2024-01-15T17:42:10.634 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-15T17:42:10.634 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_rishabh@teuthology --description fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} --archive /home/teuthworker/archive/rishabh-2024-01-15_17:34:44-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517095 --verbose --name rishabh-2024-01-15_17:34:44-fs:volumes-rishabh-clone-stats-testing-default-smithi -- /home/teuthworker/archive/rishabh-2024-01-15_17:34:44-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517095/orig.config.yaml 2024-01-15T17:42:10.674 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-15T17:42:10.748 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-01-15_17:34:44-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517095 branch: rishabh-clone-stats description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} email: null first_in_suite: false job_id: '7517095' kernel: flavor: default kdb: true sha1: 2668949e0971849893a3fb7f177ce9032fc65780 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2024-01-15_17:34:44-fs:volumes-rishabh-clone-stats-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: rishabh-clone-stats ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 95c90f413c4bb65e3e56909855f2308fd9bda05b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 95c90f413c4bb65e3e56909855f2308fd9bda05b thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: rishabh-clone-stats sha1: 95c90f413c4bb65e3e56909855f2308fd9bda05b owner: scheduled_rishabh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 1760 sha1: 95c90f413c4bb65e3e56909855f2308fd9bda05b sleep_before_teardown: 0 suite: fs:volumes suite_branch: rishabh-clone-stats suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 95c90f413c4bb65e3e56909855f2308fd9bda05b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi027.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAI2qU5KCRsR9Mi+3w/gmSQ0CY1fK8zX3xasJHG5TMacJFxEA5yy5YXPW72BFKwHcZiTD1GhG0NIszB++jnF8Zg= smithi093.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDogDNdQ6bvThoHNs52dfNHRHe7uqvJnSoKggoh25Kct9v5aHtzSGiaGbpXRFTsuZc7w2kiZGssL/Ml4pOFOSnw= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes.TestCloneProgress teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-15_17:34:44 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-15T17:42:10.748 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa; will attempt to use it 2024-01-15T17:42:10.751 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks 2024-01-15T17:42:10.751 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-15T17:42:10.753 INFO:teuthology.task.internal:Checking packages... 2024-01-15T17:42:10.773 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '95c90f413c4bb65e3e56909855f2308fd9bda05b' 2024-01-15T17:42:10.773 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-15T17:42:10.773 INFO:teuthology.packaging:ref: None 2024-01-15T17:42:10.773 INFO:teuthology.packaging:tag: None 2024-01-15T17:42:10.773 INFO:teuthology.packaging:branch: rishabh-clone-stats 2024-01-15T17:42:10.773 INFO:teuthology.packaging:sha1: 95c90f413c4bb65e3e56909855f2308fd9bda05b 2024-01-15T17:42:10.774 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=rishabh-clone-stats 2024-01-15T17:42:10.919 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7629-g95c90f41-1jammy 2024-01-15T17:42:10.920 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-15T17:42:10.926 INFO:teuthology.task.internal:no buildpackages task found 2024-01-15T17:42:10.926 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-15T17:42:10.931 INFO:teuthology.task.internal:Saving configuration 2024-01-15T17:42:10.946 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-15T17:42:10.951 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-15T17:42:10.971 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-15_17:34:44-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517095', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-15 17:35:48.121859', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEGoMT3QJQSd4mZckAxJ3LfOD3xKa+a8oHOEDOMO2EHb'} 2024-01-15T17:42:10.993 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi093.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-15_17:34:44-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517095', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-15 17:35:48.124229', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAKnYf5v9mtupWNgXUl5I54Q3Vgo5N5guMZn6LwwdC+m'} 2024-01-15T17:42:10.993 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-15T17:42:10.999 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-01-15T17:42:10.999 INFO:teuthology.task.internal:roles: ubuntu@smithi093.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-01-15T17:42:10.999 INFO:teuthology.run_tasks:Running task console_log... 2024-01-15T17:42:11.066 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f69b30e3550>, signals=[15]) 2024-01-15T17:42:11.066 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-15T17:42:11.074 INFO:teuthology.task.internal:Opening connections... 2024-01-15T17:42:11.074 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T17:42:11.076 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:42:11.159 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T17:42:11.160 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi093.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:42:11.235 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-15T17:42:11.241 DEBUG:teuthology.orchestra.run.smithi027:> uname -m 2024-01-15T17:42:11.246 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64 2024-01-15T17:42:11.246 DEBUG:teuthology.orchestra.run.smithi027:> cat /etc/os-release 2024-01-15T17:42:11.299 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-01-15T17:42:11.299 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="Ubuntu" 2024-01-15T17:42:11.299 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="22.04" 2024-01-15T17:42:11.299 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-01-15T17:42:11.299 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_CODENAME=jammy 2024-01-15T17:42:11.299 INFO:teuthology.orchestra.run.smithi027.stdout:ID=ubuntu 2024-01-15T17:42:11.299 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE=debian 2024-01-15T17:42:11.299 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://www.ubuntu.com/" 2024-01-15T17:42:11.299 INFO:teuthology.orchestra.run.smithi027.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-01-15T17:42:11.299 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-01-15T17:42:11.300 INFO:teuthology.orchestra.run.smithi027.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-01-15T17:42:11.300 INFO:teuthology.orchestra.run.smithi027.stdout:UBUNTU_CODENAME=jammy 2024-01-15T17:42:11.300 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2024-01-15T17:42:11.325 DEBUG:teuthology.orchestra.run.smithi093:> uname -m 2024-01-15T17:42:11.331 INFO:teuthology.orchestra.run.smithi093.stdout:x86_64 2024-01-15T17:42:11.331 DEBUG:teuthology.orchestra.run.smithi093:> cat /etc/os-release 2024-01-15T17:42:11.378 INFO:teuthology.orchestra.run.smithi093.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-01-15T17:42:11.378 INFO:teuthology.orchestra.run.smithi093.stdout:NAME="Ubuntu" 2024-01-15T17:42:11.379 INFO:teuthology.orchestra.run.smithi093.stdout:VERSION_ID="22.04" 2024-01-15T17:42:11.379 INFO:teuthology.orchestra.run.smithi093.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-01-15T17:42:11.379 INFO:teuthology.orchestra.run.smithi093.stdout:VERSION_CODENAME=jammy 2024-01-15T17:42:11.379 INFO:teuthology.orchestra.run.smithi093.stdout:ID=ubuntu 2024-01-15T17:42:11.379 INFO:teuthology.orchestra.run.smithi093.stdout:ID_LIKE=debian 2024-01-15T17:42:11.379 INFO:teuthology.orchestra.run.smithi093.stdout:HOME_URL="https://www.ubuntu.com/" 2024-01-15T17:42:11.379 INFO:teuthology.orchestra.run.smithi093.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-01-15T17:42:11.379 INFO:teuthology.orchestra.run.smithi093.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-01-15T17:42:11.379 INFO:teuthology.orchestra.run.smithi093.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-01-15T17:42:11.379 INFO:teuthology.orchestra.run.smithi093.stdout:UBUNTU_CODENAME=jammy 2024-01-15T17:42:11.379 INFO:teuthology.lock.ops:Updating smithi093.front.sepia.ceph.com on lock server 2024-01-15T17:42:11.400 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-15T17:42:11.408 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-15T17:42:11.413 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-15T17:42:11.413 DEBUG:teuthology.orchestra.run.smithi027:> test '!' -e /home/ubuntu/cephtest 2024-01-15T17:42:11.415 DEBUG:teuthology.orchestra.run.smithi093:> test '!' -e /home/ubuntu/cephtest 2024-01-15T17:42:11.426 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-15T17:42:11.431 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-15T17:42:11.432 DEBUG:teuthology.orchestra.run.smithi027:> test -z $(ls -A /var/lib/ceph) 2024-01-15T17:42:11.460 DEBUG:teuthology.orchestra.run.smithi093:> test -z $(ls -A /var/lib/ceph) 2024-01-15T17:42:11.477 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-15T17:42:11.527 INFO:teuthology.run_tasks:Running task kernel... 2024-01-15T17:42:11.540 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'} 2024-01-15T17:42:11.540 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}, 'mon.b': {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}}, timeout 300 2024-01-15T17:42:11.541 DEBUG:teuthology.orchestra.run.smithi027:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-15T17:42:11.541 DEBUG:teuthology.orchestra.run.smithi093:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-15T17:42:11.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T17:42:11.546 DEBUG:teuthology.task.kernel:sha1 for mon.a is 2668949e0971849893a3fb7f177ce9032fc65780 2024-01-15T17:42:11.546 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "2668949e0971849893a3fb7f177ce9032fc65780"... 2024-01-15T17:42:11.546 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-01-15T17:42:11.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T17:42:11.547 DEBUG:teuthology.task.kernel:sha1 for mon.b is 2668949e0971849893a3fb7f177ce9032fc65780 2024-01-15T17:42:11.547 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "2668949e0971849893a3fb7f177ce9032fc65780"... 2024-01-15T17:42:11.547 DEBUG:teuthology.orchestra.run.smithi093:> uname -r 2024-01-15T17:42:11.595 INFO:teuthology.orchestra.run.smithi093.stdout:5.15.0-56-generic 2024-01-15T17:42:11.595 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 2668949e0971849893a3fb7f177ce9032fc65780 2024-01-15T17:42:11.595 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-01-15T17:42:11.595 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=2668949e0971849893a3fb7f177ce9032fc65780 2024-01-15T17:42:11.598 INFO:teuthology.orchestra.run.smithi027.stdout:5.15.0-56-generic 2024-01-15T17:42:11.599 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 2668949e0971849893a3fb7f177ce9032fc65780 2024-01-15T17:42:11.599 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-01-15T17:42:11.599 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=2668949e0971849893a3fb7f177ce9032fc65780 2024-01-15T17:42:11.741 INFO:teuthology.task.kernel:Installing linux-firmware on mon.a... 2024-01-15T17:42:11.741 DEBUG:teuthology.orchestra.run.smithi027:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2024-01-15T17:42:11.757 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2024-01-15T17:42:11.916 INFO:teuthology.orchestra.run.smithi027.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2024-01-15T17:42:11.916 INFO:teuthology.orchestra.run.smithi027.stderr:hint: is subject to change. To configure the initial branch name to use in all 2024-01-15T17:42:11.917 INFO:teuthology.orchestra.run.smithi027.stderr:hint: of your new repositories, which will suppress this warning, call: 2024-01-15T17:42:11.917 INFO:teuthology.orchestra.run.smithi027.stderr:hint: 2024-01-15T17:42:11.917 INFO:teuthology.orchestra.run.smithi027.stderr:hint: git config --global init.defaultBranch 2024-01-15T17:42:11.917 INFO:teuthology.orchestra.run.smithi027.stderr:hint: 2024-01-15T17:42:11.917 INFO:teuthology.orchestra.run.smithi027.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-01-15T17:42:11.917 INFO:teuthology.orchestra.run.smithi027.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2024-01-15T17:42:11.917 INFO:teuthology.orchestra.run.smithi027.stderr:hint: 2024-01-15T17:42:11.917 INFO:teuthology.orchestra.run.smithi027.stderr:hint: git branch -m 2024-01-15T17:42:11.918 INFO:teuthology.orchestra.run.smithi027.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2024-01-15T17:42:11.921 DEBUG:teuthology.orchestra.run.smithi027:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2024-01-15T17:42:11.988 INFO:teuthology.task.kernel:Installing linux-firmware on mon.b... 2024-01-15T17:42:11.988 DEBUG:teuthology.orchestra.run.smithi093:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2024-01-15T17:42:11.992 DEBUG:teuthology.orchestra.run.smithi027:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2024-01-15T17:42:12.006 INFO:teuthology.orchestra.run.smithi093.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2024-01-15T17:42:12.048 DEBUG:teuthology.orchestra.run.smithi027:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2024-01-15T17:42:12.153 INFO:teuthology.orchestra.run.smithi093.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2024-01-15T17:42:12.153 INFO:teuthology.orchestra.run.smithi093.stderr:hint: is subject to change. To configure the initial branch name to use in all 2024-01-15T17:42:12.153 INFO:teuthology.orchestra.run.smithi093.stderr:hint: of your new repositories, which will suppress this warning, call: 2024-01-15T17:42:12.153 INFO:teuthology.orchestra.run.smithi093.stderr:hint: 2024-01-15T17:42:12.153 INFO:teuthology.orchestra.run.smithi093.stderr:hint: git config --global init.defaultBranch 2024-01-15T17:42:12.153 INFO:teuthology.orchestra.run.smithi093.stderr:hint: 2024-01-15T17:42:12.153 INFO:teuthology.orchestra.run.smithi093.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-01-15T17:42:12.153 INFO:teuthology.orchestra.run.smithi093.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2024-01-15T17:42:12.153 INFO:teuthology.orchestra.run.smithi093.stderr:hint: 2024-01-15T17:42:12.153 INFO:teuthology.orchestra.run.smithi093.stderr:hint: git branch -m 2024-01-15T17:42:12.155 INFO:teuthology.orchestra.run.smithi093.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2024-01-15T17:42:12.157 DEBUG:teuthology.orchestra.run.smithi093:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2024-01-15T17:42:12.222 DEBUG:teuthology.orchestra.run.smithi093:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2024-01-15T17:42:12.280 DEBUG:teuthology.orchestra.run.smithi093:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2024-01-15T17:49:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2024-01-15T17:49:48.422 INFO:teuthology.orchestra.run.smithi027.stderr: * [new branch] main -> origin/main 2024-01-15T17:49:48.422 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20240115 -> 20240115 2024-01-15T17:49:48.455 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20190312 -> 20190312 2024-01-15T17:49:48.455 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20190416 -> 20190416 2024-01-15T17:49:48.455 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20190514 -> 20190514 2024-01-15T17:49:48.456 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20190618 -> 20190618 2024-01-15T17:49:48.456 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20190717 -> 20190717 2024-01-15T17:49:48.456 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20190815 -> 20190815 2024-01-15T17:49:48.456 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20190923 -> 20190923 2024-01-15T17:49:48.456 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20191022 -> 20191022 2024-01-15T17:49:48.456 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20191215 -> 20191215 2024-01-15T17:49:48.456 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20200122 -> 20200122 2024-01-15T17:49:48.456 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20200316 -> 20200316 2024-01-15T17:49:48.456 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20200421 -> 20200421 2024-01-15T17:49:48.456 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20200519 -> 20200519 2024-01-15T17:49:48.457 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20200619 -> 20200619 2024-01-15T17:49:48.457 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20200721 -> 20200721 2024-01-15T17:49:48.457 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20200817 -> 20200817 2024-01-15T17:49:48.457 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20200918 -> 20200918 2024-01-15T17:49:48.457 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20201022 -> 20201022 2024-01-15T17:49:48.461 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20201118 -> 20201118 2024-01-15T17:49:48.461 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20201218 -> 20201218 2024-01-15T17:49:48.461 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20210208 -> 20210208 2024-01-15T17:49:48.461 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20210315 -> 20210315 2024-01-15T17:49:48.462 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20210511 -> 20210511 2024-01-15T17:49:48.462 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20210716 -> 20210716 2024-01-15T17:49:48.462 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20210818 -> 20210818 2024-01-15T17:49:48.462 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20210919 -> 20210919 2024-01-15T17:49:48.462 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20211027 -> 20211027 2024-01-15T17:49:48.462 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20211216 -> 20211216 2024-01-15T17:49:48.462 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20220209 -> 20220209 2024-01-15T17:49:48.462 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20220310 -> 20220310 2024-01-15T17:49:48.462 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20220411 -> 20220411 2024-01-15T17:49:48.462 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20220509 -> 20220509 2024-01-15T17:49:48.462 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20220610 -> 20220610 2024-01-15T17:49:48.463 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20220708 -> 20220708 2024-01-15T17:49:48.463 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20220815 -> 20220815 2024-01-15T17:49:48.463 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20220913 -> 20220913 2024-01-15T17:49:48.463 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20221012 -> 20221012 2024-01-15T17:49:48.463 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20221109 -> 20221109 2024-01-15T17:49:48.463 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20221214 -> 20221214 2024-01-15T17:49:48.463 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20230117 -> 20230117 2024-01-15T17:49:48.463 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20230210 -> 20230210 2024-01-15T17:49:48.463 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20230310 -> 20230310 2024-01-15T17:49:48.463 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20230404 -> 20230404 2024-01-15T17:49:48.463 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20230515 -> 20230515 2024-01-15T17:49:48.464 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20230625 -> 20230625 2024-01-15T17:49:48.464 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20230804 -> 20230804 2024-01-15T17:49:48.464 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20230919 -> 20230919 2024-01-15T17:49:48.464 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20231030 -> 20231030 2024-01-15T17:49:48.464 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20231111 -> 20231111 2024-01-15T17:49:48.464 INFO:teuthology.orchestra.run.smithi027.stderr: * [new tag] 20231211 -> 20231211 2024-01-15T17:49:52.774 INFO:teuthology.orchestra.run.smithi027.stderr:Updating files: 47% (1626/3424) Updating files: 48% (1644/3424) Updating files: 49% (1678/3424) Updating files: 50% (1712/3424) Updating files: 51% (1747/3424) Updating files: 52% (1781/3424) Updating files: 53% (1815/3424) Updating files: 54% (1849/3424) Updating files: 55% (1884/3424) Updating files: 56% (1918/3424) Updating files: 57% (1952/3424) Updating files: 58% (1986/3424) Updating files: 59% (2021/3424) Updating files: 60% (2055/3424) Updating files: 61% (2089/3424) Updating files: 62% (2123/3424) Updating files: 63% (2158/3424) Updating files: 63% (2159/3424) Updating files: 64% (2192/3424) Updating files: 65% (2226/3424) Updating files: 66% (2260/3424) Updating files: 67% (2295/3424) Updating files: 68% (2329/3424) Updating files: 69% (2363/3424) Updating files: 70% (2397/3424) Updating files: 71% (2432/3424) Updating files: 72% (2466/3424) Updating files: 73% (2500/3424) Updating files: 74% (2534/3424) Updating files: 75% (2568/3424) Updating files: 76% (2603/3424) Updating files: 76% (2616/3424) Updating files: 77% (2637/3424) Updating files: 78% (2671/3424) Updating files: 79% (2705/3424) Updating files: 80% (2740/3424) Updating files: 81% (2774/3424) Updating files: 82% (2808/3424) Updating files: 83% (2842/3424) Updating files: 83% (2856/3424) Updating files: 84% (2877/3424) Updating files: 85% (2911/3424) Updating files: 86% (2945/3424) Updating files: 87% (2979/3424) Updating files: 88% (3014/3424) Updating files: 89% (3048/3424) Updating files: 90% (3082/3424) Updating files: 91% (3116/3424) Updating files: 92% (3151/3424) Updating files: 93% (3185/3424) Updating files: 94% (3219/3424) Updating files: 95% (3253/3424) Updating files: 96% (3288/3424) Updating files: 97% (3322/3424) Updating files: 98% (3356/3424) Updating files: 99% (3390/3424) Updating files: 100% (3424/3424) Updating files: 100% (3424/3424), done. 2024-01-15T17:49:52.779 INFO:teuthology.orchestra.run.smithi027.stdout:HEAD is now at 9b6d0b08 Merge branch 'cirrus' into 'main' 2024-01-15T17:49:52.802 INFO:teuthology.task.kernel:Downloading kernel 2668949e0971849893a3fb7f177ce9032fc65780 on mon.a... 2024-01-15T17:49:52.802 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=2668949e0971849893a3fb7f177ce9032fc65780 2024-01-15T17:49:52.945 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/ubuntu/jammy/flavors/default/pool/main/l/linux-6.7.0-g2668949e0971/ 2024-01-15T17:49:52.945 DEBUG:teuthology.orchestra.run.smithi027:> rm -f /tmp/linux-image.deb && echo linux-image-6.7.0-g2668949e0971_6.7.0-g2668949e0971-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/ubuntu/jammy/flavors/default/pool/main/l/linux-6.7.0-g2668949e0971/ --input-file=- 2024-01-15T17:49:52.949 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi027.front.sepia.ceph.com to complete... 2024-01-15T17:49:58.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15 17:49:58 URL:https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/ubuntu/jammy/flavors/default/pool/main/l/linux-6.7.0-g2668949e0971/linux-image-6.7.0-g2668949e0971_6.7.0-g2668949e0971-1_amd64.deb [39096058/39096058] -> "/tmp/linux-image.deb" [1] 2024-01-15T17:49:58.977 INFO:teuthology.orchestra.run.smithi027.stderr:FINISHED --2024-01-15 17:49:58-- 2024-01-15T17:49:58.977 INFO:teuthology.orchestra.run.smithi027.stderr:Total wall clock time: 6.0s 2024-01-15T17:49:58.977 INFO:teuthology.orchestra.run.smithi027.stderr:Downloaded: 1 files, 37M in 5.6s (6.65 MB/s) 2024-01-15T17:49:58.981 INFO:teuthology.task.kernel:Installing kernel 2668949e0971849893a3fb7f177ce9032fc65780 on mon.a... 2024-01-15T17:49:58.982 DEBUG:teuthology.orchestra.run.smithi027:> sudo dpkg -i /tmp/linux-image.deb 2024-01-15T17:49:59.238 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package linux-image-6.7.0-g2668949e0971. 2024-01-15T17:49:59.814 INFO:teuthology.orchestra.run.smithi093.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2024-01-15T17:49:59.815 INFO:teuthology.orchestra.run.smithi093.stderr: * [new branch] main -> origin/main 2024-01-15T17:49:59.815 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20240115 -> 20240115 2024-01-15T17:49:59.846 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20190312 -> 20190312 2024-01-15T17:49:59.846 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20190416 -> 20190416 2024-01-15T17:49:59.846 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20190514 -> 20190514 2024-01-15T17:49:59.846 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20190618 -> 20190618 2024-01-15T17:49:59.847 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20190717 -> 20190717 2024-01-15T17:49:59.847 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20190815 -> 20190815 2024-01-15T17:49:59.847 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20190923 -> 20190923 2024-01-15T17:49:59.847 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20191022 -> 20191022 2024-01-15T17:49:59.847 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20191215 -> 20191215 2024-01-15T17:49:59.847 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20200122 -> 20200122 2024-01-15T17:49:59.847 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20200316 -> 20200316 2024-01-15T17:49:59.850 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20200421 -> 20200421 2024-01-15T17:49:59.850 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20200519 -> 20200519 2024-01-15T17:49:59.850 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20200619 -> 20200619 2024-01-15T17:49:59.851 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20200721 -> 20200721 2024-01-15T17:49:59.851 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20200817 -> 20200817 2024-01-15T17:49:59.851 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20200918 -> 20200918 2024-01-15T17:49:59.851 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20201022 -> 20201022 2024-01-15T17:49:59.851 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20201118 -> 20201118 2024-01-15T17:49:59.851 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20201218 -> 20201218 2024-01-15T17:49:59.851 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20210208 -> 20210208 2024-01-15T17:49:59.851 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20210315 -> 20210315 2024-01-15T17:49:59.852 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20210511 -> 20210511 2024-01-15T17:49:59.852 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20210716 -> 20210716 2024-01-15T17:49:59.852 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20210818 -> 20210818 2024-01-15T17:49:59.852 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20210919 -> 20210919 2024-01-15T17:49:59.852 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20211027 -> 20211027 2024-01-15T17:49:59.852 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20211216 -> 20211216 2024-01-15T17:49:59.852 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20220209 -> 20220209 2024-01-15T17:49:59.852 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20220310 -> 20220310 2024-01-15T17:49:59.852 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20220411 -> 20220411 2024-01-15T17:49:59.852 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20220509 -> 20220509 2024-01-15T17:49:59.852 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20220610 -> 20220610 2024-01-15T17:49:59.852 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20220708 -> 20220708 2024-01-15T17:49:59.853 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20220815 -> 20220815 2024-01-15T17:49:59.853 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20220913 -> 20220913 2024-01-15T17:49:59.853 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20221012 -> 20221012 2024-01-15T17:49:59.853 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20221109 -> 20221109 2024-01-15T17:49:59.853 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20221214 -> 20221214 2024-01-15T17:49:59.855 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20230117 -> 20230117 2024-01-15T17:49:59.855 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20230210 -> 20230210 2024-01-15T17:49:59.855 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20230310 -> 20230310 2024-01-15T17:49:59.855 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20230404 -> 20230404 2024-01-15T17:49:59.855 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20230515 -> 20230515 2024-01-15T17:49:59.855 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20230625 -> 20230625 2024-01-15T17:49:59.855 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20230804 -> 20230804 2024-01-15T17:49:59.855 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20230919 -> 20230919 2024-01-15T17:49:59.856 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20231030 -> 20231030 2024-01-15T17:49:59.856 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20231111 -> 20231111 2024-01-15T17:49:59.856 INFO:teuthology.orchestra.run.smithi093.stderr: * [new tag] 20231211 -> 20231211 2024-01-15T17:50:01.268 INFO:teuthology.orchestra.run.smithi027.stdout:(Reading database ... 129542 files and directories currently installed.) 2024-01-15T17:50:01.272 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack /tmp/linux-image.deb ... 2024-01-15T17:50:01.416 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking linux-image-6.7.0-g2668949e0971 (6.7.0-g2668949e0971-1) ... 2024-01-15T17:50:03.932 INFO:teuthology.orchestra.run.smithi093.stderr:Updating files: 49% (1701/3424) Updating files: 50% (1712/3424) Updating files: 51% (1747/3424) Updating files: 52% (1781/3424) Updating files: 53% (1815/3424) Updating files: 54% (1849/3424) Updating files: 55% (1884/3424) Updating files: 56% (1918/3424) Updating files: 57% (1952/3424) Updating files: 58% (1986/3424) Updating files: 59% (2021/3424) Updating files: 60% (2055/3424) Updating files: 61% (2089/3424) Updating files: 62% (2123/3424) Updating files: 63% (2158/3424) Updating files: 64% (2192/3424) Updating files: 64% (2215/3424) Updating files: 65% (2226/3424) Updating files: 66% (2260/3424) Updating files: 67% (2295/3424) Updating files: 68% (2329/3424) Updating files: 69% (2363/3424) Updating files: 70% (2397/3424) Updating files: 71% (2432/3424) Updating files: 72% (2466/3424) Updating files: 73% (2500/3424) Updating files: 74% (2534/3424) Updating files: 75% (2568/3424) Updating files: 76% (2603/3424) Updating files: 77% (2637/3424) Updating files: 78% (2671/3424) Updating files: 79% (2705/3424) Updating files: 80% (2740/3424) Updating files: 81% (2774/3424) Updating files: 81% (2783/3424) Updating files: 82% (2808/3424) Updating files: 83% (2842/3424) Updating files: 84% (2877/3424) Updating files: 85% (2911/3424) Updating files: 86% (2945/3424) Updating files: 87% (2979/3424) Updating files: 88% (3014/3424) Updating files: 89% (3048/3424) Updating files: 90% (3082/3424) Updating files: 91% (3116/3424) Updating files: 92% (3151/3424) Updating files: 93% (3185/3424) Updating files: 94% (3219/3424) Updating files: 95% (3253/3424) Updating files: 96% (3288/3424) Updating files: 96% (3304/3424) Updating files: 97% (3322/3424) Updating files: 98% (3356/3424) Updating files: 99% (3390/3424) Updating files: 100% (3424/3424) Updating files: 100% (3424/3424), done. 2024-01-15T17:50:03.937 INFO:teuthology.orchestra.run.smithi093.stdout:HEAD is now at 9b6d0b08 Merge branch 'cirrus' into 'main' 2024-01-15T17:50:03.957 INFO:teuthology.task.kernel:Downloading kernel 2668949e0971849893a3fb7f177ce9032fc65780 on mon.b... 2024-01-15T17:50:03.958 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=2668949e0971849893a3fb7f177ce9032fc65780 2024-01-15T17:50:04.094 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/ubuntu/jammy/flavors/default/pool/main/l/linux-6.7.0-g2668949e0971/ 2024-01-15T17:50:04.095 DEBUG:teuthology.orchestra.run.smithi093:> rm -f /tmp/linux-image.deb && echo linux-image-6.7.0-g2668949e0971_6.7.0-g2668949e0971-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/ubuntu/jammy/flavors/default/pool/main/l/linux-6.7.0-g2668949e0971/ --input-file=- 2024-01-15T17:50:04.097 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi093.front.sepia.ceph.com to complete... 2024-01-15T17:50:05.038 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up linux-image-6.7.0-g2668949e0971 (6.7.0-g2668949e0971-1) ... 2024-01-15T17:50:05.144 INFO:teuthology.orchestra.run.smithi027.stderr:update-initramfs: Generating /boot/initrd.img-6.7.0-g2668949e0971 2024-01-15T17:50:05.602 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15 17:50:05 URL:https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/ubuntu/jammy/flavors/default/pool/main/l/linux-6.7.0-g2668949e0971/linux-image-6.7.0-g2668949e0971_6.7.0-g2668949e0971-1_amd64.deb [39096058/39096058] -> "/tmp/linux-image.deb" [1] 2024-01-15T17:50:05.602 INFO:teuthology.orchestra.run.smithi093.stderr:FINISHED --2024-01-15 17:50:05-- 2024-01-15T17:50:05.602 INFO:teuthology.orchestra.run.smithi093.stderr:Total wall clock time: 1.5s 2024-01-15T17:50:05.602 INFO:teuthology.orchestra.run.smithi093.stderr:Downloaded: 1 files, 37M in 1.4s (27.5 MB/s) 2024-01-15T17:50:05.607 INFO:teuthology.task.kernel:Installing kernel 2668949e0971849893a3fb7f177ce9032fc65780 on mon.b... 2024-01-15T17:50:05.607 DEBUG:teuthology.orchestra.run.smithi093:> sudo dpkg -i /tmp/linux-image.deb 2024-01-15T17:50:05.839 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package linux-image-6.7.0-g2668949e0971. 2024-01-15T17:50:07.756 INFO:teuthology.orchestra.run.smithi093.stdout:(Reading database ... 129542 files and directories currently installed.) 2024-01-15T17:50:07.760 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack /tmp/linux-image.deb ... 2024-01-15T17:50:07.887 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking linux-image-6.7.0-g2668949e0971 (6.7.0-g2668949e0971-1) ... 2024-01-15T17:50:11.137 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up linux-image-6.7.0-g2668949e0971 (6.7.0-g2668949e0971-1) ... 2024-01-15T17:50:11.246 INFO:teuthology.orchestra.run.smithi093.stderr:update-initramfs: Generating /boot/initrd.img-6.7.0-g2668949e0971 2024-01-15T17:50:26.422 INFO:teuthology.orchestra.run.smithi027.stderr:Sourcing file `/etc/default/grub' 2024-01-15T17:50:26.436 INFO:teuthology.orchestra.run.smithi027.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-01-15T17:50:26.470 INFO:teuthology.orchestra.run.smithi027.stderr:Generating grub configuration file ... 2024-01-15T17:50:26.838 INFO:teuthology.orchestra.run.smithi027.stderr:Found linux image: /boot/vmlinuz-6.7.0-g2668949e0971 2024-01-15T17:50:26.857 INFO:teuthology.orchestra.run.smithi027.stderr:Found initrd image: /boot/initrd.img-6.7.0-g2668949e0971 2024-01-15T17:50:27.157 INFO:teuthology.orchestra.run.smithi027.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-01-15T17:50:27.170 INFO:teuthology.orchestra.run.smithi027.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-01-15T17:50:27.234 INFO:teuthology.orchestra.run.smithi027.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-01-15T17:50:27.240 INFO:teuthology.orchestra.run.smithi027.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-01-15T17:50:27.391 INFO:teuthology.orchestra.run.smithi027.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-01-15T17:50:27.391 INFO:teuthology.orchestra.run.smithi027.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-01-15T17:50:27.391 INFO:teuthology.orchestra.run.smithi027.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-01-15T17:50:27.423 INFO:teuthology.orchestra.run.smithi027.stderr:done 2024-01-15T17:50:27.638 DEBUG:teuthology.orchestra.run.smithi027:> dpkg-deb -c /tmp/linux-image.deb 2024-01-15T17:50:27.646 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./ 2024-01-15T17:50:27.646 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./boot/ 2024-01-15T17:50:27.646 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 4444641 2024-01-15 16:53 ./boot/System.map-6.7.0-g2668949e0971 2024-01-15T17:50:27.659 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 170149 2024-01-15 16:53 ./boot/config-6.7.0-g2668949e0971 2024-01-15T17:50:27.660 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11441152 2024-01-15 16:53 ./boot/vmlinuz-6.7.0-g2668949e0971 2024-01-15T17:50:27.681 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./etc/ 2024-01-15T17:50:27.681 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./etc/kernel/ 2024-01-15T17:50:27.681 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./etc/kernel/postinst.d/ 2024-01-15T17:50:27.681 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./etc/kernel/postrm.d/ 2024-01-15T17:50:27.681 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./etc/kernel/preinst.d/ 2024-01-15T17:50:27.681 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./etc/kernel/prerm.d/ 2024-01-15T17:50:27.681 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/ 2024-01-15T17:50:27.681 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/ 2024-01-15T17:50:27.681 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/ 2024-01-15T17:50:27.682 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/ 2024-01-15T17:50:27.682 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/ 2024-01-15T17:50:27.682 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/ 2024-01-15T17:50:27.682 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/ 2024-01-15T17:50:27.682 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 490345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/aesni-intel.ko 2024-01-15T17:50:27.684 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 25329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-01-15T17:50:27.684 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 29529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-01-15T17:50:27.685 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-01-15T17:50:27.685 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 93113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-01-15T17:50:27.685 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 106729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-01-15T17:50:27.685 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 152241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-01-15T17:50:27.685 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12553 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-01-15T17:50:27.685 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-01-15T17:50:27.685 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 83889 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-01-15T17:50:27.685 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-01-15T17:50:27.685 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 47225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-01-15T17:50:27.685 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 47209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-avx2.ko 2024-01-15T17:50:27.685 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 51801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-01-15T17:50:27.685 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 39585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-01-15T17:50:27.685 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 42529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-01-15T17:50:27.686 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 58585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-01-15T17:50:27.686 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 50497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-01-15T17:50:27.686 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 32065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-01-15T17:50:27.686 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-01-15T17:50:27.686 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/ 2024-01-15T17:50:27.686 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/ 2024-01-15T17:50:27.686 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/mce/ 2024-01-15T17:50:27.686 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 40113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-01-15T17:50:27.686 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpuid.ko 2024-01-15T17:50:27.686 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/msr.ko 2024-01-15T17:50:27.686 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/ 2024-01-15T17:50:27.686 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 353705 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm-amd.ko 2024-01-15T17:50:27.687 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 881649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm-intel.ko 2024-01-15T17:50:27.687 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 2276625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm.ko 2024-01-15T17:50:27.703 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/ 2024-01-15T17:50:27.703 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 53721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/af_alg.ko 2024-01-15T17:50:27.703 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23513 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/algif_hash.ko 2024-01-15T17:50:27.703 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/algif_skcipher.ko 2024-01-15T17:50:27.704 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/ansi_cprng.ko 2024-01-15T17:50:27.704 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19641 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/anubis.ko 2024-01-15T17:50:27.704 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/arc4.ko 2024-01-15T17:50:27.704 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/asymmetric_keys/ 2024-01-15T17:50:27.704 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2024-01-15T17:50:27.704 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/ 2024-01-15T17:50:27.704 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_memcpy.ko 2024-01-15T17:50:27.704 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_pq.ko 2024-01-15T17:50:27.704 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21561 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_raid6_recov.ko 2024-01-15T17:50:27.704 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_tx.ko 2024-01-15T17:50:27.704 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_xor.ko 2024-01-15T17:50:27.704 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/raid6test.ko 2024-01-15T17:50:27.704 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/authenc.ko 2024-01-15T17:50:27.705 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/authencesn.ko 2024-01-15T17:50:27.705 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/blake2b_generic.ko 2024-01-15T17:50:27.705 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/blowfish_common.ko 2024-01-15T17:50:27.705 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/blowfish_generic.ko 2024-01-15T17:50:27.705 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 38809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/camellia_generic.ko 2024-01-15T17:50:27.705 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 27321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast5_generic.ko 2024-01-15T17:50:27.705 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast6_generic.ko 2024-01-15T17:50:27.705 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast_common.ko 2024-01-15T17:50:27.705 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 35273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/ccm.ko 2024-01-15T17:50:27.705 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/cmac.ko 2024-01-15T17:50:27.705 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/crc32_generic.ko 2024-01-15T17:50:27.705 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 47769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/cryptd.ko 2024-01-15T17:50:27.705 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 39505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_engine.ko 2024-01-15T17:50:27.705 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_simd.ko 2024-01-15T17:50:27.706 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 25041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_user.ko 2024-01-15T17:50:27.706 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/cts.ko 2024-01-15T17:50:27.706 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/des_generic.ko 2024-01-15T17:50:27.706 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/dh_generic.ko 2024-01-15T17:50:27.706 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11001 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/echainiv.ko 2024-01-15T17:50:27.706 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 25321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/essiv.ko 2024-01-15T17:50:27.706 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/fcrypt.ko 2024-01-15T17:50:27.706 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26897 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/khazad.ko 2024-01-15T17:50:27.706 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/lrw.ko 2024-01-15T17:50:27.706 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11513 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/lz4.ko 2024-01-15T17:50:27.706 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/lz4hc.ko 2024-01-15T17:50:27.706 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11697 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/md4.ko 2024-01-15T17:50:27.706 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/michael_mic.ko 2024-01-15T17:50:27.706 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/pcbc.ko 2024-01-15T17:50:27.707 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/pcrypt.ko 2024-01-15T17:50:27.707 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/rmd160.ko 2024-01-15T17:50:27.707 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 29393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/seed.ko 2024-01-15T17:50:27.707 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 27081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/serpent_generic.ko 2024-01-15T17:50:27.707 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 149481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/tcrypt.ko 2024-01-15T17:50:27.707 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/tea.ko 2024-01-15T17:50:27.707 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/twofish_common.ko 2024-01-15T17:50:27.707 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/twofish_generic.ko 2024-01-15T17:50:27.707 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/vmac.ko 2024-01-15T17:50:27.707 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 36185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/wp512.ko 2024-01-15T17:50:27.707 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/xcbc.ko 2024-01-15T17:50:27.707 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/xor.ko 2024-01-15T17:50:27.707 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/xts.ko 2024-01-15T17:50:27.708 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/xxhash_generic.ko 2024-01-15T17:50:27.708 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ 2024-01-15T17:50:27.708 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/ 2024-01-15T17:50:27.708 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_extlog.ko 2024-01-15T17:50:27.708 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_ipmi.ko 2024-01-15T17:50:27.708 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_pad.ko 2024-01-15T17:50:27.708 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/apei/ 2024-01-15T17:50:27.708 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 31241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/apei/einj.ko 2024-01-15T17:50:27.708 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/ec_sys.ko 2024-01-15T17:50:27.708 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/platform_profile.ko 2024-01-15T17:50:27.708 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/sbs.ko 2024-01-15T17:50:27.708 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/sbshc.ko 2024-01-15T17:50:27.708 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 104729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/video.ko 2024-01-15T17:50:27.708 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ 2024-01-15T17:50:27.709 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/acard-ahci.ko 2024-01-15T17:50:27.709 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 89649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ahci.ko 2024-01-15T17:50:27.709 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ahci_platform.ko 2024-01-15T17:50:27.709 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 88401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/libahci.ko 2024-01-15T17:50:27.709 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 34985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/libahci_platform.ko 2024-01-15T17:50:27.709 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_acpi.ko 2024-01-15T17:50:27.709 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ali.ko 2024-01-15T17:50:27.709 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 33505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_amd.ko 2024-01-15T17:50:27.709 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_artop.ko 2024-01-15T17:50:27.709 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_atiixp.ko 2024-01-15T17:50:27.709 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_atp867x.ko 2024-01-15T17:50:27.709 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cmd640.ko 2024-01-15T17:50:27.709 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 25185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cmd64x.ko 2024-01-15T17:50:27.709 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cypress.ko 2024-01-15T17:50:27.709 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_efar.ko 2024-01-15T17:50:27.709 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt366.ko 2024-01-15T17:50:27.710 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 33977 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt37x.ko 2024-01-15T17:50:27.710 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt3x2n.ko 2024-01-15T17:50:27.710 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt3x3.ko 2024-01-15T17:50:27.710 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_it8213.ko 2024-01-15T17:50:27.710 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 35273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_it821x.ko 2024-01-15T17:50:27.710 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12889 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_jmicron.ko 2024-01-15T17:50:27.710 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 51481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_legacy.ko 2024-01-15T17:50:27.710 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_marvell.ko 2024-01-15T17:50:27.710 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16281 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_mpiix.ko 2024-01-15T17:50:27.710 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_netcell.ko 2024-01-15T17:50:27.710 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ninja32.ko 2024-01-15T17:50:27.710 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ns87410.ko 2024-01-15T17:50:27.710 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17361 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ns87415.ko 2024-01-15T17:50:27.710 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_oldpiix.ko 2024-01-15T17:50:27.710 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15377 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_opti.ko 2024-01-15T17:50:27.711 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_optidma.ko 2024-01-15T17:50:27.711 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 32585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pcmcia.ko 2024-01-15T17:50:27.711 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 37793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pdc2027x.ko 2024-01-15T17:50:27.711 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20561 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-01-15T17:50:27.711 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_piccolo.ko 2024-01-15T17:50:27.711 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_radisys.ko 2024-01-15T17:50:27.711 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_rdc.ko 2024-01-15T17:50:27.711 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_rz1000.ko 2024-01-15T17:50:27.711 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sch.ko 2024-01-15T17:50:27.711 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_serverworks.ko 2024-01-15T17:50:27.711 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23457 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sil680.ko 2024-01-15T17:50:27.711 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19697 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sl82c105.ko 2024-01-15T17:50:27.711 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_triflex.ko 2024-01-15T17:50:27.711 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_via.ko 2024-01-15T17:50:27.711 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pdc_adma.ko 2024-01-15T17:50:27.712 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 29121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_inic162x.ko 2024-01-15T17:50:27.712 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 83345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_mv.ko 2024-01-15T17:50:27.712 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 70393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_nv.ko 2024-01-15T17:50:27.712 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 37377 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_promise.ko 2024-01-15T17:50:27.712 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_qstor.ko 2024-01-15T17:50:27.712 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 29273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sil.ko 2024-01-15T17:50:27.712 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 38337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sil24.ko 2024-01-15T17:50:27.712 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sis.ko 2024-01-15T17:50:27.712 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_svw.ko 2024-01-15T17:50:27.712 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 44793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sx4.ko 2024-01-15T17:50:27.712 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_uli.ko 2024-01-15T17:50:27.712 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 35065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_via.ko 2024-01-15T17:50:27.712 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_vsc.ko 2024-01-15T17:50:27.712 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/auxdisplay/ 2024-01-15T17:50:27.712 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/auxdisplay/ks0108.ko 2024-01-15T17:50:27.713 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/bcma/ 2024-01-15T17:50:27.713 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 124473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/bcma/bcma.ko 2024-01-15T17:50:27.713 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/ 2024-01-15T17:50:27.713 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/aoe/ 2024-01-15T17:50:27.713 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 113753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/aoe/aoe.ko 2024-01-15T17:50:27.713 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/drbd/ 2024-01-15T17:50:27.713 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 965305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/drbd/drbd.ko 2024-01-15T17:50:27.713 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 173353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/floppy.ko 2024-01-15T17:50:27.748 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/mtip32xx/ 2024-01-15T17:50:27.748 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 94297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-01-15T17:50:27.748 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 116321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/nbd.ko 2024-01-15T17:50:27.749 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/null_blk/ 2024-01-15T17:50:27.749 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 110737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/null_blk/null_blk.ko 2024-01-15T17:50:27.749 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 98281 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/pktcdvd.ko 2024-01-15T17:50:27.749 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 263769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/rbd.ko 2024-01-15T17:50:27.749 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkback/ 2024-01-15T17:50:27.749 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 98185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-01-15T17:50:27.749 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/zram/ 2024-01-15T17:50:27.749 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 60217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/zram/zram.ko 2024-01-15T17:50:27.749 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ 2024-01-15T17:50:27.749 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30705 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/applicom.ko 2024-01-15T17:50:27.749 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hangcheck-timer.ko 2024-01-15T17:50:27.749 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/ 2024-01-15T17:50:27.749 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13889 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/amd-rng.ko 2024-01-15T17:50:27.750 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/intel-rng.ko 2024-01-15T17:50:27.750 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-01-15T17:50:27.750 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/via-rng.ko 2024-01-15T17:50:27.750 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/virtio-rng.ko 2024-01-15T17:50:27.750 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ 2024-01-15T17:50:27.750 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30713 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-01-15T17:50:27.750 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 148609 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-01-15T17:50:27.750 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-01-15T17:50:27.750 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 139313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_si.ko 2024-01-15T17:50:27.750 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 70969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-01-15T17:50:27.750 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 53513 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-01-15T17:50:27.750 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 36321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/lp.ko 2024-01-15T17:50:27.750 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/mwave/ 2024-01-15T17:50:27.751 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 84097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/mwave/mwave.ko 2024-01-15T17:50:27.751 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 25873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/nvram.ko 2024-01-15T17:50:27.751 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 32169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ppdev.ko 2024-01-15T17:50:27.751 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 51113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tlclk.ko 2024-01-15T17:50:27.751 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/ 2024-01-15T17:50:27.751 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_atmel.ko 2024-01-15T17:50:27.751 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2024-01-15T17:50:27.751 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2024-01-15T17:50:27.751 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 32985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2024-01-15T17:50:27.751 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_infineon.ko 2024-01-15T17:50:27.751 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22697 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_nsc.ko 2024-01-15T17:50:27.751 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23561 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/xen-tpmfront.ko 2024-01-15T17:50:27.751 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/xillybus/ 2024-01-15T17:50:27.751 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17001 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/xillybus/xillybus_class.ko 2024-01-15T17:50:27.751 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 50945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/xillybus/xillybus_core.ko 2024-01-15T17:50:27.752 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/xillybus/xillybus_pcie.ko 2024-01-15T17:50:27.752 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/clk/ 2024-01-15T17:50:27.752 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/clk/clk-palmas.ko 2024-01-15T17:50:27.752 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 47761 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/clk/clk-si5351.ko 2024-01-15T17:50:27.752 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11457 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/clk/clk-twl6040.ko 2024-01-15T17:50:27.752 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/clk/clk-wm831x.ko 2024-01-15T17:50:27.752 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/ 2024-01-15T17:50:27.752 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-01-15T17:50:27.752 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/p4-clockmod.ko 2024-01-15T17:50:27.752 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 25297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/speedstep-lib.ko 2024-01-15T17:50:27.752 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ 2024-01-15T17:50:27.752 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ 2024-01-15T17:50:27.752 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 70737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-01-15T17:50:27.753 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 246841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ccp.ko 2024-01-15T17:50:27.753 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/ 2024-01-15T17:50:27.753 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/ 2024-01-15T17:50:27.753 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_common/ 2024-01-15T17:50:27.753 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 517033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-01-15T17:50:27.753 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2024-01-15T17:50:27.753 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-01-15T17:50:27.753 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/padlock-aes.ko 2024-01-15T17:50:27.753 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/padlock-sha.ko 2024-01-15T17:50:27.753 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/virtio/ 2024-01-15T17:50:27.753 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 70233 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-01-15T17:50:27.753 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/dca/ 2024-01-15T17:50:27.753 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 29785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/dca/dca.ko 2024-01-15T17:50:27.753 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ 2024-01-15T17:50:27.754 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/dw/ 2024-01-15T17:50:27.754 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/dw/dw_dmac.ko 2024-01-15T17:50:27.754 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ioat/ 2024-01-15T17:50:27.754 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 122569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ioat/ioatdma.ko 2024-01-15T17:50:27.754 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/ 2024-01-15T17:50:27.754 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 79953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/amd64_edac.ko 2024-01-15T17:50:27.754 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 35769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/e752x_edac.ko 2024-01-15T17:50:27.754 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 63617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/edac_mce_amd.ko 2024-01-15T17:50:27.754 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i3000_edac.ko 2024-01-15T17:50:27.754 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i3200_edac.ko 2024-01-15T17:50:27.754 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i5100_edac.ko 2024-01-15T17:50:27.754 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 25657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i5400_edac.ko 2024-01-15T17:50:27.754 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i7300_edac.ko 2024-01-15T17:50:27.754 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 52041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i7core_edac.ko 2024-01-15T17:50:27.754 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i82975x_edac.ko 2024-01-15T17:50:27.755 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/ie31200_edac.ko 2024-01-15T17:50:27.755 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 56825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/sb_edac.ko 2024-01-15T17:50:27.755 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/x38_edac.ko 2024-01-15T17:50:27.755 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/extcon/ 2024-01-15T17:50:27.755 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/extcon/extcon-gpio.ko 2024-01-15T17:50:27.755 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 27185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/extcon/extcon-max14577.ko 2024-01-15T17:50:27.755 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/extcon/extcon-max77693.ko 2024-01-15T17:50:27.755 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/extcon/extcon-max8997.ko 2024-01-15T17:50:27.755 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/extcon/extcon-palmas.ko 2024-01-15T17:50:27.755 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23713 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/extcon/extcon-rt8973a.ko 2024-01-15T17:50:27.755 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 27785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/extcon/extcon-sm5502.ko 2024-01-15T17:50:27.755 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/ 2024-01-15T17:50:27.755 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 34977 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/dmi-sysfs.ko 2024-01-15T17:50:27.755 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/efi/ 2024-01-15T17:50:27.756 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/efi/efi-pstore.ko 2024-01-15T17:50:27.756 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/iscsi_ibft.ko 2024-01-15T17:50:27.756 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/ 2024-01-15T17:50:27.756 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-adp5520.ko 2024-01-15T17:50:27.756 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20153 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-amd8111.ko 2024-01-15T17:50:27.756 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-arizona.ko 2024-01-15T17:50:27.756 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-da9052.ko 2024-01-15T17:50:27.756 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-da9055.ko 2024-01-15T17:50:27.756 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-dln2.ko 2024-01-15T17:50:27.756 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 77849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-f7188x.ko 2024-01-15T17:50:27.756 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-generic.ko 2024-01-15T17:50:27.756 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-ich.ko 2024-01-15T17:50:27.756 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-janz-ttl.ko 2024-01-15T17:50:27.756 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-kempld.ko 2024-01-15T17:50:27.757 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-lp3943.ko 2024-01-15T17:50:27.757 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-max7300.ko 2024-01-15T17:50:27.757 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-max730x.ko 2024-01-15T17:50:27.757 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-max732x.ko 2024-01-15T17:50:27.757 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 25145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-ml-ioh.ko 2024-01-15T17:50:27.757 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 47201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-pca953x.ko 2024-01-15T17:50:27.757 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-pcf857x.ko 2024-01-15T17:50:27.757 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14377 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-rdc321x.ko 2024-01-15T17:50:27.757 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-sch.ko 2024-01-15T17:50:27.757 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-sch311x.ko 2024-01-15T17:50:27.757 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-tps65912.ko 2024-01-15T17:50:27.757 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-twl4030.ko 2024-01-15T17:50:27.757 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-twl6040.ko 2024-01-15T17:50:27.757 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-viperboard.ko 2024-01-15T17:50:27.757 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15377 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-vx855.ko 2024-01-15T17:50:27.758 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16281 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-wm831x.ko 2024-01-15T17:50:27.758 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-wm8350.ko 2024-01-15T17:50:27.758 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-wm8994.ko 2024-01-15T17:50:27.758 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/ 2024-01-15T17:50:27.758 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-a4tech.ko 2024-01-15T17:50:27.758 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 39513 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-apple.ko 2024-01-15T17:50:27.758 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-appleir.ko 2024-01-15T17:50:27.758 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-aureal.ko 2024-01-15T17:50:27.758 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-axff.ko 2024-01-15T17:50:27.758 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-belkin.ko 2024-01-15T17:50:27.758 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-cherry.ko 2024-01-15T17:50:27.758 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-chicony.ko 2024-01-15T17:50:27.758 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 60761 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-cp2112.ko 2024-01-15T17:50:27.759 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-cypress.ko 2024-01-15T17:50:27.759 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-dr.ko 2024-01-15T17:50:27.759 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10697 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-elecom.ko 2024-01-15T17:50:27.759 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-elo.ko 2024-01-15T17:50:27.759 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13457 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-emsff.ko 2024-01-15T17:50:27.759 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-ezkey.ko 2024-01-15T17:50:27.759 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gaff.ko 2024-01-15T17:50:27.759 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-generic.ko 2024-01-15T17:50:27.759 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gt683r.ko 2024-01-15T17:50:27.759 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gyration.ko 2024-01-15T17:50:27.759 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10161 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtek-kbd.ko 2024-01-15T17:50:27.759 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10457 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtek-mouse.ko 2024-01-15T17:50:27.759 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtekff.ko 2024-01-15T17:50:27.759 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-hyperv.ko 2024-01-15T17:50:27.759 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-icade.ko 2024-01-15T17:50:27.759 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-kensington.ko 2024-01-15T17:50:27.760 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8545 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-keytouch.ko 2024-01-15T17:50:27.760 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-kye.ko 2024-01-15T17:50:27.760 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lcpower.ko 2024-01-15T17:50:27.760 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-led.ko 2024-01-15T17:50:27.760 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 42945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lenovo.ko 2024-01-15T17:50:27.760 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30697 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lg-g15.ko 2024-01-15T17:50:27.760 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 58073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech-dj.ko 2024-01-15T17:50:27.760 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 100649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-01-15T17:50:27.760 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 80993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech.ko 2024-01-15T17:50:27.760 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-magicmouse.ko 2024-01-15T17:50:27.760 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-microsoft.ko 2024-01-15T17:50:27.760 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10361 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-monterey.ko 2024-01-15T17:50:27.760 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 50273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-multitouch.ko 2024-01-15T17:50:27.760 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 32593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-ntrig.ko 2024-01-15T17:50:27.761 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-ortek.ko 2024-01-15T17:50:27.761 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-penmount.ko 2024-01-15T17:50:27.761 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-petalynx.ko 2024-01-15T17:50:27.761 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 72937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-picolcd.ko 2024-01-15T17:50:27.761 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-pl.ko 2024-01-15T17:50:27.761 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-plantronics.ko 2024-01-15T17:50:27.761 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8417 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-primax.ko 2024-01-15T17:50:27.761 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 35393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-rmi.ko 2024-01-15T17:50:27.761 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-arvo.ko 2024-01-15T17:50:27.761 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-common.ko 2024-01-15T17:50:27.761 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 31425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-isku.ko 2024-01-15T17:50:27.761 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 29729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-kone.ko 2024-01-15T17:50:27.761 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 31129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-01-15T17:50:27.761 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-konepure.ko 2024-01-15T17:50:27.762 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 29489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-01-15T17:50:27.762 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-lua.ko 2024-01-15T17:50:27.762 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 29313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-pyra.ko 2024-01-15T17:50:27.762 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 27937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-ryos.ko 2024-01-15T17:50:27.762 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-savu.ko 2024-01-15T17:50:27.762 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat.ko 2024-01-15T17:50:27.762 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-saitek.ko 2024-01-15T17:50:27.762 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-samsung.ko 2024-01-15T17:50:27.762 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 36817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sensor-hub.ko 2024-01-15T17:50:27.762 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sjoy.ko 2024-01-15T17:50:27.762 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 51249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sony.ko 2024-01-15T17:50:27.762 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-speedlink.ko 2024-01-15T17:50:27.762 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 25297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-steelseries.ko 2024-01-15T17:50:27.762 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sunplus.ko 2024-01-15T17:50:27.762 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-thrustmaster.ko 2024-01-15T17:50:27.763 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-tivo.ko 2024-01-15T17:50:27.763 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-tmff.ko 2024-01-15T17:50:27.763 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-topseed.ko 2024-01-15T17:50:27.763 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-twinhan.ko 2024-01-15T17:50:27.763 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 71073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-uclogic.ko 2024-01-15T17:50:27.763 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-waltop.ko 2024-01-15T17:50:27.763 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 143473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-wiimote.ko 2024-01-15T17:50:27.763 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8153 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-xinmo.ko 2024-01-15T17:50:27.763 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-zpff.ko 2024-01-15T17:50:27.763 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-zydacron.ko 2024-01-15T17:50:27.763 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 269281 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid.ko 2024-01-15T17:50:27.763 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 32321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/uhid.ko 2024-01-15T17:50:27.763 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/usbhid/ 2024-01-15T17:50:27.763 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 125785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/usbhid/usbhid.ko 2024-01-15T17:50:27.763 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/usbhid/usbkbd.ko 2024-01-15T17:50:27.764 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/usbhid/usbmouse.ko 2024-01-15T17:50:27.764 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 217873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/wacom.ko 2024-01-15T17:50:27.764 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/ 2024-01-15T17:50:27.764 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 64201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_balloon.ko 2024-01-15T17:50:27.764 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 89625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_utils.ko 2024-01-15T17:50:27.764 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 305321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_vmbus.ko 2024-01-15T17:50:27.764 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ 2024-01-15T17:50:27.764 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 66521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/abituguru.ko 2024-01-15T17:50:27.764 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 61393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/abituguru3.ko 2024-01-15T17:50:27.764 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 34673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/acpi_power_meter.ko 2024-01-15T17:50:27.764 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ad7414.ko 2024-01-15T17:50:27.764 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ad7418.ko 2024-01-15T17:50:27.764 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 25257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adc128d818.ko 2024-01-15T17:50:27.764 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 31729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1025.ko 2024-01-15T17:50:27.764 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 97977 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1026.ko 2024-01-15T17:50:27.765 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1029.ko 2024-01-15T17:50:27.765 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 45553 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1031.ko 2024-01-15T17:50:27.765 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm9240.ko 2024-01-15T17:50:27.765 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ads7828.ko 2024-01-15T17:50:27.765 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7410.ko 2024-01-15T17:50:27.765 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7411.ko 2024-01-15T17:50:27.765 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 74409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7462.ko 2024-01-15T17:50:27.765 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 39585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7470.ko 2024-01-15T17:50:27.765 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 69401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7475.ko 2024-01-15T17:50:27.765 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7x10.ko 2024-01-15T17:50:27.765 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 39073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/amc6821.ko 2024-01-15T17:50:27.765 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 55201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/applesmc.ko 2024-01-15T17:50:27.765 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 49913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asb100.ko 2024-01-15T17:50:27.765 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 58617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asc7621.ko 2024-01-15T17:50:27.765 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 40825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asus_atk0110.ko 2024-01-15T17:50:27.765 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/atxp1.ko 2024-01-15T17:50:27.766 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/coretemp.ko 2024-01-15T17:50:27.766 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/da9052-hwmon.ko 2024-01-15T17:50:27.766 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/da9055-hwmon.ko 2024-01-15T17:50:27.766 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 44441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-01-15T17:50:27.766 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 99601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/dme1737.ko 2024-01-15T17:50:27.766 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ds1621.ko 2024-01-15T17:50:27.766 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ds620.ko 2024-01-15T17:50:27.766 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 29777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc1403.ko 2024-01-15T17:50:27.766 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 33185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc2103.ko 2024-01-15T17:50:27.766 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc6w201.ko 2024-01-15T17:50:27.766 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 66985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f71805f.ko 2024-01-15T17:50:27.766 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 94689 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f71882fg.ko 2024-01-15T17:50:27.766 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 41401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f75375s.ko 2024-01-15T17:50:27.766 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/fam15h_power.ko 2024-01-15T17:50:27.767 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 55449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/fschmd.ko 2024-01-15T17:50:27.767 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/g760a.ko 2024-01-15T17:50:27.767 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/g762.ko 2024-01-15T17:50:27.767 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 46801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/gl518sm.ko 2024-01-15T17:50:27.767 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 45529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/gl520sm.ko 2024-01-15T17:50:27.767 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/hih6130.ko 2024-01-15T17:50:27.767 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/hwmon-vid.ko 2024-01-15T17:50:27.767 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11553 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/i5500_temp.ko 2024-01-15T17:50:27.767 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/i5k_amb.ko 2024-01-15T17:50:27.767 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 34529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ibmaem.ko 2024-01-15T17:50:27.767 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ibmpex.ko 2024-01-15T17:50:27.767 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ina209.ko 2024-01-15T17:50:27.767 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ina2xx.ko 2024-01-15T17:50:27.767 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 130449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/it87.ko 2024-01-15T17:50:27.767 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/jc42.ko 2024-01-15T17:50:27.767 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/k10temp.ko 2024-01-15T17:50:27.768 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13553 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/k8temp.ko 2024-01-15T17:50:27.768 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lineage-pem.ko 2024-01-15T17:50:27.768 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 48601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm63.ko 2024-01-15T17:50:27.768 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16697 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm73.ko 2024-01-15T17:50:27.768 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 36777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm75.ko 2024-01-15T17:50:27.768 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm77.ko 2024-01-15T17:50:27.768 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 34593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm78.ko 2024-01-15T17:50:27.768 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 31745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm80.ko 2024-01-15T17:50:27.768 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16889 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm83.ko 2024-01-15T17:50:27.768 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 72833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm85.ko 2024-01-15T17:50:27.768 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 46153 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm87.ko 2024-01-15T17:50:27.768 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 37913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm90.ko 2024-01-15T17:50:27.768 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm92.ko 2024-01-15T17:50:27.768 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 108841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm93.ko 2024-01-15T17:50:27.769 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 33393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95234.ko 2024-01-15T17:50:27.769 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95241.ko 2024-01-15T17:50:27.769 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95245.ko 2024-01-15T17:50:27.769 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc2945.ko 2024-01-15T17:50:27.769 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4151.ko 2024-01-15T17:50:27.769 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4215.ko 2024-01-15T17:50:27.769 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14889 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4222.ko 2024-01-15T17:50:27.769 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4245.ko 2024-01-15T17:50:27.769 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4260.ko 2024-01-15T17:50:27.769 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4261.ko 2024-01-15T17:50:27.769 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 33489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max16065.ko 2024-01-15T17:50:27.769 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max1619.ko 2024-01-15T17:50:27.769 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max1668.ko 2024-01-15T17:50:27.769 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max197.ko 2024-01-15T17:50:27.769 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 27129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6639.ko 2024-01-15T17:50:27.770 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6650.ko 2024-01-15T17:50:27.770 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 35777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6697.ko 2024-01-15T17:50:27.770 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/mc13783-adc.ko 2024-01-15T17:50:27.770 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/mcp3021.ko 2024-01-15T17:50:27.770 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2024-01-15T17:50:27.770 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 44153 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6683.ko 2024-01-15T17:50:27.770 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 119689 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6775-core.ko 2024-01-15T17:50:27.770 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 60233 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6775.ko 2024-01-15T17:50:27.770 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 59737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct7802.ko 2024-01-15T17:50:27.770 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 82337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pc87360.ko 2024-01-15T17:50:27.770 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 54433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pc87427.ko 2024-01-15T17:50:27.770 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pcf8591.ko 2024-01-15T17:50:27.770 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ 2024-01-15T17:50:27.770 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-01-15T17:50:27.770 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-01-15T17:50:27.771 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-01-15T17:50:27.771 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max16064.ko 2024-01-15T17:50:27.771 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max34440.ko 2024-01-15T17:50:27.771 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max8688.ko 2024-01-15T17:50:27.771 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-01-15T17:50:27.771 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 82577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-01-15T17:50:27.771 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-01-15T17:50:27.771 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-01-15T17:50:27.771 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-01-15T17:50:27.771 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-01-15T17:50:27.771 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/powr1220.ko 2024-01-15T17:50:27.771 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch5627.ko 2024-01-15T17:50:27.771 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch5636.ko 2024-01-15T17:50:27.771 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 29249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch56xx-common.ko 2024-01-15T17:50:27.771 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 32137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sht15.ko 2024-01-15T17:50:27.772 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sht21.ko 2024-01-15T17:50:27.772 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/shtc1.ko 2024-01-15T17:50:27.772 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 40105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sis5595.ko 2024-01-15T17:50:27.772 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47b397.ko 2024-01-15T17:50:27.772 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 39193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47m1.ko 2024-01-15T17:50:27.772 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 37385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47m192.ko 2024-01-15T17:50:27.772 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26553 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/thmc50.ko 2024-01-15T17:50:27.772 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp102.ko 2024-01-15T17:50:27.772 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp103.ko 2024-01-15T17:50:27.772 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp401.ko 2024-01-15T17:50:27.772 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp421.ko 2024-01-15T17:50:27.772 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/via-cputemp.ko 2024-01-15T17:50:27.772 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 41945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/via686a.ko 2024-01-15T17:50:27.772 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 52457 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/vt1211.ko 2024-01-15T17:50:27.773 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 51545 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/vt8231.ko 2024-01-15T17:50:27.773 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 57449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83627ehf.ko 2024-01-15T17:50:27.773 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 78825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83627hf.ko 2024-01-15T17:50:27.773 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 71345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83781d.ko 2024-01-15T17:50:27.773 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 63225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83791d.ko 2024-01-15T17:50:27.773 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 71321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83792d.ko 2024-01-15T17:50:27.773 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 101473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83793.ko 2024-01-15T17:50:27.773 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 119505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83795.ko 2024-01-15T17:50:27.773 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83l785ts.ko 2024-01-15T17:50:27.773 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 35625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83l786ng.ko 2024-01-15T17:50:27.773 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/wm831x-hwmon.ko 2024-01-15T17:50:27.773 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/wm8350-hwmon.ko 2024-01-15T17:50:27.773 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/ 2024-01-15T17:50:27.773 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/ 2024-01-15T17:50:27.773 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-01-15T17:50:27.774 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-01-15T17:50:27.774 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/ 2024-01-15T17:50:27.774 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 27969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ali1535.ko 2024-01-15T17:50:27.774 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ali1563.ko 2024-01-15T17:50:27.774 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 27777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2024-01-15T17:50:27.774 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-01-15T17:50:27.774 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-01-15T17:50:27.774 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-01-15T17:50:27.774 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15153 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2024-01-15T17:50:27.774 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 6929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-01-15T17:50:27.774 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2024-01-15T17:50:27.774 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 50401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-01-15T17:50:27.774 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-01-15T17:50:27.774 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20001 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-01-15T17:50:27.775 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-01-15T17:50:27.775 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-dln2.ko 2024-01-15T17:50:27.775 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-gpio.ko 2024-01-15T17:50:27.775 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 62681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-i801.ko 2024-01-15T17:50:27.775 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-isch.ko 2024-01-15T17:50:27.775 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 47889 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-01-15T17:50:27.775 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-kempld.ko 2024-01-15T17:50:27.775 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-01-15T17:50:27.775 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-01-15T17:50:27.775 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ocores.ko 2024-01-15T17:50:27.775 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 25145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-parport.ko 2024-01-15T17:50:27.775 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-01-15T17:50:27.775 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 47241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-01-15T17:50:27.775 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2024-01-15T17:50:27.776 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-01-15T17:50:27.776 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-01-15T17:50:27.776 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-sis5595.ko 2024-01-15T17:50:27.776 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 29361 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-sis630.ko 2024-01-15T17:50:27.776 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-01-15T17:50:27.776 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2024-01-15T17:50:27.776 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-01-15T17:50:27.776 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-via.ko 2024-01-15T17:50:27.776 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-01-15T17:50:27.776 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-01-15T17:50:27.776 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 47657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-xiic.ko 2024-01-15T17:50:27.776 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-mux.ko 2024-01-15T17:50:27.776 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-smbus.ko 2024-01-15T17:50:27.777 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 27073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-stub.ko 2024-01-15T17:50:27.777 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/muxes/ 2024-01-15T17:50:27.777 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2024-01-15T17:50:27.777 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13457 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2024-01-15T17:50:27.777 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 29737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2024-01-15T17:50:27.777 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ 2024-01-15T17:50:27.777 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ 2024-01-15T17:50:27.777 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 273609 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_cm.ko 2024-01-15T17:50:27.777 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 883681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_core.ko 2024-01-15T17:50:27.777 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 69273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_umad.ko 2024-01-15T17:50:27.777 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 324329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-01-15T17:50:27.777 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 97921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/iw_cm.ko 2024-01-15T17:50:27.777 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 282601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/rdma_cm.ko 2024-01-15T17:50:27.777 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 66137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-01-15T17:50:27.778 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/ 2024-01-15T17:50:27.778 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/cxgb4/ 2024-01-15T17:50:27.778 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 493217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-01-15T17:50:27.778 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx4/ 2024-01-15T17:50:27.778 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 420561 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-01-15T17:50:27.778 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx5/ 2024-01-15T17:50:27.778 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 741449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-01-15T17:50:27.778 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mthca/ 2024-01-15T17:50:27.778 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 296881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-01-15T17:50:27.778 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/ocrdma/ 2024-01-15T17:50:27.778 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 153801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2024-01-15T17:50:27.778 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/usnic/ 2024-01-15T17:50:27.778 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 168969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2024-01-15T17:50:27.778 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ 2024-01-15T17:50:27.778 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ipoib/ 2024-01-15T17:50:27.779 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 263081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-01-15T17:50:27.802 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/iser/ 2024-01-15T17:50:27.802 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 100353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-01-15T17:50:27.803 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/isert/ 2024-01-15T17:50:27.803 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 111425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-01-15T17:50:27.803 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srp/ 2024-01-15T17:50:27.803 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 128753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-01-15T17:50:27.803 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srpt/ 2024-01-15T17:50:27.803 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 129529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-01-15T17:50:27.803 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/ 2024-01-15T17:50:27.803 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/evbug.ko 2024-01-15T17:50:27.803 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/ff-memless.ko 2024-01-15T17:50:27.803 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/ 2024-01-15T17:50:27.803 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-01-15T17:50:27.803 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/fm801-gp.ko 2024-01-15T17:50:27.803 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 41241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/gameport.ko 2024-01-15T17:50:27.803 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 41801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/joydev.ko 2024-01-15T17:50:27.804 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/ 2024-01-15T17:50:27.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/adp5520-keys.ko 2024-01-15T17:50:27.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12705 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/adp5588-keys.ko 2024-01-15T17:50:27.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/adp5589-keys.ko 2024-01-15T17:50:27.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2024-01-15T17:50:27.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 36665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/gpio_keys.ko 2024-01-15T17:50:27.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2024-01-15T17:50:27.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/lkkbd.ko 2024-01-15T17:50:27.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/lm8323.ko 2024-01-15T17:50:27.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/lm8333.ko 2024-01-15T17:50:27.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/matrix_keypad.ko 2024-01-15T17:50:27.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/max7359_keypad.ko 2024-01-15T17:50:27.804 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/mcs_touchkey.ko 2024-01-15T17:50:27.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2024-01-15T17:50:27.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/newtonkbd.ko 2024-01-15T17:50:27.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/opencores-kbd.ko 2024-01-15T17:50:27.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14161 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/qt1070.ko 2024-01-15T17:50:27.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/qt2160.ko 2024-01-15T17:50:27.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/samsung-keypad.ko 2024-01-15T17:50:27.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/stowaway.ko 2024-01-15T17:50:27.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/sunkbd.ko 2024-01-15T17:50:27.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/tca6416-keypad.ko 2024-01-15T17:50:27.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16177 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/tca8418_keypad.ko 2024-01-15T17:50:27.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/twl4030_keypad.ko 2024-01-15T17:50:27.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/xtkbd.ko 2024-01-15T17:50:27.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/matrix-keymap.ko 2024-01-15T17:50:27.805 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/ 2024-01-15T17:50:27.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 34329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/appletouch.ko 2024-01-15T17:50:27.805 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 34177 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/bcm5974.ko 2024-01-15T17:50:27.806 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 123417 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/cyapatp.ko 2024-01-15T17:50:27.806 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 89449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/elan_i2c.ko 2024-01-15T17:50:27.806 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/gpio_mouse.ko 2024-01-15T17:50:27.806 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 335953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/psmouse.ko 2024-01-15T17:50:27.806 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/sermouse.ko 2024-01-15T17:50:27.806 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-01-15T17:50:27.806 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/synaptics_usb.ko 2024-01-15T17:50:27.806 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/vsxxxaa.ko 2024-01-15T17:50:27.806 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/rmi4/ 2024-01-15T17:50:27.806 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 109873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/rmi4/rmi_core.ko 2024-01-15T17:50:27.806 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/ 2024-01-15T17:50:27.806 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/altera_ps2.ko 2024-01-15T17:50:27.806 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/arc_ps2.ko 2024-01-15T17:50:27.806 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/ct82c710.ko 2024-01-15T17:50:27.807 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-01-15T17:50:27.807 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/parkbd.ko 2024-01-15T17:50:27.807 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/pcips2.ko 2024-01-15T17:50:27.807 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20153 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/ps2mult.ko 2024-01-15T17:50:27.807 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24153 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/serio_raw.ko 2024-01-15T17:50:27.807 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17977 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/serport.ko 2024-01-15T17:50:27.807 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/sparse-keymap.ko 2024-01-15T17:50:27.807 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/ 2024-01-15T17:50:27.807 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-88pm860x.ko 2024-01-15T17:50:27.807 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-adp5520.ko 2024-01-15T17:50:27.807 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 59753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-bd2802.ko 2024-01-15T17:50:27.807 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-blinkm.ko 2024-01-15T17:50:27.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-da903x.ko 2024-01-15T17:50:27.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-da9052.ko 2024-01-15T17:50:27.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-gpio.ko 2024-01-15T17:50:27.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lm3530.ko 2024-01-15T17:50:27.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lm3533.ko 2024-01-15T17:50:27.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lm355x.ko 2024-01-15T17:50:27.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lm3642.ko 2024-01-15T17:50:27.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lp3944.ko 2024-01-15T17:50:27.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11233 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lp8788.ko 2024-01-15T17:50:27.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-max8997.ko 2024-01-15T17:50:27.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12545 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-mc13783.ko 2024-01-15T17:50:27.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-menf21bmc.ko 2024-01-15T17:50:27.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-pca9532.ko 2024-01-15T17:50:27.808 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-pca955x.ko 2024-01-15T17:50:27.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-pca963x.ko 2024-01-15T17:50:27.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 34433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-ss4200.ko 2024-01-15T17:50:27.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-tca6507.ko 2024-01-15T17:50:27.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-wm831x-status.ko 2024-01-15T17:50:27.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-wm8350.ko 2024-01-15T17:50:27.809 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ 2024-01-15T17:50:27.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-01-15T17:50:27.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-01-15T17:50:27.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-01-15T17:50:27.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-01-15T17:50:27.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-01-15T17:50:27.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-01-15T17:50:27.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-01-15T17:50:27.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11889 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-01-15T17:50:27.809 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-01-15T17:50:27.810 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/ 2024-01-15T17:50:27.810 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/bcache/ 2024-01-15T17:50:27.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 647193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/bcache/bcache.ko 2024-01-15T17:50:27.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 40577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-bio-prison.ko 2024-01-15T17:50:27.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 93409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-bufio.ko 2024-01-15T17:50:27.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 47529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-cache-smq.ko 2024-01-15T17:50:27.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 155897 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-cache.ko 2024-01-15T17:50:27.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 107257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-crypt.ko 2024-01-15T17:50:27.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-delay.ko 2024-01-15T17:50:27.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 53857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-era.ko 2024-01-15T17:50:27.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-flakey.ko 2024-01-15T17:50:27.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 41529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-log-userspace.ko 2024-01-15T17:50:27.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 32737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-log.ko 2024-01-15T17:50:27.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 49545 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-mirror.ko 2024-01-15T17:50:27.810 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 78033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-multipath.ko 2024-01-15T17:50:27.811 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-queue-length.ko 2024-01-15T17:50:27.811 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 71073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-raid.ko 2024-01-15T17:50:27.811 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 36297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-region-hash.ko 2024-01-15T17:50:27.811 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-round-robin.ko 2024-01-15T17:50:27.811 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-service-time.ko 2024-01-15T17:50:27.811 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 109529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-snapshot.ko 2024-01-15T17:50:27.811 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19361 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-switch.ko 2024-01-15T17:50:27.811 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 181865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-thin-pool.ko 2024-01-15T17:50:27.811 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 52073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-verity.ko 2024-01-15T17:50:27.811 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8561 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-zero.ko 2024-01-15T17:50:27.811 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/faulty.ko 2024-01-15T17:50:27.811 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19369 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/linear.ko 2024-01-15T17:50:27.811 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 32417 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/multipath.ko 2024-01-15T17:50:27.811 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/persistent-data/ 2024-01-15T17:50:27.811 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 199937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-01-15T17:50:27.812 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 37401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid0.ko 2024-01-15T17:50:27.812 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 113001 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid1.ko 2024-01-15T17:50:27.812 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 158657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid10.ko 2024-01-15T17:50:27.812 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 393777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid456.ko 2024-01-15T17:50:27.812 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/ 2024-01-15T17:50:27.812 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/ 2024-01-15T17:50:27.812 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 178137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptbase.ko 2024-01-15T17:50:27.812 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 63321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptctl.ko 2024-01-15T17:50:27.812 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 46337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptfc.ko 2024-01-15T17:50:27.812 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 37225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptlan.ko 2024-01-15T17:50:27.812 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 123321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptsas.ko 2024-01-15T17:50:27.812 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 81009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptscsih.ko 2024-01-15T17:50:27.812 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 52953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptspi.ko 2024-01-15T17:50:27.812 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/ 2024-01-15T17:50:27.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/88pm800.ko 2024-01-15T17:50:27.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12761 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/88pm805.ko 2024-01-15T17:50:27.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/88pm80x.ko 2024-01-15T17:50:27.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/arizona-i2c.ko 2024-01-15T17:50:27.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 123721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/arizona.ko 2024-01-15T17:50:27.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/bcm590xx.ko 2024-01-15T17:50:27.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/cros_ec_dev.ko 2024-01-15T17:50:27.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 34937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/dln2.ko 2024-01-15T17:50:27.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16689 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/janz-cmodio.ko 2024-01-15T17:50:27.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 53585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/kempld-core.ko 2024-01-15T17:50:27.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 27529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lm3533-core.ko 2024-01-15T17:50:27.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lm3533-ctrlbank.ko 2024-01-15T17:50:27.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lp3943.ko 2024-01-15T17:50:27.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 57057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lpc_ich.ko 2024-01-15T17:50:27.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lpc_sch.ko 2024-01-15T17:50:27.813 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/max8907.ko 2024-01-15T17:50:27.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 29857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/mc13xxx-core.ko 2024-01-15T17:50:27.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/mc13xxx-i2c.ko 2024-01-15T17:50:27.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/menf21bmc.ko 2024-01-15T17:50:27.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/pcf50633-adc.ko 2024-01-15T17:50:27.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9153 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/pcf50633-gpio.ko 2024-01-15T17:50:27.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 34873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/pcf50633.ko 2024-01-15T17:50:27.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/rdc321x-southbridge.ko 2024-01-15T17:50:27.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/retu-mfd.ko 2024-01-15T17:50:27.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 80425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/si476x-core.ko 2024-01-15T17:50:27.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 48457 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/sm501.ko 2024-01-15T17:50:27.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/ti_am335x_tscadc.ko 2024-01-15T17:50:27.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 51137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/tps65010.ko 2024-01-15T17:50:27.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/tps6507x.ko 2024-01-15T17:50:27.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/viperboard.ko 2024-01-15T17:50:27.814 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/vx855.ko 2024-01-15T17:50:27.815 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/wl1273-core.ko 2024-01-15T17:50:27.815 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 44345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/wm8994.ko 2024-01-15T17:50:27.815 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/ 2024-01-15T17:50:27.815 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/ad525x_dpot-i2c.ko 2024-01-15T17:50:27.815 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 48489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/ad525x_dpot.ko 2024-01-15T17:50:27.815 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/altera-stapl/ 2024-01-15T17:50:27.815 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 46585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-01-15T17:50:27.815 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/apds9802als.ko 2024-01-15T17:50:27.815 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 36729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/apds990x.ko 2024-01-15T17:50:27.815 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 45593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/bh1770glc.ko 2024-01-15T17:50:27.815 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/c2port/ 2024-01-15T17:50:27.815 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2024-01-15T17:50:27.815 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 35281 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/c2port/core.ko 2024-01-15T17:50:27.815 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/cb710/ 2024-01-15T17:50:27.815 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 25161 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/cb710/cb710.ko 2024-01-15T17:50:27.816 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/ds1682.ko 2024-01-15T17:50:27.816 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8897 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/dummy-irq.ko 2024-01-15T17:50:27.816 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/echo/ 2024-01-15T17:50:27.816 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/echo/echo.ko 2024-01-15T17:50:27.816 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/ 2024-01-15T17:50:27.816 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 38785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/at24.ko 2024-01-15T17:50:27.816 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-01-15T17:50:27.816 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/max6875.ko 2024-01-15T17:50:27.816 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 33009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/enclosure.ko 2024-01-15T17:50:27.816 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/genwqe/ 2024-01-15T17:50:27.816 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 152201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/genwqe/genwqe_card.ko 2024-01-15T17:50:27.816 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/hmc6352.ko 2024-01-15T17:50:27.816 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 31257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/hpilo.ko 2024-01-15T17:50:27.816 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/ibmasm/ 2024-01-15T17:50:27.816 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 72601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/ibmasm/ibmasm.ko 2024-01-15T17:50:27.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/ics932s401.ko 2024-01-15T17:50:27.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/isl29003.ko 2024-01-15T17:50:27.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/isl29020.ko 2024-01-15T17:50:27.817 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/ 2024-01-15T17:50:27.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 42513 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-01-15T17:50:27.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-01-15T17:50:27.817 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/ 2024-01-15T17:50:27.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 125945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei-me.ko 2024-01-15T17:50:27.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 56521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei-txe.ko 2024-01-15T17:50:27.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 313641 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei.ko 2024-01-15T17:50:27.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 31489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/phantom.ko 2024-01-15T17:50:27.817 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/ti-st/ 2024-01-15T17:50:27.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 74377 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/ti-st/st_drv.ko 2024-01-15T17:50:27.817 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tifm_7xx1.ko 2024-01-15T17:50:27.818 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tifm_core.ko 2024-01-15T17:50:27.818 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tsl2550.ko 2024-01-15T17:50:27.818 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 48441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_balloon.ko 2024-01-15T17:50:27.818 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_vmci/ 2024-01-15T17:50:27.818 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 183609 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-01-15T17:50:27.818 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ 2024-01-15T17:50:27.818 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/bonding/ 2024-01-15T17:50:27.818 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 490849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/bonding/bonding.ko 2024-01-15T17:50:27.818 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/ 2024-01-15T17:50:27.818 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/b53/ 2024-01-15T17:50:27.818 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 106033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/b53/b53_common.ko 2024-01-15T17:50:27.818 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 40305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/bcm-sf2.ko 2024-01-15T17:50:27.818 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6060.ko 2024-01-15T17:50:27.818 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6xxx/ 2024-01-15T17:50:27.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 455065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-01-15T17:50:27.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dummy.ko 2024-01-15T17:50:27.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/eql.ko 2024-01-15T17:50:27.819 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ 2024-01-15T17:50:27.819 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/ 2024-01-15T17:50:27.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 45785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-01-15T17:50:27.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 42105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-01-15T17:50:27.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 130553 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-01-15T17:50:27.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 68849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-01-15T17:50:27.819 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/ 2024-01-15T17:50:27.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 37649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/8390.ko 2024-01-15T17:50:27.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 59689 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-01-15T17:50:27.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 27201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-01-15T17:50:27.819 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 99049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-01-15T17:50:27.819 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/adaptec/ 2024-01-15T17:50:27.820 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 54521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-01-15T17:50:27.820 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/agere/ 2024-01-15T17:50:27.820 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 69137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/agere/et131x.ko 2024-01-15T17:50:27.820 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/alteon/ 2024-01-15T17:50:27.820 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 59585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-01-15T17:50:27.820 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/altera/ 2024-01-15T17:50:27.820 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 41545 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-01-15T17:50:27.820 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/ 2024-01-15T17:50:27.820 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 54609 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-01-15T17:50:27.820 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 49289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-01-15T17:50:27.820 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 87297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-01-15T17:50:27.820 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/ 2024-01-15T17:50:27.820 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/alx/ 2024-01-15T17:50:27.821 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 97337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-01-15T17:50:27.821 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1c/ 2024-01-15T17:50:27.821 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 102497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-01-15T17:50:27.821 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1e/ 2024-01-15T17:50:27.821 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 90345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-01-15T17:50:27.821 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/ 2024-01-15T17:50:27.821 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 81569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-01-15T17:50:27.821 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 69633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-01-15T17:50:27.821 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/ 2024-01-15T17:50:27.821 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 78401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-01-15T17:50:27.821 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 176761 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-01-15T17:50:27.821 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2024-01-15T17:50:27.821 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 1421529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-01-15T17:50:27.821 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 128569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-01-15T17:50:27.821 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/genet/ 2024-01-15T17:50:27.821 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 131337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-01-15T17:50:27.822 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 314553 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-01-15T17:50:27.822 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/ 2024-01-15T17:50:27.822 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/bna/ 2024-01-15T17:50:27.822 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 323745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-01-15T17:50:27.822 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/ 2024-01-15T17:50:27.822 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb/ 2024-01-15T17:50:27.822 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 193497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-01-15T17:50:27.822 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2024-01-15T17:50:27.822 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 347257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-01-15T17:50:27.822 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2024-01-15T17:50:27.822 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 1044793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-01-15T17:50:27.822 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2024-01-15T17:50:27.822 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 166193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-01-15T17:50:27.822 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2024-01-15T17:50:27.823 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 34881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-01-15T17:50:27.823 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/ 2024-01-15T17:50:27.823 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/enic/ 2024-01-15T17:50:27.823 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 204345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-01-15T17:50:27.823 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/ 2024-01-15T17:50:27.823 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/ 2024-01-15T17:50:27.823 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 69449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-01-15T17:50:27.823 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 61489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-01-15T17:50:27.823 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 141225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-01-15T17:50:27.823 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 51497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-01-15T17:50:27.823 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 62817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-01-15T17:50:27.823 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 31369 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-01-15T17:50:27.823 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/ 2024-01-15T17:50:27.823 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 60873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-01-15T17:50:27.823 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 62833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-01-15T17:50:27.824 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 45721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dnet.ko 2024-01-15T17:50:27.824 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 25185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ec_bhf.ko 2024-01-15T17:50:27.824 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/ 2024-01-15T17:50:27.824 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/benet/ 2024-01-15T17:50:27.824 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 281137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-01-15T17:50:27.824 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 49481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ethoc.ko 2024-01-15T17:50:27.824 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 51737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/fealnx.ko 2024-01-15T17:50:27.824 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/fujitsu/ 2024-01-15T17:50:27.824 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 45345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2024-01-15T17:50:27.824 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ 2024-01-15T17:50:27.824 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 90369 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e100.ko 2024-01-15T17:50:27.824 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000/ 2024-01-15T17:50:27.824 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 295481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-01-15T17:50:27.825 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000e/ 2024-01-15T17:50:27.825 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 584497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-01-15T17:50:27.825 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/fm10k/ 2024-01-15T17:50:27.825 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 246209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-01-15T17:50:27.825 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/i40e/ 2024-01-15T17:50:27.825 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 937673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-01-15T17:50:27.825 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/iavf/ 2024-01-15T17:50:27.825 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 287305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-01-15T17:50:27.825 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igb/ 2024-01-15T17:50:27.825 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 514569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-01-15T17:50:27.825 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igbvf/ 2024-01-15T17:50:27.825 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 93497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-01-15T17:50:27.825 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbe/ 2024-01-15T17:50:27.825 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 734353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-01-15T17:50:27.825 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbevf/ 2024-01-15T17:50:27.826 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 151769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-01-15T17:50:27.826 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 86921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/jme.ko 2024-01-15T17:50:27.826 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/ 2024-01-15T17:50:27.826 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-01-15T17:50:27.826 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 102289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/skge.ko 2024-01-15T17:50:27.826 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 122601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-01-15T17:50:27.826 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/ 2024-01-15T17:50:27.826 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/ 2024-01-15T17:50:27.826 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 727473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-01-15T17:50:27.855 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 278961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-01-15T17:50:27.856 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/ 2024-01-15T17:50:27.856 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2024-01-15T17:50:27.856 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 1118201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-01-15T17:50:27.856 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel/ 2024-01-15T17:50:27.856 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 53105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel/ks8842.ko 2024-01-15T17:50:27.856 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 41041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2024-01-15T17:50:27.856 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2024-01-15T17:50:27.856 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 95665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-01-15T17:50:27.856 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/ 2024-01-15T17:50:27.856 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/myri10ge/ 2024-01-15T17:50:27.856 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 108649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-01-15T17:50:27.856 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/ 2024-01-15T17:50:27.856 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 73441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-01-15T17:50:27.856 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 47777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-01-15T17:50:27.856 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/neterion/ 2024-01-15T17:50:27.857 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 150905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-01-15T17:50:27.857 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/nvidia/ 2024-01-15T17:50:27.857 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 136721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-01-15T17:50:27.857 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/ 2024-01-15T17:50:27.857 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 50481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-01-15T17:50:27.857 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 43145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-01-15T17:50:27.857 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/ 2024-01-15T17:50:27.857 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/netxen/ 2024-01-15T17:50:27.857 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 227897 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-01-15T17:50:27.857 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 87105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-01-15T17:50:27.857 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2024-01-15T17:50:27.857 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 671937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-01-15T17:50:27.857 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rdc/ 2024-01-15T17:50:27.857 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 43017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-01-15T17:50:27.858 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/ 2024-01-15T17:50:27.858 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 67385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-01-15T17:50:27.858 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 83985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-01-15T17:50:27.858 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 29321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/atp.ko 2024-01-15T17:50:27.858 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 192353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-01-15T17:50:27.858 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/samsung/ 2024-01-15T17:50:27.858 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/samsung/sxgbe/ 2024-01-15T17:50:27.858 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 104241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2024-01-15T17:50:27.858 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sfc/ 2024-01-15T17:50:27.858 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 1064737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-01-15T17:50:27.858 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/silan/ 2024-01-15T17:50:27.858 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 47097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-01-15T17:50:27.858 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/ 2024-01-15T17:50:27.858 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 54817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/sis190.ko 2024-01-15T17:50:27.859 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 63337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/sis900.ko 2024-01-15T17:50:27.859 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/ 2024-01-15T17:50:27.859 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 61537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-01-15T17:50:27.859 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 67193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-01-15T17:50:27.859 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 73409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-01-15T17:50:27.859 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 68177 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-01-15T17:50:27.859 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/ 2024-01-15T17:50:27.859 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/ 2024-01-15T17:50:27.859 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2024-01-15T17:50:27.859 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 39905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-01-15T17:50:27.859 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2024-01-15T17:50:27.859 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 487593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-01-15T17:50:27.859 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/ 2024-01-15T17:50:27.859 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 103337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/cassini.ko 2024-01-15T17:50:27.860 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 166097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/niu.ko 2024-01-15T17:50:27.860 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 66329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/sungem.ko 2024-01-15T17:50:27.860 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 78809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-01-15T17:50:27.860 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/tehuti/ 2024-01-15T17:50:27.860 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 50057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-01-15T17:50:27.860 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ti/ 2024-01-15T17:50:27.860 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 86529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ti/tlan.ko 2024-01-15T17:50:27.860 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/ 2024-01-15T17:50:27.860 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 80273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-01-15T17:50:27.860 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 83793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-01-15T17:50:27.860 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/ 2024-01-15T17:50:27.860 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 46217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-01-15T17:50:27.860 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-01-15T17:50:27.860 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/xircom/ 2024-01-15T17:50:27.861 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 59993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-01-15T17:50:27.861 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/fddi/ 2024-01-15T17:50:27.861 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 41001 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/fddi/defxx.ko 2024-01-15T17:50:27.861 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/fddi/skfp/ 2024-01-15T17:50:27.861 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 216985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/fddi/skfp/skfp.ko 2024-01-15T17:50:27.861 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 74569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/geneve.ko 2024-01-15T17:50:27.861 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/hyperv/ 2024-01-15T17:50:27.861 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 236225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-01-15T17:50:27.861 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ieee802154/ 2024-01-15T17:50:27.861 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ieee802154/fakelb.ko 2024-01-15T17:50:27.861 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 25593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ifb.ko 2024-01-15T17:50:27.861 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ipvlan/ 2024-01-15T17:50:27.861 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 85817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ipvlan/ipvlan.ko 2024-01-15T17:50:27.861 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 62769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/macvlan.ko 2024-01-15T17:50:27.861 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/macvtap.ko 2024-01-15T17:50:27.861 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/ 2024-01-15T17:50:27.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-01-15T17:50:27.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14233 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-01-15T17:50:27.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-gpio.ko 2024-01-15T17:50:27.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-regmap.ko 2024-01-15T17:50:27.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio.ko 2024-01-15T17:50:27.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20513 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mii.ko 2024-01-15T17:50:27.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 45089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/netconsole.ko 2024-01-15T17:50:27.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/nlmon.ko 2024-01-15T17:50:27.862 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/ 2024-01-15T17:50:27.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/pcs-lynx.ko 2024-01-15T17:50:27.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 39217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-01-15T17:50:27.862 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/ 2024-01-15T17:50:27.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/amd.ko 2024-01-15T17:50:27.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 50033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-01-15T17:50:27.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 44721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm7xxx.ko 2024-01-15T17:50:27.862 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11641 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm87xx.ko 2024-01-15T17:50:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 44857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/broadcom.ko 2024-01-15T17:50:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/cicada.ko 2024-01-15T17:50:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/davicom.ko 2024-01-15T17:50:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/et1011c.ko 2024-01-15T17:50:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/icplus.ko 2024-01-15T17:50:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/lxt.ko 2024-01-15T17:50:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 82049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/marvell.ko 2024-01-15T17:50:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 76409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/micrel.ko 2024-01-15T17:50:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/national.ko 2024-01-15T17:50:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 95561 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/phylink.ko 2024-01-15T17:50:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/qsemi.ko 2024-01-15T17:50:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 50065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/realtek.ko 2024-01-15T17:50:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/smsc.ko 2024-01-15T17:50:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/ste10Xp.ko 2024-01-15T17:50:27.863 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/vitesse.ko 2024-01-15T17:50:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/plip/ 2024-01-15T17:50:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 43505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/plip/plip.ko 2024-01-15T17:50:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ 2024-01-15T17:50:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/bsd_comp.ko 2024-01-15T17:50:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 29713 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_async.ko 2024-01-15T17:50:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_deflate.ko 2024-01-15T17:50:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_mppe.ko 2024-01-15T17:50:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_synctty.ko 2024-01-15T17:50:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 42849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pppoe.ko 2024-01-15T17:50:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pppox.ko 2024-01-15T17:50:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 32737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pptp.ko 2024-01-15T17:50:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/sb1000.ko 2024-01-15T17:50:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip/ 2024-01-15T17:50:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 42465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip/slip.ko 2024-01-15T17:50:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 27049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/sungem_phy.ko 2024-01-15T17:50:27.864 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 64585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/tap.ko 2024-01-15T17:50:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/ 2024-01-15T17:50:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 107089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team.ko 2024-01-15T17:50:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_activebackup.ko 2024-01-15T17:50:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_broadcast.ko 2024-01-15T17:50:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-01-15T17:50:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9641 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_random.ko 2024-01-15T17:50:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-01-15T17:50:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 86945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/veth.ko 2024-01-15T17:50:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vmxnet3/ 2024-01-15T17:50:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 144361 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-01-15T17:50:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vxlan/ 2024-01-15T17:50:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 301281 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vxlan/vxlan.ko 2024-01-15T17:50:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netback/ 2024-01-15T17:50:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 144001 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netback/xen-netback.ko 2024-01-15T17:50:27.865 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ntb/ 2024-01-15T17:50:27.866 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ntb/ntb.ko 2024-01-15T17:50:27.866 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/ 2024-01-15T17:50:27.866 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/ 2024-01-15T17:50:27.866 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 265777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/nvme-core.ko 2024-01-15T17:50:27.866 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 102545 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/nvme.ko 2024-01-15T17:50:27.866 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/ 2024-01-15T17:50:27.866 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 131249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport.ko 2024-01-15T17:50:27.866 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14561 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_cs.ko 2024-01-15T17:50:27.866 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 85377 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_pc.ko 2024-01-15T17:50:27.866 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_serial.ko 2024-01-15T17:50:27.866 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/ 2024-01-15T17:50:27.866 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug/ 2024-01-15T17:50:27.866 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 25009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-01-15T17:50:27.866 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2024-01-15T17:50:27.866 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2024-01-15T17:50:27.866 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/pci-stub.ko 2024-01-15T17:50:27.867 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 46465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/xen-pcifront.ko 2024-01-15T17:50:27.867 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/ 2024-01-15T17:50:27.867 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/i82092.ko 2024-01-15T17:50:27.867 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 146289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/pcmcia.ko 2024-01-15T17:50:27.867 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 62161 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/pcmcia_core.ko 2024-01-15T17:50:27.867 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 34345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2024-01-15T17:50:27.867 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 34265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/pd6729.ko 2024-01-15T17:50:27.867 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 112889 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/yenta_socket.ko 2024-01-15T17:50:27.867 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/phy/ 2024-01-15T17:50:27.867 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/phy/broadcom/ 2024-01-15T17:50:27.867 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2024-01-15T17:50:27.867 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/phy/samsung/ 2024-01-15T17:50:27.867 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2024-01-15T17:50:27.867 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/ 2024-01-15T17:50:27.867 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/intel/ 2024-01-15T17:50:27.868 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 69137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-01-15T17:50:27.868 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/ 2024-01-15T17:50:27.868 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/chrome/ 2024-01-15T17:50:27.868 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 50889 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2024-01-15T17:50:27.868 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/chrome/cros_ec.ko 2024-01-15T17:50:27.868 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2024-01-15T17:50:27.868 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 25465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2024-01-15T17:50:27.868 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2024-01-15T17:50:27.868 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2024-01-15T17:50:27.868 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17361 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2024-01-15T17:50:27.868 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/ 2024-01-15T17:50:27.868 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 67097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/acer-wmi.ko 2024-01-15T17:50:27.868 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 39145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/acerhdf.ko 2024-01-15T17:50:27.868 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 37025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/apple-gmux.ko 2024-01-15T17:50:27.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 59665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-laptop.ko 2024-01-15T17:50:27.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 31481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-01-15T17:50:27.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 132025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-wmi.ko 2024-01-15T17:50:27.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 36849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/classmate-laptop.ko 2024-01-15T17:50:27.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 49121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-01-15T17:50:27.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-01-15T17:50:27.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 40265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-01-15T17:50:27.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-01-15T17:50:27.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 36593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/hdaps.ko 2024-01-15T17:50:27.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/ibm_rtl.ko 2024-01-15T17:50:27.869 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel/ 2024-01-15T17:50:27.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-01-15T17:50:27.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2024-01-15T17:50:27.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 43497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel_ips.ko 2024-01-15T17:50:27.869 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/msi-wmi.ko 2024-01-15T17:50:27.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10377 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/mxm-wmi.ko 2024-01-15T17:50:27.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 34801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-01-15T17:50:27.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 42329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/samsung-laptop.ko 2024-01-15T17:50:27.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/samsung-q10.ko 2024-01-15T17:50:27.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/topstar-laptop.ko 2024-01-15T17:50:27.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-01-15T17:50:27.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/toshiba_haps.ko 2024-01-15T17:50:27.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/wmi-bmof.ko 2024-01-15T17:50:27.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 56905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/wmi.ko 2024-01-15T17:50:27.870 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/ 2024-01-15T17:50:27.870 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/ 2024-01-15T17:50:27.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30233 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/88pm860x_battery.ko 2024-01-15T17:50:27.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 33985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/88pm860x_charger.ko 2024-01-15T17:50:27.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 55057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/bq2415x_charger.ko 2024-01-15T17:50:27.870 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 54081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/bq24190_charger.ko 2024-01-15T17:50:27.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21553 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/bq24735-charger.ko 2024-01-15T17:50:27.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/cros_peripheral_charger.ko 2024-01-15T17:50:27.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19609 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/da9030_battery.ko 2024-01-15T17:50:27.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/da9052-battery.ko 2024-01-15T17:50:27.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 29281 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/ds2760_battery.ko 2024-01-15T17:50:27.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/ds2780_battery.ko 2024-01-15T17:50:27.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/ds2781_battery.ko 2024-01-15T17:50:27.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/ds2782_battery.ko 2024-01-15T17:50:27.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/gpio-charger.ko 2024-01-15T17:50:27.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/isp1704_charger.ko 2024-01-15T17:50:27.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/lp8727_charger.ko 2024-01-15T17:50:27.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/max14577_charger.ko 2024-01-15T17:50:27.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/max17042_battery.ko 2024-01-15T17:50:27.871 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/max8903_charger.ko 2024-01-15T17:50:27.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/max8925_power.ko 2024-01-15T17:50:27.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/pcf50633-charger.ko 2024-01-15T17:50:27.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 37937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/sbs-battery.ko 2024-01-15T17:50:27.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/test_power.ko 2024-01-15T17:50:27.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/tps65090-charger.ko 2024-01-15T17:50:27.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/wm831x_backup.ko 2024-01-15T17:50:27.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24457 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/wm831x_power.ko 2024-01-15T17:50:27.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22417 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/wm8350_power.ko 2024-01-15T17:50:27.872 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/ 2024-01-15T17:50:27.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 58001 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/intel_rapl_common.ko 2024-01-15T17:50:27.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/intel_rapl_msr.ko 2024-01-15T17:50:27.872 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/ 2024-01-15T17:50:27.872 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/ 2024-01-15T17:50:27.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps-gpio.ko 2024-01-15T17:50:27.872 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps-ldisc.ko 2024-01-15T17:50:27.873 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps_parport.ko 2024-01-15T17:50:27.873 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 41873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/pps_core.ko 2024-01-15T17:50:27.873 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ 2024-01-15T17:50:27.873 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 70937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ptp.ko 2024-01-15T17:50:27.873 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ptp_kvm.ko 2024-01-15T17:50:27.873 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/ 2024-01-15T17:50:27.873 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-88pm80x.ko 2024-01-15T17:50:27.873 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-88pm860x.ko 2024-01-15T17:50:27.873 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-bq32k.ko 2024-01-15T17:50:27.873 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-da9052.ko 2024-01-15T17:50:27.873 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-da9055.ko 2024-01-15T17:50:27.873 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-da9063.ko 2024-01-15T17:50:27.873 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16457 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1286.ko 2024-01-15T17:50:27.873 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 68577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1307.ko 2024-01-15T17:50:27.873 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1374.ko 2024-01-15T17:50:27.874 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1511.ko 2024-01-15T17:50:27.874 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1553.ko 2024-01-15T17:50:27.874 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1672.ko 2024-01-15T17:50:27.874 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13369 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1742.ko 2024-01-15T17:50:27.874 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14697 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds2404.ko 2024-01-15T17:50:27.874 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23705 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds3232.ko 2024-01-15T17:50:27.874 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-em3027.ko 2024-01-15T17:50:27.874 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-fm3130.ko 2024-01-15T17:50:27.874 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-isl12022.ko 2024-01-15T17:50:27.874 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 35969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-isl1208.ko 2024-01-15T17:50:27.874 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-lp8788.ko 2024-01-15T17:50:27.874 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 38097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m41t80.ko 2024-01-15T17:50:27.874 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t35.ko 2024-01-15T17:50:27.874 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 25945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t59.ko 2024-01-15T17:50:27.875 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t86.ko 2024-01-15T17:50:27.875 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-max6900.ko 2024-01-15T17:50:27.875 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-max8907.ko 2024-01-15T17:50:27.875 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-max8925.ko 2024-01-15T17:50:27.875 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 25809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-max8997.ko 2024-01-15T17:50:27.875 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-max8998.ko 2024-01-15T17:50:27.875 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18513 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-mc13xxx.ko 2024-01-15T17:50:27.875 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-msm6242.ko 2024-01-15T17:50:27.875 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-palmas.ko 2024-01-15T17:50:27.875 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 46633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf2127.ko 2024-01-15T17:50:27.875 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17977 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf50633.ko 2024-01-15T17:50:27.875 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf85063.ko 2024-01-15T17:50:27.875 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8523.ko 2024-01-15T17:50:27.875 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 29817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8563.ko 2024-01-15T17:50:27.875 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8583.ko 2024-01-15T17:50:27.876 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rc5t583.ko 2024-01-15T17:50:27.876 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rp5c01.ko 2024-01-15T17:50:27.876 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 33753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rs5c372.ko 2024-01-15T17:50:27.876 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 27833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-01-15T17:50:27.876 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 25257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rx8025.ko 2024-01-15T17:50:27.876 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rx8581.ko 2024-01-15T17:50:27.876 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-s35390a.ko 2024-01-15T17:50:27.876 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-01-15T17:50:27.876 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17889 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-tps6586x.ko 2024-01-15T17:50:27.876 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-tps65910.ko 2024-01-15T17:50:27.876 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20545 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-wm831x.ko 2024-01-15T17:50:27.876 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-wm8350.ko 2024-01-15T17:50:27.876 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 27873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-x1205.ko 2024-01-15T17:50:27.876 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ 2024-01-15T17:50:27.876 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 77617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-9xxx.ko 2024-01-15T17:50:27.877 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 57665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-sas.ko 2024-01-15T17:50:27.877 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 55105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-xxxx.ko 2024-01-15T17:50:27.877 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 135185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/BusLogic.ko 2024-01-15T17:50:27.877 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/a100u2w.ko 2024-01-15T17:50:27.877 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aacraid/ 2024-01-15T17:50:27.877 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 271113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aacraid/aacraid.ko 2024-01-15T17:50:27.877 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 100617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/advansys.ko 2024-01-15T17:50:27.877 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/ 2024-01-15T17:50:27.877 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 356929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-01-15T17:50:27.877 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 296049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-01-15T17:50:27.877 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic94xx/ 2024-01-15T17:50:27.877 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 189185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic94xx/aic94xx.ko 2024-01-15T17:50:27.877 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/am53c974.ko 2024-01-15T17:50:27.877 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/arcmsr/ 2024-01-15T17:50:27.877 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 106881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-01-15T17:50:27.878 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 38185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/atp870u.ko 2024-01-15T17:50:27.878 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/be2iscsi/ 2024-01-15T17:50:27.878 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 247009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-01-15T17:50:27.878 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bfa/ 2024-01-15T17:50:27.878 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 1171945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bfa/bfa.ko 2024-01-15T17:50:27.878 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2fc/ 2024-01-15T17:50:27.878 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 237721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-01-15T17:50:27.878 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2i/ 2024-01-15T17:50:27.878 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 115369 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-01-15T17:50:27.878 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 40585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ch.ko 2024-01-15T17:50:27.878 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/csiostor/ 2024-01-15T17:50:27.878 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 308921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/csiostor/csiostor.ko 2024-01-15T17:50:27.878 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/ 2024-01-15T17:50:27.878 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb3i/ 2024-01-15T17:50:27.878 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 76145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-01-15T17:50:27.879 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb4i/ 2024-01-15T17:50:27.879 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 101449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-01-15T17:50:27.879 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 146625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-01-15T17:50:27.879 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 75553 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/dc395x.ko 2024-01-15T17:50:27.879 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 38073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/dmx3191d.ko 2024-01-15T17:50:27.879 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esas2r/ 2024-01-15T17:50:27.879 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 212425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esas2r/esas2r.ko 2024-01-15T17:50:27.879 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 60593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esp_scsi.ko 2024-01-15T17:50:27.879 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/ 2024-01-15T17:50:27.879 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 78209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/fcoe.ko 2024-01-15T17:50:27.879 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 158505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-01-15T17:50:27.879 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fdomain.ko 2024-01-15T17:50:27.879 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fnic/ 2024-01-15T17:50:27.879 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 263489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fnic/fnic.ko 2024-01-15T17:50:27.880 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 201817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hpsa.ko 2024-01-15T17:50:27.880 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 51209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hptiop.ko 2024-01-15T17:50:27.880 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 49041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hv_storvsc.ko 2024-01-15T17:50:27.880 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 36217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/imm.ko 2024-01-15T17:50:27.880 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 44057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/initio.ko 2024-01-15T17:50:27.880 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 266537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ipr.ko 2024-01-15T17:50:27.880 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 101473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ips.ko 2024-01-15T17:50:27.880 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/isci/ 2024-01-15T17:50:27.880 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 317153 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/isci/isci.ko 2024-01-15T17:50:27.880 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-01-15T17:50:27.880 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 52961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/iscsi_tcp.ko 2024-01-15T17:50:27.880 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libfc/ 2024-01-15T17:50:27.880 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 315209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libfc/libfc.ko 2024-01-15T17:50:27.880 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 145585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libiscsi.ko 2024-01-15T17:50:27.881 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 53905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libiscsi_tcp.ko 2024-01-15T17:50:27.881 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libsas/ 2024-01-15T17:50:27.881 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 216441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libsas/libsas.ko 2024-01-15T17:50:27.881 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/lpfc/ 2024-01-15T17:50:27.881 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 2197481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/lpfc/lpfc.ko 2024-01-15T17:50:27.881 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/ 2024-01-15T17:50:27.881 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 73569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-01-15T17:50:27.915 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 33529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-01-15T17:50:27.915 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 310841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-01-15T17:50:27.915 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 84649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid.ko 2024-01-15T17:50:27.915 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mpt3sas/ 2024-01-15T17:50:27.915 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 659801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-01-15T17:50:27.915 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvsas/ 2024-01-15T17:50:27.915 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 119561 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvsas/mvsas.ko 2024-01-15T17:50:27.916 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 78129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvumi.ko 2024-01-15T17:50:27.916 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pcmcia/ 2024-01-15T17:50:27.916 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 81649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2024-01-15T17:50:27.916 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11713 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2024-01-15T17:50:27.916 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2024-01-15T17:50:27.916 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2024-01-15T17:50:27.916 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pm8001/ 2024-01-15T17:50:27.916 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 366401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-01-15T17:50:27.916 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 124633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pmcraid.ko 2024-01-15T17:50:27.916 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 34513 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ppa.ko 2024-01-15T17:50:27.916 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 55425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla1280.ko 2024-01-15T17:50:27.916 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/ 2024-01-15T17:50:27.916 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 1876633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-01-15T17:50:27.916 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 85289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-01-15T17:50:27.917 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla4xxx/ 2024-01-15T17:50:27.917 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 559033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-01-15T17:50:27.917 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qlogicfas408.ko 2024-01-15T17:50:27.917 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/raid_class.ko 2024-01-15T17:50:27.917 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 261241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_debug.ko 2024-01-15T17:50:27.917 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 173937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_fc.ko 2024-01-15T17:50:27.917 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 279313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-01-15T17:50:27.917 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 97729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_sas.ko 2024-01-15T17:50:27.917 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 73969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_spi.ko 2024-01-15T17:50:27.917 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 44993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_srp.ko 2024-01-15T17:50:27.917 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ses.ko 2024-01-15T17:50:27.917 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 115889 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/st.ko 2024-01-15T17:50:27.917 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 46825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/stex.ko 2024-01-15T17:50:27.917 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/sym53c8xx_2/ 2024-01-15T17:50:27.918 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 164961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-01-15T17:50:27.918 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 42297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/virtio_scsi.ko 2024-01-15T17:50:27.918 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 50321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/vmw_pvscsi.ko 2024-01-15T17:50:27.918 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 39961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/wd719x.ko 2024-01-15T17:50:27.918 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 50025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/xen-scsifront.ko 2024-01-15T17:50:27.918 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ssb/ 2024-01-15T17:50:27.918 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 129441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ssb/ssb.ko 2024-01-15T17:50:27.918 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/ 2024-01-15T17:50:27.918 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/iscsi/ 2024-01-15T17:50:27.918 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 736769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-01-15T17:50:27.918 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/loopback/ 2024-01-15T17:50:27.918 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 49969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/loopback/tcm_loop.ko 2024-01-15T17:50:27.918 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 39713 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_file.ko 2024-01-15T17:50:27.918 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 43025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_iblock.ko 2024-01-15T17:50:27.918 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 930697 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_mod.ko 2024-01-15T17:50:27.919 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 44241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_pscsi.ko 2024-01-15T17:50:27.919 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 122521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_user.ko 2024-01-15T17:50:27.919 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/tcm_fc/ 2024-01-15T17:50:27.919 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 75593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-01-15T17:50:27.919 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/ 2024-01-15T17:50:27.919 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/ 2024-01-15T17:50:27.919 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/ 2024-01-15T17:50:27.919 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-01-15T17:50:27.919 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 29321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-01-15T17:50:27.919 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-01-15T17:50:27.919 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-01-15T17:50:27.919 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-01-15T17:50:27.919 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-01-15T17:50:27.919 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-01-15T17:50:27.920 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-01-15T17:50:27.920 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-01-15T17:50:27.920 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-01-15T17:50:27.920 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-01-15T17:50:27.920 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13001 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-01-15T17:50:27.920 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 37065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-01-15T17:50:27.920 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-01-15T17:50:27.920 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-01-15T17:50:27.920 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_powerclamp.ko 2024-01-15T17:50:27.920 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-01-15T17:50:27.920 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-01-15T17:50:27.920 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-01-15T17:50:27.921 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ 2024-01-15T17:50:27.921 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ipwireless/ 2024-01-15T17:50:27.921 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 78425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-01-15T17:50:27.921 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 57865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/moxa.ko 2024-01-15T17:50:27.921 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 50953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/mxser.ko 2024-01-15T17:50:27.921 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 108601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/n_gsm.ko 2024-01-15T17:50:27.921 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 31961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/n_hdlc.ko 2024-01-15T17:50:27.921 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 67929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/nozomi.ko 2024-01-15T17:50:27.921 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/ 2024-01-15T17:50:27.921 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250/ 2024-01-15T17:50:27.921 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 34769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250/8250_dw.ko 2024-01-15T17:50:27.921 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 59497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-01-15T17:50:27.921 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/altera_jtaguart.ko 2024-01-15T17:50:27.921 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/altera_uart.ko 2024-01-15T17:50:27.922 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/arc_uart.ko 2024-01-15T17:50:27.922 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 72425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/fsl_lpuart.ko 2024-01-15T17:50:27.922 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/jsm/ 2024-01-15T17:50:27.922 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 103073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/jsm/jsm.ko 2024-01-15T17:50:27.922 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/rp2.ko 2024-01-15T17:50:27.922 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 53169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/sc16is7xx.ko 2024-01-15T17:50:27.922 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/ 2024-01-15T17:50:27.922 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/core/ 2024-01-15T17:50:27.922 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 535985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/core/ufshcd-core.ko 2024-01-15T17:50:27.922 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host/ 2024-01-15T17:50:27.922 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-01-15T17:50:27.922 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2024-01-15T17:50:27.922 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/ 2024-01-15T17:50:27.922 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 44513 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio.ko 2024-01-15T17:50:27.922 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_aec.ko 2024-01-15T17:50:27.923 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11281 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_cif.ko 2024-01-15T17:50:27.923 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_dmem_genirq.ko 2024-01-15T17:50:27.923 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_mf624.ko 2024-01-15T17:50:27.923 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12177 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_netx.ko 2024-01-15T17:50:27.923 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_pci_generic.ko 2024-01-15T17:50:27.923 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_pdrv_genirq.ko 2024-01-15T17:50:27.923 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_sercos3.ko 2024-01-15T17:50:27.923 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/ 2024-01-15T17:50:27.923 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/c67x00/ 2024-01-15T17:50:27.923 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 73297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/c67x00/c67x00.ko 2024-01-15T17:50:27.923 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/chipidea/ 2024-01-15T17:50:27.923 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 144617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/chipidea/ci_hdrc.ko 2024-01-15T17:50:27.923 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2024-01-15T17:50:27.923 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13177 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/chipidea/ci_hdrc_npcm.ko 2024-01-15T17:50:27.923 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12713 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2024-01-15T17:50:27.924 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2024-01-15T17:50:27.924 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/ 2024-01-15T17:50:27.924 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 87073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/cdc-acm.ko 2024-01-15T17:50:27.924 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 51345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/cdc-wdm.ko 2024-01-15T17:50:27.924 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 47305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/usblp.ko 2024-01-15T17:50:27.924 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 89857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/usbtmc.ko 2024-01-15T17:50:27.924 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/common/ 2024-01-15T17:50:27.924 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/common/ulpi.ko 2024-01-15T17:50:27.924 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/dwc2/ 2024-01-15T17:50:27.924 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/dwc2/dwc2_pci.ko 2024-01-15T17:50:27.924 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/dwc3/ 2024-01-15T17:50:27.924 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/dwc3/dwc3-haps.ko 2024-01-15T17:50:27.924 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/dwc3/dwc3-pci.ko 2024-01-15T17:50:27.924 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 454385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/dwc3/dwc3.ko 2024-01-15T17:50:27.924 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/ 2024-01-15T17:50:27.925 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/ 2024-01-15T17:50:27.925 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 60217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/u_ether.ko 2024-01-15T17:50:27.925 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 51881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/u_serial.ko 2024-01-15T17:50:27.925 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2024-01-15T17:50:27.925 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 42449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2024-01-15T17:50:27.925 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 27953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2024-01-15T17:50:27.925 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 36513 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2024-01-15T17:50:27.925 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 92489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2024-01-15T17:50:27.925 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 51041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2024-01-15T17:50:27.925 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 122081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2024-01-15T17:50:27.925 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 57937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2024-01-15T17:50:27.925 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2024-01-15T17:50:27.925 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 31265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2024-01-15T17:50:27.925 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 57729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2024-01-15T17:50:27.925 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 85609 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2024-01-15T17:50:27.926 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2024-01-15T17:50:27.926 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 58113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2024-01-15T17:50:27.926 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 67337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2024-01-15T17:50:27.926 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/ 2024-01-15T17:50:27.926 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2024-01-15T17:50:27.926 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2024-01-15T17:50:27.926 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2024-01-15T17:50:27.926 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/g_ether.ko 2024-01-15T17:50:27.926 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2024-01-15T17:50:27.926 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/g_hid.ko 2024-01-15T17:50:27.926 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2024-01-15T17:50:27.926 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16977 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2024-01-15T17:50:27.926 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 32073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2024-01-15T17:50:27.926 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/g_printer.ko 2024-01-15T17:50:27.926 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/g_serial.ko 2024-01-15T17:50:27.927 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 27201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/g_zero.ko 2024-01-15T17:50:27.927 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 62361 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2024-01-15T17:50:27.927 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2024-01-15T17:50:27.927 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 161225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/libcomposite.ko 2024-01-15T17:50:27.927 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/ 2024-01-15T17:50:27.927 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17705 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2024-01-15T17:50:27.927 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/bdc/ 2024-01-15T17:50:27.927 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 157681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2024-01-15T17:50:27.927 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 36089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/goku_udc.ko 2024-01-15T17:50:27.927 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 52721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/gr_udc.ko 2024-01-15T17:50:27.927 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 69705 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2024-01-15T17:50:27.927 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 56033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/mv_udc.ko 2024-01-15T17:50:27.927 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 76057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/net2272.ko 2024-01-15T17:50:27.927 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 75073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/net2280.ko 2024-01-15T17:50:27.927 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 73169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/pch_udc.ko 2024-01-15T17:50:27.928 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 65809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2024-01-15T17:50:27.928 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 66097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2024-01-15T17:50:27.928 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 81505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2024-01-15T17:50:27.928 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 156409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/udc-core.ko 2024-01-15T17:50:27.928 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/ 2024-01-15T17:50:27.928 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/bcma-hcd.ko 2024-01-15T17:50:27.928 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 80113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/isp116x-hcd.ko 2024-01-15T17:50:27.928 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 87353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/oxu210hp-hcd.ko 2024-01-15T17:50:27.928 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 78873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/r8a66597-hcd.ko 2024-01-15T17:50:27.928 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 51329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/sl811-hcd.ko 2024-01-15T17:50:27.928 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/sl811_cs.ko 2024-01-15T17:50:27.928 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/ssb-hcd.ko 2024-01-15T17:50:27.928 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/xhci-plat-hcd.ko 2024-01-15T17:50:27.928 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/ 2024-01-15T17:50:27.929 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 34585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/mdc800.ko 2024-01-15T17:50:27.929 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/microtek.ko 2024-01-15T17:50:27.929 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ 2024-01-15T17:50:27.929 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 41721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/adutux.ko 2024-01-15T17:50:27.929 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20713 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/appledisplay.ko 2024-01-15T17:50:27.929 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/cypress_cy7c63.ko 2024-01-15T17:50:27.929 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/cytherm.ko 2024-01-15T17:50:27.929 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ehset.ko 2024-01-15T17:50:27.929 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/emi26.ko 2024-01-15T17:50:27.929 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/emi62.ko 2024-01-15T17:50:27.929 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ezusb.ko 2024-01-15T17:50:27.929 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/idmouse.ko 2024-01-15T17:50:27.929 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 36553 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/iowarrior.ko 2024-01-15T17:50:27.929 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/isight_firmware.ko 2024-01-15T17:50:27.929 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 36529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ldusb.ko 2024-01-15T17:50:27.930 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 36265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/legousbtower.ko 2024-01-15T17:50:27.930 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 27793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/lvstest.ko 2024-01-15T17:50:27.930 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/sisusbvga/ 2024-01-15T17:50:27.930 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 64841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-01-15T17:50:27.930 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/trancevibrator.ko 2024-01-15T17:50:27.930 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usb3503.ko 2024-01-15T17:50:27.930 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 25145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usblcd.ko 2024-01-15T17:50:27.930 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usbsevseg.ko 2024-01-15T17:50:27.930 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 70809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usbtest.ko 2024-01-15T17:50:27.930 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 41905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/uss720.ko 2024-01-15T17:50:27.930 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 27617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/yurex.ko 2024-01-15T17:50:27.930 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/mon/ 2024-01-15T17:50:27.930 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 72953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/mon/usbmon.ko 2024-01-15T17:50:27.930 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/musb/ 2024-01-15T17:50:27.931 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 301697 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/musb/musb_hdrc.ko 2024-01-15T17:50:27.931 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy/ 2024-01-15T17:50:27.931 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy/phy-generic.ko 2024-01-15T17:50:27.931 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20177 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2024-01-15T17:50:27.931 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy/phy-isp1301.ko 2024-01-15T17:50:27.931 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy/phy-tahvo.ko 2024-01-15T17:50:27.931 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ 2024-01-15T17:50:27.931 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/aircable.ko 2024-01-15T17:50:27.931 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 33329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ark3116.ko 2024-01-15T17:50:27.931 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/belkin_sa.ko 2024-01-15T17:50:27.931 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 38105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ch341.ko 2024-01-15T17:50:27.931 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 70257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cp210x.ko 2024-01-15T17:50:27.931 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 31057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cyberjack.ko 2024-01-15T17:50:27.931 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 58217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cypress_m8.ko 2024-01-15T17:50:27.932 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 49817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/digi_acceleport.ko 2024-01-15T17:50:27.932 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/empeg.ko 2024-01-15T17:50:27.932 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 44273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/f81232.ko 2024-01-15T17:50:27.932 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 175193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ftdi_sio.ko 2024-01-15T17:50:27.932 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 53113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/garmin_gps.ko 2024-01-15T17:50:27.932 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 137081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/io_edgeport.ko 2024-01-15T17:50:27.932 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 124833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/io_ti.ko 2024-01-15T17:50:27.932 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 67769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ipaq.ko 2024-01-15T17:50:27.932 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19161 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ipw.ko 2024-01-15T17:50:27.932 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26897 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ir-usb.ko 2024-01-15T17:50:27.932 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 58233 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-01-15T17:50:27.932 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 107521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/keyspan.ko 2024-01-15T17:50:27.932 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 35617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/keyspan_pda.ko 2024-01-15T17:50:27.932 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 25665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/kl5kusb105.ko 2024-01-15T17:50:27.933 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 33689 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/kobil_sct.ko 2024-01-15T17:50:27.933 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 32953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mct_u232.ko 2024-01-15T17:50:27.933 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/metro-usb.ko 2024-01-15T17:50:27.933 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 66449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mos7720.ko 2024-01-15T17:50:27.933 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 78217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mos7840.ko 2024-01-15T17:50:27.933 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 56337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mxuport.ko 2024-01-15T17:50:27.933 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/navman.ko 2024-01-15T17:50:27.933 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/omninet.ko 2024-01-15T17:50:27.933 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/opticon.ko 2024-01-15T17:50:27.933 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 191801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/option.ko 2024-01-15T17:50:27.933 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 39073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/oti6858.ko 2024-01-15T17:50:27.933 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 54353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/pl2303.ko 2024-01-15T17:50:27.933 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/qcaux.ko 2024-01-15T17:50:27.933 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 37825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/qcserial.ko 2024-01-15T17:50:27.934 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 34321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/quatech2.ko 2024-01-15T17:50:27.934 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/safe_serial.ko 2024-01-15T17:50:27.934 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 51625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/sierra.ko 2024-01-15T17:50:27.934 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/spcp8x5.ko 2024-01-15T17:50:27.934 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 29993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ssu100.ko 2024-01-15T17:50:27.934 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/symbolserial.ko 2024-01-15T17:50:27.934 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 61489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-01-15T17:50:27.934 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-01-15T17:50:27.934 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb_debug.ko 2024-01-15T17:50:27.934 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 33201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb_wwan.ko 2024-01-15T17:50:27.934 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 125609 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usbserial.ko 2024-01-15T17:50:27.934 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/visor.ko 2024-01-15T17:50:27.934 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 37409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/whiteheat.ko 2024-01-15T17:50:27.934 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/wishbone-serial.ko 2024-01-15T17:50:27.934 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/xsens_mt.ko 2024-01-15T17:50:27.935 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ 2024-01-15T17:50:27.935 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 52913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/uas.ko 2024-01-15T17:50:27.935 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-alauda.ko 2024-01-15T17:50:27.935 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-cypress.ko 2024-01-15T17:50:27.935 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-datafab.ko 2024-01-15T17:50:27.935 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 33497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-01-15T17:50:27.935 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-freecom.ko 2024-01-15T17:50:27.935 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-isd200.ko 2024-01-15T17:50:27.935 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19361 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-01-15T17:50:27.935 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13713 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-karma.ko 2024-01-15T17:50:27.935 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-onetouch.ko 2024-01-15T17:50:27.935 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-realtek.ko 2024-01-15T17:50:27.935 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-sddr09.ko 2024-01-15T17:50:27.935 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-sddr55.ko 2024-01-15T17:50:27.936 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 35337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-usbat.ko 2024-01-15T17:50:27.936 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 180225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/usb-storage.ko 2024-01-15T17:50:27.936 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/ 2024-01-15T17:50:27.936 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 67481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/usbip-core.ko 2024-01-15T17:50:27.936 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 71097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/usbip-host.ko 2024-01-15T17:50:27.936 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 103977 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-01-15T17:50:27.936 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/ 2024-01-15T17:50:27.936 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/ 2024-01-15T17:50:27.936 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 154297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-01-15T17:50:27.936 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 29537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/vfio-pci.ko 2024-01-15T17:50:27.936 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 103737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/vfio.ko 2024-01-15T17:50:27.936 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 77609 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-01-15T17:50:27.936 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/ 2024-01-15T17:50:27.937 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 113785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost.ko 2024-01-15T17:50:27.937 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_iotlb.ko 2024-01-15T17:50:27.937 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 58089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_net.ko 2024-01-15T17:50:27.937 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 89233 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_scsi.ko 2024-01-15T17:50:27.937 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/ 2024-01-15T17:50:27.937 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/ 2024-01-15T17:50:27.937 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15417 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/88pm860x_bl.ko 2024-01-15T17:50:27.937 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/aat2870_bl.ko 2024-01-15T17:50:27.937 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/adp5520_bl.ko 2024-01-15T17:50:27.937 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 32609 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/adp8860_bl.ko 2024-01-15T17:50:27.937 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 39993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/adp8870_bl.ko 2024-01-15T17:50:27.937 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/apple_bl.ko 2024-01-15T17:50:27.937 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/as3711_bl.ko 2024-01-15T17:50:27.937 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/bd6107.ko 2024-01-15T17:50:27.937 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/da903x_bl.ko 2024-01-15T17:50:27.937 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/da9052_bl.ko 2024-01-15T17:50:27.938 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/gpio_backlight.ko 2024-01-15T17:50:27.938 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/kb3886_bl.ko 2024-01-15T17:50:27.938 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/lcd.ko 2024-01-15T17:50:27.938 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/lm3533_bl.ko 2024-01-15T17:50:27.938 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/lm3639_bl.ko 2024-01-15T17:50:27.938 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/lv5207lp.ko 2024-01-15T17:50:27.938 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/max8925_bl.ko 2024-01-15T17:50:27.938 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/pandora_bl.ko 2024-01-15T17:50:27.938 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/pcf50633-backlight.ko 2024-01-15T17:50:27.938 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10897 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/platform_lcd.ko 2024-01-15T17:50:27.938 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/wm831x_bl.ko 2024-01-15T17:50:27.938 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/ 2024-01-15T17:50:27.938 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ 2024-01-15T17:50:27.938 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ds2482.ko 2024-01-15T17:50:27.938 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 38985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ds2490.ko 2024-01-15T17:50:27.938 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/matrox_w1.ko 2024-01-15T17:50:27.939 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12417 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/w1-gpio.ko 2024-01-15T17:50:27.939 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/ 2024-01-15T17:50:27.939 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-01-15T17:50:27.939 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-01-15T17:50:27.939 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-01-15T17:50:27.939 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-01-15T17:50:27.939 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14153 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-01-15T17:50:27.939 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-01-15T17:50:27.939 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-01-15T17:50:27.939 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-01-15T17:50:27.939 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-01-15T17:50:27.939 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_smem.ko 2024-01-15T17:50:27.939 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 56425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_therm.ko 2024-01-15T17:50:27.939 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 102673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/wire.ko 2024-01-15T17:50:27.939 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ 2024-01-15T17:50:27.940 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/acquirewdt.ko 2024-01-15T17:50:27.940 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19977 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/advantechwdt.ko 2024-01-15T17:50:27.940 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18561 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/alim1535_wdt.ko 2024-01-15T17:50:27.940 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/alim7101_wdt.ko 2024-01-15T17:50:27.940 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/cpu5wdt.ko 2024-01-15T17:50:27.940 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/da9052_wdt.ko 2024-01-15T17:50:27.940 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/da9055_wdt.ko 2024-01-15T17:50:27.940 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/da9063_wdt.ko 2024-01-15T17:50:27.940 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/dw_wdt.ko 2024-01-15T17:50:27.940 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/eurotechwdt.ko 2024-01-15T17:50:27.940 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/f71808e_wdt.ko 2024-01-15T17:50:27.940 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 27401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/hpwdt.ko 2024-01-15T17:50:27.940 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/i6300esb.ko 2024-01-15T17:50:27.940 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-01-15T17:50:27.941 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/iTCO_wdt.ko 2024-01-15T17:50:27.941 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ib700wdt.ko 2024-01-15T17:50:27.941 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18689 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ibmasr.ko 2024-01-15T17:50:27.941 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19713 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-01-15T17:50:27.941 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/it8712f_wdt.ko 2024-01-15T17:50:27.941 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/it87_wdt.ko 2024-01-15T17:50:27.941 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/kempld_wdt.ko 2024-01-15T17:50:27.941 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/machzwd.ko 2024-01-15T17:50:27.941 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/mena21_wdt.ko 2024-01-15T17:50:27.941 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/menf21bmc_wdt.ko 2024-01-15T17:50:27.941 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/nv_tco.ko 2024-01-15T17:50:27.941 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/of_xilinx_wdt.ko 2024-01-15T17:50:27.941 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pc87413_wdt.ko 2024-01-15T17:50:27.941 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 38657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pcwd_pci.ko 2024-01-15T17:50:27.941 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 31825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pcwd_usb.ko 2024-01-15T17:50:27.941 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/retu_wdt.ko 2024-01-15T17:50:27.942 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sbc60xxwdt.ko 2024-01-15T17:50:27.942 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sbc_epx_c3.ko 2024-01-15T17:50:27.942 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-01-15T17:50:27.942 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sc1200wdt.ko 2024-01-15T17:50:27.942 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sch311x_wdt.ko 2024-01-15T17:50:27.942 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/smsc37b787_wdt.ko 2024-01-15T17:50:27.942 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/softdog.ko 2024-01-15T17:50:27.942 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sp5100_tco.ko 2024-01-15T17:50:27.942 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/twl4030_wdt.ko 2024-01-15T17:50:27.942 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/via_wdt.ko 2024-01-15T17:50:27.942 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19161 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-01-15T17:50:27.942 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83877f_wdt.ko 2024-01-15T17:50:27.942 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 25409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83977f_wdt.ko 2024-01-15T17:50:27.942 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/wafer5823wdt.ko 2024-01-15T17:50:27.943 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/wdt_pci.ko 2024-01-15T17:50:27.943 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/wm831x_wdt.ko 2024-01-15T17:50:27.943 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/wm8350_wdt.ko 2024-01-15T17:50:27.943 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15697 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/xen_wdt.ko 2024-01-15T17:50:27.943 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/ 2024-01-15T17:50:27.943 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-evtchn.ko 2024-01-15T17:50:27.943 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-gntalloc.ko 2024-01-15T17:50:27.943 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 47865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-gntdev.ko 2024-01-15T17:50:27.943 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-pciback/ 2024-01-15T17:50:27.943 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 176441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-01-15T17:50:27.943 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 40185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-privcmd.ko 2024-01-15T17:50:27.943 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 71249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-scsiback.ko 2024-01-15T17:50:27.944 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xenfs/ 2024-01-15T17:50:27.944 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xenfs/xenfs.ko 2024-01-15T17:50:27.944 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ 2024-01-15T17:50:27.944 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/9p/ 2024-01-15T17:50:27.944 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 166753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/9p/9p.ko 2024-01-15T17:50:27.944 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/adfs/ 2024-01-15T17:50:27.944 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 61649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/adfs/adfs.ko 2024-01-15T17:50:27.944 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/affs/ 2024-01-15T17:50:27.944 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 163449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/affs/affs.ko 2024-01-15T17:50:27.944 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/afs/ 2024-01-15T17:50:27.944 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 935033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/afs/kafs.ko 2024-01-15T17:50:27.944 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/autofs/ 2024-01-15T17:50:27.944 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 103401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/autofs/autofs4.ko 2024-01-15T17:50:27.945 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/befs/ 2024-01-15T17:50:27.945 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 64681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/befs/befs.ko 2024-01-15T17:50:27.945 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/bfs/ 2024-01-15T17:50:27.945 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 41025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/bfs/bfs.ko 2024-01-15T17:50:27.945 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 38745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/binfmt_misc.ko 2024-01-15T17:50:27.945 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/btrfs/ 2024-01-15T17:50:27.945 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 3455857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/btrfs/btrfs.ko 2024-01-15T17:50:27.945 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/cachefiles/ 2024-01-15T17:50:27.945 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 306009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/cachefiles/cachefiles.ko 2024-01-15T17:50:27.945 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ceph/ 2024-01-15T17:50:27.945 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 1335129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ceph/ceph.ko 2024-01-15T17:50:27.945 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/coda/ 2024-01-15T17:50:27.945 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 100801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/coda/coda.ko 2024-01-15T17:50:27.945 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/configfs/ 2024-01-15T17:50:27.945 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 108641 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/configfs/configfs.ko 2024-01-15T17:50:27.945 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/cramfs/ 2024-01-15T17:50:27.946 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 29673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/cramfs/cramfs.ko 2024-01-15T17:50:27.946 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/dlm/ 2024-01-15T17:50:27.946 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 644073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/dlm/dlm.ko 2024-01-15T17:50:27.946 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/efs/ 2024-01-15T17:50:27.946 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 36785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/efs/efs.ko 2024-01-15T17:50:27.946 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/f2fs/ 2024-01-15T17:50:27.946 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 1506641 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/f2fs/f2fs.ko 2024-01-15T17:50:27.953 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/ 2024-01-15T17:50:27.953 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 27929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/msdos.ko 2024-01-15T17:50:27.954 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/freevxfs/ 2024-01-15T17:50:27.954 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 42609 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/freevxfs/freevxfs.ko 2024-01-15T17:50:27.954 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/fscache/ 2024-01-15T17:50:27.954 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 226481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/fscache/fscache.ko 2024-01-15T17:50:27.954 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse/ 2024-01-15T17:50:27.954 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse/cuse.ko 2024-01-15T17:50:27.954 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/gfs2/ 2024-01-15T17:50:27.954 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 1229913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/gfs2/gfs2.ko 2024-01-15T17:50:27.954 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/hfs/ 2024-01-15T17:50:27.954 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 141985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/hfs/hfs.ko 2024-01-15T17:50:27.954 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/hfsplus/ 2024-01-15T17:50:27.954 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 223225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/hfsplus/hfsplus.ko 2024-01-15T17:50:27.954 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/hpfs/ 2024-01-15T17:50:27.954 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 193569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/hpfs/hpfs.ko 2024-01-15T17:50:27.954 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/isofs/ 2024-01-15T17:50:27.955 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 74785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/isofs/isofs.ko 2024-01-15T17:50:27.955 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/jfs/ 2024-01-15T17:50:27.955 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 419881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/jfs/jfs.ko 2024-01-15T17:50:27.955 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/lockd/ 2024-01-15T17:50:27.955 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 282049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/lockd/lockd.ko 2024-01-15T17:50:27.955 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/minix/ 2024-01-15T17:50:27.955 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 89761 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/minix/minix.ko 2024-01-15T17:50:27.955 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/netfs/ 2024-01-15T17:50:27.955 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 124649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/netfs/netfs.ko 2024-01-15T17:50:27.955 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/ 2024-01-15T17:50:27.955 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/blocklayout/ 2024-01-15T17:50:27.955 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 81513 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-01-15T17:50:27.955 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/filelayout/ 2024-01-15T17:50:27.955 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 59449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-01-15T17:50:27.955 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/flexfilelayout/ 2024-01-15T17:50:27.955 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 117465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-01-15T17:50:27.956 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 1057969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfs.ko 2024-01-15T17:50:27.956 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 66441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv2.ko 2024-01-15T17:50:27.956 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 120265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv3.ko 2024-01-15T17:50:27.956 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 2042153 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv4.ko 2024-01-15T17:50:27.956 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/ 2024-01-15T17:50:27.956 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/grace.ko 2024-01-15T17:50:27.956 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/nfs_acl.ko 2024-01-15T17:50:27.956 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfsd/ 2024-01-15T17:50:27.956 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 1563929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfsd/nfsd.ko 2024-01-15T17:50:27.956 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nilfs2/ 2024-01-15T17:50:27.956 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 590713 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nilfs2/nilfs2.ko 2024-01-15T17:50:27.956 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/ 2024-01-15T17:50:27.956 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-celtic.ko 2024-01-15T17:50:27.957 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-centeuro.ko 2024-01-15T17:50:27.957 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-croatian.ko 2024-01-15T17:50:27.957 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-cyrillic.ko 2024-01-15T17:50:27.957 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-gaelic.ko 2024-01-15T17:50:27.957 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-greek.ko 2024-01-15T17:50:27.957 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-iceland.ko 2024-01-15T17:50:27.957 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-inuit.ko 2024-01-15T17:50:27.957 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-roman.ko 2024-01-15T17:50:27.957 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-romanian.ko 2024-01-15T17:50:27.957 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-turkish.ko 2024-01-15T17:50:27.957 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_ascii.ko 2024-01-15T17:50:27.957 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1250.ko 2024-01-15T17:50:27.957 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12233 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1251.ko 2024-01-15T17:50:27.957 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1255.ko 2024-01-15T17:50:27.957 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp737.ko 2024-01-15T17:50:27.958 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp775.ko 2024-01-15T17:50:27.958 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp850.ko 2024-01-15T17:50:27.958 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp852.ko 2024-01-15T17:50:27.958 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp855.ko 2024-01-15T17:50:27.958 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp857.ko 2024-01-15T17:50:27.958 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp860.ko 2024-01-15T17:50:27.958 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13161 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp861.ko 2024-01-15T17:50:27.958 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp862.ko 2024-01-15T17:50:27.958 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13161 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp863.ko 2024-01-15T17:50:27.958 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp864.ko 2024-01-15T17:50:27.958 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13161 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp865.ko 2024-01-15T17:50:27.958 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp866.ko 2024-01-15T17:50:27.958 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp869.ko 2024-01-15T17:50:27.958 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp874.ko 2024-01-15T17:50:27.958 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 94137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp932.ko 2024-01-15T17:50:27.959 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 142673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp936.ko 2024-01-15T17:50:27.959 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 165745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp949.ko 2024-01-15T17:50:27.959 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 118121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp950.ko 2024-01-15T17:50:27.959 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12457 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_euc-jp.ko 2024-01-15T17:50:27.959 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-1.ko 2024-01-15T17:50:27.959 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-13.ko 2024-01-15T17:50:27.959 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-14.ko 2024-01-15T17:50:27.959 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-15.ko 2024-01-15T17:50:27.959 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-2.ko 2024-01-15T17:50:27.959 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-3.ko 2024-01-15T17:50:27.959 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-4.ko 2024-01-15T17:50:27.959 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-5.ko 2024-01-15T17:50:27.959 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-6.ko 2024-01-15T17:50:27.959 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12233 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-7.ko 2024-01-15T17:50:27.960 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-9.ko 2024-01-15T17:50:27.960 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-r.ko 2024-01-15T17:50:27.960 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-ru.ko 2024-01-15T17:50:27.960 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-u.ko 2024-01-15T17:50:27.960 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_ucs2_utils.ko 2024-01-15T17:50:27.960 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_utf8.ko 2024-01-15T17:50:27.960 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ntfs/ 2024-01-15T17:50:27.960 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 220553 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ntfs/ntfs.ko 2024-01-15T17:50:27.960 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ 2024-01-15T17:50:27.960 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/cluster/ 2024-01-15T17:50:27.960 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 244737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-01-15T17:50:27.960 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlm/ 2024-01-15T17:50:27.960 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 440737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-01-15T17:50:27.960 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlmfs/ 2024-01-15T17:50:27.960 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 54265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-01-15T17:50:27.961 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 2593521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2.ko 2024-01-15T17:50:27.961 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-01-15T17:50:27.961 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 31729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-01-15T17:50:27.961 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 35897 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-01-15T17:50:27.961 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/omfs/ 2024-01-15T17:50:27.961 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 53769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/omfs/omfs.ko 2024-01-15T17:50:27.961 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/overlayfs/ 2024-01-15T17:50:27.961 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 370697 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/overlayfs/overlay.ko 2024-01-15T17:50:27.961 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/pstore/ 2024-01-15T17:50:27.961 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 54809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/pstore/ramoops.ko 2024-01-15T17:50:27.961 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/qnx4/ 2024-01-15T17:50:27.961 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/qnx4/qnx4.ko 2024-01-15T17:50:27.961 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/qnx6/ 2024-01-15T17:50:27.961 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 46689 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/qnx6/qnx6.ko 2024-01-15T17:50:27.961 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/quota/ 2024-01-15T17:50:27.962 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/quota/quota_tree.ko 2024-01-15T17:50:27.962 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/quota/quota_v1.ko 2024-01-15T17:50:27.962 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/quota/quota_v2.ko 2024-01-15T17:50:27.962 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/reiserfs/ 2024-01-15T17:50:27.962 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 532025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/reiserfs/reiserfs.ko 2024-01-15T17:50:27.962 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/romfs/ 2024-01-15T17:50:27.962 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/romfs/romfs.ko 2024-01-15T17:50:27.962 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/ 2024-01-15T17:50:27.962 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/client/ 2024-01-15T17:50:27.962 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 2683249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/client/cifs.ko 2024-01-15T17:50:27.962 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/ 2024-01-15T17:50:27.962 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7161 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/cifs_arc4.ko 2024-01-15T17:50:27.962 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/cifs_md4.ko 2024-01-15T17:50:27.962 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/squashfs/ 2024-01-15T17:50:27.963 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 134777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/squashfs/squashfs.ko 2024-01-15T17:50:27.963 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/sysv/ 2024-01-15T17:50:27.963 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 91249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/sysv/sysv.ko 2024-01-15T17:50:27.963 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/udf/ 2024-01-15T17:50:27.963 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 248425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/udf/udf.ko 2024-01-15T17:50:27.963 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ufs/ 2024-01-15T17:50:27.963 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 162537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ufs/ufs.ko 2024-01-15T17:50:27.963 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/xfs/ 2024-01-15T17:50:27.963 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 4014433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/xfs/xfs.ko 2024-01-15T17:50:27.981 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/kernel/ 2024-01-15T17:50:27.981 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/kernel/locking/ 2024-01-15T17:50:27.981 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 80585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/kernel/locking/locktorture.ko 2024-01-15T17:50:27.982 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/kernel/time/ 2024-01-15T17:50:27.982 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/kernel/time/test_udelay.ko 2024-01-15T17:50:27.982 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 60353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/kernel/torture.ko 2024-01-15T17:50:27.982 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/ 2024-01-15T17:50:27.982 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 6833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/crc-itu-t.ko 2024-01-15T17:50:27.982 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 6529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/crc7.ko 2024-01-15T17:50:27.982 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/crc8.ko 2024-01-15T17:50:27.982 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/ 2024-01-15T17:50:27.982 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/libarc4.ko 2024-01-15T17:50:27.982 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 37081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/libdes.ko 2024-01-15T17:50:27.982 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/interval_tree_test.ko 2024-01-15T17:50:27.982 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/libcrc32c.ko 2024-01-15T17:50:27.982 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/lru_cache.ko 2024-01-15T17:50:27.982 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/ 2024-01-15T17:50:27.983 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 33801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/lz4_compress.ko 2024-01-15T17:50:27.983 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/lz4hc_compress.ko 2024-01-15T17:50:27.983 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/math/ 2024-01-15T17:50:27.983 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 6617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/math/cordic.ko 2024-01-15T17:50:27.983 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8233 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/memory-notifier-error-inject.ko 2024-01-15T17:50:27.983 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/notifier-error-inject.ko 2024-01-15T17:50:27.983 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/percpu_test.ko 2024-01-15T17:50:27.983 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/pm-notifier-error-inject.ko 2024-01-15T17:50:27.983 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/raid6/ 2024-01-15T17:50:27.983 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 143353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/raid6/raid6_pq.ko 2024-01-15T17:50:27.983 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/rbtree_test.ko 2024-01-15T17:50:27.983 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/reed_solomon/ 2024-01-15T17:50:27.983 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/reed_solomon/reed_solomon.ko 2024-01-15T17:50:27.984 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 38841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/test-kstrtox.ko 2024-01-15T17:50:27.984 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/test-string_helpers.ko 2024-01-15T17:50:27.984 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 4876633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/test_bpf.ko 2024-01-15T17:50:27.984 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 49089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/test_firmware.ko 2024-01-15T17:50:27.984 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 6369 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/test_module.ko 2024-01-15T17:50:27.984 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/test_user_copy.ko 2024-01-15T17:50:27.984 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_bm.ko 2024-01-15T17:50:27.984 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_fsm.ko 2024-01-15T17:50:27.984 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_kmp.ko 2024-01-15T17:50:27.984 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/xz/ 2024-01-15T17:50:27.984 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13545 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/xz/xz_dec_test.ko 2024-01-15T17:50:27.984 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/mm/ 2024-01-15T17:50:27.984 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/mm/hwpoison-inject.ko 2024-01-15T17:50:27.984 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ 2024-01-15T17:50:27.985 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/ 2024-01-15T17:50:27.985 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 55737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/6lowpan.ko 2024-01-15T17:50:27.985 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_dest.ko 2024-01-15T17:50:27.985 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_fragment.ko 2024-01-15T17:50:27.985 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_hop.ko 2024-01-15T17:50:27.985 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 6985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_ipv6.ko 2024-01-15T17:50:27.985 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_mobility.ko 2024-01-15T17:50:27.985 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7001 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_routing.ko 2024-01-15T17:50:27.985 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_udp.ko 2024-01-15T17:50:27.985 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/802/ 2024-01-15T17:50:27.985 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/802/garp.ko 2024-01-15T17:50:27.985 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 34929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/802/mrp.ko 2024-01-15T17:50:27.985 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/802/p8022.ko 2024-01-15T17:50:27.985 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/802/psnap.ko 2024-01-15T17:50:27.985 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/802/stp.ko 2024-01-15T17:50:27.986 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/8021q/ 2024-01-15T17:50:27.986 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 82593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/8021q/8021q.ko 2024-01-15T17:50:27.986 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/9p/ 2024-01-15T17:50:27.986 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 198905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet.ko 2024-01-15T17:50:27.986 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 36233 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_fd.ko 2024-01-15T17:50:27.986 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_rdma.ko 2024-01-15T17:50:27.986 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 32337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_virtio.ko 2024-01-15T17:50:27.986 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/batman-adv/ 2024-01-15T17:50:27.986 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 649977 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/batman-adv/batman-adv.ko 2024-01-15T17:50:27.986 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/ 2024-01-15T17:50:27.986 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 65137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/br_netfilter.ko 2024-01-15T17:50:27.986 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 785817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/bridge.ko 2024-01-15T17:50:27.986 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ 2024-01-15T17:50:27.986 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-01-15T17:50:27.987 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_among.ko 2024-01-15T17:50:27.987 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_arp.ko 2024-01-15T17:50:27.987 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-01-15T17:50:27.987 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-01-15T17:50:27.987 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_ip.ko 2024-01-15T17:50:27.987 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-01-15T17:50:27.987 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_limit.ko 2024-01-15T17:50:27.987 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_log.ko 2024-01-15T17:50:27.987 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_mark.ko 2024-01-15T17:50:27.987 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-01-15T17:50:27.987 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-01-15T17:50:27.987 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-01-15T17:50:27.987 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-01-15T17:50:27.987 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_snat.ko 2024-01-15T17:50:27.987 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_stp.ko 2024-01-15T17:50:27.988 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-01-15T17:50:27.988 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11641 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-01-15T17:50:27.988 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-01-15T17:50:27.988 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-01-15T17:50:27.988 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 56033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtables.ko 2024-01-15T17:50:27.988 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-01-15T17:50:27.988 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-01-15T17:50:27.988 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ceph/ 2024-01-15T17:50:27.988 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 1046969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ceph/libceph.ko 2024-01-15T17:50:27.988 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/core/ 2024-01-15T17:50:27.988 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 126041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/core/pktgen.ko 2024-01-15T17:50:27.988 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/ 2024-01-15T17:50:27.988 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 194529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp.ko 2024-01-15T17:50:27.989 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_diag.ko 2024-01-15T17:50:27.989 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 58841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_ipv4.ko 2024-01-15T17:50:27.989 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 60873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_ipv6.ko 2024-01-15T17:50:27.989 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/ 2024-01-15T17:50:27.989 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 390129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/dsa_core.ko 2024-01-15T17:50:27.989 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_brcm.ko 2024-01-15T17:50:27.989 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_dsa.ko 2024-01-15T17:50:27.989 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_none.ko 2024-01-15T17:50:27.989 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_trailer.ko 2024-01-15T17:50:27.989 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/hsr/ 2024-01-15T17:50:27.989 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 99881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/hsr/hsr.ko 2024-01-15T17:50:27.989 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ 2024-01-15T17:50:27.989 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/6lowpan/ 2024-01-15T17:50:27.989 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 45537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-01-15T17:50:27.989 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 255849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ieee802154.ko 2024-01-15T17:50:27.990 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 47985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ieee802154_socket.ko 2024-01-15T17:50:27.990 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ 2024-01-15T17:50:27.990 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24761 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ah4.ko 2024-01-15T17:50:27.990 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 39065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/esp4.ko 2024-01-15T17:50:27.990 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 48489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/fou.ko 2024-01-15T17:50:27.990 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/gre.ko 2024-01-15T17:50:27.990 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 49601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/inet_diag.ko 2024-01-15T17:50:27.990 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 61273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_gre.ko 2024-01-15T17:50:27.990 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 61401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_tunnel.ko 2024-01-15T17:50:27.990 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 35009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_vti.ko 2024-01-15T17:50:27.990 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ipcomp.ko 2024-01-15T17:50:27.990 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 29001 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ipip.ko 2024-01-15T17:50:27.990 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ 2024-01-15T17:50:27.990 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 37945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arp_tables.ko 2024-01-15T17:50:27.990 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-01-15T17:50:27.990 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-01-15T17:50:27.991 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 41929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ip_tables.ko 2024-01-15T17:50:27.991 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-01-15T17:50:27.991 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2024-01-15T17:50:27.991 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-01-15T17:50:27.991 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11177 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-01-15T17:50:27.991 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-01-15T17:50:27.991 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_filter.ko 2024-01-15T17:50:27.991 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-01-15T17:50:27.991 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-01-15T17:50:27.991 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-01-15T17:50:27.991 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_security.ko 2024-01-15T17:50:27.991 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12545 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-01-15T17:50:27.991 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-01-15T17:50:27.991 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 27625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-01-15T17:50:27.991 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-01-15T17:50:27.992 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-01-15T17:50:27.992 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21001 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2024-01-15T17:50:27.992 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-01-15T17:50:27.992 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15177 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-01-15T17:50:27.992 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-01-15T17:50:27.992 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-01-15T17:50:27.992 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-01-15T17:50:27.992 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_bic.ko 2024-01-15T17:50:27.992 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_dctcp.ko 2024-01-15T17:50:27.992 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_diag.ko 2024-01-15T17:50:27.992 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_highspeed.ko 2024-01-15T17:50:27.992 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_htcp.ko 2024-01-15T17:50:27.992 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_hybla.ko 2024-01-15T17:50:27.992 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_illinois.ko 2024-01-15T17:50:27.992 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_lp.ko 2024-01-15T17:50:27.993 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_scalable.ko 2024-01-15T17:50:27.993 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_vegas.ko 2024-01-15T17:50:27.993 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_veno.ko 2024-01-15T17:50:27.993 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_westwood.ko 2024-01-15T17:50:27.993 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_yeah.ko 2024-01-15T17:50:27.993 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tunnel4.ko 2024-01-15T17:50:27.993 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/udp_diag.ko 2024-01-15T17:50:27.993 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 44977 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/udp_tunnel.ko 2024-01-15T17:50:27.993 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11361 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/xfrm4_tunnel.ko 2024-01-15T17:50:27.993 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ 2024-01-15T17:50:27.993 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ah6.ko 2024-01-15T17:50:27.993 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 39705 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/esp6.ko 2024-01-15T17:50:27.993 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/fou6.ko 2024-01-15T17:50:27.993 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 80105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_gre.ko 2024-01-15T17:50:27.993 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 93977 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_tunnel.ko 2024-01-15T17:50:27.994 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-01-15T17:50:27.994 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 54313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_vti.ko 2024-01-15T17:50:27.994 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ipcomp6.ko 2024-01-15T17:50:27.994 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/mip6.ko 2024-01-15T17:50:27.994 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ 2024-01-15T17:50:27.994 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 42489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-01-15T17:50:27.994 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2024-01-15T17:50:27.994 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-01-15T17:50:27.994 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-01-15T17:50:27.994 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-01-15T17:50:27.994 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-01-15T17:50:27.994 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-01-15T17:50:27.994 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-01-15T17:50:27.994 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-01-15T17:50:27.995 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9897 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-01-15T17:50:27.995 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-01-15T17:50:27.995 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10897 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-01-15T17:50:27.995 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-01-15T17:50:27.995 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-01-15T17:50:27.995 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-01-15T17:50:27.995 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-01-15T17:50:27.995 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-01-15T17:50:27.995 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 39225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-01-15T17:50:27.995 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-01-15T17:50:27.995 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 25225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-01-15T17:50:27.995 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-01-15T17:50:27.995 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-01-15T17:50:27.995 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-01-15T17:50:27.995 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-01-15T17:50:27.995 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-01-15T17:50:27.996 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 74921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/sit.ko 2024-01-15T17:50:27.996 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/tunnel6.ko 2024-01-15T17:50:27.996 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/xfrm6_tunnel.ko 2024-01-15T17:50:27.996 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/key/ 2024-01-15T17:50:27.996 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 84433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/key/af_key.ko 2024-01-15T17:50:27.996 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/ 2024-01-15T17:50:27.996 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 125305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_core.ko 2024-01-15T17:50:27.996 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20177 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_debugfs.ko 2024-01-15T17:50:27.996 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 25953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_eth.ko 2024-01-15T17:50:27.996 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 36585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ip.ko 2024-01-15T17:50:27.996 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 37465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ip6.ko 2024-01-15T17:50:27.996 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 33569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_netlink.ko 2024-01-15T17:50:27.996 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 53321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ppp.ko 2024-01-15T17:50:27.996 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/lapb/ 2024-01-15T17:50:27.997 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 48121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/lapb/lapb.ko 2024-01-15T17:50:27.997 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/llc/ 2024-01-15T17:50:27.997 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/llc/llc.ko 2024-01-15T17:50:27.997 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 268265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/llc/llc2.ko 2024-01-15T17:50:27.997 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/mac802154/ 2024-01-15T17:50:27.997 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 290441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/mac802154/mac802154.ko 2024-01-15T17:50:27.997 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/mpls/ 2024-01-15T17:50:27.997 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/mpls/mpls_gso.ko 2024-01-15T17:50:27.997 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ 2024-01-15T17:50:27.997 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ 2024-01-15T17:50:27.997 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 118801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set.ko 2024-01-15T17:50:27.997 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-01-15T17:50:27.997 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-01-15T17:50:27.997 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26369 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-01-15T17:50:27.997 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 100297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-01-15T17:50:27.997 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 100633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-01-15T17:50:27.998 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 102881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-01-15T17:50:27.998 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 102649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-01-15T17:50:27.998 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 112017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-01-15T17:50:27.998 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 59585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-01-15T17:50:27.998 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 108761 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-01-15T17:50:27.998 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 112297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-01-15T17:50:27.998 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 112649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-01-15T17:50:27.998 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 110753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-01-15T17:50:27.998 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 113633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-01-15T17:50:27.998 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-01-15T17:50:27.998 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ 2024-01-15T17:50:27.998 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 412561 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs.ko 2024-01-15T17:50:27.998 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-01-15T17:50:27.998 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-01-15T17:50:27.998 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-01-15T17:50:27.999 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-01-15T17:50:27.999 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-01-15T17:50:27.999 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-01-15T17:50:27.999 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-01-15T17:50:27.999 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-01-15T17:50:27.999 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-01-15T17:50:27.999 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-01-15T17:50:27.999 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-01-15T17:50:27.999 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-01-15T17:50:27.999 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-01-15T17:50:27.999 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 49265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conncount.ko 2024-01-15T17:50:27.999 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 411489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack.ko 2024-01-15T17:50:27.999 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-01-15T17:50:27.999 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-01-15T17:50:27.999 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 33761 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-01-15T17:50:28.000 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 118193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_h323.ko 2024-01-15T17:50:28.000 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_irc.ko 2024-01-15T17:50:28.000 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-01-15T17:50:28.000 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 108105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-01-15T17:50:28.000 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 33681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-01-15T17:50:28.000 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_sane.ko 2024-01-15T17:50:28.000 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 66625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_sip.ko 2024-01-15T17:50:28.000 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11233 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-01-15T17:50:28.000 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-01-15T17:50:28.000 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_dup_netdev.ko 2024-01-15T17:50:28.000 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 91737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_flow_table.ko 2024-01-15T17:50:28.000 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_flow_table_inet.ko 2024-01-15T17:50:28.000 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 37297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_log_syslog.ko 2024-01-15T17:50:28.000 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 96545 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat.ko 2024-01-15T17:50:28.000 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_amanda.ko 2024-01-15T17:50:28.001 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_ftp.ko 2024-01-15T17:50:28.001 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_irc.ko 2024-01-15T17:50:28.001 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_sip.ko 2024-01-15T17:50:28.001 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_tftp.ko 2024-01-15T17:50:28.001 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 44297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_synproxy_core.ko 2024-01-15T17:50:28.001 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 614025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_tables.ko 2024-01-15T17:50:28.016 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 42609 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink.ko 2024-01-15T17:50:28.016 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_acct.ko 2024-01-15T17:50:28.016 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_cttimeout.ko 2024-01-15T17:50:28.016 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 39785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_log.ko 2024-01-15T17:50:28.016 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_osf.ko 2024-01-15T17:50:28.016 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 49401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_queue.ko 2024-01-15T17:50:28.016 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_chain_nat.ko 2024-01-15T17:50:28.017 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 33209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_compat.ko 2024-01-15T17:50:28.017 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 40769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_ct.ko 2024-01-15T17:50:28.017 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_dup_netdev.ko 2024-01-15T17:50:28.017 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11513 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_fib.ko 2024-01-15T17:50:28.017 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_fwd_netdev.ko 2024-01-15T17:50:28.017 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_hash.ko 2024-01-15T17:50:28.017 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_limit.ko 2024-01-15T17:50:28.017 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_log.ko 2024-01-15T17:50:28.017 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_masq.ko 2024-01-15T17:50:28.017 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_nat.ko 2024-01-15T17:50:28.017 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_numgen.ko 2024-01-15T17:50:28.017 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_osf.ko 2024-01-15T17:50:28.017 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_queue.ko 2024-01-15T17:50:28.017 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_redir.ko 2024-01-15T17:50:28.017 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_reject.ko 2024-01-15T17:50:28.018 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_reject_inet.ko 2024-01-15T17:50:28.018 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_socket.ko 2024-01-15T17:50:28.018 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20369 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_synproxy.ko 2024-01-15T17:50:28.018 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_tproxy.ko 2024-01-15T17:50:28.018 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 89313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/x_tables.ko 2024-01-15T17:50:28.018 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_AUDIT.ko 2024-01-15T17:50:28.018 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CHECKSUM.ko 2024-01-15T17:50:28.018 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CLASSIFY.ko 2024-01-15T17:50:28.018 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-01-15T17:50:28.018 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 18169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CT.ko 2024-01-15T17:50:28.018 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_DSCP.ko 2024-01-15T17:50:28.018 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_HL.ko 2024-01-15T17:50:28.018 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13369 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_HMARK.ko 2024-01-15T17:50:28.018 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 38873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_IDLETIMER.ko 2024-01-15T17:50:28.019 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13889 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_LED.ko 2024-01-15T17:50:28.019 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_LOG.ko 2024-01-15T17:50:28.019 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_MASQUERADE.ko 2024-01-15T17:50:28.019 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NETMAP.ko 2024-01-15T17:50:28.019 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NFLOG.ko 2024-01-15T17:50:28.019 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NFQUEUE.ko 2024-01-15T17:50:28.019 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_RATEEST.ko 2024-01-15T17:50:28.019 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_REDIRECT.ko 2024-01-15T17:50:28.019 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_SECMARK.ko 2024-01-15T17:50:28.019 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TCPMSS.ko 2024-01-15T17:50:28.019 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-01-15T17:50:28.019 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TEE.ko 2024-01-15T17:50:28.019 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TPROXY.ko 2024-01-15T17:50:28.019 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TRACE.ko 2024-01-15T17:50:28.020 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_addrtype.ko 2024-01-15T17:50:28.020 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13897 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_bpf.ko 2024-01-15T17:50:28.020 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cgroup.ko 2024-01-15T17:50:28.020 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cluster.ko 2024-01-15T17:50:28.020 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_comment.ko 2024-01-15T17:50:28.020 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connbytes.ko 2024-01-15T17:50:28.020 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connlabel.ko 2024-01-15T17:50:28.020 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connlimit.ko 2024-01-15T17:50:28.020 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connmark.ko 2024-01-15T17:50:28.020 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_conntrack.ko 2024-01-15T17:50:28.020 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cpu.ko 2024-01-15T17:50:28.020 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_dccp.ko 2024-01-15T17:50:28.020 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_devgroup.ko 2024-01-15T17:50:28.020 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_dscp.ko 2024-01-15T17:50:28.021 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ecn.ko 2024-01-15T17:50:28.021 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_esp.ko 2024-01-15T17:50:28.021 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 43297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_hashlimit.ko 2024-01-15T17:50:28.021 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_helper.ko 2024-01-15T17:50:28.021 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_hl.ko 2024-01-15T17:50:28.021 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ipcomp.ko 2024-01-15T17:50:28.021 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_iprange.ko 2024-01-15T17:50:28.021 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ipvs.ko 2024-01-15T17:50:28.021 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_l2tp.ko 2024-01-15T17:50:28.021 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8001 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_length.ko 2024-01-15T17:50:28.021 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_limit.ko 2024-01-15T17:50:28.021 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7545 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_mac.ko 2024-01-15T17:50:28.021 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_mark.ko 2024-01-15T17:50:28.021 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_multiport.ko 2024-01-15T17:50:28.022 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_nat.ko 2024-01-15T17:50:28.022 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_nfacct.ko 2024-01-15T17:50:28.022 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_osf.ko 2024-01-15T17:50:28.022 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_owner.ko 2024-01-15T17:50:28.022 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_physdev.ko 2024-01-15T17:50:28.022 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_pkttype.ko 2024-01-15T17:50:28.022 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_policy.ko 2024-01-15T17:50:28.022 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_quota.ko 2024-01-15T17:50:28.022 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_rateest.ko 2024-01-15T17:50:28.022 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_realm.ko 2024-01-15T17:50:28.022 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 36457 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_recent.ko 2024-01-15T17:50:28.022 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_sctp.ko 2024-01-15T17:50:28.022 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_set.ko 2024-01-15T17:50:28.022 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_socket.ko 2024-01-15T17:50:28.023 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_state.ko 2024-01-15T17:50:28.023 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_statistic.ko 2024-01-15T17:50:28.023 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_string.ko 2024-01-15T17:50:28.023 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 8553 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_tcpmss.ko 2024-01-15T17:50:28.023 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_tcpudp.ko 2024-01-15T17:50:28.023 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_time.ko 2024-01-15T17:50:28.023 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_u32.ko 2024-01-15T17:50:28.023 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netlink/ 2024-01-15T17:50:28.023 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 14537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netlink/netlink_diag.ko 2024-01-15T17:50:28.023 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/nsh/ 2024-01-15T17:50:28.023 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 12425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/nsh/nsh.ko 2024-01-15T17:50:28.023 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/ 2024-01-15T17:50:28.023 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 383481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/openvswitch.ko 2024-01-15T17:50:28.023 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-geneve.ko 2024-01-15T17:50:28.023 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-gre.ko 2024-01-15T17:50:28.024 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-vxlan.ko 2024-01-15T17:50:28.024 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/packet/ 2024-01-15T17:50:28.024 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/packet/af_packet_diag.ko 2024-01-15T17:50:28.024 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/phonet/ 2024-01-15T17:50:28.024 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 112057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/phonet/phonet.ko 2024-01-15T17:50:28.024 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 54857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/phonet/pn_pep.ko 2024-01-15T17:50:28.024 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/rds/ 2024-01-15T17:50:28.024 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 252953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds.ko 2024-01-15T17:50:28.024 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 164937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds_rdma.ko 2024-01-15T17:50:28.024 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 68161 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds_tcp.ko 2024-01-15T17:50:28.024 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/rxrpc/ 2024-01-15T17:50:28.024 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 818777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/rxrpc/rxrpc.ko 2024-01-15T17:50:28.024 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/ 2024-01-15T17:50:28.025 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 32241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_csum.ko 2024-01-15T17:50:28.025 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_gact.ko 2024-01-15T17:50:28.025 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26897 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_ipt.ko 2024-01-15T17:50:28.025 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 34097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_mirred.ko 2024-01-15T17:50:28.025 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_nat.ko 2024-01-15T17:50:28.025 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 43449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_pedit.ko 2024-01-15T17:50:28.025 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 31961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_police.ko 2024-01-15T17:50:28.025 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_simple.ko 2024-01-15T17:50:28.025 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 35297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_skbedit.ko 2024-01-15T17:50:28.025 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 36665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_vlan.ko 2024-01-15T17:50:28.025 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_basic.ko 2024-01-15T17:50:28.025 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 34353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_bpf.ko 2024-01-15T17:50:28.025 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 19281 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_cgroup.ko 2024-01-15T17:50:28.025 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 31489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_flow.ko 2024-01-15T17:50:28.026 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 32377 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_fw.ko 2024-01-15T17:50:28.026 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 40009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_route.ko 2024-01-15T17:50:28.026 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 61529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_u32.ko 2024-01-15T17:50:28.026 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_cmp.ko 2024-01-15T17:50:28.026 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 13409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_ipset.ko 2024-01-15T17:50:28.026 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 33313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_meta.ko 2024-01-15T17:50:28.026 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7897 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_nbyte.ko 2024-01-15T17:50:28.026 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_text.ko 2024-01-15T17:50:28.026 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_u32.ko 2024-01-15T17:50:28.026 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 21769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_choke.ko 2024-01-15T17:50:28.026 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_codel.ko 2024-01-15T17:50:28.026 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 34681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_drr.ko 2024-01-15T17:50:28.026 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 35817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_fq.ko 2024-01-15T17:50:28.026 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 31617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_fq_codel.ko 2024-01-15T17:50:28.026 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 35545 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_gred.ko 2024-01-15T17:50:28.027 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 49937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_hfsc.ko 2024-01-15T17:50:28.027 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 22089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_hhf.ko 2024-01-15T17:50:28.027 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 66577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_htb.ko 2024-01-15T17:50:28.027 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_ingress.ko 2024-01-15T17:50:28.027 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 30385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_mqprio.ko 2024-01-15T17:50:28.027 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_mqprio_lib.ko 2024-01-15T17:50:28.027 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_multiq.ko 2024-01-15T17:50:28.027 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 36249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_netem.ko 2024-01-15T17:50:28.027 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 27497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_pie.ko 2024-01-15T17:50:28.027 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_plug.ko 2024-01-15T17:50:28.027 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 24841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_prio.ko 2024-01-15T17:50:28.027 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 48625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_qfq.ko 2024-01-15T17:50:28.027 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 33017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_red.ko 2024-01-15T17:50:28.027 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 26881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_sfb.ko 2024-01-15T17:50:28.027 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 31409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_sfq.ko 2024-01-15T17:50:28.027 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_tbf.ko 2024-01-15T17:50:28.028 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 28153 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_teql.ko 2024-01-15T17:50:28.028 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/ 2024-01-15T17:50:28.028 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 932169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/sctp.ko 2024-01-15T17:50:28.028 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/sctp_diag.ko 2024-01-15T17:50:28.028 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/ 2024-01-15T17:50:28.028 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/ 2024-01-15T17:50:28.028 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 336777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-01-15T17:50:28.028 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 54993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-01-15T17:50:28.028 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 1518417 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/sunrpc.ko 2024-01-15T17:50:28.028 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/xprtrdma/ 2024-01-15T17:50:28.028 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 708281 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-01-15T17:50:28.028 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/ 2024-01-15T17:50:28.028 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 10489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/diag.ko 2024-01-15T17:50:28.028 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 866345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/tipc.ko 2024-01-15T17:50:28.029 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/unix/ 2024-01-15T17:50:28.029 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 16033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/unix/unix_diag.ko 2024-01-15T17:50:28.029 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/ 2024-01-15T17:50:28.029 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 86417 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-01-15T17:50:28.029 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 68049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-01-15T17:50:28.029 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 98017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock.ko 2024-01-15T17:50:28.029 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 11177 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock_diag.ko 2024-01-15T17:50:28.029 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 17761 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock_loopback.ko 2024-01-15T17:50:28.029 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/x25/ 2024-01-15T17:50:28.029 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 125409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/x25/x25.ko 2024-01-15T17:50:28.029 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm/ 2024-01-15T17:50:28.029 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm/xfrm_algo.ko 2024-01-15T17:50:28.029 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 20649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm/xfrm_ipcomp.ko 2024-01-15T17:50:28.029 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 104161 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm/xfrm_user.ko 2024-01-15T17:50:28.030 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/virt/ 2024-01-15T17:50:28.030 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/virt/lib/ 2024-01-15T17:50:28.030 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 15537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/virt/lib/irqbypass.ko 2024-01-15T17:50:28.030 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 641009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.alias 2024-01-15T17:50:28.030 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 625033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.alias.bin 2024-01-15T17:50:28.030 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 7677 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.builtin 2024-01-15T17:50:28.030 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 23634 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.builtin.alias.bin 2024-01-15T17:50:28.030 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 9916 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.builtin.bin 2024-01-15T17:50:28.030 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 70653 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.builtin.modinfo 2024-01-15T17:50:28.030 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 147734 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.dep 2024-01-15T17:50:28.030 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 234045 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.dep.bin 2024-01-15T17:50:28.030 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 210 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.devname 2024-01-15T17:50:28.030 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 72797 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.order 2024-01-15T17:50:28.030 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 879 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.softdep 2024-01-15T17:50:28.030 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 255175 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.symbols 2024-01-15T17:50:28.031 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 309826 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.symbols.bin 2024-01-15T17:50:28.031 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./usr/ 2024-01-15T17:50:28.031 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./usr/share/ 2024-01-15T17:50:28.031 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./usr/share/doc/ 2024-01-15T17:50:28.031 INFO:teuthology.orchestra.run.smithi027.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./usr/share/doc/linux-image-6.7.0-g2668949e0971/ 2024-01-15T17:50:28.031 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 171 2024-01-15 16:53 ./usr/share/doc/linux-image-6.7.0-g2668949e0971/changelog.Debian.gz 2024-01-15T17:50:28.031 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r-- root/root 691 2024-01-15 16:53 ./usr/share/doc/linux-image-6.7.0-g2668949e0971/copyright 2024-01-15T17:50:28.032 DEBUG:teuthology.task.kernel:get_image_version: 6.7.0-g2668949e0971 2024-01-15T17:50:28.032 INFO:teuthology.task.kernel:searching for kernel 6.7.0-g2668949e0971 2024-01-15T17:50:28.032 DEBUG:teuthology.orchestra.run.smithi027:> egrep '(submenu|menuentry.*6.7.0-g2668949e0971).*{' /boot/grub/grub.cfg 2024-01-15T17:50:28.064 INFO:teuthology.orchestra.run.smithi027.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-01-15T17:50:28.065 INFO:teuthology.orchestra.run.smithi027.stdout: menuentry 'Ubuntu, with Linux 6.7.0-g2668949e0971' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.7.0-g2668949e0971-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-01-15T17:50:28.065 INFO:teuthology.orchestra.run.smithi027.stdout: menuentry 'Ubuntu, with Linux 6.7.0-g2668949e0971 (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.7.0-g2668949e0971-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-01-15T17:50:28.066 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2024-01-15T17:50:28.066 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.7.0-g2668949e0971 2024-01-15T17:50:28.066 DEBUG:teuthology.orchestra.run.smithi027:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2024-01-15T17:50:28.108 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi027.front.sepia.ceph.com to complete... 2024-01-15T17:50:28.561 INFO:teuthology.orchestra.run.smithi027.stderr:Sourcing file `/etc/default/grub' 2024-01-15T17:50:28.562 INFO:teuthology.orchestra.run.smithi027.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-01-15T17:50:28.575 INFO:teuthology.orchestra.run.smithi027.stderr:Generating grub configuration file ... 2024-01-15T17:50:28.757 INFO:teuthology.orchestra.run.smithi027.stderr:Found linux image: /boot/vmlinuz-6.7.0-g2668949e0971 2024-01-15T17:50:28.763 INFO:teuthology.orchestra.run.smithi027.stderr:Found initrd image: /boot/initrd.img-6.7.0-g2668949e0971 2024-01-15T17:50:29.037 INFO:teuthology.orchestra.run.smithi027.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-01-15T17:50:29.042 INFO:teuthology.orchestra.run.smithi027.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-01-15T17:50:29.086 INFO:teuthology.orchestra.run.smithi027.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-01-15T17:50:29.092 INFO:teuthology.orchestra.run.smithi027.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-01-15T17:50:29.232 INFO:teuthology.orchestra.run.smithi027.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-01-15T17:50:29.232 INFO:teuthology.orchestra.run.smithi027.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-01-15T17:50:29.232 INFO:teuthology.orchestra.run.smithi027.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-01-15T17:50:29.249 INFO:teuthology.orchestra.run.smithi027.stderr:done 2024-01-15T17:50:32.408 INFO:teuthology.orchestra.run.smithi093.stderr:Sourcing file `/etc/default/grub' 2024-01-15T17:50:32.426 INFO:teuthology.orchestra.run.smithi093.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-01-15T17:50:32.446 INFO:teuthology.orchestra.run.smithi093.stderr:Generating grub configuration file ... 2024-01-15T17:50:32.822 INFO:teuthology.orchestra.run.smithi093.stderr:Found linux image: /boot/vmlinuz-6.7.0-g2668949e0971 2024-01-15T17:50:32.840 INFO:teuthology.orchestra.run.smithi093.stderr:Found initrd image: /boot/initrd.img-6.7.0-g2668949e0971 2024-01-15T17:50:33.160 INFO:teuthology.orchestra.run.smithi093.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-01-15T17:50:33.167 INFO:teuthology.orchestra.run.smithi093.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-01-15T17:50:33.232 INFO:teuthology.orchestra.run.smithi093.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-01-15T17:50:33.237 INFO:teuthology.orchestra.run.smithi093.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-01-15T17:50:33.385 INFO:teuthology.orchestra.run.smithi093.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-01-15T17:50:33.385 INFO:teuthology.orchestra.run.smithi093.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-01-15T17:50:33.385 INFO:teuthology.orchestra.run.smithi093.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-01-15T17:50:33.428 INFO:teuthology.orchestra.run.smithi093.stderr:done 2024-01-15T17:50:33.594 DEBUG:teuthology.orchestra.run.smithi093:> dpkg-deb -c /tmp/linux-image.deb 2024-01-15T17:50:33.602 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./ 2024-01-15T17:50:33.603 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./boot/ 2024-01-15T17:50:33.603 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 4444641 2024-01-15 16:53 ./boot/System.map-6.7.0-g2668949e0971 2024-01-15T17:50:33.614 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 170149 2024-01-15 16:53 ./boot/config-6.7.0-g2668949e0971 2024-01-15T17:50:33.614 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11441152 2024-01-15 16:53 ./boot/vmlinuz-6.7.0-g2668949e0971 2024-01-15T17:50:33.634 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./etc/ 2024-01-15T17:50:33.634 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./etc/kernel/ 2024-01-15T17:50:33.634 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./etc/kernel/postinst.d/ 2024-01-15T17:50:33.634 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./etc/kernel/postrm.d/ 2024-01-15T17:50:33.634 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./etc/kernel/preinst.d/ 2024-01-15T17:50:33.634 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./etc/kernel/prerm.d/ 2024-01-15T17:50:33.634 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/ 2024-01-15T17:50:33.635 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/ 2024-01-15T17:50:33.635 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/ 2024-01-15T17:50:33.635 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/ 2024-01-15T17:50:33.635 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/ 2024-01-15T17:50:33.635 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/ 2024-01-15T17:50:33.635 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/ 2024-01-15T17:50:33.635 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 490345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/aesni-intel.ko 2024-01-15T17:50:33.635 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 25329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-01-15T17:50:33.635 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 29529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-01-15T17:50:33.635 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-01-15T17:50:33.635 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 93113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-01-15T17:50:33.635 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 106729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-01-15T17:50:33.635 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 152241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-01-15T17:50:33.637 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12553 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-01-15T17:50:33.637 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-01-15T17:50:33.638 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 83889 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-01-15T17:50:33.638 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-01-15T17:50:33.638 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 47225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-01-15T17:50:33.638 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 47209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-avx2.ko 2024-01-15T17:50:33.638 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 51801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-01-15T17:50:33.638 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 39585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-01-15T17:50:33.638 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 42529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-01-15T17:50:33.638 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 58585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-01-15T17:50:33.638 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 50497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-01-15T17:50:33.638 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 32065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-01-15T17:50:33.638 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-01-15T17:50:33.638 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/ 2024-01-15T17:50:33.638 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/ 2024-01-15T17:50:33.638 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/mce/ 2024-01-15T17:50:33.639 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 40113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-01-15T17:50:33.639 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpuid.ko 2024-01-15T17:50:33.639 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/msr.ko 2024-01-15T17:50:33.639 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/ 2024-01-15T17:50:33.639 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 353705 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm-amd.ko 2024-01-15T17:50:33.639 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 881649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm-intel.ko 2024-01-15T17:50:33.639 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 2276625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm.ko 2024-01-15T17:50:33.655 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/ 2024-01-15T17:50:33.655 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 53721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/af_alg.ko 2024-01-15T17:50:33.655 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23513 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/algif_hash.ko 2024-01-15T17:50:33.655 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/algif_skcipher.ko 2024-01-15T17:50:33.655 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/ansi_cprng.ko 2024-01-15T17:50:33.655 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19641 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/anubis.ko 2024-01-15T17:50:33.655 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/arc4.ko 2024-01-15T17:50:33.655 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/asymmetric_keys/ 2024-01-15T17:50:33.655 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2024-01-15T17:50:33.655 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/ 2024-01-15T17:50:33.656 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_memcpy.ko 2024-01-15T17:50:33.656 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_pq.ko 2024-01-15T17:50:33.656 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21561 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_raid6_recov.ko 2024-01-15T17:50:33.656 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_tx.ko 2024-01-15T17:50:33.656 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_xor.ko 2024-01-15T17:50:33.656 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/raid6test.ko 2024-01-15T17:50:33.656 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/authenc.ko 2024-01-15T17:50:33.656 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/authencesn.ko 2024-01-15T17:50:33.656 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/blake2b_generic.ko 2024-01-15T17:50:33.656 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/blowfish_common.ko 2024-01-15T17:50:33.656 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/blowfish_generic.ko 2024-01-15T17:50:33.656 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 38809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/camellia_generic.ko 2024-01-15T17:50:33.656 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 27321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast5_generic.ko 2024-01-15T17:50:33.656 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast6_generic.ko 2024-01-15T17:50:33.657 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast_common.ko 2024-01-15T17:50:33.657 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 35273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/ccm.ko 2024-01-15T17:50:33.657 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/cmac.ko 2024-01-15T17:50:33.657 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/crc32_generic.ko 2024-01-15T17:50:33.657 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 47769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/cryptd.ko 2024-01-15T17:50:33.657 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 39505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_engine.ko 2024-01-15T17:50:33.657 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_simd.ko 2024-01-15T17:50:33.657 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 25041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_user.ko 2024-01-15T17:50:33.657 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/cts.ko 2024-01-15T17:50:33.657 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/des_generic.ko 2024-01-15T17:50:33.657 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/dh_generic.ko 2024-01-15T17:50:33.657 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11001 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/echainiv.ko 2024-01-15T17:50:33.657 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 25321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/essiv.ko 2024-01-15T17:50:33.657 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/fcrypt.ko 2024-01-15T17:50:33.658 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26897 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/khazad.ko 2024-01-15T17:50:33.658 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/lrw.ko 2024-01-15T17:50:33.658 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11513 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/lz4.ko 2024-01-15T17:50:33.658 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/lz4hc.ko 2024-01-15T17:50:33.658 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11697 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/md4.ko 2024-01-15T17:50:33.658 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/michael_mic.ko 2024-01-15T17:50:33.658 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/pcbc.ko 2024-01-15T17:50:33.658 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/pcrypt.ko 2024-01-15T17:50:33.658 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/rmd160.ko 2024-01-15T17:50:33.658 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 29393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/seed.ko 2024-01-15T17:50:33.658 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 27081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/serpent_generic.ko 2024-01-15T17:50:33.658 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 149481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/tcrypt.ko 2024-01-15T17:50:33.658 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/tea.ko 2024-01-15T17:50:33.658 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/twofish_common.ko 2024-01-15T17:50:33.659 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/twofish_generic.ko 2024-01-15T17:50:33.659 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/vmac.ko 2024-01-15T17:50:33.659 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 36185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/wp512.ko 2024-01-15T17:50:33.659 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/xcbc.ko 2024-01-15T17:50:33.659 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/xor.ko 2024-01-15T17:50:33.659 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/xts.ko 2024-01-15T17:50:33.659 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/crypto/xxhash_generic.ko 2024-01-15T17:50:33.659 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ 2024-01-15T17:50:33.659 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/ 2024-01-15T17:50:33.659 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_extlog.ko 2024-01-15T17:50:33.659 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_ipmi.ko 2024-01-15T17:50:33.659 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_pad.ko 2024-01-15T17:50:33.659 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/apei/ 2024-01-15T17:50:33.660 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 31241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/apei/einj.ko 2024-01-15T17:50:33.660 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/ec_sys.ko 2024-01-15T17:50:33.660 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/platform_profile.ko 2024-01-15T17:50:33.660 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/sbs.ko 2024-01-15T17:50:33.660 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/sbshc.ko 2024-01-15T17:50:33.660 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 104729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/video.ko 2024-01-15T17:50:33.660 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ 2024-01-15T17:50:33.660 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/acard-ahci.ko 2024-01-15T17:50:33.660 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 89649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ahci.ko 2024-01-15T17:50:33.660 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ahci_platform.ko 2024-01-15T17:50:33.660 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 88401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/libahci.ko 2024-01-15T17:50:33.660 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 34985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/libahci_platform.ko 2024-01-15T17:50:33.661 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_acpi.ko 2024-01-15T17:50:33.661 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ali.ko 2024-01-15T17:50:33.661 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 33505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_amd.ko 2024-01-15T17:50:33.661 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_artop.ko 2024-01-15T17:50:33.661 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_atiixp.ko 2024-01-15T17:50:33.661 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_atp867x.ko 2024-01-15T17:50:33.661 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cmd640.ko 2024-01-15T17:50:33.661 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 25185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cmd64x.ko 2024-01-15T17:50:33.661 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cypress.ko 2024-01-15T17:50:33.661 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_efar.ko 2024-01-15T17:50:33.661 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt366.ko 2024-01-15T17:50:33.661 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 33977 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt37x.ko 2024-01-15T17:50:33.661 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt3x2n.ko 2024-01-15T17:50:33.661 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt3x3.ko 2024-01-15T17:50:33.662 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_it8213.ko 2024-01-15T17:50:33.662 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 35273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_it821x.ko 2024-01-15T17:50:33.662 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12889 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_jmicron.ko 2024-01-15T17:50:33.662 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 51481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_legacy.ko 2024-01-15T17:50:33.662 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_marvell.ko 2024-01-15T17:50:33.662 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16281 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_mpiix.ko 2024-01-15T17:50:33.662 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_netcell.ko 2024-01-15T17:50:33.662 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ninja32.ko 2024-01-15T17:50:33.662 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ns87410.ko 2024-01-15T17:50:33.662 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17361 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ns87415.ko 2024-01-15T17:50:33.662 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_oldpiix.ko 2024-01-15T17:50:33.662 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15377 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_opti.ko 2024-01-15T17:50:33.662 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_optidma.ko 2024-01-15T17:50:33.662 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 32585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pcmcia.ko 2024-01-15T17:50:33.663 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 37793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pdc2027x.ko 2024-01-15T17:50:33.663 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20561 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-01-15T17:50:33.663 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_piccolo.ko 2024-01-15T17:50:33.663 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_radisys.ko 2024-01-15T17:50:33.663 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_rdc.ko 2024-01-15T17:50:33.663 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_rz1000.ko 2024-01-15T17:50:33.663 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sch.ko 2024-01-15T17:50:33.663 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_serverworks.ko 2024-01-15T17:50:33.663 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23457 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sil680.ko 2024-01-15T17:50:33.663 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19697 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sl82c105.ko 2024-01-15T17:50:33.663 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_triflex.ko 2024-01-15T17:50:33.663 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_via.ko 2024-01-15T17:50:33.663 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pdc_adma.ko 2024-01-15T17:50:33.663 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 29121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_inic162x.ko 2024-01-15T17:50:33.663 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 83345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_mv.ko 2024-01-15T17:50:33.663 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 70393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_nv.ko 2024-01-15T17:50:33.664 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 37377 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_promise.ko 2024-01-15T17:50:33.664 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_qstor.ko 2024-01-15T17:50:33.664 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 29273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sil.ko 2024-01-15T17:50:33.664 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 38337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sil24.ko 2024-01-15T17:50:33.664 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sis.ko 2024-01-15T17:50:33.664 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_svw.ko 2024-01-15T17:50:33.664 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 44793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sx4.ko 2024-01-15T17:50:33.664 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_uli.ko 2024-01-15T17:50:33.664 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 35065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_via.ko 2024-01-15T17:50:33.664 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_vsc.ko 2024-01-15T17:50:33.664 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/auxdisplay/ 2024-01-15T17:50:33.664 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/auxdisplay/ks0108.ko 2024-01-15T17:50:33.664 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/bcma/ 2024-01-15T17:50:33.664 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 124473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/bcma/bcma.ko 2024-01-15T17:50:33.665 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/ 2024-01-15T17:50:33.665 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/aoe/ 2024-01-15T17:50:33.665 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 113753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/aoe/aoe.ko 2024-01-15T17:50:33.665 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/drbd/ 2024-01-15T17:50:33.665 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 965305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/drbd/drbd.ko 2024-01-15T17:50:33.665 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 173353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/floppy.ko 2024-01-15T17:50:33.665 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/mtip32xx/ 2024-01-15T17:50:33.665 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 94297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-01-15T17:50:33.665 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 116321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/nbd.ko 2024-01-15T17:50:33.665 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/null_blk/ 2024-01-15T17:50:33.665 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 110737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/null_blk/null_blk.ko 2024-01-15T17:50:33.665 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 98281 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/pktcdvd.ko 2024-01-15T17:50:33.665 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 263769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/rbd.ko 2024-01-15T17:50:33.665 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkback/ 2024-01-15T17:50:33.665 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 98185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-01-15T17:50:33.665 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/zram/ 2024-01-15T17:50:33.666 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 60217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/zram/zram.ko 2024-01-15T17:50:33.666 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ 2024-01-15T17:50:33.666 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30705 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/applicom.ko 2024-01-15T17:50:33.666 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hangcheck-timer.ko 2024-01-15T17:50:33.666 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/ 2024-01-15T17:50:33.666 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13889 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/amd-rng.ko 2024-01-15T17:50:33.666 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/intel-rng.ko 2024-01-15T17:50:33.666 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-01-15T17:50:33.666 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/via-rng.ko 2024-01-15T17:50:33.666 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/virtio-rng.ko 2024-01-15T17:50:33.666 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ 2024-01-15T17:50:33.666 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30713 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-01-15T17:50:33.666 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 148609 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-01-15T17:50:33.666 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-01-15T17:50:33.666 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 139313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_si.ko 2024-01-15T17:50:33.666 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 70969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-01-15T17:50:33.731 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 53513 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-01-15T17:50:33.731 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 36321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/lp.ko 2024-01-15T17:50:33.731 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/mwave/ 2024-01-15T17:50:33.731 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 84097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/mwave/mwave.ko 2024-01-15T17:50:33.731 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 25873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/nvram.ko 2024-01-15T17:50:33.732 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 32169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ppdev.ko 2024-01-15T17:50:33.732 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 51113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tlclk.ko 2024-01-15T17:50:33.732 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/ 2024-01-15T17:50:33.732 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_atmel.ko 2024-01-15T17:50:33.732 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2024-01-15T17:50:33.732 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2024-01-15T17:50:33.732 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 32985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2024-01-15T17:50:33.732 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_infineon.ko 2024-01-15T17:50:33.732 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22697 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_nsc.ko 2024-01-15T17:50:33.732 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23561 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/xen-tpmfront.ko 2024-01-15T17:50:33.732 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/xillybus/ 2024-01-15T17:50:33.732 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17001 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/xillybus/xillybus_class.ko 2024-01-15T17:50:33.732 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 50945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/xillybus/xillybus_core.ko 2024-01-15T17:50:33.733 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/xillybus/xillybus_pcie.ko 2024-01-15T17:50:33.733 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/clk/ 2024-01-15T17:50:33.733 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/clk/clk-palmas.ko 2024-01-15T17:50:33.733 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 47761 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/clk/clk-si5351.ko 2024-01-15T17:50:33.733 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11457 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/clk/clk-twl6040.ko 2024-01-15T17:50:33.733 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/clk/clk-wm831x.ko 2024-01-15T17:50:33.733 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/ 2024-01-15T17:50:33.733 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-01-15T17:50:33.733 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/p4-clockmod.ko 2024-01-15T17:50:33.733 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 25297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/speedstep-lib.ko 2024-01-15T17:50:33.733 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ 2024-01-15T17:50:33.733 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ 2024-01-15T17:50:33.733 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 70737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-01-15T17:50:33.734 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 246841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ccp.ko 2024-01-15T17:50:33.734 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/ 2024-01-15T17:50:33.734 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/ 2024-01-15T17:50:33.734 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_common/ 2024-01-15T17:50:33.734 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 517033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-01-15T17:50:33.734 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2024-01-15T17:50:33.734 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-01-15T17:50:33.734 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/padlock-aes.ko 2024-01-15T17:50:33.734 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/padlock-sha.ko 2024-01-15T17:50:33.734 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/virtio/ 2024-01-15T17:50:33.734 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 70233 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-01-15T17:50:33.734 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/dca/ 2024-01-15T17:50:33.734 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 29785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/dca/dca.ko 2024-01-15T17:50:33.734 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ 2024-01-15T17:50:33.735 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/dw/ 2024-01-15T17:50:33.735 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/dw/dw_dmac.ko 2024-01-15T17:50:33.735 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ioat/ 2024-01-15T17:50:33.735 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 122569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ioat/ioatdma.ko 2024-01-15T17:50:33.735 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/ 2024-01-15T17:50:33.735 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 79953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/amd64_edac.ko 2024-01-15T17:50:33.735 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 35769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/e752x_edac.ko 2024-01-15T17:50:33.735 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 63617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/edac_mce_amd.ko 2024-01-15T17:50:33.735 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i3000_edac.ko 2024-01-15T17:50:33.735 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i3200_edac.ko 2024-01-15T17:50:33.735 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i5100_edac.ko 2024-01-15T17:50:33.735 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 25657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i5400_edac.ko 2024-01-15T17:50:33.735 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i7300_edac.ko 2024-01-15T17:50:33.735 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 52041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i7core_edac.ko 2024-01-15T17:50:33.736 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i82975x_edac.ko 2024-01-15T17:50:33.736 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/ie31200_edac.ko 2024-01-15T17:50:33.736 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 56825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/sb_edac.ko 2024-01-15T17:50:33.736 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/x38_edac.ko 2024-01-15T17:50:33.736 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/extcon/ 2024-01-15T17:50:33.736 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/extcon/extcon-gpio.ko 2024-01-15T17:50:33.736 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 27185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/extcon/extcon-max14577.ko 2024-01-15T17:50:33.736 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/extcon/extcon-max77693.ko 2024-01-15T17:50:33.736 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/extcon/extcon-max8997.ko 2024-01-15T17:50:33.736 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/extcon/extcon-palmas.ko 2024-01-15T17:50:33.736 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23713 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/extcon/extcon-rt8973a.ko 2024-01-15T17:50:33.736 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 27785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/extcon/extcon-sm5502.ko 2024-01-15T17:50:33.736 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/ 2024-01-15T17:50:33.737 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 34977 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/dmi-sysfs.ko 2024-01-15T17:50:33.737 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/efi/ 2024-01-15T17:50:33.737 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/efi/efi-pstore.ko 2024-01-15T17:50:33.737 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/iscsi_ibft.ko 2024-01-15T17:50:33.737 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/ 2024-01-15T17:50:33.737 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-adp5520.ko 2024-01-15T17:50:33.737 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20153 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-amd8111.ko 2024-01-15T17:50:33.737 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-arizona.ko 2024-01-15T17:50:33.737 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-da9052.ko 2024-01-15T17:50:33.737 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-da9055.ko 2024-01-15T17:50:33.737 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-dln2.ko 2024-01-15T17:50:33.737 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 77849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-f7188x.ko 2024-01-15T17:50:33.737 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-generic.ko 2024-01-15T17:50:33.737 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-ich.ko 2024-01-15T17:50:33.738 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-janz-ttl.ko 2024-01-15T17:50:33.738 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-kempld.ko 2024-01-15T17:50:33.738 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-lp3943.ko 2024-01-15T17:50:33.738 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-max7300.ko 2024-01-15T17:50:33.738 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-max730x.ko 2024-01-15T17:50:33.738 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-max732x.ko 2024-01-15T17:50:33.738 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 25145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-ml-ioh.ko 2024-01-15T17:50:33.738 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 47201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-pca953x.ko 2024-01-15T17:50:33.738 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-pcf857x.ko 2024-01-15T17:50:33.738 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14377 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-rdc321x.ko 2024-01-15T17:50:33.738 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-sch.ko 2024-01-15T17:50:33.738 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-sch311x.ko 2024-01-15T17:50:33.738 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-tps65912.ko 2024-01-15T17:50:33.738 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-twl4030.ko 2024-01-15T17:50:33.738 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-twl6040.ko 2024-01-15T17:50:33.739 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-viperboard.ko 2024-01-15T17:50:33.739 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15377 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-vx855.ko 2024-01-15T17:50:33.739 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16281 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-wm831x.ko 2024-01-15T17:50:33.739 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-wm8350.ko 2024-01-15T17:50:33.739 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-wm8994.ko 2024-01-15T17:50:33.739 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/ 2024-01-15T17:50:33.739 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-a4tech.ko 2024-01-15T17:50:33.739 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 39513 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-apple.ko 2024-01-15T17:50:33.739 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-appleir.ko 2024-01-15T17:50:33.739 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-aureal.ko 2024-01-15T17:50:33.739 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-axff.ko 2024-01-15T17:50:33.739 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-belkin.ko 2024-01-15T17:50:33.739 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-cherry.ko 2024-01-15T17:50:33.739 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-chicony.ko 2024-01-15T17:50:33.740 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 60761 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-cp2112.ko 2024-01-15T17:50:33.740 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-cypress.ko 2024-01-15T17:50:33.740 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-dr.ko 2024-01-15T17:50:33.740 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10697 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-elecom.ko 2024-01-15T17:50:33.740 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-elo.ko 2024-01-15T17:50:33.740 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13457 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-emsff.ko 2024-01-15T17:50:33.740 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-ezkey.ko 2024-01-15T17:50:33.740 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gaff.ko 2024-01-15T17:50:33.740 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-generic.ko 2024-01-15T17:50:33.740 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gt683r.ko 2024-01-15T17:50:33.740 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gyration.ko 2024-01-15T17:50:33.740 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10161 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtek-kbd.ko 2024-01-15T17:50:33.740 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10457 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtek-mouse.ko 2024-01-15T17:50:33.741 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtekff.ko 2024-01-15T17:50:33.741 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-hyperv.ko 2024-01-15T17:50:33.741 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-icade.ko 2024-01-15T17:50:33.741 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-kensington.ko 2024-01-15T17:50:33.741 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8545 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-keytouch.ko 2024-01-15T17:50:33.741 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-kye.ko 2024-01-15T17:50:33.741 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lcpower.ko 2024-01-15T17:50:33.741 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-led.ko 2024-01-15T17:50:33.741 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 42945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lenovo.ko 2024-01-15T17:50:33.741 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30697 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lg-g15.ko 2024-01-15T17:50:33.741 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 58073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech-dj.ko 2024-01-15T17:50:33.741 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 100649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-01-15T17:50:33.742 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 80993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech.ko 2024-01-15T17:50:33.742 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-magicmouse.ko 2024-01-15T17:50:33.742 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-microsoft.ko 2024-01-15T17:50:33.742 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10361 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-monterey.ko 2024-01-15T17:50:33.742 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 50273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-multitouch.ko 2024-01-15T17:50:33.742 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 32593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-ntrig.ko 2024-01-15T17:50:33.742 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-ortek.ko 2024-01-15T17:50:33.742 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-penmount.ko 2024-01-15T17:50:33.742 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-petalynx.ko 2024-01-15T17:50:33.742 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 72937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-picolcd.ko 2024-01-15T17:50:33.742 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-pl.ko 2024-01-15T17:50:33.742 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-plantronics.ko 2024-01-15T17:50:33.742 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8417 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-primax.ko 2024-01-15T17:50:33.743 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 35393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-rmi.ko 2024-01-15T17:50:33.743 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-arvo.ko 2024-01-15T17:50:33.743 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-common.ko 2024-01-15T17:50:33.743 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 31425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-isku.ko 2024-01-15T17:50:33.743 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 29729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-kone.ko 2024-01-15T17:50:33.743 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 31129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-01-15T17:50:33.743 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-konepure.ko 2024-01-15T17:50:33.743 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 29489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-01-15T17:50:33.743 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-lua.ko 2024-01-15T17:50:33.743 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 29313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-pyra.ko 2024-01-15T17:50:33.743 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 27937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-ryos.ko 2024-01-15T17:50:33.743 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-savu.ko 2024-01-15T17:50:33.743 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat.ko 2024-01-15T17:50:33.743 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-saitek.ko 2024-01-15T17:50:33.743 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-samsung.ko 2024-01-15T17:50:33.743 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 36817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sensor-hub.ko 2024-01-15T17:50:33.744 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sjoy.ko 2024-01-15T17:50:33.744 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 51249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sony.ko 2024-01-15T17:50:33.744 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-speedlink.ko 2024-01-15T17:50:33.744 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 25297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-steelseries.ko 2024-01-15T17:50:33.744 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sunplus.ko 2024-01-15T17:50:33.744 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-thrustmaster.ko 2024-01-15T17:50:33.744 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-tivo.ko 2024-01-15T17:50:33.744 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-tmff.ko 2024-01-15T17:50:33.744 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-topseed.ko 2024-01-15T17:50:33.744 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-twinhan.ko 2024-01-15T17:50:33.744 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 71073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-uclogic.ko 2024-01-15T17:50:33.744 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-waltop.ko 2024-01-15T17:50:33.744 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 143473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-wiimote.ko 2024-01-15T17:50:33.744 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8153 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-xinmo.ko 2024-01-15T17:50:33.745 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-zpff.ko 2024-01-15T17:50:33.745 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-zydacron.ko 2024-01-15T17:50:33.745 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 269281 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid.ko 2024-01-15T17:50:33.745 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 32321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/uhid.ko 2024-01-15T17:50:33.745 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/usbhid/ 2024-01-15T17:50:33.745 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 125785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/usbhid/usbhid.ko 2024-01-15T17:50:33.745 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/usbhid/usbkbd.ko 2024-01-15T17:50:33.745 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/usbhid/usbmouse.ko 2024-01-15T17:50:33.745 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 217873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/wacom.ko 2024-01-15T17:50:33.745 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/ 2024-01-15T17:50:33.745 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 64201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_balloon.ko 2024-01-15T17:50:33.745 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 89625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_utils.ko 2024-01-15T17:50:33.745 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 305321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_vmbus.ko 2024-01-15T17:50:33.745 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ 2024-01-15T17:50:33.746 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 66521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/abituguru.ko 2024-01-15T17:50:33.746 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 61393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/abituguru3.ko 2024-01-15T17:50:33.746 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 34673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/acpi_power_meter.ko 2024-01-15T17:50:33.746 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ad7414.ko 2024-01-15T17:50:33.746 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ad7418.ko 2024-01-15T17:50:33.746 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 25257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adc128d818.ko 2024-01-15T17:50:33.746 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 31729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1025.ko 2024-01-15T17:50:33.746 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 97977 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1026.ko 2024-01-15T17:50:33.746 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1029.ko 2024-01-15T17:50:33.746 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 45553 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1031.ko 2024-01-15T17:50:33.746 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm9240.ko 2024-01-15T17:50:33.746 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ads7828.ko 2024-01-15T17:50:33.746 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7410.ko 2024-01-15T17:50:33.746 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7411.ko 2024-01-15T17:50:33.746 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 74409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7462.ko 2024-01-15T17:50:33.747 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 39585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7470.ko 2024-01-15T17:50:33.747 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 69401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7475.ko 2024-01-15T17:50:33.747 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7x10.ko 2024-01-15T17:50:33.747 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 39073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/amc6821.ko 2024-01-15T17:50:33.747 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 55201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/applesmc.ko 2024-01-15T17:50:33.747 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 49913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asb100.ko 2024-01-15T17:50:33.747 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 58617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asc7621.ko 2024-01-15T17:50:33.747 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 40825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asus_atk0110.ko 2024-01-15T17:50:33.747 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/atxp1.ko 2024-01-15T17:50:33.747 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/coretemp.ko 2024-01-15T17:50:33.747 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/da9052-hwmon.ko 2024-01-15T17:50:33.747 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/da9055-hwmon.ko 2024-01-15T17:50:33.747 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 44441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-01-15T17:50:33.747 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 99601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/dme1737.ko 2024-01-15T17:50:33.748 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ds1621.ko 2024-01-15T17:50:33.748 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ds620.ko 2024-01-15T17:50:33.748 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 29777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc1403.ko 2024-01-15T17:50:33.748 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 33185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc2103.ko 2024-01-15T17:50:33.748 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc6w201.ko 2024-01-15T17:50:33.748 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 66985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f71805f.ko 2024-01-15T17:50:33.748 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 94689 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f71882fg.ko 2024-01-15T17:50:33.748 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 41401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f75375s.ko 2024-01-15T17:50:33.748 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/fam15h_power.ko 2024-01-15T17:50:33.748 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 55449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/fschmd.ko 2024-01-15T17:50:33.748 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/g760a.ko 2024-01-15T17:50:33.748 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/g762.ko 2024-01-15T17:50:33.748 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 46801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/gl518sm.ko 2024-01-15T17:50:33.748 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 45529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/gl520sm.ko 2024-01-15T17:50:33.749 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/hih6130.ko 2024-01-15T17:50:33.749 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/hwmon-vid.ko 2024-01-15T17:50:33.749 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11553 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/i5500_temp.ko 2024-01-15T17:50:33.749 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/i5k_amb.ko 2024-01-15T17:50:33.749 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 34529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ibmaem.ko 2024-01-15T17:50:33.749 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ibmpex.ko 2024-01-15T17:50:33.749 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ina209.ko 2024-01-15T17:50:33.749 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ina2xx.ko 2024-01-15T17:50:33.749 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 130449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/it87.ko 2024-01-15T17:50:33.749 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/jc42.ko 2024-01-15T17:50:33.749 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/k10temp.ko 2024-01-15T17:50:33.749 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13553 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/k8temp.ko 2024-01-15T17:50:33.749 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lineage-pem.ko 2024-01-15T17:50:33.749 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 48601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm63.ko 2024-01-15T17:50:33.750 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16697 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm73.ko 2024-01-15T17:50:33.750 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 36777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm75.ko 2024-01-15T17:50:33.750 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm77.ko 2024-01-15T17:50:33.750 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 34593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm78.ko 2024-01-15T17:50:33.750 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 31745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm80.ko 2024-01-15T17:50:33.750 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16889 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm83.ko 2024-01-15T17:50:33.750 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 72833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm85.ko 2024-01-15T17:50:33.750 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 46153 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm87.ko 2024-01-15T17:50:33.750 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 37913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm90.ko 2024-01-15T17:50:33.750 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm92.ko 2024-01-15T17:50:33.750 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 108841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm93.ko 2024-01-15T17:50:33.750 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 33393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95234.ko 2024-01-15T17:50:33.750 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95241.ko 2024-01-15T17:50:33.750 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95245.ko 2024-01-15T17:50:33.751 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc2945.ko 2024-01-15T17:50:33.751 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4151.ko 2024-01-15T17:50:33.751 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4215.ko 2024-01-15T17:50:33.751 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14889 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4222.ko 2024-01-15T17:50:33.751 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4245.ko 2024-01-15T17:50:33.751 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4260.ko 2024-01-15T17:50:33.751 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4261.ko 2024-01-15T17:50:33.751 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 33489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max16065.ko 2024-01-15T17:50:33.751 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max1619.ko 2024-01-15T17:50:33.751 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max1668.ko 2024-01-15T17:50:33.751 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max197.ko 2024-01-15T17:50:33.751 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 27129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6639.ko 2024-01-15T17:50:33.751 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6650.ko 2024-01-15T17:50:33.751 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 35777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6697.ko 2024-01-15T17:50:33.751 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/mc13783-adc.ko 2024-01-15T17:50:33.752 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/mcp3021.ko 2024-01-15T17:50:33.752 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2024-01-15T17:50:33.752 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 44153 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6683.ko 2024-01-15T17:50:33.752 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 119689 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6775-core.ko 2024-01-15T17:50:33.752 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 60233 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6775.ko 2024-01-15T17:50:33.752 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 59737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct7802.ko 2024-01-15T17:50:33.752 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 82337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pc87360.ko 2024-01-15T17:50:33.752 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 54433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pc87427.ko 2024-01-15T17:50:33.752 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pcf8591.ko 2024-01-15T17:50:33.752 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ 2024-01-15T17:50:33.752 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-01-15T17:50:33.752 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-01-15T17:50:33.752 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-01-15T17:50:33.753 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max16064.ko 2024-01-15T17:50:33.753 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max34440.ko 2024-01-15T17:50:33.753 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max8688.ko 2024-01-15T17:50:33.753 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-01-15T17:50:33.753 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 82577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-01-15T17:50:33.753 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-01-15T17:50:33.753 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-01-15T17:50:33.753 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-01-15T17:50:33.753 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-01-15T17:50:33.753 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/powr1220.ko 2024-01-15T17:50:33.753 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch5627.ko 2024-01-15T17:50:33.753 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch5636.ko 2024-01-15T17:50:33.753 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 29249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch56xx-common.ko 2024-01-15T17:50:33.753 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 32137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sht15.ko 2024-01-15T17:50:33.754 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sht21.ko 2024-01-15T17:50:33.754 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/shtc1.ko 2024-01-15T17:50:33.754 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 40105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sis5595.ko 2024-01-15T17:50:33.754 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47b397.ko 2024-01-15T17:50:33.754 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 39193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47m1.ko 2024-01-15T17:50:33.754 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 37385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47m192.ko 2024-01-15T17:50:33.754 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26553 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/thmc50.ko 2024-01-15T17:50:33.754 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp102.ko 2024-01-15T17:50:33.754 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp103.ko 2024-01-15T17:50:33.754 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp401.ko 2024-01-15T17:50:33.754 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp421.ko 2024-01-15T17:50:33.754 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/via-cputemp.ko 2024-01-15T17:50:33.754 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 41945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/via686a.ko 2024-01-15T17:50:33.754 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 52457 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/vt1211.ko 2024-01-15T17:50:33.755 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 51545 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/vt8231.ko 2024-01-15T17:50:33.755 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 57449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83627ehf.ko 2024-01-15T17:50:33.755 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 78825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83627hf.ko 2024-01-15T17:50:33.755 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 71345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83781d.ko 2024-01-15T17:50:33.755 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 63225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83791d.ko 2024-01-15T17:50:33.755 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 71321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83792d.ko 2024-01-15T17:50:33.755 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 101473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83793.ko 2024-01-15T17:50:33.755 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 119505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83795.ko 2024-01-15T17:50:33.755 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83l785ts.ko 2024-01-15T17:50:33.755 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 35625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83l786ng.ko 2024-01-15T17:50:33.755 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/wm831x-hwmon.ko 2024-01-15T17:50:33.755 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/wm8350-hwmon.ko 2024-01-15T17:50:33.755 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/ 2024-01-15T17:50:33.755 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/ 2024-01-15T17:50:33.755 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-01-15T17:50:33.756 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-01-15T17:50:33.756 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/ 2024-01-15T17:50:33.756 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 27969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ali1535.ko 2024-01-15T17:50:33.756 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ali1563.ko 2024-01-15T17:50:33.756 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 27777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2024-01-15T17:50:33.756 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-01-15T17:50:33.756 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-01-15T17:50:33.756 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-01-15T17:50:33.756 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15153 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2024-01-15T17:50:33.756 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 6929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-01-15T17:50:33.756 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2024-01-15T17:50:33.756 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 50401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-01-15T17:50:33.756 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-01-15T17:50:33.756 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20001 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-01-15T17:50:33.757 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-01-15T17:50:33.757 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-dln2.ko 2024-01-15T17:50:33.757 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-gpio.ko 2024-01-15T17:50:33.757 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 62681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-i801.ko 2024-01-15T17:50:33.757 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-isch.ko 2024-01-15T17:50:33.757 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 47889 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-01-15T17:50:33.757 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-kempld.ko 2024-01-15T17:50:33.757 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-01-15T17:50:33.757 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-01-15T17:50:33.757 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ocores.ko 2024-01-15T17:50:33.757 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 25145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-parport.ko 2024-01-15T17:50:33.757 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-01-15T17:50:33.757 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 47241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-01-15T17:50:33.757 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2024-01-15T17:50:33.758 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-01-15T17:50:33.758 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-01-15T17:50:33.758 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-sis5595.ko 2024-01-15T17:50:33.758 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 29361 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-sis630.ko 2024-01-15T17:50:33.758 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-01-15T17:50:33.758 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2024-01-15T17:50:33.758 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-01-15T17:50:33.758 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-via.ko 2024-01-15T17:50:33.758 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-01-15T17:50:33.758 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-01-15T17:50:33.758 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 47657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-xiic.ko 2024-01-15T17:50:33.758 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-mux.ko 2024-01-15T17:50:33.758 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-smbus.ko 2024-01-15T17:50:33.758 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 27073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-stub.ko 2024-01-15T17:50:33.759 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/muxes/ 2024-01-15T17:50:33.759 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2024-01-15T17:50:33.759 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13457 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2024-01-15T17:50:33.759 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 29737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2024-01-15T17:50:33.759 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ 2024-01-15T17:50:33.759 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ 2024-01-15T17:50:33.759 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 273609 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_cm.ko 2024-01-15T17:50:33.759 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 883681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_core.ko 2024-01-15T17:50:33.759 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 69273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_umad.ko 2024-01-15T17:50:33.759 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 324329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-01-15T17:50:33.759 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 97921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/iw_cm.ko 2024-01-15T17:50:33.759 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 282601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/rdma_cm.ko 2024-01-15T17:50:33.759 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 66137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-01-15T17:50:33.759 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/ 2024-01-15T17:50:33.759 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/cxgb4/ 2024-01-15T17:50:33.760 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 493217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-01-15T17:50:33.760 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx4/ 2024-01-15T17:50:33.760 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 420561 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-01-15T17:50:33.760 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx5/ 2024-01-15T17:50:33.760 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 741449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-01-15T17:50:33.760 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mthca/ 2024-01-15T17:50:33.760 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 296881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-01-15T17:50:33.760 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/ocrdma/ 2024-01-15T17:50:33.760 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 153801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2024-01-15T17:50:33.760 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/usnic/ 2024-01-15T17:50:33.760 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 168969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2024-01-15T17:50:33.760 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ 2024-01-15T17:50:33.760 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ipoib/ 2024-01-15T17:50:33.760 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 263081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-01-15T17:50:33.761 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/iser/ 2024-01-15T17:50:33.761 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 100353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-01-15T17:50:33.761 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/isert/ 2024-01-15T17:50:33.761 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 111425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-01-15T17:50:33.761 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srp/ 2024-01-15T17:50:33.761 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 128753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-01-15T17:50:33.761 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srpt/ 2024-01-15T17:50:33.761 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 129529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-01-15T17:50:33.761 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/ 2024-01-15T17:50:33.761 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/evbug.ko 2024-01-15T17:50:33.761 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/ff-memless.ko 2024-01-15T17:50:33.761 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/ 2024-01-15T17:50:33.761 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-01-15T17:50:33.762 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/fm801-gp.ko 2024-01-15T17:50:33.762 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 41241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/gameport.ko 2024-01-15T17:50:33.762 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 41801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/joydev.ko 2024-01-15T17:50:33.762 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/ 2024-01-15T17:50:33.762 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/adp5520-keys.ko 2024-01-15T17:50:33.762 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12705 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/adp5588-keys.ko 2024-01-15T17:50:33.762 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/adp5589-keys.ko 2024-01-15T17:50:33.762 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2024-01-15T17:50:33.762 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 36665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/gpio_keys.ko 2024-01-15T17:50:33.762 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2024-01-15T17:50:33.762 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/lkkbd.ko 2024-01-15T17:50:33.762 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/lm8323.ko 2024-01-15T17:50:33.762 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/lm8333.ko 2024-01-15T17:50:33.762 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/matrix_keypad.ko 2024-01-15T17:50:33.762 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/max7359_keypad.ko 2024-01-15T17:50:33.763 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/mcs_touchkey.ko 2024-01-15T17:50:33.763 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2024-01-15T17:50:33.763 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/newtonkbd.ko 2024-01-15T17:50:33.763 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/opencores-kbd.ko 2024-01-15T17:50:33.763 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14161 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/qt1070.ko 2024-01-15T17:50:33.763 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/qt2160.ko 2024-01-15T17:50:33.763 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/samsung-keypad.ko 2024-01-15T17:50:33.763 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/stowaway.ko 2024-01-15T17:50:33.763 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/sunkbd.ko 2024-01-15T17:50:33.763 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/tca6416-keypad.ko 2024-01-15T17:50:33.763 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16177 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/tca8418_keypad.ko 2024-01-15T17:50:33.763 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/twl4030_keypad.ko 2024-01-15T17:50:33.763 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/xtkbd.ko 2024-01-15T17:50:33.763 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/matrix-keymap.ko 2024-01-15T17:50:33.763 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/ 2024-01-15T17:50:33.764 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 34329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/appletouch.ko 2024-01-15T17:50:33.764 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 34177 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/bcm5974.ko 2024-01-15T17:50:33.764 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 123417 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/cyapatp.ko 2024-01-15T17:50:33.764 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 89449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/elan_i2c.ko 2024-01-15T17:50:33.764 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/gpio_mouse.ko 2024-01-15T17:50:33.764 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 335953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/psmouse.ko 2024-01-15T17:50:33.764 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/sermouse.ko 2024-01-15T17:50:33.764 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-01-15T17:50:33.764 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/synaptics_usb.ko 2024-01-15T17:50:33.764 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/vsxxxaa.ko 2024-01-15T17:50:33.764 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/rmi4/ 2024-01-15T17:50:33.764 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 109873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/rmi4/rmi_core.ko 2024-01-15T17:50:33.764 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/ 2024-01-15T17:50:33.764 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/altera_ps2.ko 2024-01-15T17:50:33.764 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/arc_ps2.ko 2024-01-15T17:50:33.765 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/ct82c710.ko 2024-01-15T17:50:33.765 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-01-15T17:50:33.765 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/parkbd.ko 2024-01-15T17:50:33.765 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/pcips2.ko 2024-01-15T17:50:33.765 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20153 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/ps2mult.ko 2024-01-15T17:50:33.765 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24153 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/serio_raw.ko 2024-01-15T17:50:33.765 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17977 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/serport.ko 2024-01-15T17:50:33.765 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/sparse-keymap.ko 2024-01-15T17:50:33.765 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/ 2024-01-15T17:50:33.765 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-88pm860x.ko 2024-01-15T17:50:33.765 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-adp5520.ko 2024-01-15T17:50:33.765 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 59753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-bd2802.ko 2024-01-15T17:50:33.765 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-blinkm.ko 2024-01-15T17:50:33.765 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-da903x.ko 2024-01-15T17:50:33.766 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-da9052.ko 2024-01-15T17:50:33.766 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-gpio.ko 2024-01-15T17:50:33.766 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lm3530.ko 2024-01-15T17:50:33.766 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lm3533.ko 2024-01-15T17:50:33.766 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lm355x.ko 2024-01-15T17:50:33.766 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lm3642.ko 2024-01-15T17:50:33.766 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lp3944.ko 2024-01-15T17:50:33.766 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11233 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lp8788.ko 2024-01-15T17:50:33.766 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-max8997.ko 2024-01-15T17:50:33.766 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12545 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-mc13783.ko 2024-01-15T17:50:33.766 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-menf21bmc.ko 2024-01-15T17:50:33.766 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-pca9532.ko 2024-01-15T17:50:33.766 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-pca955x.ko 2024-01-15T17:50:33.766 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-pca963x.ko 2024-01-15T17:50:33.766 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 34433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-ss4200.ko 2024-01-15T17:50:33.767 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-tca6507.ko 2024-01-15T17:50:33.767 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-wm831x-status.ko 2024-01-15T17:50:33.767 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-wm8350.ko 2024-01-15T17:50:33.767 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ 2024-01-15T17:50:33.767 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-01-15T17:50:33.767 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-01-15T17:50:33.767 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-01-15T17:50:33.767 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-01-15T17:50:33.767 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-01-15T17:50:33.767 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-01-15T17:50:33.767 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-01-15T17:50:33.767 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11889 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-01-15T17:50:33.767 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-01-15T17:50:33.767 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/ 2024-01-15T17:50:33.767 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/bcache/ 2024-01-15T17:50:33.768 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 647193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/bcache/bcache.ko 2024-01-15T17:50:33.768 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 40577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-bio-prison.ko 2024-01-15T17:50:33.768 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 93409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-bufio.ko 2024-01-15T17:50:33.768 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 47529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-cache-smq.ko 2024-01-15T17:50:33.768 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 155897 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-cache.ko 2024-01-15T17:50:33.768 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 107257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-crypt.ko 2024-01-15T17:50:33.768 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-delay.ko 2024-01-15T17:50:33.768 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 53857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-era.ko 2024-01-15T17:50:33.768 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-flakey.ko 2024-01-15T17:50:33.768 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 41529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-log-userspace.ko 2024-01-15T17:50:33.768 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 32737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-log.ko 2024-01-15T17:50:33.768 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 49545 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-mirror.ko 2024-01-15T17:50:33.768 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 78033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-multipath.ko 2024-01-15T17:50:33.768 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-queue-length.ko 2024-01-15T17:50:33.768 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 71073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-raid.ko 2024-01-15T17:50:33.769 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 36297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-region-hash.ko 2024-01-15T17:50:33.769 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-round-robin.ko 2024-01-15T17:50:33.769 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-service-time.ko 2024-01-15T17:50:33.769 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 109529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-snapshot.ko 2024-01-15T17:50:33.769 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19361 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-switch.ko 2024-01-15T17:50:33.769 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 181865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-thin-pool.ko 2024-01-15T17:50:33.769 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 52073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-verity.ko 2024-01-15T17:50:33.769 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8561 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-zero.ko 2024-01-15T17:50:33.769 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/faulty.ko 2024-01-15T17:50:33.769 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19369 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/linear.ko 2024-01-15T17:50:33.769 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 32417 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/multipath.ko 2024-01-15T17:50:33.769 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/persistent-data/ 2024-01-15T17:50:33.769 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 199937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-01-15T17:50:33.769 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 37401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid0.ko 2024-01-15T17:50:33.769 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 113001 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid1.ko 2024-01-15T17:50:33.770 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 158657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid10.ko 2024-01-15T17:50:33.770 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 393777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid456.ko 2024-01-15T17:50:33.770 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/ 2024-01-15T17:50:33.770 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/ 2024-01-15T17:50:33.770 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 178137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptbase.ko 2024-01-15T17:50:33.770 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 63321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptctl.ko 2024-01-15T17:50:33.770 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 46337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptfc.ko 2024-01-15T17:50:33.770 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 37225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptlan.ko 2024-01-15T17:50:33.770 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 123321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptsas.ko 2024-01-15T17:50:33.770 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 81009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptscsih.ko 2024-01-15T17:50:33.770 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 52953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptspi.ko 2024-01-15T17:50:33.770 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/ 2024-01-15T17:50:33.770 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/88pm800.ko 2024-01-15T17:50:33.770 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12761 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/88pm805.ko 2024-01-15T17:50:33.771 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/88pm80x.ko 2024-01-15T17:50:33.771 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/arizona-i2c.ko 2024-01-15T17:50:33.771 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 123721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/arizona.ko 2024-01-15T17:50:33.771 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/bcm590xx.ko 2024-01-15T17:50:33.771 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/cros_ec_dev.ko 2024-01-15T17:50:33.771 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 34937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/dln2.ko 2024-01-15T17:50:33.771 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16689 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/janz-cmodio.ko 2024-01-15T17:50:33.771 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 53585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/kempld-core.ko 2024-01-15T17:50:33.771 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 27529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lm3533-core.ko 2024-01-15T17:50:33.771 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lm3533-ctrlbank.ko 2024-01-15T17:50:33.771 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lp3943.ko 2024-01-15T17:50:33.771 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 57057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lpc_ich.ko 2024-01-15T17:50:33.771 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lpc_sch.ko 2024-01-15T17:50:33.771 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/max8907.ko 2024-01-15T17:50:33.771 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 29857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/mc13xxx-core.ko 2024-01-15T17:50:33.772 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/mc13xxx-i2c.ko 2024-01-15T17:50:33.772 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/menf21bmc.ko 2024-01-15T17:50:33.772 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/pcf50633-adc.ko 2024-01-15T17:50:33.772 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9153 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/pcf50633-gpio.ko 2024-01-15T17:50:33.772 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 34873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/pcf50633.ko 2024-01-15T17:50:33.772 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/rdc321x-southbridge.ko 2024-01-15T17:50:33.772 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/retu-mfd.ko 2024-01-15T17:50:33.772 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 80425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/si476x-core.ko 2024-01-15T17:50:33.772 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 48457 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/sm501.ko 2024-01-15T17:50:33.772 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/ti_am335x_tscadc.ko 2024-01-15T17:50:33.772 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 51137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/tps65010.ko 2024-01-15T17:50:33.772 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/tps6507x.ko 2024-01-15T17:50:33.772 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/viperboard.ko 2024-01-15T17:50:33.772 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/vx855.ko 2024-01-15T17:50:33.772 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/wl1273-core.ko 2024-01-15T17:50:33.773 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 44345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/wm8994.ko 2024-01-15T17:50:33.773 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/ 2024-01-15T17:50:33.773 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/ad525x_dpot-i2c.ko 2024-01-15T17:50:33.773 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 48489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/ad525x_dpot.ko 2024-01-15T17:50:33.773 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/altera-stapl/ 2024-01-15T17:50:33.773 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 46585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-01-15T17:50:33.773 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/apds9802als.ko 2024-01-15T17:50:33.773 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 36729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/apds990x.ko 2024-01-15T17:50:33.773 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 45593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/bh1770glc.ko 2024-01-15T17:50:33.773 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/c2port/ 2024-01-15T17:50:33.773 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2024-01-15T17:50:33.773 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 35281 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/c2port/core.ko 2024-01-15T17:50:33.773 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/cb710/ 2024-01-15T17:50:33.773 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 25161 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/cb710/cb710.ko 2024-01-15T17:50:33.773 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/ds1682.ko 2024-01-15T17:50:33.773 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8897 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/dummy-irq.ko 2024-01-15T17:50:33.774 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/echo/ 2024-01-15T17:50:33.774 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/echo/echo.ko 2024-01-15T17:50:33.774 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/ 2024-01-15T17:50:33.774 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 38785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/at24.ko 2024-01-15T17:50:33.774 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-01-15T17:50:33.774 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/max6875.ko 2024-01-15T17:50:33.774 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 33009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/enclosure.ko 2024-01-15T17:50:33.774 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/genwqe/ 2024-01-15T17:50:33.774 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 152201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/genwqe/genwqe_card.ko 2024-01-15T17:50:33.774 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/hmc6352.ko 2024-01-15T17:50:33.774 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 31257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/hpilo.ko 2024-01-15T17:50:33.774 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/ibmasm/ 2024-01-15T17:50:33.774 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 72601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/ibmasm/ibmasm.ko 2024-01-15T17:50:33.774 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/ics932s401.ko 2024-01-15T17:50:33.774 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/isl29003.ko 2024-01-15T17:50:33.775 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/isl29020.ko 2024-01-15T17:50:33.775 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/ 2024-01-15T17:50:33.775 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 42513 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-01-15T17:50:33.775 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-01-15T17:50:33.775 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/ 2024-01-15T17:50:33.775 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 125945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei-me.ko 2024-01-15T17:50:33.775 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 56521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei-txe.ko 2024-01-15T17:50:33.775 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 313641 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei.ko 2024-01-15T17:50:33.775 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 31489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/phantom.ko 2024-01-15T17:50:33.775 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/ti-st/ 2024-01-15T17:50:33.775 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 74377 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/ti-st/st_drv.ko 2024-01-15T17:50:33.775 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tifm_7xx1.ko 2024-01-15T17:50:33.775 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tifm_core.ko 2024-01-15T17:50:33.775 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tsl2550.ko 2024-01-15T17:50:33.776 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 48441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_balloon.ko 2024-01-15T17:50:33.776 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_vmci/ 2024-01-15T17:50:33.776 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 183609 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-01-15T17:50:33.776 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ 2024-01-15T17:50:33.776 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/bonding/ 2024-01-15T17:50:33.776 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 490849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/bonding/bonding.ko 2024-01-15T17:50:33.776 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/ 2024-01-15T17:50:33.776 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/b53/ 2024-01-15T17:50:33.776 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 106033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/b53/b53_common.ko 2024-01-15T17:50:33.776 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 40305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/bcm-sf2.ko 2024-01-15T17:50:33.776 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6060.ko 2024-01-15T17:50:33.776 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6xxx/ 2024-01-15T17:50:33.776 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 455065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-01-15T17:50:33.776 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dummy.ko 2024-01-15T17:50:33.777 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/eql.ko 2024-01-15T17:50:33.777 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ 2024-01-15T17:50:33.777 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/ 2024-01-15T17:50:33.777 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 45785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-01-15T17:50:33.777 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 42105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-01-15T17:50:33.777 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 130553 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-01-15T17:50:33.777 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 68849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-01-15T17:50:33.777 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/ 2024-01-15T17:50:33.777 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 37649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/8390.ko 2024-01-15T17:50:33.777 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 59689 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-01-15T17:50:33.777 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 27201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-01-15T17:50:33.777 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 99049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-01-15T17:50:33.777 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/adaptec/ 2024-01-15T17:50:33.777 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 54521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-01-15T17:50:33.778 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/agere/ 2024-01-15T17:50:33.778 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 69137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/agere/et131x.ko 2024-01-15T17:50:33.778 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/alteon/ 2024-01-15T17:50:33.778 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 59585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-01-15T17:50:33.778 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/altera/ 2024-01-15T17:50:33.778 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 41545 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-01-15T17:50:33.778 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/ 2024-01-15T17:50:33.778 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 54609 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-01-15T17:50:33.778 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 49289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-01-15T17:50:33.778 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 87297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-01-15T17:50:33.778 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/ 2024-01-15T17:50:33.778 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/alx/ 2024-01-15T17:50:33.778 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 97337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-01-15T17:50:33.778 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1c/ 2024-01-15T17:50:33.778 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 102497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-01-15T17:50:33.779 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1e/ 2024-01-15T17:50:33.779 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 90345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-01-15T17:50:33.779 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/ 2024-01-15T17:50:33.779 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 81569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-01-15T17:50:33.779 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 69633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-01-15T17:50:33.779 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/ 2024-01-15T17:50:33.779 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 78401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-01-15T17:50:33.779 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 176761 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-01-15T17:50:33.779 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2024-01-15T17:50:33.779 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 1421529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-01-15T17:50:33.779 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 128569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-01-15T17:50:33.779 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/genet/ 2024-01-15T17:50:33.779 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 131337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-01-15T17:50:33.779 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 314553 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-01-15T17:50:33.780 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/ 2024-01-15T17:50:33.780 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/bna/ 2024-01-15T17:50:33.780 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 323745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-01-15T17:50:33.780 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/ 2024-01-15T17:50:33.780 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb/ 2024-01-15T17:50:33.780 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 193497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-01-15T17:50:33.780 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2024-01-15T17:50:33.780 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 347257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-01-15T17:50:33.780 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2024-01-15T17:50:33.780 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 1044793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-01-15T17:50:33.780 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2024-01-15T17:50:33.780 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 166193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-01-15T17:50:33.780 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2024-01-15T17:50:33.780 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 34881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-01-15T17:50:33.781 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/ 2024-01-15T17:50:33.781 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/enic/ 2024-01-15T17:50:33.781 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 204345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-01-15T17:50:33.781 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/ 2024-01-15T17:50:33.781 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/ 2024-01-15T17:50:33.781 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 69449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-01-15T17:50:33.781 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 61489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-01-15T17:50:33.781 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 141225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-01-15T17:50:33.781 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 51497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-01-15T17:50:33.781 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 62817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-01-15T17:50:33.781 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 31369 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-01-15T17:50:33.781 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/ 2024-01-15T17:50:33.781 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 60873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-01-15T17:50:33.782 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 62833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-01-15T17:50:33.782 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 45721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dnet.ko 2024-01-15T17:50:33.782 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 25185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ec_bhf.ko 2024-01-15T17:50:33.782 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/ 2024-01-15T17:50:33.782 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/benet/ 2024-01-15T17:50:33.782 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 281137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-01-15T17:50:33.782 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 49481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ethoc.ko 2024-01-15T17:50:33.782 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 51737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/fealnx.ko 2024-01-15T17:50:33.782 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/fujitsu/ 2024-01-15T17:50:33.782 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 45345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2024-01-15T17:50:33.782 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ 2024-01-15T17:50:33.782 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 90369 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e100.ko 2024-01-15T17:50:33.782 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000/ 2024-01-15T17:50:33.782 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 295481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-01-15T17:50:33.782 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000e/ 2024-01-15T17:50:33.783 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 584497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-01-15T17:50:33.783 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/fm10k/ 2024-01-15T17:50:33.783 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 246209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-01-15T17:50:33.783 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/i40e/ 2024-01-15T17:50:33.783 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 937673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-01-15T17:50:33.783 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/iavf/ 2024-01-15T17:50:33.783 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 287305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-01-15T17:50:33.783 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igb/ 2024-01-15T17:50:33.783 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 514569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-01-15T17:50:33.870 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igbvf/ 2024-01-15T17:50:33.870 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 93497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-01-15T17:50:33.870 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbe/ 2024-01-15T17:50:33.870 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 734353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-01-15T17:50:33.871 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbevf/ 2024-01-15T17:50:33.871 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 151769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-01-15T17:50:33.871 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 86921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/jme.ko 2024-01-15T17:50:33.871 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/ 2024-01-15T17:50:33.871 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-01-15T17:50:33.871 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 102289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/skge.ko 2024-01-15T17:50:33.871 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 122601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-01-15T17:50:33.871 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/ 2024-01-15T17:50:33.871 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/ 2024-01-15T17:50:33.871 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 727473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-01-15T17:50:33.871 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 278961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-01-15T17:50:33.871 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/ 2024-01-15T17:50:33.871 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2024-01-15T17:50:33.872 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 1118201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-01-15T17:50:33.872 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel/ 2024-01-15T17:50:33.872 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 53105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel/ks8842.ko 2024-01-15T17:50:33.872 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 41041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2024-01-15T17:50:33.872 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2024-01-15T17:50:33.872 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 95665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-01-15T17:50:33.872 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/ 2024-01-15T17:50:33.872 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/myri10ge/ 2024-01-15T17:50:33.872 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 108649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-01-15T17:50:33.872 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/ 2024-01-15T17:50:33.872 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 73441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-01-15T17:50:33.872 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 47777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-01-15T17:50:33.872 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/neterion/ 2024-01-15T17:50:33.873 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 150905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-01-15T17:50:33.873 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/nvidia/ 2024-01-15T17:50:33.873 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 136721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-01-15T17:50:33.873 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/ 2024-01-15T17:50:33.873 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 50481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-01-15T17:50:33.873 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 43145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-01-15T17:50:33.873 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/ 2024-01-15T17:50:33.873 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/netxen/ 2024-01-15T17:50:33.873 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 227897 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-01-15T17:50:33.873 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 87105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-01-15T17:50:33.873 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2024-01-15T17:50:33.873 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 671937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-01-15T17:50:33.873 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rdc/ 2024-01-15T17:50:33.874 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 43017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-01-15T17:50:33.874 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/ 2024-01-15T17:50:33.874 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 67385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-01-15T17:50:33.874 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 83985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-01-15T17:50:33.874 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 29321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/atp.ko 2024-01-15T17:50:33.874 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 192353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-01-15T17:50:33.874 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/samsung/ 2024-01-15T17:50:33.874 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/samsung/sxgbe/ 2024-01-15T17:50:33.874 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 104241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2024-01-15T17:50:33.874 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sfc/ 2024-01-15T17:50:33.874 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 1064737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-01-15T17:50:33.874 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/silan/ 2024-01-15T17:50:33.874 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 47097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-01-15T17:50:33.874 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/ 2024-01-15T17:50:33.875 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 54817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/sis190.ko 2024-01-15T17:50:33.875 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 63337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/sis900.ko 2024-01-15T17:50:33.875 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/ 2024-01-15T17:50:33.875 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 61537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-01-15T17:50:33.875 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 67193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-01-15T17:50:33.875 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 73409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-01-15T17:50:33.875 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 68177 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-01-15T17:50:33.875 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/ 2024-01-15T17:50:33.875 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/ 2024-01-15T17:50:33.875 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2024-01-15T17:50:33.875 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 39905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-01-15T17:50:33.875 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2024-01-15T17:50:33.875 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 487593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-01-15T17:50:33.876 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/ 2024-01-15T17:50:33.876 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 103337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/cassini.ko 2024-01-15T17:50:33.876 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 166097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/niu.ko 2024-01-15T17:50:33.876 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 66329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/sungem.ko 2024-01-15T17:50:33.876 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 78809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-01-15T17:50:33.876 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/tehuti/ 2024-01-15T17:50:33.876 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 50057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-01-15T17:50:33.876 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ti/ 2024-01-15T17:50:33.876 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 86529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ti/tlan.ko 2024-01-15T17:50:33.876 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/ 2024-01-15T17:50:33.876 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 80273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-01-15T17:50:33.876 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 83793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-01-15T17:50:33.876 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/ 2024-01-15T17:50:33.877 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 46217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-01-15T17:50:33.877 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-01-15T17:50:33.877 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/xircom/ 2024-01-15T17:50:33.877 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 59993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-01-15T17:50:33.877 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/fddi/ 2024-01-15T17:50:33.877 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 41001 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/fddi/defxx.ko 2024-01-15T17:50:33.877 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/fddi/skfp/ 2024-01-15T17:50:33.877 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 216985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/fddi/skfp/skfp.ko 2024-01-15T17:50:33.877 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 74569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/geneve.ko 2024-01-15T17:50:33.877 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/hyperv/ 2024-01-15T17:50:33.877 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 236225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-01-15T17:50:33.877 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ieee802154/ 2024-01-15T17:50:33.877 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ieee802154/fakelb.ko 2024-01-15T17:50:33.877 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 25593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ifb.ko 2024-01-15T17:50:33.878 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ipvlan/ 2024-01-15T17:50:33.878 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 85817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ipvlan/ipvlan.ko 2024-01-15T17:50:33.878 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 62769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/macvlan.ko 2024-01-15T17:50:33.878 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/macvtap.ko 2024-01-15T17:50:33.878 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/ 2024-01-15T17:50:33.878 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-01-15T17:50:33.878 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14233 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-01-15T17:50:33.878 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-gpio.ko 2024-01-15T17:50:33.878 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-regmap.ko 2024-01-15T17:50:33.878 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio.ko 2024-01-15T17:50:33.878 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20513 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mii.ko 2024-01-15T17:50:33.878 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 45089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/netconsole.ko 2024-01-15T17:50:33.878 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/nlmon.ko 2024-01-15T17:50:33.878 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/ 2024-01-15T17:50:33.878 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/pcs-lynx.ko 2024-01-15T17:50:33.879 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 39217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-01-15T17:50:33.879 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/ 2024-01-15T17:50:33.879 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/amd.ko 2024-01-15T17:50:33.879 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 50033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-01-15T17:50:33.879 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 44721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm7xxx.ko 2024-01-15T17:50:33.879 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11641 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm87xx.ko 2024-01-15T17:50:33.879 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 44857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/broadcom.ko 2024-01-15T17:50:33.879 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/cicada.ko 2024-01-15T17:50:33.879 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/davicom.ko 2024-01-15T17:50:33.879 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/et1011c.ko 2024-01-15T17:50:33.879 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/icplus.ko 2024-01-15T17:50:33.879 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/lxt.ko 2024-01-15T17:50:33.879 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 82049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/marvell.ko 2024-01-15T17:50:33.879 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 76409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/micrel.ko 2024-01-15T17:50:33.880 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/national.ko 2024-01-15T17:50:33.880 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 95561 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/phylink.ko 2024-01-15T17:50:33.880 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/qsemi.ko 2024-01-15T17:50:33.880 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 50065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/realtek.ko 2024-01-15T17:50:33.880 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/smsc.ko 2024-01-15T17:50:33.880 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/ste10Xp.ko 2024-01-15T17:50:33.880 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/vitesse.ko 2024-01-15T17:50:33.880 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/plip/ 2024-01-15T17:50:33.880 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 43505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/plip/plip.ko 2024-01-15T17:50:33.880 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ 2024-01-15T17:50:33.880 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/bsd_comp.ko 2024-01-15T17:50:33.880 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 29713 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_async.ko 2024-01-15T17:50:33.880 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_deflate.ko 2024-01-15T17:50:33.880 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_mppe.ko 2024-01-15T17:50:33.880 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_synctty.ko 2024-01-15T17:50:33.881 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 42849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pppoe.ko 2024-01-15T17:50:33.881 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pppox.ko 2024-01-15T17:50:33.881 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 32737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pptp.ko 2024-01-15T17:50:33.881 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/sb1000.ko 2024-01-15T17:50:33.881 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip/ 2024-01-15T17:50:33.881 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 42465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip/slip.ko 2024-01-15T17:50:33.881 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 27049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/sungem_phy.ko 2024-01-15T17:50:33.881 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 64585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/tap.ko 2024-01-15T17:50:33.881 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/ 2024-01-15T17:50:33.881 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 107089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team.ko 2024-01-15T17:50:33.881 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_activebackup.ko 2024-01-15T17:50:33.881 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_broadcast.ko 2024-01-15T17:50:33.881 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-01-15T17:50:33.882 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9641 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_random.ko 2024-01-15T17:50:33.882 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-01-15T17:50:33.882 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 86945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/veth.ko 2024-01-15T17:50:33.882 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vmxnet3/ 2024-01-15T17:50:33.882 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 144361 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-01-15T17:50:33.882 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vxlan/ 2024-01-15T17:50:33.882 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 301281 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vxlan/vxlan.ko 2024-01-15T17:50:33.882 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netback/ 2024-01-15T17:50:33.882 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 144001 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netback/xen-netback.ko 2024-01-15T17:50:33.882 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ntb/ 2024-01-15T17:50:33.882 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ntb/ntb.ko 2024-01-15T17:50:33.882 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/ 2024-01-15T17:50:33.882 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/ 2024-01-15T17:50:33.882 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 265777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/nvme-core.ko 2024-01-15T17:50:33.882 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 102545 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/nvme.ko 2024-01-15T17:50:33.883 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/ 2024-01-15T17:50:33.883 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 131249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport.ko 2024-01-15T17:50:33.883 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14561 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_cs.ko 2024-01-15T17:50:33.883 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 85377 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_pc.ko 2024-01-15T17:50:33.883 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_serial.ko 2024-01-15T17:50:33.883 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/ 2024-01-15T17:50:33.883 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug/ 2024-01-15T17:50:33.883 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 25009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-01-15T17:50:33.883 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2024-01-15T17:50:33.883 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2024-01-15T17:50:33.883 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/pci-stub.ko 2024-01-15T17:50:33.883 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 46465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/xen-pcifront.ko 2024-01-15T17:50:33.883 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/ 2024-01-15T17:50:33.884 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/i82092.ko 2024-01-15T17:50:33.884 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 146289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/pcmcia.ko 2024-01-15T17:50:33.884 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 62161 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/pcmcia_core.ko 2024-01-15T17:50:33.884 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 34345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2024-01-15T17:50:33.884 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 34265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/pd6729.ko 2024-01-15T17:50:33.884 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 112889 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/yenta_socket.ko 2024-01-15T17:50:33.884 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/phy/ 2024-01-15T17:50:33.884 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/phy/broadcom/ 2024-01-15T17:50:33.884 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2024-01-15T17:50:33.884 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/phy/samsung/ 2024-01-15T17:50:33.884 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2024-01-15T17:50:33.884 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/ 2024-01-15T17:50:33.884 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/intel/ 2024-01-15T17:50:33.885 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 69137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-01-15T17:50:33.885 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/ 2024-01-15T17:50:33.885 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/chrome/ 2024-01-15T17:50:33.885 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 50889 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2024-01-15T17:50:33.885 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/chrome/cros_ec.ko 2024-01-15T17:50:33.885 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2024-01-15T17:50:33.885 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 25465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2024-01-15T17:50:33.885 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2024-01-15T17:50:33.885 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2024-01-15T17:50:33.885 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17361 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2024-01-15T17:50:33.885 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/ 2024-01-15T17:50:33.885 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 67097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/acer-wmi.ko 2024-01-15T17:50:33.885 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 39145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/acerhdf.ko 2024-01-15T17:50:33.886 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 37025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/apple-gmux.ko 2024-01-15T17:50:33.886 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 59665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-laptop.ko 2024-01-15T17:50:33.886 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 31481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-01-15T17:50:33.886 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 132025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-wmi.ko 2024-01-15T17:50:33.886 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 36849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/classmate-laptop.ko 2024-01-15T17:50:33.886 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 49121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-01-15T17:50:33.886 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-01-15T17:50:33.886 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 40265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-01-15T17:50:33.886 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-01-15T17:50:33.886 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 36593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/hdaps.ko 2024-01-15T17:50:33.886 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/ibm_rtl.ko 2024-01-15T17:50:33.886 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel/ 2024-01-15T17:50:33.886 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-01-15T17:50:33.886 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2024-01-15T17:50:33.887 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 43497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel_ips.ko 2024-01-15T17:50:33.887 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/msi-wmi.ko 2024-01-15T17:50:33.887 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10377 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/mxm-wmi.ko 2024-01-15T17:50:33.887 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 34801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-01-15T17:50:33.887 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 42329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/samsung-laptop.ko 2024-01-15T17:50:33.887 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/samsung-q10.ko 2024-01-15T17:50:33.887 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/topstar-laptop.ko 2024-01-15T17:50:33.887 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-01-15T17:50:33.887 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/toshiba_haps.ko 2024-01-15T17:50:33.887 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/wmi-bmof.ko 2024-01-15T17:50:33.887 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 56905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/wmi.ko 2024-01-15T17:50:33.887 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/ 2024-01-15T17:50:33.887 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/ 2024-01-15T17:50:33.887 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30233 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/88pm860x_battery.ko 2024-01-15T17:50:33.888 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 33985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/88pm860x_charger.ko 2024-01-15T17:50:33.888 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 55057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/bq2415x_charger.ko 2024-01-15T17:50:33.888 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 54081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/bq24190_charger.ko 2024-01-15T17:50:33.888 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21553 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/bq24735-charger.ko 2024-01-15T17:50:33.888 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/cros_peripheral_charger.ko 2024-01-15T17:50:33.888 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19609 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/da9030_battery.ko 2024-01-15T17:50:33.888 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/da9052-battery.ko 2024-01-15T17:50:33.888 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 29281 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/ds2760_battery.ko 2024-01-15T17:50:33.888 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/ds2780_battery.ko 2024-01-15T17:50:33.888 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/ds2781_battery.ko 2024-01-15T17:50:33.888 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/ds2782_battery.ko 2024-01-15T17:50:33.888 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/gpio-charger.ko 2024-01-15T17:50:33.888 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/isp1704_charger.ko 2024-01-15T17:50:33.889 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/lp8727_charger.ko 2024-01-15T17:50:33.889 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/max14577_charger.ko 2024-01-15T17:50:33.889 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/max17042_battery.ko 2024-01-15T17:50:33.889 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/max8903_charger.ko 2024-01-15T17:50:33.889 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/max8925_power.ko 2024-01-15T17:50:33.889 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/pcf50633-charger.ko 2024-01-15T17:50:33.889 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 37937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/sbs-battery.ko 2024-01-15T17:50:33.889 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/test_power.ko 2024-01-15T17:50:33.889 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/tps65090-charger.ko 2024-01-15T17:50:33.889 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/wm831x_backup.ko 2024-01-15T17:50:33.889 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24457 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/wm831x_power.ko 2024-01-15T17:50:33.889 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22417 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/power/supply/wm8350_power.ko 2024-01-15T17:50:33.889 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/ 2024-01-15T17:50:33.889 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 58001 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/intel_rapl_common.ko 2024-01-15T17:50:33.890 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/intel_rapl_msr.ko 2024-01-15T17:50:33.890 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/ 2024-01-15T17:50:33.890 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/ 2024-01-15T17:50:33.890 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps-gpio.ko 2024-01-15T17:50:33.890 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps-ldisc.ko 2024-01-15T17:50:33.890 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps_parport.ko 2024-01-15T17:50:33.890 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 41873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/pps_core.ko 2024-01-15T17:50:33.890 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ 2024-01-15T17:50:33.890 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 70937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ptp.ko 2024-01-15T17:50:33.890 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ptp_kvm.ko 2024-01-15T17:50:33.890 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/ 2024-01-15T17:50:33.890 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-88pm80x.ko 2024-01-15T17:50:33.890 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-88pm860x.ko 2024-01-15T17:50:33.891 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-bq32k.ko 2024-01-15T17:50:33.891 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-da9052.ko 2024-01-15T17:50:33.891 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-da9055.ko 2024-01-15T17:50:33.891 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-da9063.ko 2024-01-15T17:50:33.891 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16457 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1286.ko 2024-01-15T17:50:33.891 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 68577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1307.ko 2024-01-15T17:50:33.891 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1374.ko 2024-01-15T17:50:33.891 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1511.ko 2024-01-15T17:50:33.891 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1553.ko 2024-01-15T17:50:33.891 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1672.ko 2024-01-15T17:50:33.891 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13369 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1742.ko 2024-01-15T17:50:33.891 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14697 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds2404.ko 2024-01-15T17:50:33.891 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23705 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds3232.ko 2024-01-15T17:50:33.891 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-em3027.ko 2024-01-15T17:50:33.892 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-fm3130.ko 2024-01-15T17:50:33.892 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-isl12022.ko 2024-01-15T17:50:33.892 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 35969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-isl1208.ko 2024-01-15T17:50:33.892 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-lp8788.ko 2024-01-15T17:50:33.892 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 38097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m41t80.ko 2024-01-15T17:50:33.892 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t35.ko 2024-01-15T17:50:33.892 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 25945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t59.ko 2024-01-15T17:50:33.892 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t86.ko 2024-01-15T17:50:33.892 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-max6900.ko 2024-01-15T17:50:33.892 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-max8907.ko 2024-01-15T17:50:33.892 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-max8925.ko 2024-01-15T17:50:33.892 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 25809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-max8997.ko 2024-01-15T17:50:33.892 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-max8998.ko 2024-01-15T17:50:33.892 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18513 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-mc13xxx.ko 2024-01-15T17:50:33.893 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-msm6242.ko 2024-01-15T17:50:33.893 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-palmas.ko 2024-01-15T17:50:33.893 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 46633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf2127.ko 2024-01-15T17:50:33.893 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17977 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf50633.ko 2024-01-15T17:50:33.893 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf85063.ko 2024-01-15T17:50:33.893 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8523.ko 2024-01-15T17:50:33.893 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 29817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8563.ko 2024-01-15T17:50:33.893 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8583.ko 2024-01-15T17:50:33.893 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rc5t583.ko 2024-01-15T17:50:33.893 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rp5c01.ko 2024-01-15T17:50:33.893 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 33753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rs5c372.ko 2024-01-15T17:50:33.893 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 27833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-01-15T17:50:33.893 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 25257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rx8025.ko 2024-01-15T17:50:33.893 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rx8581.ko 2024-01-15T17:50:33.894 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-s35390a.ko 2024-01-15T17:50:33.894 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-01-15T17:50:33.894 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17889 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-tps6586x.ko 2024-01-15T17:50:33.894 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-tps65910.ko 2024-01-15T17:50:33.894 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20545 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-wm831x.ko 2024-01-15T17:50:33.894 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-wm8350.ko 2024-01-15T17:50:33.894 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 27873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-x1205.ko 2024-01-15T17:50:33.894 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ 2024-01-15T17:50:33.894 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 77617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-9xxx.ko 2024-01-15T17:50:33.894 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 57665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-sas.ko 2024-01-15T17:50:33.894 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 55105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-xxxx.ko 2024-01-15T17:50:33.894 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 135185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/BusLogic.ko 2024-01-15T17:50:33.894 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/a100u2w.ko 2024-01-15T17:50:33.894 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aacraid/ 2024-01-15T17:50:33.895 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 271113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aacraid/aacraid.ko 2024-01-15T17:50:33.895 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 100617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/advansys.ko 2024-01-15T17:50:33.895 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/ 2024-01-15T17:50:33.895 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 356929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-01-15T17:50:33.895 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 296049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-01-15T17:50:33.895 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic94xx/ 2024-01-15T17:50:33.895 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 189185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic94xx/aic94xx.ko 2024-01-15T17:50:33.895 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/am53c974.ko 2024-01-15T17:50:33.895 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/arcmsr/ 2024-01-15T17:50:33.895 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 106881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-01-15T17:50:33.895 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 38185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/atp870u.ko 2024-01-15T17:50:33.895 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/be2iscsi/ 2024-01-15T17:50:33.895 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 247009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-01-15T17:50:33.895 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bfa/ 2024-01-15T17:50:33.896 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 1171945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bfa/bfa.ko 2024-01-15T17:50:33.896 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2fc/ 2024-01-15T17:50:33.896 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 237721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-01-15T17:50:33.896 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2i/ 2024-01-15T17:50:33.896 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 115369 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-01-15T17:50:33.896 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 40585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ch.ko 2024-01-15T17:50:33.896 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/csiostor/ 2024-01-15T17:50:33.896 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 308921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/csiostor/csiostor.ko 2024-01-15T17:50:33.896 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/ 2024-01-15T17:50:33.896 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb3i/ 2024-01-15T17:50:33.896 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 76145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-01-15T17:50:33.896 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb4i/ 2024-01-15T17:50:33.896 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 101449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-01-15T17:50:33.896 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 146625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-01-15T17:50:33.897 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 75553 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/dc395x.ko 2024-01-15T17:50:33.897 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 38073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/dmx3191d.ko 2024-01-15T17:50:33.897 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esas2r/ 2024-01-15T17:50:33.897 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 212425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esas2r/esas2r.ko 2024-01-15T17:50:33.897 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 60593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esp_scsi.ko 2024-01-15T17:50:33.897 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/ 2024-01-15T17:50:33.897 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 78209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/fcoe.ko 2024-01-15T17:50:33.897 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 158505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-01-15T17:50:33.897 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fdomain.ko 2024-01-15T17:50:33.897 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fnic/ 2024-01-15T17:50:33.897 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 263489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fnic/fnic.ko 2024-01-15T17:50:33.897 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 201817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hpsa.ko 2024-01-15T17:50:33.897 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 51209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hptiop.ko 2024-01-15T17:50:33.897 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 49041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hv_storvsc.ko 2024-01-15T17:50:33.898 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 36217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/imm.ko 2024-01-15T17:50:33.898 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 44057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/initio.ko 2024-01-15T17:50:33.898 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 266537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ipr.ko 2024-01-15T17:50:33.898 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 101473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ips.ko 2024-01-15T17:50:33.898 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/isci/ 2024-01-15T17:50:33.898 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 317153 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/isci/isci.ko 2024-01-15T17:50:33.898 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-01-15T17:50:33.898 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 52961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/iscsi_tcp.ko 2024-01-15T17:50:33.898 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libfc/ 2024-01-15T17:50:33.898 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 315209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libfc/libfc.ko 2024-01-15T17:50:33.898 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 145585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libiscsi.ko 2024-01-15T17:50:33.898 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 53905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libiscsi_tcp.ko 2024-01-15T17:50:33.898 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libsas/ 2024-01-15T17:50:33.898 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 216441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libsas/libsas.ko 2024-01-15T17:50:33.898 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/lpfc/ 2024-01-15T17:50:33.899 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 2197481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/lpfc/lpfc.ko 2024-01-15T17:50:33.899 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/ 2024-01-15T17:50:33.899 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 73569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-01-15T17:50:33.899 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 33529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-01-15T17:50:33.899 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 310841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-01-15T17:50:33.899 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 84649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid.ko 2024-01-15T17:50:33.899 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mpt3sas/ 2024-01-15T17:50:33.899 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 659801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-01-15T17:50:33.899 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvsas/ 2024-01-15T17:50:33.899 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 119561 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvsas/mvsas.ko 2024-01-15T17:50:33.899 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 78129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvumi.ko 2024-01-15T17:50:33.899 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pcmcia/ 2024-01-15T17:50:33.899 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 81649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2024-01-15T17:50:33.899 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11713 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2024-01-15T17:50:33.900 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2024-01-15T17:50:33.900 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2024-01-15T17:50:33.900 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pm8001/ 2024-01-15T17:50:33.900 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 366401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-01-15T17:50:33.900 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 124633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pmcraid.ko 2024-01-15T17:50:33.900 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 34513 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ppa.ko 2024-01-15T17:50:33.900 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 55425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla1280.ko 2024-01-15T17:50:33.900 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/ 2024-01-15T17:50:33.900 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 1876633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-01-15T17:50:33.900 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 85289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-01-15T17:50:33.900 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla4xxx/ 2024-01-15T17:50:33.900 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 559033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-01-15T17:50:33.900 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qlogicfas408.ko 2024-01-15T17:50:33.900 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/raid_class.ko 2024-01-15T17:50:33.901 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 261241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_debug.ko 2024-01-15T17:50:33.901 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 173937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_fc.ko 2024-01-15T17:50:33.901 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 279313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-01-15T17:50:33.901 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 97729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_sas.ko 2024-01-15T17:50:33.901 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 73969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_spi.ko 2024-01-15T17:50:33.901 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 44993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_srp.ko 2024-01-15T17:50:33.901 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ses.ko 2024-01-15T17:50:33.901 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 115889 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/st.ko 2024-01-15T17:50:33.901 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 46825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/stex.ko 2024-01-15T17:50:33.901 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/sym53c8xx_2/ 2024-01-15T17:50:33.901 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 164961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-01-15T17:50:33.901 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 42297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/virtio_scsi.ko 2024-01-15T17:50:33.901 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 50321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/vmw_pvscsi.ko 2024-01-15T17:50:33.901 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 39961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/wd719x.ko 2024-01-15T17:50:33.902 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 50025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/xen-scsifront.ko 2024-01-15T17:50:33.902 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ssb/ 2024-01-15T17:50:33.902 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 129441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ssb/ssb.ko 2024-01-15T17:50:33.902 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/ 2024-01-15T17:50:33.902 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/iscsi/ 2024-01-15T17:50:33.902 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 736769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-01-15T17:50:33.902 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/loopback/ 2024-01-15T17:50:33.902 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 49969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/loopback/tcm_loop.ko 2024-01-15T17:50:33.902 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 39713 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_file.ko 2024-01-15T17:50:33.902 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 43025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_iblock.ko 2024-01-15T17:50:33.902 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 930697 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_mod.ko 2024-01-15T17:50:33.902 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 44241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_pscsi.ko 2024-01-15T17:50:33.902 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 122521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_user.ko 2024-01-15T17:50:33.902 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/tcm_fc/ 2024-01-15T17:50:33.903 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 75593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-01-15T17:50:33.903 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/ 2024-01-15T17:50:33.903 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/ 2024-01-15T17:50:33.903 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/ 2024-01-15T17:50:33.903 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-01-15T17:50:33.903 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 29321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-01-15T17:50:33.903 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-01-15T17:50:33.903 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-01-15T17:50:33.903 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-01-15T17:50:33.903 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-01-15T17:50:33.903 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-01-15T17:50:33.903 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-01-15T17:50:33.903 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-01-15T17:50:33.903 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-01-15T17:50:33.904 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-01-15T17:50:33.904 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13001 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-01-15T17:50:33.904 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 37065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-01-15T17:50:33.904 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-01-15T17:50:33.904 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-01-15T17:50:33.904 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_powerclamp.ko 2024-01-15T17:50:33.904 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-01-15T17:50:33.904 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-01-15T17:50:33.904 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-01-15T17:50:33.904 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ 2024-01-15T17:50:33.904 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ipwireless/ 2024-01-15T17:50:33.904 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 78425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-01-15T17:50:33.904 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 57865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/moxa.ko 2024-01-15T17:50:33.904 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 50953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/mxser.ko 2024-01-15T17:50:33.905 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 108601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/n_gsm.ko 2024-01-15T17:50:33.905 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 31961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/n_hdlc.ko 2024-01-15T17:50:33.905 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 67929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/nozomi.ko 2024-01-15T17:50:33.905 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/ 2024-01-15T17:50:33.905 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250/ 2024-01-15T17:50:33.905 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 34769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250/8250_dw.ko 2024-01-15T17:50:33.905 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 59497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-01-15T17:50:33.905 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/altera_jtaguart.ko 2024-01-15T17:50:33.905 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/altera_uart.ko 2024-01-15T17:50:33.905 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/arc_uart.ko 2024-01-15T17:50:33.905 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 72425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/fsl_lpuart.ko 2024-01-15T17:50:33.905 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/jsm/ 2024-01-15T17:50:33.905 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 103073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/jsm/jsm.ko 2024-01-15T17:50:33.905 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/rp2.ko 2024-01-15T17:50:33.906 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 53169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/sc16is7xx.ko 2024-01-15T17:50:33.906 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/ 2024-01-15T17:50:33.906 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/core/ 2024-01-15T17:50:33.906 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 535985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/core/ufshcd-core.ko 2024-01-15T17:50:33.906 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host/ 2024-01-15T17:50:33.906 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-01-15T17:50:33.906 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2024-01-15T17:50:33.906 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/ 2024-01-15T17:50:33.906 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 44513 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio.ko 2024-01-15T17:50:33.906 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_aec.ko 2024-01-15T17:50:33.906 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11281 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_cif.ko 2024-01-15T17:50:33.906 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_dmem_genirq.ko 2024-01-15T17:50:33.906 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_mf624.ko 2024-01-15T17:50:33.906 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12177 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_netx.ko 2024-01-15T17:50:33.906 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_pci_generic.ko 2024-01-15T17:50:33.907 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_pdrv_genirq.ko 2024-01-15T17:50:33.907 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_sercos3.ko 2024-01-15T17:50:33.907 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/ 2024-01-15T17:50:33.907 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/c67x00/ 2024-01-15T17:50:33.907 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 73297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/c67x00/c67x00.ko 2024-01-15T17:50:33.907 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/chipidea/ 2024-01-15T17:50:33.907 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 144617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/chipidea/ci_hdrc.ko 2024-01-15T17:50:33.907 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2024-01-15T17:50:33.907 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13177 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/chipidea/ci_hdrc_npcm.ko 2024-01-15T17:50:33.907 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12713 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2024-01-15T17:50:33.907 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2024-01-15T17:50:33.907 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/ 2024-01-15T17:50:33.907 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 87073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/cdc-acm.ko 2024-01-15T17:50:33.907 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 51345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/cdc-wdm.ko 2024-01-15T17:50:33.908 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 47305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/usblp.ko 2024-01-15T17:50:33.908 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 89857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/usbtmc.ko 2024-01-15T17:50:33.908 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/common/ 2024-01-15T17:50:33.908 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/common/ulpi.ko 2024-01-15T17:50:33.908 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/dwc2/ 2024-01-15T17:50:33.908 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/dwc2/dwc2_pci.ko 2024-01-15T17:50:33.908 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/dwc3/ 2024-01-15T17:50:33.908 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/dwc3/dwc3-haps.ko 2024-01-15T17:50:33.908 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/dwc3/dwc3-pci.ko 2024-01-15T17:50:33.908 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 454385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/dwc3/dwc3.ko 2024-01-15T17:50:33.908 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/ 2024-01-15T17:50:33.908 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/ 2024-01-15T17:50:33.908 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 60217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/u_ether.ko 2024-01-15T17:50:33.909 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 51881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/u_serial.ko 2024-01-15T17:50:33.909 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2024-01-15T17:50:33.909 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 42449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2024-01-15T17:50:33.909 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 27953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2024-01-15T17:50:33.909 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 36513 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2024-01-15T17:50:33.909 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 92489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2024-01-15T17:50:33.909 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 51041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2024-01-15T17:50:33.909 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 122081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2024-01-15T17:50:33.909 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 57937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2024-01-15T17:50:33.909 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2024-01-15T17:50:33.909 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 31265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2024-01-15T17:50:33.909 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 57729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2024-01-15T17:50:33.909 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 85609 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2024-01-15T17:50:33.910 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2024-01-15T17:50:33.910 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 58113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2024-01-15T17:50:33.910 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 67337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2024-01-15T17:50:33.910 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/ 2024-01-15T17:50:33.910 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2024-01-15T17:50:33.910 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2024-01-15T17:50:33.910 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2024-01-15T17:50:33.910 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/g_ether.ko 2024-01-15T17:50:33.910 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2024-01-15T17:50:33.910 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/g_hid.ko 2024-01-15T17:50:33.910 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2024-01-15T17:50:33.910 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16977 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2024-01-15T17:50:33.910 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 32073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2024-01-15T17:50:33.910 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/g_printer.ko 2024-01-15T17:50:33.911 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/g_serial.ko 2024-01-15T17:50:33.911 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 27201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/g_zero.ko 2024-01-15T17:50:33.911 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 62361 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2024-01-15T17:50:33.911 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2024-01-15T17:50:33.911 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 161225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/libcomposite.ko 2024-01-15T17:50:33.911 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/ 2024-01-15T17:50:33.911 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17705 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2024-01-15T17:50:33.911 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/bdc/ 2024-01-15T17:50:33.911 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 157681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2024-01-15T17:50:33.911 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 36089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/goku_udc.ko 2024-01-15T17:50:33.911 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 52721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/gr_udc.ko 2024-01-15T17:50:33.911 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 69705 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2024-01-15T17:50:33.911 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 56033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/mv_udc.ko 2024-01-15T17:50:33.911 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 76057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/net2272.ko 2024-01-15T17:50:33.912 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 75073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/net2280.ko 2024-01-15T17:50:33.912 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 73169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/pch_udc.ko 2024-01-15T17:50:33.912 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 65809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2024-01-15T17:50:33.912 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 66097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2024-01-15T17:50:33.912 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 81505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2024-01-15T17:50:33.912 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 156409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/gadget/udc/udc-core.ko 2024-01-15T17:50:33.912 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/ 2024-01-15T17:50:33.912 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/bcma-hcd.ko 2024-01-15T17:50:33.912 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 80113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/isp116x-hcd.ko 2024-01-15T17:50:33.912 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 87353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/oxu210hp-hcd.ko 2024-01-15T17:50:33.912 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 78873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/r8a66597-hcd.ko 2024-01-15T17:50:33.912 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 51329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/sl811-hcd.ko 2024-01-15T17:50:33.912 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/sl811_cs.ko 2024-01-15T17:50:33.912 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/ssb-hcd.ko 2024-01-15T17:50:33.913 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/xhci-plat-hcd.ko 2024-01-15T17:50:33.913 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/ 2024-01-15T17:50:33.913 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 34585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/mdc800.ko 2024-01-15T17:50:33.913 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/microtek.ko 2024-01-15T17:50:33.913 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ 2024-01-15T17:50:33.913 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 41721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/adutux.ko 2024-01-15T17:50:33.913 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20713 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/appledisplay.ko 2024-01-15T17:50:33.913 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/cypress_cy7c63.ko 2024-01-15T17:50:33.913 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/cytherm.ko 2024-01-15T17:50:33.913 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ehset.ko 2024-01-15T17:50:33.913 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/emi26.ko 2024-01-15T17:50:33.913 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/emi62.ko 2024-01-15T17:50:33.913 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ezusb.ko 2024-01-15T17:50:33.913 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/idmouse.ko 2024-01-15T17:50:33.913 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 36553 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/iowarrior.ko 2024-01-15T17:50:33.914 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/isight_firmware.ko 2024-01-15T17:50:33.914 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 36529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ldusb.ko 2024-01-15T17:50:33.914 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 36265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/legousbtower.ko 2024-01-15T17:50:33.914 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 27793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/lvstest.ko 2024-01-15T17:50:33.914 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/sisusbvga/ 2024-01-15T17:50:33.914 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 64841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-01-15T17:50:33.914 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/trancevibrator.ko 2024-01-15T17:50:33.914 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usb3503.ko 2024-01-15T17:50:33.914 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 25145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usblcd.ko 2024-01-15T17:50:33.914 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usbsevseg.ko 2024-01-15T17:50:33.914 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 70809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usbtest.ko 2024-01-15T17:50:33.914 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 41905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/uss720.ko 2024-01-15T17:50:33.914 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 27617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/yurex.ko 2024-01-15T17:50:33.914 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/mon/ 2024-01-15T17:50:33.915 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 72953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/mon/usbmon.ko 2024-01-15T17:50:33.915 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/musb/ 2024-01-15T17:50:33.915 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 301697 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/musb/musb_hdrc.ko 2024-01-15T17:50:33.915 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy/ 2024-01-15T17:50:33.915 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy/phy-generic.ko 2024-01-15T17:50:33.915 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20177 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2024-01-15T17:50:33.915 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy/phy-isp1301.ko 2024-01-15T17:50:33.915 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy/phy-tahvo.ko 2024-01-15T17:50:33.915 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ 2024-01-15T17:50:33.915 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/aircable.ko 2024-01-15T17:50:33.915 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 33329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ark3116.ko 2024-01-15T17:50:33.915 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/belkin_sa.ko 2024-01-15T17:50:33.915 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 38105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ch341.ko 2024-01-15T17:50:33.915 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 70257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cp210x.ko 2024-01-15T17:50:33.916 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 31057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cyberjack.ko 2024-01-15T17:50:33.916 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 58217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cypress_m8.ko 2024-01-15T17:50:33.916 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 49817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/digi_acceleport.ko 2024-01-15T17:50:33.916 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/empeg.ko 2024-01-15T17:50:33.916 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 44273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/f81232.ko 2024-01-15T17:50:33.916 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 175193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ftdi_sio.ko 2024-01-15T17:50:33.916 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 53113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/garmin_gps.ko 2024-01-15T17:50:33.916 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 137081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/io_edgeport.ko 2024-01-15T17:50:33.916 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 124833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/io_ti.ko 2024-01-15T17:50:33.916 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 67769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ipaq.ko 2024-01-15T17:50:33.916 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19161 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ipw.ko 2024-01-15T17:50:33.916 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26897 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ir-usb.ko 2024-01-15T17:50:33.916 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 58233 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-01-15T17:50:33.916 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 107521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/keyspan.ko 2024-01-15T17:50:33.917 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 35617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/keyspan_pda.ko 2024-01-15T17:50:33.917 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 25665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/kl5kusb105.ko 2024-01-15T17:50:33.917 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 33689 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/kobil_sct.ko 2024-01-15T17:50:33.917 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 32953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mct_u232.ko 2024-01-15T17:50:33.917 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/metro-usb.ko 2024-01-15T17:50:33.917 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 66449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mos7720.ko 2024-01-15T17:50:33.917 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 78217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mos7840.ko 2024-01-15T17:50:33.917 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 56337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mxuport.ko 2024-01-15T17:50:33.917 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/navman.ko 2024-01-15T17:50:33.917 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/omninet.ko 2024-01-15T17:50:33.917 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/opticon.ko 2024-01-15T17:50:33.917 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 191801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/option.ko 2024-01-15T17:50:33.917 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 39073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/oti6858.ko 2024-01-15T17:50:33.917 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 54353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/pl2303.ko 2024-01-15T17:50:33.917 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/qcaux.ko 2024-01-15T17:50:33.918 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 37825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/qcserial.ko 2024-01-15T17:50:33.918 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 34321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/quatech2.ko 2024-01-15T17:50:33.918 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/safe_serial.ko 2024-01-15T17:50:33.918 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 51625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/sierra.ko 2024-01-15T17:50:33.918 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/spcp8x5.ko 2024-01-15T17:50:33.918 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 29993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ssu100.ko 2024-01-15T17:50:33.918 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/symbolserial.ko 2024-01-15T17:50:33.918 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 61489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-01-15T17:50:33.918 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-01-15T17:50:33.918 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb_debug.ko 2024-01-15T17:50:33.918 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 33201 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb_wwan.ko 2024-01-15T17:50:33.918 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 125609 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usbserial.ko 2024-01-15T17:50:33.918 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/visor.ko 2024-01-15T17:50:33.918 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 37409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/whiteheat.ko 2024-01-15T17:50:33.918 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/wishbone-serial.ko 2024-01-15T17:50:33.919 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/xsens_mt.ko 2024-01-15T17:50:33.919 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ 2024-01-15T17:50:33.919 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 52913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/uas.ko 2024-01-15T17:50:33.919 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-alauda.ko 2024-01-15T17:50:33.919 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-cypress.ko 2024-01-15T17:50:33.919 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-datafab.ko 2024-01-15T17:50:33.919 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 33497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-01-15T17:50:33.919 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-freecom.ko 2024-01-15T17:50:33.919 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-isd200.ko 2024-01-15T17:50:33.919 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19361 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-01-15T17:50:33.919 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13713 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-karma.ko 2024-01-15T17:50:33.919 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-onetouch.ko 2024-01-15T17:50:33.919 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-realtek.ko 2024-01-15T17:50:33.919 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-sddr09.ko 2024-01-15T17:50:33.919 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-sddr55.ko 2024-01-15T17:50:33.920 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 35337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-usbat.ko 2024-01-15T17:50:33.920 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 180225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/usb-storage.ko 2024-01-15T17:50:33.920 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/ 2024-01-15T17:50:33.920 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 67481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/usbip-core.ko 2024-01-15T17:50:33.920 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 71097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/usbip-host.ko 2024-01-15T17:50:33.920 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 103977 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-01-15T17:50:33.920 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/ 2024-01-15T17:50:33.920 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/ 2024-01-15T17:50:33.920 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 154297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-01-15T17:50:33.920 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 29537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/vfio-pci.ko 2024-01-15T17:50:33.920 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 103737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/vfio.ko 2024-01-15T17:50:33.920 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 77609 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-01-15T17:50:33.920 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/ 2024-01-15T17:50:33.920 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 113785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost.ko 2024-01-15T17:50:33.920 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_iotlb.ko 2024-01-15T17:50:33.921 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 58089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_net.ko 2024-01-15T17:50:33.921 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 89233 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_scsi.ko 2024-01-15T17:50:33.921 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/ 2024-01-15T17:50:33.921 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/ 2024-01-15T17:50:33.921 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15417 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/88pm860x_bl.ko 2024-01-15T17:50:33.921 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/aat2870_bl.ko 2024-01-15T17:50:33.921 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/adp5520_bl.ko 2024-01-15T17:50:33.921 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 32609 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/adp8860_bl.ko 2024-01-15T17:50:33.921 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 39993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/adp8870_bl.ko 2024-01-15T17:50:33.921 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/apple_bl.ko 2024-01-15T17:50:33.921 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/as3711_bl.ko 2024-01-15T17:50:33.921 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/bd6107.ko 2024-01-15T17:50:33.921 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/da903x_bl.ko 2024-01-15T17:50:33.921 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/da9052_bl.ko 2024-01-15T17:50:33.922 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/gpio_backlight.ko 2024-01-15T17:50:33.922 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/kb3886_bl.ko 2024-01-15T17:50:33.922 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/lcd.ko 2024-01-15T17:50:33.922 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/lm3533_bl.ko 2024-01-15T17:50:33.922 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/lm3639_bl.ko 2024-01-15T17:50:33.922 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/lv5207lp.ko 2024-01-15T17:50:33.922 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/max8925_bl.ko 2024-01-15T17:50:33.922 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/pandora_bl.ko 2024-01-15T17:50:33.922 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/pcf50633-backlight.ko 2024-01-15T17:50:33.922 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10897 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/platform_lcd.ko 2024-01-15T17:50:33.922 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/wm831x_bl.ko 2024-01-15T17:50:33.922 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/ 2024-01-15T17:50:33.922 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ 2024-01-15T17:50:33.922 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ds2482.ko 2024-01-15T17:50:33.922 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 38985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ds2490.ko 2024-01-15T17:50:33.922 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/matrox_w1.ko 2024-01-15T17:50:33.923 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12417 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/w1-gpio.ko 2024-01-15T17:50:33.923 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/ 2024-01-15T17:50:33.923 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-01-15T17:50:33.923 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-01-15T17:50:33.923 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-01-15T17:50:33.923 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-01-15T17:50:33.923 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14153 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-01-15T17:50:33.923 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-01-15T17:50:33.923 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-01-15T17:50:33.923 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-01-15T17:50:33.923 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-01-15T17:50:33.923 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_smem.ko 2024-01-15T17:50:33.923 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 56425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_therm.ko 2024-01-15T17:50:33.923 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 102673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/wire.ko 2024-01-15T17:50:33.923 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ 2024-01-15T17:50:33.924 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/acquirewdt.ko 2024-01-15T17:50:33.924 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19977 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/advantechwdt.ko 2024-01-15T17:50:33.924 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18561 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/alim1535_wdt.ko 2024-01-15T17:50:33.924 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/alim7101_wdt.ko 2024-01-15T17:50:33.924 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/cpu5wdt.ko 2024-01-15T17:50:33.924 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/da9052_wdt.ko 2024-01-15T17:50:33.924 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/da9055_wdt.ko 2024-01-15T17:50:33.924 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/da9063_wdt.ko 2024-01-15T17:50:33.924 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/dw_wdt.ko 2024-01-15T17:50:33.924 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/eurotechwdt.ko 2024-01-15T17:50:33.924 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/f71808e_wdt.ko 2024-01-15T17:50:33.924 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 27401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/hpwdt.ko 2024-01-15T17:50:33.924 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/i6300esb.ko 2024-01-15T17:50:33.924 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-01-15T17:50:33.924 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/iTCO_wdt.ko 2024-01-15T17:50:33.925 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ib700wdt.ko 2024-01-15T17:50:33.925 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18689 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ibmasr.ko 2024-01-15T17:50:33.925 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19713 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-01-15T17:50:33.925 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/it8712f_wdt.ko 2024-01-15T17:50:33.925 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/it87_wdt.ko 2024-01-15T17:50:33.925 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/kempld_wdt.ko 2024-01-15T17:50:33.925 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/machzwd.ko 2024-01-15T17:50:33.925 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/mena21_wdt.ko 2024-01-15T17:50:33.925 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/menf21bmc_wdt.ko 2024-01-15T17:50:33.925 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/nv_tco.ko 2024-01-15T17:50:33.925 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/of_xilinx_wdt.ko 2024-01-15T17:50:33.925 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pc87413_wdt.ko 2024-01-15T17:50:33.925 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 38657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pcwd_pci.ko 2024-01-15T17:50:33.925 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 31825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pcwd_usb.ko 2024-01-15T17:50:33.925 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/retu_wdt.ko 2024-01-15T17:50:33.926 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sbc60xxwdt.ko 2024-01-15T17:50:33.926 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sbc_epx_c3.ko 2024-01-15T17:50:33.926 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-01-15T17:50:33.926 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sc1200wdt.ko 2024-01-15T17:50:33.926 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sch311x_wdt.ko 2024-01-15T17:50:33.926 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/smsc37b787_wdt.ko 2024-01-15T17:50:33.926 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/softdog.ko 2024-01-15T17:50:33.926 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sp5100_tco.ko 2024-01-15T17:50:33.926 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/twl4030_wdt.ko 2024-01-15T17:50:33.926 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/via_wdt.ko 2024-01-15T17:50:33.926 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19161 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-01-15T17:50:33.926 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83877f_wdt.ko 2024-01-15T17:50:33.926 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 25409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83977f_wdt.ko 2024-01-15T17:50:33.926 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/wafer5823wdt.ko 2024-01-15T17:50:33.926 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/wdt_pci.ko 2024-01-15T17:50:33.927 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/wm831x_wdt.ko 2024-01-15T17:50:33.927 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/wm8350_wdt.ko 2024-01-15T17:50:33.927 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15697 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/xen_wdt.ko 2024-01-15T17:50:33.927 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/ 2024-01-15T17:50:33.927 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-evtchn.ko 2024-01-15T17:50:33.927 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-gntalloc.ko 2024-01-15T17:50:33.927 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 47865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-gntdev.ko 2024-01-15T17:50:33.927 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-pciback/ 2024-01-15T17:50:33.927 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 176441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-01-15T17:50:33.927 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 40185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-privcmd.ko 2024-01-15T17:50:33.927 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 71249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-scsiback.ko 2024-01-15T17:50:33.927 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xenfs/ 2024-01-15T17:50:33.927 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xenfs/xenfs.ko 2024-01-15T17:50:33.927 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ 2024-01-15T17:50:33.927 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/9p/ 2024-01-15T17:50:33.927 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 166753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/9p/9p.ko 2024-01-15T17:50:33.928 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/adfs/ 2024-01-15T17:50:33.928 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 61649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/adfs/adfs.ko 2024-01-15T17:50:33.928 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/affs/ 2024-01-15T17:50:33.928 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 163449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/affs/affs.ko 2024-01-15T17:50:33.928 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/afs/ 2024-01-15T17:50:33.928 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 935033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/afs/kafs.ko 2024-01-15T17:50:33.928 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/autofs/ 2024-01-15T17:50:33.928 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 103401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/autofs/autofs4.ko 2024-01-15T17:50:33.928 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/befs/ 2024-01-15T17:50:33.928 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 64681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/befs/befs.ko 2024-01-15T17:50:33.928 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/bfs/ 2024-01-15T17:50:33.928 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 41025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/bfs/bfs.ko 2024-01-15T17:50:33.928 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 38745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/binfmt_misc.ko 2024-01-15T17:50:33.928 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/btrfs/ 2024-01-15T17:50:33.928 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 3455857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/btrfs/btrfs.ko 2024-01-15T17:50:33.929 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/cachefiles/ 2024-01-15T17:50:33.929 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 306009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/cachefiles/cachefiles.ko 2024-01-15T17:50:33.929 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ceph/ 2024-01-15T17:50:33.929 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 1335129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ceph/ceph.ko 2024-01-15T17:50:33.929 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/coda/ 2024-01-15T17:50:33.929 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 100801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/coda/coda.ko 2024-01-15T17:50:33.929 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/configfs/ 2024-01-15T17:50:33.929 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 108641 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/configfs/configfs.ko 2024-01-15T17:50:33.929 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/cramfs/ 2024-01-15T17:50:33.929 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 29673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/cramfs/cramfs.ko 2024-01-15T17:50:33.929 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/dlm/ 2024-01-15T17:50:33.929 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 644073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/dlm/dlm.ko 2024-01-15T17:50:33.930 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/efs/ 2024-01-15T17:50:33.930 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 36785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/efs/efs.ko 2024-01-15T17:50:33.930 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/f2fs/ 2024-01-15T17:50:33.930 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 1506641 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/f2fs/f2fs.ko 2024-01-15T17:50:33.930 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/ 2024-01-15T17:50:33.930 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 27929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/msdos.ko 2024-01-15T17:50:33.930 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/freevxfs/ 2024-01-15T17:50:33.930 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 42609 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/freevxfs/freevxfs.ko 2024-01-15T17:50:33.930 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/fscache/ 2024-01-15T17:50:33.930 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 226481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/fscache/fscache.ko 2024-01-15T17:50:33.930 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse/ 2024-01-15T17:50:33.930 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse/cuse.ko 2024-01-15T17:50:33.930 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/gfs2/ 2024-01-15T17:50:33.930 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 1229913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/gfs2/gfs2.ko 2024-01-15T17:50:33.930 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/hfs/ 2024-01-15T17:50:33.931 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 141985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/hfs/hfs.ko 2024-01-15T17:50:33.931 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/hfsplus/ 2024-01-15T17:50:33.931 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 223225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/hfsplus/hfsplus.ko 2024-01-15T17:50:33.931 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/hpfs/ 2024-01-15T17:50:33.931 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 193569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/hpfs/hpfs.ko 2024-01-15T17:50:33.931 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/isofs/ 2024-01-15T17:50:33.931 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 74785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/isofs/isofs.ko 2024-01-15T17:50:33.931 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/jfs/ 2024-01-15T17:50:33.931 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 419881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/jfs/jfs.ko 2024-01-15T17:50:33.931 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/lockd/ 2024-01-15T17:50:33.931 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 282049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/lockd/lockd.ko 2024-01-15T17:50:33.931 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/minix/ 2024-01-15T17:50:33.931 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 89761 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/minix/minix.ko 2024-01-15T17:50:33.931 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/netfs/ 2024-01-15T17:50:33.932 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 124649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/netfs/netfs.ko 2024-01-15T17:50:33.932 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/ 2024-01-15T17:50:33.932 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/blocklayout/ 2024-01-15T17:50:33.932 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 81513 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-01-15T17:50:33.932 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/filelayout/ 2024-01-15T17:50:33.932 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 59449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-01-15T17:50:33.932 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/flexfilelayout/ 2024-01-15T17:50:33.932 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 117465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-01-15T17:50:33.932 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 1057969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfs.ko 2024-01-15T17:50:33.932 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 66441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv2.ko 2024-01-15T17:50:33.932 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 120265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv3.ko 2024-01-15T17:50:33.932 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 2042153 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv4.ko 2024-01-15T17:50:33.932 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/ 2024-01-15T17:50:33.932 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/grace.ko 2024-01-15T17:50:33.932 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/nfs_acl.ko 2024-01-15T17:50:33.932 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfsd/ 2024-01-15T17:50:33.933 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 1563929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nfsd/nfsd.ko 2024-01-15T17:50:33.933 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nilfs2/ 2024-01-15T17:50:33.933 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 590713 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nilfs2/nilfs2.ko 2024-01-15T17:50:33.933 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/ 2024-01-15T17:50:33.933 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-celtic.ko 2024-01-15T17:50:33.933 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-centeuro.ko 2024-01-15T17:50:33.933 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-croatian.ko 2024-01-15T17:50:33.933 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-cyrillic.ko 2024-01-15T17:50:33.933 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-gaelic.ko 2024-01-15T17:50:33.933 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-greek.ko 2024-01-15T17:50:33.933 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-iceland.ko 2024-01-15T17:50:33.933 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-inuit.ko 2024-01-15T17:50:33.933 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-roman.ko 2024-01-15T17:50:33.933 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-romanian.ko 2024-01-15T17:50:33.933 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13793 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-turkish.ko 2024-01-15T17:50:33.933 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_ascii.ko 2024-01-15T17:50:33.934 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1250.ko 2024-01-15T17:50:33.934 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12233 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1251.ko 2024-01-15T17:50:33.934 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1255.ko 2024-01-15T17:50:33.934 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp737.ko 2024-01-15T17:50:33.934 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp775.ko 2024-01-15T17:50:33.934 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp850.ko 2024-01-15T17:50:33.934 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp852.ko 2024-01-15T17:50:33.934 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp855.ko 2024-01-15T17:50:33.934 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp857.ko 2024-01-15T17:50:33.934 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp860.ko 2024-01-15T17:50:33.934 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13161 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp861.ko 2024-01-15T17:50:33.934 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp862.ko 2024-01-15T17:50:33.934 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13161 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp863.ko 2024-01-15T17:50:33.934 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp864.ko 2024-01-15T17:50:33.935 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13161 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp865.ko 2024-01-15T17:50:33.935 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp866.ko 2024-01-15T17:50:33.935 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp869.ko 2024-01-15T17:50:33.935 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp874.ko 2024-01-15T17:50:33.935 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 94137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp932.ko 2024-01-15T17:50:33.935 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 142673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp936.ko 2024-01-15T17:50:33.935 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 165745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp949.ko 2024-01-15T17:50:33.935 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 118121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp950.ko 2024-01-15T17:50:33.935 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12457 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_euc-jp.ko 2024-01-15T17:50:33.935 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-1.ko 2024-01-15T17:50:33.935 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-13.ko 2024-01-15T17:50:33.935 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-14.ko 2024-01-15T17:50:33.935 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-15.ko 2024-01-15T17:50:33.935 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-2.ko 2024-01-15T17:50:33.935 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-3.ko 2024-01-15T17:50:33.935 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-4.ko 2024-01-15T17:50:33.936 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-5.ko 2024-01-15T17:50:33.936 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-6.ko 2024-01-15T17:50:33.936 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12233 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-7.ko 2024-01-15T17:50:33.936 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-9.ko 2024-01-15T17:50:33.936 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-r.ko 2024-01-15T17:50:33.936 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-ru.ko 2024-01-15T17:50:33.936 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-u.ko 2024-01-15T17:50:33.936 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_ucs2_utils.ko 2024-01-15T17:50:33.936 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_utf8.ko 2024-01-15T17:50:33.936 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ntfs/ 2024-01-15T17:50:33.936 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 220553 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ntfs/ntfs.ko 2024-01-15T17:50:33.936 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ 2024-01-15T17:50:33.936 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/cluster/ 2024-01-15T17:50:33.936 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 244737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-01-15T17:50:33.936 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlm/ 2024-01-15T17:50:33.936 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 440737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-01-15T17:50:33.937 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlmfs/ 2024-01-15T17:50:33.937 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 54265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-01-15T17:50:33.937 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 2593521 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2.ko 2024-01-15T17:50:33.937 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-01-15T17:50:33.937 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 31729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-01-15T17:50:33.937 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 35897 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-01-15T17:50:33.937 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/omfs/ 2024-01-15T17:50:33.937 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 53769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/omfs/omfs.ko 2024-01-15T17:50:33.937 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/overlayfs/ 2024-01-15T17:50:33.937 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 370697 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/overlayfs/overlay.ko 2024-01-15T17:50:33.937 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/pstore/ 2024-01-15T17:50:33.937 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 54809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/pstore/ramoops.ko 2024-01-15T17:50:33.937 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/qnx4/ 2024-01-15T17:50:33.937 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/qnx4/qnx4.ko 2024-01-15T17:50:33.937 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/qnx6/ 2024-01-15T17:50:33.938 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 46689 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/qnx6/qnx6.ko 2024-01-15T17:50:33.938 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/quota/ 2024-01-15T17:50:33.938 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/quota/quota_tree.ko 2024-01-15T17:50:33.938 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/quota/quota_v1.ko 2024-01-15T17:50:33.938 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/quota/quota_v2.ko 2024-01-15T17:50:33.938 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/reiserfs/ 2024-01-15T17:50:33.938 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 532025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/reiserfs/reiserfs.ko 2024-01-15T17:50:33.938 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/romfs/ 2024-01-15T17:50:33.938 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/romfs/romfs.ko 2024-01-15T17:50:33.938 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/ 2024-01-15T17:50:33.938 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/client/ 2024-01-15T17:50:33.938 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 2683249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/client/cifs.ko 2024-01-15T17:50:33.938 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/ 2024-01-15T17:50:33.938 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7161 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/cifs_arc4.ko 2024-01-15T17:50:33.938 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/cifs_md4.ko 2024-01-15T17:50:33.938 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/squashfs/ 2024-01-15T17:50:33.939 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 134777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/squashfs/squashfs.ko 2024-01-15T17:50:33.939 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/sysv/ 2024-01-15T17:50:33.939 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 91249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/sysv/sysv.ko 2024-01-15T17:50:33.939 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/udf/ 2024-01-15T17:50:33.939 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 248425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/udf/udf.ko 2024-01-15T17:50:33.939 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ufs/ 2024-01-15T17:50:33.939 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 162537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/ufs/ufs.ko 2024-01-15T17:50:33.939 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/xfs/ 2024-01-15T17:50:33.939 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 4014433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/fs/xfs/xfs.ko 2024-01-15T17:50:33.982 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/kernel/ 2024-01-15T17:50:33.982 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/kernel/locking/ 2024-01-15T17:50:33.982 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 80585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/kernel/locking/locktorture.ko 2024-01-15T17:50:33.982 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/kernel/time/ 2024-01-15T17:50:33.982 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/kernel/time/test_udelay.ko 2024-01-15T17:50:33.982 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 60353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/kernel/torture.ko 2024-01-15T17:50:33.982 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/ 2024-01-15T17:50:33.983 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 6833 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/crc-itu-t.ko 2024-01-15T17:50:33.983 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 6529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/crc7.ko 2024-01-15T17:50:33.983 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/crc8.ko 2024-01-15T17:50:33.983 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/ 2024-01-15T17:50:33.983 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/libarc4.ko 2024-01-15T17:50:33.983 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 37081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/libdes.ko 2024-01-15T17:50:33.983 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/interval_tree_test.ko 2024-01-15T17:50:33.983 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/libcrc32c.ko 2024-01-15T17:50:33.983 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/lru_cache.ko 2024-01-15T17:50:33.983 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/ 2024-01-15T17:50:33.983 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 33801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/lz4_compress.ko 2024-01-15T17:50:33.983 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/lz4hc_compress.ko 2024-01-15T17:50:33.983 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/math/ 2024-01-15T17:50:33.983 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 6617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/math/cordic.ko 2024-01-15T17:50:33.983 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8233 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/memory-notifier-error-inject.ko 2024-01-15T17:50:33.983 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/notifier-error-inject.ko 2024-01-15T17:50:33.984 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/percpu_test.ko 2024-01-15T17:50:33.984 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/pm-notifier-error-inject.ko 2024-01-15T17:50:33.984 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/raid6/ 2024-01-15T17:50:33.984 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 143353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/raid6/raid6_pq.ko 2024-01-15T17:50:33.984 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/rbtree_test.ko 2024-01-15T17:50:33.984 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/reed_solomon/ 2024-01-15T17:50:33.984 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/reed_solomon/reed_solomon.ko 2024-01-15T17:50:33.984 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 38841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/test-kstrtox.ko 2024-01-15T17:50:33.984 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/test-string_helpers.ko 2024-01-15T17:50:33.984 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 4876633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/test_bpf.ko 2024-01-15T17:50:33.984 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 49089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/test_firmware.ko 2024-01-15T17:50:33.984 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 6369 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/test_module.ko 2024-01-15T17:50:33.985 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/test_user_copy.ko 2024-01-15T17:50:33.985 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_bm.ko 2024-01-15T17:50:33.985 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_fsm.ko 2024-01-15T17:50:33.985 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_kmp.ko 2024-01-15T17:50:33.985 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/xz/ 2024-01-15T17:50:33.985 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13545 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/lib/xz/xz_dec_test.ko 2024-01-15T17:50:33.985 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/mm/ 2024-01-15T17:50:33.985 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/mm/hwpoison-inject.ko 2024-01-15T17:50:33.985 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ 2024-01-15T17:50:33.985 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/ 2024-01-15T17:50:33.985 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 55737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/6lowpan.ko 2024-01-15T17:50:33.985 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_dest.ko 2024-01-15T17:50:33.985 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_fragment.ko 2024-01-15T17:50:33.985 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_hop.ko 2024-01-15T17:50:33.986 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 6985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_ipv6.ko 2024-01-15T17:50:33.986 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_mobility.ko 2024-01-15T17:50:33.986 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7001 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_routing.ko 2024-01-15T17:50:33.986 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_udp.ko 2024-01-15T17:50:33.986 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/802/ 2024-01-15T17:50:33.986 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/802/garp.ko 2024-01-15T17:50:33.986 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 34929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/802/mrp.ko 2024-01-15T17:50:33.986 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/802/p8022.ko 2024-01-15T17:50:33.986 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/802/psnap.ko 2024-01-15T17:50:33.986 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/802/stp.ko 2024-01-15T17:50:33.986 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/8021q/ 2024-01-15T17:50:33.986 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 82593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/8021q/8021q.ko 2024-01-15T17:50:33.986 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/9p/ 2024-01-15T17:50:33.986 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 198905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet.ko 2024-01-15T17:50:33.987 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 36233 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_fd.ko 2024-01-15T17:50:33.987 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_rdma.ko 2024-01-15T17:50:33.987 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 32337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_virtio.ko 2024-01-15T17:50:33.987 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/batman-adv/ 2024-01-15T17:50:33.987 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 649977 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/batman-adv/batman-adv.ko 2024-01-15T17:50:33.987 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/ 2024-01-15T17:50:33.987 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 65137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/br_netfilter.ko 2024-01-15T17:50:33.987 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 785817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/bridge.ko 2024-01-15T17:50:33.987 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ 2024-01-15T17:50:33.987 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-01-15T17:50:33.987 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_among.ko 2024-01-15T17:50:33.987 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_arp.ko 2024-01-15T17:50:33.987 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-01-15T17:50:33.987 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-01-15T17:50:33.987 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_ip.ko 2024-01-15T17:50:33.988 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9145 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-01-15T17:50:33.988 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_limit.ko 2024-01-15T17:50:33.988 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13217 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_log.ko 2024-01-15T17:50:33.988 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_mark.ko 2024-01-15T17:50:33.988 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-01-15T17:50:33.988 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-01-15T17:50:33.988 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-01-15T17:50:33.988 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9593 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-01-15T17:50:33.988 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_snat.ko 2024-01-15T17:50:33.988 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_stp.ko 2024-01-15T17:50:33.988 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-01-15T17:50:33.988 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11641 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-01-15T17:50:33.988 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-01-15T17:50:33.988 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-01-15T17:50:33.989 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 56033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtables.ko 2024-01-15T17:50:33.989 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-01-15T17:50:33.989 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-01-15T17:50:33.989 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ceph/ 2024-01-15T17:50:33.989 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 1046969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ceph/libceph.ko 2024-01-15T17:50:33.989 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/core/ 2024-01-15T17:50:33.989 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 126041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/core/pktgen.ko 2024-01-15T17:50:33.989 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/ 2024-01-15T17:50:33.989 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 194529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp.ko 2024-01-15T17:50:33.989 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_diag.ko 2024-01-15T17:50:33.989 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 58841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_ipv4.ko 2024-01-15T17:50:33.989 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 60873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_ipv6.ko 2024-01-15T17:50:33.989 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/ 2024-01-15T17:50:33.989 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 390129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/dsa_core.ko 2024-01-15T17:50:33.990 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_brcm.ko 2024-01-15T17:50:33.990 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_dsa.ko 2024-01-15T17:50:33.990 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7721 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_none.ko 2024-01-15T17:50:33.990 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_trailer.ko 2024-01-15T17:50:33.990 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/hsr/ 2024-01-15T17:50:33.990 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 99881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/hsr/hsr.ko 2024-01-15T17:50:33.990 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ 2024-01-15T17:50:33.990 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/6lowpan/ 2024-01-15T17:50:33.990 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 45537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-01-15T17:50:33.990 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 255849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ieee802154.ko 2024-01-15T17:50:33.990 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 47985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ieee802154_socket.ko 2024-01-15T17:50:33.990 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ 2024-01-15T17:50:33.990 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24761 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ah4.ko 2024-01-15T17:50:33.990 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 39065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/esp4.ko 2024-01-15T17:50:33.990 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 48489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/fou.ko 2024-01-15T17:50:33.991 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/gre.ko 2024-01-15T17:50:33.991 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 49601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/inet_diag.ko 2024-01-15T17:50:33.991 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 61273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_gre.ko 2024-01-15T17:50:33.991 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 61401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_tunnel.ko 2024-01-15T17:50:33.991 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 35009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_vti.ko 2024-01-15T17:50:33.991 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ipcomp.ko 2024-01-15T17:50:33.991 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 29001 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ipip.ko 2024-01-15T17:50:33.991 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ 2024-01-15T17:50:33.991 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 37945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arp_tables.ko 2024-01-15T17:50:33.991 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-01-15T17:50:33.991 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-01-15T17:50:33.991 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 41929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ip_tables.ko 2024-01-15T17:50:33.991 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-01-15T17:50:33.991 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2024-01-15T17:50:33.991 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-01-15T17:50:33.991 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11177 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-01-15T17:50:33.992 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-01-15T17:50:33.992 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_filter.ko 2024-01-15T17:50:33.992 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-01-15T17:50:33.992 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-01-15T17:50:33.992 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-01-15T17:50:33.992 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_security.ko 2024-01-15T17:50:33.992 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12545 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-01-15T17:50:33.992 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-01-15T17:50:33.992 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 27625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-01-15T17:50:33.992 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-01-15T17:50:33.992 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-01-15T17:50:33.992 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21001 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2024-01-15T17:50:33.992 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-01-15T17:50:33.992 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15177 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-01-15T17:50:33.993 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-01-15T17:50:33.993 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-01-15T17:50:33.993 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-01-15T17:50:33.993 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_bic.ko 2024-01-15T17:50:33.993 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_dctcp.ko 2024-01-15T17:50:33.993 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_diag.ko 2024-01-15T17:50:33.993 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_highspeed.ko 2024-01-15T17:50:33.993 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_htcp.ko 2024-01-15T17:50:33.993 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_hybla.ko 2024-01-15T17:50:33.993 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_illinois.ko 2024-01-15T17:50:33.993 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_lp.ko 2024-01-15T17:50:33.993 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_scalable.ko 2024-01-15T17:50:33.993 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_vegas.ko 2024-01-15T17:50:33.993 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_veno.ko 2024-01-15T17:50:33.993 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_westwood.ko 2024-01-15T17:50:33.993 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_yeah.ko 2024-01-15T17:50:33.994 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tunnel4.ko 2024-01-15T17:50:33.994 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/udp_diag.ko 2024-01-15T17:50:33.994 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 44977 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/udp_tunnel.ko 2024-01-15T17:50:33.994 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11361 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/xfrm4_tunnel.ko 2024-01-15T17:50:33.994 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ 2024-01-15T17:50:33.994 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ah6.ko 2024-01-15T17:50:33.994 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 39705 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/esp6.ko 2024-01-15T17:50:33.994 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/fou6.ko 2024-01-15T17:50:33.994 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 80105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_gre.ko 2024-01-15T17:50:33.994 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 93977 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_tunnel.ko 2024-01-15T17:50:33.994 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-01-15T17:50:33.994 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 54313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_vti.ko 2024-01-15T17:50:33.994 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ipcomp6.ko 2024-01-15T17:50:33.994 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/mip6.ko 2024-01-15T17:50:33.994 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ 2024-01-15T17:50:33.995 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 42489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-01-15T17:50:33.995 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2024-01-15T17:50:33.995 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-01-15T17:50:33.995 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14137 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-01-15T17:50:33.995 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14273 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-01-15T17:50:33.995 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-01-15T17:50:33.995 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-01-15T17:50:33.995 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-01-15T17:50:33.995 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-01-15T17:50:33.995 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9897 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-01-15T17:50:33.995 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-01-15T17:50:33.995 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10897 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-01-15T17:50:33.995 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-01-15T17:50:33.995 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-01-15T17:50:33.995 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-01-15T17:50:33.995 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-01-15T17:50:33.996 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-01-15T17:50:33.996 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 39225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-01-15T17:50:33.996 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-01-15T17:50:33.996 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 25225 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-01-15T17:50:33.996 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-01-15T17:50:33.996 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-01-15T17:50:33.996 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-01-15T17:50:33.996 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-01-15T17:50:33.996 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-01-15T17:50:33.996 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 74921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/sit.ko 2024-01-15T17:50:33.996 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/tunnel6.ko 2024-01-15T17:50:33.996 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/xfrm6_tunnel.ko 2024-01-15T17:50:33.996 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/key/ 2024-01-15T17:50:33.997 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 84433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/key/af_key.ko 2024-01-15T17:50:33.997 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/ 2024-01-15T17:50:33.997 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 125305 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_core.ko 2024-01-15T17:50:33.997 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20177 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_debugfs.ko 2024-01-15T17:50:33.997 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 25953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_eth.ko 2024-01-15T17:50:33.997 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 36585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ip.ko 2024-01-15T17:50:33.997 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 37465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ip6.ko 2024-01-15T17:50:33.997 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 33569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_netlink.ko 2024-01-15T17:50:33.997 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 53321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ppp.ko 2024-01-15T17:50:33.997 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/lapb/ 2024-01-15T17:50:33.997 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 48121 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/lapb/lapb.ko 2024-01-15T17:50:33.997 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/llc/ 2024-01-15T17:50:33.997 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/llc/llc.ko 2024-01-15T17:50:33.997 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 268265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/llc/llc2.ko 2024-01-15T17:50:33.997 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/mac802154/ 2024-01-15T17:50:33.998 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 290441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/mac802154/mac802154.ko 2024-01-15T17:50:33.998 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/mpls/ 2024-01-15T17:50:33.998 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9393 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/mpls/mpls_gso.ko 2024-01-15T17:50:33.998 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ 2024-01-15T17:50:33.998 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ 2024-01-15T17:50:33.998 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 118801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set.ko 2024-01-15T17:50:33.998 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-01-15T17:50:33.998 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-01-15T17:50:33.998 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26369 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-01-15T17:50:33.998 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 100297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-01-15T17:50:33.998 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 100633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-01-15T17:50:33.998 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 102881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-01-15T17:50:33.998 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 102649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-01-15T17:50:33.998 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 112017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-01-15T17:50:33.998 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 59585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-01-15T17:50:33.999 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 108761 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-01-15T17:50:33.999 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 112297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-01-15T17:50:33.999 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 112649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-01-15T17:50:33.999 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 110753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-01-15T17:50:33.999 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 113633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-01-15T17:50:33.999 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-01-15T17:50:33.999 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ 2024-01-15T17:50:33.999 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 412561 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs.ko 2024-01-15T17:50:33.999 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-01-15T17:50:33.999 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-01-15T17:50:33.999 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-01-15T17:50:33.999 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-01-15T17:50:33.999 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-01-15T17:50:33.999 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-01-15T17:50:34.000 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-01-15T17:50:34.000 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-01-15T17:50:34.000 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-01-15T17:50:34.000 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-01-15T17:50:34.000 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-01-15T17:50:34.000 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-01-15T17:50:34.000 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-01-15T17:50:34.000 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 49265 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conncount.ko 2024-01-15T17:50:34.000 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 411489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack.ko 2024-01-15T17:50:34.000 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-01-15T17:50:34.000 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-01-15T17:50:34.000 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 33761 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-01-15T17:50:34.000 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 118193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_h323.ko 2024-01-15T17:50:34.001 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_irc.ko 2024-01-15T17:50:34.001 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-01-15T17:50:34.001 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 108105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-01-15T17:50:34.001 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 33681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-01-15T17:50:34.001 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_sane.ko 2024-01-15T17:50:34.001 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 66625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_sip.ko 2024-01-15T17:50:34.001 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11233 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-01-15T17:50:34.001 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-01-15T17:50:34.001 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10113 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_dup_netdev.ko 2024-01-15T17:50:34.001 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 91737 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_flow_table.ko 2024-01-15T17:50:34.001 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_flow_table_inet.ko 2024-01-15T17:50:34.001 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 37297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_log_syslog.ko 2024-01-15T17:50:34.001 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 96545 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat.ko 2024-01-15T17:50:34.001 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10129 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_amanda.ko 2024-01-15T17:50:34.001 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_ftp.ko 2024-01-15T17:50:34.002 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_irc.ko 2024-01-15T17:50:34.002 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_sip.ko 2024-01-15T17:50:34.002 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_tftp.ko 2024-01-15T17:50:34.002 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 44297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_synproxy_core.ko 2024-01-15T17:50:34.002 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 614025 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_tables.ko 2024-01-15T17:50:34.002 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 42609 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink.ko 2024-01-15T17:50:34.002 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_acct.ko 2024-01-15T17:50:34.002 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_cttimeout.ko 2024-01-15T17:50:34.002 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 39785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_log.ko 2024-01-15T17:50:34.002 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_osf.ko 2024-01-15T17:50:34.002 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 49401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_queue.ko 2024-01-15T17:50:34.002 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_chain_nat.ko 2024-01-15T17:50:34.003 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 33209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_compat.ko 2024-01-15T17:50:34.003 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 40769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_ct.ko 2024-01-15T17:50:34.003 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9601 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_dup_netdev.ko 2024-01-15T17:50:34.003 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11513 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_fib.ko 2024-01-15T17:50:34.003 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_fwd_netdev.ko 2024-01-15T17:50:34.003 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_hash.ko 2024-01-15T17:50:34.003 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_limit.ko 2024-01-15T17:50:34.003 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_log.ko 2024-01-15T17:50:34.003 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_masq.ko 2024-01-15T17:50:34.003 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_nat.ko 2024-01-15T17:50:34.003 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13673 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_numgen.ko 2024-01-15T17:50:34.003 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_osf.ko 2024-01-15T17:50:34.003 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_queue.ko 2024-01-15T17:50:34.003 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_redir.ko 2024-01-15T17:50:34.004 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_reject.ko 2024-01-15T17:50:34.004 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10081 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_reject_inet.ko 2024-01-15T17:50:34.004 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_socket.ko 2024-01-15T17:50:34.004 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20369 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_synproxy.ko 2024-01-15T17:50:34.004 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_tproxy.ko 2024-01-15T17:50:34.004 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 89313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/x_tables.ko 2024-01-15T17:50:34.004 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_AUDIT.ko 2024-01-15T17:50:34.004 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CHECKSUM.ko 2024-01-15T17:50:34.004 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CLASSIFY.ko 2024-01-15T17:50:34.004 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-01-15T17:50:34.004 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 18169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CT.ko 2024-01-15T17:50:34.004 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_DSCP.ko 2024-01-15T17:50:34.004 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_HL.ko 2024-01-15T17:50:34.004 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13369 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_HMARK.ko 2024-01-15T17:50:34.005 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 38873 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_IDLETIMER.ko 2024-01-15T17:50:34.005 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13889 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_LED.ko 2024-01-15T17:50:34.005 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_LOG.ko 2024-01-15T17:50:34.005 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11753 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_MASQUERADE.ko 2024-01-15T17:50:34.005 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NETMAP.ko 2024-01-15T17:50:34.005 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NFLOG.ko 2024-01-15T17:50:34.005 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NFQUEUE.ko 2024-01-15T17:50:34.005 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_RATEEST.ko 2024-01-15T17:50:34.005 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11073 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_REDIRECT.ko 2024-01-15T17:50:34.005 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14337 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_SECMARK.ko 2024-01-15T17:50:34.005 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TCPMSS.ko 2024-01-15T17:50:34.005 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10433 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-01-15T17:50:34.005 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TEE.ko 2024-01-15T17:50:34.005 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TPROXY.ko 2024-01-15T17:50:34.005 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8321 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TRACE.ko 2024-01-15T17:50:34.005 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13193 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_addrtype.ko 2024-01-15T17:50:34.006 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13897 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_bpf.ko 2024-01-15T17:50:34.006 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cgroup.ko 2024-01-15T17:50:34.006 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cluster.ko 2024-01-15T17:50:34.006 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_comment.ko 2024-01-15T17:50:34.006 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connbytes.ko 2024-01-15T17:50:34.006 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connlabel.ko 2024-01-15T17:50:34.006 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connlimit.ko 2024-01-15T17:50:34.006 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connmark.ko 2024-01-15T17:50:34.006 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_conntrack.ko 2024-01-15T17:50:34.006 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cpu.ko 2024-01-15T17:50:34.006 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10969 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_dccp.ko 2024-01-15T17:50:34.006 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_devgroup.ko 2024-01-15T17:50:34.006 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8801 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_dscp.ko 2024-01-15T17:50:34.007 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10657 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ecn.ko 2024-01-15T17:50:34.007 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_esp.ko 2024-01-15T17:50:34.007 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 43297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_hashlimit.ko 2024-01-15T17:50:34.007 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_helper.ko 2024-01-15T17:50:34.007 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7865 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_hl.ko 2024-01-15T17:50:34.007 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11921 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ipcomp.ko 2024-01-15T17:50:34.007 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_iprange.ko 2024-01-15T17:50:34.007 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11785 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ipvs.ko 2024-01-15T17:50:34.007 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_l2tp.ko 2024-01-15T17:50:34.007 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8001 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_length.ko 2024-01-15T17:50:34.007 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10209 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_limit.ko 2024-01-15T17:50:34.007 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7545 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_mac.ko 2024-01-15T17:50:34.007 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_mark.ko 2024-01-15T17:50:34.007 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11825 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_multiport.ko 2024-01-15T17:50:34.007 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_nat.ko 2024-01-15T17:50:34.008 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10105 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_nfacct.ko 2024-01-15T17:50:34.008 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_osf.ko 2024-01-15T17:50:34.008 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_owner.ko 2024-01-15T17:50:34.008 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9729 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_physdev.ko 2024-01-15T17:50:34.008 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7633 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_pkttype.ko 2024-01-15T17:50:34.008 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11985 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_policy.ko 2024-01-15T17:50:34.008 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_quota.ko 2024-01-15T17:50:34.008 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_rateest.ko 2024-01-15T17:50:34.008 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8257 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_realm.ko 2024-01-15T17:50:34.008 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 36457 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_recent.ko 2024-01-15T17:50:34.008 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13329 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_sctp.ko 2024-01-15T17:50:34.008 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_set.ko 2024-01-15T17:50:34.008 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16913 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_socket.ko 2024-01-15T17:50:34.008 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9505 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_state.ko 2024-01-15T17:50:34.008 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9473 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_statistic.ko 2024-01-15T17:50:34.009 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8809 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_string.ko 2024-01-15T17:50:34.009 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 8553 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_tcpmss.ko 2024-01-15T17:50:34.009 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_tcpudp.ko 2024-01-15T17:50:34.009 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11065 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_time.ko 2024-01-15T17:50:34.009 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_u32.ko 2024-01-15T17:50:34.009 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netlink/ 2024-01-15T17:50:34.009 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 14537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/netlink/netlink_diag.ko 2024-01-15T17:50:34.009 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/nsh/ 2024-01-15T17:50:34.009 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 12425 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/nsh/nsh.ko 2024-01-15T17:50:34.009 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/ 2024-01-15T17:50:34.009 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 383481 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/openvswitch.ko 2024-01-15T17:50:34.009 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-geneve.ko 2024-01-15T17:50:34.009 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9585 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-gre.ko 2024-01-15T17:50:34.009 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11929 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-vxlan.ko 2024-01-15T17:50:34.010 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/packet/ 2024-01-15T17:50:34.010 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13569 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/packet/af_packet_diag.ko 2024-01-15T17:50:34.010 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/phonet/ 2024-01-15T17:50:34.010 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 112057 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/phonet/phonet.ko 2024-01-15T17:50:34.010 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 54857 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/phonet/pn_pep.ko 2024-01-15T17:50:34.010 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/rds/ 2024-01-15T17:50:34.010 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 252953 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds.ko 2024-01-15T17:50:34.010 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 164937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds_rdma.ko 2024-01-15T17:50:34.010 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 68161 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds_tcp.ko 2024-01-15T17:50:34.010 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/rxrpc/ 2024-01-15T17:50:34.010 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 818777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/rxrpc/rxrpc.ko 2024-01-15T17:50:34.010 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/ 2024-01-15T17:50:34.010 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 32241 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_csum.ko 2024-01-15T17:50:34.010 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_gact.ko 2024-01-15T17:50:34.010 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26897 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_ipt.ko 2024-01-15T17:50:34.010 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 34097 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_mirred.ko 2024-01-15T17:50:34.011 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23401 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_nat.ko 2024-01-15T17:50:34.011 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 43449 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_pedit.ko 2024-01-15T17:50:34.011 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 31961 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_police.ko 2024-01-15T17:50:34.011 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20289 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_simple.ko 2024-01-15T17:50:34.011 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 35297 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_skbedit.ko 2024-01-15T17:50:34.011 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 36665 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_vlan.ko 2024-01-15T17:50:34.011 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26465 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_basic.ko 2024-01-15T17:50:34.011 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 34353 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_bpf.ko 2024-01-15T17:50:34.011 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 19281 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_cgroup.ko 2024-01-15T17:50:34.011 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 31489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_flow.ko 2024-01-15T17:50:34.011 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 32377 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_fw.ko 2024-01-15T17:50:34.011 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 40009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_route.ko 2024-01-15T17:50:34.011 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 61529 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_u32.ko 2024-01-15T17:50:34.011 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7441 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_cmp.ko 2024-01-15T17:50:34.012 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 13409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_ipset.ko 2024-01-15T17:50:34.012 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 33313 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_meta.ko 2024-01-15T17:50:34.012 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7897 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_nbyte.ko 2024-01-15T17:50:34.012 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_text.ko 2024-01-15T17:50:34.012 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_u32.ko 2024-01-15T17:50:34.012 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 21769 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_choke.ko 2024-01-15T17:50:34.012 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20849 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_codel.ko 2024-01-15T17:50:34.012 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 34681 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_drr.ko 2024-01-15T17:50:34.012 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 35817 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_fq.ko 2024-01-15T17:50:34.012 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 31617 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_fq_codel.ko 2024-01-15T17:50:34.012 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 35545 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_gred.ko 2024-01-15T17:50:34.012 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 49937 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_hfsc.ko 2024-01-15T17:50:34.012 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 22089 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_hhf.ko 2024-01-15T17:50:34.013 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 66577 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_htb.ko 2024-01-15T17:50:34.013 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_ingress.ko 2024-01-15T17:50:34.013 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 30385 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_mqprio.ko 2024-01-15T17:50:34.013 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10905 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_mqprio_lib.ko 2024-01-15T17:50:34.013 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26185 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_multiq.ko 2024-01-15T17:50:34.013 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 36249 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_netem.ko 2024-01-15T17:50:34.013 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 27497 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_pie.ko 2024-01-15T17:50:34.013 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10745 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_plug.ko 2024-01-15T17:50:34.013 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 24841 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_prio.ko 2024-01-15T17:50:34.013 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 48625 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_qfq.ko 2024-01-15T17:50:34.013 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 33017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_red.ko 2024-01-15T17:50:34.013 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 26881 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_sfb.ko 2024-01-15T17:50:34.013 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 31409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_sfq.ko 2024-01-15T17:50:34.013 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28945 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_tbf.ko 2024-01-15T17:50:34.013 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 28153 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_teql.ko 2024-01-15T17:50:34.014 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/ 2024-01-15T17:50:34.014 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 932169 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/sctp.ko 2024-01-15T17:50:34.014 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/sctp_diag.ko 2024-01-15T17:50:34.014 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/ 2024-01-15T17:50:34.014 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/ 2024-01-15T17:50:34.014 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 336777 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-01-15T17:50:34.014 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 54993 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-01-15T17:50:34.014 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 1518417 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/sunrpc.ko 2024-01-15T17:50:34.014 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/xprtrdma/ 2024-01-15T17:50:34.014 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 708281 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-01-15T17:50:34.014 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/ 2024-01-15T17:50:34.014 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 10489 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/diag.ko 2024-01-15T17:50:34.014 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 866345 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/tipc.ko 2024-01-15T17:50:34.014 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/unix/ 2024-01-15T17:50:34.014 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 16033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/unix/unix_diag.ko 2024-01-15T17:50:34.014 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/ 2024-01-15T17:50:34.015 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 86417 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-01-15T17:50:34.015 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 68049 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-01-15T17:50:34.015 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 98017 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock.ko 2024-01-15T17:50:34.015 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 11177 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock_diag.ko 2024-01-15T17:50:34.015 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 17761 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock_loopback.ko 2024-01-15T17:50:34.015 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/x25/ 2024-01-15T17:50:34.015 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 125409 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/x25/x25.ko 2024-01-15T17:50:34.015 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm/ 2024-01-15T17:50:34.015 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23041 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm/xfrm_algo.ko 2024-01-15T17:50:34.015 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 20649 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm/xfrm_ipcomp.ko 2024-01-15T17:50:34.015 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 104161 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm/xfrm_user.ko 2024-01-15T17:50:34.015 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/virt/ 2024-01-15T17:50:34.015 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/virt/lib/ 2024-01-15T17:50:34.015 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 15537 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/kernel/virt/lib/irqbypass.ko 2024-01-15T17:50:34.015 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 641009 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.alias 2024-01-15T17:50:34.016 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 625033 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.alias.bin 2024-01-15T17:50:34.016 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 7677 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.builtin 2024-01-15T17:50:34.016 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 23634 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.builtin.alias.bin 2024-01-15T17:50:34.016 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 9916 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.builtin.bin 2024-01-15T17:50:34.016 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 70653 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.builtin.modinfo 2024-01-15T17:50:34.016 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 147734 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.dep 2024-01-15T17:50:34.016 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 234045 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.dep.bin 2024-01-15T17:50:34.016 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 210 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.devname 2024-01-15T17:50:34.016 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 72797 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.order 2024-01-15T17:50:34.016 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 879 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.softdep 2024-01-15T17:50:34.016 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 255175 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.symbols 2024-01-15T17:50:34.016 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 309826 2024-01-15 16:53 ./lib/modules/6.7.0-g2668949e0971/modules.symbols.bin 2024-01-15T17:50:34.016 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./usr/ 2024-01-15T17:50:34.016 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./usr/share/ 2024-01-15T17:50:34.016 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./usr/share/doc/ 2024-01-15T17:50:34.017 INFO:teuthology.orchestra.run.smithi093.stdout:drwxr-xr-x root/root 0 2024-01-15 16:53 ./usr/share/doc/linux-image-6.7.0-g2668949e0971/ 2024-01-15T17:50:34.017 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 171 2024-01-15 16:53 ./usr/share/doc/linux-image-6.7.0-g2668949e0971/changelog.Debian.gz 2024-01-15T17:50:34.017 INFO:teuthology.orchestra.run.smithi093.stdout:-rw-r--r-- root/root 691 2024-01-15 16:53 ./usr/share/doc/linux-image-6.7.0-g2668949e0971/copyright 2024-01-15T17:50:34.017 DEBUG:teuthology.task.kernel:get_image_version: 6.7.0-g2668949e0971 2024-01-15T17:50:34.017 INFO:teuthology.task.kernel:searching for kernel 6.7.0-g2668949e0971 2024-01-15T17:50:34.018 DEBUG:teuthology.orchestra.run.smithi093:> egrep '(submenu|menuentry.*6.7.0-g2668949e0971).*{' /boot/grub/grub.cfg 2024-01-15T17:50:34.028 INFO:teuthology.orchestra.run.smithi093.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-01-15T17:50:34.029 INFO:teuthology.orchestra.run.smithi093.stdout: menuentry 'Ubuntu, with Linux 6.7.0-g2668949e0971' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.7.0-g2668949e0971-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-01-15T17:50:34.029 INFO:teuthology.orchestra.run.smithi093.stdout: menuentry 'Ubuntu, with Linux 6.7.0-g2668949e0971 (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.7.0-g2668949e0971-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-01-15T17:50:34.029 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2024-01-15T17:50:34.029 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.7.0-g2668949e0971 2024-01-15T17:50:34.029 DEBUG:teuthology.orchestra.run.smithi093:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2024-01-15T17:50:34.073 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi093.front.sepia.ceph.com to complete... 2024-01-15T17:50:34.471 INFO:teuthology.orchestra.run.smithi093.stderr:Sourcing file `/etc/default/grub' 2024-01-15T17:50:34.473 INFO:teuthology.orchestra.run.smithi093.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-01-15T17:50:34.500 INFO:teuthology.orchestra.run.smithi093.stderr:Generating grub configuration file ... 2024-01-15T17:50:34.717 INFO:teuthology.orchestra.run.smithi093.stderr:Found linux image: /boot/vmlinuz-6.7.0-g2668949e0971 2024-01-15T17:50:34.724 INFO:teuthology.orchestra.run.smithi093.stderr:Found initrd image: /boot/initrd.img-6.7.0-g2668949e0971 2024-01-15T17:50:34.963 INFO:teuthology.orchestra.run.smithi093.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-01-15T17:50:34.968 INFO:teuthology.orchestra.run.smithi093.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-01-15T17:50:35.009 INFO:teuthology.orchestra.run.smithi093.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-01-15T17:50:35.014 INFO:teuthology.orchestra.run.smithi093.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-01-15T17:50:35.140 INFO:teuthology.orchestra.run.smithi093.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-01-15T17:50:35.140 INFO:teuthology.orchestra.run.smithi093.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-01-15T17:50:35.140 INFO:teuthology.orchestra.run.smithi093.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-01-15T17:50:35.153 INFO:teuthology.orchestra.run.smithi093.stderr:done 2024-01-15T17:51:00.408 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-01-15T17:51:00.409 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-01-15T17:51:00.410 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:51:06.284 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-01-15T17:51:06.284 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi093.front.sepia.ceph.com' 2024-01-15T17:51:06.285 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi093.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:51:18.844 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-01-15T17:51:24.732 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.93 2024-01-15T17:51:27.849 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-01-15T17:51:27.850 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:51:30.908 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-01-15T17:51:33.736 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi093.front.sepia.ceph.com' 2024-01-15T17:51:33.736 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi093.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:51:36.796 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.93 2024-01-15T17:51:42.914 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-01-15T17:51:42.914 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:51:45.980 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-01-15T17:51:48.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi093.front.sepia.ceph.com' 2024-01-15T17:51:48.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi093.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:52:00.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-01-15T17:52:00.993 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:52:07.228 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.93 2024-01-15T17:52:19.516 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-01-15T17:52:22.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi093.front.sepia.ceph.com' 2024-01-15T17:52:22.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi093.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:52:22.800 DEBUG:teuthology.orchestra.run.smithi093:> true 2024-01-15T17:52:23.582 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi093.front.sepia.ceph.com' 2024-01-15T17:52:23.582 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.7.0-g2668949e0971"... 2024-01-15T17:52:23.582 DEBUG:teuthology.orchestra.run.smithi093:> uname -r 2024-01-15T17:52:23.633 INFO:teuthology.orchestra.run.smithi093.stdout:6.7.0-g2668949e0971 2024-01-15T17:52:23.634 DEBUG:teuthology.task.kernel:current kernel version is 6.7.0-g2668949e0971 vs 6.7.0-g2668949e0971 2024-01-15T17:52:23.634 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-15T17:52:23.634 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-01-15T17:52:24.635 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-01-15T17:52:24.635 DEBUG:teuthology.orchestra.run.smithi093:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-15T17:52:24.780 INFO:teuthology.orchestra.run.smithi093.stdout:ttyS1 2024-01-15T17:52:24.841 DEBUG:teuthology.parallel:result is None 2024-01-15T17:52:37.517 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-01-15T17:52:37.518 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:52:39.344 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-01-15T17:53:00.346 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-01-15T17:53:00.346 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:53:00.607 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-01-15T17:53:01.369 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-01-15T17:53:01.370 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.7.0-g2668949e0971"... 2024-01-15T17:53:01.370 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-01-15T17:53:01.419 INFO:teuthology.orchestra.run.smithi027.stdout:6.7.0-g2668949e0971 2024-01-15T17:53:01.419 DEBUG:teuthology.task.kernel:current kernel version is 6.7.0-g2668949e0971 vs 6.7.0-g2668949e0971 2024-01-15T17:53:01.419 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-15T17:53:01.420 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-01-15T17:53:02.420 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-01-15T17:53:02.420 DEBUG:teuthology.orchestra.run.smithi027:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-15T17:53:02.513 INFO:teuthology.orchestra.run.smithi027.stdout:ttyS1 2024-01-15T17:53:02.564 DEBUG:teuthology.parallel:result is None 2024-01-15T17:53:02.565 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-15T17:53:02.572 INFO:teuthology.task.internal:Creating test directory... 2024-01-15T17:53:02.572 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-15T17:53:02.576 DEBUG:teuthology.orchestra.run.smithi093:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-15T17:53:02.585 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-15T17:53:02.596 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-15T17:53:02.609 INFO:teuthology.task.internal:Creating archive directory... 2024-01-15T17:53:02.609 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-15T17:53:02.626 DEBUG:teuthology.orchestra.run.smithi093:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-15T17:53:02.648 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-15T17:53:02.659 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-15T17:53:02.659 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-15T17:53:02.686 DEBUG:teuthology.orchestra.run.smithi093:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-15T17:53:02.709 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-15T17:53:02.723 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-15T17:53:02.723 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-15T17:53:02.736 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-15T17:53:02.738 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-15T17:53:02.749 INFO:teuthology.task.internal:Configuring sudo... 2024-01-15T17:53:02.749 DEBUG:teuthology.orchestra.run.smithi027:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-15T17:53:02.775 DEBUG:teuthology.orchestra.run.smithi093:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-15T17:53:02.802 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-15T17:53:02.812 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-15T17:53:02.812 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-15T17:53:02.842 DEBUG:teuthology.orchestra.run.smithi093:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-15T17:53:02.853 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-15T17:53:02.900 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-15T17:53:02.952 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T17:53:02.952 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-15T17:53:03.021 DEBUG:teuthology.orchestra.run.smithi093:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-15T17:53:03.030 DEBUG:teuthology.orchestra.run.smithi093:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-15T17:53:03.082 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T17:53:03.082 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-15T17:53:03.160 DEBUG:teuthology.orchestra.run.smithi027:> sudo service rsyslog restart 2024-01-15T17:53:03.164 DEBUG:teuthology.orchestra.run.smithi093:> sudo service rsyslog restart 2024-01-15T17:53:03.275 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-15T17:53:03.283 INFO:teuthology.task.internal:Starting timer... 2024-01-15T17:53:03.283 INFO:teuthology.run_tasks:Running task pcp... 2024-01-15T17:53:03.299 INFO:teuthology.run_tasks:Running task selinux... 2024-01-15T17:53:03.380 DEBUG:teuthology.task.selinux:Excluding smithi027: OS 'ubuntu' does not support SELinux 2024-01-15T17:53:03.380 DEBUG:teuthology.task.selinux:Excluding smithi093: OS 'ubuntu' does not support SELinux 2024-01-15T17:53:03.380 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-15T17:53:03.380 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-01-15T17:53:03.380 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-15T17:53:03.380 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-15T17:53:03.395 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-15T17:53:03.409 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-15T17:53:03.410 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi027.front.sepia.ceph.com,smithi093.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-15T18:00:32.939 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi027.front.sepia.ceph.com'), Remote(name='ubuntu@smithi093.front.sepia.ceph.com')] 2024-01-15T18:00:32.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-01-15T18:00:32.941 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T18:00:33.025 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-01-15T18:00:33.159 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-01-15T18:00:33.159 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi093.front.sepia.ceph.com' 2024-01-15T18:00:33.159 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi093.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T18:00:33.236 DEBUG:teuthology.orchestra.run.smithi093:> true 2024-01-15T18:00:33.363 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi093.front.sepia.ceph.com' 2024-01-15T18:00:33.363 INFO:teuthology.run_tasks:Running task clock... 2024-01-15T18:00:33.374 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-15T18:00:33.375 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-15T18:00:33.375 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-15T18:00:33.378 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-15T18:00:33.378 DEBUG:teuthology.orchestra.run.smithi093:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-15T18:00:33.414 INFO:teuthology.orchestra.run.smithi027.stdout:15 Jan 18:00:33 ntpd[16733]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-01-15T18:00:33.414 INFO:teuthology.orchestra.run.smithi027.stdout:15 Jan 18:00:33 ntpd[16733]: Command line: ntpd -gq 2024-01-15T18:00:33.414 INFO:teuthology.orchestra.run.smithi027.stdout:15 Jan 18:00:33 ntpd[16733]: ---------------------------------------------------- 2024-01-15T18:00:33.414 INFO:teuthology.orchestra.run.smithi027.stdout:15 Jan 18:00:33 ntpd[16733]: ntp-4 is maintained by Network Time Foundation, 2024-01-15T18:00:33.414 INFO:teuthology.orchestra.run.smithi027.stdout:15 Jan 18:00:33 ntpd[16733]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-01-15T18:00:33.415 INFO:teuthology.orchestra.run.smithi027.stdout:15 Jan 18:00:33 ntpd[16733]: corporation. Support and training for ntp-4 are 2024-01-15T18:00:33.415 INFO:teuthology.orchestra.run.smithi027.stdout:15 Jan 18:00:33 ntpd[16733]: available at https://www.nwtime.org/support 2024-01-15T18:00:33.415 INFO:teuthology.orchestra.run.smithi027.stdout:15 Jan 18:00:33 ntpd[16733]: ---------------------------------------------------- 2024-01-15T18:00:33.416 INFO:teuthology.orchestra.run.smithi027.stdout:15 Jan 18:00:33 ntpd[16733]: proto: precision = 0.042 usec (-24) 2024-01-15T18:00:33.416 INFO:teuthology.orchestra.run.smithi027.stdout:15 Jan 18:00:33 ntpd[16733]: basedate set to 2022-02-04 2024-01-15T18:00:33.416 INFO:teuthology.orchestra.run.smithi027.stdout:15 Jan 18:00:33 ntpd[16733]: gps base set to 2022-02-06 (week 2196) 2024-01-15T18:00:33.416 INFO:teuthology.orchestra.run.smithi027.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-01-15T18:00:33.417 INFO:teuthology.orchestra.run.smithi027.stdout:15 Jan 18:00:33 ntpd[16733]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-01-15T18:00:33.417 INFO:teuthology.orchestra.run.smithi027.stdout:15 Jan 18:00:33 ntpd[16733]: restrict ::: KOD does nothing without LIMITED. 2024-01-15T18:00:33.417 INFO:teuthology.orchestra.run.smithi027.stderr:restrict ::: KOD does nothing without LIMITED. 2024-01-15T18:00:33.418 INFO:teuthology.orchestra.run.smithi027.stdout:15 Jan 18:00:33 ntpd[16733]: Listen and drop on 0 v6wildcard [::]:123 2024-01-15T18:00:33.418 INFO:teuthology.orchestra.run.smithi027.stdout:15 Jan 18:00:33 ntpd[16733]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-01-15T18:00:33.418 INFO:teuthology.orchestra.run.smithi027.stdout:15 Jan 18:00:33 ntpd[16733]: Listen normally on 2 lo 127.0.0.1:123 2024-01-15T18:00:33.418 INFO:teuthology.orchestra.run.smithi027.stdout:15 Jan 18:00:33 ntpd[16733]: Listen normally on 3 ens1f0 172.21.15.27:123 2024-01-15T18:00:33.418 INFO:teuthology.orchestra.run.smithi027.stdout:15 Jan 18:00:33 ntpd[16733]: Listen normally on 4 lo [::1]:123 2024-01-15T18:00:33.418 INFO:teuthology.orchestra.run.smithi027.stdout:15 Jan 18:00:33 ntpd[16733]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fef8:395a%4]:123 2024-01-15T18:00:33.418 INFO:teuthology.orchestra.run.smithi027.stdout:15 Jan 18:00:33 ntpd[16733]: Listening on routing socket on fd #22 for interface updates 2024-01-15T18:00:33.442 INFO:teuthology.orchestra.run.smithi093.stdout:15 Jan 18:00:33 ntpd[16831]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-01-15T18:00:33.442 INFO:teuthology.orchestra.run.smithi093.stdout:15 Jan 18:00:33 ntpd[16831]: Command line: ntpd -gq 2024-01-15T18:00:33.442 INFO:teuthology.orchestra.run.smithi093.stdout:15 Jan 18:00:33 ntpd[16831]: ---------------------------------------------------- 2024-01-15T18:00:33.442 INFO:teuthology.orchestra.run.smithi093.stdout:15 Jan 18:00:33 ntpd[16831]: ntp-4 is maintained by Network Time Foundation, 2024-01-15T18:00:33.442 INFO:teuthology.orchestra.run.smithi093.stdout:15 Jan 18:00:33 ntpd[16831]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-01-15T18:00:33.443 INFO:teuthology.orchestra.run.smithi093.stdout:15 Jan 18:00:33 ntpd[16831]: corporation. Support and training for ntp-4 are 2024-01-15T18:00:33.443 INFO:teuthology.orchestra.run.smithi093.stdout:15 Jan 18:00:33 ntpd[16831]: available at https://www.nwtime.org/support 2024-01-15T18:00:33.443 INFO:teuthology.orchestra.run.smithi093.stdout:15 Jan 18:00:33 ntpd[16831]: ---------------------------------------------------- 2024-01-15T18:00:33.444 INFO:teuthology.orchestra.run.smithi093.stdout:15 Jan 18:00:33 ntpd[16831]: proto: precision = 0.041 usec (-24) 2024-01-15T18:00:33.444 INFO:teuthology.orchestra.run.smithi093.stdout:15 Jan 18:00:33 ntpd[16831]: basedate set to 2022-02-04 2024-01-15T18:00:33.444 INFO:teuthology.orchestra.run.smithi093.stdout:15 Jan 18:00:33 ntpd[16831]: gps base set to 2022-02-06 (week 2196) 2024-01-15T18:00:33.444 INFO:teuthology.orchestra.run.smithi093.stdout:15 Jan 18:00:33 ntpd[16831]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-01-15T18:00:33.444 INFO:teuthology.orchestra.run.smithi093.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-01-15T18:00:33.444 INFO:teuthology.orchestra.run.smithi093.stderr:restrict ::: KOD does nothing without LIMITED. 2024-01-15T18:00:33.445 INFO:teuthology.orchestra.run.smithi093.stdout:15 Jan 18:00:33 ntpd[16831]: restrict ::: KOD does nothing without LIMITED. 2024-01-15T18:00:33.446 INFO:teuthology.orchestra.run.smithi093.stdout:15 Jan 18:00:33 ntpd[16831]: Listen and drop on 0 v6wildcard [::]:123 2024-01-15T18:00:33.446 INFO:teuthology.orchestra.run.smithi093.stdout:15 Jan 18:00:33 ntpd[16831]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-01-15T18:00:33.446 INFO:teuthology.orchestra.run.smithi093.stdout:15 Jan 18:00:33 ntpd[16831]: Listen normally on 2 lo 127.0.0.1:123 2024-01-15T18:00:33.446 INFO:teuthology.orchestra.run.smithi093.stdout:15 Jan 18:00:33 ntpd[16831]: Listen normally on 3 enp3s0f1 172.21.15.93:123 2024-01-15T18:00:33.446 INFO:teuthology.orchestra.run.smithi093.stdout:15 Jan 18:00:33 ntpd[16831]: Listen normally on 4 lo [::1]:123 2024-01-15T18:00:33.446 INFO:teuthology.orchestra.run.smithi093.stdout:15 Jan 18:00:33 ntpd[16831]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d635%5]:123 2024-01-15T18:00:33.446 INFO:teuthology.orchestra.run.smithi093.stdout:15 Jan 18:00:33 ntpd[16831]: Listening on routing socket on fd #22 for interface updates 2024-01-15T18:00:34.416 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:34 ntpd[16733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:34.416 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:34 ntpd[16733]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:34.416 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:34 ntpd[16733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:34.416 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:34 ntpd[16733]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:34.417 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:34 ntpd[16733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:34.417 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:34 ntpd[16733]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:34.417 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:34 ntpd[16733]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-15T18:00:34.417 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:34 ntpd[16733]: can't open /var/log/ntpstats/peerstats.20240115: Permission denied 2024-01-15T18:00:34.443 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:34 ntpd[16831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:34.443 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:34 ntpd[16831]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:36.418 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:36 ntpd[16733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:36.419 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:36 ntpd[16733]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:36.419 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:36 ntpd[16733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:36.419 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:36 ntpd[16733]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:36.419 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:36 ntpd[16733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:36.420 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:36 ntpd[16733]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:36.420 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:36 ntpd[16733]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-15T18:00:36.420 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:36 ntpd[16733]: can't open /var/log/ntpstats/peerstats.20240115: Permission denied 2024-01-15T18:00:36.420 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:36 ntpd[16733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:36.421 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:36 ntpd[16733]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:36.421 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:36 ntpd[16733]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-15T18:00:36.421 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:36 ntpd[16733]: can't open /var/log/ntpstats/peerstats.20240115: Permission denied 2024-01-15T18:00:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:37 ntpd[16733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:37.415 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:37 ntpd[16733]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:37.416 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:37 ntpd[16733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:37.416 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:37 ntpd[16733]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:37.416 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:37 ntpd[16733]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-15T18:00:37.416 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:37 ntpd[16733]: can't open /var/log/ntpstats/peerstats.20240115: Permission denied 2024-01-15T18:00:37.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:37 ntpd[16831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:37.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:37 ntpd[16831]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:37.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:37 ntpd[16831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:37.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:37 ntpd[16831]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:37.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:37 ntpd[16831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:37.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:37 ntpd[16831]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:37.446 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:37 ntpd[16831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:37.446 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:37 ntpd[16831]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:37.446 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:37 ntpd[16831]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-15T18:00:37.446 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:37 ntpd[16831]: can't open /var/log/ntpstats/peerstats.20240115: Permission denied 2024-01-15T18:00:37.446 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:37 ntpd[16831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:37.446 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:37 ntpd[16831]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:37.446 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:37 ntpd[16831]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-15T18:00:37.446 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:37 ntpd[16831]: can't open /var/log/ntpstats/peerstats.20240115: Permission denied 2024-01-15T18:00:37.446 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:37 ntpd[16831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:37.447 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:37 ntpd[16831]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:37.447 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:37 ntpd[16831]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-15T18:00:37.447 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:37 ntpd[16831]: can't open /var/log/ntpstats/peerstats.20240115: Permission denied 2024-01-15T18:00:38.416 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:38 ntpd[16733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:38.416 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:38 ntpd[16733]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:38.416 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:38 ntpd[16733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:38.416 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:38 ntpd[16733]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:38.416 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:38 ntpd[16733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:38.416 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:38 ntpd[16733]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:38.416 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:38 ntpd[16733]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-15T18:00:38.417 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:38 ntpd[16733]: can't open /var/log/ntpstats/peerstats.20240115: Permission denied 2024-01-15T18:00:38.417 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:38 ntpd[16733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:38.417 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:38 ntpd[16733]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:38.417 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:38 ntpd[16733]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-15T18:00:38.417 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:38 ntpd[16733]: can't open /var/log/ntpstats/peerstats.20240115: Permission denied 2024-01-15T18:00:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:39 ntpd[16733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:39 ntpd[16733]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:39 ntpd[16733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:39 ntpd[16733]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:39 ntpd[16733]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-15T18:00:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:39 ntpd[16733]: can't open /var/log/ntpstats/peerstats.20240115: Permission denied 2024-01-15T18:00:39.444 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:39 ntpd[16831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:39.444 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:39 ntpd[16831]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:39.444 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:39 ntpd[16831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:39.444 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:39 ntpd[16831]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:39.444 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:39 ntpd[16831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:39.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:39 ntpd[16831]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:39.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:39 ntpd[16831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:39.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:39 ntpd[16831]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:39.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:39 ntpd[16831]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-15T18:00:39.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:39 ntpd[16831]: can't open /var/log/ntpstats/peerstats.20240115: Permission denied 2024-01-15T18:00:39.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:39 ntpd[16831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:39.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:39 ntpd[16831]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:39.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:39 ntpd[16831]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-15T18:00:39.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:39 ntpd[16831]: can't open /var/log/ntpstats/peerstats.20240115: Permission denied 2024-01-15T18:00:39.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:39 ntpd[16831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:39.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:39 ntpd[16831]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:39.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:39 ntpd[16831]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-15T18:00:39.446 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:39 ntpd[16831]: can't open /var/log/ntpstats/peerstats.20240115: Permission denied 2024-01-15T18:00:40.416 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:40 ntpd[16733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:40.416 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:40 ntpd[16733]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:40.416 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:40 ntpd[16733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:40.416 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:40 ntpd[16733]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:40.416 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:40 ntpd[16733]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:40.416 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:40 ntpd[16733]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:40.416 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:40 ntpd[16733]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-15T18:00:40.417 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:40 ntpd[16733]: can't open /var/log/ntpstats/peerstats.20240115: Permission denied 2024-01-15T18:00:40.417 INFO:teuthology.orchestra.run.smithi027.stdout:15 Jan 18:00:40 ntpd[16733]: ntpd: time slew -0.000310 s 2024-01-15T18:00:40.417 INFO:teuthology.orchestra.run.smithi027.stdout:ntpd: time slew -0.000310s 2024-01-15T18:00:40.418 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:40 ntpd[16733]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-01-15T18:00:40.418 INFO:teuthology.orchestra.run.smithi027.stderr:15 Jan 18:00:40 ntpd[16733]: can't open /var/log/ntpstats/loopstats.20240115: Permission denied 2024-01-15T18:00:40.519 INFO:teuthology.orchestra.run.smithi027.stdout: remote refid st t when poll reach delay offset jitter 2024-01-15T18:00:40.519 INFO:teuthology.orchestra.run.smithi027.stdout:============================================================================== 2024-01-15T18:00:40.519 INFO:teuthology.orchestra.run.smithi027.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-01-15T18:00:40.520 INFO:teuthology.orchestra.run.smithi027.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-01-15T18:00:40.520 INFO:teuthology.orchestra.run.smithi027.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-01-15T18:00:40.520 INFO:teuthology.orchestra.run.smithi027.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-01-15T18:00:41.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:41 ntpd[16831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:41.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:41 ntpd[16831]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:41.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:41 ntpd[16831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:41.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:41 ntpd[16831]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:41.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:41 ntpd[16831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:41.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:41 ntpd[16831]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:41.446 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:41 ntpd[16831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:41.446 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:41 ntpd[16831]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:41.446 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:41 ntpd[16831]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-15T18:00:41.446 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:41 ntpd[16831]: can't open /var/log/ntpstats/peerstats.20240115: Permission denied 2024-01-15T18:00:41.446 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:41 ntpd[16831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:41.446 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:41 ntpd[16831]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:41.446 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:41 ntpd[16831]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-15T18:00:41.446 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:41 ntpd[16831]: can't open /var/log/ntpstats/peerstats.20240115: Permission denied 2024-01-15T18:00:41.446 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:41 ntpd[16831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:41.446 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:41 ntpd[16831]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:41.446 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:41 ntpd[16831]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-15T18:00:41.447 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:41 ntpd[16831]: can't open /var/log/ntpstats/peerstats.20240115: Permission denied 2024-01-15T18:00:43.444 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:43 ntpd[16831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:43.444 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:43 ntpd[16831]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:43.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:43 ntpd[16831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:43.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:43 ntpd[16831]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:43.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:43 ntpd[16831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:43.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:43 ntpd[16831]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:43.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:43 ntpd[16831]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-01-15T18:00:43.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:43 ntpd[16831]: can't open /var/log/ntpstats/rawstats.20240115: Permission denied 2024-01-15T18:00:43.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:43 ntpd[16831]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-01-15T18:00:43.445 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:43 ntpd[16831]: can't open /var/log/ntpstats/peerstats.20240115: Permission denied 2024-01-15T18:00:43.446 INFO:teuthology.orchestra.run.smithi093.stdout:15 Jan 18:00:43 ntpd[16831]: ntpd: time slew -0.002113 s 2024-01-15T18:00:43.446 INFO:teuthology.orchestra.run.smithi093.stdout:ntpd: time slew -0.002113s 2024-01-15T18:00:43.446 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:43 ntpd[16831]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-01-15T18:00:43.446 INFO:teuthology.orchestra.run.smithi093.stderr:15 Jan 18:00:43 ntpd[16831]: can't open /var/log/ntpstats/loopstats.20240115: Permission denied 2024-01-15T18:00:43.547 INFO:teuthology.orchestra.run.smithi093.stdout: remote refid st t when poll reach delay offset jitter 2024-01-15T18:00:43.547 INFO:teuthology.orchestra.run.smithi093.stdout:============================================================================== 2024-01-15T18:00:43.547 INFO:teuthology.orchestra.run.smithi093.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-01-15T18:00:43.547 INFO:teuthology.orchestra.run.smithi093.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-01-15T18:00:43.547 INFO:teuthology.orchestra.run.smithi093.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-01-15T18:00:43.547 INFO:teuthology.orchestra.run.smithi093.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-01-15T18:00:43.549 INFO:teuthology.run_tasks:Running task install... 2024-01-15T18:00:43.559 DEBUG:teuthology.task.install:project ceph 2024-01-15T18:00:43.559 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b'}} 2024-01-15T18:00:43.559 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b'} 2024-01-15T18:00:43.559 INFO:teuthology.task.install:Using flavor: default 2024-01-15T18:00:43.566 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-15T18:00:43.566 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-01-15T18:00:43.566 DEBUG:teuthology.orchestra.run.smithi027:> sudo apt-key list | grep Ceph 2024-01-15T18:00:43.567 DEBUG:teuthology.orchestra.run.smithi093:> sudo apt-key list | grep Ceph 2024-01-15T18:00:43.676 INFO:teuthology.orchestra.run.smithi027.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-01-15T18:00:43.693 INFO:teuthology.orchestra.run.smithi093.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-01-15T18:00:43.729 INFO:teuthology.orchestra.run.smithi027.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-01-15T18:00:43.729 INFO:teuthology.orchestra.run.smithi027.stdout:uid [ unknown] Ceph.com (release key) 2024-01-15T18:00:43.730 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-01-15T18:00:43.730 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-01-15T18:00:43.731 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=95c90f413c4bb65e3e56909855f2308fd9bda05b 2024-01-15T18:00:43.744 INFO:teuthology.orchestra.run.smithi093.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-01-15T18:00:43.744 INFO:teuthology.orchestra.run.smithi093.stdout:uid [ unknown] Ceph.com (release key) 2024-01-15T18:00:43.745 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-01-15T18:00:43.746 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-01-15T18:00:43.746 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=95c90f413c4bb65e3e56909855f2308fd9bda05b 2024-01-15T18:00:44.078 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default/ 2024-01-15T18:00:44.078 INFO:teuthology.task.install.deb:Package version is 18.0.0-7629-g95c90f41-1jammy 2024-01-15T18:00:44.082 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default/ 2024-01-15T18:00:44.083 INFO:teuthology.task.install.deb:Package version is 18.0.0-7629-g95c90f41-1jammy 2024-01-15T18:00:44.193 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:00:44.194 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-01-15T18:00:44.214 DEBUG:teuthology.orchestra.run.smithi027:> sudo apt-get update 2024-01-15T18:00:44.353 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:00:44.353 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-01-15T18:00:44.374 DEBUG:teuthology.orchestra.run.smithi093:> sudo apt-get update 2024-01-15T18:00:44.462 INFO:teuthology.orchestra.run.smithi027.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-01-15T18:00:44.463 INFO:teuthology.orchestra.run.smithi027.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy InRelease 2024-01-15T18:00:44.483 INFO:teuthology.orchestra.run.smithi027.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-01-15T18:00:44.505 INFO:teuthology.orchestra.run.smithi027.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy Release.gpg 2024-01-15T18:00:44.521 INFO:teuthology.orchestra.run.smithi027.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-01-15T18:00:44.526 INFO:teuthology.orchestra.run.smithi027.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,412 B] 2024-01-15T18:00:44.558 INFO:teuthology.orchestra.run.smithi027.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2024-01-15T18:00:44.603 INFO:teuthology.orchestra.run.smithi027.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-01-15T18:00:44.615 INFO:teuthology.orchestra.run.smithi093.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy InRelease 2024-01-15T18:00:44.636 INFO:teuthology.orchestra.run.smithi093.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-01-15T18:00:44.658 INFO:teuthology.orchestra.run.smithi093.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy Release.gpg 2024-01-15T18:00:44.684 INFO:teuthology.orchestra.run.smithi027.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-01-15T18:00:44.691 INFO:teuthology.orchestra.run.smithi093.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-01-15T18:00:44.691 INFO:teuthology.orchestra.run.smithi093.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-01-15T18:00:44.691 INFO:teuthology.orchestra.run.smithi093.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2024-01-15T18:00:44.722 INFO:teuthology.orchestra.run.smithi093.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,412 B] 2024-01-15T18:00:44.761 INFO:teuthology.orchestra.run.smithi093.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-01-15T18:00:44.843 INFO:teuthology.orchestra.run.smithi093.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-01-15T18:00:45.069 INFO:teuthology.orchestra.run.smithi027.stdout:Fetched 28.6 kB in 1s (45.8 kB/s) 2024-01-15T18:00:45.274 INFO:teuthology.orchestra.run.smithi093.stdout:Fetched 28.6 kB in 1s (42.3 kB/s) 2024-01-15T18:00:46.065 INFO:teuthology.orchestra.run.smithi027.stdout:Reading package lists... 2024-01-15T18:00:46.082 DEBUG:teuthology.orchestra.run.smithi027:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-7629-g95c90f41-1jammy cephadm=18.0.0-7629-g95c90f41-1jammy ceph-mds=18.0.0-7629-g95c90f41-1jammy ceph-mgr=18.0.0-7629-g95c90f41-1jammy ceph-common=18.0.0-7629-g95c90f41-1jammy ceph-fuse=18.0.0-7629-g95c90f41-1jammy ceph-test=18.0.0-7629-g95c90f41-1jammy ceph-volume=18.0.0-7629-g95c90f41-1jammy radosgw=18.0.0-7629-g95c90f41-1jammy python3-rados=18.0.0-7629-g95c90f41-1jammy python3-rgw=18.0.0-7629-g95c90f41-1jammy python3-cephfs=18.0.0-7629-g95c90f41-1jammy python3-rbd=18.0.0-7629-g95c90f41-1jammy libcephfs2=18.0.0-7629-g95c90f41-1jammy libcephfs-dev=18.0.0-7629-g95c90f41-1jammy librados2=18.0.0-7629-g95c90f41-1jammy librbd1=18.0.0-7629-g95c90f41-1jammy rbd-fuse=18.0.0-7629-g95c90f41-1jammy python3-cephfs=18.0.0-7629-g95c90f41-1jammy cephfs-shell=18.0.0-7629-g95c90f41-1jammy cephfs-top=18.0.0-7629-g95c90f41-1jammy cephfs-mirror=18.0.0-7629-g95c90f41-1jammy 2024-01-15T18:00:46.158 INFO:teuthology.orchestra.run.smithi027.stdout:Reading package lists... 2024-01-15T18:00:46.191 INFO:teuthology.orchestra.run.smithi093.stdout:Reading package lists... 2024-01-15T18:00:46.205 DEBUG:teuthology.orchestra.run.smithi093:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-7629-g95c90f41-1jammy cephadm=18.0.0-7629-g95c90f41-1jammy ceph-mds=18.0.0-7629-g95c90f41-1jammy ceph-mgr=18.0.0-7629-g95c90f41-1jammy ceph-common=18.0.0-7629-g95c90f41-1jammy ceph-fuse=18.0.0-7629-g95c90f41-1jammy ceph-test=18.0.0-7629-g95c90f41-1jammy ceph-volume=18.0.0-7629-g95c90f41-1jammy radosgw=18.0.0-7629-g95c90f41-1jammy python3-rados=18.0.0-7629-g95c90f41-1jammy python3-rgw=18.0.0-7629-g95c90f41-1jammy python3-cephfs=18.0.0-7629-g95c90f41-1jammy python3-rbd=18.0.0-7629-g95c90f41-1jammy libcephfs2=18.0.0-7629-g95c90f41-1jammy libcephfs-dev=18.0.0-7629-g95c90f41-1jammy librados2=18.0.0-7629-g95c90f41-1jammy librbd1=18.0.0-7629-g95c90f41-1jammy rbd-fuse=18.0.0-7629-g95c90f41-1jammy python3-cephfs=18.0.0-7629-g95c90f41-1jammy cephfs-shell=18.0.0-7629-g95c90f41-1jammy cephfs-top=18.0.0-7629-g95c90f41-1jammy cephfs-mirror=18.0.0-7629-g95c90f41-1jammy 2024-01-15T18:00:46.278 INFO:teuthology.orchestra.run.smithi093.stdout:Reading package lists... 2024-01-15T18:00:46.325 INFO:teuthology.orchestra.run.smithi027.stdout:Building dependency tree... 2024-01-15T18:00:46.325 INFO:teuthology.orchestra.run.smithi027.stdout:Reading state information... 2024-01-15T18:00:46.441 INFO:teuthology.orchestra.run.smithi093.stdout:Building dependency tree... 2024-01-15T18:00:46.441 INFO:teuthology.orchestra.run.smithi093.stdout:Reading state information... 2024-01-15T18:00:46.486 INFO:teuthology.orchestra.run.smithi027.stdout:The following packages were automatically installed and are no longer required: 2024-01-15T18:00:46.486 INFO:teuthology.orchestra.run.smithi027.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-01-15T18:00:46.486 INFO:teuthology.orchestra.run.smithi027.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-01-15T18:00:46.486 INFO:teuthology.orchestra.run.smithi027.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-01-15T18:00:46.486 INFO:teuthology.orchestra.run.smithi027.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-01-15T18:00:46.486 INFO:teuthology.orchestra.run.smithi027.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-01-15T18:00:46.486 INFO:teuthology.orchestra.run.smithi027.stdout:Use 'sudo apt autoremove' to remove them. 2024-01-15T18:00:46.489 INFO:teuthology.orchestra.run.smithi027.stdout:The following additional packages will be installed: 2024-01-15T18:00:46.489 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-01-15T18:00:46.489 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-01-15T18:00:46.489 INFO:teuthology.orchestra.run.smithi027.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-01-15T18:00:46.489 INFO:teuthology.orchestra.run.smithi027.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-01-15T18:00:46.489 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-01-15T18:00:46.489 INFO:teuthology.orchestra.run.smithi027.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-01-15T18:00:46.490 INFO:teuthology.orchestra.run.smithi027.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-01-15T18:00:46.490 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-01-15T18:00:46.490 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-01-15T18:00:46.490 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-01-15T18:00:46.490 INFO:teuthology.orchestra.run.smithi027.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-01-15T18:00:46.490 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-01-15T18:00:46.490 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-01-15T18:00:46.490 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-01-15T18:00:46.490 INFO:teuthology.orchestra.run.smithi027.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-01-15T18:00:46.490 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-01-15T18:00:46.490 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-01-15T18:00:46.490 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-01-15T18:00:46.491 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-01-15T18:00:46.491 INFO:teuthology.orchestra.run.smithi027.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-01-15T18:00:46.491 INFO:teuthology.orchestra.run.smithi027.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-01-15T18:00:46.491 INFO:teuthology.orchestra.run.smithi027.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-01-15T18:00:46.491 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-01-15T18:00:46.491 INFO:teuthology.orchestra.run.smithi027.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-01-15T18:00:46.491 INFO:teuthology.orchestra.run.smithi027.stdout:Suggested packages: 2024-01-15T18:00:46.491 INFO:teuthology.orchestra.run.smithi027.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-01-15T18:00:46.491 INFO:teuthology.orchestra.run.smithi027.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-01-15T18:00:46.491 INFO:teuthology.orchestra.run.smithi027.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-01-15T18:00:46.491 INFO:teuthology.orchestra.run.smithi027.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-01-15T18:00:46.491 INFO:teuthology.orchestra.run.smithi027.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-01-15T18:00:46.492 INFO:teuthology.orchestra.run.smithi027.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-01-15T18:00:46.492 INFO:teuthology.orchestra.run.smithi027.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-01-15T18:00:46.492 INFO:teuthology.orchestra.run.smithi027.stdout:Recommended packages: 2024-01-15T18:00:46.492 INFO:teuthology.orchestra.run.smithi027.stdout: btrfs-tools 2024-01-15T18:00:46.594 INFO:teuthology.orchestra.run.smithi093.stdout:The following packages were automatically installed and are no longer required: 2024-01-15T18:00:46.594 INFO:teuthology.orchestra.run.smithi093.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-01-15T18:00:46.594 INFO:teuthology.orchestra.run.smithi093.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-01-15T18:00:46.594 INFO:teuthology.orchestra.run.smithi093.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-01-15T18:00:46.594 INFO:teuthology.orchestra.run.smithi093.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-01-15T18:00:46.597 INFO:teuthology.orchestra.run.smithi093.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-01-15T18:00:46.597 INFO:teuthology.orchestra.run.smithi093.stdout:Use 'sudo apt autoremove' to remove them. 2024-01-15T18:00:46.598 INFO:teuthology.orchestra.run.smithi093.stdout:The following additional packages will be installed: 2024-01-15T18:00:46.598 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-01-15T18:00:46.598 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-01-15T18:00:46.598 INFO:teuthology.orchestra.run.smithi093.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-01-15T18:00:46.598 INFO:teuthology.orchestra.run.smithi093.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-01-15T18:00:46.598 INFO:teuthology.orchestra.run.smithi093.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-01-15T18:00:46.598 INFO:teuthology.orchestra.run.smithi093.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-01-15T18:00:46.598 INFO:teuthology.orchestra.run.smithi093.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-01-15T18:00:46.598 INFO:teuthology.orchestra.run.smithi093.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-01-15T18:00:46.598 INFO:teuthology.orchestra.run.smithi093.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-01-15T18:00:46.598 INFO:teuthology.orchestra.run.smithi093.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-01-15T18:00:46.598 INFO:teuthology.orchestra.run.smithi093.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-01-15T18:00:46.599 INFO:teuthology.orchestra.run.smithi093.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-01-15T18:00:46.599 INFO:teuthology.orchestra.run.smithi093.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-01-15T18:00:46.599 INFO:teuthology.orchestra.run.smithi093.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-01-15T18:00:46.599 INFO:teuthology.orchestra.run.smithi093.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-01-15T18:00:46.599 INFO:teuthology.orchestra.run.smithi093.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-01-15T18:00:46.599 INFO:teuthology.orchestra.run.smithi093.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-01-15T18:00:46.599 INFO:teuthology.orchestra.run.smithi093.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-01-15T18:00:46.599 INFO:teuthology.orchestra.run.smithi093.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-01-15T18:00:46.599 INFO:teuthology.orchestra.run.smithi093.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-01-15T18:00:46.599 INFO:teuthology.orchestra.run.smithi093.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-01-15T18:00:46.599 INFO:teuthology.orchestra.run.smithi093.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-01-15T18:00:46.599 INFO:teuthology.orchestra.run.smithi093.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-01-15T18:00:46.599 INFO:teuthology.orchestra.run.smithi093.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-01-15T18:00:46.600 INFO:teuthology.orchestra.run.smithi093.stdout:Suggested packages: 2024-01-15T18:00:46.600 INFO:teuthology.orchestra.run.smithi093.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-01-15T18:00:46.600 INFO:teuthology.orchestra.run.smithi093.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-01-15T18:00:46.600 INFO:teuthology.orchestra.run.smithi093.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-01-15T18:00:46.600 INFO:teuthology.orchestra.run.smithi093.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-01-15T18:00:46.600 INFO:teuthology.orchestra.run.smithi093.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-01-15T18:00:46.600 INFO:teuthology.orchestra.run.smithi093.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-01-15T18:00:46.600 INFO:teuthology.orchestra.run.smithi093.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-01-15T18:00:46.600 INFO:teuthology.orchestra.run.smithi093.stdout:Recommended packages: 2024-01-15T18:00:46.600 INFO:teuthology.orchestra.run.smithi093.stdout: btrfs-tools 2024-01-15T18:00:46.642 INFO:teuthology.orchestra.run.smithi027.stdout:The following NEW packages will be installed: 2024-01-15T18:00:46.642 INFO:teuthology.orchestra.run.smithi027.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-01-15T18:00:46.642 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-01-15T18:00:46.642 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-01-15T18:00:46.642 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-01-15T18:00:46.642 INFO:teuthology.orchestra.run.smithi027.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-01-15T18:00:46.642 INFO:teuthology.orchestra.run.smithi027.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-01-15T18:00:46.643 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-01-15T18:00:46.643 INFO:teuthology.orchestra.run.smithi027.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-01-15T18:00:46.643 INFO:teuthology.orchestra.run.smithi027.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-01-15T18:00:46.643 INFO:teuthology.orchestra.run.smithi027.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-01-15T18:00:46.643 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-01-15T18:00:46.643 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-01-15T18:00:46.643 INFO:teuthology.orchestra.run.smithi027.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-01-15T18:00:46.643 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-01-15T18:00:46.643 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-01-15T18:00:46.643 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-01-15T18:00:46.643 INFO:teuthology.orchestra.run.smithi027.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-01-15T18:00:46.644 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-01-15T18:00:46.644 INFO:teuthology.orchestra.run.smithi027.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-01-15T18:00:46.644 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-01-15T18:00:46.644 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-01-15T18:00:46.644 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-01-15T18:00:46.644 INFO:teuthology.orchestra.run.smithi027.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-01-15T18:00:46.644 INFO:teuthology.orchestra.run.smithi027.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-01-15T18:00:46.644 INFO:teuthology.orchestra.run.smithi027.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-01-15T18:00:46.644 INFO:teuthology.orchestra.run.smithi027.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-01-15T18:00:46.644 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-01-15T18:00:46.644 INFO:teuthology.orchestra.run.smithi027.stdout: socat unzip xmlstarlet zip 2024-01-15T18:00:46.746 INFO:teuthology.orchestra.run.smithi093.stdout:The following NEW packages will be installed: 2024-01-15T18:00:46.746 INFO:teuthology.orchestra.run.smithi093.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-01-15T18:00:46.746 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-01-15T18:00:46.746 INFO:teuthology.orchestra.run.smithi093.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-01-15T18:00:46.746 INFO:teuthology.orchestra.run.smithi093.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-01-15T18:00:46.746 INFO:teuthology.orchestra.run.smithi093.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-01-15T18:00:46.746 INFO:teuthology.orchestra.run.smithi093.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-01-15T18:00:46.746 INFO:teuthology.orchestra.run.smithi093.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-01-15T18:00:46.747 INFO:teuthology.orchestra.run.smithi093.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-01-15T18:00:46.747 INFO:teuthology.orchestra.run.smithi093.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-01-15T18:00:46.747 INFO:teuthology.orchestra.run.smithi093.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-01-15T18:00:46.747 INFO:teuthology.orchestra.run.smithi093.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-01-15T18:00:46.747 INFO:teuthology.orchestra.run.smithi093.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-01-15T18:00:46.747 INFO:teuthology.orchestra.run.smithi093.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-01-15T18:00:46.747 INFO:teuthology.orchestra.run.smithi093.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-01-15T18:00:46.747 INFO:teuthology.orchestra.run.smithi093.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-01-15T18:00:46.747 INFO:teuthology.orchestra.run.smithi093.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-01-15T18:00:46.747 INFO:teuthology.orchestra.run.smithi093.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-01-15T18:00:46.748 INFO:teuthology.orchestra.run.smithi093.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-01-15T18:00:46.748 INFO:teuthology.orchestra.run.smithi093.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-01-15T18:00:46.748 INFO:teuthology.orchestra.run.smithi093.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-01-15T18:00:46.748 INFO:teuthology.orchestra.run.smithi093.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-01-15T18:00:46.748 INFO:teuthology.orchestra.run.smithi093.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-01-15T18:00:46.748 INFO:teuthology.orchestra.run.smithi093.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-01-15T18:00:46.748 INFO:teuthology.orchestra.run.smithi093.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-01-15T18:00:46.748 INFO:teuthology.orchestra.run.smithi093.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-01-15T18:00:46.748 INFO:teuthology.orchestra.run.smithi093.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-01-15T18:00:46.748 INFO:teuthology.orchestra.run.smithi093.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-01-15T18:00:46.748 INFO:teuthology.orchestra.run.smithi093.stdout: socat unzip xmlstarlet zip 2024-01-15T18:00:46.792 INFO:teuthology.orchestra.run.smithi027.stdout:0 upgraded, 124 newly installed, 0 to remove and 269 not upgraded. 2024-01-15T18:00:46.792 INFO:teuthology.orchestra.run.smithi027.stdout:Need to get 187 MB of archives. 2024-01-15T18:00:46.792 INFO:teuthology.orchestra.run.smithi027.stdout:After this operation, 835 MB of additional disk space will be used. 2024-01-15T18:00:46.792 INFO:teuthology.orchestra.run.smithi027.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-7629-g95c90f41-1jammy [3,468 kB] 2024-01-15T18:00:46.838 INFO:teuthology.orchestra.run.smithi027.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-01-15T18:00:46.841 INFO:teuthology.orchestra.run.smithi093.stdout:0 upgraded, 124 newly installed, 0 to remove and 269 not upgraded. 2024-01-15T18:00:46.841 INFO:teuthology.orchestra.run.smithi093.stdout:Need to get 187 MB of archives. 2024-01-15T18:00:46.841 INFO:teuthology.orchestra.run.smithi093.stdout:After this operation, 835 MB of additional disk space will be used. 2024-01-15T18:00:46.841 INFO:teuthology.orchestra.run.smithi093.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-01-15T18:00:46.873 INFO:teuthology.orchestra.run.smithi093.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-01-15T18:00:46.892 INFO:teuthology.orchestra.run.smithi093.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-01-15T18:00:46.921 INFO:teuthology.orchestra.run.smithi027.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-01-15T18:00:46.967 INFO:teuthology.orchestra.run.smithi027.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-01-15T18:00:46.976 INFO:teuthology.orchestra.run.smithi093.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-01-15T18:00:47.023 INFO:teuthology.orchestra.run.smithi027.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-7629-g95c90f41-1jammy [3,112 kB] 2024-01-15T18:00:47.023 INFO:teuthology.orchestra.run.smithi093.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-01-15T18:00:47.028 INFO:teuthology.orchestra.run.smithi093.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-01-15T18:00:47.048 INFO:teuthology.orchestra.run.smithi093.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-01-15T18:00:47.088 INFO:teuthology.orchestra.run.smithi093.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-7629-g95c90f41-1jammy [3,468 kB] 2024-01-15T18:00:47.123 INFO:teuthology.orchestra.run.smithi027.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-7629-g95c90f41-1jammy [690 kB] 2024-01-15T18:00:47.135 INFO:teuthology.orchestra.run.smithi093.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-01-15T18:00:47.138 INFO:teuthology.orchestra.run.smithi093.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-01-15T18:00:47.147 INFO:teuthology.orchestra.run.smithi027.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-7629-g95c90f41-1jammy [356 kB] 2024-01-15T18:00:47.153 INFO:teuthology.orchestra.run.smithi093.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-01-15T18:00:47.159 INFO:teuthology.orchestra.run.smithi027.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-7629-g95c90f41-1jammy [32.8 kB] 2024-01-15T18:00:47.160 INFO:teuthology.orchestra.run.smithi027.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-7629-g95c90f41-1jammy [180 kB] 2024-01-15T18:00:47.162 INFO:teuthology.orchestra.run.smithi093.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-01-15T18:00:47.162 INFO:teuthology.orchestra.run.smithi093.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-01-15T18:00:47.167 INFO:teuthology.orchestra.run.smithi027.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-7629-g95c90f41-1jammy [63.5 kB] 2024-01-15T18:00:47.168 INFO:teuthology.orchestra.run.smithi027.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-7629-g95c90f41-1jammy [330 kB] 2024-01-15T18:00:47.170 INFO:teuthology.orchestra.run.smithi093.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-01-15T18:00:47.172 INFO:teuthology.orchestra.run.smithi027.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-01-15T18:00:47.177 INFO:teuthology.orchestra.run.smithi093.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-01-15T18:00:47.180 INFO:teuthology.orchestra.run.smithi093.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-01-15T18:00:47.183 INFO:teuthology.orchestra.run.smithi093.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-01-15T18:00:47.183 INFO:teuthology.orchestra.run.smithi027.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-7629-g95c90f41-1jammy [6,223 kB] 2024-01-15T18:00:47.184 INFO:teuthology.orchestra.run.smithi093.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-01-15T18:00:47.210 INFO:teuthology.orchestra.run.smithi093.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB] 2024-01-15T18:00:47.212 INFO:teuthology.orchestra.run.smithi093.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-01-15T18:00:47.216 INFO:teuthology.orchestra.run.smithi093.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-01-15T18:00:47.218 INFO:teuthology.orchestra.run.smithi093.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-01-15T18:00:47.218 INFO:teuthology.orchestra.run.smithi093.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-01-15T18:00:47.221 INFO:teuthology.orchestra.run.smithi093.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-01-15T18:00:47.221 INFO:teuthology.orchestra.run.smithi093.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-01-15T18:00:47.222 INFO:teuthology.orchestra.run.smithi093.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-01-15T18:00:47.222 INFO:teuthology.orchestra.run.smithi093.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-01-15T18:00:47.241 INFO:teuthology.orchestra.run.smithi093.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-01-15T18:00:47.241 INFO:teuthology.orchestra.run.smithi093.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-01-15T18:00:47.267 INFO:teuthology.orchestra.run.smithi093.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-01-15T18:00:47.269 INFO:teuthology.orchestra.run.smithi093.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-01-15T18:00:47.270 INFO:teuthology.orchestra.run.smithi093.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-01-15T18:00:47.271 INFO:teuthology.orchestra.run.smithi093.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-01-15T18:00:47.271 INFO:teuthology.orchestra.run.smithi093.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-01-15T18:00:47.272 INFO:teuthology.orchestra.run.smithi093.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-01-15T18:00:47.272 INFO:teuthology.orchestra.run.smithi093.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-01-15T18:00:47.272 INFO:teuthology.orchestra.run.smithi093.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-01-15T18:00:47.273 INFO:teuthology.orchestra.run.smithi027.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-01-15T18:00:47.284 INFO:teuthology.orchestra.run.smithi027.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-01-15T18:00:47.294 INFO:teuthology.orchestra.run.smithi093.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-01-15T18:00:47.294 INFO:teuthology.orchestra.run.smithi093.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-01-15T18:00:47.320 INFO:teuthology.orchestra.run.smithi093.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-01-15T18:00:47.328 INFO:teuthology.orchestra.run.smithi093.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-01-15T18:00:47.328 INFO:teuthology.orchestra.run.smithi093.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-01-15T18:00:47.329 INFO:teuthology.orchestra.run.smithi093.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-01-15T18:00:47.329 INFO:teuthology.orchestra.run.smithi093.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-01-15T18:00:47.330 INFO:teuthology.orchestra.run.smithi093.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-01-15T18:00:47.332 INFO:teuthology.orchestra.run.smithi093.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-01-15T18:00:47.333 INFO:teuthology.orchestra.run.smithi093.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-01-15T18:00:47.338 INFO:teuthology.orchestra.run.smithi027.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-01-15T18:00:47.350 INFO:teuthology.orchestra.run.smithi093.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-01-15T18:00:47.350 INFO:teuthology.orchestra.run.smithi093.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-01-15T18:00:47.377 INFO:teuthology.orchestra.run.smithi093.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-01-15T18:00:47.377 INFO:teuthology.orchestra.run.smithi093.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-01-15T18:00:47.378 INFO:teuthology.orchestra.run.smithi093.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-01-15T18:00:47.381 INFO:teuthology.orchestra.run.smithi093.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-01-15T18:00:47.407 INFO:teuthology.orchestra.run.smithi027.stdout:Get:17 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-7629-g95c90f41-1jammy [112 kB] 2024-01-15T18:00:47.408 INFO:teuthology.orchestra.run.smithi027.stdout:Get:18 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-7629-g95c90f41-1jammy [439 kB] 2024-01-15T18:00:47.423 INFO:teuthology.orchestra.run.smithi027.stdout:Get:19 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-7629-g95c90f41-1jammy [22.6 MB] 2024-01-15T18:00:47.461 INFO:teuthology.orchestra.run.smithi093.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-01-15T18:00:47.462 INFO:teuthology.orchestra.run.smithi093.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2024-01-15T18:00:47.469 INFO:teuthology.orchestra.run.smithi093.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-01-15T18:00:47.469 INFO:teuthology.orchestra.run.smithi093.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-01-15T18:00:47.470 INFO:teuthology.orchestra.run.smithi093.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-01-15T18:00:47.505 INFO:teuthology.orchestra.run.smithi093.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-01-15T18:00:47.507 INFO:teuthology.orchestra.run.smithi093.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-01-15T18:00:47.508 INFO:teuthology.orchestra.run.smithi093.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-01-15T18:00:47.512 INFO:teuthology.orchestra.run.smithi027.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-01-15T18:00:47.522 INFO:teuthology.orchestra.run.smithi027.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-01-15T18:00:47.549 INFO:teuthology.orchestra.run.smithi027.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-01-15T18:00:47.558 INFO:teuthology.orchestra.run.smithi027.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-01-15T18:00:47.566 INFO:teuthology.orchestra.run.smithi093.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-01-15T18:00:47.566 INFO:teuthology.orchestra.run.smithi093.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-01-15T18:00:47.567 INFO:teuthology.orchestra.run.smithi093.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-01-15T18:00:47.567 INFO:teuthology.orchestra.run.smithi093.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-01-15T18:00:47.568 INFO:teuthology.orchestra.run.smithi093.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-01-15T18:00:47.569 INFO:teuthology.orchestra.run.smithi093.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-01-15T18:00:47.569 INFO:teuthology.orchestra.run.smithi093.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-01-15T18:00:47.570 INFO:teuthology.orchestra.run.smithi093.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-01-15T18:00:47.576 INFO:teuthology.orchestra.run.smithi093.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-01-15T18:00:47.597 INFO:teuthology.orchestra.run.smithi093.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-01-15T18:00:47.599 INFO:teuthology.orchestra.run.smithi093.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-01-15T18:00:47.599 INFO:teuthology.orchestra.run.smithi093.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-01-15T18:00:47.603 INFO:teuthology.orchestra.run.smithi093.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-01-15T18:00:47.607 INFO:teuthology.orchestra.run.smithi093.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-01-15T18:00:47.627 INFO:teuthology.orchestra.run.smithi093.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-01-15T18:00:47.628 INFO:teuthology.orchestra.run.smithi093.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-01-15T18:00:47.628 INFO:teuthology.orchestra.run.smithi093.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-01-15T18:00:47.637 INFO:teuthology.orchestra.run.smithi093.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-01-15T18:00:47.638 INFO:teuthology.orchestra.run.smithi093.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-01-15T18:00:47.658 INFO:teuthology.orchestra.run.smithi093.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-01-15T18:00:47.659 INFO:teuthology.orchestra.run.smithi093.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-01-15T18:00:47.659 INFO:teuthology.orchestra.run.smithi093.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-01-15T18:00:47.660 INFO:teuthology.orchestra.run.smithi093.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-01-15T18:00:47.661 INFO:teuthology.orchestra.run.smithi093.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-01-15T18:00:47.687 INFO:teuthology.orchestra.run.smithi093.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-01-15T18:00:47.697 INFO:teuthology.orchestra.run.smithi093.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-01-15T18:00:47.697 INFO:teuthology.orchestra.run.smithi093.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-01-15T18:00:47.697 INFO:teuthology.orchestra.run.smithi093.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-01-15T18:00:47.700 INFO:teuthology.orchestra.run.smithi093.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-01-15T18:00:47.715 INFO:teuthology.orchestra.run.smithi093.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-01-15T18:00:47.721 INFO:teuthology.orchestra.run.smithi027.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-01-15T18:00:48.260 INFO:teuthology.orchestra.run.smithi027.stdout:Get:25 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-7629-g95c90f41-1jammy [6,900 kB] 2024-01-15T18:00:48.303 INFO:teuthology.orchestra.run.smithi027.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-01-15T18:00:48.319 INFO:teuthology.orchestra.run.smithi027.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-01-15T18:00:48.334 INFO:teuthology.orchestra.run.smithi027.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-01-15T18:00:48.344 INFO:teuthology.orchestra.run.smithi027.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-01-15T18:00:48.344 INFO:teuthology.orchestra.run.smithi027.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-01-15T18:00:48.360 INFO:teuthology.orchestra.run.smithi027.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB] 2024-01-15T18:00:48.397 INFO:teuthology.orchestra.run.smithi027.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-01-15T18:00:48.410 INFO:teuthology.orchestra.run.smithi027.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-01-15T18:00:48.415 INFO:teuthology.orchestra.run.smithi027.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-01-15T18:00:48.415 INFO:teuthology.orchestra.run.smithi027.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-01-15T18:00:48.514 INFO:teuthology.orchestra.run.smithi027.stdout:Get:36 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-7629-g95c90f41-1jammy [242 kB] 2024-01-15T18:00:48.523 INFO:teuthology.orchestra.run.smithi027.stdout:Get:37 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-7629-g95c90f41-1jammy [124 kB] 2024-01-15T18:00:48.526 INFO:teuthology.orchestra.run.smithi027.stdout:Get:38 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-7629-g95c90f41-1jammy [1,496 kB] 2024-01-15T18:00:48.583 INFO:teuthology.orchestra.run.smithi027.stdout:Get:39 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-7629-g95c90f41-1jammy [9,955 kB] 2024-01-15T18:00:48.584 INFO:teuthology.orchestra.run.smithi027.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-01-15T18:00:48.667 INFO:teuthology.orchestra.run.smithi027.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-01-15T18:00:48.690 INFO:teuthology.orchestra.run.smithi027.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-01-15T18:00:48.748 INFO:teuthology.orchestra.run.smithi027.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-01-15T18:00:48.795 INFO:teuthology.orchestra.run.smithi027.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-01-15T18:00:48.817 INFO:teuthology.orchestra.run.smithi027.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-01-15T18:00:48.839 INFO:teuthology.orchestra.run.smithi027.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-01-15T18:00:48.967 INFO:teuthology.orchestra.run.smithi027.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-7629-g95c90f41-1jammy [31.8 MB] 2024-01-15T18:00:49.012 INFO:teuthology.orchestra.run.smithi027.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-01-15T18:00:49.028 INFO:teuthology.orchestra.run.smithi027.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-01-15T18:00:49.035 INFO:teuthology.orchestra.run.smithi027.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-01-15T18:00:49.040 INFO:teuthology.orchestra.run.smithi027.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-01-15T18:00:49.237 INFO:teuthology.orchestra.run.smithi027.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-01-15T18:00:49.286 INFO:teuthology.orchestra.run.smithi093.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-7629-g95c90f41-1jammy [3,112 kB] 2024-01-15T18:00:49.319 INFO:teuthology.orchestra.run.smithi027.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-01-15T18:00:49.335 INFO:teuthology.orchestra.run.smithi027.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-01-15T18:00:49.520 INFO:teuthology.orchestra.run.smithi027.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-01-15T18:00:49.571 INFO:teuthology.orchestra.run.smithi027.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-01-15T18:00:49.584 INFO:teuthology.orchestra.run.smithi027.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-01-15T18:00:49.742 INFO:teuthology.orchestra.run.smithi027.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-01-15T18:00:49.743 INFO:teuthology.orchestra.run.smithi027.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-01-15T18:00:49.747 INFO:teuthology.orchestra.run.smithi027.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-01-15T18:00:49.753 INFO:teuthology.orchestra.run.smithi027.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-01-15T18:00:49.767 INFO:teuthology.orchestra.run.smithi027.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-01-15T18:00:49.951 INFO:teuthology.orchestra.run.smithi027.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-01-15T18:00:50.168 INFO:teuthology.orchestra.run.smithi093.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-7629-g95c90f41-1jammy [690 kB] 2024-01-15T18:00:50.206 INFO:teuthology.orchestra.run.smithi027.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-01-15T18:00:50.225 INFO:teuthology.orchestra.run.smithi027.stdout:Get:65 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-7629-g95c90f41-1jammy [14.1 kB] 2024-01-15T18:00:50.226 INFO:teuthology.orchestra.run.smithi027.stdout:Get:66 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-7629-g95c90f41-1jammy [839 kB] 2024-01-15T18:00:50.249 INFO:teuthology.orchestra.run.smithi027.stdout:Get:67 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-7629-g95c90f41-1jammy [2,144 kB] 2024-01-15T18:00:50.301 INFO:teuthology.orchestra.run.smithi093.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-7629-g95c90f41-1jammy [356 kB] 2024-01-15T18:00:50.338 INFO:teuthology.orchestra.run.smithi027.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-01-15T18:00:50.350 INFO:teuthology.orchestra.run.smithi093.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-7629-g95c90f41-1jammy [32.8 kB] 2024-01-15T18:00:50.355 INFO:teuthology.orchestra.run.smithi093.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-7629-g95c90f41-1jammy [180 kB] 2024-01-15T18:00:50.374 INFO:teuthology.orchestra.run.smithi093.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-7629-g95c90f41-1jammy [63.5 kB] 2024-01-15T18:00:50.379 INFO:teuthology.orchestra.run.smithi093.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-7629-g95c90f41-1jammy [330 kB] 2024-01-15T18:00:50.382 INFO:teuthology.orchestra.run.smithi027.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-01-15T18:00:50.421 INFO:teuthology.orchestra.run.smithi093.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-7629-g95c90f41-1jammy [6,223 kB] 2024-01-15T18:00:50.472 INFO:teuthology.orchestra.run.smithi027.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-01-15T18:00:50.482 INFO:teuthology.orchestra.run.smithi027.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-01-15T18:00:50.494 INFO:teuthology.orchestra.run.smithi027.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-01-15T18:00:50.537 INFO:teuthology.orchestra.run.smithi027.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-01-15T18:00:50.567 INFO:teuthology.orchestra.run.smithi027.stdout:Get:74 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-7629-g95c90f41-1jammy [805 kB] 2024-01-15T18:00:50.596 INFO:teuthology.orchestra.run.smithi027.stdout:Get:75 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-7629-g95c90f41-1jammy [139 kB] 2024-01-15T18:00:50.604 INFO:teuthology.orchestra.run.smithi027.stdout:Get:76 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-7629-g95c90f41-1jammy [1,900 kB] 2024-01-15T18:00:50.746 INFO:teuthology.orchestra.run.smithi027.stdout:Get:77 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-7629-g95c90f41-1jammy [8,625 kB] 2024-01-15T18:00:50.801 INFO:teuthology.orchestra.run.smithi027.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-01-15T18:00:50.802 INFO:teuthology.orchestra.run.smithi027.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2024-01-15T18:00:50.804 INFO:teuthology.orchestra.run.smithi027.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-01-15T18:00:50.968 INFO:teuthology.orchestra.run.smithi027.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-01-15T18:00:51.069 INFO:teuthology.orchestra.run.smithi027.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-7629-g95c90f41-1jammy [14.2 kB] 2024-01-15T18:00:51.069 INFO:teuthology.orchestra.run.smithi027.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-7629-g95c90f41-1jammy [48.6 MB] 2024-01-15T18:00:51.312 INFO:teuthology.orchestra.run.smithi027.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-01-15T18:00:51.450 INFO:teuthology.orchestra.run.smithi093.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-7629-g95c90f41-1jammy [112 kB] 2024-01-15T18:00:51.459 INFO:teuthology.orchestra.run.smithi093.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-7629-g95c90f41-1jammy [439 kB] 2024-01-15T18:00:51.537 INFO:teuthology.orchestra.run.smithi093.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-7629-g95c90f41-1jammy [22.6 MB] 2024-01-15T18:00:51.715 INFO:teuthology.orchestra.run.smithi027.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-01-15T18:00:51.724 INFO:teuthology.orchestra.run.smithi027.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-01-15T18:00:51.725 INFO:teuthology.orchestra.run.smithi027.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-01-15T18:00:51.793 INFO:teuthology.orchestra.run.smithi027.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-01-15T18:00:51.793 INFO:teuthology.orchestra.run.smithi027.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-01-15T18:00:51.795 INFO:teuthology.orchestra.run.smithi027.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-01-15T18:00:51.797 INFO:teuthology.orchestra.run.smithi027.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-01-15T18:00:51.797 INFO:teuthology.orchestra.run.smithi027.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-01-15T18:00:51.799 INFO:teuthology.orchestra.run.smithi027.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-01-15T18:00:51.963 INFO:teuthology.orchestra.run.smithi027.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-01-15T18:00:52.178 INFO:teuthology.orchestra.run.smithi027.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-01-15T18:00:52.507 INFO:teuthology.orchestra.run.smithi027.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-01-15T18:00:52.554 INFO:teuthology.orchestra.run.smithi027.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-01-15T18:00:52.581 INFO:teuthology.orchestra.run.smithi027.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-01-15T18:00:52.591 INFO:teuthology.orchestra.run.smithi027.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-01-15T18:00:52.637 INFO:teuthology.orchestra.run.smithi027.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-01-15T18:00:52.663 INFO:teuthology.orchestra.run.smithi027.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-01-15T18:00:52.665 INFO:teuthology.orchestra.run.smithi027.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-01-15T18:00:52.671 INFO:teuthology.orchestra.run.smithi027.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-01-15T18:00:52.674 INFO:teuthology.orchestra.run.smithi027.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-01-15T18:00:52.871 INFO:teuthology.orchestra.run.smithi027.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-01-15T18:00:53.133 INFO:teuthology.orchestra.run.smithi027.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-01-15T18:00:53.307 INFO:teuthology.orchestra.run.smithi027.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-7629-g95c90f41-1jammy [121 kB] 2024-01-15T18:00:53.308 INFO:teuthology.orchestra.run.smithi027.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.0.0-7629-g95c90f41-1jammy [226 kB] 2024-01-15T18:00:53.310 INFO:teuthology.orchestra.run.smithi027.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.0.0-7629-g95c90f41-1jammy [34.0 kB] 2024-01-15T18:00:53.310 INFO:teuthology.orchestra.run.smithi027.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.0.0-7629-g95c90f41-1jammy [23.4 kB] 2024-01-15T18:00:53.311 INFO:teuthology.orchestra.run.smithi027.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-7629-g95c90f41-1jammy [38.3 kB] 2024-01-15T18:00:53.311 INFO:teuthology.orchestra.run.smithi027.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-7629-g95c90f41-1jammy [12.3 MB] 2024-01-15T18:00:53.395 INFO:teuthology.orchestra.run.smithi027.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-01-15T18:00:53.397 INFO:teuthology.orchestra.run.smithi027.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-01-15T18:00:53.412 INFO:teuthology.orchestra.run.smithi027.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-01-15T18:00:53.418 INFO:teuthology.orchestra.run.smithi027.stdout:Get:116 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-01-15T18:00:53.455 INFO:teuthology.orchestra.run.smithi027.stdout:Get:117 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-01-15T18:00:53.470 INFO:teuthology.orchestra.run.smithi027.stdout:Get:118 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-01-15T18:00:53.560 INFO:teuthology.orchestra.run.smithi027.stdout:Get:119 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-01-15T18:00:53.561 INFO:teuthology.orchestra.run.smithi027.stdout:Get:120 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-01-15T18:00:53.563 INFO:teuthology.orchestra.run.smithi027.stdout:Get:121 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-01-15T18:00:53.742 INFO:teuthology.orchestra.run.smithi027.stdout:Get:122 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-01-15T18:00:53.850 INFO:teuthology.orchestra.run.smithi027.stdout:Get:123 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-7629-g95c90f41-1jammy [91.4 kB] 2024-01-15T18:00:54.015 INFO:teuthology.orchestra.run.smithi027.stdout:Get:124 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-01-15T18:00:55.033 INFO:teuthology.orchestra.run.smithi093.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-7629-g95c90f41-1jammy [6,900 kB] 2024-01-15T18:00:55.122 INFO:teuthology.orchestra.run.smithi027.stdout:Fetched 187 MB in 8s (23.9 MB/s) 2024-01-15T18:00:55.271 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-pyperclip. 2024-01-15T18:00:55.343 INFO:teuthology.orchestra.run.smithi027.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132930 files and directories currently installed.) 2024-01-15T18:00:55.346 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-01-15T18:00:55.386 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-01-15T18:00:56.039 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-wcwidth. 2024-01-15T18:00:56.054 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-01-15T18:00:56.097 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-01-15T18:00:56.331 INFO:teuthology.orchestra.run.smithi093.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-7629-g95c90f41-1jammy [242 kB] 2024-01-15T18:00:56.402 INFO:teuthology.orchestra.run.smithi093.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-7629-g95c90f41-1jammy [124 kB] 2024-01-15T18:00:56.413 INFO:teuthology.orchestra.run.smithi093.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-7629-g95c90f41-1jammy [1,496 kB] 2024-01-15T18:00:56.458 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-cmd2. 2024-01-15T18:00:56.473 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-01-15T18:00:56.525 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-01-15T18:00:56.945 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-01-15T18:00:56.960 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-01-15T18:00:57.017 INFO:teuthology.orchestra.run.smithi093.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-7629-g95c90f41-1jammy [9,955 kB] 2024-01-15T18:00:57.061 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-01-15T18:00:57.498 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-01-15T18:00:57.514 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-01-15T18:00:57.556 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-01-15T18:00:57.901 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-01-15T18:00:57.917 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-01-15T18:00:57.951 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-01-15T18:00:58.388 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-01-15T18:00:58.403 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-01-15T18:00:58.454 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-01-15T18:00:58.841 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-01-15T18:00:58.856 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-01-15T18:00:58.899 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-01-15T18:00:59.227 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-01-15T18:00:59.242 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-01-15T18:00:59.277 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-01-15T18:00:59.605 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-01-15T18:00:59.621 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-01-15T18:00:59.659 INFO:teuthology.orchestra.run.smithi093.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-7629-g95c90f41-1jammy [31.8 MB] 2024-01-15T18:00:59.663 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-01-15T18:01:00.008 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package librados2. 2024-01-15T18:01:00.023 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../010-librados2_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:00.065 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking librados2 (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:00.493 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package librbd1. 2024-01-15T18:01:00.508 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../011-librbd1_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:00.576 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking librbd1 (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:00.988 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libcephfs2. 2024-01-15T18:01:01.003 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../012-libcephfs2_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:01.038 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libcephfs2 (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:01.291 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-rados. 2024-01-15T18:01:01.306 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../013-python3-rados_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:01.348 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-rados (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:01.668 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-01-15T18:01:01.684 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../014-python3-ceph-argparse_18.0.0-7629-g95c90f41-1jammy_all.deb ... 2024-01-15T18:01:01.751 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-ceph-argparse (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:02.004 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-cephfs. 2024-01-15T18:01:02.019 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../015-python3-cephfs_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:02.061 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-cephfs (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:02.340 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-ceph-common. 2024-01-15T18:01:02.355 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../016-python3-ceph-common_18.0.0-7629-g95c90f41-1jammy_all.deb ... 2024-01-15T18:01:02.389 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-ceph-common (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:02.667 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-prettytable. 2024-01-15T18:01:02.683 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-01-15T18:01:02.725 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-01-15T18:01:03.028 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-rbd. 2024-01-15T18:01:03.043 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../018-python3-rbd_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:03.086 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-rbd (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:03.439 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-01-15T18:01:03.455 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-01-15T18:01:03.498 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-01-15T18:01:03.750 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-01-15T18:01:03.765 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-01-15T18:01:03.800 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-01-15T18:01:04.069 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-01-15T18:01:04.084 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-01-15T18:01:04.119 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-01-15T18:01:04.464 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package lua5.1. 2024-01-15T18:01:04.480 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-01-15T18:01:04.522 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-01-15T18:01:04.834 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package lua-any. 2024-01-15T18:01:04.849 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-01-15T18:01:04.892 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking lua-any (27ubuntu1) ... 2024-01-15T18:01:05.145 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package zip. 2024-01-15T18:01:05.160 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-01-15T18:01:05.194 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking zip (3.0-12build2) ... 2024-01-15T18:01:05.531 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package unzip. 2024-01-15T18:01:05.547 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.1_amd64.deb ... 2024-01-15T18:01:05.589 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking unzip (6.0-26ubuntu3.1) ... 2024-01-15T18:01:05.976 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package luarocks. 2024-01-15T18:01:05.992 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-01-15T18:01:06.034 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-01-15T18:01:06.484 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package librgw2. 2024-01-15T18:01:06.499 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../027-librgw2_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:06.542 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking librgw2 (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:06.986 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-rgw. 2024-01-15T18:01:07.002 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../028-python3-rgw_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:07.044 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-rgw (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:07.406 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package liboath0:amd64. 2024-01-15T18:01:07.422 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-01-15T18:01:07.464 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-01-15T18:01:07.725 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libradosstriper1. 2024-01-15T18:01:07.741 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../030-libradosstriper1_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:07.783 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libradosstriper1 (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:08.203 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-common. 2024-01-15T18:01:08.219 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../031-ceph-common_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:08.261 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-common (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:08.445 INFO:teuthology.orchestra.run.smithi093.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-7629-g95c90f41-1jammy [14.1 kB] 2024-01-15T18:01:08.455 INFO:teuthology.orchestra.run.smithi093.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-7629-g95c90f41-1jammy [839 kB] 2024-01-15T18:01:08.728 INFO:teuthology.orchestra.run.smithi093.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-7629-g95c90f41-1jammy [2,144 kB] 2024-01-15T18:01:09.309 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-base. 2024-01-15T18:01:09.324 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../032-ceph-base_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:09.376 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-base (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:09.457 INFO:teuthology.orchestra.run.smithi093.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-7629-g95c90f41-1jammy [805 kB] 2024-01-15T18:01:09.751 INFO:teuthology.orchestra.run.smithi093.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-7629-g95c90f41-1jammy [139 kB] 2024-01-15T18:01:09.801 INFO:teuthology.orchestra.run.smithi093.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-7629-g95c90f41-1jammy [1,900 kB] 2024-01-15T18:01:10.055 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-01-15T18:01:10.069 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-01-15T18:01:10.104 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-01-15T18:01:10.343 INFO:teuthology.orchestra.run.smithi093.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-7629-g95c90f41-1jammy [8,625 kB] 2024-01-15T18:01:10.374 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-cheroot. 2024-01-15T18:01:10.388 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-01-15T18:01:10.432 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-01-15T18:01:10.752 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-01-15T18:01:10.766 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-01-15T18:01:10.835 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-01-15T18:01:11.388 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-jaraco.text. 2024-01-15T18:01:11.403 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-01-15T18:01:11.446 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-01-15T18:01:11.724 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-01-15T18:01:11.740 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-01-15T18:01:11.774 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-01-15T18:01:12.060 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-tempora. 2024-01-15T18:01:12.076 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-01-15T18:01:12.118 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-01-15T18:01:12.404 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-portend. 2024-01-15T18:01:12.420 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-01-15T18:01:12.462 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-01-15T18:01:12.493 INFO:teuthology.orchestra.run.smithi093.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-7629-g95c90f41-1jammy [14.2 kB] 2024-01-15T18:01:12.494 INFO:teuthology.orchestra.run.smithi093.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-7629-g95c90f41-1jammy [48.6 MB] 2024-01-15T18:01:12.749 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-01-15T18:01:12.764 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-01-15T18:01:12.807 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-01-15T18:01:13.085 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-cherrypy3. 2024-01-15T18:01:13.101 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-01-15T18:01:13.134 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-01-15T18:01:13.471 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-natsort. 2024-01-15T18:01:13.487 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-01-15T18:01:13.530 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-01-15T18:01:13.825 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-logutils. 2024-01-15T18:01:13.840 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-01-15T18:01:13.883 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-01-15T18:01:14.177 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-markupsafe. 2024-01-15T18:01:14.193 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-01-15T18:01:14.227 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-01-15T18:01:14.530 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-mako. 2024-01-15T18:01:14.546 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-01-15T18:01:14.588 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-01-15T18:01:14.916 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-simplegeneric. 2024-01-15T18:01:14.932 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-01-15T18:01:14.974 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-01-15T18:01:15.302 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-singledispatch. 2024-01-15T18:01:15.318 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-01-15T18:01:15.360 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-01-15T18:01:15.613 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-webob. 2024-01-15T18:01:15.629 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-01-15T18:01:15.671 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-01-15T18:01:16.008 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-waitress. 2024-01-15T18:01:16.024 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-01-15T18:01:16.069 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-01-15T18:01:16.361 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-tempita. 2024-01-15T18:01:16.376 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-01-15T18:01:16.452 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-01-15T18:01:17.372 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-paste. 2024-01-15T18:01:17.388 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-01-15T18:01:17.430 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-01-15T18:01:17.775 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-01-15T18:01:17.790 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-01-15T18:01:17.833 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-01-15T18:01:18.136 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-pastedeploy. 2024-01-15T18:01:18.152 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-01-15T18:01:18.194 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-01-15T18:01:18.464 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-webtest. 2024-01-15T18:01:18.479 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-01-15T18:01:18.513 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-01-15T18:01:18.842 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-pecan. 2024-01-15T18:01:18.858 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-01-15T18:01:18.900 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-01-15T18:01:19.213 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-certifi. 2024-01-15T18:01:19.228 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-01-15T18:01:19.271 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-01-15T18:01:19.591 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-idna. 2024-01-15T18:01:19.607 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-01-15T18:01:19.649 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-idna (3.3-1) ... 2024-01-15T18:01:19.968 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-urllib3. 2024-01-15T18:01:19.984 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-01-15T18:01:20.026 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-01-15T18:01:20.355 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-requests. 2024-01-15T18:01:20.371 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-01-15T18:01:20.413 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-01-15T18:01:20.724 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-werkzeug. 2024-01-15T18:01:20.740 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-01-15T18:01:20.774 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-01-15T18:01:21.102 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-01-15T18:01:21.118 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_18.0.0-7629-g95c90f41-1jammy_all.deb ... 2024-01-15T18:01:21.294 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-mgr-modules-core (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:21.657 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-01-15T18:01:21.672 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:21.715 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:22.068 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-bcrypt. 2024-01-15T18:01:22.084 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-01-15T18:01:22.125 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-01-15T18:01:22.412 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-openssl. 2024-01-15T18:01:22.428 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-01-15T18:01:22.470 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-01-15T18:01:22.740 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-mgr. 2024-01-15T18:01:22.756 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../065-ceph-mgr_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:22.789 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-mgr (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:23.193 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-mon. 2024-01-15T18:01:23.209 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../066-ceph-mon_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:23.251 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-mon (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:24.122 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-osd. 2024-01-15T18:01:24.137 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../067-ceph-osd_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:24.179 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-osd (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:25.341 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph. 2024-01-15T18:01:25.357 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../068-ceph_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:25.424 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:25.685 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-fuse. 2024-01-15T18:01:25.701 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../069-ceph-fuse_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:25.735 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-fuse (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:26.129 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-mds. 2024-01-15T18:01:26.145 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../070-ceph-mds_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:26.188 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-mds (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:26.574 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package cephadm. 2024-01-15T18:01:26.589 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../071-cephadm_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:26.641 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking cephadm (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:26.952 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-asyncssh. 2024-01-15T18:01:26.967 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-01-15T18:01:27.010 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-01-15T18:01:27.296 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python-babel-localedata. 2024-01-15T18:01:27.311 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-01-15T18:01:27.354 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-01-15T18:01:28.351 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-babel. 2024-01-15T18:01:28.365 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-01-15T18:01:28.408 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-01-15T18:01:28.720 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-jinja2. 2024-01-15T18:01:28.734 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1_all.deb ... 2024-01-15T18:01:28.803 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2024-01-15T18:01:29.148 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-01-15T18:01:29.162 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_18.0.0-7629-g95c90f41-1jammy_all.deb ... 2024-01-15T18:01:29.214 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-mgr-cephadm (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:29.317 INFO:teuthology.orchestra.run.smithi093.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-7629-g95c90f41-1jammy [121 kB] 2024-01-15T18:01:29.335 INFO:teuthology.orchestra.run.smithi093.stdout:Get:119 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.0.0-7629-g95c90f41-1jammy [226 kB] 2024-01-15T18:01:29.396 INFO:teuthology.orchestra.run.smithi093.stdout:Get:120 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.0.0-7629-g95c90f41-1jammy [34.0 kB] 2024-01-15T18:01:29.397 INFO:teuthology.orchestra.run.smithi093.stdout:Get:121 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.0.0-7629-g95c90f41-1jammy [23.4 kB] 2024-01-15T18:01:29.397 INFO:teuthology.orchestra.run.smithi093.stdout:Get:122 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-7629-g95c90f41-1jammy [38.3 kB] 2024-01-15T18:01:29.414 INFO:teuthology.orchestra.run.smithi093.stdout:Get:123 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-7629-g95c90f41-1jammy [12.3 MB] 2024-01-15T18:01:29.528 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-repoze.lru. 2024-01-15T18:01:29.543 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-01-15T18:01:29.578 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-01-15T18:01:29.847 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-routes. 2024-01-15T18:01:29.862 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-01-15T18:01:29.929 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-01-15T18:01:30.209 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-01-15T18:01:30.224 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_18.0.0-7629-g95c90f41-1jammy_all.deb ... 2024-01-15T18:01:30.258 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-mgr-dashboard (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:30.687 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-01-15T18:01:30.702 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-01-15T18:01:30.745 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-01-15T18:01:31.232 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-joblib. 2024-01-15T18:01:31.247 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-01-15T18:01:31.282 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-01-15T18:01:31.644 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-01-15T18:01:31.659 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-01-15T18:01:31.779 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-01-15T18:01:32.049 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-sklearn. 2024-01-15T18:01:32.065 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-01-15T18:01:32.098 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-01-15T18:01:32.414 INFO:teuthology.orchestra.run.smithi093.stdout:Get:124 https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-7629-g95c90f41-1jammy [91.4 kB] 2024-01-15T18:01:32.808 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-01-15T18:01:32.823 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_18.0.0-7629-g95c90f41-1jammy_all.deb ... 2024-01-15T18:01:32.875 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:33.074 INFO:teuthology.orchestra.run.smithi093.stdout:Fetched 187 MB in 46s (4,090 kB/s) 2024-01-15T18:01:33.200 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-pyperclip. 2024-01-15T18:01:33.267 INFO:teuthology.orchestra.run.smithi093.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132930 files and directories currently installed.) 2024-01-15T18:01:33.271 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-01-15T18:01:33.307 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-01-15T18:01:33.577 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-wcwidth. 2024-01-15T18:01:33.590 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-01-15T18:01:33.635 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-01-15T18:01:33.937 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-cachetools. 2024-01-15T18:01:33.938 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-cmd2. 2024-01-15T18:01:33.951 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-01-15T18:01:33.953 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-01-15T18:01:33.995 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-01-15T18:01:34.003 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-01-15T18:01:34.357 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-01-15T18:01:34.370 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-01-15T18:01:34.373 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-pyasn1. 2024-01-15T18:01:34.389 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-01-15T18:01:34.439 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-01-15T18:01:34.457 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-01-15T18:01:34.810 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-01-15T18:01:34.818 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-01-15T18:01:34.824 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-01-15T18:01:34.834 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-01-15T18:01:34.868 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-01-15T18:01:34.876 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-01-15T18:01:35.179 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-01-15T18:01:35.193 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-01-15T18:01:35.196 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-rsa. 2024-01-15T18:01:35.212 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-01-15T18:01:35.254 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-rsa (4.8-1) ... 2024-01-15T18:01:35.279 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-01-15T18:01:35.583 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-google-auth. 2024-01-15T18:01:35.599 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-01-15T18:01:35.654 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-01-15T18:01:35.716 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-01-15T18:01:35.731 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-01-15T18:01:35.782 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-01-15T18:01:35.919 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-01-15T18:01:35.936 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-01-15T18:01:35.977 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-01-15T18:01:36.203 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-01-15T18:01:36.217 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-01-15T18:01:36.252 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-01-15T18:01:36.272 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-websocket. 2024-01-15T18:01:36.288 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-01-15T18:01:36.330 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-01-15T18:01:36.597 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-01-15T18:01:36.611 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-01-15T18:01:36.655 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-01-15T18:01:36.900 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-kubernetes. 2024-01-15T18:01:36.916 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-01-15T18:01:36.959 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-01-15T18:01:36.999 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-01-15T18:01:37.014 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-01-15T18:01:37.057 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-01-15T18:01:37.410 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package librados2. 2024-01-15T18:01:37.425 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../010-librados2_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:37.468 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking librados2 (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:37.639 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-01-15T18:01:37.654 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_18.0.0-7629-g95c90f41-1jammy_all.deb ... 2024-01-15T18:01:37.697 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:37.971 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package librbd1. 2024-01-15T18:01:37.985 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../011-librbd1_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:38.021 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking librbd1 (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:38.125 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libonig5:amd64. 2024-01-15T18:01:38.150 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-01-15T18:01:38.175 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-01-15T18:01:38.415 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libcephfs2. 2024-01-15T18:01:38.429 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../012-libcephfs2_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:38.479 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libjq1:amd64. 2024-01-15T18:01:38.494 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-01-15T18:01:38.507 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libcephfs2 (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:38.537 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-01-15T18:01:38.801 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-rados. 2024-01-15T18:01:38.806 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package jq. 2024-01-15T18:01:38.816 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../013-python3-rados_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:38.822 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-01-15T18:01:38.859 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-rados (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:38.864 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-01-15T18:01:39.142 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package socat. 2024-01-15T18:01:39.159 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-01-15T18:01:39.203 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-01-15T18:01:39.218 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../014-python3-ceph-argparse_18.0.0-7629-g95c90f41-1jammy_all.deb ... 2024-01-15T18:01:39.225 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-01-15T18:01:39.262 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-ceph-argparse (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:39.497 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-cephfs. 2024-01-15T18:01:39.512 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../015-python3-cephfs_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:39.520 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package xmlstarlet. 2024-01-15T18:01:39.536 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-01-15T18:01:39.547 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-cephfs (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:39.570 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-01-15T18:01:39.875 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-ceph-common. 2024-01-15T18:01:39.889 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../016-python3-ceph-common_18.0.0-7629-g95c90f41-1jammy_all.deb ... 2024-01-15T18:01:39.899 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-test. 2024-01-15T18:01:39.914 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../099-ceph-test_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:39.933 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-ceph-common (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:39.957 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-test (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:40.244 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-prettytable. 2024-01-15T18:01:40.259 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-01-15T18:01:40.302 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-01-15T18:01:40.588 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-rbd. 2024-01-15T18:01:40.603 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../018-python3-rbd_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:40.646 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-rbd (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:40.982 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-01-15T18:01:40.997 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-01-15T18:01:41.032 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-01-15T18:01:41.293 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-01-15T18:01:41.308 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-01-15T18:01:41.351 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-01-15T18:01:41.662 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-01-15T18:01:41.677 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-01-15T18:01:41.720 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-01-15T18:01:41.762 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ceph-volume. 2024-01-15T18:01:41.778 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../100-ceph-volume_18.0.0-7629-g95c90f41-1jammy_all.deb ... 2024-01-15T18:01:41.820 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ceph-volume (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:42.040 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package lua5.1. 2024-01-15T18:01:42.055 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-01-15T18:01:42.098 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-01-15T18:01:42.157 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package cephfs-mirror. 2024-01-15T18:01:42.174 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../101-cephfs-mirror_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:42.207 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking cephfs-mirror (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:42.443 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package lua-any. 2024-01-15T18:01:42.457 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-01-15T18:01:42.493 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking lua-any (27ubuntu1) ... 2024-01-15T18:01:42.502 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-colorama. 2024-01-15T18:01:42.518 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-01-15T18:01:42.560 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-01-15T18:01:42.712 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package zip. 2024-01-15T18:01:42.727 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-01-15T18:01:42.770 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking zip (3.0-12build2) ... 2024-01-15T18:01:42.846 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package cephfs-shell. 2024-01-15T18:01:42.862 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../103-cephfs-shell_18.0.0-7629-g95c90f41-1jammy_all.deb ... 2024-01-15T18:01:42.904 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking cephfs-shell (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:43.098 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package unzip. 2024-01-15T18:01:43.113 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.1_amd64.deb ... 2024-01-15T18:01:43.156 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking unzip (6.0-26ubuntu3.1) ... 2024-01-15T18:01:43.157 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package cephfs-top. 2024-01-15T18:01:43.173 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../104-cephfs-top_18.0.0-7629-g95c90f41-1jammy_all.deb ... 2024-01-15T18:01:43.215 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking cephfs-top (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:43.459 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libcephfs-dev. 2024-01-15T18:01:43.476 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../105-libcephfs-dev_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:43.517 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libcephfs-dev (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:43.585 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package luarocks. 2024-01-15T18:01:43.600 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-01-15T18:01:43.643 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-01-15T18:01:43.846 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package lua-socket:amd64. 2024-01-15T18:01:43.862 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-01-15T18:01:43.904 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-01-15T18:01:44.066 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package librgw2. 2024-01-15T18:01:44.081 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../027-librgw2_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:44.116 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking librgw2 (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:44.233 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package lua-sec:amd64. 2024-01-15T18:01:44.249 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-01-15T18:01:44.291 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-01-15T18:01:44.552 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-rgw. 2024-01-15T18:01:44.566 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../028-python3-rgw_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:44.610 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-rgw (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:44.654 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package nvme-cli. 2024-01-15T18:01:44.670 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-01-15T18:01:44.712 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-01-15T18:01:44.946 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package liboath0:amd64. 2024-01-15T18:01:44.961 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-01-15T18:01:45.004 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-01-15T18:01:45.100 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package pkg-config. 2024-01-15T18:01:45.116 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-01-15T18:01:45.150 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-01-15T18:01:45.357 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libradosstriper1. 2024-01-15T18:01:45.372 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../030-libradosstriper1_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:45.378 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-01-15T18:01:45.395 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-01-15T18:01:45.415 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libradosstriper1 (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:45.436 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-01-15T18:01:45.827 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package ceph-common. 2024-01-15T18:01:45.840 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-iniconfig. 2024-01-15T18:01:45.842 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../031-ceph-common_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:45.856 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-01-15T18:01:45.876 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking ceph-common (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:45.897 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-01-15T18:01:46.184 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-pastescript. 2024-01-15T18:01:46.200 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-01-15T18:01:46.233 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-01-15T18:01:46.529 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-pluggy. 2024-01-15T18:01:46.545 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-01-15T18:01:46.587 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-01-15T18:01:46.899 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-psutil. 2024-01-15T18:01:46.907 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package ceph-base. 2024-01-15T18:01:46.915 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-01-15T18:01:46.922 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../032-ceph-base_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:46.956 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-01-15T18:01:47.157 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking ceph-base (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:47.301 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-py. 2024-01-15T18:01:47.316 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-01-15T18:01:47.351 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-py (1.10.0-1) ... 2024-01-15T18:01:47.688 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-pygments. 2024-01-15T18:01:47.704 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-01-15T18:01:47.746 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-01-15T18:01:47.844 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-01-15T18:01:47.859 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-01-15T18:01:47.902 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-01-15T18:01:48.188 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-cheroot. 2024-01-15T18:01:48.200 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-pyinotify. 2024-01-15T18:01:48.203 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-01-15T18:01:48.215 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-01-15T18:01:48.246 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-01-15T18:01:48.258 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-01-15T18:01:48.578 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-toml. 2024-01-15T18:01:48.595 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-01-15T18:01:48.599 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-01-15T18:01:48.614 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-01-15T18:01:48.636 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-01-15T18:01:48.657 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-01-15T18:01:48.889 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-pytest. 2024-01-15T18:01:48.905 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-01-15T18:01:48.947 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-01-15T18:01:48.968 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-jaraco.text. 2024-01-15T18:01:48.983 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-01-15T18:01:49.051 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-01-15T18:01:49.276 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package python3-simplejson. 2024-01-15T18:01:49.292 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-01-15T18:01:49.334 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-01-15T18:01:49.346 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-01-15T18:01:49.360 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-01-15T18:01:49.395 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-01-15T18:01:49.628 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-01-15T18:01:49.645 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-01-15T18:01:49.686 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-01-15T18:01:49.706 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-tempora. 2024-01-15T18:01:49.721 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-01-15T18:01:49.765 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-01-15T18:01:50.067 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-portend. 2024-01-15T18:01:50.081 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-01-15T18:01:50.125 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-01-15T18:01:50.391 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package radosgw. 2024-01-15T18:01:50.406 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../122-radosgw_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:50.449 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking radosgw (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:50.462 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-01-15T18:01:50.476 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-01-15T18:01:50.520 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-01-15T18:01:50.814 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-cherrypy3. 2024-01-15T18:01:50.829 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-01-15T18:01:50.864 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-01-15T18:01:50.985 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package rbd-fuse. 2024-01-15T18:01:51.001 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../123-rbd-fuse_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:51.035 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking rbd-fuse (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:51.234 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-natsort. 2024-01-15T18:01:51.248 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-01-15T18:01:51.292 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-01-15T18:01:51.321 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-01-15T18:01:51.615 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-01-15T18:01:51.629 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-logutils. 2024-01-15T18:01:51.643 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-01-15T18:01:51.687 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-01-15T18:01:51.766 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-01-15T18:01:52.048 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-markupsafe. 2024-01-15T18:01:52.063 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-01-15T18:01:52.106 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-01-15T18:01:52.265 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-01-15T18:01:52.434 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-mako. 2024-01-15T18:01:52.449 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-01-15T18:01:52.500 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-01-15T18:01:52.629 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-01-15T18:01:52.912 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-simplegeneric. 2024-01-15T18:01:52.927 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-01-15T18:01:52.978 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-01-15T18:01:53.204 INFO:teuthology.orchestra.run.smithi027.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-01-15T18:01:53.252 INFO:teuthology.orchestra.run.smithi027.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2024-01-15T18:01:53.293 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up cephadm (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:53.372 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-singledispatch. 2024-01-15T18:01:53.387 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-01-15T18:01:53.431 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-01-15T18:01:53.675 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-webob. 2024-01-15T18:01:53.689 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-01-15T18:01:53.733 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-01-15T18:01:53.804 INFO:teuthology.orchestra.run.smithi027.stdout:Adding system user cephadm....done 2024-01-15T18:01:53.982 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-01-15T18:01:54.103 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-waitress. 2024-01-15T18:01:54.117 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-01-15T18:01:54.164 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-01-15T18:01:54.267 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-01-15T18:01:54.472 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-tempita. 2024-01-15T18:01:54.486 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-01-15T18:01:54.530 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-01-15T18:01:54.535 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-01-15T18:01:54.661 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-01-15T18:01:54.875 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-paste. 2024-01-15T18:01:54.889 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-01-15T18:01:54.932 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-01-15T18:01:54.937 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-01-15T18:01:55.197 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-01-15T18:01:55.269 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-01-15T18:01:55.284 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-01-15T18:01:55.322 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-py (1.10.0-1) ... 2024-01-15T18:01:55.335 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-01-15T18:01:55.625 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-01-15T18:01:55.705 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-pastedeploy. 2024-01-15T18:01:55.720 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-01-15T18:01:55.771 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-01-15T18:01:55.993 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-01-15T18:01:56.157 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-webtest. 2024-01-15T18:01:56.172 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-01-15T18:01:56.224 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-01-15T18:01:56.269 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-01-15T18:01:56.546 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-01-15T18:01:56.627 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-pecan. 2024-01-15T18:01:56.642 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-01-15T18:01:56.693 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-01-15T18:01:56.822 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up unzip (6.0-26ubuntu3.1) ... 2024-01-15T18:01:57.056 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-certifi. 2024-01-15T18:01:57.057 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-01-15T18:01:57.071 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-01-15T18:01:57.148 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-01-15T18:01:57.325 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-01-15T18:01:57.475 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-idna. 2024-01-15T18:01:57.490 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-01-15T18:01:57.525 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-idna (3.3-1) ... 2024-01-15T18:01:57.651 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-01-15T18:01:57.853 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-urllib3. 2024-01-15T18:01:57.868 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-01-15T18:01:57.911 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-01-15T18:01:57.918 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-ceph-argparse (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:58.172 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-requests. 2024-01-15T18:01:58.187 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-01-15T18:01:58.203 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-01-15T18:01:58.230 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-01-15T18:01:58.329 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-01-15T18:01:58.480 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-01-15T18:01:58.567 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-werkzeug. 2024-01-15T18:01:58.581 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-01-15T18:01:58.624 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-01-15T18:01:58.748 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-01-15T18:01:58.874 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-01-15T18:01:58.903 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-01-15T18:01:58.918 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_18.0.0-7629-g95c90f41-1jammy_all.deb ... 2024-01-15T18:01:58.952 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking ceph-mgr-modules-core (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:59.141 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-01-15T18:01:59.348 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-01-15T18:01:59.363 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:01:59.407 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:01:59.510 INFO:teuthology.orchestra.run.smithi027.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-01-15T18:01:59.570 INFO:teuthology.orchestra.run.smithi027.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-01-15T18:01:59.636 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-01-15T18:01:59.717 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-bcrypt. 2024-01-15T18:01:59.732 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-01-15T18:01:59.762 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-01-15T18:01:59.767 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-01-15T18:02:00.045 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-openssl. 2024-01-15T18:02:00.060 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-01-15T18:02:00.103 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-01-15T18:02:00.139 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-01-15T18:02:00.264 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-01-15T18:02:00.456 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package ceph-mgr. 2024-01-15T18:02:00.471 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../065-ceph-mgr_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:02:00.514 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking ceph-mgr (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:00.541 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-01-15T18:02:00.809 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-01-15T18:02:00.967 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package ceph-mon. 2024-01-15T18:02:00.982 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../066-ceph-mon_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:02:01.034 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking ceph-mon (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:01.118 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up zip (3.0-12build2) ... 2024-01-15T18:02:01.244 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-01-15T18:02:01.804 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package ceph-osd. 2024-01-15T18:02:01.819 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../067-ceph-osd_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:02:01.862 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking ceph-osd (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:01.971 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-01-15T18:02:02.255 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-01-15T18:02:02.382 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-01-15T18:02:02.507 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-01-15T18:02:02.859 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-01-15T18:02:03.040 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package ceph. 2024-01-15T18:02:03.056 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../068-ceph_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:02:03.098 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking ceph (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:03.302 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-01-15T18:02:03.459 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package ceph-fuse. 2024-01-15T18:02:03.473 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../069-ceph-fuse_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:02:03.517 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking ceph-fuse (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:03.594 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-01-15T18:02:03.837 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package ceph-mds. 2024-01-15T18:02:03.852 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../070-ceph-mds_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:02:03.894 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking ceph-mds (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:04.021 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-01-15T18:02:04.264 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package cephadm. 2024-01-15T18:02:04.279 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../071-cephadm_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:02:04.322 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking cephadm (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:04.356 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-01-15T18:02:04.665 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-asyncssh. 2024-01-15T18:02:04.680 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-01-15T18:02:04.682 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-01-15T18:02:04.724 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-01-15T18:02:05.042 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-01-15T18:02:05.061 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python-babel-localedata. 2024-01-15T18:02:05.076 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-01-15T18:02:05.110 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-01-15T18:02:05.301 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-01-15T18:02:05.427 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-ceph-common (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:05.737 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-idna (3.3-1) ... 2024-01-15T18:02:06.080 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-01-15T18:02:06.123 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-babel. 2024-01-15T18:02:06.137 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-01-15T18:02:06.182 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-01-15T18:02:06.525 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-jinja2. 2024-01-15T18:02:06.539 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1_all.deb ... 2024-01-15T18:02:06.583 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2024-01-15T18:02:06.945 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-01-15T18:02:06.959 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_18.0.0-7629-g95c90f41-1jammy_all.deb ... 2024-01-15T18:02:07.003 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking ceph-mgr-cephadm (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:07.336 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-01-15T18:02:07.364 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-repoze.lru. 2024-01-15T18:02:07.379 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-01-15T18:02:07.422 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-01-15T18:02:07.612 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-01-15T18:02:07.708 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-routes. 2024-01-15T18:02:07.722 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-toml (0.10.2-1) ... 2024-01-15T18:02:07.724 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-01-15T18:02:07.791 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-01-15T18:02:07.981 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-01-15T18:02:08.082 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-01-15T18:02:08.130 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-01-15T18:02:08.145 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_18.0.0-7629-g95c90f41-1jammy_all.deb ... 2024-01-15T18:02:08.183 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-01-15T18:02:08.188 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking ceph-mgr-dashboard (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:08.485 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-01-15T18:02:08.625 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-01-15T18:02:08.639 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-01-15T18:02:08.683 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-01-15T18:02:08.761 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-01-15T18:02:09.029 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-01-15T18:02:09.155 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-01-15T18:02:09.186 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-joblib. 2024-01-15T18:02:09.201 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-01-15T18:02:09.244 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-01-15T18:02:09.481 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-01-15T18:02:09.631 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-01-15T18:02:09.645 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-01-15T18:02:09.766 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-01-15T18:02:09.781 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-01-15T18:02:10.050 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-01-15T18:02:10.069 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-sklearn. 2024-01-15T18:02:10.084 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-01-15T18:02:10.127 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-01-15T18:02:10.309 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-01-15T18:02:10.577 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-01-15T18:02:10.769 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-01-15T18:02:10.784 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_18.0.0-7629-g95c90f41-1jammy_all.deb ... 2024-01-15T18:02:10.845 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:10.853 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-01-15T18:02:11.023 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-01-15T18:02:11.307 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-01-15T18:02:11.433 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-01-15T18:02:11.735 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-01-15T18:02:11.790 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-cachetools. 2024-01-15T18:02:11.805 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-01-15T18:02:11.848 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-01-15T18:02:12.077 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-01-15T18:02:12.175 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-pyasn1. 2024-01-15T18:02:12.191 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-01-15T18:02:12.233 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-01-15T18:02:12.353 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-01-15T18:02:12.454 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-01-15T18:02:12.553 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-01-15T18:02:12.555 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-01-15T18:02:12.569 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-01-15T18:02:12.612 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-01-15T18:02:12.636 INFO:teuthology.orchestra.run.smithi027.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-01-15T18:02:12.891 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-01-15T18:02:12.940 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-rsa. 2024-01-15T18:02:12.955 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-01-15T18:02:12.998 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-rsa (4.8-1) ... 2024-01-15T18:02:13.017 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-01-15T18:02:13.318 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-google-auth. 2024-01-15T18:02:13.334 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-01-15T18:02:13.393 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-01-15T18:02:13.435 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-01-15T18:02:13.663 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-01-15T18:02:13.678 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-01-15T18:02:13.711 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up lua-any (27ubuntu1) ... 2024-01-15T18:02:13.729 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-01-15T18:02:13.836 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-portend (3.0.0-1) ... 2024-01-15T18:02:14.049 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-websocket. 2024-01-15T18:02:14.065 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-01-15T18:02:14.088 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-01-15T18:02:14.107 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-01-15T18:02:14.197 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2024-01-15T18:02:14.435 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-kubernetes. 2024-01-15T18:02:14.451 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-01-15T18:02:14.494 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-01-15T18:02:14.507 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-01-15T18:02:14.808 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-01-15T18:02:15.092 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-01-15T18:02:15.175 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-01-15T18:02:15.189 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_18.0.0-7629-g95c90f41-1jammy_all.deb ... 2024-01-15T18:02:15.262 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:15.304 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-rsa (4.8-1) ... 2024-01-15T18:02:15.560 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libonig5:amd64. 2024-01-15T18:02:15.572 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-01-15T18:02:15.574 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-01-15T18:02:15.618 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-01-15T18:02:15.831 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-01-15T18:02:15.962 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libjq1:amd64. 2024-01-15T18:02:15.977 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-01-15T18:02:16.021 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-01-15T18:02:16.100 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-01-15T18:02:16.249 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package jq. 2024-01-15T18:02:16.264 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-01-15T18:02:16.307 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-01-15T18:02:16.485 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-01-15T18:02:16.585 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package socat. 2024-01-15T18:02:16.600 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-01-15T18:02:16.643 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-01-15T18:02:16.811 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-01-15T18:02:16.937 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package xmlstarlet. 2024-01-15T18:02:16.953 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-01-15T18:02:16.995 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-01-15T18:02:17.172 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-01-15T18:02:17.290 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up librados2 (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:17.366 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package ceph-test. 2024-01-15T18:02:17.381 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../099-ceph-test_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:02:17.391 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libsqlite3-mod-ceph (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:17.424 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking ceph-test (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:17.500 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-01-15T18:02:17.870 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libcephfs2 (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:17.996 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libradosstriper1 (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:18.122 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-01-15T18:02:18.440 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up librbd1 (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:18.541 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-mgr-modules-core (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:18.651 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-fuse (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:18.869 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-01-15T18:02:18.869 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-01-15T18:02:19.470 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package ceph-volume. 2024-01-15T18:02:19.486 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../100-ceph-volume_18.0.0-7629-g95c90f41-1jammy_all.deb ... 2024-01-15T18:02:19.486 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libcephfs-dev (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:19.520 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking ceph-volume (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:19.611 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-rados (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:19.737 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up librgw2 (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:19.863 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up cephfs-top (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:19.896 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package cephfs-mirror. 2024-01-15T18:02:19.912 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../101-cephfs-mirror_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:02:19.957 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking cephfs-mirror (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:20.023 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-01-15T18:02:20.210 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-colorama. 2024-01-15T18:02:20.225 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-01-15T18:02:20.268 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-01-15T18:02:20.562 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package cephfs-shell. 2024-01-15T18:02:20.578 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../103-cephfs-shell_18.0.0-7629-g95c90f41-1jammy_all.deb ... 2024-01-15T18:02:20.620 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking cephfs-shell (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:20.882 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package cephfs-top. 2024-01-15T18:02:20.897 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../104-cephfs-top_18.0.0-7629-g95c90f41-1jammy_all.deb ... 2024-01-15T18:02:20.940 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking cephfs-top (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:21.209 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libcephfs-dev. 2024-01-15T18:02:21.224 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../105-libcephfs-dev_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:02:21.267 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libcephfs-dev (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:21.359 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-rbd (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:21.485 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up rbd-fuse (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:21.595 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package lua-socket:amd64. 2024-01-15T18:02:21.610 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-01-15T18:02:21.611 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-rgw (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:21.653 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-01-15T18:02:21.712 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up python3-cephfs (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:21.813 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up cephfs-shell (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:21.931 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-common (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:22.007 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package lua-sec:amd64. 2024-01-15T18:02:22.023 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-01-15T18:02:22.065 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-01-15T18:02:22.419 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package nvme-cli. 2024-01-15T18:02:22.435 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-01-15T18:02:22.469 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-01-15T18:02:22.477 INFO:teuthology.orchestra.run.smithi027.stdout:Adding group ceph....done 2024-01-15T18:02:22.833 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package pkg-config. 2024-01-15T18:02:22.848 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-01-15T18:02:22.891 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-01-15T18:02:23.169 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-01-15T18:02:23.169 INFO:teuthology.orchestra.run.smithi027.stdout:Adding system user ceph....done 2024-01-15T18:02:23.184 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-01-15T18:02:23.227 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-01-15T18:02:23.347 INFO:teuthology.orchestra.run.smithi027.stdout:Setting system user ceph properties....done 2024-01-15T18:02:23.367 INFO:teuthology.orchestra.run.smithi027.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-01-15T18:02:23.517 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-01-15T18:02:23.680 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-iniconfig. 2024-01-15T18:02:23.695 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-01-15T18:02:23.746 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-01-15T18:02:23.910 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-01-15T18:02:23.999 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-pastescript. 2024-01-15T18:02:24.015 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-01-15T18:02:24.049 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-01-15T18:02:24.386 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-pluggy. 2024-01-15T18:02:24.401 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-01-15T18:02:24.443 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-01-15T18:02:24.575 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up cephfs-mirror (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:24.755 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-psutil. 2024-01-15T18:02:24.771 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-01-15T18:02:24.783 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-01-15T18:02:24.783 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-01-15T18:02:24.821 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-01-15T18:02:25.191 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-py. 2024-01-15T18:02:25.206 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-01-15T18:02:25.249 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-py (1.10.0-1) ... 2024-01-15T18:02:25.434 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-test (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:25.560 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up radosgw (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:25.594 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-pygments. 2024-01-15T18:02:25.610 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-01-15T18:02:25.652 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-01-15T18:02:26.106 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-pyinotify. 2024-01-15T18:02:26.121 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-01-15T18:02:26.132 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-01-15T18:02:26.132 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-01-15T18:02:26.156 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-01-15T18:02:26.434 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-toml. 2024-01-15T18:02:26.449 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-01-15T18:02:26.491 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-01-15T18:02:26.753 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-base (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:26.811 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-pytest. 2024-01-15T18:02:26.826 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-01-15T18:02:26.869 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-01-15T18:02:27.119 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-01-15T18:02:27.173 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package python3-simplejson. 2024-01-15T18:02:27.188 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-01-15T18:02:27.231 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-01-15T18:02:27.525 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-01-15T18:02:27.541 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-01-15T18:02:27.583 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-01-15T18:02:27.781 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-mds (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:28.022 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-01-15T18:02:28.022 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-01-15T18:02:28.320 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package radosgw. 2024-01-15T18:02:28.335 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../122-radosgw_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:02:28.378 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking radosgw (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:28.649 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-mgr (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:28.852 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-01-15T18:02:28.853 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-01-15T18:02:28.973 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package rbd-fuse. 2024-01-15T18:02:28.988 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../123-rbd-fuse_18.0.0-7629-g95c90f41-1jammy_amd64.deb ... 2024-01-15T18:02:29.032 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking rbd-fuse (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:29.350 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-01-15T18:02:29.434 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-osd (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:29.620 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-01-15T18:02:29.729 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-01-15T18:02:29.755 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-01-15T18:02:29.755 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-01-15T18:02:30.015 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-01-15T18:02:30.386 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-mgr-k8sevents (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:30.397 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-01-15T18:02:30.512 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:30.646 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-mon (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:30.854 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-01-15T18:02:30.854 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-01-15T18:02:30.963 INFO:teuthology.orchestra.run.smithi093.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-01-15T18:02:31.036 INFO:teuthology.orchestra.run.smithi093.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2024-01-15T18:02:31.090 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up cephadm (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:31.447 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-mgr-cephadm (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:31.582 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:31.634 INFO:teuthology.orchestra.run.smithi093.stdout:Adding system user cephadm....done 2024-01-15T18:02:31.682 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-mgr-dashboard (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:31.803 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-01-15T18:02:31.809 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ceph-volume (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:32.055 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-01-15T18:02:32.170 INFO:teuthology.orchestra.run.smithi027.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-01-15T18:02:32.270 INFO:teuthology.orchestra.run.smithi027.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-01-15T18:02:32.340 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-01-15T18:02:32.406 INFO:teuthology.orchestra.run.smithi027.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-01-15T18:02:32.466 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-01-15T18:02:32.725 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-01-15T18:02:32.993 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-01-15T18:02:33.119 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-py (1.10.0-1) ... 2024-01-15T18:02:33.437 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-01-15T18:02:33.500 INFO:teuthology.orchestra.run.smithi027.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-01-15T18:02:33.789 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-01-15T18:02:34.057 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-01-15T18:02:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-01-15T18:02:34.196 DEBUG:teuthology.orchestra.run.smithi027:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-01-15T18:02:34.272 INFO:teuthology.orchestra.run.smithi027.stdout:Reading package lists... 2024-01-15T18:02:34.392 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-01-15T18:02:34.441 INFO:teuthology.orchestra.run.smithi027.stdout:Building dependency tree... 2024-01-15T18:02:34.442 INFO:teuthology.orchestra.run.smithi027.stdout:Reading state information... 2024-01-15T18:02:34.601 INFO:teuthology.orchestra.run.smithi027.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-01-15T18:02:34.601 INFO:teuthology.orchestra.run.smithi027.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-01-15T18:02:34.601 INFO:teuthology.orchestra.run.smithi027.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2024-01-15T18:02:34.601 INFO:teuthology.orchestra.run.smithi027.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2024-01-15T18:02:34.601 INFO:teuthology.orchestra.run.smithi027.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-01-15T18:02:34.601 INFO:teuthology.orchestra.run.smithi027.stdout:The following packages were automatically installed and are no longer required: 2024-01-15T18:02:34.601 INFO:teuthology.orchestra.run.smithi027.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-01-15T18:02:34.602 INFO:teuthology.orchestra.run.smithi027.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-01-15T18:02:34.602 INFO:teuthology.orchestra.run.smithi027.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-01-15T18:02:34.602 INFO:teuthology.orchestra.run.smithi027.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-01-15T18:02:34.602 INFO:teuthology.orchestra.run.smithi027.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-01-15T18:02:34.602 INFO:teuthology.orchestra.run.smithi027.stdout:Use 'sudo apt autoremove' to remove them. 2024-01-15T18:02:34.603 INFO:teuthology.orchestra.run.smithi027.stdout:The following additional packages will be installed: 2024-01-15T18:02:34.603 INFO:teuthology.orchestra.run.smithi027.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-01-15T18:02:34.603 INFO:teuthology.orchestra.run.smithi027.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-01-15T18:02:34.603 INFO:teuthology.orchestra.run.smithi027.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2024-01-15T18:02:34.603 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-01-15T18:02:34.603 INFO:teuthology.orchestra.run.smithi027.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-01-15T18:02:34.604 INFO:teuthology.orchestra.run.smithi027.stdout:Suggested packages: 2024-01-15T18:02:34.604 INFO:teuthology.orchestra.run.smithi027.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-01-15T18:02:34.605 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2024-01-15T18:02:34.605 INFO:teuthology.orchestra.run.smithi027.stdout: libengine-pkcs11-openssl 2024-01-15T18:02:34.655 INFO:teuthology.orchestra.run.smithi027.stdout:The following NEW packages will be installed: 2024-01-15T18:02:34.655 INFO:teuthology.orchestra.run.smithi027.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-01-15T18:02:34.655 INFO:teuthology.orchestra.run.smithi027.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-01-15T18:02:34.656 INFO:teuthology.orchestra.run.smithi027.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-01-15T18:02:34.656 INFO:teuthology.orchestra.run.smithi027.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-01-15T18:02:34.656 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-01-15T18:02:34.656 INFO:teuthology.orchestra.run.smithi027.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-01-15T18:02:34.656 INFO:teuthology.orchestra.run.smithi027.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-01-15T18:02:34.657 INFO:teuthology.orchestra.run.smithi027.stdout:The following packages will be upgraded: 2024-01-15T18:02:34.657 INFO:teuthology.orchestra.run.smithi027.stdout: libssl3 2024-01-15T18:02:34.676 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up unzip (6.0-26ubuntu3.1) ... 2024-01-15T18:02:34.751 INFO:teuthology.orchestra.run.smithi027.stdout:1 upgraded, 39 newly installed, 0 to remove and 268 not upgraded. 2024-01-15T18:02:34.751 INFO:teuthology.orchestra.run.smithi027.stdout:Need to get 54.6 MB of archives. 2024-01-15T18:02:34.751 INFO:teuthology.orchestra.run.smithi027.stdout:After this operation, 202 MB of additional disk space will be used. 2024-01-15T18:02:34.751 INFO:teuthology.orchestra.run.smithi027.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-01-15T18:02:34.952 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-01-15T18:02:34.962 INFO:teuthology.orchestra.run.smithi027.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.12 [1,901 kB] 2024-01-15T18:02:35.070 INFO:teuthology.orchestra.run.smithi027.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-01-15T18:02:35.089 INFO:teuthology.orchestra.run.smithi027.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-01-15T18:02:35.089 INFO:teuthology.orchestra.run.smithi027.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-01-15T18:02:35.093 INFO:teuthology.orchestra.run.smithi027.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-01-15T18:02:35.094 INFO:teuthology.orchestra.run.smithi027.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-01-15T18:02:35.094 INFO:teuthology.orchestra.run.smithi027.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-01-15T18:02:35.097 INFO:teuthology.orchestra.run.smithi027.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-01-15T18:02:35.099 INFO:teuthology.orchestra.run.smithi027.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-01-15T18:02:35.099 INFO:teuthology.orchestra.run.smithi027.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-01-15T18:02:35.111 INFO:teuthology.orchestra.run.smithi027.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-01-15T18:02:35.131 INFO:teuthology.orchestra.run.smithi027.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-01-15T18:02:35.220 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-01-15T18:02:35.471 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-01-15T18:02:35.581 INFO:teuthology.orchestra.run.smithi027.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-01-15T18:02:35.583 INFO:teuthology.orchestra.run.smithi027.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-01-15T18:02:35.583 INFO:teuthology.orchestra.run.smithi027.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2024-01-15T18:02:35.586 INFO:teuthology.orchestra.run.smithi027.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-01-15T18:02:35.586 INFO:teuthology.orchestra.run.smithi027.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-01-15T18:02:35.591 INFO:teuthology.orchestra.run.smithi027.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.10-0ubuntu0.22.04.1 [143 kB] 2024-01-15T18:02:35.592 INFO:teuthology.orchestra.run.smithi027.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-01-15T18:02:35.603 INFO:teuthology.orchestra.run.smithi027.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-01-15T18:02:35.603 INFO:teuthology.orchestra.run.smithi027.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.12 [2,373 kB] 2024-01-15T18:02:35.644 INFO:teuthology.orchestra.run.smithi027.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-01-15T18:02:35.652 INFO:teuthology.orchestra.run.smithi027.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2024-01-15T18:02:35.663 INFO:teuthology.orchestra.run.smithi027.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-01-15T18:02:35.696 INFO:teuthology.orchestra.run.smithi027.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-01-15T18:02:35.739 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-ceph-argparse (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:35.743 INFO:teuthology.orchestra.run.smithi027.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-01-15T18:02:35.743 INFO:teuthology.orchestra.run.smithi027.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-01-15T18:02:35.744 INFO:teuthology.orchestra.run.smithi027.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-01-15T18:02:35.744 INFO:teuthology.orchestra.run.smithi027.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.10-0ubuntu0.22.04.1 [1,223 kB] 2024-01-15T18:02:35.756 INFO:teuthology.orchestra.run.smithi027.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-01-15T18:02:35.758 INFO:teuthology.orchestra.run.smithi027.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.10-0ubuntu0.22.04.1 [16.1 MB] 2024-01-15T18:02:36.014 INFO:teuthology.orchestra.run.smithi027.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-01-15T18:02:36.014 INFO:teuthology.orchestra.run.smithi027.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-01-15T18:02:36.015 INFO:teuthology.orchestra.run.smithi027.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-01-15T18:02:36.015 INFO:teuthology.orchestra.run.smithi027.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-01-15T18:02:36.015 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-01-15T18:02:36.016 INFO:teuthology.orchestra.run.smithi027.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-01-15T18:02:36.016 INFO:teuthology.orchestra.run.smithi027.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-01-15T18:02:36.030 INFO:teuthology.orchestra.run.smithi027.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-01-15T18:02:36.030 INFO:teuthology.orchestra.run.smithi027.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-01-15T18:02:36.141 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-01-15T18:02:36.267 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-01-15T18:02:36.423 INFO:teuthology.orchestra.run.smithi027.stdout:Preconfiguring packages ... 2024-01-15T18:02:36.543 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-01-15T18:02:36.572 INFO:teuthology.orchestra.run.smithi027.stdout:Fetched 54.6 MB in 1s (40.0 MB/s) 2024-01-15T18:02:36.694 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-01-15T18:02:36.714 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package flex. 2024-01-15T18:02:36.787 INFO:teuthology.orchestra.run.smithi027.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140566 files and directories currently installed.) 2024-01-15T18:02:36.791 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-01-15T18:02:36.829 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking flex (2.6.4-8build2) ... 2024-01-15T18:02:36.962 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-01-15T18:02:37.340 INFO:teuthology.orchestra.run.smithi093.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-01-15T18:02:37.375 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.12_amd64.deb ... 2024-01-15T18:02:37.392 INFO:teuthology.orchestra.run.smithi093.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-01-15T18:02:37.457 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-01-15T18:02:37.552 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.12) over (3.0.2-0ubuntu1.7) ... 2024-01-15T18:02:37.583 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-01-15T18:02:37.943 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.12) ... 2024-01-15T18:02:37.959 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-01-15T18:02:38.084 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-01-15T18:02:38.361 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-01-15T18:02:38.630 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package bison. 2024-01-15T18:02:38.653 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-01-15T18:02:38.704 INFO:teuthology.orchestra.run.smithi027.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140639 files and directories currently installed.) 2024-01-15T18:02:38.708 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-01-15T18:02:38.749 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-01-15T18:02:38.955 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up zip (3.0-12build2) ... 2024-01-15T18:02:39.091 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-01-15T18:02:39.106 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-01-15T18:02:39.114 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-01-15T18:02:39.141 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-01-15T18:02:39.511 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package dump. 2024-01-15T18:02:39.526 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-01-15T18:02:39.569 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking dump (0.4b47-3) ... 2024-01-15T18:02:39.808 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-01-15T18:02:39.856 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package indent. 2024-01-15T18:02:39.871 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-01-15T18:02:39.914 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-01-15T18:02:40.084 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-01-15T18:02:40.142 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-01-15T18:02:40.158 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-01-15T18:02:40.192 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-01-15T18:02:40.210 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-01-15T18:02:40.352 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-01-15T18:02:40.545 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-01-15T18:02:40.562 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-01-15T18:02:40.604 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-01-15T18:02:40.662 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-01-15T18:02:40.865 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-01-15T18:02:40.881 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-01-15T18:02:40.923 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-01-15T18:02:41.038 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-01-15T18:02:41.218 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libfl2:amd64. 2024-01-15T18:02:41.234 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-01-15T18:02:41.268 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-01-15T18:02:41.306 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-01-15T18:02:41.521 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-01-15T18:02:41.537 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-01-15T18:02:41.579 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-01-15T18:02:41.682 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-01-15T18:02:41.831 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libjson-perl. 2024-01-15T18:02:41.848 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-01-15T18:02:41.898 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-01-15T18:02:41.967 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-01-15T18:02:42.210 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libllvm14:amd64. 2024-01-15T18:02:42.226 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-01-15T18:02:42.268 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-01-15T18:02:42.285 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-01-15T18:02:42.636 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-01-15T18:02:42.937 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-01-15T18:02:43.072 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-ceph-common (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:43.373 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-idna (3.3-1) ... 2024-01-15T18:02:43.732 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-01-15T18:02:43.804 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-01-15T18:02:43.821 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-01-15T18:02:43.854 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-01-15T18:02:44.115 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libmbim-proxy. 2024-01-15T18:02:44.132 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-01-15T18:02:44.173 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-01-15T18:02:44.560 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-01-15T18:02:44.576 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2024-01-15T18:02:44.626 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2024-01-15T18:02:44.930 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-01-15T18:02:45.046 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libndp0:amd64. 2024-01-15T18:02:45.062 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-01-15T18:02:45.112 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-01-15T18:02:45.198 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-01-15T18:02:45.349 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-toml (0.10.2-1) ... 2024-01-15T18:02:45.465 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libnm0:amd64. 2024-01-15T18:02:45.482 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-01-15T18:02:45.523 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-01-15T18:02:45.608 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-01-15T18:02:45.734 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-01-15T18:02:45.818 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libpq5:amd64. 2024-01-15T18:02:45.834 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../16-libpq5_14.10-0ubuntu0.22.04.1_amd64.deb ... 2024-01-15T18:02:45.876 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libpq5:amd64 (14.10-0ubuntu0.22.04.1) ... 2024-01-15T18:02:45.893 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-01-15T18:02:46.187 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-01-15T18:02:46.221 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-01-15T18:02:46.237 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-01-15T18:02:46.279 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-01-15T18:02:46.471 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-01-15T18:02:46.556 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libqmi-proxy. 2024-01-15T18:02:46.573 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-01-15T18:02:46.623 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-01-15T18:02:46.731 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-01-15T18:02:46.857 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-01-15T18:02:46.867 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-01-15T18:02:46.883 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.12_amd64.deb ... 2024-01-15T18:02:46.925 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2024-01-15T18:02:47.166 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-01-15T18:02:47.412 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-01-15T18:02:47.417 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-01-15T18:02:47.428 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-01-15T18:02:47.470 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-01-15T18:02:47.710 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-01-15T18:02:47.823 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package modemmanager. 2024-01-15T18:02:47.840 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2024-01-15T18:02:47.890 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2024-01-15T18:02:47.978 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-01-15T18:02:48.229 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-01-15T18:02:48.521 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-01-15T18:02:48.597 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package network-manager. 2024-01-15T18:02:48.614 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-01-15T18:02:48.648 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-01-15T18:02:48.664 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-01-15T18:02:48.949 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-01-15T18:02:49.074 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-01-15T18:02:49.280 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package ppp. 2024-01-15T18:02:49.296 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-01-15T18:02:49.367 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-01-15T18:02:49.632 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-01-15T18:02:49.677 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-01-15T18:02:49.928 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-01-15T18:02:49.951 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package pptp-linux. 2024-01-15T18:02:49.966 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-01-15T18:02:50.009 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-01-15T18:02:50.028 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-01-15T18:02:50.180 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-01-15T18:02:50.253 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package network-manager-pptp. 2024-01-15T18:02:50.268 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-01-15T18:02:50.276 INFO:teuthology.orchestra.run.smithi093.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-01-15T18:02:50.311 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-01-15T18:02:50.515 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-01-15T18:02:50.632 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-01-15T18:02:50.640 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package postgresql-client-common. 2024-01-15T18:02:50.655 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-01-15T18:02:50.698 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking postgresql-client-common (238) ... 2024-01-15T18:02:50.952 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package postgresql-client-14. 2024-01-15T18:02:50.968 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../27-postgresql-client-14_14.10-0ubuntu0.22.04.1_amd64.deb ... 2024-01-15T18:02:51.002 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking postgresql-client-14 (14.10-0ubuntu0.22.04.1) ... 2024-01-15T18:02:51.009 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-01-15T18:02:51.276 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up lua-any (27ubuntu1) ... 2024-01-15T18:02:51.427 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-portend (3.0.0-1) ... 2024-01-15T18:02:51.573 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package postgresql-common. 2024-01-15T18:02:51.590 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-01-15T18:02:51.650 INFO:teuthology.orchestra.run.smithi027.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-01-15T18:02:51.670 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-01-15T18:02:51.702 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking postgresql-common (238) ... 2024-01-15T18:02:51.788 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2024-01-15T18:02:52.122 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-01-15T18:02:52.155 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package postgresql-14. 2024-01-15T18:02:52.172 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../29-postgresql-14_14.10-0ubuntu0.22.04.1_amd64.deb ... 2024-01-15T18:02:52.216 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking postgresql-14 (14.10-0ubuntu0.22.04.1) ... 2024-01-15T18:02:52.432 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-01-15T18:02:52.708 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-01-15T18:02:52.919 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-rsa (4.8-1) ... 2024-01-15T18:02:53.187 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-01-15T18:02:53.284 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package postgresql. 2024-01-15T18:02:53.300 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-01-15T18:02:53.491 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-01-15T18:02:53.742 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-01-15T18:02:53.792 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking postgresql (14+238) ... 2024-01-15T18:02:54.096 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package postgresql-client. 2024-01-15T18:02:54.112 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-01-15T18:02:54.119 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-01-15T18:02:54.146 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking postgresql-client (14+238) ... 2024-01-15T18:02:54.340 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package postgresql-contrib. 2024-01-15T18:02:54.355 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-01-15T18:02:54.389 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking postgresql-contrib (14+238) ... 2024-01-15T18:02:54.645 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-01-15T18:02:54.692 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package tcl8.6. 2024-01-15T18:02:54.709 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-01-15T18:02:54.750 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-01-15T18:02:55.006 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-01-15T18:02:55.053 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package tcl. 2024-01-15T18:02:55.070 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-01-15T18:02:55.150 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-01-15T18:02:55.156 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up librados2 (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:55.283 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libsqlite3-mod-ceph (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:55.409 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-01-15T18:02:55.456 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package wpasupplicant. 2024-01-15T18:02:55.473 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-01-15T18:02:55.506 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-01-15T18:02:55.728 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libcephfs2 (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:55.843 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-01-15T18:02:55.854 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libradosstriper1 (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:55.861 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-01-15T18:02:55.902 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-01-15T18:02:55.980 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-01-15T18:02:56.273 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up librbd1 (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:56.305 INFO:teuthology.orchestra.run.smithi027.stdout:Selecting previously unselected package usb-modeswitch. 2024-01-15T18:02:56.322 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-01-15T18:02:56.372 INFO:teuthology.orchestra.run.smithi027.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-01-15T18:02:56.408 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up ceph-mgr-modules-core (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:56.534 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up ceph-fuse (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:56.661 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up postgresql-client-common (238) ... 2024-01-15T18:02:56.744 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-01-15T18:02:56.745 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-01-15T18:02:56.871 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up flex (2.6.4-8build2) ... 2024-01-15T18:02:56.997 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-01-15T18:02:57.123 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libpq5:amd64 (14.10-0ubuntu0.22.04.1) ... 2024-01-15T18:02:57.249 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-01-15T18:02:57.368 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libcephfs-dev (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:57.402 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-01-15T18:02:57.486 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-rados (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:57.518 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up postgresql-client-14 (14.10-0ubuntu0.22.04.1) ... 2024-01-15T18:02:57.620 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up librgw2 (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:57.738 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up cephfs-top (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:57.863 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-01-15T18:02:58.479 INFO:teuthology.orchestra.run.smithi027.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-01-15T18:02:58.582 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-01-15T18:02:58.975 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-01-15T18:02:58.975 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-01-15T18:02:59.134 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-rbd (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:59.260 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up rbd-fuse (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:59.386 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-rgw (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:59.537 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up python3-cephfs (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:59.671 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up cephfs-shell (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:59.676 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-01-15T18:02:59.797 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up ceph-common (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:02:59.802 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2024-01-15T18:02:59.928 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-01-15T18:03:00.054 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-01-15T18:03:00.197 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-01-15T18:03:00.342 INFO:teuthology.orchestra.run.smithi093.stdout:Adding group ceph....done 2024-01-15T18:03:00.431 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-01-15T18:03:00.557 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-01-15T18:03:00.801 INFO:teuthology.orchestra.run.smithi093.stdout:Adding system user ceph....done 2024-01-15T18:03:00.987 INFO:teuthology.orchestra.run.smithi093.stdout:Setting system user ceph properties....done 2024-01-15T18:03:01.007 INFO:teuthology.orchestra.run.smithi093.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-01-15T18:03:01.147 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-01-15T18:03:01.212 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-01-15T18:03:01.338 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up dump (0.4b47-3) ... 2024-01-15T18:03:01.434 INFO:teuthology.orchestra.run.smithi027.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-01-15T18:03:01.498 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2024-01-15T18:03:01.540 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-01-15T18:03:01.624 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-01-15T18:03:01.750 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up tcl (8.6.11+1build2) ... 2024-01-15T18:03:01.851 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libjson-perl (4.04000-1) ... 2024-01-15T18:03:01.994 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-01-15T18:03:02.084 INFO:teuthology.orchestra.run.smithi027.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-01-15T18:03:02.154 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-01-15T18:03:02.181 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up cephfs-mirror (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:03:02.279 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-01-15T18:03:02.380 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-01-15T18:03:02.380 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-01-15T18:03:02.456 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-01-15T18:03:02.582 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-01-15T18:03:02.708 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-01-15T18:03:02.851 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up postgresql-client (14+238) ... 2024-01-15T18:03:02.969 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-01-15T18:03:02.983 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up ceph-test (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:03:03.142 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up radosgw (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:03:03.613 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:03.613 INFO:teuthology.orchestra.run.smithi027.stdout:The following network interfaces were found in /etc/network/interfaces 2024-01-15T18:03:03.613 INFO:teuthology.orchestra.run.smithi027.stdout:which means they are currently configured by ifupdown: 2024-01-15T18:03:03.613 INFO:teuthology.orchestra.run.smithi027.stdout:- ens1f0 2024-01-15T18:03:03.613 INFO:teuthology.orchestra.run.smithi027.stdout:If you want to manage those interfaces with NetworkManager instead 2024-01-15T18:03:03.614 INFO:teuthology.orchestra.run.smithi027.stdout:remove their configuration from /etc/network/interfaces. 2024-01-15T18:03:03.614 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:03.678 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-01-15T18:03:03.678 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-01-15T18:03:03.751 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-01-15T18:03:04.151 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-01-15T18:03:04.462 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up ceph-base (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:03:04.540 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-01-15T18:03:04.821 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-01-15T18:03:05.197 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-01-15T18:03:05.381 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-01-15T18:03:05.476 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up ceph-mds (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:03:05.507 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-01-15T18:03:05.624 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up postgresql-common (238) ... 2024-01-15T18:03:05.705 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-01-15T18:03:05.706 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-01-15T18:03:06.327 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up ceph-mgr (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:03:06.523 INFO:teuthology.orchestra.run.smithi027.stdout:Adding user postgres to group ssl-cert 2024-01-15T18:03:06.539 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-01-15T18:03:06.539 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-01-15T18:03:06.783 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:06.783 INFO:teuthology.orchestra.run.smithi027.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-01-15T18:03:07.035 INFO:teuthology.orchestra.run.smithi027.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-01-15T18:03:07.035 INFO:teuthology.orchestra.run.smithi027.stdout:Removing obsolete dictionary files: 2024-01-15T18:03:07.145 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up ceph-osd (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:03:07.455 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-01-15T18:03:07.455 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-01-15T18:03:07.741 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-01-15T18:03:08.080 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up ceph-mgr-k8sevents (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:03:08.159 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up postgresql-14 (14.10-0ubuntu0.22.04.1) ... 2024-01-15T18:03:08.214 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:03:08.365 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up ceph-mon (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:03:08.494 INFO:teuthology.orchestra.run.smithi027.stdout:Creating new PostgreSQL cluster 14/main ... 2024-01-15T18:03:08.510 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-01-15T18:03:08.565 INFO:teuthology.orchestra.run.smithi027.stdout:The files belonging to this database system will be owned by user "postgres". 2024-01-15T18:03:08.565 INFO:teuthology.orchestra.run.smithi027.stdout:This user must also own the server process. 2024-01-15T18:03:08.565 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:08.566 INFO:teuthology.orchestra.run.smithi027.stdout:The database cluster will be initialized with locale "en_US". 2024-01-15T18:03:08.566 INFO:teuthology.orchestra.run.smithi027.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-01-15T18:03:08.566 INFO:teuthology.orchestra.run.smithi027.stdout:The default text search configuration will be set to "english". 2024-01-15T18:03:08.566 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:08.566 INFO:teuthology.orchestra.run.smithi027.stdout:Data page checksums are disabled. 2024-01-15T18:03:08.566 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:08.566 INFO:teuthology.orchestra.run.smithi027.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-01-15T18:03:08.567 INFO:teuthology.orchestra.run.smithi027.stdout:creating subdirectories ... ok 2024-01-15T18:03:08.567 INFO:teuthology.orchestra.run.smithi027.stdout:selecting dynamic shared memory implementation ... posix 2024-01-15T18:03:08.571 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-01-15T18:03:08.572 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-01-15T18:03:08.591 INFO:teuthology.orchestra.run.smithi027.stdout:selecting default max_connections ... 100 2024-01-15T18:03:08.626 INFO:teuthology.orchestra.run.smithi027.stdout:selecting default shared_buffers ... 128MB 2024-01-15T18:03:08.640 INFO:teuthology.orchestra.run.smithi027.stdout:selecting default time zone ... Etc/UTC 2024-01-15T18:03:08.641 INFO:teuthology.orchestra.run.smithi027.stdout:creating configuration files ... ok 2024-01-15T18:03:08.814 INFO:teuthology.orchestra.run.smithi027.stdout:running bootstrap script ... ok 2024-01-15T18:03:09.175 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up ceph-mgr-cephadm (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:03:09.334 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up ceph (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:03:09.446 INFO:teuthology.orchestra.run.smithi027.stdout:performing post-bootstrap initialization ... ok 2024-01-15T18:03:09.452 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up ceph-mgr-dashboard (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:03:09.586 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up ceph-volume (18.0.0-7629-g95c90f41-1jammy) ... 2024-01-15T18:03:09.947 INFO:teuthology.orchestra.run.smithi093.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-01-15T18:03:10.027 INFO:teuthology.orchestra.run.smithi027.stdout:syncing data to disk ... ok 2024-01-15T18:03:10.056 INFO:teuthology.orchestra.run.smithi093.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-01-15T18:03:10.208 INFO:teuthology.orchestra.run.smithi093.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-01-15T18:03:10.506 INFO:teuthology.orchestra.run.smithi027.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-01-15T18:03:11.384 INFO:teuthology.orchestra.run.smithi093.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-01-15T18:03:12.103 INFO:teuthology.orchestra.run.smithi093.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-01-15T18:03:12.108 DEBUG:teuthology.orchestra.run.smithi093:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-01-15T18:03:12.181 INFO:teuthology.orchestra.run.smithi093.stdout:Reading package lists... 2024-01-15T18:03:12.345 INFO:teuthology.orchestra.run.smithi093.stdout:Building dependency tree... 2024-01-15T18:03:12.346 INFO:teuthology.orchestra.run.smithi093.stdout:Reading state information... 2024-01-15T18:03:12.495 INFO:teuthology.orchestra.run.smithi093.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-01-15T18:03:12.495 INFO:teuthology.orchestra.run.smithi093.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-01-15T18:03:12.495 INFO:teuthology.orchestra.run.smithi093.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2024-01-15T18:03:12.495 INFO:teuthology.orchestra.run.smithi093.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2024-01-15T18:03:12.495 INFO:teuthology.orchestra.run.smithi093.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-01-15T18:03:12.495 INFO:teuthology.orchestra.run.smithi093.stdout:The following packages were automatically installed and are no longer required: 2024-01-15T18:03:12.495 INFO:teuthology.orchestra.run.smithi093.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-01-15T18:03:12.496 INFO:teuthology.orchestra.run.smithi093.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-01-15T18:03:12.496 INFO:teuthology.orchestra.run.smithi093.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-01-15T18:03:12.496 INFO:teuthology.orchestra.run.smithi093.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-01-15T18:03:12.496 INFO:teuthology.orchestra.run.smithi093.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-01-15T18:03:12.496 INFO:teuthology.orchestra.run.smithi093.stdout:Use 'sudo apt autoremove' to remove them. 2024-01-15T18:03:12.497 INFO:teuthology.orchestra.run.smithi093.stdout:The following additional packages will be installed: 2024-01-15T18:03:12.498 INFO:teuthology.orchestra.run.smithi093.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-01-15T18:03:12.498 INFO:teuthology.orchestra.run.smithi093.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-01-15T18:03:12.498 INFO:teuthology.orchestra.run.smithi093.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2024-01-15T18:03:12.498 INFO:teuthology.orchestra.run.smithi093.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-01-15T18:03:12.498 INFO:teuthology.orchestra.run.smithi093.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-01-15T18:03:12.498 INFO:teuthology.orchestra.run.smithi093.stdout:Suggested packages: 2024-01-15T18:03:12.498 INFO:teuthology.orchestra.run.smithi093.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-01-15T18:03:12.498 INFO:teuthology.orchestra.run.smithi093.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2024-01-15T18:03:12.498 INFO:teuthology.orchestra.run.smithi093.stdout: libengine-pkcs11-openssl 2024-01-15T18:03:12.546 INFO:teuthology.orchestra.run.smithi093.stdout:The following NEW packages will be installed: 2024-01-15T18:03:12.546 INFO:teuthology.orchestra.run.smithi093.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-01-15T18:03:12.546 INFO:teuthology.orchestra.run.smithi093.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-01-15T18:03:12.547 INFO:teuthology.orchestra.run.smithi093.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-01-15T18:03:12.548 INFO:teuthology.orchestra.run.smithi093.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-01-15T18:03:12.548 INFO:teuthology.orchestra.run.smithi093.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-01-15T18:03:12.548 INFO:teuthology.orchestra.run.smithi093.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-01-15T18:03:12.548 INFO:teuthology.orchestra.run.smithi093.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-01-15T18:03:12.549 INFO:teuthology.orchestra.run.smithi093.stdout:The following packages will be upgraded: 2024-01-15T18:03:12.549 INFO:teuthology.orchestra.run.smithi093.stdout: libssl3 2024-01-15T18:03:12.615 INFO:teuthology.orchestra.run.smithi093.stdout:1 upgraded, 39 newly installed, 0 to remove and 268 not upgraded. 2024-01-15T18:03:12.615 INFO:teuthology.orchestra.run.smithi093.stdout:Need to get 54.6 MB of archives. 2024-01-15T18:03:12.615 INFO:teuthology.orchestra.run.smithi093.stdout:After this operation, 202 MB of additional disk space will be used. 2024-01-15T18:03:12.615 INFO:teuthology.orchestra.run.smithi093.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-01-15T18:03:12.774 INFO:teuthology.orchestra.run.smithi093.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.12 [1,901 kB] 2024-01-15T18:03:12.862 INFO:teuthology.orchestra.run.smithi093.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-01-15T18:03:12.877 INFO:teuthology.orchestra.run.smithi093.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-01-15T18:03:12.878 INFO:teuthology.orchestra.run.smithi093.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-01-15T18:03:12.880 INFO:teuthology.orchestra.run.smithi093.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-01-15T18:03:12.881 INFO:teuthology.orchestra.run.smithi093.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-01-15T18:03:12.882 INFO:teuthology.orchestra.run.smithi093.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-01-15T18:03:12.883 INFO:teuthology.orchestra.run.smithi093.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-01-15T18:03:12.884 INFO:teuthology.orchestra.run.smithi093.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-01-15T18:03:12.885 INFO:teuthology.orchestra.run.smithi093.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-01-15T18:03:12.892 INFO:teuthology.orchestra.run.smithi093.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-01-15T18:03:12.908 INFO:teuthology.orchestra.run.smithi093.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-01-15T18:03:13.279 INFO:teuthology.orchestra.run.smithi093.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-01-15T18:03:13.282 INFO:teuthology.orchestra.run.smithi093.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-01-15T18:03:13.282 INFO:teuthology.orchestra.run.smithi093.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2024-01-15T18:03:13.284 INFO:teuthology.orchestra.run.smithi093.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-01-15T18:03:13.285 INFO:teuthology.orchestra.run.smithi093.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-01-15T18:03:13.289 INFO:teuthology.orchestra.run.smithi093.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.10-0ubuntu0.22.04.1 [143 kB] 2024-01-15T18:03:13.291 INFO:teuthology.orchestra.run.smithi093.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-01-15T18:03:13.299 INFO:teuthology.orchestra.run.smithi093.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-01-15T18:03:13.300 INFO:teuthology.orchestra.run.smithi093.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.12 [2,373 kB] 2024-01-15T18:03:13.335 INFO:teuthology.orchestra.run.smithi093.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-01-15T18:03:13.341 INFO:teuthology.orchestra.run.smithi093.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2024-01-15T18:03:13.353 INFO:teuthology.orchestra.run.smithi093.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-01-15T18:03:13.382 INFO:teuthology.orchestra.run.smithi093.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-01-15T18:03:13.389 INFO:teuthology.orchestra.run.smithi093.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-01-15T18:03:13.392 INFO:teuthology.orchestra.run.smithi093.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-01-15T18:03:13.393 INFO:teuthology.orchestra.run.smithi093.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-01-15T18:03:13.394 INFO:teuthology.orchestra.run.smithi093.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.10-0ubuntu0.22.04.1 [1,223 kB] 2024-01-15T18:03:13.407 INFO:teuthology.orchestra.run.smithi093.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-01-15T18:03:13.421 INFO:teuthology.orchestra.run.smithi093.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.10-0ubuntu0.22.04.1 [16.1 MB] 2024-01-15T18:03:13.662 INFO:teuthology.orchestra.run.smithi093.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-01-15T18:03:13.662 INFO:teuthology.orchestra.run.smithi093.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-01-15T18:03:13.663 INFO:teuthology.orchestra.run.smithi093.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-01-15T18:03:13.663 INFO:teuthology.orchestra.run.smithi093.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-01-15T18:03:13.663 INFO:teuthology.orchestra.run.smithi093.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-01-15T18:03:13.664 INFO:teuthology.orchestra.run.smithi093.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-01-15T18:03:13.680 INFO:teuthology.orchestra.run.smithi093.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-01-15T18:03:13.681 INFO:teuthology.orchestra.run.smithi093.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-01-15T18:03:13.867 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-01-15T18:03:13.967 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up postgresql-contrib (14+238) ... 2024-01-15T18:03:14.048 INFO:teuthology.orchestra.run.smithi093.stdout:Preconfiguring packages ... 2024-01-15T18:03:14.093 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up postgresql (14+238) ... 2024-01-15T18:03:14.219 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-01-15T18:03:14.222 INFO:teuthology.orchestra.run.smithi093.stdout:Fetched 54.6 MB in 1s (48.5 MB/s) 2024-01-15T18:03:14.346 INFO:teuthology.orchestra.run.smithi027.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2024-01-15T18:03:14.365 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package flex. 2024-01-15T18:03:14.435 INFO:teuthology.orchestra.run.smithi093.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140566 files and directories currently installed.) 2024-01-15T18:03:14.439 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-01-15T18:03:14.488 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking flex (2.6.4-8build2) ... 2024-01-15T18:03:14.638 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-01-15T18:03:14.638 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-01-15T18:03:15.066 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.12_amd64.deb ... 2024-01-15T18:03:15.236 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.12) over (3.0.2-0ubuntu1.7) ... 2024-01-15T18:03:15.340 INFO:teuthology.orchestra.run.smithi027.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-01-15T18:03:15.648 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.12) ... 2024-01-15T18:03:15.683 INFO:teuthology.orchestra.run.smithi027.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-01-15T18:03:15.818 INFO:teuthology.orchestra.run.smithi027.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-01-15T18:03:16.186 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package bison. 2024-01-15T18:03:16.254 INFO:teuthology.orchestra.run.smithi093.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140639 files and directories currently installed.) 2024-01-15T18:03:16.258 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-01-15T18:03:16.296 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-01-15T18:03:16.722 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-01-15T18:03:16.736 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-01-15T18:03:16.780 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-01-15T18:03:17.067 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package dump. 2024-01-15T18:03:17.081 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-01-15T18:03:17.116 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking dump (0.4b47-3) ... 2024-01-15T18:03:17.204 INFO:teuthology.orchestra.run.smithi027.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-01-15T18:03:17.304 INFO:teuthology.orchestra.run.smithi027.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-01-15T18:03:17.411 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package indent. 2024-01-15T18:03:17.426 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-01-15T18:03:17.470 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-01-15T18:03:17.730 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-01-15T18:03:17.745 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-01-15T18:03:17.789 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-01-15T18:03:17.828 INFO:teuthology.orchestra.run.smithi027.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-01-15T18:03:17.836 DEBUG:teuthology.parallel:result is None 2024-01-15T18:03:18.142 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-01-15T18:03:18.156 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-01-15T18:03:18.200 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-01-15T18:03:18.419 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-01-15T18:03:18.434 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-01-15T18:03:18.469 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-01-15T18:03:18.822 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libfl2:amd64. 2024-01-15T18:03:18.837 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-01-15T18:03:18.881 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-01-15T18:03:19.150 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-01-15T18:03:19.164 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-01-15T18:03:19.208 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-01-15T18:03:19.461 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libjson-perl. 2024-01-15T18:03:19.475 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-01-15T18:03:19.511 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-01-15T18:03:19.788 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libllvm14:amd64. 2024-01-15T18:03:19.802 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-01-15T18:03:19.838 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-01-15T18:03:21.166 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-01-15T18:03:21.181 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-01-15T18:03:21.224 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-01-15T18:03:21.502 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libmbim-proxy. 2024-01-15T18:03:21.517 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-01-15T18:03:21.560 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-01-15T18:03:21.879 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-01-15T18:03:21.894 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2024-01-15T18:03:21.937 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2024-01-15T18:03:22.240 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libndp0:amd64. 2024-01-15T18:03:22.255 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-01-15T18:03:22.298 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-01-15T18:03:22.626 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libnm0:amd64. 2024-01-15T18:03:22.641 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-01-15T18:03:22.709 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-01-15T18:03:23.053 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libpq5:amd64. 2024-01-15T18:03:23.069 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../16-libpq5_14.10-0ubuntu0.22.04.1_amd64.deb ... 2024-01-15T18:03:23.120 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libpq5:amd64 (14.10-0ubuntu0.22.04.1) ... 2024-01-15T18:03:23.473 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-01-15T18:03:23.488 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-01-15T18:03:23.531 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-01-15T18:03:23.825 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libqmi-proxy. 2024-01-15T18:03:23.840 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-01-15T18:03:23.883 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-01-15T18:03:24.152 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-01-15T18:03:24.166 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.12_amd64.deb ... 2024-01-15T18:03:24.236 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2024-01-15T18:03:24.713 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-01-15T18:03:24.728 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-01-15T18:03:24.771 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-01-15T18:03:25.083 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package modemmanager. 2024-01-15T18:03:25.097 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2024-01-15T18:03:25.158 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2024-01-15T18:03:25.788 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package network-manager. 2024-01-15T18:03:25.803 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-01-15T18:03:25.846 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-01-15T18:03:26.445 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package ppp. 2024-01-15T18:03:26.459 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-01-15T18:03:26.779 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-01-15T18:03:27.133 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package pptp-linux. 2024-01-15T18:03:27.147 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-01-15T18:03:27.191 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-01-15T18:03:27.418 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package network-manager-pptp. 2024-01-15T18:03:27.433 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-01-15T18:03:27.468 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-01-15T18:03:27.796 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package postgresql-client-common. 2024-01-15T18:03:27.810 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-01-15T18:03:27.855 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking postgresql-client-common (238) ... 2024-01-15T18:03:28.158 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package postgresql-client-14. 2024-01-15T18:03:28.172 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../27-postgresql-client-14_14.10-0ubuntu0.22.04.1_amd64.deb ... 2024-01-15T18:03:28.248 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking postgresql-client-14 (14.10-0ubuntu0.22.04.1) ... 2024-01-15T18:03:28.729 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package postgresql-common. 2024-01-15T18:03:28.744 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-01-15T18:03:28.797 INFO:teuthology.orchestra.run.smithi093.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-01-15T18:03:28.819 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking postgresql-common (238) ... 2024-01-15T18:03:29.261 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package postgresql-14. 2024-01-15T18:03:29.276 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../29-postgresql-14_14.10-0ubuntu0.22.04.1_amd64.deb ... 2024-01-15T18:03:29.314 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking postgresql-14 (14.10-0ubuntu0.22.04.1) ... 2024-01-15T18:03:30.413 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package postgresql. 2024-01-15T18:03:30.428 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-01-15T18:03:30.472 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking postgresql (14+238) ... 2024-01-15T18:03:30.732 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package postgresql-client. 2024-01-15T18:03:30.746 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-01-15T18:03:30.782 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking postgresql-client (14+238) ... 2024-01-15T18:03:31.017 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package postgresql-contrib. 2024-01-15T18:03:31.032 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-01-15T18:03:31.076 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking postgresql-contrib (14+238) ... 2024-01-15T18:03:31.453 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package tcl8.6. 2024-01-15T18:03:31.468 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-01-15T18:03:31.512 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-01-15T18:03:31.798 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package tcl. 2024-01-15T18:03:31.813 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-01-15T18:03:31.860 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-01-15T18:03:32.200 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package wpasupplicant. 2024-01-15T18:03:32.216 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-01-15T18:03:32.250 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-01-15T18:03:32.588 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-01-15T18:03:32.602 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-01-15T18:03:32.646 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-01-15T18:03:33.007 INFO:teuthology.orchestra.run.smithi093.stdout:Selecting previously unselected package usb-modeswitch. 2024-01-15T18:03:33.021 INFO:teuthology.orchestra.run.smithi093.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-01-15T18:03:33.072 INFO:teuthology.orchestra.run.smithi093.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-01-15T18:03:33.391 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up postgresql-client-common (238) ... 2024-01-15T18:03:33.596 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up flex (2.6.4-8build2) ... 2024-01-15T18:03:33.722 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-01-15T18:03:33.873 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libpq5:amd64 (14.10-0ubuntu0.22.04.1) ... 2024-01-15T18:03:33.999 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-01-15T18:03:34.117 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-01-15T18:03:34.218 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up postgresql-client-14 (14.10-0ubuntu0.22.04.1) ... 2024-01-15T18:03:35.174 INFO:teuthology.orchestra.run.smithi093.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-01-15T18:03:35.265 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-01-15T18:03:35.625 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-01-15T18:03:35.625 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-01-15T18:03:36.296 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-01-15T18:03:36.419 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2024-01-15T18:03:36.545 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-01-15T18:03:36.670 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-01-15T18:03:36.918 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-01-15T18:03:37.047 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-01-15T18:03:37.173 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-01-15T18:03:37.847 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-01-15T18:03:37.973 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up dump (0.4b47-3) ... 2024-01-15T18:03:38.067 INFO:teuthology.orchestra.run.smithi093.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-01-15T18:03:38.132 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2024-01-15T18:03:38.275 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-01-15T18:03:38.376 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up tcl (8.6.11+1build2) ... 2024-01-15T18:03:38.502 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libjson-perl (4.04000-1) ... 2024-01-15T18:03:38.628 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-01-15T18:03:38.718 INFO:teuthology.orchestra.run.smithi093.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-01-15T18:03:38.813 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-01-15T18:03:38.939 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-01-15T18:03:39.115 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-01-15T18:03:39.241 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-01-15T18:03:39.359 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-01-15T18:03:39.527 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up postgresql-client (14+238) ... 2024-01-15T18:03:39.653 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-01-15T18:03:40.217 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:40.217 INFO:teuthology.orchestra.run.smithi093.stdout:The following network interfaces were found in /etc/network/interfaces 2024-01-15T18:03:40.217 INFO:teuthology.orchestra.run.smithi093.stdout:which means they are currently configured by ifupdown: 2024-01-15T18:03:40.217 INFO:teuthology.orchestra.run.smithi093.stdout:- enp3s0f1 2024-01-15T18:03:40.217 INFO:teuthology.orchestra.run.smithi093.stdout:If you want to manage those interfaces with NetworkManager instead 2024-01-15T18:03:40.217 INFO:teuthology.orchestra.run.smithi093.stdout:remove their configuration from /etc/network/interfaces. 2024-01-15T18:03:40.217 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:40.347 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-01-15T18:03:40.731 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-01-15T18:03:41.142 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-01-15T18:03:41.822 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-01-15T18:03:41.990 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-01-15T18:03:42.115 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-01-15T18:03:42.241 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up postgresql-common (238) ... 2024-01-15T18:03:43.134 INFO:teuthology.orchestra.run.smithi093.stdout:Adding user postgres to group ssl-cert 2024-01-15T18:03:43.399 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:43.399 INFO:teuthology.orchestra.run.smithi093.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-01-15T18:03:43.643 INFO:teuthology.orchestra.run.smithi093.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-01-15T18:03:43.643 INFO:teuthology.orchestra.run.smithi093.stdout:Removing obsolete dictionary files: 2024-01-15T18:03:44.333 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-01-15T18:03:44.792 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up postgresql-14 (14.10-0ubuntu0.22.04.1) ... 2024-01-15T18:03:45.135 INFO:teuthology.orchestra.run.smithi093.stdout:Creating new PostgreSQL cluster 14/main ... 2024-01-15T18:03:45.154 INFO:teuthology.orchestra.run.smithi093.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-01-15T18:03:45.203 INFO:teuthology.orchestra.run.smithi093.stdout:The files belonging to this database system will be owned by user "postgres". 2024-01-15T18:03:45.203 INFO:teuthology.orchestra.run.smithi093.stdout:This user must also own the server process. 2024-01-15T18:03:45.203 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:45.203 INFO:teuthology.orchestra.run.smithi093.stdout:The database cluster will be initialized with locale "en_US". 2024-01-15T18:03:45.203 INFO:teuthology.orchestra.run.smithi093.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-01-15T18:03:45.203 INFO:teuthology.orchestra.run.smithi093.stdout:The default text search configuration will be set to "english". 2024-01-15T18:03:45.203 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:45.203 INFO:teuthology.orchestra.run.smithi093.stdout:Data page checksums are disabled. 2024-01-15T18:03:45.204 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:45.204 INFO:teuthology.orchestra.run.smithi093.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-01-15T18:03:45.204 INFO:teuthology.orchestra.run.smithi093.stdout:creating subdirectories ... ok 2024-01-15T18:03:45.205 INFO:teuthology.orchestra.run.smithi093.stdout:selecting dynamic shared memory implementation ... posix 2024-01-15T18:03:45.227 INFO:teuthology.orchestra.run.smithi093.stdout:selecting default max_connections ... 100 2024-01-15T18:03:45.261 INFO:teuthology.orchestra.run.smithi093.stdout:selecting default shared_buffers ... 128MB 2024-01-15T18:03:45.275 INFO:teuthology.orchestra.run.smithi093.stdout:selecting default time zone ... Etc/UTC 2024-01-15T18:03:45.276 INFO:teuthology.orchestra.run.smithi093.stdout:creating configuration files ... ok 2024-01-15T18:03:45.439 INFO:teuthology.orchestra.run.smithi093.stdout:running bootstrap script ... ok 2024-01-15T18:03:46.060 INFO:teuthology.orchestra.run.smithi093.stdout:performing post-bootstrap initialization ... ok 2024-01-15T18:03:46.554 INFO:teuthology.orchestra.run.smithi093.stdout:syncing data to disk ... ok 2024-01-15T18:03:47.061 INFO:teuthology.orchestra.run.smithi093.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-01-15T18:03:50.157 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-01-15T18:03:50.283 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up postgresql-contrib (14+238) ... 2024-01-15T18:03:50.417 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up postgresql (14+238) ... 2024-01-15T18:03:50.543 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-01-15T18:03:50.669 INFO:teuthology.orchestra.run.smithi093.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2024-01-15T18:03:50.936 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-01-15T18:03:50.937 INFO:teuthology.orchestra.run.smithi093.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-01-15T18:03:51.663 INFO:teuthology.orchestra.run.smithi093.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-01-15T18:03:52.011 INFO:teuthology.orchestra.run.smithi093.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-01-15T18:03:52.166 INFO:teuthology.orchestra.run.smithi093.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-01-15T18:03:53.644 INFO:teuthology.orchestra.run.smithi093.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-01-15T18:03:53.745 INFO:teuthology.orchestra.run.smithi093.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-01-15T18:03:54.258 INFO:teuthology.orchestra.run.smithi093.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-01-15T18:03:54.266 DEBUG:teuthology.parallel:result is None 2024-01-15T18:03:54.267 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=95c90f413c4bb65e3e56909855f2308fd9bda05b 2024-01-15T18:03:54.530 DEBUG:teuthology.orchestra.run.smithi027:> dpkg-query -W -f '${Version}' ceph 2024-01-15T18:03:54.555 INFO:teuthology.orchestra.run.smithi027.stdout:18.0.0-7629-g95c90f41-1jammy 2024-01-15T18:03:54.557 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7629-g95c90f41-1jammy 2024-01-15T18:03:54.557 INFO:teuthology.task.install:The correct ceph version 18.0.0-7629-g95c90f41-1jammy is installed. 2024-01-15T18:03:54.559 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=95c90f413c4bb65e3e56909855f2308fd9bda05b 2024-01-15T18:03:54.797 DEBUG:teuthology.orchestra.run.smithi093:> dpkg-query -W -f '${Version}' ceph 2024-01-15T18:03:54.820 INFO:teuthology.orchestra.run.smithi093.stdout:18.0.0-7629-g95c90f41-1jammy 2024-01-15T18:03:54.820 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7629-g95c90f41-1jammy 2024-01-15T18:03:54.820 INFO:teuthology.task.install:The correct ceph version 18.0.0-7629-g95c90f41-1jammy is installed. 2024-01-15T18:03:54.822 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-15T18:03:54.822 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:03:54.822 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-15T18:03:54.845 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:03:54.845 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-15T18:03:54.880 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-15T18:03:54.880 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:03:54.881 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/daemon-helper 2024-01-15T18:03:54.909 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-15T18:03:54.973 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:03:54.973 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd of=/usr/bin/daemon-helper 2024-01-15T18:03:54.993 DEBUG:teuthology.orchestra.run.smithi093:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-15T18:03:55.059 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-15T18:03:55.059 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:03:55.059 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-15T18:03:55.081 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-15T18:03:55.149 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:03:55.149 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-15T18:03:55.169 DEBUG:teuthology.orchestra.run.smithi093:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-15T18:03:55.235 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-15T18:03:55.235 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:03:55.235 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/stdin-killer 2024-01-15T18:03:55.257 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-15T18:03:55.325 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:03:55.325 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd of=/usr/bin/stdin-killer 2024-01-15T18:03:55.344 DEBUG:teuthology.orchestra.run.smithi093:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-15T18:03:55.406 INFO:teuthology.run_tasks:Running task ceph... 2024-01-15T18:03:55.498 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-15T18:03:55.498 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 777 /var/log/ceph 2024-01-15T18:03:55.501 DEBUG:teuthology.orchestra.run.smithi093:> sudo chmod 777 /var/log/ceph 2024-01-15T18:03:55.520 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-15T18:03:55.520 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-15T18:03:55.567 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-15T18:03:55.585 INFO:tasks.ceph:Creating extra log directories... 2024-01-15T18:03:55.586 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-15T18:03:55.635 DEBUG:teuthology.orchestra.run.smithi093:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-15T18:03:55.660 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-01-15T18:03:55.660 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-01-15T18:03:55.660 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-01-15T18:03:55.660 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:03:55.660 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-01-15T18:03:55.704 DEBUG:teuthology.orchestra.run.smithi027:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-01-15T18:03:55.798 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:03:55.798 DEBUG:teuthology.orchestra.run.smithi093:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-01-15T18:03:55.806 DEBUG:teuthology.orchestra.run.smithi093:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-01-15T18:03:55.898 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-15T18:03:55.898 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-15T18:03:55.898 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2024-01-15_17:34:44-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517095', 'branch': 'rishabh-clone-stats', 'description': 'fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}}', 'email': None, 'first_in_suite': False, 'job_id': '7517095', 'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'rishabh-2024-01-15_17:34:44-fs:volumes-rishabh-clone-stats-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'rishabh-clone-stats'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'rishabh-clone-stats', 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 50, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 1760, 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b', 'sleep_before_teardown': 0, 'suite': 'fs:volumes', 'suite_branch': 'rishabh-clone-stats', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi027.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAI2qU5KCRsR9Mi+3w/gmSQ0CY1fK8zX3xasJHG5TMacJFxEA5yy5YXPW72BFKwHcZiTD1GhG0NIszB++jnF8Zg=', 'smithi093.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDogDNdQ6bvThoHNs52dfNHRHe7uqvJnSoKggoh25Kct9v5aHtzSGiaGbpXRFTsuZc7w2kiZGssL/Ml4pOFOSnw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_volumes.TestCloneProgress']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'cd45576300487d997e5a85abed65500b9f5d143b', 'timestamp': '2024-01-15_17:34:44', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-15T18:03:55.899 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-15T18:03:55.903 DEBUG:teuthology.orchestra.run.smithi093:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-15T18:03:55.912 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m0777 -- /var/run/ceph 2024-01-15T18:03:55.954 DEBUG:teuthology.orchestra.run.smithi093:> sudo install -d -m0777 -- /var/run/ceph 2024-01-15T18:03:55.975 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:03:55.975 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2024-01-15T18:03:56.029 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-15T18:03:56.029 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_1 2024-01-15T18:03:56.081 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-15T18:03:56.081 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T18:03:56.081 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 890 Links: 1 2024-01-15T18:03:56.081 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T18:03:56.081 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-01-15 17:56:57.622503181 +0000 2024-01-15T18:03:56.081 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-01-15 17:56:57.350505610 +0000 2024-01-15T18:03:56.082 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-01-15 17:56:57.350505610 +0000 2024-01-15T18:03:56.082 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-01-15 17:56:57.350505610 +0000 2024-01-15T18:03:56.082 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-15T18:03:56.143 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-01-15T18:03:56.143 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-01-15T18:03:56.143 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000402756 s, 1.3 MB/s 2024-01-15T18:03:56.145 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-15T18:03:56.199 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_2 2024-01-15T18:03:56.249 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-15T18:03:56.249 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T18:03:56.249 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 908 Links: 1 2024-01-15T18:03:56.249 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T18:03:56.249 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-01-15 17:56:58.150498464 +0000 2024-01-15T18:03:56.249 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-01-15 17:56:57.890500786 +0000 2024-01-15T18:03:56.249 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-01-15 17:56:57.890500786 +0000 2024-01-15T18:03:56.249 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-01-15 17:56:57.890500786 +0000 2024-01-15T18:03:56.249 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-15T18:03:56.311 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-01-15T18:03:56.312 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-01-15T18:03:56.312 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000642391 s, 797 kB/s 2024-01-15T18:03:56.314 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-15T18:03:56.366 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_3 2024-01-15T18:03:56.416 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-15T18:03:56.417 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T18:03:56.417 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 923 Links: 1 2024-01-15T18:03:56.417 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T18:03:56.417 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-01-15 17:56:58.678493749 +0000 2024-01-15T18:03:56.417 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-01-15 17:56:58.418496071 +0000 2024-01-15T18:03:56.417 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-01-15 17:56:58.418496071 +0000 2024-01-15T18:03:56.417 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-01-15 17:56:58.418496071 +0000 2024-01-15T18:03:56.417 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-15T18:03:56.479 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-01-15T18:03:56.479 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-01-15T18:03:56.479 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000368404 s, 1.4 MB/s 2024-01-15T18:03:56.481 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-15T18:03:56.538 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_4 2024-01-15T18:03:56.589 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-15T18:03:56.589 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T18:03:56.589 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 941 Links: 1 2024-01-15T18:03:56.589 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T18:03:56.589 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-01-15 17:56:59.178489283 +0000 2024-01-15T18:03:56.590 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-01-15 17:56:58.926491533 +0000 2024-01-15T18:03:56.590 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-01-15 17:56:58.926491533 +0000 2024-01-15T18:03:56.590 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-01-15 17:56:58.926491533 +0000 2024-01-15T18:03:56.590 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-15T18:03:56.651 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-01-15T18:03:56.651 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-01-15T18:03:56.651 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000387779 s, 1.3 MB/s 2024-01-15T18:03:56.653 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-15T18:03:56.707 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-15T18:03:56.707 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:03:56.707 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/scratch_devs of=/dev/stdout 2024-01-15T18:03:56.716 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-15T18:03:56.717 DEBUG:teuthology.orchestra.run.smithi093:> stat /dev/vg_nvme/lv_1 2024-01-15T18:03:56.769 INFO:teuthology.orchestra.run.smithi093.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-15T18:03:56.769 INFO:teuthology.orchestra.run.smithi093.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T18:03:56.769 INFO:teuthology.orchestra.run.smithi093.stdout:Device: 5h/5d Inode: 899 Links: 1 2024-01-15T18:03:56.769 INFO:teuthology.orchestra.run.smithi093.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T18:03:56.769 INFO:teuthology.orchestra.run.smithi093.stdout:Access: 2024-01-15 17:56:52.859661669 +0000 2024-01-15T18:03:56.769 INFO:teuthology.orchestra.run.smithi093.stdout:Modify: 2024-01-15 17:56:52.607663919 +0000 2024-01-15T18:03:56.769 INFO:teuthology.orchestra.run.smithi093.stdout:Change: 2024-01-15 17:56:52.607663919 +0000 2024-01-15T18:03:56.770 INFO:teuthology.orchestra.run.smithi093.stdout: Birth: 2024-01-15 17:56:52.607663919 +0000 2024-01-15T18:03:56.770 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-15T18:03:56.830 INFO:teuthology.orchestra.run.smithi093.stderr:1+0 records in 2024-01-15T18:03:56.830 INFO:teuthology.orchestra.run.smithi093.stderr:1+0 records out 2024-01-15T18:03:56.830 INFO:teuthology.orchestra.run.smithi093.stderr:512 bytes copied, 0.000375606 s, 1.4 MB/s 2024-01-15T18:03:56.832 DEBUG:teuthology.orchestra.run.smithi093:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-15T18:03:56.890 DEBUG:teuthology.orchestra.run.smithi093:> stat /dev/vg_nvme/lv_2 2024-01-15T18:03:56.941 INFO:teuthology.orchestra.run.smithi093.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-15T18:03:56.941 INFO:teuthology.orchestra.run.smithi093.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T18:03:56.941 INFO:teuthology.orchestra.run.smithi093.stdout:Device: 5h/5d Inode: 914 Links: 1 2024-01-15T18:03:56.941 INFO:teuthology.orchestra.run.smithi093.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T18:03:56.941 INFO:teuthology.orchestra.run.smithi093.stdout:Access: 2024-01-15 17:56:53.371657096 +0000 2024-01-15T18:03:56.941 INFO:teuthology.orchestra.run.smithi093.stdout:Modify: 2024-01-15 17:56:53.119659347 +0000 2024-01-15T18:03:56.941 INFO:teuthology.orchestra.run.smithi093.stdout:Change: 2024-01-15 17:56:53.119659347 +0000 2024-01-15T18:03:56.941 INFO:teuthology.orchestra.run.smithi093.stdout: Birth: 2024-01-15 17:56:53.119659347 +0000 2024-01-15T18:03:56.942 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-15T18:03:57.002 INFO:teuthology.orchestra.run.smithi093.stderr:1+0 records in 2024-01-15T18:03:57.002 INFO:teuthology.orchestra.run.smithi093.stderr:1+0 records out 2024-01-15T18:03:57.002 INFO:teuthology.orchestra.run.smithi093.stderr:512 bytes copied, 0.000362356 s, 1.4 MB/s 2024-01-15T18:03:57.004 DEBUG:teuthology.orchestra.run.smithi093:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-15T18:03:57.058 DEBUG:teuthology.orchestra.run.smithi093:> stat /dev/vg_nvme/lv_3 2024-01-15T18:03:57.109 INFO:teuthology.orchestra.run.smithi093.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-15T18:03:57.109 INFO:teuthology.orchestra.run.smithi093.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T18:03:57.109 INFO:teuthology.orchestra.run.smithi093.stdout:Device: 5h/5d Inode: 933 Links: 1 2024-01-15T18:03:57.109 INFO:teuthology.orchestra.run.smithi093.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T18:03:57.109 INFO:teuthology.orchestra.run.smithi093.stdout:Access: 2024-01-15 17:56:53.915652236 +0000 2024-01-15T18:03:57.110 INFO:teuthology.orchestra.run.smithi093.stdout:Modify: 2024-01-15 17:56:53.667654451 +0000 2024-01-15T18:03:57.110 INFO:teuthology.orchestra.run.smithi093.stdout:Change: 2024-01-15 17:56:53.667654451 +0000 2024-01-15T18:03:57.110 INFO:teuthology.orchestra.run.smithi093.stdout: Birth: 2024-01-15 17:56:53.667654451 +0000 2024-01-15T18:03:57.110 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-15T18:03:57.171 INFO:teuthology.orchestra.run.smithi093.stderr:1+0 records in 2024-01-15T18:03:57.171 INFO:teuthology.orchestra.run.smithi093.stderr:1+0 records out 2024-01-15T18:03:57.171 INFO:teuthology.orchestra.run.smithi093.stderr:512 bytes copied, 0.000360091 s, 1.4 MB/s 2024-01-15T18:03:57.173 DEBUG:teuthology.orchestra.run.smithi093:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-15T18:03:57.230 DEBUG:teuthology.orchestra.run.smithi093:> stat /dev/vg_nvme/lv_4 2024-01-15T18:03:57.285 INFO:teuthology.orchestra.run.smithi093.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-15T18:03:57.285 INFO:teuthology.orchestra.run.smithi093.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T18:03:57.285 INFO:teuthology.orchestra.run.smithi093.stdout:Device: 5h/5d Inode: 950 Links: 1 2024-01-15T18:03:57.285 INFO:teuthology.orchestra.run.smithi093.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T18:03:57.285 INFO:teuthology.orchestra.run.smithi093.stdout:Access: 2024-01-15 17:56:54.459647377 +0000 2024-01-15T18:03:57.286 INFO:teuthology.orchestra.run.smithi093.stdout:Modify: 2024-01-15 17:56:54.207649627 +0000 2024-01-15T18:03:57.286 INFO:teuthology.orchestra.run.smithi093.stdout:Change: 2024-01-15 17:56:54.207649627 +0000 2024-01-15T18:03:57.286 INFO:teuthology.orchestra.run.smithi093.stdout: Birth: 2024-01-15 17:56:54.207649627 +0000 2024-01-15T18:03:57.286 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-15T18:03:57.346 INFO:teuthology.orchestra.run.smithi093.stderr:1+0 records in 2024-01-15T18:03:57.347 INFO:teuthology.orchestra.run.smithi093.stderr:1+0 records out 2024-01-15T18:03:57.347 INFO:teuthology.orchestra.run.smithi093.stderr:512 bytes copied, 0.000375674 s, 1.4 MB/s 2024-01-15T18:03:57.348 DEBUG:teuthology.orchestra.run.smithi093:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-15T18:03:57.402 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-15T18:03:57.402 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi027.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi093.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-15T18:03:57.402 INFO:tasks.ceph:Generating config... 2024-01-15T18:03:57.404 INFO:tasks.ceph:[client] client mount timeout = 600 2024-01-15T18:03:57.404 INFO:tasks.ceph:[client] debug client = 20 2024-01-15T18:03:57.404 INFO:tasks.ceph:[client] debug ms = 1 2024-01-15T18:03:57.404 INFO:tasks.ceph:[client] pid file = 2024-01-15T18:03:57.404 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-01-15T18:03:57.405 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-01-15T18:03:57.405 INFO:tasks.ceph:[mds] debug mds = 20 2024-01-15T18:03:57.405 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-01-15T18:03:57.405 INFO:tasks.ceph:[mds] debug ms = 1 2024-01-15T18:03:57.405 INFO:tasks.ceph:[mds] mds debug frag = True 2024-01-15T18:03:57.405 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-01-15T18:03:57.405 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-01-15T18:03:57.405 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-01-15T18:03:57.405 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-01-15T18:03:57.405 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-01-15T18:03:57.406 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-01-15T18:03:57.406 INFO:tasks.ceph:[mgr] debug client = 20 2024-01-15T18:03:57.406 INFO:tasks.ceph:[mgr] debug finisher = 20 2024-01-15T18:03:57.406 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-15T18:03:57.406 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-15T18:03:57.406 INFO:tasks.ceph:[mon] debug mon = 20 2024-01-15T18:03:57.406 INFO:tasks.ceph:[mon] debug ms = 1 2024-01-15T18:03:57.406 INFO:tasks.ceph:[mon] debug paxos = 20 2024-01-15T18:03:57.406 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-01-15T18:03:57.406 INFO:tasks.ceph:[osd] bdev async discard = True 2024-01-15T18:03:57.406 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-01-15T18:03:57.407 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-01-15T18:03:57.407 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-01-15T18:03:57.407 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-01-15T18:03:57.407 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-01-15T18:03:57.407 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-01-15T18:03:57.407 INFO:tasks.ceph:[osd] debug ms = 1 2024-01-15T18:03:57.407 INFO:tasks.ceph:[osd] debug osd = 20 2024-01-15T18:03:57.407 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-01-15T18:03:57.407 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-01-15T18:03:57.407 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-01-15T18:03:57.408 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-01-15T18:03:57.408 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-01-15T18:03:57.408 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-01-15T18:03:57.408 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-01-15T18:03:57.408 INFO:tasks.ceph:Setting up mon.a... 2024-01-15T18:03:57.408 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-15T18:03:57.456 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/ceph.keyring 2024-01-15T18:03:57.472 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-15T18:03:57.524 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-15T18:03:57.545 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.27'), ('mon.b', '172.21.15.93'), ('mon.c', '[v2:172.21.15.93:3301,v1:172.21.15.93:6790]')] 2024-01-15T18:03:57.545 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.27,172.21.15.93,[v2:172.21.15.93:3301,v1:172.21.15.93:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-01-15T18:03:57.546 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:03:57.547 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-15T18:03:57.600 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.27 --add b 172.21.15.93 --addv c '[v2:172.21.15.93:3301,v1:172.21.15.93:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-01-15T18:03:57.692 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-15T18:03:57.692 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: generated fsid fce09647-6947-4a73-bd3d-e3d154552087 2024-01-15T18:03:57.692 INFO:teuthology.orchestra.run.smithi027.stdout:setting min_mon_release = pacific 2024-01-15T18:03:57.692 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 0 2024-01-15T18:03:57.693 INFO:teuthology.orchestra.run.smithi027.stdout:fsid fce09647-6947-4a73-bd3d-e3d154552087 2024-01-15T18:03:57.693 INFO:teuthology.orchestra.run.smithi027.stdout:last_changed 2024-01-15T18:03:57.692510+0000 2024-01-15T18:03:57.693 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-01-15T18:03:57.692510+0000 2024-01-15T18:03:57.693 INFO:teuthology.orchestra.run.smithi027.stdout:min_mon_release 16 (pacific) 2024-01-15T18:03:57.693 INFO:teuthology.orchestra.run.smithi027.stdout:election_strategy: 1 2024-01-15T18:03:57.693 INFO:teuthology.orchestra.run.smithi027.stdout:0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-01-15T18:03:57.693 INFO:teuthology.orchestra.run.smithi027.stdout:1: [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] mon.b 2024-01-15T18:03:57.693 INFO:teuthology.orchestra.run.smithi027.stdout:2: [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] mon.c 2024-01-15T18:03:57.693 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-01-15T18:03:57.695 DEBUG:teuthology.orchestra.run.smithi027:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-15T18:03:57.704 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID fce09647-6947-4a73-bd3d-e3d154552087... 2024-01-15T18:03:57.705 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-15T18:03:57.751 DEBUG:teuthology.orchestra.run.smithi093:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-15T18:03:57.791 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2024-01-15T18:03:57.791 INFO:teuthology.orchestra.run.smithi027.stdout: chdir = "" 2024-01-15T18:03:57.791 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-15T18:03:57.791 INFO:teuthology.orchestra.run.smithi027.stdout: auth supported = cephx 2024-01-15T18:03:57.792 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.792 INFO:teuthology.orchestra.run.smithi027.stdout: filestore xattr use omap = true 2024-01-15T18:03:57.792 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.792 INFO:teuthology.orchestra.run.smithi027.stdout: mon clock drift allowed = 1.000 2024-01-15T18:03:57.792 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.792 INFO:teuthology.orchestra.run.smithi027.stdout: osd crush chooseleaf type = 0 2024-01-15T18:03:57.792 INFO:teuthology.orchestra.run.smithi027.stdout: auth debug = true 2024-01-15T18:03:57.793 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.793 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on old message = true 2024-01-15T18:03:57.793 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on bug = true 2024-01-15T18:03:57.793 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.793 INFO:teuthology.orchestra.run.smithi027.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-15T18:03:57.793 INFO:teuthology.orchestra.run.smithi027.stdout: mon pg warn max object skew = 0 2024-01-15T18:03:57.793 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.793 INFO:teuthology.orchestra.run.smithi027.stdout: # disable pg_autoscaler by default for new pools 2024-01-15T18:03:57.793 INFO:teuthology.orchestra.run.smithi027.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-15T18:03:57.793 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.794 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default size = 2 2024-01-15T18:03:57.794 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.794 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow primary affinity = true 2024-01-15T18:03:57.794 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow pg remap = true 2024-01-15T18:03:57.794 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on legacy crush tunables = false 2024-01-15T18:03:57.794 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on crush straw calc version zero = false 2024-01-15T18:03:57.794 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on no sortbitwise = false 2024-01-15T18:03:57.794 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on osd down out interval zero = false 2024-01-15T18:03:57.794 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on too few osds = false 2024-01-15T18:03:57.794 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-15T18:03:57.795 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-15T18:03:57.795 INFO:teuthology.orchestra.run.smithi027.stdout: mon_allow_pool_size_one = true 2024-01-15T18:03:57.795 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.795 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-15T18:03:57.795 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.795 INFO:teuthology.orchestra.run.smithi027.stdout: osd default data pool replay window = 5 2024-01-15T18:03:57.795 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.795 INFO:teuthology.orchestra.run.smithi027.stdout: mon allow pool delete = true 2024-01-15T18:03:57.795 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.795 INFO:teuthology.orchestra.run.smithi027.stdout: mon cluster log file level = debug 2024-01-15T18:03:57.796 INFO:teuthology.orchestra.run.smithi027.stdout: debug asserts on shutdown = true 2024-01-15T18:03:57.796 INFO:teuthology.orchestra.run.smithi027.stdout: mon health detail to clog = false 2024-01-15T18:03:57.796 INFO:teuthology.orchestra.run.smithi027.stdout: mon host = "172.21.15.27,172.21.15.93,[v2:172.21.15.93:3301,v1:172.21.15.93:6790]" 2024-01-15T18:03:57.796 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = fce09647-6947-4a73-bd3d-e3d154552087 2024-01-15T18:03:57.796 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.796 INFO:teuthology.orchestra.run.smithi027.stdout:[osd] 2024-01-15T18:03:57.796 INFO:teuthology.orchestra.run.smithi027.stdout: osd journal size = 100 2024-01-15T18:03:57.796 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.796 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub load threshold = 5.0 2024-01-15T18:03:57.797 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub max interval = 600 2024-01-15T18:03:57.797 INFO:teuthology.orchestra.run.smithi027.stdout: osd mclock profile = high_recovery_ops 2024-01-15T18:03:57.797 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.797 INFO:teuthology.orchestra.run.smithi027.stdout: osd recover clone overlap = true 2024-01-15T18:03:57.797 INFO:teuthology.orchestra.run.smithi027.stdout: osd recovery max chunk = 1048576 2024-01-15T18:03:57.797 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.797 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug shutdown = true 2024-01-15T18:03:57.797 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug op order = true 2024-01-15T18:03:57.797 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug verify stray on activate = true 2024-01-15T18:03:57.797 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug trim objects = true 2024-01-15T18:03:57.797 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.798 INFO:teuthology.orchestra.run.smithi027.stdout: osd open classes on start = true 2024-01-15T18:03:57.798 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug pg log writeout = true 2024-01-15T18:03:57.798 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.798 INFO:teuthology.orchestra.run.smithi027.stdout: osd deep scrub update digest min age = 30 2024-01-15T18:03:57.798 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.798 INFO:teuthology.orchestra.run.smithi027.stdout: osd map max advance = 10 2024-01-15T18:03:57.798 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.798 INFO:teuthology.orchestra.run.smithi027.stdout: journal zero on create = true 2024-01-15T18:03:57.798 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.798 INFO:teuthology.orchestra.run.smithi027.stdout: filestore ondisk finisher threads = 3 2024-01-15T18:03:57.799 INFO:teuthology.orchestra.run.smithi027.stdout: filestore apply finisher threads = 3 2024-01-15T18:03:57.799 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.799 INFO:teuthology.orchestra.run.smithi027.stdout: bdev debug aio = true 2024-01-15T18:03:57.799 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug misdirected ops = true 2024-01-15T18:03:57.799 INFO:teuthology.orchestra.run.smithi027.stdout: bdev async discard = True 2024-01-15T18:03:57.799 INFO:teuthology.orchestra.run.smithi027.stdout: bdev enable discard = True 2024-01-15T18:03:57.799 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore allocator = bitmap 2024-01-15T18:03:57.799 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore block size = 96636764160 2024-01-15T18:03:57.799 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore fsck on mount = True 2024-01-15T18:03:57.799 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluefs = 1/20 2024-01-15T18:03:57.800 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluestore = 1/20 2024-01-15T18:03:57.800 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-01-15T18:03:57.800 INFO:teuthology.orchestra.run.smithi027.stdout: debug osd = 20 2024-01-15T18:03:57.800 INFO:teuthology.orchestra.run.smithi027.stdout: debug rocksdb = 4/10 2024-01-15T18:03:57.800 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd backfillfull_ratio = 0.85 2024-01-15T18:03:57.800 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd full ratio = 0.9 2024-01-15T18:03:57.800 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd nearfull ratio = 0.8 2024-01-15T18:03:57.800 INFO:teuthology.orchestra.run.smithi027.stdout: osd failsafe full ratio = 0.95 2024-01-15T18:03:57.800 INFO:teuthology.orchestra.run.smithi027.stdout: osd objectstore = bluestore 2024-01-15T18:03:57.801 INFO:teuthology.orchestra.run.smithi027.stdout: osd op complaint time = 180 2024-01-15T18:03:57.801 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.801 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr] 2024-01-15T18:03:57.801 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-01-15T18:03:57.801 INFO:teuthology.orchestra.run.smithi027.stdout: debug mgr = 20 2024-01-15T18:03:57.801 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-01-15T18:03:57.801 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-01-15T18:03:57.801 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-01-15T18:03:57.801 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-01-15T18:03:57.801 INFO:teuthology.orchestra.run.smithi027.stdout: mgr/telemetry/nag = false 2024-01-15T18:03:57.802 INFO:teuthology.orchestra.run.smithi027.stdout: debug client = 20 2024-01-15T18:03:57.802 INFO:teuthology.orchestra.run.smithi027.stdout: debug finisher = 20 2024-01-15T18:03:57.802 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.802 INFO:teuthology.orchestra.run.smithi027.stdout:[mon] 2024-01-15T18:03:57.802 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-01-15T18:03:57.802 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-01-15T18:03:57.802 INFO:teuthology.orchestra.run.smithi027.stdout: debug paxos = 20 2024-01-15T18:03:57.802 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-01-15T18:03:57.802 INFO:teuthology.orchestra.run.smithi027.stdout: mon data avail warn = 5 2024-01-15T18:03:57.802 INFO:teuthology.orchestra.run.smithi027.stdout: mon mgr mkfs grace = 240 2024-01-15T18:03:57.803 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-01-15T18:03:57.803 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd reporter subtree level = osd 2024-01-15T18:03:57.803 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd prime pg temp = true 2024-01-15T18:03:57.803 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-01-15T18:03:57.803 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.803 INFO:teuthology.orchestra.run.smithi027.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-15T18:03:57.803 INFO:teuthology.orchestra.run.smithi027.stdout: auth mon ticket ttl = 660 # 11m 2024-01-15T18:03:57.803 INFO:teuthology.orchestra.run.smithi027.stdout: auth service ticket ttl = 240 # 4m 2024-01-15T18:03:57.803 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.803 INFO:teuthology.orchestra.run.smithi027.stdout: # don't complain about insecure global_id in the test suite 2024-01-15T18:03:57.803 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-15T18:03:57.804 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-15T18:03:57.804 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.804 INFO:teuthology.orchestra.run.smithi027.stdout: # 1m isn't quite enough 2024-01-15T18:03:57.804 INFO:teuthology.orchestra.run.smithi027.stdout: mon_down_mkfs_grace = 2m 2024-01-15T18:03:57.804 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.804 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_filestore_osds = false 2024-01-15T18:03:57.804 INFO:teuthology.orchestra.run.smithi027.stdout: mon op complaint time = 120 2024-01-15T18:03:57.804 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:03:57.804 INFO:teuthology.orchestra.run.smithi027.stdout:[client] 2024-01-15T18:03:57.804 INFO:teuthology.orchestra.run.smithi027.stdout: rgw cache enabled = true 2024-01-15T18:03:57.805 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable ops log = true 2024-01-15T18:03:57.805 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable usage log = true 2024-01-15T18:03:57.805 INFO:teuthology.orchestra.run.smithi027.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-15T18:03:57.805 INFO:teuthology.orchestra.run.smithi027.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-15T18:03:57.805 INFO:teuthology.orchestra.run.smithi027.stdout: client mount timeout = 600 2024-01-15T18:03:57.805 INFO:teuthology.orchestra.run.smithi027.stdout: debug client = 20 2024-01-15T18:03:57.805 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-01-15T18:03:57.805 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = "" 2024-01-15T18:03:57.805 INFO:teuthology.orchestra.run.smithi027.stdout: rados mon op timeout = 900 2024-01-15T18:03:57.805 INFO:teuthology.orchestra.run.smithi027.stdout: rados osd op timeout = 900 2024-01-15T18:03:57.805 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.a] 2024-01-15T18:03:57.806 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.b] 2024-01-15T18:03:57.806 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.c] 2024-01-15T18:03:57.806 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.a] 2024-01-15T18:03:57.806 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.c] 2024-01-15T18:03:57.806 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.b] 2024-01-15T18:03:57.806 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.d] 2024-01-15T18:03:57.806 INFO:teuthology.orchestra.run.smithi027.stdout:[mds] 2024-01-15T18:03:57.806 INFO:teuthology.orchestra.run.smithi027.stdout: debug mds = 20 2024-01-15T18:03:57.806 INFO:teuthology.orchestra.run.smithi027.stdout: debug mds balancer = 20 2024-01-15T18:03:57.806 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-01-15T18:03:57.806 INFO:teuthology.orchestra.run.smithi027.stdout: mds debug frag = True 2024-01-15T18:03:57.806 INFO:teuthology.orchestra.run.smithi027.stdout: mds debug scatterstat = True 2024-01-15T18:03:57.807 INFO:teuthology.orchestra.run.smithi027.stdout: mds op complaint time = 180 2024-01-15T18:03:57.807 INFO:teuthology.orchestra.run.smithi027.stdout: mds verify scatter = True 2024-01-15T18:03:57.807 INFO:teuthology.orchestra.run.smithi027.stdout: osd op complaint time = 180 2024-01-15T18:03:57.807 INFO:teuthology.orchestra.run.smithi027.stdout: rados mon op timeout = 900 2024-01-15T18:03:57.807 INFO:teuthology.orchestra.run.smithi027.stdout: rados osd op timeout = 900 2024-01-15T18:03:57.808 INFO:teuthology.orchestra.run.smithi093.stdout:[global] 2024-01-15T18:03:57.808 INFO:teuthology.orchestra.run.smithi093.stdout: chdir = "" 2024-01-15T18:03:57.808 INFO:teuthology.orchestra.run.smithi093.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-15T18:03:57.808 INFO:teuthology.orchestra.run.smithi093.stdout: auth supported = cephx 2024-01-15T18:03:57.808 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.808 INFO:teuthology.orchestra.run.smithi093.stdout: filestore xattr use omap = true 2024-01-15T18:03:57.808 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.808 INFO:teuthology.orchestra.run.smithi093.stdout: mon clock drift allowed = 1.000 2024-01-15T18:03:57.808 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.808 INFO:teuthology.orchestra.run.smithi093.stdout: osd crush chooseleaf type = 0 2024-01-15T18:03:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: auth debug = true 2024-01-15T18:03:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: ms die on old message = true 2024-01-15T18:03:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: ms die on bug = true 2024-01-15T18:03:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-15T18:03:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: mon pg warn max object skew = 0 2024-01-15T18:03:57.809 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.810 INFO:teuthology.orchestra.run.smithi093.stdout: # disable pg_autoscaler by default for new pools 2024-01-15T18:03:57.810 INFO:teuthology.orchestra.run.smithi093.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-15T18:03:57.810 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.810 INFO:teuthology.orchestra.run.smithi093.stdout: osd pool default size = 2 2024-01-15T18:03:57.810 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.810 INFO:teuthology.orchestra.run.smithi093.stdout: mon osd allow primary affinity = true 2024-01-15T18:03:57.810 INFO:teuthology.orchestra.run.smithi093.stdout: mon osd allow pg remap = true 2024-01-15T18:03:57.810 INFO:teuthology.orchestra.run.smithi093.stdout: mon warn on legacy crush tunables = false 2024-01-15T18:03:57.810 INFO:teuthology.orchestra.run.smithi093.stdout: mon warn on crush straw calc version zero = false 2024-01-15T18:03:57.810 INFO:teuthology.orchestra.run.smithi093.stdout: mon warn on no sortbitwise = false 2024-01-15T18:03:57.810 INFO:teuthology.orchestra.run.smithi093.stdout: mon warn on osd down out interval zero = false 2024-01-15T18:03:57.811 INFO:teuthology.orchestra.run.smithi093.stdout: mon warn on too few osds = false 2024-01-15T18:03:57.811 INFO:teuthology.orchestra.run.smithi093.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-15T18:03:57.811 INFO:teuthology.orchestra.run.smithi093.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-15T18:03:57.811 INFO:teuthology.orchestra.run.smithi093.stdout: mon_allow_pool_size_one = true 2024-01-15T18:03:57.811 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.811 INFO:teuthology.orchestra.run.smithi093.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-15T18:03:57.811 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.811 INFO:teuthology.orchestra.run.smithi093.stdout: osd default data pool replay window = 5 2024-01-15T18:03:57.811 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.811 INFO:teuthology.orchestra.run.smithi093.stdout: mon allow pool delete = true 2024-01-15T18:03:57.811 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.812 INFO:teuthology.orchestra.run.smithi093.stdout: mon cluster log file level = debug 2024-01-15T18:03:57.812 INFO:teuthology.orchestra.run.smithi093.stdout: debug asserts on shutdown = true 2024-01-15T18:03:57.812 INFO:teuthology.orchestra.run.smithi093.stdout: mon health detail to clog = false 2024-01-15T18:03:57.812 INFO:teuthology.orchestra.run.smithi093.stdout: mon host = "172.21.15.27,172.21.15.93,[v2:172.21.15.93:3301,v1:172.21.15.93:6790]" 2024-01-15T18:03:57.812 INFO:teuthology.orchestra.run.smithi093.stdout: fsid = fce09647-6947-4a73-bd3d-e3d154552087 2024-01-15T18:03:57.812 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.812 INFO:teuthology.orchestra.run.smithi093.stdout:[osd] 2024-01-15T18:03:57.812 INFO:teuthology.orchestra.run.smithi093.stdout: osd journal size = 100 2024-01-15T18:03:57.812 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.812 INFO:teuthology.orchestra.run.smithi093.stdout: osd scrub load threshold = 5.0 2024-01-15T18:03:57.813 INFO:teuthology.orchestra.run.smithi093.stdout: osd scrub max interval = 600 2024-01-15T18:03:57.813 INFO:teuthology.orchestra.run.smithi093.stdout: osd mclock profile = high_recovery_ops 2024-01-15T18:03:57.813 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.813 INFO:teuthology.orchestra.run.smithi093.stdout: osd recover clone overlap = true 2024-01-15T18:03:57.813 INFO:teuthology.orchestra.run.smithi093.stdout: osd recovery max chunk = 1048576 2024-01-15T18:03:57.813 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.813 INFO:teuthology.orchestra.run.smithi093.stdout: osd debug shutdown = true 2024-01-15T18:03:57.813 INFO:teuthology.orchestra.run.smithi093.stdout: osd debug op order = true 2024-01-15T18:03:57.813 INFO:teuthology.orchestra.run.smithi093.stdout: osd debug verify stray on activate = true 2024-01-15T18:03:57.813 INFO:teuthology.orchestra.run.smithi093.stdout: osd debug trim objects = true 2024-01-15T18:03:57.814 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.814 INFO:teuthology.orchestra.run.smithi093.stdout: osd open classes on start = true 2024-01-15T18:03:57.814 INFO:teuthology.orchestra.run.smithi093.stdout: osd debug pg log writeout = true 2024-01-15T18:03:57.814 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.814 INFO:teuthology.orchestra.run.smithi093.stdout: osd deep scrub update digest min age = 30 2024-01-15T18:03:57.814 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.814 INFO:teuthology.orchestra.run.smithi093.stdout: osd map max advance = 10 2024-01-15T18:03:57.814 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.814 INFO:teuthology.orchestra.run.smithi093.stdout: journal zero on create = true 2024-01-15T18:03:57.814 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.814 INFO:teuthology.orchestra.run.smithi093.stdout: filestore ondisk finisher threads = 3 2024-01-15T18:03:57.815 INFO:teuthology.orchestra.run.smithi093.stdout: filestore apply finisher threads = 3 2024-01-15T18:03:57.815 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.815 INFO:teuthology.orchestra.run.smithi093.stdout: bdev debug aio = true 2024-01-15T18:03:57.815 INFO:teuthology.orchestra.run.smithi093.stdout: osd debug misdirected ops = true 2024-01-15T18:03:57.815 INFO:teuthology.orchestra.run.smithi093.stdout: bdev async discard = True 2024-01-15T18:03:57.815 INFO:teuthology.orchestra.run.smithi093.stdout: bdev enable discard = True 2024-01-15T18:03:57.815 INFO:teuthology.orchestra.run.smithi093.stdout: bluestore allocator = bitmap 2024-01-15T18:03:57.815 INFO:teuthology.orchestra.run.smithi093.stdout: bluestore block size = 96636764160 2024-01-15T18:03:57.815 INFO:teuthology.orchestra.run.smithi093.stdout: bluestore fsck on mount = True 2024-01-15T18:03:57.815 INFO:teuthology.orchestra.run.smithi093.stdout: debug bluefs = 1/20 2024-01-15T18:03:57.815 INFO:teuthology.orchestra.run.smithi093.stdout: debug bluestore = 1/20 2024-01-15T18:03:57.815 INFO:teuthology.orchestra.run.smithi093.stdout: debug ms = 1 2024-01-15T18:03:57.816 INFO:teuthology.orchestra.run.smithi093.stdout: debug osd = 20 2024-01-15T18:03:57.816 INFO:teuthology.orchestra.run.smithi093.stdout: debug rocksdb = 4/10 2024-01-15T18:03:57.816 INFO:teuthology.orchestra.run.smithi093.stdout: mon osd backfillfull_ratio = 0.85 2024-01-15T18:03:57.816 INFO:teuthology.orchestra.run.smithi093.stdout: mon osd full ratio = 0.9 2024-01-15T18:03:57.816 INFO:teuthology.orchestra.run.smithi093.stdout: mon osd nearfull ratio = 0.8 2024-01-15T18:03:57.816 INFO:teuthology.orchestra.run.smithi093.stdout: osd failsafe full ratio = 0.95 2024-01-15T18:03:57.816 INFO:teuthology.orchestra.run.smithi093.stdout: osd objectstore = bluestore 2024-01-15T18:03:57.816 INFO:teuthology.orchestra.run.smithi093.stdout: osd op complaint time = 180 2024-01-15T18:03:57.816 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.816 INFO:teuthology.orchestra.run.smithi093.stdout:[mgr] 2024-01-15T18:03:57.816 INFO:teuthology.orchestra.run.smithi093.stdout: debug ms = 1 2024-01-15T18:03:57.817 INFO:teuthology.orchestra.run.smithi093.stdout: debug mgr = 20 2024-01-15T18:03:57.817 INFO:teuthology.orchestra.run.smithi093.stdout: debug mon = 20 2024-01-15T18:03:57.817 INFO:teuthology.orchestra.run.smithi093.stdout: debug auth = 20 2024-01-15T18:03:57.817 INFO:teuthology.orchestra.run.smithi093.stdout: mon reweight min pgs per osd = 4 2024-01-15T18:03:57.817 INFO:teuthology.orchestra.run.smithi093.stdout: mon reweight min bytes per osd = 10 2024-01-15T18:03:57.817 INFO:teuthology.orchestra.run.smithi093.stdout: mgr/telemetry/nag = false 2024-01-15T18:03:57.817 INFO:teuthology.orchestra.run.smithi093.stdout: debug client = 20 2024-01-15T18:03:57.817 INFO:teuthology.orchestra.run.smithi093.stdout: debug finisher = 20 2024-01-15T18:03:57.817 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.817 INFO:teuthology.orchestra.run.smithi093.stdout:[mon] 2024-01-15T18:03:57.817 INFO:teuthology.orchestra.run.smithi093.stdout: debug ms = 1 2024-01-15T18:03:57.818 INFO:teuthology.orchestra.run.smithi093.stdout: debug mon = 20 2024-01-15T18:03:57.818 INFO:teuthology.orchestra.run.smithi093.stdout: debug paxos = 20 2024-01-15T18:03:57.818 INFO:teuthology.orchestra.run.smithi093.stdout: debug auth = 20 2024-01-15T18:03:57.818 INFO:teuthology.orchestra.run.smithi093.stdout: mon data avail warn = 5 2024-01-15T18:03:57.818 INFO:teuthology.orchestra.run.smithi093.stdout: mon mgr mkfs grace = 240 2024-01-15T18:03:57.818 INFO:teuthology.orchestra.run.smithi093.stdout: mon reweight min pgs per osd = 4 2024-01-15T18:03:57.818 INFO:teuthology.orchestra.run.smithi093.stdout: mon osd reporter subtree level = osd 2024-01-15T18:03:57.818 INFO:teuthology.orchestra.run.smithi093.stdout: mon osd prime pg temp = true 2024-01-15T18:03:57.818 INFO:teuthology.orchestra.run.smithi093.stdout: mon reweight min bytes per osd = 10 2024-01-15T18:03:57.818 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.818 INFO:teuthology.orchestra.run.smithi093.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-15T18:03:57.818 INFO:teuthology.orchestra.run.smithi093.stdout: auth mon ticket ttl = 660 # 11m 2024-01-15T18:03:57.819 INFO:teuthology.orchestra.run.smithi093.stdout: auth service ticket ttl = 240 # 4m 2024-01-15T18:03:57.819 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.819 INFO:teuthology.orchestra.run.smithi093.stdout: # don't complain about insecure global_id in the test suite 2024-01-15T18:03:57.819 INFO:teuthology.orchestra.run.smithi093.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-15T18:03:57.819 INFO:teuthology.orchestra.run.smithi093.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-15T18:03:57.819 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.819 INFO:teuthology.orchestra.run.smithi093.stdout: # 1m isn't quite enough 2024-01-15T18:03:57.819 INFO:teuthology.orchestra.run.smithi093.stdout: mon_down_mkfs_grace = 2m 2024-01-15T18:03:57.819 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.819 INFO:teuthology.orchestra.run.smithi093.stdout: mon_warn_on_filestore_osds = false 2024-01-15T18:03:57.819 INFO:teuthology.orchestra.run.smithi093.stdout: mon op complaint time = 120 2024-01-15T18:03:57.819 INFO:teuthology.orchestra.run.smithi093.stdout: 2024-01-15T18:03:57.820 INFO:teuthology.orchestra.run.smithi093.stdout:[client] 2024-01-15T18:03:57.820 INFO:teuthology.orchestra.run.smithi093.stdout: rgw cache enabled = true 2024-01-15T18:03:57.820 INFO:teuthology.orchestra.run.smithi093.stdout: rgw enable ops log = true 2024-01-15T18:03:57.820 INFO:teuthology.orchestra.run.smithi093.stdout: rgw enable usage log = true 2024-01-15T18:03:57.820 INFO:teuthology.orchestra.run.smithi093.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-15T18:03:57.820 INFO:teuthology.orchestra.run.smithi093.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-15T18:03:57.820 INFO:teuthology.orchestra.run.smithi093.stdout: client mount timeout = 600 2024-01-15T18:03:57.820 INFO:teuthology.orchestra.run.smithi093.stdout: debug client = 20 2024-01-15T18:03:57.820 INFO:teuthology.orchestra.run.smithi093.stdout: debug ms = 1 2024-01-15T18:03:57.820 INFO:teuthology.orchestra.run.smithi093.stdout: pid file = "" 2024-01-15T18:03:57.820 INFO:teuthology.orchestra.run.smithi093.stdout: rados mon op timeout = 900 2024-01-15T18:03:57.821 INFO:teuthology.orchestra.run.smithi093.stdout: rados osd op timeout = 900 2024-01-15T18:03:57.821 INFO:teuthology.orchestra.run.smithi093.stdout:[mon.a] 2024-01-15T18:03:57.821 INFO:teuthology.orchestra.run.smithi093.stdout:[mon.b] 2024-01-15T18:03:57.821 INFO:teuthology.orchestra.run.smithi093.stdout:[mon.c] 2024-01-15T18:03:57.821 INFO:teuthology.orchestra.run.smithi093.stdout:[mds.a] 2024-01-15T18:03:57.821 INFO:teuthology.orchestra.run.smithi093.stdout:[mds.c] 2024-01-15T18:03:57.821 INFO:teuthology.orchestra.run.smithi093.stdout:[mds.b] 2024-01-15T18:03:57.821 INFO:teuthology.orchestra.run.smithi093.stdout:[mds.d] 2024-01-15T18:03:57.821 INFO:teuthology.orchestra.run.smithi093.stdout:[mds] 2024-01-15T18:03:57.821 INFO:teuthology.orchestra.run.smithi093.stdout: debug mds = 20 2024-01-15T18:03:57.821 INFO:teuthology.orchestra.run.smithi093.stdout: debug mds balancer = 20 2024-01-15T18:03:57.821 INFO:teuthology.orchestra.run.smithi093.stdout: debug ms = 1 2024-01-15T18:03:57.822 INFO:teuthology.orchestra.run.smithi093.stdout: mds debug frag = True 2024-01-15T18:03:57.822 INFO:teuthology.orchestra.run.smithi093.stdout: mds debug scatterstat = True 2024-01-15T18:03:57.822 INFO:teuthology.orchestra.run.smithi093.stdout: mds op complaint time = 180 2024-01-15T18:03:57.822 INFO:teuthology.orchestra.run.smithi093.stdout: mds verify scatter = True 2024-01-15T18:03:57.822 INFO:teuthology.orchestra.run.smithi093.stdout: osd op complaint time = 180 2024-01-15T18:03:57.822 INFO:teuthology.orchestra.run.smithi093.stdout: rados mon op timeout = 900 2024-01-15T18:03:57.822 INFO:teuthology.orchestra.run.smithi093.stdout: rados osd op timeout = 900 2024-01-15T18:03:57.824 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-15T18:03:57.824 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-15T18:03:57.927 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-15T18:03:57.927 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:03:57.927 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-15T18:03:57.937 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:03:57.937 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-15T18:03:57.988 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T18:03:57.989 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:03:57.989 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-15T18:03:57.989 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-15T18:03:58.064 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:03:58.064 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-15T18:03:58.120 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T18:03:58.120 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:03:58.121 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-15T18:03:58.121 DEBUG:teuthology.orchestra.run.smithi093:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-15T18:03:58.156 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:03:58.156 DEBUG:teuthology.orchestra.run.smithi093:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-15T18:03:58.209 INFO:tasks.ceph:Setting up mon nodes... 2024-01-15T18:03:58.209 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-15T18:03:58.209 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-01-15T18:03:58.266 INFO:teuthology.orchestra.run.smithi027.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-01-15T18:03:58.272 DEBUG:teuthology.orchestra.run.smithi093:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-15T18:03:58.330 INFO:teuthology.orchestra.run.smithi093.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-15T18:03:58.349 INFO:tasks.ceph:Setting up mds nodes... 2024-01-15T18:03:58.350 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-01-15T18:03:58.406 INFO:teuthology.orchestra.run.smithi027.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-01-15T18:03:58.411 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-01-15T18:03:58.432 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-01-15T18:03:58.531 INFO:teuthology.orchestra.run.smithi027.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-01-15T18:03:58.536 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-01-15T18:03:58.557 DEBUG:teuthology.orchestra.run.smithi093:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-01-15T18:03:58.612 INFO:teuthology.orchestra.run.smithi093.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-01-15T18:03:58.617 DEBUG:teuthology.orchestra.run.smithi093:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-01-15T18:03:58.637 DEBUG:teuthology.orchestra.run.smithi093:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-01-15T18:03:58.733 INFO:teuthology.orchestra.run.smithi093.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-01-15T18:03:58.738 DEBUG:teuthology.orchestra.run.smithi093:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-01-15T18:03:58.757 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-15T18:03:58.757 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-15T18:03:58.800 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-15T18:03:58.818 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-01-15T18:03:58.902 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-01-15T18:03:58.920 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-01-15T18:03:58.962 INFO:teuthology.orchestra.run.smithi093.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-01-15T18:03:58.979 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-01-15T18:03:59.064 INFO:teuthology.orchestra.run.smithi093.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-01-15T18:03:59.081 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-15T18:03:59.082 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi027.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi093.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-15T18:03:59.082 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-15T18:03:59.103 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-15T18:03:59.103 INFO:tasks.ceph:role: osd.0 2024-01-15T18:03:59.103 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T18:03:59.103 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-15T18:03:59.171 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T18:03:59.171 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T18:03:59.171 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T18:03:59.171 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-15T18:03:59.172 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T18:03:59.172 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-01-15T18:03:59.172 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T18:03:59.172 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-15T18:03:59.172 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T18:03:59.172 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T18:03:59.624 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-01-15T18:03:59.626 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-01-15T18:03:59.627 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-15T18:03:59.734 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-15T18:03:59.751 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: /sbin/restorecon: command not found 2024-01-15T18:03:59.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:03:59.752 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-15T18:03:59.819 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-15T18:03:59.820 INFO:tasks.ceph:role: osd.1 2024-01-15T18:03:59.820 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T18:03:59.820 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-15T18:03:59.887 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T18:03:59.887 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T18:03:59.887 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T18:03:59.887 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-15T18:03:59.887 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T18:03:59.888 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-01-15T18:03:59.888 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T18:03:59.888 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-15T18:03:59.888 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T18:03:59.888 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T18:04:00.348 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-01-15T18:04:00.351 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-01-15T18:04:00.351 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-15T18:04:00.509 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-15T18:04:00.525 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: /sbin/restorecon: command not found 2024-01-15T18:04:00.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:04:00.527 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-15T18:04:00.587 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-15T18:04:00.587 INFO:tasks.ceph:role: osd.2 2024-01-15T18:04:00.588 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T18:04:00.588 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-15T18:04:00.655 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T18:04:00.656 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T18:04:00.656 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T18:04:00.656 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-15T18:04:00.656 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T18:04:00.656 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-01-15T18:04:00.656 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T18:04:00.656 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-15T18:04:00.656 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T18:04:00.656 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T18:04:01.141 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-01-15T18:04:01.144 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-01-15T18:04:01.144 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-01-15T18:04:01.285 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-15T18:04:01.303 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: /sbin/restorecon: command not found 2024-01-15T18:04:01.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:04:01.305 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-01-15T18:04:01.372 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-15T18:04:01.373 INFO:tasks.ceph:role: osd.3 2024-01-15T18:04:01.373 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T18:04:01.373 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-15T18:04:01.444 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T18:04:01.444 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T18:04:01.445 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T18:04:01.445 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-15T18:04:01.445 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T18:04:01.445 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-01-15T18:04:01.445 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T18:04:01.445 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-15T18:04:01.445 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T18:04:01.445 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T18:04:01.899 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-01-15T18:04:01.901 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-01-15T18:04:01.902 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-01-15T18:04:02.016 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-01-15T18:04:02.033 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: /sbin/restorecon: command not found 2024-01-15T18:04:02.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:04:02.034 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T18:04:02.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:02.159+0000 7f6c4f38f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-01-15T18:04:02.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:02.163+0000 7f6c4f38f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-01-15T18:04:02.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:02.163+0000 7f6c4f38f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-15T18:04:02.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:02.163+0000 7f6c4f38f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-15T18:04:02.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:02.163+0000 7f6c4f38f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-15T18:04:02.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:02.171+0000 7f6c4f38f640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-01-15T18:04:05.170 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-15T18:04:05.192 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T18:04:05.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:05.295+0000 7f370ebc6640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-01-15T18:04:05.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:05.295+0000 7f370ebc6640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-01-15T18:04:05.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:05.295+0000 7f370ebc6640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-15T18:04:05.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:05.295+0000 7f370ebc6640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-15T18:04:05.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:05.295+0000 7f370ebc6640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-15T18:04:05.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:05.295+0000 7f370ebc6640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-01-15T18:04:08.285 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-15T18:04:08.306 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T18:04:08.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:08.403+0000 7fcd927fd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-01-15T18:04:08.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:08.403+0000 7fcd927fd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-01-15T18:04:08.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:08.403+0000 7fcd927fd640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-15T18:04:08.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:08.403+0000 7fcd927fd640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-15T18:04:08.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:08.403+0000 7fcd927fd640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-15T18:04:08.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:08.403+0000 7fcd927fd640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-01-15T18:04:11.397 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-15T18:04:11.419 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T18:04:11.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:11.519+0000 7f7f2d33a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-01-15T18:04:11.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:11.519+0000 7f7f2d33a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-01-15T18:04:11.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:11.519+0000 7f7f2d33a640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-15T18:04:11.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:11.519+0000 7f7f2d33a640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-15T18:04:11.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:11.519+0000 7f7f2d33a640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-15T18:04:11.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:11.519+0000 7f7f2d33a640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-01-15T18:04:14.511 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-01-15T18:04:14.533 DEBUG:teuthology.orchestra.run.smithi093:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-01-15T18:04:14.555 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-15T18:04:14.555 INFO:tasks.ceph:role: osd.4 2024-01-15T18:04:14.555 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T18:04:14.555 DEBUG:teuthology.orchestra.run.smithi093:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-15T18:04:14.625 INFO:teuthology.orchestra.run.smithi093.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T18:04:14.625 INFO:teuthology.orchestra.run.smithi093.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T18:04:14.625 INFO:teuthology.orchestra.run.smithi093.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T18:04:14.625 INFO:teuthology.orchestra.run.smithi093.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-15T18:04:14.625 INFO:teuthology.orchestra.run.smithi093.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T18:04:14.625 INFO:teuthology.orchestra.run.smithi093.stdout: = sunit=0 swidth=0 blks 2024-01-15T18:04:14.625 INFO:teuthology.orchestra.run.smithi093.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T18:04:14.625 INFO:teuthology.orchestra.run.smithi093.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-15T18:04:14.626 INFO:teuthology.orchestra.run.smithi093.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T18:04:14.626 INFO:teuthology.orchestra.run.smithi093.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T18:04:15.097 INFO:teuthology.orchestra.run.smithi093.stdout:Discarding blocks...Done. 2024-01-15T18:04:15.100 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi093.front.sepia.ceph.com -o noatime 2024-01-15T18:04:15.100 DEBUG:teuthology.orchestra.run.smithi093:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-01-15T18:04:15.248 DEBUG:teuthology.orchestra.run.smithi093:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-01-15T18:04:15.266 INFO:teuthology.orchestra.run.smithi093.stderr:sudo: /sbin/restorecon: command not found 2024-01-15T18:04:15.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:04:15.267 DEBUG:teuthology.orchestra.run.smithi093:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-01-15T18:04:15.333 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-15T18:04:15.334 INFO:tasks.ceph:role: osd.5 2024-01-15T18:04:15.334 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T18:04:15.334 DEBUG:teuthology.orchestra.run.smithi093:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-15T18:04:15.405 INFO:teuthology.orchestra.run.smithi093.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T18:04:15.405 INFO:teuthology.orchestra.run.smithi093.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T18:04:15.405 INFO:teuthology.orchestra.run.smithi093.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T18:04:15.405 INFO:teuthology.orchestra.run.smithi093.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-15T18:04:15.405 INFO:teuthology.orchestra.run.smithi093.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T18:04:15.406 INFO:teuthology.orchestra.run.smithi093.stdout: = sunit=0 swidth=0 blks 2024-01-15T18:04:15.406 INFO:teuthology.orchestra.run.smithi093.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T18:04:15.406 INFO:teuthology.orchestra.run.smithi093.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-15T18:04:15.406 INFO:teuthology.orchestra.run.smithi093.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T18:04:15.406 INFO:teuthology.orchestra.run.smithi093.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T18:04:15.838 INFO:teuthology.orchestra.run.smithi093.stdout:Discarding blocks...Done. 2024-01-15T18:04:15.841 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi093.front.sepia.ceph.com -o noatime 2024-01-15T18:04:15.841 DEBUG:teuthology.orchestra.run.smithi093:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-01-15T18:04:15.950 DEBUG:teuthology.orchestra.run.smithi093:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-01-15T18:04:15.967 INFO:teuthology.orchestra.run.smithi093.stderr:sudo: /sbin/restorecon: command not found 2024-01-15T18:04:15.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:04:15.968 DEBUG:teuthology.orchestra.run.smithi093:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-01-15T18:04:16.036 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-15T18:04:16.036 INFO:tasks.ceph:role: osd.6 2024-01-15T18:04:16.037 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T18:04:16.037 DEBUG:teuthology.orchestra.run.smithi093:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-15T18:04:16.113 INFO:teuthology.orchestra.run.smithi093.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T18:04:16.113 INFO:teuthology.orchestra.run.smithi093.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T18:04:16.113 INFO:teuthology.orchestra.run.smithi093.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T18:04:16.113 INFO:teuthology.orchestra.run.smithi093.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-15T18:04:16.113 INFO:teuthology.orchestra.run.smithi093.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T18:04:16.113 INFO:teuthology.orchestra.run.smithi093.stdout: = sunit=0 swidth=0 blks 2024-01-15T18:04:16.113 INFO:teuthology.orchestra.run.smithi093.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T18:04:16.113 INFO:teuthology.orchestra.run.smithi093.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-15T18:04:16.114 INFO:teuthology.orchestra.run.smithi093.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T18:04:16.114 INFO:teuthology.orchestra.run.smithi093.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T18:04:16.582 INFO:teuthology.orchestra.run.smithi093.stdout:Discarding blocks...Done. 2024-01-15T18:04:16.585 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi093.front.sepia.ceph.com -o noatime 2024-01-15T18:04:16.585 DEBUG:teuthology.orchestra.run.smithi093:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-01-15T18:04:16.728 DEBUG:teuthology.orchestra.run.smithi093:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-01-15T18:04:16.744 INFO:teuthology.orchestra.run.smithi093.stderr:sudo: /sbin/restorecon: command not found 2024-01-15T18:04:16.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:04:16.745 DEBUG:teuthology.orchestra.run.smithi093:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-01-15T18:04:16.812 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-15T18:04:16.812 INFO:tasks.ceph:role: osd.7 2024-01-15T18:04:16.812 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T18:04:16.812 DEBUG:teuthology.orchestra.run.smithi093:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-15T18:04:16.884 INFO:teuthology.orchestra.run.smithi093.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T18:04:16.884 INFO:teuthology.orchestra.run.smithi093.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T18:04:16.884 INFO:teuthology.orchestra.run.smithi093.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T18:04:16.884 INFO:teuthology.orchestra.run.smithi093.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-15T18:04:16.884 INFO:teuthology.orchestra.run.smithi093.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T18:04:16.884 INFO:teuthology.orchestra.run.smithi093.stdout: = sunit=0 swidth=0 blks 2024-01-15T18:04:16.884 INFO:teuthology.orchestra.run.smithi093.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T18:04:16.884 INFO:teuthology.orchestra.run.smithi093.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-15T18:04:16.884 INFO:teuthology.orchestra.run.smithi093.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T18:04:16.884 INFO:teuthology.orchestra.run.smithi093.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T18:04:17.388 INFO:teuthology.orchestra.run.smithi093.stdout:Discarding blocks...Done. 2024-01-15T18:04:17.392 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi093.front.sepia.ceph.com -o noatime 2024-01-15T18:04:17.392 DEBUG:teuthology.orchestra.run.smithi093:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-01-15T18:04:17.527 DEBUG:teuthology.orchestra.run.smithi093:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-01-15T18:04:17.543 INFO:teuthology.orchestra.run.smithi093.stderr:sudo: /sbin/restorecon: command not found 2024-01-15T18:04:17.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:04:17.544 DEBUG:teuthology.orchestra.run.smithi093:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T18:04:17.681 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:17.676+0000 7f8c43f94640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-01-15T18:04:17.683 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:17.676+0000 7f8c43f94640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-01-15T18:04:17.684 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:17.676+0000 7f8c43f94640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-15T18:04:17.684 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:17.676+0000 7f8c43f94640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-15T18:04:17.684 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:17.676+0000 7f8c43f94640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-15T18:04:17.686 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:17.680+0000 7f8c43f94640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-01-15T18:04:20.681 DEBUG:teuthology.orchestra.run.smithi093:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-01-15T18:04:20.702 DEBUG:teuthology.orchestra.run.smithi093:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T18:04:20.803 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:20.800+0000 7ff6c0de8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-01-15T18:04:20.804 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:20.800+0000 7ff6c0de8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-01-15T18:04:20.804 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:20.800+0000 7ff6c0de8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-15T18:04:20.804 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:20.800+0000 7ff6c0de8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-15T18:04:20.804 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:20.800+0000 7ff6c0de8640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-15T18:04:20.805 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:20.804+0000 7ff6c0de8640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-01-15T18:04:23.785 DEBUG:teuthology.orchestra.run.smithi093:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-01-15T18:04:23.805 DEBUG:teuthology.orchestra.run.smithi093:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T18:04:23.902 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:23.900+0000 7febf342b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-01-15T18:04:23.902 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:23.900+0000 7febf342b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-01-15T18:04:23.903 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:23.900+0000 7febf342b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-15T18:04:23.903 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:23.900+0000 7febf342b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-15T18:04:23.903 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:23.900+0000 7febf342b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-15T18:04:23.904 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:23.900+0000 7febf342b640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-01-15T18:04:25.900 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:04:25.904 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:04:26.904 DEBUG:teuthology.orchestra.run.smithi093:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-01-15T18:04:26.927 DEBUG:teuthology.orchestra.run.smithi093:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T18:04:27.021 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:27.016+0000 7f6bed5b6640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-01-15T18:04:27.022 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:27.020+0000 7f6bed5b6640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-01-15T18:04:27.022 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:27.020+0000 7f6bed5b6640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-15T18:04:27.022 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:27.020+0000 7f6bed5b6640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-15T18:04:27.022 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:27.020+0000 7f6bed5b6640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-15T18:04:27.023 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:27.020+0000 7f6bed5b6640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-01-15T18:04:30.008 DEBUG:teuthology.orchestra.run.smithi093:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-01-15T18:04:30.028 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-15T18:04:30.029 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:04:30.029 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-01-15T18:04:30.051 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:04:30.051 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-01-15T18:04:30.115 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:04:30.115 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-01-15T18:04:30.178 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:04:30.178 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-15T18:04:30.242 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:04:30.242 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-15T18:04:30.307 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:04:30.307 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-15T18:04:30.370 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:04:30.371 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-01-15T18:04:30.434 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:04:30.434 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-15T18:04:30.456 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:04:30.456 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-01-15T18:04:30.517 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:04:30.517 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-01-15T18:04:30.581 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:04:30.581 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-01-15T18:04:30.645 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:04:30.645 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-01-15T18:04:30.709 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:04:30.709 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-01-15T18:04:30.769 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:04:30.769 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-01-15T18:04:30.833 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:04:30.833 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-15T18:04:30.843 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:04:30.843 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-01-15T18:04:30.894 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:04:30.894 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-01-15T18:04:30.903 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:04:30.903 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-01-15T18:04:30.953 INFO:tasks.ceph:Adding keys to all mons... 2024-01-15T18:04:30.954 DEBUG:teuthology.orchestra.run.smithi027:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-15T18:04:30.957 DEBUG:teuthology.orchestra.run.smithi093:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-15T18:04:31.002 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr.y] 2024-01-15T18:04:31.002 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOc6VlzafrDxAAVVEBuX8BDT3cB9NfD2xTdQ== 2024-01-15T18:04:31.002 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.a] 2024-01-15T18:04:31.002 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOc6VlbMZIGBAAhSU4y2DGJd3MH0v6v3LsDg== 2024-01-15T18:04:31.002 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.c] 2024-01-15T18:04:31.002 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOc6Vla6a3HxAA/AHc6FSTccj0JGwiR4Vl7g== 2024-01-15T18:04:31.002 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.0] 2024-01-15T18:04:31.002 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCSc6Vly4TACRAAYBmMIcsZsRp4CZ5a3xCWgA== 2024-01-15T18:04:31.002 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.1] 2024-01-15T18:04:31.003 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCVc6VlW/HEERAAOBgPz+eGpHwNXDe8fkYo7w== 2024-01-15T18:04:31.003 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.2] 2024-01-15T18:04:31.003 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCYc6Vl+Ec4GBAAWRe/6u7u4RXApd57H8ihng== 2024-01-15T18:04:31.003 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.3] 2024-01-15T18:04:31.003 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCbc6VlNdMPHxAAHzJvzLltsTl1wrEnI/iyrQ== 2024-01-15T18:04:31.003 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr.x] 2024-01-15T18:04:31.003 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOc6VlQbinExAAWtUdgkUNR3cCKV+ytcbUuQ== 2024-01-15T18:04:31.003 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.b] 2024-01-15T18:04:31.003 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOc6VlC7B9JBAAsXmBjV4KwPZejOmowXO2Zg== 2024-01-15T18:04:31.003 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.d] 2024-01-15T18:04:31.004 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOc6Vl9nqzKxAAx2aDA8UwidAUrU9DBxGldA== 2024-01-15T18:04:31.004 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.4] 2024-01-15T18:04:31.004 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQChc6Vlg8eVKBAAwZWAncCE2Lra+CxmSIxZkA== 2024-01-15T18:04:31.004 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.5] 2024-01-15T18:04:31.004 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCkc6VlIXDOLxAAFISn6ILqq/aNFK07RxNnaw== 2024-01-15T18:04:31.004 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.6] 2024-01-15T18:04:31.004 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCnc6Vl9Ru2NRAAl996VBDudw23SGiFPusIcw== 2024-01-15T18:04:31.004 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.7] 2024-01-15T18:04:31.004 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCrc6VlHoAsARAAIoPMridLsIcpU3I9Hr5XLQ== 2024-01-15T18:04:31.004 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-01-15T18:04:31.005 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOc6VlH+DCLxAA1sKVut+3WjA/B+aYJ6/Jeg== 2024-01-15T18:04:31.005 INFO:teuthology.orchestra.run.smithi027.stdout:[client.2] 2024-01-15T18:04:31.005 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOc6VlDbLQNRAA82Lwz+pKY+3nHPAr1DichA== 2024-01-15T18:04:31.005 INFO:teuthology.orchestra.run.smithi027.stdout:[client.1] 2024-01-15T18:04:31.005 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOc6VlaRBaORAAijn3UJLHbWm8ol2NiKNSaw== 2024-01-15T18:04:31.005 INFO:teuthology.orchestra.run.smithi027.stdout:[client.3] 2024-01-15T18:04:31.005 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCPc6VlOyDaAxAADea0K1OEDTfrzsM7VEkixg== 2024-01-15T18:04:31.015 INFO:teuthology.orchestra.run.smithi093.stdout:[mgr.y] 2024-01-15T18:04:31.015 INFO:teuthology.orchestra.run.smithi093.stdout: key = AQCOc6VlzafrDxAAVVEBuX8BDT3cB9NfD2xTdQ== 2024-01-15T18:04:31.015 INFO:teuthology.orchestra.run.smithi093.stdout:[mds.a] 2024-01-15T18:04:31.015 INFO:teuthology.orchestra.run.smithi093.stdout: key = AQCOc6VlbMZIGBAAhSU4y2DGJd3MH0v6v3LsDg== 2024-01-15T18:04:31.015 INFO:teuthology.orchestra.run.smithi093.stdout:[mds.c] 2024-01-15T18:04:31.015 INFO:teuthology.orchestra.run.smithi093.stdout: key = AQCOc6Vla6a3HxAA/AHc6FSTccj0JGwiR4Vl7g== 2024-01-15T18:04:31.016 INFO:teuthology.orchestra.run.smithi093.stdout:[osd.0] 2024-01-15T18:04:31.016 INFO:teuthology.orchestra.run.smithi093.stdout: key = AQCSc6Vly4TACRAAYBmMIcsZsRp4CZ5a3xCWgA== 2024-01-15T18:04:31.016 INFO:teuthology.orchestra.run.smithi093.stdout:[osd.1] 2024-01-15T18:04:31.016 INFO:teuthology.orchestra.run.smithi093.stdout: key = AQCVc6VlW/HEERAAOBgPz+eGpHwNXDe8fkYo7w== 2024-01-15T18:04:31.016 INFO:teuthology.orchestra.run.smithi093.stdout:[osd.2] 2024-01-15T18:04:31.016 INFO:teuthology.orchestra.run.smithi093.stdout: key = AQCYc6Vl+Ec4GBAAWRe/6u7u4RXApd57H8ihng== 2024-01-15T18:04:31.016 INFO:teuthology.orchestra.run.smithi093.stdout:[osd.3] 2024-01-15T18:04:31.016 INFO:teuthology.orchestra.run.smithi093.stdout: key = AQCbc6VlNdMPHxAAHzJvzLltsTl1wrEnI/iyrQ== 2024-01-15T18:04:31.016 INFO:teuthology.orchestra.run.smithi093.stdout:[mgr.x] 2024-01-15T18:04:31.016 INFO:teuthology.orchestra.run.smithi093.stdout: key = AQCOc6VlQbinExAAWtUdgkUNR3cCKV+ytcbUuQ== 2024-01-15T18:04:31.017 INFO:teuthology.orchestra.run.smithi093.stdout:[mds.b] 2024-01-15T18:04:31.017 INFO:teuthology.orchestra.run.smithi093.stdout: key = AQCOc6VlC7B9JBAAsXmBjV4KwPZejOmowXO2Zg== 2024-01-15T18:04:31.017 INFO:teuthology.orchestra.run.smithi093.stdout:[mds.d] 2024-01-15T18:04:31.017 INFO:teuthology.orchestra.run.smithi093.stdout: key = AQCOc6Vl9nqzKxAAx2aDA8UwidAUrU9DBxGldA== 2024-01-15T18:04:31.017 INFO:teuthology.orchestra.run.smithi093.stdout:[osd.4] 2024-01-15T18:04:31.017 INFO:teuthology.orchestra.run.smithi093.stdout: key = AQChc6Vlg8eVKBAAwZWAncCE2Lra+CxmSIxZkA== 2024-01-15T18:04:31.017 INFO:teuthology.orchestra.run.smithi093.stdout:[osd.5] 2024-01-15T18:04:31.017 INFO:teuthology.orchestra.run.smithi093.stdout: key = AQCkc6VlIXDOLxAAFISn6ILqq/aNFK07RxNnaw== 2024-01-15T18:04:31.017 INFO:teuthology.orchestra.run.smithi093.stdout:[osd.6] 2024-01-15T18:04:31.017 INFO:teuthology.orchestra.run.smithi093.stdout: key = AQCnc6Vl9Ru2NRAAl996VBDudw23SGiFPusIcw== 2024-01-15T18:04:31.017 INFO:teuthology.orchestra.run.smithi093.stdout:[osd.7] 2024-01-15T18:04:31.017 INFO:teuthology.orchestra.run.smithi093.stdout: key = AQCrc6VlHoAsARAAIoPMridLsIcpU3I9Hr5XLQ== 2024-01-15T18:04:31.018 INFO:teuthology.orchestra.run.smithi093.stdout:[client.0] 2024-01-15T18:04:31.018 INFO:teuthology.orchestra.run.smithi093.stdout: key = AQCOc6VlH+DCLxAA1sKVut+3WjA/B+aYJ6/Jeg== 2024-01-15T18:04:31.018 INFO:teuthology.orchestra.run.smithi093.stdout:[client.2] 2024-01-15T18:04:31.018 INFO:teuthology.orchestra.run.smithi093.stdout: key = AQCOc6VlDbLQNRAA82Lwz+pKY+3nHPAr1DichA== 2024-01-15T18:04:31.018 INFO:teuthology.orchestra.run.smithi093.stdout:[client.1] 2024-01-15T18:04:31.018 INFO:teuthology.orchestra.run.smithi093.stdout: key = AQCOc6VlaRBaORAAijn3UJLHbWm8ol2NiKNSaw== 2024-01-15T18:04:31.018 INFO:teuthology.orchestra.run.smithi093.stdout:[client.3] 2024-01-15T18:04:31.018 INFO:teuthology.orchestra.run.smithi093.stdout: key = AQCPc6VlOyDaAxAADea0K1OEDTfrzsM7VEkixg== 2024-01-15T18:04:31.019 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-15T18:04:31.053 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-15T18:04:31.117 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T18:04:31.121 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T18:04:31.179 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T18:04:31.183 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T18:04:31.240 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T18:04:31.244 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T18:04:31.297 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T18:04:31.301 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T18:04:31.354 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T18:04:31.359 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T18:04:31.413 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T18:04:31.417 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T18:04:31.471 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-15T18:04:31.475 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-15T18:04:31.528 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T18:04:31.532 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T18:04:31.588 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T18:04:31.592 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T18:04:31.649 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T18:04:31.652 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T18:04:31.706 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T18:04:31.710 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T18:04:31.764 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T18:04:31.767 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T18:04:31.820 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T18:04:31.824 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T18:04:31.877 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T18:04:31.880 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T18:04:31.936 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T18:04:31.939 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T18:04:31.996 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T18:04:32.000 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T18:04:32.056 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T18:04:32.059 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T18:04:32.115 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-15T18:04:32.115 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-15T18:04:32.135 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-15T18:04:32.249 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-15T18:04:32.270 DEBUG:teuthology.orchestra.run.smithi093:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-01-15T18:04:32.289 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-15T18:04:32.405 DEBUG:teuthology.orchestra.run.smithi093:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-01-15T18:04:32.424 DEBUG:teuthology.orchestra.run.smithi093:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-01-15T18:04:32.484 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-15T18:04:32.595 DEBUG:teuthology.orchestra.run.smithi093:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-01-15T18:04:32.614 DEBUG:teuthology.orchestra.run.smithi027:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-15T18:04:32.618 DEBUG:teuthology.orchestra.run.smithi093:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-15T18:04:32.664 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-15T18:04:32.665 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-15T18:04:32.665 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-01-15T18:04:32.668 INFO:tasks.ceph.mon.a:Started 2024-01-15T18:04:32.668 INFO:tasks.ceph.mon.b:Restarting daemon 2024-01-15T18:04:32.668 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-01-15T18:04:32.708 INFO:tasks.ceph.mon.b:Started 2024-01-15T18:04:32.708 INFO:tasks.ceph.mon.c:Restarting daemon 2024-01-15T18:04:32.708 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-01-15T18:04:32.711 INFO:tasks.ceph.mon.c:Started 2024-01-15T18:04:32.711 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-15T18:04:32.711 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-01-15T18:04:32.711 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-01-15T18:04:32.715 INFO:tasks.ceph.mgr.y:Started 2024-01-15T18:04:32.715 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-15T18:04:32.715 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-01-15T18:04:32.718 INFO:tasks.ceph.mgr.x:Started 2024-01-15T18:04:32.718 INFO:tasks.ceph:Setting crush tunables to default 2024-01-15T18:04:32.718 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd crush tunables default 2024-01-15T18:04:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:32.853+0000 7f9245800640 1 Processor -- start 2024-01-15T18:04:32.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:32.853+0000 7f9245800640 1 -- start start 2024-01-15T18:04:32.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:32.853+0000 7f9245800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f924015d480 0x7f92400cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:32.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:32.853+0000 7f9245800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f924006c740 0x7f92400d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:32.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:32.853+0000 7f9245800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f9240099a10 0x7f9240099ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:32.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:32.853+0000 7f9245800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f924009a400 con 0x7f924015d480 2024-01-15T18:04:32.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:32.853+0000 7f9245800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f924015d860 con 0x7f9240099a10 2024-01-15T18:04:32.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:32.853+0000 7f9245800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f924006b6a0 con 0x7f924006c740 2024-01-15T18:04:32.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:32.853+0000 7f923ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f924015d480 0x7f92400cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:32.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:32.853+0000 7f923ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f924015d480 0x7f92400cfbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55862/0 (socket says 172.21.15.27:55862) 2024-01-15T18:04:32.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:32.853+0000 7f923ea00640 1 -- 172.21.15.27:0/1283264961 learned_addr learned my addr 172.21.15.27:0/1283264961 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:32.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:32.853+0000 7f923e000640 1 --2- 172.21.15.27:0/1283264961 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f924006c740 0x7f92400d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:32.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:32.853+0000 7f923ea00640 1 -- 172.21.15.27:0/1283264961 >> v2:172.21.15.27:3300/0 conn(0x7f924015d480 msgr2=0x7f92400cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-15T18:04:32.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:32.857+0000 7f923ea00640 1 -- 172.21.15.27:0/1283264961 >> v2:172.21.15.27:3300/0 conn(0x7f924015d480 msgr2=0x7f92400cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-15T18:04:32.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:32.857+0000 7f923ea00640 1 --2- 172.21.15.27:0/1283264961 >> v2:172.21.15.27:3300/0 conn(0x7f924015d480 0x7f92400cfbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-15T18:04:32.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:32.857+0000 7f923e000640 1 -- 172.21.15.27:0/1283264961 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f924006c740 msgr2=0x7f92400d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-15T18:04:32.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:32.857+0000 7f923e000640 1 -- 172.21.15.27:0/1283264961 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f924006c740 msgr2=0x7f92400d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-15T18:04:32.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:32.857+0000 7f923e000640 1 --2- 172.21.15.27:0/1283264961 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f924006c740 0x7f92400d0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-15T18:04:32.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:32.857+0000 7f923e000640 1 --2- 172.21.15.27:0/1283264961 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f924006c740 0x7f92400d0130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-15T18:04:32.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:32.857+0000 7f923ea00640 1 --2- 172.21.15.27:0/1283264961 >> v2:172.21.15.27:3300/0 conn(0x7f924015d480 0x7f92400cfbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-15T18:04:32.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:32.857+0000 7f923f400640 1 --2- 172.21.15.27:0/1283264961 >> v2:172.21.15.93:3300/0 conn(0x7f9240099a10 0x7f9240099ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:32.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:32.857+0000 7f923f400640 1 -- 172.21.15.27:0/1283264961 >> v2:172.21.15.93:3300/0 conn(0x7f9240099a10 msgr2=0x7f9240099ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-01-15T18:04:32.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:32.857+0000 7f923f400640 1 -- 172.21.15.27:0/1283264961 >> v2:172.21.15.93:3300/0 conn(0x7f9240099a10 msgr2=0x7f9240099ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-15T18:04:32.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:32.857+0000 7f923f400640 1 --2- 172.21.15.27:0/1283264961 >> v2:172.21.15.93:3300/0 conn(0x7f9240099a10 0x7f9240099ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-15T18:04:32.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:32.857+0000 7f923f400640 1 --2- 172.21.15.27:0/1283264961 >> v2:172.21.15.93:3300/0 conn(0x7f9240099a10 0x7f9240099ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-15T18:04:33.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.061+0000 7f923ea00640 1 --2- 172.21.15.27:0/1283264961 >> v2:172.21.15.27:3300/0 conn(0x7f924015d480 0x7f92400cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:33.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.061+0000 7f923f400640 1 --2- 172.21.15.27:0/1283264961 >> v2:172.21.15.93:3300/0 conn(0x7f9240099a10 0x7f9240099ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:33.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.061+0000 7f923e000640 1 --2- 172.21.15.27:0/1283264961 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f924006c740 0x7f92400d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:33.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.061+0000 7f923ea00640 1 -- 172.21.15.27:0/1283264961 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f924006c740 msgr2=0x7f92400d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:33.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.061+0000 7f923ea00640 1 --2- 172.21.15.27:0/1283264961 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f924006c740 0x7f92400d0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.061+0000 7f923ea00640 1 -- 172.21.15.27:0/1283264961 >> v2:172.21.15.93:3300/0 conn(0x7f9240099a10 msgr2=0x7f9240099ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:33.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.061+0000 7f923ea00640 1 --2- 172.21.15.27:0/1283264961 >> v2:172.21.15.93:3300/0 conn(0x7f9240099a10 0x7f9240099ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.061+0000 7f923ea00640 1 -- 172.21.15.27:0/1283264961 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f924006b980 con 0x7f924015d480 2024-01-15T18:04:33.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.061+0000 7f923e000640 1 --2- 172.21.15.27:0/1283264961 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f924006c740 0x7f92400d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:04:33.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.061+0000 7f923ea00640 1 --2- 172.21.15.27:0/1283264961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f924015d480 0x7f92400cfbf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f92400719a0 tx=0x7f923001c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=639dba2f01df8d03 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:33.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.061+0000 7f923d600640 1 -- 172.21.15.27:0/1283264961 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92300a57e0 con 0x7f924015d480 2024-01-15T18:04:33.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.061+0000 7f923d600640 1 -- 172.21.15.27:0/1283264961 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92300a5940 con 0x7f924015d480 2024-01-15T18:04:33.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.061+0000 7f923d600640 1 -- 172.21.15.27:0/1283264961 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92300a5cf0 con 0x7f924015d480 2024-01-15T18:04:33.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.065+0000 7f9245800640 1 -- 172.21.15.27:0/1283264961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f924015d480 msgr2=0x7f92400cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:33.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.065+0000 7f9245800640 1 --2- 172.21.15.27:0/1283264961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f924015d480 0x7f92400cfbf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f92400719a0 tx=0x7f923001c760 comp rx=0 tx=0).stop 2024-01-15T18:04:33.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.065+0000 7f9245800640 1 -- 172.21.15.27:0/1283264961 shutdown_connections 2024-01-15T18:04:33.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.065+0000 7f9245800640 1 --2- 172.21.15.27:0/1283264961 >> v2:172.21.15.93:3300/0 conn(0x7f9240099a10 0x7f9240099ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.065+0000 7f9245800640 1 --2- 172.21.15.27:0/1283264961 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f924006c740 0x7f92400d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.065+0000 7f9245800640 1 --2- 172.21.15.27:0/1283264961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f924015d480 0x7f92400cfbf0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.065+0000 7f9245800640 1 -- 172.21.15.27:0/1283264961 >> 172.21.15.27:0/1283264961 conn(0x7f92400b2a80 msgr2=0x7f92400b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:33.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.065+0000 7f9245800640 1 -- 172.21.15.27:0/1283264961 shutdown_connections 2024-01-15T18:04:33.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.065+0000 7f9245800640 1 -- 172.21.15.27:0/1283264961 wait complete. 2024-01-15T18:04:33.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.065+0000 7f9245800640 1 Processor -- start 2024-01-15T18:04:33.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.065+0000 7f9245800640 1 -- start start 2024-01-15T18:04:33.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.065+0000 7f9245800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f924006c740 0x7f9240154ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:33.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.065+0000 7f9245800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9240099a10 0x7f924014ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:33.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.065+0000 7f9245800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f924015d480 0x7f924014d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:33.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.065+0000 7f9245800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9240152810 con 0x7f924006c740 2024-01-15T18:04:33.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.065+0000 7f9245800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924015b090 con 0x7f9240099a10 2024-01-15T18:04:33.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.065+0000 7f9245800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f924014d750 con 0x7f924015d480 2024-01-15T18:04:33.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.065+0000 7f923ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f924006c740 0x7f9240154ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:33.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.065+0000 7f923ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f924006c740 0x7f9240154ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55888/0 (socket says 172.21.15.27:55888) 2024-01-15T18:04:33.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.065+0000 7f923ea00640 1 -- 172.21.15.27:0/3783755656 learned_addr learned my addr 172.21.15.27:0/3783755656 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:33.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.069+0000 7f923e000640 1 --2- 172.21.15.27:0/3783755656 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9240099a10 0x7f924014ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:33.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.069+0000 7f923f400640 1 --2- 172.21.15.27:0/3783755656 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f924015d480 0x7f924014d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:33.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.069+0000 7f923ea00640 1 -- 172.21.15.27:0/3783755656 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f924015d480 msgr2=0x7f924014d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:33.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.069+0000 7f923ea00640 1 --2- 172.21.15.27:0/3783755656 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f924015d480 0x7f924014d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.069+0000 7f923ea00640 1 -- 172.21.15.27:0/3783755656 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9240099a10 msgr2=0x7f924014ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:33.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.069+0000 7f923ea00640 1 --2- 172.21.15.27:0/3783755656 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9240099a10 0x7f924014ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.069+0000 7f923ea00640 1 -- 172.21.15.27:0/3783755656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9230009530 con 0x7f924006c740 2024-01-15T18:04:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.069+0000 7f923f400640 1 --2- 172.21.15.27:0/3783755656 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f924015d480 0x7f924014d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:04:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.069+0000 7f923ea00640 1 --2- 172.21.15.27:0/3783755656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f924006c740 0x7f9240154ac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f923001ae10 tx=0x7f9230019500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.069+0000 7f9236a00640 1 -- 172.21.15.27:0/3783755656 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9230004440 con 0x7f924006c740 2024-01-15T18:04:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.069+0000 7f9236a00640 1 -- 172.21.15.27:0/3783755656 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9230003620 con 0x7f924006c740 2024-01-15T18:04:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.069+0000 7f9236a00640 1 -- 172.21.15.27:0/3783755656 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9230003990 con 0x7f924006c740 2024-01-15T18:04:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.069+0000 7f9245800640 1 -- 172.21.15.27:0/3783755656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f924015adb0 con 0x7f924006c740 2024-01-15T18:04:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.069+0000 7f9245800640 1 -- 172.21.15.27:0/3783755656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9240152430 con 0x7f924006c740 2024-01-15T18:04:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.069+0000 7f9236a00640 1 -- 172.21.15.27:0/3783755656 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f92300b4050 con 0x7f924006c740 2024-01-15T18:04:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.069+0000 7f9236a00640 1 -- 172.21.15.27:0/3783755656 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f92300ada20 con 0x7f924006c740 2024-01-15T18:04:33.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.069+0000 7f9245800640 1 -- 172.21.15.27:0/3783755656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f924004e850 con 0x7f924006c740 2024-01-15T18:04:33.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.073+0000 7f9236a00640 1 -- 172.21.15.27:0/3783755656 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f92300c0070 con 0x7f924006c740 2024-01-15T18:04:33.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.133+0000 7f9245800640 1 -- 172.21.15.27:0/3783755656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f92400d09e0 con 0x7f924006c740 2024-01-15T18:04:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.137+0000 7f9236a00640 1 -- 172.21.15.27:0/3783755656 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f92300b2070 con 0x7f924006c740 2024-01-15T18:04:33.144 INFO:teuthology.orchestra.run.smithi027.stderr:adjusted tunables profile to default 2024-01-15T18:04:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.137+0000 7f9245800640 1 -- 172.21.15.27:0/3783755656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f924006c740 msgr2=0x7f9240154ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.137+0000 7f9245800640 1 --2- 172.21.15.27:0/3783755656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f924006c740 0x7f9240154ac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f923001ae10 tx=0x7f9230019500 comp rx=0 tx=0).stop 2024-01-15T18:04:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.137+0000 7f9245800640 1 -- 172.21.15.27:0/3783755656 shutdown_connections 2024-01-15T18:04:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.137+0000 7f9245800640 1 --2- 172.21.15.27:0/3783755656 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f924015d480 0x7f924014d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.137+0000 7f9245800640 1 --2- 172.21.15.27:0/3783755656 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9240099a10 0x7f924014ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.137+0000 7f9245800640 1 --2- 172.21.15.27:0/3783755656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f924006c740 0x7f9240154ac0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.137+0000 7f9245800640 1 -- 172.21.15.27:0/3783755656 >> 172.21.15.27:0/3783755656 conn(0x7f92400b2a80 msgr2=0x7f924015e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.137+0000 7f9245800640 1 -- 172.21.15.27:0/3783755656 shutdown_connections 2024-01-15T18:04:33.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:33.137+0000 7f9245800640 1 -- 172.21.15.27:0/3783755656 wait complete. 2024-01-15T18:04:33.164 INFO:tasks.ceph:check_enable_crimson: False 2024-01-15T18:04:33.164 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-15T18:04:33.165 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:04:33.165 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-15T18:04:33.184 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:04:33.185 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-15T18:04:33.242 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:04:33.239+0000 7f5a768cf180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-15T18:04:33.244 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:04:33.244 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-15T18:04:33.269 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-15T18:04:33.261+0000 7f49aec8d180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-15T18:04:33.289 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:04:33.289 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-01-15T18:04:33.348 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:04:33.349 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-01-15T18:04:33.369 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:04:33.369 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-01-15T18:04:33.428 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:04:33.428 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-01-15T18:04:33.488 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:04:33.488 DEBUG:teuthology.orchestra.run.smithi093:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-01-15T18:04:33.548 DEBUG:teuthology.orchestra.run.smithi093:> sudo ceph --cluster ceph osd new ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf 0 2024-01-15T18:04:33.727 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.723+0000 7fe8fec00640 1 Processor -- start 2024-01-15T18:04:33.727 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.723+0000 7fe8fec00640 1 -- start start 2024-01-15T18:04:33.728 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.723+0000 7fe8fec00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8f806c050 0x7fe8f815e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:33.728 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.723+0000 7fe8fec00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe8f806c430 0x7fe8f815cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:33.728 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.723+0000 7fe8fec00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f80ce730 con 0x7fe8f815d0f0 2024-01-15T18:04:33.729 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.723+0000 7fe8fec00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8f815e790 con 0x7fe8f806c430 2024-01-15T18:04:33.729 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.723+0000 7fe8fec00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f80cdb20 con 0x7fe8f806c050 2024-01-15T18:04:33.729 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.723+0000 7fe8f7e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8f806c050 0x7fe8f815e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:33.729 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.723+0000 7fe8f7e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8f806c050 0x7fe8f815e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.93:37884/0 (socket says 172.21.15.93:37884) 2024-01-15T18:04:33.729 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.723+0000 7fe8f7e00640 1 -- 172.21.15.93:0/3294150336 learned_addr learned my addr 172.21.15.93:0/3294150336 (peer_addr_for_me v2:172.21.15.93:0/0) 2024-01-15T18:04:33.729 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.723+0000 7fe8f7e00640 1 -- 172.21.15.93:0/3294150336 >> v2:172.21.15.27:3300/0 conn(0x7fe8f806c430 msgr2=0x7fe8f815cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:33.729 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.723+0000 7fe8f7e00640 1 --2- 172.21.15.93:0/3294150336 >> v2:172.21.15.27:3300/0 conn(0x7fe8f806c430 0x7fe8f815cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.730 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.723+0000 7fe8f7e00640 1 -- 172.21.15.93:0/3294150336 >> v1:172.21.15.27:6789/0 conn(0x7fe8f815d0f0 legacy=0x7fe8f8160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:33.730 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.723+0000 7fe8f7e00640 1 -- 172.21.15.93:0/3294150336 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f80cde00 con 0x7fe8f806c050 2024-01-15T18:04:33.731 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.727+0000 7fe8f7e00640 1 --2- 172.21.15.93:0/3294150336 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8f806c050 0x7fe8f815e250 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe8e8009870 tx=0x7fe8e801c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b55697eabac7f221 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:33.732 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.727+0000 7fe8f6a00640 1 -- 172.21.15.93:0/3294150336 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e80aa5f0 con 0x7fe8f806c050 2024-01-15T18:04:33.732 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.727+0000 7fe8f6a00640 1 -- 172.21.15.93:0/3294150336 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8e80aa750 con 0x7fe8f806c050 2024-01-15T18:04:33.732 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.727+0000 7fe8f6a00640 1 -- 172.21.15.93:0/3294150336 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e80aaae0 con 0x7fe8f806c050 2024-01-15T18:04:33.733 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.727+0000 7fe8fec00640 1 -- 172.21.15.93:0/3294150336 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8f806c050 msgr2=0x7fe8f815e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:33.733 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.727+0000 7fe8fec00640 1 --2- 172.21.15.93:0/3294150336 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8f806c050 0x7fe8f815e250 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe8e8009870 tx=0x7fe8e801c4d0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.733 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.727+0000 7fe8fec00640 1 -- 172.21.15.93:0/3294150336 shutdown_connections 2024-01-15T18:04:33.733 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.727+0000 7fe8fec00640 1 --2- 172.21.15.93:0/3294150336 >> v2:172.21.15.27:3300/0 conn(0x7fe8f806c430 0x7fe8f815cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.733 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.727+0000 7fe8fec00640 1 --2- 172.21.15.93:0/3294150336 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8f806c050 0x7fe8f815e250 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.733 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.727+0000 7fe8fec00640 1 -- 172.21.15.93:0/3294150336 >> 172.21.15.93:0/3294150336 conn(0x7fe8f8099a10 msgr2=0x7fe8f8099e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:33.733 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.727+0000 7fe8fec00640 1 -- 172.21.15.93:0/3294150336 shutdown_connections 2024-01-15T18:04:33.733 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.727+0000 7fe8fec00640 1 -- 172.21.15.93:0/3294150336 wait complete. 2024-01-15T18:04:33.735 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.731+0000 7fe8fec00640 1 Processor -- start 2024-01-15T18:04:33.735 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.731+0000 7fe8fec00640 1 -- start start 2024-01-15T18:04:33.736 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.731+0000 7fe8fec00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8f806c050 0x7fe8f8151760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:33.736 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.731+0000 7fe8fec00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe8f806c430 0x7fe8f8156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:33.736 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.731+0000 7fe8fec00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8f815d0f0 0x7fe8f8157330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:33.736 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.731+0000 7fe8fec00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f80b15c0 con 0x7fe8f806c050 2024-01-15T18:04:33.736 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.731+0000 7fe8fec00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f80b2430 con 0x7fe8f806c430 2024-01-15T18:04:33.737 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.731+0000 7fe8fec00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f80b28c0 con 0x7fe8f815d0f0 2024-01-15T18:04:33.737 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.731+0000 7fe8f7400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe8f806c430 0x7fe8f8156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:33.737 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.731+0000 7fe8fcc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8f815d0f0 0x7fe8f8157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:33.737 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.731+0000 7fe8f7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8f806c050 0x7fe8f8151760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:33.737 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.731+0000 7fe8f7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8f806c050 0x7fe8f8151760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.93:39654/0 (socket says 172.21.15.93:39654) 2024-01-15T18:04:33.737 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.731+0000 7fe8fcc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8f815d0f0 0x7fe8f8157330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.93:37896/0 (socket says 172.21.15.93:37896) 2024-01-15T18:04:33.737 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.731+0000 7fe8f7e00640 1 -- 172.21.15.93:0/3705141581 learned_addr learned my addr 172.21.15.93:0/3705141581 (peer_addr_for_me v2:172.21.15.93:0/0) 2024-01-15T18:04:33.737 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.731+0000 7fe8f7400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe8f806c430 0x7fe8f8156df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:46858/0 (socket says 172.21.15.93:46858) 2024-01-15T18:04:33.738 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.731+0000 7fe8fcc00640 1 -- 172.21.15.93:0/3705141581 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe8f806c430 msgr2=0x7fe8f8156df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:33.738 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.731+0000 7fe8fcc00640 1 --2- 172.21.15.93:0/3705141581 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe8f806c430 0x7fe8f8156df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.738 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.731+0000 7fe8fcc00640 1 -- 172.21.15.93:0/3705141581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8f806c050 msgr2=0x7fe8f8151760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:33.738 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.731+0000 7fe8fcc00640 1 --2- 172.21.15.93:0/3705141581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8f806c050 0x7fe8f8151760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.739 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.731+0000 7fe8fcc00640 1 -- 172.21.15.93:0/3705141581 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e80094d0 con 0x7fe8f815d0f0 2024-01-15T18:04:33.739 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.731+0000 7fe8f7400640 1 --2- 172.21.15.93:0/3705141581 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe8f806c430 0x7fe8f8156df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:04:33.739 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.731+0000 7fe8fcc00640 1 --2- 172.21.15.93:0/3705141581 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8f815d0f0 0x7fe8f8157330 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe8f000be50 tx=0x7fe8f000bf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:33.739 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.735+0000 7fe8efe00640 1 -- 172.21.15.93:0/3705141581 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f0014440 con 0x7fe8f815d0f0 2024-01-15T18:04:33.739 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.735+0000 7fe8efe00640 1 -- 172.21.15.93:0/3705141581 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f00145a0 con 0x7fe8f815d0f0 2024-01-15T18:04:33.739 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.735+0000 7fe8efe00640 1 -- 172.21.15.93:0/3705141581 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f0014950 con 0x7fe8f815d0f0 2024-01-15T18:04:33.739 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.735+0000 7fe8fec00640 1 -- 172.21.15.93:0/3705141581 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f80b1b00 con 0x7fe8f815d0f0 2024-01-15T18:04:33.739 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.735+0000 7fe8fec00640 1 -- 172.21.15.93:0/3705141581 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f80b20c0 con 0x7fe8f815d0f0 2024-01-15T18:04:33.739 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.735+0000 7fe8efe00640 1 -- 172.21.15.93:0/3705141581 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe8f0003980 con 0x7fe8f815d0f0 2024-01-15T18:04:33.739 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.735+0000 7fe8efe00640 1 -- 172.21.15.93:0/3705141581 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe8f0004460 con 0x7fe8f815d0f0 2024-01-15T18:04:33.739 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.735+0000 7fe8fec00640 1 -- 172.21.15.93:0/3705141581 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f804e850 con 0x7fe8f815d0f0 2024-01-15T18:04:33.763 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.759+0000 7fe8efe00640 1 -- 172.21.15.93:0/3705141581 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fe8f0019070 con 0x7fe8f815d0f0 2024-01-15T18:04:33.771 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-15T18:04:33.765+0000 7f49aec8d180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-15T18:04:33.783 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:04:33.779+0000 7f5a768cf180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-15T18:04:33.829 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.823+0000 7fe8fec00640 1 -- 172.21.15.93:0/3705141581 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf", "id": 0} v 0) v1 -- 0x7fe8f80b2570 con 0x7fe8f815d0f0 2024-01-15T18:04:33.834 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.831+0000 7fe8efe00640 1 -- 172.21.15.93:0/3705141581 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe8f0004620 con 0x7fe8f815d0f0 2024-01-15T18:04:33.834 INFO:teuthology.orchestra.run.smithi093.stdout:0 2024-01-15T18:04:33.835 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.831+0000 7fe8fec00640 1 -- 172.21.15.93:0/3705141581 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8f815d0f0 msgr2=0x7fe8f8157330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:33.835 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.831+0000 7fe8fec00640 1 --2- 172.21.15.93:0/3705141581 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8f815d0f0 0x7fe8f8157330 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe8f000be50 tx=0x7fe8f000bf40 comp rx=0 tx=0).stop 2024-01-15T18:04:33.835 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.831+0000 7fe8fec00640 1 -- 172.21.15.93:0/3705141581 shutdown_connections 2024-01-15T18:04:33.835 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.831+0000 7fe8fec00640 1 --2- 172.21.15.93:0/3705141581 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8f815d0f0 0x7fe8f8157330 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.835 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.831+0000 7fe8fec00640 1 --2- 172.21.15.93:0/3705141581 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe8f806c430 0x7fe8f8156df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.835 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.831+0000 7fe8fec00640 1 --2- 172.21.15.93:0/3705141581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8f806c050 0x7fe8f8151760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.835 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.831+0000 7fe8fec00640 1 -- 172.21.15.93:0/3705141581 >> 172.21.15.93:0/3705141581 conn(0x7fe8f8099a10 msgr2=0x7fe8f815e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:33.836 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.831+0000 7fe8fec00640 1 -- 172.21.15.93:0/3705141581 shutdown_connections 2024-01-15T18:04:33.836 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.831+0000 7fe8fec00640 1 -- 172.21.15.93:0/3705141581 wait complete. 2024-01-15T18:04:33.856 DEBUG:teuthology.orchestra.run.smithi093:> sudo ceph --cluster ceph osd new 619e3be6-1fe3-4d1a-a7ad-a613949aceb7 1 2024-01-15T18:04:33.990 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.987+0000 7fa8b3400640 1 Processor -- start 2024-01-15T18:04:33.990 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.987+0000 7fa8b3400640 1 -- start start 2024-01-15T18:04:33.992 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.987+0000 7fa8b3400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa8ac0ccf60 0x7fa8ac0cd340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:33.992 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.987+0000 7fa8b3400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa8ac0cc8a0 0x7fa8ac15cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:33.992 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.987+0000 7fa8b3400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8ac0cad40 con 0x7fa8ac0cc440 2024-01-15T18:04:33.992 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.987+0000 7fa8b3400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8ac0674f0 con 0x7fa8ac0ccf60 2024-01-15T18:04:33.992 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.987+0000 7fa8b3400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8ac0b1970 con 0x7fa8ac0cc8a0 2024-01-15T18:04:33.992 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.987+0000 7fa8b1400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa8ac0cc8a0 0x7fa8ac15cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:33.992 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.987+0000 7fa8b1400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa8ac0cc8a0 0x7fa8ac15cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:46866/0 (socket says 172.21.15.93:46866) 2024-01-15T18:04:33.992 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.987+0000 7fa8b1400640 1 -- 172.21.15.93:0/3038149655 learned_addr learned my addr 172.21.15.93:0/3038149655 (peer_addr_for_me v2:172.21.15.93:0/0) 2024-01-15T18:04:33.992 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.987+0000 7fa8b0a00640 1 --2- 172.21.15.93:0/3038149655 >> v2:172.21.15.27:3300/0 conn(0x7fa8ac0ccf60 0x7fa8ac0cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:33.993 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.987+0000 7fa8b1400640 1 -- 172.21.15.93:0/3038149655 >> v2:172.21.15.27:3300/0 conn(0x7fa8ac0ccf60 msgr2=0x7fa8ac0cd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:33.993 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.987+0000 7fa8b1400640 1 --2- 172.21.15.93:0/3038149655 >> v2:172.21.15.27:3300/0 conn(0x7fa8ac0ccf60 0x7fa8ac0cd340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.993 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.987+0000 7fa8b1400640 1 -- 172.21.15.93:0/3038149655 >> v1:172.21.15.27:6789/0 conn(0x7fa8ac0cc440 legacy=0x7fa8ac0ca190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:33.993 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.987+0000 7fa8b1400640 1 -- 172.21.15.93:0/3038149655 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ac067700 con 0x7fa8ac0cc8a0 2024-01-15T18:04:33.997 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.991+0000 7fa8b1400640 1 --2- 172.21.15.93:0/3038149655 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8ac0cc8a0 0x7fa8ac15cb90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac070150 tx=0x7fa8a401ea80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45cc181e1d57a0dd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:33.997 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.991+0000 7fa8ab400640 1 -- 172.21.15.93:0/3038149655 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a40a7b70 con 0x7fa8ac0cc8a0 2024-01-15T18:04:33.997 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.991+0000 7fa8ab400640 1 -- 172.21.15.93:0/3038149655 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a40ab070 con 0x7fa8ac0cc8a0 2024-01-15T18:04:33.997 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.991+0000 7fa8ab400640 1 -- 172.21.15.93:0/3038149655 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a40b1400 con 0x7fa8ac0cc8a0 2024-01-15T18:04:33.998 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.995+0000 7fa8b3400640 1 -- 172.21.15.93:0/3038149655 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8ac0cc8a0 msgr2=0x7fa8ac15cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:33.998 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.995+0000 7fa8b3400640 1 --2- 172.21.15.93:0/3038149655 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8ac0cc8a0 0x7fa8ac15cb90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac070150 tx=0x7fa8a401ea80 comp rx=0 tx=0).stop 2024-01-15T18:04:33.999 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.995+0000 7fa8b3400640 1 -- 172.21.15.93:0/3038149655 shutdown_connections 2024-01-15T18:04:33.999 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.995+0000 7fa8b3400640 1 --2- 172.21.15.93:0/3038149655 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8ac0cc8a0 0x7fa8ac15cb90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.999 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.995+0000 7fa8b3400640 1 --2- 172.21.15.93:0/3038149655 >> v2:172.21.15.27:3300/0 conn(0x7fa8ac0ccf60 0x7fa8ac0cd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.999 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.995+0000 7fa8b3400640 1 -- 172.21.15.93:0/3038149655 >> 172.21.15.93:0/3038149655 conn(0x7fa8ac0b0d20 msgr2=0x7fa8ac0b1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:33.999 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.995+0000 7fa8b3400640 1 -- 172.21.15.93:0/3038149655 shutdown_connections 2024-01-15T18:04:33.999 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.995+0000 7fa8b3400640 1 -- 172.21.15.93:0/3038149655 wait complete. 2024-01-15T18:04:34.000 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.995+0000 7fa8b3400640 1 Processor -- start 2024-01-15T18:04:34.000 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.995+0000 7fa8b3400640 1 -- start start 2024-01-15T18:04:34.001 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.995+0000 7fa8b3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8ac0cc440 0x7fa8ac157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:34.001 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.995+0000 7fa8b3400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8ac0cc8a0 0x7fa8ac0ae760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:34.002 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.995+0000 7fa8b3400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8ac0ccf60 0x7fa8ac0c51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:34.002 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.995+0000 7fa8b3400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ac0adde0 con 0x7fa8ac0cc440 2024-01-15T18:04:34.002 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.995+0000 7fa8b3400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ac0ad1d0 con 0x7fa8ac0cc8a0 2024-01-15T18:04:34.002 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.995+0000 7fa8b3400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8ac0ad4b0 con 0x7fa8ac0ccf60 2024-01-15T18:04:34.003 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.999+0000 7fa8abe00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8ac0cc8a0 0x7fa8ac0ae760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:34.003 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.999+0000 7fa8abe00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8ac0cc8a0 0x7fa8ac0ae760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:46870/0 (socket says 172.21.15.93:46870) 2024-01-15T18:04:34.003 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.999+0000 7fa8abe00640 1 -- 172.21.15.93:0/1928787063 learned_addr learned my addr 172.21.15.93:0/1928787063 (peer_addr_for_me v2:172.21.15.93:0/0) 2024-01-15T18:04:34.003 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.999+0000 7fa8b1400640 1 --2- 172.21.15.93:0/1928787063 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8ac0ccf60 0x7fa8ac0c51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:34.004 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.999+0000 7fa8b0a00640 1 --2- 172.21.15.93:0/1928787063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8ac0cc440 0x7fa8ac157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:34.004 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.999+0000 7fa8abe00640 1 -- 172.21.15.93:0/1928787063 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8ac0ccf60 msgr2=0x7fa8ac0c51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:34.004 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.999+0000 7fa8abe00640 1 --2- 172.21.15.93:0/1928787063 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8ac0ccf60 0x7fa8ac0c51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.004 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.999+0000 7fa8abe00640 1 -- 172.21.15.93:0/1928787063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8ac0cc440 msgr2=0x7fa8ac157d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:34.004 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.999+0000 7fa8abe00640 1 --2- 172.21.15.93:0/1928787063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8ac0cc440 0x7fa8ac157d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.004 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.999+0000 7fa8abe00640 1 -- 172.21.15.93:0/1928787063 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a400bd40 con 0x7fa8ac0cc8a0 2024-01-15T18:04:34.004 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.999+0000 7fa8b0a00640 1 --2- 172.21.15.93:0/1928787063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8ac0cc440 0x7fa8ac157d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:04:34.004 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.999+0000 7fa8abe00640 1 --2- 172.21.15.93:0/1928787063 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8ac0cc8a0 0x7fa8ac0ae760 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa89400c500 tx=0x7fa89400c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:34.004 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.999+0000 7fa8a8c00640 1 -- 172.21.15.93:0/1928787063 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa894005220 con 0x7fa8ac0cc8a0 2024-01-15T18:04:34.004 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.999+0000 7fa8a8c00640 1 -- 172.21.15.93:0/1928787063 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa894005380 con 0x7fa8ac0cc8a0 2024-01-15T18:04:34.004 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.999+0000 7fa8a8c00640 1 -- 172.21.15.93:0/1928787063 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa894007270 con 0x7fa8ac0cc8a0 2024-01-15T18:04:34.004 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.999+0000 7fa8b3400640 1 -- 172.21.15.93:0/1928787063 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8ac0ad790 con 0x7fa8ac0cc8a0 2024-01-15T18:04:34.004 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.999+0000 7fa8b3400640 1 -- 172.21.15.93:0/1928787063 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8ac158250 con 0x7fa8ac0cc8a0 2024-01-15T18:04:34.005 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.999+0000 7fa8a8c00640 1 -- 172.21.15.93:0/1928787063 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa8940067e0 con 0x7fa8ac0cc8a0 2024-01-15T18:04:34.005 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.999+0000 7fa8a8c00640 1 -- 172.21.15.93:0/1928787063 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7fa894012af0 con 0x7fa8ac0cc8a0 2024-01-15T18:04:34.005 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-15T18:04:33.997+0000 7f49aec8d180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-15T18:04:34.005 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:33.999+0000 7fa8b3400640 1 -- 172.21.15.93:0/1928787063 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8ac04e850 con 0x7fa8ac0cc8a0 2024-01-15T18:04:34.007 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.003+0000 7fa8a8c00640 1 -- 172.21.15.93:0/1928787063 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fa894016020 con 0x7fa8ac0cc8a0 2024-01-15T18:04:34.040 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:04:34.035+0000 7f5a768cf180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-15T18:04:34.100 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.095+0000 7fa8b3400640 1 -- 172.21.15.93:0/1928787063 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd new", "uuid": "619e3be6-1fe3-4d1a-a7ad-a613949aceb7", "id": 1} v 0) v1 -- 0x7fa8ac072870 con 0x7fa8ac0cc8a0 2024-01-15T18:04:34.109 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.103+0000 7fa8a8c00640 1 -- 172.21.15.93:0/1928787063 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "619e3be6-1fe3-4d1a-a7ad-a613949aceb7", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa89402a0b0 con 0x7fa8ac0cc8a0 2024-01-15T18:04:34.109 INFO:teuthology.orchestra.run.smithi093.stdout:1 2024-01-15T18:04:34.110 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.107+0000 7fa8b3400640 1 -- 172.21.15.93:0/1928787063 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8ac0cc8a0 msgr2=0x7fa8ac0ae760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:34.111 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.107+0000 7fa8b3400640 1 --2- 172.21.15.93:0/1928787063 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8ac0cc8a0 0x7fa8ac0ae760 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa89400c500 tx=0x7fa89400c9c0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.111 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.107+0000 7fa8b3400640 1 -- 172.21.15.93:0/1928787063 shutdown_connections 2024-01-15T18:04:34.111 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.107+0000 7fa8b3400640 1 --2- 172.21.15.93:0/1928787063 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8ac0ccf60 0x7fa8ac0c51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.111 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.107+0000 7fa8b3400640 1 --2- 172.21.15.93:0/1928787063 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8ac0cc8a0 0x7fa8ac0ae760 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.111 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.107+0000 7fa8b3400640 1 --2- 172.21.15.93:0/1928787063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8ac0cc440 0x7fa8ac157d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.111 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.107+0000 7fa8b3400640 1 -- 172.21.15.93:0/1928787063 >> 172.21.15.93:0/1928787063 conn(0x7fa8ac0b0d20 msgr2=0x7fa8ac158ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:34.111 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.107+0000 7fa8b3400640 1 -- 172.21.15.93:0/1928787063 shutdown_connections 2024-01-15T18:04:34.111 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.107+0000 7fa8b3400640 1 -- 172.21.15.93:0/1928787063 wait complete. 2024-01-15T18:04:34.130 DEBUG:teuthology.orchestra.run.smithi093:> sudo ceph --cluster ceph osd new 2996e145-a7bb-4e3d-9159-da33ce9f4851 2 2024-01-15T18:04:34.263 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.259+0000 7f7f26600640 1 Processor -- start 2024-01-15T18:04:34.263 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.259+0000 7f7f26600640 1 -- start start 2024-01-15T18:04:34.265 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.259+0000 7f7f26600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7f2015a240 0x7f7f2006b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:34.265 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.259+0000 7f7f26600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f7f200ccd80 0x7f7f2015e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:34.265 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.259+0000 7f7f26600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f200b2ef0 con 0x7f7f2015dbe0 2024-01-15T18:04:34.265 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.259+0000 7f7f26600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f200aef70 con 0x7f7f2015a240 2024-01-15T18:04:34.265 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.259+0000 7f7f26600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f2006bd70 con 0x7f7f200ccd80 2024-01-15T18:04:34.265 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.259+0000 7f7f1fe00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f7f200ccd80 0x7f7f2015e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:34.265 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.259+0000 7f7f1fe00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f7f200ccd80 0x7f7f2015e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:46878/0 (socket says 172.21.15.93:46878) 2024-01-15T18:04:34.265 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.259+0000 7f7f1fe00640 1 -- 172.21.15.93:0/1175809598 learned_addr learned my addr 172.21.15.93:0/1175809598 (peer_addr_for_me v2:172.21.15.93:0/0) 2024-01-15T18:04:34.265 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.259+0000 7f7f1ea00640 1 --2- 172.21.15.93:0/1175809598 >> v2:172.21.15.27:3300/0 conn(0x7f7f2015a240 0x7f7f2006b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:34.266 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.259+0000 7f7f1fe00640 1 -- 172.21.15.93:0/1175809598 >> v2:172.21.15.27:3300/0 conn(0x7f7f2015a240 msgr2=0x7f7f2006b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:34.266 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.259+0000 7f7f1fe00640 1 --2- 172.21.15.93:0/1175809598 >> v2:172.21.15.27:3300/0 conn(0x7f7f2015a240 0x7f7f2006b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.266 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.259+0000 7f7f1fe00640 1 -- 172.21.15.93:0/1175809598 >> v1:172.21.15.27:6789/0 conn(0x7f7f2015dbe0 legacy=0x7f7f2015dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:34.266 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.259+0000 7f7f1fe00640 1 -- 172.21.15.93:0/1175809598 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f200b25c0 con 0x7f7f200ccd80 2024-01-15T18:04:34.268 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.263+0000 7f7f1fe00640 1 --2- 172.21.15.93:0/1175809598 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f200ccd80 0x7f7f2015e6d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7f20066800 tx=0x7f7f1801c9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47cef819aaca1420 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:34.269 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.263+0000 7f7f1e000640 1 -- 172.21.15.93:0/1175809598 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f180a5860 con 0x7f7f200ccd80 2024-01-15T18:04:34.269 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.263+0000 7f7f1e000640 1 -- 172.21.15.93:0/1175809598 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f180a59c0 con 0x7f7f200ccd80 2024-01-15T18:04:34.269 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.263+0000 7f7f1e000640 1 -- 172.21.15.93:0/1175809598 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f180a5d50 con 0x7f7f200ccd80 2024-01-15T18:04:34.271 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.263+0000 7f7f26600640 1 -- 172.21.15.93:0/1175809598 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f200ccd80 msgr2=0x7f7f2015e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:34.271 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.263+0000 7f7f26600640 1 --2- 172.21.15.93:0/1175809598 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f200ccd80 0x7f7f2015e6d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7f20066800 tx=0x7f7f1801c9a0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.271 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.263+0000 7f7f26600640 1 -- 172.21.15.93:0/1175809598 shutdown_connections 2024-01-15T18:04:34.271 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.263+0000 7f7f26600640 1 --2- 172.21.15.93:0/1175809598 >> v2:172.21.15.27:3300/0 conn(0x7f7f2015a240 0x7f7f2006b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.271 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.263+0000 7f7f26600640 1 --2- 172.21.15.93:0/1175809598 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f200ccd80 0x7f7f2015e6d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.271 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.263+0000 7f7f26600640 1 -- 172.21.15.93:0/1175809598 >> 172.21.15.93:0/1175809598 conn(0x7f7f200cf510 msgr2=0x7f7f200cf920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:34.272 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.267+0000 7f7f26600640 1 -- 172.21.15.93:0/1175809598 shutdown_connections 2024-01-15T18:04:34.272 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.267+0000 7f7f26600640 1 -- 172.21.15.93:0/1175809598 wait complete. 2024-01-15T18:04:34.272 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.267+0000 7f7f26600640 1 Processor -- start 2024-01-15T18:04:34.272 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.267+0000 7f7f26600640 1 -- start start 2024-01-15T18:04:34.274 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.267+0000 7f7f26600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f200ccd80 0x7f7f20151660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:34.274 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.267+0000 7f7f26600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f2015a240 0x7f7f2014ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:34.274 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.267+0000 7f7f26600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7f2015dbe0 0x7f7f2014e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:34.274 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.267+0000 7f7f26600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f20151ba0 con 0x7f7f200ccd80 2024-01-15T18:04:34.274 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.267+0000 7f7f26600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f20152750 con 0x7f7f2015a240 2024-01-15T18:04:34.274 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.267+0000 7f7f26600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2015c2b0 con 0x7f7f2015dbe0 2024-01-15T18:04:34.274 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.267+0000 7f7f1ea00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f2015a240 0x7f7f2014ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:34.274 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.267+0000 7f7f1fe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7f2015dbe0 0x7f7f2014e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:34.274 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.267+0000 7f7f1ea00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f2015a240 0x7f7f2014ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:46880/0 (socket says 172.21.15.93:46880) 2024-01-15T18:04:34.275 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.267+0000 7f7f1ea00640 1 -- 172.21.15.93:0/1771983154 learned_addr learned my addr 172.21.15.93:0/1771983154 (peer_addr_for_me v2:172.21.15.93:0/0) 2024-01-15T18:04:34.275 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.267+0000 7f7f1f400640 1 --2- 172.21.15.93:0/1771983154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f200ccd80 0x7f7f20151660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:34.275 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.267+0000 7f7f1ea00640 1 -- 172.21.15.93:0/1771983154 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7f2015dbe0 msgr2=0x7f7f2014e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:34.275 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.267+0000 7f7f1ea00640 1 --2- 172.21.15.93:0/1771983154 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7f2015dbe0 0x7f7f2014e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.275 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.267+0000 7f7f1ea00640 1 -- 172.21.15.93:0/1771983154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f200ccd80 msgr2=0x7f7f20151660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:34.275 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.267+0000 7f7f1ea00640 1 --2- 172.21.15.93:0/1771983154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f200ccd80 0x7f7f20151660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.275 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.267+0000 7f7f1ea00640 1 -- 172.21.15.93:0/1771983154 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f18009b30 con 0x7f7f2015a240 2024-01-15T18:04:34.275 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.267+0000 7f7f1ea00640 1 --2- 172.21.15.93:0/1771983154 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f2015a240 0x7f7f2014ce10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7f0800e930 tx=0x7f7f0800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:34.275 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.271+0000 7f7f17400640 1 -- 172.21.15.93:0/1771983154 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f08009690 con 0x7f7f2015a240 2024-01-15T18:04:34.275 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.271+0000 7f7f17400640 1 -- 172.21.15.93:0/1771983154 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f0800ca00 con 0x7f7f2015a240 2024-01-15T18:04:34.275 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.271+0000 7f7f17400640 1 -- 172.21.15.93:0/1771983154 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0800cd70 con 0x7f7f2015a240 2024-01-15T18:04:34.275 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.271+0000 7f7f26600640 1 -- 172.21.15.93:0/1771983154 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f2015c590 con 0x7f7f2015a240 2024-01-15T18:04:34.275 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.271+0000 7f7f26600640 1 -- 172.21.15.93:0/1771983154 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f2015cb50 con 0x7f7f2015a240 2024-01-15T18:04:34.275 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.271+0000 7f7f17400640 1 -- 172.21.15.93:0/1771983154 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7f08005ca0 con 0x7f7f2015a240 2024-01-15T18:04:34.275 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.271+0000 7f7f17400640 1 -- 172.21.15.93:0/1771983154 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f7f08013890 con 0x7f7f2015a240 2024-01-15T18:04:34.276 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.271+0000 7f7f26600640 1 -- 172.21.15.93:0/1771983154 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f2014b060 con 0x7f7f2015a240 2024-01-15T18:04:34.277 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.271+0000 7f7f17400640 1 -- 172.21.15.93:0/1771983154 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f7f08021060 con 0x7f7f2015a240 2024-01-15T18:04:34.342 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.339+0000 7f7f26600640 1 -- 172.21.15.93:0/1771983154 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2996e145-a7bb-4e3d-9159-da33ce9f4851", "id": 2} v 0) v1 -- 0x7f7f2004e850 con 0x7f7f2015a240 2024-01-15T18:04:34.351 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.347+0000 7f7f17400640 1 -- 172.21.15.93:0/1771983154 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2996e145-a7bb-4e3d-9159-da33ce9f4851", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7f08017070 con 0x7f7f2015a240 2024-01-15T18:04:34.351 INFO:teuthology.orchestra.run.smithi093.stdout:2 2024-01-15T18:04:34.352 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.347+0000 7f7f26600640 1 -- 172.21.15.93:0/1771983154 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f2015a240 msgr2=0x7f7f2014ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:34.352 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.347+0000 7f7f26600640 1 --2- 172.21.15.93:0/1771983154 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f2015a240 0x7f7f2014ce10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7f0800e930 tx=0x7f7f0800edf0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.352 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.347+0000 7f7f26600640 1 -- 172.21.15.93:0/1771983154 shutdown_connections 2024-01-15T18:04:34.353 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.347+0000 7f7f26600640 1 --2- 172.21.15.93:0/1771983154 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7f2015dbe0 0x7f7f2014e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.353 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.347+0000 7f7f26600640 1 --2- 172.21.15.93:0/1771983154 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7f2015a240 0x7f7f2014ce10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.353 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.347+0000 7f7f26600640 1 --2- 172.21.15.93:0/1771983154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7f200ccd80 0x7f7f20151660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.353 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.347+0000 7f7f26600640 1 -- 172.21.15.93:0/1771983154 >> 172.21.15.93:0/1771983154 conn(0x7f7f200cf510 msgr2=0x7f7f200ce310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:34.353 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.347+0000 7f7f26600640 1 -- 172.21.15.93:0/1771983154 shutdown_connections 2024-01-15T18:04:34.353 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.347+0000 7f7f26600640 1 -- 172.21.15.93:0/1771983154 wait complete. 2024-01-15T18:04:34.370 DEBUG:teuthology.orchestra.run.smithi093:> sudo ceph --cluster ceph osd new d3b5a604-2395-4cef-b9ec-bba975a26f40 3 2024-01-15T18:04:34.505 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.499+0000 7f917f600640 1 Processor -- start 2024-01-15T18:04:34.505 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.499+0000 7f917f600640 1 -- start start 2024-01-15T18:04:34.506 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.503+0000 7f917f600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f9178153ba0 0x7f917806b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:34.507 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.503+0000 7f917f600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f917806d4a0 0x7f91780b2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:34.507 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.503+0000 7f917f600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f917806db10 con 0x7f917806ba40 2024-01-15T18:04:34.507 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.503+0000 7f917f600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f917806bef0 con 0x7f917806d4a0 2024-01-15T18:04:34.507 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.503+0000 7f917f600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f917806e690 con 0x7f9178153ba0 2024-01-15T18:04:34.507 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.503+0000 7f917cc00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f9178153ba0 0x7f917806b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:34.507 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.503+0000 7f917cc00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f9178153ba0 0x7f917806b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:46882/0 (socket says 172.21.15.93:46882) 2024-01-15T18:04:34.507 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.503+0000 7f917cc00640 1 -- 172.21.15.93:0/1828118589 learned_addr learned my addr 172.21.15.93:0/1828118589 (peer_addr_for_me v2:172.21.15.93:0/0) 2024-01-15T18:04:34.507 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.503+0000 7f917d600640 1 --2- 172.21.15.93:0/1828118589 >> v2:172.21.15.27:3300/0 conn(0x7f917806d4a0 0x7f91780b2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:34.507 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.503+0000 7f917cc00640 1 -- 172.21.15.93:0/1828118589 >> v2:172.21.15.27:3300/0 conn(0x7f917806d4a0 msgr2=0x7f91780b2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:34.507 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.503+0000 7f917cc00640 1 --2- 172.21.15.93:0/1828118589 >> v2:172.21.15.27:3300/0 conn(0x7f917806d4a0 0x7f91780b2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.507 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.503+0000 7f917cc00640 1 -- 172.21.15.93:0/1828118589 >> v1:172.21.15.93:6789/0 conn(0x7f917806ba40 legacy=0x7f917806cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:34.507 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.503+0000 7f917cc00640 1 -- 172.21.15.93:0/1828118589 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f917806ddf0 con 0x7f9178153ba0 2024-01-15T18:04:34.509 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.507+0000 7f917cc00640 1 --2- 172.21.15.93:0/1828118589 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9178153ba0 0x7f917806b500 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9178072270 tx=0x7f916801c500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4dea5bef730110d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:34.510 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.507+0000 7f9177400640 1 -- 172.21.15.93:0/1828118589 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91680ab600 con 0x7f9178153ba0 2024-01-15T18:04:34.510 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.507+0000 7f9177400640 1 -- 172.21.15.93:0/1828118589 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91680a7050 con 0x7f9178153ba0 2024-01-15T18:04:34.510 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.507+0000 7f9177400640 1 -- 172.21.15.93:0/1828118589 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91680ab980 con 0x7f9178153ba0 2024-01-15T18:04:34.511 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.507+0000 7f917f600640 1 -- 172.21.15.93:0/1828118589 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9178153ba0 msgr2=0x7f917806b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:34.511 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.507+0000 7f917f600640 1 --2- 172.21.15.93:0/1828118589 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9178153ba0 0x7f917806b500 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9178072270 tx=0x7f916801c500 comp rx=0 tx=0).stop 2024-01-15T18:04:34.511 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.507+0000 7f917f600640 1 -- 172.21.15.93:0/1828118589 shutdown_connections 2024-01-15T18:04:34.511 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.507+0000 7f917f600640 1 --2- 172.21.15.93:0/1828118589 >> v2:172.21.15.27:3300/0 conn(0x7f917806d4a0 0x7f91780b2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.511 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.507+0000 7f917f600640 1 --2- 172.21.15.93:0/1828118589 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9178153ba0 0x7f917806b500 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.511 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.507+0000 7f917f600640 1 -- 172.21.15.93:0/1828118589 >> 172.21.15.93:0/1828118589 conn(0x7f9178070ad0 msgr2=0x7f9178070ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:34.511 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.507+0000 7f917f600640 1 -- 172.21.15.93:0/1828118589 shutdown_connections 2024-01-15T18:04:34.511 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.507+0000 7f917f600640 1 -- 172.21.15.93:0/1828118589 wait complete. 2024-01-15T18:04:34.513 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.507+0000 7f917f600640 1 Processor -- start 2024-01-15T18:04:34.513 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.507+0000 7f917f600640 1 -- start start 2024-01-15T18:04:34.514 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.511+0000 7f917f600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917806ba40 0x7f917813e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:34.514 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.511+0000 7f917f600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f917806d4a0 0x7f91781472b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:34.514 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.511+0000 7f917f600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9178153ba0 0x7f9178145750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:34.514 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.511+0000 7f917f600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9178146e20 con 0x7f917806ba40 2024-01-15T18:04:34.514 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.511+0000 7f917f600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917813e780 con 0x7f917806d4a0 2024-01-15T18:04:34.515 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.511+0000 7f917f600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9178146210 con 0x7f9178153ba0 2024-01-15T18:04:34.515 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.511+0000 7f917d600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9178153ba0 0x7f9178145750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:34.515 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.511+0000 7f9177e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f917806d4a0 0x7f91781472b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:34.515 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.511+0000 7f917d600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9178153ba0 0x7f9178145750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.93:37922/0 (socket says 172.21.15.93:37922) 2024-01-15T18:04:34.515 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.511+0000 7f917d600640 1 -- 172.21.15.93:0/1876041761 learned_addr learned my addr 172.21.15.93:0/1876041761 (peer_addr_for_me v2:172.21.15.93:0/0) 2024-01-15T18:04:34.515 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.511+0000 7f917cc00640 1 --2- 172.21.15.93:0/1876041761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917806ba40 0x7f917813e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:34.516 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.511+0000 7f917d600640 1 -- 172.21.15.93:0/1876041761 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f917806d4a0 msgr2=0x7f91781472b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:34.516 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.511+0000 7f917d600640 1 --2- 172.21.15.93:0/1876041761 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f917806d4a0 0x7f91781472b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.516 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.511+0000 7f917d600640 1 -- 172.21.15.93:0/1876041761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917806ba40 msgr2=0x7f917813e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:34.516 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.511+0000 7f917d600640 1 --2- 172.21.15.93:0/1876041761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917806ba40 0x7f917813e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.517 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.511+0000 7f917d600640 1 -- 172.21.15.93:0/1876041761 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91680094d0 con 0x7f9178153ba0 2024-01-15T18:04:34.517 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.511+0000 7f917d600640 1 --2- 172.21.15.93:0/1876041761 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9178153ba0 0x7f9178145750 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9170007e40 tx=0x7f917000e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:34.517 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.511+0000 7f9174c00640 1 -- 172.21.15.93:0/1876041761 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9170014620 con 0x7f9178153ba0 2024-01-15T18:04:34.517 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.511+0000 7f9174c00640 1 -- 172.21.15.93:0/1876041761 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f917000ee90 con 0x7f9178153ba0 2024-01-15T18:04:34.517 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.511+0000 7f9174c00640 1 -- 172.21.15.93:0/1876041761 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91700149e0 con 0x7f9178153ba0 2024-01-15T18:04:34.517 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.511+0000 7f917f600640 1 -- 172.21.15.93:0/1876041761 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91781464f0 con 0x7f9178153ba0 2024-01-15T18:04:34.517 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.511+0000 7f917f600640 1 -- 172.21.15.93:0/1876041761 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9178146ab0 con 0x7f9178153ba0 2024-01-15T18:04:34.517 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.511+0000 7f9174c00640 1 -- 172.21.15.93:0/1876041761 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f917001e460 con 0x7f9178153ba0 2024-01-15T18:04:34.517 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.511+0000 7f9174c00640 1 -- 172.21.15.93:0/1876041761 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f917001d850 con 0x7f9178153ba0 2024-01-15T18:04:34.517 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.511+0000 7f917f600640 1 -- 172.21.15.93:0/1876041761 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9178146f90 con 0x7f9178153ba0 2024-01-15T18:04:34.521 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.515+0000 7f9174c00640 1 -- 172.21.15.93:0/1876041761 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f917001d3c0 con 0x7f9178153ba0 2024-01-15T18:04:34.587 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.583+0000 7f917f600640 1 -- 172.21.15.93:0/1876041761 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d3b5a604-2395-4cef-b9ec-bba975a26f40", "id": 3} v 0) v1 -- 0x7f917804e850 con 0x7f9178153ba0 2024-01-15T18:04:34.596 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.591+0000 7f9174c00640 1 -- 172.21.15.93:0/1876041761 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d3b5a604-2395-4cef-b9ec-bba975a26f40", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f917001db30 con 0x7f9178153ba0 2024-01-15T18:04:34.596 INFO:teuthology.orchestra.run.smithi093.stdout:3 2024-01-15T18:04:34.597 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.591+0000 7f917f600640 1 -- 172.21.15.93:0/1876041761 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9178153ba0 msgr2=0x7f9178145750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:34.597 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.591+0000 7f917f600640 1 --2- 172.21.15.93:0/1876041761 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9178153ba0 0x7f9178145750 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9170007e40 tx=0x7f917000e4b0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.597 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.591+0000 7f917f600640 1 -- 172.21.15.93:0/1876041761 shutdown_connections 2024-01-15T18:04:34.597 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.591+0000 7f917f600640 1 --2- 172.21.15.93:0/1876041761 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9178153ba0 0x7f9178145750 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.598 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.591+0000 7f917f600640 1 --2- 172.21.15.93:0/1876041761 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f917806d4a0 0x7f91781472b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.598 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.591+0000 7f917f600640 1 --2- 172.21.15.93:0/1876041761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917806ba40 0x7f917813e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.598 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.591+0000 7f917f600640 1 -- 172.21.15.93:0/1876041761 >> 172.21.15.93:0/1876041761 conn(0x7f9178070ad0 msgr2=0x7f91781516d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:34.598 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.591+0000 7f917f600640 1 -- 172.21.15.93:0/1876041761 shutdown_connections 2024-01-15T18:04:34.598 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.591+0000 7f917f600640 1 -- 172.21.15.93:0/1876041761 wait complete. 2024-01-15T18:04:34.617 DEBUG:teuthology.orchestra.run.smithi093:> sudo ceph --cluster ceph osd new 04c19746-6de6-45b1-aaa3-3d8e96469592 4 2024-01-15T18:04:34.707 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:04:34.703+0000 7f5a768cf180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-15T18:04:34.714 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-15T18:04:34.709+0000 7f49aec8d180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-15T18:04:34.751 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.747+0000 7f5b32000640 1 Processor -- start 2024-01-15T18:04:34.751 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.747+0000 7f5b32000640 1 -- start start 2024-01-15T18:04:34.753 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.747+0000 7f5b32000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b2c06ac00 0x7f5b2c15e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:34.753 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.747+0000 7f5b32000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2c0b2870 con 0x7f5b2c157a50 2024-01-15T18:04:34.753 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.747+0000 7f5b32000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2c06b0b0 con 0x7f5b2c070d50 2024-01-15T18:04:34.753 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.747+0000 7f5b32000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c15d8e0 con 0x7f5b2c06ac00 2024-01-15T18:04:34.753 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.747+0000 7f5b2aa00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b2c06ac00 0x7f5b2c15e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:34.753 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.747+0000 7f5b2be00640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7f5b2c070d50 0x7f5b2c160dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.93:41424/0 (socket says 172.21.15.93:41424) 2024-01-15T18:04:34.753 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.747+0000 7f5b2be00640 1 -- 172.21.15.93:0/1729929276 learned_addr learned my addr 172.21.15.93:0/1729929276 (peer_addr_for_me v1:172.21.15.93:0/0) 2024-01-15T18:04:34.753 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.747+0000 7f5b2aa00640 1 -- 172.21.15.93:0/1729929276 >> v1:172.21.15.93:6789/0 conn(0x7f5b2c070d50 legacy=0x7f5b2c160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:34.754 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.747+0000 7f5b2a000640 1 -- 172.21.15.93:0/1729929276 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1675423757 0 0) 0x7f5b2c06b0b0 con 0x7f5b2c070d50 2024-01-15T18:04:34.754 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.747+0000 7f5b2aa00640 1 -- 172.21.15.93:0/1729929276 >> v1:172.21.15.27:6789/0 conn(0x7f5b2c157a50 legacy=0x7f5b2c06a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:34.754 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.747+0000 7f5b2aa00640 1 -- 172.21.15.93:0/1729929276 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b2c15dbc0 con 0x7f5b2c06ac00 2024-01-15T18:04:34.756 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.751+0000 7f5b2aa00640 1 --2- 172.21.15.93:0/1729929276 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b2c06ac00 0x7f5b2c15e3c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5b140097f0 tx=0x7f5b1401cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cc089dd1634f425 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:34.756 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.751+0000 7f5b2a000640 1 -- 172.21.15.93:0/1729929276 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b140aa610 con 0x7f5b2c06ac00 2024-01-15T18:04:34.756 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.751+0000 7f5b2a000640 1 -- 172.21.15.93:0/1729929276 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b140a6040 con 0x7f5b2c06ac00 2024-01-15T18:04:34.756 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.751+0000 7f5b2a000640 1 -- 172.21.15.93:0/1729929276 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b140aa990 con 0x7f5b2c06ac00 2024-01-15T18:04:34.757 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.751+0000 7f5b32000640 1 -- 172.21.15.93:0/1729929276 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b2c06ac00 msgr2=0x7f5b2c15e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:34.757 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.751+0000 7f5b32000640 1 --2- 172.21.15.93:0/1729929276 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b2c06ac00 0x7f5b2c15e3c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5b140097f0 tx=0x7f5b1401cef0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.758 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.751+0000 7f5b32000640 1 -- 172.21.15.93:0/1729929276 shutdown_connections 2024-01-15T18:04:34.758 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.751+0000 7f5b32000640 1 --2- 172.21.15.93:0/1729929276 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b2c06ac00 0x7f5b2c15e3c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.758 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.751+0000 7f5b32000640 1 -- 172.21.15.93:0/1729929276 >> 172.21.15.93:0/1729929276 conn(0x7f5b2c0b1620 msgr2=0x7f5b2c0b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:34.758 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.755+0000 7f5b32000640 1 -- 172.21.15.93:0/1729929276 shutdown_connections 2024-01-15T18:04:34.758 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.755+0000 7f5b32000640 1 -- 172.21.15.93:0/1729929276 wait complete. 2024-01-15T18:04:34.759 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.755+0000 7f5b32000640 1 Processor -- start 2024-01-15T18:04:34.759 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.755+0000 7f5b32000640 1 -- start start 2024-01-15T18:04:34.761 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.755+0000 7f5b32000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b2c06ac00 0x7f5b2c153e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:34.761 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.755+0000 7f5b32000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b2c070d50 0x7f5b2c1504e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:34.761 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.755+0000 7f5b32000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b2c157a50 0x7f5b2c14e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:34.761 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.755+0000 7f5b32000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c152a90 con 0x7f5b2c157a50 2024-01-15T18:04:34.761 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.755+0000 7f5b32000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c150ce0 con 0x7f5b2c06ac00 2024-01-15T18:04:34.761 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.755+0000 7f5b32000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c152130 con 0x7f5b2c070d50 2024-01-15T18:04:34.761 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.755+0000 7f5b2aa00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b2c070d50 0x7f5b2c1504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:34.761 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.755+0000 7f5b2b400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b2c06ac00 0x7f5b2c153e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:34.761 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.755+0000 7f5b2aa00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b2c070d50 0x7f5b2c1504e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.93:37948/0 (socket says 172.21.15.93:37948) 2024-01-15T18:04:34.761 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.755+0000 7f5b2aa00640 1 -- 172.21.15.93:0/2456144435 learned_addr learned my addr 172.21.15.93:0/2456144435 (peer_addr_for_me v2:172.21.15.93:0/0) 2024-01-15T18:04:34.761 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.755+0000 7f5b2be00640 1 --2- 172.21.15.93:0/2456144435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b2c157a50 0x7f5b2c14e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:34.763 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.755+0000 7f5b2b400640 1 -- 172.21.15.93:0/2456144435 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b2c070d50 msgr2=0x7f5b2c1504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:34.763 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.755+0000 7f5b2b400640 1 --2- 172.21.15.93:0/2456144435 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b2c070d50 0x7f5b2c1504e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.763 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.755+0000 7f5b2b400640 1 -- 172.21.15.93:0/2456144435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b2c157a50 msgr2=0x7f5b2c14e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:34.763 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.755+0000 7f5b2b400640 1 --2- 172.21.15.93:0/2456144435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b2c157a50 0x7f5b2c14e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.763 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.755+0000 7f5b2b400640 1 -- 172.21.15.93:0/2456144435 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b14009420 con 0x7f5b2c06ac00 2024-01-15T18:04:34.763 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.755+0000 7f5b2aa00640 1 --2- 172.21.15.93:0/2456144435 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b2c070d50 0x7f5b2c1504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:04:34.763 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.759+0000 7f5b2b400640 1 --2- 172.21.15.93:0/2456144435 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b2c06ac00 0x7f5b2c153e10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c00c390 tx=0x7f5b1c00c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:34.763 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.759+0000 7f5b23400640 1 -- 172.21.15.93:0/2456144435 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c0052d0 con 0x7f5b2c06ac00 2024-01-15T18:04:34.763 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.759+0000 7f5b23400640 1 -- 172.21.15.93:0/2456144435 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1c005430 con 0x7f5b2c06ac00 2024-01-15T18:04:34.763 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.759+0000 7f5b32000640 1 -- 172.21.15.93:0/2456144435 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b2c150a20 con 0x7f5b2c06ac00 2024-01-15T18:04:34.763 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.759+0000 7f5b23400640 1 -- 172.21.15.93:0/2456144435 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c003840 con 0x7f5b2c06ac00 2024-01-15T18:04:34.763 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.759+0000 7f5b32000640 1 -- 172.21.15.93:0/2456144435 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b2c1526b0 con 0x7f5b2c06ac00 2024-01-15T18:04:34.763 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.759+0000 7f5b23400640 1 -- 172.21.15.93:0/2456144435 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5b1c003ca0 con 0x7f5b2c06ac00 2024-01-15T18:04:34.763 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.759+0000 7f5b23400640 1 -- 172.21.15.93:0/2456144435 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f5b1c012930 con 0x7f5b2c06ac00 2024-01-15T18:04:34.764 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.759+0000 7f5b32000640 1 -- 172.21.15.93:0/2456144435 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b2c04e850 con 0x7f5b2c06ac00 2024-01-15T18:04:34.765 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.763+0000 7f5b23400640 1 -- 172.21.15.93:0/2456144435 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f5b1c017070 con 0x7f5b2c06ac00 2024-01-15T18:04:34.775 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:04:34.771+0000 7f5a768cf180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-15T18:04:34.780 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-15T18:04:34.773+0000 7f49aec8d180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-15T18:04:34.831 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.827+0000 7f5b32000640 1 -- 172.21.15.93:0/2456144435 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd new", "uuid": "04c19746-6de6-45b1-aaa3-3d8e96469592", "id": 4} v 0) v1 -- 0x7f5b2c06e970 con 0x7f5b2c06ac00 2024-01-15T18:04:34.836 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:04:34.831+0000 7f5a768cf180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-15T18:04:34.838 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.835+0000 7f5b23400640 1 -- 172.21.15.93:0/2456144435 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "04c19746-6de6-45b1-aaa3-3d8e96469592", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5b1c012b40 con 0x7f5b2c06ac00 2024-01-15T18:04:34.838 INFO:teuthology.orchestra.run.smithi093.stdout:4 2024-01-15T18:04:34.839 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.835+0000 7f5b32000640 1 -- 172.21.15.93:0/2456144435 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b2c06ac00 msgr2=0x7f5b2c153e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:34.839 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.835+0000 7f5b32000640 1 --2- 172.21.15.93:0/2456144435 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b2c06ac00 0x7f5b2c153e10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c00c390 tx=0x7f5b1c00c850 comp rx=0 tx=0).stop 2024-01-15T18:04:34.840 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.835+0000 7f5b32000640 1 -- 172.21.15.93:0/2456144435 shutdown_connections 2024-01-15T18:04:34.840 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.835+0000 7f5b32000640 1 --2- 172.21.15.93:0/2456144435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b2c157a50 0x7f5b2c14e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.840 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.835+0000 7f5b32000640 1 --2- 172.21.15.93:0/2456144435 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b2c070d50 0x7f5b2c1504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.840 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.835+0000 7f5b32000640 1 --2- 172.21.15.93:0/2456144435 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b2c06ac00 0x7f5b2c153e10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.840 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.835+0000 7f5b32000640 1 -- 172.21.15.93:0/2456144435 >> 172.21.15.93:0/2456144435 conn(0x7f5b2c0b1620 msgr2=0x7f5b2c06a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:34.840 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.835+0000 7f5b32000640 1 -- 172.21.15.93:0/2456144435 shutdown_connections 2024-01-15T18:04:34.840 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.835+0000 7f5b32000640 1 -- 172.21.15.93:0/2456144435 wait complete. 2024-01-15T18:04:34.844 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-15T18:04:34.837+0000 7f49aec8d180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-15T18:04:34.858 DEBUG:teuthology.orchestra.run.smithi093:> sudo ceph --cluster ceph osd new 82211717-bb0b-4c7e-9e48-bdc0dd39bd97 5 2024-01-15T18:04:34.929 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:04:34.923+0000 7f5a768cf180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-15T18:04:34.936 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-15T18:04:34.929+0000 7f49aec8d180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-15T18:04:34.991 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:04:34.987+0000 7f5a768cf180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-15T18:04:34.993 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.987+0000 7f1d03a00640 1 Processor -- start 2024-01-15T18:04:34.993 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.987+0000 7f1d03a00640 1 -- start start 2024-01-15T18:04:34.995 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.991+0000 7f1d03a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1cfc143b00 0x7f1cfc0cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:34.995 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.991+0000 7f1d03a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cfc15e150 con 0x7f1cfc143720 2024-01-15T18:04:34.995 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.991+0000 7f1d03a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cfc15e430 con 0x7f1cfc071ab0 2024-01-15T18:04:34.995 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.991+0000 7f1d03a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cfc071380 con 0x7f1cfc143b00 2024-01-15T18:04:34.995 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.991+0000 7f1d01a00640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7f1cfc071ab0 0x7f1cfc15eae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.93:41434/0 (socket says 172.21.15.93:41434) 2024-01-15T18:04:34.995 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.991+0000 7f1cfbe00640 1 --2- 172.21.15.93:0/2667308684 >> v2:172.21.15.27:3300/0 conn(0x7f1cfc143b00 0x7f1cfc0cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:34.995 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.991+0000 7f1d01a00640 1 -- 172.21.15.93:0/2667308684 learned_addr learned my addr 172.21.15.93:0/2667308684 (peer_addr_for_me v1:172.21.15.93:0/0) 2024-01-15T18:04:34.996 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.991+0000 7f1cfbe00640 1 -- 172.21.15.93:0/2667308684 >> v1:172.21.15.93:6789/0 conn(0x7f1cfc071ab0 legacy=0x7f1cfc15eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:34.996 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.991+0000 7f1cfbe00640 1 -- 172.21.15.93:0/2667308684 >> v1:172.21.15.27:6789/0 conn(0x7f1cfc143720 legacy=0x7f1cfc15d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:34.996 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.991+0000 7f1cfbe00640 1 -- 172.21.15.93:0/2667308684 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cfc15e710 con 0x7f1cfc143b00 2024-01-15T18:04:34.996 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.991+0000 7f1cfb400640 1 -- 172.21.15.93:0/2667308684 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3429295684 0 0) 0x7f1cfc15e150 con 0x7f1cfc143720 2024-01-15T18:04:34.998 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.995+0000 7f1cfbe00640 1 --2- 172.21.15.93:0/2667308684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cfc143b00 0x7f1cfc0cde40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1ce40097c0 tx=0x7f1ce401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7df6963b98610d4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:34.999 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-15T18:04:34.993+0000 7f49aec8d180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-15T18:04:34.999 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.995+0000 7f1cfb400640 1 -- 172.21.15.93:0/2667308684 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce40ad660 con 0x7f1cfc143b00 2024-01-15T18:04:34.999 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.995+0000 7f1cfb400640 1 -- 172.21.15.93:0/2667308684 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce40ad7c0 con 0x7f1cfc143b00 2024-01-15T18:04:34.999 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.995+0000 7f1cfb400640 1 -- 172.21.15.93:0/2667308684 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce40adb50 con 0x7f1cfc143b00 2024-01-15T18:04:35.000 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.995+0000 7f1d03a00640 1 -- 172.21.15.93:0/2667308684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cfc143b00 msgr2=0x7f1cfc0cde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:35.000 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.995+0000 7f1d03a00640 1 --2- 172.21.15.93:0/2667308684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cfc143b00 0x7f1cfc0cde40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1ce40097c0 tx=0x7f1ce401c450 comp rx=0 tx=0).stop 2024-01-15T18:04:35.000 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.995+0000 7f1d03a00640 1 -- 172.21.15.93:0/2667308684 shutdown_connections 2024-01-15T18:04:35.000 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.995+0000 7f1d03a00640 1 --2- 172.21.15.93:0/2667308684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cfc143b00 0x7f1cfc0cde40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.000 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.995+0000 7f1d03a00640 1 -- 172.21.15.93:0/2667308684 >> 172.21.15.93:0/2667308684 conn(0x7f1cfc06b600 msgr2=0x7f1cfc06ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:35.001 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.995+0000 7f1d03a00640 1 -- 172.21.15.93:0/2667308684 shutdown_connections 2024-01-15T18:04:35.001 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.995+0000 7f1d03a00640 1 -- 172.21.15.93:0/2667308684 wait complete. 2024-01-15T18:04:35.002 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1d03a00640 1 Processor -- start 2024-01-15T18:04:35.002 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1d03a00640 1 -- start start 2024-01-15T18:04:35.005 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1d03a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1cfc071ab0 0x7f1cfc15cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:35.005 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1d03a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cfc143720 0x7f1cfc15a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:35.005 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1d03a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1cfc143b00 0x7f1cfc0b5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:35.005 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1d03a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cfc0b5eb0 con 0x7f1cfc143720 2024-01-15T18:04:35.005 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1d03a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cfc0ad350 con 0x7f1cfc071ab0 2024-01-15T18:04:35.005 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1d03a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cfc0aca20 con 0x7f1cfc143b00 2024-01-15T18:04:35.005 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1d01000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1cfc071ab0 0x7f1cfc15cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:35.005 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1d01a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1cfc143b00 0x7f1cfc0b5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:35.005 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1cfbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cfc143720 0x7f1cfc15a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:35.005 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1cfbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cfc143720 0x7f1cfc15a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.93:39728/0 (socket says 172.21.15.93:39728) 2024-01-15T18:04:35.006 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1d01a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1cfc143b00 0x7f1cfc0b5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.93:37956/0 (socket says 172.21.15.93:37956) 2024-01-15T18:04:35.006 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1d01a00640 1 -- 172.21.15.93:0/648254313 learned_addr learned my addr 172.21.15.93:0/648254313 (peer_addr_for_me v2:172.21.15.93:0/0) 2024-01-15T18:04:35.006 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1d01000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1cfc071ab0 0x7f1cfc15cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:46898/0 (socket says 172.21.15.93:46898) 2024-01-15T18:04:35.006 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1d01000640 1 -- 172.21.15.93:0/648254313 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1cfc143b00 msgr2=0x7f1cfc0b5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:35.006 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1d01000640 1 --2- 172.21.15.93:0/648254313 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1cfc143b00 0x7f1cfc0b5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.006 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1d01000640 1 -- 172.21.15.93:0/648254313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cfc143720 msgr2=0x7f1cfc15a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:35.006 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1d01000640 1 --2- 172.21.15.93:0/648254313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cfc143720 0x7f1cfc15a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.006 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1d01000640 1 -- 172.21.15.93:0/648254313 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce4009420 con 0x7f1cfc071ab0 2024-01-15T18:04:35.006 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1d01a00640 1 --2- 172.21.15.93:0/648254313 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1cfc143b00 0x7f1cfc0b5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:04:35.006 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1cfbe00640 1 --2- 172.21.15.93:0/648254313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cfc143720 0x7f1cfc15a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:04:35.006 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1d01000640 1 --2- 172.21.15.93:0/648254313 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1cfc071ab0 0x7f1cfc15cd00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1cec00c430 tx=0x7f1cec00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:35.006 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1cf8c00640 1 -- 172.21.15.93:0/648254313 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec0054e0 con 0x7f1cfc071ab0 2024-01-15T18:04:35.006 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1cf8c00640 1 -- 172.21.15.93:0/648254313 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cec003600 con 0x7f1cfc071ab0 2024-01-15T18:04:35.006 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1cf8c00640 1 -- 172.21.15.93:0/648254313 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec0039f0 con 0x7f1cfc071ab0 2024-01-15T18:04:35.006 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1d03a00640 1 -- 172.21.15.93:0/648254313 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cfc0acd00 con 0x7f1cfc071ab0 2024-01-15T18:04:35.007 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1d03a00640 1 -- 172.21.15.93:0/648254313 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cfc15a580 con 0x7f1cfc071ab0 2024-01-15T18:04:35.007 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1cf8c00640 1 -- 172.21.15.93:0/648254313 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f1cec01c440 con 0x7f1cfc071ab0 2024-01-15T18:04:35.007 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:34.999+0000 7f1cf8c00640 1 -- 172.21.15.93:0/648254313 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f1cec012980 con 0x7f1cfc071ab0 2024-01-15T18:04:35.007 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.003+0000 7f1d03a00640 1 -- 172.21.15.93:0/648254313 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cfc04e850 con 0x7f1cfc071ab0 2024-01-15T18:04:35.010 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.007+0000 7f1cf8c00640 1 -- 172.21.15.93:0/648254313 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f1cec017070 con 0x7f1cfc071ab0 2024-01-15T18:04:35.079 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.075+0000 7f1d03a00640 1 -- 172.21.15.93:0/648254313 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd new", "uuid": "82211717-bb0b-4c7e-9e48-bdc0dd39bd97", "id": 5} v 0) v1 -- 0x7f1cfc13f230 con 0x7f1cfc071ab0 2024-01-15T18:04:35.084 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.079+0000 7f1cf8c00640 1 -- 172.21.15.93:0/648254313 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "82211717-bb0b-4c7e-9e48-bdc0dd39bd97", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1cec012400 con 0x7f1cfc071ab0 2024-01-15T18:04:35.084 INFO:teuthology.orchestra.run.smithi093.stdout:5 2024-01-15T18:04:35.086 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.079+0000 7f1d03a00640 1 -- 172.21.15.93:0/648254313 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1cfc071ab0 msgr2=0x7f1cfc15cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:35.086 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.079+0000 7f1d03a00640 1 --2- 172.21.15.93:0/648254313 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1cfc071ab0 0x7f1cfc15cd00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1cec00c430 tx=0x7f1cec00c8f0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.086 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.083+0000 7f1d03a00640 1 -- 172.21.15.93:0/648254313 shutdown_connections 2024-01-15T18:04:35.086 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.083+0000 7f1d03a00640 1 --2- 172.21.15.93:0/648254313 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1cfc143b00 0x7f1cfc0b5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.086 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.083+0000 7f1d03a00640 1 --2- 172.21.15.93:0/648254313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cfc143720 0x7f1cfc15a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.086 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.083+0000 7f1d03a00640 1 --2- 172.21.15.93:0/648254313 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1cfc071ab0 0x7f1cfc15cd00 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.086 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.083+0000 7f1d03a00640 1 -- 172.21.15.93:0/648254313 >> 172.21.15.93:0/648254313 conn(0x7f1cfc06b600 msgr2=0x7f1cfc0aff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:35.086 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.083+0000 7f1d03a00640 1 -- 172.21.15.93:0/648254313 shutdown_connections 2024-01-15T18:04:35.086 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.083+0000 7f1d03a00640 1 -- 172.21.15.93:0/648254313 wait complete. 2024-01-15T18:04:35.090 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:04:35.087+0000 7f5a768cf180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-15T18:04:35.093 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-15T18:04:35.085+0000 7f49aec8d180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-15T18:04:35.105 DEBUG:teuthology.orchestra.run.smithi093:> sudo ceph --cluster ceph osd new 1fabf1d4-0289-42f3-88e4-90fede3a06c7 6 2024-01-15T18:04:35.153 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:04:35.147+0000 7f5a768cf180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-15T18:04:35.155 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-15T18:04:35.149+0000 7f49aec8d180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-15T18:04:35.239 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.235+0000 7fc009400640 1 Processor -- start 2024-01-15T18:04:35.239 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.235+0000 7fc009400640 1 -- start start 2024-01-15T18:04:35.240 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.235+0000 7fc009400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc0040d1560 0x7fc0040b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:35.240 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.235+0000 7fc009400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0040cb920 con 0x7fc00415b400 2024-01-15T18:04:35.241 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.235+0000 7fc009400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0040ce7d0 con 0x7fc00415b020 2024-01-15T18:04:35.241 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.235+0000 7fc009400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0040d0cf0 con 0x7fc0040d1560 2024-01-15T18:04:35.241 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.235+0000 7fc002800640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7fc00415b020 0x7fc0040cd6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.93:41440/0 (socket says 172.21.15.93:41440) 2024-01-15T18:04:35.241 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.235+0000 7fc002800640 1 -- 172.21.15.93:0/4107721636 learned_addr learned my addr 172.21.15.93:0/4107721636 (peer_addr_for_me v1:172.21.15.93:0/0) 2024-01-15T18:04:35.241 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.235+0000 7fc003200640 1 --2- 172.21.15.93:0/4107721636 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc0040d1560 0x7fc0040b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:35.241 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.235+0000 7fc001400640 1 -- 172.21.15.93:0/4107721636 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636490596 0 0) 0x7fc0040ce7d0 con 0x7fc00415b020 2024-01-15T18:04:35.241 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.235+0000 7fc003200640 1 -- 172.21.15.93:0/4107721636 >> v1:172.21.15.93:6789/0 conn(0x7fc00415b020 legacy=0x7fc0040cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:35.242 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.235+0000 7fc003200640 1 -- 172.21.15.93:0/4107721636 >> v1:172.21.15.27:6789/0 conn(0x7fc00415b400 legacy=0x7fc0040d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:35.242 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.235+0000 7fc003200640 1 -- 172.21.15.93:0/4107721636 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0040cdf30 con 0x7fc0040d1560 2024-01-15T18:04:35.244 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.239+0000 7fc003200640 1 --2- 172.21.15.93:0/4107721636 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc0040d1560 0x7fc0040b0a30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbffc009dc0 tx=0x7fbffc01c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c71778a80f8a3545 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:35.244 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.239+0000 7fc001400640 1 -- 172.21.15.93:0/4107721636 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc0a5860 con 0x7fc0040d1560 2024-01-15T18:04:35.244 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.239+0000 7fc001400640 1 -- 172.21.15.93:0/4107721636 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbffc0a59c0 con 0x7fc0040d1560 2024-01-15T18:04:35.244 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.239+0000 7fc001400640 1 -- 172.21.15.93:0/4107721636 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbffc0a5d50 con 0x7fc0040d1560 2024-01-15T18:04:35.245 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.239+0000 7fc009400640 1 -- 172.21.15.93:0/4107721636 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc0040d1560 msgr2=0x7fc0040b0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:35.245 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.239+0000 7fc009400640 1 --2- 172.21.15.93:0/4107721636 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc0040d1560 0x7fc0040b0a30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbffc009dc0 tx=0x7fbffc01c800 comp rx=0 tx=0).stop 2024-01-15T18:04:35.245 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.239+0000 7fc009400640 1 -- 172.21.15.93:0/4107721636 shutdown_connections 2024-01-15T18:04:35.245 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.239+0000 7fc009400640 1 --2- 172.21.15.93:0/4107721636 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc0040d1560 0x7fc0040b0a30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.245 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.239+0000 7fc009400640 1 -- 172.21.15.93:0/4107721636 >> 172.21.15.93:0/4107721636 conn(0x7fc004099a10 msgr2=0x7fc004099e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:35.246 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.243+0000 7fc009400640 1 -- 172.21.15.93:0/4107721636 shutdown_connections 2024-01-15T18:04:35.246 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.243+0000 7fc009400640 1 -- 172.21.15.93:0/4107721636 wait complete. 2024-01-15T18:04:35.247 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.243+0000 7fc009400640 1 Processor -- start 2024-01-15T18:04:35.247 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.243+0000 7fc009400640 1 -- start start 2024-01-15T18:04:35.249 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.243+0000 7fc009400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc0040d1560 0x7fc0040b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:35.249 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.243+0000 7fc009400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc00415b020 0x7fc0040b58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:35.249 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.243+0000 7fc009400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc00415b400 0x7fc0040b5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:35.249 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.243+0000 7fc009400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0040c7480 con 0x7fc00415b400 2024-01-15T18:04:35.249 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.243+0000 7fc009400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0040c6900 con 0x7fc00415b020 2024-01-15T18:04:35.249 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.243+0000 7fc009400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0040c6be0 con 0x7fc0040d1560 2024-01-15T18:04:35.249 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.243+0000 7fc002800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc0040d1560 0x7fc0040b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:35.250 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.243+0000 7fc001e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc00415b020 0x7fc0040b58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:35.250 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.243+0000 7fc002800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc0040d1560 0x7fc0040b3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.93:37970/0 (socket says 172.21.15.93:37970) 2024-01-15T18:04:35.250 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.243+0000 7fc002800640 1 -- 172.21.15.93:0/2651858216 learned_addr learned my addr 172.21.15.93:0/2651858216 (peer_addr_for_me v2:172.21.15.93:0/0) 2024-01-15T18:04:35.250 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.243+0000 7fc001e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc00415b020 0x7fc0040b58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:46900/0 (socket says 172.21.15.93:46900) 2024-01-15T18:04:35.250 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.243+0000 7fc003200640 1 --2- 172.21.15.93:0/2651858216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc00415b400 0x7fc0040b5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:35.250 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.243+0000 7fc002800640 1 -- 172.21.15.93:0/2651858216 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc00415b020 msgr2=0x7fc0040b58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:35.250 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.243+0000 7fc002800640 1 --2- 172.21.15.93:0/2651858216 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc00415b020 0x7fc0040b58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.250 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.243+0000 7fc002800640 1 -- 172.21.15.93:0/2651858216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc00415b400 msgr2=0x7fc0040b5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:35.250 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.243+0000 7fc002800640 1 --2- 172.21.15.93:0/2651858216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc00415b400 0x7fc0040b5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.250 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.243+0000 7fc002800640 1 -- 172.21.15.93:0/2651858216 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbffc009a70 con 0x7fc0040d1560 2024-01-15T18:04:35.250 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.243+0000 7fc001e00640 1 --2- 172.21.15.93:0/2651858216 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc00415b020 0x7fc0040b58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:04:35.250 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.243+0000 7fc003200640 1 --2- 172.21.15.93:0/2651858216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc00415b400 0x7fc0040b5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:04:35.251 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.247+0000 7fc002800640 1 --2- 172.21.15.93:0/2651858216 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc0040d1560 0x7fc0040b3bd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbff400c4a0 tx=0x7fbff400c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:35.251 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.247+0000 7fbffaa00640 1 -- 172.21.15.93:0/2651858216 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff40074e0 con 0x7fc0040d1560 2024-01-15T18:04:35.251 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.247+0000 7fbffaa00640 1 -- 172.21.15.93:0/2651858216 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff4007640 con 0x7fc0040d1560 2024-01-15T18:04:35.251 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.247+0000 7fbffaa00640 1 -- 172.21.15.93:0/2651858216 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff4004a30 con 0x7fc0040d1560 2024-01-15T18:04:35.252 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.247+0000 7fc009400640 1 -- 172.21.15.93:0/2651858216 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0040c6ec0 con 0x7fc0040d1560 2024-01-15T18:04:35.252 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.247+0000 7fc009400640 1 -- 172.21.15.93:0/2651858216 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0041f3d00 con 0x7fc0040d1560 2024-01-15T18:04:35.252 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.247+0000 7fbffaa00640 1 -- 172.21.15.93:0/2651858216 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fbff4012400 con 0x7fc0040d1560 2024-01-15T18:04:35.252 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.247+0000 7fbffaa00640 1 -- 172.21.15.93:0/2651858216 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fbff401bac0 con 0x7fc0040d1560 2024-01-15T18:04:35.252 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.247+0000 7fc009400640 1 -- 172.21.15.93:0/2651858216 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc00404e850 con 0x7fc0040d1560 2024-01-15T18:04:35.253 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.247+0000 7fbffaa00640 1 -- 172.21.15.93:0/2651858216 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fbff4021060 con 0x7fc0040d1560 2024-01-15T18:04:35.316 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:04:35.311+0000 7f5a768cf180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-15T18:04:35.318 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.315+0000 7fc009400640 1 -- 172.21.15.93:0/2651858216 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1fabf1d4-0289-42f3-88e4-90fede3a06c7", "id": 6} v 0) v1 -- 0x7fc0040b6ec0 con 0x7fc0040d1560 2024-01-15T18:04:35.320 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-15T18:04:35.313+0000 7f49aec8d180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-15T18:04:35.324 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.319+0000 7fbffaa00640 1 -- 172.21.15.93:0/2651858216 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1fabf1d4-0289-42f3-88e4-90fede3a06c7", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fbff4018070 con 0x7fc0040d1560 2024-01-15T18:04:35.324 INFO:teuthology.orchestra.run.smithi093.stdout:6 2024-01-15T18:04:35.325 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.319+0000 7fc009400640 1 -- 172.21.15.93:0/2651858216 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc0040d1560 msgr2=0x7fc0040b3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:35.325 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.319+0000 7fc009400640 1 --2- 172.21.15.93:0/2651858216 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc0040d1560 0x7fc0040b3bd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbff400c4a0 tx=0x7fbff400c960 comp rx=0 tx=0).stop 2024-01-15T18:04:35.325 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.319+0000 7fc009400640 1 -- 172.21.15.93:0/2651858216 shutdown_connections 2024-01-15T18:04:35.325 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.319+0000 7fc009400640 1 --2- 172.21.15.93:0/2651858216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc00415b400 0x7fc0040b5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.325 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.319+0000 7fc009400640 1 --2- 172.21.15.93:0/2651858216 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc00415b020 0x7fc0040b58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.326 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.319+0000 7fc009400640 1 --2- 172.21.15.93:0/2651858216 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc0040d1560 0x7fc0040b3bd0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.326 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.319+0000 7fc009400640 1 -- 172.21.15.93:0/2651858216 >> 172.21.15.93:0/2651858216 conn(0x7fc004099a10 msgr2=0x7fc00415a690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:35.326 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.319+0000 7fc009400640 1 -- 172.21.15.93:0/2651858216 shutdown_connections 2024-01-15T18:04:35.326 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.323+0000 7fc009400640 1 -- 172.21.15.93:0/2651858216 wait complete. 2024-01-15T18:04:35.344 DEBUG:teuthology.orchestra.run.smithi093:> sudo ceph --cluster ceph osd new e3626db0-8611-4ead-86f0-1b9ae0163da4 7 2024-01-15T18:04:35.388 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:04:35.383+0000 7f5a768cf180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-15T18:04:35.393 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-15T18:04:35.385+0000 7f49aec8d180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-15T18:04:35.477 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.475+0000 7ff8e3c00640 1 Processor -- start 2024-01-15T18:04:35.478 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.475+0000 7ff8e3c00640 1 -- start start 2024-01-15T18:04:35.480 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.475+0000 7ff8e3c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff8dc0ce7c0 0x7ff8dc0cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:35.480 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.475+0000 7ff8e3c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8dc0b5c00 con 0x7ff8dc15a620 2024-01-15T18:04:35.480 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.475+0000 7ff8e3c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8dc0b5ee0 con 0x7ff8dc0b1c80 2024-01-15T18:04:35.480 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.475+0000 7ff8e3c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc0cdfc0 con 0x7ff8dc0ce7c0 2024-01-15T18:04:35.480 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.475+0000 7ff8e1200640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7ff8dc0b1c80 0x7ff8dc0b2060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.93:41456/0 (socket says 172.21.15.93:41456) 2024-01-15T18:04:35.480 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.475+0000 7ff8e1200640 1 -- 172.21.15.93:0/2153886175 learned_addr learned my addr 172.21.15.93:0/2153886175 (peer_addr_for_me v1:172.21.15.93:0/0) 2024-01-15T18:04:35.480 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.475+0000 7ff8dbe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff8dc0ce7c0 0x7ff8dc0cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:35.480 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.475+0000 7ff8db400640 1 -- 172.21.15.93:0/2153886175 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2106860886 0 0) 0x7ff8dc0b5ee0 con 0x7ff8dc0b1c80 2024-01-15T18:04:35.481 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.475+0000 7ff8db400640 1 -- 172.21.15.93:0/2153886175 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8bc005740 con 0x7ff8dc0b1c80 2024-01-15T18:04:35.481 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.475+0000 7ff8dbe00640 1 -- 172.21.15.93:0/2153886175 >> v1:172.21.15.93:6789/0 conn(0x7ff8dc0b1c80 legacy=0x7ff8dc0b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:35.481 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.475+0000 7ff8dbe00640 1 -- 172.21.15.93:0/2153886175 >> v1:172.21.15.27:6789/0 conn(0x7ff8dc15a620 legacy=0x7ff8dc0b4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:35.481 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.475+0000 7ff8db400640 1 -- 172.21.15.93:0/2153886175 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808444590 0 0) 0x7ff8dc0b5c00 con 0x7ff8dc15a620 2024-01-15T18:04:35.481 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.475+0000 7ff8dbe00640 1 -- 172.21.15.93:0/2153886175 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8dc0ce2a0 con 0x7ff8dc0ce7c0 2024-01-15T18:04:35.483 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.479+0000 7ff8dbe00640 1 --2- 172.21.15.93:0/2153886175 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff8dc0ce7c0 0x7ff8dc0cec20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40097f0 tx=0x7ff8c401c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2258d6816372aad1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:35.483 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.479+0000 7ff8db400640 1 -- 172.21.15.93:0/2153886175 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c40ab600 con 0x7ff8dc0ce7c0 2024-01-15T18:04:35.483 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.479+0000 7ff8db400640 1 -- 172.21.15.93:0/2153886175 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c40a7050 con 0x7ff8dc0ce7c0 2024-01-15T18:04:35.483 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.479+0000 7ff8db400640 1 -- 172.21.15.93:0/2153886175 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c40ab980 con 0x7ff8dc0ce7c0 2024-01-15T18:04:35.484 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.479+0000 7ff8e3c00640 1 -- 172.21.15.93:0/2153886175 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff8dc0ce7c0 msgr2=0x7ff8dc0cec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:35.484 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.479+0000 7ff8e3c00640 1 --2- 172.21.15.93:0/2153886175 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff8dc0ce7c0 0x7ff8dc0cec20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40097f0 tx=0x7ff8c401c4c0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.484 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.479+0000 7ff8e3c00640 1 -- 172.21.15.93:0/2153886175 shutdown_connections 2024-01-15T18:04:35.484 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.479+0000 7ff8e3c00640 1 --2- 172.21.15.93:0/2153886175 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff8dc0ce7c0 0x7ff8dc0cec20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.484 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.479+0000 7ff8e3c00640 1 -- 172.21.15.93:0/2153886175 >> 172.21.15.93:0/2153886175 conn(0x7ff8dc0669e0 msgr2=0x7ff8dc06b630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:35.484 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.479+0000 7ff8e3c00640 1 -- 172.21.15.93:0/2153886175 shutdown_connections 2024-01-15T18:04:35.484 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.479+0000 7ff8e3c00640 1 -- 172.21.15.93:0/2153886175 wait complete. 2024-01-15T18:04:35.486 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8e3c00640 1 Processor -- start 2024-01-15T18:04:35.486 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8e3c00640 1 -- start start 2024-01-15T18:04:35.489 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8e3c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff8dc0ce7c0 0x7ff8dc142890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:35.489 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8e3c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff8dc15a620 0x7ff8dc159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:35.489 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8e3c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8dc137490 0x7ff8dc0af070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:35.489 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8e3c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc13bcc0 con 0x7ff8dc137490 2024-01-15T18:04:35.489 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8e3c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc0ac920 con 0x7ff8dc0ce7c0 2024-01-15T18:04:35.489 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8e3c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc0acc00 con 0x7ff8dc15a620 2024-01-15T18:04:35.489 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8e1200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff8dc0ce7c0 0x7ff8dc142890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:35.489 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8dbe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff8dc15a620 0x7ff8dc159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:35.489 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8e1200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff8dc0ce7c0 0x7ff8dc142890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.93:46902/0 (socket says 172.21.15.93:46902) 2024-01-15T18:04:35.489 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8e1200640 1 -- 172.21.15.93:0/3599718399 learned_addr learned my addr 172.21.15.93:0/3599718399 (peer_addr_for_me v2:172.21.15.93:0/0) 2024-01-15T18:04:35.489 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8e1c00640 1 --2- 172.21.15.93:0/3599718399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8dc137490 0x7ff8dc0af070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:35.490 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8e1200640 1 -- 172.21.15.93:0/3599718399 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff8dc15a620 msgr2=0x7ff8dc159b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:35.490 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8e1200640 1 --2- 172.21.15.93:0/3599718399 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff8dc15a620 0x7ff8dc159b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.490 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8e1200640 1 -- 172.21.15.93:0/3599718399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8dc137490 msgr2=0x7ff8dc0af070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:35.490 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8e1200640 1 --2- 172.21.15.93:0/3599718399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8dc137490 0x7ff8dc0af070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.490 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8e1200640 1 -- 172.21.15.93:0/3599718399 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c4009420 con 0x7ff8dc0ce7c0 2024-01-15T18:04:35.490 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8dbe00640 1 --2- 172.21.15.93:0/3599718399 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff8dc15a620 0x7ff8dc159b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:04:35.490 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8e1200640 1 --2- 172.21.15.93:0/3599718399 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff8dc0ce7c0 0x7ff8dc142890 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc00c450 tx=0x7ff8cc00c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:35.490 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8d8c00640 1 -- 172.21.15.93:0/3599718399 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc005cf0 con 0x7ff8dc0ce7c0 2024-01-15T18:04:35.490 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8d8c00640 1 -- 172.21.15.93:0/3599718399 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8cc005e50 con 0x7ff8dc0ce7c0 2024-01-15T18:04:35.490 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8d8c00640 1 -- 172.21.15.93:0/3599718399 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc007290 con 0x7ff8dc0ce7c0 2024-01-15T18:04:35.490 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8e3c00640 1 -- 172.21.15.93:0/3599718399 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8dc0acee0 con 0x7ff8dc0ce7c0 2024-01-15T18:04:35.490 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8e3c00640 1 -- 172.21.15.93:0/3599718399 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8dc137af0 con 0x7ff8dc0ce7c0 2024-01-15T18:04:35.490 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8d8c00640 1 -- 172.21.15.93:0/3599718399 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff8cc01c440 con 0x7ff8dc0ce7c0 2024-01-15T18:04:35.490 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.483+0000 7ff8d8c00640 1 -- 172.21.15.93:0/3599718399 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7ff8cc012a60 con 0x7ff8dc0ce7c0 2024-01-15T18:04:35.491 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.487+0000 7ff8e3c00640 1 -- 172.21.15.93:0/3599718399 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8dc0ad0d0 con 0x7ff8dc0ce7c0 2024-01-15T18:04:35.492 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.487+0000 7ff8d8c00640 1 -- 172.21.15.93:0/3599718399 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7ff8cc017070 con 0x7ff8dc0ce7c0 2024-01-15T18:04:35.557 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.551+0000 7ff8e3c00640 1 -- 172.21.15.93:0/3599718399 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e3626db0-8611-4ead-86f0-1b9ae0163da4", "id": 7} v 0) v1 -- 0x7ff8dc04e850 con 0x7ff8dc0ce7c0 2024-01-15T18:04:35.562 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.559+0000 7ff8d8c00640 1 -- 172.21.15.93:0/3599718399 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e3626db0-8611-4ead-86f0-1b9ae0163da4", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff8cc0124d0 con 0x7ff8dc0ce7c0 2024-01-15T18:04:35.562 INFO:teuthology.orchestra.run.smithi093.stdout:7 2024-01-15T18:04:35.564 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.559+0000 7ff8e3c00640 1 -- 172.21.15.93:0/3599718399 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff8dc0ce7c0 msgr2=0x7ff8dc142890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:35.564 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.559+0000 7ff8e3c00640 1 --2- 172.21.15.93:0/3599718399 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff8dc0ce7c0 0x7ff8dc142890 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc00c450 tx=0x7ff8cc00c910 comp rx=0 tx=0).stop 2024-01-15T18:04:35.564 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.559+0000 7ff8e3c00640 1 -- 172.21.15.93:0/3599718399 shutdown_connections 2024-01-15T18:04:35.564 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.559+0000 7ff8e3c00640 1 --2- 172.21.15.93:0/3599718399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8dc137490 0x7ff8dc0af070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.564 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.559+0000 7ff8e3c00640 1 --2- 172.21.15.93:0/3599718399 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff8dc15a620 0x7ff8dc159b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.564 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.559+0000 7ff8e3c00640 1 --2- 172.21.15.93:0/3599718399 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff8dc0ce7c0 0x7ff8dc142890 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.564 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.559+0000 7ff8e3c00640 1 -- 172.21.15.93:0/3599718399 >> 172.21.15.93:0/3599718399 conn(0x7ff8dc0b1c80 msgr2=0x7ff8dc06b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:35.564 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.559+0000 7ff8e3c00640 1 -- 172.21.15.93:0/3599718399 shutdown_connections 2024-01-15T18:04:35.565 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:04:35.559+0000 7ff8e3c00640 1 -- 172.21.15.93:0/3599718399 wait complete. 2024-01-15T18:04:35.582 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-15T18:04:35.583 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-01-15T18:04:35.586 INFO:tasks.ceph.osd.0:Started 2024-01-15T18:04:35.586 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-15T18:04:35.586 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-01-15T18:04:35.589 INFO:tasks.ceph.osd.1:Started 2024-01-15T18:04:35.590 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-15T18:04:35.590 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-01-15T18:04:35.593 INFO:tasks.ceph.osd.2:Started 2024-01-15T18:04:35.593 INFO:tasks.ceph.osd.3:Restarting daemon 2024-01-15T18:04:35.593 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-01-15T18:04:35.596 INFO:tasks.ceph.osd.3:Started 2024-01-15T18:04:35.596 INFO:tasks.ceph.osd.4:Restarting daemon 2024-01-15T18:04:35.596 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-01-15T18:04:35.628 INFO:tasks.ceph.osd.4:Started 2024-01-15T18:04:35.628 INFO:tasks.ceph.osd.5:Restarting daemon 2024-01-15T18:04:35.628 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-01-15T18:04:35.631 INFO:tasks.ceph.osd.5:Started 2024-01-15T18:04:35.631 INFO:tasks.ceph.osd.6:Restarting daemon 2024-01-15T18:04:35.631 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-01-15T18:04:35.634 INFO:tasks.ceph.osd.6:Started 2024-01-15T18:04:35.634 INFO:tasks.ceph.osd.7:Restarting daemon 2024-01-15T18:04:35.634 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-01-15T18:04:35.639 INFO:tasks.ceph.osd.7:Started 2024-01-15T18:04:35.639 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:04:35.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.829+0000 7f2246c00640 1 Processor -- start 2024-01-15T18:04:35.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.829+0000 7f2246c00640 1 -- start start 2024-01-15T18:04:35.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.833+0000 7f2246c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f2240072380 0x7f22400b4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:35.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.833+0000 7f2246c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f22401618b0 0x7f224006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:35.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.833+0000 7f2246c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f224006a620 con 0x7f22401614d0 2024-01-15T18:04:35.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.833+0000 7f2246c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f224006a900 con 0x7f22401618b0 2024-01-15T18:04:35.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.833+0000 7f2246c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f224006abe0 con 0x7f2240072380 2024-01-15T18:04:35.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.833+0000 7f223fe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f22401614d0 0x7f2240071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40104/0 (socket says 172.21.15.27:40104) 2024-01-15T18:04:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.833+0000 7f223fe00640 1 -- 172.21.15.27:0/4220467160 learned_addr learned my addr 172.21.15.27:0/4220467160 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:04:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.833+0000 7f2245600640 1 --2- 172.21.15.27:0/4220467160 >> v2:172.21.15.27:3300/0 conn(0x7f22401618b0 0x7f224006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.833+0000 7f223f400640 1 --2- 172.21.15.27:0/4220467160 >> v2:172.21.15.93:3300/0 conn(0x7f2240072380 0x7f22400b4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.833+0000 7f2245600640 1 -- 172.21.15.27:0/4220467160 >> v2:172.21.15.93:3300/0 conn(0x7f2240072380 msgr2=0x7f22400b4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.833+0000 7f2245600640 1 --2- 172.21.15.27:0/4220467160 >> v2:172.21.15.93:3300/0 conn(0x7f2240072380 0x7f22400b4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.833+0000 7f2245600640 1 -- 172.21.15.27:0/4220467160 >> v1:172.21.15.27:6789/0 conn(0x7f22401614d0 legacy=0x7f2240071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.833+0000 7f2245600640 1 -- 172.21.15.27:0/4220467160 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224006aec0 con 0x7f22401618b0 2024-01-15T18:04:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.833+0000 7f223ea00640 1 -- 172.21.15.27:0/4220467160 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126625595 0 0) 0x7f224006a620 con 0x7f22401614d0 2024-01-15T18:04:35.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.837+0000 7f2245600640 1 --2- 172.21.15.27:0/4220467160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22401618b0 0x7f224006dcd0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f22400afcb0 tx=0x7f223801c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38c324b9c434894a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:35.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.837+0000 7f223ea00640 1 -- 172.21.15.27:0/4220467160 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22380a5930 con 0x7f22401618b0 2024-01-15T18:04:35.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.837+0000 7f223ea00640 1 -- 172.21.15.27:0/4220467160 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22380a5a90 con 0x7f22401618b0 2024-01-15T18:04:35.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.837+0000 7f223ea00640 1 -- 172.21.15.27:0/4220467160 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22380a5e20 con 0x7f22401618b0 2024-01-15T18:04:35.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.837+0000 7f2246c00640 1 -- 172.21.15.27:0/4220467160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22401618b0 msgr2=0x7f224006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:35.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.837+0000 7f2246c00640 1 --2- 172.21.15.27:0/4220467160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22401618b0 0x7f224006dcd0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f22400afcb0 tx=0x7f223801c850 comp rx=0 tx=0).stop 2024-01-15T18:04:35.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.837+0000 7f2246c00640 1 -- 172.21.15.27:0/4220467160 shutdown_connections 2024-01-15T18:04:35.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.837+0000 7f2246c00640 1 --2- 172.21.15.27:0/4220467160 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22401618b0 0x7f224006dcd0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.837+0000 7f2246c00640 1 --2- 172.21.15.27:0/4220467160 >> v2:172.21.15.93:3300/0 conn(0x7f2240072380 0x7f22400b4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.837+0000 7f2246c00640 1 -- 172.21.15.27:0/4220467160 >> 172.21.15.27:0/4220467160 conn(0x7f22400b2360 msgr2=0x7f22400b2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:35.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.841+0000 7f2246c00640 1 -- 172.21.15.27:0/4220467160 shutdown_connections 2024-01-15T18:04:35.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.841+0000 7f2246c00640 1 -- 172.21.15.27:0/4220467160 wait complete. 2024-01-15T18:04:35.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.841+0000 7f2246c00640 1 Processor -- start 2024-01-15T18:04:35.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.841+0000 7f2246c00640 1 -- start start 2024-01-15T18:04:35.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.845+0000 7f2246c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2240072380 0x7f224009d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:35.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.845+0000 7f2246c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22401614d0 0x7f224014a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:35.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.845+0000 7f2246c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f22401618b0 0x7f224009b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:35.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.845+0000 7f2246c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224015cb70 con 0x7f2240072380 2024-01-15T18:04:35.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.845+0000 7f2246c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224014ada0 con 0x7f22401614d0 2024-01-15T18:04:35.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.845+0000 7f2246c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f224015c7d0 con 0x7f22401618b0 2024-01-15T18:04:35.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.845+0000 7f223fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2240072380 0x7f224009d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:35.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.845+0000 7f223fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2240072380 0x7f224009d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36770/0 (socket says 172.21.15.27:36770) 2024-01-15T18:04:35.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.845+0000 7f223fe00640 1 -- 172.21.15.27:0/2626234131 learned_addr learned my addr 172.21.15.27:0/2626234131 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:35.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.845+0000 7f2245600640 1 --2- 172.21.15.27:0/2626234131 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f22401618b0 0x7f224009b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:35.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.845+0000 7f223f400640 1 --2- 172.21.15.27:0/2626234131 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22401614d0 0x7f224014a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:35.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.845+0000 7f223fe00640 1 -- 172.21.15.27:0/2626234131 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f22401618b0 msgr2=0x7f224009b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:35.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.845+0000 7f223fe00640 1 --2- 172.21.15.27:0/2626234131 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f22401618b0 0x7f224009b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.845+0000 7f223fe00640 1 -- 172.21.15.27:0/2626234131 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22401614d0 msgr2=0x7f224014a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:35.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.845+0000 7f223fe00640 1 --2- 172.21.15.27:0/2626234131 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22401614d0 0x7f224014a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.845+0000 7f223fe00640 1 -- 172.21.15.27:0/2626234131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2238009a70 con 0x7f2240072380 2024-01-15T18:04:35.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.845+0000 7f223fe00640 1 --2- 172.21.15.27:0/2626234131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2240072380 0x7f224009d950 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f22300011f0 tx=0x7f223000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:35.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.845+0000 7f2237e00640 1 -- 172.21.15.27:0/2626234131 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22300074e0 con 0x7f2240072380 2024-01-15T18:04:35.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.845+0000 7f2237e00640 1 -- 172.21.15.27:0/2626234131 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2230007640 con 0x7f2240072380 2024-01-15T18:04:35.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.845+0000 7f2237e00640 1 -- 172.21.15.27:0/2626234131 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2230004a10 con 0x7f2240072380 2024-01-15T18:04:35.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.845+0000 7f2246c00640 1 -- 172.21.15.27:0/2626234131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f224009df30 con 0x7f2240072380 2024-01-15T18:04:35.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.845+0000 7f2246c00640 1 -- 172.21.15.27:0/2626234131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f224009e4f0 con 0x7f2240072380 2024-01-15T18:04:35.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.845+0000 7f2237e00640 1 -- 172.21.15.27:0/2626234131 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2230012400 con 0x7f2240072380 2024-01-15T18:04:35.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.845+0000 7f2237e00640 1 -- 172.21.15.27:0/2626234131 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f223001bb10 con 0x7f2240072380 2024-01-15T18:04:35.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.845+0000 7f2246c00640 1 -- 172.21.15.27:0/2626234131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2240074820 con 0x7f2240072380 2024-01-15T18:04:35.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.857+0000 7f2237e00640 1 -- 172.21.15.27:0/2626234131 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f2230021060 con 0x7f2240072380 2024-01-15T18:04:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.913+0000 7f2246c00640 1 -- 172.21.15.27:0/2626234131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f224006ff90 con 0x7f2240072380 2024-01-15T18:04:35.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.913+0000 7f2237e00640 1 -- 172.21.15.27:0/2626234131 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f2230018070 con 0x7f2240072380 2024-01-15T18:04:35.921 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:04:35.921 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":10,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:04:35.557795+0000","last_up_change":"0.000000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:04:35.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.917+0000 7f2246c00640 1 -- 172.21.15.27:0/2626234131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2240072380 msgr2=0x7f224009d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.917+0000 7f2246c00640 1 --2- 172.21.15.27:0/2626234131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2240072380 0x7f224009d950 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f22300011f0 tx=0x7f223000c9b0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.917+0000 7f2246c00640 1 -- 172.21.15.27:0/2626234131 shutdown_connections 2024-01-15T18:04:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.917+0000 7f2246c00640 1 --2- 172.21.15.27:0/2626234131 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f22401618b0 0x7f224009b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.917+0000 7f2246c00640 1 --2- 172.21.15.27:0/2626234131 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22401614d0 0x7f224014a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.917+0000 7f2246c00640 1 --2- 172.21.15.27:0/2626234131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2240072380 0x7f224009d950 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.917+0000 7f2246c00640 1 -- 172.21.15.27:0/2626234131 >> 172.21.15.27:0/2626234131 conn(0x7f22400b2360 msgr2=0x7f22400d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.917+0000 7f2246c00640 1 -- 172.21.15.27:0/2626234131 shutdown_connections 2024-01-15T18:04:35.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:35.917+0000 7f2246c00640 1 -- 172.21.15.27:0/2626234131 wait complete. 2024-01-15T18:04:35.942 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-15T18:04:35.942 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-15T18:04:36.091 INFO:tasks.ceph.mgr.x.smithi093.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-01-15T18:04:36.091 INFO:tasks.ceph.mgr.x.smithi093.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-01-15T18:04:36.091 INFO:tasks.ceph.mgr.x.smithi093.stderr: from numpy import show_config as show_numpy_config 2024-01-15T18:04:36.124 INFO:tasks.ceph.mgr.y.smithi027.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-01-15T18:04:36.124 INFO:tasks.ceph.mgr.y.smithi027.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-01-15T18:04:36.124 INFO:tasks.ceph.mgr.y.smithi027.stderr: from numpy import show_config as show_numpy_config 2024-01-15T18:04:36.143 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:04:36.139+0000 7f5a768cf180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-15T18:04:36.184 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-15T18:04:36.177+0000 7f49aec8d180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-15T18:04:36.247 INFO:tasks.ceph.osd.1.smithi027.stderr:2024-01-15T18:04:36.241+0000 7f102c662640 -1 Falling back to public interface 2024-01-15T18:04:36.248 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-01-15T18:04:36.241+0000 7fa0304cd640 -1 Falling back to public interface 2024-01-15T18:04:36.259 INFO:tasks.ceph.osd.3.smithi027.stderr:2024-01-15T18:04:36.253+0000 7f70efa04640 -1 Falling back to public interface 2024-01-15T18:04:36.271 INFO:tasks.ceph.osd.5.smithi093.stderr:2024-01-15T18:04:36.267+0000 7f7b8f680640 -1 Falling back to public interface 2024-01-15T18:04:36.271 INFO:tasks.ceph.osd.4.smithi093.stderr:2024-01-15T18:04:36.267+0000 7f3d4b8c9640 -1 Falling back to public interface 2024-01-15T18:04:36.271 INFO:tasks.ceph.osd.6.smithi093.stderr:2024-01-15T18:04:36.267+0000 7f78c3e2f640 -1 Falling back to public interface 2024-01-15T18:04:36.275 INFO:tasks.ceph.osd.2.smithi027.stderr:2024-01-15T18:04:36.269+0000 7fc983ed8640 -1 Falling back to public interface 2024-01-15T18:04:36.280 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:04:36.275+0000 7f5a768cf180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-15T18:04:36.336 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-15T18:04:36.329+0000 7f49aec8d180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-15T18:04:36.351 INFO:tasks.ceph.osd.7.smithi093.stderr:2024-01-15T18:04:36.347+0000 7f3891743640 -1 Falling back to public interface 2024-01-15T18:04:36.457 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:04:36.451+0000 7f5a768cf180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-15T18:04:36.529 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:04:36.523+0000 7f5a768cf180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-15T18:04:36.534 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-15T18:04:36.529+0000 7f49aec8d180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-15T18:04:36.612 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-15T18:04:36.605+0000 7f49aec8d180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-15T18:04:36.668 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:04:36.663+0000 7f5a768cf180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-15T18:04:36.746 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-15T18:04:36.757 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-15T18:04:36.749+0000 7f49aec8d180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-15T18:04:36.879 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.873+0000 7fa359e00640 1 Processor -- start 2024-01-15T18:04:36.879 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.873+0000 7fa359e00640 1 -- start start 2024-01-15T18:04:36.881 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.873+0000 7fa359e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa35406bdd0 0x7fa35415bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:36.881 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.873+0000 7fa359e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3540af030 con 0x7fa35415c4e0 2024-01-15T18:04:36.881 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.873+0000 7fa359e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3540af310 con 0x7fa3540aeac0 2024-01-15T18:04:36.881 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.873+0000 7fa359e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa35406c1b0 con 0x7fa35406bdd0 2024-01-15T18:04:36.881 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.873+0000 7fa352000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa35415c4e0 0x7fa3540ace50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40120/0 (socket says 172.21.15.27:40120) 2024-01-15T18:04:36.881 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.873+0000 7fa352000640 1 -- 172.21.15.27:0/1944030568 learned_addr learned my addr 172.21.15.27:0/1944030568 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:04:36.881 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.873+0000 7fa352a00640 1 --2- 172.21.15.27:0/1944030568 >> v2:172.21.15.93:3300/0 conn(0x7fa35406bdd0 0x7fa35415bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:36.881 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.873+0000 7fa351600640 1 -- 172.21.15.27:0/1944030568 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285440908 0 0) 0x7fa3540af030 con 0x7fa35415c4e0 2024-01-15T18:04:36.881 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.873+0000 7fa351600640 1 -- 172.21.15.27:0/1944030568 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa334003530 con 0x7fa35415c4e0 2024-01-15T18:04:36.882 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.873+0000 7fa352a00640 1 -- 172.21.15.27:0/1944030568 >> v1:172.21.15.93:6789/0 conn(0x7fa3540aeac0 legacy=0x7fa3540713a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:36.882 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.873+0000 7fa352a00640 1 -- 172.21.15.27:0/1944030568 >> v1:172.21.15.27:6789/0 conn(0x7fa35415c4e0 legacy=0x7fa3540ace50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:36.882 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.873+0000 7fa352a00640 1 -- 172.21.15.27:0/1944030568 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3540af5f0 con 0x7fa35406bdd0 2024-01-15T18:04:36.886 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.881+0000 7fa352a00640 1 --2- 172.21.15.27:0/1944030568 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa35406bdd0 0x7fa35415bfa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa3440097c0 tx=0x7fa34401c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c90d4677e901406 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:36.886 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.881+0000 7fa351600640 1 -- 172.21.15.27:0/1944030568 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3440ab610 con 0x7fa35406bdd0 2024-01-15T18:04:36.886 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.881+0000 7fa351600640 1 -- 172.21.15.27:0/1944030568 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3440ab770 con 0x7fa35406bdd0 2024-01-15T18:04:36.886 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.881+0000 7fa351600640 1 -- 172.21.15.27:0/1944030568 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3440abae0 con 0x7fa35406bdd0 2024-01-15T18:04:36.887 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.881+0000 7fa359e00640 1 -- 172.21.15.27:0/1944030568 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa35406bdd0 msgr2=0x7fa35415bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:36.887 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.881+0000 7fa359e00640 1 --2- 172.21.15.27:0/1944030568 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa35406bdd0 0x7fa35415bfa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa3440097c0 tx=0x7fa34401c450 comp rx=0 tx=0).stop 2024-01-15T18:04:36.887 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.881+0000 7fa359e00640 1 -- 172.21.15.27:0/1944030568 shutdown_connections 2024-01-15T18:04:36.887 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.881+0000 7fa359e00640 1 --2- 172.21.15.27:0/1944030568 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa35406bdd0 0x7fa35415bfa0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:36.887 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.881+0000 7fa359e00640 1 -- 172.21.15.27:0/1944030568 >> 172.21.15.27:0/1944030568 conn(0x7fa3540b1a00 msgr2=0x7fa3540b1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:36.888 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.881+0000 7fa359e00640 1 -- 172.21.15.27:0/1944030568 shutdown_connections 2024-01-15T18:04:36.888 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.881+0000 7fa359e00640 1 -- 172.21.15.27:0/1944030568 wait complete. 2024-01-15T18:04:36.889 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.881+0000 7fa359e00640 1 Processor -- start 2024-01-15T18:04:36.889 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.881+0000 7fa359e00640 1 -- start start 2024-01-15T18:04:36.891 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.885+0000 7fa359e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa35406bdd0 0x7fa35414d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:36.891 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.885+0000 7fa359e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa3540aeac0 0x7fa3541511f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:36.891 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.885+0000 7fa359e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa35414e850 0x7fa3540cc820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:36.891 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.885+0000 7fa359e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa35414bf10 con 0x7fa35406bdd0 2024-01-15T18:04:36.891 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.885+0000 7fa359e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa35414ed30 con 0x7fa3540aeac0 2024-01-15T18:04:36.891 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.885+0000 7fa359e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa35414f010 con 0x7fa35414e850 2024-01-15T18:04:36.891 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.885+0000 7fa352a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa35406bdd0 0x7fa35414d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:36.891 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.885+0000 7fa352a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa35406bdd0 0x7fa35414d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36780/0 (socket says 172.21.15.27:36780) 2024-01-15T18:04:36.892 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.885+0000 7fa352a00640 1 -- 172.21.15.27:0/910141131 learned_addr learned my addr 172.21.15.27:0/910141131 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:36.892 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.885+0000 7fa353400640 1 --2- 172.21.15.27:0/910141131 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa35414e850 0x7fa3540cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:36.892 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.885+0000 7fa352000640 1 --2- 172.21.15.27:0/910141131 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa3540aeac0 0x7fa3541511f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:36.892 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.885+0000 7fa352a00640 1 -- 172.21.15.27:0/910141131 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa35414e850 msgr2=0x7fa3540cc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:36.892 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.885+0000 7fa352a00640 1 --2- 172.21.15.27:0/910141131 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa35414e850 0x7fa3540cc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:36.892 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.885+0000 7fa352a00640 1 -- 172.21.15.27:0/910141131 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa3540aeac0 msgr2=0x7fa3541511f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:36.892 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.885+0000 7fa352a00640 1 --2- 172.21.15.27:0/910141131 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa3540aeac0 0x7fa3541511f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:36.892 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.885+0000 7fa352a00640 1 -- 172.21.15.27:0/910141131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa344009420 con 0x7fa35406bdd0 2024-01-15T18:04:36.892 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.885+0000 7fa352a00640 1 --2- 172.21.15.27:0/910141131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa35406bdd0 0x7fa35414d090 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa34401a080 tx=0x7fa34401ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:36.893 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.885+0000 7fa34b400640 1 -- 172.21.15.27:0/910141131 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa344004220 con 0x7fa35406bdd0 2024-01-15T18:04:36.893 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.885+0000 7fa34b400640 1 -- 172.21.15.27:0/910141131 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3440b9070 con 0x7fa35406bdd0 2024-01-15T18:04:36.893 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.885+0000 7fa34b400640 1 -- 172.21.15.27:0/910141131 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa344003620 con 0x7fa35406bdd0 2024-01-15T18:04:36.893 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.885+0000 7fa359e00640 1 -- 172.21.15.27:0/910141131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa35414f2f0 con 0x7fa35406bdd0 2024-01-15T18:04:36.893 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.885+0000 7fa359e00640 1 -- 172.21.15.27:0/910141131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa35414f8b0 con 0x7fa35406bdd0 2024-01-15T18:04:36.893 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.885+0000 7fa34b400640 1 -- 172.21.15.27:0/910141131 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa344003780 con 0x7fa35406bdd0 2024-01-15T18:04:36.893 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.885+0000 7fa34b400640 1 -- 172.21.15.27:0/910141131 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fa3440bb070 con 0x7fa35406bdd0 2024-01-15T18:04:36.893 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.885+0000 7fa359e00640 1 -- 172.21.15.27:0/910141131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa35414f150 con 0x7fa35406bdd0 2024-01-15T18:04:36.895 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.889+0000 7fa34b400640 1 -- 172.21.15.27:0/910141131 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fa35414f150 con 0x7fa35406bdd0 2024-01-15T18:04:36.954 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.949+0000 7fa359e00640 1 -- 172.21.15.27:0/910141131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa354071a70 con 0x7fa35406bdd0 2024-01-15T18:04:36.955 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.949+0000 7fa34b400640 1 -- 172.21.15.27:0/910141131 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fa3440c4060 con 0x7fa35406bdd0 2024-01-15T18:04:36.955 INFO:teuthology.misc.health.smithi027.stdout: 2024-01-15T18:04:36.955 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":10,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:04:35.557795+0000","last_up_change":"0.000000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:04:36.956 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.949+0000 7fa359e00640 1 -- 172.21.15.27:0/910141131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa35406bdd0 msgr2=0x7fa35414d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:36.956 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.949+0000 7fa359e00640 1 --2- 172.21.15.27:0/910141131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa35406bdd0 0x7fa35414d090 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa34401a080 tx=0x7fa34401ac90 comp rx=0 tx=0).stop 2024-01-15T18:04:36.956 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.949+0000 7fa359e00640 1 -- 172.21.15.27:0/910141131 shutdown_connections 2024-01-15T18:04:36.956 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.949+0000 7fa359e00640 1 --2- 172.21.15.27:0/910141131 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa35414e850 0x7fa3540cc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:36.956 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.949+0000 7fa359e00640 1 --2- 172.21.15.27:0/910141131 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa3540aeac0 0x7fa3541511f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:36.957 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.949+0000 7fa359e00640 1 --2- 172.21.15.27:0/910141131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa35406bdd0 0x7fa35414d090 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:36.957 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.949+0000 7fa359e00640 1 -- 172.21.15.27:0/910141131 >> 172.21.15.27:0/910141131 conn(0x7fa3540b1a00 msgr2=0x7fa3540b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:36.957 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.949+0000 7fa359e00640 1 -- 172.21.15.27:0/910141131 shutdown_connections 2024-01-15T18:04:36.957 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:36.949+0000 7fa359e00640 1 -- 172.21.15.27:0/910141131 wait complete. 2024-01-15T18:04:36.974 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-15T18:04:37.363 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:04:37.359+0000 7f5a768cf180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-15T18:04:37.449 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:04:37.443+0000 7f5a768cf180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-15T18:04:37.474 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-15T18:04:37.469+0000 7f49aec8d180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-15T18:04:37.566 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-15T18:04:37.561+0000 7f49aec8d180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-15T18:04:37.643 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:04:37.639+0000 7f5a768cf180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-15T18:04:37.727 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:04:37.723+0000 7f5a768cf180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-15T18:04:37.768 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-15T18:04:37.761+0000 7f49aec8d180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-15T18:04:37.857 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-15T18:04:37.849+0000 7f49aec8d180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-15T18:04:38.053 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:04:38.047+0000 7f5a768cf180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-15T18:04:38.122 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:04:38.119+0000 7f5a768cf180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-15T18:04:38.205 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-15T18:04:38.197+0000 7f49aec8d180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-15T18:04:38.276 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-01-15T18:04:38.269+0000 7f49aec8d180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-15T18:04:39.144 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:04:39.139+0000 7f5a1cc00640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-01-15T18:04:39.145 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:04:39.139+0000 7f5a1cc00640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-01-15T18:04:40.078 INFO:tasks.ceph.osd.5.smithi093.stderr:2024-01-15T18:04:40.071+0000 7f7b8f680640 -1 osd.5 0 log_to_monitors true 2024-01-15T18:04:40.082 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-01-15T18:04:40.077+0000 7fa0304cd640 -1 osd.0 0 log_to_monitors true 2024-01-15T18:04:40.137 INFO:tasks.ceph.osd.6.smithi093.stderr:2024-01-15T18:04:40.131+0000 7f78c3e2f640 -1 osd.6 0 log_to_monitors true 2024-01-15T18:04:40.224 INFO:tasks.ceph.osd.1.smithi027.stderr:2024-01-15T18:04:40.217+0000 7f102c662640 -1 osd.1 0 log_to_monitors true 2024-01-15T18:04:40.233 INFO:tasks.ceph.osd.7.smithi093.stderr:2024-01-15T18:04:40.227+0000 7f3891743640 -1 osd.7 0 log_to_monitors true 2024-01-15T18:04:40.242 INFO:tasks.ceph.osd.4.smithi093.stderr:2024-01-15T18:04:40.239+0000 7f3d4b8c9640 -1 osd.4 0 log_to_monitors true 2024-01-15T18:04:40.284 INFO:tasks.ceph.osd.2.smithi027.stderr:2024-01-15T18:04:40.277+0000 7fc983ed8640 -1 osd.2 0 log_to_monitors true 2024-01-15T18:04:40.301 INFO:tasks.ceph.osd.3.smithi027.stderr:2024-01-15T18:04:40.293+0000 7f70efa04640 -1 osd.3 0 log_to_monitors true 2024-01-15T18:04:42.264 INFO:tasks.ceph.osd.5.smithi093.stderr:2024-01-15T18:04:42.259+0000 7f7b89c00640 -1 osd.5 0 waiting for initial osdmap 2024-01-15T18:04:42.393 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-01-15T18:04:42.385+0000 7fa02c800640 -1 osd.0 0 waiting for initial osdmap 2024-01-15T18:04:42.431 INFO:tasks.ceph.osd.5.smithi093.stderr:2024-01-15T18:04:42.427+0000 7f7b80600640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T18:04:42.471 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-01-15T18:04:42.465+0000 7fa021400640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T18:04:43.783 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-15T18:04:43.987 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.981+0000 7fdcce800640 1 Processor -- start 2024-01-15T18:04:43.987 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.981+0000 7fdcce800640 1 -- start start 2024-01-15T18:04:43.988 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.981+0000 7fdcce800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdcc815ed10 0x7fdcc815f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:43.989 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.981+0000 7fdcce800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc80cb9d0 con 0x7fdcc80b1df0 2024-01-15T18:04:43.989 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.981+0000 7fdcce800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc80cc810 con 0x7fdcc815b230 2024-01-15T18:04:43.989 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.981+0000 7fdcce800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc815f6d0 con 0x7fdcc815ed10 2024-01-15T18:04:43.989 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.981+0000 7fdccca00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdcc815ed10 0x7fdcc815f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:43.989 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.981+0000 7fdccd400640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7fdcc815b230 0x7fdcc80ceb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:40308/0 (socket says 172.21.15.27:40308) 2024-01-15T18:04:43.990 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.981+0000 7fdccd400640 1 -- 172.21.15.27:0/3822536366 learned_addr learned my addr 172.21.15.27:0/3822536366 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:04:43.990 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.981+0000 7fdcc7e00640 1 -- 172.21.15.27:0/3822536366 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975568335 0 0) 0x7fdcc80cc810 con 0x7fdcc815b230 2024-01-15T18:04:43.990 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.981+0000 7fdcc7e00640 1 -- 172.21.15.27:0/3822536366 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdca8003530 con 0x7fdcc815b230 2024-01-15T18:04:43.991 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.985+0000 7fdcc7e00640 1 -- 172.21.15.27:0/3822536366 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767025836 0 0) 0x7fdcc80cb9d0 con 0x7fdcc80b1df0 2024-01-15T18:04:43.991 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.985+0000 7fdcc7e00640 1 -- 172.21.15.27:0/3822536366 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc80cc810 con 0x7fdcc80b1df0 2024-01-15T18:04:43.991 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.985+0000 7fdccca00640 1 -- 172.21.15.27:0/3822536366 >> v1:172.21.15.93:6789/0 conn(0x7fdcc815b230 legacy=0x7fdcc80ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:43.991 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.985+0000 7fdcc7e00640 1 -- 172.21.15.27:0/3822536366 <== mon.1 v1:172.21.15.93:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2292027652 0 0) 0x7fdca8003530 con 0x7fdcc815b230 2024-01-15T18:04:43.991 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.985+0000 7fdccca00640 1 -- 172.21.15.27:0/3822536366 >> v1:172.21.15.27:6789/0 conn(0x7fdcc80b1df0 legacy=0x7fdcc80b2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:43.991 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.985+0000 7fdccca00640 1 -- 172.21.15.27:0/3822536366 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc815b610 con 0x7fdcc815ed10 2024-01-15T18:04:43.995 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.989+0000 7fdccca00640 1 --2- 172.21.15.27:0/3822536366 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdcc815ed10 0x7fdcc815f190 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8009770 tx=0x7fdcb801c2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=597ff156577c68e8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:43.996 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.989+0000 7fdcc7e00640 1 -- 172.21.15.27:0/3822536366 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0001a00 con 0x7fdcc815ed10 2024-01-15T18:04:43.996 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.989+0000 7fdcc7e00640 1 -- 172.21.15.27:0/3822536366 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb80a7050 con 0x7fdcc815ed10 2024-01-15T18:04:43.996 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.989+0000 7fdcc7e00640 1 -- 172.21.15.27:0/3822536366 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb80ab620 con 0x7fdcc815ed10 2024-01-15T18:04:43.997 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.989+0000 7fdcce800640 1 -- 172.21.15.27:0/3822536366 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdcc815ed10 msgr2=0x7fdcc815f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:43.997 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.989+0000 7fdcce800640 1 --2- 172.21.15.27:0/3822536366 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdcc815ed10 0x7fdcc815f190 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8009770 tx=0x7fdcb801c2f0 comp rx=0 tx=0).stop 2024-01-15T18:04:43.998 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.989+0000 7fdcce800640 1 -- 172.21.15.27:0/3822536366 shutdown_connections 2024-01-15T18:04:43.998 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.989+0000 7fdcce800640 1 --2- 172.21.15.27:0/3822536366 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdcc815ed10 0x7fdcc815f190 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:43.998 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.989+0000 7fdcce800640 1 -- 172.21.15.27:0/3822536366 >> 172.21.15.27:0/3822536366 conn(0x7fdcc80d0f30 msgr2=0x7fdcc80d1340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:43.998 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.989+0000 7fdcce800640 1 -- 172.21.15.27:0/3822536366 shutdown_connections 2024-01-15T18:04:43.998 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.993+0000 7fdcce800640 1 -- 172.21.15.27:0/3822536366 wait complete. 2024-01-15T18:04:44.001 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.993+0000 7fdcce800640 1 Processor -- start 2024-01-15T18:04:44.001 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.993+0000 7fdcce800640 1 -- start start 2024-01-15T18:04:44.003 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.997+0000 7fdcce800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcc80b1df0 0x7fdcc8148d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:44.005 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.997+0000 7fdcce800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdcc815b230 0x7fdcc815c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:44.005 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.997+0000 7fdcce800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdcc815ed10 0x7fdcc815cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:44.005 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.997+0000 7fdcce800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc815d2c0 con 0x7fdcc80b1df0 2024-01-15T18:04:44.005 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.997+0000 7fdcce800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc80cab40 con 0x7fdcc815ed10 2024-01-15T18:04:44.005 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.997+0000 7fdcce800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc80c7670 con 0x7fdcc815b230 2024-01-15T18:04:44.005 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.997+0000 7fdccd400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcc80b1df0 0x7fdcc8148d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:44.005 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.997+0000 7fdccca00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdcc815b230 0x7fdcc815c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:44.005 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.997+0000 7fdccca00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdcc815b230 0x7fdcc815c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:54206/0 (socket says 172.21.15.27:54206) 2024-01-15T18:04:44.005 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.997+0000 7fdccca00640 1 -- 172.21.15.27:0/261999263 learned_addr learned my addr 172.21.15.27:0/261999263 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:44.005 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.997+0000 7fdccde00640 1 --2- 172.21.15.27:0/261999263 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdcc815ed10 0x7fdcc815cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:44.006 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.997+0000 7fdccde00640 1 -- 172.21.15.27:0/261999263 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdcc815b230 msgr2=0x7fdcc815c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:44.006 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.997+0000 7fdccde00640 1 --2- 172.21.15.27:0/261999263 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdcc815b230 0x7fdcc815c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:44.007 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.997+0000 7fdccde00640 1 -- 172.21.15.27:0/261999263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcc80b1df0 msgr2=0x7fdcc8148d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:44.007 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.997+0000 7fdccde00640 1 --2- 172.21.15.27:0/261999263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcc80b1df0 0x7fdcc8148d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:44.007 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.997+0000 7fdccde00640 1 -- 172.21.15.27:0/261999263 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb8009420 con 0x7fdcc815ed10 2024-01-15T18:04:44.007 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.997+0000 7fdccca00640 1 --2- 172.21.15.27:0/261999263 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdcc815b230 0x7fdcc815c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:04:44.007 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.997+0000 7fdccd400640 1 --2- 172.21.15.27:0/261999263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcc80b1df0 0x7fdcc8148d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:04:44.007 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.997+0000 7fdccde00640 1 --2- 172.21.15.27:0/261999263 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdcc815ed10 0x7fdcc815cd80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdcb000c3c0 tx=0x7fdcb0001e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:44.007 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.997+0000 7fdcc5600640 1 -- 172.21.15.27:0/261999263 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb00052d0 con 0x7fdcc815ed10 2024-01-15T18:04:44.007 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.997+0000 7fdcc5600640 1 -- 172.21.15.27:0/261999263 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb0005430 con 0x7fdcc815ed10 2024-01-15T18:04:44.007 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:43.997+0000 7fdcc5600640 1 -- 172.21.15.27:0/261999263 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb0003820 con 0x7fdcc815ed10 2024-01-15T18:04:44.007 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:44.001+0000 7fdcce800640 1 -- 172.21.15.27:0/261999263 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc80c7c30 con 0x7fdcc815ed10 2024-01-15T18:04:44.007 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:44.001+0000 7fdcce800640 1 -- 172.21.15.27:0/261999263 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc81492d0 con 0x7fdcc815ed10 2024-01-15T18:04:44.008 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:44.001+0000 7fdcce800640 1 -- 172.21.15.27:0/261999263 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc8067a10 con 0x7fdcc815ed10 2024-01-15T18:04:44.010 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:44.001+0000 7fdcc5600640 1 -- 172.21.15.27:0/261999263 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 4) v1 ==== 86104+0+0 (secure 0 0 0) 0x7fdcb0003980 con 0x7fdcc815ed10 2024-01-15T18:04:44.011 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:44.005+0000 7fdcc5600640 1 --2- 172.21.15.27:0/261999263 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdca00668b0 0x7fdca0068d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:44.012 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:44.005+0000 7fdcc5600640 1 -- 172.21.15.27:0/261999263 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2791+0+0 (secure 0 0 0) 0x7fdcb0082c00 con 0x7fdcc815ed10 2024-01-15T18:04:44.012 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:44.005+0000 7fdccd400640 1 --2- 172.21.15.27:0/261999263 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdca00668b0 0x7fdca0068d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:44.012 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:44.005+0000 7fdccd400640 1 --2- 172.21.15.27:0/261999263 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdca00668b0 0x7fdca0068d70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0001570 tx=0x7fdcc0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:44.021 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:44.013+0000 7fdcc5600640 1 -- 172.21.15.27:0/261999263 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fdcb0050ba0 con 0x7fdcc815ed10 2024-01-15T18:04:44.043 INFO:tasks.ceph.osd.4.smithi093.stderr:2024-01-15T18:04:44.039+0000 7f3d45e00640 -1 osd.4 0 waiting for initial osdmap 2024-01-15T18:04:44.050 INFO:tasks.ceph.osd.3.smithi027.stderr:2024-01-15T18:04:44.045+0000 7f70e9e00640 -1 osd.3 0 waiting for initial osdmap 2024-01-15T18:04:44.055 INFO:tasks.ceph.osd.1.smithi027.stderr:2024-01-15T18:04:44.049+0000 7f1026c00640 -1 osd.1 0 waiting for initial osdmap 2024-01-15T18:04:44.079 INFO:tasks.ceph.osd.4.smithi093.stderr:2024-01-15T18:04:44.075+0000 7f3d3c800640 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T18:04:44.095 INFO:tasks.ceph.osd.3.smithi027.stderr:2024-01-15T18:04:44.089+0000 7f70e0800640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T18:04:44.096 INFO:tasks.ceph.osd.1.smithi027.stderr:2024-01-15T18:04:44.089+0000 7f101d600640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T18:04:44.100 INFO:tasks.ceph.osd.6.smithi093.stderr:2024-01-15T18:04:44.095+0000 7f78c0000640 -1 osd.6 0 waiting for initial osdmap 2024-01-15T18:04:44.123 INFO:tasks.ceph.osd.6.smithi093.stderr:2024-01-15T18:04:44.119+0000 7f78b4c00640 -1 osd.6 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T18:04:44.125 INFO:tasks.ceph.osd.7.smithi093.stderr:2024-01-15T18:04:44.119+0000 7f388bc00640 -1 osd.7 0 waiting for initial osdmap 2024-01-15T18:04:44.175 INFO:tasks.ceph.osd.7.smithi093.stderr:2024-01-15T18:04:44.171+0000 7f3882600640 -1 osd.7 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T18:04:44.179 INFO:tasks.ceph.osd.2.smithi027.stderr:2024-01-15T18:04:44.173+0000 7fc97e400640 -1 osd.2 0 waiting for initial osdmap 2024-01-15T18:04:44.201 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:44.193+0000 7fdcce800640 1 -- 172.21.15.27:0/261999263 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcc815d400 con 0x7fdcc815ed10 2024-01-15T18:04:44.205 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:44.197+0000 7fdcc5600640 1 -- 172.21.15.27:0/261999263 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+8016 (secure 0 0 0) 0x7fdcc815d400 con 0x7fdcc815ed10 2024-01-15T18:04:44.206 INFO:teuthology.misc.health.smithi027.stdout: 2024-01-15T18:04:44.206 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":14,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:04:43.440439+0000","last_up_change":"2024-01-15T18:04:43.440439+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:04:44.208 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:44.201+0000 7fdcce800640 1 -- 172.21.15.27:0/261999263 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdca00668b0 msgr2=0x7fdca0068d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:44.208 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:44.201+0000 7fdcce800640 1 --2- 172.21.15.27:0/261999263 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdca00668b0 0x7fdca0068d70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0001570 tx=0x7fdcc0009040 comp rx=0 tx=0).stop 2024-01-15T18:04:44.208 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:44.201+0000 7fdcce800640 1 -- 172.21.15.27:0/261999263 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdcc815ed10 msgr2=0x7fdcc815cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:44.208 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:44.201+0000 7fdcce800640 1 --2- 172.21.15.27:0/261999263 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdcc815ed10 0x7fdcc815cd80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdcb000c3c0 tx=0x7fdcb0001e80 comp rx=0 tx=0).stop 2024-01-15T18:04:44.209 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:44.201+0000 7fdcce800640 1 -- 172.21.15.27:0/261999263 shutdown_connections 2024-01-15T18:04:44.209 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:44.201+0000 7fdcce800640 1 --2- 172.21.15.27:0/261999263 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdca00668b0 0x7fdca0068d70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:44.209 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:44.201+0000 7fdcce800640 1 --2- 172.21.15.27:0/261999263 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdcc815ed10 0x7fdcc815cd80 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:44.209 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:44.201+0000 7fdcce800640 1 --2- 172.21.15.27:0/261999263 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdcc815b230 0x7fdcc815c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:44.209 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:44.201+0000 7fdcce800640 1 --2- 172.21.15.27:0/261999263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdcc80b1df0 0x7fdcc8148d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:44.209 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:44.201+0000 7fdcce800640 1 -- 172.21.15.27:0/261999263 >> 172.21.15.27:0/261999263 conn(0x7fdcc80d0f30 msgr2=0x7fdcc806bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:44.209 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:44.201+0000 7fdcce800640 1 -- 172.21.15.27:0/261999263 shutdown_connections 2024-01-15T18:04:44.209 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:44.201+0000 7fdcce800640 1 -- 172.21.15.27:0/261999263 wait complete. 2024-01-15T18:04:44.226 DEBUG:teuthology.misc:2 of 8 OSDs are up 2024-01-15T18:04:44.235 INFO:tasks.ceph.osd.2.smithi027.stderr:2024-01-15T18:04:44.229+0000 7fc974e00640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T18:04:51.032 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-15T18:04:51.175 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.168+0000 7f9177e00640 1 Processor -- start 2024-01-15T18:04:51.175 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.168+0000 7f9177e00640 1 -- start start 2024-01-15T18:04:51.176 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.168+0000 7f9177e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f91700b0500 0x7f9170165c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:51.176 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.168+0000 7f9177e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f917015b540 con 0x7f917015b160 2024-01-15T18:04:51.176 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.168+0000 7f9177e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91700ae1f0 con 0x7f91700b2570 2024-01-15T18:04:51.176 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.168+0000 7f9177e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91700cb780 con 0x7f91700b0500 2024-01-15T18:04:51.176 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.168+0000 7f9175e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f91700b0500 0x7f9170165c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:51.176 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.168+0000 7f9175400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f917015b160 0x7f91700adae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52850/0 (socket says 172.21.15.27:52850) 2024-01-15T18:04:51.176 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.168+0000 7f9175400640 1 -- 172.21.15.27:0/2408003484 learned_addr learned my addr 172.21.15.27:0/2408003484 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:04:51.177 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.168+0000 7f9175e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f91700b0500 0x7f9170165c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56680/0 (socket says 172.21.15.27:56680) 2024-01-15T18:04:51.177 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.168+0000 7f9175e00640 1 -- 172.21.15.27:0/2408003484 >> v1:172.21.15.93:6789/0 conn(0x7f91700b2570 legacy=0x7f91700d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:51.177 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.168+0000 7f9175e00640 1 -- 172.21.15.27:0/2408003484 >> v1:172.21.15.27:6789/0 conn(0x7f917015b160 legacy=0x7f91700adae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:51.177 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.168+0000 7f9175e00640 1 -- 172.21.15.27:0/2408003484 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91700ae450 con 0x7f91700b0500 2024-01-15T18:04:51.180 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.172+0000 7f9175e00640 1 --2- 172.21.15.27:0/2408003484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91700b0500 0x7f9170165c20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9168009d90 tx=0x7f916801c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a9121ea925082d7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:51.180 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.172+0000 7f916fe00640 1 -- 172.21.15.27:0/2408003484 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91680a5950 con 0x7f91700b0500 2024-01-15T18:04:51.180 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.172+0000 7f916fe00640 1 -- 172.21.15.27:0/2408003484 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91680a5ab0 con 0x7f91700b0500 2024-01-15T18:04:51.180 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.172+0000 7f916fe00640 1 -- 172.21.15.27:0/2408003484 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91680a5e40 con 0x7f91700b0500 2024-01-15T18:04:51.181 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.172+0000 7f9177e00640 1 -- 172.21.15.27:0/2408003484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91700b0500 msgr2=0x7f9170165c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:51.181 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.172+0000 7f9177e00640 1 --2- 172.21.15.27:0/2408003484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91700b0500 0x7f9170165c20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9168009d90 tx=0x7f916801c890 comp rx=0 tx=0).stop 2024-01-15T18:04:51.181 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.172+0000 7f9177e00640 1 -- 172.21.15.27:0/2408003484 shutdown_connections 2024-01-15T18:04:51.181 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.172+0000 7f9177e00640 1 --2- 172.21.15.27:0/2408003484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91700b0500 0x7f9170165c20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:51.181 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.172+0000 7f9177e00640 1 -- 172.21.15.27:0/2408003484 >> 172.21.15.27:0/2408003484 conn(0x7f9170099a10 msgr2=0x7f9170099e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:51.181 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.172+0000 7f9177e00640 1 -- 172.21.15.27:0/2408003484 shutdown_connections 2024-01-15T18:04:51.181 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.172+0000 7f9177e00640 1 -- 172.21.15.27:0/2408003484 wait complete. 2024-01-15T18:04:51.183 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.176+0000 7f9177e00640 1 Processor -- start 2024-01-15T18:04:51.183 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.176+0000 7f9177e00640 1 -- start start 2024-01-15T18:04:51.185 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.176+0000 7f9177e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f91700b0500 0x7f9170149040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:51.185 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.176+0000 7f9177e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f917015b160 0x7f917014f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:51.185 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.176+0000 7f9177e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917014fae0 0x7f9170152590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:51.185 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.176+0000 7f9177e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91701f6660 con 0x7f917014fae0 2024-01-15T18:04:51.185 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.176+0000 7f9177e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917014e1e0 con 0x7f917015b160 2024-01-15T18:04:51.185 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.176+0000 7f9177e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f917014e4c0 con 0x7f91700b0500 2024-01-15T18:04:51.186 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.176+0000 7f9175e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917014fae0 0x7f9170152590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:51.186 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.176+0000 7f9175e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917014fae0 0x7f9170152590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56692/0 (socket says 172.21.15.27:56692) 2024-01-15T18:04:51.186 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.176+0000 7f9175e00640 1 -- 172.21.15.27:0/3551586312 learned_addr learned my addr 172.21.15.27:0/3551586312 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:51.186 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.176+0000 7f9175400640 1 --2- 172.21.15.27:0/3551586312 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f91700b0500 0x7f9170149040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:51.186 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.176+0000 7f9174a00640 1 --2- 172.21.15.27:0/3551586312 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f917015b160 0x7f917014f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:51.186 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.176+0000 7f9175e00640 1 -- 172.21.15.27:0/3551586312 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f91700b0500 msgr2=0x7f9170149040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:51.186 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.176+0000 7f9175e00640 1 --2- 172.21.15.27:0/3551586312 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f91700b0500 0x7f9170149040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:51.186 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.176+0000 7f9175e00640 1 -- 172.21.15.27:0/3551586312 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f917015b160 msgr2=0x7f917014f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:51.186 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.176+0000 7f9175e00640 1 --2- 172.21.15.27:0/3551586312 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f917015b160 0x7f917014f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:51.186 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.176+0000 7f9175e00640 1 -- 172.21.15.27:0/3551586312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9168009a70 con 0x7f917014fae0 2024-01-15T18:04:51.186 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.176+0000 7f9175e00640 1 --2- 172.21.15.27:0/3551586312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917014fae0 0x7f9170152590 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f91680b0040 tx=0x7f9168019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:51.186 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.176+0000 7f916d600640 1 -- 172.21.15.27:0/3551586312 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91680ade40 con 0x7f917014fae0 2024-01-15T18:04:51.186 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.176+0000 7f916d600640 1 -- 172.21.15.27:0/3551586312 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91680b9450 con 0x7f917014fae0 2024-01-15T18:04:51.186 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.176+0000 7f9177e00640 1 -- 172.21.15.27:0/3551586312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f917014e7a0 con 0x7f917014fae0 2024-01-15T18:04:51.186 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.176+0000 7f916d600640 1 -- 172.21.15.27:0/3551586312 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91680b9800 con 0x7f917014fae0 2024-01-15T18:04:51.187 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.176+0000 7f9177e00640 1 -- 172.21.15.27:0/3551586312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f917014ed60 con 0x7f917014fae0 2024-01-15T18:04:51.187 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.180+0000 7f9177e00640 1 -- 172.21.15.27:0/3551586312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f917014e600 con 0x7f917014fae0 2024-01-15T18:04:51.187 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.180+0000 7f916d600640 1 -- 172.21.15.27:0/3551586312 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f91680b9960 con 0x7f917014fae0 2024-01-15T18:04:51.188 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.180+0000 7f916d600640 1 --2- 172.21.15.27:0/3551586312 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f91480668f0 0x7f9148068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:51.188 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.180+0000 7f916d600640 1 -- 172.21.15.27:0/3551586312 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f916811fa30 con 0x7f917014fae0 2024-01-15T18:04:51.188 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.180+0000 7f9175400640 1 --2- 172.21.15.27:0/3551586312 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f91480668f0 0x7f9148068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:51.189 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.180+0000 7f9175400640 1 --2- 172.21.15.27:0/3551586312 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f91480668f0 0x7f9148068db0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9160005890 tx=0x7f9160009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:51.193 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.184+0000 7f916d600640 1 -- 172.21.15.27:0/3551586312 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f917014e600 con 0x7f917014fae0 2024-01-15T18:04:51.351 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.344+0000 7f9177e00640 1 -- 172.21.15.27:0/3551586312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f917014e600 con 0x7f917014fae0 2024-01-15T18:04:51.352 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.344+0000 7f916d600640 1 -- 172.21.15.27:0/3551586312 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12754 (secure 0 0 0) 0x7f917014e600 con 0x7f917014fae0 2024-01-15T18:04:51.352 INFO:teuthology.misc.health.smithi027.stdout: 2024-01-15T18:04:51.352 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":17,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:04:46.449830+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:04:51.354 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.348+0000 7f9177e00640 1 -- 172.21.15.27:0/3551586312 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f91480668f0 msgr2=0x7f9148068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:51.355 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.348+0000 7f9177e00640 1 --2- 172.21.15.27:0/3551586312 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f91480668f0 0x7f9148068db0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9160005890 tx=0x7f9160009040 comp rx=0 tx=0).stop 2024-01-15T18:04:51.355 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.348+0000 7f9177e00640 1 -- 172.21.15.27:0/3551586312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917014fae0 msgr2=0x7f9170152590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:51.355 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.348+0000 7f9177e00640 1 --2- 172.21.15.27:0/3551586312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917014fae0 0x7f9170152590 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f91680b0040 tx=0x7f9168019630 comp rx=0 tx=0).stop 2024-01-15T18:04:51.355 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.348+0000 7f9177e00640 1 -- 172.21.15.27:0/3551586312 shutdown_connections 2024-01-15T18:04:51.355 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.348+0000 7f9177e00640 1 --2- 172.21.15.27:0/3551586312 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f91480668f0 0x7f9148068db0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:51.355 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.348+0000 7f9177e00640 1 --2- 172.21.15.27:0/3551586312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f917014fae0 0x7f9170152590 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:51.355 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.348+0000 7f9177e00640 1 --2- 172.21.15.27:0/3551586312 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f917015b160 0x7f917014f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:51.355 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.348+0000 7f9177e00640 1 --2- 172.21.15.27:0/3551586312 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f91700b0500 0x7f9170149040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:51.355 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.348+0000 7f9177e00640 1 -- 172.21.15.27:0/3551586312 >> 172.21.15.27:0/3551586312 conn(0x7f9170099a10 msgr2=0x7f91700b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:51.355 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.348+0000 7f9177e00640 1 -- 172.21.15.27:0/3551586312 shutdown_connections 2024-01-15T18:04:51.355 INFO:teuthology.misc.health.smithi027.stderr:2024-01-15T18:04:51.348+0000 7f9177e00640 1 -- 172.21.15.27:0/3551586312 wait complete. 2024-01-15T18:04:51.372 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-01-15T18:04:51.372 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-15T18:04:51.372 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-15T18:04:51.373 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-15T18:04:51.376 INFO:tasks.ceph.mds.a:Started 2024-01-15T18:04:51.376 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-15T18:04:51.376 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-15T18:04:51.379 INFO:tasks.ceph.mds.c:Started 2024-01-15T18:04:51.379 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-15T18:04:51.379 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-15T18:04:51.382 INFO:tasks.ceph.mds.b:Started 2024-01-15T18:04:51.382 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-15T18:04:51.383 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-15T18:04:51.385 INFO:tasks.ceph.mds.d:Started 2024-01-15T18:04:51.386 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-01-15T18:04:51.386 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:04:51.480 INFO:tasks.ceph.mds.b.smithi093.stdout:starting mds.b at 2024-01-15T18:04:51.481 INFO:tasks.ceph.mds.d.smithi093.stdout:starting mds.d at 2024-01-15T18:04:51.484 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-15T18:04:51.484 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-15T18:04:51.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.528+0000 7fab77800640 1 Processor -- start 2024-01-15T18:04:51.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.528+0000 7fab77800640 1 -- start start 2024-01-15T18:04:51.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.528+0000 7fab77800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fab7006bb80 0x7fab700cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:51.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.528+0000 7fab77800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab700b2770 con 0x7fab700b2390 2024-01-15T18:04:51.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.528+0000 7fab77800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab7006c060 con 0x7fab700cddc0 2024-01-15T18:04:51.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.528+0000 7fab77800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab700ce270 con 0x7fab7006bb80 2024-01-15T18:04:51.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.528+0000 7fab6fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fab7006bb80 0x7fab700cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:51.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.528+0000 7fab74e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fab700b2390 0x7fab700cf780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52894/0 (socket says 172.21.15.27:52894) 2024-01-15T18:04:51.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.528+0000 7fab74e00640 1 -- 172.21.15.27:0/1025121258 learned_addr learned my addr 172.21.15.27:0/1025121258 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:04:51.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.528+0000 7fab6fe00640 1 -- 172.21.15.27:0/1025121258 >> v1:172.21.15.93:6789/0 conn(0x7fab700cddc0 legacy=0x7fab700b0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:51.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.528+0000 7fab6f400640 1 -- 172.21.15.27:0/1025121258 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616982811 0 0) 0x7fab700b2770 con 0x7fab700b2390 2024-01-15T18:04:51.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.528+0000 7fab6fe00640 1 -- 172.21.15.27:0/1025121258 >> v1:172.21.15.27:6789/0 conn(0x7fab700b2390 legacy=0x7fab700cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:51.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.528+0000 7fab6fe00640 1 -- 172.21.15.27:0/1025121258 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7006c340 con 0x7fab7006bb80 2024-01-15T18:04:51.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.532+0000 7fab6fe00640 1 --2- 172.21.15.27:0/1025121258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab7006bb80 0x7fab700cd880 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fab580097c0 tx=0x7fab5801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e98534682a561fbb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:51.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.532+0000 7fab6f400640 1 -- 172.21.15.27:0/1025121258 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab580ab650 con 0x7fab7006bb80 2024-01-15T18:04:51.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.532+0000 7fab6f400640 1 -- 172.21.15.27:0/1025121258 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab580ab7b0 con 0x7fab7006bb80 2024-01-15T18:04:51.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.532+0000 7fab6f400640 1 -- 172.21.15.27:0/1025121258 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab580abb40 con 0x7fab7006bb80 2024-01-15T18:04:51.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.532+0000 7fab77800640 1 -- 172.21.15.27:0/1025121258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab7006bb80 msgr2=0x7fab700cd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:51.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.532+0000 7fab77800640 1 --2- 172.21.15.27:0/1025121258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab7006bb80 0x7fab700cd880 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fab580097c0 tx=0x7fab5801c450 comp rx=0 tx=0).stop 2024-01-15T18:04:51.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.532+0000 7fab77800640 1 -- 172.21.15.27:0/1025121258 shutdown_connections 2024-01-15T18:04:51.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.532+0000 7fab77800640 1 --2- 172.21.15.27:0/1025121258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab7006bb80 0x7fab700cd880 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:51.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.532+0000 7fab77800640 1 -- 172.21.15.27:0/1025121258 >> 172.21.15.27:0/1025121258 conn(0x7fab700b2a10 msgr2=0x7fab700b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:51.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.532+0000 7fab77800640 1 -- 172.21.15.27:0/1025121258 shutdown_connections 2024-01-15T18:04:51.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.532+0000 7fab77800640 1 -- 172.21.15.27:0/1025121258 wait complete. 2024-01-15T18:04:51.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab77800640 1 Processor -- start 2024-01-15T18:04:51.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab77800640 1 -- start start 2024-01-15T18:04:51.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab77800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab7006bb80 0x7fab7006ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:51.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab77800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fab700b2390 0x7fab700c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:51.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab77800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab700cddc0 0x7fab70151060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:51.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab77800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab700c73d0 con 0x7fab7006bb80 2024-01-15T18:04:51.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab77800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab700c6750 con 0x7fab700cddc0 2024-01-15T18:04:51.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab77800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab700c6a30 con 0x7fab700b2390 2024-01-15T18:04:51.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab74e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab7006bb80 0x7fab7006ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:51.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab74e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab7006bb80 0x7fab7006ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56720/0 (socket says 172.21.15.27:56720) 2024-01-15T18:04:51.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab74e00640 1 -- 172.21.15.27:0/3997741113 learned_addr learned my addr 172.21.15.27:0/3997741113 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:51.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab74e00640 1 -- 172.21.15.27:0/3997741113 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fab700b2390 msgr2=0x7fab700c5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:51.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab6fe00640 1 --2- 172.21.15.27:0/3997741113 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fab700b2390 0x7fab700c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:51.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab75800640 1 --2- 172.21.15.27:0/3997741113 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab700cddc0 0x7fab70151060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:51.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab74e00640 1 --2- 172.21.15.27:0/3997741113 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fab700b2390 0x7fab700c5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:51.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab74e00640 1 -- 172.21.15.27:0/3997741113 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab700cddc0 msgr2=0x7fab70151060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:51.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab74e00640 1 --2- 172.21.15.27:0/3997741113 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab700cddc0 0x7fab70151060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:51.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab74e00640 1 -- 172.21.15.27:0/3997741113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab58009420 con 0x7fab7006bb80 2024-01-15T18:04:51.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab6fe00640 1 --2- 172.21.15.27:0/3997741113 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fab700b2390 0x7fab700c5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:04:51.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab75800640 1 --2- 172.21.15.27:0/3997741113 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab700cddc0 0x7fab70151060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:04:51.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab74e00640 1 --2- 172.21.15.27:0/3997741113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab7006bb80 0x7fab7006ebf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fab6000c4a0 tx=0x7fab6000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:51.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab6cc00640 1 -- 172.21.15.27:0/3997741113 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab600074e0 con 0x7fab7006bb80 2024-01-15T18:04:51.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab6cc00640 1 -- 172.21.15.27:0/3997741113 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab60007640 con 0x7fab7006bb80 2024-01-15T18:04:51.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab6cc00640 1 -- 172.21.15.27:0/3997741113 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab60004a30 con 0x7fab7006bb80 2024-01-15T18:04:51.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab77800640 1 -- 172.21.15.27:0/3997741113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab700c6d10 con 0x7fab7006bb80 2024-01-15T18:04:51.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.536+0000 7fab77800640 1 -- 172.21.15.27:0/3997741113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab700c4b50 con 0x7fab7006bb80 2024-01-15T18:04:51.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.540+0000 7fab77800640 1 -- 172.21.15.27:0/3997741113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab7004e850 con 0x7fab7006bb80 2024-01-15T18:04:51.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.540+0000 7fab6cc00640 1 -- 172.21.15.27:0/3997741113 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fab60004b90 con 0x7fab7006bb80 2024-01-15T18:04:51.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.540+0000 7fab6cc00640 1 --2- 172.21.15.27:0/3997741113 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fab480668f0 0x7fab48068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:51.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.540+0000 7fab6cc00640 1 -- 172.21.15.27:0/3997741113 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fab60083310 con 0x7fab7006bb80 2024-01-15T18:04:51.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.540+0000 7fab6fe00640 1 --2- 172.21.15.27:0/3997741113 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fab480668f0 0x7fab48068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:51.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.540+0000 7fab6fe00640 1 --2- 172.21.15.27:0/3997741113 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fab480668f0 0x7fab48068db0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fab580af000 tx=0x7fab580a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:51.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.544+0000 7fab6cc00640 1 -- 172.21.15.27:0/3997741113 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fab60050820 con 0x7fab7006bb80 2024-01-15T18:04:51.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.700+0000 7fab77800640 1 -- 172.21.15.27:0/3997741113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab700af1b0 con 0x7fab7006bb80 2024-01-15T18:04:51.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.704+0000 7fab6cc00640 1 -- 172.21.15.27:0/3997741113 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12754 (secure 0 0 0) 0x7fab60018020 con 0x7fab7006bb80 2024-01-15T18:04:51.710 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:04:51.710 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:04:46.449830+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:04:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.704+0000 7fab77800640 1 -- 172.21.15.27:0/3997741113 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fab480668f0 msgr2=0x7fab48068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.704+0000 7fab77800640 1 --2- 172.21.15.27:0/3997741113 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fab480668f0 0x7fab48068db0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fab580af000 tx=0x7fab580a7000 comp rx=0 tx=0).stop 2024-01-15T18:04:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.704+0000 7fab77800640 1 -- 172.21.15.27:0/3997741113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab7006bb80 msgr2=0x7fab7006ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.704+0000 7fab77800640 1 --2- 172.21.15.27:0/3997741113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab7006bb80 0x7fab7006ebf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fab6000c4a0 tx=0x7fab6000c960 comp rx=0 tx=0).stop 2024-01-15T18:04:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.704+0000 7fab77800640 1 -- 172.21.15.27:0/3997741113 shutdown_connections 2024-01-15T18:04:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.704+0000 7fab77800640 1 --2- 172.21.15.27:0/3997741113 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fab480668f0 0x7fab48068db0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.704+0000 7fab77800640 1 --2- 172.21.15.27:0/3997741113 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab700cddc0 0x7fab70151060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.704+0000 7fab77800640 1 --2- 172.21.15.27:0/3997741113 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fab700b2390 0x7fab700c5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.704+0000 7fab77800640 1 --2- 172.21.15.27:0/3997741113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab7006bb80 0x7fab7006ebf0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.704+0000 7fab77800640 1 -- 172.21.15.27:0/3997741113 >> 172.21.15.27:0/3997741113 conn(0x7fab700b2a10 msgr2=0x7fab700af6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.704+0000 7fab77800640 1 -- 172.21.15.27:0/3997741113 shutdown_connections 2024-01-15T18:04:51.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.704+0000 7fab77800640 1 -- 172.21.15.27:0/3997741113 wait complete. 2024-01-15T18:04:51.732 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:04:51.732 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:04:51.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.868+0000 7f0ca8800640 1 Processor -- start 2024-01-15T18:04:51.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.868+0000 7f0ca8800640 1 -- start start 2024-01-15T18:04:51.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.872+0000 7f0ca8800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0ca00cec60 0x7f0ca00b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:51.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.872+0000 7f0ca8800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca00d19a0 con 0x7f0ca00cf460 2024-01-15T18:04:51.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.872+0000 7f0ca8800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca00d1c80 con 0x7f0ca015b4b0 2024-01-15T18:04:51.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.872+0000 7f0ca8800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca00b07c0 con 0x7f0ca00cec60 2024-01-15T18:04:51.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.872+0000 7f0ca5e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0ca00cf460 0x7f0ca00cf840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52904/0 (socket says 172.21.15.27:52904) 2024-01-15T18:04:51.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.872+0000 7f0ca5e00640 1 -- 172.21.15.27:0/2248664347 learned_addr learned my addr 172.21.15.27:0/2248664347 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:04:51.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.872+0000 7f0ca6800640 1 --2- 172.21.15.27:0/2248664347 >> v2:172.21.15.93:3300/0 conn(0x7f0ca00cec60 0x7f0ca00b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:51.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.872+0000 7f0ca4a00640 1 -- 172.21.15.27:0/2248664347 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 576507192 0 0) 0x7f0ca00d19a0 con 0x7f0ca00cf460 2024-01-15T18:04:51.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.872+0000 7f0ca4a00640 1 -- 172.21.15.27:0/2248664347 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c80003530 con 0x7f0ca00cf460 2024-01-15T18:04:51.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.872+0000 7f0ca6800640 1 -- 172.21.15.27:0/2248664347 >> v1:172.21.15.93:6789/0 conn(0x7f0ca015b4b0 legacy=0x7f0ca00b1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:51.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.872+0000 7f0ca4a00640 1 -- 172.21.15.27:0/2248664347 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2378740091 0 0) 0x7f0ca00d1c80 con 0x7f0ca015b4b0 2024-01-15T18:04:51.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.872+0000 7f0ca6800640 1 -- 172.21.15.27:0/2248664347 >> v1:172.21.15.27:6789/0 conn(0x7f0ca00cf460 legacy=0x7f0ca00cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:51.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.872+0000 7f0ca6800640 1 -- 172.21.15.27:0/2248664347 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca00b0aa0 con 0x7f0ca00cec60 2024-01-15T18:04:51.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.876+0000 7f0ca6800640 1 --2- 172.21.15.27:0/2248664347 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ca00cec60 0x7f0ca00b1c90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0c98009d90 tx=0x7f0c9801c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2e5da58f5300122 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:51.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.876+0000 7f0ca4a00640 1 -- 172.21.15.27:0/2248664347 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c980a5a10 con 0x7f0ca00cec60 2024-01-15T18:04:51.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.876+0000 7f0ca4a00640 1 -- 172.21.15.27:0/2248664347 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c980a5b70 con 0x7f0ca00cec60 2024-01-15T18:04:51.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.876+0000 7f0ca4a00640 1 -- 172.21.15.27:0/2248664347 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c980ad440 con 0x7f0ca00cec60 2024-01-15T18:04:51.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.876+0000 7f0ca8800640 1 -- 172.21.15.27:0/2248664347 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ca00cec60 msgr2=0x7f0ca00b1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:51.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.876+0000 7f0ca8800640 1 --2- 172.21.15.27:0/2248664347 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ca00cec60 0x7f0ca00b1c90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0c98009d90 tx=0x7f0c9801c950 comp rx=0 tx=0).stop 2024-01-15T18:04:51.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.876+0000 7f0ca8800640 1 -- 172.21.15.27:0/2248664347 shutdown_connections 2024-01-15T18:04:51.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.876+0000 7f0ca8800640 1 --2- 172.21.15.27:0/2248664347 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ca00cec60 0x7f0ca00b1c90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:51.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.876+0000 7f0ca8800640 1 -- 172.21.15.27:0/2248664347 >> 172.21.15.27:0/2248664347 conn(0x7f0ca00d0e80 msgr2=0x7f0ca00d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:51.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.876+0000 7f0ca8800640 1 -- 172.21.15.27:0/2248664347 shutdown_connections 2024-01-15T18:04:51.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.876+0000 7f0ca8800640 1 -- 172.21.15.27:0/2248664347 wait complete. 2024-01-15T18:04:51.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.876+0000 7f0ca8800640 1 Processor -- start 2024-01-15T18:04:51.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.876+0000 7f0ca8800640 1 -- start start 2024-01-15T18:04:51.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.880+0000 7f0ca8800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ca00cec60 0x7f0ca00af630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:51.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.880+0000 7f0ca8800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ca00cf460 0x7f0ca0150300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:51.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.880+0000 7f0ca8800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ca015b4b0 0x7f0ca00adbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:51.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.880+0000 7f0ca8800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca00ae590 con 0x7f0ca00cf460 2024-01-15T18:04:51.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.880+0000 7f0ca8800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca00ae870 con 0x7f0ca00cec60 2024-01-15T18:04:51.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.880+0000 7f0ca8800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca00aeb50 con 0x7f0ca015b4b0 2024-01-15T18:04:51.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.880+0000 7f0ca5400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ca00cf460 0x7f0ca0150300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:51.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.880+0000 7f0ca5400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ca00cf460 0x7f0ca0150300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56728/0 (socket says 172.21.15.27:56728) 2024-01-15T18:04:51.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.880+0000 7f0ca5400640 1 -- 172.21.15.27:0/2339338059 learned_addr learned my addr 172.21.15.27:0/2339338059 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:51.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.880+0000 7f0ca6800640 1 --2- 172.21.15.27:0/2339338059 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ca015b4b0 0x7f0ca00adbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.880+0000 7f0ca5e00640 1 --2- 172.21.15.27:0/2339338059 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ca00cec60 0x7f0ca00af630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.880+0000 7f0ca5400640 1 -- 172.21.15.27:0/2339338059 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ca015b4b0 msgr2=0x7f0ca00adbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.880+0000 7f0ca5400640 1 --2- 172.21.15.27:0/2339338059 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ca015b4b0 0x7f0ca00adbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.880+0000 7f0ca5400640 1 -- 172.21.15.27:0/2339338059 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ca00cec60 msgr2=0x7f0ca00af630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.880+0000 7f0ca5400640 1 --2- 172.21.15.27:0/2339338059 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ca00cec60 0x7f0ca00af630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.880+0000 7f0ca5400640 1 -- 172.21.15.27:0/2339338059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c98009a70 con 0x7f0ca00cf460 2024-01-15T18:04:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.880+0000 7f0ca5400640 1 --2- 172.21.15.27:0/2339338059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ca00cf460 0x7f0ca0150300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0c8800c4d0 tx=0x7f0c8800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.880+0000 7f0c9e000640 1 -- 172.21.15.27:0/2339338059 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c88005480 con 0x7f0ca00cf460 2024-01-15T18:04:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.880+0000 7f0c9e000640 1 -- 172.21.15.27:0/2339338059 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c88003600 con 0x7f0ca00cf460 2024-01-15T18:04:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.880+0000 7f0ca8800640 1 -- 172.21.15.27:0/2339338059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca00aee30 con 0x7f0ca00cf460 2024-01-15T18:04:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.880+0000 7f0c9e000640 1 -- 172.21.15.27:0/2339338059 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c880039b0 con 0x7f0ca00cf460 2024-01-15T18:04:51.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.880+0000 7f0ca8800640 1 -- 172.21.15.27:0/2339338059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca0150840 con 0x7f0ca00cf460 2024-01-15T18:04:51.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.880+0000 7f0ca8800640 1 -- 172.21.15.27:0/2339338059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ca00afb70 con 0x7f0ca00cf460 2024-01-15T18:04:51.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.880+0000 7f0c9e000640 1 -- 172.21.15.27:0/2339338059 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f0c88003b10 con 0x7f0ca00cf460 2024-01-15T18:04:51.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.884+0000 7f0c9e000640 1 --2- 172.21.15.27:0/2339338059 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c780668f0 0x7f0c78068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:51.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.884+0000 7f0c9e000640 1 -- 172.21.15.27:0/2339338059 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0c88083760 con 0x7f0ca00cf460 2024-01-15T18:04:51.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.884+0000 7f0ca5e00640 1 --2- 172.21.15.27:0/2339338059 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c780668f0 0x7f0c78068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:51.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.884+0000 7f0ca5e00640 1 --2- 172.21.15.27:0/2339338059 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c780668f0 0x7f0c78068db0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0c900011f0 tx=0x7f0c90009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:51.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:51.888+0000 7f0c9e000640 1 -- 172.21.15.27:0/2339338059 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0c88050d10 con 0x7f0ca00cf460 2024-01-15T18:04:52.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.044+0000 7f0ca8800640 1 -- 172.21.15.27:0/2339338059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0ca004e850 con 0x7f0ca00cf460 2024-01-15T18:04:52.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.044+0000 7f0c9e000640 1 -- 172.21.15.27:0/2339338059 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0c88017020 con 0x7f0ca00cf460 2024-01-15T18:04:52.052 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:04:52.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.048+0000 7f0ca8800640 1 -- 172.21.15.27:0/2339338059 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c780668f0 msgr2=0x7f0c78068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:52.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.048+0000 7f0ca8800640 1 --2- 172.21.15.27:0/2339338059 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c780668f0 0x7f0c78068db0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0c900011f0 tx=0x7f0c90009040 comp rx=0 tx=0).stop 2024-01-15T18:04:52.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.048+0000 7f0ca8800640 1 -- 172.21.15.27:0/2339338059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ca00cf460 msgr2=0x7f0ca0150300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:52.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.048+0000 7f0ca8800640 1 --2- 172.21.15.27:0/2339338059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ca00cf460 0x7f0ca0150300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0c8800c4d0 tx=0x7f0c8800c990 comp rx=0 tx=0).stop 2024-01-15T18:04:52.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.048+0000 7f0ca8800640 1 -- 172.21.15.27:0/2339338059 shutdown_connections 2024-01-15T18:04:52.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.048+0000 7f0ca8800640 1 --2- 172.21.15.27:0/2339338059 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c780668f0 0x7f0c78068db0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:52.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.048+0000 7f0ca8800640 1 --2- 172.21.15.27:0/2339338059 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ca015b4b0 0x7f0ca00adbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:52.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.048+0000 7f0ca8800640 1 --2- 172.21.15.27:0/2339338059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ca00cf460 0x7f0ca0150300 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:52.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.048+0000 7f0ca8800640 1 --2- 172.21.15.27:0/2339338059 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ca00cec60 0x7f0ca00af630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:52.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.048+0000 7f0ca8800640 1 -- 172.21.15.27:0/2339338059 >> 172.21.15.27:0/2339338059 conn(0x7f0ca00d0e80 msgr2=0x7f0ca00cf840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:52.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.048+0000 7f0ca8800640 1 -- 172.21.15.27:0/2339338059 shutdown_connections 2024-01-15T18:04:52.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.048+0000 7f0ca8800640 1 -- 172.21.15.27:0/2339338059 wait complete. 2024-01-15T18:04:52.074 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:04:52.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.208+0000 7f9f4aa00640 1 Processor -- start 2024-01-15T18:04:52.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.208+0000 7f9f4aa00640 1 -- start start 2024-01-15T18:04:52.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.208+0000 7f9f4aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f4406ac00 0x7f9f440b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:52.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.208+0000 7f9f4aa00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9f440cdd30 0x7f9f440b4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:52.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.208+0000 7f9f4aa00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f440cc9e0 con 0x7f9f4415b2c0 2024-01-15T18:04:52.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.208+0000 7f9f4aa00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f440cd660 con 0x7f9f4406ac00 2024-01-15T18:04:52.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.208+0000 7f9f4aa00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f440cccc0 con 0x7f9f440cdd30 2024-01-15T18:04:52.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.208+0000 7f9f43400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f4406ac00 0x7f9f440b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:52.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.208+0000 7f9f43400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f4406ac00 0x7f9f440b3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56738/0 (socket says 172.21.15.27:56738) 2024-01-15T18:04:52.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.208+0000 7f9f43400640 1 -- 172.21.15.27:0/3029091391 learned_addr learned my addr 172.21.15.27:0/3029091391 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:52.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.208+0000 7f9f48a00640 1 --2- 172.21.15.27:0/3029091391 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9f440cdd30 0x7f9f440b4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:52.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.212+0000 7f9f43400640 1 -- 172.21.15.27:0/3029091391 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9f440cdd30 msgr2=0x7f9f440b4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:52.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.212+0000 7f9f43400640 1 --2- 172.21.15.27:0/3029091391 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9f440cdd30 0x7f9f440b4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:52.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.212+0000 7f9f43400640 1 -- 172.21.15.27:0/3029091391 >> v1:172.21.15.93:6789/0 conn(0x7f9f4415b2c0 legacy=0x7f9f4406a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:52.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.212+0000 7f9f43400640 1 -- 172.21.15.27:0/3029091391 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f440ccfa0 con 0x7f9f4406ac00 2024-01-15T18:04:52.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.212+0000 7f9f43400640 1 --2- 172.21.15.27:0/3029091391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f4406ac00 0x7f9f440b3ff0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c009870 tx=0x7f9f2c01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1697d03bb7713e2c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:52.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.212+0000 7f9f42a00640 1 -- 172.21.15.27:0/3029091391 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c0ab620 con 0x7f9f4406ac00 2024-01-15T18:04:52.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.212+0000 7f9f42a00640 1 -- 172.21.15.27:0/3029091391 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f2c0ab780 con 0x7f9f4406ac00 2024-01-15T18:04:52.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.212+0000 7f9f42a00640 1 -- 172.21.15.27:0/3029091391 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c0abb10 con 0x7f9f4406ac00 2024-01-15T18:04:52.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.212+0000 7f9f4aa00640 1 -- 172.21.15.27:0/3029091391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f4406ac00 msgr2=0x7f9f440b3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:52.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.212+0000 7f9f4aa00640 1 --2- 172.21.15.27:0/3029091391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f4406ac00 0x7f9f440b3ff0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c009870 tx=0x7f9f2c01c510 comp rx=0 tx=0).stop 2024-01-15T18:04:52.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.216+0000 7f9f4aa00640 1 -- 172.21.15.27:0/3029091391 shutdown_connections 2024-01-15T18:04:52.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.216+0000 7f9f4aa00640 1 --2- 172.21.15.27:0/3029091391 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9f440cdd30 0x7f9f440b4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:52.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.216+0000 7f9f4aa00640 1 --2- 172.21.15.27:0/3029091391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f4406ac00 0x7f9f440b3ff0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:52.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.216+0000 7f9f4aa00640 1 -- 172.21.15.27:0/3029091391 >> 172.21.15.27:0/3029091391 conn(0x7f9f440b1ae0 msgr2=0x7f9f440b1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:52.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.216+0000 7f9f4aa00640 1 -- 172.21.15.27:0/3029091391 shutdown_connections 2024-01-15T18:04:52.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.216+0000 7f9f4aa00640 1 -- 172.21.15.27:0/3029091391 wait complete. 2024-01-15T18:04:52.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.216+0000 7f9f4aa00640 1 Processor -- start 2024-01-15T18:04:52.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.216+0000 7f9f4aa00640 1 -- start start 2024-01-15T18:04:52.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.216+0000 7f9f4aa00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9f4406ac00 0x7f9f44156350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:52.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.216+0000 7f9f4aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f440cdd30 0x7f9f44156890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.216+0000 7f9f4aa00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f4415b2c0 0x7f9f4414e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.216+0000 7f9f4aa00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4415e660 con 0x7f9f440cdd30 2024-01-15T18:04:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.216+0000 7f9f4aa00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f44156dd0 con 0x7f9f4415b2c0 2024-01-15T18:04:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.216+0000 7f9f4aa00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f44157080 con 0x7f9f4406ac00 2024-01-15T18:04:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.216+0000 7f9f43400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f440cdd30 0x7f9f44156890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.216+0000 7f9f43e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9f4406ac00 0x7f9f44156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.216+0000 7f9f43400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f440cdd30 0x7f9f44156890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56750/0 (socket says 172.21.15.27:56750) 2024-01-15T18:04:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.216+0000 7f9f43400640 1 -- 172.21.15.27:0/403598960 learned_addr learned my addr 172.21.15.27:0/403598960 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.216+0000 7f9f48a00640 1 --2- 172.21.15.27:0/403598960 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f4415b2c0 0x7f9f4414e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.216+0000 7f9f43400640 1 -- 172.21.15.27:0/403598960 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9f4406ac00 msgr2=0x7f9f44156350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.216+0000 7f9f43400640 1 --2- 172.21.15.27:0/403598960 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9f4406ac00 0x7f9f44156350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.216+0000 7f9f43400640 1 -- 172.21.15.27:0/403598960 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f4415b2c0 msgr2=0x7f9f4414e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:52.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.216+0000 7f9f43400640 1 --2- 172.21.15.27:0/403598960 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f4415b2c0 0x7f9f4414e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:52.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.216+0000 7f9f43400640 1 -- 172.21.15.27:0/403598960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f2c0094d0 con 0x7f9f440cdd30 2024-01-15T18:04:52.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.220+0000 7f9f43400640 1 --2- 172.21.15.27:0/403598960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f440cdd30 0x7f9f44156890 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c01a140 tx=0x7f9f2c01ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:52.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.220+0000 7f9f3be00640 1 -- 172.21.15.27:0/403598960 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c0be3f0 con 0x7f9f440cdd30 2024-01-15T18:04:52.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.220+0000 7f9f3be00640 1 -- 172.21.15.27:0/403598960 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f2c0be550 con 0x7f9f440cdd30 2024-01-15T18:04:52.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.220+0000 7f9f3be00640 1 -- 172.21.15.27:0/403598960 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c0be900 con 0x7f9f440cdd30 2024-01-15T18:04:52.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.220+0000 7f9f4aa00640 1 -- 172.21.15.27:0/403598960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f44157360 con 0x7f9f440cdd30 2024-01-15T18:04:52.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.220+0000 7f9f4aa00640 1 -- 172.21.15.27:0/403598960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f44157920 con 0x7f9f440cdd30 2024-01-15T18:04:52.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.220+0000 7f9f4aa00640 1 -- 172.21.15.27:0/403598960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f44067a10 con 0x7f9f440cdd30 2024-01-15T18:04:52.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.220+0000 7f9f3be00640 1 -- 172.21.15.27:0/403598960 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f9f2c0b43f0 con 0x7f9f440cdd30 2024-01-15T18:04:52.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.220+0000 7f9f3be00640 1 --2- 172.21.15.27:0/403598960 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9f1c0668f0 0x7f9f1c068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:52.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.220+0000 7f9f3be00640 1 -- 172.21.15.27:0/403598960 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9f2c11e390 con 0x7f9f440cdd30 2024-01-15T18:04:52.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.220+0000 7f9f43e00640 1 --2- 172.21.15.27:0/403598960 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9f1c0668f0 0x7f9f1c068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:52.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.224+0000 7f9f43e00640 1 --2- 172.21.15.27:0/403598960 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9f1c0668f0 0x7f9f1c068db0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9f44150a60 tx=0x7f9f34009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:52.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.224+0000 7f9f3be00640 1 -- 172.21.15.27:0/403598960 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9f2c0eb930 con 0x7f9f440cdd30 2024-01-15T18:04:52.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.412+0000 7f9f4aa00640 1 -- 172.21.15.27:0/403598960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f441520b0 con 0x7f9f440cdd30 2024-01-15T18:04:52.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.416+0000 7f9f3be00640 1 -- 172.21.15.27:0/403598960 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1321 (secure 0 0 0) 0x7f9f441520b0 con 0x7f9f440cdd30 2024-01-15T18:04:52.422 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:04:52.422 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/393360880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":393360880},{"type":"v1","addr":"172.21.15.93:6836","nonce":393360880}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-01-15T18:04:52.423 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 2 2024-01-15T18:04:52.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.416+0000 7f9f4aa00640 1 -- 172.21.15.27:0/403598960 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9f1c0668f0 msgr2=0x7f9f1c068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:52.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.416+0000 7f9f4aa00640 1 --2- 172.21.15.27:0/403598960 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9f1c0668f0 0x7f9f1c068db0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9f44150a60 tx=0x7f9f34009040 comp rx=0 tx=0).stop 2024-01-15T18:04:52.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.416+0000 7f9f4aa00640 1 -- 172.21.15.27:0/403598960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f440cdd30 msgr2=0x7f9f44156890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:52.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.416+0000 7f9f4aa00640 1 --2- 172.21.15.27:0/403598960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f440cdd30 0x7f9f44156890 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c01a140 tx=0x7f9f2c01ad50 comp rx=0 tx=0).stop 2024-01-15T18:04:52.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.416+0000 7f9f4aa00640 1 -- 172.21.15.27:0/403598960 shutdown_connections 2024-01-15T18:04:52.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.416+0000 7f9f4aa00640 1 --2- 172.21.15.27:0/403598960 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9f1c0668f0 0x7f9f1c068db0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:52.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.416+0000 7f9f4aa00640 1 --2- 172.21.15.27:0/403598960 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f4415b2c0 0x7f9f4414e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:52.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.416+0000 7f9f4aa00640 1 --2- 172.21.15.27:0/403598960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f440cdd30 0x7f9f44156890 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:52.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.416+0000 7f9f4aa00640 1 --2- 172.21.15.27:0/403598960 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9f4406ac00 0x7f9f44156350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:52.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.416+0000 7f9f4aa00640 1 -- 172.21.15.27:0/403598960 >> 172.21.15.27:0/403598960 conn(0x7f9f440b1ae0 msgr2=0x7f9f441584c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:52.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.416+0000 7f9f4aa00640 1 -- 172.21.15.27:0/403598960 shutdown_connections 2024-01-15T18:04:52.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:52.416+0000 7f9f4aa00640 1 -- 172.21.15.27:0/403598960 wait complete. 2024-01-15T18:04:54.446 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:04:54.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.596+0000 7fc806400640 1 Processor -- start 2024-01-15T18:04:54.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.596+0000 7fc806400640 1 -- start start 2024-01-15T18:04:54.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.596+0000 7fc806400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fc80015b1b0 0x7fc8000b14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:54.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.596+0000 7fc806400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc8000ce1f0 0x7fc8000b2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:54.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.596+0000 7fc806400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc800072840 con 0x7fc80015b590 2024-01-15T18:04:54.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.596+0000 7fc806400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc800071c30 con 0x7fc8000ce1f0 2024-01-15T18:04:54.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.596+0000 7fc806400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc800071f10 con 0x7fc80015b1b0 2024-01-15T18:04:54.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.596+0000 7fc7ffe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc8000ce1f0 0x7fc8000b2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:54.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.596+0000 7fc7ffe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc8000ce1f0 0x7fc8000b2ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46408/0 (socket says 172.21.15.27:46408) 2024-01-15T18:04:54.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.596+0000 7fc7ffe00640 1 -- 172.21.15.27:0/3809211001 learned_addr learned my addr 172.21.15.27:0/3809211001 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:54.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.596+0000 7fc7ff400640 1 --2- 172.21.15.27:0/3809211001 >> v2:172.21.15.93:3300/0 conn(0x7fc80015b1b0 0x7fc8000b14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:54.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.596+0000 7fc7ffe00640 1 -- 172.21.15.27:0/3809211001 >> v2:172.21.15.93:3300/0 conn(0x7fc80015b1b0 msgr2=0x7fc8000b14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:54.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.596+0000 7fc7ffe00640 1 --2- 172.21.15.27:0/3809211001 >> v2:172.21.15.93:3300/0 conn(0x7fc80015b1b0 0x7fc8000b14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:54.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.596+0000 7fc7ffe00640 1 -- 172.21.15.27:0/3809211001 >> v1:172.21.15.93:6789/0 conn(0x7fc80015b590 legacy=0x7fc8000cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:54.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.596+0000 7fc7ffe00640 1 -- 172.21.15.27:0/3809211001 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8000721f0 con 0x7fc8000ce1f0 2024-01-15T18:04:54.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.600+0000 7fc7ffe00640 1 --2- 172.21.15.27:0/3809211001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8000ce1f0 0x7fc8000b2ef0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc80006ef80 tx=0x7fc7f801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc7bebc5257baa3e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:54.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.600+0000 7fc7fe000640 1 -- 172.21.15.27:0/3809211001 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f80a56e0 con 0x7fc8000ce1f0 2024-01-15T18:04:54.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.600+0000 7fc7fe000640 1 -- 172.21.15.27:0/3809211001 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f80a5840 con 0x7fc8000ce1f0 2024-01-15T18:04:54.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.600+0000 7fc7fe000640 1 -- 172.21.15.27:0/3809211001 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f80a5bd0 con 0x7fc8000ce1f0 2024-01-15T18:04:54.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.600+0000 7fc806400640 1 -- 172.21.15.27:0/3809211001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8000ce1f0 msgr2=0x7fc8000b2ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:54.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.600+0000 7fc806400640 1 --2- 172.21.15.27:0/3809211001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8000ce1f0 0x7fc8000b2ef0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc80006ef80 tx=0x7fc7f801c910 comp rx=0 tx=0).stop 2024-01-15T18:04:54.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.600+0000 7fc806400640 1 -- 172.21.15.27:0/3809211001 shutdown_connections 2024-01-15T18:04:54.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.600+0000 7fc806400640 1 --2- 172.21.15.27:0/3809211001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc8000ce1f0 0x7fc8000b2ef0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:54.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.600+0000 7fc806400640 1 --2- 172.21.15.27:0/3809211001 >> v2:172.21.15.93:3300/0 conn(0x7fc80015b1b0 0x7fc8000b14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:54.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.600+0000 7fc806400640 1 -- 172.21.15.27:0/3809211001 >> 172.21.15.27:0/3809211001 conn(0x7fc8000b47d0 msgr2=0x7fc8000b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:54.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.600+0000 7fc806400640 1 -- 172.21.15.27:0/3809211001 shutdown_connections 2024-01-15T18:04:54.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.600+0000 7fc806400640 1 -- 172.21.15.27:0/3809211001 wait complete. 2024-01-15T18:04:54.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.604+0000 7fc806400640 1 Processor -- start 2024-01-15T18:04:54.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.604+0000 7fc806400640 1 -- start start 2024-01-15T18:04:54.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.604+0000 7fc806400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc8000ce1f0 0x7fc8000ac070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:54.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.604+0000 7fc806400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc80015b1b0 0x7fc8001545c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:54.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.604+0000 7fc806400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc80015b590 0x7fc800154b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:54.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.604+0000 7fc806400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8000aed70 con 0x7fc80015b590 2024-01-15T18:04:54.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.604+0000 7fc806400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8000af050 con 0x7fc8000ce1f0 2024-01-15T18:04:54.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.604+0000 7fc806400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8000af330 con 0x7fc80015b1b0 2024-01-15T18:04:54.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.604+0000 7fc7ffe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc80015b590 0x7fc800154b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:54.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.604+0000 7fc7ffe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc80015b590 0x7fc800154b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46418/0 (socket says 172.21.15.27:46418) 2024-01-15T18:04:54.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.604+0000 7fc7ffe00640 1 -- 172.21.15.27:0/1400602099 learned_addr learned my addr 172.21.15.27:0/1400602099 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:54.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.604+0000 7fc7ff400640 1 --2- 172.21.15.27:0/1400602099 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc8000ce1f0 0x7fc8000ac070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:54.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.604+0000 7fc7fea00640 1 --2- 172.21.15.27:0/1400602099 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc80015b1b0 0x7fc8001545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:54.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.604+0000 7fc7ffe00640 1 -- 172.21.15.27:0/1400602099 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc80015b1b0 msgr2=0x7fc8001545c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:54.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.604+0000 7fc7ffe00640 1 --2- 172.21.15.27:0/1400602099 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc80015b1b0 0x7fc8001545c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:54.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.604+0000 7fc7ffe00640 1 -- 172.21.15.27:0/1400602099 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc8000ce1f0 msgr2=0x7fc8000ac070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:54.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.604+0000 7fc7ffe00640 1 --2- 172.21.15.27:0/1400602099 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc8000ce1f0 0x7fc8000ac070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:54.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.604+0000 7fc7ffe00640 1 -- 172.21.15.27:0/1400602099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f8009b30 con 0x7fc80015b590 2024-01-15T18:04:54.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.604+0000 7fc7ffe00640 1 --2- 172.21.15.27:0/1400602099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc80015b590 0x7fc800154b00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80af040 tx=0x7fc7f80196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:54.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.604+0000 7fc7f7400640 1 -- 172.21.15.27:0/1400602099 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f80b8670 con 0x7fc80015b590 2024-01-15T18:04:54.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.604+0000 7fc7f7400640 1 -- 172.21.15.27:0/1400602099 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f80b87d0 con 0x7fc80015b590 2024-01-15T18:04:54.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.604+0000 7fc7f7400640 1 -- 172.21.15.27:0/1400602099 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f80b8b60 con 0x7fc80015b590 2024-01-15T18:04:54.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.604+0000 7fc806400640 1 -- 172.21.15.27:0/1400602099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8000af610 con 0x7fc80015b590 2024-01-15T18:04:54.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.604+0000 7fc806400640 1 -- 172.21.15.27:0/1400602099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8000ae990 con 0x7fc80015b590 2024-01-15T18:04:54.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.608+0000 7fc806400640 1 -- 172.21.15.27:0/1400602099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8000aeb20 con 0x7fc80015b590 2024-01-15T18:04:54.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.608+0000 7fc7f7400640 1 -- 172.21.15.27:0/1400602099 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fc7f80b8cc0 con 0x7fc80015b590 2024-01-15T18:04:54.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.608+0000 7fc7f7400640 1 --2- 172.21.15.27:0/1400602099 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc7d80668f0 0x7fc7d8068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:54.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.608+0000 7fc7f7400640 1 -- 172.21.15.27:0/1400602099 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc7f811f4c0 con 0x7fc80015b590 2024-01-15T18:04:54.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.608+0000 7fc7ff400640 1 --2- 172.21.15.27:0/1400602099 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc7d80668f0 0x7fc7d8068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:54.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.608+0000 7fc7ff400640 1 --2- 172.21.15.27:0/1400602099 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc7d80668f0 0x7fc7d8068db0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc800157470 tx=0x7fc7f0009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:54.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.612+0000 7fc7f7400640 1 -- 172.21.15.27:0/1400602099 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc7f80ecb00 con 0x7fc80015b590 2024-01-15T18:04:54.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.804+0000 7fc806400640 1 -- 172.21.15.27:0/1400602099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8000aeeb0 con 0x7fc80015b590 2024-01-15T18:04:54.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.804+0000 7fc7f7400640 1 -- 172.21.15.27:0/1400602099 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3691 (secure 0 0 0) 0x7fc8000aeeb0 con 0x7fc80015b590 2024-01-15T18:04:54.810 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:04:54.810 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/393360880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":393360880},{"type":"v1","addr":"172.21.15.93:6836","nonce":393360880}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4229,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4063308188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4063308188},{"type":"v1","addr":"172.21.15.27:6835","nonce":4063308188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4305,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/2343128056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":2343128056},{"type":"v1","addr":"172.21.15.93:6837","nonce":2343128056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4308,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/2657553192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2657553192},{"type":"v1","addr":"172.21.15.27:6834","nonce":2657553192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-01-15T18:04:54.810 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 3 2024-01-15T18:04:54.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.808+0000 7fc806400640 1 -- 172.21.15.27:0/1400602099 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc7d80668f0 msgr2=0x7fc7d8068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:54.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.808+0000 7fc806400640 1 --2- 172.21.15.27:0/1400602099 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc7d80668f0 0x7fc7d8068db0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc800157470 tx=0x7fc7f0009190 comp rx=0 tx=0).stop 2024-01-15T18:04:54.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.808+0000 7fc806400640 1 -- 172.21.15.27:0/1400602099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc80015b590 msgr2=0x7fc800154b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:54.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.808+0000 7fc806400640 1 --2- 172.21.15.27:0/1400602099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc80015b590 0x7fc800154b00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80af040 tx=0x7fc7f80196b0 comp rx=0 tx=0).stop 2024-01-15T18:04:54.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.808+0000 7fc806400640 1 -- 172.21.15.27:0/1400602099 shutdown_connections 2024-01-15T18:04:54.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.808+0000 7fc806400640 1 --2- 172.21.15.27:0/1400602099 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc7d80668f0 0x7fc7d8068db0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:54.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.808+0000 7fc806400640 1 --2- 172.21.15.27:0/1400602099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc80015b590 0x7fc800154b00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:54.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.808+0000 7fc806400640 1 --2- 172.21.15.27:0/1400602099 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc80015b1b0 0x7fc8001545c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:54.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.808+0000 7fc806400640 1 --2- 172.21.15.27:0/1400602099 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc8000ce1f0 0x7fc8000ac070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:54.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.808+0000 7fc806400640 1 -- 172.21.15.27:0/1400602099 >> 172.21.15.27:0/1400602099 conn(0x7fc8000b47d0 msgr2=0x7fc800099340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:54.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.808+0000 7fc806400640 1 -- 172.21.15.27:0/1400602099 shutdown_connections 2024-01-15T18:04:54.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.808+0000 7fc806400640 1 -- 172.21.15.27:0/1400602099 wait complete. 2024-01-15T18:04:54.833 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:04:54.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.968+0000 7fd484200640 1 Processor -- start 2024-01-15T18:04:54.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.968+0000 7fd484200640 1 -- start start 2024-01-15T18:04:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.972+0000 7fd484200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fd47c160a20 0x7fd47c0b1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.972+0000 7fd484200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd47c0cd7d0 0x7fd47c0cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.972+0000 7fd484200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd47c0d1a70 con 0x7fd47c0ce190 2024-01-15T18:04:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.972+0000 7fd484200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd47c160e00 con 0x7fd47c0cd7d0 2024-01-15T18:04:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.972+0000 7fd484200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd47c0d1cc0 con 0x7fd47c160a20 2024-01-15T18:04:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.972+0000 7fd480e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd47c0cd7d0 0x7fd47c0cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.972+0000 7fd480e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd47c0cd7d0 0x7fd47c0cdc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46426/0 (socket says 172.21.15.27:46426) 2024-01-15T18:04:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.972+0000 7fd480e00640 1 -- 172.21.15.27:0/412633101 learned_addr learned my addr 172.21.15.27:0/412633101 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.972+0000 7fd481800640 1 --2- 172.21.15.27:0/412633101 >> v2:172.21.15.93:3300/0 conn(0x7fd47c160a20 0x7fd47c0b1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.972+0000 7fd480e00640 1 -- 172.21.15.27:0/412633101 >> v2:172.21.15.93:3300/0 conn(0x7fd47c160a20 msgr2=0x7fd47c0b1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.972+0000 7fd480e00640 1 --2- 172.21.15.27:0/412633101 >> v2:172.21.15.93:3300/0 conn(0x7fd47c160a20 0x7fd47c0b1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.972+0000 7fd480e00640 1 -- 172.21.15.27:0/412633101 >> v1:172.21.15.93:6789/0 conn(0x7fd47c0ce190 legacy=0x7fd47c0cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.972+0000 7fd480e00640 1 -- 172.21.15.27:0/412633101 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47c06c010 con 0x7fd47c0cd7d0 2024-01-15T18:04:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.972+0000 7fd481800640 1 --2- 172.21.15.27:0/412633101 >> v2:172.21.15.93:3300/0 conn(0x7fd47c160a20 0x7fd47c0b1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:04:54.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.976+0000 7fd480e00640 1 --2- 172.21.15.27:0/412633101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd47c0cd7d0 0x7fd47c0cdc50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd47c073de0 tx=0x7fd46401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca8282b952a1d4b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:54.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.976+0000 7fd47be00640 1 -- 172.21.15.27:0/412633101 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4640ab5f0 con 0x7fd47c0cd7d0 2024-01-15T18:04:54.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.976+0000 7fd47be00640 1 -- 172.21.15.27:0/412633101 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4640ab750 con 0x7fd47c0cd7d0 2024-01-15T18:04:54.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.976+0000 7fd47be00640 1 -- 172.21.15.27:0/412633101 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4640abae0 con 0x7fd47c0cd7d0 2024-01-15T18:04:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.976+0000 7fd484200640 1 -- 172.21.15.27:0/412633101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd47c0cd7d0 msgr2=0x7fd47c0cdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.976+0000 7fd484200640 1 --2- 172.21.15.27:0/412633101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd47c0cd7d0 0x7fd47c0cdc50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd47c073de0 tx=0x7fd46401c500 comp rx=0 tx=0).stop 2024-01-15T18:04:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.976+0000 7fd484200640 1 -- 172.21.15.27:0/412633101 shutdown_connections 2024-01-15T18:04:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.976+0000 7fd484200640 1 --2- 172.21.15.27:0/412633101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd47c0cd7d0 0x7fd47c0cdc50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:54.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.976+0000 7fd484200640 1 --2- 172.21.15.27:0/412633101 >> v2:172.21.15.93:3300/0 conn(0x7fd47c160a20 0x7fd47c0b1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:54.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.976+0000 7fd484200640 1 -- 172.21.15.27:0/412633101 >> 172.21.15.27:0/412633101 conn(0x7fd47c0d0f50 msgr2=0x7fd47c0d1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:54.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.976+0000 7fd484200640 1 -- 172.21.15.27:0/412633101 shutdown_connections 2024-01-15T18:04:54.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.976+0000 7fd484200640 1 -- 172.21.15.27:0/412633101 wait complete. 2024-01-15T18:04:54.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.976+0000 7fd484200640 1 Processor -- start 2024-01-15T18:04:54.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.976+0000 7fd484200640 1 -- start start 2024-01-15T18:04:54.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.980+0000 7fd484200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd47c0cd7d0 0x7fd47c15c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:54.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.980+0000 7fd484200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd47c0ce190 0x7fd47c1453b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:54.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.980+0000 7fd484200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd47c160a20 0x7fd47c0c7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:54.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.980+0000 7fd484200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c1458f0 con 0x7fd47c0ce190 2024-01-15T18:04:54.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.980+0000 7fd484200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c0c7870 con 0x7fd47c160a20 2024-01-15T18:04:54.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.980+0000 7fd484200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd47c15c8e0 con 0x7fd47c0cd7d0 2024-01-15T18:04:54.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.980+0000 7fd480e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd47c0ce190 0x7fd47c1453b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:54.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.980+0000 7fd480e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd47c0ce190 0x7fd47c1453b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46438/0 (socket says 172.21.15.27:46438) 2024-01-15T18:04:54.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.980+0000 7fd480e00640 1 -- 172.21.15.27:0/921254998 learned_addr learned my addr 172.21.15.27:0/921254998 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:54.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.980+0000 7fd482200640 1 --2- 172.21.15.27:0/921254998 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd47c160a20 0x7fd47c0c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:54.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.980+0000 7fd481800640 1 --2- 172.21.15.27:0/921254998 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd47c0cd7d0 0x7fd47c15c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:54.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.980+0000 7fd480e00640 1 -- 172.21.15.27:0/921254998 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd47c0cd7d0 msgr2=0x7fd47c15c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:54.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.980+0000 7fd480e00640 1 --2- 172.21.15.27:0/921254998 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd47c0cd7d0 0x7fd47c15c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:54.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.980+0000 7fd480e00640 1 -- 172.21.15.27:0/921254998 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd47c160a20 msgr2=0x7fd47c0c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:54.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.980+0000 7fd480e00640 1 --2- 172.21.15.27:0/921254998 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd47c160a20 0x7fd47c0c7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:54.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.980+0000 7fd480e00640 1 -- 172.21.15.27:0/921254998 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4640094d0 con 0x7fd47c0ce190 2024-01-15T18:04:54.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.980+0000 7fd480e00640 1 --2- 172.21.15.27:0/921254998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd47c0ce190 0x7fd47c1453b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd464019110 tx=0x7fd46401ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:54.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.980+0000 7fd479600640 1 -- 172.21.15.27:0/921254998 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4640be5f0 con 0x7fd47c0ce190 2024-01-15T18:04:54.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.980+0000 7fd479600640 1 -- 172.21.15.27:0/921254998 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4640be770 con 0x7fd47c0ce190 2024-01-15T18:04:54.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.980+0000 7fd484200640 1 -- 172.21.15.27:0/921254998 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd47c15cbc0 con 0x7fd47c0ce190 2024-01-15T18:04:54.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.980+0000 7fd479600640 1 -- 172.21.15.27:0/921254998 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4640beb20 con 0x7fd47c0ce190 2024-01-15T18:04:54.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.980+0000 7fd484200640 1 -- 172.21.15.27:0/921254998 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd47c15d180 con 0x7fd47c0ce190 2024-01-15T18:04:54.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.980+0000 7fd484200640 1 -- 172.21.15.27:0/921254998 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd47c04e850 con 0x7fd47c0ce190 2024-01-15T18:04:54.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.984+0000 7fd479600640 1 -- 172.21.15.27:0/921254998 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fd4640b43f0 con 0x7fd47c0ce190 2024-01-15T18:04:54.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.984+0000 7fd479600640 1 --2- 172.21.15.27:0/921254998 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd4540668f0 0x7fd454068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:54.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.984+0000 7fd479600640 1 -- 172.21.15.27:0/921254998 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd4640a8050 con 0x7fd47c0ce190 2024-01-15T18:04:54.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.984+0000 7fd481800640 1 --2- 172.21.15.27:0/921254998 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd4540668f0 0x7fd454068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:54.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.984+0000 7fd481800640 1 --2- 172.21.15.27:0/921254998 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd4540668f0 0x7fd454068db0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd46c006e90 tx=0x7fd46c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:54.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:54.988+0000 7fd479600640 1 -- 172.21.15.27:0/921254998 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd4640ebac0 con 0x7fd47c0ce190 2024-01-15T18:04:55.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.144+0000 7fd484200640 1 -- 172.21.15.27:0/921254998 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd47c0cdd80 con 0x7fd47c0ce190 2024-01-15T18:04:55.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.144+0000 7fd479600640 1 -- 172.21.15.27:0/921254998 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12754 (secure 0 0 0) 0x7fd4640ef0e0 con 0x7fd47c0ce190 2024-01-15T18:04:55.150 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:04:55.150 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:04:46.449830+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:04:55.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.148+0000 7fd484200640 1 -- 172.21.15.27:0/921254998 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd4540668f0 msgr2=0x7fd454068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:55.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.148+0000 7fd484200640 1 --2- 172.21.15.27:0/921254998 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd4540668f0 0x7fd454068db0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd46c006e90 tx=0x7fd46c008040 comp rx=0 tx=0).stop 2024-01-15T18:04:55.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.148+0000 7fd484200640 1 -- 172.21.15.27:0/921254998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd47c0ce190 msgr2=0x7fd47c1453b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:55.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.148+0000 7fd484200640 1 --2- 172.21.15.27:0/921254998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd47c0ce190 0x7fd47c1453b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd464019110 tx=0x7fd46401ad40 comp rx=0 tx=0).stop 2024-01-15T18:04:55.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.148+0000 7fd484200640 1 -- 172.21.15.27:0/921254998 shutdown_connections 2024-01-15T18:04:55.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.148+0000 7fd484200640 1 --2- 172.21.15.27:0/921254998 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd4540668f0 0x7fd454068db0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.148+0000 7fd484200640 1 --2- 172.21.15.27:0/921254998 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd47c160a20 0x7fd47c0c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.148+0000 7fd484200640 1 --2- 172.21.15.27:0/921254998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd47c0ce190 0x7fd47c1453b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.148+0000 7fd484200640 1 --2- 172.21.15.27:0/921254998 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd47c0cd7d0 0x7fd47c15c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.148+0000 7fd484200640 1 -- 172.21.15.27:0/921254998 >> 172.21.15.27:0/921254998 conn(0x7fd47c0d0f50 msgr2=0x7fd47c0d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:55.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.148+0000 7fd484200640 1 -- 172.21.15.27:0/921254998 shutdown_connections 2024-01-15T18:04:55.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.148+0000 7fd484200640 1 -- 172.21.15.27:0/921254998 wait complete. 2024-01-15T18:04:55.172 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:04:55.173 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:04:55.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.312+0000 7f0377a00640 1 Processor -- start 2024-01-15T18:04:55.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.312+0000 7f0377a00640 1 -- start start 2024-01-15T18:04:55.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.312+0000 7f0377a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f037006c430 0x7f0370070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:55.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.312+0000 7f0377a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03700ce760 0x7f03700b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:55.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.312+0000 7f0377a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03700ccfb0 con 0x7f03700b1db0 2024-01-15T18:04:55.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.312+0000 7f0377a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03700cd290 con 0x7f037006c430 2024-01-15T18:04:55.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.312+0000 7f0377a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03700cd570 con 0x7f03700ce760 2024-01-15T18:04:55.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.316+0000 7f0375000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f037006c430 0x7f0370070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:55.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.316+0000 7f0375000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f037006c430 0x7f0370070ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46440/0 (socket says 172.21.15.27:46440) 2024-01-15T18:04:55.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.316+0000 7f0375000640 1 -- 172.21.15.27:0/3420228060 learned_addr learned my addr 172.21.15.27:0/3420228060 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:55.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.316+0000 7f0375a00640 1 --2- 172.21.15.27:0/3420228060 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03700ce760 0x7f03700b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:55.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.316+0000 7f0375000640 1 -- 172.21.15.27:0/3420228060 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03700ce760 msgr2=0x7f03700b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:55.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.316+0000 7f0375000640 1 --2- 172.21.15.27:0/3420228060 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03700ce760 0x7f03700b2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.316+0000 7f0375000640 1 -- 172.21.15.27:0/3420228060 >> v1:172.21.15.93:6789/0 conn(0x7f03700b1db0 legacy=0x7f03700ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:55.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.316+0000 7f0375000640 1 -- 172.21.15.27:0/3420228060 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03700cd850 con 0x7f037006c430 2024-01-15T18:04:55.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.316+0000 7f0375000640 1 --2- 172.21.15.27:0/3420228060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f037006c430 0x7f0370070ba0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f036000aa40 tx=0x7f036001d830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e175321824982e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:55.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.316+0000 7f036f400640 1 -- 172.21.15.27:0/3420228060 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03600a68a0 con 0x7f037006c430 2024-01-15T18:04:55.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.316+0000 7f036f400640 1 -- 172.21.15.27:0/3420228060 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03600a6a00 con 0x7f037006c430 2024-01-15T18:04:55.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.316+0000 7f036f400640 1 -- 172.21.15.27:0/3420228060 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03600a6d90 con 0x7f037006c430 2024-01-15T18:04:55.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.320+0000 7f0377a00640 1 -- 172.21.15.27:0/3420228060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f037006c430 msgr2=0x7f0370070ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:55.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.320+0000 7f0377a00640 1 --2- 172.21.15.27:0/3420228060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f037006c430 0x7f0370070ba0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f036000aa40 tx=0x7f036001d830 comp rx=0 tx=0).stop 2024-01-15T18:04:55.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.320+0000 7f0377a00640 1 -- 172.21.15.27:0/3420228060 shutdown_connections 2024-01-15T18:04:55.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.320+0000 7f0377a00640 1 --2- 172.21.15.27:0/3420228060 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03700ce760 0x7f03700b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.320+0000 7f0377a00640 1 --2- 172.21.15.27:0/3420228060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f037006c430 0x7f0370070ba0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.320+0000 7f0377a00640 1 -- 172.21.15.27:0/3420228060 >> 172.21.15.27:0/3420228060 conn(0x7f03700cb700 msgr2=0x7f03700cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:55.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.320+0000 7f0377a00640 1 -- 172.21.15.27:0/3420228060 shutdown_connections 2024-01-15T18:04:55.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.320+0000 7f0377a00640 1 -- 172.21.15.27:0/3420228060 wait complete. 2024-01-15T18:04:55.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.320+0000 7f0377a00640 1 Processor -- start 2024-01-15T18:04:55.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.320+0000 7f0377a00640 1 -- start start 2024-01-15T18:04:55.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.320+0000 7f0377a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f037006c430 0x7f03701433e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:55.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.320+0000 7f0377a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03700b1db0 0x7f03700af810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:55.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.320+0000 7f0377a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03700ce760 0x7f03700afd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:55.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.320+0000 7f0377a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037013ef10 con 0x7f03700ce760 2024-01-15T18:04:55.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.320+0000 7f0377a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0370143920 con 0x7f03700b1db0 2024-01-15T18:04:55.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.320+0000 7f0377a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f037014a6c0 con 0x7f037006c430 2024-01-15T18:04:55.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.320+0000 7f0375a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03700ce760 0x7f03700afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:55.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.320+0000 7f0375a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03700ce760 0x7f03700afd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46444/0 (socket says 172.21.15.27:46444) 2024-01-15T18:04:55.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.320+0000 7f0375a00640 1 -- 172.21.15.27:0/2403902030 learned_addr learned my addr 172.21.15.27:0/2403902030 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:55.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.324+0000 7f0375000640 1 --2- 172.21.15.27:0/2403902030 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f037006c430 0x7f03701433e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:55.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.324+0000 7f036fe00640 1 --2- 172.21.15.27:0/2403902030 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03700b1db0 0x7f03700af810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:55.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.324+0000 7f0375a00640 1 -- 172.21.15.27:0/2403902030 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f037006c430 msgr2=0x7f03701433e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:55.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.324+0000 7f0375a00640 1 --2- 172.21.15.27:0/2403902030 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f037006c430 0x7f03701433e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.324+0000 7f0375a00640 1 -- 172.21.15.27:0/2403902030 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03700b1db0 msgr2=0x7f03700af810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:55.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.324+0000 7f0375a00640 1 --2- 172.21.15.27:0/2403902030 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03700b1db0 0x7f03700af810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.324+0000 7f0375a00640 1 -- 172.21.15.27:0/2403902030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036000a670 con 0x7f03700ce760 2024-01-15T18:04:55.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.324+0000 7f0375a00640 1 --2- 172.21.15.27:0/2403902030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03700ce760 0x7f03700afd50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f036800ef90 tx=0x7f036800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:55.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.324+0000 7f036cc00640 1 -- 172.21.15.27:0/2403902030 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0368009b20 con 0x7f03700ce760 2024-01-15T18:04:55.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.324+0000 7f036cc00640 1 -- 172.21.15.27:0/2403902030 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f036800f040 con 0x7f03700ce760 2024-01-15T18:04:55.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.324+0000 7f036cc00640 1 -- 172.21.15.27:0/2403902030 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03680146c0 con 0x7f03700ce760 2024-01-15T18:04:55.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.324+0000 7f0377a00640 1 -- 172.21.15.27:0/2403902030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03700712f0 con 0x7f03700ce760 2024-01-15T18:04:55.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.324+0000 7f0377a00640 1 -- 172.21.15.27:0/2403902030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0370142900 con 0x7f03700ce760 2024-01-15T18:04:55.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.324+0000 7f0377a00640 1 -- 172.21.15.27:0/2403902030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f037004e850 con 0x7f03700ce760 2024-01-15T18:04:55.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.324+0000 7f036cc00640 1 -- 172.21.15.27:0/2403902030 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f0368014820 con 0x7f03700ce760 2024-01-15T18:04:55.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.324+0000 7f036cc00640 1 --2- 172.21.15.27:0/2403902030 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03480668f0 0x7f0348068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:55.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.328+0000 7f036cc00640 1 -- 172.21.15.27:0/2403902030 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0368084cc0 con 0x7f03700ce760 2024-01-15T18:04:55.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.328+0000 7f0375000640 1 --2- 172.21.15.27:0/2403902030 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03480668f0 0x7f0348068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:55.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.328+0000 7f0375000640 1 --2- 172.21.15.27:0/2403902030 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03480668f0 0x7f0348068db0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f03600b0000 tx=0x7f0360008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:55.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.332+0000 7f036cc00640 1 -- 172.21.15.27:0/2403902030 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0368052300 con 0x7f03700ce760 2024-01-15T18:04:55.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.488+0000 7f0377a00640 1 -- 172.21.15.27:0/2403902030 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f037015ced0 con 0x7f03700ce760 2024-01-15T18:04:55.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.488+0000 7f036cc00640 1 -- 172.21.15.27:0/2403902030 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f03680180a0 con 0x7f03700ce760 2024-01-15T18:04:55.493 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:04:55.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.492+0000 7f0377a00640 1 -- 172.21.15.27:0/2403902030 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03480668f0 msgr2=0x7f0348068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:55.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.492+0000 7f0377a00640 1 --2- 172.21.15.27:0/2403902030 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03480668f0 0x7f0348068db0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f03600b0000 tx=0x7f0360008040 comp rx=0 tx=0).stop 2024-01-15T18:04:55.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.492+0000 7f0377a00640 1 -- 172.21.15.27:0/2403902030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03700ce760 msgr2=0x7f03700afd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:55.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.492+0000 7f0377a00640 1 --2- 172.21.15.27:0/2403902030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03700ce760 0x7f03700afd50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f036800ef90 tx=0x7f036800c490 comp rx=0 tx=0).stop 2024-01-15T18:04:55.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.492+0000 7f0377a00640 1 -- 172.21.15.27:0/2403902030 shutdown_connections 2024-01-15T18:04:55.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.492+0000 7f0377a00640 1 --2- 172.21.15.27:0/2403902030 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03480668f0 0x7f0348068db0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.492+0000 7f0377a00640 1 --2- 172.21.15.27:0/2403902030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03700ce760 0x7f03700afd50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.492+0000 7f0377a00640 1 --2- 172.21.15.27:0/2403902030 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03700b1db0 0x7f03700af810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.492+0000 7f0377a00640 1 --2- 172.21.15.27:0/2403902030 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f037006c430 0x7f03701433e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.492+0000 7f0377a00640 1 -- 172.21.15.27:0/2403902030 >> 172.21.15.27:0/2403902030 conn(0x7f03700cb700 msgr2=0x7f03701505e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:55.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.492+0000 7f0377a00640 1 -- 172.21.15.27:0/2403902030 shutdown_connections 2024-01-15T18:04:55.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.492+0000 7f0377a00640 1 -- 172.21.15.27:0/2403902030 wait complete. 2024-01-15T18:04:55.514 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-15T18:04:55.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.652+0000 7fe544600640 1 Processor -- start 2024-01-15T18:04:55.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.652+0000 7fe544600640 1 -- start start 2024-01-15T18:04:55.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.652+0000 7fe544600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe53c0b0130 0x7fe53c15f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:55.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.652+0000 7fe544600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53c0d1ec0 con 0x7fe53c0cf080 2024-01-15T18:04:55.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.652+0000 7fe544600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53c0ae160 con 0x7fe53c0b24a0 2024-01-15T18:04:55.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.652+0000 7fe544600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c0cfb70 con 0x7fe53c0b0130 2024-01-15T18:04:55.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.652+0000 7fe541c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe53c0cf080 0x7fe53c0cf460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53022/0 (socket says 172.21.15.27:53022) 2024-01-15T18:04:55.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.652+0000 7fe541c00640 1 -- 172.21.15.27:0/3632754975 learned_addr learned my addr 172.21.15.27:0/3632754975 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:04:55.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.652+0000 7fe541200640 1 --2- 172.21.15.27:0/3632754975 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe53c0b0130 0x7fe53c15f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:55.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.652+0000 7fe53be00640 1 -- 172.21.15.27:0/3632754975 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298525374 0 0) 0x7fe53c0d1ec0 con 0x7fe53c0cf080 2024-01-15T18:04:55.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.652+0000 7fe53be00640 1 -- 172.21.15.27:0/3632754975 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe51c003530 con 0x7fe53c0cf080 2024-01-15T18:04:55.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.652+0000 7fe53be00640 1 -- 172.21.15.27:0/3632754975 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645393093 0 0) 0x7fe53c0ae160 con 0x7fe53c0b24a0 2024-01-15T18:04:55.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.652+0000 7fe53be00640 1 -- 172.21.15.27:0/3632754975 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe53c0d1ec0 con 0x7fe53c0b24a0 2024-01-15T18:04:55.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.652+0000 7fe541200640 1 -- 172.21.15.27:0/3632754975 >> v1:172.21.15.93:6789/0 conn(0x7fe53c0b24a0 legacy=0x7fe53c15f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:55.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.652+0000 7fe541200640 1 -- 172.21.15.27:0/3632754975 >> v1:172.21.15.27:6789/0 conn(0x7fe53c0cf080 legacy=0x7fe53c0cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:55.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.652+0000 7fe541200640 1 -- 172.21.15.27:0/3632754975 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53c0d1be0 con 0x7fe53c0b0130 2024-01-15T18:04:55.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.656+0000 7fe541200640 1 --2- 172.21.15.27:0/3632754975 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe53c0b0130 0x7fe53c15f300 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe5240097f0 tx=0x7fe52401c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d746b7fb6ab09df server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:55.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.656+0000 7fe53be00640 1 -- 172.21.15.27:0/3632754975 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5240aa610 con 0x7fe53c0b0130 2024-01-15T18:04:55.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.656+0000 7fe53be00640 1 -- 172.21.15.27:0/3632754975 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5240a6040 con 0x7fe53c0b0130 2024-01-15T18:04:55.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.656+0000 7fe53be00640 1 -- 172.21.15.27:0/3632754975 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5240aa990 con 0x7fe53c0b0130 2024-01-15T18:04:55.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.656+0000 7fe544600640 1 -- 172.21.15.27:0/3632754975 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe53c0b0130 msgr2=0x7fe53c15f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:55.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.656+0000 7fe544600640 1 --2- 172.21.15.27:0/3632754975 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe53c0b0130 0x7fe53c15f300 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe5240097f0 tx=0x7fe52401c490 comp rx=0 tx=0).stop 2024-01-15T18:04:55.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.656+0000 7fe544600640 1 -- 172.21.15.27:0/3632754975 shutdown_connections 2024-01-15T18:04:55.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.656+0000 7fe544600640 1 --2- 172.21.15.27:0/3632754975 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe53c0b0130 0x7fe53c15f300 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.656+0000 7fe544600640 1 -- 172.21.15.27:0/3632754975 >> 172.21.15.27:0/3632754975 conn(0x7fe53c0d0f20 msgr2=0x7fe53c0d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:55.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.656+0000 7fe544600640 1 -- 172.21.15.27:0/3632754975 shutdown_connections 2024-01-15T18:04:55.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.656+0000 7fe544600640 1 -- 172.21.15.27:0/3632754975 wait complete. 2024-01-15T18:04:55.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.660+0000 7fe544600640 1 Processor -- start 2024-01-15T18:04:55.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.660+0000 7fe544600640 1 -- start start 2024-01-15T18:04:55.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.660+0000 7fe544600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe53c0b0130 0x7fe53c1599b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:55.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.660+0000 7fe544600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe53c0b24a0 0x7fe53c06b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:55.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.660+0000 7fe544600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe53c0cf080 0x7fe53c06c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:55.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.660+0000 7fe544600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c06bd50 con 0x7fe53c0cf080 2024-01-15T18:04:55.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.660+0000 7fe544600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c15a050 con 0x7fe53c0b0130 2024-01-15T18:04:55.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.660+0000 7fe544600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c06ce50 con 0x7fe53c0b24a0 2024-01-15T18:04:55.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.660+0000 7fe542600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe53c0cf080 0x7fe53c06c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:55.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.660+0000 7fe542600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe53c0cf080 0x7fe53c06c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46450/0 (socket says 172.21.15.27:46450) 2024-01-15T18:04:55.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.660+0000 7fe542600640 1 -- 172.21.15.27:0/2734221314 learned_addr learned my addr 172.21.15.27:0/2734221314 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:55.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.660+0000 7fe541200640 1 --2- 172.21.15.27:0/2734221314 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe53c0b24a0 0x7fe53c06b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:55.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.660+0000 7fe541c00640 1 --2- 172.21.15.27:0/2734221314 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe53c0b0130 0x7fe53c1599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:55.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.660+0000 7fe542600640 1 -- 172.21.15.27:0/2734221314 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe53c0b24a0 msgr2=0x7fe53c06b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:55.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.660+0000 7fe542600640 1 --2- 172.21.15.27:0/2734221314 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe53c0b24a0 0x7fe53c06b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.660+0000 7fe542600640 1 -- 172.21.15.27:0/2734221314 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe53c0b0130 msgr2=0x7fe53c1599b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:55.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.660+0000 7fe542600640 1 --2- 172.21.15.27:0/2734221314 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe53c0b0130 0x7fe53c1599b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.660+0000 7fe542600640 1 -- 172.21.15.27:0/2734221314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe524009420 con 0x7fe53c0cf080 2024-01-15T18:04:55.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.660+0000 7fe542600640 1 --2- 172.21.15.27:0/2734221314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe53c0cf080 0x7fe53c06c500 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe53400c790 tx=0x7fe53400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:55.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.660+0000 7fe539600640 1 -- 172.21.15.27:0/2734221314 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53400cea0 con 0x7fe53c0cf080 2024-01-15T18:04:55.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.660+0000 7fe539600640 1 -- 172.21.15.27:0/2734221314 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe534003b80 con 0x7fe53c0cf080 2024-01-15T18:04:55.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.660+0000 7fe544600640 1 -- 172.21.15.27:0/2734221314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe53c06d130 con 0x7fe53c0cf080 2024-01-15T18:04:55.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.660+0000 7fe539600640 1 -- 172.21.15.27:0/2734221314 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53400a850 con 0x7fe53c0cf080 2024-01-15T18:04:55.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.660+0000 7fe544600640 1 -- 172.21.15.27:0/2734221314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe53c06ca40 con 0x7fe53c0cf080 2024-01-15T18:04:55.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.664+0000 7fe544600640 1 -- 172.21.15.27:0/2734221314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe53c155a00 con 0x7fe53c0cf080 2024-01-15T18:04:55.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.664+0000 7fe539600640 1 -- 172.21.15.27:0/2734221314 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fe53400a9b0 con 0x7fe53c0cf080 2024-01-15T18:04:55.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.664+0000 7fe539600640 1 --2- 172.21.15.27:0/2734221314 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe5140668f0 0x7fe514068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:55.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.664+0000 7fe539600640 1 -- 172.21.15.27:0/2734221314 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe534083e20 con 0x7fe53c0cf080 2024-01-15T18:04:55.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.664+0000 7fe541c00640 1 --2- 172.21.15.27:0/2734221314 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe5140668f0 0x7fe514068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:55.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.664+0000 7fe541c00640 1 --2- 172.21.15.27:0/2734221314 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe5140668f0 0x7fe514068db0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe52c0011f0 tx=0x7fe52c007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:55.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.668+0000 7fe539600640 1 -- 172.21.15.27:0/2734221314 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe53c155a00 con 0x7fe53c0cf080 2024-01-15T18:04:55.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.828+0000 7fe544600640 1 -- 172.21.15.27:0/2734221314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe53c155a00 con 0x7fe53c0cf080 2024-01-15T18:04:55.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.828+0000 7fe539600640 1 -- 172.21.15.27:0/2734221314 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe53c155a00 con 0x7fe53c0cf080 2024-01-15T18:04:55.830 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:04:55.831 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:04:55.831 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:04:55.831 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-01-15T18:04:55.831 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-01-15T18:04:55.831 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:04:55.831 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:04:55.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.828+0000 7fe544600640 1 -- 172.21.15.27:0/2734221314 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe5140668f0 msgr2=0x7fe514068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:55.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.828+0000 7fe544600640 1 --2- 172.21.15.27:0/2734221314 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe5140668f0 0x7fe514068db0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe52c0011f0 tx=0x7fe52c007d40 comp rx=0 tx=0).stop 2024-01-15T18:04:55.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.828+0000 7fe544600640 1 -- 172.21.15.27:0/2734221314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe53c0cf080 msgr2=0x7fe53c06c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:55.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.828+0000 7fe544600640 1 --2- 172.21.15.27:0/2734221314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe53c0cf080 0x7fe53c06c500 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe53400c790 tx=0x7fe53400cc50 comp rx=0 tx=0).stop 2024-01-15T18:04:55.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.828+0000 7fe544600640 1 -- 172.21.15.27:0/2734221314 shutdown_connections 2024-01-15T18:04:55.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.828+0000 7fe544600640 1 --2- 172.21.15.27:0/2734221314 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe5140668f0 0x7fe514068db0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.828+0000 7fe544600640 1 --2- 172.21.15.27:0/2734221314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe53c0cf080 0x7fe53c06c500 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.828+0000 7fe544600640 1 --2- 172.21.15.27:0/2734221314 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe53c0b24a0 0x7fe53c06b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.828+0000 7fe544600640 1 --2- 172.21.15.27:0/2734221314 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe53c0b0130 0x7fe53c1599b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.828+0000 7fe544600640 1 -- 172.21.15.27:0/2734221314 >> 172.21.15.27:0/2734221314 conn(0x7fe53c0d0f20 msgr2=0x7fe53c0ca890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:55.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.828+0000 7fe544600640 1 -- 172.21.15.27:0/2734221314 shutdown_connections 2024-01-15T18:04:55.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.828+0000 7fe544600640 1 -- 172.21.15.27:0/2734221314 wait complete. 2024-01-15T18:04:55.852 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-15T18:04:55.852 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-15T18:04:55.990 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:04:55.993 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:04:55.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.992+0000 7fb0ca600640 1 Processor -- start 2024-01-15T18:04:55.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.992+0000 7fb0ca600640 1 -- start start 2024-01-15T18:04:56.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.996+0000 7fb0ca600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb0c406a720 0x7fb0c406e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:56.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.996+0000 7fb0ca600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fb0c415e190 0x7fb0c4150780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:56.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.996+0000 7fb0ca600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c406e860 con 0x7fb0c406ab00 2024-01-15T18:04:56.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.996+0000 7fb0ca600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c415e850 con 0x7fb0c406a720 2024-01-15T18:04:56.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.996+0000 7fb0ca600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c406b390 con 0x7fb0c415e190 2024-01-15T18:04:56.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.996+0000 7fb0c3400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb0c406a720 0x7fb0c406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:56.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.996+0000 7fb0c3400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb0c406a720 0x7fb0c406e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46452/0 (socket says 172.21.15.27:46452) 2024-01-15T18:04:56.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.996+0000 7fb0c3400640 1 -- 172.21.15.27:0/3223737818 learned_addr learned my addr 172.21.15.27:0/3223737818 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:56.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.996+0000 7fb0c3e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fb0c415e190 0x7fb0c4150780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:56.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.996+0000 7fb0c3400640 1 -- 172.21.15.27:0/3223737818 >> v2:172.21.15.93:3300/0 conn(0x7fb0c415e190 msgr2=0x7fb0c4150780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:56.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.996+0000 7fb0c3400640 1 --2- 172.21.15.27:0/3223737818 >> v2:172.21.15.93:3300/0 conn(0x7fb0c415e190 0x7fb0c4150780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:56.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.996+0000 7fb0c3400640 1 -- 172.21.15.27:0/3223737818 >> v1:172.21.15.93:6789/0 conn(0x7fb0c406ab00 legacy=0x7fb0c415da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:56.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:55.996+0000 7fb0c3400640 1 -- 172.21.15.27:0/3223737818 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c406b670 con 0x7fb0c406a720 2024-01-15T18:04:56.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.000+0000 7fb0c3400640 1 --2- 172.21.15.27:0/3223737818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c406a720 0x7fb0c406e320 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb0c406d6d0 tx=0x7fb0b401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8137f3a07bd9ddfc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:56.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.000+0000 7fb0c2000640 1 -- 172.21.15.27:0/3223737818 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b40aa660 con 0x7fb0c406a720 2024-01-15T18:04:56.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.000+0000 7fb0c2000640 1 -- 172.21.15.27:0/3223737818 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b40aa7c0 con 0x7fb0c406a720 2024-01-15T18:04:56.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.000+0000 7fb0c2000640 1 -- 172.21.15.27:0/3223737818 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b40aab50 con 0x7fb0c406a720 2024-01-15T18:04:56.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.000+0000 7fb0ca600640 1 -- 172.21.15.27:0/3223737818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c406a720 msgr2=0x7fb0c406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:56.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.000+0000 7fb0ca600640 1 --2- 172.21.15.27:0/3223737818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c406a720 0x7fb0c406e320 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb0c406d6d0 tx=0x7fb0b401c500 comp rx=0 tx=0).stop 2024-01-15T18:04:56.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.000+0000 7fb0ca600640 1 -- 172.21.15.27:0/3223737818 shutdown_connections 2024-01-15T18:04:56.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.000+0000 7fb0ca600640 1 --2- 172.21.15.27:0/3223737818 >> v2:172.21.15.93:3300/0 conn(0x7fb0c415e190 0x7fb0c4150780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:56.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.000+0000 7fb0ca600640 1 --2- 172.21.15.27:0/3223737818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c406a720 0x7fb0c406e320 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:56.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.000+0000 7fb0ca600640 1 -- 172.21.15.27:0/3223737818 >> 172.21.15.27:0/3223737818 conn(0x7fb0c40b1620 msgr2=0x7fb0c40b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:56.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.000+0000 7fb0ca600640 1 -- 172.21.15.27:0/3223737818 shutdown_connections 2024-01-15T18:04:56.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.000+0000 7fb0ca600640 1 -- 172.21.15.27:0/3223737818 wait complete. 2024-01-15T18:04:56.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.004+0000 7fb0ca600640 1 Processor -- start 2024-01-15T18:04:56.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.004+0000 7fb0ca600640 1 -- start start 2024-01-15T18:04:56.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.004+0000 7fb0ca600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c406a720 0x7fb0c4149200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:56.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.004+0000 7fb0ca600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0c406ab00 0x7fb0c415c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:56.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.004+0000 7fb0ca600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0c415e190 0x7fb0c415c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:56.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.004+0000 7fb0ca600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c4149740 con 0x7fb0c406a720 2024-01-15T18:04:56.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.004+0000 7fb0ca600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c414a2c0 con 0x7fb0c406ab00 2024-01-15T18:04:56.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.004+0000 7fb0ca600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c4149990 con 0x7fb0c415e190 2024-01-15T18:04:56.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.004+0000 7fb0c3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c406a720 0x7fb0c4149200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:56.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.004+0000 7fb0c3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c406a720 0x7fb0c4149200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46456/0 (socket says 172.21.15.27:46456) 2024-01-15T18:04:56.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.004+0000 7fb0c3400640 1 -- 172.21.15.27:0/233155434 learned_addr learned my addr 172.21.15.27:0/233155434 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:56.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.008+0000 7fb0c2a00640 1 --2- 172.21.15.27:0/233155434 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0c406ab00 0x7fb0c415c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:56.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.008+0000 7fb0c3e00640 1 --2- 172.21.15.27:0/233155434 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0c415e190 0x7fb0c415c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:56.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.008+0000 7fb0c3400640 1 -- 172.21.15.27:0/233155434 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0c415e190 msgr2=0x7fb0c415c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:56.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.008+0000 7fb0c3400640 1 --2- 172.21.15.27:0/233155434 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0c415e190 0x7fb0c415c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:56.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.008+0000 7fb0c3400640 1 -- 172.21.15.27:0/233155434 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0c406ab00 msgr2=0x7fb0c415c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:56.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.008+0000 7fb0c3400640 1 --2- 172.21.15.27:0/233155434 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0c406ab00 0x7fb0c415c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:56.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.008+0000 7fb0c3400640 1 -- 172.21.15.27:0/233155434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b40094d0 con 0x7fb0c406a720 2024-01-15T18:04:56.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.008+0000 7fb0c3400640 1 --2- 172.21.15.27:0/233155434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c406a720 0x7fb0c4149200 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb0b40ac040 tx=0x7fb0b401ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.008+0000 7fb0bb400640 1 -- 172.21.15.27:0/233155434 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b40b3800 con 0x7fb0c406a720 2024-01-15T18:04:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.008+0000 7fb0bb400640 1 -- 172.21.15.27:0/233155434 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b40b3980 con 0x7fb0c406a720 2024-01-15T18:04:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.008+0000 7fb0bb400640 1 -- 172.21.15.27:0/233155434 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b40b3d30 con 0x7fb0c406a720 2024-01-15T18:04:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.008+0000 7fb0ca600640 1 -- 172.21.15.27:0/233155434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c4149c70 con 0x7fb0c406a720 2024-01-15T18:04:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.008+0000 7fb0ca600640 1 -- 172.21.15.27:0/233155434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c414d120 con 0x7fb0c406a720 2024-01-15T18:04:56.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.008+0000 7fb0ca600640 1 -- 172.21.15.27:0/233155434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c406ea00 con 0x7fb0c406a720 2024-01-15T18:04:56.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.008+0000 7fb0bb400640 1 -- 172.21.15.27:0/233155434 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fb0b40b2380 con 0x7fb0c406a720 2024-01-15T18:04:56.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.012+0000 7fb0bb400640 1 --2- 172.21.15.27:0/233155434 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb09c0668f0 0x7fb09c068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:56.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.012+0000 7fb0bb400640 1 -- 172.21.15.27:0/233155434 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb0b40a6040 con 0x7fb0c406a720 2024-01-15T18:04:56.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.012+0000 7fb0c2a00640 1 --2- 172.21.15.27:0/233155434 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb09c0668f0 0x7fb09c068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:56.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.012+0000 7fb0c2a00640 1 --2- 172.21.15.27:0/233155434 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb09c0668f0 0x7fb09c068db0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb0c406d470 tx=0x7fb0ac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:56.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.012+0000 7fb0bb400640 1 -- 172.21.15.27:0/233155434 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb0b40ebbf0 con 0x7fb0c406a720 2024-01-15T18:04:56.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.172+0000 7fb0ca600640 1 -- 172.21.15.27:0/233155434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fb0c4149ad0 con 0x7fb0c406a720 2024-01-15T18:04:56.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.800+0000 7fb0bb400640 1 -- 172.21.15.27:0/233155434 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb0c4149ad0 con 0x7fb0c406a720 2024-01-15T18:04:56.805 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-01-15T18:04:56.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.804+0000 7fb0ca600640 1 -- 172.21.15.27:0/233155434 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb09c0668f0 msgr2=0x7fb09c068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:56.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.804+0000 7fb0ca600640 1 --2- 172.21.15.27:0/233155434 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb09c0668f0 0x7fb09c068db0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb0c406d470 tx=0x7fb0ac009040 comp rx=0 tx=0).stop 2024-01-15T18:04:56.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.804+0000 7fb0ca600640 1 -- 172.21.15.27:0/233155434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c406a720 msgr2=0x7fb0c4149200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:56.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.804+0000 7fb0ca600640 1 --2- 172.21.15.27:0/233155434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c406a720 0x7fb0c4149200 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb0b40ac040 tx=0x7fb0b401ad40 comp rx=0 tx=0).stop 2024-01-15T18:04:56.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.804+0000 7fb0ca600640 1 -- 172.21.15.27:0/233155434 shutdown_connections 2024-01-15T18:04:56.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.804+0000 7fb0ca600640 1 --2- 172.21.15.27:0/233155434 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb09c0668f0 0x7fb09c068db0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:56.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.804+0000 7fb0ca600640 1 --2- 172.21.15.27:0/233155434 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0c415e190 0x7fb0c415c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:56.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.804+0000 7fb0ca600640 1 --2- 172.21.15.27:0/233155434 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0c406ab00 0x7fb0c415c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:56.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.804+0000 7fb0ca600640 1 --2- 172.21.15.27:0/233155434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0c406a720 0x7fb0c4149200 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:56.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.804+0000 7fb0ca600640 1 -- 172.21.15.27:0/233155434 >> 172.21.15.27:0/233155434 conn(0x7fb0c40b1620 msgr2=0x7fb0c40ced60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:56.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.804+0000 7fb0ca600640 1 -- 172.21.15.27:0/233155434 shutdown_connections 2024-01-15T18:04:56.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.804+0000 7fb0ca600640 1 -- 172.21.15.27:0/233155434 wait complete. 2024-01-15T18:04:56.839 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-15T18:04:56.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.988+0000 7fe904600640 1 Processor -- start 2024-01-15T18:04:56.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.988+0000 7fe904600640 1 -- start start 2024-01-15T18:04:56.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.988+0000 7fe904600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8fc15b100 0x7fe8fc070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:56.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.988+0000 7fe904600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe8fc071420 0x7fe8fc15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:56.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.988+0000 7fe904600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8fc15a610 con 0x7fe8fc15ad20 2024-01-15T18:04:56.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.988+0000 7fe904600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8fc0718b0 con 0x7fe8fc071420 2024-01-15T18:04:56.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.988+0000 7fe904600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8fc15e3c0 con 0x7fe8fc15b100 2024-01-15T18:04:56.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.988+0000 7fe902600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe8fc071420 0x7fe8fc15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:56.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.988+0000 7fe902600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe8fc071420 0x7fe8fc15bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46462/0 (socket says 172.21.15.27:46462) 2024-01-15T18:04:56.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.988+0000 7fe902600640 1 -- 172.21.15.27:0/353678041 learned_addr learned my addr 172.21.15.27:0/353678041 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:56.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.988+0000 7fe901200640 1 --2- 172.21.15.27:0/353678041 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8fc15b100 0x7fe8fc070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:56.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.988+0000 7fe902600640 1 -- 172.21.15.27:0/353678041 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8fc15b100 msgr2=0x7fe8fc070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:56.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.988+0000 7fe902600640 1 --2- 172.21.15.27:0/353678041 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8fc15b100 0x7fe8fc070ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:56.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.988+0000 7fe902600640 1 -- 172.21.15.27:0/353678041 >> v1:172.21.15.93:6789/0 conn(0x7fe8fc15ad20 legacy=0x7fe8fc15d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:56.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.988+0000 7fe902600640 1 -- 172.21.15.27:0/353678041 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8fc15e6a0 con 0x7fe8fc071420 2024-01-15T18:04:56.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.992+0000 7fe902600640 1 --2- 172.21.15.27:0/353678041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8fc071420 0x7fe8fc15bee0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe8f400d150 tx=0x7fe8f401ebe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c43fdd6f6add88 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:56.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.992+0000 7fe8fbe00640 1 -- 172.21.15.27:0/353678041 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f400b2d0 con 0x7fe8fc071420 2024-01-15T18:04:56.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.992+0000 7fe8fbe00640 1 -- 172.21.15.27:0/353678041 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f400b430 con 0x7fe8fc071420 2024-01-15T18:04:56.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.992+0000 7fe8fbe00640 1 -- 172.21.15.27:0/353678041 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f40a7be0 con 0x7fe8fc071420 2024-01-15T18:04:56.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.992+0000 7fe904600640 1 -- 172.21.15.27:0/353678041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8fc071420 msgr2=0x7fe8fc15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:56.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.992+0000 7fe904600640 1 --2- 172.21.15.27:0/353678041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8fc071420 0x7fe8fc15bee0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe8f400d150 tx=0x7fe8f401ebe0 comp rx=0 tx=0).stop 2024-01-15T18:04:56.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.992+0000 7fe904600640 1 -- 172.21.15.27:0/353678041 shutdown_connections 2024-01-15T18:04:56.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.992+0000 7fe904600640 1 --2- 172.21.15.27:0/353678041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8fc071420 0x7fe8fc15bee0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:56.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.992+0000 7fe904600640 1 --2- 172.21.15.27:0/353678041 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8fc15b100 0x7fe8fc070ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:56.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.992+0000 7fe904600640 1 -- 172.21.15.27:0/353678041 >> 172.21.15.27:0/353678041 conn(0x7fe8fc0cb700 msgr2=0x7fe8fc0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:56.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.992+0000 7fe904600640 1 -- 172.21.15.27:0/353678041 shutdown_connections 2024-01-15T18:04:56.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.992+0000 7fe904600640 1 -- 172.21.15.27:0/353678041 wait complete. 2024-01-15T18:04:56.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.996+0000 7fe904600640 1 Processor -- start 2024-01-15T18:04:56.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.996+0000 7fe904600640 1 -- start start 2024-01-15T18:04:57.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.996+0000 7fe904600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8fc071420 0x7fe8fc156c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:57.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.996+0000 7fe904600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8fc15ad20 0x7fe8fc0cdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:57.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.996+0000 7fe904600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe8fc15b100 0x7fe8fc153510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:57.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.996+0000 7fe904600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8fc155c60 con 0x7fe8fc15ad20 2024-01-15T18:04:57.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.996+0000 7fe904600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8fc157150 con 0x7fe8fc15b100 2024-01-15T18:04:57.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.996+0000 7fe904600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8fc154080 con 0x7fe8fc071420 2024-01-15T18:04:57.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.996+0000 7fe901200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8fc15ad20 0x7fe8fc0cdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.996+0000 7fe901200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8fc15ad20 0x7fe8fc0cdbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46468/0 (socket says 172.21.15.27:46468) 2024-01-15T18:04:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.996+0000 7fe901200640 1 -- 172.21.15.27:0/2237705426 learned_addr learned my addr 172.21.15.27:0/2237705426 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.996+0000 7fe901c00640 1 --2- 172.21.15.27:0/2237705426 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8fc071420 0x7fe8fc156c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.996+0000 7fe902600640 1 --2- 172.21.15.27:0/2237705426 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe8fc15b100 0x7fe8fc153510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.996+0000 7fe901200640 1 -- 172.21.15.27:0/2237705426 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8fc071420 msgr2=0x7fe8fc156c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.996+0000 7fe901200640 1 --2- 172.21.15.27:0/2237705426 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8fc071420 0x7fe8fc156c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.996+0000 7fe901200640 1 -- 172.21.15.27:0/2237705426 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe8fc15b100 msgr2=0x7fe8fc153510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.996+0000 7fe901200640 1 --2- 172.21.15.27:0/2237705426 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe8fc15b100 0x7fe8fc153510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.996+0000 7fe901200640 1 -- 172.21.15.27:0/2237705426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f400bd40 con 0x7fe8fc15ad20 2024-01-15T18:04:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.996+0000 7fe901200640 1 --2- 172.21.15.27:0/2237705426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8fc15ad20 0x7fe8fc0cdbe0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe8e400e870 tx=0x7fe8e400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.996+0000 7fe8f9600640 1 -- 172.21.15.27:0/2237705426 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e4009510 con 0x7fe8fc15ad20 2024-01-15T18:04:57.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.996+0000 7fe8f9600640 1 -- 172.21.15.27:0/2237705426 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8e400eec0 con 0x7fe8fc15ad20 2024-01-15T18:04:57.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.996+0000 7fe904600640 1 -- 172.21.15.27:0/2237705426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8fc0b4aa0 con 0x7fe8fc15ad20 2024-01-15T18:04:57.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.996+0000 7fe8f9600640 1 -- 172.21.15.27:0/2237705426 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e400cb40 con 0x7fe8fc15ad20 2024-01-15T18:04:57.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:56.996+0000 7fe904600640 1 -- 172.21.15.27:0/2237705426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8fc156410 con 0x7fe8fc15ad20 2024-01-15T18:04:57.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:57.000+0000 7fe904600640 1 -- 172.21.15.27:0/2237705426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8fc04e850 con 0x7fe8fc15ad20 2024-01-15T18:04:57.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:57.000+0000 7fe8f9600640 1 -- 172.21.15.27:0/2237705426 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fe8e400cca0 con 0x7fe8fc15ad20 2024-01-15T18:04:57.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:57.000+0000 7fe8f9600640 1 --2- 172.21.15.27:0/2237705426 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe8d40668f0 0x7fe8d4068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:57.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:57.000+0000 7fe8f9600640 1 -- 172.21.15.27:0/2237705426 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fe8e4084210 con 0x7fe8fc15ad20 2024-01-15T18:04:57.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:57.000+0000 7fe901c00640 1 --2- 172.21.15.27:0/2237705426 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe8d40668f0 0x7fe8d4068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:57.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:57.004+0000 7fe901c00640 1 --2- 172.21.15.27:0/2237705426 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe8d40668f0 0x7fe8d4068db0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec007fd0 tx=0x7fe8ec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:57.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:57.004+0000 7fe8f9600640 1 -- 172.21.15.27:0/2237705426 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe8e4051640 con 0x7fe8fc15ad20 2024-01-15T18:04:57.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:57.164+0000 7fe904600640 1 -- 172.21.15.27:0/2237705426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fe8fc0b4a40 con 0x7fe8fc15ad20 2024-01-15T18:04:57.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:57.804+0000 7fe8f9600640 1 -- 172.21.15.27:0/2237705426 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) v1 ==== 170+0+0 (secure 0 0 0) 0x7fe8e4054c60 con 0x7fe8fc15ad20 2024-01-15T18:04:57.809 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-01-15T18:04:57.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:57.808+0000 7fe904600640 1 -- 172.21.15.27:0/2237705426 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe8d40668f0 msgr2=0x7fe8d4068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:57.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:57.808+0000 7fe904600640 1 --2- 172.21.15.27:0/2237705426 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe8d40668f0 0x7fe8d4068db0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec007fd0 tx=0x7fe8ec009040 comp rx=0 tx=0).stop 2024-01-15T18:04:57.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:57.808+0000 7fe904600640 1 -- 172.21.15.27:0/2237705426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8fc15ad20 msgr2=0x7fe8fc0cdbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:57.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:57.808+0000 7fe904600640 1 --2- 172.21.15.27:0/2237705426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8fc15ad20 0x7fe8fc0cdbe0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe8e400e870 tx=0x7fe8e400ec70 comp rx=0 tx=0).stop 2024-01-15T18:04:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:57.808+0000 7fe904600640 1 -- 172.21.15.27:0/2237705426 shutdown_connections 2024-01-15T18:04:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:57.808+0000 7fe904600640 1 --2- 172.21.15.27:0/2237705426 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe8d40668f0 0x7fe8d4068db0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:57.808+0000 7fe904600640 1 --2- 172.21.15.27:0/2237705426 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe8fc15b100 0x7fe8fc153510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:57.808+0000 7fe904600640 1 --2- 172.21.15.27:0/2237705426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8fc15ad20 0x7fe8fc0cdbe0 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe8e400e870 tx=0x7fe8e400ec70 comp rx=0 tx=0).stop 2024-01-15T18:04:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:57.808+0000 7fe904600640 1 --2- 172.21.15.27:0/2237705426 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8fc071420 0x7fe8fc156c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:57.808+0000 7fe904600640 1 -- 172.21.15.27:0/2237705426 >> 172.21.15.27:0/2237705426 conn(0x7fe8fc0cb700 msgr2=0x7fe8fc1642d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:57.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:57.812+0000 7fe904600640 1 -- 172.21.15.27:0/2237705426 shutdown_connections 2024-01-15T18:04:57.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:57.812+0000 7fe904600640 1 -- 172.21.15.27:0/2237705426 wait complete. 2024-01-15T18:04:57.855 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-15T18:04:58.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:57.996+0000 7fc84c600640 1 Processor -- start 2024-01-15T18:04:58.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:57.996+0000 7fc84c600640 1 -- start start 2024-01-15T18:04:58.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.000+0000 7fc84c600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fc84415d480 0x7fc8440cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:58.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.000+0000 7fc84c600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc84406c740 0x7fc8440d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:58.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.000+0000 7fc84c600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc84415d860 con 0x7fc844099a10 2024-01-15T18:04:58.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.000+0000 7fc84c600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc84406b6a0 con 0x7fc84406c740 2024-01-15T18:04:58.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.000+0000 7fc84c600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc84406b980 con 0x7fc84415d480 2024-01-15T18:04:58.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.000+0000 7fc849200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc84406c740 0x7fc8440d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:58.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.000+0000 7fc84a600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc844099a10 0x7fc844160f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53028/0 (socket says 172.21.15.27:53028) 2024-01-15T18:04:58.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.000+0000 7fc84a600640 1 -- 172.21.15.27:0/1270522963 learned_addr learned my addr 172.21.15.27:0/1270522963 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:04:58.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.000+0000 7fc849c00640 1 --2- 172.21.15.27:0/1270522963 >> v2:172.21.15.93:3300/0 conn(0x7fc84415d480 0x7fc8440cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:58.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.000+0000 7fc849200640 1 -- 172.21.15.27:0/1270522963 >> v2:172.21.15.93:3300/0 conn(0x7fc84415d480 msgr2=0x7fc8440cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:58.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.000+0000 7fc849200640 1 --2- 172.21.15.27:0/1270522963 >> v2:172.21.15.93:3300/0 conn(0x7fc84415d480 0x7fc8440cfbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:58.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.000+0000 7fc849200640 1 -- 172.21.15.27:0/1270522963 >> v1:172.21.15.27:6789/0 conn(0x7fc844099a10 legacy=0x7fc844160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:58.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.000+0000 7fc849200640 1 -- 172.21.15.27:0/1270522963 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc84406bc60 con 0x7fc84406c740 2024-01-15T18:04:58.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.000+0000 7fc843e00640 1 -- 172.21.15.27:0/1270522963 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017370077 0 0) 0x7fc84415d860 con 0x7fc844099a10 2024-01-15T18:04:58.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.004+0000 7fc849200640 1 --2- 172.21.15.27:0/1270522963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc84406c740 0x7fc8440d0130 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc82c0097c0 tx=0x7fc82c01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc3b33943485bb4a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:58.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.004+0000 7fc843e00640 1 -- 172.21.15.27:0/1270522963 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82c0ab600 con 0x7fc84406c740 2024-01-15T18:04:58.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.004+0000 7fc843e00640 1 -- 172.21.15.27:0/1270522963 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc82c0a7050 con 0x7fc84406c740 2024-01-15T18:04:58.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.004+0000 7fc843e00640 1 -- 172.21.15.27:0/1270522963 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82c0ab980 con 0x7fc84406c740 2024-01-15T18:04:58.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.004+0000 7fc84c600640 1 -- 172.21.15.27:0/1270522963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc84406c740 msgr2=0x7fc8440d0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:58.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.004+0000 7fc84c600640 1 --2- 172.21.15.27:0/1270522963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc84406c740 0x7fc8440d0130 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc82c0097c0 tx=0x7fc82c01c440 comp rx=0 tx=0).stop 2024-01-15T18:04:58.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.004+0000 7fc84c600640 1 -- 172.21.15.27:0/1270522963 shutdown_connections 2024-01-15T18:04:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.004+0000 7fc84c600640 1 --2- 172.21.15.27:0/1270522963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc84406c740 0x7fc8440d0130 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.004+0000 7fc84c600640 1 --2- 172.21.15.27:0/1270522963 >> v2:172.21.15.93:3300/0 conn(0x7fc84415d480 0x7fc8440cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.004+0000 7fc84c600640 1 -- 172.21.15.27:0/1270522963 >> 172.21.15.27:0/1270522963 conn(0x7fc8440b2a80 msgr2=0x7fc8440b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.004+0000 7fc84c600640 1 -- 172.21.15.27:0/1270522963 shutdown_connections 2024-01-15T18:04:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.004+0000 7fc84c600640 1 -- 172.21.15.27:0/1270522963 wait complete. 2024-01-15T18:04:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.004+0000 7fc84c600640 1 Processor -- start 2024-01-15T18:04:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.004+0000 7fc84c600640 1 -- start start 2024-01-15T18:04:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.008+0000 7fc84c600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc84406c740 0x7fc8441534e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.008+0000 7fc84c600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc844099a10 0x7fc84415a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.008+0000 7fc84c600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc84415d480 0x7fc8440cf170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.008+0000 7fc84c600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc844153a20 con 0x7fc84406c740 2024-01-15T18:04:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.008+0000 7fc84c600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84415b620 con 0x7fc844099a10 2024-01-15T18:04:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.008+0000 7fc84c600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc84415ab00 con 0x7fc84415d480 2024-01-15T18:04:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.008+0000 7fc849c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc84406c740 0x7fc8441534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.008+0000 7fc849c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc84406c740 0x7fc8441534e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46484/0 (socket says 172.21.15.27:46484) 2024-01-15T18:04:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.008+0000 7fc849c00640 1 -- 172.21.15.27:0/2224313460 learned_addr learned my addr 172.21.15.27:0/2224313460 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.008+0000 7fc84a600640 1 --2- 172.21.15.27:0/2224313460 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc84415d480 0x7fc8440cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:58.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.008+0000 7fc849200640 1 --2- 172.21.15.27:0/2224313460 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc844099a10 0x7fc84415a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:58.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.008+0000 7fc849c00640 1 -- 172.21.15.27:0/2224313460 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc84415d480 msgr2=0x7fc8440cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:58.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.008+0000 7fc849c00640 1 --2- 172.21.15.27:0/2224313460 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc84415d480 0x7fc8440cf170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:58.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.008+0000 7fc849c00640 1 -- 172.21.15.27:0/2224313460 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc844099a10 msgr2=0x7fc84415a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:58.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.008+0000 7fc849c00640 1 --2- 172.21.15.27:0/2224313460 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc844099a10 0x7fc84415a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:58.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.008+0000 7fc849c00640 1 -- 172.21.15.27:0/2224313460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc82c009420 con 0x7fc84406c740 2024-01-15T18:04:58.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.008+0000 7fc849c00640 1 --2- 172.21.15.27:0/2224313460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc84406c740 0x7fc8441534e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc83400e8a0 tx=0x7fc83400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:58.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.008+0000 7fc841600640 1 -- 172.21.15.27:0/2224313460 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc834009530 con 0x7fc84406c740 2024-01-15T18:04:58.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.008+0000 7fc841600640 1 -- 172.21.15.27:0/2224313460 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc83400c940 con 0x7fc84406c740 2024-01-15T18:04:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.008+0000 7fc841600640 1 -- 172.21.15.27:0/2224313460 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83400ccd0 con 0x7fc84406c740 2024-01-15T18:04:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.008+0000 7fc84c600640 1 -- 172.21.15.27:0/2224313460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc84415ad80 con 0x7fc84406c740 2024-01-15T18:04:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.008+0000 7fc84c600640 1 -- 172.21.15.27:0/2224313460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc84415b340 con 0x7fc84406c740 2024-01-15T18:04:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.008+0000 7fc84c600640 1 -- 172.21.15.27:0/2224313460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc84415cfd0 con 0x7fc84406c740 2024-01-15T18:04:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.012+0000 7fc841600640 1 -- 172.21.15.27:0/2224313460 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fc834013470 con 0x7fc84406c740 2024-01-15T18:04:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.012+0000 7fc841600640 1 --2- 172.21.15.27:0/2224313460 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc81c066920 0x7fc81c068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.012+0000 7fc841600640 1 -- 172.21.15.27:0/2224313460 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fc834084ae0 con 0x7fc84406c740 2024-01-15T18:04:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.012+0000 7fc849200640 1 --2- 172.21.15.27:0/2224313460 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc81c066920 0x7fc81c068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.012+0000 7fc849200640 1 --2- 172.21.15.27:0/2224313460 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc81c066920 0x7fc81c068de0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc82c0ad000 tx=0x7fc82c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:58.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.016+0000 7fc841600640 1 -- 172.21.15.27:0/2224313460 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc834051fa0 con 0x7fc84406c740 2024-01-15T18:04:58.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.204+0000 7fc84c600640 1 -- 172.21.15.27:0/2224313460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc84404e850 con 0x7fc84406c740 2024-01-15T18:04:58.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.832+0000 7fc841600640 1 -- 172.21.15.27:0/2224313460 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7fc8340170c0 con 0x7fc84406c740 2024-01-15T18:04:58.842 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 2 and data pool 3 2024-01-15T18:04:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.844+0000 7fc84c600640 1 -- 172.21.15.27:0/2224313460 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc81c066920 msgr2=0x7fc81c068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.844+0000 7fc84c600640 1 --2- 172.21.15.27:0/2224313460 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc81c066920 0x7fc81c068de0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc82c0ad000 tx=0x7fc82c008040 comp rx=0 tx=0).stop 2024-01-15T18:04:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.844+0000 7fc84c600640 1 -- 172.21.15.27:0/2224313460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc84406c740 msgr2=0x7fc8441534e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.844+0000 7fc84c600640 1 --2- 172.21.15.27:0/2224313460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc84406c740 0x7fc8441534e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc83400e8a0 tx=0x7fc83400ed60 comp rx=0 tx=0).stop 2024-01-15T18:04:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.844+0000 7fc84c600640 1 -- 172.21.15.27:0/2224313460 shutdown_connections 2024-01-15T18:04:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.844+0000 7fc84c600640 1 --2- 172.21.15.27:0/2224313460 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc81c066920 0x7fc81c068de0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.844+0000 7fc84c600640 1 --2- 172.21.15.27:0/2224313460 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc84415d480 0x7fc8440cf170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.844+0000 7fc84c600640 1 --2- 172.21.15.27:0/2224313460 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc844099a10 0x7fc84415a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.844+0000 7fc84c600640 1 --2- 172.21.15.27:0/2224313460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc84406c740 0x7fc8441534e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.844+0000 7fc84c600640 1 -- 172.21.15.27:0/2224313460 >> 172.21.15.27:0/2224313460 conn(0x7fc8440b2a80 msgr2=0x7fc84406f810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:58.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.844+0000 7fc84c600640 1 -- 172.21.15.27:0/2224313460 shutdown_connections 2024-01-15T18:04:58.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:58.844+0000 7fc84c600640 1 -- 172.21.15.27:0/2224313460 wait complete. 2024-01-15T18:04:58.891 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:04:59.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.028+0000 7f9087400640 1 Processor -- start 2024-01-15T18:04:59.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.028+0000 7f9087400640 1 -- start start 2024-01-15T18:04:59.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.028+0000 7f9087400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f908015a080 0x7f90800b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:59.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.028+0000 7f9087400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908015a460 con 0x7f908015db90 2024-01-15T18:04:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.028+0000 7f9087400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f908006dcd0 con 0x7f90800cca00 2024-01-15T18:04:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.028+0000 7f9087400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f908006e8c0 con 0x7f908015a080 2024-01-15T18:04:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.028+0000 7f9084a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f908015a080 0x7f90800b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.028+0000 7f9077e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f908015db90 0x7f90800b3f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53034/0 (socket says 172.21.15.27:53034) 2024-01-15T18:04:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.028+0000 7f9077e00640 1 -- 172.21.15.27:0/3324057118 learned_addr learned my addr 172.21.15.27:0/3324057118 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:04:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.032+0000 7f9084a00640 1 -- 172.21.15.27:0/3324057118 >> v1:172.21.15.93:6789/0 conn(0x7f90800cca00 legacy=0x7f908006ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.032+0000 7f9084a00640 1 -- 172.21.15.27:0/3324057118 >> v1:172.21.15.27:6789/0 conn(0x7f908015db90 legacy=0x7f90800b3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.032+0000 7f9084a00640 1 -- 172.21.15.27:0/3324057118 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908006df20 con 0x7f908015a080 2024-01-15T18:04:59.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.032+0000 7f907fe00640 1 -- 172.21.15.27:0/3324057118 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1205828532 0 0) 0x7f908015a460 con 0x7f908015db90 2024-01-15T18:04:59.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.032+0000 7f9084a00640 1 --2- 172.21.15.27:0/3324057118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f908015a080 0x7f90800b3a00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f90700097c0 tx=0x7f907001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56cd2e7c161a437a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:59.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.032+0000 7f907fe00640 1 -- 172.21.15.27:0/3324057118 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90700ab640 con 0x7f908015a080 2024-01-15T18:04:59.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.032+0000 7f907fe00640 1 -- 172.21.15.27:0/3324057118 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90700a7070 con 0x7f908015a080 2024-01-15T18:04:59.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.032+0000 7f907fe00640 1 -- 172.21.15.27:0/3324057118 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90700ab9c0 con 0x7f908015a080 2024-01-15T18:04:59.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.036+0000 7f9087400640 1 -- 172.21.15.27:0/3324057118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f908015a080 msgr2=0x7f90800b3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:59.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.036+0000 7f9087400640 1 --2- 172.21.15.27:0/3324057118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f908015a080 0x7f90800b3a00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f90700097c0 tx=0x7f907001c440 comp rx=0 tx=0).stop 2024-01-15T18:04:59.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.036+0000 7f9087400640 1 -- 172.21.15.27:0/3324057118 shutdown_connections 2024-01-15T18:04:59.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.036+0000 7f9087400640 1 --2- 172.21.15.27:0/3324057118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f908015a080 0x7f90800b3a00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.036+0000 7f9087400640 1 -- 172.21.15.27:0/3324057118 >> 172.21.15.27:0/3324057118 conn(0x7f90800b2360 msgr2=0x7f90800b2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:59.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.036+0000 7f9087400640 1 -- 172.21.15.27:0/3324057118 shutdown_connections 2024-01-15T18:04:59.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.036+0000 7f9087400640 1 -- 172.21.15.27:0/3324057118 wait complete. 2024-01-15T18:04:59.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.036+0000 7f9087400640 1 Processor -- start 2024-01-15T18:04:59.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.036+0000 7f9087400640 1 -- start start 2024-01-15T18:04:59.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.036+0000 7f9087400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f90800cca00 0x7f9080150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:59.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.036+0000 7f9087400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f908015a080 0x7f9080150110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:59.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.036+0000 7f9087400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9080150fb0 0x7f90800cf050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:59.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.036+0000 7f9087400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9080154990 con 0x7f908015a080 2024-01-15T18:04:59.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.036+0000 7f9087400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90801543d0 con 0x7f90800cca00 2024-01-15T18:04:59.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.036+0000 7f9087400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90801546b0 con 0x7f9080150fb0 2024-01-15T18:04:59.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.036+0000 7f9077e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f908015a080 0x7f9080150110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:59.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.036+0000 7f9077e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f908015a080 0x7f9080150110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46500/0 (socket says 172.21.15.27:46500) 2024-01-15T18:04:59.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.036+0000 7f9077e00640 1 -- 172.21.15.27:0/278900027 learned_addr learned my addr 172.21.15.27:0/278900027 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:59.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.036+0000 7f9084a00640 1 --2- 172.21.15.27:0/278900027 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f90800cca00 0x7f9080150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:59.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.036+0000 7f9085e00640 1 --2- 172.21.15.27:0/278900027 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9080150fb0 0x7f90800cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:59.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.036+0000 7f9077e00640 1 -- 172.21.15.27:0/278900027 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9080150fb0 msgr2=0x7f90800cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:59.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.036+0000 7f9077e00640 1 --2- 172.21.15.27:0/278900027 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9080150fb0 0x7f90800cf050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.040+0000 7f9077e00640 1 -- 172.21.15.27:0/278900027 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f90800cca00 msgr2=0x7f9080150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:59.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.040+0000 7f9077e00640 1 --2- 172.21.15.27:0/278900027 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f90800cca00 0x7f9080150960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.040+0000 7f9077e00640 1 -- 172.21.15.27:0/278900027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9070009420 con 0x7f908015a080 2024-01-15T18:04:59.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.040+0000 7f9084a00640 1 --2- 172.21.15.27:0/278900027 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f90800cca00 0x7f9080150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:04:59.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.040+0000 7f9077e00640 1 --2- 172.21.15.27:0/278900027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f908015a080 0x7f9080150110 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f906800c410 tx=0x7f906800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:59.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.040+0000 7f907d600640 1 -- 172.21.15.27:0/278900027 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90680074e0 con 0x7f908015a080 2024-01-15T18:04:59.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.040+0000 7f907d600640 1 -- 172.21.15.27:0/278900027 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9068007640 con 0x7f908015a080 2024-01-15T18:04:59.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.040+0000 7f9087400640 1 -- 172.21.15.27:0/278900027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9080156360 con 0x7f908015a080 2024-01-15T18:04:59.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.040+0000 7f907d600640 1 -- 172.21.15.27:0/278900027 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9068004a30 con 0x7f908015a080 2024-01-15T18:04:59.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.040+0000 7f9087400640 1 -- 172.21.15.27:0/278900027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f908009e190 con 0x7f908015a080 2024-01-15T18:04:59.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.040+0000 7f9087400640 1 -- 172.21.15.27:0/278900027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90801561c0 con 0x7f908015a080 2024-01-15T18:04:59.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.040+0000 7f907d600640 1 -- 172.21.15.27:0/278900027 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f9068004b90 con 0x7f908015a080 2024-01-15T18:04:59.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.040+0000 7f907d600640 1 --2- 172.21.15.27:0/278900027 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f90580668f0 0x7f9058068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:59.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.040+0000 7f907d600640 1 -- 172.21.15.27:0/278900027 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90680842e0 con 0x7f908015a080 2024-01-15T18:04:59.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.044+0000 7f9084a00640 1 --2- 172.21.15.27:0/278900027 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f90580668f0 0x7f9058068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:59.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.044+0000 7f9084a00640 1 --2- 172.21.15.27:0/278900027 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f90580668f0 0x7f9058068db0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f90700af000 tx=0x7f9070008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:59.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.044+0000 7f907d600640 1 -- 172.21.15.27:0/278900027 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f90801561c0 con 0x7f908015a080 2024-01-15T18:04:59.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.204+0000 7f9087400640 1 -- 172.21.15.27:0/278900027 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90801561c0 con 0x7f908015a080 2024-01-15T18:04:59.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.208+0000 7f907d600640 1 -- 172.21.15.27:0/278900027 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16422 (secure 0 0 0) 0x7f90801561c0 con 0x7f908015a080 2024-01-15T18:04:59.210 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:04:59.210 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:04:58.207697+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:04:56.220608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T18:04:57.210421+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:04:59.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.208+0000 7f9087400640 1 -- 172.21.15.27:0/278900027 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f90580668f0 msgr2=0x7f9058068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:59.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.208+0000 7f9087400640 1 --2- 172.21.15.27:0/278900027 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f90580668f0 0x7f9058068db0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f90700af000 tx=0x7f9070008040 comp rx=0 tx=0).stop 2024-01-15T18:04:59.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.208+0000 7f9087400640 1 -- 172.21.15.27:0/278900027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f908015a080 msgr2=0x7f9080150110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:59.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.208+0000 7f9087400640 1 --2- 172.21.15.27:0/278900027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f908015a080 0x7f9080150110 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f906800c410 tx=0x7f906800c8d0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.208+0000 7f9087400640 1 -- 172.21.15.27:0/278900027 shutdown_connections 2024-01-15T18:04:59.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.208+0000 7f9087400640 1 --2- 172.21.15.27:0/278900027 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f90580668f0 0x7f9058068db0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.208+0000 7f9087400640 1 --2- 172.21.15.27:0/278900027 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9080150fb0 0x7f90800cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.208+0000 7f9087400640 1 --2- 172.21.15.27:0/278900027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f908015a080 0x7f9080150110 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.208+0000 7f9087400640 1 --2- 172.21.15.27:0/278900027 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f90800cca00 0x7f9080150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.208+0000 7f9087400640 1 -- 172.21.15.27:0/278900027 >> 172.21.15.27:0/278900027 conn(0x7f90800b2360 msgr2=0x7f908006acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:59.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.208+0000 7f9087400640 1 -- 172.21.15.27:0/278900027 shutdown_connections 2024-01-15T18:04:59.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.208+0000 7f9087400640 1 -- 172.21.15.27:0/278900027 wait complete. 2024-01-15T18:04:59.232 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:04:59.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.368+0000 7ff932000640 1 Processor -- start 2024-01-15T18:04:59.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.368+0000 7ff932000640 1 -- start start 2024-01-15T18:04:59.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.368+0000 7ff932000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff92c0cedd0 0x7ff92c0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:59.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.368+0000 7ff932000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff92c0cd8e0 0x7ff92c0cdd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:59.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.372+0000 7ff932000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff92c0cbf40 con 0x7ff92c15b220 2024-01-15T18:04:59.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.372+0000 7ff932000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff92c0ce290 con 0x7ff92c0cedd0 2024-01-15T18:04:59.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.372+0000 7ff932000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c0b1250 con 0x7ff92c0cd8e0 2024-01-15T18:04:59.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.372+0000 7ff92be00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff92c0cd8e0 0x7ff92c0cdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:59.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.372+0000 7ff92aa00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff92c0cedd0 0x7ff92c0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:59.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.372+0000 7ff92aa00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff92c0cedd0 0x7ff92c0cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:41440/0 (socket says 172.21.15.27:41440) 2024-01-15T18:04:59.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.372+0000 7ff92aa00640 1 -- 172.21.15.27:0/2791223966 learned_addr learned my addr 172.21.15.27:0/2791223966 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:59.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.372+0000 7ff92be00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff92c0cd8e0 0x7ff92c0cdd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:56876/0 (socket says 172.21.15.27:56876) 2024-01-15T18:04:59.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.372+0000 7ff92b400640 1 --1- 172.21.15.27:0/2791223966 >> v1:172.21.15.93:6789/0 conn(0x7ff92c15b220 0x7ff92c0b0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:47774/0 (socket says 172.21.15.27:47774) 2024-01-15T18:04:59.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.372+0000 7ff92aa00640 1 -- 172.21.15.27:0/2791223966 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff92c0cd8e0 msgr2=0x7ff92c0cdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:59.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.372+0000 7ff92aa00640 1 --2- 172.21.15.27:0/2791223966 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff92c0cd8e0 0x7ff92c0cdd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.372+0000 7ff92aa00640 1 -- 172.21.15.27:0/2791223966 >> v1:172.21.15.93:6789/0 conn(0x7ff92c15b220 legacy=0x7ff92c0b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:59.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.372+0000 7ff92aa00640 1 -- 172.21.15.27:0/2791223966 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff92c0cf890 con 0x7ff92c0cedd0 2024-01-15T18:04:59.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.372+0000 7ff92a000640 1 -- 172.21.15.27:0/2791223966 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225045940 0 0) 0x7ff92c0cbf40 con 0x7ff92c15b220 2024-01-15T18:04:59.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.376+0000 7ff92aa00640 1 --2- 172.21.15.27:0/2791223966 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff92c0cedd0 0x7ff92c0cf230 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff914009870 tx=0x7ff91401c510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31a77f8490ac6008 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:59.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.376+0000 7ff92a000640 1 -- 172.21.15.27:0/2791223966 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9140aa670 con 0x7ff92c0cedd0 2024-01-15T18:04:59.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.376+0000 7ff92a000640 1 -- 172.21.15.27:0/2791223966 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9140aa7d0 con 0x7ff92c0cedd0 2024-01-15T18:04:59.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.376+0000 7ff92a000640 1 -- 172.21.15.27:0/2791223966 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9140aab60 con 0x7ff92c0cedd0 2024-01-15T18:04:59.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.376+0000 7ff932000640 1 -- 172.21.15.27:0/2791223966 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff92c0cedd0 msgr2=0x7ff92c0cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:59.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.376+0000 7ff932000640 1 --2- 172.21.15.27:0/2791223966 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff92c0cedd0 0x7ff92c0cf230 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff914009870 tx=0x7ff91401c510 comp rx=0 tx=0).stop 2024-01-15T18:04:59.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.376+0000 7ff932000640 1 -- 172.21.15.27:0/2791223966 shutdown_connections 2024-01-15T18:04:59.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.376+0000 7ff932000640 1 --2- 172.21.15.27:0/2791223966 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff92c0cd8e0 0x7ff92c0cdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.376+0000 7ff932000640 1 --2- 172.21.15.27:0/2791223966 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff92c0cedd0 0x7ff92c0cf230 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.376+0000 7ff932000640 1 -- 172.21.15.27:0/2791223966 >> 172.21.15.27:0/2791223966 conn(0x7ff92c099a10 msgr2=0x7ff92c099e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:59.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.376+0000 7ff932000640 1 -- 172.21.15.27:0/2791223966 shutdown_connections 2024-01-15T18:04:59.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.376+0000 7ff932000640 1 -- 172.21.15.27:0/2791223966 wait complete. 2024-01-15T18:04:59.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.376+0000 7ff932000640 1 Processor -- start 2024-01-15T18:04:59.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.376+0000 7ff932000640 1 -- start start 2024-01-15T18:04:59.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff932000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff92c0cd8e0 0x7ff92c14df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:59.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff932000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff92c0cedd0 0x7ff92c06b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:59.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff932000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff92c15b220 0x7ff92c0af610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:59.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff932000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c06c1e0 con 0x7ff92c0cd8e0 2024-01-15T18:04:59.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff932000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c06bf00 con 0x7ff92c15b220 2024-01-15T18:04:59.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff932000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c0c4a80 con 0x7ff92c0cedd0 2024-01-15T18:04:59.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff92b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff92c0cd8e0 0x7ff92c14df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:59.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff92b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff92c0cd8e0 0x7ff92c14df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46512/0 (socket says 172.21.15.27:46512) 2024-01-15T18:04:59.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff92b400640 1 -- 172.21.15.27:0/3943359645 learned_addr learned my addr 172.21.15.27:0/3943359645 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff92aa00640 1 --2- 172.21.15.27:0/3943359645 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff92c0cedd0 0x7ff92c06b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff92be00640 1 --2- 172.21.15.27:0/3943359645 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff92c15b220 0x7ff92c0af610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff92b400640 1 -- 172.21.15.27:0/3943359645 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff92c0cedd0 msgr2=0x7ff92c06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff92b400640 1 --2- 172.21.15.27:0/3943359645 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff92c0cedd0 0x7ff92c06b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff92b400640 1 -- 172.21.15.27:0/3943359645 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff92c15b220 msgr2=0x7ff92c0af610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff92b400640 1 --2- 172.21.15.27:0/3943359645 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff92c15b220 0x7ff92c0af610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff92b400640 1 -- 172.21.15.27:0/3943359645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9140094d0 con 0x7ff92c0cd8e0 2024-01-15T18:04:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff92aa00640 1 --2- 172.21.15.27:0/3943359645 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff92c0cedd0 0x7ff92c06b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:04:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff92be00640 1 --2- 172.21.15.27:0/3943359645 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff92c15b220 0x7ff92c0af610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:04:59.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff92b400640 1 --2- 172.21.15.27:0/3943359645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff92c0cd8e0 0x7ff92c14df40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff91c0011f0 tx=0x7ff91c00c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff923400640 1 -- 172.21.15.27:0/3943359645 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c0074e0 con 0x7ff92c0cd8e0 2024-01-15T18:04:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff923400640 1 -- 172.21.15.27:0/3943359645 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff91c007640 con 0x7ff92c0cd8e0 2024-01-15T18:04:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff932000640 1 -- 172.21.15.27:0/3943359645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff92c0c4d60 con 0x7ff92c0cd8e0 2024-01-15T18:04:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff923400640 1 -- 172.21.15.27:0/3943359645 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c004a10 con 0x7ff92c0cd8e0 2024-01-15T18:04:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff932000640 1 -- 172.21.15.27:0/3943359645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff92c06bb40 con 0x7ff92c0cd8e0 2024-01-15T18:04:59.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff932000640 1 -- 172.21.15.27:0/3943359645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff92c0c48e0 con 0x7ff92c0cd8e0 2024-01-15T18:04:59.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.380+0000 7ff923400640 1 -- 172.21.15.27:0/3943359645 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7ff91c004b70 con 0x7ff92c0cd8e0 2024-01-15T18:04:59.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.384+0000 7ff923400640 1 --2- 172.21.15.27:0/3943359645 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff9040668f0 0x7ff904068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:59.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.384+0000 7ff923400640 1 -- 172.21.15.27:0/3943359645 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff91c083940 con 0x7ff92c0cd8e0 2024-01-15T18:04:59.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.384+0000 7ff92aa00640 1 --2- 172.21.15.27:0/3943359645 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff9040668f0 0x7ff904068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:59.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.384+0000 7ff92aa00640 1 --2- 172.21.15.27:0/3943359645 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff9040668f0 0x7ff904068db0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff9140ae000 tx=0x7ff9140a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:59.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.388+0000 7ff923400640 1 -- 172.21.15.27:0/3943359645 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff92c0c48e0 con 0x7ff92c0cd8e0 2024-01-15T18:04:59.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.544+0000 7ff932000640 1 -- 172.21.15.27:0/3943359645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff92c0c48e0 con 0x7ff92c0cd8e0 2024-01-15T18:04:59.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.544+0000 7ff923400640 1 -- 172.21.15.27:0/3943359645 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16422 (secure 0 0 0) 0x7ff92c0c48e0 con 0x7ff92c0cd8e0 2024-01-15T18:04:59.550 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:04:59.550 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:04:58.207697+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:04:56.220608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T18:04:57.210421+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:04:59.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.548+0000 7ff932000640 1 -- 172.21.15.27:0/3943359645 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff9040668f0 msgr2=0x7ff904068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:59.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.548+0000 7ff932000640 1 --2- 172.21.15.27:0/3943359645 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff9040668f0 0x7ff904068db0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff9140ae000 tx=0x7ff9140a7000 comp rx=0 tx=0).stop 2024-01-15T18:04:59.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.548+0000 7ff932000640 1 -- 172.21.15.27:0/3943359645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff92c0cd8e0 msgr2=0x7ff92c14df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:59.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.548+0000 7ff932000640 1 --2- 172.21.15.27:0/3943359645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff92c0cd8e0 0x7ff92c14df40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff91c0011f0 tx=0x7ff91c00c9b0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.548+0000 7ff932000640 1 -- 172.21.15.27:0/3943359645 shutdown_connections 2024-01-15T18:04:59.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.548+0000 7ff932000640 1 --2- 172.21.15.27:0/3943359645 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff9040668f0 0x7ff904068db0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.548+0000 7ff932000640 1 --2- 172.21.15.27:0/3943359645 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff92c15b220 0x7ff92c0af610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.548+0000 7ff932000640 1 --2- 172.21.15.27:0/3943359645 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff92c0cedd0 0x7ff92c06b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.548+0000 7ff932000640 1 --2- 172.21.15.27:0/3943359645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff92c0cd8e0 0x7ff92c14df40 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.548+0000 7ff932000640 1 -- 172.21.15.27:0/3943359645 >> 172.21.15.27:0/3943359645 conn(0x7ff92c099a10 msgr2=0x7ff92c0adc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:59.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.548+0000 7ff932000640 1 -- 172.21.15.27:0/3943359645 shutdown_connections 2024-01-15T18:04:59.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.548+0000 7ff932000640 1 -- 172.21.15.27:0/3943359645 wait complete. 2024-01-15T18:04:59.573 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-15T18:04:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.712+0000 7f0943000640 1 Processor -- start 2024-01-15T18:04:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.712+0000 7f0943000640 1 -- start start 2024-01-15T18:04:59.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.712+0000 7f0943000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f093c0cde10 0x7f093c0cbf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:59.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.712+0000 7f0943000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f093c0cfc30 0x7f093c0cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:59.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.712+0000 7f0943000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f093c0cb4d0 con 0x7f093c15d680 2024-01-15T18:04:59.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.712+0000 7f0943000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f093c0d0120 con 0x7f093c0cde10 2024-01-15T18:04:59.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.712+0000 7f0943000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f093c0d0400 con 0x7f093c0cfc30 2024-01-15T18:04:59.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.712+0000 7f093b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f093c0cde10 0x7f093c0cbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:59.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.712+0000 7f093b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f093c0cde10 0x7f093c0cbf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46526/0 (socket says 172.21.15.27:46526) 2024-01-15T18:04:59.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.712+0000 7f093b400640 1 -- 172.21.15.27:0/2193501021 learned_addr learned my addr 172.21.15.27:0/2193501021 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:59.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.712+0000 7f0941000640 1 --2- 172.21.15.27:0/2193501021 >> v2:172.21.15.93:3300/0 conn(0x7f093c0cfc30 0x7f093c0cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:59.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.716+0000 7f093b400640 1 -- 172.21.15.27:0/2193501021 >> v2:172.21.15.93:3300/0 conn(0x7f093c0cfc30 msgr2=0x7f093c0cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:59.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.716+0000 7f093b400640 1 --2- 172.21.15.27:0/2193501021 >> v2:172.21.15.93:3300/0 conn(0x7f093c0cfc30 0x7f093c0cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.716+0000 7f093b400640 1 -- 172.21.15.27:0/2193501021 >> v1:172.21.15.93:6789/0 conn(0x7f093c15d680 legacy=0x7f093c0cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:59.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.716+0000 7f093b400640 1 -- 172.21.15.27:0/2193501021 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f093c0cb770 con 0x7f093c0cde10 2024-01-15T18:04:59.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.716+0000 7f093b400640 1 --2- 172.21.15.27:0/2193501021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f093c0cde10 0x7f093c0cbf70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f093c071f30 tx=0x7f092401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b866c31fe3aa367 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:59.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.716+0000 7f093aa00640 1 -- 172.21.15.27:0/2193501021 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093c0ce850 con 0x7f093c0cde10 2024-01-15T18:04:59.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.716+0000 7f093aa00640 1 -- 172.21.15.27:0/2193501021 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09240aa650 con 0x7f093c0cde10 2024-01-15T18:04:59.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.716+0000 7f093aa00640 1 -- 172.21.15.27:0/2193501021 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09240aa9e0 con 0x7f093c0cde10 2024-01-15T18:04:59.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.716+0000 7f0943000640 1 -- 172.21.15.27:0/2193501021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f093c0cde10 msgr2=0x7f093c0cbf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:59.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.716+0000 7f0943000640 1 --2- 172.21.15.27:0/2193501021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f093c0cde10 0x7f093c0cbf70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f093c071f30 tx=0x7f092401c4d0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.720+0000 7f0943000640 1 -- 172.21.15.27:0/2193501021 shutdown_connections 2024-01-15T18:04:59.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.720+0000 7f0943000640 1 --2- 172.21.15.27:0/2193501021 >> v2:172.21.15.93:3300/0 conn(0x7f093c0cfc30 0x7f093c0cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.720+0000 7f0943000640 1 --2- 172.21.15.27:0/2193501021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f093c0cde10 0x7f093c0cbf70 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.720+0000 7f0943000640 1 -- 172.21.15.27:0/2193501021 >> 172.21.15.27:0/2193501021 conn(0x7f093c0d0e80 msgr2=0x7f093c0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:59.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.720+0000 7f0943000640 1 -- 172.21.15.27:0/2193501021 shutdown_connections 2024-01-15T18:04:59.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.720+0000 7f0943000640 1 -- 172.21.15.27:0/2193501021 wait complete. 2024-01-15T18:04:59.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.720+0000 7f0943000640 1 Processor -- start 2024-01-15T18:04:59.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.720+0000 7f0943000640 1 -- start start 2024-01-15T18:04:59.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.720+0000 7f0943000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f093c0cde10 0x7f093c15e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:59.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.720+0000 7f0943000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f093c0cfc30 0x7f093c0c6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:59.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.720+0000 7f0943000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f093c15d680 0x7f093c0c6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:59.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.720+0000 7f0943000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093c15ee30 con 0x7f093c15d680 2024-01-15T18:04:59.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.720+0000 7f0943000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093c15f110 con 0x7f093c0cde10 2024-01-15T18:04:59.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.720+0000 7f0943000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f093c15f3f0 con 0x7f093c0cfc30 2024-01-15T18:04:59.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.720+0000 7f0941000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f093c15d680 0x7f093c0c6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:59.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.720+0000 7f0941000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f093c15d680 0x7f093c0c6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46536/0 (socket says 172.21.15.27:46536) 2024-01-15T18:04:59.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.720+0000 7f0941000640 1 -- 172.21.15.27:0/2425292519 learned_addr learned my addr 172.21.15.27:0/2425292519 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:04:59.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.720+0000 7f093b400640 1 --2- 172.21.15.27:0/2425292519 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f093c0cfc30 0x7f093c0c6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:59.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.720+0000 7f093be00640 1 --2- 172.21.15.27:0/2425292519 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f093c0cde10 0x7f093c15e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:59.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.720+0000 7f0941000640 1 -- 172.21.15.27:0/2425292519 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f093c0cfc30 msgr2=0x7f093c0c6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:59.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.720+0000 7f0941000640 1 --2- 172.21.15.27:0/2425292519 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f093c0cfc30 0x7f093c0c6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.720+0000 7f0941000640 1 -- 172.21.15.27:0/2425292519 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f093c0cde10 msgr2=0x7f093c15e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:59.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.720+0000 7f0941000640 1 --2- 172.21.15.27:0/2425292519 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f093c0cde10 0x7f093c15e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.720+0000 7f0941000640 1 -- 172.21.15.27:0/2425292519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09240094d0 con 0x7f093c15d680 2024-01-15T18:04:59.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.724+0000 7f0941000640 1 --2- 172.21.15.27:0/2425292519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f093c15d680 0x7f093c0c6c70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f093400efc0 tx=0x7f093400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:59.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.724+0000 7f0933e00640 1 -- 172.21.15.27:0/2425292519 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0934014480 con 0x7f093c15d680 2024-01-15T18:04:59.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.724+0000 7f0933e00640 1 -- 172.21.15.27:0/2425292519 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f093400f040 con 0x7f093c15d680 2024-01-15T18:04:59.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.724+0000 7f0943000640 1 -- 172.21.15.27:0/2425292519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f093c15f6d0 con 0x7f093c15d680 2024-01-15T18:04:59.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.724+0000 7f0933e00640 1 -- 172.21.15.27:0/2425292519 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0934014820 con 0x7f093c15d680 2024-01-15T18:04:59.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.724+0000 7f0943000640 1 -- 172.21.15.27:0/2425292519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f093c15ea00 con 0x7f093c15d680 2024-01-15T18:04:59.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.724+0000 7f0943000640 1 -- 172.21.15.27:0/2425292519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f093c0cdc40 con 0x7f093c15d680 2024-01-15T18:04:59.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.724+0000 7f0933e00640 1 -- 172.21.15.27:0/2425292519 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f0934014980 con 0x7f093c15d680 2024-01-15T18:04:59.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.728+0000 7f0933e00640 1 --2- 172.21.15.27:0/2425292519 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f09140668f0 0x7f0914068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:59.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.728+0000 7f0933e00640 1 -- 172.21.15.27:0/2425292519 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0934084e70 con 0x7f093c15d680 2024-01-15T18:04:59.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.728+0000 7f093be00640 1 --2- 172.21.15.27:0/2425292519 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f09140668f0 0x7f0914068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:59.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.728+0000 7f093be00640 1 --2- 172.21.15.27:0/2425292519 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f09140668f0 0x7f0914068db0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f092c007fd0 tx=0x7f092c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:59.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.732+0000 7f0933e00640 1 -- 172.21.15.27:0/2425292519 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0934052150 con 0x7f093c15d680 2024-01-15T18:04:59.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:04:59.924+0000 7f0943000640 1 -- 172.21.15.27:0/2425292519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f093c04e850 con 0x7f093c15d680 2024-01-15T18:05:00.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:00.848+0000 7f0933e00640 1 -- 172.21.15.27:0/2425292519 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f09340180b0 con 0x7f093c15d680 2024-01-15T18:05:00.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:00.848+0000 7f0943000640 1 -- 172.21.15.27:0/2425292519 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f09140668f0 msgr2=0x7f0914068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:00.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:00.848+0000 7f0943000640 1 --2- 172.21.15.27:0/2425292519 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f09140668f0 0x7f0914068db0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f092c007fd0 tx=0x7f092c009040 comp rx=0 tx=0).stop 2024-01-15T18:05:00.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:00.848+0000 7f0943000640 1 -- 172.21.15.27:0/2425292519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f093c15d680 msgr2=0x7f093c0c6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:00.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:00.848+0000 7f0943000640 1 --2- 172.21.15.27:0/2425292519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f093c15d680 0x7f093c0c6c70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f093400efc0 tx=0x7f093400c490 comp rx=0 tx=0).stop 2024-01-15T18:05:00.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:00.848+0000 7f0943000640 1 -- 172.21.15.27:0/2425292519 shutdown_connections 2024-01-15T18:05:00.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:00.848+0000 7f0943000640 1 --2- 172.21.15.27:0/2425292519 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f09140668f0 0x7f0914068db0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:00.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:00.848+0000 7f0943000640 1 --2- 172.21.15.27:0/2425292519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f093c15d680 0x7f093c0c6c70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:00.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:00.848+0000 7f0943000640 1 --2- 172.21.15.27:0/2425292519 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f093c0cfc30 0x7f093c0c6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:00.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:00.848+0000 7f0943000640 1 --2- 172.21.15.27:0/2425292519 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f093c0cde10 0x7f093c15e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:00.848+0000 7f0943000640 1 -- 172.21.15.27:0/2425292519 >> 172.21.15.27:0/2425292519 conn(0x7f093c0d0e80 msgr2=0x7f093c0d1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:00.848+0000 7f0943000640 1 -- 172.21.15.27:0/2425292519 shutdown_connections 2024-01-15T18:05:00.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:00.848+0000 7f0943000640 1 -- 172.21.15.27:0/2425292519 wait complete. 2024-01-15T18:05:00.874 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-15T18:05:00.875 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-15T18:05:01.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.012+0000 7f854d000640 1 Processor -- start 2024-01-15T18:05:01.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.016+0000 7f854d000640 1 -- start start 2024-01-15T18:05:01.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.016+0000 7f854d000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8548158900 0x7f854815f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:01.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.016+0000 7f854d000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f85480cf2e0 0x7f85480701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:01.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.016+0000 7f854d000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85480b1cb0 con 0x7f854815bf00 2024-01-15T18:05:01.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.016+0000 7f854d000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85480b1f00 con 0x7f8548158900 2024-01-15T18:05:01.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.016+0000 7f854d000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85480b21e0 con 0x7f85480cf2e0 2024-01-15T18:05:01.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.016+0000 7f8545e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8548158900 0x7f854815f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:01.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.016+0000 7f8545e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8548158900 0x7f854815f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46550/0 (socket says 172.21.15.27:46550) 2024-01-15T18:05:01.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.016+0000 7f8545e00640 1 -- 172.21.15.27:0/3437966358 learned_addr learned my addr 172.21.15.27:0/3437966358 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:01.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.016+0000 7f8546800640 1 --2- 172.21.15.27:0/3437966358 >> v2:172.21.15.93:3300/0 conn(0x7f85480cf2e0 0x7f85480701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:01.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.016+0000 7f8545e00640 1 -- 172.21.15.27:0/3437966358 >> v2:172.21.15.93:3300/0 conn(0x7f85480cf2e0 msgr2=0x7f85480701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:01.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.016+0000 7f8545e00640 1 --2- 172.21.15.27:0/3437966358 >> v2:172.21.15.93:3300/0 conn(0x7f85480cf2e0 0x7f85480701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:01.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.016+0000 7f8545e00640 1 -- 172.21.15.27:0/3437966358 >> v1:172.21.15.93:6789/0 conn(0x7f854815bf00 legacy=0x7f85480cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:01.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.016+0000 7f8545e00640 1 -- 172.21.15.27:0/3437966358 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854815ea70 con 0x7f8548158900 2024-01-15T18:05:01.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.020+0000 7f8545e00640 1 --2- 172.21.15.27:0/3437966358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8548158900 0x7f854815f550 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f85480b30c0 tx=0x7f853801c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ba871b7b0e26536 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:01.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.020+0000 7f8544a00640 1 -- 172.21.15.27:0/3437966358 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8548070f30 con 0x7f8548158900 2024-01-15T18:05:01.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.020+0000 7f8544a00640 1 -- 172.21.15.27:0/3437966358 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85380ac410 con 0x7f8548158900 2024-01-15T18:05:01.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.020+0000 7f8544a00640 1 -- 172.21.15.27:0/3437966358 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85380ac7a0 con 0x7f8548158900 2024-01-15T18:05:01.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.020+0000 7f854d000640 1 -- 172.21.15.27:0/3437966358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8548158900 msgr2=0x7f854815f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:01.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.020+0000 7f854d000640 1 --2- 172.21.15.27:0/3437966358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8548158900 0x7f854815f550 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f85480b30c0 tx=0x7f853801c4c0 comp rx=0 tx=0).stop 2024-01-15T18:05:01.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.020+0000 7f854d000640 1 -- 172.21.15.27:0/3437966358 shutdown_connections 2024-01-15T18:05:01.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.020+0000 7f854d000640 1 --2- 172.21.15.27:0/3437966358 >> v2:172.21.15.93:3300/0 conn(0x7f85480cf2e0 0x7f85480701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:01.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.020+0000 7f854d000640 1 --2- 172.21.15.27:0/3437966358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8548158900 0x7f854815f550 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:01.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.020+0000 7f854d000640 1 -- 172.21.15.27:0/3437966358 >> 172.21.15.27:0/3437966358 conn(0x7f85480cb700 msgr2=0x7f85480cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:01.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.020+0000 7f854d000640 1 -- 172.21.15.27:0/3437966358 shutdown_connections 2024-01-15T18:05:01.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.020+0000 7f854d000640 1 -- 172.21.15.27:0/3437966358 wait complete. 2024-01-15T18:05:01.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.020+0000 7f854d000640 1 Processor -- start 2024-01-15T18:05:01.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.024+0000 7f854d000640 1 -- start start 2024-01-15T18:05:01.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.024+0000 7f854d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85480cf2e0 0x7f854806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:01.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.024+0000 7f854d000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8548158900 0x7f8548158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:01.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.024+0000 7f854d000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f854815bf00 0x7f854814b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:01.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.024+0000 7f854d000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8548136f20 con 0x7f85480cf2e0 2024-01-15T18:05:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.024+0000 7f854d000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85480af740 con 0x7f8548158900 2024-01-15T18:05:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.024+0000 7f854d000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85480aeb30 con 0x7f854815bf00 2024-01-15T18:05:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.024+0000 7f8545e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85480cf2e0 0x7f854806c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.024+0000 7f8545e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85480cf2e0 0x7f854806c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46564/0 (socket says 172.21.15.27:46564) 2024-01-15T18:05:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.024+0000 7f8545e00640 1 -- 172.21.15.27:0/2787852787 learned_addr learned my addr 172.21.15.27:0/2787852787 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.024+0000 7f8546800640 1 --2- 172.21.15.27:0/2787852787 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f854815bf00 0x7f854814b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.024+0000 7f8545400640 1 --2- 172.21.15.27:0/2787852787 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8548158900 0x7f8548158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.024+0000 7f8545e00640 1 -- 172.21.15.27:0/2787852787 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f854815bf00 msgr2=0x7f854814b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.024+0000 7f8545e00640 1 --2- 172.21.15.27:0/2787852787 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f854815bf00 0x7f854814b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.024+0000 7f8545e00640 1 -- 172.21.15.27:0/2787852787 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8548158900 msgr2=0x7f8548158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.024+0000 7f8545e00640 1 --2- 172.21.15.27:0/2787852787 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8548158900 0x7f8548158fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.024+0000 7f8545e00640 1 -- 172.21.15.27:0/2787852787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85380094d0 con 0x7f85480cf2e0 2024-01-15T18:05:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.024+0000 7f8545400640 1 --2- 172.21.15.27:0/2787852787 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8548158900 0x7f8548158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.024+0000 7f8545e00640 1 --2- 172.21.15.27:0/2787852787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85480cf2e0 0x7f854806c690 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f85380af040 tx=0x7f8538019260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.024+0000 7f853e000640 1 -- 172.21.15.27:0/2787852787 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85380b7470 con 0x7f85480cf2e0 2024-01-15T18:05:01.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.024+0000 7f853e000640 1 -- 172.21.15.27:0/2787852787 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85380b75d0 con 0x7f85480cf2e0 2024-01-15T18:05:01.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.024+0000 7f854d000640 1 -- 172.21.15.27:0/2787852787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85480aee10 con 0x7f85480cf2e0 2024-01-15T18:05:01.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.024+0000 7f853e000640 1 -- 172.21.15.27:0/2787852787 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85380b7960 con 0x7f85480cf2e0 2024-01-15T18:05:01.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.024+0000 7f854d000640 1 -- 172.21.15.27:0/2787852787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85480af3d0 con 0x7f85480cf2e0 2024-01-15T18:05:01.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.024+0000 7f854d000640 1 -- 172.21.15.27:0/2787852787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f854804e850 con 0x7f85480cf2e0 2024-01-15T18:05:01.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.028+0000 7f853e000640 1 -- 172.21.15.27:0/2787852787 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f85380b3070 con 0x7f85480cf2e0 2024-01-15T18:05:01.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.028+0000 7f853e000640 1 --2- 172.21.15.27:0/2787852787 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f85200668f0 0x7f8520068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:01.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.028+0000 7f853e000640 1 -- 172.21.15.27:0/2787852787 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f853811ead0 con 0x7f85480cf2e0 2024-01-15T18:05:01.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.028+0000 7f8545400640 1 --2- 172.21.15.27:0/2787852787 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f85200668f0 0x7f8520068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:01.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.028+0000 7f8545400640 1 --2- 172.21.15.27:0/2787852787 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f85200668f0 0x7f8520068db0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f854814cf30 tx=0x7f8530008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:01.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.032+0000 7f853e000640 1 -- 172.21.15.27:0/2787852787 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f85380ebd00 con 0x7f85480cf2e0 2024-01-15T18:05:01.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.224+0000 7f854d000640 1 -- 172.21.15.27:0/2787852787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f854806de00 con 0x7f85480cf2e0 2024-01-15T18:05:01.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.852+0000 7f853e000640 1 -- 172.21.15.27:0/2787852787 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f85380b10c0 con 0x7f85480cf2e0 2024-01-15T18:05:01.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.856+0000 7f854d000640 1 -- 172.21.15.27:0/2787852787 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f85200668f0 msgr2=0x7f8520068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:01.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.856+0000 7f854d000640 1 --2- 172.21.15.27:0/2787852787 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f85200668f0 0x7f8520068db0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f854814cf30 tx=0x7f8530008040 comp rx=0 tx=0).stop 2024-01-15T18:05:01.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.856+0000 7f854d000640 1 -- 172.21.15.27:0/2787852787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85480cf2e0 msgr2=0x7f854806c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:01.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.856+0000 7f854d000640 1 --2- 172.21.15.27:0/2787852787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85480cf2e0 0x7f854806c690 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f85380af040 tx=0x7f8538019260 comp rx=0 tx=0).stop 2024-01-15T18:05:01.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.856+0000 7f854d000640 1 -- 172.21.15.27:0/2787852787 shutdown_connections 2024-01-15T18:05:01.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.856+0000 7f854d000640 1 --2- 172.21.15.27:0/2787852787 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f85200668f0 0x7f8520068db0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:01.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.856+0000 7f854d000640 1 --2- 172.21.15.27:0/2787852787 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f854815bf00 0x7f854814b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:01.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.856+0000 7f854d000640 1 --2- 172.21.15.27:0/2787852787 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8548158900 0x7f8548158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:01.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.856+0000 7f854d000640 1 --2- 172.21.15.27:0/2787852787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85480cf2e0 0x7f854806c690 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:01.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.856+0000 7f854d000640 1 -- 172.21.15.27:0/2787852787 >> 172.21.15.27:0/2787852787 conn(0x7f85480cb700 msgr2=0x7f8548072080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:01.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.856+0000 7f854d000640 1 -- 172.21.15.27:0/2787852787 shutdown_connections 2024-01-15T18:05:01.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:01.856+0000 7f854d000640 1 -- 172.21.15.27:0/2787852787 wait complete. 2024-01-15T18:05:01.883 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-15T18:05:01.883 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:05:02.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.020+0000 7fb835800640 1 Processor -- start 2024-01-15T18:05:02.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.020+0000 7fb835800640 1 -- start start 2024-01-15T18:05:02.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.024+0000 7fb835800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb83006a4f0 0x7fb8300d0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:02.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.024+0000 7fb835800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fb8300ca3f0 0x7fb83006ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:02.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.024+0000 7fb835800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb830160bf0 con 0x7fb83006a8d0 2024-01-15T18:05:02.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.024+0000 7fb835800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8300ce9f0 con 0x7fb83006a4f0 2024-01-15T18:05:02.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.024+0000 7fb835800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb830161ab0 con 0x7fb8300ca3f0 2024-01-15T18:05:02.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.024+0000 7fb82ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb83006a4f0 0x7fb8300d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:02.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.024+0000 7fb82f400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb83006a8d0 0x7fb8300713f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53040/0 (socket says 172.21.15.27:53040) 2024-01-15T18:05:02.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.024+0000 7fb82f400640 1 -- 172.21.15.27:0/3598422544 learned_addr learned my addr 172.21.15.27:0/3598422544 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:02.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.024+0000 7fb82e000640 1 --2- 172.21.15.27:0/3598422544 >> v2:172.21.15.93:3300/0 conn(0x7fb8300ca3f0 0x7fb83006ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:02.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.024+0000 7fb82d600640 1 -- 172.21.15.27:0/3598422544 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727201196 0 0) 0x7fb8280014a0 con 0x7fb83006a8d0 2024-01-15T18:05:02.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.024+0000 7fb82d600640 1 -- 172.21.15.27:0/3598422544 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb810003530 con 0x7fb83006a8d0 2024-01-15T18:05:02.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.024+0000 7fb82ea00640 1 -- 172.21.15.27:0/3598422544 >> v2:172.21.15.93:3300/0 conn(0x7fb8300ca3f0 msgr2=0x7fb83006ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:02.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.024+0000 7fb82ea00640 1 --2- 172.21.15.27:0/3598422544 >> v2:172.21.15.93:3300/0 conn(0x7fb8300ca3f0 0x7fb83006ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.024+0000 7fb82ea00640 1 -- 172.21.15.27:0/3598422544 >> v1:172.21.15.27:6789/0 conn(0x7fb83006a8d0 legacy=0x7fb8300713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:02.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.024+0000 7fb82ea00640 1 -- 172.21.15.27:0/3598422544 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8300cc660 con 0x7fb83006a4f0 2024-01-15T18:05:02.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.024+0000 7fb82e000640 1 --2- 172.21.15.27:0/3598422544 >> v2:172.21.15.93:3300/0 conn(0x7fb8300ca3f0 0x7fb83006ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.028+0000 7fb82ea00640 1 --2- 172.21.15.27:0/3598422544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb83006a4f0 0x7fb8300d0930 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb8300cc7f0 tx=0x7fb82001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73cb8a24a0885b12 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.028+0000 7fb82d600640 1 -- 172.21.15.27:0/3598422544 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8200ab600 con 0x7fb83006a4f0 2024-01-15T18:05:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.028+0000 7fb82d600640 1 -- 172.21.15.27:0/3598422544 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8200a7050 con 0x7fb83006a4f0 2024-01-15T18:05:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.028+0000 7fb82d600640 1 -- 172.21.15.27:0/3598422544 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8200ab980 con 0x7fb83006a4f0 2024-01-15T18:05:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.028+0000 7fb835800640 1 -- 172.21.15.27:0/3598422544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb83006a4f0 msgr2=0x7fb8300d0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:02.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.028+0000 7fb835800640 1 --2- 172.21.15.27:0/3598422544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb83006a4f0 0x7fb8300d0930 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb8300cc7f0 tx=0x7fb82001c440 comp rx=0 tx=0).stop 2024-01-15T18:05:02.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.028+0000 7fb835800640 1 -- 172.21.15.27:0/3598422544 shutdown_connections 2024-01-15T18:05:02.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.028+0000 7fb835800640 1 --2- 172.21.15.27:0/3598422544 >> v2:172.21.15.93:3300/0 conn(0x7fb8300ca3f0 0x7fb83006ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.028+0000 7fb835800640 1 --2- 172.21.15.27:0/3598422544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb83006a4f0 0x7fb8300d0930 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.028+0000 7fb835800640 1 -- 172.21.15.27:0/3598422544 >> 172.21.15.27:0/3598422544 conn(0x7fb8300cfda0 msgr2=0x7fb8300d01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:02.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.028+0000 7fb835800640 1 -- 172.21.15.27:0/3598422544 shutdown_connections 2024-01-15T18:05:02.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.028+0000 7fb835800640 1 -- 172.21.15.27:0/3598422544 wait complete. 2024-01-15T18:05:02.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.028+0000 7fb835800640 1 Processor -- start 2024-01-15T18:05:02.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.028+0000 7fb835800640 1 -- start start 2024-01-15T18:05:02.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.032+0000 7fb835800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb83006a4f0 0x7fb8300ae4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:02.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.032+0000 7fb835800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb83006a8d0 0x7fb8300ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:02.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.032+0000 7fb835800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb8300ca3f0 0x7fb83014f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:02.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.032+0000 7fb835800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8300c3a80 con 0x7fb83006a8d0 2024-01-15T18:05:02.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.032+0000 7fb835800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8300c3450 con 0x7fb8300ca3f0 2024-01-15T18:05:02.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.032+0000 7fb835800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8300c92c0 con 0x7fb83006a4f0 2024-01-15T18:05:02.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.032+0000 7fb82e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb83006a8d0 0x7fb8300ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:02.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.032+0000 7fb82e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb83006a8d0 0x7fb8300ae9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46574/0 (socket says 172.21.15.27:46574) 2024-01-15T18:05:02.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.032+0000 7fb82e000640 1 -- 172.21.15.27:0/1424438748 learned_addr learned my addr 172.21.15.27:0/1424438748 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:02.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.032+0000 7fb82f400640 1 --2- 172.21.15.27:0/1424438748 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb8300ca3f0 0x7fb83014f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:02.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.032+0000 7fb82ea00640 1 --2- 172.21.15.27:0/1424438748 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb83006a4f0 0x7fb8300ae4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:02.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.032+0000 7fb82e000640 1 -- 172.21.15.27:0/1424438748 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb83006a4f0 msgr2=0x7fb8300ae4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:02.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.032+0000 7fb82e000640 1 --2- 172.21.15.27:0/1424438748 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb83006a4f0 0x7fb8300ae4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.032+0000 7fb82e000640 1 -- 172.21.15.27:0/1424438748 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb8300ca3f0 msgr2=0x7fb83014f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:02.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.032+0000 7fb82e000640 1 --2- 172.21.15.27:0/1424438748 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb8300ca3f0 0x7fb83014f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.032+0000 7fb82e000640 1 -- 172.21.15.27:0/1424438748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb820009420 con 0x7fb83006a8d0 2024-01-15T18:05:02.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.032+0000 7fb82e000640 1 --2- 172.21.15.27:0/1424438748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb83006a8d0 0x7fb8300ae9e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb81800e810 tx=0x7fb81800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:02.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.032+0000 7fb826a00640 1 -- 172.21.15.27:0/1424438748 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81800c8d0 con 0x7fb83006a8d0 2024-01-15T18:05:02.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.032+0000 7fb826a00640 1 -- 172.21.15.27:0/1424438748 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb81800ca30 con 0x7fb83006a8d0 2024-01-15T18:05:02.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.032+0000 7fb826a00640 1 -- 172.21.15.27:0/1424438748 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81800cdc0 con 0x7fb83006a8d0 2024-01-15T18:05:02.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.032+0000 7fb835800640 1 -- 172.21.15.27:0/1424438748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8300c95a0 con 0x7fb83006a8d0 2024-01-15T18:05:02.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.032+0000 7fb835800640 1 -- 172.21.15.27:0/1424438748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8300c8e50 con 0x7fb83006a8d0 2024-01-15T18:05:02.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.032+0000 7fb835800640 1 -- 172.21.15.27:0/1424438748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb83004e850 con 0x7fb83006a8d0 2024-01-15T18:05:02.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.036+0000 7fb826a00640 1 -- 172.21.15.27:0/1424438748 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fb818013410 con 0x7fb83006a8d0 2024-01-15T18:05:02.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.036+0000 7fb826a00640 1 --2- 172.21.15.27:0/1424438748 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb8080668f0 0x7fb808068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:02.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.036+0000 7fb826a00640 1 -- 172.21.15.27:0/1424438748 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb818084220 con 0x7fb83006a8d0 2024-01-15T18:05:02.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.036+0000 7fb82ea00640 1 --2- 172.21.15.27:0/1424438748 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb8080668f0 0x7fb808068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:02.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.036+0000 7fb82ea00640 1 --2- 172.21.15.27:0/1424438748 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb8080668f0 0x7fb808068db0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb8200ad000 tx=0x7fb820008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:02.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.040+0000 7fb826a00640 1 -- 172.21.15.27:0/1424438748 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb818051450 con 0x7fb83006a8d0 2024-01-15T18:05:02.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.228+0000 7fb835800640 1 -- 172.21.15.27:0/1424438748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb83006e9e0 con 0x7fb83006a8d0 2024-01-15T18:05:02.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.228+0000 7fb826a00640 1 -- 172.21.15.27:0/1424438748 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7fb818009480 con 0x7fb83006a8d0 2024-01-15T18:05:02.233 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:02.233 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/393360880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":393360880},{"type":"v1","addr":"172.21.15.93:6836","nonce":393360880}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4229,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4063308188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4063308188},{"type":"v1","addr":"172.21.15.27:6835","nonce":4063308188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4305,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/2343128056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":2343128056},{"type":"v1","addr":"172.21.15.93:6837","nonce":2343128056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:04:58.208165+0000","modified":"2024-01-15T18:05:01.225419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2657553192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2657553192},{"type":"v1","addr":"172.21.15.27:6834","nonce":2657553192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T18:05:02.233 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 8 2024-01-15T18:05:02.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.232+0000 7fb835800640 1 -- 172.21.15.27:0/1424438748 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb8080668f0 msgr2=0x7fb808068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:02.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.232+0000 7fb835800640 1 --2- 172.21.15.27:0/1424438748 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb8080668f0 0x7fb808068db0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb8200ad000 tx=0x7fb820008040 comp rx=0 tx=0).stop 2024-01-15T18:05:02.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.232+0000 7fb835800640 1 -- 172.21.15.27:0/1424438748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb83006a8d0 msgr2=0x7fb8300ae9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:02.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.232+0000 7fb835800640 1 --2- 172.21.15.27:0/1424438748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb83006a8d0 0x7fb8300ae9e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb81800e810 tx=0x7fb81800ecd0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.232+0000 7fb835800640 1 -- 172.21.15.27:0/1424438748 shutdown_connections 2024-01-15T18:05:02.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.232+0000 7fb835800640 1 --2- 172.21.15.27:0/1424438748 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb8080668f0 0x7fb808068db0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.232+0000 7fb835800640 1 --2- 172.21.15.27:0/1424438748 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb8300ca3f0 0x7fb83014f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.232+0000 7fb835800640 1 --2- 172.21.15.27:0/1424438748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb83006a8d0 0x7fb8300ae9e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.232+0000 7fb835800640 1 --2- 172.21.15.27:0/1424438748 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb83006a4f0 0x7fb8300ae4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.232+0000 7fb835800640 1 -- 172.21.15.27:0/1424438748 >> 172.21.15.27:0/1424438748 conn(0x7fb8300cfda0 msgr2=0x7fb8300cf180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:02.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.232+0000 7fb835800640 1 -- 172.21.15.27:0/1424438748 shutdown_connections 2024-01-15T18:05:02.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.232+0000 7fb835800640 1 -- 172.21.15.27:0/1424438748 wait complete. 2024-01-15T18:05:02.256 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:02.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.392+0000 7f3e3fc00640 1 Processor -- start 2024-01-15T18:05:02.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.392+0000 7f3e3fc00640 1 -- start start 2024-01-15T18:05:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.396+0000 7f3e3fc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3e3815d320 0x7f3e380d0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.396+0000 7f3e3fc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e380cdc30 0x7f3e380d1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.396+0000 7f3e3fc00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e380b2c90 con 0x7f3e380d2dd0 2024-01-15T18:05:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.396+0000 7f3e3fc00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e380b4130 con 0x7f3e3815d320 2024-01-15T18:05:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.396+0000 7f3e3fc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e380b29e0 con 0x7f3e380cdc30 2024-01-15T18:05:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.396+0000 7f3e3d200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3e3815d320 0x7f3e380d0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.396+0000 7f3e3dc00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3e380d2dd0 0x7f3e380d3280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53042/0 (socket says 172.21.15.27:53042) 2024-01-15T18:05:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.396+0000 7f3e3dc00640 1 -- 172.21.15.27:0/260457281 learned_addr learned my addr 172.21.15.27:0/260457281 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:02.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.396+0000 7f3e37e00640 1 --2- 172.21.15.27:0/260457281 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e380cdc30 0x7f3e380d1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:02.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.396+0000 7f3e3d200640 1 -- 172.21.15.27:0/260457281 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e380cdc30 msgr2=0x7f3e380d1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:02.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.396+0000 7f3e3d200640 1 --2- 172.21.15.27:0/260457281 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e380cdc30 0x7f3e380d1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.396+0000 7f3e3d200640 1 -- 172.21.15.27:0/260457281 >> v1:172.21.15.27:6789/0 conn(0x7f3e380d2dd0 legacy=0x7f3e380d3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:02.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.396+0000 7f3e37400640 1 -- 172.21.15.27:0/260457281 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424715836 0 0) 0x7f3e380b2c90 con 0x7f3e380d2dd0 2024-01-15T18:05:02.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.396+0000 7f3e3d200640 1 -- 172.21.15.27:0/260457281 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3815d700 con 0x7f3e3815d320 2024-01-15T18:05:02.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.396+0000 7f3e3d200640 1 --2- 172.21.15.27:0/260457281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e3815d320 0x7f3e380d0cf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3e280097c0 tx=0x7f3e2801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5705043038996761 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:02.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.400+0000 7f3e37400640 1 -- 172.21.15.27:0/260457281 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e280ab600 con 0x7f3e3815d320 2024-01-15T18:05:02.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.400+0000 7f3e37400640 1 -- 172.21.15.27:0/260457281 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e280a7050 con 0x7f3e3815d320 2024-01-15T18:05:02.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.400+0000 7f3e37400640 1 -- 172.21.15.27:0/260457281 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e280ab980 con 0x7f3e3815d320 2024-01-15T18:05:02.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.400+0000 7f3e3fc00640 1 -- 172.21.15.27:0/260457281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e3815d320 msgr2=0x7f3e380d0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:02.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.400+0000 7f3e3fc00640 1 --2- 172.21.15.27:0/260457281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e3815d320 0x7f3e380d0cf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3e280097c0 tx=0x7f3e2801c3f0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.400+0000 7f3e3fc00640 1 -- 172.21.15.27:0/260457281 shutdown_connections 2024-01-15T18:05:02.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.400+0000 7f3e3fc00640 1 --2- 172.21.15.27:0/260457281 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e380cdc30 0x7f3e380d1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.400+0000 7f3e3fc00640 1 --2- 172.21.15.27:0/260457281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e3815d320 0x7f3e380d0cf0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.400+0000 7f3e3fc00640 1 -- 172.21.15.27:0/260457281 >> 172.21.15.27:0/260457281 conn(0x7f3e3808cad0 msgr2=0x7f3e3809c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.400+0000 7f3e3fc00640 1 -- 172.21.15.27:0/260457281 shutdown_connections 2024-01-15T18:05:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.400+0000 7f3e3fc00640 1 -- 172.21.15.27:0/260457281 wait complete. 2024-01-15T18:05:02.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.400+0000 7f3e3fc00640 1 Processor -- start 2024-01-15T18:05:02.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.400+0000 7f3e3fc00640 1 -- start start 2024-01-15T18:05:02.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.404+0000 7f3e3fc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3e380cdc30 0x7f3e380b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:02.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.404+0000 7f3e3fc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e380d2dd0 0x7f3e380b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:02.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.404+0000 7f3e3fc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e3815d320 0x7f3e3815a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:02.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.404+0000 7f3e3fc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e380ca5b0 con 0x7f3e380d2dd0 2024-01-15T18:05:02.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.404+0000 7f3e3fc00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e380cb130 con 0x7f3e380cdc30 2024-01-15T18:05:02.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.404+0000 7f3e3fc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e380ca800 con 0x7f3e3815d320 2024-01-15T18:05:02.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.404+0000 7f3e37e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e380d2dd0 0x7f3e380b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:02.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.404+0000 7f3e37e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e380d2dd0 0x7f3e380b12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46600/0 (socket says 172.21.15.27:46600) 2024-01-15T18:05:02.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.404+0000 7f3e37e00640 1 -- 172.21.15.27:0/942337418 learned_addr learned my addr 172.21.15.27:0/942337418 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:02.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.404+0000 7f3e3d200640 1 --2- 172.21.15.27:0/942337418 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3e380cdc30 0x7f3e380b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:02.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.404+0000 7f3e3dc00640 1 --2- 172.21.15.27:0/942337418 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e3815d320 0x7f3e3815a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:02.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.404+0000 7f3e37e00640 1 -- 172.21.15.27:0/942337418 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e3815d320 msgr2=0x7f3e3815a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:02.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.404+0000 7f3e37e00640 1 --2- 172.21.15.27:0/942337418 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e3815d320 0x7f3e3815a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.404+0000 7f3e37e00640 1 -- 172.21.15.27:0/942337418 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3e380cdc30 msgr2=0x7f3e380b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:02.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.404+0000 7f3e37e00640 1 --2- 172.21.15.27:0/942337418 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3e380cdc30 0x7f3e380b1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.404+0000 7f3e37e00640 1 -- 172.21.15.27:0/942337418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e28009420 con 0x7f3e380d2dd0 2024-01-15T18:05:02.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.404+0000 7f3e37e00640 1 --2- 172.21.15.27:0/942337418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e380d2dd0 0x7f3e380b12e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3e2000d800 tx=0x7f3e2000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:02.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.404+0000 7f3e34c00640 1 -- 172.21.15.27:0/942337418 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2000b6c0 con 0x7f3e380d2dd0 2024-01-15T18:05:02.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.404+0000 7f3e34c00640 1 -- 172.21.15.27:0/942337418 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e2000b820 con 0x7f3e380d2dd0 2024-01-15T18:05:02.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.404+0000 7f3e34c00640 1 -- 172.21.15.27:0/942337418 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2000bb90 con 0x7f3e380d2dd0 2024-01-15T18:05:02.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.404+0000 7f3e3fc00640 1 -- 172.21.15.27:0/942337418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e380caae0 con 0x7f3e380d2dd0 2024-01-15T18:05:02.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.404+0000 7f3e3fc00640 1 -- 172.21.15.27:0/942337418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e380b24e0 con 0x7f3e380d2dd0 2024-01-15T18:05:02.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.404+0000 7f3e3fc00640 1 -- 172.21.15.27:0/942337418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e3804e850 con 0x7f3e380d2dd0 2024-01-15T18:05:02.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.404+0000 7f3e34c00640 1 -- 172.21.15.27:0/942337418 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f3e2000bcf0 con 0x7f3e380d2dd0 2024-01-15T18:05:02.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.408+0000 7f3e34c00640 1 --2- 172.21.15.27:0/942337418 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e100668f0 0x7f3e10068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:02.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.408+0000 7f3e34c00640 1 -- 172.21.15.27:0/942337418 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e20084920 con 0x7f3e380d2dd0 2024-01-15T18:05:02.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.408+0000 7f3e3d200640 1 --2- 172.21.15.27:0/942337418 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e100668f0 0x7f3e10068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:02.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.408+0000 7f3e3d200640 1 --2- 172.21.15.27:0/942337418 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e100668f0 0x7f3e10068db0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3e280ad000 tx=0x7f3e28008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:02.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.412+0000 7f3e34c00640 1 -- 172.21.15.27:0/942337418 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3e20051af0 con 0x7f3e380d2dd0 2024-01-15T18:05:02.501 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-01-15T18:05:02.496+0000 7fcf00200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-15T18:05:02.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.572+0000 7f3e3fc00640 1 -- 172.21.15.27:0/942337418 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e38167b60 con 0x7f3e380d2dd0 2024-01-15T18:05:02.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.572+0000 7f3e34c00640 1 -- 172.21.15.27:0/942337418 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16409 (secure 0 0 0) 0x7f3e20055110 con 0x7f3e380d2dd0 2024-01-15T18:05:02.576 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:02.576 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:04:59.837052+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:04:56.220608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T18:04:57.210421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:02.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.572+0000 7f3e3fc00640 1 -- 172.21.15.27:0/942337418 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e100668f0 msgr2=0x7f3e10068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:02.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.572+0000 7f3e3fc00640 1 --2- 172.21.15.27:0/942337418 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e100668f0 0x7f3e10068db0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3e280ad000 tx=0x7f3e28008040 comp rx=0 tx=0).stop 2024-01-15T18:05:02.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.572+0000 7f3e3fc00640 1 -- 172.21.15.27:0/942337418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e380d2dd0 msgr2=0x7f3e380b12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:02.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.572+0000 7f3e3fc00640 1 --2- 172.21.15.27:0/942337418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e380d2dd0 0x7f3e380b12e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3e2000d800 tx=0x7f3e2000dcc0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.572+0000 7f3e3fc00640 1 -- 172.21.15.27:0/942337418 shutdown_connections 2024-01-15T18:05:02.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.572+0000 7f3e3fc00640 1 --2- 172.21.15.27:0/942337418 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e100668f0 0x7f3e10068db0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.572+0000 7f3e3fc00640 1 --2- 172.21.15.27:0/942337418 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e3815d320 0x7f3e3815a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.572+0000 7f3e3fc00640 1 --2- 172.21.15.27:0/942337418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e380d2dd0 0x7f3e380b12e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.572+0000 7f3e3fc00640 1 --2- 172.21.15.27:0/942337418 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3e380cdc30 0x7f3e380b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.572+0000 7f3e3fc00640 1 -- 172.21.15.27:0/942337418 >> 172.21.15.27:0/942337418 conn(0x7f3e3808cad0 msgr2=0x7f3e3806c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:02.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.572+0000 7f3e3fc00640 1 -- 172.21.15.27:0/942337418 shutdown_connections 2024-01-15T18:05:02.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.572+0000 7f3e3fc00640 1 -- 172.21.15.27:0/942337418 wait complete. 2024-01-15T18:05:02.598 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-15T18:05:02.598 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:05:02.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.736+0000 7fceb2200640 1 Processor -- start 2024-01-15T18:05:02.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.736+0000 7fceb2200640 1 -- start start 2024-01-15T18:05:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.736+0000 7fceb2200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fceac15e730 0x7fceac0cb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.736+0000 7fceb2200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fceac0aea00 0x7fceac06af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.736+0000 7fceb2200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceac06e450 con 0x7fceac06b440 2024-01-15T18:05:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.736+0000 7fceb2200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceac06e980 con 0x7fceac15e730 2024-01-15T18:05:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.736+0000 7fceb2200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceac0c9bf0 con 0x7fceac0aea00 2024-01-15T18:05:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.736+0000 7fceabe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fceac06b440 0x7fceac15eb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53052/0 (socket says 172.21.15.27:53052) 2024-01-15T18:05:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.736+0000 7fceabe00640 1 -- 172.21.15.27:0/460802278 learned_addr learned my addr 172.21.15.27:0/460802278 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.736+0000 7fceab400640 1 --2- 172.21.15.27:0/460802278 >> v2:172.21.15.93:3300/0 conn(0x7fceac15e730 0x7fceac0cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.736+0000 7fceaaa00640 1 --2- 172.21.15.27:0/460802278 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fceac0aea00 0x7fceac06af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.736+0000 7fceaa000640 1 -- 172.21.15.27:0/460802278 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685167974 0 0) 0x7fceac06e450 con 0x7fceac06b440 2024-01-15T18:05:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.736+0000 7fceaa000640 1 -- 172.21.15.27:0/460802278 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce8c003530 con 0x7fceac06b440 2024-01-15T18:05:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.736+0000 7fceaaa00640 1 -- 172.21.15.27:0/460802278 >> v2:172.21.15.93:3300/0 conn(0x7fceac15e730 msgr2=0x7fceac0cb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:02.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.736+0000 7fceaaa00640 1 --2- 172.21.15.27:0/460802278 >> v2:172.21.15.93:3300/0 conn(0x7fceac15e730 0x7fceac0cb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.736+0000 7fceaaa00640 1 -- 172.21.15.27:0/460802278 >> v1:172.21.15.27:6789/0 conn(0x7fceac06b440 legacy=0x7fceac15eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:02.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.736+0000 7fceaaa00640 1 -- 172.21.15.27:0/460802278 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceac0c9ed0 con 0x7fceac0aea00 2024-01-15T18:05:02.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.740+0000 7fceaaa00640 1 --2- 172.21.15.27:0/460802278 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fceac0aea00 0x7fceac06af00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fce940097c0 tx=0x7fce9401c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e685009d31c642f9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:02.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.740+0000 7fceaa000640 1 -- 172.21.15.27:0/460802278 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce940ab600 con 0x7fceac0aea00 2024-01-15T18:05:02.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.740+0000 7fceaa000640 1 -- 172.21.15.27:0/460802278 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce940a7050 con 0x7fceac0aea00 2024-01-15T18:05:02.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.740+0000 7fceaa000640 1 -- 172.21.15.27:0/460802278 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce940ab980 con 0x7fceac0aea00 2024-01-15T18:05:02.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.740+0000 7fceb2200640 1 -- 172.21.15.27:0/460802278 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fceac0aea00 msgr2=0x7fceac06af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:02.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.740+0000 7fceb2200640 1 --2- 172.21.15.27:0/460802278 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fceac0aea00 0x7fceac06af00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fce940097c0 tx=0x7fce9401c330 comp rx=0 tx=0).stop 2024-01-15T18:05:02.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.740+0000 7fceb2200640 1 -- 172.21.15.27:0/460802278 shutdown_connections 2024-01-15T18:05:02.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.740+0000 7fceb2200640 1 --2- 172.21.15.27:0/460802278 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fceac0aea00 0x7fceac06af00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.740+0000 7fceb2200640 1 --2- 172.21.15.27:0/460802278 >> v2:172.21.15.93:3300/0 conn(0x7fceac15e730 0x7fceac0cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.740+0000 7fceb2200640 1 -- 172.21.15.27:0/460802278 >> 172.21.15.27:0/460802278 conn(0x7fceac0cbf80 msgr2=0x7fceac0cc360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.740+0000 7fceb2200640 1 -- 172.21.15.27:0/460802278 shutdown_connections 2024-01-15T18:05:02.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.740+0000 7fceb2200640 1 -- 172.21.15.27:0/460802278 wait complete. 2024-01-15T18:05:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.744+0000 7fceb2200640 1 Processor -- start 2024-01-15T18:05:02.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.744+0000 7fceb2200640 1 -- start start 2024-01-15T18:05:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.744+0000 7fceb2200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fceac06b440 0x7fceac151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.744+0000 7fceb2200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fceac0aea00 0x7fceac159b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:02.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.744+0000 7fceb2200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceac15e730 0x7fceac14dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.744+0000 7fceb2200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceac14c240 con 0x7fceac15e730 2024-01-15T18:05:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.744+0000 7fceb2200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceac15d910 con 0x7fceac0aea00 2024-01-15T18:05:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.744+0000 7fceb2200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceac15dbf0 con 0x7fceac06b440 2024-01-15T18:05:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.744+0000 7fceabe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceac15e730 0x7fceac14dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.744+0000 7fceabe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceac15e730 0x7fceac14dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46606/0 (socket says 172.21.15.27:46606) 2024-01-15T18:05:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.744+0000 7fceabe00640 1 -- 172.21.15.27:0/4109265986 learned_addr learned my addr 172.21.15.27:0/4109265986 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.744+0000 7fceab400640 1 --2- 172.21.15.27:0/4109265986 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fceac06b440 0x7fceac151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.744+0000 7fceaaa00640 1 --2- 172.21.15.27:0/4109265986 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fceac0aea00 0x7fceac159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.744+0000 7fceabe00640 1 -- 172.21.15.27:0/4109265986 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fceac06b440 msgr2=0x7fceac151cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.744+0000 7fceabe00640 1 --2- 172.21.15.27:0/4109265986 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fceac06b440 0x7fceac151cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.744+0000 7fceabe00640 1 -- 172.21.15.27:0/4109265986 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fceac0aea00 msgr2=0x7fceac159b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.744+0000 7fceabe00640 1 --2- 172.21.15.27:0/4109265986 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fceac0aea00 0x7fceac159b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.744+0000 7fceabe00640 1 -- 172.21.15.27:0/4109265986 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce94009420 con 0x7fceac15e730 2024-01-15T18:05:02.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.744+0000 7fceabe00640 1 --2- 172.21.15.27:0/4109265986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceac15e730 0x7fceac14dce0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcea400caa0 tx=0x7fcea400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:02.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.744+0000 7fcea3400640 1 -- 172.21.15.27:0/4109265986 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea400ac00 con 0x7fceac15e730 2024-01-15T18:05:02.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.744+0000 7fcea3400640 1 -- 172.21.15.27:0/4109265986 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea400ad60 con 0x7fceac15e730 2024-01-15T18:05:02.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.744+0000 7fceb2200640 1 -- 172.21.15.27:0/4109265986 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceac14c4b0 con 0x7fceac15e730 2024-01-15T18:05:02.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.744+0000 7fcea3400640 1 -- 172.21.15.27:0/4109265986 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea4012620 con 0x7fceac15e730 2024-01-15T18:05:02.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.744+0000 7fceb2200640 1 -- 172.21.15.27:0/4109265986 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceac15ded0 con 0x7fceac15e730 2024-01-15T18:05:02.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.748+0000 7fceb2200640 1 -- 172.21.15.27:0/4109265986 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceac14c6f0 con 0x7fceac15e730 2024-01-15T18:05:02.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.748+0000 7fcea3400640 1 -- 172.21.15.27:0/4109265986 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fcea4007070 con 0x7fceac15e730 2024-01-15T18:05:02.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.748+0000 7fcea3400640 1 --2- 172.21.15.27:0/4109265986 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce840668f0 0x7fce84068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:02.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.748+0000 7fcea3400640 1 -- 172.21.15.27:0/4109265986 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcea40843f0 con 0x7fceac15e730 2024-01-15T18:05:02.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.748+0000 7fceab400640 1 --2- 172.21.15.27:0/4109265986 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce840668f0 0x7fce84068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:02.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.748+0000 7fceab400640 1 --2- 172.21.15.27:0/4109265986 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce840668f0 0x7fce84068db0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fce9c009b60 tx=0x7fce9c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:02.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.752+0000 7fcea3400640 1 -- 172.21.15.27:0/4109265986 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fceac14c6f0 con 0x7fceac15e730 2024-01-15T18:05:02.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.908+0000 7fceb2200640 1 -- 172.21.15.27:0/4109265986 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fceac0714c0 con 0x7fce840668f0 2024-01-15T18:05:02.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.916+0000 7fcea3400640 1 -- 172.21.15.27:0/4109265986 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264648 (secure 0 0 0) 0x7fceac0714c0 con 0x7fce840668f0 2024-01-15T18:05:02.923 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:02.925 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:05:02.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.920+0000 7fceb2200640 1 -- 172.21.15.27:0/4109265986 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce840668f0 msgr2=0x7fce84068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:02.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.920+0000 7fceb2200640 1 --2- 172.21.15.27:0/4109265986 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce840668f0 0x7fce84068db0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fce9c009b60 tx=0x7fce9c009190 comp rx=0 tx=0).stop 2024-01-15T18:05:02.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.920+0000 7fceb2200640 1 -- 172.21.15.27:0/4109265986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceac15e730 msgr2=0x7fceac14dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:02.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.920+0000 7fceb2200640 1 --2- 172.21.15.27:0/4109265986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceac15e730 0x7fceac14dce0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcea400caa0 tx=0x7fcea400cf60 comp rx=0 tx=0).stop 2024-01-15T18:05:02.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.920+0000 7fceb2200640 1 -- 172.21.15.27:0/4109265986 shutdown_connections 2024-01-15T18:05:02.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.920+0000 7fceb2200640 1 --2- 172.21.15.27:0/4109265986 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce840668f0 0x7fce84068db0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.920+0000 7fceb2200640 1 --2- 172.21.15.27:0/4109265986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fceac15e730 0x7fceac14dce0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.920+0000 7fceb2200640 1 --2- 172.21.15.27:0/4109265986 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fceac0aea00 0x7fceac159b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.920+0000 7fceb2200640 1 --2- 172.21.15.27:0/4109265986 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fceac06b440 0x7fceac151cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.920+0000 7fceb2200640 1 -- 172.21.15.27:0/4109265986 >> 172.21.15.27:0/4109265986 conn(0x7fceac0cbf80 msgr2=0x7fceac070ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:02.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.920+0000 7fceb2200640 1 -- 172.21.15.27:0/4109265986 shutdown_connections 2024-01-15T18:05:02.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:02.920+0000 7fceb2200640 1 -- 172.21.15.27:0/4109265986 wait complete. 2024-01-15T18:05:02.944 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-15T18:05:02.139338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":121,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218952,"kb_used_data":2780,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755768,"statfs":{"total":773094113280,"available":772869906432,"internally_reserved":0,"allocated":2846720,"data_stored":1724431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.329975"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032071+0000","last_change":"2024-01-15T18:04:58.862012+0000","last_active":"2024-01-15T18:05:00.032071+0000","last_peered":"2024-01-15T18:05:00.032071+0000","last_clean":"2024-01-15T18:05:00.032071+0000","last_became_active":"2024-01-15T18:04:58.861713+0000","last_became_peered":"2024-01-15T18:04:58.861713+0000","last_unstale":"2024-01-15T18:05:00.032071+0000","last_undegraded":"2024-01-15T18:05:00.032071+0000","last_fullsized":"2024-01-15T18:05:00.032071+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:30:29.323987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849437+0000","last_change":"2024-01-15T18:04:58.870910+0000","last_active":"2024-01-15T18:04:59.849437+0000","last_peered":"2024-01-15T18:04:59.849437+0000","last_clean":"2024-01-15T18:04:59.849437+0000","last_became_active":"2024-01-15T18:04:58.869957+0000","last_became_peered":"2024-01-15T18:04:58.869957+0000","last_unstale":"2024-01-15T18:04:59.849437+0000","last_undegraded":"2024-01-15T18:04:59.849437+0000","last_fullsized":"2024-01-15T18:04:59.849437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:10:24.519754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.862393+0000","last_change":"2024-01-15T18:04:58.854305+0000","last_active":"2024-01-15T18:04:58.862393+0000","last_peered":"2024-01-15T18:04:58.862393+0000","last_clean":"2024-01-15T18:04:58.862393+0000","last_became_active":"2024-01-15T18:04:58.853375+0000","last_became_peered":"2024-01-15T18:04:58.853375+0000","last_unstale":"2024-01-15T18:04:58.862393+0000","last_undegraded":"2024-01-15T18:04:58.862393+0000","last_fullsized":"2024-01-15T18:04:58.862393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:36:10.997513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.852061+0000","last_change":"2024-01-15T18:04:58.844639+0000","last_active":"2024-01-15T18:04:58.852061+0000","last_peered":"2024-01-15T18:04:58.852061+0000","last_clean":"2024-01-15T18:04:58.852061+0000","last_became_active":"2024-01-15T18:04:58.844174+0000","last_became_peered":"2024-01-15T18:04:58.844174+0000","last_unstale":"2024-01-15T18:04:58.852061+0000","last_undegraded":"2024-01-15T18:04:58.852061+0000","last_fullsized":"2024-01-15T18:04:58.852061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:29:02.731583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031678+0000","last_change":"2024-01-15T18:04:58.852004+0000","last_active":"2024-01-15T18:05:00.031678+0000","last_peered":"2024-01-15T18:05:00.031678+0000","last_clean":"2024-01-15T18:05:00.031678+0000","last_became_active":"2024-01-15T18:04:58.849191+0000","last_became_peered":"2024-01-15T18:04:58.849191+0000","last_unstale":"2024-01-15T18:05:00.031678+0000","last_undegraded":"2024-01-15T18:05:00.031678+0000","last_fullsized":"2024-01-15T18:05:00.031678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:22:03.084714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.863975+0000","last_change":"2024-01-15T18:04:58.854461+0000","last_active":"2024-01-15T18:04:58.863975+0000","last_peered":"2024-01-15T18:04:58.863975+0000","last_clean":"2024-01-15T18:04:58.863975+0000","last_became_active":"2024-01-15T18:04:58.854170+0000","last_became_peered":"2024-01-15T18:04:58.854170+0000","last_unstale":"2024-01-15T18:04:58.863975+0000","last_undegraded":"2024-01-15T18:04:58.863975+0000","last_fullsized":"2024-01-15T18:04:58.863975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:21:44.926560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031425+0000","last_change":"2024-01-15T18:04:58.857161+0000","last_active":"2024-01-15T18:05:00.031425+0000","last_peered":"2024-01-15T18:05:00.031425+0000","last_clean":"2024-01-15T18:05:00.031425+0000","last_became_active":"2024-01-15T18:04:58.856839+0000","last_became_peered":"2024-01-15T18:04:58.856839+0000","last_unstale":"2024-01-15T18:05:00.031425+0000","last_undegraded":"2024-01-15T18:05:00.031425+0000","last_fullsized":"2024-01-15T18:05:00.031425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:53:23.471036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031831+0000","last_change":"2024-01-15T18:04:58.861226+0000","last_active":"2024-01-15T18:05:00.031831+0000","last_peered":"2024-01-15T18:05:00.031831+0000","last_clean":"2024-01-15T18:05:00.031831+0000","last_became_active":"2024-01-15T18:04:58.860964+0000","last_became_peered":"2024-01-15T18:04:58.860964+0000","last_unstale":"2024-01-15T18:05:00.031831+0000","last_undegraded":"2024-01-15T18:05:00.031831+0000","last_fullsized":"2024-01-15T18:05:00.031831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:11:49.903930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032115+0000","last_change":"2024-01-15T18:04:58.856674+0000","last_active":"2024-01-15T18:05:00.032115+0000","last_peered":"2024-01-15T18:05:00.032115+0000","last_clean":"2024-01-15T18:05:00.032115+0000","last_became_active":"2024-01-15T18:04:58.856365+0000","last_became_peered":"2024-01-15T18:04:58.856365+0000","last_unstale":"2024-01-15T18:05:00.032115+0000","last_undegraded":"2024-01-15T18:05:00.032115+0000","last_fullsized":"2024-01-15T18:05:00.032115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:09:03.416070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030934+0000","last_change":"2024-01-15T18:04:58.853185+0000","last_active":"2024-01-15T18:05:00.030934+0000","last_peered":"2024-01-15T18:05:00.030934+0000","last_clean":"2024-01-15T18:05:00.030934+0000","last_became_active":"2024-01-15T18:04:58.852888+0000","last_became_peered":"2024-01-15T18:04:58.852888+0000","last_unstale":"2024-01-15T18:05:00.030934+0000","last_undegraded":"2024-01-15T18:05:00.030934+0000","last_fullsized":"2024-01-15T18:05:00.030934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:10:24.519754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030810+0000","last_change":"2024-01-15T18:04:58.848009+0000","last_active":"2024-01-15T18:05:00.030810+0000","last_peered":"2024-01-15T18:05:00.030810+0000","last_clean":"2024-01-15T18:05:00.030810+0000","last_became_active":"2024-01-15T18:04:58.847765+0000","last_became_peered":"2024-01-15T18:04:58.847765+0000","last_unstale":"2024-01-15T18:05:00.030810+0000","last_undegraded":"2024-01-15T18:05:00.030810+0000","last_fullsized":"2024-01-15T18:05:00.030810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:15:34.376159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847406+0000","last_change":"2024-01-15T18:04:58.847259+0000","last_active":"2024-01-15T18:04:59.847406+0000","last_peered":"2024-01-15T18:04:59.847406+0000","last_clean":"2024-01-15T18:04:59.847406+0000","last_became_active":"2024-01-15T18:04:58.845619+0000","last_became_peered":"2024-01-15T18:04:58.845619+0000","last_unstale":"2024-01-15T18:04:59.847406+0000","last_undegraded":"2024-01-15T18:04:59.847406+0000","last_fullsized":"2024-01-15T18:04:59.847406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:16:30.446952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032165+0000","last_change":"2024-01-15T18:04:58.842389+0000","last_active":"2024-01-15T18:05:00.032165+0000","last_peered":"2024-01-15T18:05:00.032165+0000","last_clean":"2024-01-15T18:05:00.032165+0000","last_became_active":"2024-01-15T18:04:58.841378+0000","last_became_peered":"2024-01-15T18:04:58.841378+0000","last_unstale":"2024-01-15T18:05:00.032165+0000","last_undegraded":"2024-01-15T18:05:00.032165+0000","last_fullsized":"2024-01-15T18:05:00.032165+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:36.680373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.868487+0000","last_change":"2024-01-15T18:04:58.864203+0000","last_active":"2024-01-15T18:04:58.868487+0000","last_peered":"2024-01-15T18:04:58.868487+0000","last_clean":"2024-01-15T18:04:58.868487+0000","last_became_active":"2024-01-15T18:04:58.863956+0000","last_became_peered":"2024-01-15T18:04:58.863956+0000","last_unstale":"2024-01-15T18:04:58.868487+0000","last_undegraded":"2024-01-15T18:04:58.868487+0000","last_fullsized":"2024-01-15T18:04:58.868487+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:02:33.077446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849113+0000","last_change":"2024-01-15T18:04:58.836384+0000","last_active":"2024-01-15T18:04:59.849113+0000","last_peered":"2024-01-15T18:04:59.849113+0000","last_clean":"2024-01-15T18:04:59.849113+0000","last_became_active":"2024-01-15T18:04:58.836081+0000","last_became_peered":"2024-01-15T18:04:58.836081+0000","last_unstale":"2024-01-15T18:04:59.849113+0000","last_undegraded":"2024-01-15T18:04:59.849113+0000","last_fullsized":"2024-01-15T18:04:59.849113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:15:34.376159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.870295+0000","last_change":"2024-01-15T18:04:58.868380+0000","last_active":"2024-01-15T18:04:58.870295+0000","last_peered":"2024-01-15T18:04:58.870295+0000","last_clean":"2024-01-15T18:04:58.870295+0000","last_became_active":"2024-01-15T18:04:58.868103+0000","last_became_peered":"2024-01-15T18:04:58.868103+0000","last_unstale":"2024-01-15T18:04:58.870295+0000","last_undegraded":"2024-01-15T18:04:58.870295+0000","last_fullsized":"2024-01-15T18:04:58.870295+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:11.967136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.865954+0000","last_change":"2024-01-15T18:04:58.858404+0000","last_active":"2024-01-15T18:04:58.865954+0000","last_peered":"2024-01-15T18:04:58.865954+0000","last_clean":"2024-01-15T18:04:58.865954+0000","last_became_active":"2024-01-15T18:04:58.857755+0000","last_became_peered":"2024-01-15T18:04:58.857755+0000","last_unstale":"2024-01-15T18:04:58.865954+0000","last_undegraded":"2024-01-15T18:04:58.865954+0000","last_fullsized":"2024-01-15T18:04:58.865954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:58:42.218891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030907+0000","last_change":"2024-01-15T18:04:58.844392+0000","last_active":"2024-01-15T18:05:00.030907+0000","last_peered":"2024-01-15T18:05:00.030907+0000","last_clean":"2024-01-15T18:05:00.030907+0000","last_became_active":"2024-01-15T18:04:58.844086+0000","last_became_peered":"2024-01-15T18:04:58.844086+0000","last_unstale":"2024-01-15T18:05:00.030907+0000","last_undegraded":"2024-01-15T18:05:00.030907+0000","last_fullsized":"2024-01-15T18:05:00.030907+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:16:30.446952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031976+0000","last_change":"2024-01-15T18:04:58.861264+0000","last_active":"2024-01-15T18:05:00.031976+0000","last_peered":"2024-01-15T18:05:00.031976+0000","last_clean":"2024-01-15T18:05:00.031976+0000","last_became_active":"2024-01-15T18:04:58.860695+0000","last_became_peered":"2024-01-15T18:04:58.860695+0000","last_unstale":"2024-01-15T18:05:00.031976+0000","last_undegraded":"2024-01-15T18:05:00.031976+0000","last_fullsized":"2024-01-15T18:05:00.031976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:47:51.247174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030783+0000","last_change":"2024-01-15T18:04:58.846632+0000","last_active":"2024-01-15T18:05:00.030783+0000","last_peered":"2024-01-15T18:05:00.030783+0000","last_clean":"2024-01-15T18:05:00.030783+0000","last_became_active":"2024-01-15T18:04:58.846359+0000","last_became_peered":"2024-01-15T18:04:58.846359+0000","last_unstale":"2024-01-15T18:05:00.030783+0000","last_undegraded":"2024-01-15T18:05:00.030783+0000","last_fullsized":"2024-01-15T18:05:00.030783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:12:00.696642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.868985+0000","last_change":"2024-01-15T18:04:58.858817+0000","last_active":"2024-01-15T18:04:58.868985+0000","last_peered":"2024-01-15T18:04:58.868985+0000","last_clean":"2024-01-15T18:04:58.868985+0000","last_became_active":"2024-01-15T18:04:58.858539+0000","last_became_peered":"2024-01-15T18:04:58.858539+0000","last_unstale":"2024-01-15T18:04:58.868985+0000","last_undegraded":"2024-01-15T18:04:58.868985+0000","last_fullsized":"2024-01-15T18:04:58.868985+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:16:02.098975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.870284+0000","last_change":"2024-01-15T18:04:58.867761+0000","last_active":"2024-01-15T18:04:58.870284+0000","last_peered":"2024-01-15T18:04:58.870284+0000","last_clean":"2024-01-15T18:04:58.870284+0000","last_became_active":"2024-01-15T18:04:58.867331+0000","last_became_peered":"2024-01-15T18:04:58.867331+0000","last_unstale":"2024-01-15T18:04:58.870284+0000","last_undegraded":"2024-01-15T18:04:58.870284+0000","last_fullsized":"2024-01-15T18:04:58.870284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:14:59.421400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031915+0000","last_change":"2024-01-15T18:04:58.839963+0000","last_active":"2024-01-15T18:05:00.031915+0000","last_peered":"2024-01-15T18:05:00.031915+0000","last_clean":"2024-01-15T18:05:00.031915+0000","last_became_active":"2024-01-15T18:04:58.839597+0000","last_became_peered":"2024-01-15T18:04:58.839597+0000","last_unstale":"2024-01-15T18:05:00.031915+0000","last_undegraded":"2024-01-15T18:05:00.031915+0000","last_fullsized":"2024-01-15T18:05:00.031915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:47.713249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.847369+0000","last_change":"2024-01-15T18:04:58.841274+0000","last_active":"2024-01-15T18:04:58.847369+0000","last_peered":"2024-01-15T18:04:58.847369+0000","last_clean":"2024-01-15T18:04:58.847369+0000","last_became_active":"2024-01-15T18:04:58.840866+0000","last_became_peered":"2024-01-15T18:04:58.840866+0000","last_unstale":"2024-01-15T18:04:58.847369+0000","last_undegraded":"2024-01-15T18:04:58.847369+0000","last_fullsized":"2024-01-15T18:04:58.847369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:11.539894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032528+0000","last_change":"2024-01-15T18:04:58.855334+0000","last_active":"2024-01-15T18:05:00.032528+0000","last_peered":"2024-01-15T18:05:00.032528+0000","last_clean":"2024-01-15T18:05:00.032528+0000","last_became_active":"2024-01-15T18:04:58.855074+0000","last_became_peered":"2024-01-15T18:04:58.855074+0000","last_unstale":"2024-01-15T18:05:00.032528+0000","last_undegraded":"2024-01-15T18:05:00.032528+0000","last_fullsized":"2024-01-15T18:05:00.032528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:44:17.933005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031069+0000","last_change":"2024-01-15T18:04:58.861125+0000","last_active":"2024-01-15T18:05:00.031069+0000","last_peered":"2024-01-15T18:05:00.031069+0000","last_clean":"2024-01-15T18:05:00.031069+0000","last_became_active":"2024-01-15T18:04:58.860819+0000","last_became_peered":"2024-01-15T18:04:58.860819+0000","last_unstale":"2024-01-15T18:05:00.031069+0000","last_undegraded":"2024-01-15T18:05:00.031069+0000","last_fullsized":"2024-01-15T18:05:00.031069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:04:27.769262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849316+0000","last_change":"2024-01-15T18:04:58.837356+0000","last_active":"2024-01-15T18:04:59.849316+0000","last_peered":"2024-01-15T18:04:59.849316+0000","last_clean":"2024-01-15T18:04:59.849316+0000","last_became_active":"2024-01-15T18:04:58.837071+0000","last_became_peered":"2024-01-15T18:04:58.837071+0000","last_unstale":"2024-01-15T18:04:59.849316+0000","last_undegraded":"2024-01-15T18:04:59.849316+0000","last_fullsized":"2024-01-15T18:04:59.849316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:36.680373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032362+0000","last_change":"2024-01-15T18:04:58.858404+0000","last_active":"2024-01-15T18:05:00.032362+0000","last_peered":"2024-01-15T18:05:00.032362+0000","last_clean":"2024-01-15T18:05:00.032362+0000","last_became_active":"2024-01-15T18:04:58.857239+0000","last_became_peered":"2024-01-15T18:04:58.857239+0000","last_unstale":"2024-01-15T18:05:00.032362+0000","last_undegraded":"2024-01-15T18:05:00.032362+0000","last_fullsized":"2024-01-15T18:05:00.032362+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:04:27.769262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.869022+0000","last_change":"2024-01-15T18:04:58.861031+0000","last_active":"2024-01-15T18:04:58.869022+0000","last_peered":"2024-01-15T18:04:58.869022+0000","last_clean":"2024-01-15T18:04:58.869022+0000","last_became_active":"2024-01-15T18:04:58.860718+0000","last_became_peered":"2024-01-15T18:04:58.860718+0000","last_unstale":"2024-01-15T18:04:58.869022+0000","last_undegraded":"2024-01-15T18:04:58.869022+0000","last_fullsized":"2024-01-15T18:04:58.869022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:35:59.532764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.847378+0000","last_change":"2024-01-15T18:04:58.841212+0000","last_active":"2024-01-15T18:04:58.847378+0000","last_peered":"2024-01-15T18:04:58.847378+0000","last_clean":"2024-01-15T18:04:58.847378+0000","last_became_active":"2024-01-15T18:04:58.840913+0000","last_became_peered":"2024-01-15T18:04:58.840913+0000","last_unstale":"2024-01-15T18:04:58.847378+0000","last_undegraded":"2024-01-15T18:04:58.847378+0000","last_fullsized":"2024-01-15T18:04:58.847378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:25:34.152008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030927+0000","last_change":"2024-01-15T18:04:58.847922+0000","last_active":"2024-01-15T18:05:00.030927+0000","last_peered":"2024-01-15T18:05:00.030927+0000","last_clean":"2024-01-15T18:05:00.030927+0000","last_became_active":"2024-01-15T18:04:58.847596+0000","last_became_peered":"2024-01-15T18:04:58.847596+0000","last_unstale":"2024-01-15T18:05:00.030927+0000","last_undegraded":"2024-01-15T18:05:00.030927+0000","last_fullsized":"2024-01-15T18:05:00.030927+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:24:34.638590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030579+0000","last_change":"2024-01-15T18:04:58.866119+0000","last_active":"2024-01-15T18:05:00.030579+0000","last_peered":"2024-01-15T18:05:00.030579+0000","last_clean":"2024-01-15T18:05:00.030579+0000","last_became_active":"2024-01-15T18:04:58.865811+0000","last_became_peered":"2024-01-15T18:04:58.865811+0000","last_unstale":"2024-01-15T18:05:00.030579+0000","last_undegraded":"2024-01-15T18:05:00.030579+0000","last_fullsized":"2024-01-15T18:05:00.030579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:10.719289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847206+0000","last_change":"2024-01-15T18:04:58.839281+0000","last_active":"2024-01-15T18:04:59.847206+0000","last_peered":"2024-01-15T18:04:59.847206+0000","last_clean":"2024-01-15T18:04:59.847206+0000","last_became_active":"2024-01-15T18:04:58.838304+0000","last_became_peered":"2024-01-15T18:04:58.838304+0000","last_unstale":"2024-01-15T18:04:59.847206+0000","last_undegraded":"2024-01-15T18:04:59.847206+0000","last_fullsized":"2024-01-15T18:04:59.847206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:08:43.279198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032260+0000","last_change":"2024-01-15T18:04:57.840527+0000","last_active":"2024-01-15T18:05:00.032260+0000","last_peered":"2024-01-15T18:05:00.032260+0000","last_clean":"2024-01-15T18:05:00.032260+0000","last_became_active":"2024-01-15T18:04:57.840153+0000","last_became_peered":"2024-01-15T18:04:57.840153+0000","last_unstale":"2024-01-15T18:05:00.032260+0000","last_undegraded":"2024-01-15T18:05:00.032260+0000","last_fullsized":"2024-01-15T18:05:00.032260+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:17:50.737228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.869005+0000","last_change":"2024-01-15T18:04:58.859368+0000","last_active":"2024-01-15T18:04:58.869005+0000","last_peered":"2024-01-15T18:04:58.869005+0000","last_clean":"2024-01-15T18:04:58.869005+0000","last_became_active":"2024-01-15T18:04:58.859112+0000","last_became_peered":"2024-01-15T18:04:58.859112+0000","last_unstale":"2024-01-15T18:04:58.869005+0000","last_undegraded":"2024-01-15T18:04:58.869005+0000","last_fullsized":"2024-01-15T18:04:58.869005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:12:46.662813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030936+0000","last_change":"2024-01-15T18:04:57.843777+0000","last_active":"2024-01-15T18:05:00.030936+0000","last_peered":"2024-01-15T18:05:00.030936+0000","last_clean":"2024-01-15T18:05:00.030936+0000","last_became_active":"2024-01-15T18:04:57.843062+0000","last_became_peered":"2024-01-15T18:04:57.843062+0000","last_unstale":"2024-01-15T18:05:00.030936+0000","last_undegraded":"2024-01-15T18:05:00.030936+0000","last_fullsized":"2024-01-15T18:05:00.030936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:58:15.645144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031986+0000","last_change":"2024-01-15T18:04:57.833749+0000","last_active":"2024-01-15T18:05:00.031986+0000","last_peered":"2024-01-15T18:05:00.031986+0000","last_clean":"2024-01-15T18:05:00.031986+0000","last_became_active":"2024-01-15T18:04:57.832655+0000","last_became_peered":"2024-01-15T18:04:57.832655+0000","last_unstale":"2024-01-15T18:05:00.031986+0000","last_undegraded":"2024-01-15T18:05:00.031986+0000","last_fullsized":"2024-01-15T18:05:00.031986+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:42:16.957837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031910+0000","last_change":"2024-01-15T18:04:58.862938+0000","last_active":"2024-01-15T18:05:00.031910+0000","last_peered":"2024-01-15T18:05:00.031910+0000","last_clean":"2024-01-15T18:05:00.031910+0000","last_became_active":"2024-01-15T18:04:58.862733+0000","last_became_peered":"2024-01-15T18:04:58.862733+0000","last_unstale":"2024-01-15T18:05:00.031910+0000","last_undegraded":"2024-01-15T18:05:00.031910+0000","last_fullsized":"2024-01-15T18:05:00.031910+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:10.719289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.831953+0000","last_change":"2024-01-15T18:04:57.828200+0000","last_active":"2024-01-15T18:04:58.831953+0000","last_peered":"2024-01-15T18:04:58.831953+0000","last_clean":"2024-01-15T18:04:58.831953+0000","last_became_active":"2024-01-15T18:04:57.827883+0000","last_became_peered":"2024-01-15T18:04:57.827883+0000","last_unstale":"2024-01-15T18:04:58.831953+0000","last_undegraded":"2024-01-15T18:04:58.831953+0000","last_fullsized":"2024-01-15T18:04:58.831953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:48:16.392862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849021+0000","last_change":"2024-01-15T18:04:58.842387+0000","last_active":"2024-01-15T18:04:59.849021+0000","last_peered":"2024-01-15T18:04:59.849021+0000","last_clean":"2024-01-15T18:04:59.849021+0000","last_became_active":"2024-01-15T18:04:58.842093+0000","last_became_peered":"2024-01-15T18:04:58.842093+0000","last_unstale":"2024-01-15T18:04:59.849021+0000","last_undegraded":"2024-01-15T18:04:59.849021+0000","last_fullsized":"2024-01-15T18:04:59.849021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:47.713249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030983+0000","last_change":"2024-01-15T18:04:57.837812+0000","last_active":"2024-01-15T18:05:00.030983+0000","last_peered":"2024-01-15T18:05:00.030983+0000","last_clean":"2024-01-15T18:05:00.030983+0000","last_became_active":"2024-01-15T18:04:57.837524+0000","last_became_peered":"2024-01-15T18:04:57.837524+0000","last_unstale":"2024-01-15T18:05:00.030983+0000","last_undegraded":"2024-01-15T18:05:00.030983+0000","last_fullsized":"2024-01-15T18:05:00.030983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:37:44.430593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.859275+0000","last_change":"2024-01-15T18:04:58.854115+0000","last_active":"2024-01-15T18:04:58.859275+0000","last_peered":"2024-01-15T18:04:58.859275+0000","last_clean":"2024-01-15T18:04:58.859275+0000","last_became_active":"2024-01-15T18:04:58.853039+0000","last_became_peered":"2024-01-15T18:04:58.853039+0000","last_unstale":"2024-01-15T18:04:58.859275+0000","last_undegraded":"2024-01-15T18:04:58.859275+0000","last_fullsized":"2024-01-15T18:04:58.859275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:26:45.940779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.829328+0000","last_change":"2024-01-15T18:04:57.839191+0000","last_active":"2024-01-15T18:04:58.829328+0000","last_peered":"2024-01-15T18:04:58.829328+0000","last_clean":"2024-01-15T18:04:58.829328+0000","last_became_active":"2024-01-15T18:04:57.838884+0000","last_became_peered":"2024-01-15T18:04:57.838884+0000","last_unstale":"2024-01-15T18:04:58.829328+0000","last_undegraded":"2024-01-15T18:04:58.829328+0000","last_fullsized":"2024-01-15T18:04:58.829328+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:37:39.211256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.871789+0000","last_change":"2024-01-15T18:04:58.860345+0000","last_active":"2024-01-15T18:04:58.871789+0000","last_peered":"2024-01-15T18:04:58.871789+0000","last_clean":"2024-01-15T18:04:58.871789+0000","last_became_active":"2024-01-15T18:04:58.859997+0000","last_became_peered":"2024-01-15T18:04:58.859997+0000","last_unstale":"2024-01-15T18:04:58.871789+0000","last_undegraded":"2024-01-15T18:04:58.871789+0000","last_fullsized":"2024-01-15T18:04:58.871789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:55:53.031409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849266+0000","last_change":"2024-01-15T18:04:57.835121+0000","last_active":"2024-01-15T18:04:59.849266+0000","last_peered":"2024-01-15T18:04:59.849266+0000","last_clean":"2024-01-15T18:04:59.849266+0000","last_became_active":"2024-01-15T18:04:57.834822+0000","last_became_peered":"2024-01-15T18:04:57.834822+0000","last_unstale":"2024-01-15T18:04:59.849266+0000","last_undegraded":"2024-01-15T18:04:59.849266+0000","last_fullsized":"2024-01-15T18:04:59.849266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:17:50.737228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847554+0000","last_change":"2024-01-15T18:04:58.859000+0000","last_active":"2024-01-15T18:04:59.847554+0000","last_peered":"2024-01-15T18:04:59.847554+0000","last_clean":"2024-01-15T18:04:59.847554+0000","last_became_active":"2024-01-15T18:04:58.854577+0000","last_became_peered":"2024-01-15T18:04:58.854577+0000","last_unstale":"2024-01-15T18:04:59.847554+0000","last_undegraded":"2024-01-15T18:04:59.847554+0000","last_fullsized":"2024-01-15T18:04:59.847554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:24:34.638590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030767+0000","last_change":"2024-01-15T18:04:57.836570+0000","last_active":"2024-01-15T18:05:00.030767+0000","last_peered":"2024-01-15T18:05:00.030767+0000","last_clean":"2024-01-15T18:05:00.030767+0000","last_became_active":"2024-01-15T18:04:57.836027+0000","last_became_peered":"2024-01-15T18:04:57.836027+0000","last_unstale":"2024-01-15T18:05:00.030767+0000","last_undegraded":"2024-01-15T18:05:00.030767+0000","last_fullsized":"2024-01-15T18:05:00.030767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:17:50.737228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031727+0000","last_change":"2024-01-15T18:04:58.872987+0000","last_active":"2024-01-15T18:05:00.031727+0000","last_peered":"2024-01-15T18:05:00.031727+0000","last_clean":"2024-01-15T18:05:00.031727+0000","last_became_active":"2024-01-15T18:04:58.872671+0000","last_became_peered":"2024-01-15T18:04:58.872671+0000","last_unstale":"2024-01-15T18:05:00.031727+0000","last_undegraded":"2024-01-15T18:05:00.031727+0000","last_fullsized":"2024-01-15T18:05:00.031727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:42:12.458758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.871345+0000","last_change":"2024-01-15T18:04:57.839505+0000","last_active":"2024-01-15T18:04:58.871345+0000","last_peered":"2024-01-15T18:04:58.871345+0000","last_clean":"2024-01-15T18:04:58.871345+0000","last_became_active":"2024-01-15T18:04:57.839194+0000","last_became_peered":"2024-01-15T18:04:57.839194+0000","last_unstale":"2024-01-15T18:04:58.871345+0000","last_undegraded":"2024-01-15T18:04:58.871345+0000","last_fullsized":"2024-01-15T18:04:58.871345+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:10.960718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.867419+0000","last_change":"2024-01-15T18:04:58.858064+0000","last_active":"2024-01-15T18:04:58.867419+0000","last_peered":"2024-01-15T18:04:58.867419+0000","last_clean":"2024-01-15T18:04:58.867419+0000","last_became_active":"2024-01-15T18:04:58.857697+0000","last_became_peered":"2024-01-15T18:04:58.857697+0000","last_unstale":"2024-01-15T18:04:58.867419+0000","last_undegraded":"2024-01-15T18:04:58.867419+0000","last_fullsized":"2024-01-15T18:04:58.867419+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:11.539894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.830236+0000","last_change":"2024-01-15T18:04:57.836035+0000","last_active":"2024-01-15T18:04:58.830236+0000","last_peered":"2024-01-15T18:04:58.830236+0000","last_clean":"2024-01-15T18:04:58.830236+0000","last_became_active":"2024-01-15T18:04:57.835722+0000","last_became_peered":"2024-01-15T18:04:57.835722+0000","last_unstale":"2024-01-15T18:04:58.830236+0000","last_undegraded":"2024-01-15T18:04:58.830236+0000","last_fullsized":"2024-01-15T18:04:58.830236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:24:59.847823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031861+0000","last_change":"2024-01-15T18:04:58.852717+0000","last_active":"2024-01-15T18:05:00.031861+0000","last_peered":"2024-01-15T18:05:00.031861+0000","last_clean":"2024-01-15T18:05:00.031861+0000","last_became_active":"2024-01-15T18:04:58.852351+0000","last_became_peered":"2024-01-15T18:04:58.852351+0000","last_unstale":"2024-01-15T18:05:00.031861+0000","last_undegraded":"2024-01-15T18:05:00.031861+0000","last_fullsized":"2024-01-15T18:05:00.031861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:47:03.033950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030528+0000","last_change":"2024-01-15T18:04:57.831525+0000","last_active":"2024-01-15T18:05:00.030528+0000","last_peered":"2024-01-15T18:05:00.030528+0000","last_clean":"2024-01-15T18:05:00.030528+0000","last_became_active":"2024-01-15T18:04:57.831053+0000","last_became_peered":"2024-01-15T18:04:57.831053+0000","last_unstale":"2024-01-15T18:05:00.030528+0000","last_undegraded":"2024-01-15T18:05:00.030528+0000","last_fullsized":"2024-01-15T18:05:00.030528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:24:33.632172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030589+0000","last_change":"2024-01-15T18:04:58.866415+0000","last_active":"2024-01-15T18:05:00.030589+0000","last_peered":"2024-01-15T18:05:00.030589+0000","last_clean":"2024-01-15T18:05:00.030589+0000","last_became_active":"2024-01-15T18:04:58.865778+0000","last_became_peered":"2024-01-15T18:04:58.865778+0000","last_unstale":"2024-01-15T18:05:00.030589+0000","last_undegraded":"2024-01-15T18:05:00.030589+0000","last_fullsized":"2024-01-15T18:05:00.030589+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:47.713249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030622+0000","last_change":"2024-01-15T18:04:57.821766+0000","last_active":"2024-01-15T18:05:00.030622+0000","last_peered":"2024-01-15T18:05:00.030622+0000","last_clean":"2024-01-15T18:05:00.030622+0000","last_became_active":"2024-01-15T18:04:57.820957+0000","last_became_peered":"2024-01-15T18:04:57.820957+0000","last_unstale":"2024-01-15T18:05:00.030622+0000","last_undegraded":"2024-01-15T18:05:00.030622+0000","last_fullsized":"2024-01-15T18:05:00.030622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:08:42.272780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031415+0000","last_change":"2024-01-15T18:04:58.862675+0000","last_active":"2024-01-15T18:05:00.031415+0000","last_peered":"2024-01-15T18:05:00.031415+0000","last_clean":"2024-01-15T18:05:00.031415+0000","last_became_active":"2024-01-15T18:04:58.862420+0000","last_became_peered":"2024-01-15T18:04:58.862420+0000","last_unstale":"2024-01-15T18:05:00.031415+0000","last_undegraded":"2024-01-15T18:05:00.031415+0000","last_fullsized":"2024-01-15T18:05:00.031415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:44:17.933005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847491+0000","last_change":"2024-01-15T18:04:57.839139+0000","last_active":"2024-01-15T18:04:59.847491+0000","last_peered":"2024-01-15T18:04:59.847491+0000","last_clean":"2024-01-15T18:04:59.847491+0000","last_became_active":"2024-01-15T18:04:57.838847+0000","last_became_peered":"2024-01-15T18:04:57.838847+0000","last_unstale":"2024-01-15T18:04:59.847491+0000","last_undegraded":"2024-01-15T18:04:59.847491+0000","last_fullsized":"2024-01-15T18:04:59.847491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:44:30.299838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031909+0000","last_change":"2024-01-15T18:04:58.870909+0000","last_active":"2024-01-15T18:05:00.031909+0000","last_peered":"2024-01-15T18:05:00.031909+0000","last_clean":"2024-01-15T18:05:00.031909+0000","last_became_active":"2024-01-15T18:04:58.868734+0000","last_became_peered":"2024-01-15T18:04:58.868734+0000","last_unstale":"2024-01-15T18:05:00.031909+0000","last_undegraded":"2024-01-15T18:05:00.031909+0000","last_fullsized":"2024-01-15T18:05:00.031909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:58:21.135065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030997+0000","last_change":"2024-01-15T18:04:57.829932+0000","last_active":"2024-01-15T18:05:00.030997+0000","last_peered":"2024-01-15T18:05:00.030997+0000","last_clean":"2024-01-15T18:05:00.030997+0000","last_became_active":"2024-01-15T18:04:57.829636+0000","last_became_peered":"2024-01-15T18:04:57.829636+0000","last_unstale":"2024-01-15T18:05:00.030997+0000","last_undegraded":"2024-01-15T18:05:00.030997+0000","last_fullsized":"2024-01-15T18:05:00.030997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:04:13.242693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032156+0000","last_change":"2024-01-15T18:04:58.869477+0000","last_active":"2024-01-15T18:05:00.032156+0000","last_peered":"2024-01-15T18:05:00.032156+0000","last_clean":"2024-01-15T18:05:00.032156+0000","last_became_active":"2024-01-15T18:04:58.869133+0000","last_became_peered":"2024-01-15T18:04:58.869133+0000","last_unstale":"2024-01-15T18:05:00.032156+0000","last_undegraded":"2024-01-15T18:05:00.032156+0000","last_fullsized":"2024-01-15T18:05:00.032156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:39:08.175871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849313+0000","last_change":"2024-01-15T18:04:57.830731+0000","last_active":"2024-01-15T18:04:59.849313+0000","last_peered":"2024-01-15T18:04:59.849313+0000","last_clean":"2024-01-15T18:04:59.849313+0000","last_became_active":"2024-01-15T18:04:57.830234+0000","last_became_peered":"2024-01-15T18:04:57.830234+0000","last_unstale":"2024-01-15T18:04:59.849313+0000","last_undegraded":"2024-01-15T18:04:59.849313+0000","last_fullsized":"2024-01-15T18:04:59.849313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:37:44.430593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.853920+0000","last_change":"2024-01-15T18:04:58.843965+0000","last_active":"2024-01-15T18:04:58.853920+0000","last_peered":"2024-01-15T18:04:58.853920+0000","last_clean":"2024-01-15T18:04:58.853920+0000","last_became_active":"2024-01-15T18:04:58.841866+0000","last_became_peered":"2024-01-15T18:04:58.841866+0000","last_unstale":"2024-01-15T18:04:58.853920+0000","last_undegraded":"2024-01-15T18:04:58.853920+0000","last_fullsized":"2024-01-15T18:04:58.853920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:02:33.077446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030724+0000","last_change":"2024-01-15T18:04:57.836644+0000","last_active":"2024-01-15T18:05:00.030724+0000","last_peered":"2024-01-15T18:05:00.030724+0000","last_clean":"2024-01-15T18:05:00.030724+0000","last_became_active":"2024-01-15T18:04:57.835872+0000","last_became_peered":"2024-01-15T18:04:57.835872+0000","last_unstale":"2024-01-15T18:05:00.030724+0000","last_undegraded":"2024-01-15T18:05:00.030724+0000","last_fullsized":"2024-01-15T18:05:00.030724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:44:30.299838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031872+0000","last_change":"2024-01-15T18:04:58.857962+0000","last_active":"2024-01-15T18:05:00.031872+0000","last_peered":"2024-01-15T18:05:00.031872+0000","last_clean":"2024-01-15T18:05:00.031872+0000","last_became_active":"2024-01-15T18:04:58.857644+0000","last_became_peered":"2024-01-15T18:04:58.857644+0000","last_unstale":"2024-01-15T18:05:00.031872+0000","last_undegraded":"2024-01-15T18:05:00.031872+0000","last_fullsized":"2024-01-15T18:05:00.031872+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:02:19.202437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.836746+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:04:58.836746+0000","last_peered":"2024-01-15T18:04:58.836746+0000","last_clean":"2024-01-15T18:04:58.836746+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:04:58.836746+0000","last_undegraded":"2024-01-15T18:04:58.836746+0000","last_fullsized":"2024-01-15T18:04:58.836746+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.827351+0000","last_change":"2024-01-15T18:04:57.829926+0000","last_active":"2024-01-15T18:04:58.827351+0000","last_peered":"2024-01-15T18:04:58.827351+0000","last_clean":"2024-01-15T18:04:58.827351+0000","last_became_active":"2024-01-15T18:04:57.829578+0000","last_became_peered":"2024-01-15T18:04:57.829578+0000","last_unstale":"2024-01-15T18:04:58.827351+0000","last_undegraded":"2024-01-15T18:04:58.827351+0000","last_fullsized":"2024-01-15T18:04:58.827351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:55:52.024992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032132+0000","last_change":"2024-01-15T18:04:58.862009+0000","last_active":"2024-01-15T18:05:00.032132+0000","last_peered":"2024-01-15T18:05:00.032132+0000","last_clean":"2024-01-15T18:05:00.032132+0000","last_became_active":"2024-01-15T18:04:58.861745+0000","last_became_peered":"2024-01-15T18:04:58.861745+0000","last_unstale":"2024-01-15T18:05:00.032132+0000","last_undegraded":"2024-01-15T18:05:00.032132+0000","last_fullsized":"2024-01-15T18:05:00.032132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:15:34.376159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031331+0000","last_change":"2024-01-15T18:04:57.842536+0000","last_active":"2024-01-15T18:05:00.031331+0000","last_peered":"2024-01-15T18:05:00.031331+0000","last_clean":"2024-01-15T18:05:00.031331+0000","last_became_active":"2024-01-15T18:04:57.842236+0000","last_became_peered":"2024-01-15T18:04:57.842236+0000","last_unstale":"2024-01-15T18:05:00.031331+0000","last_undegraded":"2024-01-15T18:05:00.031331+0000","last_fullsized":"2024-01-15T18:05:00.031331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:24:33.632172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.867604+0000","last_change":"2024-01-15T18:04:58.859545+0000","last_active":"2024-01-15T18:04:58.867604+0000","last_peered":"2024-01-15T18:04:58.867604+0000","last_clean":"2024-01-15T18:04:58.867604+0000","last_became_active":"2024-01-15T18:04:58.859282+0000","last_became_peered":"2024-01-15T18:04:58.859282+0000","last_unstale":"2024-01-15T18:04:58.867604+0000","last_undegraded":"2024-01-15T18:04:58.867604+0000","last_fullsized":"2024-01-15T18:04:58.867604+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:26:45.940779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030970+0000","last_change":"2024-01-15T18:04:57.826351+0000","last_active":"2024-01-15T18:05:00.030970+0000","last_peered":"2024-01-15T18:05:00.030970+0000","last_clean":"2024-01-15T18:05:00.030970+0000","last_became_active":"2024-01-15T18:04:57.825983+0000","last_became_peered":"2024-01-15T18:04:57.825983+0000","last_unstale":"2024-01-15T18:05:00.030970+0000","last_undegraded":"2024-01-15T18:05:00.030970+0000","last_fullsized":"2024-01-15T18:05:00.030970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:30:11.561882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849021+0000","last_change":"2024-01-15T18:04:58.834900+0000","last_active":"2024-01-15T18:04:59.849021+0000","last_peered":"2024-01-15T18:04:59.849021+0000","last_clean":"2024-01-15T18:04:59.849021+0000","last_became_active":"2024-01-15T18:04:58.834561+0000","last_became_peered":"2024-01-15T18:04:58.834561+0000","last_unstale":"2024-01-15T18:04:59.849021+0000","last_undegraded":"2024-01-15T18:04:59.849021+0000","last_fullsized":"2024-01-15T18:04:59.849021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:10.719289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.830421+0000","last_change":"2024-01-15T18:04:57.840353+0000","last_active":"2024-01-15T18:04:58.830421+0000","last_peered":"2024-01-15T18:04:58.830421+0000","last_clean":"2024-01-15T18:04:58.830421+0000","last_became_active":"2024-01-15T18:04:57.840032+0000","last_became_peered":"2024-01-15T18:04:57.840032+0000","last_unstale":"2024-01-15T18:04:58.830421+0000","last_undegraded":"2024-01-15T18:04:58.830421+0000","last_fullsized":"2024-01-15T18:04:58.830421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:21:05.984228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847638+0000","last_change":"2024-01-15T18:04:58.862393+0000","last_active":"2024-01-15T18:04:59.847638+0000","last_peered":"2024-01-15T18:04:59.847638+0000","last_clean":"2024-01-15T18:04:59.847638+0000","last_became_active":"2024-01-15T18:04:58.861776+0000","last_became_peered":"2024-01-15T18:04:58.861776+0000","last_unstale":"2024-01-15T18:04:59.847638+0000","last_undegraded":"2024-01-15T18:04:59.847638+0000","last_fullsized":"2024-01-15T18:04:59.847638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:58:16.651562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849271+0000","last_change":"2024-01-15T18:04:57.832247+0000","last_active":"2024-01-15T18:04:59.849271+0000","last_peered":"2024-01-15T18:04:59.849271+0000","last_clean":"2024-01-15T18:04:59.849271+0000","last_became_active":"2024-01-15T18:04:57.831962+0000","last_became_peered":"2024-01-15T18:04:57.831962+0000","last_unstale":"2024-01-15T18:04:59.849271+0000","last_undegraded":"2024-01-15T18:04:59.849271+0000","last_fullsized":"2024-01-15T18:04:59.849271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:08:42.272780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.850248+0000","last_change":"2024-01-15T18:04:58.842320+0000","last_active":"2024-01-15T18:04:58.850248+0000","last_peered":"2024-01-15T18:04:58.850248+0000","last_clean":"2024-01-15T18:04:58.850248+0000","last_became_active":"2024-01-15T18:04:58.841887+0000","last_became_peered":"2024-01-15T18:04:58.841887+0000","last_unstale":"2024-01-15T18:04:58.850248+0000","last_undegraded":"2024-01-15T18:04:58.850248+0000","last_fullsized":"2024-01-15T18:04:58.850248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:02:12.562587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032034+0000","last_change":"2024-01-15T18:04:57.843600+0000","last_active":"2024-01-15T18:05:00.032034+0000","last_peered":"2024-01-15T18:05:00.032034+0000","last_clean":"2024-01-15T18:05:00.032034+0000","last_became_active":"2024-01-15T18:04:57.843179+0000","last_became_peered":"2024-01-15T18:04:57.843179+0000","last_unstale":"2024-01-15T18:05:00.032034+0000","last_undegraded":"2024-01-15T18:05:00.032034+0000","last_fullsized":"2024-01-15T18:05:00.032034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:10:23.513336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849104+0000","last_change":"2024-01-15T18:04:58.834860+0000","last_active":"2024-01-15T18:04:59.849104+0000","last_peered":"2024-01-15T18:04:59.849104+0000","last_clean":"2024-01-15T18:04:59.849104+0000","last_became_active":"2024-01-15T18:04:58.834570+0000","last_became_peered":"2024-01-15T18:04:58.834570+0000","last_unstale":"2024-01-15T18:04:59.849104+0000","last_undegraded":"2024-01-15T18:04:59.849104+0000","last_fullsized":"2024-01-15T18:04:59.849104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:12:00.696642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849515+0000","last_change":"2024-01-15T18:04:57.832391+0000","last_active":"2024-01-15T18:04:59.849515+0000","last_peered":"2024-01-15T18:04:59.849515+0000","last_clean":"2024-01-15T18:04:59.849515+0000","last_became_active":"2024-01-15T18:04:57.832005+0000","last_became_peered":"2024-01-15T18:04:57.832005+0000","last_unstale":"2024-01-15T18:04:59.849515+0000","last_undegraded":"2024-01-15T18:04:59.849515+0000","last_fullsized":"2024-01-15T18:04:59.849515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:24:33.632172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032511+0000","last_change":"2024-01-15T18:04:58.863289+0000","last_active":"2024-01-15T18:05:00.032511+0000","last_peered":"2024-01-15T18:05:00.032511+0000","last_clean":"2024-01-15T18:05:00.032511+0000","last_became_active":"2024-01-15T18:04:58.862985+0000","last_became_peered":"2024-01-15T18:04:58.862985+0000","last_unstale":"2024-01-15T18:05:00.032511+0000","last_undegraded":"2024-01-15T18:05:00.032511+0000","last_fullsized":"2024-01-15T18:05:00.032511+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:48:27.499654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031100+0000","last_change":"2024-01-15T18:04:57.835018+0000","last_active":"2024-01-15T18:05:00.031100+0000","last_peered":"2024-01-15T18:05:00.031100+0000","last_clean":"2024-01-15T18:05:00.031100+0000","last_became_active":"2024-01-15T18:04:57.834706+0000","last_became_peered":"2024-01-15T18:04:57.834706+0000","last_unstale":"2024-01-15T18:05:00.031100+0000","last_undegraded":"2024-01-15T18:05:00.031100+0000","last_fullsized":"2024-01-15T18:05:00.031100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:11:48.897513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.871874+0000","last_change":"2024-01-15T18:04:58.866805+0000","last_active":"2024-01-15T18:04:58.871874+0000","last_peered":"2024-01-15T18:04:58.871874+0000","last_clean":"2024-01-15T18:04:58.871874+0000","last_became_active":"2024-01-15T18:04:58.866475+0000","last_became_peered":"2024-01-15T18:04:58.866475+0000","last_unstale":"2024-01-15T18:04:58.871874+0000","last_undegraded":"2024-01-15T18:04:58.871874+0000","last_fullsized":"2024-01-15T18:04:58.871874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:14:59.421400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031326+0000","last_change":"2024-01-15T18:04:57.839780+0000","last_active":"2024-01-15T18:05:00.031326+0000","last_peered":"2024-01-15T18:05:00.031326+0000","last_clean":"2024-01-15T18:05:00.031326+0000","last_became_active":"2024-01-15T18:04:57.839392+0000","last_became_peered":"2024-01-15T18:04:57.839392+0000","last_unstale":"2024-01-15T18:05:00.031326+0000","last_undegraded":"2024-01-15T18:05:00.031326+0000","last_fullsized":"2024-01-15T18:05:00.031326+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:16:29.440534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847071+0000","last_change":"2024-01-15T18:04:58.835003+0000","last_active":"2024-01-15T18:04:59.847071+0000","last_peered":"2024-01-15T18:04:59.847071+0000","last_clean":"2024-01-15T18:04:59.847071+0000","last_became_active":"2024-01-15T18:04:58.834678+0000","last_became_peered":"2024-01-15T18:04:58.834678+0000","last_unstale":"2024-01-15T18:04:59.847071+0000","last_undegraded":"2024-01-15T18:04:59.847071+0000","last_fullsized":"2024-01-15T18:04:59.847071+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:04:14.249111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849091+0000","last_change":"2024-01-15T18:04:57.837454+0000","last_active":"2024-01-15T18:04:59.849091+0000","last_peered":"2024-01-15T18:04:59.849091+0000","last_clean":"2024-01-15T18:04:59.849091+0000","last_became_active":"2024-01-15T18:04:57.836947+0000","last_became_peered":"2024-01-15T18:04:57.836947+0000","last_unstale":"2024-01-15T18:04:59.849091+0000","last_undegraded":"2024-01-15T18:04:59.849091+0000","last_fullsized":"2024-01-15T18:04:59.849091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:11:48.897513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031977+0000","last_change":"2024-01-15T18:04:58.854684+0000","last_active":"2024-01-15T18:05:00.031977+0000","last_peered":"2024-01-15T18:05:00.031977+0000","last_clean":"2024-01-15T18:05:00.031977+0000","last_became_active":"2024-01-15T18:04:58.844371+0000","last_became_peered":"2024-01-15T18:04:58.844371+0000","last_unstale":"2024-01-15T18:05:00.031977+0000","last_undegraded":"2024-01-15T18:05:00.031977+0000","last_fullsized":"2024-01-15T18:05:00.031977+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:12:00.696642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849194+0000","last_change":"2024-01-15T18:04:57.837432+0000","last_active":"2024-01-15T18:04:59.849194+0000","last_peered":"2024-01-15T18:04:59.849194+0000","last_clean":"2024-01-15T18:04:59.849194+0000","last_became_active":"2024-01-15T18:04:57.837105+0000","last_became_peered":"2024-01-15T18:04:57.837105+0000","last_unstale":"2024-01-15T18:04:59.849194+0000","last_undegraded":"2024-01-15T18:04:59.849194+0000","last_fullsized":"2024-01-15T18:04:59.849194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:04:13.242693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032216+0000","last_change":"2024-01-15T18:04:58.859567+0000","last_active":"2024-01-15T18:05:00.032216+0000","last_peered":"2024-01-15T18:05:00.032216+0000","last_clean":"2024-01-15T18:05:00.032216+0000","last_became_active":"2024-01-15T18:04:58.859199+0000","last_became_peered":"2024-01-15T18:04:58.859199+0000","last_unstale":"2024-01-15T18:05:00.032216+0000","last_undegraded":"2024-01-15T18:05:00.032216+0000","last_fullsized":"2024-01-15T18:05:00.032216+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:10:24.519754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031791+0000","last_change":"2024-01-15T18:04:57.843450+0000","last_active":"2024-01-15T18:05:00.031791+0000","last_peered":"2024-01-15T18:05:00.031791+0000","last_clean":"2024-01-15T18:05:00.031791+0000","last_became_active":"2024-01-15T18:04:57.841883+0000","last_became_peered":"2024-01-15T18:04:57.841883+0000","last_unstale":"2024-01-15T18:05:00.031791+0000","last_undegraded":"2024-01-15T18:05:00.031791+0000","last_fullsized":"2024-01-15T18:05:00.031791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:17:50.737228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030851+0000","last_change":"2024-01-15T18:04:58.845515+0000","last_active":"2024-01-15T18:05:00.030851+0000","last_peered":"2024-01-15T18:05:00.030851+0000","last_clean":"2024-01-15T18:05:00.030851+0000","last_became_active":"2024-01-15T18:04:58.845242+0000","last_became_peered":"2024-01-15T18:04:58.845242+0000","last_unstale":"2024-01-15T18:05:00.030851+0000","last_undegraded":"2024-01-15T18:05:00.030851+0000","last_fullsized":"2024-01-15T18:05:00.030851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:36.680373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.846973+0000","last_change":"2024-01-15T18:04:58.840618+0000","last_active":"2024-01-15T18:04:58.846973+0000","last_peered":"2024-01-15T18:04:58.846973+0000","last_clean":"2024-01-15T18:04:58.846973+0000","last_became_active":"2024-01-15T18:04:58.840306+0000","last_became_peered":"2024-01-15T18:04:58.840306+0000","last_unstale":"2024-01-15T18:04:58.846973+0000","last_undegraded":"2024-01-15T18:04:58.846973+0000","last_fullsized":"2024-01-15T18:04:58.846973+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:27:32.756552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031572+0000","last_change":"2024-01-15T18:04:57.842676+0000","last_active":"2024-01-15T18:05:00.031572+0000","last_peered":"2024-01-15T18:05:00.031572+0000","last_clean":"2024-01-15T18:05:00.031572+0000","last_became_active":"2024-01-15T18:04:57.842402+0000","last_became_peered":"2024-01-15T18:04:57.842402+0000","last_unstale":"2024-01-15T18:05:00.031572+0000","last_undegraded":"2024-01-15T18:05:00.031572+0000","last_fullsized":"2024-01-15T18:05:00.031572+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:09.712871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.847833+0000","last_change":"2024-01-15T18:04:58.837433+0000","last_active":"2024-01-15T18:04:58.847833+0000","last_peered":"2024-01-15T18:04:58.847833+0000","last_clean":"2024-01-15T18:04:58.847833+0000","last_became_active":"2024-01-15T18:04:58.836989+0000","last_became_peered":"2024-01-15T18:04:58.836989+0000","last_unstale":"2024-01-15T18:04:58.847833+0000","last_undegraded":"2024-01-15T18:04:58.847833+0000","last_fullsized":"2024-01-15T18:04:58.847833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:36:10.997513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031908+0000","last_change":"2024-01-15T18:04:57.834129+0000","last_active":"2024-01-15T18:05:00.031908+0000","last_peered":"2024-01-15T18:05:00.031908+0000","last_clean":"2024-01-15T18:05:00.031908+0000","last_became_active":"2024-01-15T18:04:57.833804+0000","last_became_peered":"2024-01-15T18:04:57.833804+0000","last_unstale":"2024-01-15T18:05:00.031908+0000","last_undegraded":"2024-01-15T18:05:00.031908+0000","last_fullsized":"2024-01-15T18:05:00.031908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:11:59.690224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.871769+0000","last_change":"2024-01-15T18:04:58.860153+0000","last_active":"2024-01-15T18:04:58.871769+0000","last_peered":"2024-01-15T18:04:58.871769+0000","last_clean":"2024-01-15T18:04:58.871769+0000","last_became_active":"2024-01-15T18:04:58.855401+0000","last_became_peered":"2024-01-15T18:04:58.855401+0000","last_unstale":"2024-01-15T18:04:58.871769+0000","last_undegraded":"2024-01-15T18:04:58.871769+0000","last_fullsized":"2024-01-15T18:04:58.871769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:02:12.562587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032517+0000","last_change":"2024-01-15T18:04:57.830136+0000","last_active":"2024-01-15T18:05:00.032517+0000","last_peered":"2024-01-15T18:05:00.032517+0000","last_clean":"2024-01-15T18:05:00.032517+0000","last_became_active":"2024-01-15T18:04:57.829333+0000","last_became_peered":"2024-01-15T18:04:57.829333+0000","last_unstale":"2024-01-15T18:05:00.032517+0000","last_undegraded":"2024-01-15T18:05:00.032517+0000","last_fullsized":"2024-01-15T18:05:00.032517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:37:44.430593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032421+0000","last_change":"2024-01-15T18:04:58.855872+0000","last_active":"2024-01-15T18:05:00.032421+0000","last_peered":"2024-01-15T18:05:00.032421+0000","last_clean":"2024-01-15T18:05:00.032421+0000","last_became_active":"2024-01-15T18:04:58.854897+0000","last_became_peered":"2024-01-15T18:04:58.854897+0000","last_unstale":"2024-01-15T18:05:00.032421+0000","last_undegraded":"2024-01-15T18:05:00.032421+0000","last_fullsized":"2024-01-15T18:05:00.032421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:33:52.636110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.878477+0000","last_change":"2024-01-15T18:04:57.843229+0000","last_active":"2024-01-15T18:04:58.878477+0000","last_peered":"2024-01-15T18:04:58.878477+0000","last_clean":"2024-01-15T18:04:58.878477+0000","last_became_active":"2024-01-15T18:04:57.842720+0000","last_became_peered":"2024-01-15T18:04:57.842720+0000","last_unstale":"2024-01-15T18:04:58.878477+0000","last_undegraded":"2024-01-15T18:04:58.878477+0000","last_fullsized":"2024-01-15T18:04:58.878477+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:25:33.145590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509445,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":806912,"data_stored":663105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80400000000000005}]}]},{"osd":6,"up_from":15,"seq":64424509445,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":73767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]}]},{"osd":4,"up_from":15,"seq":64424509445,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":65981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]}]},{"osd":3,"up_from":15,"seq":64424509444,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":67467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]}]},{"osd":2,"up_from":15,"seq":64424509444,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":65855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]}]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]}]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":184320,"data_stored":59080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":802816,"data_stored":656861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:05:02.949 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:05:03.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.120+0000 7f8bee800640 1 Processor -- start 2024-01-15T18:05:03.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.120+0000 7f8bee800640 1 -- start start 2024-01-15T18:05:03.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.124+0000 7f8bee800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8be815d0a0 0x7f8be806c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:03.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.124+0000 7f8bee800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8be80c94c0 0x7f8be80c9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:03.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.124+0000 7f8bee800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8be80ad180 con 0x7f8be80ce710 2024-01-15T18:05:03.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.124+0000 7f8bee800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8be80c9e90 con 0x7f8be80c94c0 2024-01-15T18:05:03.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.124+0000 7f8bee800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8be80ac890 con 0x7f8be815d0a0 2024-01-15T18:05:03.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.124+0000 7f8be7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8be80c94c0 0x7f8be80c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:03.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.124+0000 7f8be7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8be80c94c0 0x7f8be80c9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46620/0 (socket says 172.21.15.27:46620) 2024-01-15T18:05:03.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.124+0000 7f8be7e00640 1 -- 172.21.15.27:0/2942632513 learned_addr learned my addr 172.21.15.27:0/2942632513 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:03.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.124+0000 7f8be7e00640 1 -- 172.21.15.27:0/2942632513 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8be815d0a0 msgr2=0x7f8be806c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:03.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.124+0000 7f8be7400640 1 --2- 172.21.15.27:0/2942632513 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8be815d0a0 0x7f8be806c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:03.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.124+0000 7f8be7e00640 1 --2- 172.21.15.27:0/2942632513 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8be815d0a0 0x7f8be806c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.124+0000 7f8be7e00640 1 -- 172.21.15.27:0/2942632513 >> v1:172.21.15.93:6789/0 conn(0x7f8be80ce710 legacy=0x7f8be80cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:03.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.124+0000 7f8be7e00640 1 -- 172.21.15.27:0/2942632513 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be80cda90 con 0x7f8be80c94c0 2024-01-15T18:05:03.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.124+0000 7f8be7400640 1 --2- 172.21.15.27:0/2942632513 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8be815d0a0 0x7f8be806c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:03.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.128+0000 7f8be7e00640 1 --2- 172.21.15.27:0/2942632513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8be80c94c0 0x7f8be80c9950 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8be0009e50 tx=0x7f8be001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=337805abcb995fd0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:03.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.128+0000 7f8be6000640 1 -- 172.21.15.27:0/2942632513 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be00a5820 con 0x7f8be80c94c0 2024-01-15T18:05:03.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.128+0000 7f8be6000640 1 -- 172.21.15.27:0/2942632513 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be00a5980 con 0x7f8be80c94c0 2024-01-15T18:05:03.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.128+0000 7f8be6000640 1 -- 172.21.15.27:0/2942632513 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be00a5d10 con 0x7f8be80c94c0 2024-01-15T18:05:03.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.128+0000 7f8bee800640 1 -- 172.21.15.27:0/2942632513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8be80c94c0 msgr2=0x7f8be80c9950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:03.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.128+0000 7f8bee800640 1 --2- 172.21.15.27:0/2942632513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8be80c94c0 0x7f8be80c9950 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8be0009e50 tx=0x7f8be001c9a0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.128+0000 7f8bee800640 1 -- 172.21.15.27:0/2942632513 shutdown_connections 2024-01-15T18:05:03.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.128+0000 7f8bee800640 1 --2- 172.21.15.27:0/2942632513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8be80c94c0 0x7f8be80c9950 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.128+0000 7f8bee800640 1 --2- 172.21.15.27:0/2942632513 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8be815d0a0 0x7f8be806c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.128+0000 7f8bee800640 1 -- 172.21.15.27:0/2942632513 >> 172.21.15.27:0/2942632513 conn(0x7f8be80cec60 msgr2=0x7f8be80cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:03.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.128+0000 7f8bee800640 1 -- 172.21.15.27:0/2942632513 shutdown_connections 2024-01-15T18:05:03.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.128+0000 7f8bee800640 1 -- 172.21.15.27:0/2942632513 wait complete. 2024-01-15T18:05:03.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.128+0000 7f8bee800640 1 Processor -- start 2024-01-15T18:05:03.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.128+0000 7f8bee800640 1 -- start start 2024-01-15T18:05:03.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.132+0000 7f8bee800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8be80c94c0 0x7f8be81679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:03.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.132+0000 7f8bee800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8be80ce710 0x7f8be8158410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:03.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.132+0000 7f8bee800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8be815d0a0 0x7f8be814c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:03.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.132+0000 7f8bee800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be8158950 con 0x7f8be80ce710 2024-01-15T18:05:03.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.132+0000 7f8bee800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be814c990 con 0x7f8be815d0a0 2024-01-15T18:05:03.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.132+0000 7f8bee800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8be815c220 con 0x7f8be80c94c0 2024-01-15T18:05:03.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.132+0000 7f8be6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8be80ce710 0x7f8be8158410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:03.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.132+0000 7f8be6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8be80ce710 0x7f8be8158410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46630/0 (socket says 172.21.15.27:46630) 2024-01-15T18:05:03.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.132+0000 7f8be6a00640 1 -- 172.21.15.27:0/1713064991 learned_addr learned my addr 172.21.15.27:0/1713064991 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.132+0000 7f8be7e00640 1 --2- 172.21.15.27:0/1713064991 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8be815d0a0 0x7f8be814c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.132+0000 7f8be7400640 1 --2- 172.21.15.27:0/1713064991 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8be80c94c0 0x7f8be81679a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.132+0000 7f8be6a00640 1 -- 172.21.15.27:0/1713064991 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8be80c94c0 msgr2=0x7f8be81679a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.132+0000 7f8be6a00640 1 --2- 172.21.15.27:0/1713064991 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8be80c94c0 0x7f8be81679a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.132+0000 7f8be6a00640 1 -- 172.21.15.27:0/1713064991 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8be815d0a0 msgr2=0x7f8be814c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.132+0000 7f8be6a00640 1 --2- 172.21.15.27:0/1713064991 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8be815d0a0 0x7f8be814c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.132+0000 7f8be6a00640 1 -- 172.21.15.27:0/1713064991 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be0009b30 con 0x7f8be80ce710 2024-01-15T18:05:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.132+0000 7f8be6a00640 1 --2- 172.21.15.27:0/1713064991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8be80ce710 0x7f8be8158410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8bd000b410 tx=0x7f8bd000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.132+0000 7f8bdf400640 1 -- 172.21.15.27:0/1713064991 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd0004020 con 0x7f8be80ce710 2024-01-15T18:05:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.132+0000 7f8bdf400640 1 -- 172.21.15.27:0/1713064991 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bd0004180 con 0x7f8be80ce710 2024-01-15T18:05:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.132+0000 7f8bee800640 1 -- 172.21.15.27:0/1713064991 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8be815d690 con 0x7f8be80ce710 2024-01-15T18:05:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.132+0000 7f8bdf400640 1 -- 172.21.15.27:0/1713064991 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd0011630 con 0x7f8be80ce710 2024-01-15T18:05:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.132+0000 7f8bee800640 1 -- 172.21.15.27:0/1713064991 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8be8157c10 con 0x7f8be80ce710 2024-01-15T18:05:03.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.132+0000 7f8bee800640 1 -- 172.21.15.27:0/1713064991 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8be80c8cb0 con 0x7f8be80ce710 2024-01-15T18:05:03.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.132+0000 7f8bdf400640 1 -- 172.21.15.27:0/1713064991 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f8bd0011790 con 0x7f8be80ce710 2024-01-15T18:05:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.136+0000 7f8bdf400640 1 --2- 172.21.15.27:0/1713064991 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8bc00668f0 0x7f8bc0068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:03.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.136+0000 7f8bdf400640 1 -- 172.21.15.27:0/1713064991 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8bd00840d0 con 0x7f8be80ce710 2024-01-15T18:05:03.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.136+0000 7f8be7400640 1 --2- 172.21.15.27:0/1713064991 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8bc00668f0 0x7f8bc0068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:03.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.136+0000 7f8be7400640 1 --2- 172.21.15.27:0/1713064991 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8bc00668f0 0x7f8bc0068db0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8006fd0 tx=0x7f8bd8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:03.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.140+0000 7f8bdf400640 1 -- 172.21.15.27:0/1713064991 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8bd0051320 con 0x7f8be80ce710 2024-01-15T18:05:03.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.292+0000 7f8bee800640 1 -- 172.21.15.27:0/1713064991 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8be804e850 con 0x7f8bc00668f0 2024-01-15T18:05:03.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.300+0000 7f8bdf400640 1 -- 172.21.15.27:0/1713064991 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264648 (secure 0 0 0) 0x7f8be804e850 con 0x7f8bc00668f0 2024-01-15T18:05:03.307 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:03.308 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:05:03.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.304+0000 7f8bee800640 1 -- 172.21.15.27:0/1713064991 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8bc00668f0 msgr2=0x7f8bc0068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:03.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.304+0000 7f8bee800640 1 --2- 172.21.15.27:0/1713064991 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8bc00668f0 0x7f8bc0068db0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8006fd0 tx=0x7f8bd8008040 comp rx=0 tx=0).stop 2024-01-15T18:05:03.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.304+0000 7f8bee800640 1 -- 172.21.15.27:0/1713064991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8be80ce710 msgr2=0x7f8be8158410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:03.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.304+0000 7f8bee800640 1 --2- 172.21.15.27:0/1713064991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8be80ce710 0x7f8be8158410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8bd000b410 tx=0x7f8bd000b8d0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.304+0000 7f8bee800640 1 -- 172.21.15.27:0/1713064991 shutdown_connections 2024-01-15T18:05:03.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.304+0000 7f8bee800640 1 --2- 172.21.15.27:0/1713064991 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8bc00668f0 0x7f8bc0068db0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.304+0000 7f8bee800640 1 --2- 172.21.15.27:0/1713064991 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8be815d0a0 0x7f8be814c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.304+0000 7f8bee800640 1 --2- 172.21.15.27:0/1713064991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8be80ce710 0x7f8be8158410 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.304+0000 7f8bee800640 1 --2- 172.21.15.27:0/1713064991 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8be80c94c0 0x7f8be81679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.304+0000 7f8bee800640 1 -- 172.21.15.27:0/1713064991 >> 172.21.15.27:0/1713064991 conn(0x7f8be80cec60 msgr2=0x7f8be806dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:03.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.304+0000 7f8bee800640 1 -- 172.21.15.27:0/1713064991 shutdown_connections 2024-01-15T18:05:03.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.304+0000 7f8bee800640 1 -- 172.21.15.27:0/1713064991 wait complete. 2024-01-15T18:05:03.328 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-15T18:05:02.139338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":121,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218952,"kb_used_data":2780,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755768,"statfs":{"total":773094113280,"available":772869906432,"internally_reserved":0,"allocated":2846720,"data_stored":1724431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.329975"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032071+0000","last_change":"2024-01-15T18:04:58.862012+0000","last_active":"2024-01-15T18:05:00.032071+0000","last_peered":"2024-01-15T18:05:00.032071+0000","last_clean":"2024-01-15T18:05:00.032071+0000","last_became_active":"2024-01-15T18:04:58.861713+0000","last_became_peered":"2024-01-15T18:04:58.861713+0000","last_unstale":"2024-01-15T18:05:00.032071+0000","last_undegraded":"2024-01-15T18:05:00.032071+0000","last_fullsized":"2024-01-15T18:05:00.032071+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:30:29.323987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849437+0000","last_change":"2024-01-15T18:04:58.870910+0000","last_active":"2024-01-15T18:04:59.849437+0000","last_peered":"2024-01-15T18:04:59.849437+0000","last_clean":"2024-01-15T18:04:59.849437+0000","last_became_active":"2024-01-15T18:04:58.869957+0000","last_became_peered":"2024-01-15T18:04:58.869957+0000","last_unstale":"2024-01-15T18:04:59.849437+0000","last_undegraded":"2024-01-15T18:04:59.849437+0000","last_fullsized":"2024-01-15T18:04:59.849437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:10:24.519754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.862393+0000","last_change":"2024-01-15T18:04:58.854305+0000","last_active":"2024-01-15T18:04:58.862393+0000","last_peered":"2024-01-15T18:04:58.862393+0000","last_clean":"2024-01-15T18:04:58.862393+0000","last_became_active":"2024-01-15T18:04:58.853375+0000","last_became_peered":"2024-01-15T18:04:58.853375+0000","last_unstale":"2024-01-15T18:04:58.862393+0000","last_undegraded":"2024-01-15T18:04:58.862393+0000","last_fullsized":"2024-01-15T18:04:58.862393+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:36:10.997513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.852061+0000","last_change":"2024-01-15T18:04:58.844639+0000","last_active":"2024-01-15T18:04:58.852061+0000","last_peered":"2024-01-15T18:04:58.852061+0000","last_clean":"2024-01-15T18:04:58.852061+0000","last_became_active":"2024-01-15T18:04:58.844174+0000","last_became_peered":"2024-01-15T18:04:58.844174+0000","last_unstale":"2024-01-15T18:04:58.852061+0000","last_undegraded":"2024-01-15T18:04:58.852061+0000","last_fullsized":"2024-01-15T18:04:58.852061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:29:02.731583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031678+0000","last_change":"2024-01-15T18:04:58.852004+0000","last_active":"2024-01-15T18:05:00.031678+0000","last_peered":"2024-01-15T18:05:00.031678+0000","last_clean":"2024-01-15T18:05:00.031678+0000","last_became_active":"2024-01-15T18:04:58.849191+0000","last_became_peered":"2024-01-15T18:04:58.849191+0000","last_unstale":"2024-01-15T18:05:00.031678+0000","last_undegraded":"2024-01-15T18:05:00.031678+0000","last_fullsized":"2024-01-15T18:05:00.031678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:22:03.084714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.863975+0000","last_change":"2024-01-15T18:04:58.854461+0000","last_active":"2024-01-15T18:04:58.863975+0000","last_peered":"2024-01-15T18:04:58.863975+0000","last_clean":"2024-01-15T18:04:58.863975+0000","last_became_active":"2024-01-15T18:04:58.854170+0000","last_became_peered":"2024-01-15T18:04:58.854170+0000","last_unstale":"2024-01-15T18:04:58.863975+0000","last_undegraded":"2024-01-15T18:04:58.863975+0000","last_fullsized":"2024-01-15T18:04:58.863975+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:21:44.926560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031425+0000","last_change":"2024-01-15T18:04:58.857161+0000","last_active":"2024-01-15T18:05:00.031425+0000","last_peered":"2024-01-15T18:05:00.031425+0000","last_clean":"2024-01-15T18:05:00.031425+0000","last_became_active":"2024-01-15T18:04:58.856839+0000","last_became_peered":"2024-01-15T18:04:58.856839+0000","last_unstale":"2024-01-15T18:05:00.031425+0000","last_undegraded":"2024-01-15T18:05:00.031425+0000","last_fullsized":"2024-01-15T18:05:00.031425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:53:23.471036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031831+0000","last_change":"2024-01-15T18:04:58.861226+0000","last_active":"2024-01-15T18:05:00.031831+0000","last_peered":"2024-01-15T18:05:00.031831+0000","last_clean":"2024-01-15T18:05:00.031831+0000","last_became_active":"2024-01-15T18:04:58.860964+0000","last_became_peered":"2024-01-15T18:04:58.860964+0000","last_unstale":"2024-01-15T18:05:00.031831+0000","last_undegraded":"2024-01-15T18:05:00.031831+0000","last_fullsized":"2024-01-15T18:05:00.031831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:11:49.903930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032115+0000","last_change":"2024-01-15T18:04:58.856674+0000","last_active":"2024-01-15T18:05:00.032115+0000","last_peered":"2024-01-15T18:05:00.032115+0000","last_clean":"2024-01-15T18:05:00.032115+0000","last_became_active":"2024-01-15T18:04:58.856365+0000","last_became_peered":"2024-01-15T18:04:58.856365+0000","last_unstale":"2024-01-15T18:05:00.032115+0000","last_undegraded":"2024-01-15T18:05:00.032115+0000","last_fullsized":"2024-01-15T18:05:00.032115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:09:03.416070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030934+0000","last_change":"2024-01-15T18:04:58.853185+0000","last_active":"2024-01-15T18:05:00.030934+0000","last_peered":"2024-01-15T18:05:00.030934+0000","last_clean":"2024-01-15T18:05:00.030934+0000","last_became_active":"2024-01-15T18:04:58.852888+0000","last_became_peered":"2024-01-15T18:04:58.852888+0000","last_unstale":"2024-01-15T18:05:00.030934+0000","last_undegraded":"2024-01-15T18:05:00.030934+0000","last_fullsized":"2024-01-15T18:05:00.030934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:10:24.519754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030810+0000","last_change":"2024-01-15T18:04:58.848009+0000","last_active":"2024-01-15T18:05:00.030810+0000","last_peered":"2024-01-15T18:05:00.030810+0000","last_clean":"2024-01-15T18:05:00.030810+0000","last_became_active":"2024-01-15T18:04:58.847765+0000","last_became_peered":"2024-01-15T18:04:58.847765+0000","last_unstale":"2024-01-15T18:05:00.030810+0000","last_undegraded":"2024-01-15T18:05:00.030810+0000","last_fullsized":"2024-01-15T18:05:00.030810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:15:34.376159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847406+0000","last_change":"2024-01-15T18:04:58.847259+0000","last_active":"2024-01-15T18:04:59.847406+0000","last_peered":"2024-01-15T18:04:59.847406+0000","last_clean":"2024-01-15T18:04:59.847406+0000","last_became_active":"2024-01-15T18:04:58.845619+0000","last_became_peered":"2024-01-15T18:04:58.845619+0000","last_unstale":"2024-01-15T18:04:59.847406+0000","last_undegraded":"2024-01-15T18:04:59.847406+0000","last_fullsized":"2024-01-15T18:04:59.847406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:16:30.446952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032165+0000","last_change":"2024-01-15T18:04:58.842389+0000","last_active":"2024-01-15T18:05:00.032165+0000","last_peered":"2024-01-15T18:05:00.032165+0000","last_clean":"2024-01-15T18:05:00.032165+0000","last_became_active":"2024-01-15T18:04:58.841378+0000","last_became_peered":"2024-01-15T18:04:58.841378+0000","last_unstale":"2024-01-15T18:05:00.032165+0000","last_undegraded":"2024-01-15T18:05:00.032165+0000","last_fullsized":"2024-01-15T18:05:00.032165+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:36.680373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.868487+0000","last_change":"2024-01-15T18:04:58.864203+0000","last_active":"2024-01-15T18:04:58.868487+0000","last_peered":"2024-01-15T18:04:58.868487+0000","last_clean":"2024-01-15T18:04:58.868487+0000","last_became_active":"2024-01-15T18:04:58.863956+0000","last_became_peered":"2024-01-15T18:04:58.863956+0000","last_unstale":"2024-01-15T18:04:58.868487+0000","last_undegraded":"2024-01-15T18:04:58.868487+0000","last_fullsized":"2024-01-15T18:04:58.868487+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:02:33.077446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849113+0000","last_change":"2024-01-15T18:04:58.836384+0000","last_active":"2024-01-15T18:04:59.849113+0000","last_peered":"2024-01-15T18:04:59.849113+0000","last_clean":"2024-01-15T18:04:59.849113+0000","last_became_active":"2024-01-15T18:04:58.836081+0000","last_became_peered":"2024-01-15T18:04:58.836081+0000","last_unstale":"2024-01-15T18:04:59.849113+0000","last_undegraded":"2024-01-15T18:04:59.849113+0000","last_fullsized":"2024-01-15T18:04:59.849113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:15:34.376159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.870295+0000","last_change":"2024-01-15T18:04:58.868380+0000","last_active":"2024-01-15T18:04:58.870295+0000","last_peered":"2024-01-15T18:04:58.870295+0000","last_clean":"2024-01-15T18:04:58.870295+0000","last_became_active":"2024-01-15T18:04:58.868103+0000","last_became_peered":"2024-01-15T18:04:58.868103+0000","last_unstale":"2024-01-15T18:04:58.870295+0000","last_undegraded":"2024-01-15T18:04:58.870295+0000","last_fullsized":"2024-01-15T18:04:58.870295+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:11.967136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.865954+0000","last_change":"2024-01-15T18:04:58.858404+0000","last_active":"2024-01-15T18:04:58.865954+0000","last_peered":"2024-01-15T18:04:58.865954+0000","last_clean":"2024-01-15T18:04:58.865954+0000","last_became_active":"2024-01-15T18:04:58.857755+0000","last_became_peered":"2024-01-15T18:04:58.857755+0000","last_unstale":"2024-01-15T18:04:58.865954+0000","last_undegraded":"2024-01-15T18:04:58.865954+0000","last_fullsized":"2024-01-15T18:04:58.865954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:58:42.218891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030907+0000","last_change":"2024-01-15T18:04:58.844392+0000","last_active":"2024-01-15T18:05:00.030907+0000","last_peered":"2024-01-15T18:05:00.030907+0000","last_clean":"2024-01-15T18:05:00.030907+0000","last_became_active":"2024-01-15T18:04:58.844086+0000","last_became_peered":"2024-01-15T18:04:58.844086+0000","last_unstale":"2024-01-15T18:05:00.030907+0000","last_undegraded":"2024-01-15T18:05:00.030907+0000","last_fullsized":"2024-01-15T18:05:00.030907+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:16:30.446952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031976+0000","last_change":"2024-01-15T18:04:58.861264+0000","last_active":"2024-01-15T18:05:00.031976+0000","last_peered":"2024-01-15T18:05:00.031976+0000","last_clean":"2024-01-15T18:05:00.031976+0000","last_became_active":"2024-01-15T18:04:58.860695+0000","last_became_peered":"2024-01-15T18:04:58.860695+0000","last_unstale":"2024-01-15T18:05:00.031976+0000","last_undegraded":"2024-01-15T18:05:00.031976+0000","last_fullsized":"2024-01-15T18:05:00.031976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:47:51.247174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030783+0000","last_change":"2024-01-15T18:04:58.846632+0000","last_active":"2024-01-15T18:05:00.030783+0000","last_peered":"2024-01-15T18:05:00.030783+0000","last_clean":"2024-01-15T18:05:00.030783+0000","last_became_active":"2024-01-15T18:04:58.846359+0000","last_became_peered":"2024-01-15T18:04:58.846359+0000","last_unstale":"2024-01-15T18:05:00.030783+0000","last_undegraded":"2024-01-15T18:05:00.030783+0000","last_fullsized":"2024-01-15T18:05:00.030783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:12:00.696642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.868985+0000","last_change":"2024-01-15T18:04:58.858817+0000","last_active":"2024-01-15T18:04:58.868985+0000","last_peered":"2024-01-15T18:04:58.868985+0000","last_clean":"2024-01-15T18:04:58.868985+0000","last_became_active":"2024-01-15T18:04:58.858539+0000","last_became_peered":"2024-01-15T18:04:58.858539+0000","last_unstale":"2024-01-15T18:04:58.868985+0000","last_undegraded":"2024-01-15T18:04:58.868985+0000","last_fullsized":"2024-01-15T18:04:58.868985+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:16:02.098975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.870284+0000","last_change":"2024-01-15T18:04:58.867761+0000","last_active":"2024-01-15T18:04:58.870284+0000","last_peered":"2024-01-15T18:04:58.870284+0000","last_clean":"2024-01-15T18:04:58.870284+0000","last_became_active":"2024-01-15T18:04:58.867331+0000","last_became_peered":"2024-01-15T18:04:58.867331+0000","last_unstale":"2024-01-15T18:04:58.870284+0000","last_undegraded":"2024-01-15T18:04:58.870284+0000","last_fullsized":"2024-01-15T18:04:58.870284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:14:59.421400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031915+0000","last_change":"2024-01-15T18:04:58.839963+0000","last_active":"2024-01-15T18:05:00.031915+0000","last_peered":"2024-01-15T18:05:00.031915+0000","last_clean":"2024-01-15T18:05:00.031915+0000","last_became_active":"2024-01-15T18:04:58.839597+0000","last_became_peered":"2024-01-15T18:04:58.839597+0000","last_unstale":"2024-01-15T18:05:00.031915+0000","last_undegraded":"2024-01-15T18:05:00.031915+0000","last_fullsized":"2024-01-15T18:05:00.031915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:47.713249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.847369+0000","last_change":"2024-01-15T18:04:58.841274+0000","last_active":"2024-01-15T18:04:58.847369+0000","last_peered":"2024-01-15T18:04:58.847369+0000","last_clean":"2024-01-15T18:04:58.847369+0000","last_became_active":"2024-01-15T18:04:58.840866+0000","last_became_peered":"2024-01-15T18:04:58.840866+0000","last_unstale":"2024-01-15T18:04:58.847369+0000","last_undegraded":"2024-01-15T18:04:58.847369+0000","last_fullsized":"2024-01-15T18:04:58.847369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:11.539894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032528+0000","last_change":"2024-01-15T18:04:58.855334+0000","last_active":"2024-01-15T18:05:00.032528+0000","last_peered":"2024-01-15T18:05:00.032528+0000","last_clean":"2024-01-15T18:05:00.032528+0000","last_became_active":"2024-01-15T18:04:58.855074+0000","last_became_peered":"2024-01-15T18:04:58.855074+0000","last_unstale":"2024-01-15T18:05:00.032528+0000","last_undegraded":"2024-01-15T18:05:00.032528+0000","last_fullsized":"2024-01-15T18:05:00.032528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:44:17.933005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031069+0000","last_change":"2024-01-15T18:04:58.861125+0000","last_active":"2024-01-15T18:05:00.031069+0000","last_peered":"2024-01-15T18:05:00.031069+0000","last_clean":"2024-01-15T18:05:00.031069+0000","last_became_active":"2024-01-15T18:04:58.860819+0000","last_became_peered":"2024-01-15T18:04:58.860819+0000","last_unstale":"2024-01-15T18:05:00.031069+0000","last_undegraded":"2024-01-15T18:05:00.031069+0000","last_fullsized":"2024-01-15T18:05:00.031069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:04:27.769262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849316+0000","last_change":"2024-01-15T18:04:58.837356+0000","last_active":"2024-01-15T18:04:59.849316+0000","last_peered":"2024-01-15T18:04:59.849316+0000","last_clean":"2024-01-15T18:04:59.849316+0000","last_became_active":"2024-01-15T18:04:58.837071+0000","last_became_peered":"2024-01-15T18:04:58.837071+0000","last_unstale":"2024-01-15T18:04:59.849316+0000","last_undegraded":"2024-01-15T18:04:59.849316+0000","last_fullsized":"2024-01-15T18:04:59.849316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:36.680373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032362+0000","last_change":"2024-01-15T18:04:58.858404+0000","last_active":"2024-01-15T18:05:00.032362+0000","last_peered":"2024-01-15T18:05:00.032362+0000","last_clean":"2024-01-15T18:05:00.032362+0000","last_became_active":"2024-01-15T18:04:58.857239+0000","last_became_peered":"2024-01-15T18:04:58.857239+0000","last_unstale":"2024-01-15T18:05:00.032362+0000","last_undegraded":"2024-01-15T18:05:00.032362+0000","last_fullsized":"2024-01-15T18:05:00.032362+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:04:27.769262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.869022+0000","last_change":"2024-01-15T18:04:58.861031+0000","last_active":"2024-01-15T18:04:58.869022+0000","last_peered":"2024-01-15T18:04:58.869022+0000","last_clean":"2024-01-15T18:04:58.869022+0000","last_became_active":"2024-01-15T18:04:58.860718+0000","last_became_peered":"2024-01-15T18:04:58.860718+0000","last_unstale":"2024-01-15T18:04:58.869022+0000","last_undegraded":"2024-01-15T18:04:58.869022+0000","last_fullsized":"2024-01-15T18:04:58.869022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:35:59.532764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.847378+0000","last_change":"2024-01-15T18:04:58.841212+0000","last_active":"2024-01-15T18:04:58.847378+0000","last_peered":"2024-01-15T18:04:58.847378+0000","last_clean":"2024-01-15T18:04:58.847378+0000","last_became_active":"2024-01-15T18:04:58.840913+0000","last_became_peered":"2024-01-15T18:04:58.840913+0000","last_unstale":"2024-01-15T18:04:58.847378+0000","last_undegraded":"2024-01-15T18:04:58.847378+0000","last_fullsized":"2024-01-15T18:04:58.847378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:25:34.152008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030927+0000","last_change":"2024-01-15T18:04:58.847922+0000","last_active":"2024-01-15T18:05:00.030927+0000","last_peered":"2024-01-15T18:05:00.030927+0000","last_clean":"2024-01-15T18:05:00.030927+0000","last_became_active":"2024-01-15T18:04:58.847596+0000","last_became_peered":"2024-01-15T18:04:58.847596+0000","last_unstale":"2024-01-15T18:05:00.030927+0000","last_undegraded":"2024-01-15T18:05:00.030927+0000","last_fullsized":"2024-01-15T18:05:00.030927+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:24:34.638590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030579+0000","last_change":"2024-01-15T18:04:58.866119+0000","last_active":"2024-01-15T18:05:00.030579+0000","last_peered":"2024-01-15T18:05:00.030579+0000","last_clean":"2024-01-15T18:05:00.030579+0000","last_became_active":"2024-01-15T18:04:58.865811+0000","last_became_peered":"2024-01-15T18:04:58.865811+0000","last_unstale":"2024-01-15T18:05:00.030579+0000","last_undegraded":"2024-01-15T18:05:00.030579+0000","last_fullsized":"2024-01-15T18:05:00.030579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:10.719289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847206+0000","last_change":"2024-01-15T18:04:58.839281+0000","last_active":"2024-01-15T18:04:59.847206+0000","last_peered":"2024-01-15T18:04:59.847206+0000","last_clean":"2024-01-15T18:04:59.847206+0000","last_became_active":"2024-01-15T18:04:58.838304+0000","last_became_peered":"2024-01-15T18:04:58.838304+0000","last_unstale":"2024-01-15T18:04:59.847206+0000","last_undegraded":"2024-01-15T18:04:59.847206+0000","last_fullsized":"2024-01-15T18:04:59.847206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:08:43.279198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032260+0000","last_change":"2024-01-15T18:04:57.840527+0000","last_active":"2024-01-15T18:05:00.032260+0000","last_peered":"2024-01-15T18:05:00.032260+0000","last_clean":"2024-01-15T18:05:00.032260+0000","last_became_active":"2024-01-15T18:04:57.840153+0000","last_became_peered":"2024-01-15T18:04:57.840153+0000","last_unstale":"2024-01-15T18:05:00.032260+0000","last_undegraded":"2024-01-15T18:05:00.032260+0000","last_fullsized":"2024-01-15T18:05:00.032260+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:17:50.737228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.869005+0000","last_change":"2024-01-15T18:04:58.859368+0000","last_active":"2024-01-15T18:04:58.869005+0000","last_peered":"2024-01-15T18:04:58.869005+0000","last_clean":"2024-01-15T18:04:58.869005+0000","last_became_active":"2024-01-15T18:04:58.859112+0000","last_became_peered":"2024-01-15T18:04:58.859112+0000","last_unstale":"2024-01-15T18:04:58.869005+0000","last_undegraded":"2024-01-15T18:04:58.869005+0000","last_fullsized":"2024-01-15T18:04:58.869005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:12:46.662813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030936+0000","last_change":"2024-01-15T18:04:57.843777+0000","last_active":"2024-01-15T18:05:00.030936+0000","last_peered":"2024-01-15T18:05:00.030936+0000","last_clean":"2024-01-15T18:05:00.030936+0000","last_became_active":"2024-01-15T18:04:57.843062+0000","last_became_peered":"2024-01-15T18:04:57.843062+0000","last_unstale":"2024-01-15T18:05:00.030936+0000","last_undegraded":"2024-01-15T18:05:00.030936+0000","last_fullsized":"2024-01-15T18:05:00.030936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:58:15.645144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031986+0000","last_change":"2024-01-15T18:04:57.833749+0000","last_active":"2024-01-15T18:05:00.031986+0000","last_peered":"2024-01-15T18:05:00.031986+0000","last_clean":"2024-01-15T18:05:00.031986+0000","last_became_active":"2024-01-15T18:04:57.832655+0000","last_became_peered":"2024-01-15T18:04:57.832655+0000","last_unstale":"2024-01-15T18:05:00.031986+0000","last_undegraded":"2024-01-15T18:05:00.031986+0000","last_fullsized":"2024-01-15T18:05:00.031986+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:42:16.957837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031910+0000","last_change":"2024-01-15T18:04:58.862938+0000","last_active":"2024-01-15T18:05:00.031910+0000","last_peered":"2024-01-15T18:05:00.031910+0000","last_clean":"2024-01-15T18:05:00.031910+0000","last_became_active":"2024-01-15T18:04:58.862733+0000","last_became_peered":"2024-01-15T18:04:58.862733+0000","last_unstale":"2024-01-15T18:05:00.031910+0000","last_undegraded":"2024-01-15T18:05:00.031910+0000","last_fullsized":"2024-01-15T18:05:00.031910+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:10.719289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.831953+0000","last_change":"2024-01-15T18:04:57.828200+0000","last_active":"2024-01-15T18:04:58.831953+0000","last_peered":"2024-01-15T18:04:58.831953+0000","last_clean":"2024-01-15T18:04:58.831953+0000","last_became_active":"2024-01-15T18:04:57.827883+0000","last_became_peered":"2024-01-15T18:04:57.827883+0000","last_unstale":"2024-01-15T18:04:58.831953+0000","last_undegraded":"2024-01-15T18:04:58.831953+0000","last_fullsized":"2024-01-15T18:04:58.831953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:48:16.392862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849021+0000","last_change":"2024-01-15T18:04:58.842387+0000","last_active":"2024-01-15T18:04:59.849021+0000","last_peered":"2024-01-15T18:04:59.849021+0000","last_clean":"2024-01-15T18:04:59.849021+0000","last_became_active":"2024-01-15T18:04:58.842093+0000","last_became_peered":"2024-01-15T18:04:58.842093+0000","last_unstale":"2024-01-15T18:04:59.849021+0000","last_undegraded":"2024-01-15T18:04:59.849021+0000","last_fullsized":"2024-01-15T18:04:59.849021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:47.713249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030983+0000","last_change":"2024-01-15T18:04:57.837812+0000","last_active":"2024-01-15T18:05:00.030983+0000","last_peered":"2024-01-15T18:05:00.030983+0000","last_clean":"2024-01-15T18:05:00.030983+0000","last_became_active":"2024-01-15T18:04:57.837524+0000","last_became_peered":"2024-01-15T18:04:57.837524+0000","last_unstale":"2024-01-15T18:05:00.030983+0000","last_undegraded":"2024-01-15T18:05:00.030983+0000","last_fullsized":"2024-01-15T18:05:00.030983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:37:44.430593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.859275+0000","last_change":"2024-01-15T18:04:58.854115+0000","last_active":"2024-01-15T18:04:58.859275+0000","last_peered":"2024-01-15T18:04:58.859275+0000","last_clean":"2024-01-15T18:04:58.859275+0000","last_became_active":"2024-01-15T18:04:58.853039+0000","last_became_peered":"2024-01-15T18:04:58.853039+0000","last_unstale":"2024-01-15T18:04:58.859275+0000","last_undegraded":"2024-01-15T18:04:58.859275+0000","last_fullsized":"2024-01-15T18:04:58.859275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:26:45.940779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.829328+0000","last_change":"2024-01-15T18:04:57.839191+0000","last_active":"2024-01-15T18:04:58.829328+0000","last_peered":"2024-01-15T18:04:58.829328+0000","last_clean":"2024-01-15T18:04:58.829328+0000","last_became_active":"2024-01-15T18:04:57.838884+0000","last_became_peered":"2024-01-15T18:04:57.838884+0000","last_unstale":"2024-01-15T18:04:58.829328+0000","last_undegraded":"2024-01-15T18:04:58.829328+0000","last_fullsized":"2024-01-15T18:04:58.829328+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:37:39.211256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.871789+0000","last_change":"2024-01-15T18:04:58.860345+0000","last_active":"2024-01-15T18:04:58.871789+0000","last_peered":"2024-01-15T18:04:58.871789+0000","last_clean":"2024-01-15T18:04:58.871789+0000","last_became_active":"2024-01-15T18:04:58.859997+0000","last_became_peered":"2024-01-15T18:04:58.859997+0000","last_unstale":"2024-01-15T18:04:58.871789+0000","last_undegraded":"2024-01-15T18:04:58.871789+0000","last_fullsized":"2024-01-15T18:04:58.871789+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:55:53.031409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849266+0000","last_change":"2024-01-15T18:04:57.835121+0000","last_active":"2024-01-15T18:04:59.849266+0000","last_peered":"2024-01-15T18:04:59.849266+0000","last_clean":"2024-01-15T18:04:59.849266+0000","last_became_active":"2024-01-15T18:04:57.834822+0000","last_became_peered":"2024-01-15T18:04:57.834822+0000","last_unstale":"2024-01-15T18:04:59.849266+0000","last_undegraded":"2024-01-15T18:04:59.849266+0000","last_fullsized":"2024-01-15T18:04:59.849266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:17:50.737228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847554+0000","last_change":"2024-01-15T18:04:58.859000+0000","last_active":"2024-01-15T18:04:59.847554+0000","last_peered":"2024-01-15T18:04:59.847554+0000","last_clean":"2024-01-15T18:04:59.847554+0000","last_became_active":"2024-01-15T18:04:58.854577+0000","last_became_peered":"2024-01-15T18:04:58.854577+0000","last_unstale":"2024-01-15T18:04:59.847554+0000","last_undegraded":"2024-01-15T18:04:59.847554+0000","last_fullsized":"2024-01-15T18:04:59.847554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:24:34.638590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030767+0000","last_change":"2024-01-15T18:04:57.836570+0000","last_active":"2024-01-15T18:05:00.030767+0000","last_peered":"2024-01-15T18:05:00.030767+0000","last_clean":"2024-01-15T18:05:00.030767+0000","last_became_active":"2024-01-15T18:04:57.836027+0000","last_became_peered":"2024-01-15T18:04:57.836027+0000","last_unstale":"2024-01-15T18:05:00.030767+0000","last_undegraded":"2024-01-15T18:05:00.030767+0000","last_fullsized":"2024-01-15T18:05:00.030767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:17:50.737228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031727+0000","last_change":"2024-01-15T18:04:58.872987+0000","last_active":"2024-01-15T18:05:00.031727+0000","last_peered":"2024-01-15T18:05:00.031727+0000","last_clean":"2024-01-15T18:05:00.031727+0000","last_became_active":"2024-01-15T18:04:58.872671+0000","last_became_peered":"2024-01-15T18:04:58.872671+0000","last_unstale":"2024-01-15T18:05:00.031727+0000","last_undegraded":"2024-01-15T18:05:00.031727+0000","last_fullsized":"2024-01-15T18:05:00.031727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:42:12.458758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.871345+0000","last_change":"2024-01-15T18:04:57.839505+0000","last_active":"2024-01-15T18:04:58.871345+0000","last_peered":"2024-01-15T18:04:58.871345+0000","last_clean":"2024-01-15T18:04:58.871345+0000","last_became_active":"2024-01-15T18:04:57.839194+0000","last_became_peered":"2024-01-15T18:04:57.839194+0000","last_unstale":"2024-01-15T18:04:58.871345+0000","last_undegraded":"2024-01-15T18:04:58.871345+0000","last_fullsized":"2024-01-15T18:04:58.871345+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:10.960718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.867419+0000","last_change":"2024-01-15T18:04:58.858064+0000","last_active":"2024-01-15T18:04:58.867419+0000","last_peered":"2024-01-15T18:04:58.867419+0000","last_clean":"2024-01-15T18:04:58.867419+0000","last_became_active":"2024-01-15T18:04:58.857697+0000","last_became_peered":"2024-01-15T18:04:58.857697+0000","last_unstale":"2024-01-15T18:04:58.867419+0000","last_undegraded":"2024-01-15T18:04:58.867419+0000","last_fullsized":"2024-01-15T18:04:58.867419+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:11.539894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.830236+0000","last_change":"2024-01-15T18:04:57.836035+0000","last_active":"2024-01-15T18:04:58.830236+0000","last_peered":"2024-01-15T18:04:58.830236+0000","last_clean":"2024-01-15T18:04:58.830236+0000","last_became_active":"2024-01-15T18:04:57.835722+0000","last_became_peered":"2024-01-15T18:04:57.835722+0000","last_unstale":"2024-01-15T18:04:58.830236+0000","last_undegraded":"2024-01-15T18:04:58.830236+0000","last_fullsized":"2024-01-15T18:04:58.830236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:24:59.847823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031861+0000","last_change":"2024-01-15T18:04:58.852717+0000","last_active":"2024-01-15T18:05:00.031861+0000","last_peered":"2024-01-15T18:05:00.031861+0000","last_clean":"2024-01-15T18:05:00.031861+0000","last_became_active":"2024-01-15T18:04:58.852351+0000","last_became_peered":"2024-01-15T18:04:58.852351+0000","last_unstale":"2024-01-15T18:05:00.031861+0000","last_undegraded":"2024-01-15T18:05:00.031861+0000","last_fullsized":"2024-01-15T18:05:00.031861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:47:03.033950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030528+0000","last_change":"2024-01-15T18:04:57.831525+0000","last_active":"2024-01-15T18:05:00.030528+0000","last_peered":"2024-01-15T18:05:00.030528+0000","last_clean":"2024-01-15T18:05:00.030528+0000","last_became_active":"2024-01-15T18:04:57.831053+0000","last_became_peered":"2024-01-15T18:04:57.831053+0000","last_unstale":"2024-01-15T18:05:00.030528+0000","last_undegraded":"2024-01-15T18:05:00.030528+0000","last_fullsized":"2024-01-15T18:05:00.030528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:24:33.632172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030589+0000","last_change":"2024-01-15T18:04:58.866415+0000","last_active":"2024-01-15T18:05:00.030589+0000","last_peered":"2024-01-15T18:05:00.030589+0000","last_clean":"2024-01-15T18:05:00.030589+0000","last_became_active":"2024-01-15T18:04:58.865778+0000","last_became_peered":"2024-01-15T18:04:58.865778+0000","last_unstale":"2024-01-15T18:05:00.030589+0000","last_undegraded":"2024-01-15T18:05:00.030589+0000","last_fullsized":"2024-01-15T18:05:00.030589+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:47.713249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030622+0000","last_change":"2024-01-15T18:04:57.821766+0000","last_active":"2024-01-15T18:05:00.030622+0000","last_peered":"2024-01-15T18:05:00.030622+0000","last_clean":"2024-01-15T18:05:00.030622+0000","last_became_active":"2024-01-15T18:04:57.820957+0000","last_became_peered":"2024-01-15T18:04:57.820957+0000","last_unstale":"2024-01-15T18:05:00.030622+0000","last_undegraded":"2024-01-15T18:05:00.030622+0000","last_fullsized":"2024-01-15T18:05:00.030622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:08:42.272780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031415+0000","last_change":"2024-01-15T18:04:58.862675+0000","last_active":"2024-01-15T18:05:00.031415+0000","last_peered":"2024-01-15T18:05:00.031415+0000","last_clean":"2024-01-15T18:05:00.031415+0000","last_became_active":"2024-01-15T18:04:58.862420+0000","last_became_peered":"2024-01-15T18:04:58.862420+0000","last_unstale":"2024-01-15T18:05:00.031415+0000","last_undegraded":"2024-01-15T18:05:00.031415+0000","last_fullsized":"2024-01-15T18:05:00.031415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:44:17.933005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847491+0000","last_change":"2024-01-15T18:04:57.839139+0000","last_active":"2024-01-15T18:04:59.847491+0000","last_peered":"2024-01-15T18:04:59.847491+0000","last_clean":"2024-01-15T18:04:59.847491+0000","last_became_active":"2024-01-15T18:04:57.838847+0000","last_became_peered":"2024-01-15T18:04:57.838847+0000","last_unstale":"2024-01-15T18:04:59.847491+0000","last_undegraded":"2024-01-15T18:04:59.847491+0000","last_fullsized":"2024-01-15T18:04:59.847491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:44:30.299838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031909+0000","last_change":"2024-01-15T18:04:58.870909+0000","last_active":"2024-01-15T18:05:00.031909+0000","last_peered":"2024-01-15T18:05:00.031909+0000","last_clean":"2024-01-15T18:05:00.031909+0000","last_became_active":"2024-01-15T18:04:58.868734+0000","last_became_peered":"2024-01-15T18:04:58.868734+0000","last_unstale":"2024-01-15T18:05:00.031909+0000","last_undegraded":"2024-01-15T18:05:00.031909+0000","last_fullsized":"2024-01-15T18:05:00.031909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:58:21.135065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030997+0000","last_change":"2024-01-15T18:04:57.829932+0000","last_active":"2024-01-15T18:05:00.030997+0000","last_peered":"2024-01-15T18:05:00.030997+0000","last_clean":"2024-01-15T18:05:00.030997+0000","last_became_active":"2024-01-15T18:04:57.829636+0000","last_became_peered":"2024-01-15T18:04:57.829636+0000","last_unstale":"2024-01-15T18:05:00.030997+0000","last_undegraded":"2024-01-15T18:05:00.030997+0000","last_fullsized":"2024-01-15T18:05:00.030997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:04:13.242693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032156+0000","last_change":"2024-01-15T18:04:58.869477+0000","last_active":"2024-01-15T18:05:00.032156+0000","last_peered":"2024-01-15T18:05:00.032156+0000","last_clean":"2024-01-15T18:05:00.032156+0000","last_became_active":"2024-01-15T18:04:58.869133+0000","last_became_peered":"2024-01-15T18:04:58.869133+0000","last_unstale":"2024-01-15T18:05:00.032156+0000","last_undegraded":"2024-01-15T18:05:00.032156+0000","last_fullsized":"2024-01-15T18:05:00.032156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:39:08.175871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849313+0000","last_change":"2024-01-15T18:04:57.830731+0000","last_active":"2024-01-15T18:04:59.849313+0000","last_peered":"2024-01-15T18:04:59.849313+0000","last_clean":"2024-01-15T18:04:59.849313+0000","last_became_active":"2024-01-15T18:04:57.830234+0000","last_became_peered":"2024-01-15T18:04:57.830234+0000","last_unstale":"2024-01-15T18:04:59.849313+0000","last_undegraded":"2024-01-15T18:04:59.849313+0000","last_fullsized":"2024-01-15T18:04:59.849313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:37:44.430593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.853920+0000","last_change":"2024-01-15T18:04:58.843965+0000","last_active":"2024-01-15T18:04:58.853920+0000","last_peered":"2024-01-15T18:04:58.853920+0000","last_clean":"2024-01-15T18:04:58.853920+0000","last_became_active":"2024-01-15T18:04:58.841866+0000","last_became_peered":"2024-01-15T18:04:58.841866+0000","last_unstale":"2024-01-15T18:04:58.853920+0000","last_undegraded":"2024-01-15T18:04:58.853920+0000","last_fullsized":"2024-01-15T18:04:58.853920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:02:33.077446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030724+0000","last_change":"2024-01-15T18:04:57.836644+0000","last_active":"2024-01-15T18:05:00.030724+0000","last_peered":"2024-01-15T18:05:00.030724+0000","last_clean":"2024-01-15T18:05:00.030724+0000","last_became_active":"2024-01-15T18:04:57.835872+0000","last_became_peered":"2024-01-15T18:04:57.835872+0000","last_unstale":"2024-01-15T18:05:00.030724+0000","last_undegraded":"2024-01-15T18:05:00.030724+0000","last_fullsized":"2024-01-15T18:05:00.030724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:44:30.299838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031872+0000","last_change":"2024-01-15T18:04:58.857962+0000","last_active":"2024-01-15T18:05:00.031872+0000","last_peered":"2024-01-15T18:05:00.031872+0000","last_clean":"2024-01-15T18:05:00.031872+0000","last_became_active":"2024-01-15T18:04:58.857644+0000","last_became_peered":"2024-01-15T18:04:58.857644+0000","last_unstale":"2024-01-15T18:05:00.031872+0000","last_undegraded":"2024-01-15T18:05:00.031872+0000","last_fullsized":"2024-01-15T18:05:00.031872+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:02:19.202437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.836746+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:04:58.836746+0000","last_peered":"2024-01-15T18:04:58.836746+0000","last_clean":"2024-01-15T18:04:58.836746+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:04:58.836746+0000","last_undegraded":"2024-01-15T18:04:58.836746+0000","last_fullsized":"2024-01-15T18:04:58.836746+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.827351+0000","last_change":"2024-01-15T18:04:57.829926+0000","last_active":"2024-01-15T18:04:58.827351+0000","last_peered":"2024-01-15T18:04:58.827351+0000","last_clean":"2024-01-15T18:04:58.827351+0000","last_became_active":"2024-01-15T18:04:57.829578+0000","last_became_peered":"2024-01-15T18:04:57.829578+0000","last_unstale":"2024-01-15T18:04:58.827351+0000","last_undegraded":"2024-01-15T18:04:58.827351+0000","last_fullsized":"2024-01-15T18:04:58.827351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:55:52.024992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032132+0000","last_change":"2024-01-15T18:04:58.862009+0000","last_active":"2024-01-15T18:05:00.032132+0000","last_peered":"2024-01-15T18:05:00.032132+0000","last_clean":"2024-01-15T18:05:00.032132+0000","last_became_active":"2024-01-15T18:04:58.861745+0000","last_became_peered":"2024-01-15T18:04:58.861745+0000","last_unstale":"2024-01-15T18:05:00.032132+0000","last_undegraded":"2024-01-15T18:05:00.032132+0000","last_fullsized":"2024-01-15T18:05:00.032132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:15:34.376159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031331+0000","last_change":"2024-01-15T18:04:57.842536+0000","last_active":"2024-01-15T18:05:00.031331+0000","last_peered":"2024-01-15T18:05:00.031331+0000","last_clean":"2024-01-15T18:05:00.031331+0000","last_became_active":"2024-01-15T18:04:57.842236+0000","last_became_peered":"2024-01-15T18:04:57.842236+0000","last_unstale":"2024-01-15T18:05:00.031331+0000","last_undegraded":"2024-01-15T18:05:00.031331+0000","last_fullsized":"2024-01-15T18:05:00.031331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:24:33.632172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.867604+0000","last_change":"2024-01-15T18:04:58.859545+0000","last_active":"2024-01-15T18:04:58.867604+0000","last_peered":"2024-01-15T18:04:58.867604+0000","last_clean":"2024-01-15T18:04:58.867604+0000","last_became_active":"2024-01-15T18:04:58.859282+0000","last_became_peered":"2024-01-15T18:04:58.859282+0000","last_unstale":"2024-01-15T18:04:58.867604+0000","last_undegraded":"2024-01-15T18:04:58.867604+0000","last_fullsized":"2024-01-15T18:04:58.867604+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:26:45.940779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030970+0000","last_change":"2024-01-15T18:04:57.826351+0000","last_active":"2024-01-15T18:05:00.030970+0000","last_peered":"2024-01-15T18:05:00.030970+0000","last_clean":"2024-01-15T18:05:00.030970+0000","last_became_active":"2024-01-15T18:04:57.825983+0000","last_became_peered":"2024-01-15T18:04:57.825983+0000","last_unstale":"2024-01-15T18:05:00.030970+0000","last_undegraded":"2024-01-15T18:05:00.030970+0000","last_fullsized":"2024-01-15T18:05:00.030970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:30:11.561882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849021+0000","last_change":"2024-01-15T18:04:58.834900+0000","last_active":"2024-01-15T18:04:59.849021+0000","last_peered":"2024-01-15T18:04:59.849021+0000","last_clean":"2024-01-15T18:04:59.849021+0000","last_became_active":"2024-01-15T18:04:58.834561+0000","last_became_peered":"2024-01-15T18:04:58.834561+0000","last_unstale":"2024-01-15T18:04:59.849021+0000","last_undegraded":"2024-01-15T18:04:59.849021+0000","last_fullsized":"2024-01-15T18:04:59.849021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:10.719289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.830421+0000","last_change":"2024-01-15T18:04:57.840353+0000","last_active":"2024-01-15T18:04:58.830421+0000","last_peered":"2024-01-15T18:04:58.830421+0000","last_clean":"2024-01-15T18:04:58.830421+0000","last_became_active":"2024-01-15T18:04:57.840032+0000","last_became_peered":"2024-01-15T18:04:57.840032+0000","last_unstale":"2024-01-15T18:04:58.830421+0000","last_undegraded":"2024-01-15T18:04:58.830421+0000","last_fullsized":"2024-01-15T18:04:58.830421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:21:05.984228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847638+0000","last_change":"2024-01-15T18:04:58.862393+0000","last_active":"2024-01-15T18:04:59.847638+0000","last_peered":"2024-01-15T18:04:59.847638+0000","last_clean":"2024-01-15T18:04:59.847638+0000","last_became_active":"2024-01-15T18:04:58.861776+0000","last_became_peered":"2024-01-15T18:04:58.861776+0000","last_unstale":"2024-01-15T18:04:59.847638+0000","last_undegraded":"2024-01-15T18:04:59.847638+0000","last_fullsized":"2024-01-15T18:04:59.847638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:58:16.651562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849271+0000","last_change":"2024-01-15T18:04:57.832247+0000","last_active":"2024-01-15T18:04:59.849271+0000","last_peered":"2024-01-15T18:04:59.849271+0000","last_clean":"2024-01-15T18:04:59.849271+0000","last_became_active":"2024-01-15T18:04:57.831962+0000","last_became_peered":"2024-01-15T18:04:57.831962+0000","last_unstale":"2024-01-15T18:04:59.849271+0000","last_undegraded":"2024-01-15T18:04:59.849271+0000","last_fullsized":"2024-01-15T18:04:59.849271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:08:42.272780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.850248+0000","last_change":"2024-01-15T18:04:58.842320+0000","last_active":"2024-01-15T18:04:58.850248+0000","last_peered":"2024-01-15T18:04:58.850248+0000","last_clean":"2024-01-15T18:04:58.850248+0000","last_became_active":"2024-01-15T18:04:58.841887+0000","last_became_peered":"2024-01-15T18:04:58.841887+0000","last_unstale":"2024-01-15T18:04:58.850248+0000","last_undegraded":"2024-01-15T18:04:58.850248+0000","last_fullsized":"2024-01-15T18:04:58.850248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:02:12.562587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032034+0000","last_change":"2024-01-15T18:04:57.843600+0000","last_active":"2024-01-15T18:05:00.032034+0000","last_peered":"2024-01-15T18:05:00.032034+0000","last_clean":"2024-01-15T18:05:00.032034+0000","last_became_active":"2024-01-15T18:04:57.843179+0000","last_became_peered":"2024-01-15T18:04:57.843179+0000","last_unstale":"2024-01-15T18:05:00.032034+0000","last_undegraded":"2024-01-15T18:05:00.032034+0000","last_fullsized":"2024-01-15T18:05:00.032034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:10:23.513336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849104+0000","last_change":"2024-01-15T18:04:58.834860+0000","last_active":"2024-01-15T18:04:59.849104+0000","last_peered":"2024-01-15T18:04:59.849104+0000","last_clean":"2024-01-15T18:04:59.849104+0000","last_became_active":"2024-01-15T18:04:58.834570+0000","last_became_peered":"2024-01-15T18:04:58.834570+0000","last_unstale":"2024-01-15T18:04:59.849104+0000","last_undegraded":"2024-01-15T18:04:59.849104+0000","last_fullsized":"2024-01-15T18:04:59.849104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:12:00.696642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849515+0000","last_change":"2024-01-15T18:04:57.832391+0000","last_active":"2024-01-15T18:04:59.849515+0000","last_peered":"2024-01-15T18:04:59.849515+0000","last_clean":"2024-01-15T18:04:59.849515+0000","last_became_active":"2024-01-15T18:04:57.832005+0000","last_became_peered":"2024-01-15T18:04:57.832005+0000","last_unstale":"2024-01-15T18:04:59.849515+0000","last_undegraded":"2024-01-15T18:04:59.849515+0000","last_fullsized":"2024-01-15T18:04:59.849515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:24:33.632172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032511+0000","last_change":"2024-01-15T18:04:58.863289+0000","last_active":"2024-01-15T18:05:00.032511+0000","last_peered":"2024-01-15T18:05:00.032511+0000","last_clean":"2024-01-15T18:05:00.032511+0000","last_became_active":"2024-01-15T18:04:58.862985+0000","last_became_peered":"2024-01-15T18:04:58.862985+0000","last_unstale":"2024-01-15T18:05:00.032511+0000","last_undegraded":"2024-01-15T18:05:00.032511+0000","last_fullsized":"2024-01-15T18:05:00.032511+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:48:27.499654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031100+0000","last_change":"2024-01-15T18:04:57.835018+0000","last_active":"2024-01-15T18:05:00.031100+0000","last_peered":"2024-01-15T18:05:00.031100+0000","last_clean":"2024-01-15T18:05:00.031100+0000","last_became_active":"2024-01-15T18:04:57.834706+0000","last_became_peered":"2024-01-15T18:04:57.834706+0000","last_unstale":"2024-01-15T18:05:00.031100+0000","last_undegraded":"2024-01-15T18:05:00.031100+0000","last_fullsized":"2024-01-15T18:05:00.031100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:11:48.897513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.871874+0000","last_change":"2024-01-15T18:04:58.866805+0000","last_active":"2024-01-15T18:04:58.871874+0000","last_peered":"2024-01-15T18:04:58.871874+0000","last_clean":"2024-01-15T18:04:58.871874+0000","last_became_active":"2024-01-15T18:04:58.866475+0000","last_became_peered":"2024-01-15T18:04:58.866475+0000","last_unstale":"2024-01-15T18:04:58.871874+0000","last_undegraded":"2024-01-15T18:04:58.871874+0000","last_fullsized":"2024-01-15T18:04:58.871874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:14:59.421400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031326+0000","last_change":"2024-01-15T18:04:57.839780+0000","last_active":"2024-01-15T18:05:00.031326+0000","last_peered":"2024-01-15T18:05:00.031326+0000","last_clean":"2024-01-15T18:05:00.031326+0000","last_became_active":"2024-01-15T18:04:57.839392+0000","last_became_peered":"2024-01-15T18:04:57.839392+0000","last_unstale":"2024-01-15T18:05:00.031326+0000","last_undegraded":"2024-01-15T18:05:00.031326+0000","last_fullsized":"2024-01-15T18:05:00.031326+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:16:29.440534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847071+0000","last_change":"2024-01-15T18:04:58.835003+0000","last_active":"2024-01-15T18:04:59.847071+0000","last_peered":"2024-01-15T18:04:59.847071+0000","last_clean":"2024-01-15T18:04:59.847071+0000","last_became_active":"2024-01-15T18:04:58.834678+0000","last_became_peered":"2024-01-15T18:04:58.834678+0000","last_unstale":"2024-01-15T18:04:59.847071+0000","last_undegraded":"2024-01-15T18:04:59.847071+0000","last_fullsized":"2024-01-15T18:04:59.847071+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:04:14.249111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849091+0000","last_change":"2024-01-15T18:04:57.837454+0000","last_active":"2024-01-15T18:04:59.849091+0000","last_peered":"2024-01-15T18:04:59.849091+0000","last_clean":"2024-01-15T18:04:59.849091+0000","last_became_active":"2024-01-15T18:04:57.836947+0000","last_became_peered":"2024-01-15T18:04:57.836947+0000","last_unstale":"2024-01-15T18:04:59.849091+0000","last_undegraded":"2024-01-15T18:04:59.849091+0000","last_fullsized":"2024-01-15T18:04:59.849091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:11:48.897513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031977+0000","last_change":"2024-01-15T18:04:58.854684+0000","last_active":"2024-01-15T18:05:00.031977+0000","last_peered":"2024-01-15T18:05:00.031977+0000","last_clean":"2024-01-15T18:05:00.031977+0000","last_became_active":"2024-01-15T18:04:58.844371+0000","last_became_peered":"2024-01-15T18:04:58.844371+0000","last_unstale":"2024-01-15T18:05:00.031977+0000","last_undegraded":"2024-01-15T18:05:00.031977+0000","last_fullsized":"2024-01-15T18:05:00.031977+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:12:00.696642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849194+0000","last_change":"2024-01-15T18:04:57.837432+0000","last_active":"2024-01-15T18:04:59.849194+0000","last_peered":"2024-01-15T18:04:59.849194+0000","last_clean":"2024-01-15T18:04:59.849194+0000","last_became_active":"2024-01-15T18:04:57.837105+0000","last_became_peered":"2024-01-15T18:04:57.837105+0000","last_unstale":"2024-01-15T18:04:59.849194+0000","last_undegraded":"2024-01-15T18:04:59.849194+0000","last_fullsized":"2024-01-15T18:04:59.849194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:04:13.242693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032216+0000","last_change":"2024-01-15T18:04:58.859567+0000","last_active":"2024-01-15T18:05:00.032216+0000","last_peered":"2024-01-15T18:05:00.032216+0000","last_clean":"2024-01-15T18:05:00.032216+0000","last_became_active":"2024-01-15T18:04:58.859199+0000","last_became_peered":"2024-01-15T18:04:58.859199+0000","last_unstale":"2024-01-15T18:05:00.032216+0000","last_undegraded":"2024-01-15T18:05:00.032216+0000","last_fullsized":"2024-01-15T18:05:00.032216+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:10:24.519754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031791+0000","last_change":"2024-01-15T18:04:57.843450+0000","last_active":"2024-01-15T18:05:00.031791+0000","last_peered":"2024-01-15T18:05:00.031791+0000","last_clean":"2024-01-15T18:05:00.031791+0000","last_became_active":"2024-01-15T18:04:57.841883+0000","last_became_peered":"2024-01-15T18:04:57.841883+0000","last_unstale":"2024-01-15T18:05:00.031791+0000","last_undegraded":"2024-01-15T18:05:00.031791+0000","last_fullsized":"2024-01-15T18:05:00.031791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:17:50.737228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030851+0000","last_change":"2024-01-15T18:04:58.845515+0000","last_active":"2024-01-15T18:05:00.030851+0000","last_peered":"2024-01-15T18:05:00.030851+0000","last_clean":"2024-01-15T18:05:00.030851+0000","last_became_active":"2024-01-15T18:04:58.845242+0000","last_became_peered":"2024-01-15T18:04:58.845242+0000","last_unstale":"2024-01-15T18:05:00.030851+0000","last_undegraded":"2024-01-15T18:05:00.030851+0000","last_fullsized":"2024-01-15T18:05:00.030851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:36.680373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.846973+0000","last_change":"2024-01-15T18:04:58.840618+0000","last_active":"2024-01-15T18:04:58.846973+0000","last_peered":"2024-01-15T18:04:58.846973+0000","last_clean":"2024-01-15T18:04:58.846973+0000","last_became_active":"2024-01-15T18:04:58.840306+0000","last_became_peered":"2024-01-15T18:04:58.840306+0000","last_unstale":"2024-01-15T18:04:58.846973+0000","last_undegraded":"2024-01-15T18:04:58.846973+0000","last_fullsized":"2024-01-15T18:04:58.846973+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:27:32.756552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031572+0000","last_change":"2024-01-15T18:04:57.842676+0000","last_active":"2024-01-15T18:05:00.031572+0000","last_peered":"2024-01-15T18:05:00.031572+0000","last_clean":"2024-01-15T18:05:00.031572+0000","last_became_active":"2024-01-15T18:04:57.842402+0000","last_became_peered":"2024-01-15T18:04:57.842402+0000","last_unstale":"2024-01-15T18:05:00.031572+0000","last_undegraded":"2024-01-15T18:05:00.031572+0000","last_fullsized":"2024-01-15T18:05:00.031572+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:09.712871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.847833+0000","last_change":"2024-01-15T18:04:58.837433+0000","last_active":"2024-01-15T18:04:58.847833+0000","last_peered":"2024-01-15T18:04:58.847833+0000","last_clean":"2024-01-15T18:04:58.847833+0000","last_became_active":"2024-01-15T18:04:58.836989+0000","last_became_peered":"2024-01-15T18:04:58.836989+0000","last_unstale":"2024-01-15T18:04:58.847833+0000","last_undegraded":"2024-01-15T18:04:58.847833+0000","last_fullsized":"2024-01-15T18:04:58.847833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:36:10.997513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031908+0000","last_change":"2024-01-15T18:04:57.834129+0000","last_active":"2024-01-15T18:05:00.031908+0000","last_peered":"2024-01-15T18:05:00.031908+0000","last_clean":"2024-01-15T18:05:00.031908+0000","last_became_active":"2024-01-15T18:04:57.833804+0000","last_became_peered":"2024-01-15T18:04:57.833804+0000","last_unstale":"2024-01-15T18:05:00.031908+0000","last_undegraded":"2024-01-15T18:05:00.031908+0000","last_fullsized":"2024-01-15T18:05:00.031908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:11:59.690224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.871769+0000","last_change":"2024-01-15T18:04:58.860153+0000","last_active":"2024-01-15T18:04:58.871769+0000","last_peered":"2024-01-15T18:04:58.871769+0000","last_clean":"2024-01-15T18:04:58.871769+0000","last_became_active":"2024-01-15T18:04:58.855401+0000","last_became_peered":"2024-01-15T18:04:58.855401+0000","last_unstale":"2024-01-15T18:04:58.871769+0000","last_undegraded":"2024-01-15T18:04:58.871769+0000","last_fullsized":"2024-01-15T18:04:58.871769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:02:12.562587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032517+0000","last_change":"2024-01-15T18:04:57.830136+0000","last_active":"2024-01-15T18:05:00.032517+0000","last_peered":"2024-01-15T18:05:00.032517+0000","last_clean":"2024-01-15T18:05:00.032517+0000","last_became_active":"2024-01-15T18:04:57.829333+0000","last_became_peered":"2024-01-15T18:04:57.829333+0000","last_unstale":"2024-01-15T18:05:00.032517+0000","last_undegraded":"2024-01-15T18:05:00.032517+0000","last_fullsized":"2024-01-15T18:05:00.032517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:37:44.430593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032421+0000","last_change":"2024-01-15T18:04:58.855872+0000","last_active":"2024-01-15T18:05:00.032421+0000","last_peered":"2024-01-15T18:05:00.032421+0000","last_clean":"2024-01-15T18:05:00.032421+0000","last_became_active":"2024-01-15T18:04:58.854897+0000","last_became_peered":"2024-01-15T18:04:58.854897+0000","last_unstale":"2024-01-15T18:05:00.032421+0000","last_undegraded":"2024-01-15T18:05:00.032421+0000","last_fullsized":"2024-01-15T18:05:00.032421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:33:52.636110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.878477+0000","last_change":"2024-01-15T18:04:57.843229+0000","last_active":"2024-01-15T18:04:58.878477+0000","last_peered":"2024-01-15T18:04:58.878477+0000","last_clean":"2024-01-15T18:04:58.878477+0000","last_became_active":"2024-01-15T18:04:57.842720+0000","last_became_peered":"2024-01-15T18:04:57.842720+0000","last_unstale":"2024-01-15T18:04:58.878477+0000","last_undegraded":"2024-01-15T18:04:58.878477+0000","last_fullsized":"2024-01-15T18:04:58.878477+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:25:33.145590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509445,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":806912,"data_stored":663105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80400000000000005}]}]},{"osd":6,"up_from":15,"seq":64424509445,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":73767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.876}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]}]},{"osd":4,"up_from":15,"seq":64424509445,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":65981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]}]},{"osd":3,"up_from":15,"seq":64424509444,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":67467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]}]},{"osd":2,"up_from":15,"seq":64424509444,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":65855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71799999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]}]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]}]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":184320,"data_stored":59080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":802816,"data_stored":656861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:05:03.332 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-15T18:05:03.332 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-15T18:05:03.332 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-15T18:05:03.332 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-15T18:05:03.333 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-15T18:05:03.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.508+0000 7fb69c200640 1 Processor -- start 2024-01-15T18:05:03.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.508+0000 7fb69c200640 1 -- start start 2024-01-15T18:05:03.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.508+0000 7fb69c200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6940cf9b0 0x7fb6940b10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:03.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.508+0000 7fb69c200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fb6940cedc0 0x7fb6940b1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:03.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.508+0000 7fb69c200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6940b38b0 con 0x7fb69415e7d0 2024-01-15T18:05:03.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.508+0000 7fb69c200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6940cd550 con 0x7fb6940cedc0 2024-01-15T18:05:03.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.508+0000 7fb69c200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6940cd270 con 0x7fb6940cf9b0 2024-01-15T18:05:03.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.508+0000 7fb698e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6940cf9b0 0x7fb6940b10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:03.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.508+0000 7fb69a200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fb6940cedc0 0x7fb6940b1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:03.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.508+0000 7fb698e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6940cf9b0 0x7fb6940b10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:56974/0 (socket says 172.21.15.27:56974) 2024-01-15T18:05:03.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.508+0000 7fb698e00640 1 -- 172.21.15.27:0/3408612970 learned_addr learned my addr 172.21.15.27:0/3408612970 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:03.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.508+0000 7fb69a200640 1 -- 172.21.15.27:0/3408612970 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6940cf9b0 msgr2=0x7fb6940b10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:03.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.508+0000 7fb69a200640 1 --2- 172.21.15.27:0/3408612970 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6940cf9b0 0x7fb6940b10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.508+0000 7fb69a200640 1 -- 172.21.15.27:0/3408612970 >> v1:172.21.15.93:6789/0 conn(0x7fb69415e7d0 legacy=0x7fb6940cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:03.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.508+0000 7fb69a200640 1 -- 172.21.15.27:0/3408612970 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6940b35d0 con 0x7fb6940cedc0 2024-01-15T18:05:03.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.508+0000 7fb698e00640 1 --2- 172.21.15.27:0/3408612970 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6940cf9b0 0x7fb6940b10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:03.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.512+0000 7fb69a200640 1 --2- 172.21.15.27:0/3408612970 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb6940cedc0 0x7fb6940b1600 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009d90 tx=0x7fb68c01c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0ca2395caa77204 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:03.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.512+0000 7fb693e00640 1 -- 172.21.15.27:0/3408612970 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c0a5900 con 0x7fb6940cedc0 2024-01-15T18:05:03.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.512+0000 7fb693e00640 1 -- 172.21.15.27:0/3408612970 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb68c0a5a60 con 0x7fb6940cedc0 2024-01-15T18:05:03.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.512+0000 7fb693e00640 1 -- 172.21.15.27:0/3408612970 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c0a5df0 con 0x7fb6940cedc0 2024-01-15T18:05:03.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.512+0000 7fb69c200640 1 -- 172.21.15.27:0/3408612970 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb6940cedc0 msgr2=0x7fb6940b1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:03.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.512+0000 7fb69c200640 1 --2- 172.21.15.27:0/3408612970 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb6940cedc0 0x7fb6940b1600 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009d90 tx=0x7fb68c01c950 comp rx=0 tx=0).stop 2024-01-15T18:05:03.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.512+0000 7fb69c200640 1 -- 172.21.15.27:0/3408612970 shutdown_connections 2024-01-15T18:05:03.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.512+0000 7fb69c200640 1 --2- 172.21.15.27:0/3408612970 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb6940cedc0 0x7fb6940b1600 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.512+0000 7fb69c200640 1 --2- 172.21.15.27:0/3408612970 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6940cf9b0 0x7fb6940b10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.512+0000 7fb69c200640 1 -- 172.21.15.27:0/3408612970 >> 172.21.15.27:0/3408612970 conn(0x7fb6940b2a10 msgr2=0x7fb6940b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:03.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.512+0000 7fb69c200640 1 -- 172.21.15.27:0/3408612970 shutdown_connections 2024-01-15T18:05:03.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.512+0000 7fb69c200640 1 -- 172.21.15.27:0/3408612970 wait complete. 2024-01-15T18:05:03.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.516+0000 7fb69c200640 1 Processor -- start 2024-01-15T18:05:03.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.516+0000 7fb69c200640 1 -- start start 2024-01-15T18:05:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.516+0000 7fb69c200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6940cedc0 0x7fb694151f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.516+0000 7fb69c200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6940cf9b0 0x7fb6940ca530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.516+0000 7fb69c200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb69415e7d0 0x7fb69406c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.516+0000 7fb69c200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69406d060 con 0x7fb6940cedc0 2024-01-15T18:05:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.516+0000 7fb69c200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb694152450 con 0x7fb69415e7d0 2024-01-15T18:05:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.516+0000 7fb69c200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6940c9490 con 0x7fb6940cf9b0 2024-01-15T18:05:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.516+0000 7fb699800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6940cedc0 0x7fb694151f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.516+0000 7fb699800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6940cedc0 0x7fb694151f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46636/0 (socket says 172.21.15.27:46636) 2024-01-15T18:05:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.516+0000 7fb698e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6940cf9b0 0x7fb6940ca530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.516+0000 7fb699800640 1 -- 172.21.15.27:0/216765498 learned_addr learned my addr 172.21.15.27:0/216765498 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.516+0000 7fb69a200640 1 --2- 172.21.15.27:0/216765498 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb69415e7d0 0x7fb69406c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.516+0000 7fb699800640 1 -- 172.21.15.27:0/216765498 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6940cf9b0 msgr2=0x7fb6940ca530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.516+0000 7fb699800640 1 --2- 172.21.15.27:0/216765498 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6940cf9b0 0x7fb6940ca530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.516+0000 7fb699800640 1 -- 172.21.15.27:0/216765498 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb69415e7d0 msgr2=0x7fb69406c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:03.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.516+0000 7fb699800640 1 --2- 172.21.15.27:0/216765498 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb69415e7d0 0x7fb69406c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.516+0000 7fb699800640 1 -- 172.21.15.27:0/216765498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c0094d0 con 0x7fb6940cedc0 2024-01-15T18:05:03.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.516+0000 7fb698e00640 1 --2- 172.21.15.27:0/216765498 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6940cf9b0 0x7fb6940ca530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:03.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.516+0000 7fb699800640 1 --2- 172.21.15.27:0/216765498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6940cedc0 0x7fb694151f10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb68400c420 tx=0x7fb68400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:03.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.516+0000 7fb691600640 1 -- 172.21.15.27:0/216765498 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6840052d0 con 0x7fb6940cedc0 2024-01-15T18:05:03.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.516+0000 7fb691600640 1 -- 172.21.15.27:0/216765498 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb684005430 con 0x7fb6940cedc0 2024-01-15T18:05:03.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.516+0000 7fb69c200640 1 -- 172.21.15.27:0/216765498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb68c009a70 con 0x7fb6940cedc0 2024-01-15T18:05:03.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.516+0000 7fb691600640 1 -- 172.21.15.27:0/216765498 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb684007260 con 0x7fb6940cedc0 2024-01-15T18:05:03.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.516+0000 7fb69c200640 1 -- 172.21.15.27:0/216765498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6940c9a50 con 0x7fb6940cedc0 2024-01-15T18:05:03.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.520+0000 7fb69c200640 1 -- 172.21.15.27:0/216765498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb69404e850 con 0x7fb6940cedc0 2024-01-15T18:05:03.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.520+0000 7fb691600640 1 -- 172.21.15.27:0/216765498 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fb6840073c0 con 0x7fb6940cedc0 2024-01-15T18:05:03.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.520+0000 7fb691600640 1 --2- 172.21.15.27:0/216765498 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb66c0668f0 0x7fb66c068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:03.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.520+0000 7fb691600640 1 -- 172.21.15.27:0/216765498 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb6840838b0 con 0x7fb6940cedc0 2024-01-15T18:05:03.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.520+0000 7fb698e00640 1 --2- 172.21.15.27:0/216765498 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb66c0668f0 0x7fb66c068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:03.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.520+0000 7fb698e00640 1 --2- 172.21.15.27:0/216765498 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb66c0668f0 0x7fb66c068db0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb6940c8d40 tx=0x7fb67c009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:03.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.524+0000 7fb691600640 1 -- 172.21.15.27:0/216765498 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb684050a80 con 0x7fb6940cedc0 2024-01-15T18:05:03.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.724+0000 7fb69c200640 1 -- 172.21.15.27:0/216765498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fb694067740 con 0x7fb6940cedc0 2024-01-15T18:05:03.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.728+0000 7fb691600640 1 -- 172.21.15.27:0/216765498 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+164957 (secure 0 0 0) 0x7fb6840540a0 con 0x7fb6940cedc0 2024-01-15T18:05:03.732 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:03.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.728+0000 7fb69c200640 1 -- 172.21.15.27:0/216765498 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb66c0668f0 msgr2=0x7fb66c068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:03.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.728+0000 7fb69c200640 1 --2- 172.21.15.27:0/216765498 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb66c0668f0 0x7fb66c068db0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb6940c8d40 tx=0x7fb67c009240 comp rx=0 tx=0).stop 2024-01-15T18:05:03.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.728+0000 7fb69c200640 1 -- 172.21.15.27:0/216765498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6940cedc0 msgr2=0x7fb694151f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:03.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.728+0000 7fb69c200640 1 --2- 172.21.15.27:0/216765498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6940cedc0 0x7fb694151f10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb68400c420 tx=0x7fb68400c8e0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.728+0000 7fb69c200640 1 -- 172.21.15.27:0/216765498 shutdown_connections 2024-01-15T18:05:03.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.728+0000 7fb69c200640 1 --2- 172.21.15.27:0/216765498 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb66c0668f0 0x7fb66c068db0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.728+0000 7fb69c200640 1 --2- 172.21.15.27:0/216765498 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb69415e7d0 0x7fb69406c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.728+0000 7fb69c200640 1 --2- 172.21.15.27:0/216765498 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6940cf9b0 0x7fb6940ca530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.728+0000 7fb69c200640 1 --2- 172.21.15.27:0/216765498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6940cedc0 0x7fb694151f10 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.728+0000 7fb69c200640 1 -- 172.21.15.27:0/216765498 >> 172.21.15.27:0/216765498 conn(0x7fb6940b2a10 msgr2=0x7fb6940cc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:03.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.728+0000 7fb69c200640 1 -- 172.21.15.27:0/216765498 shutdown_connections 2024-01-15T18:05:03.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.728+0000 7fb69c200640 1 -- 172.21.15.27:0/216765498 wait complete. 2024-01-15T18:05:03.752 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":5,"active_gid":4106,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6832","nonce":1477920209},{"type":"v1","addr":"172.21.15.93:6833","nonce":1477920209}]},"active_addr":"172.21.15.93:6833/1477920209","active_change":"2024-01-15T18:04:38.123999+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4104,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"clone_delay_every_regfile":{"name":"clone_delay_every_regfile","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Sleep for specified seconds after copying a regfile","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"clone_delay_every_regfile":{"name":"clone_delay_every_regfile","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Sleep for specified seconds after copying a regfile","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.93:0","nonce":3790795822}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.93:0","nonce":3380164630}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.93:0","nonce":885365292}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.93:0","nonce":1876951210}]}]} 2024-01-15T18:05:03.756 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-15T18:05:03.756 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-15T18:05:03.756 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:03.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.896+0000 7fcd8de00640 1 Processor -- start 2024-01-15T18:05:03.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.896+0000 7fcd8de00640 1 -- start start 2024-01-15T18:05:03.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.896+0000 7fcd8de00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fcd880ce390 0x7fcd880cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:03.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.896+0000 7fcd8de00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd880d0f80 0x7fcd880d13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:03.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.896+0000 7fcd8de00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcd880cceb0 0x7fcd880cd320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:03.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.896+0000 7fcd8de00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd880ce770 con 0x7fcd880cceb0 2024-01-15T18:05:03.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.896+0000 7fcd8de00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd880d0af0 con 0x7fcd880ce390 2024-01-15T18:05:03.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.896+0000 7fcd8de00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd880cd860 con 0x7fcd880d0f80 2024-01-15T18:05:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.896+0000 7fcd87400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcd880cceb0 0x7fcd880cd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.896+0000 7fcd87400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcd880cceb0 0x7fcd880cd320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59024/0 (socket says 172.21.15.27:59024) 2024-01-15T18:05:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.896+0000 7fcd87400640 1 -- 172.21.15.27:0/1120414379 learned_addr learned my addr 172.21.15.27:0/1120414379 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.896+0000 7fcd86000640 1 --2- 172.21.15.27:0/1120414379 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd880d0f80 0x7fcd880d13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.896+0000 7fcd86a00640 1 --2- 172.21.15.27:0/1120414379 >> v2:172.21.15.93:3300/0 conn(0x7fcd880ce390 0x7fcd880cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.896+0000 7fcd87400640 1 -- 172.21.15.27:0/1120414379 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd880d0f80 msgr2=0x7fcd880d13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.896+0000 7fcd87400640 1 --2- 172.21.15.27:0/1120414379 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd880d0f80 0x7fcd880d13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.896+0000 7fcd87400640 1 -- 172.21.15.27:0/1120414379 >> v2:172.21.15.93:3300/0 conn(0x7fcd880ce390 msgr2=0x7fcd880cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.896+0000 7fcd87400640 1 --2- 172.21.15.27:0/1120414379 >> v2:172.21.15.93:3300/0 conn(0x7fcd880ce390 0x7fcd880cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.896+0000 7fcd87400640 1 -- 172.21.15.27:0/1120414379 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd880cfad0 con 0x7fcd880cceb0 2024-01-15T18:05:03.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.900+0000 7fcd87400640 1 --2- 172.21.15.27:0/1120414379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd880cceb0 0x7fcd880cd320 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcd880ce9b0 tx=0x7fcd8001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e5d033cdf8c8e06 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:03.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.900+0000 7fcd85600640 1 -- 172.21.15.27:0/1120414379 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd800a5850 con 0x7fcd880cceb0 2024-01-15T18:05:03.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.900+0000 7fcd85600640 1 -- 172.21.15.27:0/1120414379 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd800a59b0 con 0x7fcd880cceb0 2024-01-15T18:05:03.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.900+0000 7fcd85600640 1 -- 172.21.15.27:0/1120414379 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd800a5d40 con 0x7fcd880cceb0 2024-01-15T18:05:03.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.900+0000 7fcd8de00640 1 -- 172.21.15.27:0/1120414379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd880cceb0 msgr2=0x7fcd880cd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:03.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.900+0000 7fcd8de00640 1 --2- 172.21.15.27:0/1120414379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd880cceb0 0x7fcd880cd320 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcd880ce9b0 tx=0x7fcd8001c9d0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.900+0000 7fcd8de00640 1 -- 172.21.15.27:0/1120414379 shutdown_connections 2024-01-15T18:05:03.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.900+0000 7fcd8de00640 1 --2- 172.21.15.27:0/1120414379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd880cceb0 0x7fcd880cd320 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.900+0000 7fcd8de00640 1 --2- 172.21.15.27:0/1120414379 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd880d0f80 0x7fcd880d13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.900+0000 7fcd8de00640 1 --2- 172.21.15.27:0/1120414379 >> v2:172.21.15.93:3300/0 conn(0x7fcd880ce390 0x7fcd880cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.900+0000 7fcd8de00640 1 -- 172.21.15.27:0/1120414379 >> 172.21.15.27:0/1120414379 conn(0x7fcd880662a0 msgr2=0x7fcd880666b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:03.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.900+0000 7fcd8de00640 1 -- 172.21.15.27:0/1120414379 shutdown_connections 2024-01-15T18:05:03.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.900+0000 7fcd8de00640 1 -- 172.21.15.27:0/1120414379 wait complete. 2024-01-15T18:05:03.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.904+0000 7fcd8de00640 1 Processor -- start 2024-01-15T18:05:03.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.904+0000 7fcd8de00640 1 -- start start 2024-01-15T18:05:03.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.904+0000 7fcd8de00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd880cceb0 0x7fcd88149af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:03.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.904+0000 7fcd8de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd880ce390 0x7fcd880b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:03.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.904+0000 7fcd8de00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcd880d0f80 0x7fcd880b43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:03.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.904+0000 7fcd8de00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd880b4d40 con 0x7fcd880ce390 2024-01-15T18:05:03.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.904+0000 7fcd8de00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd881473c0 con 0x7fcd880d0f80 2024-01-15T18:05:03.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.904+0000 7fcd8de00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd880b61a0 con 0x7fcd880cceb0 2024-01-15T18:05:03.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.904+0000 7fcd86000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd880ce390 0x7fcd880b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:03.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.904+0000 7fcd86000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd880ce390 0x7fcd880b3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59038/0 (socket says 172.21.15.27:59038) 2024-01-15T18:05:03.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.904+0000 7fcd86000640 1 -- 172.21.15.27:0/2176203842 learned_addr learned my addr 172.21.15.27:0/2176203842 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:03.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.904+0000 7fcd87400640 1 --2- 172.21.15.27:0/2176203842 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcd880d0f80 0x7fcd880b43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:03.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.904+0000 7fcd86a00640 1 --2- 172.21.15.27:0/2176203842 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd880cceb0 0x7fcd88149af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:03.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.904+0000 7fcd86000640 1 -- 172.21.15.27:0/2176203842 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd880cceb0 msgr2=0x7fcd88149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:03.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.904+0000 7fcd86000640 1 --2- 172.21.15.27:0/2176203842 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd880cceb0 0x7fcd88149af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.904+0000 7fcd86000640 1 -- 172.21.15.27:0/2176203842 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcd880d0f80 msgr2=0x7fcd880b43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.904+0000 7fcd86000640 1 --2- 172.21.15.27:0/2176203842 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcd880d0f80 0x7fcd880b43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.904+0000 7fcd86000640 1 -- 172.21.15.27:0/2176203842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd80009b30 con 0x7fcd880ce390 2024-01-15T18:05:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.904+0000 7fcd86000640 1 --2- 172.21.15.27:0/2176203842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd880ce390 0x7fcd880b3e90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcd7000d5f0 tx=0x7fcd7000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.904+0000 7fcd7f400640 1 -- 172.21.15.27:0/2176203842 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd700125f0 con 0x7fcd880ce390 2024-01-15T18:05:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.904+0000 7fcd7f400640 1 -- 172.21.15.27:0/2176203842 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd70012750 con 0x7fcd880ce390 2024-01-15T18:05:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.904+0000 7fcd7f400640 1 -- 172.21.15.27:0/2176203842 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd70012ae0 con 0x7fcd880ce390 2024-01-15T18:05:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.904+0000 7fcd8de00640 1 -- 172.21.15.27:0/2176203842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd880b0d70 con 0x7fcd880ce390 2024-01-15T18:05:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.904+0000 7fcd8de00640 1 -- 172.21.15.27:0/2176203842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd880b6720 con 0x7fcd880ce390 2024-01-15T18:05:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.904+0000 7fcd8de00640 1 -- 172.21.15.27:0/2176203842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd88068860 con 0x7fcd880ce390 2024-01-15T18:05:03.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.908+0000 7fcd7f400640 1 -- 172.21.15.27:0/2176203842 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fcd70012c40 con 0x7fcd880ce390 2024-01-15T18:05:03.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.908+0000 7fcd7f400640 1 --2- 172.21.15.27:0/2176203842 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcd600668f0 0x7fcd60068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:03.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.908+0000 7fcd7f400640 1 -- 172.21.15.27:0/2176203842 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcd70084860 con 0x7fcd880ce390 2024-01-15T18:05:03.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.908+0000 7fcd86a00640 1 --2- 172.21.15.27:0/2176203842 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcd600668f0 0x7fcd60068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:03.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.908+0000 7fcd86a00640 1 --2- 172.21.15.27:0/2176203842 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcd600668f0 0x7fcd60068db0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcd780045f0 tx=0x7fcd78008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:03.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:03.912+0000 7fcd7f400640 1 -- 172.21.15.27:0/2176203842 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fcd70051ab0 con 0x7fcd880ce390 2024-01-15T18:05:04.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.080+0000 7fcd8de00640 1 -- 172.21.15.27:0/2176203842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd8804e850 con 0x7fcd880ce390 2024-01-15T18:05:04.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.080+0000 7fcd7f400640 1 -- 172.21.15.27:0/2176203842 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16409 (secure 0 0 0) 0x7fcd70007380 con 0x7fcd880ce390 2024-01-15T18:05:04.084 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:04.084 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:04:59.837052+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:04:56.220608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T18:04:57.210421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:04.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.084+0000 7fcd8de00640 1 -- 172.21.15.27:0/2176203842 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcd600668f0 msgr2=0x7fcd60068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.084+0000 7fcd8de00640 1 --2- 172.21.15.27:0/2176203842 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcd600668f0 0x7fcd60068db0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcd780045f0 tx=0x7fcd78008040 comp rx=0 tx=0).stop 2024-01-15T18:05:04.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.084+0000 7fcd8de00640 1 -- 172.21.15.27:0/2176203842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd880ce390 msgr2=0x7fcd880b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.084+0000 7fcd8de00640 1 --2- 172.21.15.27:0/2176203842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd880ce390 0x7fcd880b3e90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcd7000d5f0 tx=0x7fcd7000dab0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.084+0000 7fcd8de00640 1 -- 172.21.15.27:0/2176203842 shutdown_connections 2024-01-15T18:05:04.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.084+0000 7fcd8de00640 1 --2- 172.21.15.27:0/2176203842 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcd600668f0 0x7fcd60068db0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.084+0000 7fcd8de00640 1 --2- 172.21.15.27:0/2176203842 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcd880d0f80 0x7fcd880b43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.084+0000 7fcd8de00640 1 --2- 172.21.15.27:0/2176203842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd880ce390 0x7fcd880b3e90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.084+0000 7fcd8de00640 1 --2- 172.21.15.27:0/2176203842 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd880cceb0 0x7fcd88149af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.084+0000 7fcd8de00640 1 -- 172.21.15.27:0/2176203842 >> 172.21.15.27:0/2176203842 conn(0x7fcd880662a0 msgr2=0x7fcd8815d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:04.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.084+0000 7fcd8de00640 1 -- 172.21.15.27:0/2176203842 shutdown_connections 2024-01-15T18:05:04.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.084+0000 7fcd8de00640 1 -- 172.21.15.27:0/2176203842 wait complete. 2024-01-15T18:05:04.106 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-15T18:05:04.106 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f3420e00640 1 Processor -- start 2024-01-15T18:05:04.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f3420e00640 1 -- start start 2024-01-15T18:05:04.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f3420e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f341c0cdf90 0x7f341c168450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f3420e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f341c0b2770 con 0x7f341c0b2390 2024-01-15T18:05:04.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f3420e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f341c06c060 con 0x7f341c06bb80 2024-01-15T18:05:04.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f3420e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f341c06c340 con 0x7f341c0cdf90 2024-01-15T18:05:04.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f3419e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f341c0b2390 0x7f341c0cf780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42282/0 (socket says 172.21.15.27:42282) 2024-01-15T18:05:04.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f3419e00640 1 -- 172.21.15.27:0/586908284 learned_addr learned my addr 172.21.15.27:0/586908284 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:04.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f341a800640 1 --2- 172.21.15.27:0/586908284 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f341c0cdf90 0x7f341c168450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f3418a00640 1 -- 172.21.15.27:0/586908284 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3165902201 0 0) 0x7f341c0b2770 con 0x7f341c0b2390 2024-01-15T18:05:04.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f3418a00640 1 -- 172.21.15.27:0/586908284 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33fc003530 con 0x7f341c0b2390 2024-01-15T18:05:04.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f3418a00640 1 -- 172.21.15.27:0/586908284 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061978324 0 0) 0x7f341c06c060 con 0x7f341c06bb80 2024-01-15T18:05:04.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f3418a00640 1 -- 172.21.15.27:0/586908284 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f341c0b2770 con 0x7f341c06bb80 2024-01-15T18:05:04.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f3418a00640 1 -- 172.21.15.27:0/586908284 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1072699094 0 0) 0x7f33fc003530 con 0x7f341c0b2390 2024-01-15T18:05:04.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f3418a00640 1 -- 172.21.15.27:0/586908284 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f341c0cdf90 msgr2=0x7f341c168450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:04.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f3418a00640 1 --2- 172.21.15.27:0/586908284 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f341c0cdf90 0x7f341c168450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f3418a00640 1 -- 172.21.15.27:0/586908284 >> v1:172.21.15.93:6789/0 conn(0x7f341c06bb80 legacy=0x7f341c0cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f3418a00640 1 -- 172.21.15.27:0/586908284 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341c06c620 con 0x7f341c0b2390 2024-01-15T18:05:04.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f3418a00640 1 -- 172.21.15.27:0/586908284 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f340c001680 con 0x7f341c0b2390 2024-01-15T18:05:04.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f3418a00640 1 -- 172.21.15.27:0/586908284 >> v1:172.21.15.27:6789/0 conn(0x7f341c0b2390 legacy=0x7f341c0cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f3418a00640 1 --2- 172.21.15.27:0/586908284 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33fc003bb0 0x7f33fc003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f3418a00640 1 --2- 172.21.15.27:0/586908284 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f341c0cdf90 0x7f33fc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f3418a00640 1 --2- 172.21.15.27:0/586908284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33fc004d10 0x7f33fc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f3418a00640 1 -- 172.21.15.27:0/586908284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341c06c340 con 0x7f33fc004d10 2024-01-15T18:05:04.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f3418a00640 1 -- 172.21.15.27:0/586908284 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc00baf0 con 0x7f33fc003bb0 2024-01-15T18:05:04.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.244+0000 7f3418a00640 1 -- 172.21.15.27:0/586908284 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc00bcb0 con 0x7f341c0cdf90 2024-01-15T18:05:04.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.248+0000 7f341a800640 1 --2- 172.21.15.27:0/586908284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33fc004d10 0x7f33fc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.248+0000 7f3419e00640 1 --2- 172.21.15.27:0/586908284 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33fc003bb0 0x7f33fc003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.248+0000 7f3419400640 1 --2- 172.21.15.27:0/586908284 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f341c0cdf90 0x7f33fc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.248+0000 7f341a800640 1 -- 172.21.15.27:0/586908284 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f341c0cdf90 msgr2=0x7f33fc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:04.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.248+0000 7f341a800640 1 --2- 172.21.15.27:0/586908284 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f341c0cdf90 0x7f33fc0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.248+0000 7f341a800640 1 -- 172.21.15.27:0/586908284 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33fc003bb0 msgr2=0x7f33fc003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:04.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.248+0000 7f341a800640 1 --2- 172.21.15.27:0/586908284 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33fc003bb0 0x7f33fc003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.248+0000 7f341a800640 1 -- 172.21.15.27:0/586908284 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33fc003530 con 0x7f33fc004d10 2024-01-15T18:05:04.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.248+0000 7f341a800640 1 --2- 172.21.15.27:0/586908284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33fc004d10 0x7f33fc005190 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3414009fc0 tx=0x7f341401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=880d06f47e557746 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.248+0000 7f3418a00640 1 -- 172.21.15.27:0/586908284 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34140a5b60 con 0x7f33fc004d10 2024-01-15T18:05:04.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.248+0000 7f3418a00640 1 -- 172.21.15.27:0/586908284 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341400f040 con 0x7f33fc004d10 2024-01-15T18:05:04.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.248+0000 7f3418a00640 1 -- 172.21.15.27:0/586908284 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34140ad440 con 0x7f33fc004d10 2024-01-15T18:05:04.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.252+0000 7f3420e00640 1 -- 172.21.15.27:0/586908284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33fc004d10 msgr2=0x7f33fc005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.252+0000 7f3420e00640 1 --2- 172.21.15.27:0/586908284 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33fc004d10 0x7f33fc005190 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3414009fc0 tx=0x7f341401cc60 comp rx=0 tx=0).stop 2024-01-15T18:05:04.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.252+0000 7f341a800640 1 -- 172.21.15.27:0/586908284 reap_dead start 2024-01-15T18:05:04.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.252+0000 7f3420e00640 1 -- 172.21.15.27:0/586908284 shutdown_connections 2024-01-15T18:05:04.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.252+0000 7f3420e00640 1 -- 172.21.15.27:0/586908284 >> 172.21.15.27:0/586908284 conn(0x7f341c0b2a10 msgr2=0x7f341c0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:04.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.252+0000 7f3420e00640 1 -- 172.21.15.27:0/586908284 shutdown_connections 2024-01-15T18:05:04.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.252+0000 7f3420e00640 1 -- 172.21.15.27:0/586908284 wait complete. 2024-01-15T18:05:04.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.252+0000 7f3420e00640 1 Processor -- start 2024-01-15T18:05:04.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.252+0000 7f3420e00640 1 -- start start 2024-01-15T18:05:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.252+0000 7f3420e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33fc004d10 0x7f341c0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.252+0000 7f3420e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f341c0c5270 0x7f341c0c56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.252+0000 7f3420e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f341c0ae710 0x7f341c0aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.252+0000 7f3420e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341c0ca4d0 con 0x7f33fc004d10 2024-01-15T18:05:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.252+0000 7f3420e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341c0af280 con 0x7f341c0ae710 2024-01-15T18:05:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.252+0000 7f3420e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f341c0af560 con 0x7f341c0c5270 2024-01-15T18:05:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.252+0000 7f3419e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33fc004d10 0x7f341c0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.252+0000 7f3419e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33fc004d10 0x7f341c0c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59054/0 (socket says 172.21.15.27:59054) 2024-01-15T18:05:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.252+0000 7f3419e00640 1 -- 172.21.15.27:0/3263194649 learned_addr learned my addr 172.21.15.27:0/3263194649 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.252+0000 7f341a800640 1 --2- 172.21.15.27:0/3263194649 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f341c0ae710 0x7f341c0aeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.252+0000 7f3419400640 1 --2- 172.21.15.27:0/3263194649 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f341c0c5270 0x7f341c0c56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.256+0000 7f3419e00640 1 -- 172.21.15.27:0/3263194649 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f341c0c5270 msgr2=0x7f341c0c56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.256+0000 7f3419e00640 1 --2- 172.21.15.27:0/3263194649 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f341c0c5270 0x7f341c0c56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.256+0000 7f3419e00640 1 -- 172.21.15.27:0/3263194649 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f341c0ae710 msgr2=0x7f341c0aeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.256+0000 7f3419e00640 1 --2- 172.21.15.27:0/3263194649 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f341c0ae710 0x7f341c0aeb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.256+0000 7f3419e00640 1 -- 172.21.15.27:0/3263194649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341c0c8030 con 0x7f33fc004d10 2024-01-15T18:05:04.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.256+0000 7f341a800640 1 --2- 172.21.15.27:0/3263194649 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f341c0ae710 0x7f341c0aeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:04.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.256+0000 7f3419e00640 1 --2- 172.21.15.27:0/3263194649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33fc004d10 0x7f341c0c9f90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f340c012490 tx=0x7f340c012950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.256+0000 7f3412000640 1 -- 172.21.15.27:0/3263194649 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340c014810 con 0x7f33fc004d10 2024-01-15T18:05:04.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.256+0000 7f3412000640 1 -- 172.21.15.27:0/3263194649 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340c014970 con 0x7f33fc004d10 2024-01-15T18:05:04.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.256+0000 7f3412000640 1 -- 172.21.15.27:0/3263194649 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340c014ce0 con 0x7f33fc004d10 2024-01-15T18:05:04.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.256+0000 7f3420e00640 1 -- 172.21.15.27:0/3263194649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f341c0c8310 con 0x7f33fc004d10 2024-01-15T18:05:04.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.256+0000 7f3420e00640 1 -- 172.21.15.27:0/3263194649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f341c14ab70 con 0x7f33fc004d10 2024-01-15T18:05:04.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.256+0000 7f3420e00640 1 -- 172.21.15.27:0/3263194649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f341c067a10 con 0x7f33fc004d10 2024-01-15T18:05:04.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.256+0000 7f3412000640 1 -- 172.21.15.27:0/3263194649 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f340c0153e0 con 0x7f33fc004d10 2024-01-15T18:05:04.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.256+0000 7f3412000640 1 --2- 172.21.15.27:0/3263194649 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f33f4066920 0x7f33f4068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.256+0000 7f3412000640 1 -- 172.21.15.27:0/3263194649 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f340c086da0 con 0x7f33fc004d10 2024-01-15T18:05:04.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.260+0000 7f3419400640 1 --2- 172.21.15.27:0/3263194649 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f33f4066920 0x7f33f4068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.260+0000 7f3419400640 1 --2- 172.21.15.27:0/3263194649 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f33f4066920 0x7f33f4068de0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f341c15b2f0 tx=0x7f3404009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.260+0000 7f3412000640 1 -- 172.21.15.27:0/3263194649 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f340c054080 con 0x7f33fc004d10 2024-01-15T18:05:04.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.416+0000 7f3420e00640 1 -- 172.21.15.27:0/3263194649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f341c0b1870 con 0x7f33fc004d10 2024-01-15T18:05:04.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.420+0000 7f3412000640 1 -- 172.21.15.27:0/3263194649 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16409 (secure 0 0 0) 0x7f341c0b1870 con 0x7f33fc004d10 2024-01-15T18:05:04.423 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:04.423 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:04:59.837052+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:04:56.220608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T18:04:57.210421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:04.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.420+0000 7f3420e00640 1 -- 172.21.15.27:0/3263194649 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f33f4066920 msgr2=0x7f33f4068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.420+0000 7f3420e00640 1 --2- 172.21.15.27:0/3263194649 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f33f4066920 0x7f33f4068de0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f341c15b2f0 tx=0x7f3404009040 comp rx=0 tx=0).stop 2024-01-15T18:05:04.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.420+0000 7f3420e00640 1 -- 172.21.15.27:0/3263194649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33fc004d10 msgr2=0x7f341c0c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.420+0000 7f3420e00640 1 --2- 172.21.15.27:0/3263194649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33fc004d10 0x7f341c0c9f90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f340c012490 tx=0x7f340c012950 comp rx=0 tx=0).stop 2024-01-15T18:05:04.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.420+0000 7f3420e00640 1 -- 172.21.15.27:0/3263194649 shutdown_connections 2024-01-15T18:05:04.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.420+0000 7f3420e00640 1 --2- 172.21.15.27:0/3263194649 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f33f4066920 0x7f33f4068de0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.420+0000 7f3420e00640 1 --2- 172.21.15.27:0/3263194649 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f341c0ae710 0x7f341c0aeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.420+0000 7f3420e00640 1 --2- 172.21.15.27:0/3263194649 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f341c0c5270 0x7f341c0c56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.420+0000 7f3420e00640 1 --2- 172.21.15.27:0/3263194649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33fc004d10 0x7f341c0c9f90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.420+0000 7f3420e00640 1 -- 172.21.15.27:0/3263194649 >> 172.21.15.27:0/3263194649 conn(0x7f341c0b2a10 msgr2=0x7f341c0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:04.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.420+0000 7f3420e00640 1 -- 172.21.15.27:0/3263194649 shutdown_connections 2024-01-15T18:05:04.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.420+0000 7f3420e00640 1 -- 172.21.15.27:0/3263194649 wait complete. 2024-01-15T18:05:04.446 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-15T18:05:04.446 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-15T18:05:04.447 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-15T18:05:04.447 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-01-15T18:05:04.448 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-01-15T18:05:04.448 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-01-15T18:05:04.448 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-01-15T18:05:04.449 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-01-15T18:05:04.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.692+0000 7f78f2400640 1 Processor -- start 2024-01-15T18:05:04.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.692+0000 7f78f2400640 1 -- start start 2024-01-15T18:05:04.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.692+0000 7f78f2400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f78e40e3260 0x7f78e4056fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.692+0000 7f78f2400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78e40e3640 con 0x7f78e40543b0 2024-01-15T18:05:04.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.692+0000 7f78f2400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78e40578c0 con 0x7f78e403e8d0 2024-01-15T18:05:04.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.692+0000 7f78f2400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78e4057ba0 con 0x7f78e40e3260 2024-01-15T18:05:04.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.692+0000 7f78ebe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f78e40543b0 0x7f78e403c070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42288/0 (socket says 172.21.15.27:42288) 2024-01-15T18:05:04.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.692+0000 7f78ebe00640 1 -- 172.21.15.27:0/218872553 learned_addr learned my addr 172.21.15.27:0/218872553 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:04.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.692+0000 7f78eb400640 1 --2- 172.21.15.27:0/218872553 >> v2:172.21.15.93:3300/0 conn(0x7f78e40e3260 0x7f78e4056fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.696+0000 7fcc4d200640 1 Processor -- start 2024-01-15T18:05:04.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.696+0000 7f0aa6800640 1 Processor -- start 2024-01-15T18:05:04.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.696+0000 7f78eb400640 1 -- 172.21.15.27:0/218872553 >> v1:172.21.15.93:6789/0 conn(0x7f78e403e8d0 legacy=0x7f78e4053ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.696+0000 7f78eb400640 1 -- 172.21.15.27:0/218872553 >> v1:172.21.15.27:6789/0 conn(0x7f78e40543b0 legacy=0x7f78e403c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.696+0000 7f78eb400640 1 -- 172.21.15.27:0/218872553 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e4057e80 con 0x7f78e40e3260 2024-01-15T18:05:04.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.696+0000 7f78ea000640 1 -- 172.21.15.27:0/218872553 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565805162 0 0) 0x7f78e40e3640 con 0x7f78e40543b0 2024-01-15T18:05:04.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.696+0000 7fcc4d200640 1 -- start start 2024-01-15T18:05:04.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.696+0000 7fcc4d200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fcc480b1370 0x7fcc480ce160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.696+0000 7fcc4d200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc4815cd70 0x7fcc4815d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.696+0000 7fcc4d200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc480b24d0 con 0x7fcc4806cc10 2024-01-15T18:05:04.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.696+0000 7fcc4d200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc4815d740 con 0x7fcc4815cd70 2024-01-15T18:05:04.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.696+0000 7fcc4d200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc480cb6e0 con 0x7fcc480b1370 2024-01-15T18:05:04.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.696+0000 7f0aa6800640 1 -- start start 2024-01-15T18:05:04.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.696+0000 7f0aa6800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0a980e7490 0x7f0a98008040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.696+0000 7f0aa6800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0a98050520 0x7f0a980f1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.696+0000 7fcc45e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fcc480b1370 0x7fcc480ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.696+0000 7fcc45400640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7fcc4806cc10 0x7fcc48159050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:57014/0 (socket says 172.21.15.27:57014) 2024-01-15T18:05:04.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.696+0000 7fcc45400640 1 -- 172.21.15.27:0/1806860683 learned_addr learned my addr 172.21.15.27:0/1806860683 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:04.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.696+0000 7f0a9f400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0a980e7490 0x7f0a98008040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.696+0000 7f0a9f400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0a980e7490 0x7f0a98008040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:51420/0 (socket says 172.21.15.27:51420) 2024-01-15T18:05:04.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.696+0000 7f4295000640 1 Processor -- start 2024-01-15T18:05:04.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f4295000640 1 -- start start 2024-01-15T18:05:04.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7fcc46800640 1 --2- 172.21.15.27:0/1806860683 >> v2:172.21.15.27:3300/0 conn(0x7fcc4815cd70 0x7fcc4815d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7fcc44a00640 1 -- 172.21.15.27:0/1806860683 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376126599 0 0) 0x7fcc480b24d0 con 0x7fcc4806cc10 2024-01-15T18:05:04.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7fcc44a00640 1 -- 172.21.15.27:0/1806860683 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc28003530 con 0x7fcc4806cc10 2024-01-15T18:05:04.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f0a9fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0a98050520 0x7f0a980f1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f0a9fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0a98050520 0x7f0a980f1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59068/0 (socket says 172.21.15.27:59068) 2024-01-15T18:05:04.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f0a9f400640 1 -- 172.21.15.27:0/2186595764 learned_addr learned my addr 172.21.15.27:0/2186595764 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:04.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f0aa6800640 1 -- 172.21.15.27:0/2186595764 --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a98051650 con 0x7f0a9804f980 2024-01-15T18:05:04.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f0aa6800640 1 -- 172.21.15.27:0/2186595764 --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a98051c10 con 0x7f0a98050520 2024-01-15T18:05:04.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f0aa6800640 1 -- 172.21.15.27:0/2186595764 --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a980514b0 con 0x7f0a980e7490 2024-01-15T18:05:04.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f78eb400640 1 --2- 172.21.15.27:0/218872553 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78e40e3260 0x7f78e4056fc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f78dc0097c0 tx=0x7f78dc01ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d63db95e03b61b6b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f78ea000640 1 -- 172.21.15.27:0/218872553 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78dc0a8990 con 0x7f78e40e3260 2024-01-15T18:05:04.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f78ea000640 1 -- 172.21.15.27:0/218872553 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78dc0a8af0 con 0x7f78e40e3260 2024-01-15T18:05:04.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f78ea000640 1 -- 172.21.15.27:0/218872553 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78dc0a8e80 con 0x7f78e40e3260 2024-01-15T18:05:04.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f4295000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f42901592b0 0x7f42900ce160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f4295000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f429006cc10 0x7f429015cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7fcc45e00640 1 -- 172.21.15.27:0/1806860683 >> v2:172.21.15.27:3300/0 conn(0x7fcc4815cd70 msgr2=0x7fcc4815d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:04.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7fcc45e00640 1 --2- 172.21.15.27:0/1806860683 >> v2:172.21.15.27:3300/0 conn(0x7fcc4815cd70 0x7fcc4815d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7fcc45e00640 1 -- 172.21.15.27:0/1806860683 >> v1:172.21.15.93:6789/0 conn(0x7fcc4806cc10 legacy=0x7fcc48159050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7fcc45e00640 1 -- 172.21.15.27:0/1806860683 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4806d640 con 0x7fcc480b1370 2024-01-15T18:05:04.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f4295000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42900b24d0 con 0x7f429015d2b0 2024-01-15T18:05:04.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f4295000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4290159690 con 0x7f42901592b0 2024-01-15T18:05:04.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f4295000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f429015d740 con 0x7f429006cc10 2024-01-15T18:05:04.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f428e800640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7f429015d2b0 0x7f42900701c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:57020/0 (socket says 172.21.15.27:57020) 2024-01-15T18:05:04.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f428e800640 1 -- 172.21.15.27:0/4000390859 learned_addr learned my addr 172.21.15.27:0/4000390859 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:04.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f78f2400640 1 -- 172.21.15.27:0/218872553 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78e40e3260 msgr2=0x7f78e4056fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f78f2400640 1 --2- 172.21.15.27:0/218872553 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78e40e3260 0x7f78e4056fc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f78dc0097c0 tx=0x7f78dc01ec90 comp rx=0 tx=0).stop 2024-01-15T18:05:04.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f428ca00640 1 -- 172.21.15.27:0/4000390859 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 196428631 0 0) 0x7f42900b24d0 con 0x7f429015d2b0 2024-01-15T18:05:04.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f428d400640 1 --2- 172.21.15.27:0/4000390859 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f429006cc10 0x7f429015cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f428ca00640 1 -- 172.21.15.27:0/4000390859 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4270003530 con 0x7f429015d2b0 2024-01-15T18:05:04.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f78f2400640 1 -- 172.21.15.27:0/218872553 shutdown_connections 2024-01-15T18:05:04.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f78f2400640 1 --2- 172.21.15.27:0/218872553 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78e40e3260 0x7f78e4056fc0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f78f2400640 1 -- 172.21.15.27:0/218872553 >> 172.21.15.27:0/218872553 conn(0x7f78e4058ed0 msgr2=0x7f78e40592e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:04.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f428ca00640 1 -- 172.21.15.27:0/4000390859 <== mon.1 v1:172.21.15.93:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1408057563 0 0) 0x7f4270003530 con 0x7f429015d2b0 2024-01-15T18:05:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f428ca00640 1 -- 172.21.15.27:0/4000390859 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f429006cc10 msgr2=0x7f429015cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f428ca00640 1 --2- 172.21.15.27:0/4000390859 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f429006cc10 0x7f429015cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f428ca00640 1 -- 172.21.15.27:0/4000390859 >> v2:172.21.15.27:3300/0 conn(0x7f42901592b0 msgr2=0x7f42900ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f428ca00640 1 --2- 172.21.15.27:0/4000390859 >> v2:172.21.15.27:3300/0 conn(0x7f42901592b0 0x7f42900ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f428ca00640 1 -- 172.21.15.27:0/4000390859 --> v1:172.21.15.93:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429006d360 con 0x7f429015d2b0 2024-01-15T18:05:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f428ca00640 1 -- 172.21.15.27:0/4000390859 <== mon.1 v1:172.21.15.93:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f4288001a00 con 0x7f429015d2b0 2024-01-15T18:05:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f428ca00640 1 -- 172.21.15.27:0/4000390859 >> v1:172.21.15.93:6789/0 conn(0x7f429015d2b0 legacy=0x7f42900701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f428ca00640 1 --2- 172.21.15.27:0/4000390859 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42700039f0 0x7f4270003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f428ca00640 1 --2- 172.21.15.27:0/4000390859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42700047d0 0x7f4270004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f428ca00640 1 --2- 172.21.15.27:0/4000390859 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f429006cc10 0x7f42700051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f428ca00640 1 -- 172.21.15.27:0/4000390859 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4290159690 con 0x7f42700047d0 2024-01-15T18:05:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f428ca00640 1 -- 172.21.15.27:0/4000390859 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429015d740 con 0x7f42700039f0 2024-01-15T18:05:04.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f428ca00640 1 -- 172.21.15.27:0/4000390859 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f427000bbb0 con 0x7f429006cc10 2024-01-15T18:05:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f3f17200640 1 Processor -- start 2024-01-15T18:05:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f3f17200640 1 -- start start 2024-01-15T18:05:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f060f200640 1 Processor -- start 2024-01-15T18:05:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f78f2400640 1 -- 172.21.15.27:0/218872553 shutdown_connections 2024-01-15T18:05:04.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f78f2400640 1 -- 172.21.15.27:0/218872553 wait complete. 2024-01-15T18:05:04.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f060f200640 1 -- start start 2024-01-15T18:05:04.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f428e800640 1 --2- 172.21.15.27:0/4000390859 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f429006cc10 0x7f42700051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f428de00640 1 --2- 172.21.15.27:0/4000390859 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42700039f0 0x7f4270003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f0a9fe00640 1 -- 172.21.15.27:0/2186595764 >> v2:172.21.15.93:3300/0 conn(0x7f0a980e7490 msgr2=0x7f0a98008040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:04.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f0a9fe00640 1 --2- 172.21.15.27:0/2186595764 >> v2:172.21.15.93:3300/0 conn(0x7f0a980e7490 0x7f0a98008040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f0a9fe00640 1 -- 172.21.15.27:0/2186595764 >> v1:172.21.15.27:6789/0 conn(0x7f0a9804f980 legacy=0x7f0a9804fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:04.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.700+0000 7f0a9fe00640 1 -- 172.21.15.27:0/2186595764 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a98051930 con 0x7f0a98050520 2024-01-15T18:05:04.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f0a9f400640 1 --2- 172.21.15.27:0/2186595764 >> v2:172.21.15.93:3300/0 conn(0x7f0a980e7490 0x7f0a98008040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:04.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f060f200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0608161a10 0x7f06080706a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f060f200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0608070f90 0x7f06080b3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f060f200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f06080b3bb0 0x7f0608165550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f060f200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06080b4040 con 0x7f06080b3bb0 2024-01-15T18:05:04.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f060f200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06080acb10 con 0x7f0608161a10 2024-01-15T18:05:04.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f060f200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0608071490 con 0x7f0608070f90 2024-01-15T18:05:04.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f428e800640 1 -- 172.21.15.27:0/4000390859 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42700039f0 msgr2=0x7f4270003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:04.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f428e800640 1 --2- 172.21.15.27:0/4000390859 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42700039f0 0x7f4270003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f428e800640 1 -- 172.21.15.27:0/4000390859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42700047d0 msgr2=0x7f4270004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:04.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f428e800640 1 --2- 172.21.15.27:0/4000390859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42700047d0 0x7f4270004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f0607e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0608161a10 0x7f06080706a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f428e800640 1 -- 172.21.15.27:0/4000390859 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4270003530 con 0x7f429006cc10 2024-01-15T18:05:04.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f0607e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0608161a10 0x7f06080706a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:51444/0 (socket says 172.21.15.27:51444) 2024-01-15T18:05:04.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f0607e00640 1 -- 172.21.15.27:0/3996893591 learned_addr learned my addr 172.21.15.27:0/3996893591 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:04.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f3f17200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f3f18159600 0x7f3f180b2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f78f2400640 1 Processor -- start 2024-01-15T18:05:04.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f3f17200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3f18070d50 0x7f3f1806c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7fcc45e00640 1 --2- 172.21.15.27:0/1806860683 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcc480b1370 0x7fcc480ce160 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcc480cfcd0 tx=0x7fcc3001c270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4ebdb45bd659f21 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.704+0000 7f060d200640 1 --2- 172.21.15.27:0/3996893591 >> v2:172.21.15.27:3300/0 conn(0x7f06080b3bb0 0x7f0608165550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f16800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3f18070d50 0x7f3f1806c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f16800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3f18070d50 0x7f3f1806c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:42536/0 (socket says 172.21.15.27:42536) 2024-01-15T18:05:04.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f16800640 1 -- 172.21.15.27:0/3584989554 learned_addr learned my addr 172.21.15.27:0/3584989554 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:04.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f0607e00640 1 -- 172.21.15.27:0/3996893591 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0608070f90 msgr2=0x7f06080b3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:04.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f0607e00640 1 --2- 172.21.15.27:0/3996893591 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0608070f90 0x7f06080b3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7fcc44a00640 1 -- 172.21.15.27:0/1806860683 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc38001680 con 0x7fcc480b1370 2024-01-15T18:05:04.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f0607e00640 1 -- 172.21.15.27:0/3996893591 >> v2:172.21.15.27:3300/0 conn(0x7f06080b3bb0 msgr2=0x7f0608165550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:04.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f0607e00640 1 --2- 172.21.15.27:0/3996893591 >> v2:172.21.15.27:3300/0 conn(0x7f06080b3bb0 0x7f0608165550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f0a9fe00640 1 --2- 172.21.15.27:0/2186595764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a98050520 0x7f0a980f1050 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0a9800bc00 tx=0x7f0a8801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16226e41845fbbca server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f0a9e000640 1 -- 172.21.15.27:0/2186595764 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a880ab600 con 0x7f0a98050520 2024-01-15T18:05:04.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f0a9e000640 1 -- 172.21.15.27:0/2186595764 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a880a7050 con 0x7f0a98050520 2024-01-15T18:05:04.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f0a9e000640 1 -- 172.21.15.27:0/2186595764 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a880ab980 con 0x7f0a98050520 2024-01-15T18:05:04.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f78f2400640 1 -- start start 2024-01-15T18:05:04.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f78f2400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f78e403e8d0 0x7f78e403a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f78f2400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78e40543b0 0x7f78e4180700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f78f2400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78e40e3260 0x7f78e4039830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f15e00640 1 --2- 172.21.15.27:0/3584989554 >> v2:172.21.15.93:3300/0 conn(0x7f3f18159600 0x7f3f180b2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f17200640 1 -- 172.21.15.27:0/3584989554 --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f180b3960 con 0x7f3f180ce390 2024-01-15T18:05:04.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f17200640 1 -- 172.21.15.27:0/3584989554 --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f1806c3f0 con 0x7f3f18159600 2024-01-15T18:05:04.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f17200640 1 -- 172.21.15.27:0/3584989554 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f180b30c0 con 0x7f3f18070d50 2024-01-15T18:05:04.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f14a00640 1 -- 172.21.15.27:0/3584989554 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146470775 0 0) 0x7f3f00001120 con 0x7f3f180ce390 2024-01-15T18:05:04.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f14a00640 1 -- 172.21.15.27:0/3584989554 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ef8005740 con 0x7f3f180ce390 2024-01-15T18:05:04.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f14a00640 1 -- 172.21.15.27:0/3584989554 <== mon.1 v1:172.21.15.93:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2252122349 0 0) 0x7f3ef8005740 con 0x7f3f180ce390 2024-01-15T18:05:04.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f14a00640 1 -- 172.21.15.27:0/3584989554 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3f18070d50 msgr2=0x7f3f1806c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:04.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f14a00640 1 --2- 172.21.15.27:0/3584989554 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3f18070d50 0x7f3f1806c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f14a00640 1 -- 172.21.15.27:0/3584989554 >> v2:172.21.15.93:3300/0 conn(0x7f3f18159600 msgr2=0x7f3f180b2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:04.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f14a00640 1 --2- 172.21.15.27:0/3584989554 >> v2:172.21.15.93:3300/0 conn(0x7f3f18159600 0x7f3f180b2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f14a00640 1 -- 172.21.15.27:0/3584989554 --> v1:172.21.15.93:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f180b33a0 con 0x7f3f180ce390 2024-01-15T18:05:04.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f14a00640 1 -- 172.21.15.27:0/3584989554 <== mon.1 v1:172.21.15.93:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f3f00001810 con 0x7f3f180ce390 2024-01-15T18:05:04.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f14a00640 1 -- 172.21.15.27:0/3584989554 >> v1:172.21.15.93:6789/0 conn(0x7f3f180ce390 legacy=0x7f3f18070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f14a00640 1 --2- 172.21.15.27:0/3584989554 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ef8005c00 0x7f3ef8006020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f14a00640 1 --2- 172.21.15.27:0/3584989554 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ef80069e0 0x7f3ef8006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f14a00640 1 --2- 172.21.15.27:0/3584989554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ef80073a0 0x7f3ef800dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f14a00640 1 -- 172.21.15.27:0/3584989554 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1806c3f0 con 0x7f3ef80073a0 2024-01-15T18:05:04.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f14a00640 1 -- 172.21.15.27:0/3584989554 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f180b30c0 con 0x7f3ef8005c00 2024-01-15T18:05:04.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f14a00640 1 -- 172.21.15.27:0/3584989554 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef800e0e0 con 0x7f3ef80069e0 2024-01-15T18:05:04.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f15e00640 1 --2- 172.21.15.27:0/3584989554 >> v2:172.21.15.93:3300/0 conn(0x7f3f18159600 0x7f3f180b2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:04.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f0607e00640 1 -- 172.21.15.27:0/3996893591 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0608071770 con 0x7f0608161a10 2024-01-15T18:05:04.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7fa8dc600640 1 Processor -- start 2024-01-15T18:05:04.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7fa8dc600640 1 -- start start 2024-01-15T18:05:04.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f78f2400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e4181150 con 0x7f78e40e3260 2024-01-15T18:05:04.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f16800640 1 --2- 172.21.15.27:0/3584989554 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3f18070d50 0x7f3f1806c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:04.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f78f2400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e4181430 con 0x7f78e40543b0 2024-01-15T18:05:04.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f15400640 1 --2- 172.21.15.27:0/3584989554 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ef80069e0 0x7f3ef8006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f78ebe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78e40e3260 0x7f78e4039830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f78f2400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78e4050160 con 0x7f78e403e8d0 2024-01-15T18:05:04.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f78ebe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78e40e3260 0x7f78e4039830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59104/0 (socket says 172.21.15.27:59104) 2024-01-15T18:05:04.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f78ebe00640 1 -- 172.21.15.27:0/2375858441 learned_addr learned my addr 172.21.15.27:0/2375858441 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:04.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f15e00640 1 --2- 172.21.15.27:0/3584989554 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ef8005c00 0x7f3ef8006020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7fcc44a00640 1 -- 172.21.15.27:0/1806860683 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc300a7050 con 0x7fcc480b1370 2024-01-15T18:05:04.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7fcc44a00640 1 -- 172.21.15.27:0/1806860683 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc300ab620 con 0x7fcc480b1370 2024-01-15T18:05:04.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7fcc4d200640 1 -- 172.21.15.27:0/1806860683 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcc480b1370 msgr2=0x7fcc480ce160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7fcc4d200640 1 --2- 172.21.15.27:0/1806860683 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcc480b1370 0x7fcc480ce160 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcc480cfcd0 tx=0x7fcc3001c270 comp rx=0 tx=0).stop 2024-01-15T18:05:04.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7fcc4d200640 1 -- 172.21.15.27:0/1806860683 shutdown_connections 2024-01-15T18:05:04.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f78eaa00640 1 --2- 172.21.15.27:0/2375858441 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78e40543b0 0x7f78e4180700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f060d200640 1 --2- 172.21.15.27:0/3996893591 >> v2:172.21.15.27:3300/0 conn(0x7f06080b3bb0 0x7f0608165550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:04.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7fcc4d200640 1 --2- 172.21.15.27:0/1806860683 >> v2:172.21.15.27:3300/0 conn(0x7fcc4815cd70 0x7fcc4815d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7fcc4d200640 1 --2- 172.21.15.27:0/1806860683 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcc480b1370 0x7fcc480ce160 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7fcc4d200640 1 -- 172.21.15.27:0/1806860683 >> 172.21.15.27:0/1806860683 conn(0x7fcc480b0850 msgr2=0x7fcc480b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:04.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7fcc4d200640 1 -- 172.21.15.27:0/1806860683 shutdown_connections 2024-01-15T18:05:04.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f0aa6800640 1 -- 172.21.15.27:0/2186595764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a98050520 msgr2=0x7f0a980f1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f0aa6800640 1 --2- 172.21.15.27:0/2186595764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a98050520 0x7f0a980f1050 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0a9800bc00 tx=0x7f0a8801c440 comp rx=0 tx=0).stop 2024-01-15T18:05:04.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f15400640 1 -- 172.21.15.27:0/3584989554 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ef8005c00 msgr2=0x7f3ef8006020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:04.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f15400640 1 --2- 172.21.15.27:0/3584989554 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ef8005c00 0x7f3ef8006020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f15400640 1 -- 172.21.15.27:0/3584989554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ef80073a0 msgr2=0x7f3ef800dba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-15T18:05:04.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7fcc4d200640 1 -- 172.21.15.27:0/1806860683 wait complete. 2024-01-15T18:05:04.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f78eb400640 1 --2- 172.21.15.27:0/2375858441 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f78e403e8d0 0x7f78e403a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f16800640 1 --2- 172.21.15.27:0/3584989554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ef80073a0 0x7f3ef800dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f78eaa00640 1 -- 172.21.15.27:0/2375858441 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f78e403e8d0 msgr2=0x7f78e403a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f78eaa00640 1 --2- 172.21.15.27:0/2375858441 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f78e403e8d0 0x7f78e403a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f78eaa00640 1 -- 172.21.15.27:0/2375858441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78e40e3260 msgr2=0x7f78e4039830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f78eaa00640 1 --2- 172.21.15.27:0/2375858441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78e40e3260 0x7f78e4039830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f15400640 1 --2- 172.21.15.27:0/3584989554 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ef80073a0 0x7f3ef800dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f3f15400640 1 -- 172.21.15.27:0/3584989554 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f180b33a0 con 0x7f3ef80069e0 2024-01-15T18:05:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.708+0000 7f78eaa00640 1 -- 172.21.15.27:0/2375858441 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78dc009420 con 0x7f78e40543b0 2024-01-15T18:05:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7f78eaa00640 1 --2- 172.21.15.27:0/2375858441 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78e40543b0 0x7f78e4180700 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f78d400c3c0 tx=0x7f78d400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7f78e3400640 1 -- 172.21.15.27:0/2375858441 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d40052d0 con 0x7f78e40543b0 2024-01-15T18:05:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7fa8dc600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa8d4156e20 0x7fa8d40ce160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7fa8dc600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8d4070bd0 0x7fa8d406c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7fa8dc600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d4157200 con 0x7fa8d415cd90 2024-01-15T18:05:04.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7fa8dc600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8d40ce6a0 con 0x7fa8d4156e20 2024-01-15T18:05:04.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7fa8dc600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d406c740 con 0x7fa8d4070bd0 2024-01-15T18:05:04.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7f78e3400640 1 -- 172.21.15.27:0/2375858441 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d4005430 con 0x7f78e40543b0 2024-01-15T18:05:04.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7fa8da600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa8d415cd90 0x7fa8d40b3290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42310/0 (socket says 172.21.15.27:42310) 2024-01-15T18:05:04.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7fa8da600640 1 -- 172.21.15.27:0/123376231 learned_addr learned my addr 172.21.15.27:0/123376231 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:04.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7fa8d9c00640 1 --2- 172.21.15.27:0/123376231 >> v2:172.21.15.93:3300/0 conn(0x7fa8d4156e20 0x7fa8d40ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7fa8d9200640 1 --2- 172.21.15.27:0/123376231 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8d4070bd0 0x7fa8d406c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7f78f2400640 1 -- 172.21.15.27:0/2375858441 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78e4050440 con 0x7f78e40543b0 2024-01-15T18:05:04.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7f78f2400640 1 -- 172.21.15.27:0/2375858441 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78e40e60f0 con 0x7f78e40543b0 2024-01-15T18:05:04.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7f78e3400640 1 -- 172.21.15.27:0/2375858441 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d4007260 con 0x7f78e40543b0 2024-01-15T18:05:04.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7fa8d3e00640 1 -- 172.21.15.27:0/123376231 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1697383791 0 0) 0x7fa8d4157200 con 0x7fa8d415cd90 2024-01-15T18:05:04.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7fa8d9c00640 1 -- 172.21.15.27:0/123376231 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8d4070bd0 msgr2=0x7fa8d406c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:04.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7fa8d9c00640 1 --2- 172.21.15.27:0/123376231 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8d4070bd0 0x7fa8d406c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7fa8d9c00640 1 -- 172.21.15.27:0/123376231 >> v1:172.21.15.27:6789/0 conn(0x7fa8d415cd90 legacy=0x7fa8d40b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7fa8d9c00640 1 -- 172.21.15.27:0/123376231 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d40b24d0 con 0x7fa8d4156e20 2024-01-15T18:05:04.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7f78f2400640 1 -- 172.21.15.27:0/2375858441 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f78bc000fd0 con 0x7f78e40543b0 2024-01-15T18:05:04.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7fcc4d200640 1 Processor -- start 2024-01-15T18:05:04.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7f0aa6800640 1 -- 172.21.15.27:0/2186595764 shutdown_connections 2024-01-15T18:05:04.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7f0aa6800640 1 --2- 172.21.15.27:0/2186595764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a98050520 0x7f0a980f1050 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7f0aa6800640 1 --2- 172.21.15.27:0/2186595764 >> v2:172.21.15.93:3300/0 conn(0x7f0a980e7490 0x7f0a98008040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7f0aa6800640 1 -- 172.21.15.27:0/2186595764 >> 172.21.15.27:0/2186595764 conn(0x7f0a98055330 msgr2=0x7f0a98055740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:04.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.712+0000 7fcc4d200640 1 -- start start 2024-01-15T18:05:04.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f0aa6800640 1 -- 172.21.15.27:0/2186595764 shutdown_connections 2024-01-15T18:05:04.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f0aa6800640 1 -- 172.21.15.27:0/2186595764 wait complete. 2024-01-15T18:05:04.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f0607e00640 1 --2- 172.21.15.27:0/3996893591 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0608161a10 0x7f06080706a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f06080b1dc0 tx=0x7f05f801c180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fdacd236c393b8f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f396f200640 1 Processor -- start 2024-01-15T18:05:04.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f3f15400640 1 --2- 172.21.15.27:0/3584989554 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ef80069e0 0x7f3ef8006e60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3f0000c190 tx=0x7f3f0001cb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5434e38be7bbc771 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f396f200640 1 -- start start 2024-01-15T18:05:04.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f3f14a00640 1 -- 172.21.15.27:0/3584989554 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f000a5aa0 con 0x7f3ef80069e0 2024-01-15T18:05:04.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f3f14a00640 1 -- 172.21.15.27:0/3584989554 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f000a5c20 con 0x7f3ef80069e0 2024-01-15T18:05:04.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f3f14a00640 1 -- 172.21.15.27:0/3584989554 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f000ad440 con 0x7f3ef80069e0 2024-01-15T18:05:04.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f428e800640 1 --2- 172.21.15.27:0/4000390859 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f429006cc10 0x7f42700051b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4288005220 tx=0x7f428801e190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de85e661579bf16d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f0606a00640 1 -- 172.21.15.27:0/3996893591 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f80ab600 con 0x7f0608161a10 2024-01-15T18:05:04.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f78e3400640 1 -- 172.21.15.27:0/2375858441 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f78d40073c0 con 0x7f78e40543b0 2024-01-15T18:05:04.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f0606a00640 1 -- 172.21.15.27:0/3996893591 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f80a7050 con 0x7f0608161a10 2024-01-15T18:05:04.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f0606a00640 1 -- 172.21.15.27:0/3996893591 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f80ab600 con 0x7f0608161a10 2024-01-15T18:05:04.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7fcc4d200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcc4806cc10 0x7fcc48149760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7fcc4d200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcc480b1370 0x7fcc4814b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f428ca00640 1 -- 172.21.15.27:0/4000390859 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4288008270 con 0x7f429006cc10 2024-01-15T18:05:04.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f428ca00640 1 -- 172.21.15.27:0/4000390859 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42880a9070 con 0x7f429006cc10 2024-01-15T18:05:04.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f428ca00640 1 -- 172.21.15.27:0/4000390859 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4288006730 con 0x7f429006cc10 2024-01-15T18:05:04.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f3f17200640 1 -- 172.21.15.27:0/3584989554 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ef80069e0 msgr2=0x7f3ef8006e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f3f17200640 1 --2- 172.21.15.27:0/3584989554 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ef80069e0 0x7f3ef8006e60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3f0000c190 tx=0x7f3f0001cb40 comp rx=0 tx=0).stop 2024-01-15T18:05:04.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7fa8d9c00640 1 --2- 172.21.15.27:0/123376231 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8d4156e20 0x7fa8d40ce160 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc00c2e0 tx=0x7fa8cc01de30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6641284f6677c4f7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f3f16800640 1 -- 172.21.15.27:0/3584989554 reap_dead start 2024-01-15T18:05:04.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7fcc45e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcc4806cc10 0x7fcc48149760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7fcc45e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcc4806cc10 0x7fcc48149760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:42576/0 (socket says 172.21.15.27:42576) 2024-01-15T18:05:04.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7fcc45400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcc480b1370 0x7fcc4814b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7fcc45400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcc480b1370 0x7fcc4814b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:51512/0 (socket says 172.21.15.27:51512) 2024-01-15T18:05:04.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7fcc45e00640 1 -- 172.21.15.27:0/2064329409 learned_addr learned my addr 172.21.15.27:0/2064329409 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:04.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f3f17200640 1 -- 172.21.15.27:0/3584989554 shutdown_connections 2024-01-15T18:05:04.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f3f17200640 1 -- 172.21.15.27:0/3584989554 >> 172.21.15.27:0/3584989554 conn(0x7f3f180b0850 msgr2=0x7f3f180b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:04.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7fcc4d200640 1 --2- 172.21.15.27:0/2064329409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc4815cd70 0x7fcc481f7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7fa8d3e00640 1 -- 172.21.15.27:0/123376231 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc0a6e00 con 0x7fa8d4156e20 2024-01-15T18:05:04.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7fa8d3e00640 1 -- 172.21.15.27:0/123376231 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8cc00a0d0 con 0x7fa8d4156e20 2024-01-15T18:05:04.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7fa8d3e00640 1 -- 172.21.15.27:0/123376231 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc00a460 con 0x7fa8d4156e20 2024-01-15T18:05:04.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7fcc4d200640 1 -- 172.21.15.27:0/2064329409 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4814caf0 con 0x7fcc4815cd70 2024-01-15T18:05:04.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7fcc4d200640 1 -- 172.21.15.27:0/2064329409 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4814bee0 con 0x7fcc480b1370 2024-01-15T18:05:04.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f0aa6800640 1 Processor -- start 2024-01-15T18:05:04.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7fcc4d200640 1 -- 172.21.15.27:0/2064329409 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4814c1c0 con 0x7fcc4806cc10 2024-01-15T18:05:04.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f4295000640 1 -- 172.21.15.27:0/4000390859 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f429006cc10 msgr2=0x7f42700051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f3f17200640 1 -- 172.21.15.27:0/3584989554 shutdown_connections 2024-01-15T18:05:04.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f4295000640 1 --2- 172.21.15.27:0/4000390859 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f429006cc10 0x7f42700051b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4288005220 tx=0x7f428801e190 comp rx=0 tx=0).stop 2024-01-15T18:05:04.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f3f17200640 1 -- 172.21.15.27:0/3584989554 wait complete. 2024-01-15T18:05:04.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7fcc46800640 1 --2- 172.21.15.27:0/2064329409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc4815cd70 0x7fcc481f7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7fcc45400640 1 -- 172.21.15.27:0/2064329409 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcc4806cc10 msgr2=0x7fcc48149760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7fcc45400640 1 --2- 172.21.15.27:0/2064329409 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcc4806cc10 0x7fcc48149760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7fcc45400640 1 -- 172.21.15.27:0/2064329409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc4815cd70 msgr2=0x7fcc481f7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7fcc45400640 1 --2- 172.21.15.27:0/2064329409 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc4815cd70 0x7fcc481f7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7fcc45400640 1 -- 172.21.15.27:0/2064329409 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc30009420 con 0x7fcc480b1370 2024-01-15T18:05:04.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f428e800640 1 -- 172.21.15.27:0/4000390859 reap_dead start 2024-01-15T18:05:04.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f4295000640 1 -- 172.21.15.27:0/4000390859 shutdown_connections 2024-01-15T18:05:04.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f4295000640 1 -- 172.21.15.27:0/4000390859 >> 172.21.15.27:0/4000390859 conn(0x7f42900b0850 msgr2=0x7f42900b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:04.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f396f200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f39700d1ab0 0x7f39700b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f396f200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39700d3890 con 0x7f3970160cb0 2024-01-15T18:05:04.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f396f200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39700d3b00 con 0x7f39701608d0 2024-01-15T18:05:04.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f396f200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39700d3de0 con 0x7f39700d1ab0 2024-01-15T18:05:04.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f396de00640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7f39701608d0 0x7f39700d0450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:57030/0 (socket says 172.21.15.27:57030) 2024-01-15T18:05:04.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f396de00640 1 -- 172.21.15.27:0/4243007658 learned_addr learned my addr 172.21.15.27:0/4243007658 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:04.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f396e800640 1 --2- 172.21.15.27:0/4243007658 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f39700d1ab0 0x7f39700b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7fcc45400640 1 --2- 172.21.15.27:0/2064329409 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcc480b1370 0x7fcc4814b5e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcc380011f0 tx=0x7fcc3800cb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f0aa6800640 1 -- start start 2024-01-15T18:05:04.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.716+0000 7f78e3400640 1 --2- 172.21.15.27:0/2375858441 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f78c4066940 0x7f78c4068e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f78e3400640 1 -- 172.21.15.27:0/2375858441 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f78d4083550 con 0x7f78e40543b0 2024-01-15T18:05:04.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f78e3400640 1 --2- 172.21.15.27:0/2375858441 >> [v2:172.21.15.27:6824/366025692,v1:172.21.15.27:6825/366025692] conn(0x7f78c406eae0 0x7f78c4070f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f78e3400640 1 -- 172.21.15.27:0/2375858441 --> [v2:172.21.15.27:6824/366025692,v1:172.21.15.27:6825/366025692] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f78c40715b0 con 0x7f78c406eae0 2024-01-15T18:05:04.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f4295000640 1 -- 172.21.15.27:0/4000390859 shutdown_connections 2024-01-15T18:05:04.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f4295000640 1 -- 172.21.15.27:0/4000390859 wait complete. 2024-01-15T18:05:04.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7fa8dc600640 1 -- 172.21.15.27:0/123376231 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8d4156e20 msgr2=0x7fa8d40ce160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7fa8dc600640 1 --2- 172.21.15.27:0/123376231 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8d4156e20 0x7fa8d40ce160 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc00c2e0 tx=0x7fa8cc01de30 comp rx=0 tx=0).stop 2024-01-15T18:05:04.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f396ca00640 1 -- 172.21.15.27:0/4243007658 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926154863 0 0) 0x7f39700d3b00 con 0x7f39701608d0 2024-01-15T18:05:04.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f396ca00640 1 -- 172.21.15.27:0/4243007658 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3950003530 con 0x7f39701608d0 2024-01-15T18:05:04.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f396ca00640 1 -- 172.21.15.27:0/4243007658 <== mon.1 v1:172.21.15.93:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4085357678 0 0) 0x7f3950003530 con 0x7f39701608d0 2024-01-15T18:05:04.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f396ca00640 1 -- 172.21.15.27:0/4243007658 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f39700d1ab0 msgr2=0x7f39700b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:04.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f396ca00640 1 --2- 172.21.15.27:0/4243007658 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f39700d1ab0 0x7f39700b2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f396ca00640 1 -- 172.21.15.27:0/4243007658 >> v1:172.21.15.27:6789/0 conn(0x7f3970160cb0 legacy=0x7f39700d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:04.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f396ca00640 1 -- 172.21.15.27:0/4243007658 --> v1:172.21.15.93:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39700cdf90 con 0x7f39701608d0 2024-01-15T18:05:04.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f396ca00640 1 -- 172.21.15.27:0/4243007658 <== mon.1 v1:172.21.15.93:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f3958001680 con 0x7f39701608d0 2024-01-15T18:05:04.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f396ca00640 1 -- 172.21.15.27:0/4243007658 >> v1:172.21.15.93:6789/0 conn(0x7f39701608d0 legacy=0x7f39700d0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f396ca00640 1 --2- 172.21.15.27:0/4243007658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39500039f0 0x7f3950003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f396ca00640 1 --2- 172.21.15.27:0/4243007658 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f39500047d0 0x7f3950004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f396ca00640 1 --2- 172.21.15.27:0/4243007658 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3950005190 0x7f395000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f396ca00640 1 -- 172.21.15.27:0/4243007658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39700d3de0 con 0x7f39500039f0 2024-01-15T18:05:04.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f396ca00640 1 -- 172.21.15.27:0/4243007658 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395000c060 con 0x7f39500047d0 2024-01-15T18:05:04.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f396ca00640 1 -- 172.21.15.27:0/4243007658 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f395000c220 con 0x7f3950005190 2024-01-15T18:05:04.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f396de00640 1 --2- 172.21.15.27:0/4243007658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39500039f0 0x7f3950003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f396e800640 1 --2- 172.21.15.27:0/4243007658 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3950005190 0x7f395000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7fcc3e000640 1 -- 172.21.15.27:0/2064329409 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc380072d0 con 0x7fcc480b1370 2024-01-15T18:05:04.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7fcc3e000640 1 -- 172.21.15.27:0/2064329409 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc38007430 con 0x7fcc480b1370 2024-01-15T18:05:04.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f060f200640 1 -- 172.21.15.27:0/3996893591 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0608161a10 msgr2=0x7f06080706a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f060f200640 1 --2- 172.21.15.27:0/3996893591 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0608161a10 0x7f06080706a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f06080b1dc0 tx=0x7f05f801c180 comp rx=0 tx=0).stop 2024-01-15T18:05:04.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7fcc3e000640 1 -- 172.21.15.27:0/2064329409 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc380047f0 con 0x7fcc480b1370 2024-01-15T18:05:04.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f78eb400640 1 --2- 172.21.15.27:0/2375858441 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f78c4066940 0x7f78c4068e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f78e3400640 1 -- 172.21.15.27:0/2375858441 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78d4050920 con 0x7f78e40543b0 2024-01-15T18:05:04.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7fa8dc600640 1 -- 172.21.15.27:0/123376231 shutdown_connections 2024-01-15T18:05:04.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f396e800640 1 -- 172.21.15.27:0/4243007658 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f39500047d0 msgr2=0x7f3950004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:04.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f396e800640 1 --2- 172.21.15.27:0/4243007658 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f39500047d0 0x7f3950004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f396e800640 1 -- 172.21.15.27:0/4243007658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39500039f0 msgr2=0x7f3950003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:04.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f396e800640 1 --2- 172.21.15.27:0/4243007658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39500039f0 0x7f3950003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f396e800640 1 -- 172.21.15.27:0/4243007658 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39700cdf90 con 0x7f3950005190 2024-01-15T18:05:04.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7fcc4d200640 1 -- 172.21.15.27:0/2064329409 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc4814c4a0 con 0x7fcc480b1370 2024-01-15T18:05:04.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7fcc4d200640 1 -- 172.21.15.27:0/2064329409 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc4814bb20 con 0x7fcc480b1370 2024-01-15T18:05:04.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f78ebe00640 1 --2- 172.21.15.27:0/2375858441 >> [v2:172.21.15.27:6824/366025692,v1:172.21.15.27:6825/366025692] conn(0x7f78c406eae0 0x7f78c4070f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7fa8dc600640 1 --2- 172.21.15.27:0/123376231 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8d4070bd0 0x7fa8d406c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7fa8dc600640 1 --2- 172.21.15.27:0/123376231 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8d4156e20 0x7fa8d40ce160 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7fa8dc600640 1 -- 172.21.15.27:0/123376231 >> 172.21.15.27:0/123376231 conn(0x7fa8d40b0850 msgr2=0x7fa8d40b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:04.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f0aa6800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0a9804f980 0x7f0a980389c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f0aa6800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0a98050520 0x7f0a9804d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7fcc37400640 1 -- 172.21.15.27:0/2064329409 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcc4806e7e0 con 0x7fcc480b1370 2024-01-15T18:05:04.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f060f200640 1 -- 172.21.15.27:0/3996893591 shutdown_connections 2024-01-15T18:05:04.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f060f200640 1 --2- 172.21.15.27:0/3996893591 >> v2:172.21.15.27:3300/0 conn(0x7f06080b3bb0 0x7f0608165550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7fa8dc600640 1 -- 172.21.15.27:0/123376231 shutdown_connections 2024-01-15T18:05:04.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7fa8dc600640 1 -- 172.21.15.27:0/123376231 wait complete. 2024-01-15T18:05:04.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f060f200640 1 --2- 172.21.15.27:0/3996893591 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0608070f90 0x7f06080b3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f060f200640 1 --2- 172.21.15.27:0/3996893591 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0608161a10 0x7f06080706a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f0a9f400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0a9804f980 0x7f0a980389c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f0aa6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a980e7490 0x7f0a9804d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f0a9f400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0a9804f980 0x7f0a980389c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:42596/0 (socket says 172.21.15.27:42596) 2024-01-15T18:05:04.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f0a9ea00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0a98050520 0x7f0a9804d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f0a9ea00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0a98050520 0x7f0a9804d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:51540/0 (socket says 172.21.15.27:51540) 2024-01-15T18:05:04.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f0aa6800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a98189b00 con 0x7f0a980e7490 2024-01-15T18:05:04.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f0aa6800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a98189da0 con 0x7f0a98050520 2024-01-15T18:05:04.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f0aa6800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9818a080 con 0x7f0a9804f980 2024-01-15T18:05:04.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f0a9fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a980e7490 0x7f0a9804d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.720+0000 7f78ebe00640 1 --2- 172.21.15.27:0/2375858441 >> [v2:172.21.15.27:6824/366025692,v1:172.21.15.27:6825/366025692] conn(0x7f78c406eae0 0x7f78c4070f00 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f060f200640 1 -- 172.21.15.27:0/3996893591 >> 172.21.15.27:0/3996893591 conn(0x7f06080b0d90 msgr2=0x7f06080b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:04.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f78eb400640 1 --2- 172.21.15.27:0/2375858441 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f78c4066940 0x7f78c4068e00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f78dc0b2000 tx=0x7f78dc0aa070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f3f17200640 1 Processor -- start 2024-01-15T18:05:04.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f3f17200640 1 -- start start 2024-01-15T18:05:04.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f060f200640 1 -- 172.21.15.27:0/3996893591 shutdown_connections 2024-01-15T18:05:04.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f060f200640 1 -- 172.21.15.27:0/3996893591 wait complete. 2024-01-15T18:05:04.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7fa8dc600640 1 Processor -- start 2024-01-15T18:05:04.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f3f17200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ef80069e0 0x7f3f18155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f4295000640 1 Processor -- start 2024-01-15T18:05:04.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f3f17200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f181f7bf0 0x7f3f1814cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f4295000640 1 -- start start 2024-01-15T18:05:04.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f3f17200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3f1815bd50 0x7f3f181501a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f3f15e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ef80069e0 0x7f3f18155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f3f15e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ef80069e0 0x7f3f18155c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:51546/0 (socket says 172.21.15.27:51546) 2024-01-15T18:05:04.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f3f15400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f181f7bf0 0x7f3f1814cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f78e3400640 1 -- 172.21.15.27:0/2375858441 <== osd.2 v2:172.21.15.27:6824/366025692 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f78c40715b0 con 0x7f78c406eae0 2024-01-15T18:05:04.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f0a9fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a980e7490 0x7f0a9804d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59134/0 (socket says 172.21.15.27:59134) 2024-01-15T18:05:04.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f0a9fe00640 1 -- 172.21.15.27:0/3222726296 learned_addr learned my addr 172.21.15.27:0/3222726296 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:04.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f0a9fe00640 1 -- 172.21.15.27:0/3222726296 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0a9804f980 msgr2=0x7f0a980389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f0a9fe00640 1 --2- 172.21.15.27:0/3222726296 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0a9804f980 0x7f0a980389c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f0a9fe00640 1 -- 172.21.15.27:0/3222726296 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0a98050520 msgr2=0x7f0a9804d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f0a9fe00640 1 --2- 172.21.15.27:0/3222726296 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0a98050520 0x7f0a9804d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f0a9fe00640 1 -- 172.21.15.27:0/3222726296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a88009420 con 0x7f0a980e7490 2024-01-15T18:05:04.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f0a9fe00640 1 --2- 172.21.15.27:0/3222726296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a980e7490 0x7f0a9804d830 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0a880ae040 tx=0x7f0a880191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f0a97400640 1 -- 172.21.15.27:0/3222726296 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a880b4410 con 0x7f0a980e7490 2024-01-15T18:05:04.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f0a97400640 1 -- 172.21.15.27:0/3222726296 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a880b4570 con 0x7f0a980e7490 2024-01-15T18:05:04.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f0a97400640 1 -- 172.21.15.27:0/3222726296 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a880b4410 con 0x7f0a980e7490 2024-01-15T18:05:04.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7fa8dc600640 1 -- start start 2024-01-15T18:05:04.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f0aa6800640 1 -- 172.21.15.27:0/3222726296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a98055090 con 0x7f0a980e7490 2024-01-15T18:05:04.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f060f200640 1 Processor -- start 2024-01-15T18:05:04.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f396e800640 1 --2- 172.21.15.27:0/4243007658 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3950005190 0x7f395000b990 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3968009f00 tx=0x7f396801cca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89c055f3bd96cbba server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f0aa6800640 1 -- 172.21.15.27:0/3222726296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a980dd070 con 0x7f0a980e7490 2024-01-15T18:05:04.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f060f200640 1 -- start start 2024-01-15T18:05:04.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f4295000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f429006cc10 0x7f42900cd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f4295000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4290157890 0x7f4290157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f4295000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42901469d0 0x7f4290146e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f4295000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4290154a90 con 0x7f42901469d0 2024-01-15T18:05:04.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f4295000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42901583e0 con 0x7f4290157890 2024-01-15T18:05:04.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f4295000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42901586c0 con 0x7f429006cc10 2024-01-15T18:05:04.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f428d400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4290157890 0x7f4290157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f428d400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4290157890 0x7f4290157d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:51560/0 (socket says 172.21.15.27:51560) 2024-01-15T18:05:04.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7fa8dc600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8d4070bd0 0x7fa8d41629a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7fa8dc600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8d4156e20 0x7fa8d4148880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7fa8dc600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8d415cd90 0x7fa8d4159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f396ca00640 1 -- 172.21.15.27:0/4243007658 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39680a5a70 con 0x7f3950005190 2024-01-15T18:05:04.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f396ca00640 1 -- 172.21.15.27:0/4243007658 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39680a5bd0 con 0x7f3950005190 2024-01-15T18:05:04.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f396ca00640 1 -- 172.21.15.27:0/4243007658 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39680ad440 con 0x7f3950005190 2024-01-15T18:05:04.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7fa8d9200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8d4156e20 0x7fa8d4148880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7fa8d9200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8d4156e20 0x7fa8d4148880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59170/0 (socket says 172.21.15.27:59170) 2024-01-15T18:05:04.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7fa8d9c00640 1 --2- 172.21.15.27:0/1474593609 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8d4070bd0 0x7fa8d41629a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f060f200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0608070f90 0x7f0608158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7fa8d9200640 1 -- 172.21.15.27:0/1474593609 learned_addr learned my addr 172.21.15.27:0/1474593609 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:04.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7fcc3e000640 1 -- 172.21.15.27:0/2064329409 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fcc38004b60 con 0x7fcc480b1370 2024-01-15T18:05:04.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7fcc3e000640 1 --2- 172.21.15.27:0/2064329409 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcc200669f0 0x7fcc20068eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7fcc3e000640 1 -- 172.21.15.27:0/2064329409 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc38083de0 con 0x7fcc480b1370 2024-01-15T18:05:04.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7fcc3e000640 1 --2- 172.21.15.27:0/2064329409 >> [v2:172.21.15.27:6802/464948094,v1:172.21.15.27:6804/464948094] conn(0x7fcc2006ec10 0x7fcc20071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7fcc3e000640 1 -- 172.21.15.27:0/2064329409 --> [v2:172.21.15.27:6802/464948094,v1:172.21.15.27:6804/464948094] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcc200716e0 con 0x7fcc2006ec10 2024-01-15T18:05:04.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7fcc3e000640 1 -- 172.21.15.27:0/2064329409 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc380893d0 con 0x7fcc480b1370 2024-01-15T18:05:04.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f428d400640 1 -- 172.21.15.27:0/3841749268 learned_addr learned my addr 172.21.15.27:0/3841749268 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:04.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f428de00640 1 --2- 172.21.15.27:0/3841749268 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f429006cc10 0x7f42900cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7fa8da600640 1 --2- 172.21.15.27:0/1474593609 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8d415cd90 0x7fa8d4159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f428d400640 1 -- 172.21.15.27:0/3841749268 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f429006cc10 msgr2=0x7f42900cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f428d400640 1 --2- 172.21.15.27:0/3841749268 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f429006cc10 0x7f42900cd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f428d400640 1 -- 172.21.15.27:0/3841749268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42901469d0 msgr2=0x7f4290146e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f428d400640 1 --2- 172.21.15.27:0/3841749268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42901469d0 0x7f4290146e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7fa8dc600640 1 -- 172.21.15.27:0/1474593609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d4162ee0 con 0x7fa8d4156e20 2024-01-15T18:05:04.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f428d400640 1 -- 172.21.15.27:0/3841749268 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42901589a0 con 0x7f4290157890 2024-01-15T18:05:04.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f428d400640 1 --2- 172.21.15.27:0/3841749268 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4290157890 0x7f4290157d10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f428000e7c0 tx=0x7f428000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7fa8dc600640 1 -- 172.21.15.27:0/1474593609 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d41483f0 con 0x7fa8d4070bd0 2024-01-15T18:05:04.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7fa8dc600640 1 -- 172.21.15.27:0/1474593609 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d4148dc0 con 0x7fa8d415cd90 2024-01-15T18:05:04.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7fcc45e00640 1 --2- 172.21.15.27:0/2064329409 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcc200669f0 0x7fcc20068eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.724+0000 7f3f15400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f181f7bf0 0x7f3f1814cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59144/0 (socket says 172.21.15.27:59144) 2024-01-15T18:05:04.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f3f15e00640 1 -- 172.21.15.27:0/810295683 learned_addr learned my addr 172.21.15.27:0/810295683 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:04.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f3f17200640 1 -- 172.21.15.27:0/810295683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1814d020 con 0x7f3f181f7bf0 2024-01-15T18:05:04.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f3f17200640 1 -- 172.21.15.27:0/810295683 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f18156a70 con 0x7f3ef80069e0 2024-01-15T18:05:04.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f3f17200640 1 -- 172.21.15.27:0/810295683 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f181f8070 con 0x7f3f1815bd50 2024-01-15T18:05:04.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7fa8da600640 1 -- 172.21.15.27:0/1474593609 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8d4070bd0 msgr2=0x7fa8d41629a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7fa8da600640 1 --2- 172.21.15.27:0/1474593609 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8d4070bd0 0x7fa8d41629a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7fa8da600640 1 -- 172.21.15.27:0/1474593609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8d4156e20 msgr2=0x7fa8d4148880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7fa8da600640 1 --2- 172.21.15.27:0/1474593609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8d4156e20 0x7fa8d4148880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7fa8da600640 1 -- 172.21.15.27:0/1474593609 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8cc00c050 con 0x7fa8d415cd90 2024-01-15T18:05:04.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f3f16800640 1 --2- 172.21.15.27:0/810295683 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3f1815bd50 0x7f3f181501a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7fcc46800640 1 --2- 172.21.15.27:0/2064329409 >> [v2:172.21.15.27:6802/464948094,v1:172.21.15.27:6804/464948094] conn(0x7fcc2006ec10 0x7fcc20071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f4286000640 1 -- 172.21.15.27:0/3841749268 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428000ee70 con 0x7f4290157890 2024-01-15T18:05:04.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f4286000640 1 -- 172.21.15.27:0/3841749268 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f428000c940 con 0x7f4290157890 2024-01-15T18:05:04.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f4286000640 1 -- 172.21.15.27:0/3841749268 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428000ccf0 con 0x7f4290157890 2024-01-15T18:05:04.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7fa8d9c00640 1 --2- 172.21.15.27:0/1474593609 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8d4070bd0 0x7fa8d41629a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:04.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f0607e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0608070f90 0x7f0608158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f0607e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0608070f90 0x7f0608158910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:51570/0 (socket says 172.21.15.27:51570) 2024-01-15T18:05:04.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f0607e00640 1 -- 172.21.15.27:0/633850887 learned_addr learned my addr 172.21.15.27:0/633850887 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:04.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f396f200640 1 -- 172.21.15.27:0/4243007658 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3950005190 msgr2=0x7f395000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f396f200640 1 --2- 172.21.15.27:0/4243007658 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3950005190 0x7f395000b990 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3968009f00 tx=0x7f396801cca0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f3f15400640 1 -- 172.21.15.27:0/810295683 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3f1815bd50 msgr2=0x7f3f181501a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f3f15400640 1 --2- 172.21.15.27:0/810295683 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3f1815bd50 0x7f3f181501a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f3f15400640 1 -- 172.21.15.27:0/810295683 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ef80069e0 msgr2=0x7f3f18155c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f3f15400640 1 --2- 172.21.15.27:0/810295683 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ef80069e0 0x7f3f18155c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f3f15400640 1 -- 172.21.15.27:0/810295683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f180b3ec0 con 0x7f3f181f7bf0 2024-01-15T18:05:04.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f396e800640 1 -- 172.21.15.27:0/4243007658 reap_dead start 2024-01-15T18:05:04.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7fcc45e00640 1 --2- 172.21.15.27:0/2064329409 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcc200669f0 0x7fcc20068eb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcc3001c2b0 tx=0x7fcc30008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f3f15e00640 1 --2- 172.21.15.27:0/810295683 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ef80069e0 0x7f3f18155c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:04.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7fa8da600640 1 --2- 172.21.15.27:0/1474593609 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8d415cd90 0x7fa8d4159bb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc00c410 tx=0x7fa8bc00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f4295000640 1 -- 172.21.15.27:0/3841749268 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4290154ca0 con 0x7f4290157890 2024-01-15T18:05:04.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.728+0000 7f4295000640 1 -- 172.21.15.27:0/3841749268 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f429014d5a0 con 0x7f4290157890 2024-01-15T18:05:04.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7fcc46800640 1 --2- 172.21.15.27:0/2064329409 >> [v2:172.21.15.27:6802/464948094,v1:172.21.15.27:6804/464948094] conn(0x7fcc2006ec10 0x7fcc20071030 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7f3f15400640 1 --2- 172.21.15.27:0/810295683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f181f7bf0 0x7f3f1814cae0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3f000ac000 tx=0x7f3ef8007a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7f060f200640 1 --2- 172.21.15.27:0/633850887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06080b3bb0 0x7f0608158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7fa8d1600640 1 -- 172.21.15.27:0/1474593609 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc0074e0 con 0x7fa8d415cd90 2024-01-15T18:05:04.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7fa8d1600640 1 -- 172.21.15.27:0/1474593609 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8bc007640 con 0x7fa8d415cd90 2024-01-15T18:05:04.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7fa8d1600640 1 -- 172.21.15.27:0/1474593609 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc004a10 con 0x7fa8d415cd90 2024-01-15T18:05:04.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7f396f200640 1 -- 172.21.15.27:0/4243007658 shutdown_connections 2024-01-15T18:05:04.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7f396f200640 1 -- 172.21.15.27:0/4243007658 >> 172.21.15.27:0/4243007658 conn(0x7f397009c410 msgr2=0x7f39700d30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:04.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7f0a94c00640 1 -- 172.21.15.27:0/3222726296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0a9804fe10 con 0x7f0a980e7490 2024-01-15T18:05:04.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7f4295000640 1 -- 172.21.15.27:0/3841749268 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4290158be0 con 0x7f4290157890 2024-01-15T18:05:04.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7f060f200640 1 --2- 172.21.15.27:0/633850887 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0608161a10 0x7f06080ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7fa8dc600640 1 -- 172.21.15.27:0/1474593609 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d41477e0 con 0x7fa8d415cd90 2024-01-15T18:05:04.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7fa8dc600640 1 -- 172.21.15.27:0/1474593609 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d4147da0 con 0x7fa8d415cd90 2024-01-15T18:05:04.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7f396f200640 1 -- 172.21.15.27:0/4243007658 shutdown_connections 2024-01-15T18:05:04.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7f060f200640 1 -- 172.21.15.27:0/633850887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060809a1b0 con 0x7f06080b3bb0 2024-01-15T18:05:04.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7f060d200640 1 --2- 172.21.15.27:0/633850887 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0608161a10 0x7f06080ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7f060f200640 1 -- 172.21.15.27:0/633850887 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060815a150 con 0x7f0608070f90 2024-01-15T18:05:04.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7f060f200640 1 -- 172.21.15.27:0/633850887 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06080cdef0 con 0x7f0608161a10 2024-01-15T18:05:04.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7f0607400640 1 --2- 172.21.15.27:0/633850887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06080b3bb0 0x7f0608158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7f396f200640 1 -- 172.21.15.27:0/4243007658 wait complete. 2024-01-15T18:05:04.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7fcc3e000640 1 -- 172.21.15.27:0/2064329409 <== osd.0 v2:172.21.15.27:6802/464948094 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fcc200716e0 con 0x7fcc2006ec10 2024-01-15T18:05:04.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7f4286000640 1 -- 172.21.15.27:0/3841749268 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f4280013470 con 0x7f4290157890 2024-01-15T18:05:04.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7fa8caa00640 1 -- 172.21.15.27:0/1474593609 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa8d4072980 con 0x7fa8d415cd90 2024-01-15T18:05:04.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7f0607400640 1 -- 172.21.15.27:0/633850887 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0608161a10 msgr2=0x7f06080ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7f0607400640 1 --2- 172.21.15.27:0/633850887 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0608161a10 0x7f06080ccd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7f0607400640 1 -- 172.21.15.27:0/633850887 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0608070f90 msgr2=0x7f0608158910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7f0607400640 1 --2- 172.21.15.27:0/633850887 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0608070f90 0x7f0608158910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7f0607400640 1 -- 172.21.15.27:0/633850887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f80094d0 con 0x7f06080b3bb0 2024-01-15T18:05:04.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.732+0000 7f396f200640 1 Processor -- start 2024-01-15T18:05:04.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f0607400640 1 --2- 172.21.15.27:0/633850887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06080b3bb0 0x7f0608158e50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f05f000b3f0 tx=0x7f05f000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7fa8d1600640 1 -- 172.21.15.27:0/1474593609 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fa8bc004b70 con 0x7fa8d415cd90 2024-01-15T18:05:04.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f4286000640 1 --2- 172.21.15.27:0/3841749268 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4268066900 0x7f4268068dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f4286000640 1 -- 172.21.15.27:0/3841749268 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4280084300 con 0x7f4290157890 2024-01-15T18:05:04.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f05ffe00640 1 -- 172.21.15.27:0/633850887 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f0011830 con 0x7f06080b3bb0 2024-01-15T18:05:04.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f05ffe00640 1 -- 172.21.15.27:0/633850887 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f0011990 con 0x7f06080b3bb0 2024-01-15T18:05:04.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f05ffe00640 1 -- 172.21.15.27:0/633850887 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f0011d40 con 0x7f06080b3bb0 2024-01-15T18:05:04.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f4286000640 1 --2- 172.21.15.27:0/3841749268 >> [v2:172.21.15.27:6816/934967897,v1:172.21.15.27:6817/934967897] conn(0x7f426806eb10 0x7f4268070f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f4286000640 1 -- 172.21.15.27:0/3841749268 --> [v2:172.21.15.27:6816/934967897,v1:172.21.15.27:6817/934967897] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f42680715e0 con 0x7f426806eb10 2024-01-15T18:05:04.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f4286000640 1 -- 172.21.15.27:0/3841749268 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4280086920 con 0x7f4290157890 2024-01-15T18:05:04.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f428de00640 1 --2- 172.21.15.27:0/3841749268 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4268066900 0x7f4268068dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f396f200640 1 -- start start 2024-01-15T18:05:04.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7fa8d1600640 1 --2- 172.21.15.27:0/1474593609 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa8ac066940 0x7fa8ac068e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7fa8d1600640 1 -- 172.21.15.27:0/1474593609 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8bc083ff0 con 0x7fa8d415cd90 2024-01-15T18:05:04.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f428e800640 1 --2- 172.21.15.27:0/3841749268 >> [v2:172.21.15.27:6816/934967897,v1:172.21.15.27:6817/934967897] conn(0x7f426806eb10 0x7f4268070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7fa8d1600640 1 --2- 172.21.15.27:0/1474593609 >> [v2:172.21.15.93:6805/838236833,v1:172.21.15.93:6809/838236833] conn(0x7fa8ac06eb60 0x7fa8ac070f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7fa8d1600640 1 -- 172.21.15.27:0/1474593609 --> [v2:172.21.15.93:6805/838236833,v1:172.21.15.93:6809/838236833] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa8ac071630 con 0x7fa8ac06eb60 2024-01-15T18:05:04.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7fa8d1600640 1 -- 172.21.15.27:0/1474593609 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa8bc086610 con 0x7fa8d415cd90 2024-01-15T18:05:04.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7fa8d9c00640 1 --2- 172.21.15.27:0/1474593609 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa8ac066940 0x7fa8ac068e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f396f200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3950005190 0x7f39700ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f396f200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f397015c620 0x7f397009e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f396f200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f397009fef0 0x7f39700a0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f396f200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397009eaf0 con 0x7f397015c620 2024-01-15T18:05:04.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f396f200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39700ca840 con 0x7f397009fef0 2024-01-15T18:05:04.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f396f200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39700cab20 con 0x7f3950005190 2024-01-15T18:05:04.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f3f0e000640 1 -- 172.21.15.27:0/810295683 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f000054e0 con 0x7f3f181f7bf0 2024-01-15T18:05:04.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f3f0e000640 1 -- 172.21.15.27:0/810295683 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f000a5a60 con 0x7f3f181f7bf0 2024-01-15T18:05:04.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f3f0e000640 1 -- 172.21.15.27:0/810295683 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f00006a00 con 0x7f3f181f7bf0 2024-01-15T18:05:04.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f3f17200640 1 -- 172.21.15.27:0/810295683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f18156d10 con 0x7f3f181f7bf0 2024-01-15T18:05:04.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f3f17200640 1 -- 172.21.15.27:0/810295683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f181572d0 con 0x7f3f181f7bf0 2024-01-15T18:05:04.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7fa8d9200640 1 --2- 172.21.15.27:0/1474593609 >> [v2:172.21.15.93:6805/838236833,v1:172.21.15.93:6809/838236833] conn(0x7fa8ac06eb60 0x7fa8ac070f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f396e800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f397009fef0 0x7f39700a0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f396e800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f397009fef0 0x7f39700a0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:51584/0 (socket says 172.21.15.27:51584) 2024-01-15T18:05:04.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f396e800640 1 -- 172.21.15.27:0/1445334659 learned_addr learned my addr 172.21.15.27:0/1445334659 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:04.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f3f17200640 1 -- 172.21.15.27:0/810295683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3f180b3d40 con 0x7f3f181f7bf0 2024-01-15T18:05:04.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f428e800640 1 --2- 172.21.15.27:0/3841749268 >> [v2:172.21.15.27:6816/934967897,v1:172.21.15.27:6817/934967897] conn(0x7f426806eb10 0x7f4268070f30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f428de00640 1 --2- 172.21.15.27:0/3841749268 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4268066900 0x7f4268068dc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4278004710 tx=0x7f4278009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7fa8d9c00640 1 --2- 172.21.15.27:0/1474593609 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa8ac066940 0x7fa8ac068e00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc0af000 tx=0x7fa8cc0a8070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f396de00640 1 --2- 172.21.15.27:0/1445334659 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3950005190 0x7f39700ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7fa8d9200640 1 --2- 172.21.15.27:0/1474593609 >> [v2:172.21.15.93:6805/838236833,v1:172.21.15.93:6809/838236833] conn(0x7fa8ac06eb60 0x7fa8ac070f80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7f396e800640 1 -- 172.21.15.27:0/1445334659 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3950005190 msgr2=0x7f39700ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7f396e800640 1 --2- 172.21.15.27:0/1445334659 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3950005190 0x7f39700ca1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7f396e800640 1 -- 172.21.15.27:0/1445334659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f397015c620 msgr2=0x7f397009e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7f396e800640 1 --2- 172.21.15.27:0/1445334659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f397015c620 0x7f397009e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7f396e800640 1 -- 172.21.15.27:0/1445334659 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39700cae00 con 0x7f397009fef0 2024-01-15T18:05:04.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.736+0000 7f0a97400640 1 -- 172.21.15.27:0/3222726296 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f0a880b2040 con 0x7f0a980e7490 2024-01-15T18:05:04.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7f0a97400640 1 --2- 172.21.15.27:0/3222726296 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0a78066980 0x7f0a78068e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7f0a97400640 1 -- 172.21.15.27:0/3222726296 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a8811e500 con 0x7f0a980e7490 2024-01-15T18:05:04.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7f396e800640 1 --2- 172.21.15.27:0/1445334659 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f397009fef0 0x7f39700a0370 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f396801c710 tx=0x7f3950005800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7f060f200640 1 -- 172.21.15.27:0/633850887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06080ce1d0 con 0x7f06080b3bb0 2024-01-15T18:05:04.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7f060f200640 1 -- 172.21.15.27:0/633850887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06080cd2a0 con 0x7f06080b3bb0 2024-01-15T18:05:04.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7f060f200640 1 -- 172.21.15.27:0/633850887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0608068000 con 0x7f06080b3bb0 2024-01-15T18:05:04.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7f0a97400640 1 --2- 172.21.15.27:0/3222726296 >> [v2:172.21.15.27:6800/3601044148,v1:172.21.15.27:6801/3601044148] conn(0x7f0a7806eb20 0x7f0a78070f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7f0a97400640 1 -- 172.21.15.27:0/3222726296 --> [v2:172.21.15.27:6800/3601044148,v1:172.21.15.27:6801/3601044148] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0a780715f0 con 0x7f0a7806eb20 2024-01-15T18:05:04.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7f0a97400640 1 -- 172.21.15.27:0/3222726296 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a880eb7e0 con 0x7f0a980e7490 2024-01-15T18:05:04.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7f0a9f400640 1 --2- 172.21.15.27:0/3222726296 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0a78066980 0x7f0a78068e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7f3966000640 1 -- 172.21.15.27:0/1445334659 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3968004240 con 0x7f397009fef0 2024-01-15T18:05:04.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7f0a9ea00640 1 --2- 172.21.15.27:0/3222726296 >> [v2:172.21.15.27:6800/3601044148,v1:172.21.15.27:6801/3601044148] conn(0x7f0a7806eb20 0x7f0a78070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7f3966000640 1 -- 172.21.15.27:0/1445334659 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39680043a0 con 0x7f397009fef0 2024-01-15T18:05:04.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7f396f200640 1 -- 172.21.15.27:0/1445334659 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39700cb0e0 con 0x7f397009fef0 2024-01-15T18:05:04.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7fa8d1600640 1 -- 172.21.15.27:0/1474593609 <== osd.6 v2:172.21.15.93:6805/838236833 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fa8ac071630 con 0x7fa8ac06eb60 2024-01-15T18:05:04.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7f3966000640 1 -- 172.21.15.27:0/1445334659 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3968004710 con 0x7f397009fef0 2024-01-15T18:05:04.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7f0a9f400640 1 --2- 172.21.15.27:0/3222726296 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0a78066980 0x7f0a78068e40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0aa002cef0 tx=0x7f0aa008c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7f396f200640 1 -- 172.21.15.27:0/1445334659 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39700a0d60 con 0x7f397009fef0 2024-01-15T18:05:04.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7f4286000640 1 -- 172.21.15.27:0/3841749268 <== osd.3 v2:172.21.15.27:6816/934967897 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f42680715e0 con 0x7f426806eb10 2024-01-15T18:05:04.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7f3f0e000640 1 -- 172.21.15.27:0/810295683 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f3f000b4040 con 0x7f3f181f7bf0 2024-01-15T18:05:04.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.744+0000 7f3f0e000640 1 --2- 172.21.15.27:0/810295683 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3ef00669f0 0x7f3ef0068eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.744+0000 7f3f0e000640 1 -- 172.21.15.27:0/810295683 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3f0011e030 con 0x7f3f181f7bf0 2024-01-15T18:05:04.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.744+0000 7f3f15e00640 1 --2- 172.21.15.27:0/810295683 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3ef00669f0 0x7f3ef0068eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.744+0000 7f3f0e000640 1 --2- 172.21.15.27:0/810295683 >> [v2:172.21.15.93:6803/1254182834,v1:172.21.15.93:6806/1254182834] conn(0x7f3ef006ec10 0x7f3ef0071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.744+0000 7f3f0e000640 1 -- 172.21.15.27:0/810295683 --> [v2:172.21.15.93:6803/1254182834,v1:172.21.15.93:6806/1254182834] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ef00716e0 con 0x7f3ef006ec10 2024-01-15T18:05:04.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.744+0000 7f3f0e000640 1 -- 172.21.15.27:0/810295683 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f0011e3e0 con 0x7f3f181f7bf0 2024-01-15T18:05:04.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.740+0000 7f396f200640 1 -- 172.21.15.27:0/1445334659 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f397004e850 con 0x7f397009fef0 2024-01-15T18:05:04.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.744+0000 7f0a9ea00640 1 --2- 172.21.15.27:0/3222726296 >> [v2:172.21.15.27:6800/3601044148,v1:172.21.15.27:6801/3601044148] conn(0x7f0a7806eb20 0x7f0a78070f40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.744+0000 7f3f15e00640 1 --2- 172.21.15.27:0/810295683 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3ef00669f0 0x7f3ef0068eb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3f18156900 tx=0x7f3f0800c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.744+0000 7f3966000640 1 -- 172.21.15.27:0/1445334659 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f39680c1460 con 0x7f397009fef0 2024-01-15T18:05:04.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.744+0000 7f05ffe00640 1 -- 172.21.15.27:0/633850887 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f05f001a400 con 0x7f06080b3bb0 2024-01-15T18:05:04.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.744+0000 7f05ffe00640 1 --2- 172.21.15.27:0/633850887 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05e0066970 0x7f05e0068e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.744+0000 7f3f16800640 1 --2- 172.21.15.27:0/810295683 >> [v2:172.21.15.93:6803/1254182834,v1:172.21.15.93:6806/1254182834] conn(0x7f3ef006ec10 0x7f3ef0071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.744+0000 7f05ffe00640 1 -- 172.21.15.27:0/633850887 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05f0084920 con 0x7f06080b3bb0 2024-01-15T18:05:04.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.744+0000 7f0607e00640 1 --2- 172.21.15.27:0/633850887 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05e0066970 0x7f05e0068e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.744+0000 7f3966000640 1 --2- 172.21.15.27:0/1445334659 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f39480669c0 0x7f3948068e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.744+0000 7f3966000640 1 -- 172.21.15.27:0/1445334659 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f396811f0e0 con 0x7f397009fef0 2024-01-15T18:05:04.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.744+0000 7f3966000640 1 --2- 172.21.15.27:0/1445334659 >> [v2:172.21.15.93:6824/978428062,v1:172.21.15.93:6825/978428062] conn(0x7f394806ebe0 0x7f3948071000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.744+0000 7f3966000640 1 -- 172.21.15.27:0/1445334659 --> [v2:172.21.15.93:6824/978428062,v1:172.21.15.93:6825/978428062] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f39480716b0 con 0x7f394806ebe0 2024-01-15T18:05:04.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.744+0000 7f3966000640 1 -- 172.21.15.27:0/1445334659 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3968121640 con 0x7f397009fef0 2024-01-15T18:05:04.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.744+0000 7f05ffe00640 1 --2- 172.21.15.27:0/633850887 >> [v2:172.21.15.93:6800/2348354791,v1:172.21.15.93:6801/2348354791] conn(0x7f05e006eb90 0x7f05e0070fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:04.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.744+0000 7f396de00640 1 --2- 172.21.15.27:0/1445334659 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f39480669c0 0x7f3948068e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.748+0000 7f396d400640 1 --2- 172.21.15.27:0/1445334659 >> [v2:172.21.15.93:6824/978428062,v1:172.21.15.93:6825/978428062] conn(0x7f394806ebe0 0x7f3948071000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.748+0000 7f0a97400640 1 -- 172.21.15.27:0/3222726296 <== osd.1 v2:172.21.15.27:6800/3601044148 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f0a780715f0 con 0x7f0a7806eb20 2024-01-15T18:05:04.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.748+0000 7f3f16800640 1 --2- 172.21.15.27:0/810295683 >> [v2:172.21.15.93:6803/1254182834,v1:172.21.15.93:6806/1254182834] conn(0x7f3ef006ec10 0x7f3ef0071030 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.748+0000 7f05ffe00640 1 -- 172.21.15.27:0/633850887 --> [v2:172.21.15.93:6800/2348354791,v1:172.21.15.93:6801/2348354791] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f05e0071660 con 0x7f05e006eb90 2024-01-15T18:05:04.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.748+0000 7f05ffe00640 1 -- 172.21.15.27:0/633850887 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05f0084dc0 con 0x7f06080b3bb0 2024-01-15T18:05:04.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.748+0000 7f0607e00640 1 --2- 172.21.15.27:0/633850887 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05e0066970 0x7f05e0068e30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f05f80090b0 tx=0x7f05f80a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.748+0000 7f396de00640 1 --2- 172.21.15.27:0/1445334659 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f39480669c0 0x7f3948068e80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3958000fe0 tx=0x7f39580013c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.748+0000 7f060d200640 1 --2- 172.21.15.27:0/633850887 >> [v2:172.21.15.93:6800/2348354791,v1:172.21.15.93:6801/2348354791] conn(0x7f05e006eb90 0x7f05e0070fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:04.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.748+0000 7f396d400640 1 --2- 172.21.15.27:0/1445334659 >> [v2:172.21.15.93:6824/978428062,v1:172.21.15.93:6825/978428062] conn(0x7f394806ebe0 0x7f3948071000 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.748+0000 7f3f0e000640 1 -- 172.21.15.27:0/810295683 <== osd.4 v2:172.21.15.93:6803/1254182834 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f3ef00716e0 con 0x7f3ef006ec10 2024-01-15T18:05:04.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.748+0000 7f3966000640 1 -- 172.21.15.27:0/1445334659 <== osd.7 v2:172.21.15.93:6824/978428062 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f39480716b0 con 0x7f394806ebe0 2024-01-15T18:05:04.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.748+0000 7f060d200640 1 --2- 172.21.15.27:0/633850887 >> [v2:172.21.15.93:6800/2348354791,v1:172.21.15.93:6801/2348354791] conn(0x7f05e006eb90 0x7f05e0070fb0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:04.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.756+0000 7f05ffe00640 1 -- 172.21.15.27:0/633850887 <== osd.5 v2:172.21.15.93:6800/2348354791 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f05e0071660 con 0x7f05e006eb90 2024-01-15T18:05:04.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.756+0000 7f78f2400640 1 -- 172.21.15.27:0/2375858441 --> [v2:172.21.15.27:6824/366025692,v1:172.21.15.27:6825/366025692] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f78bc002cc0 con 0x7f78c406eae0 2024-01-15T18:05:04.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.760+0000 7f78e3400640 1 -- 172.21.15.27:0/2375858441 <== osd.2 v2:172.21.15.27:6824/366025692 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f78bc002cc0 con 0x7f78c406eae0 2024-01-15T18:05:04.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7f78f2400640 1 -- 172.21.15.27:0/2375858441 >> [v2:172.21.15.27:6824/366025692,v1:172.21.15.27:6825/366025692] conn(0x7f78c406eae0 msgr2=0x7f78c4070f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7f78f2400640 1 --2- 172.21.15.27:0/2375858441 >> [v2:172.21.15.27:6824/366025692,v1:172.21.15.27:6825/366025692] conn(0x7f78c406eae0 0x7f78c4070f00 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.760+0000 7f4295000640 1 -- 172.21.15.27:0/3841749268 --> [v2:172.21.15.27:6816/934967897,v1:172.21.15.27:6817/934967897] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f42900b3900 con 0x7f426806eb10 2024-01-15T18:05:04.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7fcc37400640 1 -- 172.21.15.27:0/2064329409 --> [v2:172.21.15.27:6802/464948094,v1:172.21.15.27:6804/464948094] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcc4814cc60 con 0x7fcc2006ec10 2024-01-15T18:05:04.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7f4286000640 1 -- 172.21.15.27:0/3841749268 <== osd.3 v2:172.21.15.27:6816/934967897 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f42900b3900 con 0x7f426806eb10 2024-01-15T18:05:04.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7fcc3e000640 1 -- 172.21.15.27:0/2064329409 <== osd.0 v2:172.21.15.27:6802/464948094 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcc4814cc60 con 0x7fcc2006ec10 2024-01-15T18:05:04.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7f78f2400640 1 -- 172.21.15.27:0/2375858441 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f78c4066940 msgr2=0x7f78c4068e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7f78f2400640 1 --2- 172.21.15.27:0/2375858441 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f78c4066940 0x7f78c4068e00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f78dc0b2000 tx=0x7f78dc0aa070 comp rx=0 tx=0).stop 2024-01-15T18:05:04.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7f78f2400640 1 -- 172.21.15.27:0/2375858441 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78e40543b0 msgr2=0x7f78e4180700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7f78f2400640 1 --2- 172.21.15.27:0/2375858441 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78e40543b0 0x7f78e4180700 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f78d400c3c0 tx=0x7f78d400c880 comp rx=0 tx=0).stop 2024-01-15T18:05:04.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7fcc37400640 1 -- 172.21.15.27:0/2064329409 >> [v2:172.21.15.27:6802/464948094,v1:172.21.15.27:6804/464948094] conn(0x7fcc2006ec10 msgr2=0x7fcc20071030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7fcc37400640 1 --2- 172.21.15.27:0/2064329409 >> [v2:172.21.15.27:6802/464948094,v1:172.21.15.27:6804/464948094] conn(0x7fcc2006ec10 0x7fcc20071030 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7f78ebe00640 1 -- 172.21.15.27:0/2375858441 reap_dead start 2024-01-15T18:05:04.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7f4295000640 1 -- 172.21.15.27:0/3841749268 >> [v2:172.21.15.27:6816/934967897,v1:172.21.15.27:6817/934967897] conn(0x7f426806eb10 msgr2=0x7f4268070f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7f4295000640 1 --2- 172.21.15.27:0/3841749268 >> [v2:172.21.15.27:6816/934967897,v1:172.21.15.27:6817/934967897] conn(0x7f426806eb10 0x7f4268070f30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7fcc37400640 1 -- 172.21.15.27:0/2064329409 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcc200669f0 msgr2=0x7fcc20068eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7fcc37400640 1 --2- 172.21.15.27:0/2064329409 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcc200669f0 0x7fcc20068eb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcc3001c2b0 tx=0x7fcc30008040 comp rx=0 tx=0).stop 2024-01-15T18:05:04.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7fcc37400640 1 -- 172.21.15.27:0/2064329409 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcc480b1370 msgr2=0x7fcc4814b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7fcc37400640 1 --2- 172.21.15.27:0/2064329409 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcc480b1370 0x7fcc4814b5e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcc380011f0 tx=0x7fcc3800cb90 comp rx=0 tx=0).stop 2024-01-15T18:05:04.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7fcc46800640 1 -- 172.21.15.27:0/2064329409 reap_dead start 2024-01-15T18:05:04.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7f78f2400640 1 -- 172.21.15.27:0/2375858441 shutdown_connections 2024-01-15T18:05:04.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7f78f2400640 1 -- 172.21.15.27:0/2375858441 >> 172.21.15.27:0/2375858441 conn(0x7f78e4058ed0 msgr2=0x7f78e404b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:04.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7fcc37400640 1 -- 172.21.15.27:0/2064329409 shutdown_connections 2024-01-15T18:05:04.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7fcc37400640 1 -- 172.21.15.27:0/2064329409 >> 172.21.15.27:0/2064329409 conn(0x7fcc480b0850 msgr2=0x7fcc480aff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:04.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7f78f2400640 1 -- 172.21.15.27:0/2375858441 shutdown_connections 2024-01-15T18:05:04.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7f78f2400640 1 -- 172.21.15.27:0/2375858441 wait complete. 2024-01-15T18:05:04.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7f4295000640 1 -- 172.21.15.27:0/3841749268 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4268066900 msgr2=0x7f4268068dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7f4295000640 1 --2- 172.21.15.27:0/3841749268 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4268066900 0x7f4268068dc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4278004710 tx=0x7f4278009060 comp rx=0 tx=0).stop 2024-01-15T18:05:04.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7f4295000640 1 -- 172.21.15.27:0/3841749268 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4290157890 msgr2=0x7f4290157d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7f4295000640 1 --2- 172.21.15.27:0/3841749268 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4290157890 0x7f4290157d10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f428000e7c0 tx=0x7f428000ec80 comp rx=0 tx=0).stop 2024-01-15T18:05:04.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.764+0000 7f428e800640 1 -- 172.21.15.27:0/3841749268 reap_dead start 2024-01-15T18:05:04.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.768+0000 7fcc37400640 1 -- 172.21.15.27:0/2064329409 shutdown_connections 2024-01-15T18:05:04.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.768+0000 7fcc37400640 1 -- 172.21.15.27:0/2064329409 wait complete. 2024-01-15T18:05:04.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.768+0000 7f4295000640 1 -- 172.21.15.27:0/3841749268 shutdown_connections 2024-01-15T18:05:04.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.768+0000 7f4295000640 1 -- 172.21.15.27:0/3841749268 >> 172.21.15.27:0/3841749268 conn(0x7f42900b0850 msgr2=0x7f42900b31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:04.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.768+0000 7f4295000640 1 -- 172.21.15.27:0/3841749268 shutdown_connections 2024-01-15T18:05:04.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.768+0000 7f4295000640 1 -- 172.21.15.27:0/3841749268 wait complete. 2024-01-15T18:05:04.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.772+0000 7fa8caa00640 1 -- 172.21.15.27:0/1474593609 --> [v2:172.21.15.93:6805/838236833,v1:172.21.15.93:6809/838236833] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa8d415d8c0 con 0x7fa8ac06eb60 2024-01-15T18:05:04.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.776+0000 7fa8d1600640 1 -- 172.21.15.27:0/1474593609 <== osd.6 v2:172.21.15.93:6805/838236833 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa8d415d8c0 con 0x7fa8ac06eb60 2024-01-15T18:05:04.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.776+0000 7f3f17200640 1 -- 172.21.15.27:0/810295683 --> [v2:172.21.15.93:6803/1254182834,v1:172.21.15.93:6806/1254182834] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3f180b1370 con 0x7f3ef006ec10 2024-01-15T18:05:04.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.776+0000 7f396f200640 1 -- 172.21.15.27:0/1445334659 --> [v2:172.21.15.93:6824/978428062,v1:172.21.15.93:6825/978428062] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f39700cac60 con 0x7f394806ebe0 2024-01-15T18:05:04.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7fa8caa00640 1 -- 172.21.15.27:0/1474593609 >> [v2:172.21.15.93:6805/838236833,v1:172.21.15.93:6809/838236833] conn(0x7fa8ac06eb60 msgr2=0x7fa8ac070f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7fa8caa00640 1 --2- 172.21.15.27:0/1474593609 >> [v2:172.21.15.93:6805/838236833,v1:172.21.15.93:6809/838236833] conn(0x7fa8ac06eb60 0x7fa8ac070f80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7f3f0e000640 1 -- 172.21.15.27:0/810295683 <== osd.4 v2:172.21.15.93:6803/1254182834 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3f10009d50 con 0x7f3ef006ec10 2024-01-15T18:05:04.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7f3966000640 1 -- 172.21.15.27:0/1445334659 <== osd.7 v2:172.21.15.93:6824/978428062 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f39700cac60 con 0x7f394806ebe0 2024-01-15T18:05:04.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7f0a94c00640 1 -- 172.21.15.27:0/3222726296 --> [v2:172.21.15.27:6800/3601044148,v1:172.21.15.27:6801/3601044148] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0a9804fe10 con 0x7f0a7806eb20 2024-01-15T18:05:04.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7fa8caa00640 1 -- 172.21.15.27:0/1474593609 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa8ac066940 msgr2=0x7fa8ac068e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7fa8caa00640 1 --2- 172.21.15.27:0/1474593609 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa8ac066940 0x7fa8ac068e00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc0af000 tx=0x7fa8cc0a8070 comp rx=0 tx=0).stop 2024-01-15T18:05:04.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7fa8caa00640 1 -- 172.21.15.27:0/1474593609 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8d415cd90 msgr2=0x7fa8d4159bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7fa8caa00640 1 --2- 172.21.15.27:0/1474593609 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8d415cd90 0x7fa8d4159bb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc00c410 tx=0x7fa8bc00c8d0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7fa8da600640 1 -- 172.21.15.27:0/1474593609 reap_dead start 2024-01-15T18:05:04.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7fa8caa00640 1 -- 172.21.15.27:0/1474593609 shutdown_connections 2024-01-15T18:05:04.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7fa8caa00640 1 -- 172.21.15.27:0/1474593609 >> 172.21.15.27:0/1474593609 conn(0x7fa8d40b0850 msgr2=0x7fa8d40aea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:04.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7f395f400640 1 -- 172.21.15.27:0/1445334659 >> [v2:172.21.15.93:6824/978428062,v1:172.21.15.93:6825/978428062] conn(0x7f394806ebe0 msgr2=0x7f3948071000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7f395f400640 1 --2- 172.21.15.27:0/1445334659 >> [v2:172.21.15.93:6824/978428062,v1:172.21.15.93:6825/978428062] conn(0x7f394806ebe0 0x7f3948071000 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7f0a97400640 1 -- 172.21.15.27:0/3222726296 <== osd.1 v2:172.21.15.27:6800/3601044148 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0a9804fe10 con 0x7f0a7806eb20 2024-01-15T18:05:04.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7f3f17200640 1 -- 172.21.15.27:0/810295683 >> [v2:172.21.15.93:6803/1254182834,v1:172.21.15.93:6806/1254182834] conn(0x7f3ef006ec10 msgr2=0x7f3ef0071030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7f3f17200640 1 --2- 172.21.15.27:0/810295683 >> [v2:172.21.15.93:6803/1254182834,v1:172.21.15.93:6806/1254182834] conn(0x7f3ef006ec10 0x7f3ef0071030 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7fa8caa00640 1 -- 172.21.15.27:0/1474593609 shutdown_connections 2024-01-15T18:05:04.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7fa8caa00640 1 -- 172.21.15.27:0/1474593609 wait complete. 2024-01-15T18:05:04.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7f395f400640 1 -- 172.21.15.27:0/1445334659 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f39480669c0 msgr2=0x7f3948068e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7f395f400640 1 --2- 172.21.15.27:0/1445334659 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f39480669c0 0x7f3948068e80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3958000fe0 tx=0x7f39580013c0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7f395f400640 1 -- 172.21.15.27:0/1445334659 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f397009fef0 msgr2=0x7f39700a0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7f395f400640 1 --2- 172.21.15.27:0/1445334659 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f397009fef0 0x7f39700a0370 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f396801c710 tx=0x7f3950005800 comp rx=0 tx=0).stop 2024-01-15T18:05:04.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7f396e800640 1 -- 172.21.15.27:0/1445334659 reap_dead start 2024-01-15T18:05:04.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7f3f17200640 1 -- 172.21.15.27:0/810295683 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3ef00669f0 msgr2=0x7f3ef0068eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7f3f17200640 1 --2- 172.21.15.27:0/810295683 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3ef00669f0 0x7f3ef0068eb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3f18156900 tx=0x7f3f0800c040 comp rx=0 tx=0).stop 2024-01-15T18:05:04.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7f3f17200640 1 -- 172.21.15.27:0/810295683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f181f7bf0 msgr2=0x7f3f1814cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7f3f17200640 1 --2- 172.21.15.27:0/810295683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f181f7bf0 0x7f3f1814cae0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3f000ac000 tx=0x7f3ef8007a10 comp rx=0 tx=0).stop 2024-01-15T18:05:04.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7f3f16800640 1 -- 172.21.15.27:0/810295683 reap_dead start 2024-01-15T18:05:04.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7f395f400640 1 -- 172.21.15.27:0/1445334659 shutdown_connections 2024-01-15T18:05:04.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7f395f400640 1 -- 172.21.15.27:0/1445334659 >> 172.21.15.27:0/1445334659 conn(0x7f397009c410 msgr2=0x7f397015f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:04.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7f3f17200640 1 -- 172.21.15.27:0/810295683 shutdown_connections 2024-01-15T18:05:04.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.780+0000 7f3f17200640 1 -- 172.21.15.27:0/810295683 >> 172.21.15.27:0/810295683 conn(0x7f3f180b0850 msgr2=0x7f3f180cc680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:04.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.784+0000 7f0a94c00640 1 -- 172.21.15.27:0/3222726296 >> [v2:172.21.15.27:6800/3601044148,v1:172.21.15.27:6801/3601044148] conn(0x7f0a7806eb20 msgr2=0x7f0a78070f40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.784+0000 7f0a94c00640 1 --2- 172.21.15.27:0/3222726296 >> [v2:172.21.15.27:6800/3601044148,v1:172.21.15.27:6801/3601044148] conn(0x7f0a7806eb20 0x7f0a78070f40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.784+0000 7f3f17200640 1 -- 172.21.15.27:0/810295683 shutdown_connections 2024-01-15T18:05:04.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.784+0000 7f3f17200640 1 -- 172.21.15.27:0/810295683 wait complete. 2024-01-15T18:05:04.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.784+0000 7f395f400640 1 -- 172.21.15.27:0/1445334659 shutdown_connections 2024-01-15T18:05:04.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.784+0000 7f395f400640 1 -- 172.21.15.27:0/1445334659 wait complete. 2024-01-15T18:05:04.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.784+0000 7f0a94c00640 1 -- 172.21.15.27:0/3222726296 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0a78066980 msgr2=0x7f0a78068e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.784+0000 7f0a94c00640 1 --2- 172.21.15.27:0/3222726296 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0a78066980 0x7f0a78068e40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0aa002cef0 tx=0x7f0aa008c040 comp rx=0 tx=0).stop 2024-01-15T18:05:04.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.784+0000 7f0a94c00640 1 -- 172.21.15.27:0/3222726296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a980e7490 msgr2=0x7f0a9804d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.784+0000 7f0a94c00640 1 --2- 172.21.15.27:0/3222726296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a980e7490 0x7f0a9804d830 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0a880ae040 tx=0x7f0a880191e0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.784+0000 7f0a9fe00640 1 -- 172.21.15.27:0/3222726296 reap_dead start 2024-01-15T18:05:04.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.784+0000 7f0a94c00640 1 -- 172.21.15.27:0/3222726296 shutdown_connections 2024-01-15T18:05:04.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.784+0000 7f0a94c00640 1 -- 172.21.15.27:0/3222726296 >> 172.21.15.27:0/3222726296 conn(0x7f0a98055330 msgr2=0x7f0a98033f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:04.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.784+0000 7f0a94c00640 1 -- 172.21.15.27:0/3222726296 shutdown_connections 2024-01-15T18:05:04.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.784+0000 7f0a94c00640 1 -- 172.21.15.27:0/3222726296 wait complete. 2024-01-15T18:05:04.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.784+0000 7f060f200640 1 -- 172.21.15.27:0/633850887 --> [v2:172.21.15.93:6800/2348354791,v1:172.21.15.93:6801/2348354791] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f06081654e0 con 0x7f05e006eb90 2024-01-15T18:05:04.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.788+0000 7f05ffe00640 1 -- 172.21.15.27:0/633850887 <== osd.5 v2:172.21.15.93:6800/2348354791 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f06081654e0 con 0x7f05e006eb90 2024-01-15T18:05:04.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.788+0000 7f060f200640 1 -- 172.21.15.27:0/633850887 >> [v2:172.21.15.93:6800/2348354791,v1:172.21.15.93:6801/2348354791] conn(0x7f05e006eb90 msgr2=0x7f05e0070fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.788+0000 7f060f200640 1 --2- 172.21.15.27:0/633850887 >> [v2:172.21.15.93:6800/2348354791,v1:172.21.15.93:6801/2348354791] conn(0x7f05e006eb90 0x7f05e0070fb0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.788+0000 7f060f200640 1 -- 172.21.15.27:0/633850887 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05e0066970 msgr2=0x7f05e0068e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.788+0000 7f060f200640 1 --2- 172.21.15.27:0/633850887 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05e0066970 0x7f05e0068e30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f05f80090b0 tx=0x7f05f80a6000 comp rx=0 tx=0).stop 2024-01-15T18:05:04.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.788+0000 7f060f200640 1 -- 172.21.15.27:0/633850887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06080b3bb0 msgr2=0x7f0608158e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:04.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.788+0000 7f060f200640 1 --2- 172.21.15.27:0/633850887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06080b3bb0 0x7f0608158e50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f05f000b3f0 tx=0x7f05f000b8b0 comp rx=0 tx=0).stop 2024-01-15T18:05:04.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.788+0000 7f060d200640 1 -- 172.21.15.27:0/633850887 reap_dead start 2024-01-15T18:05:04.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.792+0000 7f060f200640 1 -- 172.21.15.27:0/633850887 shutdown_connections 2024-01-15T18:05:04.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.792+0000 7f060f200640 1 -- 172.21.15.27:0/633850887 >> 172.21.15.27:0/633850887 conn(0x7f06080b0d90 msgr2=0x7f06080aed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:04.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.792+0000 7f060f200640 1 -- 172.21.15.27:0/633850887 shutdown_connections 2024-01-15T18:05:04.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:04.792+0000 7f060f200640 1 -- 172.21.15.27:0/633850887 wait complete. 2024-01-15T18:05:04.811 INFO:teuthology.orchestra.run.smithi027.stdout:64424509445 2024-01-15T18:05:04.811 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-15T18:05:04.812 INFO:teuthology.orchestra.run.smithi027.stdout:64424509445 2024-01-15T18:05:04.812 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-15T18:05:04.819 INFO:teuthology.orchestra.run.smithi027.stdout:60129542151 2024-01-15T18:05:04.820 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-15T18:05:04.825 INFO:teuthology.orchestra.run.smithi027.stdout:64424509446 2024-01-15T18:05:04.825 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-15T18:05:04.831 INFO:teuthology.orchestra.run.smithi027.stdout:64424509447 2024-01-15T18:05:04.831 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-15T18:05:04.831 INFO:teuthology.orchestra.run.smithi027.stdout:64424509446 2024-01-15T18:05:04.832 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-15T18:05:04.834 INFO:teuthology.orchestra.run.smithi027.stdout:64424509446 2024-01-15T18:05:04.834 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-15T18:05:04.841 INFO:teuthology.orchestra.run.smithi027.stdout:60129542151 2024-01-15T18:05:04.841 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-15T18:05:05.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.044+0000 7fa2e5200640 1 Processor -- start 2024-01-15T18:05:05.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.044+0000 7fa2e5200640 1 -- start start 2024-01-15T18:05:05.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.048+0000 7fa2e5200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa2d81048f0 0x7fa2d8007630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.048+0000 7fa2e5200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa2d8006090 0x7fa2d8104cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.048+0000 7fa2e5200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2d80071a0 con 0x7fa2d804f560 2024-01-15T18:05:05.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.048+0000 7fa2e5200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2d80fed40 con 0x7fa2d81048f0 2024-01-15T18:05:05.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.048+0000 7fa2e5200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2d8050250 con 0x7fa2d8006090 2024-01-15T18:05:05.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.048+0000 7fa2dd400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa2d8006090 0x7fa2d8104cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.048+0000 7fa2df200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa2d81048f0 0x7fa2d8007630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.048+0000 7fa2dd400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa2d8006090 0x7fa2d8104cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:51590/0 (socket says 172.21.15.27:51590) 2024-01-15T18:05:05.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.048+0000 7fa2dd400640 1 -- 172.21.15.27:0/2925088019 learned_addr learned my addr 172.21.15.27:0/2925088019 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:05.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.048+0000 7fa2dd400640 1 -- 172.21.15.27:0/2925088019 >> v2:172.21.15.27:3300/0 conn(0x7fa2d81048f0 msgr2=0x7fa2d8007630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:05.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.048+0000 7fa2dd400640 1 --2- 172.21.15.27:0/2925088019 >> v2:172.21.15.27:3300/0 conn(0x7fa2d81048f0 0x7fa2d8007630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.048+0000 7fa2dd400640 1 -- 172.21.15.27:0/2925088019 >> v1:172.21.15.27:6789/0 conn(0x7fa2d804f560 legacy=0x7fa2d804fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.048+0000 7fa2dd400640 1 -- 172.21.15.27:0/2925088019 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d8105210 con 0x7fa2d8006090 2024-01-15T18:05:05.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.052+0000 7fa2df200640 1 --2- 172.21.15.27:0/2925088019 >> v2:172.21.15.27:3300/0 conn(0x7fa2d81048f0 0x7fa2d8007630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:05.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.052+0000 7fb55b200640 1 Processor -- start 2024-01-15T18:05:05.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.052+0000 7fb55b200640 1 -- start start 2024-01-15T18:05:05.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.056+0000 7fa2dd400640 1 --2- 172.21.15.27:0/2925088019 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa2d8006090 0x7fa2d8104cd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa2d803a9c0 tx=0x7fa2d0021400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33e8b609966c33ca server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.056+0000 7fa2dca00640 1 -- 172.21.15.27:0/2925088019 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d00b1600 con 0x7fa2d8006090 2024-01-15T18:05:05.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.056+0000 7fa2dca00640 1 -- 172.21.15.27:0/2925088019 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d00ad050 con 0x7fa2d8006090 2024-01-15T18:05:05.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.056+0000 7fb55b200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb5540b2d90 0x7fb5540b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.056+0000 7fb55b200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb554157510 0x7fb55415a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.056+0000 7fa2dca00640 1 -- 172.21.15.27:0/2925088019 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d00b1980 con 0x7fa2d8006090 2024-01-15T18:05:05.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.056+0000 7fb55b200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5541579c0 con 0x7fb554157130 2024-01-15T18:05:05.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.056+0000 7fb55b200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb55415d520 con 0x7fb554157510 2024-01-15T18:05:05.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.056+0000 7fb55b200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb55415aeb0 con 0x7fb5540b2d90 2024-01-15T18:05:05.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.056+0000 7fb559400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb5540b2d90 0x7fb5540b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.056+0000 7fb559400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb5540b2d90 0x7fb5540b1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:42648/0 (socket says 172.21.15.27:42648) 2024-01-15T18:05:05.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.056+0000 7fa2e5200640 1 -- 172.21.15.27:0/2925088019 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa2d8006090 msgr2=0x7fa2d8104cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.056+0000 7fa2e5200640 1 --2- 172.21.15.27:0/2925088019 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa2d8006090 0x7fa2d8104cd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa2d803a9c0 tx=0x7fa2d0021400 comp rx=0 tx=0).stop 2024-01-15T18:05:05.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.056+0000 7fb559400640 1 -- 172.21.15.27:0/1904957227 learned_addr learned my addr 172.21.15.27:0/1904957227 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:05.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.056+0000 7fb559e00640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7fb554157130 0x7fb5540b2680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:57042/0 (socket says 172.21.15.27:57042) 2024-01-15T18:05:05.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.056+0000 7fb559400640 1 -- 172.21.15.27:0/1904957227 >> v2:172.21.15.27:3300/0 conn(0x7fb554157510 msgr2=0x7fb55415a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:05.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.056+0000 7fb559400640 1 --2- 172.21.15.27:0/1904957227 >> v2:172.21.15.27:3300/0 conn(0x7fb554157510 0x7fb55415a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.056+0000 7fb559400640 1 -- 172.21.15.27:0/1904957227 >> v1:172.21.15.93:6789/0 conn(0x7fb554157130 legacy=0x7fb5540b2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.056+0000 7fb559400640 1 -- 172.21.15.27:0/1904957227 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5540b21f0 con 0x7fb5540b2d90 2024-01-15T18:05:05.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.060+0000 7fa2e5200640 1 -- 172.21.15.27:0/2925088019 shutdown_connections 2024-01-15T18:05:05.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.060+0000 7fa2e5200640 1 --2- 172.21.15.27:0/2925088019 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa2d8006090 0x7fa2d8104cd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.060+0000 7fa2e5200640 1 --2- 172.21.15.27:0/2925088019 >> v2:172.21.15.27:3300/0 conn(0x7fa2d81048f0 0x7fa2d8007630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.060+0000 7fa2e5200640 1 -- 172.21.15.27:0/2925088019 >> 172.21.15.27:0/2925088019 conn(0x7fa2d8054c70 msgr2=0x7fa2d8055080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:05.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.060+0000 7fa2e5200640 1 -- 172.21.15.27:0/2925088019 shutdown_connections 2024-01-15T18:05:05.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.060+0000 7fa2e5200640 1 -- 172.21.15.27:0/2925088019 wait complete. 2024-01-15T18:05:05.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.060+0000 7f5c7ae00640 1 Processor -- start 2024-01-15T18:05:05.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.064+0000 7f5c7ae00640 1 -- start start 2024-01-15T18:05:05.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.064+0000 7fb559400640 1 --2- 172.21.15.27:0/1904957227 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb5540b2d90 0x7fb5540b1b50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb544006fd0 tx=0x7fb544020d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ba54409af973750 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.064+0000 7f5c7ae00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c740701c0 0x7f5c74070600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.064+0000 7f5c7ae00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c740b1b50 0x7f5c740b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.064+0000 7f5c7ae00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c74070b40 con 0x7f5c7414afa0 2024-01-15T18:05:05.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.064+0000 7f5c7ae00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c7406d940 con 0x7f5c740b1b50 2024-01-15T18:05:05.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.064+0000 7f5c7ae00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7414b380 con 0x7f5c740701c0 2024-01-15T18:05:05.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.064+0000 7fa2e5200640 1 Processor -- start 2024-01-15T18:05:05.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.064+0000 7fa2e5200640 1 -- start start 2024-01-15T18:05:05.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.064+0000 7f27c9000640 1 Processor -- start 2024-01-15T18:05:05.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.064+0000 7f5c73400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c740701c0 0x7f5c74070600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.064+0000 7f5c73e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5c7414afa0 0x7f5c7406c200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42326/0 (socket says 172.21.15.27:42326) 2024-01-15T18:05:05.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.064+0000 7f5c73e00640 1 -- 172.21.15.27:0/1645721016 learned_addr learned my addr 172.21.15.27:0/1645721016 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:05.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.064+0000 7f27c9000640 1 -- start start 2024-01-15T18:05:05.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fb558a00640 1 -- 172.21.15.27:0/1904957227 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5440a9d90 con 0x7fb5540b2d90 2024-01-15T18:05:05.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fb558a00640 1 -- 172.21.15.27:0/1904957227 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5440ad070 con 0x7fb5540b2d90 2024-01-15T18:05:05.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fb558a00640 1 -- 172.21.15.27:0/1904957227 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb544004230 con 0x7fb5540b2d90 2024-01-15T18:05:05.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7f5c72a00640 1 -- 172.21.15.27:0/1645721016 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2038045431 0 0) 0x7f5c74070b40 con 0x7f5c7414afa0 2024-01-15T18:05:05.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7f5c72a00640 1 -- 172.21.15.27:0/1645721016 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c54003530 con 0x7f5c7414afa0 2024-01-15T18:05:05.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7f5c78e00640 1 --2- 172.21.15.27:0/1645721016 >> v2:172.21.15.27:3300/0 conn(0x7f5c740b1b50 0x7f5c740b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7f5c73400640 1 -- 172.21.15.27:0/1645721016 >> v2:172.21.15.27:3300/0 conn(0x7f5c740b1b50 msgr2=0x7f5c740b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:05.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7f5c72a00640 1 -- 172.21.15.27:0/1645721016 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3151513048 0 0) 0x7f5c54003530 con 0x7f5c7414afa0 2024-01-15T18:05:05.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7f27c9000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27bc033e10 0x7f27bc0eb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7f27c9000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f27bc005ec0 0x7f27bc006310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fa2e5200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa2d8006090 0x7fa2d80ec9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fa2e5200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2d804f560 0x7fa2d80ef220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fa2e5200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa2d81048f0 0x7fa2d80ef760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fa2e5200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d80f00d0 con 0x7fa2d804f560 2024-01-15T18:05:05.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fa2e5200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d80ecf30 con 0x7fa2d81048f0 2024-01-15T18:05:05.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fa2e5200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d804d9d0 con 0x7fa2d8006090 2024-01-15T18:05:05.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fa2df200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa2d8006090 0x7fa2d80ec9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fa2df200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa2d8006090 0x7fa2d80ec9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:42682/0 (socket says 172.21.15.27:42682) 2024-01-15T18:05:05.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fa2df200640 1 -- 172.21.15.27:0/3150040607 learned_addr learned my addr 172.21.15.27:0/3150040607 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:05.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fa2dd400640 1 --2- 172.21.15.27:0/3150040607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2d804f560 0x7fa2d80ef220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fa2dde00640 1 --2- 172.21.15.27:0/3150040607 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa2d81048f0 0x7fa2d80ef760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7f27c9000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc037b90 con 0x7f27bc047bf0 2024-01-15T18:05:05.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7f27c9000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27bc0501e0 con 0x7f27bc005ec0 2024-01-15T18:05:05.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7f27c9000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc04f060 con 0x7f27bc033e10 2024-01-15T18:05:05.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7f27c1e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27bc033e10 0x7f27bc0eb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7f27c1e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27bc033e10 0x7f27bc0eb150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:42680/0 (socket says 172.21.15.27:42680) 2024-01-15T18:05:05.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7f27c1e00640 1 -- 172.21.15.27:0/560594561 learned_addr learned my addr 172.21.15.27:0/560594561 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:05.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fa2df200640 1 -- 172.21.15.27:0/3150040607 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa2d81048f0 msgr2=0x7fa2d80ef760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fa2df200640 1 --2- 172.21.15.27:0/3150040607 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa2d81048f0 0x7fa2d80ef760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fa2df200640 1 -- 172.21.15.27:0/3150040607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2d804f560 msgr2=0x7fa2d80ef220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fa2df200640 1 --2- 172.21.15.27:0/3150040607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2d804f560 0x7fa2d80ef220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fa2df200640 1 -- 172.21.15.27:0/3150040607 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d000b770 con 0x7fa2d8006090 2024-01-15T18:05:05.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fb55b200640 1 -- 172.21.15.27:0/1904957227 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb5540b2d90 msgr2=0x7fb5540b1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fb55b200640 1 --2- 172.21.15.27:0/1904957227 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb5540b2d90 0x7fb5540b1b50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb544006fd0 tx=0x7fb544020d20 comp rx=0 tx=0).stop 2024-01-15T18:05:05.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fb55b200640 1 -- 172.21.15.27:0/1904957227 shutdown_connections 2024-01-15T18:05:05.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fb55b200640 1 --2- 172.21.15.27:0/1904957227 >> v2:172.21.15.27:3300/0 conn(0x7fb554157510 0x7fb55415a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fb55b200640 1 --2- 172.21.15.27:0/1904957227 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb5540b2d90 0x7fb5540b1b50 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb544006fd0 tx=0x7fb544020d20 comp rx=0 tx=0).stop 2024-01-15T18:05:05.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fb55b200640 1 -- 172.21.15.27:0/1904957227 >> 172.21.15.27:0/1904957227 conn(0x7fb55406aaf0 msgr2=0x7fb5540b0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:05.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7f5c73400640 1 --2- 172.21.15.27:0/1645721016 >> v2:172.21.15.27:3300/0 conn(0x7f5c740b1b50 0x7f5c740b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fa2df200640 1 --2- 172.21.15.27:0/3150040607 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa2d8006090 0x7fa2d80ec9f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa2e004f6a0 tx=0x7fa2e0074ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7f5c73400640 1 -- 172.21.15.27:0/1645721016 >> v1:172.21.15.27:6789/0 conn(0x7f5c7414afa0 legacy=0x7f5c7406c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7f40dea00640 1 Processor -- start 2024-01-15T18:05:05.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7f5c73400640 1 -- 172.21.15.27:0/1645721016 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7406ccc0 con 0x7f5c740701c0 2024-01-15T18:05:05.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7f5c78e00640 1 --2- 172.21.15.27:0/1645721016 >> v2:172.21.15.27:3300/0 conn(0x7f5c740b1b50 0x7f5c740b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:05.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7f27c2800640 1 --2- 172.21.15.27:0/560594561 >> v2:172.21.15.27:3300/0 conn(0x7f27bc005ec0 0x7f27bc006310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.072+0000 7f27c1e00640 1 -- 172.21.15.27:0/560594561 >> v2:172.21.15.27:3300/0 conn(0x7f27bc005ec0 msgr2=0x7f27bc006310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:05.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.072+0000 7f27c1e00640 1 --2- 172.21.15.27:0/560594561 >> v2:172.21.15.27:3300/0 conn(0x7f27bc005ec0 0x7f27bc006310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.072+0000 7f27c1e00640 1 -- 172.21.15.27:0/560594561 >> v1:172.21.15.93:6789/0 conn(0x7f27bc047bf0 legacy=0x7f27bc0eb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.072+0000 7f27c1e00640 1 -- 172.21.15.27:0/560594561 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc04fd50 con 0x7f27bc033e10 2024-01-15T18:05:05.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.072+0000 7f27c0a00640 1 -- 172.21.15.27:0/560594561 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610364678 0 0) 0x7f27bc037b90 con 0x7f27bc047bf0 2024-01-15T18:05:05.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.072+0000 7f40dea00640 1 -- start start 2024-01-15T18:05:05.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.068+0000 7fa2d6000640 1 -- 172.21.15.27:0/3150040607 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e0082470 con 0x7fa2d8006090 2024-01-15T18:05:05.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.072+0000 7fb55b200640 1 -- 172.21.15.27:0/1904957227 shutdown_connections 2024-01-15T18:05:05.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.072+0000 7fb55b200640 1 -- 172.21.15.27:0/1904957227 wait complete. 2024-01-15T18:05:05.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.072+0000 7fa2d6000640 1 -- 172.21.15.27:0/3150040607 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e0071050 con 0x7fa2d8006090 2024-01-15T18:05:05.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.072+0000 7fa2e5200640 1 -- 172.21.15.27:0/3150040607 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2d804cdc0 con 0x7fa2d8006090 2024-01-15T18:05:05.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.072+0000 7fa2e5200640 1 -- 172.21.15.27:0/3150040607 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2d804d380 con 0x7fa2d8006090 2024-01-15T18:05:05.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.072+0000 7fa2d6000640 1 -- 172.21.15.27:0/3150040607 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e0082810 con 0x7fa2d8006090 2024-01-15T18:05:05.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.072+0000 7fa2e5200640 1 -- 172.21.15.27:0/3150040607 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2b0005260 con 0x7fa2d8006090 2024-01-15T18:05:05.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.072+0000 7fb55b200640 1 Processor -- start 2024-01-15T18:05:05.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.072+0000 7fa2d6000640 1 -- 172.21.15.27:0/3150040607 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fa2e007f070 con 0x7fa2d8006090 2024-01-15T18:05:05.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.072+0000 7fb55b200640 1 -- start start 2024-01-15T18:05:05.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.072+0000 7f40dea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f40d80b1b50 0x7f40d80cd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.072+0000 7f40dea00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f40d80b2000 0x7f40d80cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.072+0000 7f40dea00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40d806c450 con 0x7f40d815a0c0 2024-01-15T18:05:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.072+0000 7f40dea00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40d80712a0 con 0x7f40d80b1b50 2024-01-15T18:05:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.072+0000 7f40dea00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40d80cc640 con 0x7f40d80b2000 2024-01-15T18:05:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7f40de000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f40d80b2000 0x7f40d80cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7f40de000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f40d80b2000 0x7f40d80cd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:51608/0 (socket says 172.21.15.27:51608) 2024-01-15T18:05:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7f40de000640 1 -- 172.21.15.27:0/2039514872 learned_addr learned my addr 172.21.15.27:0/2039514872 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7fa2d6000640 1 --2- 172.21.15.27:0/3150040607 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa2b80669c0 0x7fa2b8068e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7fa2d6000640 1 -- 172.21.15.27:0/3150040607 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2e00ed0b0 con 0x7fa2d8006090 2024-01-15T18:05:05.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7fa2dd400640 1 --2- 172.21.15.27:0/3150040607 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa2b80669c0 0x7fa2b8068e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7f5c73400640 1 --2- 172.21.15.27:0/1645721016 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c740701c0 0x7f5c74070600 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5c640097c0 tx=0x7f5c6401c370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ef1f8aba432fba6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7f40de000640 1 -- 172.21.15.27:0/2039514872 >> v2:172.21.15.27:3300/0 conn(0x7f40d80b1b50 msgr2=0x7f40d80cd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:05.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7f40de000640 1 --2- 172.21.15.27:0/2039514872 >> v2:172.21.15.27:3300/0 conn(0x7f40d80b1b50 0x7f40d80cd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7f40de000640 1 -- 172.21.15.27:0/2039514872 >> v1:172.21.15.27:6789/0 conn(0x7f40d815a0c0 legacy=0x7f40d8070b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:05.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7f40de000640 1 -- 172.21.15.27:0/2039514872 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d80ce4b0 con 0x7f40d80b2000 2024-01-15T18:05:05.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7fb55b200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb554157130 0x7fb5541482a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7fa2dd400640 1 --2- 172.21.15.27:0/3150040607 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa2b80669c0 0x7fa2b8068e80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa2d00b3000 tx=0x7fa2d00ac000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7fb55b200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb554157510 0x7fb554152550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7f27c1e00640 1 --2- 172.21.15.27:0/560594561 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27bc033e10 0x7f27bc0eb150 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f27b4009870 tx=0x7f27b401ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4db39fe5611c93 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7fb55b200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb554152a90 0x7fb5540cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7fb55b200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb554149690 con 0x7fb554157510 2024-01-15T18:05:05.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7fb55b200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb554148a10 con 0x7fb554157130 2024-01-15T18:05:05.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7fb55b200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb554148cf0 con 0x7fb554152a90 2024-01-15T18:05:05.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7fb559e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb554157130 0x7fb5541482a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7fb55a800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb554152a90 0x7fb5540cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7fb55a800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb554152a90 0x7fb5540cbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:42684/0 (socket says 172.21.15.27:42684) 2024-01-15T18:05:05.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7fb55a800640 1 -- 172.21.15.27:0/1592273490 learned_addr learned my addr 172.21.15.27:0/1592273490 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:05.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7fb559400640 1 --2- 172.21.15.27:0/1592273490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb554157510 0x7fb554152550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7f8f21400640 1 Processor -- start 2024-01-15T18:05:05.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7fb55a800640 1 -- 172.21.15.27:0/1592273490 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb554157130 msgr2=0x7fb5541482a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7fb55a800640 1 --2- 172.21.15.27:0/1592273490 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb554157130 0x7fb5541482a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7fb55a800640 1 -- 172.21.15.27:0/1592273490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb554157510 msgr2=0x7fb554152550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7fb55a800640 1 --2- 172.21.15.27:0/1592273490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb554157510 0x7fb554152550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7fb55a800640 1 -- 172.21.15.27:0/1592273490 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54400b780 con 0x7fb554152a90 2024-01-15T18:05:05.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7fb559e00640 1 --2- 172.21.15.27:0/1592273490 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb554157130 0x7fb5541482a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:05.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7f8f21400640 1 -- start start 2024-01-15T18:05:05.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.076+0000 7fb55a800640 1 --2- 172.21.15.27:0/1592273490 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb554152a90 0x7fb5540cbc20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb54c00b970 tx=0x7fb54c00be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f40de000640 1 --2- 172.21.15.27:0/2039514872 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40d80b2000 0x7f40d80cd620 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f40d8067e70 tx=0x7f40d001ead0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd466e21c0bb11c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f8f21400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f8f1c15e450 0x7f8f1c168660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f8f21400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c070430 con 0x7f8f1c15dfd0 2024-01-15T18:05:05.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f8f21400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c0b2580 con 0x7f8f1c15dbf0 2024-01-15T18:05:05.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f8f21400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f1c06d970 con 0x7f8f1c15e450 2024-01-15T18:05:05.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f40d7e00640 1 -- 172.21.15.27:0/2039514872 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d000b2d0 con 0x7f40d80b2000 2024-01-15T18:05:05.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f40d7e00640 1 -- 172.21.15.27:0/2039514872 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d000b430 con 0x7f40d80b2000 2024-01-15T18:05:05.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f40d7e00640 1 -- 172.21.15.27:0/2039514872 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d00a7bc0 con 0x7f40d80b2000 2024-01-15T18:05:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f8f1b400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8f1c15dfd0 0x7f8f1c164f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42354/0 (socket says 172.21.15.27:42354) 2024-01-15T18:05:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f8f1b400640 1 -- 172.21.15.27:0/1995290076 learned_addr learned my addr 172.21.15.27:0/1995290076 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f8f1be00640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7f8f1c15dbf0 0x7f8f1c0710c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:57046/0 (socket says 172.21.15.27:57046) 2024-01-15T18:05:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f8f20a00640 1 --2- 172.21.15.27:0/1995290076 >> v2:172.21.15.93:3300/0 conn(0x7f8f1c15e450 0x7f8f1c168660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f8f1aa00640 1 -- 172.21.15.27:0/1995290076 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970665703 0 0) 0x7f8f1c070430 con 0x7f8f1c15dfd0 2024-01-15T18:05:05.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f8f1aa00640 1 -- 172.21.15.27:0/1995290076 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8efc003530 con 0x7f8f1c15dfd0 2024-01-15T18:05:05.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f8f1aa00640 1 -- 172.21.15.27:0/1995290076 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154755411 0 0) 0x7f8f1c0b2580 con 0x7f8f1c15dbf0 2024-01-15T18:05:05.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f8f1aa00640 1 -- 172.21.15.27:0/1995290076 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f1c070430 con 0x7f8f1c15dbf0 2024-01-15T18:05:05.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f40dea00640 1 -- 172.21.15.27:0/2039514872 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40d80b2000 msgr2=0x7f40d80cd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f40dea00640 1 --2- 172.21.15.27:0/2039514872 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40d80b2000 0x7f40d80cd620 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f40d8067e70 tx=0x7f40d001ead0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f8f20a00640 1 -- 172.21.15.27:0/1995290076 >> v1:172.21.15.93:6789/0 conn(0x7f8f1c15dbf0 legacy=0x7f8f1c0710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f27c0a00640 1 -- 172.21.15.27:0/560594561 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b40a8a30 con 0x7f27bc033e10 2024-01-15T18:05:05.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f27c0a00640 1 -- 172.21.15.27:0/560594561 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b40a8b90 con 0x7f27bc033e10 2024-01-15T18:05:05.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f40dea00640 1 -- 172.21.15.27:0/2039514872 shutdown_connections 2024-01-15T18:05:05.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f40dea00640 1 --2- 172.21.15.27:0/2039514872 >> v2:172.21.15.27:3300/0 conn(0x7f40d80b1b50 0x7f40d80cd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f40dea00640 1 --2- 172.21.15.27:0/2039514872 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40d80b2000 0x7f40d80cd620 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f40dea00640 1 -- 172.21.15.27:0/2039514872 >> 172.21.15.27:0/2039514872 conn(0x7f40d80b0850 msgr2=0x7f40d80b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:05.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7fa2d6000640 1 -- 172.21.15.27:0/3150040607 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa2e00ba3c0 con 0x7fa2d8006090 2024-01-15T18:05:05.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f27c0a00640 1 -- 172.21.15.27:0/560594561 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b40b0440 con 0x7f27bc033e10 2024-01-15T18:05:05.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f5c72a00640 1 -- 172.21.15.27:0/1645721016 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5c001660 con 0x7f5c740701c0 2024-01-15T18:05:05.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f5c72a00640 1 -- 172.21.15.27:0/1645721016 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c640a7050 con 0x7f5c740701c0 2024-01-15T18:05:05.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f5c72a00640 1 -- 172.21.15.27:0/1645721016 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c640ab620 con 0x7f5c740701c0 2024-01-15T18:05:05.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.084+0000 7f27c9000640 1 -- 172.21.15.27:0/560594561 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27bc033e10 msgr2=0x7f27bc0eb150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.084+0000 7f27c9000640 1 --2- 172.21.15.27:0/560594561 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27bc033e10 0x7f27bc0eb150 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f27b4009870 tx=0x7f27b401ed30 comp rx=0 tx=0).stop 2024-01-15T18:05:05.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.084+0000 7f5c7ae00640 1 -- 172.21.15.27:0/1645721016 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c740701c0 msgr2=0x7f5c74070600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.084+0000 7f5c7ae00640 1 --2- 172.21.15.27:0/1645721016 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c740701c0 0x7f5c74070600 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5c640097c0 tx=0x7f5c6401c370 comp rx=0 tx=0).stop 2024-01-15T18:05:05.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f40dea00640 1 -- 172.21.15.27:0/2039514872 shutdown_connections 2024-01-15T18:05:05.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.084+0000 7f40dea00640 1 -- 172.21.15.27:0/2039514872 wait complete. 2024-01-15T18:05:05.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.084+0000 7f5c7ae00640 1 -- 172.21.15.27:0/1645721016 shutdown_connections 2024-01-15T18:05:05.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.084+0000 7f5c7ae00640 1 --2- 172.21.15.27:0/1645721016 >> v2:172.21.15.27:3300/0 conn(0x7f5c740b1b50 0x7f5c740b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f8f20a00640 1 -- 172.21.15.27:0/1995290076 >> v1:172.21.15.27:6789/0 conn(0x7f8f1c15dfd0 legacy=0x7f8f1c164f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7f8f20a00640 1 -- 172.21.15.27:0/1995290076 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1c0b2860 con 0x7f8f1c15e450 2024-01-15T18:05:05.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7fb552000640 1 -- 172.21.15.27:0/1592273490 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c00cb40 con 0x7fb554152a90 2024-01-15T18:05:05.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7fb552000640 1 -- 172.21.15.27:0/1592273490 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb54c00ccc0 con 0x7fb554152a90 2024-01-15T18:05:05.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.080+0000 7fb552000640 1 -- 172.21.15.27:0/1592273490 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c014680 con 0x7fb554152a90 2024-01-15T18:05:05.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.084+0000 7f5c7ae00640 1 --2- 172.21.15.27:0/1645721016 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c740701c0 0x7f5c74070600 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.084+0000 7f5c7ae00640 1 -- 172.21.15.27:0/1645721016 >> 172.21.15.27:0/1645721016 conn(0x7f5c740b0850 msgr2=0x7f5c740b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:05.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.084+0000 7fb55b200640 1 -- 172.21.15.27:0/1592273490 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb554148fd0 con 0x7fb554152a90 2024-01-15T18:05:05.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.084+0000 7fb55b200640 1 -- 172.21.15.27:0/1592273490 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb55414e850 con 0x7fb554152a90 2024-01-15T18:05:05.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.084+0000 7f27c9000640 1 -- 172.21.15.27:0/560594561 shutdown_connections 2024-01-15T18:05:05.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.084+0000 7f27c9000640 1 --2- 172.21.15.27:0/560594561 >> v2:172.21.15.27:3300/0 conn(0x7f27bc005ec0 0x7f27bc006310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.084+0000 7f27c9000640 1 --2- 172.21.15.27:0/560594561 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27bc033e10 0x7f27bc0eb150 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.084+0000 7f27c9000640 1 -- 172.21.15.27:0/560594561 >> 172.21.15.27:0/560594561 conn(0x7f27bc009f90 msgr2=0x7f27bc00a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:05.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.084+0000 7f5c7ae00640 1 -- 172.21.15.27:0/1645721016 shutdown_connections 2024-01-15T18:05:05.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.084+0000 7f27c9000640 1 -- 172.21.15.27:0/560594561 shutdown_connections 2024-01-15T18:05:05.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.084+0000 7f5c7ae00640 1 -- 172.21.15.27:0/1645721016 wait complete. 2024-01-15T18:05:05.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.084+0000 7fb55b200640 1 -- 172.21.15.27:0/1592273490 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5541497d0 con 0x7fb554152a90 2024-01-15T18:05:05.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.088+0000 7f8f20a00640 1 --2- 172.21.15.27:0/1995290076 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f1c15e450 0x7f8f1c168660 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8f14009d90 tx=0x7f8f1401c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42d095ba8ceb028b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.084+0000 7f27c9000640 1 -- 172.21.15.27:0/560594561 wait complete. 2024-01-15T18:05:05.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.088+0000 7f27c9000640 1 Processor -- start 2024-01-15T18:05:05.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.088+0000 7f5c7ae00640 1 Processor -- start 2024-01-15T18:05:05.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.088+0000 7fb552000640 1 -- 172.21.15.27:0/1592273490 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fb54c01d3b0 con 0x7fb554152a90 2024-01-15T18:05:05.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.088+0000 7f5c7ae00640 1 -- start start 2024-01-15T18:05:05.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.088+0000 7f27c9000640 1 -- start start 2024-01-15T18:05:05.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.088+0000 7f40dea00640 1 Processor -- start 2024-01-15T18:05:05.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.088+0000 7f8f1aa00640 1 -- 172.21.15.27:0/1995290076 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f140a5900 con 0x7f8f1c15e450 2024-01-15T18:05:05.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.088+0000 7f40dea00640 1 -- start start 2024-01-15T18:05:05.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.088+0000 7f8f1aa00640 1 -- 172.21.15.27:0/1995290076 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f140a5a60 con 0x7f8f1c15e450 2024-01-15T18:05:05.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.088+0000 7f8f1aa00640 1 -- 172.21.15.27:0/1995290076 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f140a5df0 con 0x7f8f1c15e450 2024-01-15T18:05:05.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.088+0000 7fb552000640 1 --2- 172.21.15.27:0/1592273490 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb52c066940 0x7fb52c068e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.088+0000 7fb552000640 1 -- 172.21.15.27:0/1592273490 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb54c084240 con 0x7fb554152a90 2024-01-15T18:05:05.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.088+0000 7fb559e00640 1 --2- 172.21.15.27:0/1592273490 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb52c066940 0x7fb52c068e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.088+0000 7f8f21400640 1 -- 172.21.15.27:0/1995290076 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f1c15e450 msgr2=0x7f8f1c168660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.088+0000 7f8f21400640 1 --2- 172.21.15.27:0/1995290076 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f1c15e450 0x7f8f1c168660 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8f14009d90 tx=0x7f8f1401c990 comp rx=0 tx=0).stop 2024-01-15T18:05:05.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.088+0000 7f5c7ae00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5c740b1b50 0x7f5c740ae6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f8f21400640 1 -- 172.21.15.27:0/1995290076 shutdown_connections 2024-01-15T18:05:05.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f8f21400640 1 --2- 172.21.15.27:0/1995290076 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f1c15e450 0x7f8f1c168660 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f8f21400640 1 -- 172.21.15.27:0/1995290076 >> 172.21.15.27:0/1995290076 conn(0x7f8f1c06a6c0 msgr2=0x7f8f1c06aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:05.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f5c7ae00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7414afa0 0x7f5c740aec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f8f21400640 1 -- 172.21.15.27:0/1995290076 shutdown_connections 2024-01-15T18:05:05.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f8f21400640 1 -- 172.21.15.27:0/1995290076 wait complete. 2024-01-15T18:05:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f5c7ae00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c74141d50 0x7f5c740cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f5c7ae00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c74142da0 con 0x7f5c7414afa0 2024-01-15T18:05:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f5c7ae00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c74143080 con 0x7f5c740b1b50 2024-01-15T18:05:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f5c7ae00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c74148850 con 0x7f5c74141d50 2024-01-15T18:05:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f5c73e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5c740b1b50 0x7f5c740ae6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f5c78e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c74141d50 0x7f5c740cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f5c78e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c74141d50 0x7f5c740cccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:42698/0 (socket says 172.21.15.27:42698) 2024-01-15T18:05:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f5c78e00640 1 -- 172.21.15.27:0/2108322595 learned_addr learned my addr 172.21.15.27:0/2108322595 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f5c78e00640 1 -- 172.21.15.27:0/2108322595 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5c740b1b50 msgr2=0x7f5c740ae6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f5c78e00640 1 --2- 172.21.15.27:0/2108322595 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5c740b1b50 0x7f5c740ae6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f5c78e00640 1 -- 172.21.15.27:0/2108322595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7414afa0 msgr2=0x7f5c740aec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f5c78e00640 1 --2- 172.21.15.27:0/2108322595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7414afa0 0x7f5c740aec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f5c78e00640 1 -- 172.21.15.27:0/2108322595 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c64009420 con 0x7f5c74141d50 2024-01-15T18:05:05.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f27c9000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27bc005ec0 0x7f27bc04dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f27c9000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27bc033e10 0x7f27bc04bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f27c9000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27bc047bf0 0x7f27bc17e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f27c9000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc04e2a0 con 0x7f27bc047bf0 2024-01-15T18:05:05.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f27c9000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc04c0b0 con 0x7f27bc005ec0 2024-01-15T18:05:05.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f27c9000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc04c2e0 con 0x7f27bc033e10 2024-01-15T18:05:05.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f40dea00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40d80b1b50 0x7f40d8152690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f40dea00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f40d80b2000 0x7f40d814d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f40dea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40d815a0c0 0x7f40d8070180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f40dea00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d8152270 con 0x7f40d815a0c0 2024-01-15T18:05:05.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f40dea00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d814d600 con 0x7f40d80b1b50 2024-01-15T18:05:05.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f40dea00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40d8153820 con 0x7f40d80b2000 2024-01-15T18:05:05.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f5c78e00640 1 --2- 172.21.15.27:0/2108322595 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c74141d50 0x7f5c740cccc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c00cef0 tx=0x7f5c6c00a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f8f21400640 1 Processor -- start 2024-01-15T18:05:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f27c1400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27bc033e10 0x7f27bc04bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f27c1400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27bc033e10 0x7f27bc04bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:42702/0 (socket says 172.21.15.27:42702) 2024-01-15T18:05:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f27c1400640 1 -- 172.21.15.27:0/2281312730 learned_addr learned my addr 172.21.15.27:0/2281312730 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f27c1e00640 1 --2- 172.21.15.27:0/2281312730 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27bc005ec0 0x7f27bc04dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7fb559e00640 1 --2- 172.21.15.27:0/1592273490 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb52c066940 0x7fb52c068e00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb53c000f70 tx=0x7fb53c007e50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f5c6be00640 1 -- 172.21.15.27:0/2108322595 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c00d840 con 0x7f5c74141d50 2024-01-15T18:05:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f27c2800640 1 --2- 172.21.15.27:0/2281312730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27bc047bf0 0x7f27bc17e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f27c1400640 1 -- 172.21.15.27:0/2281312730 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27bc005ec0 msgr2=0x7f27bc04dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f27c1400640 1 --2- 172.21.15.27:0/2281312730 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27bc005ec0 0x7f27bc04dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f27c1400640 1 -- 172.21.15.27:0/2281312730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27bc047bf0 msgr2=0x7f27bc17e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f27c1400640 1 --2- 172.21.15.27:0/2281312730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27bc047bf0 0x7f27bc17e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f27c1400640 1 -- 172.21.15.27:0/2281312730 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b40094d0 con 0x7f27bc033e10 2024-01-15T18:05:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f27c1400640 1 --2- 172.21.15.27:0/2281312730 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27bc033e10 0x7f27bc04bb70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f27ac001ec0 tx=0x7f27ac00c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f8f21400640 1 -- start start 2024-01-15T18:05:05.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f8f21400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f1c15dbf0 0x7f8f1c14e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f8f21400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f1c15dfd0 0x7f8f1c157530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f8f21400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8f1c15e450 0x7f8f1c157fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f40de000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40d815a0c0 0x7f40d8070180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f40de000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40d815a0c0 0x7f40d8070180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59252/0 (socket says 172.21.15.27:59252) 2024-01-15T18:05:05.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f40de000640 1 -- 172.21.15.27:0/1385217403 learned_addr learned my addr 172.21.15.27:0/1385217403 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:05.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f40dcc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f40d80b2000 0x7f40d814d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f40dd600640 1 --2- 172.21.15.27:0/1385217403 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40d80b1b50 0x7f40d8152690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f5c7ae00640 1 -- 172.21.15.27:0/2108322595 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c7406ca90 con 0x7f5c74141d50 2024-01-15T18:05:05.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f5c7ae00640 1 -- 172.21.15.27:0/2108322595 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c74148e10 con 0x7f5c74141d50 2024-01-15T18:05:05.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f5c6be00640 1 -- 172.21.15.27:0/2108322595 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6c00d9a0 con 0x7f5c74141d50 2024-01-15T18:05:05.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.092+0000 7f5c6be00640 1 -- 172.21.15.27:0/2108322595 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c00dd50 con 0x7f5c74141d50 2024-01-15T18:05:05.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7fb552000640 1 -- 172.21.15.27:0/1592273490 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb5541497d0 con 0x7fb554152a90 2024-01-15T18:05:05.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f27ba000640 1 -- 172.21.15.27:0/2281312730 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ac0054e0 con 0x7f27bc033e10 2024-01-15T18:05:05.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f27ba000640 1 -- 172.21.15.27:0/2281312730 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27ac003600 con 0x7f27bc033e10 2024-01-15T18:05:05.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f27ba000640 1 -- 172.21.15.27:0/2281312730 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ac003990 con 0x7f27bc033e10 2024-01-15T18:05:05.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f8f21400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c1677f0 con 0x7f8f1c15dfd0 2024-01-15T18:05:05.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f8f21400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c167a40 con 0x7f8f1c15dbf0 2024-01-15T18:05:05.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f8f1be00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f1c15dbf0 0x7f8f1c14e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f8f21400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c156a50 con 0x7f8f1c15e450 2024-01-15T18:05:05.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f8f1be00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f1c15dbf0 0x7f8f1c14e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:51638/0 (socket says 172.21.15.27:51638) 2024-01-15T18:05:05.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f8f1be00640 1 -- 172.21.15.27:0/986328679 learned_addr learned my addr 172.21.15.27:0/986328679 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:05.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f8f20a00640 1 --2- 172.21.15.27:0/986328679 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8f1c15e450 0x7f8f1c157fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f8f1b400640 1 --2- 172.21.15.27:0/986328679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f1c15dfd0 0x7f8f1c157530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f8f20a00640 1 -- 172.21.15.27:0/986328679 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f1c15dbf0 msgr2=0x7f8f1c14e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f8f20a00640 1 --2- 172.21.15.27:0/986328679 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f1c15dbf0 0x7f8f1c14e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f8f20a00640 1 -- 172.21.15.27:0/986328679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f1c15dfd0 msgr2=0x7f8f1c157530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f8f20a00640 1 --2- 172.21.15.27:0/986328679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f1c15dfd0 0x7f8f1c157530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f8f20a00640 1 -- 172.21.15.27:0/986328679 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f14009a70 con 0x7f8f1c15e450 2024-01-15T18:05:05.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f40dcc00640 1 -- 172.21.15.27:0/1385217403 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40d80b1b50 msgr2=0x7f40d8152690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f40dcc00640 1 --2- 172.21.15.27:0/1385217403 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40d80b1b50 0x7f40d8152690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f40dcc00640 1 -- 172.21.15.27:0/1385217403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40d815a0c0 msgr2=0x7f40d8070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f40dcc00640 1 --2- 172.21.15.27:0/1385217403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40d815a0c0 0x7f40d8070180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f40dcc00640 1 -- 172.21.15.27:0/1385217403 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d000bd40 con 0x7f40d80b2000 2024-01-15T18:05:05.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f40dcc00640 1 --2- 172.21.15.27:0/1385217403 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f40d80b2000 0x7f40d814d0c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f40c000b3f0 tx=0x7f40c000b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f8f1be00640 1 --2- 172.21.15.27:0/986328679 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f1c15dbf0 0x7f8f1c14e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:05.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f40d5600640 1 -- 172.21.15.27:0/1385217403 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c0011830 con 0x7f40d80b2000 2024-01-15T18:05:05.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f40d5600640 1 -- 172.21.15.27:0/1385217403 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40c0011990 con 0x7f40d80b2000 2024-01-15T18:05:05.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f40d5600640 1 -- 172.21.15.27:0/1385217403 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c0011d20 con 0x7f40d80b2000 2024-01-15T18:05:05.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f8f20a00640 1 --2- 172.21.15.27:0/986328679 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8f1c15e450 0x7f8f1c157fe0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8f140ac000 tx=0x7f8f14019730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f8f13e00640 1 -- 172.21.15.27:0/986328679 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f140bd070 con 0x7f8f1c15e450 2024-01-15T18:05:05.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f8f13e00640 1 -- 172.21.15.27:0/986328679 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f140ad440 con 0x7f8f1c15e450 2024-01-15T18:05:05.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f8f21400640 1 -- 172.21.15.27:0/986328679 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f1c156d30 con 0x7f8f1c15e450 2024-01-15T18:05:05.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f8f21400640 1 -- 172.21.15.27:0/986328679 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f1c14ece0 con 0x7f8f1c15e450 2024-01-15T18:05:05.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.100+0000 7f8f13e00640 1 -- 172.21.15.27:0/986328679 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f140ad7d0 con 0x7f8f1c15e450 2024-01-15T18:05:05.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f27c9000640 1 -- 172.21.15.27:0/2281312730 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27bc04c5c0 con 0x7f27bc033e10 2024-01-15T18:05:05.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.096+0000 7f27c9000640 1 -- 172.21.15.27:0/2281312730 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27bc04cb80 con 0x7f27bc033e10 2024-01-15T18:05:05.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.100+0000 7f27b3400640 1 -- 172.21.15.27:0/2281312730 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f278c005260 con 0x7f27bc033e10 2024-01-15T18:05:05.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.100+0000 7f40dea00640 1 -- 172.21.15.27:0/1385217403 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40d8151680 con 0x7f40d80b2000 2024-01-15T18:05:05.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.100+0000 7f40dea00640 1 -- 172.21.15.27:0/1385217403 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40d81518d0 con 0x7f40d80b2000 2024-01-15T18:05:05.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.100+0000 7f5c7ae00640 1 -- 172.21.15.27:0/2108322595 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c44005260 con 0x7f5c74141d50 2024-01-15T18:05:05.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.100+0000 7f5c6be00640 1 -- 172.21.15.27:0/2108322595 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f5c6c01e3b0 con 0x7f5c74141d50 2024-01-15T18:05:05.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.100+0000 7f8f13e00640 1 -- 172.21.15.27:0/986328679 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f8f140b4070 con 0x7f8f1c15e450 2024-01-15T18:05:05.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.100+0000 7f27ba000640 1 -- 172.21.15.27:0/2281312730 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f27ac003af0 con 0x7f27bc033e10 2024-01-15T18:05:05.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.100+0000 7f5c6be00640 1 --2- 172.21.15.27:0/2108322595 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5c4c0669c0 0x7f5c4c068e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.100+0000 7f8f11600640 1 -- 172.21.15.27:0/986328679 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f1c04e850 con 0x7f8f1c15e450 2024-01-15T18:05:05.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.104+0000 7f40dea00640 1 -- 172.21.15.27:0/1385217403 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40d81523b0 con 0x7f40d80b2000 2024-01-15T18:05:05.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.100+0000 7f5c6be00640 1 -- 172.21.15.27:0/2108322595 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c6c084cc0 con 0x7f5c74141d50 2024-01-15T18:05:05.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.104+0000 7f8f13e00640 1 --2- 172.21.15.27:0/986328679 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8ef40669c0 0x7f8ef4068e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.104+0000 7f5c73e00640 1 --2- 172.21.15.27:0/2108322595 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5c4c0669c0 0x7f5c4c068e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.104+0000 7f8f13e00640 1 -- 172.21.15.27:0/986328679 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8f1411e850 con 0x7f8f1c15e450 2024-01-15T18:05:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.104+0000 7f27ba000640 1 --2- 172.21.15.27:0/2281312730 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f279c0668f0 0x7f279c068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.104+0000 7f8f1be00640 1 --2- 172.21.15.27:0/986328679 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8ef40669c0 0x7f8ef4068e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.104+0000 7f27ba000640 1 -- 172.21.15.27:0/2281312730 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f27ac083d00 con 0x7f27bc033e10 2024-01-15T18:05:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.104+0000 7f27c1e00640 1 --2- 172.21.15.27:0/2281312730 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f279c0668f0 0x7f279c068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.104+0000 7f5c73e00640 1 --2- 172.21.15.27:0/2108322595 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5c4c0669c0 0x7f5c4c068e80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c001790 tx=0x7f5c5c0015c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.104+0000 7f27ba000640 1 -- 172.21.15.27:0/2281312730 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f27ac0510d0 con 0x7f27bc033e10 2024-01-15T18:05:05.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.104+0000 7f8f1be00640 1 --2- 172.21.15.27:0/986328679 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8ef40669c0 0x7f8ef4068e80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8f040075f0 tx=0x7f8f04007c30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.104+0000 7f27c1e00640 1 --2- 172.21.15.27:0/2281312730 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f279c0668f0 0x7f279c068db0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f27b40b2000 tx=0x7f27b40aa070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.108+0000 7f8f13e00640 1 -- 172.21.15.27:0/986328679 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8f140ebb00 con 0x7f8f1c15e450 2024-01-15T18:05:05.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.108+0000 7f40d5600640 1 -- 172.21.15.27:0/1385217403 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f40c001a400 con 0x7f40d80b2000 2024-01-15T18:05:05.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.112+0000 7f40d5600640 1 --2- 172.21.15.27:0/1385217403 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f40b0066940 0x7f40b0068e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.112+0000 7f40dd600640 1 --2- 172.21.15.27:0/1385217403 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f40b0066940 0x7f40b0068e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.112+0000 7f40dd600640 1 --2- 172.21.15.27:0/1385217403 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f40b0066940 0x7f40b0068e00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f40c80098e0 tx=0x7f40c8009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.116+0000 7f40d5600640 1 -- 172.21.15.27:0/1385217403 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40c0055f40 con 0x7f40d80b2000 2024-01-15T18:05:05.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.116+0000 7f5c6be00640 1 -- 172.21.15.27:0/2108322595 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5c6c052020 con 0x7f5c74141d50 2024-01-15T18:05:05.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.120+0000 7f40d5600640 1 -- 172.21.15.27:0/1385217403 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f40c0051b80 con 0x7f40d80b2000 2024-01-15T18:05:05.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.132+0000 7f6e0a800640 1 Processor -- start 2024-01-15T18:05:05.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.132+0000 7f3ac7200640 1 Processor -- start 2024-01-15T18:05:05.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.132+0000 7f6e0a800640 1 -- start start 2024-01-15T18:05:05.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.132+0000 7f3ac7200640 1 -- start start 2024-01-15T18:05:05.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.132+0000 7f6e0a800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6e0415dbf0 0x7f6e040710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.132+0000 7f3ac7200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f3ac806c5a0 0x7f3ac815cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.132+0000 7f3ac7200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ac815d4a0 0x7f3ac80ce390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.132+0000 7f6e0a800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e04070430 con 0x7f6e0415e450 2024-01-15T18:05:05.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.132+0000 7f6e0a800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e040b2580 con 0x7f6e0415dfd0 2024-01-15T18:05:05.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.132+0000 7f6e0a800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e040b2860 con 0x7f6e0415dbf0 2024-01-15T18:05:05.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.132+0000 7f3ac5400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f3ac806c5a0 0x7f3ac815cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.136+0000 7f3ac5400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f3ac806c5a0 0x7f3ac815cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:51642/0 (socket says 172.21.15.27:51642) 2024-01-15T18:05:05.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.136+0000 7f6e09e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6e0415e450 0x7f6e0406d070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42366/0 (socket says 172.21.15.27:42366) 2024-01-15T18:05:05.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.136+0000 7f6e09e00640 1 -- 172.21.15.27:0/3498888719 learned_addr learned my addr 172.21.15.27:0/3498888719 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:05.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.136+0000 7f3ac5e00640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7f3ac806c1c0 0x7f3ac81536d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:57052/0 (socket says 172.21.15.27:57052) 2024-01-15T18:05:05.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.136+0000 7f3ac5e00640 1 -- 172.21.15.27:0/1589151992 learned_addr learned my addr 172.21.15.27:0/1589151992 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:05.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.136+0000 7f3ac6800640 1 --2- 172.21.15.27:0/1589151992 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ac815d4a0 0x7f3ac80ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.136+0000 7f3ac7200640 1 -- 172.21.15.27:0/1589151992 --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac8070c10 con 0x7f3ac806c1c0 2024-01-15T18:05:05.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.136+0000 7f3ac7200640 1 -- 172.21.15.27:0/1589151992 --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ac80b2f80 con 0x7f3ac806c5a0 2024-01-15T18:05:05.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.136+0000 7f3ac7200640 1 -- 172.21.15.27:0/1589151992 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac815d910 con 0x7f3ac815d4a0 2024-01-15T18:05:05.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.136+0000 7f3ac4a00640 1 -- 172.21.15.27:0/1589151992 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265452545 0 0) 0x7f3ac8070c10 con 0x7f3ac806c1c0 2024-01-15T18:05:05.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.136+0000 7f3ac4a00640 1 -- 172.21.15.27:0/1589151992 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aa8003530 con 0x7f3ac806c1c0 2024-01-15T18:05:05.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.136+0000 7f3ac6800640 1 -- 172.21.15.27:0/1589151992 >> v2:172.21.15.93:3300/0 conn(0x7f3ac806c5a0 msgr2=0x7f3ac815cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:05.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.136+0000 7f3ac6800640 1 --2- 172.21.15.27:0/1589151992 >> v2:172.21.15.93:3300/0 conn(0x7f3ac806c5a0 0x7f3ac815cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.136+0000 7f3ac6800640 1 -- 172.21.15.27:0/1589151992 >> v1:172.21.15.93:6789/0 conn(0x7f3ac806c1c0 legacy=0x7f3ac81536d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.136+0000 7f3ac6800640 1 -- 172.21.15.27:0/1589151992 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac8153de0 con 0x7f3ac815d4a0 2024-01-15T18:05:05.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f6e03e00640 1 -- 172.21.15.27:0/3498888719 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1387398846 0 0) 0x7f6e040b2580 con 0x7f6e0415dfd0 2024-01-15T18:05:05.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f6e03e00640 1 -- 172.21.15.27:0/3498888719 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6de4003530 con 0x7f6e0415dfd0 2024-01-15T18:05:05.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f6e09400640 1 --2- 172.21.15.27:0/3498888719 >> v2:172.21.15.27:3300/0 conn(0x7f6e0415dbf0 0x7f6e040710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f6e03e00640 1 -- 172.21.15.27:0/3498888719 <== mon.1 v1:172.21.15.93:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2967701755 0 0) 0x7f6de4003530 con 0x7f6e0415dfd0 2024-01-15T18:05:05.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f6e03e00640 1 -- 172.21.15.27:0/3498888719 >> v2:172.21.15.27:3300/0 conn(0x7f6e0415dbf0 msgr2=0x7f6e040710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:05.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f6e03e00640 1 --2- 172.21.15.27:0/3498888719 >> v2:172.21.15.27:3300/0 conn(0x7f6e0415dbf0 0x7f6e040710c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f6e03e00640 1 -- 172.21.15.27:0/3498888719 >> v1:172.21.15.27:6789/0 conn(0x7f6e0415e450 legacy=0x7f6e0406d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f6e03e00640 1 -- 172.21.15.27:0/3498888719 --> v1:172.21.15.93:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e040b2b40 con 0x7f6e0415dfd0 2024-01-15T18:05:05.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f6e03e00640 1 -- 172.21.15.27:0/3498888719 <== mon.1 v1:172.21.15.93:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f6dec001680 con 0x7f6e0415dfd0 2024-01-15T18:05:05.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f6e09400640 1 --2- 172.21.15.27:0/3498888719 >> v2:172.21.15.27:3300/0 conn(0x7f6e0415dbf0 0x7f6e040710c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:05.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f6e03e00640 1 -- 172.21.15.27:0/3498888719 >> v1:172.21.15.93:6789/0 conn(0x7f6e0415dfd0 legacy=0x7f6e0406c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f6e03e00640 1 --2- 172.21.15.27:0/3498888719 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6de40039f0 0x7f6de4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f6e03e00640 1 --2- 172.21.15.27:0/3498888719 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6de40047d0 0x7f6de4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f6e03e00640 1 --2- 172.21.15.27:0/3498888719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6de4005190 0x7f6de400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f6e03e00640 1 -- 172.21.15.27:0/3498888719 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e040b2860 con 0x7f6de4005190 2024-01-15T18:05:05.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f6e03e00640 1 -- 172.21.15.27:0/3498888719 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de400c060 con 0x7f6de40047d0 2024-01-15T18:05:05.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f6e03e00640 1 -- 172.21.15.27:0/3498888719 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6de400c220 con 0x7f6de40039f0 2024-01-15T18:05:05.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f6e09400640 1 --2- 172.21.15.27:0/3498888719 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6de40039f0 0x7f6de4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f6e08a00640 1 --2- 172.21.15.27:0/3498888719 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6de40047d0 0x7f6de4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f6e09e00640 1 --2- 172.21.15.27:0/3498888719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6de4005190 0x7f6de400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f3ac6800640 1 --2- 172.21.15.27:0/1589151992 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ac815d4a0 0x7f3ac80ce390 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3ac8153f70 tx=0x7f3ac001c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97d24748fd455d60 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f3ac4a00640 1 -- 172.21.15.27:0/1589151992 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac00a59c0 con 0x7f3ac815d4a0 2024-01-15T18:05:05.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f3ac4a00640 1 -- 172.21.15.27:0/1589151992 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac00a5b20 con 0x7f3ac815d4a0 2024-01-15T18:05:05.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f6e08a00640 1 -- 172.21.15.27:0/3498888719 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6de40039f0 msgr2=0x7f6de4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:05.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f6e08a00640 1 --2- 172.21.15.27:0/3498888719 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6de40039f0 0x7f6de4003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f6e08a00640 1 -- 172.21.15.27:0/3498888719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6de4005190 msgr2=0x7f6de400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:05.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f6e08a00640 1 --2- 172.21.15.27:0/3498888719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6de4005190 0x7f6de400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.140+0000 7f6e08a00640 1 -- 172.21.15.27:0/3498888719 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de4003530 con 0x7f6de40047d0 2024-01-15T18:05:05.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.144+0000 7f3ac4a00640 1 -- 172.21.15.27:0/1589151992 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac00a59c0 con 0x7f3ac815d4a0 2024-01-15T18:05:05.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.144+0000 7f3ac7200640 1 -- 172.21.15.27:0/1589151992 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ac815d4a0 msgr2=0x7f3ac80ce390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.144+0000 7f3ac7200640 1 --2- 172.21.15.27:0/1589151992 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ac815d4a0 0x7f3ac80ce390 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3ac8153f70 tx=0x7f3ac001c940 comp rx=0 tx=0).stop 2024-01-15T18:05:05.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.144+0000 7f6e09e00640 1 --2- 172.21.15.27:0/3498888719 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6de4005190 0x7f6de400b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:05.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.144+0000 7f6e09e00640 1 -- 172.21.15.27:0/3498888719 reap_dead start 2024-01-15T18:05:05.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.144+0000 7f3ac7200640 1 -- 172.21.15.27:0/1589151992 shutdown_connections 2024-01-15T18:05:05.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.144+0000 7f3ac7200640 1 --2- 172.21.15.27:0/1589151992 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ac815d4a0 0x7f3ac80ce390 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.144+0000 7f3ac7200640 1 --2- 172.21.15.27:0/1589151992 >> v2:172.21.15.93:3300/0 conn(0x7f3ac806c5a0 0x7f3ac815cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.144+0000 7f3ac7200640 1 -- 172.21.15.27:0/1589151992 >> 172.21.15.27:0/1589151992 conn(0x7f3ac806a6c0 msgr2=0x7f3ac806aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:05.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.144+0000 7f3ac7200640 1 -- 172.21.15.27:0/1589151992 shutdown_connections 2024-01-15T18:05:05.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.144+0000 7f3ac7200640 1 -- 172.21.15.27:0/1589151992 wait complete. 2024-01-15T18:05:05.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.148+0000 7f3ac7200640 1 Processor -- start 2024-01-15T18:05:05.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.148+0000 7f3ac7200640 1 -- start start 2024-01-15T18:05:05.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.148+0000 7f6e08a00640 1 --2- 172.21.15.27:0/3498888719 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6de40047d0 0x7f6de4004c50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6dec007b20 tx=0x7f6dec027a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58d4e18e9d2fb1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.152+0000 7f6e03e00640 1 -- 172.21.15.27:0/3498888719 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dec0b2630 con 0x7f6de40047d0 2024-01-15T18:05:05.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.152+0000 7f6e03e00640 1 -- 172.21.15.27:0/3498888719 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dec0b2790 con 0x7f6de40047d0 2024-01-15T18:05:05.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.152+0000 7f6e03e00640 1 -- 172.21.15.27:0/3498888719 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dec0b2bc0 con 0x7f6de40047d0 2024-01-15T18:05:05.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.152+0000 7f3ac7200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ac806c1c0 0x7f3ac82023d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.152+0000 7f6e0a800640 1 -- 172.21.15.27:0/3498888719 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6de40047d0 msgr2=0x7f6de4004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.152+0000 7f6e0a800640 1 --2- 172.21.15.27:0/3498888719 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6de40047d0 0x7f6de4004c50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6dec007b20 tx=0x7f6dec027a00 comp rx=0 tx=0).stop 2024-01-15T18:05:05.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.152+0000 7f3ac7200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ac815d4a0 0x7f3ac815b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.152+0000 7f3ac5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ac806c1c0 0x7f3ac82023d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.152+0000 7f3ac7200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ac815bdb0 0x7f3ac814ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.152+0000 7f3ac5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ac806c1c0 0x7f3ac82023d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59276/0 (socket says 172.21.15.27:59276) 2024-01-15T18:05:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.152+0000 7f3ac5e00640 1 -- 172.21.15.27:0/1284107258 learned_addr learned my addr 172.21.15.27:0/1284107258 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.152+0000 7f3ac7200640 1 -- 172.21.15.27:0/1284107258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac814c5b0 con 0x7f3ac806c1c0 2024-01-15T18:05:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.152+0000 7f3ac6800640 1 --2- 172.21.15.27:0/1284107258 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ac815bdb0 0x7f3ac814ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.152+0000 7f3ac5400640 1 --2- 172.21.15.27:0/1284107258 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ac815d4a0 0x7f3ac815b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.152+0000 7f3ac7200640 1 -- 172.21.15.27:0/1284107258 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac815c5f0 con 0x7f3ac815bdb0 2024-01-15T18:05:05.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.152+0000 7f3ac7200640 1 -- 172.21.15.27:0/1284107258 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac814bf60 con 0x7f3ac815d4a0 2024-01-15T18:05:05.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.152+0000 7f3ac5e00640 1 -- 172.21.15.27:0/1284107258 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ac815d4a0 msgr2=0x7f3ac815b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.152+0000 7f3ac5e00640 1 --2- 172.21.15.27:0/1284107258 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ac815d4a0 0x7f3ac815b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.152+0000 7f3ac5e00640 1 -- 172.21.15.27:0/1284107258 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ac815bdb0 msgr2=0x7f3ac814ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.152+0000 7f3ac5e00640 1 --2- 172.21.15.27:0/1284107258 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ac815bdb0 0x7f3ac814ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.152+0000 7f3ac5e00640 1 -- 172.21.15.27:0/1284107258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac0009a70 con 0x7f3ac806c1c0 2024-01-15T18:05:05.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.152+0000 7f3ac6800640 1 --2- 172.21.15.27:0/1284107258 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ac815bdb0 0x7f3ac814ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:05.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.152+0000 7f3ac5e00640 1 --2- 172.21.15.27:0/1284107258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ac806c1c0 0x7f3ac82023d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3ab000c420 tx=0x7f3ab000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.156+0000 7f3abe000640 1 -- 172.21.15.27:0/1284107258 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0005cf0 con 0x7f3ac806c1c0 2024-01-15T18:05:05.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.156+0000 7f3abe000640 1 -- 172.21.15.27:0/1284107258 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ab0005e50 con 0x7f3ac806c1c0 2024-01-15T18:05:05.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.156+0000 7f3abe000640 1 -- 172.21.15.27:0/1284107258 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0005cf0 con 0x7f3ac806c1c0 2024-01-15T18:05:05.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.156+0000 7f3ac7200640 1 -- 172.21.15.27:0/1284107258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac814c240 con 0x7f3ac806c1c0 2024-01-15T18:05:05.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.156+0000 7f3ac7200640 1 -- 172.21.15.27:0/1284107258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac8202910 con 0x7f3ac806c1c0 2024-01-15T18:05:05.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.156+0000 7f3ab7400640 1 -- 172.21.15.27:0/1284107258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac814c3d0 con 0x7f3ac806c1c0 2024-01-15T18:05:05.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.160+0000 7f6e0a800640 1 -- 172.21.15.27:0/3498888719 shutdown_connections 2024-01-15T18:05:05.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.160+0000 7f6e0a800640 1 --2- 172.21.15.27:0/3498888719 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6de40047d0 0x7f6de4004c50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.160+0000 7f6e0a800640 1 -- 172.21.15.27:0/3498888719 >> 172.21.15.27:0/3498888719 conn(0x7f6e0406a6c0 msgr2=0x7f6e0406aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:05.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.160+0000 7f6e0a800640 1 -- 172.21.15.27:0/3498888719 shutdown_connections 2024-01-15T18:05:05.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.160+0000 7f6e0a800640 1 -- 172.21.15.27:0/3498888719 wait complete. 2024-01-15T18:05:05.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.160+0000 7f6e0a800640 1 Processor -- start 2024-01-15T18:05:05.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.160+0000 7f6e0a800640 1 -- start start 2024-01-15T18:05:05.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.160+0000 7f3abe000640 1 -- 172.21.15.27:0/1284107258 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f3ab00070d0 con 0x7f3ac806c1c0 2024-01-15T18:05:05.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.160+0000 7f3abe000640 1 --2- 172.21.15.27:0/1284107258 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3aa0066950 0x7f3aa0068e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.164+0000 7f6e0a800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6de40047d0 0x7f6e0415f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.164+0000 7f6e0a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e04152320 0x7f6e0415fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.164+0000 7f6e0a800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6e040aa8a0 0x7f6e040aad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.164+0000 7f6e09400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6de40047d0 0x7f6e0415f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.164+0000 7f6e09400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6de40047d0 0x7f6e0415f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:42758/0 (socket says 172.21.15.27:42758) 2024-01-15T18:05:05.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.164+0000 7f3ac5400640 1 --2- 172.21.15.27:0/1284107258 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3aa0066950 0x7f3aa0068e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.164+0000 7f6e09400640 1 -- 172.21.15.27:0/3913017560 learned_addr learned my addr 172.21.15.27:0/3913017560 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:05.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.164+0000 7f6e0a800640 1 -- 172.21.15.27:0/3913017560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e040ab290 con 0x7f6e04152320 2024-01-15T18:05:05.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.164+0000 7f6e0a800640 1 -- 172.21.15.27:0/3913017560 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e04151560 con 0x7f6e040aa8a0 2024-01-15T18:05:05.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.164+0000 7f6e09e00640 1 --2- 172.21.15.27:0/3913017560 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6e040aa8a0 0x7f6e040aad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.164+0000 7f6e0a800640 1 -- 172.21.15.27:0/3913017560 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e04151840 con 0x7f6de40047d0 2024-01-15T18:05:05.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.164+0000 7f3abe000640 1 -- 172.21.15.27:0/1284107258 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ab0083680 con 0x7f3ac806c1c0 2024-01-15T18:05:05.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.164+0000 7f3ac5400640 1 --2- 172.21.15.27:0/1284107258 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3aa0066950 0x7f3aa0068e10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3ac80cd790 tx=0x7f3ab8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.164+0000 7f6e09e00640 1 -- 172.21.15.27:0/3913017560 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6de40047d0 msgr2=0x7f6e0415f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.164+0000 7f6e09e00640 1 --2- 172.21.15.27:0/3913017560 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6de40047d0 0x7f6e0415f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.164+0000 7f6e09e00640 1 -- 172.21.15.27:0/3913017560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e04152320 msgr2=0x7f6e0415fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.164+0000 7f6e09e00640 1 --2- 172.21.15.27:0/3913017560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e04152320 0x7f6e0415fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.164+0000 7f6e09e00640 1 -- 172.21.15.27:0/3913017560 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e04151b20 con 0x7f6e040aa8a0 2024-01-15T18:05:05.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.168+0000 7f6e09e00640 1 --2- 172.21.15.27:0/3913017560 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6e040aa8a0 0x7f6e040aad50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6e040ae590 tx=0x7f6dfc00f5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.168+0000 7f6e01600640 1 -- 172.21.15.27:0/3913017560 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc007af0 con 0x7f6e040aa8a0 2024-01-15T18:05:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.168+0000 7f6e01600640 1 -- 172.21.15.27:0/3913017560 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dfc00fdb0 con 0x7f6e040aa8a0 2024-01-15T18:05:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.168+0000 7f6e01600640 1 -- 172.21.15.27:0/3913017560 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc013630 con 0x7f6e040aa8a0 2024-01-15T18:05:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.168+0000 7f6e0a800640 1 -- 172.21.15.27:0/3913017560 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e04151e00 con 0x7f6e040aa8a0 2024-01-15T18:05:05.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.168+0000 7f6e0a800640 1 -- 172.21.15.27:0/3913017560 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e041f1f10 con 0x7f6e040aa8a0 2024-01-15T18:05:05.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.172+0000 7f6e0a800640 1 -- 172.21.15.27:0/3913017560 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e0404e850 con 0x7f6e040aa8a0 2024-01-15T18:05:05.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.172+0000 7f6e01600640 1 -- 172.21.15.27:0/3913017560 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f6dfc013790 con 0x7f6e040aa8a0 2024-01-15T18:05:05.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.172+0000 7f6e01600640 1 --2- 172.21.15.27:0/3913017560 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6ddc0669f0 0x7f6ddc068eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.172+0000 7f6e01600640 1 -- 172.21.15.27:0/3913017560 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6dfc084600 con 0x7f6e040aa8a0 2024-01-15T18:05:05.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.172+0000 7f6e09400640 1 --2- 172.21.15.27:0/3913017560 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6ddc0669f0 0x7f6ddc068eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.176+0000 7f6e09400640 1 --2- 172.21.15.27:0/3913017560 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6ddc0669f0 0x7f6ddc068eb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6df4009df0 tx=0x7f6df4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.176+0000 7f6e01600640 1 -- 172.21.15.27:0/3913017560 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6dfc0518b0 con 0x7f6e040aa8a0 2024-01-15T18:05:05.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.176+0000 7f3abe000640 1 -- 172.21.15.27:0/1284107258 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3ab0050840 con 0x7f3ac806c1c0 2024-01-15T18:05:05.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.424+0000 7fa2e5200640 1 -- 172.21.15.27:0/3150040607 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fa2b0005510 con 0x7fa2d8006090 2024-01-15T18:05:05.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.424+0000 7fa2d6000640 1 -- 172.21.15.27:0/3150040607 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa2e00c2e60 con 0x7fa2d8006090 2024-01-15T18:05:05.430 INFO:teuthology.orchestra.run.smithi027.stdout:64424509444 2024-01-15T18:05:05.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.428+0000 7f8f11600640 1 -- 172.21.15.27:0/986328679 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f8f1c1571e0 con 0x7f8f1c15e450 2024-01-15T18:05:05.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.428+0000 7fa2cf400640 1 -- 172.21.15.27:0/3150040607 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa2b80669c0 msgr2=0x7fa2b8068e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.428+0000 7fa2cf400640 1 --2- 172.21.15.27:0/3150040607 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa2b80669c0 0x7fa2b8068e80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa2d00b3000 tx=0x7fa2d00ac000 comp rx=0 tx=0).stop 2024-01-15T18:05:05.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.428+0000 7fa2cf400640 1 -- 172.21.15.27:0/3150040607 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa2d8006090 msgr2=0x7fa2d80ec9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.428+0000 7fa2cf400640 1 --2- 172.21.15.27:0/3150040607 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa2d8006090 0x7fa2d80ec9f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa2e004f6a0 tx=0x7fa2e0074ad0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.428+0000 7fa2cf400640 1 -- 172.21.15.27:0/3150040607 shutdown_connections 2024-01-15T18:05:05.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.428+0000 7fa2cf400640 1 --2- 172.21.15.27:0/3150040607 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa2b80669c0 0x7fa2b8068e80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.428+0000 7fa2cf400640 1 --2- 172.21.15.27:0/3150040607 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa2d81048f0 0x7fa2d80ef760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.428+0000 7fa2cf400640 1 --2- 172.21.15.27:0/3150040607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2d804f560 0x7fa2d80ef220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.428+0000 7fa2cf400640 1 --2- 172.21.15.27:0/3150040607 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa2d8006090 0x7fa2d80ec9f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.428+0000 7fa2cf400640 1 -- 172.21.15.27:0/3150040607 >> 172.21.15.27:0/3150040607 conn(0x7fa2d8054c70 msgr2=0x7fa2d8102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:05.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.428+0000 7f8f13e00640 1 -- 172.21.15.27:0/986328679 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8f140b2090 con 0x7f8f1c15e450 2024-01-15T18:05:05.435 INFO:teuthology.orchestra.run.smithi027.stdout:64424509446 2024-01-15T18:05:05.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.432+0000 7fa2cf400640 1 -- 172.21.15.27:0/3150040607 shutdown_connections 2024-01-15T18:05:05.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.432+0000 7fa2cf400640 1 -- 172.21.15.27:0/3150040607 wait complete. 2024-01-15T18:05:05.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.432+0000 7f8f21400640 1 -- 172.21.15.27:0/986328679 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8ef40669c0 msgr2=0x7f8ef4068e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.432+0000 7f8f21400640 1 --2- 172.21.15.27:0/986328679 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8ef40669c0 0x7f8ef4068e80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8f040075f0 tx=0x7f8f04007c30 comp rx=0 tx=0).stop 2024-01-15T18:05:05.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.432+0000 7f8f21400640 1 -- 172.21.15.27:0/986328679 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8f1c15e450 msgr2=0x7f8f1c157fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.432+0000 7f8f21400640 1 --2- 172.21.15.27:0/986328679 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8f1c15e450 0x7f8f1c157fe0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8f140ac000 tx=0x7f8f14019730 comp rx=0 tx=0).stop 2024-01-15T18:05:05.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.432+0000 7f8f21400640 1 -- 172.21.15.27:0/986328679 shutdown_connections 2024-01-15T18:05:05.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.432+0000 7f8f21400640 1 --2- 172.21.15.27:0/986328679 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8ef40669c0 0x7f8ef4068e80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.432+0000 7f8f21400640 1 --2- 172.21.15.27:0/986328679 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8f1c15e450 0x7f8f1c157fe0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.432+0000 7f8f21400640 1 --2- 172.21.15.27:0/986328679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f1c15dfd0 0x7f8f1c157530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.432+0000 7f8f21400640 1 --2- 172.21.15.27:0/986328679 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f1c15dbf0 0x7f8f1c14e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.432+0000 7f8f21400640 1 -- 172.21.15.27:0/986328679 >> 172.21.15.27:0/986328679 conn(0x7f8f1c06a6c0 msgr2=0x7f8f1c0b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:05.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.436+0000 7f8f21400640 1 -- 172.21.15.27:0/986328679 shutdown_connections 2024-01-15T18:05:05.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.436+0000 7f8f21400640 1 -- 172.21.15.27:0/986328679 wait complete. 2024-01-15T18:05:05.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.444+0000 7f27b3400640 1 -- 172.21.15.27:0/2281312730 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f278c0057e0 con 0x7f27bc033e10 2024-01-15T18:05:05.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.444+0000 7f27ba000640 1 -- 172.21.15.27:0/2281312730 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f27ac017020 con 0x7f27bc033e10 2024-01-15T18:05:05.449 INFO:teuthology.orchestra.run.smithi027.stdout:64424509445 2024-01-15T18:05:05.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.448+0000 7f27b3400640 1 -- 172.21.15.27:0/2281312730 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f279c0668f0 msgr2=0x7f279c068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.448+0000 7f27b3400640 1 --2- 172.21.15.27:0/2281312730 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f279c0668f0 0x7f279c068db0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f27b40b2000 tx=0x7f27b40aa070 comp rx=0 tx=0).stop 2024-01-15T18:05:05.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.448+0000 7f27b3400640 1 -- 172.21.15.27:0/2281312730 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27bc033e10 msgr2=0x7f27bc04bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.448+0000 7f27b3400640 1 --2- 172.21.15.27:0/2281312730 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27bc033e10 0x7f27bc04bb70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f27ac001ec0 tx=0x7f27ac00c8e0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.448+0000 7f27b3400640 1 -- 172.21.15.27:0/2281312730 shutdown_connections 2024-01-15T18:05:05.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.448+0000 7f27b3400640 1 --2- 172.21.15.27:0/2281312730 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f279c0668f0 0x7f279c068db0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.448+0000 7f27b3400640 1 --2- 172.21.15.27:0/2281312730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27bc047bf0 0x7f27bc17e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.448+0000 7f27b3400640 1 --2- 172.21.15.27:0/2281312730 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27bc033e10 0x7f27bc04bb70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.448+0000 7f27b3400640 1 --2- 172.21.15.27:0/2281312730 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27bc005ec0 0x7f27bc04dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.448+0000 7f27b3400640 1 -- 172.21.15.27:0/2281312730 >> 172.21.15.27:0/2281312730 conn(0x7f27bc009f90 msgr2=0x7f27bc0e6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:05.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.452+0000 7f27b3400640 1 -- 172.21.15.27:0/2281312730 shutdown_connections 2024-01-15T18:05:05.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.452+0000 7f27b3400640 1 -- 172.21.15.27:0/2281312730 wait complete. 2024-01-15T18:05:05.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.452+0000 7fb55b200640 1 -- 172.21.15.27:0/1592273490 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fb5541497d0 con 0x7fb554152a90 2024-01-15T18:05:05.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.452+0000 7f40dea00640 1 -- 172.21.15.27:0/1385217403 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f40d81523b0 con 0x7f40d80b2000 2024-01-15T18:05:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.456+0000 7fb552000640 1 -- 172.21.15.27:0/1592273490 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb54c019020 con 0x7fb554152a90 2024-01-15T18:05:05.462 INFO:teuthology.orchestra.run.smithi027.stdout:60129542149 2024-01-15T18:05:05.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.456+0000 7f40d5600640 1 -- 172.21.15.27:0/1385217403 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f40d81523b0 con 0x7f40d80b2000 2024-01-15T18:05:05.462 INFO:teuthology.orchestra.run.smithi027.stdout:64424509444 2024-01-15T18:05:05.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.460+0000 7f40dea00640 1 -- 172.21.15.27:0/1385217403 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f40b0066940 msgr2=0x7f40b0068e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.460+0000 7f40dea00640 1 --2- 172.21.15.27:0/1385217403 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f40b0066940 0x7f40b0068e00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f40c80098e0 tx=0x7f40c8009060 comp rx=0 tx=0).stop 2024-01-15T18:05:05.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.460+0000 7f40dea00640 1 -- 172.21.15.27:0/1385217403 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f40d80b2000 msgr2=0x7f40d814d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.460+0000 7f40dea00640 1 --2- 172.21.15.27:0/1385217403 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f40d80b2000 0x7f40d814d0c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f40c000b3f0 tx=0x7f40c000b8b0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.460+0000 7fb55b200640 1 -- 172.21.15.27:0/1592273490 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb52c066940 msgr2=0x7fb52c068e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.460+0000 7fb55b200640 1 --2- 172.21.15.27:0/1592273490 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb52c066940 0x7fb52c068e00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb53c000f70 tx=0x7fb53c007e50 comp rx=0 tx=0).stop 2024-01-15T18:05:05.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.460+0000 7fb55b200640 1 -- 172.21.15.27:0/1592273490 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb554152a90 msgr2=0x7fb5540cbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.460+0000 7fb55b200640 1 --2- 172.21.15.27:0/1592273490 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb554152a90 0x7fb5540cbc20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb54c00b970 tx=0x7fb54c00be30 comp rx=0 tx=0).stop 2024-01-15T18:05:05.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.460+0000 7f40dea00640 1 -- 172.21.15.27:0/1385217403 shutdown_connections 2024-01-15T18:05:05.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.460+0000 7fb55b200640 1 -- 172.21.15.27:0/1592273490 shutdown_connections 2024-01-15T18:05:05.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.460+0000 7fb55b200640 1 --2- 172.21.15.27:0/1592273490 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb52c066940 0x7fb52c068e00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.460+0000 7fb55b200640 1 --2- 172.21.15.27:0/1592273490 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb554152a90 0x7fb5540cbc20 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.460+0000 7fb55b200640 1 --2- 172.21.15.27:0/1592273490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb554157510 0x7fb554152550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.460+0000 7fb55b200640 1 --2- 172.21.15.27:0/1592273490 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb554157130 0x7fb5541482a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.460+0000 7fb55b200640 1 -- 172.21.15.27:0/1592273490 >> 172.21.15.27:0/1592273490 conn(0x7fb55406aaf0 msgr2=0x7fb554159fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.460+0000 7f40dea00640 1 --2- 172.21.15.27:0/1385217403 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f40b0066940 0x7f40b0068e00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.460+0000 7f40dea00640 1 --2- 172.21.15.27:0/1385217403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f40d815a0c0 0x7f40d8070180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.460+0000 7f40dea00640 1 --2- 172.21.15.27:0/1385217403 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f40d80b2000 0x7f40d814d0c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.460+0000 7f40dea00640 1 --2- 172.21.15.27:0/1385217403 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f40d80b1b50 0x7f40d8152690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.460+0000 7f40dea00640 1 -- 172.21.15.27:0/1385217403 >> 172.21.15.27:0/1385217403 conn(0x7f40d80b0850 msgr2=0x7f40d8150ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:05.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.460+0000 7f40dea00640 1 -- 172.21.15.27:0/1385217403 shutdown_connections 2024-01-15T18:05:05.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.460+0000 7f40dea00640 1 -- 172.21.15.27:0/1385217403 wait complete. 2024-01-15T18:05:05.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.460+0000 7fb55b200640 1 -- 172.21.15.27:0/1592273490 shutdown_connections 2024-01-15T18:05:05.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.460+0000 7fb55b200640 1 -- 172.21.15.27:0/1592273490 wait complete. 2024-01-15T18:05:05.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.460+0000 7f5c7ae00640 1 -- 172.21.15.27:0/2108322595 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f5c44005510 con 0x7f5c74141d50 2024-01-15T18:05:05.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.464+0000 7f5c6be00640 1 -- 172.21.15.27:0/2108322595 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5c6c01a090 con 0x7f5c74141d50 2024-01-15T18:05:05.470 INFO:teuthology.orchestra.run.smithi027.stdout:64424509445 2024-01-15T18:05:05.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.468+0000 7f5c7ae00640 1 -- 172.21.15.27:0/2108322595 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5c4c0669c0 msgr2=0x7f5c4c068e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.468+0000 7f5c7ae00640 1 --2- 172.21.15.27:0/2108322595 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5c4c0669c0 0x7f5c4c068e80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c001790 tx=0x7f5c5c0015c0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.468+0000 7f5c7ae00640 1 -- 172.21.15.27:0/2108322595 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c74141d50 msgr2=0x7f5c740cccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.468+0000 7f5c7ae00640 1 --2- 172.21.15.27:0/2108322595 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c74141d50 0x7f5c740cccc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c00cef0 tx=0x7f5c6c00a710 comp rx=0 tx=0).stop 2024-01-15T18:05:05.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.468+0000 7f5c7ae00640 1 -- 172.21.15.27:0/2108322595 shutdown_connections 2024-01-15T18:05:05.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.468+0000 7f5c7ae00640 1 --2- 172.21.15.27:0/2108322595 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5c4c0669c0 0x7f5c4c068e80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.468+0000 7f5c7ae00640 1 --2- 172.21.15.27:0/2108322595 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c74141d50 0x7f5c740cccc0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.468+0000 7f5c7ae00640 1 --2- 172.21.15.27:0/2108322595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7414afa0 0x7f5c740aec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.468+0000 7f5c7ae00640 1 --2- 172.21.15.27:0/2108322595 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5c740b1b50 0x7f5c740ae6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.468+0000 7f5c7ae00640 1 -- 172.21.15.27:0/2108322595 >> 172.21.15.27:0/2108322595 conn(0x7f5c740b0850 msgr2=0x7f5c74162d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:05.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.468+0000 7f5c7ae00640 1 -- 172.21.15.27:0/2108322595 shutdown_connections 2024-01-15T18:05:05.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.468+0000 7f5c7ae00640 1 -- 172.21.15.27:0/2108322595 wait complete. 2024-01-15T18:05:05.480 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.1 2024-01-15T18:05:05.481 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509444 for osd.3 2024-01-15T18:05:05.493 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.6 2024-01-15T18:05:05.494 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509444 for osd.2 2024-01-15T18:05:05.497 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.0 2024-01-15T18:05:05.503 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.4 2024-01-15T18:05:05.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.504+0000 7f6e0a800640 1 -- 172.21.15.27:0/3913017560 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f6e040716e0 con 0x7f6e040aa8a0 2024-01-15T18:05:05.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.504+0000 7f6e01600640 1 -- 172.21.15.27:0/3913017560 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6dfc001fb0 con 0x7f6e040aa8a0 2024-01-15T18:05:05.509 INFO:teuthology.orchestra.run.smithi027.stdout:60129542149 2024-01-15T18:05:05.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.508+0000 7f6e0a800640 1 -- 172.21.15.27:0/3913017560 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6ddc0669f0 msgr2=0x7f6ddc068eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.508+0000 7f6e0a800640 1 --2- 172.21.15.27:0/3913017560 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6ddc0669f0 0x7f6ddc068eb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6df4009df0 tx=0x7f6df4008040 comp rx=0 tx=0).stop 2024-01-15T18:05:05.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.508+0000 7f6e0a800640 1 -- 172.21.15.27:0/3913017560 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6e040aa8a0 msgr2=0x7f6e040aad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.508+0000 7f6e0a800640 1 --2- 172.21.15.27:0/3913017560 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6e040aa8a0 0x7f6e040aad50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6e040ae590 tx=0x7f6dfc00f5a0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.508+0000 7f6e0a800640 1 -- 172.21.15.27:0/3913017560 shutdown_connections 2024-01-15T18:05:05.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.508+0000 7f6e0a800640 1 --2- 172.21.15.27:0/3913017560 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6ddc0669f0 0x7f6ddc068eb0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.508+0000 7f6e0a800640 1 --2- 172.21.15.27:0/3913017560 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6e040aa8a0 0x7f6e040aad50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.508+0000 7f6e0a800640 1 --2- 172.21.15.27:0/3913017560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e04152320 0x7f6e0415fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.508+0000 7f6e0a800640 1 --2- 172.21.15.27:0/3913017560 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6de40047d0 0x7f6e0415f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.508+0000 7f6e0a800640 1 -- 172.21.15.27:0/3913017560 >> 172.21.15.27:0/3913017560 conn(0x7f6e0406a6c0 msgr2=0x7f6e040af860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:05.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.508+0000 7f6e0a800640 1 -- 172.21.15.27:0/3913017560 shutdown_connections 2024-01-15T18:05:05.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.508+0000 7f6e0a800640 1 -- 172.21.15.27:0/3913017560 wait complete. 2024-01-15T18:05:05.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.516+0000 7f3ab7400640 1 -- 172.21.15.27:0/1284107258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f3ac804e850 con 0x7f3ac806c1c0 2024-01-15T18:05:05.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.516+0000 7f3abe000640 1 -- 172.21.15.27:0/1284107258 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3ab0017090 con 0x7f3ac806c1c0 2024-01-15T18:05:05.520 INFO:teuthology.orchestra.run.smithi027.stdout:64424509445 2024-01-15T18:05:05.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.520+0000 7f3ab7400640 1 -- 172.21.15.27:0/1284107258 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3aa0066950 msgr2=0x7f3aa0068e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.520+0000 7f3ab7400640 1 --2- 172.21.15.27:0/1284107258 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3aa0066950 0x7f3aa0068e10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3ac80cd790 tx=0x7f3ab8008040 comp rx=0 tx=0).stop 2024-01-15T18:05:05.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.520+0000 7f3ab7400640 1 -- 172.21.15.27:0/1284107258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ac806c1c0 msgr2=0x7f3ac82023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.520+0000 7f3ab7400640 1 --2- 172.21.15.27:0/1284107258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ac806c1c0 0x7f3ac82023d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3ab000c420 tx=0x7f3ab000c8e0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.520+0000 7f3ab7400640 1 -- 172.21.15.27:0/1284107258 shutdown_connections 2024-01-15T18:05:05.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.520+0000 7f3ab7400640 1 --2- 172.21.15.27:0/1284107258 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3aa0066950 0x7f3aa0068e10 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.520+0000 7f3ab7400640 1 --2- 172.21.15.27:0/1284107258 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ac815bdb0 0x7f3ac814ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.520+0000 7f3ab7400640 1 --2- 172.21.15.27:0/1284107258 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ac815d4a0 0x7f3ac815b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.520+0000 7f3ab7400640 1 --2- 172.21.15.27:0/1284107258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ac806c1c0 0x7f3ac82023d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.520+0000 7f3ab7400640 1 -- 172.21.15.27:0/1284107258 >> 172.21.15.27:0/1284107258 conn(0x7f3ac806a6c0 msgr2=0x7f3ac80cb980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:05.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.520+0000 7f3ab7400640 1 -- 172.21.15.27:0/1284107258 shutdown_connections 2024-01-15T18:05:05.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:05.520+0000 7f3ab7400640 1 -- 172.21.15.27:0/1284107258 wait complete. 2024-01-15T18:05:05.533 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.5 2024-01-15T18:05:05.542 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.7 2024-01-15T18:05:06.480 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-15T18:05:06.481 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-15T18:05:06.494 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-15T18:05:06.495 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-15T18:05:06.497 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-15T18:05:06.504 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-15T18:05:06.534 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-15T18:05:06.543 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-15T18:05:06.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.688+0000 7f5e62200640 1 Processor -- start 2024-01-15T18:05:06.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.688+0000 7f5e62200640 1 -- start start 2024-01-15T18:05:06.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.692+0000 7f3749400640 1 Processor -- start 2024-01-15T18:05:06.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.692+0000 7f3749400640 1 -- start start 2024-01-15T18:05:06.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.692+0000 7f5e62200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e5c06dd10 0x7f5e5c06b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.692+0000 7f5e62200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f5e5c06e800 0x7f5e5c0ad640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.692+0000 7f5e62200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5c0b3890 con 0x7f5e5c08aaa0 2024-01-15T18:05:06.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.692+0000 7f5e62200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e5c06aed0 con 0x7f5e5c06e800 2024-01-15T18:05:06.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.692+0000 7f5e62200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c0cc850 con 0x7f5e5c06dd10 2024-01-15T18:05:06.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.692+0000 7f5e60c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f5e5c06e800 0x7f5e5c0ad640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.692+0000 7f5e60c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f5e5c06e800 0x7f5e5c0ad640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:51696/0 (socket says 172.21.15.27:51696) 2024-01-15T18:05:06.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.692+0000 7f5e60c00640 1 -- 172.21.15.27:0/4068372955 learned_addr learned my addr 172.21.15.27:0/4068372955 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:06.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.692+0000 7f5e5b400640 1 --2- 172.21.15.27:0/4068372955 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e5c06dd10 0x7f5e5c06b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.692+0000 7f5e60c00640 1 -- 172.21.15.27:0/4068372955 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e5c06dd10 msgr2=0x7f5e5c06b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:06.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.696+0000 7f5e60c00640 1 --2- 172.21.15.27:0/4068372955 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e5c06dd10 0x7f5e5c06b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.696+0000 7f5e60c00640 1 -- 172.21.15.27:0/4068372955 >> v1:172.21.15.93:6789/0 conn(0x7f5e5c08aaa0 legacy=0x7f5e5c06e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:06.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.696+0000 7f5e60c00640 1 -- 172.21.15.27:0/4068372955 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c06f590 con 0x7f5e5c06e800 2024-01-15T18:05:06.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.696+0000 7f3749400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f373c0e2790 0x7f373c053e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.696+0000 7f3749400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f373c0e34b0 0x7f373c0380f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.696+0000 7f3749400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f373c007210 con 0x7f373c006090 2024-01-15T18:05:06.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.696+0000 7f3749400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f373c037be0 con 0x7f373c0e2790 2024-01-15T18:05:06.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.696+0000 7f3749400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f373c0e2b70 con 0x7f373c0e34b0 2024-01-15T18:05:06.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.696+0000 7f3742800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f373c0e2790 0x7f373c053e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.696+0000 7f3742800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f373c0e2790 0x7f373c053e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59292/0 (socket says 172.21.15.27:59292) 2024-01-15T18:05:06.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.696+0000 7f3742800640 1 -- 172.21.15.27:0/3969714214 learned_addr learned my addr 172.21.15.27:0/3969714214 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:06.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.696+0000 7f3742800640 1 -- 172.21.15.27:0/3969714214 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f373c0e34b0 msgr2=0x7f373c0380f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T18:05:06.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.696+0000 7f3742800640 1 --2- 172.21.15.27:0/3969714214 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f373c0e34b0 0x7f373c0380f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.696+0000 7f3742800640 1 -- 172.21.15.27:0/3969714214 >> v1:172.21.15.93:6789/0 conn(0x7f373c006090 legacy=0x7f373c0e2da0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T18:05:06.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.696+0000 7f3742800640 1 -- 172.21.15.27:0/3969714214 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f373c006600 con 0x7f373c0e2790 2024-01-15T18:05:06.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.700+0000 7f5e60c00640 1 --2- 172.21.15.27:0/4068372955 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5e5c06e800 0x7f5e5c0ad640 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5e44009870 tx=0x7f5e4401c3d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0f1f4b4f62d249b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:06.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.700+0000 7f5e5a000640 1 -- 172.21.15.27:0/4068372955 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5c0ae390 con 0x7f5e5c06e800 2024-01-15T18:05:06.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.700+0000 7f5e5a000640 1 -- 172.21.15.27:0/4068372955 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e440a7050 con 0x7f5e5c06e800 2024-01-15T18:05:06.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.700+0000 7f5e5a000640 1 -- 172.21.15.27:0/4068372955 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e440ab820 con 0x7f5e5c06e800 2024-01-15T18:05:06.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.700+0000 7f5e62200640 1 -- 172.21.15.27:0/4068372955 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5e5c06e800 msgr2=0x7f5e5c0ad640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.700+0000 7f5e62200640 1 --2- 172.21.15.27:0/4068372955 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5e5c06e800 0x7f5e5c0ad640 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5e44009870 tx=0x7f5e4401c3d0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.700+0000 7f5e62200640 1 -- 172.21.15.27:0/4068372955 shutdown_connections 2024-01-15T18:05:06.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.700+0000 7f5e62200640 1 --2- 172.21.15.27:0/4068372955 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5e5c06e800 0x7f5e5c0ad640 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.700+0000 7f5e62200640 1 --2- 172.21.15.27:0/4068372955 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e5c06dd10 0x7f5e5c06b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.700+0000 7f5e62200640 1 -- 172.21.15.27:0/4068372955 >> 172.21.15.27:0/4068372955 conn(0x7f5e5c0721b0 msgr2=0x7f5e5c099a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:06.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.700+0000 7f5e62200640 1 -- 172.21.15.27:0/4068372955 shutdown_connections 2024-01-15T18:05:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.700+0000 7f5e62200640 1 -- 172.21.15.27:0/4068372955 wait complete. 2024-01-15T18:05:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.700+0000 7f3742800640 1 --2- 172.21.15.27:0/3969714214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f373c0e2790 0x7f373c053e90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f37440730c0 tx=0x7f3744096b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2db3f61accf5a6fc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.704+0000 7f3741400640 1 -- 172.21.15.27:0/3969714214 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374411f960 con 0x7f373c0e2790 2024-01-15T18:05:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.704+0000 7f3741400640 1 -- 172.21.15.27:0/3969714214 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f374411fac0 con 0x7f373c0e2790 2024-01-15T18:05:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.704+0000 7f3741400640 1 -- 172.21.15.27:0/3969714214 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3744127440 con 0x7f373c0e2790 2024-01-15T18:05:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.704+0000 7f3749400640 1 -- 172.21.15.27:0/3969714214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f373c0e2790 msgr2=0x7f373c053e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.704+0000 7f3749400640 1 --2- 172.21.15.27:0/3969714214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f373c0e2790 0x7f373c053e90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f37440730c0 tx=0x7f3744096b00 comp rx=0 tx=0).stop 2024-01-15T18:05:06.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.704+0000 7f5e62200640 1 Processor -- start 2024-01-15T18:05:06.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.704+0000 7f5e62200640 1 -- start start 2024-01-15T18:05:06.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.704+0000 7f3749400640 1 -- 172.21.15.27:0/3969714214 shutdown_connections 2024-01-15T18:05:06.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.704+0000 7f3749400640 1 --2- 172.21.15.27:0/3969714214 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f373c0e34b0 0x7f373c0380f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.704+0000 7f3749400640 1 --2- 172.21.15.27:0/3969714214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f373c0e2790 0x7f373c053e90 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.704+0000 7f3749400640 1 -- 172.21.15.27:0/3969714214 >> 172.21.15.27:0/3969714214 conn(0x7f373c054c70 msgr2=0x7f373c055080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:06.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.708+0000 7f5e62200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e5c06dd10 0x7f5e5c16ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.708+0000 7f5e62200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5e5c06e800 0x7f5e5c16b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.708+0000 7f5e62200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e5c08aaa0 0x7f5e5c0873d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.708+0000 7f5e62200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c16b9b0 con 0x7f5e5c08aaa0 2024-01-15T18:05:06.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.708+0000 7f5e62200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c16bc90 con 0x7f5e5c06e800 2024-01-15T18:05:06.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.708+0000 7f5e62200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c16bf70 con 0x7f5e5c06dd10 2024-01-15T18:05:06.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.708+0000 7f3749400640 1 -- 172.21.15.27:0/3969714214 shutdown_connections 2024-01-15T18:05:06.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.708+0000 7f3749400640 1 -- 172.21.15.27:0/3969714214 wait complete. 2024-01-15T18:05:06.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.708+0000 7f5e5b400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e5c06dd10 0x7f5e5c16ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.708+0000 7f5e5b400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e5c06dd10 0x7f5e5c16ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:42796/0 (socket says 172.21.15.27:42796) 2024-01-15T18:05:06.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.708+0000 7f5e5b400640 1 -- 172.21.15.27:0/3017366036 learned_addr learned my addr 172.21.15.27:0/3017366036 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:06.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.708+0000 7f5e5aa00640 1 --2- 172.21.15.27:0/3017366036 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5e5c06e800 0x7f5e5c16b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.708+0000 7f5e60c00640 1 --2- 172.21.15.27:0/3017366036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e5c08aaa0 0x7f5e5c0873d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.708+0000 7f5e5aa00640 1 -- 172.21.15.27:0/3017366036 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e5c06dd10 msgr2=0x7f5e5c16ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.708+0000 7f5e5aa00640 1 --2- 172.21.15.27:0/3017366036 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e5c06dd10 0x7f5e5c16ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.708+0000 7f5e5aa00640 1 -- 172.21.15.27:0/3017366036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e5c08aaa0 msgr2=0x7f5e5c0873d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.708+0000 7f5e5aa00640 1 --2- 172.21.15.27:0/3017366036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e5c08aaa0 0x7f5e5c0873d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.708+0000 7f5e5aa00640 1 -- 172.21.15.27:0/3017366036 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e440094d0 con 0x7f5e5c06e800 2024-01-15T18:05:06.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.708+0000 7f5e5aa00640 1 --2- 172.21.15.27:0/3017366036 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5e5c06e800 0x7f5e5c16b320 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c00c780 tx=0x7f5e4c00cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:06.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.708+0000 7f5e53400640 1 -- 172.21.15.27:0/3017366036 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c003d90 con 0x7f5e5c06e800 2024-01-15T18:05:06.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.708+0000 7f5e53400640 1 -- 172.21.15.27:0/3017366036 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e4c00ce30 con 0x7f5e5c06e800 2024-01-15T18:05:06.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.708+0000 7f5e53400640 1 -- 172.21.15.27:0/3017366036 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c005290 con 0x7f5e5c06e800 2024-01-15T18:05:06.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.712+0000 7f5e62200640 1 -- 172.21.15.27:0/3017366036 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5c16c250 con 0x7f5e5c06e800 2024-01-15T18:05:06.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.712+0000 7f5e62200640 1 -- 172.21.15.27:0/3017366036 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5c207d30 con 0x7f5e5c06e800 2024-01-15T18:05:06.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.712+0000 7f3749400640 1 Processor -- start 2024-01-15T18:05:06.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.712+0000 7f3749400640 1 -- start start 2024-01-15T18:05:06.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.716+0000 7f3749400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f373c006090 0x7f373c0cd760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.716+0000 7f5e53400640 1 -- 172.21.15.27:0/3017366036 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f5e4c0053f0 con 0x7f5e5c06e800 2024-01-15T18:05:06.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.716+0000 7f5e50c00640 1 -- 172.21.15.27:0/3017366036 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e24005260 con 0x7f5e5c06e800 2024-01-15T18:05:06.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.716+0000 7f3749400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f373c0e2790 0x7f373c0db3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.716+0000 7f3749400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f373c0e34b0 0x7f373c0db920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.716+0000 7f3749400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f373c0cdca0 con 0x7f373c006090 2024-01-15T18:05:06.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.716+0000 7f5e53400640 1 --2- 172.21.15.27:0/3017366036 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5e34066940 0x7f5e34068e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.716+0000 7f3749400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f373c037a60 con 0x7f373c0e34b0 2024-01-15T18:05:06.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.716+0000 7f3749400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f373c0d3ed0 con 0x7f373c0e2790 2024-01-15T18:05:06.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.716+0000 7f3741e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f373c0e2790 0x7f373c0db3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.716+0000 7f3748a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f373c0e34b0 0x7f373c0db920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.716+0000 7f3741e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f373c0e2790 0x7f373c0db3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:42812/0 (socket says 172.21.15.27:42812) 2024-01-15T18:05:06.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.716+0000 7f3741e00640 1 -- 172.21.15.27:0/251453535 learned_addr learned my addr 172.21.15.27:0/251453535 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:06.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.716+0000 7f3742800640 1 --2- 172.21.15.27:0/251453535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f373c006090 0x7f373c0cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.716+0000 7f5e53400640 1 -- 172.21.15.27:0/3017366036 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e4c054b70 con 0x7f5e5c06e800 2024-01-15T18:05:06.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.716+0000 7f3748a00640 1 -- 172.21.15.27:0/251453535 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f373c0e2790 msgr2=0x7f373c0db3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.716+0000 7f3748a00640 1 --2- 172.21.15.27:0/251453535 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f373c0e2790 0x7f373c0db3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.716+0000 7f3748a00640 1 -- 172.21.15.27:0/251453535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f373c006090 msgr2=0x7f373c0cd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.716+0000 7f3748a00640 1 --2- 172.21.15.27:0/251453535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f373c006090 0x7f373c0cd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.716+0000 7f3748a00640 1 -- 172.21.15.27:0/251453535 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3744068640 con 0x7f373c0e34b0 2024-01-15T18:05:06.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.716+0000 7f5e5b400640 1 --2- 172.21.15.27:0/3017366036 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5e34066940 0x7f5e34068e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.716+0000 7f3748a00640 1 --2- 172.21.15.27:0/251453535 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f373c0e34b0 0x7f373c0db920 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f373c00df40 tx=0x7f373400b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:06.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.716+0000 7f5e5b400640 1 --2- 172.21.15.27:0/3017366036 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5e34066940 0x7f5e34068e00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5e5400b240 tx=0x7f5e5400b3a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:06.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.720+0000 7f373aa00640 1 -- 172.21.15.27:0/251453535 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3734011830 con 0x7f373c0e34b0 2024-01-15T18:05:06.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.720+0000 7f373aa00640 1 -- 172.21.15.27:0/251453535 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37340119b0 con 0x7f373c0e34b0 2024-01-15T18:05:06.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.720+0000 7f373aa00640 1 -- 172.21.15.27:0/251453535 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3734011d60 con 0x7f373c0e34b0 2024-01-15T18:05:06.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.720+0000 7f5e53400640 1 -- 172.21.15.27:0/3017366036 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5e4c051640 con 0x7f5e5c06e800 2024-01-15T18:05:06.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.724+0000 7f3749400640 1 -- 172.21.15.27:0/251453535 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f373c0d3620 con 0x7f373c0e34b0 2024-01-15T18:05:06.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.724+0000 7f3749400640 1 -- 172.21.15.27:0/251453535 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f373c037780 con 0x7f373c0e34b0 2024-01-15T18:05:06.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.724+0000 7f3749400640 1 -- 172.21.15.27:0/251453535 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f373c0d4060 con 0x7f373c0e34b0 2024-01-15T18:05:06.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.724+0000 7f373aa00640 1 -- 172.21.15.27:0/251453535 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f373401a400 con 0x7f373c0e34b0 2024-01-15T18:05:06.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.728+0000 7f373aa00640 1 --2- 172.21.15.27:0/251453535 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f371c066940 0x7f371c068e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.728+0000 7f373aa00640 1 -- 172.21.15.27:0/251453535 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37340842b0 con 0x7f373c0e34b0 2024-01-15T18:05:06.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.732+0000 7f3742800640 1 --2- 172.21.15.27:0/251453535 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f371c066940 0x7f371c068e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.732+0000 7f373aa00640 1 -- 172.21.15.27:0/251453535 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f37340515d0 con 0x7f373c0e34b0 2024-01-15T18:05:06.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.736+0000 7f3742800640 1 --2- 172.21.15.27:0/251453535 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f371c066940 0x7f371c068e00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f373c055d70 tx=0x7f3744121070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:06.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.760+0000 7f52d6800640 1 Processor -- start 2024-01-15T18:05:06.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.764+0000 7f52d6800640 1 -- start start 2024-01-15T18:05:06.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.764+0000 7f52d6800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f52c8006090 0x7f52c80e2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.764+0000 7f52d6800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f52c80076a0 0x7f52c80e32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.764+0000 7f52d6800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52c8007210 con 0x7f52c80e2790 2024-01-15T18:05:06.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.764+0000 7f52d6800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c8037be0 con 0x7f52c8006090 2024-01-15T18:05:06.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.764+0000 7f52d6800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c80e2b70 con 0x7f52c80076a0 2024-01-15T18:05:06.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.764+0000 7f52cf400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f52c80076a0 0x7f52c80e32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.764+0000 7f52cf400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f52c80076a0 0x7f52c80e32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:42822/0 (socket says 172.21.15.27:42822) 2024-01-15T18:05:06.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.764+0000 7f52cf400640 1 -- 172.21.15.27:0/3983054985 learned_addr learned my addr 172.21.15.27:0/3983054985 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:06.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.768+0000 7f52cea00640 1 --2- 172.21.15.27:0/3983054985 >> v2:172.21.15.93:3300/0 conn(0x7f52c8006090 0x7f52c80e2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.768+0000 7f52cf400640 1 -- 172.21.15.27:0/3983054985 >> v2:172.21.15.93:3300/0 conn(0x7f52c8006090 msgr2=0x7f52c80e2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:06.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.768+0000 7f52cf400640 1 --2- 172.21.15.27:0/3983054985 >> v2:172.21.15.93:3300/0 conn(0x7f52c8006090 0x7f52c80e2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.768+0000 7f52cf400640 1 -- 172.21.15.27:0/3983054985 >> v1:172.21.15.27:6789/0 conn(0x7f52c80e2790 legacy=0x7f52c8053e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:06.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.768+0000 7f52cf400640 1 -- 172.21.15.27:0/3983054985 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c8006600 con 0x7f52c80076a0 2024-01-15T18:05:06.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.772+0000 7f982aa00640 1 Processor -- start 2024-01-15T18:05:06.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.772+0000 7f982aa00640 1 -- start start 2024-01-15T18:05:06.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.772+0000 7f982aa00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f981c0544d0 0x7f981c005ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.772+0000 7f9823e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f981c0e30a0 0x7f981c04fa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42386/0 (socket says 172.21.15.27:42386) 2024-01-15T18:05:06.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.772+0000 7f9823e00640 1 -- 172.21.15.27:0/4286372007 learned_addr learned my addr 172.21.15.27:0/4286372007 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:06.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.772+0000 7f982aa00640 1 -- 172.21.15.27:0/4286372007 --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f981c037b90 con 0x7f981c0e30a0 2024-01-15T18:05:06.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.772+0000 7f9823400640 1 --2- 172.21.15.27:0/4286372007 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f981c0544d0 0x7f981c005ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.772+0000 7f52cf400640 1 --2- 172.21.15.27:0/3983054985 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f52c80076a0 0x7f52c80e32e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f52c00088a0 tx=0x7f52c0020bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90dbb1924b62b2a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:06.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f982aa00640 1 -- 172.21.15.27:0/4286372007 --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f981c054290 con 0x7f981c04f2d0 2024-01-15T18:05:06.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f982aa00640 1 -- 172.21.15.27:0/4286372007 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f981c050180 con 0x7f981c0544d0 2024-01-15T18:05:06.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f9822a00640 1 -- 172.21.15.27:0/4286372007 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046791442 0 0) 0x7f981c037b90 con 0x7f981c0e30a0 2024-01-15T18:05:06.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f9822a00640 1 -- 172.21.15.27:0/4286372007 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9804003530 con 0x7f981c0e30a0 2024-01-15T18:05:06.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f52ce000640 1 -- 172.21.15.27:0/3983054985 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c00a9c60 con 0x7f52c80076a0 2024-01-15T18:05:06.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f52ce000640 1 -- 172.21.15.27:0/3983054985 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c00ad070 con 0x7f52c80076a0 2024-01-15T18:05:06.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f9822a00640 1 -- 172.21.15.27:0/4286372007 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 424735959 0 0) 0x7f9804003530 con 0x7f981c0e30a0 2024-01-15T18:05:06.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f9822a00640 1 -- 172.21.15.27:0/4286372007 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f981c0544d0 msgr2=0x7f981c005ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:06.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f9822a00640 1 --2- 172.21.15.27:0/4286372007 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f981c0544d0 0x7f981c005ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f9822a00640 1 -- 172.21.15.27:0/4286372007 >> v1:172.21.15.93:6789/0 conn(0x7f981c04f2d0 legacy=0x7f981c0e3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f9822a00640 1 -- 172.21.15.27:0/4286372007 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981c0e7820 con 0x7f981c0e30a0 2024-01-15T18:05:06.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f9822a00640 1 -- 172.21.15.27:0/4286372007 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f9814001680 con 0x7f981c0e30a0 2024-01-15T18:05:06.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f9822a00640 1 -- 172.21.15.27:0/4286372007 >> v1:172.21.15.27:6789/0 conn(0x7f981c0e30a0 legacy=0x7f981c04fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f9822a00640 1 --2- 172.21.15.27:0/4286372007 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f98040039b0 0x7f9804003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f9822a00640 1 --2- 172.21.15.27:0/4286372007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98040047d0 0x7f9804004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f9822a00640 1 --2- 172.21.15.27:0/4286372007 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9804005190 0x7f980400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f9822a00640 1 -- 172.21.15.27:0/4286372007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c050180 con 0x7f98040047d0 2024-01-15T18:05:06.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f9822a00640 1 -- 172.21.15.27:0/4286372007 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980400bfe0 con 0x7f9804005190 2024-01-15T18:05:06.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f9822a00640 1 -- 172.21.15.27:0/4286372007 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980400c1a0 con 0x7f98040039b0 2024-01-15T18:05:06.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f52d6800640 1 -- 172.21.15.27:0/3983054985 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f52c80076a0 msgr2=0x7f52c80e32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f9828a00640 1 --2- 172.21.15.27:0/4286372007 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9804005190 0x7f980400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f52ce000640 1 -- 172.21.15.27:0/3983054985 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c00b3400 con 0x7f52c80076a0 2024-01-15T18:05:06.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f52d6800640 1 --2- 172.21.15.27:0/3983054985 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f52c80076a0 0x7f52c80e32e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f52c00088a0 tx=0x7f52c0020bd0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f9828a00640 1 -- 172.21.15.27:0/4286372007 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f98040039b0 msgr2=0x7f9804003dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-15T18:05:06.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f9828a00640 1 --2- 172.21.15.27:0/4286372007 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f98040039b0 0x7f9804003dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f9828a00640 1 -- 172.21.15.27:0/4286372007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98040047d0 msgr2=0x7f9804004c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-15T18:05:06.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f9828a00640 1 --2- 172.21.15.27:0/4286372007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98040047d0 0x7f9804004c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f9828a00640 1 -- 172.21.15.27:0/4286372007 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9804003530 con 0x7f9804005190 2024-01-15T18:05:06.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.776+0000 7f9823400640 1 --2- 172.21.15.27:0/4286372007 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f981c0544d0 0x7f981c005ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:06.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.780+0000 7f52d6800640 1 -- 172.21.15.27:0/3983054985 shutdown_connections 2024-01-15T18:05:06.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.780+0000 7f52d6800640 1 --2- 172.21.15.27:0/3983054985 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f52c80076a0 0x7f52c80e32e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.780+0000 7f52d6800640 1 --2- 172.21.15.27:0/3983054985 >> v2:172.21.15.93:3300/0 conn(0x7f52c8006090 0x7f52c80e2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.780+0000 7f52d6800640 1 -- 172.21.15.27:0/3983054985 >> 172.21.15.27:0/3983054985 conn(0x7f52c8054c70 msgr2=0x7f52c8055080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:06.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.780+0000 7f52d6800640 1 -- 172.21.15.27:0/3983054985 shutdown_connections 2024-01-15T18:05:06.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.780+0000 7f52d6800640 1 -- 172.21.15.27:0/3983054985 wait complete. 2024-01-15T18:05:06.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.780+0000 7f9828a00640 1 --2- 172.21.15.27:0/4286372007 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9804005190 0x7f980400b990 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9824068e40 tx=0x7f9824096ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4ee698cccf17722 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:06.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.780+0000 7f9822a00640 1 -- 172.21.15.27:0/4286372007 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982411f8d0 con 0x7f9804005190 2024-01-15T18:05:06.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.780+0000 7f9822a00640 1 -- 172.21.15.27:0/4286372007 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982411fa50 con 0x7f9804005190 2024-01-15T18:05:06.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.780+0000 7f9822a00640 1 -- 172.21.15.27:0/4286372007 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982411fe00 con 0x7f9804005190 2024-01-15T18:05:06.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.784+0000 7f982aa00640 1 -- 172.21.15.27:0/4286372007 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9804005190 msgr2=0x7f980400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.784+0000 7f982aa00640 1 --2- 172.21.15.27:0/4286372007 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9804005190 0x7f980400b990 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9824068e40 tx=0x7f9824096ac0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.784+0000 7f9828a00640 1 -- 172.21.15.27:0/4286372007 reap_dead start 2024-01-15T18:05:06.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.784+0000 7f52d6800640 1 Processor -- start 2024-01-15T18:05:06.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.784+0000 7fc083a00640 1 Processor -- start 2024-01-15T18:05:06.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.784+0000 7fc083a00640 1 -- start start 2024-01-15T18:05:06.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.784+0000 7f982aa00640 1 -- 172.21.15.27:0/4286372007 shutdown_connections 2024-01-15T18:05:06.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.784+0000 7f982aa00640 1 -- 172.21.15.27:0/4286372007 >> 172.21.15.27:0/4286372007 conn(0x7f981c009f90 msgr2=0x7f981c00a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:06.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.784+0000 7f982aa00640 1 -- 172.21.15.27:0/4286372007 shutdown_connections 2024-01-15T18:05:06.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.784+0000 7f982aa00640 1 -- 172.21.15.27:0/4286372007 wait complete. 2024-01-15T18:05:06.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.784+0000 7f52d6800640 1 -- start start 2024-01-15T18:05:06.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.784+0000 7f52d6800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f52c8006090 0x7f52c80cda30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.784+0000 7f52d6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c80076a0 0x7f52c80cbe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.784+0000 7f52d6800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52c80e2790 0x7f52c80d9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.784+0000 7f52d6800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c80ce8c0 con 0x7f52c80076a0 2024-01-15T18:05:06.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.784+0000 7f52d6800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c80d1bc0 con 0x7f52c80e2790 2024-01-15T18:05:06.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.784+0000 7f52d6800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c80d9830 con 0x7f52c8006090 2024-01-15T18:05:06.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7fc083a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fc07c157b30 0x7fc07c15d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f52d5200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f52c8006090 0x7f52c80cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f52cea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c80076a0 0x7f52c80cbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f52d5200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f52c8006090 0x7f52c80cda30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:42830/0 (socket says 172.21.15.27:42830) 2024-01-15T18:05:06.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f52d5200640 1 -- 172.21.15.27:0/368879889 learned_addr learned my addr 172.21.15.27:0/368879889 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:06.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f52cf400640 1 --2- 172.21.15.27:0/368879889 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52c80e2790 0x7f52c80d9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f982aa00640 1 Processor -- start 2024-01-15T18:05:06.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7fc083a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc07c06fa20 0x7fc07c0b4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7fc083a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc07c06a4f0 con 0x7fc07c06ef30 2024-01-15T18:05:06.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7fc083a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc07c06fe50 con 0x7fc07c06fa20 2024-01-15T18:05:06.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7fc083a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc07c0afff0 con 0x7fc07c157b30 2024-01-15T18:05:06.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f982aa00640 1 -- start start 2024-01-15T18:05:06.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f52cf400640 1 -- 172.21.15.27:0/368879889 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f52c8006090 msgr2=0x7f52c80cda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f52cf400640 1 --2- 172.21.15.27:0/368879889 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f52c8006090 0x7f52c80cda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f52cf400640 1 -- 172.21.15.27:0/368879889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c80076a0 msgr2=0x7f52c80cbe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f52cf400640 1 --2- 172.21.15.27:0/368879889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c80076a0 0x7f52c80cbe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f52cf400640 1 -- 172.21.15.27:0/368879889 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c0008550 con 0x7f52c80e2790 2024-01-15T18:05:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7fc081000640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7fc07c06ef30 0x7fc07c06f310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:57106/0 (socket says 172.21.15.27:57106) 2024-01-15T18:05:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7fc081000640 1 -- 172.21.15.27:0/3072912645 learned_addr learned my addr 172.21.15.27:0/3072912645 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f52cea00640 1 --2- 172.21.15.27:0/368879889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c80076a0 0x7f52c80cbe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7fc082400640 1 --2- 172.21.15.27:0/3072912645 >> v2:172.21.15.27:3300/0 conn(0x7fc07c06fa20 0x7fc07c0b4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f52cf400640 1 --2- 172.21.15.27:0/368879889 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52c80e2790 0x7f52c80d9ff0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f52c00b1000 tx=0x7f52c001f410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f982aa00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9804005190 0x7f981c0dd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f982aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f981c0dc1f0 0x7f981c04cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f982aa00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f981c185710 0x7f981c04d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f982aa00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c0de570 con 0x7f981c0dc1f0 2024-01-15T18:05:06.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f982aa00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c0dd960 con 0x7f981c185710 2024-01-15T18:05:06.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f982aa00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f981c0ddc40 con 0x7f9804005190 2024-01-15T18:05:06.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7fc07b400640 1 -- 172.21.15.27:0/3072912645 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99903956 0 0) 0x7fc07c06a4f0 con 0x7fc07c06ef30 2024-01-15T18:05:06.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7fc07b400640 1 -- 172.21.15.27:0/3072912645 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc05c003530 con 0x7fc07c06ef30 2024-01-15T18:05:06.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f52c7400640 1 -- 172.21.15.27:0/368879889 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c00c2070 con 0x7f52c80e2790 2024-01-15T18:05:06.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f52c7400640 1 -- 172.21.15.27:0/368879889 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c00b39c0 con 0x7f52c80e2790 2024-01-15T18:05:06.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f52c7400640 1 -- 172.21.15.27:0/368879889 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c00c2070 con 0x7f52c80e2790 2024-01-15T18:05:06.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7fc07be00640 1 --2- 172.21.15.27:0/3072912645 >> v2:172.21.15.93:3300/0 conn(0x7fc07c157b30 0x7fc07c15d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f52d6800640 1 -- 172.21.15.27:0/368879889 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c80d1600 con 0x7f52c80e2790 2024-01-15T18:05:06.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f9828a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f981c185710 0x7f981c04d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.788+0000 7f9823e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9804005190 0x7f981c0dd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7f9823e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9804005190 0x7f981c0dd0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:42834/0 (socket says 172.21.15.27:42834) 2024-01-15T18:05:06.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7f9823e00640 1 -- 172.21.15.27:0/756265673 learned_addr learned my addr 172.21.15.27:0/756265673 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:06.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7f52d6800640 1 -- 172.21.15.27:0/368879889 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c80d9ad0 con 0x7f52c80e2790 2024-01-15T18:05:06.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7f9828a00640 1 -- 172.21.15.27:0/756265673 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9804005190 msgr2=0x7f981c0dd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7fc07b400640 1 -- 172.21.15.27:0/3072912645 <== mon.1 v1:172.21.15.93:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2910064227 0 0) 0x7fc05c003530 con 0x7fc07c06ef30 2024-01-15T18:05:06.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7fc07b400640 1 -- 172.21.15.27:0/3072912645 >> v2:172.21.15.93:3300/0 conn(0x7fc07c157b30 msgr2=0x7fc07c15d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:06.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7f9823400640 1 --2- 172.21.15.27:0/756265673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f981c0dc1f0 0x7f981c04cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7fc07b400640 1 --2- 172.21.15.27:0/3072912645 >> v2:172.21.15.93:3300/0 conn(0x7fc07c157b30 0x7fc07c15d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7fc07b400640 1 -- 172.21.15.27:0/3072912645 >> v2:172.21.15.27:3300/0 conn(0x7fc07c06fa20 msgr2=0x7fc07c0b4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:06.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7fc07b400640 1 --2- 172.21.15.27:0/3072912645 >> v2:172.21.15.27:3300/0 conn(0x7fc07c06fa20 0x7fc07c0b4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7f9828a00640 1 --2- 172.21.15.27:0/756265673 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9804005190 0x7f981c0dd0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7fc07b400640 1 -- 172.21.15.27:0/3072912645 --> v1:172.21.15.93:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07c06a7d0 con 0x7fc07c06ef30 2024-01-15T18:05:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7f9828a00640 1 -- 172.21.15.27:0/756265673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f981c0dc1f0 msgr2=0x7f981c04cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7f9828a00640 1 --2- 172.21.15.27:0/756265673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f981c0dc1f0 0x7f981c04cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7f9828a00640 1 -- 172.21.15.27:0/756265673 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981c0ddf20 con 0x7f981c185710 2024-01-15T18:05:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7fc07b400640 1 -- 172.21.15.27:0/3072912645 <== mon.1 v1:172.21.15.93:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fc06c001660 con 0x7fc07c06ef30 2024-01-15T18:05:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7fc07b400640 1 -- 172.21.15.27:0/3072912645 >> v1:172.21.15.93:6789/0 conn(0x7fc07c06ef30 legacy=0x7fc07c06f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7fc07b400640 1 --2- 172.21.15.27:0/3072912645 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc05c003a00 0x7fc05c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7fc07b400640 1 --2- 172.21.15.27:0/3072912645 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc05c0047d0 0x7fc05c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7fc07b400640 1 --2- 172.21.15.27:0/3072912645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc05c005200 0x7fc05c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7fc07b400640 1 -- 172.21.15.27:0/3072912645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07c06fe50 con 0x7fc05c005200 2024-01-15T18:05:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7fc07b400640 1 -- 172.21.15.27:0/3072912645 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07c0afff0 con 0x7fc05c003a00 2024-01-15T18:05:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7fc07b400640 1 -- 172.21.15.27:0/3072912645 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc05c00c1e0 con 0x7fc05c0047d0 2024-01-15T18:05:06.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7fc082400640 1 --2- 172.21.15.27:0/3072912645 >> v2:172.21.15.27:3300/0 conn(0x7fc07c06fa20 0x7fc07c0b4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:06.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7f9823e00640 1 --2- 172.21.15.27:0/756265673 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9804005190 0x7f981c0dd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:06.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7f9823400640 1 --2- 172.21.15.27:0/756265673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f981c0dc1f0 0x7f981c04cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:06.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7fc081000640 1 --2- 172.21.15.27:0/3072912645 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc05c003a00 0x7fc05c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7fc07be00640 1 --2- 172.21.15.27:0/3072912645 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc05c0047d0 0x7fc05c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7f9828a00640 1 --2- 172.21.15.27:0/756265673 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f981c185710 0x7f981c04d1c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9824050190 tx=0x7f9804005800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:06.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.796+0000 7fc07be00640 1 -- 172.21.15.27:0/3072912645 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc05c003a00 msgr2=0x7fc05c003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:06.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.796+0000 7fc07be00640 1 --2- 172.21.15.27:0/3072912645 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc05c003a00 0x7fc05c003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.796+0000 7fc07be00640 1 -- 172.21.15.27:0/3072912645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc05c005200 msgr2=0x7fc05c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:06.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.796+0000 7fc082400640 1 --2- 172.21.15.27:0/3072912645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc05c005200 0x7fc05c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.796+0000 7f981be00640 1 -- 172.21.15.27:0/756265673 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9824127650 con 0x7f981c185710 2024-01-15T18:05:06.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.796+0000 7fc07be00640 1 --2- 172.21.15.27:0/3072912645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc05c005200 0x7fc05c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.796+0000 7fc07be00640 1 -- 172.21.15.27:0/3072912645 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc05c003530 con 0x7fc05c0047d0 2024-01-15T18:05:06.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.796+0000 7f981be00640 1 -- 172.21.15.27:0/756265673 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98241277b0 con 0x7f981c185710 2024-01-15T18:05:06.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.796+0000 7f981be00640 1 -- 172.21.15.27:0/756265673 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9824127650 con 0x7f981c185710 2024-01-15T18:05:06.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.796+0000 7f982aa00640 1 -- 172.21.15.27:0/756265673 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f981c0de200 con 0x7f981c185710 2024-01-15T18:05:06.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.796+0000 7f982aa00640 1 -- 172.21.15.27:0/756265673 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f981c04bb70 con 0x7f981c185710 2024-01-15T18:05:06.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.792+0000 7f52c7400640 1 -- 172.21.15.27:0/368879889 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f52c00b3b30 con 0x7f52c80e2790 2024-01-15T18:05:06.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.796+0000 7f52d6800640 1 -- 172.21.15.27:0/368879889 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c8008b80 con 0x7f52c80e2790 2024-01-15T18:05:06.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.796+0000 7f52c7400640 1 --2- 172.21.15.27:0/368879889 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f52a8066950 0x7f52a8068e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.796+0000 7f52c7400640 1 -- 172.21.15.27:0/368879889 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f52c0122730 con 0x7f52c80e2790 2024-01-15T18:05:06.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.796+0000 7f52d5200640 1 --2- 172.21.15.27:0/368879889 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f52a8066950 0x7f52a8068e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.796+0000 7f982aa00640 1 -- 172.21.15.27:0/756265673 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f981c0de390 con 0x7f981c185710 2024-01-15T18:05:06.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.800+0000 7f981be00640 1 -- 172.21.15.27:0/756265673 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f982412f070 con 0x7f981c185710 2024-01-15T18:05:06.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.800+0000 7f52d5200640 1 --2- 172.21.15.27:0/368879889 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f52a8066950 0x7f52a8068e10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f52c80d6260 tx=0x7f52b80090e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:06.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.800+0000 7fc081000640 1 --2- 172.21.15.27:0/3072912645 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc05c003a00 0x7fc05c003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:06.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.800+0000 7fc082400640 1 --2- 172.21.15.27:0/3072912645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc05c005200 0x7fc05c00b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:06.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.800+0000 7fc082400640 1 -- 172.21.15.27:0/3072912645 reap_dead start 2024-01-15T18:05:06.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.800+0000 7f516e400640 1 Processor -- start 2024-01-15T18:05:06.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.800+0000 7f516e400640 1 -- start start 2024-01-15T18:05:06.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.800+0000 7f981be00640 1 --2- 172.21.15.27:0/756265673 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f97fc0669e0 0x7f97fc068ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.800+0000 7f981be00640 1 -- 172.21.15.27:0/756265673 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98241978a0 con 0x7f981c185710 2024-01-15T18:05:06.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.800+0000 7f9823e00640 1 --2- 172.21.15.27:0/756265673 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f97fc0669e0 0x7f97fc068ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.800+0000 7f516e400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f516806c0a0 0x7f5168165ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.800+0000 7f516e400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f516806b190 0x7f5168166000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.800+0000 7fc07be00640 1 --2- 172.21.15.27:0/3072912645 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc05c0047d0 0x7fc05c004c50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc06400ba30 tx=0x7fc06401c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70d37029cf85a61e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:06.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.800+0000 7f516e400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51680725d0 con 0x7f516806c480 2024-01-15T18:05:06.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.800+0000 7f516e400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51680d2c00 con 0x7f516806c0a0 2024-01-15T18:05:06.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.800+0000 7f516e400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5168072a60 con 0x7f516806b190 2024-01-15T18:05:06.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.800+0000 7f52c7400640 1 -- 172.21.15.27:0/368879889 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f52c00efaa0 con 0x7f52c80e2790 2024-01-15T18:05:06.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.800+0000 7f5167400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f516806c0a0 0x7f5168165ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.800+0000 7f5167400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f516806c0a0 0x7f5168165ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59350/0 (socket says 172.21.15.27:59350) 2024-01-15T18:05:06.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.800+0000 7f5167400640 1 -- 172.21.15.27:0/1257110312 learned_addr learned my addr 172.21.15.27:0/1257110312 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:06.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.800+0000 7fc07b400640 1 -- 172.21.15.27:0/3072912645 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0640a9660 con 0x7fc05c0047d0 2024-01-15T18:05:06.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.800+0000 7f5166a00640 1 --2- 172.21.15.27:0/1257110312 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f516806b190 0x7f5168166000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.800+0000 7fc07b400640 1 -- 172.21.15.27:0/3072912645 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0640a97c0 con 0x7fc05c0047d0 2024-01-15T18:05:06.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.804+0000 7f9823e00640 1 --2- 172.21.15.27:0/756265673 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f97fc0669e0 0x7f97fc068ea0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9814001ec0 tx=0x7f9814001a50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:06.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.804+0000 7f5167400640 1 -- 172.21.15.27:0/1257110312 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f516806b190 msgr2=0x7f5168166000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:06.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.804+0000 7f5167400640 1 --2- 172.21.15.27:0/1257110312 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f516806b190 0x7f5168166000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.804+0000 7f5167400640 1 -- 172.21.15.27:0/1257110312 >> v1:172.21.15.93:6789/0 conn(0x7f516806c480 legacy=0x7f51680d1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:06.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.804+0000 7f5167400640 1 -- 172.21.15.27:0/1257110312 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516809c6e0 con 0x7f516806c0a0 2024-01-15T18:05:06.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.804+0000 7fc083a00640 1 -- 172.21.15.27:0/3072912645 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc05c0047d0 msgr2=0x7fc05c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.804+0000 7fc07b400640 1 -- 172.21.15.27:0/3072912645 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0640a9660 con 0x7fc05c0047d0 2024-01-15T18:05:06.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.804+0000 7fc083a00640 1 --2- 172.21.15.27:0/3072912645 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc05c0047d0 0x7fc05c004c50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc06400ba30 tx=0x7fc06401c6e0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.804+0000 7fc083a00640 1 -- 172.21.15.27:0/3072912645 shutdown_connections 2024-01-15T18:05:06.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.804+0000 7fc083a00640 1 --2- 172.21.15.27:0/3072912645 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc05c0047d0 0x7fc05c004c50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.804+0000 7fc083a00640 1 -- 172.21.15.27:0/3072912645 >> 172.21.15.27:0/3072912645 conn(0x7fc07c0721b0 msgr2=0x7fc07c099a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:06.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.804+0000 7fc083a00640 1 -- 172.21.15.27:0/3072912645 shutdown_connections 2024-01-15T18:05:06.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.804+0000 7fc083a00640 1 -- 172.21.15.27:0/3072912645 wait complete. 2024-01-15T18:05:06.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.804+0000 7f981be00640 1 -- 172.21.15.27:0/756265673 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9824164a80 con 0x7f981c185710 2024-01-15T18:05:06.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.808+0000 7f5167400640 1 --2- 172.21.15.27:0/1257110312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f516806c0a0 0x7f5168165ac0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f51500098a0 tx=0x7f515001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c44601288ed76906 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:06.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.808+0000 7fc083a00640 1 Processor -- start 2024-01-15T18:05:06.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.808+0000 7fc083a00640 1 -- start start 2024-01-15T18:05:06.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.808+0000 7f5166000640 1 -- 172.21.15.27:0/1257110312 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51500ab600 con 0x7f516806c0a0 2024-01-15T18:05:06.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.812+0000 7f5166000640 1 -- 172.21.15.27:0/1257110312 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51500ab760 con 0x7f516806c0a0 2024-01-15T18:05:06.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.812+0000 7f5166000640 1 -- 172.21.15.27:0/1257110312 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51500abaf0 con 0x7f516806c0a0 2024-01-15T18:05:06.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.812+0000 7fc083a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc05c0047d0 0x7fc07c153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.812+0000 7fc083a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc07c09ceb0 0x7fc07c0cc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.812+0000 7f516e400640 1 -- 172.21.15.27:0/1257110312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f516806c0a0 msgr2=0x7f5168165ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.812+0000 7fc081000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc05c0047d0 0x7fc07c153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.812+0000 7fc081000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc05c0047d0 0x7fc07c153f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:42852/0 (socket says 172.21.15.27:42852) 2024-01-15T18:05:06.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.812+0000 7fc081000640 1 -- 172.21.15.27:0/4108521706 learned_addr learned my addr 172.21.15.27:0/4108521706 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:06.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.812+0000 7f516e400640 1 --2- 172.21.15.27:0/1257110312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f516806c0a0 0x7f5168165ac0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f51500098a0 tx=0x7f515001c4f0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.812+0000 7fc083a00640 1 --2- 172.21.15.27:0/4108521706 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc07c0cc8b0 0x7fc07c1f2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.812+0000 7fc07be00640 1 --2- 172.21.15.27:0/4108521706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc07c09ceb0 0x7fc07c0cc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.812+0000 7fc082400640 1 --2- 172.21.15.27:0/4108521706 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc07c0cc8b0 0x7fc07c1f2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.812+0000 7fc083a00640 1 -- 172.21.15.27:0/4108521706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07c0ca5d0 con 0x7fc07c09ceb0 2024-01-15T18:05:06.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.812+0000 7fc083a00640 1 -- 172.21.15.27:0/4108521706 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07c0c99e0 con 0x7fc07c0cc8b0 2024-01-15T18:05:06.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.812+0000 7fc083a00640 1 -- 172.21.15.27:0/4108521706 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc07c154460 con 0x7fc05c0047d0 2024-01-15T18:05:06.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.812+0000 7fc082400640 1 -- 172.21.15.27:0/4108521706 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc05c0047d0 msgr2=0x7fc07c153f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.812+0000 7fc082400640 1 --2- 172.21.15.27:0/4108521706 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc05c0047d0 0x7fc07c153f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.812+0000 7fc082400640 1 -- 172.21.15.27:0/4108521706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc07c09ceb0 msgr2=0x7fc07c0cc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.812+0000 7fc082400640 1 --2- 172.21.15.27:0/4108521706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc07c09ceb0 0x7fc07c0cc370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.812+0000 7fc082400640 1 -- 172.21.15.27:0/4108521706 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07c0c9c30 con 0x7fc07c0cc8b0 2024-01-15T18:05:06.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.812+0000 7fc081000640 1 --2- 172.21.15.27:0/4108521706 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc05c0047d0 0x7fc07c153f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:06.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.816+0000 7fc082400640 1 --2- 172.21.15.27:0/4108521706 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc07c0cc8b0 0x7fc07c1f2ce0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc07c071c00 tx=0x7fc07400f520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:06.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.816+0000 7fc078c00640 1 -- 172.21.15.27:0/4108521706 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07400fd80 con 0x7fc07c0cc8b0 2024-01-15T18:05:06.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.816+0000 7fc083a00640 1 -- 172.21.15.27:0/4108521706 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc07c0c9f10 con 0x7fc07c0cc8b0 2024-01-15T18:05:06.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.816+0000 7fc083a00640 1 -- 172.21.15.27:0/4108521706 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc07c1f3220 con 0x7fc07c0cc8b0 2024-01-15T18:05:06.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.816+0000 7fc078c00640 1 -- 172.21.15.27:0/4108521706 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0740143f0 con 0x7fc07c0cc8b0 2024-01-15T18:05:06.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.816+0000 7fc078c00640 1 -- 172.21.15.27:0/4108521706 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07400fee0 con 0x7fc07c0cc8b0 2024-01-15T18:05:06.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.816+0000 7fc083a00640 1 -- 172.21.15.27:0/4108521706 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc07c0d15d0 con 0x7fc07c0cc8b0 2024-01-15T18:05:06.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.816+0000 7f516e400640 1 -- 172.21.15.27:0/1257110312 shutdown_connections 2024-01-15T18:05:06.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.816+0000 7f516e400640 1 --2- 172.21.15.27:0/1257110312 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f516806b190 0x7f5168166000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.816+0000 7f516e400640 1 --2- 172.21.15.27:0/1257110312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f516806c0a0 0x7f5168165ac0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.816+0000 7f516e400640 1 -- 172.21.15.27:0/1257110312 >> 172.21.15.27:0/1257110312 conn(0x7f51680d34a0 msgr2=0x7f51680d38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:06.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.816+0000 7f516e400640 1 -- 172.21.15.27:0/1257110312 shutdown_connections 2024-01-15T18:05:06.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.816+0000 7f516e400640 1 -- 172.21.15.27:0/1257110312 wait complete. 2024-01-15T18:05:06.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.816+0000 7f516e400640 1 Processor -- start 2024-01-15T18:05:06.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.820+0000 7f516e400640 1 -- start start 2024-01-15T18:05:06.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.820+0000 7f516e400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f516806b190 0x7f5168159810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.820+0000 7f516e400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f516806c0a0 0x7f516815e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.820+0000 7f5167400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f516806b190 0x7f5168159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.820+0000 7f5167400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f516806b190 0x7f5168159810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59358/0 (socket says 172.21.15.27:59358) 2024-01-15T18:05:06.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.820+0000 7f5167400640 1 -- 172.21.15.27:0/3913248009 learned_addr learned my addr 172.21.15.27:0/3913248009 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:06.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.820+0000 7f5166a00640 1 --2- 172.21.15.27:0/3913248009 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f516806c0a0 0x7f516815e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.820+0000 7f516e400640 1 --2- 172.21.15.27:0/3913248009 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f516806c480 0x7f5168159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.820+0000 7fc078c00640 1 -- 172.21.15.27:0/4108521706 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fc074014780 con 0x7fc07c0cc8b0 2024-01-15T18:05:06.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.824+0000 7fc078c00640 1 --2- 172.21.15.27:0/4108521706 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc054066970 0x7fc054068e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.824+0000 7fc078c00640 1 -- 172.21.15.27:0/4108521706 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc0740554f0 con 0x7fc07c0cc8b0 2024-01-15T18:05:06.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.824+0000 7fc081000640 1 --2- 172.21.15.27:0/4108521706 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc054066970 0x7fc054068e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.824+0000 7f5167e00640 1 --2- 172.21.15.27:0/3913248009 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f516806c480 0x7f5168159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.824+0000 7f516e400640 1 -- 172.21.15.27:0/3913248009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516815b9c0 con 0x7f516806b190 2024-01-15T18:05:06.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.824+0000 7f516e400640 1 -- 172.21.15.27:0/3913248009 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51680c9aa0 con 0x7f516806c0a0 2024-01-15T18:05:06.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.824+0000 7fc081000640 1 --2- 172.21.15.27:0/4108521706 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc054066970 0x7fc054068e30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc06c007b80 tx=0x7fc06c007a70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:06.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.824+0000 7f516e400640 1 -- 172.21.15.27:0/3913248009 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51681595d0 con 0x7f516806c480 2024-01-15T18:05:06.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.824+0000 7f5167e00640 1 -- 172.21.15.27:0/3913248009 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f516806c0a0 msgr2=0x7f516815e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.824+0000 7f5167e00640 1 --2- 172.21.15.27:0/3913248009 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f516806c0a0 0x7f516815e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.824+0000 7f5167e00640 1 -- 172.21.15.27:0/3913248009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f516806b190 msgr2=0x7f5168159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.824+0000 7f5167e00640 1 --2- 172.21.15.27:0/3913248009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f516806b190 0x7f5168159810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.824+0000 7f5167e00640 1 -- 172.21.15.27:0/3913248009 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51500094d0 con 0x7f516806c480 2024-01-15T18:05:06.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.824+0000 7f5166a00640 1 --2- 172.21.15.27:0/3913248009 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f516806c0a0 0x7f516815e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:06.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.824+0000 7f5167e00640 1 --2- 172.21.15.27:0/3913248009 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f516806c480 0x7f5168159090 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f516000c730 tx=0x7f516000cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:06.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.824+0000 7f515f400640 1 -- 172.21.15.27:0/3913248009 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5160006d60 con 0x7f516806c480 2024-01-15T18:05:06.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.824+0000 7fc078c00640 1 -- 172.21.15.27:0/4108521706 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc074089060 con 0x7fc07c0cc8b0 2024-01-15T18:05:06.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.824+0000 7f515f400640 1 -- 172.21.15.27:0/3913248009 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5160003980 con 0x7f516806c480 2024-01-15T18:05:06.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.824+0000 7f515f400640 1 -- 172.21.15.27:0/3913248009 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5160003d70 con 0x7f516806c480 2024-01-15T18:05:06.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.824+0000 7f516e400640 1 -- 172.21.15.27:0/3913248009 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51680c97c0 con 0x7f516806c480 2024-01-15T18:05:06.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.828+0000 7f516e400640 1 -- 172.21.15.27:0/3913248009 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51680c8ed0 con 0x7f516806c480 2024-01-15T18:05:06.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.828+0000 7f515f400640 1 -- 172.21.15.27:0/3913248009 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f51600052e0 con 0x7f516806c480 2024-01-15T18:05:06.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.832+0000 7f515cc00640 1 -- 172.21.15.27:0/3913248009 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51680c8a50 con 0x7f516806c480 2024-01-15T18:05:06.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.832+0000 7f515f400640 1 --2- 172.21.15.27:0/3913248009 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5140066920 0x7f5140068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.832+0000 7f515f400640 1 -- 172.21.15.27:0/3913248009 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5160083fa0 con 0x7f516806c480 2024-01-15T18:05:06.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.832+0000 7f5167400640 1 --2- 172.21.15.27:0/3913248009 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5140066920 0x7f5140068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.832+0000 7f5167400640 1 --2- 172.21.15.27:0/3913248009 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5140066920 0x7f5140068de0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f515001aba0 tx=0x7f51500b6040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:06.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.836+0000 7f515f400640 1 -- 172.21.15.27:0/3913248009 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f51680c8a50 con 0x7f516806c480 2024-01-15T18:05:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.848+0000 7fd797800640 1 Processor -- start 2024-01-15T18:05:06.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.852+0000 7fd797800640 1 -- start start 2024-01-15T18:05:06.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.852+0000 7fd797800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd7900b3a70 0x7fd790161150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.852+0000 7fd78fe00640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7fd790072060 0x7fd79006cd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:57132/0 (socket says 172.21.15.27:57132) 2024-01-15T18:05:06.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.852+0000 7fd78fe00640 1 -- 172.21.15.27:0/3817402340 learned_addr learned my addr 172.21.15.27:0/3817402340 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:06.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.852+0000 7fd795800640 1 --2- 172.21.15.27:0/3817402340 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd7900b3a70 0x7fd790161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.852+0000 7fd797800640 1 -- 172.21.15.27:0/3817402340 --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7900b2cb0 con 0x7fd79015d480 2024-01-15T18:05:06.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.852+0000 7fd797800640 1 -- 172.21.15.27:0/3817402340 --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7900b2f90 con 0x7fd790072060 2024-01-15T18:05:06.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.856+0000 7fd797800640 1 -- 172.21.15.27:0/3817402340 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7900b2aa0 con 0x7fd7900b3a70 2024-01-15T18:05:06.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.856+0000 7fd795800640 1 -- 172.21.15.27:0/3817402340 >> v1:172.21.15.93:6789/0 conn(0x7fd790072060 legacy=0x7fd79006cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.856+0000 7fd795800640 1 -- 172.21.15.27:0/3817402340 >> v1:172.21.15.27:6789/0 conn(0x7fd79015d480 legacy=0x7fd79006c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:06.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.856+0000 7fd78f400640 1 -- 172.21.15.27:0/3817402340 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756922939 0 0) 0x7fd7900b2f90 con 0x7fd790072060 2024-01-15T18:05:06.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.856+0000 7fd795800640 1 -- 172.21.15.27:0/3817402340 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7900b3270 con 0x7fd7900b3a70 2024-01-15T18:05:06.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.856+0000 7f2c84800640 1 Processor -- start 2024-01-15T18:05:06.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.856+0000 7f2c84800640 1 -- start start 2024-01-15T18:05:06.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.860+0000 7f2c84800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c7c15d480 0x7f2c7c0ce160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.860+0000 7f2c82800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2c7c06cc10 0x7f2c7c0b2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42404/0 (socket says 172.21.15.27:42404) 2024-01-15T18:05:06.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.860+0000 7f2c82800640 1 -- 172.21.15.27:0/2865387364 learned_addr learned my addr 172.21.15.27:0/2865387364 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:06.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.860+0000 7f2c84800640 1 -- 172.21.15.27:0/2865387364 --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7c0b24d0 con 0x7f2c7c06cc10 2024-01-15T18:05:06.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.860+0000 7f2c84800640 1 -- 172.21.15.27:0/2865387364 --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7c06d0c0 con 0x7f2c7c1592b0 2024-01-15T18:05:06.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.860+0000 7f2c84800640 1 -- 172.21.15.27:0/2865387364 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c159690 con 0x7f2c7c15d480 2024-01-15T18:05:06.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.860+0000 7f2c81400640 1 --2- 172.21.15.27:0/2865387364 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c7c15d480 0x7f2c7c0ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.860+0000 7f2c80a00640 1 -- 172.21.15.27:0/2865387364 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798956526 0 0) 0x7f2c7c0b24d0 con 0x7f2c7c06cc10 2024-01-15T18:05:06.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.860+0000 7f2c80a00640 1 -- 172.21.15.27:0/2865387364 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c5c004640 con 0x7f2c7c06cc10 2024-01-15T18:05:06.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7fd795800640 1 --2- 172.21.15.27:0/3817402340 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd7900b3a70 0x7fd790161150 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd788009dc0 tx=0x7fd78801c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=769521679735604c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:06.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.860+0000 7f2c80a00640 1 -- 172.21.15.27:0/2865387364 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3000024148 0 0) 0x7f2c7c06d0c0 con 0x7f2c7c1592b0 2024-01-15T18:05:06.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.860+0000 7f2c80a00640 1 -- 172.21.15.27:0/2865387364 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c7c0b24d0 con 0x7f2c7c1592b0 2024-01-15T18:05:06.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7f2c80a00640 1 -- 172.21.15.27:0/2865387364 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3520618791 0 0) 0x7f2c5c004640 con 0x7f2c7c06cc10 2024-01-15T18:05:06.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7f2c80a00640 1 -- 172.21.15.27:0/2865387364 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c7c15d480 msgr2=0x7f2c7c0ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:06.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7f2c80a00640 1 --2- 172.21.15.27:0/2865387364 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c7c15d480 0x7f2c7c0ce160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7f2c80a00640 1 -- 172.21.15.27:0/2865387364 >> v1:172.21.15.93:6789/0 conn(0x7f2c7c1592b0 legacy=0x7f2c7c15cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7f2c80a00640 1 -- 172.21.15.27:0/2865387364 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7c06d360 con 0x7f2c7c06cc10 2024-01-15T18:05:06.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7f2c80a00640 1 -- 172.21.15.27:0/2865387364 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f2c6c004ce0 con 0x7f2c7c06cc10 2024-01-15T18:05:06.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7f2c80a00640 1 -- 172.21.15.27:0/2865387364 >> v1:172.21.15.27:6789/0 conn(0x7f2c7c06cc10 legacy=0x7f2c7c0b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7f2c80a00640 1 --2- 172.21.15.27:0/2865387364 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c5c004cc0 0x7f2c5c0050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7fd78f400640 1 -- 172.21.15.27:0/3817402340 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7880a5860 con 0x7fd7900b3a70 2024-01-15T18:05:06.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7fd78f400640 1 -- 172.21.15.27:0/3817402340 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7880a59c0 con 0x7fd7900b3a70 2024-01-15T18:05:06.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7fd78f400640 1 -- 172.21.15.27:0/3817402340 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7880a5d50 con 0x7fd7900b3a70 2024-01-15T18:05:06.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7fd797800640 1 -- 172.21.15.27:0/3817402340 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd7900b3a70 msgr2=0x7fd790161150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7fd797800640 1 --2- 172.21.15.27:0/3817402340 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd7900b3a70 0x7fd790161150 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd788009dc0 tx=0x7fd78801c7e0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7f2c80a00640 1 --2- 172.21.15.27:0/2865387364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c5c0058e0 0x7f2c5c005d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7f2c80a00640 1 --2- 172.21.15.27:0/2865387364 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c5c0062f0 0x7f2c5c003450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7f2c80a00640 1 -- 172.21.15.27:0/2865387364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c159690 con 0x7f2c5c0058e0 2024-01-15T18:05:06.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7f2c80a00640 1 -- 172.21.15.27:0/2865387364 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c003aa0 con 0x7f2c5c004cc0 2024-01-15T18:05:06.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7f2c80a00640 1 -- 172.21.15.27:0/2865387364 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5c003c60 con 0x7f2c5c0062f0 2024-01-15T18:05:06.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7f2c81400640 1 --2- 172.21.15.27:0/2865387364 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c7c15d480 0x7f2c7c0ce160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:06.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7f2c82800640 1 --2- 172.21.15.27:0/2865387364 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c5c0062f0 0x7f2c5c003450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7f2c81e00640 1 --2- 172.21.15.27:0/2865387364 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c5c004cc0 0x7f2c5c0050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7f2c81400640 1 --2- 172.21.15.27:0/2865387364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c5c0058e0 0x7f2c5c005d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7f2c82800640 1 -- 172.21.15.27:0/2865387364 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c5c004cc0 msgr2=0x7f2c5c0050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:06.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7f2c82800640 1 --2- 172.21.15.27:0/2865387364 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c5c004cc0 0x7f2c5c0050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7f2c82800640 1 -- 172.21.15.27:0/2865387364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c5c0058e0 msgr2=0x7f2c5c005d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:06.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7f2c82800640 1 --2- 172.21.15.27:0/2865387364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c5c0058e0 0x7f2c5c005d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7fd797800640 1 -- 172.21.15.27:0/3817402340 shutdown_connections 2024-01-15T18:05:06.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7fd797800640 1 --2- 172.21.15.27:0/3817402340 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd7900b3a70 0x7fd790161150 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7fd797800640 1 -- 172.21.15.27:0/3817402340 >> 172.21.15.27:0/3817402340 conn(0x7fd790070aa0 msgr2=0x7fd790071950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:06.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7f2c82800640 1 -- 172.21.15.27:0/2865387364 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5c004640 con 0x7f2c5c0062f0 2024-01-15T18:05:06.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7f2c81e00640 1 --2- 172.21.15.27:0/2865387364 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c5c004cc0 0x7f2c5c0050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:06.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.864+0000 7f2c81400640 1 --2- 172.21.15.27:0/2865387364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c5c0058e0 0x7f2c5c005d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:06.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.868+0000 7fd797800640 1 -- 172.21.15.27:0/3817402340 shutdown_connections 2024-01-15T18:05:06.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.868+0000 7fd797800640 1 -- 172.21.15.27:0/3817402340 wait complete. 2024-01-15T18:05:06.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.868+0000 7f2c82800640 1 --2- 172.21.15.27:0/2865387364 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c5c0062f0 0x7f2c5c003450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c001e50 tx=0x7f2c6c01f8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72fbea14259ba3f8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:06.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.872+0000 7f2c80a00640 1 -- 172.21.15.27:0/2865387364 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c0aa630 con 0x7f2c5c0062f0 2024-01-15T18:05:06.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.872+0000 7f2c80a00640 1 -- 172.21.15.27:0/2865387364 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c6c0aa7b0 con 0x7f2c5c0062f0 2024-01-15T18:05:06.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.872+0000 7f2c80a00640 1 -- 172.21.15.27:0/2865387364 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c0aab60 con 0x7f2c5c0062f0 2024-01-15T18:05:06.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.872+0000 7fd797800640 1 Processor -- start 2024-01-15T18:05:06.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.872+0000 7fd797800640 1 -- start start 2024-01-15T18:05:06.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.872+0000 7fd797800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd790072060 0x7fd79015c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.872+0000 7fd797800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7900b3a70 0x7fd790098790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.872+0000 7f2c84800640 1 -- 172.21.15.27:0/2865387364 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c5c0062f0 msgr2=0x7f2c5c003450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.872+0000 7f2c84800640 1 --2- 172.21.15.27:0/2865387364 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c5c0062f0 0x7f2c5c003450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c001e50 tx=0x7f2c6c01f8f0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.872+0000 7f2c82800640 1 -- 172.21.15.27:0/2865387364 reap_dead start 2024-01-15T18:05:06.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.872+0000 7fd797800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd79015d480 0x7fd7900b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.872+0000 7fd78fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7900b3a70 0x7fd790098790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.872+0000 7fd794e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd790072060 0x7fd79015c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.872+0000 7fd794e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd790072060 0x7fd79015c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:42876/0 (socket says 172.21.15.27:42876) 2024-01-15T18:05:06.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.876+0000 7fd797800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd790098cd0 con 0x7fd7900b3a70 2024-01-15T18:05:06.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.876+0000 7fd797800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd790151f10 con 0x7fd79015d480 2024-01-15T18:05:06.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.876+0000 7fd797800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7901521f0 con 0x7fd790072060 2024-01-15T18:05:06.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.876+0000 7fd794e00640 1 -- 172.21.15.27:0/981498573 learned_addr learned my addr 172.21.15.27:0/981498573 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:06.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.876+0000 7fd795800640 1 --2- 172.21.15.27:0/981498573 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd79015d480 0x7fd7900b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.876+0000 7f2c84800640 1 -- 172.21.15.27:0/2865387364 shutdown_connections 2024-01-15T18:05:06.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.876+0000 7f2c84800640 1 -- 172.21.15.27:0/2865387364 >> 172.21.15.27:0/2865387364 conn(0x7f2c7c0b0850 msgr2=0x7f2c7c0b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:06.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.876+0000 7f2c84800640 1 -- 172.21.15.27:0/2865387364 shutdown_connections 2024-01-15T18:05:06.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.876+0000 7fd794e00640 1 -- 172.21.15.27:0/981498573 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd79015d480 msgr2=0x7fd7900b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.876+0000 7fd794e00640 1 --2- 172.21.15.27:0/981498573 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd79015d480 0x7fd7900b00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.876+0000 7fd794e00640 1 -- 172.21.15.27:0/981498573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7900b3a70 msgr2=0x7fd790098790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.876+0000 7fd794e00640 1 --2- 172.21.15.27:0/981498573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7900b3a70 0x7fd790098790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.876+0000 7fd794e00640 1 -- 172.21.15.27:0/981498573 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd788009a70 con 0x7fd790072060 2024-01-15T18:05:06.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.876+0000 7f2c84800640 1 -- 172.21.15.27:0/2865387364 wait complete. 2024-01-15T18:05:06.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.876+0000 7fd795800640 1 --2- 172.21.15.27:0/981498573 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd79015d480 0x7fd7900b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:06.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.876+0000 7fd794e00640 1 --2- 172.21.15.27:0/981498573 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd790072060 0x7fd79015c2a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd77800b410 tx=0x7fd77800b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:06.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.876+0000 7fd78cc00640 1 -- 172.21.15.27:0/981498573 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd778004020 con 0x7fd790072060 2024-01-15T18:05:06.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.876+0000 7fd78cc00640 1 -- 172.21.15.27:0/981498573 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd778004180 con 0x7fd790072060 2024-01-15T18:05:06.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.876+0000 7fd78cc00640 1 -- 172.21.15.27:0/981498573 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd778011630 con 0x7fd790072060 2024-01-15T18:05:06.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.876+0000 7fd797800640 1 -- 172.21.15.27:0/981498573 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7901524d0 con 0x7fd790072060 2024-01-15T18:05:06.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.876+0000 7fd797800640 1 -- 172.21.15.27:0/981498573 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd790152a90 con 0x7fd790072060 2024-01-15T18:05:06.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.876+0000 7fd797800640 1 -- 172.21.15.27:0/981498573 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd790074820 con 0x7fd790072060 2024-01-15T18:05:06.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.876+0000 7f2c84800640 1 Processor -- start 2024-01-15T18:05:06.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.880+0000 7f2c84800640 1 -- start start 2024-01-15T18:05:06.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.880+0000 7f2c84800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c5c0062f0 0x7f2c7c15c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.880+0000 7f2c84800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c7c0cc080 0x7f2c7c1efd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.880+0000 7f2c84800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c7c0cd0d0 0x7f2c7c0cd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.880+0000 7fd78cc00640 1 -- 172.21.15.27:0/981498573 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fd778011790 con 0x7fd790072060 2024-01-15T18:05:06.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.880+0000 7f2c84800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c0cda90 con 0x7f2c5c0062f0 2024-01-15T18:05:06.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.880+0000 7f2c84800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c159870 con 0x7f2c7c0cc080 2024-01-15T18:05:06.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.880+0000 7f2c84800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c159ae0 con 0x7f2c7c0cd0d0 2024-01-15T18:05:06.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.880+0000 7f2c81e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c5c0062f0 0x7f2c7c15c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.880+0000 7f2c81400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c7c0cc080 0x7f2c7c1efd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.880+0000 7f2c81e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c5c0062f0 0x7f2c7c15c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59378/0 (socket says 172.21.15.27:59378) 2024-01-15T18:05:06.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.880+0000 7f2c81e00640 1 -- 172.21.15.27:0/2238771426 learned_addr learned my addr 172.21.15.27:0/2238771426 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:06.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.880+0000 7f2c81400640 1 --2- 172.21.15.27:0/2238771426 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c7c0cc080 0x7f2c7c1efd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:51806/0 (socket says 172.21.15.27:51806) 2024-01-15T18:05:06.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.880+0000 7f2c82800640 1 --2- 172.21.15.27:0/2238771426 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c7c0cd0d0 0x7f2c7c0cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.880+0000 7f2c81e00640 1 -- 172.21.15.27:0/2238771426 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c7c0cd0d0 msgr2=0x7f2c7c0cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.880+0000 7f2c81e00640 1 --2- 172.21.15.27:0/2238771426 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c7c0cd0d0 0x7f2c7c0cd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.880+0000 7f2c81e00640 1 -- 172.21.15.27:0/2238771426 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c7c0cc080 msgr2=0x7f2c7c1efd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:06.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.880+0000 7f2c81e00640 1 --2- 172.21.15.27:0/2238771426 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c7c0cc080 0x7f2c7c1efd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:06.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.880+0000 7f2c81e00640 1 -- 172.21.15.27:0/2238771426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7c159dc0 con 0x7f2c5c0062f0 2024-01-15T18:05:06.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.884+0000 7f2c81e00640 1 --2- 172.21.15.27:0/2238771426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c5c0062f0 0x7f2c7c15c260 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2c7400fc40 tx=0x7f2c7400d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:06.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.884+0000 7f2c7a000640 1 -- 172.21.15.27:0/2238771426 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7400dbd0 con 0x7f2c5c0062f0 2024-01-15T18:05:06.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.884+0000 7fd78cc00640 1 --2- 172.21.15.27:0/981498573 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd768066940 0x7fd768068e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.884+0000 7f2c84800640 1 -- 172.21.15.27:0/2238771426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c7c15a0a0 con 0x7f2c5c0062f0 2024-01-15T18:05:06.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.884+0000 7f2c84800640 1 -- 172.21.15.27:0/2238771426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c7c0cc500 con 0x7f2c5c0062f0 2024-01-15T18:05:06.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.884+0000 7f2c7a000640 1 -- 172.21.15.27:0/2238771426 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7400dd30 con 0x7f2c5c0062f0 2024-01-15T18:05:06.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.884+0000 7fd78cc00640 1 -- 172.21.15.27:0/981498573 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd778083fe0 con 0x7fd790072060 2024-01-15T18:05:06.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.884+0000 7fd78fe00640 1 --2- 172.21.15.27:0/981498573 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd768066940 0x7fd768068e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.884+0000 7f2c7a000640 1 -- 172.21.15.27:0/2238771426 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7400dea0 con 0x7f2c5c0062f0 2024-01-15T18:05:06.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.888+0000 7f2c73400640 1 -- 172.21.15.27:0/2238771426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c7c04e850 con 0x7f2c5c0062f0 2024-01-15T18:05:06.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.888+0000 7fd78fe00640 1 --2- 172.21.15.27:0/981498573 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd768066940 0x7fd768068e00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd7900666d0 tx=0x7fd780000f20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:06.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.888+0000 7fd78cc00640 1 -- 172.21.15.27:0/981498573 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd7780512d0 con 0x7fd790072060 2024-01-15T18:05:06.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.888+0000 7f2c7a000640 1 -- 172.21.15.27:0/2238771426 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f2c74003b80 con 0x7f2c5c0062f0 2024-01-15T18:05:06.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.892+0000 7f2c7a000640 1 --2- 172.21.15.27:0/2238771426 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2c54066920 0x7f2c54068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:06.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.892+0000 7f2c81400640 1 --2- 172.21.15.27:0/2238771426 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2c54066920 0x7f2c54068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:06.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.892+0000 7f2c7a000640 1 -- 172.21.15.27:0/2238771426 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c74003e30 con 0x7f2c5c0062f0 2024-01-15T18:05:06.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.892+0000 7f2c81400640 1 --2- 172.21.15.27:0/2238771426 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2c54066920 0x7f2c54068de0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c150770 tx=0x7f2c64013040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:06.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:06.900+0000 7f2c7a000640 1 -- 172.21.15.27:0/2238771426 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2c74052160 con 0x7f2c5c0062f0 2024-01-15T18:05:07.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.060+0000 7f3749400640 1 -- 172.21.15.27:0/251453535 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f373c008b80 con 0x7f373c0e34b0 2024-01-15T18:05:07.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.060+0000 7f373aa00640 1 -- 172.21.15.27:0/251453535 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3734054bf0 con 0x7f373c0e34b0 2024-01-15T18:05:07.066 INFO:teuthology.orchestra.run.smithi027.stdout:64424509446 2024-01-15T18:05:07.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.064+0000 7f5e50c00640 1 -- 172.21.15.27:0/3017366036 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f5e240057e0 con 0x7f5e5c06e800 2024-01-15T18:05:07.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.064+0000 7f5e53400640 1 -- 172.21.15.27:0/3017366036 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5e4c0517e0 con 0x7f5e5c06e800 2024-01-15T18:05:07.069 INFO:teuthology.orchestra.run.smithi027.stdout:64424509448 2024-01-15T18:05:07.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.064+0000 7f3749400640 1 -- 172.21.15.27:0/251453535 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f371c066940 msgr2=0x7f371c068e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.064+0000 7f3749400640 1 --2- 172.21.15.27:0/251453535 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f371c066940 0x7f371c068e00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f373c055d70 tx=0x7f3744121070 comp rx=0 tx=0).stop 2024-01-15T18:05:07.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.068+0000 7f3749400640 1 -- 172.21.15.27:0/251453535 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f373c0e34b0 msgr2=0x7f373c0db920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.068+0000 7f3749400640 1 --2- 172.21.15.27:0/251453535 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f373c0e34b0 0x7f373c0db920 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f373c00df40 tx=0x7f373400b870 comp rx=0 tx=0).stop 2024-01-15T18:05:07.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.068+0000 7f5e50c00640 1 -- 172.21.15.27:0/3017366036 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5e34066940 msgr2=0x7f5e34068e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.068+0000 7f5e50c00640 1 --2- 172.21.15.27:0/3017366036 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5e34066940 0x7f5e34068e00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5e5400b240 tx=0x7f5e5400b3a0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.068+0000 7f5e50c00640 1 -- 172.21.15.27:0/3017366036 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5e5c06e800 msgr2=0x7f5e5c16b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.068+0000 7f5e50c00640 1 --2- 172.21.15.27:0/3017366036 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5e5c06e800 0x7f5e5c16b320 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c00c780 tx=0x7f5e4c00cc40 comp rx=0 tx=0).stop 2024-01-15T18:05:07.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.068+0000 7f5e50c00640 1 -- 172.21.15.27:0/3017366036 shutdown_connections 2024-01-15T18:05:07.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.068+0000 7f5e50c00640 1 --2- 172.21.15.27:0/3017366036 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5e34066940 0x7f5e34068e00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.068+0000 7f5e50c00640 1 --2- 172.21.15.27:0/3017366036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e5c08aaa0 0x7f5e5c0873d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.068+0000 7f5e50c00640 1 --2- 172.21.15.27:0/3017366036 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5e5c06e800 0x7f5e5c16b320 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.068+0000 7f5e50c00640 1 --2- 172.21.15.27:0/3017366036 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e5c06dd10 0x7f5e5c16ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.068+0000 7f5e50c00640 1 -- 172.21.15.27:0/3017366036 >> 172.21.15.27:0/3017366036 conn(0x7f5e5c0721b0 msgr2=0x7f5e5c164ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:07.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.068+0000 7f3749400640 1 -- 172.21.15.27:0/251453535 shutdown_connections 2024-01-15T18:05:07.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.068+0000 7f3749400640 1 --2- 172.21.15.27:0/251453535 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f371c066940 0x7f371c068e00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.068+0000 7f3749400640 1 --2- 172.21.15.27:0/251453535 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f373c0e34b0 0x7f373c0db920 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.068+0000 7f3749400640 1 --2- 172.21.15.27:0/251453535 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f373c0e2790 0x7f373c0db3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.068+0000 7f3749400640 1 --2- 172.21.15.27:0/251453535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f373c006090 0x7f373c0cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.068+0000 7f3749400640 1 -- 172.21.15.27:0/251453535 >> 172.21.15.27:0/251453535 conn(0x7f373c054c70 msgr2=0x7f373c055050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:07.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.068+0000 7f3749400640 1 -- 172.21.15.27:0/251453535 shutdown_connections 2024-01-15T18:05:07.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.072+0000 7f3749400640 1 -- 172.21.15.27:0/251453535 wait complete. 2024-01-15T18:05:07.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.072+0000 7f5e50c00640 1 -- 172.21.15.27:0/3017366036 shutdown_connections 2024-01-15T18:05:07.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.072+0000 7f5e50c00640 1 -- 172.21.15.27:0/3017366036 wait complete. 2024-01-15T18:05:07.110 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509448 for osd.1 2024-01-15T18:05:07.111 DEBUG:teuthology.parallel:result is None 2024-01-15T18:05:07.111 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509446 for osd.3 2024-01-15T18:05:07.111 DEBUG:teuthology.parallel:result is None 2024-01-15T18:05:07.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.136+0000 7fc083a00640 1 -- 172.21.15.27:0/4108521706 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fc07c0ca710 con 0x7fc07c0cc8b0 2024-01-15T18:05:07.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.136+0000 7fc078c00640 1 -- 172.21.15.27:0/4108521706 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc074089020 con 0x7fc07c0cc8b0 2024-01-15T18:05:07.141 INFO:teuthology.orchestra.run.smithi027.stdout:60129542151 2024-01-15T18:05:07.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.140+0000 7fc083a00640 1 -- 172.21.15.27:0/4108521706 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc054066970 msgr2=0x7fc054068e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.140+0000 7fc083a00640 1 --2- 172.21.15.27:0/4108521706 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc054066970 0x7fc054068e30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc06c007b80 tx=0x7fc06c007a70 comp rx=0 tx=0).stop 2024-01-15T18:05:07.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.140+0000 7fc083a00640 1 -- 172.21.15.27:0/4108521706 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc07c0cc8b0 msgr2=0x7fc07c1f2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.140+0000 7fc083a00640 1 --2- 172.21.15.27:0/4108521706 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc07c0cc8b0 0x7fc07c1f2ce0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc07c071c00 tx=0x7fc07400f520 comp rx=0 tx=0).stop 2024-01-15T18:05:07.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.140+0000 7fc083a00640 1 -- 172.21.15.27:0/4108521706 shutdown_connections 2024-01-15T18:05:07.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.140+0000 7fc083a00640 1 --2- 172.21.15.27:0/4108521706 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc054066970 0x7fc054068e30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.140+0000 7fc083a00640 1 --2- 172.21.15.27:0/4108521706 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc07c0cc8b0 0x7fc07c1f2ce0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.140+0000 7fc083a00640 1 --2- 172.21.15.27:0/4108521706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc07c09ceb0 0x7fc07c0cc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.140+0000 7fc083a00640 1 --2- 172.21.15.27:0/4108521706 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc05c0047d0 0x7fc07c153f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.140+0000 7fc083a00640 1 -- 172.21.15.27:0/4108521706 >> 172.21.15.27:0/4108521706 conn(0x7fc07c0721b0 msgr2=0x7fc07c0b1b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:07.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.140+0000 7fc083a00640 1 -- 172.21.15.27:0/4108521706 shutdown_connections 2024-01-15T18:05:07.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.140+0000 7fc083a00640 1 -- 172.21.15.27:0/4108521706 wait complete. 2024-01-15T18:05:07.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.144+0000 7f515cc00640 1 -- 172.21.15.27:0/3913248009 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f51680c8a50 con 0x7f516806c480 2024-01-15T18:05:07.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.144+0000 7f515f400640 1 -- 172.21.15.27:0/3913248009 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f51680c8a50 con 0x7f516806c480 2024-01-15T18:05:07.148 INFO:teuthology.orchestra.run.smithi027.stdout:64424509447 2024-01-15T18:05:07.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.148+0000 7f515cc00640 1 -- 172.21.15.27:0/3913248009 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5140066920 msgr2=0x7f5140068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.148+0000 7f515cc00640 1 --2- 172.21.15.27:0/3913248009 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5140066920 0x7f5140068de0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f515001aba0 tx=0x7f51500b6040 comp rx=0 tx=0).stop 2024-01-15T18:05:07.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.148+0000 7f515cc00640 1 -- 172.21.15.27:0/3913248009 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f516806c480 msgr2=0x7f5168159090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.148+0000 7f515cc00640 1 --2- 172.21.15.27:0/3913248009 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f516806c480 0x7f5168159090 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f516000c730 tx=0x7f516000cbf0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.148+0000 7f515cc00640 1 -- 172.21.15.27:0/3913248009 shutdown_connections 2024-01-15T18:05:07.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.148+0000 7f515cc00640 1 --2- 172.21.15.27:0/3913248009 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5140066920 0x7f5140068de0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.148+0000 7f515cc00640 1 --2- 172.21.15.27:0/3913248009 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f516806c480 0x7f5168159090 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.148+0000 7f515cc00640 1 --2- 172.21.15.27:0/3913248009 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f516806c0a0 0x7f516815e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.148+0000 7f515cc00640 1 --2- 172.21.15.27:0/3913248009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f516806b190 0x7f5168159810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.148+0000 7f515cc00640 1 -- 172.21.15.27:0/3913248009 >> 172.21.15.27:0/3913248009 conn(0x7f51680d34a0 msgr2=0x7f51680b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:07.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.148+0000 7f515cc00640 1 -- 172.21.15.27:0/3913248009 shutdown_connections 2024-01-15T18:05:07.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.148+0000 7f515cc00640 1 -- 172.21.15.27:0/3913248009 wait complete. 2024-01-15T18:05:07.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.148+0000 7f52d6800640 1 -- 172.21.15.27:0/368879889 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f52c8005470 con 0x7f52c80e2790 2024-01-15T18:05:07.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.148+0000 7f52c7400640 1 -- 172.21.15.27:0/368879889 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f52c00af070 con 0x7f52c80e2790 2024-01-15T18:05:07.154 INFO:teuthology.orchestra.run.smithi027.stdout:64424509446 2024-01-15T18:05:07.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.152+0000 7f52d6800640 1 -- 172.21.15.27:0/368879889 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f52a8066950 msgr2=0x7f52a8068e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.152+0000 7f52d6800640 1 --2- 172.21.15.27:0/368879889 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f52a8066950 0x7f52a8068e10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f52c80d6260 tx=0x7f52b80090e0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.152+0000 7f52d6800640 1 -- 172.21.15.27:0/368879889 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52c80e2790 msgr2=0x7f52c80d9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.152+0000 7f52d6800640 1 --2- 172.21.15.27:0/368879889 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52c80e2790 0x7f52c80d9ff0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f52c00b1000 tx=0x7f52c001f410 comp rx=0 tx=0).stop 2024-01-15T18:05:07.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.152+0000 7f52d6800640 1 -- 172.21.15.27:0/368879889 shutdown_connections 2024-01-15T18:05:07.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.152+0000 7f52d6800640 1 --2- 172.21.15.27:0/368879889 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f52a8066950 0x7f52a8068e10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.152+0000 7f52d6800640 1 --2- 172.21.15.27:0/368879889 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52c80e2790 0x7f52c80d9ff0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.152+0000 7f52d6800640 1 --2- 172.21.15.27:0/368879889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c80076a0 0x7f52c80cbe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.152+0000 7f52d6800640 1 --2- 172.21.15.27:0/368879889 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f52c8006090 0x7f52c80cda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.152+0000 7f52d6800640 1 -- 172.21.15.27:0/368879889 >> 172.21.15.27:0/368879889 conn(0x7f52c8054c70 msgr2=0x7f52c80e15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:07.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.152+0000 7f52d6800640 1 -- 172.21.15.27:0/368879889 shutdown_connections 2024-01-15T18:05:07.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.152+0000 7f52d6800640 1 -- 172.21.15.27:0/368879889 wait complete. 2024-01-15T18:05:07.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.160+0000 7f982aa00640 1 -- 172.21.15.27:0/756265673 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f981c0de390 con 0x7f981c185710 2024-01-15T18:05:07.164 INFO:teuthology.orchestra.run.smithi027.stdout:64424509447 2024-01-15T18:05:07.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.160+0000 7f981be00640 1 -- 172.21.15.27:0/756265673 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9824136070 con 0x7f981c185710 2024-01-15T18:05:07.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.164+0000 7f982aa00640 1 -- 172.21.15.27:0/756265673 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f97fc0669e0 msgr2=0x7f97fc068ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.164+0000 7f982aa00640 1 --2- 172.21.15.27:0/756265673 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f97fc0669e0 0x7f97fc068ea0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9814001ec0 tx=0x7f9814001a50 comp rx=0 tx=0).stop 2024-01-15T18:05:07.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.164+0000 7f982aa00640 1 -- 172.21.15.27:0/756265673 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f981c185710 msgr2=0x7f981c04d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.164+0000 7f982aa00640 1 --2- 172.21.15.27:0/756265673 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f981c185710 0x7f981c04d1c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9824050190 tx=0x7f9804005800 comp rx=0 tx=0).stop 2024-01-15T18:05:07.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.164+0000 7f982aa00640 1 -- 172.21.15.27:0/756265673 shutdown_connections 2024-01-15T18:05:07.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.164+0000 7f982aa00640 1 --2- 172.21.15.27:0/756265673 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f97fc0669e0 0x7f97fc068ea0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.164+0000 7f982aa00640 1 --2- 172.21.15.27:0/756265673 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f981c185710 0x7f981c04d1c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.164+0000 7f982aa00640 1 --2- 172.21.15.27:0/756265673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f981c0dc1f0 0x7f981c04cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.164+0000 7f982aa00640 1 --2- 172.21.15.27:0/756265673 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9804005190 0x7f981c0dd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.164+0000 7f982aa00640 1 -- 172.21.15.27:0/756265673 >> 172.21.15.27:0/756265673 conn(0x7f981c009f90 msgr2=0x7f981c033760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:07.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.164+0000 7f982aa00640 1 -- 172.21.15.27:0/756265673 shutdown_connections 2024-01-15T18:05:07.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.164+0000 7f982aa00640 1 -- 172.21.15.27:0/756265673 wait complete. 2024-01-15T18:05:07.172 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.0 2024-01-15T18:05:07.172 DEBUG:teuthology.parallel:result is None 2024-01-15T18:05:07.179 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509447 for osd.4 2024-01-15T18:05:07.179 DEBUG:teuthology.parallel:result is None 2024-01-15T18:05:07.193 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509446 for osd.2 2024-01-15T18:05:07.193 DEBUG:teuthology.parallel:result is None 2024-01-15T18:05:07.199 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509447 for osd.6 2024-01-15T18:05:07.200 DEBUG:teuthology.parallel:result is None 2024-01-15T18:05:07.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.204+0000 7fd797800640 1 -- 172.21.15.27:0/981498573 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fd790152330 con 0x7fd790072060 2024-01-15T18:05:07.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.204+0000 7fd78cc00640 1 -- 172.21.15.27:0/981498573 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd790152330 con 0x7fd790072060 2024-01-15T18:05:07.210 INFO:teuthology.orchestra.run.smithi027.stdout:60129542151 2024-01-15T18:05:07.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.208+0000 7fd797800640 1 -- 172.21.15.27:0/981498573 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd768066940 msgr2=0x7fd768068e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.208+0000 7fd797800640 1 --2- 172.21.15.27:0/981498573 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd768066940 0x7fd768068e00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd7900666d0 tx=0x7fd780000f20 comp rx=0 tx=0).stop 2024-01-15T18:05:07.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.208+0000 7fd797800640 1 -- 172.21.15.27:0/981498573 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd790072060 msgr2=0x7fd79015c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.208+0000 7fd797800640 1 --2- 172.21.15.27:0/981498573 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd790072060 0x7fd79015c2a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd77800b410 tx=0x7fd77800b8d0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.208+0000 7fd797800640 1 -- 172.21.15.27:0/981498573 shutdown_connections 2024-01-15T18:05:07.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.208+0000 7fd797800640 1 --2- 172.21.15.27:0/981498573 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd768066940 0x7fd768068e00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.208+0000 7fd797800640 1 --2- 172.21.15.27:0/981498573 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd79015d480 0x7fd7900b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.208+0000 7fd797800640 1 --2- 172.21.15.27:0/981498573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7900b3a70 0x7fd790098790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.208+0000 7fd797800640 1 --2- 172.21.15.27:0/981498573 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd790072060 0x7fd79015c2a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.208+0000 7fd797800640 1 -- 172.21.15.27:0/981498573 >> 172.21.15.27:0/981498573 conn(0x7fd790070aa0 msgr2=0x7fd7900b1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:07.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.208+0000 7fd797800640 1 -- 172.21.15.27:0/981498573 shutdown_connections 2024-01-15T18:05:07.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.208+0000 7fd797800640 1 -- 172.21.15.27:0/981498573 wait complete. 2024-01-15T18:05:07.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.216+0000 7f2c73400640 1 -- 172.21.15.27:0/2238771426 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f2c7c0b1120 con 0x7f2c5c0062f0 2024-01-15T18:05:07.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.216+0000 7f2c7a000640 1 -- 172.21.15.27:0/2238771426 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2c74052300 con 0x7f2c5c0062f0 2024-01-15T18:05:07.219 INFO:teuthology.orchestra.run.smithi027.stdout:64424509447 2024-01-15T18:05:07.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.216+0000 7f2c73400640 1 -- 172.21.15.27:0/2238771426 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2c54066920 msgr2=0x7f2c54068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.216+0000 7f2c73400640 1 --2- 172.21.15.27:0/2238771426 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2c54066920 0x7f2c54068de0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c150770 tx=0x7f2c64013040 comp rx=0 tx=0).stop 2024-01-15T18:05:07.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.216+0000 7f2c73400640 1 -- 172.21.15.27:0/2238771426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c5c0062f0 msgr2=0x7f2c7c15c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.216+0000 7f2c73400640 1 --2- 172.21.15.27:0/2238771426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c5c0062f0 0x7f2c7c15c260 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2c7400fc40 tx=0x7f2c7400d520 comp rx=0 tx=0).stop 2024-01-15T18:05:07.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.216+0000 7f2c73400640 1 -- 172.21.15.27:0/2238771426 shutdown_connections 2024-01-15T18:05:07.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.216+0000 7f2c73400640 1 --2- 172.21.15.27:0/2238771426 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2c54066920 0x7f2c54068de0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.216+0000 7f2c73400640 1 --2- 172.21.15.27:0/2238771426 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c7c0cd0d0 0x7f2c7c0cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.216+0000 7f2c73400640 1 --2- 172.21.15.27:0/2238771426 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c7c0cc080 0x7f2c7c1efd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.216+0000 7f2c73400640 1 --2- 172.21.15.27:0/2238771426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c5c0062f0 0x7f2c7c15c260 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.216+0000 7f2c73400640 1 -- 172.21.15.27:0/2238771426 >> 172.21.15.27:0/2238771426 conn(0x7f2c7c0b0850 msgr2=0x7f2c7c0b3310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:07.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.216+0000 7f2c73400640 1 -- 172.21.15.27:0/2238771426 shutdown_connections 2024-01-15T18:05:07.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.216+0000 7f2c73400640 1 -- 172.21.15.27:0/2238771426 wait complete. 2024-01-15T18:05:07.233 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.5 2024-01-15T18:05:07.233 DEBUG:teuthology.parallel:result is None 2024-01-15T18:05:07.242 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509447 for osd.7 2024-01-15T18:05:07.242 DEBUG:teuthology.parallel:result is None 2024-01-15T18:05:07.242 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-15T18:05:07.242 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:05:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.420+0000 7f96b8000640 1 Processor -- start 2024-01-15T18:05:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.420+0000 7f96b8000640 1 -- start start 2024-01-15T18:05:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.420+0000 7f96b8000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f96b00cf080 0x7f96b00cf460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.424+0000 7f96b8000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f96b00b0130 0x7f96b00b0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:07.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.424+0000 7f96b8000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b00d1ec0 con 0x7f96b00b24a0 2024-01-15T18:05:07.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.424+0000 7f96b8000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96b00ae160 con 0x7f96b00b0130 2024-01-15T18:05:07.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.424+0000 7f96b8000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96b00d1be0 con 0x7f96b00cf080 2024-01-15T18:05:07.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.424+0000 7f96b4c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f96b00b0130 0x7f96b00b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:07.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.424+0000 7f96b4c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f96b00b0130 0x7f96b00b0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59384/0 (socket says 172.21.15.27:59384) 2024-01-15T18:05:07.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.424+0000 7f96b4c00640 1 -- 172.21.15.27:0/2159167259 learned_addr learned my addr 172.21.15.27:0/2159167259 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:07.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.424+0000 7f96b5600640 1 --2- 172.21.15.27:0/2159167259 >> v2:172.21.15.93:3300/0 conn(0x7f96b00cf080 0x7f96b00cf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:07.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.424+0000 7f96b4c00640 1 -- 172.21.15.27:0/2159167259 >> v2:172.21.15.93:3300/0 conn(0x7f96b00cf080 msgr2=0x7f96b00cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:07.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.424+0000 7f96b4c00640 1 --2- 172.21.15.27:0/2159167259 >> v2:172.21.15.93:3300/0 conn(0x7f96b00cf080 0x7f96b00cf460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.424+0000 7f96b4c00640 1 -- 172.21.15.27:0/2159167259 >> v1:172.21.15.93:6789/0 conn(0x7f96b00b24a0 legacy=0x7f96b00b0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.424+0000 7f96b4c00640 1 -- 172.21.15.27:0/2159167259 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b00addf0 con 0x7f96b00b0130 2024-01-15T18:05:07.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.424+0000 7f96b4c00640 1 --2- 172.21.15.27:0/2159167259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b00b0130 0x7f96b00b0880 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f96b0071400 tx=0x7f969801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f944df8a4c5676f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:07.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.428+0000 7f96afe00640 1 -- 172.21.15.27:0/2159167259 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96980aa680 con 0x7f96b00b0130 2024-01-15T18:05:07.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.428+0000 7f96afe00640 1 -- 172.21.15.27:0/2159167259 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96980aa7e0 con 0x7f96b00b0130 2024-01-15T18:05:07.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.428+0000 7f96afe00640 1 -- 172.21.15.27:0/2159167259 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96980aab70 con 0x7f96b00b0130 2024-01-15T18:05:07.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.428+0000 7f96b8000640 1 -- 172.21.15.27:0/2159167259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b00b0130 msgr2=0x7f96b00b0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.428+0000 7f96b8000640 1 --2- 172.21.15.27:0/2159167259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b00b0130 0x7f96b00b0880 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f96b0071400 tx=0x7f969801c500 comp rx=0 tx=0).stop 2024-01-15T18:05:07.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.428+0000 7f96b8000640 1 -- 172.21.15.27:0/2159167259 shutdown_connections 2024-01-15T18:05:07.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.428+0000 7f96b8000640 1 --2- 172.21.15.27:0/2159167259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b00b0130 0x7f96b00b0880 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.428+0000 7f96b8000640 1 --2- 172.21.15.27:0/2159167259 >> v2:172.21.15.93:3300/0 conn(0x7f96b00cf080 0x7f96b00cf460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.428+0000 7f96b8000640 1 -- 172.21.15.27:0/2159167259 >> 172.21.15.27:0/2159167259 conn(0x7f96b00d0f20 msgr2=0x7f96b00d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:07.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.428+0000 7f96b8000640 1 -- 172.21.15.27:0/2159167259 shutdown_connections 2024-01-15T18:05:07.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.428+0000 7f96b8000640 1 -- 172.21.15.27:0/2159167259 wait complete. 2024-01-15T18:05:07.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.428+0000 7f96b8000640 1 Processor -- start 2024-01-15T18:05:07.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.428+0000 7f96b8000640 1 -- start start 2024-01-15T18:05:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.428+0000 7f96b8000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b00b0130 0x7f96b0151370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.428+0000 7f96b8000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f96b00b24a0 0x7f96b0156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.428+0000 7f96b8000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f96b00cf080 0x7f96b0156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.428+0000 7f96b8000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b0156260 con 0x7f96b00b0130 2024-01-15T18:05:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.428+0000 7f96b8000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b0155650 con 0x7f96b00cf080 2024-01-15T18:05:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.428+0000 7f96b8000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b0155930 con 0x7f96b00b24a0 2024-01-15T18:05:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.432+0000 7f96b5600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b00b0130 0x7f96b0151370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.432+0000 7f96b5600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b00b0130 0x7f96b0151370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59394/0 (socket says 172.21.15.27:59394) 2024-01-15T18:05:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.432+0000 7f96b5600640 1 -- 172.21.15.27:0/774303790 learned_addr learned my addr 172.21.15.27:0/774303790 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.432+0000 7f96b4c00640 1 --2- 172.21.15.27:0/774303790 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f96b00b24a0 0x7f96b0156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.432+0000 7f96b6000640 1 --2- 172.21.15.27:0/774303790 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f96b00cf080 0x7f96b0156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.432+0000 7f96b5600640 1 -- 172.21.15.27:0/774303790 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f96b00b24a0 msgr2=0x7f96b0156a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.432+0000 7f96b5600640 1 --2- 172.21.15.27:0/774303790 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f96b00b24a0 0x7f96b0156a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.432+0000 7f96b5600640 1 -- 172.21.15.27:0/774303790 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f96b00cf080 msgr2=0x7f96b0156f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.432+0000 7f96b5600640 1 --2- 172.21.15.27:0/774303790 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f96b00cf080 0x7f96b0156f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.432+0000 7f96b5600640 1 -- 172.21.15.27:0/774303790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96980094d0 con 0x7f96b00b0130 2024-01-15T18:05:07.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.432+0000 7f96b5600640 1 --2- 172.21.15.27:0/774303790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b00b0130 0x7f96b0151370 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f96a000e870 tx=0x7f96a000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:07.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.432+0000 7f96ad600640 1 -- 172.21.15.27:0/774303790 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a0009510 con 0x7f96b00b0130 2024-01-15T18:05:07.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.432+0000 7f96ad600640 1 -- 172.21.15.27:0/774303790 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a000eec0 con 0x7f96b00b0130 2024-01-15T18:05:07.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.432+0000 7f96ad600640 1 -- 172.21.15.27:0/774303790 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a000cb40 con 0x7f96b00b0130 2024-01-15T18:05:07.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.432+0000 7f96b8000640 1 -- 172.21.15.27:0/774303790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b0155c10 con 0x7f96b00b0130 2024-01-15T18:05:07.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.432+0000 7f96b8000640 1 -- 172.21.15.27:0/774303790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b01518b0 con 0x7f96b00b0130 2024-01-15T18:05:07.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.432+0000 7f96b8000640 1 -- 172.21.15.27:0/774303790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96b004e850 con 0x7f96b00b0130 2024-01-15T18:05:07.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.432+0000 7f96ad600640 1 -- 172.21.15.27:0/774303790 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f96a000cca0 con 0x7f96b00b0130 2024-01-15T18:05:07.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.436+0000 7f96ad600640 1 --2- 172.21.15.27:0/774303790 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9688066940 0x7f9688068e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:07.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.436+0000 7f96ad600640 1 -- 172.21.15.27:0/774303790 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f96a00843d0 con 0x7f96b00b0130 2024-01-15T18:05:07.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.436+0000 7f96b4c00640 1 --2- 172.21.15.27:0/774303790 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9688066940 0x7f9688068e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:07.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.436+0000 7f96b4c00640 1 --2- 172.21.15.27:0/774303790 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9688066940 0x7f9688068e00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f96980ae000 tx=0x7f96980b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:07.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.440+0000 7f96ad600640 1 -- 172.21.15.27:0/774303790 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f96a00517a0 con 0x7f96b00b0130 2024-01-15T18:05:07.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.592+0000 7f96b8000640 1 -- 172.21.15.27:0/774303790 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f96b01563a0 con 0x7f9688066940 2024-01-15T18:05:07.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.600+0000 7f96ad600640 1 -- 172.21.15.27:0/774303790 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264557 (secure 0 0 0) 0x7f96980aa950 con 0x7f9688066940 2024-01-15T18:05:07.608 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:07.609 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:05:07.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.604+0000 7f96b8000640 1 -- 172.21.15.27:0/774303790 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9688066940 msgr2=0x7f9688068e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.604+0000 7f96b8000640 1 --2- 172.21.15.27:0/774303790 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9688066940 0x7f9688068e00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f96980ae000 tx=0x7f96980b5040 comp rx=0 tx=0).stop 2024-01-15T18:05:07.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.604+0000 7f96b8000640 1 -- 172.21.15.27:0/774303790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b00b0130 msgr2=0x7f96b0151370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.604+0000 7f96b8000640 1 --2- 172.21.15.27:0/774303790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b00b0130 0x7f96b0151370 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f96a000e870 tx=0x7f96a000ec70 comp rx=0 tx=0).stop 2024-01-15T18:05:07.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.604+0000 7f96b8000640 1 -- 172.21.15.27:0/774303790 shutdown_connections 2024-01-15T18:05:07.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.604+0000 7f96b8000640 1 --2- 172.21.15.27:0/774303790 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9688066940 0x7f9688068e00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.604+0000 7f96b8000640 1 --2- 172.21.15.27:0/774303790 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f96b00cf080 0x7f96b0156f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.604+0000 7f96b8000640 1 --2- 172.21.15.27:0/774303790 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f96b00b24a0 0x7f96b0156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.604+0000 7f96b8000640 1 --2- 172.21.15.27:0/774303790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f96b00b0130 0x7f96b0151370 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.604+0000 7f96b8000640 1 -- 172.21.15.27:0/774303790 >> 172.21.15.27:0/774303790 conn(0x7f96b00d0f20 msgr2=0x7f96b00cd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:07.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.604+0000 7f96b8000640 1 -- 172.21.15.27:0/774303790 shutdown_connections 2024-01-15T18:05:07.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.604+0000 7f96b8000640 1 -- 172.21.15.27:0/774303790 wait complete. 2024-01-15T18:05:07.629 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-01-15T18:05:06.140488+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219052,"kb_used_data":2856,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755668,"statfs":{"total":773094113280,"available":772869804032,"internally_reserved":0,"allocated":2924544,"data_stored":1762796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002178"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032071+0000","last_change":"2024-01-15T18:04:58.862012+0000","last_active":"2024-01-15T18:05:00.032071+0000","last_peered":"2024-01-15T18:05:00.032071+0000","last_clean":"2024-01-15T18:05:00.032071+0000","last_became_active":"2024-01-15T18:04:58.861713+0000","last_became_peered":"2024-01-15T18:04:58.861713+0000","last_unstale":"2024-01-15T18:05:00.032071+0000","last_undegraded":"2024-01-15T18:05:00.032071+0000","last_fullsized":"2024-01-15T18:05:00.032071+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:30:29.323987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849437+0000","last_change":"2024-01-15T18:04:58.870910+0000","last_active":"2024-01-15T18:04:59.849437+0000","last_peered":"2024-01-15T18:04:59.849437+0000","last_clean":"2024-01-15T18:04:59.849437+0000","last_became_active":"2024-01-15T18:04:58.869957+0000","last_became_peered":"2024-01-15T18:04:58.869957+0000","last_unstale":"2024-01-15T18:04:59.849437+0000","last_undegraded":"2024-01-15T18:04:59.849437+0000","last_fullsized":"2024-01-15T18:04:59.849437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:10:24.519754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031283+0000","last_change":"2024-01-15T18:04:58.854305+0000","last_active":"2024-01-15T18:05:00.031283+0000","last_peered":"2024-01-15T18:05:00.031283+0000","last_clean":"2024-01-15T18:05:00.031283+0000","last_became_active":"2024-01-15T18:04:58.853375+0000","last_became_peered":"2024-01-15T18:04:58.853375+0000","last_unstale":"2024-01-15T18:05:00.031283+0000","last_undegraded":"2024-01-15T18:05:00.031283+0000","last_fullsized":"2024-01-15T18:05:00.031283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:27:32.756552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847538+0000","last_change":"2024-01-15T18:04:58.844639+0000","last_active":"2024-01-15T18:04:59.847538+0000","last_peered":"2024-01-15T18:04:59.847538+0000","last_clean":"2024-01-15T18:04:59.847538+0000","last_became_active":"2024-01-15T18:04:58.844174+0000","last_became_peered":"2024-01-15T18:04:58.844174+0000","last_unstale":"2024-01-15T18:04:59.847538+0000","last_undegraded":"2024-01-15T18:04:59.847538+0000","last_fullsized":"2024-01-15T18:04:59.847538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:18:25.022013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031678+0000","last_change":"2024-01-15T18:04:58.852004+0000","last_active":"2024-01-15T18:05:00.031678+0000","last_peered":"2024-01-15T18:05:00.031678+0000","last_clean":"2024-01-15T18:05:00.031678+0000","last_became_active":"2024-01-15T18:04:58.849191+0000","last_became_peered":"2024-01-15T18:04:58.849191+0000","last_unstale":"2024-01-15T18:05:00.031678+0000","last_undegraded":"2024-01-15T18:05:00.031678+0000","last_fullsized":"2024-01-15T18:05:00.031678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:22:03.084714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.846852+0000","last_change":"2024-01-15T18:04:58.854461+0000","last_active":"2024-01-15T18:04:59.846852+0000","last_peered":"2024-01-15T18:04:59.846852+0000","last_clean":"2024-01-15T18:04:59.846852+0000","last_became_active":"2024-01-15T18:04:58.854170+0000","last_became_peered":"2024-01-15T18:04:58.854170+0000","last_unstale":"2024-01-15T18:04:59.846852+0000","last_undegraded":"2024-01-15T18:04:59.846852+0000","last_fullsized":"2024-01-15T18:04:59.846852+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:05:31.164011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031425+0000","last_change":"2024-01-15T18:04:58.857161+0000","last_active":"2024-01-15T18:05:00.031425+0000","last_peered":"2024-01-15T18:05:00.031425+0000","last_clean":"2024-01-15T18:05:00.031425+0000","last_became_active":"2024-01-15T18:04:58.856839+0000","last_became_peered":"2024-01-15T18:04:58.856839+0000","last_unstale":"2024-01-15T18:05:00.031425+0000","last_undegraded":"2024-01-15T18:05:00.031425+0000","last_fullsized":"2024-01-15T18:05:00.031425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:53:23.471036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031831+0000","last_change":"2024-01-15T18:04:58.861226+0000","last_active":"2024-01-15T18:05:00.031831+0000","last_peered":"2024-01-15T18:05:00.031831+0000","last_clean":"2024-01-15T18:05:00.031831+0000","last_became_active":"2024-01-15T18:04:58.860964+0000","last_became_peered":"2024-01-15T18:04:58.860964+0000","last_unstale":"2024-01-15T18:05:00.031831+0000","last_undegraded":"2024-01-15T18:05:00.031831+0000","last_fullsized":"2024-01-15T18:05:00.031831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:11:49.903930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032115+0000","last_change":"2024-01-15T18:04:58.856674+0000","last_active":"2024-01-15T18:05:00.032115+0000","last_peered":"2024-01-15T18:05:00.032115+0000","last_clean":"2024-01-15T18:05:00.032115+0000","last_became_active":"2024-01-15T18:04:58.856365+0000","last_became_peered":"2024-01-15T18:04:58.856365+0000","last_unstale":"2024-01-15T18:05:00.032115+0000","last_undegraded":"2024-01-15T18:05:00.032115+0000","last_fullsized":"2024-01-15T18:05:00.032115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:09:03.416070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030934+0000","last_change":"2024-01-15T18:04:58.853185+0000","last_active":"2024-01-15T18:05:00.030934+0000","last_peered":"2024-01-15T18:05:00.030934+0000","last_clean":"2024-01-15T18:05:00.030934+0000","last_became_active":"2024-01-15T18:04:58.852888+0000","last_became_peered":"2024-01-15T18:04:58.852888+0000","last_unstale":"2024-01-15T18:05:00.030934+0000","last_undegraded":"2024-01-15T18:05:00.030934+0000","last_fullsized":"2024-01-15T18:05:00.030934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:10:24.519754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030810+0000","last_change":"2024-01-15T18:04:58.848009+0000","last_active":"2024-01-15T18:05:00.030810+0000","last_peered":"2024-01-15T18:05:00.030810+0000","last_clean":"2024-01-15T18:05:00.030810+0000","last_became_active":"2024-01-15T18:04:58.847765+0000","last_became_peered":"2024-01-15T18:04:58.847765+0000","last_unstale":"2024-01-15T18:05:00.030810+0000","last_undegraded":"2024-01-15T18:05:00.030810+0000","last_fullsized":"2024-01-15T18:05:00.030810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:15:34.376159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847406+0000","last_change":"2024-01-15T18:04:58.847259+0000","last_active":"2024-01-15T18:04:59.847406+0000","last_peered":"2024-01-15T18:04:59.847406+0000","last_clean":"2024-01-15T18:04:59.847406+0000","last_became_active":"2024-01-15T18:04:58.845619+0000","last_became_peered":"2024-01-15T18:04:58.845619+0000","last_unstale":"2024-01-15T18:04:59.847406+0000","last_undegraded":"2024-01-15T18:04:59.847406+0000","last_fullsized":"2024-01-15T18:04:59.847406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:16:30.446952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032165+0000","last_change":"2024-01-15T18:04:58.842389+0000","last_active":"2024-01-15T18:05:00.032165+0000","last_peered":"2024-01-15T18:05:00.032165+0000","last_clean":"2024-01-15T18:05:00.032165+0000","last_became_active":"2024-01-15T18:04:58.841378+0000","last_became_peered":"2024-01-15T18:04:58.841378+0000","last_unstale":"2024-01-15T18:05:00.032165+0000","last_undegraded":"2024-01-15T18:05:00.032165+0000","last_fullsized":"2024-01-15T18:05:00.032165+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:36.680373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030952+0000","last_change":"2024-01-15T18:04:58.864203+0000","last_active":"2024-01-15T18:05:00.030952+0000","last_peered":"2024-01-15T18:05:00.030952+0000","last_clean":"2024-01-15T18:05:00.030952+0000","last_became_active":"2024-01-15T18:04:58.863956+0000","last_became_peered":"2024-01-15T18:04:58.863956+0000","last_unstale":"2024-01-15T18:05:00.030952+0000","last_undegraded":"2024-01-15T18:05:00.030952+0000","last_fullsized":"2024-01-15T18:05:00.030952+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:37:40.217674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849113+0000","last_change":"2024-01-15T18:04:58.836384+0000","last_active":"2024-01-15T18:04:59.849113+0000","last_peered":"2024-01-15T18:04:59.849113+0000","last_clean":"2024-01-15T18:04:59.849113+0000","last_became_active":"2024-01-15T18:04:58.836081+0000","last_became_peered":"2024-01-15T18:04:58.836081+0000","last_unstale":"2024-01-15T18:04:59.849113+0000","last_undegraded":"2024-01-15T18:04:59.849113+0000","last_fullsized":"2024-01-15T18:04:59.849113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:15:34.376159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.846539+0000","last_change":"2024-01-15T18:04:58.868380+0000","last_active":"2024-01-15T18:04:59.846539+0000","last_peered":"2024-01-15T18:04:59.846539+0000","last_clean":"2024-01-15T18:04:59.846539+0000","last_became_active":"2024-01-15T18:04:58.868103+0000","last_became_peered":"2024-01-15T18:04:58.868103+0000","last_unstale":"2024-01-15T18:04:59.846539+0000","last_undegraded":"2024-01-15T18:04:59.846539+0000","last_fullsized":"2024-01-15T18:04:59.846539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:48:25.476425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.848460+0000","last_change":"2024-01-15T18:04:58.858404+0000","last_active":"2024-01-15T18:04:59.848460+0000","last_peered":"2024-01-15T18:04:59.848460+0000","last_clean":"2024-01-15T18:04:59.848460+0000","last_became_active":"2024-01-15T18:04:58.857755+0000","last_became_peered":"2024-01-15T18:04:58.857755+0000","last_unstale":"2024-01-15T18:04:59.848460+0000","last_undegraded":"2024-01-15T18:04:59.848460+0000","last_fullsized":"2024-01-15T18:04:59.848460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:38:44.912913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030907+0000","last_change":"2024-01-15T18:04:58.844392+0000","last_active":"2024-01-15T18:05:00.030907+0000","last_peered":"2024-01-15T18:05:00.030907+0000","last_clean":"2024-01-15T18:05:00.030907+0000","last_became_active":"2024-01-15T18:04:58.844086+0000","last_became_peered":"2024-01-15T18:04:58.844086+0000","last_unstale":"2024-01-15T18:05:00.030907+0000","last_undegraded":"2024-01-15T18:05:00.030907+0000","last_fullsized":"2024-01-15T18:05:00.030907+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:16:30.446952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031976+0000","last_change":"2024-01-15T18:04:58.861264+0000","last_active":"2024-01-15T18:05:00.031976+0000","last_peered":"2024-01-15T18:05:00.031976+0000","last_clean":"2024-01-15T18:05:00.031976+0000","last_became_active":"2024-01-15T18:04:58.860695+0000","last_became_peered":"2024-01-15T18:04:58.860695+0000","last_unstale":"2024-01-15T18:05:00.031976+0000","last_undegraded":"2024-01-15T18:05:00.031976+0000","last_fullsized":"2024-01-15T18:05:00.031976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:47:51.247174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030783+0000","last_change":"2024-01-15T18:04:58.846632+0000","last_active":"2024-01-15T18:05:00.030783+0000","last_peered":"2024-01-15T18:05:00.030783+0000","last_clean":"2024-01-15T18:05:00.030783+0000","last_became_active":"2024-01-15T18:04:58.846359+0000","last_became_peered":"2024-01-15T18:04:58.846359+0000","last_unstale":"2024-01-15T18:05:00.030783+0000","last_undegraded":"2024-01-15T18:05:00.030783+0000","last_fullsized":"2024-01-15T18:05:00.030783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:12:00.696642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031213+0000","last_change":"2024-01-15T18:04:58.858817+0000","last_active":"2024-01-15T18:05:00.031213+0000","last_peered":"2024-01-15T18:05:00.031213+0000","last_clean":"2024-01-15T18:05:00.031213+0000","last_became_active":"2024-01-15T18:04:58.858539+0000","last_became_peered":"2024-01-15T18:04:58.858539+0000","last_unstale":"2024-01-15T18:05:00.031213+0000","last_undegraded":"2024-01-15T18:05:00.031213+0000","last_fullsized":"2024-01-15T18:05:00.031213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:25:00.854241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847131+0000","last_change":"2024-01-15T18:04:58.867761+0000","last_active":"2024-01-15T18:04:59.847131+0000","last_peered":"2024-01-15T18:04:59.847131+0000","last_clean":"2024-01-15T18:04:59.847131+0000","last_became_active":"2024-01-15T18:04:58.867331+0000","last_became_peered":"2024-01-15T18:04:58.867331+0000","last_unstale":"2024-01-15T18:04:59.847131+0000","last_undegraded":"2024-01-15T18:04:59.847131+0000","last_fullsized":"2024-01-15T18:04:59.847131+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:14:47.495474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031915+0000","last_change":"2024-01-15T18:04:58.839963+0000","last_active":"2024-01-15T18:05:00.031915+0000","last_peered":"2024-01-15T18:05:00.031915+0000","last_clean":"2024-01-15T18:05:00.031915+0000","last_became_active":"2024-01-15T18:04:58.839597+0000","last_became_peered":"2024-01-15T18:04:58.839597+0000","last_unstale":"2024-01-15T18:05:00.031915+0000","last_undegraded":"2024-01-15T18:05:00.031915+0000","last_fullsized":"2024-01-15T18:05:00.031915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:47.713249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.846576+0000","last_change":"2024-01-15T18:04:58.841274+0000","last_active":"2024-01-15T18:04:59.846576+0000","last_peered":"2024-01-15T18:04:59.846576+0000","last_clean":"2024-01-15T18:04:59.846576+0000","last_became_active":"2024-01-15T18:04:58.840866+0000","last_became_peered":"2024-01-15T18:04:58.840866+0000","last_unstale":"2024-01-15T18:04:59.846576+0000","last_undegraded":"2024-01-15T18:04:59.846576+0000","last_fullsized":"2024-01-15T18:04:59.846576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:12:02.202965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032528+0000","last_change":"2024-01-15T18:04:58.855334+0000","last_active":"2024-01-15T18:05:00.032528+0000","last_peered":"2024-01-15T18:05:00.032528+0000","last_clean":"2024-01-15T18:05:00.032528+0000","last_became_active":"2024-01-15T18:04:58.855074+0000","last_became_peered":"2024-01-15T18:04:58.855074+0000","last_unstale":"2024-01-15T18:05:00.032528+0000","last_undegraded":"2024-01-15T18:05:00.032528+0000","last_fullsized":"2024-01-15T18:05:00.032528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:44:17.933005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031069+0000","last_change":"2024-01-15T18:04:58.861125+0000","last_active":"2024-01-15T18:05:00.031069+0000","last_peered":"2024-01-15T18:05:00.031069+0000","last_clean":"2024-01-15T18:05:00.031069+0000","last_became_active":"2024-01-15T18:04:58.860819+0000","last_became_peered":"2024-01-15T18:04:58.860819+0000","last_unstale":"2024-01-15T18:05:00.031069+0000","last_undegraded":"2024-01-15T18:05:00.031069+0000","last_fullsized":"2024-01-15T18:05:00.031069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:04:27.769262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849316+0000","last_change":"2024-01-15T18:04:58.837356+0000","last_active":"2024-01-15T18:04:59.849316+0000","last_peered":"2024-01-15T18:04:59.849316+0000","last_clean":"2024-01-15T18:04:59.849316+0000","last_became_active":"2024-01-15T18:04:58.837071+0000","last_became_peered":"2024-01-15T18:04:58.837071+0000","last_unstale":"2024-01-15T18:04:59.849316+0000","last_undegraded":"2024-01-15T18:04:59.849316+0000","last_fullsized":"2024-01-15T18:04:59.849316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:36.680373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032362+0000","last_change":"2024-01-15T18:04:58.858404+0000","last_active":"2024-01-15T18:05:00.032362+0000","last_peered":"2024-01-15T18:05:00.032362+0000","last_clean":"2024-01-15T18:05:00.032362+0000","last_became_active":"2024-01-15T18:04:58.857239+0000","last_became_peered":"2024-01-15T18:04:58.857239+0000","last_unstale":"2024-01-15T18:05:00.032362+0000","last_undegraded":"2024-01-15T18:05:00.032362+0000","last_fullsized":"2024-01-15T18:05:00.032362+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:04:27.769262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031219+0000","last_change":"2024-01-15T18:04:58.861031+0000","last_active":"2024-01-15T18:05:00.031219+0000","last_peered":"2024-01-15T18:05:00.031219+0000","last_clean":"2024-01-15T18:05:00.031219+0000","last_became_active":"2024-01-15T18:04:58.860718+0000","last_became_peered":"2024-01-15T18:04:58.860718+0000","last_unstale":"2024-01-15T18:05:00.031219+0000","last_undegraded":"2024-01-15T18:05:00.031219+0000","last_fullsized":"2024-01-15T18:05:00.031219+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:21:06.990646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847261+0000","last_change":"2024-01-15T18:04:58.841212+0000","last_active":"2024-01-15T18:04:59.847261+0000","last_peered":"2024-01-15T18:04:59.847261+0000","last_clean":"2024-01-15T18:04:59.847261+0000","last_became_active":"2024-01-15T18:04:58.840913+0000","last_became_peered":"2024-01-15T18:04:58.840913+0000","last_unstale":"2024-01-15T18:04:59.847261+0000","last_undegraded":"2024-01-15T18:04:59.847261+0000","last_fullsized":"2024-01-15T18:04:59.847261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:17:56.224397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030927+0000","last_change":"2024-01-15T18:04:58.847922+0000","last_active":"2024-01-15T18:05:00.030927+0000","last_peered":"2024-01-15T18:05:00.030927+0000","last_clean":"2024-01-15T18:05:00.030927+0000","last_became_active":"2024-01-15T18:04:58.847596+0000","last_became_peered":"2024-01-15T18:04:58.847596+0000","last_unstale":"2024-01-15T18:05:00.030927+0000","last_undegraded":"2024-01-15T18:05:00.030927+0000","last_fullsized":"2024-01-15T18:05:00.030927+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:24:34.638590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030579+0000","last_change":"2024-01-15T18:04:58.866119+0000","last_active":"2024-01-15T18:05:00.030579+0000","last_peered":"2024-01-15T18:05:00.030579+0000","last_clean":"2024-01-15T18:05:00.030579+0000","last_became_active":"2024-01-15T18:04:58.865811+0000","last_became_peered":"2024-01-15T18:04:58.865811+0000","last_unstale":"2024-01-15T18:05:00.030579+0000","last_undegraded":"2024-01-15T18:05:00.030579+0000","last_fullsized":"2024-01-15T18:05:00.030579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:10.719289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847206+0000","last_change":"2024-01-15T18:04:58.839281+0000","last_active":"2024-01-15T18:04:59.847206+0000","last_peered":"2024-01-15T18:04:59.847206+0000","last_clean":"2024-01-15T18:04:59.847206+0000","last_became_active":"2024-01-15T18:04:58.838304+0000","last_became_peered":"2024-01-15T18:04:58.838304+0000","last_unstale":"2024-01-15T18:04:59.847206+0000","last_undegraded":"2024-01-15T18:04:59.847206+0000","last_fullsized":"2024-01-15T18:04:59.847206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:08:43.279198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032260+0000","last_change":"2024-01-15T18:04:57.840527+0000","last_active":"2024-01-15T18:05:00.032260+0000","last_peered":"2024-01-15T18:05:00.032260+0000","last_clean":"2024-01-15T18:05:00.032260+0000","last_became_active":"2024-01-15T18:04:57.840153+0000","last_became_peered":"2024-01-15T18:04:57.840153+0000","last_unstale":"2024-01-15T18:05:00.032260+0000","last_undegraded":"2024-01-15T18:05:00.032260+0000","last_fullsized":"2024-01-15T18:05:00.032260+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:17:50.737228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031360+0000","last_change":"2024-01-15T18:04:58.859368+0000","last_active":"2024-01-15T18:05:00.031360+0000","last_peered":"2024-01-15T18:05:00.031360+0000","last_clean":"2024-01-15T18:05:00.031360+0000","last_became_active":"2024-01-15T18:04:58.859112+0000","last_became_peered":"2024-01-15T18:04:58.859112+0000","last_unstale":"2024-01-15T18:05:00.031360+0000","last_undegraded":"2024-01-15T18:05:00.031360+0000","last_fullsized":"2024-01-15T18:05:00.031360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:48:25.476425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030936+0000","last_change":"2024-01-15T18:04:57.843777+0000","last_active":"2024-01-15T18:05:00.030936+0000","last_peered":"2024-01-15T18:05:00.030936+0000","last_clean":"2024-01-15T18:05:00.030936+0000","last_became_active":"2024-01-15T18:04:57.843062+0000","last_became_peered":"2024-01-15T18:04:57.843062+0000","last_unstale":"2024-01-15T18:05:00.030936+0000","last_undegraded":"2024-01-15T18:05:00.030936+0000","last_fullsized":"2024-01-15T18:05:00.030936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:58:15.645144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031986+0000","last_change":"2024-01-15T18:04:57.833749+0000","last_active":"2024-01-15T18:05:00.031986+0000","last_peered":"2024-01-15T18:05:00.031986+0000","last_clean":"2024-01-15T18:05:00.031986+0000","last_became_active":"2024-01-15T18:04:57.832655+0000","last_became_peered":"2024-01-15T18:04:57.832655+0000","last_unstale":"2024-01-15T18:05:00.031986+0000","last_undegraded":"2024-01-15T18:05:00.031986+0000","last_fullsized":"2024-01-15T18:05:00.031986+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:42:16.957837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031910+0000","last_change":"2024-01-15T18:04:58.862938+0000","last_active":"2024-01-15T18:05:00.031910+0000","last_peered":"2024-01-15T18:05:00.031910+0000","last_clean":"2024-01-15T18:05:00.031910+0000","last_became_active":"2024-01-15T18:04:58.862733+0000","last_became_peered":"2024-01-15T18:04:58.862733+0000","last_unstale":"2024-01-15T18:05:00.031910+0000","last_undegraded":"2024-01-15T18:05:00.031910+0000","last_fullsized":"2024-01-15T18:05:00.031910+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:10.719289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.161858+0000","last_change":"2024-01-15T18:04:57.828200+0000","last_active":"2024-01-15T18:04:59.161858+0000","last_peered":"2024-01-15T18:04:59.161858+0000","last_clean":"2024-01-15T18:04:59.161858+0000","last_became_active":"2024-01-15T18:04:57.827883+0000","last_became_peered":"2024-01-15T18:04:57.827883+0000","last_unstale":"2024-01-15T18:04:59.161858+0000","last_undegraded":"2024-01-15T18:04:59.161858+0000","last_fullsized":"2024-01-15T18:04:59.161858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:48:16.392862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849021+0000","last_change":"2024-01-15T18:04:58.842387+0000","last_active":"2024-01-15T18:04:59.849021+0000","last_peered":"2024-01-15T18:04:59.849021+0000","last_clean":"2024-01-15T18:04:59.849021+0000","last_became_active":"2024-01-15T18:04:58.842093+0000","last_became_peered":"2024-01-15T18:04:58.842093+0000","last_unstale":"2024-01-15T18:04:59.849021+0000","last_undegraded":"2024-01-15T18:04:59.849021+0000","last_fullsized":"2024-01-15T18:04:59.849021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:47.713249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030983+0000","last_change":"2024-01-15T18:04:57.837812+0000","last_active":"2024-01-15T18:05:00.030983+0000","last_peered":"2024-01-15T18:05:00.030983+0000","last_clean":"2024-01-15T18:05:00.030983+0000","last_became_active":"2024-01-15T18:04:57.837524+0000","last_became_peered":"2024-01-15T18:04:57.837524+0000","last_unstale":"2024-01-15T18:05:00.030983+0000","last_undegraded":"2024-01-15T18:05:00.030983+0000","last_fullsized":"2024-01-15T18:05:00.030983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:37:44.430593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.848322+0000","last_change":"2024-01-15T18:04:58.854115+0000","last_active":"2024-01-15T18:04:59.848322+0000","last_peered":"2024-01-15T18:04:59.848322+0000","last_clean":"2024-01-15T18:04:59.848322+0000","last_became_active":"2024-01-15T18:04:58.853039+0000","last_became_peered":"2024-01-15T18:04:58.853039+0000","last_unstale":"2024-01-15T18:04:59.848322+0000","last_undegraded":"2024-01-15T18:04:59.848322+0000","last_fullsized":"2024-01-15T18:04:59.848322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:36:36.331502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.162008+0000","last_change":"2024-01-15T18:04:57.839191+0000","last_active":"2024-01-15T18:04:59.162008+0000","last_peered":"2024-01-15T18:04:59.162008+0000","last_clean":"2024-01-15T18:04:59.162008+0000","last_became_active":"2024-01-15T18:04:57.838884+0000","last_became_peered":"2024-01-15T18:04:57.838884+0000","last_unstale":"2024-01-15T18:04:59.162008+0000","last_undegraded":"2024-01-15T18:04:59.162008+0000","last_fullsized":"2024-01-15T18:04:59.162008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:37:39.211256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031433+0000","last_change":"2024-01-15T18:04:58.860345+0000","last_active":"2024-01-15T18:05:00.031433+0000","last_peered":"2024-01-15T18:05:00.031433+0000","last_clean":"2024-01-15T18:05:00.031433+0000","last_became_active":"2024-01-15T18:04:58.859997+0000","last_became_peered":"2024-01-15T18:04:58.859997+0000","last_unstale":"2024-01-15T18:05:00.031433+0000","last_undegraded":"2024-01-15T18:05:00.031433+0000","last_fullsized":"2024-01-15T18:05:00.031433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:05:31.164011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849266+0000","last_change":"2024-01-15T18:04:57.835121+0000","last_active":"2024-01-15T18:04:59.849266+0000","last_peered":"2024-01-15T18:04:59.849266+0000","last_clean":"2024-01-15T18:04:59.849266+0000","last_became_active":"2024-01-15T18:04:57.834822+0000","last_became_peered":"2024-01-15T18:04:57.834822+0000","last_unstale":"2024-01-15T18:04:59.849266+0000","last_undegraded":"2024-01-15T18:04:59.849266+0000","last_fullsized":"2024-01-15T18:04:59.849266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:17:50.737228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847554+0000","last_change":"2024-01-15T18:04:58.859000+0000","last_active":"2024-01-15T18:04:59.847554+0000","last_peered":"2024-01-15T18:04:59.847554+0000","last_clean":"2024-01-15T18:04:59.847554+0000","last_became_active":"2024-01-15T18:04:58.854577+0000","last_became_peered":"2024-01-15T18:04:58.854577+0000","last_unstale":"2024-01-15T18:04:59.847554+0000","last_undegraded":"2024-01-15T18:04:59.847554+0000","last_fullsized":"2024-01-15T18:04:59.847554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:24:34.638590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030767+0000","last_change":"2024-01-15T18:04:57.836570+0000","last_active":"2024-01-15T18:05:00.030767+0000","last_peered":"2024-01-15T18:05:00.030767+0000","last_clean":"2024-01-15T18:05:00.030767+0000","last_became_active":"2024-01-15T18:04:57.836027+0000","last_became_peered":"2024-01-15T18:04:57.836027+0000","last_unstale":"2024-01-15T18:05:00.030767+0000","last_undegraded":"2024-01-15T18:05:00.030767+0000","last_fullsized":"2024-01-15T18:05:00.030767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:17:50.737228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031727+0000","last_change":"2024-01-15T18:04:58.872987+0000","last_active":"2024-01-15T18:05:00.031727+0000","last_peered":"2024-01-15T18:05:00.031727+0000","last_clean":"2024-01-15T18:05:00.031727+0000","last_became_active":"2024-01-15T18:04:58.872671+0000","last_became_peered":"2024-01-15T18:04:58.872671+0000","last_unstale":"2024-01-15T18:05:00.031727+0000","last_undegraded":"2024-01-15T18:05:00.031727+0000","last_fullsized":"2024-01-15T18:05:00.031727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:42:12.458758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.159257+0000","last_change":"2024-01-15T18:04:57.839505+0000","last_active":"2024-01-15T18:04:59.159257+0000","last_peered":"2024-01-15T18:04:59.159257+0000","last_clean":"2024-01-15T18:04:59.159257+0000","last_became_active":"2024-01-15T18:04:57.839194+0000","last_became_peered":"2024-01-15T18:04:57.839194+0000","last_unstale":"2024-01-15T18:04:59.159257+0000","last_undegraded":"2024-01-15T18:04:59.159257+0000","last_fullsized":"2024-01-15T18:04:59.159257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:10.960718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031101+0000","last_change":"2024-01-15T18:04:58.858064+0000","last_active":"2024-01-15T18:05:00.031101+0000","last_peered":"2024-01-15T18:05:00.031101+0000","last_clean":"2024-01-15T18:05:00.031101+0000","last_became_active":"2024-01-15T18:04:58.857697+0000","last_became_peered":"2024-01-15T18:04:58.857697+0000","last_unstale":"2024-01-15T18:05:00.031101+0000","last_undegraded":"2024-01-15T18:05:00.031101+0000","last_fullsized":"2024-01-15T18:05:00.031101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:21:44.926560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.161896+0000","last_change":"2024-01-15T18:04:57.836035+0000","last_active":"2024-01-15T18:04:59.161896+0000","last_peered":"2024-01-15T18:04:59.161896+0000","last_clean":"2024-01-15T18:04:59.161896+0000","last_became_active":"2024-01-15T18:04:57.835722+0000","last_became_peered":"2024-01-15T18:04:57.835722+0000","last_unstale":"2024-01-15T18:04:59.161896+0000","last_undegraded":"2024-01-15T18:04:59.161896+0000","last_fullsized":"2024-01-15T18:04:59.161896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:24:59.847823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031861+0000","last_change":"2024-01-15T18:04:58.852717+0000","last_active":"2024-01-15T18:05:00.031861+0000","last_peered":"2024-01-15T18:05:00.031861+0000","last_clean":"2024-01-15T18:05:00.031861+0000","last_became_active":"2024-01-15T18:04:58.852351+0000","last_became_peered":"2024-01-15T18:04:58.852351+0000","last_unstale":"2024-01-15T18:05:00.031861+0000","last_undegraded":"2024-01-15T18:05:00.031861+0000","last_fullsized":"2024-01-15T18:05:00.031861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:47:03.033950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030528+0000","last_change":"2024-01-15T18:04:57.831525+0000","last_active":"2024-01-15T18:05:00.030528+0000","last_peered":"2024-01-15T18:05:00.030528+0000","last_clean":"2024-01-15T18:05:00.030528+0000","last_became_active":"2024-01-15T18:04:57.831053+0000","last_became_peered":"2024-01-15T18:04:57.831053+0000","last_unstale":"2024-01-15T18:05:00.030528+0000","last_undegraded":"2024-01-15T18:05:00.030528+0000","last_fullsized":"2024-01-15T18:05:00.030528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:24:33.632172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030589+0000","last_change":"2024-01-15T18:04:58.866415+0000","last_active":"2024-01-15T18:05:00.030589+0000","last_peered":"2024-01-15T18:05:00.030589+0000","last_clean":"2024-01-15T18:05:00.030589+0000","last_became_active":"2024-01-15T18:04:58.865778+0000","last_became_peered":"2024-01-15T18:04:58.865778+0000","last_unstale":"2024-01-15T18:05:00.030589+0000","last_undegraded":"2024-01-15T18:05:00.030589+0000","last_fullsized":"2024-01-15T18:05:00.030589+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:47.713249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030622+0000","last_change":"2024-01-15T18:04:57.821766+0000","last_active":"2024-01-15T18:05:00.030622+0000","last_peered":"2024-01-15T18:05:00.030622+0000","last_clean":"2024-01-15T18:05:00.030622+0000","last_became_active":"2024-01-15T18:04:57.820957+0000","last_became_peered":"2024-01-15T18:04:57.820957+0000","last_unstale":"2024-01-15T18:05:00.030622+0000","last_undegraded":"2024-01-15T18:05:00.030622+0000","last_fullsized":"2024-01-15T18:05:00.030622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:08:42.272780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031415+0000","last_change":"2024-01-15T18:04:58.862675+0000","last_active":"2024-01-15T18:05:00.031415+0000","last_peered":"2024-01-15T18:05:00.031415+0000","last_clean":"2024-01-15T18:05:00.031415+0000","last_became_active":"2024-01-15T18:04:58.862420+0000","last_became_peered":"2024-01-15T18:04:58.862420+0000","last_unstale":"2024-01-15T18:05:00.031415+0000","last_undegraded":"2024-01-15T18:05:00.031415+0000","last_fullsized":"2024-01-15T18:05:00.031415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:44:17.933005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847491+0000","last_change":"2024-01-15T18:04:57.839139+0000","last_active":"2024-01-15T18:04:59.847491+0000","last_peered":"2024-01-15T18:04:59.847491+0000","last_clean":"2024-01-15T18:04:59.847491+0000","last_became_active":"2024-01-15T18:04:57.838847+0000","last_became_peered":"2024-01-15T18:04:57.838847+0000","last_unstale":"2024-01-15T18:04:59.847491+0000","last_undegraded":"2024-01-15T18:04:59.847491+0000","last_fullsized":"2024-01-15T18:04:59.847491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:44:30.299838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031909+0000","last_change":"2024-01-15T18:04:58.870909+0000","last_active":"2024-01-15T18:05:00.031909+0000","last_peered":"2024-01-15T18:05:00.031909+0000","last_clean":"2024-01-15T18:05:00.031909+0000","last_became_active":"2024-01-15T18:04:58.868734+0000","last_became_peered":"2024-01-15T18:04:58.868734+0000","last_unstale":"2024-01-15T18:05:00.031909+0000","last_undegraded":"2024-01-15T18:05:00.031909+0000","last_fullsized":"2024-01-15T18:05:00.031909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:58:21.135065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030997+0000","last_change":"2024-01-15T18:04:57.829932+0000","last_active":"2024-01-15T18:05:00.030997+0000","last_peered":"2024-01-15T18:05:00.030997+0000","last_clean":"2024-01-15T18:05:00.030997+0000","last_became_active":"2024-01-15T18:04:57.829636+0000","last_became_peered":"2024-01-15T18:04:57.829636+0000","last_unstale":"2024-01-15T18:05:00.030997+0000","last_undegraded":"2024-01-15T18:05:00.030997+0000","last_fullsized":"2024-01-15T18:05:00.030997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:04:13.242693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032156+0000","last_change":"2024-01-15T18:04:58.869477+0000","last_active":"2024-01-15T18:05:00.032156+0000","last_peered":"2024-01-15T18:05:00.032156+0000","last_clean":"2024-01-15T18:05:00.032156+0000","last_became_active":"2024-01-15T18:04:58.869133+0000","last_became_peered":"2024-01-15T18:04:58.869133+0000","last_unstale":"2024-01-15T18:05:00.032156+0000","last_undegraded":"2024-01-15T18:05:00.032156+0000","last_fullsized":"2024-01-15T18:05:00.032156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:39:08.175871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849313+0000","last_change":"2024-01-15T18:04:57.830731+0000","last_active":"2024-01-15T18:04:59.849313+0000","last_peered":"2024-01-15T18:04:59.849313+0000","last_clean":"2024-01-15T18:04:59.849313+0000","last_became_active":"2024-01-15T18:04:57.830234+0000","last_became_peered":"2024-01-15T18:04:57.830234+0000","last_unstale":"2024-01-15T18:04:59.849313+0000","last_undegraded":"2024-01-15T18:04:59.849313+0000","last_fullsized":"2024-01-15T18:04:59.849313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:37:44.430593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847812+0000","last_change":"2024-01-15T18:04:58.843965+0000","last_active":"2024-01-15T18:04:59.847812+0000","last_peered":"2024-01-15T18:04:59.847812+0000","last_clean":"2024-01-15T18:04:59.847812+0000","last_became_active":"2024-01-15T18:04:58.841866+0000","last_became_peered":"2024-01-15T18:04:58.841866+0000","last_unstale":"2024-01-15T18:04:59.847812+0000","last_undegraded":"2024-01-15T18:04:59.847812+0000","last_fullsized":"2024-01-15T18:04:59.847812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:16:41.986399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030724+0000","last_change":"2024-01-15T18:04:57.836644+0000","last_active":"2024-01-15T18:05:00.030724+0000","last_peered":"2024-01-15T18:05:00.030724+0000","last_clean":"2024-01-15T18:05:00.030724+0000","last_became_active":"2024-01-15T18:04:57.835872+0000","last_became_peered":"2024-01-15T18:04:57.835872+0000","last_unstale":"2024-01-15T18:05:00.030724+0000","last_undegraded":"2024-01-15T18:05:00.030724+0000","last_fullsized":"2024-01-15T18:05:00.030724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:44:30.299838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031872+0000","last_change":"2024-01-15T18:04:58.857962+0000","last_active":"2024-01-15T18:05:00.031872+0000","last_peered":"2024-01-15T18:05:00.031872+0000","last_clean":"2024-01-15T18:05:00.031872+0000","last_became_active":"2024-01-15T18:04:58.857644+0000","last_became_peered":"2024-01-15T18:04:58.857644+0000","last_unstale":"2024-01-15T18:05:00.031872+0000","last_undegraded":"2024-01-15T18:05:00.031872+0000","last_fullsized":"2024-01-15T18:05:00.031872+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:02:19.202437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.836746+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:04:58.836746+0000","last_peered":"2024-01-15T18:04:58.836746+0000","last_clean":"2024-01-15T18:04:58.836746+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:04:58.836746+0000","last_undegraded":"2024-01-15T18:04:58.836746+0000","last_fullsized":"2024-01-15T18:04:58.836746+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.161917+0000","last_change":"2024-01-15T18:04:57.829926+0000","last_active":"2024-01-15T18:04:59.161917+0000","last_peered":"2024-01-15T18:04:59.161917+0000","last_clean":"2024-01-15T18:04:59.161917+0000","last_became_active":"2024-01-15T18:04:57.829578+0000","last_became_peered":"2024-01-15T18:04:57.829578+0000","last_unstale":"2024-01-15T18:04:59.161917+0000","last_undegraded":"2024-01-15T18:04:59.161917+0000","last_fullsized":"2024-01-15T18:04:59.161917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:55:52.024992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032132+0000","last_change":"2024-01-15T18:04:58.862009+0000","last_active":"2024-01-15T18:05:00.032132+0000","last_peered":"2024-01-15T18:05:00.032132+0000","last_clean":"2024-01-15T18:05:00.032132+0000","last_became_active":"2024-01-15T18:04:58.861745+0000","last_became_peered":"2024-01-15T18:04:58.861745+0000","last_unstale":"2024-01-15T18:05:00.032132+0000","last_undegraded":"2024-01-15T18:05:00.032132+0000","last_fullsized":"2024-01-15T18:05:00.032132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:15:34.376159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031331+0000","last_change":"2024-01-15T18:04:57.842536+0000","last_active":"2024-01-15T18:05:00.031331+0000","last_peered":"2024-01-15T18:05:00.031331+0000","last_clean":"2024-01-15T18:05:00.031331+0000","last_became_active":"2024-01-15T18:04:57.842236+0000","last_became_peered":"2024-01-15T18:04:57.842236+0000","last_unstale":"2024-01-15T18:05:00.031331+0000","last_undegraded":"2024-01-15T18:05:00.031331+0000","last_fullsized":"2024-01-15T18:05:00.031331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:24:33.632172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030699+0000","last_change":"2024-01-15T18:04:58.859545+0000","last_active":"2024-01-15T18:05:00.030699+0000","last_peered":"2024-01-15T18:05:00.030699+0000","last_clean":"2024-01-15T18:05:00.030699+0000","last_became_active":"2024-01-15T18:04:58.859282+0000","last_became_peered":"2024-01-15T18:04:58.859282+0000","last_unstale":"2024-01-15T18:05:00.030699+0000","last_undegraded":"2024-01-15T18:05:00.030699+0000","last_fullsized":"2024-01-15T18:05:00.030699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:29:02.731583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030970+0000","last_change":"2024-01-15T18:04:57.826351+0000","last_active":"2024-01-15T18:05:00.030970+0000","last_peered":"2024-01-15T18:05:00.030970+0000","last_clean":"2024-01-15T18:05:00.030970+0000","last_became_active":"2024-01-15T18:04:57.825983+0000","last_became_peered":"2024-01-15T18:04:57.825983+0000","last_unstale":"2024-01-15T18:05:00.030970+0000","last_undegraded":"2024-01-15T18:05:00.030970+0000","last_fullsized":"2024-01-15T18:05:00.030970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:30:11.561882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849021+0000","last_change":"2024-01-15T18:04:58.834900+0000","last_active":"2024-01-15T18:04:59.849021+0000","last_peered":"2024-01-15T18:04:59.849021+0000","last_clean":"2024-01-15T18:04:59.849021+0000","last_became_active":"2024-01-15T18:04:58.834561+0000","last_became_peered":"2024-01-15T18:04:58.834561+0000","last_unstale":"2024-01-15T18:04:59.849021+0000","last_undegraded":"2024-01-15T18:04:59.849021+0000","last_fullsized":"2024-01-15T18:04:59.849021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:10.719289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.161907+0000","last_change":"2024-01-15T18:04:57.840353+0000","last_active":"2024-01-15T18:04:59.161907+0000","last_peered":"2024-01-15T18:04:59.161907+0000","last_clean":"2024-01-15T18:04:59.161907+0000","last_became_active":"2024-01-15T18:04:57.840032+0000","last_became_peered":"2024-01-15T18:04:57.840032+0000","last_unstale":"2024-01-15T18:04:59.161907+0000","last_undegraded":"2024-01-15T18:04:59.161907+0000","last_fullsized":"2024-01-15T18:04:59.161907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:21:05.984228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847638+0000","last_change":"2024-01-15T18:04:58.862393+0000","last_active":"2024-01-15T18:04:59.847638+0000","last_peered":"2024-01-15T18:04:59.847638+0000","last_clean":"2024-01-15T18:04:59.847638+0000","last_became_active":"2024-01-15T18:04:58.861776+0000","last_became_peered":"2024-01-15T18:04:58.861776+0000","last_unstale":"2024-01-15T18:04:59.847638+0000","last_undegraded":"2024-01-15T18:04:59.847638+0000","last_fullsized":"2024-01-15T18:04:59.847638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:58:16.651562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849271+0000","last_change":"2024-01-15T18:04:57.832247+0000","last_active":"2024-01-15T18:04:59.849271+0000","last_peered":"2024-01-15T18:04:59.849271+0000","last_clean":"2024-01-15T18:04:59.849271+0000","last_became_active":"2024-01-15T18:04:57.831962+0000","last_became_peered":"2024-01-15T18:04:57.831962+0000","last_unstale":"2024-01-15T18:04:59.849271+0000","last_undegraded":"2024-01-15T18:04:59.849271+0000","last_fullsized":"2024-01-15T18:04:59.849271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:08:42.272780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.848124+0000","last_change":"2024-01-15T18:04:58.842320+0000","last_active":"2024-01-15T18:04:59.848124+0000","last_peered":"2024-01-15T18:04:59.848124+0000","last_clean":"2024-01-15T18:04:59.848124+0000","last_became_active":"2024-01-15T18:04:58.841887+0000","last_became_peered":"2024-01-15T18:04:58.841887+0000","last_unstale":"2024-01-15T18:04:59.848124+0000","last_undegraded":"2024-01-15T18:04:59.848124+0000","last_fullsized":"2024-01-15T18:04:59.848124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:19.608361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032034+0000","last_change":"2024-01-15T18:04:57.843600+0000","last_active":"2024-01-15T18:05:00.032034+0000","last_peered":"2024-01-15T18:05:00.032034+0000","last_clean":"2024-01-15T18:05:00.032034+0000","last_became_active":"2024-01-15T18:04:57.843179+0000","last_became_peered":"2024-01-15T18:04:57.843179+0000","last_unstale":"2024-01-15T18:05:00.032034+0000","last_undegraded":"2024-01-15T18:05:00.032034+0000","last_fullsized":"2024-01-15T18:05:00.032034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:10:23.513336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849104+0000","last_change":"2024-01-15T18:04:58.834860+0000","last_active":"2024-01-15T18:04:59.849104+0000","last_peered":"2024-01-15T18:04:59.849104+0000","last_clean":"2024-01-15T18:04:59.849104+0000","last_became_active":"2024-01-15T18:04:58.834570+0000","last_became_peered":"2024-01-15T18:04:58.834570+0000","last_unstale":"2024-01-15T18:04:59.849104+0000","last_undegraded":"2024-01-15T18:04:59.849104+0000","last_fullsized":"2024-01-15T18:04:59.849104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:12:00.696642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.162824+0000","last_change":"2024-01-15T18:04:57.832391+0000","last_active":"2024-01-15T18:05:00.162824+0000","last_peered":"2024-01-15T18:05:00.162824+0000","last_clean":"2024-01-15T18:05:00.162824+0000","last_became_active":"2024-01-15T18:04:57.832005+0000","last_became_peered":"2024-01-15T18:04:57.832005+0000","last_unstale":"2024-01-15T18:05:00.162824+0000","last_undegraded":"2024-01-15T18:05:00.162824+0000","last_fullsized":"2024-01-15T18:05:00.162824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:24:33.632172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032511+0000","last_change":"2024-01-15T18:04:58.863289+0000","last_active":"2024-01-15T18:05:00.032511+0000","last_peered":"2024-01-15T18:05:00.032511+0000","last_clean":"2024-01-15T18:05:00.032511+0000","last_became_active":"2024-01-15T18:04:58.862985+0000","last_became_peered":"2024-01-15T18:04:58.862985+0000","last_unstale":"2024-01-15T18:05:00.032511+0000","last_undegraded":"2024-01-15T18:05:00.032511+0000","last_fullsized":"2024-01-15T18:05:00.032511+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:48:27.499654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031100+0000","last_change":"2024-01-15T18:04:57.835018+0000","last_active":"2024-01-15T18:05:00.031100+0000","last_peered":"2024-01-15T18:05:00.031100+0000","last_clean":"2024-01-15T18:05:00.031100+0000","last_became_active":"2024-01-15T18:04:57.834706+0000","last_became_peered":"2024-01-15T18:04:57.834706+0000","last_unstale":"2024-01-15T18:05:00.031100+0000","last_undegraded":"2024-01-15T18:05:00.031100+0000","last_fullsized":"2024-01-15T18:05:00.031100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:11:48.897513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031377+0000","last_change":"2024-01-15T18:04:58.866805+0000","last_active":"2024-01-15T18:05:00.031377+0000","last_peered":"2024-01-15T18:05:00.031377+0000","last_clean":"2024-01-15T18:05:00.031377+0000","last_became_active":"2024-01-15T18:04:58.866475+0000","last_became_peered":"2024-01-15T18:04:58.866475+0000","last_unstale":"2024-01-15T18:05:00.031377+0000","last_undegraded":"2024-01-15T18:05:00.031377+0000","last_fullsized":"2024-01-15T18:05:00.031377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:12:02.202965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031326+0000","last_change":"2024-01-15T18:04:57.839780+0000","last_active":"2024-01-15T18:05:00.031326+0000","last_peered":"2024-01-15T18:05:00.031326+0000","last_clean":"2024-01-15T18:05:00.031326+0000","last_became_active":"2024-01-15T18:04:57.839392+0000","last_became_peered":"2024-01-15T18:04:57.839392+0000","last_unstale":"2024-01-15T18:05:00.031326+0000","last_undegraded":"2024-01-15T18:05:00.031326+0000","last_fullsized":"2024-01-15T18:05:00.031326+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:16:29.440534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847071+0000","last_change":"2024-01-15T18:04:58.835003+0000","last_active":"2024-01-15T18:04:59.847071+0000","last_peered":"2024-01-15T18:04:59.847071+0000","last_clean":"2024-01-15T18:04:59.847071+0000","last_became_active":"2024-01-15T18:04:58.834678+0000","last_became_peered":"2024-01-15T18:04:58.834678+0000","last_unstale":"2024-01-15T18:04:59.847071+0000","last_undegraded":"2024-01-15T18:04:59.847071+0000","last_fullsized":"2024-01-15T18:04:59.847071+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:04:14.249111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849091+0000","last_change":"2024-01-15T18:04:57.837454+0000","last_active":"2024-01-15T18:04:59.849091+0000","last_peered":"2024-01-15T18:04:59.849091+0000","last_clean":"2024-01-15T18:04:59.849091+0000","last_became_active":"2024-01-15T18:04:57.836947+0000","last_became_peered":"2024-01-15T18:04:57.836947+0000","last_unstale":"2024-01-15T18:04:59.849091+0000","last_undegraded":"2024-01-15T18:04:59.849091+0000","last_fullsized":"2024-01-15T18:04:59.849091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:11:48.897513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031977+0000","last_change":"2024-01-15T18:04:58.854684+0000","last_active":"2024-01-15T18:05:00.031977+0000","last_peered":"2024-01-15T18:05:00.031977+0000","last_clean":"2024-01-15T18:05:00.031977+0000","last_became_active":"2024-01-15T18:04:58.844371+0000","last_became_peered":"2024-01-15T18:04:58.844371+0000","last_unstale":"2024-01-15T18:05:00.031977+0000","last_undegraded":"2024-01-15T18:05:00.031977+0000","last_fullsized":"2024-01-15T18:05:00.031977+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:12:00.696642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849194+0000","last_change":"2024-01-15T18:04:57.837432+0000","last_active":"2024-01-15T18:04:59.849194+0000","last_peered":"2024-01-15T18:04:59.849194+0000","last_clean":"2024-01-15T18:04:59.849194+0000","last_became_active":"2024-01-15T18:04:57.837105+0000","last_became_peered":"2024-01-15T18:04:57.837105+0000","last_unstale":"2024-01-15T18:04:59.849194+0000","last_undegraded":"2024-01-15T18:04:59.849194+0000","last_fullsized":"2024-01-15T18:04:59.849194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:04:13.242693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032216+0000","last_change":"2024-01-15T18:04:58.859567+0000","last_active":"2024-01-15T18:05:00.032216+0000","last_peered":"2024-01-15T18:05:00.032216+0000","last_clean":"2024-01-15T18:05:00.032216+0000","last_became_active":"2024-01-15T18:04:58.859199+0000","last_became_peered":"2024-01-15T18:04:58.859199+0000","last_unstale":"2024-01-15T18:05:00.032216+0000","last_undegraded":"2024-01-15T18:05:00.032216+0000","last_fullsized":"2024-01-15T18:05:00.032216+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:10:24.519754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031791+0000","last_change":"2024-01-15T18:04:57.843450+0000","last_active":"2024-01-15T18:05:00.031791+0000","last_peered":"2024-01-15T18:05:00.031791+0000","last_clean":"2024-01-15T18:05:00.031791+0000","last_became_active":"2024-01-15T18:04:57.841883+0000","last_became_peered":"2024-01-15T18:04:57.841883+0000","last_unstale":"2024-01-15T18:05:00.031791+0000","last_undegraded":"2024-01-15T18:05:00.031791+0000","last_fullsized":"2024-01-15T18:05:00.031791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:17:50.737228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030851+0000","last_change":"2024-01-15T18:04:58.845515+0000","last_active":"2024-01-15T18:05:00.030851+0000","last_peered":"2024-01-15T18:05:00.030851+0000","last_clean":"2024-01-15T18:05:00.030851+0000","last_became_active":"2024-01-15T18:04:58.845242+0000","last_became_peered":"2024-01-15T18:04:58.845242+0000","last_unstale":"2024-01-15T18:05:00.030851+0000","last_undegraded":"2024-01-15T18:05:00.030851+0000","last_fullsized":"2024-01-15T18:05:00.030851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:36.680373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.848091+0000","last_change":"2024-01-15T18:04:58.840618+0000","last_active":"2024-01-15T18:04:59.848091+0000","last_peered":"2024-01-15T18:04:59.848091+0000","last_clean":"2024-01-15T18:04:59.848091+0000","last_became_active":"2024-01-15T18:04:58.840306+0000","last_became_peered":"2024-01-15T18:04:58.840306+0000","last_unstale":"2024-01-15T18:04:59.848091+0000","last_undegraded":"2024-01-15T18:04:59.848091+0000","last_fullsized":"2024-01-15T18:04:59.848091+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:30:56.054090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031572+0000","last_change":"2024-01-15T18:04:57.842676+0000","last_active":"2024-01-15T18:05:00.031572+0000","last_peered":"2024-01-15T18:05:00.031572+0000","last_clean":"2024-01-15T18:05:00.031572+0000","last_became_active":"2024-01-15T18:04:57.842402+0000","last_became_peered":"2024-01-15T18:04:57.842402+0000","last_unstale":"2024-01-15T18:05:00.031572+0000","last_undegraded":"2024-01-15T18:05:00.031572+0000","last_fullsized":"2024-01-15T18:05:00.031572+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:09.712871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847691+0000","last_change":"2024-01-15T18:04:58.837433+0000","last_active":"2024-01-15T18:04:59.847691+0000","last_peered":"2024-01-15T18:04:59.847691+0000","last_clean":"2024-01-15T18:04:59.847691+0000","last_became_active":"2024-01-15T18:04:58.836989+0000","last_became_peered":"2024-01-15T18:04:58.836989+0000","last_unstale":"2024-01-15T18:04:59.847691+0000","last_undegraded":"2024-01-15T18:04:59.847691+0000","last_fullsized":"2024-01-15T18:04:59.847691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:25:24.113321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031908+0000","last_change":"2024-01-15T18:04:57.834129+0000","last_active":"2024-01-15T18:05:00.031908+0000","last_peered":"2024-01-15T18:05:00.031908+0000","last_clean":"2024-01-15T18:05:00.031908+0000","last_became_active":"2024-01-15T18:04:57.833804+0000","last_became_peered":"2024-01-15T18:04:57.833804+0000","last_unstale":"2024-01-15T18:05:00.031908+0000","last_undegraded":"2024-01-15T18:05:00.031908+0000","last_fullsized":"2024-01-15T18:05:00.031908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:11:59.690224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031347+0000","last_change":"2024-01-15T18:04:58.860153+0000","last_active":"2024-01-15T18:05:00.031347+0000","last_peered":"2024-01-15T18:05:00.031347+0000","last_clean":"2024-01-15T18:05:00.031347+0000","last_became_active":"2024-01-15T18:04:58.855401+0000","last_became_peered":"2024-01-15T18:04:58.855401+0000","last_unstale":"2024-01-15T18:05:00.031347+0000","last_undegraded":"2024-01-15T18:05:00.031347+0000","last_fullsized":"2024-01-15T18:05:00.031347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:48:17.399280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032517+0000","last_change":"2024-01-15T18:04:57.830136+0000","last_active":"2024-01-15T18:05:00.032517+0000","last_peered":"2024-01-15T18:05:00.032517+0000","last_clean":"2024-01-15T18:05:00.032517+0000","last_became_active":"2024-01-15T18:04:57.829333+0000","last_became_peered":"2024-01-15T18:04:57.829333+0000","last_unstale":"2024-01-15T18:05:00.032517+0000","last_undegraded":"2024-01-15T18:05:00.032517+0000","last_fullsized":"2024-01-15T18:05:00.032517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:37:44.430593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032421+0000","last_change":"2024-01-15T18:04:58.855872+0000","last_active":"2024-01-15T18:05:00.032421+0000","last_peered":"2024-01-15T18:05:00.032421+0000","last_clean":"2024-01-15T18:05:00.032421+0000","last_became_active":"2024-01-15T18:04:58.854897+0000","last_became_peered":"2024-01-15T18:04:58.854897+0000","last_unstale":"2024-01-15T18:05:00.032421+0000","last_undegraded":"2024-01-15T18:05:00.032421+0000","last_fullsized":"2024-01-15T18:05:00.032421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:33:52.636110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.159371+0000","last_change":"2024-01-15T18:04:57.843229+0000","last_active":"2024-01-15T18:04:59.159371+0000","last_peered":"2024-01-15T18:04:59.159371+0000","last_clean":"2024-01-15T18:04:59.159371+0000","last_became_active":"2024-01-15T18:04:57.842720+0000","last_became_peered":"2024-01-15T18:04:57.842720+0000","last_unstale":"2024-01-15T18:04:59.159371+0000","last_undegraded":"2024-01-15T18:04:59.159371+0000","last_fullsized":"2024-01-15T18:04:59.159371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:25:33.145590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":806912,"data_stored":663105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93300000000000005}]}]},{"osd":6,"up_from":15,"seq":64424509447,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":73767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.181}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4220000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.012}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]}]},{"osd":4,"up_from":15,"seq":64424509447,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":72281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]}]},{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":73767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]}]},{"osd":2,"up_from":15,"seq":64424509446,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":72155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]}]},{"osd":1,"up_from":15,"seq":64424509448,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":212992,"data_stored":72245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]}]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":815104,"data_stored":663161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.462}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0960000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6739999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6890000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6559999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7450000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:05:07.634 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:05:07.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.804+0000 7f275b600640 1 Processor -- start 2024-01-15T18:05:07.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.804+0000 7f275b600640 1 -- start start 2024-01-15T18:05:07.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.808+0000 7f275b600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f2754071e40 0x7f27540c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:07.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.808+0000 7f275b600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f275406d430 0x7f27540cd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:07.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.808+0000 7f275b600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2754159a50 con 0x7f2754159bf0 2024-01-15T18:05:07.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.808+0000 7f275b600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27540cce00 con 0x7f275406d430 2024-01-15T18:05:07.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.808+0000 7f275b600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f275406d980 con 0x7f2754071e40 2024-01-15T18:05:07.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.808+0000 7f2759600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f275406d430 0x7f27540cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:07.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.808+0000 7f2758c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2754159bf0 0x7f2754159fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42412/0 (socket says 172.21.15.27:42412) 2024-01-15T18:05:07.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.808+0000 7f2758c00640 1 -- 172.21.15.27:0/291104029 learned_addr learned my addr 172.21.15.27:0/291104029 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:07.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.808+0000 7f2753e00640 1 --2- 172.21.15.27:0/291104029 >> v2:172.21.15.93:3300/0 conn(0x7f2754071e40 0x7f27540c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:07.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.808+0000 7f2759600640 1 -- 172.21.15.27:0/291104029 >> v2:172.21.15.93:3300/0 conn(0x7f2754071e40 msgr2=0x7f27540c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:07.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.808+0000 7f2759600640 1 --2- 172.21.15.27:0/291104029 >> v2:172.21.15.93:3300/0 conn(0x7f2754071e40 0x7f27540c9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.808+0000 7f2759600640 1 -- 172.21.15.27:0/291104029 >> v1:172.21.15.27:6789/0 conn(0x7f2754159bf0 legacy=0x7f2754159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.808+0000 7f2759600640 1 -- 172.21.15.27:0/291104029 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f275406dc60 con 0x7f275406d430 2024-01-15T18:05:07.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.808+0000 7f2759600640 1 --2- 172.21.15.27:0/291104029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f275406d430 0x7f27540cd250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f27540cfb10 tx=0x7f274c01f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a41484f9edfcd21d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:07.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.808+0000 7f2753400640 1 -- 172.21.15.27:0/291104029 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c0b0440 con 0x7f275406d430 2024-01-15T18:05:07.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.812+0000 7f2753400640 1 -- 172.21.15.27:0/291104029 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f274c0b05a0 con 0x7f275406d430 2024-01-15T18:05:07.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.812+0000 7f2753400640 1 -- 172.21.15.27:0/291104029 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c0b0930 con 0x7f275406d430 2024-01-15T18:05:07.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.812+0000 7f275b600640 1 -- 172.21.15.27:0/291104029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f275406d430 msgr2=0x7f27540cd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.812+0000 7f275b600640 1 --2- 172.21.15.27:0/291104029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f275406d430 0x7f27540cd250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f27540cfb10 tx=0x7f274c01f0f0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.812+0000 7f275b600640 1 -- 172.21.15.27:0/291104029 shutdown_connections 2024-01-15T18:05:07.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.812+0000 7f275b600640 1 --2- 172.21.15.27:0/291104029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f275406d430 0x7f27540cd250 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.812+0000 7f275b600640 1 --2- 172.21.15.27:0/291104029 >> v2:172.21.15.93:3300/0 conn(0x7f2754071e40 0x7f27540c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.812+0000 7f275b600640 1 -- 172.21.15.27:0/291104029 >> 172.21.15.27:0/291104029 conn(0x7f27540cec60 msgr2=0x7f27540cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:07.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.812+0000 7f275b600640 1 -- 172.21.15.27:0/291104029 shutdown_connections 2024-01-15T18:05:07.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.812+0000 7f275b600640 1 -- 172.21.15.27:0/291104029 wait complete. 2024-01-15T18:05:07.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.812+0000 7f275b600640 1 Processor -- start 2024-01-15T18:05:07.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.812+0000 7f275b600640 1 -- start start 2024-01-15T18:05:07.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.812+0000 7f275b600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f275406d430 0x7f27540c3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:07.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.812+0000 7f275b600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2754159bf0 0x7f2754159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:07.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.812+0000 7f275b600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27540c2800 0x7f27540c2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:07.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.812+0000 7f275b600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2754159760 con 0x7f275406d430 2024-01-15T18:05:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.812+0000 7f275b600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2754157ea0 con 0x7f2754159bf0 2024-01-15T18:05:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.812+0000 7f275b600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2754158180 con 0x7f27540c2800 2024-01-15T18:05:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.816+0000 7f2758c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f275406d430 0x7f27540c3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.816+0000 7f2758c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f275406d430 0x7f27540c3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59408/0 (socket says 172.21.15.27:59408) 2024-01-15T18:05:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.816+0000 7f2758c00640 1 -- 172.21.15.27:0/3965952799 learned_addr learned my addr 172.21.15.27:0/3965952799 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.816+0000 7f2759600640 1 --2- 172.21.15.27:0/3965952799 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27540c2800 0x7f27540c2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.816+0000 7f2753e00640 1 --2- 172.21.15.27:0/3965952799 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2754159bf0 0x7f2754159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.816+0000 7f2758c00640 1 -- 172.21.15.27:0/3965952799 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27540c2800 msgr2=0x7f27540c2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.816+0000 7f2758c00640 1 --2- 172.21.15.27:0/3965952799 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27540c2800 0x7f27540c2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.816+0000 7f2758c00640 1 -- 172.21.15.27:0/3965952799 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2754159bf0 msgr2=0x7f2754159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.816+0000 7f2758c00640 1 --2- 172.21.15.27:0/3965952799 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2754159bf0 0x7f2754159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.816+0000 7f2758c00640 1 -- 172.21.15.27:0/3965952799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f274c009a70 con 0x7f275406d430 2024-01-15T18:05:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.816+0000 7f2758c00640 1 --2- 172.21.15.27:0/3965952799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f275406d430 0x7f27540c3d30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f274400c400 tx=0x7f274400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.816+0000 7f2750c00640 1 -- 172.21.15.27:0/3965952799 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2744005cf0 con 0x7f275406d430 2024-01-15T18:05:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.816+0000 7f2750c00640 1 -- 172.21.15.27:0/3965952799 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2744005e50 con 0x7f275406d430 2024-01-15T18:05:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.816+0000 7f2750c00640 1 -- 172.21.15.27:0/3965952799 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27440072d0 con 0x7f275406d430 2024-01-15T18:05:07.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.816+0000 7f275b600640 1 -- 172.21.15.27:0/3965952799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2754158460 con 0x7f275406d430 2024-01-15T18:05:07.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.816+0000 7f275b600640 1 -- 172.21.15.27:0/3965952799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2754158a20 con 0x7f275406d430 2024-01-15T18:05:07.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.816+0000 7f275b600640 1 -- 172.21.15.27:0/3965952799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f275404e850 con 0x7f275406d430 2024-01-15T18:05:07.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.816+0000 7f2750c00640 1 -- 172.21.15.27:0/3965952799 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f2744007430 con 0x7f275406d430 2024-01-15T18:05:07.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.820+0000 7f2750c00640 1 --2- 172.21.15.27:0/3965952799 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f272c0668f0 0x7f272c068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:07.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.820+0000 7f2750c00640 1 -- 172.21.15.27:0/3965952799 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2744083780 con 0x7f275406d430 2024-01-15T18:05:07.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.820+0000 7f2753e00640 1 --2- 172.21.15.27:0/3965952799 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f272c0668f0 0x7f272c068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:07.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.820+0000 7f2753e00640 1 --2- 172.21.15.27:0/3965952799 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f272c0668f0 0x7f272c068db0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2754168650 tx=0x7f273c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:07.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.824+0000 7f2750c00640 1 -- 172.21.15.27:0/3965952799 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f27440509d0 con 0x7f275406d430 2024-01-15T18:05:07.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.976+0000 7f275b600640 1 -- 172.21.15.27:0/3965952799 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f27540ca120 con 0x7f272c0668f0 2024-01-15T18:05:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.984+0000 7f2750c00640 1 -- 172.21.15.27:0/3965952799 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264557 (secure 0 0 0) 0x7f273c010960 con 0x7f272c0668f0 2024-01-15T18:05:07.991 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:07.993 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:05:07.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.988+0000 7f275b600640 1 -- 172.21.15.27:0/3965952799 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f272c0668f0 msgr2=0x7f272c068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.988+0000 7f275b600640 1 --2- 172.21.15.27:0/3965952799 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f272c0668f0 0x7f272c068db0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2754168650 tx=0x7f273c008040 comp rx=0 tx=0).stop 2024-01-15T18:05:07.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.988+0000 7f275b600640 1 -- 172.21.15.27:0/3965952799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f275406d430 msgr2=0x7f27540c3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.988+0000 7f275b600640 1 --2- 172.21.15.27:0/3965952799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f275406d430 0x7f27540c3d30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f274400c400 tx=0x7f274400c8c0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.988+0000 7f275b600640 1 -- 172.21.15.27:0/3965952799 shutdown_connections 2024-01-15T18:05:07.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.988+0000 7f275b600640 1 --2- 172.21.15.27:0/3965952799 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f272c0668f0 0x7f272c068db0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.988+0000 7f275b600640 1 --2- 172.21.15.27:0/3965952799 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27540c2800 0x7f27540c2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.988+0000 7f275b600640 1 --2- 172.21.15.27:0/3965952799 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2754159bf0 0x7f2754159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.988+0000 7f275b600640 1 --2- 172.21.15.27:0/3965952799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f275406d430 0x7f27540c3d30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.988+0000 7f275b600640 1 -- 172.21.15.27:0/3965952799 >> 172.21.15.27:0/3965952799 conn(0x7f27540cec60 msgr2=0x7f27540cf040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:07.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.988+0000 7f275b600640 1 -- 172.21.15.27:0/3965952799 shutdown_connections 2024-01-15T18:05:07.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:07.988+0000 7f275b600640 1 -- 172.21.15.27:0/3965952799 wait complete. 2024-01-15T18:05:08.012 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-01-15T18:05:06.140488+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219052,"kb_used_data":2856,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755668,"statfs":{"total":773094113280,"available":772869804032,"internally_reserved":0,"allocated":2924544,"data_stored":1762796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002178"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032071+0000","last_change":"2024-01-15T18:04:58.862012+0000","last_active":"2024-01-15T18:05:00.032071+0000","last_peered":"2024-01-15T18:05:00.032071+0000","last_clean":"2024-01-15T18:05:00.032071+0000","last_became_active":"2024-01-15T18:04:58.861713+0000","last_became_peered":"2024-01-15T18:04:58.861713+0000","last_unstale":"2024-01-15T18:05:00.032071+0000","last_undegraded":"2024-01-15T18:05:00.032071+0000","last_fullsized":"2024-01-15T18:05:00.032071+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:30:29.323987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849437+0000","last_change":"2024-01-15T18:04:58.870910+0000","last_active":"2024-01-15T18:04:59.849437+0000","last_peered":"2024-01-15T18:04:59.849437+0000","last_clean":"2024-01-15T18:04:59.849437+0000","last_became_active":"2024-01-15T18:04:58.869957+0000","last_became_peered":"2024-01-15T18:04:58.869957+0000","last_unstale":"2024-01-15T18:04:59.849437+0000","last_undegraded":"2024-01-15T18:04:59.849437+0000","last_fullsized":"2024-01-15T18:04:59.849437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:10:24.519754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031283+0000","last_change":"2024-01-15T18:04:58.854305+0000","last_active":"2024-01-15T18:05:00.031283+0000","last_peered":"2024-01-15T18:05:00.031283+0000","last_clean":"2024-01-15T18:05:00.031283+0000","last_became_active":"2024-01-15T18:04:58.853375+0000","last_became_peered":"2024-01-15T18:04:58.853375+0000","last_unstale":"2024-01-15T18:05:00.031283+0000","last_undegraded":"2024-01-15T18:05:00.031283+0000","last_fullsized":"2024-01-15T18:05:00.031283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:27:32.756552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847538+0000","last_change":"2024-01-15T18:04:58.844639+0000","last_active":"2024-01-15T18:04:59.847538+0000","last_peered":"2024-01-15T18:04:59.847538+0000","last_clean":"2024-01-15T18:04:59.847538+0000","last_became_active":"2024-01-15T18:04:58.844174+0000","last_became_peered":"2024-01-15T18:04:58.844174+0000","last_unstale":"2024-01-15T18:04:59.847538+0000","last_undegraded":"2024-01-15T18:04:59.847538+0000","last_fullsized":"2024-01-15T18:04:59.847538+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:18:25.022013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031678+0000","last_change":"2024-01-15T18:04:58.852004+0000","last_active":"2024-01-15T18:05:00.031678+0000","last_peered":"2024-01-15T18:05:00.031678+0000","last_clean":"2024-01-15T18:05:00.031678+0000","last_became_active":"2024-01-15T18:04:58.849191+0000","last_became_peered":"2024-01-15T18:04:58.849191+0000","last_unstale":"2024-01-15T18:05:00.031678+0000","last_undegraded":"2024-01-15T18:05:00.031678+0000","last_fullsized":"2024-01-15T18:05:00.031678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:22:03.084714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.846852+0000","last_change":"2024-01-15T18:04:58.854461+0000","last_active":"2024-01-15T18:04:59.846852+0000","last_peered":"2024-01-15T18:04:59.846852+0000","last_clean":"2024-01-15T18:04:59.846852+0000","last_became_active":"2024-01-15T18:04:58.854170+0000","last_became_peered":"2024-01-15T18:04:58.854170+0000","last_unstale":"2024-01-15T18:04:59.846852+0000","last_undegraded":"2024-01-15T18:04:59.846852+0000","last_fullsized":"2024-01-15T18:04:59.846852+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:05:31.164011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031425+0000","last_change":"2024-01-15T18:04:58.857161+0000","last_active":"2024-01-15T18:05:00.031425+0000","last_peered":"2024-01-15T18:05:00.031425+0000","last_clean":"2024-01-15T18:05:00.031425+0000","last_became_active":"2024-01-15T18:04:58.856839+0000","last_became_peered":"2024-01-15T18:04:58.856839+0000","last_unstale":"2024-01-15T18:05:00.031425+0000","last_undegraded":"2024-01-15T18:05:00.031425+0000","last_fullsized":"2024-01-15T18:05:00.031425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:53:23.471036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031831+0000","last_change":"2024-01-15T18:04:58.861226+0000","last_active":"2024-01-15T18:05:00.031831+0000","last_peered":"2024-01-15T18:05:00.031831+0000","last_clean":"2024-01-15T18:05:00.031831+0000","last_became_active":"2024-01-15T18:04:58.860964+0000","last_became_peered":"2024-01-15T18:04:58.860964+0000","last_unstale":"2024-01-15T18:05:00.031831+0000","last_undegraded":"2024-01-15T18:05:00.031831+0000","last_fullsized":"2024-01-15T18:05:00.031831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:11:49.903930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032115+0000","last_change":"2024-01-15T18:04:58.856674+0000","last_active":"2024-01-15T18:05:00.032115+0000","last_peered":"2024-01-15T18:05:00.032115+0000","last_clean":"2024-01-15T18:05:00.032115+0000","last_became_active":"2024-01-15T18:04:58.856365+0000","last_became_peered":"2024-01-15T18:04:58.856365+0000","last_unstale":"2024-01-15T18:05:00.032115+0000","last_undegraded":"2024-01-15T18:05:00.032115+0000","last_fullsized":"2024-01-15T18:05:00.032115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:09:03.416070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030934+0000","last_change":"2024-01-15T18:04:58.853185+0000","last_active":"2024-01-15T18:05:00.030934+0000","last_peered":"2024-01-15T18:05:00.030934+0000","last_clean":"2024-01-15T18:05:00.030934+0000","last_became_active":"2024-01-15T18:04:58.852888+0000","last_became_peered":"2024-01-15T18:04:58.852888+0000","last_unstale":"2024-01-15T18:05:00.030934+0000","last_undegraded":"2024-01-15T18:05:00.030934+0000","last_fullsized":"2024-01-15T18:05:00.030934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:10:24.519754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030810+0000","last_change":"2024-01-15T18:04:58.848009+0000","last_active":"2024-01-15T18:05:00.030810+0000","last_peered":"2024-01-15T18:05:00.030810+0000","last_clean":"2024-01-15T18:05:00.030810+0000","last_became_active":"2024-01-15T18:04:58.847765+0000","last_became_peered":"2024-01-15T18:04:58.847765+0000","last_unstale":"2024-01-15T18:05:00.030810+0000","last_undegraded":"2024-01-15T18:05:00.030810+0000","last_fullsized":"2024-01-15T18:05:00.030810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:15:34.376159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847406+0000","last_change":"2024-01-15T18:04:58.847259+0000","last_active":"2024-01-15T18:04:59.847406+0000","last_peered":"2024-01-15T18:04:59.847406+0000","last_clean":"2024-01-15T18:04:59.847406+0000","last_became_active":"2024-01-15T18:04:58.845619+0000","last_became_peered":"2024-01-15T18:04:58.845619+0000","last_unstale":"2024-01-15T18:04:59.847406+0000","last_undegraded":"2024-01-15T18:04:59.847406+0000","last_fullsized":"2024-01-15T18:04:59.847406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:16:30.446952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032165+0000","last_change":"2024-01-15T18:04:58.842389+0000","last_active":"2024-01-15T18:05:00.032165+0000","last_peered":"2024-01-15T18:05:00.032165+0000","last_clean":"2024-01-15T18:05:00.032165+0000","last_became_active":"2024-01-15T18:04:58.841378+0000","last_became_peered":"2024-01-15T18:04:58.841378+0000","last_unstale":"2024-01-15T18:05:00.032165+0000","last_undegraded":"2024-01-15T18:05:00.032165+0000","last_fullsized":"2024-01-15T18:05:00.032165+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:36.680373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030952+0000","last_change":"2024-01-15T18:04:58.864203+0000","last_active":"2024-01-15T18:05:00.030952+0000","last_peered":"2024-01-15T18:05:00.030952+0000","last_clean":"2024-01-15T18:05:00.030952+0000","last_became_active":"2024-01-15T18:04:58.863956+0000","last_became_peered":"2024-01-15T18:04:58.863956+0000","last_unstale":"2024-01-15T18:05:00.030952+0000","last_undegraded":"2024-01-15T18:05:00.030952+0000","last_fullsized":"2024-01-15T18:05:00.030952+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:37:40.217674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849113+0000","last_change":"2024-01-15T18:04:58.836384+0000","last_active":"2024-01-15T18:04:59.849113+0000","last_peered":"2024-01-15T18:04:59.849113+0000","last_clean":"2024-01-15T18:04:59.849113+0000","last_became_active":"2024-01-15T18:04:58.836081+0000","last_became_peered":"2024-01-15T18:04:58.836081+0000","last_unstale":"2024-01-15T18:04:59.849113+0000","last_undegraded":"2024-01-15T18:04:59.849113+0000","last_fullsized":"2024-01-15T18:04:59.849113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:15:34.376159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.846539+0000","last_change":"2024-01-15T18:04:58.868380+0000","last_active":"2024-01-15T18:04:59.846539+0000","last_peered":"2024-01-15T18:04:59.846539+0000","last_clean":"2024-01-15T18:04:59.846539+0000","last_became_active":"2024-01-15T18:04:58.868103+0000","last_became_peered":"2024-01-15T18:04:58.868103+0000","last_unstale":"2024-01-15T18:04:59.846539+0000","last_undegraded":"2024-01-15T18:04:59.846539+0000","last_fullsized":"2024-01-15T18:04:59.846539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:48:25.476425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.848460+0000","last_change":"2024-01-15T18:04:58.858404+0000","last_active":"2024-01-15T18:04:59.848460+0000","last_peered":"2024-01-15T18:04:59.848460+0000","last_clean":"2024-01-15T18:04:59.848460+0000","last_became_active":"2024-01-15T18:04:58.857755+0000","last_became_peered":"2024-01-15T18:04:58.857755+0000","last_unstale":"2024-01-15T18:04:59.848460+0000","last_undegraded":"2024-01-15T18:04:59.848460+0000","last_fullsized":"2024-01-15T18:04:59.848460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:38:44.912913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030907+0000","last_change":"2024-01-15T18:04:58.844392+0000","last_active":"2024-01-15T18:05:00.030907+0000","last_peered":"2024-01-15T18:05:00.030907+0000","last_clean":"2024-01-15T18:05:00.030907+0000","last_became_active":"2024-01-15T18:04:58.844086+0000","last_became_peered":"2024-01-15T18:04:58.844086+0000","last_unstale":"2024-01-15T18:05:00.030907+0000","last_undegraded":"2024-01-15T18:05:00.030907+0000","last_fullsized":"2024-01-15T18:05:00.030907+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:16:30.446952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031976+0000","last_change":"2024-01-15T18:04:58.861264+0000","last_active":"2024-01-15T18:05:00.031976+0000","last_peered":"2024-01-15T18:05:00.031976+0000","last_clean":"2024-01-15T18:05:00.031976+0000","last_became_active":"2024-01-15T18:04:58.860695+0000","last_became_peered":"2024-01-15T18:04:58.860695+0000","last_unstale":"2024-01-15T18:05:00.031976+0000","last_undegraded":"2024-01-15T18:05:00.031976+0000","last_fullsized":"2024-01-15T18:05:00.031976+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:47:51.247174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030783+0000","last_change":"2024-01-15T18:04:58.846632+0000","last_active":"2024-01-15T18:05:00.030783+0000","last_peered":"2024-01-15T18:05:00.030783+0000","last_clean":"2024-01-15T18:05:00.030783+0000","last_became_active":"2024-01-15T18:04:58.846359+0000","last_became_peered":"2024-01-15T18:04:58.846359+0000","last_unstale":"2024-01-15T18:05:00.030783+0000","last_undegraded":"2024-01-15T18:05:00.030783+0000","last_fullsized":"2024-01-15T18:05:00.030783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:12:00.696642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031213+0000","last_change":"2024-01-15T18:04:58.858817+0000","last_active":"2024-01-15T18:05:00.031213+0000","last_peered":"2024-01-15T18:05:00.031213+0000","last_clean":"2024-01-15T18:05:00.031213+0000","last_became_active":"2024-01-15T18:04:58.858539+0000","last_became_peered":"2024-01-15T18:04:58.858539+0000","last_unstale":"2024-01-15T18:05:00.031213+0000","last_undegraded":"2024-01-15T18:05:00.031213+0000","last_fullsized":"2024-01-15T18:05:00.031213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:25:00.854241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847131+0000","last_change":"2024-01-15T18:04:58.867761+0000","last_active":"2024-01-15T18:04:59.847131+0000","last_peered":"2024-01-15T18:04:59.847131+0000","last_clean":"2024-01-15T18:04:59.847131+0000","last_became_active":"2024-01-15T18:04:58.867331+0000","last_became_peered":"2024-01-15T18:04:58.867331+0000","last_unstale":"2024-01-15T18:04:59.847131+0000","last_undegraded":"2024-01-15T18:04:59.847131+0000","last_fullsized":"2024-01-15T18:04:59.847131+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:14:47.495474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031915+0000","last_change":"2024-01-15T18:04:58.839963+0000","last_active":"2024-01-15T18:05:00.031915+0000","last_peered":"2024-01-15T18:05:00.031915+0000","last_clean":"2024-01-15T18:05:00.031915+0000","last_became_active":"2024-01-15T18:04:58.839597+0000","last_became_peered":"2024-01-15T18:04:58.839597+0000","last_unstale":"2024-01-15T18:05:00.031915+0000","last_undegraded":"2024-01-15T18:05:00.031915+0000","last_fullsized":"2024-01-15T18:05:00.031915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:47.713249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.846576+0000","last_change":"2024-01-15T18:04:58.841274+0000","last_active":"2024-01-15T18:04:59.846576+0000","last_peered":"2024-01-15T18:04:59.846576+0000","last_clean":"2024-01-15T18:04:59.846576+0000","last_became_active":"2024-01-15T18:04:58.840866+0000","last_became_peered":"2024-01-15T18:04:58.840866+0000","last_unstale":"2024-01-15T18:04:59.846576+0000","last_undegraded":"2024-01-15T18:04:59.846576+0000","last_fullsized":"2024-01-15T18:04:59.846576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:12:02.202965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032528+0000","last_change":"2024-01-15T18:04:58.855334+0000","last_active":"2024-01-15T18:05:00.032528+0000","last_peered":"2024-01-15T18:05:00.032528+0000","last_clean":"2024-01-15T18:05:00.032528+0000","last_became_active":"2024-01-15T18:04:58.855074+0000","last_became_peered":"2024-01-15T18:04:58.855074+0000","last_unstale":"2024-01-15T18:05:00.032528+0000","last_undegraded":"2024-01-15T18:05:00.032528+0000","last_fullsized":"2024-01-15T18:05:00.032528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:44:17.933005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031069+0000","last_change":"2024-01-15T18:04:58.861125+0000","last_active":"2024-01-15T18:05:00.031069+0000","last_peered":"2024-01-15T18:05:00.031069+0000","last_clean":"2024-01-15T18:05:00.031069+0000","last_became_active":"2024-01-15T18:04:58.860819+0000","last_became_peered":"2024-01-15T18:04:58.860819+0000","last_unstale":"2024-01-15T18:05:00.031069+0000","last_undegraded":"2024-01-15T18:05:00.031069+0000","last_fullsized":"2024-01-15T18:05:00.031069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:04:27.769262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849316+0000","last_change":"2024-01-15T18:04:58.837356+0000","last_active":"2024-01-15T18:04:59.849316+0000","last_peered":"2024-01-15T18:04:59.849316+0000","last_clean":"2024-01-15T18:04:59.849316+0000","last_became_active":"2024-01-15T18:04:58.837071+0000","last_became_peered":"2024-01-15T18:04:58.837071+0000","last_unstale":"2024-01-15T18:04:59.849316+0000","last_undegraded":"2024-01-15T18:04:59.849316+0000","last_fullsized":"2024-01-15T18:04:59.849316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:36.680373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032362+0000","last_change":"2024-01-15T18:04:58.858404+0000","last_active":"2024-01-15T18:05:00.032362+0000","last_peered":"2024-01-15T18:05:00.032362+0000","last_clean":"2024-01-15T18:05:00.032362+0000","last_became_active":"2024-01-15T18:04:58.857239+0000","last_became_peered":"2024-01-15T18:04:58.857239+0000","last_unstale":"2024-01-15T18:05:00.032362+0000","last_undegraded":"2024-01-15T18:05:00.032362+0000","last_fullsized":"2024-01-15T18:05:00.032362+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:04:27.769262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031219+0000","last_change":"2024-01-15T18:04:58.861031+0000","last_active":"2024-01-15T18:05:00.031219+0000","last_peered":"2024-01-15T18:05:00.031219+0000","last_clean":"2024-01-15T18:05:00.031219+0000","last_became_active":"2024-01-15T18:04:58.860718+0000","last_became_peered":"2024-01-15T18:04:58.860718+0000","last_unstale":"2024-01-15T18:05:00.031219+0000","last_undegraded":"2024-01-15T18:05:00.031219+0000","last_fullsized":"2024-01-15T18:05:00.031219+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:21:06.990646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847261+0000","last_change":"2024-01-15T18:04:58.841212+0000","last_active":"2024-01-15T18:04:59.847261+0000","last_peered":"2024-01-15T18:04:59.847261+0000","last_clean":"2024-01-15T18:04:59.847261+0000","last_became_active":"2024-01-15T18:04:58.840913+0000","last_became_peered":"2024-01-15T18:04:58.840913+0000","last_unstale":"2024-01-15T18:04:59.847261+0000","last_undegraded":"2024-01-15T18:04:59.847261+0000","last_fullsized":"2024-01-15T18:04:59.847261+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:17:56.224397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030927+0000","last_change":"2024-01-15T18:04:58.847922+0000","last_active":"2024-01-15T18:05:00.030927+0000","last_peered":"2024-01-15T18:05:00.030927+0000","last_clean":"2024-01-15T18:05:00.030927+0000","last_became_active":"2024-01-15T18:04:58.847596+0000","last_became_peered":"2024-01-15T18:04:58.847596+0000","last_unstale":"2024-01-15T18:05:00.030927+0000","last_undegraded":"2024-01-15T18:05:00.030927+0000","last_fullsized":"2024-01-15T18:05:00.030927+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:24:34.638590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030579+0000","last_change":"2024-01-15T18:04:58.866119+0000","last_active":"2024-01-15T18:05:00.030579+0000","last_peered":"2024-01-15T18:05:00.030579+0000","last_clean":"2024-01-15T18:05:00.030579+0000","last_became_active":"2024-01-15T18:04:58.865811+0000","last_became_peered":"2024-01-15T18:04:58.865811+0000","last_unstale":"2024-01-15T18:05:00.030579+0000","last_undegraded":"2024-01-15T18:05:00.030579+0000","last_fullsized":"2024-01-15T18:05:00.030579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:10.719289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847206+0000","last_change":"2024-01-15T18:04:58.839281+0000","last_active":"2024-01-15T18:04:59.847206+0000","last_peered":"2024-01-15T18:04:59.847206+0000","last_clean":"2024-01-15T18:04:59.847206+0000","last_became_active":"2024-01-15T18:04:58.838304+0000","last_became_peered":"2024-01-15T18:04:58.838304+0000","last_unstale":"2024-01-15T18:04:59.847206+0000","last_undegraded":"2024-01-15T18:04:59.847206+0000","last_fullsized":"2024-01-15T18:04:59.847206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:08:43.279198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032260+0000","last_change":"2024-01-15T18:04:57.840527+0000","last_active":"2024-01-15T18:05:00.032260+0000","last_peered":"2024-01-15T18:05:00.032260+0000","last_clean":"2024-01-15T18:05:00.032260+0000","last_became_active":"2024-01-15T18:04:57.840153+0000","last_became_peered":"2024-01-15T18:04:57.840153+0000","last_unstale":"2024-01-15T18:05:00.032260+0000","last_undegraded":"2024-01-15T18:05:00.032260+0000","last_fullsized":"2024-01-15T18:05:00.032260+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:17:50.737228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031360+0000","last_change":"2024-01-15T18:04:58.859368+0000","last_active":"2024-01-15T18:05:00.031360+0000","last_peered":"2024-01-15T18:05:00.031360+0000","last_clean":"2024-01-15T18:05:00.031360+0000","last_became_active":"2024-01-15T18:04:58.859112+0000","last_became_peered":"2024-01-15T18:04:58.859112+0000","last_unstale":"2024-01-15T18:05:00.031360+0000","last_undegraded":"2024-01-15T18:05:00.031360+0000","last_fullsized":"2024-01-15T18:05:00.031360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:48:25.476425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030936+0000","last_change":"2024-01-15T18:04:57.843777+0000","last_active":"2024-01-15T18:05:00.030936+0000","last_peered":"2024-01-15T18:05:00.030936+0000","last_clean":"2024-01-15T18:05:00.030936+0000","last_became_active":"2024-01-15T18:04:57.843062+0000","last_became_peered":"2024-01-15T18:04:57.843062+0000","last_unstale":"2024-01-15T18:05:00.030936+0000","last_undegraded":"2024-01-15T18:05:00.030936+0000","last_fullsized":"2024-01-15T18:05:00.030936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:58:15.645144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031986+0000","last_change":"2024-01-15T18:04:57.833749+0000","last_active":"2024-01-15T18:05:00.031986+0000","last_peered":"2024-01-15T18:05:00.031986+0000","last_clean":"2024-01-15T18:05:00.031986+0000","last_became_active":"2024-01-15T18:04:57.832655+0000","last_became_peered":"2024-01-15T18:04:57.832655+0000","last_unstale":"2024-01-15T18:05:00.031986+0000","last_undegraded":"2024-01-15T18:05:00.031986+0000","last_fullsized":"2024-01-15T18:05:00.031986+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:42:16.957837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031910+0000","last_change":"2024-01-15T18:04:58.862938+0000","last_active":"2024-01-15T18:05:00.031910+0000","last_peered":"2024-01-15T18:05:00.031910+0000","last_clean":"2024-01-15T18:05:00.031910+0000","last_became_active":"2024-01-15T18:04:58.862733+0000","last_became_peered":"2024-01-15T18:04:58.862733+0000","last_unstale":"2024-01-15T18:05:00.031910+0000","last_undegraded":"2024-01-15T18:05:00.031910+0000","last_fullsized":"2024-01-15T18:05:00.031910+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:10.719289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.161858+0000","last_change":"2024-01-15T18:04:57.828200+0000","last_active":"2024-01-15T18:04:59.161858+0000","last_peered":"2024-01-15T18:04:59.161858+0000","last_clean":"2024-01-15T18:04:59.161858+0000","last_became_active":"2024-01-15T18:04:57.827883+0000","last_became_peered":"2024-01-15T18:04:57.827883+0000","last_unstale":"2024-01-15T18:04:59.161858+0000","last_undegraded":"2024-01-15T18:04:59.161858+0000","last_fullsized":"2024-01-15T18:04:59.161858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:48:16.392862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849021+0000","last_change":"2024-01-15T18:04:58.842387+0000","last_active":"2024-01-15T18:04:59.849021+0000","last_peered":"2024-01-15T18:04:59.849021+0000","last_clean":"2024-01-15T18:04:59.849021+0000","last_became_active":"2024-01-15T18:04:58.842093+0000","last_became_peered":"2024-01-15T18:04:58.842093+0000","last_unstale":"2024-01-15T18:04:59.849021+0000","last_undegraded":"2024-01-15T18:04:59.849021+0000","last_fullsized":"2024-01-15T18:04:59.849021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:47.713249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030983+0000","last_change":"2024-01-15T18:04:57.837812+0000","last_active":"2024-01-15T18:05:00.030983+0000","last_peered":"2024-01-15T18:05:00.030983+0000","last_clean":"2024-01-15T18:05:00.030983+0000","last_became_active":"2024-01-15T18:04:57.837524+0000","last_became_peered":"2024-01-15T18:04:57.837524+0000","last_unstale":"2024-01-15T18:05:00.030983+0000","last_undegraded":"2024-01-15T18:05:00.030983+0000","last_fullsized":"2024-01-15T18:05:00.030983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:37:44.430593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.848322+0000","last_change":"2024-01-15T18:04:58.854115+0000","last_active":"2024-01-15T18:04:59.848322+0000","last_peered":"2024-01-15T18:04:59.848322+0000","last_clean":"2024-01-15T18:04:59.848322+0000","last_became_active":"2024-01-15T18:04:58.853039+0000","last_became_peered":"2024-01-15T18:04:58.853039+0000","last_unstale":"2024-01-15T18:04:59.848322+0000","last_undegraded":"2024-01-15T18:04:59.848322+0000","last_fullsized":"2024-01-15T18:04:59.848322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:36:36.331502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.162008+0000","last_change":"2024-01-15T18:04:57.839191+0000","last_active":"2024-01-15T18:04:59.162008+0000","last_peered":"2024-01-15T18:04:59.162008+0000","last_clean":"2024-01-15T18:04:59.162008+0000","last_became_active":"2024-01-15T18:04:57.838884+0000","last_became_peered":"2024-01-15T18:04:57.838884+0000","last_unstale":"2024-01-15T18:04:59.162008+0000","last_undegraded":"2024-01-15T18:04:59.162008+0000","last_fullsized":"2024-01-15T18:04:59.162008+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:37:39.211256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031433+0000","last_change":"2024-01-15T18:04:58.860345+0000","last_active":"2024-01-15T18:05:00.031433+0000","last_peered":"2024-01-15T18:05:00.031433+0000","last_clean":"2024-01-15T18:05:00.031433+0000","last_became_active":"2024-01-15T18:04:58.859997+0000","last_became_peered":"2024-01-15T18:04:58.859997+0000","last_unstale":"2024-01-15T18:05:00.031433+0000","last_undegraded":"2024-01-15T18:05:00.031433+0000","last_fullsized":"2024-01-15T18:05:00.031433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:05:31.164011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849266+0000","last_change":"2024-01-15T18:04:57.835121+0000","last_active":"2024-01-15T18:04:59.849266+0000","last_peered":"2024-01-15T18:04:59.849266+0000","last_clean":"2024-01-15T18:04:59.849266+0000","last_became_active":"2024-01-15T18:04:57.834822+0000","last_became_peered":"2024-01-15T18:04:57.834822+0000","last_unstale":"2024-01-15T18:04:59.849266+0000","last_undegraded":"2024-01-15T18:04:59.849266+0000","last_fullsized":"2024-01-15T18:04:59.849266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:17:50.737228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847554+0000","last_change":"2024-01-15T18:04:58.859000+0000","last_active":"2024-01-15T18:04:59.847554+0000","last_peered":"2024-01-15T18:04:59.847554+0000","last_clean":"2024-01-15T18:04:59.847554+0000","last_became_active":"2024-01-15T18:04:58.854577+0000","last_became_peered":"2024-01-15T18:04:58.854577+0000","last_unstale":"2024-01-15T18:04:59.847554+0000","last_undegraded":"2024-01-15T18:04:59.847554+0000","last_fullsized":"2024-01-15T18:04:59.847554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:24:34.638590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030767+0000","last_change":"2024-01-15T18:04:57.836570+0000","last_active":"2024-01-15T18:05:00.030767+0000","last_peered":"2024-01-15T18:05:00.030767+0000","last_clean":"2024-01-15T18:05:00.030767+0000","last_became_active":"2024-01-15T18:04:57.836027+0000","last_became_peered":"2024-01-15T18:04:57.836027+0000","last_unstale":"2024-01-15T18:05:00.030767+0000","last_undegraded":"2024-01-15T18:05:00.030767+0000","last_fullsized":"2024-01-15T18:05:00.030767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:17:50.737228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031727+0000","last_change":"2024-01-15T18:04:58.872987+0000","last_active":"2024-01-15T18:05:00.031727+0000","last_peered":"2024-01-15T18:05:00.031727+0000","last_clean":"2024-01-15T18:05:00.031727+0000","last_became_active":"2024-01-15T18:04:58.872671+0000","last_became_peered":"2024-01-15T18:04:58.872671+0000","last_unstale":"2024-01-15T18:05:00.031727+0000","last_undegraded":"2024-01-15T18:05:00.031727+0000","last_fullsized":"2024-01-15T18:05:00.031727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:42:12.458758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.159257+0000","last_change":"2024-01-15T18:04:57.839505+0000","last_active":"2024-01-15T18:04:59.159257+0000","last_peered":"2024-01-15T18:04:59.159257+0000","last_clean":"2024-01-15T18:04:59.159257+0000","last_became_active":"2024-01-15T18:04:57.839194+0000","last_became_peered":"2024-01-15T18:04:57.839194+0000","last_unstale":"2024-01-15T18:04:59.159257+0000","last_undegraded":"2024-01-15T18:04:59.159257+0000","last_fullsized":"2024-01-15T18:04:59.159257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:10.960718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031101+0000","last_change":"2024-01-15T18:04:58.858064+0000","last_active":"2024-01-15T18:05:00.031101+0000","last_peered":"2024-01-15T18:05:00.031101+0000","last_clean":"2024-01-15T18:05:00.031101+0000","last_became_active":"2024-01-15T18:04:58.857697+0000","last_became_peered":"2024-01-15T18:04:58.857697+0000","last_unstale":"2024-01-15T18:05:00.031101+0000","last_undegraded":"2024-01-15T18:05:00.031101+0000","last_fullsized":"2024-01-15T18:05:00.031101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:21:44.926560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.161896+0000","last_change":"2024-01-15T18:04:57.836035+0000","last_active":"2024-01-15T18:04:59.161896+0000","last_peered":"2024-01-15T18:04:59.161896+0000","last_clean":"2024-01-15T18:04:59.161896+0000","last_became_active":"2024-01-15T18:04:57.835722+0000","last_became_peered":"2024-01-15T18:04:57.835722+0000","last_unstale":"2024-01-15T18:04:59.161896+0000","last_undegraded":"2024-01-15T18:04:59.161896+0000","last_fullsized":"2024-01-15T18:04:59.161896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:24:59.847823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031861+0000","last_change":"2024-01-15T18:04:58.852717+0000","last_active":"2024-01-15T18:05:00.031861+0000","last_peered":"2024-01-15T18:05:00.031861+0000","last_clean":"2024-01-15T18:05:00.031861+0000","last_became_active":"2024-01-15T18:04:58.852351+0000","last_became_peered":"2024-01-15T18:04:58.852351+0000","last_unstale":"2024-01-15T18:05:00.031861+0000","last_undegraded":"2024-01-15T18:05:00.031861+0000","last_fullsized":"2024-01-15T18:05:00.031861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:47:03.033950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030528+0000","last_change":"2024-01-15T18:04:57.831525+0000","last_active":"2024-01-15T18:05:00.030528+0000","last_peered":"2024-01-15T18:05:00.030528+0000","last_clean":"2024-01-15T18:05:00.030528+0000","last_became_active":"2024-01-15T18:04:57.831053+0000","last_became_peered":"2024-01-15T18:04:57.831053+0000","last_unstale":"2024-01-15T18:05:00.030528+0000","last_undegraded":"2024-01-15T18:05:00.030528+0000","last_fullsized":"2024-01-15T18:05:00.030528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:24:33.632172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030589+0000","last_change":"2024-01-15T18:04:58.866415+0000","last_active":"2024-01-15T18:05:00.030589+0000","last_peered":"2024-01-15T18:05:00.030589+0000","last_clean":"2024-01-15T18:05:00.030589+0000","last_became_active":"2024-01-15T18:04:58.865778+0000","last_became_peered":"2024-01-15T18:04:58.865778+0000","last_unstale":"2024-01-15T18:05:00.030589+0000","last_undegraded":"2024-01-15T18:05:00.030589+0000","last_fullsized":"2024-01-15T18:05:00.030589+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:47.713249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030622+0000","last_change":"2024-01-15T18:04:57.821766+0000","last_active":"2024-01-15T18:05:00.030622+0000","last_peered":"2024-01-15T18:05:00.030622+0000","last_clean":"2024-01-15T18:05:00.030622+0000","last_became_active":"2024-01-15T18:04:57.820957+0000","last_became_peered":"2024-01-15T18:04:57.820957+0000","last_unstale":"2024-01-15T18:05:00.030622+0000","last_undegraded":"2024-01-15T18:05:00.030622+0000","last_fullsized":"2024-01-15T18:05:00.030622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:08:42.272780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031415+0000","last_change":"2024-01-15T18:04:58.862675+0000","last_active":"2024-01-15T18:05:00.031415+0000","last_peered":"2024-01-15T18:05:00.031415+0000","last_clean":"2024-01-15T18:05:00.031415+0000","last_became_active":"2024-01-15T18:04:58.862420+0000","last_became_peered":"2024-01-15T18:04:58.862420+0000","last_unstale":"2024-01-15T18:05:00.031415+0000","last_undegraded":"2024-01-15T18:05:00.031415+0000","last_fullsized":"2024-01-15T18:05:00.031415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:44:17.933005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847491+0000","last_change":"2024-01-15T18:04:57.839139+0000","last_active":"2024-01-15T18:04:59.847491+0000","last_peered":"2024-01-15T18:04:59.847491+0000","last_clean":"2024-01-15T18:04:59.847491+0000","last_became_active":"2024-01-15T18:04:57.838847+0000","last_became_peered":"2024-01-15T18:04:57.838847+0000","last_unstale":"2024-01-15T18:04:59.847491+0000","last_undegraded":"2024-01-15T18:04:59.847491+0000","last_fullsized":"2024-01-15T18:04:59.847491+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:44:30.299838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031909+0000","last_change":"2024-01-15T18:04:58.870909+0000","last_active":"2024-01-15T18:05:00.031909+0000","last_peered":"2024-01-15T18:05:00.031909+0000","last_clean":"2024-01-15T18:05:00.031909+0000","last_became_active":"2024-01-15T18:04:58.868734+0000","last_became_peered":"2024-01-15T18:04:58.868734+0000","last_unstale":"2024-01-15T18:05:00.031909+0000","last_undegraded":"2024-01-15T18:05:00.031909+0000","last_fullsized":"2024-01-15T18:05:00.031909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:58:21.135065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030997+0000","last_change":"2024-01-15T18:04:57.829932+0000","last_active":"2024-01-15T18:05:00.030997+0000","last_peered":"2024-01-15T18:05:00.030997+0000","last_clean":"2024-01-15T18:05:00.030997+0000","last_became_active":"2024-01-15T18:04:57.829636+0000","last_became_peered":"2024-01-15T18:04:57.829636+0000","last_unstale":"2024-01-15T18:05:00.030997+0000","last_undegraded":"2024-01-15T18:05:00.030997+0000","last_fullsized":"2024-01-15T18:05:00.030997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:04:13.242693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032156+0000","last_change":"2024-01-15T18:04:58.869477+0000","last_active":"2024-01-15T18:05:00.032156+0000","last_peered":"2024-01-15T18:05:00.032156+0000","last_clean":"2024-01-15T18:05:00.032156+0000","last_became_active":"2024-01-15T18:04:58.869133+0000","last_became_peered":"2024-01-15T18:04:58.869133+0000","last_unstale":"2024-01-15T18:05:00.032156+0000","last_undegraded":"2024-01-15T18:05:00.032156+0000","last_fullsized":"2024-01-15T18:05:00.032156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:39:08.175871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849313+0000","last_change":"2024-01-15T18:04:57.830731+0000","last_active":"2024-01-15T18:04:59.849313+0000","last_peered":"2024-01-15T18:04:59.849313+0000","last_clean":"2024-01-15T18:04:59.849313+0000","last_became_active":"2024-01-15T18:04:57.830234+0000","last_became_peered":"2024-01-15T18:04:57.830234+0000","last_unstale":"2024-01-15T18:04:59.849313+0000","last_undegraded":"2024-01-15T18:04:59.849313+0000","last_fullsized":"2024-01-15T18:04:59.849313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:37:44.430593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847812+0000","last_change":"2024-01-15T18:04:58.843965+0000","last_active":"2024-01-15T18:04:59.847812+0000","last_peered":"2024-01-15T18:04:59.847812+0000","last_clean":"2024-01-15T18:04:59.847812+0000","last_became_active":"2024-01-15T18:04:58.841866+0000","last_became_peered":"2024-01-15T18:04:58.841866+0000","last_unstale":"2024-01-15T18:04:59.847812+0000","last_undegraded":"2024-01-15T18:04:59.847812+0000","last_fullsized":"2024-01-15T18:04:59.847812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:16:41.986399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030724+0000","last_change":"2024-01-15T18:04:57.836644+0000","last_active":"2024-01-15T18:05:00.030724+0000","last_peered":"2024-01-15T18:05:00.030724+0000","last_clean":"2024-01-15T18:05:00.030724+0000","last_became_active":"2024-01-15T18:04:57.835872+0000","last_became_peered":"2024-01-15T18:04:57.835872+0000","last_unstale":"2024-01-15T18:05:00.030724+0000","last_undegraded":"2024-01-15T18:05:00.030724+0000","last_fullsized":"2024-01-15T18:05:00.030724+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:44:30.299838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031872+0000","last_change":"2024-01-15T18:04:58.857962+0000","last_active":"2024-01-15T18:05:00.031872+0000","last_peered":"2024-01-15T18:05:00.031872+0000","last_clean":"2024-01-15T18:05:00.031872+0000","last_became_active":"2024-01-15T18:04:58.857644+0000","last_became_peered":"2024-01-15T18:04:58.857644+0000","last_unstale":"2024-01-15T18:05:00.031872+0000","last_undegraded":"2024-01-15T18:05:00.031872+0000","last_fullsized":"2024-01-15T18:05:00.031872+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:02:19.202437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:58.836746+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:04:58.836746+0000","last_peered":"2024-01-15T18:04:58.836746+0000","last_clean":"2024-01-15T18:04:58.836746+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:04:58.836746+0000","last_undegraded":"2024-01-15T18:04:58.836746+0000","last_fullsized":"2024-01-15T18:04:58.836746+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.161917+0000","last_change":"2024-01-15T18:04:57.829926+0000","last_active":"2024-01-15T18:04:59.161917+0000","last_peered":"2024-01-15T18:04:59.161917+0000","last_clean":"2024-01-15T18:04:59.161917+0000","last_became_active":"2024-01-15T18:04:57.829578+0000","last_became_peered":"2024-01-15T18:04:57.829578+0000","last_unstale":"2024-01-15T18:04:59.161917+0000","last_undegraded":"2024-01-15T18:04:59.161917+0000","last_fullsized":"2024-01-15T18:04:59.161917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:55:52.024992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032132+0000","last_change":"2024-01-15T18:04:58.862009+0000","last_active":"2024-01-15T18:05:00.032132+0000","last_peered":"2024-01-15T18:05:00.032132+0000","last_clean":"2024-01-15T18:05:00.032132+0000","last_became_active":"2024-01-15T18:04:58.861745+0000","last_became_peered":"2024-01-15T18:04:58.861745+0000","last_unstale":"2024-01-15T18:05:00.032132+0000","last_undegraded":"2024-01-15T18:05:00.032132+0000","last_fullsized":"2024-01-15T18:05:00.032132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:15:34.376159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031331+0000","last_change":"2024-01-15T18:04:57.842536+0000","last_active":"2024-01-15T18:05:00.031331+0000","last_peered":"2024-01-15T18:05:00.031331+0000","last_clean":"2024-01-15T18:05:00.031331+0000","last_became_active":"2024-01-15T18:04:57.842236+0000","last_became_peered":"2024-01-15T18:04:57.842236+0000","last_unstale":"2024-01-15T18:05:00.031331+0000","last_undegraded":"2024-01-15T18:05:00.031331+0000","last_fullsized":"2024-01-15T18:05:00.031331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:24:33.632172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030699+0000","last_change":"2024-01-15T18:04:58.859545+0000","last_active":"2024-01-15T18:05:00.030699+0000","last_peered":"2024-01-15T18:05:00.030699+0000","last_clean":"2024-01-15T18:05:00.030699+0000","last_became_active":"2024-01-15T18:04:58.859282+0000","last_became_peered":"2024-01-15T18:04:58.859282+0000","last_unstale":"2024-01-15T18:05:00.030699+0000","last_undegraded":"2024-01-15T18:05:00.030699+0000","last_fullsized":"2024-01-15T18:05:00.030699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:29:02.731583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030970+0000","last_change":"2024-01-15T18:04:57.826351+0000","last_active":"2024-01-15T18:05:00.030970+0000","last_peered":"2024-01-15T18:05:00.030970+0000","last_clean":"2024-01-15T18:05:00.030970+0000","last_became_active":"2024-01-15T18:04:57.825983+0000","last_became_peered":"2024-01-15T18:04:57.825983+0000","last_unstale":"2024-01-15T18:05:00.030970+0000","last_undegraded":"2024-01-15T18:05:00.030970+0000","last_fullsized":"2024-01-15T18:05:00.030970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:30:11.561882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849021+0000","last_change":"2024-01-15T18:04:58.834900+0000","last_active":"2024-01-15T18:04:59.849021+0000","last_peered":"2024-01-15T18:04:59.849021+0000","last_clean":"2024-01-15T18:04:59.849021+0000","last_became_active":"2024-01-15T18:04:58.834561+0000","last_became_peered":"2024-01-15T18:04:58.834561+0000","last_unstale":"2024-01-15T18:04:59.849021+0000","last_undegraded":"2024-01-15T18:04:59.849021+0000","last_fullsized":"2024-01-15T18:04:59.849021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:10.719289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.161907+0000","last_change":"2024-01-15T18:04:57.840353+0000","last_active":"2024-01-15T18:04:59.161907+0000","last_peered":"2024-01-15T18:04:59.161907+0000","last_clean":"2024-01-15T18:04:59.161907+0000","last_became_active":"2024-01-15T18:04:57.840032+0000","last_became_peered":"2024-01-15T18:04:57.840032+0000","last_unstale":"2024-01-15T18:04:59.161907+0000","last_undegraded":"2024-01-15T18:04:59.161907+0000","last_fullsized":"2024-01-15T18:04:59.161907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:21:05.984228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847638+0000","last_change":"2024-01-15T18:04:58.862393+0000","last_active":"2024-01-15T18:04:59.847638+0000","last_peered":"2024-01-15T18:04:59.847638+0000","last_clean":"2024-01-15T18:04:59.847638+0000","last_became_active":"2024-01-15T18:04:58.861776+0000","last_became_peered":"2024-01-15T18:04:58.861776+0000","last_unstale":"2024-01-15T18:04:59.847638+0000","last_undegraded":"2024-01-15T18:04:59.847638+0000","last_fullsized":"2024-01-15T18:04:59.847638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:58:16.651562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849271+0000","last_change":"2024-01-15T18:04:57.832247+0000","last_active":"2024-01-15T18:04:59.849271+0000","last_peered":"2024-01-15T18:04:59.849271+0000","last_clean":"2024-01-15T18:04:59.849271+0000","last_became_active":"2024-01-15T18:04:57.831962+0000","last_became_peered":"2024-01-15T18:04:57.831962+0000","last_unstale":"2024-01-15T18:04:59.849271+0000","last_undegraded":"2024-01-15T18:04:59.849271+0000","last_fullsized":"2024-01-15T18:04:59.849271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:08:42.272780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.848124+0000","last_change":"2024-01-15T18:04:58.842320+0000","last_active":"2024-01-15T18:04:59.848124+0000","last_peered":"2024-01-15T18:04:59.848124+0000","last_clean":"2024-01-15T18:04:59.848124+0000","last_became_active":"2024-01-15T18:04:58.841887+0000","last_became_peered":"2024-01-15T18:04:58.841887+0000","last_unstale":"2024-01-15T18:04:59.848124+0000","last_undegraded":"2024-01-15T18:04:59.848124+0000","last_fullsized":"2024-01-15T18:04:59.848124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:19.608361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032034+0000","last_change":"2024-01-15T18:04:57.843600+0000","last_active":"2024-01-15T18:05:00.032034+0000","last_peered":"2024-01-15T18:05:00.032034+0000","last_clean":"2024-01-15T18:05:00.032034+0000","last_became_active":"2024-01-15T18:04:57.843179+0000","last_became_peered":"2024-01-15T18:04:57.843179+0000","last_unstale":"2024-01-15T18:05:00.032034+0000","last_undegraded":"2024-01-15T18:05:00.032034+0000","last_fullsized":"2024-01-15T18:05:00.032034+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:10:23.513336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849104+0000","last_change":"2024-01-15T18:04:58.834860+0000","last_active":"2024-01-15T18:04:59.849104+0000","last_peered":"2024-01-15T18:04:59.849104+0000","last_clean":"2024-01-15T18:04:59.849104+0000","last_became_active":"2024-01-15T18:04:58.834570+0000","last_became_peered":"2024-01-15T18:04:58.834570+0000","last_unstale":"2024-01-15T18:04:59.849104+0000","last_undegraded":"2024-01-15T18:04:59.849104+0000","last_fullsized":"2024-01-15T18:04:59.849104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:12:00.696642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.162824+0000","last_change":"2024-01-15T18:04:57.832391+0000","last_active":"2024-01-15T18:05:00.162824+0000","last_peered":"2024-01-15T18:05:00.162824+0000","last_clean":"2024-01-15T18:05:00.162824+0000","last_became_active":"2024-01-15T18:04:57.832005+0000","last_became_peered":"2024-01-15T18:04:57.832005+0000","last_unstale":"2024-01-15T18:05:00.162824+0000","last_undegraded":"2024-01-15T18:05:00.162824+0000","last_fullsized":"2024-01-15T18:05:00.162824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:24:33.632172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032511+0000","last_change":"2024-01-15T18:04:58.863289+0000","last_active":"2024-01-15T18:05:00.032511+0000","last_peered":"2024-01-15T18:05:00.032511+0000","last_clean":"2024-01-15T18:05:00.032511+0000","last_became_active":"2024-01-15T18:04:58.862985+0000","last_became_peered":"2024-01-15T18:04:58.862985+0000","last_unstale":"2024-01-15T18:05:00.032511+0000","last_undegraded":"2024-01-15T18:05:00.032511+0000","last_fullsized":"2024-01-15T18:05:00.032511+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:48:27.499654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031100+0000","last_change":"2024-01-15T18:04:57.835018+0000","last_active":"2024-01-15T18:05:00.031100+0000","last_peered":"2024-01-15T18:05:00.031100+0000","last_clean":"2024-01-15T18:05:00.031100+0000","last_became_active":"2024-01-15T18:04:57.834706+0000","last_became_peered":"2024-01-15T18:04:57.834706+0000","last_unstale":"2024-01-15T18:05:00.031100+0000","last_undegraded":"2024-01-15T18:05:00.031100+0000","last_fullsized":"2024-01-15T18:05:00.031100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:11:48.897513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031377+0000","last_change":"2024-01-15T18:04:58.866805+0000","last_active":"2024-01-15T18:05:00.031377+0000","last_peered":"2024-01-15T18:05:00.031377+0000","last_clean":"2024-01-15T18:05:00.031377+0000","last_became_active":"2024-01-15T18:04:58.866475+0000","last_became_peered":"2024-01-15T18:04:58.866475+0000","last_unstale":"2024-01-15T18:05:00.031377+0000","last_undegraded":"2024-01-15T18:05:00.031377+0000","last_fullsized":"2024-01-15T18:05:00.031377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:12:02.202965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031326+0000","last_change":"2024-01-15T18:04:57.839780+0000","last_active":"2024-01-15T18:05:00.031326+0000","last_peered":"2024-01-15T18:05:00.031326+0000","last_clean":"2024-01-15T18:05:00.031326+0000","last_became_active":"2024-01-15T18:04:57.839392+0000","last_became_peered":"2024-01-15T18:04:57.839392+0000","last_unstale":"2024-01-15T18:05:00.031326+0000","last_undegraded":"2024-01-15T18:05:00.031326+0000","last_fullsized":"2024-01-15T18:05:00.031326+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:16:29.440534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847071+0000","last_change":"2024-01-15T18:04:58.835003+0000","last_active":"2024-01-15T18:04:59.847071+0000","last_peered":"2024-01-15T18:04:59.847071+0000","last_clean":"2024-01-15T18:04:59.847071+0000","last_became_active":"2024-01-15T18:04:58.834678+0000","last_became_peered":"2024-01-15T18:04:58.834678+0000","last_unstale":"2024-01-15T18:04:59.847071+0000","last_undegraded":"2024-01-15T18:04:59.847071+0000","last_fullsized":"2024-01-15T18:04:59.847071+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:04:14.249111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849091+0000","last_change":"2024-01-15T18:04:57.837454+0000","last_active":"2024-01-15T18:04:59.849091+0000","last_peered":"2024-01-15T18:04:59.849091+0000","last_clean":"2024-01-15T18:04:59.849091+0000","last_became_active":"2024-01-15T18:04:57.836947+0000","last_became_peered":"2024-01-15T18:04:57.836947+0000","last_unstale":"2024-01-15T18:04:59.849091+0000","last_undegraded":"2024-01-15T18:04:59.849091+0000","last_fullsized":"2024-01-15T18:04:59.849091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:11:48.897513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031977+0000","last_change":"2024-01-15T18:04:58.854684+0000","last_active":"2024-01-15T18:05:00.031977+0000","last_peered":"2024-01-15T18:05:00.031977+0000","last_clean":"2024-01-15T18:05:00.031977+0000","last_became_active":"2024-01-15T18:04:58.844371+0000","last_became_peered":"2024-01-15T18:04:58.844371+0000","last_unstale":"2024-01-15T18:05:00.031977+0000","last_undegraded":"2024-01-15T18:05:00.031977+0000","last_fullsized":"2024-01-15T18:05:00.031977+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:12:00.696642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.849194+0000","last_change":"2024-01-15T18:04:57.837432+0000","last_active":"2024-01-15T18:04:59.849194+0000","last_peered":"2024-01-15T18:04:59.849194+0000","last_clean":"2024-01-15T18:04:59.849194+0000","last_became_active":"2024-01-15T18:04:57.837105+0000","last_became_peered":"2024-01-15T18:04:57.837105+0000","last_unstale":"2024-01-15T18:04:59.849194+0000","last_undegraded":"2024-01-15T18:04:59.849194+0000","last_fullsized":"2024-01-15T18:04:59.849194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:04:13.242693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032216+0000","last_change":"2024-01-15T18:04:58.859567+0000","last_active":"2024-01-15T18:05:00.032216+0000","last_peered":"2024-01-15T18:05:00.032216+0000","last_clean":"2024-01-15T18:05:00.032216+0000","last_became_active":"2024-01-15T18:04:58.859199+0000","last_became_peered":"2024-01-15T18:04:58.859199+0000","last_unstale":"2024-01-15T18:05:00.032216+0000","last_undegraded":"2024-01-15T18:05:00.032216+0000","last_fullsized":"2024-01-15T18:05:00.032216+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:10:24.519754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031791+0000","last_change":"2024-01-15T18:04:57.843450+0000","last_active":"2024-01-15T18:05:00.031791+0000","last_peered":"2024-01-15T18:05:00.031791+0000","last_clean":"2024-01-15T18:05:00.031791+0000","last_became_active":"2024-01-15T18:04:57.841883+0000","last_became_peered":"2024-01-15T18:04:57.841883+0000","last_unstale":"2024-01-15T18:05:00.031791+0000","last_undegraded":"2024-01-15T18:05:00.031791+0000","last_fullsized":"2024-01-15T18:05:00.031791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:17:50.737228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.030851+0000","last_change":"2024-01-15T18:04:58.845515+0000","last_active":"2024-01-15T18:05:00.030851+0000","last_peered":"2024-01-15T18:05:00.030851+0000","last_clean":"2024-01-15T18:05:00.030851+0000","last_became_active":"2024-01-15T18:04:58.845242+0000","last_became_peered":"2024-01-15T18:04:58.845242+0000","last_unstale":"2024-01-15T18:05:00.030851+0000","last_undegraded":"2024-01-15T18:05:00.030851+0000","last_fullsized":"2024-01-15T18:05:00.030851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:36.680373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.848091+0000","last_change":"2024-01-15T18:04:58.840618+0000","last_active":"2024-01-15T18:04:59.848091+0000","last_peered":"2024-01-15T18:04:59.848091+0000","last_clean":"2024-01-15T18:04:59.848091+0000","last_became_active":"2024-01-15T18:04:58.840306+0000","last_became_peered":"2024-01-15T18:04:58.840306+0000","last_unstale":"2024-01-15T18:04:59.848091+0000","last_undegraded":"2024-01-15T18:04:59.848091+0000","last_fullsized":"2024-01-15T18:04:59.848091+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:30:56.054090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031572+0000","last_change":"2024-01-15T18:04:57.842676+0000","last_active":"2024-01-15T18:05:00.031572+0000","last_peered":"2024-01-15T18:05:00.031572+0000","last_clean":"2024-01-15T18:05:00.031572+0000","last_became_active":"2024-01-15T18:04:57.842402+0000","last_became_peered":"2024-01-15T18:04:57.842402+0000","last_unstale":"2024-01-15T18:05:00.031572+0000","last_undegraded":"2024-01-15T18:05:00.031572+0000","last_fullsized":"2024-01-15T18:05:00.031572+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:09.712871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.847691+0000","last_change":"2024-01-15T18:04:58.837433+0000","last_active":"2024-01-15T18:04:59.847691+0000","last_peered":"2024-01-15T18:04:59.847691+0000","last_clean":"2024-01-15T18:04:59.847691+0000","last_became_active":"2024-01-15T18:04:58.836989+0000","last_became_peered":"2024-01-15T18:04:58.836989+0000","last_unstale":"2024-01-15T18:04:59.847691+0000","last_undegraded":"2024-01-15T18:04:59.847691+0000","last_fullsized":"2024-01-15T18:04:59.847691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:25:24.113321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031908+0000","last_change":"2024-01-15T18:04:57.834129+0000","last_active":"2024-01-15T18:05:00.031908+0000","last_peered":"2024-01-15T18:05:00.031908+0000","last_clean":"2024-01-15T18:05:00.031908+0000","last_became_active":"2024-01-15T18:04:57.833804+0000","last_became_peered":"2024-01-15T18:04:57.833804+0000","last_unstale":"2024-01-15T18:05:00.031908+0000","last_undegraded":"2024-01-15T18:05:00.031908+0000","last_fullsized":"2024-01-15T18:05:00.031908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:11:59.690224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.031347+0000","last_change":"2024-01-15T18:04:58.860153+0000","last_active":"2024-01-15T18:05:00.031347+0000","last_peered":"2024-01-15T18:05:00.031347+0000","last_clean":"2024-01-15T18:05:00.031347+0000","last_became_active":"2024-01-15T18:04:58.855401+0000","last_became_peered":"2024-01-15T18:04:58.855401+0000","last_unstale":"2024-01-15T18:05:00.031347+0000","last_undegraded":"2024-01-15T18:05:00.031347+0000","last_fullsized":"2024-01-15T18:05:00.031347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:48:17.399280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032517+0000","last_change":"2024-01-15T18:04:57.830136+0000","last_active":"2024-01-15T18:05:00.032517+0000","last_peered":"2024-01-15T18:05:00.032517+0000","last_clean":"2024-01-15T18:05:00.032517+0000","last_became_active":"2024-01-15T18:04:57.829333+0000","last_became_peered":"2024-01-15T18:04:57.829333+0000","last_unstale":"2024-01-15T18:05:00.032517+0000","last_undegraded":"2024-01-15T18:05:00.032517+0000","last_fullsized":"2024-01-15T18:05:00.032517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:37:44.430593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T18:05:00.032421+0000","last_change":"2024-01-15T18:04:58.855872+0000","last_active":"2024-01-15T18:05:00.032421+0000","last_peered":"2024-01-15T18:05:00.032421+0000","last_clean":"2024-01-15T18:05:00.032421+0000","last_became_active":"2024-01-15T18:04:58.854897+0000","last_became_peered":"2024-01-15T18:04:58.854897+0000","last_unstale":"2024-01-15T18:05:00.032421+0000","last_undegraded":"2024-01-15T18:05:00.032421+0000","last_fullsized":"2024-01-15T18:05:00.032421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:57.803969+0000","last_clean_scrub_stamp":"2024-01-15T18:04:57.803969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:33:52.636110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-15T18:04:59.159371+0000","last_change":"2024-01-15T18:04:57.843229+0000","last_active":"2024-01-15T18:04:59.159371+0000","last_peered":"2024-01-15T18:04:59.159371+0000","last_clean":"2024-01-15T18:04:59.159371+0000","last_became_active":"2024-01-15T18:04:57.842720+0000","last_became_peered":"2024-01-15T18:04:57.842720+0000","last_unstale":"2024-01-15T18:04:59.159371+0000","last_undegraded":"2024-01-15T18:04:59.159371+0000","last_fullsized":"2024-01-15T18:04:59.159371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:56.797552+0000","last_clean_scrub_stamp":"2024-01-15T18:04:56.797552+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:25:33.145590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":806912,"data_stored":663105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93300000000000005}]}]},{"osd":6,"up_from":15,"seq":64424509447,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":73767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.181}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4220000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.012}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]}]},{"osd":4,"up_from":15,"seq":64424509447,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":72281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]}]},{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":73767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]}]},{"osd":2,"up_from":15,"seq":64424509446,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":72155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]}]},{"osd":1,"up_from":15,"seq":64424509448,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":212992,"data_stored":72245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]}]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":815104,"data_stored":663161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.462}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0960000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6739999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6890000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6559999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7450000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:05:08.016 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-15T18:05:08.016 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-15T18:05:08.017 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-01-15T18:05:08.017 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-01-15T18:05:08.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.188+0000 7ff5e9600640 1 Processor -- start 2024-01-15T18:05:08.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.188+0000 7ff5e9600640 1 -- start start 2024-01-15T18:05:08.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.192+0000 7ff5e9600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff5e406bb80 0x7ff5e40cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:08.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.192+0000 7ff5e9600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff5e40cddc0 0x7ff5e40b0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:08.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.192+0000 7ff5e9600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5e40b2770 con 0x7ff5e40b2390 2024-01-15T18:05:08.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.192+0000 7ff5e9600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5e40ce270 con 0x7ff5e40cddc0 2024-01-15T18:05:08.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.192+0000 7ff5e9600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e406c060 con 0x7ff5e406bb80 2024-01-15T18:05:08.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.192+0000 7ff5e2a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff5e40b2390 0x7ff5e40cf780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42428/0 (socket says 172.21.15.27:42428) 2024-01-15T18:05:08.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.192+0000 7ff5e2a00640 1 -- 172.21.15.27:0/28897056 learned_addr learned my addr 172.21.15.27:0/28897056 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:08.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.192+0000 7ff5e2000640 1 --2- 172.21.15.27:0/28897056 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff5e406bb80 0x7ff5e40cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:08.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.192+0000 7ff5e3400640 1 --2- 172.21.15.27:0/28897056 >> v2:172.21.15.93:3300/0 conn(0x7ff5e40cddc0 0x7ff5e40b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:08.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.192+0000 7ff5e1600640 1 -- 172.21.15.27:0/28897056 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990683978 0 0) 0x7ff5e40b2770 con 0x7ff5e40b2390 2024-01-15T18:05:08.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.192+0000 7ff5e1600640 1 -- 172.21.15.27:0/28897056 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5c4003530 con 0x7ff5e40b2390 2024-01-15T18:05:08.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.192+0000 7ff5e3400640 1 -- 172.21.15.27:0/28897056 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff5e406bb80 msgr2=0x7ff5e40cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:08.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.192+0000 7ff5e3400640 1 --2- 172.21.15.27:0/28897056 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff5e406bb80 0x7ff5e40cd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.192+0000 7ff5e3400640 1 -- 172.21.15.27:0/28897056 >> v1:172.21.15.27:6789/0 conn(0x7ff5e40b2390 legacy=0x7ff5e40cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:08.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.192+0000 7ff5e3400640 1 -- 172.21.15.27:0/28897056 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e406c340 con 0x7ff5e40cddc0 2024-01-15T18:05:08.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.196+0000 7ff5e3400640 1 --2- 172.21.15.27:0/28897056 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff5e40cddc0 0x7ff5e40b0880 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc009d90 tx=0x7ff5dc01c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8465080a785405e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:08.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.196+0000 7ff5e1600640 1 -- 172.21.15.27:0/28897056 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc0a59e0 con 0x7ff5e40cddc0 2024-01-15T18:05:08.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.196+0000 7ff5e1600640 1 -- 172.21.15.27:0/28897056 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5dc0a5b40 con 0x7ff5e40cddc0 2024-01-15T18:05:08.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.196+0000 7ff5e1600640 1 -- 172.21.15.27:0/28897056 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc0ad440 con 0x7ff5e40cddc0 2024-01-15T18:05:08.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.196+0000 7ff5e9600640 1 -- 172.21.15.27:0/28897056 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff5e40cddc0 msgr2=0x7ff5e40b0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:08.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.196+0000 7ff5e9600640 1 --2- 172.21.15.27:0/28897056 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff5e40cddc0 0x7ff5e40b0880 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc009d90 tx=0x7ff5dc01c920 comp rx=0 tx=0).stop 2024-01-15T18:05:08.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.196+0000 7ff5e9600640 1 -- 172.21.15.27:0/28897056 shutdown_connections 2024-01-15T18:05:08.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.196+0000 7ff5e9600640 1 --2- 172.21.15.27:0/28897056 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff5e406bb80 0x7ff5e40cd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.196+0000 7ff5e9600640 1 --2- 172.21.15.27:0/28897056 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff5e40cddc0 0x7ff5e40b0880 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.196+0000 7ff5e9600640 1 -- 172.21.15.27:0/28897056 >> 172.21.15.27:0/28897056 conn(0x7ff5e40b2a10 msgr2=0x7ff5e40b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:08.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.196+0000 7ff5e9600640 1 -- 172.21.15.27:0/28897056 shutdown_connections 2024-01-15T18:05:08.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.196+0000 7ff5e9600640 1 -- 172.21.15.27:0/28897056 wait complete. 2024-01-15T18:05:08.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.196+0000 7ff5e9600640 1 Processor -- start 2024-01-15T18:05:08.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.200+0000 7ff5e9600640 1 -- start start 2024-01-15T18:05:08.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.200+0000 7ff5e9600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5e406bb80 0x7ff5e41568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:08.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.200+0000 7ff5e9600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff5e40b2390 0x7ff5e40c5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:08.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.200+0000 7ff5e9600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff5e40cddc0 0x7ff5e41596b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:08.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.200+0000 7ff5e9600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e4156e10 con 0x7ff5e406bb80 2024-01-15T18:05:08.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.200+0000 7ff5e9600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e40c6030 con 0x7ff5e40b2390 2024-01-15T18:05:08.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.200+0000 7ff5e9600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e415c6b0 con 0x7ff5e40cddc0 2024-01-15T18:05:08.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.200+0000 7ff5e2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5e406bb80 0x7ff5e41568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:08.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.200+0000 7ff5e2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5e406bb80 0x7ff5e41568d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59424/0 (socket says 172.21.15.27:59424) 2024-01-15T18:05:08.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.200+0000 7ff5e2a00640 1 -- 172.21.15.27:0/3240381071 learned_addr learned my addr 172.21.15.27:0/3240381071 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:08.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.200+0000 7ff5e3400640 1 --2- 172.21.15.27:0/3240381071 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff5e40cddc0 0x7ff5e41596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:08.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.200+0000 7ff5e2000640 1 --2- 172.21.15.27:0/3240381071 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff5e40b2390 0x7ff5e40c5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:08.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.200+0000 7ff5e2a00640 1 -- 172.21.15.27:0/3240381071 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff5e40cddc0 msgr2=0x7ff5e41596b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:08.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.200+0000 7ff5e2a00640 1 --2- 172.21.15.27:0/3240381071 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff5e40cddc0 0x7ff5e41596b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.200+0000 7ff5e2a00640 1 -- 172.21.15.27:0/3240381071 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff5e40b2390 msgr2=0x7ff5e40c5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:08.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.200+0000 7ff5e2a00640 1 --2- 172.21.15.27:0/3240381071 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff5e40b2390 0x7ff5e40c5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.200+0000 7ff5e2a00640 1 -- 172.21.15.27:0/3240381071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5dc009a70 con 0x7ff5e406bb80 2024-01-15T18:05:08.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.200+0000 7ff5e2a00640 1 --2- 172.21.15.27:0/3240381071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5e406bb80 0x7ff5e41568d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff5d400c4d0 tx=0x7ff5d400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:08.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.200+0000 7ff5daa00640 1 -- 172.21.15.27:0/3240381071 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4005cf0 con 0x7ff5e406bb80 2024-01-15T18:05:08.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.200+0000 7ff5daa00640 1 -- 172.21.15.27:0/3240381071 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d4005e50 con 0x7ff5e406bb80 2024-01-15T18:05:08.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.200+0000 7ff5e9600640 1 -- 172.21.15.27:0/3240381071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5e415c990 con 0x7ff5e406bb80 2024-01-15T18:05:08.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.200+0000 7ff5daa00640 1 -- 172.21.15.27:0/3240381071 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4007270 con 0x7ff5e406bb80 2024-01-15T18:05:08.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.200+0000 7ff5e9600640 1 -- 172.21.15.27:0/3240381071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5e415cf50 con 0x7ff5e406bb80 2024-01-15T18:05:08.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.200+0000 7ff5e9600640 1 -- 172.21.15.27:0/3240381071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5e40d07c0 con 0x7ff5e406bb80 2024-01-15T18:05:08.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.204+0000 7ff5daa00640 1 -- 172.21.15.27:0/3240381071 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7ff5d40073d0 con 0x7ff5e406bb80 2024-01-15T18:05:08.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.204+0000 7ff5daa00640 1 --2- 172.21.15.27:0/3240381071 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff5bc0668f0 0x7ff5bc068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:08.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.204+0000 7ff5daa00640 1 -- 172.21.15.27:0/3240381071 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff5d40839a0 con 0x7ff5e406bb80 2024-01-15T18:05:08.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.204+0000 7ff5e2000640 1 --2- 172.21.15.27:0/3240381071 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff5bc0668f0 0x7ff5bc068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:08.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.204+0000 7ff5e2000640 1 --2- 172.21.15.27:0/3240381071 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff5bc0668f0 0x7ff5bc068db0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff5e415aba0 tx=0x7ff5cc009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:08.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.208+0000 7ff5daa00640 1 -- 172.21.15.27:0/3240381071 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff5d4050bf0 con 0x7ff5e406bb80 2024-01-15T18:05:08.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.400+0000 7ff5e9600640 1 -- 172.21.15.27:0/3240381071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff5e404e850 con 0x7ff5e406bb80 2024-01-15T18:05:08.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.400+0000 7ff5daa00640 1 -- 172.21.15.27:0/3240381071 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7ff5d4017080 con 0x7ff5e406bb80 2024-01-15T18:05:08.403 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:08.404 INFO:teuthology.orchestra.run.smithi027.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-15T18:05:08.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.400+0000 7ff5e9600640 1 -- 172.21.15.27:0/3240381071 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff5bc0668f0 msgr2=0x7ff5bc068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:08.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.400+0000 7ff5e9600640 1 --2- 172.21.15.27:0/3240381071 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff5bc0668f0 0x7ff5bc068db0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff5e415aba0 tx=0x7ff5cc009190 comp rx=0 tx=0).stop 2024-01-15T18:05:08.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.400+0000 7ff5e9600640 1 -- 172.21.15.27:0/3240381071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5e406bb80 msgr2=0x7ff5e41568d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:08.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.400+0000 7ff5e9600640 1 --2- 172.21.15.27:0/3240381071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5e406bb80 0x7ff5e41568d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff5d400c4d0 tx=0x7ff5d400c990 comp rx=0 tx=0).stop 2024-01-15T18:05:08.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.400+0000 7ff5e9600640 1 -- 172.21.15.27:0/3240381071 shutdown_connections 2024-01-15T18:05:08.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.400+0000 7ff5e9600640 1 --2- 172.21.15.27:0/3240381071 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff5bc0668f0 0x7ff5bc068db0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.400+0000 7ff5e9600640 1 --2- 172.21.15.27:0/3240381071 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff5e40cddc0 0x7ff5e41596b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.400+0000 7ff5e9600640 1 --2- 172.21.15.27:0/3240381071 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff5e40b2390 0x7ff5e40c5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.400+0000 7ff5e9600640 1 --2- 172.21.15.27:0/3240381071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5e406bb80 0x7ff5e41568d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.400+0000 7ff5e9600640 1 -- 172.21.15.27:0/3240381071 >> 172.21.15.27:0/3240381071 conn(0x7ff5e40b2a10 msgr2=0x7ff5e406ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:08.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.400+0000 7ff5e9600640 1 -- 172.21.15.27:0/3240381071 shutdown_connections 2024-01-15T18:05:08.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.400+0000 7ff5e9600640 1 -- 172.21.15.27:0/3240381071 wait complete. 2024-01-15T18:05:08.426 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-01-15T18:05:08.426 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:08.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd64e00640 1 Processor -- start 2024-01-15T18:05:08.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd64e00640 1 -- start start 2024-01-15T18:05:08.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd64e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7efd600cf9b0 0x7efd600b10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:08.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd64e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd600b38b0 con 0x7efd6015e7d0 2024-01-15T18:05:08.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd64e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd600cd550 con 0x7efd600cedc0 2024-01-15T18:05:08.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd64e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd600cd270 con 0x7efd600cf9b0 2024-01-15T18:05:08.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5de00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7efd6015e7d0 0x7efd600cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42434/0 (socket says 172.21.15.27:42434) 2024-01-15T18:05:08.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5de00640 1 -- 172.21.15.27:0/2593145665 learned_addr learned my addr 172.21.15.27:0/2593145665 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:08.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5ca00640 1 -- 172.21.15.27:0/2593145665 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 753335704 0 0) 0x7efd600b38b0 con 0x7efd6015e7d0 2024-01-15T18:05:08.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5ca00640 1 -- 172.21.15.27:0/2593145665 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd40003530 con 0x7efd6015e7d0 2024-01-15T18:05:08.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5d400640 1 --2- 172.21.15.27:0/2593145665 >> v2:172.21.15.93:3300/0 conn(0x7efd600cf9b0 0x7efd600b10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:08.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5ca00640 1 -- 172.21.15.27:0/2593145665 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 518296929 0 0) 0x7efd40003530 con 0x7efd6015e7d0 2024-01-15T18:05:08.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5ca00640 1 -- 172.21.15.27:0/2593145665 >> v2:172.21.15.93:3300/0 conn(0x7efd600cf9b0 msgr2=0x7efd600b10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:08.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5ca00640 1 --2- 172.21.15.27:0/2593145665 >> v2:172.21.15.93:3300/0 conn(0x7efd600cf9b0 0x7efd600b10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5ca00640 1 -- 172.21.15.27:0/2593145665 >> v1:172.21.15.93:6789/0 conn(0x7efd600cedc0 legacy=0x7efd600b1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:08.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5ca00640 1 -- 172.21.15.27:0/2593145665 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd600b35d0 con 0x7efd6015e7d0 2024-01-15T18:05:08.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5ca00640 1 -- 172.21.15.27:0/2593145665 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7efd50001680 con 0x7efd6015e7d0 2024-01-15T18:05:08.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5ca00640 1 -- 172.21.15.27:0/2593145665 >> v1:172.21.15.27:6789/0 conn(0x7efd6015e7d0 legacy=0x7efd600cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:08.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5ca00640 1 --2- 172.21.15.27:0/2593145665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd400039f0 0x7efd40003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:08.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5ca00640 1 --2- 172.21.15.27:0/2593145665 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efd400047d0 0x7efd40004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:08.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5ca00640 1 --2- 172.21.15.27:0/2593145665 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efd40005200 0x7efd4000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:08.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5ca00640 1 -- 172.21.15.27:0/2593145665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd600cd270 con 0x7efd400039f0 2024-01-15T18:05:08.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5ca00640 1 -- 172.21.15.27:0/2593145665 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4000c160 con 0x7efd40005200 2024-01-15T18:05:08.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5ca00640 1 -- 172.21.15.27:0/2593145665 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd4000c320 con 0x7efd400047d0 2024-01-15T18:05:08.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5de00640 1 --2- 172.21.15.27:0/2593145665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd400039f0 0x7efd40003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:08.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5d400640 1 --2- 172.21.15.27:0/2593145665 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efd400047d0 0x7efd40004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:08.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5e800640 1 --2- 172.21.15.27:0/2593145665 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efd40005200 0x7efd4000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:08.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5de00640 1 -- 172.21.15.27:0/2593145665 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efd400047d0 msgr2=0x7efd40004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:08.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5de00640 1 --2- 172.21.15.27:0/2593145665 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efd400047d0 0x7efd40004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5de00640 1 -- 172.21.15.27:0/2593145665 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efd40005200 msgr2=0x7efd4000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:08.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5de00640 1 --2- 172.21.15.27:0/2593145665 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efd40005200 0x7efd4000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5de00640 1 -- 172.21.15.27:0/2593145665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd40003530 con 0x7efd400039f0 2024-01-15T18:05:08.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.564+0000 7efd5e800640 1 -- 172.21.15.27:0/2593145665 reap_dead start 2024-01-15T18:05:08.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.568+0000 7efd5de00640 1 --2- 172.21.15.27:0/2593145665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd400039f0 0x7efd40003e10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efd5000c240 tx=0x7efd5001cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0ee7c952eb2818 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:08.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.568+0000 7efd5ca00640 1 -- 172.21.15.27:0/2593145665 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd50005ae0 con 0x7efd400039f0 2024-01-15T18:05:08.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.568+0000 7efd5ca00640 1 -- 172.21.15.27:0/2593145665 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd50005c40 con 0x7efd400039f0 2024-01-15T18:05:08.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.568+0000 7efd5ca00640 1 -- 172.21.15.27:0/2593145665 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd500a5a40 con 0x7efd400039f0 2024-01-15T18:05:08.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.568+0000 7efd64e00640 1 -- 172.21.15.27:0/2593145665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd400039f0 msgr2=0x7efd40003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:08.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.568+0000 7efd64e00640 1 --2- 172.21.15.27:0/2593145665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd400039f0 0x7efd40003e10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efd5000c240 tx=0x7efd5001cb80 comp rx=0 tx=0).stop 2024-01-15T18:05:08.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.568+0000 7efd64e00640 1 -- 172.21.15.27:0/2593145665 shutdown_connections 2024-01-15T18:05:08.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.568+0000 7efd64e00640 1 --2- 172.21.15.27:0/2593145665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd400039f0 0x7efd40003e10 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.568+0000 7efd64e00640 1 -- 172.21.15.27:0/2593145665 >> 172.21.15.27:0/2593145665 conn(0x7efd600b2a10 msgr2=0x7efd600b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:08.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.568+0000 7efd64e00640 1 -- 172.21.15.27:0/2593145665 shutdown_connections 2024-01-15T18:05:08.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.568+0000 7efd64e00640 1 -- 172.21.15.27:0/2593145665 wait complete. 2024-01-15T18:05:08.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.572+0000 7efd64e00640 1 Processor -- start 2024-01-15T18:05:08.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.572+0000 7efd64e00640 1 -- start start 2024-01-15T18:05:08.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.572+0000 7efd64e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efd400039f0 0x7efd60159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:08.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.572+0000 7efd64e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efd600b5260 0x7efd600b56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:08.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.572+0000 7efd64e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd600b3ad0 0x7efd600b3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:08.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.572+0000 7efd64e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd600b5c20 con 0x7efd600b3ad0 2024-01-15T18:05:08.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.572+0000 7efd64e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd600b4490 con 0x7efd600b5260 2024-01-15T18:05:08.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.572+0000 7efd64e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd600c49b0 con 0x7efd400039f0 2024-01-15T18:05:08.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.572+0000 7efd5e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd600b3ad0 0x7efd600b3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:08.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.572+0000 7efd5e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd600b3ad0 0x7efd600b3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59450/0 (socket says 172.21.15.27:59450) 2024-01-15T18:05:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.572+0000 7efd5e800640 1 -- 172.21.15.27:0/4193582465 learned_addr learned my addr 172.21.15.27:0/4193582465 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.572+0000 7efd5de00640 1 --2- 172.21.15.27:0/4193582465 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efd400039f0 0x7efd60159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.572+0000 7efd5d400640 1 --2- 172.21.15.27:0/4193582465 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efd600b5260 0x7efd600b56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.572+0000 7efd5e800640 1 -- 172.21.15.27:0/4193582465 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efd400039f0 msgr2=0x7efd60159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.572+0000 7efd5e800640 1 --2- 172.21.15.27:0/4193582465 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efd400039f0 0x7efd60159540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.572+0000 7efd5e800640 1 -- 172.21.15.27:0/4193582465 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efd600b5260 msgr2=0x7efd600b56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.572+0000 7efd5e800640 1 --2- 172.21.15.27:0/4193582465 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efd600b5260 0x7efd600b56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.572+0000 7efd5e800640 1 -- 172.21.15.27:0/4193582465 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd600c8590 con 0x7efd600b3ad0 2024-01-15T18:05:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.572+0000 7efd5e800640 1 --2- 172.21.15.27:0/4193582465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd600b3ad0 0x7efd600b3f50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efd600723e0 tx=0x7efd5800fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.572+0000 7efd56000640 1 -- 172.21.15.27:0/4193582465 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5800fec0 con 0x7efd600b3ad0 2024-01-15T18:05:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.572+0000 7efd56000640 1 -- 172.21.15.27:0/4193582465 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd58006d70 con 0x7efd600b3ad0 2024-01-15T18:05:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.572+0000 7efd56000640 1 -- 172.21.15.27:0/4193582465 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd58003ba0 con 0x7efd600b3ad0 2024-01-15T18:05:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.572+0000 7efd64e00640 1 -- 172.21.15.27:0/4193582465 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd600cb230 con 0x7efd600b3ad0 2024-01-15T18:05:08.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.572+0000 7efd64e00640 1 -- 172.21.15.27:0/4193582465 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd600b4990 con 0x7efd600b3ad0 2024-01-15T18:05:08.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.576+0000 7efd64e00640 1 -- 172.21.15.27:0/4193582465 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd6004e850 con 0x7efd600b3ad0 2024-01-15T18:05:08.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.576+0000 7efd56000640 1 -- 172.21.15.27:0/4193582465 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7efd58003d00 con 0x7efd600b3ad0 2024-01-15T18:05:08.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.576+0000 7efd56000640 1 --2- 172.21.15.27:0/4193582465 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efd38066920 0x7efd38068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:08.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.576+0000 7efd56000640 1 -- 172.21.15.27:0/4193582465 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd580837b0 con 0x7efd600b3ad0 2024-01-15T18:05:08.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.576+0000 7efd5de00640 1 --2- 172.21.15.27:0/4193582465 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efd38066920 0x7efd38068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:08.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.576+0000 7efd5de00640 1 --2- 172.21.15.27:0/4193582465 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efd38066920 0x7efd38068de0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efd500ab000 tx=0x7efd500b1040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:08.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.580+0000 7efd56000640 1 -- 172.21.15.27:0/4193582465 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7efd58050a90 con 0x7efd600b3ad0 2024-01-15T18:05:08.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.736+0000 7efd64e00640 1 -- 172.21.15.27:0/4193582465 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd6015fab0 con 0x7efd600b3ad0 2024-01-15T18:05:08.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.740+0000 7efd56000640 1 -- 172.21.15.27:0/4193582465 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16409 (secure 0 0 0) 0x7efd580540b0 con 0x7efd600b3ad0 2024-01-15T18:05:08.743 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:08.743 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:04:59.837052+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:04:56.220608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T18:04:57.210421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:08.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.740+0000 7efd64e00640 1 -- 172.21.15.27:0/4193582465 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efd38066920 msgr2=0x7efd38068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:08.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.740+0000 7efd64e00640 1 --2- 172.21.15.27:0/4193582465 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efd38066920 0x7efd38068de0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efd500ab000 tx=0x7efd500b1040 comp rx=0 tx=0).stop 2024-01-15T18:05:08.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.740+0000 7efd64e00640 1 -- 172.21.15.27:0/4193582465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd600b3ad0 msgr2=0x7efd600b3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:08.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.740+0000 7efd64e00640 1 --2- 172.21.15.27:0/4193582465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd600b3ad0 0x7efd600b3f50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efd600723e0 tx=0x7efd5800fbb0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.740+0000 7efd64e00640 1 -- 172.21.15.27:0/4193582465 shutdown_connections 2024-01-15T18:05:08.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.740+0000 7efd64e00640 1 --2- 172.21.15.27:0/4193582465 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efd38066920 0x7efd38068de0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.740+0000 7efd64e00640 1 --2- 172.21.15.27:0/4193582465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd600b3ad0 0x7efd600b3f50 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.740+0000 7efd64e00640 1 --2- 172.21.15.27:0/4193582465 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efd600b5260 0x7efd600b56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.740+0000 7efd64e00640 1 --2- 172.21.15.27:0/4193582465 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efd400039f0 0x7efd60159540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.740+0000 7efd64e00640 1 -- 172.21.15.27:0/4193582465 >> 172.21.15.27:0/4193582465 conn(0x7efd600b2a10 msgr2=0x7efd6006f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:08.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.740+0000 7efd64e00640 1 -- 172.21.15.27:0/4193582465 shutdown_connections 2024-01-15T18:05:08.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.740+0000 7efd64e00640 1 -- 172.21.15.27:0/4193582465 wait complete. 2024-01-15T18:05:08.765 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T18:04:56.220608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T18:04:57.210421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:05:08.765 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:05:08.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.904+0000 7fc5c2e00640 1 Processor -- start 2024-01-15T18:05:08.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.904+0000 7fc5c2e00640 1 -- start start 2024-01-15T18:05:08.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.908+0000 7fc5c2e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fc5bc0cf460 0x7fc5bc0cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:08.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.908+0000 7fc5c2e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5bc15b4b0 0x7fc5bc0cba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:08.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.908+0000 7fc5c2e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5bc0cfd80 con 0x7fc5bc0cec60 2024-01-15T18:05:08.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.908+0000 7fc5c2e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5bc0d19a0 con 0x7fc5bc0cf460 2024-01-15T18:05:08.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.908+0000 7fc5c2e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc0d1c80 con 0x7fc5bc15b4b0 2024-01-15T18:05:08.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.908+0000 7fc5c0e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc5bc0cec60 0x7fc5bc0cbf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42446/0 (socket says 172.21.15.27:42446) 2024-01-15T18:05:08.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.908+0000 7fc5c0e00640 1 -- 172.21.15.27:0/874496665 learned_addr learned my addr 172.21.15.27:0/874496665 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:08.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.908+0000 7fc5bb400640 1 --2- 172.21.15.27:0/874496665 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5bc15b4b0 0x7fc5bc0cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:08.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.908+0000 7fc5bbe00640 1 --2- 172.21.15.27:0/874496665 >> v2:172.21.15.93:3300/0 conn(0x7fc5bc0cf460 0x7fc5bc0cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:08.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.908+0000 7fc5baa00640 1 -- 172.21.15.27:0/874496665 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3254528311 0 0) 0x7fc5bc0cfd80 con 0x7fc5bc0cec60 2024-01-15T18:05:08.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.908+0000 7fc5baa00640 1 -- 172.21.15.27:0/874496665 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc59c003530 con 0x7fc5bc0cec60 2024-01-15T18:05:08.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.908+0000 7fc5bbe00640 1 -- 172.21.15.27:0/874496665 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5bc15b4b0 msgr2=0x7fc5bc0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:08.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.908+0000 7fc5bbe00640 1 --2- 172.21.15.27:0/874496665 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5bc15b4b0 0x7fc5bc0cba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.908+0000 7fc5bbe00640 1 -- 172.21.15.27:0/874496665 >> v1:172.21.15.27:6789/0 conn(0x7fc5bc0cec60 legacy=0x7fc5bc0cbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:08.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.908+0000 7fc5bbe00640 1 -- 172.21.15.27:0/874496665 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5bc0b07c0 con 0x7fc5bc0cf460 2024-01-15T18:05:08.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.908+0000 7fc5bb400640 1 --2- 172.21.15.27:0/874496665 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5bc15b4b0 0x7fc5bc0cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:08.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.912+0000 7fc5bbe00640 1 --2- 172.21.15.27:0/874496665 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc5bc0cf460 0x7fc5bc0cf840 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac0097c0 tx=0x7fc5ac01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd8006f74a97c88 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:08.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.912+0000 7fc5baa00640 1 -- 172.21.15.27:0/874496665 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac0ab680 con 0x7fc5bc0cf460 2024-01-15T18:05:08.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.912+0000 7fc5baa00640 1 -- 172.21.15.27:0/874496665 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5ac0ab7e0 con 0x7fc5bc0cf460 2024-01-15T18:05:08.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.912+0000 7fc5baa00640 1 -- 172.21.15.27:0/874496665 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac0abb50 con 0x7fc5bc0cf460 2024-01-15T18:05:08.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.912+0000 7fc5c2e00640 1 -- 172.21.15.27:0/874496665 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc5bc0cf460 msgr2=0x7fc5bc0cf840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:08.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.912+0000 7fc5c2e00640 1 --2- 172.21.15.27:0/874496665 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc5bc0cf460 0x7fc5bc0cf840 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac0097c0 tx=0x7fc5ac01c480 comp rx=0 tx=0).stop 2024-01-15T18:05:08.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.912+0000 7fc5c2e00640 1 -- 172.21.15.27:0/874496665 shutdown_connections 2024-01-15T18:05:08.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.912+0000 7fc5c2e00640 1 --2- 172.21.15.27:0/874496665 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5bc15b4b0 0x7fc5bc0cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.912+0000 7fc5c2e00640 1 --2- 172.21.15.27:0/874496665 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc5bc0cf460 0x7fc5bc0cf840 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.912+0000 7fc5c2e00640 1 -- 172.21.15.27:0/874496665 >> 172.21.15.27:0/874496665 conn(0x7fc5bc0d0e80 msgr2=0x7fc5bc0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:08.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.912+0000 7fc5c2e00640 1 -- 172.21.15.27:0/874496665 shutdown_connections 2024-01-15T18:05:08.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.912+0000 7fc5c2e00640 1 -- 172.21.15.27:0/874496665 wait complete. 2024-01-15T18:05:08.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.912+0000 7fc5c2e00640 1 Processor -- start 2024-01-15T18:05:08.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.912+0000 7fc5c2e00640 1 -- start start 2024-01-15T18:05:08.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.916+0000 7fc5c2e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5bc0cec60 0x7fc5bc14a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:08.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.916+0000 7fc5c2e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc5bc0cf460 0x7fc5bc06cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:08.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.916+0000 7fc5c2e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5bc15b4b0 0x7fc5bc152a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:08.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.916+0000 7fc5c2e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc06d680 con 0x7fc5bc15b4b0 2024-01-15T18:05:08.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.916+0000 7fc5c2e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc159900 con 0x7fc5bc0cf460 2024-01-15T18:05:08.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.916+0000 7fc5c2e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5bc14acc0 con 0x7fc5bc0cec60 2024-01-15T18:05:08.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.916+0000 7fc5c0e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5bc15b4b0 0x7fc5bc152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:08.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.916+0000 7fc5c0e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5bc15b4b0 0x7fc5bc152a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59456/0 (socket says 172.21.15.27:59456) 2024-01-15T18:05:08.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.916+0000 7fc5c0e00640 1 -- 172.21.15.27:0/3827191894 learned_addr learned my addr 172.21.15.27:0/3827191894 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:08.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.916+0000 7fc5bbe00640 1 --2- 172.21.15.27:0/3827191894 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5bc0cec60 0x7fc5bc14a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:08.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.916+0000 7fc5bb400640 1 --2- 172.21.15.27:0/3827191894 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc5bc0cf460 0x7fc5bc06cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:08.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.916+0000 7fc5c0e00640 1 -- 172.21.15.27:0/3827191894 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5bc0cec60 msgr2=0x7fc5bc14a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:08.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.916+0000 7fc5c0e00640 1 --2- 172.21.15.27:0/3827191894 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5bc0cec60 0x7fc5bc14a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.916+0000 7fc5c0e00640 1 -- 172.21.15.27:0/3827191894 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc5bc0cf460 msgr2=0x7fc5bc06cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:08.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.916+0000 7fc5c0e00640 1 --2- 172.21.15.27:0/3827191894 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc5bc0cf460 0x7fc5bc06cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.916+0000 7fc5c0e00640 1 -- 172.21.15.27:0/3827191894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a4009420 con 0x7fc5bc15b4b0 2024-01-15T18:05:08.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.916+0000 7fc5bbe00640 1 --2- 172.21.15.27:0/3827191894 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5bc0cec60 0x7fc5bc14a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:08.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.916+0000 7fc5c0e00640 1 --2- 172.21.15.27:0/3827191894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5bc15b4b0 0x7fc5bc152a00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc5b400cad0 tx=0x7fc5b400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:08.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.916+0000 7fc5b3e00640 1 -- 172.21.15.27:0/3827191894 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b400ac30 con 0x7fc5bc15b4b0 2024-01-15T18:05:08.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.916+0000 7fc5c2e00640 1 -- 172.21.15.27:0/3827191894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5ac009420 con 0x7fc5bc15b4b0 2024-01-15T18:05:08.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.916+0000 7fc5b3e00640 1 -- 172.21.15.27:0/3827191894 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b400ad90 con 0x7fc5bc15b4b0 2024-01-15T18:05:08.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.916+0000 7fc5b3e00640 1 -- 172.21.15.27:0/3827191894 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4012620 con 0x7fc5bc15b4b0 2024-01-15T18:05:08.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.916+0000 7fc5c2e00640 1 -- 172.21.15.27:0/3827191894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5bc159340 con 0x7fc5bc15b4b0 2024-01-15T18:05:08.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.916+0000 7fc5c2e00640 1 -- 172.21.15.27:0/3827191894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5bc04e850 con 0x7fc5bc15b4b0 2024-01-15T18:05:08.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.916+0000 7fc5b3e00640 1 -- 172.21.15.27:0/3827191894 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fc5b4007070 con 0x7fc5bc15b4b0 2024-01-15T18:05:08.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.920+0000 7fc5b3e00640 1 --2- 172.21.15.27:0/3827191894 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc5940668f0 0x7fc594068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:08.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.920+0000 7fc5b3e00640 1 -- 172.21.15.27:0/3827191894 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc5b40843e0 con 0x7fc5bc15b4b0 2024-01-15T18:05:08.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.920+0000 7fc5bbe00640 1 --2- 172.21.15.27:0/3827191894 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc5940668f0 0x7fc594068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:08.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.920+0000 7fc5bbe00640 1 --2- 172.21.15.27:0/3827191894 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc5940668f0 0x7fc594068db0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac0af000 tx=0x7fc5ac0a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:08.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:08.924+0000 7fc5b3e00640 1 -- 172.21.15.27:0/3827191894 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc5b40515b0 con 0x7fc5bc15b4b0 2024-01-15T18:05:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.080+0000 7fc5c2e00640 1 -- 172.21.15.27:0/3827191894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc5bc0cf240 con 0x7fc5bc15b4b0 2024-01-15T18:05:09.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.080+0000 7fc5b3e00640 1 -- 172.21.15.27:0/3827191894 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc5b4054bd0 con 0x7fc5bc15b4b0 2024-01-15T18:05:09.083 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:05:09.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.080+0000 7fc5c2e00640 1 -- 172.21.15.27:0/3827191894 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc5940668f0 msgr2=0x7fc594068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:09.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.080+0000 7fc5c2e00640 1 --2- 172.21.15.27:0/3827191894 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc5940668f0 0x7fc594068db0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac0af000 tx=0x7fc5ac0a7000 comp rx=0 tx=0).stop 2024-01-15T18:05:09.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.080+0000 7fc5c2e00640 1 -- 172.21.15.27:0/3827191894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5bc15b4b0 msgr2=0x7fc5bc152a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:09.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.080+0000 7fc5c2e00640 1 --2- 172.21.15.27:0/3827191894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5bc15b4b0 0x7fc5bc152a00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc5b400cad0 tx=0x7fc5b400cf90 comp rx=0 tx=0).stop 2024-01-15T18:05:09.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.080+0000 7fc5c2e00640 1 -- 172.21.15.27:0/3827191894 shutdown_connections 2024-01-15T18:05:09.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.080+0000 7fc5c2e00640 1 --2- 172.21.15.27:0/3827191894 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc5940668f0 0x7fc594068db0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.080+0000 7fc5c2e00640 1 --2- 172.21.15.27:0/3827191894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5bc15b4b0 0x7fc5bc152a00 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.080+0000 7fc5c2e00640 1 --2- 172.21.15.27:0/3827191894 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc5bc0cf460 0x7fc5bc06cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.080+0000 7fc5c2e00640 1 --2- 172.21.15.27:0/3827191894 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5bc0cec60 0x7fc5bc14a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.080+0000 7fc5c2e00640 1 -- 172.21.15.27:0/3827191894 >> 172.21.15.27:0/3827191894 conn(0x7fc5bc0d0e80 msgr2=0x7fc5bc0adb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:09.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.080+0000 7fc5c2e00640 1 -- 172.21.15.27:0/3827191894 shutdown_connections 2024-01-15T18:05:09.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.080+0000 7fc5c2e00640 1 -- 172.21.15.27:0/3827191894 wait complete. 2024-01-15T18:05:09.105 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T18:05:09.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.244+0000 7f5c1ea00640 1 Processor -- start 2024-01-15T18:05:09.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.244+0000 7f5c1ea00640 1 -- start start 2024-01-15T18:05:09.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.244+0000 7f5c1ea00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c1815d680 0x7f5c180cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:09.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.244+0000 7f5c1ea00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c1806c750 con 0x7f5c180cfc30 2024-01-15T18:05:09.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.244+0000 7f5c1ea00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c180cb4d0 con 0x7f5c180ccf80 2024-01-15T18:05:09.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.244+0000 7f5c1ea00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c180d0120 con 0x7f5c1815d680 2024-01-15T18:05:09.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.248+0000 7f5c17400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5c180cfc30 0x7f5c1806b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42450/0 (socket says 172.21.15.27:42450) 2024-01-15T18:05:09.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.248+0000 7f5c17400640 1 -- 172.21.15.27:0/614712857 learned_addr learned my addr 172.21.15.27:0/614712857 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:09.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.248+0000 7f5c17e00640 1 --2- 172.21.15.27:0/614712857 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c1815d680 0x7f5c180cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:09.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.248+0000 7f5c16a00640 1 -- 172.21.15.27:0/614712857 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4064251291 0 0) 0x7f5c1806c750 con 0x7f5c180cfc30 2024-01-15T18:05:09.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.248+0000 7f5c16a00640 1 -- 172.21.15.27:0/614712857 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bf8003530 con 0x7f5c180cfc30 2024-01-15T18:05:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.248+0000 7f5c16a00640 1 -- 172.21.15.27:0/614712857 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2805416657 0 0) 0x7f5c180cb4d0 con 0x7f5c180ccf80 2024-01-15T18:05:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.248+0000 7f5c16a00640 1 -- 172.21.15.27:0/614712857 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c1806c750 con 0x7f5c180ccf80 2024-01-15T18:05:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.248+0000 7f5c17e00640 1 -- 172.21.15.27:0/614712857 >> v1:172.21.15.93:6789/0 conn(0x7f5c180ccf80 legacy=0x7f5c1806c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.248+0000 7f5c17e00640 1 -- 172.21.15.27:0/614712857 >> v1:172.21.15.27:6789/0 conn(0x7f5c180cfc30 legacy=0x7f5c1806b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:09.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.248+0000 7f5c17e00640 1 -- 172.21.15.27:0/614712857 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c180d0400 con 0x7f5c1815d680 2024-01-15T18:05:09.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.248+0000 7f5c17e00640 1 --2- 172.21.15.27:0/614712857 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c1815d680 0x7f5c180cd700 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5c080097f0 tx=0x7f5c0801c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8ec1d4b27409324 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:09.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.248+0000 7f5c16a00640 1 -- 172.21.15.27:0/614712857 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c080aa5f0 con 0x7f5c1815d680 2024-01-15T18:05:09.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.248+0000 7f5c16a00640 1 -- 172.21.15.27:0/614712857 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c080aa750 con 0x7f5c1815d680 2024-01-15T18:05:09.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.252+0000 7f5c16a00640 1 -- 172.21.15.27:0/614712857 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c080aaac0 con 0x7f5c1815d680 2024-01-15T18:05:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.252+0000 7f5c1ea00640 1 -- 172.21.15.27:0/614712857 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c1815d680 msgr2=0x7f5c180cd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.252+0000 7f5c1ea00640 1 --2- 172.21.15.27:0/614712857 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c1815d680 0x7f5c180cd700 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5c080097f0 tx=0x7f5c0801c490 comp rx=0 tx=0).stop 2024-01-15T18:05:09.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.252+0000 7f5c1ea00640 1 -- 172.21.15.27:0/614712857 shutdown_connections 2024-01-15T18:05:09.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.252+0000 7f5c1ea00640 1 --2- 172.21.15.27:0/614712857 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c1815d680 0x7f5c180cd700 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.252+0000 7f5c1ea00640 1 -- 172.21.15.27:0/614712857 >> 172.21.15.27:0/614712857 conn(0x7f5c180d0e80 msgr2=0x7f5c180d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:09.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.252+0000 7f5c1ea00640 1 -- 172.21.15.27:0/614712857 shutdown_connections 2024-01-15T18:05:09.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.252+0000 7f5c1ea00640 1 -- 172.21.15.27:0/614712857 wait complete. 2024-01-15T18:05:09.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.252+0000 7f5c1ea00640 1 Processor -- start 2024-01-15T18:05:09.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.252+0000 7f5c1ea00640 1 -- start start 2024-01-15T18:05:09.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.252+0000 7f5c1ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c180ccf80 0x7f5c18156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:09.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.252+0000 7f5c1ea00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5c180cfc30 0x7f5c18154390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:09.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.252+0000 7f5c1ea00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c1815d680 0x7f5c181f7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:09.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.252+0000 7f5c1ea00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c180adc10 con 0x7f5c180ccf80 2024-01-15T18:05:09.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.252+0000 7f5c1ea00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c180adef0 con 0x7f5c180cfc30 2024-01-15T18:05:09.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.252+0000 7f5c1ea00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c180ae1d0 con 0x7f5c1815d680 2024-01-15T18:05:09.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.256+0000 7f5c17e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c180ccf80 0x7f5c18156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:09.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.256+0000 7f5c17e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c180ccf80 0x7f5c18156380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59458/0 (socket says 172.21.15.27:59458) 2024-01-15T18:05:09.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.256+0000 7f5c17e00640 1 -- 172.21.15.27:0/3849235335 learned_addr learned my addr 172.21.15.27:0/3849235335 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:09.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.256+0000 7f5c1ca00640 1 --2- 172.21.15.27:0/3849235335 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c1815d680 0x7f5c181f7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:09.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.256+0000 7f5c17400640 1 --2- 172.21.15.27:0/3849235335 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5c180cfc30 0x7f5c18154390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:09.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.256+0000 7f5c17e00640 1 -- 172.21.15.27:0/3849235335 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c1815d680 msgr2=0x7f5c181f7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:09.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.256+0000 7f5c17e00640 1 --2- 172.21.15.27:0/3849235335 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c1815d680 0x7f5c181f7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.256+0000 7f5c17e00640 1 -- 172.21.15.27:0/3849235335 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5c180cfc30 msgr2=0x7f5c18154390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:09.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.256+0000 7f5c17e00640 1 --2- 172.21.15.27:0/3849235335 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5c180cfc30 0x7f5c18154390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.256+0000 7f5c17e00640 1 -- 172.21.15.27:0/3849235335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c08009420 con 0x7f5c180ccf80 2024-01-15T18:05:09.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.256+0000 7f5c17400640 1 --2- 172.21.15.27:0/3849235335 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5c180cfc30 0x7f5c18154390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:09.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.256+0000 7f5c17e00640 1 --2- 172.21.15.27:0/3849235335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c180ccf80 0x7f5c18156380 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5c0801a0c0 tx=0x7f5c08019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:09.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.256+0000 7f5c0fe00640 1 -- 172.21.15.27:0/3849235335 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c080aacd0 con 0x7f5c180ccf80 2024-01-15T18:05:09.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.256+0000 7f5c0fe00640 1 -- 172.21.15.27:0/3849235335 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c080aae30 con 0x7f5c180ccf80 2024-01-15T18:05:09.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.256+0000 7f5c0fe00640 1 -- 172.21.15.27:0/3849235335 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c080bd610 con 0x7f5c180ccf80 2024-01-15T18:05:09.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.256+0000 7f5c1ea00640 1 -- 172.21.15.27:0/3849235335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c180ae4b0 con 0x7f5c180ccf80 2024-01-15T18:05:09.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.256+0000 7f5c1ea00640 1 -- 172.21.15.27:0/3849235335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c181568c0 con 0x7f5c180ccf80 2024-01-15T18:05:09.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.256+0000 7f5c1ea00640 1 -- 172.21.15.27:0/3849235335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c180ae310 con 0x7f5c180ccf80 2024-01-15T18:05:09.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.256+0000 7f5c0fe00640 1 -- 172.21.15.27:0/3849235335 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f5c080bc3a0 con 0x7f5c180ccf80 2024-01-15T18:05:09.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.260+0000 7f5c0fe00640 1 --2- 172.21.15.27:0/3849235335 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5bf00668f0 0x7f5bf0068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:09.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.260+0000 7f5c0fe00640 1 -- 172.21.15.27:0/3849235335 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c080a7070 con 0x7f5c180ccf80 2024-01-15T18:05:09.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.260+0000 7f5c17400640 1 --2- 172.21.15.27:0/3849235335 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5bf00668f0 0x7f5bf0068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:09.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.260+0000 7f5c17400640 1 --2- 172.21.15.27:0/3849235335 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5bf00668f0 0x7f5bf0068db0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5c1814ba30 tx=0x7f5c00009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:09.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.264+0000 7f5c0fe00640 1 -- 172.21.15.27:0/3849235335 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5c180ae310 con 0x7f5c180ccf80 2024-01-15T18:05:09.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.420+0000 7f5c1ea00640 1 -- 172.21.15.27:0/3849235335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5c180ae310 con 0x7f5c180ccf80 2024-01-15T18:05:09.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.420+0000 7f5c0fe00640 1 -- 172.21.15.27:0/3849235335 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5c180ae310 con 0x7f5c180ccf80 2024-01-15T18:05:09.425 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-15T18:05:09.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.424+0000 7f5c1ea00640 1 -- 172.21.15.27:0/3849235335 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5bf00668f0 msgr2=0x7f5bf0068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:09.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.424+0000 7f5c1ea00640 1 --2- 172.21.15.27:0/3849235335 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5bf00668f0 0x7f5bf0068db0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5c1814ba30 tx=0x7f5c00009040 comp rx=0 tx=0).stop 2024-01-15T18:05:09.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.424+0000 7f5c1ea00640 1 -- 172.21.15.27:0/3849235335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c180ccf80 msgr2=0x7f5c18156380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:09.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.424+0000 7f5c1ea00640 1 --2- 172.21.15.27:0/3849235335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c180ccf80 0x7f5c18156380 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5c0801a0c0 tx=0x7f5c08019230 comp rx=0 tx=0).stop 2024-01-15T18:05:09.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.424+0000 7f5c1ea00640 1 -- 172.21.15.27:0/3849235335 shutdown_connections 2024-01-15T18:05:09.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.424+0000 7f5c1ea00640 1 --2- 172.21.15.27:0/3849235335 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5bf00668f0 0x7f5bf0068db0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.424+0000 7f5c1ea00640 1 --2- 172.21.15.27:0/3849235335 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c1815d680 0x7f5c181f7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.424+0000 7f5c1ea00640 1 --2- 172.21.15.27:0/3849235335 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5c180cfc30 0x7f5c18154390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.424+0000 7f5c1ea00640 1 --2- 172.21.15.27:0/3849235335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c180ccf80 0x7f5c18156380 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.424+0000 7f5c1ea00640 1 -- 172.21.15.27:0/3849235335 >> 172.21.15.27:0/3849235335 conn(0x7f5c180d0e80 msgr2=0x7f5c1806e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:09.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.424+0000 7f5c1ea00640 1 -- 172.21.15.27:0/3849235335 shutdown_connections 2024-01-15T18:05:09.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.424+0000 7f5c1ea00640 1 -- 172.21.15.27:0/3849235335 wait complete. 2024-01-15T18:05:09.447 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T18:05:09.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.588+0000 7f5a48400640 1 Processor -- start 2024-01-15T18:05:09.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.588+0000 7f5a48400640 1 -- start start 2024-01-15T18:05:09.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.588+0000 7f5a48400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f5a4015f550 0x7f5a400cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:09.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.588+0000 7f5a48400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a400b1cb0 con 0x7f5a400cf110 2024-01-15T18:05:09.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.588+0000 7f5a48400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4015fa00 con 0x7f5a40158900 2024-01-15T18:05:09.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.588+0000 7f5a48400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a400cf580 con 0x7f5a4015f550 2024-01-15T18:05:09.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.588+0000 7f5a46400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5a400cf110 0x7f5a400701e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42458/0 (socket says 172.21.15.27:42458) 2024-01-15T18:05:09.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.588+0000 7f5a46400640 1 -- 172.21.15.27:0/2983372464 learned_addr learned my addr 172.21.15.27:0/2983372464 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:09.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.588+0000 7f5a45000640 1 --2- 172.21.15.27:0/2983372464 >> v2:172.21.15.93:3300/0 conn(0x7f5a4015f550 0x7f5a400cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:09.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.588+0000 7f5a3fe00640 1 -- 172.21.15.27:0/2983372464 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1730253743 0 0) 0x7f5a400b1cb0 con 0x7f5a400cf110 2024-01-15T18:05:09.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.588+0000 7f5a3fe00640 1 -- 172.21.15.27:0/2983372464 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a20003530 con 0x7f5a400cf110 2024-01-15T18:05:09.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.588+0000 7f5a3fe00640 1 -- 172.21.15.27:0/2983372464 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2087861321 0 0) 0x7f5a20003530 con 0x7f5a400cf110 2024-01-15T18:05:09.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.588+0000 7f5a3fe00640 1 -- 172.21.15.27:0/2983372464 >> v2:172.21.15.93:3300/0 conn(0x7f5a4015f550 msgr2=0x7f5a400cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:09.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.588+0000 7f5a3fe00640 1 --2- 172.21.15.27:0/2983372464 >> v2:172.21.15.93:3300/0 conn(0x7f5a4015f550 0x7f5a400cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.588+0000 7f5a3fe00640 1 -- 172.21.15.27:0/2983372464 >> v1:172.21.15.93:6789/0 conn(0x7f5a40158900 legacy=0x7f5a4015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:09.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.588+0000 7f5a3fe00640 1 -- 172.21.15.27:0/2983372464 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a400b1f00 con 0x7f5a400cf110 2024-01-15T18:05:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.588+0000 7f5a3fe00640 1 -- 172.21.15.27:0/2983372464 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f5a38001a00 con 0x7f5a400cf110 2024-01-15T18:05:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.588+0000 7f5a3fe00640 1 -- 172.21.15.27:0/2983372464 >> v1:172.21.15.27:6789/0 conn(0x7f5a400cf110 legacy=0x7f5a400701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.588+0000 7f5a3fe00640 1 --2- 172.21.15.27:0/2983372464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a200039f0 0x7f5a20003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.588+0000 7f5a3fe00640 1 --2- 172.21.15.27:0/2983372464 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a200047d0 0x7f5a20004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.588+0000 7f5a3fe00640 1 --2- 172.21.15.27:0/2983372464 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a20005190 0x7f5a2000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.588+0000 7f5a3fe00640 1 -- 172.21.15.27:0/2983372464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a400cf580 con 0x7f5a200039f0 2024-01-15T18:05:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.588+0000 7f5a3fe00640 1 -- 172.21.15.27:0/2983372464 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2000c0c0 con 0x7f5a20005190 2024-01-15T18:05:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.588+0000 7f5a3fe00640 1 -- 172.21.15.27:0/2983372464 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2000c280 con 0x7f5a200047d0 2024-01-15T18:05:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.588+0000 7f5a45a00640 1 --2- 172.21.15.27:0/2983372464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a200039f0 0x7f5a20003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:09.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.592+0000 7f5a46400640 1 --2- 172.21.15.27:0/2983372464 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a20005190 0x7f5a2000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:09.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.592+0000 7f5a45000640 1 --2- 172.21.15.27:0/2983372464 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a200047d0 0x7f5a20004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:09.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.592+0000 7f5a45a00640 1 -- 172.21.15.27:0/2983372464 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a200047d0 msgr2=0x7f5a20004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:09.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.592+0000 7f5a45a00640 1 --2- 172.21.15.27:0/2983372464 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a200047d0 0x7f5a20004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.592+0000 7f5a45a00640 1 -- 172.21.15.27:0/2983372464 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a20005190 msgr2=0x7f5a2000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:09.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.592+0000 7f5a45a00640 1 --2- 172.21.15.27:0/2983372464 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a20005190 0x7f5a2000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.592+0000 7f5a45a00640 1 -- 172.21.15.27:0/2983372464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a20003530 con 0x7f5a200039f0 2024-01-15T18:05:09.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.592+0000 7f5a46400640 1 -- 172.21.15.27:0/2983372464 reap_dead start 2024-01-15T18:05:09.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.592+0000 7f5a45a00640 1 --2- 172.21.15.27:0/2983372464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a200039f0 0x7f5a20003e10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5a30001780 tx=0x7f5a3001c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7988985287700441 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:09.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.592+0000 7f5a3fe00640 1 -- 172.21.15.27:0/2983372464 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a300052d0 con 0x7f5a200039f0 2024-01-15T18:05:09.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.592+0000 7f5a3fe00640 1 -- 172.21.15.27:0/2983372464 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a300a7050 con 0x7f5a200039f0 2024-01-15T18:05:09.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.592+0000 7f5a3fe00640 1 -- 172.21.15.27:0/2983372464 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a30003600 con 0x7f5a200039f0 2024-01-15T18:05:09.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.596+0000 7f5a48400640 1 -- 172.21.15.27:0/2983372464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a200039f0 msgr2=0x7f5a20003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:09.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.596+0000 7f5a48400640 1 --2- 172.21.15.27:0/2983372464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a200039f0 0x7f5a20003e10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5a30001780 tx=0x7f5a3001c3b0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.596+0000 7f5a48400640 1 -- 172.21.15.27:0/2983372464 shutdown_connections 2024-01-15T18:05:09.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.596+0000 7f5a48400640 1 --2- 172.21.15.27:0/2983372464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a200039f0 0x7f5a20003e10 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.596+0000 7f5a48400640 1 -- 172.21.15.27:0/2983372464 >> 172.21.15.27:0/2983372464 conn(0x7f5a400cb700 msgr2=0x7f5a400cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:09.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.596+0000 7f5a48400640 1 -- 172.21.15.27:0/2983372464 shutdown_connections 2024-01-15T18:05:09.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.596+0000 7f5a48400640 1 -- 172.21.15.27:0/2983372464 wait complete. 2024-01-15T18:05:09.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.596+0000 7f5a48400640 1 Processor -- start 2024-01-15T18:05:09.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.596+0000 7f5a48400640 1 -- start start 2024-01-15T18:05:09.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.596+0000 7f5a48400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a200039f0 0x7f5a40141450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:09.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.596+0000 7f5a48400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a40151310 0x7f5a40151770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:09.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.596+0000 7f5a48400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a400cc810 0x7f5a400ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:09.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.596+0000 7f5a48400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a40151cb0 con 0x7f5a400cc810 2024-01-15T18:05:09.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.596+0000 7f5a48400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a400cd540 con 0x7f5a40151310 2024-01-15T18:05:09.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.596+0000 7f5a48400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a400cd820 con 0x7f5a200039f0 2024-01-15T18:05:09.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.596+0000 7f5a46400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a400cc810 0x7f5a400ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:09.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.596+0000 7f5a46400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a400cc810 0x7f5a400ccc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59486/0 (socket says 172.21.15.27:59486) 2024-01-15T18:05:09.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.596+0000 7f5a46400640 1 -- 172.21.15.27:0/2957728217 learned_addr learned my addr 172.21.15.27:0/2957728217 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:09.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.596+0000 7f5a45a00640 1 --2- 172.21.15.27:0/2957728217 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a200039f0 0x7f5a40141450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:09.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.596+0000 7f5a45000640 1 --2- 172.21.15.27:0/2957728217 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a40151310 0x7f5a40151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:09.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.600+0000 7f5a46400640 1 -- 172.21.15.27:0/2957728217 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a200039f0 msgr2=0x7f5a40141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:09.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.600+0000 7f5a46400640 1 --2- 172.21.15.27:0/2957728217 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a200039f0 0x7f5a40141450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.600+0000 7f5a46400640 1 -- 172.21.15.27:0/2957728217 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a40151310 msgr2=0x7f5a40151770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:09.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.600+0000 7f5a46400640 1 --2- 172.21.15.27:0/2957728217 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a40151310 0x7f5a40151770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.600+0000 7f5a46400640 1 -- 172.21.15.27:0/2957728217 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a4015c490 con 0x7f5a400cc810 2024-01-15T18:05:09.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.600+0000 7f5a46400640 1 --2- 172.21.15.27:0/2957728217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a400cc810 0x7f5a400ccc90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5a38002020 tx=0x7f5a38007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:09.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.600+0000 7f5a3d600640 1 -- 172.21.15.27:0/2957728217 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3800ab10 con 0x7f5a400cc810 2024-01-15T18:05:09.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.600+0000 7f5a3d600640 1 -- 172.21.15.27:0/2957728217 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3800ac90 con 0x7f5a400cc810 2024-01-15T18:05:09.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.600+0000 7f5a3d600640 1 -- 172.21.15.27:0/2957728217 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a380133f0 con 0x7f5a400cc810 2024-01-15T18:05:09.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.600+0000 7f5a48400640 1 -- 172.21.15.27:0/2957728217 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a400cdb00 con 0x7f5a400cc810 2024-01-15T18:05:09.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.600+0000 7f5a48400640 1 -- 172.21.15.27:0/2957728217 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a400ce0c0 con 0x7f5a400cc810 2024-01-15T18:05:09.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.600+0000 7f5a48400640 1 -- 172.21.15.27:0/2957728217 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a4004e850 con 0x7f5a400cc810 2024-01-15T18:05:09.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.600+0000 7f5a3d600640 1 -- 172.21.15.27:0/2957728217 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f5a38013550 con 0x7f5a400cc810 2024-01-15T18:05:09.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.604+0000 7f5a3d600640 1 --2- 172.21.15.27:0/2957728217 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5a180668f0 0x7f5a18068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:09.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.604+0000 7f5a3d600640 1 -- 172.21.15.27:0/2957728217 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5a380849f0 con 0x7f5a400cc810 2024-01-15T18:05:09.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.604+0000 7f5a45a00640 1 --2- 172.21.15.27:0/2957728217 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5a180668f0 0x7f5a18068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:09.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.604+0000 7f5a45a00640 1 --2- 172.21.15.27:0/2957728217 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5a180668f0 0x7f5a18068db0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5a30001750 tx=0x7f5a300011a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.608+0000 7f5a3d600640 1 -- 172.21.15.27:0/2957728217 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5a38051bc0 con 0x7f5a400cc810 2024-01-15T18:05:09.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.764+0000 7f5a48400640 1 -- 172.21.15.27:0/2957728217 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5a4006d540 con 0x7f5a400cc810 2024-01-15T18:05:09.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.764+0000 7f5a3d600640 1 -- 172.21.15.27:0/2957728217 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5a380551e0 con 0x7f5a400cc810 2024-01-15T18:05:09.768 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-15T18:05:09.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.764+0000 7f5a48400640 1 -- 172.21.15.27:0/2957728217 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5a180668f0 msgr2=0x7f5a18068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:09.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.764+0000 7f5a48400640 1 --2- 172.21.15.27:0/2957728217 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5a180668f0 0x7f5a18068db0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5a30001750 tx=0x7f5a300011a0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.764+0000 7f5a48400640 1 -- 172.21.15.27:0/2957728217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a400cc810 msgr2=0x7f5a400ccc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:09.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.764+0000 7f5a48400640 1 --2- 172.21.15.27:0/2957728217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a400cc810 0x7f5a400ccc90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5a38002020 tx=0x7f5a38007b20 comp rx=0 tx=0).stop 2024-01-15T18:05:09.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.764+0000 7f5a48400640 1 -- 172.21.15.27:0/2957728217 shutdown_connections 2024-01-15T18:05:09.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.764+0000 7f5a48400640 1 --2- 172.21.15.27:0/2957728217 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5a180668f0 0x7f5a18068db0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.764+0000 7f5a48400640 1 --2- 172.21.15.27:0/2957728217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a400cc810 0x7f5a400ccc90 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.764+0000 7f5a48400640 1 --2- 172.21.15.27:0/2957728217 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a40151310 0x7f5a40151770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.764+0000 7f5a48400640 1 --2- 172.21.15.27:0/2957728217 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a200039f0 0x7f5a40141450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.764+0000 7f5a48400640 1 -- 172.21.15.27:0/2957728217 >> 172.21.15.27:0/2957728217 conn(0x7f5a400cb700 msgr2=0x7f5a40150910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:09.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.764+0000 7f5a48400640 1 -- 172.21.15.27:0/2957728217 shutdown_connections 2024-01-15T18:05:09.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.764+0000 7f5a48400640 1 -- 172.21.15.27:0/2957728217 wait complete. 2024-01-15T18:05:09.790 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:05:09.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.928+0000 7f4b42c00640 1 Processor -- start 2024-01-15T18:05:09.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.928+0000 7f4b42c00640 1 -- start start 2024-01-15T18:05:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.928+0000 7f4b42c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f4b3c15e7d0 0x7f4b3c0cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.928+0000 7f4b42c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4b3c0cedc0 0x7f4b3c0cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.928+0000 7f4b42c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4b3c0ce2e0 0x7f4b3c0cba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.928+0000 7f4b42c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b3c0b38b0 con 0x7f4b3c0cedc0 2024-01-15T18:05:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.928+0000 7f4b42c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b3c0cd550 con 0x7f4b3c15e7d0 2024-01-15T18:05:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.928+0000 7f4b42c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c0cd270 con 0x7f4b3c0ce2e0 2024-01-15T18:05:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.928+0000 7f4b3b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4b3c0cedc0 0x7f4b3c0cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.928+0000 7f4b3b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4b3c0cedc0 0x7f4b3c0cf7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59494/0 (socket says 172.21.15.27:59494) 2024-01-15T18:05:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.928+0000 7f4b3b400640 1 -- 172.21.15.27:0/3557846253 learned_addr learned my addr 172.21.15.27:0/3557846253 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:09.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.928+0000 7f4b3be00640 1 --2- 172.21.15.27:0/3557846253 >> v2:172.21.15.93:3300/0 conn(0x7f4b3c15e7d0 0x7f4b3c0cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:09.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.928+0000 7f4b40c00640 1 --2- 172.21.15.27:0/3557846253 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4b3c0ce2e0 0x7f4b3c0cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:09.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.928+0000 7f4b3b400640 1 -- 172.21.15.27:0/3557846253 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4b3c0ce2e0 msgr2=0x7f4b3c0cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:09.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.928+0000 7f4b3b400640 1 --2- 172.21.15.27:0/3557846253 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4b3c0ce2e0 0x7f4b3c0cba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.928+0000 7f4b3b400640 1 -- 172.21.15.27:0/3557846253 >> v2:172.21.15.93:3300/0 conn(0x7f4b3c15e7d0 msgr2=0x7f4b3c0cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:09.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.928+0000 7f4b3b400640 1 --2- 172.21.15.27:0/3557846253 >> v2:172.21.15.93:3300/0 conn(0x7f4b3c15e7d0 0x7f4b3c0cf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.928+0000 7f4b3b400640 1 -- 172.21.15.27:0/3557846253 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b3c0b35d0 con 0x7f4b3c0cedc0 2024-01-15T18:05:09.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.928+0000 7f4b40c00640 1 --2- 172.21.15.27:0/3557846253 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4b3c0ce2e0 0x7f4b3c0cba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:09.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.928+0000 7f4b3be00640 1 --2- 172.21.15.27:0/3557846253 >> v2:172.21.15.93:3300/0 conn(0x7f4b3c15e7d0 0x7f4b3c0cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:09.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.932+0000 7f4b3b400640 1 --2- 172.21.15.27:0/3557846253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b3c0cedc0 0x7f4b3c0cf7e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c071b00 tx=0x7f4b2401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcf2941cbfed7cfb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:09.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.932+0000 7f4b3aa00640 1 -- 172.21.15.27:0/3557846253 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b240ac410 con 0x7f4b3c0cedc0 2024-01-15T18:05:09.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.932+0000 7f4b3aa00640 1 -- 172.21.15.27:0/3557846253 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b240ac570 con 0x7f4b3c0cedc0 2024-01-15T18:05:09.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.932+0000 7f4b3aa00640 1 -- 172.21.15.27:0/3557846253 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b240ac900 con 0x7f4b3c0cedc0 2024-01-15T18:05:09.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.932+0000 7f4b42c00640 1 -- 172.21.15.27:0/3557846253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b3c0cedc0 msgr2=0x7f4b3c0cf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:09.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.932+0000 7f4b42c00640 1 --2- 172.21.15.27:0/3557846253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b3c0cedc0 0x7f4b3c0cf7e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c071b00 tx=0x7f4b2401c500 comp rx=0 tx=0).stop 2024-01-15T18:05:09.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.932+0000 7f4b42c00640 1 -- 172.21.15.27:0/3557846253 shutdown_connections 2024-01-15T18:05:09.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.932+0000 7f4b42c00640 1 --2- 172.21.15.27:0/3557846253 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4b3c0ce2e0 0x7f4b3c0cba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.932+0000 7f4b42c00640 1 --2- 172.21.15.27:0/3557846253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b3c0cedc0 0x7f4b3c0cf7e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.932+0000 7f4b42c00640 1 --2- 172.21.15.27:0/3557846253 >> v2:172.21.15.93:3300/0 conn(0x7f4b3c15e7d0 0x7f4b3c0cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.932+0000 7f4b42c00640 1 -- 172.21.15.27:0/3557846253 >> 172.21.15.27:0/3557846253 conn(0x7f4b3c0b2a10 msgr2=0x7f4b3c0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:09.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.932+0000 7f4b42c00640 1 -- 172.21.15.27:0/3557846253 shutdown_connections 2024-01-15T18:05:09.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.932+0000 7f4b42c00640 1 -- 172.21.15.27:0/3557846253 wait complete. 2024-01-15T18:05:09.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.936+0000 7f4b42c00640 1 Processor -- start 2024-01-15T18:05:09.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.936+0000 7f4b42c00640 1 -- start start 2024-01-15T18:05:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.936+0000 7f4b42c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4b3c0ce2e0 0x7f4b3c0b6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.936+0000 7f4b42c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4b3c0cedc0 0x7f4b3c14e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.936+0000 7f4b42c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b3c15e7d0 0x7f4b3c163270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.936+0000 7f4b42c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c0b4540 con 0x7f4b3c15e7d0 2024-01-15T18:05:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.936+0000 7f4b42c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c158d70 con 0x7f4b3c0cedc0 2024-01-15T18:05:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.936+0000 7f4b42c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c0b3c30 con 0x7f4b3c0ce2e0 2024-01-15T18:05:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.936+0000 7f4b40c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b3c15e7d0 0x7f4b3c163270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.936+0000 7f4b40c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b3c15e7d0 0x7f4b3c163270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59508/0 (socket says 172.21.15.27:59508) 2024-01-15T18:05:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.936+0000 7f4b40c00640 1 -- 172.21.15.27:0/62748295 learned_addr learned my addr 172.21.15.27:0/62748295 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:09.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.936+0000 7f4b3be00640 1 --2- 172.21.15.27:0/62748295 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4b3c0ce2e0 0x7f4b3c0b6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:09.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.936+0000 7f4b3b400640 1 --2- 172.21.15.27:0/62748295 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4b3c0cedc0 0x7f4b3c14e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:09.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.936+0000 7f4b40c00640 1 -- 172.21.15.27:0/62748295 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4b3c0ce2e0 msgr2=0x7f4b3c0b6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:09.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.936+0000 7f4b40c00640 1 --2- 172.21.15.27:0/62748295 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4b3c0ce2e0 0x7f4b3c0b6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.936+0000 7f4b40c00640 1 -- 172.21.15.27:0/62748295 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4b3c0cedc0 msgr2=0x7f4b3c14e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:09.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.936+0000 7f4b40c00640 1 --2- 172.21.15.27:0/62748295 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4b3c0cedc0 0x7f4b3c14e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:09.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.936+0000 7f4b40c00640 1 -- 172.21.15.27:0/62748295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b240094d0 con 0x7f4b3c15e7d0 2024-01-15T18:05:09.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.936+0000 7f4b40c00640 1 --2- 172.21.15.27:0/62748295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b3c15e7d0 0x7f4b3c163270 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4b34007e10 tx=0x7f4b3400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:09.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.936+0000 7f4b33e00640 1 -- 172.21.15.27:0/62748295 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b34003ba0 con 0x7f4b3c15e7d0 2024-01-15T18:05:09.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.936+0000 7f4b33e00640 1 -- 172.21.15.27:0/62748295 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b3400f040 con 0x7f4b3c15e7d0 2024-01-15T18:05:09.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.936+0000 7f4b42c00640 1 -- 172.21.15.27:0/62748295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b3c0c78b0 con 0x7f4b3c15e7d0 2024-01-15T18:05:09.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.936+0000 7f4b33e00640 1 -- 172.21.15.27:0/62748295 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3400de80 con 0x7f4b3c15e7d0 2024-01-15T18:05:09.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.936+0000 7f4b42c00640 1 -- 172.21.15.27:0/62748295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b3c0b4160 con 0x7f4b3c15e7d0 2024-01-15T18:05:09.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.940+0000 7f4b42c00640 1 -- 172.21.15.27:0/62748295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b3c0b4680 con 0x7f4b3c15e7d0 2024-01-15T18:05:09.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.940+0000 7f4b33e00640 1 -- 172.21.15.27:0/62748295 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f4b34004030 con 0x7f4b3c15e7d0 2024-01-15T18:05:09.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.940+0000 7f4b33e00640 1 --2- 172.21.15.27:0/62748295 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4b14066920 0x7f4b14068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:09.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.940+0000 7f4b33e00640 1 -- 172.21.15.27:0/62748295 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4b34084660 con 0x7f4b3c15e7d0 2024-01-15T18:05:09.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.940+0000 7f4b3be00640 1 --2- 172.21.15.27:0/62748295 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4b14066920 0x7f4b14068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:09.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.940+0000 7f4b3be00640 1 --2- 172.21.15.27:0/62748295 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4b14066920 0x7f4b14068de0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c00ba10 tx=0x7f4b2c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:09.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:09.944+0000 7f4b33e00640 1 -- 172.21.15.27:0/62748295 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4b3c0b4680 con 0x7f4b3c15e7d0 2024-01-15T18:05:10.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.136+0000 7f4b42c00640 1 -- 172.21.15.27:0/62748295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b3c0b4680 con 0x7f4b3c15e7d0 2024-01-15T18:05:10.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.136+0000 7f4b33e00640 1 -- 172.21.15.27:0/62748295 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7f4b3c0b4680 con 0x7f4b3c15e7d0 2024-01-15T18:05:10.140 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:10.140 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/393360880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":393360880},{"type":"v1","addr":"172.21.15.93:6836","nonce":393360880}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4229,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4063308188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4063308188},{"type":"v1","addr":"172.21.15.27:6835","nonce":4063308188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4305,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/2343128056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":2343128056},{"type":"v1","addr":"172.21.15.93:6837","nonce":2343128056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:04:58.208165+0000","modified":"2024-01-15T18:05:01.225419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2657553192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2657553192},{"type":"v1","addr":"172.21.15.27:6834","nonce":2657553192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T18:05:10.140 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 8 2024-01-15T18:05:10.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.136+0000 7f4b42c00640 1 -- 172.21.15.27:0/62748295 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4b14066920 msgr2=0x7f4b14068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:10.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.136+0000 7f4b42c00640 1 --2- 172.21.15.27:0/62748295 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4b14066920 0x7f4b14068de0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c00ba10 tx=0x7f4b2c008040 comp rx=0 tx=0).stop 2024-01-15T18:05:10.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.136+0000 7f4b42c00640 1 -- 172.21.15.27:0/62748295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b3c15e7d0 msgr2=0x7f4b3c163270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:10.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.136+0000 7f4b42c00640 1 --2- 172.21.15.27:0/62748295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b3c15e7d0 0x7f4b3c163270 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4b34007e10 tx=0x7f4b3400d4a0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.136+0000 7f4b42c00640 1 -- 172.21.15.27:0/62748295 shutdown_connections 2024-01-15T18:05:10.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.136+0000 7f4b42c00640 1 --2- 172.21.15.27:0/62748295 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4b14066920 0x7f4b14068de0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.136+0000 7f4b42c00640 1 --2- 172.21.15.27:0/62748295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b3c15e7d0 0x7f4b3c163270 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.136+0000 7f4b42c00640 1 --2- 172.21.15.27:0/62748295 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4b3c0cedc0 0x7f4b3c14e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.136+0000 7f4b42c00640 1 --2- 172.21.15.27:0/62748295 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4b3c0ce2e0 0x7f4b3c0b6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.136+0000 7f4b42c00640 1 -- 172.21.15.27:0/62748295 >> 172.21.15.27:0/62748295 conn(0x7f4b3c0b2a10 msgr2=0x7f4b3c06e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:10.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.136+0000 7f4b42c00640 1 -- 172.21.15.27:0/62748295 shutdown_connections 2024-01-15T18:05:10.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.136+0000 7f4b42c00640 1 -- 172.21.15.27:0/62748295 wait complete. 2024-01-15T18:05:10.162 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:10.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.300+0000 7f505f000640 1 Processor -- start 2024-01-15T18:05:10.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.300+0000 7f505f000640 1 -- start start 2024-01-15T18:05:10.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.300+0000 7f505f000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f50580b2390 0x7f50580cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.300+0000 7f505f000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f505806bb80 0x7f50580cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.300+0000 7f505f000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f50580cfcc0 0x7f50580cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.300+0000 7f505f000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50580b2770 con 0x7f50580cfcc0 2024-01-15T18:05:10.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.300+0000 7f505f000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f505806c060 con 0x7f505806bb80 2024-01-15T18:05:10.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.300+0000 7f505f000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f505806c340 con 0x7f50580b2390 2024-01-15T18:05:10.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.300+0000 7f505d000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f50580cfcc0 0x7f50580cddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.300+0000 7f505d000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f50580cfcc0 0x7f50580cddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59512/0 (socket says 172.21.15.27:59512) 2024-01-15T18:05:10.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.300+0000 7f505d000640 1 -- 172.21.15.27:0/235299449 learned_addr learned my addr 172.21.15.27:0/235299449 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:10.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.300+0000 7f5057400640 1 --2- 172.21.15.27:0/235299449 >> v2:172.21.15.93:3300/0 conn(0x7f505806bb80 0x7f50580cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.300+0000 7f5057e00640 1 --2- 172.21.15.27:0/235299449 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f50580b2390 0x7f50580cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.300+0000 7f505d000640 1 -- 172.21.15.27:0/235299449 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f50580b2390 msgr2=0x7f50580cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:10.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.300+0000 7f505d000640 1 --2- 172.21.15.27:0/235299449 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f50580b2390 0x7f50580cf780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.300+0000 7f505d000640 1 -- 172.21.15.27:0/235299449 >> v2:172.21.15.93:3300/0 conn(0x7f505806bb80 msgr2=0x7f50580cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:10.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.300+0000 7f505d000640 1 --2- 172.21.15.27:0/235299449 >> v2:172.21.15.93:3300/0 conn(0x7f505806bb80 0x7f50580cd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.300+0000 7f505d000640 1 -- 172.21.15.27:0/235299449 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505806c620 con 0x7f50580cfcc0 2024-01-15T18:05:10.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.304+0000 7f505d000640 1 --2- 172.21.15.27:0/235299449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50580cfcc0 0x7f50580cddc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f505806dd50 tx=0x7f505001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69ed1673f1078e6e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:10.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.304+0000 7f5056a00640 1 -- 172.21.15.27:0/235299449 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50500a5820 con 0x7f50580cfcc0 2024-01-15T18:05:10.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.304+0000 7f5056a00640 1 -- 172.21.15.27:0/235299449 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50500a5980 con 0x7f50580cfcc0 2024-01-15T18:05:10.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.304+0000 7f5056a00640 1 -- 172.21.15.27:0/235299449 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50500a5d10 con 0x7f50580cfcc0 2024-01-15T18:05:10.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.304+0000 7f505f000640 1 -- 172.21.15.27:0/235299449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50580cfcc0 msgr2=0x7f50580cddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:10.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.304+0000 7f505f000640 1 --2- 172.21.15.27:0/235299449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50580cfcc0 0x7f50580cddc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f505806dd50 tx=0x7f505001c9a0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.304+0000 7f505f000640 1 -- 172.21.15.27:0/235299449 shutdown_connections 2024-01-15T18:05:10.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.304+0000 7f505f000640 1 --2- 172.21.15.27:0/235299449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50580cfcc0 0x7f50580cddc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.304+0000 7f505f000640 1 --2- 172.21.15.27:0/235299449 >> v2:172.21.15.93:3300/0 conn(0x7f505806bb80 0x7f50580cd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.304+0000 7f505f000640 1 --2- 172.21.15.27:0/235299449 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f50580b2390 0x7f50580cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.304+0000 7f505f000640 1 -- 172.21.15.27:0/235299449 >> 172.21.15.27:0/235299449 conn(0x7f50580b2a10 msgr2=0x7f50580b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:10.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.304+0000 7f505f000640 1 -- 172.21.15.27:0/235299449 shutdown_connections 2024-01-15T18:05:10.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.304+0000 7f505f000640 1 -- 172.21.15.27:0/235299449 wait complete. 2024-01-15T18:05:10.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.308+0000 7f505f000640 1 Processor -- start 2024-01-15T18:05:10.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.308+0000 7f505f000640 1 -- start start 2024-01-15T18:05:10.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.308+0000 7f505f000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f505806bb80 0x7f50580b6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.308+0000 7f505f000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f50580b2390 0x7f505815a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.308+0000 7f505f000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f50580cfcc0 0x7f50580ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.308+0000 7f505f000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50580c7360 con 0x7f505806bb80 2024-01-15T18:05:10.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.308+0000 7f505f000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505809ee10 con 0x7f50580b2390 2024-01-15T18:05:10.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.308+0000 7f505f000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50580c7a90 con 0x7f50580cfcc0 2024-01-15T18:05:10.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.308+0000 7f5057e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f505806bb80 0x7f50580b6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.308+0000 7f5057e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f505806bb80 0x7f50580b6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59516/0 (socket says 172.21.15.27:59516) 2024-01-15T18:05:10.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.308+0000 7f5057e00640 1 -- 172.21.15.27:0/1880268759 learned_addr learned my addr 172.21.15.27:0/1880268759 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:10.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.308+0000 7f5057400640 1 --2- 172.21.15.27:0/1880268759 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f50580b2390 0x7f505815a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.308+0000 7f505d000640 1 --2- 172.21.15.27:0/1880268759 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f50580cfcc0 0x7f50580ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.308+0000 7f5057e00640 1 -- 172.21.15.27:0/1880268759 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f50580cfcc0 msgr2=0x7f50580ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:10.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.308+0000 7f5057e00640 1 --2- 172.21.15.27:0/1880268759 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f50580cfcc0 0x7f50580ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.308+0000 7f5057e00640 1 -- 172.21.15.27:0/1880268759 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f50580b2390 msgr2=0x7f505815a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:10.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.308+0000 7f5057e00640 1 --2- 172.21.15.27:0/1880268759 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f50580b2390 0x7f505815a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.308+0000 7f5057e00640 1 -- 172.21.15.27:0/1880268759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5050009b30 con 0x7f505806bb80 2024-01-15T18:05:10.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.308+0000 7f5057e00640 1 --2- 172.21.15.27:0/1880268759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f505806bb80 0x7f50580b6b10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f504800e8a0 tx=0x7f504800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:10.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.308+0000 7f504fe00640 1 -- 172.21.15.27:0/1880268759 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5048009510 con 0x7f505806bb80 2024-01-15T18:05:10.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.308+0000 7f504fe00640 1 -- 172.21.15.27:0/1880268759 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f504800c940 con 0x7f505806bb80 2024-01-15T18:05:10.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.308+0000 7f504fe00640 1 -- 172.21.15.27:0/1880268759 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504800ccb0 con 0x7f505806bb80 2024-01-15T18:05:10.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.308+0000 7f505f000640 1 -- 172.21.15.27:0/1880268759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50580b3b00 con 0x7f505806bb80 2024-01-15T18:05:10.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.308+0000 7f505f000640 1 -- 172.21.15.27:0/1880268759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50580b40c0 con 0x7f505806bb80 2024-01-15T18:05:10.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.312+0000 7f505f000640 1 -- 172.21.15.27:0/1880268759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50580d07c0 con 0x7f505806bb80 2024-01-15T18:05:10.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.312+0000 7f504fe00640 1 -- 172.21.15.27:0/1880268759 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f5048013470 con 0x7f505806bb80 2024-01-15T18:05:10.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.312+0000 7f504fe00640 1 --2- 172.21.15.27:0/1880268759 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f50300668f0 0x7f5030068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.312+0000 7f504fe00640 1 -- 172.21.15.27:0/1880268759 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5048084440 con 0x7f505806bb80 2024-01-15T18:05:10.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.312+0000 7f5057400640 1 --2- 172.21.15.27:0/1880268759 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f50300668f0 0x7f5030068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.316+0000 7f5057400640 1 --2- 172.21.15.27:0/1880268759 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f50300668f0 0x7f5030068db0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f50580c7ca0 tx=0x7f5040009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:10.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.316+0000 7f504fe00640 1 -- 172.21.15.27:0/1880268759 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5048051720 con 0x7f505806bb80 2024-01-15T18:05:10.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.476+0000 7f505f000640 1 -- 172.21.15.27:0/1880268759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50580b4250 con 0x7f505806bb80 2024-01-15T18:05:10.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.476+0000 7f504fe00640 1 -- 172.21.15.27:0/1880268759 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16409 (secure 0 0 0) 0x7f5048054d40 con 0x7f505806bb80 2024-01-15T18:05:10.482 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:10.482 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:04:59.837052+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:04:56.220608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T18:04:57.210421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:10.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.480+0000 7f505f000640 1 -- 172.21.15.27:0/1880268759 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f50300668f0 msgr2=0x7f5030068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:10.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.480+0000 7f505f000640 1 --2- 172.21.15.27:0/1880268759 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f50300668f0 0x7f5030068db0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f50580c7ca0 tx=0x7f5040009190 comp rx=0 tx=0).stop 2024-01-15T18:05:10.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.480+0000 7f505f000640 1 -- 172.21.15.27:0/1880268759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f505806bb80 msgr2=0x7f50580b6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:10.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.480+0000 7f505f000640 1 --2- 172.21.15.27:0/1880268759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f505806bb80 0x7f50580b6b10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f504800e8a0 tx=0x7f504800eca0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.480+0000 7f505f000640 1 -- 172.21.15.27:0/1880268759 shutdown_connections 2024-01-15T18:05:10.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.480+0000 7f505f000640 1 --2- 172.21.15.27:0/1880268759 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f50300668f0 0x7f5030068db0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.480+0000 7f505f000640 1 --2- 172.21.15.27:0/1880268759 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f50580cfcc0 0x7f50580ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.480+0000 7f505f000640 1 --2- 172.21.15.27:0/1880268759 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f50580b2390 0x7f505815a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.480+0000 7f505f000640 1 --2- 172.21.15.27:0/1880268759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f505806bb80 0x7f50580b6b10 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.480+0000 7f505f000640 1 -- 172.21.15.27:0/1880268759 >> 172.21.15.27:0/1880268759 conn(0x7f50580b2a10 msgr2=0x7f50580b1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:10.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.480+0000 7f505f000640 1 -- 172.21.15.27:0/1880268759 shutdown_connections 2024-01-15T18:05:10.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.480+0000 7f505f000640 1 -- 172.21.15.27:0/1880268759 wait complete. 2024-01-15T18:05:10.505 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T18:04:56.220608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T18:04:57.210421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:05:10.505 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:05:10.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.644+0000 7f748b400640 1 Processor -- start 2024-01-15T18:05:10.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.644+0000 7f748b400640 1 -- start start 2024-01-15T18:05:10.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f748b400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f748415d600 0x7f7484161110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f748b400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f748415d9e0 con 0x7f74840d1ab0 2024-01-15T18:05:10.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f748b400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f748406b9b0 con 0x7f74840b2020 2024-01-15T18:05:10.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f748b400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f748406bc90 con 0x7f748415d600 2024-01-15T18:05:10.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7483e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f74840d1ab0 0x7f7484161650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42462/0 (socket says 172.21.15.27:42462) 2024-01-15T18:05:10.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7483e00640 1 -- 172.21.15.27:0/2390663805 learned_addr learned my addr 172.21.15.27:0/2390663805 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:10.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7483400640 1 -- 172.21.15.27:0/2390663805 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109410226 0 0) 0x7f748415d9e0 con 0x7f74840d1ab0 2024-01-15T18:05:10.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7488a00640 1 --2- 172.21.15.27:0/2390663805 >> v2:172.21.15.93:3300/0 conn(0x7f748415d600 0x7f7484161110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7483400640 1 -- 172.21.15.27:0/2390663805 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7464003530 con 0x7f74840d1ab0 2024-01-15T18:05:10.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7483400640 1 -- 172.21.15.27:0/2390663805 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3932386105 0 0) 0x7f7464003530 con 0x7f74840d1ab0 2024-01-15T18:05:10.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7483400640 1 -- 172.21.15.27:0/2390663805 >> v2:172.21.15.93:3300/0 conn(0x7f748415d600 msgr2=0x7f7484161110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:10.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7483400640 1 --2- 172.21.15.27:0/2390663805 >> v2:172.21.15.93:3300/0 conn(0x7f748415d600 0x7f7484161110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7483400640 1 -- 172.21.15.27:0/2390663805 >> v1:172.21.15.93:6789/0 conn(0x7f74840b2020 legacy=0x7f74840b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:10.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7483400640 1 -- 172.21.15.27:0/2390663805 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74840b3ea0 con 0x7f74840d1ab0 2024-01-15T18:05:10.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7483400640 1 -- 172.21.15.27:0/2390663805 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f746c001680 con 0x7f74840d1ab0 2024-01-15T18:05:10.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7483400640 1 -- 172.21.15.27:0/2390663805 >> v1:172.21.15.27:6789/0 conn(0x7f74840d1ab0 legacy=0x7f7484161650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:10.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7483400640 1 --2- 172.21.15.27:0/2390663805 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7464003a40 0x7f7464003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7483400640 1 --2- 172.21.15.27:0/2390663805 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f74640047d0 0x7f7464004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7483400640 1 --2- 172.21.15.27:0/2390663805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74640051e0 0x7f746400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7483400640 1 -- 172.21.15.27:0/2390663805 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748406bc90 con 0x7f74640051e0 2024-01-15T18:05:10.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7483400640 1 -- 172.21.15.27:0/2390663805 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746400c110 con 0x7f74640047d0 2024-01-15T18:05:10.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7483400640 1 -- 172.21.15.27:0/2390663805 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f746400c2d0 con 0x7f7464003a40 2024-01-15T18:05:10.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7489400640 1 --2- 172.21.15.27:0/2390663805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74640051e0 0x7f746400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7488a00640 1 --2- 172.21.15.27:0/2390663805 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7464003a40 0x7f7464003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7483e00640 1 --2- 172.21.15.27:0/2390663805 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f74640047d0 0x7f7464004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7489400640 1 -- 172.21.15.27:0/2390663805 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7464003a40 msgr2=0x7f7464003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:10.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7489400640 1 --2- 172.21.15.27:0/2390663805 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7464003a40 0x7f7464003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7489400640 1 -- 172.21.15.27:0/2390663805 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f74640047d0 msgr2=0x7f7464004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:10.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7489400640 1 --2- 172.21.15.27:0/2390663805 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f74640047d0 0x7f7464004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7489400640 1 -- 172.21.15.27:0/2390663805 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7464003530 con 0x7f74640051e0 2024-01-15T18:05:10.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.648+0000 7f7488a00640 1 --2- 172.21.15.27:0/2390663805 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7464003a40 0x7f7464003e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:10.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.652+0000 7f7489400640 1 -- 172.21.15.27:0/2390663805 reap_dead start 2024-01-15T18:05:10.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.652+0000 7f7489400640 1 --2- 172.21.15.27:0/2390663805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74640051e0 0x7f746400b990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f747c001e70 tx=0x7f747c01c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60e40d3034ad0b81 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:10.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.652+0000 7f7483400640 1 -- 172.21.15.27:0/2390663805 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c003b80 con 0x7f74640051e0 2024-01-15T18:05:10.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.652+0000 7f7483400640 1 -- 172.21.15.27:0/2390663805 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f747c003d20 con 0x7f74640051e0 2024-01-15T18:05:10.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.652+0000 7f7483400640 1 -- 172.21.15.27:0/2390663805 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c005260 con 0x7f74640051e0 2024-01-15T18:05:10.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.652+0000 7f748b400640 1 -- 172.21.15.27:0/2390663805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74640051e0 msgr2=0x7f746400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:10.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.652+0000 7f748b400640 1 --2- 172.21.15.27:0/2390663805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74640051e0 0x7f746400b990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f747c001e70 tx=0x7f747c01c760 comp rx=0 tx=0).stop 2024-01-15T18:05:10.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.652+0000 7f748b400640 1 -- 172.21.15.27:0/2390663805 shutdown_connections 2024-01-15T18:05:10.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.652+0000 7f748b400640 1 --2- 172.21.15.27:0/2390663805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74640051e0 0x7f746400b990 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.652+0000 7f748b400640 1 -- 172.21.15.27:0/2390663805 >> 172.21.15.27:0/2390663805 conn(0x7f74840d0eb0 msgr2=0x7f74840d12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:10.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.652+0000 7f748b400640 1 -- 172.21.15.27:0/2390663805 shutdown_connections 2024-01-15T18:05:10.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.652+0000 7f748b400640 1 -- 172.21.15.27:0/2390663805 wait complete. 2024-01-15T18:05:10.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.652+0000 7f748b400640 1 Processor -- start 2024-01-15T18:05:10.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.652+0000 7f748b400640 1 -- start start 2024-01-15T18:05:10.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.656+0000 7f748b400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f74640051e0 0x7f748415ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.656+0000 7f748b400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f74840c64e0 0x7f74840c6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.656+0000 7f748b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74841549e0 0x7f7484154e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.656+0000 7f748b400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74840c6ea0 con 0x7f74841549e0 2024-01-15T18:05:10.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.656+0000 7f748b400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748415ddc0 con 0x7f74640051e0 2024-01-15T18:05:10.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.656+0000 7f748b400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f748415e0a0 con 0x7f74840c64e0 2024-01-15T18:05:10.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.656+0000 7f7489400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74841549e0 0x7f7484154e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.656+0000 7f7489400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74841549e0 0x7f7484154e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59528/0 (socket says 172.21.15.27:59528) 2024-01-15T18:05:10.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.656+0000 7f7489400640 1 -- 172.21.15.27:0/2846211573 learned_addr learned my addr 172.21.15.27:0/2846211573 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:10.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.656+0000 7f7488a00640 1 --2- 172.21.15.27:0/2846211573 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f74640051e0 0x7f748415ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.656+0000 7f7483e00640 1 --2- 172.21.15.27:0/2846211573 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f74840c64e0 0x7f74840c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.656+0000 7f7489400640 1 -- 172.21.15.27:0/2846211573 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f74840c64e0 msgr2=0x7f74840c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:10.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.656+0000 7f7489400640 1 --2- 172.21.15.27:0/2846211573 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f74840c64e0 0x7f74840c6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.656+0000 7f7489400640 1 -- 172.21.15.27:0/2846211573 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f74640051e0 msgr2=0x7f748415ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:10.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.656+0000 7f7489400640 1 --2- 172.21.15.27:0/2846211573 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f74640051e0 0x7f748415ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.656+0000 7f7489400640 1 -- 172.21.15.27:0/2846211573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f748415e380 con 0x7f74841549e0 2024-01-15T18:05:10.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.656+0000 7f7488a00640 1 --2- 172.21.15.27:0/2846211573 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f74640051e0 0x7f748415ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:10.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.656+0000 7f7489400640 1 --2- 172.21.15.27:0/2846211573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74841549e0 0x7f7484154e90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f747c01cc40 tx=0x7f7464005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:10.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.656+0000 7f7480c00640 1 -- 172.21.15.27:0/2846211573 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c0ac070 con 0x7f74841549e0 2024-01-15T18:05:10.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.656+0000 7f7480c00640 1 -- 172.21.15.27:0/2846211573 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f747c003e90 con 0x7f74841549e0 2024-01-15T18:05:10.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.656+0000 7f748b400640 1 -- 172.21.15.27:0/2846211573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f748415e660 con 0x7f74841549e0 2024-01-15T18:05:10.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.656+0000 7f7480c00640 1 -- 172.21.15.27:0/2846211573 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c006d60 con 0x7f74841549e0 2024-01-15T18:05:10.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.656+0000 7f748b400640 1 -- 172.21.15.27:0/2846211573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74841553d0 con 0x7f74841549e0 2024-01-15T18:05:10.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.656+0000 7f748b400640 1 -- 172.21.15.27:0/2846211573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f748404e850 con 0x7f74841549e0 2024-01-15T18:05:10.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.660+0000 7f7480c00640 1 -- 172.21.15.27:0/2846211573 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f747c007030 con 0x7f74841549e0 2024-01-15T18:05:10.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.660+0000 7f7480c00640 1 --2- 172.21.15.27:0/2846211573 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f745c066920 0x7f745c068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.660+0000 7f7480c00640 1 -- 172.21.15.27:0/2846211573 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f747c11d130 con 0x7f74841549e0 2024-01-15T18:05:10.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.660+0000 7f7488a00640 1 --2- 172.21.15.27:0/2846211573 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f745c066920 0x7f745c068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.660+0000 7f7488a00640 1 --2- 172.21.15.27:0/2846211573 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f745c066920 0x7f745c068de0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7474009b60 tx=0x7f7474008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.664+0000 7f7480c00640 1 -- 172.21.15.27:0/2846211573 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f747c0ea2d0 con 0x7f74841549e0 2024-01-15T18:05:10.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.820+0000 7f748b400640 1 -- 172.21.15.27:0/2846211573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f748406f590 con 0x7f74841549e0 2024-01-15T18:05:10.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.820+0000 7f7480c00640 1 -- 172.21.15.27:0/2846211573 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f747c0ed8f0 con 0x7f74841549e0 2024-01-15T18:05:10.825 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:05:10.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.824+0000 7f748b400640 1 -- 172.21.15.27:0/2846211573 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f745c066920 msgr2=0x7f745c068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:10.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.824+0000 7f748b400640 1 --2- 172.21.15.27:0/2846211573 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f745c066920 0x7f745c068de0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7474009b60 tx=0x7f7474008040 comp rx=0 tx=0).stop 2024-01-15T18:05:10.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.824+0000 7f748b400640 1 -- 172.21.15.27:0/2846211573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74841549e0 msgr2=0x7f7484154e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:10.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.824+0000 7f748b400640 1 --2- 172.21.15.27:0/2846211573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74841549e0 0x7f7484154e90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f747c01cc40 tx=0x7f7464005800 comp rx=0 tx=0).stop 2024-01-15T18:05:10.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.824+0000 7f748b400640 1 -- 172.21.15.27:0/2846211573 shutdown_connections 2024-01-15T18:05:10.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.824+0000 7f748b400640 1 --2- 172.21.15.27:0/2846211573 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f745c066920 0x7f745c068de0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.824+0000 7f748b400640 1 --2- 172.21.15.27:0/2846211573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74841549e0 0x7f7484154e90 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.824+0000 7f748b400640 1 --2- 172.21.15.27:0/2846211573 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f74840c64e0 0x7f74840c6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.824+0000 7f748b400640 1 --2- 172.21.15.27:0/2846211573 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f74640051e0 0x7f748415ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.824+0000 7f748b400640 1 -- 172.21.15.27:0/2846211573 >> 172.21.15.27:0/2846211573 conn(0x7f74840d0eb0 msgr2=0x7f748406c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:10.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.824+0000 7f748b400640 1 -- 172.21.15.27:0/2846211573 shutdown_connections 2024-01-15T18:05:10.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.824+0000 7f748b400640 1 -- 172.21.15.27:0/2846211573 wait complete. 2024-01-15T18:05:10.847 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T18:05:10.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.988+0000 7fdf8dc00640 1 Processor -- start 2024-01-15T18:05:10.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.988+0000 7fdf8dc00640 1 -- start start 2024-01-15T18:05:10.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.992+0000 7fdf8dc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdf8815d330 0x7fdf880b0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.992+0000 7fdf8dc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdf880cec30 0x7fdf880cf430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.992+0000 7fdf8dc00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf880cc550 con 0x7fdf880cf970 2024-01-15T18:05:10.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.992+0000 7fdf8dc00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf880cfe00 con 0x7fdf880cec30 2024-01-15T18:05:10.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.992+0000 7fdf8dc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8806e430 con 0x7fdf8815d330 2024-01-15T18:05:10.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.992+0000 7fdf87400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdf880cec30 0x7fdf880cf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.992+0000 7fdf87400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdf880cec30 0x7fdf880cf430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59538/0 (socket says 172.21.15.27:59538) 2024-01-15T18:05:10.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.992+0000 7fdf87400640 1 -- 172.21.15.27:0/2819157185 learned_addr learned my addr 172.21.15.27:0/2819157185 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:10.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.992+0000 7fdf87400640 1 -- 172.21.15.27:0/2819157185 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdf8815d330 msgr2=0x7fdf880b0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:10.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.992+0000 7fdf87e00640 1 --2- 172.21.15.27:0/2819157185 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdf8815d330 0x7fdf880b0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.992+0000 7fdf87400640 1 --2- 172.21.15.27:0/2819157185 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdf8815d330 0x7fdf880b0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.992+0000 7fdf87400640 1 -- 172.21.15.27:0/2819157185 >> v1:172.21.15.93:6789/0 conn(0x7fdf880cf970 legacy=0x7fdf880cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:10.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.992+0000 7fdf87400640 1 -- 172.21.15.27:0/2819157185 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf8815d710 con 0x7fdf880cec30 2024-01-15T18:05:10.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.992+0000 7fdf87e00640 1 --2- 172.21.15.27:0/2819157185 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdf8815d330 0x7fdf880b0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:10.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.996+0000 7fdf87400640 1 --2- 172.21.15.27:0/2819157185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf880cec30 0x7fdf880cf430 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdf70009870 tx=0x7fdf7001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c2d48da71621fd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:10.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.996+0000 7fdf86a00640 1 -- 172.21.15.27:0/2819157185 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf700aa5f0 con 0x7fdf880cec30 2024-01-15T18:05:10.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.996+0000 7fdf86a00640 1 -- 172.21.15.27:0/2819157185 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf700aa750 con 0x7fdf880cec30 2024-01-15T18:05:10.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.996+0000 7fdf86a00640 1 -- 172.21.15.27:0/2819157185 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf700aaae0 con 0x7fdf880cec30 2024-01-15T18:05:11.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.996+0000 7fdf8dc00640 1 -- 172.21.15.27:0/2819157185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf880cec30 msgr2=0x7fdf880cf430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:11.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.996+0000 7fdf8dc00640 1 --2- 172.21.15.27:0/2819157185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf880cec30 0x7fdf880cf430 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdf70009870 tx=0x7fdf7001c4d0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.996+0000 7fdf8dc00640 1 -- 172.21.15.27:0/2819157185 shutdown_connections 2024-01-15T18:05:11.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.996+0000 7fdf8dc00640 1 --2- 172.21.15.27:0/2819157185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf880cec30 0x7fdf880cf430 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.996+0000 7fdf8dc00640 1 --2- 172.21.15.27:0/2819157185 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdf8815d330 0x7fdf880b0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.996+0000 7fdf8dc00640 1 -- 172.21.15.27:0/2819157185 >> 172.21.15.27:0/2819157185 conn(0x7fdf880d0e50 msgr2=0x7fdf880d1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:11.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.996+0000 7fdf8dc00640 1 -- 172.21.15.27:0/2819157185 shutdown_connections 2024-01-15T18:05:11.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.996+0000 7fdf8dc00640 1 -- 172.21.15.27:0/2819157185 wait complete. 2024-01-15T18:05:11.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.996+0000 7fdf8dc00640 1 Processor -- start 2024-01-15T18:05:11.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:10.996+0000 7fdf8dc00640 1 -- start start 2024-01-15T18:05:11.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.000+0000 7fdf8dc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdf880cec30 0x7fdf88152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:11.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.000+0000 7fdf8dc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf880cf970 0x7fdf8815ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:11.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.000+0000 7fdf8dc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdf8815d330 0x7fdf880c4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:11.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.000+0000 7fdf8dc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8815a9d0 con 0x7fdf880cf970 2024-01-15T18:05:11.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.000+0000 7fdf8dc00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8815acb0 con 0x7fdf880cec30 2024-01-15T18:05:11.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.000+0000 7fdf8dc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8815af90 con 0x7fdf8815d330 2024-01-15T18:05:11.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.000+0000 7fdf87400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf880cf970 0x7fdf8815ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:11.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.000+0000 7fdf87400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf880cf970 0x7fdf8815ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59548/0 (socket says 172.21.15.27:59548) 2024-01-15T18:05:11.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.000+0000 7fdf87400640 1 -- 172.21.15.27:0/3819749257 learned_addr learned my addr 172.21.15.27:0/3819749257 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:11.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.000+0000 7fdf87e00640 1 --2- 172.21.15.27:0/3819749257 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdf880cec30 0x7fdf88152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:11.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.000+0000 7fdf8d200640 1 --2- 172.21.15.27:0/3819749257 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdf8815d330 0x7fdf880c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:11.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.000+0000 7fdf87400640 1 -- 172.21.15.27:0/3819749257 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdf8815d330 msgr2=0x7fdf880c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:11.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.000+0000 7fdf87400640 1 --2- 172.21.15.27:0/3819749257 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdf8815d330 0x7fdf880c4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.000+0000 7fdf87400640 1 -- 172.21.15.27:0/3819749257 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdf880cec30 msgr2=0x7fdf88152220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:11.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.000+0000 7fdf87400640 1 --2- 172.21.15.27:0/3819749257 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdf880cec30 0x7fdf88152220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.000+0000 7fdf87400640 1 -- 172.21.15.27:0/3819749257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf700094d0 con 0x7fdf880cf970 2024-01-15T18:05:11.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.000+0000 7fdf8d200640 1 --2- 172.21.15.27:0/3819749257 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdf8815d330 0x7fdf880c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:11.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.000+0000 7fdf87400640 1 --2- 172.21.15.27:0/3819749257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf880cf970 0x7fdf8815ba70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fdf700ac040 tx=0x7fdf7001ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:11.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.000+0000 7fdf7fe00640 1 -- 172.21.15.27:0/3819749257 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf700b3800 con 0x7fdf880cf970 2024-01-15T18:05:11.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.000+0000 7fdf7fe00640 1 -- 172.21.15.27:0/3819749257 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf700b3960 con 0x7fdf880cf970 2024-01-15T18:05:11.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.000+0000 7fdf7fe00640 1 -- 172.21.15.27:0/3819749257 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf700b3d10 con 0x7fdf880cf970 2024-01-15T18:05:11.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.000+0000 7fdf8dc00640 1 -- 172.21.15.27:0/3819749257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf8815b270 con 0x7fdf880cf970 2024-01-15T18:05:11.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.000+0000 7fdf8dc00640 1 -- 172.21.15.27:0/3819749257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf88152760 con 0x7fdf880cf970 2024-01-15T18:05:11.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.000+0000 7fdf8dc00640 1 -- 172.21.15.27:0/3819749257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf8804e850 con 0x7fdf880cf970 2024-01-15T18:05:11.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.000+0000 7fdf7fe00640 1 -- 172.21.15.27:0/3819749257 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fdf700b2380 con 0x7fdf880cf970 2024-01-15T18:05:11.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.004+0000 7fdf7fe00640 1 --2- 172.21.15.27:0/3819749257 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdf60066920 0x7fdf60068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:11.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.004+0000 7fdf7fe00640 1 -- 172.21.15.27:0/3819749257 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdf700a7070 con 0x7fdf880cf970 2024-01-15T18:05:11.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.004+0000 7fdf87e00640 1 --2- 172.21.15.27:0/3819749257 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdf60066920 0x7fdf60068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:11.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.004+0000 7fdf87e00640 1 --2- 172.21.15.27:0/3819749257 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdf60066920 0x7fdf60068de0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdf880b6e80 tx=0x7fdf78008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:11.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.008+0000 7fdf7fe00640 1 -- 172.21.15.27:0/3819749257 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fdf700eb940 con 0x7fdf880cf970 2024-01-15T18:05:11.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.168+0000 7fdf8dc00640 1 -- 172.21.15.27:0/3819749257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdf8815b0d0 con 0x7fdf880cf970 2024-01-15T18:05:11.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.168+0000 7fdf7fe00640 1 -- 172.21.15.27:0/3819749257 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdf8815b0d0 con 0x7fdf880cf970 2024-01-15T18:05:11.173 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-15T18:05:11.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.172+0000 7fdf8dc00640 1 -- 172.21.15.27:0/3819749257 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdf60066920 msgr2=0x7fdf60068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:11.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.172+0000 7fdf8dc00640 1 --2- 172.21.15.27:0/3819749257 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdf60066920 0x7fdf60068de0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdf880b6e80 tx=0x7fdf78008040 comp rx=0 tx=0).stop 2024-01-15T18:05:11.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.172+0000 7fdf8dc00640 1 -- 172.21.15.27:0/3819749257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf880cf970 msgr2=0x7fdf8815ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:11.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.172+0000 7fdf8dc00640 1 --2- 172.21.15.27:0/3819749257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf880cf970 0x7fdf8815ba70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fdf700ac040 tx=0x7fdf7001ad10 comp rx=0 tx=0).stop 2024-01-15T18:05:11.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.172+0000 7fdf8dc00640 1 -- 172.21.15.27:0/3819749257 shutdown_connections 2024-01-15T18:05:11.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.172+0000 7fdf8dc00640 1 --2- 172.21.15.27:0/3819749257 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdf60066920 0x7fdf60068de0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.172+0000 7fdf8dc00640 1 --2- 172.21.15.27:0/3819749257 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdf8815d330 0x7fdf880c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.172+0000 7fdf8dc00640 1 --2- 172.21.15.27:0/3819749257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf880cf970 0x7fdf8815ba70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.172+0000 7fdf8dc00640 1 --2- 172.21.15.27:0/3819749257 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdf880cec30 0x7fdf88152220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.172+0000 7fdf8dc00640 1 -- 172.21.15.27:0/3819749257 >> 172.21.15.27:0/3819749257 conn(0x7fdf880d0e50 msgr2=0x7fdf880b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:11.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.172+0000 7fdf8dc00640 1 -- 172.21.15.27:0/3819749257 shutdown_connections 2024-01-15T18:05:11.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.172+0000 7fdf8dc00640 1 -- 172.21.15.27:0/3819749257 wait complete. 2024-01-15T18:05:11.195 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T18:05:11.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.336+0000 7f56cb000640 1 Processor -- start 2024-01-15T18:05:11.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.336+0000 7f56cb000640 1 -- start start 2024-01-15T18:05:11.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.336+0000 7f56cb000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56c40cf450 0x7f56c40d1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:11.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.336+0000 7f56cb000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f56c40cba60 0x7f56c40cf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:11.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.336+0000 7f56cb000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c40cefa0 con 0x7f56c415ee30 2024-01-15T18:05:11.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.336+0000 7f56cb000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56c40ce240 con 0x7f56c40cba60 2024-01-15T18:05:11.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.336+0000 7f56cb000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c40cfd70 con 0x7f56c40cf450 2024-01-15T18:05:11.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.336+0000 7f56c9000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f56c415ee30 0x7f56c415f2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42474/0 (socket says 172.21.15.27:42474) 2024-01-15T18:05:11.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.336+0000 7f56c9000640 1 -- 172.21.15.27:0/429598773 learned_addr learned my addr 172.21.15.27:0/429598773 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:11.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.336+0000 7f56c3400640 1 --2- 172.21.15.27:0/429598773 >> v2:172.21.15.93:3300/0 conn(0x7f56c40cba60 0x7f56c40cf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:11.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.336+0000 7f56c3e00640 1 --2- 172.21.15.27:0/429598773 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56c40cf450 0x7f56c40d1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:11.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.336+0000 7f56c2a00640 1 -- 172.21.15.27:0/429598773 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713150393 0 0) 0x7f56c40cefa0 con 0x7f56c415ee30 2024-01-15T18:05:11.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.336+0000 7f56c2a00640 1 -- 172.21.15.27:0/429598773 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56a4003530 con 0x7f56c415ee30 2024-01-15T18:05:11.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.336+0000 7f56c3400640 1 -- 172.21.15.27:0/429598773 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56c40cf450 msgr2=0x7f56c40d1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:11.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.336+0000 7f56c3400640 1 --2- 172.21.15.27:0/429598773 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56c40cf450 0x7f56c40d1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.336+0000 7f56c3400640 1 -- 172.21.15.27:0/429598773 >> v1:172.21.15.27:6789/0 conn(0x7f56c415ee30 legacy=0x7f56c415f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:11.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.336+0000 7f56c3400640 1 -- 172.21.15.27:0/429598773 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c40ccd70 con 0x7f56c40cba60 2024-01-15T18:05:11.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.340+0000 7f56c3400640 1 --2- 172.21.15.27:0/429598773 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56c40cba60 0x7f56c40cf830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0097c0 tx=0x7f56ac01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48e044b153fbbf83 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:11.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.340+0000 7f56c2a00640 1 -- 172.21.15.27:0/429598773 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac0ab600 con 0x7f56c40cba60 2024-01-15T18:05:11.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.340+0000 7f56c2a00640 1 -- 172.21.15.27:0/429598773 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56ac0a7050 con 0x7f56c40cba60 2024-01-15T18:05:11.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.340+0000 7f56c2a00640 1 -- 172.21.15.27:0/429598773 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac0ab980 con 0x7f56c40cba60 2024-01-15T18:05:11.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.340+0000 7f56cb000640 1 -- 172.21.15.27:0/429598773 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56c40cba60 msgr2=0x7f56c40cf830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:11.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.340+0000 7f56cb000640 1 --2- 172.21.15.27:0/429598773 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56c40cba60 0x7f56c40cf830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0097c0 tx=0x7f56ac01c480 comp rx=0 tx=0).stop 2024-01-15T18:05:11.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.340+0000 7f56cb000640 1 -- 172.21.15.27:0/429598773 shutdown_connections 2024-01-15T18:05:11.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.340+0000 7f56cb000640 1 --2- 172.21.15.27:0/429598773 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56c40cba60 0x7f56c40cf830 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.340+0000 7f56cb000640 1 --2- 172.21.15.27:0/429598773 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56c40cf450 0x7f56c40d1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.340+0000 7f56cb000640 1 -- 172.21.15.27:0/429598773 >> 172.21.15.27:0/429598773 conn(0x7f56c40d0e70 msgr2=0x7f56c40d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:11.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.340+0000 7f56cb000640 1 -- 172.21.15.27:0/429598773 shutdown_connections 2024-01-15T18:05:11.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.340+0000 7f56cb000640 1 -- 172.21.15.27:0/429598773 wait complete. 2024-01-15T18:05:11.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.344+0000 7f56cb000640 1 Processor -- start 2024-01-15T18:05:11.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.344+0000 7f56cb000640 1 -- start start 2024-01-15T18:05:11.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.344+0000 7f56cb000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56c40cba60 0x7f56c4151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:11.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.344+0000 7f56cb000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c40cf450 0x7f56c40ca980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:11.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.344+0000 7f56cb000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56c415ee30 0x7f56c41501c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:11.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.344+0000 7f56cb000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c40caec0 con 0x7f56c40cf450 2024-01-15T18:05:11.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.344+0000 7f56cb000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c4157f00 con 0x7f56c40cba60 2024-01-15T18:05:11.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.344+0000 7f56cb000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c41581a0 con 0x7f56c415ee30 2024-01-15T18:05:11.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.344+0000 7f56c3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c40cf450 0x7f56c40ca980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:11.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.344+0000 7f56c3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c40cf450 0x7f56c40ca980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59560/0 (socket says 172.21.15.27:59560) 2024-01-15T18:05:11.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.344+0000 7f56c3400640 1 -- 172.21.15.27:0/3814028241 learned_addr learned my addr 172.21.15.27:0/3814028241 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:11.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.344+0000 7f56c3e00640 1 --2- 172.21.15.27:0/3814028241 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56c40cba60 0x7f56c4151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:11.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.344+0000 7f56c9000640 1 --2- 172.21.15.27:0/3814028241 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56c415ee30 0x7f56c41501c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:11.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.344+0000 7f56c3400640 1 -- 172.21.15.27:0/3814028241 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56c415ee30 msgr2=0x7f56c41501c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:11.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.344+0000 7f56c3400640 1 --2- 172.21.15.27:0/3814028241 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56c415ee30 0x7f56c41501c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.344+0000 7f56c3400640 1 -- 172.21.15.27:0/3814028241 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56c40cba60 msgr2=0x7f56c4151e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:11.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.344+0000 7f56c3400640 1 --2- 172.21.15.27:0/3814028241 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56c40cba60 0x7f56c4151e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.344+0000 7f56c3400640 1 -- 172.21.15.27:0/3814028241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ac009420 con 0x7f56c40cf450 2024-01-15T18:05:11.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.344+0000 7f56c3400640 1 --2- 172.21.15.27:0/3814028241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c40cf450 0x7f56c40ca980 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0ae040 tx=0x7f56ac019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:11.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.344+0000 7f56bbe00640 1 -- 172.21.15.27:0/3814028241 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac0b4430 con 0x7f56c40cf450 2024-01-15T18:05:11.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.344+0000 7f56bbe00640 1 -- 172.21.15.27:0/3814028241 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56ac0b45b0 con 0x7f56c40cf450 2024-01-15T18:05:11.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.344+0000 7f56bbe00640 1 -- 172.21.15.27:0/3814028241 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac0b4960 con 0x7f56c40cf450 2024-01-15T18:05:11.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.344+0000 7f56cb000640 1 -- 172.21.15.27:0/3814028241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56c4151380 con 0x7f56c40cf450 2024-01-15T18:05:11.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.344+0000 7f56cb000640 1 -- 172.21.15.27:0/3814028241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c4151940 con 0x7f56c40cf450 2024-01-15T18:05:11.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.348+0000 7f56cb000640 1 -- 172.21.15.27:0/3814028241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56c4067f30 con 0x7f56c40cf450 2024-01-15T18:05:11.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.348+0000 7f56bbe00640 1 -- 172.21.15.27:0/3814028241 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f56ac0b2040 con 0x7f56c40cf450 2024-01-15T18:05:11.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.348+0000 7f56bbe00640 1 --2- 172.21.15.27:0/3814028241 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f569c0668f0 0x7f569c068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:11.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.348+0000 7f56bbe00640 1 -- 172.21.15.27:0/3814028241 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56ac11e0e0 con 0x7f56c40cf450 2024-01-15T18:05:11.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.348+0000 7f56c3e00640 1 --2- 172.21.15.27:0/3814028241 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f569c0668f0 0x7f569c068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:11.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.348+0000 7f56c3e00640 1 --2- 172.21.15.27:0/3814028241 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f569c0668f0 0x7f569c068db0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f56b4009ac0 tx=0x7f56b4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:11.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.352+0000 7f56bbe00640 1 -- 172.21.15.27:0/3814028241 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f56c4067f30 con 0x7f56c40cf450 2024-01-15T18:05:11.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.512+0000 7f56cb000640 1 -- 172.21.15.27:0/3814028241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f56c4067f30 con 0x7f56c40cf450 2024-01-15T18:05:11.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.512+0000 7f56bbe00640 1 -- 172.21.15.27:0/3814028241 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f56c4067f30 con 0x7f56c40cf450 2024-01-15T18:05:11.515 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-15T18:05:11.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.512+0000 7f56cb000640 1 -- 172.21.15.27:0/3814028241 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f569c0668f0 msgr2=0x7f569c068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:11.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.512+0000 7f56cb000640 1 --2- 172.21.15.27:0/3814028241 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f569c0668f0 0x7f569c068db0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f56b4009ac0 tx=0x7f56b4009190 comp rx=0 tx=0).stop 2024-01-15T18:05:11.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.512+0000 7f56cb000640 1 -- 172.21.15.27:0/3814028241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c40cf450 msgr2=0x7f56c40ca980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:11.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.512+0000 7f56cb000640 1 --2- 172.21.15.27:0/3814028241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c40cf450 0x7f56c40ca980 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0ae040 tx=0x7f56ac019220 comp rx=0 tx=0).stop 2024-01-15T18:05:11.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.512+0000 7f56cb000640 1 -- 172.21.15.27:0/3814028241 shutdown_connections 2024-01-15T18:05:11.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.512+0000 7f56cb000640 1 --2- 172.21.15.27:0/3814028241 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f569c0668f0 0x7f569c068db0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.512+0000 7f56cb000640 1 --2- 172.21.15.27:0/3814028241 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56c415ee30 0x7f56c41501c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.512+0000 7f56cb000640 1 --2- 172.21.15.27:0/3814028241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56c40cf450 0x7f56c40ca980 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.512+0000 7f56cb000640 1 --2- 172.21.15.27:0/3814028241 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56c40cba60 0x7f56c4151e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.512+0000 7f56cb000640 1 -- 172.21.15.27:0/3814028241 >> 172.21.15.27:0/3814028241 conn(0x7f56c40d0e70 msgr2=0x7f56c406eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:11.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.512+0000 7f56cb000640 1 -- 172.21.15.27:0/3814028241 shutdown_connections 2024-01-15T18:05:11.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.512+0000 7f56cb000640 1 -- 172.21.15.27:0/3814028241 wait complete. 2024-01-15T18:05:11.538 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:05:11.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.676+0000 7f355e800640 1 Processor -- start 2024-01-15T18:05:11.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.676+0000 7f355e800640 1 -- start start 2024-01-15T18:05:11.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.676+0000 7f355e800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f355815d1d0 0x7f35580cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:11.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.676+0000 7f355e800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f35580cd830 0x7f35580cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:11.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.676+0000 7f355e800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35580cf220 con 0x7f35581619d0 2024-01-15T18:05:11.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.676+0000 7f355e800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35580ce220 con 0x7f35580cd830 2024-01-15T18:05:11.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.676+0000 7f355e800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35580cf500 con 0x7f355815d1d0 2024-01-15T18:05:11.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.676+0000 7f3557e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f35580cd830 0x7f35580cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:11.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.676+0000 7f3557e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f35580cd830 0x7f35580cdce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59574/0 (socket says 172.21.15.27:59574) 2024-01-15T18:05:11.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.676+0000 7f3557e00640 1 -- 172.21.15.27:0/3140747412 learned_addr learned my addr 172.21.15.27:0/3140747412 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:11.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.676+0000 7f3557400640 1 --2- 172.21.15.27:0/3140747412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f355815d1d0 0x7f35580cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:11.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.676+0000 7f3557e00640 1 -- 172.21.15.27:0/3140747412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f355815d1d0 msgr2=0x7f35580cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:11.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.676+0000 7f3557e00640 1 --2- 172.21.15.27:0/3140747412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f355815d1d0 0x7f35580cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.676+0000 7f3557e00640 1 -- 172.21.15.27:0/3140747412 >> v1:172.21.15.93:6789/0 conn(0x7f35581619d0 legacy=0x7f355806bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:11.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.676+0000 7f3557e00640 1 -- 172.21.15.27:0/3140747412 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35580cf7e0 con 0x7f35580cd830 2024-01-15T18:05:11.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.680+0000 7f3557e00640 1 --2- 172.21.15.27:0/3140747412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35580cd830 0x7f35580cdce0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3550009e50 tx=0x7f355001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e071f3a930c0193 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:11.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.680+0000 7f3556000640 1 -- 172.21.15.27:0/3140747412 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35500a5650 con 0x7f35580cd830 2024-01-15T18:05:11.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.680+0000 7f3556000640 1 -- 172.21.15.27:0/3140747412 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35500a57b0 con 0x7f35580cd830 2024-01-15T18:05:11.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.680+0000 7f3556000640 1 -- 172.21.15.27:0/3140747412 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35500a5b40 con 0x7f35580cd830 2024-01-15T18:05:11.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.680+0000 7f355e800640 1 -- 172.21.15.27:0/3140747412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35580cd830 msgr2=0x7f35580cdce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:11.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.680+0000 7f355e800640 1 --2- 172.21.15.27:0/3140747412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35580cd830 0x7f35580cdce0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3550009e50 tx=0x7f355001c9d0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.680+0000 7f355e800640 1 -- 172.21.15.27:0/3140747412 shutdown_connections 2024-01-15T18:05:11.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.680+0000 7f355e800640 1 --2- 172.21.15.27:0/3140747412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35580cd830 0x7f35580cdce0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.680+0000 7f355e800640 1 --2- 172.21.15.27:0/3140747412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f355815d1d0 0x7f35580cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.680+0000 7f355e800640 1 -- 172.21.15.27:0/3140747412 >> 172.21.15.27:0/3140747412 conn(0x7f35580b2af0 msgr2=0x7f35580b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:11.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.680+0000 7f355e800640 1 -- 172.21.15.27:0/3140747412 shutdown_connections 2024-01-15T18:05:11.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.680+0000 7f355e800640 1 -- 172.21.15.27:0/3140747412 wait complete. 2024-01-15T18:05:11.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.680+0000 7f355e800640 1 Processor -- start 2024-01-15T18:05:11.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.680+0000 7f355e800640 1 -- start start 2024-01-15T18:05:11.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.684+0000 7f355e800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35580cd830 0x7f355815bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:11.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.684+0000 7f355e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f355815d1d0 0x7f355814e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:11.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.684+0000 7f355e800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f35581619d0 0x7f3558155e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:11.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.684+0000 7f355e800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35580c8080 con 0x7f355815d1d0 2024-01-15T18:05:11.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.684+0000 7f355e800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355814ea80 con 0x7f35581619d0 2024-01-15T18:05:11.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.684+0000 7f355e800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355814ed60 con 0x7f35580cd830 2024-01-15T18:05:11.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.684+0000 7f3556a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f355815d1d0 0x7f355814e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:11.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.684+0000 7f3556a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f355815d1d0 0x7f355814e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59576/0 (socket says 172.21.15.27:59576) 2024-01-15T18:05:11.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.684+0000 7f3556a00640 1 -- 172.21.15.27:0/2432313412 learned_addr learned my addr 172.21.15.27:0/2432313412 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:11.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.684+0000 7f3557400640 1 --2- 172.21.15.27:0/2432313412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35580cd830 0x7f355815bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:11.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.684+0000 7f3557e00640 1 --2- 172.21.15.27:0/2432313412 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f35581619d0 0x7f3558155e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:11.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.684+0000 7f3556a00640 1 -- 172.21.15.27:0/2432313412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35580cd830 msgr2=0x7f355815bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:11.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.684+0000 7f3556a00640 1 --2- 172.21.15.27:0/2432313412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35580cd830 0x7f355815bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.684+0000 7f3556a00640 1 -- 172.21.15.27:0/2432313412 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f35581619d0 msgr2=0x7f3558155e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:11.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.684+0000 7f3556a00640 1 --2- 172.21.15.27:0/2432313412 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f35581619d0 0x7f3558155e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.684+0000 7f3556a00640 1 -- 172.21.15.27:0/2432313412 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3550009b30 con 0x7f355815d1d0 2024-01-15T18:05:11.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.684+0000 7f3557400640 1 --2- 172.21.15.27:0/2432313412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35580cd830 0x7f355815bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:11.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.684+0000 7f3556a00640 1 --2- 172.21.15.27:0/2432313412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f355815d1d0 0x7f355814e110 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f354000b3e0 tx=0x7f354000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:11.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.684+0000 7f354f400640 1 -- 172.21.15.27:0/2432313412 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3540011830 con 0x7f355815d1d0 2024-01-15T18:05:11.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.684+0000 7f354f400640 1 -- 172.21.15.27:0/2432313412 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3540011990 con 0x7f355815d1d0 2024-01-15T18:05:11.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.684+0000 7f355e800640 1 -- 172.21.15.27:0/2432313412 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f355814f040 con 0x7f355815d1d0 2024-01-15T18:05:11.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.684+0000 7f354f400640 1 -- 172.21.15.27:0/2432313412 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3540011d80 con 0x7f355815d1d0 2024-01-15T18:05:11.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.684+0000 7f355e800640 1 -- 172.21.15.27:0/2432313412 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f355814e650 con 0x7f355815d1d0 2024-01-15T18:05:11.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.684+0000 7f355e800640 1 -- 172.21.15.27:0/2432313412 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f355804e850 con 0x7f355815d1d0 2024-01-15T18:05:11.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.688+0000 7f354f400640 1 -- 172.21.15.27:0/2432313412 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f354001a400 con 0x7f355815d1d0 2024-01-15T18:05:11.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.688+0000 7f354f400640 1 --2- 172.21.15.27:0/2432313412 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f35300668f0 0x7f3530068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:11.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.688+0000 7f354f400640 1 -- 172.21.15.27:0/2432313412 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35400840c0 con 0x7f355815d1d0 2024-01-15T18:05:11.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.688+0000 7f3557400640 1 --2- 172.21.15.27:0/2432313412 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f35300668f0 0x7f3530068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:11.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.688+0000 7f3557400640 1 --2- 172.21.15.27:0/2432313412 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f35300668f0 0x7f3530068db0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3548007e50 tx=0x7f3548008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:11.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.692+0000 7f354f400640 1 -- 172.21.15.27:0/2432313412 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f35400513a0 con 0x7f355815d1d0 2024-01-15T18:05:11.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.880+0000 7f355e800640 1 -- 172.21.15.27:0/2432313412 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35580cd210 con 0x7f355815d1d0 2024-01-15T18:05:11.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.880+0000 7f354f400640 1 -- 172.21.15.27:0/2432313412 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7f35400549c0 con 0x7f355815d1d0 2024-01-15T18:05:11.884 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:11.884 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/393360880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":393360880},{"type":"v1","addr":"172.21.15.93:6836","nonce":393360880}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4229,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4063308188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4063308188},{"type":"v1","addr":"172.21.15.27:6835","nonce":4063308188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4305,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/2343128056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":2343128056},{"type":"v1","addr":"172.21.15.93:6837","nonce":2343128056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:04:58.208165+0000","modified":"2024-01-15T18:05:01.225419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2657553192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2657553192},{"type":"v1","addr":"172.21.15.27:6834","nonce":2657553192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T18:05:11.885 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 8 2024-01-15T18:05:11.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.880+0000 7f355e800640 1 -- 172.21.15.27:0/2432313412 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f35300668f0 msgr2=0x7f3530068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:11.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.880+0000 7f355e800640 1 --2- 172.21.15.27:0/2432313412 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f35300668f0 0x7f3530068db0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3548007e50 tx=0x7f3548008040 comp rx=0 tx=0).stop 2024-01-15T18:05:11.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.880+0000 7f355e800640 1 -- 172.21.15.27:0/2432313412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f355815d1d0 msgr2=0x7f355814e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:11.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.880+0000 7f355e800640 1 --2- 172.21.15.27:0/2432313412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f355815d1d0 0x7f355814e110 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f354000b3e0 tx=0x7f354000b8a0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.880+0000 7f355e800640 1 -- 172.21.15.27:0/2432313412 shutdown_connections 2024-01-15T18:05:11.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.880+0000 7f355e800640 1 --2- 172.21.15.27:0/2432313412 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f35300668f0 0x7f3530068db0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.880+0000 7f355e800640 1 --2- 172.21.15.27:0/2432313412 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f35581619d0 0x7f3558155e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.880+0000 7f355e800640 1 --2- 172.21.15.27:0/2432313412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f355815d1d0 0x7f355814e110 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.880+0000 7f355e800640 1 --2- 172.21.15.27:0/2432313412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35580cd830 0x7f355815bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.880+0000 7f355e800640 1 -- 172.21.15.27:0/2432313412 >> 172.21.15.27:0/2432313412 conn(0x7f35580b2af0 msgr2=0x7f35581598a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:11.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.880+0000 7f355e800640 1 -- 172.21.15.27:0/2432313412 shutdown_connections 2024-01-15T18:05:11.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:11.884+0000 7f355e800640 1 -- 172.21.15.27:0/2432313412 wait complete. 2024-01-15T18:05:11.906 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:12.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.044+0000 7f7632800640 1 Processor -- start 2024-01-15T18:05:12.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.044+0000 7f7632800640 1 -- start start 2024-01-15T18:05:12.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.044+0000 7f7632800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f762c0cf9b0 0x7f762c0b10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:12.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.044+0000 7f7632800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f762c0cedc0 0x7f762c0b1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:12.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.044+0000 7f7632800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f762c0b38b0 con 0x7f762c15e7d0 2024-01-15T18:05:12.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.044+0000 7f7632800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f762c0cd550 con 0x7f762c0cedc0 2024-01-15T18:05:12.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.044+0000 7f7632800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f762c0cd270 con 0x7f762c0cf9b0 2024-01-15T18:05:12.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.044+0000 7f762b400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f762c15e7d0 0x7f762c0cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42478/0 (socket says 172.21.15.27:42478) 2024-01-15T18:05:12.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.044+0000 7f762be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f762c0cedc0 0x7f762c0b1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:12.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.044+0000 7f762b400640 1 -- 172.21.15.27:0/788172904 learned_addr learned my addr 172.21.15.27:0/788172904 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:12.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.044+0000 7f762aa00640 1 --2- 172.21.15.27:0/788172904 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f762c0cf9b0 0x7f762c0b10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:12.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.048+0000 7f762be00640 1 -- 172.21.15.27:0/788172904 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f762c0cf9b0 msgr2=0x7f762c0b10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:12.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.048+0000 7f762be00640 1 --2- 172.21.15.27:0/788172904 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f762c0cf9b0 0x7f762c0b10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.048+0000 7f762be00640 1 -- 172.21.15.27:0/788172904 >> v1:172.21.15.27:6789/0 conn(0x7f762c15e7d0 legacy=0x7f762c0cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:12.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.048+0000 7f762be00640 1 -- 172.21.15.27:0/788172904 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762c0b35d0 con 0x7f762c0cedc0 2024-01-15T18:05:12.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.048+0000 7f762be00640 1 --2- 172.21.15.27:0/788172904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f762c0cedc0 0x7f762c0b1600 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7624009d90 tx=0x7f762401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8baf7b262cc065cf server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:12.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.048+0000 7f762a000640 1 -- 172.21.15.27:0/788172904 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76240a5900 con 0x7f762c0cedc0 2024-01-15T18:05:12.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.048+0000 7f762a000640 1 -- 172.21.15.27:0/788172904 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76240a5a60 con 0x7f762c0cedc0 2024-01-15T18:05:12.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.048+0000 7f762a000640 1 -- 172.21.15.27:0/788172904 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76240a5df0 con 0x7f762c0cedc0 2024-01-15T18:05:12.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.048+0000 7f7632800640 1 -- 172.21.15.27:0/788172904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f762c0cedc0 msgr2=0x7f762c0b1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:12.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.048+0000 7f7632800640 1 --2- 172.21.15.27:0/788172904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f762c0cedc0 0x7f762c0b1600 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7624009d90 tx=0x7f762401c950 comp rx=0 tx=0).stop 2024-01-15T18:05:12.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.052+0000 7f7632800640 1 -- 172.21.15.27:0/788172904 shutdown_connections 2024-01-15T18:05:12.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.052+0000 7f7632800640 1 --2- 172.21.15.27:0/788172904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f762c0cedc0 0x7f762c0b1600 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.052+0000 7f7632800640 1 --2- 172.21.15.27:0/788172904 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f762c0cf9b0 0x7f762c0b10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.052+0000 7f7632800640 1 -- 172.21.15.27:0/788172904 >> 172.21.15.27:0/788172904 conn(0x7f762c0b2a10 msgr2=0x7f762c0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:12.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.052+0000 7f7632800640 1 -- 172.21.15.27:0/788172904 shutdown_connections 2024-01-15T18:05:12.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.052+0000 7f7632800640 1 -- 172.21.15.27:0/788172904 wait complete. 2024-01-15T18:05:12.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.052+0000 7f7632800640 1 Processor -- start 2024-01-15T18:05:12.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.052+0000 7f7632800640 1 -- start start 2024-01-15T18:05:12.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.052+0000 7f7632800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f762c0cedc0 0x7f762c06b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:12.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.052+0000 7f7632800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f762c0cf9b0 0x7f762c1640c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:12.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.052+0000 7f7632800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f762c15e7d0 0x7f762c1f6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:12.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.052+0000 7f7632800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c164600 con 0x7f762c15e7d0 2024-01-15T18:05:12.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.052+0000 7f7632800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c1f6bc0 con 0x7f762c0cedc0 2024-01-15T18:05:12.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.052+0000 7f7632800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f762c164c50 con 0x7f762c0cf9b0 2024-01-15T18:05:12.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.052+0000 7f762be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f762c15e7d0 0x7f762c1f6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:12.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.052+0000 7f762be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f762c15e7d0 0x7f762c1f6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59596/0 (socket says 172.21.15.27:59596) 2024-01-15T18:05:12.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.052+0000 7f762be00640 1 -- 172.21.15.27:0/1156813075 learned_addr learned my addr 172.21.15.27:0/1156813075 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:12.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.052+0000 7f762aa00640 1 --2- 172.21.15.27:0/1156813075 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f762c0cf9b0 0x7f762c1640c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:12.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.052+0000 7f762b400640 1 --2- 172.21.15.27:0/1156813075 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f762c0cedc0 0x7f762c06b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:12.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.052+0000 7f762be00640 1 -- 172.21.15.27:0/1156813075 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f762c0cf9b0 msgr2=0x7f762c1640c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:12.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.052+0000 7f762be00640 1 --2- 172.21.15.27:0/1156813075 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f762c0cf9b0 0x7f762c1640c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.052+0000 7f762be00640 1 -- 172.21.15.27:0/1156813075 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f762c0cedc0 msgr2=0x7f762c06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:12.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.052+0000 7f762be00640 1 --2- 172.21.15.27:0/1156813075 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f762c0cedc0 0x7f762c06b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.052+0000 7f762be00640 1 -- 172.21.15.27:0/1156813075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7624009a70 con 0x7f762c15e7d0 2024-01-15T18:05:12.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.056+0000 7f762be00640 1 --2- 172.21.15.27:0/1156813075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f762c15e7d0 0x7f762c1f6680 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f762401b000 tx=0x7f76240196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:12.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.056+0000 7f7623400640 1 -- 172.21.15.27:0/1156813075 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76240b9860 con 0x7f762c15e7d0 2024-01-15T18:05:12.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.056+0000 7f7623400640 1 -- 172.21.15.27:0/1156813075 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76240a58f0 con 0x7f762c15e7d0 2024-01-15T18:05:12.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.056+0000 7f7632800640 1 -- 172.21.15.27:0/1156813075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f762c06bca0 con 0x7f762c15e7d0 2024-01-15T18:05:12.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.056+0000 7f7623400640 1 -- 172.21.15.27:0/1156813075 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76240b9bc0 con 0x7f762c15e7d0 2024-01-15T18:05:12.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.056+0000 7f7632800640 1 -- 172.21.15.27:0/1156813075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762c06c260 con 0x7f762c15e7d0 2024-01-15T18:05:12.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.056+0000 7f7632800640 1 -- 172.21.15.27:0/1156813075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f762c164e20 con 0x7f762c15e7d0 2024-01-15T18:05:12.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.056+0000 7f7623400640 1 -- 172.21.15.27:0/1156813075 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f76240b9d20 con 0x7f762c15e7d0 2024-01-15T18:05:12.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.056+0000 7f7623400640 1 --2- 172.21.15.27:0/1156813075 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f76040668f0 0x7f7604068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:12.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.056+0000 7f7623400640 1 -- 172.21.15.27:0/1156813075 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f762411f870 con 0x7f762c15e7d0 2024-01-15T18:05:12.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.056+0000 7f762b400640 1 --2- 172.21.15.27:0/1156813075 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f76040668f0 0x7f7604068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:12.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.060+0000 7f762b400640 1 --2- 172.21.15.27:0/1156813075 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f76040668f0 0x7f7604068db0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f761c005890 tx=0x7f761c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:12.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.060+0000 7f7623400640 1 -- 172.21.15.27:0/1156813075 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f76240eca10 con 0x7f762c15e7d0 2024-01-15T18:05:12.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.216+0000 7f7632800640 1 -- 172.21.15.27:0/1156813075 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f762c04e850 con 0x7f762c15e7d0 2024-01-15T18:05:12.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.220+0000 7f7623400640 1 -- 172.21.15.27:0/1156813075 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16409 (secure 0 0 0) 0x7f76240f0030 con 0x7f762c15e7d0 2024-01-15T18:05:12.224 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:12.224 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:04:59.837052+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:04:56.220608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T18:04:57.210421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:12.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.220+0000 7f7632800640 1 -- 172.21.15.27:0/1156813075 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f76040668f0 msgr2=0x7f7604068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:12.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.220+0000 7f7632800640 1 --2- 172.21.15.27:0/1156813075 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f76040668f0 0x7f7604068db0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f761c005890 tx=0x7f761c009040 comp rx=0 tx=0).stop 2024-01-15T18:05:12.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.220+0000 7f7632800640 1 -- 172.21.15.27:0/1156813075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f762c15e7d0 msgr2=0x7f762c1f6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:12.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.220+0000 7f7632800640 1 --2- 172.21.15.27:0/1156813075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f762c15e7d0 0x7f762c1f6680 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f762401b000 tx=0x7f76240196f0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.220+0000 7f7632800640 1 -- 172.21.15.27:0/1156813075 shutdown_connections 2024-01-15T18:05:12.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.220+0000 7f7632800640 1 --2- 172.21.15.27:0/1156813075 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f76040668f0 0x7f7604068db0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.220+0000 7f7632800640 1 --2- 172.21.15.27:0/1156813075 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f762c15e7d0 0x7f762c1f6680 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.220+0000 7f7632800640 1 --2- 172.21.15.27:0/1156813075 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f762c0cf9b0 0x7f762c1640c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.220+0000 7f7632800640 1 --2- 172.21.15.27:0/1156813075 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f762c0cedc0 0x7f762c06b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.220+0000 7f7632800640 1 -- 172.21.15.27:0/1156813075 >> 172.21.15.27:0/1156813075 conn(0x7f762c0b2a10 msgr2=0x7f762c06e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:12.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.220+0000 7f7632800640 1 -- 172.21.15.27:0/1156813075 shutdown_connections 2024-01-15T18:05:12.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.220+0000 7f7632800640 1 -- 172.21.15.27:0/1156813075 wait complete. 2024-01-15T18:05:12.246 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:05:12.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff137600640 1 Processor -- start 2024-01-15T18:05:12.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff137600640 1 -- start start 2024-01-15T18:05:12.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff137600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff13015c130 0x7ff13006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:12.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff137600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff13006e240 0x7ff13015d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:12.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff137600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff13006c4a0 con 0x7ff1300701a0 2024-01-15T18:05:12.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff137600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff13015c660 con 0x7ff13006e240 2024-01-15T18:05:12.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff137600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff13015c940 con 0x7ff13015c130 2024-01-15T18:05:12.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff134c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff1300701a0 0x7ff130070580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42492/0 (socket says 172.21.15.27:42492) 2024-01-15T18:05:12.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff134c00640 1 -- 172.21.15.27:0/714257110 learned_addr learned my addr 172.21.15.27:0/714257110 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:12.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff12fe00640 1 --2- 172.21.15.27:0/714257110 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff13015c130 0x7ff13006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:12.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff135600640 1 --2- 172.21.15.27:0/714257110 >> v2:172.21.15.93:3300/0 conn(0x7ff13006e240 0x7ff13015d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:12.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff12f400640 1 -- 172.21.15.27:0/714257110 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 50863073 0 0) 0x7ff13006c4a0 con 0x7ff1300701a0 2024-01-15T18:05:12.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff12f400640 1 -- 172.21.15.27:0/714257110 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff110003530 con 0x7ff1300701a0 2024-01-15T18:05:12.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff12f400640 1 -- 172.21.15.27:0/714257110 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1961322708 0 0) 0x7ff110003530 con 0x7ff1300701a0 2024-01-15T18:05:12.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff12f400640 1 -- 172.21.15.27:0/714257110 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff13015c130 msgr2=0x7ff13006dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:12.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff12f400640 1 --2- 172.21.15.27:0/714257110 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff13015c130 0x7ff13006dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff12f400640 1 -- 172.21.15.27:0/714257110 >> v2:172.21.15.93:3300/0 conn(0x7ff13006e240 msgr2=0x7ff13015d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:12.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff12f400640 1 --2- 172.21.15.27:0/714257110 >> v2:172.21.15.93:3300/0 conn(0x7ff13006e240 0x7ff13015d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff12f400640 1 -- 172.21.15.27:0/714257110 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13015cc20 con 0x7ff1300701a0 2024-01-15T18:05:12.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff12f400640 1 -- 172.21.15.27:0/714257110 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7ff120001680 con 0x7ff1300701a0 2024-01-15T18:05:12.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff12fe00640 1 --2- 172.21.15.27:0/714257110 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff13015c130 0x7ff13006dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:12.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff12f400640 1 -- 172.21.15.27:0/714257110 >> v1:172.21.15.27:6789/0 conn(0x7ff1300701a0 legacy=0x7ff130070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:12.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff12f400640 1 --2- 172.21.15.27:0/714257110 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1100039f0 0x7ff110003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:12.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff12f400640 1 --2- 172.21.15.27:0/714257110 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff1100047d0 0x7ff110004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:12.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff12f400640 1 --2- 172.21.15.27:0/714257110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff110005190 0x7ff11000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:12.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff12f400640 1 -- 172.21.15.27:0/714257110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13015c660 con 0x7ff110005190 2024-01-15T18:05:12.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff12f400640 1 -- 172.21.15.27:0/714257110 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13015c940 con 0x7ff1100047d0 2024-01-15T18:05:12.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff12f400640 1 -- 172.21.15.27:0/714257110 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff11000bed0 con 0x7ff1100039f0 2024-01-15T18:05:12.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.384+0000 7ff135600640 1 --2- 172.21.15.27:0/714257110 >> v2:172.21.15.93:3300/0 conn(0x7ff13006e240 0x7ff13015d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:12.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.388+0000 7ff134c00640 1 --2- 172.21.15.27:0/714257110 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1100039f0 0x7ff110003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:12.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.388+0000 7ff12fe00640 1 --2- 172.21.15.27:0/714257110 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff1100047d0 0x7ff110004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:12.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.388+0000 7ff135600640 1 --2- 172.21.15.27:0/714257110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff110005190 0x7ff11000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:12.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.388+0000 7ff134c00640 1 -- 172.21.15.27:0/714257110 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff1100047d0 msgr2=0x7ff110004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:12.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.388+0000 7ff134c00640 1 --2- 172.21.15.27:0/714257110 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff1100047d0 0x7ff110004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.388+0000 7ff134c00640 1 -- 172.21.15.27:0/714257110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff110005190 msgr2=0x7ff11000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:12.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.388+0000 7ff134c00640 1 --2- 172.21.15.27:0/714257110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff110005190 0x7ff11000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.388+0000 7ff134c00640 1 -- 172.21.15.27:0/714257110 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff110003530 con 0x7ff1100039f0 2024-01-15T18:05:12.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.388+0000 7ff135600640 1 --2- 172.21.15.27:0/714257110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff110005190 0x7ff11000b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:12.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.388+0000 7ff134c00640 1 --2- 172.21.15.27:0/714257110 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1100039f0 0x7ff110003e10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff12000c270 tx=0x7ff12001cc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13effddb1246c868 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:12.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.388+0000 7ff12f400640 1 -- 172.21.15.27:0/714257110 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1200a5960 con 0x7ff1100039f0 2024-01-15T18:05:12.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.388+0000 7ff12f400640 1 -- 172.21.15.27:0/714257110 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1200a5ac0 con 0x7ff1100039f0 2024-01-15T18:05:12.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.388+0000 7ff12f400640 1 -- 172.21.15.27:0/714257110 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1200058e0 con 0x7ff1100039f0 2024-01-15T18:05:12.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.392+0000 7ff137600640 1 -- 172.21.15.27:0/714257110 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1100039f0 msgr2=0x7ff110003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:12.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.392+0000 7ff137600640 1 --2- 172.21.15.27:0/714257110 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1100039f0 0x7ff110003e10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff12000c270 tx=0x7ff12001cc00 comp rx=0 tx=0).stop 2024-01-15T18:05:12.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.392+0000 7ff135600640 1 -- 172.21.15.27:0/714257110 reap_dead start 2024-01-15T18:05:12.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.392+0000 7ff137600640 1 -- 172.21.15.27:0/714257110 shutdown_connections 2024-01-15T18:05:12.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.392+0000 7ff137600640 1 -- 172.21.15.27:0/714257110 >> 172.21.15.27:0/714257110 conn(0x7ff1300cec60 msgr2=0x7ff1300cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:12.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.392+0000 7ff137600640 1 -- 172.21.15.27:0/714257110 shutdown_connections 2024-01-15T18:05:12.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.392+0000 7ff137600640 1 -- 172.21.15.27:0/714257110 wait complete. 2024-01-15T18:05:12.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.392+0000 7ff137600640 1 Processor -- start 2024-01-15T18:05:12.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.392+0000 7ff137600640 1 -- start start 2024-01-15T18:05:12.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.392+0000 7ff137600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1100039f0 0x7ff1301f4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:12.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.392+0000 7ff137600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1301efe70 0x7ff130144c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:12.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.392+0000 7ff137600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff1301451c0 0x7ff1300afce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:12.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.392+0000 7ff137600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff130145640 con 0x7ff1301efe70 2024-01-15T18:05:12.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.392+0000 7ff137600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff130148f70 con 0x7ff1301451c0 2024-01-15T18:05:12.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.392+0000 7ff137600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff130145830 con 0x7ff1100039f0 2024-01-15T18:05:12.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.392+0000 7ff12fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1301efe70 0x7ff130144c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:12.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.392+0000 7ff12fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1301efe70 0x7ff130144c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59602/0 (socket says 172.21.15.27:59602) 2024-01-15T18:05:12.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.392+0000 7ff12fe00640 1 -- 172.21.15.27:0/2943611199 learned_addr learned my addr 172.21.15.27:0/2943611199 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:12.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.392+0000 7ff134c00640 1 --2- 172.21.15.27:0/2943611199 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1100039f0 0x7ff1301f4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:12.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.392+0000 7ff135600640 1 --2- 172.21.15.27:0/2943611199 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff1301451c0 0x7ff1300afce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.396+0000 7ff134c00640 1 -- 172.21.15.27:0/2943611199 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff1301451c0 msgr2=0x7ff1300afce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.396+0000 7ff134c00640 1 --2- 172.21.15.27:0/2943611199 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff1301451c0 0x7ff1300afce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.396+0000 7ff134c00640 1 -- 172.21.15.27:0/2943611199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1301efe70 msgr2=0x7ff130144c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.396+0000 7ff134c00640 1 --2- 172.21.15.27:0/2943611199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1301efe70 0x7ff130144c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.396+0000 7ff134c00640 1 -- 172.21.15.27:0/2943611199 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff130145b10 con 0x7ff1100039f0 2024-01-15T18:05:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.396+0000 7ff12fe00640 1 --2- 172.21.15.27:0/2943611199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1301efe70 0x7ff130144c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.396+0000 7ff134c00640 1 --2- 172.21.15.27:0/2943611199 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1100039f0 0x7ff1301f4600 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff11000c060 tx=0x7ff110005800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.396+0000 7ff12cc00640 1 -- 172.21.15.27:0/2943611199 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1200a5960 con 0x7ff1100039f0 2024-01-15T18:05:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.396+0000 7ff12cc00640 1 -- 172.21.15.27:0/2943611199 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1200b3070 con 0x7ff1100039f0 2024-01-15T18:05:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.396+0000 7ff137600640 1 -- 172.21.15.27:0/2943611199 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1301f3b20 con 0x7ff1100039f0 2024-01-15T18:05:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.396+0000 7ff12cc00640 1 -- 172.21.15.27:0/2943611199 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff120007530 con 0x7ff1100039f0 2024-01-15T18:05:12.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.396+0000 7ff137600640 1 -- 172.21.15.27:0/2943611199 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1301f40e0 con 0x7ff1100039f0 2024-01-15T18:05:12.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.396+0000 7ff137600640 1 -- 172.21.15.27:0/2943611199 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff13004e850 con 0x7ff1100039f0 2024-01-15T18:05:12.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.396+0000 7ff12cc00640 1 -- 172.21.15.27:0/2943611199 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7ff1200d7480 con 0x7ff1100039f0 2024-01-15T18:05:12.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.400+0000 7ff12cc00640 1 --2- 172.21.15.27:0/2943611199 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff1080668f0 0x7ff108068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:12.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.400+0000 7ff12cc00640 1 -- 172.21.15.27:0/2943611199 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff1200b5070 con 0x7ff1100039f0 2024-01-15T18:05:12.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.400+0000 7ff12fe00640 1 --2- 172.21.15.27:0/2943611199 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff1080668f0 0x7ff108068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:12.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.400+0000 7ff12fe00640 1 --2- 172.21.15.27:0/2943611199 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff1080668f0 0x7ff108068db0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff130142b00 tx=0x7ff118009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:12.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.404+0000 7ff12cc00640 1 -- 172.21.15.27:0/2943611199 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff1200eb5e0 con 0x7ff1100039f0 2024-01-15T18:05:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.592+0000 7ff137600640 1 -- 172.21.15.27:0/2943611199 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1300008e0 con 0x7ff1100039f0 2024-01-15T18:05:12.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.592+0000 7ff12cc00640 1 -- 172.21.15.27:0/2943611199 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7ff120001c60 con 0x7ff1100039f0 2024-01-15T18:05:12.598 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:12.598 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/393360880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":393360880},{"type":"v1","addr":"172.21.15.93:6836","nonce":393360880}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4229,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4063308188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4063308188},{"type":"v1","addr":"172.21.15.27:6835","nonce":4063308188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4305,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/2343128056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":2343128056},{"type":"v1","addr":"172.21.15.93:6837","nonce":2343128056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:04:58.208165+0000","modified":"2024-01-15T18:05:01.225419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2657553192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2657553192},{"type":"v1","addr":"172.21.15.27:6834","nonce":2657553192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T18:05:12.599 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 8 2024-01-15T18:05:12.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.596+0000 7ff137600640 1 -- 172.21.15.27:0/2943611199 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff1080668f0 msgr2=0x7ff108068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:12.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.596+0000 7ff137600640 1 --2- 172.21.15.27:0/2943611199 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff1080668f0 0x7ff108068db0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff130142b00 tx=0x7ff118009190 comp rx=0 tx=0).stop 2024-01-15T18:05:12.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.596+0000 7ff137600640 1 -- 172.21.15.27:0/2943611199 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1100039f0 msgr2=0x7ff1301f4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:12.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.596+0000 7ff137600640 1 --2- 172.21.15.27:0/2943611199 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1100039f0 0x7ff1301f4600 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff11000c060 tx=0x7ff110005800 comp rx=0 tx=0).stop 2024-01-15T18:05:12.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.596+0000 7ff137600640 1 -- 172.21.15.27:0/2943611199 shutdown_connections 2024-01-15T18:05:12.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.596+0000 7ff137600640 1 --2- 172.21.15.27:0/2943611199 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff1080668f0 0x7ff108068db0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.596+0000 7ff137600640 1 --2- 172.21.15.27:0/2943611199 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff1301451c0 0x7ff1300afce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.596+0000 7ff137600640 1 --2- 172.21.15.27:0/2943611199 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1301efe70 0x7ff130144c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.596+0000 7ff137600640 1 --2- 172.21.15.27:0/2943611199 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1100039f0 0x7ff1301f4600 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.596+0000 7ff137600640 1 -- 172.21.15.27:0/2943611199 >> 172.21.15.27:0/2943611199 conn(0x7ff1300cec60 msgr2=0x7ff1300c9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:12.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.596+0000 7ff137600640 1 -- 172.21.15.27:0/2943611199 shutdown_connections 2024-01-15T18:05:12.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:12.596+0000 7ff137600640 1 -- 172.21.15.27:0/2943611199 wait complete. 2024-01-15T18:05:13.023 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-15T18:04:58.208165+0000', 'modified': '2024-01-15T18:05:01.225419+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4308}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4308': {'gid': 4308, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2657553192', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2657553192}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2657553192}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-15T18:05:13.024 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-15T18:05:13.024 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-15T18:05:13.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.163+0000 7fc9cae00640 1 Processor -- start 2024-01-15T18:05:13.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.163+0000 7fc9cae00640 1 -- start start 2024-01-15T18:05:13.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.163+0000 7fc9cae00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fc9c406c200 0x7fc9c406d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.163+0000 7fc9cae00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc9c40cad40 0x7fc9c40cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.163+0000 7fc9cae00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c406c750 con 0x7fc9c4143da0 2024-01-15T18:05:13.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.163+0000 7fc9cae00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9c40cb710 con 0x7fc9c406c200 2024-01-15T18:05:13.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.163+0000 7fc9cae00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c4144180 con 0x7fc9c40cad40 2024-01-15T18:05:13.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.163+0000 7fc9c8e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc9c40cad40 0x7fc9c40cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.163+0000 7fc9c3400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fc9c406c200 0x7fc9c406d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.163+0000 7fc9c8e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc9c40cad40 0x7fc9c40cb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:43152/0 (socket says 172.21.15.27:43152) 2024-01-15T18:05:13.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.163+0000 7fc9c8e00640 1 -- 172.21.15.27:0/2917635334 learned_addr learned my addr 172.21.15.27:0/2917635334 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:13.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.163+0000 7fc9c3e00640 1 --1- 172.21.15.27:0/2917635334 >> v1:172.21.15.93:6789/0 conn(0x7fc9c4143da0 0x7fc9c40cb9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:57224/0 (socket says 172.21.15.27:57224) 2024-01-15T18:05:13.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.163+0000 7fc9c3400640 1 -- 172.21.15.27:0/2917635334 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc9c40cad40 msgr2=0x7fc9c40cb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:13.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.163+0000 7fc9c3400640 1 --2- 172.21.15.27:0/2917635334 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc9c40cad40 0x7fc9c40cb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.163+0000 7fc9c3400640 1 -- 172.21.15.27:0/2917635334 >> v1:172.21.15.93:6789/0 conn(0x7fc9c4143da0 legacy=0x7fc9c40cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.163+0000 7fc9c2a00640 1 -- 172.21.15.27:0/2917635334 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3777024034 0 0) 0x7fc9c406c750 con 0x7fc9c4143da0 2024-01-15T18:05:13.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.163+0000 7fc9c3400640 1 -- 172.21.15.27:0/2917635334 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c406ca30 con 0x7fc9c406c200 2024-01-15T18:05:13.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.167+0000 7fc9c3400640 1 --2- 172.21.15.27:0/2917635334 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc9c406c200 0x7fc9c406d7f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac0097c0 tx=0x7fc9ac01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f110c819456403a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:13.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.167+0000 7fc9c2a00640 1 -- 172.21.15.27:0/2917635334 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac0ab600 con 0x7fc9c406c200 2024-01-15T18:05:13.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.167+0000 7fc9c2a00640 1 -- 172.21.15.27:0/2917635334 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9ac0a7050 con 0x7fc9c406c200 2024-01-15T18:05:13.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.167+0000 7fc9c2a00640 1 -- 172.21.15.27:0/2917635334 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac0ab980 con 0x7fc9c406c200 2024-01-15T18:05:13.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.167+0000 7fc9cae00640 1 -- 172.21.15.27:0/2917635334 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc9c406c200 msgr2=0x7fc9c406d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.167+0000 7fc9cae00640 1 --2- 172.21.15.27:0/2917635334 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc9c406c200 0x7fc9c406d7f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac0097c0 tx=0x7fc9ac01c480 comp rx=0 tx=0).stop 2024-01-15T18:05:13.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.167+0000 7fc9cae00640 1 -- 172.21.15.27:0/2917635334 shutdown_connections 2024-01-15T18:05:13.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.167+0000 7fc9cae00640 1 --2- 172.21.15.27:0/2917635334 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc9c40cad40 0x7fc9c40cb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.167+0000 7fc9cae00640 1 --2- 172.21.15.27:0/2917635334 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc9c406c200 0x7fc9c406d7f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.167+0000 7fc9cae00640 1 -- 172.21.15.27:0/2917635334 >> 172.21.15.27:0/2917635334 conn(0x7fc9c40b0850 msgr2=0x7fc9c40b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:13.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.167+0000 7fc9cae00640 1 -- 172.21.15.27:0/2917635334 shutdown_connections 2024-01-15T18:05:13.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.167+0000 7fc9cae00640 1 -- 172.21.15.27:0/2917635334 wait complete. 2024-01-15T18:05:13.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.171+0000 7fc9cae00640 1 Processor -- start 2024-01-15T18:05:13.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.171+0000 7fc9cae00640 1 -- start start 2024-01-15T18:05:13.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.171+0000 7fc9cae00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc9c406c200 0x7fc9c413df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.171+0000 7fc9cae00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc9c40cad40 0x7fc9c415c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.171+0000 7fc9cae00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9c4143da0 0x7fc9c413c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.171+0000 7fc9cae00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c413daf0 con 0x7fc9c4143da0 2024-01-15T18:05:13.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.171+0000 7fc9cae00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c413cf50 con 0x7fc9c40cad40 2024-01-15T18:05:13.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.171+0000 7fc9cae00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c413ccb0 con 0x7fc9c406c200 2024-01-15T18:05:13.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.171+0000 7fc9c8e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9c4143da0 0x7fc9c413c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.171+0000 7fc9c8e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9c4143da0 0x7fc9c413c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59604/0 (socket says 172.21.15.27:59604) 2024-01-15T18:05:13.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.171+0000 7fc9c8e00640 1 -- 172.21.15.27:0/4044204388 learned_addr learned my addr 172.21.15.27:0/4044204388 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:13.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.171+0000 7fc9c3400640 1 --2- 172.21.15.27:0/4044204388 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc9c40cad40 0x7fc9c415c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.171+0000 7fc9c3e00640 1 --2- 172.21.15.27:0/4044204388 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc9c406c200 0x7fc9c413df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.171+0000 7fc9c8e00640 1 -- 172.21.15.27:0/4044204388 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc9c406c200 msgr2=0x7fc9c413df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.171+0000 7fc9c8e00640 1 --2- 172.21.15.27:0/4044204388 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc9c406c200 0x7fc9c413df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.171+0000 7fc9c8e00640 1 -- 172.21.15.27:0/4044204388 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc9c40cad40 msgr2=0x7fc9c415c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.171+0000 7fc9c8e00640 1 --2- 172.21.15.27:0/4044204388 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc9c40cad40 0x7fc9c415c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.171+0000 7fc9c8e00640 1 -- 172.21.15.27:0/4044204388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ac009420 con 0x7fc9c4143da0 2024-01-15T18:05:13.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.171+0000 7fc9c3e00640 1 --2- 172.21.15.27:0/4044204388 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc9c406c200 0x7fc9c413df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:13.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.171+0000 7fc9c8e00640 1 --2- 172.21.15.27:0/4044204388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9c4143da0 0x7fc9c413c770 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc00efc0 tx=0x7fc9bc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:13.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.171+0000 7fc9bbe00640 1 -- 172.21.15.27:0/4044204388 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc00cd20 con 0x7fc9c4143da0 2024-01-15T18:05:13.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.171+0000 7fc9bbe00640 1 -- 172.21.15.27:0/4044204388 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9bc00f040 con 0x7fc9c4143da0 2024-01-15T18:05:13.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.171+0000 7fc9bbe00640 1 -- 172.21.15.27:0/4044204388 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc003be0 con 0x7fc9c4143da0 2024-01-15T18:05:13.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.171+0000 7fc9cae00640 1 -- 172.21.15.27:0/4044204388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c413d1c0 con 0x7fc9c4143da0 2024-01-15T18:05:13.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.175+0000 7fc9cae00640 1 -- 172.21.15.27:0/4044204388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c413d780 con 0x7fc9c4143da0 2024-01-15T18:05:13.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.175+0000 7fc9bbe00640 1 -- 172.21.15.27:0/4044204388 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fc9bc003d40 con 0x7fc9c4143da0 2024-01-15T18:05:13.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.175+0000 7fc9bbe00640 1 --2- 172.21.15.27:0/4044204388 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc99c066920 0x7fc99c068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.175+0000 7fc9bbe00640 1 -- 172.21.15.27:0/4044204388 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9bc084a10 con 0x7fc9c4143da0 2024-01-15T18:05:13.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9c3e00640 1 --2- 172.21.15.27:0/4044204388 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc99c066920 0x7fc99c068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9ce1c7000 1 Processor -- start 2024-01-15T18:05:13.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9ce1c7000 1 -- start start 2024-01-15T18:05:13.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9c3e00640 1 --2- 172.21.15.27:0/4044204388 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc99c066920 0x7fc99c068de0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4000f70 tx=0x7fc9b4007cb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:13.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9ce1c7000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x56051495ac90 0x56051495b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9ce1c7000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x56051495b5b0 0x560514910de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9ce1c7000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560514911320 0x560514911700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9ce1c7000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x56051477b130 con 0x560514911320 2024-01-15T18:05:13.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9ce1c7000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605147689a0 con 0x56051495ac90 2024-01-15T18:05:13.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9ce1c7000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x560514765ed0 con 0x56051495b5b0 2024-01-15T18:05:13.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9c8e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560514911320 0x560514911700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9c8e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560514911320 0x560514911700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59614/0 (socket says 172.21.15.27:59614) 2024-01-15T18:05:13.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9c3400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x56051495b5b0 0x560514910de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9c8e00640 1 -- 172.21.15.27:0/2974825388 learned_addr learned my addr 172.21.15.27:0/2974825388 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:13.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9c3e00640 1 --2- 172.21.15.27:0/2974825388 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x56051495ac90 0x56051495b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9c8e00640 1 -- 172.21.15.27:0/2974825388 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x56051495b5b0 msgr2=0x560514910de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:13.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9c8e00640 1 --2- 172.21.15.27:0/2974825388 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x56051495b5b0 0x560514910de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9c8e00640 1 -- 172.21.15.27:0/2974825388 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x56051495ac90 msgr2=0x56051495b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:13.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9c8e00640 1 --2- 172.21.15.27:0/2974825388 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x56051495ac90 0x56051495b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9c8e00640 1 -- 172.21.15.27:0/2974825388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560514459890 con 0x560514911320 2024-01-15T18:05:13.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9c8e00640 1 --2- 172.21.15.27:0/2974825388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560514911320 0x560514911700 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc055890 tx=0x7fc9bc056000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afbf40c9ab729835 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:13.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9b8c00640 1 -- 172.21.15.27:0/2974825388 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc06d210 con 0x560514911320 2024-01-15T18:05:13.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9b8c00640 1 -- 172.21.15.27:0/2974825388 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9bc05dc00 con 0x560514911320 2024-01-15T18:05:13.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9b8c00640 1 -- 172.21.15.27:0/2974825388 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc052010 con 0x560514911320 2024-01-15T18:05:13.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/2974825388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560514911320 msgr2=0x560514911700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9ce1c7000 1 --2- 172.21.15.27:0/2974825388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560514911320 0x560514911700 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc055890 tx=0x7fc9bc056000 comp rx=0 tx=0).stop 2024-01-15T18:05:13.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/2974825388 shutdown_connections 2024-01-15T18:05:13.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9ce1c7000 1 --2- 172.21.15.27:0/2974825388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560514911320 0x560514911700 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9ce1c7000 1 --2- 172.21.15.27:0/2974825388 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x56051495b5b0 0x560514910de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9ce1c7000 1 --2- 172.21.15.27:0/2974825388 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x56051495ac90 0x56051495b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/2974825388 >> 172.21.15.27:0/2974825388 conn(0x56051490d2d0 msgr2=0x56051490f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:13.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/2974825388 shutdown_connections 2024-01-15T18:05:13.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.179+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/2974825388 wait complete. 2024-01-15T18:05:13.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9ce1c7000 1 -- start start 2024-01-15T18:05:13.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9ce1c7000 1 Processor -- start 2024-01-15T18:05:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9ce1c7000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x560514911320 0x56051493fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9ce1c7000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051495ac90 0x56051493fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9ce1c7000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x56051495b5b0 0x560514940530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9ce1c7000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605147617c0 con 0x56051495ac90 2024-01-15T18:05:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9ce1c7000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x560514719bf0 con 0x56051495b5b0 2024-01-15T18:05:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9ce1c7000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x56051471fa90 con 0x560514911320 2024-01-15T18:05:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9ce1c7000 10 client.0 _ll_register_callbacks cb 0x5605148fc3c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:05:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9c3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051495ac90 0x56051493fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9c3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051495ac90 0x56051493fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59622/0 (socket says 172.21.15.27:59622) 2024-01-15T18:05:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9c3400640 1 -- 172.21.15.27:0/1088535349 learned_addr learned my addr 172.21.15.27:0/1088535349 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9c3e00640 1 --2- 172.21.15.27:0/1088535349 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x560514911320 0x56051493fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9c8e00640 1 --2- 172.21.15.27:0/1088535349 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x56051495b5b0 0x560514940530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9c3400640 1 -- 172.21.15.27:0/1088535349 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x560514911320 msgr2=0x56051493fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9c3400640 1 --2- 172.21.15.27:0/1088535349 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x560514911320 0x56051493fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9c3400640 1 -- 172.21.15.27:0/1088535349 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x56051495b5b0 msgr2=0x560514940530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9c3400640 1 --2- 172.21.15.27:0/1088535349 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x56051495b5b0 0x560514940530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9c3400640 1 -- 172.21.15.27:0/1088535349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c413d780 con 0x56051495ac90 2024-01-15T18:05:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9c3400640 1 --2- 172.21.15.27:0/1088535349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051495ac90 0x56051493fff0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac0097c0 tx=0x7fc9ac01acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6453fa8df97ec634 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9b1600640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-15T18:05:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9b1600640 1 -- 172.21.15.27:0/1088535349 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac0abd60 con 0x56051495ac90 2024-01-15T18:05:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/1088535349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560514723470 con 0x56051495ac90 2024-01-15T18:05:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9b1600640 1 -- 172.21.15.27:0/1088535349 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9ac0c1450 con 0x56051495ac90 2024-01-15T18:05:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9b1600640 1 -- 172.21.15.27:0/1088535349 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9ac0c1610 con 0x56051495ac90 2024-01-15T18:05:13.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9ce1c7000 10 client.4653 fetch_fsmap learned FSMap version 8 2024-01-15T18:05:13.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/1088535349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x560514722190 con 0x56051495ac90 2024-01-15T18:05:13.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9b1600640 1 -- 172.21.15.27:0/1088535349 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x560514723470 con 0x56051495ac90 2024-01-15T18:05:13.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9b1600640 1 -- 172.21.15.27:0/1088535349 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc9ac0c1770 con 0x56051495ac90 2024-01-15T18:05:13.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9ce1c7000 10 client.4653 fetch_fsmap finished waiting for FSMap version 8 2024-01-15T18:05:13.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9ce1c7000 10 client.4653 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-15T18:05:13.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9ce1c7000 20 client.4653 populate_metadata read hostname 'smithi027' 2024-01-15T18:05:13.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9ce1c7000 1 --2- 172.21.15.27:0/1088535349 >> [v2:172.21.15.27:6832/2657553192,v1:172.21.15.27:6834/2657553192] conn(0x560514940f40 0x56051496cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9ce1c7000 4 client.4653 mds_command: new command op to 4308 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-15T18:05:13.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.183+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/1088535349 --> [v2:172.21.15.27:6832/2657553192,v1:172.21.15.27:6834/2657553192] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560514726b50 con 0x560514940f40 2024-01-15T18:05:13.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.187+0000 7fc9c3e00640 1 --2- 172.21.15.27:0/1088535349 >> [v2:172.21.15.27:6832/2657553192,v1:172.21.15.27:6834/2657553192] conn(0x560514940f40 0x56051496cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.187+0000 7fc9c3e00640 1 --2- 172.21.15.27:0/1088535349 >> [v2:172.21.15.27:6832/2657553192,v1:172.21.15.27:6834/2657553192] conn(0x560514940f40 0x56051496cd40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b2b86dd009a6d2e server_cookie=df9051d3f99a9917 in_seq=0 out_seq=0 2024-01-15T18:05:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.187+0000 7fc9b1600640 10 client.4653 ms_handle_connect on v2:172.21.15.27:6832/2657553192 2024-01-15T18:05:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.187+0000 7fc9b1600640 1 -- 172.21.15.27:0/1088535349 <== mds.0 v2:172.21.15.27:6832/2657553192 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560514726b50 con 0x560514940f40 2024-01-15T18:05:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.187+0000 7fc9b1600640 10 client.4653 handle_command_reply: tid=0 2024-01-15T18:05:13.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.187+0000 7fc9ce1c7000 1 client.4653 shutdown 2024-01-15T18:05:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.187+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/1088535349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051495ac90 msgr2=0x56051493fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.187+0000 7fc9ce1c7000 1 --2- 172.21.15.27:0/1088535349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051495ac90 0x56051493fff0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac0097c0 tx=0x7fc9ac01acc0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.187+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/1088535349 shutdown_connections 2024-01-15T18:05:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.187+0000 7fc9ce1c7000 1 --2- 172.21.15.27:0/1088535349 >> [v2:172.21.15.27:6832/2657553192,v1:172.21.15.27:6834/2657553192] conn(0x560514940f40 0x56051496cd40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.187+0000 7fc9ce1c7000 1 --2- 172.21.15.27:0/1088535349 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x56051495b5b0 0x560514940530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.187+0000 7fc9ce1c7000 1 --2- 172.21.15.27:0/1088535349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051495ac90 0x56051493fff0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.187+0000 7fc9b1600640 0 client.4653 ms_handle_reset on v2:172.21.15.27:6832/2657553192 2024-01-15T18:05:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.187+0000 7fc9ce1c7000 1 --2- 172.21.15.27:0/1088535349 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x560514911320 0x56051493fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.187+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/1088535349 >> 172.21.15.27:0/1088535349 conn(0x56051490d2d0 msgr2=0x56051490f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.187+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/1088535349 shutdown_connections 2024-01-15T18:05:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.187+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/1088535349 wait complete. 2024-01-15T18:05:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.187+0000 7fc9ce1c7000 20 client.4653 trim_cache size 0 max 16384 2024-01-15T18:05:13.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.203+0000 7fc9ce1c7000 1 Processor -- start 2024-01-15T18:05:13.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.203+0000 7fc9ce1c7000 1 -- start start 2024-01-15T18:05:13.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.203+0000 7fc9ce1c7000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x56051490d2d0 0x5605149593e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.203+0000 7fc9ce1c7000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560514911320 0x560514959920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.203+0000 7fc9ce1c7000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x56051495ac90 0x560514912920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.203+0000 7fc9ce1c7000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x56051490d190 con 0x560514911320 2024-01-15T18:05:13.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.203+0000 7fc9ce1c7000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x56051441a940 con 0x56051490d2d0 2024-01-15T18:05:13.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.203+0000 7fc9ce1c7000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x560514764320 con 0x56051495ac90 2024-01-15T18:05:13.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.203+0000 7fc9c3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560514911320 0x560514959920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.203+0000 7fc9c3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560514911320 0x560514959920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59628/0 (socket says 172.21.15.27:59628) 2024-01-15T18:05:13.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.203+0000 7fc9c3400640 1 -- 172.21.15.27:0/3796210163 learned_addr learned my addr 172.21.15.27:0/3796210163 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:13.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.203+0000 7fc9c3e00640 1 --2- 172.21.15.27:0/3796210163 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x56051490d2d0 0x5605149593e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.203+0000 7fc9c8e00640 1 --2- 172.21.15.27:0/3796210163 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x56051495ac90 0x560514912920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.203+0000 7fc9c3400640 1 -- 172.21.15.27:0/3796210163 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x56051495ac90 msgr2=0x560514912920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:13.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.203+0000 7fc9c3400640 1 --2- 172.21.15.27:0/3796210163 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x56051495ac90 0x560514912920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.203+0000 7fc9c3400640 1 -- 172.21.15.27:0/3796210163 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x56051490d2d0 msgr2=0x5605149593e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:13.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.203+0000 7fc9c3400640 1 --2- 172.21.15.27:0/3796210163 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x56051490d2d0 0x5605149593e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.203+0000 7fc9c3400640 1 -- 172.21.15.27:0/3796210163 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560514708490 con 0x560514911320 2024-01-15T18:05:13.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.203+0000 7fc9c3e00640 1 --2- 172.21.15.27:0/3796210163 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x56051490d2d0 0x5605149593e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:13.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9c3400640 1 --2- 172.21.15.27:0/3796210163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560514911320 0x560514959920 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac01ad00 tx=0x7fc9ac0b8040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28558c1def6e8758 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:13.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9b9600640 1 -- 172.21.15.27:0/3796210163 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac0c1650 con 0x560514911320 2024-01-15T18:05:13.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9b9600640 1 -- 172.21.15.27:0/3796210163 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9ac0c17b0 con 0x560514911320 2024-01-15T18:05:13.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9b9600640 1 -- 172.21.15.27:0/3796210163 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac004280 con 0x560514911320 2024-01-15T18:05:13.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/3796210163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560514911320 msgr2=0x560514959920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9ce1c7000 1 --2- 172.21.15.27:0/3796210163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560514911320 0x560514959920 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac01ad00 tx=0x7fc9ac0b8040 comp rx=0 tx=0).stop 2024-01-15T18:05:13.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/3796210163 shutdown_connections 2024-01-15T18:05:13.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9ce1c7000 1 --2- 172.21.15.27:0/3796210163 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x56051495ac90 0x560514912920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9ce1c7000 1 --2- 172.21.15.27:0/3796210163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x560514911320 0x560514959920 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9ce1c7000 1 --2- 172.21.15.27:0/3796210163 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x56051490d2d0 0x5605149593e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/3796210163 >> 172.21.15.27:0/3796210163 conn(0x56051493daa0 msgr2=0x56051490f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:13.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/3796210163 shutdown_connections 2024-01-15T18:05:13.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/3796210163 wait complete. 2024-01-15T18:05:13.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9ce1c7000 1 -- start start 2024-01-15T18:05:13.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9ce1c7000 1 Processor -- start 2024-01-15T18:05:13.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9ce1c7000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051490d2d0 0x56051496e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9ce1c7000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x560514911320 0x56051497cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9ce1c7000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x56051495ac90 0x56051497d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9ce1c7000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605147208f0 con 0x56051490d2d0 2024-01-15T18:05:13.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9ce1c7000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x560514916fb0 con 0x560514911320 2024-01-15T18:05:13.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9ce1c7000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x56051475d130 con 0x56051495ac90 2024-01-15T18:05:13.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9ce1c7000 10 client.0 _ll_register_callbacks cb 0x5605147afc00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:05:13.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9c3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051490d2d0 0x56051496e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9c3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051490d2d0 0x56051496e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59640/0 (socket says 172.21.15.27:59640) 2024-01-15T18:05:13.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9c3e00640 1 -- 172.21.15.27:0/680904809 learned_addr learned my addr 172.21.15.27:0/680904809 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:13.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9c3400640 1 --2- 172.21.15.27:0/680904809 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x560514911320 0x56051497cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9c8e00640 1 --2- 172.21.15.27:0/680904809 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x56051495ac90 0x56051497d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9c3e00640 1 -- 172.21.15.27:0/680904809 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x56051495ac90 msgr2=0x56051497d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:13.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9c3e00640 1 --2- 172.21.15.27:0/680904809 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x56051495ac90 0x56051497d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9c3e00640 1 -- 172.21.15.27:0/680904809 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x560514911320 msgr2=0x56051497cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:13.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9c3e00640 1 --2- 172.21.15.27:0/680904809 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x560514911320 0x56051497cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.207+0000 7fc9c3e00640 1 -- 172.21.15.27:0/680904809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560514722190 con 0x56051490d2d0 2024-01-15T18:05:13.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9c3e00640 1 --2- 172.21.15.27:0/680904809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051490d2d0 0x56051496e180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc9b40018b0 tx=0x7fc9b4000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4654d7b1bee77c8c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:13.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9b1600640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-15T18:05:13.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9b1600640 1 -- 172.21.15.27:0/680904809 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b40277e0 con 0x56051490d2d0 2024-01-15T18:05:13.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9b1600640 1 -- 172.21.15.27:0/680904809 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9b4023420 con 0x56051490d2d0 2024-01-15T18:05:13.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/680904809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560514760110 con 0x56051490d2d0 2024-01-15T18:05:13.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9b1600640 1 -- 172.21.15.27:0/680904809 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9b40235e0 con 0x56051490d2d0 2024-01-15T18:05:13.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9ce1c7000 10 client.4659 fetch_fsmap learned FSMap version 8 2024-01-15T18:05:13.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/680904809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x560514459cc0 con 0x56051490d2d0 2024-01-15T18:05:13.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9b1600640 1 -- 172.21.15.27:0/680904809 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x560514760110 con 0x56051490d2d0 2024-01-15T18:05:13.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9b1600640 1 -- 172.21.15.27:0/680904809 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc9b4020070 con 0x56051490d2d0 2024-01-15T18:05:13.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9ce1c7000 10 client.4659 fetch_fsmap finished waiting for FSMap version 8 2024-01-15T18:05:13.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9ce1c7000 10 client.4659 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-15T18:05:13.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9ce1c7000 20 client.4659 populate_metadata read hostname 'smithi027' 2024-01-15T18:05:13.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9ce1c7000 1 --2- 172.21.15.27:0/680904809 >> [v2:172.21.15.27:6832/2657553192,v1:172.21.15.27:6834/2657553192] conn(0x56051495b5b0 0x5605149c0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9ce1c7000 4 client.4659 mds_command: new command op to 4308 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-15T18:05:13.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/680904809 --> [v2:172.21.15.27:6832/2657553192,v1:172.21.15.27:6834/2657553192] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56051478f9a0 con 0x56051495b5b0 2024-01-15T18:05:13.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9c3400640 1 --2- 172.21.15.27:0/680904809 >> [v2:172.21.15.27:6832/2657553192,v1:172.21.15.27:6834/2657553192] conn(0x56051495b5b0 0x5605149c0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9c3400640 1 --2- 172.21.15.27:0/680904809 >> [v2:172.21.15.27:6832/2657553192,v1:172.21.15.27:6834/2657553192] conn(0x56051495b5b0 0x5605149c0010 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11b6bdd1a1f566e1 server_cookie=35f388eb37789cd in_seq=0 out_seq=0 2024-01-15T18:05:13.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9b1600640 10 client.4659 ms_handle_connect on v2:172.21.15.27:6832/2657553192 2024-01-15T18:05:13.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9b1600640 1 -- 172.21.15.27:0/680904809 <== mds.0 v2:172.21.15.27:6832/2657553192 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x56051478f9a0 con 0x56051495b5b0 2024-01-15T18:05:13.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9b1600640 10 client.4659 handle_command_reply: tid=0 2024-01-15T18:05:13.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9ce1c7000 1 client.4659 shutdown 2024-01-15T18:05:13.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/680904809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051490d2d0 msgr2=0x56051496e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9ce1c7000 1 --2- 172.21.15.27:0/680904809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051490d2d0 0x56051496e180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc9b40018b0 tx=0x7fc9b4000ee0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/680904809 shutdown_connections 2024-01-15T18:05:13.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9ce1c7000 1 --2- 172.21.15.27:0/680904809 >> [v2:172.21.15.27:6832/2657553192,v1:172.21.15.27:6834/2657553192] conn(0x56051495b5b0 0x5605149c0010 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9ce1c7000 1 --2- 172.21.15.27:0/680904809 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x56051495ac90 0x56051497d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9ce1c7000 1 --2- 172.21.15.27:0/680904809 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x560514911320 0x56051497cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9b1600640 0 client.4659 ms_handle_reset on v2:172.21.15.27:6832/2657553192 2024-01-15T18:05:13.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9ce1c7000 1 --2- 172.21.15.27:0/680904809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56051490d2d0 0x56051496e180 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/680904809 >> 172.21.15.27:0/680904809 conn(0x56051493daa0 msgr2=0x56051490ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:13.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/680904809 shutdown_connections 2024-01-15T18:05:13.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9ce1c7000 1 -- 172.21.15.27:0/680904809 wait complete. 2024-01-15T18:05:13.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9ce1c7000 20 client.4659 trim_cache size 0 max 16384 2024-01-15T18:05:13.223 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:05:13.223 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:05:13.224 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-01-15T18:05:13.224 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4308, 2024-01-15T18:05:13.224 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-01-15T18:05:13.224 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-01-15T18:05:13.224 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-01-15T18:05:13.224 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 14.376668891, 2024-01-15T18:05:13.224 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 8, 2024-01-15T18:05:13.224 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 21, 2024-01-15T18:05:13.224 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 21, 2024-01-15T18:05:13.224 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 21.728533076000002 2024-01-15T18:05:13.224 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:05:13.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9cae00640 1 -- 172.21.15.27:0/4044204388 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc99c066920 msgr2=0x7fc99c068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9cae00640 1 --2- 172.21.15.27:0/4044204388 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc99c066920 0x7fc99c068de0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4000f70 tx=0x7fc9b4007cb0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9cae00640 1 -- 172.21.15.27:0/4044204388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9c4143da0 msgr2=0x7fc9c413c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.211+0000 7fc9cae00640 1 --2- 172.21.15.27:0/4044204388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9c4143da0 0x7fc9c413c770 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc00efc0 tx=0x7fc9bc00c490 comp rx=0 tx=0).stop 2024-01-15T18:05:13.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.215+0000 7fc9cae00640 1 -- 172.21.15.27:0/4044204388 shutdown_connections 2024-01-15T18:05:13.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.215+0000 7fc9cae00640 1 --2- 172.21.15.27:0/4044204388 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc99c066920 0x7fc99c068de0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.215+0000 7fc9cae00640 1 --2- 172.21.15.27:0/4044204388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9c4143da0 0x7fc9c413c770 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.215+0000 7fc9cae00640 1 --2- 172.21.15.27:0/4044204388 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc9c40cad40 0x7fc9c415c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.215+0000 7fc9cae00640 1 --2- 172.21.15.27:0/4044204388 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc9c406c200 0x7fc9c413df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.215+0000 7fc9cae00640 1 -- 172.21.15.27:0/4044204388 >> 172.21.15.27:0/4044204388 conn(0x7fc9c40b0850 msgr2=0x7fc9c4142dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:13.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.215+0000 7fc9cae00640 1 -- 172.21.15.27:0/4044204388 shutdown_connections 2024-01-15T18:05:13.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.215+0000 7fc9cae00640 1 -- 172.21.15.27:0/4044204388 wait complete. 2024-01-15T18:05:13.239 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-01-15T18:05:13.268 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-01-15T18:05:13.268 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-01-15T18:05:13.268 INFO:tasks.ceph_fuse:client.0 config is {} 2024-01-15T18:05:13.268 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-15T18:05:13.269 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-15T18:05:13.269 INFO:tasks.ceph_fuse:client.2 config is {} 2024-01-15T18:05:13.269 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-15T18:05:13.269 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-01-15T18:05:13.269 INFO:tasks.ceph_fuse:client.1 config is {} 2024-01-15T18:05:13.269 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-15T18:05:13.269 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-15T18:05:13.269 INFO:tasks.ceph_fuse:client.3 config is {} 2024-01-15T18:05:13.269 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-15T18:05:13.269 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-01-15T18:05:13.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:13.270 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-15T18:05:13.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:13.280 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link delete ceph-brx 2024-01-15T18:05:13.345 INFO:teuthology.orchestra.run.smithi027.stderr:Cannot find device "ceph-brx" 2024-01-15T18:05:13.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:05:13.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:13.348 DEBUG:teuthology.orchestra.run.smithi093:> ip netns list 2024-01-15T18:05:13.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:13.358 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip link delete ceph-brx 2024-01-15T18:05:13.425 INFO:teuthology.orchestra.run.smithi093.stderr:Cannot find device "ceph-brx" 2024-01-15T18:05:13.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:05:13.428 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-01-15T18:05:13.428 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:05:13.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.563+0000 7f5d9d200640 1 Processor -- start 2024-01-15T18:05:13.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.563+0000 7f5d9d200640 1 -- start start 2024-01-15T18:05:13.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.563+0000 7f5d9d200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5d980b1e50 0x7f5d9806b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.563+0000 7f5d9d200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d98161d10 con 0x7f5d980c3e90 2024-01-15T18:05:13.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.563+0000 7f5d9d200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d98165de0 con 0x7f5d980b23e0 2024-01-15T18:05:13.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.563+0000 7f5d9d200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d980cf260 con 0x7f5d980b1e50 2024-01-15T18:05:13.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.563+0000 7f5d96800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5d980b1e50 0x7f5d9806b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.563+0000 7f5d95e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5d980c3e90 0x7f5d980c4270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42504/0 (socket says 172.21.15.27:42504) 2024-01-15T18:05:13.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.563+0000 7f5d95e00640 1 -- 172.21.15.27:0/3748192911 learned_addr learned my addr 172.21.15.27:0/3748192911 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:13.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.563+0000 7f5d96800640 1 -- 172.21.15.27:0/3748192911 >> v1:172.21.15.93:6789/0 conn(0x7f5d980b23e0 legacy=0x7f5d980cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.563+0000 7f5d96800640 1 -- 172.21.15.27:0/3748192911 >> v1:172.21.15.27:6789/0 conn(0x7f5d980c3e90 legacy=0x7f5d980c4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.567+0000 7f5d94a00640 1 -- 172.21.15.27:0/3748192911 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3037906938 0 0) 0x7f5d88001120 con 0x7f5d980c3e90 2024-01-15T18:05:13.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.567+0000 7f5d96800640 1 -- 172.21.15.27:0/3748192911 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d98166310 con 0x7f5d980b1e50 2024-01-15T18:05:13.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.567+0000 7f5d96800640 1 --2- 172.21.15.27:0/3748192911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d980b1e50 0x7f5d9806b600 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5d90009d90 tx=0x7f5d9001c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a1b6f4995e1242d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:13.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.567+0000 7f5d94a00640 1 -- 172.21.15.27:0/3748192911 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d900a58d0 con 0x7f5d980b1e50 2024-01-15T18:05:13.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.567+0000 7f5d94a00640 1 -- 172.21.15.27:0/3748192911 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d900a5a30 con 0x7f5d980b1e50 2024-01-15T18:05:13.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.567+0000 7f5d94a00640 1 -- 172.21.15.27:0/3748192911 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d900a5dc0 con 0x7f5d980b1e50 2024-01-15T18:05:13.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.567+0000 7f5d9d200640 1 -- 172.21.15.27:0/3748192911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d980b1e50 msgr2=0x7f5d9806b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.567+0000 7f5d9d200640 1 --2- 172.21.15.27:0/3748192911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d980b1e50 0x7f5d9806b600 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5d90009d90 tx=0x7f5d9001c860 comp rx=0 tx=0).stop 2024-01-15T18:05:13.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.571+0000 7f5d9d200640 1 -- 172.21.15.27:0/3748192911 shutdown_connections 2024-01-15T18:05:13.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.571+0000 7f5d9d200640 1 --2- 172.21.15.27:0/3748192911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d980b1e50 0x7f5d9806b600 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.571+0000 7f5d9d200640 1 -- 172.21.15.27:0/3748192911 >> 172.21.15.27:0/3748192911 conn(0x7f5d980b2a10 msgr2=0x7f5d980b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:13.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.571+0000 7f5d9d200640 1 -- 172.21.15.27:0/3748192911 shutdown_connections 2024-01-15T18:05:13.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.571+0000 7f5d9d200640 1 -- 172.21.15.27:0/3748192911 wait complete. 2024-01-15T18:05:13.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.571+0000 7f5d9d200640 1 Processor -- start 2024-01-15T18:05:13.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.571+0000 7f5d9d200640 1 -- start start 2024-01-15T18:05:13.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.571+0000 7f5d9d200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5d980b1e50 0x7f5d980c8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.571+0000 7f5d9d200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5d980b23e0 0x7f5d9815b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.571+0000 7f5d9d200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d980c3e90 0x7f5d981598b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.571+0000 7f5d9d200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d981551b0 con 0x7f5d980c3e90 2024-01-15T18:05:13.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.571+0000 7f5d9d200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d981519f0 con 0x7f5d980b23e0 2024-01-15T18:05:13.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.571+0000 7f5d9d200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d980c9b20 con 0x7f5d980b1e50 2024-01-15T18:05:13.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.571+0000 7f5d96800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d980c3e90 0x7f5d981598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.571+0000 7f5d96800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d980c3e90 0x7f5d981598b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59648/0 (socket says 172.21.15.27:59648) 2024-01-15T18:05:13.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.571+0000 7f5d96800640 1 -- 172.21.15.27:0/2669620023 learned_addr learned my addr 172.21.15.27:0/2669620023 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:13.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.571+0000 7f5d95400640 1 --2- 172.21.15.27:0/2669620023 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5d980b23e0 0x7f5d9815b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.571+0000 7f5d95e00640 1 --2- 172.21.15.27:0/2669620023 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5d980b1e50 0x7f5d980c8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.571+0000 7f5d96800640 1 -- 172.21.15.27:0/2669620023 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5d980b1e50 msgr2=0x7f5d980c8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.571+0000 7f5d96800640 1 --2- 172.21.15.27:0/2669620023 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5d980b1e50 0x7f5d980c8da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.571+0000 7f5d96800640 1 -- 172.21.15.27:0/2669620023 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5d980b23e0 msgr2=0x7f5d9815b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.571+0000 7f5d96800640 1 --2- 172.21.15.27:0/2669620023 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5d980b23e0 0x7f5d9815b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.571+0000 7f5d96800640 1 -- 172.21.15.27:0/2669620023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d90009a70 con 0x7f5d980c3e90 2024-01-15T18:05:13.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.571+0000 7f5d95e00640 1 --2- 172.21.15.27:0/2669620023 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5d980b1e50 0x7f5d980c8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:13.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.571+0000 7f5d96800640 1 --2- 172.21.15.27:0/2669620023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d980c3e90 0x7f5d981598b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5d90019470 tx=0x7f5d90019600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:13.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.575+0000 7f5d8e000640 1 -- 172.21.15.27:0/2669620023 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d90003ba0 con 0x7f5d980c3e90 2024-01-15T18:05:13.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.575+0000 7f5d8e000640 1 -- 172.21.15.27:0/2669620023 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d90003d00 con 0x7f5d980c3e90 2024-01-15T18:05:13.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.575+0000 7f5d8e000640 1 -- 172.21.15.27:0/2669620023 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d900ad850 con 0x7f5d980c3e90 2024-01-15T18:05:13.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.575+0000 7f5d9d200640 1 -- 172.21.15.27:0/2669620023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d980c92e0 con 0x7f5d980c3e90 2024-01-15T18:05:13.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.575+0000 7f5d9d200640 1 -- 172.21.15.27:0/2669620023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d980c9840 con 0x7f5d980c3e90 2024-01-15T18:05:13.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.575+0000 7f5d9d200640 1 -- 172.21.15.27:0/2669620023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d9804e850 con 0x7f5d980c3e90 2024-01-15T18:05:13.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.575+0000 7f5d8e000640 1 -- 172.21.15.27:0/2669620023 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f5d900b4070 con 0x7f5d980c3e90 2024-01-15T18:05:13.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.575+0000 7f5d8e000640 1 --2- 172.21.15.27:0/2669620023 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5d700668f0 0x7f5d70068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.575+0000 7f5d8e000640 1 -- 172.21.15.27:0/2669620023 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5d9011f250 con 0x7f5d980c3e90 2024-01-15T18:05:13.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.575+0000 7f5d95e00640 1 --2- 172.21.15.27:0/2669620023 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5d700668f0 0x7f5d70068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.579+0000 7f5d95e00640 1 --2- 172.21.15.27:0/2669620023 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5d700668f0 0x7f5d70068db0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5d88000f70 tx=0x7f5d88007b00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:13.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.579+0000 7f5d8e000640 1 -- 172.21.15.27:0/2669620023 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5d900ec3f0 con 0x7f5d980c3e90 2024-01-15T18:05:13.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.771+0000 7f5d9d200640 1 -- 172.21.15.27:0/2669620023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5d980b6e40 con 0x7f5d980c3e90 2024-01-15T18:05:13.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.771+0000 7f5d8e000640 1 -- 172.21.15.27:0/2669620023 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5d900efa10 con 0x7f5d980c3e90 2024-01-15T18:05:13.775 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:05:13.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.771+0000 7f5d9d200640 1 -- 172.21.15.27:0/2669620023 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5d700668f0 msgr2=0x7f5d70068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.771+0000 7f5d9d200640 1 --2- 172.21.15.27:0/2669620023 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5d700668f0 0x7f5d70068db0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5d88000f70 tx=0x7f5d88007b00 comp rx=0 tx=0).stop 2024-01-15T18:05:13.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.771+0000 7f5d9d200640 1 -- 172.21.15.27:0/2669620023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d980c3e90 msgr2=0x7f5d981598b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.771+0000 7f5d9d200640 1 --2- 172.21.15.27:0/2669620023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d980c3e90 0x7f5d981598b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5d90019470 tx=0x7f5d90019600 comp rx=0 tx=0).stop 2024-01-15T18:05:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.771+0000 7f5d9d200640 1 -- 172.21.15.27:0/2669620023 shutdown_connections 2024-01-15T18:05:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.771+0000 7f5d9d200640 1 --2- 172.21.15.27:0/2669620023 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5d700668f0 0x7f5d70068db0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.771+0000 7f5d9d200640 1 --2- 172.21.15.27:0/2669620023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d980c3e90 0x7f5d981598b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.771+0000 7f5d9d200640 1 --2- 172.21.15.27:0/2669620023 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5d980b23e0 0x7f5d9815b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.771+0000 7f5d9d200640 1 --2- 172.21.15.27:0/2669620023 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5d980b1e50 0x7f5d980c8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.771+0000 7f5d9d200640 1 -- 172.21.15.27:0/2669620023 >> 172.21.15.27:0/2669620023 conn(0x7f5d980b2a10 msgr2=0x7f5d980ccd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.771+0000 7f5d9d200640 1 -- 172.21.15.27:0/2669620023 shutdown_connections 2024-01-15T18:05:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:13.771+0000 7f5d9d200640 1 -- 172.21.15.27:0/2669620023 wait complete. 2024-01-15T18:05:13.796 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:05:13.797 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:05:13.797 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-01-15T18:05:13.797 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-15T18:05:13.797 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-15T18:05:13.797 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:05:13.797 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:05:13.797 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:05:13.797 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:05:13.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:13.797 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-01-15T18:05:13.806 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:05:13.806 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:05:13.806 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:05:13.806 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:13.806 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-15T18:05:13.806 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:13.806 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:05:13.807 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:13.807 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-01-15T18:05:13.807 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:05:13.807 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:13.807 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-01-15T18:05:13.807 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:05:13.807 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:13.807 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-01-15T18:05:13.807 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-01-15T18:05:13.807 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42445sec preferred_lft 42445sec 2024-01-15T18:05:13.807 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-01-15T18:05:13.807 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:13.807 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:05:13.808 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:13.808 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-01-15T18:05:13.808 INFO:teuthology.orchestra.run.smithi027.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-01-15T18:05:13.808 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 02:42:65:20:14:72 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:13.808 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-01-15T18:05:13.808 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:13.809 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-15T18:05:13.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:13.809 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-15T18:05:13.809 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-15T18:05:13.810 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add name ceph-brx type bridge 2024-01-15T18:05:13.810 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr flush dev ceph-brx 2024-01-15T18:05:13.810 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set ceph-brx up 2024-01-15T18:05:13.810 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-15T18:05:13.810 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-15T18:05:13.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:13.917 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:05:13.936 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-01-15T18:05:13.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:13.939 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-01-15T18:05:14.026 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-01-15T18:05:14.026 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:05:14.026 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-01-15T18:05:14.026 INFO:teuthology.orchestra.run.smithi027.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-01-15T18:05:14.026 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-01-15T18:05:14.026 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:05:14.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:14.027 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-15T18:05:14.027 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-15T18:05:14.027 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-15T18:05:14.027 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-15T18:05:14.027 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-15T18:05:14.028 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-15T18:05:14.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:14.120 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-15T18:05:14.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:14.130 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list-id 2024-01-15T18:05:14.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:14.183 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-15T18:05:14.183 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-15T18:05:14.183 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:05:14.184 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-01-15T18:05:14.184 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-15T18:05:14.265 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-01-15T18:05:14.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:14.266 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-15T18:05:14.266 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-15T18:05:14.266 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-01-15T18:05:14.266 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-15T18:05:14.266 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-01-15T18:05:14.266 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-01-15T18:05:14.266 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-01-15T18:05:14.266 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-15T18:05:14.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:14.429 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-15T18:05:14.429 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-15T18:05:14.429 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-01-15T18:05:14.429 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set dev brx.0 master ceph-brx 2024-01-15T18:05:14.429 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-15T18:05:14.470 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-15T18:05:14.470 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:05:14.470 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-15T18:05:14.527 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:05:14.527 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:05:14.527 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:05:14.582 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-01-15T18:05:14.647 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:05:14.699 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-01-15T18:05:14.699 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-01-15T18:05:14.700 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-01-15T18:05:14.700 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-01-15T18:05:14.700 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-01-15T18:05:14.700 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-01-15T18:05:14.700 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-01-15T18:05:14.700 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-01-15T18:05:14.700 INFO:teuthology.orchestra.run.smithi027.stdout:/run/lock 2024-01-15T18:05:14.700 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-01-15T18:05:14.700 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-01-15T18:05:14.700 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-01-15T18:05:14.701 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:05:14.701 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-01-15T18:05:14.701 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-01-15T18:05:14.701 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-01-15T18:05:14.701 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-01-15T18:05:14.701 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/fs/nfsd 2024-01-15T18:05:14.701 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-01-15T18:05:14.701 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-01-15T18:05:14.701 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:05:14.701 INFO:teuthology.orchestra.run.smithi027.stdout:/run/rpc_pipefs 2024-01-15T18:05:14.701 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-01-15T18:05:14.701 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-01-15T18:05:14.702 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:05:14.702 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:05:14.702 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:05:14.702 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:05:14.702 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-01-15T18:05:14.702 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:05:14.702 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:05:14.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:14.703 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-01-15T18:05:14.758 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:05:14.758 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-15T18:05:14.809 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-01-15T18:05:14.829 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:05:14.884 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-01-15T18:05:14.884 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-01-15T18:05:14.884 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-01-15T18:05:14.884 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-01-15T18:05:14.884 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-01-15T18:05:14.885 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-01-15T18:05:14.885 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-01-15T18:05:14.885 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-01-15T18:05:14.885 INFO:teuthology.orchestra.run.smithi027.stdout:/run/lock 2024-01-15T18:05:14.885 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-01-15T18:05:14.885 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-01-15T18:05:14.885 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-01-15T18:05:14.885 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:05:14.885 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-01-15T18:05:14.885 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-01-15T18:05:14.886 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-01-15T18:05:14.886 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-01-15T18:05:14.886 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/fs/nfsd 2024-01-15T18:05:14.886 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-01-15T18:05:14.886 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-01-15T18:05:14.886 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:05:14.886 INFO:teuthology.orchestra.run.smithi027.stdout:/run/rpc_pipefs 2024-01-15T18:05:14.886 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-01-15T18:05:14.886 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-01-15T18:05:14.886 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:05:14.887 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:05:14.887 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:05:14.887 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:05:14.887 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-01-15T18:05:14.887 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:05:14.887 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:05:14.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:14.888 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-01-15T18:05:14.920 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:2024-01-15T18:05:14.915+0000 7f6f5ca46580 -1 init, newargv = 0x55e2e96cb2c0 newargc=15 2024-01-15T18:05:14.920 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[33779]: starting ceph client 2024-01-15T18:05:14.928 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[33779]: starting fuse 2024-01-15T18:05:14.929 INFO:teuthology.orchestra.run.smithi027.stdout:43 2024-01-15T18:05:14.930 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-01-15T18:05:14.930 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:05:14.930 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-01-15T18:05:14.930 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-01-15T18:05:14.930 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-01-15T18:05:14.930 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-01-15T18:05:14.931 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:05:14.931 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-01-15T18:05:14.931 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-15T18:05:14.931 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-01-15T18:05:14.931 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-15T18:05:14.931 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:05:14.931 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-01-15T18:05:14.931 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-01-15T18:05:14.931 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-01-15T18:05:14.931 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-01-15T18:05:14.931 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:05:14.931 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-01-15T18:05:14.932 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:05:14.932 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:05:14.932 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:05:14.932 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-01-15T18:05:14.932 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-01-15T18:05:14.932 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-01-15T18:05:14.932 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:05:14.932 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:05:14.932 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.0")) 2024-01-15T18:05:14.932 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-01-15T18:05:15.056 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.0.33779.asok 2024-01-15T18:05:15.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:05:15.068 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.33779.asok 2024-01-15T18:05:15.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:15.069 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.33779.asok status 2024-01-15T18:05:15.241 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:05:15.241 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-01-15T18:05:15.241 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:05:15.241 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:05:15.241 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "0", 2024-01-15T18:05:15.241 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-01-15T18:05:15.241 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-15T18:05:15.242 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "33779", 2024-01-15T18:05:15.242 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-01-15T18:05:15.242 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:15.242 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-01-15T18:05:15.242 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-01-15T18:05:15.242 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4677, 2024-01-15T18:05:15.242 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-01-15T18:05:15.242 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-01-15T18:05:15.242 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-01-15T18:05:15.242 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 4677 2024-01-15T18:05:15.243 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:15.243 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-01-15T18:05:15.243 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-01-15T18:05:15.243 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-01-15T18:05:15.243 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132683226 2024-01-15T18:05:15.243 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:15.243 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:15.243 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-01-15T18:05:15.243 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-01-15T18:05:15.244 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:0", 2024-01-15T18:05:15.244 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1132683226 2024-01-15T18:05:15.244 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:15.244 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.4677 172.21.15.27:0/1132683226", 2024-01-15T18:05:15.244 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "172.21.15.27:0/1132683226", 2024-01-15T18:05:15.244 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-01-15T18:05:15.244 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 8, 2024-01-15T18:05:15.244 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 21, 2024-01-15T18:05:15.244 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:05:15.245 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-01-15T18:05:15.245 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-01-15T18:05:15.245 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:05:15.256 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:05:15.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.391+0000 7f288ba00640 1 Processor -- start 2024-01-15T18:05:15.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.391+0000 7f288ba00640 1 -- start start 2024-01-15T18:05:15.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.391+0000 7f288ba00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f28840b2390 0x7f28840cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:15.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.391+0000 7f288ba00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28840b2770 con 0x7f288406bb80 2024-01-15T18:05:15.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.391+0000 7f288ba00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f288406c060 con 0x7f28840cdf90 2024-01-15T18:05:15.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.391+0000 7f288ba00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f288406c340 con 0x7f28840b2390 2024-01-15T18:05:15.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.395+0000 7f2889000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f28840b2390 0x7f28840cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:15.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.395+0000 7f2883e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f288406bb80 0x7f28840cd880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34074/0 (socket says 172.21.15.27:34074) 2024-01-15T18:05:15.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.395+0000 7f2883e00640 1 -- 172.21.15.27:0/510604150 learned_addr learned my addr 172.21.15.27:0/510604150 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:15.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.395+0000 7f2889000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f28840b2390 0x7f28840cf780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39454/0 (socket says 172.21.15.27:39454) 2024-01-15T18:05:15.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.395+0000 7f2889000640 1 -- 172.21.15.27:0/510604150 >> v1:172.21.15.93:6789/0 conn(0x7f28840cdf90 legacy=0x7f28840b0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:15.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.395+0000 7f2889000640 1 -- 172.21.15.27:0/510604150 >> v1:172.21.15.27:6789/0 conn(0x7f288406bb80 legacy=0x7f28840cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:15.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.395+0000 7f2889000640 1 -- 172.21.15.27:0/510604150 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f288406c620 con 0x7f28840b2390 2024-01-15T18:05:15.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.395+0000 7f2889000640 1 --2- 172.21.15.27:0/510604150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28840b2390 0x7f28840cf780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f28740097c0 tx=0x7f287401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=477b054b1d463899 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:15.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.395+0000 7f2883400640 1 -- 172.21.15.27:0/510604150 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28740ab650 con 0x7f28840b2390 2024-01-15T18:05:15.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.395+0000 7f2883400640 1 -- 172.21.15.27:0/510604150 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28740ab7b0 con 0x7f28840b2390 2024-01-15T18:05:15.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.395+0000 7f2883400640 1 -- 172.21.15.27:0/510604150 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28740abb40 con 0x7f28840b2390 2024-01-15T18:05:15.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.399+0000 7f288ba00640 1 -- 172.21.15.27:0/510604150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28840b2390 msgr2=0x7f28840cf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:15.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.399+0000 7f288ba00640 1 --2- 172.21.15.27:0/510604150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28840b2390 0x7f28840cf780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f28740097c0 tx=0x7f287401c450 comp rx=0 tx=0).stop 2024-01-15T18:05:15.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.399+0000 7f288ba00640 1 -- 172.21.15.27:0/510604150 shutdown_connections 2024-01-15T18:05:15.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.399+0000 7f288ba00640 1 --2- 172.21.15.27:0/510604150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28840b2390 0x7f28840cf780 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:15.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.399+0000 7f288ba00640 1 -- 172.21.15.27:0/510604150 >> 172.21.15.27:0/510604150 conn(0x7f28840b2a10 msgr2=0x7f28840b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:15.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.399+0000 7f288ba00640 1 -- 172.21.15.27:0/510604150 shutdown_connections 2024-01-15T18:05:15.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.399+0000 7f288ba00640 1 -- 172.21.15.27:0/510604150 wait complete. 2024-01-15T18:05:15.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.399+0000 7f288ba00640 1 Processor -- start 2024-01-15T18:05:15.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.399+0000 7f288ba00640 1 -- start start 2024-01-15T18:05:15.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.399+0000 7f288ba00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f288406bb80 0x7f28840c9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:15.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.399+0000 7f288ba00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28840b2390 0x7f28841f9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:15.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.399+0000 7f288ba00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28840cdf90 0x7f28841fa120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:15.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.399+0000 7f288ba00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28840b6460 con 0x7f288406bb80 2024-01-15T18:05:15.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.399+0000 7f288ba00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28840c4920 con 0x7f28840b2390 2024-01-15T18:05:15.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.399+0000 7f288ba00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28840c4b70 con 0x7f28840cdf90 2024-01-15T18:05:15.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.399+0000 7f2889000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f288406bb80 0x7f28840c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:15.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.399+0000 7f2889000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f288406bb80 0x7f28840c9200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39462/0 (socket says 172.21.15.27:39462) 2024-01-15T18:05:15.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.399+0000 7f2889000640 1 -- 172.21.15.27:0/1115114723 learned_addr learned my addr 172.21.15.27:0/1115114723 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:15.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.399+0000 7f2883e00640 1 --2- 172.21.15.27:0/1115114723 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28840b2390 0x7f28841f9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:15.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.399+0000 7f2889a00640 1 --2- 172.21.15.27:0/1115114723 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28840cdf90 0x7f28841fa120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:15.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.403+0000 7f2889000640 1 -- 172.21.15.27:0/1115114723 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28840cdf90 msgr2=0x7f28841fa120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:15.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.403+0000 7f2889000640 1 --2- 172.21.15.27:0/1115114723 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28840cdf90 0x7f28841fa120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:15.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.403+0000 7f2889000640 1 -- 172.21.15.27:0/1115114723 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28840b2390 msgr2=0x7f28841f9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:15.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.403+0000 7f2889000640 1 --2- 172.21.15.27:0/1115114723 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28840b2390 0x7f28841f9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:15.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.403+0000 7f2889000640 1 -- 172.21.15.27:0/1115114723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2874009420 con 0x7f288406bb80 2024-01-15T18:05:15.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.403+0000 7f2889000640 1 --2- 172.21.15.27:0/1115114723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f288406bb80 0x7f28840c9200 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f28740ad040 tx=0x7f287401ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:15.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.403+0000 7f2880c00640 1 -- 172.21.15.27:0/1115114723 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28740b5810 con 0x7f288406bb80 2024-01-15T18:05:15.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.403+0000 7f2880c00640 1 -- 172.21.15.27:0/1115114723 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28740b5990 con 0x7f288406bb80 2024-01-15T18:05:15.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.403+0000 7f288ba00640 1 -- 172.21.15.27:0/1115114723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28840c4e50 con 0x7f288406bb80 2024-01-15T18:05:15.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.403+0000 7f2880c00640 1 -- 172.21.15.27:0/1115114723 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28740b5d40 con 0x7f288406bb80 2024-01-15T18:05:15.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.403+0000 7f288ba00640 1 -- 172.21.15.27:0/1115114723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28840c5410 con 0x7f288406bb80 2024-01-15T18:05:15.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.403+0000 7f288ba00640 1 -- 172.21.15.27:0/1115114723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f288404e850 con 0x7f288406bb80 2024-01-15T18:05:15.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.403+0000 7f2880c00640 1 -- 172.21.15.27:0/1115114723 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f28740bf410 con 0x7f288406bb80 2024-01-15T18:05:15.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.407+0000 7f2880c00640 1 --2- 172.21.15.27:0/1115114723 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f285c0668f0 0x7f285c068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:15.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.407+0000 7f2880c00640 1 -- 172.21.15.27:0/1115114723 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f287411f190 con 0x7f288406bb80 2024-01-15T18:05:15.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.407+0000 7f2883e00640 1 --2- 172.21.15.27:0/1115114723 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f285c0668f0 0x7f285c068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:15.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.407+0000 7f2883e00640 1 --2- 172.21.15.27:0/1115114723 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f285c0668f0 0x7f285c068db0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f288406eee0 tx=0x7f286c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:15.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.407+0000 7f2880c00640 1 -- 172.21.15.27:0/1115114723 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f28740ec3f0 con 0x7f288406bb80 2024-01-15T18:05:15.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.599+0000 7f288ba00640 1 -- 172.21.15.27:0/1115114723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f288406ef10 con 0x7f288406bb80 2024-01-15T18:05:15.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.599+0000 7f2880c00640 1 -- 172.21.15.27:0/1115114723 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f28740efa10 con 0x7f288406bb80 2024-01-15T18:05:15.607 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:05:15.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.603+0000 7f288ba00640 1 -- 172.21.15.27:0/1115114723 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f285c0668f0 msgr2=0x7f285c068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:15.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.603+0000 7f288ba00640 1 --2- 172.21.15.27:0/1115114723 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f285c0668f0 0x7f285c068db0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f288406eee0 tx=0x7f286c009040 comp rx=0 tx=0).stop 2024-01-15T18:05:15.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.603+0000 7f288ba00640 1 -- 172.21.15.27:0/1115114723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f288406bb80 msgr2=0x7f28840c9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:15.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.603+0000 7f288ba00640 1 --2- 172.21.15.27:0/1115114723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f288406bb80 0x7f28840c9200 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f28740ad040 tx=0x7f287401ac90 comp rx=0 tx=0).stop 2024-01-15T18:05:15.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.603+0000 7f288ba00640 1 -- 172.21.15.27:0/1115114723 shutdown_connections 2024-01-15T18:05:15.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.603+0000 7f288ba00640 1 --2- 172.21.15.27:0/1115114723 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f285c0668f0 0x7f285c068db0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:15.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.603+0000 7f288ba00640 1 --2- 172.21.15.27:0/1115114723 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28840cdf90 0x7f28841fa120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:15.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.603+0000 7f288ba00640 1 --2- 172.21.15.27:0/1115114723 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28840b2390 0x7f28841f9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:15.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.603+0000 7f288ba00640 1 --2- 172.21.15.27:0/1115114723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f288406bb80 0x7f28840c9200 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:15.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.603+0000 7f288ba00640 1 -- 172.21.15.27:0/1115114723 >> 172.21.15.27:0/1115114723 conn(0x7f28840b2a10 msgr2=0x7f28841562f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:15.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.603+0000 7f288ba00640 1 -- 172.21.15.27:0/1115114723 shutdown_connections 2024-01-15T18:05:15.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:15.603+0000 7f288ba00640 1 -- 172.21.15.27:0/1115114723 wait complete. 2024-01-15T18:05:15.628 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:05:15.628 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:05:15.628 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-01-15T18:05:15.628 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-01-15T18:05:15.628 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-01-15T18:05:15.629 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:05:15.629 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:05:15.629 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:05:15.629 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-01-15T18:05:15.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:15.629 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-01-15T18:05:15.639 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:05:15.640 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:05:15.640 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:05:15.640 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:15.640 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-15T18:05:15.640 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:15.640 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:05:15.640 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:15.640 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-01-15T18:05:15.640 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:05:15.641 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:15.641 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-01-15T18:05:15.641 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:05:15.641 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:15.641 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-01-15T18:05:15.641 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-01-15T18:05:15.641 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42444sec preferred_lft 42444sec 2024-01-15T18:05:15.641 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-01-15T18:05:15.641 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:15.641 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:05:15.642 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:15.642 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-01-15T18:05:15.642 INFO:teuthology.orchestra.run.smithi027.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-01-15T18:05:15.642 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 02:42:65:20:14:72 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:15.642 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-01-15T18:05:15.642 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:15.642 INFO:teuthology.orchestra.run.smithi027.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:05:15.642 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:15.642 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:05:15.642 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:15.643 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link tentative 2024-01-15T18:05:15.643 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:15.643 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:05:15.643 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:05:15.643 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link tentative 2024-01-15T18:05:15.643 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:15.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:15.643 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:05:15.701 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-01-15T18:05:15.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:15.703 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-01-15T18:05:15.755 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-01-15T18:05:15.755 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:05:15.756 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-01-15T18:05:15.756 INFO:teuthology.orchestra.run.smithi027.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-01-15T18:05:15.756 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-01-15T18:05:15.756 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:05:15.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:15.756 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-15T18:05:15.756 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-15T18:05:15.756 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-15T18:05:15.757 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-15T18:05:15.757 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-15T18:05:15.757 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-15T18:05:15.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:15.847 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-15T18:05:15.857 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-15T18:05:15.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:15.858 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list-id 2024-01-15T18:05:15.907 INFO:teuthology.orchestra.run.smithi027.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-01-15T18:05:15.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:15.908 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-15T18:05:15.908 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-15T18:05:15.908 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:05:15.908 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-01-15T18:05:15.908 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-15T18:05:15.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:15.989 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-01-15T18:05:16.050 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:05:16.051 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:05:16.051 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:05:16.051 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:16.051 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-15T18:05:16.051 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:16.051 INFO:teuthology.orchestra.run.smithi027.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:05:16.051 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether de:ce:de:68:a5:dd brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-15T18:05:16.052 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-15T18:05:16.052 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:16.052 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::dcce:deff:fe68:a5dd/64 scope link 2024-01-15T18:05:16.052 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:16.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:16.053 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-01-15T18:05:16.115 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:05:16.116 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:05:16.116 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:05:16.116 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:16.116 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-15T18:05:16.116 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:16.116 INFO:teuthology.orchestra.run.smithi027.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:05:16.116 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether de:ce:de:68:a5:dd brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-15T18:05:16.116 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-15T18:05:16.116 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:16.117 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::dcce:deff:fe68:a5dd/64 scope link 2024-01-15T18:05:16.117 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:16.118 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-01-15T18:05:16.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:16.118 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-15T18:05:16.118 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-15T18:05:16.119 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-01-15T18:05:16.119 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-01-15T18:05:16.119 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-01-15T18:05:16.119 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-01-15T18:05:16.119 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-01-15T18:05:16.119 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-15T18:05:16.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:16.256 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-15T18:05:16.256 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-15T18:05:16.256 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.1 up 2024-01-15T18:05:16.256 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set dev brx.1 master ceph-brx 2024-01-15T18:05:16.257 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-15T18:05:16.296 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-01-15T18:05:16.296 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:05:16.296 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-01-15T18:05:16.347 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-01-15T18:05:16.347 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:05:16.347 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-01-15T18:05:16.398 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-01-15T18:05:16.462 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:05:16.516 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-01-15T18:05:16.516 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-01-15T18:05:16.516 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-01-15T18:05:16.516 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-01-15T18:05:16.516 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-01-15T18:05:16.517 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-01-15T18:05:16.517 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-01-15T18:05:16.517 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-01-15T18:05:16.517 INFO:teuthology.orchestra.run.smithi027.stdout:/run/lock 2024-01-15T18:05:16.517 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-01-15T18:05:16.517 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-01-15T18:05:16.517 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-01-15T18:05:16.517 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:05:16.517 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-01-15T18:05:16.517 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-01-15T18:05:16.517 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-01-15T18:05:16.518 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-01-15T18:05:16.518 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/fs/nfsd 2024-01-15T18:05:16.518 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-01-15T18:05:16.518 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-01-15T18:05:16.519 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:05:16.519 INFO:teuthology.orchestra.run.smithi027.stdout:/run/rpc_pipefs 2024-01-15T18:05:16.519 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-01-15T18:05:16.519 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-01-15T18:05:16.519 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:05:16.519 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:05:16.519 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:05:16.519 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:05:16.519 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-01-15T18:05:16.520 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:05:16.520 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:05:16.520 INFO:teuthology.orchestra.run.smithi027.stdout:/home/ubuntu/cephtest/mnt.0 2024-01-15T18:05:16.520 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:05:16.520 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:05:16.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:16.521 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-01-15T18:05:16.571 INFO:teuthology.orchestra.run.smithi027.stdout:43 2024-01-15T18:05:16.571 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-01-15T18:05:16.571 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-01-15T18:05:16.617 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-01-15T18:05:16.639 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:05:16.692 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-01-15T18:05:16.692 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-01-15T18:05:16.692 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-01-15T18:05:16.692 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-01-15T18:05:16.693 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-01-15T18:05:16.693 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-01-15T18:05:16.693 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-01-15T18:05:16.693 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-01-15T18:05:16.693 INFO:teuthology.orchestra.run.smithi027.stdout:/run/lock 2024-01-15T18:05:16.693 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-01-15T18:05:16.693 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-01-15T18:05:16.693 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-01-15T18:05:16.693 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:05:16.693 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-01-15T18:05:16.694 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-01-15T18:05:16.694 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-01-15T18:05:16.694 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-01-15T18:05:16.694 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/fs/nfsd 2024-01-15T18:05:16.694 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-01-15T18:05:16.694 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-01-15T18:05:16.694 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:05:16.694 INFO:teuthology.orchestra.run.smithi027.stdout:/run/rpc_pipefs 2024-01-15T18:05:16.694 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-01-15T18:05:16.694 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-01-15T18:05:16.694 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:05:16.695 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:05:16.695 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:05:16.695 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:05:16.695 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-01-15T18:05:16.695 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:05:16.695 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:05:16.695 INFO:teuthology.orchestra.run.smithi027.stdout:/home/ubuntu/cephtest/mnt.0 2024-01-15T18:05:16.695 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:05:16.695 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:05:16.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:16.697 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-01-15T18:05:16.712 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi027.stderr:2024-01-15T18:05:16.707+0000 7f2da9bd2580 -1 init, newargv = 0x5563b9d90130 newargc=15 2024-01-15T18:05:16.713 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi027.stderr:ceph-fuse[33911]: starting ceph client 2024-01-15T18:05:16.719 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi027.stderr:ceph-fuse[33911]: starting fuse 2024-01-15T18:05:16.722 INFO:teuthology.orchestra.run.smithi027.stdout:43 2024-01-15T18:05:16.722 INFO:teuthology.orchestra.run.smithi027.stdout:44 2024-01-15T18:05:16.722 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-01-15T18:05:16.722 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:05:16.723 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-01-15T18:05:16.723 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-01-15T18:05:16.723 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-01-15T18:05:16.723 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-01-15T18:05:16.723 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:05:16.723 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-01-15T18:05:16.723 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-01-15T18:05:16.723 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-01-15T18:05:16.723 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-01-15T18:05:16.723 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:05:16.723 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-01-15T18:05:16.724 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-01-15T18:05:16.724 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-01-15T18:05:16.724 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-01-15T18:05:16.724 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:05:16.724 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-01-15T18:05:16.724 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:05:16.724 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:05:16.724 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:05:16.724 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-01-15T18:05:16.725 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-01-15T18:05:16.725 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-01-15T18:05:16.725 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:05:16.725 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:05:16.725 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.2")) 2024-01-15T18:05:16.725 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-01-15T18:05:16.849 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.2.33911.asok 2024-01-15T18:05:16.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:05:16.861 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.33911.asok 2024-01-15T18:05:16.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:16.861 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.33911.asok status 2024-01-15T18:05:17.034 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:05:17.034 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-01-15T18:05:17.034 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:05:17.034 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:05:17.034 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "2", 2024-01-15T18:05:17.034 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-01-15T18:05:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-01-15T18:05:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "33911", 2024-01-15T18:05:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-01-15T18:05:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-01-15T18:05:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-01-15T18:05:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4692, 2024-01-15T18:05:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-01-15T18:05:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-01-15T18:05:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-01-15T18:05:17.035 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 4692 2024-01-15T18:05:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-01-15T18:05:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-01-15T18:05:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.2:0", 2024-01-15T18:05:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 748978069 2024-01-15T18:05:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-01-15T18:05:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-01-15T18:05:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.2:0", 2024-01-15T18:05:17.036 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 748978069 2024-01-15T18:05:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.4692 192.168.0.2:0/748978069", 2024-01-15T18:05:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.2:0/748978069", 2024-01-15T18:05:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-01-15T18:05:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 8, 2024-01-15T18:05:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 21, 2024-01-15T18:05:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:05:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-01-15T18:05:17.037 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-01-15T18:05:17.037 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:05:17.050 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:05:17.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.187+0000 7fd4a4800640 1 Processor -- start 2024-01-15T18:05:17.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.187+0000 7fd4a4800640 1 -- start start 2024-01-15T18:05:17.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.187+0000 7fd4a4800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd49c06bb80 0x7fd49c0cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:17.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.187+0000 7fd4a4800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd49c0cddc0 0x7fd49c0b0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:17.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.187+0000 7fd4a4800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd49c0b2770 con 0x7fd49c0b2390 2024-01-15T18:05:17.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.187+0000 7fd4a4800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd49c0ce270 con 0x7fd49c06bb80 2024-01-15T18:05:17.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.187+0000 7fd4a4800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c06c060 con 0x7fd49c0cddc0 2024-01-15T18:05:17.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.187+0000 7fd4a1400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd49c06bb80 0x7fd49c0cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:17.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.187+0000 7fd4a1e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd49c0b2390 0x7fd49c0cf780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34082/0 (socket says 172.21.15.27:34082) 2024-01-15T18:05:17.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.187+0000 7fd4a1400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd49c06bb80 0x7fd49c0cd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39474/0 (socket says 172.21.15.27:39474) 2024-01-15T18:05:17.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.187+0000 7fd4a1e00640 1 -- 172.21.15.27:0/1501492773 learned_addr learned my addr 172.21.15.27:0/1501492773 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:17.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.187+0000 7fd4a2800640 1 --2- 172.21.15.27:0/1501492773 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd49c0cddc0 0x7fd49c0b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:17.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.187+0000 7fd4a1400640 1 -- 172.21.15.27:0/1501492773 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd49c0cddc0 msgr2=0x7fd49c0b0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:17.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.187+0000 7fd4a1400640 1 --2- 172.21.15.27:0/1501492773 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd49c0cddc0 0x7fd49c0b0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:17.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.187+0000 7fd4a1400640 1 -- 172.21.15.27:0/1501492773 >> v1:172.21.15.27:6789/0 conn(0x7fd49c0b2390 legacy=0x7fd49c0cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:17.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.187+0000 7fd4a1400640 1 -- 172.21.15.27:0/1501492773 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd49c06c340 con 0x7fd49c06bb80 2024-01-15T18:05:17.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.187+0000 7fd4a0a00640 1 -- 172.21.15.27:0/1501492773 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132876671 0 0) 0x7fd49c0b2770 con 0x7fd49c0b2390 2024-01-15T18:05:17.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.187+0000 7fd4a2800640 1 --2- 172.21.15.27:0/1501492773 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd49c0cddc0 0x7fd49c0b0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:17.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.191+0000 7fd4a1400640 1 --2- 172.21.15.27:0/1501492773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd49c06bb80 0x7fd49c0cd880 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd4840097c0 tx=0x7fd48401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e191333d813501ca server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:17.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.191+0000 7fd4a0a00640 1 -- 172.21.15.27:0/1501492773 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4840ab650 con 0x7fd49c06bb80 2024-01-15T18:05:17.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.191+0000 7fd4a0a00640 1 -- 172.21.15.27:0/1501492773 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4840ab7b0 con 0x7fd49c06bb80 2024-01-15T18:05:17.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.191+0000 7fd4a0a00640 1 -- 172.21.15.27:0/1501492773 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4840abb40 con 0x7fd49c06bb80 2024-01-15T18:05:17.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.191+0000 7fd4a4800640 1 -- 172.21.15.27:0/1501492773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd49c06bb80 msgr2=0x7fd49c0cd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:17.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.191+0000 7fd4a4800640 1 --2- 172.21.15.27:0/1501492773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd49c06bb80 0x7fd49c0cd880 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd4840097c0 tx=0x7fd48401c450 comp rx=0 tx=0).stop 2024-01-15T18:05:17.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.195+0000 7fd4a4800640 1 -- 172.21.15.27:0/1501492773 shutdown_connections 2024-01-15T18:05:17.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.195+0000 7fd4a4800640 1 --2- 172.21.15.27:0/1501492773 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd49c0cddc0 0x7fd49c0b0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:17.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.195+0000 7fd4a4800640 1 --2- 172.21.15.27:0/1501492773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd49c06bb80 0x7fd49c0cd880 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:17.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.195+0000 7fd4a4800640 1 -- 172.21.15.27:0/1501492773 >> 172.21.15.27:0/1501492773 conn(0x7fd49c0b2a10 msgr2=0x7fd49c0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:17.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.195+0000 7fd4a4800640 1 -- 172.21.15.27:0/1501492773 shutdown_connections 2024-01-15T18:05:17.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.195+0000 7fd4a4800640 1 -- 172.21.15.27:0/1501492773 wait complete. 2024-01-15T18:05:17.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.195+0000 7fd4a4800640 1 Processor -- start 2024-01-15T18:05:17.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.195+0000 7fd4a4800640 1 -- start start 2024-01-15T18:05:17.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.195+0000 7fd4a4800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd49c06bb80 0x7fd49c158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:17.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.195+0000 7fd4a4800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd49c0b2390 0x7fd49c158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:17.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.195+0000 7fd4a4800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd49c0cddc0 0x7fd49c0af440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:17.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.195+0000 7fd4a4800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c0c81b0 con 0x7fd49c0b2390 2024-01-15T18:05:17.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.195+0000 7fd4a4800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c15a2c0 con 0x7fd49c06bb80 2024-01-15T18:05:17.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.195+0000 7fd4a4800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd49c0c9570 con 0x7fd49c0cddc0 2024-01-15T18:05:17.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.195+0000 7fd4a1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd49c0b2390 0x7fd49c158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:17.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.195+0000 7fd4a1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd49c0b2390 0x7fd49c158f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39490/0 (socket says 172.21.15.27:39490) 2024-01-15T18:05:17.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.195+0000 7fd4a1400640 1 -- 172.21.15.27:0/391054750 learned_addr learned my addr 172.21.15.27:0/391054750 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:17.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.195+0000 7fd4a1e00640 1 --2- 172.21.15.27:0/391054750 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd49c06bb80 0x7fd49c158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:17.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.195+0000 7fd4a2800640 1 --2- 172.21.15.27:0/391054750 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd49c0cddc0 0x7fd49c0af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:17.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.195+0000 7fd4a1400640 1 -- 172.21.15.27:0/391054750 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd49c0cddc0 msgr2=0x7fd49c0af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:17.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.195+0000 7fd4a1400640 1 --2- 172.21.15.27:0/391054750 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd49c0cddc0 0x7fd49c0af440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:17.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.195+0000 7fd4a1400640 1 -- 172.21.15.27:0/391054750 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd49c06bb80 msgr2=0x7fd49c158810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:17.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.195+0000 7fd4a1400640 1 --2- 172.21.15.27:0/391054750 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd49c06bb80 0x7fd49c158810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:17.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.195+0000 7fd4a1400640 1 -- 172.21.15.27:0/391054750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd484009420 con 0x7fd49c0b2390 2024-01-15T18:05:17.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.199+0000 7fd4a1400640 1 --2- 172.21.15.27:0/391054750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd49c0b2390 0x7fd49c158f80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd4840ad040 tx=0x7fd48401ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:17.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.199+0000 7fd49a000640 1 -- 172.21.15.27:0/391054750 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4840b5810 con 0x7fd49c0b2390 2024-01-15T18:05:17.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.199+0000 7fd49a000640 1 -- 172.21.15.27:0/391054750 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4840b59b0 con 0x7fd49c0b2390 2024-01-15T18:05:17.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.199+0000 7fd49a000640 1 -- 172.21.15.27:0/391054750 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4840b5d60 con 0x7fd49c0b2390 2024-01-15T18:05:17.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.199+0000 7fd4a4800640 1 -- 172.21.15.27:0/391054750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd49c158d50 con 0x7fd49c0b2390 2024-01-15T18:05:17.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.199+0000 7fd4a4800640 1 -- 172.21.15.27:0/391054750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd49c159d00 con 0x7fd49c0b2390 2024-01-15T18:05:17.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.199+0000 7fd4a4800640 1 -- 172.21.15.27:0/391054750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd49c0d07c0 con 0x7fd49c0b2390 2024-01-15T18:05:17.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.199+0000 7fd49a000640 1 -- 172.21.15.27:0/391054750 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fd4840b4390 con 0x7fd49c0b2390 2024-01-15T18:05:17.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.199+0000 7fd49a000640 1 --2- 172.21.15.27:0/391054750 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd4740668f0 0x7fd474068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:17.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.199+0000 7fd49a000640 1 -- 172.21.15.27:0/391054750 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd48411f140 con 0x7fd49c0b2390 2024-01-15T18:05:17.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.199+0000 7fd4a1e00640 1 --2- 172.21.15.27:0/391054750 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd4740668f0 0x7fd474068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:17.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.203+0000 7fd4a1e00640 1 --2- 172.21.15.27:0/391054750 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd4740668f0 0x7fd474068db0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd48c007fd0 tx=0x7fd48c007d60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:17.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.203+0000 7fd49a000640 1 -- 172.21.15.27:0/391054750 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd4840ec540 con 0x7fd49c0b2390 2024-01-15T18:05:17.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.395+0000 7fd4a4800640 1 -- 172.21.15.27:0/391054750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd49c04e850 con 0x7fd49c0b2390 2024-01-15T18:05:17.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.395+0000 7fd49a000640 1 -- 172.21.15.27:0/391054750 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd4840efb60 con 0x7fd49c0b2390 2024-01-15T18:05:17.401 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:05:17.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.399+0000 7fd4a4800640 1 -- 172.21.15.27:0/391054750 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd4740668f0 msgr2=0x7fd474068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:17.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.399+0000 7fd4a4800640 1 --2- 172.21.15.27:0/391054750 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd4740668f0 0x7fd474068db0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd48c007fd0 tx=0x7fd48c007d60 comp rx=0 tx=0).stop 2024-01-15T18:05:17.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.399+0000 7fd4a4800640 1 -- 172.21.15.27:0/391054750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd49c0b2390 msgr2=0x7fd49c158f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:17.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.399+0000 7fd4a4800640 1 --2- 172.21.15.27:0/391054750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd49c0b2390 0x7fd49c158f80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd4840ad040 tx=0x7fd48401ac90 comp rx=0 tx=0).stop 2024-01-15T18:05:17.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.399+0000 7fd4a4800640 1 -- 172.21.15.27:0/391054750 shutdown_connections 2024-01-15T18:05:17.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.399+0000 7fd4a4800640 1 --2- 172.21.15.27:0/391054750 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd4740668f0 0x7fd474068db0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:17.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.399+0000 7fd4a4800640 1 --2- 172.21.15.27:0/391054750 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd49c0cddc0 0x7fd49c0af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:17.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.399+0000 7fd4a4800640 1 --2- 172.21.15.27:0/391054750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd49c0b2390 0x7fd49c158f80 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:17.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.399+0000 7fd4a4800640 1 --2- 172.21.15.27:0/391054750 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd49c06bb80 0x7fd49c158810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:17.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.399+0000 7fd4a4800640 1 -- 172.21.15.27:0/391054750 >> 172.21.15.27:0/391054750 conn(0x7fd49c0b2a10 msgr2=0x7fd49c0b05d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:17.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.399+0000 7fd4a4800640 1 -- 172.21.15.27:0/391054750 shutdown_connections 2024-01-15T18:05:17.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:17.399+0000 7fd4a4800640 1 -- 172.21.15.27:0/391054750 wait complete. 2024-01-15T18:05:17.422 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:05:17.422 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:05:17.423 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi093.front.sepia.ceph.com 2024-01-15T18:05:17.423 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-15T18:05:17.423 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-15T18:05:17.423 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:05:17.423 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:05:17.423 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:05:17.423 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:05:17.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:17.423 DEBUG:teuthology.orchestra.run.smithi093:> ip addr 2024-01-15T18:05:17.434 INFO:teuthology.orchestra.run.smithi093.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:05:17.434 INFO:teuthology.orchestra.run.smithi093.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:05:17.434 INFO:teuthology.orchestra.run.smithi093.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:05:17.434 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:17.434 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 ::1/128 scope host 2024-01-15T18:05:17.434 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:17.434 INFO:teuthology.orchestra.run.smithi093.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:05:17.434 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:d8:da:1c brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:17.434 INFO:teuthology.orchestra.run.smithi093.stdout: altname enp7s0f0 2024-01-15T18:05:17.434 INFO:teuthology.orchestra.run.smithi093.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:05:17.435 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:d8:da:1d brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:17.435 INFO:teuthology.orchestra.run.smithi093.stdout: altname enp7s0f1 2024-01-15T18:05:17.435 INFO:teuthology.orchestra.run.smithi093.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:05:17.435 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:8f:d6:34 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:17.435 INFO:teuthology.orchestra.run.smithi093.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:05:17.435 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:8f:d6:35 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:17.435 INFO:teuthology.orchestra.run.smithi093.stdout: inet 172.21.15.93/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-01-15T18:05:17.435 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft 42421sec preferred_lft 42421sec 2024-01-15T18:05:17.435 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::ec4:7aff:fe8f:d635/64 scope link 2024-01-15T18:05:17.435 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:17.435 INFO:teuthology.orchestra.run.smithi093.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-01-15T18:05:17.435 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 02:42:2a:61:fc:7d brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:17.435 INFO:teuthology.orchestra.run.smithi093.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-01-15T18:05:17.436 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:17.436 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-15T18:05:17.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:17.436 DEBUG:teuthology.orchestra.run.smithi093:> (cd / && exec bash -c ' 2024-01-15T18:05:17.436 DEBUG:teuthology.orchestra.run.smithi093:> set -e 2024-01-15T18:05:17.436 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip link add name ceph-brx type bridge 2024-01-15T18:05:17.436 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip addr flush dev ceph-brx 2024-01-15T18:05:17.436 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip link set ceph-brx up 2024-01-15T18:05:17.436 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-15T18:05:17.437 DEBUG:teuthology.orchestra.run.smithi093:> ') 2024-01-15T18:05:17.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:17.550 DEBUG:teuthology.orchestra.run.smithi093:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:05:17.568 INFO:teuthology.orchestra.run.smithi093.stdout:1 2024-01-15T18:05:17.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:17.571 DEBUG:teuthology.orchestra.run.smithi093:> route 2024-01-15T18:05:17.652 INFO:teuthology.orchestra.run.smithi093.stdout:Kernel IP routing table 2024-01-15T18:05:17.652 INFO:teuthology.orchestra.run.smithi093.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:05:17.653 INFO:teuthology.orchestra.run.smithi093.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-01-15T18:05:17.653 INFO:teuthology.orchestra.run.smithi093.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T18:05:17.653 INFO:teuthology.orchestra.run.smithi093.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-01-15T18:05:17.653 INFO:teuthology.orchestra.run.smithi093.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T18:05:17.653 INFO:teuthology.orchestra.run.smithi093.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:05:17.653 INFO:teuthology.orchestra.run.smithi093.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:05:17.653 INFO:teuthology.orchestra.run.smithi093.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:05:17.653 INFO:teuthology.orchestra.run.smithi093.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:05:17.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:17.653 DEBUG:teuthology.orchestra.run.smithi093:> (cd / && exec bash -c ' 2024-01-15T18:05:17.654 DEBUG:teuthology.orchestra.run.smithi093:> set -e 2024-01-15T18:05:17.654 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T18:05:17.654 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T18:05:17.654 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T18:05:17.654 DEBUG:teuthology.orchestra.run.smithi093:> ') 2024-01-15T18:05:17.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:17.743 DEBUG:teuthology.orchestra.run.smithi093:> ip netns list 2024-01-15T18:05:17.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:17.753 DEBUG:teuthology.orchestra.run.smithi093:> ip netns list-id 2024-01-15T18:05:17.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:17.806 DEBUG:teuthology.orchestra.run.smithi093:> (cd / && exec bash -c ' 2024-01-15T18:05:17.806 DEBUG:teuthology.orchestra.run.smithi093:> set -e 2024-01-15T18:05:17.806 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:05:17.807 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-01-15T18:05:17.807 DEBUG:teuthology.orchestra.run.smithi093:> ') 2024-01-15T18:05:17.891 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-01-15T18:05:17.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:17.892 DEBUG:teuthology.orchestra.run.smithi093:> (cd / && exec bash -c ' 2024-01-15T18:05:17.892 DEBUG:teuthology.orchestra.run.smithi093:> set -e 2024-01-15T18:05:17.892 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-01-15T18:05:17.892 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-15T18:05:17.892 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-01-15T18:05:17.892 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-01-15T18:05:17.892 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-01-15T18:05:17.892 DEBUG:teuthology.orchestra.run.smithi093:> ') 2024-01-15T18:05:18.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:18.053 DEBUG:teuthology.orchestra.run.smithi093:> (cd / && exec bash -c ' 2024-01-15T18:05:18.054 DEBUG:teuthology.orchestra.run.smithi093:> set -e 2024-01-15T18:05:18.054 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip link set brx.0 up 2024-01-15T18:05:18.054 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip link set dev brx.0 master ceph-brx 2024-01-15T18:05:18.054 DEBUG:teuthology.orchestra.run.smithi093:> ') 2024-01-15T18:05:18.093 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-15T18:05:18.093 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:05:18.093 DEBUG:teuthology.orchestra.run.smithi093:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-15T18:05:18.146 INFO:teuthology.orchestra.run.smithi093.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:05:18.146 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:05:18.146 DEBUG:teuthology.orchestra.run.smithi093:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:05:18.202 DEBUG:teuthology.orchestra.run.smithi093:> sudo modprobe fuse 2024-01-15T18:05:18.266 DEBUG:teuthology.orchestra.run.smithi093:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:05:18.319 INFO:teuthology.orchestra.run.smithi093.stdout:/sys 2024-01-15T18:05:18.319 INFO:teuthology.orchestra.run.smithi093.stdout:/proc 2024-01-15T18:05:18.320 INFO:teuthology.orchestra.run.smithi093.stdout:/dev 2024-01-15T18:05:18.320 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/pts 2024-01-15T18:05:18.320 INFO:teuthology.orchestra.run.smithi093.stdout:/run 2024-01-15T18:05:18.320 INFO:teuthology.orchestra.run.smithi093.stdout:/ 2024-01-15T18:05:18.320 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/security 2024-01-15T18:05:18.320 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/shm 2024-01-15T18:05:18.320 INFO:teuthology.orchestra.run.smithi093.stdout:/run/lock 2024-01-15T18:05:18.320 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/cgroup 2024-01-15T18:05:18.320 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/pstore 2024-01-15T18:05:18.321 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/bpf 2024-01-15T18:05:18.321 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:05:18.321 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/hugepages 2024-01-15T18:05:18.321 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/mqueue 2024-01-15T18:05:18.321 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/debug 2024-01-15T18:05:18.321 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/tracing 2024-01-15T18:05:18.321 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/fs/nfsd 2024-01-15T18:05:18.321 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/fuse/connections 2024-01-15T18:05:18.321 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/config 2024-01-15T18:05:18.321 INFO:teuthology.orchestra.run.smithi093.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:05:18.322 INFO:teuthology.orchestra.run.smithi093.stdout:/run/rpc_pipefs 2024-01-15T18:05:18.322 INFO:teuthology.orchestra.run.smithi093.stdout:/run/user/1000 2024-01-15T18:05:18.322 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph 2024-01-15T18:05:18.322 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:05:18.322 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:05:18.322 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:05:18.322 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:05:18.322 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns 2024-01-15T18:05:18.322 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:05:18.322 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:05:18.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:18.324 DEBUG:teuthology.orchestra.run.smithi093:> ls /sys/fs/fuse/connections 2024-01-15T18:05:18.374 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:05:18.374 DEBUG:teuthology.orchestra.run.smithi093:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-15T18:05:18.421 DEBUG:teuthology.orchestra.run.smithi093:> sudo modprobe fuse 2024-01-15T18:05:18.442 DEBUG:teuthology.orchestra.run.smithi093:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:05:18.495 INFO:teuthology.orchestra.run.smithi093.stdout:/sys 2024-01-15T18:05:18.495 INFO:teuthology.orchestra.run.smithi093.stdout:/proc 2024-01-15T18:05:18.495 INFO:teuthology.orchestra.run.smithi093.stdout:/dev 2024-01-15T18:05:18.495 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/pts 2024-01-15T18:05:18.496 INFO:teuthology.orchestra.run.smithi093.stdout:/run 2024-01-15T18:05:18.496 INFO:teuthology.orchestra.run.smithi093.stdout:/ 2024-01-15T18:05:18.496 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/security 2024-01-15T18:05:18.496 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/shm 2024-01-15T18:05:18.496 INFO:teuthology.orchestra.run.smithi093.stdout:/run/lock 2024-01-15T18:05:18.496 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/cgroup 2024-01-15T18:05:18.496 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/pstore 2024-01-15T18:05:18.496 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/bpf 2024-01-15T18:05:18.496 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:05:18.496 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/hugepages 2024-01-15T18:05:18.497 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/mqueue 2024-01-15T18:05:18.497 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/debug 2024-01-15T18:05:18.497 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/tracing 2024-01-15T18:05:18.497 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/fs/nfsd 2024-01-15T18:05:18.497 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/fuse/connections 2024-01-15T18:05:18.497 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/config 2024-01-15T18:05:18.497 INFO:teuthology.orchestra.run.smithi093.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:05:18.497 INFO:teuthology.orchestra.run.smithi093.stdout:/run/rpc_pipefs 2024-01-15T18:05:18.497 INFO:teuthology.orchestra.run.smithi093.stdout:/run/user/1000 2024-01-15T18:05:18.497 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph 2024-01-15T18:05:18.498 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:05:18.498 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:05:18.498 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:05:18.498 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:05:18.498 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns 2024-01-15T18:05:18.498 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:05:18.498 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:05:18.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:18.499 DEBUG:teuthology.orchestra.run.smithi093:> ls /sys/fs/fuse/connections 2024-01-15T18:05:18.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:2024-01-15T18:05:18.518+0000 7f923c603580 -1 init, newargv = 0x55f927517430 newargc=15 2024-01-15T18:05:18.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:ceph-fuse[33018]: starting ceph client 2024-01-15T18:05:18.534 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:ceph-fuse[33018]: starting fuse 2024-01-15T18:05:19.635 DEBUG:teuthology.orchestra.run.smithi093:> sudo modprobe fuse 2024-01-15T18:05:19.659 DEBUG:teuthology.orchestra.run.smithi093:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:05:19.712 INFO:teuthology.orchestra.run.smithi093.stdout:/sys 2024-01-15T18:05:19.712 INFO:teuthology.orchestra.run.smithi093.stdout:/proc 2024-01-15T18:05:19.712 INFO:teuthology.orchestra.run.smithi093.stdout:/dev 2024-01-15T18:05:19.712 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/pts 2024-01-15T18:05:19.712 INFO:teuthology.orchestra.run.smithi093.stdout:/run 2024-01-15T18:05:19.713 INFO:teuthology.orchestra.run.smithi093.stdout:/ 2024-01-15T18:05:19.713 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/security 2024-01-15T18:05:19.713 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/shm 2024-01-15T18:05:19.713 INFO:teuthology.orchestra.run.smithi093.stdout:/run/lock 2024-01-15T18:05:19.713 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/cgroup 2024-01-15T18:05:19.713 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/pstore 2024-01-15T18:05:19.713 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/bpf 2024-01-15T18:05:19.713 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:05:19.713 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/hugepages 2024-01-15T18:05:19.713 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/mqueue 2024-01-15T18:05:19.713 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/debug 2024-01-15T18:05:19.714 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/tracing 2024-01-15T18:05:19.714 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/fs/nfsd 2024-01-15T18:05:19.714 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/fuse/connections 2024-01-15T18:05:19.714 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/config 2024-01-15T18:05:19.714 INFO:teuthology.orchestra.run.smithi093.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:05:19.714 INFO:teuthology.orchestra.run.smithi093.stdout:/run/rpc_pipefs 2024-01-15T18:05:19.714 INFO:teuthology.orchestra.run.smithi093.stdout:/run/user/1000 2024-01-15T18:05:19.714 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph 2024-01-15T18:05:19.714 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:05:19.714 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:05:19.714 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:05:19.714 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:05:19.715 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns 2024-01-15T18:05:19.715 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:05:19.715 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:05:19.715 INFO:teuthology.orchestra.run.smithi093.stdout:/home/ubuntu/cephtest/mnt.1 2024-01-15T18:05:19.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:19.716 DEBUG:teuthology.orchestra.run.smithi093:> ls /sys/fs/fuse/connections 2024-01-15T18:05:19.766 INFO:teuthology.orchestra.run.smithi093.stdout:43 2024-01-15T18:05:19.766 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-01-15T18:05:19.767 DEBUG:teuthology.orchestra.run.smithi093:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:05:19.767 DEBUG:teuthology.orchestra.run.smithi093:> import glob 2024-01-15T18:05:19.767 DEBUG:teuthology.orchestra.run.smithi093:> import re 2024-01-15T18:05:19.767 DEBUG:teuthology.orchestra.run.smithi093:> import os 2024-01-15T18:05:19.767 DEBUG:teuthology.orchestra.run.smithi093:> import subprocess 2024-01-15T18:05:19.767 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:05:19.767 DEBUG:teuthology.orchestra.run.smithi093:> def _find_admin_socket(client_name): 2024-01-15T18:05:19.767 DEBUG:teuthology.orchestra.run.smithi093:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-15T18:05:19.767 DEBUG:teuthology.orchestra.run.smithi093:> files = glob.glob(asok_path) 2024-01-15T18:05:19.767 DEBUG:teuthology.orchestra.run.smithi093:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-15T18:05:19.767 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:05:19.767 DEBUG:teuthology.orchestra.run.smithi093:> # Given a non-glob path, it better be there 2024-01-15T18:05:19.767 DEBUG:teuthology.orchestra.run.smithi093:> if "*" not in asok_path: 2024-01-15T18:05:19.768 DEBUG:teuthology.orchestra.run.smithi093:> assert(len(files) == 1) 2024-01-15T18:05:19.768 DEBUG:teuthology.orchestra.run.smithi093:> return files[0] 2024-01-15T18:05:19.768 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:05:19.768 DEBUG:teuthology.orchestra.run.smithi093:> for f in files: 2024-01-15T18:05:19.768 DEBUG:teuthology.orchestra.run.smithi093:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:05:19.768 DEBUG:teuthology.orchestra.run.smithi093:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:05:19.768 DEBUG:teuthology.orchestra.run.smithi093:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:05:19.768 DEBUG:teuthology.orchestra.run.smithi093:> contents = proc_f.read() 2024-01-15T18:05:19.768 DEBUG:teuthology.orchestra.run.smithi093:> if mountpoint in contents: 2024-01-15T18:05:19.768 DEBUG:teuthology.orchestra.run.smithi093:> return f 2024-01-15T18:05:19.768 DEBUG:teuthology.orchestra.run.smithi093:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:05:19.768 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:05:19.769 DEBUG:teuthology.orchestra.run.smithi093:> print(_find_admin_socket("client.1")) 2024-01-15T18:05:19.769 DEBUG:teuthology.orchestra.run.smithi093:> ' 2024-01-15T18:05:19.897 INFO:teuthology.orchestra.run.smithi093.stdout:/var/run/ceph/ceph-client.1.33018.asok 2024-01-15T18:05:19.900 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:05:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:05:19.909 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.33018.asok 2024-01-15T18:05:19.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:19.909 DEBUG:teuthology.orchestra.run.smithi093:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.33018.asok status 2024-01-15T18:05:20.081 INFO:teuthology.orchestra.run.smithi093.stdout:{ 2024-01-15T18:05:20.081 INFO:teuthology.orchestra.run.smithi093.stdout: "metadata": { 2024-01-15T18:05:20.082 INFO:teuthology.orchestra.run.smithi093.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:05:20.082 INFO:teuthology.orchestra.run.smithi093.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:05:20.082 INFO:teuthology.orchestra.run.smithi093.stdout: "entity_id": "1", 2024-01-15T18:05:20.082 INFO:teuthology.orchestra.run.smithi093.stdout: "hostname": "smithi093", 2024-01-15T18:05:20.082 INFO:teuthology.orchestra.run.smithi093.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-15T18:05:20.082 INFO:teuthology.orchestra.run.smithi093.stdout: "pid": "33018", 2024-01-15T18:05:20.082 INFO:teuthology.orchestra.run.smithi093.stdout: "root": "/" 2024-01-15T18:05:20.082 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:05:20.082 INFO:teuthology.orchestra.run.smithi093.stdout: "dentry_count": 0, 2024-01-15T18:05:20.082 INFO:teuthology.orchestra.run.smithi093.stdout: "dentry_pinned_count": 0, 2024-01-15T18:05:20.082 INFO:teuthology.orchestra.run.smithi093.stdout: "id": 4702, 2024-01-15T18:05:20.083 INFO:teuthology.orchestra.run.smithi093.stdout: "inst": { 2024-01-15T18:05:20.083 INFO:teuthology.orchestra.run.smithi093.stdout: "name": { 2024-01-15T18:05:20.083 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "client", 2024-01-15T18:05:20.083 INFO:teuthology.orchestra.run.smithi093.stdout: "num": 4702 2024-01-15T18:05:20.083 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:05:20.083 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": { 2024-01-15T18:05:20.083 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "v1", 2024-01-15T18:05:20.083 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:05:20.083 INFO:teuthology.orchestra.run.smithi093.stdout: "nonce": 1348509051 2024-01-15T18:05:20.083 INFO:teuthology.orchestra.run.smithi093.stdout: } 2024-01-15T18:05:20.083 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:05:20.084 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": { 2024-01-15T18:05:20.084 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "v1", 2024-01-15T18:05:20.084 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:05:20.084 INFO:teuthology.orchestra.run.smithi093.stdout: "nonce": 1348509051 2024-01-15T18:05:20.084 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:05:20.084 INFO:teuthology.orchestra.run.smithi093.stdout: "inst_str": "client.4702 192.168.0.1:0/1348509051", 2024-01-15T18:05:20.084 INFO:teuthology.orchestra.run.smithi093.stdout: "addr_str": "192.168.0.1:0/1348509051", 2024-01-15T18:05:20.084 INFO:teuthology.orchestra.run.smithi093.stdout: "inode_count": 1, 2024-01-15T18:05:20.084 INFO:teuthology.orchestra.run.smithi093.stdout: "mds_epoch": 8, 2024-01-15T18:05:20.084 INFO:teuthology.orchestra.run.smithi093.stdout: "osd_epoch": 21, 2024-01-15T18:05:20.084 INFO:teuthology.orchestra.run.smithi093.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:05:20.085 INFO:teuthology.orchestra.run.smithi093.stdout: "blocklisted": false, 2024-01-15T18:05:20.085 INFO:teuthology.orchestra.run.smithi093.stdout: "fs_name": "cephfs" 2024-01-15T18:05:20.085 INFO:teuthology.orchestra.run.smithi093.stdout:} 2024-01-15T18:05:20.096 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:05:20.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.243+0000 7f578fc00640 1 Processor -- start 2024-01-15T18:05:20.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.243+0000 7f578fc00640 1 -- start start 2024-01-15T18:05:20.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.243+0000 7f578fc00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f57880c3e90 0x7f57880b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:20.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.243+0000 7f578fc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f57880b23e0 0x7f578806b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:20.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.243+0000 7f578fc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f57880c4270 0x7f57880cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:20.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.247+0000 7f578fc00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5788161d10 con 0x7f57880b23e0 2024-01-15T18:05:20.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.247+0000 7f578fc00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5788165de0 con 0x7f57880c3e90 2024-01-15T18:05:20.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.247+0000 7f578fc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57880cf260 con 0x7f57880c4270 2024-01-15T18:05:20.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.247+0000 7f5787e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f57880b23e0 0x7f578806b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:20.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.247+0000 7f5787e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f57880b23e0 0x7f578806b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39504/0 (socket says 172.21.15.27:39504) 2024-01-15T18:05:20.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.247+0000 7f5787e00640 1 -- 172.21.15.27:0/2211305857 learned_addr learned my addr 172.21.15.27:0/2211305857 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:20.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.247+0000 7f578dc00640 1 --2- 172.21.15.27:0/2211305857 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f57880c4270 0x7f57880cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:20.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.247+0000 7f5787e00640 1 -- 172.21.15.27:0/2211305857 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f57880c4270 msgr2=0x7f57880cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:20.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.247+0000 7f578d200640 1 --2- 172.21.15.27:0/2211305857 >> v2:172.21.15.93:3300/0 conn(0x7f57880c3e90 0x7f57880b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:20.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.247+0000 7f5787e00640 1 --2- 172.21.15.27:0/2211305857 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f57880c4270 0x7f57880cda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.247+0000 7f5787e00640 1 -- 172.21.15.27:0/2211305857 >> v2:172.21.15.93:3300/0 conn(0x7f57880c3e90 msgr2=0x7f57880b1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:20.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.247+0000 7f5787e00640 1 --2- 172.21.15.27:0/2211305857 >> v2:172.21.15.93:3300/0 conn(0x7f57880c3e90 0x7f57880b1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.247+0000 7f5787e00640 1 -- 172.21.15.27:0/2211305857 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5788166310 con 0x7f57880b23e0 2024-01-15T18:05:20.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.247+0000 7f578dc00640 1 --2- 172.21.15.27:0/2211305857 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f57880c4270 0x7f57880cda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:20.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.247+0000 7f578d200640 1 --2- 172.21.15.27:0/2211305857 >> v2:172.21.15.93:3300/0 conn(0x7f57880c3e90 0x7f57880b1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:20.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.247+0000 7f5787e00640 1 --2- 172.21.15.27:0/2211305857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57880b23e0 0x7f578806b600 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f57880720c0 tx=0x7f577001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ebe02657e8552 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:20.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.247+0000 7f5787400640 1 -- 172.21.15.27:0/2211305857 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57700ac440 con 0x7f57880b23e0 2024-01-15T18:05:20.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.247+0000 7f5787400640 1 -- 172.21.15.27:0/2211305857 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57700ac5a0 con 0x7f57880b23e0 2024-01-15T18:05:20.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.247+0000 7f5787400640 1 -- 172.21.15.27:0/2211305857 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57700ac930 con 0x7f57880b23e0 2024-01-15T18:05:20.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.251+0000 7f578fc00640 1 -- 172.21.15.27:0/2211305857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57880b23e0 msgr2=0x7f578806b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:20.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.251+0000 7f578fc00640 1 --2- 172.21.15.27:0/2211305857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57880b23e0 0x7f578806b600 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f57880720c0 tx=0x7f577001c440 comp rx=0 tx=0).stop 2024-01-15T18:05:20.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.251+0000 7f578fc00640 1 -- 172.21.15.27:0/2211305857 shutdown_connections 2024-01-15T18:05:20.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.251+0000 7f578fc00640 1 --2- 172.21.15.27:0/2211305857 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f57880c4270 0x7f57880cda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.251+0000 7f578fc00640 1 --2- 172.21.15.27:0/2211305857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57880b23e0 0x7f578806b600 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.251+0000 7f578fc00640 1 --2- 172.21.15.27:0/2211305857 >> v2:172.21.15.93:3300/0 conn(0x7f57880c3e90 0x7f57880b1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.251+0000 7f578fc00640 1 -- 172.21.15.27:0/2211305857 >> 172.21.15.27:0/2211305857 conn(0x7f57880b2a10 msgr2=0x7f57880b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:20.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.251+0000 7f578fc00640 1 -- 172.21.15.27:0/2211305857 shutdown_connections 2024-01-15T18:05:20.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.251+0000 7f578fc00640 1 -- 172.21.15.27:0/2211305857 wait complete. 2024-01-15T18:05:20.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.251+0000 7f578fc00640 1 Processor -- start 2024-01-15T18:05:20.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.251+0000 7f578fc00640 1 -- start start 2024-01-15T18:05:20.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.251+0000 7f578fc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f57880b23e0 0x7f578815c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:20.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.251+0000 7f578fc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57880c3e90 0x7f57880c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:20.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.251+0000 7f578fc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f57880c4270 0x7f57880c84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:20.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.251+0000 7f578fc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578815c6f0 con 0x7f57880c3e90 2024-01-15T18:05:20.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.251+0000 7f578fc00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5788162f90 con 0x7f57880c4270 2024-01-15T18:05:20.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.251+0000 7f578fc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57880cafc0 con 0x7f57880b23e0 2024-01-15T18:05:20.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.255+0000 7f5787e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57880c3e90 0x7f57880c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:20.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.255+0000 7f578dc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f57880c4270 0x7f57880c84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:20.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.255+0000 7f5787e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57880c3e90 0x7f57880c7f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39518/0 (socket says 172.21.15.27:39518) 2024-01-15T18:05:20.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.255+0000 7f5787e00640 1 -- 172.21.15.27:0/2611703562 learned_addr learned my addr 172.21.15.27:0/2611703562 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:20.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.255+0000 7f578d200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f57880b23e0 0x7f578815c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:20.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.255+0000 7f5787e00640 1 -- 172.21.15.27:0/2611703562 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f57880b23e0 msgr2=0x7f578815c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:20.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.255+0000 7f5787e00640 1 --2- 172.21.15.27:0/2611703562 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f57880b23e0 0x7f578815c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.255+0000 7f5787e00640 1 -- 172.21.15.27:0/2611703562 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f57880c4270 msgr2=0x7f57880c84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:20.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.255+0000 7f5787e00640 1 --2- 172.21.15.27:0/2611703562 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f57880c4270 0x7f57880c84d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.255+0000 7f5787e00640 1 -- 172.21.15.27:0/2611703562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57700094d0 con 0x7f57880c3e90 2024-01-15T18:05:20.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.255+0000 7f5787e00640 1 --2- 172.21.15.27:0/2611703562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57880c3e90 0x7f57880c7f90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f57700af040 tx=0x7f577001ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:20.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.255+0000 7f5784c00640 1 -- 172.21.15.27:0/2611703562 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57700b7830 con 0x7f57880c3e90 2024-01-15T18:05:20.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.255+0000 7f5784c00640 1 -- 172.21.15.27:0/2611703562 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57700b79b0 con 0x7f57880c3e90 2024-01-15T18:05:20.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.255+0000 7f578fc00640 1 -- 172.21.15.27:0/2611703562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57880b0780 con 0x7f57880c3e90 2024-01-15T18:05:20.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.255+0000 7f5784c00640 1 -- 172.21.15.27:0/2611703562 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57700b7d60 con 0x7f57880c3e90 2024-01-15T18:05:20.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.255+0000 7f578fc00640 1 -- 172.21.15.27:0/2611703562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57880b0d40 con 0x7f57880c3e90 2024-01-15T18:05:20.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.255+0000 7f578fc00640 1 -- 172.21.15.27:0/2611703562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f578804e850 con 0x7f57880c3e90 2024-01-15T18:05:20.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.255+0000 7f5784c00640 1 -- 172.21.15.27:0/2611703562 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f57700b3040 con 0x7f57880c3e90 2024-01-15T18:05:20.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.259+0000 7f5784c00640 1 --2- 172.21.15.27:0/2611703562 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f57600668f0 0x7f5760068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:20.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.259+0000 7f5784c00640 1 -- 172.21.15.27:0/2611703562 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f577011ec30 con 0x7f57880c3e90 2024-01-15T18:05:20.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.259+0000 7f578d200640 1 --2- 172.21.15.27:0/2611703562 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f57600668f0 0x7f5760068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:20.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.259+0000 7f578d200640 1 --2- 172.21.15.27:0/2611703562 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f57600668f0 0x7f5760068db0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f57881650b0 tx=0x7f5778008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:20.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.263+0000 7f5784c00640 1 -- 172.21.15.27:0/2611703562 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f57700ebe80 con 0x7f57880c3e90 2024-01-15T18:05:20.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.455+0000 7f578fc00640 1 -- 172.21.15.27:0/2611703562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f57880b17a0 con 0x7f57880c3e90 2024-01-15T18:05:20.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.455+0000 7f5784c00640 1 -- 172.21.15.27:0/2611703562 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f57700ef4a0 con 0x7f57880c3e90 2024-01-15T18:05:20.460 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:05:20.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.455+0000 7f578fc00640 1 -- 172.21.15.27:0/2611703562 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f57600668f0 msgr2=0x7f5760068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:20.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.455+0000 7f578fc00640 1 --2- 172.21.15.27:0/2611703562 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f57600668f0 0x7f5760068db0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f57881650b0 tx=0x7f5778008040 comp rx=0 tx=0).stop 2024-01-15T18:05:20.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.455+0000 7f578fc00640 1 -- 172.21.15.27:0/2611703562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57880c3e90 msgr2=0x7f57880c7f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:20.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.455+0000 7f578fc00640 1 --2- 172.21.15.27:0/2611703562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57880c3e90 0x7f57880c7f90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f57700af040 tx=0x7f577001ac80 comp rx=0 tx=0).stop 2024-01-15T18:05:20.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.455+0000 7f578fc00640 1 -- 172.21.15.27:0/2611703562 shutdown_connections 2024-01-15T18:05:20.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.455+0000 7f578fc00640 1 --2- 172.21.15.27:0/2611703562 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f57600668f0 0x7f5760068db0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.455+0000 7f578fc00640 1 --2- 172.21.15.27:0/2611703562 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f57880c4270 0x7f57880c84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.455+0000 7f578fc00640 1 --2- 172.21.15.27:0/2611703562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57880c3e90 0x7f57880c7f90 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.455+0000 7f578fc00640 1 --2- 172.21.15.27:0/2611703562 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f57880b23e0 0x7f578815c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.455+0000 7f578fc00640 1 -- 172.21.15.27:0/2611703562 >> 172.21.15.27:0/2611703562 conn(0x7f57880b2a10 msgr2=0x7f578806cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:20.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.455+0000 7f578fc00640 1 -- 172.21.15.27:0/2611703562 shutdown_connections 2024-01-15T18:05:20.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:20.455+0000 7f578fc00640 1 -- 172.21.15.27:0/2611703562 wait complete. 2024-01-15T18:05:20.482 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:05:20.482 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:05:20.482 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi093.front.sepia.ceph.com 2024-01-15T18:05:20.482 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-01-15T18:05:20.482 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-01-15T18:05:20.482 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:05:20.482 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:05:20.482 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:05:20.483 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-01-15T18:05:20.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:20.483 DEBUG:teuthology.orchestra.run.smithi093:> ip addr 2024-01-15T18:05:20.493 INFO:teuthology.orchestra.run.smithi093.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:05:20.493 INFO:teuthology.orchestra.run.smithi093.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:05:20.493 INFO:teuthology.orchestra.run.smithi093.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:05:20.493 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:20.493 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 ::1/128 scope host 2024-01-15T18:05:20.493 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:20.494 INFO:teuthology.orchestra.run.smithi093.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:05:20.494 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:d8:da:1c brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:20.494 INFO:teuthology.orchestra.run.smithi093.stdout: altname enp7s0f0 2024-01-15T18:05:20.494 INFO:teuthology.orchestra.run.smithi093.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:05:20.494 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:d8:da:1d brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:20.494 INFO:teuthology.orchestra.run.smithi093.stdout: altname enp7s0f1 2024-01-15T18:05:20.494 INFO:teuthology.orchestra.run.smithi093.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:05:20.494 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:8f:d6:34 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:20.494 INFO:teuthology.orchestra.run.smithi093.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:05:20.494 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:8f:d6:35 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:20.494 INFO:teuthology.orchestra.run.smithi093.stdout: inet 172.21.15.93/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-01-15T18:05:20.495 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft 42418sec preferred_lft 42418sec 2024-01-15T18:05:20.495 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::ec4:7aff:fe8f:d635/64 scope link 2024-01-15T18:05:20.495 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:20.495 INFO:teuthology.orchestra.run.smithi093.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-01-15T18:05:20.495 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 02:42:2a:61:fc:7d brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:20.495 INFO:teuthology.orchestra.run.smithi093.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-01-15T18:05:20.495 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:20.495 INFO:teuthology.orchestra.run.smithi093.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:05:20.495 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:20.495 INFO:teuthology.orchestra.run.smithi093.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:05:20.495 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:20.496 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-01-15T18:05:20.496 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:20.496 INFO:teuthology.orchestra.run.smithi093.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:05:20.496 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:05:20.496 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-01-15T18:05:20.496 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:20.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:20.496 DEBUG:teuthology.orchestra.run.smithi093:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:05:20.551 INFO:teuthology.orchestra.run.smithi093.stdout:1 2024-01-15T18:05:20.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:20.554 DEBUG:teuthology.orchestra.run.smithi093:> route 2024-01-15T18:05:20.609 INFO:teuthology.orchestra.run.smithi093.stdout:Kernel IP routing table 2024-01-15T18:05:20.609 INFO:teuthology.orchestra.run.smithi093.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:05:20.610 INFO:teuthology.orchestra.run.smithi093.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-01-15T18:05:20.610 INFO:teuthology.orchestra.run.smithi093.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T18:05:20.610 INFO:teuthology.orchestra.run.smithi093.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-01-15T18:05:20.610 INFO:teuthology.orchestra.run.smithi093.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T18:05:20.610 INFO:teuthology.orchestra.run.smithi093.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:05:20.610 INFO:teuthology.orchestra.run.smithi093.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:05:20.610 INFO:teuthology.orchestra.run.smithi093.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:05:20.610 INFO:teuthology.orchestra.run.smithi093.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:05:20.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:20.611 DEBUG:teuthology.orchestra.run.smithi093:> (cd / && exec bash -c ' 2024-01-15T18:05:20.611 DEBUG:teuthology.orchestra.run.smithi093:> set -e 2024-01-15T18:05:20.611 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T18:05:20.611 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T18:05:20.611 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T18:05:20.611 DEBUG:teuthology.orchestra.run.smithi093:> ') 2024-01-15T18:05:20.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:20.700 DEBUG:teuthology.orchestra.run.smithi093:> ip netns list 2024-01-15T18:05:20.709 INFO:teuthology.orchestra.run.smithi093.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-15T18:05:20.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:20.710 DEBUG:teuthology.orchestra.run.smithi093:> ip netns list-id 2024-01-15T18:05:20.762 INFO:teuthology.orchestra.run.smithi093.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-01-15T18:05:20.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:20.763 DEBUG:teuthology.orchestra.run.smithi093:> (cd / && exec bash -c ' 2024-01-15T18:05:20.763 DEBUG:teuthology.orchestra.run.smithi093:> set -e 2024-01-15T18:05:20.763 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:05:20.763 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-01-15T18:05:20.763 DEBUG:teuthology.orchestra.run.smithi093:> ') 2024-01-15T18:05:20.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:20.843 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-01-15T18:05:20.909 INFO:teuthology.orchestra.run.smithi093.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:05:20.909 INFO:teuthology.orchestra.run.smithi093.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:05:20.909 INFO:teuthology.orchestra.run.smithi093.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:05:20.909 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:20.909 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 ::1/128 scope host 2024-01-15T18:05:20.909 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:20.909 INFO:teuthology.orchestra.run.smithi093.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:05:20.909 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 72:a7:6b:86:44:d3 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-15T18:05:20.910 INFO:teuthology.orchestra.run.smithi093.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-15T18:05:20.910 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:20.910 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::70a7:6bff:fe86:44d3/64 scope link 2024-01-15T18:05:20.910 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:20.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:20.911 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-01-15T18:05:20.973 INFO:teuthology.orchestra.run.smithi093.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:05:20.974 INFO:teuthology.orchestra.run.smithi093.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:05:20.974 INFO:teuthology.orchestra.run.smithi093.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:05:20.974 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:20.974 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 ::1/128 scope host 2024-01-15T18:05:20.974 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:20.974 INFO:teuthology.orchestra.run.smithi093.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:05:20.974 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 72:a7:6b:86:44:d3 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-15T18:05:20.974 INFO:teuthology.orchestra.run.smithi093.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-15T18:05:20.974 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:20.974 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::70a7:6bff:fe86:44d3/64 scope link 2024-01-15T18:05:20.974 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:20.975 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-01-15T18:05:20.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:20.976 DEBUG:teuthology.orchestra.run.smithi093:> (cd / && exec bash -c ' 2024-01-15T18:05:20.976 DEBUG:teuthology.orchestra.run.smithi093:> set -e 2024-01-15T18:05:20.976 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-01-15T18:05:20.976 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-01-15T18:05:20.976 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-01-15T18:05:20.976 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-01-15T18:05:20.976 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-01-15T18:05:20.976 DEBUG:teuthology.orchestra.run.smithi093:> ') 2024-01-15T18:05:21.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:21.109 DEBUG:teuthology.orchestra.run.smithi093:> (cd / && exec bash -c ' 2024-01-15T18:05:21.109 DEBUG:teuthology.orchestra.run.smithi093:> set -e 2024-01-15T18:05:21.109 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip link set brx.1 up 2024-01-15T18:05:21.110 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip link set dev brx.1 master ceph-brx 2024-01-15T18:05:21.110 DEBUG:teuthology.orchestra.run.smithi093:> ') 2024-01-15T18:05:21.147 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-01-15T18:05:21.147 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:05:21.147 DEBUG:teuthology.orchestra.run.smithi093:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-01-15T18:05:21.202 INFO:teuthology.orchestra.run.smithi093.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-01-15T18:05:21.202 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:05:21.202 DEBUG:teuthology.orchestra.run.smithi093:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-01-15T18:05:21.258 DEBUG:teuthology.orchestra.run.smithi093:> sudo modprobe fuse 2024-01-15T18:05:21.325 DEBUG:teuthology.orchestra.run.smithi093:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:05:21.379 INFO:teuthology.orchestra.run.smithi093.stdout:/sys 2024-01-15T18:05:21.379 INFO:teuthology.orchestra.run.smithi093.stdout:/proc 2024-01-15T18:05:21.379 INFO:teuthology.orchestra.run.smithi093.stdout:/dev 2024-01-15T18:05:21.379 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/pts 2024-01-15T18:05:21.379 INFO:teuthology.orchestra.run.smithi093.stdout:/run 2024-01-15T18:05:21.379 INFO:teuthology.orchestra.run.smithi093.stdout:/ 2024-01-15T18:05:21.380 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/security 2024-01-15T18:05:21.380 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/shm 2024-01-15T18:05:21.380 INFO:teuthology.orchestra.run.smithi093.stdout:/run/lock 2024-01-15T18:05:21.380 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/cgroup 2024-01-15T18:05:21.380 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/pstore 2024-01-15T18:05:21.380 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/bpf 2024-01-15T18:05:21.380 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:05:21.380 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/hugepages 2024-01-15T18:05:21.380 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/mqueue 2024-01-15T18:05:21.380 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/debug 2024-01-15T18:05:21.380 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/tracing 2024-01-15T18:05:21.381 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/fs/nfsd 2024-01-15T18:05:21.381 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/fuse/connections 2024-01-15T18:05:21.381 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/config 2024-01-15T18:05:21.381 INFO:teuthology.orchestra.run.smithi093.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:05:21.381 INFO:teuthology.orchestra.run.smithi093.stdout:/run/rpc_pipefs 2024-01-15T18:05:21.381 INFO:teuthology.orchestra.run.smithi093.stdout:/run/user/1000 2024-01-15T18:05:21.381 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph 2024-01-15T18:05:21.381 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:05:21.381 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:05:21.381 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:05:21.381 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:05:21.381 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns 2024-01-15T18:05:21.382 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:05:21.382 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:05:21.382 INFO:teuthology.orchestra.run.smithi093.stdout:/home/ubuntu/cephtest/mnt.1 2024-01-15T18:05:21.382 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:05:21.382 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:05:21.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:21.383 DEBUG:teuthology.orchestra.run.smithi093:> ls /sys/fs/fuse/connections 2024-01-15T18:05:21.434 INFO:teuthology.orchestra.run.smithi093.stdout:43 2024-01-15T18:05:21.434 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-01-15T18:05:21.434 DEBUG:teuthology.orchestra.run.smithi093:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-01-15T18:05:21.481 DEBUG:teuthology.orchestra.run.smithi093:> sudo modprobe fuse 2024-01-15T18:05:21.502 DEBUG:teuthology.orchestra.run.smithi093:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:05:21.555 INFO:teuthology.orchestra.run.smithi093.stdout:/sys 2024-01-15T18:05:21.555 INFO:teuthology.orchestra.run.smithi093.stdout:/proc 2024-01-15T18:05:21.555 INFO:teuthology.orchestra.run.smithi093.stdout:/dev 2024-01-15T18:05:21.556 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/pts 2024-01-15T18:05:21.556 INFO:teuthology.orchestra.run.smithi093.stdout:/run 2024-01-15T18:05:21.556 INFO:teuthology.orchestra.run.smithi093.stdout:/ 2024-01-15T18:05:21.556 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/security 2024-01-15T18:05:21.556 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/shm 2024-01-15T18:05:21.556 INFO:teuthology.orchestra.run.smithi093.stdout:/run/lock 2024-01-15T18:05:21.556 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/cgroup 2024-01-15T18:05:21.556 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/pstore 2024-01-15T18:05:21.556 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/bpf 2024-01-15T18:05:21.556 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:05:21.557 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/hugepages 2024-01-15T18:05:21.557 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/mqueue 2024-01-15T18:05:21.557 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/debug 2024-01-15T18:05:21.557 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/tracing 2024-01-15T18:05:21.557 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/fs/nfsd 2024-01-15T18:05:21.557 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/fuse/connections 2024-01-15T18:05:21.557 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/config 2024-01-15T18:05:21.557 INFO:teuthology.orchestra.run.smithi093.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:05:21.557 INFO:teuthology.orchestra.run.smithi093.stdout:/run/rpc_pipefs 2024-01-15T18:05:21.557 INFO:teuthology.orchestra.run.smithi093.stdout:/run/user/1000 2024-01-15T18:05:21.557 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph 2024-01-15T18:05:21.558 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:05:21.558 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:05:21.558 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:05:21.558 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:05:21.558 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns 2024-01-15T18:05:21.558 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:05:21.558 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:05:21.558 INFO:teuthology.orchestra.run.smithi093.stdout:/home/ubuntu/cephtest/mnt.1 2024-01-15T18:05:21.558 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:05:21.558 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:05:21.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:21.560 DEBUG:teuthology.orchestra.run.smithi093:> ls /sys/fs/fuse/connections 2024-01-15T18:05:21.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi093.stderr:2024-01-15T18:05:21.570+0000 7f1810e45580 -1 init, newargv = 0x55ccae2001f0 newargc=15 2024-01-15T18:05:21.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi093.stderr:ceph-fuse[33130]: starting ceph client 2024-01-15T18:05:21.583 INFO:teuthology.orchestra.run.smithi093.stdout:43 2024-01-15T18:05:21.584 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi093.stderr:ceph-fuse[33130]: starting fuse 2024-01-15T18:05:22.685 DEBUG:teuthology.orchestra.run.smithi093:> sudo modprobe fuse 2024-01-15T18:05:22.709 DEBUG:teuthology.orchestra.run.smithi093:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:05:22.760 INFO:teuthology.orchestra.run.smithi093.stdout:/sys 2024-01-15T18:05:22.760 INFO:teuthology.orchestra.run.smithi093.stdout:/proc 2024-01-15T18:05:22.760 INFO:teuthology.orchestra.run.smithi093.stdout:/dev 2024-01-15T18:05:22.760 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/pts 2024-01-15T18:05:22.760 INFO:teuthology.orchestra.run.smithi093.stdout:/run 2024-01-15T18:05:22.761 INFO:teuthology.orchestra.run.smithi093.stdout:/ 2024-01-15T18:05:22.761 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/security 2024-01-15T18:05:22.761 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/shm 2024-01-15T18:05:22.761 INFO:teuthology.orchestra.run.smithi093.stdout:/run/lock 2024-01-15T18:05:22.761 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/cgroup 2024-01-15T18:05:22.761 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/pstore 2024-01-15T18:05:22.761 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/bpf 2024-01-15T18:05:22.761 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:05:22.761 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/hugepages 2024-01-15T18:05:22.761 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/mqueue 2024-01-15T18:05:22.761 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/debug 2024-01-15T18:05:22.761 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/tracing 2024-01-15T18:05:22.761 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/fs/nfsd 2024-01-15T18:05:22.762 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/fuse/connections 2024-01-15T18:05:22.762 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/config 2024-01-15T18:05:22.762 INFO:teuthology.orchestra.run.smithi093.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:05:22.762 INFO:teuthology.orchestra.run.smithi093.stdout:/run/rpc_pipefs 2024-01-15T18:05:22.762 INFO:teuthology.orchestra.run.smithi093.stdout:/run/user/1000 2024-01-15T18:05:22.762 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph 2024-01-15T18:05:22.762 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:05:22.762 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:05:22.762 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:05:22.762 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:05:22.762 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns 2024-01-15T18:05:22.762 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:05:22.762 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:05:22.763 INFO:teuthology.orchestra.run.smithi093.stdout:/home/ubuntu/cephtest/mnt.1 2024-01-15T18:05:22.763 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:05:22.763 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:05:22.763 INFO:teuthology.orchestra.run.smithi093.stdout:/home/ubuntu/cephtest/mnt.3 2024-01-15T18:05:22.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:22.764 DEBUG:teuthology.orchestra.run.smithi093:> ls /sys/fs/fuse/connections 2024-01-15T18:05:22.818 INFO:teuthology.orchestra.run.smithi093.stdout:43 2024-01-15T18:05:22.818 INFO:teuthology.orchestra.run.smithi093.stdout:44 2024-01-15T18:05:22.818 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-01-15T18:05:22.819 DEBUG:teuthology.orchestra.run.smithi093:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:05:22.819 DEBUG:teuthology.orchestra.run.smithi093:> import glob 2024-01-15T18:05:22.819 DEBUG:teuthology.orchestra.run.smithi093:> import re 2024-01-15T18:05:22.819 DEBUG:teuthology.orchestra.run.smithi093:> import os 2024-01-15T18:05:22.819 DEBUG:teuthology.orchestra.run.smithi093:> import subprocess 2024-01-15T18:05:22.819 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:05:22.819 DEBUG:teuthology.orchestra.run.smithi093:> def _find_admin_socket(client_name): 2024-01-15T18:05:22.819 DEBUG:teuthology.orchestra.run.smithi093:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-01-15T18:05:22.819 DEBUG:teuthology.orchestra.run.smithi093:> files = glob.glob(asok_path) 2024-01-15T18:05:22.819 DEBUG:teuthology.orchestra.run.smithi093:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-01-15T18:05:22.819 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:05:22.819 DEBUG:teuthology.orchestra.run.smithi093:> # Given a non-glob path, it better be there 2024-01-15T18:05:22.820 DEBUG:teuthology.orchestra.run.smithi093:> if "*" not in asok_path: 2024-01-15T18:05:22.820 DEBUG:teuthology.orchestra.run.smithi093:> assert(len(files) == 1) 2024-01-15T18:05:22.820 DEBUG:teuthology.orchestra.run.smithi093:> return files[0] 2024-01-15T18:05:22.820 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:05:22.820 DEBUG:teuthology.orchestra.run.smithi093:> for f in files: 2024-01-15T18:05:22.820 DEBUG:teuthology.orchestra.run.smithi093:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:05:22.820 DEBUG:teuthology.orchestra.run.smithi093:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:05:22.820 DEBUG:teuthology.orchestra.run.smithi093:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:05:22.820 DEBUG:teuthology.orchestra.run.smithi093:> contents = proc_f.read() 2024-01-15T18:05:22.820 DEBUG:teuthology.orchestra.run.smithi093:> if mountpoint in contents: 2024-01-15T18:05:22.820 DEBUG:teuthology.orchestra.run.smithi093:> return f 2024-01-15T18:05:22.820 DEBUG:teuthology.orchestra.run.smithi093:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:05:22.821 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:05:22.821 DEBUG:teuthology.orchestra.run.smithi093:> print(_find_admin_socket("client.3")) 2024-01-15T18:05:22.821 DEBUG:teuthology.orchestra.run.smithi093:> ' 2024-01-15T18:05:22.951 INFO:teuthology.orchestra.run.smithi093.stdout:/var/run/ceph/ceph-client.3.33130.asok 2024-01-15T18:05:22.955 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:05:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:05:22.963 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.33130.asok 2024-01-15T18:05:22.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:22.963 DEBUG:teuthology.orchestra.run.smithi093:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.33130.asok status 2024-01-15T18:05:23.137 INFO:teuthology.orchestra.run.smithi093.stdout:{ 2024-01-15T18:05:23.137 INFO:teuthology.orchestra.run.smithi093.stdout: "metadata": { 2024-01-15T18:05:23.137 INFO:teuthology.orchestra.run.smithi093.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:05:23.137 INFO:teuthology.orchestra.run.smithi093.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:05:23.137 INFO:teuthology.orchestra.run.smithi093.stdout: "entity_id": "3", 2024-01-15T18:05:23.138 INFO:teuthology.orchestra.run.smithi093.stdout: "hostname": "smithi093", 2024-01-15T18:05:23.138 INFO:teuthology.orchestra.run.smithi093.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-01-15T18:05:23.138 INFO:teuthology.orchestra.run.smithi093.stdout: "pid": "33130", 2024-01-15T18:05:23.138 INFO:teuthology.orchestra.run.smithi093.stdout: "root": "/" 2024-01-15T18:05:23.138 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:05:23.138 INFO:teuthology.orchestra.run.smithi093.stdout: "dentry_count": 0, 2024-01-15T18:05:23.138 INFO:teuthology.orchestra.run.smithi093.stdout: "dentry_pinned_count": 0, 2024-01-15T18:05:23.138 INFO:teuthology.orchestra.run.smithi093.stdout: "id": 4714, 2024-01-15T18:05:23.138 INFO:teuthology.orchestra.run.smithi093.stdout: "inst": { 2024-01-15T18:05:23.139 INFO:teuthology.orchestra.run.smithi093.stdout: "name": { 2024-01-15T18:05:23.139 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "client", 2024-01-15T18:05:23.139 INFO:teuthology.orchestra.run.smithi093.stdout: "num": 4714 2024-01-15T18:05:23.139 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:05:23.139 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": { 2024-01-15T18:05:23.139 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "v1", 2024-01-15T18:05:23.139 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": "192.168.0.2:0", 2024-01-15T18:05:23.139 INFO:teuthology.orchestra.run.smithi093.stdout: "nonce": 3706686464 2024-01-15T18:05:23.139 INFO:teuthology.orchestra.run.smithi093.stdout: } 2024-01-15T18:05:23.140 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:05:23.140 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": { 2024-01-15T18:05:23.140 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "v1", 2024-01-15T18:05:23.140 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": "192.168.0.2:0", 2024-01-15T18:05:23.140 INFO:teuthology.orchestra.run.smithi093.stdout: "nonce": 3706686464 2024-01-15T18:05:23.140 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:05:23.140 INFO:teuthology.orchestra.run.smithi093.stdout: "inst_str": "client.4714 192.168.0.2:0/3706686464", 2024-01-15T18:05:23.140 INFO:teuthology.orchestra.run.smithi093.stdout: "addr_str": "192.168.0.2:0/3706686464", 2024-01-15T18:05:23.140 INFO:teuthology.orchestra.run.smithi093.stdout: "inode_count": 1, 2024-01-15T18:05:23.141 INFO:teuthology.orchestra.run.smithi093.stdout: "mds_epoch": 8, 2024-01-15T18:05:23.141 INFO:teuthology.orchestra.run.smithi093.stdout: "osd_epoch": 21, 2024-01-15T18:05:23.141 INFO:teuthology.orchestra.run.smithi093.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:05:23.141 INFO:teuthology.orchestra.run.smithi093.stdout: "blocklisted": false, 2024-01-15T18:05:23.141 INFO:teuthology.orchestra.run.smithi093.stdout: "fs_name": "cephfs" 2024-01-15T18:05:23.141 INFO:teuthology.orchestra.run.smithi093.stdout:} 2024-01-15T18:05:23.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:23.152 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-01-15T18:05:23.152 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-15T18:05:23.163 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-01-15T18:05:23.163 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-15T18:05:23.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:23.164 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:05:23.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:23.230 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-01-15T18:05:23.231 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.2 2024-01-15T18:05:23.284 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-01-15T18:05:23.284 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-01-15T18:05:23.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:23.285 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-01-15T18:05:23.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:23.353 DEBUG:teuthology.orchestra.run.smithi093:> stat --file-system '--printf=%T 2024-01-15T18:05:23.353 DEBUG:teuthology.orchestra.run.smithi093:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-15T18:05:23.363 INFO:teuthology.orchestra.run.smithi093.stdout:fuseblk 2024-01-15T18:05:23.363 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-15T18:05:23.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:23.363 DEBUG:teuthology.orchestra.run.smithi093:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:05:23.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:23.437 DEBUG:teuthology.orchestra.run.smithi093:> stat --file-system '--printf=%T 2024-01-15T18:05:23.437 DEBUG:teuthology.orchestra.run.smithi093:> ' -- /home/ubuntu/cephtest/mnt.3 2024-01-15T18:05:23.478 INFO:teuthology.orchestra.run.smithi093.stdout:fuseblk 2024-01-15T18:05:23.478 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-01-15T18:05:23.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:23.479 DEBUG:teuthology.orchestra.run.smithi093:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-01-15T18:05:23.541 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-01-15T18:05:23.558 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:23.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.707+0000 7f7308e00640 1 Processor -- start 2024-01-15T18:05:23.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.707+0000 7f7308e00640 1 -- start start 2024-01-15T18:05:23.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.707+0000 7f7308e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f730406db80 0x7f730406df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:23.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.707+0000 7f7308e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f73040cdd10 0x7f7304070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:23.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.707+0000 7f7308e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73040ce920 con 0x7f730415e730 2024-01-15T18:05:23.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.707+0000 7f7308e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73040cdb30 con 0x7f73040cdd10 2024-01-15T18:05:23.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.707+0000 7f7308e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f730406e4a0 con 0x7f730406db80 2024-01-15T18:05:23.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.707+0000 7f7302800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f73040cdd10 0x7f7304070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:23.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.707+0000 7f7301400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f730415e730 0x7f73040cd420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34748/0 (socket says 172.21.15.27:34748) 2024-01-15T18:05:23.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.707+0000 7f7301400640 1 -- 172.21.15.27:0/2467824645 learned_addr learned my addr 172.21.15.27:0/2467824645 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:23.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.707+0000 7f7301e00640 1 --2- 172.21.15.27:0/2467824645 >> v2:172.21.15.93:3300/0 conn(0x7f730406db80 0x7f730406df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:23.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.707+0000 7f7302800640 1 -- 172.21.15.27:0/2467824645 >> v2:172.21.15.93:3300/0 conn(0x7f730406db80 msgr2=0x7f730406df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:23.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.707+0000 7f7302800640 1 --2- 172.21.15.27:0/2467824645 >> v2:172.21.15.93:3300/0 conn(0x7f730406db80 0x7f730406df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.707+0000 7f7302800640 1 -- 172.21.15.27:0/2467824645 >> v1:172.21.15.27:6789/0 conn(0x7f730415e730 legacy=0x7f73040cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:23.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.707+0000 7f7302800640 1 -- 172.21.15.27:0/2467824645 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73040ce640 con 0x7f73040cdd10 2024-01-15T18:05:23.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.707+0000 7f7300a00640 1 -- 172.21.15.27:0/2467824645 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3872118428 0 0) 0x7f73040ce920 con 0x7f730415e730 2024-01-15T18:05:23.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.711+0000 7f7302800640 1 --2- 172.21.15.27:0/2467824645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73040cdd10 0x7f7304070c20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f730406be30 tx=0x7f72fc01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22699775fb08541 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:23.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.711+0000 7f7300a00640 1 -- 172.21.15.27:0/2467824645 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc0a5970 con 0x7f73040cdd10 2024-01-15T18:05:23.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.711+0000 7f7300a00640 1 -- 172.21.15.27:0/2467824645 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72fc0a5ad0 con 0x7f73040cdd10 2024-01-15T18:05:23.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.711+0000 7f7300a00640 1 -- 172.21.15.27:0/2467824645 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc0a5e60 con 0x7f73040cdd10 2024-01-15T18:05:23.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.711+0000 7f7308e00640 1 -- 172.21.15.27:0/2467824645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73040cdd10 msgr2=0x7f7304070c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:23.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.711+0000 7f7308e00640 1 --2- 172.21.15.27:0/2467824645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73040cdd10 0x7f7304070c20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f730406be30 tx=0x7f72fc01c8e0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.711+0000 7f7308e00640 1 -- 172.21.15.27:0/2467824645 shutdown_connections 2024-01-15T18:05:23.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.711+0000 7f7308e00640 1 --2- 172.21.15.27:0/2467824645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73040cdd10 0x7f7304070c20 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.711+0000 7f7308e00640 1 --2- 172.21.15.27:0/2467824645 >> v2:172.21.15.93:3300/0 conn(0x7f730406db80 0x7f730406df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.711+0000 7f7308e00640 1 -- 172.21.15.27:0/2467824645 >> 172.21.15.27:0/2467824645 conn(0x7f73040cec60 msgr2=0x7f73040cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:23.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.711+0000 7f7308e00640 1 -- 172.21.15.27:0/2467824645 shutdown_connections 2024-01-15T18:05:23.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.711+0000 7f7308e00640 1 -- 172.21.15.27:0/2467824645 wait complete. 2024-01-15T18:05:23.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.715+0000 7f7308e00640 1 Processor -- start 2024-01-15T18:05:23.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.715+0000 7f7308e00640 1 -- start start 2024-01-15T18:05:23.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.715+0000 7f7308e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f730406db80 0x7f7304158bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:23.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.715+0000 7f7308e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f73040cdd10 0x7f73041f59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:23.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.715+0000 7f7308e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730415e730 0x7f730415f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:23.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.715+0000 7f7308e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73041f70e0 con 0x7f730415e730 2024-01-15T18:05:23.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.715+0000 7f7308e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73041590f0 con 0x7f730406db80 2024-01-15T18:05:23.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.715+0000 7f7308e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73041f64d0 con 0x7f73040cdd10 2024-01-15T18:05:23.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.715+0000 7f7302800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730415e730 0x7f730415f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:23.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.715+0000 7f7302800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730415e730 0x7f730415f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40490/0 (socket says 172.21.15.27:40490) 2024-01-15T18:05:23.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.715+0000 7f7302800640 1 -- 172.21.15.27:0/2112366295 learned_addr learned my addr 172.21.15.27:0/2112366295 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:23.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.715+0000 7f7301e00640 1 --2- 172.21.15.27:0/2112366295 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f730406db80 0x7f7304158bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:23.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.715+0000 7f7301400640 1 --2- 172.21.15.27:0/2112366295 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f73040cdd10 0x7f73041f59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:23.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.715+0000 7f7302800640 1 -- 172.21.15.27:0/2112366295 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f73040cdd10 msgr2=0x7f73041f59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:23.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.715+0000 7f7302800640 1 --2- 172.21.15.27:0/2112366295 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f73040cdd10 0x7f73041f59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.715+0000 7f7302800640 1 -- 172.21.15.27:0/2112366295 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f730406db80 msgr2=0x7f7304158bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:23.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.715+0000 7f7302800640 1 --2- 172.21.15.27:0/2112366295 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f730406db80 0x7f7304158bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.715+0000 7f7302800640 1 -- 172.21.15.27:0/2112366295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc009a70 con 0x7f730415e730 2024-01-15T18:05:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.715+0000 7f7302800640 1 --2- 172.21.15.27:0/2112366295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730415e730 0x7f730415f390 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f72fc0194f0 tx=0x7f72fc019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.715+0000 7f72fa000640 1 -- 172.21.15.27:0/2112366295 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc0aee00 con 0x7f730415e730 2024-01-15T18:05:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.715+0000 7f72fa000640 1 -- 172.21.15.27:0/2112366295 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72fc009450 con 0x7f730415e730 2024-01-15T18:05:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.715+0000 7f7308e00640 1 -- 172.21.15.27:0/2112366295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73041f67b0 con 0x7f730415e730 2024-01-15T18:05:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.715+0000 7f72fa000640 1 -- 172.21.15.27:0/2112366295 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc0b7610 con 0x7f730415e730 2024-01-15T18:05:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.715+0000 7f7308e00640 1 -- 172.21.15.27:0/2112366295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73041f6d70 con 0x7f730415e730 2024-01-15T18:05:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.715+0000 7f7308e00640 1 -- 172.21.15.27:0/2112366295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73041f7250 con 0x7f730415e730 2024-01-15T18:05:23.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.719+0000 7f72fa000640 1 -- 172.21.15.27:0/2112366295 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f72fc0b4050 con 0x7f730415e730 2024-01-15T18:05:23.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.719+0000 7f72fa000640 1 --2- 172.21.15.27:0/2112366295 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f72dc0668f0 0x7f72dc068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:23.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.719+0000 7f72fa000640 1 -- 172.21.15.27:0/2112366295 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f72fc11f360 con 0x7f730415e730 2024-01-15T18:05:23.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.719+0000 7f7301e00640 1 --2- 172.21.15.27:0/2112366295 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f72dc0668f0 0x7f72dc068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:23.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.719+0000 7f7301e00640 1 --2- 172.21.15.27:0/2112366295 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f72dc0668f0 0x7f72dc068db0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f73040b0760 tx=0x7f72f4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:23.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.723+0000 7f72fa000640 1 -- 172.21.15.27:0/2112366295 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f72fc0ec640 con 0x7f730415e730 2024-01-15T18:05:23.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.883+0000 7f7308e00640 1 -- 172.21.15.27:0/2112366295 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f730404e850 con 0x7f730415e730 2024-01-15T18:05:23.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.883+0000 7f72fa000640 1 -- 172.21.15.27:0/2112366295 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16409 (secure 0 0 0) 0x7f72fc0efc60 con 0x7f730415e730 2024-01-15T18:05:23.889 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:23.889 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:04:59.837052+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:04:56.220608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T18:04:57.210421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:23.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.887+0000 7f7308e00640 1 -- 172.21.15.27:0/2112366295 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f72dc0668f0 msgr2=0x7f72dc068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:23.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.887+0000 7f7308e00640 1 --2- 172.21.15.27:0/2112366295 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f72dc0668f0 0x7f72dc068db0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f73040b0760 tx=0x7f72f4009190 comp rx=0 tx=0).stop 2024-01-15T18:05:23.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.887+0000 7f7308e00640 1 -- 172.21.15.27:0/2112366295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730415e730 msgr2=0x7f730415f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:23.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.887+0000 7f7308e00640 1 --2- 172.21.15.27:0/2112366295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730415e730 0x7f730415f390 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f72fc0194f0 tx=0x7f72fc019680 comp rx=0 tx=0).stop 2024-01-15T18:05:23.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.887+0000 7f7308e00640 1 -- 172.21.15.27:0/2112366295 shutdown_connections 2024-01-15T18:05:23.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.887+0000 7f7308e00640 1 --2- 172.21.15.27:0/2112366295 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f72dc0668f0 0x7f72dc068db0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.887+0000 7f7308e00640 1 --2- 172.21.15.27:0/2112366295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730415e730 0x7f730415f390 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.887+0000 7f7308e00640 1 --2- 172.21.15.27:0/2112366295 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f73040cdd10 0x7f73041f59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.887+0000 7f7308e00640 1 --2- 172.21.15.27:0/2112366295 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f730406db80 0x7f7304158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.887+0000 7f7308e00640 1 -- 172.21.15.27:0/2112366295 >> 172.21.15.27:0/2112366295 conn(0x7f73040cec60 msgr2=0x7f73040aedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:23.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.887+0000 7f7308e00640 1 -- 172.21.15.27:0/2112366295 shutdown_connections 2024-01-15T18:05:23.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:23.887+0000 7f7308e00640 1 -- 172.21.15.27:0/2112366295 wait complete. 2024-01-15T18:05:23.911 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T18:04:56.220608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T18:04:57.210421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:05:23.912 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:05:24.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.051+0000 7f0103600640 1 Processor -- start 2024-01-15T18:05:24.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.051+0000 7f0103600640 1 -- start start 2024-01-15T18:05:24.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.051+0000 7f0103600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f00fc0b1db0 0x7f00fc0b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:24.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.051+0000 7f0103600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f00fc0ce050 0x7f00fc0ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:24.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.051+0000 7f0103600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00fc0712b0 con 0x7f00fc06c430 2024-01-15T18:05:24.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.051+0000 7f0103600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00fc0cea20 con 0x7f00fc0ce050 2024-01-15T18:05:24.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.051+0000 7f0103600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc0ccfb0 con 0x7f00fc0b1db0 2024-01-15T18:05:24.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.051+0000 7f00fbe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f00fc0b1db0 0x7f00fc0b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:24.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.051+0000 7f0101600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f00fc0ce050 0x7f00fc0ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:24.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.055+0000 7f0101600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f00fc0ce050 0x7f00fc0ce4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:53684/0 (socket says 172.21.15.27:53684) 2024-01-15T18:05:24.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.055+0000 7f0101600640 1 -- 172.21.15.27:0/1626149058 learned_addr learned my addr 172.21.15.27:0/1626149058 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:24.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.055+0000 7f00fbe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f00fc0b1db0 0x7f00fc0b2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:53852/0 (socket says 172.21.15.27:53852) 2024-01-15T18:05:24.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.055+0000 7f0100c00640 1 --1- 172.21.15.27:0/1626149058 >> v1:172.21.15.93:6789/0 conn(0x7f00fc06c430 0x7f00fc070ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:57628/0 (socket says 172.21.15.27:57628) 2024-01-15T18:05:24.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.055+0000 7f00fbe00640 1 -- 172.21.15.27:0/1626149058 >> v2:172.21.15.93:3300/0 conn(0x7f00fc0ce050 msgr2=0x7f00fc0ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:24.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.055+0000 7f00fbe00640 1 --2- 172.21.15.27:0/1626149058 >> v2:172.21.15.93:3300/0 conn(0x7f00fc0ce050 0x7f00fc0ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.055+0000 7f00fbe00640 1 -- 172.21.15.27:0/1626149058 >> v1:172.21.15.93:6789/0 conn(0x7f00fc06c430 legacy=0x7f00fc070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:24.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.055+0000 7f00fbe00640 1 -- 172.21.15.27:0/1626149058 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00fc0cd290 con 0x7f00fc0b1db0 2024-01-15T18:05:24.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.055+0000 7f0101600640 1 --2- 172.21.15.27:0/1626149058 >> v2:172.21.15.93:3300/0 conn(0x7f00fc0ce050 0x7f00fc0ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:24.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.055+0000 7f00fbe00640 1 --2- 172.21.15.27:0/1626149058 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f00fc0b1db0 0x7f00fc0b2210 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f00e40097c0 tx=0x7f00e401c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3467b81ab931c1dc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:24.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.055+0000 7f00fb400640 1 -- 172.21.15.27:0/1626149058 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e40ab600 con 0x7f00fc0b1db0 2024-01-15T18:05:24.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.055+0000 7f00fb400640 1 -- 172.21.15.27:0/1626149058 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e40a7050 con 0x7f00fc0b1db0 2024-01-15T18:05:24.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.055+0000 7f00fb400640 1 -- 172.21.15.27:0/1626149058 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e40ab980 con 0x7f00fc0b1db0 2024-01-15T18:05:24.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.059+0000 7f0103600640 1 -- 172.21.15.27:0/1626149058 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f00fc0b1db0 msgr2=0x7f00fc0b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:24.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.059+0000 7f0103600640 1 --2- 172.21.15.27:0/1626149058 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f00fc0b1db0 0x7f00fc0b2210 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f00e40097c0 tx=0x7f00e401c3f0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.059+0000 7f0103600640 1 -- 172.21.15.27:0/1626149058 shutdown_connections 2024-01-15T18:05:24.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.059+0000 7f0103600640 1 --2- 172.21.15.27:0/1626149058 >> v2:172.21.15.93:3300/0 conn(0x7f00fc0ce050 0x7f00fc0ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.059+0000 7f0103600640 1 --2- 172.21.15.27:0/1626149058 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f00fc0b1db0 0x7f00fc0b2210 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.059+0000 7f0103600640 1 -- 172.21.15.27:0/1626149058 >> 172.21.15.27:0/1626149058 conn(0x7f00fc0cb700 msgr2=0x7f00fc0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:24.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.059+0000 7f0103600640 1 -- 172.21.15.27:0/1626149058 shutdown_connections 2024-01-15T18:05:24.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.059+0000 7f0103600640 1 -- 172.21.15.27:0/1626149058 wait complete. 2024-01-15T18:05:24.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.059+0000 7f0103600640 1 Processor -- start 2024-01-15T18:05:24.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.059+0000 7f0103600640 1 -- start start 2024-01-15T18:05:24.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.059+0000 7f0103600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00fc06c430 0x7f00fc14fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:24.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.059+0000 7f0103600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f00fc0b1db0 0x7f00fc1484c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:24.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.059+0000 7f0103600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f00fc0ce050 0x7f00fc0ad890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:24.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.059+0000 7f0103600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc152ea0 con 0x7f00fc06c430 2024-01-15T18:05:24.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.059+0000 7f0103600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc159ce0 con 0x7f00fc0b1db0 2024-01-15T18:05:24.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.059+0000 7f0103600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc1502a0 con 0x7f00fc0ce050 2024-01-15T18:05:24.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.059+0000 7f0100c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00fc06c430 0x7f00fc14fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:24.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.059+0000 7f0100c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00fc06c430 0x7f00fc14fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40492/0 (socket says 172.21.15.27:40492) 2024-01-15T18:05:24.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.059+0000 7f0100c00640 1 -- 172.21.15.27:0/1701494282 learned_addr learned my addr 172.21.15.27:0/1701494282 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:24.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.063+0000 7f00fbe00640 1 --2- 172.21.15.27:0/1701494282 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f00fc0b1db0 0x7f00fc1484c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:24.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.063+0000 7f0101600640 1 --2- 172.21.15.27:0/1701494282 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f00fc0ce050 0x7f00fc0ad890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:24.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.063+0000 7f0100c00640 1 -- 172.21.15.27:0/1701494282 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f00fc0ce050 msgr2=0x7f00fc0ad890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:24.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.063+0000 7f0100c00640 1 --2- 172.21.15.27:0/1701494282 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f00fc0ce050 0x7f00fc0ad890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.063+0000 7f0100c00640 1 -- 172.21.15.27:0/1701494282 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f00fc0b1db0 msgr2=0x7f00fc1484c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:24.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.063+0000 7f0100c00640 1 --2- 172.21.15.27:0/1701494282 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f00fc0b1db0 0x7f00fc1484c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.063+0000 7f0100c00640 1 -- 172.21.15.27:0/1701494282 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f4009b30 con 0x7f00fc06c430 2024-01-15T18:05:24.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.063+0000 7f0100c00640 1 --2- 172.21.15.27:0/1701494282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00fc06c430 0x7f00fc14fd60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f00ec00c490 tx=0x7f00ec00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:24.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.063+0000 7f00f8c00640 1 -- 172.21.15.27:0/1701494282 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ec007480 con 0x7f00fc06c430 2024-01-15T18:05:24.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.063+0000 7f00f8c00640 1 -- 172.21.15.27:0/1701494282 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00ec0075e0 con 0x7f00fc06c430 2024-01-15T18:05:24.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.063+0000 7f00f8c00640 1 -- 172.21.15.27:0/1701494282 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ec004a10 con 0x7f00fc06c430 2024-01-15T18:05:24.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.063+0000 7f0103600640 1 -- 172.21.15.27:0/1701494282 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00e4009420 con 0x7f00fc06c430 2024-01-15T18:05:24.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.063+0000 7f0103600640 1 -- 172.21.15.27:0/1701494282 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00fc1528e0 con 0x7f00fc06c430 2024-01-15T18:05:24.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.063+0000 7f0103600640 1 -- 172.21.15.27:0/1701494282 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00fc04e850 con 0x7f00fc06c430 2024-01-15T18:05:24.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.063+0000 7f00f8c00640 1 -- 172.21.15.27:0/1701494282 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f00ec004b70 con 0x7f00fc06c430 2024-01-15T18:05:24.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.063+0000 7f00f8c00640 1 --2- 172.21.15.27:0/1701494282 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f00d40668f0 0x7f00d4068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:24.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.067+0000 7f00f8c00640 1 -- 172.21.15.27:0/1701494282 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f00ec0834b0 con 0x7f00fc06c430 2024-01-15T18:05:24.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.067+0000 7f00fbe00640 1 --2- 172.21.15.27:0/1701494282 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f00d40668f0 0x7f00d4068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:24.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.067+0000 7f00fbe00640 1 --2- 172.21.15.27:0/1701494282 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f00d40668f0 0x7f00d4068db0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f00e40ad000 tx=0x7f00e4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:24.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.071+0000 7f00f8c00640 1 -- 172.21.15.27:0/1701494282 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f00ec050830 con 0x7f00fc06c430 2024-01-15T18:05:24.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.231+0000 7f0103600640 1 -- 172.21.15.27:0/1701494282 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f00fc15ced0 con 0x7f00fc06c430 2024-01-15T18:05:24.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.231+0000 7f00f8c00640 1 -- 172.21.15.27:0/1701494282 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f00ec018020 con 0x7f00fc06c430 2024-01-15T18:05:24.236 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:05:24.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.231+0000 7f0103600640 1 -- 172.21.15.27:0/1701494282 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f00d40668f0 msgr2=0x7f00d4068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:24.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.231+0000 7f0103600640 1 --2- 172.21.15.27:0/1701494282 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f00d40668f0 0x7f00d4068db0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f00e40ad000 tx=0x7f00e4008040 comp rx=0 tx=0).stop 2024-01-15T18:05:24.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.231+0000 7f0103600640 1 -- 172.21.15.27:0/1701494282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00fc06c430 msgr2=0x7f00fc14fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:24.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.231+0000 7f0103600640 1 --2- 172.21.15.27:0/1701494282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00fc06c430 0x7f00fc14fd60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f00ec00c490 tx=0x7f00ec00c950 comp rx=0 tx=0).stop 2024-01-15T18:05:24.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.231+0000 7f0103600640 1 -- 172.21.15.27:0/1701494282 shutdown_connections 2024-01-15T18:05:24.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.231+0000 7f0103600640 1 --2- 172.21.15.27:0/1701494282 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f00d40668f0 0x7f00d4068db0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.231+0000 7f0103600640 1 --2- 172.21.15.27:0/1701494282 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f00fc0ce050 0x7f00fc0ad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.231+0000 7f0103600640 1 --2- 172.21.15.27:0/1701494282 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f00fc0b1db0 0x7f00fc1484c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.231+0000 7f0103600640 1 --2- 172.21.15.27:0/1701494282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00fc06c430 0x7f00fc14fd60 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.231+0000 7f0103600640 1 -- 172.21.15.27:0/1701494282 >> 172.21.15.27:0/1701494282 conn(0x7f00fc0cb700 msgr2=0x7f00fc070850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:24.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.231+0000 7f0103600640 1 -- 172.21.15.27:0/1701494282 shutdown_connections 2024-01-15T18:05:24.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.231+0000 7f0103600640 1 -- 172.21.15.27:0/1701494282 wait complete. 2024-01-15T18:05:24.258 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T18:05:24.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.395+0000 7f9e7f200640 1 Processor -- start 2024-01-15T18:05:24.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.395+0000 7f9e7f200640 1 -- start start 2024-01-15T18:05:24.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.395+0000 7f9e7f200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f9e7815b4b0 0x7f9e780b1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:24.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.395+0000 7f9e7f200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9e780cec60 0x7f9e780b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:24.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.395+0000 7f9e7f200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e780d19a0 con 0x7f9e780cf460 2024-01-15T18:05:24.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.395+0000 7f9e7f200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e780d1c80 con 0x7f9e7815b4b0 2024-01-15T18:05:24.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.395+0000 7f9e7f200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e780b07c0 con 0x7f9e780cec60 2024-01-15T18:05:24.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.399+0000 7f9e7d200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9e780cec60 0x7f9e780b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:24.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.399+0000 7f9e77400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f9e7815b4b0 0x7f9e780b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:24.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.399+0000 7f9e7d200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9e780cec60 0x7f9e780b1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:53860/0 (socket says 172.21.15.27:53860) 2024-01-15T18:05:24.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.399+0000 7f9e7d200640 1 -- 172.21.15.27:0/3827890142 learned_addr learned my addr 172.21.15.27:0/3827890142 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:24.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.399+0000 7f9e77400640 1 -- 172.21.15.27:0/3827890142 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9e780cec60 msgr2=0x7f9e780b1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:24.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.399+0000 7f9e77400640 1 --2- 172.21.15.27:0/3827890142 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9e780cec60 0x7f9e780b1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.399+0000 7f9e77400640 1 -- 172.21.15.27:0/3827890142 >> v1:172.21.15.93:6789/0 conn(0x7f9e780cf460 legacy=0x7f9e780cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:24.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.399+0000 7f9e77400640 1 -- 172.21.15.27:0/3827890142 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e780b0aa0 con 0x7f9e7815b4b0 2024-01-15T18:05:24.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.399+0000 7f9e77400640 1 --2- 172.21.15.27:0/3827890142 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e7815b4b0 0x7f9e780b1580 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9e600097c0 tx=0x7f9e6001c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dc107d240ce0f0e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:24.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.403+0000 7f9e76a00640 1 -- 172.21.15.27:0/3827890142 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e780b2810 con 0x7f9e7815b4b0 2024-01-15T18:05:24.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.403+0000 7f9e76a00640 1 -- 172.21.15.27:0/3827890142 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e600ab680 con 0x7f9e7815b4b0 2024-01-15T18:05:24.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.403+0000 7f9e76a00640 1 -- 172.21.15.27:0/3827890142 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e600aba10 con 0x7f9e7815b4b0 2024-01-15T18:05:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.403+0000 7f9e7f200640 1 -- 172.21.15.27:0/3827890142 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e7815b4b0 msgr2=0x7f9e780b1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.403+0000 7f9e7f200640 1 --2- 172.21.15.27:0/3827890142 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e7815b4b0 0x7f9e780b1580 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9e600097c0 tx=0x7f9e6001c480 comp rx=0 tx=0).stop 2024-01-15T18:05:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.403+0000 7f9e7f200640 1 -- 172.21.15.27:0/3827890142 shutdown_connections 2024-01-15T18:05:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.403+0000 7f9e7f200640 1 --2- 172.21.15.27:0/3827890142 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9e780cec60 0x7f9e780b1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.403+0000 7f9e7f200640 1 --2- 172.21.15.27:0/3827890142 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e7815b4b0 0x7f9e780b1580 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.403+0000 7f9e7f200640 1 -- 172.21.15.27:0/3827890142 >> 172.21.15.27:0/3827890142 conn(0x7f9e780d0e80 msgr2=0x7f9e780d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:24.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.403+0000 7f9e7f200640 1 -- 172.21.15.27:0/3827890142 shutdown_connections 2024-01-15T18:05:24.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.403+0000 7f9e7f200640 1 -- 172.21.15.27:0/3827890142 wait complete. 2024-01-15T18:05:24.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.403+0000 7f9e7f200640 1 Processor -- start 2024-01-15T18:05:24.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.403+0000 7f9e7f200640 1 -- start start 2024-01-15T18:05:24.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.403+0000 7f9e7f200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e780cec60 0x7f9e780c8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:24.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.403+0000 7f9e7f200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e780cf460 0x7f9e780c6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:24.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.403+0000 7f9e7f200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9e7815b4b0 0x7f9e78149cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:24.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.403+0000 7f9e7f200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e780c7150 con 0x7f9e780cf460 2024-01-15T18:05:24.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.403+0000 7f9e7f200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7814a7f0 con 0x7f9e780cec60 2024-01-15T18:05:24.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.403+0000 7f9e7f200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e78148c30 con 0x7f9e7815b4b0 2024-01-15T18:05:24.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.407+0000 7f9e77400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e780cf460 0x7f9e780c6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:24.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.407+0000 7f9e77400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e780cf460 0x7f9e780c6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40500/0 (socket says 172.21.15.27:40500) 2024-01-15T18:05:24.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.407+0000 7f9e77400640 1 -- 172.21.15.27:0/108834913 learned_addr learned my addr 172.21.15.27:0/108834913 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:24.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.407+0000 7f9e7d200640 1 --2- 172.21.15.27:0/108834913 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9e7815b4b0 0x7f9e78149cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:24.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.407+0000 7f9e77e00640 1 --2- 172.21.15.27:0/108834913 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e780cec60 0x7f9e780c8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:24.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.407+0000 7f9e77400640 1 -- 172.21.15.27:0/108834913 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9e7815b4b0 msgr2=0x7f9e78149cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:24.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.407+0000 7f9e77400640 1 --2- 172.21.15.27:0/108834913 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9e7815b4b0 0x7f9e78149cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.407+0000 7f9e77400640 1 -- 172.21.15.27:0/108834913 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e780cec60 msgr2=0x7f9e780c8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:24.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.407+0000 7f9e77400640 1 --2- 172.21.15.27:0/108834913 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e780cec60 0x7f9e780c8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.407+0000 7f9e77400640 1 -- 172.21.15.27:0/108834913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e60009420 con 0x7f9e780cf460 2024-01-15T18:05:24.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.407+0000 7f9e77400640 1 --2- 172.21.15.27:0/108834913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e780cf460 0x7f9e780c6c10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9e600ad040 tx=0x7f9e6001acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.407+0000 7f9e6fe00640 1 -- 172.21.15.27:0/108834913 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e600b5610 con 0x7f9e780cf460 2024-01-15T18:05:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.407+0000 7f9e6fe00640 1 -- 172.21.15.27:0/108834913 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e600b5790 con 0x7f9e780cf460 2024-01-15T18:05:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.407+0000 7f9e7f200640 1 -- 172.21.15.27:0/108834913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e78148f10 con 0x7f9e780cf460 2024-01-15T18:05:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.407+0000 7f9e6fe00640 1 -- 172.21.15.27:0/108834913 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e600b5b40 con 0x7f9e780cf460 2024-01-15T18:05:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.407+0000 7f9e7f200640 1 -- 172.21.15.27:0/108834913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e781494d0 con 0x7f9e780cf460 2024-01-15T18:05:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.407+0000 7f9e7f200640 1 -- 172.21.15.27:0/108834913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e7804e850 con 0x7f9e780cf460 2024-01-15T18:05:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.407+0000 7f9e6fe00640 1 -- 172.21.15.27:0/108834913 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f9e600b5ca0 con 0x7f9e780cf460 2024-01-15T18:05:24.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.411+0000 7f9e6fe00640 1 --2- 172.21.15.27:0/108834913 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9e500668f0 0x7f9e50068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:24.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.411+0000 7f9e6fe00640 1 -- 172.21.15.27:0/108834913 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e6011ef60 con 0x7f9e780cf460 2024-01-15T18:05:24.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.411+0000 7f9e77e00640 1 --2- 172.21.15.27:0/108834913 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9e500668f0 0x7f9e50068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:24.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.411+0000 7f9e77e00640 1 --2- 172.21.15.27:0/108834913 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9e500668f0 0x7f9e50068db0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9e780cc990 tx=0x7f9e68009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.415+0000 7f9e6fe00640 1 -- 172.21.15.27:0/108834913 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9e600ec1c0 con 0x7f9e780cf460 2024-01-15T18:05:24.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.571+0000 7f9e7f200640 1 -- 172.21.15.27:0/108834913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9e780cf240 con 0x7f9e780cf460 2024-01-15T18:05:24.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.571+0000 7f9e6fe00640 1 -- 172.21.15.27:0/108834913 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9e600ef7e0 con 0x7f9e780cf460 2024-01-15T18:05:24.575 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-15T18:05:24.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.571+0000 7f9e7f200640 1 -- 172.21.15.27:0/108834913 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9e500668f0 msgr2=0x7f9e50068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:24.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.571+0000 7f9e7f200640 1 --2- 172.21.15.27:0/108834913 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9e500668f0 0x7f9e50068db0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9e780cc990 tx=0x7f9e68009040 comp rx=0 tx=0).stop 2024-01-15T18:05:24.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.571+0000 7f9e7f200640 1 -- 172.21.15.27:0/108834913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e780cf460 msgr2=0x7f9e780c6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:24.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.571+0000 7f9e7f200640 1 --2- 172.21.15.27:0/108834913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e780cf460 0x7f9e780c6c10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9e600ad040 tx=0x7f9e6001acc0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.571+0000 7f9e7f200640 1 -- 172.21.15.27:0/108834913 shutdown_connections 2024-01-15T18:05:24.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.571+0000 7f9e7f200640 1 --2- 172.21.15.27:0/108834913 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9e500668f0 0x7f9e50068db0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.571+0000 7f9e7f200640 1 --2- 172.21.15.27:0/108834913 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9e7815b4b0 0x7f9e78149cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.571+0000 7f9e7f200640 1 --2- 172.21.15.27:0/108834913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e780cf460 0x7f9e780c6c10 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.571+0000 7f9e7f200640 1 --2- 172.21.15.27:0/108834913 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e780cec60 0x7f9e780c8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.571+0000 7f9e7f200640 1 -- 172.21.15.27:0/108834913 >> 172.21.15.27:0/108834913 conn(0x7f9e780d0e80 msgr2=0x7f9e780c7290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:24.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.571+0000 7f9e7f200640 1 -- 172.21.15.27:0/108834913 shutdown_connections 2024-01-15T18:05:24.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.571+0000 7f9e7f200640 1 -- 172.21.15.27:0/108834913 wait complete. 2024-01-15T18:05:24.597 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T18:05:24.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.735+0000 7fd4c6800640 1 Processor -- start 2024-01-15T18:05:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.735+0000 7fd4c6800640 1 -- start start 2024-01-15T18:05:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.735+0000 7fd4c6800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4c00b1db0 0x7fd4c00b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:24.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.735+0000 7fd4c6800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c00712b0 con 0x7fd4c00ce050 2024-01-15T18:05:24.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.735+0000 7fd4c6800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4c00ccfb0 con 0x7fd4c006c430 2024-01-15T18:05:24.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.735+0000 7fd4c6800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c00cd290 con 0x7fd4c00b1db0 2024-01-15T18:05:24.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.735+0000 7fd4bfe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd4c00ce050 0x7fd4c006c810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34760/0 (socket says 172.21.15.27:34760) 2024-01-15T18:05:24.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.735+0000 7fd4bfe00640 1 -- 172.21.15.27:0/1988733442 learned_addr learned my addr 172.21.15.27:0/1988733442 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:24.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.735+0000 7fd4bea00640 1 --2- 172.21.15.27:0/1988733442 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4c00b1db0 0x7fd4c00b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:24.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.735+0000 7fd4be000640 1 -- 172.21.15.27:0/1988733442 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 926093137 0 0) 0x7fd4c00712b0 con 0x7fd4c00ce050 2024-01-15T18:05:24.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.735+0000 7fd4be000640 1 -- 172.21.15.27:0/1988733442 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4a0003530 con 0x7fd4c00ce050 2024-01-15T18:05:24.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.735+0000 7fd4be000640 1 -- 172.21.15.27:0/1988733442 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2654668551 0 0) 0x7fd4a0003530 con 0x7fd4c00ce050 2024-01-15T18:05:24.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.735+0000 7fd4be000640 1 -- 172.21.15.27:0/1988733442 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4c00b1db0 msgr2=0x7fd4c00b2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:24.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.735+0000 7fd4be000640 1 --2- 172.21.15.27:0/1988733442 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4c00b1db0 0x7fd4c00b2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.735+0000 7fd4be000640 1 -- 172.21.15.27:0/1988733442 >> v1:172.21.15.93:6789/0 conn(0x7fd4c006c430 legacy=0x7fd4c0070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:24.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.735+0000 7fd4be000640 1 -- 172.21.15.27:0/1988733442 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c00cd570 con 0x7fd4c00ce050 2024-01-15T18:05:24.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.739+0000 7fd4be000640 1 -- 172.21.15.27:0/1988733442 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fd4b8001a00 con 0x7fd4c00ce050 2024-01-15T18:05:24.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.739+0000 7fd4be000640 1 -- 172.21.15.27:0/1988733442 >> v1:172.21.15.27:6789/0 conn(0x7fd4c00ce050 legacy=0x7fd4c006c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:24.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.739+0000 7fd4be000640 1 --2- 172.21.15.27:0/1988733442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4a0003a20 0x7fd4a0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:24.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.739+0000 7fd4be000640 1 --2- 172.21.15.27:0/1988733442 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4c00b1db0 0x7fd4a00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:24.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.739+0000 7fd4be000640 1 --2- 172.21.15.27:0/1988733442 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4a0004d10 0x7fd4a0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:24.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.739+0000 7fd4be000640 1 -- 172.21.15.27:0/1988733442 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c00cd290 con 0x7fd4a0003a20 2024-01-15T18:05:24.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.739+0000 7fd4be000640 1 -- 172.21.15.27:0/1988733442 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a000bac0 con 0x7fd4a0004d10 2024-01-15T18:05:24.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.739+0000 7fd4be000640 1 -- 172.21.15.27:0/1988733442 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a000de90 con 0x7fd4c00b1db0 2024-01-15T18:05:24.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.739+0000 7fd4bf400640 1 --2- 172.21.15.27:0/1988733442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4a0003a20 0x7fd4a0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:24.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.739+0000 7fd4bfe00640 1 --2- 172.21.15.27:0/1988733442 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4a0004d10 0x7fd4a0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:24.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.739+0000 7fd4bea00640 1 --2- 172.21.15.27:0/1988733442 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4c00b1db0 0x7fd4a00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:24.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.739+0000 7fd4bf400640 1 -- 172.21.15.27:0/1988733442 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4c00b1db0 msgr2=0x7fd4a00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:24.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.739+0000 7fd4bf400640 1 --2- 172.21.15.27:0/1988733442 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4c00b1db0 0x7fd4a00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.739+0000 7fd4bf400640 1 -- 172.21.15.27:0/1988733442 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4a0004d10 msgr2=0x7fd4a0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:24.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.739+0000 7fd4bf400640 1 --2- 172.21.15.27:0/1988733442 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4a0004d10 0x7fd4a0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.739+0000 7fd4bf400640 1 -- 172.21.15.27:0/1988733442 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a0003530 con 0x7fd4a0003a20 2024-01-15T18:05:24.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.739+0000 7fd4bf400640 1 --2- 172.21.15.27:0/1988733442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4a0003a20 0x7fd4a0003e00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0001910 tx=0x7fd4b001c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c589837c7826ffb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:24.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.739+0000 7fd4be000640 1 -- 172.21.15.27:0/1988733442 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b00ad660 con 0x7fd4a0003a20 2024-01-15T18:05:24.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.739+0000 7fd4be000640 1 -- 172.21.15.27:0/1988733442 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b00ad7e0 con 0x7fd4a0003a20 2024-01-15T18:05:24.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.739+0000 7fd4be000640 1 -- 172.21.15.27:0/1988733442 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b00adb90 con 0x7fd4a0003a20 2024-01-15T18:05:24.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.743+0000 7fd4c6800640 1 -- 172.21.15.27:0/1988733442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4a0003a20 msgr2=0x7fd4a0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:24.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.743+0000 7fd4c6800640 1 --2- 172.21.15.27:0/1988733442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4a0003a20 0x7fd4a0003e00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0001910 tx=0x7fd4b001c530 comp rx=0 tx=0).stop 2024-01-15T18:05:24.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.743+0000 7fd4bfe00640 1 -- 172.21.15.27:0/1988733442 reap_dead start 2024-01-15T18:05:24.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.743+0000 7fd4c6800640 1 -- 172.21.15.27:0/1988733442 shutdown_connections 2024-01-15T18:05:24.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.743+0000 7fd4c6800640 1 -- 172.21.15.27:0/1988733442 >> 172.21.15.27:0/1988733442 conn(0x7fd4c00cb700 msgr2=0x7fd4c00cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:24.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.743+0000 7fd4c6800640 1 -- 172.21.15.27:0/1988733442 shutdown_connections 2024-01-15T18:05:24.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.743+0000 7fd4c6800640 1 -- 172.21.15.27:0/1988733442 wait complete. 2024-01-15T18:05:24.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.743+0000 7fd4c6800640 1 Processor -- start 2024-01-15T18:05:24.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.743+0000 7fd4c6800640 1 -- start start 2024-01-15T18:05:24.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.743+0000 7fd4c6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4a0003a20 0x7fd4c01f6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:24.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.743+0000 7fd4c6800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4c0154150 0x7fd4c00acc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:24.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.743+0000 7fd4c6800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4c0159b20 0x7fd4c01534d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:24.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.743+0000 7fd4c6800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c01f7240 con 0x7fd4a0003a20 2024-01-15T18:05:24.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.743+0000 7fd4c6800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c01f9870 con 0x7fd4c0159b20 2024-01-15T18:05:24.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.743+0000 7fd4c6800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c01f9b50 con 0x7fd4c0154150 2024-01-15T18:05:24.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.747+0000 7fd4bf400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4a0003a20 0x7fd4c01f6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:24.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.747+0000 7fd4bf400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4a0003a20 0x7fd4c01f6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40518/0 (socket says 172.21.15.27:40518) 2024-01-15T18:05:24.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.747+0000 7fd4bf400640 1 -- 172.21.15.27:0/1339971645 learned_addr learned my addr 172.21.15.27:0/1339971645 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:24.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.747+0000 7fd4bea00640 1 --2- 172.21.15.27:0/1339971645 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4c0154150 0x7fd4c00acc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:24.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.747+0000 7fd4bfe00640 1 --2- 172.21.15.27:0/1339971645 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4c0159b20 0x7fd4c01534d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:24.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.747+0000 7fd4bf400640 1 -- 172.21.15.27:0/1339971645 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4c0154150 msgr2=0x7fd4c00acc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:24.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.747+0000 7fd4bf400640 1 --2- 172.21.15.27:0/1339971645 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4c0154150 0x7fd4c00acc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.747+0000 7fd4bf400640 1 -- 172.21.15.27:0/1339971645 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4c0159b20 msgr2=0x7fd4c01534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:24.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.747+0000 7fd4bf400640 1 --2- 172.21.15.27:0/1339971645 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4c0159b20 0x7fd4c01534d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.747+0000 7fd4bf400640 1 -- 172.21.15.27:0/1339971645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c01f9e30 con 0x7fd4a0003a20 2024-01-15T18:05:24.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.747+0000 7fd4bf400640 1 --2- 172.21.15.27:0/1339971645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4a0003a20 0x7fd4c01f6d00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd4b00b0040 tx=0x7fd4a0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:24.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.747+0000 7fd4b7400640 1 -- 172.21.15.27:0/1339971645 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b00ad660 con 0x7fd4a0003a20 2024-01-15T18:05:24.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.747+0000 7fd4b7400640 1 -- 172.21.15.27:0/1339971645 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b0007a80 con 0x7fd4a0003a20 2024-01-15T18:05:24.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.747+0000 7fd4b7400640 1 -- 172.21.15.27:0/1339971645 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b00adb50 con 0x7fd4a0003a20 2024-01-15T18:05:24.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.747+0000 7fd4c6800640 1 -- 172.21.15.27:0/1339971645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4c01fa110 con 0x7fd4a0003a20 2024-01-15T18:05:24.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.747+0000 7fd4c6800640 1 -- 172.21.15.27:0/1339971645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c01fa6d0 con 0x7fd4a0003a20 2024-01-15T18:05:24.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.747+0000 7fd4c6800640 1 -- 172.21.15.27:0/1339971645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4c004e850 con 0x7fd4a0003a20 2024-01-15T18:05:24.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.747+0000 7fd4b7400640 1 -- 172.21.15.27:0/1339971645 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fd4b00b4050 con 0x7fd4a0003a20 2024-01-15T18:05:24.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.751+0000 7fd4b7400640 1 --2- 172.21.15.27:0/1339971645 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd4980668f0 0x7fd498068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:24.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.751+0000 7fd4b7400640 1 -- 172.21.15.27:0/1339971645 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd4b011e670 con 0x7fd4a0003a20 2024-01-15T18:05:24.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.751+0000 7fd4bea00640 1 --2- 172.21.15.27:0/1339971645 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd4980668f0 0x7fd498068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:24.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.751+0000 7fd4bea00640 1 --2- 172.21.15.27:0/1339971645 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd4980668f0 0x7fd498068db0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd4c006df00 tx=0x7fd4a8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:24.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.755+0000 7fd4b7400640 1 -- 172.21.15.27:0/1339971645 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd4b00eb950 con 0x7fd4a0003a20 2024-01-15T18:05:24.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.911+0000 7fd4c6800640 1 -- 172.21.15.27:0/1339971645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd4c015ced0 con 0x7fd4a0003a20 2024-01-15T18:05:24.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.911+0000 7fd4b7400640 1 -- 172.21.15.27:0/1339971645 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd4b00eef70 con 0x7fd4a0003a20 2024-01-15T18:05:24.917 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-15T18:05:24.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.915+0000 7fd4c6800640 1 -- 172.21.15.27:0/1339971645 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd4980668f0 msgr2=0x7fd498068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:24.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.915+0000 7fd4c6800640 1 --2- 172.21.15.27:0/1339971645 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd4980668f0 0x7fd498068db0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd4c006df00 tx=0x7fd4a8009190 comp rx=0 tx=0).stop 2024-01-15T18:05:24.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.915+0000 7fd4c6800640 1 -- 172.21.15.27:0/1339971645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4a0003a20 msgr2=0x7fd4c01f6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:24.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.915+0000 7fd4c6800640 1 --2- 172.21.15.27:0/1339971645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4a0003a20 0x7fd4c01f6d00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd4b00b0040 tx=0x7fd4a0005800 comp rx=0 tx=0).stop 2024-01-15T18:05:24.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.915+0000 7fd4c6800640 1 -- 172.21.15.27:0/1339971645 shutdown_connections 2024-01-15T18:05:24.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.915+0000 7fd4c6800640 1 --2- 172.21.15.27:0/1339971645 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd4980668f0 0x7fd498068db0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.915+0000 7fd4c6800640 1 --2- 172.21.15.27:0/1339971645 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4c0159b20 0x7fd4c01534d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.915+0000 7fd4c6800640 1 --2- 172.21.15.27:0/1339971645 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4c0154150 0x7fd4c00acc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.915+0000 7fd4c6800640 1 --2- 172.21.15.27:0/1339971645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4a0003a20 0x7fd4c01f6d00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:24.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.915+0000 7fd4c6800640 1 -- 172.21.15.27:0/1339971645 >> 172.21.15.27:0/1339971645 conn(0x7fd4c00cb700 msgr2=0x7fd4c00b3130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:24.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.915+0000 7fd4c6800640 1 -- 172.21.15.27:0/1339971645 shutdown_connections 2024-01-15T18:05:24.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:24.915+0000 7fd4c6800640 1 -- 172.21.15.27:0/1339971645 wait complete. 2024-01-15T18:05:24.939 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:25.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.075+0000 7f9781a00640 1 Processor -- start 2024-01-15T18:05:25.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.075+0000 7f9781a00640 1 -- start start 2024-01-15T18:05:25.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f9781a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f977c0b2390 0x7f977c0cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:25.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f9781a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f977c0cdf90 0x7f977c0b0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:25.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f9781a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977c0b2770 con 0x7f977c06bb80 2024-01-15T18:05:25.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f9781a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f977c06c060 con 0x7f977c0b2390 2024-01-15T18:05:25.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f9781a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f977c06c340 con 0x7f977c0cdf90 2024-01-15T18:05:25.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f977a000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f977c06bb80 0x7f977c0cd880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34762/0 (socket says 172.21.15.27:34762) 2024-01-15T18:05:25.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f977a000640 1 -- 172.21.15.27:0/1602941432 learned_addr learned my addr 172.21.15.27:0/1602941432 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:25.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f977aa00640 1 --2- 172.21.15.27:0/1602941432 >> v2:172.21.15.93:3300/0 conn(0x7f977c0b2390 0x7f977c0cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:25.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f977b400640 1 --2- 172.21.15.27:0/1602941432 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f977c0cdf90 0x7f977c0b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:25.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f9779600640 1 -- 172.21.15.27:0/1602941432 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4238512215 0 0) 0x7f977c0b2770 con 0x7f977c06bb80 2024-01-15T18:05:25.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f9779600640 1 -- 172.21.15.27:0/1602941432 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f975c003530 con 0x7f977c06bb80 2024-01-15T18:05:25.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f9779600640 1 -- 172.21.15.27:0/1602941432 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1227764664 0 0) 0x7f975c003530 con 0x7f977c06bb80 2024-01-15T18:05:25.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f9779600640 1 -- 172.21.15.27:0/1602941432 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f977c0cdf90 msgr2=0x7f977c0b0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:25.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f9779600640 1 --2- 172.21.15.27:0/1602941432 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f977c0cdf90 0x7f977c0b0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f9779600640 1 -- 172.21.15.27:0/1602941432 >> v2:172.21.15.93:3300/0 conn(0x7f977c0b2390 msgr2=0x7f977c0cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:25.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f9779600640 1 --2- 172.21.15.27:0/1602941432 >> v2:172.21.15.93:3300/0 conn(0x7f977c0b2390 0x7f977c0cf780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f9779600640 1 -- 172.21.15.27:0/1602941432 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f977c06c620 con 0x7f977c06bb80 2024-01-15T18:05:25.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f9779600640 1 -- 172.21.15.27:0/1602941432 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f9764001680 con 0x7f977c06bb80 2024-01-15T18:05:25.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f977aa00640 1 --2- 172.21.15.27:0/1602941432 >> v2:172.21.15.93:3300/0 conn(0x7f977c0b2390 0x7f977c0cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:25.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f9779600640 1 -- 172.21.15.27:0/1602941432 >> v1:172.21.15.27:6789/0 conn(0x7f977c06bb80 legacy=0x7f977c0cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:25.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f9779600640 1 --2- 172.21.15.27:0/1602941432 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f975c0039f0 0x7f975c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:25.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f9779600640 1 --2- 172.21.15.27:0/1602941432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f975c0047d0 0x7f975c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:25.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f9779600640 1 --2- 172.21.15.27:0/1602941432 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f977c0cdf90 0x7f975c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:25.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f9779600640 1 -- 172.21.15.27:0/1602941432 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c06c060 con 0x7f975c0047d0 2024-01-15T18:05:25.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f9779600640 1 -- 172.21.15.27:0/1602941432 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c06c340 con 0x7f975c0039f0 2024-01-15T18:05:25.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f9779600640 1 -- 172.21.15.27:0/1602941432 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f975c00b990 con 0x7f977c0cdf90 2024-01-15T18:05:25.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f977a000640 1 --2- 172.21.15.27:0/1602941432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f975c0047d0 0x7f975c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:25.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f977aa00640 1 --2- 172.21.15.27:0/1602941432 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f975c0039f0 0x7f975c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:25.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f977b400640 1 --2- 172.21.15.27:0/1602941432 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f977c0cdf90 0x7f975c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:25.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f977a000640 1 -- 172.21.15.27:0/1602941432 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f977c0cdf90 msgr2=0x7f975c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:25.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f977a000640 1 --2- 172.21.15.27:0/1602941432 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f977c0cdf90 0x7f975c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f977a000640 1 -- 172.21.15.27:0/1602941432 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f975c0039f0 msgr2=0x7f975c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:25.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f977a000640 1 --2- 172.21.15.27:0/1602941432 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f975c0039f0 0x7f975c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.079+0000 7f977a000640 1 -- 172.21.15.27:0/1602941432 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975c003530 con 0x7f975c0047d0 2024-01-15T18:05:25.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.083+0000 7f977a000640 1 --2- 172.21.15.27:0/1602941432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f975c0047d0 0x7f975c004c50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f976400c270 tx=0x7f976401f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f705cec7e081ed55 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:25.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.083+0000 7f9779600640 1 -- 172.21.15.27:0/1602941432 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97640a8ce0 con 0x7f975c0047d0 2024-01-15T18:05:25.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.083+0000 7f9779600640 1 -- 172.21.15.27:0/1602941432 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97640a8e40 con 0x7f975c0047d0 2024-01-15T18:05:25.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.083+0000 7f9779600640 1 -- 172.21.15.27:0/1602941432 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9764005b00 con 0x7f975c0047d0 2024-01-15T18:05:25.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.083+0000 7f9781a00640 1 -- 172.21.15.27:0/1602941432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f975c0047d0 msgr2=0x7f975c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:25.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.083+0000 7f9781a00640 1 --2- 172.21.15.27:0/1602941432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f975c0047d0 0x7f975c004c50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f976400c270 tx=0x7f976401f3b0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.083+0000 7f977b400640 1 -- 172.21.15.27:0/1602941432 reap_dead start 2024-01-15T18:05:25.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.083+0000 7f9781a00640 1 -- 172.21.15.27:0/1602941432 shutdown_connections 2024-01-15T18:05:25.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.083+0000 7f9781a00640 1 -- 172.21.15.27:0/1602941432 >> 172.21.15.27:0/1602941432 conn(0x7f977c0b2a10 msgr2=0x7f977c0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:25.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.083+0000 7f9781a00640 1 -- 172.21.15.27:0/1602941432 shutdown_connections 2024-01-15T18:05:25.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.083+0000 7f9781a00640 1 -- 172.21.15.27:0/1602941432 wait complete. 2024-01-15T18:05:25.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.087+0000 7f9781a00640 1 Processor -- start 2024-01-15T18:05:25.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.087+0000 7f9781a00640 1 -- start start 2024-01-15T18:05:25.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.087+0000 7f9781a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f975c0047d0 0x7f977c1f37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:25.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.087+0000 7f9781a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f977c1f6330 0x7f977c1534c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:25.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.087+0000 7f9781a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f977c0b5720 0x7f977c0af2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:25.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.087+0000 7f9781a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c0ca6a0 con 0x7f975c0047d0 2024-01-15T18:05:25.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.087+0000 7f9781a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c1f3ce0 con 0x7f977c1f6330 2024-01-15T18:05:25.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.087+0000 7f9781a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f977c153a00 con 0x7f977c0b5720 2024-01-15T18:05:25.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.087+0000 7f977aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f975c0047d0 0x7f977c1f37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:25.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.087+0000 7f977aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f975c0047d0 0x7f977c1f37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40542/0 (socket says 172.21.15.27:40542) 2024-01-15T18:05:25.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.087+0000 7f977aa00640 1 -- 172.21.15.27:0/1449373874 learned_addr learned my addr 172.21.15.27:0/1449373874 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:25.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.087+0000 7f977b400640 1 --2- 172.21.15.27:0/1449373874 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f977c0b5720 0x7f977c0af2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:25.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.087+0000 7f977a000640 1 --2- 172.21.15.27:0/1449373874 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f977c1f6330 0x7f977c1534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:25.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.087+0000 7f977aa00640 1 -- 172.21.15.27:0/1449373874 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f977c0b5720 msgr2=0x7f977c0af2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:25.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.087+0000 7f977aa00640 1 --2- 172.21.15.27:0/1449373874 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f977c0b5720 0x7f977c0af2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.087+0000 7f977aa00640 1 -- 172.21.15.27:0/1449373874 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f977c1f6330 msgr2=0x7f977c1534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:25.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.087+0000 7f977aa00640 1 --2- 172.21.15.27:0/1449373874 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f977c1f6330 0x7f977c1534c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.087+0000 7f977aa00640 1 -- 172.21.15.27:0/1449373874 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f977c0b5070 con 0x7f975c0047d0 2024-01-15T18:05:25.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.087+0000 7f977aa00640 1 --2- 172.21.15.27:0/1449373874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f975c0047d0 0x7f977c1f37a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f976c011ca0 tx=0x7f976c00f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:25.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.087+0000 7f9772a00640 1 -- 172.21.15.27:0/1449373874 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c00fcc0 con 0x7f975c0047d0 2024-01-15T18:05:25.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.087+0000 7f9772a00640 1 -- 172.21.15.27:0/1449373874 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f976c00fe20 con 0x7f975c0047d0 2024-01-15T18:05:25.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.087+0000 7f9772a00640 1 -- 172.21.15.27:0/1449373874 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c004240 con 0x7f975c0047d0 2024-01-15T18:05:25.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.087+0000 7f9781a00640 1 -- 172.21.15.27:0/1449373874 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f977c0c86c0 con 0x7f975c0047d0 2024-01-15T18:05:25.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.087+0000 7f9781a00640 1 -- 172.21.15.27:0/1449373874 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f977c1f6df0 con 0x7f975c0047d0 2024-01-15T18:05:25.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.087+0000 7f9781a00640 1 -- 172.21.15.27:0/1449373874 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f977c04e850 con 0x7f975c0047d0 2024-01-15T18:05:25.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.091+0000 7f9772a00640 1 -- 172.21.15.27:0/1449373874 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f976c0043a0 con 0x7f975c0047d0 2024-01-15T18:05:25.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.091+0000 7f9772a00640 1 --2- 172.21.15.27:0/1449373874 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9754066920 0x7f9754068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:25.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.091+0000 7f9772a00640 1 -- 172.21.15.27:0/1449373874 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f976c084930 con 0x7f975c0047d0 2024-01-15T18:05:25.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.091+0000 7f977a000640 1 --2- 172.21.15.27:0/1449373874 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9754066920 0x7f9754068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:25.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.091+0000 7f977a000640 1 --2- 172.21.15.27:0/1449373874 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9754066920 0x7f9754068de0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f97640ad000 tx=0x7f9764001a80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:25.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.095+0000 7f9772a00640 1 -- 172.21.15.27:0/1449373874 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f976c051b80 con 0x7f975c0047d0 2024-01-15T18:05:25.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.251+0000 7f9781a00640 1 -- 172.21.15.27:0/1449373874 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f977c0b5500 con 0x7f975c0047d0 2024-01-15T18:05:25.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.251+0000 7f9772a00640 1 -- 172.21.15.27:0/1449373874 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16409 (secure 0 0 0) 0x7f976c022070 con 0x7f975c0047d0 2024-01-15T18:05:25.259 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:25.259 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:04:59.837052+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:04:56.220608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T18:04:57.210421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:25.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.255+0000 7f9781a00640 1 -- 172.21.15.27:0/1449373874 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9754066920 msgr2=0x7f9754068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:25.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.255+0000 7f9781a00640 1 --2- 172.21.15.27:0/1449373874 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9754066920 0x7f9754068de0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f97640ad000 tx=0x7f9764001a80 comp rx=0 tx=0).stop 2024-01-15T18:05:25.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.255+0000 7f9781a00640 1 -- 172.21.15.27:0/1449373874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f975c0047d0 msgr2=0x7f977c1f37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:25.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.255+0000 7f9781a00640 1 --2- 172.21.15.27:0/1449373874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f975c0047d0 0x7f977c1f37a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f976c011ca0 tx=0x7f976c00f610 comp rx=0 tx=0).stop 2024-01-15T18:05:25.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.255+0000 7f9781a00640 1 -- 172.21.15.27:0/1449373874 shutdown_connections 2024-01-15T18:05:25.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.255+0000 7f9781a00640 1 --2- 172.21.15.27:0/1449373874 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9754066920 0x7f9754068de0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.255+0000 7f9781a00640 1 --2- 172.21.15.27:0/1449373874 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f977c0b5720 0x7f977c0af2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.255+0000 7f9781a00640 1 --2- 172.21.15.27:0/1449373874 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f977c1f6330 0x7f977c1534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.255+0000 7f9781a00640 1 --2- 172.21.15.27:0/1449373874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f975c0047d0 0x7f977c1f37a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.255+0000 7f9781a00640 1 -- 172.21.15.27:0/1449373874 >> 172.21.15.27:0/1449373874 conn(0x7f977c0b2a10 msgr2=0x7f977c0cc270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:25.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.255+0000 7f9781a00640 1 -- 172.21.15.27:0/1449373874 shutdown_connections 2024-01-15T18:05:25.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.255+0000 7f9781a00640 1 -- 172.21.15.27:0/1449373874 wait complete. 2024-01-15T18:05:25.281 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T18:04:56.220608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T18:04:57.210421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:05:25.281 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:05:25.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.419+0000 7f1050600640 1 Processor -- start 2024-01-15T18:05:25.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.419+0000 7f1050600640 1 -- start start 2024-01-15T18:05:25.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.423+0000 7f1050600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f104815b4b0 0x7f10480b1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:25.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.423+0000 7f1050600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f10480cec60 0x7f10480b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:25.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.423+0000 7f1050600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10480d19a0 con 0x7f10480cf460 2024-01-15T18:05:25.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.423+0000 7f1050600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10480d1c80 con 0x7f104815b4b0 2024-01-15T18:05:25.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.423+0000 7f1050600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10480b07c0 con 0x7f10480cec60 2024-01-15T18:05:25.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.423+0000 7f104d200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f104815b4b0 0x7f10480b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:25.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.423+0000 7f104d200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f104815b4b0 0x7f10480b1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40558/0 (socket says 172.21.15.27:40558) 2024-01-15T18:05:25.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.423+0000 7f104d200640 1 -- 172.21.15.27:0/2406292127 learned_addr learned my addr 172.21.15.27:0/2406292127 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:25.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.423+0000 7f104e600640 1 --2- 172.21.15.27:0/2406292127 >> v2:172.21.15.93:3300/0 conn(0x7f10480cec60 0x7f10480b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:25.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.423+0000 7f104d200640 1 -- 172.21.15.27:0/2406292127 >> v2:172.21.15.93:3300/0 conn(0x7f10480cec60 msgr2=0x7f10480b1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:25.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.423+0000 7f104d200640 1 --2- 172.21.15.27:0/2406292127 >> v2:172.21.15.93:3300/0 conn(0x7f10480cec60 0x7f10480b1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.423+0000 7f104d200640 1 -- 172.21.15.27:0/2406292127 >> v1:172.21.15.93:6789/0 conn(0x7f10480cf460 legacy=0x7f10480cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:25.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.423+0000 7f104d200640 1 -- 172.21.15.27:0/2406292127 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10480b0aa0 con 0x7f104815b4b0 2024-01-15T18:05:25.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.423+0000 7f104e600640 1 --2- 172.21.15.27:0/2406292127 >> v2:172.21.15.93:3300/0 conn(0x7f10480cec60 0x7f10480b1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:25.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.427+0000 7f104d200640 1 --2- 172.21.15.27:0/2406292127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f104815b4b0 0x7f10480b1580 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f104806af90 tx=0x7f103001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad87514467f28de server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:25.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.427+0000 7f1047e00640 1 -- 172.21.15.27:0/2406292127 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10480b2810 con 0x7f104815b4b0 2024-01-15T18:05:25.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.427+0000 7f1047e00640 1 -- 172.21.15.27:0/2406292127 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10300aa660 con 0x7f104815b4b0 2024-01-15T18:05:25.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.427+0000 7f1047e00640 1 -- 172.21.15.27:0/2406292127 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10300aa9f0 con 0x7f104815b4b0 2024-01-15T18:05:25.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.427+0000 7f1050600640 1 -- 172.21.15.27:0/2406292127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f104815b4b0 msgr2=0x7f10480b1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:25.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.427+0000 7f1050600640 1 --2- 172.21.15.27:0/2406292127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f104815b4b0 0x7f10480b1580 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f104806af90 tx=0x7f103001c500 comp rx=0 tx=0).stop 2024-01-15T18:05:25.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.427+0000 7f1050600640 1 -- 172.21.15.27:0/2406292127 shutdown_connections 2024-01-15T18:05:25.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.427+0000 7f1050600640 1 --2- 172.21.15.27:0/2406292127 >> v2:172.21.15.93:3300/0 conn(0x7f10480cec60 0x7f10480b1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.427+0000 7f1050600640 1 --2- 172.21.15.27:0/2406292127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f104815b4b0 0x7f10480b1580 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.427+0000 7f1050600640 1 -- 172.21.15.27:0/2406292127 >> 172.21.15.27:0/2406292127 conn(0x7f10480d0e80 msgr2=0x7f10480d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:25.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.427+0000 7f1050600640 1 -- 172.21.15.27:0/2406292127 shutdown_connections 2024-01-15T18:05:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.427+0000 7f1050600640 1 -- 172.21.15.27:0/2406292127 wait complete. 2024-01-15T18:05:25.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.427+0000 7f1050600640 1 Processor -- start 2024-01-15T18:05:25.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.427+0000 7f1050600640 1 -- start start 2024-01-15T18:05:25.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.431+0000 7f1050600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10480cec60 0x7f1048153170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:25.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.431+0000 7f1050600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f10480cf460 0x7f104814f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:25.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.431+0000 7f1050600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f104815b4b0 0x7f10480c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:25.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.431+0000 7f1050600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10481520d0 con 0x7f10480cec60 2024-01-15T18:05:25.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.431+0000 7f1050600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10481523b0 con 0x7f10480cf460 2024-01-15T18:05:25.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.431+0000 7f1050600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1048152690 con 0x7f104815b4b0 2024-01-15T18:05:25.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.431+0000 7f104dc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10480cec60 0x7f1048153170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:25.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.431+0000 7f104dc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10480cec60 0x7f1048153170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40562/0 (socket says 172.21.15.27:40562) 2024-01-15T18:05:25.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.431+0000 7f104dc00640 1 -- 172.21.15.27:0/775571444 learned_addr learned my addr 172.21.15.27:0/775571444 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:25.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.431+0000 7f104d200640 1 --2- 172.21.15.27:0/775571444 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f10480cf460 0x7f104814f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:25.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.431+0000 7f104e600640 1 --2- 172.21.15.27:0/775571444 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f104815b4b0 0x7f10480c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:25.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.431+0000 7f104dc00640 1 -- 172.21.15.27:0/775571444 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f104815b4b0 msgr2=0x7f10480c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:25.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.431+0000 7f104dc00640 1 --2- 172.21.15.27:0/775571444 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f104815b4b0 0x7f10480c7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.431+0000 7f104dc00640 1 -- 172.21.15.27:0/775571444 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f10480cf460 msgr2=0x7f104814f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:25.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.431+0000 7f104dc00640 1 --2- 172.21.15.27:0/775571444 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f10480cf460 0x7f104814f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.431+0000 7f104dc00640 1 -- 172.21.15.27:0/775571444 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10300094d0 con 0x7f10480cec60 2024-01-15T18:05:25.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.431+0000 7f104dc00640 1 --2- 172.21.15.27:0/775571444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10480cec60 0x7f1048153170 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f103800c500 tx=0x7f103800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:25.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.431+0000 7f1045600640 1 -- 172.21.15.27:0/775571444 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1038005220 con 0x7f10480cec60 2024-01-15T18:05:25.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.431+0000 7f1045600640 1 -- 172.21.15.27:0/775571444 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1038005380 con 0x7f10480cec60 2024-01-15T18:05:25.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.431+0000 7f1045600640 1 -- 172.21.15.27:0/775571444 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1038007270 con 0x7f10480cec60 2024-01-15T18:05:25.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.431+0000 7f1050600640 1 -- 172.21.15.27:0/775571444 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1048152970 con 0x7f10480cec60 2024-01-15T18:05:25.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.431+0000 7f1050600640 1 -- 172.21.15.27:0/775571444 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1048151d60 con 0x7f10480cec60 2024-01-15T18:05:25.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.431+0000 7f1050600640 1 -- 172.21.15.27:0/775571444 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f104804e850 con 0x7f10480cec60 2024-01-15T18:05:25.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.435+0000 7f1045600640 1 -- 172.21.15.27:0/775571444 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f10380073d0 con 0x7f10480cec60 2024-01-15T18:05:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.435+0000 7f1045600640 1 --2- 172.21.15.27:0/775571444 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f10200668f0 0x7f1020068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.435+0000 7f1045600640 1 -- 172.21.15.27:0/775571444 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1038083c20 con 0x7f10480cec60 2024-01-15T18:05:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.435+0000 7f104d200640 1 --2- 172.21.15.27:0/775571444 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f10200668f0 0x7f1020068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.435+0000 7f104d200640 1 --2- 172.21.15.27:0/775571444 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f10200668f0 0x7f1020068db0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f10300ae000 tx=0x7f10300b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:25.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.439+0000 7f1045600640 1 -- 172.21.15.27:0/775571444 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1038050e80 con 0x7f10480cec60 2024-01-15T18:05:25.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.599+0000 7f1050600640 1 -- 172.21.15.27:0/775571444 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f10480cf240 con 0x7f10480cec60 2024-01-15T18:05:25.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.599+0000 7f1045600640 1 -- 172.21.15.27:0/775571444 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f10380544a0 con 0x7f10480cec60 2024-01-15T18:05:25.604 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:05:25.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.599+0000 7f1050600640 1 -- 172.21.15.27:0/775571444 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f10200668f0 msgr2=0x7f1020068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:25.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.599+0000 7f1050600640 1 --2- 172.21.15.27:0/775571444 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f10200668f0 0x7f1020068db0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f10300ae000 tx=0x7f10300b5040 comp rx=0 tx=0).stop 2024-01-15T18:05:25.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.599+0000 7f1050600640 1 -- 172.21.15.27:0/775571444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10480cec60 msgr2=0x7f1048153170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:25.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.599+0000 7f1050600640 1 --2- 172.21.15.27:0/775571444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10480cec60 0x7f1048153170 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f103800c500 tx=0x7f103800c9c0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.599+0000 7f1050600640 1 -- 172.21.15.27:0/775571444 shutdown_connections 2024-01-15T18:05:25.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.599+0000 7f1050600640 1 --2- 172.21.15.27:0/775571444 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f10200668f0 0x7f1020068db0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.599+0000 7f1050600640 1 --2- 172.21.15.27:0/775571444 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f104815b4b0 0x7f10480c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.599+0000 7f1050600640 1 --2- 172.21.15.27:0/775571444 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f10480cf460 0x7f104814f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.599+0000 7f1050600640 1 --2- 172.21.15.27:0/775571444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10480cec60 0x7f1048153170 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.599+0000 7f1050600640 1 -- 172.21.15.27:0/775571444 >> 172.21.15.27:0/775571444 conn(0x7f10480d0e80 msgr2=0x7f10480d1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:25.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.599+0000 7f1050600640 1 -- 172.21.15.27:0/775571444 shutdown_connections 2024-01-15T18:05:25.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.599+0000 7f1050600640 1 -- 172.21.15.27:0/775571444 wait complete. 2024-01-15T18:05:25.626 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T18:05:25.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.763+0000 7f34c7e00640 1 Processor -- start 2024-01-15T18:05:25.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.763+0000 7f34c7e00640 1 -- start start 2024-01-15T18:05:25.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.767+0000 7f34c7e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34c006c430 0x7f34c0070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:25.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.767+0000 7f34c7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f34c00b1db0 0x7f34c00b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:25.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.767+0000 7f34c7e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c00ccfb0 con 0x7f34c00ce050 2024-01-15T18:05:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.767+0000 7f34c7e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34c00cd290 con 0x7f34c00b1db0 2024-01-15T18:05:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.767+0000 7f34c7e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34c00cd570 con 0x7f34c006c430 2024-01-15T18:05:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.767+0000 7f34c4a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f34c00b1db0 0x7f34c00b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.767+0000 7f34c4a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f34c00b1db0 0x7f34c00b2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40570/0 (socket says 172.21.15.27:40570) 2024-01-15T18:05:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.767+0000 7f34c5400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34c006c430 0x7f34c0070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.767+0000 7f34c4a00640 1 -- 172.21.15.27:0/3480973705 learned_addr learned my addr 172.21.15.27:0/3480973705 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.767+0000 7f34c4a00640 1 -- 172.21.15.27:0/3480973705 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34c006c430 msgr2=0x7f34c0070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.767+0000 7f34c4a00640 1 --2- 172.21.15.27:0/3480973705 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34c006c430 0x7f34c0070ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.767+0000 7f34c4a00640 1 -- 172.21.15.27:0/3480973705 >> v1:172.21.15.27:6789/0 conn(0x7f34c00ce050 legacy=0x7f34c006c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.767+0000 7f34c4a00640 1 -- 172.21.15.27:0/3480973705 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c00cd850 con 0x7f34c00b1db0 2024-01-15T18:05:25.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.767+0000 7f34c5400640 1 --2- 172.21.15.27:0/3480973705 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34c006c430 0x7f34c0070ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:25.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.771+0000 7f34c4a00640 1 --2- 172.21.15.27:0/3480973705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34c00b1db0 0x7f34c00b2210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f34a8009870 tx=0x7f34a801c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1377402026493ede server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:25.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.771+0000 7f34bfe00640 1 -- 172.21.15.27:0/3480973705 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a80ab600 con 0x7f34c00b1db0 2024-01-15T18:05:25.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.771+0000 7f34bfe00640 1 -- 172.21.15.27:0/3480973705 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a80a7050 con 0x7f34c00b1db0 2024-01-15T18:05:25.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.771+0000 7f34bfe00640 1 -- 172.21.15.27:0/3480973705 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a80ab980 con 0x7f34c00b1db0 2024-01-15T18:05:25.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.771+0000 7f34c7e00640 1 -- 172.21.15.27:0/3480973705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34c00b1db0 msgr2=0x7f34c00b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:25.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.771+0000 7f34c7e00640 1 --2- 172.21.15.27:0/3480973705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34c00b1db0 0x7f34c00b2210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f34a8009870 tx=0x7f34a801c4b0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.771+0000 7f34c7e00640 1 -- 172.21.15.27:0/3480973705 shutdown_connections 2024-01-15T18:05:25.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.771+0000 7f34c7e00640 1 --2- 172.21.15.27:0/3480973705 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34c00b1db0 0x7f34c00b2210 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.771+0000 7f34c7e00640 1 --2- 172.21.15.27:0/3480973705 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34c006c430 0x7f34c0070ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.771+0000 7f34c7e00640 1 -- 172.21.15.27:0/3480973705 >> 172.21.15.27:0/3480973705 conn(0x7f34c00cb700 msgr2=0x7f34c00cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:25.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.771+0000 7f34c7e00640 1 -- 172.21.15.27:0/3480973705 shutdown_connections 2024-01-15T18:05:25.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.771+0000 7f34c7e00640 1 -- 172.21.15.27:0/3480973705 wait complete. 2024-01-15T18:05:25.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34c7e00640 1 Processor -- start 2024-01-15T18:05:25.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34c7e00640 1 -- start start 2024-01-15T18:05:25.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34c7e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34c006c430 0x7f34c01500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:25.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34c7e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f34c00b1db0 0x7f34c01541e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:25.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34c7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34c00ce050 0x7f34c0155a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:25.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34c7e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c0154d10 con 0x7f34c00ce050 2024-01-15T18:05:25.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34c7e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c0154a30 con 0x7f34c00b1db0 2024-01-15T18:05:25.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34c7e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34c0159c60 con 0x7f34c006c430 2024-01-15T18:05:25.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34c5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34c00ce050 0x7f34c0155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:25.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34c5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34c00ce050 0x7f34c0155a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40584/0 (socket says 172.21.15.27:40584) 2024-01-15T18:05:25.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34c5e00640 1 -- 172.21.15.27:0/1352326911 learned_addr learned my addr 172.21.15.27:0/1352326911 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:25.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34c4a00640 1 --2- 172.21.15.27:0/1352326911 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f34c00b1db0 0x7f34c01541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:25.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34c5400640 1 --2- 172.21.15.27:0/1352326911 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34c006c430 0x7f34c01500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:25.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34c5e00640 1 -- 172.21.15.27:0/1352326911 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34c006c430 msgr2=0x7f34c01500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:25.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34c5e00640 1 --2- 172.21.15.27:0/1352326911 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34c006c430 0x7f34c01500d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34c5e00640 1 -- 172.21.15.27:0/1352326911 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f34c00b1db0 msgr2=0x7f34c01541e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:25.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34c5e00640 1 --2- 172.21.15.27:0/1352326911 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f34c00b1db0 0x7f34c01541e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34c5e00640 1 -- 172.21.15.27:0/1352326911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b0009420 con 0x7f34c00ce050 2024-01-15T18:05:25.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34c4a00640 1 --2- 172.21.15.27:0/1352326911 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f34c00b1db0 0x7f34c01541e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:25.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34c5e00640 1 --2- 172.21.15.27:0/1352326911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34c00ce050 0x7f34c0155a10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f34b800c770 tx=0x7f34b800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:25.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34bd600640 1 -- 172.21.15.27:0/1352326911 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b8003b80 con 0x7f34c00ce050 2024-01-15T18:05:25.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34bd600640 1 -- 172.21.15.27:0/1352326911 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34b8003ce0 con 0x7f34c00ce050 2024-01-15T18:05:25.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34bd600640 1 -- 172.21.15.27:0/1352326911 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b800aa90 con 0x7f34c00ce050 2024-01-15T18:05:25.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34c7e00640 1 -- 172.21.15.27:0/1352326911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a80094d0 con 0x7f34c00ce050 2024-01-15T18:05:25.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34c7e00640 1 -- 172.21.15.27:0/1352326911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34c015a220 con 0x7f34c00ce050 2024-01-15T18:05:25.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.775+0000 7f34c7e00640 1 -- 172.21.15.27:0/1352326911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34c004e850 con 0x7f34c00ce050 2024-01-15T18:05:25.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.779+0000 7f34bd600640 1 -- 172.21.15.27:0/1352326911 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f34b800abf0 con 0x7f34c00ce050 2024-01-15T18:05:25.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.779+0000 7f34bd600640 1 --2- 172.21.15.27:0/1352326911 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f34980668f0 0x7f3498068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:25.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.779+0000 7f34bd600640 1 -- 172.21.15.27:0/1352326911 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34b8084280 con 0x7f34c00ce050 2024-01-15T18:05:25.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.779+0000 7f34c5400640 1 --2- 172.21.15.27:0/1352326911 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f34980668f0 0x7f3498068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:25.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.779+0000 7f34c5400640 1 --2- 172.21.15.27:0/1352326911 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f34980668f0 0x7f3498068db0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f34c0156eb0 tx=0x7f34b0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:25.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.783+0000 7f34bd600640 1 -- 172.21.15.27:0/1352326911 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f34b80514b0 con 0x7f34c00ce050 2024-01-15T18:05:25.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.939+0000 7f34c7e00640 1 -- 172.21.15.27:0/1352326911 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f34c015ced0 con 0x7f34c00ce050 2024-01-15T18:05:25.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.939+0000 7f34bd600640 1 -- 172.21.15.27:0/1352326911 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f34b8054ad0 con 0x7f34c00ce050 2024-01-15T18:05:25.945 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-15T18:05:25.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.943+0000 7f34c7e00640 1 -- 172.21.15.27:0/1352326911 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f34980668f0 msgr2=0x7f3498068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:25.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.943+0000 7f34c7e00640 1 --2- 172.21.15.27:0/1352326911 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f34980668f0 0x7f3498068db0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f34c0156eb0 tx=0x7f34b0008040 comp rx=0 tx=0).stop 2024-01-15T18:05:25.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.943+0000 7f34c7e00640 1 -- 172.21.15.27:0/1352326911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34c00ce050 msgr2=0x7f34c0155a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:25.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.943+0000 7f34c7e00640 1 --2- 172.21.15.27:0/1352326911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34c00ce050 0x7f34c0155a10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f34b800c770 tx=0x7f34b800cc30 comp rx=0 tx=0).stop 2024-01-15T18:05:25.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.943+0000 7f34c7e00640 1 -- 172.21.15.27:0/1352326911 shutdown_connections 2024-01-15T18:05:25.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.943+0000 7f34c7e00640 1 --2- 172.21.15.27:0/1352326911 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f34980668f0 0x7f3498068db0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.943+0000 7f34c7e00640 1 --2- 172.21.15.27:0/1352326911 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34c00ce050 0x7f34c0155a10 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.943+0000 7f34c7e00640 1 --2- 172.21.15.27:0/1352326911 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f34c00b1db0 0x7f34c01541e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.943+0000 7f34c7e00640 1 --2- 172.21.15.27:0/1352326911 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34c006c430 0x7f34c01500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:25.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.943+0000 7f34c7e00640 1 -- 172.21.15.27:0/1352326911 >> 172.21.15.27:0/1352326911 conn(0x7f34c00cb700 msgr2=0x7f34c00b3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:25.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.943+0000 7f34c7e00640 1 -- 172.21.15.27:0/1352326911 shutdown_connections 2024-01-15T18:05:25.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:25.943+0000 7f34c7e00640 1 -- 172.21.15.27:0/1352326911 wait complete. 2024-01-15T18:05:25.968 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T18:05:26.055 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:05:26.058 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:05:26.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.107+0000 7f19a5000640 1 Processor -- start 2024-01-15T18:05:26.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.107+0000 7f19a5000640 1 -- start start 2024-01-15T18:05:26.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.107+0000 7f19a5000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f19a015b020 0x7f19a00b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:26.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.107+0000 7f19a5000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19a00cb920 con 0x7f19a015b400 2024-01-15T18:05:26.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.107+0000 7f19a5000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19a00ce7d0 con 0x7f19a00ceb10 2024-01-15T18:05:26.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.107+0000 7f19a5000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19a00d0cf0 con 0x7f19a015b020 2024-01-15T18:05:26.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.111+0000 7f199d400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f19a015b400 0x7f19a00cd6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34788/0 (socket says 172.21.15.27:34788) 2024-01-15T18:05:26.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.111+0000 7f199d400640 1 -- 172.21.15.27:0/1587723340 learned_addr learned my addr 172.21.15.27:0/1587723340 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.111+0000 7f199de00640 1 --2- 172.21.15.27:0/1587723340 >> v2:172.21.15.93:3300/0 conn(0x7f19a015b020 0x7f19a00b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.111+0000 7f199ca00640 1 -- 172.21.15.27:0/1587723340 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3026668947 0 0) 0x7f19a00cb920 con 0x7f19a015b400 2024-01-15T18:05:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.111+0000 7f199ca00640 1 -- 172.21.15.27:0/1587723340 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1980003530 con 0x7f19a015b400 2024-01-15T18:05:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.111+0000 7f199ca00640 1 -- 172.21.15.27:0/1587723340 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1016854694 0 0) 0x7f1980003530 con 0x7f19a015b400 2024-01-15T18:05:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.111+0000 7f199de00640 1 -- 172.21.15.27:0/1587723340 >> v1:172.21.15.93:6789/0 conn(0x7f19a00ceb10 legacy=0x7f19a00d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.111+0000 7f199de00640 1 -- 172.21.15.27:0/1587723340 >> v1:172.21.15.27:6789/0 conn(0x7f19a015b400 legacy=0x7f19a00cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.111+0000 7f199de00640 1 -- 172.21.15.27:0/1587723340 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a00cdf30 con 0x7f19a015b020 2024-01-15T18:05:26.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.111+0000 7f199de00640 1 --2- 172.21.15.27:0/1587723340 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f19a015b020 0x7f19a00b0a30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f19900097c0 tx=0x7f199001c370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8538470a4ea0fb15 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:26.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.111+0000 7f199ca00640 1 -- 172.21.15.27:0/1587723340 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1988001680 con 0x7f19a015b020 2024-01-15T18:05:26.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.111+0000 7f199ca00640 1 -- 172.21.15.27:0/1587723340 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19900a7050 con 0x7f19a015b020 2024-01-15T18:05:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.111+0000 7f199ca00640 1 -- 172.21.15.27:0/1587723340 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19900ab620 con 0x7f19a015b020 2024-01-15T18:05:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.115+0000 7f19a5000640 1 -- 172.21.15.27:0/1587723340 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f19a015b020 msgr2=0x7f19a00b0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:26.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.115+0000 7f19a5000640 1 --2- 172.21.15.27:0/1587723340 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f19a015b020 0x7f19a00b0a30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f19900097c0 tx=0x7f199001c370 comp rx=0 tx=0).stop 2024-01-15T18:05:26.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.115+0000 7f19a5000640 1 -- 172.21.15.27:0/1587723340 shutdown_connections 2024-01-15T18:05:26.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.115+0000 7f19a5000640 1 --2- 172.21.15.27:0/1587723340 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f19a015b020 0x7f19a00b0a30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:26.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.115+0000 7f19a5000640 1 -- 172.21.15.27:0/1587723340 >> 172.21.15.27:0/1587723340 conn(0x7f19a0099a10 msgr2=0x7f19a0099e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:26.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.115+0000 7f19a5000640 1 -- 172.21.15.27:0/1587723340 shutdown_connections 2024-01-15T18:05:26.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.115+0000 7f19a5000640 1 -- 172.21.15.27:0/1587723340 wait complete. 2024-01-15T18:05:26.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.115+0000 7f19a5000640 1 Processor -- start 2024-01-15T18:05:26.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.115+0000 7f19a5000640 1 -- start start 2024-01-15T18:05:26.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.115+0000 7f19a5000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19a00ceb10 0x7f19a0157c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:26.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.115+0000 7f19a5000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f19a015b020 0x7f19a0159760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:26.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.115+0000 7f19a5000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19a015b400 0x7f19a0159ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:26.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.115+0000 7f19a5000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a0149c10 con 0x7f19a00ceb10 2024-01-15T18:05:26.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.115+0000 7f19a5000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a00c5fd0 con 0x7f19a015b020 2024-01-15T18:05:26.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.115+0000 7f19a5000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a015a1e0 con 0x7f19a015b400 2024-01-15T18:05:26.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.115+0000 7f199de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19a00ceb10 0x7f19a0157c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:26.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.115+0000 7f199d400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f19a015b020 0x7f19a0159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:26.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.115+0000 7f199de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19a00ceb10 0x7f19a0157c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40590/0 (socket says 172.21.15.27:40590) 2024-01-15T18:05:26.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.115+0000 7f199de00640 1 -- 172.21.15.27:0/2885449988 learned_addr learned my addr 172.21.15.27:0/2885449988 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:26.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.115+0000 7f199e800640 1 --2- 172.21.15.27:0/2885449988 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19a015b400 0x7f19a0159ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:26.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.119+0000 7f199de00640 1 -- 172.21.15.27:0/2885449988 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19a015b400 msgr2=0x7f19a0159ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:26.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.119+0000 7f199de00640 1 --2- 172.21.15.27:0/2885449988 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19a015b400 0x7f19a0159ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:26.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.119+0000 7f199de00640 1 -- 172.21.15.27:0/2885449988 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f19a015b020 msgr2=0x7f19a0159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:26.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.119+0000 7f199de00640 1 --2- 172.21.15.27:0/2885449988 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f19a015b020 0x7f19a0159760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:26.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.119+0000 7f199de00640 1 -- 172.21.15.27:0/2885449988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1990009420 con 0x7f19a00ceb10 2024-01-15T18:05:26.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.119+0000 7f199de00640 1 --2- 172.21.15.27:0/2885449988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19a00ceb10 0x7f19a0157c30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f19900ae040 tx=0x7f199001abb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:26.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.119+0000 7f1996000640 1 -- 172.21.15.27:0/2885449988 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19900abd00 con 0x7f19a00ceb10 2024-01-15T18:05:26.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.119+0000 7f1996000640 1 -- 172.21.15.27:0/2885449988 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19900abe80 con 0x7f19a00ceb10 2024-01-15T18:05:26.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.119+0000 7f1996000640 1 -- 172.21.15.27:0/2885449988 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1990004260 con 0x7f19a00ceb10 2024-01-15T18:05:26.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.119+0000 7f19a5000640 1 -- 172.21.15.27:0/2885449988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a00c53c0 con 0x7f19a00ceb10 2024-01-15T18:05:26.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.119+0000 7f19a5000640 1 -- 172.21.15.27:0/2885449988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a00c5980 con 0x7f19a00ceb10 2024-01-15T18:05:26.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.119+0000 7f19a5000640 1 -- 172.21.15.27:0/2885449988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a004e850 con 0x7f19a00ceb10 2024-01-15T18:05:26.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.119+0000 7f1996000640 1 -- 172.21.15.27:0/2885449988 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f19900043c0 con 0x7f19a00ceb10 2024-01-15T18:05:26.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.123+0000 7f1996000640 1 --2- 172.21.15.27:0/2885449988 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f19780668f0 0x7f1978068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:26.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.123+0000 7f1996000640 1 -- 172.21.15.27:0/2885449988 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f199011e510 con 0x7f19a00ceb10 2024-01-15T18:05:26.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.123+0000 7f199d400640 1 --2- 172.21.15.27:0/2885449988 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f19780668f0 0x7f1978068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:26.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.123+0000 7f199d400640 1 --2- 172.21.15.27:0/2885449988 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f19780668f0 0x7f1978068db0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f19a0153490 tx=0x7f1988001d70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:26.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.127+0000 7f1996000640 1 -- 172.21.15.27:0/2885449988 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f19900eb7f0 con 0x7f19a00ceb10 2024-01-15T18:05:26.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.283+0000 7f19a5000640 1 -- 172.21.15.27:0/2885449988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f19a00b6ec0 con 0x7f19a00ceb10 2024-01-15T18:05:26.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.283+0000 7f1996000640 1 -- 172.21.15.27:0/2885449988 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f19900eee10 con 0x7f19a00ceb10 2024-01-15T18:05:26.288 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-15T18:05:26.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.283+0000 7f19a5000640 1 -- 172.21.15.27:0/2885449988 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f19780668f0 msgr2=0x7f1978068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:26.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.283+0000 7f19a5000640 1 --2- 172.21.15.27:0/2885449988 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f19780668f0 0x7f1978068db0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f19a0153490 tx=0x7f1988001d70 comp rx=0 tx=0).stop 2024-01-15T18:05:26.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.283+0000 7f19a5000640 1 -- 172.21.15.27:0/2885449988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19a00ceb10 msgr2=0x7f19a0157c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:26.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.283+0000 7f19a5000640 1 --2- 172.21.15.27:0/2885449988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19a00ceb10 0x7f19a0157c30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f19900ae040 tx=0x7f199001abb0 comp rx=0 tx=0).stop 2024-01-15T18:05:26.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.283+0000 7f19a5000640 1 -- 172.21.15.27:0/2885449988 shutdown_connections 2024-01-15T18:05:26.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.283+0000 7f19a5000640 1 --2- 172.21.15.27:0/2885449988 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f19780668f0 0x7f1978068db0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:26.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.283+0000 7f19a5000640 1 --2- 172.21.15.27:0/2885449988 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19a015b400 0x7f19a0159ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:26.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.283+0000 7f19a5000640 1 --2- 172.21.15.27:0/2885449988 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f19a015b020 0x7f19a0159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:26.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.283+0000 7f19a5000640 1 --2- 172.21.15.27:0/2885449988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19a00ceb10 0x7f19a0157c30 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:26.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.283+0000 7f19a5000640 1 -- 172.21.15.27:0/2885449988 >> 172.21.15.27:0/2885449988 conn(0x7f19a0099a10 msgr2=0x7f19a0099df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:26.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.283+0000 7f19a5000640 1 -- 172.21.15.27:0/2885449988 shutdown_connections 2024-01-15T18:05:26.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.283+0000 7f19a5000640 1 -- 172.21.15.27:0/2885449988 wait complete. 2024-01-15T18:05:26.310 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:26.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7741200640 1 Processor -- start 2024-01-15T18:05:26.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7741200640 1 -- start start 2024-01-15T18:05:26.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7741200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f773c159bf0 0x7f773c159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:26.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7741200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f773c06d430 0x7f773c0cd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:26.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7741200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f773c159a50 con 0x7f773c071e40 2024-01-15T18:05:26.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7741200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f773c0cce00 con 0x7f773c159bf0 2024-01-15T18:05:26.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7741200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f773c15a510 con 0x7f773c06d430 2024-01-15T18:05:26.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7739400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f773c071e40 0x7f773c0c9be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34794/0 (socket says 172.21.15.27:34794) 2024-01-15T18:05:26.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7739400640 1 -- 172.21.15.27:0/7350122 learned_addr learned my addr 172.21.15.27:0/7350122 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:26.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7739e00640 1 --2- 172.21.15.27:0/7350122 >> v2:172.21.15.93:3300/0 conn(0x7f773c159bf0 0x7f773c159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:26.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f773a800640 1 --2- 172.21.15.27:0/7350122 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f773c06d430 0x7f773c0cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:26.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7738a00640 1 -- 172.21.15.27:0/7350122 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2876644184 0 0) 0x7f7724001120 con 0x7f773c071e40 2024-01-15T18:05:26.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7738a00640 1 -- 172.21.15.27:0/7350122 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f771c003530 con 0x7f773c071e40 2024-01-15T18:05:26.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7738a00640 1 -- 172.21.15.27:0/7350122 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 604075209 0 0) 0x7f771c003530 con 0x7f773c071e40 2024-01-15T18:05:26.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7738a00640 1 -- 172.21.15.27:0/7350122 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f773c06d430 msgr2=0x7f773c0cd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:26.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7738a00640 1 --2- 172.21.15.27:0/7350122 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f773c06d430 0x7f773c0cd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:26.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7738a00640 1 -- 172.21.15.27:0/7350122 >> v2:172.21.15.93:3300/0 conn(0x7f773c159bf0 msgr2=0x7f773c159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:26.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7738a00640 1 --2- 172.21.15.27:0/7350122 >> v2:172.21.15.93:3300/0 conn(0x7f773c159bf0 0x7f773c159fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:26.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7738a00640 1 -- 172.21.15.27:0/7350122 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f773c06d980 con 0x7f773c071e40 2024-01-15T18:05:26.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7738a00640 1 -- 172.21.15.27:0/7350122 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f7724001810 con 0x7f773c071e40 2024-01-15T18:05:26.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7738a00640 1 -- 172.21.15.27:0/7350122 >> v1:172.21.15.27:6789/0 conn(0x7f773c071e40 legacy=0x7f773c0c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:26.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7738a00640 1 --2- 172.21.15.27:0/7350122 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f773c06d430 0x7f771c0039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:26.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7738a00640 1 --2- 172.21.15.27:0/7350122 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f771c003f70 0x7f771c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:26.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7738a00640 1 --2- 172.21.15.27:0/7350122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f771c004d10 0x7f771c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:26.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7738a00640 1 -- 172.21.15.27:0/7350122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c0cce00 con 0x7f771c004d10 2024-01-15T18:05:26.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7738a00640 1 -- 172.21.15.27:0/7350122 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c15a510 con 0x7f771c003f70 2024-01-15T18:05:26.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7738a00640 1 -- 172.21.15.27:0/7350122 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771c00ba40 con 0x7f773c06d430 2024-01-15T18:05:26.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f773a800640 1 --2- 172.21.15.27:0/7350122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f771c004d10 0x7f771c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:26.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7739e00640 1 --2- 172.21.15.27:0/7350122 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f773c06d430 0x7f771c0039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:26.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.495+0000 7f7739400640 1 --2- 172.21.15.27:0/7350122 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f771c003f70 0x7f771c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:26.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.499+0000 7f773a800640 1 -- 172.21.15.27:0/7350122 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f773c06d430 msgr2=0x7f771c0039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:26.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.499+0000 7f773a800640 1 --2- 172.21.15.27:0/7350122 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f773c06d430 0x7f771c0039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:26.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.499+0000 7f773a800640 1 -- 172.21.15.27:0/7350122 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f771c003f70 msgr2=0x7f771c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:26.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.499+0000 7f773a800640 1 --2- 172.21.15.27:0/7350122 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f771c003f70 0x7f771c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:26.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.499+0000 7f773a800640 1 -- 172.21.15.27:0/7350122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f771c003530 con 0x7f771c004d10 2024-01-15T18:05:26.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.499+0000 7f773a800640 1 --2- 172.21.15.27:0/7350122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f771c004d10 0x7f771c005190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7734009fc0 tx=0x7f773401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea984a9a3e362149 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:26.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.499+0000 7f7738a00640 1 -- 172.21.15.27:0/7350122 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77340a5c70 con 0x7f771c004d10 2024-01-15T18:05:26.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.499+0000 7f7738a00640 1 -- 172.21.15.27:0/7350122 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f773400f040 con 0x7f771c004d10 2024-01-15T18:05:26.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.499+0000 7f7738a00640 1 -- 172.21.15.27:0/7350122 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77340ad440 con 0x7f771c004d10 2024-01-15T18:05:26.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.499+0000 7f7741200640 1 -- 172.21.15.27:0/7350122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f771c004d10 msgr2=0x7f771c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:26.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.499+0000 7f7741200640 1 --2- 172.21.15.27:0/7350122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f771c004d10 0x7f771c005190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7734009fc0 tx=0x7f773401cc60 comp rx=0 tx=0).stop 2024-01-15T18:05:26.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.499+0000 7f773a800640 1 -- 172.21.15.27:0/7350122 reap_dead start 2024-01-15T18:05:26.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.503+0000 7f7741200640 1 -- 172.21.15.27:0/7350122 shutdown_connections 2024-01-15T18:05:26.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.503+0000 7f7741200640 1 -- 172.21.15.27:0/7350122 >> 172.21.15.27:0/7350122 conn(0x7f773c0cec60 msgr2=0x7f773c0cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:26.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.503+0000 7f7741200640 1 -- 172.21.15.27:0/7350122 shutdown_connections 2024-01-15T18:05:26.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.503+0000 7f7741200640 1 -- 172.21.15.27:0/7350122 wait complete. 2024-01-15T18:05:26.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.503+0000 7f7741200640 1 Processor -- start 2024-01-15T18:05:26.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.503+0000 7f7741200640 1 -- start start 2024-01-15T18:05:26.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.503+0000 7f7741200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f771c004d10 0x7f773c157970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:26.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.503+0000 7f7741200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f773c1581a0 0x7f773c158620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:26.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.503+0000 7f7741200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f773c0ab790 0x7f773c0abc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:26.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.503+0000 7f7741200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c0aa6f0 con 0x7f773c1581a0 2024-01-15T18:05:26.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.503+0000 7f7741200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c0aa9d0 con 0x7f771c004d10 2024-01-15T18:05:26.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.503+0000 7f7741200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f773c0aacb0 con 0x7f773c0ab790 2024-01-15T18:05:26.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.503+0000 7f7739400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f773c1581a0 0x7f773c158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:26.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.503+0000 7f7739400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f773c1581a0 0x7f773c158620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40600/0 (socket says 172.21.15.27:40600) 2024-01-15T18:05:26.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.503+0000 7f7739400640 1 -- 172.21.15.27:0/3845653931 learned_addr learned my addr 172.21.15.27:0/3845653931 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:26.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.503+0000 7f7739e00640 1 --2- 172.21.15.27:0/3845653931 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f771c004d10 0x7f773c157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:26.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.503+0000 7f773a800640 1 --2- 172.21.15.27:0/3845653931 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f773c0ab790 0x7f773c0abc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:26.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.503+0000 7f7739400640 1 -- 172.21.15.27:0/3845653931 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f773c0ab790 msgr2=0x7f773c0abc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:26.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.503+0000 7f7739400640 1 --2- 172.21.15.27:0/3845653931 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f773c0ab790 0x7f773c0abc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:26.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.503+0000 7f7739400640 1 -- 172.21.15.27:0/3845653931 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f771c004d10 msgr2=0x7f773c157970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:26.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.503+0000 7f7739400640 1 --2- 172.21.15.27:0/3845653931 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f771c004d10 0x7f773c157970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:26.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.503+0000 7f7739400640 1 -- 172.21.15.27:0/3845653931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f773c0aaf90 con 0x7f773c1581a0 2024-01-15T18:05:26.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.503+0000 7f773a800640 1 --2- 172.21.15.27:0/3845653931 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f773c0ab790 0x7f773c0abc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:26.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.507+0000 7f7739400640 1 --2- 172.21.15.27:0/3845653931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f773c1581a0 0x7f773c158620 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7724001940 tx=0x7f77240074b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:26.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.507+0000 7f7732000640 1 -- 172.21.15.27:0/3845653931 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7724007700 con 0x7f773c1581a0 2024-01-15T18:05:26.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.507+0000 7f7732000640 1 -- 172.21.15.27:0/3845653931 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772400f9c0 con 0x7f773c1581a0 2024-01-15T18:05:26.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.507+0000 7f7732000640 1 -- 172.21.15.27:0/3845653931 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772400fd70 con 0x7f773c1581a0 2024-01-15T18:05:26.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.507+0000 7f7741200640 1 -- 172.21.15.27:0/3845653931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f773c0ab270 con 0x7f773c1581a0 2024-01-15T18:05:26.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.507+0000 7f7741200640 1 -- 172.21.15.27:0/3845653931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f773c157eb0 con 0x7f773c1581a0 2024-01-15T18:05:26.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.507+0000 7f7741200640 1 -- 172.21.15.27:0/3845653931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f773c04e850 con 0x7f773c1581a0 2024-01-15T18:05:26.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.507+0000 7f7732000640 1 -- 172.21.15.27:0/3845653931 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f772401d460 con 0x7f773c1581a0 2024-01-15T18:05:26.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.507+0000 7f7732000640 1 --2- 172.21.15.27:0/3845653931 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7714066920 0x7f7714068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:26.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.507+0000 7f7732000640 1 -- 172.21.15.27:0/3845653931 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7724084480 con 0x7f773c1581a0 2024-01-15T18:05:26.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.507+0000 7f7739e00640 1 --2- 172.21.15.27:0/3845653931 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7714066920 0x7f7714068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:26.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.511+0000 7f7739e00640 1 --2- 172.21.15.27:0/3845653931 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7714066920 0x7f7714068de0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f773c158fe0 tx=0x7f772c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:26.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.511+0000 7f7732000640 1 -- 172.21.15.27:0/3845653931 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7724051620 con 0x7f773c1581a0 2024-01-15T18:05:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.675+0000 7f7741200640 1 -- 172.21.15.27:0/3845653931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f773c159a50 con 0x7f773c1581a0 2024-01-15T18:05:26.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.675+0000 7f7732000640 1 -- 172.21.15.27:0/3845653931 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16409 (secure 0 0 0) 0x7f773c159a50 con 0x7f773c1581a0 2024-01-15T18:05:26.681 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:26.681 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:04:59.837052+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:04:56.220608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T18:04:57.210421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:26.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.675+0000 7f7741200640 1 -- 172.21.15.27:0/3845653931 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7714066920 msgr2=0x7f7714068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:26.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.675+0000 7f7741200640 1 --2- 172.21.15.27:0/3845653931 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7714066920 0x7f7714068de0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f773c158fe0 tx=0x7f772c009190 comp rx=0 tx=0).stop 2024-01-15T18:05:26.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.675+0000 7f7741200640 1 -- 172.21.15.27:0/3845653931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f773c1581a0 msgr2=0x7f773c158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:26.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.675+0000 7f7741200640 1 --2- 172.21.15.27:0/3845653931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f773c1581a0 0x7f773c158620 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7724001940 tx=0x7f77240074b0 comp rx=0 tx=0).stop 2024-01-15T18:05:26.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.679+0000 7f7741200640 1 -- 172.21.15.27:0/3845653931 shutdown_connections 2024-01-15T18:05:26.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.679+0000 7f7741200640 1 --2- 172.21.15.27:0/3845653931 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7714066920 0x7f7714068de0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.679+0000 7f7741200640 1 --2- 172.21.15.27:0/3845653931 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f773c0ab790 0x7f773c0abc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.679+0000 7f7741200640 1 --2- 172.21.15.27:0/3845653931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f773c1581a0 0x7f773c158620 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.679+0000 7f7741200640 1 --2- 172.21.15.27:0/3845653931 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f771c004d10 0x7f773c157970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.679+0000 7f7741200640 1 -- 172.21.15.27:0/3845653931 >> 172.21.15.27:0/3845653931 conn(0x7f773c0cec60 msgr2=0x7f773c159130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.679+0000 7f7741200640 1 -- 172.21.15.27:0/3845653931 shutdown_connections 2024-01-15T18:05:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.679+0000 7f7741200640 1 -- 172.21.15.27:0/3845653931 wait complete. 2024-01-15T18:05:26.704 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T18:04:56.220608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T18:04:57.210421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:05:26.704 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:05:26.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.843+0000 7f9f7c200640 1 Processor -- start 2024-01-15T18:05:26.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.843+0000 7f9f7c200640 1 -- start start 2024-01-15T18:05:26.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.843+0000 7f9f7c200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f9f7415d680 0x7f9f740cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:26.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.843+0000 7f9f7c200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f740ccf80 0x7f9f740cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:26.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.843+0000 7f9f7c200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f740cb4d0 con 0x7f9f740cfc30 2024-01-15T18:05:26.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.843+0000 7f9f7c200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f740d0120 con 0x7f9f740ccf80 2024-01-15T18:05:26.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.843+0000 7f9f7c200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f740d0400 con 0x7f9f7415d680 2024-01-15T18:05:26.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.843+0000 7f9f7a200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f740ccf80 0x7f9f740cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:26.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.843+0000 7f9f7a200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f740ccf80 0x7f9f740cdc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40612/0 (socket says 172.21.15.27:40612) 2024-01-15T18:05:26.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.843+0000 7f9f7a200640 1 -- 172.21.15.27:0/2618235396 learned_addr learned my addr 172.21.15.27:0/2618235396 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:26.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.843+0000 7f9f79800640 1 --2- 172.21.15.27:0/2618235396 >> v2:172.21.15.93:3300/0 conn(0x7f9f7415d680 0x7f9f740cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:26.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.843+0000 7f9f7a200640 1 -- 172.21.15.27:0/2618235396 >> v2:172.21.15.93:3300/0 conn(0x7f9f7415d680 msgr2=0x7f9f740cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:26.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.843+0000 7f9f7a200640 1 --2- 172.21.15.27:0/2618235396 >> v2:172.21.15.93:3300/0 conn(0x7f9f7415d680 0x7f9f740cd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:26.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.843+0000 7f9f7a200640 1 -- 172.21.15.27:0/2618235396 >> v1:172.21.15.93:6789/0 conn(0x7f9f740cfc30 legacy=0x7f9f7406b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:26.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.843+0000 7f9f7a200640 1 -- 172.21.15.27:0/2618235396 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f740cb770 con 0x7f9f740ccf80 2024-01-15T18:05:26.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.847+0000 7f9f7a200640 1 --2- 172.21.15.27:0/2618235396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f740ccf80 0x7f9f740cdc40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9f740d0e50 tx=0x7f9f6c01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa26e21b14f46be server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:26.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.847+0000 7f9f73e00640 1 -- 172.21.15.27:0/2618235396 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f740ce850 con 0x7f9f740ccf80 2024-01-15T18:05:26.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.847+0000 7f9f73e00640 1 -- 172.21.15.27:0/2618235396 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6c0a5880 con 0x7f9f740ccf80 2024-01-15T18:05:26.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.847+0000 7f9f73e00640 1 -- 172.21.15.27:0/2618235396 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c0a5c10 con 0x7f9f740ccf80 2024-01-15T18:05:26.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.847+0000 7f9f7c200640 1 -- 172.21.15.27:0/2618235396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f740ccf80 msgr2=0x7f9f740cdc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:26.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.847+0000 7f9f7c200640 1 --2- 172.21.15.27:0/2618235396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f740ccf80 0x7f9f740cdc40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9f740d0e50 tx=0x7f9f6c01c9a0 comp rx=0 tx=0).stop 2024-01-15T18:05:26.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.851+0000 7f9f7c200640 1 -- 172.21.15.27:0/2618235396 shutdown_connections 2024-01-15T18:05:26.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.851+0000 7f9f7c200640 1 --2- 172.21.15.27:0/2618235396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f740ccf80 0x7f9f740cdc40 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:26.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.851+0000 7f9f7c200640 1 --2- 172.21.15.27:0/2618235396 >> v2:172.21.15.93:3300/0 conn(0x7f9f7415d680 0x7f9f740cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:26.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.851+0000 7f9f7c200640 1 -- 172.21.15.27:0/2618235396 >> 172.21.15.27:0/2618235396 conn(0x7f9f740d0e80 msgr2=0x7f9f740d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:26.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.851+0000 7f9f7c200640 1 -- 172.21.15.27:0/2618235396 shutdown_connections 2024-01-15T18:05:26.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.851+0000 7f9f7c200640 1 -- 172.21.15.27:0/2618235396 wait complete. 2024-01-15T18:05:26.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.851+0000 7f9f7c200640 1 Processor -- start 2024-01-15T18:05:26.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.851+0000 7f9f7c200640 1 -- start start 2024-01-15T18:05:26.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.851+0000 7f9f7c200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f740ccf80 0x7f9f7415e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:26.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.851+0000 7f9f7c200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9f740cfc30 0x7f9f7415fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:26.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.851+0000 7f9f7c200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f7415d680 0x7f9f74153f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:26.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.851+0000 7f9f7c200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f741534d0 con 0x7f9f7415d680 2024-01-15T18:05:26.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.851+0000 7f9f7c200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f74160740 con 0x7f9f740ccf80 2024-01-15T18:05:26.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.851+0000 7f9f7c200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f74153740 con 0x7f9f740cfc30 2024-01-15T18:05:26.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.851+0000 7f9f7a200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f7415d680 0x7f9f74153f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:26.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.851+0000 7f9f79800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f740ccf80 0x7f9f7415e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:26.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.851+0000 7f9f79800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f740ccf80 0x7f9f7415e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:53856/0 (socket says 172.21.15.27:53856) 2024-01-15T18:05:26.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.851+0000 7f9f79800640 1 -- 172.21.15.27:0/952169514 learned_addr learned my addr 172.21.15.27:0/952169514 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:26.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.851+0000 7f9f78e00640 1 --2- 172.21.15.27:0/952169514 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9f740cfc30 0x7f9f7415fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:26.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.851+0000 7f9f7a200640 1 -- 172.21.15.27:0/952169514 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9f740cfc30 msgr2=0x7f9f7415fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:26.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.851+0000 7f9f7a200640 1 --2- 172.21.15.27:0/952169514 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9f740cfc30 0x7f9f7415fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:26.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.851+0000 7f9f7a200640 1 -- 172.21.15.27:0/952169514 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f740ccf80 msgr2=0x7f9f7415e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:26.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.851+0000 7f9f7a200640 1 --2- 172.21.15.27:0/952169514 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f740ccf80 0x7f9f7415e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:26.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.851+0000 7f9f7a200640 1 -- 172.21.15.27:0/952169514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6c009b30 con 0x7f9f7415d680 2024-01-15T18:05:26.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.855+0000 7f9f7a200640 1 --2- 172.21.15.27:0/952169514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f7415d680 0x7f9f74153f40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c0ab000 tx=0x7f9f6c019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:26.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.855+0000 7f9f71600640 1 -- 172.21.15.27:0/952169514 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c0bd070 con 0x7f9f7415d680 2024-01-15T18:05:26.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.855+0000 7f9f71600640 1 -- 172.21.15.27:0/952169514 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6c0b8470 con 0x7f9f7415d680 2024-01-15T18:05:26.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.855+0000 7f9f7c200640 1 -- 172.21.15.27:0/952169514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f74153a20 con 0x7f9f7415d680 2024-01-15T18:05:26.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.855+0000 7f9f71600640 1 -- 172.21.15.27:0/952169514 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c0b8820 con 0x7f9f7415d680 2024-01-15T18:05:26.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.855+0000 7f9f7c200640 1 -- 172.21.15.27:0/952169514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f740cab90 con 0x7f9f7415d680 2024-01-15T18:05:26.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.855+0000 7f9f7c200640 1 -- 172.21.15.27:0/952169514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f74160bf0 con 0x7f9f7415d680 2024-01-15T18:05:26.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.855+0000 7f9f71600640 1 -- 172.21.15.27:0/952169514 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f9f6c0b8980 con 0x7f9f7415d680 2024-01-15T18:05:26.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.855+0000 7f9f71600640 1 --2- 172.21.15.27:0/952169514 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9f4c0668f0 0x7f9f4c068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:26.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.855+0000 7f9f71600640 1 -- 172.21.15.27:0/952169514 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9f6c11f2a0 con 0x7f9f7415d680 2024-01-15T18:05:26.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.855+0000 7f9f79800640 1 --2- 172.21.15.27:0/952169514 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9f4c0668f0 0x7f9f4c068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:26.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.859+0000 7f9f79800640 1 --2- 172.21.15.27:0/952169514 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9f4c0668f0 0x7f9f4c068db0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9f741f9ed0 tx=0x7f9f64008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:26.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:26.859+0000 7f9f71600640 1 -- 172.21.15.27:0/952169514 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9f6c0ec500 con 0x7f9f7415d680 2024-01-15T18:05:27.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.027+0000 7f9f7c200640 1 -- 172.21.15.27:0/952169514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9f7404e850 con 0x7f9f7415d680 2024-01-15T18:05:27.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.027+0000 7f9f71600640 1 -- 172.21.15.27:0/952169514 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9f6c0efb20 con 0x7f9f7415d680 2024-01-15T18:05:27.033 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:05:27.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.031+0000 7f9f7c200640 1 -- 172.21.15.27:0/952169514 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9f4c0668f0 msgr2=0x7f9f4c068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:27.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.031+0000 7f9f7c200640 1 --2- 172.21.15.27:0/952169514 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9f4c0668f0 0x7f9f4c068db0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9f741f9ed0 tx=0x7f9f64008040 comp rx=0 tx=0).stop 2024-01-15T18:05:27.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.031+0000 7f9f7c200640 1 -- 172.21.15.27:0/952169514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f7415d680 msgr2=0x7f9f74153f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:27.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.031+0000 7f9f7c200640 1 --2- 172.21.15.27:0/952169514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f7415d680 0x7f9f74153f40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c0ab000 tx=0x7f9f6c019740 comp rx=0 tx=0).stop 2024-01-15T18:05:27.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.031+0000 7f9f7c200640 1 -- 172.21.15.27:0/952169514 shutdown_connections 2024-01-15T18:05:27.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.031+0000 7f9f7c200640 1 --2- 172.21.15.27:0/952169514 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9f4c0668f0 0x7f9f4c068db0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.031+0000 7f9f7c200640 1 --2- 172.21.15.27:0/952169514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f7415d680 0x7f9f74153f40 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.031+0000 7f9f7c200640 1 --2- 172.21.15.27:0/952169514 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9f740cfc30 0x7f9f7415fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.031+0000 7f9f7c200640 1 --2- 172.21.15.27:0/952169514 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f740ccf80 0x7f9f7415e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.031+0000 7f9f7c200640 1 -- 172.21.15.27:0/952169514 >> 172.21.15.27:0/952169514 conn(0x7f9f740d0e80 msgr2=0x7f9f7406f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:27.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.031+0000 7f9f7c200640 1 -- 172.21.15.27:0/952169514 shutdown_connections 2024-01-15T18:05:27.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.031+0000 7f9f7c200640 1 -- 172.21.15.27:0/952169514 wait complete. 2024-01-15T18:05:27.055 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T18:05:27.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcddf800640 1 Processor -- start 2024-01-15T18:05:27.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcddf800640 1 -- start start 2024-01-15T18:05:27.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcddf800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fcdd80cf460 0x7fcdd80cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:27.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcddf800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd80cfd80 con 0x7fcdd80b2590 2024-01-15T18:05:27.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcddf800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd80d19a0 con 0x7fcdd815b4b0 2024-01-15T18:05:27.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcddf800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd80d1c80 con 0x7fcdd80cf460 2024-01-15T18:05:27.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcddd800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcdd80b2590 0x7fcdd80cc130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34796/0 (socket says 172.21.15.27:34796) 2024-01-15T18:05:27.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcddd800640 1 -- 172.21.15.27:0/4054641757 learned_addr learned my addr 172.21.15.27:0/4054641757 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:27.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcddce00640 1 --2- 172.21.15.27:0/4054641757 >> v2:172.21.15.93:3300/0 conn(0x7fcdd80cf460 0x7fcdd80cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:27.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcdd7400640 1 -- 172.21.15.27:0/4054641757 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3192775849 0 0) 0x7fcdd80cfd80 con 0x7fcdd80b2590 2024-01-15T18:05:27.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcdd7400640 1 -- 172.21.15.27:0/4054641757 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdb8003530 con 0x7fcdd80b2590 2024-01-15T18:05:27.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcdd7400640 1 -- 172.21.15.27:0/4054641757 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 470195463 0 0) 0x7fcdb8003530 con 0x7fcdd80b2590 2024-01-15T18:05:27.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcdd7400640 1 -- 172.21.15.27:0/4054641757 >> v2:172.21.15.93:3300/0 conn(0x7fcdd80cf460 msgr2=0x7fcdd80cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:27.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcdd7400640 1 --2- 172.21.15.27:0/4054641757 >> v2:172.21.15.93:3300/0 conn(0x7fcdd80cf460 0x7fcdd80cf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcdd7400640 1 -- 172.21.15.27:0/4054641757 >> v1:172.21.15.93:6789/0 conn(0x7fcdd815b4b0 legacy=0x7fcdd80cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:27.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcdd7400640 1 -- 172.21.15.27:0/4054641757 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd80b07c0 con 0x7fcdd80b2590 2024-01-15T18:05:27.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcdd7400640 1 -- 172.21.15.27:0/4054641757 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fcdd0001a00 con 0x7fcdd80b2590 2024-01-15T18:05:27.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcdd7400640 1 -- 172.21.15.27:0/4054641757 >> v1:172.21.15.27:6789/0 conn(0x7fcdd80b2590 legacy=0x7fcdd80cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:27.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcdd7400640 1 --2- 172.21.15.27:0/4054641757 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcdb80039f0 0x7fcdb8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:27.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcdd7400640 1 --2- 172.21.15.27:0/4054641757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdb80047d0 0x7fcdb8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:27.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcdd7400640 1 --2- 172.21.15.27:0/4054641757 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcdb8005190 0x7fcdb800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:27.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcdd7400640 1 -- 172.21.15.27:0/4054641757 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd80d1c80 con 0x7fcdb80047d0 2024-01-15T18:05:27.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcdd7400640 1 -- 172.21.15.27:0/4054641757 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb800c0c0 con 0x7fcdb80039f0 2024-01-15T18:05:27.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcdd7400640 1 -- 172.21.15.27:0/4054641757 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb800c280 con 0x7fcdb8005190 2024-01-15T18:05:27.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcdd7e00640 1 --2- 172.21.15.27:0/4054641757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdb80047d0 0x7fcdb8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:27.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcddce00640 1 --2- 172.21.15.27:0/4054641757 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcdb80039f0 0x7fcdb8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:27.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcddd800640 1 --2- 172.21.15.27:0/4054641757 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcdb8005190 0x7fcdb800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:27.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcdd7e00640 1 -- 172.21.15.27:0/4054641757 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcdb8005190 msgr2=0x7fcdb800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:27.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcdd7e00640 1 --2- 172.21.15.27:0/4054641757 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcdb8005190 0x7fcdb800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcdd7e00640 1 -- 172.21.15.27:0/4054641757 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcdb80039f0 msgr2=0x7fcdb8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:27.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcdd7e00640 1 --2- 172.21.15.27:0/4054641757 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcdb80039f0 0x7fcdb8003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcdd7e00640 1 -- 172.21.15.27:0/4054641757 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb8003530 con 0x7fcdb80047d0 2024-01-15T18:05:27.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.195+0000 7fcddd800640 1 -- 172.21.15.27:0/4054641757 reap_dead start 2024-01-15T18:05:27.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.199+0000 7fcdd7e00640 1 --2- 172.21.15.27:0/4054641757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdb80047d0 0x7fcdb8004c50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0001780 tx=0x7fcdc001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f643cfd562f822f5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.199+0000 7fcdd7400640 1 -- 172.21.15.27:0/4054641757 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc00052d0 con 0x7fcdb80047d0 2024-01-15T18:05:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.199+0000 7fcdd7400640 1 -- 172.21.15.27:0/4054641757 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc00a7050 con 0x7fcdb80047d0 2024-01-15T18:05:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.199+0000 7fcdd7400640 1 -- 172.21.15.27:0/4054641757 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc0003600 con 0x7fcdb80047d0 2024-01-15T18:05:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.199+0000 7fcddf800640 1 -- 172.21.15.27:0/4054641757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdb80047d0 msgr2=0x7fcdb8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.199+0000 7fcddf800640 1 --2- 172.21.15.27:0/4054641757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdb80047d0 0x7fcdb8004c50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcdc0001780 tx=0x7fcdc001c400 comp rx=0 tx=0).stop 2024-01-15T18:05:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.199+0000 7fcddf800640 1 -- 172.21.15.27:0/4054641757 shutdown_connections 2024-01-15T18:05:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.199+0000 7fcddf800640 1 --2- 172.21.15.27:0/4054641757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdb80047d0 0x7fcdb8004c50 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.199+0000 7fcddf800640 1 -- 172.21.15.27:0/4054641757 >> 172.21.15.27:0/4054641757 conn(0x7fcdd80d0e80 msgr2=0x7fcdd80d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.199+0000 7fcddf800640 1 -- 172.21.15.27:0/4054641757 shutdown_connections 2024-01-15T18:05:27.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.199+0000 7fcddf800640 1 -- 172.21.15.27:0/4054641757 wait complete. 2024-01-15T18:05:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.203+0000 7fcddf800640 1 Processor -- start 2024-01-15T18:05:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.203+0000 7fcddf800640 1 -- start start 2024-01-15T18:05:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.203+0000 7fcddf800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcdb80047d0 0x7fcdd81503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.203+0000 7fcddf800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcdd80c9e80 0x7fcdd806c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.203+0000 7fcddf800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd806cb90 0x7fcdd815d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.203+0000 7fcddf800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd815c7a0 con 0x7fcdd806cb90 2024-01-15T18:05:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.203+0000 7fcddf800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd806cff0 con 0x7fcdb80047d0 2024-01-15T18:05:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.203+0000 7fcddf800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd8150920 con 0x7fcdd80c9e80 2024-01-15T18:05:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.203+0000 7fcddd800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd806cb90 0x7fcdd815d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.203+0000 7fcddd800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd806cb90 0x7fcdd815d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40624/0 (socket says 172.21.15.27:40624) 2024-01-15T18:05:27.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.203+0000 7fcddd800640 1 -- 172.21.15.27:0/104449631 learned_addr learned my addr 172.21.15.27:0/104449631 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:27.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.203+0000 7fcdd7e00640 1 --2- 172.21.15.27:0/104449631 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcdd80c9e80 0x7fcdd806c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:27.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.203+0000 7fcddce00640 1 --2- 172.21.15.27:0/104449631 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcdb80047d0 0x7fcdd81503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:27.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.203+0000 7fcddd800640 1 -- 172.21.15.27:0/104449631 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcdd80c9e80 msgr2=0x7fcdd806c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:27.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.203+0000 7fcddd800640 1 --2- 172.21.15.27:0/104449631 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcdd80c9e80 0x7fcdd806c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.203+0000 7fcddd800640 1 -- 172.21.15.27:0/104449631 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcdb80047d0 msgr2=0x7fcdd81503e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:27.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.203+0000 7fcddd800640 1 --2- 172.21.15.27:0/104449631 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcdb80047d0 0x7fcdd81503e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.203+0000 7fcddd800640 1 -- 172.21.15.27:0/104449631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd815b8d0 con 0x7fcdd806cb90 2024-01-15T18:05:27.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.203+0000 7fcddd800640 1 --2- 172.21.15.27:0/104449631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd806cb90 0x7fcdd815d490 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8072390 tx=0x7fcdd0011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:27.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.203+0000 7fcdd4c00640 1 -- 172.21.15.27:0/104449631 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd000f840 con 0x7fcdd806cb90 2024-01-15T18:05:27.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.203+0000 7fcddf800640 1 -- 172.21.15.27:0/104449631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd815bb20 con 0x7fcdd806cb90 2024-01-15T18:05:27.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.203+0000 7fcdd4c00640 1 -- 172.21.15.27:0/104449631 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd000f9c0 con 0x7fcdd806cb90 2024-01-15T18:05:27.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.203+0000 7fcdd4c00640 1 -- 172.21.15.27:0/104449631 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd000fd70 con 0x7fcdd806cb90 2024-01-15T18:05:27.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.203+0000 7fcddf800640 1 -- 172.21.15.27:0/104449631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd815c0e0 con 0x7fcdd806cb90 2024-01-15T18:05:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.207+0000 7fcddf800640 1 -- 172.21.15.27:0/104449631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd815c8e0 con 0x7fcdd806cb90 2024-01-15T18:05:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.207+0000 7fcdd4c00640 1 -- 172.21.15.27:0/104449631 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fcdd001f460 con 0x7fcdd806cb90 2024-01-15T18:05:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.207+0000 7fcdd4c00640 1 --2- 172.21.15.27:0/104449631 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcdb0066920 0x7fcdb0068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.207+0000 7fcdd4c00640 1 -- 172.21.15.27:0/104449631 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcdd00871c0 con 0x7fcdd806cb90 2024-01-15T18:05:27.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.207+0000 7fcddce00640 1 --2- 172.21.15.27:0/104449631 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcdb0066920 0x7fcdb0068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:27.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.211+0000 7fcddce00640 1 --2- 172.21.15.27:0/104449631 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcdb0066920 0x7fcdb0068de0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fcdd815e800 tx=0x7fcdc8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:27.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.211+0000 7fcdd4c00640 1 -- 172.21.15.27:0/104449631 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fcdd815c8e0 con 0x7fcdd806cb90 2024-01-15T18:05:27.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.367+0000 7fcddf800640 1 -- 172.21.15.27:0/104449631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcdd815c8e0 con 0x7fcdd806cb90 2024-01-15T18:05:27.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.367+0000 7fcdd4c00640 1 -- 172.21.15.27:0/104449631 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcdd815c8e0 con 0x7fcdd806cb90 2024-01-15T18:05:27.374 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-15T18:05:27.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.371+0000 7fcddf800640 1 -- 172.21.15.27:0/104449631 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcdb0066920 msgr2=0x7fcdb0068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:27.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.371+0000 7fcddf800640 1 --2- 172.21.15.27:0/104449631 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcdb0066920 0x7fcdb0068de0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fcdd815e800 tx=0x7fcdc8009190 comp rx=0 tx=0).stop 2024-01-15T18:05:27.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.371+0000 7fcddf800640 1 -- 172.21.15.27:0/104449631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd806cb90 msgr2=0x7fcdd815d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:27.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.371+0000 7fcddf800640 1 --2- 172.21.15.27:0/104449631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd806cb90 0x7fcdd815d490 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8072390 tx=0x7fcdd0011b60 comp rx=0 tx=0).stop 2024-01-15T18:05:27.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.371+0000 7fcddf800640 1 -- 172.21.15.27:0/104449631 shutdown_connections 2024-01-15T18:05:27.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.371+0000 7fcddf800640 1 --2- 172.21.15.27:0/104449631 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcdb0066920 0x7fcdb0068de0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.371+0000 7fcddf800640 1 --2- 172.21.15.27:0/104449631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcdd806cb90 0x7fcdd815d490 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.371+0000 7fcddf800640 1 --2- 172.21.15.27:0/104449631 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcdd80c9e80 0x7fcdd806c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.371+0000 7fcddf800640 1 --2- 172.21.15.27:0/104449631 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcdb80047d0 0x7fcdd81503e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.371+0000 7fcddf800640 1 -- 172.21.15.27:0/104449631 >> 172.21.15.27:0/104449631 conn(0x7fcdd80d0e80 msgr2=0x7fcdd815a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:27.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.371+0000 7fcddf800640 1 -- 172.21.15.27:0/104449631 shutdown_connections 2024-01-15T18:05:27.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.371+0000 7fcddf800640 1 -- 172.21.15.27:0/104449631 wait complete. 2024-01-15T18:05:27.396 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T18:05:27.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.535+0000 7faab6e00640 1 Processor -- start 2024-01-15T18:05:27.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.535+0000 7faab6e00640 1 -- start start 2024-01-15T18:05:27.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.539+0000 7faab6e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7faab00cf460 0x7faab00cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:27.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.539+0000 7faab6e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faab015b4b0 0x7faab00cba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:27.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.539+0000 7faab6e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab00cfd80 con 0x7faab00cec60 2024-01-15T18:05:27.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.539+0000 7faab6e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faab00d19a0 con 0x7faab00cf460 2024-01-15T18:05:27.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.539+0000 7faab6e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab00d1c80 con 0x7faab015b4b0 2024-01-15T18:05:27.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.539+0000 7faab4e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7faab00cec60 0x7faab00cbf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34798/0 (socket says 172.21.15.27:34798) 2024-01-15T18:05:27.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.539+0000 7faab4e00640 1 -- 172.21.15.27:0/1541294336 learned_addr learned my addr 172.21.15.27:0/1541294336 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:27.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.539+0000 7faaafe00640 1 --2- 172.21.15.27:0/1541294336 >> v2:172.21.15.93:3300/0 conn(0x7faab00cf460 0x7faab00cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:27.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.539+0000 7faaaf400640 1 --2- 172.21.15.27:0/1541294336 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faab015b4b0 0x7faab00cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:27.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.539+0000 7faaaea00640 1 -- 172.21.15.27:0/1541294336 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1697838000 0 0) 0x7faab00cfd80 con 0x7faab00cec60 2024-01-15T18:05:27.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.539+0000 7faaaea00640 1 -- 172.21.15.27:0/1541294336 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa90003530 con 0x7faab00cec60 2024-01-15T18:05:27.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.539+0000 7faaafe00640 1 -- 172.21.15.27:0/1541294336 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faab015b4b0 msgr2=0x7faab00cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:27.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.539+0000 7faaafe00640 1 --2- 172.21.15.27:0/1541294336 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faab015b4b0 0x7faab00cba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.539+0000 7faaaea00640 1 -- 172.21.15.27:0/1541294336 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 873119070 0 0) 0x7faa90003530 con 0x7faab00cec60 2024-01-15T18:05:27.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.539+0000 7faaafe00640 1 -- 172.21.15.27:0/1541294336 >> v1:172.21.15.27:6789/0 conn(0x7faab00cec60 legacy=0x7faab00cbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:27.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.539+0000 7faaafe00640 1 -- 172.21.15.27:0/1541294336 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab00b07c0 con 0x7faab00cf460 2024-01-15T18:05:27.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.543+0000 7faaafe00640 1 --2- 172.21.15.27:0/1541294336 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faab00cf460 0x7faab00cf840 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7faaa00097c0 tx=0x7faaa001c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a957beb2d9f531fd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:27.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.543+0000 7faaaea00640 1 -- 172.21.15.27:0/1541294336 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa8001a00 con 0x7faab00cf460 2024-01-15T18:05:27.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.543+0000 7faaaea00640 1 -- 172.21.15.27:0/1541294336 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa00a7050 con 0x7faab00cf460 2024-01-15T18:05:27.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.543+0000 7faaaea00640 1 -- 172.21.15.27:0/1541294336 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa00ab620 con 0x7faab00cf460 2024-01-15T18:05:27.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.543+0000 7faab6e00640 1 -- 172.21.15.27:0/1541294336 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faab00cf460 msgr2=0x7faab00cf840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:27.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.543+0000 7faab6e00640 1 --2- 172.21.15.27:0/1541294336 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faab00cf460 0x7faab00cf840 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7faaa00097c0 tx=0x7faaa001c3a0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.543+0000 7faab6e00640 1 -- 172.21.15.27:0/1541294336 shutdown_connections 2024-01-15T18:05:27.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.543+0000 7faab6e00640 1 --2- 172.21.15.27:0/1541294336 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faab015b4b0 0x7faab00cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.543+0000 7faab6e00640 1 --2- 172.21.15.27:0/1541294336 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faab00cf460 0x7faab00cf840 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.543+0000 7faab6e00640 1 -- 172.21.15.27:0/1541294336 >> 172.21.15.27:0/1541294336 conn(0x7faab00d0e80 msgr2=0x7faab00d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:27.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.543+0000 7faab6e00640 1 -- 172.21.15.27:0/1541294336 shutdown_connections 2024-01-15T18:05:27.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.543+0000 7faab6e00640 1 -- 172.21.15.27:0/1541294336 wait complete. 2024-01-15T18:05:27.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.543+0000 7faab6e00640 1 Processor -- start 2024-01-15T18:05:27.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.543+0000 7faab6e00640 1 -- start start 2024-01-15T18:05:27.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faab6e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab00cec60 0x7faab014bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:27.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faab6e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faab00cf460 0x7faab0154c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:27.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faab6e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faab015b4b0 0x7faab006f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:27.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faab6e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab015b8d0 con 0x7faab00cec60 2024-01-15T18:05:27.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faab6e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab014c180 con 0x7faab00cf460 2024-01-15T18:05:27.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faab6e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab015cbc0 con 0x7faab015b4b0 2024-01-15T18:05:27.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faaafe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab00cec60 0x7faab014bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:27.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faaafe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab00cec60 0x7faab014bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40628/0 (socket says 172.21.15.27:40628) 2024-01-15T18:05:27.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faaafe00640 1 -- 172.21.15.27:0/3329054712 learned_addr learned my addr 172.21.15.27:0/3329054712 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:27.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faaaf400640 1 --2- 172.21.15.27:0/3329054712 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faab00cf460 0x7faab0154c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:27.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faab4e00640 1 --2- 172.21.15.27:0/3329054712 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faab015b4b0 0x7faab006f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:27.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faaafe00640 1 -- 172.21.15.27:0/3329054712 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faab015b4b0 msgr2=0x7faab006f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:27.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faaafe00640 1 --2- 172.21.15.27:0/3329054712 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faab015b4b0 0x7faab006f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faaafe00640 1 -- 172.21.15.27:0/3329054712 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faab00cf460 msgr2=0x7faab0154c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:27.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faaafe00640 1 --2- 172.21.15.27:0/3329054712 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faab00cf460 0x7faab0154c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faaafe00640 1 -- 172.21.15.27:0/3329054712 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa0009420 con 0x7faab00cec60 2024-01-15T18:05:27.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faab4e00640 1 --2- 172.21.15.27:0/3329054712 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faab015b4b0 0x7faab006f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:27.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faaaf400640 1 --2- 172.21.15.27:0/3329054712 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faab00cf460 0x7faab0154c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:27.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faaafe00640 1 --2- 172.21.15.27:0/3329054712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab00cec60 0x7faab014bc40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7faaa00ae040 tx=0x7faaa001abe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:27.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faaa7e00640 1 -- 172.21.15.27:0/3329054712 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa00abea0 con 0x7faab00cec60 2024-01-15T18:05:27.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faaa7e00640 1 -- 172.21.15.27:0/3329054712 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa00b7450 con 0x7faab00cec60 2024-01-15T18:05:27.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faaa7e00640 1 -- 172.21.15.27:0/3329054712 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa00b7820 con 0x7faab00cec60 2024-01-15T18:05:27.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faab6e00640 1 -- 172.21.15.27:0/3329054712 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab015bb20 con 0x7faab00cec60 2024-01-15T18:05:27.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faab6e00640 1 -- 172.21.15.27:0/3329054712 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab015c0e0 con 0x7faab00cec60 2024-01-15T18:05:27.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faab6e00640 1 -- 172.21.15.27:0/3329054712 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab004e850 con 0x7faab00cec60 2024-01-15T18:05:27.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.547+0000 7faaa7e00640 1 -- 172.21.15.27:0/3329054712 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7faaa00b7980 con 0x7faab00cec60 2024-01-15T18:05:27.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.551+0000 7faaa7e00640 1 --2- 172.21.15.27:0/3329054712 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faa880668f0 0x7faa88068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:27.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.551+0000 7faaa7e00640 1 -- 172.21.15.27:0/3329054712 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faaa011eab0 con 0x7faab00cec60 2024-01-15T18:05:27.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.551+0000 7faaaf400640 1 --2- 172.21.15.27:0/3329054712 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faa880668f0 0x7faa88068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:27.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.551+0000 7faaaf400640 1 --2- 172.21.15.27:0/3329054712 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faa880668f0 0x7faa88068db0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7faab0155cb0 tx=0x7faa98008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:27.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.555+0000 7faaa7e00640 1 -- 172.21.15.27:0/3329054712 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7faaa00ebce0 con 0x7faab00cec60 2024-01-15T18:05:27.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.711+0000 7faab6e00640 1 -- 172.21.15.27:0/3329054712 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7faab00674f0 con 0x7faab00cec60 2024-01-15T18:05:27.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.711+0000 7faaa7e00640 1 -- 172.21.15.27:0/3329054712 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7faaa00b00a0 con 0x7faab00cec60 2024-01-15T18:05:27.718 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-15T18:05:27.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.715+0000 7faab6e00640 1 -- 172.21.15.27:0/3329054712 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faa880668f0 msgr2=0x7faa88068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:27.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.715+0000 7faab6e00640 1 --2- 172.21.15.27:0/3329054712 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faa880668f0 0x7faa88068db0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7faab0155cb0 tx=0x7faa98008040 comp rx=0 tx=0).stop 2024-01-15T18:05:27.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.715+0000 7faab6e00640 1 -- 172.21.15.27:0/3329054712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab00cec60 msgr2=0x7faab014bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:27.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.715+0000 7faab6e00640 1 --2- 172.21.15.27:0/3329054712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab00cec60 0x7faab014bc40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7faaa00ae040 tx=0x7faaa001abe0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.715+0000 7faab6e00640 1 -- 172.21.15.27:0/3329054712 shutdown_connections 2024-01-15T18:05:27.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.715+0000 7faab6e00640 1 --2- 172.21.15.27:0/3329054712 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faa880668f0 0x7faa88068db0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.715+0000 7faab6e00640 1 --2- 172.21.15.27:0/3329054712 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faab015b4b0 0x7faab006f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.715+0000 7faab6e00640 1 --2- 172.21.15.27:0/3329054712 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faab00cf460 0x7faab0154c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.715+0000 7faab6e00640 1 --2- 172.21.15.27:0/3329054712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab00cec60 0x7faab014bc40 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.715+0000 7faab6e00640 1 -- 172.21.15.27:0/3329054712 >> 172.21.15.27:0/3329054712 conn(0x7faab00d0e80 msgr2=0x7faab00aec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:27.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.715+0000 7faab6e00640 1 -- 172.21.15.27:0/3329054712 shutdown_connections 2024-01-15T18:05:27.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.715+0000 7faab6e00640 1 -- 172.21.15.27:0/3329054712 wait complete. 2024-01-15T18:05:27.740 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:27.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.875+0000 7fb18da00640 1 Processor -- start 2024-01-15T18:05:27.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.875+0000 7fb18da00640 1 -- start start 2024-01-15T18:05:27.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.879+0000 7fb18da00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb18815e7d0 0x7fb1880cd730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:27.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.879+0000 7fb18da00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fb1880cdc70 0x7fb1880ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:27.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.879+0000 7fb18da00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1880d1a70 con 0x7fb1880c4d80 2024-01-15T18:05:27.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.879+0000 7fb18da00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1880ce5f0 con 0x7fb18815e7d0 2024-01-15T18:05:27.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.879+0000 7fb18da00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1880ccdd0 con 0x7fb1880cdc70 2024-01-15T18:05:27.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.879+0000 7fb186a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb18815e7d0 0x7fb1880cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:27.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.879+0000 7fb186a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb18815e7d0 0x7fb1880cd730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40630/0 (socket says 172.21.15.27:40630) 2024-01-15T18:05:27.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.879+0000 7fb187400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb1880c4d80 0x7fb1880c51f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34812/0 (socket says 172.21.15.27:34812) 2024-01-15T18:05:27.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.879+0000 7fb186a00640 1 -- 172.21.15.27:0/263587304 learned_addr learned my addr 172.21.15.27:0/263587304 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:27.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.879+0000 7fb186000640 1 --2- 172.21.15.27:0/263587304 >> v2:172.21.15.93:3300/0 conn(0x7fb1880cdc70 0x7fb1880ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:27.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.879+0000 7fb186a00640 1 -- 172.21.15.27:0/263587304 >> v2:172.21.15.93:3300/0 conn(0x7fb1880cdc70 msgr2=0x7fb1880ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:27.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.879+0000 7fb186a00640 1 --2- 172.21.15.27:0/263587304 >> v2:172.21.15.93:3300/0 conn(0x7fb1880cdc70 0x7fb1880ce0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.879+0000 7fb186a00640 1 -- 172.21.15.27:0/263587304 >> v1:172.21.15.27:6789/0 conn(0x7fb1880c4d80 legacy=0x7fb1880c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:27.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.879+0000 7fb185600640 1 -- 172.21.15.27:0/263587304 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802123724 0 0) 0x7fb1880d1a70 con 0x7fb1880c4d80 2024-01-15T18:05:27.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.879+0000 7fb186a00640 1 -- 172.21.15.27:0/263587304 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb18815ebb0 con 0x7fb18815e7d0 2024-01-15T18:05:27.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.879+0000 7fb186000640 1 --2- 172.21.15.27:0/263587304 >> v2:172.21.15.93:3300/0 conn(0x7fb1880cdc70 0x7fb1880ce0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:27.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.879+0000 7fb186a00640 1 --2- 172.21.15.27:0/263587304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb18815e7d0 0x7fb1880cd730 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb1880cb810 tx=0x7fb17801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7840ce93ec73fb6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:27.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.883+0000 7fb185600640 1 -- 172.21.15.27:0/263587304 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1780ac640 con 0x7fb18815e7d0 2024-01-15T18:05:27.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.883+0000 7fb185600640 1 -- 172.21.15.27:0/263587304 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1780ac7a0 con 0x7fb18815e7d0 2024-01-15T18:05:27.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.883+0000 7fb185600640 1 -- 172.21.15.27:0/263587304 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1780acb30 con 0x7fb18815e7d0 2024-01-15T18:05:27.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.883+0000 7fb18da00640 1 -- 172.21.15.27:0/263587304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb18815e7d0 msgr2=0x7fb1880cd730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:27.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.883+0000 7fb18da00640 1 --2- 172.21.15.27:0/263587304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb18815e7d0 0x7fb1880cd730 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb1880cb810 tx=0x7fb17801c4d0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.883+0000 7fb18da00640 1 -- 172.21.15.27:0/263587304 shutdown_connections 2024-01-15T18:05:27.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.883+0000 7fb18da00640 1 --2- 172.21.15.27:0/263587304 >> v2:172.21.15.93:3300/0 conn(0x7fb1880cdc70 0x7fb1880ce0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.883+0000 7fb18da00640 1 --2- 172.21.15.27:0/263587304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb18815e7d0 0x7fb1880cd730 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.883+0000 7fb18da00640 1 -- 172.21.15.27:0/263587304 >> 172.21.15.27:0/263587304 conn(0x7fb1880d0f50 msgr2=0x7fb1880d1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:27.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.883+0000 7fb18da00640 1 -- 172.21.15.27:0/263587304 shutdown_connections 2024-01-15T18:05:27.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.883+0000 7fb18da00640 1 -- 172.21.15.27:0/263587304 wait complete. 2024-01-15T18:05:27.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.883+0000 7fb18da00640 1 Processor -- start 2024-01-15T18:05:27.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.883+0000 7fb18da00640 1 -- start start 2024-01-15T18:05:27.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.883+0000 7fb18da00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1880c4d80 0x7fb188151c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:27.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.883+0000 7fb18da00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb1880cdc70 0x7fb18815a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:27.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.883+0000 7fb18da00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb18815e7d0 0x7fb18815aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:27.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.883+0000 7fb18da00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb188152190 con 0x7fb18815e7d0 2024-01-15T18:05:27.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.883+0000 7fb18da00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1881530a0 con 0x7fb1880c4d80 2024-01-15T18:05:27.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.883+0000 7fb18da00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1881640f0 con 0x7fb1880cdc70 2024-01-15T18:05:27.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.887+0000 7fb187400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb18815e7d0 0x7fb18815aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:27.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.887+0000 7fb187400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb18815e7d0 0x7fb18815aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40646/0 (socket says 172.21.15.27:40646) 2024-01-15T18:05:27.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.887+0000 7fb187400640 1 -- 172.21.15.27:0/2120454479 learned_addr learned my addr 172.21.15.27:0/2120454479 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:27.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.887+0000 7fb186a00640 1 --2- 172.21.15.27:0/2120454479 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1880c4d80 0x7fb188151c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:27.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.887+0000 7fb186000640 1 --2- 172.21.15.27:0/2120454479 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb1880cdc70 0x7fb18815a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:27.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.887+0000 7fb187400640 1 -- 172.21.15.27:0/2120454479 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb1880cdc70 msgr2=0x7fb18815a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:27.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.887+0000 7fb187400640 1 --2- 172.21.15.27:0/2120454479 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb1880cdc70 0x7fb18815a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.887+0000 7fb187400640 1 -- 172.21.15.27:0/2120454479 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1880c4d80 msgr2=0x7fb188151c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:27.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.887+0000 7fb187400640 1 --2- 172.21.15.27:0/2120454479 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1880c4d80 0x7fb188151c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.887+0000 7fb187400640 1 -- 172.21.15.27:0/2120454479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1780094d0 con 0x7fb18815e7d0 2024-01-15T18:05:27.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.887+0000 7fb186a00640 1 --2- 172.21.15.27:0/2120454479 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1880c4d80 0x7fb188151c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:27.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.887+0000 7fb187400640 1 --2- 172.21.15.27:0/2120454479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb18815e7d0 0x7fb18815aa30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb18000c7b0 tx=0x7fb18000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:27.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.887+0000 7fb17ea00640 1 -- 172.21.15.27:0/2120454479 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18000a640 con 0x7fb18815e7d0 2024-01-15T18:05:27.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.887+0000 7fb17ea00640 1 -- 172.21.15.27:0/2120454479 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb18000a7c0 con 0x7fb18815e7d0 2024-01-15T18:05:27.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.887+0000 7fb18da00640 1 -- 172.21.15.27:0/2120454479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1881643d0 con 0x7fb18815e7d0 2024-01-15T18:05:27.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.887+0000 7fb17ea00640 1 -- 172.21.15.27:0/2120454479 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18000ab70 con 0x7fb18815e7d0 2024-01-15T18:05:27.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.887+0000 7fb18da00640 1 -- 172.21.15.27:0/2120454479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb188153310 con 0x7fb18815e7d0 2024-01-15T18:05:27.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.887+0000 7fb18da00640 1 -- 172.21.15.27:0/2120454479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb188153530 con 0x7fb18815e7d0 2024-01-15T18:05:27.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.887+0000 7fb17ea00640 1 -- 172.21.15.27:0/2120454479 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fb18000acd0 con 0x7fb18815e7d0 2024-01-15T18:05:27.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.891+0000 7fb17ea00640 1 --2- 172.21.15.27:0/2120454479 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb1600668f0 0x7fb160068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:27.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.891+0000 7fb17ea00640 1 -- 172.21.15.27:0/2120454479 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb1800837e0 con 0x7fb18815e7d0 2024-01-15T18:05:27.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.891+0000 7fb186a00640 1 --2- 172.21.15.27:0/2120454479 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb1600668f0 0x7fb160068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:27.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.891+0000 7fb186a00640 1 --2- 172.21.15.27:0/2120454479 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb1600668f0 0x7fb160068db0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb1780ae000 tx=0x7fb1780a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:27.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:27.895+0000 7fb17ea00640 1 -- 172.21.15.27:0/2120454479 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb180050ac0 con 0x7fb18815e7d0 2024-01-15T18:05:28.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.051+0000 7fb18da00640 1 -- 172.21.15.27:0/2120454479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb18804e850 con 0x7fb18815e7d0 2024-01-15T18:05:28.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.055+0000 7fb17ea00640 1 -- 172.21.15.27:0/2120454479 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16409 (secure 0 0 0) 0x7fb180018090 con 0x7fb18815e7d0 2024-01-15T18:05:28.059 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:28.059 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:04:59.837052+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:04:56.220608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T18:04:57.210421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:28.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.055+0000 7fb18da00640 1 -- 172.21.15.27:0/2120454479 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb1600668f0 msgr2=0x7fb160068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.055+0000 7fb18da00640 1 --2- 172.21.15.27:0/2120454479 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb1600668f0 0x7fb160068db0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb1780ae000 tx=0x7fb1780a7000 comp rx=0 tx=0).stop 2024-01-15T18:05:28.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.055+0000 7fb18da00640 1 -- 172.21.15.27:0/2120454479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb18815e7d0 msgr2=0x7fb18815aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.055+0000 7fb18da00640 1 --2- 172.21.15.27:0/2120454479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb18815e7d0 0x7fb18815aa30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb18000c7b0 tx=0x7fb18000cc70 comp rx=0 tx=0).stop 2024-01-15T18:05:28.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.055+0000 7fb18da00640 1 -- 172.21.15.27:0/2120454479 shutdown_connections 2024-01-15T18:05:28.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.055+0000 7fb18da00640 1 --2- 172.21.15.27:0/2120454479 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb1600668f0 0x7fb160068db0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.055+0000 7fb18da00640 1 --2- 172.21.15.27:0/2120454479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb18815e7d0 0x7fb18815aa30 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.055+0000 7fb18da00640 1 --2- 172.21.15.27:0/2120454479 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb1880cdc70 0x7fb18815a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.055+0000 7fb18da00640 1 --2- 172.21.15.27:0/2120454479 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1880c4d80 0x7fb188151c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.055+0000 7fb18da00640 1 -- 172.21.15.27:0/2120454479 >> 172.21.15.27:0/2120454479 conn(0x7fb1880d0f50 msgr2=0x7fb18815c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:28.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.055+0000 7fb18da00640 1 -- 172.21.15.27:0/2120454479 shutdown_connections 2024-01-15T18:05:28.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.055+0000 7fb18da00640 1 -- 172.21.15.27:0/2120454479 wait complete. 2024-01-15T18:05:28.082 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T18:04:56.220608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T18:04:57.210421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:05:28.082 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:05:28.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.219+0000 7ff301400640 1 Processor -- start 2024-01-15T18:05:28.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.219+0000 7ff301400640 1 -- start start 2024-01-15T18:05:28.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.223+0000 7ff301400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff2fc15f550 0x7ff2fc0cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.223+0000 7ff301400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff2fc0cf110 0x7ff2fc0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.223+0000 7ff301400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2fc0b1cb0 con 0x7ff2fc158900 2024-01-15T18:05:28.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.223+0000 7ff301400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2fc0cf580 con 0x7ff2fc15f550 2024-01-15T18:05:28.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.223+0000 7ff301400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2fc15fa00 con 0x7ff2fc0cf110 2024-01-15T18:05:28.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.223+0000 7ff2f9e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff2fc15f550 0x7ff2fc0cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.223+0000 7ff2fa800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff2fc158900 0x7ff2fc15bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34816/0 (socket says 172.21.15.27:34816) 2024-01-15T18:05:28.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.223+0000 7ff2fa800640 1 -- 172.21.15.27:0/805090229 learned_addr learned my addr 172.21.15.27:0/805090229 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:28.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.223+0000 7ff2fb200640 1 --2- 172.21.15.27:0/805090229 >> v2:172.21.15.93:3300/0 conn(0x7ff2fc0cf110 0x7ff2fc0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.223+0000 7ff2f9e00640 1 -- 172.21.15.27:0/805090229 >> v2:172.21.15.93:3300/0 conn(0x7ff2fc0cf110 msgr2=0x7ff2fc0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:28.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.223+0000 7ff2f9e00640 1 --2- 172.21.15.27:0/805090229 >> v2:172.21.15.93:3300/0 conn(0x7ff2fc0cf110 0x7ff2fc0701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.223+0000 7ff2f9e00640 1 -- 172.21.15.27:0/805090229 >> v1:172.21.15.27:6789/0 conn(0x7ff2fc158900 legacy=0x7ff2fc15bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.223+0000 7ff2f9e00640 1 -- 172.21.15.27:0/805090229 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2fc0b1f00 con 0x7ff2fc15f550 2024-01-15T18:05:28.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.223+0000 7ff2f9400640 1 -- 172.21.15.27:0/805090229 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 612262283 0 0) 0x7ff2fc0b1cb0 con 0x7ff2fc158900 2024-01-15T18:05:28.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.223+0000 7ff2fb200640 1 --2- 172.21.15.27:0/805090229 >> v2:172.21.15.93:3300/0 conn(0x7ff2fc0cf110 0x7ff2fc0701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:28.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.227+0000 7ff2f9e00640 1 --2- 172.21.15.27:0/805090229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2fc15f550 0x7ff2fc0cebd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff2e40097c0 tx=0x7ff2e401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937e5b754b55f34d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:28.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.227+0000 7ff2f9400640 1 -- 172.21.15.27:0/805090229 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc070f30 con 0x7ff2fc15f550 2024-01-15T18:05:28.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.227+0000 7ff2f9400640 1 -- 172.21.15.27:0/805090229 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e40ab610 con 0x7ff2fc15f550 2024-01-15T18:05:28.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.227+0000 7ff2f9400640 1 -- 172.21.15.27:0/805090229 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e40ab9a0 con 0x7ff2fc15f550 2024-01-15T18:05:28.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.227+0000 7ff301400640 1 -- 172.21.15.27:0/805090229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2fc15f550 msgr2=0x7ff2fc0cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.227+0000 7ff301400640 1 --2- 172.21.15.27:0/805090229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2fc15f550 0x7ff2fc0cebd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff2e40097c0 tx=0x7ff2e401c410 comp rx=0 tx=0).stop 2024-01-15T18:05:28.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.227+0000 7ff301400640 1 -- 172.21.15.27:0/805090229 shutdown_connections 2024-01-15T18:05:28.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.227+0000 7ff301400640 1 --2- 172.21.15.27:0/805090229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2fc15f550 0x7ff2fc0cebd0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.227+0000 7ff301400640 1 --2- 172.21.15.27:0/805090229 >> v2:172.21.15.93:3300/0 conn(0x7ff2fc0cf110 0x7ff2fc0701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.227+0000 7ff301400640 1 -- 172.21.15.27:0/805090229 >> 172.21.15.27:0/805090229 conn(0x7ff2fc0cb700 msgr2=0x7ff2fc0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:28.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.227+0000 7ff301400640 1 -- 172.21.15.27:0/805090229 shutdown_connections 2024-01-15T18:05:28.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.227+0000 7ff301400640 1 -- 172.21.15.27:0/805090229 wait complete. 2024-01-15T18:05:28.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.227+0000 7ff301400640 1 Processor -- start 2024-01-15T18:05:28.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.227+0000 7ff301400640 1 -- start start 2024-01-15T18:05:28.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.231+0000 7ff301400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2fc0cf110 0x7ff2fc0adef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.231+0000 7ff301400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2fc158900 0x7ff2fc14e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.231+0000 7ff301400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2fc15f550 0x7ff2fc151240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.231+0000 7ff301400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc14ed70 con 0x7ff2fc158900 2024-01-15T18:05:28.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.231+0000 7ff301400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc1558f0 con 0x7ff2fc0cf110 2024-01-15T18:05:28.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.231+0000 7ff301400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2fc15d4f0 con 0x7ff2fc15f550 2024-01-15T18:05:28.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.231+0000 7ff2f9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2fc158900 0x7ff2fc14e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.231+0000 7ff2f9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2fc158900 0x7ff2fc14e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40658/0 (socket says 172.21.15.27:40658) 2024-01-15T18:05:28.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.231+0000 7ff2f9e00640 1 -- 172.21.15.27:0/2317500333 learned_addr learned my addr 172.21.15.27:0/2317500333 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:28.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.231+0000 7ff2fb200640 1 --2- 172.21.15.27:0/2317500333 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2fc15f550 0x7ff2fc151240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.231+0000 7ff2fa800640 1 --2- 172.21.15.27:0/2317500333 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2fc0cf110 0x7ff2fc0adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.231+0000 7ff2f9e00640 1 -- 172.21.15.27:0/2317500333 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2fc15f550 msgr2=0x7ff2fc151240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.231+0000 7ff2f9e00640 1 --2- 172.21.15.27:0/2317500333 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2fc15f550 0x7ff2fc151240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.231+0000 7ff2f9e00640 1 -- 172.21.15.27:0/2317500333 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2fc0cf110 msgr2=0x7ff2fc0adef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.231+0000 7ff2f9e00640 1 --2- 172.21.15.27:0/2317500333 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2fc0cf110 0x7ff2fc0adef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.231+0000 7ff2f9e00640 1 -- 172.21.15.27:0/2317500333 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e4009420 con 0x7ff2fc158900 2024-01-15T18:05:28.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.231+0000 7ff2fb200640 1 --2- 172.21.15.27:0/2317500333 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2fc15f550 0x7ff2fc151240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:28.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.231+0000 7ff2fa800640 1 --2- 172.21.15.27:0/2317500333 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2fc0cf110 0x7ff2fc0adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:28.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.231+0000 7ff2f9e00640 1 --2- 172.21.15.27:0/2317500333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2fc158900 0x7ff2fc14e830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff2e40097c0 tx=0x7ff2e401ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:28.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.231+0000 7ff2f2a00640 1 -- 172.21.15.27:0/2317500333 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e4004030 con 0x7ff2fc158900 2024-01-15T18:05:28.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.231+0000 7ff2f2a00640 1 -- 172.21.15.27:0/2317500333 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e40b9070 con 0x7ff2fc158900 2024-01-15T18:05:28.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.231+0000 7ff301400640 1 -- 172.21.15.27:0/2317500333 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2fc157d40 con 0x7ff2fc158900 2024-01-15T18:05:28.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.231+0000 7ff2f2a00640 1 -- 172.21.15.27:0/2317500333 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e4004390 con 0x7ff2fc158900 2024-01-15T18:05:28.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.231+0000 7ff301400640 1 -- 172.21.15.27:0/2317500333 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2fc15e290 con 0x7ff2fc158900 2024-01-15T18:05:28.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.231+0000 7ff301400640 1 -- 172.21.15.27:0/2317500333 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2fc073bd0 con 0x7ff2fc158900 2024-01-15T18:05:28.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.235+0000 7ff2f2a00640 1 -- 172.21.15.27:0/2317500333 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7ff2e4003620 con 0x7ff2fc158900 2024-01-15T18:05:28.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.235+0000 7ff2f2a00640 1 --2- 172.21.15.27:0/2317500333 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff2d40668f0 0x7ff2d4068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.235+0000 7ff2f2a00640 1 -- 172.21.15.27:0/2317500333 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2e40bb070 con 0x7ff2fc158900 2024-01-15T18:05:28.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.235+0000 7ff2fa800640 1 --2- 172.21.15.27:0/2317500333 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff2d40668f0 0x7ff2d4068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.235+0000 7ff2fa800640 1 --2- 172.21.15.27:0/2317500333 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff2d40668f0 0x7ff2d4068db0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec000f70 tx=0x7ff2ec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:28.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.239+0000 7ff2f2a00640 1 -- 172.21.15.27:0/2317500333 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff2e40ec260 con 0x7ff2fc158900 2024-01-15T18:05:28.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.399+0000 7ff301400640 1 -- 172.21.15.27:0/2317500333 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff2fc15e480 con 0x7ff2fc158900 2024-01-15T18:05:28.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.399+0000 7ff2f2a00640 1 -- 172.21.15.27:0/2317500333 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff2e40044f0 con 0x7ff2fc158900 2024-01-15T18:05:28.406 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:05:28.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.403+0000 7ff301400640 1 -- 172.21.15.27:0/2317500333 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff2d40668f0 msgr2=0x7ff2d4068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.403+0000 7ff301400640 1 --2- 172.21.15.27:0/2317500333 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff2d40668f0 0x7ff2d4068db0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec000f70 tx=0x7ff2ec009040 comp rx=0 tx=0).stop 2024-01-15T18:05:28.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.403+0000 7ff301400640 1 -- 172.21.15.27:0/2317500333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2fc158900 msgr2=0x7ff2fc14e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.403+0000 7ff301400640 1 --2- 172.21.15.27:0/2317500333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2fc158900 0x7ff2fc14e830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff2e40097c0 tx=0x7ff2e401ac50 comp rx=0 tx=0).stop 2024-01-15T18:05:28.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.403+0000 7ff301400640 1 -- 172.21.15.27:0/2317500333 shutdown_connections 2024-01-15T18:05:28.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.403+0000 7ff301400640 1 --2- 172.21.15.27:0/2317500333 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff2d40668f0 0x7ff2d4068db0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.403+0000 7ff301400640 1 --2- 172.21.15.27:0/2317500333 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2fc15f550 0x7ff2fc151240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.403+0000 7ff301400640 1 --2- 172.21.15.27:0/2317500333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2fc158900 0x7ff2fc14e830 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.403+0000 7ff301400640 1 --2- 172.21.15.27:0/2317500333 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2fc0cf110 0x7ff2fc0adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.403+0000 7ff301400640 1 -- 172.21.15.27:0/2317500333 >> 172.21.15.27:0/2317500333 conn(0x7ff2fc0cb700 msgr2=0x7ff2fc0af260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.403+0000 7ff301400640 1 -- 172.21.15.27:0/2317500333 shutdown_connections 2024-01-15T18:05:28.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.403+0000 7ff301400640 1 -- 172.21.15.27:0/2317500333 wait complete. 2024-01-15T18:05:28.429 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T18:05:28.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.567+0000 7fd936a00640 1 Processor -- start 2024-01-15T18:05:28.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.567+0000 7fd936a00640 1 -- start start 2024-01-15T18:05:28.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.567+0000 7fd936a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fd93015ad20 0x7fd93015b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.567+0000 7fd936a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd9300715f0 0x7fd93015ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.567+0000 7fd936a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd93015a610 con 0x7fd93015d900 2024-01-15T18:05:28.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.567+0000 7fd936a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd93015e3c0 con 0x7fd9300715f0 2024-01-15T18:05:28.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.567+0000 7fd936a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd93015e6a0 con 0x7fd93015ad20 2024-01-15T18:05:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.571+0000 7fd934a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd9300715f0 0x7fd93015ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.571+0000 7fd92f400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd93015d900 0x7fd930070ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34822/0 (socket says 172.21.15.27:34822) 2024-01-15T18:05:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.571+0000 7fd934a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd9300715f0 0x7fd93015ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40660/0 (socket says 172.21.15.27:40660) 2024-01-15T18:05:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.571+0000 7fd92f400640 1 -- 172.21.15.27:0/1256796211 learned_addr learned my addr 172.21.15.27:0/1256796211 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.571+0000 7fd92fe00640 1 --2- 172.21.15.27:0/1256796211 >> v2:172.21.15.93:3300/0 conn(0x7fd93015ad20 0x7fd93015b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.571+0000 7fd934a00640 1 -- 172.21.15.27:0/1256796211 >> v2:172.21.15.93:3300/0 conn(0x7fd93015ad20 msgr2=0x7fd93015b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.571+0000 7fd934a00640 1 --2- 172.21.15.27:0/1256796211 >> v2:172.21.15.93:3300/0 conn(0x7fd93015ad20 0x7fd93015b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.571+0000 7fd934a00640 1 -- 172.21.15.27:0/1256796211 >> v1:172.21.15.27:6789/0 conn(0x7fd93015d900 legacy=0x7fd930070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.571+0000 7fd934a00640 1 -- 172.21.15.27:0/1256796211 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93015b6e0 con 0x7fd9300715f0 2024-01-15T18:05:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.571+0000 7fd92fe00640 1 --2- 172.21.15.27:0/1256796211 >> v2:172.21.15.93:3300/0 conn(0x7fd93015ad20 0x7fd93015b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:28.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.571+0000 7fd934a00640 1 --2- 172.21.15.27:0/1256796211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9300715f0 0x7fd93015ddb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd928009d90 tx=0x7fd92801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd882f0db97537c6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:28.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.571+0000 7fd92ea00640 1 -- 172.21.15.27:0/1256796211 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9280a59c0 con 0x7fd9300715f0 2024-01-15T18:05:28.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.571+0000 7fd92ea00640 1 -- 172.21.15.27:0/1256796211 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9280a5b20 con 0x7fd9300715f0 2024-01-15T18:05:28.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.571+0000 7fd92ea00640 1 -- 172.21.15.27:0/1256796211 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9280ad440 con 0x7fd9300715f0 2024-01-15T18:05:28.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.575+0000 7fd936a00640 1 -- 172.21.15.27:0/1256796211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9300715f0 msgr2=0x7fd93015ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.575+0000 7fd936a00640 1 --2- 172.21.15.27:0/1256796211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9300715f0 0x7fd93015ddb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd928009d90 tx=0x7fd92801c8e0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.575+0000 7fd936a00640 1 -- 172.21.15.27:0/1256796211 shutdown_connections 2024-01-15T18:05:28.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.575+0000 7fd936a00640 1 --2- 172.21.15.27:0/1256796211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9300715f0 0x7fd93015ddb0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.575+0000 7fd936a00640 1 --2- 172.21.15.27:0/1256796211 >> v2:172.21.15.93:3300/0 conn(0x7fd93015ad20 0x7fd93015b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.575+0000 7fd936a00640 1 -- 172.21.15.27:0/1256796211 >> 172.21.15.27:0/1256796211 conn(0x7fd9300cb700 msgr2=0x7fd9300cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:28.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.575+0000 7fd936a00640 1 -- 172.21.15.27:0/1256796211 shutdown_connections 2024-01-15T18:05:28.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.575+0000 7fd936a00640 1 -- 172.21.15.27:0/1256796211 wait complete. 2024-01-15T18:05:28.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.575+0000 7fd936a00640 1 Processor -- start 2024-01-15T18:05:28.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.575+0000 7fd936a00640 1 -- start start 2024-01-15T18:05:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.575+0000 7fd936a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd9300715f0 0x7fd93014cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.575+0000 7fd936a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd93015ad20 0x7fd930155010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.575+0000 7fd936a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd93015d900 0x7fd9300b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.575+0000 7fd936a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9300b4850 con 0x7fd93015d900 2024-01-15T18:05:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.575+0000 7fd936a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93014d130 con 0x7fd93015ad20 2024-01-15T18:05:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.575+0000 7fd936a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93014d5c0 con 0x7fd9300715f0 2024-01-15T18:05:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.575+0000 7fd934a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd93015d900 0x7fd9300b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.575+0000 7fd934a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd93015d900 0x7fd9300b3dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40666/0 (socket says 172.21.15.27:40666) 2024-01-15T18:05:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.575+0000 7fd934a00640 1 -- 172.21.15.27:0/3284891029 learned_addr learned my addr 172.21.15.27:0/3284891029 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.575+0000 7fd92f400640 1 --2- 172.21.15.27:0/3284891029 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd93015ad20 0x7fd930155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.575+0000 7fd92fe00640 1 --2- 172.21.15.27:0/3284891029 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd9300715f0 0x7fd93014cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.579+0000 7fd934a00640 1 -- 172.21.15.27:0/3284891029 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd9300715f0 msgr2=0x7fd93014cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.579+0000 7fd934a00640 1 --2- 172.21.15.27:0/3284891029 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd9300715f0 0x7fd93014cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.579+0000 7fd934a00640 1 -- 172.21.15.27:0/3284891029 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd93015ad20 msgr2=0x7fd930155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.579+0000 7fd934a00640 1 --2- 172.21.15.27:0/3284891029 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd93015ad20 0x7fd930155010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.579+0000 7fd934a00640 1 -- 172.21.15.27:0/3284891029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd928009a70 con 0x7fd93015d900 2024-01-15T18:05:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.579+0000 7fd934a00640 1 --2- 172.21.15.27:0/3284891029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd93015d900 0x7fd9300b3dd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd9280b0040 tx=0x7fd928019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.579+0000 7fd927e00640 1 -- 172.21.15.27:0/3284891029 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9280b9650 con 0x7fd93015d900 2024-01-15T18:05:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.579+0000 7fd927e00640 1 -- 172.21.15.27:0/3284891029 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9280b97b0 con 0x7fd93015d900 2024-01-15T18:05:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.579+0000 7fd927e00640 1 -- 172.21.15.27:0/3284891029 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9280b9b60 con 0x7fd93015d900 2024-01-15T18:05:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.579+0000 7fd936a00640 1 -- 172.21.15.27:0/3284891029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd93014d8a0 con 0x7fd93015d900 2024-01-15T18:05:28.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.579+0000 7fd936a00640 1 -- 172.21.15.27:0/3284891029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd93014de60 con 0x7fd93015d900 2024-01-15T18:05:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.579+0000 7fd936a00640 1 -- 172.21.15.27:0/3284891029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd93006e000 con 0x7fd93015d900 2024-01-15T18:05:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.579+0000 7fd927e00640 1 -- 172.21.15.27:0/3284891029 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fd9280b9cc0 con 0x7fd93015d900 2024-01-15T18:05:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.579+0000 7fd927e00640 1 --2- 172.21.15.27:0/3284891029 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd9080668f0 0x7fd908068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.583+0000 7fd927e00640 1 -- 172.21.15.27:0/3284891029 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd92811fed0 con 0x7fd93015d900 2024-01-15T18:05:28.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.583+0000 7fd92fe00640 1 --2- 172.21.15.27:0/3284891029 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd9080668f0 0x7fd908068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.583+0000 7fd92fe00640 1 --2- 172.21.15.27:0/3284891029 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd9080668f0 0x7fd908068db0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd930157d30 tx=0x7fd920008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:28.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.583+0000 7fd927e00640 1 -- 172.21.15.27:0/3284891029 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd9280ed0a0 con 0x7fd93015d900 2024-01-15T18:05:28.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.743+0000 7fd936a00640 1 -- 172.21.15.27:0/3284891029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd93014e050 con 0x7fd93015d900 2024-01-15T18:05:28.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.743+0000 7fd927e00640 1 -- 172.21.15.27:0/3284891029 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd928009450 con 0x7fd93015d900 2024-01-15T18:05:28.747 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-15T18:05:28.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.743+0000 7fd936a00640 1 -- 172.21.15.27:0/3284891029 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd9080668f0 msgr2=0x7fd908068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.743+0000 7fd936a00640 1 --2- 172.21.15.27:0/3284891029 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd9080668f0 0x7fd908068db0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd930157d30 tx=0x7fd920008040 comp rx=0 tx=0).stop 2024-01-15T18:05:28.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.743+0000 7fd936a00640 1 -- 172.21.15.27:0/3284891029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd93015d900 msgr2=0x7fd9300b3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.743+0000 7fd936a00640 1 --2- 172.21.15.27:0/3284891029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd93015d900 0x7fd9300b3dd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd9280b0040 tx=0x7fd928019680 comp rx=0 tx=0).stop 2024-01-15T18:05:28.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.743+0000 7fd936a00640 1 -- 172.21.15.27:0/3284891029 shutdown_connections 2024-01-15T18:05:28.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.743+0000 7fd936a00640 1 --2- 172.21.15.27:0/3284891029 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd9080668f0 0x7fd908068db0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.743+0000 7fd936a00640 1 --2- 172.21.15.27:0/3284891029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd93015d900 0x7fd9300b3dd0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.743+0000 7fd936a00640 1 --2- 172.21.15.27:0/3284891029 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd93015ad20 0x7fd930155010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.743+0000 7fd936a00640 1 --2- 172.21.15.27:0/3284891029 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd9300715f0 0x7fd93014cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.743+0000 7fd936a00640 1 -- 172.21.15.27:0/3284891029 >> 172.21.15.27:0/3284891029 conn(0x7fd9300cb700 msgr2=0x7fd9301537f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:28.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.743+0000 7fd936a00640 1 -- 172.21.15.27:0/3284891029 shutdown_connections 2024-01-15T18:05:28.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.743+0000 7fd936a00640 1 -- 172.21.15.27:0/3284891029 wait complete. 2024-01-15T18:05:28.769 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T18:05:28.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.907+0000 7fd422600640 1 Processor -- start 2024-01-15T18:05:28.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.907+0000 7fd422600640 1 -- start start 2024-01-15T18:05:28.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.911+0000 7fd422600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fd41c0cf460 0x7fd41c0cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.911+0000 7fd422600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd41c0b2590 0x7fd41c0cc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.911+0000 7fd422600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd41c0cfd80 con 0x7fd41c15b4b0 2024-01-15T18:05:28.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.911+0000 7fd422600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd41c0d19a0 con 0x7fd41c0b2590 2024-01-15T18:05:28.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.911+0000 7fd422600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd41c0d1c80 con 0x7fd41c0cf460 2024-01-15T18:05:28.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.911+0000 7fd41be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd41c0b2590 0x7fd41c0cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.911+0000 7fd41aa00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd41c15b4b0 0x7fd41c0cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34830/0 (socket says 172.21.15.27:34830) 2024-01-15T18:05:28.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.911+0000 7fd41aa00640 1 -- 172.21.15.27:0/1178183426 learned_addr learned my addr 172.21.15.27:0/1178183426 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:28.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.911+0000 7fd41b400640 1 --2- 172.21.15.27:0/1178183426 >> v2:172.21.15.93:3300/0 conn(0x7fd41c0cf460 0x7fd41c0cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.911+0000 7fd41be00640 1 -- 172.21.15.27:0/1178183426 >> v2:172.21.15.93:3300/0 conn(0x7fd41c0cf460 msgr2=0x7fd41c0cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:28.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.911+0000 7fd41be00640 1 --2- 172.21.15.27:0/1178183426 >> v2:172.21.15.93:3300/0 conn(0x7fd41c0cf460 0x7fd41c0cf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.911+0000 7fd41be00640 1 -- 172.21.15.27:0/1178183426 >> v1:172.21.15.27:6789/0 conn(0x7fd41c15b4b0 legacy=0x7fd41c0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.911+0000 7fd41be00640 1 -- 172.21.15.27:0/1178183426 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd41c0b07c0 con 0x7fd41c0b2590 2024-01-15T18:05:28.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.911+0000 7fd41b400640 1 --2- 172.21.15.27:0/1178183426 >> v2:172.21.15.93:3300/0 conn(0x7fd41c0cf460 0x7fd41c0cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:28.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.911+0000 7fd41a000640 1 -- 172.21.15.27:0/1178183426 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773173772 0 0) 0x7fd41c0cfd80 con 0x7fd41c15b4b0 2024-01-15T18:05:28.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.915+0000 7fd41be00640 1 --2- 172.21.15.27:0/1178183426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd41c0b2590 0x7fd41c0cc130 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd41c0681f0 tx=0x7fd41401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cf5fd1a3fa11f1d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:28.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.915+0000 7fd41a000640 1 -- 172.21.15.27:0/1178183426 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4140a59d0 con 0x7fd41c0b2590 2024-01-15T18:05:28.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.915+0000 7fd41a000640 1 -- 172.21.15.27:0/1178183426 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4140a5b30 con 0x7fd41c0b2590 2024-01-15T18:05:28.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.915+0000 7fd41a000640 1 -- 172.21.15.27:0/1178183426 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4140ad440 con 0x7fd41c0b2590 2024-01-15T18:05:28.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.915+0000 7fd422600640 1 -- 172.21.15.27:0/1178183426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd41c0b2590 msgr2=0x7fd41c0cc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.915+0000 7fd422600640 1 --2- 172.21.15.27:0/1178183426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd41c0b2590 0x7fd41c0cc130 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd41c0681f0 tx=0x7fd41401c910 comp rx=0 tx=0).stop 2024-01-15T18:05:28.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.915+0000 7fd422600640 1 -- 172.21.15.27:0/1178183426 shutdown_connections 2024-01-15T18:05:28.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.915+0000 7fd422600640 1 --2- 172.21.15.27:0/1178183426 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd41c0b2590 0x7fd41c0cc130 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.915+0000 7fd422600640 1 --2- 172.21.15.27:0/1178183426 >> v2:172.21.15.93:3300/0 conn(0x7fd41c0cf460 0x7fd41c0cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.915+0000 7fd422600640 1 -- 172.21.15.27:0/1178183426 >> 172.21.15.27:0/1178183426 conn(0x7fd41c0d0e80 msgr2=0x7fd41c0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:28.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.915+0000 7fd422600640 1 -- 172.21.15.27:0/1178183426 shutdown_connections 2024-01-15T18:05:28.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.915+0000 7fd422600640 1 -- 172.21.15.27:0/1178183426 wait complete. 2024-01-15T18:05:28.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.915+0000 7fd422600640 1 Processor -- start 2024-01-15T18:05:28.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.915+0000 7fd422600640 1 -- start start 2024-01-15T18:05:28.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.919+0000 7fd422600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd41c0b2590 0x7fd41c0c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.919+0000 7fd422600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd41c0cf460 0x7fd41c14c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.919+0000 7fd422600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd41c15b4b0 0x7fd41c15bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.919+0000 7fd422600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd41c0c9320 con 0x7fd41c15b4b0 2024-01-15T18:05:28.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.919+0000 7fd422600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd41c0c8490 con 0x7fd41c0cf460 2024-01-15T18:05:28.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.919+0000 7fd422600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd41c14cab0 con 0x7fd41c0b2590 2024-01-15T18:05:28.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.919+0000 7fd41be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd41c15b4b0 0x7fd41c15bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.919+0000 7fd41be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd41c15b4b0 0x7fd41c15bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40678/0 (socket says 172.21.15.27:40678) 2024-01-15T18:05:28.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.919+0000 7fd41be00640 1 -- 172.21.15.27:0/3069938434 learned_addr learned my addr 172.21.15.27:0/3069938434 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:28.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.919+0000 7fd41aa00640 1 --2- 172.21.15.27:0/3069938434 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd41c0cf460 0x7fd41c14c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.919+0000 7fd41b400640 1 --2- 172.21.15.27:0/3069938434 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd41c0b2590 0x7fd41c0c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.919+0000 7fd41be00640 1 -- 172.21.15.27:0/3069938434 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd41c0b2590 msgr2=0x7fd41c0c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.919+0000 7fd41be00640 1 --2- 172.21.15.27:0/3069938434 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd41c0b2590 0x7fd41c0c7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.919+0000 7fd41be00640 1 -- 172.21.15.27:0/3069938434 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd41c0cf460 msgr2=0x7fd41c14c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.919+0000 7fd41be00640 1 --2- 172.21.15.27:0/3069938434 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd41c0cf460 0x7fd41c14c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.919+0000 7fd41be00640 1 -- 172.21.15.27:0/3069938434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd414009a70 con 0x7fd41c15b4b0 2024-01-15T18:05:28.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.919+0000 7fd41be00640 1 --2- 172.21.15.27:0/3069938434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd41c15b4b0 0x7fd41c15bfe0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd414019520 tx=0x7fd4140196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:28.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.919+0000 7fd413400640 1 -- 172.21.15.27:0/3069938434 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4140ade20 con 0x7fd41c15b4b0 2024-01-15T18:05:28.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.919+0000 7fd413400640 1 -- 172.21.15.27:0/3069938434 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd414009450 con 0x7fd41c15b4b0 2024-01-15T18:05:28.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.919+0000 7fd422600640 1 -- 172.21.15.27:0/3069938434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd41c0c8710 con 0x7fd41c15b4b0 2024-01-15T18:05:28.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.919+0000 7fd413400640 1 -- 172.21.15.27:0/3069938434 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4140b9690 con 0x7fd41c15b4b0 2024-01-15T18:05:28.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.919+0000 7fd422600640 1 -- 172.21.15.27:0/3069938434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd41c0c8cd0 con 0x7fd41c15b4b0 2024-01-15T18:05:28.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.919+0000 7fd422600640 1 -- 172.21.15.27:0/3069938434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd41c0c94d0 con 0x7fd41c15b4b0 2024-01-15T18:05:28.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.923+0000 7fd413400640 1 -- 172.21.15.27:0/3069938434 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fd4140b97f0 con 0x7fd41c15b4b0 2024-01-15T18:05:28.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.923+0000 7fd413400640 1 --2- 172.21.15.27:0/3069938434 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd3f40668f0 0x7fd3f4068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.923+0000 7fd413400640 1 -- 172.21.15.27:0/3069938434 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd41411fa30 con 0x7fd41c15b4b0 2024-01-15T18:05:28.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.923+0000 7fd41b400640 1 --2- 172.21.15.27:0/3069938434 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd3f40668f0 0x7fd3f4068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.923+0000 7fd41b400640 1 --2- 172.21.15.27:0/3069938434 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd3f40668f0 0x7fd3f4068db0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd40c009950 tx=0x7fd40c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:28.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:28.927+0000 7fd413400640 1 -- 172.21.15.27:0/3069938434 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd41c0c94d0 con 0x7fd41c15b4b0 2024-01-15T18:05:29.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.083+0000 7fd422600640 1 -- 172.21.15.27:0/3069938434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd41c0c94d0 con 0x7fd41c15b4b0 2024-01-15T18:05:29.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.083+0000 7fd413400640 1 -- 172.21.15.27:0/3069938434 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd41c0c94d0 con 0x7fd41c15b4b0 2024-01-15T18:05:29.089 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-15T18:05:29.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.087+0000 7fd422600640 1 -- 172.21.15.27:0/3069938434 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd3f40668f0 msgr2=0x7fd3f4068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:29.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.087+0000 7fd422600640 1 --2- 172.21.15.27:0/3069938434 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd3f40668f0 0x7fd3f4068db0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd40c009950 tx=0x7fd40c008040 comp rx=0 tx=0).stop 2024-01-15T18:05:29.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.087+0000 7fd422600640 1 -- 172.21.15.27:0/3069938434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd41c15b4b0 msgr2=0x7fd41c15bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:29.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.087+0000 7fd422600640 1 --2- 172.21.15.27:0/3069938434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd41c15b4b0 0x7fd41c15bfe0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd414019520 tx=0x7fd4140196b0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.087+0000 7fd422600640 1 -- 172.21.15.27:0/3069938434 shutdown_connections 2024-01-15T18:05:29.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.087+0000 7fd422600640 1 --2- 172.21.15.27:0/3069938434 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd3f40668f0 0x7fd3f4068db0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.087+0000 7fd422600640 1 --2- 172.21.15.27:0/3069938434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd41c15b4b0 0x7fd41c15bfe0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.087+0000 7fd422600640 1 --2- 172.21.15.27:0/3069938434 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd41c0cf460 0x7fd41c14c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.087+0000 7fd422600640 1 --2- 172.21.15.27:0/3069938434 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd41c0b2590 0x7fd41c0c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.087+0000 7fd422600640 1 -- 172.21.15.27:0/3069938434 >> 172.21.15.27:0/3069938434 conn(0x7fd41c0d0e80 msgr2=0x7fd41c0ade50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:29.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.087+0000 7fd422600640 1 -- 172.21.15.27:0/3069938434 shutdown_connections 2024-01-15T18:05:29.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.087+0000 7fd422600640 1 -- 172.21.15.27:0/3069938434 wait complete. 2024-01-15T18:05:29.186 INFO:tasks.cephfs_test_runner:Starting test: test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) Test that when one clone is in progress, one progress bar is printed 2024-01-15T18:05:29.186 INFO:tasks.cephfs_test_runner:test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T18:05:29.186 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:29.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.319+0000 7f8d47c00640 1 Processor -- start 2024-01-15T18:05:29.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.319+0000 7f8d47c00640 1 -- start start 2024-01-15T18:05:29.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.323+0000 7f8d47c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f8d4015e7d0 0x7f8d400cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:29.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.323+0000 7f8d47c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8d400b20d0 0x7f8d400cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:29.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.323+0000 7f8d47c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d400b38b0 con 0x7f8d400cedc0 2024-01-15T18:05:29.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.323+0000 7f8d47c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d400cd550 con 0x7f8d4015e7d0 2024-01-15T18:05:29.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.323+0000 7f8d47c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d400cd270 con 0x7f8d400b20d0 2024-01-15T18:05:29.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.323+0000 7f8d45c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8d400b20d0 0x7f8d400cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:29.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.323+0000 7f8d45200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f8d4015e7d0 0x7f8d400cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:29.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.323+0000 7f8d45c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8d400b20d0 0x7f8d400cf7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:54072/0 (socket says 172.21.15.27:54072) 2024-01-15T18:05:29.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.323+0000 7f8d45200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f8d4015e7d0 0x7f8d400cf2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:53994/0 (socket says 172.21.15.27:53994) 2024-01-15T18:05:29.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.323+0000 7f8d45c00640 1 -- 172.21.15.27:0/1112008779 learned_addr learned my addr 172.21.15.27:0/1112008779 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:29.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.323+0000 7f8d45200640 1 -- 172.21.15.27:0/1112008779 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8d400b20d0 msgr2=0x7f8d400cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:29.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.323+0000 7f8d45200640 1 --2- 172.21.15.27:0/1112008779 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8d400b20d0 0x7f8d400cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.323+0000 7f8d45200640 1 -- 172.21.15.27:0/1112008779 >> v1:172.21.15.93:6789/0 conn(0x7f8d400cedc0 legacy=0x7f8d400cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:29.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.323+0000 7f8d3f400640 1 -- 172.21.15.27:0/1112008779 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999851338 0 0) 0x7f8d400b38b0 con 0x7f8d400cedc0 2024-01-15T18:05:29.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.323+0000 7f8d45200640 1 -- 172.21.15.27:0/1112008779 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d400b35d0 con 0x7f8d4015e7d0 2024-01-15T18:05:29.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.323+0000 7f8d45c00640 1 --2- 172.21.15.27:0/1112008779 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8d400b20d0 0x7f8d400cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:29.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.327+0000 7f8d45200640 1 --2- 172.21.15.27:0/1112008779 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8d4015e7d0 0x7f8d400cf2a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8d300097c0 tx=0x7f8d3001c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea8020aa59c47376 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:29.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.327+0000 7f8d3f400640 1 -- 172.21.15.27:0/1112008779 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d300ab600 con 0x7f8d4015e7d0 2024-01-15T18:05:29.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.327+0000 7f8d3f400640 1 -- 172.21.15.27:0/1112008779 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d300a7050 con 0x7f8d4015e7d0 2024-01-15T18:05:29.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.327+0000 7f8d3f400640 1 -- 172.21.15.27:0/1112008779 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d300ab980 con 0x7f8d4015e7d0 2024-01-15T18:05:29.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.327+0000 7f8d47c00640 1 -- 172.21.15.27:0/1112008779 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8d4015e7d0 msgr2=0x7f8d400cf2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:29.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.327+0000 7f8d47c00640 1 --2- 172.21.15.27:0/1112008779 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8d4015e7d0 0x7f8d400cf2a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8d300097c0 tx=0x7f8d3001c480 comp rx=0 tx=0).stop 2024-01-15T18:05:29.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.327+0000 7f8d47c00640 1 -- 172.21.15.27:0/1112008779 shutdown_connections 2024-01-15T18:05:29.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.327+0000 7f8d47c00640 1 --2- 172.21.15.27:0/1112008779 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8d400b20d0 0x7f8d400cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.327+0000 7f8d47c00640 1 --2- 172.21.15.27:0/1112008779 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8d4015e7d0 0x7f8d400cf2a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.327+0000 7f8d47c00640 1 -- 172.21.15.27:0/1112008779 >> 172.21.15.27:0/1112008779 conn(0x7f8d400b2a10 msgr2=0x7f8d400b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:29.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.327+0000 7f8d47c00640 1 -- 172.21.15.27:0/1112008779 shutdown_connections 2024-01-15T18:05:29.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.327+0000 7f8d47c00640 1 -- 172.21.15.27:0/1112008779 wait complete. 2024-01-15T18:05:29.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.331+0000 7f8d47c00640 1 Processor -- start 2024-01-15T18:05:29.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.331+0000 7f8d47c00640 1 -- start start 2024-01-15T18:05:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.331+0000 7f8d47c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d400b20d0 0x7f8d4014b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.331+0000 7f8d47c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8d400cedc0 0x7f8d4015ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.331+0000 7f8d47c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8d4015e7d0 0x7f8d4015bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.331+0000 7f8d47c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d40163d00 con 0x7f8d400b20d0 2024-01-15T18:05:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.331+0000 7f8d47c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4014ba50 con 0x7f8d400cedc0 2024-01-15T18:05:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.331+0000 7f8d47c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4006d5f0 con 0x7f8d4015e7d0 2024-01-15T18:05:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.331+0000 7f8d45200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d400b20d0 0x7f8d4014b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.331+0000 7f8d45200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d400b20d0 0x7f8d4014b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40690/0 (socket says 172.21.15.27:40690) 2024-01-15T18:05:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.331+0000 7f8d45200640 1 -- 172.21.15.27:0/2638806120 learned_addr learned my addr 172.21.15.27:0/2638806120 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.331+0000 7f8d3fe00640 1 --2- 172.21.15.27:0/2638806120 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8d400cedc0 0x7f8d4015ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.331+0000 7f8d45c00640 1 --2- 172.21.15.27:0/2638806120 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8d4015e7d0 0x7f8d4015bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.331+0000 7f8d45200640 1 -- 172.21.15.27:0/2638806120 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8d4015e7d0 msgr2=0x7f8d4015bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.331+0000 7f8d45200640 1 --2- 172.21.15.27:0/2638806120 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8d4015e7d0 0x7f8d4015bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.331+0000 7f8d45200640 1 -- 172.21.15.27:0/2638806120 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8d400cedc0 msgr2=0x7f8d4015ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:29.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.331+0000 7f8d45200640 1 --2- 172.21.15.27:0/2638806120 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8d400cedc0 0x7f8d4015ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.331+0000 7f8d45200640 1 -- 172.21.15.27:0/2638806120 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d38009b30 con 0x7f8d400b20d0 2024-01-15T18:05:29.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.331+0000 7f8d45200640 1 --2- 172.21.15.27:0/2638806120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d400b20d0 0x7f8d4014b510 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8d300097c0 tx=0x7f8d30019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:29.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.331+0000 7f8d3cc00640 1 -- 172.21.15.27:0/2638806120 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d300b4430 con 0x7f8d400b20d0 2024-01-15T18:05:29.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.331+0000 7f8d3cc00640 1 -- 172.21.15.27:0/2638806120 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d300b4590 con 0x7f8d400b20d0 2024-01-15T18:05:29.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.331+0000 7f8d3cc00640 1 -- 172.21.15.27:0/2638806120 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d300b4940 con 0x7f8d400b20d0 2024-01-15T18:05:29.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.331+0000 7f8d47c00640 1 -- 172.21.15.27:0/2638806120 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d30009420 con 0x7f8d400b20d0 2024-01-15T18:05:29.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.331+0000 7f8d47c00640 1 -- 172.21.15.27:0/2638806120 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d4006d030 con 0x7f8d400b20d0 2024-01-15T18:05:29.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.335+0000 7f8d47c00640 1 -- 172.21.15.27:0/2638806120 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d400d03f0 con 0x7f8d400b20d0 2024-01-15T18:05:29.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.335+0000 7f8d3cc00640 1 -- 172.21.15.27:0/2638806120 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f8d300b2040 con 0x7f8d400b20d0 2024-01-15T18:05:29.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.335+0000 7f8d3cc00640 1 --2- 172.21.15.27:0/2638806120 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8d18066920 0x7f8d18068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:29.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.335+0000 7f8d3cc00640 1 -- 172.21.15.27:0/2638806120 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d3011e230 con 0x7f8d400b20d0 2024-01-15T18:05:29.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.335+0000 7f8d3fe00640 1 --2- 172.21.15.27:0/2638806120 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8d18066920 0x7f8d18068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:29.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.335+0000 7f8d3fe00640 1 --2- 172.21.15.27:0/2638806120 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8d18066920 0x7f8d18068de0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8d4006bc60 tx=0x7f8d28009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:29.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.339+0000 7f8d3cc00640 1 -- 172.21.15.27:0/2638806120 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8d300eb510 con 0x7f8d400b20d0 2024-01-15T18:05:29.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.495+0000 7f8d47c00640 1 -- 172.21.15.27:0/2638806120 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d4004e850 con 0x7f8d400b20d0 2024-01-15T18:05:29.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.495+0000 7f8d3cc00640 1 -- 172.21.15.27:0/2638806120 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16409 (secure 0 0 0) 0x7f8d300eeb30 con 0x7f8d400b20d0 2024-01-15T18:05:29.502 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:29.502 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:04:59.837052+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:04:56.220608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T18:04:57.210421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.499+0000 7f8d47c00640 1 -- 172.21.15.27:0/2638806120 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8d18066920 msgr2=0x7f8d18068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.499+0000 7f8d47c00640 1 --2- 172.21.15.27:0/2638806120 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8d18066920 0x7f8d18068de0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8d4006bc60 tx=0x7f8d28009040 comp rx=0 tx=0).stop 2024-01-15T18:05:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.499+0000 7f8d47c00640 1 -- 172.21.15.27:0/2638806120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d400b20d0 msgr2=0x7f8d4014b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.499+0000 7f8d47c00640 1 --2- 172.21.15.27:0/2638806120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d400b20d0 0x7f8d4014b510 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8d300097c0 tx=0x7f8d30019220 comp rx=0 tx=0).stop 2024-01-15T18:05:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.499+0000 7f8d47c00640 1 -- 172.21.15.27:0/2638806120 shutdown_connections 2024-01-15T18:05:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.499+0000 7f8d47c00640 1 --2- 172.21.15.27:0/2638806120 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8d18066920 0x7f8d18068de0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.499+0000 7f8d47c00640 1 --2- 172.21.15.27:0/2638806120 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8d4015e7d0 0x7f8d4015bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.499+0000 7f8d47c00640 1 --2- 172.21.15.27:0/2638806120 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8d400cedc0 0x7f8d4015ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.499+0000 7f8d47c00640 1 --2- 172.21.15.27:0/2638806120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d400b20d0 0x7f8d4014b510 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.499+0000 7f8d47c00640 1 -- 172.21.15.27:0/2638806120 >> 172.21.15.27:0/2638806120 conn(0x7f8d400b2a10 msgr2=0x7f8d4015daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.499+0000 7f8d47c00640 1 -- 172.21.15.27:0/2638806120 shutdown_connections 2024-01-15T18:05:29.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.499+0000 7f8d47c00640 1 -- 172.21.15.27:0/2638806120 wait complete. 2024-01-15T18:05:29.524 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T18:04:56.220608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T18:04:57.210421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:05:29.524 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:05:29.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.663+0000 7f3825400640 1 Processor -- start 2024-01-15T18:05:29.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.663+0000 7f3825400640 1 -- start start 2024-01-15T18:05:29.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.663+0000 7f3825400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38200d1a40 0x7f38200cba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:29.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.663+0000 7f3825400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f38200b0650 0x7f38200b0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:29.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.663+0000 7f3825400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38200cbfa0 con 0x7f38200cf450 2024-01-15T18:05:29.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.663+0000 7f3825400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38200cefa0 con 0x7f38200b0650 2024-01-15T18:05:29.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.663+0000 7f3825400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38200ce240 con 0x7f38200d1a40 2024-01-15T18:05:29.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.663+0000 7f381de00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38200d1a40 0x7f38200cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:29.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.663+0000 7f381e800640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7f38200cf450 0x7f38200cf830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:57710/0 (socket says 172.21.15.27:57710) 2024-01-15T18:05:29.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.663+0000 7f381e800640 1 -- 172.21.15.27:0/1246032578 learned_addr learned my addr 172.21.15.27:0/1246032578 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:29.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.663+0000 7f381f200640 1 --2- 172.21.15.27:0/1246032578 >> v2:172.21.15.93:3300/0 conn(0x7f38200b0650 0x7f38200b0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:29.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.663+0000 7f381d400640 1 -- 172.21.15.27:0/1246032578 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4200818233 0 0) 0x7f38200cbfa0 con 0x7f38200cf450 2024-01-15T18:05:29.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.663+0000 7f381d400640 1 -- 172.21.15.27:0/1246032578 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3800003530 con 0x7f38200cf450 2024-01-15T18:05:29.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.663+0000 7f381de00640 1 -- 172.21.15.27:0/1246032578 >> v2:172.21.15.93:3300/0 conn(0x7f38200b0650 msgr2=0x7f38200b0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:29.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.663+0000 7f381de00640 1 --2- 172.21.15.27:0/1246032578 >> v2:172.21.15.93:3300/0 conn(0x7f38200b0650 0x7f38200b0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.663+0000 7f381de00640 1 -- 172.21.15.27:0/1246032578 >> v1:172.21.15.93:6789/0 conn(0x7f38200cf450 legacy=0x7f38200cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:29.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.663+0000 7f381de00640 1 -- 172.21.15.27:0/1246032578 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38200ccd70 con 0x7f38200d1a40 2024-01-15T18:05:29.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.663+0000 7f381f200640 1 --2- 172.21.15.27:0/1246032578 >> v2:172.21.15.93:3300/0 conn(0x7f38200b0650 0x7f38200b0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:29.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.667+0000 7f381de00640 1 --2- 172.21.15.27:0/1246032578 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38200d1a40 0x7f38200cba60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f38080097c0 tx=0x7f380801c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b09dcad2cc695c59 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:29.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.667+0000 7f381d400640 1 -- 172.21.15.27:0/1246032578 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38080ab600 con 0x7f38200d1a40 2024-01-15T18:05:29.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.667+0000 7f381d400640 1 -- 172.21.15.27:0/1246032578 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38080a7050 con 0x7f38200d1a40 2024-01-15T18:05:29.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.667+0000 7f381d400640 1 -- 172.21.15.27:0/1246032578 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38080ab980 con 0x7f38200d1a40 2024-01-15T18:05:29.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.667+0000 7f3825400640 1 -- 172.21.15.27:0/1246032578 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38200d1a40 msgr2=0x7f38200cba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:29.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.667+0000 7f3825400640 1 --2- 172.21.15.27:0/1246032578 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38200d1a40 0x7f38200cba60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f38080097c0 tx=0x7f380801c480 comp rx=0 tx=0).stop 2024-01-15T18:05:29.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.671+0000 7f3825400640 1 -- 172.21.15.27:0/1246032578 shutdown_connections 2024-01-15T18:05:29.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.671+0000 7f3825400640 1 --2- 172.21.15.27:0/1246032578 >> v2:172.21.15.93:3300/0 conn(0x7f38200b0650 0x7f38200b0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.671+0000 7f3825400640 1 --2- 172.21.15.27:0/1246032578 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38200d1a40 0x7f38200cba60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.671+0000 7f3825400640 1 -- 172.21.15.27:0/1246032578 >> 172.21.15.27:0/1246032578 conn(0x7f38200d0e70 msgr2=0x7f38200d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:29.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.671+0000 7f3825400640 1 -- 172.21.15.27:0/1246032578 shutdown_connections 2024-01-15T18:05:29.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.671+0000 7f3825400640 1 -- 172.21.15.27:0/1246032578 wait complete. 2024-01-15T18:05:29.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.671+0000 7f3825400640 1 Processor -- start 2024-01-15T18:05:29.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.671+0000 7f3825400640 1 -- start start 2024-01-15T18:05:29.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.671+0000 7f3825400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38200b0650 0x7f38201655e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:29.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.671+0000 7f3825400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f38200cf450 0x7f382015c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:29.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.671+0000 7f3825400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38200d1a40 0x7f382015c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:29.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.671+0000 7f3825400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382015a2c0 con 0x7f38200d1a40 2024-01-15T18:05:29.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.671+0000 7f3825400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382015a5a0 con 0x7f38200cf450 2024-01-15T18:05:29.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.671+0000 7f3825400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f382015a880 con 0x7f38200b0650 2024-01-15T18:05:29.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.671+0000 7f381f200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38200d1a40 0x7f382015c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:29.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.671+0000 7f381de00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f38200cf450 0x7f382015c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:29.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.671+0000 7f381e800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38200b0650 0x7f38201655e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:29.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.671+0000 7f381f200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38200d1a40 0x7f382015c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40692/0 (socket says 172.21.15.27:40692) 2024-01-15T18:05:29.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.671+0000 7f381f200640 1 -- 172.21.15.27:0/67787337 learned_addr learned my addr 172.21.15.27:0/67787337 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:29.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.671+0000 7f381de00640 1 -- 172.21.15.27:0/67787337 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38200b0650 msgr2=0x7f38201655e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:29.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.671+0000 7f381de00640 1 --2- 172.21.15.27:0/67787337 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38200b0650 0x7f38201655e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.671+0000 7f381de00640 1 -- 172.21.15.27:0/67787337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38200d1a40 msgr2=0x7f382015c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:29.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.671+0000 7f381de00640 1 --2- 172.21.15.27:0/67787337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38200d1a40 0x7f382015c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.671+0000 7f381de00640 1 -- 172.21.15.27:0/67787337 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3818009a70 con 0x7f38200cf450 2024-01-15T18:05:29.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.671+0000 7f381f200640 1 --2- 172.21.15.27:0/67787337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38200d1a40 0x7f382015c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:29.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.675+0000 7f381de00640 1 --2- 172.21.15.27:0/67787337 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f38200cf450 0x7f382015c470 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f38080ab000 tx=0x7f3808019220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:29.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.675+0000 7f3816a00640 1 -- 172.21.15.27:0/67787337 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38080bd070 con 0x7f38200cf450 2024-01-15T18:05:29.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.675+0000 7f3816a00640 1 -- 172.21.15.27:0/67787337 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38080abd70 con 0x7f38200cf450 2024-01-15T18:05:29.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.675+0000 7f3816a00640 1 -- 172.21.15.27:0/67787337 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38080b56b0 con 0x7f38200cf450 2024-01-15T18:05:29.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.675+0000 7f3825400640 1 -- 172.21.15.27:0/67787337 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3808009420 con 0x7f38200cf450 2024-01-15T18:05:29.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.675+0000 7f3825400640 1 -- 172.21.15.27:0/67787337 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f382015ae40 con 0x7f38200cf450 2024-01-15T18:05:29.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.675+0000 7f3825400640 1 -- 172.21.15.27:0/67787337 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f382015b030 con 0x7f38200cf450 2024-01-15T18:05:29.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.675+0000 7f3816a00640 1 -- 172.21.15.27:0/67787337 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f38080b2040 con 0x7f38200cf450 2024-01-15T18:05:29.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.679+0000 7f3816a00640 1 --2- 172.21.15.27:0/67787337 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f37f80668f0 0x7f37f8068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:29.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.679+0000 7f3816a00640 1 -- 172.21.15.27:0/67787337 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f380811ea80 con 0x7f38200cf450 2024-01-15T18:05:29.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.679+0000 7f381e800640 1 --2- 172.21.15.27:0/67787337 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f37f80668f0 0x7f37f8068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:29.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.679+0000 7f381e800640 1 --2- 172.21.15.27:0/67787337 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f37f80668f0 0x7f37f8068db0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f38100011f0 tx=0x7f3810007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:29.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.679+0000 7f3816a00640 1 -- 172.21.15.27:0/67787337 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f38080ebe70 con 0x7f38200cf450 2024-01-15T18:05:29.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.839+0000 7f3825400640 1 -- 172.21.15.27:0/67787337 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f382004e850 con 0x7f38200cf450 2024-01-15T18:05:29.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.839+0000 7f3816a00640 1 -- 172.21.15.27:0/67787337 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f38080b00a0 con 0x7f38200cf450 2024-01-15T18:05:29.845 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:05:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.843+0000 7f3825400640 1 -- 172.21.15.27:0/67787337 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f37f80668f0 msgr2=0x7f37f8068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.843+0000 7f3825400640 1 --2- 172.21.15.27:0/67787337 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f37f80668f0 0x7f37f8068db0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f38100011f0 tx=0x7f3810007d40 comp rx=0 tx=0).stop 2024-01-15T18:05:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.843+0000 7f3825400640 1 -- 172.21.15.27:0/67787337 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f38200cf450 msgr2=0x7f382015c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.843+0000 7f3825400640 1 --2- 172.21.15.27:0/67787337 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f38200cf450 0x7f382015c470 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f38080ab000 tx=0x7f3808019220 comp rx=0 tx=0).stop 2024-01-15T18:05:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.843+0000 7f3825400640 1 -- 172.21.15.27:0/67787337 shutdown_connections 2024-01-15T18:05:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.843+0000 7f3825400640 1 --2- 172.21.15.27:0/67787337 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f37f80668f0 0x7f37f8068db0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.843+0000 7f3825400640 1 --2- 172.21.15.27:0/67787337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38200d1a40 0x7f382015c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.843+0000 7f3825400640 1 --2- 172.21.15.27:0/67787337 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f38200cf450 0x7f382015c470 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.843+0000 7f3825400640 1 --2- 172.21.15.27:0/67787337 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38200b0650 0x7f38201655e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.843+0000 7f3825400640 1 -- 172.21.15.27:0/67787337 >> 172.21.15.27:0/67787337 conn(0x7f38200d0e70 msgr2=0x7f38200af770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.843+0000 7f3825400640 1 -- 172.21.15.27:0/67787337 shutdown_connections 2024-01-15T18:05:29.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:29.843+0000 7f3825400640 1 -- 172.21.15.27:0/67787337 wait complete. 2024-01-15T18:05:29.868 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T18:05:30.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.007+0000 7f67df200640 1 Processor -- start 2024-01-15T18:05:30.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.007+0000 7f67df200640 1 -- start start 2024-01-15T18:05:30.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.011+0000 7f67df200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f67d81557e0 0x7f67d80acd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.011+0000 7f67df200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f67d80b3d10 0x7f67d815c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.011+0000 7f67df200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67d80ad410 con 0x7f67d806a4f0 2024-01-15T18:05:30.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.011+0000 7f67df200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67d80ae360 con 0x7f67d81557e0 2024-01-15T18:05:30.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.011+0000 7f67df200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67d815c570 con 0x7f67d80b3d10 2024-01-15T18:05:30.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.011+0000 7f67d7e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f67d81557e0 0x7f67d80acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.011+0000 7f67d7400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f67d80b3d10 0x7f67d815c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.011+0000 7f67d7400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f67d80b3d10 0x7f67d815c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:54104/0 (socket says 172.21.15.27:54104) 2024-01-15T18:05:30.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.011+0000 7f67d7e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f67d81557e0 0x7f67d80acd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:54026/0 (socket says 172.21.15.27:54026) 2024-01-15T18:05:30.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.011+0000 7f67d7400640 1 -- 172.21.15.27:0/4023422338 learned_addr learned my addr 172.21.15.27:0/4023422338 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:30.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.011+0000 7f67dd200640 1 --1- 172.21.15.27:0/4023422338 >> v1:172.21.15.93:6789/0 conn(0x7f67d806a4f0 0x7f67d815c730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:57716/0 (socket says 172.21.15.27:57716) 2024-01-15T18:05:30.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.011+0000 7f67d7e00640 1 -- 172.21.15.27:0/4023422338 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f67d80b3d10 msgr2=0x7f67d815c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:30.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.011+0000 7f67d7e00640 1 --2- 172.21.15.27:0/4023422338 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f67d80b3d10 0x7f67d815c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.011+0000 7f67d7e00640 1 -- 172.21.15.27:0/4023422338 >> v1:172.21.15.93:6789/0 conn(0x7f67d806a4f0 legacy=0x7f67d815c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.011+0000 7f67d7e00640 1 -- 172.21.15.27:0/4023422338 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d80b2d00 con 0x7f67d81557e0 2024-01-15T18:05:30.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.011+0000 7f67d6a00640 1 -- 172.21.15.27:0/4023422338 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912651414 0 0) 0x7f67d80ad410 con 0x7f67d806a4f0 2024-01-15T18:05:30.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.011+0000 7f67d7400640 1 --2- 172.21.15.27:0/4023422338 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f67d80b3d10 0x7f67d815c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:30.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.015+0000 7f67d7e00640 1 --2- 172.21.15.27:0/4023422338 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f67d81557e0 0x7f67d80acd50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f67c80097c0 tx=0x7f67c801c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a63b2861d2c64862 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:30.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.015+0000 7f67d6a00640 1 -- 172.21.15.27:0/4023422338 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c80ab600 con 0x7f67d81557e0 2024-01-15T18:05:30.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.015+0000 7f67d6a00640 1 -- 172.21.15.27:0/4023422338 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67c80a7050 con 0x7f67d81557e0 2024-01-15T18:05:30.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.015+0000 7f67d6a00640 1 -- 172.21.15.27:0/4023422338 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c80ab980 con 0x7f67d81557e0 2024-01-15T18:05:30.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.015+0000 7f67df200640 1 -- 172.21.15.27:0/4023422338 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f67d81557e0 msgr2=0x7f67d80acd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.015+0000 7f67df200640 1 --2- 172.21.15.27:0/4023422338 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f67d81557e0 0x7f67d80acd50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f67c80097c0 tx=0x7f67c801c480 comp rx=0 tx=0).stop 2024-01-15T18:05:30.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.015+0000 7f67df200640 1 -- 172.21.15.27:0/4023422338 shutdown_connections 2024-01-15T18:05:30.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.015+0000 7f67df200640 1 --2- 172.21.15.27:0/4023422338 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f67d80b3d10 0x7f67d815c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.015+0000 7f67df200640 1 --2- 172.21.15.27:0/4023422338 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f67d81557e0 0x7f67d80acd50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.015+0000 7f67df200640 1 -- 172.21.15.27:0/4023422338 >> 172.21.15.27:0/4023422338 conn(0x7f67d806b600 msgr2=0x7f67d806ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:30.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.015+0000 7f67df200640 1 -- 172.21.15.27:0/4023422338 shutdown_connections 2024-01-15T18:05:30.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.015+0000 7f67df200640 1 -- 172.21.15.27:0/4023422338 wait complete. 2024-01-15T18:05:30.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.015+0000 7f67df200640 1 Processor -- start 2024-01-15T18:05:30.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.015+0000 7f67df200640 1 -- start start 2024-01-15T18:05:30.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.019+0000 7f67df200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f67d806a4f0 0x7f67d814f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.019+0000 7f67df200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f67d80b3d10 0x7f67d814d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.019+0000 7f67df200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67d81557e0 0x7f67d815b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.019+0000 7f67df200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d81500c0 con 0x7f67d81557e0 2024-01-15T18:05:30.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.019+0000 7f67df200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d8159910 con 0x7f67d806a4f0 2024-01-15T18:05:30.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.019+0000 7f67df200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67d814fde0 con 0x7f67d80b3d10 2024-01-15T18:05:30.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.019+0000 7f67dd200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67d81557e0 0x7f67d815b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.019+0000 7f67dd200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67d81557e0 0x7f67d815b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40694/0 (socket says 172.21.15.27:40694) 2024-01-15T18:05:30.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.019+0000 7f67dd200640 1 -- 172.21.15.27:0/4169336763 learned_addr learned my addr 172.21.15.27:0/4169336763 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:30.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.019+0000 7f67d7e00640 1 --2- 172.21.15.27:0/4169336763 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f67d806a4f0 0x7f67d814f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.019+0000 7f67d7400640 1 --2- 172.21.15.27:0/4169336763 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f67d80b3d10 0x7f67d814d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.019+0000 7f67dd200640 1 -- 172.21.15.27:0/4169336763 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f67d80b3d10 msgr2=0x7f67d814d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.019+0000 7f67dd200640 1 --2- 172.21.15.27:0/4169336763 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f67d80b3d10 0x7f67d814d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.019+0000 7f67dd200640 1 -- 172.21.15.27:0/4169336763 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f67d806a4f0 msgr2=0x7f67d814f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.019+0000 7f67dd200640 1 --2- 172.21.15.27:0/4169336763 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f67d806a4f0 0x7f67d814f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.019+0000 7f67dd200640 1 -- 172.21.15.27:0/4169336763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c00094d0 con 0x7f67d81557e0 2024-01-15T18:05:30.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.019+0000 7f67dd200640 1 --2- 172.21.15.27:0/4169336763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67d81557e0 0x7f67d815b700 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f67d0001570 tx=0x7f67d000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:30.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.019+0000 7f67cfe00640 1 -- 172.21.15.27:0/4169336763 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d000a640 con 0x7f67d81557e0 2024-01-15T18:05:30.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.019+0000 7f67cfe00640 1 -- 172.21.15.27:0/4169336763 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d000a7c0 con 0x7f67d81557e0 2024-01-15T18:05:30.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.019+0000 7f67df200640 1 -- 172.21.15.27:0/4169336763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c8009420 con 0x7f67d81557e0 2024-01-15T18:05:30.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.019+0000 7f67cfe00640 1 -- 172.21.15.27:0/4169336763 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d000ab70 con 0x7f67d81557e0 2024-01-15T18:05:30.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.019+0000 7f67df200640 1 -- 172.21.15.27:0/4169336763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67d81595a0 con 0x7f67d81557e0 2024-01-15T18:05:30.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.019+0000 7f67df200640 1 -- 172.21.15.27:0/4169336763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67d8159a80 con 0x7f67d81557e0 2024-01-15T18:05:30.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.019+0000 7f67cfe00640 1 -- 172.21.15.27:0/4169336763 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f67d000acd0 con 0x7f67d81557e0 2024-01-15T18:05:30.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.023+0000 7f67cfe00640 1 --2- 172.21.15.27:0/4169336763 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f67b00668f0 0x7f67b0068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.023+0000 7f67cfe00640 1 -- 172.21.15.27:0/4169336763 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f67d0083920 con 0x7f67d81557e0 2024-01-15T18:05:30.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.023+0000 7f67d7e00640 1 --2- 172.21.15.27:0/4169336763 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f67b00668f0 0x7f67b0068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.023+0000 7f67d7e00640 1 --2- 172.21.15.27:0/4169336763 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f67b00668f0 0x7f67b0068db0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f67c80ad000 tx=0x7f67c80a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:30.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.027+0000 7f67cfe00640 1 -- 172.21.15.27:0/4169336763 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f67d0050c10 con 0x7f67d81557e0 2024-01-15T18:05:30.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.183+0000 7f67df200640 1 -- 172.21.15.27:0/4169336763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f67d804e850 con 0x7f67d81557e0 2024-01-15T18:05:30.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.183+0000 7f67cfe00640 1 -- 172.21.15.27:0/4169336763 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f67d0018020 con 0x7f67d81557e0 2024-01-15T18:05:30.190 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-15T18:05:30.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.187+0000 7f67df200640 1 -- 172.21.15.27:0/4169336763 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f67b00668f0 msgr2=0x7f67b0068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.187+0000 7f67df200640 1 --2- 172.21.15.27:0/4169336763 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f67b00668f0 0x7f67b0068db0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f67c80ad000 tx=0x7f67c80a6000 comp rx=0 tx=0).stop 2024-01-15T18:05:30.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.187+0000 7f67df200640 1 -- 172.21.15.27:0/4169336763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67d81557e0 msgr2=0x7f67d815b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.187+0000 7f67df200640 1 --2- 172.21.15.27:0/4169336763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67d81557e0 0x7f67d815b700 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f67d0001570 tx=0x7f67d000cca0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.187+0000 7f67df200640 1 -- 172.21.15.27:0/4169336763 shutdown_connections 2024-01-15T18:05:30.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.187+0000 7f67df200640 1 --2- 172.21.15.27:0/4169336763 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f67b00668f0 0x7f67b0068db0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.187+0000 7f67df200640 1 --2- 172.21.15.27:0/4169336763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67d81557e0 0x7f67d815b700 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.187+0000 7f67df200640 1 --2- 172.21.15.27:0/4169336763 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f67d80b3d10 0x7f67d814d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.187+0000 7f67df200640 1 --2- 172.21.15.27:0/4169336763 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f67d806a4f0 0x7f67d814f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.187+0000 7f67df200640 1 -- 172.21.15.27:0/4169336763 >> 172.21.15.27:0/4169336763 conn(0x7f67d806b600 msgr2=0x7f67d814c480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:30.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.187+0000 7f67df200640 1 -- 172.21.15.27:0/4169336763 shutdown_connections 2024-01-15T18:05:30.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.187+0000 7f67df200640 1 -- 172.21.15.27:0/4169336763 wait complete. 2024-01-15T18:05:30.212 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T18:05:30.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.351+0000 7f8ad6200640 1 Processor -- start 2024-01-15T18:05:30.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.351+0000 7f8ad6200640 1 -- start start 2024-01-15T18:05:30.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.351+0000 7f8ad6200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8ad00cf460 0x7f8ad00cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.351+0000 7f8ad6200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f8ad015b4b0 0x7f8ad00cba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.351+0000 7f8ad6200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8ad00cec60 0x7f8ad00cbf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.351+0000 7f8ad6200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ad00cfd80 con 0x7f8ad00cec60 2024-01-15T18:05:30.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.351+0000 7f8ad6200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ad00d19a0 con 0x7f8ad015b4b0 2024-01-15T18:05:30.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.351+0000 7f8ad6200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad00d1c80 con 0x7f8ad00cf460 2024-01-15T18:05:30.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.351+0000 7f8acfe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8ad00cec60 0x7f8ad00cbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.351+0000 7f8acfe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8ad00cec60 0x7f8ad00cbf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40696/0 (socket says 172.21.15.27:40696) 2024-01-15T18:05:30.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.351+0000 7f8acfe00640 1 -- 172.21.15.27:0/2374681311 learned_addr learned my addr 172.21.15.27:0/2374681311 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:30.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.351+0000 7f8acea00640 1 --2- 172.21.15.27:0/2374681311 >> v2:172.21.15.93:3300/0 conn(0x7f8ad015b4b0 0x7f8ad00cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.351+0000 7f8acf400640 1 --2- 172.21.15.27:0/2374681311 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8ad00cf460 0x7f8ad00cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.351+0000 7f8acfe00640 1 -- 172.21.15.27:0/2374681311 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8ad00cf460 msgr2=0x7f8ad00cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:30.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.351+0000 7f8acfe00640 1 --2- 172.21.15.27:0/2374681311 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8ad00cf460 0x7f8ad00cf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.351+0000 7f8acfe00640 1 -- 172.21.15.27:0/2374681311 >> v2:172.21.15.93:3300/0 conn(0x7f8ad015b4b0 msgr2=0x7f8ad00cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:30.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.351+0000 7f8acfe00640 1 --2- 172.21.15.27:0/2374681311 >> v2:172.21.15.93:3300/0 conn(0x7f8ad015b4b0 0x7f8ad00cba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.351+0000 7f8acfe00640 1 -- 172.21.15.27:0/2374681311 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad00b07c0 con 0x7f8ad00cec60 2024-01-15T18:05:30.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.355+0000 7f8acfe00640 1 --2- 172.21.15.27:0/2374681311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad00cec60 0x7f8ad00cbf60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0067280 tx=0x7f8ac801c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc1713fd4de99ff7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:30.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.355+0000 7f8ace000640 1 -- 172.21.15.27:0/2374681311 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac80a5850 con 0x7f8ad00cec60 2024-01-15T18:05:30.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.355+0000 7f8ace000640 1 -- 172.21.15.27:0/2374681311 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac80a59b0 con 0x7f8ad00cec60 2024-01-15T18:05:30.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.355+0000 7f8ace000640 1 -- 172.21.15.27:0/2374681311 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac80a5d40 con 0x7f8ad00cec60 2024-01-15T18:05:30.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.355+0000 7f8ad6200640 1 -- 172.21.15.27:0/2374681311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad00cec60 msgr2=0x7f8ad00cbf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.355+0000 7f8ad6200640 1 --2- 172.21.15.27:0/2374681311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad00cec60 0x7f8ad00cbf60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0067280 tx=0x7f8ac801c9d0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.355+0000 7f8ad6200640 1 -- 172.21.15.27:0/2374681311 shutdown_connections 2024-01-15T18:05:30.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.355+0000 7f8ad6200640 1 --2- 172.21.15.27:0/2374681311 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad00cec60 0x7f8ad00cbf60 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.355+0000 7f8ad6200640 1 --2- 172.21.15.27:0/2374681311 >> v2:172.21.15.93:3300/0 conn(0x7f8ad015b4b0 0x7f8ad00cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.355+0000 7f8ad6200640 1 --2- 172.21.15.27:0/2374681311 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8ad00cf460 0x7f8ad00cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.355+0000 7f8ad6200640 1 -- 172.21.15.27:0/2374681311 >> 172.21.15.27:0/2374681311 conn(0x7f8ad00d0e80 msgr2=0x7f8ad00d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:30.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.355+0000 7f8ad6200640 1 -- 172.21.15.27:0/2374681311 shutdown_connections 2024-01-15T18:05:30.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.355+0000 7f8ad6200640 1 -- 172.21.15.27:0/2374681311 wait complete. 2024-01-15T18:05:30.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.359+0000 7f8ad6200640 1 Processor -- start 2024-01-15T18:05:30.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.359+0000 7f8ad6200640 1 -- start start 2024-01-15T18:05:30.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.359+0000 7f8ad6200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad00cec60 0x7f8ad0150c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.359+0000 7f8ad6200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8ad00cf460 0x7f8ad0155ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.359+0000 7f8ad6200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8ad015b4b0 0x7f8ad006c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.359+0000 7f8ad6200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad015cab0 con 0x7f8ad00cec60 2024-01-15T18:05:30.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.359+0000 7f8ad6200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad015cd50 con 0x7f8ad00cf460 2024-01-15T18:05:30.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.359+0000 7f8ad6200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad015d030 con 0x7f8ad015b4b0 2024-01-15T18:05:30.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.359+0000 7f8acf400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad00cec60 0x7f8ad0150c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.359+0000 7f8acf400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad00cec60 0x7f8ad0150c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40700/0 (socket says 172.21.15.27:40700) 2024-01-15T18:05:30.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.359+0000 7f8acea00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8ad00cf460 0x7f8ad0155ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.359+0000 7f8acf400640 1 -- 172.21.15.27:0/1138605268 learned_addr learned my addr 172.21.15.27:0/1138605268 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:30.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.359+0000 7f8acfe00640 1 --2- 172.21.15.27:0/1138605268 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8ad015b4b0 0x7f8ad006c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.359+0000 7f8acea00640 1 -- 172.21.15.27:0/1138605268 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8ad015b4b0 msgr2=0x7f8ad006c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.359+0000 7f8acea00640 1 --2- 172.21.15.27:0/1138605268 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8ad015b4b0 0x7f8ad006c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.359+0000 7f8acea00640 1 -- 172.21.15.27:0/1138605268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad00cec60 msgr2=0x7f8ad0150c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.359+0000 7f8acea00640 1 --2- 172.21.15.27:0/1138605268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad00cec60 0x7f8ad0150c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.359+0000 7f8acea00640 1 -- 172.21.15.27:0/1138605268 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac8009b30 con 0x7f8ad00cf460 2024-01-15T18:05:30.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.359+0000 7f8acf400640 1 --2- 172.21.15.27:0/1138605268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad00cec60 0x7f8ad0150c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:30.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.359+0000 7f8acea00640 1 --2- 172.21.15.27:0/1138605268 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8ad00cf460 0x7f8ad0155ac0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8ab800d5d0 tx=0x7f8ab800da90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:30.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.359+0000 7f8ac7400640 1 -- 172.21.15.27:0/1138605268 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab80125f0 con 0x7f8ad00cf460 2024-01-15T18:05:30.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.359+0000 7f8ac7400640 1 -- 172.21.15.27:0/1138605268 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ab8012750 con 0x7f8ad00cf460 2024-01-15T18:05:30.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.359+0000 7f8ac7400640 1 -- 172.21.15.27:0/1138605268 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab8012ae0 con 0x7f8ad00cf460 2024-01-15T18:05:30.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.359+0000 7f8ad6200640 1 -- 172.21.15.27:0/1138605268 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ad015d310 con 0x7f8ad00cf460 2024-01-15T18:05:30.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.363+0000 7f8ad6200640 1 -- 172.21.15.27:0/1138605268 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ad015e800 con 0x7f8ad00cf460 2024-01-15T18:05:30.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.363+0000 7f8ad6200640 1 -- 172.21.15.27:0/1138605268 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ad004e850 con 0x7f8ad00cf460 2024-01-15T18:05:30.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.363+0000 7f8ac7400640 1 -- 172.21.15.27:0/1138605268 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f8ab8012c40 con 0x7f8ad00cf460 2024-01-15T18:05:30.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.363+0000 7f8ac7400640 1 --2- 172.21.15.27:0/1138605268 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8aa8066920 0x7f8aa8068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.363+0000 7f8ac7400640 1 -- 172.21.15.27:0/1138605268 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8ab80847c0 con 0x7f8ad00cf460 2024-01-15T18:05:30.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.363+0000 7f8acf400640 1 --2- 172.21.15.27:0/1138605268 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8aa8066920 0x7f8aa8068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.363+0000 7f8acf400640 1 --2- 172.21.15.27:0/1138605268 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8aa8066920 0x7f8aa8068de0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8ac000bf80 tx=0x7f8ac0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:30.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.367+0000 7f8ac7400640 1 -- 172.21.15.27:0/1138605268 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8ab8051990 con 0x7f8ad00cf460 2024-01-15T18:05:30.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.527+0000 7f8ad6200640 1 -- 172.21.15.27:0/1138605268 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8ad00d0a90 con 0x7f8ad00cf460 2024-01-15T18:05:30.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.527+0000 7f8ac7400640 1 -- 172.21.15.27:0/1138605268 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8ab8054fb0 con 0x7f8ad00cf460 2024-01-15T18:05:30.531 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-15T18:05:30.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.527+0000 7f8ad6200640 1 -- 172.21.15.27:0/1138605268 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8aa8066920 msgr2=0x7f8aa8068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.527+0000 7f8ad6200640 1 --2- 172.21.15.27:0/1138605268 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8aa8066920 0x7f8aa8068de0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8ac000bf80 tx=0x7f8ac0008040 comp rx=0 tx=0).stop 2024-01-15T18:05:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.527+0000 7f8ad6200640 1 -- 172.21.15.27:0/1138605268 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8ad00cf460 msgr2=0x7f8ad0155ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.527+0000 7f8ad6200640 1 --2- 172.21.15.27:0/1138605268 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8ad00cf460 0x7f8ad0155ac0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8ab800d5d0 tx=0x7f8ab800da90 comp rx=0 tx=0).stop 2024-01-15T18:05:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.527+0000 7f8ad6200640 1 -- 172.21.15.27:0/1138605268 shutdown_connections 2024-01-15T18:05:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.527+0000 7f8ad6200640 1 --2- 172.21.15.27:0/1138605268 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8aa8066920 0x7f8aa8068de0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.527+0000 7f8ad6200640 1 --2- 172.21.15.27:0/1138605268 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8ad015b4b0 0x7f8ad006c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.527+0000 7f8ad6200640 1 --2- 172.21.15.27:0/1138605268 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8ad00cf460 0x7f8ad0155ac0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.527+0000 7f8ad6200640 1 --2- 172.21.15.27:0/1138605268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ad00cec60 0x7f8ad0150c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.527+0000 7f8ad6200640 1 -- 172.21.15.27:0/1138605268 >> 172.21.15.27:0/1138605268 conn(0x7f8ad00d0e80 msgr2=0x7f8ad00d1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.527+0000 7f8ad6200640 1 -- 172.21.15.27:0/1138605268 shutdown_connections 2024-01-15T18:05:30.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.527+0000 7f8ad6200640 1 -- 172.21.15.27:0/1138605268 wait complete. 2024-01-15T18:05:30.553 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone' 2024-01-15T18:05:30.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.691+0000 7efcf0200640 1 Processor -- start 2024-01-15T18:05:30.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.691+0000 7efcf0200640 1 -- start start 2024-01-15T18:05:30.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.691+0000 7efcf0200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efce815e7d0 0x7efce80cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.691+0000 7efcf0200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efce80b20d0 0x7efce80cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.691+0000 7efcf0200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efce80b38b0 con 0x7efce80cedc0 2024-01-15T18:05:30.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.691+0000 7efcf0200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efce80cd550 con 0x7efce815e7d0 2024-01-15T18:05:30.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.691+0000 7efcf0200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce80cd270 con 0x7efce80b20d0 2024-01-15T18:05:30.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.691+0000 7efced800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efce815e7d0 0x7efce80cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.691+0000 7efced800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efce815e7d0 0x7efce80cf2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40716/0 (socket says 172.21.15.27:40716) 2024-01-15T18:05:30.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.691+0000 7efced800640 1 -- 172.21.15.27:0/3462411897 learned_addr learned my addr 172.21.15.27:0/3462411897 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:30.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.691+0000 7efcee200640 1 --2- 172.21.15.27:0/3462411897 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efce80b20d0 0x7efce80cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.691+0000 7efced800640 1 -- 172.21.15.27:0/3462411897 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efce80b20d0 msgr2=0x7efce80cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:30.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.691+0000 7efced800640 1 --2- 172.21.15.27:0/3462411897 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efce80b20d0 0x7efce80cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.691+0000 7efced800640 1 -- 172.21.15.27:0/3462411897 >> v1:172.21.15.27:6789/0 conn(0x7efce80cedc0 legacy=0x7efce80cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.691+0000 7efced800640 1 -- 172.21.15.27:0/3462411897 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce80b35d0 con 0x7efce815e7d0 2024-01-15T18:05:30.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.691+0000 7efcee200640 1 --2- 172.21.15.27:0/3462411897 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efce80b20d0 0x7efce80cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:30.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.695+0000 7efced800640 1 --2- 172.21.15.27:0/3462411897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efce815e7d0 0x7efce80cf2a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7efcd8009870 tx=0x7efcd801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f4f820d5cc056b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:30.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.695+0000 7efce7e00640 1 -- 172.21.15.27:0/3462411897 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd80a7070 con 0x7efce815e7d0 2024-01-15T18:05:30.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.695+0000 7efce7e00640 1 -- 172.21.15.27:0/3462411897 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd80ab430 con 0x7efce815e7d0 2024-01-15T18:05:30.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.695+0000 7efce7e00640 1 -- 172.21.15.27:0/3462411897 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd80ab7c0 con 0x7efce815e7d0 2024-01-15T18:05:30.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.695+0000 7efcf0200640 1 -- 172.21.15.27:0/3462411897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efce815e7d0 msgr2=0x7efce80cf2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.695+0000 7efcf0200640 1 --2- 172.21.15.27:0/3462411897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efce815e7d0 0x7efce80cf2a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7efcd8009870 tx=0x7efcd801c540 comp rx=0 tx=0).stop 2024-01-15T18:05:30.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.695+0000 7efcf0200640 1 -- 172.21.15.27:0/3462411897 shutdown_connections 2024-01-15T18:05:30.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.695+0000 7efcf0200640 1 --2- 172.21.15.27:0/3462411897 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efce80b20d0 0x7efce80cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.695+0000 7efcf0200640 1 --2- 172.21.15.27:0/3462411897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efce815e7d0 0x7efce80cf2a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.695+0000 7efcf0200640 1 -- 172.21.15.27:0/3462411897 >> 172.21.15.27:0/3462411897 conn(0x7efce80b2a10 msgr2=0x7efce80b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:30.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.695+0000 7efcf0200640 1 -- 172.21.15.27:0/3462411897 shutdown_connections 2024-01-15T18:05:30.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.695+0000 7efcf0200640 1 -- 172.21.15.27:0/3462411897 wait complete. 2024-01-15T18:05:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.699+0000 7efcf0200640 1 Processor -- start 2024-01-15T18:05:30.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.699+0000 7efcf0200640 1 -- start start 2024-01-15T18:05:30.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.699+0000 7efcf0200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efce80b20d0 0x7efce81494f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.699+0000 7efcf0200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efce80cedc0 0x7efce8153e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.699+0000 7efcf0200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efce815e7d0 0x7efce81550c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.699+0000 7efcf0200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce8152840 con 0x7efce80b20d0 2024-01-15T18:05:30.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.699+0000 7efcf0200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce8154910 con 0x7efce80cedc0 2024-01-15T18:05:30.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.699+0000 7efcf0200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce81f42a0 con 0x7efce815e7d0 2024-01-15T18:05:30.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.699+0000 7efced800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efce80b20d0 0x7efce81494f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.699+0000 7efced800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efce80b20d0 0x7efce81494f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40724/0 (socket says 172.21.15.27:40724) 2024-01-15T18:05:30.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.699+0000 7efced800640 1 -- 172.21.15.27:0/4208549605 learned_addr learned my addr 172.21.15.27:0/4208549605 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:30.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.699+0000 7efcee200640 1 --2- 172.21.15.27:0/4208549605 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efce815e7d0 0x7efce81550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.699+0000 7efcece00640 1 --2- 172.21.15.27:0/4208549605 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efce80cedc0 0x7efce8153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.699+0000 7efced800640 1 -- 172.21.15.27:0/4208549605 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efce815e7d0 msgr2=0x7efce81550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.699+0000 7efced800640 1 --2- 172.21.15.27:0/4208549605 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efce815e7d0 0x7efce81550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.699+0000 7efced800640 1 -- 172.21.15.27:0/4208549605 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efce80cedc0 msgr2=0x7efce8153e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.699+0000 7efced800640 1 --2- 172.21.15.27:0/4208549605 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efce80cedc0 0x7efce8153e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.699+0000 7efced800640 1 -- 172.21.15.27:0/4208549605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd80094d0 con 0x7efce80b20d0 2024-01-15T18:05:30.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.699+0000 7efced800640 1 --2- 172.21.15.27:0/4208549605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efce80b20d0 0x7efce81494f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7efcd80ad040 tx=0x7efcd80192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:30.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.699+0000 7efce5600640 1 -- 172.21.15.27:0/4208549605 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd80bc070 con 0x7efce80b20d0 2024-01-15T18:05:30.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.699+0000 7efce5600640 1 -- 172.21.15.27:0/4208549605 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd80abe10 con 0x7efce80b20d0 2024-01-15T18:05:30.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.699+0000 7efcf0200640 1 -- 172.21.15.27:0/4208549605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce81f4580 con 0x7efce80b20d0 2024-01-15T18:05:30.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.699+0000 7efce5600640 1 -- 172.21.15.27:0/4208549605 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd80b4870 con 0x7efce80b20d0 2024-01-15T18:05:30.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.699+0000 7efcf0200640 1 -- 172.21.15.27:0/4208549605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce81f4b40 con 0x7efce80b20d0 2024-01-15T18:05:30.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.703+0000 7efcf0200640 1 -- 172.21.15.27:0/4208549605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efce81f43e0 con 0x7efce80b20d0 2024-01-15T18:05:30.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.703+0000 7efce5600640 1 -- 172.21.15.27:0/4208549605 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7efcd80b49d0 con 0x7efce80b20d0 2024-01-15T18:05:30.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.703+0000 7efce5600640 1 --2- 172.21.15.27:0/4208549605 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efcc00668f0 0x7efcc0068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.703+0000 7efce5600640 1 -- 172.21.15.27:0/4208549605 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efcd811dd40 con 0x7efce80b20d0 2024-01-15T18:05:30.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.703+0000 7efcece00640 1 --2- 172.21.15.27:0/4208549605 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efcc00668f0 0x7efcc0068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.703+0000 7efcece00640 1 --2- 172.21.15.27:0/4208549605 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efcc00668f0 0x7efcc0068db0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efce80b11a0 tx=0x7efcd0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:30.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.707+0000 7efce5600640 1 -- 172.21.15.27:0/4208549605 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7efce81f43e0 con 0x7efce80b20d0 2024-01-15T18:05:30.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.923+0000 7efcf0200640 1 -- 172.21.15.27:0/4208549605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]} v 0) v1 -- 0x7efce81f43e0 con 0x7efce80b20d0 2024-01-15T18:05:30.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.967+0000 7efce5600640 1 -- 172.21.15.27:0/4208549605 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]}]=0 v50) v1 ==== 155+0+0 (secure 0 0 0) 0x7efce81f43e0 con 0x7efce80b20d0 2024-01-15T18:05:30.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.971+0000 7efcf0200640 1 -- 172.21.15.27:0/4208549605 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efcc00668f0 msgr2=0x7efcc0068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.971+0000 7efcf0200640 1 --2- 172.21.15.27:0/4208549605 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efcc00668f0 0x7efcc0068db0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efce80b11a0 tx=0x7efcd0009040 comp rx=0 tx=0).stop 2024-01-15T18:05:30.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.971+0000 7efcf0200640 1 -- 172.21.15.27:0/4208549605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efce80b20d0 msgr2=0x7efce81494f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.971+0000 7efcf0200640 1 --2- 172.21.15.27:0/4208549605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efce80b20d0 0x7efce81494f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7efcd80ad040 tx=0x7efcd80192e0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.971+0000 7efcf0200640 1 -- 172.21.15.27:0/4208549605 shutdown_connections 2024-01-15T18:05:30.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.971+0000 7efcf0200640 1 --2- 172.21.15.27:0/4208549605 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efcc00668f0 0x7efcc0068db0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.971+0000 7efcf0200640 1 --2- 172.21.15.27:0/4208549605 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efce815e7d0 0x7efce81550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.971+0000 7efcf0200640 1 --2- 172.21.15.27:0/4208549605 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efce80cedc0 0x7efce8153e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.971+0000 7efcf0200640 1 --2- 172.21.15.27:0/4208549605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efce80b20d0 0x7efce81494f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.971+0000 7efcf0200640 1 -- 172.21.15.27:0/4208549605 >> 172.21.15.27:0/4208549605 conn(0x7efce80b2a10 msgr2=0x7efce80cfdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:30.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.971+0000 7efcf0200640 1 -- 172.21.15.27:0/4208549605 shutdown_connections 2024-01-15T18:05:30.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:30.971+0000 7efcf0200640 1 -- 172.21.15.27:0/4208549605 wait complete. 2024-01-15T18:05:30.996 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:05:31.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.135+0000 7f0cc7a00640 1 Processor -- start 2024-01-15T18:05:31.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.135+0000 7f0cc7a00640 1 -- start start 2024-01-15T18:05:31.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.135+0000 7f0cc7a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0cc00cf2e0 0x7f0cc00701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:31.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.135+0000 7f0cc7a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc00b1cb0 con 0x7f0cc015f550 2024-01-15T18:05:31.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.135+0000 7f0cc7a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc015fa00 con 0x7f0cc0158900 2024-01-15T18:05:31.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.135+0000 7f0cc7a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cc00b1f00 con 0x7f0cc00cf2e0 2024-01-15T18:05:31.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.135+0000 7f0cc5a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0cc00cf2e0 0x7f0cc00701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:31.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.139+0000 7f0cc5a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0cc00cf2e0 0x7f0cc00701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40732/0 (socket says 172.21.15.27:40732) 2024-01-15T18:05:31.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.139+0000 7f0cc5a00640 1 -- 172.21.15.27:0/1365079805 learned_addr learned my addr 172.21.15.27:0/1365079805 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:31.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.139+0000 7f0cbfe00640 1 --1- 172.21.15.27:0/1365079805 >> v1:172.21.15.27:6789/0 conn(0x7f0cc015f550 0x7f0cc00cebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34844/0 (socket says 172.21.15.27:34844) 2024-01-15T18:05:31.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.139+0000 7f0cc5a00640 1 -- 172.21.15.27:0/1365079805 >> v1:172.21.15.93:6789/0 conn(0x7f0cc0158900 legacy=0x7f0cc015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:31.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.139+0000 7f0cc5a00640 1 -- 172.21.15.27:0/1365079805 >> v1:172.21.15.27:6789/0 conn(0x7f0cc015f550 legacy=0x7f0cc00cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:31.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.139+0000 7f0cc5a00640 1 -- 172.21.15.27:0/1365079805 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc00b21e0 con 0x7f0cc00cf2e0 2024-01-15T18:05:31.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.139+0000 7f0cc5a00640 1 --2- 172.21.15.27:0/1365079805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cc00cf2e0 0x7f0cc00701e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8009e50 tx=0x7f0cb801c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eda518bc2d7aec6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:31.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.139+0000 7f0cbf400640 1 -- 172.21.15.27:0/1365079805 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc0070f30 con 0x7f0cc00cf2e0 2024-01-15T18:05:31.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.139+0000 7f0cbf400640 1 -- 172.21.15.27:0/1365079805 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cb80a58a0 con 0x7f0cc00cf2e0 2024-01-15T18:05:31.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.139+0000 7f0cbf400640 1 -- 172.21.15.27:0/1365079805 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb80a5c30 con 0x7f0cc00cf2e0 2024-01-15T18:05:31.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cc7a00640 1 -- 172.21.15.27:0/1365079805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cc00cf2e0 msgr2=0x7f0cc00701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:31.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cc7a00640 1 --2- 172.21.15.27:0/1365079805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cc00cf2e0 0x7f0cc00701e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8009e50 tx=0x7f0cb801c9e0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cc7a00640 1 -- 172.21.15.27:0/1365079805 shutdown_connections 2024-01-15T18:05:31.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cc7a00640 1 --2- 172.21.15.27:0/1365079805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cc00cf2e0 0x7f0cc00701e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cc7a00640 1 -- 172.21.15.27:0/1365079805 >> 172.21.15.27:0/1365079805 conn(0x7f0cc00cb700 msgr2=0x7f0cc00cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:31.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cc7a00640 1 -- 172.21.15.27:0/1365079805 shutdown_connections 2024-01-15T18:05:31.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cc7a00640 1 -- 172.21.15.27:0/1365079805 wait complete. 2024-01-15T18:05:31.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cc7a00640 1 Processor -- start 2024-01-15T18:05:31.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cc7a00640 1 -- start start 2024-01-15T18:05:31.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cc7a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0cc00cf2e0 0x7f0cc014f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:31.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cc7a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cc0158900 0x7f0cc00af060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:31.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cc7a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0cc015f550 0x7f0cc00af5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:31.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cc7a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc0150fc0 con 0x7f0cc0158900 2024-01-15T18:05:31.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cc7a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc01512a0 con 0x7f0cc015f550 2024-01-15T18:05:31.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cc7a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc0151580 con 0x7f0cc00cf2e0 2024-01-15T18:05:31.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cbfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cc0158900 0x7f0cc00af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:31.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cbfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cc0158900 0x7f0cc00af060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40748/0 (socket says 172.21.15.27:40748) 2024-01-15T18:05:31.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cbfe00640 1 -- 172.21.15.27:0/3317600894 learned_addr learned my addr 172.21.15.27:0/3317600894 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:31.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cc5a00640 1 --2- 172.21.15.27:0/3317600894 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0cc015f550 0x7f0cc00af5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:31.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cc5000640 1 --2- 172.21.15.27:0/3317600894 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0cc00cf2e0 0x7f0cc014f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:31.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cbfe00640 1 -- 172.21.15.27:0/3317600894 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0cc00cf2e0 msgr2=0x7f0cc014f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:31.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cbfe00640 1 --2- 172.21.15.27:0/3317600894 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0cc00cf2e0 0x7f0cc014f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cbfe00640 1 -- 172.21.15.27:0/3317600894 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0cc015f550 msgr2=0x7f0cc00af5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:31.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cbfe00640 1 --2- 172.21.15.27:0/3317600894 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0cc015f550 0x7f0cc00af5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cbfe00640 1 -- 172.21.15.27:0/3317600894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb8009b30 con 0x7f0cc0158900 2024-01-15T18:05:31.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.143+0000 7f0cc5a00640 1 --2- 172.21.15.27:0/3317600894 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0cc015f550 0x7f0cc00af5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:31.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.147+0000 7f0cbfe00640 1 --2- 172.21.15.27:0/3317600894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cc0158900 0x7f0cc00af060 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0ca800c430 tx=0x7f0ca800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:31.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.147+0000 7f0cbcc00640 1 -- 172.21.15.27:0/3317600894 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca80074e0 con 0x7f0cc0158900 2024-01-15T18:05:31.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.147+0000 7f0cbcc00640 1 -- 172.21.15.27:0/3317600894 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca8007640 con 0x7f0cc0158900 2024-01-15T18:05:31.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.147+0000 7f0cc7a00640 1 -- 172.21.15.27:0/3317600894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cc0151860 con 0x7f0cc0158900 2024-01-15T18:05:31.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.147+0000 7f0cbcc00640 1 -- 172.21.15.27:0/3317600894 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca8004a10 con 0x7f0cc0158900 2024-01-15T18:05:31.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.147+0000 7f0cc7a00640 1 -- 172.21.15.27:0/3317600894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cc014f7d0 con 0x7f0cc0158900 2024-01-15T18:05:31.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.147+0000 7f0cc7a00640 1 -- 172.21.15.27:0/3317600894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cc004e850 con 0x7f0cc0158900 2024-01-15T18:05:31.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.147+0000 7f0cbcc00640 1 -- 172.21.15.27:0/3317600894 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f0ca8004b70 con 0x7f0cc0158900 2024-01-15T18:05:31.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.147+0000 7f0cbcc00640 1 --2- 172.21.15.27:0/3317600894 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c980668f0 0x7f0c98068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:31.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.147+0000 7f0cbcc00640 1 -- 172.21.15.27:0/3317600894 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ca8083d00 con 0x7f0cc0158900 2024-01-15T18:05:31.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.147+0000 7f0cc5000640 1 --2- 172.21.15.27:0/3317600894 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c980668f0 0x7f0c98068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:31.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.151+0000 7f0cc5000640 1 --2- 172.21.15.27:0/3317600894 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c980668f0 0x7f0c98068db0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0000f70 tx=0x7f0cb0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:31.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.151+0000 7f0cbcc00640 1 -- 172.21.15.27:0/3317600894 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0ca8050ea0 con 0x7f0cc0158900 2024-01-15T18:05:31.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.307+0000 7f0cc7a00640 1 -- 172.21.15.27:0/3317600894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0cc014f7d0 con 0x7f0cc0158900 2024-01-15T18:05:31.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.311+0000 7f0cbcc00640 1 -- 172.21.15.27:0/3317600894 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0cc014f7d0 con 0x7f0cc0158900 2024-01-15T18:05:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.315+0000 7f0cc7a00640 1 -- 172.21.15.27:0/3317600894 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c980668f0 msgr2=0x7f0c98068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.315+0000 7f0cc7a00640 1 --2- 172.21.15.27:0/3317600894 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c980668f0 0x7f0c98068db0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0000f70 tx=0x7f0cb0009040 comp rx=0 tx=0).stop 2024-01-15T18:05:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.315+0000 7f0cc7a00640 1 -- 172.21.15.27:0/3317600894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cc0158900 msgr2=0x7f0cc00af060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.315+0000 7f0cc7a00640 1 --2- 172.21.15.27:0/3317600894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cc0158900 0x7f0cc00af060 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0ca800c430 tx=0x7f0ca800c8f0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.315+0000 7f0cc7a00640 1 -- 172.21.15.27:0/3317600894 shutdown_connections 2024-01-15T18:05:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.315+0000 7f0cc7a00640 1 --2- 172.21.15.27:0/3317600894 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c980668f0 0x7f0c98068db0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.315+0000 7f0cc7a00640 1 --2- 172.21.15.27:0/3317600894 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0cc015f550 0x7f0cc00af5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.315+0000 7f0cc7a00640 1 --2- 172.21.15.27:0/3317600894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cc0158900 0x7f0cc00af060 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.315+0000 7f0cc7a00640 1 --2- 172.21.15.27:0/3317600894 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0cc00cf2e0 0x7f0cc014f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.315+0000 7f0cc7a00640 1 -- 172.21.15.27:0/3317600894 >> 172.21.15.27:0/3317600894 conn(0x7f0cc00cb700 msgr2=0x7f0cc00cebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.315+0000 7f0cc7a00640 1 -- 172.21.15.27:0/3317600894 shutdown_connections 2024-01-15T18:05:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.315+0000 7f0cc7a00640 1 -- 172.21.15.27:0/3317600894 wait complete. 2024-01-15T18:05:31.340 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T18:05:31.340 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:05:31.424 DEBUG:teuthology.orchestra.run.smithi093:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:05:31.485 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T18:05:31.485 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:05:31.485 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:05:31.494 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:05:31.494 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:05:31.546 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:31.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.723+0000 7f9399e00640 1 Processor -- start 2024-01-15T18:05:31.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.723+0000 7f9399e00640 1 -- start start 2024-01-15T18:05:31.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.723+0000 7f9399e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f939415d2c0 0x7f93940c62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:31.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.723+0000 7f9399e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f93940c6800 0x7f939406a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:31.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.723+0000 7f9399e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f939406aa30 0x7f93940712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:31.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.723+0000 7f9399e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f939406aec0 con 0x7f939406aa30 2024-01-15T18:05:31.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.723+0000 7f9399e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93940ce600 con 0x7f93940c6800 2024-01-15T18:05:31.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.723+0000 7f9399e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93940c5d00 con 0x7f939415d2c0 2024-01-15T18:05:31.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.723+0000 7f9393400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f939406aa30 0x7f93940712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:31.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.723+0000 7f9393400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f939406aa30 0x7f93940712b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40752/0 (socket says 172.21.15.27:40752) 2024-01-15T18:05:31.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.723+0000 7f9393400640 1 -- 172.21.15.27:0/3826428677 learned_addr learned my addr 172.21.15.27:0/3826428677 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:31.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.723+0000 7f9392a00640 1 --2- 172.21.15.27:0/3826428677 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f939415d2c0 0x7f93940c62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:31.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.723+0000 7f9392000640 1 --2- 172.21.15.27:0/3826428677 >> v2:172.21.15.93:3300/0 conn(0x7f93940c6800 0x7f939406a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:31.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.723+0000 7f9393400640 1 -- 172.21.15.27:0/3826428677 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f939415d2c0 msgr2=0x7f93940c62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:31.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.723+0000 7f9393400640 1 --2- 172.21.15.27:0/3826428677 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f939415d2c0 0x7f93940c62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.723+0000 7f9393400640 1 -- 172.21.15.27:0/3826428677 >> v2:172.21.15.93:3300/0 conn(0x7f93940c6800 msgr2=0x7f939406a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:31.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.723+0000 7f9393400640 1 --2- 172.21.15.27:0/3826428677 >> v2:172.21.15.93:3300/0 conn(0x7f93940c6800 0x7f939406a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.723+0000 7f9393400640 1 -- 172.21.15.27:0/3826428677 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f939406fc90 con 0x7f939406aa30 2024-01-15T18:05:31.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.723+0000 7f9392000640 1 --2- 172.21.15.27:0/3826428677 >> v2:172.21.15.93:3300/0 conn(0x7f93940c6800 0x7f939406a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:31.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.727+0000 7f9393400640 1 --2- 172.21.15.27:0/3826428677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f939406aa30 0x7f93940712b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f93940681f0 tx=0x7f938c01c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56f1771f31c2ed63 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:31.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.727+0000 7f9391600640 1 -- 172.21.15.27:0/3826428677 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c0a6480 con 0x7f939406aa30 2024-01-15T18:05:31.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.727+0000 7f9391600640 1 -- 172.21.15.27:0/3826428677 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f938c0a65e0 con 0x7f939406aa30 2024-01-15T18:05:31.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.727+0000 7f9391600640 1 -- 172.21.15.27:0/3826428677 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c0a6970 con 0x7f939406aa30 2024-01-15T18:05:31.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.727+0000 7f9399e00640 1 -- 172.21.15.27:0/3826428677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f939406aa30 msgr2=0x7f93940712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:31.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.727+0000 7f9399e00640 1 --2- 172.21.15.27:0/3826428677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f939406aa30 0x7f93940712b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f93940681f0 tx=0x7f938c01c6e0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.727+0000 7f9399e00640 1 -- 172.21.15.27:0/3826428677 shutdown_connections 2024-01-15T18:05:31.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.727+0000 7f9399e00640 1 --2- 172.21.15.27:0/3826428677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f939406aa30 0x7f93940712b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.727+0000 7f9399e00640 1 --2- 172.21.15.27:0/3826428677 >> v2:172.21.15.93:3300/0 conn(0x7f93940c6800 0x7f939406a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.727+0000 7f9399e00640 1 --2- 172.21.15.27:0/3826428677 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f939415d2c0 0x7f93940c62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.727+0000 7f9399e00640 1 -- 172.21.15.27:0/3826428677 >> 172.21.15.27:0/3826428677 conn(0x7f93940cfd60 msgr2=0x7f93940d0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:31.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.727+0000 7f9399e00640 1 -- 172.21.15.27:0/3826428677 shutdown_connections 2024-01-15T18:05:31.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.727+0000 7f9399e00640 1 -- 172.21.15.27:0/3826428677 wait complete. 2024-01-15T18:05:31.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.731+0000 7f9399e00640 1 Processor -- start 2024-01-15T18:05:31.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.731+0000 7f9399e00640 1 -- start start 2024-01-15T18:05:31.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.731+0000 7f9399e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f939406aa30 0x7f939414e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:31.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.731+0000 7f9399e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93940c6800 0x7f939414fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:31.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.731+0000 7f9399e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f939415d2c0 0x7f9394150220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:31.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.731+0000 7f9399e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939414ef00 con 0x7f93940c6800 2024-01-15T18:05:31.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.731+0000 7f9399e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939406e7c0 con 0x7f939406aa30 2024-01-15T18:05:31.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.731+0000 7f9399e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f939406eaa0 con 0x7f939415d2c0 2024-01-15T18:05:31.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.731+0000 7f9392000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93940c6800 0x7f939414fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:31.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.731+0000 7f9392000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93940c6800 0x7f939414fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40768/0 (socket says 172.21.15.27:40768) 2024-01-15T18:05:31.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.731+0000 7f9392000640 1 -- 172.21.15.27:0/14645613 learned_addr learned my addr 172.21.15.27:0/14645613 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:31.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.731+0000 7f9392a00640 1 --2- 172.21.15.27:0/14645613 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f939406aa30 0x7f939414e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:31.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.731+0000 7f9393400640 1 --2- 172.21.15.27:0/14645613 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f939415d2c0 0x7f9394150220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:31.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.731+0000 7f9392000640 1 -- 172.21.15.27:0/14645613 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f939415d2c0 msgr2=0x7f9394150220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:31.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.731+0000 7f9392000640 1 --2- 172.21.15.27:0/14645613 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f939415d2c0 0x7f9394150220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.731+0000 7f9392000640 1 -- 172.21.15.27:0/14645613 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f939406aa30 msgr2=0x7f939414e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:31.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.731+0000 7f9392000640 1 --2- 172.21.15.27:0/14645613 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f939406aa30 0x7f939414e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.731+0000 7f9392000640 1 -- 172.21.15.27:0/14645613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f938c009b30 con 0x7f93940c6800 2024-01-15T18:05:31.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.731+0000 7f9392000640 1 --2- 172.21.15.27:0/14645613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93940c6800 0x7f939414fce0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f937c00e810 tx=0x7f937c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:31.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.731+0000 7f938b400640 1 -- 172.21.15.27:0/14645613 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f937c00c8d0 con 0x7f93940c6800 2024-01-15T18:05:31.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.731+0000 7f938b400640 1 -- 172.21.15.27:0/14645613 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f937c00ca30 con 0x7f93940c6800 2024-01-15T18:05:31.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.731+0000 7f938b400640 1 -- 172.21.15.27:0/14645613 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f937c00cde0 con 0x7f93940c6800 2024-01-15T18:05:31.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.731+0000 7f9399e00640 1 -- 172.21.15.27:0/14645613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f939406ed80 con 0x7f93940c6800 2024-01-15T18:05:31.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.731+0000 7f9399e00640 1 -- 172.21.15.27:0/14645613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f939406da90 con 0x7f93940c6800 2024-01-15T18:05:31.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.731+0000 7f9399e00640 1 -- 172.21.15.27:0/14645613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9394067c00 con 0x7f93940c6800 2024-01-15T18:05:31.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.735+0000 7f938b400640 1 -- 172.21.15.27:0/14645613 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f937c013410 con 0x7f93940c6800 2024-01-15T18:05:31.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.735+0000 7f938b400640 1 --2- 172.21.15.27:0/14645613 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f936c0668f0 0x7f936c068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:31.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.735+0000 7f938b400640 1 -- 172.21.15.27:0/14645613 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f937c084140 con 0x7f93940c6800 2024-01-15T18:05:31.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.735+0000 7f9392a00640 1 --2- 172.21.15.27:0/14645613 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f936c0668f0 0x7f936c068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:31.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.735+0000 7f9392a00640 1 --2- 172.21.15.27:0/14645613 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f936c0668f0 0x7f936c068db0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9384009a40 tx=0x7f9384009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:31.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.739+0000 7f938b400640 1 -- 172.21.15.27:0/14645613 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f937c051420 con 0x7f93940c6800 2024-01-15T18:05:31.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.895+0000 7f9399e00640 1 -- 172.21.15.27:0/14645613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f939404e850 con 0x7f93940c6800 2024-01-15T18:05:31.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.895+0000 7f938b400640 1 -- 172.21.15.27:0/14645613 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16409 (secure 0 0 0) 0x7f937c054a40 con 0x7f93940c6800 2024-01-15T18:05:31.902 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:31.902 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:04:59.837052+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:04:56.220608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T18:04:57.210421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:31.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.899+0000 7f9399e00640 1 -- 172.21.15.27:0/14645613 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f936c0668f0 msgr2=0x7f936c068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:31.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.899+0000 7f9399e00640 1 --2- 172.21.15.27:0/14645613 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f936c0668f0 0x7f936c068db0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9384009a40 tx=0x7f9384009190 comp rx=0 tx=0).stop 2024-01-15T18:05:31.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.899+0000 7f9399e00640 1 -- 172.21.15.27:0/14645613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93940c6800 msgr2=0x7f939414fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:31.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.899+0000 7f9399e00640 1 --2- 172.21.15.27:0/14645613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93940c6800 0x7f939414fce0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f937c00e810 tx=0x7f937c00ecd0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.899+0000 7f9399e00640 1 -- 172.21.15.27:0/14645613 shutdown_connections 2024-01-15T18:05:31.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.899+0000 7f9399e00640 1 --2- 172.21.15.27:0/14645613 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f936c0668f0 0x7f936c068db0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.899+0000 7f9399e00640 1 --2- 172.21.15.27:0/14645613 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f939415d2c0 0x7f9394150220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.899+0000 7f9399e00640 1 --2- 172.21.15.27:0/14645613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93940c6800 0x7f939414fce0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.899+0000 7f9399e00640 1 --2- 172.21.15.27:0/14645613 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f939406aa30 0x7f939414e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.899+0000 7f9399e00640 1 -- 172.21.15.27:0/14645613 >> 172.21.15.27:0/14645613 conn(0x7f93940cfd60 msgr2=0x7f93940aff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:31.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.899+0000 7f9399e00640 1 -- 172.21.15.27:0/14645613 shutdown_connections 2024-01-15T18:05:31.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:31.899+0000 7f9399e00640 1 -- 172.21.15.27:0/14645613 wait complete. 2024-01-15T18:05:31.923 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T18:04:56.220608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T18:04:57.210421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:05:31.924 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:05:32.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.063+0000 7f234c400640 1 Processor -- start 2024-01-15T18:05:32.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.063+0000 7f234c400640 1 -- start start 2024-01-15T18:05:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.063+0000 7f234c400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f234415f550 0x7f23440cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.063+0000 7f234c400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23440b1cb0 con 0x7f2344158900 2024-01-15T18:05:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.063+0000 7f234c400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f234415fa00 con 0x7f23440cf110 2024-01-15T18:05:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.063+0000 7f234c400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23440cf540 con 0x7f234415f550 2024-01-15T18:05:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.063+0000 7f2349a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2344158900 0x7f234415bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34852/0 (socket says 172.21.15.27:34852) 2024-01-15T18:05:32.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.063+0000 7f2349a00640 1 -- 172.21.15.27:0/536190769 learned_addr learned my addr 172.21.15.27:0/536190769 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:32.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.063+0000 7f2349000640 1 --2- 172.21.15.27:0/536190769 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f234415f550 0x7f23440cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.063+0000 7f2343e00640 1 -- 172.21.15.27:0/536190769 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486336903 0 0) 0x7f23440b1cb0 con 0x7f2344158900 2024-01-15T18:05:32.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.063+0000 7f2343e00640 1 -- 172.21.15.27:0/536190769 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2324003530 con 0x7f2344158900 2024-01-15T18:05:32.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.063+0000 7f2343e00640 1 -- 172.21.15.27:0/536190769 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3000265093 0 0) 0x7f234415fa00 con 0x7f23440cf110 2024-01-15T18:05:32.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.063+0000 7f2343e00640 1 -- 172.21.15.27:0/536190769 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23440b1cb0 con 0x7f23440cf110 2024-01-15T18:05:32.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.063+0000 7f2349000640 1 -- 172.21.15.27:0/536190769 >> v1:172.21.15.93:6789/0 conn(0x7f23440cf110 legacy=0x7f23440701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.063+0000 7f2349000640 1 -- 172.21.15.27:0/536190769 >> v1:172.21.15.27:6789/0 conn(0x7f2344158900 legacy=0x7f234415bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.063+0000 7f2349000640 1 -- 172.21.15.27:0/536190769 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23440b1f00 con 0x7f234415f550 2024-01-15T18:05:32.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.067+0000 7f2349000640 1 --2- 172.21.15.27:0/536190769 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f234415f550 0x7f23440cebd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f232c0097f0 tx=0x7f232c01c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1951b77ba283d978 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:32.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.067+0000 7f2343e00640 1 -- 172.21.15.27:0/536190769 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232c0aa5f0 con 0x7f234415f550 2024-01-15T18:05:32.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.067+0000 7f2343e00640 1 -- 172.21.15.27:0/536190769 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f232c0aa750 con 0x7f234415f550 2024-01-15T18:05:32.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.067+0000 7f2343e00640 1 -- 172.21.15.27:0/536190769 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232c0aaae0 con 0x7f234415f550 2024-01-15T18:05:32.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.067+0000 7f234c400640 1 -- 172.21.15.27:0/536190769 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f234415f550 msgr2=0x7f23440cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.067+0000 7f234c400640 1 --2- 172.21.15.27:0/536190769 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f234415f550 0x7f23440cebd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f232c0097f0 tx=0x7f232c01c490 comp rx=0 tx=0).stop 2024-01-15T18:05:32.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.067+0000 7f234c400640 1 -- 172.21.15.27:0/536190769 shutdown_connections 2024-01-15T18:05:32.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.067+0000 7f234c400640 1 --2- 172.21.15.27:0/536190769 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f234415f550 0x7f23440cebd0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.067+0000 7f234c400640 1 -- 172.21.15.27:0/536190769 >> 172.21.15.27:0/536190769 conn(0x7f23440cb700 msgr2=0x7f23440cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:32.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.067+0000 7f234c400640 1 -- 172.21.15.27:0/536190769 shutdown_connections 2024-01-15T18:05:32.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.067+0000 7f234c400640 1 -- 172.21.15.27:0/536190769 wait complete. 2024-01-15T18:05:32.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.071+0000 7f234c400640 1 Processor -- start 2024-01-15T18:05:32.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.071+0000 7f234c400640 1 -- start start 2024-01-15T18:05:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.071+0000 7f234c400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f23440cf110 0x7f2344155c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.071+0000 7f234c400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2344158900 0x7f234414ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.071+0000 7f234c400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f234415f550 0x7f23440af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.071+0000 7f234c400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23441507e0 con 0x7f234415f550 2024-01-15T18:05:32.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.071+0000 7f234c400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23440ae810 con 0x7f2344158900 2024-01-15T18:05:32.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.071+0000 7f234c400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23440aeaf0 con 0x7f23440cf110 2024-01-15T18:05:32.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.071+0000 7f234a400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f234415f550 0x7f23440af8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.071+0000 7f234a400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f234415f550 0x7f23440af8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40782/0 (socket says 172.21.15.27:40782) 2024-01-15T18:05:32.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.071+0000 7f234a400640 1 -- 172.21.15.27:0/4287807429 learned_addr learned my addr 172.21.15.27:0/4287807429 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:32.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.071+0000 7f2349a00640 1 --2- 172.21.15.27:0/4287807429 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f23440cf110 0x7f2344155c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.071+0000 7f2349000640 1 --2- 172.21.15.27:0/4287807429 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2344158900 0x7f234414ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.071+0000 7f234a400640 1 -- 172.21.15.27:0/4287807429 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f23440cf110 msgr2=0x7f2344155c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.071+0000 7f234a400640 1 --2- 172.21.15.27:0/4287807429 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f23440cf110 0x7f2344155c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.071+0000 7f234a400640 1 -- 172.21.15.27:0/4287807429 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2344158900 msgr2=0x7f234414ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.071+0000 7f234a400640 1 --2- 172.21.15.27:0/4287807429 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2344158900 0x7f234414ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.071+0000 7f234a400640 1 -- 172.21.15.27:0/4287807429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232c009420 con 0x7f234415f550 2024-01-15T18:05:32.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.071+0000 7f234a400640 1 --2- 172.21.15.27:0/4287807429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f234415f550 0x7f23440af8b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f233c00c730 tx=0x7f233c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:32.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.071+0000 7f2341600640 1 -- 172.21.15.27:0/4287807429 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233c003b80 con 0x7f234415f550 2024-01-15T18:05:32.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.071+0000 7f2341600640 1 -- 172.21.15.27:0/4287807429 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f233c003ce0 con 0x7f234415f550 2024-01-15T18:05:32.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.071+0000 7f2341600640 1 -- 172.21.15.27:0/4287807429 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233c00aa90 con 0x7f234415f550 2024-01-15T18:05:32.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.071+0000 7f234c400640 1 -- 172.21.15.27:0/4287807429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23440aedd0 con 0x7f234415f550 2024-01-15T18:05:32.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.071+0000 7f234c400640 1 -- 172.21.15.27:0/4287807429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23440af390 con 0x7f234415f550 2024-01-15T18:05:32.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.075+0000 7f234c400640 1 -- 172.21.15.27:0/4287807429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2344071690 con 0x7f234415f550 2024-01-15T18:05:32.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.075+0000 7f2341600640 1 -- 172.21.15.27:0/4287807429 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f233c00abf0 con 0x7f234415f550 2024-01-15T18:05:32.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.075+0000 7f2341600640 1 --2- 172.21.15.27:0/4287807429 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f231c0668f0 0x7f231c068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.075+0000 7f2341600640 1 -- 172.21.15.27:0/4287807429 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f233c084110 con 0x7f234415f550 2024-01-15T18:05:32.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.075+0000 7f2349a00640 1 --2- 172.21.15.27:0/4287807429 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f231c0668f0 0x7f231c068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.075+0000 7f2349a00640 1 --2- 172.21.15.27:0/4287807429 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f231c0668f0 0x7f231c068db0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2334000f70 tx=0x7f2334007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.079+0000 7f2341600640 1 -- 172.21.15.27:0/4287807429 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f233c0513f0 con 0x7f234415f550 2024-01-15T18:05:32.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.235+0000 7f234c400640 1 -- 172.21.15.27:0/4287807429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f23440af580 con 0x7f234415f550 2024-01-15T18:05:32.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.239+0000 7f2341600640 1 -- 172.21.15.27:0/4287807429 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f233c054a10 con 0x7f234415f550 2024-01-15T18:05:32.243 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:05:32.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.239+0000 7f234c400640 1 -- 172.21.15.27:0/4287807429 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f231c0668f0 msgr2=0x7f231c068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.239+0000 7f234c400640 1 --2- 172.21.15.27:0/4287807429 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f231c0668f0 0x7f231c068db0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2334000f70 tx=0x7f2334007d40 comp rx=0 tx=0).stop 2024-01-15T18:05:32.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.239+0000 7f234c400640 1 -- 172.21.15.27:0/4287807429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f234415f550 msgr2=0x7f23440af8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.239+0000 7f234c400640 1 --2- 172.21.15.27:0/4287807429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f234415f550 0x7f23440af8b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f233c00c730 tx=0x7f233c00cbf0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.239+0000 7f234c400640 1 -- 172.21.15.27:0/4287807429 shutdown_connections 2024-01-15T18:05:32.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.239+0000 7f234c400640 1 --2- 172.21.15.27:0/4287807429 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f231c0668f0 0x7f231c068db0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.239+0000 7f234c400640 1 --2- 172.21.15.27:0/4287807429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f234415f550 0x7f23440af8b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.239+0000 7f234c400640 1 --2- 172.21.15.27:0/4287807429 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2344158900 0x7f234414ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.239+0000 7f234c400640 1 --2- 172.21.15.27:0/4287807429 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f23440cf110 0x7f2344155c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.239+0000 7f234c400640 1 -- 172.21.15.27:0/4287807429 >> 172.21.15.27:0/4287807429 conn(0x7f23440cb700 msgr2=0x7f23440adcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:32.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.239+0000 7f234c400640 1 -- 172.21.15.27:0/4287807429 shutdown_connections 2024-01-15T18:05:32.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.239+0000 7f234c400640 1 -- 172.21.15.27:0/4287807429 wait complete. 2024-01-15T18:05:32.265 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T18:05:32.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.403+0000 7fa921c00640 1 Processor -- start 2024-01-15T18:05:32.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.403+0000 7fa921c00640 1 -- start start 2024-01-15T18:05:32.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.403+0000 7fa921c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa91c15b4b0 0x7fa91c0b1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.407+0000 7fa921c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa91c0cec60 0x7fa91c0b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.407+0000 7fa921c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa91c0d19a0 con 0x7fa91c0cf460 2024-01-15T18:05:32.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.407+0000 7fa921c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa91c0d1c80 con 0x7fa91c15b4b0 2024-01-15T18:05:32.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.407+0000 7fa921c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c0b07c0 con 0x7fa91c0cec60 2024-01-15T18:05:32.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.407+0000 7fa91a000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa91c15b4b0 0x7fa91c0b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.407+0000 7fa91b400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa91c0cec60 0x7fa91c0b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.407+0000 7fa91a000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa91c15b4b0 0x7fa91c0b1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:54098/0 (socket says 172.21.15.27:54098) 2024-01-15T18:05:32.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.407+0000 7fa91a000640 1 -- 172.21.15.27:0/3949736551 learned_addr learned my addr 172.21.15.27:0/3949736551 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:32.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.407+0000 7fa91aa00640 1 --1- 172.21.15.27:0/3949736551 >> v1:172.21.15.93:6789/0 conn(0x7fa91c0cf460 0x7fa91c0cf840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:57752/0 (socket says 172.21.15.27:57752) 2024-01-15T18:05:32.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.407+0000 7fa91a000640 1 -- 172.21.15.27:0/3949736551 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa91c0cec60 msgr2=0x7fa91c0b1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:32.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.407+0000 7fa91a000640 1 --2- 172.21.15.27:0/3949736551 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa91c0cec60 0x7fa91c0b1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.407+0000 7fa91a000640 1 -- 172.21.15.27:0/3949736551 >> v1:172.21.15.93:6789/0 conn(0x7fa91c0cf460 legacy=0x7fa91c0cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.407+0000 7fa91a000640 1 -- 172.21.15.27:0/3949736551 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91c0b0aa0 con 0x7fa91c15b4b0 2024-01-15T18:05:32.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.407+0000 7fa91b400640 1 --2- 172.21.15.27:0/3949736551 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa91c0cec60 0x7fa91c0b1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:32.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.407+0000 7fa919600640 1 -- 172.21.15.27:0/3949736551 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 240172896 0 0) 0x7fa91c0d19a0 con 0x7fa91c0cf460 2024-01-15T18:05:32.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.407+0000 7fa91a000640 1 --2- 172.21.15.27:0/3949736551 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa91c15b4b0 0x7fa91c0b1580 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa904009870 tx=0x7fa90401c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12bf8ed8f178aa9c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:32.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.411+0000 7fa919600640 1 -- 172.21.15.27:0/3949736551 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c0b2810 con 0x7fa91c15b4b0 2024-01-15T18:05:32.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.411+0000 7fa919600640 1 -- 172.21.15.27:0/3949736551 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9040a9400 con 0x7fa91c15b4b0 2024-01-15T18:05:32.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.411+0000 7fa919600640 1 -- 172.21.15.27:0/3949736551 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9040a9790 con 0x7fa91c15b4b0 2024-01-15T18:05:32.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.411+0000 7fa921c00640 1 -- 172.21.15.27:0/3949736551 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa91c15b4b0 msgr2=0x7fa91c0b1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.411+0000 7fa921c00640 1 --2- 172.21.15.27:0/3949736551 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa91c15b4b0 0x7fa91c0b1580 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa904009870 tx=0x7fa90401c540 comp rx=0 tx=0).stop 2024-01-15T18:05:32.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.411+0000 7fa921c00640 1 -- 172.21.15.27:0/3949736551 shutdown_connections 2024-01-15T18:05:32.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.411+0000 7fa921c00640 1 --2- 172.21.15.27:0/3949736551 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa91c0cec60 0x7fa91c0b1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.411+0000 7fa921c00640 1 --2- 172.21.15.27:0/3949736551 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa91c15b4b0 0x7fa91c0b1580 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.411+0000 7fa921c00640 1 -- 172.21.15.27:0/3949736551 >> 172.21.15.27:0/3949736551 conn(0x7fa91c0d0e80 msgr2=0x7fa91c0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:32.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.411+0000 7fa921c00640 1 -- 172.21.15.27:0/3949736551 shutdown_connections 2024-01-15T18:05:32.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.411+0000 7fa921c00640 1 -- 172.21.15.27:0/3949736551 wait complete. 2024-01-15T18:05:32.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.411+0000 7fa921c00640 1 Processor -- start 2024-01-15T18:05:32.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.411+0000 7fa921c00640 1 -- start start 2024-01-15T18:05:32.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.411+0000 7fa921c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa91c0cec60 0x7fa91c0ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.411+0000 7fa921c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa91c0cf460 0x7fa91c14d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.411+0000 7fa921c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa91c15b4b0 0x7fa91c0c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.411+0000 7fa921c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c0cac40 con 0x7fa91c0cf460 2024-01-15T18:05:32.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.411+0000 7fa921c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c06ee90 con 0x7fa91c0cec60 2024-01-15T18:05:32.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.411+0000 7fa921c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa91c06f170 con 0x7fa91c15b4b0 2024-01-15T18:05:32.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.415+0000 7fa91a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa91c0cf460 0x7fa91c14d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.415+0000 7fa91a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa91c0cf460 0x7fa91c14d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40798/0 (socket says 172.21.15.27:40798) 2024-01-15T18:05:32.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.415+0000 7fa91a000640 1 -- 172.21.15.27:0/2841975779 learned_addr learned my addr 172.21.15.27:0/2841975779 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:32.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.415+0000 7fa91b400640 1 --2- 172.21.15.27:0/2841975779 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa91c15b4b0 0x7fa91c0c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.415+0000 7fa91aa00640 1 --2- 172.21.15.27:0/2841975779 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa91c0cec60 0x7fa91c0ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.415+0000 7fa91a000640 1 -- 172.21.15.27:0/2841975779 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa91c15b4b0 msgr2=0x7fa91c0c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.415+0000 7fa91a000640 1 --2- 172.21.15.27:0/2841975779 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa91c15b4b0 0x7fa91c0c84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.415+0000 7fa91a000640 1 -- 172.21.15.27:0/2841975779 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa91c0cec60 msgr2=0x7fa91c0ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.415+0000 7fa91a000640 1 --2- 172.21.15.27:0/2841975779 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa91c0cec60 0x7fa91c0ca560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.415+0000 7fa91a000640 1 -- 172.21.15.27:0/2841975779 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa914009a70 con 0x7fa91c0cf460 2024-01-15T18:05:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.415+0000 7fa91a000640 1 --2- 172.21.15.27:0/2841975779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa91c0cf460 0x7fa91c14d0e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa90401abf0 tx=0x7fa9040192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.415+0000 7fa912a00640 1 -- 172.21.15.27:0/2841975779 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9040bc680 con 0x7fa91c0cf460 2024-01-15T18:05:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.415+0000 7fa912a00640 1 -- 172.21.15.27:0/2841975779 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9040b5070 con 0x7fa91c0cf460 2024-01-15T18:05:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.415+0000 7fa921c00640 1 -- 172.21.15.27:0/2841975779 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9040094d0 con 0x7fa91c0cf460 2024-01-15T18:05:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.415+0000 7fa912a00640 1 -- 172.21.15.27:0/2841975779 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9040bca20 con 0x7fa91c0cf460 2024-01-15T18:05:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.415+0000 7fa921c00640 1 -- 172.21.15.27:0/2841975779 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa91c06f730 con 0x7fa91c0cf460 2024-01-15T18:05:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.415+0000 7fa921c00640 1 -- 172.21.15.27:0/2841975779 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa91c06f920 con 0x7fa91c0cf460 2024-01-15T18:05:32.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.415+0000 7fa912a00640 1 -- 172.21.15.27:0/2841975779 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fa9040bcb80 con 0x7fa91c0cf460 2024-01-15T18:05:32.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.419+0000 7fa912a00640 1 --2- 172.21.15.27:0/2841975779 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa8f40668f0 0x7fa8f4068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.419+0000 7fa912a00640 1 -- 172.21.15.27:0/2841975779 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9040b7070 con 0x7fa91c0cf460 2024-01-15T18:05:32.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.419+0000 7fa91aa00640 1 --2- 172.21.15.27:0/2841975779 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa8f40668f0 0x7fa8f4068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.419+0000 7fa91aa00640 1 --2- 172.21.15.27:0/2841975779 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa8f40668f0 0x7fa8f4068db0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa90c007fd0 tx=0x7fa90c007d60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:32.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.423+0000 7fa912a00640 1 -- 172.21.15.27:0/2841975779 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa9040ebe20 con 0x7fa91c0cf460 2024-01-15T18:05:32.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.579+0000 7fa921c00640 1 -- 172.21.15.27:0/2841975779 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa91c04e850 con 0x7fa91c0cf460 2024-01-15T18:05:32.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.579+0000 7fa912a00640 1 -- 172.21.15.27:0/2841975779 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa9040a6030 con 0x7fa91c0cf460 2024-01-15T18:05:32.583 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-15T18:05:32.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.579+0000 7fa921c00640 1 -- 172.21.15.27:0/2841975779 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa8f40668f0 msgr2=0x7fa8f4068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.579+0000 7fa921c00640 1 --2- 172.21.15.27:0/2841975779 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa8f40668f0 0x7fa8f4068db0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa90c007fd0 tx=0x7fa90c007d60 comp rx=0 tx=0).stop 2024-01-15T18:05:32.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.579+0000 7fa921c00640 1 -- 172.21.15.27:0/2841975779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa91c0cf460 msgr2=0x7fa91c14d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.579+0000 7fa921c00640 1 --2- 172.21.15.27:0/2841975779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa91c0cf460 0x7fa91c14d0e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa90401abf0 tx=0x7fa9040192e0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.579+0000 7fa921c00640 1 -- 172.21.15.27:0/2841975779 shutdown_connections 2024-01-15T18:05:32.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.579+0000 7fa921c00640 1 --2- 172.21.15.27:0/2841975779 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa8f40668f0 0x7fa8f4068db0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.579+0000 7fa921c00640 1 --2- 172.21.15.27:0/2841975779 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa91c15b4b0 0x7fa91c0c84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.579+0000 7fa921c00640 1 --2- 172.21.15.27:0/2841975779 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa91c0cf460 0x7fa91c14d0e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.579+0000 7fa921c00640 1 --2- 172.21.15.27:0/2841975779 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa91c0cec60 0x7fa91c0ca560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.579+0000 7fa921c00640 1 -- 172.21.15.27:0/2841975779 >> 172.21.15.27:0/2841975779 conn(0x7fa91c0d0e80 msgr2=0x7fa91c0b1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:32.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.579+0000 7fa921c00640 1 -- 172.21.15.27:0/2841975779 shutdown_connections 2024-01-15T18:05:32.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.579+0000 7fa921c00640 1 -- 172.21.15.27:0/2841975779 wait complete. 2024-01-15T18:05:32.605 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T18:05:32.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.743+0000 7fa121c00640 1 Processor -- start 2024-01-15T18:05:32.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.743+0000 7fa121c00640 1 -- start start 2024-01-15T18:05:32.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.747+0000 7fa121c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa11c0cc2c0 0x7fa11c06a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.747+0000 7fa121c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa11c15bb90 0x7fa11c0b4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.747+0000 7fa121c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11c06ae90 con 0x7fa11c06aa30 2024-01-15T18:05:32.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.747+0000 7fa121c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa11c0cc6a0 con 0x7fa11c0cc2c0 2024-01-15T18:05:32.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.747+0000 7fa121c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa11c13e230 con 0x7fa11c15bb90 2024-01-15T18:05:32.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.747+0000 7fa11aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa11c0cc2c0 0x7fa11c06a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.747+0000 7fa11aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa11c0cc2c0 0x7fa11c06a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40800/0 (socket says 172.21.15.27:40800) 2024-01-15T18:05:32.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.747+0000 7fa11aa00640 1 -- 172.21.15.27:0/1752815335 learned_addr learned my addr 172.21.15.27:0/1752815335 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:32.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.747+0000 7fa11b400640 1 --2- 172.21.15.27:0/1752815335 >> v2:172.21.15.93:3300/0 conn(0x7fa11c15bb90 0x7fa11c0b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.747+0000 7fa11aa00640 1 -- 172.21.15.27:0/1752815335 >> v2:172.21.15.93:3300/0 conn(0x7fa11c15bb90 msgr2=0x7fa11c0b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:32.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.747+0000 7fa11aa00640 1 --2- 172.21.15.27:0/1752815335 >> v2:172.21.15.93:3300/0 conn(0x7fa11c15bb90 0x7fa11c0b4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.747+0000 7fa11aa00640 1 -- 172.21.15.27:0/1752815335 >> v1:172.21.15.93:6789/0 conn(0x7fa11c06aa30 legacy=0x7fa11c15b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.747+0000 7fa11aa00640 1 -- 172.21.15.27:0/1752815335 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11c15bfe0 con 0x7fa11c0cc2c0 2024-01-15T18:05:32.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.747+0000 7fa11aa00640 1 --2- 172.21.15.27:0/1752815335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa11c0cc2c0 0x7fa11c06a4f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0b2b20 tx=0x7fa10c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7219fe0f48871c44 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:32.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.751+0000 7fa119600640 1 -- 172.21.15.27:0/1752815335 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10c0ab5f0 con 0x7fa11c0cc2c0 2024-01-15T18:05:32.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.751+0000 7fa119600640 1 -- 172.21.15.27:0/1752815335 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10c0ab750 con 0x7fa11c0cc2c0 2024-01-15T18:05:32.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.751+0000 7fa119600640 1 -- 172.21.15.27:0/1752815335 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10c0abae0 con 0x7fa11c0cc2c0 2024-01-15T18:05:32.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.751+0000 7fa121c00640 1 -- 172.21.15.27:0/1752815335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa11c0cc2c0 msgr2=0x7fa11c06a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.751+0000 7fa121c00640 1 --2- 172.21.15.27:0/1752815335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa11c0cc2c0 0x7fa11c06a4f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0b2b20 tx=0x7fa10c01c500 comp rx=0 tx=0).stop 2024-01-15T18:05:32.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.751+0000 7fa121c00640 1 -- 172.21.15.27:0/1752815335 shutdown_connections 2024-01-15T18:05:32.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.751+0000 7fa121c00640 1 --2- 172.21.15.27:0/1752815335 >> v2:172.21.15.93:3300/0 conn(0x7fa11c15bb90 0x7fa11c0b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.751+0000 7fa121c00640 1 --2- 172.21.15.27:0/1752815335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa11c0cc2c0 0x7fa11c06a4f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.751+0000 7fa121c00640 1 -- 172.21.15.27:0/1752815335 >> 172.21.15.27:0/1752815335 conn(0x7fa11c0cce10 msgr2=0x7fa11c0b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:32.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.751+0000 7fa121c00640 1 -- 172.21.15.27:0/1752815335 shutdown_connections 2024-01-15T18:05:32.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.751+0000 7fa121c00640 1 -- 172.21.15.27:0/1752815335 wait complete. 2024-01-15T18:05:32.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.751+0000 7fa121c00640 1 Processor -- start 2024-01-15T18:05:32.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.751+0000 7fa121c00640 1 -- start start 2024-01-15T18:05:32.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.751+0000 7fa121c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa11c06aa30 0x7fa11c138130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.751+0000 7fa121c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa11c0cc2c0 0x7fa11c1513b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.751+0000 7fa121c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa11c15bb90 0x7fa11c13d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.751+0000 7fa121c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c13ba50 con 0x7fa11c0cc2c0 2024-01-15T18:05:32.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.751+0000 7fa121c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c13f800 con 0x7fa11c06aa30 2024-01-15T18:05:32.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.751+0000 7fa121c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c13fa50 con 0x7fa11c15bb90 2024-01-15T18:05:32.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.755+0000 7fa11a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa11c0cc2c0 0x7fa11c1513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.755+0000 7fa11a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa11c0cc2c0 0x7fa11c1513b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40806/0 (socket says 172.21.15.27:40806) 2024-01-15T18:05:32.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.755+0000 7fa11a000640 1 -- 172.21.15.27:0/896441645 learned_addr learned my addr 172.21.15.27:0/896441645 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:32.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.755+0000 7fa11b400640 1 --2- 172.21.15.27:0/896441645 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa11c15bb90 0x7fa11c13d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.755+0000 7fa11aa00640 1 --2- 172.21.15.27:0/896441645 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa11c06aa30 0x7fa11c138130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.755+0000 7fa11a000640 1 -- 172.21.15.27:0/896441645 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa11c15bb90 msgr2=0x7fa11c13d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.755+0000 7fa11a000640 1 --2- 172.21.15.27:0/896441645 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa11c15bb90 0x7fa11c13d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.755+0000 7fa11a000640 1 -- 172.21.15.27:0/896441645 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa11c06aa30 msgr2=0x7fa11c138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.755+0000 7fa11a000640 1 --2- 172.21.15.27:0/896441645 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa11c06aa30 0x7fa11c138130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.755+0000 7fa11a000640 1 -- 172.21.15.27:0/896441645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa10c0094d0 con 0x7fa11c0cc2c0 2024-01-15T18:05:32.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.755+0000 7fa11b400640 1 --2- 172.21.15.27:0/896441645 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa11c15bb90 0x7fa11c13d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:32.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.755+0000 7fa11a000640 1 --2- 172.21.15.27:0/896441645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa11c0cc2c0 0x7fa11c1513b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa10400c3c0 tx=0x7fa10400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:32.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.755+0000 7fa112a00640 1 -- 172.21.15.27:0/896441645 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1040052d0 con 0x7fa11c0cc2c0 2024-01-15T18:05:32.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.755+0000 7fa112a00640 1 -- 172.21.15.27:0/896441645 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa104005430 con 0x7fa11c0cc2c0 2024-01-15T18:05:32.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.755+0000 7fa112a00640 1 -- 172.21.15.27:0/896441645 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa104003840 con 0x7fa11c0cc2c0 2024-01-15T18:05:32.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.755+0000 7fa121c00640 1 -- 172.21.15.27:0/896441645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa11c13fd30 con 0x7fa11c0cc2c0 2024-01-15T18:05:32.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.755+0000 7fa121c00640 1 -- 172.21.15.27:0/896441645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa11c1402f0 con 0x7fa11c0cc2c0 2024-01-15T18:05:32.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.755+0000 7fa121c00640 1 -- 172.21.15.27:0/896441645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa11c04e850 con 0x7fa11c0cc2c0 2024-01-15T18:05:32.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.755+0000 7fa112a00640 1 -- 172.21.15.27:0/896441645 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fa1040039a0 con 0x7fa11c0cc2c0 2024-01-15T18:05:32.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.759+0000 7fa112a00640 1 --2- 172.21.15.27:0/896441645 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0f40668f0 0x7fa0f4068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.759+0000 7fa112a00640 1 -- 172.21.15.27:0/896441645 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1040838d0 con 0x7fa11c0cc2c0 2024-01-15T18:05:32.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.759+0000 7fa11aa00640 1 --2- 172.21.15.27:0/896441645 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0f40668f0 0x7fa0f4068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.759+0000 7fa11aa00640 1 --2- 172.21.15.27:0/896441645 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0f40668f0 0x7fa0f4068db0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa10c0af000 tx=0x7fa10c0b6040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:32.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.763+0000 7fa112a00640 1 -- 172.21.15.27:0/896441645 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa104050bb0 con 0x7fa11c0cc2c0 2024-01-15T18:05:32.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.919+0000 7fa121c00640 1 -- 172.21.15.27:0/896441645 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa11c06f560 con 0x7fa11c0cc2c0 2024-01-15T18:05:32.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.919+0000 7fa112a00640 1 -- 172.21.15.27:0/896441645 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa1040170a0 con 0x7fa11c0cc2c0 2024-01-15T18:05:32.924 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-15T18:05:32.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.919+0000 7fa121c00640 1 -- 172.21.15.27:0/896441645 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0f40668f0 msgr2=0x7fa0f4068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.919+0000 7fa121c00640 1 --2- 172.21.15.27:0/896441645 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0f40668f0 0x7fa0f4068db0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa10c0af000 tx=0x7fa10c0b6040 comp rx=0 tx=0).stop 2024-01-15T18:05:32.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.919+0000 7fa121c00640 1 -- 172.21.15.27:0/896441645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa11c0cc2c0 msgr2=0x7fa11c1513b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.919+0000 7fa121c00640 1 --2- 172.21.15.27:0/896441645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa11c0cc2c0 0x7fa11c1513b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa10400c3c0 tx=0x7fa10400c880 comp rx=0 tx=0).stop 2024-01-15T18:05:32.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.919+0000 7fa121c00640 1 -- 172.21.15.27:0/896441645 shutdown_connections 2024-01-15T18:05:32.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.919+0000 7fa121c00640 1 --2- 172.21.15.27:0/896441645 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0f40668f0 0x7fa0f4068db0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.919+0000 7fa121c00640 1 --2- 172.21.15.27:0/896441645 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa11c15bb90 0x7fa11c13d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.919+0000 7fa121c00640 1 --2- 172.21.15.27:0/896441645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa11c0cc2c0 0x7fa11c1513b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.919+0000 7fa121c00640 1 --2- 172.21.15.27:0/896441645 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa11c06aa30 0x7fa11c138130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.919+0000 7fa121c00640 1 -- 172.21.15.27:0/896441645 >> 172.21.15.27:0/896441645 conn(0x7fa11c0cce10 msgr2=0x7fa11c1598b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:32.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.923+0000 7fa121c00640 1 -- 172.21.15.27:0/896441645 shutdown_connections 2024-01-15T18:05:32.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:32.923+0000 7fa121c00640 1 -- 172.21.15.27:0/896441645 wait complete. 2024-01-15T18:05:32.947 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:05:33.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.087+0000 7f5fef800640 1 Processor -- start 2024-01-15T18:05:33.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.087+0000 7f5fef800640 1 -- start start 2024-01-15T18:05:33.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.087+0000 7f5fef800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5fe815d7c0 0x7f5fe80b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.087+0000 7f5fef800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f5fe80b1d90 0x7f5fe8099a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.087+0000 7f5fef800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe80b1aa0 con 0x7f5fe806bc70 2024-01-15T18:05:33.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.087+0000 7f5fef800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fe80b1300 con 0x7f5fe815d7c0 2024-01-15T18:05:33.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.087+0000 7f5fef800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fe80b25a0 con 0x7f5fe80b1d90 2024-01-15T18:05:33.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.087+0000 7f5fece00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5fe815d7c0 0x7f5fe80b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.087+0000 7f5fece00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5fe815d7c0 0x7f5fe80b35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40808/0 (socket says 172.21.15.27:40808) 2024-01-15T18:05:33.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.087+0000 7f5fece00640 1 -- 172.21.15.27:0/670774183 learned_addr learned my addr 172.21.15.27:0/670774183 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:33.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.087+0000 7f5fe7e00640 1 --2- 172.21.15.27:0/670774183 >> v2:172.21.15.93:3300/0 conn(0x7f5fe80b1d90 0x7f5fe8099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.087+0000 7f5fece00640 1 -- 172.21.15.27:0/670774183 >> v2:172.21.15.93:3300/0 conn(0x7f5fe80b1d90 msgr2=0x7f5fe8099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:33.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.087+0000 7f5fece00640 1 --2- 172.21.15.27:0/670774183 >> v2:172.21.15.93:3300/0 conn(0x7f5fe80b1d90 0x7f5fe8099a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.087+0000 7f5fece00640 1 -- 172.21.15.27:0/670774183 >> v1:172.21.15.93:6789/0 conn(0x7f5fe806bc70 legacy=0x7f5fe8161140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:33.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.087+0000 7f5fece00640 1 -- 172.21.15.27:0/670774183 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe80b14e0 con 0x7f5fe815d7c0 2024-01-15T18:05:33.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.087+0000 7f5fe7e00640 1 --2- 172.21.15.27:0/670774183 >> v2:172.21.15.93:3300/0 conn(0x7f5fe80b1d90 0x7f5fe8099a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:33.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.091+0000 7f5fece00640 1 --2- 172.21.15.27:0/670774183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fe815d7c0 0x7f5fe80b35a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8073de0 tx=0x7f5fd801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=551ae6d25de5ce3e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:33.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.091+0000 7f5fe7400640 1 -- 172.21.15.27:0/670774183 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd80ab600 con 0x7f5fe815d7c0 2024-01-15T18:05:33.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.091+0000 7f5fe7400640 1 -- 172.21.15.27:0/670774183 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd80a7050 con 0x7f5fe815d7c0 2024-01-15T18:05:33.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.091+0000 7f5fe7400640 1 -- 172.21.15.27:0/670774183 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd80ab980 con 0x7f5fe815d7c0 2024-01-15T18:05:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.091+0000 7f5fef800640 1 -- 172.21.15.27:0/670774183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fe815d7c0 msgr2=0x7f5fe80b35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.091+0000 7f5fef800640 1 --2- 172.21.15.27:0/670774183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fe815d7c0 0x7f5fe80b35a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5fe8073de0 tx=0x7f5fd801c3a0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.091+0000 7f5fef800640 1 -- 172.21.15.27:0/670774183 shutdown_connections 2024-01-15T18:05:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.091+0000 7f5fef800640 1 --2- 172.21.15.27:0/670774183 >> v2:172.21.15.93:3300/0 conn(0x7f5fe80b1d90 0x7f5fe8099a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.091+0000 7f5fef800640 1 --2- 172.21.15.27:0/670774183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fe815d7c0 0x7f5fe80b35a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.091+0000 7f5fef800640 1 -- 172.21.15.27:0/670774183 >> 172.21.15.27:0/670774183 conn(0x7f5fe80b2a80 msgr2=0x7f5fe80b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.091+0000 7f5fef800640 1 -- 172.21.15.27:0/670774183 shutdown_connections 2024-01-15T18:05:33.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.091+0000 7f5fef800640 1 -- 172.21.15.27:0/670774183 wait complete. 2024-01-15T18:05:33.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.095+0000 7f5fef800640 1 Processor -- start 2024-01-15T18:05:33.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.095+0000 7f5fef800640 1 -- start start 2024-01-15T18:05:33.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.095+0000 7f5fef800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fe806bc70 0x7f5fe81605f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.095+0000 7f5fef800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5fe80b1d90 0x7f5fe8155490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.095+0000 7f5fef800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5fe815d7c0 0x7f5fe815e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.095+0000 7f5fef800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe815f010 con 0x7f5fe806bc70 2024-01-15T18:05:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.095+0000 7f5fef800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe8160b30 con 0x7f5fe815d7c0 2024-01-15T18:05:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.095+0000 7f5fef800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe815f830 con 0x7f5fe80b1d90 2024-01-15T18:05:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.095+0000 7f5fece00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fe806bc70 0x7f5fe81605f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.095+0000 7f5fece00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fe806bc70 0x7f5fe81605f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40822/0 (socket says 172.21.15.27:40822) 2024-01-15T18:05:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.095+0000 7f5fece00640 1 -- 172.21.15.27:0/164588463 learned_addr learned my addr 172.21.15.27:0/164588463 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.095+0000 7f5fed800640 1 --2- 172.21.15.27:0/164588463 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5fe815d7c0 0x7f5fe815e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.095+0000 7f5fe7e00640 1 --2- 172.21.15.27:0/164588463 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5fe80b1d90 0x7f5fe8155490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.095+0000 7f5fece00640 1 -- 172.21.15.27:0/164588463 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5fe80b1d90 msgr2=0x7f5fe8155490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.095+0000 7f5fece00640 1 --2- 172.21.15.27:0/164588463 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5fe80b1d90 0x7f5fe8155490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.095+0000 7f5fece00640 1 -- 172.21.15.27:0/164588463 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5fe815d7c0 msgr2=0x7f5fe815e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.095+0000 7f5fece00640 1 --2- 172.21.15.27:0/164588463 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5fe815d7c0 0x7f5fe815e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.095+0000 7f5fece00640 1 -- 172.21.15.27:0/164588463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd80094d0 con 0x7f5fe806bc70 2024-01-15T18:05:33.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.095+0000 7f5fece00640 1 --2- 172.21.15.27:0/164588463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fe806bc70 0x7f5fe81605f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5fd801aa50 tx=0x7f5fd8019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:33.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.095+0000 7f5fe4c00640 1 -- 172.21.15.27:0/164588463 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd80b3890 con 0x7f5fe806bc70 2024-01-15T18:05:33.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.095+0000 7f5fe4c00640 1 -- 172.21.15.27:0/164588463 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd80b3a10 con 0x7f5fe806bc70 2024-01-15T18:05:33.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.095+0000 7f5fe4c00640 1 -- 172.21.15.27:0/164588463 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd80b3dc0 con 0x7f5fe806bc70 2024-01-15T18:05:33.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.095+0000 7f5fef800640 1 -- 172.21.15.27:0/164588463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe815fb10 con 0x7f5fe806bc70 2024-01-15T18:05:33.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.095+0000 7f5fef800640 1 -- 172.21.15.27:0/164588463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe81600d0 con 0x7f5fe806bc70 2024-01-15T18:05:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.095+0000 7f5fef800640 1 -- 172.21.15.27:0/164588463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe804e850 con 0x7f5fe806bc70 2024-01-15T18:05:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.099+0000 7f5fe4c00640 1 -- 172.21.15.27:0/164588463 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f5fd80b2070 con 0x7f5fe806bc70 2024-01-15T18:05:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.099+0000 7f5fe4c00640 1 --2- 172.21.15.27:0/164588463 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5fc00668f0 0x7f5fc0068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.099+0000 7f5fe4c00640 1 -- 172.21.15.27:0/164588463 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5fd811e740 con 0x7f5fe806bc70 2024-01-15T18:05:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.099+0000 7f5fe7e00640 1 --2- 172.21.15.27:0/164588463 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5fc00668f0 0x7f5fc0068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.099+0000 7f5fe7e00640 1 --2- 172.21.15.27:0/164588463 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5fc00668f0 0x7f5fc0068db0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5fe815f450 tx=0x7f5fd0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:33.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.103+0000 7f5fe4c00640 1 -- 172.21.15.27:0/164588463 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5fd80eb990 con 0x7f5fe806bc70 2024-01-15T18:05:33.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.271+0000 7f5fef800640 1 -- 172.21.15.27:0/164588463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5fe80b6ed0 con 0x7f5fe806bc70 2024-01-15T18:05:33.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.271+0000 7f5fe4c00640 1 -- 172.21.15.27:0/164588463 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f5fd80eefb0 con 0x7f5fe806bc70 2024-01-15T18:05:33.275 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-01-15T18:05:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.271+0000 7f5fef800640 1 -- 172.21.15.27:0/164588463 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5fc00668f0 msgr2=0x7f5fc0068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.271+0000 7f5fef800640 1 --2- 172.21.15.27:0/164588463 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5fc00668f0 0x7f5fc0068db0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5fe815f450 tx=0x7f5fd0008040 comp rx=0 tx=0).stop 2024-01-15T18:05:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.271+0000 7f5fef800640 1 -- 172.21.15.27:0/164588463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fe806bc70 msgr2=0x7f5fe81605f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.271+0000 7f5fef800640 1 --2- 172.21.15.27:0/164588463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fe806bc70 0x7f5fe81605f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5fd801aa50 tx=0x7f5fd8019140 comp rx=0 tx=0).stop 2024-01-15T18:05:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.271+0000 7f5fef800640 1 -- 172.21.15.27:0/164588463 shutdown_connections 2024-01-15T18:05:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.271+0000 7f5fef800640 1 --2- 172.21.15.27:0/164588463 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5fc00668f0 0x7f5fc0068db0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.271+0000 7f5fef800640 1 --2- 172.21.15.27:0/164588463 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5fe815d7c0 0x7f5fe815e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.271+0000 7f5fef800640 1 --2- 172.21.15.27:0/164588463 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5fe80b1d90 0x7f5fe8155490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.271+0000 7f5fef800640 1 --2- 172.21.15.27:0/164588463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5fe806bc70 0x7f5fe81605f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.271+0000 7f5fef800640 1 -- 172.21.15.27:0/164588463 >> 172.21.15.27:0/164588463 conn(0x7f5fe80b2a80 msgr2=0x7f5fe80733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:33.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.271+0000 7f5fef800640 1 -- 172.21.15.27:0/164588463 shutdown_connections 2024-01-15T18:05:33.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.271+0000 7f5fef800640 1 -- 172.21.15.27:0/164588463 wait complete. 2024-01-15T18:05:33.298 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:05:33.298 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:05:33.308 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:33.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.491+0000 7f88ac400640 1 Processor -- start 2024-01-15T18:05:33.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.491+0000 7f88ac400640 1 -- start start 2024-01-15T18:05:33.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.491+0000 7f88ac400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f88a406ac00 0x7f88a406dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.491+0000 7f88ac400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f88a40cf0c0 0x7f88a40acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.491+0000 7f88ac400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a415d6c0 con 0x7f88a415d2e0 2024-01-15T18:05:33.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.491+0000 7f88ac400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88a406e310 con 0x7f88a406ac00 2024-01-15T18:05:33.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.491+0000 7f88ac400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88a406e5f0 con 0x7f88a40cf0c0 2024-01-15T18:05:33.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.491+0000 7f88a9a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f88a415d2e0 0x7f88a406a4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34854/0 (socket says 172.21.15.27:34854) 2024-01-15T18:05:33.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.491+0000 7f88a9a00640 1 -- 172.21.15.27:0/3847711949 learned_addr learned my addr 172.21.15.27:0/3847711949 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:33.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.491+0000 7f88a9000640 1 --2- 172.21.15.27:0/3847711949 >> v2:172.21.15.27:3300/0 conn(0x7f88a406ac00 0x7f88a406dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.491+0000 7f88aa400640 1 --2- 172.21.15.27:0/3847711949 >> v2:172.21.15.93:3300/0 conn(0x7f88a40cf0c0 0x7f88a40acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.491+0000 7f88a9000640 1 -- 172.21.15.27:0/3847711949 >> v2:172.21.15.93:3300/0 conn(0x7f88a40cf0c0 msgr2=0x7f88a40acdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:33.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.491+0000 7f88a9000640 1 --2- 172.21.15.27:0/3847711949 >> v2:172.21.15.93:3300/0 conn(0x7f88a40cf0c0 0x7f88a40acdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.491+0000 7f88a9000640 1 -- 172.21.15.27:0/3847711949 >> v1:172.21.15.27:6789/0 conn(0x7f88a415d2e0 legacy=0x7f88a406a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.491+0000 7f88a9000640 1 -- 172.21.15.27:0/3847711949 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a406e8d0 con 0x7f88a406ac00 2024-01-15T18:05:33.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.495+0000 7f88a9000640 1 --2- 172.21.15.27:0/3847711949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88a406ac00 0x7f88a406dce0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f88a40d0f00 tx=0x7f888c01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e41db81125e2fa1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:33.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.495+0000 7f88a3e00640 1 -- 172.21.15.27:0/3847711949 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888c0a7070 con 0x7f88a406ac00 2024-01-15T18:05:33.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.495+0000 7f88a3e00640 1 -- 172.21.15.27:0/3847711949 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f888c0ab430 con 0x7f88a406ac00 2024-01-15T18:05:33.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.495+0000 7f88a3e00640 1 -- 172.21.15.27:0/3847711949 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888c0ab7c0 con 0x7f88a406ac00 2024-01-15T18:05:33.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.495+0000 7f88ac400640 1 -- 172.21.15.27:0/3847711949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88a406ac00 msgr2=0x7f88a406dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.495+0000 7f88ac400640 1 --2- 172.21.15.27:0/3847711949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88a406ac00 0x7f88a406dce0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f88a40d0f00 tx=0x7f888c01c430 comp rx=0 tx=0).stop 2024-01-15T18:05:33.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.495+0000 7f88ac400640 1 -- 172.21.15.27:0/3847711949 shutdown_connections 2024-01-15T18:05:33.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.495+0000 7f88ac400640 1 --2- 172.21.15.27:0/3847711949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88a406ac00 0x7f88a406dce0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.495+0000 7f88ac400640 1 --2- 172.21.15.27:0/3847711949 >> v2:172.21.15.93:3300/0 conn(0x7f88a40cf0c0 0x7f88a40acdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.495+0000 7f88ac400640 1 -- 172.21.15.27:0/3847711949 >> 172.21.15.27:0/3847711949 conn(0x7f88a40b1ae0 msgr2=0x7f88a40b1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:33.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.495+0000 7f88ac400640 1 -- 172.21.15.27:0/3847711949 shutdown_connections 2024-01-15T18:05:33.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.495+0000 7f88ac400640 1 -- 172.21.15.27:0/3847711949 wait complete. 2024-01-15T18:05:33.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.499+0000 7f88ac400640 1 Processor -- start 2024-01-15T18:05:33.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.499+0000 7f88ac400640 1 -- start start 2024-01-15T18:05:33.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.499+0000 7f88ac400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88a406ac00 0x7f88a4150920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.499+0000 7f88ac400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88a40cf0c0 0x7f88a41500f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.499+0000 7f88ac400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88a415d2e0 0x7f88a4158370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.499+0000 7f88ac400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a4157350 con 0x7f88a40cf0c0 2024-01-15T18:05:33.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.499+0000 7f88ac400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a4150e60 con 0x7f88a406ac00 2024-01-15T18:05:33.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.499+0000 7f88ac400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a41575a0 con 0x7f88a415d2e0 2024-01-15T18:05:33.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.499+0000 7f88a9000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88a40cf0c0 0x7f88a41500f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.499+0000 7f88a9000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88a40cf0c0 0x7f88a41500f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40840/0 (socket says 172.21.15.27:40840) 2024-01-15T18:05:33.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.499+0000 7f88a9000640 1 -- 172.21.15.27:0/1624397338 learned_addr learned my addr 172.21.15.27:0/1624397338 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:33.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.499+0000 7f88aa400640 1 --2- 172.21.15.27:0/1624397338 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88a415d2e0 0x7f88a4158370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.499+0000 7f88a9a00640 1 --2- 172.21.15.27:0/1624397338 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88a406ac00 0x7f88a4150920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.499+0000 7f88a9000640 1 -- 172.21.15.27:0/1624397338 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88a415d2e0 msgr2=0x7f88a4158370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.499+0000 7f88a9000640 1 --2- 172.21.15.27:0/1624397338 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88a415d2e0 0x7f88a4158370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.499+0000 7f88a9000640 1 -- 172.21.15.27:0/1624397338 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88a406ac00 msgr2=0x7f88a4150920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.499+0000 7f88a9000640 1 --2- 172.21.15.27:0/1624397338 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88a406ac00 0x7f88a4150920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.499+0000 7f88a9000640 1 -- 172.21.15.27:0/1624397338 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888c009420 con 0x7f88a40cf0c0 2024-01-15T18:05:33.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.499+0000 7f88a9a00640 1 --2- 172.21.15.27:0/1624397338 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88a406ac00 0x7f88a4150920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:33.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.499+0000 7f88a9000640 1 --2- 172.21.15.27:0/1624397338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88a40cf0c0 0x7f88a41500f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f888c019040 tx=0x7f888c01ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:33.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.499+0000 7f88a1600640 1 -- 172.21.15.27:0/1624397338 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888c0b5810 con 0x7f88a40cf0c0 2024-01-15T18:05:33.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.499+0000 7f88a1600640 1 -- 172.21.15.27:0/1624397338 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f888c0b5990 con 0x7f88a40cf0c0 2024-01-15T18:05:33.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.499+0000 7f88a1600640 1 -- 172.21.15.27:0/1624397338 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888c0b5d40 con 0x7f88a40cf0c0 2024-01-15T18:05:33.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.499+0000 7f88ac400640 1 -- 172.21.15.27:0/1624397338 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a4157880 con 0x7f88a40cf0c0 2024-01-15T18:05:33.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.499+0000 7f88ac400640 1 -- 172.21.15.27:0/1624397338 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a40b2b10 con 0x7f88a40cf0c0 2024-01-15T18:05:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.503+0000 7f88ac400640 1 -- 172.21.15.27:0/1624397338 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a4067a10 con 0x7f88a40cf0c0 2024-01-15T18:05:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.503+0000 7f88a1600640 1 -- 172.21.15.27:0/1624397338 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f888c0b4390 con 0x7f88a40cf0c0 2024-01-15T18:05:33.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.503+0000 7f88a1600640 1 --2- 172.21.15.27:0/1624397338 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f887c0668f0 0x7f887c068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.503+0000 7f88a1600640 1 -- 172.21.15.27:0/1624397338 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f888c11ed10 con 0x7f88a40cf0c0 2024-01-15T18:05:33.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.503+0000 7f88a9a00640 1 --2- 172.21.15.27:0/1624397338 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f887c0668f0 0x7f887c068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.503+0000 7f88a9a00640 1 --2- 172.21.15.27:0/1624397338 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f887c0668f0 0x7f887c068db0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8894005890 tx=0x7f8894009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:33.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.507+0000 7f88a1600640 1 -- 172.21.15.27:0/1624397338 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f888c0ec080 con 0x7f88a40cf0c0 2024-01-15T18:05:33.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.667+0000 7f88ac400640 1 -- 172.21.15.27:0/1624397338 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88a404e850 con 0x7f88a40cf0c0 2024-01-15T18:05:33.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.671+0000 7f88a1600640 1 -- 172.21.15.27:0/1624397338 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16409 (secure 0 0 0) 0x7f888c006e20 con 0x7f88a40cf0c0 2024-01-15T18:05:33.675 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:33.676 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:04:59.837052+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:04:56.220608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T18:04:57.210421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:33.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.671+0000 7f88ac400640 1 -- 172.21.15.27:0/1624397338 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f887c0668f0 msgr2=0x7f887c068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.671+0000 7f88ac400640 1 --2- 172.21.15.27:0/1624397338 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f887c0668f0 0x7f887c068db0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8894005890 tx=0x7f8894009040 comp rx=0 tx=0).stop 2024-01-15T18:05:33.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.671+0000 7f88ac400640 1 -- 172.21.15.27:0/1624397338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88a40cf0c0 msgr2=0x7f88a41500f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.671+0000 7f88ac400640 1 --2- 172.21.15.27:0/1624397338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88a40cf0c0 0x7f88a41500f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f888c019040 tx=0x7f888c01ac70 comp rx=0 tx=0).stop 2024-01-15T18:05:33.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.671+0000 7f88ac400640 1 -- 172.21.15.27:0/1624397338 shutdown_connections 2024-01-15T18:05:33.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.671+0000 7f88ac400640 1 --2- 172.21.15.27:0/1624397338 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f887c0668f0 0x7f887c068db0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.671+0000 7f88ac400640 1 --2- 172.21.15.27:0/1624397338 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88a415d2e0 0x7f88a4158370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.671+0000 7f88ac400640 1 --2- 172.21.15.27:0/1624397338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88a40cf0c0 0x7f88a41500f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.671+0000 7f88ac400640 1 --2- 172.21.15.27:0/1624397338 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88a406ac00 0x7f88a4150920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.671+0000 7f88ac400640 1 -- 172.21.15.27:0/1624397338 >> 172.21.15.27:0/1624397338 conn(0x7f88a40b1ae0 msgr2=0x7f88a41511e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:33.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.671+0000 7f88ac400640 1 -- 172.21.15.27:0/1624397338 shutdown_connections 2024-01-15T18:05:33.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.671+0000 7f88ac400640 1 -- 172.21.15.27:0/1624397338 wait complete. 2024-01-15T18:05:33.698 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T18:04:56.220608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T18:04:57.210421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:05:33.698 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:05:33.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.839+0000 7fb0a6000640 1 Processor -- start 2024-01-15T18:05:33.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.839+0000 7fb0a6000640 1 -- start start 2024-01-15T18:05:33.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.839+0000 7fb0a6000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb0a009a5b0 0x7fb0a00cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.839+0000 7fb0a6000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a00ccd50 con 0x7fb0a0099a10 2024-01-15T18:05:33.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.839+0000 7fb0a6000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a00cebc0 con 0x7fb0a015b350 2024-01-15T18:05:33.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.839+0000 7fb0a6000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0a00ceea0 con 0x7fb0a009a5b0 2024-01-15T18:05:33.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.839+0000 7fb09ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb0a009a5b0 0x7fb0a00cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.839+0000 7fb09fe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb0a0099a10 0x7fb0a0099e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35072/0 (socket says 172.21.15.27:35072) 2024-01-15T18:05:33.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.839+0000 7fb09fe00640 1 -- 172.21.15.27:0/3301346253 learned_addr learned my addr 172.21.15.27:0/3301346253 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:33.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.839+0000 7fb09ea00640 1 -- 172.21.15.27:0/3301346253 >> v1:172.21.15.93:6789/0 conn(0x7fb0a015b350 legacy=0x7fb0a00cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:33.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.839+0000 7fb09ea00640 1 -- 172.21.15.27:0/3301346253 >> v1:172.21.15.27:6789/0 conn(0x7fb0a0099a10 legacy=0x7fb0a0099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.839+0000 7fb09ea00640 1 -- 172.21.15.27:0/3301346253 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a00cf180 con 0x7fb0a009a5b0 2024-01-15T18:05:33.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.843+0000 7fb09ea00640 1 --2- 172.21.15.27:0/3301346253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0a009a5b0 0x7fb0a00cc500 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb0880097c0 tx=0x7fb08801c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3db3795aacfb67e5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:33.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.843+0000 7fb09e000640 1 -- 172.21.15.27:0/3301346253 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0880ab600 con 0x7fb0a009a5b0 2024-01-15T18:05:33.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.843+0000 7fb09e000640 1 -- 172.21.15.27:0/3301346253 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0880a7050 con 0x7fb0a009a5b0 2024-01-15T18:05:33.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.843+0000 7fb09e000640 1 -- 172.21.15.27:0/3301346253 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0880ab980 con 0x7fb0a009a5b0 2024-01-15T18:05:33.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.843+0000 7fb0a6000640 1 -- 172.21.15.27:0/3301346253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0a009a5b0 msgr2=0x7fb0a00cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.843+0000 7fb0a6000640 1 --2- 172.21.15.27:0/3301346253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0a009a5b0 0x7fb0a00cc500 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb0880097c0 tx=0x7fb08801c380 comp rx=0 tx=0).stop 2024-01-15T18:05:33.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.843+0000 7fb0a6000640 1 -- 172.21.15.27:0/3301346253 shutdown_connections 2024-01-15T18:05:33.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.843+0000 7fb0a6000640 1 --2- 172.21.15.27:0/3301346253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0a009a5b0 0x7fb0a00cc500 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.843+0000 7fb0a6000640 1 -- 172.21.15.27:0/3301346253 >> 172.21.15.27:0/3301346253 conn(0x7fb0a00b2a80 msgr2=0x7fb0a00b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:33.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.843+0000 7fb0a6000640 1 -- 172.21.15.27:0/3301346253 shutdown_connections 2024-01-15T18:05:33.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.843+0000 7fb0a6000640 1 -- 172.21.15.27:0/3301346253 wait complete. 2024-01-15T18:05:33.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.843+0000 7fb0a6000640 1 Processor -- start 2024-01-15T18:05:33.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.847+0000 7fb0a6000640 1 -- start start 2024-01-15T18:05:33.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.847+0000 7fb0a6000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0a0099a10 0x7fb0a006ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.847+0000 7fb0a6000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0a009a5b0 0x7fb0a0153d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.847+0000 7fb0a6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0a015b350 0x7fb0a0151190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.847+0000 7fb0a6000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a006d370 con 0x7fb0a015b350 2024-01-15T18:05:33.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.847+0000 7fb0a6000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a0154290 con 0x7fb0a0099a10 2024-01-15T18:05:33.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.847+0000 7fb0a6000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a01544a0 con 0x7fb0a009a5b0 2024-01-15T18:05:33.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.847+0000 7fb09fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0a015b350 0x7fb0a0151190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.847+0000 7fb09fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0a015b350 0x7fb0a0151190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34852/0 (socket says 172.21.15.27:34852) 2024-01-15T18:05:33.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.847+0000 7fb09fe00640 1 -- 172.21.15.27:0/4002420187 learned_addr learned my addr 172.21.15.27:0/4002420187 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:33.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.847+0000 7fb09ea00640 1 --2- 172.21.15.27:0/4002420187 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0a009a5b0 0x7fb0a0153d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.847+0000 7fb09f400640 1 --2- 172.21.15.27:0/4002420187 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0a0099a10 0x7fb0a006ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.847+0000 7fb09fe00640 1 -- 172.21.15.27:0/4002420187 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0a009a5b0 msgr2=0x7fb0a0153d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.847+0000 7fb09fe00640 1 --2- 172.21.15.27:0/4002420187 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0a009a5b0 0x7fb0a0153d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.847+0000 7fb09fe00640 1 -- 172.21.15.27:0/4002420187 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0a0099a10 msgr2=0x7fb0a006ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.847+0000 7fb09fe00640 1 --2- 172.21.15.27:0/4002420187 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0a0099a10 0x7fb0a006ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.847+0000 7fb09fe00640 1 -- 172.21.15.27:0/4002420187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb088009420 con 0x7fb0a015b350 2024-01-15T18:05:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.847+0000 7fb09fe00640 1 --2- 172.21.15.27:0/4002420187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0a015b350 0x7fb0a0151190 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb09800cbc0 tx=0x7fb09800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.847+0000 7fb097400640 1 -- 172.21.15.27:0/4002420187 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09800ad40 con 0x7fb0a015b350 2024-01-15T18:05:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.847+0000 7fb097400640 1 -- 172.21.15.27:0/4002420187 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb098012400 con 0x7fb0a015b350 2024-01-15T18:05:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.847+0000 7fb097400640 1 -- 172.21.15.27:0/4002420187 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09800aea0 con 0x7fb0a015b350 2024-01-15T18:05:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.847+0000 7fb0a6000640 1 -- 172.21.15.27:0/4002420187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a0154780 con 0x7fb0a015b350 2024-01-15T18:05:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.847+0000 7fb0a6000640 1 -- 172.21.15.27:0/4002420187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a00c58c0 con 0x7fb0a015b350 2024-01-15T18:05:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.847+0000 7fb0a6000640 1 -- 172.21.15.27:0/4002420187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a00d06a0 con 0x7fb0a015b350 2024-01-15T18:05:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.851+0000 7fb097400640 1 -- 172.21.15.27:0/4002420187 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fb098007070 con 0x7fb0a015b350 2024-01-15T18:05:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.851+0000 7fb097400640 1 --2- 172.21.15.27:0/4002420187 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb0780668f0 0x7fb078068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.851+0000 7fb097400640 1 -- 172.21.15.27:0/4002420187 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb0980842d0 con 0x7fb0a015b350 2024-01-15T18:05:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.851+0000 7fb09f400640 1 --2- 172.21.15.27:0/4002420187 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb0780668f0 0x7fb078068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.851+0000 7fb09f400640 1 --2- 172.21.15.27:0/4002420187 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb0780668f0 0x7fb078068db0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb0900098e0 tx=0x7fb090009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:33.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:33.855+0000 7fb097400640 1 -- 172.21.15.27:0/4002420187 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb098051520 con 0x7fb0a015b350 2024-01-15T18:05:34.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.011+0000 7fb0a6000640 1 -- 172.21.15.27:0/4002420187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb0a00c5ab0 con 0x7fb0a015b350 2024-01-15T18:05:34.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.011+0000 7fb097400640 1 -- 172.21.15.27:0/4002420187 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb098054b40 con 0x7fb0a015b350 2024-01-15T18:05:34.018 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:05:34.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.015+0000 7fb0a6000640 1 -- 172.21.15.27:0/4002420187 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb0780668f0 msgr2=0x7fb078068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:34.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.015+0000 7fb0a6000640 1 --2- 172.21.15.27:0/4002420187 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb0780668f0 0x7fb078068db0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb0900098e0 tx=0x7fb090009060 comp rx=0 tx=0).stop 2024-01-15T18:05:34.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.015+0000 7fb0a6000640 1 -- 172.21.15.27:0/4002420187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0a015b350 msgr2=0x7fb0a0151190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:34.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.015+0000 7fb0a6000640 1 --2- 172.21.15.27:0/4002420187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0a015b350 0x7fb0a0151190 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb09800cbc0 tx=0x7fb09800a430 comp rx=0 tx=0).stop 2024-01-15T18:05:34.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.015+0000 7fb0a6000640 1 -- 172.21.15.27:0/4002420187 shutdown_connections 2024-01-15T18:05:34.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.015+0000 7fb0a6000640 1 --2- 172.21.15.27:0/4002420187 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb0780668f0 0x7fb078068db0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.015+0000 7fb0a6000640 1 --2- 172.21.15.27:0/4002420187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0a015b350 0x7fb0a0151190 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.015+0000 7fb0a6000640 1 --2- 172.21.15.27:0/4002420187 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0a009a5b0 0x7fb0a0153d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.015+0000 7fb0a6000640 1 --2- 172.21.15.27:0/4002420187 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0a0099a10 0x7fb0a006ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.015+0000 7fb0a6000640 1 -- 172.21.15.27:0/4002420187 >> 172.21.15.27:0/4002420187 conn(0x7fb0a00b2a80 msgr2=0x7fb0a00cf980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:34.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.015+0000 7fb0a6000640 1 -- 172.21.15.27:0/4002420187 shutdown_connections 2024-01-15T18:05:34.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.015+0000 7fb0a6000640 1 -- 172.21.15.27:0/4002420187 wait complete. 2024-01-15T18:05:34.041 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T18:05:34.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.179+0000 7ff1da600640 1 Processor -- start 2024-01-15T18:05:34.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.179+0000 7ff1da600640 1 -- start start 2024-01-15T18:05:34.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.179+0000 7ff1da600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1d409a5b0 0x7ff1d40cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:34.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.179+0000 7ff1da600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff1d4099a10 0x7ff1d4099e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:34.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.179+0000 7ff1da600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1d40ccd50 con 0x7ff1d415b350 2024-01-15T18:05:34.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.179+0000 7ff1da600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1d409a3c0 con 0x7ff1d4099a10 2024-01-15T18:05:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.179+0000 7ff1da600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d40cebc0 con 0x7ff1d409a5b0 2024-01-15T18:05:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.179+0000 7ff1d2a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1d409a5b0 0x7ff1d40cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.179+0000 7ff1d3e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff1d4099a10 0x7ff1d4099e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.179+0000 7ff1d2a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1d409a5b0 0x7ff1d40cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:43064/0 (socket says 172.21.15.27:43064) 2024-01-15T18:05:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.179+0000 7ff1d3e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff1d4099a10 0x7ff1d4099e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:50860/0 (socket says 172.21.15.27:50860) 2024-01-15T18:05:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.179+0000 7ff1d2a00640 1 -- 172.21.15.27:0/2094463003 learned_addr learned my addr 172.21.15.27:0/2094463003 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:34.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.179+0000 7ff1d3400640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7ff1d415b350 0x7ff1d40cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:59286/0 (socket says 172.21.15.27:59286) 2024-01-15T18:05:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.183+0000 7ff1d2a00640 1 -- 172.21.15.27:0/2094463003 >> v2:172.21.15.93:3300/0 conn(0x7ff1d4099a10 msgr2=0x7ff1d4099e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.183+0000 7ff1d2a00640 1 --2- 172.21.15.27:0/2094463003 >> v2:172.21.15.93:3300/0 conn(0x7ff1d4099a10 0x7ff1d4099e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.183+0000 7ff1d2a00640 1 -- 172.21.15.27:0/2094463003 >> v1:172.21.15.93:6789/0 conn(0x7ff1d415b350 legacy=0x7ff1d40cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:34.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.183+0000 7ff1d2a00640 1 -- 172.21.15.27:0/2094463003 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d40ceea0 con 0x7ff1d409a5b0 2024-01-15T18:05:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.183+0000 7ff1d2a00640 1 --2- 172.21.15.27:0/2094463003 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1d409a5b0 0x7ff1d40cc500 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc009870 tx=0x7ff1bc01c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a7c3ac259fb329a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.187+0000 7ff1d2000640 1 -- 172.21.15.27:0/2094463003 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc0ab680 con 0x7ff1d409a5b0 2024-01-15T18:05:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.187+0000 7ff1d2000640 1 -- 172.21.15.27:0/2094463003 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1bc0ab7e0 con 0x7ff1d409a5b0 2024-01-15T18:05:34.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.187+0000 7ff1d2000640 1 -- 172.21.15.27:0/2094463003 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc0abb70 con 0x7ff1d409a5b0 2024-01-15T18:05:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.187+0000 7ff1da600640 1 -- 172.21.15.27:0/2094463003 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1d409a5b0 msgr2=0x7ff1d40cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.187+0000 7ff1da600640 1 --2- 172.21.15.27:0/2094463003 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1d409a5b0 0x7ff1d40cc500 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc009870 tx=0x7ff1bc01c480 comp rx=0 tx=0).stop 2024-01-15T18:05:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.187+0000 7ff1da600640 1 -- 172.21.15.27:0/2094463003 shutdown_connections 2024-01-15T18:05:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.187+0000 7ff1da600640 1 --2- 172.21.15.27:0/2094463003 >> v2:172.21.15.93:3300/0 conn(0x7ff1d4099a10 0x7ff1d4099e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.187+0000 7ff1da600640 1 --2- 172.21.15.27:0/2094463003 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1d409a5b0 0x7ff1d40cc500 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.187+0000 7ff1da600640 1 -- 172.21.15.27:0/2094463003 >> 172.21.15.27:0/2094463003 conn(0x7ff1d40b2a80 msgr2=0x7ff1d40b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.187+0000 7ff1da600640 1 -- 172.21.15.27:0/2094463003 shutdown_connections 2024-01-15T18:05:34.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.187+0000 7ff1da600640 1 -- 172.21.15.27:0/2094463003 wait complete. 2024-01-15T18:05:34.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.187+0000 7ff1da600640 1 Processor -- start 2024-01-15T18:05:34.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.187+0000 7ff1da600640 1 -- start start 2024-01-15T18:05:34.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.187+0000 7ff1da600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1d4099a10 0x7ff1d4159710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:34.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.187+0000 7ff1da600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff1d409a5b0 0x7ff1d4147eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:34.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.187+0000 7ff1da600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1d415b350 0x7ff1d41463c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:34.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.187+0000 7ff1da600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d4159cf0 con 0x7ff1d415b350 2024-01-15T18:05:34.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.187+0000 7ff1da600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d4147580 con 0x7ff1d409a5b0 2024-01-15T18:05:34.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.187+0000 7ff1da600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d4147860 con 0x7ff1d4099a10 2024-01-15T18:05:34.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.191+0000 7ff1d3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1d415b350 0x7ff1d41463c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:34.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.191+0000 7ff1d3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1d415b350 0x7ff1d41463c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34860/0 (socket says 172.21.15.27:34860) 2024-01-15T18:05:34.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.191+0000 7ff1d3e00640 1 -- 172.21.15.27:0/1446888406 learned_addr learned my addr 172.21.15.27:0/1446888406 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:34.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.191+0000 7ff1d2a00640 1 --2- 172.21.15.27:0/1446888406 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff1d409a5b0 0x7ff1d4147eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:34.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.191+0000 7ff1d3400640 1 --2- 172.21.15.27:0/1446888406 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1d4099a10 0x7ff1d4159710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:34.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.191+0000 7ff1d3e00640 1 -- 172.21.15.27:0/1446888406 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1d4099a10 msgr2=0x7ff1d4159710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:34.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.191+0000 7ff1d3e00640 1 --2- 172.21.15.27:0/1446888406 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1d4099a10 0x7ff1d4159710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.191+0000 7ff1d3e00640 1 -- 172.21.15.27:0/1446888406 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff1d409a5b0 msgr2=0x7ff1d4147eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:34.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.191+0000 7ff1d3e00640 1 --2- 172.21.15.27:0/1446888406 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff1d409a5b0 0x7ff1d4147eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.191+0000 7ff1d3e00640 1 -- 172.21.15.27:0/1446888406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1bc0094d0 con 0x7ff1d415b350 2024-01-15T18:05:34.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.191+0000 7ff1d3e00640 1 --2- 172.21.15.27:0/1446888406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1d415b350 0x7ff1d41463c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc00efc0 tx=0x7ff1cc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:34.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.191+0000 7ff1cb400640 1 -- 172.21.15.27:0/1446888406 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc014480 con 0x7ff1d415b350 2024-01-15T18:05:34.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.191+0000 7ff1cb400640 1 -- 172.21.15.27:0/1446888406 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1cc00f040 con 0x7ff1d415b350 2024-01-15T18:05:34.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.191+0000 7ff1da600640 1 -- 172.21.15.27:0/1446888406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1d4147b40 con 0x7ff1d415b350 2024-01-15T18:05:34.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.191+0000 7ff1cb400640 1 -- 172.21.15.27:0/1446888406 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc014820 con 0x7ff1d415b350 2024-01-15T18:05:34.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.191+0000 7ff1da600640 1 -- 172.21.15.27:0/1446888406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1d4150760 con 0x7ff1d415b350 2024-01-15T18:05:34.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.191+0000 7ff1da600640 1 -- 172.21.15.27:0/1446888406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1d404e850 con 0x7ff1d415b350 2024-01-15T18:05:34.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.191+0000 7ff1cb400640 1 -- 172.21.15.27:0/1446888406 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7ff1cc014980 con 0x7ff1d415b350 2024-01-15T18:05:34.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.195+0000 7ff1cb400640 1 --2- 172.21.15.27:0/1446888406 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff1ac0668f0 0x7ff1ac068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:34.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.195+0000 7ff1cb400640 1 -- 172.21.15.27:0/1446888406 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff1cc084e00 con 0x7ff1d415b350 2024-01-15T18:05:34.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.195+0000 7ff1d3400640 1 --2- 172.21.15.27:0/1446888406 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff1ac0668f0 0x7ff1ac068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:34.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.195+0000 7ff1d3400640 1 --2- 172.21.15.27:0/1446888406 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff1ac0668f0 0x7ff1ac068db0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4000f70 tx=0x7ff1c4007e50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:34.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.195+0000 7ff1cb400640 1 -- 172.21.15.27:0/1446888406 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff1cc0520e0 con 0x7ff1d415b350 2024-01-15T18:05:34.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.355+0000 7ff1da600640 1 -- 172.21.15.27:0/1446888406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff1d40b6ed0 con 0x7ff1d415b350 2024-01-15T18:05:34.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.355+0000 7ff1cb400640 1 -- 172.21.15.27:0/1446888406 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff1cc018020 con 0x7ff1d415b350 2024-01-15T18:05:34.361 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-15T18:05:34.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.359+0000 7ff1da600640 1 -- 172.21.15.27:0/1446888406 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff1ac0668f0 msgr2=0x7ff1ac068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:34.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.359+0000 7ff1da600640 1 --2- 172.21.15.27:0/1446888406 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff1ac0668f0 0x7ff1ac068db0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4000f70 tx=0x7ff1c4007e50 comp rx=0 tx=0).stop 2024-01-15T18:05:34.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.359+0000 7ff1da600640 1 -- 172.21.15.27:0/1446888406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1d415b350 msgr2=0x7ff1d41463c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:34.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.359+0000 7ff1da600640 1 --2- 172.21.15.27:0/1446888406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1d415b350 0x7ff1d41463c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc00efc0 tx=0x7ff1cc00c490 comp rx=0 tx=0).stop 2024-01-15T18:05:34.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.359+0000 7ff1da600640 1 -- 172.21.15.27:0/1446888406 shutdown_connections 2024-01-15T18:05:34.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.359+0000 7ff1da600640 1 --2- 172.21.15.27:0/1446888406 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff1ac0668f0 0x7ff1ac068db0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.359+0000 7ff1da600640 1 --2- 172.21.15.27:0/1446888406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1d415b350 0x7ff1d41463c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.359+0000 7ff1da600640 1 --2- 172.21.15.27:0/1446888406 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff1d409a5b0 0x7ff1d4147eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.359+0000 7ff1da600640 1 --2- 172.21.15.27:0/1446888406 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1d4099a10 0x7ff1d4159710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.359+0000 7ff1da600640 1 -- 172.21.15.27:0/1446888406 >> 172.21.15.27:0/1446888406 conn(0x7ff1d40b2a80 msgr2=0x7ff1d40ccf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:34.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.359+0000 7ff1da600640 1 -- 172.21.15.27:0/1446888406 shutdown_connections 2024-01-15T18:05:34.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.359+0000 7ff1da600640 1 -- 172.21.15.27:0/1446888406 wait complete. 2024-01-15T18:05:34.383 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T18:05:34.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.523+0000 7fd8b5400640 1 Processor -- start 2024-01-15T18:05:34.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.523+0000 7fd8b5400640 1 -- start start 2024-01-15T18:05:34.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.523+0000 7fd8b5400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8b00cde10 0x7fd8b00cbf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:34.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.523+0000 7fd8b5400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fd8b00cfc30 0x7fd8b00cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:34.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.523+0000 7fd8b5400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b00cb4d0 con 0x7fd8b015d680 2024-01-15T18:05:34.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.523+0000 7fd8b5400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b00d0120 con 0x7fd8b00cfc30 2024-01-15T18:05:34.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.523+0000 7fd8b5400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00d0400 con 0x7fd8b00cde10 2024-01-15T18:05:34.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.523+0000 7fd8ade00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8b00cde10 0x7fd8b00cbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:34.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.523+0000 7fd8af200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fd8b00cfc30 0x7fd8b00cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:34.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.523+0000 7fd8ae800640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7fd8b015d680 0x7fd8b00cd700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:59290/0 (socket says 172.21.15.27:59290) 2024-01-15T18:05:34.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.523+0000 7fd8ae800640 1 -- 172.21.15.27:0/2449257144 learned_addr learned my addr 172.21.15.27:0/2449257144 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:34.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.523+0000 7fd8ad400640 1 -- 172.21.15.27:0/2449257144 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3091872986 0 0) 0x7fd8b00cb4d0 con 0x7fd8b015d680 2024-01-15T18:05:34.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.523+0000 7fd8ad400640 1 -- 172.21.15.27:0/2449257144 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd890003530 con 0x7fd8b015d680 2024-01-15T18:05:34.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.523+0000 7fd8af200640 1 -- 172.21.15.27:0/2449257144 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8b00cde10 msgr2=0x7fd8b00cbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:34.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.523+0000 7fd8af200640 1 --2- 172.21.15.27:0/2449257144 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8b00cde10 0x7fd8b00cbf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.523+0000 7fd8af200640 1 -- 172.21.15.27:0/2449257144 >> v1:172.21.15.93:6789/0 conn(0x7fd8b015d680 legacy=0x7fd8b00cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:34.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.523+0000 7fd8af200640 1 -- 172.21.15.27:0/2449257144 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b00cb770 con 0x7fd8b00cfc30 2024-01-15T18:05:34.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.523+0000 7fd8ade00640 1 --2- 172.21.15.27:0/2449257144 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8b00cde10 0x7fd8b00cbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:34.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.527+0000 7fd8af200640 1 --2- 172.21.15.27:0/2449257144 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd8b00cfc30 0x7fd8b00cc4b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009d90 tx=0x7fd8a801c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3205cba6ece409e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:34.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.527+0000 7fd8ad400640 1 -- 172.21.15.27:0/2449257144 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b00ce850 con 0x7fd8b00cfc30 2024-01-15T18:05:34.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.527+0000 7fd8ad400640 1 -- 172.21.15.27:0/2449257144 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a80a5a00 con 0x7fd8b00cfc30 2024-01-15T18:05:34.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.527+0000 7fd8ad400640 1 -- 172.21.15.27:0/2449257144 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a80a5d70 con 0x7fd8b00cfc30 2024-01-15T18:05:34.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.527+0000 7fd8b5400640 1 -- 172.21.15.27:0/2449257144 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd8b00cfc30 msgr2=0x7fd8b00cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:34.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.527+0000 7fd8b5400640 1 --2- 172.21.15.27:0/2449257144 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd8b00cfc30 0x7fd8b00cc4b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009d90 tx=0x7fd8a801c920 comp rx=0 tx=0).stop 2024-01-15T18:05:34.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.527+0000 7fd8b5400640 1 -- 172.21.15.27:0/2449257144 shutdown_connections 2024-01-15T18:05:34.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.527+0000 7fd8b5400640 1 --2- 172.21.15.27:0/2449257144 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd8b00cfc30 0x7fd8b00cc4b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.527+0000 7fd8b5400640 1 --2- 172.21.15.27:0/2449257144 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8b00cde10 0x7fd8b00cbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.527+0000 7fd8b5400640 1 -- 172.21.15.27:0/2449257144 >> 172.21.15.27:0/2449257144 conn(0x7fd8b00d0e80 msgr2=0x7fd8b00d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:34.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.527+0000 7fd8b5400640 1 -- 172.21.15.27:0/2449257144 shutdown_connections 2024-01-15T18:05:34.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.527+0000 7fd8b5400640 1 -- 172.21.15.27:0/2449257144 wait complete. 2024-01-15T18:05:34.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.531+0000 7fd8b5400640 1 Processor -- start 2024-01-15T18:05:34.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.531+0000 7fd8b5400640 1 -- start start 2024-01-15T18:05:34.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.531+0000 7fd8b5400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd8b00cde10 0x7fd8b0153200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:34.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.531+0000 7fd8b5400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8b00cfc30 0x7fd8b00aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:34.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.531+0000 7fd8b5400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8b015d680 0x7fd8b00af130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:34.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.531+0000 7fd8b5400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b015fe40 con 0x7fd8b00cfc30 2024-01-15T18:05:34.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.531+0000 7fd8b5400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b015ea10 con 0x7fd8b00cde10 2024-01-15T18:05:34.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.531+0000 7fd8b5400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b015de00 con 0x7fd8b015d680 2024-01-15T18:05:34.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.531+0000 7fd8ade00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8b00cfc30 0x7fd8b00aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:34.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.531+0000 7fd8ade00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8b00cfc30 0x7fd8b00aebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34866/0 (socket says 172.21.15.27:34866) 2024-01-15T18:05:34.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.531+0000 7fd8ade00640 1 -- 172.21.15.27:0/3417782234 learned_addr learned my addr 172.21.15.27:0/3417782234 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:34.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.531+0000 7fd8af200640 1 --2- 172.21.15.27:0/3417782234 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8b015d680 0x7fd8b00af130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:34.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.531+0000 7fd8ae800640 1 --2- 172.21.15.27:0/3417782234 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd8b00cde10 0x7fd8b0153200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:34.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.531+0000 7fd8ade00640 1 -- 172.21.15.27:0/3417782234 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8b015d680 msgr2=0x7fd8b00af130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:34.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.531+0000 7fd8ade00640 1 --2- 172.21.15.27:0/3417782234 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8b015d680 0x7fd8b00af130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.531+0000 7fd8ade00640 1 -- 172.21.15.27:0/3417782234 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd8b00cde10 msgr2=0x7fd8b0153200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:34.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.531+0000 7fd8ade00640 1 --2- 172.21.15.27:0/3417782234 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd8b00cde10 0x7fd8b0153200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.531+0000 7fd8ade00640 1 -- 172.21.15.27:0/3417782234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd898009420 con 0x7fd8b00cfc30 2024-01-15T18:05:34.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.531+0000 7fd8ade00640 1 --2- 172.21.15.27:0/3417782234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8b00cfc30 0x7fd8b00aebf0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd89800ebe0 tx=0x7fd89800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:34.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.531+0000 7fd8a6a00640 1 -- 172.21.15.27:0/3417782234 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89800eee0 con 0x7fd8b00cfc30 2024-01-15T18:05:34.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.531+0000 7fd8a6a00640 1 -- 172.21.15.27:0/3417782234 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd89800cc30 con 0x7fd8b00cfc30 2024-01-15T18:05:34.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.531+0000 7fd8a6a00640 1 -- 172.21.15.27:0/3417782234 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd898013470 con 0x7fd8b00cfc30 2024-01-15T18:05:34.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.531+0000 7fd8b5400640 1 -- 172.21.15.27:0/3417782234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8a8009a70 con 0x7fd8b00cfc30 2024-01-15T18:05:34.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.531+0000 7fd8b5400640 1 -- 172.21.15.27:0/3417782234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b015e3c0 con 0x7fd8b00cfc30 2024-01-15T18:05:34.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.535+0000 7fd8b5400640 1 -- 172.21.15.27:0/3417782234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b004e850 con 0x7fd8b00cfc30 2024-01-15T18:05:34.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.535+0000 7fd8a6a00640 1 -- 172.21.15.27:0/3417782234 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fd8980135d0 con 0x7fd8b00cfc30 2024-01-15T18:05:34.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.535+0000 7fd8a6a00640 1 --2- 172.21.15.27:0/3417782234 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd8880668f0 0x7fd888068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:34.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.535+0000 7fd8a6a00640 1 -- 172.21.15.27:0/3417782234 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd898084680 con 0x7fd8b00cfc30 2024-01-15T18:05:34.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.535+0000 7fd8ae800640 1 --2- 172.21.15.27:0/3417782234 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd8880668f0 0x7fd888068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:34.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.535+0000 7fd8ae800640 1 --2- 172.21.15.27:0/3417782234 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd8880668f0 0x7fd888068db0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0007fd0 tx=0x7fd8a0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:34.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.539+0000 7fd8a6a00640 1 -- 172.21.15.27:0/3417782234 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd898051970 con 0x7fd8b00cfc30 2024-01-15T18:05:34.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.695+0000 7fd8b5400640 1 -- 172.21.15.27:0/3417782234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd8b0067060 con 0x7fd8b00cfc30 2024-01-15T18:05:34.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.695+0000 7fd8a6a00640 1 -- 172.21.15.27:0/3417782234 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd898054f90 con 0x7fd8b00cfc30 2024-01-15T18:05:34.701 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-15T18:05:34.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.695+0000 7fd8b5400640 1 -- 172.21.15.27:0/3417782234 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd8880668f0 msgr2=0x7fd888068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:34.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.695+0000 7fd8b5400640 1 --2- 172.21.15.27:0/3417782234 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd8880668f0 0x7fd888068db0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0007fd0 tx=0x7fd8a0009040 comp rx=0 tx=0).stop 2024-01-15T18:05:34.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.695+0000 7fd8b5400640 1 -- 172.21.15.27:0/3417782234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8b00cfc30 msgr2=0x7fd8b00aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:34.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.695+0000 7fd8b5400640 1 --2- 172.21.15.27:0/3417782234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8b00cfc30 0x7fd8b00aebf0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd89800ebe0 tx=0x7fd89800c520 comp rx=0 tx=0).stop 2024-01-15T18:05:34.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.699+0000 7fd8b5400640 1 -- 172.21.15.27:0/3417782234 shutdown_connections 2024-01-15T18:05:34.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.699+0000 7fd8b5400640 1 --2- 172.21.15.27:0/3417782234 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd8880668f0 0x7fd888068db0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.699+0000 7fd8b5400640 1 --2- 172.21.15.27:0/3417782234 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8b015d680 0x7fd8b00af130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.699+0000 7fd8b5400640 1 --2- 172.21.15.27:0/3417782234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8b00cfc30 0x7fd8b00aebf0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.699+0000 7fd8b5400640 1 --2- 172.21.15.27:0/3417782234 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd8b00cde10 0x7fd8b0153200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.699+0000 7fd8b5400640 1 -- 172.21.15.27:0/3417782234 >> 172.21.15.27:0/3417782234 conn(0x7fd8b00d0e80 msgr2=0x7fd8b00d1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:34.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.699+0000 7fd8b5400640 1 -- 172.21.15.27:0/3417782234 shutdown_connections 2024-01-15T18:05:34.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.699+0000 7fd8b5400640 1 -- 172.21.15.27:0/3417782234 wait complete. 2024-01-15T18:05:34.722 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:05:34.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.859+0000 7f8952400640 1 Processor -- start 2024-01-15T18:05:34.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.859+0000 7f8952400640 1 -- start start 2024-01-15T18:05:34.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.859+0000 7f8952400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f894c15f290 0x7f894c15f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:34.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.859+0000 7f8952400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f894c0ce4d0 0x7f894c0ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:34.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.859+0000 7f8952400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f894c15fd00 con 0x7f894c0b1b40 2024-01-15T18:05:34.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.859+0000 7f8952400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f894c0ade90 con 0x7f894c0ce4d0 2024-01-15T18:05:34.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.859+0000 7f8952400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f894c15ffa0 con 0x7f894c15f290 2024-01-15T18:05:34.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.859+0000 7f894be00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f894c0ce4d0 0x7f894c0ae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:34.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.859+0000 7f894b400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f894c15f290 0x7f894c15f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:34.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.859+0000 7f894be00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f894c0ce4d0 0x7f894c0ae2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:50890/0 (socket says 172.21.15.27:50890) 2024-01-15T18:05:34.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.859+0000 7f894be00640 1 -- 172.21.15.27:0/1685678396 learned_addr learned my addr 172.21.15.27:0/1685678396 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:34.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.859+0000 7f894aa00640 1 --1- 172.21.15.27:0/1685678396 >> v1:172.21.15.93:6789/0 conn(0x7f894c0b1b40 0x7f894c0cddc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:59296/0 (socket says 172.21.15.27:59296) 2024-01-15T18:05:34.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.859+0000 7f894be00640 1 -- 172.21.15.27:0/1685678396 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f894c15f290 msgr2=0x7f894c15f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:34.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.859+0000 7f894be00640 1 --2- 172.21.15.27:0/1685678396 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f894c15f290 0x7f894c15f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.859+0000 7f894be00640 1 -- 172.21.15.27:0/1685678396 >> v1:172.21.15.93:6789/0 conn(0x7f894c0b1b40 legacy=0x7f894c0cddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:34.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.859+0000 7f894be00640 1 -- 172.21.15.27:0/1685678396 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f894c0b1630 con 0x7f894c0ce4d0 2024-01-15T18:05:34.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.859+0000 7f894a000640 1 -- 172.21.15.27:0/1685678396 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859039580 0 0) 0x7f894c15fd00 con 0x7f894c0b1b40 2024-01-15T18:05:34.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.859+0000 7f894b400640 1 --2- 172.21.15.27:0/1685678396 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f894c15f290 0x7f894c15f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:34.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.863+0000 7f894be00640 1 --2- 172.21.15.27:0/1685678396 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f894c0ce4d0 0x7f894c0ae2b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8944009e50 tx=0x7f894401c980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f95c54d65436b9b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:34.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.863+0000 7f894a000640 1 -- 172.21.15.27:0/1685678396 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89440a5750 con 0x7f894c0ce4d0 2024-01-15T18:05:34.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.863+0000 7f894a000640 1 -- 172.21.15.27:0/1685678396 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89440a58b0 con 0x7f894c0ce4d0 2024-01-15T18:05:34.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.863+0000 7f894a000640 1 -- 172.21.15.27:0/1685678396 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89440a5c40 con 0x7f894c0ce4d0 2024-01-15T18:05:34.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.863+0000 7f8952400640 1 -- 172.21.15.27:0/1685678396 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f894c0ce4d0 msgr2=0x7f894c0ae2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:34.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.863+0000 7f8952400640 1 --2- 172.21.15.27:0/1685678396 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f894c0ce4d0 0x7f894c0ae2b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8944009e50 tx=0x7f894401c980 comp rx=0 tx=0).stop 2024-01-15T18:05:34.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.863+0000 7f8952400640 1 -- 172.21.15.27:0/1685678396 shutdown_connections 2024-01-15T18:05:34.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.863+0000 7f8952400640 1 --2- 172.21.15.27:0/1685678396 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f894c0ce4d0 0x7f894c0ae2b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.863+0000 7f8952400640 1 --2- 172.21.15.27:0/1685678396 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f894c15f290 0x7f894c15f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.863+0000 7f8952400640 1 -- 172.21.15.27:0/1685678396 >> 172.21.15.27:0/1685678396 conn(0x7f894c0d0f40 msgr2=0x7f894c0d1350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:34.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.863+0000 7f8952400640 1 -- 172.21.15.27:0/1685678396 shutdown_connections 2024-01-15T18:05:34.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.863+0000 7f8952400640 1 -- 172.21.15.27:0/1685678396 wait complete. 2024-01-15T18:05:34.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.867+0000 7f8952400640 1 Processor -- start 2024-01-15T18:05:34.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.867+0000 7f8952400640 1 -- start start 2024-01-15T18:05:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.867+0000 7f8952400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f894c0b1b40 0x7f894c0ca820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.867+0000 7f8952400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f894c0ce4d0 0x7f894c156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.867+0000 7f8952400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f894c15f290 0x7f894c0c8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.867+0000 7f8952400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894c0cae90 con 0x7f894c15f290 2024-01-15T18:05:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.867+0000 7f8952400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894c0c8c70 con 0x7f894c0ce4d0 2024-01-15T18:05:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.867+0000 7f8952400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f894c0c8f50 con 0x7f894c0b1b40 2024-01-15T18:05:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.867+0000 7f894be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f894c15f290 0x7f894c0c8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.867+0000 7f894aa00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f894c0ce4d0 0x7f894c156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.867+0000 7f894be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f894c15f290 0x7f894c0c8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34876/0 (socket says 172.21.15.27:34876) 2024-01-15T18:05:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.867+0000 7f894be00640 1 -- 172.21.15.27:0/510133793 learned_addr learned my addr 172.21.15.27:0/510133793 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.867+0000 7f894b400640 1 --2- 172.21.15.27:0/510133793 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f894c0b1b40 0x7f894c0ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.867+0000 7f894be00640 1 -- 172.21.15.27:0/510133793 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f894c0b1b40 msgr2=0x7f894c0ca820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.867+0000 7f894be00640 1 --2- 172.21.15.27:0/510133793 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f894c0b1b40 0x7f894c0ca820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.867+0000 7f894be00640 1 -- 172.21.15.27:0/510133793 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f894c0ce4d0 msgr2=0x7f894c156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.867+0000 7f894be00640 1 --2- 172.21.15.27:0/510133793 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f894c0ce4d0 0x7f894c156900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.867+0000 7f894be00640 1 -- 172.21.15.27:0/510133793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f893c009420 con 0x7f894c15f290 2024-01-15T18:05:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.867+0000 7f894be00640 1 --2- 172.21.15.27:0/510133793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f894c15f290 0x7f894c0c8270 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f894401a5b0 tx=0x7f8944019720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:34.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.867+0000 7f8943400640 1 -- 172.21.15.27:0/510133793 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89440b8880 con 0x7f894c15f290 2024-01-15T18:05:34.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.867+0000 7f8943400640 1 -- 172.21.15.27:0/510133793 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89440b89e0 con 0x7f894c15f290 2024-01-15T18:05:34.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.867+0000 7f8952400640 1 -- 172.21.15.27:0/510133793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8944009b30 con 0x7f894c15f290 2024-01-15T18:05:34.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.867+0000 7f8943400640 1 -- 172.21.15.27:0/510133793 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89440b8d70 con 0x7f894c15f290 2024-01-15T18:05:34.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.867+0000 7f8952400640 1 -- 172.21.15.27:0/510133793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f894c1563e0 con 0x7f894c15f290 2024-01-15T18:05:34.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.871+0000 7f8952400640 1 -- 172.21.15.27:0/510133793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f894c04e850 con 0x7f894c15f290 2024-01-15T18:05:34.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.871+0000 7f8943400640 1 -- 172.21.15.27:0/510133793 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f89440b6440 con 0x7f894c15f290 2024-01-15T18:05:34.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.871+0000 7f8943400640 1 --2- 172.21.15.27:0/510133793 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8924066920 0x7f8924068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:34.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.871+0000 7f8943400640 1 -- 172.21.15.27:0/510133793 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f894411f940 con 0x7f894c15f290 2024-01-15T18:05:34.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.871+0000 7f894b400640 1 --2- 172.21.15.27:0/510133793 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8924066920 0x7f8924068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:34.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.871+0000 7f894b400640 1 --2- 172.21.15.27:0/510133793 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8924066920 0x7f8924068de0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f894c0cec10 tx=0x7f893c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:34.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:34.875+0000 7f8943400640 1 -- 172.21.15.27:0/510133793 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f89440eccc0 con 0x7f894c15f290 2024-01-15T18:05:35.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.043+0000 7f8952400640 1 -- 172.21.15.27:0/510133793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f894c072680 con 0x7f894c15f290 2024-01-15T18:05:35.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.043+0000 7f8943400640 1 -- 172.21.15.27:0/510133793 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f8944009510 con 0x7f894c15f290 2024-01-15T18:05:35.050 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-01-15T18:05:35.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.047+0000 7f8952400640 1 -- 172.21.15.27:0/510133793 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8924066920 msgr2=0x7f8924068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:35.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.047+0000 7f8952400640 1 --2- 172.21.15.27:0/510133793 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8924066920 0x7f8924068de0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f894c0cec10 tx=0x7f893c009190 comp rx=0 tx=0).stop 2024-01-15T18:05:35.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.047+0000 7f8952400640 1 -- 172.21.15.27:0/510133793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f894c15f290 msgr2=0x7f894c0c8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:35.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.047+0000 7f8952400640 1 --2- 172.21.15.27:0/510133793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f894c15f290 0x7f894c0c8270 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f894401a5b0 tx=0x7f8944019720 comp rx=0 tx=0).stop 2024-01-15T18:05:35.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.047+0000 7f8952400640 1 -- 172.21.15.27:0/510133793 shutdown_connections 2024-01-15T18:05:35.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.047+0000 7f8952400640 1 --2- 172.21.15.27:0/510133793 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8924066920 0x7f8924068de0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.047+0000 7f8952400640 1 --2- 172.21.15.27:0/510133793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f894c15f290 0x7f894c0c8270 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.047+0000 7f8952400640 1 --2- 172.21.15.27:0/510133793 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f894c0ce4d0 0x7f894c156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.047+0000 7f8952400640 1 --2- 172.21.15.27:0/510133793 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f894c0b1b40 0x7f894c0ca820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.047+0000 7f8952400640 1 -- 172.21.15.27:0/510133793 >> 172.21.15.27:0/510133793 conn(0x7f894c0d0f40 msgr2=0x7f894c157bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:35.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.047+0000 7f8952400640 1 -- 172.21.15.27:0/510133793 shutdown_connections 2024-01-15T18:05:35.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.047+0000 7f8952400640 1 -- 172.21.15.27:0/510133793 wait complete. 2024-01-15T18:05:35.072 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-01-15T18:05:35.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:35.072 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-15T18:05:35.101 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:05:35.101 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T18:05:35.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:35.101 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-15T18:05:35.161 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:05:35.161 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:05:35.170 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:05:35.170 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:05:35.222 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:35.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.359+0000 7f041ae00640 1 Processor -- start 2024-01-15T18:05:35.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.359+0000 7f041ae00640 1 -- start start 2024-01-15T18:05:35.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.359+0000 7f041ae00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f041415d0a0 0x7f041406c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.359+0000 7f041ae00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f04140ce710 0x7f04140cd240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.359+0000 7f041ae00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04140ad180 con 0x7f04140c94c0 2024-01-15T18:05:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.359+0000 7f041ae00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04140ac890 con 0x7f04140ce710 2024-01-15T18:05:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.359+0000 7f041ae00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04140cda90 con 0x7f041415d0a0 2024-01-15T18:05:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.359+0000 7f0418e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f04140c94c0 0x7f04140ad610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35076/0 (socket says 172.21.15.27:35076) 2024-01-15T18:05:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.359+0000 7f0418e00640 1 -- 172.21.15.27:0/478767052 learned_addr learned my addr 172.21.15.27:0/478767052 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.359+0000 7f0413400640 1 --2- 172.21.15.27:0/478767052 >> v2:172.21.15.93:3300/0 conn(0x7f04140ce710 0x7f04140cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.359+0000 7f0413e00640 1 --2- 172.21.15.27:0/478767052 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f041415d0a0 0x7f041406c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.359+0000 7f0412a00640 1 -- 172.21.15.27:0/478767052 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158161734 0 0) 0x7f04140ad180 con 0x7f04140c94c0 2024-01-15T18:05:35.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.359+0000 7f0412a00640 1 -- 172.21.15.27:0/478767052 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03f4003530 con 0x7f04140c94c0 2024-01-15T18:05:35.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.359+0000 7f0413e00640 1 -- 172.21.15.27:0/478767052 >> v2:172.21.15.93:3300/0 conn(0x7f04140ce710 msgr2=0x7f04140cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:35.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.359+0000 7f0413e00640 1 --2- 172.21.15.27:0/478767052 >> v2:172.21.15.93:3300/0 conn(0x7f04140ce710 0x7f04140cd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.359+0000 7f0413e00640 1 -- 172.21.15.27:0/478767052 >> v1:172.21.15.27:6789/0 conn(0x7f04140c94c0 legacy=0x7f04140ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:35.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.359+0000 7f0413e00640 1 -- 172.21.15.27:0/478767052 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04140acb30 con 0x7f041415d0a0 2024-01-15T18:05:35.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.359+0000 7f0413400640 1 --2- 172.21.15.27:0/478767052 >> v2:172.21.15.93:3300/0 conn(0x7f04140ce710 0x7f04140cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:35.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.363+0000 7f0413e00640 1 --2- 172.21.15.27:0/478767052 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f041415d0a0 0x7f041406c1e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f04040097c0 tx=0x7f040401c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd2bd4979c780b2c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:35.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.363+0000 7f0412a00640 1 -- 172.21.15.27:0/478767052 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04040ab610 con 0x7f041415d0a0 2024-01-15T18:05:35.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.363+0000 7f0412a00640 1 -- 172.21.15.27:0/478767052 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04040ab770 con 0x7f041415d0a0 2024-01-15T18:05:35.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.363+0000 7f0412a00640 1 -- 172.21.15.27:0/478767052 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04040abb00 con 0x7f041415d0a0 2024-01-15T18:05:35.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.363+0000 7f041ae00640 1 -- 172.21.15.27:0/478767052 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f041415d0a0 msgr2=0x7f041406c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:35.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.363+0000 7f041ae00640 1 --2- 172.21.15.27:0/478767052 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f041415d0a0 0x7f041406c1e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f04040097c0 tx=0x7f040401c450 comp rx=0 tx=0).stop 2024-01-15T18:05:35.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.363+0000 7f041ae00640 1 -- 172.21.15.27:0/478767052 shutdown_connections 2024-01-15T18:05:35.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.363+0000 7f041ae00640 1 --2- 172.21.15.27:0/478767052 >> v2:172.21.15.93:3300/0 conn(0x7f04140ce710 0x7f04140cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.363+0000 7f041ae00640 1 --2- 172.21.15.27:0/478767052 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f041415d0a0 0x7f041406c1e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.363+0000 7f041ae00640 1 -- 172.21.15.27:0/478767052 >> 172.21.15.27:0/478767052 conn(0x7f04140cec60 msgr2=0x7f04140cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:35.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.363+0000 7f041ae00640 1 -- 172.21.15.27:0/478767052 shutdown_connections 2024-01-15T18:05:35.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.363+0000 7f041ae00640 1 -- 172.21.15.27:0/478767052 wait complete. 2024-01-15T18:05:35.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.367+0000 7f041ae00640 1 Processor -- start 2024-01-15T18:05:35.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.367+0000 7f041ae00640 1 -- start start 2024-01-15T18:05:35.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.367+0000 7f041ae00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f04140c94c0 0x7f041415c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.367+0000 7f041ae00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04140ce710 0x7f04140c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.367+0000 7f041ae00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f041415d0a0 0x7f04140c5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.367+0000 7f041ae00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0414159ac0 con 0x7f04140ce710 2024-01-15T18:05:35.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.367+0000 7f041ae00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0414159d10 con 0x7f041415d0a0 2024-01-15T18:05:35.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.367+0000 7f041ae00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0414159ff0 con 0x7f04140c94c0 2024-01-15T18:05:35.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.367+0000 7f0413400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04140ce710 0x7f04140c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.367+0000 7f0413400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04140ce710 0x7f04140c7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34886/0 (socket says 172.21.15.27:34886) 2024-01-15T18:05:35.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.367+0000 7f0413400640 1 -- 172.21.15.27:0/3419811648 learned_addr learned my addr 172.21.15.27:0/3419811648 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:35.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.367+0000 7f0413e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f04140c94c0 0x7f041415c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.367+0000 7f0418e00640 1 --2- 172.21.15.27:0/3419811648 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f041415d0a0 0x7f04140c5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.367+0000 7f0413400640 1 -- 172.21.15.27:0/3419811648 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f04140c94c0 msgr2=0x7f041415c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:35.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.367+0000 7f0413400640 1 --2- 172.21.15.27:0/3419811648 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f04140c94c0 0x7f041415c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.367+0000 7f0413400640 1 -- 172.21.15.27:0/3419811648 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f041415d0a0 msgr2=0x7f04140c5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:35.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.367+0000 7f0413400640 1 --2- 172.21.15.27:0/3419811648 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f041415d0a0 0x7f04140c5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.367+0000 7f0413400640 1 -- 172.21.15.27:0/3419811648 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03fc009420 con 0x7f04140ce710 2024-01-15T18:05:35.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.367+0000 7f0413e00640 1 --2- 172.21.15.27:0/3419811648 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f04140c94c0 0x7f041415c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:35.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.367+0000 7f0413400640 1 --2- 172.21.15.27:0/3419811648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04140ce710 0x7f04140c7d50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f03fc00ebe0 tx=0x7f03fc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:35.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.367+0000 7f040be00640 1 -- 172.21.15.27:0/3419811648 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03fc00eee0 con 0x7f04140ce710 2024-01-15T18:05:35.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.367+0000 7f040be00640 1 -- 172.21.15.27:0/3419811648 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03fc00cc30 con 0x7f04140ce710 2024-01-15T18:05:35.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.367+0000 7f041ae00640 1 -- 172.21.15.27:0/3419811648 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0404009420 con 0x7f04140ce710 2024-01-15T18:05:35.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.367+0000 7f040be00640 1 -- 172.21.15.27:0/3419811648 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03fc013470 con 0x7f04140ce710 2024-01-15T18:05:35.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.367+0000 7f041ae00640 1 -- 172.21.15.27:0/3419811648 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04140ae2b0 con 0x7f04140ce710 2024-01-15T18:05:35.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.371+0000 7f041ae00640 1 -- 172.21.15.27:0/3419811648 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f041404e850 con 0x7f04140ce710 2024-01-15T18:05:35.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.371+0000 7f040be00640 1 -- 172.21.15.27:0/3419811648 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f03fc0135d0 con 0x7f04140ce710 2024-01-15T18:05:35.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.371+0000 7f040be00640 1 --2- 172.21.15.27:0/3419811648 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03ec066920 0x7f03ec068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.371+0000 7f040be00640 1 -- 172.21.15.27:0/3419811648 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f03fc084610 con 0x7f04140ce710 2024-01-15T18:05:35.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.371+0000 7f0413e00640 1 --2- 172.21.15.27:0/3419811648 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03ec066920 0x7f03ec068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.371+0000 7f0413e00640 1 --2- 172.21.15.27:0/3419811648 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03ec066920 0x7f03ec068de0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f04040af000 tx=0x7f04040a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:35.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.375+0000 7f040be00640 1 -- 172.21.15.27:0/3419811648 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f03fc0518f0 con 0x7f04140ce710 2024-01-15T18:05:35.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.531+0000 7f041ae00640 1 -- 172.21.15.27:0/3419811648 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04141677b0 con 0x7f04140ce710 2024-01-15T18:05:35.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.535+0000 7f040be00640 1 -- 172.21.15.27:0/3419811648 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16409 (secure 0 0 0) 0x7f03fc054f10 con 0x7f04140ce710 2024-01-15T18:05:35.539 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:35.539 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:04:59.837052+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:04:56.220608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T18:04:57.210421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:35.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.535+0000 7f041ae00640 1 -- 172.21.15.27:0/3419811648 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03ec066920 msgr2=0x7f03ec068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:35.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.535+0000 7f041ae00640 1 --2- 172.21.15.27:0/3419811648 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03ec066920 0x7f03ec068de0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f04040af000 tx=0x7f04040a7000 comp rx=0 tx=0).stop 2024-01-15T18:05:35.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.535+0000 7f041ae00640 1 -- 172.21.15.27:0/3419811648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04140ce710 msgr2=0x7f04140c7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:35.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.535+0000 7f041ae00640 1 --2- 172.21.15.27:0/3419811648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04140ce710 0x7f04140c7d50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f03fc00ebe0 tx=0x7f03fc00c520 comp rx=0 tx=0).stop 2024-01-15T18:05:35.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.535+0000 7f041ae00640 1 -- 172.21.15.27:0/3419811648 shutdown_connections 2024-01-15T18:05:35.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.535+0000 7f041ae00640 1 --2- 172.21.15.27:0/3419811648 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03ec066920 0x7f03ec068de0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.535+0000 7f041ae00640 1 --2- 172.21.15.27:0/3419811648 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f041415d0a0 0x7f04140c5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.535+0000 7f041ae00640 1 --2- 172.21.15.27:0/3419811648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04140ce710 0x7f04140c7d50 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.535+0000 7f041ae00640 1 --2- 172.21.15.27:0/3419811648 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f04140c94c0 0x7f041415c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.535+0000 7f041ae00640 1 -- 172.21.15.27:0/3419811648 >> 172.21.15.27:0/3419811648 conn(0x7f04140cec60 msgr2=0x7f04140733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:35.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.535+0000 7f041ae00640 1 -- 172.21.15.27:0/3419811648 shutdown_connections 2024-01-15T18:05:35.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.535+0000 7f041ae00640 1 -- 172.21.15.27:0/3419811648 wait complete. 2024-01-15T18:05:35.560 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T18:04:56.220608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T18:04:57.210421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:05:35.561 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:05:35.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.699+0000 7f08f9000640 1 Processor -- start 2024-01-15T18:05:35.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.699+0000 7f08f9000640 1 -- start start 2024-01-15T18:05:35.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.699+0000 7f08f9000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f08f406b050 0x7f08f406a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.699+0000 7f08f9000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f08f4071550 0x7f08f415d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.699+0000 7f08f9000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08f4161020 con 0x7f08f406bc50 2024-01-15T18:05:35.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.699+0000 7f08f9000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08f4161300 con 0x7f08f4071550 2024-01-15T18:05:35.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.699+0000 7f08f9000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08f41615e0 con 0x7f08f406b050 2024-01-15T18:05:35.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.699+0000 7f08f1400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f08f406bc50 0x7f08f4070e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35086/0 (socket says 172.21.15.27:35086) 2024-01-15T18:05:35.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.699+0000 7f08f1400640 1 -- 172.21.15.27:0/1216992002 learned_addr learned my addr 172.21.15.27:0/1216992002 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:35.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.699+0000 7f08f2800640 1 --2- 172.21.15.27:0/1216992002 >> v2:172.21.15.93:3300/0 conn(0x7f08f4071550 0x7f08f415d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.699+0000 7f08f1e00640 1 --2- 172.21.15.27:0/1216992002 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f08f406b050 0x7f08f406a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.699+0000 7f08f0a00640 1 -- 172.21.15.27:0/1216992002 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4843358 0 0) 0x7f08f4161020 con 0x7f08f406bc50 2024-01-15T18:05:35.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.703+0000 7f08f0a00640 1 -- 172.21.15.27:0/1216992002 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08d4003530 con 0x7f08f406bc50 2024-01-15T18:05:35.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.703+0000 7f08f2800640 1 -- 172.21.15.27:0/1216992002 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f08f406b050 msgr2=0x7f08f406a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:35.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.703+0000 7f08f0a00640 1 -- 172.21.15.27:0/1216992002 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3237064030 0 0) 0x7f08d4003530 con 0x7f08f406bc50 2024-01-15T18:05:35.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.703+0000 7f08f2800640 1 --2- 172.21.15.27:0/1216992002 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f08f406b050 0x7f08f406a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.703+0000 7f08f2800640 1 -- 172.21.15.27:0/1216992002 >> v1:172.21.15.27:6789/0 conn(0x7f08f406bc50 legacy=0x7f08f4070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:35.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.703+0000 7f08f2800640 1 -- 172.21.15.27:0/1216992002 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f41618c0 con 0x7f08f4071550 2024-01-15T18:05:35.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.703+0000 7f08f1e00640 1 --2- 172.21.15.27:0/1216992002 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f08f406b050 0x7f08f406a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:35.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.703+0000 7f08f2800640 1 --2- 172.21.15.27:0/1216992002 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f08f4071550 0x7f08f415d9f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f08ec009d90 tx=0x7f08ec01c7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6c82a58238b044c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:35.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.703+0000 7f08f0a00640 1 -- 172.21.15.27:0/1216992002 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08dc001680 con 0x7f08f4071550 2024-01-15T18:05:35.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.703+0000 7f08f0a00640 1 -- 172.21.15.27:0/1216992002 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ec0a55c0 con 0x7f08f4071550 2024-01-15T18:05:35.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.703+0000 7f08f0a00640 1 -- 172.21.15.27:0/1216992002 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ec0a5930 con 0x7f08f4071550 2024-01-15T18:05:35.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.707+0000 7f08f9000640 1 -- 172.21.15.27:0/1216992002 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f08f4071550 msgr2=0x7f08f415d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:35.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.707+0000 7f08f9000640 1 --2- 172.21.15.27:0/1216992002 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f08f4071550 0x7f08f415d9f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f08ec009d90 tx=0x7f08ec01c7e0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.707+0000 7f08f9000640 1 -- 172.21.15.27:0/1216992002 shutdown_connections 2024-01-15T18:05:35.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.707+0000 7f08f9000640 1 --2- 172.21.15.27:0/1216992002 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f08f4071550 0x7f08f415d9f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.707+0000 7f08f9000640 1 --2- 172.21.15.27:0/1216992002 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f08f406b050 0x7f08f406a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.707+0000 7f08f9000640 1 -- 172.21.15.27:0/1216992002 >> 172.21.15.27:0/1216992002 conn(0x7f08f40b15b0 msgr2=0x7f08f40b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:35.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.707+0000 7f08f9000640 1 -- 172.21.15.27:0/1216992002 shutdown_connections 2024-01-15T18:05:35.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.707+0000 7f08f9000640 1 -- 172.21.15.27:0/1216992002 wait complete. 2024-01-15T18:05:35.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.707+0000 7f08f9000640 1 Processor -- start 2024-01-15T18:05:35.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.707+0000 7f08f9000640 1 -- start start 2024-01-15T18:05:35.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.707+0000 7f08f9000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08f406b050 0x7f08f414f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.707+0000 7f08f9000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f08f406bc50 0x7f08f4155500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.707+0000 7f08f9000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f08f4071550 0x7f08f406dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.707+0000 7f08f9000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f414f5e0 con 0x7f08f406b050 2024-01-15T18:05:35.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.707+0000 7f08f9000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f414f8e0 con 0x7f08f4071550 2024-01-15T18:05:35.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.707+0000 7f08f9000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08f414fbc0 con 0x7f08f406bc50 2024-01-15T18:05:35.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.707+0000 7f08f1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08f406b050 0x7f08f414f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.707+0000 7f08f1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08f406b050 0x7f08f414f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34896/0 (socket says 172.21.15.27:34896) 2024-01-15T18:05:35.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.707+0000 7f08f1e00640 1 -- 172.21.15.27:0/480897395 learned_addr learned my addr 172.21.15.27:0/480897395 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:35.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.711+0000 7f08f2800640 1 --2- 172.21.15.27:0/480897395 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f08f4071550 0x7f08f406dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.711+0000 7f08f1400640 1 --2- 172.21.15.27:0/480897395 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f08f406bc50 0x7f08f4155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.711+0000 7f08f1e00640 1 -- 172.21.15.27:0/480897395 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f08f406bc50 msgr2=0x7f08f4155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:35.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.711+0000 7f08f1e00640 1 --2- 172.21.15.27:0/480897395 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f08f406bc50 0x7f08f4155500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.711+0000 7f08f1e00640 1 -- 172.21.15.27:0/480897395 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f08f4071550 msgr2=0x7f08f406dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:35.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.711+0000 7f08f1e00640 1 --2- 172.21.15.27:0/480897395 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f08f4071550 0x7f08f406dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.711+0000 7f08f1e00640 1 -- 172.21.15.27:0/480897395 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e4009420 con 0x7f08f406b050 2024-01-15T18:05:35.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.711+0000 7f08f2800640 1 --2- 172.21.15.27:0/480897395 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f08f4071550 0x7f08f406dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:35.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.711+0000 7f08f1e00640 1 --2- 172.21.15.27:0/480897395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08f406b050 0x7f08f414f0a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f08e400ebe0 tx=0x7f08e400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:35.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.711+0000 7f08ea000640 1 -- 172.21.15.27:0/480897395 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e400eee0 con 0x7f08f406b050 2024-01-15T18:05:35.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.711+0000 7f08ea000640 1 -- 172.21.15.27:0/480897395 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e400cc30 con 0x7f08f406b050 2024-01-15T18:05:35.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.711+0000 7f08f9000640 1 -- 172.21.15.27:0/480897395 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ec009a70 con 0x7f08f406b050 2024-01-15T18:05:35.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.711+0000 7f08ea000640 1 -- 172.21.15.27:0/480897395 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e4013470 con 0x7f08f406b050 2024-01-15T18:05:35.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.711+0000 7f08f9000640 1 -- 172.21.15.27:0/480897395 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08f4150180 con 0x7f08f406b050 2024-01-15T18:05:35.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.711+0000 7f08f9000640 1 -- 172.21.15.27:0/480897395 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08f4067a10 con 0x7f08f406b050 2024-01-15T18:05:35.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.711+0000 7f08ea000640 1 -- 172.21.15.27:0/480897395 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f08e40135d0 con 0x7f08f406b050 2024-01-15T18:05:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.711+0000 7f08ea000640 1 --2- 172.21.15.27:0/480897395 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f08cc0668f0 0x7f08cc068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.711+0000 7f08ea000640 1 -- 172.21.15.27:0/480897395 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f08e4084690 con 0x7f08f406b050 2024-01-15T18:05:35.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.715+0000 7f08f1400640 1 --2- 172.21.15.27:0/480897395 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f08cc0668f0 0x7f08cc068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.715+0000 7f08f1400640 1 --2- 172.21.15.27:0/480897395 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f08cc0668f0 0x7f08cc068db0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f08f4151210 tx=0x7f08dc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:35.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.715+0000 7f08ea000640 1 -- 172.21.15.27:0/480897395 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f08e40518e0 con 0x7f08f406b050 2024-01-15T18:05:35.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.875+0000 7f08f9000640 1 -- 172.21.15.27:0/480897395 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f08f404e850 con 0x7f08f406b050 2024-01-15T18:05:35.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.875+0000 7f08ea000640 1 -- 172.21.15.27:0/480897395 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f08e4054f00 con 0x7f08f406b050 2024-01-15T18:05:35.880 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:05:35.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.875+0000 7f08f9000640 1 -- 172.21.15.27:0/480897395 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f08cc0668f0 msgr2=0x7f08cc068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:35.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.875+0000 7f08f9000640 1 --2- 172.21.15.27:0/480897395 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f08cc0668f0 0x7f08cc068db0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f08f4151210 tx=0x7f08dc009040 comp rx=0 tx=0).stop 2024-01-15T18:05:35.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.875+0000 7f08f9000640 1 -- 172.21.15.27:0/480897395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08f406b050 msgr2=0x7f08f414f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:35.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.875+0000 7f08f9000640 1 --2- 172.21.15.27:0/480897395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08f406b050 0x7f08f414f0a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f08e400ebe0 tx=0x7f08e400c520 comp rx=0 tx=0).stop 2024-01-15T18:05:35.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.875+0000 7f08f9000640 1 -- 172.21.15.27:0/480897395 shutdown_connections 2024-01-15T18:05:35.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.875+0000 7f08f9000640 1 --2- 172.21.15.27:0/480897395 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f08cc0668f0 0x7f08cc068db0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.875+0000 7f08f9000640 1 --2- 172.21.15.27:0/480897395 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f08f4071550 0x7f08f406dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.875+0000 7f08f9000640 1 --2- 172.21.15.27:0/480897395 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f08f406bc50 0x7f08f4155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.875+0000 7f08f9000640 1 --2- 172.21.15.27:0/480897395 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08f406b050 0x7f08f414f0a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.875+0000 7f08f9000640 1 -- 172.21.15.27:0/480897395 >> 172.21.15.27:0/480897395 conn(0x7f08f40b15b0 msgr2=0x7f08f40ad440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:35.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.875+0000 7f08f9000640 1 -- 172.21.15.27:0/480897395 shutdown_connections 2024-01-15T18:05:35.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:35.875+0000 7f08f9000640 1 -- 172.21.15.27:0/480897395 wait complete. 2024-01-15T18:05:35.901 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T18:05:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.039+0000 7f1e19600640 1 Processor -- start 2024-01-15T18:05:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.039+0000 7f1e19600640 1 -- start start 2024-01-15T18:05:36.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.043+0000 7f1e19600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f1e1415ad20 0x7f1e1415b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:36.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.043+0000 7f1e19600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e1415d900 0x7f1e1415dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:36.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.043+0000 7f1e19600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e1415a610 con 0x7f1e14070ee0 2024-01-15T18:05:36.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.043+0000 7f1e19600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e1415e3c0 con 0x7f1e1415ad20 2024-01-15T18:05:36.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.043+0000 7f1e19600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1415e6a0 con 0x7f1e1415d900 2024-01-15T18:05:36.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.043+0000 7f1e12a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f1e1415ad20 0x7f1e1415b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:36.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.043+0000 7f1e13400640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7f1e14070ee0 0x7f1e1406d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:59306/0 (socket says 172.21.15.27:59306) 2024-01-15T18:05:36.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.043+0000 7f1e12000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e1415d900 0x7f1e1415dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:36.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.043+0000 7f1e13400640 1 -- 172.21.15.27:0/2481765727 learned_addr learned my addr 172.21.15.27:0/2481765727 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:36.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.043+0000 7f1e11600640 1 -- 172.21.15.27:0/2481765727 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173621782 0 0) 0x7f1e1415a610 con 0x7f1e14070ee0 2024-01-15T18:05:36.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.043+0000 7f1e11600640 1 -- 172.21.15.27:0/2481765727 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1df4003530 con 0x7f1e14070ee0 2024-01-15T18:05:36.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.043+0000 7f1e12000640 1 -- 172.21.15.27:0/2481765727 >> v2:172.21.15.93:3300/0 conn(0x7f1e1415ad20 msgr2=0x7f1e1415b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:36.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.043+0000 7f1e12000640 1 --2- 172.21.15.27:0/2481765727 >> v2:172.21.15.93:3300/0 conn(0x7f1e1415ad20 0x7f1e1415b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.043+0000 7f1e12000640 1 -- 172.21.15.27:0/2481765727 >> v1:172.21.15.93:6789/0 conn(0x7f1e14070ee0 legacy=0x7f1e1406d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:36.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.043+0000 7f1e12000640 1 -- 172.21.15.27:0/2481765727 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1415b6e0 con 0x7f1e1415d900 2024-01-15T18:05:36.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.043+0000 7f1e12a00640 1 --2- 172.21.15.27:0/2481765727 >> v2:172.21.15.93:3300/0 conn(0x7f1e1415ad20 0x7f1e1415b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:36.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.047+0000 7f1e12000640 1 --2- 172.21.15.27:0/2481765727 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e1415d900 0x7f1e1415dd80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc0097c0 tx=0x7f1dfc01c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ee50c29e36fc5f4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:36.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.047+0000 7f1e11600640 1 -- 172.21.15.27:0/2481765727 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc0ab670 con 0x7f1e1415d900 2024-01-15T18:05:36.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.047+0000 7f1e11600640 1 -- 172.21.15.27:0/2481765727 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dfc0ab7d0 con 0x7f1e1415d900 2024-01-15T18:05:36.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.047+0000 7f1e11600640 1 -- 172.21.15.27:0/2481765727 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc0abb60 con 0x7f1e1415d900 2024-01-15T18:05:36.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.047+0000 7f1e19600640 1 -- 172.21.15.27:0/2481765727 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e1415d900 msgr2=0x7f1e1415dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:36.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.047+0000 7f1e19600640 1 --2- 172.21.15.27:0/2481765727 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e1415d900 0x7f1e1415dd80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc0097c0 tx=0x7f1dfc01c450 comp rx=0 tx=0).stop 2024-01-15T18:05:36.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.047+0000 7f1e19600640 1 -- 172.21.15.27:0/2481765727 shutdown_connections 2024-01-15T18:05:36.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.047+0000 7f1e19600640 1 --2- 172.21.15.27:0/2481765727 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e1415d900 0x7f1e1415dd80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.047+0000 7f1e19600640 1 --2- 172.21.15.27:0/2481765727 >> v2:172.21.15.93:3300/0 conn(0x7f1e1415ad20 0x7f1e1415b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.047+0000 7f1e19600640 1 -- 172.21.15.27:0/2481765727 >> 172.21.15.27:0/2481765727 conn(0x7f1e140cb700 msgr2=0x7f1e140cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:36.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.047+0000 7f1e19600640 1 -- 172.21.15.27:0/2481765727 shutdown_connections 2024-01-15T18:05:36.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.047+0000 7f1e19600640 1 -- 172.21.15.27:0/2481765727 wait complete. 2024-01-15T18:05:36.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.051+0000 7f1e19600640 1 Processor -- start 2024-01-15T18:05:36.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.051+0000 7f1e19600640 1 -- start start 2024-01-15T18:05:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.051+0000 7f1e19600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e14070ee0 0x7f1e1414e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.051+0000 7f1e19600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1e1415ad20 0x7f1e14161950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.051+0000 7f1e19600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e1415d900 0x7f1e1415f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.051+0000 7f1e19600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e14160560 con 0x7f1e1415d900 2024-01-15T18:05:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.051+0000 7f1e19600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e14151420 con 0x7f1e1415ad20 2024-01-15T18:05:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.051+0000 7f1e19600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e14161e90 con 0x7f1e14070ee0 2024-01-15T18:05:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.051+0000 7f1e13400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e1415d900 0x7f1e1415f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.051+0000 7f1e13400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e1415d900 0x7f1e1415f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34902/0 (socket says 172.21.15.27:34902) 2024-01-15T18:05:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.051+0000 7f1e13400640 1 -- 172.21.15.27:0/3067349767 learned_addr learned my addr 172.21.15.27:0/3067349767 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.051+0000 7f1e12000640 1 --2- 172.21.15.27:0/3067349767 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1e1415ad20 0x7f1e14161950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.051+0000 7f1e12a00640 1 --2- 172.21.15.27:0/3067349767 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e14070ee0 0x7f1e1414e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.051+0000 7f1e13400640 1 -- 172.21.15.27:0/3067349767 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e14070ee0 msgr2=0x7f1e1414e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.051+0000 7f1e13400640 1 --2- 172.21.15.27:0/3067349767 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e14070ee0 0x7f1e1414e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.051+0000 7f1e13400640 1 -- 172.21.15.27:0/3067349767 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1e1415ad20 msgr2=0x7f1e14161950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.051+0000 7f1e13400640 1 --2- 172.21.15.27:0/3067349767 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1e1415ad20 0x7f1e14161950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.051+0000 7f1e13400640 1 -- 172.21.15.27:0/3067349767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e04009420 con 0x7f1e1415d900 2024-01-15T18:05:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.051+0000 7f1e13400640 1 --2- 172.21.15.27:0/3067349767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e1415d900 0x7f1e1415f7b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c00cb40 tx=0x7f1e0c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.051+0000 7f1e0aa00640 1 -- 172.21.15.27:0/3067349767 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c00aca0 con 0x7f1e1415d900 2024-01-15T18:05:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.051+0000 7f1e0aa00640 1 -- 172.21.15.27:0/3067349767 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e0c00ae00 con 0x7f1e1415d900 2024-01-15T18:05:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.051+0000 7f1e0aa00640 1 -- 172.21.15.27:0/3067349767 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c012620 con 0x7f1e1415d900 2024-01-15T18:05:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.051+0000 7f1e19600640 1 -- 172.21.15.27:0/3067349767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dfc009420 con 0x7f1e1415d900 2024-01-15T18:05:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.051+0000 7f1e19600640 1 -- 172.21.15.27:0/3067349767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e14160280 con 0x7f1e1415d900 2024-01-15T18:05:36.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.051+0000 7f1e19600640 1 -- 172.21.15.27:0/3067349767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e1404e850 con 0x7f1e1415d900 2024-01-15T18:05:36.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.055+0000 7f1e0aa00640 1 -- 172.21.15.27:0/3067349767 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f1e0c007070 con 0x7f1e1415d900 2024-01-15T18:05:36.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.055+0000 7f1e0aa00640 1 --2- 172.21.15.27:0/3067349767 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1dec066920 0x7f1dec068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:36.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.055+0000 7f1e0aa00640 1 -- 172.21.15.27:0/3067349767 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e0c084430 con 0x7f1e1415d900 2024-01-15T18:05:36.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.055+0000 7f1e12a00640 1 --2- 172.21.15.27:0/3067349767 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1dec066920 0x7f1dec068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:36.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.055+0000 7f1e12a00640 1 --2- 172.21.15.27:0/3067349767 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1dec066920 0x7f1dec068de0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1e14163600 tx=0x7f1e04009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:36.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.059+0000 7f1e0aa00640 1 -- 172.21.15.27:0/3067349767 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1e0c051680 con 0x7f1e1415d900 2024-01-15T18:05:36.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.215+0000 7f1e19600640 1 -- 172.21.15.27:0/3067349767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1e1415fcf0 con 0x7f1e1415d900 2024-01-15T18:05:36.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.215+0000 7f1e0aa00640 1 -- 172.21.15.27:0/3067349767 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1e0c054ca0 con 0x7f1e1415d900 2024-01-15T18:05:36.221 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-15T18:05:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.219+0000 7f1e19600640 1 -- 172.21.15.27:0/3067349767 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1dec066920 msgr2=0x7f1dec068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.219+0000 7f1e19600640 1 --2- 172.21.15.27:0/3067349767 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1dec066920 0x7f1dec068de0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1e14163600 tx=0x7f1e04009190 comp rx=0 tx=0).stop 2024-01-15T18:05:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.219+0000 7f1e19600640 1 -- 172.21.15.27:0/3067349767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e1415d900 msgr2=0x7f1e1415f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.219+0000 7f1e19600640 1 --2- 172.21.15.27:0/3067349767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e1415d900 0x7f1e1415f7b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c00cb40 tx=0x7f1e0c00a060 comp rx=0 tx=0).stop 2024-01-15T18:05:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.219+0000 7f1e19600640 1 -- 172.21.15.27:0/3067349767 shutdown_connections 2024-01-15T18:05:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.219+0000 7f1e19600640 1 --2- 172.21.15.27:0/3067349767 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1dec066920 0x7f1dec068de0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.219+0000 7f1e19600640 1 --2- 172.21.15.27:0/3067349767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e1415d900 0x7f1e1415f7b0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.219+0000 7f1e19600640 1 --2- 172.21.15.27:0/3067349767 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1e1415ad20 0x7f1e14161950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.219+0000 7f1e19600640 1 --2- 172.21.15.27:0/3067349767 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e14070ee0 0x7f1e1414e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.219+0000 7f1e19600640 1 -- 172.21.15.27:0/3067349767 >> 172.21.15.27:0/3067349767 conn(0x7f1e140cb700 msgr2=0x7f1e1415a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.219+0000 7f1e19600640 1 -- 172.21.15.27:0/3067349767 shutdown_connections 2024-01-15T18:05:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.219+0000 7f1e19600640 1 -- 172.21.15.27:0/3067349767 wait complete. 2024-01-15T18:05:36.242 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T18:05:36.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.383+0000 7fc4aea00640 1 Processor -- start 2024-01-15T18:05:36.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.383+0000 7fc4aea00640 1 -- start start 2024-01-15T18:05:36.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.383+0000 7fc4aea00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fc4a8158900 0x7fc4a815f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:36.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.383+0000 7fc4aea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc4a815bf00 0x7fc4a80cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:36.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.383+0000 7fc4aea00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc4a80cf110 0x7fc4a80701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:36.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.383+0000 7fc4aea00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4a80cf580 con 0x7fc4a815bf00 2024-01-15T18:05:36.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.383+0000 7fc4aea00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4a80b1cb0 con 0x7fc4a8158900 2024-01-15T18:05:36.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.383+0000 7fc4aea00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a80b1f00 con 0x7fc4a80cf110 2024-01-15T18:05:36.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.383+0000 7fc4a7400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc4a815bf00 0x7fc4a80cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:36.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.383+0000 7fc4a7400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc4a815bf00 0x7fc4a80cebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34908/0 (socket says 172.21.15.27:34908) 2024-01-15T18:05:36.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.383+0000 7fc4a7400640 1 -- 172.21.15.27:0/360625782 learned_addr learned my addr 172.21.15.27:0/360625782 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:36.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.383+0000 7fc4a7e00640 1 --2- 172.21.15.27:0/360625782 >> v2:172.21.15.93:3300/0 conn(0x7fc4a8158900 0x7fc4a815f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:36.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.383+0000 7fc4aca00640 1 --2- 172.21.15.27:0/360625782 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc4a80cf110 0x7fc4a80701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.383+0000 7fc4a7400640 1 -- 172.21.15.27:0/360625782 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc4a80cf110 msgr2=0x7fc4a80701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.383+0000 7fc4a7400640 1 --2- 172.21.15.27:0/360625782 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc4a80cf110 0x7fc4a80701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.383+0000 7fc4a7400640 1 -- 172.21.15.27:0/360625782 >> v2:172.21.15.93:3300/0 conn(0x7fc4a8158900 msgr2=0x7fc4a815f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.383+0000 7fc4a7400640 1 --2- 172.21.15.27:0/360625782 >> v2:172.21.15.93:3300/0 conn(0x7fc4a8158900 0x7fc4a815f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.383+0000 7fc4a7400640 1 -- 172.21.15.27:0/360625782 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a80b21e0 con 0x7fc4a815bf00 2024-01-15T18:05:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.383+0000 7fc4aca00640 1 --2- 172.21.15.27:0/360625782 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc4a80cf110 0x7fc4a80701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:36.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.387+0000 7fc4a7400640 1 --2- 172.21.15.27:0/360625782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4a815bf00 0x7fc4a80cebd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc4a80ceaa0 tx=0x7fc49001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=478f80091e00ed9a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:36.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.387+0000 7fc4a6a00640 1 -- 172.21.15.27:0/360625782 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8070f30 con 0x7fc4a815bf00 2024-01-15T18:05:36.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.387+0000 7fc4a6a00640 1 -- 172.21.15.27:0/360625782 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4900ac410 con 0x7fc4a815bf00 2024-01-15T18:05:36.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.387+0000 7fc4a6a00640 1 -- 172.21.15.27:0/360625782 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4900ac7a0 con 0x7fc4a815bf00 2024-01-15T18:05:36.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.387+0000 7fc4aea00640 1 -- 172.21.15.27:0/360625782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4a815bf00 msgr2=0x7fc4a80cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:36.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.387+0000 7fc4aea00640 1 --2- 172.21.15.27:0/360625782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4a815bf00 0x7fc4a80cebd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc4a80ceaa0 tx=0x7fc49001c500 comp rx=0 tx=0).stop 2024-01-15T18:05:36.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.387+0000 7fc4aea00640 1 -- 172.21.15.27:0/360625782 shutdown_connections 2024-01-15T18:05:36.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.387+0000 7fc4aea00640 1 --2- 172.21.15.27:0/360625782 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc4a80cf110 0x7fc4a80701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.387+0000 7fc4aea00640 1 --2- 172.21.15.27:0/360625782 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4a815bf00 0x7fc4a80cebd0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.387+0000 7fc4aea00640 1 --2- 172.21.15.27:0/360625782 >> v2:172.21.15.93:3300/0 conn(0x7fc4a8158900 0x7fc4a815f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.387+0000 7fc4aea00640 1 -- 172.21.15.27:0/360625782 >> 172.21.15.27:0/360625782 conn(0x7fc4a80cb700 msgr2=0x7fc4a80cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:36.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.387+0000 7fc4aea00640 1 -- 172.21.15.27:0/360625782 shutdown_connections 2024-01-15T18:05:36.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.387+0000 7fc4aea00640 1 -- 172.21.15.27:0/360625782 wait complete. 2024-01-15T18:05:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.391+0000 7fc4aea00640 1 Processor -- start 2024-01-15T18:05:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.391+0000 7fc4aea00640 1 -- start start 2024-01-15T18:05:36.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.391+0000 7fc4aea00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc4a80cf110 0x7fc4a814eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:36.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.391+0000 7fc4aea00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc4a8158900 0x7fc4a81579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.391+0000 7fc4aea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4a815bf00 0x7fc4a814fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.391+0000 7fc4aea00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a80ccd20 con 0x7fc4a815bf00 2024-01-15T18:05:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.391+0000 7fc4aea00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a814f020 con 0x7fc4a80cf110 2024-01-15T18:05:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.391+0000 7fc4aea00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a80ad790 con 0x7fc4a8158900 2024-01-15T18:05:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.391+0000 7fc4aca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4a815bf00 0x7fc4a814fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.391+0000 7fc4aca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4a815bf00 0x7fc4a814fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34920/0 (socket says 172.21.15.27:34920) 2024-01-15T18:05:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.391+0000 7fc4aca00640 1 -- 172.21.15.27:0/1334599467 learned_addr learned my addr 172.21.15.27:0/1334599467 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.391+0000 7fc4a7e00640 1 --2- 172.21.15.27:0/1334599467 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc4a80cf110 0x7fc4a814eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.391+0000 7fc4a7400640 1 --2- 172.21.15.27:0/1334599467 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc4a8158900 0x7fc4a81579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.391+0000 7fc4aca00640 1 -- 172.21.15.27:0/1334599467 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc4a8158900 msgr2=0x7fc4a81579e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.391+0000 7fc4aca00640 1 --2- 172.21.15.27:0/1334599467 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc4a8158900 0x7fc4a81579e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.391+0000 7fc4aca00640 1 -- 172.21.15.27:0/1334599467 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc4a80cf110 msgr2=0x7fc4a814eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.391+0000 7fc4aca00640 1 --2- 172.21.15.27:0/1334599467 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc4a80cf110 0x7fc4a814eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.391+0000 7fc4aca00640 1 -- 172.21.15.27:0/1334599467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4900094d0 con 0x7fc4a815bf00 2024-01-15T18:05:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.391+0000 7fc4aca00640 1 --2- 172.21.15.27:0/1334599467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4a815bf00 0x7fc4a814fd10 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0007e70 tx=0x7fc4a000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:36.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.391+0000 7fc49fe00640 1 -- 172.21.15.27:0/1334599467 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a0003ba0 con 0x7fc4a815bf00 2024-01-15T18:05:36.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.391+0000 7fc49fe00640 1 -- 172.21.15.27:0/1334599467 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a000f040 con 0x7fc4a815bf00 2024-01-15T18:05:36.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.391+0000 7fc4aea00640 1 -- 172.21.15.27:0/1334599467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a80ada70 con 0x7fc4a815bf00 2024-01-15T18:05:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.391+0000 7fc49fe00640 1 -- 172.21.15.27:0/1334599467 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a000de80 con 0x7fc4a815bf00 2024-01-15T18:05:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.391+0000 7fc4aea00640 1 -- 172.21.15.27:0/1334599467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a80ae030 con 0x7fc4a815bf00 2024-01-15T18:05:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.391+0000 7fc4aea00640 1 -- 172.21.15.27:0/1334599467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4a80ae220 con 0x7fc4a815bf00 2024-01-15T18:05:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.395+0000 7fc49fe00640 1 -- 172.21.15.27:0/1334599467 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fc4a0004030 con 0x7fc4a815bf00 2024-01-15T18:05:36.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.395+0000 7fc49fe00640 1 --2- 172.21.15.27:0/1334599467 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc480066920 0x7fc480068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:36.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.395+0000 7fc49fe00640 1 -- 172.21.15.27:0/1334599467 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc4a00846c0 con 0x7fc4a815bf00 2024-01-15T18:05:36.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.395+0000 7fc4a7e00640 1 --2- 172.21.15.27:0/1334599467 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc480066920 0x7fc480068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:36.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.395+0000 7fc4a7e00640 1 --2- 172.21.15.27:0/1334599467 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc480066920 0x7fc480068de0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc4a80706f0 tx=0x7fc498008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:36.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.399+0000 7fc49fe00640 1 -- 172.21.15.27:0/1334599467 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc4a80ae220 con 0x7fc4a815bf00 2024-01-15T18:05:36.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.555+0000 7fc4aea00640 1 -- 172.21.15.27:0/1334599467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc4a80ae220 con 0x7fc4a815bf00 2024-01-15T18:05:36.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.555+0000 7fc49fe00640 1 -- 172.21.15.27:0/1334599467 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc4a80ae220 con 0x7fc4a815bf00 2024-01-15T18:05:36.561 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-15T18:05:36.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.559+0000 7fc4aea00640 1 -- 172.21.15.27:0/1334599467 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc480066920 msgr2=0x7fc480068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:36.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.559+0000 7fc4aea00640 1 --2- 172.21.15.27:0/1334599467 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc480066920 0x7fc480068de0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc4a80706f0 tx=0x7fc498008040 comp rx=0 tx=0).stop 2024-01-15T18:05:36.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.559+0000 7fc4aea00640 1 -- 172.21.15.27:0/1334599467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4a815bf00 msgr2=0x7fc4a814fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:36.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.559+0000 7fc4aea00640 1 --2- 172.21.15.27:0/1334599467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4a815bf00 0x7fc4a814fd10 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0007e70 tx=0x7fc4a000d4a0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.559+0000 7fc4aea00640 1 -- 172.21.15.27:0/1334599467 shutdown_connections 2024-01-15T18:05:36.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.559+0000 7fc4aea00640 1 --2- 172.21.15.27:0/1334599467 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc480066920 0x7fc480068de0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.559+0000 7fc4aea00640 1 --2- 172.21.15.27:0/1334599467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4a815bf00 0x7fc4a814fd10 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.559+0000 7fc4aea00640 1 --2- 172.21.15.27:0/1334599467 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc4a8158900 0x7fc4a81579e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.559+0000 7fc4aea00640 1 --2- 172.21.15.27:0/1334599467 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc4a80cf110 0x7fc4a814eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.559+0000 7fc4aea00640 1 -- 172.21.15.27:0/1334599467 >> 172.21.15.27:0/1334599467 conn(0x7fc4a80cb700 msgr2=0x7fc4a80cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:36.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.559+0000 7fc4aea00640 1 -- 172.21.15.27:0/1334599467 shutdown_connections 2024-01-15T18:05:36.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.559+0000 7fc4aea00640 1 -- 172.21.15.27:0/1334599467 wait complete. 2024-01-15T18:05:36.583 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:05:36.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.719+0000 7f1e5ce00640 1 Processor -- start 2024-01-15T18:05:36.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.719+0000 7f1e5ce00640 1 -- start start 2024-01-15T18:05:36.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.723+0000 7f1e5ce00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1e5815d7c0 0x7f1e580b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:36.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.723+0000 7f1e5ce00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f1e580b1d90 0x7f1e58099a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:36.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.723+0000 7f1e5ce00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e580b1aa0 con 0x7f1e5806bc70 2024-01-15T18:05:36.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.723+0000 7f1e5ce00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e580b1300 con 0x7f1e5815d7c0 2024-01-15T18:05:36.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.723+0000 7f1e5ce00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e580b25a0 con 0x7f1e580b1d90 2024-01-15T18:05:36.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.723+0000 7f1e55e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1e5815d7c0 0x7f1e580b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:36.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.723+0000 7f1e56800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1e5806bc70 0x7f1e58161140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35098/0 (socket says 172.21.15.27:35098) 2024-01-15T18:05:36.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.723+0000 7f1e56800640 1 -- 172.21.15.27:0/1821645251 learned_addr learned my addr 172.21.15.27:0/1821645251 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:36.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.723+0000 7f1e55e00640 1 --2- 172.21.15.27:0/1821645251 >> v2:172.21.15.27:3300/0 conn(0x7f1e5815d7c0 0x7f1e580b35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34936/0 (socket says 172.21.15.27:34936) 2024-01-15T18:05:36.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.723+0000 7f1e55400640 1 --2- 172.21.15.27:0/1821645251 >> v2:172.21.15.93:3300/0 conn(0x7f1e580b1d90 0x7f1e58099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:36.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.723+0000 7f1e55e00640 1 -- 172.21.15.27:0/1821645251 >> v2:172.21.15.93:3300/0 conn(0x7f1e580b1d90 msgr2=0x7f1e58099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:36.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.723+0000 7f1e55e00640 1 --2- 172.21.15.27:0/1821645251 >> v2:172.21.15.93:3300/0 conn(0x7f1e580b1d90 0x7f1e58099a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.723+0000 7f1e55e00640 1 -- 172.21.15.27:0/1821645251 >> v1:172.21.15.27:6789/0 conn(0x7f1e5806bc70 legacy=0x7f1e58161140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:36.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.723+0000 7f1e54a00640 1 -- 172.21.15.27:0/1821645251 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820469015 0 0) 0x7f1e580b1aa0 con 0x7f1e5806bc70 2024-01-15T18:05:36.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.723+0000 7f1e55e00640 1 -- 172.21.15.27:0/1821645251 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e580b14e0 con 0x7f1e5815d7c0 2024-01-15T18:05:36.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.727+0000 7f1e55e00640 1 --2- 172.21.15.27:0/1821645251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e5815d7c0 0x7f1e580b35a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1e58073de0 tx=0x7f1e4801c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7e64e7062b9fe07 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:36.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.727+0000 7f1e54a00640 1 -- 172.21.15.27:0/1821645251 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e480ab600 con 0x7f1e5815d7c0 2024-01-15T18:05:36.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.727+0000 7f1e54a00640 1 -- 172.21.15.27:0/1821645251 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e480a7050 con 0x7f1e5815d7c0 2024-01-15T18:05:36.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.727+0000 7f1e54a00640 1 -- 172.21.15.27:0/1821645251 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e480ab980 con 0x7f1e5815d7c0 2024-01-15T18:05:36.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.727+0000 7f1e5ce00640 1 -- 172.21.15.27:0/1821645251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e5815d7c0 msgr2=0x7f1e580b35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:36.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.727+0000 7f1e5ce00640 1 --2- 172.21.15.27:0/1821645251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e5815d7c0 0x7f1e580b35a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1e58073de0 tx=0x7f1e4801c340 comp rx=0 tx=0).stop 2024-01-15T18:05:36.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.727+0000 7f1e5ce00640 1 -- 172.21.15.27:0/1821645251 shutdown_connections 2024-01-15T18:05:36.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.727+0000 7f1e5ce00640 1 --2- 172.21.15.27:0/1821645251 >> v2:172.21.15.93:3300/0 conn(0x7f1e580b1d90 0x7f1e58099a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.727+0000 7f1e5ce00640 1 --2- 172.21.15.27:0/1821645251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e5815d7c0 0x7f1e580b35a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.727+0000 7f1e5ce00640 1 -- 172.21.15.27:0/1821645251 >> 172.21.15.27:0/1821645251 conn(0x7f1e580b2a80 msgr2=0x7f1e580b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:36.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.727+0000 7f1e5ce00640 1 -- 172.21.15.27:0/1821645251 shutdown_connections 2024-01-15T18:05:36.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.727+0000 7f1e5ce00640 1 -- 172.21.15.27:0/1821645251 wait complete. 2024-01-15T18:05:36.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.727+0000 7f1e5ce00640 1 Processor -- start 2024-01-15T18:05:36.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.727+0000 7f1e5ce00640 1 -- start start 2024-01-15T18:05:36.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.731+0000 7f1e5ce00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e5806bc70 0x7f1e5814a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:36.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.731+0000 7f1e5ce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e580b1d90 0x7f1e58153c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:36.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.731+0000 7f1e5ce00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1e5815d7c0 0x7f1e58151420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:36.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.731+0000 7f1e5ce00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5814c7f0 con 0x7f1e580b1d90 2024-01-15T18:05:36.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.731+0000 7f1e5ce00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e58150710 con 0x7f1e5815d7c0 2024-01-15T18:05:36.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.731+0000 7f1e5ce00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e58154150 con 0x7f1e5806bc70 2024-01-15T18:05:36.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.731+0000 7f1e55400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e580b1d90 0x7f1e58153c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:36.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.731+0000 7f1e55400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e580b1d90 0x7f1e58153c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34952/0 (socket says 172.21.15.27:34952) 2024-01-15T18:05:36.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.731+0000 7f1e55400640 1 -- 172.21.15.27:0/3162872640 learned_addr learned my addr 172.21.15.27:0/3162872640 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:36.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.731+0000 7f1e55e00640 1 --2- 172.21.15.27:0/3162872640 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e5806bc70 0x7f1e5814a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:36.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.731+0000 7f1e56800640 1 --2- 172.21.15.27:0/3162872640 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1e5815d7c0 0x7f1e58151420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:36.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.731+0000 7f1e55400640 1 -- 172.21.15.27:0/3162872640 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e5806bc70 msgr2=0x7f1e5814a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:36.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.731+0000 7f1e55400640 1 --2- 172.21.15.27:0/3162872640 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e5806bc70 0x7f1e5814a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.731+0000 7f1e55400640 1 -- 172.21.15.27:0/3162872640 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1e5815d7c0 msgr2=0x7f1e58151420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:36.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.731+0000 7f1e55400640 1 --2- 172.21.15.27:0/3162872640 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1e5815d7c0 0x7f1e58151420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.731+0000 7f1e55400640 1 -- 172.21.15.27:0/3162872640 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e48009420 con 0x7f1e580b1d90 2024-01-15T18:05:36.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.731+0000 7f1e55400640 1 --2- 172.21.15.27:0/3162872640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e580b1d90 0x7f1e58153c10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1e4000d5d0 tx=0x7f1e4000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:36.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.731+0000 7f1e4e000640 1 -- 172.21.15.27:0/3162872640 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e40004240 con 0x7f1e580b1d90 2024-01-15T18:05:36.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.731+0000 7f1e4e000640 1 -- 172.21.15.27:0/3162872640 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e400043a0 con 0x7f1e580b1d90 2024-01-15T18:05:36.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.731+0000 7f1e4e000640 1 -- 172.21.15.27:0/3162872640 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e40003840 con 0x7f1e580b1d90 2024-01-15T18:05:36.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.731+0000 7f1e5ce00640 1 -- 172.21.15.27:0/3162872640 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e58151960 con 0x7f1e580b1d90 2024-01-15T18:05:36.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.731+0000 7f1e5ce00640 1 -- 172.21.15.27:0/3162872640 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e58150960 con 0x7f1e580b1d90 2024-01-15T18:05:36.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.731+0000 7f1e5ce00640 1 -- 172.21.15.27:0/3162872640 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e5804e850 con 0x7f1e580b1d90 2024-01-15T18:05:36.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.735+0000 7f1e4e000640 1 -- 172.21.15.27:0/3162872640 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f1e400039a0 con 0x7f1e580b1d90 2024-01-15T18:05:36.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.735+0000 7f1e4e000640 1 --2- 172.21.15.27:0/3162872640 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1e300668f0 0x7f1e30068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:36.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.735+0000 7f1e4e000640 1 -- 172.21.15.27:0/3162872640 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e400844d0 con 0x7f1e580b1d90 2024-01-15T18:05:36.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.735+0000 7f1e55e00640 1 --2- 172.21.15.27:0/3162872640 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1e300668f0 0x7f1e30068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:36.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.735+0000 7f1e55e00640 1 --2- 172.21.15.27:0/3162872640 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1e300668f0 0x7f1e30068db0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1e480ad000 tx=0x7f1e480a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:36.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.739+0000 7f1e4e000640 1 -- 172.21.15.27:0/3162872640 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1e40051720 con 0x7f1e580b1d90 2024-01-15T18:05:36.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.907+0000 7f1e5ce00640 1 -- 172.21.15.27:0/3162872640 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1e580b6ed0 con 0x7f1e580b1d90 2024-01-15T18:05:36.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.907+0000 7f1e4e000640 1 -- 172.21.15.27:0/3162872640 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1e40054d40 con 0x7f1e580b1d90 2024-01-15T18:05:36.914 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-01-15T18:05:36.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.911+0000 7f1e5ce00640 1 -- 172.21.15.27:0/3162872640 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1e300668f0 msgr2=0x7f1e30068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:36.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.911+0000 7f1e5ce00640 1 --2- 172.21.15.27:0/3162872640 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1e300668f0 0x7f1e30068db0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1e480ad000 tx=0x7f1e480a6000 comp rx=0 tx=0).stop 2024-01-15T18:05:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.911+0000 7f1e5ce00640 1 -- 172.21.15.27:0/3162872640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e580b1d90 msgr2=0x7f1e58153c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.911+0000 7f1e5ce00640 1 --2- 172.21.15.27:0/3162872640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e580b1d90 0x7f1e58153c10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1e4000d5d0 tx=0x7f1e4000da90 comp rx=0 tx=0).stop 2024-01-15T18:05:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.911+0000 7f1e5ce00640 1 -- 172.21.15.27:0/3162872640 shutdown_connections 2024-01-15T18:05:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.911+0000 7f1e5ce00640 1 --2- 172.21.15.27:0/3162872640 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1e300668f0 0x7f1e30068db0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.911+0000 7f1e5ce00640 1 --2- 172.21.15.27:0/3162872640 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1e5815d7c0 0x7f1e58151420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.911+0000 7f1e5ce00640 1 --2- 172.21.15.27:0/3162872640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e580b1d90 0x7f1e58153c10 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.911+0000 7f1e5ce00640 1 --2- 172.21.15.27:0/3162872640 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e5806bc70 0x7f1e5814a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.911+0000 7f1e5ce00640 1 -- 172.21.15.27:0/3162872640 >> 172.21.15.27:0/3162872640 conn(0x7f1e580b2a80 msgr2=0x7f1e5815c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.911+0000 7f1e5ce00640 1 -- 172.21.15.27:0/3162872640 shutdown_connections 2024-01-15T18:05:36.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:36.911+0000 7f1e5ce00640 1 -- 172.21.15.27:0/3162872640 wait complete. 2024-01-15T18:05:36.936 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:05:36.936 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:05:36.946 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:37.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.083+0000 7f826c000640 1 Processor -- start 2024-01-15T18:05:37.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.083+0000 7f826c000640 1 -- start start 2024-01-15T18:05:37.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.083+0000 7f826c000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f826415b1b0 0x7f82640b14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.083+0000 7f826c000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f82640ce1f0 0x7f82640b2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.083+0000 7f826c000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8264072840 con 0x7f826415b590 2024-01-15T18:05:37.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.083+0000 7f826c000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8264071c30 con 0x7f82640ce1f0 2024-01-15T18:05:37.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.083+0000 7f826c000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8264071f10 con 0x7f826415b1b0 2024-01-15T18:05:37.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.083+0000 7f8269600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f826415b1b0 0x7f82640b14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.083+0000 7f826a000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f82640ce1f0 0x7f82640b2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.083+0000 7f826a000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f82640ce1f0 0x7f82640b2ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:50988/0 (socket says 172.21.15.27:50988) 2024-01-15T18:05:37.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.083+0000 7f826a000640 1 -- 172.21.15.27:0/1740539870 learned_addr learned my addr 172.21.15.27:0/1740539870 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:37.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.083+0000 7f8269600640 1 -- 172.21.15.27:0/1740539870 >> v2:172.21.15.93:3300/0 conn(0x7f82640ce1f0 msgr2=0x7f82640b2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:37.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.083+0000 7f8269600640 1 --2- 172.21.15.27:0/1740539870 >> v2:172.21.15.93:3300/0 conn(0x7f82640ce1f0 0x7f82640b2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.083+0000 7f8269600640 1 -- 172.21.15.27:0/1740539870 >> v1:172.21.15.93:6789/0 conn(0x7f826415b590 legacy=0x7f82640cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.083+0000 7f8269600640 1 -- 172.21.15.27:0/1740539870 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82640721f0 con 0x7f826415b1b0 2024-01-15T18:05:37.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.083+0000 7f826a000640 1 --2- 172.21.15.27:0/1740539870 >> v2:172.21.15.93:3300/0 conn(0x7f82640ce1f0 0x7f82640b2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:37.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.087+0000 7f8269600640 1 --2- 172.21.15.27:0/1740539870 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f826415b1b0 0x7f82640b14a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f82540097f0 tx=0x7f825401c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b5acb969e6dded5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:37.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.087+0000 7f8263e00640 1 -- 172.21.15.27:0/1740539870 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82640ce920 con 0x7f826415b1b0 2024-01-15T18:05:37.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.087+0000 7f8263e00640 1 -- 172.21.15.27:0/1740539870 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82540ac640 con 0x7f826415b1b0 2024-01-15T18:05:37.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.087+0000 7f8263e00640 1 -- 172.21.15.27:0/1740539870 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82540ac9d0 con 0x7f826415b1b0 2024-01-15T18:05:37.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.087+0000 7f826c000640 1 -- 172.21.15.27:0/1740539870 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f826415b1b0 msgr2=0x7f82640b14a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.087+0000 7f826c000640 1 --2- 172.21.15.27:0/1740539870 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f826415b1b0 0x7f82640b14a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f82540097f0 tx=0x7f825401c3c0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.087+0000 7f826c000640 1 -- 172.21.15.27:0/1740539870 shutdown_connections 2024-01-15T18:05:37.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.087+0000 7f826c000640 1 --2- 172.21.15.27:0/1740539870 >> v2:172.21.15.93:3300/0 conn(0x7f82640ce1f0 0x7f82640b2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.087+0000 7f826c000640 1 --2- 172.21.15.27:0/1740539870 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f826415b1b0 0x7f82640b14a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.087+0000 7f826c000640 1 -- 172.21.15.27:0/1740539870 >> 172.21.15.27:0/1740539870 conn(0x7f82640b47d0 msgr2=0x7f82640b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:37.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.087+0000 7f826c000640 1 -- 172.21.15.27:0/1740539870 shutdown_connections 2024-01-15T18:05:37.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.087+0000 7f826c000640 1 -- 172.21.15.27:0/1740539870 wait complete. 2024-01-15T18:05:37.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.091+0000 7f826c000640 1 Processor -- start 2024-01-15T18:05:37.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.091+0000 7f826c000640 1 -- start start 2024-01-15T18:05:37.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.091+0000 7f826c000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f82640ce1f0 0x7f8264146fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.091+0000 7f826c000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f826415b1b0 0x7f82640c8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.091+0000 7f826c000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f826415b590 0x7f82640c9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.091+0000 7f826c000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82640c97e0 con 0x7f826415b590 2024-01-15T18:05:37.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.091+0000 7f826c000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8264099880 con 0x7f826415b1b0 2024-01-15T18:05:37.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.091+0000 7f826c000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8264099b60 con 0x7f82640ce1f0 2024-01-15T18:05:37.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.091+0000 7f826a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f826415b590 0x7f82640c9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.091+0000 7f826a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f826415b590 0x7f82640c9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34956/0 (socket says 172.21.15.27:34956) 2024-01-15T18:05:37.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.091+0000 7f826a000640 1 -- 172.21.15.27:0/2142834956 learned_addr learned my addr 172.21.15.27:0/2142834956 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:37.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.091+0000 7f8269600640 1 --2- 172.21.15.27:0/2142834956 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f82640ce1f0 0x7f8264146fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.091+0000 7f8268c00640 1 --2- 172.21.15.27:0/2142834956 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f826415b1b0 0x7f82640c8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.091+0000 7f826a000640 1 -- 172.21.15.27:0/2142834956 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f82640ce1f0 msgr2=0x7f8264146fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.091+0000 7f826a000640 1 --2- 172.21.15.27:0/2142834956 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f82640ce1f0 0x7f8264146fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.091+0000 7f826a000640 1 -- 172.21.15.27:0/2142834956 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f826415b1b0 msgr2=0x7f82640c8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.091+0000 7f826a000640 1 --2- 172.21.15.27:0/2142834956 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f826415b1b0 0x7f82640c8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.091+0000 7f826a000640 1 -- 172.21.15.27:0/2142834956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825c009b30 con 0x7f826415b590 2024-01-15T18:05:37.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.091+0000 7f826a000640 1 --2- 172.21.15.27:0/2142834956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f826415b590 0x7f82640c9c80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f825c00efc0 tx=0x7f825c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:37.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.091+0000 7f8261600640 1 -- 172.21.15.27:0/2142834956 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c00cdc0 con 0x7f826415b590 2024-01-15T18:05:37.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.095+0000 7f8261600640 1 -- 172.21.15.27:0/2142834956 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825c00f040 con 0x7f826415b590 2024-01-15T18:05:37.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.095+0000 7f826c000640 1 -- 172.21.15.27:0/2142834956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8254009420 con 0x7f826415b590 2024-01-15T18:05:37.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.095+0000 7f8261600640 1 -- 172.21.15.27:0/2142834956 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c0146c0 con 0x7f826415b590 2024-01-15T18:05:37.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.095+0000 7f826c000640 1 -- 172.21.15.27:0/2142834956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8264099550 con 0x7f826415b590 2024-01-15T18:05:37.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.095+0000 7f826c000640 1 -- 172.21.15.27:0/2142834956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f826404e850 con 0x7f826415b590 2024-01-15T18:05:37.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.095+0000 7f8261600640 1 -- 172.21.15.27:0/2142834956 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f825c014820 con 0x7f826415b590 2024-01-15T18:05:37.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.095+0000 7f8261600640 1 --2- 172.21.15.27:0/2142834956 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f823c066920 0x7f823c068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.095+0000 7f8261600640 1 -- 172.21.15.27:0/2142834956 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f825c084c00 con 0x7f826415b590 2024-01-15T18:05:37.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.095+0000 7f8269600640 1 --2- 172.21.15.27:0/2142834956 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f823c066920 0x7f823c068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.095+0000 7f8269600640 1 --2- 172.21.15.27:0/2142834956 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f823c066920 0x7f823c068de0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f82540ae000 tx=0x7f82540a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:37.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.099+0000 7f8261600640 1 -- 172.21.15.27:0/2142834956 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f825c051e50 con 0x7f826415b590 2024-01-15T18:05:37.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.259+0000 7f826c000640 1 -- 172.21.15.27:0/2142834956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f826409a010 con 0x7f826415b590 2024-01-15T18:05:37.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.263+0000 7f8261600640 1 -- 172.21.15.27:0/2142834956 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16409 (secure 0 0 0) 0x7f825c055470 con 0x7f826415b590 2024-01-15T18:05:37.267 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:37.267 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:04:59.837052+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:04:56.220608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T18:04:57.210421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:37.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.263+0000 7f826c000640 1 -- 172.21.15.27:0/2142834956 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f823c066920 msgr2=0x7f823c068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.263+0000 7f826c000640 1 --2- 172.21.15.27:0/2142834956 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f823c066920 0x7f823c068de0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f82540ae000 tx=0x7f82540a7000 comp rx=0 tx=0).stop 2024-01-15T18:05:37.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.263+0000 7f826c000640 1 -- 172.21.15.27:0/2142834956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f826415b590 msgr2=0x7f82640c9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.263+0000 7f826c000640 1 --2- 172.21.15.27:0/2142834956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f826415b590 0x7f82640c9c80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f825c00efc0 tx=0x7f825c00c490 comp rx=0 tx=0).stop 2024-01-15T18:05:37.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.263+0000 7f826c000640 1 -- 172.21.15.27:0/2142834956 shutdown_connections 2024-01-15T18:05:37.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.263+0000 7f826c000640 1 --2- 172.21.15.27:0/2142834956 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f823c066920 0x7f823c068de0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.263+0000 7f826c000640 1 --2- 172.21.15.27:0/2142834956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f826415b590 0x7f82640c9c80 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.263+0000 7f826c000640 1 --2- 172.21.15.27:0/2142834956 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f826415b1b0 0x7f82640c8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.263+0000 7f826c000640 1 --2- 172.21.15.27:0/2142834956 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f82640ce1f0 0x7f8264146fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.263+0000 7f826c000640 1 -- 172.21.15.27:0/2142834956 >> 172.21.15.27:0/2142834956 conn(0x7f82640b47d0 msgr2=0x7f8264157e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:37.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.263+0000 7f826c000640 1 -- 172.21.15.27:0/2142834956 shutdown_connections 2024-01-15T18:05:37.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.263+0000 7f826c000640 1 -- 172.21.15.27:0/2142834956 wait complete. 2024-01-15T18:05:37.290 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T18:04:56.220608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T18:04:57.210421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:05:37.290 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:05:37.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.427+0000 7f4a99800640 1 Processor -- start 2024-01-15T18:05:37.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.427+0000 7f4a99800640 1 -- start start 2024-01-15T18:05:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.431+0000 7f4a99800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4a940cf460 0x7f4a940cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.431+0000 7f4a99800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4a940b2590 0x7f4a940cc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.431+0000 7f4a99800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a940cfd80 con 0x7f4a9415b4b0 2024-01-15T18:05:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.431+0000 7f4a99800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a940d19a0 con 0x7f4a940cf460 2024-01-15T18:05:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.431+0000 7f4a99800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a940d1c80 con 0x7f4a940b2590 2024-01-15T18:05:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.431+0000 7f4a92000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4a9415b4b0 0x7f4a940cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35100/0 (socket says 172.21.15.27:35100) 2024-01-15T18:05:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.431+0000 7f4a92000640 1 -- 172.21.15.27:0/2461007142 learned_addr learned my addr 172.21.15.27:0/2461007142 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.431+0000 7f4a92a00640 1 --2- 172.21.15.27:0/2461007142 >> v2:172.21.15.27:3300/0 conn(0x7f4a940cf460 0x7f4a940cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.431+0000 7f4a93400640 1 --2- 172.21.15.27:0/2461007142 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4a940b2590 0x7f4a940cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.431+0000 7f4a91600640 1 -- 172.21.15.27:0/2461007142 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3876398501 0 0) 0x7f4a940cfd80 con 0x7f4a9415b4b0 2024-01-15T18:05:37.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.431+0000 7f4a91600640 1 -- 172.21.15.27:0/2461007142 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a74003530 con 0x7f4a9415b4b0 2024-01-15T18:05:37.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.431+0000 7f4a92a00640 1 -- 172.21.15.27:0/2461007142 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4a940b2590 msgr2=0x7f4a940cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:37.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.431+0000 7f4a92a00640 1 --2- 172.21.15.27:0/2461007142 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4a940b2590 0x7f4a940cc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.431+0000 7f4a92a00640 1 -- 172.21.15.27:0/2461007142 >> v1:172.21.15.27:6789/0 conn(0x7f4a9415b4b0 legacy=0x7f4a940cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.431+0000 7f4a92a00640 1 -- 172.21.15.27:0/2461007142 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a940b07c0 con 0x7f4a940cf460 2024-01-15T18:05:37.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.435+0000 7f4a92a00640 1 --2- 172.21.15.27:0/2461007142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a940cf460 0x7f4a940cf840 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4a840097c0 tx=0x7f4a8401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e5c40bcfbcddb9d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:37.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.435+0000 7f4a91600640 1 -- 172.21.15.27:0/2461007142 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a840ab680 con 0x7f4a940cf460 2024-01-15T18:05:37.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.435+0000 7f4a91600640 1 -- 172.21.15.27:0/2461007142 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a840ab7e0 con 0x7f4a940cf460 2024-01-15T18:05:37.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.435+0000 7f4a91600640 1 -- 172.21.15.27:0/2461007142 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a840abb70 con 0x7f4a940cf460 2024-01-15T18:05:37.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.435+0000 7f4a99800640 1 -- 172.21.15.27:0/2461007142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a940cf460 msgr2=0x7f4a940cf840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.435+0000 7f4a99800640 1 --2- 172.21.15.27:0/2461007142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a940cf460 0x7f4a940cf840 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4a840097c0 tx=0x7f4a8401c480 comp rx=0 tx=0).stop 2024-01-15T18:05:37.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.435+0000 7f4a99800640 1 -- 172.21.15.27:0/2461007142 shutdown_connections 2024-01-15T18:05:37.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.435+0000 7f4a99800640 1 --2- 172.21.15.27:0/2461007142 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4a940b2590 0x7f4a940cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.435+0000 7f4a99800640 1 --2- 172.21.15.27:0/2461007142 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a940cf460 0x7f4a940cf840 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.435+0000 7f4a99800640 1 -- 172.21.15.27:0/2461007142 >> 172.21.15.27:0/2461007142 conn(0x7f4a940d0e80 msgr2=0x7f4a940d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:37.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.435+0000 7f4a99800640 1 -- 172.21.15.27:0/2461007142 shutdown_connections 2024-01-15T18:05:37.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.435+0000 7f4a99800640 1 -- 172.21.15.27:0/2461007142 wait complete. 2024-01-15T18:05:37.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.435+0000 7f4a99800640 1 Processor -- start 2024-01-15T18:05:37.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.435+0000 7f4a99800640 1 -- start start 2024-01-15T18:05:37.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.439+0000 7f4a99800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4a940b2590 0x7f4a94148630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.439+0000 7f4a99800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4a940cf460 0x7f4a94148e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.439+0000 7f4a99800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a9415b4b0 0x7f4a94149380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.439+0000 7f4a99800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9414a6a0 con 0x7f4a9415b4b0 2024-01-15T18:05:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.439+0000 7f4a99800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a94150020 con 0x7f4a940b2590 2024-01-15T18:05:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.439+0000 7f4a99800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9415bb90 con 0x7f4a940cf460 2024-01-15T18:05:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.439+0000 7f4a93400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a9415b4b0 0x7f4a94149380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.439+0000 7f4a93400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a9415b4b0 0x7f4a94149380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34980/0 (socket says 172.21.15.27:34980) 2024-01-15T18:05:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.439+0000 7f4a93400640 1 -- 172.21.15.27:0/404525628 learned_addr learned my addr 172.21.15.27:0/404525628 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.439+0000 7f4a92000640 1 --2- 172.21.15.27:0/404525628 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4a940cf460 0x7f4a94148e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.439+0000 7f4a92a00640 1 --2- 172.21.15.27:0/404525628 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4a940b2590 0x7f4a94148630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.439+0000 7f4a93400640 1 -- 172.21.15.27:0/404525628 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4a940cf460 msgr2=0x7f4a94148e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.439+0000 7f4a93400640 1 --2- 172.21.15.27:0/404525628 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4a940cf460 0x7f4a94148e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.439+0000 7f4a93400640 1 -- 172.21.15.27:0/404525628 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4a940b2590 msgr2=0x7f4a94148630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.439+0000 7f4a93400640 1 --2- 172.21.15.27:0/404525628 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4a940b2590 0x7f4a94148630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.439+0000 7f4a93400640 1 -- 172.21.15.27:0/404525628 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a84009420 con 0x7f4a9415b4b0 2024-01-15T18:05:37.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.439+0000 7f4a92000640 1 --2- 172.21.15.27:0/404525628 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4a940cf460 0x7f4a94148e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:37.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.439+0000 7f4a92a00640 1 --2- 172.21.15.27:0/404525628 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4a940b2590 0x7f4a94148630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:37.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.439+0000 7f4a93400640 1 --2- 172.21.15.27:0/404525628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a9415b4b0 0x7f4a94149380 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00efc0 tx=0x7f4a8c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:37.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.439+0000 7f4a8aa00640 1 -- 172.21.15.27:0/404525628 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c009ba0 con 0x7f4a9415b4b0 2024-01-15T18:05:37.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.439+0000 7f4a8aa00640 1 -- 172.21.15.27:0/404525628 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a8c00f040 con 0x7f4a9415b4b0 2024-01-15T18:05:37.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.439+0000 7f4a99800640 1 -- 172.21.15.27:0/404525628 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a9415be70 con 0x7f4a9415b4b0 2024-01-15T18:05:37.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.439+0000 7f4a8aa00640 1 -- 172.21.15.27:0/404525628 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c0146c0 con 0x7f4a9415b4b0 2024-01-15T18:05:37.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.439+0000 7f4a99800640 1 -- 172.21.15.27:0/404525628 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a94148b70 con 0x7f4a9415b4b0 2024-01-15T18:05:37.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.439+0000 7f4a99800640 1 -- 172.21.15.27:0/404525628 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a9415b9f0 con 0x7f4a9415b4b0 2024-01-15T18:05:37.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.443+0000 7f4a8aa00640 1 -- 172.21.15.27:0/404525628 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f4a8c014820 con 0x7f4a9415b4b0 2024-01-15T18:05:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.443+0000 7f4a8aa00640 1 --2- 172.21.15.27:0/404525628 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4a6c0668f0 0x7f4a6c068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.443+0000 7f4a8aa00640 1 -- 172.21.15.27:0/404525628 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4a8c084fd0 con 0x7f4a9415b4b0 2024-01-15T18:05:37.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.443+0000 7f4a92a00640 1 --2- 172.21.15.27:0/404525628 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4a6c0668f0 0x7f4a6c068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.443+0000 7f4a92a00640 1 --2- 172.21.15.27:0/404525628 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4a6c0668f0 0x7f4a6c068db0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4a840af000 tx=0x7f4a84008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:37.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.447+0000 7f4a8aa00640 1 -- 172.21.15.27:0/404525628 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4a9415b9f0 con 0x7f4a9415b4b0 2024-01-15T18:05:37.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.603+0000 7f4a99800640 1 -- 172.21.15.27:0/404525628 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4a9415b9f0 con 0x7f4a9415b4b0 2024-01-15T18:05:37.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.603+0000 7f4a8aa00640 1 -- 172.21.15.27:0/404525628 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4a9415b9f0 con 0x7f4a9415b4b0 2024-01-15T18:05:37.608 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:05:37.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.603+0000 7f4a99800640 1 -- 172.21.15.27:0/404525628 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4a6c0668f0 msgr2=0x7f4a6c068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.603+0000 7f4a99800640 1 --2- 172.21.15.27:0/404525628 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4a6c0668f0 0x7f4a6c068db0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4a840af000 tx=0x7f4a84008040 comp rx=0 tx=0).stop 2024-01-15T18:05:37.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.603+0000 7f4a99800640 1 -- 172.21.15.27:0/404525628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a9415b4b0 msgr2=0x7f4a94149380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.603+0000 7f4a99800640 1 --2- 172.21.15.27:0/404525628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a9415b4b0 0x7f4a94149380 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00efc0 tx=0x7f4a8c00c490 comp rx=0 tx=0).stop 2024-01-15T18:05:37.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.603+0000 7f4a99800640 1 -- 172.21.15.27:0/404525628 shutdown_connections 2024-01-15T18:05:37.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.603+0000 7f4a99800640 1 --2- 172.21.15.27:0/404525628 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4a6c0668f0 0x7f4a6c068db0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.603+0000 7f4a99800640 1 --2- 172.21.15.27:0/404525628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a9415b4b0 0x7f4a94149380 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.603+0000 7f4a99800640 1 --2- 172.21.15.27:0/404525628 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4a940cf460 0x7f4a94148e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.603+0000 7f4a99800640 1 --2- 172.21.15.27:0/404525628 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4a940b2590 0x7f4a94148630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.603+0000 7f4a99800640 1 -- 172.21.15.27:0/404525628 >> 172.21.15.27:0/404525628 conn(0x7f4a940d0e80 msgr2=0x7f4a940ada70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:37.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.607+0000 7f4a99800640 1 -- 172.21.15.27:0/404525628 shutdown_connections 2024-01-15T18:05:37.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.607+0000 7f4a99800640 1 -- 172.21.15.27:0/404525628 wait complete. 2024-01-15T18:05:37.630 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T18:05:37.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.771+0000 7f3e49400640 1 Processor -- start 2024-01-15T18:05:37.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.771+0000 7f3e49400640 1 -- start start 2024-01-15T18:05:37.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.771+0000 7f3e49400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e4406c740 0x7f3e44099a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.771+0000 7f3e49400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3e44099f50 0x7f3e440cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.771+0000 7f3e49400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4415d860 con 0x7f3e4415d480 2024-01-15T18:05:37.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.771+0000 7f3e49400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4409a400 con 0x7f3e44099f50 2024-01-15T18:05:37.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.771+0000 7f3e49400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4406b6a0 con 0x7f3e4406c740 2024-01-15T18:05:37.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.771+0000 7f3e43200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3e44099f50 0x7f3e440cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.771+0000 7f3e43200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3e44099f50 0x7f3e440cb5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34986/0 (socket says 172.21.15.27:34986) 2024-01-15T18:05:37.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.771+0000 7f3e43200640 1 -- 172.21.15.27:0/1356318654 learned_addr learned my addr 172.21.15.27:0/1356318654 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:37.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.771+0000 7f3e41e00640 1 --2- 172.21.15.27:0/1356318654 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e4406c740 0x7f3e44099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.771+0000 7f3e43200640 1 -- 172.21.15.27:0/1356318654 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e4406c740 msgr2=0x7f3e44099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:37.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.771+0000 7f3e43200640 1 --2- 172.21.15.27:0/1356318654 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e4406c740 0x7f3e44099a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.771+0000 7f3e43200640 1 -- 172.21.15.27:0/1356318654 >> v1:172.21.15.93:6789/0 conn(0x7f3e4415d480 legacy=0x7f3e440cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.771+0000 7f3e43200640 1 -- 172.21.15.27:0/1356318654 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4406b980 con 0x7f3e44099f50 2024-01-15T18:05:37.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.775+0000 7f3e43200640 1 --2- 172.21.15.27:0/1356318654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e44099f50 0x7f3e440cb5a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c009e50 tx=0x7f3e3c01c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=665f6da36cb20d04 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:37.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.775+0000 7f3e41400640 1 -- 172.21.15.27:0/1356318654 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c0a64e0 con 0x7f3e44099f50 2024-01-15T18:05:37.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.775+0000 7f3e41400640 1 -- 172.21.15.27:0/1356318654 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3c0a6640 con 0x7f3e44099f50 2024-01-15T18:05:37.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.775+0000 7f3e41400640 1 -- 172.21.15.27:0/1356318654 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c0a69d0 con 0x7f3e44099f50 2024-01-15T18:05:37.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.775+0000 7f3e49400640 1 -- 172.21.15.27:0/1356318654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e44099f50 msgr2=0x7f3e440cb5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.775+0000 7f3e49400640 1 --2- 172.21.15.27:0/1356318654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e44099f50 0x7f3e440cb5a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c009e50 tx=0x7f3e3c01c9e0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.775+0000 7f3e49400640 1 -- 172.21.15.27:0/1356318654 shutdown_connections 2024-01-15T18:05:37.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.775+0000 7f3e49400640 1 --2- 172.21.15.27:0/1356318654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e44099f50 0x7f3e440cb5a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.775+0000 7f3e49400640 1 --2- 172.21.15.27:0/1356318654 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e4406c740 0x7f3e44099a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.775+0000 7f3e49400640 1 -- 172.21.15.27:0/1356318654 >> 172.21.15.27:0/1356318654 conn(0x7f3e440b2a80 msgr2=0x7f3e440b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:37.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.775+0000 7f3e49400640 1 -- 172.21.15.27:0/1356318654 shutdown_connections 2024-01-15T18:05:37.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.775+0000 7f3e49400640 1 -- 172.21.15.27:0/1356318654 wait complete. 2024-01-15T18:05:37.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.779+0000 7f3e49400640 1 Processor -- start 2024-01-15T18:05:37.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.779+0000 7f3e49400640 1 -- start start 2024-01-15T18:05:37.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.779+0000 7f3e49400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e4406c740 0x7f3e4414aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.779+0000 7f3e49400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3e44099f50 0x7f3e4415b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.779+0000 7f3e49400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e4415d480 0x7f3e4415bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.779+0000 7f3e49400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e44153d60 con 0x7f3e4415d480 2024-01-15T18:05:37.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.779+0000 7f3e49400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e440c60f0 con 0x7f3e44099f50 2024-01-15T18:05:37.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.779+0000 7f3e49400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e440c63d0 con 0x7f3e4406c740 2024-01-15T18:05:37.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.779+0000 7f3e43200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e4415d480 0x7f3e4415bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.779+0000 7f3e43200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e4415d480 0x7f3e4415bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34992/0 (socket says 172.21.15.27:34992) 2024-01-15T18:05:37.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.779+0000 7f3e43200640 1 -- 172.21.15.27:0/1315980429 learned_addr learned my addr 172.21.15.27:0/1315980429 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:37.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.779+0000 7f3e41e00640 1 --2- 172.21.15.27:0/1315980429 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3e44099f50 0x7f3e4415b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.779+0000 7f3e42800640 1 --2- 172.21.15.27:0/1315980429 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e4406c740 0x7f3e4414aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.779+0000 7f3e43200640 1 -- 172.21.15.27:0/1315980429 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e4406c740 msgr2=0x7f3e4414aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.779+0000 7f3e43200640 1 --2- 172.21.15.27:0/1315980429 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e4406c740 0x7f3e4414aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.779+0000 7f3e43200640 1 -- 172.21.15.27:0/1315980429 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3e44099f50 msgr2=0x7f3e4415b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.779+0000 7f3e43200640 1 --2- 172.21.15.27:0/1315980429 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3e44099f50 0x7f3e4415b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.779+0000 7f3e43200640 1 -- 172.21.15.27:0/1315980429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3c009b30 con 0x7f3e4415d480 2024-01-15T18:05:37.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.779+0000 7f3e43200640 1 --2- 172.21.15.27:0/1315980429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e4415d480 0x7f3e4415bd50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c01b090 tx=0x7f3e3c019780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.779+0000 7f3e3aa00640 1 -- 172.21.15.27:0/1315980429 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c0b4070 con 0x7f3e4415d480 2024-01-15T18:05:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.779+0000 7f3e3aa00640 1 -- 172.21.15.27:0/1315980429 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3c0091d0 con 0x7f3e4415d480 2024-01-15T18:05:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.779+0000 7f3e3aa00640 1 -- 172.21.15.27:0/1315980429 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c0a6a10 con 0x7f3e4415d480 2024-01-15T18:05:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.779+0000 7f3e49400640 1 -- 172.21.15.27:0/1315980429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e440c66b0 con 0x7f3e4415d480 2024-01-15T18:05:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.779+0000 7f3e49400640 1 -- 172.21.15.27:0/1315980429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e4414afe0 con 0x7f3e4415d480 2024-01-15T18:05:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.779+0000 7f3e49400640 1 -- 172.21.15.27:0/1315980429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e4404e850 con 0x7f3e4415d480 2024-01-15T18:05:37.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.783+0000 7f3e3aa00640 1 -- 172.21.15.27:0/1315980429 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f3e3c0b8070 con 0x7f3e4415d480 2024-01-15T18:05:37.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.783+0000 7f3e3aa00640 1 --2- 172.21.15.27:0/1315980429 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e1c0668f0 0x7f3e1c068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.783+0000 7f3e3aa00640 1 -- 172.21.15.27:0/1315980429 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e3c0aa070 con 0x7f3e4415d480 2024-01-15T18:05:37.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.783+0000 7f3e42800640 1 --2- 172.21.15.27:0/1315980429 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e1c0668f0 0x7f3e1c068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.783+0000 7f3e42800640 1 --2- 172.21.15.27:0/1315980429 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e1c0668f0 0x7f3e1c068db0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3e34000f70 tx=0x7f3e34009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:37.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.787+0000 7f3e3aa00640 1 -- 172.21.15.27:0/1315980429 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3e3c0eb160 con 0x7f3e4415d480 2024-01-15T18:05:37.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.947+0000 7f3e49400640 1 -- 172.21.15.27:0/1315980429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3e440d09e0 con 0x7f3e4415d480 2024-01-15T18:05:37.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.947+0000 7f3e3aa00640 1 -- 172.21.15.27:0/1315980429 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3e3c0ee780 con 0x7f3e4415d480 2024-01-15T18:05:37.952 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-15T18:05:37.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.947+0000 7f3e49400640 1 -- 172.21.15.27:0/1315980429 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e1c0668f0 msgr2=0x7f3e1c068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.947+0000 7f3e49400640 1 --2- 172.21.15.27:0/1315980429 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e1c0668f0 0x7f3e1c068db0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3e34000f70 tx=0x7f3e34009040 comp rx=0 tx=0).stop 2024-01-15T18:05:37.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.947+0000 7f3e49400640 1 -- 172.21.15.27:0/1315980429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e4415d480 msgr2=0x7f3e4415bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.947+0000 7f3e49400640 1 --2- 172.21.15.27:0/1315980429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e4415d480 0x7f3e4415bd50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c01b090 tx=0x7f3e3c019780 comp rx=0 tx=0).stop 2024-01-15T18:05:37.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.947+0000 7f3e49400640 1 -- 172.21.15.27:0/1315980429 shutdown_connections 2024-01-15T18:05:37.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.947+0000 7f3e49400640 1 --2- 172.21.15.27:0/1315980429 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e1c0668f0 0x7f3e1c068db0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.947+0000 7f3e49400640 1 --2- 172.21.15.27:0/1315980429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e4415d480 0x7f3e4415bd50 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.947+0000 7f3e49400640 1 --2- 172.21.15.27:0/1315980429 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3e44099f50 0x7f3e4415b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.947+0000 7f3e49400640 1 --2- 172.21.15.27:0/1315980429 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e4406c740 0x7f3e4414aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.947+0000 7f3e49400640 1 -- 172.21.15.27:0/1315980429 >> 172.21.15.27:0/1315980429 conn(0x7f3e440b2a80 msgr2=0x7f3e44161840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:37.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.947+0000 7f3e49400640 1 -- 172.21.15.27:0/1315980429 shutdown_connections 2024-01-15T18:05:37.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:37.947+0000 7f3e49400640 1 -- 172.21.15.27:0/1315980429 wait complete. 2024-01-15T18:05:37.974 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T18:05:38.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.111+0000 7f44f3800640 1 Processor -- start 2024-01-15T18:05:38.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.111+0000 7f44f3800640 1 -- start start 2024-01-15T18:05:38.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44f3800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f44ec0cf460 0x7f44ec0cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44f3800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44ec0cfd80 con 0x7f44ec15b4b0 2024-01-15T18:05:38.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44f3800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44ec0d19a0 con 0x7f44ec0b2590 2024-01-15T18:05:38.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44f3800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44ec0d1c80 con 0x7f44ec0cf460 2024-01-15T18:05:38.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44ebe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f44ec15b4b0 0x7f44ec0cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35106/0 (socket says 172.21.15.27:35106) 2024-01-15T18:05:38.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44ebe00640 1 -- 172.21.15.27:0/3665352265 learned_addr learned my addr 172.21.15.27:0/3665352265 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:38.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44f0e00640 1 --2- 172.21.15.27:0/3665352265 >> v2:172.21.15.93:3300/0 conn(0x7f44ec0cf460 0x7f44ec0cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:38.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44eb400640 1 -- 172.21.15.27:0/3665352265 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663522947 0 0) 0x7f44ec0cfd80 con 0x7f44ec15b4b0 2024-01-15T18:05:38.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44eb400640 1 -- 172.21.15.27:0/3665352265 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44cc003530 con 0x7f44ec15b4b0 2024-01-15T18:05:38.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44eb400640 1 -- 172.21.15.27:0/3665352265 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 745937046 0 0) 0x7f44cc003530 con 0x7f44ec15b4b0 2024-01-15T18:05:38.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44eb400640 1 -- 172.21.15.27:0/3665352265 >> v2:172.21.15.93:3300/0 conn(0x7f44ec0cf460 msgr2=0x7f44ec0cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:38.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44eb400640 1 --2- 172.21.15.27:0/3665352265 >> v2:172.21.15.93:3300/0 conn(0x7f44ec0cf460 0x7f44ec0cf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44eb400640 1 -- 172.21.15.27:0/3665352265 >> v1:172.21.15.93:6789/0 conn(0x7f44ec0b2590 legacy=0x7f44ec0cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:38.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44eb400640 1 -- 172.21.15.27:0/3665352265 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44ec0b07c0 con 0x7f44ec15b4b0 2024-01-15T18:05:38.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44eb400640 1 -- 172.21.15.27:0/3665352265 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f44d4001680 con 0x7f44ec15b4b0 2024-01-15T18:05:38.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44eb400640 1 -- 172.21.15.27:0/3665352265 >> v1:172.21.15.27:6789/0 conn(0x7f44ec15b4b0 legacy=0x7f44ec0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:38.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44eb400640 1 --2- 172.21.15.27:0/3665352265 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44cc003a40 0x7f44cc003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44eb400640 1 --2- 172.21.15.27:0/3665352265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44cc0047d0 0x7f44cc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44eb400640 1 --2- 172.21.15.27:0/3665352265 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44cc0051e0 0x7f44cc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44eb400640 1 -- 172.21.15.27:0/3665352265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ec0d1c80 con 0x7f44cc0047d0 2024-01-15T18:05:38.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44eb400640 1 -- 172.21.15.27:0/3665352265 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44cc00c110 con 0x7f44cc003a40 2024-01-15T18:05:38.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44eb400640 1 -- 172.21.15.27:0/3665352265 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44cc00c2d0 con 0x7f44cc0051e0 2024-01-15T18:05:38.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44ebe00640 1 --2- 172.21.15.27:0/3665352265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44cc0047d0 0x7f44cc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:38.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44f0e00640 1 --2- 172.21.15.27:0/3665352265 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44cc003a40 0x7f44cc003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:38.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44ebe00640 1 -- 172.21.15.27:0/3665352265 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44cc0051e0 msgr2=0x7f44cc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:38.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44f1800640 1 --2- 172.21.15.27:0/3665352265 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44cc0051e0 0x7f44cc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:38.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44ebe00640 1 --2- 172.21.15.27:0/3665352265 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44cc0051e0 0x7f44cc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44ebe00640 1 -- 172.21.15.27:0/3665352265 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44cc003a40 msgr2=0x7f44cc003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:38.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44ebe00640 1 --2- 172.21.15.27:0/3665352265 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44cc003a40 0x7f44cc003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44ebe00640 1 -- 172.21.15.27:0/3665352265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44cc003530 con 0x7f44cc0047d0 2024-01-15T18:05:38.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44f1800640 1 --2- 172.21.15.27:0/3665352265 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44cc0051e0 0x7f44cc00b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:38.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44f0e00640 1 --2- 172.21.15.27:0/3665352265 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44cc003a40 0x7f44cc003e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:38.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.115+0000 7f44f1800640 1 -- 172.21.15.27:0/3665352265 reap_dead start 2024-01-15T18:05:38.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.119+0000 7f44ebe00640 1 --2- 172.21.15.27:0/3665352265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44cc0047d0 0x7f44cc004c50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f44d400c0e0 tx=0x7f44d401f2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b72048fe74f48a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:38.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.119+0000 7f44eb400640 1 -- 172.21.15.27:0/3665352265 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d40058e0 con 0x7f44cc0047d0 2024-01-15T18:05:38.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.119+0000 7f44eb400640 1 -- 172.21.15.27:0/3665352265 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d4005a40 con 0x7f44cc0047d0 2024-01-15T18:05:38.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.119+0000 7f44eb400640 1 -- 172.21.15.27:0/3665352265 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d4005e70 con 0x7f44cc0047d0 2024-01-15T18:05:38.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.119+0000 7f44f3800640 1 -- 172.21.15.27:0/3665352265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44cc0047d0 msgr2=0x7f44cc004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:38.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.119+0000 7f44f3800640 1 --2- 172.21.15.27:0/3665352265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44cc0047d0 0x7f44cc004c50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f44d400c0e0 tx=0x7f44d401f2d0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.119+0000 7f44f3800640 1 -- 172.21.15.27:0/3665352265 shutdown_connections 2024-01-15T18:05:38.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.119+0000 7f44f3800640 1 --2- 172.21.15.27:0/3665352265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44cc0047d0 0x7f44cc004c50 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.119+0000 7f44f3800640 1 -- 172.21.15.27:0/3665352265 >> 172.21.15.27:0/3665352265 conn(0x7f44ec0d0e80 msgr2=0x7f44ec0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:38.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.119+0000 7f44f3800640 1 -- 172.21.15.27:0/3665352265 shutdown_connections 2024-01-15T18:05:38.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.119+0000 7f44f3800640 1 -- 172.21.15.27:0/3665352265 wait complete. 2024-01-15T18:05:38.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.123+0000 7f44f3800640 1 Processor -- start 2024-01-15T18:05:38.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.123+0000 7f44f3800640 1 -- start start 2024-01-15T18:05:38.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.123+0000 7f44f3800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44cc0047d0 0x7f44ec09bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.123+0000 7f44f3800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44ec158bf0 0x7f44ec09e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.123+0000 7f44f3800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44ec09e6f0 0x7f44ec0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.123+0000 7f44f3800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ec09c1b0 con 0x7f44cc0047d0 2024-01-15T18:05:38.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.123+0000 7f44f3800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ec09ac60 con 0x7f44ec09e6f0 2024-01-15T18:05:38.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.123+0000 7f44f3800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44ec09aeb0 con 0x7f44ec158bf0 2024-01-15T18:05:38.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.123+0000 7f44f0e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44cc0047d0 0x7f44ec09bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:38.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.123+0000 7f44f0e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44cc0047d0 0x7f44ec09bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35008/0 (socket says 172.21.15.27:35008) 2024-01-15T18:05:38.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.123+0000 7f44f0e00640 1 -- 172.21.15.27:0/3955581714 learned_addr learned my addr 172.21.15.27:0/3955581714 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:38.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.123+0000 7f44ebe00640 1 --2- 172.21.15.27:0/3955581714 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44ec158bf0 0x7f44ec09e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:38.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.123+0000 7f44f1800640 1 --2- 172.21.15.27:0/3955581714 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44ec09e6f0 0x7f44ec0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:38.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.123+0000 7f44f0e00640 1 -- 172.21.15.27:0/3955581714 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44ec158bf0 msgr2=0x7f44ec09e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:38.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.123+0000 7f44f0e00640 1 --2- 172.21.15.27:0/3955581714 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44ec158bf0 0x7f44ec09e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.123+0000 7f44f0e00640 1 -- 172.21.15.27:0/3955581714 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44ec09e6f0 msgr2=0x7f44ec0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:38.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.123+0000 7f44f0e00640 1 --2- 172.21.15.27:0/3955581714 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44ec09e6f0 0x7f44ec0cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.123+0000 7f44f0e00640 1 -- 172.21.15.27:0/3955581714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44ec09b190 con 0x7f44cc0047d0 2024-01-15T18:05:38.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.123+0000 7f44f0e00640 1 --2- 172.21.15.27:0/3955581714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44cc0047d0 0x7f44ec09bc70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f44dc011ca0 tx=0x7f44dc00f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:38.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.123+0000 7f44e8c00640 1 -- 172.21.15.27:0/3955581714 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44dc00feb0 con 0x7f44cc0047d0 2024-01-15T18:05:38.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.123+0000 7f44e8c00640 1 -- 172.21.15.27:0/3955581714 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44dc011e50 con 0x7f44cc0047d0 2024-01-15T18:05:38.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.123+0000 7f44f3800640 1 -- 172.21.15.27:0/3955581714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44ec09b470 con 0x7f44cc0047d0 2024-01-15T18:05:38.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.123+0000 7f44e8c00640 1 -- 172.21.15.27:0/3955581714 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44dc004260 con 0x7f44cc0047d0 2024-01-15T18:05:38.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.123+0000 7f44f3800640 1 -- 172.21.15.27:0/3955581714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44ec159630 con 0x7f44cc0047d0 2024-01-15T18:05:38.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.127+0000 7f44f3800640 1 -- 172.21.15.27:0/3955581714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44ec04e850 con 0x7f44cc0047d0 2024-01-15T18:05:38.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.127+0000 7f44e8c00640 1 -- 172.21.15.27:0/3955581714 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f44dc0043c0 con 0x7f44cc0047d0 2024-01-15T18:05:38.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.127+0000 7f44e8c00640 1 --2- 172.21.15.27:0/3955581714 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f44c4066920 0x7f44c4068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.127+0000 7f44e8c00640 1 -- 172.21.15.27:0/3955581714 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f44dc084ad0 con 0x7f44cc0047d0 2024-01-15T18:05:38.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.127+0000 7f44ebe00640 1 --2- 172.21.15.27:0/3955581714 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f44c4066920 0x7f44c4068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:38.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.127+0000 7f44ebe00640 1 --2- 172.21.15.27:0/3955581714 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f44c4066920 0x7f44c4068de0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f44d40ae000 tx=0x7f44d40aa070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:38.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.131+0000 7f44e8c00640 1 -- 172.21.15.27:0/3955581714 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f44dc051da0 con 0x7f44cc0047d0 2024-01-15T18:05:38.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.287+0000 7f44f3800640 1 -- 172.21.15.27:0/3955581714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f44ec0cf240 con 0x7f44cc0047d0 2024-01-15T18:05:38.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.287+0000 7f44e8c00640 1 -- 172.21.15.27:0/3955581714 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f44dc022020 con 0x7f44cc0047d0 2024-01-15T18:05:38.293 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-15T18:05:38.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.287+0000 7f44f3800640 1 -- 172.21.15.27:0/3955581714 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f44c4066920 msgr2=0x7f44c4068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:38.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.287+0000 7f44f3800640 1 --2- 172.21.15.27:0/3955581714 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f44c4066920 0x7f44c4068de0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f44d40ae000 tx=0x7f44d40aa070 comp rx=0 tx=0).stop 2024-01-15T18:05:38.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.287+0000 7f44f3800640 1 -- 172.21.15.27:0/3955581714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44cc0047d0 msgr2=0x7f44ec09bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:38.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.287+0000 7f44f3800640 1 --2- 172.21.15.27:0/3955581714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44cc0047d0 0x7f44ec09bc70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f44dc011ca0 tx=0x7f44dc00f610 comp rx=0 tx=0).stop 2024-01-15T18:05:38.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.291+0000 7f44f3800640 1 -- 172.21.15.27:0/3955581714 shutdown_connections 2024-01-15T18:05:38.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.291+0000 7f44f3800640 1 --2- 172.21.15.27:0/3955581714 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f44c4066920 0x7f44c4068de0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.291+0000 7f44f3800640 1 --2- 172.21.15.27:0/3955581714 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44ec09e6f0 0x7f44ec0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.291+0000 7f44f3800640 1 --2- 172.21.15.27:0/3955581714 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44ec158bf0 0x7f44ec09e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.291+0000 7f44f3800640 1 --2- 172.21.15.27:0/3955581714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44cc0047d0 0x7f44ec09bc70 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.291+0000 7f44f3800640 1 -- 172.21.15.27:0/3955581714 >> 172.21.15.27:0/3955581714 conn(0x7f44ec0d0e80 msgr2=0x7f44ec15a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:38.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.291+0000 7f44f3800640 1 -- 172.21.15.27:0/3955581714 shutdown_connections 2024-01-15T18:05:38.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.291+0000 7f44f3800640 1 -- 172.21.15.27:0/3955581714 wait complete. 2024-01-15T18:05:38.316 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:05:38.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.451+0000 7f4452c00640 1 Processor -- start 2024-01-15T18:05:38.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.451+0000 7f4452c00640 1 -- start start 2024-01-15T18:05:38.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.455+0000 7f4452c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f444c15ac50 0x7f444c0b2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.455+0000 7f4452c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f444c0b2ba0 0x7f444c071020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.455+0000 7f4452c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f444c071560 0x7f444c06c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.455+0000 7f4452c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f444c070b90 con 0x7f444c071560 2024-01-15T18:05:38.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.455+0000 7f4452c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f444c072020 con 0x7f444c15ac50 2024-01-15T18:05:38.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.455+0000 7f4452c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f444c072300 con 0x7f444c0b2ba0 2024-01-15T18:05:38.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.455+0000 7f4450c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f444c071560 0x7f444c06c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:38.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.455+0000 7f4450c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f444c071560 0x7f444c06c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35010/0 (socket says 172.21.15.27:35010) 2024-01-15T18:05:38.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.455+0000 7f4450c00640 1 -- 172.21.15.27:0/702543818 learned_addr learned my addr 172.21.15.27:0/702543818 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:38.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.455+0000 7f444be00640 1 --2- 172.21.15.27:0/702543818 >> v2:172.21.15.93:3300/0 conn(0x7f444c15ac50 0x7f444c0b2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:38.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.455+0000 7f444b400640 1 --2- 172.21.15.27:0/702543818 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f444c0b2ba0 0x7f444c071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:38.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.455+0000 7f4450c00640 1 -- 172.21.15.27:0/702543818 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f444c0b2ba0 msgr2=0x7f444c071020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:38.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.455+0000 7f4450c00640 1 --2- 172.21.15.27:0/702543818 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f444c0b2ba0 0x7f444c071020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.455+0000 7f4450c00640 1 -- 172.21.15.27:0/702543818 >> v2:172.21.15.93:3300/0 conn(0x7f444c15ac50 msgr2=0x7f444c0b2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:38.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.455+0000 7f4450c00640 1 --2- 172.21.15.27:0/702543818 >> v2:172.21.15.93:3300/0 conn(0x7f444c15ac50 0x7f444c0b2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.455+0000 7f4450c00640 1 -- 172.21.15.27:0/702543818 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f444c070820 con 0x7f444c071560 2024-01-15T18:05:38.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.459+0000 7f4450c00640 1 --2- 172.21.15.27:0/702543818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f444c071560 0x7f444c06c6d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f444c0cfd40 tx=0x7f444401c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8dc001963a9169 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:38.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.459+0000 7f444aa00640 1 -- 172.21.15.27:0/702543818 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44440a5760 con 0x7f444c071560 2024-01-15T18:05:38.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.459+0000 7f444aa00640 1 -- 172.21.15.27:0/702543818 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44440a58c0 con 0x7f444c071560 2024-01-15T18:05:38.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.459+0000 7f444aa00640 1 -- 172.21.15.27:0/702543818 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44440a5c50 con 0x7f444c071560 2024-01-15T18:05:38.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.459+0000 7f4452c00640 1 -- 172.21.15.27:0/702543818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f444c071560 msgr2=0x7f444c06c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:38.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.459+0000 7f4452c00640 1 --2- 172.21.15.27:0/702543818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f444c071560 0x7f444c06c6d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f444c0cfd40 tx=0x7f444401c9d0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.459+0000 7f4452c00640 1 -- 172.21.15.27:0/702543818 shutdown_connections 2024-01-15T18:05:38.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.459+0000 7f4452c00640 1 --2- 172.21.15.27:0/702543818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f444c071560 0x7f444c06c6d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.459+0000 7f4452c00640 1 --2- 172.21.15.27:0/702543818 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f444c0b2ba0 0x7f444c071020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.459+0000 7f4452c00640 1 --2- 172.21.15.27:0/702543818 >> v2:172.21.15.93:3300/0 conn(0x7f444c15ac50 0x7f444c0b2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.459+0000 7f4452c00640 1 -- 172.21.15.27:0/702543818 >> 172.21.15.27:0/702543818 conn(0x7f444c0b0850 msgr2=0x7f444c0b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:38.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.459+0000 7f4452c00640 1 -- 172.21.15.27:0/702543818 shutdown_connections 2024-01-15T18:05:38.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.459+0000 7f4452c00640 1 -- 172.21.15.27:0/702543818 wait complete. 2024-01-15T18:05:38.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.459+0000 7f4452c00640 1 Processor -- start 2024-01-15T18:05:38.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.459+0000 7f4452c00640 1 -- start start 2024-01-15T18:05:38.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.463+0000 7f4452c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f444c071560 0x7f444c151a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.463+0000 7f4452c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f444c0b2ba0 0x7f444c156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.463+0000 7f4452c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f444c15ac50 0x7f444c1565c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.463+0000 7f4452c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444c158270 con 0x7f444c0b2ba0 2024-01-15T18:05:38.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.463+0000 7f4452c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444c158550 con 0x7f444c071560 2024-01-15T18:05:38.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.463+0000 7f4452c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f444c158830 con 0x7f444c15ac50 2024-01-15T18:05:38.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.463+0000 7f444b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f444c0b2ba0 0x7f444c156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:38.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.463+0000 7f444b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f444c0b2ba0 0x7f444c156080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35018/0 (socket says 172.21.15.27:35018) 2024-01-15T18:05:38.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.463+0000 7f444b400640 1 -- 172.21.15.27:0/955218216 learned_addr learned my addr 172.21.15.27:0/955218216 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:38.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.463+0000 7f444be00640 1 --2- 172.21.15.27:0/955218216 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f444c071560 0x7f444c151a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:38.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.463+0000 7f4450c00640 1 --2- 172.21.15.27:0/955218216 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f444c15ac50 0x7f444c1565c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:38.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.463+0000 7f444b400640 1 -- 172.21.15.27:0/955218216 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f444c15ac50 msgr2=0x7f444c1565c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:38.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.463+0000 7f444b400640 1 --2- 172.21.15.27:0/955218216 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f444c15ac50 0x7f444c1565c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.463+0000 7f444b400640 1 -- 172.21.15.27:0/955218216 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f444c071560 msgr2=0x7f444c151a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:38.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.463+0000 7f444b400640 1 --2- 172.21.15.27:0/955218216 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f444c071560 0x7f444c151a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.463+0000 7f444b400640 1 -- 172.21.15.27:0/955218216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4444009b30 con 0x7f444c0b2ba0 2024-01-15T18:05:38.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.463+0000 7f444b400640 1 --2- 172.21.15.27:0/955218216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f444c0b2ba0 0x7f444c156080 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f443400d800 tx=0x7f443400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:38.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.463+0000 7f4443e00640 1 -- 172.21.15.27:0/955218216 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44340096a0 con 0x7f444c0b2ba0 2024-01-15T18:05:38.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.463+0000 7f4443e00640 1 -- 172.21.15.27:0/955218216 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4434009800 con 0x7f444c0b2ba0 2024-01-15T18:05:38.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.463+0000 7f4452c00640 1 -- 172.21.15.27:0/955218216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f444c158b10 con 0x7f444c0b2ba0 2024-01-15T18:05:38.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.463+0000 7f4443e00640 1 -- 172.21.15.27:0/955218216 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4434009b70 con 0x7f444c0b2ba0 2024-01-15T18:05:38.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.463+0000 7f4452c00640 1 -- 172.21.15.27:0/955218216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f444c157e20 con 0x7f444c0b2ba0 2024-01-15T18:05:38.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.463+0000 7f4452c00640 1 -- 172.21.15.27:0/955218216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f444c04e850 con 0x7f444c0b2ba0 2024-01-15T18:05:38.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.463+0000 7f4443e00640 1 -- 172.21.15.27:0/955218216 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f4434009cd0 con 0x7f444c0b2ba0 2024-01-15T18:05:38.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.467+0000 7f4443e00640 1 --2- 172.21.15.27:0/955218216 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4424066920 0x7f4424068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.467+0000 7f4443e00640 1 -- 172.21.15.27:0/955218216 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4434084aa0 con 0x7f444c0b2ba0 2024-01-15T18:05:38.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.467+0000 7f444be00640 1 --2- 172.21.15.27:0/955218216 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4424066920 0x7f4424068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:38.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.467+0000 7f444be00640 1 --2- 172.21.15.27:0/955218216 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4424066920 0x7f4424068de0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f444c1f9c80 tx=0x7f443c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:38.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.471+0000 7f4443e00640 1 -- 172.21.15.27:0/955218216 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4434051cf0 con 0x7f444c0b2ba0 2024-01-15T18:05:38.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.639+0000 7f4452c00640 1 -- 172.21.15.27:0/955218216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f444c0677a0 con 0x7f444c0b2ba0 2024-01-15T18:05:38.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.639+0000 7f4443e00640 1 -- 172.21.15.27:0/955218216 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4434055310 con 0x7f444c0b2ba0 2024-01-15T18:05:38.643 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-01-15T18:05:38.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.639+0000 7f4452c00640 1 -- 172.21.15.27:0/955218216 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4424066920 msgr2=0x7f4424068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:38.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.639+0000 7f4452c00640 1 --2- 172.21.15.27:0/955218216 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4424066920 0x7f4424068de0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f444c1f9c80 tx=0x7f443c008040 comp rx=0 tx=0).stop 2024-01-15T18:05:38.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.639+0000 7f4452c00640 1 -- 172.21.15.27:0/955218216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f444c0b2ba0 msgr2=0x7f444c156080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:38.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.639+0000 7f4452c00640 1 --2- 172.21.15.27:0/955218216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f444c0b2ba0 0x7f444c156080 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f443400d800 tx=0x7f443400dcc0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.639+0000 7f4452c00640 1 -- 172.21.15.27:0/955218216 shutdown_connections 2024-01-15T18:05:38.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.639+0000 7f4452c00640 1 --2- 172.21.15.27:0/955218216 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4424066920 0x7f4424068de0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.639+0000 7f4452c00640 1 --2- 172.21.15.27:0/955218216 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f444c15ac50 0x7f444c1565c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.639+0000 7f4452c00640 1 --2- 172.21.15.27:0/955218216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f444c0b2ba0 0x7f444c156080 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.639+0000 7f4452c00640 1 --2- 172.21.15.27:0/955218216 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f444c071560 0x7f444c151a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.639+0000 7f4452c00640 1 -- 172.21.15.27:0/955218216 >> 172.21.15.27:0/955218216 conn(0x7f444c0b0850 msgr2=0x7f444c1546f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:38.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.639+0000 7f4452c00640 1 -- 172.21.15.27:0/955218216 shutdown_connections 2024-01-15T18:05:38.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:38.639+0000 7f4452c00640 1 -- 172.21.15.27:0/955218216 wait complete. 2024-01-15T18:05:38.665 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi093.front.sepia.ceph.com... 2024-01-15T18:05:38.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:38.665 DEBUG:teuthology.orchestra.run.smithi093:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-15T18:05:38.702 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:05:38.703 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T18:05:38.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:38.703 DEBUG:teuthology.orchestra.run.smithi093:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-15T18:05:38.778 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:05:38.778 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:05:38.787 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:05:38.787 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:05:38.838 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:39.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.023+0000 7fd70a800640 1 Processor -- start 2024-01-15T18:05:39.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.023+0000 7fd70a800640 1 -- start start 2024-01-15T18:05:39.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.023+0000 7fd70a800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd7040701a0 0x7fd704070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.023+0000 7fd70a800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd70415c130 0x7fd70415d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.023+0000 7fd70a800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd70406c4a0 con 0x7fd7040c92b0 2024-01-15T18:05:39.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.023+0000 7fd70a800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd704070ac0 con 0x7fd7040701a0 2024-01-15T18:05:39.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.023+0000 7fd70a800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd70415c660 con 0x7fd70415c130 2024-01-15T18:05:39.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.023+0000 7fd703400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd7040701a0 0x7fd704070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.023+0000 7fd703400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd7040701a0 0x7fd704070580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35024/0 (socket says 172.21.15.27:35024) 2024-01-15T18:05:39.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.023+0000 7fd703400640 1 -- 172.21.15.27:0/3134876486 learned_addr learned my addr 172.21.15.27:0/3134876486 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:39.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.023+0000 7fd703e00640 1 --2- 172.21.15.27:0/3134876486 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd70415c130 0x7fd70415d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.023+0000 7fd703400640 1 -- 172.21.15.27:0/3134876486 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd70415c130 msgr2=0x7fd70415d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:39.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.023+0000 7fd703400640 1 --2- 172.21.15.27:0/3134876486 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd70415c130 0x7fd70415d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.023+0000 7fd703400640 1 -- 172.21.15.27:0/3134876486 >> v1:172.21.15.93:6789/0 conn(0x7fd7040c92b0 legacy=0x7fd70406dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.023+0000 7fd703400640 1 -- 172.21.15.27:0/3134876486 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd70415c940 con 0x7fd7040701a0 2024-01-15T18:05:39.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.023+0000 7fd702000640 1 -- 172.21.15.27:0/3134876486 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1219597442 0 0) 0x7fd70406c4a0 con 0x7fd7040c92b0 2024-01-15T18:05:39.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.027+0000 7fd703400640 1 --2- 172.21.15.27:0/3134876486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7040701a0 0x7fd704070580 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4009870 tx=0x7fd6f401c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36d0f367c4f54f97 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:39.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.027+0000 7fd702000640 1 -- 172.21.15.27:0/3134876486 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f40aa670 con 0x7fd7040701a0 2024-01-15T18:05:39.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.027+0000 7fd702000640 1 -- 172.21.15.27:0/3134876486 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6f40aa7d0 con 0x7fd7040701a0 2024-01-15T18:05:39.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.027+0000 7fd702000640 1 -- 172.21.15.27:0/3134876486 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f40aab60 con 0x7fd7040701a0 2024-01-15T18:05:39.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.027+0000 7fd70a800640 1 -- 172.21.15.27:0/3134876486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7040701a0 msgr2=0x7fd704070580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.027+0000 7fd70a800640 1 --2- 172.21.15.27:0/3134876486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7040701a0 0x7fd704070580 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4009870 tx=0x7fd6f401c510 comp rx=0 tx=0).stop 2024-01-15T18:05:39.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.027+0000 7fd70a800640 1 -- 172.21.15.27:0/3134876486 shutdown_connections 2024-01-15T18:05:39.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.027+0000 7fd70a800640 1 --2- 172.21.15.27:0/3134876486 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd70415c130 0x7fd70415d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.027+0000 7fd70a800640 1 --2- 172.21.15.27:0/3134876486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7040701a0 0x7fd704070580 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.027+0000 7fd70a800640 1 -- 172.21.15.27:0/3134876486 >> 172.21.15.27:0/3134876486 conn(0x7fd7040cec60 msgr2=0x7fd7040cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:39.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.027+0000 7fd70a800640 1 -- 172.21.15.27:0/3134876486 shutdown_connections 2024-01-15T18:05:39.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.027+0000 7fd70a800640 1 -- 172.21.15.27:0/3134876486 wait complete. 2024-01-15T18:05:39.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.031+0000 7fd70a800640 1 Processor -- start 2024-01-15T18:05:39.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.031+0000 7fd70a800640 1 -- start start 2024-01-15T18:05:39.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.031+0000 7fd70a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7040701a0 0x7fd704158110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.031+0000 7fd70a800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd7040c92b0 0x7fd704165250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.031+0000 7fd70a800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd70415c130 0x7fd7041516d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.031+0000 7fd70a800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7040c6640 con 0x7fd7040701a0 2024-01-15T18:05:39.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.031+0000 7fd70a800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7040c5100 con 0x7fd7040c92b0 2024-01-15T18:05:39.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.031+0000 7fd70a800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7040c5d30 con 0x7fd70415c130 2024-01-15T18:05:39.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.031+0000 7fd703400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7040701a0 0x7fd704158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.031+0000 7fd703400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7040701a0 0x7fd704158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35030/0 (socket says 172.21.15.27:35030) 2024-01-15T18:05:39.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.031+0000 7fd703400640 1 -- 172.21.15.27:0/2102059227 learned_addr learned my addr 172.21.15.27:0/2102059227 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:39.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.031+0000 7fd703e00640 1 --2- 172.21.15.27:0/2102059227 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd70415c130 0x7fd7041516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.031+0000 7fd702a00640 1 --2- 172.21.15.27:0/2102059227 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd7040c92b0 0x7fd704165250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.031+0000 7fd703400640 1 -- 172.21.15.27:0/2102059227 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd70415c130 msgr2=0x7fd7041516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.031+0000 7fd703400640 1 --2- 172.21.15.27:0/2102059227 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd70415c130 0x7fd7041516d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.031+0000 7fd703400640 1 -- 172.21.15.27:0/2102059227 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd7040c92b0 msgr2=0x7fd704165250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.031+0000 7fd703400640 1 --2- 172.21.15.27:0/2102059227 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd7040c92b0 0x7fd704165250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.031+0000 7fd703400640 1 -- 172.21.15.27:0/2102059227 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f40094d0 con 0x7fd7040701a0 2024-01-15T18:05:39.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.031+0000 7fd703400640 1 --2- 172.21.15.27:0/2102059227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7040701a0 0x7fd704158110 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd6f40ac040 tx=0x7fd6f401ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:39.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.031+0000 7fd6fb400640 1 -- 172.21.15.27:0/2102059227 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f40b3800 con 0x7fd7040701a0 2024-01-15T18:05:39.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.031+0000 7fd6fb400640 1 -- 172.21.15.27:0/2102059227 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6f40b3980 con 0x7fd7040701a0 2024-01-15T18:05:39.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.031+0000 7fd6fb400640 1 -- 172.21.15.27:0/2102059227 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f40b3d30 con 0x7fd7040701a0 2024-01-15T18:05:39.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.031+0000 7fd70a800640 1 -- 172.21.15.27:0/2102059227 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd704158960 con 0x7fd7040701a0 2024-01-15T18:05:39.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.035+0000 7fd70a800640 1 -- 172.21.15.27:0/2102059227 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7040c6260 con 0x7fd7040701a0 2024-01-15T18:05:39.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.035+0000 7fd70a800640 1 -- 172.21.15.27:0/2102059227 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd70404e850 con 0x7fd7040701a0 2024-01-15T18:05:39.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.035+0000 7fd6fb400640 1 -- 172.21.15.27:0/2102059227 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fd6f40b2380 con 0x7fd7040701a0 2024-01-15T18:05:39.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.035+0000 7fd6fb400640 1 --2- 172.21.15.27:0/2102059227 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd6dc0668f0 0x7fd6dc068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.035+0000 7fd6fb400640 1 -- 172.21.15.27:0/2102059227 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fd6f411e9c0 con 0x7fd7040701a0 2024-01-15T18:05:39.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.035+0000 7fd702a00640 1 --2- 172.21.15.27:0/2102059227 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd6dc0668f0 0x7fd6dc068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.035+0000 7fd702a00640 1 --2- 172.21.15.27:0/2102059227 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd6dc0668f0 0x7fd6dc068db0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd7040c8380 tx=0x7fd6ec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:39.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.039+0000 7fd6fb400640 1 -- 172.21.15.27:0/2102059227 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd6f40ebc10 con 0x7fd7040701a0 2024-01-15T18:05:39.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.207+0000 7fd70a800640 1 -- 172.21.15.27:0/2102059227 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7040cf8d0 con 0x7fd7040701a0 2024-01-15T18:05:39.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.207+0000 7fd6fb400640 1 -- 172.21.15.27:0/2102059227 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16705 (secure 0 0 0) 0x7fd6f40c3090 con 0x7fd7040701a0 2024-01-15T18:05:39.213 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:39.213 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":22,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:05:38.998571+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:04:56.220608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T18:04:57.210421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:39.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.211+0000 7fd70a800640 1 -- 172.21.15.27:0/2102059227 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd6dc0668f0 msgr2=0x7fd6dc068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.211+0000 7fd70a800640 1 --2- 172.21.15.27:0/2102059227 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd6dc0668f0 0x7fd6dc068db0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd7040c8380 tx=0x7fd6ec009040 comp rx=0 tx=0).stop 2024-01-15T18:05:39.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.211+0000 7fd70a800640 1 -- 172.21.15.27:0/2102059227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7040701a0 msgr2=0x7fd704158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.211+0000 7fd70a800640 1 --2- 172.21.15.27:0/2102059227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7040701a0 0x7fd704158110 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd6f40ac040 tx=0x7fd6f401ad50 comp rx=0 tx=0).stop 2024-01-15T18:05:39.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.211+0000 7fd70a800640 1 -- 172.21.15.27:0/2102059227 shutdown_connections 2024-01-15T18:05:39.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.211+0000 7fd70a800640 1 --2- 172.21.15.27:0/2102059227 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd6dc0668f0 0x7fd6dc068db0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.211+0000 7fd70a800640 1 --2- 172.21.15.27:0/2102059227 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd70415c130 0x7fd7041516d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.211+0000 7fd70a800640 1 --2- 172.21.15.27:0/2102059227 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd7040c92b0 0x7fd704165250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.211+0000 7fd70a800640 1 --2- 172.21.15.27:0/2102059227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7040701a0 0x7fd704158110 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.211+0000 7fd70a800640 1 -- 172.21.15.27:0/2102059227 >> 172.21.15.27:0/2102059227 conn(0x7fd7040cec60 msgr2=0x7fd70415b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:39.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.211+0000 7fd70a800640 1 -- 172.21.15.27:0/2102059227 shutdown_connections 2024-01-15T18:05:39.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.211+0000 7fd70a800640 1 -- 172.21.15.27:0/2102059227 wait complete. 2024-01-15T18:05:39.236 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T18:04:56.220608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T18:04:57.210421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:05:39.236 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:05:39.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.375+0000 7f602a400640 1 Processor -- start 2024-01-15T18:05:39.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.375+0000 7f602a400640 1 -- start start 2024-01-15T18:05:39.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.375+0000 7f602a400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f60240d1a40 0x7f60240cba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.375+0000 7f602a400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60240cbfa0 con 0x7f60240b0650 2024-01-15T18:05:39.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.375+0000 7f602a400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60240cefa0 con 0x7f60240cf450 2024-01-15T18:05:39.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.375+0000 7f602a400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60240ce240 con 0x7f60240d1a40 2024-01-15T18:05:39.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.375+0000 7f6023e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f60240b0650 0x7f60240b0ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35112/0 (socket says 172.21.15.27:35112) 2024-01-15T18:05:39.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.375+0000 7f6023e00640 1 -- 172.21.15.27:0/3392253587 learned_addr learned my addr 172.21.15.27:0/3392253587 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:39.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.375+0000 7f6022a00640 1 --2- 172.21.15.27:0/3392253587 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f60240d1a40 0x7f60240cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.375+0000 7f6022000640 1 -- 172.21.15.27:0/3392253587 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1560088886 0 0) 0x7f60240cbfa0 con 0x7f60240b0650 2024-01-15T18:05:39.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.375+0000 7f6022000640 1 -- 172.21.15.27:0/3392253587 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6004003530 con 0x7f60240b0650 2024-01-15T18:05:39.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.375+0000 7f6022000640 1 -- 172.21.15.27:0/3392253587 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3287356178 0 0) 0x7f6004003530 con 0x7f60240b0650 2024-01-15T18:05:39.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.375+0000 7f6022000640 1 -- 172.21.15.27:0/3392253587 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f60240d1a40 msgr2=0x7f60240cba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:39.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.375+0000 7f6022000640 1 --2- 172.21.15.27:0/3392253587 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f60240d1a40 0x7f60240cba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.375+0000 7f6022000640 1 -- 172.21.15.27:0/3392253587 >> v1:172.21.15.93:6789/0 conn(0x7f60240cf450 legacy=0x7f60240cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.375+0000 7f6022000640 1 -- 172.21.15.27:0/3392253587 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60240ccd70 con 0x7f60240b0650 2024-01-15T18:05:39.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.375+0000 7f6022000640 1 -- 172.21.15.27:0/3392253587 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f601c001a00 con 0x7f60240b0650 2024-01-15T18:05:39.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.375+0000 7f6022000640 1 -- 172.21.15.27:0/3392253587 >> v1:172.21.15.27:6789/0 conn(0x7f60240b0650 legacy=0x7f60240b0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.375+0000 7f6022000640 1 --2- 172.21.15.27:0/3392253587 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f60240d1a40 0x7f6004003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.375+0000 7f6022000640 1 --2- 172.21.15.27:0/3392253587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6004003f60 0x7f60040047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.375+0000 7f6022000640 1 --2- 172.21.15.27:0/3392253587 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6004004d10 0x7f6004005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.375+0000 7f6022000640 1 -- 172.21.15.27:0/3392253587 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60240ce240 con 0x7f6004003f60 2024-01-15T18:05:39.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.375+0000 7f6022000640 1 -- 172.21.15.27:0/3392253587 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600400bad0 con 0x7f6004004d10 2024-01-15T18:05:39.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.375+0000 7f6022000640 1 -- 172.21.15.27:0/3392253587 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f600400bc90 con 0x7f60240d1a40 2024-01-15T18:05:39.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.379+0000 7f6022a00640 1 --2- 172.21.15.27:0/3392253587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6004003f60 0x7f60040047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.379+0000 7f6023400640 1 --2- 172.21.15.27:0/3392253587 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f60240d1a40 0x7f6004003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.379+0000 7f6023e00640 1 --2- 172.21.15.27:0/3392253587 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6004004d10 0x7f6004005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.379+0000 7f6022a00640 1 -- 172.21.15.27:0/3392253587 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f60240d1a40 msgr2=0x7f6004003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:39.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.379+0000 7f6022a00640 1 --2- 172.21.15.27:0/3392253587 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f60240d1a40 0x7f6004003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.379+0000 7f6022a00640 1 -- 172.21.15.27:0/3392253587 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6004004d10 msgr2=0x7f6004005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:39.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.379+0000 7f6022a00640 1 --2- 172.21.15.27:0/3392253587 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6004004d10 0x7f6004005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.379+0000 7f6022a00640 1 -- 172.21.15.27:0/3392253587 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6004003530 con 0x7f6004003f60 2024-01-15T18:05:39.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.379+0000 7f6023e00640 1 --2- 172.21.15.27:0/3392253587 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6004004d10 0x7f6004005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:39.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.379+0000 7f6022a00640 1 --2- 172.21.15.27:0/3392253587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6004003f60 0x7f60040047d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f600c009ae0 tx=0x7f600c01c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a86b9f9fc1cfcb7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:39.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.379+0000 7f6022000640 1 -- 172.21.15.27:0/3392253587 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f600c0a5580 con 0x7f6004003f60 2024-01-15T18:05:39.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.379+0000 7f6022000640 1 -- 172.21.15.27:0/3392253587 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f600c0a56e0 con 0x7f6004003f60 2024-01-15T18:05:39.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.379+0000 7f6022000640 1 -- 172.21.15.27:0/3392253587 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f600c0a5a90 con 0x7f6004003f60 2024-01-15T18:05:39.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.383+0000 7f602a400640 1 -- 172.21.15.27:0/3392253587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6004003f60 msgr2=0x7f60040047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.383+0000 7f602a400640 1 --2- 172.21.15.27:0/3392253587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6004003f60 0x7f60040047d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f600c009ae0 tx=0x7f600c01c990 comp rx=0 tx=0).stop 2024-01-15T18:05:39.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.383+0000 7f6023e00640 1 -- 172.21.15.27:0/3392253587 reap_dead start 2024-01-15T18:05:39.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.383+0000 7f602a400640 1 -- 172.21.15.27:0/3392253587 shutdown_connections 2024-01-15T18:05:39.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.383+0000 7f602a400640 1 -- 172.21.15.27:0/3392253587 >> 172.21.15.27:0/3392253587 conn(0x7f60240d0e70 msgr2=0x7f60240d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:39.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.383+0000 7f602a400640 1 -- 172.21.15.27:0/3392253587 shutdown_connections 2024-01-15T18:05:39.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.383+0000 7f602a400640 1 -- 172.21.15.27:0/3392253587 wait complete. 2024-01-15T18:05:39.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.383+0000 7f602a400640 1 Processor -- start 2024-01-15T18:05:39.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.383+0000 7f602a400640 1 -- start start 2024-01-15T18:05:39.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.383+0000 7f602a400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6004003f60 0x7f60240b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.383+0000 7f602a400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f60240b4200 0x7f602414a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.383+0000 7f602a400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f602414a980 0x7f60240ca270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.383+0000 7f602a400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60240af230 con 0x7f6004003f60 2024-01-15T18:05:39.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.383+0000 7f602a400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60240ae620 con 0x7f60240b4200 2024-01-15T18:05:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.383+0000 7f602a400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60240ae900 con 0x7f602414a980 2024-01-15T18:05:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.383+0000 7f6023400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6004003f60 0x7f60240b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.383+0000 7f6023400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6004003f60 0x7f60240b3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35046/0 (socket says 172.21.15.27:35046) 2024-01-15T18:05:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.383+0000 7f6023400640 1 -- 172.21.15.27:0/3328537154 learned_addr learned my addr 172.21.15.27:0/3328537154 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.383+0000 7f6022a00640 1 --2- 172.21.15.27:0/3328537154 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f60240b4200 0x7f602414a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.383+0000 7f6023e00640 1 --2- 172.21.15.27:0/3328537154 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f602414a980 0x7f60240ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.387+0000 7f6023400640 1 -- 172.21.15.27:0/3328537154 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f602414a980 msgr2=0x7f60240ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.387+0000 7f6023400640 1 --2- 172.21.15.27:0/3328537154 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f602414a980 0x7f60240ca270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.387+0000 7f6023400640 1 -- 172.21.15.27:0/3328537154 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f60240b4200 msgr2=0x7f602414a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.387+0000 7f6023400640 1 --2- 172.21.15.27:0/3328537154 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f60240b4200 0x7f602414a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.387+0000 7f6023400640 1 -- 172.21.15.27:0/3328537154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60240aebe0 con 0x7f6004003f60 2024-01-15T18:05:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.387+0000 7f6023400640 1 --2- 172.21.15.27:0/3328537154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6004003f60 0x7f60240b3cc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6014011bb0 tx=0x7f601400af20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.387+0000 7f601b400640 1 -- 172.21.15.27:0/3328537154 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601400a860 con 0x7f6004003f60 2024-01-15T18:05:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.387+0000 7f601b400640 1 -- 172.21.15.27:0/3328537154 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60140052e0 con 0x7f6004003f60 2024-01-15T18:05:39.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.387+0000 7f602a400640 1 -- 172.21.15.27:0/3328537154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60240aeec0 con 0x7f6004003f60 2024-01-15T18:05:39.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.387+0000 7f601b400640 1 -- 172.21.15.27:0/3328537154 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601400fad0 con 0x7f6004003f60 2024-01-15T18:05:39.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.387+0000 7f602a400640 1 -- 172.21.15.27:0/3328537154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60240ca7b0 con 0x7f6004003f60 2024-01-15T18:05:39.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.387+0000 7f602a400640 1 -- 172.21.15.27:0/3328537154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f602404e850 con 0x7f6004003f60 2024-01-15T18:05:39.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.387+0000 7f601b400640 1 -- 172.21.15.27:0/3328537154 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f601400fc30 con 0x7f6004003f60 2024-01-15T18:05:39.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.391+0000 7f601b400640 1 --2- 172.21.15.27:0/3328537154 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5ffc066920 0x7f5ffc068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.391+0000 7f601b400640 1 -- 172.21.15.27:0/3328537154 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f6014087b40 con 0x7f6004003f60 2024-01-15T18:05:39.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.391+0000 7f6022a00640 1 --2- 172.21.15.27:0/3328537154 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5ffc066920 0x7f5ffc068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.391+0000 7f6022a00640 1 --2- 172.21.15.27:0/3328537154 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5ffc066920 0x7f5ffc068de0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f600c0af000 tx=0x7f600c0046d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:39.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.395+0000 7f601b400640 1 -- 172.21.15.27:0/3328537154 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6014054ce0 con 0x7f6004003f60 2024-01-15T18:05:39.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.551+0000 7f602a400640 1 -- 172.21.15.27:0/3328537154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f60240cf230 con 0x7f6004003f60 2024-01-15T18:05:39.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.551+0000 7f601b400640 1 -- 172.21.15.27:0/3328537154 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6014019020 con 0x7f6004003f60 2024-01-15T18:05:39.558 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:05:39.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.555+0000 7f602a400640 1 -- 172.21.15.27:0/3328537154 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5ffc066920 msgr2=0x7f5ffc068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.555+0000 7f602a400640 1 --2- 172.21.15.27:0/3328537154 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5ffc066920 0x7f5ffc068de0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f600c0af000 tx=0x7f600c0046d0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.555+0000 7f602a400640 1 -- 172.21.15.27:0/3328537154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6004003f60 msgr2=0x7f60240b3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.555+0000 7f602a400640 1 --2- 172.21.15.27:0/3328537154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6004003f60 0x7f60240b3cc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6014011bb0 tx=0x7f601400af20 comp rx=0 tx=0).stop 2024-01-15T18:05:39.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.555+0000 7f602a400640 1 -- 172.21.15.27:0/3328537154 shutdown_connections 2024-01-15T18:05:39.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.555+0000 7f602a400640 1 --2- 172.21.15.27:0/3328537154 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5ffc066920 0x7f5ffc068de0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.555+0000 7f602a400640 1 --2- 172.21.15.27:0/3328537154 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f602414a980 0x7f60240ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.555+0000 7f602a400640 1 --2- 172.21.15.27:0/3328537154 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f60240b4200 0x7f602414a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.555+0000 7f602a400640 1 --2- 172.21.15.27:0/3328537154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6004003f60 0x7f60240b3cc0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.555+0000 7f602a400640 1 -- 172.21.15.27:0/3328537154 >> 172.21.15.27:0/3328537154 conn(0x7f60240d0e70 msgr2=0x7f602406b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:39.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.555+0000 7f602a400640 1 -- 172.21.15.27:0/3328537154 shutdown_connections 2024-01-15T18:05:39.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.555+0000 7f602a400640 1 -- 172.21.15.27:0/3328537154 wait complete. 2024-01-15T18:05:39.580 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T18:05:39.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.719+0000 7f6490400640 1 Processor -- start 2024-01-15T18:05:39.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.719+0000 7f6490400640 1 -- start start 2024-01-15T18:05:39.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.719+0000 7f6490400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f64880cdb20 0x7f648806cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.719+0000 7f6490400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f648806d2d0 0x7f6488161a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.719+0000 7f6490400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f648806c5d0 con 0x7f6488161fd0 2024-01-15T18:05:39.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.719+0000 7f6490400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6488162b00 con 0x7f648806d2d0 2024-01-15T18:05:39.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.719+0000 7f6490400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6488162540 con 0x7f64880cdb20 2024-01-15T18:05:39.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.719+0000 7f648d000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f648806d2d0 0x7f6488161a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.719+0000 7f648d000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f648806d2d0 0x7f6488161a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35054/0 (socket says 172.21.15.27:35054) 2024-01-15T18:05:39.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.719+0000 7f648d000640 1 -- 172.21.15.27:0/767032685 learned_addr learned my addr 172.21.15.27:0/767032685 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:39.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.719+0000 7f648da00640 1 --2- 172.21.15.27:0/767032685 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f64880cdb20 0x7f648806cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.719+0000 7f648d000640 1 -- 172.21.15.27:0/767032685 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f64880cdb20 msgr2=0x7f648806cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:39.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.719+0000 7f648d000640 1 --2- 172.21.15.27:0/767032685 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f64880cdb20 0x7f648806cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.719+0000 7f648d000640 1 -- 172.21.15.27:0/767032685 >> v1:172.21.15.93:6789/0 conn(0x7f6488161fd0 legacy=0x7f64880ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:39.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.719+0000 7f648d000640 1 -- 172.21.15.27:0/767032685 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6488162820 con 0x7f648806d2d0 2024-01-15T18:05:39.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.723+0000 7f648d000640 1 --2- 172.21.15.27:0/767032685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f648806d2d0 0x7f6488161a90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6470009870 tx=0x7f647001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b32a04b44a907ea5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:39.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.723+0000 7f6487e00640 1 -- 172.21.15.27:0/767032685 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64700aa610 con 0x7f648806d2d0 2024-01-15T18:05:39.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.723+0000 7f6487e00640 1 -- 172.21.15.27:0/767032685 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64700a6040 con 0x7f648806d2d0 2024-01-15T18:05:39.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.723+0000 7f6487e00640 1 -- 172.21.15.27:0/767032685 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64700aa990 con 0x7f648806d2d0 2024-01-15T18:05:39.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.723+0000 7f6490400640 1 -- 172.21.15.27:0/767032685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f648806d2d0 msgr2=0x7f6488161a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.723+0000 7f6490400640 1 --2- 172.21.15.27:0/767032685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f648806d2d0 0x7f6488161a90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6470009870 tx=0x7f647001c4d0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.723+0000 7f6490400640 1 -- 172.21.15.27:0/767032685 shutdown_connections 2024-01-15T18:05:39.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.723+0000 7f6490400640 1 --2- 172.21.15.27:0/767032685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f648806d2d0 0x7f6488161a90 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.723+0000 7f6490400640 1 --2- 172.21.15.27:0/767032685 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f64880cdb20 0x7f648806cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.723+0000 7f6490400640 1 -- 172.21.15.27:0/767032685 >> 172.21.15.27:0/767032685 conn(0x7f64880cb700 msgr2=0x7f64880cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:39.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.723+0000 7f6490400640 1 -- 172.21.15.27:0/767032685 shutdown_connections 2024-01-15T18:05:39.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.723+0000 7f6490400640 1 -- 172.21.15.27:0/767032685 wait complete. 2024-01-15T18:05:39.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.727+0000 7f6490400640 1 Processor -- start 2024-01-15T18:05:39.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.727+0000 7f6490400640 1 -- start start 2024-01-15T18:05:39.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.727+0000 7f6490400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f648806d2d0 0x7f6488163850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.727+0000 7f6490400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f64880cdb20 0x7f6488137850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.727+0000 7f6490400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6488161fd0 0x7f64881644f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.727+0000 7f6490400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6488163d90 con 0x7f648806d2d0 2024-01-15T18:05:39.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.727+0000 7f6490400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64880b48c0 con 0x7f6488161fd0 2024-01-15T18:05:39.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.727+0000 7f6490400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6488158ae0 con 0x7f64880cdb20 2024-01-15T18:05:39.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.727+0000 7f648da00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f648806d2d0 0x7f6488163850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.727+0000 7f648da00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f648806d2d0 0x7f6488163850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35064/0 (socket says 172.21.15.27:35064) 2024-01-15T18:05:39.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.727+0000 7f648da00640 1 -- 172.21.15.27:0/607949470 learned_addr learned my addr 172.21.15.27:0/607949470 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:39.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.727+0000 7f648d000640 1 --2- 172.21.15.27:0/607949470 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f64880cdb20 0x7f6488137850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.727+0000 7f648e400640 1 --2- 172.21.15.27:0/607949470 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6488161fd0 0x7f64881644f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.727+0000 7f648da00640 1 -- 172.21.15.27:0/607949470 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f64880cdb20 msgr2=0x7f6488137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.727+0000 7f648da00640 1 --2- 172.21.15.27:0/607949470 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f64880cdb20 0x7f6488137850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.727+0000 7f648da00640 1 -- 172.21.15.27:0/607949470 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6488161fd0 msgr2=0x7f64881644f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.727+0000 7f648da00640 1 --2- 172.21.15.27:0/607949470 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6488161fd0 0x7f64881644f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.727+0000 7f648da00640 1 -- 172.21.15.27:0/607949470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64700094d0 con 0x7f648806d2d0 2024-01-15T18:05:39.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.727+0000 7f648da00640 1 --2- 172.21.15.27:0/607949470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f648806d2d0 0x7f6488163850 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f647800d800 tx=0x7f647800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:39.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.727+0000 7f6485600640 1 -- 172.21.15.27:0/607949470 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f647800b6c0 con 0x7f648806d2d0 2024-01-15T18:05:39.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.727+0000 7f6485600640 1 -- 172.21.15.27:0/607949470 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f647800b820 con 0x7f648806d2d0 2024-01-15T18:05:39.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.727+0000 7f6490400640 1 -- 172.21.15.27:0/607949470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f648815b210 con 0x7f648806d2d0 2024-01-15T18:05:39.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.727+0000 7f6485600640 1 -- 172.21.15.27:0/607949470 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f647800bb90 con 0x7f648806d2d0 2024-01-15T18:05:39.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.727+0000 7f6490400640 1 -- 172.21.15.27:0/607949470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f648815b7d0 con 0x7f648806d2d0 2024-01-15T18:05:39.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.731+0000 7f6490400640 1 -- 172.21.15.27:0/607949470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f648804e850 con 0x7f648806d2d0 2024-01-15T18:05:39.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.731+0000 7f6485600640 1 -- 172.21.15.27:0/607949470 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f647800bcf0 con 0x7f648806d2d0 2024-01-15T18:05:39.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.731+0000 7f6485600640 1 --2- 172.21.15.27:0/607949470 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f64600668f0 0x7f6460068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.731+0000 7f6485600640 1 -- 172.21.15.27:0/607949470 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f64780848a0 con 0x7f648806d2d0 2024-01-15T18:05:39.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.731+0000 7f648d000640 1 --2- 172.21.15.27:0/607949470 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f64600668f0 0x7f6460068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.731+0000 7f648d000640 1 --2- 172.21.15.27:0/607949470 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f64600668f0 0x7f6460068db0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f64700ae000 tx=0x7f64700b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:39.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.735+0000 7f6485600640 1 -- 172.21.15.27:0/607949470 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6478051b00 con 0x7f648806d2d0 2024-01-15T18:05:39.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.891+0000 7f6490400640 1 -- 172.21.15.27:0/607949470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f64881615b0 con 0x7f648806d2d0 2024-01-15T18:05:39.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.891+0000 7f6485600640 1 -- 172.21.15.27:0/607949470 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6478055120 con 0x7f648806d2d0 2024-01-15T18:05:39.899 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-15T18:05:39.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.895+0000 7f6490400640 1 -- 172.21.15.27:0/607949470 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f64600668f0 msgr2=0x7f6460068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.895+0000 7f6490400640 1 --2- 172.21.15.27:0/607949470 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f64600668f0 0x7f6460068db0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f64700ae000 tx=0x7f64700b5040 comp rx=0 tx=0).stop 2024-01-15T18:05:39.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.895+0000 7f6490400640 1 -- 172.21.15.27:0/607949470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f648806d2d0 msgr2=0x7f6488163850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.895+0000 7f6490400640 1 --2- 172.21.15.27:0/607949470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f648806d2d0 0x7f6488163850 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f647800d800 tx=0x7f647800dcc0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.895+0000 7f6490400640 1 -- 172.21.15.27:0/607949470 shutdown_connections 2024-01-15T18:05:39.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.895+0000 7f6490400640 1 --2- 172.21.15.27:0/607949470 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f64600668f0 0x7f6460068db0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.895+0000 7f6490400640 1 --2- 172.21.15.27:0/607949470 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6488161fd0 0x7f64881644f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.895+0000 7f6490400640 1 --2- 172.21.15.27:0/607949470 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f64880cdb20 0x7f6488137850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.895+0000 7f6490400640 1 --2- 172.21.15.27:0/607949470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f648806d2d0 0x7f6488163850 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.895+0000 7f6490400640 1 -- 172.21.15.27:0/607949470 >> 172.21.15.27:0/607949470 conn(0x7f64880cb700 msgr2=0x7f64880b0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:39.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.895+0000 7f6490400640 1 -- 172.21.15.27:0/607949470 shutdown_connections 2024-01-15T18:05:39.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:39.895+0000 7f6490400640 1 -- 172.21.15.27:0/607949470 wait complete. 2024-01-15T18:05:39.920 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T18:05:40.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.067+0000 7f42c0200640 1 Processor -- start 2024-01-15T18:05:40.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.067+0000 7f42c0200640 1 -- start start 2024-01-15T18:05:40.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.067+0000 7f42c0200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f42b815d0d0 0x7f42b80b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.067+0000 7f42c0200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b80d0400 con 0x7f42b80cfb20 2024-01-15T18:05:40.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.067+0000 7f42c0200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b8071940 con 0x7f42b80cff40 2024-01-15T18:05:40.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.067+0000 7f42c0200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42b80721c0 con 0x7f42b815d0d0 2024-01-15T18:05:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.067+0000 7f42bce00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f42b80cfb20 0x7f42b80b4880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35114/0 (socket says 172.21.15.27:35114) 2024-01-15T18:05:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.067+0000 7f42bce00640 1 -- 172.21.15.27:0/973631365 learned_addr learned my addr 172.21.15.27:0/973631365 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.067+0000 7f42bd800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f42b815d0d0 0x7f42b80b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.067+0000 7f42bd800640 1 -- 172.21.15.27:0/973631365 >> v1:172.21.15.93:6789/0 conn(0x7f42b80cff40 legacy=0x7f42b8161410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.067+0000 7f42bd800640 1 -- 172.21.15.27:0/973631365 >> v1:172.21.15.27:6789/0 conn(0x7f42b80cfb20 legacy=0x7f42b80b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.067+0000 7f42bd800640 1 -- 172.21.15.27:0/973631365 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b8161b20 con 0x7f42b815d0d0 2024-01-15T18:05:40.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.067+0000 7f42b7e00640 1 -- 172.21.15.27:0/973631365 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262947156 0 0) 0x7f42b80d0400 con 0x7f42b80cfb20 2024-01-15T18:05:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.071+0000 7f42bd800640 1 --2- 172.21.15.27:0/973631365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42b815d0d0 0x7f42b80b2fa0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f42b8161d10 tx=0x7f42a801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e039b794d9a3c68f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.071+0000 7f42b7e00640 1 -- 172.21.15.27:0/973631365 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a80ab600 con 0x7f42b815d0d0 2024-01-15T18:05:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.071+0000 7f42b7e00640 1 -- 172.21.15.27:0/973631365 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a80a7050 con 0x7f42b815d0d0 2024-01-15T18:05:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.071+0000 7f42b7e00640 1 -- 172.21.15.27:0/973631365 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a80ab980 con 0x7f42b815d0d0 2024-01-15T18:05:40.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.071+0000 7f42c0200640 1 -- 172.21.15.27:0/973631365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42b815d0d0 msgr2=0x7f42b80b2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.071+0000 7f42c0200640 1 --2- 172.21.15.27:0/973631365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42b815d0d0 0x7f42b80b2fa0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f42b8161d10 tx=0x7f42a801c3f0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.071+0000 7f42c0200640 1 -- 172.21.15.27:0/973631365 shutdown_connections 2024-01-15T18:05:40.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.071+0000 7f42c0200640 1 --2- 172.21.15.27:0/973631365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42b815d0d0 0x7f42b80b2fa0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.071+0000 7f42c0200640 1 -- 172.21.15.27:0/973631365 >> 172.21.15.27:0/973631365 conn(0x7f42b806b600 msgr2=0x7f42b806ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:40.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.071+0000 7f42c0200640 1 -- 172.21.15.27:0/973631365 shutdown_connections 2024-01-15T18:05:40.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.071+0000 7f42c0200640 1 -- 172.21.15.27:0/973631365 wait complete. 2024-01-15T18:05:40.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.075+0000 7f42c0200640 1 Processor -- start 2024-01-15T18:05:40.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.075+0000 7f42c0200640 1 -- start start 2024-01-15T18:05:40.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.075+0000 7f42c0200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42b80cfb20 0x7f42b81488d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.075+0000 7f42c0200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f42b80cff40 0x7f42b80b0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.075+0000 7f42c0200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42b815d0d0 0x7f42b8150da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.075+0000 7f42c0200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b814e820 con 0x7f42b80cfb20 2024-01-15T18:05:40.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.075+0000 7f42c0200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b814eb00 con 0x7f42b815d0d0 2024-01-15T18:05:40.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.075+0000 7f42c0200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b814ede0 con 0x7f42b80cff40 2024-01-15T18:05:40.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.075+0000 7f42bd800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42b80cfb20 0x7f42b81488d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.075+0000 7f42bd800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42b80cfb20 0x7f42b81488d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35094/0 (socket says 172.21.15.27:35094) 2024-01-15T18:05:40.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.075+0000 7f42bd800640 1 -- 172.21.15.27:0/2989911340 learned_addr learned my addr 172.21.15.27:0/2989911340 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:40.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.075+0000 7f42bce00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f42b80cff40 0x7f42b80b0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.075+0000 7f42be200640 1 --2- 172.21.15.27:0/2989911340 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42b815d0d0 0x7f42b8150da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.075+0000 7f42bd800640 1 -- 172.21.15.27:0/2989911340 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f42b80cff40 msgr2=0x7f42b80b0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.075+0000 7f42bd800640 1 --2- 172.21.15.27:0/2989911340 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f42b80cff40 0x7f42b80b0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.075+0000 7f42bd800640 1 -- 172.21.15.27:0/2989911340 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42b815d0d0 msgr2=0x7f42b8150da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.075+0000 7f42bd800640 1 --2- 172.21.15.27:0/2989911340 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42b815d0d0 0x7f42b8150da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.075+0000 7f42bd800640 1 -- 172.21.15.27:0/2989911340 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a8009420 con 0x7f42b80cfb20 2024-01-15T18:05:40.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.075+0000 7f42bd800640 1 --2- 172.21.15.27:0/2989911340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42b80cfb20 0x7f42b81488d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f42a80ae040 tx=0x7f42a801ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:40.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.075+0000 7f42b5600640 1 -- 172.21.15.27:0/2989911340 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a80b7650 con 0x7f42b80cfb20 2024-01-15T18:05:40.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.075+0000 7f42b5600640 1 -- 172.21.15.27:0/2989911340 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a80b77d0 con 0x7f42b80cfb20 2024-01-15T18:05:40.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.075+0000 7f42b5600640 1 -- 172.21.15.27:0/2989911340 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a80b7b80 con 0x7f42b80cfb20 2024-01-15T18:05:40.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.075+0000 7f42c0200640 1 -- 172.21.15.27:0/2989911340 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b814f0c0 con 0x7f42b80cfb20 2024-01-15T18:05:40.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.075+0000 7f42c0200640 1 -- 172.21.15.27:0/2989911340 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b8148e10 con 0x7f42b80cfb20 2024-01-15T18:05:40.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.079+0000 7f42c0200640 1 -- 172.21.15.27:0/2989911340 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b804e850 con 0x7f42b80cfb20 2024-01-15T18:05:40.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.079+0000 7f42b5600640 1 -- 172.21.15.27:0/2989911340 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f42a80b7ce0 con 0x7f42b80cfb20 2024-01-15T18:05:40.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.079+0000 7f42b5600640 1 --2- 172.21.15.27:0/2989911340 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f42900668f0 0x7f4290068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.079+0000 7f42b5600640 1 -- 172.21.15.27:0/2989911340 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f42a811ede0 con 0x7f42b80cfb20 2024-01-15T18:05:40.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.079+0000 7f42bce00640 1 --2- 172.21.15.27:0/2989911340 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f42900668f0 0x7f4290068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.079+0000 7f42bce00640 1 --2- 172.21.15.27:0/2989911340 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f42900668f0 0x7f4290068db0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f42b814fbd0 tx=0x7f42a0007240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:40.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.083+0000 7f42b5600640 1 -- 172.21.15.27:0/2989911340 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f42a80ebfb0 con 0x7f42b80cfb20 2024-01-15T18:05:40.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.243+0000 7f42c0200640 1 -- 172.21.15.27:0/2989911340 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f42b8149120 con 0x7f42b80cfb20 2024-01-15T18:05:40.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.243+0000 7f42b5600640 1 -- 172.21.15.27:0/2989911340 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f42a80ef5d0 con 0x7f42b80cfb20 2024-01-15T18:05:40.247 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-15T18:05:40.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.243+0000 7f42c0200640 1 -- 172.21.15.27:0/2989911340 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f42900668f0 msgr2=0x7f4290068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.243+0000 7f42c0200640 1 --2- 172.21.15.27:0/2989911340 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f42900668f0 0x7f4290068db0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f42b814fbd0 tx=0x7f42a0007240 comp rx=0 tx=0).stop 2024-01-15T18:05:40.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.243+0000 7f42c0200640 1 -- 172.21.15.27:0/2989911340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42b80cfb20 msgr2=0x7f42b81488d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.243+0000 7f42c0200640 1 --2- 172.21.15.27:0/2989911340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42b80cfb20 0x7f42b81488d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f42a80ae040 tx=0x7f42a801ac30 comp rx=0 tx=0).stop 2024-01-15T18:05:40.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.243+0000 7f42c0200640 1 -- 172.21.15.27:0/2989911340 shutdown_connections 2024-01-15T18:05:40.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.243+0000 7f42c0200640 1 --2- 172.21.15.27:0/2989911340 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f42900668f0 0x7f4290068db0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.243+0000 7f42c0200640 1 --2- 172.21.15.27:0/2989911340 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42b815d0d0 0x7f42b8150da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.243+0000 7f42c0200640 1 --2- 172.21.15.27:0/2989911340 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f42b80cff40 0x7f42b80b0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.243+0000 7f42c0200640 1 --2- 172.21.15.27:0/2989911340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42b80cfb20 0x7f42b81488d0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.243+0000 7f42c0200640 1 -- 172.21.15.27:0/2989911340 >> 172.21.15.27:0/2989911340 conn(0x7f42b806b600 msgr2=0x7f42b80b2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:40.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.243+0000 7f42c0200640 1 -- 172.21.15.27:0/2989911340 shutdown_connections 2024-01-15T18:05:40.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.243+0000 7f42c0200640 1 -- 172.21.15.27:0/2989911340 wait complete. 2024-01-15T18:05:40.270 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:05:40.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.407+0000 7fd58f000640 1 Processor -- start 2024-01-15T18:05:40.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.407+0000 7fd58f000640 1 -- start start 2024-01-15T18:05:40.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.407+0000 7fd58f000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd58809a120 0x7fd5880b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.407+0000 7fd58f000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd5880b1d90 0x7fd5880cf180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.407+0000 7fd58f000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5880b1aa0 con 0x7fd58815d7c0 2024-01-15T18:05:40.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.407+0000 7fd58f000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5880b1300 con 0x7fd58809a120 2024-01-15T18:05:40.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.407+0000 7fd58f000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5880b25a0 con 0x7fd5880b1d90 2024-01-15T18:05:40.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.407+0000 7fd587400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd58809a120 0x7fd5880b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.407+0000 7fd587400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd58809a120 0x7fd5880b35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35104/0 (socket says 172.21.15.27:35104) 2024-01-15T18:05:40.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.407+0000 7fd587400640 1 -- 172.21.15.27:0/3698130949 learned_addr learned my addr 172.21.15.27:0/3698130949 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:40.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.407+0000 7fd58d000640 1 --2- 172.21.15.27:0/3698130949 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd5880b1d90 0x7fd5880cf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.407+0000 7fd587e00640 1 --1- 172.21.15.27:0/3698130949 >> v1:172.21.15.93:6789/0 conn(0x7fd58815d7c0 0x7fd588099a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:59394/0 (socket says 172.21.15.27:59394) 2024-01-15T18:05:40.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.407+0000 7fd587400640 1 -- 172.21.15.27:0/3698130949 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd5880b1d90 msgr2=0x7fd5880cf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:40.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.407+0000 7fd587400640 1 --2- 172.21.15.27:0/3698130949 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd5880b1d90 0x7fd5880cf180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.407+0000 7fd587400640 1 -- 172.21.15.27:0/3698130949 >> v1:172.21.15.93:6789/0 conn(0x7fd58815d7c0 legacy=0x7fd588099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.407+0000 7fd587400640 1 -- 172.21.15.27:0/3698130949 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5880b14e0 con 0x7fd58809a120 2024-01-15T18:05:40.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.411+0000 7fd587400640 1 --2- 172.21.15.27:0/3698130949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd58809a120 0x7fd5880b35a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd570009870 tx=0x7fd57001c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2fa130db40d35c5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:40.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.411+0000 7fd586a00640 1 -- 172.21.15.27:0/3698130949 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5700ab600 con 0x7fd58809a120 2024-01-15T18:05:40.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.411+0000 7fd586a00640 1 -- 172.21.15.27:0/3698130949 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5700a7050 con 0x7fd58809a120 2024-01-15T18:05:40.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.411+0000 7fd586a00640 1 -- 172.21.15.27:0/3698130949 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5700ab980 con 0x7fd58809a120 2024-01-15T18:05:40.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.411+0000 7fd58f000640 1 -- 172.21.15.27:0/3698130949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd58809a120 msgr2=0x7fd5880b35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.411+0000 7fd58f000640 1 --2- 172.21.15.27:0/3698130949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd58809a120 0x7fd5880b35a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd570009870 tx=0x7fd57001c4b0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.415+0000 7fd58f000640 1 -- 172.21.15.27:0/3698130949 shutdown_connections 2024-01-15T18:05:40.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.415+0000 7fd58f000640 1 --2- 172.21.15.27:0/3698130949 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd5880b1d90 0x7fd5880cf180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.415+0000 7fd58f000640 1 --2- 172.21.15.27:0/3698130949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd58809a120 0x7fd5880b35a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.415+0000 7fd58f000640 1 -- 172.21.15.27:0/3698130949 >> 172.21.15.27:0/3698130949 conn(0x7fd5880b2a80 msgr2=0x7fd5880b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:40.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.415+0000 7fd58f000640 1 -- 172.21.15.27:0/3698130949 shutdown_connections 2024-01-15T18:05:40.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.415+0000 7fd58f000640 1 -- 172.21.15.27:0/3698130949 wait complete. 2024-01-15T18:05:40.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.415+0000 7fd58f000640 1 Processor -- start 2024-01-15T18:05:40.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.415+0000 7fd58f000640 1 -- start start 2024-01-15T18:05:40.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.415+0000 7fd58f000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd58809a120 0x7fd58815adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.415+0000 7fd58f000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5880b1d90 0x7fd588152bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.415+0000 7fd58f000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd58815d7c0 0x7fd58815f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.415+0000 7fd58f000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58815b300 con 0x7fd58809a120 2024-01-15T18:05:40.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.415+0000 7fd58f000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5880c56b0 con 0x7fd5880b1d90 2024-01-15T18:05:40.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.415+0000 7fd58f000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd58815bd90 con 0x7fd58815d7c0 2024-01-15T18:05:40.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.415+0000 7fd587e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd58809a120 0x7fd58815adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.415+0000 7fd587e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd58809a120 0x7fd58815adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35116/0 (socket says 172.21.15.27:35116) 2024-01-15T18:05:40.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.415+0000 7fd587e00640 1 -- 172.21.15.27:0/2038674394 learned_addr learned my addr 172.21.15.27:0/2038674394 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:40.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.415+0000 7fd587400640 1 --2- 172.21.15.27:0/2038674394 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5880b1d90 0x7fd588152bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.415+0000 7fd58d000640 1 --2- 172.21.15.27:0/2038674394 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd58815d7c0 0x7fd58815f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.415+0000 7fd587e00640 1 -- 172.21.15.27:0/2038674394 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd58815d7c0 msgr2=0x7fd58815f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.415+0000 7fd587e00640 1 --2- 172.21.15.27:0/2038674394 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd58815d7c0 0x7fd58815f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.415+0000 7fd587e00640 1 -- 172.21.15.27:0/2038674394 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5880b1d90 msgr2=0x7fd588152bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.415+0000 7fd587e00640 1 --2- 172.21.15.27:0/2038674394 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5880b1d90 0x7fd588152bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.415+0000 7fd587e00640 1 -- 172.21.15.27:0/2038674394 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5700094d0 con 0x7fd58809a120 2024-01-15T18:05:40.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.419+0000 7fd587e00640 1 --2- 172.21.15.27:0/2038674394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd58809a120 0x7fd58815adc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd57800c490 tx=0x7fd57800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:40.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.419+0000 7fd57fe00640 1 -- 172.21.15.27:0/2038674394 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd578007480 con 0x7fd58809a120 2024-01-15T18:05:40.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.419+0000 7fd57fe00640 1 -- 172.21.15.27:0/2038674394 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5780075e0 con 0x7fd58809a120 2024-01-15T18:05:40.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.419+0000 7fd57fe00640 1 -- 172.21.15.27:0/2038674394 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd578004a10 con 0x7fd58809a120 2024-01-15T18:05:40.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.419+0000 7fd58f000640 1 -- 172.21.15.27:0/2038674394 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5880c5970 con 0x7fd58809a120 2024-01-15T18:05:40.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.419+0000 7fd58f000640 1 -- 172.21.15.27:0/2038674394 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd58815b7d0 con 0x7fd58809a120 2024-01-15T18:05:40.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.419+0000 7fd58f000640 1 -- 172.21.15.27:0/2038674394 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd58804e850 con 0x7fd58809a120 2024-01-15T18:05:40.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.419+0000 7fd57fe00640 1 -- 172.21.15.27:0/2038674394 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fd578004b70 con 0x7fd58809a120 2024-01-15T18:05:40.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.419+0000 7fd57fe00640 1 --2- 172.21.15.27:0/2038674394 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd5600668f0 0x7fd560068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.419+0000 7fd57fe00640 1 -- 172.21.15.27:0/2038674394 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fd578083650 con 0x7fd58809a120 2024-01-15T18:05:40.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.419+0000 7fd587400640 1 --2- 172.21.15.27:0/2038674394 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd5600668f0 0x7fd560068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.423+0000 7fd587400640 1 --2- 172.21.15.27:0/2038674394 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd5600668f0 0x7fd560068db0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd5700ad000 tx=0x7fd5700a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:40.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.423+0000 7fd57fe00640 1 -- 172.21.15.27:0/2038674394 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd578050810 con 0x7fd58809a120 2024-01-15T18:05:40.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.595+0000 7fd58f000640 1 -- 172.21.15.27:0/2038674394 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd5880b6ed0 con 0x7fd58809a120 2024-01-15T18:05:40.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.595+0000 7fd57fe00640 1 -- 172.21.15.27:0/2038674394 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd578053e30 con 0x7fd58809a120 2024-01-15T18:05:40.601 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-01-15T18:05:40.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.595+0000 7fd58f000640 1 -- 172.21.15.27:0/2038674394 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd5600668f0 msgr2=0x7fd560068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.595+0000 7fd58f000640 1 --2- 172.21.15.27:0/2038674394 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd5600668f0 0x7fd560068db0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd5700ad000 tx=0x7fd5700a6000 comp rx=0 tx=0).stop 2024-01-15T18:05:40.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.595+0000 7fd58f000640 1 -- 172.21.15.27:0/2038674394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd58809a120 msgr2=0x7fd58815adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.595+0000 7fd58f000640 1 --2- 172.21.15.27:0/2038674394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd58809a120 0x7fd58815adc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd57800c490 tx=0x7fd57800c950 comp rx=0 tx=0).stop 2024-01-15T18:05:40.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.599+0000 7fd58f000640 1 -- 172.21.15.27:0/2038674394 shutdown_connections 2024-01-15T18:05:40.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.599+0000 7fd58f000640 1 --2- 172.21.15.27:0/2038674394 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd5600668f0 0x7fd560068db0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.599+0000 7fd58f000640 1 --2- 172.21.15.27:0/2038674394 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd58815d7c0 0x7fd58815f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.599+0000 7fd58f000640 1 --2- 172.21.15.27:0/2038674394 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5880b1d90 0x7fd588152bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.599+0000 7fd58f000640 1 --2- 172.21.15.27:0/2038674394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd58809a120 0x7fd58815adc0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.599+0000 7fd58f000640 1 -- 172.21.15.27:0/2038674394 >> 172.21.15.27:0/2038674394 conn(0x7fd5880b2a80 msgr2=0x7fd5880ccc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:40.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.599+0000 7fd58f000640 1 -- 172.21.15.27:0/2038674394 shutdown_connections 2024-01-15T18:05:40.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.599+0000 7fd58f000640 1 -- 172.21.15.27:0/2038674394 wait complete. 2024-01-15T18:05:40.623 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:05:40.624 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:05:40.633 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:40.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.807+0000 7f226e600640 1 Processor -- start 2024-01-15T18:05:40.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.807+0000 7f226e600640 1 -- start start 2024-01-15T18:05:40.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.811+0000 7f226e600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2268143b90 0x7f22680cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.811+0000 7f226e600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f226806d7c0 0x7f22680ad810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.811+0000 7f226e600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22680ce080 con 0x7f22680cf160 2024-01-15T18:05:40.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.811+0000 7f226e600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22680acdc0 con 0x7f2268143b90 2024-01-15T18:05:40.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.811+0000 7f226e600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22680ad380 con 0x7f226806d7c0 2024-01-15T18:05:40.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.811+0000 7f2267400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2268143b90 0x7f22680cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.811+0000 7f2267400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2268143b90 0x7f22680cec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35126/0 (socket says 172.21.15.27:35126) 2024-01-15T18:05:40.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.811+0000 7f2267400640 1 -- 172.21.15.27:0/1808565544 learned_addr learned my addr 172.21.15.27:0/1808565544 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:40.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.811+0000 7f2267e00640 1 --2- 172.21.15.27:0/1808565544 >> v2:172.21.15.93:3300/0 conn(0x7f226806d7c0 0x7f22680ad810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.811+0000 7f2267400640 1 -- 172.21.15.27:0/1808565544 >> v2:172.21.15.93:3300/0 conn(0x7f226806d7c0 msgr2=0x7f22680ad810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:40.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.811+0000 7f2267400640 1 --2- 172.21.15.27:0/1808565544 >> v2:172.21.15.93:3300/0 conn(0x7f226806d7c0 0x7f22680ad810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.811+0000 7f2267400640 1 -- 172.21.15.27:0/1808565544 >> v1:172.21.15.93:6789/0 conn(0x7f22680cf160 legacy=0x7f226806d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.811+0000 7f2267400640 1 -- 172.21.15.27:0/1808565544 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22680cf580 con 0x7f2268143b90 2024-01-15T18:05:40.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.815+0000 7f2267400640 1 --2- 172.21.15.27:0/1808565544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2268143b90 0x7f22680cec20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f22680af9c0 tx=0x7f225801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ddc5e7446478ba server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:40.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.815+0000 7f2266000640 1 -- 172.21.15.27:0/1808565544 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22580a89f0 con 0x7f2268143b90 2024-01-15T18:05:40.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.815+0000 7f2266000640 1 -- 172.21.15.27:0/1808565544 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22580a8b50 con 0x7f2268143b90 2024-01-15T18:05:40.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.815+0000 7f2266000640 1 -- 172.21.15.27:0/1808565544 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22580b0440 con 0x7f2268143b90 2024-01-15T18:05:40.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.815+0000 7f226e600640 1 -- 172.21.15.27:0/1808565544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2268143b90 msgr2=0x7f22680cec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.815+0000 7f226e600640 1 --2- 172.21.15.27:0/1808565544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2268143b90 0x7f22680cec20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f22680af9c0 tx=0x7f225801ecf0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.815+0000 7f226e600640 1 -- 172.21.15.27:0/1808565544 shutdown_connections 2024-01-15T18:05:40.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.815+0000 7f226e600640 1 --2- 172.21.15.27:0/1808565544 >> v2:172.21.15.93:3300/0 conn(0x7f226806d7c0 0x7f22680ad810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.815+0000 7f226e600640 1 --2- 172.21.15.27:0/1808565544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2268143b90 0x7f22680cec20 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.815+0000 7f226e600640 1 -- 172.21.15.27:0/1808565544 >> 172.21.15.27:0/1808565544 conn(0x7f22680b1ae0 msgr2=0x7f22680b1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:40.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.815+0000 7f226e600640 1 -- 172.21.15.27:0/1808565544 shutdown_connections 2024-01-15T18:05:40.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.815+0000 7f226e600640 1 -- 172.21.15.27:0/1808565544 wait complete. 2024-01-15T18:05:40.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.815+0000 7f226e600640 1 Processor -- start 2024-01-15T18:05:40.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.815+0000 7f226e600640 1 -- start start 2024-01-15T18:05:40.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.819+0000 7f226e600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f226806d7c0 0x7f226815d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.819+0000 7f226e600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22680cf160 0x7f22681f3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.819+0000 7f226e600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2268143b90 0x7f22681f3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.819+0000 7f226e600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226815d960 con 0x7f22680cf160 2024-01-15T18:05:40.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.819+0000 7f226e600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22681e5c60 con 0x7f226806d7c0 2024-01-15T18:05:40.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.819+0000 7f226e600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22681f3d50 con 0x7f2268143b90 2024-01-15T18:05:40.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.819+0000 7f2266a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22680cf160 0x7f22681f3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.819+0000 7f2266a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22680cf160 0x7f22681f3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35130/0 (socket says 172.21.15.27:35130) 2024-01-15T18:05:40.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.819+0000 7f2266a00640 1 -- 172.21.15.27:0/110216621 learned_addr learned my addr 172.21.15.27:0/110216621 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:40.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.819+0000 7f2267e00640 1 --2- 172.21.15.27:0/110216621 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2268143b90 0x7f22681f3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.819+0000 7f2267400640 1 --2- 172.21.15.27:0/110216621 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f226806d7c0 0x7f226815d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.819+0000 7f2266a00640 1 -- 172.21.15.27:0/110216621 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2268143b90 msgr2=0x7f22681f3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.819+0000 7f2266a00640 1 --2- 172.21.15.27:0/110216621 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2268143b90 0x7f22681f3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.819+0000 7f2266a00640 1 -- 172.21.15.27:0/110216621 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f226806d7c0 msgr2=0x7f226815d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.819+0000 7f2266a00640 1 --2- 172.21.15.27:0/110216621 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f226806d7c0 0x7f226815d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.819+0000 7f2266a00640 1 -- 172.21.15.27:0/110216621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22580094d0 con 0x7f22680cf160 2024-01-15T18:05:40.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.819+0000 7f2266a00640 1 --2- 172.21.15.27:0/110216621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22680cf160 0x7f22681f3810 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f225000c530 tx=0x7f225000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:40.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.819+0000 7f225f400640 1 -- 172.21.15.27:0/110216621 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2250005220 con 0x7f22680cf160 2024-01-15T18:05:40.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.819+0000 7f225f400640 1 -- 172.21.15.27:0/110216621 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2250005380 con 0x7f22680cf160 2024-01-15T18:05:40.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.819+0000 7f226e600640 1 -- 172.21.15.27:0/110216621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22681f4490 con 0x7f22680cf160 2024-01-15T18:05:40.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.819+0000 7f225f400640 1 -- 172.21.15.27:0/110216621 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2250007270 con 0x7f22680cf160 2024-01-15T18:05:40.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.819+0000 7f226e600640 1 -- 172.21.15.27:0/110216621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22681e5330 con 0x7f22680cf160 2024-01-15T18:05:40.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.819+0000 7f226e600640 1 -- 172.21.15.27:0/110216621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2268067a10 con 0x7f22680cf160 2024-01-15T18:05:40.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.819+0000 7f225f400640 1 -- 172.21.15.27:0/110216621 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f22500073d0 con 0x7f22680cf160 2024-01-15T18:05:40.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.823+0000 7f225f400640 1 --2- 172.21.15.27:0/110216621 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f22400668f0 0x7f2240068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.823+0000 7f225f400640 1 -- 172.21.15.27:0/110216621 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f2250083b80 con 0x7f22680cf160 2024-01-15T18:05:40.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.823+0000 7f2267400640 1 --2- 172.21.15.27:0/110216621 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f22400668f0 0x7f2240068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.823+0000 7f2267400640 1 --2- 172.21.15.27:0/110216621 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f22400668f0 0x7f2240068db0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f22580b2000 tx=0x7f2258009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:40.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.827+0000 7f225f400640 1 -- 172.21.15.27:0/110216621 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2250050e70 con 0x7f22680cf160 2024-01-15T18:05:40.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.983+0000 7f226e600640 1 -- 172.21.15.27:0/110216621 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22681e5e10 con 0x7f22680cf160 2024-01-15T18:05:40.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.987+0000 7f225f400640 1 -- 172.21.15.27:0/110216621 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16705 (secure 0 0 0) 0x7f22681e5e10 con 0x7f22680cf160 2024-01-15T18:05:40.991 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:40.991 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":23,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:05:40.008536+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:04:56.220608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T18:04:57.210421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:40.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.987+0000 7f226e600640 1 -- 172.21.15.27:0/110216621 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f22400668f0 msgr2=0x7f2240068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.987+0000 7f226e600640 1 --2- 172.21.15.27:0/110216621 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f22400668f0 0x7f2240068db0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f22580b2000 tx=0x7f2258009040 comp rx=0 tx=0).stop 2024-01-15T18:05:40.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.987+0000 7f226e600640 1 -- 172.21.15.27:0/110216621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22680cf160 msgr2=0x7f22681f3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.987+0000 7f226e600640 1 --2- 172.21.15.27:0/110216621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22680cf160 0x7f22681f3810 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f225000c530 tx=0x7f225000c9f0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.987+0000 7f226e600640 1 -- 172.21.15.27:0/110216621 shutdown_connections 2024-01-15T18:05:40.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.987+0000 7f226e600640 1 --2- 172.21.15.27:0/110216621 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f22400668f0 0x7f2240068db0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.987+0000 7f226e600640 1 --2- 172.21.15.27:0/110216621 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2268143b90 0x7f22681f3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.987+0000 7f226e600640 1 --2- 172.21.15.27:0/110216621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22680cf160 0x7f22681f3810 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.987+0000 7f226e600640 1 --2- 172.21.15.27:0/110216621 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f226806d7c0 0x7f226815d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.987+0000 7f226e600640 1 -- 172.21.15.27:0/110216621 >> 172.21.15.27:0/110216621 conn(0x7f22680b1ae0 msgr2=0x7f22680aec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:40.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.987+0000 7f226e600640 1 -- 172.21.15.27:0/110216621 shutdown_connections 2024-01-15T18:05:40.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:40.987+0000 7f226e600640 1 -- 172.21.15.27:0/110216621 wait complete. 2024-01-15T18:05:41.013 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T18:04:56.220608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T18:04:57.210421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:05:41.013 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:05:41.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.151+0000 7f8741c00640 1 Processor -- start 2024-01-15T18:05:41.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.151+0000 7f8741c00640 1 -- start start 2024-01-15T18:05:41.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.155+0000 7f8741c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f873c0cf450 0x7f873c0d1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:41.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.155+0000 7f8741c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f873c0b0650 0x7f873c0b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:41.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.155+0000 7f8741c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f873c0cefa0 con 0x7f873c0cba60 2024-01-15T18:05:41.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.155+0000 7f8741c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f873c0ce240 con 0x7f873c0b0650 2024-01-15T18:05:41.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.155+0000 7f8741c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f873c0ccd70 con 0x7f873c0cf450 2024-01-15T18:05:41.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.155+0000 7f873b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f873c0b0650 0x7f873c0b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:41.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.155+0000 7f873b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f873c0b0650 0x7f873c0b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35138/0 (socket says 172.21.15.27:35138) 2024-01-15T18:05:41.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.155+0000 7f873b400640 1 -- 172.21.15.27:0/4208596020 learned_addr learned my addr 172.21.15.27:0/4208596020 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:41.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.155+0000 7f873aa00640 1 --2- 172.21.15.27:0/4208596020 >> v2:172.21.15.93:3300/0 conn(0x7f873c0cf450 0x7f873c0d1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:41.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.155+0000 7f873b400640 1 -- 172.21.15.27:0/4208596020 >> v2:172.21.15.93:3300/0 conn(0x7f873c0cf450 msgr2=0x7f873c0d1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:41.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.155+0000 7f873b400640 1 --2- 172.21.15.27:0/4208596020 >> v2:172.21.15.93:3300/0 conn(0x7f873c0cf450 0x7f873c0d1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.155+0000 7f873b400640 1 -- 172.21.15.27:0/4208596020 >> v1:172.21.15.93:6789/0 conn(0x7f873c0cba60 legacy=0x7f873c0cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:41.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.155+0000 7f873b400640 1 -- 172.21.15.27:0/4208596020 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873c0cc1f0 con 0x7f873c0b0650 2024-01-15T18:05:41.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.159+0000 7f873b400640 1 --2- 172.21.15.27:0/4208596020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f873c0b0650 0x7f873c0b0a80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f873c0713f0 tx=0x7f873401c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0545d88e4438ac9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:41.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.159+0000 7f8739600640 1 -- 172.21.15.27:0/4208596020 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87340a5780 con 0x7f873c0b0650 2024-01-15T18:05:41.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.159+0000 7f8739600640 1 -- 172.21.15.27:0/4208596020 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87340a58e0 con 0x7f873c0b0650 2024-01-15T18:05:41.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.159+0000 7f8739600640 1 -- 172.21.15.27:0/4208596020 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87340a5c70 con 0x7f873c0b0650 2024-01-15T18:05:41.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.159+0000 7f8741c00640 1 -- 172.21.15.27:0/4208596020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f873c0b0650 msgr2=0x7f873c0b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:41.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.159+0000 7f8741c00640 1 --2- 172.21.15.27:0/4208596020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f873c0b0650 0x7f873c0b0a80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f873c0713f0 tx=0x7f873401c9d0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.159+0000 7f8741c00640 1 -- 172.21.15.27:0/4208596020 shutdown_connections 2024-01-15T18:05:41.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.159+0000 7f8741c00640 1 --2- 172.21.15.27:0/4208596020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f873c0b0650 0x7f873c0b0a80 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.159+0000 7f8741c00640 1 --2- 172.21.15.27:0/4208596020 >> v2:172.21.15.93:3300/0 conn(0x7f873c0cf450 0x7f873c0d1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.159+0000 7f8741c00640 1 -- 172.21.15.27:0/4208596020 >> 172.21.15.27:0/4208596020 conn(0x7f873c0d0e70 msgr2=0x7f873c0d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:41.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.159+0000 7f8741c00640 1 -- 172.21.15.27:0/4208596020 shutdown_connections 2024-01-15T18:05:41.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.159+0000 7f8741c00640 1 -- 172.21.15.27:0/4208596020 wait complete. 2024-01-15T18:05:41.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.159+0000 7f8741c00640 1 Processor -- start 2024-01-15T18:05:41.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.159+0000 7f8741c00640 1 -- start start 2024-01-15T18:05:41.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.163+0000 7f8741c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f873c0b0650 0x7f873c0c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:41.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.163+0000 7f8741c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f873c0cba60 0x7f873c0b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:41.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.163+0000 7f8741c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f873c0cf450 0x7f873c0b2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:41.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.163+0000 7f8741c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873c0c95f0 con 0x7f873c0cf450 2024-01-15T18:05:41.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.163+0000 7f8741c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873c0c5ba0 con 0x7f873c0cba60 2024-01-15T18:05:41.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.163+0000 7f8741c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f873c0c5df0 con 0x7f873c0b0650 2024-01-15T18:05:41.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.163+0000 7f873b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f873c0cf450 0x7f873c0b2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:41.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.163+0000 7f873b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f873c0cf450 0x7f873c0b2bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35142/0 (socket says 172.21.15.27:35142) 2024-01-15T18:05:41.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.163+0000 7f873b400640 1 -- 172.21.15.27:0/889098044 learned_addr learned my addr 172.21.15.27:0/889098044 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:41.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.163+0000 7f873aa00640 1 --2- 172.21.15.27:0/889098044 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f873c0b0650 0x7f873c0c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:41.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.163+0000 7f873a000640 1 --2- 172.21.15.27:0/889098044 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f873c0cba60 0x7f873c0b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:41.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.163+0000 7f873b400640 1 -- 172.21.15.27:0/889098044 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f873c0b0650 msgr2=0x7f873c0c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:41.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.163+0000 7f873b400640 1 --2- 172.21.15.27:0/889098044 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f873c0b0650 0x7f873c0c90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.163+0000 7f873b400640 1 -- 172.21.15.27:0/889098044 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f873c0cba60 msgr2=0x7f873c0b3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:41.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.163+0000 7f873b400640 1 --2- 172.21.15.27:0/889098044 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f873c0cba60 0x7f873c0b3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.163+0000 7f873b400640 1 -- 172.21.15.27:0/889098044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8734009b30 con 0x7f873c0cf450 2024-01-15T18:05:41.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.163+0000 7f873b400640 1 --2- 172.21.15.27:0/889098044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f873c0cf450 0x7f873c0b2bc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f87340af040 tx=0x7f8734019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:41.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.163+0000 7f8732a00640 1 -- 172.21.15.27:0/889098044 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87340b8670 con 0x7f873c0cf450 2024-01-15T18:05:41.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.163+0000 7f8732a00640 1 -- 172.21.15.27:0/889098044 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87340b87d0 con 0x7f873c0cf450 2024-01-15T18:05:41.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.163+0000 7f8741c00640 1 -- 172.21.15.27:0/889098044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f873c0c60d0 con 0x7f873c0cf450 2024-01-15T18:05:41.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.163+0000 7f8732a00640 1 -- 172.21.15.27:0/889098044 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87340b8b80 con 0x7f873c0cf450 2024-01-15T18:05:41.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.163+0000 7f8741c00640 1 -- 172.21.15.27:0/889098044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f873c0c6690 con 0x7f873c0cf450 2024-01-15T18:05:41.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.163+0000 7f8741c00640 1 -- 172.21.15.27:0/889098044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f873c04e850 con 0x7f873c0cf450 2024-01-15T18:05:41.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.167+0000 7f8732a00640 1 -- 172.21.15.27:0/889098044 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f87340b8ce0 con 0x7f873c0cf450 2024-01-15T18:05:41.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.167+0000 7f8732a00640 1 --2- 172.21.15.27:0/889098044 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f87140668f0 0x7f8714068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:41.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.167+0000 7f8732a00640 1 -- 172.21.15.27:0/889098044 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f873411f830 con 0x7f873c0cf450 2024-01-15T18:05:41.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.167+0000 7f873aa00640 1 --2- 172.21.15.27:0/889098044 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f87140668f0 0x7f8714068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:41.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.167+0000 7f873aa00640 1 --2- 172.21.15.27:0/889098044 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f87140668f0 0x7f8714068db0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f873c15ab40 tx=0x7f872c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:41.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.171+0000 7f8732a00640 1 -- 172.21.15.27:0/889098044 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f87340ecbc0 con 0x7f873c0cf450 2024-01-15T18:05:41.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.331+0000 7f8741c00640 1 -- 172.21.15.27:0/889098044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f873c0cf230 con 0x7f873c0cf450 2024-01-15T18:05:41.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.331+0000 7f8732a00640 1 -- 172.21.15.27:0/889098044 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f87340bc0a0 con 0x7f873c0cf450 2024-01-15T18:05:41.336 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:05:41.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.331+0000 7f8741c00640 1 -- 172.21.15.27:0/889098044 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f87140668f0 msgr2=0x7f8714068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:41.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.331+0000 7f8741c00640 1 --2- 172.21.15.27:0/889098044 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f87140668f0 0x7f8714068db0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f873c15ab40 tx=0x7f872c009190 comp rx=0 tx=0).stop 2024-01-15T18:05:41.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.331+0000 7f8741c00640 1 -- 172.21.15.27:0/889098044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f873c0cf450 msgr2=0x7f873c0b2bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:41.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.331+0000 7f8741c00640 1 --2- 172.21.15.27:0/889098044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f873c0cf450 0x7f873c0b2bc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f87340af040 tx=0x7f8734019770 comp rx=0 tx=0).stop 2024-01-15T18:05:41.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.331+0000 7f8741c00640 1 -- 172.21.15.27:0/889098044 shutdown_connections 2024-01-15T18:05:41.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.331+0000 7f8741c00640 1 --2- 172.21.15.27:0/889098044 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f87140668f0 0x7f8714068db0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.331+0000 7f8741c00640 1 --2- 172.21.15.27:0/889098044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f873c0cf450 0x7f873c0b2bc0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.331+0000 7f8741c00640 1 --2- 172.21.15.27:0/889098044 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f873c0cba60 0x7f873c0b3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.331+0000 7f8741c00640 1 --2- 172.21.15.27:0/889098044 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f873c0b0650 0x7f873c0c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.331+0000 7f8741c00640 1 -- 172.21.15.27:0/889098044 >> 172.21.15.27:0/889098044 conn(0x7f873c0d0e70 msgr2=0x7f873c06e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:41.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.331+0000 7f8741c00640 1 -- 172.21.15.27:0/889098044 shutdown_connections 2024-01-15T18:05:41.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.335+0000 7f8741c00640 1 -- 172.21.15.27:0/889098044 wait complete. 2024-01-15T18:05:41.358 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T18:05:41.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.495+0000 7f1227200640 1 Processor -- start 2024-01-15T18:05:41.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.495+0000 7f1227200640 1 -- start start 2024-01-15T18:05:41.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.499+0000 7f1227200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f12200cef20 0x7f12200ce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:41.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.499+0000 7f1227200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f12200d19f0 0x7f12200b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:41.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.499+0000 7f1227200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f122015b6b0 con 0x7f12200cca40 2024-01-15T18:05:41.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.499+0000 7f1227200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12200d1e50 con 0x7f12200d19f0 2024-01-15T18:05:41.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.499+0000 7f1227200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12200cb9e0 con 0x7f12200cef20 2024-01-15T18:05:41.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.499+0000 7f1225200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f12200d19f0 0x7f12200b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:41.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.499+0000 7f121f400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f12200cca40 0x7f12200cf300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35124/0 (socket says 172.21.15.27:35124) 2024-01-15T18:05:41.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.499+0000 7f121f400640 1 -- 172.21.15.27:0/3145485826 learned_addr learned my addr 172.21.15.27:0/3145485826 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:41.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.499+0000 7f121fe00640 1 --2- 172.21.15.27:0/3145485826 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f12200cef20 0x7f12200ce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:41.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.499+0000 7f1225200640 1 -- 172.21.15.27:0/3145485826 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f12200cef20 msgr2=0x7f12200ce2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:41.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.499+0000 7f1225200640 1 --2- 172.21.15.27:0/3145485826 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f12200cef20 0x7f12200ce2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.499+0000 7f1225200640 1 -- 172.21.15.27:0/3145485826 >> v1:172.21.15.27:6789/0 conn(0x7f12200cca40 legacy=0x7f12200cf300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:41.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.499+0000 7f1225200640 1 -- 172.21.15.27:0/3145485826 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12200cfdb0 con 0x7f12200d19f0 2024-01-15T18:05:41.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.499+0000 7f1225200640 1 --2- 172.21.15.27:0/3145485826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12200d19f0 0x7f12200b3d00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1218009d90 tx=0x7f121801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a86b26cc83d6968f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:41.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.503+0000 7f121ea00640 1 -- 172.21.15.27:0/3145485826 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12180a5900 con 0x7f12200d19f0 2024-01-15T18:05:41.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.503+0000 7f121ea00640 1 -- 172.21.15.27:0/3145485826 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12180a5a60 con 0x7f12200d19f0 2024-01-15T18:05:41.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.503+0000 7f121ea00640 1 -- 172.21.15.27:0/3145485826 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12180a5df0 con 0x7f12200d19f0 2024-01-15T18:05:41.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.503+0000 7f1227200640 1 -- 172.21.15.27:0/3145485826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12200d19f0 msgr2=0x7f12200b3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:41.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.503+0000 7f1227200640 1 --2- 172.21.15.27:0/3145485826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12200d19f0 0x7f12200b3d00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1218009d90 tx=0x7f121801c950 comp rx=0 tx=0).stop 2024-01-15T18:05:41.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.503+0000 7f1227200640 1 -- 172.21.15.27:0/3145485826 shutdown_connections 2024-01-15T18:05:41.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.503+0000 7f1227200640 1 --2- 172.21.15.27:0/3145485826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12200d19f0 0x7f12200b3d00 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.503+0000 7f1227200640 1 --2- 172.21.15.27:0/3145485826 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f12200cef20 0x7f12200ce2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.503+0000 7f1227200640 1 -- 172.21.15.27:0/3145485826 >> 172.21.15.27:0/3145485826 conn(0x7f12200d0ed0 msgr2=0x7f12200d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:41.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.503+0000 7f1227200640 1 -- 172.21.15.27:0/3145485826 shutdown_connections 2024-01-15T18:05:41.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.503+0000 7f1227200640 1 -- 172.21.15.27:0/3145485826 wait complete. 2024-01-15T18:05:41.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.503+0000 7f1227200640 1 Processor -- start 2024-01-15T18:05:41.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.503+0000 7f1227200640 1 -- start start 2024-01-15T18:05:41.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.503+0000 7f1227200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f12200cca40 0x7f122014ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:41.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.503+0000 7f1227200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f12200cef20 0x7f122006ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:41.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.503+0000 7f1227200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12200d19f0 0x7f122006b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:41.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.503+0000 7f1227200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122006d210 con 0x7f12200d19f0 2024-01-15T18:05:41.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.503+0000 7f1227200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f122015e380 con 0x7f12200cef20 2024-01-15T18:05:41.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.503+0000 7f1227200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f122015e660 con 0x7f12200cca40 2024-01-15T18:05:41.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.507+0000 7f1225200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12200d19f0 0x7f122006b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:41.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.507+0000 7f1225200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12200d19f0 0x7f122006b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35170/0 (socket says 172.21.15.27:35170) 2024-01-15T18:05:41.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.507+0000 7f1225200640 1 -- 172.21.15.27:0/1605940374 learned_addr learned my addr 172.21.15.27:0/1605940374 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:41.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.507+0000 7f121fe00640 1 --2- 172.21.15.27:0/1605940374 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f12200cca40 0x7f122014ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:41.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.507+0000 7f121f400640 1 --2- 172.21.15.27:0/1605940374 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f12200cef20 0x7f122006ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:41.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.507+0000 7f1225200640 1 -- 172.21.15.27:0/1605940374 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f12200cca40 msgr2=0x7f122014ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:41.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.507+0000 7f1225200640 1 --2- 172.21.15.27:0/1605940374 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f12200cca40 0x7f122014ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.507+0000 7f1225200640 1 -- 172.21.15.27:0/1605940374 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f12200cef20 msgr2=0x7f122006ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:41.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.507+0000 7f1225200640 1 --2- 172.21.15.27:0/1605940374 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f12200cef20 0x7f122006ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.507+0000 7f1225200640 1 -- 172.21.15.27:0/1605940374 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1218009a70 con 0x7f12200d19f0 2024-01-15T18:05:41.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.507+0000 7f1225200640 1 --2- 172.21.15.27:0/1605940374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12200d19f0 0x7f122006b600 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f12180b0040 tx=0x7f12180196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:41.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.507+0000 7f1217e00640 1 -- 172.21.15.27:0/1605940374 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12180b9650 con 0x7f12200d19f0 2024-01-15T18:05:41.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.507+0000 7f1217e00640 1 -- 172.21.15.27:0/1605940374 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12180b97b0 con 0x7f12200d19f0 2024-01-15T18:05:41.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.507+0000 7f1227200640 1 -- 172.21.15.27:0/1605940374 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1220163180 con 0x7f12200d19f0 2024-01-15T18:05:41.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.507+0000 7f1217e00640 1 -- 172.21.15.27:0/1605940374 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12180b9b40 con 0x7f12200d19f0 2024-01-15T18:05:41.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.507+0000 7f1227200640 1 -- 172.21.15.27:0/1605940374 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f122006bb40 con 0x7f12200d19f0 2024-01-15T18:05:41.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.507+0000 7f1227200640 1 -- 172.21.15.27:0/1605940374 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12200d09f0 con 0x7f12200d19f0 2024-01-15T18:05:41.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.507+0000 7f1217e00640 1 -- 172.21.15.27:0/1605940374 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f12180b9ca0 con 0x7f12200d19f0 2024-01-15T18:05:41.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.511+0000 7f1217e00640 1 --2- 172.21.15.27:0/1605940374 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f11f8066920 0x7f11f8068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:41.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.511+0000 7f1217e00640 1 -- 172.21.15.27:0/1605940374 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f121811f480 con 0x7f12200d19f0 2024-01-15T18:05:41.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.511+0000 7f121fe00640 1 --2- 172.21.15.27:0/1605940374 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f11f8066920 0x7f11f8068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:41.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.511+0000 7f121fe00640 1 --2- 172.21.15.27:0/1605940374 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f11f8066920 0x7f11f8068de0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f122014eaa0 tx=0x7f1210009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:41.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.515+0000 7f1217e00640 1 -- 172.21.15.27:0/1605940374 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f12180ec760 con 0x7f12200d19f0 2024-01-15T18:05:41.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.671+0000 7f1227200640 1 -- 172.21.15.27:0/1605940374 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1220151fe0 con 0x7f12200d19f0 2024-01-15T18:05:41.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.671+0000 7f1217e00640 1 -- 172.21.15.27:0/1605940374 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1220151fe0 con 0x7f12200d19f0 2024-01-15T18:05:41.677 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-15T18:05:41.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.671+0000 7f1227200640 1 -- 172.21.15.27:0/1605940374 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f11f8066920 msgr2=0x7f11f8068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:41.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.671+0000 7f1227200640 1 --2- 172.21.15.27:0/1605940374 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f11f8066920 0x7f11f8068de0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f122014eaa0 tx=0x7f1210009190 comp rx=0 tx=0).stop 2024-01-15T18:05:41.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.671+0000 7f1227200640 1 -- 172.21.15.27:0/1605940374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12200d19f0 msgr2=0x7f122006b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:41.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.671+0000 7f1227200640 1 --2- 172.21.15.27:0/1605940374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12200d19f0 0x7f122006b600 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f12180b0040 tx=0x7f12180196f0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.675+0000 7f1227200640 1 -- 172.21.15.27:0/1605940374 shutdown_connections 2024-01-15T18:05:41.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.675+0000 7f1227200640 1 --2- 172.21.15.27:0/1605940374 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f11f8066920 0x7f11f8068de0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.675+0000 7f1227200640 1 --2- 172.21.15.27:0/1605940374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12200d19f0 0x7f122006b600 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.675+0000 7f1227200640 1 --2- 172.21.15.27:0/1605940374 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f12200cef20 0x7f122006ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.675+0000 7f1227200640 1 --2- 172.21.15.27:0/1605940374 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f12200cca40 0x7f122014ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.675+0000 7f1227200640 1 -- 172.21.15.27:0/1605940374 >> 172.21.15.27:0/1605940374 conn(0x7f12200d0ed0 msgr2=0x7f1220150db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:41.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.675+0000 7f1227200640 1 -- 172.21.15.27:0/1605940374 shutdown_connections 2024-01-15T18:05:41.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.675+0000 7f1227200640 1 -- 172.21.15.27:0/1605940374 wait complete. 2024-01-15T18:05:41.698 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T18:05:41.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.835+0000 7fe8eec00640 1 Processor -- start 2024-01-15T18:05:41.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.835+0000 7fe8eec00640 1 -- start start 2024-01-15T18:05:41.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.839+0000 7fe8eec00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8e806c520 0x7fe8e80cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:41.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.839+0000 7fe8eec00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e80b0d70 con 0x7fe8e815d190 2024-01-15T18:05:41.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.839+0000 7fe8eec00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e80d0540 con 0x7fe8e815d570 2024-01-15T18:05:41.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.839+0000 7fe8eec00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e80b0440 con 0x7fe8e806c520 2024-01-15T18:05:41.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.839+0000 7fe8e7e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe8e815d190 0x7fe8e80cfcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35140/0 (socket says 172.21.15.27:35140) 2024-01-15T18:05:41.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.839+0000 7fe8e7e00640 1 -- 172.21.15.27:0/13378976 learned_addr learned my addr 172.21.15.27:0/13378976 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:41.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.839+0000 7fe8ecc00640 1 --2- 172.21.15.27:0/13378976 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8e806c520 0x7fe8e80cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:41.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.839+0000 7fe8e6a00640 1 -- 172.21.15.27:0/13378976 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4172342775 0 0) 0x7fe8e80b0d70 con 0x7fe8e815d190 2024-01-15T18:05:41.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.839+0000 7fe8e6a00640 1 -- 172.21.15.27:0/13378976 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8c8003530 con 0x7fe8e815d190 2024-01-15T18:05:41.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.839+0000 7fe8e6a00640 1 -- 172.21.15.27:0/13378976 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1383335025 0 0) 0x7fe8e80d0540 con 0x7fe8e815d570 2024-01-15T18:05:41.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.839+0000 7fe8e6a00640 1 -- 172.21.15.27:0/13378976 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8e80b0d70 con 0x7fe8e815d570 2024-01-15T18:05:41.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.839+0000 7fe8ecc00640 1 -- 172.21.15.27:0/13378976 >> v1:172.21.15.93:6789/0 conn(0x7fe8e815d570 legacy=0x7fe8e806be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:41.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.839+0000 7fe8ecc00640 1 -- 172.21.15.27:0/13378976 >> v1:172.21.15.27:6789/0 conn(0x7fe8e815d190 legacy=0x7fe8e80cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:41.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.839+0000 7fe8e6a00640 1 -- 172.21.15.27:0/13378976 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 58741362 0 0) 0x7fe8c8003530 con 0x7fe8e815d190 2024-01-15T18:05:41.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.839+0000 7fe8ecc00640 1 -- 172.21.15.27:0/13378976 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e80b0720 con 0x7fe8e806c520 2024-01-15T18:05:41.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.843+0000 7fe8ecc00640 1 --2- 172.21.15.27:0/13378976 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8e806c520 0x7fe8e80cd5e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0009dc0 tx=0x7fe8e001c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2deb2ad81ce79cd6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:41.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.843+0000 7fe8e6a00640 1 -- 172.21.15.27:0/13378976 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d8001680 con 0x7fe8e806c520 2024-01-15T18:05:41.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.843+0000 7fe8e6a00640 1 -- 172.21.15.27:0/13378976 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8e00a56c0 con 0x7fe8e806c520 2024-01-15T18:05:41.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.843+0000 7fe8e6a00640 1 -- 172.21.15.27:0/13378976 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e00a5a50 con 0x7fe8e806c520 2024-01-15T18:05:41.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.843+0000 7fe8eec00640 1 -- 172.21.15.27:0/13378976 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8e806c520 msgr2=0x7fe8e80cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:41.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.843+0000 7fe8eec00640 1 --2- 172.21.15.27:0/13378976 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8e806c520 0x7fe8e80cd5e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0009dc0 tx=0x7fe8e001c860 comp rx=0 tx=0).stop 2024-01-15T18:05:41.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.843+0000 7fe8eec00640 1 -- 172.21.15.27:0/13378976 shutdown_connections 2024-01-15T18:05:41.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.843+0000 7fe8eec00640 1 --2- 172.21.15.27:0/13378976 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8e806c520 0x7fe8e80cd5e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.843+0000 7fe8eec00640 1 -- 172.21.15.27:0/13378976 >> 172.21.15.27:0/13378976 conn(0x7fe8e80d0e70 msgr2=0x7fe8e80d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:41.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.843+0000 7fe8eec00640 1 -- 172.21.15.27:0/13378976 shutdown_connections 2024-01-15T18:05:41.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.843+0000 7fe8eec00640 1 -- 172.21.15.27:0/13378976 wait complete. 2024-01-15T18:05:41.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.843+0000 7fe8eec00640 1 Processor -- start 2024-01-15T18:05:41.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.843+0000 7fe8eec00640 1 -- start start 2024-01-15T18:05:41.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.847+0000 7fe8eec00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe8e806c520 0x7fe8e81555f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:41.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.847+0000 7fe8eec00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8e815d190 0x7fe8e80b5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:41.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.847+0000 7fe8eec00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8e815d570 0x7fe8e815d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:41.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.847+0000 7fe8eec00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e8155b30 con 0x7fe8e815d570 2024-01-15T18:05:41.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.847+0000 7fe8eec00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e80b6560 con 0x7fe8e806c520 2024-01-15T18:05:41.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.847+0000 7fe8eec00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e80b6840 con 0x7fe8e815d190 2024-01-15T18:05:41.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.847+0000 7fe8ecc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8e815d570 0x7fe8e815d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:41.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.847+0000 7fe8ecc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8e815d570 0x7fe8e815d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35178/0 (socket says 172.21.15.27:35178) 2024-01-15T18:05:41.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.847+0000 7fe8ecc00640 1 -- 172.21.15.27:0/1181452042 learned_addr learned my addr 172.21.15.27:0/1181452042 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:41.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.847+0000 7fe8e7400640 1 --2- 172.21.15.27:0/1181452042 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8e815d190 0x7fe8e80b5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:41.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.847+0000 7fe8e7e00640 1 --2- 172.21.15.27:0/1181452042 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe8e806c520 0x7fe8e81555f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:41.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.847+0000 7fe8ecc00640 1 -- 172.21.15.27:0/1181452042 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8e815d190 msgr2=0x7fe8e80b5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:41.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.847+0000 7fe8ecc00640 1 --2- 172.21.15.27:0/1181452042 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8e815d190 0x7fe8e80b5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.847+0000 7fe8ecc00640 1 -- 172.21.15.27:0/1181452042 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe8e806c520 msgr2=0x7fe8e81555f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:41.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.847+0000 7fe8ecc00640 1 --2- 172.21.15.27:0/1181452042 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe8e806c520 0x7fe8e81555f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.847+0000 7fe8ecc00640 1 -- 172.21.15.27:0/1181452042 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e0009a70 con 0x7fe8e815d570 2024-01-15T18:05:41.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.847+0000 7fe8ecc00640 1 --2- 172.21.15.27:0/1181452042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8e815d570 0x7fe8e815d970 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe8e00b0040 tx=0x7fe8e0019600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:41.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.847+0000 7fe8dfe00640 1 -- 172.21.15.27:0/1181452042 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e0003ba0 con 0x7fe8e815d570 2024-01-15T18:05:41.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.847+0000 7fe8eec00640 1 -- 172.21.15.27:0/1181452042 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e80b6b20 con 0x7fe8e815d570 2024-01-15T18:05:41.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.847+0000 7fe8dfe00640 1 -- 172.21.15.27:0/1181452042 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8e0003d00 con 0x7fe8e815d570 2024-01-15T18:05:41.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.847+0000 7fe8dfe00640 1 -- 172.21.15.27:0/1181452042 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e00b98a0 con 0x7fe8e815d570 2024-01-15T18:05:41.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.847+0000 7fe8eec00640 1 -- 172.21.15.27:0/1181452042 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e80b5130 con 0x7fe8e815d570 2024-01-15T18:05:41.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.847+0000 7fe8eec00640 1 -- 172.21.15.27:0/1181452042 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8e80b5320 con 0x7fe8e815d570 2024-01-15T18:05:41.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.851+0000 7fe8dfe00640 1 -- 172.21.15.27:0/1181452042 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fe8e00b9a00 con 0x7fe8e815d570 2024-01-15T18:05:41.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.851+0000 7fe8dfe00640 1 --2- 172.21.15.27:0/1181452042 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe8c00668f0 0x7fe8c0068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:41.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.851+0000 7fe8dfe00640 1 -- 172.21.15.27:0/1181452042 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fe8e011f400 con 0x7fe8e815d570 2024-01-15T18:05:41.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.851+0000 7fe8e7e00640 1 --2- 172.21.15.27:0/1181452042 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe8c00668f0 0x7fe8c0068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:41.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.851+0000 7fe8e7e00640 1 --2- 172.21.15.27:0/1181452042 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe8c00668f0 0x7fe8c0068db0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe8d8000f70 tx=0x7fe8d8001700 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:41.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:41.855+0000 7fe8dfe00640 1 -- 172.21.15.27:0/1181452042 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe8e00ec5d0 con 0x7fe8e815d570 2024-01-15T18:05:42.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.011+0000 7fe8eec00640 1 -- 172.21.15.27:0/1181452042 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe8e804e850 con 0x7fe8e815d570 2024-01-15T18:05:42.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.011+0000 7fe8dfe00640 1 -- 172.21.15.27:0/1181452042 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe8e0009450 con 0x7fe8e815d570 2024-01-15T18:05:42.015 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-15T18:05:42.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.011+0000 7fe8eec00640 1 -- 172.21.15.27:0/1181452042 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe8c00668f0 msgr2=0x7fe8c0068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:42.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.011+0000 7fe8eec00640 1 --2- 172.21.15.27:0/1181452042 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe8c00668f0 0x7fe8c0068db0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe8d8000f70 tx=0x7fe8d8001700 comp rx=0 tx=0).stop 2024-01-15T18:05:42.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.011+0000 7fe8eec00640 1 -- 172.21.15.27:0/1181452042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8e815d570 msgr2=0x7fe8e815d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:42.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.011+0000 7fe8eec00640 1 --2- 172.21.15.27:0/1181452042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8e815d570 0x7fe8e815d970 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe8e00b0040 tx=0x7fe8e0019600 comp rx=0 tx=0).stop 2024-01-15T18:05:42.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.011+0000 7fe8eec00640 1 -- 172.21.15.27:0/1181452042 shutdown_connections 2024-01-15T18:05:42.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.011+0000 7fe8eec00640 1 --2- 172.21.15.27:0/1181452042 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe8c00668f0 0x7fe8c0068db0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.011+0000 7fe8eec00640 1 --2- 172.21.15.27:0/1181452042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8e815d570 0x7fe8e815d970 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.011+0000 7fe8eec00640 1 --2- 172.21.15.27:0/1181452042 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8e815d190 0x7fe8e80b5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.011+0000 7fe8eec00640 1 --2- 172.21.15.27:0/1181452042 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe8e806c520 0x7fe8e81555f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.011+0000 7fe8eec00640 1 -- 172.21.15.27:0/1181452042 >> 172.21.15.27:0/1181452042 conn(0x7fe8e80d0e70 msgr2=0x7fe8e806fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:42.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.011+0000 7fe8eec00640 1 -- 172.21.15.27:0/1181452042 shutdown_connections 2024-01-15T18:05:42.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.011+0000 7fe8eec00640 1 -- 172.21.15.27:0/1181452042 wait complete. 2024-01-15T18:05:42.038 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:05:42.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.175+0000 7fbef0800640 1 Processor -- start 2024-01-15T18:05:42.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.175+0000 7fbef0800640 1 -- start start 2024-01-15T18:05:42.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.179+0000 7fbef0800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbee80b2d70 0x7fbee80aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:42.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.179+0000 7fbef0800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbee806dad0 0x7fbee806a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:42.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.179+0000 7fbef0800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee8071f70 con 0x7fbee815d0c0 2024-01-15T18:05:42.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.179+0000 7fbef0800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbee80af630 con 0x7fbee806dad0 2024-01-15T18:05:42.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.179+0000 7fbef0800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbee80d0190 con 0x7fbee80b2d70 2024-01-15T18:05:42.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.179+0000 7fbeee800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbee806dad0 0x7fbee806a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:42.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.179+0000 7fbeee800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbee806dad0 0x7fbee806a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35180/0 (socket says 172.21.15.27:35180) 2024-01-15T18:05:42.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.179+0000 7fbeee800640 1 -- 172.21.15.27:0/2778304095 learned_addr learned my addr 172.21.15.27:0/2778304095 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:42.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.179+0000 7fbeed400640 1 --2- 172.21.15.27:0/2778304095 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbee80b2d70 0x7fbee80aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:42.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.179+0000 7fbeee800640 1 -- 172.21.15.27:0/2778304095 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbee80b2d70 msgr2=0x7fbee80aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:42.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.179+0000 7fbeee800640 1 --2- 172.21.15.27:0/2778304095 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbee80b2d70 0x7fbee80aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.179+0000 7fbeee800640 1 -- 172.21.15.27:0/2778304095 >> v1:172.21.15.93:6789/0 conn(0x7fbee815d0c0 legacy=0x7fbee80cfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:42.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.179+0000 7fbeee800640 1 -- 172.21.15.27:0/2778304095 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee8072210 con 0x7fbee806dad0 2024-01-15T18:05:42.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.179+0000 7fbeee800640 1 --2- 172.21.15.27:0/2778304095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee806dad0 0x7fbee806a4f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fbee0009e80 tx=0x7fbee001d450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3823cf628d7a16db server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:42.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.183+0000 7fbeeca00640 1 -- 172.21.15.27:0/2778304095 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee00a57d0 con 0x7fbee806dad0 2024-01-15T18:05:42.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.183+0000 7fbeeca00640 1 -- 172.21.15.27:0/2778304095 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbee00a5930 con 0x7fbee806dad0 2024-01-15T18:05:42.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.183+0000 7fbeeca00640 1 -- 172.21.15.27:0/2778304095 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee00a5cc0 con 0x7fbee806dad0 2024-01-15T18:05:42.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.183+0000 7fbef0800640 1 -- 172.21.15.27:0/2778304095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee806dad0 msgr2=0x7fbee806a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:42.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.183+0000 7fbef0800640 1 --2- 172.21.15.27:0/2778304095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee806dad0 0x7fbee806a4f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fbee0009e80 tx=0x7fbee001d450 comp rx=0 tx=0).stop 2024-01-15T18:05:42.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.183+0000 7fbef0800640 1 -- 172.21.15.27:0/2778304095 shutdown_connections 2024-01-15T18:05:42.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.183+0000 7fbef0800640 1 --2- 172.21.15.27:0/2778304095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee806dad0 0x7fbee806a4f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.183+0000 7fbef0800640 1 --2- 172.21.15.27:0/2778304095 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbee80b2d70 0x7fbee80aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.183+0000 7fbef0800640 1 -- 172.21.15.27:0/2778304095 >> 172.21.15.27:0/2778304095 conn(0x7fbee806b600 msgr2=0x7fbee806ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:42.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.183+0000 7fbef0800640 1 -- 172.21.15.27:0/2778304095 shutdown_connections 2024-01-15T18:05:42.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.183+0000 7fbef0800640 1 -- 172.21.15.27:0/2778304095 wait complete. 2024-01-15T18:05:42.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.183+0000 7fbef0800640 1 Processor -- start 2024-01-15T18:05:42.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.183+0000 7fbef0800640 1 -- start start 2024-01-15T18:05:42.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbef0800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbee806dad0 0x7fbee815c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:42.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbef0800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbee80b2d70 0x7fbee8154e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:42.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbef0800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee815d0c0 0x7fbee8161dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:42.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbef0800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee815ceb0 con 0x7fbee815d0c0 2024-01-15T18:05:42.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbef0800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee815e5c0 con 0x7fbee806dad0 2024-01-15T18:05:42.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbef0800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbee815e8a0 con 0x7fbee80b2d70 2024-01-15T18:05:42.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbeed400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbee80b2d70 0x7fbee8154e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:42.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbeede00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbee806dad0 0x7fbee815c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:42.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbeee800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee815d0c0 0x7fbee8161dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:42.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbeed400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbee80b2d70 0x7fbee8154e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:43474/0 (socket says 172.21.15.27:43474) 2024-01-15T18:05:42.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbeede00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbee806dad0 0x7fbee815c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:51196/0 (socket says 172.21.15.27:51196) 2024-01-15T18:05:42.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbeed400640 1 -- 172.21.15.27:0/1270106091 learned_addr learned my addr 172.21.15.27:0/1270106091 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:42.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbeee800640 1 -- 172.21.15.27:0/1270106091 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbee80b2d70 msgr2=0x7fbee8154e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:42.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbeee800640 1 --2- 172.21.15.27:0/1270106091 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbee80b2d70 0x7fbee8154e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbeee800640 1 -- 172.21.15.27:0/1270106091 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbee806dad0 msgr2=0x7fbee815c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:42.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbeee800640 1 --2- 172.21.15.27:0/1270106091 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbee806dad0 0x7fbee815c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbeee800640 1 -- 172.21.15.27:0/1270106091 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee0009b30 con 0x7fbee815d0c0 2024-01-15T18:05:42.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbeede00640 1 --2- 172.21.15.27:0/1270106091 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbee806dad0 0x7fbee815c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:42.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbeee800640 1 --2- 172.21.15.27:0/1270106091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee815d0c0 0x7fbee8161dd0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbee00ab000 tx=0x7fbee0019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:42.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbee6000640 1 -- 172.21.15.27:0/1270106091 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee00b8470 con 0x7fbee815d0c0 2024-01-15T18:05:42.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbee6000640 1 -- 172.21.15.27:0/1270106091 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbee00b85d0 con 0x7fbee815d0c0 2024-01-15T18:05:42.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbee6000640 1 -- 172.21.15.27:0/1270106091 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee00b8940 con 0x7fbee815d0c0 2024-01-15T18:05:42.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbef0800640 1 -- 172.21.15.27:0/1270106091 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbee815eb80 con 0x7fbee815d0c0 2024-01-15T18:05:42.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbef0800640 1 -- 172.21.15.27:0/1270106091 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbee815e130 con 0x7fbee815d0c0 2024-01-15T18:05:42.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbef0800640 1 -- 172.21.15.27:0/1270106091 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbee804e850 con 0x7fbee815d0c0 2024-01-15T18:05:42.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.187+0000 7fbee6000640 1 -- 172.21.15.27:0/1270106091 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fbee00b8aa0 con 0x7fbee815d0c0 2024-01-15T18:05:42.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.191+0000 7fbee6000640 1 --2- 172.21.15.27:0/1270106091 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbec0066920 0x7fbec0068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:42.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.191+0000 7fbee6000640 1 -- 172.21.15.27:0/1270106091 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fbee011f150 con 0x7fbee815d0c0 2024-01-15T18:05:42.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.191+0000 7fbeede00640 1 --2- 172.21.15.27:0/1270106091 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbec0066920 0x7fbec0068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:42.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.191+0000 7fbeede00640 1 --2- 172.21.15.27:0/1270106091 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbec0066920 0x7fbec0068de0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fbed8007fd0 tx=0x7fbed8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:42.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.195+0000 7fbee6000640 1 -- 172.21.15.27:0/1270106091 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbee00ec4c0 con 0x7fbee815d0c0 2024-01-15T18:05:42.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.363+0000 7fbef0800640 1 -- 172.21.15.27:0/1270106091 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbee8071a80 con 0x7fbee815d0c0 2024-01-15T18:05:42.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.363+0000 7fbee6000640 1 -- 172.21.15.27:0/1270106091 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7fbee00efae0 con 0x7fbee815d0c0 2024-01-15T18:05:42.367 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-01-15T18:05:42.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.363+0000 7fbef0800640 1 -- 172.21.15.27:0/1270106091 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbec0066920 msgr2=0x7fbec0068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:42.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.363+0000 7fbef0800640 1 --2- 172.21.15.27:0/1270106091 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbec0066920 0x7fbec0068de0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fbed8007fd0 tx=0x7fbed8009040 comp rx=0 tx=0).stop 2024-01-15T18:05:42.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.363+0000 7fbef0800640 1 -- 172.21.15.27:0/1270106091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee815d0c0 msgr2=0x7fbee8161dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:42.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.363+0000 7fbef0800640 1 --2- 172.21.15.27:0/1270106091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee815d0c0 0x7fbee8161dd0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbee00ab000 tx=0x7fbee0019790 comp rx=0 tx=0).stop 2024-01-15T18:05:42.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.363+0000 7fbef0800640 1 -- 172.21.15.27:0/1270106091 shutdown_connections 2024-01-15T18:05:42.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.363+0000 7fbef0800640 1 --2- 172.21.15.27:0/1270106091 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbec0066920 0x7fbec0068de0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.363+0000 7fbef0800640 1 --2- 172.21.15.27:0/1270106091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbee815d0c0 0x7fbee8161dd0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.363+0000 7fbef0800640 1 --2- 172.21.15.27:0/1270106091 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbee80b2d70 0x7fbee8154e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.363+0000 7fbef0800640 1 --2- 172.21.15.27:0/1270106091 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbee806dad0 0x7fbee815c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.363+0000 7fbef0800640 1 -- 172.21.15.27:0/1270106091 >> 172.21.15.27:0/1270106091 conn(0x7fbee806b600 msgr2=0x7fbee806d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:42.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.363+0000 7fbef0800640 1 -- 172.21.15.27:0/1270106091 shutdown_connections 2024-01-15T18:05:42.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.363+0000 7fbef0800640 1 -- 172.21.15.27:0/1270106091 wait complete. 2024-01-15T18:05:42.389 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-01-15T18:05:42.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:42.389 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-01-15T18:05:42.410 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:05:42.410 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T18:05:42.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:42.410 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-15T18:05:42.463 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:05:42.463 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:05:42.472 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:05:42.472 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:05:42.522 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:42.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.659+0000 7f8dd8200640 1 Processor -- start 2024-01-15T18:05:42.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.659+0000 7f8dd8200640 1 -- start start 2024-01-15T18:05:42.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.659+0000 7f8dd8200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f8dd015e7d0 0x7f8dd00cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:42.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.659+0000 7f8dd8200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8dd00cedc0 0x7f8dd00cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:42.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.659+0000 7f8dd8200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd00b38b0 con 0x7f8dd00ce2e0 2024-01-15T18:05:42.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.659+0000 7f8dd8200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dd00cd550 con 0x7f8dd00cedc0 2024-01-15T18:05:42.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.659+0000 7f8dd8200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dd00cd270 con 0x7f8dd015e7d0 2024-01-15T18:05:42.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.659+0000 7f8dd4e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8dd00cedc0 0x7f8dd00cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:42.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.659+0000 7f8dd4e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8dd00cedc0 0x7f8dd00cf7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35206/0 (socket says 172.21.15.27:35206) 2024-01-15T18:05:42.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.659+0000 7f8dd4e00640 1 -- 172.21.15.27:0/4206454205 learned_addr learned my addr 172.21.15.27:0/4206454205 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:42.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.659+0000 7f8dd5800640 1 --2- 172.21.15.27:0/4206454205 >> v2:172.21.15.93:3300/0 conn(0x7f8dd015e7d0 0x7f8dd00cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:42.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.659+0000 7f8dd4e00640 1 -- 172.21.15.27:0/4206454205 >> v2:172.21.15.93:3300/0 conn(0x7f8dd015e7d0 msgr2=0x7f8dd00cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:42.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.659+0000 7f8dd4e00640 1 --2- 172.21.15.27:0/4206454205 >> v2:172.21.15.93:3300/0 conn(0x7f8dd015e7d0 0x7f8dd00cf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.659+0000 7f8dd4e00640 1 -- 172.21.15.27:0/4206454205 >> v1:172.21.15.93:6789/0 conn(0x7f8dd00ce2e0 legacy=0x7f8dd00cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:42.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.659+0000 7f8dd4e00640 1 -- 172.21.15.27:0/4206454205 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd00b35d0 con 0x7f8dd00cedc0 2024-01-15T18:05:42.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.663+0000 7f8dd4e00640 1 --2- 172.21.15.27:0/4206454205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd00cedc0 0x7f8dd00cf7e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0071b00 tx=0x7f8db801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd318d0d2b83cfb9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:42.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.663+0000 7f8dcfe00640 1 -- 172.21.15.27:0/4206454205 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db80ad660 con 0x7f8dd00cedc0 2024-01-15T18:05:42.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.663+0000 7f8dcfe00640 1 -- 172.21.15.27:0/4206454205 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db80ad7c0 con 0x7f8dd00cedc0 2024-01-15T18:05:42.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.663+0000 7f8dcfe00640 1 -- 172.21.15.27:0/4206454205 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db80adb50 con 0x7f8dd00cedc0 2024-01-15T18:05:42.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.663+0000 7f8dd8200640 1 -- 172.21.15.27:0/4206454205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd00cedc0 msgr2=0x7f8dd00cf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:42.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.663+0000 7f8dd8200640 1 --2- 172.21.15.27:0/4206454205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd00cedc0 0x7f8dd00cf7e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0071b00 tx=0x7f8db801c470 comp rx=0 tx=0).stop 2024-01-15T18:05:42.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.663+0000 7f8dd8200640 1 -- 172.21.15.27:0/4206454205 shutdown_connections 2024-01-15T18:05:42.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.663+0000 7f8dd8200640 1 --2- 172.21.15.27:0/4206454205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd00cedc0 0x7f8dd00cf7e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.663+0000 7f8dd8200640 1 --2- 172.21.15.27:0/4206454205 >> v2:172.21.15.93:3300/0 conn(0x7f8dd015e7d0 0x7f8dd00cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.663+0000 7f8dd8200640 1 -- 172.21.15.27:0/4206454205 >> 172.21.15.27:0/4206454205 conn(0x7f8dd00b2a10 msgr2=0x7f8dd00b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:42.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.663+0000 7f8dd8200640 1 -- 172.21.15.27:0/4206454205 shutdown_connections 2024-01-15T18:05:42.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.663+0000 7f8dd8200640 1 -- 172.21.15.27:0/4206454205 wait complete. 2024-01-15T18:05:42.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dd8200640 1 Processor -- start 2024-01-15T18:05:42.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dd8200640 1 -- start start 2024-01-15T18:05:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dd8200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8dd00ce2e0 0x7f8dd015be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dd8200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8dd00cedc0 0x7f8dd0155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dd8200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd015e7d0 0x7f8dd015da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dd8200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd0155e90 con 0x7f8dd015e7d0 2024-01-15T18:05:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dd8200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd015dfd0 con 0x7f8dd00ce2e0 2024-01-15T18:05:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dd8200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd015c480 con 0x7f8dd00cedc0 2024-01-15T18:05:42.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dd6200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd015e7d0 0x7f8dd015da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dd4e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8dd00cedc0 0x7f8dd0155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dd6200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd015e7d0 0x7f8dd015da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35216/0 (socket says 172.21.15.27:35216) 2024-01-15T18:05:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dd6200640 1 -- 172.21.15.27:0/3662416728 learned_addr learned my addr 172.21.15.27:0/3662416728 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dd5800640 1 --2- 172.21.15.27:0/3662416728 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8dd00ce2e0 0x7f8dd015be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dd6200640 1 -- 172.21.15.27:0/3662416728 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8dd00cedc0 msgr2=0x7f8dd0155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dd6200640 1 --2- 172.21.15.27:0/3662416728 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8dd00cedc0 0x7f8dd0155950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dd6200640 1 -- 172.21.15.27:0/3662416728 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8dd00ce2e0 msgr2=0x7f8dd015be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dd6200640 1 --2- 172.21.15.27:0/3662416728 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8dd00ce2e0 0x7f8dd015be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dd6200640 1 -- 172.21.15.27:0/3662416728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db80094d0 con 0x7f8dd015e7d0 2024-01-15T18:05:42.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dd5800640 1 --2- 172.21.15.27:0/3662416728 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8dd00ce2e0 0x7f8dd015be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:42.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dd4e00640 1 --2- 172.21.15.27:0/3662416728 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8dd00cedc0 0x7f8dd0155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:42.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dd6200640 1 --2- 172.21.15.27:0/3662416728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd015e7d0 0x7f8dd015da90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8dc800c730 tx=0x7f8dc800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:42.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dcd600640 1 -- 172.21.15.27:0/3662416728 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8003b80 con 0x7f8dd015e7d0 2024-01-15T18:05:42.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dcd600640 1 -- 172.21.15.27:0/3662416728 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc8003ce0 con 0x7f8dd015e7d0 2024-01-15T18:05:42.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dcd600640 1 -- 172.21.15.27:0/3662416728 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc800aa30 con 0x7f8dd015e7d0 2024-01-15T18:05:42.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dd8200640 1 -- 172.21.15.27:0/3662416728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd01590f0 con 0x7f8dd015e7d0 2024-01-15T18:05:42.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dd8200640 1 -- 172.21.15.27:0/3662416728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd01596b0 con 0x7f8dd015e7d0 2024-01-15T18:05:42.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.667+0000 7f8dd8200640 1 -- 172.21.15.27:0/3662416728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd01598a0 con 0x7f8dd015e7d0 2024-01-15T18:05:42.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.671+0000 7f8dcd600640 1 -- 172.21.15.27:0/3662416728 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f8dc800ab90 con 0x7f8dd015e7d0 2024-01-15T18:05:42.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.671+0000 7f8dcd600640 1 --2- 172.21.15.27:0/3662416728 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8da80668f0 0x7f8da8068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:42.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.671+0000 7f8dcd600640 1 -- 172.21.15.27:0/3662416728 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f8dc80840f0 con 0x7f8dd015e7d0 2024-01-15T18:05:42.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.671+0000 7f8dd5800640 1 --2- 172.21.15.27:0/3662416728 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8da80668f0 0x7f8da8068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.671+0000 7f8dd5800640 1 --2- 172.21.15.27:0/3662416728 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8da80668f0 0x7f8da8068db0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0007e50 tx=0x7f8dc0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:42.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.675+0000 7f8dcd600640 1 -- 172.21.15.27:0/3662416728 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8dc8051340 con 0x7f8dd015e7d0 2024-01-15T18:05:42.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.831+0000 7f8dd8200640 1 -- 172.21.15.27:0/3662416728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8dd015c5c0 con 0x7f8dd015e7d0 2024-01-15T18:05:42.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.831+0000 7f8dcd600640 1 -- 172.21.15.27:0/3662416728 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16705 (secure 0 0 0) 0x7f8dc8054960 con 0x7f8dd015e7d0 2024-01-15T18:05:42.838 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:42.838 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":23,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:05:40.008536+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:04:56.220608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T18:04:57.210421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:42.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.835+0000 7f8dd8200640 1 -- 172.21.15.27:0/3662416728 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8da80668f0 msgr2=0x7f8da8068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:42.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.835+0000 7f8dd8200640 1 --2- 172.21.15.27:0/3662416728 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8da80668f0 0x7f8da8068db0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0007e50 tx=0x7f8dc0008040 comp rx=0 tx=0).stop 2024-01-15T18:05:42.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.835+0000 7f8dd8200640 1 -- 172.21.15.27:0/3662416728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd015e7d0 msgr2=0x7f8dd015da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:42.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.835+0000 7f8dd8200640 1 --2- 172.21.15.27:0/3662416728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd015e7d0 0x7f8dd015da90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8dc800c730 tx=0x7f8dc800cbf0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.835+0000 7f8dd8200640 1 -- 172.21.15.27:0/3662416728 shutdown_connections 2024-01-15T18:05:42.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.835+0000 7f8dd8200640 1 --2- 172.21.15.27:0/3662416728 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8da80668f0 0x7f8da8068db0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.835+0000 7f8dd8200640 1 --2- 172.21.15.27:0/3662416728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8dd015e7d0 0x7f8dd015da90 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.835+0000 7f8dd8200640 1 --2- 172.21.15.27:0/3662416728 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8dd00cedc0 0x7f8dd0155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.835+0000 7f8dd8200640 1 --2- 172.21.15.27:0/3662416728 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8dd00ce2e0 0x7f8dd015be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.835+0000 7f8dd8200640 1 -- 172.21.15.27:0/3662416728 >> 172.21.15.27:0/3662416728 conn(0x7f8dd00b2a10 msgr2=0x7f8dd015ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:42.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.835+0000 7f8dd8200640 1 -- 172.21.15.27:0/3662416728 shutdown_connections 2024-01-15T18:05:42.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.835+0000 7f8dd8200640 1 -- 172.21.15.27:0/3662416728 wait complete. 2024-01-15T18:05:42.860 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T18:04:56.220608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T18:04:57.210421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:05:42.860 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:05:43.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.999+0000 7f80c0e00640 1 Processor -- start 2024-01-15T18:05:43.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:42.999+0000 7f80c0e00640 1 -- start start 2024-01-15T18:05:43.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.003+0000 7f80c0e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f80bc1557e0 0x7f80bc0acd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:43.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.003+0000 7f80c0e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80bc0ad410 con 0x7f80bc06ac00 2024-01-15T18:05:43.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.003+0000 7f80c0e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80bc0b2d00 con 0x7f80bc0b3d10 2024-01-15T18:05:43.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.003+0000 7f80c0e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80bc0ae360 con 0x7f80bc1557e0 2024-01-15T18:05:43.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.003+0000 7f80ba800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f80bc06ac00 0x7f80bc0ae820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35148/0 (socket says 172.21.15.27:35148) 2024-01-15T18:05:43.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.003+0000 7f80ba800640 1 -- 172.21.15.27:0/1946612653 learned_addr learned my addr 172.21.15.27:0/1946612653 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:43.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.003+0000 7f80b9e00640 1 --2- 172.21.15.27:0/1946612653 >> v2:172.21.15.93:3300/0 conn(0x7f80bc1557e0 0x7f80bc0acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:43.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.003+0000 7f80b8a00640 1 -- 172.21.15.27:0/1946612653 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 448459276 0 0) 0x7f80bc0ad410 con 0x7f80bc06ac00 2024-01-15T18:05:43.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.003+0000 7f80b8a00640 1 -- 172.21.15.27:0/1946612653 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f809c003530 con 0x7f80bc06ac00 2024-01-15T18:05:43.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.003+0000 7f80b9e00640 1 -- 172.21.15.27:0/1946612653 >> v1:172.21.15.93:6789/0 conn(0x7f80bc0b3d10 legacy=0x7f80bc06a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.003+0000 7f80b9e00640 1 -- 172.21.15.27:0/1946612653 >> v1:172.21.15.27:6789/0 conn(0x7f80bc06ac00 legacy=0x7f80bc0ae820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.003+0000 7f80b9e00640 1 -- 172.21.15.27:0/1946612653 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80bc0b3880 con 0x7f80bc1557e0 2024-01-15T18:05:43.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.003+0000 7f80b9e00640 1 --2- 172.21.15.27:0/1946612653 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f80bc1557e0 0x7f80bc0acd50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f80ac0097c0 tx=0x7f80ac01c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f65d814f10369cae server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:43.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.007+0000 7f80b8a00640 1 -- 172.21.15.27:0/1946612653 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac0ab650 con 0x7f80bc1557e0 2024-01-15T18:05:43.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.007+0000 7f80b8a00640 1 -- 172.21.15.27:0/1946612653 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ac0ab7b0 con 0x7f80bc1557e0 2024-01-15T18:05:43.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.007+0000 7f80b8a00640 1 -- 172.21.15.27:0/1946612653 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac0abb40 con 0x7f80bc1557e0 2024-01-15T18:05:43.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.007+0000 7f80c0e00640 1 -- 172.21.15.27:0/1946612653 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f80bc1557e0 msgr2=0x7f80bc0acd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.007+0000 7f80c0e00640 1 --2- 172.21.15.27:0/1946612653 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f80bc1557e0 0x7f80bc0acd50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f80ac0097c0 tx=0x7f80ac01c450 comp rx=0 tx=0).stop 2024-01-15T18:05:43.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.007+0000 7f80c0e00640 1 -- 172.21.15.27:0/1946612653 shutdown_connections 2024-01-15T18:05:43.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.007+0000 7f80c0e00640 1 --2- 172.21.15.27:0/1946612653 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f80bc1557e0 0x7f80bc0acd50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.007+0000 7f80c0e00640 1 -- 172.21.15.27:0/1946612653 >> 172.21.15.27:0/1946612653 conn(0x7f80bc06b600 msgr2=0x7f80bc06ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:43.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.007+0000 7f80c0e00640 1 -- 172.21.15.27:0/1946612653 shutdown_connections 2024-01-15T18:05:43.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.007+0000 7f80c0e00640 1 -- 172.21.15.27:0/1946612653 wait complete. 2024-01-15T18:05:43.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.007+0000 7f80c0e00640 1 Processor -- start 2024-01-15T18:05:43.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.007+0000 7f80c0e00640 1 -- start start 2024-01-15T18:05:43.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.011+0000 7f80c0e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80bc06ac00 0x7f80bc1582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:43.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.011+0000 7f80c0e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f80bc0b3d10 0x7f80bc155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:43.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.011+0000 7f80c0e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f80bc1557e0 0x7f80bc1526a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:43.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.011+0000 7f80c0e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80bc153f10 con 0x7f80bc06ac00 2024-01-15T18:05:43.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.011+0000 7f80c0e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80bc151600 con 0x7f80bc1557e0 2024-01-15T18:05:43.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.011+0000 7f80c0e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80bc1518e0 con 0x7f80bc0b3d10 2024-01-15T18:05:43.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.011+0000 7f80b9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80bc06ac00 0x7f80bc1582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:43.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.011+0000 7f80b9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80bc06ac00 0x7f80bc1582a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35218/0 (socket says 172.21.15.27:35218) 2024-01-15T18:05:43.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.011+0000 7f80b9e00640 1 -- 172.21.15.27:0/650559110 learned_addr learned my addr 172.21.15.27:0/650559110 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:43.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.011+0000 7f80ba800640 1 --2- 172.21.15.27:0/650559110 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f80bc1557e0 0x7f80bc1526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:43.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.011+0000 7f80b9400640 1 --2- 172.21.15.27:0/650559110 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f80bc0b3d10 0x7f80bc155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:43.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.011+0000 7f80b9e00640 1 -- 172.21.15.27:0/650559110 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f80bc0b3d10 msgr2=0x7f80bc155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.011+0000 7f80b9e00640 1 --2- 172.21.15.27:0/650559110 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f80bc0b3d10 0x7f80bc155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.011+0000 7f80b9e00640 1 -- 172.21.15.27:0/650559110 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f80bc1557e0 msgr2=0x7f80bc1526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.011+0000 7f80b9e00640 1 --2- 172.21.15.27:0/650559110 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f80bc1557e0 0x7f80bc1526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.011+0000 7f80b9e00640 1 -- 172.21.15.27:0/650559110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ac009420 con 0x7f80bc06ac00 2024-01-15T18:05:43.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.011+0000 7f80ba800640 1 --2- 172.21.15.27:0/650559110 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f80bc1557e0 0x7f80bc1526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:43.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.011+0000 7f80b9e00640 1 --2- 172.21.15.27:0/650559110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80bc06ac00 0x7f80bc1582a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f80ac0ad040 tx=0x7f80ac01ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:43.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.011+0000 7f80b2000640 1 -- 172.21.15.27:0/650559110 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac0bf610 con 0x7f80bc06ac00 2024-01-15T18:05:43.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.011+0000 7f80b2000640 1 -- 172.21.15.27:0/650559110 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ac0bf770 con 0x7f80bc06ac00 2024-01-15T18:05:43.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.011+0000 7f80c0e00640 1 -- 172.21.15.27:0/650559110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80bc151bc0 con 0x7f80bc06ac00 2024-01-15T18:05:43.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.011+0000 7f80b2000640 1 -- 172.21.15.27:0/650559110 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac0bfb20 con 0x7f80bc06ac00 2024-01-15T18:05:43.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.011+0000 7f80c0e00640 1 -- 172.21.15.27:0/650559110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80bc152180 con 0x7f80bc06ac00 2024-01-15T18:05:43.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.011+0000 7f80c0e00640 1 -- 172.21.15.27:0/650559110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80bc152370 con 0x7f80bc06ac00 2024-01-15T18:05:43.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.011+0000 7f80b2000640 1 -- 172.21.15.27:0/650559110 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f80ac0bfc80 con 0x7f80bc06ac00 2024-01-15T18:05:43.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.015+0000 7f80b2000640 1 --2- 172.21.15.27:0/650559110 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f80940668f0 0x7f8094068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:43.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.015+0000 7f80b2000640 1 -- 172.21.15.27:0/650559110 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f80ac11f350 con 0x7f80bc06ac00 2024-01-15T18:05:43.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.015+0000 7f80b9400640 1 --2- 172.21.15.27:0/650559110 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f80940668f0 0x7f8094068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:43.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.015+0000 7f80b9400640 1 --2- 172.21.15.27:0/650559110 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f80940668f0 0x7f8094068db0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f80bc151510 tx=0x7f80a4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:43.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.019+0000 7f80b2000640 1 -- 172.21.15.27:0/650559110 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f80ac0ec4f0 con 0x7f80bc06ac00 2024-01-15T18:05:43.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.175+0000 7f80c0e00640 1 -- 172.21.15.27:0/650559110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f80bc04e850 con 0x7f80bc06ac00 2024-01-15T18:05:43.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.179+0000 7f80b2000640 1 -- 172.21.15.27:0/650559110 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f80ac0efb10 con 0x7f80bc06ac00 2024-01-15T18:05:43.183 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:05:43.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.179+0000 7f80c0e00640 1 -- 172.21.15.27:0/650559110 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f80940668f0 msgr2=0x7f8094068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.179+0000 7f80c0e00640 1 --2- 172.21.15.27:0/650559110 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f80940668f0 0x7f8094068db0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f80bc151510 tx=0x7f80a4009040 comp rx=0 tx=0).stop 2024-01-15T18:05:43.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.179+0000 7f80c0e00640 1 -- 172.21.15.27:0/650559110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80bc06ac00 msgr2=0x7f80bc1582a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.179+0000 7f80c0e00640 1 --2- 172.21.15.27:0/650559110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80bc06ac00 0x7f80bc1582a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f80ac0ad040 tx=0x7f80ac01ac90 comp rx=0 tx=0).stop 2024-01-15T18:05:43.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.179+0000 7f80c0e00640 1 -- 172.21.15.27:0/650559110 shutdown_connections 2024-01-15T18:05:43.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.179+0000 7f80c0e00640 1 --2- 172.21.15.27:0/650559110 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f80940668f0 0x7f8094068db0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.179+0000 7f80c0e00640 1 --2- 172.21.15.27:0/650559110 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f80bc1557e0 0x7f80bc1526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.179+0000 7f80c0e00640 1 --2- 172.21.15.27:0/650559110 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f80bc0b3d10 0x7f80bc155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.179+0000 7f80c0e00640 1 --2- 172.21.15.27:0/650559110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80bc06ac00 0x7f80bc1582a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.179+0000 7f80c0e00640 1 -- 172.21.15.27:0/650559110 >> 172.21.15.27:0/650559110 conn(0x7f80bc06b600 msgr2=0x7f80bc06e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:43.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.179+0000 7f80c0e00640 1 -- 172.21.15.27:0/650559110 shutdown_connections 2024-01-15T18:05:43.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.179+0000 7f80c0e00640 1 -- 172.21.15.27:0/650559110 wait complete. 2024-01-15T18:05:43.204 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T18:05:43.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.343+0000 7f1c18400640 1 Processor -- start 2024-01-15T18:05:43.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.343+0000 7f1c18400640 1 -- start start 2024-01-15T18:05:43.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.343+0000 7f1c18400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1c1006ac00 0x7f1c10161120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:43.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.343+0000 7f1c18400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c100cde30 con 0x7f1c10160cb0 2024-01-15T18:05:43.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.343+0000 7f1c18400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c1006fa50 con 0x7f1c10146a60 2024-01-15T18:05:43.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.343+0000 7f1c18400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c10161820 con 0x7f1c1006ac00 2024-01-15T18:05:43.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.343+0000 7f1c16400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1c1006ac00 0x7f1c10161120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:43.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.343+0000 7f1c15000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1c10160cb0 0x7f1c1006a4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35164/0 (socket says 172.21.15.27:35164) 2024-01-15T18:05:43.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.343+0000 7f1c15000640 1 -- 172.21.15.27:0/3742627323 learned_addr learned my addr 172.21.15.27:0/3742627323 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:43.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.343+0000 7f1c16400640 1 -- 172.21.15.27:0/3742627323 >> v1:172.21.15.93:6789/0 conn(0x7f1c10146a60 legacy=0x7f1c100ac880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.343+0000 7f1c16400640 1 -- 172.21.15.27:0/3742627323 >> v1:172.21.15.27:6789/0 conn(0x7f1c10160cb0 legacy=0x7f1c1006a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.343+0000 7f1c16400640 1 -- 172.21.15.27:0/3742627323 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c10161b00 con 0x7f1c1006ac00 2024-01-15T18:05:43.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.343+0000 7f1c0fe00640 1 -- 172.21.15.27:0/3742627323 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206053492 0 0) 0x7f1c100cde30 con 0x7f1c10160cb0 2024-01-15T18:05:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.347+0000 7f1c16400640 1 --2- 172.21.15.27:0/3742627323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1006ac00 0x7f1c10161120 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1c08009d90 tx=0x7f1c0801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b76ba36719c778c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.347+0000 7f1c0fe00640 1 -- 172.21.15.27:0/3742627323 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c080a5a10 con 0x7f1c1006ac00 2024-01-15T18:05:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.347+0000 7f1c0fe00640 1 -- 172.21.15.27:0/3742627323 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c080a5b70 con 0x7f1c1006ac00 2024-01-15T18:05:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.347+0000 7f1c0fe00640 1 -- 172.21.15.27:0/3742627323 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c080ad440 con 0x7f1c1006ac00 2024-01-15T18:05:43.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.347+0000 7f1c18400640 1 -- 172.21.15.27:0/3742627323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1006ac00 msgr2=0x7f1c10161120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.347+0000 7f1c18400640 1 --2- 172.21.15.27:0/3742627323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1006ac00 0x7f1c10161120 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1c08009d90 tx=0x7f1c0801c950 comp rx=0 tx=0).stop 2024-01-15T18:05:43.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.347+0000 7f1c18400640 1 -- 172.21.15.27:0/3742627323 shutdown_connections 2024-01-15T18:05:43.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.347+0000 7f1c18400640 1 --2- 172.21.15.27:0/3742627323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1006ac00 0x7f1c10161120 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.347+0000 7f1c18400640 1 -- 172.21.15.27:0/3742627323 >> 172.21.15.27:0/3742627323 conn(0x7f1c100cce10 msgr2=0x7f1c100b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.347+0000 7f1c18400640 1 -- 172.21.15.27:0/3742627323 shutdown_connections 2024-01-15T18:05:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.347+0000 7f1c18400640 1 -- 172.21.15.27:0/3742627323 wait complete. 2024-01-15T18:05:43.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.351+0000 7f1c18400640 1 Processor -- start 2024-01-15T18:05:43.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.351+0000 7f1c18400640 1 -- start start 2024-01-15T18:05:43.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.351+0000 7f1c18400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c1006ac00 0x7f1c1013ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:43.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.351+0000 7f1c18400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1c10146a60 0x7f1c1013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:43.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.351+0000 7f1c18400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c10160cb0 0x7f1c1013f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:43.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.351+0000 7f1c18400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c100b18c0 con 0x7f1c10160cb0 2024-01-15T18:05:43.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.351+0000 7f1c18400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c10161e00 con 0x7f1c10146a60 2024-01-15T18:05:43.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.351+0000 7f1c18400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c10143260 con 0x7f1c1006ac00 2024-01-15T18:05:43.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.351+0000 7f1c16400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c10160cb0 0x7f1c1013f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:43.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.351+0000 7f1c16400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c10160cb0 0x7f1c1013f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35240/0 (socket says 172.21.15.27:35240) 2024-01-15T18:05:43.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.351+0000 7f1c16400640 1 -- 172.21.15.27:0/2066120736 learned_addr learned my addr 172.21.15.27:0/2066120736 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:43.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.351+0000 7f1c15000640 1 --2- 172.21.15.27:0/2066120736 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1c10146a60 0x7f1c1013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:43.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.351+0000 7f1c15a00640 1 --2- 172.21.15.27:0/2066120736 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c1006ac00 0x7f1c1013ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:43.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.351+0000 7f1c16400640 1 -- 172.21.15.27:0/2066120736 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c1006ac00 msgr2=0x7f1c1013ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.351+0000 7f1c16400640 1 --2- 172.21.15.27:0/2066120736 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c1006ac00 0x7f1c1013ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.351+0000 7f1c16400640 1 -- 172.21.15.27:0/2066120736 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1c10146a60 msgr2=0x7f1c1013f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.351+0000 7f1c16400640 1 --2- 172.21.15.27:0/2066120736 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1c10146a60 0x7f1c1013f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.351+0000 7f1c16400640 1 -- 172.21.15.27:0/2066120736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c08009a70 con 0x7f1c10160cb0 2024-01-15T18:05:43.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.351+0000 7f1c16400640 1 --2- 172.21.15.27:0/2066120736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c10160cb0 0x7f1c1013f810 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1c08019560 tx=0x7f1c080196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:43.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.351+0000 7f1c0d600640 1 -- 172.21.15.27:0/2066120736 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c080adea0 con 0x7f1c10160cb0 2024-01-15T18:05:43.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.351+0000 7f1c0d600640 1 -- 172.21.15.27:0/2066120736 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c080039a0 con 0x7f1c10160cb0 2024-01-15T18:05:43.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.351+0000 7f1c0d600640 1 -- 172.21.15.27:0/2066120736 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c08003d50 con 0x7f1c10160cb0 2024-01-15T18:05:43.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.351+0000 7f1c18400640 1 -- 172.21.15.27:0/2066120736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c1013ef90 con 0x7f1c10160cb0 2024-01-15T18:05:43.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.351+0000 7f1c18400640 1 -- 172.21.15.27:0/2066120736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c100ad440 con 0x7f1c10160cb0 2024-01-15T18:05:43.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.355+0000 7f1c18400640 1 -- 172.21.15.27:0/2066120736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c1004e850 con 0x7f1c10160cb0 2024-01-15T18:05:43.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.355+0000 7f1c0d600640 1 -- 172.21.15.27:0/2066120736 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f1c08004030 con 0x7f1c10160cb0 2024-01-15T18:05:43.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.355+0000 7f1c0d600640 1 --2- 172.21.15.27:0/2066120736 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1be80668f0 0x7f1be8068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:43.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.355+0000 7f1c0d600640 1 -- 172.21.15.27:0/2066120736 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f1c0811fa60 con 0x7f1c10160cb0 2024-01-15T18:05:43.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.355+0000 7f1c15a00640 1 --2- 172.21.15.27:0/2066120736 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1be80668f0 0x7f1be8068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:43.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.355+0000 7f1c15a00640 1 --2- 172.21.15.27:0/2066120736 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1be80668f0 0x7f1be8068db0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1c00000f70 tx=0x7f1c00009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:43.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.359+0000 7f1c0d600640 1 -- 172.21.15.27:0/2066120736 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1c080ecc30 con 0x7f1c10160cb0 2024-01-15T18:05:43.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.515+0000 7f1c18400640 1 -- 172.21.15.27:0/2066120736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1c10067620 con 0x7f1c10160cb0 2024-01-15T18:05:43.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.515+0000 7f1c0d600640 1 -- 172.21.15.27:0/2066120736 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1c080b2020 con 0x7f1c10160cb0 2024-01-15T18:05:43.522 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-15T18:05:43.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.519+0000 7f1c18400640 1 -- 172.21.15.27:0/2066120736 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1be80668f0 msgr2=0x7f1be8068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.519+0000 7f1c18400640 1 --2- 172.21.15.27:0/2066120736 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1be80668f0 0x7f1be8068db0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1c00000f70 tx=0x7f1c00009040 comp rx=0 tx=0).stop 2024-01-15T18:05:43.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.519+0000 7f1c18400640 1 -- 172.21.15.27:0/2066120736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c10160cb0 msgr2=0x7f1c1013f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.519+0000 7f1c18400640 1 --2- 172.21.15.27:0/2066120736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c10160cb0 0x7f1c1013f810 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1c08019560 tx=0x7f1c080196f0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.519+0000 7f1c18400640 1 -- 172.21.15.27:0/2066120736 shutdown_connections 2024-01-15T18:05:43.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.519+0000 7f1c18400640 1 --2- 172.21.15.27:0/2066120736 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1be80668f0 0x7f1be8068db0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.519+0000 7f1c18400640 1 --2- 172.21.15.27:0/2066120736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c10160cb0 0x7f1c1013f810 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.519+0000 7f1c18400640 1 --2- 172.21.15.27:0/2066120736 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1c10146a60 0x7f1c1013f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.519+0000 7f1c18400640 1 --2- 172.21.15.27:0/2066120736 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c1006ac00 0x7f1c1013ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.519+0000 7f1c18400640 1 -- 172.21.15.27:0/2066120736 >> 172.21.15.27:0/2066120736 conn(0x7f1c100cce10 msgr2=0x7f1c10144b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:43.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.519+0000 7f1c18400640 1 -- 172.21.15.27:0/2066120736 shutdown_connections 2024-01-15T18:05:43.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.519+0000 7f1c18400640 1 -- 172.21.15.27:0/2066120736 wait complete. 2024-01-15T18:05:43.544 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T18:05:43.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc2a8200640 1 Processor -- start 2024-01-15T18:05:43.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc2a8200640 1 -- start start 2024-01-15T18:05:43.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc2a8200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fc2a006c740 0x7fc2a0099a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:43.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc2a8200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2a0099f50 0x7fc2a00cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:43.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc2a8200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a015d860 con 0x7fc2a015d480 2024-01-15T18:05:43.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc2a8200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a009a3e0 con 0x7fc2a006c740 2024-01-15T18:05:43.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc2a8200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a006b6a0 con 0x7fc2a0099f50 2024-01-15T18:05:43.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc2a5800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc2a015d480 0x7fc2a00cfbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41820/0 (socket says 172.21.15.27:41820) 2024-01-15T18:05:43.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc2a5800640 1 -- 172.21.15.27:0/1630514625 learned_addr learned my addr 172.21.15.27:0/1630514625 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:43.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc2a6200640 1 --2- 172.21.15.27:0/1630514625 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2a0099f50 0x7fc2a00cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:43.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc2a4e00640 1 --2- 172.21.15.27:0/1630514625 >> v2:172.21.15.93:3300/0 conn(0x7fc2a006c740 0x7fc2a0099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:43.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc29fe00640 1 -- 172.21.15.27:0/1630514625 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177311576 0 0) 0x7fc2a015d860 con 0x7fc2a015d480 2024-01-15T18:05:43.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc29fe00640 1 -- 172.21.15.27:0/1630514625 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc280003530 con 0x7fc2a015d480 2024-01-15T18:05:43.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc29fe00640 1 -- 172.21.15.27:0/1630514625 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3411003581 0 0) 0x7fc280003530 con 0x7fc2a015d480 2024-01-15T18:05:43.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc29fe00640 1 -- 172.21.15.27:0/1630514625 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2a0099f50 msgr2=0x7fc2a00cb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:43.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc29fe00640 1 --2- 172.21.15.27:0/1630514625 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2a0099f50 0x7fc2a00cb5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc29fe00640 1 -- 172.21.15.27:0/1630514625 >> v2:172.21.15.93:3300/0 conn(0x7fc2a006c740 msgr2=0x7fc2a0099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:43.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc29fe00640 1 --2- 172.21.15.27:0/1630514625 >> v2:172.21.15.93:3300/0 conn(0x7fc2a006c740 0x7fc2a0099a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc29fe00640 1 -- 172.21.15.27:0/1630514625 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a006b980 con 0x7fc2a015d480 2024-01-15T18:05:43.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc29fe00640 1 -- 172.21.15.27:0/1630514625 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fc290001680 con 0x7fc2a015d480 2024-01-15T18:05:43.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc29fe00640 1 -- 172.21.15.27:0/1630514625 >> v1:172.21.15.27:6789/0 conn(0x7fc2a015d480 legacy=0x7fc2a00cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc29fe00640 1 --2- 172.21.15.27:0/1630514625 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2a0099f50 0x7fc2800039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:43.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc29fe00640 1 --2- 172.21.15.27:0/1630514625 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc280003f70 0x7fc2800047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:43.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc29fe00640 1 --2- 172.21.15.27:0/1630514625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc280004d10 0x7fc280005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:43.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc29fe00640 1 -- 172.21.15.27:0/1630514625 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a009a3e0 con 0x7fc280004d10 2024-01-15T18:05:43.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc29fe00640 1 -- 172.21.15.27:0/1630514625 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a006b6a0 con 0x7fc280003f70 2024-01-15T18:05:43.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc29fe00640 1 -- 172.21.15.27:0/1630514625 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc28000b9c0 con 0x7fc2a0099f50 2024-01-15T18:05:43.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc2a6200640 1 --2- 172.21.15.27:0/1630514625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc280004d10 0x7fc280005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:43.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc2a5800640 1 --2- 172.21.15.27:0/1630514625 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2a0099f50 0x7fc2800039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:43.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.683+0000 7fc2a4e00640 1 --2- 172.21.15.27:0/1630514625 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc280003f70 0x7fc2800047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:43.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.687+0000 7fc2a6200640 1 -- 172.21.15.27:0/1630514625 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2a0099f50 msgr2=0x7fc2800039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:43.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.687+0000 7fc2a6200640 1 --2- 172.21.15.27:0/1630514625 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2a0099f50 0x7fc2800039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.687+0000 7fc2a6200640 1 -- 172.21.15.27:0/1630514625 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc280003f70 msgr2=0x7fc2800047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:43.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.687+0000 7fc2a6200640 1 --2- 172.21.15.27:0/1630514625 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc280003f70 0x7fc2800047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.687+0000 7fc2a6200640 1 -- 172.21.15.27:0/1630514625 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc280003530 con 0x7fc280004d10 2024-01-15T18:05:43.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.687+0000 7fc2a6200640 1 --2- 172.21.15.27:0/1630514625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc280004d10 0x7fc280005190 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc298009fc0 tx=0x7fc29801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28b65ac59529faa6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:43.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.687+0000 7fc29fe00640 1 -- 172.21.15.27:0/1630514625 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2980a5c70 con 0x7fc280004d10 2024-01-15T18:05:43.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.687+0000 7fc29fe00640 1 -- 172.21.15.27:0/1630514625 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc29800f040 con 0x7fc280004d10 2024-01-15T18:05:43.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.687+0000 7fc29fe00640 1 -- 172.21.15.27:0/1630514625 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2980ad440 con 0x7fc280004d10 2024-01-15T18:05:43.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.691+0000 7fc2a8200640 1 -- 172.21.15.27:0/1630514625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc280004d10 msgr2=0x7fc280005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.691+0000 7fc2a8200640 1 --2- 172.21.15.27:0/1630514625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc280004d10 0x7fc280005190 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc298009fc0 tx=0x7fc29801cc60 comp rx=0 tx=0).stop 2024-01-15T18:05:43.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.691+0000 7fc2a6200640 1 -- 172.21.15.27:0/1630514625 reap_dead start 2024-01-15T18:05:43.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.691+0000 7fc2a8200640 1 -- 172.21.15.27:0/1630514625 shutdown_connections 2024-01-15T18:05:43.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.691+0000 7fc2a8200640 1 -- 172.21.15.27:0/1630514625 >> 172.21.15.27:0/1630514625 conn(0x7fc2a00b2a80 msgr2=0x7fc2a00b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:43.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.691+0000 7fc2a8200640 1 -- 172.21.15.27:0/1630514625 shutdown_connections 2024-01-15T18:05:43.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.691+0000 7fc2a8200640 1 -- 172.21.15.27:0/1630514625 wait complete. 2024-01-15T18:05:43.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.691+0000 7fc2a8200640 1 Processor -- start 2024-01-15T18:05:43.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.691+0000 7fc2a8200640 1 -- start start 2024-01-15T18:05:43.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.691+0000 7fc2a8200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc280004d10 0x7fc2a00aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:43.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.691+0000 7fc2a8200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2a014e520 0x7fc2a00b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:43.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.691+0000 7fc2a8200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2a00b64e0 0x7fc2a009da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:43.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.691+0000 7fc2a8200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a00b6960 con 0x7fc280004d10 2024-01-15T18:05:43.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.691+0000 7fc2a8200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a00ae7c0 con 0x7fc2a00b64e0 2024-01-15T18:05:43.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.691+0000 7fc2a8200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a00b16b0 con 0x7fc2a014e520 2024-01-15T18:05:43.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.691+0000 7fc2a5800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc280004d10 0x7fc2a00aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:43.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.691+0000 7fc2a5800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc280004d10 0x7fc2a00aec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38764/0 (socket says 172.21.15.27:38764) 2024-01-15T18:05:43.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.691+0000 7fc2a5800640 1 -- 172.21.15.27:0/2647329688 learned_addr learned my addr 172.21.15.27:0/2647329688 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:43.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.691+0000 7fc2a4e00640 1 --2- 172.21.15.27:0/2647329688 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2a014e520 0x7fc2a00b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:43.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.691+0000 7fc2a6200640 1 --2- 172.21.15.27:0/2647329688 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2a00b64e0 0x7fc2a009da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:43.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.691+0000 7fc2a5800640 1 -- 172.21.15.27:0/2647329688 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2a014e520 msgr2=0x7fc2a00b5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.691+0000 7fc2a5800640 1 --2- 172.21.15.27:0/2647329688 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2a014e520 0x7fc2a00b5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.691+0000 7fc2a5800640 1 -- 172.21.15.27:0/2647329688 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2a00b64e0 msgr2=0x7fc2a009da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.691+0000 7fc2a5800640 1 --2- 172.21.15.27:0/2647329688 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2a00b64e0 0x7fc2a009da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.691+0000 7fc2a5800640 1 -- 172.21.15.27:0/2647329688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a009e620 con 0x7fc280004d10 2024-01-15T18:05:43.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.695+0000 7fc2a5800640 1 --2- 172.21.15.27:0/2647329688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc280004d10 0x7fc2a00aec50 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc2900072c0 tx=0x7fc2900076c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:43.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.695+0000 7fc29d600640 1 -- 172.21.15.27:0/2647329688 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc290001450 con 0x7fc280004d10 2024-01-15T18:05:43.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.695+0000 7fc29d600640 1 -- 172.21.15.27:0/2647329688 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc29000f940 con 0x7fc280004d10 2024-01-15T18:05:43.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.695+0000 7fc29d600640 1 -- 172.21.15.27:0/2647329688 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29000fcf0 con 0x7fc280004d10 2024-01-15T18:05:43.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.695+0000 7fc2a8200640 1 -- 172.21.15.27:0/2647329688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a00ade90 con 0x7fc280004d10 2024-01-15T18:05:43.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.695+0000 7fc2a8200640 1 -- 172.21.15.27:0/2647329688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a00ae450 con 0x7fc280004d10 2024-01-15T18:05:43.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.695+0000 7fc2a8200640 1 -- 172.21.15.27:0/2647329688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a00ae5e0 con 0x7fc280004d10 2024-01-15T18:05:43.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.695+0000 7fc29d600640 1 -- 172.21.15.27:0/2647329688 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fc29001d460 con 0x7fc280004d10 2024-01-15T18:05:43.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.695+0000 7fc29d600640 1 --2- 172.21.15.27:0/2647329688 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc2780668f0 0x7fc278068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:43.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.695+0000 7fc29d600640 1 -- 172.21.15.27:0/2647329688 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fc290084620 con 0x7fc280004d10 2024-01-15T18:05:43.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.695+0000 7fc2a4e00640 1 --2- 172.21.15.27:0/2647329688 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc2780668f0 0x7fc278068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:43.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.699+0000 7fc2a4e00640 1 --2- 172.21.15.27:0/2647329688 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc2780668f0 0x7fc278068db0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0067930 tx=0x7fc288009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:43.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.699+0000 7fc29d600640 1 -- 172.21.15.27:0/2647329688 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc2900517c0 con 0x7fc280004d10 2024-01-15T18:05:43.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.859+0000 7fc2a8200640 1 -- 172.21.15.27:0/2647329688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc2a004e850 con 0x7fc280004d10 2024-01-15T18:05:43.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.859+0000 7fc29d600640 1 -- 172.21.15.27:0/2647329688 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc290054de0 con 0x7fc280004d10 2024-01-15T18:05:43.867 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-15T18:05:43.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.863+0000 7fc2a8200640 1 -- 172.21.15.27:0/2647329688 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc2780668f0 msgr2=0x7fc278068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.863+0000 7fc2a8200640 1 --2- 172.21.15.27:0/2647329688 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc2780668f0 0x7fc278068db0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0067930 tx=0x7fc288009190 comp rx=0 tx=0).stop 2024-01-15T18:05:43.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.863+0000 7fc2a8200640 1 -- 172.21.15.27:0/2647329688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc280004d10 msgr2=0x7fc2a00aec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.863+0000 7fc2a8200640 1 --2- 172.21.15.27:0/2647329688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc280004d10 0x7fc2a00aec50 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc2900072c0 tx=0x7fc2900076c0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.863+0000 7fc2a8200640 1 -- 172.21.15.27:0/2647329688 shutdown_connections 2024-01-15T18:05:43.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.863+0000 7fc2a8200640 1 --2- 172.21.15.27:0/2647329688 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc2780668f0 0x7fc278068db0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.863+0000 7fc2a8200640 1 --2- 172.21.15.27:0/2647329688 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2a00b64e0 0x7fc2a009da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.863+0000 7fc2a8200640 1 --2- 172.21.15.27:0/2647329688 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2a014e520 0x7fc2a00b5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.863+0000 7fc2a8200640 1 --2- 172.21.15.27:0/2647329688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc280004d10 0x7fc2a00aec50 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.863+0000 7fc2a8200640 1 -- 172.21.15.27:0/2647329688 >> 172.21.15.27:0/2647329688 conn(0x7fc2a00b2a80 msgr2=0x7fc2a00b3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:43.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.863+0000 7fc2a8200640 1 -- 172.21.15.27:0/2647329688 shutdown_connections 2024-01-15T18:05:43.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:43.863+0000 7fc2a8200640 1 -- 172.21.15.27:0/2647329688 wait complete. 2024-01-15T18:05:43.889 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:05:44.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.027+0000 7ff684a00640 1 Processor -- start 2024-01-15T18:05:44.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.027+0000 7ff684a00640 1 -- start start 2024-01-15T18:05:44.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.027+0000 7ff684a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff67c09a120 0x7ff67c0b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.027+0000 7ff684a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff67c0b1d90 0x7ff67c0cf180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.027+0000 7ff684a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67c0b1aa0 con 0x7ff67c15d7c0 2024-01-15T18:05:44.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.027+0000 7ff684a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff67c0b1300 con 0x7ff67c0b1d90 2024-01-15T18:05:44.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.027+0000 7ff684a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff67c0b25a0 con 0x7ff67c09a120 2024-01-15T18:05:44.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.027+0000 7ff682000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff67c15d7c0 0x7ff67c099a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41826/0 (socket says 172.21.15.27:41826) 2024-01-15T18:05:44.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.027+0000 7ff682000640 1 -- 172.21.15.27:0/1956741186 learned_addr learned my addr 172.21.15.27:0/1956741186 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:44.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.027+0000 7ff681600640 1 --2- 172.21.15.27:0/1956741186 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff67c09a120 0x7ff67c0b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.027+0000 7ff682a00640 1 --2- 172.21.15.27:0/1956741186 >> v2:172.21.15.93:3300/0 conn(0x7ff67c0b1d90 0x7ff67c0cf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.027+0000 7ff681600640 1 -- 172.21.15.27:0/1956741186 >> v2:172.21.15.93:3300/0 conn(0x7ff67c0b1d90 msgr2=0x7ff67c0cf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:44.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.027+0000 7ff681600640 1 --2- 172.21.15.27:0/1956741186 >> v2:172.21.15.93:3300/0 conn(0x7ff67c0b1d90 0x7ff67c0cf180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.027+0000 7ff681600640 1 -- 172.21.15.27:0/1956741186 >> v1:172.21.15.27:6789/0 conn(0x7ff67c15d7c0 legacy=0x7ff67c099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.027+0000 7ff681600640 1 -- 172.21.15.27:0/1956741186 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff67c0b14e0 con 0x7ff67c09a120 2024-01-15T18:05:44.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.031+0000 7ff681600640 1 --2- 172.21.15.27:0/1956741186 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff67c09a120 0x7ff67c0b35a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff6640097c0 tx=0x7ff66401c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91f9858161743d36 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:44.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.031+0000 7ff680c00640 1 -- 172.21.15.27:0/1956741186 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6640a4ea0 con 0x7ff67c09a120 2024-01-15T18:05:44.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.031+0000 7ff680c00640 1 -- 172.21.15.27:0/1956741186 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6640a7050 con 0x7ff67c09a120 2024-01-15T18:05:44.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.031+0000 7ff680c00640 1 -- 172.21.15.27:0/1956741186 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6640ab820 con 0x7ff67c09a120 2024-01-15T18:05:44.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.035+0000 7ff684a00640 1 -- 172.21.15.27:0/1956741186 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff67c09a120 msgr2=0x7ff67c0b35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.035+0000 7ff684a00640 1 --2- 172.21.15.27:0/1956741186 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff67c09a120 0x7ff67c0b35a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff6640097c0 tx=0x7ff66401c3f0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.035+0000 7ff684a00640 1 -- 172.21.15.27:0/1956741186 shutdown_connections 2024-01-15T18:05:44.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.035+0000 7ff684a00640 1 --2- 172.21.15.27:0/1956741186 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff67c09a120 0x7ff67c0b35a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.035+0000 7ff684a00640 1 --2- 172.21.15.27:0/1956741186 >> v2:172.21.15.93:3300/0 conn(0x7ff67c0b1d90 0x7ff67c0cf180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.035+0000 7ff684a00640 1 -- 172.21.15.27:0/1956741186 >> 172.21.15.27:0/1956741186 conn(0x7ff67c0b2a80 msgr2=0x7ff67c0b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:44.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.035+0000 7ff684a00640 1 -- 172.21.15.27:0/1956741186 shutdown_connections 2024-01-15T18:05:44.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.035+0000 7ff684a00640 1 -- 172.21.15.27:0/1956741186 wait complete. 2024-01-15T18:05:44.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.035+0000 7ff684a00640 1 Processor -- start 2024-01-15T18:05:44.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.035+0000 7ff684a00640 1 -- start start 2024-01-15T18:05:44.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.035+0000 7ff684a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff67c09a120 0x7ff67c158f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.035+0000 7ff684a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff67c0b1d90 0x7ff67c1535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.035+0000 7ff684a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67c15d7c0 0x7ff67c153b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.035+0000 7ff684a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67c15bf00 con 0x7ff67c15d7c0 2024-01-15T18:05:44.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.035+0000 7ff684a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67c15ba70 con 0x7ff67c09a120 2024-01-15T18:05:44.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.035+0000 7ff684a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff67c15ae60 con 0x7ff67c0b1d90 2024-01-15T18:05:44.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.035+0000 7ff682a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67c15d7c0 0x7ff67c153b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.035+0000 7ff682a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67c15d7c0 0x7ff67c153b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38770/0 (socket says 172.21.15.27:38770) 2024-01-15T18:05:44.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.035+0000 7ff682a00640 1 -- 172.21.15.27:0/2073914324 learned_addr learned my addr 172.21.15.27:0/2073914324 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:44.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.035+0000 7ff681600640 1 --2- 172.21.15.27:0/2073914324 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff67c0b1d90 0x7ff67c1535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.035+0000 7ff682000640 1 --2- 172.21.15.27:0/2073914324 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff67c09a120 0x7ff67c158f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.039+0000 7ff682a00640 1 -- 172.21.15.27:0/2073914324 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff67c0b1d90 msgr2=0x7ff67c1535f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.039+0000 7ff682a00640 1 --2- 172.21.15.27:0/2073914324 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff67c0b1d90 0x7ff67c1535f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.039+0000 7ff682a00640 1 -- 172.21.15.27:0/2073914324 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff67c09a120 msgr2=0x7ff67c158f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.039+0000 7ff682a00640 1 --2- 172.21.15.27:0/2073914324 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff67c09a120 0x7ff67c158f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.039+0000 7ff682a00640 1 -- 172.21.15.27:0/2073914324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff664009420 con 0x7ff67c15d7c0 2024-01-15T18:05:44.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.039+0000 7ff682a00640 1 --2- 172.21.15.27:0/2073914324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67c15d7c0 0x7ff67c153b30 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff67400ef90 tx=0x7ff67400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:44.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.039+0000 7ff67a000640 1 -- 172.21.15.27:0/2073914324 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff674009b20 con 0x7ff67c15d7c0 2024-01-15T18:05:44.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.039+0000 7ff67a000640 1 -- 172.21.15.27:0/2073914324 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff67400f040 con 0x7ff67c15d7c0 2024-01-15T18:05:44.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.039+0000 7ff67a000640 1 -- 172.21.15.27:0/2073914324 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6740146c0 con 0x7ff67c15d7c0 2024-01-15T18:05:44.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.039+0000 7ff684a00640 1 -- 172.21.15.27:0/2073914324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff67c15b140 con 0x7ff67c15d7c0 2024-01-15T18:05:44.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.039+0000 7ff684a00640 1 -- 172.21.15.27:0/2073914324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff67c15b700 con 0x7ff67c15d7c0 2024-01-15T18:05:44.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.039+0000 7ff684a00640 1 -- 172.21.15.27:0/2073914324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff67c15bbe0 con 0x7ff67c15d7c0 2024-01-15T18:05:44.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.039+0000 7ff67a000640 1 -- 172.21.15.27:0/2073914324 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7ff674014820 con 0x7ff67c15d7c0 2024-01-15T18:05:44.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.039+0000 7ff67a000640 1 --2- 172.21.15.27:0/2073914324 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff6540668f0 0x7ff654068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.039+0000 7ff67a000640 1 -- 172.21.15.27:0/2073914324 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7ff674085080 con 0x7ff67c15d7c0 2024-01-15T18:05:44.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.043+0000 7ff682000640 1 --2- 172.21.15.27:0/2073914324 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff6540668f0 0x7ff654068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.043+0000 7ff682000640 1 --2- 172.21.15.27:0/2073914324 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff6540668f0 0x7ff654068db0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff66c005890 tx=0x7ff66c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:44.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.043+0000 7ff67a000640 1 -- 172.21.15.27:0/2073914324 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff6740522e0 con 0x7ff67c15d7c0 2024-01-15T18:05:44.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.215+0000 7ff684a00640 1 -- 172.21.15.27:0/2073914324 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff67c04e850 con 0x7ff67c15d7c0 2024-01-15T18:05:44.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.215+0000 7ff67a000640 1 -- 172.21.15.27:0/2073914324 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff674009e60 con 0x7ff67c15d7c0 2024-01-15T18:05:44.219 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-01-15T18:05:44.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.215+0000 7ff684a00640 1 -- 172.21.15.27:0/2073914324 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff6540668f0 msgr2=0x7ff654068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.215+0000 7ff684a00640 1 --2- 172.21.15.27:0/2073914324 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff6540668f0 0x7ff654068db0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff66c005890 tx=0x7ff66c009040 comp rx=0 tx=0).stop 2024-01-15T18:05:44.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.215+0000 7ff684a00640 1 -- 172.21.15.27:0/2073914324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67c15d7c0 msgr2=0x7ff67c153b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.215+0000 7ff684a00640 1 --2- 172.21.15.27:0/2073914324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67c15d7c0 0x7ff67c153b30 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff67400ef90 tx=0x7ff67400c490 comp rx=0 tx=0).stop 2024-01-15T18:05:44.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.215+0000 7ff684a00640 1 -- 172.21.15.27:0/2073914324 shutdown_connections 2024-01-15T18:05:44.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.215+0000 7ff684a00640 1 --2- 172.21.15.27:0/2073914324 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff6540668f0 0x7ff654068db0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.215+0000 7ff684a00640 1 --2- 172.21.15.27:0/2073914324 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff67c15d7c0 0x7ff67c153b30 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.215+0000 7ff684a00640 1 --2- 172.21.15.27:0/2073914324 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff67c0b1d90 0x7ff67c1535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.215+0000 7ff684a00640 1 --2- 172.21.15.27:0/2073914324 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff67c09a120 0x7ff67c158f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.215+0000 7ff684a00640 1 -- 172.21.15.27:0/2073914324 >> 172.21.15.27:0/2073914324 conn(0x7ff67c0b2a80 msgr2=0x7ff67c099a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:44.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.215+0000 7ff684a00640 1 -- 172.21.15.27:0/2073914324 shutdown_connections 2024-01-15T18:05:44.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.215+0000 7ff684a00640 1 -- 172.21.15.27:0/2073914324 wait complete. 2024-01-15T18:05:44.242 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:05:44.242 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:05:44.252 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:44.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.387+0000 7f0260200640 1 Processor -- start 2024-01-15T18:05:44.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.387+0000 7f0260200640 1 -- start start 2024-01-15T18:05:44.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.387+0000 7f0260200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f025815e7d0 0x7f02580cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.387+0000 7f0260200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02580b38b0 con 0x7f02580b20d0 2024-01-15T18:05:44.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.387+0000 7f0260200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02580cd550 con 0x7f02580cedc0 2024-01-15T18:05:44.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.387+0000 7f0260200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02580cd270 con 0x7f025815e7d0 2024-01-15T18:05:44.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.387+0000 7f025e200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f02580b20d0 0x7f02580cc1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41832/0 (socket says 172.21.15.27:41832) 2024-01-15T18:05:44.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.387+0000 7f025e200640 1 -- 172.21.15.27:0/1354485461 learned_addr learned my addr 172.21.15.27:0/1354485461 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:44.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.387+0000 7f025d800640 1 --2- 172.21.15.27:0/1354485461 >> v2:172.21.15.93:3300/0 conn(0x7f025815e7d0 0x7f02580cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.387+0000 7f0257e00640 1 -- 172.21.15.27:0/1354485461 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3525694281 0 0) 0x7f02580b38b0 con 0x7f02580b20d0 2024-01-15T18:05:44.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.387+0000 7f0257e00640 1 -- 172.21.15.27:0/1354485461 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0238003530 con 0x7f02580b20d0 2024-01-15T18:05:44.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.387+0000 7f0257e00640 1 -- 172.21.15.27:0/1354485461 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3427574918 0 0) 0x7f02580cd550 con 0x7f02580cedc0 2024-01-15T18:05:44.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.387+0000 7f0257e00640 1 -- 172.21.15.27:0/1354485461 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02580b38b0 con 0x7f02580cedc0 2024-01-15T18:05:44.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.387+0000 7f0257e00640 1 -- 172.21.15.27:0/1354485461 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2180664639 0 0) 0x7f0238003530 con 0x7f02580b20d0 2024-01-15T18:05:44.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.387+0000 7f0257e00640 1 -- 172.21.15.27:0/1354485461 >> v2:172.21.15.93:3300/0 conn(0x7f025815e7d0 msgr2=0x7f02580cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:44.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.387+0000 7f0257e00640 1 --2- 172.21.15.27:0/1354485461 >> v2:172.21.15.93:3300/0 conn(0x7f025815e7d0 0x7f02580cf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.387+0000 7f0257e00640 1 -- 172.21.15.27:0/1354485461 >> v1:172.21.15.93:6789/0 conn(0x7f02580cedc0 legacy=0x7f02580cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.387+0000 7f0257e00640 1 -- 172.21.15.27:0/1354485461 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02580b35d0 con 0x7f02580b20d0 2024-01-15T18:05:44.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.387+0000 7f0257e00640 1 -- 172.21.15.27:0/1354485461 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f0250001a00 con 0x7f02580b20d0 2024-01-15T18:05:44.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.387+0000 7f025d800640 1 --2- 172.21.15.27:0/1354485461 >> v2:172.21.15.93:3300/0 conn(0x7f025815e7d0 0x7f02580cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:44.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.387+0000 7f0257e00640 1 -- 172.21.15.27:0/1354485461 >> v1:172.21.15.27:6789/0 conn(0x7f02580b20d0 legacy=0x7f02580cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.387+0000 7f0257e00640 1 --2- 172.21.15.27:0/1354485461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0238003b80 0x7f0238003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.387+0000 7f0257e00640 1 --2- 172.21.15.27:0/1354485461 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f02380047d0 0x7f0238004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.387+0000 7f0257e00640 1 --2- 172.21.15.27:0/1354485461 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0238005190 0x7f023800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.391+0000 7f0257e00640 1 -- 172.21.15.27:0/1354485461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02580cd270 con 0x7f0238003b80 2024-01-15T18:05:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.391+0000 7f0257e00640 1 -- 172.21.15.27:0/1354485461 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f023800c090 con 0x7f02380047d0 2024-01-15T18:05:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.391+0000 7f0257e00640 1 -- 172.21.15.27:0/1354485461 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f023800c250 con 0x7f0238005190 2024-01-15T18:05:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.391+0000 7f025d800640 1 --2- 172.21.15.27:0/1354485461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0238003b80 0x7f0238003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.391+0000 7f025ce00640 1 --2- 172.21.15.27:0/1354485461 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f02380047d0 0x7f0238004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.391+0000 7f025e200640 1 --2- 172.21.15.27:0/1354485461 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0238005190 0x7f023800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.391+0000 7f025d800640 1 -- 172.21.15.27:0/1354485461 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0238005190 msgr2=0x7f023800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.391+0000 7f025d800640 1 --2- 172.21.15.27:0/1354485461 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0238005190 0x7f023800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.391+0000 7f025d800640 1 -- 172.21.15.27:0/1354485461 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f02380047d0 msgr2=0x7f0238004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.391+0000 7f025d800640 1 --2- 172.21.15.27:0/1354485461 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f02380047d0 0x7f0238004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.391+0000 7f025d800640 1 -- 172.21.15.27:0/1354485461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0238003530 con 0x7f0238003b80 2024-01-15T18:05:44.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.391+0000 7f025e200640 1 --2- 172.21.15.27:0/1354485461 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0238005190 0x7f023800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:44.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.391+0000 7f025e200640 1 -- 172.21.15.27:0/1354485461 reap_dead start 2024-01-15T18:05:44.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.391+0000 7f025d800640 1 --2- 172.21.15.27:0/1354485461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0238003b80 0x7f0238003f80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0248009cb0 tx=0x7f024801cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=563b724358a8f110 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:44.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.391+0000 7f0257e00640 1 -- 172.21.15.27:0/1354485461 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02480a5830 con 0x7f0238003b80 2024-01-15T18:05:44.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.391+0000 7f0257e00640 1 -- 172.21.15.27:0/1354485461 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02480a5990 con 0x7f0238003b80 2024-01-15T18:05:44.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.391+0000 7f0257e00640 1 -- 172.21.15.27:0/1354485461 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02480a5d80 con 0x7f0238003b80 2024-01-15T18:05:44.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.395+0000 7f0260200640 1 -- 172.21.15.27:0/1354485461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0238003b80 msgr2=0x7f0238003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.395+0000 7f0260200640 1 --2- 172.21.15.27:0/1354485461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0238003b80 0x7f0238003f80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0248009cb0 tx=0x7f024801cbf0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.395+0000 7f0260200640 1 -- 172.21.15.27:0/1354485461 shutdown_connections 2024-01-15T18:05:44.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.395+0000 7f0260200640 1 --2- 172.21.15.27:0/1354485461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0238003b80 0x7f0238003f80 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.395+0000 7f0260200640 1 -- 172.21.15.27:0/1354485461 >> 172.21.15.27:0/1354485461 conn(0x7f02580b2a10 msgr2=0x7f02580b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:44.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.395+0000 7f0260200640 1 -- 172.21.15.27:0/1354485461 shutdown_connections 2024-01-15T18:05:44.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.395+0000 7f0260200640 1 -- 172.21.15.27:0/1354485461 wait complete. 2024-01-15T18:05:44.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.395+0000 7f0260200640 1 Processor -- start 2024-01-15T18:05:44.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.395+0000 7f0260200640 1 -- start start 2024-01-15T18:05:44.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.395+0000 7f0260200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0238003b80 0x7f025815a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.395+0000 7f0260200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f02581f5cd0 0x7f02580b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.395+0000 7f0260200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02580b42c0 0x7f02580b4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.395+0000 7f0260200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02581f7160 con 0x7f02580b42c0 2024-01-15T18:05:44.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.395+0000 7f0260200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02580b4c80 con 0x7f02581f5cd0 2024-01-15T18:05:44.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.395+0000 7f0260200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02581f6d10 con 0x7f0238003b80 2024-01-15T18:05:44.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.395+0000 7f025e200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02580b42c0 0x7f02580b4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.395+0000 7f025d800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0238003b80 0x7f025815a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.395+0000 7f025e200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02580b42c0 0x7f02580b4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38786/0 (socket says 172.21.15.27:38786) 2024-01-15T18:05:44.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.395+0000 7f025e200640 1 -- 172.21.15.27:0/3412176728 learned_addr learned my addr 172.21.15.27:0/3412176728 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:44.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.395+0000 7f025ce00640 1 --2- 172.21.15.27:0/3412176728 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f02581f5cd0 0x7f02580b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.399+0000 7f025d800640 1 -- 172.21.15.27:0/3412176728 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f02581f5cd0 msgr2=0x7f02580b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.399+0000 7f025d800640 1 --2- 172.21.15.27:0/3412176728 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f02581f5cd0 0x7f02580b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.399+0000 7f025d800640 1 -- 172.21.15.27:0/3412176728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02580b42c0 msgr2=0x7f02580b4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.399+0000 7f025d800640 1 --2- 172.21.15.27:0/3412176728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02580b42c0 0x7f02580b4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.399+0000 7f025d800640 1 -- 172.21.15.27:0/3412176728 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02580b6b30 con 0x7f0238003b80 2024-01-15T18:05:44.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.399+0000 7f025e200640 1 --2- 172.21.15.27:0/3412176728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02580b42c0 0x7f02580b4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:44.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.399+0000 7f025d800640 1 --2- 172.21.15.27:0/3412176728 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0238003b80 0x7f025815a450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f02480ac000 tx=0x7f0248019990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:44.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.399+0000 7f0255600640 1 -- 172.21.15.27:0/3412176728 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02480b8650 con 0x7f0238003b80 2024-01-15T18:05:44.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.399+0000 7f0255600640 1 -- 172.21.15.27:0/3412176728 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02480b87b0 con 0x7f0238003b80 2024-01-15T18:05:44.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.399+0000 7f0255600640 1 -- 172.21.15.27:0/3412176728 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02480b8b60 con 0x7f0238003b80 2024-01-15T18:05:44.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.399+0000 7f0260200640 1 -- 172.21.15.27:0/3412176728 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02581f6470 con 0x7f0238003b80 2024-01-15T18:05:44.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.399+0000 7f0260200640 1 -- 172.21.15.27:0/3412176728 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02581f6a30 con 0x7f0238003b80 2024-01-15T18:05:44.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.399+0000 7f0260200640 1 -- 172.21.15.27:0/3412176728 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02581f6ec0 con 0x7f0238003b80 2024-01-15T18:05:44.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.399+0000 7f0255600640 1 -- 172.21.15.27:0/3412176728 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f02480b4070 con 0x7f0238003b80 2024-01-15T18:05:44.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.399+0000 7f0255600640 1 --2- 172.21.15.27:0/3412176728 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0230066920 0x7f0230068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.399+0000 7f0255600640 1 -- 172.21.15.27:0/3412176728 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f024811e250 con 0x7f0238003b80 2024-01-15T18:05:44.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.403+0000 7f025ce00640 1 --2- 172.21.15.27:0/3412176728 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0230066920 0x7f0230068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.403+0000 7f025ce00640 1 --2- 172.21.15.27:0/3412176728 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0230066920 0x7f0230068de0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f02581f38f0 tx=0x7f0240009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:44.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.407+0000 7f0255600640 1 -- 172.21.15.27:0/3412176728 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f02581f6ec0 con 0x7f0238003b80 2024-01-15T18:05:44.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.563+0000 7f0260200640 1 -- 172.21.15.27:0/3412176728 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02581f72a0 con 0x7f0238003b80 2024-01-15T18:05:44.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.567+0000 7f0255600640 1 -- 172.21.15.27:0/3412176728 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16705 (secure 0 0 0) 0x7f02581f72a0 con 0x7f0238003b80 2024-01-15T18:05:44.571 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:44.571 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":23,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:05:40.008536+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:04:56.220608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T18:04:57.210421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:44.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.567+0000 7f0260200640 1 -- 172.21.15.27:0/3412176728 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0230066920 msgr2=0x7f0230068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.567+0000 7f0260200640 1 --2- 172.21.15.27:0/3412176728 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0230066920 0x7f0230068de0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f02581f38f0 tx=0x7f0240009040 comp rx=0 tx=0).stop 2024-01-15T18:05:44.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.567+0000 7f0260200640 1 -- 172.21.15.27:0/3412176728 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0238003b80 msgr2=0x7f025815a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.567+0000 7f0260200640 1 --2- 172.21.15.27:0/3412176728 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0238003b80 0x7f025815a450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f02480ac000 tx=0x7f0248019990 comp rx=0 tx=0).stop 2024-01-15T18:05:44.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.567+0000 7f0260200640 1 -- 172.21.15.27:0/3412176728 shutdown_connections 2024-01-15T18:05:44.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.567+0000 7f0260200640 1 --2- 172.21.15.27:0/3412176728 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0230066920 0x7f0230068de0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.567+0000 7f0260200640 1 --2- 172.21.15.27:0/3412176728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02580b42c0 0x7f02580b4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.567+0000 7f0260200640 1 --2- 172.21.15.27:0/3412176728 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f02581f5cd0 0x7f02580b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.567+0000 7f0260200640 1 --2- 172.21.15.27:0/3412176728 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0238003b80 0x7f025815a450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.567+0000 7f0260200640 1 -- 172.21.15.27:0/3412176728 >> 172.21.15.27:0/3412176728 conn(0x7f02580b2a10 msgr2=0x7f025806be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:44.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.567+0000 7f0260200640 1 -- 172.21.15.27:0/3412176728 shutdown_connections 2024-01-15T18:05:44.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.567+0000 7f0260200640 1 -- 172.21.15.27:0/3412176728 wait complete. 2024-01-15T18:05:44.594 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T18:04:56.220608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T18:04:57.210421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:05:44.594 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:05:44.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.731+0000 7f7526c00640 1 Processor -- start 2024-01-15T18:05:44.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.735+0000 7f7526c00640 1 -- start start 2024-01-15T18:05:44.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.735+0000 7f7526c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f752015d480 0x7f75200cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.735+0000 7f7526c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f752009a120 0x7f75200d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.735+0000 7f7526c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f752015d860 con 0x7f752006c740 2024-01-15T18:05:44.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.735+0000 7f7526c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f752006b6a0 con 0x7f752015d480 2024-01-15T18:05:44.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.735+0000 7f7526c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f752006b980 con 0x7f752009a120 2024-01-15T18:05:44.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.735+0000 7f751f400640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7f752006c740 0x7f7520099a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:36806/0 (socket says 172.21.15.27:36806) 2024-01-15T18:05:44.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.735+0000 7f751f400640 1 -- 172.21.15.27:0/2025036876 learned_addr learned my addr 172.21.15.27:0/2025036876 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:44.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.735+0000 7f751fe00640 1 --2- 172.21.15.27:0/2025036876 >> v2:172.21.15.93:3300/0 conn(0x7f752015d480 0x7f75200cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.735+0000 7f7524c00640 1 --2- 172.21.15.27:0/2025036876 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f752009a120 0x7f75200d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.735+0000 7f751ea00640 1 -- 172.21.15.27:0/2025036876 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906803541 0 0) 0x7f752015d860 con 0x7f752006c740 2024-01-15T18:05:44.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.735+0000 7f751ea00640 1 -- 172.21.15.27:0/2025036876 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7500003530 con 0x7f752006c740 2024-01-15T18:05:44.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.735+0000 7f751fe00640 1 -- 172.21.15.27:0/2025036876 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f752009a120 msgr2=0x7f75200d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:44.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.735+0000 7f751fe00640 1 --2- 172.21.15.27:0/2025036876 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f752009a120 0x7f75200d0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.735+0000 7f751fe00640 1 -- 172.21.15.27:0/2025036876 >> v1:172.21.15.93:6789/0 conn(0x7f752006c740 legacy=0x7f7520099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.735+0000 7f751fe00640 1 -- 172.21.15.27:0/2025036876 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f752006bc60 con 0x7f752015d480 2024-01-15T18:05:44.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.735+0000 7f7524c00640 1 --2- 172.21.15.27:0/2025036876 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f752009a120 0x7f75200d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:44.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.739+0000 7f751fe00640 1 --2- 172.21.15.27:0/2025036876 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f752015d480 0x7f75200cfbf0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f75100097f0 tx=0x7f751001c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50114d1e7f783f0b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:44.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.739+0000 7f751ea00640 1 -- 172.21.15.27:0/2025036876 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75100ab600 con 0x7f752015d480 2024-01-15T18:05:44.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.739+0000 7f751ea00640 1 -- 172.21.15.27:0/2025036876 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75100a7050 con 0x7f752015d480 2024-01-15T18:05:44.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.739+0000 7f751ea00640 1 -- 172.21.15.27:0/2025036876 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75100ab980 con 0x7f752015d480 2024-01-15T18:05:44.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.739+0000 7f7526c00640 1 -- 172.21.15.27:0/2025036876 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f752015d480 msgr2=0x7f75200cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.739+0000 7f7526c00640 1 --2- 172.21.15.27:0/2025036876 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f752015d480 0x7f75200cfbf0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f75100097f0 tx=0x7f751001c450 comp rx=0 tx=0).stop 2024-01-15T18:05:44.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.739+0000 7f7526c00640 1 -- 172.21.15.27:0/2025036876 shutdown_connections 2024-01-15T18:05:44.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.739+0000 7f7526c00640 1 --2- 172.21.15.27:0/2025036876 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f752009a120 0x7f75200d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.739+0000 7f7526c00640 1 --2- 172.21.15.27:0/2025036876 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f752015d480 0x7f75200cfbf0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.739+0000 7f7526c00640 1 -- 172.21.15.27:0/2025036876 >> 172.21.15.27:0/2025036876 conn(0x7f75200b2a80 msgr2=0x7f75200b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:44.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.739+0000 7f7526c00640 1 -- 172.21.15.27:0/2025036876 shutdown_connections 2024-01-15T18:05:44.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.739+0000 7f7526c00640 1 -- 172.21.15.27:0/2025036876 wait complete. 2024-01-15T18:05:44.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.743+0000 7f7526c00640 1 Processor -- start 2024-01-15T18:05:44.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.743+0000 7f7526c00640 1 -- start start 2024-01-15T18:05:44.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.743+0000 7f7526c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f752006c740 0x7f7520153a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.743+0000 7f7526c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f752009a120 0x7f75200b6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.743+0000 7f7526c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f752015d480 0x7f75200c7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.743+0000 7f7526c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75200c65f0 con 0x7f752015d480 2024-01-15T18:05:44.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.743+0000 7f7526c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75200c5e20 con 0x7f752009a120 2024-01-15T18:05:44.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.743+0000 7f7526c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75200b6d60 con 0x7f752006c740 2024-01-15T18:05:44.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.743+0000 7f7524c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f752015d480 0x7f75200c7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.743+0000 7f7524c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f752015d480 0x7f75200c7110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38788/0 (socket says 172.21.15.27:38788) 2024-01-15T18:05:44.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.743+0000 7f7524c00640 1 -- 172.21.15.27:0/2536512708 learned_addr learned my addr 172.21.15.27:0/2536512708 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:44.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.743+0000 7f751f400640 1 --2- 172.21.15.27:0/2536512708 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f752009a120 0x7f75200b6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.743+0000 7f751fe00640 1 --2- 172.21.15.27:0/2536512708 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f752006c740 0x7f7520153a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.743+0000 7f7524c00640 1 -- 172.21.15.27:0/2536512708 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f752006c740 msgr2=0x7f7520153a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.743+0000 7f7524c00640 1 --2- 172.21.15.27:0/2536512708 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f752006c740 0x7f7520153a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.743+0000 7f7524c00640 1 -- 172.21.15.27:0/2536512708 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f752009a120 msgr2=0x7f75200b6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.743+0000 7f7524c00640 1 --2- 172.21.15.27:0/2536512708 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f752009a120 0x7f75200b6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.743+0000 7f7524c00640 1 -- 172.21.15.27:0/2536512708 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7518009a70 con 0x7f752015d480 2024-01-15T18:05:44.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.743+0000 7f7524c00640 1 --2- 172.21.15.27:0/2536512708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f752015d480 0x7f75200c7110 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f751800efb0 tx=0x7f751800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:44.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.743+0000 7f7517e00640 1 -- 172.21.15.27:0/2536512708 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7518009d50 con 0x7f752015d480 2024-01-15T18:05:44.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.743+0000 7f7517e00640 1 -- 172.21.15.27:0/2536512708 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f751800f040 con 0x7f752015d480 2024-01-15T18:05:44.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.743+0000 7f7517e00640 1 -- 172.21.15.27:0/2536512708 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75180146a0 con 0x7f752015d480 2024-01-15T18:05:44.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.743+0000 7f7526c00640 1 -- 172.21.15.27:0/2536512708 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7510009420 con 0x7f752015d480 2024-01-15T18:05:44.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.743+0000 7f7526c00640 1 -- 172.21.15.27:0/2536512708 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75200c6c80 con 0x7f752015d480 2024-01-15T18:05:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.747+0000 7f7526c00640 1 -- 172.21.15.27:0/2536512708 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75200c6e10 con 0x7f752015d480 2024-01-15T18:05:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.747+0000 7f7517e00640 1 -- 172.21.15.27:0/2536512708 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f7518014800 con 0x7f752015d480 2024-01-15T18:05:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.747+0000 7f7517e00640 1 --2- 172.21.15.27:0/2536512708 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f74f8066920 0x7f74f8068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.747+0000 7f7517e00640 1 -- 172.21.15.27:0/2536512708 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f75180853a0 con 0x7f752015d480 2024-01-15T18:05:44.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.747+0000 7f751fe00640 1 --2- 172.21.15.27:0/2536512708 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f74f8066920 0x7f74f8068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.747+0000 7f751fe00640 1 --2- 172.21.15.27:0/2536512708 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f74f8066920 0x7f74f8068de0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f75100097c0 tx=0x7f75100a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:44.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.751+0000 7f7517e00640 1 -- 172.21.15.27:0/2536512708 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7518052560 con 0x7f752015d480 2024-01-15T18:05:44.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.911+0000 7f7526c00640 1 -- 172.21.15.27:0/2536512708 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f752004e850 con 0x7f752015d480 2024-01-15T18:05:44.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.911+0000 7f7517e00640 1 -- 172.21.15.27:0/2536512708 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7518055b80 con 0x7f752015d480 2024-01-15T18:05:44.917 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:05:44.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.915+0000 7f7526c00640 1 -- 172.21.15.27:0/2536512708 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f74f8066920 msgr2=0x7f74f8068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.915+0000 7f7526c00640 1 --2- 172.21.15.27:0/2536512708 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f74f8066920 0x7f74f8068de0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f75100097c0 tx=0x7f75100a6000 comp rx=0 tx=0).stop 2024-01-15T18:05:44.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.915+0000 7f7526c00640 1 -- 172.21.15.27:0/2536512708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f752015d480 msgr2=0x7f75200c7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.915+0000 7f7526c00640 1 --2- 172.21.15.27:0/2536512708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f752015d480 0x7f75200c7110 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f751800efb0 tx=0x7f751800c490 comp rx=0 tx=0).stop 2024-01-15T18:05:44.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.915+0000 7f7526c00640 1 -- 172.21.15.27:0/2536512708 shutdown_connections 2024-01-15T18:05:44.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.915+0000 7f7526c00640 1 --2- 172.21.15.27:0/2536512708 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f74f8066920 0x7f74f8068de0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.915+0000 7f7526c00640 1 --2- 172.21.15.27:0/2536512708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f752015d480 0x7f75200c7110 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.915+0000 7f7526c00640 1 --2- 172.21.15.27:0/2536512708 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f752009a120 0x7f75200b6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.915+0000 7f7526c00640 1 --2- 172.21.15.27:0/2536512708 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f752006c740 0x7f7520153a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.915+0000 7f7526c00640 1 -- 172.21.15.27:0/2536512708 >> 172.21.15.27:0/2536512708 conn(0x7f75200b2a80 msgr2=0x7f75201591e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:44.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.915+0000 7f7526c00640 1 -- 172.21.15.27:0/2536512708 shutdown_connections 2024-01-15T18:05:44.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:44.915+0000 7f7526c00640 1 -- 172.21.15.27:0/2536512708 wait complete. 2024-01-15T18:05:44.940 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T18:05:45.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.078+0000 7ffb19600640 1 Processor -- start 2024-01-15T18:05:45.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.078+0000 7ffb19600640 1 -- start start 2024-01-15T18:05:45.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.078+0000 7ffb19600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffb1406c430 0x7ffb14070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:45.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.078+0000 7ffb19600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ffb140b1db0 0x7ffb140b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:45.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.078+0000 7ffb19600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ffb140ce050 0x7ffb140ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:45.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.078+0000 7ffb19600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb140cea20 con 0x7ffb140b1db0 2024-01-15T18:05:45.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.078+0000 7ffb19600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb140ccfb0 con 0x7ffb140ce050 2024-01-15T18:05:45.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.078+0000 7ffb19600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb140cd290 con 0x7ffb1406c430 2024-01-15T18:05:45.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.082+0000 7ffb12000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ffb140b1db0 0x7ffb140b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:45.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.082+0000 7ffb12000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ffb140b1db0 0x7ffb140b2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38794/0 (socket says 172.21.15.27:38794) 2024-01-15T18:05:45.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.082+0000 7ffb12000640 1 -- 172.21.15.27:0/2622823519 learned_addr learned my addr 172.21.15.27:0/2622823519 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:45.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.082+0000 7ffb13400640 1 --2- 172.21.15.27:0/2622823519 >> v2:172.21.15.93:3300/0 conn(0x7ffb140ce050 0x7ffb140ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:45.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.082+0000 7ffb12a00640 1 --2- 172.21.15.27:0/2622823519 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffb1406c430 0x7ffb14070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:45.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.082+0000 7ffb12000640 1 -- 172.21.15.27:0/2622823519 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffb1406c430 msgr2=0x7ffb14070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:45.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.082+0000 7ffb12000640 1 --2- 172.21.15.27:0/2622823519 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffb1406c430 0x7ffb14070ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.082+0000 7ffb12000640 1 -- 172.21.15.27:0/2622823519 >> v2:172.21.15.93:3300/0 conn(0x7ffb140ce050 msgr2=0x7ffb140ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:45.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.082+0000 7ffb12000640 1 --2- 172.21.15.27:0/2622823519 >> v2:172.21.15.93:3300/0 conn(0x7ffb140ce050 0x7ffb140ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.082+0000 7ffb12000640 1 -- 172.21.15.27:0/2622823519 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb140cd570 con 0x7ffb140b1db0 2024-01-15T18:05:45.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.082+0000 7ffb12000640 1 --2- 172.21.15.27:0/2622823519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb140b1db0 0x7ffb140b2210 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ffb14072330 tx=0x7ffafc01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46082e706e224c74 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:45.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.082+0000 7ffb11600640 1 -- 172.21.15.27:0/2622823519 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc0ab600 con 0x7ffb140b1db0 2024-01-15T18:05:45.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.082+0000 7ffb11600640 1 -- 172.21.15.27:0/2622823519 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffafc0a7050 con 0x7ffb140b1db0 2024-01-15T18:05:45.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.082+0000 7ffb11600640 1 -- 172.21.15.27:0/2622823519 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc0ab980 con 0x7ffb140b1db0 2024-01-15T18:05:45.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.086+0000 7ffb19600640 1 -- 172.21.15.27:0/2622823519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb140b1db0 msgr2=0x7ffb140b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:45.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.086+0000 7ffb19600640 1 --2- 172.21.15.27:0/2622823519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb140b1db0 0x7ffb140b2210 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ffb14072330 tx=0x7ffafc01c470 comp rx=0 tx=0).stop 2024-01-15T18:05:45.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.086+0000 7ffb19600640 1 -- 172.21.15.27:0/2622823519 shutdown_connections 2024-01-15T18:05:45.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.086+0000 7ffb19600640 1 --2- 172.21.15.27:0/2622823519 >> v2:172.21.15.93:3300/0 conn(0x7ffb140ce050 0x7ffb140ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.086+0000 7ffb19600640 1 --2- 172.21.15.27:0/2622823519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb140b1db0 0x7ffb140b2210 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.086+0000 7ffb19600640 1 --2- 172.21.15.27:0/2622823519 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffb1406c430 0x7ffb14070ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.086+0000 7ffb19600640 1 -- 172.21.15.27:0/2622823519 >> 172.21.15.27:0/2622823519 conn(0x7ffb140cb700 msgr2=0x7ffb140cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:45.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.086+0000 7ffb19600640 1 -- 172.21.15.27:0/2622823519 shutdown_connections 2024-01-15T18:05:45.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.086+0000 7ffb19600640 1 -- 172.21.15.27:0/2622823519 wait complete. 2024-01-15T18:05:45.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.086+0000 7ffb19600640 1 Processor -- start 2024-01-15T18:05:45.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.086+0000 7ffb19600640 1 -- start start 2024-01-15T18:05:45.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.086+0000 7ffb19600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb1406c430 0x7ffb14143370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:45.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.086+0000 7ffb19600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffb140b1db0 0x7ffb141f4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:45.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.086+0000 7ffb19600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ffb140ce050 0x7ffb141f5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:45.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.086+0000 7ffb19600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb14150210 con 0x7ffb1406c430 2024-01-15T18:05:45.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.086+0000 7ffb19600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1415bfe0 con 0x7ffb140ce050 2024-01-15T18:05:45.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.086+0000 7ffb19600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1414fb50 con 0x7ffb140b1db0 2024-01-15T18:05:45.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.086+0000 7ffb12a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb1406c430 0x7ffb14143370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:45.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.086+0000 7ffb12a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb1406c430 0x7ffb14143370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38804/0 (socket says 172.21.15.27:38804) 2024-01-15T18:05:45.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.086+0000 7ffb12a00640 1 -- 172.21.15.27:0/1815629998 learned_addr learned my addr 172.21.15.27:0/1815629998 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:45.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.086+0000 7ffb12000640 1 --2- 172.21.15.27:0/1815629998 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffb140b1db0 0x7ffb141f4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.086+0000 7ffb13400640 1 --2- 172.21.15.27:0/1815629998 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ffb140ce050 0x7ffb141f5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.090+0000 7ffb12a00640 1 -- 172.21.15.27:0/1815629998 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffb140b1db0 msgr2=0x7ffb141f4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.090+0000 7ffb12a00640 1 --2- 172.21.15.27:0/1815629998 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffb140b1db0 0x7ffb141f4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.090+0000 7ffb12a00640 1 -- 172.21.15.27:0/1815629998 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ffb140ce050 msgr2=0x7ffb141f5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.090+0000 7ffb12a00640 1 --2- 172.21.15.27:0/1815629998 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ffb140ce050 0x7ffb141f5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.090+0000 7ffb12a00640 1 -- 172.21.15.27:0/1815629998 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffafc0094d0 con 0x7ffb1406c430 2024-01-15T18:05:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.090+0000 7ffb12a00640 1 --2- 172.21.15.27:0/1815629998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb1406c430 0x7ffb14143370 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ffb0400d800 tx=0x7ffb0400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.090+0000 7ffb0aa00640 1 -- 172.21.15.27:0/1815629998 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0400b6c0 con 0x7ffb1406c430 2024-01-15T18:05:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.090+0000 7ffb0aa00640 1 -- 172.21.15.27:0/1815629998 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0400b820 con 0x7ffb1406c430 2024-01-15T18:05:45.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.090+0000 7ffb0aa00640 1 -- 172.21.15.27:0/1815629998 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0400bb90 con 0x7ffb1406c430 2024-01-15T18:05:45.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.090+0000 7ffb19600640 1 -- 172.21.15.27:0/1815629998 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb1414fe30 con 0x7ffb1406c430 2024-01-15T18:05:45.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.090+0000 7ffb19600640 1 -- 172.21.15.27:0/1815629998 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1415c2c0 con 0x7ffb1406c430 2024-01-15T18:05:45.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.090+0000 7ffb19600640 1 -- 172.21.15.27:0/1815629998 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb1404e850 con 0x7ffb1406c430 2024-01-15T18:05:45.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.090+0000 7ffb0aa00640 1 -- 172.21.15.27:0/1815629998 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7ffb0400bcf0 con 0x7ffb1406c430 2024-01-15T18:05:45.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.090+0000 7ffb0aa00640 1 --2- 172.21.15.27:0/1815629998 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ffaec0668f0 0x7ffaec068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:45.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.090+0000 7ffb0aa00640 1 -- 172.21.15.27:0/1815629998 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7ffb04084830 con 0x7ffb1406c430 2024-01-15T18:05:45.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.094+0000 7ffb12000640 1 --2- 172.21.15.27:0/1815629998 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ffaec0668f0 0x7ffaec068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:45.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.094+0000 7ffb12000640 1 --2- 172.21.15.27:0/1815629998 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ffaec0668f0 0x7ffaec068db0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ffafc0ad000 tx=0x7ffafc0a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:45.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.094+0000 7ffb0aa00640 1 -- 172.21.15.27:0/1815629998 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ffb04051b20 con 0x7ffb1406c430 2024-01-15T18:05:45.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.250+0000 7ffb19600640 1 -- 172.21.15.27:0/1815629998 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffb140af700 con 0x7ffb1406c430 2024-01-15T18:05:45.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.250+0000 7ffb0aa00640 1 -- 172.21.15.27:0/1815629998 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffb04055140 con 0x7ffb1406c430 2024-01-15T18:05:45.259 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-15T18:05:45.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.254+0000 7ffb19600640 1 -- 172.21.15.27:0/1815629998 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ffaec0668f0 msgr2=0x7ffaec068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:45.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.254+0000 7ffb19600640 1 --2- 172.21.15.27:0/1815629998 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ffaec0668f0 0x7ffaec068db0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ffafc0ad000 tx=0x7ffafc0a6000 comp rx=0 tx=0).stop 2024-01-15T18:05:45.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.254+0000 7ffb19600640 1 -- 172.21.15.27:0/1815629998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb1406c430 msgr2=0x7ffb14143370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:45.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.254+0000 7ffb19600640 1 --2- 172.21.15.27:0/1815629998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb1406c430 0x7ffb14143370 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ffb0400d800 tx=0x7ffb0400dcc0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.254+0000 7ffb19600640 1 -- 172.21.15.27:0/1815629998 shutdown_connections 2024-01-15T18:05:45.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.254+0000 7ffb19600640 1 --2- 172.21.15.27:0/1815629998 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ffaec0668f0 0x7ffaec068db0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.254+0000 7ffb19600640 1 --2- 172.21.15.27:0/1815629998 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ffb140ce050 0x7ffb141f5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.254+0000 7ffb19600640 1 --2- 172.21.15.27:0/1815629998 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffb140b1db0 0x7ffb141f4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.254+0000 7ffb19600640 1 --2- 172.21.15.27:0/1815629998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb1406c430 0x7ffb14143370 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.254+0000 7ffb19600640 1 -- 172.21.15.27:0/1815629998 >> 172.21.15.27:0/1815629998 conn(0x7ffb140cb700 msgr2=0x7ffb140b2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:45.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.254+0000 7ffb19600640 1 -- 172.21.15.27:0/1815629998 shutdown_connections 2024-01-15T18:05:45.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.254+0000 7ffb19600640 1 -- 172.21.15.27:0/1815629998 wait complete. 2024-01-15T18:05:45.280 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T18:05:45.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.418+0000 7f02bf800640 1 Processor -- start 2024-01-15T18:05:45.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.418+0000 7f02bf800640 1 -- start start 2024-01-15T18:05:45.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.418+0000 7f02bf800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f02b815b310 0x7f02b80d1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:45.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.418+0000 7f02bf800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02b80cc770 con 0x7f02b80cdf90 2024-01-15T18:05:45.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.418+0000 7f02bf800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02b80ceb70 con 0x7f02b80afd00 2024-01-15T18:05:45.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.418+0000 7f02bf800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b815b6f0 con 0x7f02b815b310 2024-01-15T18:05:45.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.418+0000 7f02bce00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f02b815b310 0x7f02b80d1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:45.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.418+0000 7f02bd800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f02b80cdf90 0x7f02b80cf000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41838/0 (socket says 172.21.15.27:41838) 2024-01-15T18:05:45.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.418+0000 7f02bd800640 1 -- 172.21.15.27:0/1216524161 learned_addr learned my addr 172.21.15.27:0/1216524161 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:45.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.422+0000 7f02bce00640 1 -- 172.21.15.27:0/1216524161 >> v1:172.21.15.93:6789/0 conn(0x7f02b80afd00 legacy=0x7f02b80cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:45.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.422+0000 7f02bce00640 1 -- 172.21.15.27:0/1216524161 >> v1:172.21.15.27:6789/0 conn(0x7f02b80cdf90 legacy=0x7f02b80cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:45.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.422+0000 7f02b7400640 1 -- 172.21.15.27:0/1216524161 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630369290 0 0) 0x7f02b80ceb70 con 0x7f02b80afd00 2024-01-15T18:05:45.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.422+0000 7f02bce00640 1 -- 172.21.15.27:0/1216524161 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b80cc9c0 con 0x7f02b815b310 2024-01-15T18:05:45.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.422+0000 7f02bce00640 1 --2- 172.21.15.27:0/1216524161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b815b310 0x7f02b80d1990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f02a80097c0 tx=0x7f02a801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=946d6370e183f575 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:45.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.422+0000 7f02b7400640 1 -- 172.21.15.27:0/1216524161 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a80ab600 con 0x7f02b815b310 2024-01-15T18:05:45.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.422+0000 7f02b7400640 1 -- 172.21.15.27:0/1216524161 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a80a7050 con 0x7f02b815b310 2024-01-15T18:05:45.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.422+0000 7f02b7400640 1 -- 172.21.15.27:0/1216524161 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a80ab980 con 0x7f02b815b310 2024-01-15T18:05:45.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.422+0000 7f02bf800640 1 -- 172.21.15.27:0/1216524161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b815b310 msgr2=0x7f02b80d1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:45.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.422+0000 7f02bf800640 1 --2- 172.21.15.27:0/1216524161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b815b310 0x7f02b80d1990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f02a80097c0 tx=0x7f02a801c3a0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.426+0000 7f02bf800640 1 -- 172.21.15.27:0/1216524161 shutdown_connections 2024-01-15T18:05:45.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.426+0000 7f02bf800640 1 --2- 172.21.15.27:0/1216524161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b815b310 0x7f02b80d1990 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.426+0000 7f02bf800640 1 -- 172.21.15.27:0/1216524161 >> 172.21.15.27:0/1216524161 conn(0x7f02b80d0e70 msgr2=0x7f02b80d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:45.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.426+0000 7f02bf800640 1 -- 172.21.15.27:0/1216524161 shutdown_connections 2024-01-15T18:05:45.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.426+0000 7f02bf800640 1 -- 172.21.15.27:0/1216524161 wait complete. 2024-01-15T18:05:45.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.426+0000 7f02bf800640 1 Processor -- start 2024-01-15T18:05:45.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.426+0000 7f02bf800640 1 -- start start 2024-01-15T18:05:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.426+0000 7f02bf800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f02b80afd00 0x7f02b81583a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.426+0000 7f02bf800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b80cdf90 0x7f02b81f6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.426+0000 7f02bf800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f02b815b310 0x7f02b81f67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.426+0000 7f02bf800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b80ca1c0 con 0x7f02b80cdf90 2024-01-15T18:05:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.426+0000 7f02bf800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b815e510 con 0x7f02b815b310 2024-01-15T18:05:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.426+0000 7f02bf800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b80ca410 con 0x7f02b80afd00 2024-01-15T18:05:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.426+0000 7f02b7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b80cdf90 0x7f02b81f6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.426+0000 7f02b7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b80cdf90 0x7f02b81f6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38820/0 (socket says 172.21.15.27:38820) 2024-01-15T18:05:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.426+0000 7f02b7e00640 1 -- 172.21.15.27:0/2221940457 learned_addr learned my addr 172.21.15.27:0/2221940457 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.426+0000 7f02bd800640 1 --2- 172.21.15.27:0/2221940457 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f02b815b310 0x7f02b81f67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.426+0000 7f02bce00640 1 --2- 172.21.15.27:0/2221940457 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f02b80afd00 0x7f02b81583a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.426+0000 7f02b7e00640 1 -- 172.21.15.27:0/2221940457 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f02b80afd00 msgr2=0x7f02b81583a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:45.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.426+0000 7f02b7e00640 1 --2- 172.21.15.27:0/2221940457 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f02b80afd00 0x7f02b81583a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.430+0000 7f02b7e00640 1 -- 172.21.15.27:0/2221940457 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f02b815b310 msgr2=0x7f02b81f67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.430+0000 7f02b7e00640 1 --2- 172.21.15.27:0/2221940457 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f02b815b310 0x7f02b81f67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.430+0000 7f02b7e00640 1 -- 172.21.15.27:0/2221940457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a8009420 con 0x7f02b80cdf90 2024-01-15T18:05:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.430+0000 7f02b7e00640 1 --2- 172.21.15.27:0/2221940457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b80cdf90 0x7f02b81f6290 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f02a000c3e0 tx=0x7f02a000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.430+0000 7f02b4c00640 1 -- 172.21.15.27:0/2221940457 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a00054e0 con 0x7f02b80cdf90 2024-01-15T18:05:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.430+0000 7f02b4c00640 1 -- 172.21.15.27:0/2221940457 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a0003600 con 0x7f02b80cdf90 2024-01-15T18:05:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.430+0000 7f02b4c00640 1 -- 172.21.15.27:0/2221940457 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a0003a30 con 0x7f02b80cdf90 2024-01-15T18:05:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.430+0000 7f02bf800640 1 -- 172.21.15.27:0/2221940457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b80ca6f0 con 0x7f02b80cdf90 2024-01-15T18:05:45.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.430+0000 7f02bf800640 1 -- 172.21.15.27:0/2221940457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b814db50 con 0x7f02b80cdf90 2024-01-15T18:05:45.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.430+0000 7f02bf800640 1 -- 172.21.15.27:0/2221940457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b80af7a0 con 0x7f02b80cdf90 2024-01-15T18:05:45.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.430+0000 7f02b4c00640 1 -- 172.21.15.27:0/2221940457 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f02a0003b90 con 0x7f02b80cdf90 2024-01-15T18:05:45.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.430+0000 7f02b4c00640 1 --2- 172.21.15.27:0/2221940457 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0290066920 0x7f0290068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:45.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.430+0000 7f02b4c00640 1 -- 172.21.15.27:0/2221940457 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f02a00834f0 con 0x7f02b80cdf90 2024-01-15T18:05:45.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.434+0000 7f02bce00640 1 --2- 172.21.15.27:0/2221940457 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0290066920 0x7f0290068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:45.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.434+0000 7f02bce00640 1 --2- 172.21.15.27:0/2221940457 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0290066920 0x7f0290068de0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f02b80b0730 tx=0x7f02a80a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:45.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.434+0000 7f02b4c00640 1 -- 172.21.15.27:0/2221940457 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f02a0050740 con 0x7f02b80cdf90 2024-01-15T18:05:45.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.590+0000 7f02bf800640 1 -- 172.21.15.27:0/2221940457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f02b804e850 con 0x7f02b80cdf90 2024-01-15T18:05:45.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.590+0000 7f02b4c00640 1 -- 172.21.15.27:0/2221940457 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f02a0053d60 con 0x7f02b80cdf90 2024-01-15T18:05:45.599 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-15T18:05:45.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.594+0000 7f02bf800640 1 -- 172.21.15.27:0/2221940457 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0290066920 msgr2=0x7f0290068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:45.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.594+0000 7f02bf800640 1 --2- 172.21.15.27:0/2221940457 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0290066920 0x7f0290068de0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f02b80b0730 tx=0x7f02a80a6000 comp rx=0 tx=0).stop 2024-01-15T18:05:45.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.594+0000 7f02bf800640 1 -- 172.21.15.27:0/2221940457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b80cdf90 msgr2=0x7f02b81f6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:45.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.594+0000 7f02bf800640 1 --2- 172.21.15.27:0/2221940457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b80cdf90 0x7f02b81f6290 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f02a000c3e0 tx=0x7f02a000c8a0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.594+0000 7f02bf800640 1 -- 172.21.15.27:0/2221940457 shutdown_connections 2024-01-15T18:05:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.594+0000 7f02bf800640 1 --2- 172.21.15.27:0/2221940457 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0290066920 0x7f0290068de0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.594+0000 7f02bf800640 1 --2- 172.21.15.27:0/2221940457 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f02b815b310 0x7f02b81f67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.594+0000 7f02bf800640 1 --2- 172.21.15.27:0/2221940457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02b80cdf90 0x7f02b81f6290 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.594+0000 7f02bf800640 1 --2- 172.21.15.27:0/2221940457 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f02b80afd00 0x7f02b81583a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.594+0000 7f02bf800640 1 -- 172.21.15.27:0/2221940457 >> 172.21.15.27:0/2221940457 conn(0x7f02b80d0e70 msgr2=0x7f02b806eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.594+0000 7f02bf800640 1 -- 172.21.15.27:0/2221940457 shutdown_connections 2024-01-15T18:05:45.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.594+0000 7f02bf800640 1 -- 172.21.15.27:0/2221940457 wait complete. 2024-01-15T18:05:45.621 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:05:45.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.758+0000 7fc4bc800640 1 Processor -- start 2024-01-15T18:05:45.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.758+0000 7fc4bc800640 1 -- start start 2024-01-15T18:05:45.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.758+0000 7fc4bc800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc4b415ac50 0x7fc4b40b2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:45.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.758+0000 7fc4bc800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fc4b40b2ba0 0x7fc4b4071020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:45.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.758+0000 7fc4bc800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4b4070b90 con 0x7fc4b4071560 2024-01-15T18:05:45.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.758+0000 7fc4bc800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4b4072020 con 0x7fc4b415ac50 2024-01-15T18:05:45.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.758+0000 7fc4bc800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4b4072300 con 0x7fc4b40b2ba0 2024-01-15T18:05:45.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.758+0000 7fc4b9e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc4b415ac50 0x7fc4b40b2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:45.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.758+0000 7fc4ba800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc4b4071560 0x7fc4b406c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41844/0 (socket says 172.21.15.27:41844) 2024-01-15T18:05:45.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.758+0000 7fc4ba800640 1 -- 172.21.15.27:0/70231042 learned_addr learned my addr 172.21.15.27:0/70231042 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:45.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.758+0000 7fc4b9400640 1 --2- 172.21.15.27:0/70231042 >> v2:172.21.15.93:3300/0 conn(0x7fc4b40b2ba0 0x7fc4b4071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:45.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.758+0000 7fc4b9e00640 1 -- 172.21.15.27:0/70231042 >> v2:172.21.15.93:3300/0 conn(0x7fc4b40b2ba0 msgr2=0x7fc4b4071020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:45.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.758+0000 7fc4b9e00640 1 --2- 172.21.15.27:0/70231042 >> v2:172.21.15.93:3300/0 conn(0x7fc4b40b2ba0 0x7fc4b4071020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.758+0000 7fc4b9e00640 1 -- 172.21.15.27:0/70231042 >> v1:172.21.15.27:6789/0 conn(0x7fc4b4071560 legacy=0x7fc4b406c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:45.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.758+0000 7fc4b9e00640 1 -- 172.21.15.27:0/70231042 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b4070820 con 0x7fc4b415ac50 2024-01-15T18:05:45.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.758+0000 7fc4b9400640 1 --2- 172.21.15.27:0/70231042 >> v2:172.21.15.93:3300/0 conn(0x7fc4b40b2ba0 0x7fc4b4071020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:45.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.758+0000 7fc4b8a00640 1 -- 172.21.15.27:0/70231042 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779430515 0 0) 0x7fc4b4070b90 con 0x7fc4b4071560 2024-01-15T18:05:45.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.762+0000 7fc4b9e00640 1 --2- 172.21.15.27:0/70231042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4b415ac50 0x7fc4b40b2660 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc4b40cfc70 tx=0x7fc4a401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5871b93142a53e21 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:45.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.762+0000 7fc4b8a00640 1 -- 172.21.15.27:0/70231042 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a40ab600 con 0x7fc4b415ac50 2024-01-15T18:05:45.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.762+0000 7fc4b8a00640 1 -- 172.21.15.27:0/70231042 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a40a7050 con 0x7fc4b415ac50 2024-01-15T18:05:45.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.762+0000 7fc4b8a00640 1 -- 172.21.15.27:0/70231042 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a40ab980 con 0x7fc4b415ac50 2024-01-15T18:05:45.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.762+0000 7fc4bc800640 1 -- 172.21.15.27:0/70231042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4b415ac50 msgr2=0x7fc4b40b2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:45.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.762+0000 7fc4bc800640 1 --2- 172.21.15.27:0/70231042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4b415ac50 0x7fc4b40b2660 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc4b40cfc70 tx=0x7fc4a401c460 comp rx=0 tx=0).stop 2024-01-15T18:05:45.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.762+0000 7fc4bc800640 1 -- 172.21.15.27:0/70231042 shutdown_connections 2024-01-15T18:05:45.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.762+0000 7fc4bc800640 1 --2- 172.21.15.27:0/70231042 >> v2:172.21.15.93:3300/0 conn(0x7fc4b40b2ba0 0x7fc4b4071020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.762+0000 7fc4bc800640 1 --2- 172.21.15.27:0/70231042 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4b415ac50 0x7fc4b40b2660 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.762+0000 7fc4bc800640 1 -- 172.21.15.27:0/70231042 >> 172.21.15.27:0/70231042 conn(0x7fc4b40b0850 msgr2=0x7fc4b40b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:45.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.762+0000 7fc4bc800640 1 -- 172.21.15.27:0/70231042 shutdown_connections 2024-01-15T18:05:45.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.762+0000 7fc4bc800640 1 -- 172.21.15.27:0/70231042 wait complete. 2024-01-15T18:05:45.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.766+0000 7fc4bc800640 1 Processor -- start 2024-01-15T18:05:45.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.766+0000 7fc4bc800640 1 -- start start 2024-01-15T18:05:45.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.766+0000 7fc4bc800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4b4071560 0x7fc4b40cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:45.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.766+0000 7fc4bc800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc4b40b2ba0 0x7fc4b414a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:45.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.766+0000 7fc4bc800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc4b415ac50 0x7fc4b40cbad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:45.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.766+0000 7fc4bc800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b414b0a0 con 0x7fc4b4071560 2024-01-15T18:05:45.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.766+0000 7fc4bc800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b415b3e0 con 0x7fc4b40b2ba0 2024-01-15T18:05:45.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.766+0000 7fc4bc800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b415b6c0 con 0x7fc4b415ac50 2024-01-15T18:05:45.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.766+0000 7fc4b9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4b4071560 0x7fc4b40cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:45.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.766+0000 7fc4b9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4b4071560 0x7fc4b40cdb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38846/0 (socket says 172.21.15.27:38846) 2024-01-15T18:05:45.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.766+0000 7fc4b9e00640 1 -- 172.21.15.27:0/731167288 learned_addr learned my addr 172.21.15.27:0/731167288 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:45.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.766+0000 7fc4b9400640 1 --2- 172.21.15.27:0/731167288 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc4b40b2ba0 0x7fc4b414a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:45.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.766+0000 7fc4ba800640 1 --2- 172.21.15.27:0/731167288 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc4b415ac50 0x7fc4b40cbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:45.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.766+0000 7fc4b9e00640 1 -- 172.21.15.27:0/731167288 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc4b415ac50 msgr2=0x7fc4b40cbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:45.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.766+0000 7fc4b9e00640 1 --2- 172.21.15.27:0/731167288 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc4b415ac50 0x7fc4b40cbad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.766+0000 7fc4b9e00640 1 -- 172.21.15.27:0/731167288 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc4b40b2ba0 msgr2=0x7fc4b414a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:45.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.766+0000 7fc4b9e00640 1 --2- 172.21.15.27:0/731167288 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc4b40b2ba0 0x7fc4b414a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.766+0000 7fc4b9e00640 1 -- 172.21.15.27:0/731167288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a4009420 con 0x7fc4b4071560 2024-01-15T18:05:45.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.766+0000 7fc4b9400640 1 --2- 172.21.15.27:0/731167288 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc4b40b2ba0 0x7fc4b414a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:45.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.766+0000 7fc4b9e00640 1 --2- 172.21.15.27:0/731167288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4b4071560 0x7fc4b40cdb90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc4a40ae040 tx=0x7fc4a401aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:45.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.766+0000 7fc4b2000640 1 -- 172.21.15.27:0/731167288 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a40b7650 con 0x7fc4b4071560 2024-01-15T18:05:45.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.766+0000 7fc4b2000640 1 -- 172.21.15.27:0/731167288 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a40b77b0 con 0x7fc4b4071560 2024-01-15T18:05:45.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.766+0000 7fc4b2000640 1 -- 172.21.15.27:0/731167288 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a40b7b60 con 0x7fc4b4071560 2024-01-15T18:05:45.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.766+0000 7fc4bc800640 1 -- 172.21.15.27:0/731167288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4b415b9a0 con 0x7fc4b4071560 2024-01-15T18:05:45.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.766+0000 7fc4bc800640 1 -- 172.21.15.27:0/731167288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4b40ce0d0 con 0x7fc4b4071560 2024-01-15T18:05:45.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.770+0000 7fc4bc800640 1 -- 172.21.15.27:0/731167288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4b4071020 con 0x7fc4b4071560 2024-01-15T18:05:45.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.770+0000 7fc4b2000640 1 -- 172.21.15.27:0/731167288 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fc4a40b7cc0 con 0x7fc4b4071560 2024-01-15T18:05:45.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.770+0000 7fc4b2000640 1 --2- 172.21.15.27:0/731167288 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc48c0668f0 0x7fc48c068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:45.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.770+0000 7fc4b2000640 1 -- 172.21.15.27:0/731167288 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fc4a411ed50 con 0x7fc4b4071560 2024-01-15T18:05:45.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.770+0000 7fc4b9400640 1 --2- 172.21.15.27:0/731167288 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc48c0668f0 0x7fc48c068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:45.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.770+0000 7fc4b9400640 1 --2- 172.21.15.27:0/731167288 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc48c0668f0 0x7fc48c068db0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc4b40b15b0 tx=0x7fc49c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:45.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.774+0000 7fc4b2000640 1 -- 172.21.15.27:0/731167288 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc4a40ebef0 con 0x7fc4b4071560 2024-01-15T18:05:45.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.946+0000 7fc4bc800640 1 -- 172.21.15.27:0/731167288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc4b404e850 con 0x7fc4b4071560 2024-01-15T18:05:45.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.946+0000 7fc4b2000640 1 -- 172.21.15.27:0/731167288 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc4a40b0080 con 0x7fc4b4071560 2024-01-15T18:05:45.952 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-01-15T18:05:45.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.946+0000 7fc4bc800640 1 -- 172.21.15.27:0/731167288 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc48c0668f0 msgr2=0x7fc48c068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:45.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.946+0000 7fc4bc800640 1 --2- 172.21.15.27:0/731167288 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc48c0668f0 0x7fc48c068db0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc4b40b15b0 tx=0x7fc49c008040 comp rx=0 tx=0).stop 2024-01-15T18:05:45.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.946+0000 7fc4bc800640 1 -- 172.21.15.27:0/731167288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4b4071560 msgr2=0x7fc4b40cdb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:45.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.946+0000 7fc4bc800640 1 --2- 172.21.15.27:0/731167288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4b4071560 0x7fc4b40cdb90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc4a40ae040 tx=0x7fc4a401aca0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.946+0000 7fc4bc800640 1 -- 172.21.15.27:0/731167288 shutdown_connections 2024-01-15T18:05:45.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.946+0000 7fc4bc800640 1 --2- 172.21.15.27:0/731167288 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc48c0668f0 0x7fc48c068db0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.946+0000 7fc4bc800640 1 --2- 172.21.15.27:0/731167288 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc4b415ac50 0x7fc4b40cbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.946+0000 7fc4bc800640 1 --2- 172.21.15.27:0/731167288 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc4b40b2ba0 0x7fc4b414a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.946+0000 7fc4bc800640 1 --2- 172.21.15.27:0/731167288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4b4071560 0x7fc4b40cdb90 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.946+0000 7fc4bc800640 1 -- 172.21.15.27:0/731167288 >> 172.21.15.27:0/731167288 conn(0x7fc4b40b0850 msgr2=0x7fc4b40ae010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:45.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.946+0000 7fc4bc800640 1 -- 172.21.15.27:0/731167288 shutdown_connections 2024-01-15T18:05:45.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:45.946+0000 7fc4bc800640 1 -- 172.21.15.27:0/731167288 wait complete. 2024-01-15T18:05:45.975 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi093.front.sepia.ceph.com... 2024-01-15T18:05:45.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:45.976 DEBUG:teuthology.orchestra.run.smithi093:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-01-15T18:05:45.999 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:05:45.999 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T18:05:45.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:45.999 DEBUG:teuthology.orchestra.run.smithi093:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-15T18:05:46.052 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-15T18:05:46.052 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:05:46.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.186+0000 7f95a7800640 1 Processor -- start 2024-01-15T18:05:46.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.186+0000 7f95a7800640 1 -- start start 2024-01-15T18:05:46.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.190+0000 7f95a7800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f95a00cdd10 0x7f95a0070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.190+0000 7f95a7800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a00ce920 con 0x7f95a006db80 2024-01-15T18:05:46.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.190+0000 7f95a7800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a00ce640 con 0x7f95a015e730 2024-01-15T18:05:46.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.190+0000 7f95a7800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95a00cdb30 con 0x7f95a00cdd10 2024-01-15T18:05:46.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.190+0000 7f95a5800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f95a00cdd10 0x7f95a0070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.190+0000 7f95a4e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f95a006db80 0x7f95a006df60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41860/0 (socket says 172.21.15.27:41860) 2024-01-15T18:05:46.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.190+0000 7f95a4e00640 1 -- 172.21.15.27:0/446274561 learned_addr learned my addr 172.21.15.27:0/446274561 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:46.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.190+0000 7f95a5800640 1 -- 172.21.15.27:0/446274561 >> v1:172.21.15.93:6789/0 conn(0x7f95a015e730 legacy=0x7f95a00cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:46.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.190+0000 7f95a5800640 1 -- 172.21.15.27:0/446274561 >> v1:172.21.15.27:6789/0 conn(0x7f95a006db80 legacy=0x7f95a006df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:46.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.190+0000 7f95a5800640 1 -- 172.21.15.27:0/446274561 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a00cc940 con 0x7f95a00cdd10 2024-01-15T18:05:46.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.190+0000 7f959f400640 1 -- 172.21.15.27:0/446274561 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163194944 0 0) 0x7f95a00ce920 con 0x7f95a006db80 2024-01-15T18:05:46.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.194+0000 7f95a5800640 1 --2- 172.21.15.27:0/446274561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95a00cdd10 0x7f95a0070c20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9598009d90 tx=0x7f959801c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ca4af66cd5d2a47 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:46.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.194+0000 7f959f400640 1 -- 172.21.15.27:0/446274561 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95980a59b0 con 0x7f95a00cdd10 2024-01-15T18:05:46.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.194+0000 7f959f400640 1 -- 172.21.15.27:0/446274561 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95980a5b10 con 0x7f95a00cdd10 2024-01-15T18:05:46.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.194+0000 7f959f400640 1 -- 172.21.15.27:0/446274561 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95980a5ea0 con 0x7f95a00cdd10 2024-01-15T18:05:46.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.194+0000 7f95a7800640 1 -- 172.21.15.27:0/446274561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95a00cdd10 msgr2=0x7f95a0070c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:46.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.194+0000 7f95a7800640 1 --2- 172.21.15.27:0/446274561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95a00cdd10 0x7f95a0070c20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9598009d90 tx=0x7f959801c920 comp rx=0 tx=0).stop 2024-01-15T18:05:46.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.194+0000 7f95a7800640 1 -- 172.21.15.27:0/446274561 shutdown_connections 2024-01-15T18:05:46.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.194+0000 7f95a7800640 1 --2- 172.21.15.27:0/446274561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95a00cdd10 0x7f95a0070c20 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.194+0000 7f95a7800640 1 -- 172.21.15.27:0/446274561 >> 172.21.15.27:0/446274561 conn(0x7f95a00cec60 msgr2=0x7f95a00cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:46.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.194+0000 7f95a7800640 1 -- 172.21.15.27:0/446274561 shutdown_connections 2024-01-15T18:05:46.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.194+0000 7f95a7800640 1 -- 172.21.15.27:0/446274561 wait complete. 2024-01-15T18:05:46.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.194+0000 7f95a7800640 1 Processor -- start 2024-01-15T18:05:46.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.194+0000 7f95a7800640 1 -- start start 2024-01-15T18:05:46.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.198+0000 7f95a7800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95a006db80 0x7f95a00c9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.198+0000 7f95a7800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f95a00cdd10 0x7f95a00c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.198+0000 7f95a7800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95a015e730 0x7f95a00b3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.198+0000 7f95a7800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a00b1970 con 0x7f95a006db80 2024-01-15T18:05:46.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.198+0000 7f95a7800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a0153440 con 0x7f95a015e730 2024-01-15T18:05:46.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.198+0000 7f95a7800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a0153720 con 0x7f95a00cdd10 2024-01-15T18:05:46.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.198+0000 7f95a4e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95a006db80 0x7f95a00c9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.198+0000 7f95a4e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95a006db80 0x7f95a00c9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38862/0 (socket says 172.21.15.27:38862) 2024-01-15T18:05:46.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.198+0000 7f95a4e00640 1 -- 172.21.15.27:0/3572610110 learned_addr learned my addr 172.21.15.27:0/3572610110 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:46.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.198+0000 7f95a5800640 1 --2- 172.21.15.27:0/3572610110 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95a015e730 0x7f95a00b3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.198+0000 7f959fe00640 1 --2- 172.21.15.27:0/3572610110 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f95a00cdd10 0x7f95a00c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.198+0000 7f95a4e00640 1 -- 172.21.15.27:0/3572610110 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f95a00cdd10 msgr2=0x7f95a00c7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:46.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.198+0000 7f95a4e00640 1 --2- 172.21.15.27:0/3572610110 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f95a00cdd10 0x7f95a00c7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.198+0000 7f95a4e00640 1 -- 172.21.15.27:0/3572610110 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95a015e730 msgr2=0x7f95a00b3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:46.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.198+0000 7f95a4e00640 1 --2- 172.21.15.27:0/3572610110 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95a015e730 0x7f95a00b3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.198+0000 7f95a4e00640 1 -- 172.21.15.27:0/3572610110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9598009a70 con 0x7f95a006db80 2024-01-15T18:05:46.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.198+0000 7f95a4e00640 1 --2- 172.21.15.27:0/3572610110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95a006db80 0x7f95a00c9ad0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f959000c4a0 tx=0x7f959000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:46.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.198+0000 7f959cc00640 1 -- 172.21.15.27:0/3572610110 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95900074e0 con 0x7f95a006db80 2024-01-15T18:05:46.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.198+0000 7f959cc00640 1 -- 172.21.15.27:0/3572610110 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9590007640 con 0x7f95a006db80 2024-01-15T18:05:46.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.198+0000 7f959cc00640 1 -- 172.21.15.27:0/3572610110 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9590004a30 con 0x7f95a006db80 2024-01-15T18:05:46.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.198+0000 7f95a7800640 1 -- 172.21.15.27:0/3572610110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a0153a00 con 0x7f95a006db80 2024-01-15T18:05:46.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.198+0000 7f95a7800640 1 -- 172.21.15.27:0/3572610110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a0153fc0 con 0x7f95a006db80 2024-01-15T18:05:46.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.198+0000 7f95a7800640 1 -- 172.21.15.27:0/3572610110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a004e850 con 0x7f95a006db80 2024-01-15T18:05:46.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.202+0000 7f959cc00640 1 -- 172.21.15.27:0/3572610110 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f959001b360 con 0x7f95a006db80 2024-01-15T18:05:46.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.202+0000 7f959cc00640 1 --2- 172.21.15.27:0/3572610110 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f95780668f0 0x7f9578068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.202+0000 7f959cc00640 1 -- 172.21.15.27:0/3572610110 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f9590083a50 con 0x7f95a006db80 2024-01-15T18:05:46.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.202+0000 7f959fe00640 1 --2- 172.21.15.27:0/3572610110 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f95780668f0 0x7f9578068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.202+0000 7f959fe00640 1 --2- 172.21.15.27:0/3572610110 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f95780668f0 0x7f9578068db0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f95a00ae770 tx=0x7f9588009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:46.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.206+0000 7f959cc00640 1 -- 172.21.15.27:0/3572610110 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9590050d20 con 0x7f95a006db80 2024-01-15T18:05:46.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.402+0000 7f95a7800640 1 -- 172.21.15.27:0/3572610110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95a006c800 con 0x7f95a006db80 2024-01-15T18:05:46.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.402+0000 7f959cc00640 1 -- 172.21.15.27:0/3572610110 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7f9590018020 con 0x7f95a006db80 2024-01-15T18:05:46.409 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:46.409 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/393360880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":393360880},{"type":"v1","addr":"172.21.15.93:6836","nonce":393360880}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4229,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4063308188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4063308188},{"type":"v1","addr":"172.21.15.27:6835","nonce":4063308188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4305,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/2343128056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":2343128056},{"type":"v1","addr":"172.21.15.93:6837","nonce":2343128056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:04:58.208165+0000","modified":"2024-01-15T18:05:01.225419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2657553192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2657553192},{"type":"v1","addr":"172.21.15.27:6834","nonce":2657553192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T18:05:46.410 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 8 2024-01-15T18:05:46.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.406+0000 7f95a7800640 1 -- 172.21.15.27:0/3572610110 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f95780668f0 msgr2=0x7f9578068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:46.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.406+0000 7f95a7800640 1 --2- 172.21.15.27:0/3572610110 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f95780668f0 0x7f9578068db0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f95a00ae770 tx=0x7f9588009040 comp rx=0 tx=0).stop 2024-01-15T18:05:46.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.406+0000 7f95a7800640 1 -- 172.21.15.27:0/3572610110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95a006db80 msgr2=0x7f95a00c9ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:46.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.406+0000 7f95a7800640 1 --2- 172.21.15.27:0/3572610110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95a006db80 0x7f95a00c9ad0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f959000c4a0 tx=0x7f959000c960 comp rx=0 tx=0).stop 2024-01-15T18:05:46.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.406+0000 7f95a7800640 1 -- 172.21.15.27:0/3572610110 shutdown_connections 2024-01-15T18:05:46.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.406+0000 7f95a7800640 1 --2- 172.21.15.27:0/3572610110 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f95780668f0 0x7f9578068db0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.406+0000 7f95a7800640 1 --2- 172.21.15.27:0/3572610110 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95a015e730 0x7f95a00b3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.406+0000 7f95a7800640 1 --2- 172.21.15.27:0/3572610110 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f95a00cdd10 0x7f95a00c7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.406+0000 7f95a7800640 1 --2- 172.21.15.27:0/3572610110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95a006db80 0x7f95a00c9ad0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.406+0000 7f95a7800640 1 -- 172.21.15.27:0/3572610110 >> 172.21.15.27:0/3572610110 conn(0x7f95a00cec60 msgr2=0x7f95a006ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:46.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.406+0000 7f95a7800640 1 -- 172.21.15.27:0/3572610110 shutdown_connections 2024-01-15T18:05:46.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.406+0000 7f95a7800640 1 -- 172.21.15.27:0/3572610110 wait complete. 2024-01-15T18:05:46.432 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:46.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.566+0000 7f788f000640 1 Processor -- start 2024-01-15T18:05:46.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.566+0000 7f788f000640 1 -- start start 2024-01-15T18:05:46.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.566+0000 7f788f000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f7888160af0 0x7f788806b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.566+0000 7f788f000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f78880cfaa0 0x7f788806bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.566+0000 7f788f000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f788806c080 0x7f7888161210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.566+0000 7f788f000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78880ae280 con 0x7f78880cfaa0 2024-01-15T18:05:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.566+0000 7f788f000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78880adc50 con 0x7f7888160af0 2024-01-15T18:05:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.566+0000 7f788f000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78880cff80 con 0x7f788806c080 2024-01-15T18:05:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.570+0000 7f7887400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f78880cfaa0 0x7f788806bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.570+0000 7f7887400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f78880cfaa0 0x7f788806bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38872/0 (socket says 172.21.15.27:38872) 2024-01-15T18:05:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.570+0000 7f7887400640 1 -- 172.21.15.27:0/194461906 learned_addr learned my addr 172.21.15.27:0/194461906 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:46.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.570+0000 7f7887e00640 1 --2- 172.21.15.27:0/194461906 >> v2:172.21.15.93:3300/0 conn(0x7f7888160af0 0x7f788806b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.570+0000 7f788d000640 1 --2- 172.21.15.27:0/194461906 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f788806c080 0x7f7888161210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.570+0000 7f7887400640 1 -- 172.21.15.27:0/194461906 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f788806c080 msgr2=0x7f7888161210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.570+0000 7f7887400640 1 --2- 172.21.15.27:0/194461906 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f788806c080 0x7f7888161210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.570+0000 7f7887400640 1 -- 172.21.15.27:0/194461906 >> v2:172.21.15.93:3300/0 conn(0x7f7888160af0 msgr2=0x7f788806b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.570+0000 7f7887400640 1 --2- 172.21.15.27:0/194461906 >> v2:172.21.15.93:3300/0 conn(0x7f7888160af0 0x7f788806b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.570+0000 7f7887400640 1 -- 172.21.15.27:0/194461906 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78880d0260 con 0x7f78880cfaa0 2024-01-15T18:05:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.570+0000 7f7887e00640 1 --2- 172.21.15.27:0/194461906 >> v2:172.21.15.93:3300/0 conn(0x7f7888160af0 0x7f788806b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:46.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.570+0000 7f7887400640 1 --2- 172.21.15.27:0/194461906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78880cfaa0 0x7f788806bb40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7888071650 tx=0x7f787001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50b2f794686de7cd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:46.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.570+0000 7f7886a00640 1 -- 172.21.15.27:0/194461906 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78700ab600 con 0x7f78880cfaa0 2024-01-15T18:05:46.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.570+0000 7f7886a00640 1 -- 172.21.15.27:0/194461906 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78700a7050 con 0x7f78880cfaa0 2024-01-15T18:05:46.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.570+0000 7f7886a00640 1 -- 172.21.15.27:0/194461906 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78700ab980 con 0x7f78880cfaa0 2024-01-15T18:05:46.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.574+0000 7f788f000640 1 -- 172.21.15.27:0/194461906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78880cfaa0 msgr2=0x7f788806bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:46.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.574+0000 7f788f000640 1 --2- 172.21.15.27:0/194461906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78880cfaa0 0x7f788806bb40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7888071650 tx=0x7f787001c500 comp rx=0 tx=0).stop 2024-01-15T18:05:46.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.574+0000 7f788f000640 1 -- 172.21.15.27:0/194461906 shutdown_connections 2024-01-15T18:05:46.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.574+0000 7f788f000640 1 --2- 172.21.15.27:0/194461906 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f788806c080 0x7f7888161210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.574+0000 7f788f000640 1 --2- 172.21.15.27:0/194461906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78880cfaa0 0x7f788806bb40 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.574+0000 7f788f000640 1 --2- 172.21.15.27:0/194461906 >> v2:172.21.15.93:3300/0 conn(0x7f7888160af0 0x7f788806b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.574+0000 7f788f000640 1 -- 172.21.15.27:0/194461906 >> 172.21.15.27:0/194461906 conn(0x7f78880d0f20 msgr2=0x7f78880d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:46.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.574+0000 7f788f000640 1 -- 172.21.15.27:0/194461906 shutdown_connections 2024-01-15T18:05:46.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.574+0000 7f788f000640 1 -- 172.21.15.27:0/194461906 wait complete. 2024-01-15T18:05:46.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.574+0000 7f788f000640 1 Processor -- start 2024-01-15T18:05:46.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.574+0000 7f788f000640 1 -- start start 2024-01-15T18:05:46.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.574+0000 7f788f000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f788806c080 0x7f7888159510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.574+0000 7f788f000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78880cfaa0 0x7f788809d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.574+0000 7f788f000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7888160af0 0x7f788809b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.574+0000 7f788f000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788815af90 con 0x7f7888160af0 2024-01-15T18:05:46.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.574+0000 7f788f000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7888159a50 con 0x7f78880cfaa0 2024-01-15T18:05:46.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.574+0000 7f788f000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f788809d680 con 0x7f788806c080 2024-01-15T18:05:46.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.578+0000 7f7887400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78880cfaa0 0x7f788809d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.578+0000 7f788d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7888160af0 0x7f788809b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.578+0000 7f7887e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f788806c080 0x7f7888159510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.578+0000 7f788d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7888160af0 0x7f788809b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38888/0 (socket says 172.21.15.27:38888) 2024-01-15T18:05:46.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.578+0000 7f7887400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78880cfaa0 0x7f788809d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:39414/0 (socket says 172.21.15.27:39414) 2024-01-15T18:05:46.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.578+0000 7f788d000640 1 -- 172.21.15.27:0/1249270793 learned_addr learned my addr 172.21.15.27:0/1249270793 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:46.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.578+0000 7f788d000640 1 -- 172.21.15.27:0/1249270793 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f788806c080 msgr2=0x7f7888159510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:46.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.578+0000 7f788d000640 1 --2- 172.21.15.27:0/1249270793 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f788806c080 0x7f7888159510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.578+0000 7f788d000640 1 -- 172.21.15.27:0/1249270793 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78880cfaa0 msgr2=0x7f788809d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:46.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.578+0000 7f788d000640 1 --2- 172.21.15.27:0/1249270793 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78880cfaa0 0x7f788809d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.578+0000 7f788d000640 1 -- 172.21.15.27:0/1249270793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7878009420 con 0x7f7888160af0 2024-01-15T18:05:46.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.578+0000 7f7887400640 1 --2- 172.21.15.27:0/1249270793 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78880cfaa0 0x7f788809d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:46.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.578+0000 7f7887e00640 1 --2- 172.21.15.27:0/1249270793 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f788806c080 0x7f7888159510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:46.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.578+0000 7f788d000640 1 --2- 172.21.15.27:0/1249270793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7888160af0 0x7f788809b090 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f788000eed0 tx=0x7f788000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.578+0000 7f787fe00640 1 -- 172.21.15.27:0/1249270793 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7880010040 con 0x7f7888160af0 2024-01-15T18:05:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.578+0000 7f787fe00640 1 -- 172.21.15.27:0/1249270793 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7880014400 con 0x7f7888160af0 2024-01-15T18:05:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.578+0000 7f788f000640 1 -- 172.21.15.27:0/1249270793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78700094d0 con 0x7f7888160af0 2024-01-15T18:05:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.578+0000 7f787fe00640 1 -- 172.21.15.27:0/1249270793 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7880009ea0 con 0x7f7888160af0 2024-01-15T18:05:46.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.578+0000 7f788f000640 1 -- 172.21.15.27:0/1249270793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f788809dc20 con 0x7f7888160af0 2024-01-15T18:05:46.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.578+0000 7f788f000640 1 -- 172.21.15.27:0/1249270793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f788809e700 con 0x7f7888160af0 2024-01-15T18:05:46.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.578+0000 7f787fe00640 1 -- 172.21.15.27:0/1249270793 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f78800147b0 con 0x7f7888160af0 2024-01-15T18:05:46.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.582+0000 7f787fe00640 1 --2- 172.21.15.27:0/1249270793 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7860066920 0x7f7860068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.582+0000 7f787fe00640 1 -- 172.21.15.27:0/1249270793 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f7880084970 con 0x7f7888160af0 2024-01-15T18:05:46.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.582+0000 7f7887e00640 1 --2- 172.21.15.27:0/1249270793 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7860066920 0x7f7860068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.582+0000 7f7887e00640 1 --2- 172.21.15.27:0/1249270793 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7860066920 0x7f7860068de0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f78880cb3c0 tx=0x7f7878009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:46.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.586+0000 7f787fe00640 1 -- 172.21.15.27:0/1249270793 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f788809e700 con 0x7f7888160af0 2024-01-15T18:05:46.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.742+0000 7f788f000640 1 -- 172.21.15.27:0/1249270793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f788809e700 con 0x7f7888160af0 2024-01-15T18:05:46.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.742+0000 7f787fe00640 1 -- 172.21.15.27:0/1249270793 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16705 (secure 0 0 0) 0x7f788809e700 con 0x7f7888160af0 2024-01-15T18:05:46.750 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:46.750 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":23,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:05:40.008536+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:04:56.220608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T18:04:57.210421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:46.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.746+0000 7f788f000640 1 -- 172.21.15.27:0/1249270793 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7860066920 msgr2=0x7f7860068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:46.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.746+0000 7f788f000640 1 --2- 172.21.15.27:0/1249270793 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7860066920 0x7f7860068de0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f78880cb3c0 tx=0x7f7878009190 comp rx=0 tx=0).stop 2024-01-15T18:05:46.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.746+0000 7f788f000640 1 -- 172.21.15.27:0/1249270793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7888160af0 msgr2=0x7f788809b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:46.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.746+0000 7f788f000640 1 --2- 172.21.15.27:0/1249270793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7888160af0 0x7f788809b090 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f788000eed0 tx=0x7f788000b480 comp rx=0 tx=0).stop 2024-01-15T18:05:46.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.746+0000 7f788f000640 1 -- 172.21.15.27:0/1249270793 shutdown_connections 2024-01-15T18:05:46.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.746+0000 7f788f000640 1 --2- 172.21.15.27:0/1249270793 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7860066920 0x7f7860068de0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.746+0000 7f788f000640 1 --2- 172.21.15.27:0/1249270793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7888160af0 0x7f788809b090 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.746+0000 7f788f000640 1 --2- 172.21.15.27:0/1249270793 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f78880cfaa0 0x7f788809d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.746+0000 7f788f000640 1 --2- 172.21.15.27:0/1249270793 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f788806c080 0x7f7888159510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.746+0000 7f788f000640 1 -- 172.21.15.27:0/1249270793 >> 172.21.15.27:0/1249270793 conn(0x7f78880d0f20 msgr2=0x7f788806b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:46.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.746+0000 7f788f000640 1 -- 172.21.15.27:0/1249270793 shutdown_connections 2024-01-15T18:05:46.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.746+0000 7f788f000640 1 -- 172.21.15.27:0/1249270793 wait complete. 2024-01-15T18:05:46.773 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T18:04:56.220608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T18:04:57.210421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:05:46.774 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:05:46.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.914+0000 7ff150e00640 1 Processor -- start 2024-01-15T18:05:46.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.914+0000 7ff150e00640 1 -- start start 2024-01-15T18:05:46.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.914+0000 7ff150e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff14c0cef20 0x7ff14c0ce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.914+0000 7ff150e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff14c0cca40 0x7ff14c0d19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.914+0000 7ff150e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff14c15b6b0 con 0x7ff14c0cf300 2024-01-15T18:05:46.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.914+0000 7ff150e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff14c0cb9e0 con 0x7ff14c0cef20 2024-01-15T18:05:46.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.914+0000 7ff150e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff14c0cbc80 con 0x7ff14c0cca40 2024-01-15T18:05:46.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.914+0000 7ff149e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff14c0cef20 0x7ff14c0ce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.914+0000 7ff149e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff14c0cef20 0x7ff14c0ce2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38896/0 (socket says 172.21.15.27:38896) 2024-01-15T18:05:46.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.914+0000 7ff149e00640 1 -- 172.21.15.27:0/3449843818 learned_addr learned my addr 172.21.15.27:0/3449843818 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:46.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.918+0000 7ff149400640 1 --2- 172.21.15.27:0/3449843818 >> v2:172.21.15.93:3300/0 conn(0x7ff14c0cca40 0x7ff14c0d19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.918+0000 7ff149e00640 1 -- 172.21.15.27:0/3449843818 >> v2:172.21.15.93:3300/0 conn(0x7ff14c0cca40 msgr2=0x7ff14c0d19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:46.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.918+0000 7ff149e00640 1 --2- 172.21.15.27:0/3449843818 >> v2:172.21.15.93:3300/0 conn(0x7ff14c0cca40 0x7ff14c0d19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.918+0000 7ff149e00640 1 -- 172.21.15.27:0/3449843818 >> v1:172.21.15.93:6789/0 conn(0x7ff14c0cf300 legacy=0x7ff14c0b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:46.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.918+0000 7ff149e00640 1 -- 172.21.15.27:0/3449843818 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c0cfdb0 con 0x7ff14c0cef20 2024-01-15T18:05:46.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.918+0000 7ff149400640 1 --2- 172.21.15.27:0/3449843818 >> v2:172.21.15.93:3300/0 conn(0x7ff14c0cca40 0x7ff14c0d19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:46.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.918+0000 7ff149e00640 1 --2- 172.21.15.27:0/3449843818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff14c0cef20 0x7ff14c0ce2c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff14c070e80 tx=0x7ff13c01c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d3ce0a2ba8dcec4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:46.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.918+0000 7ff148a00640 1 -- 172.21.15.27:0/3449843818 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c0ac410 con 0x7ff14c0cef20 2024-01-15T18:05:46.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.918+0000 7ff148a00640 1 -- 172.21.15.27:0/3449843818 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff13c0ac570 con 0x7ff14c0cef20 2024-01-15T18:05:46.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.918+0000 7ff148a00640 1 -- 172.21.15.27:0/3449843818 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c0ac900 con 0x7ff14c0cef20 2024-01-15T18:05:46.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.918+0000 7ff150e00640 1 -- 172.21.15.27:0/3449843818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff14c0cef20 msgr2=0x7ff14c0ce2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:46.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.918+0000 7ff150e00640 1 --2- 172.21.15.27:0/3449843818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff14c0cef20 0x7ff14c0ce2c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff14c070e80 tx=0x7ff13c01c4c0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.922+0000 7ff150e00640 1 -- 172.21.15.27:0/3449843818 shutdown_connections 2024-01-15T18:05:46.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.922+0000 7ff150e00640 1 --2- 172.21.15.27:0/3449843818 >> v2:172.21.15.93:3300/0 conn(0x7ff14c0cca40 0x7ff14c0d19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.922+0000 7ff150e00640 1 --2- 172.21.15.27:0/3449843818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff14c0cef20 0x7ff14c0ce2c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.922+0000 7ff150e00640 1 -- 172.21.15.27:0/3449843818 >> 172.21.15.27:0/3449843818 conn(0x7ff14c0d0ed0 msgr2=0x7ff14c0d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:46.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.922+0000 7ff150e00640 1 -- 172.21.15.27:0/3449843818 shutdown_connections 2024-01-15T18:05:46.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.922+0000 7ff150e00640 1 -- 172.21.15.27:0/3449843818 wait complete. 2024-01-15T18:05:46.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.922+0000 7ff150e00640 1 Processor -- start 2024-01-15T18:05:46.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.922+0000 7ff150e00640 1 -- start start 2024-01-15T18:05:46.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.922+0000 7ff150e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff14c0cca40 0x7ff14c15e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.922+0000 7ff150e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff14c0cef20 0x7ff14c06e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.922+0000 7ff150e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff14c0cf300 0x7ff14c0c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.922+0000 7ff150e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c14e710 con 0x7ff14c0cf300 2024-01-15T18:05:46.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.922+0000 7ff150e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c165a90 con 0x7ff14c0cef20 2024-01-15T18:05:46.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.922+0000 7ff150e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff14c165d70 con 0x7ff14c0cca40 2024-01-15T18:05:46.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.922+0000 7ff14a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff14c0cf300 0x7ff14c0c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.922+0000 7ff14a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff14c0cf300 0x7ff14c0c6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38906/0 (socket says 172.21.15.27:38906) 2024-01-15T18:05:46.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.922+0000 7ff14a800640 1 -- 172.21.15.27:0/190700063 learned_addr learned my addr 172.21.15.27:0/190700063 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:46.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.922+0000 7ff149400640 1 --2- 172.21.15.27:0/190700063 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff14c0cef20 0x7ff14c06e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.922+0000 7ff149e00640 1 --2- 172.21.15.27:0/190700063 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff14c0cca40 0x7ff14c15e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.926+0000 7ff14a800640 1 -- 172.21.15.27:0/190700063 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff14c0cca40 msgr2=0x7ff14c15e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:46.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.926+0000 7ff14a800640 1 --2- 172.21.15.27:0/190700063 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff14c0cca40 0x7ff14c15e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.926+0000 7ff14a800640 1 -- 172.21.15.27:0/190700063 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff14c0cef20 msgr2=0x7ff14c06e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:46.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.926+0000 7ff14a800640 1 --2- 172.21.15.27:0/190700063 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff14c0cef20 0x7ff14c06e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.926+0000 7ff14a800640 1 -- 172.21.15.27:0/190700063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13c0094d0 con 0x7ff14c0cf300 2024-01-15T18:05:46.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.926+0000 7ff14a800640 1 --2- 172.21.15.27:0/190700063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff14c0cf300 0x7ff14c0c6c30 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff14400b970 tx=0x7ff14400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:46.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.926+0000 7ff142000640 1 -- 172.21.15.27:0/190700063 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14400cb40 con 0x7ff14c0cf300 2024-01-15T18:05:46.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.926+0000 7ff142000640 1 -- 172.21.15.27:0/190700063 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14400cca0 con 0x7ff14c0cf300 2024-01-15T18:05:46.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.926+0000 7ff142000640 1 -- 172.21.15.27:0/190700063 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff144014680 con 0x7ff14c0cf300 2024-01-15T18:05:46.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.926+0000 7ff150e00640 1 -- 172.21.15.27:0/190700063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff14c14e9d0 con 0x7ff14c0cf300 2024-01-15T18:05:46.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.926+0000 7ff150e00640 1 -- 172.21.15.27:0/190700063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14c06d1f0 con 0x7ff14c0cf300 2024-01-15T18:05:46.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.926+0000 7ff150e00640 1 -- 172.21.15.27:0/190700063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff14c0d09f0 con 0x7ff14c0cf300 2024-01-15T18:05:46.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.926+0000 7ff142000640 1 -- 172.21.15.27:0/190700063 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7ff14401d3b0 con 0x7ff14c0cf300 2024-01-15T18:05:46.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.926+0000 7ff142000640 1 --2- 172.21.15.27:0/190700063 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff1240668f0 0x7ff124068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.926+0000 7ff142000640 1 -- 172.21.15.27:0/190700063 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7ff144084250 con 0x7ff14c0cf300 2024-01-15T18:05:46.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.930+0000 7ff149e00640 1 --2- 172.21.15.27:0/190700063 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff1240668f0 0x7ff124068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.930+0000 7ff149e00640 1 --2- 172.21.15.27:0/190700063 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff1240668f0 0x7ff124068db0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff13c0ae000 tx=0x7ff13c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:46.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:46.930+0000 7ff142000640 1 -- 172.21.15.27:0/190700063 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff1440514a0 con 0x7ff14c0cf300 2024-01-15T18:05:47.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.090+0000 7ff150e00640 1 -- 172.21.15.27:0/190700063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff14c14eef0 con 0x7ff14c0cf300 2024-01-15T18:05:47.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.090+0000 7ff142000640 1 -- 172.21.15.27:0/190700063 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff14c14eef0 con 0x7ff14c0cf300 2024-01-15T18:05:47.095 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:05:47.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.090+0000 7ff150e00640 1 -- 172.21.15.27:0/190700063 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff1240668f0 msgr2=0x7ff124068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.090+0000 7ff150e00640 1 --2- 172.21.15.27:0/190700063 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff1240668f0 0x7ff124068db0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff13c0ae000 tx=0x7ff13c008040 comp rx=0 tx=0).stop 2024-01-15T18:05:47.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.090+0000 7ff150e00640 1 -- 172.21.15.27:0/190700063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff14c0cf300 msgr2=0x7ff14c0c6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.090+0000 7ff150e00640 1 --2- 172.21.15.27:0/190700063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff14c0cf300 0x7ff14c0c6c30 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff14400b970 tx=0x7ff14400be30 comp rx=0 tx=0).stop 2024-01-15T18:05:47.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.090+0000 7ff150e00640 1 -- 172.21.15.27:0/190700063 shutdown_connections 2024-01-15T18:05:47.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.090+0000 7ff150e00640 1 --2- 172.21.15.27:0/190700063 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff1240668f0 0x7ff124068db0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.090+0000 7ff150e00640 1 --2- 172.21.15.27:0/190700063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff14c0cf300 0x7ff14c0c6c30 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.090+0000 7ff150e00640 1 --2- 172.21.15.27:0/190700063 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff14c0cef20 0x7ff14c06e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.090+0000 7ff150e00640 1 --2- 172.21.15.27:0/190700063 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff14c0cca40 0x7ff14c15e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.090+0000 7ff150e00640 1 -- 172.21.15.27:0/190700063 >> 172.21.15.27:0/190700063 conn(0x7ff14c0d0ed0 msgr2=0x7ff14c0d12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:47.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.090+0000 7ff150e00640 1 -- 172.21.15.27:0/190700063 shutdown_connections 2024-01-15T18:05:47.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.090+0000 7ff150e00640 1 -- 172.21.15.27:0/190700063 wait complete. 2024-01-15T18:05:47.117 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T18:05:47.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.254+0000 7faa10600640 1 Processor -- start 2024-01-15T18:05:47.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.254+0000 7faa10600640 1 -- start start 2024-01-15T18:05:47.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.258+0000 7faa10600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faa0815d190 0x7faa080cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.258+0000 7faa10600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faa0806c520 0x7faa080cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.258+0000 7faa10600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa080b0d70 con 0x7faa0815d570 2024-01-15T18:05:47.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.258+0000 7faa10600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa080d0540 con 0x7faa0815d190 2024-01-15T18:05:47.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.258+0000 7faa10600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa080b0440 con 0x7faa0806c520 2024-01-15T18:05:47.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.258+0000 7faa0d200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7faa0815d570 0x7faa0806be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41866/0 (socket says 172.21.15.27:41866) 2024-01-15T18:05:47.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.258+0000 7faa0dc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faa0815d190 0x7faa080cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.258+0000 7faa0d200640 1 -- 172.21.15.27:0/1232444816 learned_addr learned my addr 172.21.15.27:0/1232444816 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:47.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.258+0000 7faa0e600640 1 --2- 172.21.15.27:0/1232444816 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faa0806c520 0x7faa080cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.258+0000 7faa0dc00640 1 -- 172.21.15.27:0/1232444816 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faa0806c520 msgr2=0x7faa080cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:47.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.258+0000 7faa0dc00640 1 --2- 172.21.15.27:0/1232444816 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faa0806c520 0x7faa080cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.258+0000 7faa0dc00640 1 -- 172.21.15.27:0/1232444816 >> v1:172.21.15.27:6789/0 conn(0x7faa0815d570 legacy=0x7faa0806be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.258+0000 7faa0dc00640 1 -- 172.21.15.27:0/1232444816 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa080b0720 con 0x7faa0815d190 2024-01-15T18:05:47.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.258+0000 7faa0e600640 1 --2- 172.21.15.27:0/1232444816 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faa0806c520 0x7faa080cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:47.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.258+0000 7faa0dc00640 1 --2- 172.21.15.27:0/1232444816 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa0815d190 0x7faa080cfcf0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa9f80097c0 tx=0x7fa9f801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e00a7afe201e614 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:47.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.258+0000 7faa07e00640 1 -- 172.21.15.27:0/1232444816 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f80ab610 con 0x7faa0815d190 2024-01-15T18:05:47.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.262+0000 7faa07e00640 1 -- 172.21.15.27:0/1232444816 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f80ab770 con 0x7faa0815d190 2024-01-15T18:05:47.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.262+0000 7faa07e00640 1 -- 172.21.15.27:0/1232444816 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f80abb00 con 0x7faa0815d190 2024-01-15T18:05:47.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.262+0000 7faa10600640 1 -- 172.21.15.27:0/1232444816 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa0815d190 msgr2=0x7faa080cfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.262+0000 7faa10600640 1 --2- 172.21.15.27:0/1232444816 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa0815d190 0x7faa080cfcf0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa9f80097c0 tx=0x7fa9f801c450 comp rx=0 tx=0).stop 2024-01-15T18:05:47.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.262+0000 7faa10600640 1 -- 172.21.15.27:0/1232444816 shutdown_connections 2024-01-15T18:05:47.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.262+0000 7faa10600640 1 --2- 172.21.15.27:0/1232444816 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faa0806c520 0x7faa080cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.262+0000 7faa10600640 1 --2- 172.21.15.27:0/1232444816 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa0815d190 0x7faa080cfcf0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.262+0000 7faa10600640 1 -- 172.21.15.27:0/1232444816 >> 172.21.15.27:0/1232444816 conn(0x7faa080d0e70 msgr2=0x7faa080d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:47.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.262+0000 7faa10600640 1 -- 172.21.15.27:0/1232444816 shutdown_connections 2024-01-15T18:05:47.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.262+0000 7faa10600640 1 -- 172.21.15.27:0/1232444816 wait complete. 2024-01-15T18:05:47.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.262+0000 7faa10600640 1 Processor -- start 2024-01-15T18:05:47.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.262+0000 7faa10600640 1 -- start start 2024-01-15T18:05:47.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.262+0000 7faa10600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faa0806c520 0x7faa0814b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.262+0000 7faa10600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faa0815d190 0x7faa08149bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.262+0000 7faa10600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa0815d570 0x7faa0814a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.262+0000 7faa10600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa080c7920 con 0x7faa0815d570 2024-01-15T18:05:47.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.262+0000 7faa10600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa080c7640 con 0x7faa0806c520 2024-01-15T18:05:47.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.262+0000 7faa10600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa08152ac0 con 0x7faa0815d190 2024-01-15T18:05:47.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.266+0000 7faa0e600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa0815d570 0x7faa0814a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.266+0000 7faa0e600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa0815d570 0x7faa0814a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38914/0 (socket says 172.21.15.27:38914) 2024-01-15T18:05:47.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.266+0000 7faa0e600640 1 -- 172.21.15.27:0/984426390 learned_addr learned my addr 172.21.15.27:0/984426390 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:47.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.266+0000 7faa0dc00640 1 --2- 172.21.15.27:0/984426390 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faa0806c520 0x7faa0814b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.266+0000 7faa0d200640 1 --2- 172.21.15.27:0/984426390 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faa0815d190 0x7faa08149bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.266+0000 7faa0e600640 1 -- 172.21.15.27:0/984426390 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faa0815d190 msgr2=0x7faa08149bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.266+0000 7faa0e600640 1 --2- 172.21.15.27:0/984426390 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faa0815d190 0x7faa08149bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.266+0000 7faa0e600640 1 -- 172.21.15.27:0/984426390 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faa0806c520 msgr2=0x7faa0814b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.266+0000 7faa0e600640 1 --2- 172.21.15.27:0/984426390 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faa0806c520 0x7faa0814b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.266+0000 7faa0e600640 1 -- 172.21.15.27:0/984426390 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f8009420 con 0x7faa0815d570 2024-01-15T18:05:47.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.266+0000 7faa0e600640 1 --2- 172.21.15.27:0/984426390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa0815d570 0x7faa0814a110 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7faa00007ec0 tx=0x7faa0000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:47.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.266+0000 7faa05600640 1 -- 172.21.15.27:0/984426390 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa00015600 con 0x7faa0815d570 2024-01-15T18:05:47.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.266+0000 7faa05600640 1 -- 172.21.15.27:0/984426390 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0000f040 con 0x7faa0815d570 2024-01-15T18:05:47.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.266+0000 7faa10600640 1 -- 172.21.15.27:0/984426390 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa08152da0 con 0x7faa0815d570 2024-01-15T18:05:47.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.266+0000 7faa05600640 1 -- 172.21.15.27:0/984426390 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa000159a0 con 0x7faa0815d570 2024-01-15T18:05:47.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.266+0000 7faa10600640 1 -- 172.21.15.27:0/984426390 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa0814b5b0 con 0x7faa0815d570 2024-01-15T18:05:47.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.266+0000 7faa10600640 1 -- 172.21.15.27:0/984426390 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa081532c0 con 0x7faa0815d570 2024-01-15T18:05:47.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.266+0000 7faa05600640 1 -- 172.21.15.27:0/984426390 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7faa0001e3b0 con 0x7faa0815d570 2024-01-15T18:05:47.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.270+0000 7faa05600640 1 --2- 172.21.15.27:0/984426390 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa9e00668f0 0x7fa9e0068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.270+0000 7faa05600640 1 -- 172.21.15.27:0/984426390 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7faa00084750 con 0x7faa0815d570 2024-01-15T18:05:47.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.270+0000 7faa0dc00640 1 --2- 172.21.15.27:0/984426390 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa9e00668f0 0x7fa9e0068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.270+0000 7faa0dc00640 1 --2- 172.21.15.27:0/984426390 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa9e00668f0 0x7fa9e0068db0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa9f80af000 tx=0x7fa9f80a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:47.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.270+0000 7faa05600640 1 -- 172.21.15.27:0/984426390 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7faa081532c0 con 0x7faa0815d570 2024-01-15T18:05:47.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.430+0000 7faa10600640 1 -- 172.21.15.27:0/984426390 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faa081532c0 con 0x7faa0815d570 2024-01-15T18:05:47.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.430+0000 7faa05600640 1 -- 172.21.15.27:0/984426390 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7faa081532c0 con 0x7faa0815d570 2024-01-15T18:05:47.438 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-15T18:05:47.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.434+0000 7faa10600640 1 -- 172.21.15.27:0/984426390 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa9e00668f0 msgr2=0x7fa9e0068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.434+0000 7faa10600640 1 --2- 172.21.15.27:0/984426390 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa9e00668f0 0x7fa9e0068db0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa9f80af000 tx=0x7fa9f80a7000 comp rx=0 tx=0).stop 2024-01-15T18:05:47.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.434+0000 7faa10600640 1 -- 172.21.15.27:0/984426390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa0815d570 msgr2=0x7faa0814a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.434+0000 7faa10600640 1 --2- 172.21.15.27:0/984426390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa0815d570 0x7faa0814a110 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7faa00007ec0 tx=0x7faa0000d4a0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.434+0000 7faa10600640 1 -- 172.21.15.27:0/984426390 shutdown_connections 2024-01-15T18:05:47.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.434+0000 7faa10600640 1 --2- 172.21.15.27:0/984426390 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa9e00668f0 0x7fa9e0068db0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.434+0000 7faa10600640 1 --2- 172.21.15.27:0/984426390 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa0815d570 0x7faa0814a110 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.434+0000 7faa10600640 1 --2- 172.21.15.27:0/984426390 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faa0815d190 0x7faa08149bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.434+0000 7faa10600640 1 --2- 172.21.15.27:0/984426390 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faa0806c520 0x7faa0814b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.434+0000 7faa10600640 1 -- 172.21.15.27:0/984426390 >> 172.21.15.27:0/984426390 conn(0x7faa080d0e70 msgr2=0x7faa080cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:47.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.434+0000 7faa10600640 1 -- 172.21.15.27:0/984426390 shutdown_connections 2024-01-15T18:05:47.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.434+0000 7faa10600640 1 -- 172.21.15.27:0/984426390 wait complete. 2024-01-15T18:05:47.460 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T18:05:47.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.598+0000 7f6742c00640 1 Processor -- start 2024-01-15T18:05:47.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.598+0000 7f6742c00640 1 -- start start 2024-01-15T18:05:47.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.598+0000 7f6742c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f673c06c430 0x7f673c070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.598+0000 7f6742c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f673c0b1db0 0x7f673c0b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.598+0000 7f6742c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f673c0ce050 0x7f673c0ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.598+0000 7f6742c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f673c0cea20 con 0x7f673c0b1db0 2024-01-15T18:05:47.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.598+0000 7f6742c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f673c0ccfb0 con 0x7f673c06c430 2024-01-15T18:05:47.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.598+0000 7f6742c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f673c0cd290 con 0x7f673c0ce050 2024-01-15T18:05:47.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.598+0000 7f673b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f673c0b1db0 0x7f673c0b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.598+0000 7f673be00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f673c06c430 0x7f673c070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.598+0000 7f6740c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f673c0ce050 0x7f673c0ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.598+0000 7f673b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f673c0b1db0 0x7f673c0b2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38918/0 (socket says 172.21.15.27:38918) 2024-01-15T18:05:47.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.598+0000 7f673b400640 1 -- 172.21.15.27:0/3871940150 learned_addr learned my addr 172.21.15.27:0/3871940150 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:47.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.602+0000 7f673b400640 1 -- 172.21.15.27:0/3871940150 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f673c0ce050 msgr2=0x7f673c0ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:47.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.602+0000 7f673b400640 1 --2- 172.21.15.27:0/3871940150 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f673c0ce050 0x7f673c0ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.602+0000 7f673b400640 1 -- 172.21.15.27:0/3871940150 >> v2:172.21.15.93:3300/0 conn(0x7f673c06c430 msgr2=0x7f673c070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:47.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.602+0000 7f673b400640 1 --2- 172.21.15.27:0/3871940150 >> v2:172.21.15.93:3300/0 conn(0x7f673c06c430 0x7f673c070ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.602+0000 7f673b400640 1 -- 172.21.15.27:0/3871940150 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f673c0cd570 con 0x7f673c0b1db0 2024-01-15T18:05:47.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.602+0000 7f673b400640 1 --2- 172.21.15.27:0/3871940150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f673c0b1db0 0x7f673c0b2240 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6724009870 tx=0x7f672401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f5a20bc0ecf0256 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:47.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.602+0000 7f673aa00640 1 -- 172.21.15.27:0/3871940150 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67240ad680 con 0x7f673c0b1db0 2024-01-15T18:05:47.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.602+0000 7f673aa00640 1 -- 172.21.15.27:0/3871940150 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67240ad7e0 con 0x7f673c0b1db0 2024-01-15T18:05:47.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.602+0000 7f673aa00640 1 -- 172.21.15.27:0/3871940150 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67240adb70 con 0x7f673c0b1db0 2024-01-15T18:05:47.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.606+0000 7f6742c00640 1 -- 172.21.15.27:0/3871940150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f673c0b1db0 msgr2=0x7f673c0b2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.606+0000 7f6742c00640 1 --2- 172.21.15.27:0/3871940150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f673c0b1db0 0x7f673c0b2240 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6724009870 tx=0x7f672401c440 comp rx=0 tx=0).stop 2024-01-15T18:05:47.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.606+0000 7f6742c00640 1 -- 172.21.15.27:0/3871940150 shutdown_connections 2024-01-15T18:05:47.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.606+0000 7f6742c00640 1 --2- 172.21.15.27:0/3871940150 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f673c0ce050 0x7f673c0ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.606+0000 7f6742c00640 1 --2- 172.21.15.27:0/3871940150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f673c0b1db0 0x7f673c0b2240 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.606+0000 7f6742c00640 1 --2- 172.21.15.27:0/3871940150 >> v2:172.21.15.93:3300/0 conn(0x7f673c06c430 0x7f673c070ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.606+0000 7f6742c00640 1 -- 172.21.15.27:0/3871940150 >> 172.21.15.27:0/3871940150 conn(0x7f673c0cb700 msgr2=0x7f673c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:47.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.606+0000 7f6742c00640 1 -- 172.21.15.27:0/3871940150 shutdown_connections 2024-01-15T18:05:47.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.606+0000 7f6742c00640 1 -- 172.21.15.27:0/3871940150 wait complete. 2024-01-15T18:05:47.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.606+0000 7f6742c00640 1 Processor -- start 2024-01-15T18:05:47.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.606+0000 7f6742c00640 1 -- start start 2024-01-15T18:05:47.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.606+0000 7f6742c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f673c06c430 0x7f673c14aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.606+0000 7f6742c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f673c0b1db0 0x7f673c0ac880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.606+0000 7f6742c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f673c0ce050 0x7f673c13f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.606+0000 7f6742c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c14d8d0 con 0x7f673c0ce050 2024-01-15T18:05:47.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.606+0000 7f6742c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c13fc10 con 0x7f673c06c430 2024-01-15T18:05:47.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.606+0000 7f6742c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f673c14b080 con 0x7f673c0b1db0 2024-01-15T18:05:47.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.606+0000 7f6740c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f673c0ce050 0x7f673c13f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.606+0000 7f673be00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f673c06c430 0x7f673c14aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.606+0000 7f6740c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f673c0ce050 0x7f673c13f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38928/0 (socket says 172.21.15.27:38928) 2024-01-15T18:05:47.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.606+0000 7f6740c00640 1 -- 172.21.15.27:0/601983683 learned_addr learned my addr 172.21.15.27:0/601983683 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:47.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.606+0000 7f673b400640 1 --2- 172.21.15.27:0/601983683 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f673c0b1db0 0x7f673c0ac880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.610+0000 7f6740c00640 1 -- 172.21.15.27:0/601983683 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f673c0b1db0 msgr2=0x7f673c0ac880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.610+0000 7f6740c00640 1 --2- 172.21.15.27:0/601983683 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f673c0b1db0 0x7f673c0ac880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.610+0000 7f6740c00640 1 -- 172.21.15.27:0/601983683 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f673c06c430 msgr2=0x7f673c14aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.610+0000 7f6740c00640 1 --2- 172.21.15.27:0/601983683 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f673c06c430 0x7f673c14aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.610+0000 7f6740c00640 1 -- 172.21.15.27:0/601983683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67240094d0 con 0x7f673c0ce050 2024-01-15T18:05:47.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.610+0000 7f6740c00640 1 --2- 172.21.15.27:0/601983683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f673c0ce050 0x7f673c13f6d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6734009d30 tx=0x7f673400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:47.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.610+0000 7f6733e00640 1 -- 172.21.15.27:0/601983683 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6734009ba0 con 0x7f673c0ce050 2024-01-15T18:05:47.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.610+0000 7f6733e00640 1 -- 172.21.15.27:0/601983683 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f673400f040 con 0x7f673c0ce050 2024-01-15T18:05:47.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.610+0000 7f6733e00640 1 -- 172.21.15.27:0/601983683 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67340146c0 con 0x7f673c0ce050 2024-01-15T18:05:47.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.610+0000 7f6742c00640 1 -- 172.21.15.27:0/601983683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f673c14d210 con 0x7f673c0ce050 2024-01-15T18:05:47.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.610+0000 7f6742c00640 1 -- 172.21.15.27:0/601983683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f673c14b360 con 0x7f673c0ce050 2024-01-15T18:05:47.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.610+0000 7f6742c00640 1 -- 172.21.15.27:0/601983683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f673c14d070 con 0x7f673c0ce050 2024-01-15T18:05:47.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.610+0000 7f6733e00640 1 -- 172.21.15.27:0/601983683 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f6734014820 con 0x7f673c0ce050 2024-01-15T18:05:47.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.610+0000 7f6733e00640 1 --2- 172.21.15.27:0/601983683 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f67140668f0 0x7f6714068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.610+0000 7f6733e00640 1 -- 172.21.15.27:0/601983683 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f6734084df0 con 0x7f673c0ce050 2024-01-15T18:05:47.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.614+0000 7f673be00640 1 --2- 172.21.15.27:0/601983683 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f67140668f0 0x7f6714068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.614+0000 7f673be00640 1 --2- 172.21.15.27:0/601983683 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f67140668f0 0x7f6714068db0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f672c009b60 tx=0x7f672c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:47.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.614+0000 7f6733e00640 1 -- 172.21.15.27:0/601983683 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f673c14d070 con 0x7f673c0ce050 2024-01-15T18:05:47.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.770+0000 7f6742c00640 1 -- 172.21.15.27:0/601983683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f673c14da10 con 0x7f673c0ce050 2024-01-15T18:05:47.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.770+0000 7f6733e00640 1 -- 172.21.15.27:0/601983683 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f673c14da10 con 0x7f673c0ce050 2024-01-15T18:05:47.778 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-15T18:05:47.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.774+0000 7f6742c00640 1 -- 172.21.15.27:0/601983683 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f67140668f0 msgr2=0x7f6714068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.774+0000 7f6742c00640 1 --2- 172.21.15.27:0/601983683 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f67140668f0 0x7f6714068db0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f672c009b60 tx=0x7f672c009190 comp rx=0 tx=0).stop 2024-01-15T18:05:47.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.774+0000 7f6742c00640 1 -- 172.21.15.27:0/601983683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f673c0ce050 msgr2=0x7f673c13f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.774+0000 7f6742c00640 1 --2- 172.21.15.27:0/601983683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f673c0ce050 0x7f673c13f6d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6734009d30 tx=0x7f673400c490 comp rx=0 tx=0).stop 2024-01-15T18:05:47.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.774+0000 7f6742c00640 1 -- 172.21.15.27:0/601983683 shutdown_connections 2024-01-15T18:05:47.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.774+0000 7f6742c00640 1 --2- 172.21.15.27:0/601983683 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f67140668f0 0x7f6714068db0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.774+0000 7f6742c00640 1 --2- 172.21.15.27:0/601983683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f673c0ce050 0x7f673c13f6d0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.774+0000 7f6742c00640 1 --2- 172.21.15.27:0/601983683 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f673c0b1db0 0x7f673c0ac880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.774+0000 7f6742c00640 1 --2- 172.21.15.27:0/601983683 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f673c06c430 0x7f673c14aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.774+0000 7f6742c00640 1 -- 172.21.15.27:0/601983683 >> 172.21.15.27:0/601983683 conn(0x7f673c0cb700 msgr2=0x7f673c14ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:47.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.774+0000 7f6742c00640 1 -- 172.21.15.27:0/601983683 shutdown_connections 2024-01-15T18:05:47.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.774+0000 7f6742c00640 1 -- 172.21.15.27:0/601983683 wait complete. 2024-01-15T18:05:47.800 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:05:47.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.934+0000 7fedfa800640 1 Processor -- start 2024-01-15T18:05:47.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.934+0000 7fedfa800640 1 -- start start 2024-01-15T18:05:47.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.934+0000 7fedfa800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fedf40cf080 0x7fedf40cf460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.934+0000 7fedfa800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fedf40b0130 0x7fedf40b0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.934+0000 7fedfa800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf40d1ec0 con 0x7fedf40b24a0 2024-01-15T18:05:47.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.934+0000 7fedfa800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf40ae160 con 0x7fedf40cf080 2024-01-15T18:05:47.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.934+0000 7fedfa800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf40d1be0 con 0x7fedf40b0130 2024-01-15T18:05:47.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.934+0000 7fedf3400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fedf40cf080 0x7fedf40cf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.934+0000 7fedf3e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fedf40b24a0 0x7fedf40b0dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41872/0 (socket says 172.21.15.27:41872) 2024-01-15T18:05:47.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.934+0000 7fedf3e00640 1 -- 172.21.15.27:0/4289489209 learned_addr learned my addr 172.21.15.27:0/4289489209 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:47.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.934+0000 7fedf2a00640 1 --2- 172.21.15.27:0/4289489209 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fedf40b0130 0x7fedf40b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.938+0000 7fedf3400640 1 -- 172.21.15.27:0/4289489209 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fedf40b0130 msgr2=0x7fedf40b0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:47.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.938+0000 7fedf3400640 1 --2- 172.21.15.27:0/4289489209 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fedf40b0130 0x7fedf40b0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.938+0000 7fedf2000640 1 -- 172.21.15.27:0/4289489209 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786784727 0 0) 0x7fedf40d1ec0 con 0x7fedf40b24a0 2024-01-15T18:05:47.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.938+0000 7fedf3400640 1 -- 172.21.15.27:0/4289489209 >> v1:172.21.15.27:6789/0 conn(0x7fedf40b24a0 legacy=0x7fedf40b0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.938+0000 7fedf3400640 1 -- 172.21.15.27:0/4289489209 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf40addf0 con 0x7fedf40cf080 2024-01-15T18:05:47.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.938+0000 7fedf3400640 1 --2- 172.21.15.27:0/4289489209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf40cf080 0x7fedf40cf460 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fede40097c0 tx=0x7fede401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ba7bd06343809ef server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:47.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.938+0000 7fedf2000640 1 -- 172.21.15.27:0/4289489209 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede40a6040 con 0x7fedf40cf080 2024-01-15T18:05:47.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.938+0000 7fedf2000640 1 -- 172.21.15.27:0/4289489209 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede40aa600 con 0x7fedf40cf080 2024-01-15T18:05:47.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.938+0000 7fedf2000640 1 -- 172.21.15.27:0/4289489209 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede40aa990 con 0x7fedf40cf080 2024-01-15T18:05:47.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedfa800640 1 -- 172.21.15.27:0/4289489209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf40cf080 msgr2=0x7fedf40cf460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedfa800640 1 --2- 172.21.15.27:0/4289489209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf40cf080 0x7fedf40cf460 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fede40097c0 tx=0x7fede401c480 comp rx=0 tx=0).stop 2024-01-15T18:05:47.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedfa800640 1 -- 172.21.15.27:0/4289489209 shutdown_connections 2024-01-15T18:05:47.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedfa800640 1 --2- 172.21.15.27:0/4289489209 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fedf40b0130 0x7fedf40b0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedfa800640 1 --2- 172.21.15.27:0/4289489209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf40cf080 0x7fedf40cf460 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedfa800640 1 -- 172.21.15.27:0/4289489209 >> 172.21.15.27:0/4289489209 conn(0x7fedf40d0f20 msgr2=0x7fedf40d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:47.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedfa800640 1 -- 172.21.15.27:0/4289489209 shutdown_connections 2024-01-15T18:05:47.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedfa800640 1 -- 172.21.15.27:0/4289489209 wait complete. 2024-01-15T18:05:47.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedfa800640 1 Processor -- start 2024-01-15T18:05:47.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedfa800640 1 -- start start 2024-01-15T18:05:47.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedfa800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fedf40b0130 0x7fedf40c5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedfa800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fedf40b24a0 0x7fedf4153e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedfa800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf40cf080 0x7fedf415cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedfa800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf40c6060 con 0x7fedf40cf080 2024-01-15T18:05:47.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedfa800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf40c6ea0 con 0x7fedf40b24a0 2024-01-15T18:05:47.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedfa800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf40c6290 con 0x7fedf40b0130 2024-01-15T18:05:47.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedf3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf40cf080 0x7fedf415cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedf3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf40cf080 0x7fedf415cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38942/0 (socket says 172.21.15.27:38942) 2024-01-15T18:05:47.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedf3e00640 1 -- 172.21.15.27:0/3187903811 learned_addr learned my addr 172.21.15.27:0/3187903811 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:47.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedf3400640 1 --2- 172.21.15.27:0/3187903811 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fedf40b0130 0x7fedf40c5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedf2a00640 1 --2- 172.21.15.27:0/3187903811 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fedf40b24a0 0x7fedf4153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedf3e00640 1 -- 172.21.15.27:0/3187903811 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fedf40b0130 msgr2=0x7fedf40c5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedf3e00640 1 --2- 172.21.15.27:0/3187903811 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fedf40b0130 0x7fedf40c5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedf3e00640 1 -- 172.21.15.27:0/3187903811 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fedf40b24a0 msgr2=0x7fedf4153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedf3e00640 1 --2- 172.21.15.27:0/3187903811 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fedf40b24a0 0x7fedf4153e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.942+0000 7fedf3e00640 1 -- 172.21.15.27:0/3187903811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede4009420 con 0x7fedf40cf080 2024-01-15T18:05:47.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.946+0000 7fedf3e00640 1 --2- 172.21.15.27:0/3187903811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf40cf080 0x7fedf415cee0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fedec001570 tx=0x7fedec00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:47.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.946+0000 7fedeb400640 1 -- 172.21.15.27:0/3187903811 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec00a640 con 0x7fedf40cf080 2024-01-15T18:05:47.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.946+0000 7fedeb400640 1 -- 172.21.15.27:0/3187903811 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedec00a7c0 con 0x7fedf40cf080 2024-01-15T18:05:47.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.946+0000 7fedfa800640 1 -- 172.21.15.27:0/3187903811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf40c6570 con 0x7fedf40cf080 2024-01-15T18:05:47.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.946+0000 7fedeb400640 1 -- 172.21.15.27:0/3187903811 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec00ab70 con 0x7fedf40cf080 2024-01-15T18:05:47.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.946+0000 7fedfa800640 1 -- 172.21.15.27:0/3187903811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf40c6b30 con 0x7fedf40cf080 2024-01-15T18:05:47.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.946+0000 7fedfa800640 1 -- 172.21.15.27:0/3187903811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf40c7010 con 0x7fedf40cf080 2024-01-15T18:05:47.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.946+0000 7fedeb400640 1 -- 172.21.15.27:0/3187903811 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fedec00acd0 con 0x7fedf40cf080 2024-01-15T18:05:47.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.946+0000 7fedeb400640 1 --2- 172.21.15.27:0/3187903811 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fedcc0668f0 0x7fedcc068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.946+0000 7fedeb400640 1 -- 172.21.15.27:0/3187903811 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fedec083b50 con 0x7fedf40cf080 2024-01-15T18:05:47.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.950+0000 7fedf3400640 1 --2- 172.21.15.27:0/3187903811 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fedcc0668f0 0x7fedcc068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.950+0000 7fedf3400640 1 --2- 172.21.15.27:0/3187903811 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fedcc0668f0 0x7fedcc068db0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fede40ae000 tx=0x7fede40b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:47.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:47.950+0000 7fedeb400640 1 -- 172.21.15.27:0/3187903811 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fedec050e40 con 0x7fedf40cf080 2024-01-15T18:05:48.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.146+0000 7fedfa800640 1 -- 172.21.15.27:0/3187903811 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedf404e850 con 0x7fedf40cf080 2024-01-15T18:05:48.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.146+0000 7fedeb400640 1 -- 172.21.15.27:0/3187903811 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7fedec018020 con 0x7fedf40cf080 2024-01-15T18:05:48.153 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:48.153 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/393360880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":393360880},{"type":"v1","addr":"172.21.15.93:6836","nonce":393360880}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4229,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4063308188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4063308188},{"type":"v1","addr":"172.21.15.27:6835","nonce":4063308188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4305,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/2343128056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":2343128056},{"type":"v1","addr":"172.21.15.93:6837","nonce":2343128056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:04:58.208165+0000","modified":"2024-01-15T18:05:01.225419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2657553192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2657553192},{"type":"v1","addr":"172.21.15.27:6834","nonce":2657553192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T18:05:48.153 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 8 2024-01-15T18:05:48.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.150+0000 7fedfa800640 1 -- 172.21.15.27:0/3187903811 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fedcc0668f0 msgr2=0x7fedcc068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.150+0000 7fedfa800640 1 --2- 172.21.15.27:0/3187903811 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fedcc0668f0 0x7fedcc068db0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fede40ae000 tx=0x7fede40b5040 comp rx=0 tx=0).stop 2024-01-15T18:05:48.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.150+0000 7fedfa800640 1 -- 172.21.15.27:0/3187903811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf40cf080 msgr2=0x7fedf415cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.150+0000 7fedfa800640 1 --2- 172.21.15.27:0/3187903811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf40cf080 0x7fedf415cee0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fedec001570 tx=0x7fedec00cca0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.150+0000 7fedfa800640 1 -- 172.21.15.27:0/3187903811 shutdown_connections 2024-01-15T18:05:48.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.150+0000 7fedfa800640 1 --2- 172.21.15.27:0/3187903811 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fedcc0668f0 0x7fedcc068db0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.150+0000 7fedfa800640 1 --2- 172.21.15.27:0/3187903811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedf40cf080 0x7fedf415cee0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.150+0000 7fedfa800640 1 --2- 172.21.15.27:0/3187903811 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fedf40b24a0 0x7fedf4153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.150+0000 7fedfa800640 1 --2- 172.21.15.27:0/3187903811 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fedf40b0130 0x7fedf40c5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.150+0000 7fedfa800640 1 -- 172.21.15.27:0/3187903811 >> 172.21.15.27:0/3187903811 conn(0x7fedf40d0f20 msgr2=0x7fedf40cb880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:48.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.150+0000 7fedfa800640 1 -- 172.21.15.27:0/3187903811 shutdown_connections 2024-01-15T18:05:48.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.150+0000 7fedfa800640 1 -- 172.21.15.27:0/3187903811 wait complete. 2024-01-15T18:05:48.175 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:48.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.310+0000 7fad22e00640 1 Processor -- start 2024-01-15T18:05:48.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.310+0000 7fad22e00640 1 -- start start 2024-01-15T18:05:48.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.310+0000 7fad22e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fad1c0cdf90 0x7fad1c168450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.310+0000 7fad22e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad1c0b2770 con 0x7fad1c06bb80 2024-01-15T18:05:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.310+0000 7fad22e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad1c06c060 con 0x7fad1c0b2390 2024-01-15T18:05:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.310+0000 7fad22e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad1c06c340 con 0x7fad1c0cdf90 2024-01-15T18:05:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.310+0000 7fad1b400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fad1c06bb80 0x7fad1c0cd880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41886/0 (socket says 172.21.15.27:41886) 2024-01-15T18:05:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.310+0000 7fad1b400640 1 -- 172.21.15.27:0/2230592570 learned_addr learned my addr 172.21.15.27:0/2230592570 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.310+0000 7fad20e00640 1 --2- 172.21.15.27:0/2230592570 >> v2:172.21.15.27:3300/0 conn(0x7fad1c0cdf90 0x7fad1c168450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.310+0000 7fad20e00640 1 -- 172.21.15.27:0/2230592570 >> v1:172.21.15.93:6789/0 conn(0x7fad1c0b2390 legacy=0x7fad1c0cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.310+0000 7fad20e00640 1 -- 172.21.15.27:0/2230592570 >> v1:172.21.15.27:6789/0 conn(0x7fad1c06bb80 legacy=0x7fad1c0cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.310+0000 7fad20e00640 1 -- 172.21.15.27:0/2230592570 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1c06c620 con 0x7fad1c0cdf90 2024-01-15T18:05:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.310+0000 7fad1aa00640 1 -- 172.21.15.27:0/2230592570 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1817115224 0 0) 0x7fad1c0b2770 con 0x7fad1c06bb80 2024-01-15T18:05:48.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.314+0000 7fad20e00640 1 --2- 172.21.15.27:0/2230592570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad1c0cdf90 0x7fad1c168450 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fad14009d90 tx=0x7fad1401c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=653fd933f09c1522 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:48.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.314+0000 7fad1aa00640 1 -- 172.21.15.27:0/2230592570 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad140a59e0 con 0x7fad1c0cdf90 2024-01-15T18:05:48.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.314+0000 7fad1aa00640 1 -- 172.21.15.27:0/2230592570 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad140a5b40 con 0x7fad1c0cdf90 2024-01-15T18:05:48.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.314+0000 7fad1aa00640 1 -- 172.21.15.27:0/2230592570 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad140ad440 con 0x7fad1c0cdf90 2024-01-15T18:05:48.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.314+0000 7fad22e00640 1 -- 172.21.15.27:0/2230592570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad1c0cdf90 msgr2=0x7fad1c168450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.314+0000 7fad22e00640 1 --2- 172.21.15.27:0/2230592570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad1c0cdf90 0x7fad1c168450 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fad14009d90 tx=0x7fad1401c920 comp rx=0 tx=0).stop 2024-01-15T18:05:48.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.314+0000 7fad22e00640 1 -- 172.21.15.27:0/2230592570 shutdown_connections 2024-01-15T18:05:48.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.314+0000 7fad22e00640 1 --2- 172.21.15.27:0/2230592570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad1c0cdf90 0x7fad1c168450 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.314+0000 7fad22e00640 1 -- 172.21.15.27:0/2230592570 >> 172.21.15.27:0/2230592570 conn(0x7fad1c0b2a10 msgr2=0x7fad1c0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:48.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.318+0000 7fad22e00640 1 -- 172.21.15.27:0/2230592570 shutdown_connections 2024-01-15T18:05:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.318+0000 7fad22e00640 1 -- 172.21.15.27:0/2230592570 wait complete. 2024-01-15T18:05:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.318+0000 7fad22e00640 1 Processor -- start 2024-01-15T18:05:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.318+0000 7fad22e00640 1 -- start start 2024-01-15T18:05:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.318+0000 7fad22e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fad1c06bb80 0x7fad1c14ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.318+0000 7fad22e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fad1c0b2390 0x7fad1c0c6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.318+0000 7fad22e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad1c0cdf90 0x7fad1c155bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.318+0000 7fad22e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c0b41a0 con 0x7fad1c0cdf90 2024-01-15T18:05:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.318+0000 7fad22e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c0c7230 con 0x7fad1c06bb80 2024-01-15T18:05:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.318+0000 7fad22e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c0c7510 con 0x7fad1c0b2390 2024-01-15T18:05:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.318+0000 7fad20e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad1c0cdf90 0x7fad1c155bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.318+0000 7fad20e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad1c0cdf90 0x7fad1c155bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38960/0 (socket says 172.21.15.27:38960) 2024-01-15T18:05:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.318+0000 7fad20e00640 1 -- 172.21.15.27:0/2114506863 learned_addr learned my addr 172.21.15.27:0/2114506863 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.318+0000 7fad1be00640 1 --2- 172.21.15.27:0/2114506863 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fad1c06bb80 0x7fad1c14ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.318+0000 7fad1b400640 1 --2- 172.21.15.27:0/2114506863 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fad1c0b2390 0x7fad1c0c6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.318+0000 7fad20e00640 1 -- 172.21.15.27:0/2114506863 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fad1c0b2390 msgr2=0x7fad1c0c6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.318+0000 7fad20e00640 1 --2- 172.21.15.27:0/2114506863 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fad1c0b2390 0x7fad1c0c6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.318+0000 7fad20e00640 1 -- 172.21.15.27:0/2114506863 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fad1c06bb80 msgr2=0x7fad1c14ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.318+0000 7fad20e00640 1 --2- 172.21.15.27:0/2114506863 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fad1c06bb80 0x7fad1c14ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.318+0000 7fad20e00640 1 -- 172.21.15.27:0/2114506863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad14009a70 con 0x7fad1c0cdf90 2024-01-15T18:05:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.318+0000 7fad20e00640 1 --2- 172.21.15.27:0/2114506863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad1c0cdf90 0x7fad1c155bf0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fad1401a550 tx=0x7fad140196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.322+0000 7fad13e00640 1 -- 172.21.15.27:0/2114506863 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad140aee90 con 0x7fad1c0cdf90 2024-01-15T18:05:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.322+0000 7fad13e00640 1 -- 172.21.15.27:0/2114506863 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad140039a0 con 0x7fad1c0cdf90 2024-01-15T18:05:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.322+0000 7fad13e00640 1 -- 172.21.15.27:0/2114506863 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad14003d30 con 0x7fad1c0cdf90 2024-01-15T18:05:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.322+0000 7fad22e00640 1 -- 172.21.15.27:0/2114506863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad1c0c77f0 con 0x7fad1c0cdf90 2024-01-15T18:05:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.322+0000 7fad22e00640 1 -- 172.21.15.27:0/2114506863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad1c0c6d70 con 0x7fad1c0cdf90 2024-01-15T18:05:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.322+0000 7fad22e00640 1 -- 172.21.15.27:0/2114506863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad1c0c7650 con 0x7fad1c0cdf90 2024-01-15T18:05:48.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.322+0000 7fad13e00640 1 -- 172.21.15.27:0/2114506863 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fad140b4050 con 0x7fad1c0cdf90 2024-01-15T18:05:48.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.322+0000 7fad13e00640 1 --2- 172.21.15.27:0/2114506863 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7facf4066920 0x7facf4068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:48.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.322+0000 7fad13e00640 1 -- 172.21.15.27:0/2114506863 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fad1411e780 con 0x7fad1c0cdf90 2024-01-15T18:05:48.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.322+0000 7fad1be00640 1 --2- 172.21.15.27:0/2114506863 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7facf4066920 0x7facf4068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:48.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.326+0000 7fad1be00640 1 --2- 172.21.15.27:0/2114506863 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7facf4066920 0x7facf4068de0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fad0c007fd0 tx=0x7fad0c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:48.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.326+0000 7fad13e00640 1 -- 172.21.15.27:0/2114506863 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fad1c0c7650 con 0x7fad1c0cdf90 2024-01-15T18:05:48.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.486+0000 7fad22e00640 1 -- 172.21.15.27:0/2114506863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad1c0c7650 con 0x7fad1c0cdf90 2024-01-15T18:05:48.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.486+0000 7fad13e00640 1 -- 172.21.15.27:0/2114506863 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16705 (secure 0 0 0) 0x7fad1c0c7650 con 0x7fad1c0cdf90 2024-01-15T18:05:48.493 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:48.494 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":23,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:05:40.008536+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:04:56.220608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T18:04:57.210421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:48.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.490+0000 7fad22e00640 1 -- 172.21.15.27:0/2114506863 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7facf4066920 msgr2=0x7facf4068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.490+0000 7fad22e00640 1 --2- 172.21.15.27:0/2114506863 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7facf4066920 0x7facf4068de0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fad0c007fd0 tx=0x7fad0c009040 comp rx=0 tx=0).stop 2024-01-15T18:05:48.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.490+0000 7fad22e00640 1 -- 172.21.15.27:0/2114506863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad1c0cdf90 msgr2=0x7fad1c155bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.490+0000 7fad22e00640 1 --2- 172.21.15.27:0/2114506863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad1c0cdf90 0x7fad1c155bf0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fad1401a550 tx=0x7fad140196c0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.490+0000 7fad22e00640 1 -- 172.21.15.27:0/2114506863 shutdown_connections 2024-01-15T18:05:48.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.490+0000 7fad22e00640 1 --2- 172.21.15.27:0/2114506863 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7facf4066920 0x7facf4068de0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.490+0000 7fad22e00640 1 --2- 172.21.15.27:0/2114506863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad1c0cdf90 0x7fad1c155bf0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.490+0000 7fad22e00640 1 --2- 172.21.15.27:0/2114506863 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fad1c0b2390 0x7fad1c0c6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.490+0000 7fad22e00640 1 --2- 172.21.15.27:0/2114506863 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fad1c06bb80 0x7fad1c14ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.490+0000 7fad22e00640 1 -- 172.21.15.27:0/2114506863 >> 172.21.15.27:0/2114506863 conn(0x7fad1c0b2a10 msgr2=0x7fad1c0cd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:48.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.490+0000 7fad22e00640 1 -- 172.21.15.27:0/2114506863 shutdown_connections 2024-01-15T18:05:48.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.490+0000 7fad22e00640 1 -- 172.21.15.27:0/2114506863 wait complete. 2024-01-15T18:05:48.516 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-15T18:05:48.516 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:05:48.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.650+0000 7f6615000640 1 Processor -- start 2024-01-15T18:05:48.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.650+0000 7f6615000640 1 -- start start 2024-01-15T18:05:48.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.650+0000 7f6615000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f661015d240 0x7f661006b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:48.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.650+0000 7f6615000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66100ce570 0x7f661006bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:48.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.650+0000 7f6615000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66100cfe30 con 0x7f661006c4b0 2024-01-15T18:05:48.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.650+0000 7f6615000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f661006c960 con 0x7f661015d240 2024-01-15T18:05:48.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.650+0000 7f6615000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66100cf500 con 0x7f66100ce570 2024-01-15T18:05:48.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.650+0000 7f660e800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f661006c4b0 0x7f661015d620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41898/0 (socket says 172.21.15.27:41898) 2024-01-15T18:05:48.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.650+0000 7f660e800640 1 -- 172.21.15.27:0/3917135938 learned_addr learned my addr 172.21.15.27:0/3917135938 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:48.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.654+0000 7f660d400640 1 --2- 172.21.15.27:0/3917135938 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66100ce570 0x7f661006bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:48.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.654+0000 7f660de00640 1 --2- 172.21.15.27:0/3917135938 >> v2:172.21.15.93:3300/0 conn(0x7f661015d240 0x7f661006b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:48.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.654+0000 7f660ca00640 1 -- 172.21.15.27:0/3917135938 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300939797 0 0) 0x7f66100cfe30 con 0x7f661006c4b0 2024-01-15T18:05:48.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.654+0000 7f660ca00640 1 -- 172.21.15.27:0/3917135938 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65f0003530 con 0x7f661006c4b0 2024-01-15T18:05:48.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.654+0000 7f660d400640 1 -- 172.21.15.27:0/3917135938 >> v2:172.21.15.93:3300/0 conn(0x7f661015d240 msgr2=0x7f661006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:48.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.654+0000 7f660d400640 1 --2- 172.21.15.27:0/3917135938 >> v2:172.21.15.93:3300/0 conn(0x7f661015d240 0x7f661006b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.654+0000 7f660ca00640 1 -- 172.21.15.27:0/3917135938 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1463642706 0 0) 0x7f65f0003530 con 0x7f661006c4b0 2024-01-15T18:05:48.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.654+0000 7f660d400640 1 -- 172.21.15.27:0/3917135938 >> v1:172.21.15.27:6789/0 conn(0x7f661006c4b0 legacy=0x7f661015d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.654+0000 7f660d400640 1 -- 172.21.15.27:0/3917135938 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66100cf7e0 con 0x7f66100ce570 2024-01-15T18:05:48.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.654+0000 7f660d400640 1 --2- 172.21.15.27:0/3917135938 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66100ce570 0x7f661006bf70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f65f80097f0 tx=0x7f65f801c370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d5d71e76e4e1bd8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:48.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.658+0000 7f660ca00640 1 -- 172.21.15.27:0/3917135938 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6608001a00 con 0x7f66100ce570 2024-01-15T18:05:48.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.658+0000 7f660ca00640 1 -- 172.21.15.27:0/3917135938 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f80ac440 con 0x7f66100ce570 2024-01-15T18:05:48.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.658+0000 7f660ca00640 1 -- 172.21.15.27:0/3917135938 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f80ac7d0 con 0x7f66100ce570 2024-01-15T18:05:48.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.658+0000 7f6615000640 1 -- 172.21.15.27:0/3917135938 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66100ce570 msgr2=0x7f661006bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.658+0000 7f6615000640 1 --2- 172.21.15.27:0/3917135938 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66100ce570 0x7f661006bf70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f65f80097f0 tx=0x7f65f801c370 comp rx=0 tx=0).stop 2024-01-15T18:05:48.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.658+0000 7f6615000640 1 -- 172.21.15.27:0/3917135938 shutdown_connections 2024-01-15T18:05:48.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.658+0000 7f6615000640 1 --2- 172.21.15.27:0/3917135938 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66100ce570 0x7f661006bf70 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.658+0000 7f6615000640 1 --2- 172.21.15.27:0/3917135938 >> v2:172.21.15.93:3300/0 conn(0x7f661015d240 0x7f661006b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.658+0000 7f6615000640 1 -- 172.21.15.27:0/3917135938 >> 172.21.15.27:0/3917135938 conn(0x7f66100d0f20 msgr2=0x7f66100d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:48.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.658+0000 7f6615000640 1 -- 172.21.15.27:0/3917135938 shutdown_connections 2024-01-15T18:05:48.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.658+0000 7f6615000640 1 -- 172.21.15.27:0/3917135938 wait complete. 2024-01-15T18:05:48.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.658+0000 7f6615000640 1 Processor -- start 2024-01-15T18:05:48.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.658+0000 7f6615000640 1 -- start start 2024-01-15T18:05:48.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.658+0000 7f6615000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f661006c4b0 0x7f661009c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:48.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.658+0000 7f6615000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66100ce570 0x7f661009cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:48.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.662+0000 7f6615000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f661015d240 0x7f6610152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:48.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.662+0000 7f6615000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661015c2e0 con 0x7f66100ce570 2024-01-15T18:05:48.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.662+0000 7f6615000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661015b6d0 con 0x7f661006c4b0 2024-01-15T18:05:48.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.662+0000 7f6615000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f661015b9b0 con 0x7f661015d240 2024-01-15T18:05:48.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.662+0000 7f660d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66100ce570 0x7f661009cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:48.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.662+0000 7f660d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66100ce570 0x7f661009cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38974/0 (socket says 172.21.15.27:38974) 2024-01-15T18:05:48.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.662+0000 7f660d400640 1 -- 172.21.15.27:0/386877491 learned_addr learned my addr 172.21.15.27:0/386877491 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:48.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.662+0000 7f660de00640 1 --2- 172.21.15.27:0/386877491 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f661006c4b0 0x7f661009c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:48.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.662+0000 7f660e800640 1 --2- 172.21.15.27:0/386877491 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f661015d240 0x7f6610152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:48.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.662+0000 7f660d400640 1 -- 172.21.15.27:0/386877491 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f661015d240 msgr2=0x7f6610152ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.662+0000 7f660d400640 1 --2- 172.21.15.27:0/386877491 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f661015d240 0x7f6610152ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.662+0000 7f660d400640 1 -- 172.21.15.27:0/386877491 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f661006c4b0 msgr2=0x7f661009c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.662+0000 7f660d400640 1 --2- 172.21.15.27:0/386877491 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f661006c4b0 0x7f661009c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.662+0000 7f660d400640 1 -- 172.21.15.27:0/386877491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f8009420 con 0x7f66100ce570 2024-01-15T18:05:48.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.662+0000 7f660e800640 1 --2- 172.21.15.27:0/386877491 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f661015d240 0x7f6610152ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:48.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.662+0000 7f660de00640 1 --2- 172.21.15.27:0/386877491 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f661006c4b0 0x7f661009c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:48.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.662+0000 7f660d400640 1 --2- 172.21.15.27:0/386877491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66100ce570 0x7f661009cca0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f65f801aa20 tx=0x7f65f8019110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:48.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.662+0000 7f6606000640 1 -- 172.21.15.27:0/386877491 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f80b5610 con 0x7f66100ce570 2024-01-15T18:05:48.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.662+0000 7f6606000640 1 -- 172.21.15.27:0/386877491 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f80b5790 con 0x7f66100ce570 2024-01-15T18:05:48.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.662+0000 7f6606000640 1 -- 172.21.15.27:0/386877491 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f80b5b40 con 0x7f66100ce570 2024-01-15T18:05:48.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.662+0000 7f6615000640 1 -- 172.21.15.27:0/386877491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f661015bc90 con 0x7f66100ce570 2024-01-15T18:05:48.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.662+0000 7f6615000640 1 -- 172.21.15.27:0/386877491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f661009d370 con 0x7f66100ce570 2024-01-15T18:05:48.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.662+0000 7f6615000640 1 -- 172.21.15.27:0/386877491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661006d530 con 0x7f66100ce570 2024-01-15T18:05:48.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.662+0000 7f6606000640 1 -- 172.21.15.27:0/386877491 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f65f80b3040 con 0x7f66100ce570 2024-01-15T18:05:48.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.666+0000 7f6606000640 1 --2- 172.21.15.27:0/386877491 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f65e8066920 0x7f65e8068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:48.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.666+0000 7f6606000640 1 -- 172.21.15.27:0/386877491 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f65f811de40 con 0x7f66100ce570 2024-01-15T18:05:48.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.666+0000 7f660de00640 1 --2- 172.21.15.27:0/386877491 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f65e8066920 0x7f65e8068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:48.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.666+0000 7f660de00640 1 --2- 172.21.15.27:0/386877491 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f65e8066920 0x7f65e8068de0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6600009f50 tx=0x7f6600009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:48.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.670+0000 7f6606000640 1 -- 172.21.15.27:0/386877491 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f65f80eb000 con 0x7f66100ce570 2024-01-15T18:05:48.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.858+0000 7f6615000640 1 -- 172.21.15.27:0/386877491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f661004e850 con 0x7f66100ce570 2024-01-15T18:05:48.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.862+0000 7f6606000640 1 -- 172.21.15.27:0/386877491 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7f65f80b1020 con 0x7f66100ce570 2024-01-15T18:05:48.867 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:48.867 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/393360880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":393360880},{"type":"v1","addr":"172.21.15.93:6836","nonce":393360880}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4229,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4063308188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4063308188},{"type":"v1","addr":"172.21.15.27:6835","nonce":4063308188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4305,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/2343128056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":2343128056},{"type":"v1","addr":"172.21.15.93:6837","nonce":2343128056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:04:58.208165+0000","modified":"2024-01-15T18:05:01.225419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2657553192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2657553192},{"type":"v1","addr":"172.21.15.27:6834","nonce":2657553192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T18:05:48.867 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 8 2024-01-15T18:05:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.862+0000 7f6615000640 1 -- 172.21.15.27:0/386877491 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f65e8066920 msgr2=0x7f65e8068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.862+0000 7f6615000640 1 --2- 172.21.15.27:0/386877491 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f65e8066920 0x7f65e8068de0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6600009f50 tx=0x7f6600009190 comp rx=0 tx=0).stop 2024-01-15T18:05:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.862+0000 7f6615000640 1 -- 172.21.15.27:0/386877491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66100ce570 msgr2=0x7f661009cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.862+0000 7f6615000640 1 --2- 172.21.15.27:0/386877491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66100ce570 0x7f661009cca0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f65f801aa20 tx=0x7f65f8019110 comp rx=0 tx=0).stop 2024-01-15T18:05:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.862+0000 7f6615000640 1 -- 172.21.15.27:0/386877491 shutdown_connections 2024-01-15T18:05:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.862+0000 7f6615000640 1 --2- 172.21.15.27:0/386877491 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f65e8066920 0x7f65e8068de0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.862+0000 7f6615000640 1 --2- 172.21.15.27:0/386877491 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f661015d240 0x7f6610152ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.862+0000 7f6615000640 1 --2- 172.21.15.27:0/386877491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66100ce570 0x7f661009cca0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.862+0000 7f6615000640 1 --2- 172.21.15.27:0/386877491 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f661006c4b0 0x7f661009c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.862+0000 7f6615000640 1 -- 172.21.15.27:0/386877491 >> 172.21.15.27:0/386877491 conn(0x7f66100d0f20 msgr2=0x7f66100b0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:48.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.862+0000 7f6615000640 1 -- 172.21.15.27:0/386877491 shutdown_connections 2024-01-15T18:05:48.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:48.862+0000 7f6615000640 1 -- 172.21.15.27:0/386877491 wait complete. 2024-01-15T18:05:48.890 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:05:49.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.026+0000 7fa34be00640 1 Processor -- start 2024-01-15T18:05:49.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.026+0000 7fa34be00640 1 -- start start 2024-01-15T18:05:49.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.026+0000 7fa34be00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa34415e7d0 0x7fa3440cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:49.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.026+0000 7fa34be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa3440cedc0 0x7fa3440cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:49.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.026+0000 7fa34be00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3440b38b0 con 0x7fa3440ce2e0 2024-01-15T18:05:49.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.026+0000 7fa34be00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3440cd550 con 0x7fa3440cedc0 2024-01-15T18:05:49.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.026+0000 7fa34be00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3440cd270 con 0x7fa34415e7d0 2024-01-15T18:05:49.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.026+0000 7fa349e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa3440ce2e0 0x7fa3440cba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41902/0 (socket says 172.21.15.27:41902) 2024-01-15T18:05:49.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.026+0000 7fa349e00640 1 -- 172.21.15.27:0/2317904769 learned_addr learned my addr 172.21.15.27:0/2317904769 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:49.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.026+0000 7fa348a00640 1 --2- 172.21.15.27:0/2317904769 >> v2:172.21.15.27:3300/0 conn(0x7fa3440cedc0 0x7fa3440cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:49.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.026+0000 7fa349400640 1 --2- 172.21.15.27:0/2317904769 >> v2:172.21.15.93:3300/0 conn(0x7fa34415e7d0 0x7fa3440cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:49.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.026+0000 7fa348a00640 1 -- 172.21.15.27:0/2317904769 >> v2:172.21.15.93:3300/0 conn(0x7fa34415e7d0 msgr2=0x7fa3440cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:49.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.026+0000 7fa348a00640 1 --2- 172.21.15.27:0/2317904769 >> v2:172.21.15.93:3300/0 conn(0x7fa34415e7d0 0x7fa3440cf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.026+0000 7fa348a00640 1 -- 172.21.15.27:0/2317904769 >> v1:172.21.15.27:6789/0 conn(0x7fa3440ce2e0 legacy=0x7fa3440cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.026+0000 7fa348a00640 1 -- 172.21.15.27:0/2317904769 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3440b35d0 con 0x7fa3440cedc0 2024-01-15T18:05:49.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.026+0000 7fa343e00640 1 -- 172.21.15.27:0/2317904769 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3694015556 0 0) 0x7fa3440b38b0 con 0x7fa3440ce2e0 2024-01-15T18:05:49.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.030+0000 7fa348a00640 1 --2- 172.21.15.27:0/2317904769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3440cedc0 0x7fa3440cf7e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa344071b00 tx=0x7fa32c01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c3539df8400c9c0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:49.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.030+0000 7fa343e00640 1 -- 172.21.15.27:0/2317904769 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c0ab600 con 0x7fa3440cedc0 2024-01-15T18:05:49.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.030+0000 7fa343e00640 1 -- 172.21.15.27:0/2317904769 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa32c0a7050 con 0x7fa3440cedc0 2024-01-15T18:05:49.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.030+0000 7fa343e00640 1 -- 172.21.15.27:0/2317904769 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c0ab980 con 0x7fa3440cedc0 2024-01-15T18:05:49.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.030+0000 7fa34be00640 1 -- 172.21.15.27:0/2317904769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3440cedc0 msgr2=0x7fa3440cf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.030+0000 7fa34be00640 1 --2- 172.21.15.27:0/2317904769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3440cedc0 0x7fa3440cf7e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa344071b00 tx=0x7fa32c01c3d0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.030+0000 7fa34be00640 1 -- 172.21.15.27:0/2317904769 shutdown_connections 2024-01-15T18:05:49.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.030+0000 7fa34be00640 1 --2- 172.21.15.27:0/2317904769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3440cedc0 0x7fa3440cf7e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.030+0000 7fa34be00640 1 --2- 172.21.15.27:0/2317904769 >> v2:172.21.15.93:3300/0 conn(0x7fa34415e7d0 0x7fa3440cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.030+0000 7fa34be00640 1 -- 172.21.15.27:0/2317904769 >> 172.21.15.27:0/2317904769 conn(0x7fa3440b2a10 msgr2=0x7fa3440b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:49.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.030+0000 7fa34be00640 1 -- 172.21.15.27:0/2317904769 shutdown_connections 2024-01-15T18:05:49.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.030+0000 7fa34be00640 1 -- 172.21.15.27:0/2317904769 wait complete. 2024-01-15T18:05:49.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa34be00640 1 Processor -- start 2024-01-15T18:05:49.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa34be00640 1 -- start start 2024-01-15T18:05:49.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa34be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3440ce2e0 0x7fa34414e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:49.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa34be00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa3440cedc0 0x7fa3440c6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:49.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa34be00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa34415e7d0 0x7fa3440c4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:49.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa34be00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3440c62e0 con 0x7fa3440ce2e0 2024-01-15T18:05:49.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa34be00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3440c56d0 con 0x7fa3440cedc0 2024-01-15T18:05:49.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa34be00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3440c59b0 con 0x7fa34415e7d0 2024-01-15T18:05:49.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa349400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3440ce2e0 0x7fa34414e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:49.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa349400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3440ce2e0 0x7fa34414e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38986/0 (socket says 172.21.15.27:38986) 2024-01-15T18:05:49.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa349400640 1 -- 172.21.15.27:0/3142991004 learned_addr learned my addr 172.21.15.27:0/3142991004 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:49.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa348a00640 1 --2- 172.21.15.27:0/3142991004 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa3440cedc0 0x7fa3440c6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:49.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa349e00640 1 --2- 172.21.15.27:0/3142991004 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa34415e7d0 0x7fa3440c4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:49.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa349400640 1 -- 172.21.15.27:0/3142991004 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa34415e7d0 msgr2=0x7fa3440c4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa349400640 1 --2- 172.21.15.27:0/3142991004 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa34415e7d0 0x7fa3440c4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa349400640 1 -- 172.21.15.27:0/3142991004 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa3440cedc0 msgr2=0x7fa3440c6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa349400640 1 --2- 172.21.15.27:0/3142991004 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa3440cedc0 0x7fa3440c6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa349400640 1 -- 172.21.15.27:0/3142991004 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32c009420 con 0x7fa3440ce2e0 2024-01-15T18:05:49.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa349e00640 1 --2- 172.21.15.27:0/3142991004 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa34415e7d0 0x7fa3440c4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:49.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa349400640 1 --2- 172.21.15.27:0/3142991004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3440ce2e0 0x7fa34414e640 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa33400e930 tx=0x7fa33400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:49.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa341600640 1 -- 172.21.15.27:0/3142991004 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa334009690 con 0x7fa3440ce2e0 2024-01-15T18:05:49.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa341600640 1 -- 172.21.15.27:0/3142991004 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa33400ca00 con 0x7fa3440ce2e0 2024-01-15T18:05:49.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa341600640 1 -- 172.21.15.27:0/3142991004 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33400cd70 con 0x7fa3440ce2e0 2024-01-15T18:05:49.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa34be00640 1 -- 172.21.15.27:0/3142991004 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3440c5c90 con 0x7fa3440ce2e0 2024-01-15T18:05:49.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa34be00640 1 -- 172.21.15.27:0/3142991004 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3440b4ce0 con 0x7fa3440ce2e0 2024-01-15T18:05:49.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.034+0000 7fa34be00640 1 -- 172.21.15.27:0/3142991004 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa34404e850 con 0x7fa3440ce2e0 2024-01-15T18:05:49.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.038+0000 7fa341600640 1 -- 172.21.15.27:0/3142991004 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fa334013470 con 0x7fa3440ce2e0 2024-01-15T18:05:49.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.038+0000 7fa341600640 1 --2- 172.21.15.27:0/3142991004 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa31c0668f0 0x7fa31c068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:49.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.038+0000 7fa341600640 1 -- 172.21.15.27:0/3142991004 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fa3340846a0 con 0x7fa3440ce2e0 2024-01-15T18:05:49.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.038+0000 7fa348a00640 1 --2- 172.21.15.27:0/3142991004 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa31c0668f0 0x7fa31c068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:49.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.038+0000 7fa348a00640 1 --2- 172.21.15.27:0/3142991004 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa31c0668f0 0x7fa31c068db0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa32c0ad000 tx=0x7fa32c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:49.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.042+0000 7fa341600640 1 -- 172.21.15.27:0/3142991004 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa3340519a0 con 0x7fa3440ce2e0 2024-01-15T18:05:49.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.234+0000 7fa34be00640 1 -- 172.21.15.27:0/3142991004 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3440d02e0 con 0x7fa3440ce2e0 2024-01-15T18:05:49.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.234+0000 7fa341600640 1 -- 172.21.15.27:0/3142991004 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4904 (secure 0 0 0) 0x7fa334054fc0 con 0x7fa3440ce2e0 2024-01-15T18:05:49.240 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:49.240 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/393360880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":393360880},{"type":"v1","addr":"172.21.15.93:6836","nonce":393360880}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4229,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4063308188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4063308188},{"type":"v1","addr":"172.21.15.27:6835","nonce":4063308188}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4305,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/2343128056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":2343128056},{"type":"v1","addr":"172.21.15.93:6837","nonce":2343128056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:04:58.208165+0000","modified":"2024-01-15T18:05:01.225419+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2657553192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2657553192},{"type":"v1","addr":"172.21.15.27:6834","nonce":2657553192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T18:05:49.240 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 8 2024-01-15T18:05:49.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.234+0000 7fa34be00640 1 -- 172.21.15.27:0/3142991004 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa31c0668f0 msgr2=0x7fa31c068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.234+0000 7fa34be00640 1 --2- 172.21.15.27:0/3142991004 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa31c0668f0 0x7fa31c068db0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa32c0ad000 tx=0x7fa32c008040 comp rx=0 tx=0).stop 2024-01-15T18:05:49.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.234+0000 7fa34be00640 1 -- 172.21.15.27:0/3142991004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3440ce2e0 msgr2=0x7fa34414e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.234+0000 7fa34be00640 1 --2- 172.21.15.27:0/3142991004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3440ce2e0 0x7fa34414e640 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa33400e930 tx=0x7fa33400edf0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.234+0000 7fa34be00640 1 -- 172.21.15.27:0/3142991004 shutdown_connections 2024-01-15T18:05:49.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.234+0000 7fa34be00640 1 --2- 172.21.15.27:0/3142991004 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa31c0668f0 0x7fa31c068db0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.234+0000 7fa34be00640 1 --2- 172.21.15.27:0/3142991004 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa34415e7d0 0x7fa3440c4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.234+0000 7fa34be00640 1 --2- 172.21.15.27:0/3142991004 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa3440cedc0 0x7fa3440c6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.234+0000 7fa34be00640 1 --2- 172.21.15.27:0/3142991004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3440ce2e0 0x7fa34414e640 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.234+0000 7fa34be00640 1 -- 172.21.15.27:0/3142991004 >> 172.21.15.27:0/3142991004 conn(0x7fa3440b2a10 msgr2=0x7fa34415d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:49.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.234+0000 7fa34be00640 1 -- 172.21.15.27:0/3142991004 shutdown_connections 2024-01-15T18:05:49.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.234+0000 7fa34be00640 1 -- 172.21.15.27:0/3142991004 wait complete. 2024-01-15T18:05:49.263 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:49.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.398+0000 7f62cfe00640 1 Processor -- start 2024-01-15T18:05:49.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.398+0000 7f62cfe00640 1 -- start start 2024-01-15T18:05:49.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.398+0000 7f62cfe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f62c815c130 0x7f62c806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:49.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.398+0000 7f62cfe00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c806c4a0 con 0x7f62c80701a0 2024-01-15T18:05:49.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.398+0000 7f62cfe00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c815c660 con 0x7f62c806e240 2024-01-15T18:05:49.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.398+0000 7f62cfe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62c815c940 con 0x7f62c815c130 2024-01-15T18:05:49.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.398+0000 7f62cca00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f62c815c130 0x7f62c806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:49.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.398+0000 7f62cd400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f62c80701a0 0x7f62c8070580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41918/0 (socket says 172.21.15.27:41918) 2024-01-15T18:05:49.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.398+0000 7f62cd400640 1 -- 172.21.15.27:0/2394684804 learned_addr learned my addr 172.21.15.27:0/2394684804 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:49.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.398+0000 7f62cca00640 1 -- 172.21.15.27:0/2394684804 >> v1:172.21.15.93:6789/0 conn(0x7f62c806e240 legacy=0x7f62c815d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.398+0000 7f62cca00640 1 -- 172.21.15.27:0/2394684804 >> v1:172.21.15.27:6789/0 conn(0x7f62c80701a0 legacy=0x7f62c8070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.398+0000 7f62cca00640 1 -- 172.21.15.27:0/2394684804 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c815cc20 con 0x7f62c815c130 2024-01-15T18:05:49.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.398+0000 7f62c7e00640 1 -- 172.21.15.27:0/2394684804 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268891506 0 0) 0x7f62c806c4a0 con 0x7f62c80701a0 2024-01-15T18:05:49.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.402+0000 7f62cca00640 1 --2- 172.21.15.27:0/2394684804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62c815c130 0x7f62c806dd00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f62b00097c0 tx=0x7f62b001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6b7550081808bd8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:49.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.402+0000 7f62c7e00640 1 -- 172.21.15.27:0/2394684804 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b00ab650 con 0x7f62c815c130 2024-01-15T18:05:49.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.402+0000 7f62c7e00640 1 -- 172.21.15.27:0/2394684804 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b00ab7b0 con 0x7f62c815c130 2024-01-15T18:05:49.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.402+0000 7f62c7e00640 1 -- 172.21.15.27:0/2394684804 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b00abb40 con 0x7f62c815c130 2024-01-15T18:05:49.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.402+0000 7f62cfe00640 1 -- 172.21.15.27:0/2394684804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62c815c130 msgr2=0x7f62c806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.402+0000 7f62cfe00640 1 --2- 172.21.15.27:0/2394684804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62c815c130 0x7f62c806dd00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f62b00097c0 tx=0x7f62b001c450 comp rx=0 tx=0).stop 2024-01-15T18:05:49.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.402+0000 7f62cfe00640 1 -- 172.21.15.27:0/2394684804 shutdown_connections 2024-01-15T18:05:49.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.402+0000 7f62cfe00640 1 --2- 172.21.15.27:0/2394684804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62c815c130 0x7f62c806dd00 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.402+0000 7f62cfe00640 1 -- 172.21.15.27:0/2394684804 >> 172.21.15.27:0/2394684804 conn(0x7f62c80cec60 msgr2=0x7f62c80cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:49.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.402+0000 7f62cfe00640 1 -- 172.21.15.27:0/2394684804 shutdown_connections 2024-01-15T18:05:49.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.402+0000 7f62cfe00640 1 -- 172.21.15.27:0/2394684804 wait complete. 2024-01-15T18:05:49.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.402+0000 7f62cfe00640 1 Processor -- start 2024-01-15T18:05:49.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.406+0000 7f62cfe00640 1 -- start start 2024-01-15T18:05:49.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.406+0000 7f62cfe00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62c806e240 0x7f62c80c8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:49.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.406+0000 7f62cfe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f62c80701a0 0x7f62c814b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:49.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.406+0000 7f62cfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62c815c130 0x7f62c80c6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:49.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.406+0000 7f62cfe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c80c8070 con 0x7f62c815c130 2024-01-15T18:05:49.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.406+0000 7f62cfe00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c814bc60 con 0x7f62c806e240 2024-01-15T18:05:49.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.406+0000 7f62cfe00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c80c7460 con 0x7f62c80701a0 2024-01-15T18:05:49.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.406+0000 7f62cde00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62c815c130 0x7f62c80c6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:49.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.406+0000 7f62cde00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62c815c130 0x7f62c80c6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39014/0 (socket says 172.21.15.27:39014) 2024-01-15T18:05:49.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.406+0000 7f62cde00640 1 -- 172.21.15.27:0/2895533951 learned_addr learned my addr 172.21.15.27:0/2895533951 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:49.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.406+0000 7f62cd400640 1 --2- 172.21.15.27:0/2895533951 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62c806e240 0x7f62c80c8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:49.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.406+0000 7f62cca00640 1 --2- 172.21.15.27:0/2895533951 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f62c80701a0 0x7f62c814b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:49.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.406+0000 7f62cde00640 1 -- 172.21.15.27:0/2895533951 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f62c80701a0 msgr2=0x7f62c814b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.406+0000 7f62cde00640 1 --2- 172.21.15.27:0/2895533951 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f62c80701a0 0x7f62c814b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.406+0000 7f62cde00640 1 -- 172.21.15.27:0/2895533951 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62c806e240 msgr2=0x7f62c80c8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.406+0000 7f62cde00640 1 --2- 172.21.15.27:0/2895533951 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62c806e240 0x7f62c80c8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.406+0000 7f62cde00640 1 -- 172.21.15.27:0/2895533951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b0009420 con 0x7f62c815c130 2024-01-15T18:05:49.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.406+0000 7f62cd400640 1 --2- 172.21.15.27:0/2895533951 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62c806e240 0x7f62c80c8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:49.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.406+0000 7f62cde00640 1 --2- 172.21.15.27:0/2895533951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62c815c130 0x7f62c80c6740 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f62c000c7a0 tx=0x7f62c000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:49.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.406+0000 7f62c5600640 1 -- 172.21.15.27:0/2895533951 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c0003b80 con 0x7f62c815c130 2024-01-15T18:05:49.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.406+0000 7f62c5600640 1 -- 172.21.15.27:0/2895533951 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c0003ce0 con 0x7f62c815c130 2024-01-15T18:05:49.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.406+0000 7f62c5600640 1 -- 172.21.15.27:0/2895533951 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c000aa90 con 0x7f62c815c130 2024-01-15T18:05:49.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.406+0000 7f62cfe00640 1 -- 172.21.15.27:0/2895533951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c80c7740 con 0x7f62c815c130 2024-01-15T18:05:49.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.406+0000 7f62cfe00640 1 -- 172.21.15.27:0/2895533951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c80c7d00 con 0x7f62c815c130 2024-01-15T18:05:49.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.406+0000 7f62cfe00640 1 -- 172.21.15.27:0/2895533951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c80c8220 con 0x7f62c815c130 2024-01-15T18:05:49.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.410+0000 7f62c5600640 1 -- 172.21.15.27:0/2895533951 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f62c000abf0 con 0x7f62c815c130 2024-01-15T18:05:49.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.410+0000 7f62c5600640 1 --2- 172.21.15.27:0/2895533951 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f62a00668f0 0x7f62a0068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:49.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.410+0000 7f62c5600640 1 -- 172.21.15.27:0/2895533951 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f62c00842d0 con 0x7f62c815c130 2024-01-15T18:05:49.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.410+0000 7f62cd400640 1 --2- 172.21.15.27:0/2895533951 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f62a00668f0 0x7f62a0068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:49.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.410+0000 7f62cd400640 1 --2- 172.21.15.27:0/2895533951 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f62a00668f0 0x7f62a0068db0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f62b8000f70 tx=0x7f62b8007c60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:49.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.414+0000 7f62c5600640 1 -- 172.21.15.27:0/2895533951 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f62c80c8220 con 0x7f62c815c130 2024-01-15T18:05:49.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.570+0000 7f62cfe00640 1 -- 172.21.15.27:0/2895533951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62c80c8220 con 0x7f62c815c130 2024-01-15T18:05:49.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.574+0000 7f62c5600640 1 -- 172.21.15.27:0/2895533951 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16705 (secure 0 0 0) 0x7f62c80c8220 con 0x7f62c815c130 2024-01-15T18:05:49.580 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:49.580 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":23,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:05:40.008536+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:04:56.220608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T18:04:57.210421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.574+0000 7f62cfe00640 1 -- 172.21.15.27:0/2895533951 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f62a00668f0 msgr2=0x7f62a0068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.574+0000 7f62cfe00640 1 --2- 172.21.15.27:0/2895533951 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f62a00668f0 0x7f62a0068db0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f62b8000f70 tx=0x7f62b8007c60 comp rx=0 tx=0).stop 2024-01-15T18:05:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.574+0000 7f62cfe00640 1 -- 172.21.15.27:0/2895533951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62c815c130 msgr2=0x7f62c80c6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.574+0000 7f62cfe00640 1 --2- 172.21.15.27:0/2895533951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62c815c130 0x7f62c80c6740 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f62c000c7a0 tx=0x7f62c000cc60 comp rx=0 tx=0).stop 2024-01-15T18:05:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.574+0000 7f62cfe00640 1 -- 172.21.15.27:0/2895533951 shutdown_connections 2024-01-15T18:05:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.574+0000 7f62cfe00640 1 --2- 172.21.15.27:0/2895533951 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f62a00668f0 0x7f62a0068db0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.574+0000 7f62cfe00640 1 --2- 172.21.15.27:0/2895533951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62c815c130 0x7f62c80c6740 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.574+0000 7f62cfe00640 1 --2- 172.21.15.27:0/2895533951 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f62c80701a0 0x7f62c814b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.574+0000 7f62cfe00640 1 --2- 172.21.15.27:0/2895533951 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f62c806e240 0x7f62c80c8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.574+0000 7f62cfe00640 1 -- 172.21.15.27:0/2895533951 >> 172.21.15.27:0/2895533951 conn(0x7f62c80cec60 msgr2=0x7f62c8070580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.574+0000 7f62cfe00640 1 -- 172.21.15.27:0/2895533951 shutdown_connections 2024-01-15T18:05:49.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.574+0000 7f62cfe00640 1 -- 172.21.15.27:0/2895533951 wait complete. 2024-01-15T18:05:49.601 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-15T18:05:49.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.738+0000 7f06b5c00640 1 Processor -- start 2024-01-15T18:05:49.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.738+0000 7f06b5c00640 1 -- start start 2024-01-15T18:05:49.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.738+0000 7f06b5c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f06b015d680 0x7f06b00cfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:49.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.738+0000 7f06b5c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f06b00cde10 0x7f06b00cc150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:49.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.738+0000 7f06b5c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b00b3cd0 con 0x7f06b015d2a0 2024-01-15T18:05:49.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.738+0000 7f06b5c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06b00d0260 con 0x7f06b00cde10 2024-01-15T18:05:49.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.738+0000 7f06b5c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06b00cb670 con 0x7f06b015d680 2024-01-15T18:05:49.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.738+0000 7f06af400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f06b00cde10 0x7f06b00cc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:49.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.738+0000 7f06aea00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f06b015d2a0 0x7f06b00cd700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41934/0 (socket says 172.21.15.27:41934) 2024-01-15T18:05:49.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.738+0000 7f06aea00640 1 -- 172.21.15.27:0/1820126793 learned_addr learned my addr 172.21.15.27:0/1820126793 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:49.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.738+0000 7f06ae000640 1 --2- 172.21.15.27:0/1820126793 >> v2:172.21.15.93:3300/0 conn(0x7f06b015d680 0x7f06b00cfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:49.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.738+0000 7f06af400640 1 -- 172.21.15.27:0/1820126793 >> v2:172.21.15.93:3300/0 conn(0x7f06b015d680 msgr2=0x7f06b00cfbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:49.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.738+0000 7f06af400640 1 --2- 172.21.15.27:0/1820126793 >> v2:172.21.15.93:3300/0 conn(0x7f06b015d680 0x7f06b00cfbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.738+0000 7f06af400640 1 -- 172.21.15.27:0/1820126793 >> v1:172.21.15.27:6789/0 conn(0x7f06b015d2a0 legacy=0x7f06b00cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.738+0000 7f06af400640 1 -- 172.21.15.27:0/1820126793 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b00cb950 con 0x7f06b00cde10 2024-01-15T18:05:49.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.738+0000 7f06ad600640 1 -- 172.21.15.27:0/1820126793 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1909930774 0 0) 0x7f06b00b3cd0 con 0x7f06b015d2a0 2024-01-15T18:05:49.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.742+0000 7f06af400640 1 --2- 172.21.15.27:0/1820126793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06b00cde10 0x7f06b00cc150 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f06b006d610 tx=0x7f06a801c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=758a1df5a7430a46 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:49.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.742+0000 7f06ad600640 1 -- 172.21.15.27:0/1820126793 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a80a58f0 con 0x7f06b00cde10 2024-01-15T18:05:49.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.742+0000 7f06ad600640 1 -- 172.21.15.27:0/1820126793 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a80a5a50 con 0x7f06b00cde10 2024-01-15T18:05:49.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.742+0000 7f06ad600640 1 -- 172.21.15.27:0/1820126793 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a80a5de0 con 0x7f06b00cde10 2024-01-15T18:05:49.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.742+0000 7f06b5c00640 1 -- 172.21.15.27:0/1820126793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06b00cde10 msgr2=0x7f06b00cc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.742+0000 7f06b5c00640 1 --2- 172.21.15.27:0/1820126793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06b00cde10 0x7f06b00cc150 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f06b006d610 tx=0x7f06a801c850 comp rx=0 tx=0).stop 2024-01-15T18:05:49.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.742+0000 7f06b5c00640 1 -- 172.21.15.27:0/1820126793 shutdown_connections 2024-01-15T18:05:49.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.742+0000 7f06b5c00640 1 --2- 172.21.15.27:0/1820126793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06b00cde10 0x7f06b00cc150 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.742+0000 7f06b5c00640 1 --2- 172.21.15.27:0/1820126793 >> v2:172.21.15.93:3300/0 conn(0x7f06b015d680 0x7f06b00cfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.742+0000 7f06b5c00640 1 -- 172.21.15.27:0/1820126793 >> 172.21.15.27:0/1820126793 conn(0x7f06b00d0e80 msgr2=0x7f06b00d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:49.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.742+0000 7f06b5c00640 1 -- 172.21.15.27:0/1820126793 shutdown_connections 2024-01-15T18:05:49.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.742+0000 7f06b5c00640 1 -- 172.21.15.27:0/1820126793 wait complete. 2024-01-15T18:05:49.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06b5c00640 1 Processor -- start 2024-01-15T18:05:49.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06b5c00640 1 -- start start 2024-01-15T18:05:49.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06b5c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06b00cde10 0x7f06b0159200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:49.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06b5c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06b015d2a0 0x7f06b015c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:49.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06b5c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06b015d680 0x7f06b01566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:49.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06b5c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b014de40 con 0x7f06b015d2a0 2024-01-15T18:05:49.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06b5c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b014e120 con 0x7f06b015d680 2024-01-15T18:05:49.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06b5c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b014e400 con 0x7f06b00cde10 2024-01-15T18:05:49.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06ae000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06b015d2a0 0x7f06b015c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:49.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06ae000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06b015d2a0 0x7f06b015c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39038/0 (socket says 172.21.15.27:39038) 2024-01-15T18:05:49.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06ae000640 1 -- 172.21.15.27:0/4226123860 learned_addr learned my addr 172.21.15.27:0/4226123860 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:49.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06aea00640 1 --2- 172.21.15.27:0/4226123860 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06b00cde10 0x7f06b0159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:49.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06af400640 1 --2- 172.21.15.27:0/4226123860 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06b015d680 0x7f06b01566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:49.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06ae000640 1 -- 172.21.15.27:0/4226123860 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06b00cde10 msgr2=0x7f06b0159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06ae000640 1 --2- 172.21.15.27:0/4226123860 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06b00cde10 0x7f06b0159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06ae000640 1 -- 172.21.15.27:0/4226123860 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06b015d680 msgr2=0x7f06b01566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06ae000640 1 --2- 172.21.15.27:0/4226123860 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06b015d680 0x7f06b01566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06ae000640 1 -- 172.21.15.27:0/4226123860 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a8009a70 con 0x7f06b015d2a0 2024-01-15T18:05:49.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06af400640 1 --2- 172.21.15.27:0/4226123860 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06b015d680 0x7f06b01566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:49.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06ae000640 1 --2- 172.21.15.27:0/4226123860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06b015d2a0 0x7f06b015c380 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f069800e870 tx=0x7f069800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:49.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06a6a00640 1 -- 172.21.15.27:0/4226123860 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0698009510 con 0x7f06b015d2a0 2024-01-15T18:05:49.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06a6a00640 1 -- 172.21.15.27:0/4226123860 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f069800eec0 con 0x7f06b015d2a0 2024-01-15T18:05:49.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06a6a00640 1 -- 172.21.15.27:0/4226123860 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069800cb40 con 0x7f06b015d2a0 2024-01-15T18:05:49.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06b5c00640 1 -- 172.21.15.27:0/4226123860 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b014e6e0 con 0x7f06b015d2a0 2024-01-15T18:05:49.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06b5c00640 1 -- 172.21.15.27:0/4226123860 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b014d9d0 con 0x7f06b015d2a0 2024-01-15T18:05:49.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.746+0000 7f06b5c00640 1 -- 172.21.15.27:0/4226123860 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b004e850 con 0x7f06b015d2a0 2024-01-15T18:05:49.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.750+0000 7f06a6a00640 1 -- 172.21.15.27:0/4226123860 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f069800cca0 con 0x7f06b015d2a0 2024-01-15T18:05:49.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.750+0000 7f06a6a00640 1 --2- 172.21.15.27:0/4226123860 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0688066920 0x7f0688068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:49.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.750+0000 7f06a6a00640 1 -- 172.21.15.27:0/4226123860 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f0698084530 con 0x7f06b015d2a0 2024-01-15T18:05:49.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.750+0000 7f06aea00640 1 --2- 172.21.15.27:0/4226123860 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0688066920 0x7f0688068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:49.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.750+0000 7f06aea00640 1 --2- 172.21.15.27:0/4226123860 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0688066920 0x7f0688068de0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f06a0000f70 tx=0x7f06a0007b60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:49.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.754+0000 7f06a6a00640 1 -- 172.21.15.27:0/4226123860 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f06980516d0 con 0x7f06b015d2a0 2024-01-15T18:05:49.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.942+0000 7f06b5c00640 1 -- 172.21.15.27:0/4226123860 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f06b0156130 con 0x7f06b015d2a0 2024-01-15T18:05:49.949 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-01-15T18:05:49.942+0000 7f053dc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-15T18:05:49.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.946+0000 7f06a6a00640 1 -- 172.21.15.27:0/4226123860 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0698054cf0 con 0x7f06b015d2a0 2024-01-15T18:05:49.951 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-15T18:05:49.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.946+0000 7f06b5c00640 1 -- 172.21.15.27:0/4226123860 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0688066920 msgr2=0x7f0688068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.946+0000 7f06b5c00640 1 --2- 172.21.15.27:0/4226123860 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0688066920 0x7f0688068de0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f06a0000f70 tx=0x7f06a0007b60 comp rx=0 tx=0).stop 2024-01-15T18:05:49.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.946+0000 7f06b5c00640 1 -- 172.21.15.27:0/4226123860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06b015d2a0 msgr2=0x7f06b015c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.946+0000 7f06b5c00640 1 --2- 172.21.15.27:0/4226123860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06b015d2a0 0x7f06b015c380 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f069800e870 tx=0x7f069800ec70 comp rx=0 tx=0).stop 2024-01-15T18:05:49.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.946+0000 7f06b5c00640 1 -- 172.21.15.27:0/4226123860 shutdown_connections 2024-01-15T18:05:49.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.946+0000 7f06b5c00640 1 --2- 172.21.15.27:0/4226123860 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0688066920 0x7f0688068de0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.946+0000 7f06b5c00640 1 --2- 172.21.15.27:0/4226123860 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06b015d680 0x7f06b01566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.946+0000 7f06b5c00640 1 --2- 172.21.15.27:0/4226123860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06b015d2a0 0x7f06b015c380 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.946+0000 7f06b5c00640 1 --2- 172.21.15.27:0/4226123860 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06b00cde10 0x7f06b0159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.946+0000 7f06b5c00640 1 -- 172.21.15.27:0/4226123860 >> 172.21.15.27:0/4226123860 conn(0x7f06b00d0e80 msgr2=0x7f06b015e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:49.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.950+0000 7f06b5c00640 1 -- 172.21.15.27:0/4226123860 shutdown_connections 2024-01-15T18:05:49.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:49.950+0000 7f06b5c00640 1 -- 172.21.15.27:0/4226123860 wait complete. 2024-01-15T18:05:49.961 INFO:tasks.ceph.mds.a.smithi027.stderr: -3077> 2024-01-15T18:05:02.496+0000 7fcf00200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-15T18:05:49.976 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-15T18:05:50.008 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-15T18:05:50.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.114+0000 7f1445600640 1 Processor -- start 2024-01-15T18:05:50.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.114+0000 7f1445600640 1 -- start start 2024-01-15T18:05:50.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.114+0000 7f1445600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f14400cf2e0 0x7f14400701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.114+0000 7f1445600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14400b1cb0 con 0x7f144015f550 2024-01-15T18:05:50.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.114+0000 7f1445600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f144015fa00 con 0x7f1440158900 2024-01-15T18:05:50.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.114+0000 7f1445600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14400b1f00 con 0x7f14400cf2e0 2024-01-15T18:05:50.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.114+0000 7f143e000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f144015f550 0x7f14400cebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41936/0 (socket says 172.21.15.27:41936) 2024-01-15T18:05:50.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.114+0000 7f143e000640 1 -- 172.21.15.27:0/2281990 learned_addr learned my addr 172.21.15.27:0/2281990 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:50.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.114+0000 7f143f400640 1 --2- 172.21.15.27:0/2281990 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f14400cf2e0 0x7f14400701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:50.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.114+0000 7f143d600640 1 -- 172.21.15.27:0/2281990 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 453723087 0 0) 0x7f14400b1cb0 con 0x7f144015f550 2024-01-15T18:05:50.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.114+0000 7f143d600640 1 -- 172.21.15.27:0/2281990 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1420003530 con 0x7f144015f550 2024-01-15T18:05:50.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.114+0000 7f143d600640 1 -- 172.21.15.27:0/2281990 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395769627 0 0) 0x7f144015fa00 con 0x7f1440158900 2024-01-15T18:05:50.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.114+0000 7f143d600640 1 -- 172.21.15.27:0/2281990 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14400b1cb0 con 0x7f1440158900 2024-01-15T18:05:50.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.114+0000 7f143f400640 1 -- 172.21.15.27:0/2281990 >> v1:172.21.15.93:6789/0 conn(0x7f1440158900 legacy=0x7f144015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:50.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.114+0000 7f143f400640 1 -- 172.21.15.27:0/2281990 >> v1:172.21.15.27:6789/0 conn(0x7f144015f550 legacy=0x7f14400cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:50.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.114+0000 7f143f400640 1 -- 172.21.15.27:0/2281990 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14400b21e0 con 0x7f14400cf2e0 2024-01-15T18:05:50.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.118+0000 7f143f400640 1 --2- 172.21.15.27:0/2281990 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f14400cf2e0 0x7f14400701e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1438009dc0 tx=0x7f143801c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3553a143b5b323a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:50.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.118+0000 7f143d600640 1 -- 172.21.15.27:0/2281990 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1440070f30 con 0x7f14400cf2e0 2024-01-15T18:05:50.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.118+0000 7f143d600640 1 -- 172.21.15.27:0/2281990 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14380a5a00 con 0x7f14400cf2e0 2024-01-15T18:05:50.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.118+0000 7f143d600640 1 -- 172.21.15.27:0/2281990 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14380a5d90 con 0x7f14400cf2e0 2024-01-15T18:05:50.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.118+0000 7f1445600640 1 -- 172.21.15.27:0/2281990 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f14400cf2e0 msgr2=0x7f14400701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:50.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.118+0000 7f1445600640 1 --2- 172.21.15.27:0/2281990 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f14400cf2e0 0x7f14400701e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1438009dc0 tx=0x7f143801c940 comp rx=0 tx=0).stop 2024-01-15T18:05:50.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.118+0000 7f1445600640 1 -- 172.21.15.27:0/2281990 shutdown_connections 2024-01-15T18:05:50.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.118+0000 7f1445600640 1 --2- 172.21.15.27:0/2281990 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f14400cf2e0 0x7f14400701e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.118+0000 7f1445600640 1 -- 172.21.15.27:0/2281990 >> 172.21.15.27:0/2281990 conn(0x7f14400cb700 msgr2=0x7f14400cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:50.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.118+0000 7f1445600640 1 -- 172.21.15.27:0/2281990 shutdown_connections 2024-01-15T18:05:50.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.118+0000 7f1445600640 1 -- 172.21.15.27:0/2281990 wait complete. 2024-01-15T18:05:50.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.122+0000 7f1445600640 1 Processor -- start 2024-01-15T18:05:50.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.122+0000 7f1445600640 1 -- start start 2024-01-15T18:05:50.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.122+0000 7f1445600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f14400cf2e0 0x7f14401405e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.122+0000 7f1445600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1440158900 0x7f144015d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.122+0000 7f1445600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144015f550 0x7f14401526c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.122+0000 7f1445600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144015e4f0 con 0x7f144015f550 2024-01-15T18:05:50.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.122+0000 7f1445600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144014f770 con 0x7f1440158900 2024-01-15T18:05:50.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.122+0000 7f1445600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f144015e210 con 0x7f14400cf2e0 2024-01-15T18:05:50.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.122+0000 7f143f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144015f550 0x7f14401526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:50.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.122+0000 7f143ea00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f14400cf2e0 0x7f14401405e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:50.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.122+0000 7f143e000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1440158900 0x7f144015d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:50.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.122+0000 7f143f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144015f550 0x7f14401526c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39072/0 (socket says 172.21.15.27:39072) 2024-01-15T18:05:50.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.122+0000 7f143ea00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f14400cf2e0 0x7f14401405e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:53892/0 (socket says 172.21.15.27:53892) 2024-01-15T18:05:50.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.122+0000 7f143f400640 1 -- 172.21.15.27:0/500669973 learned_addr learned my addr 172.21.15.27:0/500669973 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:50.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.122+0000 7f143f400640 1 -- 172.21.15.27:0/500669973 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f14400cf2e0 msgr2=0x7f14401405e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:50.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.122+0000 7f143f400640 1 --2- 172.21.15.27:0/500669973 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f14400cf2e0 0x7f14401405e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.122+0000 7f143f400640 1 -- 172.21.15.27:0/500669973 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1440158900 msgr2=0x7f144015d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:50.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.122+0000 7f143f400640 1 --2- 172.21.15.27:0/500669973 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1440158900 0x7f144015d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.122+0000 7f143f400640 1 -- 172.21.15.27:0/500669973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1438009a70 con 0x7f144015f550 2024-01-15T18:05:50.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.122+0000 7f143f400640 1 --2- 172.21.15.27:0/500669973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144015f550 0x7f14401526c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f143801a570 tx=0x7f14380196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:50.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.122+0000 7f1436a00640 1 -- 172.21.15.27:0/500669973 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14380b9860 con 0x7f144015f550 2024-01-15T18:05:50.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.122+0000 7f1436a00640 1 -- 172.21.15.27:0/500669973 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14380b99c0 con 0x7f144015f550 2024-01-15T18:05:50.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.122+0000 7f1436a00640 1 -- 172.21.15.27:0/500669973 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14380b9d70 con 0x7f144015f550 2024-01-15T18:05:50.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.122+0000 7f1445600640 1 -- 172.21.15.27:0/500669973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f144014f120 con 0x7f144015f550 2024-01-15T18:05:50.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.126+0000 7f1445600640 1 -- 172.21.15.27:0/500669973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f144015beb0 con 0x7f144015f550 2024-01-15T18:05:50.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.126+0000 7f1445600640 1 -- 172.21.15.27:0/500669973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f144014ef80 con 0x7f144015f550 2024-01-15T18:05:50.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.126+0000 7f1436a00640 1 -- 172.21.15.27:0/500669973 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f14380ad440 con 0x7f144015f550 2024-01-15T18:05:50.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.126+0000 7f1436a00640 1 --2- 172.21.15.27:0/500669973 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f14180668f0 0x7f1418068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.126+0000 7f1436a00640 1 -- 172.21.15.27:0/500669973 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5940+0+0 (secure 0 0 0) 0x7f143811f7e0 con 0x7f144015f550 2024-01-15T18:05:50.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.126+0000 7f143ea00640 1 --2- 172.21.15.27:0/500669973 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f14180668f0 0x7f1418068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:50.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.130+0000 7f143ea00640 1 --2- 172.21.15.27:0/500669973 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f14180668f0 0x7f1418068db0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f14300011f0 tx=0x7f1430009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:50.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.130+0000 7f1436a00640 1 -- 172.21.15.27:0/500669973 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f144014ef80 con 0x7f144015f550 2024-01-15T18:05:50.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.322+0000 7f1445600640 1 -- 172.21.15.27:0/500669973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f144014ef80 con 0x7f144015f550 2024-01-15T18:05:50.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.946+0000 7f1436a00640 1 -- 172.21.15.27:0/500669973 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7f144014ef80 con 0x7f144015f550 2024-01-15T18:05:50.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.950+0000 7f1445600640 1 -- 172.21.15.27:0/500669973 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f14180668f0 msgr2=0x7f1418068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:50.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.950+0000 7f1445600640 1 --2- 172.21.15.27:0/500669973 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f14180668f0 0x7f1418068db0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f14300011f0 tx=0x7f1430009040 comp rx=0 tx=0).stop 2024-01-15T18:05:50.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.950+0000 7f1445600640 1 -- 172.21.15.27:0/500669973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144015f550 msgr2=0x7f14401526c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:50.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.950+0000 7f1445600640 1 --2- 172.21.15.27:0/500669973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144015f550 0x7f14401526c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f143801a570 tx=0x7f14380196e0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.950+0000 7f1445600640 1 -- 172.21.15.27:0/500669973 shutdown_connections 2024-01-15T18:05:50.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.950+0000 7f1445600640 1 --2- 172.21.15.27:0/500669973 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f14180668f0 0x7f1418068db0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.950+0000 7f1445600640 1 --2- 172.21.15.27:0/500669973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f144015f550 0x7f14401526c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.950+0000 7f1445600640 1 --2- 172.21.15.27:0/500669973 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1440158900 0x7f144015d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.950+0000 7f1445600640 1 --2- 172.21.15.27:0/500669973 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f14400cf2e0 0x7f14401405e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.950+0000 7f1445600640 1 -- 172.21.15.27:0/500669973 >> 172.21.15.27:0/500669973 conn(0x7f14400cb700 msgr2=0x7f14400ad2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:50.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.950+0000 7f1445600640 1 -- 172.21.15.27:0/500669973 shutdown_connections 2024-01-15T18:05:50.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:50.950+0000 7f1445600640 1 -- 172.21.15.27:0/500669973 wait complete. 2024-01-15T18:05:50.978 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-15T18:05:51.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.162+0000 7fa8cd600640 1 Processor -- start 2024-01-15T18:05:51.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.162+0000 7fa8cd600640 1 -- start start 2024-01-15T18:05:51.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.162+0000 7fa8cd600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8c815b350 0x7fa8c80cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:51.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.162+0000 7fa8cd600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa8c8099a10 0x7fa8c8099e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:51.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.162+0000 7fa8cd600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c80cfec0 con 0x7fa8c809a5b0 2024-01-15T18:05:51.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.162+0000 7fa8cd600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8c809a3c0 con 0x7fa8c8099a10 2024-01-15T18:05:51.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.162+0000 7fa8cd600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c80ccd50 con 0x7fa8c815b350 2024-01-15T18:05:51.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.162+0000 7fa8c7400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa8c8099a10 0x7fa8c8099e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:51.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.162+0000 7fa8c7400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa8c8099a10 0x7fa8c8099e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39074/0 (socket says 172.21.15.27:39074) 2024-01-15T18:05:51.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.162+0000 7fa8c7400640 1 -- 172.21.15.27:0/3872398893 learned_addr learned my addr 172.21.15.27:0/3872398893 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:51.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.162+0000 7fa8c6a00640 1 --2- 172.21.15.27:0/3872398893 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8c815b350 0x7fa8c80cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:51.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.162+0000 7fa8c7400640 1 -- 172.21.15.27:0/3872398893 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8c815b350 msgr2=0x7fa8c80cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:51.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.162+0000 7fa8c7400640 1 --2- 172.21.15.27:0/3872398893 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8c815b350 0x7fa8c80cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:51.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.162+0000 7fa8c7400640 1 -- 172.21.15.27:0/3872398893 >> v1:172.21.15.93:6789/0 conn(0x7fa8c809a5b0 legacy=0x7fa8c80cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:51.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.162+0000 7fa8c7400640 1 -- 172.21.15.27:0/3872398893 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c80cebc0 con 0x7fa8c8099a10 2024-01-15T18:05:51.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.166+0000 7fa8c7400640 1 --2- 172.21.15.27:0/3872398893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c8099a10 0x7fa8c8099e80 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0009e50 tx=0x7fa8c001c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ff797a2b02c09a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:51.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.166+0000 7fa8c5600640 1 -- 172.21.15.27:0/3872398893 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c00a5720 con 0x7fa8c8099a10 2024-01-15T18:05:51.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.166+0000 7fa8c5600640 1 -- 172.21.15.27:0/3872398893 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c00a5880 con 0x7fa8c8099a10 2024-01-15T18:05:51.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.166+0000 7fa8c5600640 1 -- 172.21.15.27:0/3872398893 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c00a5c10 con 0x7fa8c8099a10 2024-01-15T18:05:51.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.166+0000 7fa8cd600640 1 -- 172.21.15.27:0/3872398893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c8099a10 msgr2=0x7fa8c8099e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:51.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.166+0000 7fa8cd600640 1 --2- 172.21.15.27:0/3872398893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c8099a10 0x7fa8c8099e80 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0009e50 tx=0x7fa8c001c950 comp rx=0 tx=0).stop 2024-01-15T18:05:51.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.166+0000 7fa8cd600640 1 -- 172.21.15.27:0/3872398893 shutdown_connections 2024-01-15T18:05:51.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.166+0000 7fa8cd600640 1 --2- 172.21.15.27:0/3872398893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c8099a10 0x7fa8c8099e80 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:51.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.166+0000 7fa8cd600640 1 --2- 172.21.15.27:0/3872398893 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8c815b350 0x7fa8c80cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:51.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.166+0000 7fa8cd600640 1 -- 172.21.15.27:0/3872398893 >> 172.21.15.27:0/3872398893 conn(0x7fa8c80b2a80 msgr2=0x7fa8c80b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:51.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.166+0000 7fa8cd600640 1 -- 172.21.15.27:0/3872398893 shutdown_connections 2024-01-15T18:05:51.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.166+0000 7fa8cd600640 1 -- 172.21.15.27:0/3872398893 wait complete. 2024-01-15T18:05:51.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.170+0000 7fa8cd600640 1 Processor -- start 2024-01-15T18:05:51.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.170+0000 7fa8cd600640 1 -- start start 2024-01-15T18:05:51.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.170+0000 7fa8cd600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8c8099a10 0x7fa8c8165590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:51.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.170+0000 7fa8cd600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c809a5b0 0x7fa8c8158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:51.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.170+0000 7fa8cd600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8c815b350 0x7fa8c8159630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:51.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.170+0000 7fa8cd600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c806c660 con 0x7fa8c809a5b0 2024-01-15T18:05:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.170+0000 7fa8cd600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c814ff40 con 0x7fa8c815b350 2024-01-15T18:05:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.170+0000 7fa8cd600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c806b9e0 con 0x7fa8c8099a10 2024-01-15T18:05:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.170+0000 7fa8c6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c809a5b0 0x7fa8c8158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.170+0000 7fa8c6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c809a5b0 0x7fa8c8158350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39082/0 (socket says 172.21.15.27:39082) 2024-01-15T18:05:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.170+0000 7fa8c6000640 1 -- 172.21.15.27:0/4034853710 learned_addr learned my addr 172.21.15.27:0/4034853710 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.170+0000 7fa8c6a00640 1 --2- 172.21.15.27:0/4034853710 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8c8099a10 0x7fa8c8165590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.170+0000 7fa8c7400640 1 --2- 172.21.15.27:0/4034853710 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8c815b350 0x7fa8c8159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.170+0000 7fa8c6000640 1 -- 172.21.15.27:0/4034853710 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8c8099a10 msgr2=0x7fa8c8165590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.170+0000 7fa8c6000640 1 --2- 172.21.15.27:0/4034853710 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8c8099a10 0x7fa8c8165590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.170+0000 7fa8c6000640 1 -- 172.21.15.27:0/4034853710 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8c815b350 msgr2=0x7fa8c8159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:51.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.170+0000 7fa8c6000640 1 --2- 172.21.15.27:0/4034853710 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8c815b350 0x7fa8c8159630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:51.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.170+0000 7fa8c6000640 1 -- 172.21.15.27:0/4034853710 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c0009b30 con 0x7fa8c809a5b0 2024-01-15T18:05:51.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.170+0000 7fa8c7400640 1 --2- 172.21.15.27:0/4034853710 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8c815b350 0x7fa8c8159630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:51.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.170+0000 7fa8c6000640 1 --2- 172.21.15.27:0/4034853710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c809a5b0 0x7fa8c8158350 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa8b000c500 tx=0x7fa8b000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:51.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.170+0000 7fa8bea00640 1 -- 172.21.15.27:0/4034853710 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b0005220 con 0x7fa8c809a5b0 2024-01-15T18:05:51.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.170+0000 7fa8bea00640 1 -- 172.21.15.27:0/4034853710 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b0005380 con 0x7fa8c809a5b0 2024-01-15T18:05:51.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.170+0000 7fa8cd600640 1 -- 172.21.15.27:0/4034853710 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c806bcc0 con 0x7fa8c809a5b0 2024-01-15T18:05:51.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.170+0000 7fa8bea00640 1 -- 172.21.15.27:0/4034853710 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b0007270 con 0x7fa8c809a5b0 2024-01-15T18:05:51.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.170+0000 7fa8cd600640 1 -- 172.21.15.27:0/4034853710 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c806c280 con 0x7fa8c809a5b0 2024-01-15T18:05:51.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.174+0000 7fa8cd600640 1 -- 172.21.15.27:0/4034853710 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c80d0720 con 0x7fa8c809a5b0 2024-01-15T18:05:51.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.174+0000 7fa8bea00640 1 -- 172.21.15.27:0/4034853710 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fa8b00073d0 con 0x7fa8c809a5b0 2024-01-15T18:05:51.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.174+0000 7fa8bea00640 1 --2- 172.21.15.27:0/4034853710 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa8a00668f0 0x7fa8a0068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:51.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.174+0000 7fa8bea00640 1 -- 172.21.15.27:0/4034853710 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5940+0+0 (secure 0 0 0) 0x7fa8b0083d20 con 0x7fa8c809a5b0 2024-01-15T18:05:51.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.174+0000 7fa8c6a00640 1 --2- 172.21.15.27:0/4034853710 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa8a00668f0 0x7fa8a0068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:51.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.174+0000 7fa8c6a00640 1 --2- 172.21.15.27:0/4034853710 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa8a00668f0 0x7fa8a0068db0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8007a60 tx=0x7fa8b8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:51.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.178+0000 7fa8bea00640 1 -- 172.21.15.27:0/4034853710 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa8b0050e80 con 0x7fa8c809a5b0 2024-01-15T18:05:51.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:51.338+0000 7fa8cd600640 1 -- 172.21.15.27:0/4034853710 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa8c806c7a0 con 0x7fa8c809a5b0 2024-01-15T18:05:52.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.066+0000 7fa8bea00640 1 -- 172.21.15.27:0/4034853710 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v25) v1 ==== 181+0+0 (secure 0 0 0) 0x7fa8c806c7a0 con 0x7fa8c809a5b0 2024-01-15T18:05:52.074 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2024-01-15T18:05:52.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.070+0000 7fa8cd600640 1 -- 172.21.15.27:0/4034853710 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa8a00668f0 msgr2=0x7fa8a0068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:52.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.070+0000 7fa8cd600640 1 --2- 172.21.15.27:0/4034853710 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa8a00668f0 0x7fa8a0068db0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8007a60 tx=0x7fa8b8008040 comp rx=0 tx=0).stop 2024-01-15T18:05:52.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.070+0000 7fa8cd600640 1 -- 172.21.15.27:0/4034853710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c809a5b0 msgr2=0x7fa8c8158350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:52.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.070+0000 7fa8cd600640 1 --2- 172.21.15.27:0/4034853710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c809a5b0 0x7fa8c8158350 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa8b000c500 tx=0x7fa8b000c9c0 comp rx=0 tx=0).stop 2024-01-15T18:05:52.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.070+0000 7fa8cd600640 1 -- 172.21.15.27:0/4034853710 shutdown_connections 2024-01-15T18:05:52.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.070+0000 7fa8cd600640 1 --2- 172.21.15.27:0/4034853710 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa8a00668f0 0x7fa8a0068db0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:52.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.070+0000 7fa8cd600640 1 --2- 172.21.15.27:0/4034853710 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8c815b350 0x7fa8c8159630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:52.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.070+0000 7fa8cd600640 1 --2- 172.21.15.27:0/4034853710 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8c809a5b0 0x7fa8c8158350 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:52.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.070+0000 7fa8cd600640 1 --2- 172.21.15.27:0/4034853710 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8c8099a10 0x7fa8c8165590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:52.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.070+0000 7fa8cd600640 1 -- 172.21.15.27:0/4034853710 >> 172.21.15.27:0/4034853710 conn(0x7fa8c80b2a80 msgr2=0x7fa8c8158c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:52.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.070+0000 7fa8cd600640 1 -- 172.21.15.27:0/4034853710 shutdown_connections 2024-01-15T18:05:52.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.070+0000 7fa8cd600640 1 -- 172.21.15.27:0/4034853710 wait complete. 2024-01-15T18:05:52.113 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-15T18:05:52.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.262+0000 7f811e800640 1 Processor -- start 2024-01-15T18:05:52.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.262+0000 7f811e800640 1 -- start start 2024-01-15T18:05:52.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.262+0000 7f811e800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f811815ad20 0x7f811815b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:52.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.262+0000 7f811e800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f81180715f0 0x7f811815ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:52.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.262+0000 7f811e800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f811815a610 con 0x7f811815d900 2024-01-15T18:05:52.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.262+0000 7f811e800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f811815e3c0 con 0x7f81180715f0 2024-01-15T18:05:52.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.262+0000 7f811e800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f811815e6a0 con 0x7f811815ad20 2024-01-15T18:05:52.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.262+0000 7f8117e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f81180715f0 0x7f811815ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:52.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.262+0000 7f8116a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f811815d900 0x7f8118070ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41940/0 (socket says 172.21.15.27:41940) 2024-01-15T18:05:52.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.262+0000 7f8116a00640 1 -- 172.21.15.27:0/3689830488 learned_addr learned my addr 172.21.15.27:0/3689830488 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:52.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.262+0000 7f8117400640 1 --2- 172.21.15.27:0/3689830488 >> v2:172.21.15.93:3300/0 conn(0x7f811815ad20 0x7f811815b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:52.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.262+0000 7f8117e00640 1 -- 172.21.15.27:0/3689830488 >> v2:172.21.15.93:3300/0 conn(0x7f811815ad20 msgr2=0x7f811815b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:52.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.262+0000 7f8117e00640 1 --2- 172.21.15.27:0/3689830488 >> v2:172.21.15.93:3300/0 conn(0x7f811815ad20 0x7f811815b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:52.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.262+0000 7f8117e00640 1 -- 172.21.15.27:0/3689830488 >> v1:172.21.15.27:6789/0 conn(0x7f811815d900 legacy=0x7f8118070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:52.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.262+0000 7f8117e00640 1 -- 172.21.15.27:0/3689830488 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811815b6e0 con 0x7f81180715f0 2024-01-15T18:05:52.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.262+0000 7f8116000640 1 -- 172.21.15.27:0/3689830488 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412837615 0 0) 0x7f811815a610 con 0x7f811815d900 2024-01-15T18:05:52.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.266+0000 7f8117e00640 1 --2- 172.21.15.27:0/3689830488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81180715f0 0x7f811815ddb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8110009d90 tx=0x7f811001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5201d85aa6403460 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:52.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.266+0000 7f8116000640 1 -- 172.21.15.27:0/3689830488 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81100a59c0 con 0x7f81180715f0 2024-01-15T18:05:52.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.266+0000 7f8116000640 1 -- 172.21.15.27:0/3689830488 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81100a5b20 con 0x7f81180715f0 2024-01-15T18:05:52.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.266+0000 7f8116000640 1 -- 172.21.15.27:0/3689830488 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81100ad440 con 0x7f81180715f0 2024-01-15T18:05:52.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.266+0000 7f811e800640 1 -- 172.21.15.27:0/3689830488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81180715f0 msgr2=0x7f811815ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:52.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.266+0000 7f811e800640 1 --2- 172.21.15.27:0/3689830488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81180715f0 0x7f811815ddb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8110009d90 tx=0x7f811001c8e0 comp rx=0 tx=0).stop 2024-01-15T18:05:52.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.266+0000 7f811e800640 1 -- 172.21.15.27:0/3689830488 shutdown_connections 2024-01-15T18:05:52.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.266+0000 7f811e800640 1 --2- 172.21.15.27:0/3689830488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81180715f0 0x7f811815ddb0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:52.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.266+0000 7f811e800640 1 --2- 172.21.15.27:0/3689830488 >> v2:172.21.15.93:3300/0 conn(0x7f811815ad20 0x7f811815b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:52.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.266+0000 7f811e800640 1 -- 172.21.15.27:0/3689830488 >> 172.21.15.27:0/3689830488 conn(0x7f81180cb700 msgr2=0x7f81180cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:52.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.266+0000 7f811e800640 1 -- 172.21.15.27:0/3689830488 shutdown_connections 2024-01-15T18:05:52.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.266+0000 7f811e800640 1 -- 172.21.15.27:0/3689830488 wait complete. 2024-01-15T18:05:52.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.270+0000 7f811e800640 1 Processor -- start 2024-01-15T18:05:52.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.270+0000 7f811e800640 1 -- start start 2024-01-15T18:05:52.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.270+0000 7f811e800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f81180715f0 0x7f811814c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:52.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.270+0000 7f811e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f811815ad20 0x7f811814fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:52.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.270+0000 7f811e800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f811815d900 0x7f8118150170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:52.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.270+0000 7f811e800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81180b42d0 con 0x7f811815ad20 2024-01-15T18:05:52.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.270+0000 7f811e800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81180b36c0 con 0x7f811815d900 2024-01-15T18:05:52.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.270+0000 7f811e800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81180b39a0 con 0x7f81180715f0 2024-01-15T18:05:52.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.270+0000 7f8116a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f811815ad20 0x7f811814fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:52.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.270+0000 7f8116a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f811815ad20 0x7f811814fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39088/0 (socket says 172.21.15.27:39088) 2024-01-15T18:05:52.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.270+0000 7f8116a00640 1 -- 172.21.15.27:0/3619485139 learned_addr learned my addr 172.21.15.27:0/3619485139 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:52.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.270+0000 7f8117400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f81180715f0 0x7f811814c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:52.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.270+0000 7f8117e00640 1 --2- 172.21.15.27:0/3619485139 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f811815d900 0x7f8118150170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:52.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.270+0000 7f8116a00640 1 -- 172.21.15.27:0/3619485139 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f81180715f0 msgr2=0x7f811814c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:52.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.270+0000 7f8116a00640 1 --2- 172.21.15.27:0/3619485139 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f81180715f0 0x7f811814c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:52.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.270+0000 7f8116a00640 1 -- 172.21.15.27:0/3619485139 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f811815d900 msgr2=0x7f8118150170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:52.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.270+0000 7f8116a00640 1 --2- 172.21.15.27:0/3619485139 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f811815d900 0x7f8118150170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:52.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.270+0000 7f8116a00640 1 -- 172.21.15.27:0/3619485139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8110009a70 con 0x7f811815ad20 2024-01-15T18:05:52.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.270+0000 7f8116a00640 1 --2- 172.21.15.27:0/3619485139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f811815ad20 0x7f811814fc30 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f810000c480 tx=0x7f810000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:52.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.270+0000 7f810f400640 1 -- 172.21.15.27:0/3619485139 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81000074e0 con 0x7f811815ad20 2024-01-15T18:05:52.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.270+0000 7f810f400640 1 -- 172.21.15.27:0/3619485139 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8100007660 con 0x7f811815ad20 2024-01-15T18:05:52.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.270+0000 7f811e800640 1 -- 172.21.15.27:0/3619485139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81180b3c80 con 0x7f811815ad20 2024-01-15T18:05:52.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.270+0000 7f810f400640 1 -- 172.21.15.27:0/3619485139 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8100004a30 con 0x7f811815ad20 2024-01-15T18:05:52.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.270+0000 7f811e800640 1 -- 172.21.15.27:0/3619485139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f811814cf00 con 0x7f811815ad20 2024-01-15T18:05:52.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.274+0000 7f811e800640 1 -- 172.21.15.27:0/3619485139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f811804e850 con 0x7f811815ad20 2024-01-15T18:05:52.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.274+0000 7f810f400640 1 -- 172.21.15.27:0/3619485139 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f8100004b90 con 0x7f811815ad20 2024-01-15T18:05:52.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.274+0000 7f810f400640 1 --2- 172.21.15.27:0/3619485139 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f80f0066920 0x7f80f0068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:52.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.274+0000 7f810f400640 1 -- 172.21.15.27:0/3619485139 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f8100083750 con 0x7f811815ad20 2024-01-15T18:05:52.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.274+0000 7f8117400640 1 --2- 172.21.15.27:0/3619485139 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f80f0066920 0x7f80f0068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:52.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.274+0000 7f8117400640 1 --2- 172.21.15.27:0/3619485139 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f80f0066920 0x7f80f0068de0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f811814d660 tx=0x7f8108009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:52.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.278+0000 7f810f400640 1 -- 172.21.15.27:0/3619485139 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8100050b80 con 0x7f811815ad20 2024-01-15T18:05:52.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:52.434+0000 7f811e800640 1 -- 172.21.15.27:0/3619485139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f81180cf160 con 0x7f811815ad20 2024-01-15T18:05:53.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.078+0000 7f810f400640 1 -- 172.21.15.27:0/3619485139 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v26) v1 ==== 169+0+0 (secure 0 0 0) 0x7f81000180d0 con 0x7f811815ad20 2024-01-15T18:05:53.083 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2024-01-15T18:05:53.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.078+0000 7f811e800640 1 -- 172.21.15.27:0/3619485139 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f80f0066920 msgr2=0x7f80f0068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:53.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.078+0000 7f811e800640 1 --2- 172.21.15.27:0/3619485139 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f80f0066920 0x7f80f0068de0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f811814d660 tx=0x7f8108009190 comp rx=0 tx=0).stop 2024-01-15T18:05:53.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.078+0000 7f811e800640 1 -- 172.21.15.27:0/3619485139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f811815ad20 msgr2=0x7f811814fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:53.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.078+0000 7f811e800640 1 --2- 172.21.15.27:0/3619485139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f811815ad20 0x7f811814fc30 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f810000c480 tx=0x7f810000c940 comp rx=0 tx=0).stop 2024-01-15T18:05:53.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.078+0000 7f811e800640 1 -- 172.21.15.27:0/3619485139 shutdown_connections 2024-01-15T18:05:53.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.078+0000 7f811e800640 1 --2- 172.21.15.27:0/3619485139 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f80f0066920 0x7f80f0068de0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:53.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.078+0000 7f811e800640 1 --2- 172.21.15.27:0/3619485139 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f811815d900 0x7f8118150170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:53.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.078+0000 7f811e800640 1 --2- 172.21.15.27:0/3619485139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f811815ad20 0x7f811814fc30 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:53.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.078+0000 7f811e800640 1 --2- 172.21.15.27:0/3619485139 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f81180715f0 0x7f811814c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:53.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.078+0000 7f811e800640 1 -- 172.21.15.27:0/3619485139 >> 172.21.15.27:0/3619485139 conn(0x7f81180cb700 msgr2=0x7f81181642d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:53.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.078+0000 7f811e800640 1 -- 172.21.15.27:0/3619485139 shutdown_connections 2024-01-15T18:05:53.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.078+0000 7f811e800640 1 -- 172.21.15.27:0/3619485139 wait complete. 2024-01-15T18:05:53.115 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-15T18:05:53.115 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-15T18:05:53.115 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-15T18:05:53.115 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:05:53.116 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-15T18:05:53.116 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-15T18:05:53.116 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-15T18:05:53.116 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:05:53.116 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-15T18:05:53.117 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-15T18:05:53.117 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-15T18:05:53.117 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:05:53.117 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-15T18:05:53.117 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-15T18:05:53.118 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-15T18:05:53.118 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:05:53.132 INFO:tasks.ceph.mds.a:Stopped 2024-01-15T18:05:53.132 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-15T18:05:53.133 INFO:tasks.ceph.mds.c:Stopped 2024-01-15T18:05:53.133 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-15T18:05:53.136 INFO:tasks.ceph.mds.b:Stopped 2024-01-15T18:05:53.136 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-15T18:05:53.138 INFO:tasks.ceph.mds.d:Stopped 2024-01-15T18:05:53.138 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-15T18:05:53.177 INFO:tasks.ceph.mds.a:Started 2024-01-15T18:05:53.178 DEBUG:teuthology.parallel:result is None 2024-01-15T18:05:53.179 INFO:tasks.ceph.mds.c:Started 2024-01-15T18:05:53.179 DEBUG:teuthology.parallel:result is None 2024-01-15T18:05:53.184 INFO:tasks.ceph.mds.b:Started 2024-01-15T18:05:53.185 DEBUG:teuthology.parallel:result is None 2024-01-15T18:05:53.187 INFO:tasks.ceph.mds.d:Started 2024-01-15T18:05:53.187 DEBUG:teuthology.parallel:result is None 2024-01-15T18:05:53.187 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-15T18:05:53.265 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-15T18:05:53.266 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-15T18:05:53.272 INFO:tasks.ceph.mds.d.smithi093.stdout:starting mds.d at 2024-01-15T18:05:53.273 INFO:tasks.ceph.mds.b.smithi093.stdout:starting mds.b at 2024-01-15T18:05:53.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.326+0000 7fc09cc00640 1 Processor -- start 2024-01-15T18:05:53.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.326+0000 7fc09cc00640 1 -- start start 2024-01-15T18:05:53.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.330+0000 7fc09cc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc09809a120 0x7fc0980cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:53.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.330+0000 7fc09cc00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09809a700 con 0x7fc0980b35a0 2024-01-15T18:05:53.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.330+0000 7fc09cc00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0980cd590 con 0x7fc09815e7d0 2024-01-15T18:05:53.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.330+0000 7fc09cc00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0980cd870 con 0x7fc09809a120 2024-01-15T18:05:53.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.330+0000 7fc095400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc09809a120 0x7fc0980cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:53.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.330+0000 7fc096800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc0980b35a0 0x7fc0980cc530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41976/0 (socket says 172.21.15.27:41976) 2024-01-15T18:05:53.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.330+0000 7fc096800640 1 -- 172.21.15.27:0/2466322582 learned_addr learned my addr 172.21.15.27:0/2466322582 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:53.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.330+0000 7fc095400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc09809a120 0x7fc0980cbff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39110/0 (socket says 172.21.15.27:39110) 2024-01-15T18:05:53.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.330+0000 7fc095400640 1 -- 172.21.15.27:0/2466322582 >> v1:172.21.15.93:6789/0 conn(0x7fc09815e7d0 legacy=0x7fc098099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:53.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.330+0000 7fc095400640 1 -- 172.21.15.27:0/2466322582 >> v1:172.21.15.27:6789/0 conn(0x7fc0980b35a0 legacy=0x7fc0980cc530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:53.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.330+0000 7fc095400640 1 -- 172.21.15.27:0/2466322582 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0980cfa00 con 0x7fc09809a120 2024-01-15T18:05:53.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.330+0000 7fc094a00640 1 -- 172.21.15.27:0/2466322582 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1826020713 0 0) 0x7fc09809a700 con 0x7fc0980b35a0 2024-01-15T18:05:53.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.334+0000 7fc095400640 1 --2- 172.21.15.27:0/2466322582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc09809a120 0x7fc0980cbff0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc0800097c0 tx=0x7fc08001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae0a88d9b316e36 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:53.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.334+0000 7fc094a00640 1 -- 172.21.15.27:0/2466322582 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0800ab670 con 0x7fc09809a120 2024-01-15T18:05:53.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.334+0000 7fc094a00640 1 -- 172.21.15.27:0/2466322582 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0800ab7d0 con 0x7fc09809a120 2024-01-15T18:05:53.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.334+0000 7fc094a00640 1 -- 172.21.15.27:0/2466322582 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0800abb60 con 0x7fc09809a120 2024-01-15T18:05:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.334+0000 7fc09cc00640 1 -- 172.21.15.27:0/2466322582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc09809a120 msgr2=0x7fc0980cbff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.334+0000 7fc09cc00640 1 --2- 172.21.15.27:0/2466322582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc09809a120 0x7fc0980cbff0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc0800097c0 tx=0x7fc08001c450 comp rx=0 tx=0).stop 2024-01-15T18:05:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.334+0000 7fc09cc00640 1 -- 172.21.15.27:0/2466322582 shutdown_connections 2024-01-15T18:05:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.334+0000 7fc09cc00640 1 --2- 172.21.15.27:0/2466322582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc09809a120 0x7fc0980cbff0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.334+0000 7fc09cc00640 1 -- 172.21.15.27:0/2466322582 >> 172.21.15.27:0/2466322582 conn(0x7fc0980b2a80 msgr2=0x7fc0980b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:53.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.334+0000 7fc09cc00640 1 -- 172.21.15.27:0/2466322582 shutdown_connections 2024-01-15T18:05:53.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.334+0000 7fc09cc00640 1 -- 172.21.15.27:0/2466322582 wait complete. 2024-01-15T18:05:53.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.334+0000 7fc09cc00640 1 Processor -- start 2024-01-15T18:05:53.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.334+0000 7fc09cc00640 1 -- start start 2024-01-15T18:05:53.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.338+0000 7fc09cc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc09809a120 0x7fc09815cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:53.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.338+0000 7fc09cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0980b35a0 0x7fc0981548d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:53.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.338+0000 7fc09cc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc09815e7d0 0x7fc09815c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:53.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.338+0000 7fc09cc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0981657c0 con 0x7fc0980b35a0 2024-01-15T18:05:53.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.338+0000 7fc09cc00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09815d2e0 con 0x7fc09809a120 2024-01-15T18:05:53.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.338+0000 7fc09cc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc098164eb0 con 0x7fc09815e7d0 2024-01-15T18:05:53.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.338+0000 7fc095400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0980b35a0 0x7fc0981548d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:53.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.338+0000 7fc095400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0980b35a0 0x7fc0981548d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39126/0 (socket says 172.21.15.27:39126) 2024-01-15T18:05:53.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.338+0000 7fc095400640 1 -- 172.21.15.27:0/3549668138 learned_addr learned my addr 172.21.15.27:0/3549668138 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:53.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.338+0000 7fc096800640 1 --2- 172.21.15.27:0/3549668138 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc09815e7d0 0x7fc09815c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:53.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.338+0000 7fc095e00640 1 --2- 172.21.15.27:0/3549668138 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc09809a120 0x7fc09815cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:53.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.338+0000 7fc096800640 1 -- 172.21.15.27:0/3549668138 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc09809a120 msgr2=0x7fc09815cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:53.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.338+0000 7fc096800640 1 --2- 172.21.15.27:0/3549668138 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc09809a120 0x7fc09815cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:53.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.338+0000 7fc096800640 1 -- 172.21.15.27:0/3549668138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0980b35a0 msgr2=0x7fc0981548d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:53.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.338+0000 7fc096800640 1 --2- 172.21.15.27:0/3549668138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0980b35a0 0x7fc0981548d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:53.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.338+0000 7fc096800640 1 -- 172.21.15.27:0/3549668138 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc080009420 con 0x7fc09815e7d0 2024-01-15T18:05:53.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.338+0000 7fc095400640 1 --2- 172.21.15.27:0/3549668138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0980b35a0 0x7fc0981548d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:53.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.338+0000 7fc096800640 1 --2- 172.21.15.27:0/3549668138 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc09815e7d0 0x7fc09815c190 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc090002240 tx=0x7fc09000cbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:53.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.338+0000 7fc08e000640 1 -- 172.21.15.27:0/3549668138 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09000ce30 con 0x7fc09815e7d0 2024-01-15T18:05:53.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.338+0000 7fc08e000640 1 -- 172.21.15.27:0/3549668138 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09000a640 con 0x7fc09815e7d0 2024-01-15T18:05:53.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.338+0000 7fc08e000640 1 -- 172.21.15.27:0/3549668138 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09000a9f0 con 0x7fc09815e7d0 2024-01-15T18:05:53.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.338+0000 7fc09cc00640 1 -- 172.21.15.27:0/3549668138 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc09806ce50 con 0x7fc09815e7d0 2024-01-15T18:05:53.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.338+0000 7fc09cc00640 1 -- 172.21.15.27:0/3549668138 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0981653e0 con 0x7fc09815e7d0 2024-01-15T18:05:53.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.338+0000 7fc09cc00640 1 -- 172.21.15.27:0/3549668138 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc098165900 con 0x7fc09815e7d0 2024-01-15T18:05:53.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.342+0000 7fc08e000640 1 -- 172.21.15.27:0/3549668138 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fc09000ab50 con 0x7fc09815e7d0 2024-01-15T18:05:53.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.342+0000 7fc08e000640 1 --2- 172.21.15.27:0/3549668138 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc0700668f0 0x7fc070068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:53.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.342+0000 7fc08e000640 1 -- 172.21.15.27:0/3549668138 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc090083640 con 0x7fc09815e7d0 2024-01-15T18:05:53.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.342+0000 7fc095e00640 1 --2- 172.21.15.27:0/3549668138 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc0700668f0 0x7fc070068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:53.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.342+0000 7fc095e00640 1 --2- 172.21.15.27:0/3549668138 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc0700668f0 0x7fc070068db0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc088007fd0 tx=0x7fc088009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:53.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.346+0000 7fc08e000640 1 -- 172.21.15.27:0/3549668138 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc098165900 con 0x7fc09815e7d0 2024-01-15T18:05:53.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:53.514+0000 7fc09cc00640 1 -- 172.21.15.27:0/3549668138 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc098165900 con 0x7fc09815e7d0 2024-01-15T18:05:54.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.098+0000 7fc08e000640 1 -- 172.21.15.27:0/3549668138 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v28) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc098165900 con 0x7fc09815e7d0 2024-01-15T18:05:54.107 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-01-15T18:05:54.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.106+0000 7fc09cc00640 1 -- 172.21.15.27:0/3549668138 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc0700668f0 msgr2=0x7fc070068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:54.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.106+0000 7fc09cc00640 1 --2- 172.21.15.27:0/3549668138 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc0700668f0 0x7fc070068db0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc088007fd0 tx=0x7fc088009040 comp rx=0 tx=0).stop 2024-01-15T18:05:54.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.106+0000 7fc09cc00640 1 -- 172.21.15.27:0/3549668138 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc09815e7d0 msgr2=0x7fc09815c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:54.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.106+0000 7fc09cc00640 1 --2- 172.21.15.27:0/3549668138 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc09815e7d0 0x7fc09815c190 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc090002240 tx=0x7fc09000cbe0 comp rx=0 tx=0).stop 2024-01-15T18:05:54.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.106+0000 7fc09cc00640 1 -- 172.21.15.27:0/3549668138 shutdown_connections 2024-01-15T18:05:54.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.106+0000 7fc09cc00640 1 --2- 172.21.15.27:0/3549668138 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc0700668f0 0x7fc070068db0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:54.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.106+0000 7fc09cc00640 1 --2- 172.21.15.27:0/3549668138 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc09815e7d0 0x7fc09815c190 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:54.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.106+0000 7fc09cc00640 1 --2- 172.21.15.27:0/3549668138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0980b35a0 0x7fc0981548d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:54.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.106+0000 7fc09cc00640 1 --2- 172.21.15.27:0/3549668138 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc09809a120 0x7fc09815cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:54.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.106+0000 7fc09cc00640 1 -- 172.21.15.27:0/3549668138 >> 172.21.15.27:0/3549668138 conn(0x7fc0980b2a80 msgr2=0x7fc0980cbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:54.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.106+0000 7fc09cc00640 1 -- 172.21.15.27:0/3549668138 shutdown_connections 2024-01-15T18:05:54.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.106+0000 7fc09cc00640 1 -- 172.21.15.27:0/3549668138 wait complete. 2024-01-15T18:05:54.168 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-15T18:05:54.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.302+0000 7f0222e00640 1 Processor -- start 2024-01-15T18:05:54.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.302+0000 7f0222e00640 1 -- start start 2024-01-15T18:05:54.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.306+0000 7f0222e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f021c160a20 0x7f021c0b1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:54.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.306+0000 7f0222e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f021c0ce360 0x7f021c0cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:54.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.306+0000 7f0222e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f021c0d1a70 con 0x7f021c0cd7d0 2024-01-15T18:05:54.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.306+0000 7f0222e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f021c160e00 con 0x7f021c0ce360 2024-01-15T18:05:54.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.306+0000 7f0222e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f021c0d1cc0 con 0x7f021c160a20 2024-01-15T18:05:54.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.306+0000 7f0220e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f021c0ce360 0x7f021c0cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:54.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.306+0000 7f0220e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f021c0ce360 0x7f021c0cb880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52658/0 (socket says 172.21.15.27:52658) 2024-01-15T18:05:54.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.306+0000 7f0220e00640 1 -- 172.21.15.27:0/839644204 learned_addr learned my addr 172.21.15.27:0/839644204 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:54.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.306+0000 7f021be00640 1 --2- 172.21.15.27:0/839644204 >> v2:172.21.15.93:3300/0 conn(0x7f021c160a20 0x7f021c0b1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:54.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.306+0000 7f0220e00640 1 -- 172.21.15.27:0/839644204 >> v2:172.21.15.93:3300/0 conn(0x7f021c160a20 msgr2=0x7f021c0b1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:54.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.306+0000 7f0220e00640 1 --2- 172.21.15.27:0/839644204 >> v2:172.21.15.93:3300/0 conn(0x7f021c160a20 0x7f021c0b1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:54.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.306+0000 7f0220e00640 1 -- 172.21.15.27:0/839644204 >> v1:172.21.15.93:6789/0 conn(0x7f021c0cd7d0 legacy=0x7f021c0cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:54.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.306+0000 7f0220e00640 1 -- 172.21.15.27:0/839644204 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f021c06c010 con 0x7f021c0ce360 2024-01-15T18:05:54.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.306+0000 7f021be00640 1 --2- 172.21.15.27:0/839644204 >> v2:172.21.15.93:3300/0 conn(0x7f021c160a20 0x7f021c0b1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:54.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.306+0000 7f0220e00640 1 --2- 172.21.15.27:0/839644204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f021c0ce360 0x7f021c0cb880 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f021c073de0 tx=0x7f021401c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4cc4f7e4fbf7cb6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:54.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.306+0000 7f021aa00640 1 -- 172.21.15.27:0/839644204 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02140a5780 con 0x7f021c0ce360 2024-01-15T18:05:54.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.310+0000 7f021aa00640 1 -- 172.21.15.27:0/839644204 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02140a58e0 con 0x7f021c0ce360 2024-01-15T18:05:54.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.310+0000 7f021aa00640 1 -- 172.21.15.27:0/839644204 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02140a5c70 con 0x7f021c0ce360 2024-01-15T18:05:54.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.310+0000 7f0222e00640 1 -- 172.21.15.27:0/839644204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f021c0ce360 msgr2=0x7f021c0cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:54.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.310+0000 7f0222e00640 1 --2- 172.21.15.27:0/839644204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f021c0ce360 0x7f021c0cb880 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f021c073de0 tx=0x7f021401c9d0 comp rx=0 tx=0).stop 2024-01-15T18:05:54.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.310+0000 7f0222e00640 1 -- 172.21.15.27:0/839644204 shutdown_connections 2024-01-15T18:05:54.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.310+0000 7f0222e00640 1 --2- 172.21.15.27:0/839644204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f021c0ce360 0x7f021c0cb880 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:54.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.310+0000 7f0222e00640 1 --2- 172.21.15.27:0/839644204 >> v2:172.21.15.93:3300/0 conn(0x7f021c160a20 0x7f021c0b1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:54.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.310+0000 7f0222e00640 1 -- 172.21.15.27:0/839644204 >> 172.21.15.27:0/839644204 conn(0x7f021c0d0f50 msgr2=0x7f021c0d1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:54.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.310+0000 7f0222e00640 1 -- 172.21.15.27:0/839644204 shutdown_connections 2024-01-15T18:05:54.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.310+0000 7f0222e00640 1 -- 172.21.15.27:0/839644204 wait complete. 2024-01-15T18:05:54.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.310+0000 7f0222e00640 1 Processor -- start 2024-01-15T18:05:54.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.310+0000 7f0222e00640 1 -- start start 2024-01-15T18:05:54.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.310+0000 7f0222e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f021c0cd7d0 0x7f021c0ca750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:54.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.310+0000 7f0222e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f021c0ce360 0x7f021c156ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:54.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.310+0000 7f0222e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f021c160a20 0x7f021c159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:54.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.310+0000 7f0222e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c15ed60 con 0x7f021c160a20 2024-01-15T18:05:54.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.310+0000 7f0222e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c158740 con 0x7f021c0ce360 2024-01-15T18:05:54.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.310+0000 7f0222e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f021c158a20 con 0x7f021c0cd7d0 2024-01-15T18:05:54.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.314+0000 7f021be00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f021c0cd7d0 0x7f021c0ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:54.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.314+0000 7f021be00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f021c0cd7d0 0x7f021c0ca750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:36098/0 (socket says 172.21.15.27:36098) 2024-01-15T18:05:54.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.314+0000 7f021be00640 1 -- 172.21.15.27:0/1955275597 learned_addr learned my addr 172.21.15.27:0/1955275597 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:54.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.314+0000 7f021b400640 1 --2- 172.21.15.27:0/1955275597 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f021c0ce360 0x7f021c156ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:54.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.314+0000 7f0220e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f021c160a20 0x7f021c159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:54.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.314+0000 7f0220e00640 1 -- 172.21.15.27:0/1955275597 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f021c0cd7d0 msgr2=0x7f021c0ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:54.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.314+0000 7f0220e00640 1 --2- 172.21.15.27:0/1955275597 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f021c0cd7d0 0x7f021c0ca750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:54.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.314+0000 7f0220e00640 1 -- 172.21.15.27:0/1955275597 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f021c0ce360 msgr2=0x7f021c156ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:54.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.314+0000 7f0220e00640 1 --2- 172.21.15.27:0/1955275597 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f021c0ce360 0x7f021c156ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:54.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.314+0000 7f0220e00640 1 -- 172.21.15.27:0/1955275597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0214009b30 con 0x7f021c160a20 2024-01-15T18:05:54.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.314+0000 7f0220e00640 1 --2- 172.21.15.27:0/1955275597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f021c160a20 0x7f021c159500 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f02140ab000 tx=0x7f0214019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:54.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.314+0000 7f0213e00640 1 -- 172.21.15.27:0/1955275597 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02140bd070 con 0x7f021c160a20 2024-01-15T18:05:54.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.314+0000 7f0213e00640 1 -- 172.21.15.27:0/1955275597 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02140b8470 con 0x7f021c160a20 2024-01-15T18:05:54.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.314+0000 7f0213e00640 1 -- 172.21.15.27:0/1955275597 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02140b8800 con 0x7f021c160a20 2024-01-15T18:05:54.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.314+0000 7f0222e00640 1 -- 172.21.15.27:0/1955275597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f021c158d00 con 0x7f021c160a20 2024-01-15T18:05:54.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.314+0000 7f0222e00640 1 -- 172.21.15.27:0/1955275597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f021c0cae30 con 0x7f021c160a20 2024-01-15T18:05:54.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.314+0000 7f0222e00640 1 -- 172.21.15.27:0/1955275597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f021c04e850 con 0x7f021c160a20 2024-01-15T18:05:54.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.314+0000 7f0213e00640 1 -- 172.21.15.27:0/1955275597 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f02140b8960 con 0x7f021c160a20 2024-01-15T18:05:54.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.318+0000 7f0213e00640 1 --2- 172.21.15.27:0/1955275597 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f01f40668f0 0x7f01f4068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:54.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.318+0000 7f0213e00640 1 -- 172.21.15.27:0/1955275597 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f021411eb90 con 0x7f021c160a20 2024-01-15T18:05:54.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.318+0000 7f021be00640 1 --2- 172.21.15.27:0/1955275597 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f01f40668f0 0x7f01f4068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:54.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.318+0000 7f021be00640 1 --2- 172.21.15.27:0/1955275597 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f01f40668f0 0x7f01f4068db0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f020c007a70 tx=0x7f020c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:54.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.322+0000 7f0213e00640 1 -- 172.21.15.27:0/1955275597 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f02140ec390 con 0x7f021c160a20 2024-01-15T18:05:54.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.530+0000 7f0222e00640 1 -- 172.21.15.27:0/1955275597 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f021c158b60 con 0x7f021c160a20 2024-01-15T18:05:54.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.530+0000 7f0213e00640 1 -- 172.21.15.27:0/1955275597 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v2) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f021c158b60 con 0x7f021c160a20 2024-01-15T18:05:54.536 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:54.536 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:05:54.537 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-01-15T18:05:54.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:54.537 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-01-15T18:05:54.537 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlbMZIGBAAhSU4y2DGJd3MH0v6v3LsDg==", 2024-01-15T18:05:54.537 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:05:54.537 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:05:54.537 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:05:54.537 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:05:54.537 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:05:54.537 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:54.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:54.538 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-01-15T18:05:54.538 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlC7B9JBAAsXmBjV4KwPZejOmowXO2Zg==", 2024-01-15T18:05:54.538 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:05:54.538 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:05:54.538 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:05:54.538 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:05:54.538 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:05:54.539 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:54.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:54.539 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-01-15T18:05:54.539 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6Vla6a3HxAA/AHc6FSTccj0JGwiR4Vl7g==", 2024-01-15T18:05:54.539 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:05:54.539 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:05:54.539 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:05:54.539 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:05:54.540 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:05:54.540 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:54.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:54.540 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-01-15T18:05:54.540 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6Vl9nqzKxAAx2aDA8UwidAUrU9DBxGldA==", 2024-01-15T18:05:54.540 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:05:54.540 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:05:54.540 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:05:54.541 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:05:54.541 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:05:54.541 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:54.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:54.541 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-01-15T18:05:54.541 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCSc6Vly4TACRAAYBmMIcsZsRp4CZ5a3xCWgA==", 2024-01-15T18:05:54.541 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:05:54.541 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:05:54.541 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:05:54.541 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:05:54.542 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:54.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:54.542 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-01-15T18:05:54.542 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCVc6VlW/HEERAAOBgPz+eGpHwNXDe8fkYo7w==", 2024-01-15T18:05:54.542 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:05:54.542 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:05:54.542 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:05:54.542 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:05:54.542 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:54.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:54.543 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-01-15T18:05:54.543 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCYc6Vl+Ec4GBAAWRe/6u7u4RXApd57H8ihng==", 2024-01-15T18:05:54.543 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:05:54.543 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:05:54.543 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:05:54.543 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:05:54.543 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:54.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:54.544 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-01-15T18:05:54.544 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCbc6VlNdMPHxAAHzJvzLltsTl1wrEnI/iyrQ==", 2024-01-15T18:05:54.544 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:05:54.544 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:05:54.544 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:05:54.544 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:05:54.544 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:54.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:54.545 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-01-15T18:05:54.545 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQChc6Vlg8eVKBAAwZWAncCE2Lra+CxmSIxZkA==", 2024-01-15T18:05:54.545 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:05:54.545 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:05:54.545 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:05:54.545 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:05:54.545 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:54.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:54.545 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-01-15T18:05:54.546 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCkc6VlIXDOLxAAFISn6ILqq/aNFK07RxNnaw==", 2024-01-15T18:05:54.546 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:05:54.546 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:05:54.546 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:05:54.546 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:05:54.546 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:54.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:54.546 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-01-15T18:05:54.546 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCnc6Vl9Ru2NRAAl996VBDudw23SGiFPusIcw==", 2024-01-15T18:05:54.546 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:05:54.547 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:05:54.547 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:05:54.547 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:05:54.547 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.547 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:54.547 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:54.547 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-01-15T18:05:54.547 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCrc6VlHoAsARAAIoPMridLsIcpU3I9Hr5XLQ==", 2024-01-15T18:05:54.547 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:05:54.547 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:05:54.548 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:05:54.548 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:05:54.548 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.548 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:54.548 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:54.548 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-01-15T18:05:54.548 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlH+DCLxAA1sKVut+3WjA/B+aYJ6/Jeg==", 2024-01-15T18:05:54.548 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:05:54.548 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:05:54.549 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-15T18:05:54.549 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-15T18:05:54.549 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-15T18:05:54.549 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.549 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:54.549 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:54.549 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-01-15T18:05:54.549 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlaRBaORAAijn3UJLHbWm8ol2NiKNSaw==", 2024-01-15T18:05:54.549 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:05:54.549 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:05:54.549 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-15T18:05:54.550 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-15T18:05:54.550 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-15T18:05:54.550 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.550 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:54.550 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:54.550 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-01-15T18:05:54.550 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlDbLQNRAA82Lwz+pKY+3nHPAr1DichA==", 2024-01-15T18:05:54.550 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:05:54.550 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:05:54.550 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-15T18:05:54.550 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-15T18:05:54.551 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-15T18:05:54.551 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:54.551 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:54.551 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-01-15T18:05:54.551 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCPc6VlOyDaAxAADea0K1OEDTfrzsM7VEkixg==", 2024-01-15T18:05:54.551 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:05:54.551 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:05:54.551 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-15T18:05:54.551 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-15T18:05:54.552 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-15T18:05:54.552 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:54.552 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:54.552 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-01-15T18:05:54.552 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNc6VlK70BNxAAgYXkW/pMlzy8GZoZhX9NMA==", 2024-01-15T18:05:54.552 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:05:54.552 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-15T18:05:54.552 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:05:54.552 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:05:54.553 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:05:54.553 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.553 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:54.553 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:54.553 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.bootstrap-mds", 2024-01-15T18:05:54.553 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCxc6Vl5k+mABAAZUgx6c8ybR74w1AR94GC/w==", 2024-01-15T18:05:54.553 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:05:54.553 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile bootstrap-mds" 2024-01-15T18:05:54.553 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:54.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:54.554 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.bootstrap-mgr", 2024-01-15T18:05:54.554 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCxc6VlXlqmABAA71T0sRhKk98NhLv7NsQK5Q==", 2024-01-15T18:05:54.554 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:05:54.554 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile bootstrap-mgr" 2024-01-15T18:05:54.554 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.554 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:54.554 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:54.554 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.bootstrap-osd", 2024-01-15T18:05:54.554 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCxc6VlB2OmABAAKKiyF5JyzypCQJnFqBSW6w==", 2024-01-15T18:05:54.555 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:05:54.555 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile bootstrap-osd" 2024-01-15T18:05:54.555 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:54.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:54.555 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.bootstrap-rbd", 2024-01-15T18:05:54.555 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCxc6VleWumABAA6NlffjY0uC9mDKpa8STIlA==", 2024-01-15T18:05:54.555 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:05:54.555 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile bootstrap-rbd" 2024-01-15T18:05:54.555 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.555 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:54.555 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:54.556 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-01-15T18:05:54.556 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCxc6VlEHSmABAAUket+1gLsZVn7YNTLC0n9A==", 2024-01-15T18:05:54.556 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:05:54.556 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-01-15T18:05:54.556 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.556 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:54.556 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:54.556 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.bootstrap-rgw", 2024-01-15T18:05:54.556 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCxc6VlAn6mABAADQgcNgF5ODzke21tCyCVdA==", 2024-01-15T18:05:54.556 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:05:54.557 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile bootstrap-rgw" 2024-01-15T18:05:54.557 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:54.557 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:54.557 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-01-15T18:05:54.557 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlQbinExAAWtUdgkUNR3cCKV+ytcbUuQ==", 2024-01-15T18:05:54.557 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:05:54.557 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-15T18:05:54.557 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-15T18:05:54.557 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:05:54.557 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.557 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:05:54.558 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:54.558 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-01-15T18:05:54.558 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlzafrDxAAVVEBuX8BDT3cB9NfD2xTdQ==", 2024-01-15T18:05:54.558 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:05:54.558 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-15T18:05:54.558 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-15T18:05:54.558 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:05:54.558 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.558 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:54.558 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-01-15T18:05:54.558 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:05:54.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.530+0000 7f0222e00640 1 -- 172.21.15.27:0/1955275597 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f01f40668f0 msgr2=0x7f01f4068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:54.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.530+0000 7f0222e00640 1 --2- 172.21.15.27:0/1955275597 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f01f40668f0 0x7f01f4068db0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f020c007a70 tx=0x7f020c008040 comp rx=0 tx=0).stop 2024-01-15T18:05:54.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.530+0000 7f0222e00640 1 -- 172.21.15.27:0/1955275597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f021c160a20 msgr2=0x7f021c159500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:54.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.530+0000 7f0222e00640 1 --2- 172.21.15.27:0/1955275597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f021c160a20 0x7f021c159500 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f02140ab000 tx=0x7f0214019770 comp rx=0 tx=0).stop 2024-01-15T18:05:54.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.530+0000 7f0222e00640 1 -- 172.21.15.27:0/1955275597 shutdown_connections 2024-01-15T18:05:54.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.530+0000 7f0222e00640 1 --2- 172.21.15.27:0/1955275597 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f01f40668f0 0x7f01f4068db0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:54.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.530+0000 7f0222e00640 1 --2- 172.21.15.27:0/1955275597 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f021c160a20 0x7f021c159500 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:54.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.530+0000 7f0222e00640 1 --2- 172.21.15.27:0/1955275597 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f021c0ce360 0x7f021c156ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:54.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.530+0000 7f0222e00640 1 --2- 172.21.15.27:0/1955275597 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f021c0cd7d0 0x7f021c0ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:54.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.530+0000 7f0222e00640 1 -- 172.21.15.27:0/1955275597 >> 172.21.15.27:0/1955275597 conn(0x7f021c0d0f50 msgr2=0x7f021c0cf110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:54.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.530+0000 7f0222e00640 1 -- 172.21.15.27:0/1955275597 shutdown_connections 2024-01-15T18:05:54.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.530+0000 7f0222e00640 1 -- 172.21.15.27:0/1955275597 wait complete. 2024-01-15T18:05:54.562 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-01-15T18:05:54.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.698+0000 7f024ce00640 1 Processor -- start 2024-01-15T18:05:54.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.698+0000 7f024ce00640 1 -- start start 2024-01-15T18:05:54.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.702+0000 7f024ce00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f024815d680 0x7f02480cfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:54.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.702+0000 7f024ce00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f02480cde10 0x7f02480cc150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:54.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.702+0000 7f024ce00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02480b3cd0 con 0x7f024815d2a0 2024-01-15T18:05:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.702+0000 7f024ce00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02480d0260 con 0x7f024815d680 2024-01-15T18:05:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.702+0000 7f024ce00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02480cb670 con 0x7f02480cde10 2024-01-15T18:05:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.702+0000 7f0245e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f024815d2a0 0x7f02480cd700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51798/0 (socket says 172.21.15.27:51798) 2024-01-15T18:05:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.702+0000 7f0245e00640 1 -- 172.21.15.27:0/184762580 learned_addr learned my addr 172.21.15.27:0/184762580 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.702+0000 7f0245400640 1 --2- 172.21.15.27:0/184762580 >> v2:172.21.15.93:3300/0 conn(0x7f024815d680 0x7f02480cfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.702+0000 7f0246800640 1 --2- 172.21.15.27:0/184762580 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f02480cde10 0x7f02480cc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.702+0000 7f0244a00640 1 -- 172.21.15.27:0/184762580 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 850029225 0 0) 0x7f02480b3cd0 con 0x7f024815d2a0 2024-01-15T18:05:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.702+0000 7f0244a00640 1 -- 172.21.15.27:0/184762580 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0228003530 con 0x7f024815d2a0 2024-01-15T18:05:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.702+0000 7f0245400640 1 -- 172.21.15.27:0/184762580 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f02480cde10 msgr2=0x7f02480cc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.702+0000 7f0245400640 1 --2- 172.21.15.27:0/184762580 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f02480cde10 0x7f02480cc150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:54.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.702+0000 7f0245400640 1 -- 172.21.15.27:0/184762580 >> v1:172.21.15.27:6789/0 conn(0x7f024815d2a0 legacy=0x7f02480cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:54.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.702+0000 7f0245400640 1 -- 172.21.15.27:0/184762580 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02480cb950 con 0x7f024815d680 2024-01-15T18:05:54.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.706+0000 7f0245400640 1 --2- 172.21.15.27:0/184762580 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f024815d680 0x7f02480cfbd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f02300097c0 tx=0x7f023001c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f75a386cab1c922 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:54.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.706+0000 7f0244a00640 1 -- 172.21.15.27:0/184762580 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02300ac600 con 0x7f024815d680 2024-01-15T18:05:54.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.706+0000 7f0244a00640 1 -- 172.21.15.27:0/184762580 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02300a8050 con 0x7f024815d680 2024-01-15T18:05:54.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.706+0000 7f0244a00640 1 -- 172.21.15.27:0/184762580 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02300ac980 con 0x7f024815d680 2024-01-15T18:05:54.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.706+0000 7f024ce00640 1 -- 172.21.15.27:0/184762580 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f024815d680 msgr2=0x7f02480cfbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:54.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.706+0000 7f024ce00640 1 --2- 172.21.15.27:0/184762580 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f024815d680 0x7f02480cfbd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f02300097c0 tx=0x7f023001c3c0 comp rx=0 tx=0).stop 2024-01-15T18:05:54.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.706+0000 7f024ce00640 1 -- 172.21.15.27:0/184762580 shutdown_connections 2024-01-15T18:05:54.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.706+0000 7f024ce00640 1 --2- 172.21.15.27:0/184762580 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f02480cde10 0x7f02480cc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:54.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.706+0000 7f024ce00640 1 --2- 172.21.15.27:0/184762580 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f024815d680 0x7f02480cfbd0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:54.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.706+0000 7f024ce00640 1 -- 172.21.15.27:0/184762580 >> 172.21.15.27:0/184762580 conn(0x7f02480d0e80 msgr2=0x7f02480d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:54.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.706+0000 7f024ce00640 1 -- 172.21.15.27:0/184762580 shutdown_connections 2024-01-15T18:05:54.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.706+0000 7f024ce00640 1 -- 172.21.15.27:0/184762580 wait complete. 2024-01-15T18:05:54.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.706+0000 7f024ce00640 1 Processor -- start 2024-01-15T18:05:54.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.706+0000 7f024ce00640 1 -- start start 2024-01-15T18:05:54.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.710+0000 7f024ce00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f02480cde10 0x7f0248155500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:54.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.710+0000 7f024ce00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f024815d2a0 0x7f024815af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:54.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.710+0000 7f024ce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f024815d680 0x7f024814a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:54.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.710+0000 7f024ce00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f024815aae0 con 0x7f024815d680 2024-01-15T18:05:54.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.710+0000 7f024ce00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f024815a220 con 0x7f024815d2a0 2024-01-15T18:05:54.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.710+0000 7f024ce00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f024815b4b0 con 0x7f02480cde10 2024-01-15T18:05:54.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.710+0000 7f0246800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f024815d680 0x7f024814a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:54.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.710+0000 7f0246800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f024815d680 0x7f024814a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52682/0 (socket says 172.21.15.27:52682) 2024-01-15T18:05:54.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.710+0000 7f0246800640 1 -- 172.21.15.27:0/599738416 learned_addr learned my addr 172.21.15.27:0/599738416 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:54.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.710+0000 7f0245e00640 1 --2- 172.21.15.27:0/599738416 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f02480cde10 0x7f0248155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:54.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.710+0000 7f0245400640 1 --2- 172.21.15.27:0/599738416 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f024815d2a0 0x7f024815af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:54.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.710+0000 7f0246800640 1 -- 172.21.15.27:0/599738416 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f02480cde10 msgr2=0x7f0248155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:54.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.710+0000 7f0246800640 1 --2- 172.21.15.27:0/599738416 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f02480cde10 0x7f0248155500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:54.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.710+0000 7f0246800640 1 -- 172.21.15.27:0/599738416 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f024815d2a0 msgr2=0x7f024815af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:54.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.710+0000 7f0246800640 1 --2- 172.21.15.27:0/599738416 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f024815d2a0 0x7f024815af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:54.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.710+0000 7f0246800640 1 -- 172.21.15.27:0/599738416 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0230009420 con 0x7f024815d680 2024-01-15T18:05:54.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.710+0000 7f0246800640 1 --2- 172.21.15.27:0/599738416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f024815d680 0x7f024814a0e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0240009dd0 tx=0x7f024000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:54.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.710+0000 7f023e000640 1 -- 172.21.15.27:0/599738416 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0240014480 con 0x7f024815d680 2024-01-15T18:05:54.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.710+0000 7f023e000640 1 -- 172.21.15.27:0/599738416 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f024000f040 con 0x7f024815d680 2024-01-15T18:05:54.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.710+0000 7f023e000640 1 -- 172.21.15.27:0/599738416 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0240014840 con 0x7f024815d680 2024-01-15T18:05:54.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.710+0000 7f024ce00640 1 -- 172.21.15.27:0/599738416 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f024815a490 con 0x7f024815d680 2024-01-15T18:05:54.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.710+0000 7f024ce00640 1 -- 172.21.15.27:0/599738416 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02480c8350 con 0x7f024815d680 2024-01-15T18:05:54.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.710+0000 7f024ce00640 1 -- 172.21.15.27:0/599738416 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f024815ac90 con 0x7f024815d680 2024-01-15T18:05:54.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.714+0000 7f023e000640 1 -- 172.21.15.27:0/599738416 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f02400149a0 con 0x7f024815d680 2024-01-15T18:05:54.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.714+0000 7f023e000640 1 --2- 172.21.15.27:0/599738416 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0220066920 0x7f0220068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:54.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.714+0000 7f023e000640 1 -- 172.21.15.27:0/599738416 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0240084cd0 con 0x7f024815d680 2024-01-15T18:05:54.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.714+0000 7f0245e00640 1 --2- 172.21.15.27:0/599738416 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0220066920 0x7f0220068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:54.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.714+0000 7f0245e00640 1 --2- 172.21.15.27:0/599738416 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0220066920 0x7f0220068de0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f02380011f0 tx=0x7f0238009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:54.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.718+0000 7f023e000640 1 -- 172.21.15.27:0/599738416 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f02400521c0 con 0x7f024815d680 2024-01-15T18:05:54.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.934+0000 7f024ce00640 1 -- 172.21.15.27:0/599738416 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f024804e850 con 0x7f024815d680 2024-01-15T18:05:54.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.938+0000 7f023e000640 1 -- 172.21.15.27:0/599738416 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v3) v1 ==== 90+0+0 (secure 0 0 0) 0x7f02400557e0 con 0x7f024815d680 2024-01-15T18:05:54.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.942+0000 7f024ce00640 1 -- 172.21.15.27:0/599738416 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0220066920 msgr2=0x7f0220068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:54.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.942+0000 7f024ce00640 1 --2- 172.21.15.27:0/599738416 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0220066920 0x7f0220068de0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f02380011f0 tx=0x7f0238009040 comp rx=0 tx=0).stop 2024-01-15T18:05:54.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.942+0000 7f024ce00640 1 -- 172.21.15.27:0/599738416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f024815d680 msgr2=0x7f024814a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:54.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.942+0000 7f024ce00640 1 --2- 172.21.15.27:0/599738416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f024815d680 0x7f024814a0e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0240009dd0 tx=0x7f024000c490 comp rx=0 tx=0).stop 2024-01-15T18:05:54.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.942+0000 7f024ce00640 1 -- 172.21.15.27:0/599738416 shutdown_connections 2024-01-15T18:05:54.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.942+0000 7f024ce00640 1 --2- 172.21.15.27:0/599738416 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0220066920 0x7f0220068de0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:54.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.942+0000 7f024ce00640 1 --2- 172.21.15.27:0/599738416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f024815d680 0x7f024814a0e0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:54.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.942+0000 7f024ce00640 1 --2- 172.21.15.27:0/599738416 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f024815d2a0 0x7f024815af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:54.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.942+0000 7f024ce00640 1 --2- 172.21.15.27:0/599738416 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f02480cde10 0x7f0248155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:54.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.942+0000 7f024ce00640 1 -- 172.21.15.27:0/599738416 >> 172.21.15.27:0/599738416 conn(0x7f02480d0e80 msgr2=0x7f024806fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:54.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.942+0000 7f024ce00640 1 -- 172.21.15.27:0/599738416 shutdown_connections 2024-01-15T18:05:54.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:54.942+0000 7f024ce00640 1 -- 172.21.15.27:0/599738416 wait complete. 2024-01-15T18:05:54.967 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-01-15T18:05:55.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.106+0000 7ff0ff800640 1 Processor -- start 2024-01-15T18:05:55.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.106+0000 7ff0ff800640 1 -- start start 2024-01-15T18:05:55.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.106+0000 7ff0ff800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff0f81438e0 0x7ff0f8071d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.106+0000 7ff0ff800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0f806d640 con 0x7ff0f806ac00 2024-01-15T18:05:55.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.106+0000 7ff0ff800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0f8072370 con 0x7ff0f80cdb90 2024-01-15T18:05:55.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.106+0000 7ff0ff800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0f8072650 con 0x7ff0f81438e0 2024-01-15T18:05:55.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.106+0000 7ff0fce00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff0f81438e0 0x7ff0f8071d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.106+0000 7ff0fd800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff0f806ac00 0x7ff0f806e0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51808/0 (socket says 172.21.15.27:51808) 2024-01-15T18:05:55.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.106+0000 7ff0fce00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff0f81438e0 0x7ff0f8071d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52694/0 (socket says 172.21.15.27:52694) 2024-01-15T18:05:55.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.106+0000 7ff0fd800640 1 -- 172.21.15.27:0/466142403 learned_addr learned my addr 172.21.15.27:0/466142403 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:55.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.106+0000 7ff0fce00640 1 -- 172.21.15.27:0/466142403 >> v1:172.21.15.93:6789/0 conn(0x7ff0f80cdb90 legacy=0x7ff0f806a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.106+0000 7ff0fce00640 1 -- 172.21.15.27:0/466142403 >> v1:172.21.15.27:6789/0 conn(0x7ff0f806ac00 legacy=0x7ff0f806e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.106+0000 7ff0fce00640 1 -- 172.21.15.27:0/466142403 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f806d8e0 con 0x7ff0f81438e0 2024-01-15T18:05:55.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.110+0000 7ff0fce00640 1 --2- 172.21.15.27:0/466142403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0f81438e0 0x7ff0f8071d00 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff0e80097c0 tx=0x7ff0e801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93e573e22aa28893 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:55.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.110+0000 7ff0f7400640 1 -- 172.21.15.27:0/466142403 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e80ac600 con 0x7ff0f81438e0 2024-01-15T18:05:55.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.110+0000 7ff0f7400640 1 -- 172.21.15.27:0/466142403 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e80a8050 con 0x7ff0f81438e0 2024-01-15T18:05:55.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.110+0000 7ff0f7400640 1 -- 172.21.15.27:0/466142403 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e80ac980 con 0x7ff0f81438e0 2024-01-15T18:05:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.110+0000 7ff0ff800640 1 -- 172.21.15.27:0/466142403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0f81438e0 msgr2=0x7ff0f8071d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.110+0000 7ff0ff800640 1 --2- 172.21.15.27:0/466142403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0f81438e0 0x7ff0f8071d00 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff0e80097c0 tx=0x7ff0e801c3c0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.114+0000 7ff0ff800640 1 -- 172.21.15.27:0/466142403 shutdown_connections 2024-01-15T18:05:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.114+0000 7ff0ff800640 1 --2- 172.21.15.27:0/466142403 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0f81438e0 0x7ff0f8071d00 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.114+0000 7ff0ff800640 1 -- 172.21.15.27:0/466142403 >> 172.21.15.27:0/466142403 conn(0x7ff0f80cfd90 msgr2=0x7ff0f80d01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.114+0000 7ff0ff800640 1 -- 172.21.15.27:0/466142403 shutdown_connections 2024-01-15T18:05:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.114+0000 7ff0ff800640 1 -- 172.21.15.27:0/466142403 wait complete. 2024-01-15T18:05:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.114+0000 7ff0ff800640 1 Processor -- start 2024-01-15T18:05:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.114+0000 7ff0ff800640 1 -- start start 2024-01-15T18:05:55.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.114+0000 7ff0ff800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0f806ac00 0x7ff0f815a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.114+0000 7ff0ff800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0f80cdb90 0x7ff0f813e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.114+0000 7ff0ff800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0f81438e0 0x7ff0f813ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.114+0000 7ff0ff800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f813f3b0 con 0x7ff0f806ac00 2024-01-15T18:05:55.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.114+0000 7ff0ff800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f813b3e0 con 0x7ff0f80cdb90 2024-01-15T18:05:55.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.114+0000 7ff0ff800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f813b6c0 con 0x7ff0f81438e0 2024-01-15T18:05:55.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.114+0000 7ff0fce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0f806ac00 0x7ff0f815a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.114+0000 7ff0fce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0f806ac00 0x7ff0f815a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52702/0 (socket says 172.21.15.27:52702) 2024-01-15T18:05:55.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.114+0000 7ff0fce00640 1 -- 172.21.15.27:0/1685157981 learned_addr learned my addr 172.21.15.27:0/1685157981 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:55.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.114+0000 7ff0fd800640 1 --2- 172.21.15.27:0/1685157981 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0f81438e0 0x7ff0f813ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.114+0000 7ff0f7e00640 1 --2- 172.21.15.27:0/1685157981 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0f80cdb90 0x7ff0f813e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.114+0000 7ff0fce00640 1 -- 172.21.15.27:0/1685157981 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0f81438e0 msgr2=0x7ff0f813ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.114+0000 7ff0fce00640 1 --2- 172.21.15.27:0/1685157981 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0f81438e0 0x7ff0f813ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.114+0000 7ff0fce00640 1 -- 172.21.15.27:0/1685157981 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0f80cdb90 msgr2=0x7ff0f813e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.114+0000 7ff0fce00640 1 --2- 172.21.15.27:0/1685157981 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0f80cdb90 0x7ff0f813e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.114+0000 7ff0fce00640 1 -- 172.21.15.27:0/1685157981 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e8009420 con 0x7ff0f806ac00 2024-01-15T18:05:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.114+0000 7ff0fce00640 1 --2- 172.21.15.27:0/1685157981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0f806ac00 0x7ff0f815a1e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8018fd0 tx=0x7ff0e801ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.118+0000 7ff0f4c00640 1 -- 172.21.15.27:0/1685157981 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e80b7620 con 0x7ff0f806ac00 2024-01-15T18:05:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.118+0000 7ff0f4c00640 1 -- 172.21.15.27:0/1685157981 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e80b77a0 con 0x7ff0f806ac00 2024-01-15T18:05:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.118+0000 7ff0ff800640 1 -- 172.21.15.27:0/1685157981 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f813b9a0 con 0x7ff0f806ac00 2024-01-15T18:05:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.118+0000 7ff0f4c00640 1 -- 172.21.15.27:0/1685157981 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e80b7b50 con 0x7ff0f806ac00 2024-01-15T18:05:55.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.118+0000 7ff0ff800640 1 -- 172.21.15.27:0/1685157981 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f813bf60 con 0x7ff0f806ac00 2024-01-15T18:05:55.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.118+0000 7ff0ff800640 1 -- 172.21.15.27:0/1685157981 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f813c150 con 0x7ff0f806ac00 2024-01-15T18:05:55.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.118+0000 7ff0f4c00640 1 -- 172.21.15.27:0/1685157981 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7ff0e80b3040 con 0x7ff0f806ac00 2024-01-15T18:05:55.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.118+0000 7ff0f4c00640 1 --2- 172.21.15.27:0/1685157981 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff0d00668f0 0x7ff0d0068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.122+0000 7ff0f4c00640 1 -- 172.21.15.27:0/1685157981 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff0e811ea50 con 0x7ff0f806ac00 2024-01-15T18:05:55.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.122+0000 7ff0f7e00640 1 --2- 172.21.15.27:0/1685157981 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff0d00668f0 0x7ff0d0068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.122+0000 7ff0f7e00640 1 --2- 172.21.15.27:0/1685157981 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff0d00668f0 0x7ff0d0068db0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8142950 tx=0x7ff0e0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:55.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.122+0000 7ff0f4c00640 1 -- 172.21.15.27:0/1685157981 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff0e80ec190 con 0x7ff0f806ac00 2024-01-15T18:05:55.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.334+0000 7ff0ff800640 1 -- 172.21.15.27:0/1685157981 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7ff0f813b800 con 0x7ff0f806ac00 2024-01-15T18:05:55.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.338+0000 7ff0f4c00640 1 -- 172.21.15.27:0/1685157981 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v4) v1 ==== 90+0+0 (secure 0 0 0) 0x7ff0f813b800 con 0x7ff0f806ac00 2024-01-15T18:05:55.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.342+0000 7ff0ff800640 1 -- 172.21.15.27:0/1685157981 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff0d00668f0 msgr2=0x7ff0d0068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.342+0000 7ff0ff800640 1 --2- 172.21.15.27:0/1685157981 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff0d00668f0 0x7ff0d0068db0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8142950 tx=0x7ff0e0009040 comp rx=0 tx=0).stop 2024-01-15T18:05:55.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.342+0000 7ff0ff800640 1 -- 172.21.15.27:0/1685157981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0f806ac00 msgr2=0x7ff0f815a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.342+0000 7ff0ff800640 1 --2- 172.21.15.27:0/1685157981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0f806ac00 0x7ff0f815a1e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8018fd0 tx=0x7ff0e801ac00 comp rx=0 tx=0).stop 2024-01-15T18:05:55.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.342+0000 7ff0ff800640 1 -- 172.21.15.27:0/1685157981 shutdown_connections 2024-01-15T18:05:55.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.342+0000 7ff0ff800640 1 --2- 172.21.15.27:0/1685157981 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff0d00668f0 0x7ff0d0068db0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.342+0000 7ff0ff800640 1 --2- 172.21.15.27:0/1685157981 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0f81438e0 0x7ff0f813ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.342+0000 7ff0ff800640 1 --2- 172.21.15.27:0/1685157981 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0f80cdb90 0x7ff0f813e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.342+0000 7ff0ff800640 1 --2- 172.21.15.27:0/1685157981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0f806ac00 0x7ff0f815a1e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.342+0000 7ff0ff800640 1 -- 172.21.15.27:0/1685157981 >> 172.21.15.27:0/1685157981 conn(0x7ff0f80cfd90 msgr2=0x7ff0f80d0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:55.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.342+0000 7ff0ff800640 1 -- 172.21.15.27:0/1685157981 shutdown_connections 2024-01-15T18:05:55.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.342+0000 7ff0ff800640 1 -- 172.21.15.27:0/1685157981 wait complete. 2024-01-15T18:05:55.368 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-01-15T18:05:55.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.502+0000 7f3bf1800640 1 Processor -- start 2024-01-15T18:05:55.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.502+0000 7f3bf1800640 1 -- start start 2024-01-15T18:05:55.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.506+0000 7f3bf1800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3bec15d2a0 0x7f3bec0cfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.506+0000 7f3bf1800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f3bec15d680 0x7f3bec0cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.506+0000 7f3bf1800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bec0b3cd0 con 0x7f3bec0cdc40 2024-01-15T18:05:55.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.506+0000 7f3bf1800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bec0d0260 con 0x7f3bec15d2a0 2024-01-15T18:05:55.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.506+0000 7f3bf1800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bec0cb670 con 0x7f3bec15d680 2024-01-15T18:05:55.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.506+0000 7f3beaa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3bec15d2a0 0x7f3bec0cfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.506+0000 7f3beaa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3bec15d2a0 0x7f3bec0cfbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52704/0 (socket says 172.21.15.27:52704) 2024-01-15T18:05:55.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.506+0000 7f3beaa00640 1 -- 172.21.15.27:0/3508646982 learned_addr learned my addr 172.21.15.27:0/3508646982 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:55.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.506+0000 7f3bea000640 1 --2- 172.21.15.27:0/3508646982 >> v2:172.21.15.93:3300/0 conn(0x7f3bec15d680 0x7f3bec0cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.506+0000 7f3beaa00640 1 -- 172.21.15.27:0/3508646982 >> v2:172.21.15.93:3300/0 conn(0x7f3bec15d680 msgr2=0x7f3bec0cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:55.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.506+0000 7f3beaa00640 1 --2- 172.21.15.27:0/3508646982 >> v2:172.21.15.93:3300/0 conn(0x7f3bec15d680 0x7f3bec0cd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.506+0000 7f3beaa00640 1 -- 172.21.15.27:0/3508646982 >> v1:172.21.15.93:6789/0 conn(0x7f3bec0cdc40 legacy=0x7f3bec06b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:55.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.506+0000 7f3beaa00640 1 -- 172.21.15.27:0/3508646982 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bec0cb950 con 0x7f3bec15d2a0 2024-01-15T18:05:55.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.506+0000 7f3bea000640 1 --2- 172.21.15.27:0/3508646982 >> v2:172.21.15.93:3300/0 conn(0x7f3bec15d680 0x7f3bec0cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:55.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.510+0000 7f3beaa00640 1 --2- 172.21.15.27:0/3508646982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bec15d2a0 0x7f3bec0cfbd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3bec06d610 tx=0x7f3bdc01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73a9c2b0cdb4990d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:55.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.510+0000 7f3be9600640 1 -- 172.21.15.27:0/3508646982 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc0ac640 con 0x7f3bec15d2a0 2024-01-15T18:05:55.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.510+0000 7f3be9600640 1 -- 172.21.15.27:0/3508646982 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bdc0ac7a0 con 0x7f3bec15d2a0 2024-01-15T18:05:55.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.510+0000 7f3be9600640 1 -- 172.21.15.27:0/3508646982 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc0acb30 con 0x7f3bec15d2a0 2024-01-15T18:05:55.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.510+0000 7f3bf1800640 1 -- 172.21.15.27:0/3508646982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bec15d2a0 msgr2=0x7f3bec0cfbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.510+0000 7f3bf1800640 1 --2- 172.21.15.27:0/3508646982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bec15d2a0 0x7f3bec0cfbd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3bec06d610 tx=0x7f3bdc01c400 comp rx=0 tx=0).stop 2024-01-15T18:05:55.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.510+0000 7f3bf1800640 1 -- 172.21.15.27:0/3508646982 shutdown_connections 2024-01-15T18:05:55.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.510+0000 7f3bf1800640 1 --2- 172.21.15.27:0/3508646982 >> v2:172.21.15.93:3300/0 conn(0x7f3bec15d680 0x7f3bec0cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.510+0000 7f3bf1800640 1 --2- 172.21.15.27:0/3508646982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bec15d2a0 0x7f3bec0cfbd0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.510+0000 7f3bf1800640 1 -- 172.21.15.27:0/3508646982 >> 172.21.15.27:0/3508646982 conn(0x7f3bec0d0e80 msgr2=0x7f3bec0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:55.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.510+0000 7f3bf1800640 1 -- 172.21.15.27:0/3508646982 shutdown_connections 2024-01-15T18:05:55.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.510+0000 7f3bf1800640 1 -- 172.21.15.27:0/3508646982 wait complete. 2024-01-15T18:05:55.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.510+0000 7f3bf1800640 1 Processor -- start 2024-01-15T18:05:55.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.510+0000 7f3bf1800640 1 -- start start 2024-01-15T18:05:55.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.514+0000 7f3bf1800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3bec0cdc40 0x7f3bec14b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.514+0000 7f3bf1800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bec15d2a0 0x7f3bec14c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.514+0000 7f3bf1800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3bec15d680 0x7f3bec14c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.514+0000 7f3bf1800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec14e610 con 0x7f3bec15d2a0 2024-01-15T18:05:55.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.514+0000 7f3bf1800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec14dd40 con 0x7f3bec0cdc40 2024-01-15T18:05:55.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.514+0000 7f3bf1800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec152a00 con 0x7f3bec15d680 2024-01-15T18:05:55.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.514+0000 7f3bea000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bec15d2a0 0x7f3bec14c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.514+0000 7f3bea000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bec15d2a0 0x7f3bec14c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52706/0 (socket says 172.21.15.27:52706) 2024-01-15T18:05:55.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.514+0000 7f3bea000640 1 -- 172.21.15.27:0/2128333262 learned_addr learned my addr 172.21.15.27:0/2128333262 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:55.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.514+0000 7f3beaa00640 1 --2- 172.21.15.27:0/2128333262 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3bec0cdc40 0x7f3bec14b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.514+0000 7f3beb400640 1 --2- 172.21.15.27:0/2128333262 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3bec15d680 0x7f3bec14c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.514+0000 7f3bea000640 1 -- 172.21.15.27:0/2128333262 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3bec15d680 msgr2=0x7f3bec14c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.514+0000 7f3bea000640 1 --2- 172.21.15.27:0/2128333262 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3bec15d680 0x7f3bec14c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.514+0000 7f3bea000640 1 -- 172.21.15.27:0/2128333262 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3bec0cdc40 msgr2=0x7f3bec14b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.514+0000 7f3bea000640 1 --2- 172.21.15.27:0/2128333262 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3bec0cdc40 0x7f3bec14b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.514+0000 7f3bea000640 1 -- 172.21.15.27:0/2128333262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bdc0094d0 con 0x7f3bec15d2a0 2024-01-15T18:05:55.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.514+0000 7f3beaa00640 1 --2- 172.21.15.27:0/2128333262 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3bec0cdc40 0x7f3bec14b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:55.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.514+0000 7f3bea000640 1 --2- 172.21.15.27:0/2128333262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bec15d2a0 0x7f3bec14c0b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3bd400c7f0 tx=0x7f3bd400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:55.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.514+0000 7f3be2a00640 1 -- 172.21.15.27:0/2128333262 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd400cea0 con 0x7f3bec15d2a0 2024-01-15T18:05:55.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.514+0000 7f3be2a00640 1 -- 172.21.15.27:0/2128333262 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd4007a30 con 0x7f3bec15d2a0 2024-01-15T18:05:55.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.514+0000 7f3bf1800640 1 -- 172.21.15.27:0/2128333262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bec14dfc0 con 0x7f3bec15d2a0 2024-01-15T18:05:55.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.514+0000 7f3be2a00640 1 -- 172.21.15.27:0/2128333262 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd4007de0 con 0x7f3bec15d2a0 2024-01-15T18:05:55.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.514+0000 7f3bf1800640 1 -- 172.21.15.27:0/2128333262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bec14be40 con 0x7f3bec15d2a0 2024-01-15T18:05:55.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.514+0000 7f3bf1800640 1 -- 172.21.15.27:0/2128333262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bec14e7c0 con 0x7f3bec15d2a0 2024-01-15T18:05:55.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.514+0000 7f3be2a00640 1 -- 172.21.15.27:0/2128333262 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f3bd40113f0 con 0x7f3bec15d2a0 2024-01-15T18:05:55.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.518+0000 7f3be2a00640 1 --2- 172.21.15.27:0/2128333262 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3bc4066920 0x7f3bc4068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.518+0000 7f3be2a00640 1 -- 172.21.15.27:0/2128333262 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3bd4084130 con 0x7f3bec15d2a0 2024-01-15T18:05:55.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.518+0000 7f3beaa00640 1 --2- 172.21.15.27:0/2128333262 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3bc4066920 0x7f3bc4068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.518+0000 7f3beaa00640 1 --2- 172.21.15.27:0/2128333262 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3bc4066920 0x7f3bc4068de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc0ae000 tx=0x7f3bdc0a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:55.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.522+0000 7f3be2a00640 1 -- 172.21.15.27:0/2128333262 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3bec14e7c0 con 0x7f3bec15d2a0 2024-01-15T18:05:55.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.730+0000 7f3bf1800640 1 -- 172.21.15.27:0/2128333262 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f3bec14e7c0 con 0x7f3bec15d2a0 2024-01-15T18:05:55.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.734+0000 7f3be2a00640 1 -- 172.21.15.27:0/2128333262 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f3bec14e7c0 con 0x7f3bec15d2a0 2024-01-15T18:05:55.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.738+0000 7f3bf1800640 1 -- 172.21.15.27:0/2128333262 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3bc4066920 msgr2=0x7f3bc4068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.738+0000 7f3bf1800640 1 --2- 172.21.15.27:0/2128333262 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3bc4066920 0x7f3bc4068de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc0ae000 tx=0x7f3bdc0a7000 comp rx=0 tx=0).stop 2024-01-15T18:05:55.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.738+0000 7f3bf1800640 1 -- 172.21.15.27:0/2128333262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bec15d2a0 msgr2=0x7f3bec14c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.738+0000 7f3bf1800640 1 --2- 172.21.15.27:0/2128333262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bec15d2a0 0x7f3bec14c0b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3bd400c7f0 tx=0x7f3bd400ccb0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.738+0000 7f3bf1800640 1 -- 172.21.15.27:0/2128333262 shutdown_connections 2024-01-15T18:05:55.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.738+0000 7f3bf1800640 1 --2- 172.21.15.27:0/2128333262 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3bc4066920 0x7f3bc4068de0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.738+0000 7f3bf1800640 1 --2- 172.21.15.27:0/2128333262 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3bec15d680 0x7f3bec14c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.738+0000 7f3bf1800640 1 --2- 172.21.15.27:0/2128333262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bec15d2a0 0x7f3bec14c0b0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.738+0000 7f3bf1800640 1 --2- 172.21.15.27:0/2128333262 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3bec0cdc40 0x7f3bec14b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.738+0000 7f3bf1800640 1 -- 172.21.15.27:0/2128333262 >> 172.21.15.27:0/2128333262 conn(0x7f3bec0d0e80 msgr2=0x7f3bec0cf080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:55.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.738+0000 7f3bf1800640 1 -- 172.21.15.27:0/2128333262 shutdown_connections 2024-01-15T18:05:55.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.738+0000 7f3bf1800640 1 -- 172.21.15.27:0/2128333262 wait complete. 2024-01-15T18:05:55.763 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-01-15T18:05:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.898+0000 7f720e600640 1 Processor -- start 2024-01-15T18:05:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.898+0000 7f720e600640 1 -- start start 2024-01-15T18:05:55.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.898+0000 7f720e600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f720809a120 0x7f72080b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.898+0000 7f720e600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f72080b1d90 0x7f72080cf180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.898+0000 7f720e600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72080b1aa0 con 0x7f720815d7c0 2024-01-15T18:05:55.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.898+0000 7f720e600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72080b1300 con 0x7f72080b1d90 2024-01-15T18:05:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.898+0000 7f720e600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72080b25a0 con 0x7f720809a120 2024-01-15T18:05:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7207400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f720815d7c0 0x7f7208099a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51812/0 (socket says 172.21.15.27:51812) 2024-01-15T18:05:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7207400640 1 -- 172.21.15.27:0/1835820805 learned_addr learned my addr 172.21.15.27:0/1835820805 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7206a00640 1 --2- 172.21.15.27:0/1835820805 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f720809a120 0x7f72080b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7207e00640 1 --2- 172.21.15.27:0/1835820805 >> v2:172.21.15.93:3300/0 conn(0x7f72080b1d90 0x7f72080cf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7206000640 1 -- 172.21.15.27:0/1835820805 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197156846 0 0) 0x7f72080b1aa0 con 0x7f720815d7c0 2024-01-15T18:05:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7206000640 1 -- 172.21.15.27:0/1835820805 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71e8003530 con 0x7f720815d7c0 2024-01-15T18:05:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7206000640 1 -- 172.21.15.27:0/1835820805 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2540928277 0 0) 0x7f71e8003530 con 0x7f720815d7c0 2024-01-15T18:05:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7206000640 1 -- 172.21.15.27:0/1835820805 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f720809a120 msgr2=0x7f72080b35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7206000640 1 --2- 172.21.15.27:0/1835820805 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f720809a120 0x7f72080b35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7206000640 1 -- 172.21.15.27:0/1835820805 >> v2:172.21.15.93:3300/0 conn(0x7f72080b1d90 msgr2=0x7f72080cf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7206000640 1 --2- 172.21.15.27:0/1835820805 >> v2:172.21.15.93:3300/0 conn(0x7f72080b1d90 0x7f72080cf180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7206000640 1 -- 172.21.15.27:0/1835820805 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72080b14e0 con 0x7f720815d7c0 2024-01-15T18:05:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7206000640 1 -- 172.21.15.27:0/1835820805 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f71f8001680 con 0x7f720815d7c0 2024-01-15T18:05:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7206000640 1 -- 172.21.15.27:0/1835820805 >> v1:172.21.15.27:6789/0 conn(0x7f720815d7c0 legacy=0x7f7208099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7206000640 1 --2- 172.21.15.27:0/1835820805 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f71e80039f0 0x7f71e8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7206000640 1 --2- 172.21.15.27:0/1835820805 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f720809a120 0x7f71e80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7206000640 1 --2- 172.21.15.27:0/1835820805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71e8004d10 0x7f71e8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7206000640 1 -- 172.21.15.27:0/1835820805 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72080b1300 con 0x7f71e8004d10 2024-01-15T18:05:55.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7206000640 1 -- 172.21.15.27:0/1835820805 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72080b25a0 con 0x7f71e80039f0 2024-01-15T18:05:55.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7206000640 1 -- 172.21.15.27:0/1835820805 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e800b9e0 con 0x7f720809a120 2024-01-15T18:05:55.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7207e00640 1 --2- 172.21.15.27:0/1835820805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71e8004d10 0x7f71e8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7206a00640 1 --2- 172.21.15.27:0/1835820805 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f720809a120 0x7f71e80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7207400640 1 --2- 172.21.15.27:0/1835820805 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f71e80039f0 0x7f71e8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7207e00640 1 -- 172.21.15.27:0/1835820805 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f720809a120 msgr2=0x7f71e80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:55.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7207e00640 1 --2- 172.21.15.27:0/1835820805 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f720809a120 0x7f71e80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7207e00640 1 -- 172.21.15.27:0/1835820805 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f71e80039f0 msgr2=0x7f71e8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:55.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7207e00640 1 --2- 172.21.15.27:0/1835820805 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f71e80039f0 0x7f71e8003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.902+0000 7f7207e00640 1 -- 172.21.15.27:0/1835820805 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e8003530 con 0x7f71e8004d10 2024-01-15T18:05:55.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.906+0000 7f7207e00640 1 --2- 172.21.15.27:0/1835820805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71e8004d10 0x7f71e8005190 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f720806e450 tx=0x7f720001cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1058b1001c367c6b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:55.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.906+0000 7f7206000640 1 -- 172.21.15.27:0/1835820805 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72000a5c50 con 0x7f71e8004d10 2024-01-15T18:05:55.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.906+0000 7f7206000640 1 -- 172.21.15.27:0/1835820805 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f720000f040 con 0x7f71e8004d10 2024-01-15T18:05:55.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.906+0000 7f7206000640 1 -- 172.21.15.27:0/1835820805 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72000ad440 con 0x7f71e8004d10 2024-01-15T18:05:55.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.906+0000 7f720e600640 1 -- 172.21.15.27:0/1835820805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71e8004d10 msgr2=0x7f71e8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.906+0000 7f720e600640 1 --2- 172.21.15.27:0/1835820805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71e8004d10 0x7f71e8005190 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f720806e450 tx=0x7f720001cc40 comp rx=0 tx=0).stop 2024-01-15T18:05:55.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.906+0000 7f7207e00640 1 -- 172.21.15.27:0/1835820805 reap_dead start 2024-01-15T18:05:55.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.906+0000 7f720e600640 1 -- 172.21.15.27:0/1835820805 shutdown_connections 2024-01-15T18:05:55.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.906+0000 7f720e600640 1 -- 172.21.15.27:0/1835820805 >> 172.21.15.27:0/1835820805 conn(0x7f72080b2a80 msgr2=0x7f72080b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:55.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.906+0000 7f720e600640 1 -- 172.21.15.27:0/1835820805 shutdown_connections 2024-01-15T18:05:55.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.906+0000 7f720e600640 1 -- 172.21.15.27:0/1835820805 wait complete. 2024-01-15T18:05:55.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.906+0000 7f720e600640 1 Processor -- start 2024-01-15T18:05:55.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.906+0000 7f720e600640 1 -- start start 2024-01-15T18:05:55.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.910+0000 7f720e600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71e8004d10 0x7f72081f3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.910+0000 7f720e600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f72080b6820 0x7f72080b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.910+0000 7f720e600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f72080b5240 0x7f72080c8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.910+0000 7f720e600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72080b6400 con 0x7f71e8004d10 2024-01-15T18:05:55.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.910+0000 7f720e600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72080b6ca0 con 0x7f72080b5240 2024-01-15T18:05:55.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.910+0000 7f720e600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72080b5780 con 0x7f72080b6820 2024-01-15T18:05:55.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.910+0000 7f7207400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71e8004d10 0x7f72081f3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.910+0000 7f7207400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71e8004d10 0x7f72081f3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52726/0 (socket says 172.21.15.27:52726) 2024-01-15T18:05:55.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.910+0000 7f7207400640 1 -- 172.21.15.27:0/2836263983 learned_addr learned my addr 172.21.15.27:0/2836263983 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:55.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.910+0000 7f7207e00640 1 --2- 172.21.15.27:0/2836263983 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f72080b5240 0x7f72080c8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.910+0000 7f7206a00640 1 --2- 172.21.15.27:0/2836263983 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f72080b6820 0x7f72080b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.910+0000 7f7207400640 1 -- 172.21.15.27:0/2836263983 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f72080b6820 msgr2=0x7f72080b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.910+0000 7f7207400640 1 --2- 172.21.15.27:0/2836263983 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f72080b6820 0x7f72080b4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.910+0000 7f7207400640 1 -- 172.21.15.27:0/2836263983 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f72080b5240 msgr2=0x7f72080c8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.910+0000 7f7207400640 1 --2- 172.21.15.27:0/2836263983 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f72080b5240 0x7f72080c8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.910+0000 7f7207400640 1 -- 172.21.15.27:0/2836263983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72080b5a60 con 0x7f71e8004d10 2024-01-15T18:05:55.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.910+0000 7f7207400640 1 --2- 172.21.15.27:0/2836263983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71e8004d10 0x7f72081f3480 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f71f8012540 tx=0x7f71f8012940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:55.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.910+0000 7f71ff400640 1 -- 172.21.15.27:0/2836263983 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f8014750 con 0x7f71e8004d10 2024-01-15T18:05:55.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.910+0000 7f71ff400640 1 -- 172.21.15.27:0/2836263983 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f80148b0 con 0x7f71e8004d10 2024-01-15T18:05:55.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.910+0000 7f71ff400640 1 -- 172.21.15.27:0/2836263983 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f8014c60 con 0x7f71e8004d10 2024-01-15T18:05:55.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.910+0000 7f720e600640 1 -- 172.21.15.27:0/2836263983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72080b5d40 con 0x7f71e8004d10 2024-01-15T18:05:55.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.910+0000 7f720e600640 1 -- 172.21.15.27:0/2836263983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72080cac60 con 0x7f71e8004d10 2024-01-15T18:05:55.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.910+0000 7f720e600640 1 -- 172.21.15.27:0/2836263983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72080b6540 con 0x7f71e8004d10 2024-01-15T18:05:55.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.914+0000 7f71ff400640 1 -- 172.21.15.27:0/2836263983 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f71f80153e0 con 0x7f71e8004d10 2024-01-15T18:05:55.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.914+0000 7f71ff400640 1 --2- 172.21.15.27:0/2836263983 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f71e00668f0 0x7f71e0068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.914+0000 7f71ff400640 1 -- 172.21.15.27:0/2836263983 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f71f8086870 con 0x7f71e8004d10 2024-01-15T18:05:55.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.914+0000 7f7206a00640 1 --2- 172.21.15.27:0/2836263983 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f71e00668f0 0x7f71e0068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.914+0000 7f7206a00640 1 --2- 172.21.15.27:0/2836263983 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f71e00668f0 0x7f71e0068db0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f720809f040 tx=0x7f71f0009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:55.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:55.918+0000 7f71ff400640 1 -- 172.21.15.27:0/2836263983 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f72080b6540 con 0x7f71e8004d10 2024-01-15T18:05:56.112 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:05:56.115 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:05:56.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.130+0000 7f720e600640 1 -- 172.21.15.27:0/2836263983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f720806bc30 con 0x7f71e8004d10 2024-01-15T18:05:56.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.134+0000 7f71ff400640 1 -- 172.21.15.27:0/2836263983 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f71f801c0a0 con 0x7f71e8004d10 2024-01-15T18:05:56.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.138+0000 7f720e600640 1 -- 172.21.15.27:0/2836263983 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f71e00668f0 msgr2=0x7f71e0068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.138+0000 7f720e600640 1 --2- 172.21.15.27:0/2836263983 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f71e00668f0 0x7f71e0068db0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f720809f040 tx=0x7f71f0009190 comp rx=0 tx=0).stop 2024-01-15T18:05:56.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.138+0000 7f720e600640 1 -- 172.21.15.27:0/2836263983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71e8004d10 msgr2=0x7f72081f3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.138+0000 7f720e600640 1 --2- 172.21.15.27:0/2836263983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71e8004d10 0x7f72081f3480 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f71f8012540 tx=0x7f71f8012940 comp rx=0 tx=0).stop 2024-01-15T18:05:56.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.138+0000 7f720e600640 1 -- 172.21.15.27:0/2836263983 shutdown_connections 2024-01-15T18:05:56.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.138+0000 7f720e600640 1 --2- 172.21.15.27:0/2836263983 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f71e00668f0 0x7f71e0068db0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.138+0000 7f720e600640 1 --2- 172.21.15.27:0/2836263983 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f72080b5240 0x7f72080c8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.138+0000 7f720e600640 1 --2- 172.21.15.27:0/2836263983 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f72080b6820 0x7f72080b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.138+0000 7f720e600640 1 --2- 172.21.15.27:0/2836263983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71e8004d10 0x7f72081f3480 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.138+0000 7f720e600640 1 -- 172.21.15.27:0/2836263983 >> 172.21.15.27:0/2836263983 conn(0x7f72080b2a80 msgr2=0x7f72080cd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:56.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.138+0000 7f720e600640 1 -- 172.21.15.27:0/2836263983 shutdown_connections 2024-01-15T18:05:56.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.138+0000 7f720e600640 1 -- 172.21.15.27:0/2836263983 wait complete. 2024-01-15T18:05:56.163 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-01-15T18:05:56.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.342+0000 7f94b2400640 1 Processor -- start 2024-01-15T18:05:56.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.342+0000 7f94b2400640 1 -- start start 2024-01-15T18:05:56.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.342+0000 7f94b2400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f94ac0719f0 0x7f94ac15a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.342+0000 7f94b2400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f94ac15d9e0 0x7f94ac15bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.342+0000 7f94b2400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ac15dfa0 con 0x7f94ac15b060 2024-01-15T18:05:56.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.342+0000 7f94b2400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94ac15b440 con 0x7f94ac0719f0 2024-01-15T18:05:56.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.342+0000 7f94b2400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94ac15e280 con 0x7f94ac15d9e0 2024-01-15T18:05:56.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.342+0000 7f94aaa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f94ac0719f0 0x7f94ac15a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:56.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.342+0000 7f94aaa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f94ac0719f0 0x7f94ac15a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52730/0 (socket says 172.21.15.27:52730) 2024-01-15T18:05:56.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.342+0000 7f94aaa00640 1 -- 172.21.15.27:0/311547184 learned_addr learned my addr 172.21.15.27:0/311547184 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:56.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.342+0000 7f94ab400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f94ac15b060 0x7f94ac0712e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51826/0 (socket says 172.21.15.27:51826) 2024-01-15T18:05:56.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.342+0000 7f94abe00640 1 --2- 172.21.15.27:0/311547184 >> v2:172.21.15.93:3300/0 conn(0x7f94ac15d9e0 0x7f94ac15bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:56.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.342+0000 7f94aaa00640 1 -- 172.21.15.27:0/311547184 >> v2:172.21.15.93:3300/0 conn(0x7f94ac15d9e0 msgr2=0x7f94ac15bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:56.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.342+0000 7f94aaa00640 1 --2- 172.21.15.27:0/311547184 >> v2:172.21.15.93:3300/0 conn(0x7f94ac15d9e0 0x7f94ac15bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.342+0000 7f94aaa00640 1 -- 172.21.15.27:0/311547184 >> v1:172.21.15.27:6789/0 conn(0x7f94ac15b060 legacy=0x7f94ac0712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.342+0000 7f94aaa00640 1 -- 172.21.15.27:0/311547184 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ac15e560 con 0x7f94ac0719f0 2024-01-15T18:05:56.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.342+0000 7f94aa000640 1 -- 172.21.15.27:0/311547184 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3118567509 0 0) 0x7f94ac15dfa0 con 0x7f94ac15b060 2024-01-15T18:05:56.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.342+0000 7f94abe00640 1 --2- 172.21.15.27:0/311547184 >> v2:172.21.15.93:3300/0 conn(0x7f94ac15d9e0 0x7f94ac15bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:56.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.346+0000 7f94aaa00640 1 --2- 172.21.15.27:0/311547184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94ac0719f0 0x7f94ac15a3f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f94ac0cb710 tx=0x7f949401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89b6633ca382fdca server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:56.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.346+0000 7f94aa000640 1 -- 172.21.15.27:0/311547184 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94940ab640 con 0x7f94ac0719f0 2024-01-15T18:05:56.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.346+0000 7f94aa000640 1 -- 172.21.15.27:0/311547184 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94940a7070 con 0x7f94ac0719f0 2024-01-15T18:05:56.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.346+0000 7f94aa000640 1 -- 172.21.15.27:0/311547184 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94940ab9c0 con 0x7f94ac0719f0 2024-01-15T18:05:56.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.346+0000 7f94b2400640 1 -- 172.21.15.27:0/311547184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94ac0719f0 msgr2=0x7f94ac15a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.346+0000 7f94b2400640 1 --2- 172.21.15.27:0/311547184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94ac0719f0 0x7f94ac15a3f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f94ac0cb710 tx=0x7f949401c440 comp rx=0 tx=0).stop 2024-01-15T18:05:56.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.346+0000 7f94b2400640 1 -- 172.21.15.27:0/311547184 shutdown_connections 2024-01-15T18:05:56.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.346+0000 7f94b2400640 1 --2- 172.21.15.27:0/311547184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94ac0719f0 0x7f94ac15a3f0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.346+0000 7f94b2400640 1 --2- 172.21.15.27:0/311547184 >> v2:172.21.15.93:3300/0 conn(0x7f94ac15d9e0 0x7f94ac15bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.346+0000 7f94b2400640 1 -- 172.21.15.27:0/311547184 >> 172.21.15.27:0/311547184 conn(0x7f94ac0b0850 msgr2=0x7f94ac0b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:56.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.346+0000 7f94b2400640 1 -- 172.21.15.27:0/311547184 shutdown_connections 2024-01-15T18:05:56.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.346+0000 7f94b2400640 1 -- 172.21.15.27:0/311547184 wait complete. 2024-01-15T18:05:56.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.350+0000 7f94b2400640 1 Processor -- start 2024-01-15T18:05:56.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.350+0000 7f94b2400640 1 -- start start 2024-01-15T18:05:56.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.350+0000 7f94b2400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f94ac0719f0 0x7f94ac1f5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.350+0000 7f94b2400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94ac15b060 0x7f94ac150800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.350+0000 7f94b2400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f94ac15d9e0 0x7f94ac150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.350+0000 7f94b2400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac0cdc60 con 0x7f94ac15b060 2024-01-15T18:05:56.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.350+0000 7f94b2400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac0cdf40 con 0x7f94ac0719f0 2024-01-15T18:05:56.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.350+0000 7f94b2400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94ac14bc80 con 0x7f94ac15d9e0 2024-01-15T18:05:56.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.350+0000 7f94aaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94ac15b060 0x7f94ac150800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:56.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.350+0000 7f94aaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94ac15b060 0x7f94ac150800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52742/0 (socket says 172.21.15.27:52742) 2024-01-15T18:05:56.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.350+0000 7f94aaa00640 1 -- 172.21.15.27:0/1742990873 learned_addr learned my addr 172.21.15.27:0/1742990873 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:56.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.350+0000 7f94abe00640 1 --2- 172.21.15.27:0/1742990873 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f94ac15d9e0 0x7f94ac150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:56.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.350+0000 7f94ab400640 1 --2- 172.21.15.27:0/1742990873 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f94ac0719f0 0x7f94ac1f5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:56.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.350+0000 7f94aaa00640 1 -- 172.21.15.27:0/1742990873 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f94ac15d9e0 msgr2=0x7f94ac150d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.350+0000 7f94aaa00640 1 --2- 172.21.15.27:0/1742990873 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f94ac15d9e0 0x7f94ac150d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.350+0000 7f94aaa00640 1 -- 172.21.15.27:0/1742990873 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f94ac0719f0 msgr2=0x7f94ac1f5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.350+0000 7f94aaa00640 1 --2- 172.21.15.27:0/1742990873 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f94ac0719f0 0x7f94ac1f5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.350+0000 7f94aaa00640 1 -- 172.21.15.27:0/1742990873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94940094d0 con 0x7f94ac15b060 2024-01-15T18:05:56.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.350+0000 7f94aaa00640 1 --2- 172.21.15.27:0/1742990873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94ac15b060 0x7f94ac150800 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9494019050 tx=0x7f949401ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:56.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.350+0000 7f94a3400640 1 -- 172.21.15.27:0/1742990873 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94940be3f0 con 0x7f94ac15b060 2024-01-15T18:05:56.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.350+0000 7f94a3400640 1 -- 172.21.15.27:0/1742990873 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94940be550 con 0x7f94ac15b060 2024-01-15T18:05:56.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.350+0000 7f94b2400640 1 -- 172.21.15.27:0/1742990873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94ac14bf60 con 0x7f94ac15b060 2024-01-15T18:05:56.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.350+0000 7f94a3400640 1 -- 172.21.15.27:0/1742990873 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94940be8e0 con 0x7f94ac15b060 2024-01-15T18:05:56.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.350+0000 7f94b2400640 1 -- 172.21.15.27:0/1742990873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94ac159b50 con 0x7f94ac15b060 2024-01-15T18:05:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.354+0000 7f94b2400640 1 -- 172.21.15.27:0/1742990873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94ac14bae0 con 0x7f94ac15b060 2024-01-15T18:05:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.354+0000 7f94a3400640 1 -- 172.21.15.27:0/1742990873 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f94940b43f0 con 0x7f94ac15b060 2024-01-15T18:05:56.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.354+0000 7f94a3400640 1 --2- 172.21.15.27:0/1742990873 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f94840668f0 0x7f9484068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.354+0000 7f94a3400640 1 -- 172.21.15.27:0/1742990873 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f949411e8c0 con 0x7f94ac15b060 2024-01-15T18:05:56.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.354+0000 7f94ab400640 1 --2- 172.21.15.27:0/1742990873 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f94840668f0 0x7f9484068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:56.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.354+0000 7f94ab400640 1 --2- 172.21.15.27:0/1742990873 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f94840668f0 0x7f9484068db0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f949c000f70 tx=0x7f949c007b60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:56.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.358+0000 7f94a3400640 1 -- 172.21.15.27:0/1742990873 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f94ac14bae0 con 0x7f94ac15b060 2024-01-15T18:05:56.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.570+0000 7f94b2400640 1 -- 172.21.15.27:0/1742990873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f94ac14bae0 con 0x7f94ac15b060 2024-01-15T18:05:56.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.574+0000 7f94a3400640 1 -- 172.21.15.27:0/1742990873 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v7) v1 ==== 97+0+0 (secure 0 0 0) 0x7f94ac14bae0 con 0x7f94ac15b060 2024-01-15T18:05:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.574+0000 7f94b2400640 1 -- 172.21.15.27:0/1742990873 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f94840668f0 msgr2=0x7f9484068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.574+0000 7f94b2400640 1 --2- 172.21.15.27:0/1742990873 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f94840668f0 0x7f9484068db0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f949c000f70 tx=0x7f949c007b60 comp rx=0 tx=0).stop 2024-01-15T18:05:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.574+0000 7f94b2400640 1 -- 172.21.15.27:0/1742990873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94ac15b060 msgr2=0x7f94ac150800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.574+0000 7f94b2400640 1 --2- 172.21.15.27:0/1742990873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94ac15b060 0x7f94ac150800 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9494019050 tx=0x7f949401ac80 comp rx=0 tx=0).stop 2024-01-15T18:05:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.574+0000 7f94b2400640 1 -- 172.21.15.27:0/1742990873 shutdown_connections 2024-01-15T18:05:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.574+0000 7f94b2400640 1 --2- 172.21.15.27:0/1742990873 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f94840668f0 0x7f9484068db0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.574+0000 7f94b2400640 1 --2- 172.21.15.27:0/1742990873 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f94ac15d9e0 0x7f94ac150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.574+0000 7f94b2400640 1 --2- 172.21.15.27:0/1742990873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94ac15b060 0x7f94ac150800 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.574+0000 7f94b2400640 1 --2- 172.21.15.27:0/1742990873 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f94ac0719f0 0x7f94ac1f5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.574+0000 7f94b2400640 1 -- 172.21.15.27:0/1742990873 >> 172.21.15.27:0/1742990873 conn(0x7f94ac0b0850 msgr2=0x7f94ac0cec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.574+0000 7f94b2400640 1 -- 172.21.15.27:0/1742990873 shutdown_connections 2024-01-15T18:05:56.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.574+0000 7f94b2400640 1 -- 172.21.15.27:0/1742990873 wait complete. 2024-01-15T18:05:56.602 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-01-15T18:05:56.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.738+0000 7f5e89a00640 1 Processor -- start 2024-01-15T18:05:56.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.738+0000 7f5e89a00640 1 -- start start 2024-01-15T18:05:56.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.738+0000 7f5e89a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e84143da0 0x7f5e8406d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.738+0000 7f5e89a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5e840cad40 0x7f5e840cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.738+0000 7f5e89a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e8406c750 con 0x7f5e8406c200 2024-01-15T18:05:56.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.738+0000 7f5e89a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e840cb710 con 0x7f5e840cad40 2024-01-15T18:05:56.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.738+0000 7f5e89a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e84144180 con 0x7f5e84143da0 2024-01-15T18:05:56.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.738+0000 7f5e83400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5e840cad40 0x7f5e840cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:56.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.738+0000 7f5e82000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5e8406c200 0x7f5e840cb9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51836/0 (socket says 172.21.15.27:51836) 2024-01-15T18:05:56.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.738+0000 7f5e82000640 1 -- 172.21.15.27:0/3222544885 learned_addr learned my addr 172.21.15.27:0/3222544885 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:56.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.738+0000 7f5e82a00640 1 --2- 172.21.15.27:0/3222544885 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e84143da0 0x7f5e8406d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:56.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.738+0000 7f5e83400640 1 -- 172.21.15.27:0/3222544885 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e84143da0 msgr2=0x7f5e8406d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:56.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.738+0000 7f5e83400640 1 --2- 172.21.15.27:0/3222544885 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e84143da0 0x7f5e8406d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.738+0000 7f5e83400640 1 -- 172.21.15.27:0/3222544885 >> v1:172.21.15.27:6789/0 conn(0x7f5e8406c200 legacy=0x7f5e840cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.738+0000 7f5e83400640 1 -- 172.21.15.27:0/3222544885 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8406ca30 con 0x7f5e840cad40 2024-01-15T18:05:56.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.738+0000 7f5e82a00640 1 --2- 172.21.15.27:0/3222544885 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e84143da0 0x7f5e8406d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:56.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.742+0000 7f5e83400640 1 --2- 172.21.15.27:0/3222544885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e840cad40 0x7f5e840cb1d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c009d90 tx=0x7f5e7c01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58d7d791a1b567ef server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:56.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.742+0000 7f5e81600640 1 -- 172.21.15.27:0/3222544885 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c0a5900 con 0x7f5e840cad40 2024-01-15T18:05:56.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.742+0000 7f5e81600640 1 -- 172.21.15.27:0/3222544885 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7c0a5a60 con 0x7f5e840cad40 2024-01-15T18:05:56.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.742+0000 7f5e81600640 1 -- 172.21.15.27:0/3222544885 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c0a5df0 con 0x7f5e840cad40 2024-01-15T18:05:56.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.742+0000 7f5e89a00640 1 -- 172.21.15.27:0/3222544885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e840cad40 msgr2=0x7f5e840cb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.742+0000 7f5e89a00640 1 --2- 172.21.15.27:0/3222544885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e840cad40 0x7f5e840cb1d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c009d90 tx=0x7f5e7c01c950 comp rx=0 tx=0).stop 2024-01-15T18:05:56.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.742+0000 7f5e89a00640 1 -- 172.21.15.27:0/3222544885 shutdown_connections 2024-01-15T18:05:56.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.742+0000 7f5e89a00640 1 --2- 172.21.15.27:0/3222544885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e840cad40 0x7f5e840cb1d0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.742+0000 7f5e89a00640 1 --2- 172.21.15.27:0/3222544885 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e84143da0 0x7f5e8406d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.742+0000 7f5e89a00640 1 -- 172.21.15.27:0/3222544885 >> 172.21.15.27:0/3222544885 conn(0x7f5e840b0850 msgr2=0x7f5e840b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:56.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.742+0000 7f5e89a00640 1 -- 172.21.15.27:0/3222544885 shutdown_connections 2024-01-15T18:05:56.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.742+0000 7f5e89a00640 1 -- 172.21.15.27:0/3222544885 wait complete. 2024-01-15T18:05:56.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.746+0000 7f5e89a00640 1 Processor -- start 2024-01-15T18:05:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.746+0000 7f5e89a00640 1 -- start start 2024-01-15T18:05:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.746+0000 7f5e89a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e8406c200 0x7f5e8415da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.746+0000 7f5e89a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e840cad40 0x7f5e84139260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.746+0000 7f5e89a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5e84143da0 0x7f5e841397a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.746+0000 7f5e89a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8413fe00 con 0x7f5e840cad40 2024-01-15T18:05:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.746+0000 7f5e89a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e841400e0 con 0x7f5e84143da0 2024-01-15T18:05:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.746+0000 7f5e89a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e841403c0 con 0x7f5e8406c200 2024-01-15T18:05:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.746+0000 7f5e82000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e840cad40 0x7f5e84139260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:56.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.746+0000 7f5e82000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e840cad40 0x7f5e84139260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52756/0 (socket says 172.21.15.27:52756) 2024-01-15T18:05:56.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.746+0000 7f5e82000640 1 -- 172.21.15.27:0/1475530893 learned_addr learned my addr 172.21.15.27:0/1475530893 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:56.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.746+0000 7f5e83400640 1 --2- 172.21.15.27:0/1475530893 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5e84143da0 0x7f5e841397a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:56.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.746+0000 7f5e82a00640 1 --2- 172.21.15.27:0/1475530893 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e8406c200 0x7f5e8415da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:56.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.746+0000 7f5e82000640 1 -- 172.21.15.27:0/1475530893 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e8406c200 msgr2=0x7f5e8415da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.746+0000 7f5e82000640 1 --2- 172.21.15.27:0/1475530893 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e8406c200 0x7f5e8415da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.746+0000 7f5e82000640 1 -- 172.21.15.27:0/1475530893 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5e84143da0 msgr2=0x7f5e841397a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.746+0000 7f5e82000640 1 --2- 172.21.15.27:0/1475530893 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5e84143da0 0x7f5e841397a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.746+0000 7f5e82000640 1 -- 172.21.15.27:0/1475530893 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e7c009a70 con 0x7f5e840cad40 2024-01-15T18:05:56.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.746+0000 7f5e82000640 1 --2- 172.21.15.27:0/1475530893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e840cad40 0x7f5e84139260 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c00c500 tx=0x7f5e6c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:56.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.746+0000 7f5e7aa00640 1 -- 172.21.15.27:0/1475530893 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c005cf0 con 0x7f5e840cad40 2024-01-15T18:05:56.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.746+0000 7f5e7aa00640 1 -- 172.21.15.27:0/1475530893 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6c005e50 con 0x7f5e840cad40 2024-01-15T18:05:56.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.746+0000 7f5e7aa00640 1 -- 172.21.15.27:0/1475530893 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c0072f0 con 0x7f5e840cad40 2024-01-15T18:05:56.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.746+0000 7f5e89a00640 1 -- 172.21.15.27:0/1475530893 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e841406a0 con 0x7f5e840cad40 2024-01-15T18:05:56.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.750+0000 7f5e89a00640 1 -- 172.21.15.27:0/1475530893 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e8413ae10 con 0x7f5e840cad40 2024-01-15T18:05:56.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.750+0000 7f5e89a00640 1 -- 172.21.15.27:0/1475530893 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e840ac910 con 0x7f5e840cad40 2024-01-15T18:05:56.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.750+0000 7f5e7aa00640 1 -- 172.21.15.27:0/1475530893 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f5e6c007450 con 0x7f5e840cad40 2024-01-15T18:05:56.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.750+0000 7f5e7aa00640 1 --2- 172.21.15.27:0/1475530893 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5e5c0668f0 0x7f5e5c068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.750+0000 7f5e7aa00640 1 -- 172.21.15.27:0/1475530893 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5e6c083630 con 0x7f5e840cad40 2024-01-15T18:05:56.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.750+0000 7f5e82a00640 1 --2- 172.21.15.27:0/1475530893 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5e5c0668f0 0x7f5e5c068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:56.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.750+0000 7f5e82a00640 1 --2- 172.21.15.27:0/1475530893 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5e5c0668f0 0x7f5e5c068db0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5e74006fd0 tx=0x7f5e74008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:56.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.754+0000 7f5e7aa00640 1 -- 172.21.15.27:0/1475530893 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5e840ac910 con 0x7f5e840cad40 2024-01-15T18:05:56.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.970+0000 7f5e89a00640 1 -- 172.21.15.27:0/1475530893 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f5e840ac910 con 0x7f5e840cad40 2024-01-15T18:05:56.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.974+0000 7f5e7aa00640 1 -- 172.21.15.27:0/1475530893 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f5e840ac910 con 0x7f5e840cad40 2024-01-15T18:05:56.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.974+0000 7f5e89a00640 1 -- 172.21.15.27:0/1475530893 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5e5c0668f0 msgr2=0x7f5e5c068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.974+0000 7f5e89a00640 1 --2- 172.21.15.27:0/1475530893 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5e5c0668f0 0x7f5e5c068db0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5e74006fd0 tx=0x7f5e74008040 comp rx=0 tx=0).stop 2024-01-15T18:05:56.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.974+0000 7f5e89a00640 1 -- 172.21.15.27:0/1475530893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e840cad40 msgr2=0x7f5e84139260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.974+0000 7f5e89a00640 1 --2- 172.21.15.27:0/1475530893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e840cad40 0x7f5e84139260 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c00c500 tx=0x7f5e6c00c9c0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.974+0000 7f5e89a00640 1 -- 172.21.15.27:0/1475530893 shutdown_connections 2024-01-15T18:05:56.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.974+0000 7f5e89a00640 1 --2- 172.21.15.27:0/1475530893 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5e5c0668f0 0x7f5e5c068db0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.974+0000 7f5e89a00640 1 --2- 172.21.15.27:0/1475530893 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5e84143da0 0x7f5e841397a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.974+0000 7f5e89a00640 1 --2- 172.21.15.27:0/1475530893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e840cad40 0x7f5e84139260 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.974+0000 7f5e89a00640 1 --2- 172.21.15.27:0/1475530893 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e8406c200 0x7f5e8415da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.974+0000 7f5e89a00640 1 -- 172.21.15.27:0/1475530893 >> 172.21.15.27:0/1475530893 conn(0x7f5e840b0850 msgr2=0x7f5e840afd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:56.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.974+0000 7f5e89a00640 1 -- 172.21.15.27:0/1475530893 shutdown_connections 2024-01-15T18:05:56.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:56.974+0000 7f5e89a00640 1 -- 172.21.15.27:0/1475530893 wait complete. 2024-01-15T18:05:57.001 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:57.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.138+0000 7f27d9200640 1 Processor -- start 2024-01-15T18:05:57.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.138+0000 7f27d9200640 1 -- start start 2024-01-15T18:05:57.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.138+0000 7f27d9200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27d40b2390 0x7f27d40cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.138+0000 7f27d9200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f27d406bb80 0x7f27d40cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.138+0000 7f27d9200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f27d40cfcc0 0x7f27d40cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.138+0000 7f27d9200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27d40b2770 con 0x7f27d406bb80 2024-01-15T18:05:57.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.138+0000 7f27d9200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27d406c060 con 0x7f27d40cfcc0 2024-01-15T18:05:57.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.138+0000 7f27d9200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27d406c340 con 0x7f27d40b2390 2024-01-15T18:05:57.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.138+0000 7f27d1400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f27d406bb80 0x7f27d40cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.138+0000 7f27d1e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27d40b2390 0x7f27d40cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.138+0000 7f27d2800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f27d40cfcc0 0x7f27d40cddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.138+0000 7f27d1e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27d40b2390 0x7f27d40cf780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:36204/0 (socket says 172.21.15.27:36204) 2024-01-15T18:05:57.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.138+0000 7f27d1e00640 1 -- 172.21.15.27:0/4056201383 learned_addr learned my addr 172.21.15.27:0/4056201383 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:57.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.138+0000 7f27d1e00640 1 -- 172.21.15.27:0/4056201383 >> v2:172.21.15.93:3300/0 conn(0x7f27d40cfcc0 msgr2=0x7f27d40cddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:57.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.138+0000 7f27d1e00640 1 --2- 172.21.15.27:0/4056201383 >> v2:172.21.15.93:3300/0 conn(0x7f27d40cfcc0 0x7f27d40cddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.138+0000 7f27d1e00640 1 -- 172.21.15.27:0/4056201383 >> v2:172.21.15.27:3300/0 conn(0x7f27d406bb80 msgr2=0x7f27d40cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:57.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.138+0000 7f27d1e00640 1 --2- 172.21.15.27:0/4056201383 >> v2:172.21.15.27:3300/0 conn(0x7f27d406bb80 0x7f27d40cd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.138+0000 7f27d1e00640 1 -- 172.21.15.27:0/4056201383 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d406c620 con 0x7f27d40b2390 2024-01-15T18:05:57.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.142+0000 7f27d1e00640 1 --2- 172.21.15.27:0/4056201383 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27d40b2390 0x7f27d40cf780 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f27d406d100 tx=0x7f27c401c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9c90d4815ec7a92 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:57.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.142+0000 7f27d0a00640 1 -- 172.21.15.27:0/4056201383 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c40aa5f0 con 0x7f27d40b2390 2024-01-15T18:05:57.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.142+0000 7f27d0a00640 1 -- 172.21.15.27:0/4056201383 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c40aa750 con 0x7f27d40b2390 2024-01-15T18:05:57.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.142+0000 7f27d0a00640 1 -- 172.21.15.27:0/4056201383 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c40aaae0 con 0x7f27d40b2390 2024-01-15T18:05:57.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.142+0000 7f27d9200640 1 -- 172.21.15.27:0/4056201383 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27d40b2390 msgr2=0x7f27d40cf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:57.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.142+0000 7f27d9200640 1 --2- 172.21.15.27:0/4056201383 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27d40b2390 0x7f27d40cf780 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f27d406d100 tx=0x7f27c401c4d0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.142+0000 7f27d9200640 1 -- 172.21.15.27:0/4056201383 shutdown_connections 2024-01-15T18:05:57.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.142+0000 7f27d9200640 1 --2- 172.21.15.27:0/4056201383 >> v2:172.21.15.93:3300/0 conn(0x7f27d40cfcc0 0x7f27d40cddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.142+0000 7f27d9200640 1 --2- 172.21.15.27:0/4056201383 >> v2:172.21.15.27:3300/0 conn(0x7f27d406bb80 0x7f27d40cd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.142+0000 7f27d9200640 1 --2- 172.21.15.27:0/4056201383 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27d40b2390 0x7f27d40cf780 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.142+0000 7f27d9200640 1 -- 172.21.15.27:0/4056201383 >> 172.21.15.27:0/4056201383 conn(0x7f27d40b2a10 msgr2=0x7f27d40b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:57.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.142+0000 7f27d9200640 1 -- 172.21.15.27:0/4056201383 shutdown_connections 2024-01-15T18:05:57.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.142+0000 7f27d9200640 1 -- 172.21.15.27:0/4056201383 wait complete. 2024-01-15T18:05:57.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.146+0000 7f27d9200640 1 Processor -- start 2024-01-15T18:05:57.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.146+0000 7f27d9200640 1 -- start start 2024-01-15T18:05:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.146+0000 7f27d9200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27d406bb80 0x7f27d40c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.146+0000 7f27d9200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27d40b2390 0x7f27d414b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.146+0000 7f27d9200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27d40cfcc0 0x7f27d40b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.146+0000 7f27d9200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d40c9380 con 0x7f27d406bb80 2024-01-15T18:05:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.146+0000 7f27d9200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d40c84d0 con 0x7f27d40b2390 2024-01-15T18:05:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.146+0000 7f27d9200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27d40c8770 con 0x7f27d40cfcc0 2024-01-15T18:05:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.146+0000 7f27d1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27d406bb80 0x7f27d40c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.146+0000 7f27d1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27d406bb80 0x7f27d40c7f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52778/0 (socket says 172.21.15.27:52778) 2024-01-15T18:05:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.146+0000 7f27d1e00640 1 -- 172.21.15.27:0/3340570895 learned_addr learned my addr 172.21.15.27:0/3340570895 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:57.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.146+0000 7f27d1400640 1 --2- 172.21.15.27:0/3340570895 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27d40b2390 0x7f27d414b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.146+0000 7f27d2800640 1 --2- 172.21.15.27:0/3340570895 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27d40cfcc0 0x7f27d40b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.146+0000 7f27d1e00640 1 -- 172.21.15.27:0/3340570895 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27d40cfcc0 msgr2=0x7f27d40b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:57.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.146+0000 7f27d1e00640 1 --2- 172.21.15.27:0/3340570895 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27d40cfcc0 0x7f27d40b3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.146+0000 7f27d1e00640 1 -- 172.21.15.27:0/3340570895 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27d40b2390 msgr2=0x7f27d414b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:57.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.146+0000 7f27d1e00640 1 --2- 172.21.15.27:0/3340570895 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27d40b2390 0x7f27d414b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.146+0000 7f27d1e00640 1 -- 172.21.15.27:0/3340570895 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c40094d0 con 0x7f27d406bb80 2024-01-15T18:05:57.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.146+0000 7f27d1400640 1 --2- 172.21.15.27:0/3340570895 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27d40b2390 0x7f27d414b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:57.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.146+0000 7f27d1e00640 1 --2- 172.21.15.27:0/3340570895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27d406bb80 0x7f27d40c7f90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f27c401ab80 tx=0x7f27c401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:57.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.146+0000 7f27ca000640 1 -- 172.21.15.27:0/3340570895 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c40bd5f0 con 0x7f27d406bb80 2024-01-15T18:05:57.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.146+0000 7f27ca000640 1 -- 172.21.15.27:0/3340570895 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c40bd770 con 0x7f27d406bb80 2024-01-15T18:05:57.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.146+0000 7f27ca000640 1 -- 172.21.15.27:0/3340570895 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c40bdb20 con 0x7f27d406bb80 2024-01-15T18:05:57.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.146+0000 7f27d9200640 1 -- 172.21.15.27:0/3340570895 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d40c8a50 con 0x7f27d406bb80 2024-01-15T18:05:57.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.146+0000 7f27d9200640 1 -- 172.21.15.27:0/3340570895 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d40c9010 con 0x7f27d406bb80 2024-01-15T18:05:57.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.150+0000 7f27d9200640 1 -- 172.21.15.27:0/3340570895 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27d40d07c0 con 0x7f27d406bb80 2024-01-15T18:05:57.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.150+0000 7f27ca000640 1 -- 172.21.15.27:0/3340570895 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f27c40b33f0 con 0x7f27d406bb80 2024-01-15T18:05:57.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.150+0000 7f27ca000640 1 --2- 172.21.15.27:0/3340570895 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f27ac066920 0x7f27ac068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.150+0000 7f27ca000640 1 -- 172.21.15.27:0/3340570895 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f27c40a7070 con 0x7f27d406bb80 2024-01-15T18:05:57.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.150+0000 7f27d1400640 1 --2- 172.21.15.27:0/3340570895 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f27ac066920 0x7f27ac068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.150+0000 7f27d1400640 1 --2- 172.21.15.27:0/3340570895 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f27ac066920 0x7f27ac068de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f27d41f98c0 tx=0x7f27bc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:57.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.154+0000 7f27ca000640 1 -- 172.21.15.27:0/3340570895 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f27c40ec150 con 0x7f27d406bb80 2024-01-15T18:05:57.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.310+0000 7f27d9200640 1 -- 172.21.15.27:0/3340570895 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27d404e850 con 0x7f27d406bb80 2024-01-15T18:05:57.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.314+0000 7f27ca000640 1 -- 172.21.15.27:0/3340570895 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+12761 (secure 0 0 0) 0x7f27c40ef770 con 0x7f27d406bb80 2024-01-15T18:05:57.319 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:57.319 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":30,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:05:54.110126+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:57.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.314+0000 7f27d9200640 1 -- 172.21.15.27:0/3340570895 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f27ac066920 msgr2=0x7f27ac068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:57.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.314+0000 7f27d9200640 1 --2- 172.21.15.27:0/3340570895 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f27ac066920 0x7f27ac068de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f27d41f98c0 tx=0x7f27bc008040 comp rx=0 tx=0).stop 2024-01-15T18:05:57.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.314+0000 7f27d9200640 1 -- 172.21.15.27:0/3340570895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27d406bb80 msgr2=0x7f27d40c7f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:57.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.314+0000 7f27d9200640 1 --2- 172.21.15.27:0/3340570895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27d406bb80 0x7f27d40c7f90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f27c401ab80 tx=0x7f27c401ad10 comp rx=0 tx=0).stop 2024-01-15T18:05:57.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.314+0000 7f27d9200640 1 -- 172.21.15.27:0/3340570895 shutdown_connections 2024-01-15T18:05:57.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.314+0000 7f27d9200640 1 --2- 172.21.15.27:0/3340570895 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f27ac066920 0x7f27ac068de0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.314+0000 7f27d9200640 1 --2- 172.21.15.27:0/3340570895 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27d40cfcc0 0x7f27d40b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.314+0000 7f27d9200640 1 --2- 172.21.15.27:0/3340570895 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27d40b2390 0x7f27d414b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.314+0000 7f27d9200640 1 --2- 172.21.15.27:0/3340570895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27d406bb80 0x7f27d40c7f90 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.314+0000 7f27d9200640 1 -- 172.21.15.27:0/3340570895 >> 172.21.15.27:0/3340570895 conn(0x7f27d40b2a10 msgr2=0x7f27d40b2df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:57.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.314+0000 7f27d9200640 1 -- 172.21.15.27:0/3340570895 shutdown_connections 2024-01-15T18:05:57.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.314+0000 7f27d9200640 1 -- 172.21.15.27:0/3340570895 wait complete. 2024-01-15T18:05:57.341 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:05:57.342 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:05:57.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.478+0000 7f61c0600640 1 Processor -- start 2024-01-15T18:05:57.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.478+0000 7f61c0600640 1 -- start start 2024-01-15T18:05:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61c0600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61b80cef20 0x7f61b80ce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61c0600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f61b80d19f0 0x7f61b80b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61c0600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b815b6b0 con 0x7f61b80cca40 2024-01-15T18:05:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61c0600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b80d1e50 con 0x7f61b80d19f0 2024-01-15T18:05:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61c0600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b80cb9e0 con 0x7f61b80cef20 2024-01-15T18:05:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61bd200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f61b80cca40 0x7f61b80cf300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51846/0 (socket says 172.21.15.27:51846) 2024-01-15T18:05:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61bd200640 1 -- 172.21.15.27:0/4057121726 learned_addr learned my addr 172.21.15.27:0/4057121726 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61be600640 1 --2- 172.21.15.27:0/4057121726 >> v2:172.21.15.93:3300/0 conn(0x7f61b80d19f0 0x7f61b80b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61b7e00640 1 -- 172.21.15.27:0/4057121726 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084661410 0 0) 0x7f61b815b6b0 con 0x7f61b80cca40 2024-01-15T18:05:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61bdc00640 1 --2- 172.21.15.27:0/4057121726 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61b80cef20 0x7f61b80ce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61b7e00640 1 -- 172.21.15.27:0/4057121726 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6198003530 con 0x7f61b80cca40 2024-01-15T18:05:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61b7e00640 1 -- 172.21.15.27:0/4057121726 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2838234117 0 0) 0x7f6198003530 con 0x7f61b80cca40 2024-01-15T18:05:57.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61b7e00640 1 -- 172.21.15.27:0/4057121726 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61b80cef20 msgr2=0x7f61b80ce2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:57.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61b7e00640 1 --2- 172.21.15.27:0/4057121726 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61b80cef20 0x7f61b80ce2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61b7e00640 1 -- 172.21.15.27:0/4057121726 >> v2:172.21.15.93:3300/0 conn(0x7f61b80d19f0 msgr2=0x7f61b80b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:57.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61b7e00640 1 --2- 172.21.15.27:0/4057121726 >> v2:172.21.15.93:3300/0 conn(0x7f61b80d19f0 0x7f61b80b3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61b7e00640 1 -- 172.21.15.27:0/4057121726 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b80cfdb0 con 0x7f61b80cca40 2024-01-15T18:05:57.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61b7e00640 1 -- 172.21.15.27:0/4057121726 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f61a0001680 con 0x7f61b80cca40 2024-01-15T18:05:57.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61be600640 1 --2- 172.21.15.27:0/4057121726 >> v2:172.21.15.93:3300/0 conn(0x7f61b80d19f0 0x7f61b80b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:57.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61b7e00640 1 -- 172.21.15.27:0/4057121726 >> v1:172.21.15.27:6789/0 conn(0x7f61b80cca40 legacy=0x7f61b80cf300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:57.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61b7e00640 1 --2- 172.21.15.27:0/4057121726 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61b80cef20 0x7f61980039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61b7e00640 1 --2- 172.21.15.27:0/4057121726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6198003f70 0x7f61980047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61b7e00640 1 --2- 172.21.15.27:0/4057121726 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6198004d10 0x7f6198005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61b7e00640 1 -- 172.21.15.27:0/4057121726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b80d1e50 con 0x7f6198003f70 2024-01-15T18:05:57.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61b7e00640 1 -- 172.21.15.27:0/4057121726 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b80cb9e0 con 0x7f6198004d10 2024-01-15T18:05:57.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61b7e00640 1 -- 172.21.15.27:0/4057121726 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f619800ba70 con 0x7f61b80cef20 2024-01-15T18:05:57.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61bd200640 1 --2- 172.21.15.27:0/4057121726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6198003f70 0x7f61980047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61bdc00640 1 --2- 172.21.15.27:0/4057121726 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61b80cef20 0x7f61980039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61be600640 1 --2- 172.21.15.27:0/4057121726 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6198004d10 0x7f6198005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61bd200640 1 -- 172.21.15.27:0/4057121726 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61b80cef20 msgr2=0x7f61980039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:57.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61bd200640 1 --2- 172.21.15.27:0/4057121726 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61b80cef20 0x7f61980039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61bd200640 1 -- 172.21.15.27:0/4057121726 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6198004d10 msgr2=0x7f6198005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:57.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61bd200640 1 --2- 172.21.15.27:0/4057121726 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6198004d10 0x7f6198005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.482+0000 7f61bd200640 1 -- 172.21.15.27:0/4057121726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6198003530 con 0x7f6198003f70 2024-01-15T18:05:57.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.486+0000 7f61bd200640 1 --2- 172.21.15.27:0/4057121726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6198003f70 0x7f61980047d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f61a000c290 tx=0x7f61a001f430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77f3d3df71e86064 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:57.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.486+0000 7f61b7e00640 1 -- 172.21.15.27:0/4057121726 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a0005ae0 con 0x7f6198003f70 2024-01-15T18:05:57.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.486+0000 7f61b7e00640 1 -- 172.21.15.27:0/4057121726 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a00ab070 con 0x7f6198003f70 2024-01-15T18:05:57.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.486+0000 7f61b7e00640 1 -- 172.21.15.27:0/4057121726 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a0005e80 con 0x7f6198003f70 2024-01-15T18:05:57.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.486+0000 7f61c0600640 1 -- 172.21.15.27:0/4057121726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6198003f70 msgr2=0x7f61980047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:57.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.486+0000 7f61c0600640 1 --2- 172.21.15.27:0/4057121726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6198003f70 0x7f61980047d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f61a000c290 tx=0x7f61a001f430 comp rx=0 tx=0).stop 2024-01-15T18:05:57.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.486+0000 7f61be600640 1 -- 172.21.15.27:0/4057121726 reap_dead start 2024-01-15T18:05:57.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.486+0000 7f61c0600640 1 -- 172.21.15.27:0/4057121726 shutdown_connections 2024-01-15T18:05:57.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.486+0000 7f61c0600640 1 -- 172.21.15.27:0/4057121726 >> 172.21.15.27:0/4057121726 conn(0x7f61b80d0ed0 msgr2=0x7f61b80d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:57.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.486+0000 7f61c0600640 1 -- 172.21.15.27:0/4057121726 shutdown_connections 2024-01-15T18:05:57.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.486+0000 7f61c0600640 1 -- 172.21.15.27:0/4057121726 wait complete. 2024-01-15T18:05:57.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.490+0000 7f61c0600640 1 Processor -- start 2024-01-15T18:05:57.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.490+0000 7f61c0600640 1 -- start start 2024-01-15T18:05:57.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.490+0000 7f61c0600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6198003f70 0x7f61b81f3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.490+0000 7f61c0600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f61b80c8e30 0x7f61b806f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.490+0000 7f61c0600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61b806fd10 0x7f61b806b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.490+0000 7f61c0600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b8070190 con 0x7f6198003f70 2024-01-15T18:05:57.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.490+0000 7f61c0600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b80ca020 con 0x7f61b80c8e30 2024-01-15T18:05:57.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.490+0000 7f61c0600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b80ca300 con 0x7f61b806fd10 2024-01-15T18:05:57.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.490+0000 7f61bdc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6198003f70 0x7f61b81f3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.490+0000 7f61bdc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6198003f70 0x7f61b81f3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52798/0 (socket says 172.21.15.27:52798) 2024-01-15T18:05:57.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.490+0000 7f61bdc00640 1 -- 172.21.15.27:0/4158125414 learned_addr learned my addr 172.21.15.27:0/4158125414 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:57.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.490+0000 7f61bd200640 1 --2- 172.21.15.27:0/4158125414 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f61b80c8e30 0x7f61b806f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.490+0000 7f61be600640 1 --2- 172.21.15.27:0/4158125414 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61b806fd10 0x7f61b806b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.490+0000 7f61bdc00640 1 -- 172.21.15.27:0/4158125414 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61b806fd10 msgr2=0x7f61b806b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:57.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.490+0000 7f61bdc00640 1 --2- 172.21.15.27:0/4158125414 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61b806fd10 0x7f61b806b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.490+0000 7f61bdc00640 1 -- 172.21.15.27:0/4158125414 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f61b80c8e30 msgr2=0x7f61b806f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:57.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.490+0000 7f61bdc00640 1 --2- 172.21.15.27:0/4158125414 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f61b80c8e30 0x7f61b806f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.490+0000 7f61bdc00640 1 -- 172.21.15.27:0/4158125414 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b80ca5e0 con 0x7f6198003f70 2024-01-15T18:05:57.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.490+0000 7f61bdc00640 1 --2- 172.21.15.27:0/4158125414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6198003f70 0x7f61b81f3380 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f61a80118e0 tx=0x7f61a8011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:57.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.490+0000 7f61b5600640 1 -- 172.21.15.27:0/4158125414 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a800faf0 con 0x7f6198003f70 2024-01-15T18:05:57.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.490+0000 7f61b5600640 1 -- 172.21.15.27:0/4158125414 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a800fc50 con 0x7f6198003f70 2024-01-15T18:05:57.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.490+0000 7f61c0600640 1 -- 172.21.15.27:0/4158125414 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b80ca8c0 con 0x7f6198003f70 2024-01-15T18:05:57.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.490+0000 7f61b5600640 1 -- 172.21.15.27:0/4158125414 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a8014440 con 0x7f6198003f70 2024-01-15T18:05:57.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.490+0000 7f61c0600640 1 -- 172.21.15.27:0/4158125414 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b80c92b0 con 0x7f6198003f70 2024-01-15T18:05:57.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.494+0000 7f61c0600640 1 -- 172.21.15.27:0/4158125414 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b80d09f0 con 0x7f6198003f70 2024-01-15T18:05:57.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.494+0000 7f61b5600640 1 -- 172.21.15.27:0/4158125414 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f61a80145a0 con 0x7f6198003f70 2024-01-15T18:05:57.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.494+0000 7f61b5600640 1 --2- 172.21.15.27:0/4158125414 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6190066920 0x7f6190068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.494+0000 7f61b5600640 1 -- 172.21.15.27:0/4158125414 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f61a80849b0 con 0x7f6198003f70 2024-01-15T18:05:57.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.494+0000 7f61bd200640 1 --2- 172.21.15.27:0/4158125414 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6190066920 0x7f6190068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.494+0000 7f61bd200640 1 --2- 172.21.15.27:0/4158125414 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6190066920 0x7f6190068de0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f61a00ae000 tx=0x7f61a0001a10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:57.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.498+0000 7f61b5600640 1 -- 172.21.15.27:0/4158125414 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f61a8051e20 con 0x7f6198003f70 2024-01-15T18:05:57.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.658+0000 7f61c0600640 1 -- 172.21.15.27:0/4158125414 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f61b804e850 con 0x7f6198003f70 2024-01-15T18:05:57.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.658+0000 7f61b5600640 1 -- 172.21.15.27:0/4158125414 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) v1 ==== 93+0+10 (secure 0 0 0) 0x7f61a8055440 con 0x7f6198003f70 2024-01-15T18:05:57.665 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:05:57.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.662+0000 7f61c0600640 1 -- 172.21.15.27:0/4158125414 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6190066920 msgr2=0x7f6190068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:57.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.662+0000 7f61c0600640 1 --2- 172.21.15.27:0/4158125414 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6190066920 0x7f6190068de0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f61a00ae000 tx=0x7f61a0001a10 comp rx=0 tx=0).stop 2024-01-15T18:05:57.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.662+0000 7f61c0600640 1 -- 172.21.15.27:0/4158125414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6198003f70 msgr2=0x7f61b81f3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:57.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.662+0000 7f61c0600640 1 --2- 172.21.15.27:0/4158125414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6198003f70 0x7f61b81f3380 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f61a80118e0 tx=0x7f61a8011da0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.662+0000 7f61c0600640 1 -- 172.21.15.27:0/4158125414 shutdown_connections 2024-01-15T18:05:57.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.662+0000 7f61c0600640 1 --2- 172.21.15.27:0/4158125414 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6190066920 0x7f6190068de0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.662+0000 7f61c0600640 1 --2- 172.21.15.27:0/4158125414 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61b806fd10 0x7f61b806b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.662+0000 7f61c0600640 1 --2- 172.21.15.27:0/4158125414 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f61b80c8e30 0x7f61b806f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.662+0000 7f61c0600640 1 --2- 172.21.15.27:0/4158125414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6198003f70 0x7f61b81f3380 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.662+0000 7f61c0600640 1 -- 172.21.15.27:0/4158125414 >> 172.21.15.27:0/4158125414 conn(0x7f61b80d0ed0 msgr2=0x7f61b815a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:57.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.662+0000 7f61c0600640 1 -- 172.21.15.27:0/4158125414 shutdown_connections 2024-01-15T18:05:57.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.662+0000 7f61c0600640 1 -- 172.21.15.27:0/4158125414 wait complete. 2024-01-15T18:05:57.687 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-15T18:05:57.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.822+0000 7fb058800640 1 Processor -- start 2024-01-15T18:05:57.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.822+0000 7fb058800640 1 -- start start 2024-01-15T18:05:57.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.822+0000 7fb058800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fb0500c92b0 0x7fb05015d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.822+0000 7fb058800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb05006c4a0 con 0x7fb05015c130 2024-01-15T18:05:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb058800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb05015c660 con 0x7fb0500701a0 2024-01-15T18:05:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb058800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb05015c940 con 0x7fb0500c92b0 2024-01-15T18:05:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb055400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb05015c130 0x7fb05006dd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51848/0 (socket says 172.21.15.27:51848) 2024-01-15T18:05:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb055400640 1 -- 172.21.15.27:0/2565735588 learned_addr learned my addr 172.21.15.27:0/2565735588 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:05:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb056800640 1 --2- 172.21.15.27:0/2565735588 >> v2:172.21.15.93:3300/0 conn(0x7fb0500c92b0 0x7fb05015d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb054a00640 1 -- 172.21.15.27:0/2565735588 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755281337 0 0) 0x7fb05006c4a0 con 0x7fb05015c130 2024-01-15T18:05:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb054a00640 1 -- 172.21.15.27:0/2565735588 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb030003530 con 0x7fb05015c130 2024-01-15T18:05:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb054a00640 1 -- 172.21.15.27:0/2565735588 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 683036088 0 0) 0x7fb030003530 con 0x7fb05015c130 2024-01-15T18:05:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb054a00640 1 -- 172.21.15.27:0/2565735588 >> v2:172.21.15.93:3300/0 conn(0x7fb0500c92b0 msgr2=0x7fb05015d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb054a00640 1 --2- 172.21.15.27:0/2565735588 >> v2:172.21.15.93:3300/0 conn(0x7fb0500c92b0 0x7fb05015d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb054a00640 1 -- 172.21.15.27:0/2565735588 >> v1:172.21.15.93:6789/0 conn(0x7fb0500701a0 legacy=0x7fb050070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb054a00640 1 -- 172.21.15.27:0/2565735588 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05015cc20 con 0x7fb05015c130 2024-01-15T18:05:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb054a00640 1 -- 172.21.15.27:0/2565735588 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fb038001680 con 0x7fb05015c130 2024-01-15T18:05:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb054a00640 1 -- 172.21.15.27:0/2565735588 >> v1:172.21.15.27:6789/0 conn(0x7fb05015c130 legacy=0x7fb05006dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb054a00640 1 --2- 172.21.15.27:0/2565735588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0300039f0 0x7fb030003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb054a00640 1 --2- 172.21.15.27:0/2565735588 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0300047d0 0x7fb030004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb054a00640 1 --2- 172.21.15.27:0/2565735588 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb030005200 0x7fb03000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb054a00640 1 -- 172.21.15.27:0/2565735588 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05015c940 con 0x7fb0300039f0 2024-01-15T18:05:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb054a00640 1 -- 172.21.15.27:0/2565735588 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03000c160 con 0x7fb030005200 2024-01-15T18:05:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb054a00640 1 -- 172.21.15.27:0/2565735588 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb03000c320 con 0x7fb0300047d0 2024-01-15T18:05:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb055e00640 1 --2- 172.21.15.27:0/2565735588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0300039f0 0x7fb030003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb055400640 1 --2- 172.21.15.27:0/2565735588 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0300047d0 0x7fb030004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb056800640 1 --2- 172.21.15.27:0/2565735588 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb030005200 0x7fb03000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb055e00640 1 -- 172.21.15.27:0/2565735588 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0300047d0 msgr2=0x7fb030004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:57.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb055e00640 1 --2- 172.21.15.27:0/2565735588 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0300047d0 0x7fb030004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb055e00640 1 -- 172.21.15.27:0/2565735588 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb030005200 msgr2=0x7fb03000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:57.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb055e00640 1 --2- 172.21.15.27:0/2565735588 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb030005200 0x7fb03000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb055e00640 1 -- 172.21.15.27:0/2565735588 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb030003530 con 0x7fb0300039f0 2024-01-15T18:05:57.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb056800640 1 --2- 172.21.15.27:0/2565735588 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb030005200 0x7fb03000b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:57.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.826+0000 7fb056800640 1 -- 172.21.15.27:0/2565735588 reap_dead start 2024-01-15T18:05:57.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.830+0000 7fb055e00640 1 --2- 172.21.15.27:0/2565735588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0300039f0 0x7fb030003e10 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb040001780 tx=0x7fb04001c2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba833982905f33ba server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.830+0000 7fb054a00640 1 -- 172.21.15.27:0/2565735588 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0400052d0 con 0x7fb0300039f0 2024-01-15T18:05:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.830+0000 7fb054a00640 1 -- 172.21.15.27:0/2565735588 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0400a7050 con 0x7fb0300039f0 2024-01-15T18:05:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.830+0000 7fb054a00640 1 -- 172.21.15.27:0/2565735588 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb040003600 con 0x7fb0300039f0 2024-01-15T18:05:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.830+0000 7fb058800640 1 -- 172.21.15.27:0/2565735588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0300039f0 msgr2=0x7fb030003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.830+0000 7fb058800640 1 --2- 172.21.15.27:0/2565735588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0300039f0 0x7fb030003e10 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb040001780 tx=0x7fb04001c2f0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.830+0000 7fb058800640 1 -- 172.21.15.27:0/2565735588 shutdown_connections 2024-01-15T18:05:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.830+0000 7fb058800640 1 --2- 172.21.15.27:0/2565735588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0300039f0 0x7fb030003e10 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.830+0000 7fb058800640 1 -- 172.21.15.27:0/2565735588 >> 172.21.15.27:0/2565735588 conn(0x7fb0500cec60 msgr2=0x7fb0500cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.830+0000 7fb058800640 1 -- 172.21.15.27:0/2565735588 shutdown_connections 2024-01-15T18:05:57.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.830+0000 7fb058800640 1 -- 172.21.15.27:0/2565735588 wait complete. 2024-01-15T18:05:57.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb058800640 1 Processor -- start 2024-01-15T18:05:57.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb058800640 1 -- start start 2024-01-15T18:05:57.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb058800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0300039f0 0x7fb050150200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb058800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0500c8410 0x7fb0500c7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb058800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0500c7d70 0x7fb0500ac920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb058800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0500cd240 con 0x7fb0300039f0 2024-01-15T18:05:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb058800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0500c8200 con 0x7fb0500c7d70 2024-01-15T18:05:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb058800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb050150740 con 0x7fb0500c8410 2024-01-15T18:05:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb055e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0300039f0 0x7fb050150200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb055e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0300039f0 0x7fb050150200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52822/0 (socket says 172.21.15.27:52822) 2024-01-15T18:05:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb055e00640 1 -- 172.21.15.27:0/272284583 learned_addr learned my addr 172.21.15.27:0/272284583 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb055400640 1 --2- 172.21.15.27:0/272284583 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0500c8410 0x7fb0500c7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb056800640 1 --2- 172.21.15.27:0/272284583 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0500c7d70 0x7fb0500ac920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb055e00640 1 -- 172.21.15.27:0/272284583 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0500c8410 msgr2=0x7fb0500c7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb055e00640 1 --2- 172.21.15.27:0/272284583 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0500c8410 0x7fb0500c7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb055e00640 1 -- 172.21.15.27:0/272284583 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0500c7d70 msgr2=0x7fb0500ac920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:57.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb055e00640 1 --2- 172.21.15.27:0/272284583 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0500c7d70 0x7fb0500ac920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb055e00640 1 -- 172.21.15.27:0/272284583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0500cd420 con 0x7fb0300039f0 2024-01-15T18:05:57.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb055400640 1 --2- 172.21.15.27:0/272284583 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0500c8410 0x7fb0500c7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:57.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb056800640 1 --2- 172.21.15.27:0/272284583 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0500c7d70 0x7fb0500ac920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:57.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb055e00640 1 --2- 172.21.15.27:0/272284583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0300039f0 0x7fb050150200 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb040005040 tx=0x7fb030005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:57.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb04e000640 1 -- 172.21.15.27:0/272284583 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb040007390 con 0x7fb0300039f0 2024-01-15T18:05:57.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb058800640 1 -- 172.21.15.27:0/272284583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0500cd700 con 0x7fb0300039f0 2024-01-15T18:05:57.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb04e000640 1 -- 172.21.15.27:0/272284583 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0400074f0 con 0x7fb0300039f0 2024-01-15T18:05:57.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb04e000640 1 -- 172.21.15.27:0/272284583 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0400069e0 con 0x7fb0300039f0 2024-01-15T18:05:57.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb058800640 1 -- 172.21.15.27:0/272284583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0500cdcc0 con 0x7fb0300039f0 2024-01-15T18:05:57.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.834+0000 7fb058800640 1 -- 172.21.15.27:0/272284583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb05004e850 con 0x7fb0300039f0 2024-01-15T18:05:57.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.838+0000 7fb04e000640 1 -- 172.21.15.27:0/272284583 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fb040006b40 con 0x7fb0300039f0 2024-01-15T18:05:57.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.838+0000 7fb04e000640 1 --2- 172.21.15.27:0/272284583 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb0280668f0 0x7fb028068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.838+0000 7fb04e000640 1 -- 172.21.15.27:0/272284583 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb04011e720 con 0x7fb0300039f0 2024-01-15T18:05:57.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.838+0000 7fb055400640 1 --2- 172.21.15.27:0/272284583 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb0280668f0 0x7fb028068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.838+0000 7fb055400640 1 --2- 172.21.15.27:0/272284583 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb0280668f0 0x7fb028068db0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb0500ce9e0 tx=0x7fb038001440 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.842+0000 7fb04e000640 1 -- 172.21.15.27:0/272284583 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb0400ef6b0 con 0x7fb0300039f0 2024-01-15T18:05:58.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.998+0000 7fb058800640 1 -- 172.21.15.27:0/272284583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb0500af720 con 0x7fb0300039f0 2024-01-15T18:05:58.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:57.998+0000 7fb04e000640 1 -- 172.21.15.27:0/272284583 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v30) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb0400ef240 con 0x7fb0300039f0 2024-01-15T18:05:58.006 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:05:58.006 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:05:58.007 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:05:58.007 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-01-15T18:05:58.007 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-01-15T18:05:58.007 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:05:58.007 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:05:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.002+0000 7fb058800640 1 -- 172.21.15.27:0/272284583 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb0280668f0 msgr2=0x7fb028068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.002+0000 7fb058800640 1 --2- 172.21.15.27:0/272284583 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb0280668f0 0x7fb028068db0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb0500ce9e0 tx=0x7fb038001440 comp rx=0 tx=0).stop 2024-01-15T18:05:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.002+0000 7fb058800640 1 -- 172.21.15.27:0/272284583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0300039f0 msgr2=0x7fb050150200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.002+0000 7fb058800640 1 --2- 172.21.15.27:0/272284583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0300039f0 0x7fb050150200 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb040005040 tx=0x7fb030005800 comp rx=0 tx=0).stop 2024-01-15T18:05:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.002+0000 7fb058800640 1 -- 172.21.15.27:0/272284583 shutdown_connections 2024-01-15T18:05:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.002+0000 7fb058800640 1 --2- 172.21.15.27:0/272284583 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb0280668f0 0x7fb028068db0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.002+0000 7fb058800640 1 --2- 172.21.15.27:0/272284583 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0500c7d70 0x7fb0500ac920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.002+0000 7fb058800640 1 --2- 172.21.15.27:0/272284583 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0500c8410 0x7fb0500c7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:58.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.002+0000 7fb058800640 1 --2- 172.21.15.27:0/272284583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0300039f0 0x7fb050150200 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.002+0000 7fb058800640 1 -- 172.21.15.27:0/272284583 >> 172.21.15.27:0/272284583 conn(0x7fb0500cec60 msgr2=0x7fb05015d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.002+0000 7fb058800640 1 -- 172.21.15.27:0/272284583 shutdown_connections 2024-01-15T18:05:58.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.002+0000 7fb058800640 1 -- 172.21.15.27:0/272284583 wait complete. 2024-01-15T18:05:58.028 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-15T18:05:58.028 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-15T18:05:58.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.166+0000 7f75e8000640 1 Processor -- start 2024-01-15T18:05:58.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.166+0000 7f75e8000640 1 -- start start 2024-01-15T18:05:58.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.166+0000 7f75e8000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f75e0158900 0x7f75e015f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:58.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.166+0000 7f75e8000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f75e015bf00 0x7f75e00cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:58.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.166+0000 7f75e8000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f75e00cf110 0x7f75e00701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:58.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.166+0000 7f75e8000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75e00cf580 con 0x7f75e015bf00 2024-01-15T18:05:58.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.166+0000 7f75e8000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75e00b1cb0 con 0x7f75e0158900 2024-01-15T18:05:58.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.166+0000 7f75e8000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75e00b1f00 con 0x7f75e00cf110 2024-01-15T18:05:58.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.166+0000 7f75e4c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f75e015bf00 0x7f75e00cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:58.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.166+0000 7f75e4c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f75e015bf00 0x7f75e00cebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52832/0 (socket says 172.21.15.27:52832) 2024-01-15T18:05:58.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.166+0000 7f75e4c00640 1 -- 172.21.15.27:0/2242892960 learned_addr learned my addr 172.21.15.27:0/2242892960 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:58.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.170+0000 7f75e6000640 1 --2- 172.21.15.27:0/2242892960 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f75e00cf110 0x7f75e00701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:58.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.170+0000 7f75e5600640 1 --2- 172.21.15.27:0/2242892960 >> v2:172.21.15.93:3300/0 conn(0x7f75e0158900 0x7f75e015f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:58.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.170+0000 7f75e4c00640 1 -- 172.21.15.27:0/2242892960 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f75e00cf110 msgr2=0x7f75e00701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:58.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.170+0000 7f75e4c00640 1 --2- 172.21.15.27:0/2242892960 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f75e00cf110 0x7f75e00701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:58.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.170+0000 7f75e4c00640 1 -- 172.21.15.27:0/2242892960 >> v2:172.21.15.93:3300/0 conn(0x7f75e0158900 msgr2=0x7f75e015f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:58.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.170+0000 7f75e4c00640 1 --2- 172.21.15.27:0/2242892960 >> v2:172.21.15.93:3300/0 conn(0x7f75e0158900 0x7f75e015f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:58.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.170+0000 7f75e4c00640 1 -- 172.21.15.27:0/2242892960 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e00b21e0 con 0x7f75e015bf00 2024-01-15T18:05:58.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.170+0000 7f75e6000640 1 --2- 172.21.15.27:0/2242892960 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f75e00cf110 0x7f75e00701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:58.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.170+0000 7f75e5600640 1 --2- 172.21.15.27:0/2242892960 >> v2:172.21.15.93:3300/0 conn(0x7f75e0158900 0x7f75e015f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.170+0000 7f75e4c00640 1 --2- 172.21.15.27:0/2242892960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75e015bf00 0x7f75e00cebd0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f75e00ceaa0 tx=0x7f75c801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b829911901468652 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.170+0000 7f75dfe00640 1 -- 172.21.15.27:0/2242892960 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e0070f30 con 0x7f75e015bf00 2024-01-15T18:05:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.170+0000 7f75dfe00640 1 -- 172.21.15.27:0/2242892960 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c80ac410 con 0x7f75e015bf00 2024-01-15T18:05:58.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.170+0000 7f75dfe00640 1 -- 172.21.15.27:0/2242892960 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c80ac7a0 con 0x7f75e015bf00 2024-01-15T18:05:58.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.174+0000 7f75e8000640 1 -- 172.21.15.27:0/2242892960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75e015bf00 msgr2=0x7f75e00cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:58.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.174+0000 7f75e8000640 1 --2- 172.21.15.27:0/2242892960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75e015bf00 0x7f75e00cebd0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f75e00ceaa0 tx=0x7f75c801c500 comp rx=0 tx=0).stop 2024-01-15T18:05:58.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.174+0000 7f75e8000640 1 -- 172.21.15.27:0/2242892960 shutdown_connections 2024-01-15T18:05:58.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.174+0000 7f75e8000640 1 --2- 172.21.15.27:0/2242892960 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f75e00cf110 0x7f75e00701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:58.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.174+0000 7f75e8000640 1 --2- 172.21.15.27:0/2242892960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75e015bf00 0x7f75e00cebd0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:58.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.174+0000 7f75e8000640 1 --2- 172.21.15.27:0/2242892960 >> v2:172.21.15.93:3300/0 conn(0x7f75e0158900 0x7f75e015f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:58.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.174+0000 7f75e8000640 1 -- 172.21.15.27:0/2242892960 >> 172.21.15.27:0/2242892960 conn(0x7f75e00cb700 msgr2=0x7f75e00cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:58.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.174+0000 7f75e8000640 1 -- 172.21.15.27:0/2242892960 shutdown_connections 2024-01-15T18:05:58.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.174+0000 7f75e8000640 1 -- 172.21.15.27:0/2242892960 wait complete. 2024-01-15T18:05:58.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.174+0000 7f75e8000640 1 Processor -- start 2024-01-15T18:05:58.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.174+0000 7f75e8000640 1 -- start start 2024-01-15T18:05:58.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.174+0000 7f75e8000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f75e00cf110 0x7f75e014c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:58.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.174+0000 7f75e8000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75e0158900 0x7f75e014ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:58.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.174+0000 7f75e8000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f75e015bf00 0x7f75e014bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:58.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.174+0000 7f75e8000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e0150650 con 0x7f75e0158900 2024-01-15T18:05:58.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.174+0000 7f75e8000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e0150930 con 0x7f75e015bf00 2024-01-15T18:05:58.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.174+0000 7f75e8000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75e0150c10 con 0x7f75e00cf110 2024-01-15T18:05:58.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.174+0000 7f75e4c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75e0158900 0x7f75e014ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:58.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.174+0000 7f75e4c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75e0158900 0x7f75e014ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52842/0 (socket says 172.21.15.27:52842) 2024-01-15T18:05:58.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.174+0000 7f75e4c00640 1 -- 172.21.15.27:0/811991976 learned_addr learned my addr 172.21.15.27:0/811991976 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:58.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.178+0000 7f75e5600640 1 --2- 172.21.15.27:0/811991976 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f75e00cf110 0x7f75e014c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:58.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.178+0000 7f75e6000640 1 --2- 172.21.15.27:0/811991976 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f75e015bf00 0x7f75e014bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:58.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.178+0000 7f75e4c00640 1 -- 172.21.15.27:0/811991976 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f75e00cf110 msgr2=0x7f75e014c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:58.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.178+0000 7f75e4c00640 1 --2- 172.21.15.27:0/811991976 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f75e00cf110 0x7f75e014c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:58.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.178+0000 7f75e4c00640 1 -- 172.21.15.27:0/811991976 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f75e015bf00 msgr2=0x7f75e014bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:58.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.178+0000 7f75e4c00640 1 --2- 172.21.15.27:0/811991976 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f75e015bf00 0x7f75e014bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:58.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.178+0000 7f75e4c00640 1 -- 172.21.15.27:0/811991976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c80094d0 con 0x7f75e0158900 2024-01-15T18:05:58.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.178+0000 7f75e5600640 1 --2- 172.21.15.27:0/811991976 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f75e00cf110 0x7f75e014c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:58.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.178+0000 7f75e6000640 1 --2- 172.21.15.27:0/811991976 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f75e015bf00 0x7f75e014bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:58.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.178+0000 7f75e4c00640 1 --2- 172.21.15.27:0/811991976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75e0158900 0x7f75e014ff70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f75c801abb0 tx=0x7f75c80192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:58.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.178+0000 7f75dd600640 1 -- 172.21.15.27:0/811991976 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c80ace20 con 0x7f75e0158900 2024-01-15T18:05:58.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.178+0000 7f75dd600640 1 -- 172.21.15.27:0/811991976 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c80b65f0 con 0x7f75e0158900 2024-01-15T18:05:58.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.178+0000 7f75dd600640 1 -- 172.21.15.27:0/811991976 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c80b69a0 con 0x7f75e0158900 2024-01-15T18:05:58.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.178+0000 7f75e8000640 1 -- 172.21.15.27:0/811991976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e0150ef0 con 0x7f75e0158900 2024-01-15T18:05:58.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.178+0000 7f75e8000640 1 -- 172.21.15.27:0/811991976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e014c080 con 0x7f75e0158900 2024-01-15T18:05:58.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.178+0000 7f75e8000640 1 -- 172.21.15.27:0/811991976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75e004e850 con 0x7f75e0158900 2024-01-15T18:05:58.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.178+0000 7f75dd600640 1 -- 172.21.15.27:0/811991976 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f75c80b6b00 con 0x7f75e0158900 2024-01-15T18:05:58.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.178+0000 7f75dd600640 1 --2- 172.21.15.27:0/811991976 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f75b80668f0 0x7f75b8068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:58.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.178+0000 7f75dd600640 1 -- 172.21.15.27:0/811991976 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f75c811ed00 con 0x7f75e0158900 2024-01-15T18:05:58.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.182+0000 7f75e5600640 1 --2- 172.21.15.27:0/811991976 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f75b80668f0 0x7f75b8068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:58.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.182+0000 7f75e5600640 1 --2- 172.21.15.27:0/811991976 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f75b80668f0 0x7f75b8068db0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f75e006cf60 tx=0x7f75d0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:58.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.182+0000 7f75dd600640 1 -- 172.21.15.27:0/811991976 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f75c80ec3a0 con 0x7f75e0158900 2024-01-15T18:05:58.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:58.342+0000 7f75e8000640 1 -- 172.21.15.27:0/811991976 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f75e01513a0 con 0x7f75e0158900 2024-01-15T18:05:59.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.146+0000 7f75dd600640 1 -- 172.21.15.27:0/811991976 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v31) v1 ==== 138+0+0 (secure 0 0 0) 0x7f75c80b6db0 con 0x7f75e0158900 2024-01-15T18:05:59.154 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-01-15T18:05:59.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.150+0000 7f75e8000640 1 -- 172.21.15.27:0/811991976 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f75b80668f0 msgr2=0x7f75b8068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:59.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.150+0000 7f75e8000640 1 --2- 172.21.15.27:0/811991976 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f75b80668f0 0x7f75b8068db0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f75e006cf60 tx=0x7f75d0008040 comp rx=0 tx=0).stop 2024-01-15T18:05:59.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.150+0000 7f75e8000640 1 -- 172.21.15.27:0/811991976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75e0158900 msgr2=0x7f75e014ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:59.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.150+0000 7f75e8000640 1 --2- 172.21.15.27:0/811991976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75e0158900 0x7f75e014ff70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f75c801abb0 tx=0x7f75c80192a0 comp rx=0 tx=0).stop 2024-01-15T18:05:59.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.150+0000 7f75e8000640 1 -- 172.21.15.27:0/811991976 shutdown_connections 2024-01-15T18:05:59.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.150+0000 7f75e8000640 1 --2- 172.21.15.27:0/811991976 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f75b80668f0 0x7f75b8068db0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:59.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.150+0000 7f75e8000640 1 --2- 172.21.15.27:0/811991976 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f75e015bf00 0x7f75e014bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:59.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.150+0000 7f75e8000640 1 --2- 172.21.15.27:0/811991976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75e0158900 0x7f75e014ff70 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:59.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.150+0000 7f75e8000640 1 --2- 172.21.15.27:0/811991976 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f75e00cf110 0x7f75e014c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:59.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.150+0000 7f75e8000640 1 -- 172.21.15.27:0/811991976 >> 172.21.15.27:0/811991976 conn(0x7f75e00cb700 msgr2=0x7f75e0073480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:59.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.150+0000 7f75e8000640 1 -- 172.21.15.27:0/811991976 shutdown_connections 2024-01-15T18:05:59.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.150+0000 7f75e8000640 1 -- 172.21.15.27:0/811991976 wait complete. 2024-01-15T18:05:59.186 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-15T18:05:59.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.330+0000 7ff441400640 1 Processor -- start 2024-01-15T18:05:59.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.330+0000 7ff441400640 1 -- start start 2024-01-15T18:05:59.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.334+0000 7ff441400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff43c158900 0x7ff43c15f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:59.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.334+0000 7ff441400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff43c0cf2e0 0x7ff43c0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:59.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.334+0000 7ff441400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43c0b1cb0 con 0x7ff43c15bf00 2024-01-15T18:05:59.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.334+0000 7ff441400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff43c0b1f00 con 0x7ff43c0cf2e0 2024-01-15T18:05:59.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.334+0000 7ff441400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff43c0b21e0 con 0x7ff43c158900 2024-01-15T18:05:59.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.334+0000 7ff43b200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff43c0cf2e0 0x7ff43c0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:59.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.334+0000 7ff43b200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff43c0cf2e0 0x7ff43c0701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52850/0 (socket says 172.21.15.27:52850) 2024-01-15T18:05:59.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.334+0000 7ff43b200640 1 -- 172.21.15.27:0/1422259595 learned_addr learned my addr 172.21.15.27:0/1422259595 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:59.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.334+0000 7ff43a800640 1 --2- 172.21.15.27:0/1422259595 >> v2:172.21.15.93:3300/0 conn(0x7ff43c158900 0x7ff43c15f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:59.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.334+0000 7ff43b200640 1 -- 172.21.15.27:0/1422259595 >> v2:172.21.15.93:3300/0 conn(0x7ff43c158900 msgr2=0x7ff43c15f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:59.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.334+0000 7ff43b200640 1 --2- 172.21.15.27:0/1422259595 >> v2:172.21.15.93:3300/0 conn(0x7ff43c158900 0x7ff43c15f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:59.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.334+0000 7ff43b200640 1 -- 172.21.15.27:0/1422259595 >> v1:172.21.15.93:6789/0 conn(0x7ff43c15bf00 legacy=0x7ff43c0cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:59.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.334+0000 7ff43b200640 1 -- 172.21.15.27:0/1422259595 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43c15ea70 con 0x7ff43c0cf2e0 2024-01-15T18:05:59.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.338+0000 7ff43b200640 1 --2- 172.21.15.27:0/1422259595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff43c0cf2e0 0x7ff43c0701e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0ceaa0 tx=0x7ff43401c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=275f24279101559a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:59.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.338+0000 7ff439400640 1 -- 172.21.15.27:0/1422259595 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43c070f30 con 0x7ff43c0cf2e0 2024-01-15T18:05:59.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.338+0000 7ff439400640 1 -- 172.21.15.27:0/1422259595 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4340a5780 con 0x7ff43c0cf2e0 2024-01-15T18:05:59.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.338+0000 7ff439400640 1 -- 172.21.15.27:0/1422259595 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4340a5b10 con 0x7ff43c0cf2e0 2024-01-15T18:05:59.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.338+0000 7ff441400640 1 -- 172.21.15.27:0/1422259595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff43c0cf2e0 msgr2=0x7ff43c0701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:59.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.338+0000 7ff441400640 1 --2- 172.21.15.27:0/1422259595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff43c0cf2e0 0x7ff43c0701e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0ceaa0 tx=0x7ff43401c9d0 comp rx=0 tx=0).stop 2024-01-15T18:05:59.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.338+0000 7ff441400640 1 -- 172.21.15.27:0/1422259595 shutdown_connections 2024-01-15T18:05:59.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.338+0000 7ff441400640 1 --2- 172.21.15.27:0/1422259595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff43c0cf2e0 0x7ff43c0701e0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:59.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.338+0000 7ff441400640 1 --2- 172.21.15.27:0/1422259595 >> v2:172.21.15.93:3300/0 conn(0x7ff43c158900 0x7ff43c15f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:59.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.338+0000 7ff441400640 1 -- 172.21.15.27:0/1422259595 >> 172.21.15.27:0/1422259595 conn(0x7ff43c0cb700 msgr2=0x7ff43c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:59.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.338+0000 7ff441400640 1 -- 172.21.15.27:0/1422259595 shutdown_connections 2024-01-15T18:05:59.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.338+0000 7ff441400640 1 -- 172.21.15.27:0/1422259595 wait complete. 2024-01-15T18:05:59.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.338+0000 7ff441400640 1 Processor -- start 2024-01-15T18:05:59.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.342+0000 7ff441400640 1 -- start start 2024-01-15T18:05:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.342+0000 7ff441400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff43c0cf2e0 0x7ff43c1526b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.342+0000 7ff441400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff43c158900 0x7ff43c14e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.342+0000 7ff441400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff43c15bf00 0x7ff43c14e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.342+0000 7ff441400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c1562f0 con 0x7ff43c0cf2e0 2024-01-15T18:05:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.342+0000 7ff441400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c152bf0 con 0x7ff43c15bf00 2024-01-15T18:05:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.342+0000 7ff441400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c1531f0 con 0x7ff43c158900 2024-01-15T18:05:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.342+0000 7ff43a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff43c0cf2e0 0x7ff43c1526b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:59.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.342+0000 7ff43a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff43c0cf2e0 0x7ff43c1526b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52866/0 (socket says 172.21.15.27:52866) 2024-01-15T18:05:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.342+0000 7ff43a800640 1 -- 172.21.15.27:0/506194581 learned_addr learned my addr 172.21.15.27:0/506194581 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:05:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.342+0000 7ff43b200640 1 --2- 172.21.15.27:0/506194581 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff43c15bf00 0x7ff43c14e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.342+0000 7ff439e00640 1 --2- 172.21.15.27:0/506194581 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff43c158900 0x7ff43c14e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.342+0000 7ff43a800640 1 -- 172.21.15.27:0/506194581 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff43c158900 msgr2=0x7ff43c14e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.342+0000 7ff43a800640 1 --2- 172.21.15.27:0/506194581 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff43c158900 0x7ff43c14e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.342+0000 7ff43a800640 1 -- 172.21.15.27:0/506194581 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff43c15bf00 msgr2=0x7ff43c14e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.342+0000 7ff43a800640 1 --2- 172.21.15.27:0/506194581 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff43c15bf00 0x7ff43c14e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.342+0000 7ff43a800640 1 -- 172.21.15.27:0/506194581 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff434009b30 con 0x7ff43c0cf2e0 2024-01-15T18:05:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.342+0000 7ff43a800640 1 --2- 172.21.15.27:0/506194581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff43c0cf2e0 0x7ff43c1526b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff42c00d800 tx=0x7ff42c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.342+0000 7ff432a00640 1 -- 172.21.15.27:0/506194581 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42c00b6c0 con 0x7ff43c0cf2e0 2024-01-15T18:05:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.342+0000 7ff432a00640 1 -- 172.21.15.27:0/506194581 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42c00b820 con 0x7ff43c0cf2e0 2024-01-15T18:05:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.342+0000 7ff441400640 1 -- 172.21.15.27:0/506194581 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff43c1534d0 con 0x7ff43c0cf2e0 2024-01-15T18:05:59.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.342+0000 7ff432a00640 1 -- 172.21.15.27:0/506194581 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42c00bb90 con 0x7ff43c0cf2e0 2024-01-15T18:05:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.342+0000 7ff441400640 1 -- 172.21.15.27:0/506194581 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff43c155f10 con 0x7ff43c0cf2e0 2024-01-15T18:05:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.342+0000 7ff441400640 1 -- 172.21.15.27:0/506194581 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff43c1560a0 con 0x7ff43c0cf2e0 2024-01-15T18:05:59.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.346+0000 7ff432a00640 1 -- 172.21.15.27:0/506194581 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7ff42c00bcf0 con 0x7ff43c0cf2e0 2024-01-15T18:05:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.346+0000 7ff432a00640 1 --2- 172.21.15.27:0/506194581 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff4140668f0 0x7ff414068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.346+0000 7ff432a00640 1 -- 172.21.15.27:0/506194581 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5227+0+0 (secure 0 0 0) 0x7ff42c084660 con 0x7ff43c0cf2e0 2024-01-15T18:05:59.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.346+0000 7ff439e00640 1 --2- 172.21.15.27:0/506194581 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff4140668f0 0x7ff414068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:59.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.346+0000 7ff439e00640 1 --2- 172.21.15.27:0/506194581 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff4140668f0 0x7ff414068db0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff43c152fe0 tx=0x7ff424009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:59.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.350+0000 7ff432a00640 1 -- 172.21.15.27:0/506194581 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff42c051b20 con 0x7ff43c0cf2e0 2024-01-15T18:05:59.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:05:59.506+0000 7ff441400640 1 -- 172.21.15.27:0/506194581 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff43c04e850 con 0x7ff43c0cf2e0 2024-01-15T18:06:00.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.154+0000 7ff432a00640 1 -- 172.21.15.27:0/506194581 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v32) v1 ==== 170+0+0 (secure 0 0 0) 0x7ff42c055140 con 0x7ff43c0cf2e0 2024-01-15T18:06:00.161 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-01-15T18:06:00.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.158+0000 7ff441400640 1 -- 172.21.15.27:0/506194581 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff4140668f0 msgr2=0x7ff414068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:00.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.158+0000 7ff441400640 1 --2- 172.21.15.27:0/506194581 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff4140668f0 0x7ff414068db0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff43c152fe0 tx=0x7ff424009040 comp rx=0 tx=0).stop 2024-01-15T18:06:00.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.158+0000 7ff441400640 1 -- 172.21.15.27:0/506194581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff43c0cf2e0 msgr2=0x7ff43c1526b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:00.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.158+0000 7ff441400640 1 --2- 172.21.15.27:0/506194581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff43c0cf2e0 0x7ff43c1526b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff42c00d800 tx=0x7ff42c00dcc0 comp rx=0 tx=0).stop 2024-01-15T18:06:00.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.158+0000 7ff441400640 1 -- 172.21.15.27:0/506194581 shutdown_connections 2024-01-15T18:06:00.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.158+0000 7ff441400640 1 --2- 172.21.15.27:0/506194581 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff4140668f0 0x7ff414068db0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:00.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.158+0000 7ff441400640 1 --2- 172.21.15.27:0/506194581 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff43c15bf00 0x7ff43c14e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:00.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.158+0000 7ff441400640 1 --2- 172.21.15.27:0/506194581 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff43c158900 0x7ff43c14e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:00.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.158+0000 7ff441400640 1 --2- 172.21.15.27:0/506194581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff43c0cf2e0 0x7ff43c1526b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:00.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.158+0000 7ff441400640 1 -- 172.21.15.27:0/506194581 >> 172.21.15.27:0/506194581 conn(0x7ff43c0cb700 msgr2=0x7ff43c0cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:00.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.158+0000 7ff441400640 1 -- 172.21.15.27:0/506194581 shutdown_connections 2024-01-15T18:06:00.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.158+0000 7ff441400640 1 -- 172.21.15.27:0/506194581 wait complete. 2024-01-15T18:06:00.207 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-15T18:06:00.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.346+0000 7f6805c00640 1 Processor -- start 2024-01-15T18:06:00.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.346+0000 7f6805c00640 1 -- start start 2024-01-15T18:06:00.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.346+0000 7f6805c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f68000cf460 0x7f68000cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:00.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.346+0000 7f6805c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f680015b4b0 0x7f68000cba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:00.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.346+0000 7f6805c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68000cfd80 con 0x7f68000cec60 2024-01-15T18:06:00.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.346+0000 7f6805c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68000d19a0 con 0x7f68000cf460 2024-01-15T18:06:00.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.346+0000 7f6805c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68000d1c80 con 0x7f680015b4b0 2024-01-15T18:06:00.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.346+0000 7f67fea00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f68000cf460 0x7f68000cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:00.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.346+0000 7f67fe000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f680015b4b0 0x7f68000cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:00.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.346+0000 7f67ff400640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7f68000cec60 0x7f68000cbf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:40132/0 (socket says 172.21.15.27:40132) 2024-01-15T18:06:00.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.346+0000 7f67ff400640 1 -- 172.21.15.27:0/3649662039 learned_addr learned my addr 172.21.15.27:0/3649662039 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:06:00.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.346+0000 7f67fd600640 1 -- 172.21.15.27:0/3649662039 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253093161 0 0) 0x7f68000cfd80 con 0x7f68000cec60 2024-01-15T18:06:00.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.346+0000 7f67fd600640 1 -- 172.21.15.27:0/3649662039 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67e0003530 con 0x7f68000cec60 2024-01-15T18:06:00.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.346+0000 7f67fe000640 1 -- 172.21.15.27:0/3649662039 >> v2:172.21.15.93:3300/0 conn(0x7f68000cf460 msgr2=0x7f68000cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:00.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.346+0000 7f67fe000640 1 --2- 172.21.15.27:0/3649662039 >> v2:172.21.15.93:3300/0 conn(0x7f68000cf460 0x7f68000cf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:00.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.346+0000 7f67fe000640 1 -- 172.21.15.27:0/3649662039 >> v1:172.21.15.93:6789/0 conn(0x7f68000cec60 legacy=0x7f68000cbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:00.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.346+0000 7f67fe000640 1 -- 172.21.15.27:0/3649662039 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68000b07c0 con 0x7f680015b4b0 2024-01-15T18:06:00.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.346+0000 7f67fea00640 1 --2- 172.21.15.27:0/3649662039 >> v2:172.21.15.93:3300/0 conn(0x7f68000cf460 0x7f68000cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:06:00.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.350+0000 7f67fe000640 1 --2- 172.21.15.27:0/3649662039 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f680015b4b0 0x7f68000cba20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f67e80097c0 tx=0x7f67e801c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac95655bd6101ed0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:00.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.350+0000 7f67fd600640 1 -- 172.21.15.27:0/3649662039 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e80ab680 con 0x7f680015b4b0 2024-01-15T18:06:00.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.350+0000 7f67fd600640 1 -- 172.21.15.27:0/3649662039 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e80ab7e0 con 0x7f680015b4b0 2024-01-15T18:06:00.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.350+0000 7f67fd600640 1 -- 172.21.15.27:0/3649662039 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e80abb70 con 0x7f680015b4b0 2024-01-15T18:06:00.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.350+0000 7f6805c00640 1 -- 172.21.15.27:0/3649662039 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f680015b4b0 msgr2=0x7f68000cba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:00.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.350+0000 7f6805c00640 1 --2- 172.21.15.27:0/3649662039 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f680015b4b0 0x7f68000cba20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f67e80097c0 tx=0x7f67e801c480 comp rx=0 tx=0).stop 2024-01-15T18:06:00.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.350+0000 7f6805c00640 1 -- 172.21.15.27:0/3649662039 shutdown_connections 2024-01-15T18:06:00.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.350+0000 7f6805c00640 1 --2- 172.21.15.27:0/3649662039 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f680015b4b0 0x7f68000cba20 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:00.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.350+0000 7f6805c00640 1 --2- 172.21.15.27:0/3649662039 >> v2:172.21.15.93:3300/0 conn(0x7f68000cf460 0x7f68000cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:00.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.350+0000 7f6805c00640 1 -- 172.21.15.27:0/3649662039 >> 172.21.15.27:0/3649662039 conn(0x7f68000d0e80 msgr2=0x7f68000d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:00.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.350+0000 7f6805c00640 1 -- 172.21.15.27:0/3649662039 shutdown_connections 2024-01-15T18:06:00.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.350+0000 7f6805c00640 1 -- 172.21.15.27:0/3649662039 wait complete. 2024-01-15T18:06:00.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.354+0000 7f6805c00640 1 Processor -- start 2024-01-15T18:06:00.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.354+0000 7f6805c00640 1 -- start start 2024-01-15T18:06:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.354+0000 7f6805c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68000cec60 0x7f6800159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.354+0000 7f6805c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f68000cf460 0x7f68000c6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.354+0000 7f6805c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f680015b4b0 0x7f680009ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.354+0000 7f6805c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68000c74f0 con 0x7f68000cec60 2024-01-15T18:06:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.354+0000 7f6805c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680009eb80 con 0x7f68000cf460 2024-01-15T18:06:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.354+0000 7f6805c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f680009d360 con 0x7f680015b4b0 2024-01-15T18:06:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.354+0000 7f67fea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68000cec60 0x7f6800159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.354+0000 7f67fe000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f68000cf460 0x7f68000c6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:00.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.354+0000 7f67fe000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f68000cf460 0x7f68000c6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:40118/0 (socket says 172.21.15.27:40118) 2024-01-15T18:06:00.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.354+0000 7f67fe000640 1 -- 172.21.15.27:0/699056832 learned_addr learned my addr 172.21.15.27:0/699056832 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:00.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.354+0000 7f67ff400640 1 --2- 172.21.15.27:0/699056832 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f680015b4b0 0x7f680009ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:00.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.354+0000 7f67fe000640 1 -- 172.21.15.27:0/699056832 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f680015b4b0 msgr2=0x7f680009ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:00.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.354+0000 7f67fe000640 1 --2- 172.21.15.27:0/699056832 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f680015b4b0 0x7f680009ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:00.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.354+0000 7f67fe000640 1 -- 172.21.15.27:0/699056832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68000cec60 msgr2=0x7f6800159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:00.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.354+0000 7f67fe000640 1 --2- 172.21.15.27:0/699056832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68000cec60 0x7f6800159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:00.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.354+0000 7f67fe000640 1 -- 172.21.15.27:0/699056832 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f0009420 con 0x7f68000cf460 2024-01-15T18:06:00.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.354+0000 7f67fea00640 1 --2- 172.21.15.27:0/699056832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68000cec60 0x7f6800159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:06:00.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.354+0000 7f67fe000640 1 --2- 172.21.15.27:0/699056832 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f68000cf460 0x7f68000c6fb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f67e80ad040 tx=0x7f67e801acc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:00.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.354+0000 7f67f6a00640 1 -- 172.21.15.27:0/699056832 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e80bf610 con 0x7f68000cf460 2024-01-15T18:06:00.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.354+0000 7f67f6a00640 1 -- 172.21.15.27:0/699056832 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e80bf790 con 0x7f68000cf460 2024-01-15T18:06:00.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.354+0000 7f6805c00640 1 -- 172.21.15.27:0/699056832 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67e8009420 con 0x7f68000cf460 2024-01-15T18:06:00.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.354+0000 7f67f6a00640 1 -- 172.21.15.27:0/699056832 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e80bfb40 con 0x7f68000cf460 2024-01-15T18:06:00.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.354+0000 7f6805c00640 1 -- 172.21.15.27:0/699056832 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f680015a480 con 0x7f68000cf460 2024-01-15T18:06:00.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.358+0000 7f6805c00640 1 -- 172.21.15.27:0/699056832 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f680004e850 con 0x7f68000cf460 2024-01-15T18:06:00.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.358+0000 7f67f6a00640 1 -- 172.21.15.27:0/699056832 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f67e80de470 con 0x7f68000cf460 2024-01-15T18:06:00.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.358+0000 7f67f6a00640 1 --2- 172.21.15.27:0/699056832 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f67d80668f0 0x7f67d8068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:00.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.358+0000 7f67f6a00640 1 -- 172.21.15.27:0/699056832 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f67e811f120 con 0x7f68000cf460 2024-01-15T18:06:00.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.358+0000 7f67fea00640 1 --2- 172.21.15.27:0/699056832 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f67d80668f0 0x7f67d8068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:00.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.358+0000 7f67fea00640 1 --2- 172.21.15.27:0/699056832 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f67d80668f0 0x7f67d8068db0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f67f0009fd0 tx=0x7f67f0009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:00.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.362+0000 7f67f6a00640 1 -- 172.21.15.27:0/699056832 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f67e80ec4e0 con 0x7f68000cf460 2024-01-15T18:06:00.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:00.550+0000 7f6805c00640 1 -- 172.21.15.27:0/699056832 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f68000cf240 con 0x7f68000cf460 2024-01-15T18:06:01.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.170+0000 7f67f6a00640 1 -- 172.21.15.27:0/699056832 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7f67e80efb00 con 0x7f68000cf460 2024-01-15T18:06:01.179 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 4 and data pool 5 2024-01-15T18:06:01.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.178+0000 7f6805c00640 1 -- 172.21.15.27:0/699056832 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f67d80668f0 msgr2=0x7f67d8068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:01.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.178+0000 7f6805c00640 1 --2- 172.21.15.27:0/699056832 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f67d80668f0 0x7f67d8068db0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f67f0009fd0 tx=0x7f67f0009190 comp rx=0 tx=0).stop 2024-01-15T18:06:01.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.178+0000 7f6805c00640 1 -- 172.21.15.27:0/699056832 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f68000cf460 msgr2=0x7f68000c6fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:01.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.178+0000 7f6805c00640 1 --2- 172.21.15.27:0/699056832 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f68000cf460 0x7f68000c6fb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f67e80ad040 tx=0x7f67e801acc0 comp rx=0 tx=0).stop 2024-01-15T18:06:01.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.178+0000 7f6805c00640 1 -- 172.21.15.27:0/699056832 shutdown_connections 2024-01-15T18:06:01.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.178+0000 7f6805c00640 1 --2- 172.21.15.27:0/699056832 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f67d80668f0 0x7f67d8068db0 secure :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f67f0009fd0 tx=0x7f67f0009190 comp rx=0 tx=0).stop 2024-01-15T18:06:01.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.178+0000 7f6805c00640 1 --2- 172.21.15.27:0/699056832 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f680015b4b0 0x7f680009ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:01.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.178+0000 7f6805c00640 1 --2- 172.21.15.27:0/699056832 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f68000cf460 0x7f68000c6fb0 secure :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f67e80ad040 tx=0x7f67e801acc0 comp rx=0 tx=0).stop 2024-01-15T18:06:01.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.178+0000 7f6805c00640 1 --2- 172.21.15.27:0/699056832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68000cec60 0x7f6800159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:01.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.178+0000 7f6805c00640 1 -- 172.21.15.27:0/699056832 >> 172.21.15.27:0/699056832 conn(0x7f68000d0e80 msgr2=0x7f680006e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:01.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.182+0000 7f6805c00640 1 -- 172.21.15.27:0/699056832 shutdown_connections 2024-01-15T18:06:01.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.182+0000 7f6805c00640 1 -- 172.21.15.27:0/699056832 wait complete. 2024-01-15T18:06:01.229 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:06:01.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.366+0000 7f683be00640 1 Processor -- start 2024-01-15T18:06:01.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.366+0000 7f683be00640 1 -- start start 2024-01-15T18:06:01.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.366+0000 7f683be00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f68340701a0 0x7f6834070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:01.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.366+0000 7f683be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f68340c92b0 0x7f683406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:01.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.366+0000 7f683be00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f683406c4a0 con 0x7f683406e240 2024-01-15T18:06:01.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.366+0000 7f683be00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6834070ac0 con 0x7f68340c92b0 2024-01-15T18:06:01.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.366+0000 7f683be00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f683415c660 con 0x7f68340701a0 2024-01-15T18:06:01.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.366+0000 7f6838a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f68340c92b0 0x7f683406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:01.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.366+0000 7f6838a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f68340c92b0 0x7f683406dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52880/0 (socket says 172.21.15.27:52880) 2024-01-15T18:06:01.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.366+0000 7f6838a00640 1 -- 172.21.15.27:0/3705827200 learned_addr learned my addr 172.21.15.27:0/3705827200 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:01.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.370+0000 7f6839400640 1 --2- 172.21.15.27:0/3705827200 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f68340701a0 0x7f6834070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:01.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.370+0000 7f6838a00640 1 -- 172.21.15.27:0/3705827200 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f68340701a0 msgr2=0x7f6834070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:01.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.370+0000 7f6838a00640 1 --2- 172.21.15.27:0/3705827200 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f68340701a0 0x7f6834070580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:01.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.370+0000 7f6838a00640 1 -- 172.21.15.27:0/3705827200 >> v1:172.21.15.93:6789/0 conn(0x7f683406e240 legacy=0x7f683415d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:01.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.370+0000 7f6838a00640 1 -- 172.21.15.27:0/3705827200 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f683415c940 con 0x7f68340c92b0 2024-01-15T18:06:01.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.370+0000 7f6838a00640 1 --2- 172.21.15.27:0/3705827200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68340c92b0 0x7f683406dd00 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f681c009870 tx=0x7f681c01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=473b7f6809665bba server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:01.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.370+0000 7f6833e00640 1 -- 172.21.15.27:0/3705827200 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c0aa670 con 0x7f68340c92b0 2024-01-15T18:06:01.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.370+0000 7f6833e00640 1 -- 172.21.15.27:0/3705827200 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681c0aa7d0 con 0x7f68340c92b0 2024-01-15T18:06:01.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.370+0000 7f6833e00640 1 -- 172.21.15.27:0/3705827200 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c0aab60 con 0x7f68340c92b0 2024-01-15T18:06:01.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.374+0000 7f683be00640 1 -- 172.21.15.27:0/3705827200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68340c92b0 msgr2=0x7f683406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.374+0000 7f683be00640 1 --2- 172.21.15.27:0/3705827200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68340c92b0 0x7f683406dd00 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f681c009870 tx=0x7f681c01c510 comp rx=0 tx=0).stop 2024-01-15T18:06:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.374+0000 7f683be00640 1 -- 172.21.15.27:0/3705827200 shutdown_connections 2024-01-15T18:06:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.374+0000 7f683be00640 1 --2- 172.21.15.27:0/3705827200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68340c92b0 0x7f683406dd00 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.374+0000 7f683be00640 1 --2- 172.21.15.27:0/3705827200 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f68340701a0 0x7f6834070580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.374+0000 7f683be00640 1 -- 172.21.15.27:0/3705827200 >> 172.21.15.27:0/3705827200 conn(0x7f68340cec60 msgr2=0x7f68340cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.374+0000 7f683be00640 1 -- 172.21.15.27:0/3705827200 shutdown_connections 2024-01-15T18:06:01.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.374+0000 7f683be00640 1 -- 172.21.15.27:0/3705827200 wait complete. 2024-01-15T18:06:01.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.374+0000 7f683be00640 1 Processor -- start 2024-01-15T18:06:01.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.374+0000 7f683be00640 1 -- start start 2024-01-15T18:06:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.374+0000 7f683be00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f683406e240 0x7f6834143eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.374+0000 7f683be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68340701a0 0x7f68340cde20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.374+0000 7f683be00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f68340c92b0 0x7f68340ce360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.374+0000 7f683be00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6834159cf0 con 0x7f68340701a0 2024-01-15T18:06:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.374+0000 7f683be00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6834159fd0 con 0x7f683406e240 2024-01-15T18:06:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.374+0000 7f683be00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f683415a2b0 con 0x7f68340c92b0 2024-01-15T18:06:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.374+0000 7f6838a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68340701a0 0x7f68340cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.374+0000 7f6838a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68340701a0 0x7f68340cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52894/0 (socket says 172.21.15.27:52894) 2024-01-15T18:06:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.374+0000 7f6838a00640 1 -- 172.21.15.27:0/2303332118 learned_addr learned my addr 172.21.15.27:0/2303332118 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.378+0000 7f6839400640 1 --2- 172.21.15.27:0/2303332118 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f683406e240 0x7f6834143eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.378+0000 7f6839e00640 1 --2- 172.21.15.27:0/2303332118 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f68340c92b0 0x7f68340ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.378+0000 7f6838a00640 1 -- 172.21.15.27:0/2303332118 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f68340c92b0 msgr2=0x7f68340ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.378+0000 7f6838a00640 1 --2- 172.21.15.27:0/2303332118 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f68340c92b0 0x7f68340ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.378+0000 7f6838a00640 1 -- 172.21.15.27:0/2303332118 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f683406e240 msgr2=0x7f6834143eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.378+0000 7f6838a00640 1 --2- 172.21.15.27:0/2303332118 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f683406e240 0x7f6834143eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.378+0000 7f6838a00640 1 -- 172.21.15.27:0/2303332118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c0094d0 con 0x7f68340701a0 2024-01-15T18:06:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.378+0000 7f6839e00640 1 --2- 172.21.15.27:0/2303332118 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f68340c92b0 0x7f68340ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:06:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.378+0000 7f6838a00640 1 --2- 172.21.15.27:0/2303332118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68340701a0 0x7f68340cde20 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f681c01a140 tx=0x7f681c01ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.378+0000 7f6831600640 1 -- 172.21.15.27:0/2303332118 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c0b3800 con 0x7f68340701a0 2024-01-15T18:06:01.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.378+0000 7f6831600640 1 -- 172.21.15.27:0/2303332118 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681c0b3980 con 0x7f68340701a0 2024-01-15T18:06:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.378+0000 7f6831600640 1 -- 172.21.15.27:0/2303332118 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c0b3d30 con 0x7f68340701a0 2024-01-15T18:06:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.378+0000 7f683be00640 1 -- 172.21.15.27:0/2303332118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f683415a590 con 0x7f68340701a0 2024-01-15T18:06:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.378+0000 7f683be00640 1 -- 172.21.15.27:0/2303332118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68341599a0 con 0x7f68340701a0 2024-01-15T18:06:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.378+0000 7f683be00640 1 -- 172.21.15.27:0/2303332118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f683404e850 con 0x7f68340701a0 2024-01-15T18:06:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.378+0000 7f6831600640 1 -- 172.21.15.27:0/2303332118 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f681c0b2380 con 0x7f68340701a0 2024-01-15T18:06:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.378+0000 7f6831600640 1 --2- 172.21.15.27:0/2303332118 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f680c0668f0 0x7f680c068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.378+0000 7f6831600640 1 -- 172.21.15.27:0/2303332118 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f681c11eb80 con 0x7f68340701a0 2024-01-15T18:06:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.382+0000 7f6839400640 1 --2- 172.21.15.27:0/2303332118 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f680c0668f0 0x7f680c068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.382+0000 7f6839400640 1 --2- 172.21.15.27:0/2303332118 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f680c0668f0 0x7f680c068db0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6824009af0 tx=0x7f6824008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:01.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.386+0000 7f6831600640 1 -- 172.21.15.27:0/2303332118 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f681c0ebdd0 con 0x7f68340701a0 2024-01-15T18:06:01.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.542+0000 7f683be00640 1 -- 172.21.15.27:0/2303332118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6834159e30 con 0x7f68340701a0 2024-01-15T18:06:01.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.542+0000 7f6831600640 1 -- 172.21.15.27:0/2303332118 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16486 (secure 0 0 0) 0x7f6834159e30 con 0x7f68340701a0 2024-01-15T18:06:01.550 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:06:01.550 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":33,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:06:00.557789+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:05:58.389641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-15T18:05:59.555643+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:06:01.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.546+0000 7f683be00640 1 -- 172.21.15.27:0/2303332118 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f680c0668f0 msgr2=0x7f680c068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:01.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.546+0000 7f683be00640 1 --2- 172.21.15.27:0/2303332118 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f680c0668f0 0x7f680c068db0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6824009af0 tx=0x7f6824008040 comp rx=0 tx=0).stop 2024-01-15T18:06:01.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.546+0000 7f683be00640 1 -- 172.21.15.27:0/2303332118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68340701a0 msgr2=0x7f68340cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:01.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.546+0000 7f683be00640 1 --2- 172.21.15.27:0/2303332118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68340701a0 0x7f68340cde20 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f681c01a140 tx=0x7f681c01ad50 comp rx=0 tx=0).stop 2024-01-15T18:06:01.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.546+0000 7f683be00640 1 -- 172.21.15.27:0/2303332118 shutdown_connections 2024-01-15T18:06:01.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.546+0000 7f683be00640 1 --2- 172.21.15.27:0/2303332118 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f680c0668f0 0x7f680c068db0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:01.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.546+0000 7f683be00640 1 --2- 172.21.15.27:0/2303332118 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f68340c92b0 0x7f68340ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:01.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.546+0000 7f683be00640 1 --2- 172.21.15.27:0/2303332118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68340701a0 0x7f68340cde20 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:01.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.546+0000 7f683be00640 1 --2- 172.21.15.27:0/2303332118 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f683406e240 0x7f6834143eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:01.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.546+0000 7f683be00640 1 -- 172.21.15.27:0/2303332118 >> 172.21.15.27:0/2303332118 conn(0x7f68340cec60 msgr2=0x7f68340afc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:01.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.546+0000 7f683be00640 1 -- 172.21.15.27:0/2303332118 shutdown_connections 2024-01-15T18:06:01.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.546+0000 7f683be00640 1 -- 172.21.15.27:0/2303332118 wait complete. 2024-01-15T18:06:01.573 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:06:01.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.706+0000 7fa67c800640 1 Processor -- start 2024-01-15T18:06:01.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.706+0000 7fa67c800640 1 -- start start 2024-01-15T18:06:01.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.710+0000 7fa67c800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa67406db80 0x7fa67406df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:01.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.710+0000 7fa67c800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa6740cdd10 0x7fa674070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:01.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.710+0000 7fa67c800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6740ce920 con 0x7fa67415e730 2024-01-15T18:06:01.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.710+0000 7fa67c800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6740cdb30 con 0x7fa67406db80 2024-01-15T18:06:01.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.710+0000 7fa67c800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa67406e4a0 con 0x7fa6740cdd10 2024-01-15T18:06:01.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.710+0000 7fa679e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa67406db80 0x7fa67406df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:01.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.710+0000 7fa679e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa67406db80 0x7fa67406df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52898/0 (socket says 172.21.15.27:52898) 2024-01-15T18:06:01.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.710+0000 7fa679400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa67415e730 0x7fa6740cd420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51860/0 (socket says 172.21.15.27:51860) 2024-01-15T18:06:01.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.710+0000 7fa679e00640 1 -- 172.21.15.27:0/3478993353 learned_addr learned my addr 172.21.15.27:0/3478993353 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:01.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.710+0000 7fa67a800640 1 --2- 172.21.15.27:0/3478993353 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa6740cdd10 0x7fa674070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:01.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.710+0000 7fa679e00640 1 -- 172.21.15.27:0/3478993353 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa6740cdd10 msgr2=0x7fa674070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:01.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.710+0000 7fa679e00640 1 --2- 172.21.15.27:0/3478993353 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa6740cdd10 0x7fa674070c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:01.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.710+0000 7fa679e00640 1 -- 172.21.15.27:0/3478993353 >> v1:172.21.15.27:6789/0 conn(0x7fa67415e730 legacy=0x7fa6740cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:01.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.710+0000 7fa679e00640 1 -- 172.21.15.27:0/3478993353 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6740ce640 con 0x7fa67406db80 2024-01-15T18:06:01.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.710+0000 7fa678a00640 1 -- 172.21.15.27:0/3478993353 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825425388 0 0) 0x7fa6740ce920 con 0x7fa67415e730 2024-01-15T18:06:01.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.714+0000 7fa679e00640 1 --2- 172.21.15.27:0/3478993353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa67406db80 0x7fa67406df60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa664009870 tx=0x7fa66401c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee97f98cbca4cefc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:01.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.714+0000 7fa678a00640 1 -- 172.21.15.27:0/3478993353 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6640aa640 con 0x7fa67406db80 2024-01-15T18:06:01.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.714+0000 7fa678a00640 1 -- 172.21.15.27:0/3478993353 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6640aa7a0 con 0x7fa67406db80 2024-01-15T18:06:01.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.714+0000 7fa678a00640 1 -- 172.21.15.27:0/3478993353 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6640aab30 con 0x7fa67406db80 2024-01-15T18:06:01.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.714+0000 7fa67c800640 1 -- 172.21.15.27:0/3478993353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa67406db80 msgr2=0x7fa67406df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:01.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.714+0000 7fa67c800640 1 --2- 172.21.15.27:0/3478993353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa67406db80 0x7fa67406df60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa664009870 tx=0x7fa66401c510 comp rx=0 tx=0).stop 2024-01-15T18:06:01.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.714+0000 7fa67c800640 1 -- 172.21.15.27:0/3478993353 shutdown_connections 2024-01-15T18:06:01.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.714+0000 7fa67c800640 1 --2- 172.21.15.27:0/3478993353 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa6740cdd10 0x7fa674070c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:01.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.714+0000 7fa67c800640 1 --2- 172.21.15.27:0/3478993353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa67406db80 0x7fa67406df60 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:01.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.714+0000 7fa67c800640 1 -- 172.21.15.27:0/3478993353 >> 172.21.15.27:0/3478993353 conn(0x7fa6740cec60 msgr2=0x7fa6740cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:01.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.714+0000 7fa67c800640 1 -- 172.21.15.27:0/3478993353 shutdown_connections 2024-01-15T18:06:01.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.714+0000 7fa67c800640 1 -- 172.21.15.27:0/3478993353 wait complete. 2024-01-15T18:06:01.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.714+0000 7fa67c800640 1 Processor -- start 2024-01-15T18:06:01.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.714+0000 7fa67c800640 1 -- start start 2024-01-15T18:06:01.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.718+0000 7fa67c800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa67406db80 0x7fa6740c6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:01.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.718+0000 7fa67c800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa6740cdd10 0x7fa674158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:01.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.718+0000 7fa67c800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa67415e730 0x7fa674153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:01.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.718+0000 7fa67c800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6740c7410 con 0x7fa67415e730 2024-01-15T18:06:01.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.718+0000 7fa67c800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa674157bd0 con 0x7fa67406db80 2024-01-15T18:06:01.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.718+0000 7fa67c800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6740c84e0 con 0x7fa6740cdd10 2024-01-15T18:06:01.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.718+0000 7fa67a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa67415e730 0x7fa674153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:01.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.718+0000 7fa67a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa67415e730 0x7fa674153ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52902/0 (socket says 172.21.15.27:52902) 2024-01-15T18:06:01.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.718+0000 7fa67a800640 1 -- 172.21.15.27:0/827903661 learned_addr learned my addr 172.21.15.27:0/827903661 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:01.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.718+0000 7fa679e00640 1 --2- 172.21.15.27:0/827903661 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa67406db80 0x7fa6740c6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:01.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.718+0000 7fa679400640 1 --2- 172.21.15.27:0/827903661 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa6740cdd10 0x7fa674158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:01.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.718+0000 7fa67a800640 1 -- 172.21.15.27:0/827903661 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa6740cdd10 msgr2=0x7fa674158060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:01.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.718+0000 7fa67a800640 1 --2- 172.21.15.27:0/827903661 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa6740cdd10 0x7fa674158060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:01.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.718+0000 7fa67a800640 1 -- 172.21.15.27:0/827903661 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa67406db80 msgr2=0x7fa6740c6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:01.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.718+0000 7fa67a800640 1 --2- 172.21.15.27:0/827903661 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa67406db80 0x7fa6740c6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:01.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.718+0000 7fa67a800640 1 -- 172.21.15.27:0/827903661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6640094d0 con 0x7fa67415e730 2024-01-15T18:06:01.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.718+0000 7fa67a800640 1 --2- 172.21.15.27:0/827903661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa67415e730 0x7fa674153ea0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa66c007e40 tx=0x7fa66c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:01.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.718+0000 7fa672000640 1 -- 172.21.15.27:0/827903661 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66c014620 con 0x7fa67415e730 2024-01-15T18:06:01.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.718+0000 7fa672000640 1 -- 172.21.15.27:0/827903661 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa66c00ee90 con 0x7fa67415e730 2024-01-15T18:06:01.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.718+0000 7fa672000640 1 -- 172.21.15.27:0/827903661 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66c0149e0 con 0x7fa67415e730 2024-01-15T18:06:01.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.718+0000 7fa67c800640 1 -- 172.21.15.27:0/827903661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa674157580 con 0x7fa67415e730 2024-01-15T18:06:01.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.718+0000 7fa67c800640 1 -- 172.21.15.27:0/827903661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6741571a0 con 0x7fa67415e730 2024-01-15T18:06:01.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.718+0000 7fa67c800640 1 -- 172.21.15.27:0/827903661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6741573c0 con 0x7fa67415e730 2024-01-15T18:06:01.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.722+0000 7fa672000640 1 -- 172.21.15.27:0/827903661 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fa66c014b40 con 0x7fa67415e730 2024-01-15T18:06:01.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.722+0000 7fa672000640 1 --2- 172.21.15.27:0/827903661 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa64c0668f0 0x7fa64c068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:01.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.722+0000 7fa672000640 1 -- 172.21.15.27:0/827903661 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa66c084c60 con 0x7fa67415e730 2024-01-15T18:06:01.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.722+0000 7fa679e00640 1 --2- 172.21.15.27:0/827903661 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa64c0668f0 0x7fa64c068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:01.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.722+0000 7fa679e00640 1 --2- 172.21.15.27:0/827903661 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa64c0668f0 0x7fa64c068db0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa6640ae000 tx=0x7fa6640a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:01.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.726+0000 7fa672000640 1 -- 172.21.15.27:0/827903661 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa66c051f40 con 0x7fa67415e730 2024-01-15T18:06:01.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.882+0000 7fa67c800640 1 -- 172.21.15.27:0/827903661 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa674157d40 con 0x7fa67415e730 2024-01-15T18:06:01.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.882+0000 7fa672000640 1 -- 172.21.15.27:0/827903661 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16486 (secure 0 0 0) 0x7fa66c07ce60 con 0x7fa67415e730 2024-01-15T18:06:01.889 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:06:01.889 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":33,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:06:00.557789+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:05:58.389641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-15T18:05:59.555643+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:06:01.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.886+0000 7fa67c800640 1 -- 172.21.15.27:0/827903661 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa64c0668f0 msgr2=0x7fa64c068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:01.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.886+0000 7fa67c800640 1 --2- 172.21.15.27:0/827903661 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa64c0668f0 0x7fa64c068db0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa6640ae000 tx=0x7fa6640a7000 comp rx=0 tx=0).stop 2024-01-15T18:06:01.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.886+0000 7fa67c800640 1 -- 172.21.15.27:0/827903661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa67415e730 msgr2=0x7fa674153ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:01.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.886+0000 7fa67c800640 1 --2- 172.21.15.27:0/827903661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa67415e730 0x7fa674153ea0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa66c007e40 tx=0x7fa66c00e4b0 comp rx=0 tx=0).stop 2024-01-15T18:06:01.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.886+0000 7fa67c800640 1 -- 172.21.15.27:0/827903661 shutdown_connections 2024-01-15T18:06:01.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.886+0000 7fa67c800640 1 --2- 172.21.15.27:0/827903661 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa64c0668f0 0x7fa64c068db0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:01.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.886+0000 7fa67c800640 1 --2- 172.21.15.27:0/827903661 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa67415e730 0x7fa674153ea0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:01.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.886+0000 7fa67c800640 1 --2- 172.21.15.27:0/827903661 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa6740cdd10 0x7fa674158060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:01.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.886+0000 7fa67c800640 1 --2- 172.21.15.27:0/827903661 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa67406db80 0x7fa6740c6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:01.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.886+0000 7fa67c800640 1 -- 172.21.15.27:0/827903661 >> 172.21.15.27:0/827903661 conn(0x7fa6740cec60 msgr2=0x7fa6740cf040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:01.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.886+0000 7fa67c800640 1 -- 172.21.15.27:0/827903661 shutdown_connections 2024-01-15T18:06:01.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:01.886+0000 7fa67c800640 1 -- 172.21.15.27:0/827903661 wait complete. 2024-01-15T18:06:01.912 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-15T18:06:02.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.050+0000 7f0162400640 1 Processor -- start 2024-01-15T18:06:02.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.050+0000 7f0162400640 1 -- start start 2024-01-15T18:06:02.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.050+0000 7f0162400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f015c15b4b0 0x7f015c0b1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:02.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.050+0000 7f0162400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f015c0cec60 0x7f015c0b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:02.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.050+0000 7f0162400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f015c0d19a0 con 0x7f015c0cf460 2024-01-15T18:06:02.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.050+0000 7f0162400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f015c0d1c80 con 0x7f015c15b4b0 2024-01-15T18:06:02.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.050+0000 7f0162400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015c0b07c0 con 0x7f015c0cec60 2024-01-15T18:06:02.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.050+0000 7f015aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f015c15b4b0 0x7f015c0b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:02.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.050+0000 7f015aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f015c15b4b0 0x7f015c0b1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52906/0 (socket says 172.21.15.27:52906) 2024-01-15T18:06:02.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.050+0000 7f015aa00640 1 -- 172.21.15.27:0/405759940 learned_addr learned my addr 172.21.15.27:0/405759940 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:02.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.050+0000 7f015be00640 1 --2- 172.21.15.27:0/405759940 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f015c0cec60 0x7f015c0b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:02.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.050+0000 7f015aa00640 1 -- 172.21.15.27:0/405759940 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f015c0cec60 msgr2=0x7f015c0b1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:02.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.050+0000 7f015aa00640 1 --2- 172.21.15.27:0/405759940 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f015c0cec60 0x7f015c0b1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:02.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.050+0000 7f015aa00640 1 -- 172.21.15.27:0/405759940 >> v1:172.21.15.93:6789/0 conn(0x7f015c0cf460 legacy=0x7f015c0cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:02.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.050+0000 7f015aa00640 1 -- 172.21.15.27:0/405759940 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015c0b0aa0 con 0x7f015c15b4b0 2024-01-15T18:06:02.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.050+0000 7f015be00640 1 --2- 172.21.15.27:0/405759940 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f015c0cec60 0x7f015c0b1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:06:02.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.054+0000 7f015aa00640 1 --2- 172.21.15.27:0/405759940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f015c15b4b0 0x7f015c0b1580 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0144009870 tx=0x7f014401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e33848f77ba1033 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:02.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.054+0000 7f015a000640 1 -- 172.21.15.27:0/405759940 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015c0b2810 con 0x7f015c15b4b0 2024-01-15T18:06:02.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.054+0000 7f015a000640 1 -- 172.21.15.27:0/405759940 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01440a9400 con 0x7f015c15b4b0 2024-01-15T18:06:02.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.054+0000 7f015a000640 1 -- 172.21.15.27:0/405759940 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01440a9790 con 0x7f015c15b4b0 2024-01-15T18:06:02.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.054+0000 7f0162400640 1 -- 172.21.15.27:0/405759940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f015c15b4b0 msgr2=0x7f015c0b1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:02.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.054+0000 7f0162400640 1 --2- 172.21.15.27:0/405759940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f015c15b4b0 0x7f015c0b1580 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0144009870 tx=0x7f014401c540 comp rx=0 tx=0).stop 2024-01-15T18:06:02.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.054+0000 7f0162400640 1 -- 172.21.15.27:0/405759940 shutdown_connections 2024-01-15T18:06:02.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.054+0000 7f0162400640 1 --2- 172.21.15.27:0/405759940 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f015c0cec60 0x7f015c0b1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:02.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.054+0000 7f0162400640 1 --2- 172.21.15.27:0/405759940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f015c15b4b0 0x7f015c0b1580 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:02.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.054+0000 7f0162400640 1 -- 172.21.15.27:0/405759940 >> 172.21.15.27:0/405759940 conn(0x7f015c0d0e80 msgr2=0x7f015c0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:02.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.054+0000 7f0162400640 1 -- 172.21.15.27:0/405759940 shutdown_connections 2024-01-15T18:06:02.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.054+0000 7f0162400640 1 -- 172.21.15.27:0/405759940 wait complete. 2024-01-15T18:06:02.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f0162400640 1 Processor -- start 2024-01-15T18:06:02.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f0162400640 1 -- start start 2024-01-15T18:06:02.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f0162400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f015c0cec60 0x7f015c15dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:02.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f0162400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f015c0cf460 0x7f015c1527a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:02.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f0162400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f015c15b4b0 0x7f015c152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:02.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f0162400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c153220 con 0x7f015c0cf460 2024-01-15T18:06:02.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f0162400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c1f6f10 con 0x7f015c0cec60 2024-01-15T18:06:02.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f0162400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015c15d3d0 con 0x7f015c15b4b0 2024-01-15T18:06:02.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f015aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f015c0cf460 0x7f015c1527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:02.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f015aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f015c0cf460 0x7f015c1527a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52918/0 (socket says 172.21.15.27:52918) 2024-01-15T18:06:02.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f015aa00640 1 -- 172.21.15.27:0/1329427335 learned_addr learned my addr 172.21.15.27:0/1329427335 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:02.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f015be00640 1 --2- 172.21.15.27:0/1329427335 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f015c15b4b0 0x7f015c152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:02.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f015b400640 1 --2- 172.21.15.27:0/1329427335 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f015c0cec60 0x7f015c15dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:02.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f015aa00640 1 -- 172.21.15.27:0/1329427335 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f015c15b4b0 msgr2=0x7f015c152ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:02.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f015aa00640 1 --2- 172.21.15.27:0/1329427335 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f015c15b4b0 0x7f015c152ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:02.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f015aa00640 1 -- 172.21.15.27:0/1329427335 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f015c0cec60 msgr2=0x7f015c15dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:02.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f015aa00640 1 --2- 172.21.15.27:0/1329427335 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f015c0cec60 0x7f015c15dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:02.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f015aa00640 1 -- 172.21.15.27:0/1329427335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01440094d0 con 0x7f015c0cf460 2024-01-15T18:06:02.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f015be00640 1 --2- 172.21.15.27:0/1329427335 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f015c15b4b0 0x7f015c152ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:06:02.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f015b400640 1 --2- 172.21.15.27:0/1329427335 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f015c0cec60 0x7f015c15dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:06:02.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f015aa00640 1 --2- 172.21.15.27:0/1329427335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f015c0cf460 0x7f015c1527a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f014401abf0 tx=0x7f01440192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:02.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f0153400640 1 -- 172.21.15.27:0/1329427335 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01440b95f0 con 0x7f015c0cf460 2024-01-15T18:06:02.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f0153400640 1 -- 172.21.15.27:0/1329427335 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01440b9750 con 0x7f015c0cf460 2024-01-15T18:06:02.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f0153400640 1 -- 172.21.15.27:0/1329427335 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01440b9ac0 con 0x7f015c0cf460 2024-01-15T18:06:02.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f0162400640 1 -- 172.21.15.27:0/1329427335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f015c1f65e0 con 0x7f015c0cf460 2024-01-15T18:06:02.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.058+0000 7f0162400640 1 -- 172.21.15.27:0/1329427335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f015c1f6ba0 con 0x7f015c0cf460 2024-01-15T18:06:02.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.062+0000 7f0162400640 1 -- 172.21.15.27:0/1329427335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f015c1f6d30 con 0x7f015c0cf460 2024-01-15T18:06:02.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.062+0000 7f0153400640 1 -- 172.21.15.27:0/1329427335 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f01440b1600 con 0x7f015c0cf460 2024-01-15T18:06:02.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.062+0000 7f0153400640 1 --2- 172.21.15.27:0/1329427335 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f01340668f0 0x7f0134068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:02.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.062+0000 7f0153400640 1 -- 172.21.15.27:0/1329427335 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f014411ef30 con 0x7f015c0cf460 2024-01-15T18:06:02.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.062+0000 7f015b400640 1 --2- 172.21.15.27:0/1329427335 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f01340668f0 0x7f0134068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:02.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.062+0000 7f015b400640 1 --2- 172.21.15.27:0/1329427335 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f01340668f0 0x7f0134068db0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f014c000f70 tx=0x7f014c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:02.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.066+0000 7f0153400640 1 -- 172.21.15.27:0/1329427335 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f01440ec160 con 0x7f015c0cf460 2024-01-15T18:06:02.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:02.262+0000 7f0162400640 1 -- 172.21.15.27:0/1329427335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f015c1f7080 con 0x7f015c0cf460 2024-01-15T18:06:03.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.202+0000 7f0153400640 1 -- 172.21.15.27:0/1329427335 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7f01440a6030 con 0x7f015c0cf460 2024-01-15T18:06:03.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.202+0000 7f0162400640 1 -- 172.21.15.27:0/1329427335 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f01340668f0 msgr2=0x7f0134068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:03.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.202+0000 7f0162400640 1 --2- 172.21.15.27:0/1329427335 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f01340668f0 0x7f0134068db0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f014c000f70 tx=0x7f014c009040 comp rx=0 tx=0).stop 2024-01-15T18:06:03.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.202+0000 7f0162400640 1 -- 172.21.15.27:0/1329427335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f015c0cf460 msgr2=0x7f015c1527a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:03.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.202+0000 7f0162400640 1 --2- 172.21.15.27:0/1329427335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f015c0cf460 0x7f015c1527a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f014401abf0 tx=0x7f01440192e0 comp rx=0 tx=0).stop 2024-01-15T18:06:03.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.202+0000 7f0162400640 1 -- 172.21.15.27:0/1329427335 shutdown_connections 2024-01-15T18:06:03.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.206+0000 7f0162400640 1 --2- 172.21.15.27:0/1329427335 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f01340668f0 0x7f0134068db0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:03.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.206+0000 7f0162400640 1 --2- 172.21.15.27:0/1329427335 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f015c15b4b0 0x7f015c152ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:03.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.206+0000 7f0162400640 1 --2- 172.21.15.27:0/1329427335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f015c0cf460 0x7f015c1527a0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:03.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.206+0000 7f0162400640 1 --2- 172.21.15.27:0/1329427335 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f015c0cec60 0x7f015c15dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:03.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.206+0000 7f0162400640 1 -- 172.21.15.27:0/1329427335 >> 172.21.15.27:0/1329427335 conn(0x7f015c0d0e80 msgr2=0x7f015c15a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:03.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.206+0000 7f0162400640 1 -- 172.21.15.27:0/1329427335 shutdown_connections 2024-01-15T18:06:03.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.206+0000 7f0162400640 1 -- 172.21.15.27:0/1329427335 wait complete. 2024-01-15T18:06:03.234 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-15T18:06:03.234 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-15T18:06:03.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f4216a00640 1 Processor -- start 2024-01-15T18:06:03.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f4216a00640 1 -- start start 2024-01-15T18:06:03.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f4216a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f421015f550 0x7f42100cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:03.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f4216a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42100b1cb0 con 0x7f4210158900 2024-01-15T18:06:03.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f4216a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421015fa00 con 0x7f42100cf110 2024-01-15T18:06:03.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f4216a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42100cf540 con 0x7f421015f550 2024-01-15T18:06:03.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f420fe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4210158900 0x7f421015bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51868/0 (socket says 172.21.15.27:51868) 2024-01-15T18:06:03.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f420fe00640 1 -- 172.21.15.27:0/3194679386 learned_addr learned my addr 172.21.15.27:0/3194679386 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:06:03.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f420f400640 1 --2- 172.21.15.27:0/3194679386 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f421015f550 0x7f42100cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:03.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f420ea00640 1 -- 172.21.15.27:0/3194679386 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685879658 0 0) 0x7f42100b1cb0 con 0x7f4210158900 2024-01-15T18:06:03.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f420ea00640 1 -- 172.21.15.27:0/3194679386 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41f0003530 con 0x7f4210158900 2024-01-15T18:06:03.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f420ea00640 1 -- 172.21.15.27:0/3194679386 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927466215 0 0) 0x7f421015fa00 con 0x7f42100cf110 2024-01-15T18:06:03.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f420ea00640 1 -- 172.21.15.27:0/3194679386 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42100b1cb0 con 0x7f42100cf110 2024-01-15T18:06:03.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f420ea00640 1 -- 172.21.15.27:0/3194679386 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 368650348 0 0) 0x7f41f0003530 con 0x7f4210158900 2024-01-15T18:06:03.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f420ea00640 1 -- 172.21.15.27:0/3194679386 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f421015f550 msgr2=0x7f42100cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:03.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f420ea00640 1 --2- 172.21.15.27:0/3194679386 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f421015f550 0x7f42100cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:03.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f420ea00640 1 -- 172.21.15.27:0/3194679386 >> v1:172.21.15.93:6789/0 conn(0x7f42100cf110 legacy=0x7f42100701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:03.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f420ea00640 1 -- 172.21.15.27:0/3194679386 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42100b1f00 con 0x7f4210158900 2024-01-15T18:06:03.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f420ea00640 1 -- 172.21.15.27:0/3194679386 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f4200001680 con 0x7f4210158900 2024-01-15T18:06:03.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f420ea00640 1 -- 172.21.15.27:0/3194679386 >> v1:172.21.15.27:6789/0 conn(0x7f4210158900 legacy=0x7f421015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:03.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f420ea00640 1 --2- 172.21.15.27:0/3194679386 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f41f0003bb0 0x7f41f0003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:03.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f420ea00640 1 --2- 172.21.15.27:0/3194679386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41f00047d0 0x7f41f0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:03.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f420ea00640 1 --2- 172.21.15.27:0/3194679386 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f421015f550 0x7f41f0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:03.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f420ea00640 1 -- 172.21.15.27:0/3194679386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42100cf540 con 0x7f41f00047d0 2024-01-15T18:06:03.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f420ea00640 1 -- 172.21.15.27:0/3194679386 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f000baa0 con 0x7f41f0003bb0 2024-01-15T18:06:03.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f420ea00640 1 -- 172.21.15.27:0/3194679386 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f000bc60 con 0x7f421015f550 2024-01-15T18:06:03.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.378+0000 7f420f400640 1 --2- 172.21.15.27:0/3194679386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41f00047d0 0x7f41f0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:03.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.382+0000 7f420fe00640 1 --2- 172.21.15.27:0/3194679386 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f41f0003bb0 0x7f41f0003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:03.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.382+0000 7f4214a00640 1 --2- 172.21.15.27:0/3194679386 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f421015f550 0x7f41f0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:03.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.382+0000 7f420f400640 1 -- 172.21.15.27:0/3194679386 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f421015f550 msgr2=0x7f41f0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:03.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.382+0000 7f420f400640 1 --2- 172.21.15.27:0/3194679386 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f421015f550 0x7f41f0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:03.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.382+0000 7f420f400640 1 -- 172.21.15.27:0/3194679386 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f41f0003bb0 msgr2=0x7f41f0003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:03.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.382+0000 7f420f400640 1 --2- 172.21.15.27:0/3194679386 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f41f0003bb0 0x7f41f0003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:03.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.382+0000 7f420f400640 1 -- 172.21.15.27:0/3194679386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f0003530 con 0x7f41f00047d0 2024-01-15T18:06:03.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.382+0000 7f420f400640 1 --2- 172.21.15.27:0/3194679386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41f00047d0 0x7f41f0004c50 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f41f8009ab0 tx=0x7f41f801ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=946fedb82f0491e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:03.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.382+0000 7f420ea00640 1 -- 172.21.15.27:0/3194679386 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f80a5600 con 0x7f41f00047d0 2024-01-15T18:06:03.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.382+0000 7f420ea00640 1 -- 172.21.15.27:0/3194679386 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f80a5760 con 0x7f41f00047d0 2024-01-15T18:06:03.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.382+0000 7f420ea00640 1 -- 172.21.15.27:0/3194679386 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f80a5b10 con 0x7f41f00047d0 2024-01-15T18:06:03.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.382+0000 7f4216a00640 1 -- 172.21.15.27:0/3194679386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41f00047d0 msgr2=0x7f41f0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:03.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.382+0000 7f4216a00640 1 --2- 172.21.15.27:0/3194679386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41f00047d0 0x7f41f0004c50 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f41f8009ab0 tx=0x7f41f801ca10 comp rx=0 tx=0).stop 2024-01-15T18:06:03.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.382+0000 7f4214a00640 1 -- 172.21.15.27:0/3194679386 reap_dead start 2024-01-15T18:06:03.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.386+0000 7f4216a00640 1 -- 172.21.15.27:0/3194679386 shutdown_connections 2024-01-15T18:06:03.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.386+0000 7f4216a00640 1 -- 172.21.15.27:0/3194679386 >> 172.21.15.27:0/3194679386 conn(0x7f42100cb700 msgr2=0x7f42100cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:03.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.386+0000 7f4216a00640 1 -- 172.21.15.27:0/3194679386 shutdown_connections 2024-01-15T18:06:03.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.386+0000 7f4216a00640 1 -- 172.21.15.27:0/3194679386 wait complete. 2024-01-15T18:06:03.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.386+0000 7f4216a00640 1 Processor -- start 2024-01-15T18:06:03.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.386+0000 7f4216a00640 1 -- start start 2024-01-15T18:06:03.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.386+0000 7f4216a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f41f00047d0 0x7f4210143350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:03.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.386+0000 7f4216a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42101f69c0 0x7f42100aff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:03.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.386+0000 7f4216a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4210159f80 0x7f421015a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:03.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.386+0000 7f4216a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421015a940 con 0x7f42101f69c0 2024-01-15T18:06:03.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.386+0000 7f4216a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4210156080 con 0x7f41f00047d0 2024-01-15T18:06:03.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.386+0000 7f4216a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4210155540 con 0x7f4210159f80 2024-01-15T18:06:03.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.386+0000 7f420f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42101f69c0 0x7f42100aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:03.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.386+0000 7f420f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42101f69c0 0x7f42100aff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52938/0 (socket says 172.21.15.27:52938) 2024-01-15T18:06:03.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.386+0000 7f420f400640 1 -- 172.21.15.27:0/1077059592 learned_addr learned my addr 172.21.15.27:0/1077059592 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:03.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.386+0000 7f420fe00640 1 --2- 172.21.15.27:0/1077059592 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f41f00047d0 0x7f4210143350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:03.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.386+0000 7f4214a00640 1 --2- 172.21.15.27:0/1077059592 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4210159f80 0x7f421015a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:03.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.386+0000 7f420f400640 1 -- 172.21.15.27:0/1077059592 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4210159f80 msgr2=0x7f421015a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:03.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.386+0000 7f420f400640 1 --2- 172.21.15.27:0/1077059592 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4210159f80 0x7f421015a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:03.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.386+0000 7f420f400640 1 -- 172.21.15.27:0/1077059592 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f41f00047d0 msgr2=0x7f4210143350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:03.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.386+0000 7f420f400640 1 --2- 172.21.15.27:0/1077059592 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f41f00047d0 0x7f4210143350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:03.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.386+0000 7f420f400640 1 -- 172.21.15.27:0/1077059592 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42101557e0 con 0x7f42101f69c0 2024-01-15T18:06:03.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.390+0000 7f420f400640 1 --2- 172.21.15.27:0/1077059592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42101f69c0 0x7f42100aff70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f41f801c840 tx=0x7f41f0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:03.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.390+0000 7f4207e00640 1 -- 172.21.15.27:0/1077059592 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f8004230 con 0x7f42101f69c0 2024-01-15T18:06:03.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.390+0000 7f4207e00640 1 -- 172.21.15.27:0/1077059592 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f8004390 con 0x7f42101f69c0 2024-01-15T18:06:03.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.390+0000 7f4216a00640 1 -- 172.21.15.27:0/1077059592 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4210155ac0 con 0x7f42101f69c0 2024-01-15T18:06:03.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.390+0000 7f4207e00640 1 -- 172.21.15.27:0/1077059592 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f80ad850 con 0x7f42101f69c0 2024-01-15T18:06:03.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.390+0000 7f4216a00640 1 -- 172.21.15.27:0/1077059592 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42100b04b0 con 0x7f42101f69c0 2024-01-15T18:06:03.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.390+0000 7f4216a00640 1 -- 172.21.15.27:0/1077059592 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f421004e850 con 0x7f42101f69c0 2024-01-15T18:06:03.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.390+0000 7f4207e00640 1 -- 172.21.15.27:0/1077059592 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f41f80b4040 con 0x7f42101f69c0 2024-01-15T18:06:03.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.390+0000 7f4207e00640 1 --2- 172.21.15.27:0/1077059592 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f41e80668f0 0x7f41e8068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:03.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.390+0000 7f4207e00640 1 -- 172.21.15.27:0/1077059592 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41f811e8d0 con 0x7f42101f69c0 2024-01-15T18:06:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.394+0000 7f420fe00640 1 --2- 172.21.15.27:0/1077059592 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f41e80668f0 0x7f41e8068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.394+0000 7f420fe00640 1 --2- 172.21.15.27:0/1077059592 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f41e80668f0 0x7f41e8068db0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4200001ec0 tx=0x7f4200007040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:03.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.394+0000 7f4207e00640 1 -- 172.21.15.27:0/1077059592 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f41f80ebd70 con 0x7f42101f69c0 2024-01-15T18:06:03.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:03.586+0000 7f4216a00640 1 -- 172.21.15.27:0/1077059592 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f42100b4f90 con 0x7f42101f69c0 2024-01-15T18:06:04.093 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-01-15T18:06:04.086+0000 7f2bd1000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-15T18:06:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.210+0000 7f4207e00640 1 -- 172.21.15.27:0/1077059592 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) v1 ==== 122+0+0 (secure 0 0 0) 0x7f41f80b20c0 con 0x7f42101f69c0 2024-01-15T18:06:04.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.210+0000 7f4216a00640 1 -- 172.21.15.27:0/1077059592 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f41e80668f0 msgr2=0x7f41e8068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:04.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.210+0000 7f4216a00640 1 --2- 172.21.15.27:0/1077059592 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f41e80668f0 0x7f41e8068db0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4200001ec0 tx=0x7f4200007040 comp rx=0 tx=0).stop 2024-01-15T18:06:04.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.210+0000 7f4216a00640 1 -- 172.21.15.27:0/1077059592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42101f69c0 msgr2=0x7f42100aff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:04.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.210+0000 7f4216a00640 1 --2- 172.21.15.27:0/1077059592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42101f69c0 0x7f42100aff70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f41f801c840 tx=0x7f41f0005800 comp rx=0 tx=0).stop 2024-01-15T18:06:04.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.210+0000 7f4216a00640 1 -- 172.21.15.27:0/1077059592 shutdown_connections 2024-01-15T18:06:04.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.210+0000 7f4216a00640 1 --2- 172.21.15.27:0/1077059592 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f41e80668f0 0x7f41e8068db0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:04.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.210+0000 7f4216a00640 1 --2- 172.21.15.27:0/1077059592 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4210159f80 0x7f421015a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:04.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.210+0000 7f4216a00640 1 --2- 172.21.15.27:0/1077059592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42101f69c0 0x7f42100aff70 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:04.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.210+0000 7f4216a00640 1 --2- 172.21.15.27:0/1077059592 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f41f00047d0 0x7f4210143350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:04.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.210+0000 7f4216a00640 1 -- 172.21.15.27:0/1077059592 >> 172.21.15.27:0/1077059592 conn(0x7f42100cb700 msgr2=0x7f42100b37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:04.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.210+0000 7f4216a00640 1 -- 172.21.15.27:0/1077059592 shutdown_connections 2024-01-15T18:06:04.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.210+0000 7f4216a00640 1 -- 172.21.15.27:0/1077059592 wait complete. 2024-01-15T18:06:04.240 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-15T18:06:04.240 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:06:04.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.378+0000 7f101be00640 1 Processor -- start 2024-01-15T18:06:04.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.378+0000 7f101be00640 1 -- start start 2024-01-15T18:06:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.378+0000 7f101be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f10140c92b0 0x7f101415d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.378+0000 7f101be00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101406c4a0 con 0x7f101415c130 2024-01-15T18:06:04.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.378+0000 7f101be00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f101415c660 con 0x7f10140701a0 2024-01-15T18:06:04.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.378+0000 7f101be00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f101415c940 con 0x7f10140c92b0 2024-01-15T18:06:04.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.382+0000 7f1018a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f101415c130 0x7f101406dd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49648/0 (socket says 172.21.15.27:49648) 2024-01-15T18:06:04.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.382+0000 7f1018a00640 1 -- 172.21.15.27:0/1409958846 learned_addr learned my addr 172.21.15.27:0/1409958846 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:06:04.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.382+0000 7f1019e00640 1 --2- 172.21.15.27:0/1409958846 >> v2:172.21.15.27:3300/0 conn(0x7f10140c92b0 0x7f101415d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.382+0000 7f1013e00640 1 -- 172.21.15.27:0/1409958846 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 351791730 0 0) 0x7f101406c4a0 con 0x7f101415c130 2024-01-15T18:06:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.382+0000 7f1013e00640 1 -- 172.21.15.27:0/1409958846 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ff4003530 con 0x7f101415c130 2024-01-15T18:06:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.382+0000 7f1019e00640 1 -- 172.21.15.27:0/1409958846 >> v1:172.21.15.93:6789/0 conn(0x7f10140701a0 legacy=0x7f1014070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.382+0000 7f1019e00640 1 -- 172.21.15.27:0/1409958846 >> v1:172.21.15.27:6789/0 conn(0x7f101415c130 legacy=0x7f101406dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:04.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.382+0000 7f1019e00640 1 -- 172.21.15.27:0/1409958846 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101415cc20 con 0x7f10140c92b0 2024-01-15T18:06:04.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.382+0000 7f1019e00640 1 --2- 172.21.15.27:0/1409958846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10140c92b0 0x7f101415d9e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f100c009d90 tx=0x7f100c01c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac7e7e1e171c8ccc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:04.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.382+0000 7f1013e00640 1 -- 172.21.15.27:0/1409958846 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100c0a5950 con 0x7f10140c92b0 2024-01-15T18:06:04.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.382+0000 7f1013e00640 1 -- 172.21.15.27:0/1409958846 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f100c0a5ab0 con 0x7f10140c92b0 2024-01-15T18:06:04.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.382+0000 7f1013e00640 1 -- 172.21.15.27:0/1409958846 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100c0a5e40 con 0x7f10140c92b0 2024-01-15T18:06:04.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.386+0000 7f101be00640 1 -- 172.21.15.27:0/1409958846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10140c92b0 msgr2=0x7f101415d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:04.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.386+0000 7f101be00640 1 --2- 172.21.15.27:0/1409958846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10140c92b0 0x7f101415d9e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f100c009d90 tx=0x7f100c01c890 comp rx=0 tx=0).stop 2024-01-15T18:06:04.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.386+0000 7f101be00640 1 -- 172.21.15.27:0/1409958846 shutdown_connections 2024-01-15T18:06:04.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.386+0000 7f101be00640 1 --2- 172.21.15.27:0/1409958846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10140c92b0 0x7f101415d9e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:04.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.386+0000 7f101be00640 1 -- 172.21.15.27:0/1409958846 >> 172.21.15.27:0/1409958846 conn(0x7f10140cec60 msgr2=0x7f10140cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:04.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.386+0000 7f101be00640 1 -- 172.21.15.27:0/1409958846 shutdown_connections 2024-01-15T18:06:04.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.386+0000 7f101be00640 1 -- 172.21.15.27:0/1409958846 wait complete. 2024-01-15T18:06:04.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.386+0000 7f101be00640 1 Processor -- start 2024-01-15T18:06:04.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.386+0000 7f101be00640 1 -- start start 2024-01-15T18:06:04.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.386+0000 7f101be00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f10140701a0 0x7f101414b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:04.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.386+0000 7f101be00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f10140c92b0 0x7f10140b0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.386+0000 7f101be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f101415c130 0x7f10140ae5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.386+0000 7f101be00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101414b6f0 con 0x7f101415c130 2024-01-15T18:06:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.386+0000 7f101be00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10140afbd0 con 0x7f10140c92b0 2024-01-15T18:06:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.386+0000 7f101be00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10140aefc0 con 0x7f10140701a0 2024-01-15T18:06:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.386+0000 7f1019e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f101415c130 0x7f10140ae5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.386+0000 7f1019e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f101415c130 0x7f10140ae5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56584/0 (socket says 172.21.15.27:56584) 2024-01-15T18:06:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.386+0000 7f1019e00640 1 -- 172.21.15.27:0/624269317 learned_addr learned my addr 172.21.15.27:0/624269317 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.386+0000 7f1018a00640 1 --2- 172.21.15.27:0/624269317 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f10140c92b0 0x7f10140b0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.390+0000 7f1019400640 1 --2- 172.21.15.27:0/624269317 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f10140701a0 0x7f101414b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.390+0000 7f1019e00640 1 -- 172.21.15.27:0/624269317 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f10140701a0 msgr2=0x7f101414b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.390+0000 7f1019e00640 1 --2- 172.21.15.27:0/624269317 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f10140701a0 0x7f101414b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:04.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.390+0000 7f1019e00640 1 -- 172.21.15.27:0/624269317 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f10140c92b0 msgr2=0x7f10140b0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:04.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.390+0000 7f1019e00640 1 --2- 172.21.15.27:0/624269317 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f10140c92b0 0x7f10140b0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:04.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.390+0000 7f1019e00640 1 -- 172.21.15.27:0/624269317 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f100c009a70 con 0x7f101415c130 2024-01-15T18:06:04.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.390+0000 7f1019400640 1 --2- 172.21.15.27:0/624269317 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f10140701a0 0x7f101414b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:06:04.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.390+0000 7f1019e00640 1 --2- 172.21.15.27:0/624269317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f101415c130 0x7f10140ae5e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f100c01a4c0 tx=0x7f100c019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:04.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.390+0000 7f1011600640 1 -- 172.21.15.27:0/624269317 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100c0add90 con 0x7f101415c130 2024-01-15T18:06:04.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.390+0000 7f1011600640 1 -- 172.21.15.27:0/624269317 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f100c009450 con 0x7f101415c130 2024-01-15T18:06:04.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.390+0000 7f101be00640 1 -- 172.21.15.27:0/624269317 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10140af2a0 con 0x7f101415c130 2024-01-15T18:06:04.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.390+0000 7f1011600640 1 -- 172.21.15.27:0/624269317 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100c0b9670 con 0x7f101415c130 2024-01-15T18:06:04.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.390+0000 7f101be00640 1 -- 172.21.15.27:0/624269317 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10140af860 con 0x7f101415c130 2024-01-15T18:06:04.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.390+0000 7f101be00640 1 -- 172.21.15.27:0/624269317 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10140afd40 con 0x7f101415c130 2024-01-15T18:06:04.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.390+0000 7f1011600640 1 -- 172.21.15.27:0/624269317 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f100c0039a0 con 0x7f101415c130 2024-01-15T18:06:04.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.390+0000 7f1011600640 1 --2- 172.21.15.27:0/624269317 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0fec066920 0x7f0fec068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:04.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.390+0000 7f1011600640 1 -- 172.21.15.27:0/624269317 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f100c11f9e0 con 0x7f101415c130 2024-01-15T18:06:04.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.394+0000 7f1019400640 1 --2- 172.21.15.27:0/624269317 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0fec066920 0x7f0fec068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:04.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.394+0000 7f1019400640 1 --2- 172.21.15.27:0/624269317 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0fec066920 0x7f0fec068de0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1004000f70 tx=0x7f1004009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:04.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.394+0000 7f1011600640 1 -- 172.21.15.27:0/624269317 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f100c0ecca0 con 0x7f101415c130 2024-01-15T18:06:04.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.582+0000 7f101be00640 1 -- 172.21.15.27:0/624269317 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f101404e850 con 0x7f101415c130 2024-01-15T18:06:04.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.586+0000 7f1011600640 1 -- 172.21.15.27:0/624269317 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4913 (secure 0 0 0) 0x7f100c0f02c0 con 0x7f101415c130 2024-01-15T18:06:04.591 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:06:04.591 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4910,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4204677709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4204677709},{"type":"v1","addr":"172.21.15.27:6835","nonce":4204677709}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4913,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/1766105879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":1766105879},{"type":"v1","addr":"172.21.15.93:6836","nonce":1766105879}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/2799226725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":2799226725},{"type":"v1","addr":"172.21.15.93:6837","nonce":2799226725}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:06:00.558126+0000","modified":"2024-01-15T18:06:03.589849+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5232":{"gid":5232,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2187944510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2187944510},{"type":"v1","addr":"172.21.15.27:6834","nonce":2187944510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-01-15T18:06:04.592 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-01-15T18:06:04.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.586+0000 7f101be00640 1 -- 172.21.15.27:0/624269317 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0fec066920 msgr2=0x7f0fec068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:04.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.586+0000 7f101be00640 1 --2- 172.21.15.27:0/624269317 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0fec066920 0x7f0fec068de0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1004000f70 tx=0x7f1004009040 comp rx=0 tx=0).stop 2024-01-15T18:06:04.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.586+0000 7f101be00640 1 -- 172.21.15.27:0/624269317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f101415c130 msgr2=0x7f10140ae5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:04.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.586+0000 7f101be00640 1 --2- 172.21.15.27:0/624269317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f101415c130 0x7f10140ae5e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f100c01a4c0 tx=0x7f100c019630 comp rx=0 tx=0).stop 2024-01-15T18:06:04.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.586+0000 7f101be00640 1 -- 172.21.15.27:0/624269317 shutdown_connections 2024-01-15T18:06:04.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.586+0000 7f101be00640 1 --2- 172.21.15.27:0/624269317 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0fec066920 0x7f0fec068de0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:04.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.586+0000 7f101be00640 1 --2- 172.21.15.27:0/624269317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f101415c130 0x7f10140ae5e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:04.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.586+0000 7f101be00640 1 --2- 172.21.15.27:0/624269317 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f10140c92b0 0x7f10140b0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:04.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.586+0000 7f101be00640 1 --2- 172.21.15.27:0/624269317 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f10140701a0 0x7f101414b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:04.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.586+0000 7f101be00640 1 -- 172.21.15.27:0/624269317 >> 172.21.15.27:0/624269317 conn(0x7f10140cec60 msgr2=0x7f1014154b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:04.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.586+0000 7f101be00640 1 -- 172.21.15.27:0/624269317 shutdown_connections 2024-01-15T18:06:04.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.586+0000 7f101be00640 1 -- 172.21.15.27:0/624269317 wait complete. 2024-01-15T18:06:04.613 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:06:04.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.746+0000 7fda35200640 1 Processor -- start 2024-01-15T18:06:04.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.746+0000 7fda35200640 1 -- start start 2024-01-15T18:06:04.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.750+0000 7fda35200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fda3006c010 0x7fda3006b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:04.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.750+0000 7fda35200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda3015caf0 0x7fda3015cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:04.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.750+0000 7fda35200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda300cc310 con 0x7fda300b3cd0 2024-01-15T18:06:04.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.750+0000 7fda35200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda300cc5d0 con 0x7fda3006c010 2024-01-15T18:06:04.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.750+0000 7fda35200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda300cc8b0 con 0x7fda3015caf0 2024-01-15T18:06:04.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.750+0000 7fda2de00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fda3006c010 0x7fda3006b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:04.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.750+0000 7fda2e800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda3015caf0 0x7fda3015cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:04.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.750+0000 7fda2de00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fda3006c010 0x7fda3006b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:49430/0 (socket says 172.21.15.27:49430) 2024-01-15T18:06:04.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.750+0000 7fda2de00640 1 -- 172.21.15.27:0/457611941 learned_addr learned my addr 172.21.15.27:0/457611941 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:04.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.750+0000 7fda2d400640 1 --1- 172.21.15.27:0/457611941 >> v1:172.21.15.93:6789/0 conn(0x7fda300b3cd0 0x7fda3006c3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:43658/0 (socket says 172.21.15.27:43658) 2024-01-15T18:06:04.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.750+0000 7fda2de00640 1 -- 172.21.15.27:0/457611941 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda3015caf0 msgr2=0x7fda3015cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:04.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.750+0000 7fda2de00640 1 --2- 172.21.15.27:0/457611941 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda3015caf0 0x7fda3015cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:04.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.750+0000 7fda2de00640 1 -- 172.21.15.27:0/457611941 >> v1:172.21.15.93:6789/0 conn(0x7fda300b3cd0 legacy=0x7fda3006c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:04.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.750+0000 7fda2de00640 1 -- 172.21.15.27:0/457611941 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda300ccb90 con 0x7fda3006c010 2024-01-15T18:06:04.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.750+0000 7fda2ca00640 1 -- 172.21.15.27:0/457611941 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658616976 0 0) 0x7fda300cc310 con 0x7fda300b3cd0 2024-01-15T18:06:04.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.750+0000 7fda2e800640 1 --2- 172.21.15.27:0/457611941 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda3015caf0 0x7fda3015cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:06:04.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.754+0000 7fda2de00640 1 --2- 172.21.15.27:0/457611941 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda3006c010 0x7fda3006b810 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fda20009870 tx=0x7fda2001c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13597e66638d52a1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:04.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.754+0000 7fda2ca00640 1 -- 172.21.15.27:0/457611941 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda200ab600 con 0x7fda3006c010 2024-01-15T18:06:04.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.754+0000 7fda2ca00640 1 -- 172.21.15.27:0/457611941 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda200a7050 con 0x7fda3006c010 2024-01-15T18:06:04.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.754+0000 7fda2ca00640 1 -- 172.21.15.27:0/457611941 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda200ab980 con 0x7fda3006c010 2024-01-15T18:06:04.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.754+0000 7fda35200640 1 -- 172.21.15.27:0/457611941 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda3006c010 msgr2=0x7fda3006b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:04.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.754+0000 7fda35200640 1 --2- 172.21.15.27:0/457611941 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda3006c010 0x7fda3006b810 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fda20009870 tx=0x7fda2001c480 comp rx=0 tx=0).stop 2024-01-15T18:06:04.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.754+0000 7fda35200640 1 -- 172.21.15.27:0/457611941 shutdown_connections 2024-01-15T18:06:04.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.754+0000 7fda35200640 1 --2- 172.21.15.27:0/457611941 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda3015caf0 0x7fda3015cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:04.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.754+0000 7fda35200640 1 --2- 172.21.15.27:0/457611941 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda3006c010 0x7fda3006b810 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:04.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.754+0000 7fda35200640 1 -- 172.21.15.27:0/457611941 >> 172.21.15.27:0/457611941 conn(0x7fda300d0e80 msgr2=0x7fda300d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:04.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.754+0000 7fda35200640 1 -- 172.21.15.27:0/457611941 shutdown_connections 2024-01-15T18:06:04.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.754+0000 7fda35200640 1 -- 172.21.15.27:0/457611941 wait complete. 2024-01-15T18:06:04.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.754+0000 7fda35200640 1 Processor -- start 2024-01-15T18:06:04.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.754+0000 7fda35200640 1 -- start start 2024-01-15T18:06:04.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.758+0000 7fda35200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda3006c010 0x7fda3015c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:04.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.758+0000 7fda35200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda300b3cd0 0x7fda30154d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:04.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.758+0000 7fda35200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3015caf0 0x7fda30150490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:04.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.758+0000 7fda35200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3015c8a0 con 0x7fda3015caf0 2024-01-15T18:06:04.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.758+0000 7fda35200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3015b910 con 0x7fda300b3cd0 2024-01-15T18:06:04.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.758+0000 7fda35200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3015bed0 con 0x7fda3006c010 2024-01-15T18:06:04.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.758+0000 7fda2e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3015caf0 0x7fda30150490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:04.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.758+0000 7fda2e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3015caf0 0x7fda30150490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56590/0 (socket says 172.21.15.27:56590) 2024-01-15T18:06:04.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.758+0000 7fda2e800640 1 -- 172.21.15.27:0/3357690155 learned_addr learned my addr 172.21.15.27:0/3357690155 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:04.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.758+0000 7fda2de00640 1 --2- 172.21.15.27:0/3357690155 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda3006c010 0x7fda3015c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:04.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.758+0000 7fda2d400640 1 --2- 172.21.15.27:0/3357690155 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda300b3cd0 0x7fda30154d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:04.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.758+0000 7fda2e800640 1 -- 172.21.15.27:0/3357690155 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda3006c010 msgr2=0x7fda3015c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:04.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.758+0000 7fda2e800640 1 --2- 172.21.15.27:0/3357690155 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda3006c010 0x7fda3015c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:04.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.758+0000 7fda2e800640 1 -- 172.21.15.27:0/3357690155 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda300b3cd0 msgr2=0x7fda30154d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:04.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.758+0000 7fda2e800640 1 --2- 172.21.15.27:0/3357690155 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda300b3cd0 0x7fda30154d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:04.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.758+0000 7fda2e800640 1 -- 172.21.15.27:0/3357690155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda28009a70 con 0x7fda3015caf0 2024-01-15T18:06:04.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.758+0000 7fda2e800640 1 --2- 172.21.15.27:0/3357690155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3015caf0 0x7fda30150490 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fda2800efc0 tx=0x7fda2800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:04.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.758+0000 7fda26000640 1 -- 172.21.15.27:0/3357690155 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28009cd0 con 0x7fda3015caf0 2024-01-15T18:06:04.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.758+0000 7fda26000640 1 -- 172.21.15.27:0/3357690155 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2800f040 con 0x7fda3015caf0 2024-01-15T18:06:04.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.758+0000 7fda26000640 1 -- 172.21.15.27:0/3357690155 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda280039a0 con 0x7fda3015caf0 2024-01-15T18:06:04.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.758+0000 7fda35200640 1 -- 172.21.15.27:0/3357690155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda200094d0 con 0x7fda3015caf0 2024-01-15T18:06:04.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.758+0000 7fda35200640 1 -- 172.21.15.27:0/3357690155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda301564d0 con 0x7fda3015caf0 2024-01-15T18:06:04.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.758+0000 7fda35200640 1 -- 172.21.15.27:0/3357690155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda3015c040 con 0x7fda3015caf0 2024-01-15T18:06:04.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.762+0000 7fda26000640 1 -- 172.21.15.27:0/3357690155 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fda28014480 con 0x7fda3015caf0 2024-01-15T18:06:04.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.762+0000 7fda26000640 1 --2- 172.21.15.27:0/3357690155 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fda080668f0 0x7fda08068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:04.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.762+0000 7fda26000640 1 -- 172.21.15.27:0/3357690155 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda28084f50 con 0x7fda3015caf0 2024-01-15T18:06:04.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.762+0000 7fda2de00640 1 --2- 172.21.15.27:0/3357690155 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fda080668f0 0x7fda08068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:04.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.762+0000 7fda2de00640 1 --2- 172.21.15.27:0/3357690155 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fda080668f0 0x7fda08068db0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fda200ad000 tx=0x7fda20008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:04.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.766+0000 7fda26000640 1 -- 172.21.15.27:0/3357690155 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fda280521a0 con 0x7fda3015caf0 2024-01-15T18:06:04.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.922+0000 7fda35200640 1 -- 172.21.15.27:0/3357690155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda3004e850 con 0x7fda3015caf0 2024-01-15T18:06:04.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.926+0000 7fda26000640 1 -- 172.21.15.27:0/3357690155 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16473 (secure 0 0 0) 0x7fda280557c0 con 0x7fda3015caf0 2024-01-15T18:06:04.932 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:06:04.932 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":34,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:06:02.169536+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:05:58.389641+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-15T18:05:59.555643+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:06:04.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.926+0000 7fda35200640 1 -- 172.21.15.27:0/3357690155 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fda080668f0 msgr2=0x7fda08068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:04.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.926+0000 7fda35200640 1 --2- 172.21.15.27:0/3357690155 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fda080668f0 0x7fda08068db0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fda200ad000 tx=0x7fda20008040 comp rx=0 tx=0).stop 2024-01-15T18:06:04.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.926+0000 7fda35200640 1 -- 172.21.15.27:0/3357690155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3015caf0 msgr2=0x7fda30150490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:04.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.926+0000 7fda35200640 1 --2- 172.21.15.27:0/3357690155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3015caf0 0x7fda30150490 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fda2800efc0 tx=0x7fda2800c490 comp rx=0 tx=0).stop 2024-01-15T18:06:04.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.926+0000 7fda35200640 1 -- 172.21.15.27:0/3357690155 shutdown_connections 2024-01-15T18:06:04.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.926+0000 7fda35200640 1 --2- 172.21.15.27:0/3357690155 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fda080668f0 0x7fda08068db0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:04.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.926+0000 7fda35200640 1 --2- 172.21.15.27:0/3357690155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda3015caf0 0x7fda30150490 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:04.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.926+0000 7fda35200640 1 --2- 172.21.15.27:0/3357690155 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda300b3cd0 0x7fda30154d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:04.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.926+0000 7fda35200640 1 --2- 172.21.15.27:0/3357690155 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda3006c010 0x7fda3015c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:04.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.926+0000 7fda35200640 1 -- 172.21.15.27:0/3357690155 >> 172.21.15.27:0/3357690155 conn(0x7fda300d0e80 msgr2=0x7fda30159dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:04.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.926+0000 7fda35200640 1 -- 172.21.15.27:0/3357690155 shutdown_connections 2024-01-15T18:06:04.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:04.926+0000 7fda35200640 1 -- 172.21.15.27:0/3357690155 wait complete. 2024-01-15T18:06:04.955 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-15T18:06:04.955 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:06:05.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.090+0000 7f066c400640 1 Processor -- start 2024-01-15T18:06:05.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.090+0000 7f066c400640 1 -- start start 2024-01-15T18:06:05.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.090+0000 7f066c400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f06640701a0 0x7f0664070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:05.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.090+0000 7f066c400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f06640c92b0 0x7f066406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:05.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.090+0000 7f066c400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066406c4a0 con 0x7f066406e240 2024-01-15T18:06:05.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.090+0000 7f066c400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0664070ac0 con 0x7f06640701a0 2024-01-15T18:06:05.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.090+0000 7f066c400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f066415c660 con 0x7f06640c92b0 2024-01-15T18:06:05.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.094+0000 7f0669a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f06640701a0 0x7f0664070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:05.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.094+0000 7f066a400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f066406e240 0x7f066415d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49660/0 (socket says 172.21.15.27:49660) 2024-01-15T18:06:05.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.094+0000 7f066a400640 1 -- 172.21.15.27:0/1920110349 learned_addr learned my addr 172.21.15.27:0/1920110349 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:06:05.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.094+0000 7f0669000640 1 --2- 172.21.15.27:0/1920110349 >> v2:172.21.15.93:3300/0 conn(0x7f06640c92b0 0x7f066406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:05.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.094+0000 7f0669a00640 1 -- 172.21.15.27:0/1920110349 >> v2:172.21.15.93:3300/0 conn(0x7f06640c92b0 msgr2=0x7f066406dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:05.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.094+0000 7f0669a00640 1 --2- 172.21.15.27:0/1920110349 >> v2:172.21.15.93:3300/0 conn(0x7f06640c92b0 0x7f066406dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:05.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.094+0000 7f0669a00640 1 -- 172.21.15.27:0/1920110349 >> v1:172.21.15.27:6789/0 conn(0x7f066406e240 legacy=0x7f066415d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:05.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.094+0000 7f0663e00640 1 -- 172.21.15.27:0/1920110349 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1053166362 0 0) 0x7f066406c4a0 con 0x7f066406e240 2024-01-15T18:06:05.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.094+0000 7f0669a00640 1 -- 172.21.15.27:0/1920110349 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066415c940 con 0x7f06640701a0 2024-01-15T18:06:05.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.094+0000 7f0669a00640 1 --2- 172.21.15.27:0/1920110349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06640701a0 0x7f0664070580 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0664067d80 tx=0x7f065401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f991e581ccb8827 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:05.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.094+0000 7f0663e00640 1 -- 172.21.15.27:0/1920110349 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06540a7070 con 0x7f06640701a0 2024-01-15T18:06:05.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.094+0000 7f0663e00640 1 -- 172.21.15.27:0/1920110349 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06540ab430 con 0x7f06640701a0 2024-01-15T18:06:05.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.094+0000 7f0663e00640 1 -- 172.21.15.27:0/1920110349 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06540ab7c0 con 0x7f06640701a0 2024-01-15T18:06:05.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.098+0000 7f066c400640 1 -- 172.21.15.27:0/1920110349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06640701a0 msgr2=0x7f0664070580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:05.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.098+0000 7f066c400640 1 --2- 172.21.15.27:0/1920110349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06640701a0 0x7f0664070580 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0664067d80 tx=0x7f065401c430 comp rx=0 tx=0).stop 2024-01-15T18:06:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.098+0000 7f066c400640 1 -- 172.21.15.27:0/1920110349 shutdown_connections 2024-01-15T18:06:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.098+0000 7f066c400640 1 --2- 172.21.15.27:0/1920110349 >> v2:172.21.15.93:3300/0 conn(0x7f06640c92b0 0x7f066406dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.098+0000 7f066c400640 1 --2- 172.21.15.27:0/1920110349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06640701a0 0x7f0664070580 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.098+0000 7f066c400640 1 -- 172.21.15.27:0/1920110349 >> 172.21.15.27:0/1920110349 conn(0x7f06640cec60 msgr2=0x7f06640cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.098+0000 7f066c400640 1 -- 172.21.15.27:0/1920110349 shutdown_connections 2024-01-15T18:06:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.098+0000 7f066c400640 1 -- 172.21.15.27:0/1920110349 wait complete. 2024-01-15T18:06:05.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.098+0000 7f066c400640 1 Processor -- start 2024-01-15T18:06:05.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.098+0000 7f066c400640 1 -- start start 2024-01-15T18:06:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.098+0000 7f066c400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f066406e240 0x7f06640aed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.098+0000 7f066c400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06640701a0 0x7f06640c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.098+0000 7f066c400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06640c92b0 0x7f06641581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.098+0000 7f066c400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06640c7020 con 0x7f06640c92b0 2024-01-15T18:06:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.098+0000 7f066c400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06640ad190 con 0x7f06640701a0 2024-01-15T18:06:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.098+0000 7f066c400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06640ad470 con 0x7f066406e240 2024-01-15T18:06:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.098+0000 7f066a400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06640c92b0 0x7f06641581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.098+0000 7f066a400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06640c92b0 0x7f06641581b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56606/0 (socket says 172.21.15.27:56606) 2024-01-15T18:06:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.098+0000 7f066a400640 1 -- 172.21.15.27:0/2075849338 learned_addr learned my addr 172.21.15.27:0/2075849338 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.102+0000 7f0669000640 1 --2- 172.21.15.27:0/2075849338 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06640701a0 0x7f06640c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.102+0000 7f0669a00640 1 --2- 172.21.15.27:0/2075849338 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f066406e240 0x7f06640aed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.102+0000 7f066a400640 1 -- 172.21.15.27:0/2075849338 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f066406e240 msgr2=0x7f06640aed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.102+0000 7f066a400640 1 --2- 172.21.15.27:0/2075849338 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f066406e240 0x7f06640aed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.102+0000 7f066a400640 1 -- 172.21.15.27:0/2075849338 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06640701a0 msgr2=0x7f06640c66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.102+0000 7f066a400640 1 --2- 172.21.15.27:0/2075849338 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06640701a0 0x7f06640c66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.102+0000 7f066a400640 1 -- 172.21.15.27:0/2075849338 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0654009420 con 0x7f06640c92b0 2024-01-15T18:06:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.102+0000 7f066a400640 1 --2- 172.21.15.27:0/2075849338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06640c92b0 0x7f06641581b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f065c001570 tx=0x7f065c00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:05.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.102+0000 7f0661600640 1 -- 172.21.15.27:0/2075849338 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065c00a640 con 0x7f06640c92b0 2024-01-15T18:06:05.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.102+0000 7f0661600640 1 -- 172.21.15.27:0/2075849338 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f065c00a7c0 con 0x7f06640c92b0 2024-01-15T18:06:05.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.102+0000 7f0661600640 1 -- 172.21.15.27:0/2075849338 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065c00ab70 con 0x7f06640c92b0 2024-01-15T18:06:05.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.102+0000 7f066c400640 1 -- 172.21.15.27:0/2075849338 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06640c7290 con 0x7f06640c92b0 2024-01-15T18:06:05.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.102+0000 7f066c400640 1 -- 172.21.15.27:0/2075849338 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06640ad750 con 0x7f06640c92b0 2024-01-15T18:06:05.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.102+0000 7f066c400640 1 -- 172.21.15.27:0/2075849338 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06640c74b0 con 0x7f06640c92b0 2024-01-15T18:06:05.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.102+0000 7f0661600640 1 -- 172.21.15.27:0/2075849338 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f065c00acd0 con 0x7f06640c92b0 2024-01-15T18:06:05.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.106+0000 7f0661600640 1 --2- 172.21.15.27:0/2075849338 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f063c0668f0 0x7f063c068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:05.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.106+0000 7f0661600640 1 -- 172.21.15.27:0/2075849338 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f065c083aa0 con 0x7f06640c92b0 2024-01-15T18:06:05.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.106+0000 7f0669a00640 1 --2- 172.21.15.27:0/2075849338 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f063c0668f0 0x7f063c068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:05.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.106+0000 7f0669a00640 1 --2- 172.21.15.27:0/2075849338 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f063c0668f0 0x7f063c068db0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f06540af000 tx=0x7f06540a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:05.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.110+0000 7f0661600640 1 -- 172.21.15.27:0/2075849338 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f065c050c40 con 0x7f06640c92b0 2024-01-15T18:06:05.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.266+0000 7f066c400640 1 -- 172.21.15.27:0/2075849338 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f06640cab70 con 0x7f063c0668f0 2024-01-15T18:06:05.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.274+0000 7f0661600640 1 -- 172.21.15.27:0/2075849338 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+280246 (secure 0 0 0) 0x7f06640cab70 con 0x7f063c0668f0 2024-01-15T18:06:05.284 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:06:05.285 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:06:05.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.278+0000 7f066c400640 1 -- 172.21.15.27:0/2075849338 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f063c0668f0 msgr2=0x7f063c068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:05.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.278+0000 7f066c400640 1 --2- 172.21.15.27:0/2075849338 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f063c0668f0 0x7f063c068db0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f06540af000 tx=0x7f06540a7000 comp rx=0 tx=0).stop 2024-01-15T18:06:05.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.278+0000 7f066c400640 1 -- 172.21.15.27:0/2075849338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06640c92b0 msgr2=0x7f06641581b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:05.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.278+0000 7f066c400640 1 --2- 172.21.15.27:0/2075849338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06640c92b0 0x7f06641581b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f065c001570 tx=0x7f065c00cca0 comp rx=0 tx=0).stop 2024-01-15T18:06:05.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.278+0000 7f066c400640 1 -- 172.21.15.27:0/2075849338 shutdown_connections 2024-01-15T18:06:05.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.278+0000 7f066c400640 1 --2- 172.21.15.27:0/2075849338 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f063c0668f0 0x7f063c068db0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:05.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.278+0000 7f066c400640 1 --2- 172.21.15.27:0/2075849338 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06640c92b0 0x7f06641581b0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:05.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.278+0000 7f066c400640 1 --2- 172.21.15.27:0/2075849338 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06640701a0 0x7f06640c66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:05.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.278+0000 7f066c400640 1 --2- 172.21.15.27:0/2075849338 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f066406e240 0x7f06640aed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:05.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.278+0000 7f066c400640 1 -- 172.21.15.27:0/2075849338 >> 172.21.15.27:0/2075849338 conn(0x7f06640cec60 msgr2=0x7f06640afa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:05.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.278+0000 7f066c400640 1 -- 172.21.15.27:0/2075849338 shutdown_connections 2024-01-15T18:06:05.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.278+0000 7f066c400640 1 -- 172.21.15.27:0/2075849338 wait complete. 2024-01-15T18:06:05.304 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-01-15T18:06:04.153664+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":134,"acting":134,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219968,"kb_used_data":3792,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754752,"statfs":{"total":773094113280,"available":772868866048,"internally_reserved":0,"allocated":3883008,"data_stored":2198294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.000451"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.171586+0000","last_change":"2024-01-15T18:06:00.171586+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.171586+0000","last_undegraded":"2024-01-15T18:06:00.171586+0000","last_fullsized":"2024-01-15T18:06:00.171586+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.179571+0000","last_change":"2024-01-15T18:06:00.179571+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.179571+0000","last_undegraded":"2024-01-15T18:06:00.179571+0000","last_fullsized":"2024-01-15T18:06:00.179571+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.165981+0000","last_change":"2024-01-15T18:06:00.165981+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.165981+0000","last_undegraded":"2024-01-15T18:06:00.165981+0000","last_fullsized":"2024-01-15T18:06:00.165981+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.172568+0000","last_change":"2024-01-15T18:06:00.172568+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.172568+0000","last_undegraded":"2024-01-15T18:06:00.172568+0000","last_fullsized":"2024-01-15T18:06:00.172568+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.180507+0000","last_change":"2024-01-15T18:06:00.180507+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.180507+0000","last_undegraded":"2024-01-15T18:06:00.180507+0000","last_fullsized":"2024-01-15T18:06:00.180507+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.165821+0000","last_change":"2024-01-15T18:06:00.165821+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.165821+0000","last_undegraded":"2024-01-15T18:06:00.165821+0000","last_fullsized":"2024-01-15T18:06:00.165821+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.165786+0000","last_change":"2024-01-15T18:06:00.165786+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.165786+0000","last_undegraded":"2024-01-15T18:06:00.165786+0000","last_fullsized":"2024-01-15T18:06:00.165786+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.172055+0000","last_change":"2024-01-15T18:06:00.172055+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.172055+0000","last_undegraded":"2024-01-15T18:06:00.172055+0000","last_fullsized":"2024-01-15T18:06:00.172055+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.172208+0000","last_change":"2024-01-15T18:06:00.172208+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.172208+0000","last_undegraded":"2024-01-15T18:06:00.172208+0000","last_fullsized":"2024-01-15T18:06:00.172208+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.172461+0000","last_change":"2024-01-15T18:06:00.172461+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.172461+0000","last_undegraded":"2024-01-15T18:06:00.172461+0000","last_fullsized":"2024-01-15T18:06:00.172461+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.179869+0000","last_change":"2024-01-15T18:06:00.179869+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.179869+0000","last_undegraded":"2024-01-15T18:06:00.179869+0000","last_fullsized":"2024-01-15T18:06:00.179869+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.169391+0000","last_change":"2024-01-15T18:06:00.169391+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.169391+0000","last_undegraded":"2024-01-15T18:06:00.169391+0000","last_fullsized":"2024-01-15T18:06:00.169391+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.173714+0000","last_change":"2024-01-15T18:06:00.173714+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.173714+0000","last_undegraded":"2024-01-15T18:06:00.173714+0000","last_fullsized":"2024-01-15T18:06:00.173714+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.167370+0000","last_change":"2024-01-15T18:06:00.167370+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.167370+0000","last_undegraded":"2024-01-15T18:06:00.167370+0000","last_fullsized":"2024-01-15T18:06:00.167370+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.166345+0000","last_change":"2024-01-15T18:06:00.166345+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.166345+0000","last_undegraded":"2024-01-15T18:06:00.166345+0000","last_fullsized":"2024-01-15T18:06:00.166345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.170414+0000","last_change":"2024-01-15T18:06:00.170414+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.170414+0000","last_undegraded":"2024-01-15T18:06:00.170414+0000","last_fullsized":"2024-01-15T18:06:00.170414+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.165628+0000","last_change":"2024-01-15T18:06:00.165628+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.165628+0000","last_undegraded":"2024-01-15T18:06:00.165628+0000","last_fullsized":"2024-01-15T18:06:00.165628+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.165370+0000","last_change":"2024-01-15T18:06:00.165370+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.165370+0000","last_undegraded":"2024-01-15T18:06:00.165370+0000","last_fullsized":"2024-01-15T18:06:00.165370+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.170608+0000","last_change":"2024-01-15T18:06:00.170608+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.170608+0000","last_undegraded":"2024-01-15T18:06:00.170608+0000","last_fullsized":"2024-01-15T18:06:00.170608+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-01-15T18:05:59.161092+0000","last_change":"2024-01-15T18:05:59.161092+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.161092+0000","last_undegraded":"2024-01-15T18:05:59.161092+0000","last_fullsized":"2024-01-15T18:05:59.161092+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.171119+0000","last_change":"2024-01-15T18:06:00.171119+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.171119+0000","last_undegraded":"2024-01-15T18:06:00.171119+0000","last_fullsized":"2024-01-15T18:06:00.171119+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:05:59.145608+0000","last_change":"2024-01-15T18:05:59.145608+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.145608+0000","last_undegraded":"2024-01-15T18:05:59.145608+0000","last_fullsized":"2024-01-15T18:05:59.145608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.164402+0000","last_change":"2024-01-15T18:06:00.164402+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.164402+0000","last_undegraded":"2024-01-15T18:06:00.164402+0000","last_fullsized":"2024-01-15T18:06:00.164402+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.180303+0000","last_change":"2024-01-15T18:06:00.180303+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.180303+0000","last_undegraded":"2024-01-15T18:06:00.180303+0000","last_fullsized":"2024-01-15T18:06:00.180303+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:38:07.317901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.165003+0000","last_change":"2024-01-15T18:06:00.165003+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.165003+0000","last_undegraded":"2024-01-15T18:06:00.165003+0000","last_fullsized":"2024-01-15T18:06:00.165003+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.168374+0000","last_change":"2024-01-15T18:06:00.168374+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.168374+0000","last_undegraded":"2024-01-15T18:06:00.168374+0000","last_fullsized":"2024-01-15T18:06:00.168374+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:31:30.665626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.174622+0000","last_change":"2024-01-15T18:06:00.174622+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.174622+0000","last_undegraded":"2024-01-15T18:06:00.174622+0000","last_fullsized":"2024-01-15T18:06:00.174622+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:39:25.962651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.164626+0000","last_change":"2024-01-15T18:06:00.164626+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.164626+0000","last_undegraded":"2024-01-15T18:06:00.164626+0000","last_fullsized":"2024-01-15T18:06:00.164626+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:48:52.588812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.164265+0000","last_change":"2024-01-15T18:06:00.164265+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.164265+0000","last_undegraded":"2024-01-15T18:06:00.164265+0000","last_fullsized":"2024-01-15T18:06:00.164265+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.171075+0000","last_change":"2024-01-15T18:06:00.171075+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.171075+0000","last_undegraded":"2024-01-15T18:06:00.171075+0000","last_fullsized":"2024-01-15T18:06:00.171075+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:40:56.123776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.164936+0000","last_change":"2024-01-15T18:06:00.164936+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.164936+0000","last_undegraded":"2024-01-15T18:06:00.164936+0000","last_fullsized":"2024-01-15T18:06:00.164936+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.173385+0000","last_change":"2024-01-15T18:06:00.173385+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.173385+0000","last_undegraded":"2024-01-15T18:06:00.173385+0000","last_fullsized":"2024-01-15T18:06:00.173385+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:35:19.522019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":4,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.183420+0000","last_change":"2024-01-15T18:06:00.171516+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.183420+0000","last_undegraded":"2024-01-15T18:06:00.183420+0000","last_fullsized":"2024-01-15T18:06:00.183420+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:31:04.425248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.172541+0000","last_change":"2024-01-15T18:06:00.172541+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.172541+0000","last_undegraded":"2024-01-15T18:06:00.172541+0000","last_fullsized":"2024-01-15T18:06:00.172541+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:05:59.145608+0000","last_change":"2024-01-15T18:05:59.145608+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.145608+0000","last_undegraded":"2024-01-15T18:05:59.145608+0000","last_fullsized":"2024-01-15T18:05:59.145608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.179315+0000","last_change":"2024-01-15T18:06:00.179315+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.179315+0000","last_undegraded":"2024-01-15T18:06:00.179315+0000","last_fullsized":"2024-01-15T18:06:00.179315+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-01-15T18:05:59.156144+0000","last_change":"2024-01-15T18:05:59.156144+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.156144+0000","last_undegraded":"2024-01-15T18:05:59.156144+0000","last_fullsized":"2024-01-15T18:05:59.156144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":4,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.179743+0000","last_change":"2024-01-15T18:06:00.171716+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.179743+0000","last_undegraded":"2024-01-15T18:06:00.179743+0000","last_fullsized":"2024-01-15T18:06:00.179743+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:26:09.652268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.165709+0000","last_change":"2024-01-15T18:06:00.165709+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.165709+0000","last_undegraded":"2024-01-15T18:06:00.165709+0000","last_fullsized":"2024-01-15T18:06:00.165709+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.164330+0000","last_change":"2024-01-15T18:06:00.164330+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.164330+0000","last_undegraded":"2024-01-15T18:06:00.164330+0000","last_fullsized":"2024-01-15T18:06:00.164330+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:03:20.544076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.170139+0000","last_change":"2024-01-15T18:06:00.170139+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.170139+0000","last_undegraded":"2024-01-15T18:06:00.170139+0000","last_fullsized":"2024-01-15T18:06:00.170139+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.162418+0000","last_change":"2024-01-15T18:06:00.162418+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.162418+0000","last_undegraded":"2024-01-15T18:06:00.162418+0000","last_fullsized":"2024-01-15T18:06:00.162418+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:56.736968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":4,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.183486+0000","last_change":"2024-01-15T18:06:00.170635+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.183486+0000","last_undegraded":"2024-01-15T18:06:00.183486+0000","last_fullsized":"2024-01-15T18:06:00.183486+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:56.736968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.168794+0000","last_change":"2024-01-15T18:06:00.168794+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.168794+0000","last_undegraded":"2024-01-15T18:06:00.168794+0000","last_fullsized":"2024-01-15T18:06:00.168794+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.179395+0000","last_change":"2024-01-15T18:06:00.179395+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.179395+0000","last_undegraded":"2024-01-15T18:06:00.179395+0000","last_fullsized":"2024-01-15T18:06:00.179395+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:51:46.033055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.164841+0000","last_change":"2024-01-15T18:06:00.164841+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.164841+0000","last_undegraded":"2024-01-15T18:06:00.164841+0000","last_fullsized":"2024-01-15T18:06:00.164841+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.162240+0000","last_change":"2024-01-15T18:06:00.162240+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.162240+0000","last_undegraded":"2024-01-15T18:06:00.162240+0000","last_fullsized":"2024-01-15T18:06:00.162240+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:08:08.490734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.166559+0000","last_change":"2024-01-15T18:06:00.166559+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.166559+0000","last_undegraded":"2024-01-15T18:06:00.166559+0000","last_fullsized":"2024-01-15T18:06:00.166559+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:11:04.167913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.164909+0000","last_change":"2024-01-15T18:06:00.164909+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.164909+0000","last_undegraded":"2024-01-15T18:06:00.164909+0000","last_fullsized":"2024-01-15T18:06:00.164909+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":8,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T18:06:00.176281+0000","last_change":"2024-01-15T18:06:00.176281+0000","last_active":"2024-01-15T18:06:00.176281+0000","last_peered":"2024-01-15T18:06:00.176281+0000","last_clean":"2024-01-15T18:06:00.176281+0000","last_became_active":"2024-01-15T18:06:00.174503+0000","last_became_peered":"2024-01-15T18:06:00.174503+0000","last_unstale":"2024-01-15T18:06:00.176281+0000","last_undegraded":"2024-01-15T18:06:00.176281+0000","last_fullsized":"2024-01-15T18:06:00.176281+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:10:04.757709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.168628+0000","last_change":"2024-01-15T18:06:00.168628+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.168628+0000","last_undegraded":"2024-01-15T18:06:00.168628+0000","last_fullsized":"2024-01-15T18:06:00.168628+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.164364+0000","last_change":"2024-01-15T18:06:00.164364+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.164364+0000","last_undegraded":"2024-01-15T18:06:00.164364+0000","last_fullsized":"2024-01-15T18:06:00.164364+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:59:22.476704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.168573+0000","last_change":"2024-01-15T18:06:00.168573+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.168573+0000","last_undegraded":"2024-01-15T18:06:00.168573+0000","last_fullsized":"2024-01-15T18:06:00.168573+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:05:59.145608+0000","last_change":"2024-01-15T18:05:59.145608+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.145608+0000","last_undegraded":"2024-01-15T18:05:59.145608+0000","last_fullsized":"2024-01-15T18:05:59.145608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.168105+0000","last_change":"2024-01-15T18:06:00.168105+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.168105+0000","last_undegraded":"2024-01-15T18:06:00.168105+0000","last_fullsized":"2024-01-15T18:06:00.168105+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-01-15T18:05:59.161004+0000","last_change":"2024-01-15T18:05:59.161004+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.161004+0000","last_undegraded":"2024-01-15T18:05:59.161004+0000","last_fullsized":"2024-01-15T18:05:59.161004+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.168020+0000","last_change":"2024-01-15T18:06:00.168020+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.168020+0000","last_undegraded":"2024-01-15T18:06:00.168020+0000","last_fullsized":"2024-01-15T18:06:00.168020+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.165635+0000","last_change":"2024-01-15T18:06:00.165635+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.165635+0000","last_undegraded":"2024-01-15T18:06:00.165635+0000","last_fullsized":"2024-01-15T18:06:00.165635+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:31:30.665626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-01-15T18:05:59.155857+0000","last_change":"2024-01-15T18:05:59.155857+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.155857+0000","last_undegraded":"2024-01-15T18:05:59.155857+0000","last_fullsized":"2024-01-15T18:05:59.155857+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.168806+0000","last_change":"2024-01-15T18:06:00.168806+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.168806+0000","last_undegraded":"2024-01-15T18:06:00.168806+0000","last_fullsized":"2024-01-15T18:06:00.168806+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":8,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T18:06:00.176082+0000","last_change":"2024-01-15T18:06:00.176082+0000","last_active":"2024-01-15T18:06:00.176082+0000","last_peered":"2024-01-15T18:06:00.176082+0000","last_clean":"2024-01-15T18:06:00.176082+0000","last_became_active":"2024-01-15T18:06:00.175734+0000","last_became_peered":"2024-01-15T18:06:00.175734+0000","last_unstale":"2024-01-15T18:06:00.176082+0000","last_undegraded":"2024-01-15T18:06:00.176082+0000","last_fullsized":"2024-01-15T18:06:00.176082+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:12.325308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:05:59.145608+0000","last_change":"2024-01-15T18:05:59.145608+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.145608+0000","last_undegraded":"2024-01-15T18:05:59.145608+0000","last_fullsized":"2024-01-15T18:05:59.145608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.162200+0000","last_change":"2024-01-15T18:06:00.162200+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.162200+0000","last_undegraded":"2024-01-15T18:06:00.162200+0000","last_fullsized":"2024-01-15T18:06:00.162200+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:46:52.165002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.172639+0000","last_change":"2024-01-15T18:06:00.172639+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.172639+0000","last_undegraded":"2024-01-15T18:06:00.172639+0000","last_fullsized":"2024-01-15T18:06:00.172639+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:05:59.145608+0000","last_change":"2024-01-15T18:05:59.145608+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.145608+0000","last_undegraded":"2024-01-15T18:05:59.145608+0000","last_fullsized":"2024-01-15T18:05:59.145608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":8,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T18:06:00.176729+0000","last_change":"2024-01-15T18:06:00.176729+0000","last_active":"2024-01-15T18:06:00.176729+0000","last_peered":"2024-01-15T18:06:00.176729+0000","last_clean":"2024-01-15T18:06:00.176729+0000","last_became_active":"2024-01-15T18:06:00.175928+0000","last_became_peered":"2024-01-15T18:06:00.175928+0000","last_unstale":"2024-01-15T18:06:00.176729+0000","last_undegraded":"2024-01-15T18:06:00.176729+0000","last_fullsized":"2024-01-15T18:06:00.176729+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:59:02.797271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.170431+0000","last_change":"2024-01-15T18:06:00.170431+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.170431+0000","last_undegraded":"2024-01-15T18:06:00.170431+0000","last_fullsized":"2024-01-15T18:06:00.170431+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":8,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T18:06:00.177964+0000","last_change":"2024-01-15T18:06:00.178477+0000","last_active":"2024-01-15T18:06:00.177964+0000","last_peered":"2024-01-15T18:06:00.177964+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"2024-01-15T18:06:00.176994+0000","last_became_peered":"2024-01-15T18:06:00.176994+0000","last_unstale":"2024-01-15T18:06:00.177964+0000","last_undegraded":"2024-01-15T18:06:00.177964+0000","last_fullsized":"2024-01-15T18:06:00.177964+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:40:09.517509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.166107+0000","last_change":"2024-01-15T18:06:00.166107+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.166107+0000","last_undegraded":"2024-01-15T18:06:00.166107+0000","last_fullsized":"2024-01-15T18:06:00.166107+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":8,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T18:06:00.184467+0000","last_change":"2024-01-15T18:06:00.184467+0000","last_active":"2024-01-15T18:06:00.184467+0000","last_peered":"2024-01-15T18:06:00.184467+0000","last_clean":"2024-01-15T18:06:00.184467+0000","last_became_active":"2024-01-15T18:06:00.184044+0000","last_became_peered":"2024-01-15T18:06:00.184044+0000","last_unstale":"2024-01-15T18:06:00.184467+0000","last_undegraded":"2024-01-15T18:06:00.184467+0000","last_fullsized":"2024-01-15T18:06:00.184467+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:12.325308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-01-15T18:05:59.161012+0000","last_change":"2024-01-15T18:05:59.161012+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.161012+0000","last_undegraded":"2024-01-15T18:05:59.161012+0000","last_fullsized":"2024-01-15T18:05:59.161012+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":172,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-15T18:05:54.653336+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:05:54.653336+0000","last_peered":"2024-01-15T18:05:54.653336+0000","last_clean":"2024-01-15T18:05:54.653336+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:05:54.653336+0000","last_undegraded":"2024-01-15T18:05:54.653336+0000","last_fullsized":"2024-01-15T18:05:54.653336+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":78,"acting":78,"num_store_stats":5},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":54,"acting":54,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509458,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27932,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343908,"statfs":{"total":96636764160,"available":96608161792,"internally_reserved":0,"allocated":933888,"data_stored":719017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.84999999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.90400000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.83299999999999996}]},{"osd":3,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.73699999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.61499999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.70499999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.75600000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509458,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":339968,"data_stored":128649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.59299999999999997}]},{"osd":1,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":2.4220000000000002,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.391,"5min":1.391,"15min":1.391},"last":0.745}]},{"osd":2,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.8700000000000001,"5min":2.8700000000000001,"15min":2.8700000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":0.52400000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":2.012,"5min":2.012,"15min":2.012},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.532,"5min":2.532,"15min":2.532},"last":0.81799999999999995}]},{"osd":4,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.879},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.68600000000000005}]},{"osd":5,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.91400000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.85399999999999998}]}]},{"osd":4,"up_from":15,"seq":64424509458,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":339968,"data_stored":128649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.64000000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.88}]},{"osd":2,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.66000000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.77000000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.74399999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.92100000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.90600000000000003}]}]},{"osd":3,"up_from":15,"seq":64424509457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":339968,"data_stored":128649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.42099999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.47199999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.56399999999999995}]},{"osd":4,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.76900000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.61699999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.67000000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.67900000000000005}]}]},{"osd":2,"up_from":15,"seq":64424509457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":339968,"data_stored":128649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.47799999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.42699999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.56899999999999995}]},{"osd":4,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.466,"5min":1.466,"15min":1.466},"last":0.77400000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.90400000000000003}]},{"osd":6,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.516,"5min":1.516,"15min":1.516},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.71199999999999997}]},{"osd":7,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.6279999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.81200000000000006}]}]},{"osd":1,"up_from":15,"seq":64424509459,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":339968,"data_stored":128649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.317}]},{"osd":2,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":4.2450000000000001,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.435}]},{"osd":3,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.917,"5min":1.917,"15min":1.917},"last":0.54300000000000004}]},{"osd":4,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.73499999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.56399999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.69699999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.753},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.66500000000000004}]}]},{"osd":5,"up_from":14,"seq":60129542162,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":327680,"data_stored":122832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":2.802,"5min":2.802,"15min":2.802},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.68600000000000005}]},{"osd":1,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.54100000000000004}]},{"osd":2,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":0.57099999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":2.3849999999999998,"5min":2.3849999999999998,"15min":2.3849999999999998},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":0.55400000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.64100000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.621}]},{"osd":7,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.67000000000000004}]}]},{"osd":0,"up_from":14,"seq":60129542162,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27920,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343920,"statfs":{"total":96636764160,"available":96608174080,"internally_reserved":0,"allocated":921600,"data_stored":713200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.34399999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.495},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.51600000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.325,"5min":1.325,"15min":1.325},"last":0.53900000000000003}]},{"osd":4,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.63700000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:05:43 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.9079999999999999,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":0.70799999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.764,"5min":1.764,"15min":1.764},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.806,"5min":1.806,"15min":1.806},"last":0.72499999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.722,"5min":1.722,"15min":1.722},"last":0.77900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:06:05.309 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:06:05.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.482+0000 7f7ec6600640 1 Processor -- start 2024-01-15T18:06:05.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.482+0000 7f7ec6600640 1 -- start start 2024-01-15T18:06:05.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.486+0000 7f7ec6600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ec00afe70 0x7f7ec00cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:05.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.486+0000 7f7ec6600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7ec00cb7b0 0x7f7ec00cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:05.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.486+0000 7f7ec6600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec00d05b0 con 0x7f7ec0160b10 2024-01-15T18:06:05.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.486+0000 7f7ec6600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ec00ce7d0 con 0x7f7ec00cb7b0 2024-01-15T18:06:05.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.486+0000 7f7ec6600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec00cfc80 con 0x7f7ec00afe70 2024-01-15T18:06:05.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.486+0000 7f7ebfe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7ec00cb7b0 0x7f7ec00cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:05.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.486+0000 7f7ebfe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7ec00cb7b0 0x7f7ec00cbc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56612/0 (socket says 172.21.15.27:56612) 2024-01-15T18:06:05.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.486+0000 7f7ebfe00640 1 -- 172.21.15.27:0/3588257092 learned_addr learned my addr 172.21.15.27:0/3588257092 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:05.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.486+0000 7f7ebea00640 1 --2- 172.21.15.27:0/3588257092 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ec00afe70 0x7f7ec00cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:05.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.486+0000 7f7ebfe00640 1 -- 172.21.15.27:0/3588257092 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ec00afe70 msgr2=0x7f7ec00cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:05.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.486+0000 7f7ebfe00640 1 --2- 172.21.15.27:0/3588257092 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ec00afe70 0x7f7ec00cdeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:05.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.486+0000 7f7ebfe00640 1 -- 172.21.15.27:0/3588257092 >> v1:172.21.15.93:6789/0 conn(0x7f7ec0160b10 legacy=0x7f7ec00cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:05.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.486+0000 7f7ebfe00640 1 -- 172.21.15.27:0/3588257092 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec00cff60 con 0x7f7ec00cb7b0 2024-01-15T18:06:05.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.486+0000 7f7ebea00640 1 --2- 172.21.15.27:0/3588257092 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ec00afe70 0x7f7ec00cdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:06:05.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.490+0000 7f7ebfe00640 1 --2- 172.21.15.27:0/3588257092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec00cb7b0 0x7f7ec00cbc40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8009e50 tx=0x7f7eb801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beb4286350c16811 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:05.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.490+0000 7f7ebe000640 1 -- 172.21.15.27:0/3588257092 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb80a5720 con 0x7f7ec00cb7b0 2024-01-15T18:06:05.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.490+0000 7f7ebe000640 1 -- 172.21.15.27:0/3588257092 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb80a5880 con 0x7f7ec00cb7b0 2024-01-15T18:06:05.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.490+0000 7f7ebe000640 1 -- 172.21.15.27:0/3588257092 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb80a5c10 con 0x7f7ec00cb7b0 2024-01-15T18:06:05.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.490+0000 7f7ec6600640 1 -- 172.21.15.27:0/3588257092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec00cb7b0 msgr2=0x7f7ec00cbc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:05.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.490+0000 7f7ec6600640 1 --2- 172.21.15.27:0/3588257092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec00cb7b0 0x7f7ec00cbc40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8009e50 tx=0x7f7eb801c950 comp rx=0 tx=0).stop 2024-01-15T18:06:05.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.490+0000 7f7ec6600640 1 -- 172.21.15.27:0/3588257092 shutdown_connections 2024-01-15T18:06:05.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.490+0000 7f7ec6600640 1 --2- 172.21.15.27:0/3588257092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec00cb7b0 0x7f7ec00cbc40 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:05.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.490+0000 7f7ec6600640 1 --2- 172.21.15.27:0/3588257092 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ec00afe70 0x7f7ec00cdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:05.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.490+0000 7f7ec6600640 1 -- 172.21.15.27:0/3588257092 >> 172.21.15.27:0/3588257092 conn(0x7f7ec00d0f20 msgr2=0x7f7ec00d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:05.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.490+0000 7f7ec6600640 1 -- 172.21.15.27:0/3588257092 shutdown_connections 2024-01-15T18:06:05.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.490+0000 7f7ec6600640 1 -- 172.21.15.27:0/3588257092 wait complete. 2024-01-15T18:06:05.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.490+0000 7f7ec6600640 1 Processor -- start 2024-01-15T18:06:05.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.490+0000 7f7ec6600640 1 -- start start 2024-01-15T18:06:05.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.494+0000 7f7ec6600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ec00afe70 0x7f7ec00c77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:05.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.494+0000 7f7ec6600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ec00cb7b0 0x7f7ec0150ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:05.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.494+0000 7f7ec6600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec0160b10 0x7f7ec006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:05.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.494+0000 7f7ec6600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec0154010 con 0x7f7ec0160b10 2024-01-15T18:06:05.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.494+0000 7f7ec6600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec0153440 con 0x7f7ec00afe70 2024-01-15T18:06:05.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.494+0000 7f7ec6600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec01536e0 con 0x7f7ec00cb7b0 2024-01-15T18:06:05.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.494+0000 7f7ebfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec0160b10 0x7f7ec006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:05.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.494+0000 7f7ebfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec0160b10 0x7f7ec006e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56622/0 (socket says 172.21.15.27:56622) 2024-01-15T18:06:05.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.494+0000 7f7ebfe00640 1 -- 172.21.15.27:0/925913825 learned_addr learned my addr 172.21.15.27:0/925913825 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:05.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.494+0000 7f7ebea00640 1 --2- 172.21.15.27:0/925913825 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ec00cb7b0 0x7f7ec0150ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:05.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.494+0000 7f7ebf400640 1 --2- 172.21.15.27:0/925913825 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ec00afe70 0x7f7ec00c77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:05.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.494+0000 7f7ebfe00640 1 -- 172.21.15.27:0/925913825 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ec00cb7b0 msgr2=0x7f7ec0150ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:05.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.494+0000 7f7ebfe00640 1 --2- 172.21.15.27:0/925913825 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ec00cb7b0 0x7f7ec0150ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:05.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.494+0000 7f7ebfe00640 1 -- 172.21.15.27:0/925913825 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ec00afe70 msgr2=0x7f7ec00c77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:05.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.494+0000 7f7ebfe00640 1 --2- 172.21.15.27:0/925913825 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ec00afe70 0x7f7ec00c77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:05.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.494+0000 7f7ebfe00640 1 -- 172.21.15.27:0/925913825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb8009b30 con 0x7f7ec0160b10 2024-01-15T18:06:05.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.494+0000 7f7ebfe00640 1 --2- 172.21.15.27:0/925913825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec0160b10 0x7f7ec006e470 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f7eb80af040 tx=0x7f7eb80196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:05.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.494+0000 7f7eb7400640 1 -- 172.21.15.27:0/925913825 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb80b8670 con 0x7f7ec0160b10 2024-01-15T18:06:05.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.494+0000 7f7eb7400640 1 -- 172.21.15.27:0/925913825 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb80b87d0 con 0x7f7ec0160b10 2024-01-15T18:06:05.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.494+0000 7f7eb7400640 1 -- 172.21.15.27:0/925913825 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb80b8b80 con 0x7f7ec0160b10 2024-01-15T18:06:05.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.494+0000 7f7ec6600640 1 -- 172.21.15.27:0/925913825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec01539c0 con 0x7f7ec0160b10 2024-01-15T18:06:05.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.494+0000 7f7ec6600640 1 -- 172.21.15.27:0/925913825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec014e5a0 con 0x7f7ec0160b10 2024-01-15T18:06:05.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.494+0000 7f7ec6600640 1 -- 172.21.15.27:0/925913825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec014e850 con 0x7f7ec0160b10 2024-01-15T18:06:05.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.494+0000 7f7eb7400640 1 -- 172.21.15.27:0/925913825 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f7eb80b8ce0 con 0x7f7ec0160b10 2024-01-15T18:06:05.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.498+0000 7f7eb7400640 1 --2- 172.21.15.27:0/925913825 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7e980668f0 0x7f7e98068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:05.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.498+0000 7f7eb7400640 1 -- 172.21.15.27:0/925913825 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7eb811f9b0 con 0x7f7ec0160b10 2024-01-15T18:06:05.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.498+0000 7f7ebf400640 1 --2- 172.21.15.27:0/925913825 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7e980668f0 0x7f7e98068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:05.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.498+0000 7f7ebf400640 1 --2- 172.21.15.27:0/925913825 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7e980668f0 0x7f7e98068db0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7eb0000f70 tx=0x7f7eb0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:05.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.502+0000 7f7eb7400640 1 -- 172.21.15.27:0/925913825 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7eb80ecd20 con 0x7f7ec0160b10 2024-01-15T18:06:05.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.654+0000 7f7ec6600640 1 -- 172.21.15.27:0/925913825 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7ec00b6c80 con 0x7f7e980668f0 2024-01-15T18:06:05.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.662+0000 7f7eb7400640 1 -- 172.21.15.27:0/925913825 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+280246 (secure 0 0 0) 0x7f7ec00b6c80 con 0x7f7e980668f0 2024-01-15T18:06:05.670 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:06:05.672 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:06:05.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.662+0000 7f7ec6600640 1 -- 172.21.15.27:0/925913825 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7e980668f0 msgr2=0x7f7e98068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:05.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.662+0000 7f7ec6600640 1 --2- 172.21.15.27:0/925913825 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7e980668f0 0x7f7e98068db0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7eb0000f70 tx=0x7f7eb0009040 comp rx=0 tx=0).stop 2024-01-15T18:06:05.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.662+0000 7f7ec6600640 1 -- 172.21.15.27:0/925913825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec0160b10 msgr2=0x7f7ec006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:05.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.662+0000 7f7ec6600640 1 --2- 172.21.15.27:0/925913825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec0160b10 0x7f7ec006e470 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f7eb80af040 tx=0x7f7eb80196f0 comp rx=0 tx=0).stop 2024-01-15T18:06:05.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.666+0000 7f7ec6600640 1 -- 172.21.15.27:0/925913825 shutdown_connections 2024-01-15T18:06:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.666+0000 7f7ec6600640 1 --2- 172.21.15.27:0/925913825 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7e980668f0 0x7f7e98068db0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.666+0000 7f7ec6600640 1 --2- 172.21.15.27:0/925913825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec0160b10 0x7f7ec006e470 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.666+0000 7f7ec6600640 1 --2- 172.21.15.27:0/925913825 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ec00cb7b0 0x7f7ec0150ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.666+0000 7f7ec6600640 1 --2- 172.21.15.27:0/925913825 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ec00afe70 0x7f7ec00c77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.666+0000 7f7ec6600640 1 -- 172.21.15.27:0/925913825 >> 172.21.15.27:0/925913825 conn(0x7f7ec00d0f20 msgr2=0x7f7ec006fcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.666+0000 7f7ec6600640 1 -- 172.21.15.27:0/925913825 shutdown_connections 2024-01-15T18:06:05.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.666+0000 7f7ec6600640 1 -- 172.21.15.27:0/925913825 wait complete. 2024-01-15T18:06:05.691 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-01-15T18:06:04.153664+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":134,"acting":134,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219968,"kb_used_data":3792,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754752,"statfs":{"total":773094113280,"available":772868866048,"internally_reserved":0,"allocated":3883008,"data_stored":2198294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.000451"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.171586+0000","last_change":"2024-01-15T18:06:00.171586+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.171586+0000","last_undegraded":"2024-01-15T18:06:00.171586+0000","last_fullsized":"2024-01-15T18:06:00.171586+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.179571+0000","last_change":"2024-01-15T18:06:00.179571+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.179571+0000","last_undegraded":"2024-01-15T18:06:00.179571+0000","last_fullsized":"2024-01-15T18:06:00.179571+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.165981+0000","last_change":"2024-01-15T18:06:00.165981+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.165981+0000","last_undegraded":"2024-01-15T18:06:00.165981+0000","last_fullsized":"2024-01-15T18:06:00.165981+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.172568+0000","last_change":"2024-01-15T18:06:00.172568+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.172568+0000","last_undegraded":"2024-01-15T18:06:00.172568+0000","last_fullsized":"2024-01-15T18:06:00.172568+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.180507+0000","last_change":"2024-01-15T18:06:00.180507+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.180507+0000","last_undegraded":"2024-01-15T18:06:00.180507+0000","last_fullsized":"2024-01-15T18:06:00.180507+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.165821+0000","last_change":"2024-01-15T18:06:00.165821+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.165821+0000","last_undegraded":"2024-01-15T18:06:00.165821+0000","last_fullsized":"2024-01-15T18:06:00.165821+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.165786+0000","last_change":"2024-01-15T18:06:00.165786+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.165786+0000","last_undegraded":"2024-01-15T18:06:00.165786+0000","last_fullsized":"2024-01-15T18:06:00.165786+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.172055+0000","last_change":"2024-01-15T18:06:00.172055+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.172055+0000","last_undegraded":"2024-01-15T18:06:00.172055+0000","last_fullsized":"2024-01-15T18:06:00.172055+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.172208+0000","last_change":"2024-01-15T18:06:00.172208+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.172208+0000","last_undegraded":"2024-01-15T18:06:00.172208+0000","last_fullsized":"2024-01-15T18:06:00.172208+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.172461+0000","last_change":"2024-01-15T18:06:00.172461+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.172461+0000","last_undegraded":"2024-01-15T18:06:00.172461+0000","last_fullsized":"2024-01-15T18:06:00.172461+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.179869+0000","last_change":"2024-01-15T18:06:00.179869+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.179869+0000","last_undegraded":"2024-01-15T18:06:00.179869+0000","last_fullsized":"2024-01-15T18:06:00.179869+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.169391+0000","last_change":"2024-01-15T18:06:00.169391+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.169391+0000","last_undegraded":"2024-01-15T18:06:00.169391+0000","last_fullsized":"2024-01-15T18:06:00.169391+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.173714+0000","last_change":"2024-01-15T18:06:00.173714+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.173714+0000","last_undegraded":"2024-01-15T18:06:00.173714+0000","last_fullsized":"2024-01-15T18:06:00.173714+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.167370+0000","last_change":"2024-01-15T18:06:00.167370+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.167370+0000","last_undegraded":"2024-01-15T18:06:00.167370+0000","last_fullsized":"2024-01-15T18:06:00.167370+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.166345+0000","last_change":"2024-01-15T18:06:00.166345+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.166345+0000","last_undegraded":"2024-01-15T18:06:00.166345+0000","last_fullsized":"2024-01-15T18:06:00.166345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.170414+0000","last_change":"2024-01-15T18:06:00.170414+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.170414+0000","last_undegraded":"2024-01-15T18:06:00.170414+0000","last_fullsized":"2024-01-15T18:06:00.170414+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.165628+0000","last_change":"2024-01-15T18:06:00.165628+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.165628+0000","last_undegraded":"2024-01-15T18:06:00.165628+0000","last_fullsized":"2024-01-15T18:06:00.165628+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.165370+0000","last_change":"2024-01-15T18:06:00.165370+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.165370+0000","last_undegraded":"2024-01-15T18:06:00.165370+0000","last_fullsized":"2024-01-15T18:06:00.165370+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.170608+0000","last_change":"2024-01-15T18:06:00.170608+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.170608+0000","last_undegraded":"2024-01-15T18:06:00.170608+0000","last_fullsized":"2024-01-15T18:06:00.170608+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-01-15T18:05:59.161092+0000","last_change":"2024-01-15T18:05:59.161092+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.161092+0000","last_undegraded":"2024-01-15T18:05:59.161092+0000","last_fullsized":"2024-01-15T18:05:59.161092+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.171119+0000","last_change":"2024-01-15T18:06:00.171119+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.171119+0000","last_undegraded":"2024-01-15T18:06:00.171119+0000","last_fullsized":"2024-01-15T18:06:00.171119+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:05:59.145608+0000","last_change":"2024-01-15T18:05:59.145608+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.145608+0000","last_undegraded":"2024-01-15T18:05:59.145608+0000","last_fullsized":"2024-01-15T18:05:59.145608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.164402+0000","last_change":"2024-01-15T18:06:00.164402+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.164402+0000","last_undegraded":"2024-01-15T18:06:00.164402+0000","last_fullsized":"2024-01-15T18:06:00.164402+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.180303+0000","last_change":"2024-01-15T18:06:00.180303+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.180303+0000","last_undegraded":"2024-01-15T18:06:00.180303+0000","last_fullsized":"2024-01-15T18:06:00.180303+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:38:07.317901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.165003+0000","last_change":"2024-01-15T18:06:00.165003+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.165003+0000","last_undegraded":"2024-01-15T18:06:00.165003+0000","last_fullsized":"2024-01-15T18:06:00.165003+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.168374+0000","last_change":"2024-01-15T18:06:00.168374+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.168374+0000","last_undegraded":"2024-01-15T18:06:00.168374+0000","last_fullsized":"2024-01-15T18:06:00.168374+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:31:30.665626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.174622+0000","last_change":"2024-01-15T18:06:00.174622+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.174622+0000","last_undegraded":"2024-01-15T18:06:00.174622+0000","last_fullsized":"2024-01-15T18:06:00.174622+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:39:25.962651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.164626+0000","last_change":"2024-01-15T18:06:00.164626+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.164626+0000","last_undegraded":"2024-01-15T18:06:00.164626+0000","last_fullsized":"2024-01-15T18:06:00.164626+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:48:52.588812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.164265+0000","last_change":"2024-01-15T18:06:00.164265+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.164265+0000","last_undegraded":"2024-01-15T18:06:00.164265+0000","last_fullsized":"2024-01-15T18:06:00.164265+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.171075+0000","last_change":"2024-01-15T18:06:00.171075+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.171075+0000","last_undegraded":"2024-01-15T18:06:00.171075+0000","last_fullsized":"2024-01-15T18:06:00.171075+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:40:56.123776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.164936+0000","last_change":"2024-01-15T18:06:00.164936+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.164936+0000","last_undegraded":"2024-01-15T18:06:00.164936+0000","last_fullsized":"2024-01-15T18:06:00.164936+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.173385+0000","last_change":"2024-01-15T18:06:00.173385+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.173385+0000","last_undegraded":"2024-01-15T18:06:00.173385+0000","last_fullsized":"2024-01-15T18:06:00.173385+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:35:19.522019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":4,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.183420+0000","last_change":"2024-01-15T18:06:00.171516+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.183420+0000","last_undegraded":"2024-01-15T18:06:00.183420+0000","last_fullsized":"2024-01-15T18:06:00.183420+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:31:04.425248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.172541+0000","last_change":"2024-01-15T18:06:00.172541+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.172541+0000","last_undegraded":"2024-01-15T18:06:00.172541+0000","last_fullsized":"2024-01-15T18:06:00.172541+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:05:59.145608+0000","last_change":"2024-01-15T18:05:59.145608+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.145608+0000","last_undegraded":"2024-01-15T18:05:59.145608+0000","last_fullsized":"2024-01-15T18:05:59.145608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.179315+0000","last_change":"2024-01-15T18:06:00.179315+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.179315+0000","last_undegraded":"2024-01-15T18:06:00.179315+0000","last_fullsized":"2024-01-15T18:06:00.179315+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-01-15T18:05:59.156144+0000","last_change":"2024-01-15T18:05:59.156144+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.156144+0000","last_undegraded":"2024-01-15T18:05:59.156144+0000","last_fullsized":"2024-01-15T18:05:59.156144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":4,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.179743+0000","last_change":"2024-01-15T18:06:00.171716+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.179743+0000","last_undegraded":"2024-01-15T18:06:00.179743+0000","last_fullsized":"2024-01-15T18:06:00.179743+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:26:09.652268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.165709+0000","last_change":"2024-01-15T18:06:00.165709+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.165709+0000","last_undegraded":"2024-01-15T18:06:00.165709+0000","last_fullsized":"2024-01-15T18:06:00.165709+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.164330+0000","last_change":"2024-01-15T18:06:00.164330+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.164330+0000","last_undegraded":"2024-01-15T18:06:00.164330+0000","last_fullsized":"2024-01-15T18:06:00.164330+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:03:20.544076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.170139+0000","last_change":"2024-01-15T18:06:00.170139+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.170139+0000","last_undegraded":"2024-01-15T18:06:00.170139+0000","last_fullsized":"2024-01-15T18:06:00.170139+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.162418+0000","last_change":"2024-01-15T18:06:00.162418+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.162418+0000","last_undegraded":"2024-01-15T18:06:00.162418+0000","last_fullsized":"2024-01-15T18:06:00.162418+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:56.736968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":4,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.183486+0000","last_change":"2024-01-15T18:06:00.170635+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.183486+0000","last_undegraded":"2024-01-15T18:06:00.183486+0000","last_fullsized":"2024-01-15T18:06:00.183486+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:56.736968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.168794+0000","last_change":"2024-01-15T18:06:00.168794+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.168794+0000","last_undegraded":"2024-01-15T18:06:00.168794+0000","last_fullsized":"2024-01-15T18:06:00.168794+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.179395+0000","last_change":"2024-01-15T18:06:00.179395+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.179395+0000","last_undegraded":"2024-01-15T18:06:00.179395+0000","last_fullsized":"2024-01-15T18:06:00.179395+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:51:46.033055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.164841+0000","last_change":"2024-01-15T18:06:00.164841+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.164841+0000","last_undegraded":"2024-01-15T18:06:00.164841+0000","last_fullsized":"2024-01-15T18:06:00.164841+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.162240+0000","last_change":"2024-01-15T18:06:00.162240+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.162240+0000","last_undegraded":"2024-01-15T18:06:00.162240+0000","last_fullsized":"2024-01-15T18:06:00.162240+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:08:08.490734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.166559+0000","last_change":"2024-01-15T18:06:00.166559+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.166559+0000","last_undegraded":"2024-01-15T18:06:00.166559+0000","last_fullsized":"2024-01-15T18:06:00.166559+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:11:04.167913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.164909+0000","last_change":"2024-01-15T18:06:00.164909+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.164909+0000","last_undegraded":"2024-01-15T18:06:00.164909+0000","last_fullsized":"2024-01-15T18:06:00.164909+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":8,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T18:06:00.176281+0000","last_change":"2024-01-15T18:06:00.176281+0000","last_active":"2024-01-15T18:06:00.176281+0000","last_peered":"2024-01-15T18:06:00.176281+0000","last_clean":"2024-01-15T18:06:00.176281+0000","last_became_active":"2024-01-15T18:06:00.174503+0000","last_became_peered":"2024-01-15T18:06:00.174503+0000","last_unstale":"2024-01-15T18:06:00.176281+0000","last_undegraded":"2024-01-15T18:06:00.176281+0000","last_fullsized":"2024-01-15T18:06:00.176281+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:10:04.757709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.168628+0000","last_change":"2024-01-15T18:06:00.168628+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.168628+0000","last_undegraded":"2024-01-15T18:06:00.168628+0000","last_fullsized":"2024-01-15T18:06:00.168628+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.164364+0000","last_change":"2024-01-15T18:06:00.164364+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.164364+0000","last_undegraded":"2024-01-15T18:06:00.164364+0000","last_fullsized":"2024-01-15T18:06:00.164364+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:59:22.476704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.168573+0000","last_change":"2024-01-15T18:06:00.168573+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.168573+0000","last_undegraded":"2024-01-15T18:06:00.168573+0000","last_fullsized":"2024-01-15T18:06:00.168573+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:05:59.145608+0000","last_change":"2024-01-15T18:05:59.145608+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.145608+0000","last_undegraded":"2024-01-15T18:05:59.145608+0000","last_fullsized":"2024-01-15T18:05:59.145608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.168105+0000","last_change":"2024-01-15T18:06:00.168105+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.168105+0000","last_undegraded":"2024-01-15T18:06:00.168105+0000","last_fullsized":"2024-01-15T18:06:00.168105+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-01-15T18:05:59.161004+0000","last_change":"2024-01-15T18:05:59.161004+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.161004+0000","last_undegraded":"2024-01-15T18:05:59.161004+0000","last_fullsized":"2024-01-15T18:05:59.161004+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.168020+0000","last_change":"2024-01-15T18:06:00.168020+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.168020+0000","last_undegraded":"2024-01-15T18:06:00.168020+0000","last_fullsized":"2024-01-15T18:06:00.168020+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.165635+0000","last_change":"2024-01-15T18:06:00.165635+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.165635+0000","last_undegraded":"2024-01-15T18:06:00.165635+0000","last_fullsized":"2024-01-15T18:06:00.165635+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:31:30.665626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-01-15T18:05:59.155857+0000","last_change":"2024-01-15T18:05:59.155857+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.155857+0000","last_undegraded":"2024-01-15T18:05:59.155857+0000","last_fullsized":"2024-01-15T18:05:59.155857+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.168806+0000","last_change":"2024-01-15T18:06:00.168806+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.168806+0000","last_undegraded":"2024-01-15T18:06:00.168806+0000","last_fullsized":"2024-01-15T18:06:00.168806+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":8,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T18:06:00.176082+0000","last_change":"2024-01-15T18:06:00.176082+0000","last_active":"2024-01-15T18:06:00.176082+0000","last_peered":"2024-01-15T18:06:00.176082+0000","last_clean":"2024-01-15T18:06:00.176082+0000","last_became_active":"2024-01-15T18:06:00.175734+0000","last_became_peered":"2024-01-15T18:06:00.175734+0000","last_unstale":"2024-01-15T18:06:00.176082+0000","last_undegraded":"2024-01-15T18:06:00.176082+0000","last_fullsized":"2024-01-15T18:06:00.176082+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:12.325308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:05:59.145608+0000","last_change":"2024-01-15T18:05:59.145608+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.145608+0000","last_undegraded":"2024-01-15T18:05:59.145608+0000","last_fullsized":"2024-01-15T18:05:59.145608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.162200+0000","last_change":"2024-01-15T18:06:00.162200+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.162200+0000","last_undegraded":"2024-01-15T18:06:00.162200+0000","last_fullsized":"2024-01-15T18:06:00.162200+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:46:52.165002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.172639+0000","last_change":"2024-01-15T18:06:00.172639+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.172639+0000","last_undegraded":"2024-01-15T18:06:00.172639+0000","last_fullsized":"2024-01-15T18:06:00.172639+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:05:59.145608+0000","last_change":"2024-01-15T18:05:59.145608+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.145608+0000","last_undegraded":"2024-01-15T18:05:59.145608+0000","last_fullsized":"2024-01-15T18:05:59.145608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":8,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T18:06:00.176729+0000","last_change":"2024-01-15T18:06:00.176729+0000","last_active":"2024-01-15T18:06:00.176729+0000","last_peered":"2024-01-15T18:06:00.176729+0000","last_clean":"2024-01-15T18:06:00.176729+0000","last_became_active":"2024-01-15T18:06:00.175928+0000","last_became_peered":"2024-01-15T18:06:00.175928+0000","last_unstale":"2024-01-15T18:06:00.176729+0000","last_undegraded":"2024-01-15T18:06:00.176729+0000","last_fullsized":"2024-01-15T18:06:00.176729+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:59:02.797271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.170431+0000","last_change":"2024-01-15T18:06:00.170431+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.170431+0000","last_undegraded":"2024-01-15T18:06:00.170431+0000","last_fullsized":"2024-01-15T18:06:00.170431+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":8,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T18:06:00.177964+0000","last_change":"2024-01-15T18:06:00.178477+0000","last_active":"2024-01-15T18:06:00.177964+0000","last_peered":"2024-01-15T18:06:00.177964+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"2024-01-15T18:06:00.176994+0000","last_became_peered":"2024-01-15T18:06:00.176994+0000","last_unstale":"2024-01-15T18:06:00.177964+0000","last_undegraded":"2024-01-15T18:06:00.177964+0000","last_fullsized":"2024-01-15T18:06:00.177964+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:40:09.517509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.166107+0000","last_change":"2024-01-15T18:06:00.166107+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.166107+0000","last_undegraded":"2024-01-15T18:06:00.166107+0000","last_fullsized":"2024-01-15T18:06:00.166107+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":8,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T18:06:00.184467+0000","last_change":"2024-01-15T18:06:00.184467+0000","last_active":"2024-01-15T18:06:00.184467+0000","last_peered":"2024-01-15T18:06:00.184467+0000","last_clean":"2024-01-15T18:06:00.184467+0000","last_became_active":"2024-01-15T18:06:00.184044+0000","last_became_peered":"2024-01-15T18:06:00.184044+0000","last_unstale":"2024-01-15T18:06:00.184467+0000","last_undegraded":"2024-01-15T18:06:00.184467+0000","last_fullsized":"2024-01-15T18:06:00.184467+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:12.325308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-01-15T18:05:59.161012+0000","last_change":"2024-01-15T18:05:59.161012+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.161012+0000","last_undegraded":"2024-01-15T18:05:59.161012+0000","last_fullsized":"2024-01-15T18:05:59.161012+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":172,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-15T18:05:54.653336+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:05:54.653336+0000","last_peered":"2024-01-15T18:05:54.653336+0000","last_clean":"2024-01-15T18:05:54.653336+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:05:54.653336+0000","last_undegraded":"2024-01-15T18:05:54.653336+0000","last_fullsized":"2024-01-15T18:05:54.653336+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":78,"acting":78,"num_store_stats":5},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":54,"acting":54,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509458,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27932,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343908,"statfs":{"total":96636764160,"available":96608161792,"internally_reserved":0,"allocated":933888,"data_stored":719017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.84999999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.90400000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.83299999999999996}]},{"osd":3,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.73699999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.61499999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.70499999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.75600000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509458,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":339968,"data_stored":128649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.59299999999999997}]},{"osd":1,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":2.4220000000000002,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.391,"5min":1.391,"15min":1.391},"last":0.745}]},{"osd":2,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.8700000000000001,"5min":2.8700000000000001,"15min":2.8700000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":0.52400000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":2.012,"5min":2.012,"15min":2.012},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.532,"5min":2.532,"15min":2.532},"last":0.81799999999999995}]},{"osd":4,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.879},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.68600000000000005}]},{"osd":5,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.91400000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.85399999999999998}]}]},{"osd":4,"up_from":15,"seq":64424509458,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":339968,"data_stored":128649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.64000000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.88}]},{"osd":2,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.66000000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.77000000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.74399999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.92100000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.90600000000000003}]}]},{"osd":3,"up_from":15,"seq":64424509457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":339968,"data_stored":128649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.42099999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.47199999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.56399999999999995}]},{"osd":4,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.76900000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.61699999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.67000000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.67900000000000005}]}]},{"osd":2,"up_from":15,"seq":64424509457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":339968,"data_stored":128649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.47799999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.42699999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.56899999999999995}]},{"osd":4,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.466,"5min":1.466,"15min":1.466},"last":0.77400000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.90400000000000003}]},{"osd":6,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.516,"5min":1.516,"15min":1.516},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.71199999999999997}]},{"osd":7,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.6279999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.81200000000000006}]}]},{"osd":1,"up_from":15,"seq":64424509459,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":339968,"data_stored":128649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.317}]},{"osd":2,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":4.2450000000000001,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.435}]},{"osd":3,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.917,"5min":1.917,"15min":1.917},"last":0.54300000000000004}]},{"osd":4,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.73499999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.56399999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.69699999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.753},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.66500000000000004}]}]},{"osd":5,"up_from":14,"seq":60129542162,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":327680,"data_stored":122832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":2.802,"5min":2.802,"15min":2.802},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.68600000000000005}]},{"osd":1,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.54100000000000004}]},{"osd":2,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":0.57099999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":2.3849999999999998,"5min":2.3849999999999998,"15min":2.3849999999999998},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":0.55400000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.64100000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.621}]},{"osd":7,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.67000000000000004}]}]},{"osd":0,"up_from":14,"seq":60129542162,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27920,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343920,"statfs":{"total":96636764160,"available":96608174080,"internally_reserved":0,"allocated":921600,"data_stored":713200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.34399999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.495},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.51600000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.325,"5min":1.325,"15min":1.325},"last":0.53900000000000003}]},{"osd":4,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.63700000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:05:43 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.9079999999999999,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":0.70799999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.764,"5min":1.764,"15min":1.764},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.806,"5min":1.806,"15min":1.806},"last":0.72499999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.722,"5min":1.722,"15min":1.722},"last":0.77900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:06:05.695 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:06:05.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.874+0000 7f0cac000640 1 Processor -- start 2024-01-15T18:06:05.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.874+0000 7f0cac000640 1 -- start start 2024-01-15T18:06:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.874+0000 7f0cac000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0ca40ce570 0x7f0ca406bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.874+0000 7f0cac000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ca406c4b0 0x7f0ca4166dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.874+0000 7f0cac000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca40cfe30 con 0x7f0ca415d240 2024-01-15T18:06:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.874+0000 7f0cac000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca406bd10 con 0x7f0ca40ce570 2024-01-15T18:06:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.874+0000 7f0cac000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca406c960 con 0x7f0ca406c4b0 2024-01-15T18:06:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.874+0000 7f0ca8c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0ca40ce570 0x7f0ca406bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.874+0000 7f0caa000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ca406c4b0 0x7f0ca4166dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.874+0000 7f0ca8c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0ca40ce570 0x7f0ca406bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:49490/0 (socket says 172.21.15.27:49490) 2024-01-15T18:06:05.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.874+0000 7f0ca9600640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7f0ca415d240 0x7f0ca406b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:43674/0 (socket says 172.21.15.27:43674) 2024-01-15T18:06:05.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.874+0000 7f0ca8c00640 1 -- 172.21.15.27:0/481404672 learned_addr learned my addr 172.21.15.27:0/481404672 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:05.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.874+0000 7f0ca8c00640 1 -- 172.21.15.27:0/481404672 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ca406c4b0 msgr2=0x7f0ca4166dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:05.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.874+0000 7f0ca8c00640 1 --2- 172.21.15.27:0/481404672 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ca406c4b0 0x7f0ca4166dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:05.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.874+0000 7f0ca8c00640 1 -- 172.21.15.27:0/481404672 >> v1:172.21.15.93:6789/0 conn(0x7f0ca415d240 legacy=0x7f0ca406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:05.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.874+0000 7f0ca8c00640 1 -- 172.21.15.27:0/481404672 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca415e630 con 0x7f0ca40ce570 2024-01-15T18:06:05.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.878+0000 7f0ca8c00640 1 --2- 172.21.15.27:0/481404672 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ca40ce570 0x7f0ca406bf70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c009870 tx=0x7f0c8c01c510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dc4068d8a06063d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:05.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.878+0000 7f0ca3e00640 1 -- 172.21.15.27:0/481404672 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c0aa670 con 0x7f0ca40ce570 2024-01-15T18:06:05.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.878+0000 7f0ca3e00640 1 -- 172.21.15.27:0/481404672 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8c0aa7d0 con 0x7f0ca40ce570 2024-01-15T18:06:05.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.878+0000 7f0ca3e00640 1 -- 172.21.15.27:0/481404672 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c0aab60 con 0x7f0ca40ce570 2024-01-15T18:06:05.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.878+0000 7f0cac000640 1 -- 172.21.15.27:0/481404672 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ca40ce570 msgr2=0x7f0ca406bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:05.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.878+0000 7f0cac000640 1 --2- 172.21.15.27:0/481404672 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ca40ce570 0x7f0ca406bf70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c009870 tx=0x7f0c8c01c510 comp rx=0 tx=0).stop 2024-01-15T18:06:05.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.878+0000 7f0cac000640 1 -- 172.21.15.27:0/481404672 shutdown_connections 2024-01-15T18:06:05.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.878+0000 7f0cac000640 1 --2- 172.21.15.27:0/481404672 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ca406c4b0 0x7f0ca4166dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:05.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.878+0000 7f0cac000640 1 --2- 172.21.15.27:0/481404672 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ca40ce570 0x7f0ca406bf70 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:05.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.878+0000 7f0cac000640 1 -- 172.21.15.27:0/481404672 >> 172.21.15.27:0/481404672 conn(0x7f0ca40d0f20 msgr2=0x7f0ca40d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:05.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.878+0000 7f0cac000640 1 -- 172.21.15.27:0/481404672 shutdown_connections 2024-01-15T18:06:05.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.878+0000 7f0cac000640 1 -- 172.21.15.27:0/481404672 wait complete. 2024-01-15T18:06:05.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.882+0000 7f0cac000640 1 Processor -- start 2024-01-15T18:06:05.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.882+0000 7f0cac000640 1 -- start start 2024-01-15T18:06:05.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.882+0000 7f0cac000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ca406c4b0 0x7f0ca414b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:05.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.882+0000 7f0cac000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ca40ce570 0x7f0ca4154cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:05.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.882+0000 7f0cac000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ca415d240 0x7f0ca414da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:05.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.882+0000 7f0cac000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca414b820 con 0x7f0ca406c4b0 2024-01-15T18:06:05.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.882+0000 7f0cac000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca41577f0 con 0x7f0ca415d240 2024-01-15T18:06:05.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.882+0000 7f0cac000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca4155210 con 0x7f0ca40ce570 2024-01-15T18:06:05.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.882+0000 7f0ca9600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ca406c4b0 0x7f0ca414b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:05.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.882+0000 7f0ca9600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ca406c4b0 0x7f0ca414b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56632/0 (socket says 172.21.15.27:56632) 2024-01-15T18:06:05.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.882+0000 7f0ca9600640 1 -- 172.21.15.27:0/1153518631 learned_addr learned my addr 172.21.15.27:0/1153518631 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:05.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.882+0000 7f0caa000640 1 --2- 172.21.15.27:0/1153518631 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ca415d240 0x7f0ca414da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:05.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.882+0000 7f0ca8c00640 1 --2- 172.21.15.27:0/1153518631 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ca40ce570 0x7f0ca4154cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:05.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.882+0000 7f0ca9600640 1 -- 172.21.15.27:0/1153518631 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ca40ce570 msgr2=0x7f0ca4154cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:05.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.882+0000 7f0ca9600640 1 --2- 172.21.15.27:0/1153518631 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ca40ce570 0x7f0ca4154cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:05.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.882+0000 7f0ca9600640 1 -- 172.21.15.27:0/1153518631 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ca415d240 msgr2=0x7f0ca414da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:05.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.882+0000 7f0ca9600640 1 --2- 172.21.15.27:0/1153518631 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ca415d240 0x7f0ca414da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:05.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.882+0000 7f0ca9600640 1 -- 172.21.15.27:0/1153518631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c8c0094d0 con 0x7f0ca406c4b0 2024-01-15T18:06:05.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.882+0000 7f0ca9600640 1 --2- 172.21.15.27:0/1153518631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ca406c4b0 0x7f0ca414b2e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0c9400c430 tx=0x7f0c9400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:05.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.882+0000 7f0ca1600640 1 -- 172.21.15.27:0/1153518631 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c940074e0 con 0x7f0ca406c4b0 2024-01-15T18:06:05.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.882+0000 7f0ca1600640 1 -- 172.21.15.27:0/1153518631 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c94007640 con 0x7f0ca406c4b0 2024-01-15T18:06:05.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.882+0000 7f0ca1600640 1 -- 172.21.15.27:0/1153518631 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c94004a10 con 0x7f0ca406c4b0 2024-01-15T18:06:05.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.882+0000 7f0cac000640 1 -- 172.21.15.27:0/1153518631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca414df40 con 0x7f0ca406c4b0 2024-01-15T18:06:05.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.882+0000 7f0cac000640 1 -- 172.21.15.27:0/1153518631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca41571a0 con 0x7f0ca406c4b0 2024-01-15T18:06:05.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.886+0000 7f0cac000640 1 -- 172.21.15.27:0/1153518631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ca406d530 con 0x7f0ca406c4b0 2024-01-15T18:06:05.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.886+0000 7f0ca1600640 1 -- 172.21.15.27:0/1153518631 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f0c94004b70 con 0x7f0ca406c4b0 2024-01-15T18:06:05.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.886+0000 7f0ca1600640 1 --2- 172.21.15.27:0/1153518631 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c7c0668f0 0x7f0c7c068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:05.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.886+0000 7f0ca1600640 1 -- 172.21.15.27:0/1153518631 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c94083ca0 con 0x7f0ca406c4b0 2024-01-15T18:06:05.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.886+0000 7f0ca8c00640 1 --2- 172.21.15.27:0/1153518631 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c7c0668f0 0x7f0c7c068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:05.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.886+0000 7f0ca8c00640 1 --2- 172.21.15.27:0/1153518631 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c7c0668f0 0x7f0c7c068db0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c0ae000 tx=0x7f0c8c0b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:05.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:05.890+0000 7f0ca1600640 1 -- 172.21.15.27:0/1153518631 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0c94050f90 con 0x7f0ca406c4b0 2024-01-15T18:06:06.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.046+0000 7f0cac000640 1 -- 172.21.15.27:0/1153518631 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0ca4160070 con 0x7f0c7c0668f0 2024-01-15T18:06:06.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.054+0000 7f0ca1600640 1 -- 172.21.15.27:0/1153518631 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+280246 (secure 0 0 0) 0x7f0ca4160070 con 0x7f0c7c0668f0 2024-01-15T18:06:06.062 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:06:06.064 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:06:06.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.058+0000 7f0cac000640 1 -- 172.21.15.27:0/1153518631 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c7c0668f0 msgr2=0x7f0c7c068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:06.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.058+0000 7f0cac000640 1 --2- 172.21.15.27:0/1153518631 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c7c0668f0 0x7f0c7c068db0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c0ae000 tx=0x7f0c8c0b5040 comp rx=0 tx=0).stop 2024-01-15T18:06:06.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.058+0000 7f0cac000640 1 -- 172.21.15.27:0/1153518631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ca406c4b0 msgr2=0x7f0ca414b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:06.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.058+0000 7f0cac000640 1 --2- 172.21.15.27:0/1153518631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ca406c4b0 0x7f0ca414b2e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0c9400c430 tx=0x7f0c9400c8f0 comp rx=0 tx=0).stop 2024-01-15T18:06:06.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.058+0000 7f0cac000640 1 -- 172.21.15.27:0/1153518631 shutdown_connections 2024-01-15T18:06:06.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.058+0000 7f0cac000640 1 --2- 172.21.15.27:0/1153518631 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c7c0668f0 0x7f0c7c068db0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:06.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.058+0000 7f0cac000640 1 --2- 172.21.15.27:0/1153518631 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ca415d240 0x7f0ca414da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:06.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.058+0000 7f0cac000640 1 --2- 172.21.15.27:0/1153518631 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ca40ce570 0x7f0ca4154cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:06.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.058+0000 7f0cac000640 1 --2- 172.21.15.27:0/1153518631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ca406c4b0 0x7f0ca414b2e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:06.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.058+0000 7f0cac000640 1 -- 172.21.15.27:0/1153518631 >> 172.21.15.27:0/1153518631 conn(0x7f0ca40d0f20 msgr2=0x7f0ca4155ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:06.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.058+0000 7f0cac000640 1 -- 172.21.15.27:0/1153518631 shutdown_connections 2024-01-15T18:06:06.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.058+0000 7f0cac000640 1 -- 172.21.15.27:0/1153518631 wait complete. 2024-01-15T18:06:06.083 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-01-15T18:06:04.153664+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":134,"acting":134,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219968,"kb_used_data":3792,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754752,"statfs":{"total":773094113280,"available":772868866048,"internally_reserved":0,"allocated":3883008,"data_stored":2198294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.000451"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.171586+0000","last_change":"2024-01-15T18:06:00.171586+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.171586+0000","last_undegraded":"2024-01-15T18:06:00.171586+0000","last_fullsized":"2024-01-15T18:06:00.171586+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.179571+0000","last_change":"2024-01-15T18:06:00.179571+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.179571+0000","last_undegraded":"2024-01-15T18:06:00.179571+0000","last_fullsized":"2024-01-15T18:06:00.179571+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.165981+0000","last_change":"2024-01-15T18:06:00.165981+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.165981+0000","last_undegraded":"2024-01-15T18:06:00.165981+0000","last_fullsized":"2024-01-15T18:06:00.165981+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.172568+0000","last_change":"2024-01-15T18:06:00.172568+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.172568+0000","last_undegraded":"2024-01-15T18:06:00.172568+0000","last_fullsized":"2024-01-15T18:06:00.172568+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.180507+0000","last_change":"2024-01-15T18:06:00.180507+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.180507+0000","last_undegraded":"2024-01-15T18:06:00.180507+0000","last_fullsized":"2024-01-15T18:06:00.180507+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.165821+0000","last_change":"2024-01-15T18:06:00.165821+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.165821+0000","last_undegraded":"2024-01-15T18:06:00.165821+0000","last_fullsized":"2024-01-15T18:06:00.165821+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.165786+0000","last_change":"2024-01-15T18:06:00.165786+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.165786+0000","last_undegraded":"2024-01-15T18:06:00.165786+0000","last_fullsized":"2024-01-15T18:06:00.165786+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.172055+0000","last_change":"2024-01-15T18:06:00.172055+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.172055+0000","last_undegraded":"2024-01-15T18:06:00.172055+0000","last_fullsized":"2024-01-15T18:06:00.172055+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.172208+0000","last_change":"2024-01-15T18:06:00.172208+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.172208+0000","last_undegraded":"2024-01-15T18:06:00.172208+0000","last_fullsized":"2024-01-15T18:06:00.172208+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.172461+0000","last_change":"2024-01-15T18:06:00.172461+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.172461+0000","last_undegraded":"2024-01-15T18:06:00.172461+0000","last_fullsized":"2024-01-15T18:06:00.172461+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.179869+0000","last_change":"2024-01-15T18:06:00.179869+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.179869+0000","last_undegraded":"2024-01-15T18:06:00.179869+0000","last_fullsized":"2024-01-15T18:06:00.179869+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.169391+0000","last_change":"2024-01-15T18:06:00.169391+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.169391+0000","last_undegraded":"2024-01-15T18:06:00.169391+0000","last_fullsized":"2024-01-15T18:06:00.169391+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.173714+0000","last_change":"2024-01-15T18:06:00.173714+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.173714+0000","last_undegraded":"2024-01-15T18:06:00.173714+0000","last_fullsized":"2024-01-15T18:06:00.173714+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.167370+0000","last_change":"2024-01-15T18:06:00.167370+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.167370+0000","last_undegraded":"2024-01-15T18:06:00.167370+0000","last_fullsized":"2024-01-15T18:06:00.167370+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.166345+0000","last_change":"2024-01-15T18:06:00.166345+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.166345+0000","last_undegraded":"2024-01-15T18:06:00.166345+0000","last_fullsized":"2024-01-15T18:06:00.166345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.170414+0000","last_change":"2024-01-15T18:06:00.170414+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.170414+0000","last_undegraded":"2024-01-15T18:06:00.170414+0000","last_fullsized":"2024-01-15T18:06:00.170414+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.165628+0000","last_change":"2024-01-15T18:06:00.165628+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.165628+0000","last_undegraded":"2024-01-15T18:06:00.165628+0000","last_fullsized":"2024-01-15T18:06:00.165628+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.165370+0000","last_change":"2024-01-15T18:06:00.165370+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.165370+0000","last_undegraded":"2024-01-15T18:06:00.165370+0000","last_fullsized":"2024-01-15T18:06:00.165370+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.170608+0000","last_change":"2024-01-15T18:06:00.170608+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.170608+0000","last_undegraded":"2024-01-15T18:06:00.170608+0000","last_fullsized":"2024-01-15T18:06:00.170608+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-01-15T18:05:59.161092+0000","last_change":"2024-01-15T18:05:59.161092+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.161092+0000","last_undegraded":"2024-01-15T18:05:59.161092+0000","last_fullsized":"2024-01-15T18:05:59.161092+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.171119+0000","last_change":"2024-01-15T18:06:00.171119+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.171119+0000","last_undegraded":"2024-01-15T18:06:00.171119+0000","last_fullsized":"2024-01-15T18:06:00.171119+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:05:59.145608+0000","last_change":"2024-01-15T18:05:59.145608+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.145608+0000","last_undegraded":"2024-01-15T18:05:59.145608+0000","last_fullsized":"2024-01-15T18:05:59.145608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.164402+0000","last_change":"2024-01-15T18:06:00.164402+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.164402+0000","last_undegraded":"2024-01-15T18:06:00.164402+0000","last_fullsized":"2024-01-15T18:06:00.164402+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.180303+0000","last_change":"2024-01-15T18:06:00.180303+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.180303+0000","last_undegraded":"2024-01-15T18:06:00.180303+0000","last_fullsized":"2024-01-15T18:06:00.180303+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:38:07.317901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.165003+0000","last_change":"2024-01-15T18:06:00.165003+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.165003+0000","last_undegraded":"2024-01-15T18:06:00.165003+0000","last_fullsized":"2024-01-15T18:06:00.165003+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.168374+0000","last_change":"2024-01-15T18:06:00.168374+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.168374+0000","last_undegraded":"2024-01-15T18:06:00.168374+0000","last_fullsized":"2024-01-15T18:06:00.168374+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:31:30.665626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.174622+0000","last_change":"2024-01-15T18:06:00.174622+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.174622+0000","last_undegraded":"2024-01-15T18:06:00.174622+0000","last_fullsized":"2024-01-15T18:06:00.174622+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:39:25.962651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.164626+0000","last_change":"2024-01-15T18:06:00.164626+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.164626+0000","last_undegraded":"2024-01-15T18:06:00.164626+0000","last_fullsized":"2024-01-15T18:06:00.164626+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:48:52.588812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.164265+0000","last_change":"2024-01-15T18:06:00.164265+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.164265+0000","last_undegraded":"2024-01-15T18:06:00.164265+0000","last_fullsized":"2024-01-15T18:06:00.164265+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.171075+0000","last_change":"2024-01-15T18:06:00.171075+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.171075+0000","last_undegraded":"2024-01-15T18:06:00.171075+0000","last_fullsized":"2024-01-15T18:06:00.171075+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:40:56.123776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.164936+0000","last_change":"2024-01-15T18:06:00.164936+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.164936+0000","last_undegraded":"2024-01-15T18:06:00.164936+0000","last_fullsized":"2024-01-15T18:06:00.164936+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.173385+0000","last_change":"2024-01-15T18:06:00.173385+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.173385+0000","last_undegraded":"2024-01-15T18:06:00.173385+0000","last_fullsized":"2024-01-15T18:06:00.173385+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:35:19.522019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":4,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.183420+0000","last_change":"2024-01-15T18:06:00.171516+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.183420+0000","last_undegraded":"2024-01-15T18:06:00.183420+0000","last_fullsized":"2024-01-15T18:06:00.183420+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:31:04.425248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.172541+0000","last_change":"2024-01-15T18:06:00.172541+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.172541+0000","last_undegraded":"2024-01-15T18:06:00.172541+0000","last_fullsized":"2024-01-15T18:06:00.172541+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:05:59.145608+0000","last_change":"2024-01-15T18:05:59.145608+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.145608+0000","last_undegraded":"2024-01-15T18:05:59.145608+0000","last_fullsized":"2024-01-15T18:05:59.145608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.179315+0000","last_change":"2024-01-15T18:06:00.179315+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.179315+0000","last_undegraded":"2024-01-15T18:06:00.179315+0000","last_fullsized":"2024-01-15T18:06:00.179315+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-01-15T18:05:59.156144+0000","last_change":"2024-01-15T18:05:59.156144+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.156144+0000","last_undegraded":"2024-01-15T18:05:59.156144+0000","last_fullsized":"2024-01-15T18:05:59.156144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":4,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.179743+0000","last_change":"2024-01-15T18:06:00.171716+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.179743+0000","last_undegraded":"2024-01-15T18:06:00.179743+0000","last_fullsized":"2024-01-15T18:06:00.179743+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:26:09.652268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.165709+0000","last_change":"2024-01-15T18:06:00.165709+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.165709+0000","last_undegraded":"2024-01-15T18:06:00.165709+0000","last_fullsized":"2024-01-15T18:06:00.165709+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.164330+0000","last_change":"2024-01-15T18:06:00.164330+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.164330+0000","last_undegraded":"2024-01-15T18:06:00.164330+0000","last_fullsized":"2024-01-15T18:06:00.164330+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:03:20.544076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.170139+0000","last_change":"2024-01-15T18:06:00.170139+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.170139+0000","last_undegraded":"2024-01-15T18:06:00.170139+0000","last_fullsized":"2024-01-15T18:06:00.170139+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.162418+0000","last_change":"2024-01-15T18:06:00.162418+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.162418+0000","last_undegraded":"2024-01-15T18:06:00.162418+0000","last_fullsized":"2024-01-15T18:06:00.162418+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:56.736968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":4,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.183486+0000","last_change":"2024-01-15T18:06:00.170635+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.183486+0000","last_undegraded":"2024-01-15T18:06:00.183486+0000","last_fullsized":"2024-01-15T18:06:00.183486+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:56.736968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.168794+0000","last_change":"2024-01-15T18:06:00.168794+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.168794+0000","last_undegraded":"2024-01-15T18:06:00.168794+0000","last_fullsized":"2024-01-15T18:06:00.168794+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.179395+0000","last_change":"2024-01-15T18:06:00.179395+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.179395+0000","last_undegraded":"2024-01-15T18:06:00.179395+0000","last_fullsized":"2024-01-15T18:06:00.179395+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:51:46.033055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.164841+0000","last_change":"2024-01-15T18:06:00.164841+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.164841+0000","last_undegraded":"2024-01-15T18:06:00.164841+0000","last_fullsized":"2024-01-15T18:06:00.164841+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.162240+0000","last_change":"2024-01-15T18:06:00.162240+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.162240+0000","last_undegraded":"2024-01-15T18:06:00.162240+0000","last_fullsized":"2024-01-15T18:06:00.162240+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:08:08.490734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.166559+0000","last_change":"2024-01-15T18:06:00.166559+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.166559+0000","last_undegraded":"2024-01-15T18:06:00.166559+0000","last_fullsized":"2024-01-15T18:06:00.166559+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:11:04.167913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.164909+0000","last_change":"2024-01-15T18:06:00.164909+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.164909+0000","last_undegraded":"2024-01-15T18:06:00.164909+0000","last_fullsized":"2024-01-15T18:06:00.164909+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":8,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T18:06:00.176281+0000","last_change":"2024-01-15T18:06:00.176281+0000","last_active":"2024-01-15T18:06:00.176281+0000","last_peered":"2024-01-15T18:06:00.176281+0000","last_clean":"2024-01-15T18:06:00.176281+0000","last_became_active":"2024-01-15T18:06:00.174503+0000","last_became_peered":"2024-01-15T18:06:00.174503+0000","last_unstale":"2024-01-15T18:06:00.176281+0000","last_undegraded":"2024-01-15T18:06:00.176281+0000","last_fullsized":"2024-01-15T18:06:00.176281+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:10:04.757709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.168628+0000","last_change":"2024-01-15T18:06:00.168628+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.168628+0000","last_undegraded":"2024-01-15T18:06:00.168628+0000","last_fullsized":"2024-01-15T18:06:00.168628+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.164364+0000","last_change":"2024-01-15T18:06:00.164364+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.164364+0000","last_undegraded":"2024-01-15T18:06:00.164364+0000","last_fullsized":"2024-01-15T18:06:00.164364+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:59:22.476704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.168573+0000","last_change":"2024-01-15T18:06:00.168573+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.168573+0000","last_undegraded":"2024-01-15T18:06:00.168573+0000","last_fullsized":"2024-01-15T18:06:00.168573+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:05:59.145608+0000","last_change":"2024-01-15T18:05:59.145608+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.145608+0000","last_undegraded":"2024-01-15T18:05:59.145608+0000","last_fullsized":"2024-01-15T18:05:59.145608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.168105+0000","last_change":"2024-01-15T18:06:00.168105+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.168105+0000","last_undegraded":"2024-01-15T18:06:00.168105+0000","last_fullsized":"2024-01-15T18:06:00.168105+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-01-15T18:05:59.161004+0000","last_change":"2024-01-15T18:05:59.161004+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.161004+0000","last_undegraded":"2024-01-15T18:05:59.161004+0000","last_fullsized":"2024-01-15T18:05:59.161004+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.168020+0000","last_change":"2024-01-15T18:06:00.168020+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.168020+0000","last_undegraded":"2024-01-15T18:06:00.168020+0000","last_fullsized":"2024-01-15T18:06:00.168020+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.165635+0000","last_change":"2024-01-15T18:06:00.165635+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.165635+0000","last_undegraded":"2024-01-15T18:06:00.165635+0000","last_fullsized":"2024-01-15T18:06:00.165635+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:31:30.665626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-01-15T18:05:59.155857+0000","last_change":"2024-01-15T18:05:59.155857+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.155857+0000","last_undegraded":"2024-01-15T18:05:59.155857+0000","last_fullsized":"2024-01-15T18:05:59.155857+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.168806+0000","last_change":"2024-01-15T18:06:00.168806+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.168806+0000","last_undegraded":"2024-01-15T18:06:00.168806+0000","last_fullsized":"2024-01-15T18:06:00.168806+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":8,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T18:06:00.176082+0000","last_change":"2024-01-15T18:06:00.176082+0000","last_active":"2024-01-15T18:06:00.176082+0000","last_peered":"2024-01-15T18:06:00.176082+0000","last_clean":"2024-01-15T18:06:00.176082+0000","last_became_active":"2024-01-15T18:06:00.175734+0000","last_became_peered":"2024-01-15T18:06:00.175734+0000","last_unstale":"2024-01-15T18:06:00.176082+0000","last_undegraded":"2024-01-15T18:06:00.176082+0000","last_fullsized":"2024-01-15T18:06:00.176082+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:12.325308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:05:59.145608+0000","last_change":"2024-01-15T18:05:59.145608+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.145608+0000","last_undegraded":"2024-01-15T18:05:59.145608+0000","last_fullsized":"2024-01-15T18:05:59.145608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":3,"reported_epoch":32,"state":"creating+activating","last_fresh":"2024-01-15T18:06:00.162200+0000","last_change":"2024-01-15T18:06:00.162200+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.162200+0000","last_undegraded":"2024-01-15T18:06:00.162200+0000","last_fullsized":"2024-01-15T18:06:00.162200+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:46:52.165002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.172639+0000","last_change":"2024-01-15T18:06:00.172639+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.172639+0000","last_undegraded":"2024-01-15T18:06:00.172639+0000","last_fullsized":"2024-01-15T18:06:00.172639+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:05:59.145608+0000","last_change":"2024-01-15T18:05:59.145608+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.145608+0000","last_undegraded":"2024-01-15T18:05:59.145608+0000","last_fullsized":"2024-01-15T18:05:59.145608+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":8,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T18:06:00.176729+0000","last_change":"2024-01-15T18:06:00.176729+0000","last_active":"2024-01-15T18:06:00.176729+0000","last_peered":"2024-01-15T18:06:00.176729+0000","last_clean":"2024-01-15T18:06:00.176729+0000","last_became_active":"2024-01-15T18:06:00.175928+0000","last_became_peered":"2024-01-15T18:06:00.175928+0000","last_unstale":"2024-01-15T18:06:00.176729+0000","last_undegraded":"2024-01-15T18:06:00.176729+0000","last_fullsized":"2024-01-15T18:06:00.176729+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:59:02.797271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.170431+0000","last_change":"2024-01-15T18:06:00.170431+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.170431+0000","last_undegraded":"2024-01-15T18:06:00.170431+0000","last_fullsized":"2024-01-15T18:06:00.170431+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":8,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T18:06:00.177964+0000","last_change":"2024-01-15T18:06:00.178477+0000","last_active":"2024-01-15T18:06:00.177964+0000","last_peered":"2024-01-15T18:06:00.177964+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"2024-01-15T18:06:00.176994+0000","last_became_peered":"2024-01-15T18:06:00.176994+0000","last_unstale":"2024-01-15T18:06:00.177964+0000","last_undegraded":"2024-01-15T18:06:00.177964+0000","last_fullsized":"2024-01-15T18:06:00.177964+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:40:09.517509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2024-01-15T18:06:00.166107+0000","last_change":"2024-01-15T18:06:00.166107+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.166107+0000","last_undegraded":"2024-01-15T18:06:00.166107+0000","last_fullsized":"2024-01-15T18:06:00.166107+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":8,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T18:06:00.184467+0000","last_change":"2024-01-15T18:06:00.184467+0000","last_active":"2024-01-15T18:06:00.184467+0000","last_peered":"2024-01-15T18:06:00.184467+0000","last_clean":"2024-01-15T18:06:00.184467+0000","last_became_active":"2024-01-15T18:06:00.184044+0000","last_became_peered":"2024-01-15T18:06:00.184044+0000","last_unstale":"2024-01-15T18:06:00.184467+0000","last_undegraded":"2024-01-15T18:06:00.184467+0000","last_fullsized":"2024-01-15T18:06:00.184467+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:12.325308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:06:00.152476+0000","last_change":"2024-01-15T18:06:00.152476+0000","last_active":"2024-01-15T18:06:00.152476+0000","last_peered":"2024-01-15T18:06:00.152476+0000","last_clean":"2024-01-15T18:06:00.152476+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:06:00.152476+0000","last_undegraded":"2024-01-15T18:06:00.152476+0000","last_fullsized":"2024-01-15T18:06:00.152476+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-01-15T18:05:59.161012+0000","last_change":"2024-01-15T18:05:59.161012+0000","last_active":"2024-01-15T18:05:59.145608+0000","last_peered":"2024-01-15T18:05:59.145608+0000","last_clean":"2024-01-15T18:05:59.145608+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:05:59.161012+0000","last_undegraded":"2024-01-15T18:05:59.161012+0000","last_fullsized":"2024-01-15T18:05:59.161012+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":172,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-15T18:05:54.653336+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:05:54.653336+0000","last_peered":"2024-01-15T18:05:54.653336+0000","last_clean":"2024-01-15T18:05:54.653336+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:05:54.653336+0000","last_undegraded":"2024-01-15T18:05:54.653336+0000","last_fullsized":"2024-01-15T18:05:54.653336+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":78,"acting":78,"num_store_stats":5},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":54,"acting":54,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509458,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27932,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343908,"statfs":{"total":96636764160,"available":96608161792,"internally_reserved":0,"allocated":933888,"data_stored":719017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.84999999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.90400000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.83299999999999996}]},{"osd":3,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.73699999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.61499999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.70499999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.75600000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509458,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":339968,"data_stored":128649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.59299999999999997}]},{"osd":1,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":2.4220000000000002,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.391,"5min":1.391,"15min":1.391},"last":0.745}]},{"osd":2,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.8700000000000001,"5min":2.8700000000000001,"15min":2.8700000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":0.52400000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":2.012,"5min":2.012,"15min":2.012},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.532,"5min":2.532,"15min":2.532},"last":0.81799999999999995}]},{"osd":4,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.879},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.68600000000000005}]},{"osd":5,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.91400000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.85399999999999998}]}]},{"osd":4,"up_from":15,"seq":64424509458,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":339968,"data_stored":128649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.64000000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.88}]},{"osd":2,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.66000000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.77000000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.74399999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.92100000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.90600000000000003}]}]},{"osd":3,"up_from":15,"seq":64424509457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":339968,"data_stored":128649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.42099999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.47199999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.56399999999999995}]},{"osd":4,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.76900000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.61699999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.67000000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.67900000000000005}]}]},{"osd":2,"up_from":15,"seq":64424509457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":339968,"data_stored":128649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.47799999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.42699999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.56899999999999995}]},{"osd":4,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.466,"5min":1.466,"15min":1.466},"last":0.77400000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.90400000000000003}]},{"osd":6,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.516,"5min":1.516,"15min":1.516},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.71199999999999997}]},{"osd":7,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.6279999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.81200000000000006}]}]},{"osd":1,"up_from":15,"seq":64424509459,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":339968,"data_stored":128649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.317}]},{"osd":2,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":4.2450000000000001,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.435}]},{"osd":3,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.917,"5min":1.917,"15min":1.917},"last":0.54300000000000004}]},{"osd":4,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.73499999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.56399999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.69699999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.753},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.66500000000000004}]}]},{"osd":5,"up_from":14,"seq":60129542162,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":327680,"data_stored":122832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":2.802,"5min":2.802,"15min":2.802},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.68600000000000005}]},{"osd":1,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.54100000000000004}]},{"osd":2,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":0.57099999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":2.3849999999999998,"5min":2.3849999999999998,"15min":2.3849999999999998},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":0.55400000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.64100000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.621}]},{"osd":7,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.67000000000000004}]}]},{"osd":0,"up_from":14,"seq":60129542162,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27920,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343920,"statfs":{"total":96636764160,"available":96608174080,"internally_reserved":0,"allocated":921600,"data_stored":713200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.34399999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.495},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.51600000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.325,"5min":1.325,"15min":1.325},"last":0.53900000000000003}]},{"osd":4,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.63700000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:05:43 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.9079999999999999,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":0.70799999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.764,"5min":1.764,"15min":1.764},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.806,"5min":1.806,"15min":1.806},"last":0.72499999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.722,"5min":1.722,"15min":1.722},"last":0.77900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:06:06.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3f is not active+clean 2024-01-15T18:06:06.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.171586+0000', 'last_change': '2024-01-15T18:06:00.171586+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.171586+0000', 'last_undegraded': '2024-01-15T18:06:00.171586+0000', 'last_fullsized': '2024-01-15T18:06:00.171586+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:06:06.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3e is not active+clean 2024-01-15T18:06:06.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.179571+0000', 'last_change': '2024-01-15T18:06:00.179571+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.179571+0000', 'last_undegraded': '2024-01-15T18:06:00.179571+0000', 'last_fullsized': '2024-01-15T18:06:00.179571+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:06:06.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3d is not active+clean 2024-01-15T18:06:06.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:06:00.152476+0000', 'last_change': '2024-01-15T18:06:00.152476+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.152476+0000', 'last_undegraded': '2024-01-15T18:06:00.152476+0000', 'last_fullsized': '2024-01-15T18:06:00.152476+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.088 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3c is not active+clean 2024-01-15T18:06:06.088 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.165981+0000', 'last_change': '2024-01-15T18:06:00.165981+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.165981+0000', 'last_undegraded': '2024-01-15T18:06:00.165981+0000', 'last_fullsized': '2024-01-15T18:06:00.165981+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:06:06.089 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3b is not active+clean 2024-01-15T18:06:06.089 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.172568+0000', 'last_change': '2024-01-15T18:06:00.172568+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.172568+0000', 'last_undegraded': '2024-01-15T18:06:00.172568+0000', 'last_fullsized': '2024-01-15T18:06:00.172568+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:06:06.089 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3a is not active+clean 2024-01-15T18:06:06.089 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.180507+0000', 'last_change': '2024-01-15T18:06:00.180507+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.180507+0000', 'last_undegraded': '2024-01-15T18:06:00.180507+0000', 'last_fullsized': '2024-01-15T18:06:00.180507+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:06:06.089 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.39 is not active+clean 2024-01-15T18:06:06.089 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:06:00.152476+0000', 'last_change': '2024-01-15T18:06:00.152476+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.152476+0000', 'last_undegraded': '2024-01-15T18:06:00.152476+0000', 'last_fullsized': '2024-01-15T18:06:00.152476+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.089 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.38 is not active+clean 2024-01-15T18:06:06.089 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:06:00.152476+0000', 'last_change': '2024-01-15T18:06:00.152476+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.152476+0000', 'last_undegraded': '2024-01-15T18:06:00.152476+0000', 'last_fullsized': '2024-01-15T18:06:00.152476+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.089 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.37 is not active+clean 2024-01-15T18:06:06.089 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.165821+0000', 'last_change': '2024-01-15T18:06:00.165821+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.165821+0000', 'last_undegraded': '2024-01-15T18:06:00.165821+0000', 'last_fullsized': '2024-01-15T18:06:00.165821+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:06:06.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.36 is not active+clean 2024-01-15T18:06:06.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.165786+0000', 'last_change': '2024-01-15T18:06:00.165786+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.165786+0000', 'last_undegraded': '2024-01-15T18:06:00.165786+0000', 'last_fullsized': '2024-01-15T18:06:00.165786+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:06:06.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.35 is not active+clean 2024-01-15T18:06:06.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:06:00.152476+0000', 'last_change': '2024-01-15T18:06:00.152476+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.152476+0000', 'last_undegraded': '2024-01-15T18:06:00.152476+0000', 'last_fullsized': '2024-01-15T18:06:00.152476+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.34 is not active+clean 2024-01-15T18:06:06.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:06:00.152476+0000', 'last_change': '2024-01-15T18:06:00.152476+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.152476+0000', 'last_undegraded': '2024-01-15T18:06:00.152476+0000', 'last_fullsized': '2024-01-15T18:06:00.152476+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.33 is not active+clean 2024-01-15T18:06:06.090 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.172055+0000', 'last_change': '2024-01-15T18:06:00.172055+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.172055+0000', 'last_undegraded': '2024-01-15T18:06:00.172055+0000', 'last_fullsized': '2024-01-15T18:06:00.172055+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:06:06.090 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.32 is not active+clean 2024-01-15T18:06:06.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.172208+0000', 'last_change': '2024-01-15T18:06:00.172208+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.172208+0000', 'last_undegraded': '2024-01-15T18:06:00.172208+0000', 'last_fullsized': '2024-01-15T18:06:00.172208+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:06:06.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.31 is not active+clean 2024-01-15T18:06:06.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:06:00.152476+0000', 'last_change': '2024-01-15T18:06:00.152476+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.152476+0000', 'last_undegraded': '2024-01-15T18:06:00.152476+0000', 'last_fullsized': '2024-01-15T18:06:00.152476+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.30 is not active+clean 2024-01-15T18:06:06.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:06:00.152476+0000', 'last_change': '2024-01-15T18:06:00.152476+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.152476+0000', 'last_undegraded': '2024-01-15T18:06:00.152476+0000', 'last_fullsized': '2024-01-15T18:06:00.152476+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2f is not active+clean 2024-01-15T18:06:06.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:06:00.152476+0000', 'last_change': '2024-01-15T18:06:00.152476+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.152476+0000', 'last_undegraded': '2024-01-15T18:06:00.152476+0000', 'last_fullsized': '2024-01-15T18:06:00.152476+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2e is not active+clean 2024-01-15T18:06:06.091 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.172461+0000', 'last_change': '2024-01-15T18:06:00.172461+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.172461+0000', 'last_undegraded': '2024-01-15T18:06:00.172461+0000', 'last_fullsized': '2024-01-15T18:06:00.172461+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:06:06.091 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2d is not active+clean 2024-01-15T18:06:06.092 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.179869+0000', 'last_change': '2024-01-15T18:06:00.179869+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.179869+0000', 'last_undegraded': '2024-01-15T18:06:00.179869+0000', 'last_fullsized': '2024-01-15T18:06:00.179869+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:06:06.092 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2c is not active+clean 2024-01-15T18:06:06.092 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.169391+0000', 'last_change': '2024-01-15T18:06:00.169391+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.169391+0000', 'last_undegraded': '2024-01-15T18:06:00.169391+0000', 'last_fullsized': '2024-01-15T18:06:00.169391+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:06:06.092 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2b is not active+clean 2024-01-15T18:06:06.092 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.173714+0000', 'last_change': '2024-01-15T18:06:00.173714+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.173714+0000', 'last_undegraded': '2024-01-15T18:06:00.173714+0000', 'last_fullsized': '2024-01-15T18:06:00.173714+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:06:06.092 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2a is not active+clean 2024-01-15T18:06:06.092 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.167370+0000', 'last_change': '2024-01-15T18:06:00.167370+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.167370+0000', 'last_undegraded': '2024-01-15T18:06:00.167370+0000', 'last_fullsized': '2024-01-15T18:06:00.167370+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:06:06.092 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.29 is not active+clean 2024-01-15T18:06:06.092 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:06:00.152476+0000', 'last_change': '2024-01-15T18:06:00.152476+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.152476+0000', 'last_undegraded': '2024-01-15T18:06:00.152476+0000', 'last_fullsized': '2024-01-15T18:06:00.152476+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.28 is not active+clean 2024-01-15T18:06:06.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:06:00.152476+0000', 'last_change': '2024-01-15T18:06:00.152476+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.152476+0000', 'last_undegraded': '2024-01-15T18:06:00.152476+0000', 'last_fullsized': '2024-01-15T18:06:00.152476+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.27 is not active+clean 2024-01-15T18:06:06.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.166345+0000', 'last_change': '2024-01-15T18:06:00.166345+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.166345+0000', 'last_undegraded': '2024-01-15T18:06:00.166345+0000', 'last_fullsized': '2024-01-15T18:06:00.166345+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:06:06.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.26 is not active+clean 2024-01-15T18:06:06.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.170414+0000', 'last_change': '2024-01-15T18:06:00.170414+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.170414+0000', 'last_undegraded': '2024-01-15T18:06:00.170414+0000', 'last_fullsized': '2024-01-15T18:06:00.170414+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:06:06.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.25 is not active+clean 2024-01-15T18:06:06.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.165628+0000', 'last_change': '2024-01-15T18:06:00.165628+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.165628+0000', 'last_undegraded': '2024-01-15T18:06:00.165628+0000', 'last_fullsized': '2024-01-15T18:06:00.165628+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:06:06.093 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.24 is not active+clean 2024-01-15T18:06:06.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:06:00.152476+0000', 'last_change': '2024-01-15T18:06:00.152476+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.152476+0000', 'last_undegraded': '2024-01-15T18:06:00.152476+0000', 'last_fullsized': '2024-01-15T18:06:00.152476+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.094 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.23 is not active+clean 2024-01-15T18:06:06.094 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.165370+0000', 'last_change': '2024-01-15T18:06:00.165370+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.165370+0000', 'last_undegraded': '2024-01-15T18:06:00.165370+0000', 'last_fullsized': '2024-01-15T18:06:00.165370+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:06:06.094 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.22 is not active+clean 2024-01-15T18:06:06.094 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:06:00.152476+0000', 'last_change': '2024-01-15T18:06:00.152476+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.152476+0000', 'last_undegraded': '2024-01-15T18:06:00.152476+0000', 'last_fullsized': '2024-01-15T18:06:00.152476+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.094 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.21 is not active+clean 2024-01-15T18:06:06.094 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.170608+0000', 'last_change': '2024-01-15T18:06:00.170608+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.170608+0000', 'last_undegraded': '2024-01-15T18:06:00.170608+0000', 'last_fullsized': '2024-01-15T18:06:00.170608+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:06:06.094 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.20 is not active+clean 2024-01-15T18:06:06.094 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:06:00.152476+0000', 'last_change': '2024-01-15T18:06:00.152476+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.152476+0000', 'last_undegraded': '2024-01-15T18:06:00.152476+0000', 'last_fullsized': '2024-01-15T18:06:00.152476+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.094 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1e is not active+clean 2024-01-15T18:06:06.094 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:06:00.152476+0000', 'last_change': '2024-01-15T18:06:00.152476+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.152476+0000', 'last_undegraded': '2024-01-15T18:06:00.152476+0000', 'last_fullsized': '2024-01-15T18:06:00.152476+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.095 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1f is not active+clean 2024-01-15T18:06:06.095 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:05:59.161092+0000', 'last_change': '2024-01-15T18:05:59.161092+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:05:59.161092+0000', 'last_undegraded': '2024-01-15T18:05:59.161092+0000', 'last_fullsized': '2024-01-15T18:05:59.161092+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:06:06.095 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1f is not active+clean 2024-01-15T18:06:06.095 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.171119+0000', 'last_change': '2024-01-15T18:06:00.171119+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.171119+0000', 'last_undegraded': '2024-01-15T18:06:00.171119+0000', 'last_fullsized': '2024-01-15T18:06:00.171119+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:06:06.095 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1e is not active+clean 2024-01-15T18:06:06.095 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:05:59.145608+0000', 'last_change': '2024-01-15T18:05:59.145608+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:05:59.145608+0000', 'last_undegraded': '2024-01-15T18:05:59.145608+0000', 'last_fullsized': '2024-01-15T18:05:59.145608+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.095 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1c is not active+clean 2024-01-15T18:06:06.095 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.164402+0000', 'last_change': '2024-01-15T18:06:00.164402+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.164402+0000', 'last_undegraded': '2024-01-15T18:06:00.164402+0000', 'last_fullsized': '2024-01-15T18:06:00.164402+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:06:06.095 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1d is not active+clean 2024-01-15T18:06:06.095 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1d', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-01-15T18:06:00.180303+0000', 'last_change': '2024-01-15T18:06:00.180303+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.180303+0000', 'last_undegraded': '2024-01-15T18:06:00.180303+0000', 'last_fullsized': '2024-01-15T18:06:00.180303+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-17T00:38:07.317901+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': ['3', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:06:06.096 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1d is not active+clean 2024-01-15T18:06:06.096 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.165003+0000', 'last_change': '2024-01-15T18:06:00.165003+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.165003+0000', 'last_undegraded': '2024-01-15T18:06:00.165003+0000', 'last_fullsized': '2024-01-15T18:06:00.165003+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:06:06.096 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1c is not active+clean 2024-01-15T18:06:06.096 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1c', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-01-15T18:06:00.168374+0000', 'last_change': '2024-01-15T18:06:00.168374+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.168374+0000', 'last_undegraded': '2024-01-15T18:06:00.168374+0000', 'last_fullsized': '2024-01-15T18:06:00.168374+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-16T18:31:30.665626+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:06:06.096 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1a is not active+clean 2024-01-15T18:06:06.096 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:06:00.152476+0000', 'last_change': '2024-01-15T18:06:00.152476+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.152476+0000', 'last_undegraded': '2024-01-15T18:06:00.152476+0000', 'last_fullsized': '2024-01-15T18:06:00.152476+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.096 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1b is not active+clean 2024-01-15T18:06:06.096 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1b', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-01-15T18:06:00.174622+0000', 'last_change': '2024-01-15T18:06:00.174622+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.174622+0000', 'last_undegraded': '2024-01-15T18:06:00.174622+0000', 'last_fullsized': '2024-01-15T18:06:00.174622+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-16T21:39:25.962651+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': ['4', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:06:06.096 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1b is not active+clean 2024-01-15T18:06:06.096 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:06:00.152476+0000', 'last_change': '2024-01-15T18:06:00.152476+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.152476+0000', 'last_undegraded': '2024-01-15T18:06:00.152476+0000', 'last_fullsized': '2024-01-15T18:06:00.152476+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.097 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1a is not active+clean 2024-01-15T18:06:06.097 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1a', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-01-15T18:06:00.164626+0000', 'last_change': '2024-01-15T18:06:00.164626+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.164626+0000', 'last_undegraded': '2024-01-15T18:06:00.164626+0000', 'last_fullsized': '2024-01-15T18:06:00.164626+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-17T00:48:52.588812+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': ['4', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:06:06.097 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.18 is not active+clean 2024-01-15T18:06:06.097 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.164265+0000', 'last_change': '2024-01-15T18:06:00.164265+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.164265+0000', 'last_undegraded': '2024-01-15T18:06:00.164265+0000', 'last_fullsized': '2024-01-15T18:06:00.164265+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:06:06.097 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.19 is not active+clean 2024-01-15T18:06:06.097 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.19', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-01-15T18:06:00.171075+0000', 'last_change': '2024-01-15T18:06:00.171075+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.171075+0000', 'last_undegraded': '2024-01-15T18:06:00.171075+0000', 'last_fullsized': '2024-01-15T18:06:00.171075+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-16T23:40:56.123776+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['3', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:06:06.097 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.19 is not active+clean 2024-01-15T18:06:06.097 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.164936+0000', 'last_change': '2024-01-15T18:06:00.164936+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.164936+0000', 'last_undegraded': '2024-01-15T18:06:00.164936+0000', 'last_fullsized': '2024-01-15T18:06:00.164936+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:06:06.097 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.18 is not active+clean 2024-01-15T18:06:06.097 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.18', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-01-15T18:06:00.173385+0000', 'last_change': '2024-01-15T18:06:00.173385+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.173385+0000', 'last_undegraded': '2024-01-15T18:06:00.173385+0000', 'last_fullsized': '2024-01-15T18:06:00.173385+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-16T18:35:19.522019+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': ['4', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:06:06.098 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.16 is not active+clean 2024-01-15T18:06:06.098 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:06:00.152476+0000', 'last_change': '2024-01-15T18:06:00.152476+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.152476+0000', 'last_undegraded': '2024-01-15T18:06:00.152476+0000', 'last_fullsized': '2024-01-15T18:06:00.152476+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.098 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.17 is not active+clean 2024-01-15T18:06:06.098 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.17', 'version': "0'0", 'reported_seq': 4, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-01-15T18:06:00.183420+0000', 'last_change': '2024-01-15T18:06:00.171516+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.183420+0000', 'last_undegraded': '2024-01-15T18:06:00.183420+0000', 'last_fullsized': '2024-01-15T18:06:00.183420+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-17T03:31:04.425248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': ['3', '7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:06:06.098 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.17 is not active+clean 2024-01-15T18:06:06.098 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.172541+0000', 'last_change': '2024-01-15T18:06:00.172541+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.172541+0000', 'last_undegraded': '2024-01-15T18:06:00.172541+0000', 'last_fullsized': '2024-01-15T18:06:00.172541+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:06:06.098 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.16 is not active+clean 2024-01-15T18:06:06.098 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:05:59.145608+0000', 'last_change': '2024-01-15T18:05:59.145608+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:05:59.145608+0000', 'last_undegraded': '2024-01-15T18:05:59.145608+0000', 'last_fullsized': '2024-01-15T18:05:59.145608+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.098 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.14 is not active+clean 2024-01-15T18:06:06.098 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.179315+0000', 'last_change': '2024-01-15T18:06:00.179315+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.179315+0000', 'last_undegraded': '2024-01-15T18:06:00.179315+0000', 'last_fullsized': '2024-01-15T18:06:00.179315+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:06:06.098 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.15 is not active+clean 2024-01-15T18:06:06.099 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:05:59.156144+0000', 'last_change': '2024-01-15T18:05:59.156144+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:05:59.156144+0000', 'last_undegraded': '2024-01-15T18:05:59.156144+0000', 'last_fullsized': '2024-01-15T18:05:59.156144+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T18:06:06.099 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.15 is not active+clean 2024-01-15T18:06:06.099 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:06:00.152476+0000', 'last_change': '2024-01-15T18:06:00.152476+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.152476+0000', 'last_undegraded': '2024-01-15T18:06:00.152476+0000', 'last_fullsized': '2024-01-15T18:06:00.152476+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.099 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.14 is not active+clean 2024-01-15T18:06:06.099 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.14', 'version': "0'0", 'reported_seq': 4, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-01-15T18:06:00.179743+0000', 'last_change': '2024-01-15T18:06:00.171716+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.179743+0000', 'last_undegraded': '2024-01-15T18:06:00.179743+0000', 'last_fullsized': '2024-01-15T18:06:00.179743+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-16T19:26:09.652268+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': ['3', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:06:06.099 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.12 is not active+clean 2024-01-15T18:06:06.099 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.165709+0000', 'last_change': '2024-01-15T18:06:00.165709+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.165709+0000', 'last_undegraded': '2024-01-15T18:06:00.165709+0000', 'last_fullsized': '2024-01-15T18:06:00.165709+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:06:06.099 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.13 is not active+clean 2024-01-15T18:06:06.099 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.13', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-01-15T18:06:00.164330+0000', 'last_change': '2024-01-15T18:06:00.164330+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.164330+0000', 'last_undegraded': '2024-01-15T18:06:00.164330+0000', 'last_fullsized': '2024-01-15T18:06:00.164330+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-16T23:03:20.544076+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': ['4', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:06:06.099 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.13 is not active+clean 2024-01-15T18:06:06.100 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.170139+0000', 'last_change': '2024-01-15T18:06:00.170139+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.170139+0000', 'last_undegraded': '2024-01-15T18:06:00.170139+0000', 'last_fullsized': '2024-01-15T18:06:00.170139+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:06:06.100 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.12 is not active+clean 2024-01-15T18:06:06.100 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.12', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-01-15T18:06:00.162418+0000', 'last_change': '2024-01-15T18:06:00.162418+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.162418+0000', 'last_undegraded': '2024-01-15T18:06:00.162418+0000', 'last_fullsized': '2024-01-15T18:06:00.162418+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-17T01:11:56.736968+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': ['1', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:06:06.100 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.10 is not active+clean 2024-01-15T18:06:06.100 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:06:00.152476+0000', 'last_change': '2024-01-15T18:06:00.152476+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.152476+0000', 'last_undegraded': '2024-01-15T18:06:00.152476+0000', 'last_fullsized': '2024-01-15T18:06:00.152476+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.100 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.11 is not active+clean 2024-01-15T18:06:06.100 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.11', 'version': "0'0", 'reported_seq': 4, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-01-15T18:06:00.183486+0000', 'last_change': '2024-01-15T18:06:00.170635+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.183486+0000', 'last_undegraded': '2024-01-15T18:06:00.183486+0000', 'last_fullsized': '2024-01-15T18:06:00.183486+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-17T01:11:56.736968+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': ['3', '7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:06:06.100 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.11 is not active+clean 2024-01-15T18:06:06.100 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.168794+0000', 'last_change': '2024-01-15T18:06:00.168794+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.168794+0000', 'last_undegraded': '2024-01-15T18:06:00.168794+0000', 'last_fullsized': '2024-01-15T18:06:00.168794+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:06:06.101 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.10 is not active+clean 2024-01-15T18:06:06.101 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.10', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-01-15T18:06:00.179395+0000', 'last_change': '2024-01-15T18:06:00.179395+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.179395+0000', 'last_undegraded': '2024-01-15T18:06:00.179395+0000', 'last_fullsized': '2024-01-15T18:06:00.179395+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-17T05:51:46.033055+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': ['3', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:06:06.101 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.e is not active+clean 2024-01-15T18:06:06.101 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.164841+0000', 'last_change': '2024-01-15T18:06:00.164841+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.164841+0000', 'last_undegraded': '2024-01-15T18:06:00.164841+0000', 'last_fullsized': '2024-01-15T18:06:00.164841+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:06:06.101 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.f is not active+clean 2024-01-15T18:06:06.101 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.f', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-01-15T18:06:00.162240+0000', 'last_change': '2024-01-15T18:06:00.162240+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.162240+0000', 'last_undegraded': '2024-01-15T18:06:00.162240+0000', 'last_fullsized': '2024-01-15T18:06:00.162240+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-16T23:08:08.490734+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': ['1', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:06:06.101 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.f is not active+clean 2024-01-15T18:06:06.101 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:06:00.152476+0000', 'last_change': '2024-01-15T18:06:00.152476+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.152476+0000', 'last_undegraded': '2024-01-15T18:06:00.152476+0000', 'last_fullsized': '2024-01-15T18:06:00.152476+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.101 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.e is not active+clean 2024-01-15T18:06:06.101 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.e', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-01-15T18:06:00.166559+0000', 'last_change': '2024-01-15T18:06:00.166559+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.166559+0000', 'last_undegraded': '2024-01-15T18:06:00.166559+0000', 'last_fullsized': '2024-01-15T18:06:00.166559+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-16T18:11:04.167913+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': ['4', '6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:06:06.101 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.c is not active+clean 2024-01-15T18:06:06.102 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.164909+0000', 'last_change': '2024-01-15T18:06:00.164909+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.164909+0000', 'last_undegraded': '2024-01-15T18:06:00.164909+0000', 'last_fullsized': '2024-01-15T18:06:00.164909+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:06:06.102 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.d is not active+clean 2024-01-15T18:06:06.102 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.168628+0000', 'last_change': '2024-01-15T18:06:00.168628+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.168628+0000', 'last_undegraded': '2024-01-15T18:06:00.168628+0000', 'last_fullsized': '2024-01-15T18:06:00.168628+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [7], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:06:06.102 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.c is not active+clean 2024-01-15T18:06:06.102 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.c', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-01-15T18:06:00.164364+0000', 'last_change': '2024-01-15T18:06:00.164364+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.164364+0000', 'last_undegraded': '2024-01-15T18:06:00.164364+0000', 'last_fullsized': '2024-01-15T18:06:00.164364+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-16T19:59:22.476704+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': ['4', '3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:06:06.102 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.a is not active+clean 2024-01-15T18:06:06.102 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.168573+0000', 'last_change': '2024-01-15T18:06:00.168573+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.168573+0000', 'last_undegraded': '2024-01-15T18:06:00.168573+0000', 'last_fullsized': '2024-01-15T18:06:00.168573+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:06:06.102 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.b is not active+clean 2024-01-15T18:06:06.102 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:05:59.145608+0000', 'last_change': '2024-01-15T18:05:59.145608+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:05:59.145608+0000', 'last_undegraded': '2024-01-15T18:05:59.145608+0000', 'last_fullsized': '2024-01-15T18:05:59.145608+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.102 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.b is not active+clean 2024-01-15T18:06:06.103 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.168105+0000', 'last_change': '2024-01-15T18:06:00.168105+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.168105+0000', 'last_undegraded': '2024-01-15T18:06:00.168105+0000', 'last_fullsized': '2024-01-15T18:06:00.168105+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:06:06.103 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.a is not active+clean 2024-01-15T18:06:06.103 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:05:59.161004+0000', 'last_change': '2024-01-15T18:05:59.161004+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:05:59.161004+0000', 'last_undegraded': '2024-01-15T18:05:59.161004+0000', 'last_fullsized': '2024-01-15T18:05:59.161004+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:06:06.103 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.8 is not active+clean 2024-01-15T18:06:06.103 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.168020+0000', 'last_change': '2024-01-15T18:06:00.168020+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.168020+0000', 'last_undegraded': '2024-01-15T18:06:00.168020+0000', 'last_fullsized': '2024-01-15T18:06:00.168020+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:06:06.103 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.9 is not active+clean 2024-01-15T18:06:06.103 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.9', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-01-15T18:06:00.165635+0000', 'last_change': '2024-01-15T18:06:00.165635+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.165635+0000', 'last_undegraded': '2024-01-15T18:06:00.165635+0000', 'last_fullsized': '2024-01-15T18:06:00.165635+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-16T18:31:30.665626+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': ['4', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:06:06.103 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.9 is not active+clean 2024-01-15T18:06:06.103 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:06:00.152476+0000', 'last_change': '2024-01-15T18:06:00.152476+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.152476+0000', 'last_undegraded': '2024-01-15T18:06:00.152476+0000', 'last_fullsized': '2024-01-15T18:06:00.152476+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.103 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.8 is not active+clean 2024-01-15T18:06:06.103 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:05:59.155857+0000', 'last_change': '2024-01-15T18:05:59.155857+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:05:59.155857+0000', 'last_undegraded': '2024-01-15T18:05:59.155857+0000', 'last_fullsized': '2024-01-15T18:05:59.155857+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T18:06:06.104 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.6 is not active+clean 2024-01-15T18:06:06.104 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.168806+0000', 'last_change': '2024-01-15T18:06:00.168806+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.168806+0000', 'last_undegraded': '2024-01-15T18:06:00.168806+0000', 'last_fullsized': '2024-01-15T18:06:00.168806+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:06:06.104 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.7 is not active+clean 2024-01-15T18:06:06.104 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:06:00.152476+0000', 'last_change': '2024-01-15T18:06:00.152476+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.152476+0000', 'last_undegraded': '2024-01-15T18:06:00.152476+0000', 'last_fullsized': '2024-01-15T18:06:00.152476+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.104 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.6 is not active+clean 2024-01-15T18:06:06.104 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:05:59.145608+0000', 'last_change': '2024-01-15T18:05:59.145608+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:05:59.145608+0000', 'last_undegraded': '2024-01-15T18:05:59.145608+0000', 'last_fullsized': '2024-01-15T18:05:59.145608+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.104 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.5 is not active+clean 2024-01-15T18:06:06.104 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:06:00.152476+0000', 'last_change': '2024-01-15T18:06:00.152476+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.152476+0000', 'last_undegraded': '2024-01-15T18:06:00.152476+0000', 'last_fullsized': '2024-01-15T18:06:00.152476+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.104 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.4 is not active+clean 2024-01-15T18:06:06.105 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.4', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 32, 'state': 'creating+activating', 'last_fresh': '2024-01-15T18:06:00.162200+0000', 'last_change': '2024-01-15T18:06:00.162200+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.162200+0000', 'last_undegraded': '2024-01-15T18:06:00.162200+0000', 'last_fullsized': '2024-01-15T18:06:00.162200+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-01-16T19:46:52.165002+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': ['1', '2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:06:06.105 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2 is not active+clean 2024-01-15T18:06:06.105 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.172639+0000', 'last_change': '2024-01-15T18:06:00.172639+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.172639+0000', 'last_undegraded': '2024-01-15T18:06:00.172639+0000', 'last_fullsized': '2024-01-15T18:06:00.172639+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:06:06.105 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.3 is not active+clean 2024-01-15T18:06:06.105 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:05:59.145608+0000', 'last_change': '2024-01-15T18:05:59.145608+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:05:59.145608+0000', 'last_undegraded': '2024-01-15T18:05:59.145608+0000', 'last_fullsized': '2024-01-15T18:05:59.145608+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.105 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3 is not active+clean 2024-01-15T18:06:06.105 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:06:00.152476+0000', 'last_change': '2024-01-15T18:06:00.152476+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.152476+0000', 'last_undegraded': '2024-01-15T18:06:00.152476+0000', 'last_fullsized': '2024-01-15T18:06:00.152476+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.105 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.0 is not active+clean 2024-01-15T18:06:06.105 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.170431+0000', 'last_change': '2024-01-15T18:06:00.170431+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.170431+0000', 'last_undegraded': '2024-01-15T18:06:00.170431+0000', 'last_fullsized': '2024-01-15T18:06:00.170431+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:06:06.105 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1 is not active+clean 2024-01-15T18:06:06.105 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:06:00.166107+0000', 'last_change': '2024-01-15T18:06:00.166107+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.166107+0000', 'last_undegraded': '2024-01-15T18:06:00.166107+0000', 'last_fullsized': '2024-01-15T18:06:00.166107+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:06:06.106 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.4 is not active+clean 2024-01-15T18:06:06.106 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:06:00.152476+0000', 'last_change': '2024-01-15T18:06:00.152476+0000', 'last_active': '2024-01-15T18:06:00.152476+0000', 'last_peered': '2024-01-15T18:06:00.152476+0000', 'last_clean': '2024-01-15T18:06:00.152476+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:06:00.152476+0000', 'last_undegraded': '2024-01-15T18:06:00.152476+0000', 'last_fullsized': '2024-01-15T18:06:00.152476+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'last_clean_scrub_stamp': '2024-01-15T18:06:00.152476+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:06:06.106 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.5 is not active+clean 2024-01-15T18:06:06.106 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:05:59.161012+0000', 'last_change': '2024-01-15T18:05:59.161012+0000', 'last_active': '2024-01-15T18:05:59.145608+0000', 'last_peered': '2024-01-15T18:05:59.145608+0000', 'last_clean': '2024-01-15T18:05:59.145608+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:05:59.161012+0000', 'last_undegraded': '2024-01-15T18:05:59.161012+0000', 'last_fullsized': '2024-01-15T18:05:59.161012+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'last_clean_scrub_stamp': '2024-01-15T18:05:59.145608+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:06:06.106 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-15T18:06:06.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.262+0000 7f15fbe00640 1 Processor -- start 2024-01-15T18:06:06.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.262+0000 7f15fbe00640 1 -- start start 2024-01-15T18:06:06.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.266+0000 7f15fbe00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f15f40af720 0x7f15f4168540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:06.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.266+0000 7f15fbe00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f40ae280 con 0x7f15f4160af0 2024-01-15T18:06:06.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.266+0000 7f15fbe00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f406c420 con 0x7f15f40cfaa0 2024-01-15T18:06:06.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.266+0000 7f15fbe00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15f40adc50 con 0x7f15f40af720 2024-01-15T18:06:06.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.266+0000 7f15f9400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f15f4160af0 0x7f15f406b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49672/0 (socket says 172.21.15.27:49672) 2024-01-15T18:06:06.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.266+0000 7f15f9400640 1 -- 172.21.15.27:0/3197929277 learned_addr learned my addr 172.21.15.27:0/3197929277 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:06:06.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.266+0000 7f15f9e00640 1 --2- 172.21.15.27:0/3197929277 >> v2:172.21.15.93:3300/0 conn(0x7f15f40af720 0x7f15f4168540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:06.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.266+0000 7f15f3e00640 1 -- 172.21.15.27:0/3197929277 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 988017615 0 0) 0x7f15f40ae280 con 0x7f15f4160af0 2024-01-15T18:06:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.266+0000 7f15f3e00640 1 -- 172.21.15.27:0/3197929277 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15d4003530 con 0x7f15f4160af0 2024-01-15T18:06:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.266+0000 7f15f3e00640 1 -- 172.21.15.27:0/3197929277 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2301287111 0 0) 0x7f15f406c420 con 0x7f15f40cfaa0 2024-01-15T18:06:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.266+0000 7f15f3e00640 1 -- 172.21.15.27:0/3197929277 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f40ae280 con 0x7f15f40cfaa0 2024-01-15T18:06:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.266+0000 7f15f9e00640 1 -- 172.21.15.27:0/3197929277 >> v1:172.21.15.93:6789/0 conn(0x7f15f40cfaa0 legacy=0x7f15f406bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.266+0000 7f15f9e00640 1 -- 172.21.15.27:0/3197929277 >> v1:172.21.15.27:6789/0 conn(0x7f15f4160af0 legacy=0x7f15f406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.266+0000 7f15f9e00640 1 -- 172.21.15.27:0/3197929277 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f40cff80 con 0x7f15f40af720 2024-01-15T18:06:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.270+0000 7f15f9e00640 1 --2- 172.21.15.27:0/3197929277 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f15f40af720 0x7f15f4168540 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f15ec009d90 tx=0x7f15ec01c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1183134322c17ce server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:06.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.270+0000 7f15f3e00640 1 -- 172.21.15.27:0/3197929277 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ec0a5900 con 0x7f15f40af720 2024-01-15T18:06:06.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.270+0000 7f15f3e00640 1 -- 172.21.15.27:0/3197929277 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15ec0a5a60 con 0x7f15f40af720 2024-01-15T18:06:06.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.270+0000 7f15f3e00640 1 -- 172.21.15.27:0/3197929277 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ec0a5df0 con 0x7f15f40af720 2024-01-15T18:06:06.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.270+0000 7f15fbe00640 1 -- 172.21.15.27:0/3197929277 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f15f40af720 msgr2=0x7f15f4168540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:06.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.270+0000 7f15fbe00640 1 --2- 172.21.15.27:0/3197929277 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f15f40af720 0x7f15f4168540 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f15ec009d90 tx=0x7f15ec01c950 comp rx=0 tx=0).stop 2024-01-15T18:06:06.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.270+0000 7f15fbe00640 1 -- 172.21.15.27:0/3197929277 shutdown_connections 2024-01-15T18:06:06.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.270+0000 7f15fbe00640 1 --2- 172.21.15.27:0/3197929277 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f15f40af720 0x7f15f4168540 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:06.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.270+0000 7f15fbe00640 1 -- 172.21.15.27:0/3197929277 >> 172.21.15.27:0/3197929277 conn(0x7f15f40d0f20 msgr2=0x7f15f40d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:06.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.270+0000 7f15fbe00640 1 -- 172.21.15.27:0/3197929277 shutdown_connections 2024-01-15T18:06:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.270+0000 7f15fbe00640 1 -- 172.21.15.27:0/3197929277 wait complete. 2024-01-15T18:06:06.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.274+0000 7f15fbe00640 1 Processor -- start 2024-01-15T18:06:06.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.274+0000 7f15fbe00640 1 -- start start 2024-01-15T18:06:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.274+0000 7f15fbe00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f15f40af720 0x7f15f4154220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.274+0000 7f15fbe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f15f40cfaa0 0x7f15f40b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.274+0000 7f15fbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15f4160af0 0x7f15f40ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.274+0000 7f15fbe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f4154760 con 0x7f15f4160af0 2024-01-15T18:06:06.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.274+0000 7f15fbe00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f40b5480 con 0x7f15f40af720 2024-01-15T18:06:06.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.274+0000 7f15fbe00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15f40b45f0 con 0x7f15f40cfaa0 2024-01-15T18:06:06.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.274+0000 7f15f9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15f4160af0 0x7f15f40ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:06.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.274+0000 7f15f9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15f4160af0 0x7f15f40ccca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56634/0 (socket says 172.21.15.27:56634) 2024-01-15T18:06:06.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.274+0000 7f15f9e00640 1 -- 172.21.15.27:0/3754840702 learned_addr learned my addr 172.21.15.27:0/3754840702 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:06.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.274+0000 7f15f9400640 1 --2- 172.21.15.27:0/3754840702 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f15f40af720 0x7f15f4154220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:06.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.274+0000 7f15f8a00640 1 --2- 172.21.15.27:0/3754840702 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f15f40cfaa0 0x7f15f40b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:06.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.274+0000 7f15f9e00640 1 -- 172.21.15.27:0/3754840702 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f15f40cfaa0 msgr2=0x7f15f40b29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:06.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.274+0000 7f15f9e00640 1 --2- 172.21.15.27:0/3754840702 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f15f40cfaa0 0x7f15f40b29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:06.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.274+0000 7f15f9e00640 1 -- 172.21.15.27:0/3754840702 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f15f40af720 msgr2=0x7f15f4154220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:06.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.274+0000 7f15f9e00640 1 --2- 172.21.15.27:0/3754840702 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f15f40af720 0x7f15f4154220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.274+0000 7f15f9e00640 1 -- 172.21.15.27:0/3754840702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ec009a70 con 0x7f15f4160af0 2024-01-15T18:06:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.274+0000 7f15f9e00640 1 --2- 172.21.15.27:0/3754840702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15f4160af0 0x7f15f40ccca0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f15ec0b0040 tx=0x7f15ec0196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.274+0000 7f15f1600640 1 -- 172.21.15.27:0/3754840702 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ec0b9650 con 0x7f15f4160af0 2024-01-15T18:06:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.274+0000 7f15f1600640 1 -- 172.21.15.27:0/3754840702 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15ec0b97b0 con 0x7f15f4160af0 2024-01-15T18:06:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.274+0000 7f15f1600640 1 -- 172.21.15.27:0/3754840702 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15ec0b9b60 con 0x7f15f4160af0 2024-01-15T18:06:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.274+0000 7f15fbe00640 1 -- 172.21.15.27:0/3754840702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15f40b4870 con 0x7f15f4160af0 2024-01-15T18:06:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.274+0000 7f15fbe00640 1 -- 172.21.15.27:0/3754840702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f40b4e30 con 0x7f15f4160af0 2024-01-15T18:06:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.274+0000 7f15fbe00640 1 -- 172.21.15.27:0/3754840702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15f404e850 con 0x7f15f4160af0 2024-01-15T18:06:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.278+0000 7f15f1600640 1 -- 172.21.15.27:0/3754840702 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f15ec0b9cc0 con 0x7f15f4160af0 2024-01-15T18:06:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.278+0000 7f15f1600640 1 --2- 172.21.15.27:0/3754840702 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f15cc0668f0 0x7f15cc068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.278+0000 7f15f1600640 1 -- 172.21.15.27:0/3754840702 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f15ec11f500 con 0x7f15f4160af0 2024-01-15T18:06:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.278+0000 7f15f9400640 1 --2- 172.21.15.27:0/3754840702 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f15cc0668f0 0x7f15cc068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:06.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.278+0000 7f15f9400640 1 --2- 172.21.15.27:0/3754840702 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f15cc0668f0 0x7f15cc068db0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f15e4007fd0 tx=0x7f15e4007d60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:06.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.282+0000 7f15f1600640 1 -- 172.21.15.27:0/3754840702 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f15ec0ec7e0 con 0x7f15f4160af0 2024-01-15T18:06:06.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.494+0000 7f15fbe00640 1 -- 172.21.15.27:0/3754840702 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f15f4072620 con 0x7f15f4160af0 2024-01-15T18:06:06.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.494+0000 7f15f1600640 1 -- 172.21.15.27:0/3754840702 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1172 (secure 0 0 0) 0x7f15ec0efe00 con 0x7f15f4160af0 2024-01-15T18:06:06.500 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:06:06.500 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":93,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":34,"num_osds":8,"num_up_osds":8,"osd_up_since":1705341884,"num_in_osds":8,"osd_in_since":1705341875,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":229892096,"bytes_avail":772864221184,"bytes_total":773094113280,"write_bytes_sec":2388,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":17,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"a","status":"up:active","gid":5232}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-01-15T18:05:54.151625+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-01-15T18:06:06.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.494+0000 7f15fbe00640 1 -- 172.21.15.27:0/3754840702 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f15cc0668f0 msgr2=0x7f15cc068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:06.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.494+0000 7f15fbe00640 1 --2- 172.21.15.27:0/3754840702 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f15cc0668f0 0x7f15cc068db0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f15e4007fd0 tx=0x7f15e4007d60 comp rx=0 tx=0).stop 2024-01-15T18:06:06.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.494+0000 7f15fbe00640 1 -- 172.21.15.27:0/3754840702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15f4160af0 msgr2=0x7f15f40ccca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:06.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.494+0000 7f15fbe00640 1 --2- 172.21.15.27:0/3754840702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15f4160af0 0x7f15f40ccca0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f15ec0b0040 tx=0x7f15ec0196f0 comp rx=0 tx=0).stop 2024-01-15T18:06:06.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.494+0000 7f15fbe00640 1 -- 172.21.15.27:0/3754840702 shutdown_connections 2024-01-15T18:06:06.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.494+0000 7f15fbe00640 1 --2- 172.21.15.27:0/3754840702 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f15cc0668f0 0x7f15cc068db0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:06.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.494+0000 7f15fbe00640 1 --2- 172.21.15.27:0/3754840702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15f4160af0 0x7f15f40ccca0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:06.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.494+0000 7f15fbe00640 1 --2- 172.21.15.27:0/3754840702 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f15f40cfaa0 0x7f15f40b29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:06.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.494+0000 7f15fbe00640 1 --2- 172.21.15.27:0/3754840702 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f15f40af720 0x7f15f4154220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:06.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.494+0000 7f15fbe00640 1 -- 172.21.15.27:0/3754840702 >> 172.21.15.27:0/3754840702 conn(0x7f15f40d0f20 msgr2=0x7f15f40cac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:06.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.494+0000 7f15fbe00640 1 -- 172.21.15.27:0/3754840702 shutdown_connections 2024-01-15T18:06:06.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.494+0000 7f15fbe00640 1 -- 172.21.15.27:0/3754840702 wait complete. 2024-01-15T18:06:06.522 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-01-15T18:06:06.522 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:06:06.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.658+0000 7f63d1e00640 1 Processor -- start 2024-01-15T18:06:06.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.658+0000 7f63d1e00640 1 -- start start 2024-01-15T18:06:06.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.658+0000 7f63d1e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f63cc15e730 0x7f63cc0cb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:06.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.658+0000 7f63d1e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f63cc06b610 0x7f63cc06cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.658+0000 7f63d1e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63cc06e450 con 0x7f63cc0aea00 2024-01-15T18:06:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.658+0000 7f63d1e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63cc06e980 con 0x7f63cc15e730 2024-01-15T18:06:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.658+0000 7f63d1e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63cc0c9bf0 con 0x7f63cc06b610 2024-01-15T18:06:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.658+0000 7f63caa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f63cc15e730 0x7f63cc0cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.658+0000 7f63caa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f63cc15e730 0x7f63cc0cb970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56642/0 (socket says 172.21.15.27:56642) 2024-01-15T18:06:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.658+0000 7f63caa00640 1 -- 172.21.15.27:0/1849963039 learned_addr learned my addr 172.21.15.27:0/1849963039 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.658+0000 7f63cb400640 1 --2- 172.21.15.27:0/1849963039 >> v2:172.21.15.93:3300/0 conn(0x7f63cc06b610 0x7f63cc06cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.658+0000 7f63caa00640 1 -- 172.21.15.27:0/1849963039 >> v2:172.21.15.93:3300/0 conn(0x7f63cc06b610 msgr2=0x7f63cc06cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.658+0000 7f63caa00640 1 --2- 172.21.15.27:0/1849963039 >> v2:172.21.15.93:3300/0 conn(0x7f63cc06b610 0x7f63cc06cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.658+0000 7f63caa00640 1 -- 172.21.15.27:0/1849963039 >> v1:172.21.15.93:6789/0 conn(0x7f63cc0aea00 legacy=0x7f63cc06af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:06.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.658+0000 7f63caa00640 1 -- 172.21.15.27:0/1849963039 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63cc0c9ed0 con 0x7f63cc15e730 2024-01-15T18:06:06.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.662+0000 7f63caa00640 1 --2- 172.21.15.27:0/1849963039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63cc15e730 0x7f63cc0cb970 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f63cc0ccfe0 tx=0x7f63bc01c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e30d8a319340b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:06.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.662+0000 7f63c9600640 1 -- 172.21.15.27:0/1849963039 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc0ac630 con 0x7f63cc15e730 2024-01-15T18:06:06.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.662+0000 7f63c9600640 1 -- 172.21.15.27:0/1849963039 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63bc0ac790 con 0x7f63cc15e730 2024-01-15T18:06:06.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.662+0000 7f63c9600640 1 -- 172.21.15.27:0/1849963039 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc0acb20 con 0x7f63cc15e730 2024-01-15T18:06:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.662+0000 7f63d1e00640 1 -- 172.21.15.27:0/1849963039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63cc15e730 msgr2=0x7f63cc0cb970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.662+0000 7f63d1e00640 1 --2- 172.21.15.27:0/1849963039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63cc15e730 0x7f63cc0cb970 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f63cc0ccfe0 tx=0x7f63bc01c3b0 comp rx=0 tx=0).stop 2024-01-15T18:06:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.662+0000 7f63d1e00640 1 -- 172.21.15.27:0/1849963039 shutdown_connections 2024-01-15T18:06:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.662+0000 7f63d1e00640 1 --2- 172.21.15.27:0/1849963039 >> v2:172.21.15.93:3300/0 conn(0x7f63cc06b610 0x7f63cc06cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.662+0000 7f63d1e00640 1 --2- 172.21.15.27:0/1849963039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63cc15e730 0x7f63cc0cb970 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.662+0000 7f63d1e00640 1 -- 172.21.15.27:0/1849963039 >> 172.21.15.27:0/1849963039 conn(0x7f63cc0cbf80 msgr2=0x7f63cc0cc360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.662+0000 7f63d1e00640 1 -- 172.21.15.27:0/1849963039 shutdown_connections 2024-01-15T18:06:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.662+0000 7f63d1e00640 1 -- 172.21.15.27:0/1849963039 wait complete. 2024-01-15T18:06:06.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63d1e00640 1 Processor -- start 2024-01-15T18:06:06.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63d1e00640 1 -- start start 2024-01-15T18:06:06.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63d1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63cc06b610 0x7f63cc14f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:06.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63d1e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f63cc0aea00 0x7f63cc0ad390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:06.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63d1e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63cc15e730 0x7f63cc0ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:06.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63d1e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc14fc80 con 0x7f63cc06b610 2024-01-15T18:06:06.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63d1e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc152470 con 0x7f63cc15e730 2024-01-15T18:06:06.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63d1e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc152750 con 0x7f63cc0aea00 2024-01-15T18:06:06.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63caa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63cc06b610 0x7f63cc14f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:06.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63caa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63cc06b610 0x7f63cc14f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56648/0 (socket says 172.21.15.27:56648) 2024-01-15T18:06:06.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63caa00640 1 -- 172.21.15.27:0/3268716654 learned_addr learned my addr 172.21.15.27:0/3268716654 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:06.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63cb400640 1 --2- 172.21.15.27:0/3268716654 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63cc15e730 0x7f63cc0ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:06.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63caa00640 1 -- 172.21.15.27:0/3268716654 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f63cc0aea00 msgr2=0x7f63cc0ad390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:06.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63ca000640 1 --2- 172.21.15.27:0/3268716654 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f63cc0aea00 0x7f63cc0ad390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:06.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63caa00640 1 --2- 172.21.15.27:0/3268716654 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f63cc0aea00 0x7f63cc0ad390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:06.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63caa00640 1 -- 172.21.15.27:0/3268716654 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63cc15e730 msgr2=0x7f63cc0ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:06.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63caa00640 1 --2- 172.21.15.27:0/3268716654 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63cc15e730 0x7f63cc0ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:06.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63caa00640 1 -- 172.21.15.27:0/3268716654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63bc0094d0 con 0x7f63cc06b610 2024-01-15T18:06:06.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63cb400640 1 --2- 172.21.15.27:0/3268716654 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63cc15e730 0x7f63cc0ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:06:06.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63ca000640 1 --2- 172.21.15.27:0/3268716654 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f63cc0aea00 0x7f63cc0ad390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:06:06.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63caa00640 1 --2- 172.21.15.27:0/3268716654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63cc06b610 0x7f63cc14f740 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f63bc0af040 tx=0x7f63bc01abf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:06.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63c3400640 1 -- 172.21.15.27:0/3268716654 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc0b7620 con 0x7f63cc06b610 2024-01-15T18:06:06.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63c3400640 1 -- 172.21.15.27:0/3268716654 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63bc0b77a0 con 0x7f63cc06b610 2024-01-15T18:06:06.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63d1e00640 1 -- 172.21.15.27:0/3268716654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63cc152a30 con 0x7f63cc06b610 2024-01-15T18:06:06.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63c3400640 1 -- 172.21.15.27:0/3268716654 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc0b7b50 con 0x7f63cc06b610 2024-01-15T18:06:06.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.666+0000 7f63d1e00640 1 -- 172.21.15.27:0/3268716654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63cc152ff0 con 0x7f63cc06b610 2024-01-15T18:06:06.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.670+0000 7f63d1e00640 1 -- 172.21.15.27:0/3268716654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63cc1f68c0 con 0x7f63cc06b610 2024-01-15T18:06:06.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.670+0000 7f63c3400640 1 -- 172.21.15.27:0/3268716654 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f63bc0b3040 con 0x7f63cc06b610 2024-01-15T18:06:06.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.670+0000 7f63c3400640 1 --2- 172.21.15.27:0/3268716654 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f63a40668f0 0x7f63a4068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:06.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.670+0000 7f63c3400640 1 -- 172.21.15.27:0/3268716654 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63bc11e980 con 0x7f63cc06b610 2024-01-15T18:06:06.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.670+0000 7f63ca000640 1 --2- 172.21.15.27:0/3268716654 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f63a40668f0 0x7f63a4068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:06.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.670+0000 7f63ca000640 1 --2- 172.21.15.27:0/3268716654 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f63a40668f0 0x7f63a4068db0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f63cc0af200 tx=0x7f63b4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:06.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.674+0000 7f63c3400640 1 -- 172.21.15.27:0/3268716654 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f63bc0ebbb0 con 0x7f63cc06b610 2024-01-15T18:06:06.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.830+0000 7f63d1e00640 1 -- 172.21.15.27:0/3268716654 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f63cc04e850 con 0x7f63a40668f0 2024-01-15T18:06:06.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.838+0000 7f63c3400640 1 -- 172.21.15.27:0/3268716654 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289009 (secure 0 0 0) 0x7f63cc04e850 con 0x7f63a40668f0 2024-01-15T18:06:06.847 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:06:06.849 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:06:06.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.842+0000 7f63d1e00640 1 -- 172.21.15.27:0/3268716654 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f63a40668f0 msgr2=0x7f63a4068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.842+0000 7f63d1e00640 1 --2- 172.21.15.27:0/3268716654 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f63a40668f0 0x7f63a4068db0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f63cc0af200 tx=0x7f63b4009040 comp rx=0 tx=0).stop 2024-01-15T18:06:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.842+0000 7f63d1e00640 1 -- 172.21.15.27:0/3268716654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63cc06b610 msgr2=0x7f63cc14f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.842+0000 7f63d1e00640 1 --2- 172.21.15.27:0/3268716654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63cc06b610 0x7f63cc14f740 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f63bc0af040 tx=0x7f63bc01abf0 comp rx=0 tx=0).stop 2024-01-15T18:06:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.842+0000 7f63d1e00640 1 -- 172.21.15.27:0/3268716654 shutdown_connections 2024-01-15T18:06:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.842+0000 7f63d1e00640 1 --2- 172.21.15.27:0/3268716654 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f63a40668f0 0x7f63a4068db0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.842+0000 7f63d1e00640 1 --2- 172.21.15.27:0/3268716654 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63cc15e730 0x7f63cc0ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.842+0000 7f63d1e00640 1 --2- 172.21.15.27:0/3268716654 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f63cc0aea00 0x7f63cc0ad390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.842+0000 7f63d1e00640 1 --2- 172.21.15.27:0/3268716654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63cc06b610 0x7f63cc14f740 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:06.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.842+0000 7f63d1e00640 1 -- 172.21.15.27:0/3268716654 >> 172.21.15.27:0/3268716654 conn(0x7f63cc0cbf80 msgr2=0x7f63cc0ce870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:06.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.842+0000 7f63d1e00640 1 -- 172.21.15.27:0/3268716654 shutdown_connections 2024-01-15T18:06:06.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:06.842+0000 7f63d1e00640 1 -- 172.21.15.27:0/3268716654 wait complete. 2024-01-15T18:06:06.868 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-01-15T18:06:06.154062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224504,"kb_used_data":4168,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750216,"statfs":{"total":773094113280,"available":772864221184,"internally_reserved":0,"allocated":4268032,"data_stored":2369484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001436"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258756+0000","last_change":"2024-01-15T18:06:01.189891+0000","last_active":"2024-01-15T18:06:02.258756+0000","last_peered":"2024-01-15T18:06:02.258756+0000","last_clean":"2024-01-15T18:06:02.258756+0000","last_became_active":"2024-01-15T18:06:01.189621+0000","last_became_peered":"2024-01-15T18:06:01.189621+0000","last_unstale":"2024-01-15T18:06:02.258756+0000","last_undegraded":"2024-01-15T18:06:02.258756+0000","last_fullsized":"2024-01-15T18:06:02.258756+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:26:25.916306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.260287+0000","last_change":"2024-01-15T18:06:01.193972+0000","last_active":"2024-01-15T18:06:02.260287+0000","last_peered":"2024-01-15T18:06:02.260287+0000","last_clean":"2024-01-15T18:06:02.260287+0000","last_became_active":"2024-01-15T18:06:01.193681+0000","last_became_peered":"2024-01-15T18:06:01.193681+0000","last_unstale":"2024-01-15T18:06:02.260287+0000","last_undegraded":"2024-01-15T18:06:02.260287+0000","last_fullsized":"2024-01-15T18:06:02.260287+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:58:04.778601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.176014+0000","last_change":"2024-01-15T18:06:01.189390+0000","last_active":"2024-01-15T18:06:02.176014+0000","last_peered":"2024-01-15T18:06:02.176014+0000","last_clean":"2024-01-15T18:06:02.176014+0000","last_became_active":"2024-01-15T18:06:01.188747+0000","last_became_peered":"2024-01-15T18:06:01.188747+0000","last_unstale":"2024-01-15T18:06:02.176014+0000","last_undegraded":"2024-01-15T18:06:02.176014+0000","last_fullsized":"2024-01-15T18:06:02.176014+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:38:31.001343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.174594+0000","last_change":"2024-01-15T18:06:01.183654+0000","last_active":"2024-01-15T18:06:02.174594+0000","last_peered":"2024-01-15T18:06:02.174594+0000","last_clean":"2024-01-15T18:06:02.174594+0000","last_became_active":"2024-01-15T18:06:01.183228+0000","last_became_peered":"2024-01-15T18:06:01.183228+0000","last_unstale":"2024-01-15T18:06:02.174594+0000","last_undegraded":"2024-01-15T18:06:02.174594+0000","last_fullsized":"2024-01-15T18:06:02.174594+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:41:31.415431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258875+0000","last_change":"2024-01-15T18:06:01.178251+0000","last_active":"2024-01-15T18:06:02.258875+0000","last_peered":"2024-01-15T18:06:02.258875+0000","last_clean":"2024-01-15T18:06:02.258875+0000","last_became_active":"2024-01-15T18:06:01.175580+0000","last_became_peered":"2024-01-15T18:06:01.175580+0000","last_unstale":"2024-01-15T18:06:02.258875+0000","last_undegraded":"2024-01-15T18:06:02.258875+0000","last_fullsized":"2024-01-15T18:06:02.258875+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:13.332175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259387+0000","last_change":"2024-01-15T18:06:01.187886+0000","last_active":"2024-01-15T18:06:02.259387+0000","last_peered":"2024-01-15T18:06:02.259387+0000","last_clean":"2024-01-15T18:06:02.259387+0000","last_became_active":"2024-01-15T18:06:01.187610+0000","last_became_peered":"2024-01-15T18:06:01.187610+0000","last_unstale":"2024-01-15T18:06:02.259387+0000","last_undegraded":"2024-01-15T18:06:02.259387+0000","last_fullsized":"2024-01-15T18:06:02.259387+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:16:22.277642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259320+0000","last_change":"2024-01-15T18:06:01.193492+0000","last_active":"2024-01-15T18:06:02.259320+0000","last_peered":"2024-01-15T18:06:02.259320+0000","last_clean":"2024-01-15T18:06:02.259320+0000","last_became_active":"2024-01-15T18:06:01.190348+0000","last_became_peered":"2024-01-15T18:06:01.190348+0000","last_unstale":"2024-01-15T18:06:02.259320+0000","last_undegraded":"2024-01-15T18:06:02.259320+0000","last_fullsized":"2024-01-15T18:06:02.259320+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:09:25.929285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258606+0000","last_change":"2024-01-15T18:06:01.188695+0000","last_active":"2024-01-15T18:06:02.258606+0000","last_peered":"2024-01-15T18:06:02.258606+0000","last_clean":"2024-01-15T18:06:02.258606+0000","last_became_active":"2024-01-15T18:06:01.184708+0000","last_became_peered":"2024-01-15T18:06:01.184708+0000","last_unstale":"2024-01-15T18:06:02.258606+0000","last_undegraded":"2024-01-15T18:06:02.258606+0000","last_fullsized":"2024-01-15T18:06:02.258606+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:12:07.957753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.177002+0000","last_change":"2024-01-15T18:06:01.182729+0000","last_active":"2024-01-15T18:06:02.177002+0000","last_peered":"2024-01-15T18:06:02.177002+0000","last_clean":"2024-01-15T18:06:02.177002+0000","last_became_active":"2024-01-15T18:06:01.182433+0000","last_became_peered":"2024-01-15T18:06:01.182433+0000","last_unstale":"2024-01-15T18:06:02.177002+0000","last_undegraded":"2024-01-15T18:06:02.177002+0000","last_fullsized":"2024-01-15T18:06:02.177002+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:15.311517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.174963+0000","last_change":"2024-01-15T18:06:01.179590+0000","last_active":"2024-01-15T18:06:02.174963+0000","last_peered":"2024-01-15T18:06:02.174963+0000","last_clean":"2024-01-15T18:06:02.174963+0000","last_became_active":"2024-01-15T18:06:01.179235+0000","last_became_peered":"2024-01-15T18:06:01.179235+0000","last_unstale":"2024-01-15T18:06:02.174963+0000","last_undegraded":"2024-01-15T18:06:02.174963+0000","last_fullsized":"2024-01-15T18:06:02.174963+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:08:45.088678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259464+0000","last_change":"2024-01-15T18:06:01.193835+0000","last_active":"2024-01-15T18:06:02.259464+0000","last_peered":"2024-01-15T18:06:02.259464+0000","last_clean":"2024-01-15T18:06:02.259464+0000","last_became_active":"2024-01-15T18:06:01.193229+0000","last_became_peered":"2024-01-15T18:06:01.193229+0000","last_unstale":"2024-01-15T18:06:02.259464+0000","last_undegraded":"2024-01-15T18:06:02.259464+0000","last_fullsized":"2024-01-15T18:06:02.259464+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:15:23.520645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.175901+0000","last_change":"2024-01-15T18:06:01.189227+0000","last_active":"2024-01-15T18:06:02.175901+0000","last_peered":"2024-01-15T18:06:02.175901+0000","last_clean":"2024-01-15T18:06:02.175901+0000","last_became_active":"2024-01-15T18:06:01.188807+0000","last_became_peered":"2024-01-15T18:06:01.188807+0000","last_unstale":"2024-01-15T18:06:02.175901+0000","last_undegraded":"2024-01-15T18:06:02.175901+0000","last_fullsized":"2024-01-15T18:06:02.175901+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:49:18.322259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259000+0000","last_change":"2024-01-15T18:06:01.184266+0000","last_active":"2024-01-15T18:06:02.259000+0000","last_peered":"2024-01-15T18:06:02.259000+0000","last_clean":"2024-01-15T18:06:02.259000+0000","last_became_active":"2024-01-15T18:06:01.183933+0000","last_became_peered":"2024-01-15T18:06:01.183933+0000","last_unstale":"2024-01-15T18:06:02.259000+0000","last_undegraded":"2024-01-15T18:06:02.259000+0000","last_fullsized":"2024-01-15T18:06:02.259000+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:40:57.130644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259845+0000","last_change":"2024-01-15T18:06:01.175248+0000","last_active":"2024-01-15T18:06:02.259845+0000","last_peered":"2024-01-15T18:06:02.259845+0000","last_clean":"2024-01-15T18:06:02.259845+0000","last_became_active":"2024-01-15T18:06:01.174941+0000","last_became_peered":"2024-01-15T18:06:01.174941+0000","last_unstale":"2024-01-15T18:06:02.259845+0000","last_undegraded":"2024-01-15T18:06:02.259845+0000","last_fullsized":"2024-01-15T18:06:02.259845+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:20:58.455307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.175860+0000","last_change":"2024-01-15T18:06:01.183569+0000","last_active":"2024-01-15T18:06:02.175860+0000","last_peered":"2024-01-15T18:06:02.175860+0000","last_clean":"2024-01-15T18:06:02.175860+0000","last_became_active":"2024-01-15T18:06:01.183236+0000","last_became_peered":"2024-01-15T18:06:01.183236+0000","last_unstale":"2024-01-15T18:06:02.175860+0000","last_undegraded":"2024-01-15T18:06:02.175860+0000","last_fullsized":"2024-01-15T18:06:02.175860+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:43:51.863373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258744+0000","last_change":"2024-01-15T18:06:01.192491+0000","last_active":"2024-01-15T18:06:02.258744+0000","last_peered":"2024-01-15T18:06:02.258744+0000","last_clean":"2024-01-15T18:06:02.258744+0000","last_became_active":"2024-01-15T18:06:01.192015+0000","last_became_peered":"2024-01-15T18:06:01.192015+0000","last_unstale":"2024-01-15T18:06:02.258744+0000","last_undegraded":"2024-01-15T18:06:02.258744+0000","last_fullsized":"2024-01-15T18:06:02.258744+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:16:22.277642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259443+0000","last_change":"2024-01-15T18:06:01.194135+0000","last_active":"2024-01-15T18:06:02.259443+0000","last_peered":"2024-01-15T18:06:02.259443+0000","last_clean":"2024-01-15T18:06:02.259443+0000","last_became_active":"2024-01-15T18:06:01.193839+0000","last_became_peered":"2024-01-15T18:06:01.193839+0000","last_unstale":"2024-01-15T18:06:02.259443+0000","last_undegraded":"2024-01-15T18:06:02.259443+0000","last_fullsized":"2024-01-15T18:06:02.259443+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:18:47.499122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.260288+0000","last_change":"2024-01-15T18:06:01.174507+0000","last_active":"2024-01-15T18:06:02.260288+0000","last_peered":"2024-01-15T18:06:02.260288+0000","last_clean":"2024-01-15T18:06:02.260288+0000","last_became_active":"2024-01-15T18:06:01.174222+0000","last_became_peered":"2024-01-15T18:06:01.174222+0000","last_unstale":"2024-01-15T18:06:02.260288+0000","last_undegraded":"2024-01-15T18:06:02.260288+0000","last_fullsized":"2024-01-15T18:06:02.260288+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:15:23.520645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259924+0000","last_change":"2024-01-15T18:06:01.184671+0000","last_active":"2024-01-15T18:06:02.259924+0000","last_peered":"2024-01-15T18:06:02.259924+0000","last_clean":"2024-01-15T18:06:02.259924+0000","last_became_active":"2024-01-15T18:06:01.184407+0000","last_became_peered":"2024-01-15T18:06:01.184407+0000","last_unstale":"2024-01-15T18:06:02.259924+0000","last_undegraded":"2024-01-15T18:06:02.259924+0000","last_fullsized":"2024-01-15T18:06:02.259924+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:00:34.319443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259643+0000","last_change":"2024-01-15T18:06:01.183764+0000","last_active":"2024-01-15T18:06:02.259643+0000","last_peered":"2024-01-15T18:06:02.259643+0000","last_clean":"2024-01-15T18:06:02.259643+0000","last_became_active":"2024-01-15T18:06:01.183530+0000","last_became_peered":"2024-01-15T18:06:01.183530+0000","last_unstale":"2024-01-15T18:06:02.259643+0000","last_undegraded":"2024-01-15T18:06:02.259643+0000","last_fullsized":"2024-01-15T18:06:02.259643+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:38:08.324768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.177103+0000","last_change":"2024-01-15T18:06:01.188277+0000","last_active":"2024-01-15T18:06:02.177103+0000","last_peered":"2024-01-15T18:06:02.177103+0000","last_clean":"2024-01-15T18:06:02.177103+0000","last_became_active":"2024-01-15T18:06:01.187945+0000","last_became_peered":"2024-01-15T18:06:01.187945+0000","last_unstale":"2024-01-15T18:06:02.177103+0000","last_undegraded":"2024-01-15T18:06:02.177103+0000","last_fullsized":"2024-01-15T18:06:02.177103+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:08:45.088678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.176680+0000","last_change":"2024-01-15T18:06:01.172951+0000","last_active":"2024-01-15T18:06:02.176680+0000","last_peered":"2024-01-15T18:06:02.176680+0000","last_clean":"2024-01-15T18:06:02.176680+0000","last_became_active":"2024-01-15T18:06:01.172634+0000","last_became_peered":"2024-01-15T18:06:01.172634+0000","last_unstale":"2024-01-15T18:06:02.176680+0000","last_undegraded":"2024-01-15T18:06:02.176680+0000","last_fullsized":"2024-01-15T18:06:02.176680+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:52:51.980421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258714+0000","last_change":"2024-01-15T18:06:01.192000+0000","last_active":"2024-01-15T18:06:02.258714+0000","last_peered":"2024-01-15T18:06:02.258714+0000","last_clean":"2024-01-15T18:06:02.258714+0000","last_became_active":"2024-01-15T18:06:01.191712+0000","last_became_peered":"2024-01-15T18:06:01.191712+0000","last_unstale":"2024-01-15T18:06:02.258714+0000","last_undegraded":"2024-01-15T18:06:02.258714+0000","last_fullsized":"2024-01-15T18:06:02.258714+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:49:34.739781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.175593+0000","last_change":"2024-01-15T18:06:01.177362+0000","last_active":"2024-01-15T18:06:02.175593+0000","last_peered":"2024-01-15T18:06:02.175593+0000","last_clean":"2024-01-15T18:06:02.175593+0000","last_became_active":"2024-01-15T18:06:01.177047+0000","last_became_peered":"2024-01-15T18:06:01.177047+0000","last_unstale":"2024-01-15T18:06:02.175593+0000","last_undegraded":"2024-01-15T18:06:02.175593+0000","last_fullsized":"2024-01-15T18:06:02.175593+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:15.719166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.175027+0000","last_change":"2024-01-15T18:06:01.191258+0000","last_active":"2024-01-15T18:06:02.175027+0000","last_peered":"2024-01-15T18:06:02.175027+0000","last_clean":"2024-01-15T18:06:02.175027+0000","last_became_active":"2024-01-15T18:06:01.190908+0000","last_became_peered":"2024-01-15T18:06:01.190908+0000","last_unstale":"2024-01-15T18:06:02.175027+0000","last_undegraded":"2024-01-15T18:06:02.175027+0000","last_fullsized":"2024-01-15T18:06:02.175027+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:50:20.923618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.260378+0000","last_change":"2024-01-15T18:06:01.190720+0000","last_active":"2024-01-15T18:06:02.260378+0000","last_peered":"2024-01-15T18:06:02.260378+0000","last_clean":"2024-01-15T18:06:02.260378+0000","last_became_active":"2024-01-15T18:06:01.190391+0000","last_became_peered":"2024-01-15T18:06:01.190391+0000","last_unstale":"2024-01-15T18:06:02.260378+0000","last_undegraded":"2024-01-15T18:06:02.260378+0000","last_fullsized":"2024-01-15T18:06:02.260378+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:43:52.778742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.174985+0000","last_change":"2024-01-15T18:06:01.186658+0000","last_active":"2024-01-15T18:06:02.174985+0000","last_peered":"2024-01-15T18:06:02.174985+0000","last_clean":"2024-01-15T18:06:02.174985+0000","last_became_active":"2024-01-15T18:06:01.183065+0000","last_became_peered":"2024-01-15T18:06:01.183065+0000","last_unstale":"2024-01-15T18:06:02.174985+0000","last_undegraded":"2024-01-15T18:06:02.174985+0000","last_fullsized":"2024-01-15T18:06:02.174985+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:02.935448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.175887+0000","last_change":"2024-01-15T18:06:01.194852+0000","last_active":"2024-01-15T18:06:02.175887+0000","last_peered":"2024-01-15T18:06:02.175887+0000","last_clean":"2024-01-15T18:06:02.175887+0000","last_became_active":"2024-01-15T18:06:01.194509+0000","last_became_peered":"2024-01-15T18:06:01.194509+0000","last_unstale":"2024-01-15T18:06:02.175887+0000","last_undegraded":"2024-01-15T18:06:02.175887+0000","last_fullsized":"2024-01-15T18:06:02.175887+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:48:05.723516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.176765+0000","last_change":"2024-01-15T18:06:01.180504+0000","last_active":"2024-01-15T18:06:02.176765+0000","last_peered":"2024-01-15T18:06:02.176765+0000","last_clean":"2024-01-15T18:06:02.176765+0000","last_became_active":"2024-01-15T18:06:01.180161+0000","last_became_peered":"2024-01-15T18:06:01.180161+0000","last_unstale":"2024-01-15T18:06:02.176765+0000","last_undegraded":"2024-01-15T18:06:02.176765+0000","last_fullsized":"2024-01-15T18:06:02.176765+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:25:51.645494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259370+0000","last_change":"2024-01-15T18:06:01.182624+0000","last_active":"2024-01-15T18:06:02.259370+0000","last_peered":"2024-01-15T18:06:02.259370+0000","last_clean":"2024-01-15T18:06:02.259370+0000","last_became_active":"2024-01-15T18:06:01.182328+0000","last_became_peered":"2024-01-15T18:06:01.182328+0000","last_unstale":"2024-01-15T18:06:02.259370+0000","last_undegraded":"2024-01-15T18:06:02.259370+0000","last_fullsized":"2024-01-15T18:06:02.259370+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:23:07.800413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258805+0000","last_change":"2024-01-15T18:06:01.185113+0000","last_active":"2024-01-15T18:06:02.258805+0000","last_peered":"2024-01-15T18:06:02.258805+0000","last_clean":"2024-01-15T18:06:02.258805+0000","last_became_active":"2024-01-15T18:06:01.184857+0000","last_became_peered":"2024-01-15T18:06:01.184857+0000","last_unstale":"2024-01-15T18:06:02.258805+0000","last_undegraded":"2024-01-15T18:06:02.258805+0000","last_fullsized":"2024-01-15T18:06:02.258805+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:43:35.040533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258552+0000","last_change":"2024-01-15T18:06:01.189709+0000","last_active":"2024-01-15T18:06:02.258552+0000","last_peered":"2024-01-15T18:06:02.258552+0000","last_clean":"2024-01-15T18:06:02.258552+0000","last_became_active":"2024-01-15T18:06:01.185069+0000","last_became_peered":"2024-01-15T18:06:01.185069+0000","last_unstale":"2024-01-15T18:06:02.258552+0000","last_undegraded":"2024-01-15T18:06:02.258552+0000","last_fullsized":"2024-01-15T18:06:02.258552+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:42:55.460579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259388+0000","last_change":"2024-01-15T18:06:01.191979+0000","last_active":"2024-01-15T18:06:02.259388+0000","last_peered":"2024-01-15T18:06:02.259388+0000","last_clean":"2024-01-15T18:06:02.259388+0000","last_became_active":"2024-01-15T18:06:01.191642+0000","last_became_peered":"2024-01-15T18:06:01.191642+0000","last_unstale":"2024-01-15T18:06:02.259388+0000","last_undegraded":"2024-01-15T18:06:02.259388+0000","last_fullsized":"2024-01-15T18:06:02.259388+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:01:18.922210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.208366+0000","last_change":"2024-01-15T18:06:00.191056+0000","last_active":"2024-01-15T18:06:01.208366+0000","last_peered":"2024-01-15T18:06:01.208366+0000","last_clean":"2024-01-15T18:06:01.208366+0000","last_became_active":"2024-01-15T18:06:00.190695+0000","last_became_peered":"2024-01-15T18:06:00.190695+0000","last_unstale":"2024-01-15T18:06:01.208366+0000","last_undegraded":"2024-01-15T18:06:01.208366+0000","last_fullsized":"2024-01-15T18:06:01.208366+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:51:46.033055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258879+0000","last_change":"2024-01-15T18:06:01.184402+0000","last_active":"2024-01-15T18:06:02.258879+0000","last_peered":"2024-01-15T18:06:02.258879+0000","last_clean":"2024-01-15T18:06:02.258879+0000","last_became_active":"2024-01-15T18:06:01.184128+0000","last_became_peered":"2024-01-15T18:06:01.184128+0000","last_unstale":"2024-01-15T18:06:02.258879+0000","last_undegraded":"2024-01-15T18:06:02.258879+0000","last_fullsized":"2024-01-15T18:06:02.258879+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:55:20.748447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259476+0000","last_change":"2024-01-15T18:06:00.187711+0000","last_active":"2024-01-15T18:06:02.259476+0000","last_peered":"2024-01-15T18:06:02.259476+0000","last_clean":"2024-01-15T18:06:02.259476+0000","last_became_active":"2024-01-15T18:06:00.187351+0000","last_became_peered":"2024-01-15T18:06:00.187351+0000","last_unstale":"2024-01-15T18:06:02.259476+0000","last_undegraded":"2024-01-15T18:06:02.259476+0000","last_fullsized":"2024-01-15T18:06:02.259476+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:45:53.529417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.176955+0000","last_change":"2024-01-15T18:06:01.175024+0000","last_active":"2024-01-15T18:06:02.176955+0000","last_peered":"2024-01-15T18:06:02.176955+0000","last_clean":"2024-01-15T18:06:02.176955+0000","last_became_active":"2024-01-15T18:06:01.174755+0000","last_became_peered":"2024-01-15T18:06:01.174755+0000","last_unstale":"2024-01-15T18:06:02.176955+0000","last_undegraded":"2024-01-15T18:06:02.176955+0000","last_fullsized":"2024-01-15T18:06:02.176955+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:41:31.415431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.210775+0000","last_change":"2024-01-15T18:06:00.194271+0000","last_active":"2024-01-15T18:06:01.210775+0000","last_peered":"2024-01-15T18:06:01.210775+0000","last_clean":"2024-01-15T18:06:01.210775+0000","last_became_active":"2024-01-15T18:06:00.193996+0000","last_became_peered":"2024-01-15T18:06:00.193996+0000","last_unstale":"2024-01-15T18:06:01.210775+0000","last_undegraded":"2024-01-15T18:06:01.210775+0000","last_fullsized":"2024-01-15T18:06:01.210775+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:01.928581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.175025+0000","last_change":"2024-01-15T18:06:01.184800+0000","last_active":"2024-01-15T18:06:02.175025+0000","last_peered":"2024-01-15T18:06:02.175025+0000","last_clean":"2024-01-15T18:06:02.175025+0000","last_became_active":"2024-01-15T18:06:01.183424+0000","last_became_peered":"2024-01-15T18:06:01.183424+0000","last_unstale":"2024-01-15T18:06:02.175025+0000","last_undegraded":"2024-01-15T18:06:02.175025+0000","last_fullsized":"2024-01-15T18:06:02.175025+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:42:55.460579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.205682+0000","last_change":"2024-01-15T18:06:00.178063+0000","last_active":"2024-01-15T18:06:01.205682+0000","last_peered":"2024-01-15T18:06:01.205682+0000","last_clean":"2024-01-15T18:06:01.205682+0000","last_became_active":"2024-01-15T18:06:00.177628+0000","last_became_peered":"2024-01-15T18:06:00.177628+0000","last_unstale":"2024-01-15T18:06:01.205682+0000","last_undegraded":"2024-01-15T18:06:01.205682+0000","last_fullsized":"2024-01-15T18:06:01.205682+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:08:08.490734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259226+0000","last_change":"2024-01-15T18:06:01.196654+0000","last_active":"2024-01-15T18:06:02.259226+0000","last_peered":"2024-01-15T18:06:02.259226+0000","last_clean":"2024-01-15T18:06:02.259226+0000","last_became_active":"2024-01-15T18:06:01.196278+0000","last_became_peered":"2024-01-15T18:06:01.196278+0000","last_unstale":"2024-01-15T18:06:02.259226+0000","last_undegraded":"2024-01-15T18:06:02.259226+0000","last_fullsized":"2024-01-15T18:06:02.259226+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:00:34.319443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.209388+0000","last_change":"2024-01-15T18:06:00.193082+0000","last_active":"2024-01-15T18:06:01.209388+0000","last_peered":"2024-01-15T18:06:01.209388+0000","last_clean":"2024-01-15T18:06:01.209388+0000","last_became_active":"2024-01-15T18:06:00.192750+0000","last_became_peered":"2024-01-15T18:06:00.192750+0000","last_unstale":"2024-01-15T18:06:01.209388+0000","last_undegraded":"2024-01-15T18:06:01.209388+0000","last_fullsized":"2024-01-15T18:06:01.209388+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:43:34.033665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.176654+0000","last_change":"2024-01-15T18:06:01.195416+0000","last_active":"2024-01-15T18:06:02.176654+0000","last_peered":"2024-01-15T18:06:02.176654+0000","last_clean":"2024-01-15T18:06:02.176654+0000","last_became_active":"2024-01-15T18:06:01.195155+0000","last_became_peered":"2024-01-15T18:06:01.195155+0000","last_unstale":"2024-01-15T18:06:02.176654+0000","last_undegraded":"2024-01-15T18:06:02.176654+0000","last_fullsized":"2024-01-15T18:06:02.176654+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:22:40.694092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.204726+0000","last_change":"2024-01-15T18:06:00.191810+0000","last_active":"2024-01-15T18:06:01.204726+0000","last_peered":"2024-01-15T18:06:01.204726+0000","last_clean":"2024-01-15T18:06:01.204726+0000","last_became_active":"2024-01-15T18:06:00.191470+0000","last_became_peered":"2024-01-15T18:06:00.191470+0000","last_unstale":"2024-01-15T18:06:01.204726+0000","last_undegraded":"2024-01-15T18:06:01.204726+0000","last_fullsized":"2024-01-15T18:06:01.204726+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:27:03.729350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.176435+0000","last_change":"2024-01-15T18:06:01.181141+0000","last_active":"2024-01-15T18:06:02.176435+0000","last_peered":"2024-01-15T18:06:02.176435+0000","last_clean":"2024-01-15T18:06:02.176435+0000","last_became_active":"2024-01-15T18:06:01.180747+0000","last_became_peered":"2024-01-15T18:06:01.180747+0000","last_unstale":"2024-01-15T18:06:02.176435+0000","last_undegraded":"2024-01-15T18:06:02.176435+0000","last_fullsized":"2024-01-15T18:06:02.176435+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:43:46.415533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.159942+0000","last_change":"2024-01-15T18:06:00.190904+0000","last_active":"2024-01-15T18:06:01.159942+0000","last_peered":"2024-01-15T18:06:01.159942+0000","last_clean":"2024-01-15T18:06:01.159942+0000","last_became_active":"2024-01-15T18:06:00.190620+0000","last_became_peered":"2024-01-15T18:06:00.190620+0000","last_unstale":"2024-01-15T18:06:01.159942+0000","last_undegraded":"2024-01-15T18:06:01.159942+0000","last_fullsized":"2024-01-15T18:06:01.159942+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:26:24.909439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.174392+0000","last_change":"2024-01-15T18:06:01.182798+0000","last_active":"2024-01-15T18:06:02.174392+0000","last_peered":"2024-01-15T18:06:02.174392+0000","last_clean":"2024-01-15T18:06:02.174392+0000","last_became_active":"2024-01-15T18:06:01.182517+0000","last_became_peered":"2024-01-15T18:06:01.182517+0000","last_unstale":"2024-01-15T18:06:02.174392+0000","last_undegraded":"2024-01-15T18:06:02.174392+0000","last_fullsized":"2024-01-15T18:06:02.174392+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:25:51.645494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.165733+0000","last_change":"2024-01-15T18:06:00.184942+0000","last_active":"2024-01-15T18:06:01.165733+0000","last_peered":"2024-01-15T18:06:01.165733+0000","last_clean":"2024-01-15T18:06:01.165733+0000","last_became_active":"2024-01-15T18:06:00.184628+0000","last_became_peered":"2024-01-15T18:06:00.184628+0000","last_unstale":"2024-01-15T18:06:01.165733+0000","last_undegraded":"2024-01-15T18:06:01.165733+0000","last_fullsized":"2024-01-15T18:06:01.165733+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:36:46.224457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.176014+0000","last_change":"2024-01-15T18:06:01.186936+0000","last_active":"2024-01-15T18:06:02.176014+0000","last_peered":"2024-01-15T18:06:02.176014+0000","last_clean":"2024-01-15T18:06:02.176014+0000","last_became_active":"2024-01-15T18:06:01.186613+0000","last_became_peered":"2024-01-15T18:06:01.186613+0000","last_unstale":"2024-01-15T18:06:02.176014+0000","last_undegraded":"2024-01-15T18:06:02.176014+0000","last_fullsized":"2024-01-15T18:06:02.176014+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:57.989668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.161217+0000","last_change":"2024-01-15T18:06:00.189934+0000","last_active":"2024-01-15T18:06:01.161217+0000","last_peered":"2024-01-15T18:06:01.161217+0000","last_clean":"2024-01-15T18:06:01.161217+0000","last_became_active":"2024-01-15T18:06:00.189306+0000","last_became_peered":"2024-01-15T18:06:00.189306+0000","last_unstale":"2024-01-15T18:06:01.161217+0000","last_undegraded":"2024-01-15T18:06:01.161217+0000","last_fullsized":"2024-01-15T18:06:01.161217+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:08:44.081810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.260274+0000","last_change":"2024-01-15T18:06:01.176311+0000","last_active":"2024-01-15T18:06:02.260274+0000","last_peered":"2024-01-15T18:06:02.260274+0000","last_clean":"2024-01-15T18:06:02.260274+0000","last_became_active":"2024-01-15T18:06:01.175647+0000","last_became_peered":"2024-01-15T18:06:01.175647+0000","last_unstale":"2024-01-15T18:06:02.260274+0000","last_undegraded":"2024-01-15T18:06:02.260274+0000","last_fullsized":"2024-01-15T18:06:02.260274+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:18:47.499122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259624+0000","last_change":"2024-01-15T18:06:00.193942+0000","last_active":"2024-01-15T18:06:02.259624+0000","last_peered":"2024-01-15T18:06:02.259624+0000","last_clean":"2024-01-15T18:06:02.259624+0000","last_became_active":"2024-01-15T18:06:00.193102+0000","last_became_peered":"2024-01-15T18:06:00.193102+0000","last_unstale":"2024-01-15T18:06:02.259624+0000","last_undegraded":"2024-01-15T18:06:02.259624+0000","last_fullsized":"2024-01-15T18:06:02.259624+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:36:35.312505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.260071+0000","last_change":"2024-01-15T18:06:01.183098+0000","last_active":"2024-01-15T18:06:02.260071+0000","last_peered":"2024-01-15T18:06:02.260071+0000","last_clean":"2024-01-15T18:06:02.260071+0000","last_became_active":"2024-01-15T18:06:01.182836+0000","last_became_peered":"2024-01-15T18:06:01.182836+0000","last_unstale":"2024-01-15T18:06:02.260071+0000","last_undegraded":"2024-01-15T18:06:02.260071+0000","last_fullsized":"2024-01-15T18:06:02.260071+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:09:25.929285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.209871+0000","last_change":"2024-01-15T18:06:00.182124+0000","last_active":"2024-01-15T18:06:01.209871+0000","last_peered":"2024-01-15T18:06:01.209871+0000","last_clean":"2024-01-15T18:06:01.209871+0000","last_became_active":"2024-01-15T18:06:00.181820+0000","last_became_peered":"2024-01-15T18:06:00.181820+0000","last_unstale":"2024-01-15T18:06:01.209871+0000","last_undegraded":"2024-01-15T18:06:01.209871+0000","last_fullsized":"2024-01-15T18:06:01.209871+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:15:39.265433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.175925+0000","last_change":"2024-01-15T18:06:01.189648+0000","last_active":"2024-01-15T18:06:02.175925+0000","last_peered":"2024-01-15T18:06:02.175925+0000","last_clean":"2024-01-15T18:06:02.175925+0000","last_became_active":"2024-01-15T18:06:01.189124+0000","last_became_peered":"2024-01-15T18:06:01.189124+0000","last_unstale":"2024-01-15T18:06:02.175925+0000","last_undegraded":"2024-01-15T18:06:02.175925+0000","last_fullsized":"2024-01-15T18:06:02.175925+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:25:05.480462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.215933+0000","last_change":"2024-01-15T18:06:00.191070+0000","last_active":"2024-01-15T18:06:01.215933+0000","last_peered":"2024-01-15T18:06:01.215933+0000","last_clean":"2024-01-15T18:06:01.215933+0000","last_became_active":"2024-01-15T18:06:00.190787+0000","last_became_peered":"2024-01-15T18:06:00.190787+0000","last_unstale":"2024-01-15T18:06:01.215933+0000","last_undegraded":"2024-01-15T18:06:01.215933+0000","last_fullsized":"2024-01-15T18:06:01.215933+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:15:36.318765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.174392+0000","last_change":"2024-01-15T18:06:01.185323+0000","last_active":"2024-01-15T18:06:02.174392+0000","last_peered":"2024-01-15T18:06:02.174392+0000","last_clean":"2024-01-15T18:06:02.174392+0000","last_became_active":"2024-01-15T18:06:01.184980+0000","last_became_peered":"2024-01-15T18:06:01.184980+0000","last_unstale":"2024-01-15T18:06:02.174392+0000","last_undegraded":"2024-01-15T18:06:02.174392+0000","last_fullsized":"2024-01-15T18:06:02.174392+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:52:51.980421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.207074+0000","last_change":"2024-01-15T18:06:00.180527+0000","last_active":"2024-01-15T18:06:01.207074+0000","last_peered":"2024-01-15T18:06:01.207074+0000","last_clean":"2024-01-15T18:06:01.207074+0000","last_became_active":"2024-01-15T18:06:00.180161+0000","last_became_peered":"2024-01-15T18:06:00.180161+0000","last_unstale":"2024-01-15T18:06:01.207074+0000","last_undegraded":"2024-01-15T18:06:01.207074+0000","last_fullsized":"2024-01-15T18:06:01.207074+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:36:16.189750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.260221+0000","last_change":"2024-01-15T18:06:01.195303+0000","last_active":"2024-01-15T18:06:02.260221+0000","last_peered":"2024-01-15T18:06:02.260221+0000","last_clean":"2024-01-15T18:06:02.260221+0000","last_became_active":"2024-01-15T18:06:01.195043+0000","last_became_peered":"2024-01-15T18:06:01.195043+0000","last_unstale":"2024-01-15T18:06:02.260221+0000","last_undegraded":"2024-01-15T18:06:02.260221+0000","last_fullsized":"2024-01-15T18:06:02.260221+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:23:07.800413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.168995+0000","last_change":"2024-01-15T18:06:00.184666+0000","last_active":"2024-01-15T18:06:01.168995+0000","last_peered":"2024-01-15T18:06:01.168995+0000","last_clean":"2024-01-15T18:06:01.168995+0000","last_became_active":"2024-01-15T18:06:00.184339+0000","last_became_peered":"2024-01-15T18:06:00.184339+0000","last_unstale":"2024-01-15T18:06:01.168995+0000","last_undegraded":"2024-01-15T18:06:01.168995+0000","last_fullsized":"2024-01-15T18:06:01.168995+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:36:16.189750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258684+0000","last_change":"2024-01-15T18:06:01.192289+0000","last_active":"2024-01-15T18:06:02.258684+0000","last_peered":"2024-01-15T18:06:02.258684+0000","last_clean":"2024-01-15T18:06:02.258684+0000","last_became_active":"2024-01-15T18:06:01.191604+0000","last_became_peered":"2024-01-15T18:06:01.191604+0000","last_unstale":"2024-01-15T18:06:02.258684+0000","last_undegraded":"2024-01-15T18:06:02.258684+0000","last_fullsized":"2024-01-15T18:06:02.258684+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:41.300029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.160464+0000","last_change":"2024-01-15T18:06:00.189891+0000","last_active":"2024-01-15T18:06:01.160464+0000","last_peered":"2024-01-15T18:06:01.160464+0000","last_clean":"2024-01-15T18:06:01.160464+0000","last_became_active":"2024-01-15T18:06:00.189621+0000","last_became_peered":"2024-01-15T18:06:00.189621+0000","last_unstale":"2024-01-15T18:06:01.160464+0000","last_undegraded":"2024-01-15T18:06:01.160464+0000","last_fullsized":"2024-01-15T18:06:01.160464+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:44:09.010095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258960+0000","last_change":"2024-01-15T18:06:01.182869+0000","last_active":"2024-01-15T18:06:02.258960+0000","last_peered":"2024-01-15T18:06:02.258960+0000","last_clean":"2024-01-15T18:06:02.258960+0000","last_became_active":"2024-01-15T18:06:01.182557+0000","last_became_peered":"2024-01-15T18:06:01.182557+0000","last_unstale":"2024-01-15T18:06:02.258960+0000","last_undegraded":"2024-01-15T18:06:02.258960+0000","last_fullsized":"2024-01-15T18:06:02.258960+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:27:04.736218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.213575+0000","last_change":"2024-01-15T18:06:00.194127+0000","last_active":"2024-01-15T18:06:01.213575+0000","last_peered":"2024-01-15T18:06:01.213575+0000","last_clean":"2024-01-15T18:06:01.213575+0000","last_became_active":"2024-01-15T18:06:00.193851+0000","last_became_peered":"2024-01-15T18:06:00.193851+0000","last_unstale":"2024-01-15T18:06:01.213575+0000","last_undegraded":"2024-01-15T18:06:01.213575+0000","last_fullsized":"2024-01-15T18:06:01.213575+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:36:46.224457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.176657+0000","last_change":"2024-01-15T18:06:01.183977+0000","last_active":"2024-01-15T18:06:02.176657+0000","last_peered":"2024-01-15T18:06:02.176657+0000","last_clean":"2024-01-15T18:06:02.176657+0000","last_became_active":"2024-01-15T18:06:01.183643+0000","last_became_peered":"2024-01-15T18:06:01.183643+0000","last_unstale":"2024-01-15T18:06:02.176657+0000","last_undegraded":"2024-01-15T18:06:02.176657+0000","last_fullsized":"2024-01-15T18:06:02.176657+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:44:10.016963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.209611+0000","last_change":"2024-01-15T18:06:00.195128+0000","last_active":"2024-01-15T18:06:01.209611+0000","last_peered":"2024-01-15T18:06:01.209611+0000","last_clean":"2024-01-15T18:06:01.209611+0000","last_became_active":"2024-01-15T18:06:00.194872+0000","last_became_peered":"2024-01-15T18:06:00.194872+0000","last_unstale":"2024-01-15T18:06:01.209611+0000","last_undegraded":"2024-01-15T18:06:01.209611+0000","last_fullsized":"2024-01-15T18:06:01.209611+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:15:36.318765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.176469+0000","last_change":"2024-01-15T18:06:01.184378+0000","last_active":"2024-01-15T18:06:02.176469+0000","last_peered":"2024-01-15T18:06:02.176469+0000","last_clean":"2024-01-15T18:06:02.176469+0000","last_became_active":"2024-01-15T18:06:01.184115+0000","last_became_peered":"2024-01-15T18:06:01.184115+0000","last_unstale":"2024-01-15T18:06:02.176469+0000","last_undegraded":"2024-01-15T18:06:02.176469+0000","last_fullsized":"2024-01-15T18:06:02.176469+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:35:51.820223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.204985+0000","last_change":"2024-01-15T18:06:00.181570+0000","last_active":"2024-01-15T18:06:01.204985+0000","last_peered":"2024-01-15T18:06:01.204985+0000","last_clean":"2024-01-15T18:06:01.204985+0000","last_became_active":"2024-01-15T18:06:00.180958+0000","last_became_peered":"2024-01-15T18:06:00.180958+0000","last_unstale":"2024-01-15T18:06:01.204985+0000","last_undegraded":"2024-01-15T18:06:01.204985+0000","last_fullsized":"2024-01-15T18:06:01.204985+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:40:56.123776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.174633+0000","last_change":"2024-01-15T18:06:01.174681+0000","last_active":"2024-01-15T18:06:02.174633+0000","last_peered":"2024-01-15T18:06:02.174633+0000","last_clean":"2024-01-15T18:06:02.174633+0000","last_became_active":"2024-01-15T18:06:01.174387+0000","last_became_peered":"2024-01-15T18:06:01.174387+0000","last_unstale":"2024-01-15T18:06:02.174633+0000","last_undegraded":"2024-01-15T18:06:02.174633+0000","last_fullsized":"2024-01-15T18:06:02.174633+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:15.311517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.207366+0000","last_change":"2024-01-15T18:06:00.176281+0000","last_active":"2024-01-15T18:06:01.207366+0000","last_peered":"2024-01-15T18:06:01.207366+0000","last_clean":"2024-01-15T18:06:01.207366+0000","last_became_active":"2024-01-15T18:06:00.174503+0000","last_became_peered":"2024-01-15T18:06:00.174503+0000","last_unstale":"2024-01-15T18:06:01.207366+0000","last_undegraded":"2024-01-15T18:06:01.207366+0000","last_fullsized":"2024-01-15T18:06:01.207366+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:31:04.425248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259554+0000","last_change":"2024-01-15T18:06:01.181443+0000","last_active":"2024-01-15T18:06:02.259554+0000","last_peered":"2024-01-15T18:06:02.259554+0000","last_clean":"2024-01-15T18:06:02.259554+0000","last_became_active":"2024-01-15T18:06:01.181122+0000","last_became_peered":"2024-01-15T18:06:01.181122+0000","last_unstale":"2024-01-15T18:06:02.259554+0000","last_undegraded":"2024-01-15T18:06:02.259554+0000","last_fullsized":"2024-01-15T18:06:02.259554+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:51:47.039923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.163735+0000","last_change":"2024-01-15T18:06:00.190091+0000","last_active":"2024-01-15T18:06:01.163735+0000","last_peered":"2024-01-15T18:06:01.163735+0000","last_clean":"2024-01-15T18:06:01.163735+0000","last_became_active":"2024-01-15T18:06:00.189619+0000","last_became_peered":"2024-01-15T18:06:00.189619+0000","last_unstale":"2024-01-15T18:06:01.163735+0000","last_undegraded":"2024-01-15T18:06:01.163735+0000","last_fullsized":"2024-01-15T18:06:01.163735+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:59:02.797271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259156+0000","last_change":"2024-01-15T18:06:01.172767+0000","last_active":"2024-01-15T18:06:02.259156+0000","last_peered":"2024-01-15T18:06:02.259156+0000","last_clean":"2024-01-15T18:06:02.259156+0000","last_became_active":"2024-01-15T18:06:01.172483+0000","last_became_peered":"2024-01-15T18:06:01.172483+0000","last_unstale":"2024-01-15T18:06:02.259156+0000","last_undegraded":"2024-01-15T18:06:02.259156+0000","last_fullsized":"2024-01-15T18:06:02.259156+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:31:05.432116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259204+0000","last_change":"2024-01-15T18:06:00.190248+0000","last_active":"2024-01-15T18:06:02.259204+0000","last_peered":"2024-01-15T18:06:02.259204+0000","last_clean":"2024-01-15T18:06:02.259204+0000","last_became_active":"2024-01-15T18:06:00.189972+0000","last_became_peered":"2024-01-15T18:06:00.189972+0000","last_unstale":"2024-01-15T18:06:02.259204+0000","last_undegraded":"2024-01-15T18:06:02.259204+0000","last_fullsized":"2024-01-15T18:06:02.259204+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:46:26.889479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259484+0000","last_change":"2024-01-15T18:06:01.195352+0000","last_active":"2024-01-15T18:06:02.259484+0000","last_peered":"2024-01-15T18:06:02.259484+0000","last_clean":"2024-01-15T18:06:02.259484+0000","last_became_active":"2024-01-15T18:06:01.195053+0000","last_became_peered":"2024-01-15T18:06:01.195053+0000","last_unstale":"2024-01-15T18:06:02.259484+0000","last_undegraded":"2024-01-15T18:06:02.259484+0000","last_fullsized":"2024-01-15T18:06:02.259484+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:26:10.659136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.176825+0000","last_change":"2024-01-15T18:06:00.190356+0000","last_active":"2024-01-15T18:06:01.176825+0000","last_peered":"2024-01-15T18:06:01.176825+0000","last_clean":"2024-01-15T18:06:01.176825+0000","last_became_active":"2024-01-15T18:06:00.189527+0000","last_became_peered":"2024-01-15T18:06:00.189527+0000","last_unstale":"2024-01-15T18:06:01.176825+0000","last_undegraded":"2024-01-15T18:06:01.176825+0000","last_fullsized":"2024-01-15T18:06:01.176825+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:36:46.224457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258955+0000","last_change":"2024-01-15T18:06:01.189227+0000","last_active":"2024-01-15T18:06:02.258955+0000","last_peered":"2024-01-15T18:06:02.258955+0000","last_clean":"2024-01-15T18:06:02.258955+0000","last_became_active":"2024-01-15T18:06:01.188599+0000","last_became_peered":"2024-01-15T18:06:01.188599+0000","last_unstale":"2024-01-15T18:06:02.258955+0000","last_undegraded":"2024-01-15T18:06:02.258955+0000","last_fullsized":"2024-01-15T18:06:02.258955+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:57.743836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.163863+0000","last_change":"2024-01-15T18:06:00.179035+0000","last_active":"2024-01-15T18:06:01.163863+0000","last_peered":"2024-01-15T18:06:01.163863+0000","last_clean":"2024-01-15T18:06:01.163863+0000","last_became_active":"2024-01-15T18:06:00.178771+0000","last_became_peered":"2024-01-15T18:06:00.178771+0000","last_unstale":"2024-01-15T18:06:01.163863+0000","last_undegraded":"2024-01-15T18:06:01.163863+0000","last_fullsized":"2024-01-15T18:06:01.163863+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:12.325308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258926+0000","last_change":"2024-01-15T18:06:01.193458+0000","last_active":"2024-01-15T18:06:02.258926+0000","last_peered":"2024-01-15T18:06:02.258926+0000","last_clean":"2024-01-15T18:06:02.258926+0000","last_became_active":"2024-01-15T18:06:01.191417+0000","last_became_peered":"2024-01-15T18:06:01.191417+0000","last_unstale":"2024-01-15T18:06:02.258926+0000","last_undegraded":"2024-01-15T18:06:02.258926+0000","last_fullsized":"2024-01-15T18:06:02.258926+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:20:58.455307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.171988+0000","last_change":"2024-01-15T18:06:00.182290+0000","last_active":"2024-01-15T18:06:01.171988+0000","last_peered":"2024-01-15T18:06:01.171988+0000","last_clean":"2024-01-15T18:06:01.171988+0000","last_became_active":"2024-01-15T18:06:00.182016+0000","last_became_peered":"2024-01-15T18:06:00.182016+0000","last_unstale":"2024-01-15T18:06:01.171988+0000","last_undegraded":"2024-01-15T18:06:01.171988+0000","last_fullsized":"2024-01-15T18:06:01.171988+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:23:13.587394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259753+0000","last_change":"2024-01-15T18:06:01.184496+0000","last_active":"2024-01-15T18:06:02.259753+0000","last_peered":"2024-01-15T18:06:02.259753+0000","last_clean":"2024-01-15T18:06:02.259753+0000","last_became_active":"2024-01-15T18:06:01.184224+0000","last_became_peered":"2024-01-15T18:06:01.184224+0000","last_unstale":"2024-01-15T18:06:02.259753+0000","last_undegraded":"2024-01-15T18:06:02.259753+0000","last_fullsized":"2024-01-15T18:06:02.259753+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:36:47.231325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.208763+0000","last_change":"2024-01-15T18:06:00.176082+0000","last_active":"2024-01-15T18:06:01.208763+0000","last_peered":"2024-01-15T18:06:01.208763+0000","last_clean":"2024-01-15T18:06:01.208763+0000","last_became_active":"2024-01-15T18:06:00.175734+0000","last_became_peered":"2024-01-15T18:06:00.175734+0000","last_unstale":"2024-01-15T18:06:01.208763+0000","last_undegraded":"2024-01-15T18:06:01.208763+0000","last_fullsized":"2024-01-15T18:06:01.208763+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:43:45.408666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.176424+0000","last_change":"2024-01-15T18:06:01.187160+0000","last_active":"2024-01-15T18:06:02.176424+0000","last_peered":"2024-01-15T18:06:02.176424+0000","last_clean":"2024-01-15T18:06:02.176424+0000","last_became_active":"2024-01-15T18:06:01.186806+0000","last_became_peered":"2024-01-15T18:06:01.186806+0000","last_unstale":"2024-01-15T18:06:02.176424+0000","last_undegraded":"2024-01-15T18:06:02.176424+0000","last_fullsized":"2024-01-15T18:06:02.176424+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:37:32.291136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:03.273111+0000","last_change":"2024-01-15T18:06:00.183191+0000","last_active":"2024-01-15T18:06:03.273111+0000","last_peered":"2024-01-15T18:06:03.273111+0000","last_clean":"2024-01-15T18:06:03.273111+0000","last_became_active":"2024-01-15T18:06:00.182692+0000","last_became_peered":"2024-01-15T18:06:00.182692+0000","last_unstale":"2024-01-15T18:06:03.273111+0000","last_undegraded":"2024-01-15T18:06:03.273111+0000","last_fullsized":"2024-01-15T18:06:03.273111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:41:12.249000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259096+0000","last_change":"2024-01-15T18:06:01.194408+0000","last_active":"2024-01-15T18:06:02.259096+0000","last_peered":"2024-01-15T18:06:02.259096+0000","last_clean":"2024-01-15T18:06:02.259096+0000","last_became_active":"2024-01-15T18:06:01.194050+0000","last_became_peered":"2024-01-15T18:06:01.194050+0000","last_unstale":"2024-01-15T18:06:02.259096+0000","last_undegraded":"2024-01-15T18:06:02.259096+0000","last_fullsized":"2024-01-15T18:06:02.259096+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:55.114370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.169108+0000","last_change":"2024-01-15T18:06:00.181270+0000","last_active":"2024-01-15T18:06:01.169108+0000","last_peered":"2024-01-15T18:06:01.169108+0000","last_clean":"2024-01-15T18:06:01.169108+0000","last_became_active":"2024-01-15T18:06:00.180946+0000","last_became_peered":"2024-01-15T18:06:00.180946+0000","last_unstale":"2024-01-15T18:06:01.169108+0000","last_undegraded":"2024-01-15T18:06:01.169108+0000","last_fullsized":"2024-01-15T18:06:01.169108+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:55:19.741579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258840+0000","last_change":"2024-01-15T18:06:01.187228+0000","last_active":"2024-01-15T18:06:02.258840+0000","last_peered":"2024-01-15T18:06:02.258840+0000","last_clean":"2024-01-15T18:06:02.258840+0000","last_became_active":"2024-01-15T18:06:01.187018+0000","last_became_peered":"2024-01-15T18:06:01.187018+0000","last_unstale":"2024-01-15T18:06:02.258840+0000","last_undegraded":"2024-01-15T18:06:02.258840+0000","last_fullsized":"2024-01-15T18:06:02.258840+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:36:17.196617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259363+0000","last_change":"2024-01-15T18:06:00.188148+0000","last_active":"2024-01-15T18:06:02.259363+0000","last_peered":"2024-01-15T18:06:02.259363+0000","last_clean":"2024-01-15T18:06:02.259363+0000","last_became_active":"2024-01-15T18:06:00.187598+0000","last_became_peered":"2024-01-15T18:06:00.187598+0000","last_unstale":"2024-01-15T18:06:02.259363+0000","last_undegraded":"2024-01-15T18:06:02.259363+0000","last_fullsized":"2024-01-15T18:06:02.259363+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:51:01.940485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259121+0000","last_change":"2024-01-15T18:06:01.195224+0000","last_active":"2024-01-15T18:06:02.259121+0000","last_peered":"2024-01-15T18:06:02.259121+0000","last_clean":"2024-01-15T18:06:02.259121+0000","last_became_active":"2024-01-15T18:06:01.194852+0000","last_became_peered":"2024-01-15T18:06:01.194852+0000","last_unstale":"2024-01-15T18:06:02.259121+0000","last_undegraded":"2024-01-15T18:06:02.259121+0000","last_fullsized":"2024-01-15T18:06:02.259121+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:29.256704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.169150+0000","last_change":"2024-01-15T18:06:00.176729+0000","last_active":"2024-01-15T18:06:01.169150+0000","last_peered":"2024-01-15T18:06:01.169150+0000","last_clean":"2024-01-15T18:06:01.169150+0000","last_became_active":"2024-01-15T18:06:00.175928+0000","last_became_peered":"2024-01-15T18:06:00.175928+0000","last_unstale":"2024-01-15T18:06:01.169150+0000","last_undegraded":"2024-01-15T18:06:01.169150+0000","last_fullsized":"2024-01-15T18:06:01.169150+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:27:03.729350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259269+0000","last_change":"2024-01-15T18:06:01.176150+0000","last_active":"2024-01-15T18:06:02.259269+0000","last_peered":"2024-01-15T18:06:02.259269+0000","last_clean":"2024-01-15T18:06:02.259269+0000","last_became_active":"2024-01-15T18:06:01.175559+0000","last_became_peered":"2024-01-15T18:06:01.175559+0000","last_unstale":"2024-01-15T18:06:02.259269+0000","last_undegraded":"2024-01-15T18:06:02.259269+0000","last_fullsized":"2024-01-15T18:06:02.259269+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:41.300029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.163997+0000","last_change":"2024-01-15T18:06:00.178477+0000","last_active":"2024-01-15T18:06:01.163997+0000","last_peered":"2024-01-15T18:06:01.163997+0000","last_clean":"2024-01-15T18:06:01.163997+0000","last_became_active":"2024-01-15T18:06:00.176994+0000","last_became_peered":"2024-01-15T18:06:00.176994+0000","last_unstale":"2024-01-15T18:06:01.163997+0000","last_undegraded":"2024-01-15T18:06:01.163997+0000","last_fullsized":"2024-01-15T18:06:01.163997+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:56.736968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.176381+0000","last_change":"2024-01-15T18:06:01.173689+0000","last_active":"2024-01-15T18:06:02.176381+0000","last_peered":"2024-01-15T18:06:02.176381+0000","last_clean":"2024-01-15T18:06:02.176381+0000","last_became_active":"2024-01-15T18:06:01.172390+0000","last_became_peered":"2024-01-15T18:06:01.172390+0000","last_unstale":"2024-01-15T18:06:02.176381+0000","last_undegraded":"2024-01-15T18:06:02.176381+0000","last_fullsized":"2024-01-15T18:06:02.176381+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:15:37.325633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.160483+0000","last_change":"2024-01-15T18:06:00.184467+0000","last_active":"2024-01-15T18:06:01.160483+0000","last_peered":"2024-01-15T18:06:01.160483+0000","last_clean":"2024-01-15T18:06:01.160483+0000","last_became_active":"2024-01-15T18:06:00.184044+0000","last_became_peered":"2024-01-15T18:06:00.184044+0000","last_unstale":"2024-01-15T18:06:01.160483+0000","last_undegraded":"2024-01-15T18:06:01.160483+0000","last_fullsized":"2024-01-15T18:06:01.160483+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:52:50.973554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259499+0000","last_change":"2024-01-15T18:06:01.190807+0000","last_active":"2024-01-15T18:06:02.259499+0000","last_peered":"2024-01-15T18:06:02.259499+0000","last_clean":"2024-01-15T18:06:02.259499+0000","last_became_active":"2024-01-15T18:06:01.188659+0000","last_became_peered":"2024-01-15T18:06:01.188659+0000","last_unstale":"2024-01-15T18:06:02.259499+0000","last_undegraded":"2024-01-15T18:06:02.259499+0000","last_fullsized":"2024-01-15T18:06:02.259499+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:58:04.778601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.176553+0000","last_change":"2024-01-15T18:06:00.190042+0000","last_active":"2024-01-15T18:06:01.176553+0000","last_peered":"2024-01-15T18:06:01.176553+0000","last_clean":"2024-01-15T18:06:01.176553+0000","last_became_active":"2024-01-15T18:06:00.189749+0000","last_became_peered":"2024-01-15T18:06:00.189749+0000","last_unstale":"2024-01-15T18:06:01.176553+0000","last_undegraded":"2024-01-15T18:06:01.176553+0000","last_fullsized":"2024-01-15T18:06:01.176553+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:38:07.317901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":175,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.181400+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:06:01.181400+0000","last_peered":"2024-01-15T18:06:01.181400+0000","last_clean":"2024-01-15T18:06:01.181400+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:06:01.181400+0000","last_undegraded":"2024-01-15T18:06:01.181400+0000","last_fullsized":"2024-01-15T18:06:01.181400+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509459,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27980,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343860,"statfs":{"total":96636764160,"available":96608112640,"internally_reserved":0,"allocated":974848,"data_stored":738449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.77300000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.85199999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.747},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.874}]},{"osd":3,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.72499999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.502},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.58999999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.61099999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.52500000000000002}]}]},{"osd":6,"up_from":15,"seq":64424509459,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":385024,"data_stored":148607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.61899999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":2.4220000000000002,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.391,"5min":1.391,"15min":1.391},"last":0.73399999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.8700000000000001,"5min":2.8700000000000001,"15min":2.8700000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":0.67400000000000004}]},{"osd":3,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":2.012,"5min":2.012,"15min":2.012},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.532,"5min":2.532,"15min":2.532},"last":0.64000000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.38100000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.443},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.65600000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.503}]}]},{"osd":4,"up_from":15,"seq":64424509459,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":385024,"data_stored":148095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.67100000000000004}]},{"osd":1,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.68999999999999995}]},{"osd":2,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.81100000000000005}]},{"osd":3,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.751},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.71099999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.498},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.57299999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.72799999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.86499999999999999}]}]},{"osd":3,"up_from":15,"seq":64424509458,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":149625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.46899999999999997}]},{"osd":1,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.442},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.505}]},{"osd":2,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.45300000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.78800000000000003}]},{"osd":5,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.75900000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.67300000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.83699999999999997}]}]},{"osd":2,"up_from":15,"seq":64424509458,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":148061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.34999999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.443}]},{"osd":3,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.46200000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":1.109},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.466,"5min":1.466,"15min":1.466},"last":0.94299999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.89200000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.516,"5min":1.516,"15min":1.516},"last":1.0489999999999999},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":1.0089999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.6279999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":1.1299999999999999},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":1.1519999999999999}]}]},{"osd":1,"up_from":15,"seq":64424509460,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":149625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.50700000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":4.2450000000000001,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.59999999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.917,"5min":1.917,"15min":1.917},"last":0.46000000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.81899999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.79500000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.96099999999999997}]},{"osd":7,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.84299999999999997}]}]},{"osd":5,"up_from":14,"seq":60129542163,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":385024,"data_stored":148663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":2.802,"5min":2.802,"15min":2.802},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.63300000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.65400000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":0.82599999999999996}]},{"osd":3,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":2.3849999999999998,"5min":2.3849999999999998,"15min":2.3849999999999998},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":0.73399999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.75700000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.54700000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.57799999999999996}]}]},{"osd":0,"up_from":14,"seq":60129542163,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":738359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.50800000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.52800000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.325,"5min":1.325,"15min":1.325},"last":0.58799999999999997}]},{"osd":4,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.75700000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:05:43 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.9079999999999999,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":0.79900000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.764,"5min":1.764,"15min":1.764},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.806,"5min":1.806,"15min":1.806},"last":0.74399999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.722,"5min":1.722,"15min":1.722},"last":0.83899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:06:09.875 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:06:10.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.022+0000 7f2d36400640 1 Processor -- start 2024-01-15T18:06:10.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.022+0000 7f2d36400640 1 -- start start 2024-01-15T18:06:10.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.022+0000 7f2d36400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2d30071c70 0x7f2d3015ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:10.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.022+0000 7f2d36400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d300aff50 con 0x7f2d300b4a80 2024-01-15T18:06:10.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.022+0000 7f2d36400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d300b4ff0 con 0x7f2d30160dd0 2024-01-15T18:06:10.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.022+0000 7f2d36400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d300b5c00 con 0x7f2d30071c70 2024-01-15T18:06:10.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.026+0000 7f2d2f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2d30071c70 0x7f2d3015ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:10.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.026+0000 7f2d2f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2d30071c70 0x7f2d3015ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56650/0 (socket says 172.21.15.27:56650) 2024-01-15T18:06:10.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.026+0000 7f2d2f400640 1 -- 172.21.15.27:0/3941143486 learned_addr learned my addr 172.21.15.27:0/3941143486 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:10.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.026+0000 7f2d2f400640 1 -- 172.21.15.27:0/3941143486 >> v1:172.21.15.93:6789/0 conn(0x7f2d30160dd0 legacy=0x7f2d3006a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:10.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.026+0000 7f2d2f400640 1 -- 172.21.15.27:0/3941143486 >> v1:172.21.15.27:6789/0 conn(0x7f2d300b4a80 legacy=0x7f2d30072050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:10.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.026+0000 7f2d2f400640 1 -- 172.21.15.27:0/3941143486 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d300b52d0 con 0x7f2d30071c70 2024-01-15T18:06:10.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.026+0000 7f2d2f400640 1 --2- 172.21.15.27:0/3941143486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d30071c70 0x7f2d3015ebd0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2d20009870 tx=0x7f2d2001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bcbbe032b21ca36 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:10.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.026+0000 7f2d2e000640 1 -- 172.21.15.27:0/3941143486 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d200ab620 con 0x7f2d30071c70 2024-01-15T18:06:10.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.026+0000 7f2d2e000640 1 -- 172.21.15.27:0/3941143486 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d200ab780 con 0x7f2d30071c70 2024-01-15T18:06:10.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.026+0000 7f2d2e000640 1 -- 172.21.15.27:0/3941143486 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d200abb10 con 0x7f2d30071c70 2024-01-15T18:06:10.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.030+0000 7f2d36400640 1 -- 172.21.15.27:0/3941143486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d30071c70 msgr2=0x7f2d3015ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:10.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.030+0000 7f2d36400640 1 --2- 172.21.15.27:0/3941143486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d30071c70 0x7f2d3015ebd0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2d20009870 tx=0x7f2d2001c420 comp rx=0 tx=0).stop 2024-01-15T18:06:10.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.030+0000 7f2d36400640 1 -- 172.21.15.27:0/3941143486 shutdown_connections 2024-01-15T18:06:10.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.030+0000 7f2d36400640 1 --2- 172.21.15.27:0/3941143486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d30071c70 0x7f2d3015ebd0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:10.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.030+0000 7f2d36400640 1 -- 172.21.15.27:0/3941143486 >> 172.21.15.27:0/3941143486 conn(0x7f2d300b2360 msgr2=0x7f2d300b2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:10.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.030+0000 7f2d36400640 1 -- 172.21.15.27:0/3941143486 shutdown_connections 2024-01-15T18:06:10.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.030+0000 7f2d36400640 1 -- 172.21.15.27:0/3941143486 wait complete. 2024-01-15T18:06:10.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.030+0000 7f2d36400640 1 Processor -- start 2024-01-15T18:06:10.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.030+0000 7f2d36400640 1 -- start start 2024-01-15T18:06:10.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.030+0000 7f2d36400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d30071c70 0x7f2d3009c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.030+0000 7f2d36400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2d300b4a80 0x7f2d30154130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.030+0000 7f2d36400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2d30160dd0 0x7f2d301512b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.030+0000 7f2d36400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3009d290 con 0x7f2d30071c70 2024-01-15T18:06:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.030+0000 7f2d36400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d30153d10 con 0x7f2d300b4a80 2024-01-15T18:06:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.030+0000 7f2d36400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d30153090 con 0x7f2d30160dd0 2024-01-15T18:06:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.030+0000 7f2d2f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d30071c70 0x7f2d3009c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.030+0000 7f2d2f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d30071c70 0x7f2d3009c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56658/0 (socket says 172.21.15.27:56658) 2024-01-15T18:06:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.030+0000 7f2d2f400640 1 -- 172.21.15.27:0/1001159617 learned_addr learned my addr 172.21.15.27:0/1001159617 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.030+0000 7f2d2ea00640 1 --2- 172.21.15.27:0/1001159617 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2d300b4a80 0x7f2d30154130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.034+0000 7f2d34e00640 1 --2- 172.21.15.27:0/1001159617 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2d30160dd0 0x7f2d301512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.034+0000 7f2d2f400640 1 -- 172.21.15.27:0/1001159617 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2d30160dd0 msgr2=0x7f2d301512b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.034+0000 7f2d2f400640 1 --2- 172.21.15.27:0/1001159617 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2d30160dd0 0x7f2d301512b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.034+0000 7f2d2f400640 1 -- 172.21.15.27:0/1001159617 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2d300b4a80 msgr2=0x7f2d30154130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.034+0000 7f2d2f400640 1 --2- 172.21.15.27:0/1001159617 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2d300b4a80 0x7f2d30154130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:10.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.034+0000 7f2d2f400640 1 -- 172.21.15.27:0/1001159617 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d200094d0 con 0x7f2d30071c70 2024-01-15T18:06:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.034+0000 7f2d2f400640 1 --2- 172.21.15.27:0/1001159617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d30071c70 0x7f2d3009c170 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2d200ad040 tx=0x7f2d2001ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:10.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.034+0000 7f2d27400640 1 -- 172.21.15.27:0/1001159617 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d200be3f0 con 0x7f2d30071c70 2024-01-15T18:06:10.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.034+0000 7f2d27400640 1 -- 172.21.15.27:0/1001159617 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d200be550 con 0x7f2d30071c70 2024-01-15T18:06:10.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.034+0000 7f2d36400640 1 -- 172.21.15.27:0/1001159617 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d30153370 con 0x7f2d30071c70 2024-01-15T18:06:10.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.034+0000 7f2d27400640 1 -- 172.21.15.27:0/1001159617 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d200be8e0 con 0x7f2d30071c70 2024-01-15T18:06:10.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.034+0000 7f2d36400640 1 -- 172.21.15.27:0/1001159617 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d30153930 con 0x7f2d30071c70 2024-01-15T18:06:10.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.034+0000 7f2d36400640 1 -- 172.21.15.27:0/1001159617 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d30067a10 con 0x7f2d30071c70 2024-01-15T18:06:10.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.034+0000 7f2d27400640 1 -- 172.21.15.27:0/1001159617 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f2d200c7440 con 0x7f2d30071c70 2024-01-15T18:06:10.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.038+0000 7f2d27400640 1 --2- 172.21.15.27:0/1001159617 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2d08066920 0x7f2d08068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:10.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.038+0000 7f2d27400640 1 -- 172.21.15.27:0/1001159617 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2d2011eb60 con 0x7f2d30071c70 2024-01-15T18:06:10.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.038+0000 7f2d2ea00640 1 --2- 172.21.15.27:0/1001159617 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2d08066920 0x7f2d08068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:10.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.038+0000 7f2d2ea00640 1 --2- 172.21.15.27:0/1001159617 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2d08066920 0x7f2d08068de0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2d300ae350 tx=0x7f2d18009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:10.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.042+0000 7f2d27400640 1 -- 172.21.15.27:0/1001159617 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2d200ebf60 con 0x7f2d30071c70 2024-01-15T18:06:10.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.194+0000 7f2d36400640 1 -- 172.21.15.27:0/1001159617 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2d3015e630 con 0x7f2d08066920 2024-01-15T18:06:10.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.202+0000 7f2d27400640 1 -- 172.21.15.27:0/1001159617 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288971 (secure 0 0 0) 0x7f2d18001450 con 0x7f2d08066920 2024-01-15T18:06:10.211 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:06:10.213 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:06:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.206+0000 7f2d36400640 1 -- 172.21.15.27:0/1001159617 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2d08066920 msgr2=0x7f2d08068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.206+0000 7f2d36400640 1 --2- 172.21.15.27:0/1001159617 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2d08066920 0x7f2d08068de0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2d300ae350 tx=0x7f2d18009040 comp rx=0 tx=0).stop 2024-01-15T18:06:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.206+0000 7f2d36400640 1 -- 172.21.15.27:0/1001159617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d30071c70 msgr2=0x7f2d3009c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.206+0000 7f2d36400640 1 --2- 172.21.15.27:0/1001159617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d30071c70 0x7f2d3009c170 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2d200ad040 tx=0x7f2d2001ac60 comp rx=0 tx=0).stop 2024-01-15T18:06:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.206+0000 7f2d36400640 1 -- 172.21.15.27:0/1001159617 shutdown_connections 2024-01-15T18:06:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.206+0000 7f2d36400640 1 --2- 172.21.15.27:0/1001159617 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2d08066920 0x7f2d08068de0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:10.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.206+0000 7f2d36400640 1 --2- 172.21.15.27:0/1001159617 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2d30160dd0 0x7f2d301512b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:10.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.206+0000 7f2d36400640 1 --2- 172.21.15.27:0/1001159617 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2d300b4a80 0x7f2d30154130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:10.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.206+0000 7f2d36400640 1 --2- 172.21.15.27:0/1001159617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d30071c70 0x7f2d3009c170 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:10.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.206+0000 7f2d36400640 1 -- 172.21.15.27:0/1001159617 >> 172.21.15.27:0/1001159617 conn(0x7f2d300b2360 msgr2=0x7f2d300d0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:10.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.206+0000 7f2d36400640 1 -- 172.21.15.27:0/1001159617 shutdown_connections 2024-01-15T18:06:10.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.206+0000 7f2d36400640 1 -- 172.21.15.27:0/1001159617 wait complete. 2024-01-15T18:06:10.234 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":70,"stamp":"2024-01-15T18:06:10.155015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224504,"kb_used_data":4168,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750216,"statfs":{"total":773094113280,"available":772864221184,"internally_reserved":0,"allocated":4268032,"data_stored":2369484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.991919"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258756+0000","last_change":"2024-01-15T18:06:01.189891+0000","last_active":"2024-01-15T18:06:02.258756+0000","last_peered":"2024-01-15T18:06:02.258756+0000","last_clean":"2024-01-15T18:06:02.258756+0000","last_became_active":"2024-01-15T18:06:01.189621+0000","last_became_peered":"2024-01-15T18:06:01.189621+0000","last_unstale":"2024-01-15T18:06:02.258756+0000","last_undegraded":"2024-01-15T18:06:02.258756+0000","last_fullsized":"2024-01-15T18:06:02.258756+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:26:25.916306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.260287+0000","last_change":"2024-01-15T18:06:01.193972+0000","last_active":"2024-01-15T18:06:02.260287+0000","last_peered":"2024-01-15T18:06:02.260287+0000","last_clean":"2024-01-15T18:06:02.260287+0000","last_became_active":"2024-01-15T18:06:01.193681+0000","last_became_peered":"2024-01-15T18:06:01.193681+0000","last_unstale":"2024-01-15T18:06:02.260287+0000","last_undegraded":"2024-01-15T18:06:02.260287+0000","last_fullsized":"2024-01-15T18:06:02.260287+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:58:04.778601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.176014+0000","last_change":"2024-01-15T18:06:01.189390+0000","last_active":"2024-01-15T18:06:02.176014+0000","last_peered":"2024-01-15T18:06:02.176014+0000","last_clean":"2024-01-15T18:06:02.176014+0000","last_became_active":"2024-01-15T18:06:01.188747+0000","last_became_peered":"2024-01-15T18:06:01.188747+0000","last_unstale":"2024-01-15T18:06:02.176014+0000","last_undegraded":"2024-01-15T18:06:02.176014+0000","last_fullsized":"2024-01-15T18:06:02.176014+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:38:31.001343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.174594+0000","last_change":"2024-01-15T18:06:01.183654+0000","last_active":"2024-01-15T18:06:02.174594+0000","last_peered":"2024-01-15T18:06:02.174594+0000","last_clean":"2024-01-15T18:06:02.174594+0000","last_became_active":"2024-01-15T18:06:01.183228+0000","last_became_peered":"2024-01-15T18:06:01.183228+0000","last_unstale":"2024-01-15T18:06:02.174594+0000","last_undegraded":"2024-01-15T18:06:02.174594+0000","last_fullsized":"2024-01-15T18:06:02.174594+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:41:31.415431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258875+0000","last_change":"2024-01-15T18:06:01.178251+0000","last_active":"2024-01-15T18:06:02.258875+0000","last_peered":"2024-01-15T18:06:02.258875+0000","last_clean":"2024-01-15T18:06:02.258875+0000","last_became_active":"2024-01-15T18:06:01.175580+0000","last_became_peered":"2024-01-15T18:06:01.175580+0000","last_unstale":"2024-01-15T18:06:02.258875+0000","last_undegraded":"2024-01-15T18:06:02.258875+0000","last_fullsized":"2024-01-15T18:06:02.258875+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:13.332175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259387+0000","last_change":"2024-01-15T18:06:01.187886+0000","last_active":"2024-01-15T18:06:02.259387+0000","last_peered":"2024-01-15T18:06:02.259387+0000","last_clean":"2024-01-15T18:06:02.259387+0000","last_became_active":"2024-01-15T18:06:01.187610+0000","last_became_peered":"2024-01-15T18:06:01.187610+0000","last_unstale":"2024-01-15T18:06:02.259387+0000","last_undegraded":"2024-01-15T18:06:02.259387+0000","last_fullsized":"2024-01-15T18:06:02.259387+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:16:22.277642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259320+0000","last_change":"2024-01-15T18:06:01.193492+0000","last_active":"2024-01-15T18:06:02.259320+0000","last_peered":"2024-01-15T18:06:02.259320+0000","last_clean":"2024-01-15T18:06:02.259320+0000","last_became_active":"2024-01-15T18:06:01.190348+0000","last_became_peered":"2024-01-15T18:06:01.190348+0000","last_unstale":"2024-01-15T18:06:02.259320+0000","last_undegraded":"2024-01-15T18:06:02.259320+0000","last_fullsized":"2024-01-15T18:06:02.259320+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:09:25.929285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258606+0000","last_change":"2024-01-15T18:06:01.188695+0000","last_active":"2024-01-15T18:06:02.258606+0000","last_peered":"2024-01-15T18:06:02.258606+0000","last_clean":"2024-01-15T18:06:02.258606+0000","last_became_active":"2024-01-15T18:06:01.184708+0000","last_became_peered":"2024-01-15T18:06:01.184708+0000","last_unstale":"2024-01-15T18:06:02.258606+0000","last_undegraded":"2024-01-15T18:06:02.258606+0000","last_fullsized":"2024-01-15T18:06:02.258606+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:12:07.957753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.177002+0000","last_change":"2024-01-15T18:06:01.182729+0000","last_active":"2024-01-15T18:06:02.177002+0000","last_peered":"2024-01-15T18:06:02.177002+0000","last_clean":"2024-01-15T18:06:02.177002+0000","last_became_active":"2024-01-15T18:06:01.182433+0000","last_became_peered":"2024-01-15T18:06:01.182433+0000","last_unstale":"2024-01-15T18:06:02.177002+0000","last_undegraded":"2024-01-15T18:06:02.177002+0000","last_fullsized":"2024-01-15T18:06:02.177002+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:15.311517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.174963+0000","last_change":"2024-01-15T18:06:01.179590+0000","last_active":"2024-01-15T18:06:02.174963+0000","last_peered":"2024-01-15T18:06:02.174963+0000","last_clean":"2024-01-15T18:06:02.174963+0000","last_became_active":"2024-01-15T18:06:01.179235+0000","last_became_peered":"2024-01-15T18:06:01.179235+0000","last_unstale":"2024-01-15T18:06:02.174963+0000","last_undegraded":"2024-01-15T18:06:02.174963+0000","last_fullsized":"2024-01-15T18:06:02.174963+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:08:45.088678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259464+0000","last_change":"2024-01-15T18:06:01.193835+0000","last_active":"2024-01-15T18:06:02.259464+0000","last_peered":"2024-01-15T18:06:02.259464+0000","last_clean":"2024-01-15T18:06:02.259464+0000","last_became_active":"2024-01-15T18:06:01.193229+0000","last_became_peered":"2024-01-15T18:06:01.193229+0000","last_unstale":"2024-01-15T18:06:02.259464+0000","last_undegraded":"2024-01-15T18:06:02.259464+0000","last_fullsized":"2024-01-15T18:06:02.259464+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:15:23.520645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.175901+0000","last_change":"2024-01-15T18:06:01.189227+0000","last_active":"2024-01-15T18:06:02.175901+0000","last_peered":"2024-01-15T18:06:02.175901+0000","last_clean":"2024-01-15T18:06:02.175901+0000","last_became_active":"2024-01-15T18:06:01.188807+0000","last_became_peered":"2024-01-15T18:06:01.188807+0000","last_unstale":"2024-01-15T18:06:02.175901+0000","last_undegraded":"2024-01-15T18:06:02.175901+0000","last_fullsized":"2024-01-15T18:06:02.175901+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:49:18.322259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259000+0000","last_change":"2024-01-15T18:06:01.184266+0000","last_active":"2024-01-15T18:06:02.259000+0000","last_peered":"2024-01-15T18:06:02.259000+0000","last_clean":"2024-01-15T18:06:02.259000+0000","last_became_active":"2024-01-15T18:06:01.183933+0000","last_became_peered":"2024-01-15T18:06:01.183933+0000","last_unstale":"2024-01-15T18:06:02.259000+0000","last_undegraded":"2024-01-15T18:06:02.259000+0000","last_fullsized":"2024-01-15T18:06:02.259000+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:40:57.130644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259845+0000","last_change":"2024-01-15T18:06:01.175248+0000","last_active":"2024-01-15T18:06:02.259845+0000","last_peered":"2024-01-15T18:06:02.259845+0000","last_clean":"2024-01-15T18:06:02.259845+0000","last_became_active":"2024-01-15T18:06:01.174941+0000","last_became_peered":"2024-01-15T18:06:01.174941+0000","last_unstale":"2024-01-15T18:06:02.259845+0000","last_undegraded":"2024-01-15T18:06:02.259845+0000","last_fullsized":"2024-01-15T18:06:02.259845+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:20:58.455307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.175860+0000","last_change":"2024-01-15T18:06:01.183569+0000","last_active":"2024-01-15T18:06:02.175860+0000","last_peered":"2024-01-15T18:06:02.175860+0000","last_clean":"2024-01-15T18:06:02.175860+0000","last_became_active":"2024-01-15T18:06:01.183236+0000","last_became_peered":"2024-01-15T18:06:01.183236+0000","last_unstale":"2024-01-15T18:06:02.175860+0000","last_undegraded":"2024-01-15T18:06:02.175860+0000","last_fullsized":"2024-01-15T18:06:02.175860+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:43:51.863373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258744+0000","last_change":"2024-01-15T18:06:01.192491+0000","last_active":"2024-01-15T18:06:02.258744+0000","last_peered":"2024-01-15T18:06:02.258744+0000","last_clean":"2024-01-15T18:06:02.258744+0000","last_became_active":"2024-01-15T18:06:01.192015+0000","last_became_peered":"2024-01-15T18:06:01.192015+0000","last_unstale":"2024-01-15T18:06:02.258744+0000","last_undegraded":"2024-01-15T18:06:02.258744+0000","last_fullsized":"2024-01-15T18:06:02.258744+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:16:22.277642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259443+0000","last_change":"2024-01-15T18:06:01.194135+0000","last_active":"2024-01-15T18:06:02.259443+0000","last_peered":"2024-01-15T18:06:02.259443+0000","last_clean":"2024-01-15T18:06:02.259443+0000","last_became_active":"2024-01-15T18:06:01.193839+0000","last_became_peered":"2024-01-15T18:06:01.193839+0000","last_unstale":"2024-01-15T18:06:02.259443+0000","last_undegraded":"2024-01-15T18:06:02.259443+0000","last_fullsized":"2024-01-15T18:06:02.259443+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:18:47.499122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.260288+0000","last_change":"2024-01-15T18:06:01.174507+0000","last_active":"2024-01-15T18:06:02.260288+0000","last_peered":"2024-01-15T18:06:02.260288+0000","last_clean":"2024-01-15T18:06:02.260288+0000","last_became_active":"2024-01-15T18:06:01.174222+0000","last_became_peered":"2024-01-15T18:06:01.174222+0000","last_unstale":"2024-01-15T18:06:02.260288+0000","last_undegraded":"2024-01-15T18:06:02.260288+0000","last_fullsized":"2024-01-15T18:06:02.260288+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:15:23.520645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259924+0000","last_change":"2024-01-15T18:06:01.184671+0000","last_active":"2024-01-15T18:06:02.259924+0000","last_peered":"2024-01-15T18:06:02.259924+0000","last_clean":"2024-01-15T18:06:02.259924+0000","last_became_active":"2024-01-15T18:06:01.184407+0000","last_became_peered":"2024-01-15T18:06:01.184407+0000","last_unstale":"2024-01-15T18:06:02.259924+0000","last_undegraded":"2024-01-15T18:06:02.259924+0000","last_fullsized":"2024-01-15T18:06:02.259924+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:00:34.319443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259643+0000","last_change":"2024-01-15T18:06:01.183764+0000","last_active":"2024-01-15T18:06:02.259643+0000","last_peered":"2024-01-15T18:06:02.259643+0000","last_clean":"2024-01-15T18:06:02.259643+0000","last_became_active":"2024-01-15T18:06:01.183530+0000","last_became_peered":"2024-01-15T18:06:01.183530+0000","last_unstale":"2024-01-15T18:06:02.259643+0000","last_undegraded":"2024-01-15T18:06:02.259643+0000","last_fullsized":"2024-01-15T18:06:02.259643+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:38:08.324768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.177103+0000","last_change":"2024-01-15T18:06:01.188277+0000","last_active":"2024-01-15T18:06:02.177103+0000","last_peered":"2024-01-15T18:06:02.177103+0000","last_clean":"2024-01-15T18:06:02.177103+0000","last_became_active":"2024-01-15T18:06:01.187945+0000","last_became_peered":"2024-01-15T18:06:01.187945+0000","last_unstale":"2024-01-15T18:06:02.177103+0000","last_undegraded":"2024-01-15T18:06:02.177103+0000","last_fullsized":"2024-01-15T18:06:02.177103+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:08:45.088678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.176680+0000","last_change":"2024-01-15T18:06:01.172951+0000","last_active":"2024-01-15T18:06:02.176680+0000","last_peered":"2024-01-15T18:06:02.176680+0000","last_clean":"2024-01-15T18:06:02.176680+0000","last_became_active":"2024-01-15T18:06:01.172634+0000","last_became_peered":"2024-01-15T18:06:01.172634+0000","last_unstale":"2024-01-15T18:06:02.176680+0000","last_undegraded":"2024-01-15T18:06:02.176680+0000","last_fullsized":"2024-01-15T18:06:02.176680+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:52:51.980421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258714+0000","last_change":"2024-01-15T18:06:01.192000+0000","last_active":"2024-01-15T18:06:02.258714+0000","last_peered":"2024-01-15T18:06:02.258714+0000","last_clean":"2024-01-15T18:06:02.258714+0000","last_became_active":"2024-01-15T18:06:01.191712+0000","last_became_peered":"2024-01-15T18:06:01.191712+0000","last_unstale":"2024-01-15T18:06:02.258714+0000","last_undegraded":"2024-01-15T18:06:02.258714+0000","last_fullsized":"2024-01-15T18:06:02.258714+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:49:34.739781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.175593+0000","last_change":"2024-01-15T18:06:01.177362+0000","last_active":"2024-01-15T18:06:02.175593+0000","last_peered":"2024-01-15T18:06:02.175593+0000","last_clean":"2024-01-15T18:06:02.175593+0000","last_became_active":"2024-01-15T18:06:01.177047+0000","last_became_peered":"2024-01-15T18:06:01.177047+0000","last_unstale":"2024-01-15T18:06:02.175593+0000","last_undegraded":"2024-01-15T18:06:02.175593+0000","last_fullsized":"2024-01-15T18:06:02.175593+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:15.719166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.175027+0000","last_change":"2024-01-15T18:06:01.191258+0000","last_active":"2024-01-15T18:06:02.175027+0000","last_peered":"2024-01-15T18:06:02.175027+0000","last_clean":"2024-01-15T18:06:02.175027+0000","last_became_active":"2024-01-15T18:06:01.190908+0000","last_became_peered":"2024-01-15T18:06:01.190908+0000","last_unstale":"2024-01-15T18:06:02.175027+0000","last_undegraded":"2024-01-15T18:06:02.175027+0000","last_fullsized":"2024-01-15T18:06:02.175027+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:50:20.923618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.260378+0000","last_change":"2024-01-15T18:06:01.190720+0000","last_active":"2024-01-15T18:06:02.260378+0000","last_peered":"2024-01-15T18:06:02.260378+0000","last_clean":"2024-01-15T18:06:02.260378+0000","last_became_active":"2024-01-15T18:06:01.190391+0000","last_became_peered":"2024-01-15T18:06:01.190391+0000","last_unstale":"2024-01-15T18:06:02.260378+0000","last_undegraded":"2024-01-15T18:06:02.260378+0000","last_fullsized":"2024-01-15T18:06:02.260378+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:43:52.778742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.174985+0000","last_change":"2024-01-15T18:06:01.186658+0000","last_active":"2024-01-15T18:06:02.174985+0000","last_peered":"2024-01-15T18:06:02.174985+0000","last_clean":"2024-01-15T18:06:02.174985+0000","last_became_active":"2024-01-15T18:06:01.183065+0000","last_became_peered":"2024-01-15T18:06:01.183065+0000","last_unstale":"2024-01-15T18:06:02.174985+0000","last_undegraded":"2024-01-15T18:06:02.174985+0000","last_fullsized":"2024-01-15T18:06:02.174985+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:02.935448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.175887+0000","last_change":"2024-01-15T18:06:01.194852+0000","last_active":"2024-01-15T18:06:02.175887+0000","last_peered":"2024-01-15T18:06:02.175887+0000","last_clean":"2024-01-15T18:06:02.175887+0000","last_became_active":"2024-01-15T18:06:01.194509+0000","last_became_peered":"2024-01-15T18:06:01.194509+0000","last_unstale":"2024-01-15T18:06:02.175887+0000","last_undegraded":"2024-01-15T18:06:02.175887+0000","last_fullsized":"2024-01-15T18:06:02.175887+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:48:05.723516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.176765+0000","last_change":"2024-01-15T18:06:01.180504+0000","last_active":"2024-01-15T18:06:02.176765+0000","last_peered":"2024-01-15T18:06:02.176765+0000","last_clean":"2024-01-15T18:06:02.176765+0000","last_became_active":"2024-01-15T18:06:01.180161+0000","last_became_peered":"2024-01-15T18:06:01.180161+0000","last_unstale":"2024-01-15T18:06:02.176765+0000","last_undegraded":"2024-01-15T18:06:02.176765+0000","last_fullsized":"2024-01-15T18:06:02.176765+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:25:51.645494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259370+0000","last_change":"2024-01-15T18:06:01.182624+0000","last_active":"2024-01-15T18:06:02.259370+0000","last_peered":"2024-01-15T18:06:02.259370+0000","last_clean":"2024-01-15T18:06:02.259370+0000","last_became_active":"2024-01-15T18:06:01.182328+0000","last_became_peered":"2024-01-15T18:06:01.182328+0000","last_unstale":"2024-01-15T18:06:02.259370+0000","last_undegraded":"2024-01-15T18:06:02.259370+0000","last_fullsized":"2024-01-15T18:06:02.259370+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:23:07.800413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258805+0000","last_change":"2024-01-15T18:06:01.185113+0000","last_active":"2024-01-15T18:06:02.258805+0000","last_peered":"2024-01-15T18:06:02.258805+0000","last_clean":"2024-01-15T18:06:02.258805+0000","last_became_active":"2024-01-15T18:06:01.184857+0000","last_became_peered":"2024-01-15T18:06:01.184857+0000","last_unstale":"2024-01-15T18:06:02.258805+0000","last_undegraded":"2024-01-15T18:06:02.258805+0000","last_fullsized":"2024-01-15T18:06:02.258805+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:43:35.040533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258552+0000","last_change":"2024-01-15T18:06:01.189709+0000","last_active":"2024-01-15T18:06:02.258552+0000","last_peered":"2024-01-15T18:06:02.258552+0000","last_clean":"2024-01-15T18:06:02.258552+0000","last_became_active":"2024-01-15T18:06:01.185069+0000","last_became_peered":"2024-01-15T18:06:01.185069+0000","last_unstale":"2024-01-15T18:06:02.258552+0000","last_undegraded":"2024-01-15T18:06:02.258552+0000","last_fullsized":"2024-01-15T18:06:02.258552+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:42:55.460579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259388+0000","last_change":"2024-01-15T18:06:01.191979+0000","last_active":"2024-01-15T18:06:02.259388+0000","last_peered":"2024-01-15T18:06:02.259388+0000","last_clean":"2024-01-15T18:06:02.259388+0000","last_became_active":"2024-01-15T18:06:01.191642+0000","last_became_peered":"2024-01-15T18:06:01.191642+0000","last_unstale":"2024-01-15T18:06:02.259388+0000","last_undegraded":"2024-01-15T18:06:02.259388+0000","last_fullsized":"2024-01-15T18:06:02.259388+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:01:18.922210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.208366+0000","last_change":"2024-01-15T18:06:00.191056+0000","last_active":"2024-01-15T18:06:01.208366+0000","last_peered":"2024-01-15T18:06:01.208366+0000","last_clean":"2024-01-15T18:06:01.208366+0000","last_became_active":"2024-01-15T18:06:00.190695+0000","last_became_peered":"2024-01-15T18:06:00.190695+0000","last_unstale":"2024-01-15T18:06:01.208366+0000","last_undegraded":"2024-01-15T18:06:01.208366+0000","last_fullsized":"2024-01-15T18:06:01.208366+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:51:46.033055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258879+0000","last_change":"2024-01-15T18:06:01.184402+0000","last_active":"2024-01-15T18:06:02.258879+0000","last_peered":"2024-01-15T18:06:02.258879+0000","last_clean":"2024-01-15T18:06:02.258879+0000","last_became_active":"2024-01-15T18:06:01.184128+0000","last_became_peered":"2024-01-15T18:06:01.184128+0000","last_unstale":"2024-01-15T18:06:02.258879+0000","last_undegraded":"2024-01-15T18:06:02.258879+0000","last_fullsized":"2024-01-15T18:06:02.258879+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:55:20.748447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259476+0000","last_change":"2024-01-15T18:06:00.187711+0000","last_active":"2024-01-15T18:06:02.259476+0000","last_peered":"2024-01-15T18:06:02.259476+0000","last_clean":"2024-01-15T18:06:02.259476+0000","last_became_active":"2024-01-15T18:06:00.187351+0000","last_became_peered":"2024-01-15T18:06:00.187351+0000","last_unstale":"2024-01-15T18:06:02.259476+0000","last_undegraded":"2024-01-15T18:06:02.259476+0000","last_fullsized":"2024-01-15T18:06:02.259476+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:45:53.529417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.176955+0000","last_change":"2024-01-15T18:06:01.175024+0000","last_active":"2024-01-15T18:06:02.176955+0000","last_peered":"2024-01-15T18:06:02.176955+0000","last_clean":"2024-01-15T18:06:02.176955+0000","last_became_active":"2024-01-15T18:06:01.174755+0000","last_became_peered":"2024-01-15T18:06:01.174755+0000","last_unstale":"2024-01-15T18:06:02.176955+0000","last_undegraded":"2024-01-15T18:06:02.176955+0000","last_fullsized":"2024-01-15T18:06:02.176955+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:41:31.415431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.210775+0000","last_change":"2024-01-15T18:06:00.194271+0000","last_active":"2024-01-15T18:06:01.210775+0000","last_peered":"2024-01-15T18:06:01.210775+0000","last_clean":"2024-01-15T18:06:01.210775+0000","last_became_active":"2024-01-15T18:06:00.193996+0000","last_became_peered":"2024-01-15T18:06:00.193996+0000","last_unstale":"2024-01-15T18:06:01.210775+0000","last_undegraded":"2024-01-15T18:06:01.210775+0000","last_fullsized":"2024-01-15T18:06:01.210775+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:01.928581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.175025+0000","last_change":"2024-01-15T18:06:01.184800+0000","last_active":"2024-01-15T18:06:02.175025+0000","last_peered":"2024-01-15T18:06:02.175025+0000","last_clean":"2024-01-15T18:06:02.175025+0000","last_became_active":"2024-01-15T18:06:01.183424+0000","last_became_peered":"2024-01-15T18:06:01.183424+0000","last_unstale":"2024-01-15T18:06:02.175025+0000","last_undegraded":"2024-01-15T18:06:02.175025+0000","last_fullsized":"2024-01-15T18:06:02.175025+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:42:55.460579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.205682+0000","last_change":"2024-01-15T18:06:00.178063+0000","last_active":"2024-01-15T18:06:01.205682+0000","last_peered":"2024-01-15T18:06:01.205682+0000","last_clean":"2024-01-15T18:06:01.205682+0000","last_became_active":"2024-01-15T18:06:00.177628+0000","last_became_peered":"2024-01-15T18:06:00.177628+0000","last_unstale":"2024-01-15T18:06:01.205682+0000","last_undegraded":"2024-01-15T18:06:01.205682+0000","last_fullsized":"2024-01-15T18:06:01.205682+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:08:08.490734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259226+0000","last_change":"2024-01-15T18:06:01.196654+0000","last_active":"2024-01-15T18:06:02.259226+0000","last_peered":"2024-01-15T18:06:02.259226+0000","last_clean":"2024-01-15T18:06:02.259226+0000","last_became_active":"2024-01-15T18:06:01.196278+0000","last_became_peered":"2024-01-15T18:06:01.196278+0000","last_unstale":"2024-01-15T18:06:02.259226+0000","last_undegraded":"2024-01-15T18:06:02.259226+0000","last_fullsized":"2024-01-15T18:06:02.259226+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:00:34.319443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.209388+0000","last_change":"2024-01-15T18:06:00.193082+0000","last_active":"2024-01-15T18:06:01.209388+0000","last_peered":"2024-01-15T18:06:01.209388+0000","last_clean":"2024-01-15T18:06:01.209388+0000","last_became_active":"2024-01-15T18:06:00.192750+0000","last_became_peered":"2024-01-15T18:06:00.192750+0000","last_unstale":"2024-01-15T18:06:01.209388+0000","last_undegraded":"2024-01-15T18:06:01.209388+0000","last_fullsized":"2024-01-15T18:06:01.209388+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:43:34.033665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.176654+0000","last_change":"2024-01-15T18:06:01.195416+0000","last_active":"2024-01-15T18:06:02.176654+0000","last_peered":"2024-01-15T18:06:02.176654+0000","last_clean":"2024-01-15T18:06:02.176654+0000","last_became_active":"2024-01-15T18:06:01.195155+0000","last_became_peered":"2024-01-15T18:06:01.195155+0000","last_unstale":"2024-01-15T18:06:02.176654+0000","last_undegraded":"2024-01-15T18:06:02.176654+0000","last_fullsized":"2024-01-15T18:06:02.176654+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:22:40.694092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.204726+0000","last_change":"2024-01-15T18:06:00.191810+0000","last_active":"2024-01-15T18:06:01.204726+0000","last_peered":"2024-01-15T18:06:01.204726+0000","last_clean":"2024-01-15T18:06:01.204726+0000","last_became_active":"2024-01-15T18:06:00.191470+0000","last_became_peered":"2024-01-15T18:06:00.191470+0000","last_unstale":"2024-01-15T18:06:01.204726+0000","last_undegraded":"2024-01-15T18:06:01.204726+0000","last_fullsized":"2024-01-15T18:06:01.204726+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:27:03.729350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.176435+0000","last_change":"2024-01-15T18:06:01.181141+0000","last_active":"2024-01-15T18:06:02.176435+0000","last_peered":"2024-01-15T18:06:02.176435+0000","last_clean":"2024-01-15T18:06:02.176435+0000","last_became_active":"2024-01-15T18:06:01.180747+0000","last_became_peered":"2024-01-15T18:06:01.180747+0000","last_unstale":"2024-01-15T18:06:02.176435+0000","last_undegraded":"2024-01-15T18:06:02.176435+0000","last_fullsized":"2024-01-15T18:06:02.176435+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:43:46.415533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.159942+0000","last_change":"2024-01-15T18:06:00.190904+0000","last_active":"2024-01-15T18:06:01.159942+0000","last_peered":"2024-01-15T18:06:01.159942+0000","last_clean":"2024-01-15T18:06:01.159942+0000","last_became_active":"2024-01-15T18:06:00.190620+0000","last_became_peered":"2024-01-15T18:06:00.190620+0000","last_unstale":"2024-01-15T18:06:01.159942+0000","last_undegraded":"2024-01-15T18:06:01.159942+0000","last_fullsized":"2024-01-15T18:06:01.159942+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:26:24.909439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.174392+0000","last_change":"2024-01-15T18:06:01.182798+0000","last_active":"2024-01-15T18:06:02.174392+0000","last_peered":"2024-01-15T18:06:02.174392+0000","last_clean":"2024-01-15T18:06:02.174392+0000","last_became_active":"2024-01-15T18:06:01.182517+0000","last_became_peered":"2024-01-15T18:06:01.182517+0000","last_unstale":"2024-01-15T18:06:02.174392+0000","last_undegraded":"2024-01-15T18:06:02.174392+0000","last_fullsized":"2024-01-15T18:06:02.174392+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:25:51.645494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.165733+0000","last_change":"2024-01-15T18:06:00.184942+0000","last_active":"2024-01-15T18:06:01.165733+0000","last_peered":"2024-01-15T18:06:01.165733+0000","last_clean":"2024-01-15T18:06:01.165733+0000","last_became_active":"2024-01-15T18:06:00.184628+0000","last_became_peered":"2024-01-15T18:06:00.184628+0000","last_unstale":"2024-01-15T18:06:01.165733+0000","last_undegraded":"2024-01-15T18:06:01.165733+0000","last_fullsized":"2024-01-15T18:06:01.165733+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:36:46.224457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.176014+0000","last_change":"2024-01-15T18:06:01.186936+0000","last_active":"2024-01-15T18:06:02.176014+0000","last_peered":"2024-01-15T18:06:02.176014+0000","last_clean":"2024-01-15T18:06:02.176014+0000","last_became_active":"2024-01-15T18:06:01.186613+0000","last_became_peered":"2024-01-15T18:06:01.186613+0000","last_unstale":"2024-01-15T18:06:02.176014+0000","last_undegraded":"2024-01-15T18:06:02.176014+0000","last_fullsized":"2024-01-15T18:06:02.176014+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:57.989668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.161217+0000","last_change":"2024-01-15T18:06:00.189934+0000","last_active":"2024-01-15T18:06:01.161217+0000","last_peered":"2024-01-15T18:06:01.161217+0000","last_clean":"2024-01-15T18:06:01.161217+0000","last_became_active":"2024-01-15T18:06:00.189306+0000","last_became_peered":"2024-01-15T18:06:00.189306+0000","last_unstale":"2024-01-15T18:06:01.161217+0000","last_undegraded":"2024-01-15T18:06:01.161217+0000","last_fullsized":"2024-01-15T18:06:01.161217+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:08:44.081810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.260274+0000","last_change":"2024-01-15T18:06:01.176311+0000","last_active":"2024-01-15T18:06:02.260274+0000","last_peered":"2024-01-15T18:06:02.260274+0000","last_clean":"2024-01-15T18:06:02.260274+0000","last_became_active":"2024-01-15T18:06:01.175647+0000","last_became_peered":"2024-01-15T18:06:01.175647+0000","last_unstale":"2024-01-15T18:06:02.260274+0000","last_undegraded":"2024-01-15T18:06:02.260274+0000","last_fullsized":"2024-01-15T18:06:02.260274+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:18:47.499122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259624+0000","last_change":"2024-01-15T18:06:00.193942+0000","last_active":"2024-01-15T18:06:02.259624+0000","last_peered":"2024-01-15T18:06:02.259624+0000","last_clean":"2024-01-15T18:06:02.259624+0000","last_became_active":"2024-01-15T18:06:00.193102+0000","last_became_peered":"2024-01-15T18:06:00.193102+0000","last_unstale":"2024-01-15T18:06:02.259624+0000","last_undegraded":"2024-01-15T18:06:02.259624+0000","last_fullsized":"2024-01-15T18:06:02.259624+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:36:35.312505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.260071+0000","last_change":"2024-01-15T18:06:01.183098+0000","last_active":"2024-01-15T18:06:02.260071+0000","last_peered":"2024-01-15T18:06:02.260071+0000","last_clean":"2024-01-15T18:06:02.260071+0000","last_became_active":"2024-01-15T18:06:01.182836+0000","last_became_peered":"2024-01-15T18:06:01.182836+0000","last_unstale":"2024-01-15T18:06:02.260071+0000","last_undegraded":"2024-01-15T18:06:02.260071+0000","last_fullsized":"2024-01-15T18:06:02.260071+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:09:25.929285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.209871+0000","last_change":"2024-01-15T18:06:00.182124+0000","last_active":"2024-01-15T18:06:01.209871+0000","last_peered":"2024-01-15T18:06:01.209871+0000","last_clean":"2024-01-15T18:06:01.209871+0000","last_became_active":"2024-01-15T18:06:00.181820+0000","last_became_peered":"2024-01-15T18:06:00.181820+0000","last_unstale":"2024-01-15T18:06:01.209871+0000","last_undegraded":"2024-01-15T18:06:01.209871+0000","last_fullsized":"2024-01-15T18:06:01.209871+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:15:39.265433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.175925+0000","last_change":"2024-01-15T18:06:01.189648+0000","last_active":"2024-01-15T18:06:02.175925+0000","last_peered":"2024-01-15T18:06:02.175925+0000","last_clean":"2024-01-15T18:06:02.175925+0000","last_became_active":"2024-01-15T18:06:01.189124+0000","last_became_peered":"2024-01-15T18:06:01.189124+0000","last_unstale":"2024-01-15T18:06:02.175925+0000","last_undegraded":"2024-01-15T18:06:02.175925+0000","last_fullsized":"2024-01-15T18:06:02.175925+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:25:05.480462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.215933+0000","last_change":"2024-01-15T18:06:00.191070+0000","last_active":"2024-01-15T18:06:01.215933+0000","last_peered":"2024-01-15T18:06:01.215933+0000","last_clean":"2024-01-15T18:06:01.215933+0000","last_became_active":"2024-01-15T18:06:00.190787+0000","last_became_peered":"2024-01-15T18:06:00.190787+0000","last_unstale":"2024-01-15T18:06:01.215933+0000","last_undegraded":"2024-01-15T18:06:01.215933+0000","last_fullsized":"2024-01-15T18:06:01.215933+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:15:36.318765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.174392+0000","last_change":"2024-01-15T18:06:01.185323+0000","last_active":"2024-01-15T18:06:02.174392+0000","last_peered":"2024-01-15T18:06:02.174392+0000","last_clean":"2024-01-15T18:06:02.174392+0000","last_became_active":"2024-01-15T18:06:01.184980+0000","last_became_peered":"2024-01-15T18:06:01.184980+0000","last_unstale":"2024-01-15T18:06:02.174392+0000","last_undegraded":"2024-01-15T18:06:02.174392+0000","last_fullsized":"2024-01-15T18:06:02.174392+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:52:51.980421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.207074+0000","last_change":"2024-01-15T18:06:00.180527+0000","last_active":"2024-01-15T18:06:01.207074+0000","last_peered":"2024-01-15T18:06:01.207074+0000","last_clean":"2024-01-15T18:06:01.207074+0000","last_became_active":"2024-01-15T18:06:00.180161+0000","last_became_peered":"2024-01-15T18:06:00.180161+0000","last_unstale":"2024-01-15T18:06:01.207074+0000","last_undegraded":"2024-01-15T18:06:01.207074+0000","last_fullsized":"2024-01-15T18:06:01.207074+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:36:16.189750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.260221+0000","last_change":"2024-01-15T18:06:01.195303+0000","last_active":"2024-01-15T18:06:02.260221+0000","last_peered":"2024-01-15T18:06:02.260221+0000","last_clean":"2024-01-15T18:06:02.260221+0000","last_became_active":"2024-01-15T18:06:01.195043+0000","last_became_peered":"2024-01-15T18:06:01.195043+0000","last_unstale":"2024-01-15T18:06:02.260221+0000","last_undegraded":"2024-01-15T18:06:02.260221+0000","last_fullsized":"2024-01-15T18:06:02.260221+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:23:07.800413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.168995+0000","last_change":"2024-01-15T18:06:00.184666+0000","last_active":"2024-01-15T18:06:01.168995+0000","last_peered":"2024-01-15T18:06:01.168995+0000","last_clean":"2024-01-15T18:06:01.168995+0000","last_became_active":"2024-01-15T18:06:00.184339+0000","last_became_peered":"2024-01-15T18:06:00.184339+0000","last_unstale":"2024-01-15T18:06:01.168995+0000","last_undegraded":"2024-01-15T18:06:01.168995+0000","last_fullsized":"2024-01-15T18:06:01.168995+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:36:16.189750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258684+0000","last_change":"2024-01-15T18:06:01.192289+0000","last_active":"2024-01-15T18:06:02.258684+0000","last_peered":"2024-01-15T18:06:02.258684+0000","last_clean":"2024-01-15T18:06:02.258684+0000","last_became_active":"2024-01-15T18:06:01.191604+0000","last_became_peered":"2024-01-15T18:06:01.191604+0000","last_unstale":"2024-01-15T18:06:02.258684+0000","last_undegraded":"2024-01-15T18:06:02.258684+0000","last_fullsized":"2024-01-15T18:06:02.258684+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:41.300029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.160464+0000","last_change":"2024-01-15T18:06:00.189891+0000","last_active":"2024-01-15T18:06:01.160464+0000","last_peered":"2024-01-15T18:06:01.160464+0000","last_clean":"2024-01-15T18:06:01.160464+0000","last_became_active":"2024-01-15T18:06:00.189621+0000","last_became_peered":"2024-01-15T18:06:00.189621+0000","last_unstale":"2024-01-15T18:06:01.160464+0000","last_undegraded":"2024-01-15T18:06:01.160464+0000","last_fullsized":"2024-01-15T18:06:01.160464+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:44:09.010095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258960+0000","last_change":"2024-01-15T18:06:01.182869+0000","last_active":"2024-01-15T18:06:02.258960+0000","last_peered":"2024-01-15T18:06:02.258960+0000","last_clean":"2024-01-15T18:06:02.258960+0000","last_became_active":"2024-01-15T18:06:01.182557+0000","last_became_peered":"2024-01-15T18:06:01.182557+0000","last_unstale":"2024-01-15T18:06:02.258960+0000","last_undegraded":"2024-01-15T18:06:02.258960+0000","last_fullsized":"2024-01-15T18:06:02.258960+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:27:04.736218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.213575+0000","last_change":"2024-01-15T18:06:00.194127+0000","last_active":"2024-01-15T18:06:01.213575+0000","last_peered":"2024-01-15T18:06:01.213575+0000","last_clean":"2024-01-15T18:06:01.213575+0000","last_became_active":"2024-01-15T18:06:00.193851+0000","last_became_peered":"2024-01-15T18:06:00.193851+0000","last_unstale":"2024-01-15T18:06:01.213575+0000","last_undegraded":"2024-01-15T18:06:01.213575+0000","last_fullsized":"2024-01-15T18:06:01.213575+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:36:46.224457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.176657+0000","last_change":"2024-01-15T18:06:01.183977+0000","last_active":"2024-01-15T18:06:02.176657+0000","last_peered":"2024-01-15T18:06:02.176657+0000","last_clean":"2024-01-15T18:06:02.176657+0000","last_became_active":"2024-01-15T18:06:01.183643+0000","last_became_peered":"2024-01-15T18:06:01.183643+0000","last_unstale":"2024-01-15T18:06:02.176657+0000","last_undegraded":"2024-01-15T18:06:02.176657+0000","last_fullsized":"2024-01-15T18:06:02.176657+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:44:10.016963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.209611+0000","last_change":"2024-01-15T18:06:00.195128+0000","last_active":"2024-01-15T18:06:01.209611+0000","last_peered":"2024-01-15T18:06:01.209611+0000","last_clean":"2024-01-15T18:06:01.209611+0000","last_became_active":"2024-01-15T18:06:00.194872+0000","last_became_peered":"2024-01-15T18:06:00.194872+0000","last_unstale":"2024-01-15T18:06:01.209611+0000","last_undegraded":"2024-01-15T18:06:01.209611+0000","last_fullsized":"2024-01-15T18:06:01.209611+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:15:36.318765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.176469+0000","last_change":"2024-01-15T18:06:01.184378+0000","last_active":"2024-01-15T18:06:02.176469+0000","last_peered":"2024-01-15T18:06:02.176469+0000","last_clean":"2024-01-15T18:06:02.176469+0000","last_became_active":"2024-01-15T18:06:01.184115+0000","last_became_peered":"2024-01-15T18:06:01.184115+0000","last_unstale":"2024-01-15T18:06:02.176469+0000","last_undegraded":"2024-01-15T18:06:02.176469+0000","last_fullsized":"2024-01-15T18:06:02.176469+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:35:51.820223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.204985+0000","last_change":"2024-01-15T18:06:00.181570+0000","last_active":"2024-01-15T18:06:01.204985+0000","last_peered":"2024-01-15T18:06:01.204985+0000","last_clean":"2024-01-15T18:06:01.204985+0000","last_became_active":"2024-01-15T18:06:00.180958+0000","last_became_peered":"2024-01-15T18:06:00.180958+0000","last_unstale":"2024-01-15T18:06:01.204985+0000","last_undegraded":"2024-01-15T18:06:01.204985+0000","last_fullsized":"2024-01-15T18:06:01.204985+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:40:56.123776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.174633+0000","last_change":"2024-01-15T18:06:01.174681+0000","last_active":"2024-01-15T18:06:02.174633+0000","last_peered":"2024-01-15T18:06:02.174633+0000","last_clean":"2024-01-15T18:06:02.174633+0000","last_became_active":"2024-01-15T18:06:01.174387+0000","last_became_peered":"2024-01-15T18:06:01.174387+0000","last_unstale":"2024-01-15T18:06:02.174633+0000","last_undegraded":"2024-01-15T18:06:02.174633+0000","last_fullsized":"2024-01-15T18:06:02.174633+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:15.311517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.207366+0000","last_change":"2024-01-15T18:06:00.176281+0000","last_active":"2024-01-15T18:06:01.207366+0000","last_peered":"2024-01-15T18:06:01.207366+0000","last_clean":"2024-01-15T18:06:01.207366+0000","last_became_active":"2024-01-15T18:06:00.174503+0000","last_became_peered":"2024-01-15T18:06:00.174503+0000","last_unstale":"2024-01-15T18:06:01.207366+0000","last_undegraded":"2024-01-15T18:06:01.207366+0000","last_fullsized":"2024-01-15T18:06:01.207366+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:31:04.425248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259554+0000","last_change":"2024-01-15T18:06:01.181443+0000","last_active":"2024-01-15T18:06:02.259554+0000","last_peered":"2024-01-15T18:06:02.259554+0000","last_clean":"2024-01-15T18:06:02.259554+0000","last_became_active":"2024-01-15T18:06:01.181122+0000","last_became_peered":"2024-01-15T18:06:01.181122+0000","last_unstale":"2024-01-15T18:06:02.259554+0000","last_undegraded":"2024-01-15T18:06:02.259554+0000","last_fullsized":"2024-01-15T18:06:02.259554+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:51:47.039923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.163735+0000","last_change":"2024-01-15T18:06:00.190091+0000","last_active":"2024-01-15T18:06:01.163735+0000","last_peered":"2024-01-15T18:06:01.163735+0000","last_clean":"2024-01-15T18:06:01.163735+0000","last_became_active":"2024-01-15T18:06:00.189619+0000","last_became_peered":"2024-01-15T18:06:00.189619+0000","last_unstale":"2024-01-15T18:06:01.163735+0000","last_undegraded":"2024-01-15T18:06:01.163735+0000","last_fullsized":"2024-01-15T18:06:01.163735+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:59:02.797271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259156+0000","last_change":"2024-01-15T18:06:01.172767+0000","last_active":"2024-01-15T18:06:02.259156+0000","last_peered":"2024-01-15T18:06:02.259156+0000","last_clean":"2024-01-15T18:06:02.259156+0000","last_became_active":"2024-01-15T18:06:01.172483+0000","last_became_peered":"2024-01-15T18:06:01.172483+0000","last_unstale":"2024-01-15T18:06:02.259156+0000","last_undegraded":"2024-01-15T18:06:02.259156+0000","last_fullsized":"2024-01-15T18:06:02.259156+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:31:05.432116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259204+0000","last_change":"2024-01-15T18:06:00.190248+0000","last_active":"2024-01-15T18:06:02.259204+0000","last_peered":"2024-01-15T18:06:02.259204+0000","last_clean":"2024-01-15T18:06:02.259204+0000","last_became_active":"2024-01-15T18:06:00.189972+0000","last_became_peered":"2024-01-15T18:06:00.189972+0000","last_unstale":"2024-01-15T18:06:02.259204+0000","last_undegraded":"2024-01-15T18:06:02.259204+0000","last_fullsized":"2024-01-15T18:06:02.259204+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:46:26.889479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259484+0000","last_change":"2024-01-15T18:06:01.195352+0000","last_active":"2024-01-15T18:06:02.259484+0000","last_peered":"2024-01-15T18:06:02.259484+0000","last_clean":"2024-01-15T18:06:02.259484+0000","last_became_active":"2024-01-15T18:06:01.195053+0000","last_became_peered":"2024-01-15T18:06:01.195053+0000","last_unstale":"2024-01-15T18:06:02.259484+0000","last_undegraded":"2024-01-15T18:06:02.259484+0000","last_fullsized":"2024-01-15T18:06:02.259484+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:26:10.659136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.176825+0000","last_change":"2024-01-15T18:06:00.190356+0000","last_active":"2024-01-15T18:06:01.176825+0000","last_peered":"2024-01-15T18:06:01.176825+0000","last_clean":"2024-01-15T18:06:01.176825+0000","last_became_active":"2024-01-15T18:06:00.189527+0000","last_became_peered":"2024-01-15T18:06:00.189527+0000","last_unstale":"2024-01-15T18:06:01.176825+0000","last_undegraded":"2024-01-15T18:06:01.176825+0000","last_fullsized":"2024-01-15T18:06:01.176825+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:36:46.224457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258955+0000","last_change":"2024-01-15T18:06:01.189227+0000","last_active":"2024-01-15T18:06:02.258955+0000","last_peered":"2024-01-15T18:06:02.258955+0000","last_clean":"2024-01-15T18:06:02.258955+0000","last_became_active":"2024-01-15T18:06:01.188599+0000","last_became_peered":"2024-01-15T18:06:01.188599+0000","last_unstale":"2024-01-15T18:06:02.258955+0000","last_undegraded":"2024-01-15T18:06:02.258955+0000","last_fullsized":"2024-01-15T18:06:02.258955+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:57.743836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.163863+0000","last_change":"2024-01-15T18:06:00.179035+0000","last_active":"2024-01-15T18:06:01.163863+0000","last_peered":"2024-01-15T18:06:01.163863+0000","last_clean":"2024-01-15T18:06:01.163863+0000","last_became_active":"2024-01-15T18:06:00.178771+0000","last_became_peered":"2024-01-15T18:06:00.178771+0000","last_unstale":"2024-01-15T18:06:01.163863+0000","last_undegraded":"2024-01-15T18:06:01.163863+0000","last_fullsized":"2024-01-15T18:06:01.163863+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:12.325308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258926+0000","last_change":"2024-01-15T18:06:01.193458+0000","last_active":"2024-01-15T18:06:02.258926+0000","last_peered":"2024-01-15T18:06:02.258926+0000","last_clean":"2024-01-15T18:06:02.258926+0000","last_became_active":"2024-01-15T18:06:01.191417+0000","last_became_peered":"2024-01-15T18:06:01.191417+0000","last_unstale":"2024-01-15T18:06:02.258926+0000","last_undegraded":"2024-01-15T18:06:02.258926+0000","last_fullsized":"2024-01-15T18:06:02.258926+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:20:58.455307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.171988+0000","last_change":"2024-01-15T18:06:00.182290+0000","last_active":"2024-01-15T18:06:01.171988+0000","last_peered":"2024-01-15T18:06:01.171988+0000","last_clean":"2024-01-15T18:06:01.171988+0000","last_became_active":"2024-01-15T18:06:00.182016+0000","last_became_peered":"2024-01-15T18:06:00.182016+0000","last_unstale":"2024-01-15T18:06:01.171988+0000","last_undegraded":"2024-01-15T18:06:01.171988+0000","last_fullsized":"2024-01-15T18:06:01.171988+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:23:13.587394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259753+0000","last_change":"2024-01-15T18:06:01.184496+0000","last_active":"2024-01-15T18:06:02.259753+0000","last_peered":"2024-01-15T18:06:02.259753+0000","last_clean":"2024-01-15T18:06:02.259753+0000","last_became_active":"2024-01-15T18:06:01.184224+0000","last_became_peered":"2024-01-15T18:06:01.184224+0000","last_unstale":"2024-01-15T18:06:02.259753+0000","last_undegraded":"2024-01-15T18:06:02.259753+0000","last_fullsized":"2024-01-15T18:06:02.259753+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:36:47.231325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.208763+0000","last_change":"2024-01-15T18:06:00.176082+0000","last_active":"2024-01-15T18:06:01.208763+0000","last_peered":"2024-01-15T18:06:01.208763+0000","last_clean":"2024-01-15T18:06:01.208763+0000","last_became_active":"2024-01-15T18:06:00.175734+0000","last_became_peered":"2024-01-15T18:06:00.175734+0000","last_unstale":"2024-01-15T18:06:01.208763+0000","last_undegraded":"2024-01-15T18:06:01.208763+0000","last_fullsized":"2024-01-15T18:06:01.208763+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:43:45.408666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.176424+0000","last_change":"2024-01-15T18:06:01.187160+0000","last_active":"2024-01-15T18:06:02.176424+0000","last_peered":"2024-01-15T18:06:02.176424+0000","last_clean":"2024-01-15T18:06:02.176424+0000","last_became_active":"2024-01-15T18:06:01.186806+0000","last_became_peered":"2024-01-15T18:06:01.186806+0000","last_unstale":"2024-01-15T18:06:02.176424+0000","last_undegraded":"2024-01-15T18:06:02.176424+0000","last_fullsized":"2024-01-15T18:06:02.176424+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:37:32.291136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:03.273111+0000","last_change":"2024-01-15T18:06:00.183191+0000","last_active":"2024-01-15T18:06:03.273111+0000","last_peered":"2024-01-15T18:06:03.273111+0000","last_clean":"2024-01-15T18:06:03.273111+0000","last_became_active":"2024-01-15T18:06:00.182692+0000","last_became_peered":"2024-01-15T18:06:00.182692+0000","last_unstale":"2024-01-15T18:06:03.273111+0000","last_undegraded":"2024-01-15T18:06:03.273111+0000","last_fullsized":"2024-01-15T18:06:03.273111+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:41:12.249000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259096+0000","last_change":"2024-01-15T18:06:01.194408+0000","last_active":"2024-01-15T18:06:02.259096+0000","last_peered":"2024-01-15T18:06:02.259096+0000","last_clean":"2024-01-15T18:06:02.259096+0000","last_became_active":"2024-01-15T18:06:01.194050+0000","last_became_peered":"2024-01-15T18:06:01.194050+0000","last_unstale":"2024-01-15T18:06:02.259096+0000","last_undegraded":"2024-01-15T18:06:02.259096+0000","last_fullsized":"2024-01-15T18:06:02.259096+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:55.114370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.169108+0000","last_change":"2024-01-15T18:06:00.181270+0000","last_active":"2024-01-15T18:06:01.169108+0000","last_peered":"2024-01-15T18:06:01.169108+0000","last_clean":"2024-01-15T18:06:01.169108+0000","last_became_active":"2024-01-15T18:06:00.180946+0000","last_became_peered":"2024-01-15T18:06:00.180946+0000","last_unstale":"2024-01-15T18:06:01.169108+0000","last_undegraded":"2024-01-15T18:06:01.169108+0000","last_fullsized":"2024-01-15T18:06:01.169108+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:55:19.741579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.258840+0000","last_change":"2024-01-15T18:06:01.187228+0000","last_active":"2024-01-15T18:06:02.258840+0000","last_peered":"2024-01-15T18:06:02.258840+0000","last_clean":"2024-01-15T18:06:02.258840+0000","last_became_active":"2024-01-15T18:06:01.187018+0000","last_became_peered":"2024-01-15T18:06:01.187018+0000","last_unstale":"2024-01-15T18:06:02.258840+0000","last_undegraded":"2024-01-15T18:06:02.258840+0000","last_fullsized":"2024-01-15T18:06:02.258840+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:36:17.196617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259363+0000","last_change":"2024-01-15T18:06:00.188148+0000","last_active":"2024-01-15T18:06:02.259363+0000","last_peered":"2024-01-15T18:06:02.259363+0000","last_clean":"2024-01-15T18:06:02.259363+0000","last_became_active":"2024-01-15T18:06:00.187598+0000","last_became_peered":"2024-01-15T18:06:00.187598+0000","last_unstale":"2024-01-15T18:06:02.259363+0000","last_undegraded":"2024-01-15T18:06:02.259363+0000","last_fullsized":"2024-01-15T18:06:02.259363+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:51:01.940485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259121+0000","last_change":"2024-01-15T18:06:01.195224+0000","last_active":"2024-01-15T18:06:02.259121+0000","last_peered":"2024-01-15T18:06:02.259121+0000","last_clean":"2024-01-15T18:06:02.259121+0000","last_became_active":"2024-01-15T18:06:01.194852+0000","last_became_peered":"2024-01-15T18:06:01.194852+0000","last_unstale":"2024-01-15T18:06:02.259121+0000","last_undegraded":"2024-01-15T18:06:02.259121+0000","last_fullsized":"2024-01-15T18:06:02.259121+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:29.256704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.169150+0000","last_change":"2024-01-15T18:06:00.176729+0000","last_active":"2024-01-15T18:06:01.169150+0000","last_peered":"2024-01-15T18:06:01.169150+0000","last_clean":"2024-01-15T18:06:01.169150+0000","last_became_active":"2024-01-15T18:06:00.175928+0000","last_became_peered":"2024-01-15T18:06:00.175928+0000","last_unstale":"2024-01-15T18:06:01.169150+0000","last_undegraded":"2024-01-15T18:06:01.169150+0000","last_fullsized":"2024-01-15T18:06:01.169150+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:27:03.729350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259269+0000","last_change":"2024-01-15T18:06:01.176150+0000","last_active":"2024-01-15T18:06:02.259269+0000","last_peered":"2024-01-15T18:06:02.259269+0000","last_clean":"2024-01-15T18:06:02.259269+0000","last_became_active":"2024-01-15T18:06:01.175559+0000","last_became_peered":"2024-01-15T18:06:01.175559+0000","last_unstale":"2024-01-15T18:06:02.259269+0000","last_undegraded":"2024-01-15T18:06:02.259269+0000","last_fullsized":"2024-01-15T18:06:02.259269+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:41.300029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.163997+0000","last_change":"2024-01-15T18:06:00.178477+0000","last_active":"2024-01-15T18:06:01.163997+0000","last_peered":"2024-01-15T18:06:01.163997+0000","last_clean":"2024-01-15T18:06:01.163997+0000","last_became_active":"2024-01-15T18:06:00.176994+0000","last_became_peered":"2024-01-15T18:06:00.176994+0000","last_unstale":"2024-01-15T18:06:01.163997+0000","last_undegraded":"2024-01-15T18:06:01.163997+0000","last_fullsized":"2024-01-15T18:06:01.163997+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:56.736968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.176381+0000","last_change":"2024-01-15T18:06:01.173689+0000","last_active":"2024-01-15T18:06:02.176381+0000","last_peered":"2024-01-15T18:06:02.176381+0000","last_clean":"2024-01-15T18:06:02.176381+0000","last_became_active":"2024-01-15T18:06:01.172390+0000","last_became_peered":"2024-01-15T18:06:01.172390+0000","last_unstale":"2024-01-15T18:06:02.176381+0000","last_undegraded":"2024-01-15T18:06:02.176381+0000","last_fullsized":"2024-01-15T18:06:02.176381+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:15:37.325633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.160483+0000","last_change":"2024-01-15T18:06:00.184467+0000","last_active":"2024-01-15T18:06:01.160483+0000","last_peered":"2024-01-15T18:06:01.160483+0000","last_clean":"2024-01-15T18:06:01.160483+0000","last_became_active":"2024-01-15T18:06:00.184044+0000","last_became_peered":"2024-01-15T18:06:00.184044+0000","last_unstale":"2024-01-15T18:06:01.160483+0000","last_undegraded":"2024-01-15T18:06:01.160483+0000","last_fullsized":"2024-01-15T18:06:01.160483+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:52:50.973554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T18:06:02.259499+0000","last_change":"2024-01-15T18:06:01.190807+0000","last_active":"2024-01-15T18:06:02.259499+0000","last_peered":"2024-01-15T18:06:02.259499+0000","last_clean":"2024-01-15T18:06:02.259499+0000","last_became_active":"2024-01-15T18:06:01.188659+0000","last_became_peered":"2024-01-15T18:06:01.188659+0000","last_unstale":"2024-01-15T18:06:02.259499+0000","last_undegraded":"2024-01-15T18:06:02.259499+0000","last_fullsized":"2024-01-15T18:06:02.259499+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:06:00.152476+0000","last_clean_scrub_stamp":"2024-01-15T18:06:00.152476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:58:04.778601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.176553+0000","last_change":"2024-01-15T18:06:00.190042+0000","last_active":"2024-01-15T18:06:01.176553+0000","last_peered":"2024-01-15T18:06:01.176553+0000","last_clean":"2024-01-15T18:06:01.176553+0000","last_became_active":"2024-01-15T18:06:00.189749+0000","last_became_peered":"2024-01-15T18:06:00.189749+0000","last_unstale":"2024-01-15T18:06:01.176553+0000","last_undegraded":"2024-01-15T18:06:01.176553+0000","last_fullsized":"2024-01-15T18:06:01.176553+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:59.145608+0000","last_clean_scrub_stamp":"2024-01-15T18:05:59.145608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:38:07.317901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":175,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T18:06:01.181400+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:06:01.181400+0000","last_peered":"2024-01-15T18:06:01.181400+0000","last_clean":"2024-01-15T18:06:01.181400+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:06:01.181400+0000","last_undegraded":"2024-01-15T18:06:01.181400+0000","last_fullsized":"2024-01-15T18:06:01.181400+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509459,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27980,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343860,"statfs":{"total":96636764160,"available":96608112640,"internally_reserved":0,"allocated":974848,"data_stored":738449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.77300000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.85199999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.747},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.874}]},{"osd":3,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.72499999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.502},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.58999999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.61099999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.52500000000000002}]}]},{"osd":6,"up_from":15,"seq":64424509459,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":385024,"data_stored":148607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.61899999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":2.4220000000000002,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.391,"5min":1.391,"15min":1.391},"last":0.73399999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":2.8700000000000001,"5min":2.8700000000000001,"15min":2.8700000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.857,"5min":1.857,"15min":1.857},"last":0.67400000000000004}]},{"osd":3,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":2.012,"5min":2.012,"15min":2.012},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.532,"5min":2.532,"15min":2.532},"last":0.64000000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.2430000000000001,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.38100000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.443},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.65600000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:05:49 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.503}]}]},{"osd":4,"up_from":15,"seq":64424509459,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":385024,"data_stored":148095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.67100000000000004}]},{"osd":1,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.68999999999999995}]},{"osd":2,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.81100000000000005}]},{"osd":3,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.751},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.71099999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.498},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.57299999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.72799999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.86499999999999999}]}]},{"osd":3,"up_from":15,"seq":64424509458,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":149625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.46899999999999997}]},{"osd":1,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.442},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.505}]},{"osd":2,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.45300000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.78800000000000003}]},{"osd":5,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.75900000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.67300000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.83699999999999997}]}]},{"osd":2,"up_from":15,"seq":64424509458,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":148061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.34999999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.443}]},{"osd":3,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.46200000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":1.109},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.466,"5min":1.466,"15min":1.466},"last":0.94299999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.89200000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.516,"5min":1.516,"15min":1.516},"last":1.0489999999999999},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.7849999999999999,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":1.0089999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.6279999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":1.1299999999999999},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":1.1519999999999999}]}]},{"osd":1,"up_from":15,"seq":64424509460,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":149625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.50700000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":4.2450000000000001,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.59999999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.871,"5min":1.871,"15min":1.871},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.917,"5min":1.917,"15min":1.917},"last":0.46000000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.81899999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.79500000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.216,"5min":1.216,"15min":1.216},"last":0.96099999999999997}]},{"osd":7,"last update":"Mon Jan 15 18:05:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.84299999999999997}]}]},{"osd":5,"up_from":14,"seq":60129542164,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":385024,"data_stored":148663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":2.802,"5min":2.802,"15min":2.802},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.83099999999999996}]},{"osd":1,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.51100000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":0.70699999999999996}]},{"osd":3,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":2.3849999999999998,"5min":2.3849999999999998,"15min":2.3849999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":2.1970000000000001,"5min":2.1970000000000001,"15min":2.1970000000000001},"last":0.77500000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.746}]},{"osd":6,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.89500000000000002}]},{"osd":7,"last update":"Mon Jan 15 18:05:47 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.80500000000000005}]}]},{"osd":0,"up_from":14,"seq":60129542164,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":738359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.58199999999999996}]},{"osd":2,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.40000000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.325,"5min":1.325,"15min":1.325},"last":0.54400000000000004}]},{"osd":4,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.86399999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:05:43 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.9079999999999999,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":0.82799999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.764,"5min":1.764,"15min":1.764},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.806,"5min":1.806,"15min":1.806},"last":0.84699999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.722,"5min":1.722,"15min":1.722},"last":0.78500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:06:10.238 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-15T18:06:10.238 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-15T18:06:10.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.418+0000 7f3a48400640 1 Processor -- start 2024-01-15T18:06:10.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.418+0000 7f3a48400640 1 -- start start 2024-01-15T18:06:10.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.418+0000 7f3a48400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f3a400d1a40 0x7f3a400cba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:10.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.418+0000 7f3a48400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3a400b0650 0x7f3a400b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:10.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.418+0000 7f3a48400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a400cbfa0 con 0x7f3a400cf450 2024-01-15T18:06:10.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.418+0000 7f3a48400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a400cefa0 con 0x7f3a400b0650 2024-01-15T18:06:10.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.418+0000 7f3a48400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a400ce240 con 0x7f3a400d1a40 2024-01-15T18:06:10.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.422+0000 7f3a46400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3a400b0650 0x7f3a400b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:10.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.422+0000 7f3a46400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3a400b0650 0x7f3a400b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56674/0 (socket says 172.21.15.27:56674) 2024-01-15T18:06:10.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.422+0000 7f3a46400640 1 -- 172.21.15.27:0/3867260697 learned_addr learned my addr 172.21.15.27:0/3867260697 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:10.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.422+0000 7f3a45000640 1 --2- 172.21.15.27:0/3867260697 >> v2:172.21.15.93:3300/0 conn(0x7f3a400d1a40 0x7f3a400cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:10.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.422+0000 7f3a46400640 1 -- 172.21.15.27:0/3867260697 >> v2:172.21.15.93:3300/0 conn(0x7f3a400d1a40 msgr2=0x7f3a400cba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:10.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.422+0000 7f3a46400640 1 --2- 172.21.15.27:0/3867260697 >> v2:172.21.15.93:3300/0 conn(0x7f3a400d1a40 0x7f3a400cba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:10.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.422+0000 7f3a46400640 1 -- 172.21.15.27:0/3867260697 >> v1:172.21.15.93:6789/0 conn(0x7f3a400cf450 legacy=0x7f3a400cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:10.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.422+0000 7f3a46400640 1 -- 172.21.15.27:0/3867260697 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a400ccd70 con 0x7f3a400b0650 2024-01-15T18:06:10.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.422+0000 7f3a46400640 1 --2- 172.21.15.27:0/3867260697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a400b0650 0x7f3a400b0a80 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3a4009aa00 tx=0x7f3a3801c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34876d167bf8c52f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:10.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.422+0000 7f3a3fe00640 1 -- 172.21.15.27:0/3867260697 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a380a5690 con 0x7f3a400b0650 2024-01-15T18:06:10.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.422+0000 7f3a3fe00640 1 -- 172.21.15.27:0/3867260697 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a380a57f0 con 0x7f3a400b0650 2024-01-15T18:06:10.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.422+0000 7f3a3fe00640 1 -- 172.21.15.27:0/3867260697 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a380a5b80 con 0x7f3a400b0650 2024-01-15T18:06:10.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.426+0000 7f3a48400640 1 -- 172.21.15.27:0/3867260697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a400b0650 msgr2=0x7f3a400b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:10.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.426+0000 7f3a48400640 1 --2- 172.21.15.27:0/3867260697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a400b0650 0x7f3a400b0a80 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3a4009aa00 tx=0x7f3a3801c9d0 comp rx=0 tx=0).stop 2024-01-15T18:06:10.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.426+0000 7f3a48400640 1 -- 172.21.15.27:0/3867260697 shutdown_connections 2024-01-15T18:06:10.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.426+0000 7f3a48400640 1 --2- 172.21.15.27:0/3867260697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a400b0650 0x7f3a400b0a80 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:10.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.426+0000 7f3a48400640 1 --2- 172.21.15.27:0/3867260697 >> v2:172.21.15.93:3300/0 conn(0x7f3a400d1a40 0x7f3a400cba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:10.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.426+0000 7f3a48400640 1 -- 172.21.15.27:0/3867260697 >> 172.21.15.27:0/3867260697 conn(0x7f3a400d0e70 msgr2=0x7f3a400d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:10.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.426+0000 7f3a48400640 1 -- 172.21.15.27:0/3867260697 shutdown_connections 2024-01-15T18:06:10.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.426+0000 7f3a48400640 1 -- 172.21.15.27:0/3867260697 wait complete. 2024-01-15T18:06:10.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.426+0000 7f3a48400640 1 Processor -- start 2024-01-15T18:06:10.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.426+0000 7f3a48400640 1 -- start start 2024-01-15T18:06:10.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.426+0000 7f3a48400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a400b0650 0x7f3a400caa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:10.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.426+0000 7f3a48400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a400cf450 0x7f3a40156570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:10.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.426+0000 7f3a48400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3a400d1a40 0x7f3a400c7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:10.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.426+0000 7f3a48400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a401595c0 con 0x7f3a400b0650 2024-01-15T18:06:10.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.426+0000 7f3a48400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a400c6e20 con 0x7f3a400d1a40 2024-01-15T18:06:10.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.426+0000 7f3a48400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a400c7100 con 0x7f3a400cf450 2024-01-15T18:06:10.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.426+0000 7f3a45a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a400b0650 0x7f3a400caa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:10.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.426+0000 7f3a45000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a400cf450 0x7f3a40156570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:10.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.426+0000 7f3a45a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a400b0650 0x7f3a400caa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56680/0 (socket says 172.21.15.27:56680) 2024-01-15T18:06:10.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.426+0000 7f3a45a00640 1 -- 172.21.15.27:0/1507657657 learned_addr learned my addr 172.21.15.27:0/1507657657 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:10.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.426+0000 7f3a46400640 1 --2- 172.21.15.27:0/1507657657 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3a400d1a40 0x7f3a400c7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:10.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.430+0000 7f3a45a00640 1 -- 172.21.15.27:0/1507657657 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a400cf450 msgr2=0x7f3a40156570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:10.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.430+0000 7f3a45a00640 1 --2- 172.21.15.27:0/1507657657 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a400cf450 0x7f3a40156570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:10.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.430+0000 7f3a45a00640 1 -- 172.21.15.27:0/1507657657 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3a400d1a40 msgr2=0x7f3a400c7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:10.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.430+0000 7f3a45a00640 1 --2- 172.21.15.27:0/1507657657 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3a400d1a40 0x7f3a400c7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:10.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.430+0000 7f3a45a00640 1 -- 172.21.15.27:0/1507657657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a38009b30 con 0x7f3a400b0650 2024-01-15T18:06:10.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.430+0000 7f3a45a00640 1 --2- 172.21.15.27:0/1507657657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a400b0650 0x7f3a400caa30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000c3c0 tx=0x7f3a3000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.430+0000 7f3a3d600640 1 -- 172.21.15.27:0/1507657657 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a300052d0 con 0x7f3a400b0650 2024-01-15T18:06:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.430+0000 7f3a3d600640 1 -- 172.21.15.27:0/1507657657 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a30005430 con 0x7f3a400b0650 2024-01-15T18:06:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.430+0000 7f3a3d600640 1 -- 172.21.15.27:0/1507657657 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a30003840 con 0x7f3a400b0650 2024-01-15T18:06:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.430+0000 7f3a48400640 1 -- 172.21.15.27:0/1507657657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a40156eb0 con 0x7f3a400b0650 2024-01-15T18:06:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.430+0000 7f3a48400640 1 -- 172.21.15.27:0/1507657657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a40156ab0 con 0x7f3a400b0650 2024-01-15T18:06:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.430+0000 7f3a48400640 1 -- 172.21.15.27:0/1507657657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a401573d0 con 0x7f3a400b0650 2024-01-15T18:06:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.430+0000 7f3a3d600640 1 -- 172.21.15.27:0/1507657657 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f3a300039a0 con 0x7f3a400b0650 2024-01-15T18:06:10.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.430+0000 7f3a3d600640 1 --2- 172.21.15.27:0/1507657657 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a18066920 0x7f3a18068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:10.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.430+0000 7f3a3d600640 1 -- 172.21.15.27:0/1507657657 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3a300838b0 con 0x7f3a400b0650 2024-01-15T18:06:10.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.434+0000 7f3a45000640 1 --2- 172.21.15.27:0/1507657657 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a18066920 0x7f3a18068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:10.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.434+0000 7f3a45000640 1 --2- 172.21.15.27:0/1507657657 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a18066920 0x7f3a18068de0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3a400ca9c0 tx=0x7f3a28009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:10.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.434+0000 7f3a3d600640 1 -- 172.21.15.27:0/1507657657 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3a401573d0 con 0x7f3a400b0650 2024-01-15T18:06:10.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.650+0000 7f3a48400640 1 -- 172.21.15.27:0/1507657657 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f3a401573d0 con 0x7f3a400b0650 2024-01-15T18:06:10.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.654+0000 7f3a3d600640 1 -- 172.21.15.27:0/1507657657 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v9) v1 ==== 190+0+147 (secure 0 0 0) 0x7f3a300170b0 con 0x7f3a400b0650 2024-01-15T18:06:10.659 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-01-15T18:06:10.659 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOc6VlH+DCLxAA1sKVut+3WjA/B+aYJ6/Jeg== 2024-01-15T18:06:10.659 INFO:teuthology.orchestra.run.smithi027.stdout: caps mds = "allow" 2024-01-15T18:06:10.659 INFO:teuthology.orchestra.run.smithi027.stdout: caps mon = "allow r" 2024-01-15T18:06:10.659 INFO:teuthology.orchestra.run.smithi027.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-15T18:06:10.659 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-01-15T18:06:10.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.654+0000 7f3a48400640 1 -- 172.21.15.27:0/1507657657 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a18066920 msgr2=0x7f3a18068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:10.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.654+0000 7f3a48400640 1 --2- 172.21.15.27:0/1507657657 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a18066920 0x7f3a18068de0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3a400ca9c0 tx=0x7f3a28009190 comp rx=0 tx=0).stop 2024-01-15T18:06:10.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.654+0000 7f3a48400640 1 -- 172.21.15.27:0/1507657657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a400b0650 msgr2=0x7f3a400caa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:10.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.654+0000 7f3a48400640 1 --2- 172.21.15.27:0/1507657657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a400b0650 0x7f3a400caa30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000c3c0 tx=0x7f3a3000c880 comp rx=0 tx=0).stop 2024-01-15T18:06:10.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.654+0000 7f3a48400640 1 -- 172.21.15.27:0/1507657657 shutdown_connections 2024-01-15T18:06:10.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.654+0000 7f3a48400640 1 --2- 172.21.15.27:0/1507657657 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a18066920 0x7f3a18068de0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:10.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.654+0000 7f3a48400640 1 --2- 172.21.15.27:0/1507657657 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3a400d1a40 0x7f3a400c7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:10.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.654+0000 7f3a48400640 1 --2- 172.21.15.27:0/1507657657 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a400cf450 0x7f3a40156570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:10.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.654+0000 7f3a48400640 1 --2- 172.21.15.27:0/1507657657 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a400b0650 0x7f3a400caa30 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:10.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.654+0000 7f3a48400640 1 -- 172.21.15.27:0/1507657657 >> 172.21.15.27:0/1507657657 conn(0x7f3a400d0e70 msgr2=0x7f3a4006e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:10.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.654+0000 7f3a48400640 1 -- 172.21.15.27:0/1507657657 shutdown_connections 2024-01-15T18:06:10.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.654+0000 7f3a48400640 1 -- 172.21.15.27:0/1507657657 wait complete. 2024-01-15T18:06:10.681 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:06:10.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.814+0000 7fda39000640 1 Processor -- start 2024-01-15T18:06:10.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.814+0000 7fda39000640 1 -- start start 2024-01-15T18:06:10.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.818+0000 7fda39000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fda340cf9b0 0x7fda340b10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:10.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.818+0000 7fda39000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda340b38b0 con 0x7fda3415e7d0 2024-01-15T18:06:10.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.818+0000 7fda39000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda340cd550 con 0x7fda340cedc0 2024-01-15T18:06:10.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.818+0000 7fda39000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda340cd270 con 0x7fda340cf9b0 2024-01-15T18:06:10.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.818+0000 7fda31e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fda3415e7d0 0x7fda340cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49686/0 (socket says 172.21.15.27:49686) 2024-01-15T18:06:10.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.818+0000 7fda31e00640 1 -- 172.21.15.27:0/3500577883 learned_addr learned my addr 172.21.15.27:0/3500577883 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:06:10.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.818+0000 7fda31400640 1 --2- 172.21.15.27:0/3500577883 >> v2:172.21.15.93:3300/0 conn(0x7fda340cf9b0 0x7fda340b10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:10.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.818+0000 7fda30a00640 1 -- 172.21.15.27:0/3500577883 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581757090 0 0) 0x7fda340b38b0 con 0x7fda3415e7d0 2024-01-15T18:06:10.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.818+0000 7fda30a00640 1 -- 172.21.15.27:0/3500577883 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda14003530 con 0x7fda3415e7d0 2024-01-15T18:06:10.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.818+0000 7fda30a00640 1 -- 172.21.15.27:0/3500577883 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3387168888 0 0) 0x7fda340cd550 con 0x7fda340cedc0 2024-01-15T18:06:10.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.818+0000 7fda30a00640 1 -- 172.21.15.27:0/3500577883 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda340b38b0 con 0x7fda340cedc0 2024-01-15T18:06:10.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.818+0000 7fda31400640 1 -- 172.21.15.27:0/3500577883 >> v1:172.21.15.93:6789/0 conn(0x7fda340cedc0 legacy=0x7fda340b1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:10.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.818+0000 7fda31400640 1 -- 172.21.15.27:0/3500577883 >> v1:172.21.15.27:6789/0 conn(0x7fda3415e7d0 legacy=0x7fda340cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:10.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.818+0000 7fda31400640 1 -- 172.21.15.27:0/3500577883 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda340b35d0 con 0x7fda340cf9b0 2024-01-15T18:06:10.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.822+0000 7fda31400640 1 --2- 172.21.15.27:0/3500577883 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda340cf9b0 0x7fda340b10c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fda1c0097c0 tx=0x7fda340b2520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d47be5845246903f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:10.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.822+0000 7fda30a00640 1 -- 172.21.15.27:0/3500577883 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c0ab600 con 0x7fda340cf9b0 2024-01-15T18:06:10.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.822+0000 7fda30a00640 1 -- 172.21.15.27:0/3500577883 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1c0a7050 con 0x7fda340cf9b0 2024-01-15T18:06:10.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.822+0000 7fda30a00640 1 -- 172.21.15.27:0/3500577883 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c0ab980 con 0x7fda340cf9b0 2024-01-15T18:06:10.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.822+0000 7fda39000640 1 -- 172.21.15.27:0/3500577883 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda340cf9b0 msgr2=0x7fda340b10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:10.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.822+0000 7fda39000640 1 --2- 172.21.15.27:0/3500577883 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda340cf9b0 0x7fda340b10c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fda1c0097c0 tx=0x7fda340b2520 comp rx=0 tx=0).stop 2024-01-15T18:06:10.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.822+0000 7fda39000640 1 -- 172.21.15.27:0/3500577883 shutdown_connections 2024-01-15T18:06:10.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.822+0000 7fda39000640 1 --2- 172.21.15.27:0/3500577883 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda340cf9b0 0x7fda340b10c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:10.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.822+0000 7fda39000640 1 -- 172.21.15.27:0/3500577883 >> 172.21.15.27:0/3500577883 conn(0x7fda340b2a10 msgr2=0x7fda340b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:10.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.822+0000 7fda39000640 1 -- 172.21.15.27:0/3500577883 shutdown_connections 2024-01-15T18:06:10.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.822+0000 7fda39000640 1 -- 172.21.15.27:0/3500577883 wait complete. 2024-01-15T18:06:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.822+0000 7fda39000640 1 Processor -- start 2024-01-15T18:06:10.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.822+0000 7fda39000640 1 -- start start 2024-01-15T18:06:10.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.826+0000 7fda39000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda340cedc0 0x7fda341551d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:10.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.826+0000 7fda39000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda340cf9b0 0x7fda340b4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:10.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.826+0000 7fda39000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda3415e7d0 0x7fda340adcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:10.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.826+0000 7fda39000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda34157300 con 0x7fda340cf9b0 2024-01-15T18:06:10.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.826+0000 7fda39000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda340ae860 con 0x7fda340cedc0 2024-01-15T18:06:10.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.826+0000 7fda39000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda340aeb40 con 0x7fda3415e7d0 2024-01-15T18:06:10.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.826+0000 7fda31400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda340cf9b0 0x7fda340b4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:10.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.826+0000 7fda31400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda340cf9b0 0x7fda340b4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56686/0 (socket says 172.21.15.27:56686) 2024-01-15T18:06:10.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.826+0000 7fda31400640 1 -- 172.21.15.27:0/2601912927 learned_addr learned my addr 172.21.15.27:0/2601912927 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:10.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.826+0000 7fda31e00640 1 --2- 172.21.15.27:0/2601912927 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda340cedc0 0x7fda341551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:10.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.826+0000 7fda32800640 1 --2- 172.21.15.27:0/2601912927 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda3415e7d0 0x7fda340adcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:10.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.826+0000 7fda31400640 1 -- 172.21.15.27:0/2601912927 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda3415e7d0 msgr2=0x7fda340adcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:10.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.826+0000 7fda31400640 1 --2- 172.21.15.27:0/2601912927 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda3415e7d0 0x7fda340adcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:10.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.826+0000 7fda31400640 1 -- 172.21.15.27:0/2601912927 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda340cedc0 msgr2=0x7fda341551d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:10.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.826+0000 7fda31400640 1 --2- 172.21.15.27:0/2601912927 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda340cedc0 0x7fda341551d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:10.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.826+0000 7fda31400640 1 -- 172.21.15.27:0/2601912927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda1c009420 con 0x7fda340cf9b0 2024-01-15T18:06:10.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.826+0000 7fda31e00640 1 --2- 172.21.15.27:0/2601912927 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda340cedc0 0x7fda341551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:06:10.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.826+0000 7fda31400640 1 --2- 172.21.15.27:0/2601912927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda340cf9b0 0x7fda340b4c10 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fda1c0ae040 tx=0x7fda1c019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:10.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.826+0000 7fda2a000640 1 -- 172.21.15.27:0/2601912927 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c0abe80 con 0x7fda340cf9b0 2024-01-15T18:06:10.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.826+0000 7fda2a000640 1 -- 172.21.15.27:0/2601912927 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1c0b4430 con 0x7fda340cf9b0 2024-01-15T18:06:10.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.826+0000 7fda39000640 1 -- 172.21.15.27:0/2601912927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda34156f90 con 0x7fda340cf9b0 2024-01-15T18:06:10.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.826+0000 7fda2a000640 1 -- 172.21.15.27:0/2601912927 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c0b47c0 con 0x7fda340cf9b0 2024-01-15T18:06:10.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.826+0000 7fda39000640 1 -- 172.21.15.27:0/2601912927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda34155710 con 0x7fda340cf9b0 2024-01-15T18:06:10.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.826+0000 7fda39000640 1 -- 172.21.15.27:0/2601912927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda34156df0 con 0x7fda340cf9b0 2024-01-15T18:06:10.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.830+0000 7fda2a000640 1 -- 172.21.15.27:0/2601912927 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fda1c0b2040 con 0x7fda340cf9b0 2024-01-15T18:06:10.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.830+0000 7fda2a000640 1 --2- 172.21.15.27:0/2601912927 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fda0c0668f0 0x7fda0c068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:10.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.830+0000 7fda2a000640 1 -- 172.21.15.27:0/2601912927 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda1c11e110 con 0x7fda340cf9b0 2024-01-15T18:06:10.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.830+0000 7fda31e00640 1 --2- 172.21.15.27:0/2601912927 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fda0c0668f0 0x7fda0c068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:10.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.830+0000 7fda31e00640 1 --2- 172.21.15.27:0/2601912927 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fda0c0668f0 0x7fda0c068db0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fda24007fd0 tx=0x7fda24007d80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:10.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:10.834+0000 7fda2a000640 1 -- 172.21.15.27:0/2601912927 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fda34156df0 con 0x7fda340cf9b0 2024-01-15T18:06:11.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.022+0000 7fda39000640 1 -- 172.21.15.27:0/2601912927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda34156df0 con 0x7fda340cf9b0 2024-01-15T18:06:11.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.026+0000 7fda2a000640 1 -- 172.21.15.27:0/2601912927 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4913 (secure 0 0 0) 0x7fda34156df0 con 0x7fda340cf9b0 2024-01-15T18:06:11.030 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:06:11.031 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4910,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4204677709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4204677709},{"type":"v1","addr":"172.21.15.27:6835","nonce":4204677709}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4913,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/1766105879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":1766105879},{"type":"v1","addr":"172.21.15.93:6836","nonce":1766105879}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/2799226725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":2799226725},{"type":"v1","addr":"172.21.15.93:6837","nonce":2799226725}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:06:00.558126+0000","modified":"2024-01-15T18:06:03.589849+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5232":{"gid":5232,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/2187944510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2187944510},{"type":"v1","addr":"172.21.15.27:6834","nonce":2187944510}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-01-15T18:06:11.031 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-01-15T18:06:11.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.026+0000 7fda39000640 1 -- 172.21.15.27:0/2601912927 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fda0c0668f0 msgr2=0x7fda0c068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:11.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.026+0000 7fda39000640 1 --2- 172.21.15.27:0/2601912927 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fda0c0668f0 0x7fda0c068db0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fda24007fd0 tx=0x7fda24007d80 comp rx=0 tx=0).stop 2024-01-15T18:06:11.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.026+0000 7fda39000640 1 -- 172.21.15.27:0/2601912927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda340cf9b0 msgr2=0x7fda340b4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.026+0000 7fda39000640 1 --2- 172.21.15.27:0/2601912927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda340cf9b0 0x7fda340b4c10 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fda1c0ae040 tx=0x7fda1c019190 comp rx=0 tx=0).stop 2024-01-15T18:06:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.026+0000 7fda39000640 1 -- 172.21.15.27:0/2601912927 shutdown_connections 2024-01-15T18:06:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.026+0000 7fda39000640 1 --2- 172.21.15.27:0/2601912927 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fda0c0668f0 0x7fda0c068db0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.026+0000 7fda39000640 1 --2- 172.21.15.27:0/2601912927 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda3415e7d0 0x7fda340adcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.026+0000 7fda39000640 1 --2- 172.21.15.27:0/2601912927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda340cf9b0 0x7fda340b4c10 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.026+0000 7fda39000640 1 --2- 172.21.15.27:0/2601912927 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda340cedc0 0x7fda341551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.026+0000 7fda39000640 1 -- 172.21.15.27:0/2601912927 >> 172.21.15.27:0/2601912927 conn(0x7fda340b2a10 msgr2=0x7fda340cc450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.026+0000 7fda39000640 1 -- 172.21.15.27:0/2601912927 shutdown_connections 2024-01-15T18:06:11.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.026+0000 7fda39000640 1 -- 172.21.15.27:0/2601912927 wait complete. 2024-01-15T18:06:11.455 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-15T18:06:00.558126+0000', 'modified': '2024-01-15T18:06:03.589849+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5232}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5232': {'gid': 5232, 'name': 'a', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/2187944510', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 2187944510}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 2187944510}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-15T18:06:11.455 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-15T18:06:11.455 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-15T18:06:11.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.594+0000 7f7c28e00640 1 Processor -- start 2024-01-15T18:06:11.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.594+0000 7f7c28e00640 1 -- start start 2024-01-15T18:06:11.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.594+0000 7f7c28e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7c2415b290 0x7f7c240d19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.594+0000 7f7c28e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f7c240cfbc0 0x7f7c240cd720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.594+0000 7f7c28e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2415b670 con 0x7f7c240cdc60 2024-01-15T18:06:11.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.594+0000 7f7c28e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c240cc470 con 0x7f7c2415b290 2024-01-15T18:06:11.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.594+0000 7f7c28e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c240ccd10 con 0x7f7c240cfbc0 2024-01-15T18:06:11.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.594+0000 7f7c21e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7c2415b290 0x7f7c240d19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:11.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.594+0000 7f7c21e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7c2415b290 0x7f7c240d19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56696/0 (socket says 172.21.15.27:56696) 2024-01-15T18:06:11.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.594+0000 7f7c21e00640 1 -- 172.21.15.27:0/549777726 learned_addr learned my addr 172.21.15.27:0/549777726 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:11.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.594+0000 7f7c22800640 1 --1- 172.21.15.27:0/549777726 >> v1:172.21.15.27:6789/0 conn(0x7f7c240cdc60 0x7f7c240cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49700/0 (socket says 172.21.15.27:49700) 2024-01-15T18:06:11.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.594+0000 7f7c21400640 1 --2- 172.21.15.27:0/549777726 >> v2:172.21.15.93:3300/0 conn(0x7f7c240cfbc0 0x7f7c240cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:11.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.594+0000 7f7c21e00640 1 -- 172.21.15.27:0/549777726 >> v2:172.21.15.93:3300/0 conn(0x7f7c240cfbc0 msgr2=0x7f7c240cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:11.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.594+0000 7f7c21e00640 1 --2- 172.21.15.27:0/549777726 >> v2:172.21.15.93:3300/0 conn(0x7f7c240cfbc0 0x7f7c240cd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.594+0000 7f7c21e00640 1 -- 172.21.15.27:0/549777726 >> v1:172.21.15.27:6789/0 conn(0x7f7c240cdc60 legacy=0x7f7c240cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:11.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.594+0000 7f7c20a00640 1 -- 172.21.15.27:0/549777726 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2521981666 0 0) 0x7f7c2415b670 con 0x7f7c240cdc60 2024-01-15T18:06:11.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.594+0000 7f7c21e00640 1 -- 172.21.15.27:0/549777726 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c240ce630 con 0x7f7c2415b290 2024-01-15T18:06:11.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.598+0000 7f7c21e00640 1 --2- 172.21.15.27:0/549777726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c2415b290 0x7f7c240d19b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7c240728e0 tx=0x7f7c1401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea450df18e1c5ebf server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:11.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.598+0000 7f7c20a00640 1 -- 172.21.15.27:0/549777726 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c140aa430 con 0x7f7c2415b290 2024-01-15T18:06:11.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.598+0000 7f7c20a00640 1 -- 172.21.15.27:0/549777726 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c140aa590 con 0x7f7c2415b290 2024-01-15T18:06:11.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.598+0000 7f7c20a00640 1 -- 172.21.15.27:0/549777726 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c140aa920 con 0x7f7c2415b290 2024-01-15T18:06:11.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.598+0000 7f7c28e00640 1 -- 172.21.15.27:0/549777726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c2415b290 msgr2=0x7f7c240d19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:11.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.598+0000 7f7c28e00640 1 --2- 172.21.15.27:0/549777726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c2415b290 0x7f7c240d19b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7c240728e0 tx=0x7f7c1401c4d0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.598+0000 7f7c28e00640 1 -- 172.21.15.27:0/549777726 shutdown_connections 2024-01-15T18:06:11.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.598+0000 7f7c28e00640 1 --2- 172.21.15.27:0/549777726 >> v2:172.21.15.93:3300/0 conn(0x7f7c240cfbc0 0x7f7c240cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.598+0000 7f7c28e00640 1 --2- 172.21.15.27:0/549777726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c2415b290 0x7f7c240d19b0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.598+0000 7f7c28e00640 1 -- 172.21.15.27:0/549777726 >> 172.21.15.27:0/549777726 conn(0x7f7c240d0e90 msgr2=0x7f7c240d12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:11.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.598+0000 7f7c28e00640 1 -- 172.21.15.27:0/549777726 shutdown_connections 2024-01-15T18:06:11.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.598+0000 7f7c28e00640 1 -- 172.21.15.27:0/549777726 wait complete. 2024-01-15T18:06:11.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.602+0000 7f7c28e00640 1 Processor -- start 2024-01-15T18:06:11.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.602+0000 7f7c28e00640 1 -- start start 2024-01-15T18:06:11.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.602+0000 7f7c28e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7c240cdc60 0x7f7c2406fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.602+0000 7f7c28e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7c240cfbc0 0x7f7c24152e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.602+0000 7f7c28e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c2415b290 0x7f7c240d14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.602+0000 7f7c28e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c240c8060 con 0x7f7c2415b290 2024-01-15T18:06:11.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.602+0000 7f7c28e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c24153cd0 con 0x7f7c240cdc60 2024-01-15T18:06:11.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.602+0000 7f7c28e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c24153fb0 con 0x7f7c240cfbc0 2024-01-15T18:06:11.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.602+0000 7f7c22800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c2415b290 0x7f7c240d14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:11.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.602+0000 7f7c22800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c2415b290 0x7f7c240d14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56710/0 (socket says 172.21.15.27:56710) 2024-01-15T18:06:11.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.602+0000 7f7c22800640 1 -- 172.21.15.27:0/3270760899 learned_addr learned my addr 172.21.15.27:0/3270760899 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:11.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.602+0000 7f7c21400640 1 --2- 172.21.15.27:0/3270760899 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7c240cfbc0 0x7f7c24152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:11.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.602+0000 7f7c21e00640 1 --2- 172.21.15.27:0/3270760899 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7c240cdc60 0x7f7c2406fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:11.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.602+0000 7f7c22800640 1 -- 172.21.15.27:0/3270760899 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7c240cfbc0 msgr2=0x7f7c24152e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:11.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.602+0000 7f7c22800640 1 --2- 172.21.15.27:0/3270760899 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7c240cfbc0 0x7f7c24152e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.602+0000 7f7c22800640 1 -- 172.21.15.27:0/3270760899 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7c240cdc60 msgr2=0x7f7c2406fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:11.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.602+0000 7f7c22800640 1 --2- 172.21.15.27:0/3270760899 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7c240cdc60 0x7f7c2406fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.602+0000 7f7c22800640 1 -- 172.21.15.27:0/3270760899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c140094d0 con 0x7f7c2415b290 2024-01-15T18:06:11.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.602+0000 7f7c22800640 1 --2- 172.21.15.27:0/3270760899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c2415b290 0x7f7c240d14d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c00c780 tx=0x7f7c1c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:11.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.602+0000 7f7c1a000640 1 -- 172.21.15.27:0/3270760899 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c00ce90 con 0x7f7c2415b290 2024-01-15T18:06:11.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.602+0000 7f7c1a000640 1 -- 172.21.15.27:0/3270760899 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1c00a640 con 0x7f7c2415b290 2024-01-15T18:06:11.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.602+0000 7f7c28e00640 1 -- 172.21.15.27:0/3270760899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c24154290 con 0x7f7c2415b290 2024-01-15T18:06:11.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.602+0000 7f7c1a000640 1 -- 172.21.15.27:0/3270760899 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c00a9d0 con 0x7f7c2415b290 2024-01-15T18:06:11.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.602+0000 7f7c28e00640 1 -- 172.21.15.27:0/3270760899 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c24154850 con 0x7f7c2415b290 2024-01-15T18:06:11.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.606+0000 7f7c1a000640 1 -- 172.21.15.27:0/3270760899 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f7c1c00ab30 con 0x7f7c2415b290 2024-01-15T18:06:11.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.606+0000 7f7c1a000640 1 --2- 172.21.15.27:0/3270760899 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7bfc066920 0x7f7bfc068de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.606+0000 7f7c1a000640 1 -- 172.21.15.27:0/3270760899 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c1c083b70 con 0x7f7c2415b290 2024-01-15T18:06:11.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.606+0000 7f7c2c2a9000 1 Processor -- start 2024-01-15T18:06:11.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.606+0000 7f7c2c2a9000 1 -- start start 2024-01-15T18:06:11.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.606+0000 7f7c21e00640 1 --2- 172.21.15.27:0/3270760899 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7bfc066920 0x7f7bfc068de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:11.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.606+0000 7f7c2c2a9000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5565015c5c90 0x5565015c6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.606+0000 7f7c2c2a9000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5565015c65b0 0x55650157bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.606+0000 7f7c2c2a9000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55650157c320 0x55650157c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.606+0000 7f7c2c2a9000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565013e6130 con 0x55650157c320 2024-01-15T18:06:11.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.606+0000 7f7c2c2a9000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565013d39a0 con 0x5565015c65b0 2024-01-15T18:06:11.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.606+0000 7f7c2c2a9000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x5565013d0ed0 con 0x5565015c5c90 2024-01-15T18:06:11.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.606+0000 7f7c21e00640 1 --2- 172.21.15.27:0/3270760899 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7bfc066920 0x7f7bfc068de0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7c140ae000 tx=0x7f7c140b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:11.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.606+0000 7f7c22800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55650157c320 0x55650157c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:11.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.606+0000 7f7c22800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55650157c320 0x55650157c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56716/0 (socket says 172.21.15.27:56716) 2024-01-15T18:06:11.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.606+0000 7f7c22800640 1 -- 172.21.15.27:0/1849628565 learned_addr learned my addr 172.21.15.27:0/1849628565 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:11.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.606+0000 7f7c21e00640 1 --2- 172.21.15.27:0/1849628565 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5565015c5c90 0x5565015c6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:11.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.606+0000 7f7c21400640 1 --2- 172.21.15.27:0/1849628565 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5565015c65b0 0x55650157bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:11.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c22800640 1 -- 172.21.15.27:0/1849628565 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5565015c5c90 msgr2=0x5565015c6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:11.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c22800640 1 --2- 172.21.15.27:0/1849628565 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5565015c5c90 0x5565015c6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c22800640 1 -- 172.21.15.27:0/1849628565 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5565015c65b0 msgr2=0x55650157bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:11.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c22800640 1 --2- 172.21.15.27:0/1849628565 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5565015c65b0 0x55650157bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c22800640 1 -- 172.21.15.27:0/1849628565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565010c4890 con 0x55650157c320 2024-01-15T18:06:11.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c21400640 1 --2- 172.21.15.27:0/1849628565 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5565015c65b0 0x55650157bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:06:11.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c22800640 1 --2- 172.21.15.27:0/1849628565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55650157c320 0x55650157c700 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c054810 tx=0x7f7c1c054b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b715a76700eee1c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:11.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c12a00640 1 -- 172.21.15.27:0/1849628565 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c052eb0 con 0x55650157c320 2024-01-15T18:06:11.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c12a00640 1 -- 172.21.15.27:0/1849628565 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1c082110 con 0x55650157c320 2024-01-15T18:06:11.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c12a00640 1 -- 172.21.15.27:0/1849628565 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c05c8d0 con 0x55650157c320 2024-01-15T18:06:11.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/1849628565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55650157c320 msgr2=0x55650157c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:11.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c2c2a9000 1 --2- 172.21.15.27:0/1849628565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55650157c320 0x55650157c700 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c054810 tx=0x7f7c1c054b10 comp rx=0 tx=0).stop 2024-01-15T18:06:11.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/1849628565 shutdown_connections 2024-01-15T18:06:11.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c2c2a9000 1 --2- 172.21.15.27:0/1849628565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55650157c320 0x55650157c700 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c2c2a9000 1 --2- 172.21.15.27:0/1849628565 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5565015c65b0 0x55650157bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c2c2a9000 1 --2- 172.21.15.27:0/1849628565 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5565015c5c90 0x5565015c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/1849628565 >> 172.21.15.27:0/1849628565 conn(0x5565015782d0 msgr2=0x55650157a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:11.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/1849628565 shutdown_connections 2024-01-15T18:06:11.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/1849628565 wait complete. 2024-01-15T18:06:11.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c2c2a9000 1 -- start start 2024-01-15T18:06:11.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c2c2a9000 1 Processor -- start 2024-01-15T18:06:11.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c2c2a9000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55650157c320 0x5565015ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c2c2a9000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5565015c5c90 0x5565015aec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c2c2a9000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5565015c65b0 0x5565015af150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c2c2a9000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565013cc7c0 con 0x55650157c320 2024-01-15T18:06:11.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c2c2a9000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x556501384bf0 con 0x5565015c65b0 2024-01-15T18:06:11.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c2c2a9000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x55650138aa90 con 0x5565015c5c90 2024-01-15T18:06:11.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.610+0000 7f7c2c2a9000 10 client.0 _ll_register_callbacks cb 0x5565015673c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:06:11.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c21e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55650157c320 0x5565015ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:11.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c21400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5565015c5c90 0x5565015aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:11.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c22800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5565015c65b0 0x5565015af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:11.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c21e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55650157c320 0x5565015ae6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56730/0 (socket says 172.21.15.27:56730) 2024-01-15T18:06:11.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c21e00640 1 -- 172.21.15.27:0/2666978717 learned_addr learned my addr 172.21.15.27:0/2666978717 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:11.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c21400640 1 -- 172.21.15.27:0/2666978717 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5565015c65b0 msgr2=0x5565015af150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:11.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c21400640 1 --2- 172.21.15.27:0/2666978717 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5565015c65b0 0x5565015af150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c21400640 1 -- 172.21.15.27:0/2666978717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55650157c320 msgr2=0x5565015ae6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:11.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c21400640 1 --2- 172.21.15.27:0/2666978717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55650157c320 0x5565015ae6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c21400640 1 -- 172.21.15.27:0/2666978717 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c24154850 con 0x5565015c5c90 2024-01-15T18:06:11.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c21e00640 1 --2- 172.21.15.27:0/2666978717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55650157c320 0x5565015ae6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:06:11.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c21400640 1 --2- 172.21.15.27:0/2666978717 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5565015c5c90 0x5565015aec10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7c24067a00 tx=0x7f7c0c014ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=719eec82aaf8e203 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:11.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c0b400640 10 client.0 ms_handle_connect on v2:172.21.15.93:3301/0 2024-01-15T18:06:11.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c0b400640 1 -- 172.21.15.27:0/2666978717 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c0167c0 con 0x5565015c5c90 2024-01-15T18:06:11.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c0b400640 1 -- 172.21.15.27:0/2666978717 <== mon.2 v2:172.21.15.93:3301/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c0c01a070 con 0x5565015c5c90 2024-01-15T18:06:11.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/2666978717 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55650138e470 con 0x5565015c5c90 2024-01-15T18:06:11.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c0b400640 1 -- 172.21.15.27:0/2666978717 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c0c016920 con 0x5565015c5c90 2024-01-15T18:06:11.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c2c2a9000 10 client.5033 fetch_fsmap learned FSMap version 17 2024-01-15T18:06:11.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/2666978717 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x55650138d190 con 0x5565015c5c90 2024-01-15T18:06:11.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c0b400640 1 -- 172.21.15.27:0/2666978717 <== mon.2 v2:172.21.15.93:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55650138e470 con 0x5565015c5c90 2024-01-15T18:06:11.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c0b400640 1 -- 172.21.15.27:0/2666978717 <== mon.2 v2:172.21.15.93:3301/0 5 ==== fsmap(e 17) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7c0c011030 con 0x5565015c5c90 2024-01-15T18:06:11.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c2c2a9000 10 client.5033 fetch_fsmap finished waiting for FSMap version 17 2024-01-15T18:06:11.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c2c2a9000 10 client.5033 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-15T18:06:11.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c2c2a9000 20 client.5033 populate_metadata read hostname 'smithi027' 2024-01-15T18:06:11.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c2c2a9000 1 --2- 172.21.15.27:0/2666978717 >> [v2:172.21.15.27:6832/2187944510,v1:172.21.15.27:6834/2187944510] conn(0x5565015d9f80 0x5565015dc370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c2c2a9000 4 client.5033 mds_command: new command op to 5232 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-15T18:06:11.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/2666978717 --> [v2:172.21.15.27:6832/2187944510,v1:172.21.15.27:6834/2187944510] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556501391b50 con 0x5565015d9f80 2024-01-15T18:06:11.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c21e00640 1 --2- 172.21.15.27:0/2666978717 >> [v2:172.21.15.27:6832/2187944510,v1:172.21.15.27:6834/2187944510] conn(0x5565015d9f80 0x5565015dc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:11.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c21e00640 1 --2- 172.21.15.27:0/2666978717 >> [v2:172.21.15.27:6832/2187944510,v1:172.21.15.27:6834/2187944510] conn(0x5565015d9f80 0x5565015dc370 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c36062decc90a8a server_cookie=1c7f5384648b50af in_seq=0 out_seq=0 2024-01-15T18:06:11.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c0b400640 10 client.5033 ms_handle_connect on v2:172.21.15.27:6832/2187944510 2024-01-15T18:06:11.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c0b400640 1 -- 172.21.15.27:0/2666978717 <== mds.0 v2:172.21.15.27:6832/2187944510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556501391b50 con 0x5565015d9f80 2024-01-15T18:06:11.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c0b400640 10 client.5033 handle_command_reply: tid=0 2024-01-15T18:06:11.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.614+0000 7f7c2c2a9000 1 client.5033 shutdown 2024-01-15T18:06:11.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.618+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/2666978717 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5565015c5c90 msgr2=0x5565015aec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:11.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.618+0000 7f7c2c2a9000 1 --2- 172.21.15.27:0/2666978717 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5565015c5c90 0x5565015aec10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7c24067a00 tx=0x7f7c0c014ae0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.618+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/2666978717 shutdown_connections 2024-01-15T18:06:11.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.618+0000 7f7c2c2a9000 1 --2- 172.21.15.27:0/2666978717 >> [v2:172.21.15.27:6832/2187944510,v1:172.21.15.27:6834/2187944510] conn(0x5565015d9f80 0x5565015dc370 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.618+0000 7f7c2c2a9000 1 --2- 172.21.15.27:0/2666978717 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5565015c65b0 0x5565015af150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.618+0000 7f7c2c2a9000 1 --2- 172.21.15.27:0/2666978717 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5565015c5c90 0x5565015aec10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.618+0000 7f7c2c2a9000 1 --2- 172.21.15.27:0/2666978717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55650157c320 0x5565015ae6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.618+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/2666978717 >> 172.21.15.27:0/2666978717 conn(0x5565015782d0 msgr2=0x556501580b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:11.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.618+0000 7f7c0b400640 0 client.5033 ms_handle_reset on v2:172.21.15.27:6832/2187944510 2024-01-15T18:06:11.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.618+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/2666978717 shutdown_connections 2024-01-15T18:06:11.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.618+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/2666978717 wait complete. 2024-01-15T18:06:11.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.618+0000 7f7c2c2a9000 20 client.5033 trim_cache size 0 max 16384 2024-01-15T18:06:11.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c2c2a9000 1 Processor -- start 2024-01-15T18:06:11.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c2c2a9000 1 -- start start 2024-01-15T18:06:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c2c2a9000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55650157c320 0x55650157b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c2c2a9000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565015c5c90 0x55650157bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c2c2a9000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5565015c65b0 0x5565015c3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c2c2a9000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x556501085940 con 0x5565015c5c90 2024-01-15T18:06:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c2c2a9000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565013a6e30 con 0x55650157c320 2024-01-15T18:06:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c2c2a9000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x5565015ab270 con 0x5565015c65b0 2024-01-15T18:06:11.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c21400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565015c5c90 0x55650157bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:11.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c21400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565015c5c90 0x55650157bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56742/0 (socket says 172.21.15.27:56742) 2024-01-15T18:06:11.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c21400640 1 -- 172.21.15.27:0/1047539010 learned_addr learned my addr 172.21.15.27:0/1047539010 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c21400640 1 -- 172.21.15.27:0/1047539010 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5565015c65b0 msgr2=0x5565015c3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c21400640 1 --2- 172.21.15.27:0/1047539010 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5565015c65b0 0x5565015c3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c21e00640 1 --2- 172.21.15.27:0/1047539010 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55650157c320 0x55650157b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c21400640 1 -- 172.21.15.27:0/1047539010 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55650157c320 msgr2=0x55650157b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c21400640 1 --2- 172.21.15.27:0/1047539010 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55650157c320 0x55650157b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c21400640 1 -- 172.21.15.27:0/1047539010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556501373490 con 0x5565015c5c90 2024-01-15T18:06:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c21e00640 1 --2- 172.21.15.27:0/1047539010 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55650157c320 0x55650157b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:06:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c21400640 1 --2- 172.21.15.27:0/1047539010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565015c5c90 0x55650157bc10 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c009310 tx=0x7f7c0c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8990e347e831e377 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c13400640 1 -- 172.21.15.27:0/1047539010 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c009e40 con 0x5565015c5c90 2024-01-15T18:06:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c13400640 1 -- 172.21.15.27:0/1047539010 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0c0167c0 con 0x5565015c5c90 2024-01-15T18:06:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c13400640 1 -- 172.21.15.27:0/1047539010 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c004230 con 0x5565015c5c90 2024-01-15T18:06:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/1047539010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565015c5c90 msgr2=0x55650157bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c2c2a9000 1 --2- 172.21.15.27:0/1047539010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565015c5c90 0x55650157bc10 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c009310 tx=0x7f7c0c019000 comp rx=0 tx=0).stop 2024-01-15T18:06:11.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/1047539010 shutdown_connections 2024-01-15T18:06:11.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c2c2a9000 1 --2- 172.21.15.27:0/1047539010 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5565015c65b0 0x5565015c3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c2c2a9000 1 --2- 172.21.15.27:0/1047539010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565015c5c90 0x55650157bc10 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c2c2a9000 1 --2- 172.21.15.27:0/1047539010 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55650157c320 0x55650157b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/1047539010 >> 172.21.15.27:0/1047539010 conn(0x5565015782d0 msgr2=0x5565015d6a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:11.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/1047539010 shutdown_connections 2024-01-15T18:06:11.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.634+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/1047539010 wait complete. 2024-01-15T18:06:11.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c2c2a9000 1 -- start start 2024-01-15T18:06:11.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c2c2a9000 1 Processor -- start 2024-01-15T18:06:11.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c2c2a9000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55650157c320 0x5565015dd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c2c2a9000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565015c5c90 0x5565015dd900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c2c2a9000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5565015c65b0 0x5565015e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c2c2a9000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565013cb650 con 0x5565015c5c90 2024-01-15T18:06:11.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c2c2a9000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565013a5e10 con 0x55650157c320 2024-01-15T18:06:11.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c2c2a9000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x5565013c85c0 con 0x5565015c65b0 2024-01-15T18:06:11.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c2c2a9000 10 client.0 _ll_register_callbacks cb 0x55650141ac00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:06:11.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c21400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565015c5c90 0x5565015dd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:11.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c21400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565015c5c90 0x5565015dd900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56758/0 (socket says 172.21.15.27:56758) 2024-01-15T18:06:11.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c21400640 1 -- 172.21.15.27:0/826543958 learned_addr learned my addr 172.21.15.27:0/826543958 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:11.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c21e00640 1 --2- 172.21.15.27:0/826543958 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55650157c320 0x5565015dd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:11.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c22800640 1 --2- 172.21.15.27:0/826543958 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5565015c65b0 0x5565015e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:11.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c21400640 1 -- 172.21.15.27:0/826543958 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5565015c65b0 msgr2=0x5565015e7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:11.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c21400640 1 --2- 172.21.15.27:0/826543958 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5565015c65b0 0x5565015e7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c21400640 1 -- 172.21.15.27:0/826543958 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55650157c320 msgr2=0x5565015dd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:11.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c21400640 1 --2- 172.21.15.27:0/826543958 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55650157c320 0x5565015dd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c21400640 1 -- 172.21.15.27:0/826543958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c0c000e70 con 0x5565015c5c90 2024-01-15T18:06:11.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c21400640 1 --2- 172.21.15.27:0/826543958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565015c5c90 0x5565015dd900 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c014fe0 tx=0x7f7c0c0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea3eda09dfc2290 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:11.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c0b400640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-15T18:06:11.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c0b400640 1 -- 172.21.15.27:0/826543958 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c005ba0 con 0x5565015c5c90 2024-01-15T18:06:11.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c0b400640 1 -- 172.21.15.27:0/826543958 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c0c005d00 con 0x5565015c5c90 2024-01-15T18:06:11.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/826543958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5565013cb110 con 0x5565015c5c90 2024-01-15T18:06:11.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c0b400640 1 -- 172.21.15.27:0/826543958 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c0c009580 con 0x5565015c5c90 2024-01-15T18:06:11.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c2c2a9000 10 client.5445 fetch_fsmap learned FSMap version 17 2024-01-15T18:06:11.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/826543958 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) v3 -- 0x5565010c4cc0 con 0x5565015c5c90 2024-01-15T18:06:11.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c0b400640 1 -- 172.21.15.27:0/826543958 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5565013cb110 con 0x5565015c5c90 2024-01-15T18:06:11.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c0b400640 1 -- 172.21.15.27:0/826543958 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 17) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7c0c011070 con 0x5565015c5c90 2024-01-15T18:06:11.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c2c2a9000 10 client.5445 fetch_fsmap finished waiting for FSMap version 17 2024-01-15T18:06:11.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c2c2a9000 10 client.5445 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-15T18:06:11.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c2c2a9000 20 client.5445 populate_metadata read hostname 'smithi027' 2024-01-15T18:06:11.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c2c2a9000 1 --2- 172.21.15.27:0/826543958 >> [v2:172.21.15.27:6832/2187944510,v1:172.21.15.27:6834/2187944510] conn(0x5565015e8340 0x5565015e8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c2c2a9000 4 client.5445 mds_command: new command op to 5232 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-15T18:06:11.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.638+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/826543958 --> [v2:172.21.15.27:6832/2187944510,v1:172.21.15.27:6834/2187944510] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5565013fa9a0 con 0x5565015e8340 2024-01-15T18:06:11.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c21e00640 1 --2- 172.21.15.27:0/826543958 >> [v2:172.21.15.27:6832/2187944510,v1:172.21.15.27:6834/2187944510] conn(0x5565015e8340 0x5565015e8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:11.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c21e00640 1 --2- 172.21.15.27:0/826543958 >> [v2:172.21.15.27:6832/2187944510,v1:172.21.15.27:6834/2187944510] conn(0x5565015e8340 0x5565015e8720 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=964ebfb73ee0b744 server_cookie=5da68ae2cb37b547 in_seq=0 out_seq=0 2024-01-15T18:06:11.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c0b400640 10 client.5445 ms_handle_connect on v2:172.21.15.27:6832/2187944510 2024-01-15T18:06:11.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c0b400640 1 -- 172.21.15.27:0/826543958 <== mds.0 v2:172.21.15.27:6832/2187944510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+319 (crc 0 0 0) 0x5565013fa9a0 con 0x5565015e8340 2024-01-15T18:06:11.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c0b400640 10 client.5445 handle_command_reply: tid=0 2024-01-15T18:06:11.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c2c2a9000 1 client.5445 shutdown 2024-01-15T18:06:11.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/826543958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565015c5c90 msgr2=0x5565015dd900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:11.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c2c2a9000 1 --2- 172.21.15.27:0/826543958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565015c5c90 0x5565015dd900 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c014fe0 tx=0x7f7c0c0046d0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/826543958 shutdown_connections 2024-01-15T18:06:11.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c2c2a9000 1 --2- 172.21.15.27:0/826543958 >> [v2:172.21.15.27:6832/2187944510,v1:172.21.15.27:6834/2187944510] conn(0x5565015e8340 0x5565015e8720 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c2c2a9000 1 --2- 172.21.15.27:0/826543958 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5565015c65b0 0x5565015e7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c2c2a9000 1 --2- 172.21.15.27:0/826543958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5565015c5c90 0x5565015dd900 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c2c2a9000 1 --2- 172.21.15.27:0/826543958 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55650157c320 0x5565015dd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c0b400640 0 client.5445 ms_handle_reset on v2:172.21.15.27:6832/2187944510 2024-01-15T18:06:11.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/826543958 >> 172.21.15.27:0/826543958 conn(0x5565015782d0 msgr2=0x5565015d72a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:11.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/826543958 shutdown_connections 2024-01-15T18:06:11.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c2c2a9000 1 -- 172.21.15.27:0/826543958 wait complete. 2024-01-15T18:06:11.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c2c2a9000 20 client.5445 trim_cache size 0 max 16384 2024-01-15T18:06:11.651 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:06:11.651 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:06:11.651 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-01-15T18:06:11.651 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 5232, 2024-01-15T18:06:11.651 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-01-15T18:06:11.651 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-01-15T18:06:11.651 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-01-15T18:06:11.651 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 10.474211549, 2024-01-15T18:06:11.652 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 17, 2024-01-15T18:06:11.652 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 34, 2024-01-15T18:06:11.652 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 34, 2024-01-15T18:06:11.652 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 18.377409468 2024-01-15T18:06:11.652 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:06:11.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c28e00640 1 -- 172.21.15.27:0/3270760899 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7bfc066920 msgr2=0x7f7bfc068de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:11.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c28e00640 1 --2- 172.21.15.27:0/3270760899 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7bfc066920 0x7f7bfc068de0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7c140ae000 tx=0x7f7c140b5040 comp rx=0 tx=0).stop 2024-01-15T18:06:11.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c28e00640 1 -- 172.21.15.27:0/3270760899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c2415b290 msgr2=0x7f7c240d14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:11.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c28e00640 1 --2- 172.21.15.27:0/3270760899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c2415b290 0x7f7c240d14d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c00c780 tx=0x7f7c1c00cc40 comp rx=0 tx=0).stop 2024-01-15T18:06:11.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c28e00640 1 -- 172.21.15.27:0/3270760899 shutdown_connections 2024-01-15T18:06:11.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c28e00640 1 --2- 172.21.15.27:0/3270760899 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7bfc066920 0x7f7bfc068de0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c28e00640 1 --2- 172.21.15.27:0/3270760899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c2415b290 0x7f7c240d14d0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c28e00640 1 --2- 172.21.15.27:0/3270760899 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7c240cfbc0 0x7f7c24152e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c28e00640 1 --2- 172.21.15.27:0/3270760899 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7c240cdc60 0x7f7c2406fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c28e00640 1 -- 172.21.15.27:0/3270760899 >> 172.21.15.27:0/3270760899 conn(0x7f7c240d0e90 msgr2=0x7f7c240cea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:11.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c28e00640 1 -- 172.21.15.27:0/3270760899 shutdown_connections 2024-01-15T18:06:11.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.642+0000 7f7c28e00640 1 -- 172.21.15.27:0/3270760899 wait complete. 2024-01-15T18:06:11.669 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:06:11.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.802+0000 7f9880c00640 1 Processor -- start 2024-01-15T18:06:11.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.802+0000 7f9880c00640 1 -- start start 2024-01-15T18:06:11.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.806+0000 7f9880c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f987c06bb80 0x7f987c0cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.806+0000 7f9880c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f987c0cddc0 0x7f987c0b0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.806+0000 7f9880c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f987c0b2770 con 0x7f987c0b2390 2024-01-15T18:06:11.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.806+0000 7f9880c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f987c0ce270 con 0x7f987c0cddc0 2024-01-15T18:06:11.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.806+0000 7f9880c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987c06c060 con 0x7f987c06bb80 2024-01-15T18:06:11.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.806+0000 7f9879e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f987c0b2390 0x7f987c0cf780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49702/0 (socket says 172.21.15.27:49702) 2024-01-15T18:06:11.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.806+0000 7f9879e00640 1 -- 172.21.15.27:0/1228125498 learned_addr learned my addr 172.21.15.27:0/1228125498 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:06:11.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.806+0000 7f9879400640 1 --2- 172.21.15.27:0/1228125498 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f987c06bb80 0x7f987c0cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:11.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.806+0000 7f987a800640 1 --2- 172.21.15.27:0/1228125498 >> v2:172.21.15.93:3300/0 conn(0x7f987c0cddc0 0x7f987c0b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:11.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.806+0000 7f9878a00640 1 -- 172.21.15.27:0/1228125498 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318250002 0 0) 0x7f987c0b2770 con 0x7f987c0b2390 2024-01-15T18:06:11.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.806+0000 7f9878a00640 1 -- 172.21.15.27:0/1228125498 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f985c003530 con 0x7f987c0b2390 2024-01-15T18:06:11.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.806+0000 7f9879400640 1 -- 172.21.15.27:0/1228125498 >> v2:172.21.15.93:3300/0 conn(0x7f987c0cddc0 msgr2=0x7f987c0b0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:11.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.806+0000 7f9879400640 1 --2- 172.21.15.27:0/1228125498 >> v2:172.21.15.93:3300/0 conn(0x7f987c0cddc0 0x7f987c0b0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.806+0000 7f9879400640 1 -- 172.21.15.27:0/1228125498 >> v1:172.21.15.27:6789/0 conn(0x7f987c0b2390 legacy=0x7f987c0cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:11.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.806+0000 7f9879400640 1 -- 172.21.15.27:0/1228125498 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987c06c340 con 0x7f987c06bb80 2024-01-15T18:06:11.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.810+0000 7f9879400640 1 --2- 172.21.15.27:0/1228125498 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f987c06bb80 0x7f987c0cd880 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f98640097c0 tx=0x7f986401c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67ab54f2a400582e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:11.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.810+0000 7f9878a00640 1 -- 172.21.15.27:0/1228125498 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98640ab650 con 0x7f987c06bb80 2024-01-15T18:06:11.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.810+0000 7f9878a00640 1 -- 172.21.15.27:0/1228125498 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98640ab7b0 con 0x7f987c06bb80 2024-01-15T18:06:11.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.810+0000 7f9878a00640 1 -- 172.21.15.27:0/1228125498 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98640abb40 con 0x7f987c06bb80 2024-01-15T18:06:11.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.810+0000 7f9880c00640 1 -- 172.21.15.27:0/1228125498 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f987c06bb80 msgr2=0x7f987c0cd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:11.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.810+0000 7f9880c00640 1 --2- 172.21.15.27:0/1228125498 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f987c06bb80 0x7f987c0cd880 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f98640097c0 tx=0x7f986401c450 comp rx=0 tx=0).stop 2024-01-15T18:06:11.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.814+0000 7f9880c00640 1 -- 172.21.15.27:0/1228125498 shutdown_connections 2024-01-15T18:06:11.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.814+0000 7f9880c00640 1 --2- 172.21.15.27:0/1228125498 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f987c06bb80 0x7f987c0cd880 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.814+0000 7f9880c00640 1 --2- 172.21.15.27:0/1228125498 >> v2:172.21.15.93:3300/0 conn(0x7f987c0cddc0 0x7f987c0b0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.814+0000 7f9880c00640 1 -- 172.21.15.27:0/1228125498 >> 172.21.15.27:0/1228125498 conn(0x7f987c0b2a10 msgr2=0x7f987c0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:11.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.814+0000 7f9880c00640 1 -- 172.21.15.27:0/1228125498 shutdown_connections 2024-01-15T18:06:11.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.814+0000 7f9880c00640 1 -- 172.21.15.27:0/1228125498 wait complete. 2024-01-15T18:06:11.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.814+0000 7f9880c00640 1 Processor -- start 2024-01-15T18:06:11.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.814+0000 7f9880c00640 1 -- start start 2024-01-15T18:06:11.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.814+0000 7f9880c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f987c06bb80 0x7f987c0cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.814+0000 7f9880c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987c0b2390 0x7f987c15acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.814+0000 7f9880c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f987c0cddc0 0x7f987c15b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.814+0000 7f9880c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987c15c040 con 0x7f987c0b2390 2024-01-15T18:06:11.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.814+0000 7f9880c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987c1547b0 con 0x7f987c0cddc0 2024-01-15T18:06:11.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.814+0000 7f9880c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987c154a90 con 0x7f987c06bb80 2024-01-15T18:06:11.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.814+0000 7f9879400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987c0b2390 0x7f987c15acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:11.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.814+0000 7f9879400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987c0b2390 0x7f987c15acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56766/0 (socket says 172.21.15.27:56766) 2024-01-15T18:06:11.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.814+0000 7f9879400640 1 -- 172.21.15.27:0/3842613715 learned_addr learned my addr 172.21.15.27:0/3842613715 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:11.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.814+0000 7f9879e00640 1 --2- 172.21.15.27:0/3842613715 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f987c06bb80 0x7f987c0cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:11.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.814+0000 7f987a800640 1 --2- 172.21.15.27:0/3842613715 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f987c0cddc0 0x7f987c15b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:11.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.814+0000 7f9879400640 1 -- 172.21.15.27:0/3842613715 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f987c06bb80 msgr2=0x7f987c0cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:11.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.814+0000 7f9879400640 1 --2- 172.21.15.27:0/3842613715 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f987c06bb80 0x7f987c0cc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.814+0000 7f9879400640 1 -- 172.21.15.27:0/3842613715 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f987c0cddc0 msgr2=0x7f987c15b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:11.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.814+0000 7f9879400640 1 --2- 172.21.15.27:0/3842613715 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f987c0cddc0 0x7f987c15b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:11.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.814+0000 7f9879400640 1 -- 172.21.15.27:0/3842613715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9864009420 con 0x7f987c0b2390 2024-01-15T18:06:11.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.818+0000 7f9879400640 1 --2- 172.21.15.27:0/3842613715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987c0b2390 0x7f987c15acf0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f98640ad040 tx=0x7f986401ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:11.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.818+0000 7f9872000640 1 -- 172.21.15.27:0/3842613715 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98640bf610 con 0x7f987c0b2390 2024-01-15T18:06:11.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.818+0000 7f9872000640 1 -- 172.21.15.27:0/3842613715 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98640bf7b0 con 0x7f987c0b2390 2024-01-15T18:06:11.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.818+0000 7f9872000640 1 -- 172.21.15.27:0/3842613715 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98640bfb60 con 0x7f987c0b2390 2024-01-15T18:06:11.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.818+0000 7f9880c00640 1 -- 172.21.15.27:0/3842613715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f987c154d70 con 0x7f987c0b2390 2024-01-15T18:06:11.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.818+0000 7f9880c00640 1 -- 172.21.15.27:0/3842613715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f987c1562c0 con 0x7f987c0b2390 2024-01-15T18:06:11.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.818+0000 7f9880c00640 1 -- 172.21.15.27:0/3842613715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f987c0d07c0 con 0x7f987c0b2390 2024-01-15T18:06:11.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.818+0000 7f9872000640 1 -- 172.21.15.27:0/3842613715 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f98640b5400 con 0x7f987c0b2390 2024-01-15T18:06:11.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.818+0000 7f9872000640 1 --2- 172.21.15.27:0/3842613715 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f98540668f0 0x7f9854068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:11.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.818+0000 7f9872000640 1 -- 172.21.15.27:0/3842613715 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f986411f190 con 0x7f987c0b2390 2024-01-15T18:06:11.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.822+0000 7f9879e00640 1 --2- 172.21.15.27:0/3842613715 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f98540668f0 0x7f9854068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:11.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.822+0000 7f9879e00640 1 --2- 172.21.15.27:0/3842613715 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f98540668f0 0x7f9854068db0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f986c0011f0 tx=0x7f986c007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:11.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:11.822+0000 7f9872000640 1 -- 172.21.15.27:0/3842613715 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f98640ec630 con 0x7f987c0b2390 2024-01-15T18:06:12.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:12.014+0000 7f9880c00640 1 -- 172.21.15.27:0/3842613715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f987c154bd0 con 0x7f987c0b2390 2024-01-15T18:06:12.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:12.014+0000 7f9872000640 1 -- 172.21.15.27:0/3842613715 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7f987c154bd0 con 0x7f987c0b2390 2024-01-15T18:06:12.022 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:06:12.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:12.018+0000 7f9880c00640 1 -- 172.21.15.27:0/3842613715 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f98540668f0 msgr2=0x7f9854068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:12.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:12.018+0000 7f9880c00640 1 --2- 172.21.15.27:0/3842613715 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f98540668f0 0x7f9854068db0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f986c0011f0 tx=0x7f986c007d40 comp rx=0 tx=0).stop 2024-01-15T18:06:12.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:12.018+0000 7f9880c00640 1 -- 172.21.15.27:0/3842613715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987c0b2390 msgr2=0x7f987c15acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:12.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:12.018+0000 7f9880c00640 1 --2- 172.21.15.27:0/3842613715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987c0b2390 0x7f987c15acf0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f98640ad040 tx=0x7f986401ac90 comp rx=0 tx=0).stop 2024-01-15T18:06:12.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:12.018+0000 7f9880c00640 1 -- 172.21.15.27:0/3842613715 shutdown_connections 2024-01-15T18:06:12.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:12.018+0000 7f9880c00640 1 --2- 172.21.15.27:0/3842613715 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f98540668f0 0x7f9854068db0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:12.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:12.018+0000 7f9880c00640 1 --2- 172.21.15.27:0/3842613715 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f987c0cddc0 0x7f987c15b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:12.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:12.018+0000 7f9880c00640 1 --2- 172.21.15.27:0/3842613715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987c0b2390 0x7f987c15acf0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:12.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:12.018+0000 7f9880c00640 1 --2- 172.21.15.27:0/3842613715 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f987c06bb80 0x7f987c0cc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:12.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:12.018+0000 7f9880c00640 1 -- 172.21.15.27:0/3842613715 >> 172.21.15.27:0/3842613715 conn(0x7f987c0b2a10 msgr2=0x7f987c0af150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:12.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:12.018+0000 7f9880c00640 1 -- 172.21.15.27:0/3842613715 shutdown_connections 2024-01-15T18:06:12.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:12.018+0000 7f9880c00640 1 -- 172.21.15.27:0/3842613715 wait complete. 2024-01-15T18:06:12.044 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:06:12.044 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:06:12.044 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-01-15T18:06:12.044 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-15T18:06:12.044 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-15T18:06:12.044 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:06:12.045 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:06:12.045 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:06:12.045 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:06:12.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:12.045 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-01-15T18:06:12.055 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:06:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:06:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:06:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:06:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-15T18:06:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:06:12.055 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:06:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:06:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-01-15T18:06:12.055 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:06:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:06:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-01-15T18:06:12.055 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:06:12.055 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-01-15T18:06:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-01-15T18:06:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-01-15T18:06:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42387sec preferred_lft 42387sec 2024-01-15T18:06:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-01-15T18:06:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:06:12.056 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:06:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-01-15T18:06:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-01-15T18:06:12.056 INFO:teuthology.orchestra.run.smithi027.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-01-15T18:06:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 02:42:65:20:14:72 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:06:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-01-15T18:06:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:06:12.056 INFO:teuthology.orchestra.run.smithi027.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:06:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-01-15T18:06:12.056 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:06:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:06:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-01-15T18:06:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:06:12.057 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:06:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:06:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-01-15T18:06:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:06:12.057 INFO:teuthology.orchestra.run.smithi027.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:06:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:06:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-01-15T18:06:12.057 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:06:12.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:12.057 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:06:12.115 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-01-15T18:06:12.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:12.118 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-01-15T18:06:12.167 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-01-15T18:06:12.167 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:06:12.167 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-01-15T18:06:12.168 INFO:teuthology.orchestra.run.smithi027.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-01-15T18:06:12.168 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-01-15T18:06:12.168 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:06:12.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:12.168 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-15T18:06:12.168 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-15T18:06:12.168 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-15T18:06:12.168 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-15T18:06:12.168 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-15T18:06:12.168 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-15T18:06:12.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:12.259 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-15T18:06:12.269 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-15T18:06:12.269 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-15T18:06:12.269 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:06:12.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:12.269 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-01-15T18:06:12.329 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-15T18:06:12.330 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:06:12.330 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-15T18:06:12.378 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:06:12.378 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:06:12.378 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:06:12.429 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-01-15T18:06:12.490 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:06:12.547 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-01-15T18:06:12.547 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-01-15T18:06:12.547 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-01-15T18:06:12.548 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-01-15T18:06:12.548 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-01-15T18:06:12.548 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-01-15T18:06:12.548 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-01-15T18:06:12.548 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-01-15T18:06:12.548 INFO:teuthology.orchestra.run.smithi027.stdout:/run/lock 2024-01-15T18:06:12.548 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-01-15T18:06:12.548 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-01-15T18:06:12.548 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-01-15T18:06:12.548 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:06:12.548 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-01-15T18:06:12.548 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-01-15T18:06:12.548 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-01-15T18:06:12.548 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-01-15T18:06:12.549 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/fs/nfsd 2024-01-15T18:06:12.549 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-01-15T18:06:12.549 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-01-15T18:06:12.549 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:06:12.549 INFO:teuthology.orchestra.run.smithi027.stdout:/run/rpc_pipefs 2024-01-15T18:06:12.549 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-01-15T18:06:12.549 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-01-15T18:06:12.549 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:06:12.549 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:06:12.549 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:06:12.549 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:06:12.549 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-01-15T18:06:12.549 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:06:12.549 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:06:12.549 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:06:12.550 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:06:12.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:12.550 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-01-15T18:06:12.601 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:06:12.601 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-15T18:06:12.644 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-01-15T18:06:12.665 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:06:12.715 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-01-15T18:06:12.715 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-01-15T18:06:12.715 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-01-15T18:06:12.716 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-01-15T18:06:12.716 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-01-15T18:06:12.716 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-01-15T18:06:12.716 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-01-15T18:06:12.716 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-01-15T18:06:12.716 INFO:teuthology.orchestra.run.smithi027.stdout:/run/lock 2024-01-15T18:06:12.716 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-01-15T18:06:12.716 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-01-15T18:06:12.716 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-01-15T18:06:12.716 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:06:12.716 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-01-15T18:06:12.716 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-01-15T18:06:12.716 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-01-15T18:06:12.716 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-01-15T18:06:12.717 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/fs/nfsd 2024-01-15T18:06:12.717 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-01-15T18:06:12.717 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-01-15T18:06:12.717 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:06:12.717 INFO:teuthology.orchestra.run.smithi027.stdout:/run/rpc_pipefs 2024-01-15T18:06:12.717 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-01-15T18:06:12.717 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-01-15T18:06:12.717 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:06:12.717 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:06:12.717 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:06:12.717 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:06:12.717 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-01-15T18:06:12.717 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:06:12.717 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:06:12.717 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:06:12.718 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:06:12.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:12.718 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-01-15T18:06:12.740 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:2024-01-15T18:06:12.734+0000 7f7023648580 -1 init, newargv = 0x555f39029360 newargc=15 2024-01-15T18:06:12.741 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[36928]: starting ceph client 2024-01-15T18:06:12.747 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[36928]: starting fuse 2024-01-15T18:06:12.750 INFO:teuthology.orchestra.run.smithi027.stdout:43 2024-01-15T18:06:12.750 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-01-15T18:06:12.750 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:06:12.750 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-01-15T18:06:12.750 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-01-15T18:06:12.750 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-01-15T18:06:12.750 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-01-15T18:06:12.750 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:06:12.750 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-01-15T18:06:12.750 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-15T18:06:12.750 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-01-15T18:06:12.751 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-15T18:06:12.751 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:06:12.751 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-01-15T18:06:12.751 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-01-15T18:06:12.751 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-01-15T18:06:12.751 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-01-15T18:06:12.751 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:06:12.751 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-01-15T18:06:12.751 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:06:12.751 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:06:12.751 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:06:12.751 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-01-15T18:06:12.751 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-01-15T18:06:12.751 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-01-15T18:06:12.751 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:06:12.751 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:06:12.751 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.0")) 2024-01-15T18:06:12.751 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-01-15T18:06:12.877 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.0.36928.asok 2024-01-15T18:06:12.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:06:12.888 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.36928.asok 2024-01-15T18:06:12.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:12.888 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.36928.asok status 2024-01-15T18:06:13.065 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:06:13.065 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-01-15T18:06:13.065 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:06:13.065 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:06:13.065 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "0", 2024-01-15T18:06:13.065 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-01-15T18:06:13.065 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-15T18:06:13.066 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "36928", 2024-01-15T18:06:13.066 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-01-15T18:06:13.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:06:13.066 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-01-15T18:06:13.066 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-01-15T18:06:13.066 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 5457, 2024-01-15T18:06:13.066 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-01-15T18:06:13.066 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-01-15T18:06:13.066 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-01-15T18:06:13.066 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 5457 2024-01-15T18:06:13.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:06:13.066 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-01-15T18:06:13.066 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-01-15T18:06:13.067 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:06:13.067 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3730295433 2024-01-15T18:06:13.067 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:13.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:06:13.067 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-01-15T18:06:13.067 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-01-15T18:06:13.067 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:06:13.067 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3730295433 2024-01-15T18:06:13.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:06:13.067 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.5457 192.168.0.1:0/3730295433", 2024-01-15T18:06:13.067 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.1:0/3730295433", 2024-01-15T18:06:13.067 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-01-15T18:06:13.067 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 17, 2024-01-15T18:06:13.068 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 34, 2024-01-15T18:06:13.068 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:06:13.068 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-01-15T18:06:13.068 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-01-15T18:06:13.068 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:06:13.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:13.080 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-01-15T18:06:13.080 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-15T18:06:13.090 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-01-15T18:06:13.090 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-15T18:06:13.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:13.090 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:06:13.154 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:06:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.338+0000 7f3099e00640 1 Processor -- start 2024-01-15T18:06:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.338+0000 7f3099e00640 1 -- start start 2024-01-15T18:06:13.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.338+0000 7f3099e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30940b2390 0x7f30940cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:13.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.338+0000 7f3099e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f30940cdf90 0x7f30940b0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:13.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.338+0000 7f3099e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30940b2770 con 0x7f309406bb80 2024-01-15T18:06:13.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.338+0000 7f3099e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f309406c060 con 0x7f30940cdf90 2024-01-15T18:06:13.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.338+0000 7f3099e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309406c340 con 0x7f30940b2390 2024-01-15T18:06:13.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.338+0000 7f3093400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f30940cdf90 0x7f30940b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:13.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.338+0000 7f3092000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f309406bb80 0x7f30940cd880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49706/0 (socket says 172.21.15.27:49706) 2024-01-15T18:06:13.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.338+0000 7f3092000640 1 -- 172.21.15.27:0/2213970576 learned_addr learned my addr 172.21.15.27:0/2213970576 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:06:13.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.338+0000 7f3092a00640 1 --2- 172.21.15.27:0/2213970576 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30940b2390 0x7f30940cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:13.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.338+0000 7f3093400640 1 -- 172.21.15.27:0/2213970576 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30940b2390 msgr2=0x7f30940cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:13.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.338+0000 7f3093400640 1 --2- 172.21.15.27:0/2213970576 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30940b2390 0x7f30940cf780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:13.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.338+0000 7f3093400640 1 -- 172.21.15.27:0/2213970576 >> v1:172.21.15.27:6789/0 conn(0x7f309406bb80 legacy=0x7f30940cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:13.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.338+0000 7f3093400640 1 -- 172.21.15.27:0/2213970576 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309406c620 con 0x7f30940cdf90 2024-01-15T18:06:13.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.338+0000 7f3091600640 1 -- 172.21.15.27:0/2213970576 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506475945 0 0) 0x7f30940b2770 con 0x7f309406bb80 2024-01-15T18:06:13.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.342+0000 7f3093400640 1 --2- 172.21.15.27:0/2213970576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30940cdf90 0x7f30940b0880 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f308c009d90 tx=0x7f308c01c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2248efcf3bcce54d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:13.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.342+0000 7f3091600640 1 -- 172.21.15.27:0/2213970576 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c0a59e0 con 0x7f30940cdf90 2024-01-15T18:06:13.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.342+0000 7f3091600640 1 -- 172.21.15.27:0/2213970576 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f308c0a5b40 con 0x7f30940cdf90 2024-01-15T18:06:13.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.342+0000 7f3091600640 1 -- 172.21.15.27:0/2213970576 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308c0ad440 con 0x7f30940cdf90 2024-01-15T18:06:13.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.342+0000 7f3099e00640 1 -- 172.21.15.27:0/2213970576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30940cdf90 msgr2=0x7f30940b0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:13.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.342+0000 7f3099e00640 1 --2- 172.21.15.27:0/2213970576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30940cdf90 0x7f30940b0880 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f308c009d90 tx=0x7f308c01c920 comp rx=0 tx=0).stop 2024-01-15T18:06:13.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.342+0000 7f3099e00640 1 -- 172.21.15.27:0/2213970576 shutdown_connections 2024-01-15T18:06:13.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.342+0000 7f3099e00640 1 --2- 172.21.15.27:0/2213970576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30940cdf90 0x7f30940b0880 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:13.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.342+0000 7f3099e00640 1 --2- 172.21.15.27:0/2213970576 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30940b2390 0x7f30940cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:13.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.342+0000 7f3099e00640 1 -- 172.21.15.27:0/2213970576 >> 172.21.15.27:0/2213970576 conn(0x7f30940b2a10 msgr2=0x7f30940b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:13.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.342+0000 7f3099e00640 1 -- 172.21.15.27:0/2213970576 shutdown_connections 2024-01-15T18:06:13.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.342+0000 7f3099e00640 1 -- 172.21.15.27:0/2213970576 wait complete. 2024-01-15T18:06:13.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.346+0000 7f3099e00640 1 Processor -- start 2024-01-15T18:06:13.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.346+0000 7f3099e00640 1 -- start start 2024-01-15T18:06:13.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.346+0000 7f3099e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f309406bb80 0x7f30940c4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:13.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.346+0000 7f3099e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30940b2390 0x7f30940c8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:13.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.346+0000 7f3099e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30940cdf90 0x7f30940c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:13.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.346+0000 7f3099e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30940c6000 con 0x7f309406bb80 2024-01-15T18:06:13.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.346+0000 7f3099e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30940c53f0 con 0x7f30940cdf90 2024-01-15T18:06:13.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.346+0000 7f3099e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30940c56d0 con 0x7f30940b2390 2024-01-15T18:06:13.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.346+0000 7f3092a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f309406bb80 0x7f30940c4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:13.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.346+0000 7f3092a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f309406bb80 0x7f30940c4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56796/0 (socket says 172.21.15.27:56796) 2024-01-15T18:06:13.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.346+0000 7f3092a00640 1 -- 172.21.15.27:0/3775229822 learned_addr learned my addr 172.21.15.27:0/3775229822 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:13.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.346+0000 7f3092a00640 1 -- 172.21.15.27:0/3775229822 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30940b2390 msgr2=0x7f30940c8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:13.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.346+0000 7f3092a00640 1 --2- 172.21.15.27:0/3775229822 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30940b2390 0x7f30940c8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:13.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.346+0000 7f3092a00640 1 -- 172.21.15.27:0/3775229822 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30940cdf90 msgr2=0x7f30940c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:13.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.346+0000 7f3092a00640 1 --2- 172.21.15.27:0/3775229822 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30940cdf90 0x7f30940c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:13.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.346+0000 7f3092a00640 1 -- 172.21.15.27:0/3775229822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308c009a70 con 0x7f309406bb80 2024-01-15T18:06:13.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.346+0000 7f3092a00640 1 --2- 172.21.15.27:0/3775229822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f309406bb80 0x7f30940c4d80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f308400d5f0 tx=0x7f308400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:13.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.346+0000 7f308b400640 1 -- 172.21.15.27:0/3775229822 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30840125f0 con 0x7f309406bb80 2024-01-15T18:06:13.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.346+0000 7f308b400640 1 -- 172.21.15.27:0/3775229822 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3084012750 con 0x7f309406bb80 2024-01-15T18:06:13.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.346+0000 7f3099e00640 1 -- 172.21.15.27:0/3775229822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30940c59b0 con 0x7f309406bb80 2024-01-15T18:06:13.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.346+0000 7f308b400640 1 -- 172.21.15.27:0/3775229822 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3084012ae0 con 0x7f309406bb80 2024-01-15T18:06:13.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.346+0000 7f3099e00640 1 -- 172.21.15.27:0/3775229822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30940c8ac0 con 0x7f309406bb80 2024-01-15T18:06:13.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.346+0000 7f3099e00640 1 -- 172.21.15.27:0/3775229822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30940d07c0 con 0x7f309406bb80 2024-01-15T18:06:13.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.350+0000 7f308b400640 1 -- 172.21.15.27:0/3775229822 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f3084012c40 con 0x7f309406bb80 2024-01-15T18:06:13.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.350+0000 7f308b400640 1 --2- 172.21.15.27:0/3775229822 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f306c0668f0 0x7f306c068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:13.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.350+0000 7f308b400640 1 -- 172.21.15.27:0/3775229822 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f30840848f0 con 0x7f309406bb80 2024-01-15T18:06:13.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.350+0000 7f3092000640 1 --2- 172.21.15.27:0/3775229822 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f306c0668f0 0x7f306c068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:13.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.350+0000 7f3092000640 1 --2- 172.21.15.27:0/3775229822 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f306c0668f0 0x7f306c068db0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f30940c7a20 tx=0x7f307c007970 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:13.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.354+0000 7f308b400640 1 -- 172.21.15.27:0/3775229822 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3084051b20 con 0x7f309406bb80 2024-01-15T18:06:13.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.550+0000 7f3099e00640 1 -- 172.21.15.27:0/3775229822 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f309404e850 con 0x7f309406bb80 2024-01-15T18:06:13.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.550+0000 7f308b400640 1 -- 172.21.15.27:0/3775229822 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3084007380 con 0x7f309406bb80 2024-01-15T18:06:13.555 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:06:13.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.550+0000 7f3099e00640 1 -- 172.21.15.27:0/3775229822 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f306c0668f0 msgr2=0x7f306c068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:13.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.550+0000 7f3099e00640 1 --2- 172.21.15.27:0/3775229822 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f306c0668f0 0x7f306c068db0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f30940c7a20 tx=0x7f307c007970 comp rx=0 tx=0).stop 2024-01-15T18:06:13.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.550+0000 7f3099e00640 1 -- 172.21.15.27:0/3775229822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f309406bb80 msgr2=0x7f30940c4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:13.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.550+0000 7f3099e00640 1 --2- 172.21.15.27:0/3775229822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f309406bb80 0x7f30940c4d80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f308400d5f0 tx=0x7f308400dab0 comp rx=0 tx=0).stop 2024-01-15T18:06:13.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.550+0000 7f3099e00640 1 -- 172.21.15.27:0/3775229822 shutdown_connections 2024-01-15T18:06:13.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.550+0000 7f3099e00640 1 --2- 172.21.15.27:0/3775229822 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f306c0668f0 0x7f306c068db0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:13.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.550+0000 7f3099e00640 1 --2- 172.21.15.27:0/3775229822 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30940cdf90 0x7f30940c8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:13.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.550+0000 7f3099e00640 1 --2- 172.21.15.27:0/3775229822 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30940b2390 0x7f30940c8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:13.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.550+0000 7f3099e00640 1 --2- 172.21.15.27:0/3775229822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f309406bb80 0x7f30940c4d80 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:13.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.550+0000 7f3099e00640 1 -- 172.21.15.27:0/3775229822 >> 172.21.15.27:0/3775229822 conn(0x7f30940b2a10 msgr2=0x7f30940733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:13.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.550+0000 7f3099e00640 1 -- 172.21.15.27:0/3775229822 shutdown_connections 2024-01-15T18:06:13.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:13.550+0000 7f3099e00640 1 -- 172.21.15.27:0/3775229822 wait complete. 2024-01-15T18:06:13.578 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:06:13.578 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:06:13.578 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi093.front.sepia.ceph.com 2024-01-15T18:06:13.578 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-15T18:06:13.579 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-15T18:06:13.579 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:06:13.579 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:06:13.579 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:06:13.579 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:06:13.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:13.579 DEBUG:teuthology.orchestra.run.smithi093:> ip addr 2024-01-15T18:06:13.590 INFO:teuthology.orchestra.run.smithi093.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:06:13.590 INFO:teuthology.orchestra.run.smithi093.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:06:13.590 INFO:teuthology.orchestra.run.smithi093.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:06:13.590 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:06:13.590 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 ::1/128 scope host 2024-01-15T18:06:13.590 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:06:13.591 INFO:teuthology.orchestra.run.smithi093.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:06:13.591 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:d8:da:1c brd ff:ff:ff:ff:ff:ff 2024-01-15T18:06:13.591 INFO:teuthology.orchestra.run.smithi093.stdout: altname enp7s0f0 2024-01-15T18:06:13.591 INFO:teuthology.orchestra.run.smithi093.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:06:13.591 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:d8:da:1d brd ff:ff:ff:ff:ff:ff 2024-01-15T18:06:13.591 INFO:teuthology.orchestra.run.smithi093.stdout: altname enp7s0f1 2024-01-15T18:06:13.591 INFO:teuthology.orchestra.run.smithi093.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:06:13.591 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:8f:d6:34 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:06:13.591 INFO:teuthology.orchestra.run.smithi093.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:06:13.591 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:8f:d6:35 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:06:13.591 INFO:teuthology.orchestra.run.smithi093.stdout: inet 172.21.15.93/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-01-15T18:06:13.592 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft 42365sec preferred_lft 42365sec 2024-01-15T18:06:13.592 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::ec4:7aff:fe8f:d635/64 scope link 2024-01-15T18:06:13.592 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:06:13.592 INFO:teuthology.orchestra.run.smithi093.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-01-15T18:06:13.592 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 02:42:2a:61:fc:7d brd ff:ff:ff:ff:ff:ff 2024-01-15T18:06:13.592 INFO:teuthology.orchestra.run.smithi093.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-01-15T18:06:13.592 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:06:13.592 INFO:teuthology.orchestra.run.smithi093.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:06:13.592 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-01-15T18:06:13.593 INFO:teuthology.orchestra.run.smithi093.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:06:13.593 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:06:13.593 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-01-15T18:06:13.593 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:06:13.593 INFO:teuthology.orchestra.run.smithi093.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:06:13.593 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:06:13.593 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-01-15T18:06:13.593 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:06:13.593 INFO:teuthology.orchestra.run.smithi093.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:06:13.593 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:06:13.593 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-01-15T18:06:13.594 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:06:13.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:13.595 DEBUG:teuthology.orchestra.run.smithi093:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:06:13.657 INFO:teuthology.orchestra.run.smithi093.stdout:1 2024-01-15T18:06:13.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:13.660 DEBUG:teuthology.orchestra.run.smithi093:> route 2024-01-15T18:06:13.713 INFO:teuthology.orchestra.run.smithi093.stdout:Kernel IP routing table 2024-01-15T18:06:13.713 INFO:teuthology.orchestra.run.smithi093.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:06:13.713 INFO:teuthology.orchestra.run.smithi093.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-01-15T18:06:13.713 INFO:teuthology.orchestra.run.smithi093.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T18:06:13.713 INFO:teuthology.orchestra.run.smithi093.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-01-15T18:06:13.713 INFO:teuthology.orchestra.run.smithi093.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T18:06:13.713 INFO:teuthology.orchestra.run.smithi093.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:06:13.714 INFO:teuthology.orchestra.run.smithi093.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:06:13.714 INFO:teuthology.orchestra.run.smithi093.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:06:13.714 INFO:teuthology.orchestra.run.smithi093.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:06:13.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:13.714 DEBUG:teuthology.orchestra.run.smithi093:> (cd / && exec bash -c ' 2024-01-15T18:06:13.714 DEBUG:teuthology.orchestra.run.smithi093:> set -e 2024-01-15T18:06:13.714 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T18:06:13.715 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T18:06:13.715 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T18:06:13.715 DEBUG:teuthology.orchestra.run.smithi093:> ') 2024-01-15T18:06:13.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:13.806 DEBUG:teuthology.orchestra.run.smithi093:> ip netns list 2024-01-15T18:06:13.815 INFO:teuthology.orchestra.run.smithi093.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-15T18:06:13.815 INFO:teuthology.orchestra.run.smithi093.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-15T18:06:13.816 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:06:13.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:13.816 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip link set brx.0 up 2024-01-15T18:06:13.878 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-15T18:06:13.879 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:06:13.879 DEBUG:teuthology.orchestra.run.smithi093:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-15T18:06:13.930 INFO:teuthology.orchestra.run.smithi093.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:06:13.930 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:06:13.930 DEBUG:teuthology.orchestra.run.smithi093:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:06:13.981 DEBUG:teuthology.orchestra.run.smithi093:> sudo modprobe fuse 2024-01-15T18:06:14.042 DEBUG:teuthology.orchestra.run.smithi093:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:06:14.100 INFO:teuthology.orchestra.run.smithi093.stdout:/sys 2024-01-15T18:06:14.100 INFO:teuthology.orchestra.run.smithi093.stdout:/proc 2024-01-15T18:06:14.100 INFO:teuthology.orchestra.run.smithi093.stdout:/dev 2024-01-15T18:06:14.100 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/pts 2024-01-15T18:06:14.100 INFO:teuthology.orchestra.run.smithi093.stdout:/run 2024-01-15T18:06:14.100 INFO:teuthology.orchestra.run.smithi093.stdout:/ 2024-01-15T18:06:14.100 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/security 2024-01-15T18:06:14.100 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/shm 2024-01-15T18:06:14.100 INFO:teuthology.orchestra.run.smithi093.stdout:/run/lock 2024-01-15T18:06:14.101 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/cgroup 2024-01-15T18:06:14.101 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/pstore 2024-01-15T18:06:14.101 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/bpf 2024-01-15T18:06:14.101 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:06:14.101 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/hugepages 2024-01-15T18:06:14.101 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/mqueue 2024-01-15T18:06:14.101 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/debug 2024-01-15T18:06:14.101 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/tracing 2024-01-15T18:06:14.101 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/fs/nfsd 2024-01-15T18:06:14.101 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/fuse/connections 2024-01-15T18:06:14.101 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/config 2024-01-15T18:06:14.102 INFO:teuthology.orchestra.run.smithi093.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:06:14.102 INFO:teuthology.orchestra.run.smithi093.stdout:/run/rpc_pipefs 2024-01-15T18:06:14.102 INFO:teuthology.orchestra.run.smithi093.stdout:/run/user/1000 2024-01-15T18:06:14.102 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph 2024-01-15T18:06:14.102 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:06:14.102 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:06:14.102 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:06:14.102 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:06:14.102 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns 2024-01-15T18:06:14.102 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:06:14.102 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:06:14.103 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:06:14.103 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:06:14.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:14.104 DEBUG:teuthology.orchestra.run.smithi093:> ls /sys/fs/fuse/connections 2024-01-15T18:06:14.154 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:06:14.154 DEBUG:teuthology.orchestra.run.smithi093:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-15T18:06:14.201 DEBUG:teuthology.orchestra.run.smithi093:> sudo modprobe fuse 2024-01-15T18:06:14.223 DEBUG:teuthology.orchestra.run.smithi093:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:06:14.279 INFO:teuthology.orchestra.run.smithi093.stdout:/sys 2024-01-15T18:06:14.279 INFO:teuthology.orchestra.run.smithi093.stdout:/proc 2024-01-15T18:06:14.280 INFO:teuthology.orchestra.run.smithi093.stdout:/dev 2024-01-15T18:06:14.280 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/pts 2024-01-15T18:06:14.280 INFO:teuthology.orchestra.run.smithi093.stdout:/run 2024-01-15T18:06:14.280 INFO:teuthology.orchestra.run.smithi093.stdout:/ 2024-01-15T18:06:14.280 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/security 2024-01-15T18:06:14.280 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/shm 2024-01-15T18:06:14.280 INFO:teuthology.orchestra.run.smithi093.stdout:/run/lock 2024-01-15T18:06:14.280 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/cgroup 2024-01-15T18:06:14.280 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/pstore 2024-01-15T18:06:14.281 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/bpf 2024-01-15T18:06:14.281 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:06:14.281 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/hugepages 2024-01-15T18:06:14.281 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/mqueue 2024-01-15T18:06:14.281 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/debug 2024-01-15T18:06:14.281 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/tracing 2024-01-15T18:06:14.281 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/fs/nfsd 2024-01-15T18:06:14.281 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/fuse/connections 2024-01-15T18:06:14.281 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/config 2024-01-15T18:06:14.281 INFO:teuthology.orchestra.run.smithi093.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:06:14.281 INFO:teuthology.orchestra.run.smithi093.stdout:/run/rpc_pipefs 2024-01-15T18:06:14.282 INFO:teuthology.orchestra.run.smithi093.stdout:/run/user/1000 2024-01-15T18:06:14.282 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph 2024-01-15T18:06:14.282 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:06:14.282 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:06:14.282 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:06:14.282 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:06:14.282 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns 2024-01-15T18:06:14.282 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:06:14.282 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:06:14.282 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:06:14.283 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:06:14.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:14.284 DEBUG:teuthology.orchestra.run.smithi093:> ls /sys/fs/fuse/connections 2024-01-15T18:06:14.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:2024-01-15T18:06:14.289+0000 7f2cd6dc6580 -1 init, newargv = 0x5649008ed860 newargc=15 2024-01-15T18:06:14.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:ceph-fuse[33287]: starting ceph client 2024-01-15T18:06:14.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:ceph-fuse[33287]: starting fuse 2024-01-15T18:06:14.305 INFO:teuthology.orchestra.run.smithi093.stdout:43 2024-01-15T18:06:14.305 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-01-15T18:06:14.306 DEBUG:teuthology.orchestra.run.smithi093:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:06:14.306 DEBUG:teuthology.orchestra.run.smithi093:> import glob 2024-01-15T18:06:14.306 DEBUG:teuthology.orchestra.run.smithi093:> import re 2024-01-15T18:06:14.306 DEBUG:teuthology.orchestra.run.smithi093:> import os 2024-01-15T18:06:14.306 DEBUG:teuthology.orchestra.run.smithi093:> import subprocess 2024-01-15T18:06:14.306 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:06:14.306 DEBUG:teuthology.orchestra.run.smithi093:> def _find_admin_socket(client_name): 2024-01-15T18:06:14.306 DEBUG:teuthology.orchestra.run.smithi093:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-15T18:06:14.306 DEBUG:teuthology.orchestra.run.smithi093:> files = glob.glob(asok_path) 2024-01-15T18:06:14.306 DEBUG:teuthology.orchestra.run.smithi093:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-15T18:06:14.307 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:06:14.307 DEBUG:teuthology.orchestra.run.smithi093:> # Given a non-glob path, it better be there 2024-01-15T18:06:14.307 DEBUG:teuthology.orchestra.run.smithi093:> if "*" not in asok_path: 2024-01-15T18:06:14.307 DEBUG:teuthology.orchestra.run.smithi093:> assert(len(files) == 1) 2024-01-15T18:06:14.307 DEBUG:teuthology.orchestra.run.smithi093:> return files[0] 2024-01-15T18:06:14.307 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:06:14.307 DEBUG:teuthology.orchestra.run.smithi093:> for f in files: 2024-01-15T18:06:14.307 DEBUG:teuthology.orchestra.run.smithi093:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:06:14.307 DEBUG:teuthology.orchestra.run.smithi093:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:06:14.307 DEBUG:teuthology.orchestra.run.smithi093:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:06:14.307 DEBUG:teuthology.orchestra.run.smithi093:> contents = proc_f.read() 2024-01-15T18:06:14.308 DEBUG:teuthology.orchestra.run.smithi093:> if mountpoint in contents: 2024-01-15T18:06:14.308 DEBUG:teuthology.orchestra.run.smithi093:> return f 2024-01-15T18:06:14.308 DEBUG:teuthology.orchestra.run.smithi093:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:06:14.308 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:06:14.308 DEBUG:teuthology.orchestra.run.smithi093:> print(_find_admin_socket("client.1")) 2024-01-15T18:06:14.308 DEBUG:teuthology.orchestra.run.smithi093:> ' 2024-01-15T18:06:14.430 INFO:teuthology.orchestra.run.smithi093.stdout:/var/run/ceph/ceph-client.1.33287.asok 2024-01-15T18:06:14.433 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:06:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:06:14.441 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.33287.asok 2024-01-15T18:06:14.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:14.442 DEBUG:teuthology.orchestra.run.smithi093:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.33287.asok status 2024-01-15T18:06:14.612 INFO:teuthology.orchestra.run.smithi093.stdout:{ 2024-01-15T18:06:14.612 INFO:teuthology.orchestra.run.smithi093.stdout: "metadata": { 2024-01-15T18:06:14.612 INFO:teuthology.orchestra.run.smithi093.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:06:14.612 INFO:teuthology.orchestra.run.smithi093.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:06:14.612 INFO:teuthology.orchestra.run.smithi093.stdout: "entity_id": "1", 2024-01-15T18:06:14.613 INFO:teuthology.orchestra.run.smithi093.stdout: "hostname": "smithi093", 2024-01-15T18:06:14.613 INFO:teuthology.orchestra.run.smithi093.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-15T18:06:14.613 INFO:teuthology.orchestra.run.smithi093.stdout: "pid": "33287", 2024-01-15T18:06:14.613 INFO:teuthology.orchestra.run.smithi093.stdout: "root": "/" 2024-01-15T18:06:14.613 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:06:14.613 INFO:teuthology.orchestra.run.smithi093.stdout: "dentry_count": 0, 2024-01-15T18:06:14.613 INFO:teuthology.orchestra.run.smithi093.stdout: "dentry_pinned_count": 0, 2024-01-15T18:06:14.613 INFO:teuthology.orchestra.run.smithi093.stdout: "id": 5425, 2024-01-15T18:06:14.613 INFO:teuthology.orchestra.run.smithi093.stdout: "inst": { 2024-01-15T18:06:14.613 INFO:teuthology.orchestra.run.smithi093.stdout: "name": { 2024-01-15T18:06:14.613 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "client", 2024-01-15T18:06:14.614 INFO:teuthology.orchestra.run.smithi093.stdout: "num": 5425 2024-01-15T18:06:14.614 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:06:14.614 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": { 2024-01-15T18:06:14.614 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "v1", 2024-01-15T18:06:14.614 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:06:14.614 INFO:teuthology.orchestra.run.smithi093.stdout: "nonce": 4164229478 2024-01-15T18:06:14.614 INFO:teuthology.orchestra.run.smithi093.stdout: } 2024-01-15T18:06:14.614 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:06:14.614 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": { 2024-01-15T18:06:14.614 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "v1", 2024-01-15T18:06:14.614 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:06:14.615 INFO:teuthology.orchestra.run.smithi093.stdout: "nonce": 4164229478 2024-01-15T18:06:14.615 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:06:14.615 INFO:teuthology.orchestra.run.smithi093.stdout: "inst_str": "client.5425 192.168.0.1:0/4164229478", 2024-01-15T18:06:14.615 INFO:teuthology.orchestra.run.smithi093.stdout: "addr_str": "192.168.0.1:0/4164229478", 2024-01-15T18:06:14.615 INFO:teuthology.orchestra.run.smithi093.stdout: "inode_count": 1, 2024-01-15T18:06:14.615 INFO:teuthology.orchestra.run.smithi093.stdout: "mds_epoch": 17, 2024-01-15T18:06:14.615 INFO:teuthology.orchestra.run.smithi093.stdout: "osd_epoch": 34, 2024-01-15T18:06:14.615 INFO:teuthology.orchestra.run.smithi093.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:06:14.615 INFO:teuthology.orchestra.run.smithi093.stdout: "blocklisted": false, 2024-01-15T18:06:14.615 INFO:teuthology.orchestra.run.smithi093.stdout: "fs_name": "cephfs" 2024-01-15T18:06:14.616 INFO:teuthology.orchestra.run.smithi093.stdout:} 2024-01-15T18:06:14.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:14.627 DEBUG:teuthology.orchestra.run.smithi093:> stat --file-system '--printf=%T 2024-01-15T18:06:14.627 DEBUG:teuthology.orchestra.run.smithi093:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-15T18:06:14.636 INFO:teuthology.orchestra.run.smithi093.stdout:fuseblk 2024-01-15T18:06:14.636 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-15T18:06:14.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:14.637 DEBUG:teuthology.orchestra.run.smithi093:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:06:14.697 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-15T18:06:14.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.850+0000 7f0550600640 1 Processor -- start 2024-01-15T18:06:14.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.850+0000 7f0550600640 1 -- start start 2024-01-15T18:06:14.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.850+0000 7f0550600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f054806c430 0x7f0548070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:14.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.850+0000 7f0550600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f05480ce760 0x7f05480b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:14.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.850+0000 7f0550600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05480ccfb0 con 0x7f05480b1db0 2024-01-15T18:06:14.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.850+0000 7f0550600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05480cd290 con 0x7f05480ce760 2024-01-15T18:06:14.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.850+0000 7f0550600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05480cd570 con 0x7f054806c430 2024-01-15T18:06:14.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.850+0000 7f054d200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f05480b1db0 0x7f05480ce050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57688/0 (socket says 172.21.15.27:57688) 2024-01-15T18:06:14.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.850+0000 7f054d200640 1 -- 172.21.15.27:0/1163736147 learned_addr learned my addr 172.21.15.27:0/1163736147 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:06:14.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.850+0000 7f054e600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f05480ce760 0x7f05480b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:14.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.850+0000 7f054dc00640 1 --2- 172.21.15.27:0/1163736147 >> v2:172.21.15.93:3300/0 conn(0x7f054806c430 0x7f0548070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:14.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.850+0000 7f054e600640 1 -- 172.21.15.27:0/1163736147 >> v2:172.21.15.93:3300/0 conn(0x7f054806c430 msgr2=0x7f0548070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:14.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.850+0000 7f054e600640 1 --2- 172.21.15.27:0/1163736147 >> v2:172.21.15.93:3300/0 conn(0x7f054806c430 0x7f0548070ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:14.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.850+0000 7f054e600640 1 -- 172.21.15.27:0/1163736147 >> v1:172.21.15.27:6789/0 conn(0x7f05480b1db0 legacy=0x7f05480ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:14.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.850+0000 7f054e600640 1 -- 172.21.15.27:0/1163736147 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05480cd850 con 0x7f05480ce760 2024-01-15T18:06:14.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.850+0000 7f0547e00640 1 -- 172.21.15.27:0/1163736147 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3330597748 0 0) 0x7f05480ccfb0 con 0x7f05480b1db0 2024-01-15T18:06:14.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.854+0000 7f054e600640 1 --2- 172.21.15.27:0/1163736147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05480ce760 0x7f05480b2240 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f0540009d90 tx=0x7f054001c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f69e047425fcd0b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:14.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.854+0000 7f0547e00640 1 -- 172.21.15.27:0/1163736147 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05400a58b0 con 0x7f05480ce760 2024-01-15T18:06:14.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.854+0000 7f0547e00640 1 -- 172.21.15.27:0/1163736147 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05400a5a10 con 0x7f05480ce760 2024-01-15T18:06:14.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.854+0000 7f0547e00640 1 -- 172.21.15.27:0/1163736147 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05400a5da0 con 0x7f05480ce760 2024-01-15T18:06:14.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.854+0000 7f0550600640 1 -- 172.21.15.27:0/1163736147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05480ce760 msgr2=0x7f05480b2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:14.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.854+0000 7f0550600640 1 --2- 172.21.15.27:0/1163736147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05480ce760 0x7f05480b2240 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f0540009d90 tx=0x7f054001c850 comp rx=0 tx=0).stop 2024-01-15T18:06:14.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.854+0000 7f0550600640 1 -- 172.21.15.27:0/1163736147 shutdown_connections 2024-01-15T18:06:14.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.854+0000 7f0550600640 1 --2- 172.21.15.27:0/1163736147 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05480ce760 0x7f05480b2240 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:14.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.854+0000 7f0550600640 1 --2- 172.21.15.27:0/1163736147 >> v2:172.21.15.93:3300/0 conn(0x7f054806c430 0x7f0548070ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:14.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.854+0000 7f0550600640 1 -- 172.21.15.27:0/1163736147 >> 172.21.15.27:0/1163736147 conn(0x7f05480cb700 msgr2=0x7f05480cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:14.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.858+0000 7f0550600640 1 -- 172.21.15.27:0/1163736147 shutdown_connections 2024-01-15T18:06:14.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.858+0000 7f0550600640 1 -- 172.21.15.27:0/1163736147 wait complete. 2024-01-15T18:06:14.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.858+0000 7f0550600640 1 Processor -- start 2024-01-15T18:06:14.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.858+0000 7f0550600640 1 -- start start 2024-01-15T18:06:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.858+0000 7f0550600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f054806c430 0x7f0548152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.858+0000 7f0550600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f05480b1db0 0x7f0548147670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.858+0000 7f0550600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05480ce760 0x7f054815c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.858+0000 7f0550600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054815be10 con 0x7f05480ce760 2024-01-15T18:06:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.858+0000 7f0550600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054815b200 con 0x7f054806c430 2024-01-15T18:06:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.858+0000 7f0550600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f054815b4e0 con 0x7f05480b1db0 2024-01-15T18:06:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.858+0000 7f054e600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05480ce760 0x7f054815c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.858+0000 7f054e600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05480ce760 0x7f054815c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54316/0 (socket says 172.21.15.27:54316) 2024-01-15T18:06:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.858+0000 7f054e600640 1 -- 172.21.15.27:0/1163529940 learned_addr learned my addr 172.21.15.27:0/1163529940 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.858+0000 7f054dc00640 1 --2- 172.21.15.27:0/1163529940 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f054806c430 0x7f0548152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.858+0000 7f054d200640 1 --2- 172.21.15.27:0/1163529940 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f05480b1db0 0x7f0548147670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.858+0000 7f054e600640 1 -- 172.21.15.27:0/1163529940 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f05480b1db0 msgr2=0x7f0548147670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.858+0000 7f054e600640 1 --2- 172.21.15.27:0/1163529940 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f05480b1db0 0x7f0548147670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.858+0000 7f054e600640 1 -- 172.21.15.27:0/1163529940 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f054806c430 msgr2=0x7f0548152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.858+0000 7f054e600640 1 --2- 172.21.15.27:0/1163529940 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f054806c430 0x7f0548152680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.858+0000 7f054e600640 1 -- 172.21.15.27:0/1163529940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0540009a70 con 0x7f05480ce760 2024-01-15T18:06:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.862+0000 7f054e600640 1 --2- 172.21.15.27:0/1163529940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05480ce760 0x7f054815c2a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f05400b0040 tx=0x7f05400195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.862+0000 7f0545600640 1 -- 172.21.15.27:0/1163529940 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05400b9860 con 0x7f05480ce760 2024-01-15T18:06:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.862+0000 7f0545600640 1 -- 172.21.15.27:0/1163529940 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05400b99c0 con 0x7f05480ce760 2024-01-15T18:06:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.862+0000 7f0545600640 1 -- 172.21.15.27:0/1163529940 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05400b9d70 con 0x7f05480ce760 2024-01-15T18:06:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.862+0000 7f0550600640 1 -- 172.21.15.27:0/1163529940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05480712f0 con 0x7f05480ce760 2024-01-15T18:06:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.862+0000 7f0550600640 1 -- 172.21.15.27:0/1163529940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f054815baa0 con 0x7f05480ce760 2024-01-15T18:06:14.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.862+0000 7f0550600640 1 -- 172.21.15.27:0/1163529940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f054815bf80 con 0x7f05480ce760 2024-01-15T18:06:14.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.862+0000 7f0545600640 1 -- 172.21.15.27:0/1163529940 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f05400ad440 con 0x7f05480ce760 2024-01-15T18:06:14.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.862+0000 7f0545600640 1 --2- 172.21.15.27:0/1163529940 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05200668f0 0x7f0520068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.862+0000 7f0545600640 1 -- 172.21.15.27:0/1163529940 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f054011f720 con 0x7f05480ce760 2024-01-15T18:06:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.866+0000 7f054dc00640 1 --2- 172.21.15.27:0/1163529940 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05200668f0 0x7f0520068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.866+0000 7f054dc00640 1 --2- 172.21.15.27:0/1163529940 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05200668f0 0x7f0520068db0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0548156030 tx=0x7f0538008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:14.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:14.870+0000 7f0545600640 1 -- 172.21.15.27:0/1163529940 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f05400ecb40 con 0x7f05480ce760 2024-01-15T18:06:15.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.062+0000 7f0550600640 1 -- 172.21.15.27:0/1163529940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f054804e850 con 0x7f05480ce760 2024-01-15T18:06:15.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.818+0000 7f0545600640 1 -- 172.21.15.27:0/1163529940 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v18) v1 ==== 136+0+0 (secure 0 0 0) 0x7f05400f0160 con 0x7f05480ce760 2024-01-15T18:06:15.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.822+0000 7f0550600640 1 -- 172.21.15.27:0/1163529940 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05200668f0 msgr2=0x7f0520068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:15.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.822+0000 7f0550600640 1 --2- 172.21.15.27:0/1163529940 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05200668f0 0x7f0520068db0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0548156030 tx=0x7f0538008040 comp rx=0 tx=0).stop 2024-01-15T18:06:15.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.822+0000 7f0550600640 1 -- 172.21.15.27:0/1163529940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05480ce760 msgr2=0x7f054815c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:15.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.822+0000 7f0550600640 1 --2- 172.21.15.27:0/1163529940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05480ce760 0x7f054815c2a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f05400b0040 tx=0x7f05400195f0 comp rx=0 tx=0).stop 2024-01-15T18:06:15.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.822+0000 7f0550600640 1 -- 172.21.15.27:0/1163529940 shutdown_connections 2024-01-15T18:06:15.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.822+0000 7f0550600640 1 --2- 172.21.15.27:0/1163529940 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05200668f0 0x7f0520068db0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:15.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.822+0000 7f0550600640 1 --2- 172.21.15.27:0/1163529940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05480ce760 0x7f054815c2a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:15.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.822+0000 7f0550600640 1 --2- 172.21.15.27:0/1163529940 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f05480b1db0 0x7f0548147670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:15.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.822+0000 7f0550600640 1 --2- 172.21.15.27:0/1163529940 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f054806c430 0x7f0548152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:15.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.822+0000 7f0550600640 1 -- 172.21.15.27:0/1163529940 >> 172.21.15.27:0/1163529940 conn(0x7f05480cb700 msgr2=0x7f05480ac880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:15.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.822+0000 7f0550600640 1 -- 172.21.15.27:0/1163529940 shutdown_connections 2024-01-15T18:06:15.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.822+0000 7f0550600640 1 -- 172.21.15.27:0/1163529940 wait complete. 2024-01-15T18:06:15.852 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:06:15.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.994+0000 7f2601600640 1 Processor -- start 2024-01-15T18:06:15.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.994+0000 7f2601600640 1 -- start start 2024-01-15T18:06:16.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.994+0000 7f2601600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f25fc0b2190 0x7f25fc0b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:16.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.994+0000 7f2601600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f25fc0b1490 0x7f25fc0b3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:16.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.994+0000 7f2601600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25fc0d03a0 con 0x7f25fc161390 2024-01-15T18:06:16.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.994+0000 7f2601600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25fc0cfb70 con 0x7f25fc0b1490 2024-01-15T18:06:16.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.994+0000 7f2601600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc0b2650 con 0x7f25fc0b2190 2024-01-15T18:06:16.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.994+0000 7f25fb400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f25fc0b1490 0x7f25fc0b3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:16.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.994+0000 7f25faa00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f25fc161390 0x7f25fc161770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57692/0 (socket says 172.21.15.27:57692) 2024-01-15T18:06:16.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.994+0000 7f25faa00640 1 -- 172.21.15.27:0/716924161 learned_addr learned my addr 172.21.15.27:0/716924161 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:06:16.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.994+0000 7f25fa000640 1 --2- 172.21.15.27:0/716924161 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f25fc0b2190 0x7f25fc0b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:16.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.994+0000 7f25fb400640 1 -- 172.21.15.27:0/716924161 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f25fc0b2190 msgr2=0x7f25fc0b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:16.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.994+0000 7f25fb400640 1 --2- 172.21.15.27:0/716924161 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f25fc0b2190 0x7f25fc0b0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:16.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.994+0000 7f25fb400640 1 -- 172.21.15.27:0/716924161 >> v1:172.21.15.27:6789/0 conn(0x7f25fc161390 legacy=0x7f25fc161770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:16.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.994+0000 7f25fb400640 1 -- 172.21.15.27:0/716924161 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25fc0cfd50 con 0x7f25fc0b1490 2024-01-15T18:06:16.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.994+0000 7f25fa000640 1 --2- 172.21.15.27:0/716924161 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f25fc0b2190 0x7f25fc0b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:06:16.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.998+0000 7f25fb400640 1 --2- 172.21.15.27:0/716924161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25fc0b1490 0x7f25fc0b3d90 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f25f4009d90 tx=0x7f25f401c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dbce1375fdd4d3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:16.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.998+0000 7f25f9600640 1 -- 172.21.15.27:0/716924161 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f40a5740 con 0x7f25fc0b1490 2024-01-15T18:06:16.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.998+0000 7f25f9600640 1 -- 172.21.15.27:0/716924161 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25f40a58a0 con 0x7f25fc0b1490 2024-01-15T18:06:16.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.998+0000 7f25f9600640 1 -- 172.21.15.27:0/716924161 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f40a5c30 con 0x7f25fc0b1490 2024-01-15T18:06:16.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.998+0000 7f2601600640 1 -- 172.21.15.27:0/716924161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25fc0b1490 msgr2=0x7f25fc0b3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:16.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.998+0000 7f2601600640 1 --2- 172.21.15.27:0/716924161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25fc0b1490 0x7f25fc0b3d90 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f25f4009d90 tx=0x7f25f401c720 comp rx=0 tx=0).stop 2024-01-15T18:06:16.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.998+0000 7f2601600640 1 -- 172.21.15.27:0/716924161 shutdown_connections 2024-01-15T18:06:16.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.998+0000 7f2601600640 1 --2- 172.21.15.27:0/716924161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25fc0b1490 0x7f25fc0b3d90 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:16.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.998+0000 7f2601600640 1 --2- 172.21.15.27:0/716924161 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f25fc0b2190 0x7f25fc0b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:16.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.998+0000 7f2601600640 1 -- 172.21.15.27:0/716924161 >> 172.21.15.27:0/716924161 conn(0x7f25fc0d0f40 msgr2=0x7f25fc0d1350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:16.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.998+0000 7f2601600640 1 -- 172.21.15.27:0/716924161 shutdown_connections 2024-01-15T18:06:16.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:15.998+0000 7f2601600640 1 -- 172.21.15.27:0/716924161 wait complete. 2024-01-15T18:06:16.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.002+0000 7f2601600640 1 Processor -- start 2024-01-15T18:06:16.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.002+0000 7f2601600640 1 -- start start 2024-01-15T18:06:16.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.002+0000 7f2601600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f25fc0b1490 0x7f25fc155f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:16.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.002+0000 7f2601600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f25fc0b2190 0x7f25fc0c6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:16.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.002+0000 7f2601600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25fc161390 0x7f25fc0c6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:16.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.002+0000 7f2601600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc15dbf0 con 0x7f25fc161390 2024-01-15T18:06:16.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.002+0000 7f2601600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc15e490 con 0x7f25fc0b1490 2024-01-15T18:06:16.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.002+0000 7f2601600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc0c6190 con 0x7f25fc0b2190 2024-01-15T18:06:16.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.002+0000 7f25fb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25fc161390 0x7f25fc0c6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:16.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.002+0000 7f25fb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25fc161390 0x7f25fc0c6f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54334/0 (socket says 172.21.15.27:54334) 2024-01-15T18:06:16.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.002+0000 7f25fb400640 1 -- 172.21.15.27:0/1560940936 learned_addr learned my addr 172.21.15.27:0/1560940936 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:16.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.002+0000 7f25fa000640 1 --2- 172.21.15.27:0/1560940936 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f25fc0b2190 0x7f25fc0c6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:16.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.002+0000 7f25faa00640 1 --2- 172.21.15.27:0/1560940936 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f25fc0b1490 0x7f25fc155f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:16.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.002+0000 7f25fb400640 1 -- 172.21.15.27:0/1560940936 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f25fc0b2190 msgr2=0x7f25fc0c6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:16.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.002+0000 7f25fb400640 1 --2- 172.21.15.27:0/1560940936 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f25fc0b2190 0x7f25fc0c6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:16.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.002+0000 7f25fb400640 1 -- 172.21.15.27:0/1560940936 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f25fc0b1490 msgr2=0x7f25fc155f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:16.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.002+0000 7f25fb400640 1 --2- 172.21.15.27:0/1560940936 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f25fc0b1490 0x7f25fc155f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:16.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.002+0000 7f25fb400640 1 -- 172.21.15.27:0/1560940936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f4009a70 con 0x7f25fc161390 2024-01-15T18:06:16.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.002+0000 7f25fb400640 1 --2- 172.21.15.27:0/1560940936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25fc161390 0x7f25fc0c6f80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f25f40b0040 tx=0x7f25f40194c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:16.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.002+0000 7f25f2a00640 1 -- 172.21.15.27:0/1560940936 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f40ad850 con 0x7f25fc161390 2024-01-15T18:06:16.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.002+0000 7f25f2a00640 1 -- 172.21.15.27:0/1560940936 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25f40ad9b0 con 0x7f25fc161390 2024-01-15T18:06:16.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.002+0000 7f2601600640 1 -- 172.21.15.27:0/1560940936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25fc15ddd0 con 0x7f25fc161390 2024-01-15T18:06:16.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.002+0000 7f25f2a00640 1 -- 172.21.15.27:0/1560940936 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f40adda0 con 0x7f25fc161390 2024-01-15T18:06:16.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.002+0000 7f2601600640 1 -- 172.21.15.27:0/1560940936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25fc156460 con 0x7f25fc161390 2024-01-15T18:06:16.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.002+0000 7f2601600640 1 -- 172.21.15.27:0/1560940936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25fc15e5d0 con 0x7f25fc161390 2024-01-15T18:06:16.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.006+0000 7f25f2a00640 1 -- 172.21.15.27:0/1560940936 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f25f40b4050 con 0x7f25fc161390 2024-01-15T18:06:16.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.006+0000 7f25f2a00640 1 --2- 172.21.15.27:0/1560940936 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f25d40668f0 0x7f25d4068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:16.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.006+0000 7f25f2a00640 1 -- 172.21.15.27:0/1560940936 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25f411f570 con 0x7f25fc161390 2024-01-15T18:06:16.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.006+0000 7f25faa00640 1 --2- 172.21.15.27:0/1560940936 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f25d40668f0 0x7f25d4068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:16.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.006+0000 7f25faa00640 1 --2- 172.21.15.27:0/1560940936 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f25d40668f0 0x7f25d4068db0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f25ec000f70 tx=0x7f25ec007e10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:16.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.010+0000 7f25f2a00640 1 -- 172.21.15.27:0/1560940936 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f25fc15e5d0 con 0x7f25fc161390 2024-01-15T18:06:16.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.190+0000 7f2601600640 1 -- 172.21.15.27:0/1560940936 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f25fc0b4ae0 con 0x7f25d40668f0 2024-01-15T18:06:16.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.190+0000 7f25f2a00640 1 -- 172.21.15.27:0/1560940936 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f25ec0072c0 con 0x7f25d40668f0 2024-01-15T18:06:16.197 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:06:16.197 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:06:16.197 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-15T18:06:16.197 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:16.197 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:06:16.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.194+0000 7f2601600640 1 -- 172.21.15.27:0/1560940936 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f25d40668f0 msgr2=0x7f25d4068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:16.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.194+0000 7f2601600640 1 --2- 172.21.15.27:0/1560940936 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f25d40668f0 0x7f25d4068db0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f25ec000f70 tx=0x7f25ec007e10 comp rx=0 tx=0).stop 2024-01-15T18:06:16.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.194+0000 7f2601600640 1 -- 172.21.15.27:0/1560940936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25fc161390 msgr2=0x7f25fc0c6f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:16.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.194+0000 7f2601600640 1 --2- 172.21.15.27:0/1560940936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25fc161390 0x7f25fc0c6f80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f25f40b0040 tx=0x7f25f40194c0 comp rx=0 tx=0).stop 2024-01-15T18:06:16.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.194+0000 7f2601600640 1 -- 172.21.15.27:0/1560940936 shutdown_connections 2024-01-15T18:06:16.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.194+0000 7f2601600640 1 --2- 172.21.15.27:0/1560940936 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f25d40668f0 0x7f25d4068db0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:16.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.194+0000 7f2601600640 1 --2- 172.21.15.27:0/1560940936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25fc161390 0x7f25fc0c6f80 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:16.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.194+0000 7f2601600640 1 --2- 172.21.15.27:0/1560940936 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f25fc0b2190 0x7f25fc0c6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:16.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.194+0000 7f2601600640 1 --2- 172.21.15.27:0/1560940936 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f25fc0b1490 0x7f25fc155f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:16.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.194+0000 7f2601600640 1 -- 172.21.15.27:0/1560940936 >> 172.21.15.27:0/1560940936 conn(0x7f25fc0d0f40 msgr2=0x7f25fc161770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:16.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.194+0000 7f2601600640 1 -- 172.21.15.27:0/1560940936 shutdown_connections 2024-01-15T18:06:16.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.194+0000 7f2601600640 1 -- 172.21.15.27:0/1560940936 wait complete. 2024-01-15T18:06:16.220 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:06:16.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.358+0000 7fe0ade00640 1 Processor -- start 2024-01-15T18:06:16.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.358+0000 7fe0ade00640 1 -- start start 2024-01-15T18:06:16.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.358+0000 7fe0ade00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fe0a80cf460 0x7fe0a80cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:16.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.358+0000 7fe0ade00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a80cfd80 con 0x7fe0a815b4b0 2024-01-15T18:06:16.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.358+0000 7fe0ade00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a80d19a0 con 0x7fe0a80b2590 2024-01-15T18:06:16.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.358+0000 7fe0ade00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0a80d1c80 con 0x7fe0a80cf460 2024-01-15T18:06:16.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.358+0000 7fe0a6000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe0a815b4b0 0x7fe0a80cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57696/0 (socket says 172.21.15.27:57696) 2024-01-15T18:06:16.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.358+0000 7fe0a6000640 1 -- 172.21.15.27:0/974987387 learned_addr learned my addr 172.21.15.27:0/974987387 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:06:16.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.358+0000 7fe0a6a00640 1 --2- 172.21.15.27:0/974987387 >> v2:172.21.15.93:3300/0 conn(0x7fe0a80cf460 0x7fe0a80cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:16.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.358+0000 7fe0a5600640 1 -- 172.21.15.27:0/974987387 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358474177 0 0) 0x7fe0a80cfd80 con 0x7fe0a815b4b0 2024-01-15T18:06:16.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.358+0000 7fe0a5600640 1 -- 172.21.15.27:0/974987387 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe088003530 con 0x7fe0a815b4b0 2024-01-15T18:06:16.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.362+0000 7fe0a5600640 1 -- 172.21.15.27:0/974987387 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504353098 0 0) 0x7fe0a80d19a0 con 0x7fe0a80b2590 2024-01-15T18:06:16.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.362+0000 7fe0a5600640 1 -- 172.21.15.27:0/974987387 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a80cfd80 con 0x7fe0a80b2590 2024-01-15T18:06:16.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.362+0000 7fe0a5600640 1 -- 172.21.15.27:0/974987387 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1610419683 0 0) 0x7fe088003530 con 0x7fe0a815b4b0 2024-01-15T18:06:16.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.362+0000 7fe0a6a00640 1 -- 172.21.15.27:0/974987387 >> v1:172.21.15.93:6789/0 conn(0x7fe0a80b2590 legacy=0x7fe0a80cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:16.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.362+0000 7fe0a6a00640 1 -- 172.21.15.27:0/974987387 >> v1:172.21.15.27:6789/0 conn(0x7fe0a815b4b0 legacy=0x7fe0a80cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:16.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.362+0000 7fe0a6a00640 1 -- 172.21.15.27:0/974987387 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a80b07c0 con 0x7fe0a80cf460 2024-01-15T18:06:16.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.362+0000 7fe0a6a00640 1 --2- 172.21.15.27:0/974987387 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0a80cf460 0x7fe0a80cf840 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe0980097c0 tx=0x7fe09801c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d82b63b7b095696a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:16.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.362+0000 7fe0a5600640 1 -- 172.21.15.27:0/974987387 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe090001600 con 0x7fe0a80cf460 2024-01-15T18:06:16.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.362+0000 7fe0a5600640 1 -- 172.21.15.27:0/974987387 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0980a7050 con 0x7fe0a80cf460 2024-01-15T18:06:16.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.362+0000 7fe0a5600640 1 -- 172.21.15.27:0/974987387 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0980ab620 con 0x7fe0a80cf460 2024-01-15T18:06:16.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.366+0000 7fe0ade00640 1 -- 172.21.15.27:0/974987387 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0a80cf460 msgr2=0x7fe0a80cf840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:16.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.366+0000 7fe0ade00640 1 --2- 172.21.15.27:0/974987387 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0a80cf460 0x7fe0a80cf840 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe0980097c0 tx=0x7fe09801c3a0 comp rx=0 tx=0).stop 2024-01-15T18:06:16.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.366+0000 7fe0ade00640 1 -- 172.21.15.27:0/974987387 shutdown_connections 2024-01-15T18:06:16.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.366+0000 7fe0ade00640 1 --2- 172.21.15.27:0/974987387 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0a80cf460 0x7fe0a80cf840 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:16.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.366+0000 7fe0ade00640 1 -- 172.21.15.27:0/974987387 >> 172.21.15.27:0/974987387 conn(0x7fe0a80d0e80 msgr2=0x7fe0a80d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:16.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.366+0000 7fe0ade00640 1 -- 172.21.15.27:0/974987387 shutdown_connections 2024-01-15T18:06:16.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.366+0000 7fe0ade00640 1 -- 172.21.15.27:0/974987387 wait complete. 2024-01-15T18:06:16.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.366+0000 7fe0ade00640 1 Processor -- start 2024-01-15T18:06:16.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.366+0000 7fe0ade00640 1 -- start start 2024-01-15T18:06:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.366+0000 7fe0ade00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0a80b2590 0x7fe0a8148740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.366+0000 7fe0ade00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0a80cf460 0x7fe0a814fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.366+0000 7fe0ade00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe0a815b4b0 0x7fe0a8151730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.366+0000 7fe0ade00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a8148c80 con 0x7fe0a80b2590 2024-01-15T18:06:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.366+0000 7fe0ade00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a815dfc0 con 0x7fe0a80cf460 2024-01-15T18:06:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.366+0000 7fe0ade00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a815eb00 con 0x7fe0a815b4b0 2024-01-15T18:06:16.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.366+0000 7fe0a6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0a80b2590 0x7fe0a8148740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:16.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.366+0000 7fe0a6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0a80b2590 0x7fe0a8148740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54342/0 (socket says 172.21.15.27:54342) 2024-01-15T18:06:16.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.366+0000 7fe0a6a00640 1 -- 172.21.15.27:0/2531921687 learned_addr learned my addr 172.21.15.27:0/2531921687 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:16.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.366+0000 7fe0a6000640 1 --2- 172.21.15.27:0/2531921687 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0a80cf460 0x7fe0a814fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:16.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.366+0000 7fe0a7400640 1 --2- 172.21.15.27:0/2531921687 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe0a815b4b0 0x7fe0a8151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:16.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.366+0000 7fe0a6a00640 1 -- 172.21.15.27:0/2531921687 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe0a815b4b0 msgr2=0x7fe0a8151730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:16.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.366+0000 7fe0a6a00640 1 --2- 172.21.15.27:0/2531921687 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe0a815b4b0 0x7fe0a8151730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:16.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.366+0000 7fe0a6a00640 1 -- 172.21.15.27:0/2531921687 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0a80cf460 msgr2=0x7fe0a814fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:16.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.366+0000 7fe0a6a00640 1 --2- 172.21.15.27:0/2531921687 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0a80cf460 0x7fe0a814fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:16.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.366+0000 7fe0a6a00640 1 -- 172.21.15.27:0/2531921687 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe098009420 con 0x7fe0a80b2590 2024-01-15T18:06:16.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.370+0000 7fe0a7400640 1 --2- 172.21.15.27:0/2531921687 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe0a815b4b0 0x7fe0a8151730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:06:16.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.370+0000 7fe0a6a00640 1 --2- 172.21.15.27:0/2531921687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0a80b2590 0x7fe0a8148740 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fe0980097c0 tx=0x7fe09801abe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:16.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.370+0000 7fe09f400640 1 -- 172.21.15.27:0/2531921687 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0980b7450 con 0x7fe0a80b2590 2024-01-15T18:06:16.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.370+0000 7fe09f400640 1 -- 172.21.15.27:0/2531921687 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0980b75d0 con 0x7fe0a80b2590 2024-01-15T18:06:16.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.370+0000 7fe09f400640 1 -- 172.21.15.27:0/2531921687 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0980b7980 con 0x7fe0a80b2590 2024-01-15T18:06:16.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.370+0000 7fe0ade00640 1 -- 172.21.15.27:0/2531921687 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a815e260 con 0x7fe0a80b2590 2024-01-15T18:06:16.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.370+0000 7fe0ade00640 1 -- 172.21.15.27:0/2531921687 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a815e820 con 0x7fe0a80b2590 2024-01-15T18:06:16.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.370+0000 7fe0ade00640 1 -- 172.21.15.27:0/2531921687 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0a804e850 con 0x7fe0a80b2590 2024-01-15T18:06:16.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.370+0000 7fe09f400640 1 -- 172.21.15.27:0/2531921687 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7fe0980b7ae0 con 0x7fe0a80b2590 2024-01-15T18:06:16.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.370+0000 7fe09f400640 1 --2- 172.21.15.27:0/2531921687 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe0800668f0 0x7fe080068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:16.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.370+0000 7fe09f400640 1 -- 172.21.15.27:0/2531921687 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe09811ea00 con 0x7fe0a80b2590 2024-01-15T18:06:16.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.370+0000 7fe0a6000640 1 --2- 172.21.15.27:0/2531921687 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe0800668f0 0x7fe080068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:16.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.374+0000 7fe0a6000640 1 --2- 172.21.15.27:0/2531921687 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe0800668f0 0x7fe080068db0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe090000fe0 tx=0x7fe090001380 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:16.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.374+0000 7fe09f400640 1 -- 172.21.15.27:0/2531921687 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe0980ebc30 con 0x7fe0a80b2590 2024-01-15T18:06:16.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.530+0000 7fe0ade00640 1 -- 172.21.15.27:0/2531921687 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe0a80cf240 con 0x7fe0a80b2590 2024-01-15T18:06:16.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.530+0000 7fe09f400640 1 -- 172.21.15.27:0/2531921687 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe0980b00c0 con 0x7fe0a80b2590 2024-01-15T18:06:16.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.534+0000 7fe0ade00640 1 -- 172.21.15.27:0/2531921687 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe0800668f0 msgr2=0x7fe080068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:16.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.534+0000 7fe0ade00640 1 --2- 172.21.15.27:0/2531921687 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe0800668f0 0x7fe080068db0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe090000fe0 tx=0x7fe090001380 comp rx=0 tx=0).stop 2024-01-15T18:06:16.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.534+0000 7fe0ade00640 1 -- 172.21.15.27:0/2531921687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0a80b2590 msgr2=0x7fe0a8148740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:16.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.534+0000 7fe0ade00640 1 --2- 172.21.15.27:0/2531921687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0a80b2590 0x7fe0a8148740 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fe0980097c0 tx=0x7fe09801abe0 comp rx=0 tx=0).stop 2024-01-15T18:06:16.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.534+0000 7fe0ade00640 1 -- 172.21.15.27:0/2531921687 shutdown_connections 2024-01-15T18:06:16.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.534+0000 7fe0ade00640 1 --2- 172.21.15.27:0/2531921687 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe0800668f0 0x7fe080068db0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:16.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.534+0000 7fe0ade00640 1 --2- 172.21.15.27:0/2531921687 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe0a815b4b0 0x7fe0a8151730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:16.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.534+0000 7fe0ade00640 1 --2- 172.21.15.27:0/2531921687 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0a80cf460 0x7fe0a814fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:16.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.534+0000 7fe0ade00640 1 --2- 172.21.15.27:0/2531921687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0a80b2590 0x7fe0a8148740 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:16.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.534+0000 7fe0ade00640 1 -- 172.21.15.27:0/2531921687 >> 172.21.15.27:0/2531921687 conn(0x7fe0a80d0e80 msgr2=0x7fe0a80cd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:16.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.534+0000 7fe0ade00640 1 -- 172.21.15.27:0/2531921687 shutdown_connections 2024-01-15T18:06:16.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.534+0000 7fe0ade00640 1 -- 172.21.15.27:0/2531921687 wait complete. 2024-01-15T18:06:16.559 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-15T18:06:16.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.698+0000 7f6d2ee00640 1 Processor -- start 2024-01-15T18:06:16.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.698+0000 7f6d2ee00640 1 -- start start 2024-01-15T18:06:16.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.698+0000 7f6d2ee00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6d280d19a0 0x7f6d280cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:16.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.698+0000 7f6d2ee00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f6d280c3e50 0x7f6d280c42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:16.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.698+0000 7f6d2ee00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d280cc910 con 0x7f6d2814b420 2024-01-15T18:06:16.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.698+0000 7f6d2ee00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d280c4800 con 0x7f6d280d19a0 2024-01-15T18:06:16.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.698+0000 7f6d2ee00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d280cbd00 con 0x7f6d280c3e50 2024-01-15T18:06:16.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.698+0000 7f6d27400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6d280d19a0 0x7f6d280cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:16.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.698+0000 7f6d27400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6d280d19a0 0x7f6d280cb4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54344/0 (socket says 172.21.15.27:54344) 2024-01-15T18:06:16.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.698+0000 7f6d27400640 1 -- 172.21.15.27:0/1455474850 learned_addr learned my addr 172.21.15.27:0/1455474850 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:16.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.698+0000 7f6d2ce00640 1 --2- 172.21.15.27:0/1455474850 >> v2:172.21.15.93:3300/0 conn(0x7f6d280c3e50 0x7f6d280c42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:16.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.698+0000 7f6d27400640 1 -- 172.21.15.27:0/1455474850 >> v2:172.21.15.93:3300/0 conn(0x7f6d280c3e50 msgr2=0x7f6d280c42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:16.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.698+0000 7f6d27400640 1 --2- 172.21.15.27:0/1455474850 >> v2:172.21.15.93:3300/0 conn(0x7f6d280c3e50 0x7f6d280c42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:16.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.698+0000 7f6d27400640 1 -- 172.21.15.27:0/1455474850 >> v1:172.21.15.93:6789/0 conn(0x7f6d2814b420 legacy=0x7f6d280ccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:16.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.698+0000 7f6d27400640 1 -- 172.21.15.27:0/1455474850 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d280cbfe0 con 0x7f6d280d19a0 2024-01-15T18:06:16.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.702+0000 7f6d27400640 1 --2- 172.21.15.27:0/1455474850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d280d19a0 0x7f6d280cb4d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6d280717e0 tx=0x7f6d1001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c35a6db1be4ec8a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:16.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.702+0000 7f6d26a00640 1 -- 172.21.15.27:0/1455474850 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d100ab640 con 0x7f6d280d19a0 2024-01-15T18:06:16.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.702+0000 7f6d26a00640 1 -- 172.21.15.27:0/1455474850 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d100a7070 con 0x7f6d280d19a0 2024-01-15T18:06:16.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.702+0000 7f6d26a00640 1 -- 172.21.15.27:0/1455474850 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d100ab9c0 con 0x7f6d280d19a0 2024-01-15T18:06:16.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.702+0000 7f6d2ee00640 1 -- 172.21.15.27:0/1455474850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d280d19a0 msgr2=0x7f6d280cb4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:16.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.702+0000 7f6d2ee00640 1 --2- 172.21.15.27:0/1455474850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d280d19a0 0x7f6d280cb4d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6d280717e0 tx=0x7f6d1001c440 comp rx=0 tx=0).stop 2024-01-15T18:06:16.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.702+0000 7f6d2ee00640 1 -- 172.21.15.27:0/1455474850 shutdown_connections 2024-01-15T18:06:16.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.702+0000 7f6d2ee00640 1 --2- 172.21.15.27:0/1455474850 >> v2:172.21.15.93:3300/0 conn(0x7f6d280c3e50 0x7f6d280c42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:16.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.702+0000 7f6d2ee00640 1 --2- 172.21.15.27:0/1455474850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d280d19a0 0x7f6d280cb4d0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:16.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.702+0000 7f6d2ee00640 1 -- 172.21.15.27:0/1455474850 >> 172.21.15.27:0/1455474850 conn(0x7f6d280d0e80 msgr2=0x7f6d280d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:16.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.702+0000 7f6d2ee00640 1 -- 172.21.15.27:0/1455474850 shutdown_connections 2024-01-15T18:06:16.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.702+0000 7f6d2ee00640 1 -- 172.21.15.27:0/1455474850 wait complete. 2024-01-15T18:06:16.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.706+0000 7f6d2ee00640 1 Processor -- start 2024-01-15T18:06:16.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.706+0000 7f6d2ee00640 1 -- start start 2024-01-15T18:06:16.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.706+0000 7f6d2ee00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d280c3e50 0x7f6d2814a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:16.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.706+0000 7f6d2ee00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6d280d19a0 0x7f6d2815d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:16.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.706+0000 7f6d2ee00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6d2814b420 0x7f6d280c8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:16.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.706+0000 7f6d2ee00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d28162180 con 0x7f6d280c3e50 2024-01-15T18:06:16.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.706+0000 7f6d2ee00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d280c7040 con 0x7f6d280d19a0 2024-01-15T18:06:16.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.706+0000 7f6d2ee00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d280c7320 con 0x7f6d2814b420 2024-01-15T18:06:16.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.706+0000 7f6d27e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d280c3e50 0x7f6d2814a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:16.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.706+0000 7f6d27e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d280c3e50 0x7f6d2814a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54350/0 (socket says 172.21.15.27:54350) 2024-01-15T18:06:16.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.706+0000 7f6d27e00640 1 -- 172.21.15.27:0/208202348 learned_addr learned my addr 172.21.15.27:0/208202348 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:16.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.706+0000 7f6d27400640 1 --2- 172.21.15.27:0/208202348 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6d280d19a0 0x7f6d2815d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:16.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.706+0000 7f6d2ce00640 1 --2- 172.21.15.27:0/208202348 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6d2814b420 0x7f6d280c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:16.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.706+0000 7f6d27e00640 1 -- 172.21.15.27:0/208202348 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6d2814b420 msgr2=0x7f6d280c8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:16.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.706+0000 7f6d27e00640 1 --2- 172.21.15.27:0/208202348 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6d2814b420 0x7f6d280c8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:16.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.706+0000 7f6d27e00640 1 -- 172.21.15.27:0/208202348 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6d280d19a0 msgr2=0x7f6d2815d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:16.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.706+0000 7f6d27e00640 1 --2- 172.21.15.27:0/208202348 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6d280d19a0 0x7f6d2815d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:16.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.706+0000 7f6d27e00640 1 -- 172.21.15.27:0/208202348 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d100094d0 con 0x7f6d280c3e50 2024-01-15T18:06:16.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.706+0000 7f6d27e00640 1 --2- 172.21.15.27:0/208202348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d280c3e50 0x7f6d2814a240 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6d1800c3c0 tx=0x7f6d1800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:16.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.710+0000 7f6d1fe00640 1 -- 172.21.15.27:0/208202348 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d180052d0 con 0x7f6d280c3e50 2024-01-15T18:06:16.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.710+0000 7f6d1fe00640 1 -- 172.21.15.27:0/208202348 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d18005430 con 0x7f6d280c3e50 2024-01-15T18:06:16.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.710+0000 7f6d2ee00640 1 -- 172.21.15.27:0/208202348 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d280c7600 con 0x7f6d280c3e50 2024-01-15T18:06:16.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.710+0000 7f6d1fe00640 1 -- 172.21.15.27:0/208202348 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d18003840 con 0x7f6d280c3e50 2024-01-15T18:06:16.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.710+0000 7f6d2ee00640 1 -- 172.21.15.27:0/208202348 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d280c6ac0 con 0x7f6d280c3e50 2024-01-15T18:06:16.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.710+0000 7f6d2ee00640 1 -- 172.21.15.27:0/208202348 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d2804e850 con 0x7f6d280c3e50 2024-01-15T18:06:16.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.710+0000 7f6d1fe00640 1 -- 172.21.15.27:0/208202348 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f6d180039a0 con 0x7f6d280c3e50 2024-01-15T18:06:16.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.710+0000 7f6d1fe00640 1 --2- 172.21.15.27:0/208202348 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6d000668f0 0x7f6d00068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:16.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.710+0000 7f6d1fe00640 1 -- 172.21.15.27:0/208202348 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d18083920 con 0x7f6d280c3e50 2024-01-15T18:06:16.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.710+0000 7f6d27400640 1 --2- 172.21.15.27:0/208202348 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6d000668f0 0x7f6d00068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:16.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.714+0000 7f6d27400640 1 --2- 172.21.15.27:0/208202348 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6d000668f0 0x7f6d00068db0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6d100af000 tx=0x7f6d100b6040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:16.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.714+0000 7f6d1fe00640 1 -- 172.21.15.27:0/208202348 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6d18050b90 con 0x7f6d280c3e50 2024-01-15T18:06:16.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:16.898+0000 7f6d2ee00640 1 -- 172.21.15.27:0/208202348 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d2806d530 con 0x7f6d000668f0 2024-01-15T18:06:16.911 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:06:16.905+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:06:16.911 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:06:16.905+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:06:16.911 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:06:16.905+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:06:16.911 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:06:16.905+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:06:16.911 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:06:16.905+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:06:17.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.054+0000 7f6d1fe00640 1 -- 172.21.15.27:0/208202348 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6d2806d530 con 0x7f6d000668f0 2024-01-15T18:06:17.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.058+0000 7f6d2ee00640 1 -- 172.21.15.27:0/208202348 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6d000668f0 msgr2=0x7f6d00068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:17.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.058+0000 7f6d2ee00640 1 --2- 172.21.15.27:0/208202348 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6d000668f0 0x7f6d00068db0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6d100af000 tx=0x7f6d100b6040 comp rx=0 tx=0).stop 2024-01-15T18:06:17.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.058+0000 7f6d2ee00640 1 -- 172.21.15.27:0/208202348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d280c3e50 msgr2=0x7f6d2814a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:17.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.058+0000 7f6d2ee00640 1 --2- 172.21.15.27:0/208202348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d280c3e50 0x7f6d2814a240 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6d1800c3c0 tx=0x7f6d1800c880 comp rx=0 tx=0).stop 2024-01-15T18:06:17.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.058+0000 7f6d2ee00640 1 -- 172.21.15.27:0/208202348 shutdown_connections 2024-01-15T18:06:17.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.058+0000 7f6d2ee00640 1 --2- 172.21.15.27:0/208202348 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6d000668f0 0x7f6d00068db0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:17.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.058+0000 7f6d2ee00640 1 --2- 172.21.15.27:0/208202348 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6d2814b420 0x7f6d280c8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:17.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.058+0000 7f6d2ee00640 1 --2- 172.21.15.27:0/208202348 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6d280d19a0 0x7f6d2815d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:17.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.058+0000 7f6d2ee00640 1 --2- 172.21.15.27:0/208202348 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d280c3e50 0x7f6d2814a240 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:17.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.058+0000 7f6d2ee00640 1 -- 172.21.15.27:0/208202348 >> 172.21.15.27:0/208202348 conn(0x7f6d280d0e80 msgr2=0x7f6d281613e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:17.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.058+0000 7f6d2ee00640 1 -- 172.21.15.27:0/208202348 shutdown_connections 2024-01-15T18:06:17.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.058+0000 7f6d2ee00640 1 -- 172.21.15.27:0/208202348 wait complete. 2024-01-15T18:06:17.085 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:06:17.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.226+0000 7f28f7400640 1 Processor -- start 2024-01-15T18:06:17.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.226+0000 7f28f7400640 1 -- start start 2024-01-15T18:06:17.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.226+0000 7f28f7400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28f00cf2e0 0x7f28f00701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:17.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.226+0000 7f28f7400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28f00b1cb0 con 0x7f28f0158900 2024-01-15T18:06:17.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.226+0000 7f28f7400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28f015fa00 con 0x7f28f015f550 2024-01-15T18:06:17.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.226+0000 7f28f7400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f00b1f00 con 0x7f28f00cf2e0 2024-01-15T18:06:17.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.226+0000 7f28f4a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f28f0158900 0x7f28f015bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57706/0 (socket says 172.21.15.27:57706) 2024-01-15T18:06:17.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.226+0000 7f28f4a00640 1 -- 172.21.15.27:0/2793768567 learned_addr learned my addr 172.21.15.27:0/2793768567 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:06:17.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.226+0000 7f28f5400640 1 --2- 172.21.15.27:0/2793768567 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28f00cf2e0 0x7f28f00701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:17.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.226+0000 7f28ef400640 1 -- 172.21.15.27:0/2793768567 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1912335813 0 0) 0x7f28f00b1cb0 con 0x7f28f0158900 2024-01-15T18:06:17.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.226+0000 7f28ef400640 1 -- 172.21.15.27:0/2793768567 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28d0003530 con 0x7f28f0158900 2024-01-15T18:06:17.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.226+0000 7f28ef400640 1 -- 172.21.15.27:0/2793768567 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304262129 0 0) 0x7f28f015fa00 con 0x7f28f015f550 2024-01-15T18:06:17.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.226+0000 7f28ef400640 1 -- 172.21.15.27:0/2793768567 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28f00b1cb0 con 0x7f28f015f550 2024-01-15T18:06:17.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.226+0000 7f28f5400640 1 -- 172.21.15.27:0/2793768567 >> v1:172.21.15.93:6789/0 conn(0x7f28f015f550 legacy=0x7f28f00cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:17.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.230+0000 7f28f5400640 1 -- 172.21.15.27:0/2793768567 >> v1:172.21.15.27:6789/0 conn(0x7f28f0158900 legacy=0x7f28f015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:17.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.230+0000 7f28f5400640 1 -- 172.21.15.27:0/2793768567 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f00b21e0 con 0x7f28f00cf2e0 2024-01-15T18:06:17.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.230+0000 7f28f5400640 1 --2- 172.21.15.27:0/2793768567 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28f00cf2e0 0x7f28f00701e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009dc0 tx=0x7f28e801c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f78d7a2cf470fa4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:17.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.230+0000 7f28ef400640 1 -- 172.21.15.27:0/2793768567 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f0070f30 con 0x7f28f00cf2e0 2024-01-15T18:06:17.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.230+0000 7f28ef400640 1 -- 172.21.15.27:0/2793768567 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e80a5a00 con 0x7f28f00cf2e0 2024-01-15T18:06:17.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.230+0000 7f28ef400640 1 -- 172.21.15.27:0/2793768567 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e80a5d90 con 0x7f28f00cf2e0 2024-01-15T18:06:17.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.234+0000 7f28f7400640 1 -- 172.21.15.27:0/2793768567 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28f00cf2e0 msgr2=0x7f28f00701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:17.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.234+0000 7f28f7400640 1 --2- 172.21.15.27:0/2793768567 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28f00cf2e0 0x7f28f00701e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009dc0 tx=0x7f28e801c940 comp rx=0 tx=0).stop 2024-01-15T18:06:17.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.234+0000 7f28f7400640 1 -- 172.21.15.27:0/2793768567 shutdown_connections 2024-01-15T18:06:17.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.234+0000 7f28f7400640 1 --2- 172.21.15.27:0/2793768567 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28f00cf2e0 0x7f28f00701e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:17.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.234+0000 7f28f7400640 1 -- 172.21.15.27:0/2793768567 >> 172.21.15.27:0/2793768567 conn(0x7f28f00cb700 msgr2=0x7f28f00cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:17.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.234+0000 7f28f7400640 1 -- 172.21.15.27:0/2793768567 shutdown_connections 2024-01-15T18:06:17.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.234+0000 7f28f7400640 1 -- 172.21.15.27:0/2793768567 wait complete. 2024-01-15T18:06:17.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.234+0000 7f28f7400640 1 Processor -- start 2024-01-15T18:06:17.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.234+0000 7f28f7400640 1 -- start start 2024-01-15T18:06:17.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.234+0000 7f28f7400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28f00cf2e0 0x7f28f014ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:17.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.234+0000 7f28f7400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f0158900 0x7f28f00cd220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:17.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.234+0000 7f28f7400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28f015f550 0x7f28f01597a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:17.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.234+0000 7f28f7400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f00ce660 con 0x7f28f0158900 2024-01-15T18:06:17.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.234+0000 7f28f7400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f00cda50 con 0x7f28f015f550 2024-01-15T18:06:17.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.234+0000 7f28f7400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f00cdd30 con 0x7f28f00cf2e0 2024-01-15T18:06:17.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.234+0000 7f28efe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f0158900 0x7f28f00cd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:17.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.234+0000 7f28f4a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28f00cf2e0 0x7f28f014ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:17.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.234+0000 7f28efe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f0158900 0x7f28f00cd220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54360/0 (socket says 172.21.15.27:54360) 2024-01-15T18:06:17.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.234+0000 7f28efe00640 1 -- 172.21.15.27:0/2017722613 learned_addr learned my addr 172.21.15.27:0/2017722613 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.234+0000 7f28f5400640 1 --2- 172.21.15.27:0/2017722613 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28f015f550 0x7f28f01597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.238+0000 7f28efe00640 1 -- 172.21.15.27:0/2017722613 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28f00cf2e0 msgr2=0x7f28f014ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.238+0000 7f28efe00640 1 --2- 172.21.15.27:0/2017722613 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28f00cf2e0 0x7f28f014ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.238+0000 7f28efe00640 1 -- 172.21.15.27:0/2017722613 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28f015f550 msgr2=0x7f28f01597a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.238+0000 7f28efe00640 1 --2- 172.21.15.27:0/2017722613 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28f015f550 0x7f28f01597a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.238+0000 7f28efe00640 1 -- 172.21.15.27:0/2017722613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e8009a70 con 0x7f28f0158900 2024-01-15T18:06:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.238+0000 7f28f5400640 1 --2- 172.21.15.27:0/2017722613 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28f015f550 0x7f28f01597a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:06:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.238+0000 7f28efe00640 1 --2- 172.21.15.27:0/2017722613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f0158900 0x7f28f00cd220 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f28d800c430 tx=0x7f28d800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.238+0000 7f28ecc00640 1 -- 172.21.15.27:0/2017722613 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d80052d0 con 0x7f28f0158900 2024-01-15T18:06:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.238+0000 7f28f7400640 1 -- 172.21.15.27:0/2017722613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f00ce010 con 0x7f28f0158900 2024-01-15T18:06:17.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.238+0000 7f28ecc00640 1 -- 172.21.15.27:0/2017722613 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d8005450 con 0x7f28f0158900 2024-01-15T18:06:17.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.238+0000 7f28ecc00640 1 -- 172.21.15.27:0/2017722613 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d8003840 con 0x7f28f0158900 2024-01-15T18:06:17.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.238+0000 7f28f7400640 1 -- 172.21.15.27:0/2017722613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f00cd760 con 0x7f28f0158900 2024-01-15T18:06:17.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.238+0000 7f28f7400640 1 -- 172.21.15.27:0/2017722613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f00ce7d0 con 0x7f28f0158900 2024-01-15T18:06:17.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.238+0000 7f28ecc00640 1 -- 172.21.15.27:0/2017722613 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 86160+0+0 (secure 0 0 0) 0x7f28d80039a0 con 0x7f28f0158900 2024-01-15T18:06:17.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.242+0000 7f28ecc00640 1 --2- 172.21.15.27:0/2017722613 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f28c80668f0 0x7f28c8068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:17.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.242+0000 7f28ecc00640 1 -- 172.21.15.27:0/2017722613 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f28d8083ac0 con 0x7f28f0158900 2024-01-15T18:06:17.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.242+0000 7f28f4a00640 1 --2- 172.21.15.27:0/2017722613 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f28c80668f0 0x7f28c8068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:17.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.242+0000 7f28f4a00640 1 --2- 172.21.15.27:0/2017722613 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f28c80668f0 0x7f28c8068db0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f28e0007fd0 tx=0x7f28e0007d80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:17.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.246+0000 7f28ecc00640 1 -- 172.21.15.27:0/2017722613 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f28d8050c90 con 0x7f28f0158900 2024-01-15T18:06:17.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.426+0000 7f28f7400640 1 -- 172.21.15.27:0/2017722613 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f28f0071cc0 con 0x7f28c80668f0 2024-01-15T18:06:17.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.434+0000 7f28ecc00640 1 -- 172.21.15.27:0/2017722613 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f28f0071cc0 con 0x7f28c80668f0 2024-01-15T18:06:17.439 INFO:teuthology.orchestra.run.smithi027.stdout:/volumes/_nogroup/sv1/6d734a73-9593-40e4-bc54-648b1a9e9fa0 2024-01-15T18:06:17.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.434+0000 7f28f7400640 1 -- 172.21.15.27:0/2017722613 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f28c80668f0 msgr2=0x7f28c8068db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:17.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.434+0000 7f28f7400640 1 --2- 172.21.15.27:0/2017722613 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f28c80668f0 0x7f28c8068db0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f28e0007fd0 tx=0x7f28e0007d80 comp rx=0 tx=0).stop 2024-01-15T18:06:17.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.434+0000 7f28f7400640 1 -- 172.21.15.27:0/2017722613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f0158900 msgr2=0x7f28f00cd220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:17.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.434+0000 7f28f7400640 1 --2- 172.21.15.27:0/2017722613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f0158900 0x7f28f00cd220 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f28d800c430 tx=0x7f28d800c8f0 comp rx=0 tx=0).stop 2024-01-15T18:06:17.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.434+0000 7f28f7400640 1 -- 172.21.15.27:0/2017722613 shutdown_connections 2024-01-15T18:06:17.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.434+0000 7f28f7400640 1 --2- 172.21.15.27:0/2017722613 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f28c80668f0 0x7f28c8068db0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:17.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.434+0000 7f28f7400640 1 --2- 172.21.15.27:0/2017722613 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28f015f550 0x7f28f01597a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:17.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.434+0000 7f28f7400640 1 --2- 172.21.15.27:0/2017722613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f0158900 0x7f28f00cd220 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:17.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.434+0000 7f28f7400640 1 --2- 172.21.15.27:0/2017722613 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28f00cf2e0 0x7f28f014ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:17.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.434+0000 7f28f7400640 1 -- 172.21.15.27:0/2017722613 >> 172.21.15.27:0/2017722613 conn(0x7f28f00cb700 msgr2=0x7f28f00b3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:17.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.434+0000 7f28f7400640 1 -- 172.21.15.27:0/2017722613 shutdown_connections 2024-01-15T18:06:17.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:17.434+0000 7f28f7400640 1 -- 172.21.15.27:0/2017722613 wait complete. 2024-01-15T18:06:17.462 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/6d734a73-9593-40e4-bc54-648b1a9e9fa0 2024-01-15T18:06:17.462 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:06:17.462 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:06:17.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:17.473 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/6d734a73-9593-40e4-bc54-648b1a9e9fa0/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:06:24.561 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records in 2024-01-15T18:06:24.561 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records out 2024-01-15T18:06:24.561 INFO:teuthology.orchestra.run.smithi027.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.02194 s, 153 MB/s 2024-01-15T18:06:24.563 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:06:24.563 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:06:24.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:24.573 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/6d734a73-9593-40e4-bc54-648b1a9e9fa0/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:06:26.168 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:06:26.174 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:06:31.453 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records in 2024-01-15T18:06:31.453 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records out 2024-01-15T18:06:31.453 INFO:teuthology.orchestra.run.smithi027.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.82726 s, 157 MB/s 2024-01-15T18:06:31.454 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:06:31.455 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:06:31.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:31.463 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/6d734a73-9593-40e4-bc54-648b1a9e9fa0/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:06:38.303 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records in 2024-01-15T18:06:38.304 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records out 2024-01-15T18:06:38.304 INFO:teuthology.orchestra.run.smithi027.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.78068 s, 158 MB/s 2024-01-15T18:06:38.305 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 2 2024-01-15T18:06:38.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.441+0000 7f09d4600640 1 Processor -- start 2024-01-15T18:06:38.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.441+0000 7f09d4600640 1 -- start start 2024-01-15T18:06:38.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.441+0000 7f09d4600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f09cc15d570 0x7f09cc06be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:38.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.441+0000 7f09d4600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f09cc06b810 0x7f09cc06c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:38.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.441+0000 7f09d4600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09cc0b0d70 con 0x7f09cc15d190 2024-01-15T18:06:38.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.441+0000 7f09d4600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09cc0d0540 con 0x7f09cc15d570 2024-01-15T18:06:38.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.441+0000 7f09d4600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc0b0440 con 0x7f09cc06b810 2024-01-15T18:06:38.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.441+0000 7f09d1200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f09cc15d570 0x7f09cc06be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:38.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.441+0000 7f09d1c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f09cc15d190 0x7f09cc0cfcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34596/0 (socket says 172.21.15.27:34596) 2024-01-15T18:06:38.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.441+0000 7f09d1c00640 1 -- 172.21.15.27:0/3922495975 learned_addr learned my addr 172.21.15.27:0/3922495975 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:06:38.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.441+0000 7f09d2600640 1 --2- 172.21.15.27:0/3922495975 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f09cc06b810 0x7f09cc06c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:38.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.445+0000 7f09d1200640 1 -- 172.21.15.27:0/3922495975 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f09cc06b810 msgr2=0x7f09cc06c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:38.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.445+0000 7f09d1200640 1 --2- 172.21.15.27:0/3922495975 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f09cc06b810 0x7f09cc06c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:38.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.445+0000 7f09d1200640 1 -- 172.21.15.27:0/3922495975 >> v1:172.21.15.27:6789/0 conn(0x7f09cc15d190 legacy=0x7f09cc0cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:38.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.445+0000 7f09cbe00640 1 -- 172.21.15.27:0/3922495975 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 513954798 0 0) 0x7f09cc0b0d70 con 0x7f09cc15d190 2024-01-15T18:06:38.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.445+0000 7f09d1200640 1 -- 172.21.15.27:0/3922495975 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09cc0b0720 con 0x7f09cc15d570 2024-01-15T18:06:38.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.445+0000 7f09d1200640 1 --2- 172.21.15.27:0/3922495975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09cc15d570 0x7f09cc06be10 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f09b40097c0 tx=0x7f09b401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2f36305c60a7d14 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:38.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.445+0000 7f09cbe00640 1 -- 172.21.15.27:0/3922495975 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b40ab670 con 0x7f09cc15d570 2024-01-15T18:06:38.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.445+0000 7f09cbe00640 1 -- 172.21.15.27:0/3922495975 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b40ab7d0 con 0x7f09cc15d570 2024-01-15T18:06:38.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.445+0000 7f09cbe00640 1 -- 172.21.15.27:0/3922495975 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b40abb60 con 0x7f09cc15d570 2024-01-15T18:06:38.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.449+0000 7f09d4600640 1 -- 172.21.15.27:0/3922495975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09cc15d570 msgr2=0x7f09cc06be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:38.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.449+0000 7f09d4600640 1 --2- 172.21.15.27:0/3922495975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09cc15d570 0x7f09cc06be10 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f09b40097c0 tx=0x7f09b401c450 comp rx=0 tx=0).stop 2024-01-15T18:06:38.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.449+0000 7f09d4600640 1 -- 172.21.15.27:0/3922495975 shutdown_connections 2024-01-15T18:06:38.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.449+0000 7f09d4600640 1 --2- 172.21.15.27:0/3922495975 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f09cc06b810 0x7f09cc06c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:38.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.449+0000 7f09d4600640 1 --2- 172.21.15.27:0/3922495975 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09cc15d570 0x7f09cc06be10 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:38.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.449+0000 7f09d4600640 1 -- 172.21.15.27:0/3922495975 >> 172.21.15.27:0/3922495975 conn(0x7f09cc0d0e70 msgr2=0x7f09cc0d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:38.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.449+0000 7f09d4600640 1 -- 172.21.15.27:0/3922495975 shutdown_connections 2024-01-15T18:06:38.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.449+0000 7f09d4600640 1 -- 172.21.15.27:0/3922495975 wait complete. 2024-01-15T18:06:38.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.449+0000 7f09d4600640 1 Processor -- start 2024-01-15T18:06:38.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.449+0000 7f09d4600640 1 -- start start 2024-01-15T18:06:38.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.449+0000 7f09d4600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09cc06b810 0x7f09cc157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:38.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.449+0000 7f09d4600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f09cc15d190 0x7f09cc14d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:38.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.449+0000 7f09d4600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f09cc15d570 0x7f09cc0c6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:38.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.449+0000 7f09d4600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc155ed0 con 0x7f09cc06b810 2024-01-15T18:06:38.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.449+0000 7f09d4600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc156ca0 con 0x7f09cc15d570 2024-01-15T18:06:38.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.449+0000 7f09d4600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc156120 con 0x7f09cc15d190 2024-01-15T18:06:38.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.449+0000 7f09d1c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09cc06b810 0x7f09cc157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:38.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.449+0000 7f09d1c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09cc06b810 0x7f09cc157360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47260/0 (socket says 172.21.15.27:47260) 2024-01-15T18:06:38.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.449+0000 7f09d1c00640 1 -- 172.21.15.27:0/1475186964 learned_addr learned my addr 172.21.15.27:0/1475186964 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:38.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.449+0000 7f09d2600640 1 --2- 172.21.15.27:0/1475186964 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f09cc15d570 0x7f09cc0c6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:38.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.449+0000 7f09d1200640 1 --2- 172.21.15.27:0/1475186964 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f09cc15d190 0x7f09cc14d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:38.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.453+0000 7f09d1c00640 1 -- 172.21.15.27:0/1475186964 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f09cc15d190 msgr2=0x7f09cc14d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:38.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.453+0000 7f09d1c00640 1 --2- 172.21.15.27:0/1475186964 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f09cc15d190 0x7f09cc14d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:38.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.453+0000 7f09d1c00640 1 -- 172.21.15.27:0/1475186964 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f09cc15d570 msgr2=0x7f09cc0c6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:38.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.453+0000 7f09d1c00640 1 --2- 172.21.15.27:0/1475186964 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f09cc15d570 0x7f09cc0c6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:38.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.453+0000 7f09d1c00640 1 -- 172.21.15.27:0/1475186964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b4009420 con 0x7f09cc06b810 2024-01-15T18:06:38.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.453+0000 7f09d1c00640 1 --2- 172.21.15.27:0/1475186964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09cc06b810 0x7f09cc157360 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f09bc0011f0 tx=0x7f09bc00c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:38.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.453+0000 7f09c9600640 1 -- 172.21.15.27:0/1475186964 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc0074e0 con 0x7f09cc06b810 2024-01-15T18:06:38.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.453+0000 7f09c9600640 1 -- 172.21.15.27:0/1475186964 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09bc007640 con 0x7f09cc06b810 2024-01-15T18:06:38.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.453+0000 7f09c9600640 1 -- 172.21.15.27:0/1475186964 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc004a10 con 0x7f09cc06b810 2024-01-15T18:06:38.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.453+0000 7f09d4600640 1 -- 172.21.15.27:0/1475186964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09cc156400 con 0x7f09cc06b810 2024-01-15T18:06:38.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.453+0000 7f09d4600640 1 -- 172.21.15.27:0/1475186964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09cc1569c0 con 0x7f09cc06b810 2024-01-15T18:06:38.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.453+0000 7f09d4600640 1 -- 172.21.15.27:0/1475186964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09cc0d0c90 con 0x7f09cc06b810 2024-01-15T18:06:38.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.453+0000 7f09c9600640 1 -- 172.21.15.27:0/1475186964 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 6) v1 ==== 86211+0+0 (secure 0 0 0) 0x7f09bc004b70 con 0x7f09cc06b810 2024-01-15T18:06:38.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.453+0000 7f09c9600640 1 --2- 172.21.15.27:0/1475186964 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f09a40669b0 0x7f09a4068e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:38.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.453+0000 7f09c9600640 1 -- 172.21.15.27:0/1475186964 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09bc0835c0 con 0x7f09cc06b810 2024-01-15T18:06:38.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.453+0000 7f09d1200640 1 --2- 172.21.15.27:0/1475186964 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f09a40669b0 0x7f09a4068e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:38.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.457+0000 7f09d1200640 1 --2- 172.21.15.27:0/1475186964 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f09a40669b0 0x7f09a4068e70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f09b40af000 tx=0x7f09b40b6040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:38.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.461+0000 7f09c9600640 1 -- 172.21.15.27:0/1475186964 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f09bc050850 con 0x7f09cc06b810 2024-01-15T18:06:38.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.617+0000 7f09d4600640 1 -- 172.21.15.27:0/1475186964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f09cc0b11e0 con 0x7f09cc06b810 2024-01-15T18:06:38.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.621+0000 7f09c9600640 1 -- 172.21.15.27:0/1475186964 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v11)=0 v11) v1 ==== 135+0+0 (secure 0 0 0) 0x7f09cc0b11e0 con 0x7f09cc06b810 2024-01-15T18:06:38.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.621+0000 7f09d4600640 1 -- 172.21.15.27:0/1475186964 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f09a40669b0 msgr2=0x7f09a4068e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:38.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.621+0000 7f09d4600640 1 --2- 172.21.15.27:0/1475186964 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f09a40669b0 0x7f09a4068e70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f09b40af000 tx=0x7f09b40b6040 comp rx=0 tx=0).stop 2024-01-15T18:06:38.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.621+0000 7f09d4600640 1 -- 172.21.15.27:0/1475186964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09cc06b810 msgr2=0x7f09cc157360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:38.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.621+0000 7f09d4600640 1 --2- 172.21.15.27:0/1475186964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09cc06b810 0x7f09cc157360 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f09bc0011f0 tx=0x7f09bc00c9b0 comp rx=0 tx=0).stop 2024-01-15T18:06:38.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.621+0000 7f09d4600640 1 -- 172.21.15.27:0/1475186964 shutdown_connections 2024-01-15T18:06:38.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.621+0000 7f09d4600640 1 --2- 172.21.15.27:0/1475186964 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f09a40669b0 0x7f09a4068e70 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:38.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.621+0000 7f09d4600640 1 --2- 172.21.15.27:0/1475186964 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f09cc15d570 0x7f09cc0c6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:38.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.621+0000 7f09d4600640 1 --2- 172.21.15.27:0/1475186964 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f09cc15d190 0x7f09cc14d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:38.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.621+0000 7f09d4600640 1 --2- 172.21.15.27:0/1475186964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09cc06b810 0x7f09cc157360 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:38.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.621+0000 7f09d4600640 1 -- 172.21.15.27:0/1475186964 >> 172.21.15.27:0/1475186964 conn(0x7f09cc0d0e70 msgr2=0x7f09cc158430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:38.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.621+0000 7f09d4600640 1 -- 172.21.15.27:0/1475186964 shutdown_connections 2024-01-15T18:06:38.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.621+0000 7f09d4600640 1 -- 172.21.15.27:0/1475186964 wait complete. 2024-01-15T18:06:38.649 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-15T18:06:38.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.789+0000 7fa2c0600640 1 Processor -- start 2024-01-15T18:06:38.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.789+0000 7fa2c0600640 1 -- start start 2024-01-15T18:06:38.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.789+0000 7fa2c0600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa2b806c740 0x7fa2b8099a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:38.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.789+0000 7fa2c0600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa2b8099f50 0x7fa2b80cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:38.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.789+0000 7fa2c0600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b815d860 con 0x7fa2b815d480 2024-01-15T18:06:38.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.789+0000 7fa2c0600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b809a3e0 con 0x7fa2b806c740 2024-01-15T18:06:38.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.789+0000 7fa2c0600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b806b6a0 con 0x7fa2b8099f50 2024-01-15T18:06:38.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.789+0000 7fa2bdc00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa2b815d480 0x7fa2b80cfbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34598/0 (socket says 172.21.15.27:34598) 2024-01-15T18:06:38.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.789+0000 7fa2bdc00640 1 -- 172.21.15.27:0/1511309238 learned_addr learned my addr 172.21.15.27:0/1511309238 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:06:38.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.789+0000 7fa2bd200640 1 --2- 172.21.15.27:0/1511309238 >> v2:172.21.15.93:3300/0 conn(0x7fa2b806c740 0x7fa2b8099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:38.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.789+0000 7fa2be600640 1 --2- 172.21.15.27:0/1511309238 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa2b8099f50 0x7fa2b80cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:38.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.789+0000 7fa2b7e00640 1 -- 172.21.15.27:0/1511309238 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411591776 0 0) 0x7fa2b815d860 con 0x7fa2b815d480 2024-01-15T18:06:38.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.789+0000 7fa2b7e00640 1 -- 172.21.15.27:0/1511309238 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa298003530 con 0x7fa2b815d480 2024-01-15T18:06:38.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.789+0000 7fa2bd200640 1 -- 172.21.15.27:0/1511309238 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa2b8099f50 msgr2=0x7fa2b80cb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:38.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.789+0000 7fa2bd200640 1 --2- 172.21.15.27:0/1511309238 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa2b8099f50 0x7fa2b80cb5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:38.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.789+0000 7fa2bd200640 1 -- 172.21.15.27:0/1511309238 >> v1:172.21.15.27:6789/0 conn(0x7fa2b815d480 legacy=0x7fa2b80cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:38.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.789+0000 7fa2bd200640 1 -- 172.21.15.27:0/1511309238 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b806b980 con 0x7fa2b806c740 2024-01-15T18:06:38.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.793+0000 7fa2bd200640 1 --2- 172.21.15.27:0/1511309238 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa2b806c740 0x7fa2b8099a10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa2a00097f0 tx=0x7fa2a001c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67c14667e4e744bc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:38.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.793+0000 7fa2b7e00640 1 -- 172.21.15.27:0/1511309238 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a00ab600 con 0x7fa2b806c740 2024-01-15T18:06:38.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.793+0000 7fa2b7e00640 1 -- 172.21.15.27:0/1511309238 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a00a7050 con 0x7fa2b806c740 2024-01-15T18:06:38.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.793+0000 7fa2b7e00640 1 -- 172.21.15.27:0/1511309238 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a00ab980 con 0x7fa2b806c740 2024-01-15T18:06:38.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.793+0000 7fa2c0600640 1 -- 172.21.15.27:0/1511309238 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa2b806c740 msgr2=0x7fa2b8099a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:38.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.793+0000 7fa2c0600640 1 --2- 172.21.15.27:0/1511309238 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa2b806c740 0x7fa2b8099a10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa2a00097f0 tx=0x7fa2a001c450 comp rx=0 tx=0).stop 2024-01-15T18:06:38.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.793+0000 7fa2c0600640 1 -- 172.21.15.27:0/1511309238 shutdown_connections 2024-01-15T18:06:38.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.793+0000 7fa2c0600640 1 --2- 172.21.15.27:0/1511309238 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa2b8099f50 0x7fa2b80cb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:38.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.793+0000 7fa2c0600640 1 --2- 172.21.15.27:0/1511309238 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa2b806c740 0x7fa2b8099a10 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:38.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.793+0000 7fa2c0600640 1 -- 172.21.15.27:0/1511309238 >> 172.21.15.27:0/1511309238 conn(0x7fa2b80b2a80 msgr2=0x7fa2b80b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:38.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.793+0000 7fa2c0600640 1 -- 172.21.15.27:0/1511309238 shutdown_connections 2024-01-15T18:06:38.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.793+0000 7fa2c0600640 1 -- 172.21.15.27:0/1511309238 wait complete. 2024-01-15T18:06:38.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.797+0000 7fa2c0600640 1 Processor -- start 2024-01-15T18:06:38.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.797+0000 7fa2c0600640 1 -- start start 2024-01-15T18:06:38.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.797+0000 7fa2c0600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2b806c740 0x7fa2b80b4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:38.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.797+0000 7fa2c0600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa2b8099f50 0x7fa2b8152e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:38.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.797+0000 7fa2c0600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa2b815d480 0x7fa2b80b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:38.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.797+0000 7fa2c0600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b80b4980 con 0x7fa2b806c740 2024-01-15T18:06:38.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.797+0000 7fa2c0600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b80c7850 con 0x7fa2b8099f50 2024-01-15T18:06:38.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.797+0000 7fa2c0600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b8153360 con 0x7fa2b815d480 2024-01-15T18:06:38.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.797+0000 7fa2bdc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2b806c740 0x7fa2b80b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:38.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.797+0000 7fa2bdc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2b806c740 0x7fa2b80b4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47274/0 (socket says 172.21.15.27:47274) 2024-01-15T18:06:38.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.797+0000 7fa2bdc00640 1 -- 172.21.15.27:0/438321840 learned_addr learned my addr 172.21.15.27:0/438321840 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:38.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.797+0000 7fa2be600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa2b815d480 0x7fa2b80b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:38.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.797+0000 7fa2bd200640 1 --2- 172.21.15.27:0/438321840 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa2b8099f50 0x7fa2b8152e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:38.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.797+0000 7fa2be600640 1 -- 172.21.15.27:0/438321840 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa2b8099f50 msgr2=0x7fa2b8152e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:38.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.797+0000 7fa2be600640 1 --2- 172.21.15.27:0/438321840 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa2b8099f50 0x7fa2b8152e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:38.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.797+0000 7fa2be600640 1 -- 172.21.15.27:0/438321840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2b806c740 msgr2=0x7fa2b80b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:38.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.797+0000 7fa2be600640 1 --2- 172.21.15.27:0/438321840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2b806c740 0x7fa2b80b4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:38.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.797+0000 7fa2be600640 1 -- 172.21.15.27:0/438321840 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a0009420 con 0x7fa2b815d480 2024-01-15T18:06:38.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.797+0000 7fa2bdc00640 1 --2- 172.21.15.27:0/438321840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2b806c740 0x7fa2b80b4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:06:38.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.797+0000 7fa2be600640 1 --2- 172.21.15.27:0/438321840 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa2b815d480 0x7fa2b80b58f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa2b000efc0 tx=0x7fa2b000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:38.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.797+0000 7fa2b5600640 1 -- 172.21.15.27:0/438321840 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b0009c00 con 0x7fa2b815d480 2024-01-15T18:06:38.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.797+0000 7fa2b5600640 1 -- 172.21.15.27:0/438321840 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2b000f040 con 0x7fa2b815d480 2024-01-15T18:06:38.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.797+0000 7fa2b5600640 1 -- 172.21.15.27:0/438321840 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b00146c0 con 0x7fa2b815d480 2024-01-15T18:06:38.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.797+0000 7fa2c0600640 1 -- 172.21.15.27:0/438321840 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b80b4b70 con 0x7fa2b815d480 2024-01-15T18:06:38.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.797+0000 7fa2c0600640 1 -- 172.21.15.27:0/438321840 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b80c49b0 con 0x7fa2b815d480 2024-01-15T18:06:38.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.801+0000 7fa2c0600640 1 -- 172.21.15.27:0/438321840 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2b804e850 con 0x7fa2b815d480 2024-01-15T18:06:38.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.801+0000 7fa2b5600640 1 -- 172.21.15.27:0/438321840 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 6) v1 ==== 86211+0+0 (secure 0 0 0) 0x7fa2b0014820 con 0x7fa2b815d480 2024-01-15T18:06:38.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.801+0000 7fa2b5600640 1 --2- 172.21.15.27:0/438321840 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa290066980 0x7fa290068e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:38.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.801+0000 7fa2b5600640 1 -- 172.21.15.27:0/438321840 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2b00853c0 con 0x7fa2b815d480 2024-01-15T18:06:38.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.801+0000 7fa2bdc00640 1 --2- 172.21.15.27:0/438321840 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa290066980 0x7fa290068e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:38.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.801+0000 7fa2bdc00640 1 --2- 172.21.15.27:0/438321840 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa290066980 0x7fa290068e40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa2a80011f0 tx=0x7fa2a8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:38.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.805+0000 7fa2b5600640 1 -- 172.21.15.27:0/438321840 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa2b00524b0 con 0x7fa2b815d480 2024-01-15T18:06:39.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:38.993+0000 7fa2c0600640 1 -- 172.21.15.27:0/438321840 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2b8067580 con 0x7fa290066980 2024-01-15T18:06:39.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.009+0000 7fa2b5600640 1 -- 172.21.15.27:0/438321840 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa2b8067580 con 0x7fa290066980 2024-01-15T18:06:39.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.013+0000 7fa2c0600640 1 -- 172.21.15.27:0/438321840 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa290066980 msgr2=0x7fa290068e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:39.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.013+0000 7fa2c0600640 1 --2- 172.21.15.27:0/438321840 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa290066980 0x7fa290068e40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa2a80011f0 tx=0x7fa2a8009040 comp rx=0 tx=0).stop 2024-01-15T18:06:39.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.013+0000 7fa2c0600640 1 -- 172.21.15.27:0/438321840 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa2b815d480 msgr2=0x7fa2b80b58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:39.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.013+0000 7fa2c0600640 1 --2- 172.21.15.27:0/438321840 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa2b815d480 0x7fa2b80b58f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa2b000efc0 tx=0x7fa2b000c490 comp rx=0 tx=0).stop 2024-01-15T18:06:39.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.013+0000 7fa2c0600640 1 -- 172.21.15.27:0/438321840 shutdown_connections 2024-01-15T18:06:39.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.013+0000 7fa2c0600640 1 --2- 172.21.15.27:0/438321840 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa290066980 0x7fa290068e40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:39.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.013+0000 7fa2c0600640 1 --2- 172.21.15.27:0/438321840 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa2b815d480 0x7fa2b80b58f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:39.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.013+0000 7fa2c0600640 1 --2- 172.21.15.27:0/438321840 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa2b8099f50 0x7fa2b8152e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:39.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.013+0000 7fa2c0600640 1 --2- 172.21.15.27:0/438321840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2b806c740 0x7fa2b80b4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:39.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.013+0000 7fa2c0600640 1 -- 172.21.15.27:0/438321840 >> 172.21.15.27:0/438321840 conn(0x7fa2b80b2a80 msgr2=0x7fa2b815c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:39.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.013+0000 7fa2c0600640 1 -- 172.21.15.27:0/438321840 shutdown_connections 2024-01-15T18:06:39.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.013+0000 7fa2c0600640 1 -- 172.21.15.27:0/438321840 wait complete. 2024-01-15T18:06:39.039 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:06:39.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.177+0000 7f567a800640 1 Processor -- start 2024-01-15T18:06:39.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.177+0000 7f567a800640 1 -- start start 2024-01-15T18:06:39.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.177+0000 7f567a800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f567415d680 0x7f56740cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:39.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.181+0000 7f567a800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f56740cfc30 0x7f56740cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:39.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.181+0000 7f567a800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56740d19a0 0x7f567406b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:39.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.181+0000 7f567a800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56740cb4d0 con 0x7f567415d680 2024-01-15T18:06:39.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.181+0000 7f567a800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56740d0120 con 0x7f56740cfc30 2024-01-15T18:06:39.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.181+0000 7f567a800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56740d0400 con 0x7f56740d19a0 2024-01-15T18:06:39.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.181+0000 7f5673400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f567415d680 0x7f56740cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:39.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.181+0000 7f5673400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f567415d680 0x7f56740cd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47284/0 (socket says 172.21.15.27:47284) 2024-01-15T18:06:39.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.181+0000 7f5673400640 1 -- 172.21.15.27:0/1167458174 learned_addr learned my addr 172.21.15.27:0/1167458174 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:39.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.181+0000 7f5672a00640 1 --2- 172.21.15.27:0/1167458174 >> v2:172.21.15.93:3300/0 conn(0x7f56740cfc30 0x7f56740cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:39.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.181+0000 7f5673e00640 1 --2- 172.21.15.27:0/1167458174 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56740d19a0 0x7f567406b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.181+0000 7f5673400640 1 -- 172.21.15.27:0/1167458174 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56740d19a0 msgr2=0x7f567406b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.181+0000 7f5673400640 1 --2- 172.21.15.27:0/1167458174 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56740d19a0 0x7f567406b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.181+0000 7f5673400640 1 -- 172.21.15.27:0/1167458174 >> v2:172.21.15.93:3300/0 conn(0x7f56740cfc30 msgr2=0x7f56740cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.181+0000 7f5673400640 1 --2- 172.21.15.27:0/1167458174 >> v2:172.21.15.93:3300/0 conn(0x7f56740cfc30 0x7f56740cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.181+0000 7f5673400640 1 -- 172.21.15.27:0/1167458174 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56740cb770 con 0x7f567415d680 2024-01-15T18:06:39.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.181+0000 7f5672a00640 1 --2- 172.21.15.27:0/1167458174 >> v2:172.21.15.93:3300/0 conn(0x7f56740cfc30 0x7f56740cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:06:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.181+0000 7f5673400640 1 --2- 172.21.15.27:0/1167458174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f567415d680 0x7f56740cd700 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f567406afb0 tx=0x7f566401c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=641e286fdf6a6dab server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.181+0000 7f5672000640 1 -- 172.21.15.27:0/1167458174 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56740ce850 con 0x7f567415d680 2024-01-15T18:06:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.181+0000 7f5672000640 1 -- 172.21.15.27:0/1167458174 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56640a7050 con 0x7f567415d680 2024-01-15T18:06:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.181+0000 7f5672000640 1 -- 172.21.15.27:0/1167458174 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56640ab820 con 0x7f567415d680 2024-01-15T18:06:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.185+0000 7f567a800640 1 -- 172.21.15.27:0/1167458174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f567415d680 msgr2=0x7f56740cd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.185+0000 7f567a800640 1 --2- 172.21.15.27:0/1167458174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f567415d680 0x7f56740cd700 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f567406afb0 tx=0x7f566401c210 comp rx=0 tx=0).stop 2024-01-15T18:06:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.185+0000 7f567a800640 1 -- 172.21.15.27:0/1167458174 shutdown_connections 2024-01-15T18:06:39.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.185+0000 7f567a800640 1 --2- 172.21.15.27:0/1167458174 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56740d19a0 0x7f567406b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:39.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.185+0000 7f567a800640 1 --2- 172.21.15.27:0/1167458174 >> v2:172.21.15.93:3300/0 conn(0x7f56740cfc30 0x7f56740cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:39.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.185+0000 7f567a800640 1 --2- 172.21.15.27:0/1167458174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f567415d680 0x7f56740cd700 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:39.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.185+0000 7f567a800640 1 -- 172.21.15.27:0/1167458174 >> 172.21.15.27:0/1167458174 conn(0x7f56740d0e80 msgr2=0x7f56740d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:39.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.185+0000 7f567a800640 1 -- 172.21.15.27:0/1167458174 shutdown_connections 2024-01-15T18:06:39.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.185+0000 7f567a800640 1 -- 172.21.15.27:0/1167458174 wait complete. 2024-01-15T18:06:39.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.185+0000 7f567a800640 1 Processor -- start 2024-01-15T18:06:39.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.185+0000 7f567a800640 1 -- start start 2024-01-15T18:06:39.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.185+0000 7f567a800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56740cfc30 0x7f5674146290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:39.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.185+0000 7f567a800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56740d19a0 0x7f5674156c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:39.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.185+0000 7f567a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f567415d680 0x7f567414dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:39.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.185+0000 7f567a800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5674146110 con 0x7f567415d680 2024-01-15T18:06:39.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.185+0000 7f567a800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56741551f0 con 0x7f56740d19a0 2024-01-15T18:06:39.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.185+0000 7f567a800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56741554d0 con 0x7f56740cfc30 2024-01-15T18:06:39.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.189+0000 7f5673400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56740cfc30 0x7f5674146290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:39.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.189+0000 7f5673400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56740cfc30 0x7f5674146290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:53346/0 (socket says 172.21.15.27:53346) 2024-01-15T18:06:39.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.189+0000 7f5673400640 1 -- 172.21.15.27:0/2535884486 learned_addr learned my addr 172.21.15.27:0/2535884486 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:39.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.189+0000 7f5672a00640 1 --2- 172.21.15.27:0/2535884486 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56740d19a0 0x7f5674156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:39.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.189+0000 7f5673e00640 1 --2- 172.21.15.27:0/2535884486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f567415d680 0x7f567414dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:39.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.189+0000 7f5673400640 1 -- 172.21.15.27:0/2535884486 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56740d19a0 msgr2=0x7f5674156c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:39.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.189+0000 7f5673400640 1 --2- 172.21.15.27:0/2535884486 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56740d19a0 0x7f5674156c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:39.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.189+0000 7f5673400640 1 -- 172.21.15.27:0/2535884486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f567415d680 msgr2=0x7f567414dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:39.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.189+0000 7f5673400640 1 --2- 172.21.15.27:0/2535884486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f567415d680 0x7f567414dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:39.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.189+0000 7f5673400640 1 -- 172.21.15.27:0/2535884486 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56640094d0 con 0x7f56740cfc30 2024-01-15T18:06:39.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.189+0000 7f5673400640 1 --2- 172.21.15.27:0/2535884486 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56740cfc30 0x7f5674146290 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f566401a8c0 tx=0x7f5664018fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:39.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.189+0000 7f566b400640 1 -- 172.21.15.27:0/2535884486 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56640b4420 con 0x7f56740cfc30 2024-01-15T18:06:39.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.189+0000 7f566b400640 1 -- 172.21.15.27:0/2535884486 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56640b4580 con 0x7f56740cfc30 2024-01-15T18:06:39.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.189+0000 7f566b400640 1 -- 172.21.15.27:0/2535884486 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56640b4930 con 0x7f56740cfc30 2024-01-15T18:06:39.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.189+0000 7f567a800640 1 -- 172.21.15.27:0/2535884486 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56741557b0 con 0x7f56740cfc30 2024-01-15T18:06:39.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.189+0000 7f567a800640 1 -- 172.21.15.27:0/2535884486 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f567414e200 con 0x7f56740cfc30 2024-01-15T18:06:39.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.189+0000 7f567a800640 1 -- 172.21.15.27:0/2535884486 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f567414e3f0 con 0x7f56740cfc30 2024-01-15T18:06:39.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.189+0000 7f566b400640 1 -- 172.21.15.27:0/2535884486 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 6) v1 ==== 86211+0+0 (secure 0 0 0) 0x7f56640b2070 con 0x7f56740cfc30 2024-01-15T18:06:39.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.193+0000 7f566b400640 1 --2- 172.21.15.27:0/2535884486 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f564c066980 0x7f564c068e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:39.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.193+0000 7f566b400640 1 -- 172.21.15.27:0/2535884486 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f566411d680 con 0x7f56740cfc30 2024-01-15T18:06:39.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.193+0000 7f5672a00640 1 --2- 172.21.15.27:0/2535884486 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f564c066980 0x7f564c068e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:39.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.193+0000 7f5672a00640 1 --2- 172.21.15.27:0/2535884486 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f564c066980 0x7f564c068e40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f56740ce0f0 tx=0x7f565c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:39.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.193+0000 7f566b400640 1 -- 172.21.15.27:0/2535884486 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f567414e3f0 con 0x7f56740cfc30 2024-01-15T18:06:39.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.381+0000 7f567a800640 1 -- 172.21.15.27:0/2535884486 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f567414e3f0 con 0x7f564c066980 2024-01-15T18:06:39.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.385+0000 7f566b400640 1 -- 172.21.15.27:0/2535884486 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f567414e3f0 con 0x7f564c066980 2024-01-15T18:06:39.393 INFO:teuthology.orchestra.run.smithi027.stdout:/volumes/_nogroup/sv1/6d734a73-9593-40e4-bc54-648b1a9e9fa0 2024-01-15T18:06:39.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.385+0000 7f567a800640 1 -- 172.21.15.27:0/2535884486 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f564c066980 msgr2=0x7f564c068e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:39.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.385+0000 7f567a800640 1 --2- 172.21.15.27:0/2535884486 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f564c066980 0x7f564c068e40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f56740ce0f0 tx=0x7f565c008040 comp rx=0 tx=0).stop 2024-01-15T18:06:39.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.385+0000 7f567a800640 1 -- 172.21.15.27:0/2535884486 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56740cfc30 msgr2=0x7f5674146290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:39.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.385+0000 7f567a800640 1 --2- 172.21.15.27:0/2535884486 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56740cfc30 0x7f5674146290 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f566401a8c0 tx=0x7f5664018fb0 comp rx=0 tx=0).stop 2024-01-15T18:06:39.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.385+0000 7f567a800640 1 -- 172.21.15.27:0/2535884486 shutdown_connections 2024-01-15T18:06:39.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.385+0000 7f567a800640 1 --2- 172.21.15.27:0/2535884486 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f564c066980 0x7f564c068e40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:39.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.385+0000 7f567a800640 1 --2- 172.21.15.27:0/2535884486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f567415d680 0x7f567414dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:39.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.385+0000 7f567a800640 1 --2- 172.21.15.27:0/2535884486 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56740d19a0 0x7f5674156c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:39.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.385+0000 7f567a800640 1 --2- 172.21.15.27:0/2535884486 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56740cfc30 0x7f5674146290 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:39.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.389+0000 7f567a800640 1 -- 172.21.15.27:0/2535884486 >> 172.21.15.27:0/2535884486 conn(0x7f56740d0e80 msgr2=0x7f56740aee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:39.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.389+0000 7f567a800640 1 -- 172.21.15.27:0/2535884486 shutdown_connections 2024-01-15T18:06:39.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:39.389+0000 7f567a800640 1 -- 172.21.15.27:0/2535884486 wait complete. 2024-01-15T18:06:39.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:39.414 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/6d734a73-9593-40e4-bc54-648b1a9e9fa0) 2024-01-15T18:06:39.664 INFO:teuthology.orchestra.run.smithi027.stdout:# file: volumes/_nogroup/sv1/6d734a73-9593-40e4-bc54-648b1a9e9fa0 2024-01-15T18:06:39.664 INFO:teuthology.orchestra.run.smithi027.stdout:ceph.dir.rbytes="3138256896" 2024-01-15T18:06:39.664 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:06:39.665 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-15T18:06:41.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:41.666 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/6d734a73-9593-40e4-bc54-648b1a9e9fa0) 2024-01-15T18:06:41.680 INFO:teuthology.orchestra.run.smithi027.stdout:# file: volumes/_nogroup/sv1/6d734a73-9593-40e4-bc54-648b1a9e9fa0 2024-01-15T18:06:41.681 INFO:teuthology.orchestra.run.smithi027.stdout:ceph.dir.rbytes="3138256896" 2024-01-15T18:06:41.681 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:06:41.682 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-15T18:06:43.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:43.683 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/6d734a73-9593-40e4-bc54-648b1a9e9fa0) 2024-01-15T18:06:43.698 INFO:teuthology.orchestra.run.smithi027.stdout:# file: volumes/_nogroup/sv1/6d734a73-9593-40e4-bc54-648b1a9e9fa0 2024-01-15T18:06:43.698 INFO:teuthology.orchestra.run.smithi027.stdout:ceph.dir.rbytes="3221225472" 2024-01-15T18:06:43.698 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:06:43.699 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-15T18:06:43.700 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 ss1clone1 2024-01-15T18:06:43.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.893+0000 7f34a7800640 1 Processor -- start 2024-01-15T18:06:43.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.893+0000 7f34a7800640 1 -- start start 2024-01-15T18:06:43.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.893+0000 7f34a7800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34a015d190 0x7f34a00cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:43.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.893+0000 7f34a7800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a00b0d70 con 0x7f34a015d570 2024-01-15T18:06:43.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.893+0000 7f34a7800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a00d0540 con 0x7f34a006c520 2024-01-15T18:06:43.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.893+0000 7f34a7800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a00b0440 con 0x7f34a015d190 2024-01-15T18:06:43.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.893+0000 7f349fe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f34a015d570 0x7f34a006be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36834/0 (socket says 172.21.15.27:36834) 2024-01-15T18:06:43.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.893+0000 7f349fe00640 1 -- 172.21.15.27:0/1857270088 learned_addr learned my addr 172.21.15.27:0/1857270088 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:06:43.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.893+0000 7f34a4e00640 1 --2- 172.21.15.27:0/1857270088 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34a015d190 0x7f34a00cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:43.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.893+0000 7f349f400640 1 -- 172.21.15.27:0/1857270088 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354710100 0 0) 0x7f34a00b0d70 con 0x7f34a015d570 2024-01-15T18:06:43.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.893+0000 7f349f400640 1 -- 172.21.15.27:0/1857270088 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3480003530 con 0x7f34a015d570 2024-01-15T18:06:43.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.893+0000 7f349f400640 1 -- 172.21.15.27:0/1857270088 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 210432510 0 0) 0x7f34a00d0540 con 0x7f34a006c520 2024-01-15T18:06:43.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.893+0000 7f349f400640 1 -- 172.21.15.27:0/1857270088 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34a00b0d70 con 0x7f34a006c520 2024-01-15T18:06:43.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.893+0000 7f349f400640 1 -- 172.21.15.27:0/1857270088 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 780509683 0 0) 0x7f3480003530 con 0x7f34a015d570 2024-01-15T18:06:43.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.893+0000 7f349f400640 1 -- 172.21.15.27:0/1857270088 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34a015d190 msgr2=0x7f34a00cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:43.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.893+0000 7f349f400640 1 --2- 172.21.15.27:0/1857270088 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34a015d190 0x7f34a00cfcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:43.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.893+0000 7f349f400640 1 -- 172.21.15.27:0/1857270088 >> v1:172.21.15.93:6789/0 conn(0x7f34a006c520 legacy=0x7f34a00cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:43.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.893+0000 7f349f400640 1 -- 172.21.15.27:0/1857270088 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a00b0720 con 0x7f34a015d570 2024-01-15T18:06:43.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.893+0000 7f349f400640 1 -- 172.21.15.27:0/1857270088 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f3488001680 con 0x7f34a015d570 2024-01-15T18:06:43.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.893+0000 7f349f400640 1 -- 172.21.15.27:0/1857270088 >> v1:172.21.15.27:6789/0 conn(0x7f34a015d570 legacy=0x7f34a006be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:43.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.893+0000 7f349f400640 1 --2- 172.21.15.27:0/1857270088 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34a015d190 0x7f3480003bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:43.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.893+0000 7f349f400640 1 --2- 172.21.15.27:0/1857270088 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3480004110 0x7f34800047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:43.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.897+0000 7f349f400640 1 --2- 172.21.15.27:0/1857270088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3480004d10 0x7f3480005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:43.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.897+0000 7f349f400640 1 -- 172.21.15.27:0/1857270088 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a00b0440 con 0x7f3480004d10 2024-01-15T18:06:43.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.897+0000 7f349f400640 1 -- 172.21.15.27:0/1857270088 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348000bad0 con 0x7f3480004110 2024-01-15T18:06:43.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.897+0000 7f349f400640 1 -- 172.21.15.27:0/1857270088 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f348000bc90 con 0x7f34a015d190 2024-01-15T18:06:43.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.897+0000 7f34a5800640 1 --2- 172.21.15.27:0/1857270088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3480004d10 0x7f3480005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:43.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.897+0000 7f34a4e00640 1 --2- 172.21.15.27:0/1857270088 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34a015d190 0x7f3480003bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:43.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.897+0000 7f349fe00640 1 --2- 172.21.15.27:0/1857270088 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3480004110 0x7f34800047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:43.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.897+0000 7f34a5800640 1 -- 172.21.15.27:0/1857270088 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34a015d190 msgr2=0x7f3480003bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:43.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.897+0000 7f34a5800640 1 --2- 172.21.15.27:0/1857270088 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34a015d190 0x7f3480003bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:43.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.897+0000 7f34a5800640 1 -- 172.21.15.27:0/1857270088 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3480004110 msgr2=0x7f34800047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:43.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.897+0000 7f34a5800640 1 --2- 172.21.15.27:0/1857270088 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3480004110 0x7f34800047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:43.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.897+0000 7f34a5800640 1 -- 172.21.15.27:0/1857270088 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3480003530 con 0x7f3480004d10 2024-01-15T18:06:43.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.897+0000 7f34a5800640 1 --2- 172.21.15.27:0/1857270088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3480004d10 0x7f3480005190 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f34a0071800 tx=0x7f349801c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1011b8a3b6c56a4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:43.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.897+0000 7f349f400640 1 -- 172.21.15.27:0/1857270088 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3488007ea0 con 0x7f3480004d10 2024-01-15T18:06:43.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.897+0000 7f349f400640 1 -- 172.21.15.27:0/1857270088 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3498003ba0 con 0x7f3480004d10 2024-01-15T18:06:43.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.897+0000 7f349f400640 1 -- 172.21.15.27:0/1857270088 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34980050d0 con 0x7f3480004d10 2024-01-15T18:06:43.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.901+0000 7f34a7800640 1 -- 172.21.15.27:0/1857270088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3480004d10 msgr2=0x7f3480005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:43.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.901+0000 7f34a7800640 1 --2- 172.21.15.27:0/1857270088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3480004d10 0x7f3480005190 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f34a0071800 tx=0x7f349801c8c0 comp rx=0 tx=0).stop 2024-01-15T18:06:43.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.901+0000 7f34a5800640 1 -- 172.21.15.27:0/1857270088 reap_dead start 2024-01-15T18:06:43.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.901+0000 7f34a7800640 1 -- 172.21.15.27:0/1857270088 shutdown_connections 2024-01-15T18:06:43.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.901+0000 7f34a7800640 1 -- 172.21.15.27:0/1857270088 >> 172.21.15.27:0/1857270088 conn(0x7f34a00d0e70 msgr2=0x7f34a00d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:43.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.901+0000 7f34a7800640 1 -- 172.21.15.27:0/1857270088 shutdown_connections 2024-01-15T18:06:43.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.901+0000 7f34a7800640 1 -- 172.21.15.27:0/1857270088 wait complete. 2024-01-15T18:06:43.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.901+0000 7f34a7800640 1 Processor -- start 2024-01-15T18:06:43.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.901+0000 7f34a7800640 1 -- start start 2024-01-15T18:06:43.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.901+0000 7f34a7800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3480004d10 0x7f34a01f2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:43.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.901+0000 7f34a7800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34a00b57e0 0x7f34a00b5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:43.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.901+0000 7f34a7800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34a00b2960 0x7f34a00b2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:43.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.901+0000 7f34a7800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a00b61a0 con 0x7f34a00b2960 2024-01-15T18:06:43.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.901+0000 7f34a7800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a00b3320 con 0x7f3480004d10 2024-01-15T18:06:43.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.901+0000 7f34a7800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a00c6620 con 0x7f34a00b57e0 2024-01-15T18:06:43.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.901+0000 7f34a5800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34a00b2960 0x7f34a00b2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:43.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.901+0000 7f34a5800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34a00b2960 0x7f34a00b2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40900/0 (socket says 172.21.15.27:40900) 2024-01-15T18:06:43.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.901+0000 7f34a5800640 1 -- 172.21.15.27:0/1530738775 learned_addr learned my addr 172.21.15.27:0/1530738775 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:43.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.901+0000 7f349fe00640 1 --2- 172.21.15.27:0/1530738775 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34a00b57e0 0x7f34a00b5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:43.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.901+0000 7f34a4e00640 1 --2- 172.21.15.27:0/1530738775 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3480004d10 0x7f34a01f2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:43.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.905+0000 7f34a5800640 1 -- 172.21.15.27:0/1530738775 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34a00b57e0 msgr2=0x7f34a00b5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:43.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.905+0000 7f34a5800640 1 --2- 172.21.15.27:0/1530738775 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34a00b57e0 0x7f34a00b5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:43.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.905+0000 7f34a5800640 1 -- 172.21.15.27:0/1530738775 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3480004d10 msgr2=0x7f34a01f2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:43.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.905+0000 7f34a5800640 1 --2- 172.21.15.27:0/1530738775 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3480004d10 0x7f34a01f2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:43.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.905+0000 7f34a5800640 1 -- 172.21.15.27:0/1530738775 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a00c52e0 con 0x7f34a00b2960 2024-01-15T18:06:43.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.905+0000 7f34a5800640 1 --2- 172.21.15.27:0/1530738775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34a00b2960 0x7f34a00b2de0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f349801c6f0 tx=0x7f3480005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:43.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.905+0000 7f349cc00640 1 -- 172.21.15.27:0/1530738775 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34980b4070 con 0x7f34a00b2960 2024-01-15T18:06:43.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.905+0000 7f349cc00640 1 -- 172.21.15.27:0/1530738775 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3498003d10 con 0x7f34a00b2960 2024-01-15T18:06:43.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.905+0000 7f349cc00640 1 -- 172.21.15.27:0/1530738775 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3498006d60 con 0x7f34a00b2960 2024-01-15T18:06:43.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.905+0000 7f34a7800640 1 -- 172.21.15.27:0/1530738775 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a01f2a90 con 0x7f34a00b2960 2024-01-15T18:06:43.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.905+0000 7f34a7800640 1 -- 172.21.15.27:0/1530738775 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a00c5860 con 0x7f34a00b2960 2024-01-15T18:06:43.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.905+0000 7f34a7800640 1 -- 172.21.15.27:0/1530738775 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a004e850 con 0x7f34a00b2960 2024-01-15T18:06:43.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.905+0000 7f349cc00640 1 -- 172.21.15.27:0/1530738775 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 6) v1 ==== 86211+0+0 (secure 0 0 0) 0x7f34980052e0 con 0x7f34a00b2960 2024-01-15T18:06:43.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.909+0000 7f349cc00640 1 --2- 172.21.15.27:0/1530738775 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f34780669b0 0x7f3478068e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:43.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.909+0000 7f349cc00640 1 -- 172.21.15.27:0/1530738775 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34980aa070 con 0x7f34a00b2960 2024-01-15T18:06:43.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.909+0000 7f34a4e00640 1 --2- 172.21.15.27:0/1530738775 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f34780669b0 0x7f3478068e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:43.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.909+0000 7f34a4e00640 1 --2- 172.21.15.27:0/1530738775 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f34780669b0 0x7f3478068e70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3490009be0 tx=0x7f3490009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:43.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:43.913+0000 7f349cc00640 1 -- 172.21.15.27:0/1530738775 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f34980eaa50 con 0x7f34a00b2960 2024-01-15T18:06:44.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.097+0000 7f34a7800640 1 -- 172.21.15.27:0/1530738775 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f34a00ce420 con 0x7f34780669b0 2024-01-15T18:06:44.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.161+0000 7f349cc00640 1 -- 172.21.15.27:0/1530738775 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f34a00ce420 con 0x7f34780669b0 2024-01-15T18:06:44.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.165+0000 7f34a7800640 1 -- 172.21.15.27:0/1530738775 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f34780669b0 msgr2=0x7f3478068e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.165+0000 7f34a7800640 1 --2- 172.21.15.27:0/1530738775 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f34780669b0 0x7f3478068e70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3490009be0 tx=0x7f3490009190 comp rx=0 tx=0).stop 2024-01-15T18:06:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.165+0000 7f34a7800640 1 -- 172.21.15.27:0/1530738775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34a00b2960 msgr2=0x7f34a00b2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.165+0000 7f34a7800640 1 --2- 172.21.15.27:0/1530738775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34a00b2960 0x7f34a00b2de0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f349801c6f0 tx=0x7f3480005800 comp rx=0 tx=0).stop 2024-01-15T18:06:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.165+0000 7f34a7800640 1 -- 172.21.15.27:0/1530738775 shutdown_connections 2024-01-15T18:06:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.165+0000 7f34a7800640 1 --2- 172.21.15.27:0/1530738775 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f34780669b0 0x7f3478068e70 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.165+0000 7f34a7800640 1 --2- 172.21.15.27:0/1530738775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34a00b2960 0x7f34a00b2de0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.165+0000 7f34a7800640 1 --2- 172.21.15.27:0/1530738775 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34a00b57e0 0x7f34a00b5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.165+0000 7f34a7800640 1 --2- 172.21.15.27:0/1530738775 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3480004d10 0x7f34a01f2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.165+0000 7f34a7800640 1 -- 172.21.15.27:0/1530738775 >> 172.21.15.27:0/1530738775 conn(0x7f34a00d0e70 msgr2=0x7f34a015b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.165+0000 7f34a7800640 1 -- 172.21.15.27:0/1530738775 shutdown_connections 2024-01-15T18:06:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.165+0000 7f34a7800640 1 -- 172.21.15.27:0/1530738775 wait complete. 2024-01-15T18:06:44.173 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:06:44.168+0000 7f59f9c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:06:44.173 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:06:44.168+0000 7f59f9c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:06:44.173 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:06:44.168+0000 7f59f9c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:06:44.173 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:06:44.168+0000 7f59f9c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:06:44.173 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:06:44.168+0000 7f59f9c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:06:44.192 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:06:44.188+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:06:44.192 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:06:44.188+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:06:44.192 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:06:44.188+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:06:44.192 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:06:44.188+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:06:44.192 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:06:44.188+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:06:44.194 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-15T18:06:44.194 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:06:44.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.337+0000 7f73bca00640 1 Processor -- start 2024-01-15T18:06:44.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.337+0000 7f73bca00640 1 -- start start 2024-01-15T18:06:44.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.337+0000 7f73bca00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f73b40715f0 0x7f73b4164a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:44.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.337+0000 7f73bca00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b415a610 con 0x7f73b415ad20 2024-01-15T18:06:44.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.337+0000 7f73bca00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b415e3c0 con 0x7f73b415b100 2024-01-15T18:06:44.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.337+0000 7f73bca00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73b415e6a0 con 0x7f73b40715f0 2024-01-15T18:06:44.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.337+0000 7f73ba000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f73b415ad20 0x7f73b415d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36842/0 (socket says 172.21.15.27:36842) 2024-01-15T18:06:44.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.337+0000 7f73ba000640 1 -- 172.21.15.27:0/1363047303 learned_addr learned my addr 172.21.15.27:0/1363047303 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:06:44.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.337+0000 7f73baa00640 1 --2- 172.21.15.27:0/1363047303 >> v2:172.21.15.93:3300/0 conn(0x7f73b40715f0 0x7f73b4164a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:44.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.337+0000 7f73b8c00640 1 -- 172.21.15.27:0/1363047303 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4175678787 0 0) 0x7f73b415a610 con 0x7f73b415ad20 2024-01-15T18:06:44.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.337+0000 7f73b8c00640 1 -- 172.21.15.27:0/1363047303 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7394003530 con 0x7f73b415ad20 2024-01-15T18:06:44.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.337+0000 7f73baa00640 1 -- 172.21.15.27:0/1363047303 >> v1:172.21.15.93:6789/0 conn(0x7f73b415b100 legacy=0x7f73b4070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:44.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.337+0000 7f73baa00640 1 -- 172.21.15.27:0/1363047303 >> v1:172.21.15.27:6789/0 conn(0x7f73b415ad20 legacy=0x7f73b415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:44.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.337+0000 7f73baa00640 1 -- 172.21.15.27:0/1363047303 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b415b6e0 con 0x7f73b40715f0 2024-01-15T18:06:44.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.341+0000 7f73baa00640 1 --2- 172.21.15.27:0/1363047303 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73b40715f0 0x7f73b4164a60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f73ac009dc0 tx=0x7f73ac01c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe6b0d9f52ee65b6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:44.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.341+0000 7f73b8c00640 1 -- 172.21.15.27:0/1363047303 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac0a5850 con 0x7f73b40715f0 2024-01-15T18:06:44.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.341+0000 7f73b8c00640 1 -- 172.21.15.27:0/1363047303 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73ac0a59b0 con 0x7f73b40715f0 2024-01-15T18:06:44.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.341+0000 7f73b8c00640 1 -- 172.21.15.27:0/1363047303 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac0a5d40 con 0x7f73b40715f0 2024-01-15T18:06:44.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.341+0000 7f73bca00640 1 -- 172.21.15.27:0/1363047303 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73b40715f0 msgr2=0x7f73b4164a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:44.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.341+0000 7f73bca00640 1 --2- 172.21.15.27:0/1363047303 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73b40715f0 0x7f73b4164a60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f73ac009dc0 tx=0x7f73ac01c870 comp rx=0 tx=0).stop 2024-01-15T18:06:44.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.341+0000 7f73bca00640 1 -- 172.21.15.27:0/1363047303 shutdown_connections 2024-01-15T18:06:44.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.341+0000 7f73bca00640 1 --2- 172.21.15.27:0/1363047303 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73b40715f0 0x7f73b4164a60 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.341+0000 7f73bca00640 1 -- 172.21.15.27:0/1363047303 >> 172.21.15.27:0/1363047303 conn(0x7f73b40cb700 msgr2=0x7f73b40cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:44.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.341+0000 7f73bca00640 1 -- 172.21.15.27:0/1363047303 shutdown_connections 2024-01-15T18:06:44.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.341+0000 7f73bca00640 1 -- 172.21.15.27:0/1363047303 wait complete. 2024-01-15T18:06:44.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.345+0000 7f73bca00640 1 Processor -- start 2024-01-15T18:06:44.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.345+0000 7f73bca00640 1 -- start start 2024-01-15T18:06:44.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.345+0000 7f73bca00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73b40715f0 0x7f73b414c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:44.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.345+0000 7f73bca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73b415ad20 0x7f73b4150d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:44.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.345+0000 7f73bca00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f73b415b100 0x7f73b40b3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:44.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.345+0000 7f73bca00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b414c790 con 0x7f73b415ad20 2024-01-15T18:06:44.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.345+0000 7f73bca00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b41516d0 con 0x7f73b40715f0 2024-01-15T18:06:44.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.345+0000 7f73bca00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b41519b0 con 0x7f73b415b100 2024-01-15T18:06:44.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.345+0000 7f73b9600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73b415ad20 0x7f73b4150d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:44.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.345+0000 7f73b9600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73b415ad20 0x7f73b4150d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40908/0 (socket says 172.21.15.27:40908) 2024-01-15T18:06:44.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.345+0000 7f73b9600640 1 -- 172.21.15.27:0/3184967894 learned_addr learned my addr 172.21.15.27:0/3184967894 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:44.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.345+0000 7f73ba000640 1 --2- 172.21.15.27:0/3184967894 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73b40715f0 0x7f73b414c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:44.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.345+0000 7f73baa00640 1 --2- 172.21.15.27:0/3184967894 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f73b415b100 0x7f73b40b3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:44.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.345+0000 7f73b9600640 1 -- 172.21.15.27:0/3184967894 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f73b415b100 msgr2=0x7f73b40b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:44.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.345+0000 7f73b9600640 1 --2- 172.21.15.27:0/3184967894 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f73b415b100 0x7f73b40b3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.345+0000 7f73b9600640 1 -- 172.21.15.27:0/3184967894 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73b40715f0 msgr2=0x7f73b414c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:44.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.345+0000 7f73b9600640 1 --2- 172.21.15.27:0/3184967894 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73b40715f0 0x7f73b414c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.345+0000 7f73b9600640 1 -- 172.21.15.27:0/3184967894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ac009a70 con 0x7f73b415ad20 2024-01-15T18:06:44.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.345+0000 7f73b9600640 1 --2- 172.21.15.27:0/3184967894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73b415ad20 0x7f73b4150d60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f739c00c3c0 tx=0x7f739c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:44.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.345+0000 7f73b2000640 1 -- 172.21.15.27:0/3184967894 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739c0052d0 con 0x7f73b415ad20 2024-01-15T18:06:44.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.345+0000 7f73b2000640 1 -- 172.21.15.27:0/3184967894 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f739c005430 con 0x7f73b415ad20 2024-01-15T18:06:44.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.345+0000 7f73b2000640 1 -- 172.21.15.27:0/3184967894 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739c003840 con 0x7f73b415ad20 2024-01-15T18:06:44.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.345+0000 7f73bca00640 1 -- 172.21.15.27:0/3184967894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73b4151c90 con 0x7f73b415ad20 2024-01-15T18:06:44.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.345+0000 7f73bca00640 1 -- 172.21.15.27:0/3184967894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b4152250 con 0x7f73b415ad20 2024-01-15T18:06:44.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.349+0000 7f73bca00640 1 -- 172.21.15.27:0/3184967894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b4152470 con 0x7f73b415ad20 2024-01-15T18:06:44.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.349+0000 7f73b2000640 1 -- 172.21.15.27:0/3184967894 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 6) v1 ==== 86211+0+0 (secure 0 0 0) 0x7f739c0039a0 con 0x7f73b415ad20 2024-01-15T18:06:44.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.349+0000 7f73b2000640 1 --2- 172.21.15.27:0/3184967894 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f738c066980 0x7f738c068e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:44.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.349+0000 7f73b2000640 1 -- 172.21.15.27:0/3184967894 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f739c0839f0 con 0x7f73b415ad20 2024-01-15T18:06:44.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.349+0000 7f73ba000640 1 --2- 172.21.15.27:0/3184967894 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f738c066980 0x7f738c068e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:44.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.349+0000 7f73ba000640 1 --2- 172.21.15.27:0/3184967894 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f738c066980 0x7f738c068e40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f73a40011f0 tx=0x7f73a4007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:44.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.353+0000 7f73b2000640 1 -- 172.21.15.27:0/3184967894 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f739c050bf0 con 0x7f73b415ad20 2024-01-15T18:06:44.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:44.545+0000 7f73bca00640 1 -- 172.21.15.27:0/3184967894 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f73b404e850 con 0x7f738c066980 2024-01-15T18:06:46.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:46.821+0000 7f73b2000640 1 -- 172.21.15.27:0/3184967894 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 7) v1 ==== 86313+0+0 (secure 0 0 0) 0x7f739c085d50 con 0x7f73b415ad20 2024-01-15T18:06:48.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.281+0000 7f73b2000640 1 -- 172.21.15.27:0/3184967894 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f73b404e850 con 0x7f738c066980 2024-01-15T18:06:48.290 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:06:48.290 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:06:48.290 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:06:48.290 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:06:48.290 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:06:48.290 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:06:48.290 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:06:48.290 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:48.290 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:48.290 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:06:48.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.285+0000 7f73bca00640 1 -- 172.21.15.27:0/3184967894 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f738c066980 msgr2=0x7f738c068e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:48.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.285+0000 7f73bca00640 1 --2- 172.21.15.27:0/3184967894 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f738c066980 0x7f738c068e40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f73a40011f0 tx=0x7f73a4007d40 comp rx=0 tx=0).stop 2024-01-15T18:06:48.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.285+0000 7f73bca00640 1 -- 172.21.15.27:0/3184967894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73b415ad20 msgr2=0x7f73b4150d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:48.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.285+0000 7f73bca00640 1 --2- 172.21.15.27:0/3184967894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73b415ad20 0x7f73b4150d60 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f739c00c3c0 tx=0x7f739c00c880 comp rx=0 tx=0).stop 2024-01-15T18:06:48.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.285+0000 7f73bca00640 1 -- 172.21.15.27:0/3184967894 shutdown_connections 2024-01-15T18:06:48.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.285+0000 7f73bca00640 1 --2- 172.21.15.27:0/3184967894 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f738c066980 0x7f738c068e40 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:48.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.285+0000 7f73bca00640 1 --2- 172.21.15.27:0/3184967894 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f73b415b100 0x7f73b40b3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:48.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.285+0000 7f73bca00640 1 --2- 172.21.15.27:0/3184967894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73b415ad20 0x7f73b4150d60 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:48.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.285+0000 7f73bca00640 1 --2- 172.21.15.27:0/3184967894 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f73b40715f0 0x7f73b414c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:48.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.285+0000 7f73bca00640 1 -- 172.21.15.27:0/3184967894 >> 172.21.15.27:0/3184967894 conn(0x7f73b40cb700 msgr2=0x7f73b40cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:48.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.285+0000 7f73bca00640 1 -- 172.21.15.27:0/3184967894 shutdown_connections 2024-01-15T18:06:48.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.285+0000 7f73bca00640 1 -- 172.21.15.27:0/3184967894 wait complete. 2024-01-15T18:06:48.318 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:06:48.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.453+0000 7f2c15600640 1 Processor -- start 2024-01-15T18:06:48.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.453+0000 7f2c15600640 1 -- start start 2024-01-15T18:06:48.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.457+0000 7f2c15600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f2c100c3e90 0x7f2c100b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:48.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.457+0000 7f2c15600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2c100b23e0 0x7f2c1006b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:48.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.457+0000 7f2c15600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c100c4270 0x7f2c100cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:48.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.457+0000 7f2c15600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c10161d10 con 0x7f2c100b23e0 2024-01-15T18:06:48.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.457+0000 7f2c15600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c10165de0 con 0x7f2c100c3e90 2024-01-15T18:06:48.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.457+0000 7f2c15600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c100cf260 con 0x7f2c100c4270 2024-01-15T18:06:48.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.457+0000 7f2c0e000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2c100b23e0 0x7f2c1006b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:48.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.457+0000 7f2c0e000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2c100b23e0 0x7f2c1006b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40912/0 (socket says 172.21.15.27:40912) 2024-01-15T18:06:48.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.457+0000 7f2c0e000640 1 -- 172.21.15.27:0/991446712 learned_addr learned my addr 172.21.15.27:0/991446712 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:48.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.457+0000 7f2c0f400640 1 --2- 172.21.15.27:0/991446712 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c100c4270 0x7f2c100cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:48.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.457+0000 7f2c0ea00640 1 --2- 172.21.15.27:0/991446712 >> v2:172.21.15.93:3300/0 conn(0x7f2c100c3e90 0x7f2c100b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:48.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.457+0000 7f2c0e000640 1 -- 172.21.15.27:0/991446712 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c100c4270 msgr2=0x7f2c100cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:48.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.457+0000 7f2c0e000640 1 --2- 172.21.15.27:0/991446712 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c100c4270 0x7f2c100cda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:48.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.457+0000 7f2c0e000640 1 -- 172.21.15.27:0/991446712 >> v2:172.21.15.93:3300/0 conn(0x7f2c100c3e90 msgr2=0x7f2c100b1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:48.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.457+0000 7f2c0e000640 1 --2- 172.21.15.27:0/991446712 >> v2:172.21.15.93:3300/0 conn(0x7f2c100c3e90 0x7f2c100b1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:48.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.457+0000 7f2c0e000640 1 -- 172.21.15.27:0/991446712 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c10166310 con 0x7f2c100b23e0 2024-01-15T18:06:48.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.461+0000 7f2c0e000640 1 --2- 172.21.15.27:0/991446712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c100b23e0 0x7f2c1006b600 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f2c100720c0 tx=0x7f2bf801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a9a9c252e9964d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:48.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.461+0000 7f2c0d600640 1 -- 172.21.15.27:0/991446712 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf80ac440 con 0x7f2c100b23e0 2024-01-15T18:06:48.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.461+0000 7f2c0d600640 1 -- 172.21.15.27:0/991446712 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf80ac5a0 con 0x7f2c100b23e0 2024-01-15T18:06:48.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.461+0000 7f2c0d600640 1 -- 172.21.15.27:0/991446712 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf80ac930 con 0x7f2c100b23e0 2024-01-15T18:06:48.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.461+0000 7f2c15600640 1 -- 172.21.15.27:0/991446712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c100b23e0 msgr2=0x7f2c1006b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:48.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.461+0000 7f2c15600640 1 --2- 172.21.15.27:0/991446712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c100b23e0 0x7f2c1006b600 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f2c100720c0 tx=0x7f2bf801c440 comp rx=0 tx=0).stop 2024-01-15T18:06:48.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.461+0000 7f2c15600640 1 -- 172.21.15.27:0/991446712 shutdown_connections 2024-01-15T18:06:48.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.461+0000 7f2c15600640 1 --2- 172.21.15.27:0/991446712 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c100c4270 0x7f2c100cda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:48.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.461+0000 7f2c15600640 1 --2- 172.21.15.27:0/991446712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c100b23e0 0x7f2c1006b600 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:48.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.461+0000 7f2c15600640 1 --2- 172.21.15.27:0/991446712 >> v2:172.21.15.93:3300/0 conn(0x7f2c100c3e90 0x7f2c100b1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:48.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.461+0000 7f2c15600640 1 -- 172.21.15.27:0/991446712 >> 172.21.15.27:0/991446712 conn(0x7f2c100b2a10 msgr2=0x7f2c100b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:48.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.461+0000 7f2c15600640 1 -- 172.21.15.27:0/991446712 shutdown_connections 2024-01-15T18:06:48.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.461+0000 7f2c15600640 1 -- 172.21.15.27:0/991446712 wait complete. 2024-01-15T18:06:48.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.461+0000 7f2c15600640 1 Processor -- start 2024-01-15T18:06:48.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.461+0000 7f2c15600640 1 -- start start 2024-01-15T18:06:48.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.465+0000 7f2c15600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c100b23e0 0x7f2c1014b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:48.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.465+0000 7f2c15600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c100c3e90 0x7f2c10161040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:48.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.465+0000 7f2c15600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c100c4270 0x7f2c10161580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:48.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.465+0000 7f2c15600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1014b8a0 con 0x7f2c100c3e90 2024-01-15T18:06:48.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.465+0000 7f2c15600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1014bb20 con 0x7f2c100c4270 2024-01-15T18:06:48.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.465+0000 7f2c15600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c100ae040 con 0x7f2c100b23e0 2024-01-15T18:06:48.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.465+0000 7f2c0e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c100c3e90 0x7f2c10161040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:48.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.465+0000 7f2c0e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c100c3e90 0x7f2c10161040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40920/0 (socket says 172.21.15.27:40920) 2024-01-15T18:06:48.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.465+0000 7f2c0e000640 1 -- 172.21.15.27:0/3659125516 learned_addr learned my addr 172.21.15.27:0/3659125516 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:48.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.465+0000 7f2c0f400640 1 --2- 172.21.15.27:0/3659125516 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c100c4270 0x7f2c10161580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:48.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.465+0000 7f2c0ea00640 1 --2- 172.21.15.27:0/3659125516 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c100b23e0 0x7f2c1014b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:48.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.465+0000 7f2c0e000640 1 -- 172.21.15.27:0/3659125516 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c100b23e0 msgr2=0x7f2c1014b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:48.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.465+0000 7f2c0e000640 1 --2- 172.21.15.27:0/3659125516 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c100b23e0 0x7f2c1014b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:48.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.465+0000 7f2c0e000640 1 -- 172.21.15.27:0/3659125516 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c100c4270 msgr2=0x7f2c10161580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:48.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.465+0000 7f2c0e000640 1 --2- 172.21.15.27:0/3659125516 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c100c4270 0x7f2c10161580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:48.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.465+0000 7f2c0e000640 1 -- 172.21.15.27:0/3659125516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf80094d0 con 0x7f2c100c3e90 2024-01-15T18:06:48.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.465+0000 7f2c0e000640 1 --2- 172.21.15.27:0/3659125516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c100c3e90 0x7f2c10161040 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2bf80af040 tx=0x7f2bf801ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:48.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.465+0000 7f2c06a00640 1 -- 172.21.15.27:0/3659125516 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf80ac440 con 0x7f2c100c3e90 2024-01-15T18:06:48.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.465+0000 7f2c06a00640 1 -- 172.21.15.27:0/3659125516 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf80b7850 con 0x7f2c100c3e90 2024-01-15T18:06:48.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.465+0000 7f2c15600640 1 -- 172.21.15.27:0/3659125516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c100ae320 con 0x7f2c100c3e90 2024-01-15T18:06:48.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.465+0000 7f2c06a00640 1 -- 172.21.15.27:0/3659125516 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf80b7c00 con 0x7f2c100c3e90 2024-01-15T18:06:48.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.465+0000 7f2c15600640 1 -- 172.21.15.27:0/3659125516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c100ae8e0 con 0x7f2c100c3e90 2024-01-15T18:06:48.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.465+0000 7f2c15600640 1 -- 172.21.15.27:0/3659125516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c100cafd0 con 0x7f2c100c3e90 2024-01-15T18:06:48.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.469+0000 7f2c06a00640 1 -- 172.21.15.27:0/3659125516 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) v1 ==== 86313+0+0 (secure 0 0 0) 0x7f2bf80b3040 con 0x7f2c100c3e90 2024-01-15T18:06:48.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.469+0000 7f2c06a00640 1 --2- 172.21.15.27:0/3659125516 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2be8066ad0 0x7f2be8068f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:48.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.469+0000 7f2c06a00640 1 -- 172.21.15.27:0/3659125516 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2bf811edf0 con 0x7f2c100c3e90 2024-01-15T18:06:48.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.469+0000 7f2c0ea00640 1 --2- 172.21.15.27:0/3659125516 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2be8066ad0 0x7f2be8068f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:48.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.469+0000 7f2c0ea00640 1 --2- 172.21.15.27:0/3659125516 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2be8066ad0 0x7f2be8068f90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2c1015eaa0 tx=0x7f2c00008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:48.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.473+0000 7f2c06a00640 1 -- 172.21.15.27:0/3659125516 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2c100cafd0 con 0x7f2c100c3e90 2024-01-15T18:06:48.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.685+0000 7f2c15600640 1 -- 172.21.15.27:0/3659125516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2c100cafd0 con 0x7f2c100c3e90 2024-01-15T18:06:48.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.685+0000 7f2c06a00640 1 -- 172.21.15.27:0/3659125516 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2309 (secure 0 0 0) 0x7f2bf80bb920 con 0x7f2c100c3e90 2024-01-15T18:06:48.693 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:06:48.693 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:06:48.693 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:06:48.693 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:06:48.693 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:06:48.693 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:06:48.693 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:06:48.693 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:06:48.694 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:06:48.694 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:06:48.694 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:06:48.694 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:06:48.694 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:06:48.694 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:06:48.694 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:06:48.694 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:06:48.694 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:06:48.694 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:06:48.695 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:06:48.695 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 135, 2024-01-15T18:06:48.695 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:06:48.695 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:06:48.695 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:06:48.695 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:06:48.695 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:06:48.695 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:06:48.695 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 34, 2024-01-15T18:06:48.695 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:06:48.695 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:06:48.696 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:06:48.696 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:06:48.696 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:06:48.696 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:06:48.696 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:06:48.696 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:06:48.696 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:06:48.696 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:06:48.696 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:06:48.696 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:06:48.696 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:48.697 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:06:48.697 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:06:48.697 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:06:48.697 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 795, 2024-01-15T18:06:48.697 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 3222082669, 2024-01-15T18:06:48.697 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 6689837056, 2024-01-15T18:06:48.697 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 766404276224, 2024-01-15T18:06:48.697 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:06:48.697 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 85, 2024-01-15T18:06:48.697 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 109043616, 2024-01-15T18:06:48.697 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 0, 2024-01-15T18:06:48.698 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 29 2024-01-15T18:06:48.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:06:48.698 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:06:48.698 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 18, 2024-01-15T18:06:48.698 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 2, 2024-01-15T18:06:48.698 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:06:48.698 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:06:48.698 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:06:48.698 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:06:48.698 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:06:48.698 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 2, 2024-01-15T18:06:48.699 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:06:48.699 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "a", 2024-01-15T18:06:48.699 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:06:48.699 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 5232 2024-01-15T18:06:48.699 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:48.699 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:06:48.699 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:06:48.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:06:48.699 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:06:48.699 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:06:48.699 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:06:48.700 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:06:48.700 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:06:48.700 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:06:48.700 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:06:48.700 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:06:48.700 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:06:48.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:06:48.700 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:06:48.700 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 6, 2024-01-15T18:06:48.700 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:05:54.151625+0000", 2024-01-15T18:06:48.701 INFO:teuthology.orchestra.run.smithi027.stdout: "services": { 2024-01-15T18:06:48.701 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-01-15T18:06:48.701 INFO:teuthology.orchestra.run.smithi027.stdout: "daemons": { 2024-01-15T18:06:48.701 INFO:teuthology.orchestra.run.smithi027.stdout: "summary": "", 2024-01-15T18:06:48.701 INFO:teuthology.orchestra.run.smithi027.stdout: "a": { 2024-01-15T18:06:48.701 INFO:teuthology.orchestra.run.smithi027.stdout: "start_epoch": 0, 2024-01-15T18:06:48.701 INFO:teuthology.orchestra.run.smithi027.stdout: "start_stamp": "0.000000", 2024-01-15T18:06:48.701 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 0, 2024-01-15T18:06:48.701 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "(unrecognized address family 0)/0", 2024-01-15T18:06:48.701 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": {}, 2024-01-15T18:06:48.701 INFO:teuthology.orchestra.run.smithi027.stdout: "task_status": {} 2024-01-15T18:06:48.702 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:48.702 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:48.702 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:48.702 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:48.702 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:06:48.702 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": {} 2024-01-15T18:06:48.702 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:06:48.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.689+0000 7f2c15600640 1 -- 172.21.15.27:0/3659125516 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2be8066ad0 msgr2=0x7f2be8068f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:48.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.689+0000 7f2c15600640 1 --2- 172.21.15.27:0/3659125516 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2be8066ad0 0x7f2be8068f90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2c1015eaa0 tx=0x7f2c00008040 comp rx=0 tx=0).stop 2024-01-15T18:06:48.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.689+0000 7f2c15600640 1 -- 172.21.15.27:0/3659125516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c100c3e90 msgr2=0x7f2c10161040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:48.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.689+0000 7f2c15600640 1 --2- 172.21.15.27:0/3659125516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c100c3e90 0x7f2c10161040 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2bf80af040 tx=0x7f2bf801ac80 comp rx=0 tx=0).stop 2024-01-15T18:06:48.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.689+0000 7f2c15600640 1 -- 172.21.15.27:0/3659125516 shutdown_connections 2024-01-15T18:06:48.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.689+0000 7f2c15600640 1 --2- 172.21.15.27:0/3659125516 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2be8066ad0 0x7f2be8068f90 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:48.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.689+0000 7f2c15600640 1 --2- 172.21.15.27:0/3659125516 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c100c4270 0x7f2c10161580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:48.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.689+0000 7f2c15600640 1 --2- 172.21.15.27:0/3659125516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c100c3e90 0x7f2c10161040 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:48.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.689+0000 7f2c15600640 1 --2- 172.21.15.27:0/3659125516 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c100b23e0 0x7f2c1014b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:48.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.689+0000 7f2c15600640 1 -- 172.21.15.27:0/3659125516 >> 172.21.15.27:0/3659125516 conn(0x7f2c100b2a10 msgr2=0x7f2c1006cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:48.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.689+0000 7f2c15600640 1 -- 172.21.15.27:0/3659125516 shutdown_connections 2024-01-15T18:06:48.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.689+0000 7f2c15600640 1 -- 172.21.15.27:0/3659125516 wait complete. 2024-01-15T18:06:48.715 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:06:48.715 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:06:48.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.853+0000 7f0ed9200640 1 Processor -- start 2024-01-15T18:06:48.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.853+0000 7f0ed9200640 1 -- start start 2024-01-15T18:06:48.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.853+0000 7f0ed9200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ed40cf570 0x7f0ed40b5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:48.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.853+0000 7f0ed9200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed40b5980 con 0x7f0ed415d5c0 2024-01-15T18:06:48.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.853+0000 7f0ed9200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed40ce7b0 con 0x7f0ed4072360 2024-01-15T18:06:48.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.853+0000 7f0ed9200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed40b5760 con 0x7f0ed40cf570 2024-01-15T18:06:48.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.853+0000 7f0ed1e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0ed415d5c0 0x7f0ed4071c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36856/0 (socket says 172.21.15.27:36856) 2024-01-15T18:06:48.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.853+0000 7f0ed1e00640 1 -- 172.21.15.27:0/3536402748 learned_addr learned my addr 172.21.15.27:0/3536402748 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:06:48.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.853+0000 7f0ed3200640 1 --2- 172.21.15.27:0/3536402748 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ed40cf570 0x7f0ed40b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:48.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.853+0000 7f0ed0a00640 1 -- 172.21.15.27:0/3536402748 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667434511 0 0) 0x7f0ed40ce7b0 con 0x7f0ed4072360 2024-01-15T18:06:48.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.853+0000 7f0ed0a00640 1 -- 172.21.15.27:0/3536402748 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0eb4003530 con 0x7f0ed4072360 2024-01-15T18:06:48.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.853+0000 7f0ed0a00640 1 -- 172.21.15.27:0/3536402748 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 299804798 0 0) 0x7f0ed40b5980 con 0x7f0ed415d5c0 2024-01-15T18:06:48.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.853+0000 7f0ed0a00640 1 -- 172.21.15.27:0/3536402748 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ed40ce7b0 con 0x7f0ed415d5c0 2024-01-15T18:06:48.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.853+0000 7f0ed3200640 1 -- 172.21.15.27:0/3536402748 >> v1:172.21.15.93:6789/0 conn(0x7f0ed4072360 legacy=0x7f0ed40b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:48.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.853+0000 7f0ed3200640 1 -- 172.21.15.27:0/3536402748 >> v1:172.21.15.27:6789/0 conn(0x7f0ed415d5c0 legacy=0x7f0ed4071c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:48.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.853+0000 7f0ed3200640 1 -- 172.21.15.27:0/3536402748 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed40cea90 con 0x7f0ed40cf570 2024-01-15T18:06:48.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.857+0000 7f0ed3200640 1 --2- 172.21.15.27:0/3536402748 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ed40cf570 0x7f0ed40b5220 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc009dc0 tx=0x7f0ecc01c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbc90e752a8822df server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:48.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.857+0000 7f0ed0a00640 1 -- 172.21.15.27:0/3536402748 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc0a5890 con 0x7f0ed40cf570 2024-01-15T18:06:48.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.857+0000 7f0ed0a00640 1 -- 172.21.15.27:0/3536402748 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ecc0a59f0 con 0x7f0ed40cf570 2024-01-15T18:06:48.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.857+0000 7f0ed0a00640 1 -- 172.21.15.27:0/3536402748 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc0a5d80 con 0x7f0ed40cf570 2024-01-15T18:06:48.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.857+0000 7f0ed9200640 1 -- 172.21.15.27:0/3536402748 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ed40cf570 msgr2=0x7f0ed40b5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:48.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.857+0000 7f0ed9200640 1 --2- 172.21.15.27:0/3536402748 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ed40cf570 0x7f0ed40b5220 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc009dc0 tx=0x7f0ecc01c8e0 comp rx=0 tx=0).stop 2024-01-15T18:06:48.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.861+0000 7f0ed9200640 1 -- 172.21.15.27:0/3536402748 shutdown_connections 2024-01-15T18:06:48.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.861+0000 7f0ed9200640 1 --2- 172.21.15.27:0/3536402748 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ed40cf570 0x7f0ed40b5220 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:48.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.861+0000 7f0ed9200640 1 -- 172.21.15.27:0/3536402748 >> 172.21.15.27:0/3536402748 conn(0x7f0ed4099e10 msgr2=0x7f0ed40b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:48.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.861+0000 7f0ed9200640 1 -- 172.21.15.27:0/3536402748 shutdown_connections 2024-01-15T18:06:48.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.861+0000 7f0ed9200640 1 -- 172.21.15.27:0/3536402748 wait complete. 2024-01-15T18:06:48.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.861+0000 7f0ed9200640 1 Processor -- start 2024-01-15T18:06:48.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.861+0000 7f0ed9200640 1 -- start start 2024-01-15T18:06:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.861+0000 7f0ed9200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ed4072360 0x7f0ed40b0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.861+0000 7f0ed9200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed40cf570 0x7f0ed415b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.861+0000 7f0ed9200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ed415d5c0 0x7f0ed406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.861+0000 7f0ed9200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed40b0c80 con 0x7f0ed40cf570 2024-01-15T18:06:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.861+0000 7f0ed9200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed40b0f60 con 0x7f0ed4072360 2024-01-15T18:06:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.861+0000 7f0ed9200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed40b1240 con 0x7f0ed415d5c0 2024-01-15T18:06:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.861+0000 7f0ed1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed40cf570 0x7f0ed415b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.861+0000 7f0ed1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed40cf570 0x7f0ed415b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40936/0 (socket says 172.21.15.27:40936) 2024-01-15T18:06:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.861+0000 7f0ed1400640 1 -- 172.21.15.27:0/3322425499 learned_addr learned my addr 172.21.15.27:0/3322425499 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:48.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.861+0000 7f0ed3200640 1 --2- 172.21.15.27:0/3322425499 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ed415d5c0 0x7f0ed406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.861+0000 7f0ed1e00640 1 --2- 172.21.15.27:0/3322425499 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ed4072360 0x7f0ed40b0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.861+0000 7f0ed1400640 1 -- 172.21.15.27:0/3322425499 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ed415d5c0 msgr2=0x7f0ed406eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.861+0000 7f0ed1400640 1 --2- 172.21.15.27:0/3322425499 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ed415d5c0 0x7f0ed406eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.861+0000 7f0ed1400640 1 -- 172.21.15.27:0/3322425499 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ed4072360 msgr2=0x7f0ed40b0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.861+0000 7f0ed1400640 1 --2- 172.21.15.27:0/3322425499 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ed4072360 0x7f0ed40b0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.861+0000 7f0ed1400640 1 -- 172.21.15.27:0/3322425499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ecc009a70 con 0x7f0ed40cf570 2024-01-15T18:06:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.861+0000 7f0ed1400640 1 --2- 172.21.15.27:0/3322425499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed40cf570 0x7f0ed415b850 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc0010f0 tx=0x7f0ebc00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.865+0000 7f0eca000640 1 -- 172.21.15.27:0/3322425499 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc0052d0 con 0x7f0ed40cf570 2024-01-15T18:06:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.865+0000 7f0eca000640 1 -- 172.21.15.27:0/3322425499 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ebc005430 con 0x7f0ed40cf570 2024-01-15T18:06:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.865+0000 7f0eca000640 1 -- 172.21.15.27:0/3322425499 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc012830 con 0x7f0ed40cf570 2024-01-15T18:06:48.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.865+0000 7f0ed9200640 1 -- 172.21.15.27:0/3322425499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed40b1520 con 0x7f0ed40cf570 2024-01-15T18:06:48.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.865+0000 7f0ed9200640 1 -- 172.21.15.27:0/3322425499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed415bd90 con 0x7f0ed40cf570 2024-01-15T18:06:48.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.865+0000 7f0ed9200640 1 -- 172.21.15.27:0/3322425499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed409e950 con 0x7f0ed40cf570 2024-01-15T18:06:48.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.865+0000 7f0eca000640 1 -- 172.21.15.27:0/3322425499 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) v1 ==== 86313+0+0 (secure 0 0 0) 0x7f0ebc003600 con 0x7f0ed40cf570 2024-01-15T18:06:48.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.865+0000 7f0eca000640 1 --2- 172.21.15.27:0/3322425499 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0eac066ad0 0x7f0eac068f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:48.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.865+0000 7f0eca000640 1 -- 172.21.15.27:0/3322425499 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ebc083b40 con 0x7f0ed40cf570 2024-01-15T18:06:48.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.865+0000 7f0ed1e00640 1 --2- 172.21.15.27:0/3322425499 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0eac066ad0 0x7f0eac068f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:48.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.869+0000 7f0ed1e00640 1 --2- 172.21.15.27:0/3322425499 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0eac066ad0 0x7f0eac068f90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0ec40011f0 tx=0x7f0ec4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:48.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:48.869+0000 7f0eca000640 1 -- 172.21.15.27:0/3322425499 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0ebc050bc0 con 0x7f0ed40cf570 2024-01-15T18:06:49.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:49.061+0000 7f0ed9200640 1 -- 172.21.15.27:0/3322425499 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ed40b10d0 con 0x7f0eac066ad0 2024-01-15T18:06:49.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:49.065+0000 7f0eca000640 1 -- 172.21.15.27:0/3322425499 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f0ed40b10d0 con 0x7f0eac066ad0 2024-01-15T18:06:49.074 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:06:49.075 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:06:49.075 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:06:49.075 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:06:49.075 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:06:49.075 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:06:49.075 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:06:49.075 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:06:49.075 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:06:49.075 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 0.0, 2024-01-15T18:06:49.075 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "0 B/3 GB", 2024-01-15T18:06:49.075 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "0/3", 2024-01-15T18:06:49.075 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:06:49.076 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:06:49.076 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:49.076 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:49.076 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:06:49.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:49.069+0000 7f0ed9200640 1 -- 172.21.15.27:0/3322425499 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0eac066ad0 msgr2=0x7f0eac068f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:49.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:49.069+0000 7f0ed9200640 1 --2- 172.21.15.27:0/3322425499 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0eac066ad0 0x7f0eac068f90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0ec40011f0 tx=0x7f0ec4009040 comp rx=0 tx=0).stop 2024-01-15T18:06:49.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:49.069+0000 7f0ed9200640 1 -- 172.21.15.27:0/3322425499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed40cf570 msgr2=0x7f0ed415b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:49.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:49.069+0000 7f0ed9200640 1 --2- 172.21.15.27:0/3322425499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed40cf570 0x7f0ed415b850 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc0010f0 tx=0x7f0ebc00c960 comp rx=0 tx=0).stop 2024-01-15T18:06:49.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:49.069+0000 7f0ed9200640 1 -- 172.21.15.27:0/3322425499 shutdown_connections 2024-01-15T18:06:49.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:49.069+0000 7f0ed9200640 1 --2- 172.21.15.27:0/3322425499 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0eac066ad0 0x7f0eac068f90 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:49.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:49.069+0000 7f0ed9200640 1 --2- 172.21.15.27:0/3322425499 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ed415d5c0 0x7f0ed406eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:49.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:49.069+0000 7f0ed9200640 1 --2- 172.21.15.27:0/3322425499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ed40cf570 0x7f0ed415b850 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:49.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:49.069+0000 7f0ed9200640 1 --2- 172.21.15.27:0/3322425499 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ed4072360 0x7f0ed40b0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:49.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:49.069+0000 7f0ed9200640 1 -- 172.21.15.27:0/3322425499 >> 172.21.15.27:0/3322425499 conn(0x7f0ed4099e10 msgr2=0x7f0ed414c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:49.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:49.069+0000 7f0ed9200640 1 -- 172.21.15.27:0/3322425499 shutdown_connections 2024-01-15T18:06:49.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:49.069+0000 7f0ed9200640 1 -- 172.21.15.27:0/3322425499 wait complete. 2024-01-15T18:06:49.097 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:06:50.098 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:06:50.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.245+0000 7f263ce00640 1 Processor -- start 2024-01-15T18:06:50.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.245+0000 7f263ce00640 1 -- start start 2024-01-15T18:06:50.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.249+0000 7f263ce00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f26380cf460 0x7f26380cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:50.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.249+0000 7f263ce00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26380b2590 0x7f26380cc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:50.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.249+0000 7f263ce00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26380cfd80 con 0x7f263815b4b0 2024-01-15T18:06:50.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.249+0000 7f263ce00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26380d19a0 con 0x7f26380cf460 2024-01-15T18:06:50.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.249+0000 7f263ce00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26380d1c80 con 0x7f26380b2590 2024-01-15T18:06:50.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.249+0000 7f2636800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26380b2590 0x7f26380cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:50.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.249+0000 7f2635e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f26380cf460 0x7f26380cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:50.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.249+0000 7f2635400640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7f263815b4b0 0x7f26380cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:53824/0 (socket says 172.21.15.27:53824) 2024-01-15T18:06:50.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.249+0000 7f2635400640 1 -- 172.21.15.27:0/4037246605 learned_addr learned my addr 172.21.15.27:0/4037246605 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:06:50.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.249+0000 7f2634a00640 1 -- 172.21.15.27:0/4037246605 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1613273634 0 0) 0x7f26380cfd80 con 0x7f263815b4b0 2024-01-15T18:06:50.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.249+0000 7f2636800640 1 -- 172.21.15.27:0/4037246605 >> v2:172.21.15.93:3300/0 conn(0x7f26380cf460 msgr2=0x7f26380cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:50.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.249+0000 7f2636800640 1 --2- 172.21.15.27:0/4037246605 >> v2:172.21.15.93:3300/0 conn(0x7f26380cf460 0x7f26380cf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.249+0000 7f2636800640 1 -- 172.21.15.27:0/4037246605 >> v1:172.21.15.93:6789/0 conn(0x7f263815b4b0 legacy=0x7f26380cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:50.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.249+0000 7f2636800640 1 -- 172.21.15.27:0/4037246605 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26380b07c0 con 0x7f26380b2590 2024-01-15T18:06:50.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.253+0000 7f2636800640 1 --2- 172.21.15.27:0/4037246605 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26380b2590 0x7f26380cc130 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2630009d90 tx=0x7f263001c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70f93be8b1b506d5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:50.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.253+0000 7f2634a00640 1 -- 172.21.15.27:0/4037246605 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26300a5a10 con 0x7f26380b2590 2024-01-15T18:06:50.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.253+0000 7f2634a00640 1 -- 172.21.15.27:0/4037246605 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26300a5b70 con 0x7f26380b2590 2024-01-15T18:06:50.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.253+0000 7f2634a00640 1 -- 172.21.15.27:0/4037246605 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26300ad440 con 0x7f26380b2590 2024-01-15T18:06:50.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.253+0000 7f263ce00640 1 -- 172.21.15.27:0/4037246605 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26380b2590 msgr2=0x7f26380cc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:50.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.253+0000 7f263ce00640 1 --2- 172.21.15.27:0/4037246605 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26380b2590 0x7f26380cc130 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2630009d90 tx=0x7f263001c950 comp rx=0 tx=0).stop 2024-01-15T18:06:50.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.253+0000 7f263ce00640 1 -- 172.21.15.27:0/4037246605 shutdown_connections 2024-01-15T18:06:50.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.253+0000 7f263ce00640 1 --2- 172.21.15.27:0/4037246605 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26380b2590 0x7f26380cc130 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.253+0000 7f263ce00640 1 --2- 172.21.15.27:0/4037246605 >> v2:172.21.15.93:3300/0 conn(0x7f26380cf460 0x7f26380cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.253+0000 7f263ce00640 1 -- 172.21.15.27:0/4037246605 >> 172.21.15.27:0/4037246605 conn(0x7f26380d0e80 msgr2=0x7f26380d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:50.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.253+0000 7f263ce00640 1 -- 172.21.15.27:0/4037246605 shutdown_connections 2024-01-15T18:06:50.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.253+0000 7f263ce00640 1 -- 172.21.15.27:0/4037246605 wait complete. 2024-01-15T18:06:50.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.253+0000 7f263ce00640 1 Processor -- start 2024-01-15T18:06:50.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.253+0000 7f263ce00640 1 -- start start 2024-01-15T18:06:50.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.257+0000 7f263ce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26380b2590 0x7f26380c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:50.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.257+0000 7f263ce00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f26380cf460 0x7f26381537f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:50.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.257+0000 7f263ce00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f263815b4b0 0x7f2638158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:50.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.257+0000 7f263ce00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26380c8530 con 0x7f26380b2590 2024-01-15T18:06:50.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.257+0000 7f263ce00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2638158a40 con 0x7f26380cf460 2024-01-15T18:06:50.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.257+0000 7f263ce00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f263815a870 con 0x7f263815b4b0 2024-01-15T18:06:50.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.257+0000 7f2635e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26380b2590 0x7f26380c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:50.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.257+0000 7f2635400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f26380cf460 0x7f26381537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:50.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.257+0000 7f2635e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26380b2590 0x7f26380c7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40950/0 (socket says 172.21.15.27:40950) 2024-01-15T18:06:50.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.257+0000 7f2636800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f263815b4b0 0x7f2638158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:50.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.257+0000 7f2635e00640 1 -- 172.21.15.27:0/3569426837 learned_addr learned my addr 172.21.15.27:0/3569426837 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:50.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.257+0000 7f2635400640 1 -- 172.21.15.27:0/3569426837 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f263815b4b0 msgr2=0x7f2638158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:50.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.257+0000 7f2635400640 1 --2- 172.21.15.27:0/3569426837 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f263815b4b0 0x7f2638158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.257+0000 7f2635400640 1 -- 172.21.15.27:0/3569426837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26380b2590 msgr2=0x7f26380c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:50.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.257+0000 7f2635400640 1 --2- 172.21.15.27:0/3569426837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26380b2590 0x7f26380c7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.257+0000 7f2635400640 1 -- 172.21.15.27:0/3569426837 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2630009a70 con 0x7f26380cf460 2024-01-15T18:06:50.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.257+0000 7f2636800640 1 --2- 172.21.15.27:0/3569426837 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f263815b4b0 0x7f2638158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:06:50.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.257+0000 7f2635e00640 1 --2- 172.21.15.27:0/3569426837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26380b2590 0x7f26380c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:06:50.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.257+0000 7f2635400640 1 --2- 172.21.15.27:0/3569426837 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f26380cf460 0x7f26381537f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f262000c440 tx=0x7f262000c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:50.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.257+0000 7f262e000640 1 -- 172.21.15.27:0/3569426837 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26200074e0 con 0x7f26380cf460 2024-01-15T18:06:50.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.257+0000 7f262e000640 1 -- 172.21.15.27:0/3569426837 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2620007660 con 0x7f26380cf460 2024-01-15T18:06:50.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.257+0000 7f262e000640 1 -- 172.21.15.27:0/3569426837 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2620004a30 con 0x7f26380cf460 2024-01-15T18:06:50.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.257+0000 7f263ce00640 1 -- 172.21.15.27:0/3569426837 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26380c99d0 con 0x7f26380cf460 2024-01-15T18:06:50.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.257+0000 7f263ce00640 1 -- 172.21.15.27:0/3569426837 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26380c9c70 con 0x7f26380cf460 2024-01-15T18:06:50.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.257+0000 7f263ce00640 1 -- 172.21.15.27:0/3569426837 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f263815ad50 con 0x7f26380cf460 2024-01-15T18:06:50.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.261+0000 7f262e000640 1 -- 172.21.15.27:0/3569426837 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 7) v1 ==== 86313+0+0 (secure 0 0 0) 0x7f2620004b90 con 0x7f26380cf460 2024-01-15T18:06:50.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.261+0000 7f262e000640 1 --2- 172.21.15.27:0/3569426837 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2610066ad0 0x7f2610068f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:50.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.261+0000 7f262e000640 1 -- 172.21.15.27:0/3569426837 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f26200843a0 con 0x7f26380cf460 2024-01-15T18:06:50.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.261+0000 7f2635e00640 1 --2- 172.21.15.27:0/3569426837 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2610066ad0 0x7f2610068f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:50.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.261+0000 7f2635e00640 1 --2- 172.21.15.27:0/3569426837 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2610066ad0 0x7f2610068f90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2628011310 tx=0x7f2628009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:50.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.265+0000 7f262e000640 1 -- 172.21.15.27:0/3569426837 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2620051430 con 0x7f26380cf460 2024-01-15T18:06:50.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.485+0000 7f263ce00640 1 -- 172.21.15.27:0/3569426837 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f26380c9e60 con 0x7f2610066ad0 2024-01-15T18:06:50.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.493+0000 7f262e000640 1 -- 172.21.15.27:0/3569426837 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f26380c9e60 con 0x7f2610066ad0 2024-01-15T18:06:50.500 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:06:50.500 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:06:50.501 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:06:50.501 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:06:50.501 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:06:50.501 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:06:50.501 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:06:50.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:06:50.501 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:06:50.501 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 0.0, 2024-01-15T18:06:50.501 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "0 B/3 GB", 2024-01-15T18:06:50.501 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "0/3", 2024-01-15T18:06:50.502 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:06:50.502 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:06:50.502 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:50.502 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:50.502 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:06:50.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.493+0000 7f263ce00640 1 -- 172.21.15.27:0/3569426837 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2610066ad0 msgr2=0x7f2610068f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:50.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.493+0000 7f263ce00640 1 --2- 172.21.15.27:0/3569426837 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2610066ad0 0x7f2610068f90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2628011310 tx=0x7f2628009190 comp rx=0 tx=0).stop 2024-01-15T18:06:50.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.493+0000 7f263ce00640 1 -- 172.21.15.27:0/3569426837 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f26380cf460 msgr2=0x7f26381537f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:50.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.493+0000 7f263ce00640 1 --2- 172.21.15.27:0/3569426837 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f26380cf460 0x7f26381537f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f262000c440 tx=0x7f262000c900 comp rx=0 tx=0).stop 2024-01-15T18:06:50.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.493+0000 7f263ce00640 1 -- 172.21.15.27:0/3569426837 shutdown_connections 2024-01-15T18:06:50.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.493+0000 7f263ce00640 1 --2- 172.21.15.27:0/3569426837 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2610066ad0 0x7f2610068f90 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.493+0000 7f263ce00640 1 --2- 172.21.15.27:0/3569426837 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f263815b4b0 0x7f2638158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.493+0000 7f263ce00640 1 --2- 172.21.15.27:0/3569426837 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f26380cf460 0x7f26381537f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.493+0000 7f263ce00640 1 --2- 172.21.15.27:0/3569426837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26380b2590 0x7f26380c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.493+0000 7f263ce00640 1 -- 172.21.15.27:0/3569426837 >> 172.21.15.27:0/3569426837 conn(0x7f26380d0e80 msgr2=0x7f26380ae540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:50.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.497+0000 7f263ce00640 1 -- 172.21.15.27:0/3569426837 shutdown_connections 2024-01-15T18:06:50.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:50.497+0000 7f263ce00640 1 -- 172.21.15.27:0/3569426837 wait complete. 2024-01-15T18:06:50.528 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:06:51.529 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:06:51.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.725+0000 7f7185000640 1 Processor -- start 2024-01-15T18:06:51.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.725+0000 7f7185000640 1 -- start start 2024-01-15T18:06:51.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.725+0000 7f7185000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71800cde10 0x7f71800cbf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:51.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.725+0000 7f7185000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71800cfc30 0x7f71800cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:51.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.725+0000 7f7185000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71800cb4d0 con 0x7f718015d680 2024-01-15T18:06:51.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.725+0000 7f7185000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71800d0120 con 0x7f71800cde10 2024-01-15T18:06:51.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.725+0000 7f7185000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71800d0400 con 0x7f71800cfc30 2024-01-15T18:06:51.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.725+0000 7f717d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71800cde10 0x7f71800cbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:51.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.729+0000 7f717d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71800cde10 0x7f71800cbf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40960/0 (socket says 172.21.15.27:40960) 2024-01-15T18:06:51.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.729+0000 7f717d400640 1 -- 172.21.15.27:0/1296140936 learned_addr learned my addr 172.21.15.27:0/1296140936 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:51.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.729+0000 7f717e800640 1 --2- 172.21.15.27:0/1296140936 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71800cfc30 0x7f71800cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:51.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.729+0000 7f717d400640 1 -- 172.21.15.27:0/1296140936 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71800cfc30 msgr2=0x7f71800cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:51.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.729+0000 7f717d400640 1 --2- 172.21.15.27:0/1296140936 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71800cfc30 0x7f71800cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:51.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.729+0000 7f717d400640 1 -- 172.21.15.27:0/1296140936 >> v1:172.21.15.93:6789/0 conn(0x7f718015d680 legacy=0x7f71800cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:51.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.729+0000 7f717d400640 1 -- 172.21.15.27:0/1296140936 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71800cb770 con 0x7f71800cde10 2024-01-15T18:06:51.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.729+0000 7f717d400640 1 --2- 172.21.15.27:0/1296140936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71800cde10 0x7f71800cbf70 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f7168009870 tx=0x7f716801c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b92994bdcfa82b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:51.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.733+0000 7f717ca00640 1 -- 172.21.15.27:0/1296140936 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71800ce850 con 0x7f71800cde10 2024-01-15T18:06:51.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.733+0000 7f717ca00640 1 -- 172.21.15.27:0/1296140936 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71680a95f0 con 0x7f71800cde10 2024-01-15T18:06:51.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.733+0000 7f717ca00640 1 -- 172.21.15.27:0/1296140936 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71680a9980 con 0x7f71800cde10 2024-01-15T18:06:51.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.733+0000 7f7185000640 1 -- 172.21.15.27:0/1296140936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71800cde10 msgr2=0x7f71800cbf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:51.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.733+0000 7f7185000640 1 --2- 172.21.15.27:0/1296140936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71800cde10 0x7f71800cbf70 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f7168009870 tx=0x7f716801c510 comp rx=0 tx=0).stop 2024-01-15T18:06:51.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.733+0000 7f7185000640 1 -- 172.21.15.27:0/1296140936 shutdown_connections 2024-01-15T18:06:51.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.733+0000 7f7185000640 1 --2- 172.21.15.27:0/1296140936 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71800cfc30 0x7f71800cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:51.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.733+0000 7f7185000640 1 --2- 172.21.15.27:0/1296140936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71800cde10 0x7f71800cbf70 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:51.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.733+0000 7f7185000640 1 -- 172.21.15.27:0/1296140936 >> 172.21.15.27:0/1296140936 conn(0x7f71800d0e80 msgr2=0x7f71800d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:51.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.733+0000 7f7185000640 1 -- 172.21.15.27:0/1296140936 shutdown_connections 2024-01-15T18:06:51.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.733+0000 7f7185000640 1 -- 172.21.15.27:0/1296140936 wait complete. 2024-01-15T18:06:51.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.733+0000 7f7185000640 1 Processor -- start 2024-01-15T18:06:51.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.733+0000 7f7185000640 1 -- start start 2024-01-15T18:06:51.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.737+0000 7f7185000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71800cde10 0x7f7180150370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:51.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.737+0000 7f7185000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f71800cfc30 0x7f71800c5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:51.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.737+0000 7f7185000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f718015d680 0x7f71800aeac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:51.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.737+0000 7f7185000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71801508b0 con 0x7f71800cde10 2024-01-15T18:06:51.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.737+0000 7f7185000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7180150b40 con 0x7f71800cfc30 2024-01-15T18:06:51.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.737+0000 7f7185000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7180150e20 con 0x7f718015d680 2024-01-15T18:06:51.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.737+0000 7f717de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71800cde10 0x7f7180150370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:51.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.737+0000 7f717de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71800cde10 0x7f7180150370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40964/0 (socket says 172.21.15.27:40964) 2024-01-15T18:06:51.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.737+0000 7f717de00640 1 -- 172.21.15.27:0/2100190555 learned_addr learned my addr 172.21.15.27:0/2100190555 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:51.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.737+0000 7f717de00640 1 -- 172.21.15.27:0/2100190555 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f718015d680 msgr2=0x7f71800aeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:51.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.737+0000 7f717de00640 1 --2- 172.21.15.27:0/2100190555 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f718015d680 0x7f71800aeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:51.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.737+0000 7f717de00640 1 -- 172.21.15.27:0/2100190555 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f71800cfc30 msgr2=0x7f71800c5880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-15T18:06:51.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.737+0000 7f717de00640 1 --2- 172.21.15.27:0/2100190555 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f71800cfc30 0x7f71800c5880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:51.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.737+0000 7f717de00640 1 -- 172.21.15.27:0/2100190555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71680094d0 con 0x7f71800cde10 2024-01-15T18:06:51.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.737+0000 7f717de00640 1 --2- 172.21.15.27:0/2100190555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71800cde10 0x7f7180150370 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f717000c3c0 tx=0x7f717000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:51.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.737+0000 7f7176000640 1 -- 172.21.15.27:0/2100190555 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71700052d0 con 0x7f71800cde10 2024-01-15T18:06:51.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.737+0000 7f7176000640 1 -- 172.21.15.27:0/2100190555 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7170005430 con 0x7f71800cde10 2024-01-15T18:06:51.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.737+0000 7f7176000640 1 -- 172.21.15.27:0/2100190555 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7170003840 con 0x7f71800cde10 2024-01-15T18:06:51.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.737+0000 7f7185000640 1 -- 172.21.15.27:0/2100190555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7180151100 con 0x7f71800cde10 2024-01-15T18:06:51.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.737+0000 7f7185000640 1 -- 172.21.15.27:0/2100190555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71801516c0 con 0x7f71800cde10 2024-01-15T18:06:51.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.741+0000 7f7185000640 1 -- 172.21.15.27:0/2100190555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71801518e0 con 0x7f71800cde10 2024-01-15T18:06:51.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.741+0000 7f7176000640 1 -- 172.21.15.27:0/2100190555 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) v1 ==== 86313+0+0 (secure 0 0 0) 0x7f71700039a0 con 0x7f71800cde10 2024-01-15T18:06:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.741+0000 7f7176000640 1 --2- 172.21.15.27:0/2100190555 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7158066af0 0x7f7158068fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.741+0000 7f7176000640 1 -- 172.21.15.27:0/2100190555 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7170083c60 con 0x7f71800cde10 2024-01-15T18:06:51.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.741+0000 7f717d400640 1 --2- 172.21.15.27:0/2100190555 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7158066af0 0x7f7158068fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:51.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.745+0000 7f717d400640 1 --2- 172.21.15.27:0/2100190555 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7158066af0 0x7f7158068fb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f71680ae000 tx=0x7f7168008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:51.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:51.749+0000 7f7176000640 1 -- 172.21.15.27:0/2100190555 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7170050cc0 con 0x7f71800cde10 2024-01-15T18:06:52.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:52.085+0000 7f7185000640 1 -- 172.21.15.27:0/2100190555 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7180150f90 con 0x7f7158066af0 2024-01-15T18:06:52.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:52.105+0000 7f7176000640 1 -- 172.21.15.27:0/2100190555 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f7180150f90 con 0x7f7158066af0 2024-01-15T18:06:52.112 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:06:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:06:52.112 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:06:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:06:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:06:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:06:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:06:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:06:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:06:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 0.0, 2024-01-15T18:06:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "0 B/3 GB", 2024-01-15T18:06:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "0/3", 2024-01-15T18:06:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:06:52.113 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:06:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:52.114 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:52.114 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:06:52.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:52.109+0000 7f7185000640 1 -- 172.21.15.27:0/2100190555 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7158066af0 msgr2=0x7f7158068fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:52.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:52.109+0000 7f7185000640 1 --2- 172.21.15.27:0/2100190555 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7158066af0 0x7f7158068fb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f71680ae000 tx=0x7f7168008040 comp rx=0 tx=0).stop 2024-01-15T18:06:52.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:52.109+0000 7f7185000640 1 -- 172.21.15.27:0/2100190555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71800cde10 msgr2=0x7f7180150370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:52.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:52.109+0000 7f7185000640 1 --2- 172.21.15.27:0/2100190555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71800cde10 0x7f7180150370 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f717000c3c0 tx=0x7f717000c880 comp rx=0 tx=0).stop 2024-01-15T18:06:52.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:52.109+0000 7f7185000640 1 -- 172.21.15.27:0/2100190555 shutdown_connections 2024-01-15T18:06:52.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:52.109+0000 7f7185000640 1 --2- 172.21.15.27:0/2100190555 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7158066af0 0x7f7158068fb0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:52.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:52.109+0000 7f7185000640 1 --2- 172.21.15.27:0/2100190555 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f718015d680 0x7f71800aeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:52.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:52.109+0000 7f7185000640 1 --2- 172.21.15.27:0/2100190555 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f71800cfc30 0x7f71800c5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:52.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:52.109+0000 7f7185000640 1 --2- 172.21.15.27:0/2100190555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71800cde10 0x7f7180150370 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:52.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:52.109+0000 7f7185000640 1 -- 172.21.15.27:0/2100190555 >> 172.21.15.27:0/2100190555 conn(0x7f71800d0e80 msgr2=0x7f71800d1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:52.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:52.109+0000 7f7185000640 1 -- 172.21.15.27:0/2100190555 shutdown_connections 2024-01-15T18:06:52.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:52.109+0000 7f7185000640 1 -- 172.21.15.27:0/2100190555 wait complete. 2024-01-15T18:06:52.145 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:06:53.146 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:06:53.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.293+0000 7f35b5600640 1 Processor -- start 2024-01-15T18:06:53.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.293+0000 7f35b5600640 1 -- start start 2024-01-15T18:06:53.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.293+0000 7f35b5600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f35b015d5c0 0x7f35b0071c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:53.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.293+0000 7f35b5600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f35b0072190 0x7f35b00b5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:53.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.293+0000 7f35b5600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b00b5980 con 0x7f35b00cf570 2024-01-15T18:06:53.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.293+0000 7f35b5600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b00b5760 con 0x7f35b015d5c0 2024-01-15T18:06:53.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.293+0000 7f35b5600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b00725b0 con 0x7f35b0072190 2024-01-15T18:06:53.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.293+0000 7f35aea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f35b015d5c0 0x7f35b0071c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:53.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.293+0000 7f35ae000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f35b00cf570 0x7f35b00b4b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36860/0 (socket says 172.21.15.27:36860) 2024-01-15T18:06:53.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.293+0000 7f35ae000640 1 -- 172.21.15.27:0/4083260552 learned_addr learned my addr 172.21.15.27:0/4083260552 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:06:53.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.293+0000 7f35afe00640 1 --2- 172.21.15.27:0/4083260552 >> v2:172.21.15.93:3300/0 conn(0x7f35b0072190 0x7f35b00b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:53.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.293+0000 7f35aea00640 1 -- 172.21.15.27:0/4083260552 >> v2:172.21.15.93:3300/0 conn(0x7f35b0072190 msgr2=0x7f35b00b5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:53.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.293+0000 7f35aea00640 1 --2- 172.21.15.27:0/4083260552 >> v2:172.21.15.93:3300/0 conn(0x7f35b0072190 0x7f35b00b5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:53.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.293+0000 7f35aea00640 1 -- 172.21.15.27:0/4083260552 >> v1:172.21.15.27:6789/0 conn(0x7f35b00cf570 legacy=0x7f35b00b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:53.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.293+0000 7f35aea00640 1 -- 172.21.15.27:0/4083260552 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b00ce7b0 con 0x7f35b015d5c0 2024-01-15T18:06:53.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.297+0000 7f35aea00640 1 --2- 172.21.15.27:0/4083260552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35b015d5c0 0x7f35b0071c50 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f35b0072d60 tx=0x7f35a001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa29d3cd377d34a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:53.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.297+0000 7f35ad600640 1 -- 172.21.15.27:0/4083260552 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a00ab600 con 0x7f35b015d5c0 2024-01-15T18:06:53.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.297+0000 7f35ad600640 1 -- 172.21.15.27:0/4083260552 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a00a7050 con 0x7f35b015d5c0 2024-01-15T18:06:53.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.297+0000 7f35ad600640 1 -- 172.21.15.27:0/4083260552 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a00ab980 con 0x7f35b015d5c0 2024-01-15T18:06:53.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.297+0000 7f35b5600640 1 -- 172.21.15.27:0/4083260552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35b015d5c0 msgr2=0x7f35b0071c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:53.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.297+0000 7f35b5600640 1 --2- 172.21.15.27:0/4083260552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35b015d5c0 0x7f35b0071c50 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f35b0072d60 tx=0x7f35a001c3d0 comp rx=0 tx=0).stop 2024-01-15T18:06:53.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.297+0000 7f35b5600640 1 -- 172.21.15.27:0/4083260552 shutdown_connections 2024-01-15T18:06:53.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.297+0000 7f35b5600640 1 --2- 172.21.15.27:0/4083260552 >> v2:172.21.15.93:3300/0 conn(0x7f35b0072190 0x7f35b00b5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:53.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.297+0000 7f35b5600640 1 --2- 172.21.15.27:0/4083260552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35b015d5c0 0x7f35b0071c50 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:53.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.297+0000 7f35b5600640 1 -- 172.21.15.27:0/4083260552 >> 172.21.15.27:0/4083260552 conn(0x7f35b0099e10 msgr2=0x7f35b00b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:53.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.297+0000 7f35b5600640 1 -- 172.21.15.27:0/4083260552 shutdown_connections 2024-01-15T18:06:53.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.297+0000 7f35b5600640 1 -- 172.21.15.27:0/4083260552 wait complete. 2024-01-15T18:06:53.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.301+0000 7f35b5600640 1 Processor -- start 2024-01-15T18:06:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.301+0000 7f35b5600640 1 -- start start 2024-01-15T18:06:53.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.301+0000 7f35b5600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35b0072190 0x7f35b006c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:53.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.301+0000 7f35b5600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f35b00cf570 0x7f35b0159a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:53.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.301+0000 7f35b5600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35b015d5c0 0x7f35b01494c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:53.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.301+0000 7f35b5600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b006b260 con 0x7f35b0072190 2024-01-15T18:06:53.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.301+0000 7f35b5600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b014f6a0 con 0x7f35b00cf570 2024-01-15T18:06:53.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.301+0000 7f35b5600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b01501c0 con 0x7f35b015d5c0 2024-01-15T18:06:53.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.301+0000 7f35aea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35b0072190 0x7f35b006c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:53.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.301+0000 7f35aea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35b0072190 0x7f35b006c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40984/0 (socket says 172.21.15.27:40984) 2024-01-15T18:06:53.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.301+0000 7f35aea00640 1 -- 172.21.15.27:0/4151011517 learned_addr learned my addr 172.21.15.27:0/4151011517 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:53.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.301+0000 7f35afe00640 1 --2- 172.21.15.27:0/4151011517 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35b015d5c0 0x7f35b01494c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:53.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.301+0000 7f35ae000640 1 --2- 172.21.15.27:0/4151011517 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f35b00cf570 0x7f35b0159a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:53.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.301+0000 7f35aea00640 1 -- 172.21.15.27:0/4151011517 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35b015d5c0 msgr2=0x7f35b01494c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:53.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.301+0000 7f35aea00640 1 --2- 172.21.15.27:0/4151011517 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35b015d5c0 0x7f35b01494c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:53.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.301+0000 7f35aea00640 1 -- 172.21.15.27:0/4151011517 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f35b00cf570 msgr2=0x7f35b0159a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:53.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.301+0000 7f35aea00640 1 --2- 172.21.15.27:0/4151011517 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f35b00cf570 0x7f35b0159a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:53.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.301+0000 7f35aea00640 1 -- 172.21.15.27:0/4151011517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a0009420 con 0x7f35b0072190 2024-01-15T18:06:53.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.301+0000 7f35ae000640 1 --2- 172.21.15.27:0/4151011517 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f35b00cf570 0x7f35b0159a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:06:53.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.301+0000 7f35aea00640 1 --2- 172.21.15.27:0/4151011517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35b0072190 0x7f35b006c630 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f35a00ae040 tx=0x7f35a001ac10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:53.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.301+0000 7f35a6a00640 1 -- 172.21.15.27:0/4151011517 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a00b7450 con 0x7f35b0072190 2024-01-15T18:06:53.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.301+0000 7f35a6a00640 1 -- 172.21.15.27:0/4151011517 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a00b75b0 con 0x7f35b0072190 2024-01-15T18:06:53.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.301+0000 7f35a6a00640 1 -- 172.21.15.27:0/4151011517 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a00b7960 con 0x7f35b0072190 2024-01-15T18:06:53.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.301+0000 7f35b5600640 1 -- 172.21.15.27:0/4151011517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b014a470 con 0x7f35b0072190 2024-01-15T18:06:53.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.301+0000 7f35b5600640 1 -- 172.21.15.27:0/4151011517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b0149ce0 con 0x7f35b0072190 2024-01-15T18:06:53.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.305+0000 7f35b5600640 1 -- 172.21.15.27:0/4151011517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b009e950 con 0x7f35b0072190 2024-01-15T18:06:53.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.305+0000 7f35a6a00640 1 -- 172.21.15.27:0/4151011517 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) v1 ==== 86313+0+0 (secure 0 0 0) 0x7f35a00b7ac0 con 0x7f35b0072190 2024-01-15T18:06:53.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.305+0000 7f35a6a00640 1 --2- 172.21.15.27:0/4151011517 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3588066ad0 0x7f3588068f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:53.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.305+0000 7f35a6a00640 1 -- 172.21.15.27:0/4151011517 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35a011efb0 con 0x7f35b0072190 2024-01-15T18:06:53.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.305+0000 7f35ae000640 1 --2- 172.21.15.27:0/4151011517 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3588066ad0 0x7f3588068f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:53.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.305+0000 7f35ae000640 1 --2- 172.21.15.27:0/4151011517 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3588066ad0 0x7f3588068f90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f35b00b4570 tx=0x7f3598009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:53.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.309+0000 7f35a6a00640 1 -- 172.21.15.27:0/4151011517 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f35a00ebdc0 con 0x7f35b0072190 2024-01-15T18:06:53.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.501+0000 7f35b5600640 1 -- 172.21.15.27:0/4151011517 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f35b004e930 con 0x7f3588066ad0 2024-01-15T18:06:53.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.505+0000 7f35a6a00640 1 -- 172.21.15.27:0/4151011517 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f35b004e930 con 0x7f3588066ad0 2024-01-15T18:06:53.515 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:06:53.515 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:06:53.515 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:06:53.515 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:06:53.515 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:06:53.515 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:06:53.515 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:06:53.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:06:53.516 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:06:53.516 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 33.33333333333333, 2024-01-15T18:06:53.516 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "1 GB/3 GB", 2024-01-15T18:06:53.516 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "1/3", 2024-01-15T18:06:53.516 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:06:53.516 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:06:53.516 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:53.516 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:53.516 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:06:53.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.509+0000 7f35b5600640 1 -- 172.21.15.27:0/4151011517 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3588066ad0 msgr2=0x7f3588068f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:53.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.509+0000 7f35b5600640 1 --2- 172.21.15.27:0/4151011517 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3588066ad0 0x7f3588068f90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f35b00b4570 tx=0x7f3598009040 comp rx=0 tx=0).stop 2024-01-15T18:06:53.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.509+0000 7f35b5600640 1 -- 172.21.15.27:0/4151011517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35b0072190 msgr2=0x7f35b006c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:53.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.509+0000 7f35b5600640 1 --2- 172.21.15.27:0/4151011517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35b0072190 0x7f35b006c630 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f35a00ae040 tx=0x7f35a001ac10 comp rx=0 tx=0).stop 2024-01-15T18:06:53.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.509+0000 7f35b5600640 1 -- 172.21.15.27:0/4151011517 shutdown_connections 2024-01-15T18:06:53.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.509+0000 7f35b5600640 1 --2- 172.21.15.27:0/4151011517 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3588066ad0 0x7f3588068f90 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:53.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.509+0000 7f35b5600640 1 --2- 172.21.15.27:0/4151011517 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35b015d5c0 0x7f35b01494c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:53.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.509+0000 7f35b5600640 1 --2- 172.21.15.27:0/4151011517 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f35b00cf570 0x7f35b0159a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:53.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.509+0000 7f35b5600640 1 --2- 172.21.15.27:0/4151011517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35b0072190 0x7f35b006c630 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:53.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.509+0000 7f35b5600640 1 -- 172.21.15.27:0/4151011517 >> 172.21.15.27:0/4151011517 conn(0x7f35b0099e10 msgr2=0x7f35b00cd200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:53.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.509+0000 7f35b5600640 1 -- 172.21.15.27:0/4151011517 shutdown_connections 2024-01-15T18:06:53.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:53.509+0000 7f35b5600640 1 -- 172.21.15.27:0/4151011517 wait complete. 2024-01-15T18:06:53.537 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:06:54.537 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:06:54.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.753+0000 7ff059200640 1 Processor -- start 2024-01-15T18:06:54.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.753+0000 7ff059200640 1 -- start start 2024-01-15T18:06:54.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.753+0000 7ff059200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff0540cf460 0x7ff0540cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:54.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.753+0000 7ff059200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff05415b4b0 0x7ff0540cba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:54.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.753+0000 7ff059200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0540cfd80 con 0x7ff0540cec60 2024-01-15T18:06:54.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.753+0000 7ff059200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0540d19a0 con 0x7ff0540cf460 2024-01-15T18:06:54.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.753+0000 7ff059200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0540d1c80 con 0x7ff05415b4b0 2024-01-15T18:06:54.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.757+0000 7ff051400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff05415b4b0 0x7ff0540cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:54.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.757+0000 7ff051400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff05415b4b0 0x7ff0540cba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:37696/0 (socket says 172.21.15.27:37696) 2024-01-15T18:06:54.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.757+0000 7ff051400640 1 -- 172.21.15.27:0/3147377705 learned_addr learned my addr 172.21.15.27:0/3147377705 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:54.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.757+0000 7ff051e00640 1 --2- 172.21.15.27:0/3147377705 >> v2:172.21.15.93:3300/0 conn(0x7ff0540cf460 0x7ff0540cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:54.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.757+0000 7ff051400640 1 -- 172.21.15.27:0/3147377705 >> v2:172.21.15.93:3300/0 conn(0x7ff0540cf460 msgr2=0x7ff0540cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:54.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.757+0000 7ff051400640 1 --2- 172.21.15.27:0/3147377705 >> v2:172.21.15.93:3300/0 conn(0x7ff0540cf460 0x7ff0540cf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:54.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.757+0000 7ff051400640 1 -- 172.21.15.27:0/3147377705 >> v1:172.21.15.93:6789/0 conn(0x7ff0540cec60 legacy=0x7ff0540cbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:54.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.757+0000 7ff051400640 1 -- 172.21.15.27:0/3147377705 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0540b07c0 con 0x7ff05415b4b0 2024-01-15T18:06:54.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.761+0000 7ff051400640 1 --2- 172.21.15.27:0/3147377705 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff05415b4b0 0x7ff0540cba20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff03c009870 tx=0x7ff03c01c500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=299eeff3dcbbe686 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:54.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.761+0000 7ff050a00640 1 -- 172.21.15.27:0/3147377705 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03c0aa620 con 0x7ff05415b4b0 2024-01-15T18:06:54.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.761+0000 7ff050a00640 1 -- 172.21.15.27:0/3147377705 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff03c0aa780 con 0x7ff05415b4b0 2024-01-15T18:06:54.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.761+0000 7ff050a00640 1 -- 172.21.15.27:0/3147377705 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03c0aab10 con 0x7ff05415b4b0 2024-01-15T18:06:54.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.761+0000 7ff059200640 1 -- 172.21.15.27:0/3147377705 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff05415b4b0 msgr2=0x7ff0540cba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:54.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.761+0000 7ff059200640 1 --2- 172.21.15.27:0/3147377705 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff05415b4b0 0x7ff0540cba20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff03c009870 tx=0x7ff03c01c500 comp rx=0 tx=0).stop 2024-01-15T18:06:54.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.765+0000 7ff059200640 1 -- 172.21.15.27:0/3147377705 shutdown_connections 2024-01-15T18:06:54.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.765+0000 7ff059200640 1 --2- 172.21.15.27:0/3147377705 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff05415b4b0 0x7ff0540cba20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:54.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.765+0000 7ff059200640 1 --2- 172.21.15.27:0/3147377705 >> v2:172.21.15.93:3300/0 conn(0x7ff0540cf460 0x7ff0540cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:54.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.765+0000 7ff059200640 1 -- 172.21.15.27:0/3147377705 >> 172.21.15.27:0/3147377705 conn(0x7ff0540d0e80 msgr2=0x7ff0540d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:54.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.765+0000 7ff059200640 1 -- 172.21.15.27:0/3147377705 shutdown_connections 2024-01-15T18:06:54.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.765+0000 7ff059200640 1 -- 172.21.15.27:0/3147377705 wait complete. 2024-01-15T18:06:54.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.765+0000 7ff059200640 1 Processor -- start 2024-01-15T18:06:54.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.765+0000 7ff059200640 1 -- start start 2024-01-15T18:06:54.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.769+0000 7ff059200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0540cec60 0x7ff0541550c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:54.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.769+0000 7ff059200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0540cf460 0x7ff054155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:54.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.769+0000 7ff059200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff05415b4b0 0x7ff0541f6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:54.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.769+0000 7ff059200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0540c8e40 con 0x7ff05415b4b0 2024-01-15T18:06:54.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.769+0000 7ff059200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0540c7f50 con 0x7ff0540cf460 2024-01-15T18:06:54.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.769+0000 7ff059200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0540c8230 con 0x7ff0540cec60 2024-01-15T18:06:54.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.769+0000 7ff052800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff05415b4b0 0x7ff0541f6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:54.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.769+0000 7ff052800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff05415b4b0 0x7ff0541f6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55974/0 (socket says 172.21.15.27:55974) 2024-01-15T18:06:54.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.769+0000 7ff052800640 1 -- 172.21.15.27:0/2887890060 learned_addr learned my addr 172.21.15.27:0/2887890060 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:54.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.769+0000 7ff051e00640 1 --2- 172.21.15.27:0/2887890060 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0540cec60 0x7ff0541550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:54.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.769+0000 7ff051400640 1 --2- 172.21.15.27:0/2887890060 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0540cf460 0x7ff054155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:54.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.769+0000 7ff052800640 1 -- 172.21.15.27:0/2887890060 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0540cec60 msgr2=0x7ff0541550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:54.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.769+0000 7ff052800640 1 --2- 172.21.15.27:0/2887890060 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0540cec60 0x7ff0541550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:54.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.769+0000 7ff052800640 1 -- 172.21.15.27:0/2887890060 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0540cf460 msgr2=0x7ff054155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:54.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.769+0000 7ff052800640 1 --2- 172.21.15.27:0/2887890060 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0540cf460 0x7ff054155600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:54.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.769+0000 7ff052800640 1 -- 172.21.15.27:0/2887890060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff03c0094d0 con 0x7ff05415b4b0 2024-01-15T18:06:54.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.769+0000 7ff052800640 1 --2- 172.21.15.27:0/2887890060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff05415b4b0 0x7ff0541f6a20 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff04c00b970 tx=0x7ff04c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:54.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.769+0000 7ff04a000640 1 -- 172.21.15.27:0/2887890060 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff04c00cb40 con 0x7ff05415b4b0 2024-01-15T18:06:54.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.773+0000 7ff059200640 1 -- 172.21.15.27:0/2887890060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0540c8510 con 0x7ff05415b4b0 2024-01-15T18:06:54.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.773+0000 7ff059200640 1 -- 172.21.15.27:0/2887890060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0540c8ad0 con 0x7ff05415b4b0 2024-01-15T18:06:54.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.773+0000 7ff04a000640 1 -- 172.21.15.27:0/2887890060 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff04c00cca0 con 0x7ff05415b4b0 2024-01-15T18:06:54.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.773+0000 7ff04a000640 1 -- 172.21.15.27:0/2887890060 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff04c014680 con 0x7ff05415b4b0 2024-01-15T18:06:54.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.773+0000 7ff059200640 1 -- 172.21.15.27:0/2887890060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff024005260 con 0x7ff05415b4b0 2024-01-15T18:06:54.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.773+0000 7ff04a000640 1 -- 172.21.15.27:0/2887890060 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) v1 ==== 86313+0+0 (secure 0 0 0) 0x7ff04c01d3b0 con 0x7ff05415b4b0 2024-01-15T18:06:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.777+0000 7ff04a000640 1 --2- 172.21.15.27:0/2887890060 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff02c066aa0 0x7ff02c068f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:54.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.777+0000 7ff04a000640 1 -- 172.21.15.27:0/2887890060 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff04c084370 con 0x7ff05415b4b0 2024-01-15T18:06:54.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.781+0000 7ff051e00640 1 --2- 172.21.15.27:0/2887890060 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff02c066aa0 0x7ff02c068f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:54.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.781+0000 7ff051e00640 1 --2- 172.21.15.27:0/2887890060 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff02c066aa0 0x7ff02c068f60 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff044009b60 tx=0x7ff044009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:54.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:54.781+0000 7ff04a000640 1 -- 172.21.15.27:0/2887890060 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff04c051520 con 0x7ff05415b4b0 2024-01-15T18:06:55.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:55.049+0000 7ff059200640 1 -- 172.21.15.27:0/2887890060 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff024000bd0 con 0x7ff02c066aa0 2024-01-15T18:06:55.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:55.069+0000 7ff04a000640 1 -- 172.21.15.27:0/2887890060 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7ff024000bd0 con 0x7ff02c066aa0 2024-01-15T18:06:55.077 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:06:55.077 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:06:55.078 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:06:55.078 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:06:55.078 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:06:55.078 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:06:55.078 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:06:55.078 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:06:55.078 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:06:55.078 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 33.33333333333333, 2024-01-15T18:06:55.078 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "1 GB/3 GB", 2024-01-15T18:06:55.078 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "1/3", 2024-01-15T18:06:55.078 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:06:55.078 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:06:55.078 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:55.078 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:55.079 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:06:55.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:55.073+0000 7ff059200640 1 -- 172.21.15.27:0/2887890060 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff02c066aa0 msgr2=0x7ff02c068f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:55.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:55.073+0000 7ff059200640 1 --2- 172.21.15.27:0/2887890060 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff02c066aa0 0x7ff02c068f60 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff044009b60 tx=0x7ff044009190 comp rx=0 tx=0).stop 2024-01-15T18:06:55.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:55.073+0000 7ff059200640 1 -- 172.21.15.27:0/2887890060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff05415b4b0 msgr2=0x7ff0541f6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:55.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:55.073+0000 7ff059200640 1 --2- 172.21.15.27:0/2887890060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff05415b4b0 0x7ff0541f6a20 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff04c00b970 tx=0x7ff04c00be30 comp rx=0 tx=0).stop 2024-01-15T18:06:55.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:55.077+0000 7ff059200640 1 -- 172.21.15.27:0/2887890060 shutdown_connections 2024-01-15T18:06:55.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:55.077+0000 7ff059200640 1 --2- 172.21.15.27:0/2887890060 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff02c066aa0 0x7ff02c068f60 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:55.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:55.077+0000 7ff059200640 1 --2- 172.21.15.27:0/2887890060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff05415b4b0 0x7ff0541f6a20 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:55.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:55.077+0000 7ff059200640 1 --2- 172.21.15.27:0/2887890060 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0540cf460 0x7ff054155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:55.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:55.077+0000 7ff059200640 1 --2- 172.21.15.27:0/2887890060 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0540cec60 0x7ff0541550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:55.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:55.077+0000 7ff059200640 1 -- 172.21.15.27:0/2887890060 >> 172.21.15.27:0/2887890060 conn(0x7ff0540d0e80 msgr2=0x7ff0540cb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:55.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:55.077+0000 7ff059200640 1 -- 172.21.15.27:0/2887890060 shutdown_connections 2024-01-15T18:06:55.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:55.077+0000 7ff059200640 1 -- 172.21.15.27:0/2887890060 wait complete. 2024-01-15T18:06:55.117 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:06:56.118 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:06:56.225 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:06:56.229 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:06:56.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.273+0000 7f0e41200640 1 Processor -- start 2024-01-15T18:06:56.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.273+0000 7f0e41200640 1 -- start start 2024-01-15T18:06:56.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.273+0000 7f0e41200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0e3c15d680 0x7f0e3c0cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:56.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.273+0000 7f0e41200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e3c06c750 con 0x7f0e3c0cfc30 2024-01-15T18:06:56.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.273+0000 7f0e41200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e3c0cb4d0 con 0x7f0e3c0ccf80 2024-01-15T18:06:56.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.273+0000 7f0e41200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c0d0120 con 0x7f0e3c15d680 2024-01-15T18:06:56.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.273+0000 7f0e39400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0e3c0cfc30 0x7f0e3c06b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36922/0 (socket says 172.21.15.27:36922) 2024-01-15T18:06:56.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.273+0000 7f0e39400640 1 -- 172.21.15.27:0/1466966093 learned_addr learned my addr 172.21.15.27:0/1466966093 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:06:56.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.273+0000 7f0e39e00640 1 --2- 172.21.15.27:0/1466966093 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0e3c15d680 0x7f0e3c0cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:56.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.273+0000 7f0e38a00640 1 -- 172.21.15.27:0/1466966093 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707246146 0 0) 0x7f0e3c06c750 con 0x7f0e3c0cfc30 2024-01-15T18:06:56.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.273+0000 7f0e38a00640 1 -- 172.21.15.27:0/1466966093 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e1c003530 con 0x7f0e3c0cfc30 2024-01-15T18:06:56.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.273+0000 7f0e38a00640 1 -- 172.21.15.27:0/1466966093 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2543610177 0 0) 0x7f0e1c003530 con 0x7f0e3c0cfc30 2024-01-15T18:06:56.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.273+0000 7f0e38a00640 1 -- 172.21.15.27:0/1466966093 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0e3c15d680 msgr2=0x7f0e3c0cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:56.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.273+0000 7f0e38a00640 1 --2- 172.21.15.27:0/1466966093 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0e3c15d680 0x7f0e3c0cd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:56.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.273+0000 7f0e38a00640 1 -- 172.21.15.27:0/1466966093 >> v1:172.21.15.93:6789/0 conn(0x7f0e3c0ccf80 legacy=0x7f0e3c06c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:56.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.273+0000 7f0e38a00640 1 -- 172.21.15.27:0/1466966093 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e3c0d0400 con 0x7f0e3c0cfc30 2024-01-15T18:06:56.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.273+0000 7f0e38a00640 1 -- 172.21.15.27:0/1466966093 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f0e24001680 con 0x7f0e3c0cfc30 2024-01-15T18:06:56.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.273+0000 7f0e39e00640 1 --2- 172.21.15.27:0/1466966093 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0e3c15d680 0x7f0e3c0cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:06:56.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.273+0000 7f0e38a00640 1 -- 172.21.15.27:0/1466966093 >> v1:172.21.15.27:6789/0 conn(0x7f0e3c0cfc30 legacy=0x7f0e3c06b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:56.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.273+0000 7f0e38a00640 1 --2- 172.21.15.27:0/1466966093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c003a20 0x7f0e1c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:56.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.277+0000 7f0e38a00640 1 --2- 172.21.15.27:0/1466966093 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0e1c0047d0 0x7f0e1c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:56.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.277+0000 7f0e38a00640 1 --2- 172.21.15.27:0/1466966093 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0e3c15d680 0x7f0e1c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:56.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.277+0000 7f0e38a00640 1 -- 172.21.15.27:0/1466966093 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c0d0120 con 0x7f0e1c003a20 2024-01-15T18:06:56.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.277+0000 7f0e38a00640 1 -- 172.21.15.27:0/1466966093 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c00b9c0 con 0x7f0e1c0047d0 2024-01-15T18:06:56.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.277+0000 7f0e38a00640 1 -- 172.21.15.27:0/1466966093 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c00bb80 con 0x7f0e3c15d680 2024-01-15T18:06:56.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.277+0000 7f0e39e00640 1 --2- 172.21.15.27:0/1466966093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c003a20 0x7f0e1c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:56.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.277+0000 7f0e3a800640 1 --2- 172.21.15.27:0/1466966093 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0e3c15d680 0x7f0e1c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:56.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.277+0000 7f0e39400640 1 --2- 172.21.15.27:0/1466966093 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0e1c0047d0 0x7f0e1c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:56.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.277+0000 7f0e39e00640 1 -- 172.21.15.27:0/1466966093 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0e3c15d680 msgr2=0x7f0e1c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:56.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.277+0000 7f0e39e00640 1 --2- 172.21.15.27:0/1466966093 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0e3c15d680 0x7f0e1c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:56.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.277+0000 7f0e39e00640 1 -- 172.21.15.27:0/1466966093 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0e1c0047d0 msgr2=0x7f0e1c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:56.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.277+0000 7f0e39e00640 1 --2- 172.21.15.27:0/1466966093 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0e1c0047d0 0x7f0e1c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:56.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.277+0000 7f0e39e00640 1 -- 172.21.15.27:0/1466966093 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2c009420 con 0x7f0e1c003a20 2024-01-15T18:06:56.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.277+0000 7f0e39e00640 1 --2- 172.21.15.27:0/1466966093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c003a20 0x7f0e1c003e00 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c009c80 tx=0x7f0e2c01cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9e382148756d2e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:56.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.277+0000 7f0e38a00640 1 -- 172.21.15.27:0/1466966093 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c0a5850 con 0x7f0e1c003a20 2024-01-15T18:06:56.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.277+0000 7f0e38a00640 1 -- 172.21.15.27:0/1466966093 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e2c0a59b0 con 0x7f0e1c003a20 2024-01-15T18:06:56.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.277+0000 7f0e38a00640 1 -- 172.21.15.27:0/1466966093 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c0a5d60 con 0x7f0e1c003a20 2024-01-15T18:06:56.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.281+0000 7f0e41200640 1 -- 172.21.15.27:0/1466966093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c003a20 msgr2=0x7f0e1c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:56.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.281+0000 7f0e41200640 1 --2- 172.21.15.27:0/1466966093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c003a20 0x7f0e1c003e00 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c009c80 tx=0x7f0e2c01cb90 comp rx=0 tx=0).stop 2024-01-15T18:06:56.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.281+0000 7f0e3a800640 1 -- 172.21.15.27:0/1466966093 reap_dead start 2024-01-15T18:06:56.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.281+0000 7f0e41200640 1 -- 172.21.15.27:0/1466966093 shutdown_connections 2024-01-15T18:06:56.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.281+0000 7f0e41200640 1 -- 172.21.15.27:0/1466966093 >> 172.21.15.27:0/1466966093 conn(0x7f0e3c0d0e80 msgr2=0x7f0e3c0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:56.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.281+0000 7f0e41200640 1 -- 172.21.15.27:0/1466966093 shutdown_connections 2024-01-15T18:06:56.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.281+0000 7f0e41200640 1 -- 172.21.15.27:0/1466966093 wait complete. 2024-01-15T18:06:56.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.281+0000 7f0e41200640 1 Processor -- start 2024-01-15T18:06:56.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.281+0000 7f0e41200640 1 -- start start 2024-01-15T18:06:56.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.281+0000 7f0e41200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c003a20 0x7f0e3c1501d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:56.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.281+0000 7f0e41200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0e3c0af060 0x7f0e3c0af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:56.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.281+0000 7f0e41200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0e3c15da90 0x7f0e3c15df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:56.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.281+0000 7f0e41200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c0afa20 con 0x7f0e1c003a20 2024-01-15T18:06:56.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.281+0000 7f0e41200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c15e780 con 0x7f0e3c0af060 2024-01-15T18:06:56.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.281+0000 7f0e41200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c15ea60 con 0x7f0e3c15da90 2024-01-15T18:06:56.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.281+0000 7f0e39e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c003a20 0x7f0e3c1501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:56.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.281+0000 7f0e39e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c003a20 0x7f0e3c1501d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55992/0 (socket says 172.21.15.27:55992) 2024-01-15T18:06:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.281+0000 7f0e39e00640 1 -- 172.21.15.27:0/3949745207 learned_addr learned my addr 172.21.15.27:0/3949745207 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.285+0000 7f0e3a800640 1 --2- 172.21.15.27:0/3949745207 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0e3c15da90 0x7f0e3c15df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.285+0000 7f0e39400640 1 --2- 172.21.15.27:0/3949745207 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0e3c0af060 0x7f0e3c0af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.285+0000 7f0e39e00640 1 -- 172.21.15.27:0/3949745207 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0e3c15da90 msgr2=0x7f0e3c15df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.285+0000 7f0e39e00640 1 --2- 172.21.15.27:0/3949745207 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0e3c15da90 0x7f0e3c15df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.285+0000 7f0e39e00640 1 -- 172.21.15.27:0/3949745207 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0e3c0af060 msgr2=0x7f0e3c0af4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.285+0000 7f0e39e00640 1 --2- 172.21.15.27:0/3949745207 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0e3c0af060 0x7f0e3c0af4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.285+0000 7f0e39e00640 1 -- 172.21.15.27:0/3949745207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e3c15ed40 con 0x7f0e1c003a20 2024-01-15T18:06:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.285+0000 7f0e39e00640 1 --2- 172.21.15.27:0/3949745207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c003a20 0x7f0e3c1501d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c009c80 tx=0x7f0e1c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.285+0000 7f0e32000640 1 -- 172.21.15.27:0/3949745207 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c004220 con 0x7f0e1c003a20 2024-01-15T18:06:56.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.285+0000 7f0e32000640 1 -- 172.21.15.27:0/3949745207 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e2c004380 con 0x7f0e1c003a20 2024-01-15T18:06:56.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.285+0000 7f0e41200640 1 -- 172.21.15.27:0/3949745207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e3c15f020 con 0x7f0e1c003a20 2024-01-15T18:06:56.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.285+0000 7f0e32000640 1 -- 172.21.15.27:0/3949745207 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c0ad870 con 0x7f0e1c003a20 2024-01-15T18:06:56.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.285+0000 7f0e41200640 1 -- 172.21.15.27:0/3949745207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e3c15e450 con 0x7f0e1c003a20 2024-01-15T18:06:56.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.285+0000 7f0e41200640 1 -- 172.21.15.27:0/3949745207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e3c04e850 con 0x7f0e1c003a20 2024-01-15T18:06:56.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.289+0000 7f0e32000640 1 -- 172.21.15.27:0/3949745207 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) v1 ==== 86313+0+0 (secure 0 0 0) 0x7f0e2c0b4040 con 0x7f0e1c003a20 2024-01-15T18:06:56.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.289+0000 7f0e32000640 1 --2- 172.21.15.27:0/3949745207 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0e14066aa0 0x7f0e14068f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:56.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.289+0000 7f0e32000640 1 -- 172.21.15.27:0/3949745207 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e2c11f160 con 0x7f0e1c003a20 2024-01-15T18:06:56.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.289+0000 7f0e39400640 1 --2- 172.21.15.27:0/3949745207 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0e14066aa0 0x7f0e14068f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:56.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.289+0000 7f0e39400640 1 --2- 172.21.15.27:0/3949745207 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0e14066aa0 0x7f0e14068f60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0e24001060 tx=0x7f0e24001c40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:56.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.289+0000 7f0e32000640 1 -- 172.21.15.27:0/3949745207 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0e2c0ec3f0 con 0x7f0e1c003a20 2024-01-15T18:06:56.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.481+0000 7f0e41200640 1 -- 172.21.15.27:0/3949745207 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e3c0afa20 con 0x7f0e14066aa0 2024-01-15T18:06:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.485+0000 7f0e32000640 1 -- 172.21.15.27:0/3949745207 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f0e3c0afa20 con 0x7f0e14066aa0 2024-01-15T18:06:56.494 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:06:56.494 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:06:56.494 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:06:56.494 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:06:56.494 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:06:56.494 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:06:56.494 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:06:56.494 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:06:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:06:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 66.66666666666666, 2024-01-15T18:06:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T18:06:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "2/3", 2024-01-15T18:06:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:06:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:06:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:56.495 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:56.495 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:06:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.489+0000 7f0e41200640 1 -- 172.21.15.27:0/3949745207 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0e14066aa0 msgr2=0x7f0e14068f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.489+0000 7f0e41200640 1 --2- 172.21.15.27:0/3949745207 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0e14066aa0 0x7f0e14068f60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0e24001060 tx=0x7f0e24001c40 comp rx=0 tx=0).stop 2024-01-15T18:06:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.489+0000 7f0e41200640 1 -- 172.21.15.27:0/3949745207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c003a20 msgr2=0x7f0e3c1501d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.489+0000 7f0e41200640 1 --2- 172.21.15.27:0/3949745207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c003a20 0x7f0e3c1501d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c009c80 tx=0x7f0e1c005800 comp rx=0 tx=0).stop 2024-01-15T18:06:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.489+0000 7f0e41200640 1 -- 172.21.15.27:0/3949745207 shutdown_connections 2024-01-15T18:06:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.489+0000 7f0e41200640 1 --2- 172.21.15.27:0/3949745207 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0e14066aa0 0x7f0e14068f60 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.489+0000 7f0e41200640 1 --2- 172.21.15.27:0/3949745207 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0e3c15da90 0x7f0e3c15df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.489+0000 7f0e41200640 1 --2- 172.21.15.27:0/3949745207 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0e3c0af060 0x7f0e3c0af4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.489+0000 7f0e41200640 1 --2- 172.21.15.27:0/3949745207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0e1c003a20 0x7f0e3c1501d0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.489+0000 7f0e41200640 1 -- 172.21.15.27:0/3949745207 >> 172.21.15.27:0/3949745207 conn(0x7f0e3c0d0e80 msgr2=0x7f0e3c15aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.489+0000 7f0e41200640 1 -- 172.21.15.27:0/3949745207 shutdown_connections 2024-01-15T18:06:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:56.489+0000 7f0e41200640 1 -- 172.21.15.27:0/3949745207 wait complete. 2024-01-15T18:06:56.515 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:06:57.516 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:06:57.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.665+0000 7fc5cee00640 1 Processor -- start 2024-01-15T18:06:57.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.665+0000 7fc5cee00640 1 -- start start 2024-01-15T18:06:57.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.665+0000 7fc5cee00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc5c815f550 0x7fc5c80cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:57.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.665+0000 7fc5cee00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5c80cf110 0x7fc5c80701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:57.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.665+0000 7fc5cee00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c80b1cb0 con 0x7fc5c8158900 2024-01-15T18:06:57.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.665+0000 7fc5cee00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c80cf580 con 0x7fc5c815f550 2024-01-15T18:06:57.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.665+0000 7fc5cee00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c815fa00 con 0x7fc5c80cf110 2024-01-15T18:06:57.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.665+0000 7fc5c7400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc5c815f550 0x7fc5c80cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:57.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.665+0000 7fc5c7e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc5c8158900 0x7fc5c815bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36938/0 (socket says 172.21.15.27:36938) 2024-01-15T18:06:57.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.665+0000 7fc5c7e00640 1 -- 172.21.15.27:0/2573780358 learned_addr learned my addr 172.21.15.27:0/2573780358 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:06:57.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.665+0000 7fc5cce00640 1 --2- 172.21.15.27:0/2573780358 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5c80cf110 0x7fc5c80701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:57.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.665+0000 7fc5c7400640 1 -- 172.21.15.27:0/2573780358 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5c80cf110 msgr2=0x7fc5c80701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:57.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.665+0000 7fc5c7400640 1 --2- 172.21.15.27:0/2573780358 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5c80cf110 0x7fc5c80701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:57.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.665+0000 7fc5c7400640 1 -- 172.21.15.27:0/2573780358 >> v1:172.21.15.27:6789/0 conn(0x7fc5c8158900 legacy=0x7fc5c815bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:57.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.665+0000 7fc5c7400640 1 -- 172.21.15.27:0/2573780358 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c80b1f00 con 0x7fc5c815f550 2024-01-15T18:06:57.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.669+0000 7fc5c7400640 1 --2- 172.21.15.27:0/2573780358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c815f550 0x7fc5c80cebd0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc5b00097c0 tx=0x7fc5b001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4e77465fb956a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:57.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.669+0000 7fc5c6a00640 1 -- 172.21.15.27:0/2573780358 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8070f30 con 0x7fc5c815f550 2024-01-15T18:06:57.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.669+0000 7fc5c6a00640 1 -- 172.21.15.27:0/2573780358 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b00ab650 con 0x7fc5c815f550 2024-01-15T18:06:57.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.669+0000 7fc5c6a00640 1 -- 172.21.15.27:0/2573780358 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b00ab9e0 con 0x7fc5c815f550 2024-01-15T18:06:57.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.669+0000 7fc5cee00640 1 -- 172.21.15.27:0/2573780358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c815f550 msgr2=0x7fc5c80cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:57.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.669+0000 7fc5cee00640 1 --2- 172.21.15.27:0/2573780358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c815f550 0x7fc5c80cebd0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc5b00097c0 tx=0x7fc5b001c450 comp rx=0 tx=0).stop 2024-01-15T18:06:57.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.669+0000 7fc5cee00640 1 -- 172.21.15.27:0/2573780358 shutdown_connections 2024-01-15T18:06:57.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.669+0000 7fc5cee00640 1 --2- 172.21.15.27:0/2573780358 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5c80cf110 0x7fc5c80701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:57.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.669+0000 7fc5cee00640 1 --2- 172.21.15.27:0/2573780358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c815f550 0x7fc5c80cebd0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:57.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.669+0000 7fc5cee00640 1 -- 172.21.15.27:0/2573780358 >> 172.21.15.27:0/2573780358 conn(0x7fc5c80cb700 msgr2=0x7fc5c80cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:57.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.669+0000 7fc5cee00640 1 -- 172.21.15.27:0/2573780358 shutdown_connections 2024-01-15T18:06:57.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.669+0000 7fc5cee00640 1 -- 172.21.15.27:0/2573780358 wait complete. 2024-01-15T18:06:57.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.673+0000 7fc5cee00640 1 Processor -- start 2024-01-15T18:06:57.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.673+0000 7fc5cee00640 1 -- start start 2024-01-15T18:06:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.673+0000 7fc5cee00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc5c80cf110 0x7fc5c814e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.673+0000 7fc5cee00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8158900 0x7fc5c815c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.673+0000 7fc5cee00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5c815f550 0x7fc5c8151a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.673+0000 7fc5cee00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c8139530 con 0x7fc5c8158900 2024-01-15T18:06:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.673+0000 7fc5cee00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c813bb30 con 0x7fc5c80cf110 2024-01-15T18:06:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.673+0000 7fc5cee00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c813be10 con 0x7fc5c815f550 2024-01-15T18:06:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.673+0000 7fc5c7400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8158900 0x7fc5c815c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.673+0000 7fc5c7400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8158900 0x7fc5c815c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56006/0 (socket says 172.21.15.27:56006) 2024-01-15T18:06:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.673+0000 7fc5c7400640 1 -- 172.21.15.27:0/283910294 learned_addr learned my addr 172.21.15.27:0/283910294 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.673+0000 7fc5c7e00640 1 --2- 172.21.15.27:0/283910294 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc5c80cf110 0x7fc5c814e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.673+0000 7fc5cce00640 1 --2- 172.21.15.27:0/283910294 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5c815f550 0x7fc5c8151a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.673+0000 7fc5c7400640 1 -- 172.21.15.27:0/283910294 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5c815f550 msgr2=0x7fc5c8151a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:57.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.673+0000 7fc5c7400640 1 --2- 172.21.15.27:0/283910294 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5c815f550 0x7fc5c8151a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:57.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.673+0000 7fc5c7400640 1 -- 172.21.15.27:0/283910294 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc5c80cf110 msgr2=0x7fc5c814e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:57.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.673+0000 7fc5c7400640 1 --2- 172.21.15.27:0/283910294 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc5c80cf110 0x7fc5c814e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:57.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.673+0000 7fc5c7400640 1 -- 172.21.15.27:0/283910294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b0009420 con 0x7fc5c8158900 2024-01-15T18:06:57.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.673+0000 7fc5c7400640 1 --2- 172.21.15.27:0/283910294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8158900 0x7fc5c815c000 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc5b00ad040 tx=0x7fc5b001ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:57.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.673+0000 7fc5bfe00640 1 -- 172.21.15.27:0/283910294 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b00b5610 con 0x7fc5c8158900 2024-01-15T18:06:57.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.673+0000 7fc5bfe00640 1 -- 172.21.15.27:0/283910294 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b00b5790 con 0x7fc5c8158900 2024-01-15T18:06:57.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.673+0000 7fc5cee00640 1 -- 172.21.15.27:0/283910294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c813c0f0 con 0x7fc5c8158900 2024-01-15T18:06:57.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.673+0000 7fc5bfe00640 1 -- 172.21.15.27:0/283910294 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b00b5b40 con 0x7fc5c8158900 2024-01-15T18:06:57.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.673+0000 7fc5cee00640 1 -- 172.21.15.27:0/283910294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c815c540 con 0x7fc5c8158900 2024-01-15T18:06:57.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.677+0000 7fc5cee00640 1 -- 172.21.15.27:0/283910294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c8070570 con 0x7fc5c8158900 2024-01-15T18:06:57.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.677+0000 7fc5bfe00640 1 -- 172.21.15.27:0/283910294 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) v1 ==== 86313+0+0 (secure 0 0 0) 0x7fc5b00b5ca0 con 0x7fc5c8158900 2024-01-15T18:06:57.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.677+0000 7fc5bfe00640 1 --2- 172.21.15.27:0/283910294 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc5a0066aa0 0x7fc5a0068f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:57.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.677+0000 7fc5bfe00640 1 -- 172.21.15.27:0/283910294 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc5b011f360 con 0x7fc5c8158900 2024-01-15T18:06:57.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.677+0000 7fc5c7e00640 1 --2- 172.21.15.27:0/283910294 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc5a0066aa0 0x7fc5a0068f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:57.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.677+0000 7fc5c7e00640 1 --2- 172.21.15.27:0/283910294 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc5a0066aa0 0x7fc5a0068f60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8005890 tx=0x7fc5b8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:57.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.681+0000 7fc5bfe00640 1 -- 172.21.15.27:0/283910294 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc5b00ec480 con 0x7fc5c8158900 2024-01-15T18:06:57.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.869+0000 7fc5cee00640 1 -- 172.21.15.27:0/283910294 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5c80cf990 con 0x7fc5a0066aa0 2024-01-15T18:06:57.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.877+0000 7fc5bfe00640 1 -- 172.21.15.27:0/283910294 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fc5c80cf990 con 0x7fc5a0066aa0 2024-01-15T18:06:57.884 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:06:57.884 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:06:57.884 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:06:57.885 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:06:57.885 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:06:57.885 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:06:57.885 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:06:57.885 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:06:57.885 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:06:57.885 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 66.66666666666666, 2024-01-15T18:06:57.885 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T18:06:57.885 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "2/3", 2024-01-15T18:06:57.885 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:06:57.885 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:06:57.885 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:57.885 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:57.885 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:06:57.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.881+0000 7fc5cee00640 1 -- 172.21.15.27:0/283910294 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc5a0066aa0 msgr2=0x7fc5a0068f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:57.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.881+0000 7fc5cee00640 1 --2- 172.21.15.27:0/283910294 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc5a0066aa0 0x7fc5a0068f60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8005890 tx=0x7fc5b8009040 comp rx=0 tx=0).stop 2024-01-15T18:06:57.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.881+0000 7fc5cee00640 1 -- 172.21.15.27:0/283910294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8158900 msgr2=0x7fc5c815c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:57.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.881+0000 7fc5cee00640 1 --2- 172.21.15.27:0/283910294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8158900 0x7fc5c815c000 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc5b00ad040 tx=0x7fc5b001ac90 comp rx=0 tx=0).stop 2024-01-15T18:06:57.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.881+0000 7fc5cee00640 1 -- 172.21.15.27:0/283910294 shutdown_connections 2024-01-15T18:06:57.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.881+0000 7fc5cee00640 1 --2- 172.21.15.27:0/283910294 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc5a0066aa0 0x7fc5a0068f60 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:57.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.881+0000 7fc5cee00640 1 --2- 172.21.15.27:0/283910294 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5c815f550 0x7fc5c8151a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:57.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.881+0000 7fc5cee00640 1 --2- 172.21.15.27:0/283910294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5c8158900 0x7fc5c815c000 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:57.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.881+0000 7fc5cee00640 1 --2- 172.21.15.27:0/283910294 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc5c80cf110 0x7fc5c814e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:57.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.881+0000 7fc5cee00640 1 -- 172.21.15.27:0/283910294 >> 172.21.15.27:0/283910294 conn(0x7fc5c80cb700 msgr2=0x7fc5c8155030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:57.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.881+0000 7fc5cee00640 1 -- 172.21.15.27:0/283910294 shutdown_connections 2024-01-15T18:06:57.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:57.881+0000 7fc5cee00640 1 -- 172.21.15.27:0/283910294 wait complete. 2024-01-15T18:06:57.907 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:06:58.908 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:06:59.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.097+0000 7f5e7c200640 1 Processor -- start 2024-01-15T18:06:59.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.097+0000 7f5e7c200640 1 -- start start 2024-01-15T18:06:59.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.101+0000 7f5e7c200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f5e740b4300 0x7f5e740cf940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:59.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.101+0000 7f5e7c200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e7415be40 0x7f5e740720a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:59.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.101+0000 7f5e7c200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e740725e0 con 0x7f5e74071c50 2024-01-15T18:06:59.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.101+0000 7f5e7c200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e7406a4f0 con 0x7f5e740b4300 2024-01-15T18:06:59.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.101+0000 7f5e7c200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e740cebf0 con 0x7f5e7415be40 2024-01-15T18:06:59.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.101+0000 7f5e78e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5e74071c50 0x7f5e7415b730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36946/0 (socket says 172.21.15.27:36946) 2024-01-15T18:06:59.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.101+0000 7f5e78e00640 1 -- 172.21.15.27:0/57440895 learned_addr learned my addr 172.21.15.27:0/57440895 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:06:59.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.101+0000 7f5e7ac00640 1 --2- 172.21.15.27:0/57440895 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e7415be40 0x7f5e740720a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:59.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.101+0000 7f5e79800640 1 --2- 172.21.15.27:0/57440895 >> v2:172.21.15.93:3300/0 conn(0x7f5e740b4300 0x7f5e740cf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:59.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.101+0000 7f5e73e00640 1 -- 172.21.15.27:0/57440895 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626573323 0 0) 0x7f5e5c001120 con 0x7f5e74071c50 2024-01-15T18:06:59.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.101+0000 7f5e73e00640 1 -- 172.21.15.27:0/57440895 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e54003530 con 0x7f5e74071c50 2024-01-15T18:06:59.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.101+0000 7f5e7ac00640 1 -- 172.21.15.27:0/57440895 >> v2:172.21.15.93:3300/0 conn(0x7f5e740b4300 msgr2=0x7f5e740cf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:59.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.101+0000 7f5e7ac00640 1 --2- 172.21.15.27:0/57440895 >> v2:172.21.15.93:3300/0 conn(0x7f5e740b4300 0x7f5e740cf940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:59.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.101+0000 7f5e7ac00640 1 -- 172.21.15.27:0/57440895 >> v1:172.21.15.27:6789/0 conn(0x7f5e74071c50 legacy=0x7f5e7415b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:59.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.101+0000 7f5e7ac00640 1 -- 172.21.15.27:0/57440895 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e740b46e0 con 0x7f5e7415be40 2024-01-15T18:06:59.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.105+0000 7f5e7ac00640 1 --2- 172.21.15.27:0/57440895 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e7415be40 0x7f5e740720a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c009d90 tx=0x7f5e6c01f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a5547b020582250 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:59.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.113+0000 7f5e73e00640 1 -- 172.21.15.27:0/57440895 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c0af600 con 0x7f5e7415be40 2024-01-15T18:06:59.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.113+0000 7f5e73e00640 1 -- 172.21.15.27:0/57440895 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6c0ab050 con 0x7f5e7415be40 2024-01-15T18:06:59.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.113+0000 7f5e73e00640 1 -- 172.21.15.27:0/57440895 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c0af980 con 0x7f5e7415be40 2024-01-15T18:06:59.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.113+0000 7f5e7c200640 1 -- 172.21.15.27:0/57440895 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e7415be40 msgr2=0x7f5e740720a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:59.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.113+0000 7f5e7c200640 1 --2- 172.21.15.27:0/57440895 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e7415be40 0x7f5e740720a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c009d90 tx=0x7f5e6c01f130 comp rx=0 tx=0).stop 2024-01-15T18:06:59.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.113+0000 7f5e7c200640 1 -- 172.21.15.27:0/57440895 shutdown_connections 2024-01-15T18:06:59.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.113+0000 7f5e7c200640 1 --2- 172.21.15.27:0/57440895 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e7415be40 0x7f5e740720a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:59.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.113+0000 7f5e7c200640 1 --2- 172.21.15.27:0/57440895 >> v2:172.21.15.93:3300/0 conn(0x7f5e740b4300 0x7f5e740cf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:59.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.113+0000 7f5e7c200640 1 -- 172.21.15.27:0/57440895 >> 172.21.15.27:0/57440895 conn(0x7f5e74099e10 msgr2=0x7f5e740b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:59.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.113+0000 7f5e7c200640 1 -- 172.21.15.27:0/57440895 shutdown_connections 2024-01-15T18:06:59.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.117+0000 7f5e7c200640 1 -- 172.21.15.27:0/57440895 wait complete. 2024-01-15T18:06:59.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.117+0000 7f5e7c200640 1 Processor -- start 2024-01-15T18:06:59.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.121+0000 7f5e7c200640 1 -- start start 2024-01-15T18:06:59.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.121+0000 7f5e7c200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e74071c50 0x7f5e7406b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:59.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.121+0000 7f5e7c200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e740b4300 0x7f5e74155010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:59.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.121+0000 7f5e7c200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5e7415be40 0x7f5e7406bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:59.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.121+0000 7f5e7c200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7406ada0 con 0x7f5e740b4300 2024-01-15T18:06:59.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.121+0000 7f5e7c200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e74155550 con 0x7f5e7415be40 2024-01-15T18:06:59.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.121+0000 7f5e7c200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7406c9e0 con 0x7f5e74071c50 2024-01-15T18:06:59.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.121+0000 7f5e78e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e740b4300 0x7f5e74155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:59.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.121+0000 7f5e78e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e740b4300 0x7f5e74155010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56014/0 (socket says 172.21.15.27:56014) 2024-01-15T18:06:59.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.121+0000 7f5e78e00640 1 -- 172.21.15.27:0/3044257481 learned_addr learned my addr 172.21.15.27:0/3044257481 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:06:59.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.121+0000 7f5e79800640 1 --2- 172.21.15.27:0/3044257481 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e74071c50 0x7f5e7406b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:59.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.125+0000 7f5e7ac00640 1 --2- 172.21.15.27:0/3044257481 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5e7415be40 0x7f5e7406bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:59.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.125+0000 7f5e78e00640 1 -- 172.21.15.27:0/3044257481 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e74071c50 msgr2=0x7f5e7406b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:59.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.125+0000 7f5e78e00640 1 --2- 172.21.15.27:0/3044257481 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e74071c50 0x7f5e7406b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:59.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.125+0000 7f5e78e00640 1 -- 172.21.15.27:0/3044257481 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5e7415be40 msgr2=0x7f5e7406bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:59.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.125+0000 7f5e78e00640 1 --2- 172.21.15.27:0/3044257481 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5e7415be40 0x7f5e7406bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:59.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.125+0000 7f5e78e00640 1 -- 172.21.15.27:0/3044257481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e6c009a70 con 0x7f5e740b4300 2024-01-15T18:06:59.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.125+0000 7f5e79800640 1 --2- 172.21.15.27:0/3044257481 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e74071c50 0x7f5e7406b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:06:59.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.125+0000 7f5e78e00640 1 --2- 172.21.15.27:0/3044257481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e740b4300 0x7f5e74155010 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c00c500 tx=0x7f5e5c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:59.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.125+0000 7f5e71600640 1 -- 172.21.15.27:0/3044257481 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5c005cf0 con 0x7f5e740b4300 2024-01-15T18:06:59.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.125+0000 7f5e71600640 1 -- 172.21.15.27:0/3044257481 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e5c005e50 con 0x7f5e740b4300 2024-01-15T18:06:59.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.125+0000 7f5e71600640 1 -- 172.21.15.27:0/3044257481 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5c007310 con 0x7f5e740b4300 2024-01-15T18:06:59.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.125+0000 7f5e7c200640 1 -- 172.21.15.27:0/3044257481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e7406a790 con 0x7f5e740b4300 2024-01-15T18:06:59.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.125+0000 7f5e7c200640 1 -- 172.21.15.27:0/3044257481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e7406aa30 con 0x7f5e740b4300 2024-01-15T18:06:59.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.129+0000 7f5e7c200640 1 -- 172.21.15.27:0/3044257481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e7409e950 con 0x7f5e740b4300 2024-01-15T18:06:59.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.129+0000 7f5e71600640 1 -- 172.21.15.27:0/3044257481 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) v1 ==== 86313+0+0 (secure 0 0 0) 0x7f5e5c007470 con 0x7f5e740b4300 2024-01-15T18:06:59.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.133+0000 7f5e71600640 1 --2- 172.21.15.27:0/3044257481 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5e4c066ad0 0x7f5e4c068f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:59.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.133+0000 7f5e71600640 1 -- 172.21.15.27:0/3044257481 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e5c083ea0 con 0x7f5e740b4300 2024-01-15T18:06:59.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.133+0000 7f5e79800640 1 --2- 172.21.15.27:0/3044257481 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5e4c066ad0 0x7f5e4c068f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:59.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.137+0000 7f5e79800640 1 --2- 172.21.15.27:0/3044257481 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5e4c066ad0 0x7f5e4c068f90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5e741f7cf0 tx=0x7f5e64009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:59.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.137+0000 7f5e71600640 1 -- 172.21.15.27:0/3044257481 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5e5c051080 con 0x7f5e740b4300 2024-01-15T18:06:59.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.457+0000 7f5e7c200640 1 -- 172.21.15.27:0/3044257481 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e740cf420 con 0x7f5e4c066ad0 2024-01-15T18:06:59.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.469+0000 7f5e71600640 1 -- 172.21.15.27:0/3044257481 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f5e64011e90 con 0x7f5e4c066ad0 2024-01-15T18:06:59.475 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:06:59.476 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:06:59.476 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:06:59.476 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:06:59.476 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:06:59.476 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:06:59.476 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:06:59.476 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:06:59.476 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:06:59.476 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 66.66666666666666, 2024-01-15T18:06:59.477 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T18:06:59.477 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "2/3", 2024-01-15T18:06:59.477 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:06:59.477 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:06:59.477 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:59.477 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:06:59.477 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:06:59.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.469+0000 7f5e7c200640 1 -- 172.21.15.27:0/3044257481 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5e4c066ad0 msgr2=0x7f5e4c068f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:59.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.469+0000 7f5e7c200640 1 --2- 172.21.15.27:0/3044257481 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5e4c066ad0 0x7f5e4c068f90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5e741f7cf0 tx=0x7f5e64009190 comp rx=0 tx=0).stop 2024-01-15T18:06:59.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.469+0000 7f5e7c200640 1 -- 172.21.15.27:0/3044257481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e740b4300 msgr2=0x7f5e74155010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:59.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.469+0000 7f5e7c200640 1 --2- 172.21.15.27:0/3044257481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e740b4300 0x7f5e74155010 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c00c500 tx=0x7f5e5c00c9c0 comp rx=0 tx=0).stop 2024-01-15T18:06:59.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.469+0000 7f5e7c200640 1 -- 172.21.15.27:0/3044257481 shutdown_connections 2024-01-15T18:06:59.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.469+0000 7f5e7c200640 1 --2- 172.21.15.27:0/3044257481 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5e4c066ad0 0x7f5e4c068f90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:59.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.469+0000 7f5e7c200640 1 --2- 172.21.15.27:0/3044257481 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5e7415be40 0x7f5e7406bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:59.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.469+0000 7f5e7c200640 1 --2- 172.21.15.27:0/3044257481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e740b4300 0x7f5e74155010 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:59.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.469+0000 7f5e7c200640 1 --2- 172.21.15.27:0/3044257481 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5e74071c50 0x7f5e7406b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:59.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.469+0000 7f5e7c200640 1 -- 172.21.15.27:0/3044257481 >> 172.21.15.27:0/3044257481 conn(0x7f5e74099e10 msgr2=0x7f5e7415dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:59.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.473+0000 7f5e7c200640 1 -- 172.21.15.27:0/3044257481 shutdown_connections 2024-01-15T18:06:59.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:06:59.473+0000 7f5e7c200640 1 -- 172.21.15.27:0/3044257481 wait complete. 2024-01-15T18:06:59.515 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:07:00.517 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:07:00.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.653+0000 7f43dac00640 1 Processor -- start 2024-01-15T18:07:00.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.653+0000 7f43dac00640 1 -- start start 2024-01-15T18:07:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.657+0000 7f43dac00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f43d4158900 0x7f43d415f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.657+0000 7f43dac00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f43d415bf00 0x7f43d40cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.657+0000 7f43dac00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f43d40cf110 0x7f43d40701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.657+0000 7f43dac00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43d40b1cb0 con 0x7f43d415bf00 2024-01-15T18:07:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.657+0000 7f43dac00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43d40cf510 con 0x7f43d40cf110 2024-01-15T18:07:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.657+0000 7f43dac00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d40b1f00 con 0x7f43d4158900 2024-01-15T18:07:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.657+0000 7f43d3400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f43d415bf00 0x7f43d40cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.657+0000 7f43d3400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f43d415bf00 0x7f43d40cebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56022/0 (socket says 172.21.15.27:56022) 2024-01-15T18:07:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.657+0000 7f43d3400640 1 -- 172.21.15.27:0/1812930053 learned_addr learned my addr 172.21.15.27:0/1812930053 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:00.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.657+0000 7f43d3400640 1 -- 172.21.15.27:0/1812930053 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f43d4158900 msgr2=0x7f43d415f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:00.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.657+0000 7f43d3400640 1 --2- 172.21.15.27:0/1812930053 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f43d4158900 0x7f43d415f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:00.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.657+0000 7f43d3400640 1 -- 172.21.15.27:0/1812930053 >> v2:172.21.15.93:3300/0 conn(0x7f43d40cf110 msgr2=0x7f43d40701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:00.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.657+0000 7f43d8c00640 1 --2- 172.21.15.27:0/1812930053 >> v2:172.21.15.93:3300/0 conn(0x7f43d40cf110 0x7f43d40701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:00.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.657+0000 7f43d3400640 1 --2- 172.21.15.27:0/1812930053 >> v2:172.21.15.93:3300/0 conn(0x7f43d40cf110 0x7f43d40701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:00.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.657+0000 7f43d3400640 1 -- 172.21.15.27:0/1812930053 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d40b21e0 con 0x7f43d415bf00 2024-01-15T18:07:00.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.657+0000 7f43d8c00640 1 --2- 172.21.15.27:0/1812930053 >> v2:172.21.15.93:3300/0 conn(0x7f43d40cf110 0x7f43d40701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:07:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.661+0000 7f43d3400640 1 --2- 172.21.15.27:0/1812930053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d415bf00 0x7f43d40cebd0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f43d4072050 tx=0x7f43bc01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=192df66f4c64a636 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.661+0000 7f43d2a00640 1 -- 172.21.15.27:0/1812930053 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d4070f30 con 0x7f43d415bf00 2024-01-15T18:07:00.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.661+0000 7f43d2a00640 1 -- 172.21.15.27:0/1812930053 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43bc0ac410 con 0x7f43d415bf00 2024-01-15T18:07:00.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.661+0000 7f43d2a00640 1 -- 172.21.15.27:0/1812930053 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43bc0ac7a0 con 0x7f43d415bf00 2024-01-15T18:07:00.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.661+0000 7f43dac00640 1 -- 172.21.15.27:0/1812930053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d415bf00 msgr2=0x7f43d40cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:00.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.661+0000 7f43dac00640 1 --2- 172.21.15.27:0/1812930053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d415bf00 0x7f43d40cebd0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f43d4072050 tx=0x7f43bc01c500 comp rx=0 tx=0).stop 2024-01-15T18:07:00.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.661+0000 7f43dac00640 1 -- 172.21.15.27:0/1812930053 shutdown_connections 2024-01-15T18:07:00.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.661+0000 7f43dac00640 1 --2- 172.21.15.27:0/1812930053 >> v2:172.21.15.93:3300/0 conn(0x7f43d40cf110 0x7f43d40701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:00.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.661+0000 7f43dac00640 1 --2- 172.21.15.27:0/1812930053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d415bf00 0x7f43d40cebd0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:00.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.661+0000 7f43dac00640 1 --2- 172.21.15.27:0/1812930053 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f43d4158900 0x7f43d415f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:00.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.661+0000 7f43dac00640 1 -- 172.21.15.27:0/1812930053 >> 172.21.15.27:0/1812930053 conn(0x7f43d40cb700 msgr2=0x7f43d40cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:00.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.661+0000 7f43dac00640 1 -- 172.21.15.27:0/1812930053 shutdown_connections 2024-01-15T18:07:00.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.661+0000 7f43dac00640 1 -- 172.21.15.27:0/1812930053 wait complete. 2024-01-15T18:07:00.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.661+0000 7f43dac00640 1 Processor -- start 2024-01-15T18:07:00.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.661+0000 7f43dac00640 1 -- start start 2024-01-15T18:07:00.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.665+0000 7f43dac00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f43d40cf110 0x7f43d415a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:00.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.665+0000 7f43dac00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f43d4158900 0x7f43d414faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:00.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.665+0000 7f43dac00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d415bf00 0x7f43d41572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:00.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.665+0000 7f43dac00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d414f0a0 con 0x7f43d415bf00 2024-01-15T18:07:00.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.665+0000 7f43dac00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d414d790 con 0x7f43d40cf110 2024-01-15T18:07:00.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.665+0000 7f43dac00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d4157e10 con 0x7f43d4158900 2024-01-15T18:07:00.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.665+0000 7f43d8c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d415bf00 0x7f43d41572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:00.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.665+0000 7f43d8c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d415bf00 0x7f43d41572a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56038/0 (socket says 172.21.15.27:56038) 2024-01-15T18:07:00.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.665+0000 7f43d8c00640 1 -- 172.21.15.27:0/3178369929 learned_addr learned my addr 172.21.15.27:0/3178369929 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:00.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.665+0000 7f43d3e00640 1 --2- 172.21.15.27:0/3178369929 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f43d40cf110 0x7f43d415a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:00.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.665+0000 7f43d3400640 1 --2- 172.21.15.27:0/3178369929 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f43d4158900 0x7f43d414faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:00.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.665+0000 7f43d8c00640 1 -- 172.21.15.27:0/3178369929 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f43d4158900 msgr2=0x7f43d414faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:00.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.665+0000 7f43d8c00640 1 --2- 172.21.15.27:0/3178369929 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f43d4158900 0x7f43d414faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:00.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.665+0000 7f43d8c00640 1 -- 172.21.15.27:0/3178369929 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f43d40cf110 msgr2=0x7f43d415a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:00.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.665+0000 7f43d8c00640 1 --2- 172.21.15.27:0/3178369929 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f43d40cf110 0x7f43d415a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:00.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.665+0000 7f43d8c00640 1 -- 172.21.15.27:0/3178369929 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43bc0094d0 con 0x7f43d415bf00 2024-01-15T18:07:00.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.665+0000 7f43d8c00640 1 --2- 172.21.15.27:0/3178369929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d415bf00 0x7f43d41572a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00cef0 tx=0x7f43cc00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:00.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.665+0000 7f43cbe00640 1 -- 172.21.15.27:0/3178369929 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43cc00d840 con 0x7f43d415bf00 2024-01-15T18:07:00.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.665+0000 7f43cbe00640 1 -- 172.21.15.27:0/3178369929 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43cc00d9c0 con 0x7f43d415bf00 2024-01-15T18:07:00.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.665+0000 7f43cbe00640 1 -- 172.21.15.27:0/3178369929 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43cc00dd70 con 0x7f43d415bf00 2024-01-15T18:07:00.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.665+0000 7f43dac00640 1 -- 172.21.15.27:0/3178369929 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d41580f0 con 0x7f43d415bf00 2024-01-15T18:07:00.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.665+0000 7f43dac00640 1 -- 172.21.15.27:0/3178369929 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d414f5d0 con 0x7f43d415bf00 2024-01-15T18:07:00.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.665+0000 7f43dac00640 1 -- 172.21.15.27:0/3178369929 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d414f7c0 con 0x7f43d415bf00 2024-01-15T18:07:00.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.669+0000 7f43cbe00640 1 -- 172.21.15.27:0/3178369929 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) v1 ==== 86313+0+0 (secure 0 0 0) 0x7f43cc01e3b0 con 0x7f43d415bf00 2024-01-15T18:07:00.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.669+0000 7f43cbe00640 1 --2- 172.21.15.27:0/3178369929 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f43ac066aa0 0x7f43ac068f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:00.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.669+0000 7f43cbe00640 1 -- 172.21.15.27:0/3178369929 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f43cc0850a0 con 0x7f43d415bf00 2024-01-15T18:07:00.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.669+0000 7f43d3e00640 1 --2- 172.21.15.27:0/3178369929 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f43ac066aa0 0x7f43ac068f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:00.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.669+0000 7f43d3e00640 1 --2- 172.21.15.27:0/3178369929 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f43ac066aa0 0x7f43ac068f60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f43c4009990 tx=0x7f43c4009110 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:00.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.673+0000 7f43cbe00640 1 -- 172.21.15.27:0/3178369929 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f43cc052280 con 0x7f43d415bf00 2024-01-15T18:07:00.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:00.865+0000 7f43dac00640 1 -- 172.21.15.27:0/3178369929 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f43d415c380 con 0x7f43ac066aa0 2024-01-15T18:07:01.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.093+0000 7f43cbe00640 1 -- 172.21.15.27:0/3178369929 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f43c40095e0 con 0x7f43ac066aa0 2024-01-15T18:07:01.102 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:07:01.102 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:07:01.102 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "complete" 2024-01-15T18:07:01.102 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:01.102 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:07:01.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.097+0000 7f43dac00640 1 -- 172.21.15.27:0/3178369929 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f43ac066aa0 msgr2=0x7f43ac068f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:01.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.097+0000 7f43dac00640 1 --2- 172.21.15.27:0/3178369929 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f43ac066aa0 0x7f43ac068f60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f43c4009990 tx=0x7f43c4009110 comp rx=0 tx=0).stop 2024-01-15T18:07:01.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.097+0000 7f43dac00640 1 -- 172.21.15.27:0/3178369929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d415bf00 msgr2=0x7f43d41572a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:01.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.097+0000 7f43dac00640 1 --2- 172.21.15.27:0/3178369929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d415bf00 0x7f43d41572a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00cef0 tx=0x7f43cc00a710 comp rx=0 tx=0).stop 2024-01-15T18:07:01.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.097+0000 7f43dac00640 1 -- 172.21.15.27:0/3178369929 shutdown_connections 2024-01-15T18:07:01.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.097+0000 7f43dac00640 1 --2- 172.21.15.27:0/3178369929 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f43ac066aa0 0x7f43ac068f60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:01.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.097+0000 7f43dac00640 1 --2- 172.21.15.27:0/3178369929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f43d415bf00 0x7f43d41572a0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:01.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.097+0000 7f43dac00640 1 --2- 172.21.15.27:0/3178369929 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f43d4158900 0x7f43d414faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:01.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.097+0000 7f43dac00640 1 --2- 172.21.15.27:0/3178369929 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f43d40cf110 0x7f43d415a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:01.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.097+0000 7f43dac00640 1 -- 172.21.15.27:0/3178369929 >> 172.21.15.27:0/3178369929 conn(0x7f43d40cb700 msgr2=0x7f43d40b4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:01.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.097+0000 7f43dac00640 1 -- 172.21.15.27:0/3178369929 shutdown_connections 2024-01-15T18:07:01.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.097+0000 7f43dac00640 1 -- 172.21.15.27:0/3178369929 wait complete. 2024-01-15T18:07:01.125 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:07:01.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.261+0000 7f50f2200640 1 Processor -- start 2024-01-15T18:07:01.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.261+0000 7f50f2200640 1 -- start start 2024-01-15T18:07:01.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.261+0000 7f50f2200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f50ec0cce90 0x7f50ec0cd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:01.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.261+0000 7f50f2200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50ec0cf550 con 0x7f50ec0d0310 2024-01-15T18:07:01.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.261+0000 7f50f2200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50ec0cf830 con 0x7f50ec06b810 2024-01-15T18:07:01.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.261+0000 7f50f2200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec0cfb10 con 0x7f50ec0cce90 2024-01-15T18:07:01.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.261+0000 7f50ebe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f50ec0d0310 0x7f50ec15d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36960/0 (socket says 172.21.15.27:36960) 2024-01-15T18:07:01.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.261+0000 7f50ebe00640 1 -- 172.21.15.27:0/307589372 learned_addr learned my addr 172.21.15.27:0/307589372 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:01.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.261+0000 7f50eb400640 1 --2- 172.21.15.27:0/307589372 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f50ec0cce90 0x7f50ec0cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:01.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.261+0000 7f50ea000640 1 -- 172.21.15.27:0/307589372 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412383565 0 0) 0x7f50ec0cf550 con 0x7f50ec0d0310 2024-01-15T18:07:01.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.261+0000 7f50ea000640 1 -- 172.21.15.27:0/307589372 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50cc003530 con 0x7f50ec0d0310 2024-01-15T18:07:01.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.261+0000 7f50eb400640 1 -- 172.21.15.27:0/307589372 >> v1:172.21.15.93:6789/0 conn(0x7f50ec06b810 legacy=0x7f50ec15cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:01.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.261+0000 7f50ea000640 1 -- 172.21.15.27:0/307589372 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97034253 0 0) 0x7f50ec0cf830 con 0x7f50ec06b810 2024-01-15T18:07:01.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.261+0000 7f50eb400640 1 -- 172.21.15.27:0/307589372 >> v1:172.21.15.27:6789/0 conn(0x7f50ec0d0310 legacy=0x7f50ec15d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:01.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.261+0000 7f50eb400640 1 -- 172.21.15.27:0/307589372 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ec0cfdf0 con 0x7f50ec0cce90 2024-01-15T18:07:01.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.265+0000 7f50eb400640 1 --2- 172.21.15.27:0/307589372 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f50ec0cce90 0x7f50ec0cd710 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f50dc0097f0 tx=0x7f50dc01c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=414d3fc367f680f2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:01.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.265+0000 7f50ea000640 1 -- 172.21.15.27:0/307589372 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc0aa640 con 0x7f50ec0cce90 2024-01-15T18:07:01.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.265+0000 7f50ea000640 1 -- 172.21.15.27:0/307589372 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50dc0aa7a0 con 0x7f50ec0cce90 2024-01-15T18:07:01.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.265+0000 7f50ea000640 1 -- 172.21.15.27:0/307589372 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc0aab30 con 0x7f50ec0cce90 2024-01-15T18:07:01.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.265+0000 7f50f2200640 1 -- 172.21.15.27:0/307589372 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f50ec0cce90 msgr2=0x7f50ec0cd710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:01.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.265+0000 7f50f2200640 1 --2- 172.21.15.27:0/307589372 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f50ec0cce90 0x7f50ec0cd710 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f50dc0097f0 tx=0x7f50dc01c4c0 comp rx=0 tx=0).stop 2024-01-15T18:07:01.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.265+0000 7f50f2200640 1 -- 172.21.15.27:0/307589372 shutdown_connections 2024-01-15T18:07:01.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.265+0000 7f50f2200640 1 --2- 172.21.15.27:0/307589372 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f50ec0cce90 0x7f50ec0cd710 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:01.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.265+0000 7f50f2200640 1 -- 172.21.15.27:0/307589372 >> 172.21.15.27:0/307589372 conn(0x7f50ec0d0e90 msgr2=0x7f50ec0d12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:01.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.265+0000 7f50f2200640 1 -- 172.21.15.27:0/307589372 shutdown_connections 2024-01-15T18:07:01.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.265+0000 7f50f2200640 1 -- 172.21.15.27:0/307589372 wait complete. 2024-01-15T18:07:01.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.269+0000 7f50f2200640 1 Processor -- start 2024-01-15T18:07:01.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.269+0000 7f50f2200640 1 -- start start 2024-01-15T18:07:01.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.269+0000 7f50f2200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f50ec0cce90 0x7f50ec0c4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:01.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.269+0000 7f50f2200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50ec0d0310 0x7f50ec0c50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:01.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.269+0000 7f50f2200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f50ec156370 0x7f50ec0c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:01.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.269+0000 7f50f2200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec13df90 con 0x7f50ec0d0310 2024-01-15T18:07:01.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.269+0000 7f50f2200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec15cf80 con 0x7f50ec0cce90 2024-01-15T18:07:01.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.269+0000 7f50f2200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec158420 con 0x7f50ec156370 2024-01-15T18:07:01.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.269+0000 7f50eaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50ec0d0310 0x7f50ec0c50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:01.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.269+0000 7f50ebe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f50ec156370 0x7f50ec0c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:01.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.269+0000 7f50eb400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f50ec0cce90 0x7f50ec0c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:01.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.269+0000 7f50eaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50ec0d0310 0x7f50ec0c50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56040/0 (socket says 172.21.15.27:56040) 2024-01-15T18:07:01.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.269+0000 7f50eaa00640 1 -- 172.21.15.27:0/1146741793 learned_addr learned my addr 172.21.15.27:0/1146741793 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:01.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.269+0000 7f50eaa00640 1 -- 172.21.15.27:0/1146741793 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f50ec156370 msgr2=0x7f50ec0c55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:01.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.269+0000 7f50eaa00640 1 --2- 172.21.15.27:0/1146741793 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f50ec156370 0x7f50ec0c55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:01.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.269+0000 7f50eaa00640 1 -- 172.21.15.27:0/1146741793 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f50ec0cce90 msgr2=0x7f50ec0c4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:01.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.269+0000 7f50eaa00640 1 --2- 172.21.15.27:0/1146741793 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f50ec0cce90 0x7f50ec0c4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:01.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.269+0000 7f50eaa00640 1 -- 172.21.15.27:0/1146741793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50dc009420 con 0x7f50ec0d0310 2024-01-15T18:07:01.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.269+0000 7f50eb400640 1 --2- 172.21.15.27:0/1146741793 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f50ec0cce90 0x7f50ec0c4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:07:01.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.269+0000 7f50eaa00640 1 --2- 172.21.15.27:0/1146741793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50ec0d0310 0x7f50ec0c50b0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f50d4001ec0 tx=0x7f50d400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:01.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.269+0000 7f50e3400640 1 -- 172.21.15.27:0/1146741793 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d40054e0 con 0x7f50ec0d0310 2024-01-15T18:07:01.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.269+0000 7f50e3400640 1 -- 172.21.15.27:0/1146741793 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d4003600 con 0x7f50ec0d0310 2024-01-15T18:07:01.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.269+0000 7f50e3400640 1 -- 172.21.15.27:0/1146741793 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d40039b0 con 0x7f50ec0d0310 2024-01-15T18:07:01.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.269+0000 7f50f2200640 1 -- 172.21.15.27:0/1146741793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50ec15cca0 con 0x7f50ec0d0310 2024-01-15T18:07:01.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.273+0000 7f50f2200640 1 -- 172.21.15.27:0/1146741793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50ec1580b0 con 0x7f50ec0d0310 2024-01-15T18:07:01.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.273+0000 7f50f2200640 1 -- 172.21.15.27:0/1146741793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50ec157c10 con 0x7f50ec0d0310 2024-01-15T18:07:01.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.273+0000 7f50e3400640 1 -- 172.21.15.27:0/1146741793 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) v1 ==== 86313+0+0 (secure 0 0 0) 0x7f50d4003b10 con 0x7f50ec0d0310 2024-01-15T18:07:01.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.273+0000 7f50e3400640 1 --2- 172.21.15.27:0/1146741793 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f50c4066aa0 0x7f50c4068f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:01.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.273+0000 7f50e3400640 1 -- 172.21.15.27:0/1146741793 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50d4083c90 con 0x7f50ec0d0310 2024-01-15T18:07:01.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.273+0000 7f50eb400640 1 --2- 172.21.15.27:0/1146741793 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f50c4066aa0 0x7f50c4068f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:01.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.273+0000 7f50eb400640 1 --2- 172.21.15.27:0/1146741793 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f50c4066aa0 0x7f50c4068f60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f50dc0ae000 tx=0x7f50dc0a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:01.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.277+0000 7f50e3400640 1 -- 172.21.15.27:0/1146741793 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f50d4050d10 con 0x7f50ec0d0310 2024-01-15T18:07:01.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.457+0000 7f50f2200640 1 -- 172.21.15.27:0/1146741793 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f50ec0ce060 con 0x7f50c4066aa0 2024-01-15T18:07:01.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.457+0000 7f50e3400640 1 -- 172.21.15.27:0/1146741793 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f50dc006e20 con 0x7f50c4066aa0 2024-01-15T18:07:01.466 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:07:01.466 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:07:01.466 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-15T18:07:01.467 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:01.467 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:07:01.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.461+0000 7f50f2200640 1 -- 172.21.15.27:0/1146741793 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f50c4066aa0 msgr2=0x7f50c4068f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:01.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.461+0000 7f50f2200640 1 --2- 172.21.15.27:0/1146741793 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f50c4066aa0 0x7f50c4068f60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f50dc0ae000 tx=0x7f50dc0a7000 comp rx=0 tx=0).stop 2024-01-15T18:07:01.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.461+0000 7f50f2200640 1 -- 172.21.15.27:0/1146741793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50ec0d0310 msgr2=0x7f50ec0c50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:01.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.461+0000 7f50f2200640 1 --2- 172.21.15.27:0/1146741793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50ec0d0310 0x7f50ec0c50b0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f50d4001ec0 tx=0x7f50d400c8e0 comp rx=0 tx=0).stop 2024-01-15T18:07:01.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.461+0000 7f50f2200640 1 -- 172.21.15.27:0/1146741793 shutdown_connections 2024-01-15T18:07:01.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.461+0000 7f50f2200640 1 --2- 172.21.15.27:0/1146741793 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f50c4066aa0 0x7f50c4068f60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:01.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.461+0000 7f50f2200640 1 --2- 172.21.15.27:0/1146741793 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f50ec156370 0x7f50ec0c55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:01.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.461+0000 7f50f2200640 1 --2- 172.21.15.27:0/1146741793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50ec0d0310 0x7f50ec0c50b0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:01.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.461+0000 7f50f2200640 1 --2- 172.21.15.27:0/1146741793 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f50ec0cce90 0x7f50ec0c4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:01.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.461+0000 7f50f2200640 1 -- 172.21.15.27:0/1146741793 >> 172.21.15.27:0/1146741793 conn(0x7f50ec0d0e90 msgr2=0x7f50ec06f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:01.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.461+0000 7f50f2200640 1 -- 172.21.15.27:0/1146741793 shutdown_connections 2024-01-15T18:07:01.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.461+0000 7f50f2200640 1 -- 172.21.15.27:0/1146741793 wait complete. 2024-01-15T18:07:01.489 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-15T18:07:01.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.625+0000 7f1c22000640 1 Processor -- start 2024-01-15T18:07:01.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.625+0000 7f1c22000640 1 -- start start 2024-01-15T18:07:01.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.625+0000 7f1c22000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1c1c15ad20 0x7f1c1c15b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:01.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.625+0000 7f1c22000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c1c15d900 0x7f1c1c15dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:01.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.625+0000 7f1c22000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c1c15a610 con 0x7f1c1c070ee0 2024-01-15T18:07:01.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.625+0000 7f1c22000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c1c15e3c0 con 0x7f1c1c15ad20 2024-01-15T18:07:01.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.625+0000 7f1c22000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1c15e6a0 con 0x7f1c1c15d900 2024-01-15T18:07:01.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.625+0000 7f1c1b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1c1c15ad20 0x7f1c1c15b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:01.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.625+0000 7f1c1be00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1c1c070ee0 0x7f1c1c06d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36966/0 (socket says 172.21.15.27:36966) 2024-01-15T18:07:01.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.625+0000 7f1c1be00640 1 -- 172.21.15.27:0/2021190222 learned_addr learned my addr 172.21.15.27:0/2021190222 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:01.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.625+0000 7f1c1aa00640 1 --2- 172.21.15.27:0/2021190222 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c1c15d900 0x7f1c1c15dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:01.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.625+0000 7f1c1b400640 1 -- 172.21.15.27:0/2021190222 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c1c15d900 msgr2=0x7f1c1c15dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:01.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.625+0000 7f1c1b400640 1 --2- 172.21.15.27:0/2021190222 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c1c15d900 0x7f1c1c15dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:01.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.625+0000 7f1c1b400640 1 -- 172.21.15.27:0/2021190222 >> v1:172.21.15.27:6789/0 conn(0x7f1c1c070ee0 legacy=0x7f1c1c06d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:01.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.625+0000 7f1c1b400640 1 -- 172.21.15.27:0/2021190222 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c1c15b6e0 con 0x7f1c1c15ad20 2024-01-15T18:07:01.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.625+0000 7f1c1aa00640 1 --2- 172.21.15.27:0/2021190222 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c1c15d900 0x7f1c1c15dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:07:01.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.629+0000 7f1c1b400640 1 --2- 172.21.15.27:0/2021190222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1c15ad20 0x7f1c1c15b100 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0097c0 tx=0x7f1c0c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=636d7735c1a6844 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:01.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.629+0000 7f1c1a000640 1 -- 172.21.15.27:0/2021190222 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c0ab670 con 0x7f1c1c15ad20 2024-01-15T18:07:01.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.629+0000 7f1c1a000640 1 -- 172.21.15.27:0/2021190222 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0c0ab7d0 con 0x7f1c1c15ad20 2024-01-15T18:07:01.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.629+0000 7f1c1a000640 1 -- 172.21.15.27:0/2021190222 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c0abb60 con 0x7f1c1c15ad20 2024-01-15T18:07:01.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.629+0000 7f1c22000640 1 -- 172.21.15.27:0/2021190222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1c15ad20 msgr2=0x7f1c1c15b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:01.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.629+0000 7f1c22000640 1 --2- 172.21.15.27:0/2021190222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1c15ad20 0x7f1c1c15b100 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0097c0 tx=0x7f1c0c01c450 comp rx=0 tx=0).stop 2024-01-15T18:07:01.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.629+0000 7f1c22000640 1 -- 172.21.15.27:0/2021190222 shutdown_connections 2024-01-15T18:07:01.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.629+0000 7f1c22000640 1 --2- 172.21.15.27:0/2021190222 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c1c15d900 0x7f1c1c15dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:01.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.629+0000 7f1c22000640 1 --2- 172.21.15.27:0/2021190222 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1c15ad20 0x7f1c1c15b100 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:01.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.629+0000 7f1c22000640 1 -- 172.21.15.27:0/2021190222 >> 172.21.15.27:0/2021190222 conn(0x7f1c1c0cb700 msgr2=0x7f1c1c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:01.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.629+0000 7f1c22000640 1 -- 172.21.15.27:0/2021190222 shutdown_connections 2024-01-15T18:07:01.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.629+0000 7f1c22000640 1 -- 172.21.15.27:0/2021190222 wait complete. 2024-01-15T18:07:01.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.633+0000 7f1c22000640 1 Processor -- start 2024-01-15T18:07:01.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.633+0000 7f1c22000640 1 -- start start 2024-01-15T18:07:01.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.633+0000 7f1c22000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1c070ee0 0x7f1c1c152cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:01.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.633+0000 7f1c22000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1c1c15ad20 0x7f1c1c1563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:01.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.633+0000 7f1c22000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c1c15d900 0x7f1c1c0b3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:01.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.633+0000 7f1c22000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1c0c3cc0 con 0x7f1c1c070ee0 2024-01-15T18:07:01.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.633+0000 7f1c22000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1c0c3fa0 con 0x7f1c1c15ad20 2024-01-15T18:07:01.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.633+0000 7f1c22000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1c0c4280 con 0x7f1c1c15d900 2024-01-15T18:07:01.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.633+0000 7f1c1b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1c070ee0 0x7f1c1c152cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:01.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.633+0000 7f1c1b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1c070ee0 0x7f1c1c152cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56068/0 (socket says 172.21.15.27:56068) 2024-01-15T18:07:01.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.633+0000 7f1c1b400640 1 -- 172.21.15.27:0/320679803 learned_addr learned my addr 172.21.15.27:0/320679803 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:01.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.633+0000 7f1c1aa00640 1 --2- 172.21.15.27:0/320679803 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1c1c15ad20 0x7f1c1c1563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:01.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.633+0000 7f1c1be00640 1 --2- 172.21.15.27:0/320679803 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c1c15d900 0x7f1c1c0b3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:01.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.633+0000 7f1c1b400640 1 -- 172.21.15.27:0/320679803 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c1c15d900 msgr2=0x7f1c1c0b3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:01.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.633+0000 7f1c1b400640 1 --2- 172.21.15.27:0/320679803 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c1c15d900 0x7f1c1c0b3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:01.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.633+0000 7f1c1b400640 1 -- 172.21.15.27:0/320679803 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1c1c15ad20 msgr2=0x7f1c1c1563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:01.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.633+0000 7f1c1b400640 1 --2- 172.21.15.27:0/320679803 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1c1c15ad20 0x7f1c1c1563a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:01.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.633+0000 7f1c1b400640 1 -- 172.21.15.27:0/320679803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0c009420 con 0x7f1c1c070ee0 2024-01-15T18:07:01.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.633+0000 7f1c1b400640 1 --2- 172.21.15.27:0/320679803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1c070ee0 0x7f1c1c152cc0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0ad040 tx=0x7f1c0c01ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:01.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.633+0000 7f1c13400640 1 -- 172.21.15.27:0/320679803 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c0aa8e0 con 0x7f1c1c070ee0 2024-01-15T18:07:01.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.633+0000 7f1c13400640 1 -- 172.21.15.27:0/320679803 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0c0aaa40 con 0x7f1c1c070ee0 2024-01-15T18:07:01.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.633+0000 7f1c22000640 1 -- 172.21.15.27:0/320679803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c1c0c4560 con 0x7f1c1c070ee0 2024-01-15T18:07:01.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.633+0000 7f1c13400640 1 -- 172.21.15.27:0/320679803 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c0aadf0 con 0x7f1c1c070ee0 2024-01-15T18:07:01.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.633+0000 7f1c22000640 1 -- 172.21.15.27:0/320679803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c1c153200 con 0x7f1c1c070ee0 2024-01-15T18:07:01.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.637+0000 7f1c22000640 1 -- 172.21.15.27:0/320679803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c1c04e850 con 0x7f1c1c070ee0 2024-01-15T18:07:01.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.637+0000 7f1c13400640 1 -- 172.21.15.27:0/320679803 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) v1 ==== 86313+0+0 (secure 0 0 0) 0x7f1c0c0bf410 con 0x7f1c1c070ee0 2024-01-15T18:07:01.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.637+0000 7f1c13400640 1 --2- 172.21.15.27:0/320679803 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1bf4066aa0 0x7f1bf4068f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:01.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.637+0000 7f1c13400640 1 -- 172.21.15.27:0/320679803 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1c0c11f7c0 con 0x7f1c1c070ee0 2024-01-15T18:07:01.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.637+0000 7f1c1aa00640 1 --2- 172.21.15.27:0/320679803 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1bf4066aa0 0x7f1bf4068f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:01.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.637+0000 7f1c1aa00640 1 --2- 172.21.15.27:0/320679803 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1bf4066aa0 0x7f1bf4068f60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c0c51a0 tx=0x7f1c04008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:01.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.641+0000 7f1c13400640 1 -- 172.21.15.27:0/320679803 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1c0c0ec8f0 con 0x7f1c1c070ee0 2024-01-15T18:07:01.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.821+0000 7f1c22000640 1 -- 172.21.15.27:0/320679803 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1c1c159980 con 0x7f1bf4066aa0 2024-01-15T18:07:01.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.829+0000 7f1c13400640 1 -- 172.21.15.27:0/320679803 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+79 (secure 0 0 0) 0x7f1c1c159980 con 0x7f1bf4066aa0 2024-01-15T18:07:01.835 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:07:01.835 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:07:01.835 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:07:01.835 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sv1" 2024-01-15T18:07:01.835 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:07:01.835 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:07:01.835 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ss1clone1" 2024-01-15T18:07:01.835 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:01.835 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:07:01.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.829+0000 7f1c22000640 1 -- 172.21.15.27:0/320679803 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1bf4066aa0 msgr2=0x7f1bf4068f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:01.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.829+0000 7f1c22000640 1 --2- 172.21.15.27:0/320679803 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1bf4066aa0 0x7f1bf4068f60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c0c51a0 tx=0x7f1c04008040 comp rx=0 tx=0).stop 2024-01-15T18:07:01.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.829+0000 7f1c22000640 1 -- 172.21.15.27:0/320679803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1c070ee0 msgr2=0x7f1c1c152cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:01.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.829+0000 7f1c22000640 1 --2- 172.21.15.27:0/320679803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1c070ee0 0x7f1c1c152cc0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0ad040 tx=0x7f1c0c01ac90 comp rx=0 tx=0).stop 2024-01-15T18:07:01.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.829+0000 7f1c22000640 1 -- 172.21.15.27:0/320679803 shutdown_connections 2024-01-15T18:07:01.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.829+0000 7f1c22000640 1 --2- 172.21.15.27:0/320679803 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1bf4066aa0 0x7f1bf4068f60 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:01.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.829+0000 7f1c22000640 1 --2- 172.21.15.27:0/320679803 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c1c15d900 0x7f1c1c0b3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:01.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.829+0000 7f1c22000640 1 --2- 172.21.15.27:0/320679803 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1c1c15ad20 0x7f1c1c1563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:01.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.829+0000 7f1c22000640 1 --2- 172.21.15.27:0/320679803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c1c070ee0 0x7f1c1c152cc0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:01.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.829+0000 7f1c22000640 1 -- 172.21.15.27:0/320679803 >> 172.21.15.27:0/320679803 conn(0x7f1c1c0cb700 msgr2=0x7f1c1c15c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:01.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.829+0000 7f1c22000640 1 -- 172.21.15.27:0/320679803 shutdown_connections 2024-01-15T18:07:01.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.829+0000 7f1c22000640 1 -- 172.21.15.27:0/320679803 wait complete. 2024-01-15T18:07:01.858 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-15T18:07:02.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.993+0000 7ff037400640 1 Processor -- start 2024-01-15T18:07:02.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.993+0000 7ff037400640 1 -- start start 2024-01-15T18:07:02.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.997+0000 7ff037400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0301468d0 0x7ff0300ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:02.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.997+0000 7ff037400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff0300cd070 0x7ff03006c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:02.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.997+0000 7ff037400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0300ad920 con 0x7ff0300cef30 2024-01-15T18:07:02.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.997+0000 7ff037400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0300ac880 con 0x7ff0300cd070 2024-01-15T18:07:02.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.997+0000 7ff037400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0300acb60 con 0x7ff0301468d0 2024-01-15T18:07:02.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.997+0000 7ff034a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0301468d0 0x7ff0300ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:02.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.997+0000 7ff035400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff0300cd070 0x7ff03006c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:02.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.997+0000 7ff035400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff0300cd070 0x7ff03006c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:56156/0 (socket says 172.21.15.27:56156) 2024-01-15T18:07:02.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.997+0000 7ff035400640 1 -- 172.21.15.27:0/850674417 learned_addr learned my addr 172.21.15.27:0/850674417 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:02.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.997+0000 7ff034a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0301468d0 0x7ff0300ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:37816/0 (socket says 172.21.15.27:37816) 2024-01-15T18:07:02.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.997+0000 7ff035400640 1 -- 172.21.15.27:0/850674417 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0301468d0 msgr2=0x7ff0300ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:02.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.997+0000 7ff035400640 1 --2- 172.21.15.27:0/850674417 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0301468d0 0x7ff0300ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.997+0000 7ff035400640 1 -- 172.21.15.27:0/850674417 >> v1:172.21.15.93:6789/0 conn(0x7ff0300cef30 legacy=0x7ff0300cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:02.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.997+0000 7ff035400640 1 -- 172.21.15.27:0/850674417 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0300ace40 con 0x7ff0300cd070 2024-01-15T18:07:02.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:01.997+0000 7ff034a00640 1 --2- 172.21.15.27:0/850674417 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0301468d0 0x7ff0300ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:07:02.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.001+0000 7ff035400640 1 --2- 172.21.15.27:0/850674417 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0300cd070 0x7ff03006c940 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff028009e80 tx=0x7ff02801c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9ff499326752f3d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:02.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.001+0000 7ff02f400640 1 -- 172.21.15.27:0/850674417 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0280a57a0 con 0x7ff0300cd070 2024-01-15T18:07:02.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.001+0000 7ff02f400640 1 -- 172.21.15.27:0/850674417 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0280a5900 con 0x7ff0300cd070 2024-01-15T18:07:02.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.001+0000 7ff02f400640 1 -- 172.21.15.27:0/850674417 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0280a5c90 con 0x7ff0300cd070 2024-01-15T18:07:02.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.001+0000 7ff037400640 1 -- 172.21.15.27:0/850674417 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0300cd070 msgr2=0x7ff03006c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:02.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.001+0000 7ff037400640 1 --2- 172.21.15.27:0/850674417 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0300cd070 0x7ff03006c940 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff028009e80 tx=0x7ff02801c9f0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.001+0000 7ff037400640 1 -- 172.21.15.27:0/850674417 shutdown_connections 2024-01-15T18:07:02.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.001+0000 7ff037400640 1 --2- 172.21.15.27:0/850674417 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0300cd070 0x7ff03006c940 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.001+0000 7ff037400640 1 --2- 172.21.15.27:0/850674417 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0301468d0 0x7ff0300ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.001+0000 7ff037400640 1 -- 172.21.15.27:0/850674417 >> 172.21.15.27:0/850674417 conn(0x7ff0300cb700 msgr2=0x7ff0300cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:02.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.001+0000 7ff037400640 1 -- 172.21.15.27:0/850674417 shutdown_connections 2024-01-15T18:07:02.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.001+0000 7ff037400640 1 -- 172.21.15.27:0/850674417 wait complete. 2024-01-15T18:07:02.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.005+0000 7ff037400640 1 Processor -- start 2024-01-15T18:07:02.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.005+0000 7ff037400640 1 -- start start 2024-01-15T18:07:02.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.005+0000 7ff037400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0300cd070 0x7ff0300b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:02.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.005+0000 7ff037400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0300cef30 0x7ff030137e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:02.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.005+0000 7ff037400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0301468d0 0x7ff03015a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:02.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.005+0000 7ff037400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff030072090 con 0x7ff0300cef30 2024-01-15T18:07:02.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.005+0000 7ff037400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff030155560 con 0x7ff0300cd070 2024-01-15T18:07:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.005+0000 7ff037400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff030155280 con 0x7ff0301468d0 2024-01-15T18:07:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.005+0000 7ff02fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0300cef30 0x7ff030137e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.005+0000 7ff02fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0300cef30 0x7ff030137e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56072/0 (socket says 172.21.15.27:56072) 2024-01-15T18:07:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.005+0000 7ff02fe00640 1 -- 172.21.15.27:0/2924620499 learned_addr learned my addr 172.21.15.27:0/2924620499 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.005+0000 7ff035400640 1 --2- 172.21.15.27:0/2924620499 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0301468d0 0x7ff03015a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.005+0000 7ff034a00640 1 --2- 172.21.15.27:0/2924620499 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0300cd070 0x7ff0300b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.005+0000 7ff02fe00640 1 -- 172.21.15.27:0/2924620499 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0301468d0 msgr2=0x7ff03015a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.005+0000 7ff02fe00640 1 --2- 172.21.15.27:0/2924620499 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0301468d0 0x7ff03015a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.005+0000 7ff02fe00640 1 -- 172.21.15.27:0/2924620499 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0300cd070 msgr2=0x7ff0300b02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.005+0000 7ff02fe00640 1 --2- 172.21.15.27:0/2924620499 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0300cd070 0x7ff0300b02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.005+0000 7ff02fe00640 1 -- 172.21.15.27:0/2924620499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff020009420 con 0x7ff0300cef30 2024-01-15T18:07:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.005+0000 7ff02fe00640 1 --2- 172.21.15.27:0/2924620499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0300cef30 0x7ff030137e80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff01800c500 tx=0x7ff01800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.005+0000 7ff02cc00640 1 -- 172.21.15.27:0/2924620499 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff018005220 con 0x7ff0300cef30 2024-01-15T18:07:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.005+0000 7ff02cc00640 1 -- 172.21.15.27:0/2924620499 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff018005380 con 0x7ff0300cef30 2024-01-15T18:07:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.005+0000 7ff02cc00640 1 -- 172.21.15.27:0/2924620499 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff018007270 con 0x7ff0300cef30 2024-01-15T18:07:02.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.005+0000 7ff037400640 1 -- 172.21.15.27:0/2924620499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff028009b30 con 0x7ff0300cef30 2024-01-15T18:07:02.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.005+0000 7ff037400640 1 -- 172.21.15.27:0/2924620499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0300cf490 con 0x7ff0300cef30 2024-01-15T18:07:02.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.009+0000 7ff037400640 1 -- 172.21.15.27:0/2924620499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff03004e850 con 0x7ff0300cef30 2024-01-15T18:07:02.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.009+0000 7ff02cc00640 1 -- 172.21.15.27:0/2924620499 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) v1 ==== 86313+0+0 (secure 0 0 0) 0x7ff0180073d0 con 0x7ff0300cef30 2024-01-15T18:07:02.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.009+0000 7ff02cc00640 1 --2- 172.21.15.27:0/2924620499 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff008066aa0 0x7ff008068f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:02.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.009+0000 7ff02cc00640 1 -- 172.21.15.27:0/2924620499 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff018083880 con 0x7ff0300cef30 2024-01-15T18:07:02.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.009+0000 7ff034a00640 1 --2- 172.21.15.27:0/2924620499 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff008066aa0 0x7ff008068f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:02.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.009+0000 7ff034a00640 1 --2- 172.21.15.27:0/2924620499 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff008066aa0 0x7ff008068f60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff020009c50 tx=0x7ff020009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:02.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.013+0000 7ff02cc00640 1 -- 172.21.15.27:0/2924620499 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff018050910 con 0x7ff0300cef30 2024-01-15T18:07:02.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.197+0000 7ff037400640 1 -- 172.21.15.27:0/2924620499 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff030066f30 con 0x7ff008066aa0 2024-01-15T18:07:02.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.213+0000 7ff02cc00640 1 -- 172.21.15.27:0/2924620499 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7ff030066f30 con 0x7ff008066aa0 2024-01-15T18:07:02.219 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:07:02.219 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:07:02.219 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:07:02.220 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ss1" 2024-01-15T18:07:02.220 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:02.220 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:07:02.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.213+0000 7ff037400640 1 -- 172.21.15.27:0/2924620499 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff008066aa0 msgr2=0x7ff008068f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:02.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.213+0000 7ff037400640 1 --2- 172.21.15.27:0/2924620499 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff008066aa0 0x7ff008068f60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff020009c50 tx=0x7ff020009190 comp rx=0 tx=0).stop 2024-01-15T18:07:02.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.213+0000 7ff037400640 1 -- 172.21.15.27:0/2924620499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0300cef30 msgr2=0x7ff030137e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:02.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.213+0000 7ff037400640 1 --2- 172.21.15.27:0/2924620499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0300cef30 0x7ff030137e80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff01800c500 tx=0x7ff01800c9c0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.213+0000 7ff037400640 1 -- 172.21.15.27:0/2924620499 shutdown_connections 2024-01-15T18:07:02.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.213+0000 7ff037400640 1 --2- 172.21.15.27:0/2924620499 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff008066aa0 0x7ff008068f60 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.213+0000 7ff037400640 1 --2- 172.21.15.27:0/2924620499 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0301468d0 0x7ff03015a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.213+0000 7ff037400640 1 --2- 172.21.15.27:0/2924620499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0300cef30 0x7ff030137e80 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.213+0000 7ff037400640 1 --2- 172.21.15.27:0/2924620499 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0300cd070 0x7ff0300b02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.213+0000 7ff037400640 1 -- 172.21.15.27:0/2924620499 >> 172.21.15.27:0/2924620499 conn(0x7ff0300cb700 msgr2=0x7ff0301614f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:02.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.213+0000 7ff037400640 1 -- 172.21.15.27:0/2924620499 shutdown_connections 2024-01-15T18:07:02.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.213+0000 7ff037400640 1 -- 172.21.15.27:0/2924620499 wait complete. 2024-01-15T18:07:02.243 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:07:02.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.381+0000 7f75ec000640 1 Processor -- start 2024-01-15T18:07:02.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.381+0000 7f75ec000640 1 -- start start 2024-01-15T18:07:02.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.385+0000 7f75ec000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f75e415d3b0 0x7f75e40cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:02.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.385+0000 7f75ec000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f75e406b600 0x7f75e406bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:02.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.385+0000 7f75ec000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75e40b04b0 con 0x7f75e406c220 2024-01-15T18:07:02.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.385+0000 7f75ec000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75e406c6b0 con 0x7f75e415d3b0 2024-01-15T18:07:02.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.385+0000 7f75ec000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75e40d0160 con 0x7f75e406b600 2024-01-15T18:07:02.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.385+0000 7f75e9600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f75e415d3b0 0x7f75e40cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:02.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.385+0000 7f75ea000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f75e406c220 0x7f75e4160f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36968/0 (socket says 172.21.15.27:36968) 2024-01-15T18:07:02.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.385+0000 7f75ea000640 1 -- 172.21.15.27:0/868583109 learned_addr learned my addr 172.21.15.27:0/868583109 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:02.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.385+0000 7f75e8c00640 1 --2- 172.21.15.27:0/868583109 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f75e406b600 0x7f75e406bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:02.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.385+0000 7f75e9600640 1 -- 172.21.15.27:0/868583109 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f75e406b600 msgr2=0x7f75e406bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:02.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.385+0000 7f75e9600640 1 --2- 172.21.15.27:0/868583109 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f75e406b600 0x7f75e406bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.385+0000 7f75e9600640 1 -- 172.21.15.27:0/868583109 >> v1:172.21.15.27:6789/0 conn(0x7f75e406c220 legacy=0x7f75e4160f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:02.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.385+0000 7f75e9600640 1 -- 172.21.15.27:0/868583109 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e40b0720 con 0x7f75e415d3b0 2024-01-15T18:07:02.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.385+0000 7f75e3e00640 1 -- 172.21.15.27:0/868583109 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353418 0 0) 0x7f75e40b04b0 con 0x7f75e406c220 2024-01-15T18:07:02.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.389+0000 7f75e9600640 1 --2- 172.21.15.27:0/868583109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75e415d3b0 0x7f75e40cfc20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f75d40097f0 tx=0x7f75d401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=356886e1ed3d5d69 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:02.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.389+0000 7f75e3e00640 1 -- 172.21.15.27:0/868583109 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d40ab650 con 0x7f75e415d3b0 2024-01-15T18:07:02.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.389+0000 7f75e3e00640 1 -- 172.21.15.27:0/868583109 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d40ab7b0 con 0x7f75e415d3b0 2024-01-15T18:07:02.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.389+0000 7f75e3e00640 1 -- 172.21.15.27:0/868583109 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d40abb40 con 0x7f75e415d3b0 2024-01-15T18:07:02.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.389+0000 7f75ec000640 1 -- 172.21.15.27:0/868583109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75e415d3b0 msgr2=0x7f75e40cfc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:02.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.389+0000 7f75ec000640 1 --2- 172.21.15.27:0/868583109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75e415d3b0 0x7f75e40cfc20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f75d40097f0 tx=0x7f75d401c450 comp rx=0 tx=0).stop 2024-01-15T18:07:02.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.389+0000 7f75ec000640 1 -- 172.21.15.27:0/868583109 shutdown_connections 2024-01-15T18:07:02.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.389+0000 7f75ec000640 1 --2- 172.21.15.27:0/868583109 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f75e406b600 0x7f75e406bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.389+0000 7f75ec000640 1 --2- 172.21.15.27:0/868583109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75e415d3b0 0x7f75e40cfc20 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.389+0000 7f75ec000640 1 -- 172.21.15.27:0/868583109 >> 172.21.15.27:0/868583109 conn(0x7f75e40d0ed0 msgr2=0x7f75e40d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:02.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.389+0000 7f75ec000640 1 -- 172.21.15.27:0/868583109 shutdown_connections 2024-01-15T18:07:02.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.389+0000 7f75ec000640 1 -- 172.21.15.27:0/868583109 wait complete. 2024-01-15T18:07:02.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.389+0000 7f75ec000640 1 Processor -- start 2024-01-15T18:07:02.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.389+0000 7f75ec000640 1 -- start start 2024-01-15T18:07:02.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75ec000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f75e406b600 0x7f75e409e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:02.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75ec000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75e406c220 0x7f75e415e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:02.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75ec000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f75e415d3b0 0x7f75e41596f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:02.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75ec000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e4157260 con 0x7f75e406c220 2024-01-15T18:07:02.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75ec000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e4156f30 con 0x7f75e406b600 2024-01-15T18:07:02.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75ec000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75e409eb80 con 0x7f75e415d3b0 2024-01-15T18:07:02.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75e8c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75e406c220 0x7f75e415e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:02.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75e8c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75e406c220 0x7f75e415e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56094/0 (socket says 172.21.15.27:56094) 2024-01-15T18:07:02.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75e8c00640 1 -- 172.21.15.27:0/1325964047 learned_addr learned my addr 172.21.15.27:0/1325964047 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:02.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75ea000640 1 --2- 172.21.15.27:0/1325964047 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f75e415d3b0 0x7f75e41596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75e8c00640 1 -- 172.21.15.27:0/1325964047 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f75e415d3b0 msgr2=0x7f75e41596f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75e9600640 1 --2- 172.21.15.27:0/1325964047 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f75e406b600 0x7f75e409e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75e8c00640 1 --2- 172.21.15.27:0/1325964047 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f75e415d3b0 0x7f75e41596f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75e8c00640 1 -- 172.21.15.27:0/1325964047 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f75e406b600 msgr2=0x7f75e409e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75e8c00640 1 --2- 172.21.15.27:0/1325964047 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f75e406b600 0x7f75e409e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75e8c00640 1 -- 172.21.15.27:0/1325964047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d4009420 con 0x7f75e406c220 2024-01-15T18:07:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75ea000640 1 --2- 172.21.15.27:0/1325964047 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f75e415d3b0 0x7f75e41596f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:07:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75e9600640 1 --2- 172.21.15.27:0/1325964047 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f75e406b600 0x7f75e409e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:07:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75e8c00640 1 --2- 172.21.15.27:0/1325964047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75e406c220 0x7f75e415e410 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f75cc00d800 tx=0x7f75cc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75e1600640 1 -- 172.21.15.27:0/1325964047 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75cc00b6c0 con 0x7f75e406c220 2024-01-15T18:07:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75e1600640 1 -- 172.21.15.27:0/1325964047 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75cc00b820 con 0x7f75e406c220 2024-01-15T18:07:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75e1600640 1 -- 172.21.15.27:0/1325964047 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75cc00bb90 con 0x7f75e406c220 2024-01-15T18:07:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75ec000640 1 -- 172.21.15.27:0/1325964047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e415ee20 con 0x7f75e406c220 2024-01-15T18:07:02.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75ec000640 1 -- 172.21.15.27:0/1325964047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e415f3e0 con 0x7f75e406c220 2024-01-15T18:07:02.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75ec000640 1 -- 172.21.15.27:0/1325964047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75e406d550 con 0x7f75e406c220 2024-01-15T18:07:02.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.393+0000 7f75e1600640 1 -- 172.21.15.27:0/1325964047 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) v1 ==== 86313+0+0 (secure 0 0 0) 0x7f75cc00bcf0 con 0x7f75e406c220 2024-01-15T18:07:02.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.397+0000 7f75e1600640 1 --2- 172.21.15.27:0/1325964047 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f75bc066aa0 0x7f75bc068f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:02.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.397+0000 7f75e1600640 1 -- 172.21.15.27:0/1325964047 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75cc084ce0 con 0x7f75e406c220 2024-01-15T18:07:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.397+0000 7f75e9600640 1 --2- 172.21.15.27:0/1325964047 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f75bc066aa0 0x7f75bc068f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:02.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.397+0000 7f75e9600640 1 --2- 172.21.15.27:0/1325964047 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f75bc066aa0 0x7f75bc068f60 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f75d40af000 tx=0x7f75d4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:02.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.401+0000 7f75e1600640 1 -- 172.21.15.27:0/1325964047 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f75cc051ca0 con 0x7f75e406c220 2024-01-15T18:07:02.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.585+0000 7f75ec000640 1 -- 172.21.15.27:0/1325964047 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f75e40d0430 con 0x7f75bc066aa0 2024-01-15T18:07:02.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.613+0000 7f75e1600640 1 -- 172.21.15.27:0/1325964047 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f75d40bc0f0 con 0x7f75bc066aa0 2024-01-15T18:07:02.622 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:07:02.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.617+0000 7f75ec000640 1 -- 172.21.15.27:0/1325964047 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f75bc066aa0 msgr2=0x7f75bc068f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.617+0000 7f75ec000640 1 --2- 172.21.15.27:0/1325964047 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f75bc066aa0 0x7f75bc068f60 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f75d40af000 tx=0x7f75d4008040 comp rx=0 tx=0).stop 2024-01-15T18:07:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.617+0000 7f75ec000640 1 -- 172.21.15.27:0/1325964047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75e406c220 msgr2=0x7f75e415e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.617+0000 7f75ec000640 1 --2- 172.21.15.27:0/1325964047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75e406c220 0x7f75e415e410 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f75cc00d800 tx=0x7f75cc00dcc0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.617+0000 7f75ec000640 1 -- 172.21.15.27:0/1325964047 shutdown_connections 2024-01-15T18:07:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.617+0000 7f75ec000640 1 --2- 172.21.15.27:0/1325964047 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f75bc066aa0 0x7f75bc068f60 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.617+0000 7f75ec000640 1 --2- 172.21.15.27:0/1325964047 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f75e415d3b0 0x7f75e41596f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.617+0000 7f75ec000640 1 --2- 172.21.15.27:0/1325964047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75e406c220 0x7f75e415e410 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.617+0000 7f75ec000640 1 --2- 172.21.15.27:0/1325964047 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f75e406b600 0x7f75e409e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.617+0000 7f75ec000640 1 -- 172.21.15.27:0/1325964047 >> 172.21.15.27:0/1325964047 conn(0x7f75e40d0ed0 msgr2=0x7f75e40c77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.617+0000 7f75ec000640 1 -- 172.21.15.27:0/1325964047 shutdown_connections 2024-01-15T18:07:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.617+0000 7f75ec000640 1 -- 172.21.15.27:0/1325964047 wait complete. 2024-01-15T18:07:02.644 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-15T18:07:02.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.781+0000 7f8915c00640 1 Processor -- start 2024-01-15T18:07:02.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.781+0000 7f8915c00640 1 -- start start 2024-01-15T18:07:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.781+0000 7f8915c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f891015d680 0x7f89100cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.781+0000 7f8915c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89100ccf80 0x7f89100cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.781+0000 7f8915c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89100cb4d0 con 0x7f89100cfc30 2024-01-15T18:07:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.781+0000 7f8915c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89100d0120 con 0x7f891015d680 2024-01-15T18:07:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.781+0000 7f8915c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89100d0400 con 0x7f89100ccf80 2024-01-15T18:07:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.781+0000 7f890ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f891015d680 0x7f89100cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.781+0000 7f890e000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f89100cfc30 0x7f891006b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36976/0 (socket says 172.21.15.27:36976) 2024-01-15T18:07:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.781+0000 7f890e000640 1 -- 172.21.15.27:0/3955823205 learned_addr learned my addr 172.21.15.27:0/3955823205 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.781+0000 7f890f400640 1 --2- 172.21.15.27:0/3955823205 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89100ccf80 0x7f89100cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.781+0000 7f890ea00640 1 -- 172.21.15.27:0/3955823205 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89100ccf80 msgr2=0x7f89100cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.781+0000 7f890ea00640 1 --2- 172.21.15.27:0/3955823205 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89100ccf80 0x7f89100cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.781+0000 7f890ea00640 1 -- 172.21.15.27:0/3955823205 >> v1:172.21.15.27:6789/0 conn(0x7f89100cfc30 legacy=0x7f891006b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.781+0000 7f890ea00640 1 -- 172.21.15.27:0/3955823205 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89100cb770 con 0x7f891015d680 2024-01-15T18:07:02.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.785+0000 7f890ea00640 1 --2- 172.21.15.27:0/3955823205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f891015d680 0x7f89100cd700 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f89000097c0 tx=0x7f890001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdb433a30c876e23 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:02.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.785+0000 7f890d600640 1 -- 172.21.15.27:0/3955823205 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89100ce850 con 0x7f891015d680 2024-01-15T18:07:02.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.785+0000 7f890d600640 1 -- 172.21.15.27:0/3955823205 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89000ac640 con 0x7f891015d680 2024-01-15T18:07:02.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.785+0000 7f890d600640 1 -- 172.21.15.27:0/3955823205 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89000ac9d0 con 0x7f891015d680 2024-01-15T18:07:02.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.785+0000 7f8915c00640 1 -- 172.21.15.27:0/3955823205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f891015d680 msgr2=0x7f89100cd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:02.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.785+0000 7f8915c00640 1 --2- 172.21.15.27:0/3955823205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f891015d680 0x7f89100cd700 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f89000097c0 tx=0x7f890001c3c0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.785+0000 7f8915c00640 1 -- 172.21.15.27:0/3955823205 shutdown_connections 2024-01-15T18:07:02.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.785+0000 7f8915c00640 1 --2- 172.21.15.27:0/3955823205 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89100ccf80 0x7f89100cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.785+0000 7f8915c00640 1 --2- 172.21.15.27:0/3955823205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f891015d680 0x7f89100cd700 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.785+0000 7f8915c00640 1 -- 172.21.15.27:0/3955823205 >> 172.21.15.27:0/3955823205 conn(0x7f89100d0e80 msgr2=0x7f89100d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:02.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.785+0000 7f8915c00640 1 -- 172.21.15.27:0/3955823205 shutdown_connections 2024-01-15T18:07:02.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.785+0000 7f8915c00640 1 -- 172.21.15.27:0/3955823205 wait complete. 2024-01-15T18:07:02.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.789+0000 7f8915c00640 1 Processor -- start 2024-01-15T18:07:02.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.789+0000 7f8915c00640 1 -- start start 2024-01-15T18:07:02.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.789+0000 7f8915c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89100ccf80 0x7f8910151730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:02.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.789+0000 7f8915c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89100cfc30 0x7f8910151ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:02.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.789+0000 7f8915c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f891015d680 0x7f8910152410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:02.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.789+0000 7f8915c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8910153780 con 0x7f89100ccf80 2024-01-15T18:07:02.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.789+0000 7f8915c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8910152a90 con 0x7f891015d680 2024-01-15T18:07:02.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.789+0000 7f8915c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8910152d70 con 0x7f89100cfc30 2024-01-15T18:07:02.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.789+0000 7f890ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89100ccf80 0x7f8910151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:02.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.789+0000 7f890ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89100ccf80 0x7f8910151730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56114/0 (socket says 172.21.15.27:56114) 2024-01-15T18:07:02.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.789+0000 7f890ea00640 1 -- 172.21.15.27:0/2204465296 learned_addr learned my addr 172.21.15.27:0/2204465296 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:02.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.789+0000 7f890f400640 1 --2- 172.21.15.27:0/2204465296 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f891015d680 0x7f8910152410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:02.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.789+0000 7f890e000640 1 --2- 172.21.15.27:0/2204465296 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89100cfc30 0x7f8910151ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:02.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.789+0000 7f890ea00640 1 -- 172.21.15.27:0/2204465296 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89100cfc30 msgr2=0x7f8910151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:02.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.789+0000 7f890ea00640 1 --2- 172.21.15.27:0/2204465296 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89100cfc30 0x7f8910151ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.789+0000 7f890ea00640 1 -- 172.21.15.27:0/2204465296 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f891015d680 msgr2=0x7f8910152410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:02.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.789+0000 7f890ea00640 1 --2- 172.21.15.27:0/2204465296 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f891015d680 0x7f8910152410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:02.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.789+0000 7f890ea00640 1 -- 172.21.15.27:0/2204465296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8900009420 con 0x7f89100ccf80 2024-01-15T18:07:02.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.789+0000 7f890ea00640 1 --2- 172.21.15.27:0/2204465296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89100ccf80 0x7f8910151730 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f890001aa70 tx=0x7f890001ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:02.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.789+0000 7f8906a00640 1 -- 172.21.15.27:0/2204465296 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89000b7660 con 0x7f89100ccf80 2024-01-15T18:07:02.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.789+0000 7f8906a00640 1 -- 172.21.15.27:0/2204465296 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89000b77c0 con 0x7f89100ccf80 2024-01-15T18:07:02.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.789+0000 7f8906a00640 1 -- 172.21.15.27:0/2204465296 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89000b7b70 con 0x7f89100ccf80 2024-01-15T18:07:02.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.789+0000 7f8915c00640 1 -- 172.21.15.27:0/2204465296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8910153050 con 0x7f89100ccf80 2024-01-15T18:07:02.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.789+0000 7f8915c00640 1 -- 172.21.15.27:0/2204465296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8910151c70 con 0x7f89100ccf80 2024-01-15T18:07:02.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.793+0000 7f8915c00640 1 -- 172.21.15.27:0/2204465296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f891004e850 con 0x7f89100ccf80 2024-01-15T18:07:02.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.793+0000 7f8906a00640 1 -- 172.21.15.27:0/2204465296 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) v1 ==== 86313+0+0 (secure 0 0 0) 0x7f89000b3070 con 0x7f89100ccf80 2024-01-15T18:07:02.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.793+0000 7f8906a00640 1 --2- 172.21.15.27:0/2204465296 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88e8066aa0 0x7f88e8068f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:02.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.793+0000 7f8906a00640 1 -- 172.21.15.27:0/2204465296 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f890011f610 con 0x7f89100ccf80 2024-01-15T18:07:02.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.793+0000 7f890e000640 1 --2- 172.21.15.27:0/2204465296 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88e8066aa0 0x7f88e8068f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:02.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.793+0000 7f890e000640 1 --2- 172.21.15.27:0/2204465296 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88e8066aa0 0x7f88e8068f60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f88f8005890 tx=0x7f88f8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:02.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.797+0000 7f8906a00640 1 -- 172.21.15.27:0/2204465296 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f89000ec640 con 0x7f89100ccf80 2024-01-15T18:07:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.977+0000 7f8915c00640 1 -- 172.21.15.27:0/2204465296 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8910067060 con 0x7f88e8066aa0 2024-01-15T18:07:03.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.997+0000 7f8906a00640 1 -- 172.21.15.27:0/2204465296 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8910067060 con 0x7f88e8066aa0 2024-01-15T18:07:03.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.997+0000 7f8915c00640 1 -- 172.21.15.27:0/2204465296 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88e8066aa0 msgr2=0x7f88e8068f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:03.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.997+0000 7f8915c00640 1 --2- 172.21.15.27:0/2204465296 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88e8066aa0 0x7f88e8068f60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f88f8005890 tx=0x7f88f8009040 comp rx=0 tx=0).stop 2024-01-15T18:07:03.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.997+0000 7f8915c00640 1 -- 172.21.15.27:0/2204465296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89100ccf80 msgr2=0x7f8910151730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:03.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.997+0000 7f8915c00640 1 --2- 172.21.15.27:0/2204465296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89100ccf80 0x7f8910151730 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f890001aa70 tx=0x7f890001ac00 comp rx=0 tx=0).stop 2024-01-15T18:07:03.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.997+0000 7f8915c00640 1 -- 172.21.15.27:0/2204465296 shutdown_connections 2024-01-15T18:07:03.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.997+0000 7f8915c00640 1 --2- 172.21.15.27:0/2204465296 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88e8066aa0 0x7f88e8068f60 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.997+0000 7f8915c00640 1 --2- 172.21.15.27:0/2204465296 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f891015d680 0x7f8910152410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.997+0000 7f8915c00640 1 --2- 172.21.15.27:0/2204465296 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89100cfc30 0x7f8910151ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.997+0000 7f8915c00640 1 --2- 172.21.15.27:0/2204465296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89100ccf80 0x7f8910151730 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.997+0000 7f8915c00640 1 -- 172.21.15.27:0/2204465296 >> 172.21.15.27:0/2204465296 conn(0x7f89100d0e80 msgr2=0x7f89100ae980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:03.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.997+0000 7f8915c00640 1 -- 172.21.15.27:0/2204465296 shutdown_connections 2024-01-15T18:07:03.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:02.997+0000 7f8915c00640 1 -- 172.21.15.27:0/2204465296 wait complete. 2024-01-15T18:07:03.008 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:07:03.004+0000 7f59f6000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:07:03.008 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:07:03.004+0000 7f59f6000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:07:03.008 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:07:03.004+0000 7f59f6000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:07:03.008 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:07:03.004+0000 7f59f6000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:07:03.008 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:07:03.004+0000 7f59f6000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:07:03.027 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:07:03.024+0000 7f59f5600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:07:03.027 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:07:03.024+0000 7f59f5600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:07:03.027 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:07:03.024+0000 7f59f5600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:07:03.027 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:07:03.024+0000 7f59f5600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:07:03.027 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:07:03.024+0000 7f59f5600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:07:03.028 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs ss1clone1 2024-01-15T18:07:03.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.173+0000 7fddba400640 1 Processor -- start 2024-01-15T18:07:03.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.173+0000 7fddba400640 1 -- start start 2024-01-15T18:07:03.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.173+0000 7fddba400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fddb415d3b0 0x7fddb40ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:03.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.173+0000 7fddba400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fddb40d01d0 0x7fddb40b32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:03.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.173+0000 7fddba400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb40b2e20 con 0x7fddb415d790 2024-01-15T18:07:03.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.173+0000 7fddba400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb40cf290 con 0x7fddb40d01d0 2024-01-15T18:07:03.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.173+0000 7fddba400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb40cefb0 con 0x7fddb415d3b0 2024-01-15T18:07:03.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.173+0000 7fddb9a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fddb40d01d0 0x7fddb40b32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:03.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.173+0000 7fddb9a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fddb40d01d0 0x7fddb40b32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56120/0 (socket says 172.21.15.27:56120) 2024-01-15T18:07:03.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.173+0000 7fddb9a00640 1 -- 172.21.15.27:0/2673678488 learned_addr learned my addr 172.21.15.27:0/2673678488 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:03.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.173+0000 7fddb9000640 1 --2- 172.21.15.27:0/2673678488 >> v2:172.21.15.93:3300/0 conn(0x7fddb415d3b0 0x7fddb40ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:03.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.177+0000 7fddb9a00640 1 -- 172.21.15.27:0/2673678488 >> v2:172.21.15.93:3300/0 conn(0x7fddb415d3b0 msgr2=0x7fddb40ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:03.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.177+0000 7fddb9a00640 1 --2- 172.21.15.27:0/2673678488 >> v2:172.21.15.93:3300/0 conn(0x7fddb415d3b0 0x7fddb40ce760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.177+0000 7fddb9a00640 1 -- 172.21.15.27:0/2673678488 >> v1:172.21.15.93:6789/0 conn(0x7fddb415d790 legacy=0x7fddb40cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:03.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.177+0000 7fddb9a00640 1 -- 172.21.15.27:0/2673678488 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb40b27d0 con 0x7fddb40d01d0 2024-01-15T18:07:03.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.177+0000 7fddb3400640 1 -- 172.21.15.27:0/2673678488 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353321029 0 0) 0x7fddb40b2e20 con 0x7fddb415d790 2024-01-15T18:07:03.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.177+0000 7fddb9a00640 1 --2- 172.21.15.27:0/2673678488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddb40d01d0 0x7fddb40b32b0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fddb408c4d0 tx=0x7fddac01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f0bb8fe992a5c5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:03.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.177+0000 7fddb3400640 1 -- 172.21.15.27:0/2673678488 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac0a5820 con 0x7fddb40d01d0 2024-01-15T18:07:03.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.177+0000 7fddb3400640 1 -- 172.21.15.27:0/2673678488 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddac0a5980 con 0x7fddb40d01d0 2024-01-15T18:07:03.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.177+0000 7fddb3400640 1 -- 172.21.15.27:0/2673678488 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac0a5d10 con 0x7fddb40d01d0 2024-01-15T18:07:03.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.181+0000 7fddba400640 1 -- 172.21.15.27:0/2673678488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddb40d01d0 msgr2=0x7fddb40b32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:03.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.181+0000 7fddba400640 1 --2- 172.21.15.27:0/2673678488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddb40d01d0 0x7fddb40b32b0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fddb408c4d0 tx=0x7fddac01c9a0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.181+0000 7fddba400640 1 -- 172.21.15.27:0/2673678488 shutdown_connections 2024-01-15T18:07:03.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.181+0000 7fddba400640 1 --2- 172.21.15.27:0/2673678488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddb40d01d0 0x7fddb40b32b0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.181+0000 7fddba400640 1 --2- 172.21.15.27:0/2673678488 >> v2:172.21.15.93:3300/0 conn(0x7fddb415d3b0 0x7fddb40ce760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.181+0000 7fddba400640 1 -- 172.21.15.27:0/2673678488 >> 172.21.15.27:0/2673678488 conn(0x7fddb40d30b0 msgr2=0x7fddb40d3490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:03.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.181+0000 7fddba400640 1 -- 172.21.15.27:0/2673678488 shutdown_connections 2024-01-15T18:07:03.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.181+0000 7fddba400640 1 -- 172.21.15.27:0/2673678488 wait complete. 2024-01-15T18:07:03.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.181+0000 7fddba400640 1 Processor -- start 2024-01-15T18:07:03.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.181+0000 7fddba400640 1 -- start start 2024-01-15T18:07:03.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.181+0000 7fddba400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fddb40d01d0 0x7fddb4157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:03.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.181+0000 7fddba400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddb415d3b0 0x7fddb40cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:03.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.181+0000 7fddba400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fddb415d790 0x7fddb41f8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:03.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.181+0000 7fddba400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb409f420 con 0x7fddb415d3b0 2024-01-15T18:07:03.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.181+0000 7fddba400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb4157a50 con 0x7fddb415d790 2024-01-15T18:07:03.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.181+0000 7fddba400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb409f140 con 0x7fddb40d01d0 2024-01-15T18:07:03.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.181+0000 7fddb9000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fddb40d01d0 0x7fddb4157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:03.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.181+0000 7fddb3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddb415d3b0 0x7fddb40cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:03.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.181+0000 7fddb9000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fddb40d01d0 0x7fddb4157510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:37860/0 (socket says 172.21.15.27:37860) 2024-01-15T18:07:03.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.181+0000 7fddb9000640 1 -- 172.21.15.27:0/79648995 learned_addr learned my addr 172.21.15.27:0/79648995 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:03.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.181+0000 7fddb9a00640 1 --2- 172.21.15.27:0/79648995 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fddb415d790 0x7fddb41f8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:03.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.185+0000 7fddb9000640 1 -- 172.21.15.27:0/79648995 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fddb415d790 msgr2=0x7fddb41f8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:03.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.185+0000 7fddb9000640 1 --2- 172.21.15.27:0/79648995 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fddb415d790 0x7fddb41f8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.185+0000 7fddb9000640 1 -- 172.21.15.27:0/79648995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddb415d3b0 msgr2=0x7fddb40cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:03.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.185+0000 7fddb9000640 1 --2- 172.21.15.27:0/79648995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddb415d3b0 0x7fddb40cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.185+0000 7fddb9000640 1 -- 172.21.15.27:0/79648995 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddac009b30 con 0x7fddb40d01d0 2024-01-15T18:07:03.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.185+0000 7fddb3e00640 1 --2- 172.21.15.27:0/79648995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddb415d3b0 0x7fddb40cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:07:03.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.185+0000 7fddb9000640 1 --2- 172.21.15.27:0/79648995 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fddb40d01d0 0x7fddb4157510 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdda400e930 tx=0x7fdda400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:03.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.185+0000 7fddb0c00640 1 -- 172.21.15.27:0/79648995 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda4009690 con 0x7fddb40d01d0 2024-01-15T18:07:03.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.185+0000 7fddb0c00640 1 -- 172.21.15.27:0/79648995 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda400ca00 con 0x7fddb40d01d0 2024-01-15T18:07:03.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.185+0000 7fddb0c00640 1 -- 172.21.15.27:0/79648995 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda400cd70 con 0x7fddb40d01d0 2024-01-15T18:07:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.185+0000 7fddba400640 1 -- 172.21.15.27:0/79648995 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb40ca2b0 con 0x7fddb40d01d0 2024-01-15T18:07:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.185+0000 7fddba400640 1 -- 172.21.15.27:0/79648995 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb40ca870 con 0x7fddb40d01d0 2024-01-15T18:07:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.185+0000 7fddba400640 1 -- 172.21.15.27:0/79648995 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddb404e850 con 0x7fddb40d01d0 2024-01-15T18:07:03.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.185+0000 7fddb0c00640 1 -- 172.21.15.27:0/79648995 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 7) v1 ==== 86313+0+0 (secure 0 0 0) 0x7fdda4013470 con 0x7fddb40d01d0 2024-01-15T18:07:03.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.189+0000 7fddb0c00640 1 --2- 172.21.15.27:0/79648995 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd8c066aa0 0x7fdd8c068f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:03.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.189+0000 7fddb0c00640 1 -- 172.21.15.27:0/79648995 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdda40848d0 con 0x7fddb40d01d0 2024-01-15T18:07:03.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.189+0000 7fddb3e00640 1 --2- 172.21.15.27:0/79648995 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd8c066aa0 0x7fdd8c068f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:03.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.189+0000 7fddb3e00640 1 --2- 172.21.15.27:0/79648995 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd8c066aa0 0x7fdd8c068f60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fddb40b05d0 tx=0x7fdd9c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:03.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.193+0000 7fddb0c00640 1 -- 172.21.15.27:0/79648995 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fdda4051aa0 con 0x7fddb40d01d0 2024-01-15T18:07:03.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.373+0000 7fddba400640 1 -- 172.21.15.27:0/79648995 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fddb40b0920 con 0x7fdd8c066aa0 2024-01-15T18:07:03.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.389+0000 7fddb0c00640 1 -- 172.21.15.27:0/79648995 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fddb40b0920 con 0x7fdd8c066aa0 2024-01-15T18:07:03.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.389+0000 7fddba400640 1 -- 172.21.15.27:0/79648995 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd8c066aa0 msgr2=0x7fdd8c068f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:03.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.389+0000 7fddba400640 1 --2- 172.21.15.27:0/79648995 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd8c066aa0 0x7fdd8c068f60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fddb40b05d0 tx=0x7fdd9c009040 comp rx=0 tx=0).stop 2024-01-15T18:07:03.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.389+0000 7fddba400640 1 -- 172.21.15.27:0/79648995 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fddb40d01d0 msgr2=0x7fddb4157510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.389+0000 7fddba400640 1 --2- 172.21.15.27:0/79648995 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fddb40d01d0 0x7fddb4157510 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdda400e930 tx=0x7fdda400edf0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.389+0000 7fddba400640 1 -- 172.21.15.27:0/79648995 shutdown_connections 2024-01-15T18:07:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.389+0000 7fddba400640 1 --2- 172.21.15.27:0/79648995 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd8c066aa0 0x7fdd8c068f60 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.389+0000 7fddba400640 1 --2- 172.21.15.27:0/79648995 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fddb415d790 0x7fddb41f8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.389+0000 7fddba400640 1 --2- 172.21.15.27:0/79648995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddb415d3b0 0x7fddb40cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.389+0000 7fddba400640 1 --2- 172.21.15.27:0/79648995 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fddb40d01d0 0x7fddb4157510 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.389+0000 7fddba400640 1 -- 172.21.15.27:0/79648995 >> 172.21.15.27:0/79648995 conn(0x7fddb40d30b0 msgr2=0x7fddb406c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.393+0000 7fddba400640 1 -- 172.21.15.27:0/79648995 shutdown_connections 2024-01-15T18:07:03.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.393+0000 7fddba400640 1 -- 172.21.15.27:0/79648995 wait complete. 2024-01-15T18:07:03.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:03.420 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-15T18:07:03.437 INFO:teuthology.orchestra.run.smithi027.stdout:2 2024-01-15T18:07:03.438 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-15T18:07:03.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.617+0000 7f8862400640 1 Processor -- start 2024-01-15T18:07:03.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.617+0000 7f8862400640 1 -- start start 2024-01-15T18:07:03.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.621+0000 7f8862400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f885c15d680 0x7f885c0cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:03.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.621+0000 7f8862400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f885c0ccf80 0x7f885c0cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:03.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.621+0000 7f8862400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f885c0cb4d0 con 0x7f885c0cfc30 2024-01-15T18:07:03.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.621+0000 7f8862400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f885c0d0120 con 0x7f885c15d680 2024-01-15T18:07:03.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.621+0000 7f8862400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f885c0d0400 con 0x7f885c0ccf80 2024-01-15T18:07:03.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.621+0000 7f885b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f885c15d680 0x7f885c0cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:03.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.621+0000 7f885aa00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f885c0cfc30 0x7f885c06b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57586/0 (socket says 172.21.15.27:57586) 2024-01-15T18:07:03.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.621+0000 7f885aa00640 1 -- 172.21.15.27:0/179876349 learned_addr learned my addr 172.21.15.27:0/179876349 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:03.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.621+0000 7f885be00640 1 --2- 172.21.15.27:0/179876349 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f885c0ccf80 0x7f885c0cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:03.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.621+0000 7f885b400640 1 -- 172.21.15.27:0/179876349 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f885c0ccf80 msgr2=0x7f885c0cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:03.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.621+0000 7f885b400640 1 --2- 172.21.15.27:0/179876349 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f885c0ccf80 0x7f885c0cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.621+0000 7f885b400640 1 -- 172.21.15.27:0/179876349 >> v1:172.21.15.27:6789/0 conn(0x7f885c0cfc30 legacy=0x7f885c06b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:03.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.621+0000 7f885b400640 1 -- 172.21.15.27:0/179876349 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f885c0cb770 con 0x7f885c15d680 2024-01-15T18:07:03.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.625+0000 7f885b400640 1 --2- 172.21.15.27:0/179876349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f885c15d680 0x7f885c0cd700 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f884c0097c0 tx=0x7f884c01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=289d73414f7e40bf server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:03.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.625+0000 7f885a000640 1 -- 172.21.15.27:0/179876349 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f885c0ce850 con 0x7f885c15d680 2024-01-15T18:07:03.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.625+0000 7f885a000640 1 -- 172.21.15.27:0/179876349 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f884c0ac640 con 0x7f885c15d680 2024-01-15T18:07:03.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.625+0000 7f885a000640 1 -- 172.21.15.27:0/179876349 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f884c0ac9d0 con 0x7f885c15d680 2024-01-15T18:07:03.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.625+0000 7f8862400640 1 -- 172.21.15.27:0/179876349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f885c15d680 msgr2=0x7f885c0cd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:03.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.625+0000 7f8862400640 1 --2- 172.21.15.27:0/179876349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f885c15d680 0x7f885c0cd700 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f884c0097c0 tx=0x7f884c01c3c0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.625+0000 7f8862400640 1 -- 172.21.15.27:0/179876349 shutdown_connections 2024-01-15T18:07:03.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.625+0000 7f8862400640 1 --2- 172.21.15.27:0/179876349 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f885c0ccf80 0x7f885c0cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.625+0000 7f8862400640 1 --2- 172.21.15.27:0/179876349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f885c15d680 0x7f885c0cd700 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.625+0000 7f8862400640 1 -- 172.21.15.27:0/179876349 >> 172.21.15.27:0/179876349 conn(0x7f885c0d0e80 msgr2=0x7f885c0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:03.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.625+0000 7f8862400640 1 -- 172.21.15.27:0/179876349 shutdown_connections 2024-01-15T18:07:03.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.625+0000 7f8862400640 1 -- 172.21.15.27:0/179876349 wait complete. 2024-01-15T18:07:03.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.625+0000 7f8862400640 1 Processor -- start 2024-01-15T18:07:03.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.625+0000 7f8862400640 1 -- start start 2024-01-15T18:07:03.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.629+0000 7f8862400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f885c0ccf80 0x7f885c151730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:03.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.629+0000 7f8862400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f885c0cfc30 0x7f885c151ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:03.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.629+0000 7f8862400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f885c15d680 0x7f885c152410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:03.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.629+0000 7f8862400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885c153780 con 0x7f885c15d680 2024-01-15T18:07:03.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.629+0000 7f8862400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885c152a90 con 0x7f885c0cfc30 2024-01-15T18:07:03.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.629+0000 7f8862400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f885c152d70 con 0x7f885c0ccf80 2024-01-15T18:07:03.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.629+0000 7f885be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f885c15d680 0x7f885c152410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:03.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.629+0000 7f885be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f885c15d680 0x7f885c152410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34630/0 (socket says 172.21.15.27:34630) 2024-01-15T18:07:03.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.629+0000 7f885be00640 1 -- 172.21.15.27:0/231454315 learned_addr learned my addr 172.21.15.27:0/231454315 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:03.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.629+0000 7f885aa00640 1 --2- 172.21.15.27:0/231454315 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f885c0cfc30 0x7f885c151ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:03.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.629+0000 7f885b400640 1 --2- 172.21.15.27:0/231454315 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f885c0ccf80 0x7f885c151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:03.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.629+0000 7f885be00640 1 -- 172.21.15.27:0/231454315 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f885c0ccf80 msgr2=0x7f885c151730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:03.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.629+0000 7f885be00640 1 --2- 172.21.15.27:0/231454315 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f885c0ccf80 0x7f885c151730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.629+0000 7f885be00640 1 -- 172.21.15.27:0/231454315 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f885c0cfc30 msgr2=0x7f885c151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:03.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.629+0000 7f885be00640 1 --2- 172.21.15.27:0/231454315 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f885c0cfc30 0x7f885c151ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.629+0000 7f885be00640 1 -- 172.21.15.27:0/231454315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f884c009420 con 0x7f885c15d680 2024-01-15T18:07:03.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.629+0000 7f885be00640 1 --2- 172.21.15.27:0/231454315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f885c15d680 0x7f885c152410 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8854007e40 tx=0x7f885400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:03.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.629+0000 7f8853400640 1 -- 172.21.15.27:0/231454315 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8854014620 con 0x7f885c15d680 2024-01-15T18:07:03.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.629+0000 7f8853400640 1 -- 172.21.15.27:0/231454315 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f885400ee90 con 0x7f885c15d680 2024-01-15T18:07:03.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.629+0000 7f8862400640 1 -- 172.21.15.27:0/231454315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f885c153050 con 0x7f885c15d680 2024-01-15T18:07:03.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.629+0000 7f8853400640 1 -- 172.21.15.27:0/231454315 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88540149e0 con 0x7f885c15d680 2024-01-15T18:07:03.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.629+0000 7f8862400640 1 -- 172.21.15.27:0/231454315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f885c151c70 con 0x7f885c15d680 2024-01-15T18:07:03.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.629+0000 7f8862400640 1 -- 172.21.15.27:0/231454315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f885c156850 con 0x7f885c15d680 2024-01-15T18:07:03.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.629+0000 7f8853400640 1 -- 172.21.15.27:0/231454315 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) v1 ==== 86313+0+0 (secure 0 0 0) 0x7f8854014b40 con 0x7f885c15d680 2024-01-15T18:07:03.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.633+0000 7f8853400640 1 --2- 172.21.15.27:0/231454315 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8834066aa0 0x7f8834068f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:03.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.633+0000 7f8853400640 1 -- 172.21.15.27:0/231454315 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8854084fa0 con 0x7f885c15d680 2024-01-15T18:07:03.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.633+0000 7f885b400640 1 --2- 172.21.15.27:0/231454315 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8834066aa0 0x7f8834068f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:03.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.633+0000 7f885b400640 1 --2- 172.21.15.27:0/231454315 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8834066aa0 0x7f8834068f60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f884c0ae000 tx=0x7f884c0a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:03.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.637+0000 7f8853400640 1 -- 172.21.15.27:0/231454315 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f885c156850 con 0x7f885c15d680 2024-01-15T18:07:03.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:03.813+0000 7f8862400640 1 -- 172.21.15.27:0/231454315 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7f885c156850 con 0x7f8834066aa0 2024-01-15T18:07:03.970 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-01-15T18:07:03.961+0000 7f053dc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-15T18:07:04.059 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-15T18:07:05.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:05.065+0000 7f8853400640 1 -- 172.21.15.27:0/231454315 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f88540523d0 con 0x7f885c15d680 2024-01-15T18:07:06.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:06.985+0000 7f8853400640 1 -- 172.21.15.27:0/231454315 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f885c156850 con 0x7f8834066aa0 2024-01-15T18:07:06.993 INFO:teuthology.orchestra.run.smithi027.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-15T18:07:06.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:06.989+0000 7f8862400640 1 -- 172.21.15.27:0/231454315 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8834066aa0 msgr2=0x7f8834068f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:06.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:06.989+0000 7f8862400640 1 --2- 172.21.15.27:0/231454315 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8834066aa0 0x7f8834068f60 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f884c0ae000 tx=0x7f884c0a7000 comp rx=0 tx=0).stop 2024-01-15T18:07:06.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:06.989+0000 7f8862400640 1 -- 172.21.15.27:0/231454315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f885c15d680 msgr2=0x7f885c152410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:06.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:06.989+0000 7f8862400640 1 --2- 172.21.15.27:0/231454315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f885c15d680 0x7f885c152410 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8854007e40 tx=0x7f885400e4b0 comp rx=0 tx=0).stop 2024-01-15T18:07:06.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:06.989+0000 7f8862400640 1 -- 172.21.15.27:0/231454315 shutdown_connections 2024-01-15T18:07:06.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:06.989+0000 7f8862400640 1 --2- 172.21.15.27:0/231454315 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8834066aa0 0x7f8834068f60 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:06.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:06.989+0000 7f8862400640 1 --2- 172.21.15.27:0/231454315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f885c15d680 0x7f885c152410 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:06.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:06.989+0000 7f8862400640 1 --2- 172.21.15.27:0/231454315 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f885c0cfc30 0x7f885c151ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:06.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:06.989+0000 7f8862400640 1 --2- 172.21.15.27:0/231454315 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f885c0ccf80 0x7f885c151730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:06.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:06.989+0000 7f8862400640 1 -- 172.21.15.27:0/231454315 >> 172.21.15.27:0/231454315 conn(0x7f885c0d0e80 msgr2=0x7f885c0ae980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:06.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:06.993+0000 7f8862400640 1 -- 172.21.15.27:0/231454315 shutdown_connections 2024-01-15T18:07:07.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:06.993+0000 7f8862400640 1 -- 172.21.15.27:0/231454315 wait complete. 2024-01-15T18:07:07.037 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T18:07:07.037 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:07:07.134 DEBUG:teuthology.orchestra.run.smithi093:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:07:07.237 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T18:07:07.238 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:07:07.238 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:07:07.253 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:07:07.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.433+0000 7f26f0000640 1 Processor -- start 2024-01-15T18:07:07.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.433+0000 7f26f0000640 1 -- start start 2024-01-15T18:07:07.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.433+0000 7f26f0000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f26e815e7d0 0x7f26e80cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:07.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.433+0000 7f26f0000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26e80b20d0 0x7f26e80cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:07.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.433+0000 7f26f0000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e80b38b0 con 0x7f26e80cedc0 2024-01-15T18:07:07.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.433+0000 7f26f0000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e80cd550 con 0x7f26e815e7d0 2024-01-15T18:07:07.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.433+0000 7f26f0000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e80cd270 con 0x7f26e80b20d0 2024-01-15T18:07:07.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.433+0000 7f26ecc00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f26e80cedc0 0x7f26e80cba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57596/0 (socket says 172.21.15.27:57596) 2024-01-15T18:07:07.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.433+0000 7f26ecc00640 1 -- 172.21.15.27:0/612132103 learned_addr learned my addr 172.21.15.27:0/612132103 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:07.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.433+0000 7f26ed600640 1 --2- 172.21.15.27:0/612132103 >> v2:172.21.15.93:3300/0 conn(0x7f26e815e7d0 0x7f26e80cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:07.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.433+0000 7f26ee000640 1 --2- 172.21.15.27:0/612132103 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26e80b20d0 0x7f26e80cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:07.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.437+0000 7f26e7e00640 1 -- 172.21.15.27:0/612132103 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537958437 0 0) 0x7f26e80b38b0 con 0x7f26e80cedc0 2024-01-15T18:07:07.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.437+0000 7f26e7e00640 1 -- 172.21.15.27:0/612132103 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c8003530 con 0x7f26e80cedc0 2024-01-15T18:07:07.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.437+0000 7f26ed600640 1 -- 172.21.15.27:0/612132103 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26e80b20d0 msgr2=0x7f26e80cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:07.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.437+0000 7f26ed600640 1 --2- 172.21.15.27:0/612132103 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26e80b20d0 0x7f26e80cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:07.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.437+0000 7f26ed600640 1 -- 172.21.15.27:0/612132103 >> v1:172.21.15.27:6789/0 conn(0x7f26e80cedc0 legacy=0x7f26e80cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:07.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.437+0000 7f26ed600640 1 -- 172.21.15.27:0/612132103 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e80b35d0 con 0x7f26e815e7d0 2024-01-15T18:07:07.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.437+0000 7f26ee000640 1 --2- 172.21.15.27:0/612132103 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26e80b20d0 0x7f26e80cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:07:07.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.437+0000 7f26ed600640 1 --2- 172.21.15.27:0/612132103 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f26e815e7d0 0x7f26e80cf2a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f26d80097c0 tx=0x7f26d801c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c5de669e00d8973 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:07.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.437+0000 7f26e7e00640 1 -- 172.21.15.27:0/612132103 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d80ab600 con 0x7f26e815e7d0 2024-01-15T18:07:07.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.437+0000 7f26e7e00640 1 -- 172.21.15.27:0/612132103 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d80a7050 con 0x7f26e815e7d0 2024-01-15T18:07:07.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.437+0000 7f26e7e00640 1 -- 172.21.15.27:0/612132103 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d80ab980 con 0x7f26e815e7d0 2024-01-15T18:07:07.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.441+0000 7f26f0000640 1 -- 172.21.15.27:0/612132103 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f26e815e7d0 msgr2=0x7f26e80cf2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:07.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.441+0000 7f26f0000640 1 --2- 172.21.15.27:0/612132103 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f26e815e7d0 0x7f26e80cf2a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f26d80097c0 tx=0x7f26d801c480 comp rx=0 tx=0).stop 2024-01-15T18:07:07.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.441+0000 7f26f0000640 1 -- 172.21.15.27:0/612132103 shutdown_connections 2024-01-15T18:07:07.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.441+0000 7f26f0000640 1 --2- 172.21.15.27:0/612132103 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26e80b20d0 0x7f26e80cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:07.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.441+0000 7f26f0000640 1 --2- 172.21.15.27:0/612132103 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f26e815e7d0 0x7f26e80cf2a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:07.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.441+0000 7f26f0000640 1 -- 172.21.15.27:0/612132103 >> 172.21.15.27:0/612132103 conn(0x7f26e80b2a10 msgr2=0x7f26e80b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:07.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.441+0000 7f26f0000640 1 -- 172.21.15.27:0/612132103 shutdown_connections 2024-01-15T18:07:07.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.441+0000 7f26f0000640 1 -- 172.21.15.27:0/612132103 wait complete. 2024-01-15T18:07:07.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.441+0000 7f26f0000640 1 Processor -- start 2024-01-15T18:07:07.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.445+0000 7f26f0000640 1 -- start start 2024-01-15T18:07:07.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.445+0000 7f26f0000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26e80b20d0 0x7f26e81faf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:07.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.445+0000 7f26f0000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26e80cedc0 0x7f26e80ae330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:07.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.445+0000 7f26f0000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f26e815e7d0 0x7f26e80ae870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:07.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.445+0000 7f26f0000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e81f4a30 con 0x7f26e80b20d0 2024-01-15T18:07:07.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.445+0000 7f26f0000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e8153fd0 con 0x7f26e815e7d0 2024-01-15T18:07:07.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.445+0000 7f26f0000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e8153450 con 0x7f26e80cedc0 2024-01-15T18:07:07.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.445+0000 7f26ed600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26e80b20d0 0x7f26e81faf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:07.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.445+0000 7f26ed600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26e80b20d0 0x7f26e81faf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34642/0 (socket says 172.21.15.27:34642) 2024-01-15T18:07:07.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.445+0000 7f26ed600640 1 -- 172.21.15.27:0/591977070 learned_addr learned my addr 172.21.15.27:0/591977070 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:07.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.445+0000 7f26ee000640 1 --2- 172.21.15.27:0/591977070 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f26e815e7d0 0x7f26e80ae870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:07.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.445+0000 7f26ecc00640 1 --2- 172.21.15.27:0/591977070 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26e80cedc0 0x7f26e80ae330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:07.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.445+0000 7f26ee000640 1 -- 172.21.15.27:0/591977070 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26e80cedc0 msgr2=0x7f26e80ae330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:07.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.445+0000 7f26ee000640 1 --2- 172.21.15.27:0/591977070 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26e80cedc0 0x7f26e80ae330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:07.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.445+0000 7f26ee000640 1 -- 172.21.15.27:0/591977070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26e80b20d0 msgr2=0x7f26e81faf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:07.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.445+0000 7f26ee000640 1 --2- 172.21.15.27:0/591977070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26e80b20d0 0x7f26e81faf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:07.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.445+0000 7f26ee000640 1 -- 172.21.15.27:0/591977070 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e0009a70 con 0x7f26e815e7d0 2024-01-15T18:07:07.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.445+0000 7f26ee000640 1 --2- 172.21.15.27:0/591977070 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f26e815e7d0 0x7f26e80ae870 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f26e000ef90 tx=0x7f26e000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:07.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.445+0000 7f26e5600640 1 -- 172.21.15.27:0/591977070 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e0014480 con 0x7f26e815e7d0 2024-01-15T18:07:07.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.445+0000 7f26e5600640 1 -- 172.21.15.27:0/591977070 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e000f040 con 0x7f26e815e7d0 2024-01-15T18:07:07.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.445+0000 7f26e5600640 1 -- 172.21.15.27:0/591977070 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e0014840 con 0x7f26e815e7d0 2024-01-15T18:07:07.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.445+0000 7f26f0000640 1 -- 172.21.15.27:0/591977070 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26d8009420 con 0x7f26e815e7d0 2024-01-15T18:07:07.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.445+0000 7f26f0000640 1 -- 172.21.15.27:0/591977070 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e8153a10 con 0x7f26e815e7d0 2024-01-15T18:07:07.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.449+0000 7f26f0000640 1 -- 172.21.15.27:0/591977070 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e81566e0 con 0x7f26e815e7d0 2024-01-15T18:07:07.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.449+0000 7f26e5600640 1 -- 172.21.15.27:0/591977070 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f26e00149a0 con 0x7f26e815e7d0 2024-01-15T18:07:07.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.449+0000 7f26e5600640 1 --2- 172.21.15.27:0/591977070 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f26c0066b30 0x7f26c0068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:07.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.449+0000 7f26e5600640 1 -- 172.21.15.27:0/591977070 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f26e0085390 con 0x7f26e815e7d0 2024-01-15T18:07:07.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.449+0000 7f26ed600640 1 --2- 172.21.15.27:0/591977070 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f26c0066b30 0x7f26c0068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:07.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.453+0000 7f26ed600640 1 --2- 172.21.15.27:0/591977070 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f26c0066b30 0x7f26c0068ff0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f26d801ab30 tx=0x7f26d80a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:07.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.453+0000 7f26e5600640 1 -- 172.21.15.27:0/591977070 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f26e00524e0 con 0x7f26e815e7d0 2024-01-15T18:07:07.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.749+0000 7f26f0000640 1 -- 172.21.15.27:0/591977070 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26e804e850 con 0x7f26e815e7d0 2024-01-15T18:07:07.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.749+0000 7f26e5600640 1 -- 172.21.15.27:0/591977070 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+12933 (secure 0 0 0) 0x7f26e0055b00 con 0x7f26e815e7d0 2024-01-15T18:07:07.761 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:07:07.761 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":37,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:07:06.979289+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6834/2187944510":"2024-01-16T18:07:03.967765+0000","172.21.15.27:6832/2187944510":"2024-01-16T18:07:03.967765+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:07:07.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.753+0000 7f26f0000640 1 -- 172.21.15.27:0/591977070 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f26c0066b30 msgr2=0x7f26c0068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:07.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.753+0000 7f26f0000640 1 --2- 172.21.15.27:0/591977070 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f26c0066b30 0x7f26c0068ff0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f26d801ab30 tx=0x7f26d80a6000 comp rx=0 tx=0).stop 2024-01-15T18:07:07.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.753+0000 7f26f0000640 1 -- 172.21.15.27:0/591977070 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f26e815e7d0 msgr2=0x7f26e80ae870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:07.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.753+0000 7f26f0000640 1 --2- 172.21.15.27:0/591977070 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f26e815e7d0 0x7f26e80ae870 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f26e000ef90 tx=0x7f26e000c490 comp rx=0 tx=0).stop 2024-01-15T18:07:07.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.753+0000 7f26f0000640 1 -- 172.21.15.27:0/591977070 shutdown_connections 2024-01-15T18:07:07.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.753+0000 7f26f0000640 1 --2- 172.21.15.27:0/591977070 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f26c0066b30 0x7f26c0068ff0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:07.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.753+0000 7f26f0000640 1 --2- 172.21.15.27:0/591977070 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f26e815e7d0 0x7f26e80ae870 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:07.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.753+0000 7f26f0000640 1 --2- 172.21.15.27:0/591977070 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26e80cedc0 0x7f26e80ae330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:07.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.753+0000 7f26f0000640 1 --2- 172.21.15.27:0/591977070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26e80b20d0 0x7f26e81faf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:07.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.753+0000 7f26f0000640 1 -- 172.21.15.27:0/591977070 >> 172.21.15.27:0/591977070 conn(0x7f26e80b2a10 msgr2=0x7f26e815c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:07.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.753+0000 7f26f0000640 1 -- 172.21.15.27:0/591977070 shutdown_connections 2024-01-15T18:07:07.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.753+0000 7f26f0000640 1 -- 172.21.15.27:0/591977070 wait complete. 2024-01-15T18:07:07.795 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:07:07.796 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:07:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.977+0000 7f5bed000640 1 Processor -- start 2024-01-15T18:07:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.981+0000 7f5bed000640 1 -- start start 2024-01-15T18:07:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.981+0000 7f5bed000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5be815d0d0 0x7f5be80b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.981+0000 7f5bed000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f5be80cfb20 0x7f5be80b4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.981+0000 7f5bed000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5be80cff40 0x7f5be8161410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.981+0000 7f5bed000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5be80721c0 con 0x7f5be80cff40 2024-01-15T18:07:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.981+0000 7f5bed000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5be80d0400 con 0x7f5be80cfb20 2024-01-15T18:07:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.981+0000 7f5bed000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be8071940 con 0x7f5be815d0d0 2024-01-15T18:07:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.981+0000 7f5be5e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5be815d0d0 0x7f5be80b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.981+0000 7f5be6800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5be80cff40 0x7f5be8161410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.981+0000 7f5be5400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f5be80cfb20 0x7f5be80b4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.981+0000 7f5be5e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5be815d0d0 0x7f5be80b2fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:41414/0 (socket says 172.21.15.27:41414) 2024-01-15T18:07:07.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.981+0000 7f5be5e00640 1 -- 172.21.15.27:0/2780563831 learned_addr learned my addr 172.21.15.27:0/2780563831 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:07.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.981+0000 7f5be5e00640 1 -- 172.21.15.27:0/2780563831 >> v2:172.21.15.93:3300/0 conn(0x7f5be80cfb20 msgr2=0x7f5be80b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:07.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.981+0000 7f5be5e00640 1 --2- 172.21.15.27:0/2780563831 >> v2:172.21.15.93:3300/0 conn(0x7f5be80cfb20 0x7f5be80b4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:07.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.981+0000 7f5be5e00640 1 -- 172.21.15.27:0/2780563831 >> v2:172.21.15.27:3300/0 conn(0x7f5be80cff40 msgr2=0x7f5be8161410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:07.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.981+0000 7f5be5e00640 1 --2- 172.21.15.27:0/2780563831 >> v2:172.21.15.27:3300/0 conn(0x7f5be80cff40 0x7f5be8161410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:07.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.981+0000 7f5be5e00640 1 -- 172.21.15.27:0/2780563831 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be8071c00 con 0x7f5be815d0d0 2024-01-15T18:07:07.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.981+0000 7f5be5400640 1 --2- 172.21.15.27:0/2780563831 >> v2:172.21.15.93:3300/0 conn(0x7f5be80cfb20 0x7f5be80b4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:07:07.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.981+0000 7f5be6800640 1 --2- 172.21.15.27:0/2780563831 >> v2:172.21.15.27:3300/0 conn(0x7f5be80cff40 0x7f5be8161410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:07:07.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.985+0000 7f5be5e00640 1 --2- 172.21.15.27:0/2780563831 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5be815d0d0 0x7f5be80b2fa0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5be809e0a0 tx=0x7f5bd801c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da479549ba1975c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:07.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.985+0000 7f5be4a00640 1 -- 172.21.15.27:0/2780563831 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd80ab600 con 0x7f5be815d0d0 2024-01-15T18:07:07.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.985+0000 7f5be4a00640 1 -- 172.21.15.27:0/2780563831 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd80ab760 con 0x7f5be815d0d0 2024-01-15T18:07:07.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.985+0000 7f5be4a00640 1 -- 172.21.15.27:0/2780563831 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd80abaf0 con 0x7f5be815d0d0 2024-01-15T18:07:07.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.985+0000 7f5bed000640 1 -- 172.21.15.27:0/2780563831 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5be815d0d0 msgr2=0x7f5be80b2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:07.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.985+0000 7f5bed000640 1 --2- 172.21.15.27:0/2780563831 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5be815d0d0 0x7f5be80b2fa0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5be809e0a0 tx=0x7f5bd801c440 comp rx=0 tx=0).stop 2024-01-15T18:07:07.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.985+0000 7f5bed000640 1 -- 172.21.15.27:0/2780563831 shutdown_connections 2024-01-15T18:07:07.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.985+0000 7f5bed000640 1 --2- 172.21.15.27:0/2780563831 >> v2:172.21.15.27:3300/0 conn(0x7f5be80cff40 0x7f5be8161410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:07.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.985+0000 7f5bed000640 1 --2- 172.21.15.27:0/2780563831 >> v2:172.21.15.93:3300/0 conn(0x7f5be80cfb20 0x7f5be80b4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:07.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.985+0000 7f5bed000640 1 --2- 172.21.15.27:0/2780563831 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5be815d0d0 0x7f5be80b2fa0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:07.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.985+0000 7f5bed000640 1 -- 172.21.15.27:0/2780563831 >> 172.21.15.27:0/2780563831 conn(0x7f5be806b600 msgr2=0x7f5be806ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:07.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.985+0000 7f5bed000640 1 -- 172.21.15.27:0/2780563831 shutdown_connections 2024-01-15T18:07:07.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.985+0000 7f5bed000640 1 -- 172.21.15.27:0/2780563831 wait complete. 2024-01-15T18:07:07.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.989+0000 7f5bed000640 1 Processor -- start 2024-01-15T18:07:07.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.989+0000 7f5bed000640 1 -- start start 2024-01-15T18:07:07.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.989+0000 7f5bed000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5be80cfb20 0x7f5be8151ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:07.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.989+0000 7f5bed000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5be80cff40 0x7f5be8151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:07.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.989+0000 7f5bed000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5be815d0d0 0x7f5be8155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:07.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.989+0000 7f5bed000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be8151ac0 con 0x7f5be80cff40 2024-01-15T18:07:07.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.989+0000 7f5bed000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be8152680 con 0x7f5be80cfb20 2024-01-15T18:07:07.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.989+0000 7f5bed000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be81523a0 con 0x7f5be815d0d0 2024-01-15T18:07:07.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.989+0000 7f5be5400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5be80cff40 0x7f5be8151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:07.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.989+0000 7f5be5e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5be80cfb20 0x7f5be8151ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:07.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.989+0000 7f5be6800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5be815d0d0 0x7f5be8155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:07.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.989+0000 7f5be5e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5be80cfb20 0x7f5be8151ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:47880/0 (socket says 172.21.15.27:47880) 2024-01-15T18:07:07.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.989+0000 7f5be5e00640 1 -- 172.21.15.27:0/4210564992 learned_addr learned my addr 172.21.15.27:0/4210564992 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:07.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.989+0000 7f5be6800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5be815d0d0 0x7f5be8155540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:41430/0 (socket says 172.21.15.27:41430) 2024-01-15T18:07:07.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.989+0000 7f5be5400640 1 -- 172.21.15.27:0/4210564992 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5be815d0d0 msgr2=0x7f5be8155540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:07.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.989+0000 7f5be5400640 1 --2- 172.21.15.27:0/4210564992 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5be815d0d0 0x7f5be8155540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:07.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.989+0000 7f5be5400640 1 -- 172.21.15.27:0/4210564992 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5be80cfb20 msgr2=0x7f5be8151ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:08.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.989+0000 7f5be5400640 1 --2- 172.21.15.27:0/4210564992 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5be80cfb20 0x7f5be8151ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.989+0000 7f5be5400640 1 -- 172.21.15.27:0/4210564992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd0009420 con 0x7f5be80cff40 2024-01-15T18:07:08.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.989+0000 7f5be5400640 1 --2- 172.21.15.27:0/4210564992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5be80cff40 0x7f5be8151580 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f5bd000e840 tx=0x7f5bd000ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:08.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.989+0000 7f5bde000640 1 -- 172.21.15.27:0/4210564992 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd000ee90 con 0x7f5be80cff40 2024-01-15T18:07:08.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.989+0000 7f5bde000640 1 -- 172.21.15.27:0/4210564992 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd0004030 con 0x7f5be80cff40 2024-01-15T18:07:08.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.989+0000 7f5bde000640 1 -- 172.21.15.27:0/4210564992 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd00043c0 con 0x7f5be80cff40 2024-01-15T18:07:08.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.989+0000 7f5bed000640 1 -- 172.21.15.27:0/4210564992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5be0009a70 con 0x7f5be80cff40 2024-01-15T18:07:08.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.989+0000 7f5bed000640 1 -- 172.21.15.27:0/4210564992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd80094d0 con 0x7f5be80cff40 2024-01-15T18:07:08.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.993+0000 7f5bed000640 1 -- 172.21.15.27:0/4210564992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5be804e850 con 0x7f5be80cff40 2024-01-15T18:07:08.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.993+0000 7f5bde000640 1 -- 172.21.15.27:0/4210564992 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f5bd000c940 con 0x7f5be80cff40 2024-01-15T18:07:08.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.993+0000 7f5bde000640 1 --2- 172.21.15.27:0/4210564992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5bc0066b30 0x7f5bc0068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:08.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.993+0000 7f5bde000640 1 -- 172.21.15.27:0/4210564992 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f5bd0084410 con 0x7f5be80cff40 2024-01-15T18:07:08.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.993+0000 7f5be5e00640 1 --2- 172.21.15.27:0/4210564992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5bc0066b30 0x7f5bc0068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:08.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.993+0000 7f5be5e00640 1 --2- 172.21.15.27:0/4210564992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5bc0066b30 0x7f5bc0068ff0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5bd801c480 tx=0x7f5bd80a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:08.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:07.997+0000 7f5bde000640 1 -- 172.21.15.27:0/4210564992 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5bd0051550 con 0x7f5be80cff40 2024-01-15T18:07:08.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.157+0000 7f5bed000640 1 -- 172.21.15.27:0/4210564992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5be80af600 con 0x7f5be80cff40 2024-01-15T18:07:08.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.157+0000 7f5bde000640 1 -- 172.21.15.27:0/4210564992 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5bd0020020 con 0x7f5be80cff40 2024-01-15T18:07:08.164 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:07:08.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.157+0000 7f5bed000640 1 -- 172.21.15.27:0/4210564992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5bc0066b30 msgr2=0x7f5bc0068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:08.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.157+0000 7f5bed000640 1 --2- 172.21.15.27:0/4210564992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5bc0066b30 0x7f5bc0068ff0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5bd801c480 tx=0x7f5bd80a7000 comp rx=0 tx=0).stop 2024-01-15T18:07:08.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.157+0000 7f5bed000640 1 -- 172.21.15.27:0/4210564992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5be80cff40 msgr2=0x7f5be8151580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:08.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.157+0000 7f5bed000640 1 --2- 172.21.15.27:0/4210564992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5be80cff40 0x7f5be8151580 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f5bd000e840 tx=0x7f5bd000ec40 comp rx=0 tx=0).stop 2024-01-15T18:07:08.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.157+0000 7f5bed000640 1 -- 172.21.15.27:0/4210564992 shutdown_connections 2024-01-15T18:07:08.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.157+0000 7f5bed000640 1 --2- 172.21.15.27:0/4210564992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5bc0066b30 0x7f5bc0068ff0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.157+0000 7f5bed000640 1 --2- 172.21.15.27:0/4210564992 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5be815d0d0 0x7f5be8155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.157+0000 7f5bed000640 1 --2- 172.21.15.27:0/4210564992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5be80cff40 0x7f5be8151580 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.157+0000 7f5bed000640 1 --2- 172.21.15.27:0/4210564992 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5be80cfb20 0x7f5be8151ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.157+0000 7f5bed000640 1 -- 172.21.15.27:0/4210564992 >> 172.21.15.27:0/4210564992 conn(0x7f5be806b600 msgr2=0x7f5be8156db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:08.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.157+0000 7f5bed000640 1 -- 172.21.15.27:0/4210564992 shutdown_connections 2024-01-15T18:07:08.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.157+0000 7f5bed000640 1 -- 172.21.15.27:0/4210564992 wait complete. 2024-01-15T18:07:08.187 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:07:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.321+0000 7f730da00640 1 Processor -- start 2024-01-15T18:07:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.321+0000 7f730da00640 1 -- start start 2024-01-15T18:07:08.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.325+0000 7f730da00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7308143da0 0x7f730806d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:08.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.325+0000 7f730da00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f730806c200 0x7f73080cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:08.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.325+0000 7f730da00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f730806c750 con 0x7f73080cad40 2024-01-15T18:07:08.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.325+0000 7f730da00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7308144180 con 0x7f730806c200 2024-01-15T18:07:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.325+0000 7f730da00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f730806ca30 con 0x7f7308143da0 2024-01-15T18:07:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.325+0000 7f7306000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f730806c200 0x7f73080cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.325+0000 7f7306000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f730806c200 0x7f73080cb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34682/0 (socket says 172.21.15.27:34682) 2024-01-15T18:07:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.325+0000 7f7306000640 1 -- 172.21.15.27:0/1015051029 learned_addr learned my addr 172.21.15.27:0/1015051029 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.325+0000 7f7306a00640 1 --2- 172.21.15.27:0/1015051029 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7308143da0 0x7f730806d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.325+0000 7f7306000640 1 -- 172.21.15.27:0/1015051029 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7308143da0 msgr2=0x7f730806d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.325+0000 7f7306000640 1 --2- 172.21.15.27:0/1015051029 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7308143da0 0x7f730806d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.325+0000 7f7306000640 1 -- 172.21.15.27:0/1015051029 >> v1:172.21.15.93:6789/0 conn(0x7f73080cad40 legacy=0x7f73080701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.325+0000 7f7306000640 1 -- 172.21.15.27:0/1015051029 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730806cd10 con 0x7f730806c200 2024-01-15T18:07:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.325+0000 7f7306a00640 1 --2- 172.21.15.27:0/1015051029 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7308143da0 0x7f730806d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:07:08.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.325+0000 7f7306000640 1 --2- 172.21.15.27:0/1015051029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730806c200 0x7f73080cb9c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f72f0009870 tx=0x7f72f001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4ca9bca904d0262 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:08.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.325+0000 7f7305600640 1 -- 172.21.15.27:0/1015051029 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f00aa5f0 con 0x7f730806c200 2024-01-15T18:07:08.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.329+0000 7f7305600640 1 -- 172.21.15.27:0/1015051029 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f00aa750 con 0x7f730806c200 2024-01-15T18:07:08.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.329+0000 7f7305600640 1 -- 172.21.15.27:0/1015051029 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f00aaae0 con 0x7f730806c200 2024-01-15T18:07:08.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.329+0000 7f730da00640 1 -- 172.21.15.27:0/1015051029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730806c200 msgr2=0x7f73080cb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:08.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.329+0000 7f730da00640 1 --2- 172.21.15.27:0/1015051029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730806c200 0x7f73080cb9c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f72f0009870 tx=0x7f72f001c4d0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.329+0000 7f730da00640 1 -- 172.21.15.27:0/1015051029 shutdown_connections 2024-01-15T18:07:08.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.329+0000 7f730da00640 1 --2- 172.21.15.27:0/1015051029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730806c200 0x7f73080cb9c0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.329+0000 7f730da00640 1 --2- 172.21.15.27:0/1015051029 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7308143da0 0x7f730806d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.329+0000 7f730da00640 1 -- 172.21.15.27:0/1015051029 >> 172.21.15.27:0/1015051029 conn(0x7f73080b0850 msgr2=0x7f73080b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:08.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.329+0000 7f730da00640 1 -- 172.21.15.27:0/1015051029 shutdown_connections 2024-01-15T18:07:08.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.329+0000 7f730da00640 1 -- 172.21.15.27:0/1015051029 wait complete. 2024-01-15T18:07:08.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.329+0000 7f730da00640 1 Processor -- start 2024-01-15T18:07:08.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.329+0000 7f730da00640 1 -- start start 2024-01-15T18:07:08.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.329+0000 7f730da00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730806c200 0x7f7308158560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:08.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.329+0000 7f730da00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f73080cad40 0x7f730815e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:08.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.329+0000 7f730da00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7308143da0 0x7f7308159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:08.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.329+0000 7f730da00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730813d110 con 0x7f730806c200 2024-01-15T18:07:08.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.329+0000 7f730da00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730815b600 con 0x7f7308143da0 2024-01-15T18:07:08.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.329+0000 7f730da00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f730813c820 con 0x7f73080cad40 2024-01-15T18:07:08.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.333+0000 7f7306a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730806c200 0x7f7308158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:08.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.333+0000 7f7306a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730806c200 0x7f7308158560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34698/0 (socket says 172.21.15.27:34698) 2024-01-15T18:07:08.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.333+0000 7f7306a00640 1 -- 172.21.15.27:0/2648857750 learned_addr learned my addr 172.21.15.27:0/2648857750 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:08.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.333+0000 7f7306000640 1 --2- 172.21.15.27:0/2648857750 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f73080cad40 0x7f730815e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:08.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.333+0000 7f7307400640 1 --2- 172.21.15.27:0/2648857750 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7308143da0 0x7f7308159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:08.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.333+0000 7f7306a00640 1 -- 172.21.15.27:0/2648857750 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f73080cad40 msgr2=0x7f730815e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:08.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.333+0000 7f7306a00640 1 --2- 172.21.15.27:0/2648857750 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f73080cad40 0x7f730815e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.333+0000 7f7306a00640 1 -- 172.21.15.27:0/2648857750 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7308143da0 msgr2=0x7f7308159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:08.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.333+0000 7f7306a00640 1 --2- 172.21.15.27:0/2648857750 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7308143da0 0x7f7308159360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.333+0000 7f7306a00640 1 -- 172.21.15.27:0/2648857750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f00094d0 con 0x7f730806c200 2024-01-15T18:07:08.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.333+0000 7f7306a00640 1 --2- 172.21.15.27:0/2648857750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730806c200 0x7f7308158560 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f72f800c820 tx=0x7f72f800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:08.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.333+0000 7f72fea00640 1 -- 172.21.15.27:0/2648857750 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f8007a30 con 0x7f730806c200 2024-01-15T18:07:08.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.333+0000 7f72fea00640 1 -- 172.21.15.27:0/2648857750 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f8007b90 con 0x7f730806c200 2024-01-15T18:07:08.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.333+0000 7f72fea00640 1 -- 172.21.15.27:0/2648857750 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f800ce10 con 0x7f730806c200 2024-01-15T18:07:08.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.333+0000 7f730da00640 1 -- 172.21.15.27:0/2648857750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f730813cd70 con 0x7f730806c200 2024-01-15T18:07:08.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.333+0000 7f730da00640 1 -- 172.21.15.27:0/2648857750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f730813ca90 con 0x7f730806c200 2024-01-15T18:07:08.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.333+0000 7f730da00640 1 -- 172.21.15.27:0/2648857750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f730804e850 con 0x7f730806c200 2024-01-15T18:07:08.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.333+0000 7f72fea00640 1 -- 172.21.15.27:0/2648857750 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f72f80113f0 con 0x7f730806c200 2024-01-15T18:07:08.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.337+0000 7f72fea00640 1 --2- 172.21.15.27:0/2648857750 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f72e0066b30 0x7f72e0068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:08.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.337+0000 7f72fea00640 1 -- 172.21.15.27:0/2648857750 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f72f8084820 con 0x7f730806c200 2024-01-15T18:07:08.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.337+0000 7f7306000640 1 --2- 172.21.15.27:0/2648857750 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f72e0066b30 0x7f72e0068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:08.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.337+0000 7f7306000640 1 --2- 172.21.15.27:0/2648857750 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f72e0066b30 0x7f72e0068ff0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f72f00ae000 tx=0x7f72f00b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:08.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.341+0000 7f72fea00640 1 -- 172.21.15.27:0/2648857750 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f72f80518f0 con 0x7f730806c200 2024-01-15T18:07:08.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.509+0000 7f730da00640 1 -- 172.21.15.27:0/2648857750 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f73080008e0 con 0x7f730806c200 2024-01-15T18:07:08.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.509+0000 7f72fea00640 1 -- 172.21.15.27:0/2648857750 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v37) v1 ==== 80+0+122 (secure 0 0 0) 0x7f72f8054f10 con 0x7f730806c200 2024-01-15T18:07:08.516 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6834/2187944510 2024-01-16T18:07:03.967765+0000 2024-01-15T18:07:08.516 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6832/2187944510 2024-01-16T18:07:03.967765+0000 2024-01-15T18:07:08.516 INFO:teuthology.orchestra.run.smithi027.stderr:listed 2 entries 2024-01-15T18:07:08.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.509+0000 7f730da00640 1 -- 172.21.15.27:0/2648857750 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f72e0066b30 msgr2=0x7f72e0068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:08.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.509+0000 7f730da00640 1 --2- 172.21.15.27:0/2648857750 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f72e0066b30 0x7f72e0068ff0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f72f00ae000 tx=0x7f72f00b5040 comp rx=0 tx=0).stop 2024-01-15T18:07:08.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.509+0000 7f730da00640 1 -- 172.21.15.27:0/2648857750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730806c200 msgr2=0x7f7308158560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:08.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.509+0000 7f730da00640 1 --2- 172.21.15.27:0/2648857750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730806c200 0x7f7308158560 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f72f800c820 tx=0x7f72f800cce0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.509+0000 7f730da00640 1 -- 172.21.15.27:0/2648857750 shutdown_connections 2024-01-15T18:07:08.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.509+0000 7f730da00640 1 --2- 172.21.15.27:0/2648857750 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f72e0066b30 0x7f72e0068ff0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.509+0000 7f730da00640 1 --2- 172.21.15.27:0/2648857750 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7308143da0 0x7f7308159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.509+0000 7f730da00640 1 --2- 172.21.15.27:0/2648857750 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f73080cad40 0x7f730815e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.509+0000 7f730da00640 1 --2- 172.21.15.27:0/2648857750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730806c200 0x7f7308158560 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.509+0000 7f730da00640 1 -- 172.21.15.27:0/2648857750 >> 172.21.15.27:0/2648857750 conn(0x7f73080b0850 msgr2=0x7f73080cd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:08.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.509+0000 7f730da00640 1 -- 172.21.15.27:0/2648857750 shutdown_connections 2024-01-15T18:07:08.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.509+0000 7f730da00640 1 -- 172.21.15.27:0/2648857750 wait complete. 2024-01-15T18:07:08.537 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-01-15T18:07:08.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:08.538 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-15T18:07:08.560 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T18:07:08.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:08.560 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-15T18:07:08.560 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[36928]: fuse finished with error 0 and tester_r 0 2024-01-15T18:07:08.746 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:07:08.746 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:07:08.756 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:07:08.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.921+0000 7fe6d0600640 1 Processor -- start 2024-01-15T18:07:08.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.921+0000 7fe6d0600640 1 -- start start 2024-01-15T18:07:08.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.925+0000 7fe6d0600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe6c8146a90 0x7fe6c8160da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:08.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.925+0000 7fe6d0600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe6c80b2ea0 0x7fe6c815dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:08.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.925+0000 7fe6d0600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c815db50 con 0x7fe6c815e520 2024-01-15T18:07:08.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.925+0000 7fe6d0600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6c8146e70 con 0x7fe6c8146a90 2024-01-15T18:07:08.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.925+0000 7fe6d0600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c81615f0 con 0x7fe6c80b2ea0 2024-01-15T18:07:08.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.925+0000 7fe6cdc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe6c8146a90 0x7fe6c8160da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:08.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.925+0000 7fe6cdc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe6c8146a90 0x7fe6c8160da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34712/0 (socket says 172.21.15.27:34712) 2024-01-15T18:07:08.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.925+0000 7fe6ce600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe6c815e520 0x7fe6c806a4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57602/0 (socket says 172.21.15.27:57602) 2024-01-15T18:07:08.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.925+0000 7fe6cdc00640 1 -- 172.21.15.27:0/3863820533 learned_addr learned my addr 172.21.15.27:0/3863820533 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:08.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.925+0000 7fe6cdc00640 1 -- 172.21.15.27:0/3863820533 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe6c80b2ea0 msgr2=0x7fe6c815dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:08.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.925+0000 7fe6cd200640 1 --2- 172.21.15.27:0/3863820533 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe6c80b2ea0 0x7fe6c815dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:08.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.925+0000 7fe6c7e00640 1 -- 172.21.15.27:0/3863820533 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679854499 0 0) 0x7fe6c815db50 con 0x7fe6c815e520 2024-01-15T18:07:08.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.925+0000 7fe6cdc00640 1 --2- 172.21.15.27:0/3863820533 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe6c80b2ea0 0x7fe6c815dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.925+0000 7fe6cdc00640 1 -- 172.21.15.27:0/3863820533 >> v1:172.21.15.27:6789/0 conn(0x7fe6c815e520 legacy=0x7fe6c806a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:08.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.925+0000 7fe6cdc00640 1 -- 172.21.15.27:0/3863820533 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c81618d0 con 0x7fe6c8146a90 2024-01-15T18:07:08.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.925+0000 7fe6cd200640 1 --2- 172.21.15.27:0/3863820533 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe6c80b2ea0 0x7fe6c815dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:07:08.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.925+0000 7fe6cdc00640 1 --2- 172.21.15.27:0/3863820533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6c8146a90 0x7fe6c8160da0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009870 tx=0x7fe6b801c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f257baae71e70ca9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:08.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.929+0000 7fe6c7e00640 1 -- 172.21.15.27:0/3863820533 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b80aa430 con 0x7fe6c8146a90 2024-01-15T18:07:08.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.929+0000 7fe6c7e00640 1 -- 172.21.15.27:0/3863820533 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b80aa590 con 0x7fe6c8146a90 2024-01-15T18:07:08.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.929+0000 7fe6c7e00640 1 -- 172.21.15.27:0/3863820533 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b80aa920 con 0x7fe6c8146a90 2024-01-15T18:07:08.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.929+0000 7fe6d0600640 1 -- 172.21.15.27:0/3863820533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6c8146a90 msgr2=0x7fe6c8160da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:08.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.929+0000 7fe6d0600640 1 --2- 172.21.15.27:0/3863820533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6c8146a90 0x7fe6c8160da0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009870 tx=0x7fe6b801c510 comp rx=0 tx=0).stop 2024-01-15T18:07:08.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.929+0000 7fe6d0600640 1 -- 172.21.15.27:0/3863820533 shutdown_connections 2024-01-15T18:07:08.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.929+0000 7fe6d0600640 1 --2- 172.21.15.27:0/3863820533 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe6c80b2ea0 0x7fe6c815dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.929+0000 7fe6d0600640 1 --2- 172.21.15.27:0/3863820533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6c8146a90 0x7fe6c8160da0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.929+0000 7fe6d0600640 1 -- 172.21.15.27:0/3863820533 >> 172.21.15.27:0/3863820533 conn(0x7fe6c80b1ba0 msgr2=0x7fe6c80b1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:08.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.929+0000 7fe6d0600640 1 -- 172.21.15.27:0/3863820533 shutdown_connections 2024-01-15T18:07:08.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.929+0000 7fe6d0600640 1 -- 172.21.15.27:0/3863820533 wait complete. 2024-01-15T18:07:08.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.929+0000 7fe6d0600640 1 Processor -- start 2024-01-15T18:07:08.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.929+0000 7fe6d0600640 1 -- start start 2024-01-15T18:07:08.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.929+0000 7fe6d0600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6c80b2ea0 0x7fe6c815afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:08.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.929+0000 7fe6d0600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe6c8146a90 0x7fe6c813dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:08.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.929+0000 7fe6d0600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe6c815e520 0x7fe6c8143780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:08.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.929+0000 7fe6d0600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c813d890 con 0x7fe6c80b2ea0 2024-01-15T18:07:08.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.929+0000 7fe6d0600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c813d560 con 0x7fe6c8146a90 2024-01-15T18:07:08.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.929+0000 7fe6d0600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c80aca20 con 0x7fe6c815e520 2024-01-15T18:07:08.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.933+0000 7fe6cdc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6c80b2ea0 0x7fe6c815afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:08.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.933+0000 7fe6cdc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6c80b2ea0 0x7fe6c815afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34728/0 (socket says 172.21.15.27:34728) 2024-01-15T18:07:08.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.933+0000 7fe6cdc00640 1 -- 172.21.15.27:0/1018839360 learned_addr learned my addr 172.21.15.27:0/1018839360 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:08.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.933+0000 7fe6cd200640 1 --2- 172.21.15.27:0/1018839360 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe6c8146a90 0x7fe6c813dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:08.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.933+0000 7fe6ce600640 1 --2- 172.21.15.27:0/1018839360 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe6c815e520 0x7fe6c8143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:08.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.933+0000 7fe6cdc00640 1 -- 172.21.15.27:0/1018839360 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe6c815e520 msgr2=0x7fe6c8143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:08.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.933+0000 7fe6cdc00640 1 --2- 172.21.15.27:0/1018839360 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe6c815e520 0x7fe6c8143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.933+0000 7fe6cdc00640 1 -- 172.21.15.27:0/1018839360 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe6c8146a90 msgr2=0x7fe6c813dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:08.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.933+0000 7fe6cdc00640 1 --2- 172.21.15.27:0/1018839360 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe6c8146a90 0x7fe6c813dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.933+0000 7fe6cdc00640 1 -- 172.21.15.27:0/1018839360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b80094d0 con 0x7fe6c80b2ea0 2024-01-15T18:07:08.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.933+0000 7fe6ce600640 1 --2- 172.21.15.27:0/1018839360 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe6c815e520 0x7fe6c8143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:07:08.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.933+0000 7fe6cdc00640 1 --2- 172.21.15.27:0/1018839360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6c80b2ea0 0x7fe6c815afa0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80ac040 tx=0x7fe6b801ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:08.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.933+0000 7fe6c5600640 1 -- 172.21.15.27:0/1018839360 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b80b38c0 con 0x7fe6c80b2ea0 2024-01-15T18:07:08.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.933+0000 7fe6c5600640 1 -- 172.21.15.27:0/1018839360 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b80b3a40 con 0x7fe6c80b2ea0 2024-01-15T18:07:08.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.933+0000 7fe6d0600640 1 -- 172.21.15.27:0/1018839360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6c80acd00 con 0x7fe6c80b2ea0 2024-01-15T18:07:08.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.933+0000 7fe6c5600640 1 -- 172.21.15.27:0/1018839360 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b80b3df0 con 0x7fe6c80b2ea0 2024-01-15T18:07:08.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.933+0000 7fe6d0600640 1 -- 172.21.15.27:0/1018839360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6c80acfe0 con 0x7fe6c80b2ea0 2024-01-15T18:07:08.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.933+0000 7fe6d0600640 1 -- 172.21.15.27:0/1018839360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6c804e850 con 0x7fe6c80b2ea0 2024-01-15T18:07:08.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.933+0000 7fe6c5600640 1 -- 172.21.15.27:0/1018839360 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7fe6b80c6390 con 0x7fe6c80b2ea0 2024-01-15T18:07:08.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.933+0000 7fe6c5600640 1 --2- 172.21.15.27:0/1018839360 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe6a0066b30 0x7fe6a0068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:08.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.937+0000 7fe6c5600640 1 -- 172.21.15.27:0/1018839360 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fe6b811efe0 con 0x7fe6c80b2ea0 2024-01-15T18:07:08.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.937+0000 7fe6cd200640 1 --2- 172.21.15.27:0/1018839360 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe6a0066b30 0x7fe6a0068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:08.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.937+0000 7fe6cd200640 1 --2- 172.21.15.27:0/1018839360 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe6a0066b30 0x7fe6a0068ff0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe6c813e750 tx=0x7fe6b0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:08.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:08.941+0000 7fe6c5600640 1 -- 172.21.15.27:0/1018839360 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe6b80ec170 con 0x7fe6c80b2ea0 2024-01-15T18:07:09.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.101+0000 7fe6d0600640 1 -- 172.21.15.27:0/1018839360 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6c8142040 con 0x7fe6c80b2ea0 2024-01-15T18:07:09.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.101+0000 7fe6c5600640 1 -- 172.21.15.27:0/1018839360 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+12933 (secure 0 0 0) 0x7fe6b80ef790 con 0x7fe6c80b2ea0 2024-01-15T18:07:09.107 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:07:09.108 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":37,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:07:06.979289+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6834/2187944510":"2024-01-16T18:07:03.967765+0000","172.21.15.27:6832/2187944510":"2024-01-16T18:07:03.967765+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:07:09.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.101+0000 7fe6d0600640 1 -- 172.21.15.27:0/1018839360 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe6a0066b30 msgr2=0x7fe6a0068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:09.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.101+0000 7fe6d0600640 1 --2- 172.21.15.27:0/1018839360 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe6a0066b30 0x7fe6a0068ff0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe6c813e750 tx=0x7fe6b0008040 comp rx=0 tx=0).stop 2024-01-15T18:07:09.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.101+0000 7fe6d0600640 1 -- 172.21.15.27:0/1018839360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6c80b2ea0 msgr2=0x7fe6c815afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:09.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.101+0000 7fe6d0600640 1 --2- 172.21.15.27:0/1018839360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6c80b2ea0 0x7fe6c815afa0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80ac040 tx=0x7fe6b801ad50 comp rx=0 tx=0).stop 2024-01-15T18:07:09.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.101+0000 7fe6d0600640 1 -- 172.21.15.27:0/1018839360 shutdown_connections 2024-01-15T18:07:09.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.101+0000 7fe6d0600640 1 --2- 172.21.15.27:0/1018839360 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe6a0066b30 0x7fe6a0068ff0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.101+0000 7fe6d0600640 1 --2- 172.21.15.27:0/1018839360 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe6c815e520 0x7fe6c8143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.101+0000 7fe6d0600640 1 --2- 172.21.15.27:0/1018839360 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe6c8146a90 0x7fe6c813dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.101+0000 7fe6d0600640 1 --2- 172.21.15.27:0/1018839360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6c80b2ea0 0x7fe6c815afa0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.101+0000 7fe6d0600640 1 -- 172.21.15.27:0/1018839360 >> 172.21.15.27:0/1018839360 conn(0x7fe6c80b1ba0 msgr2=0x7fe6c80cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:09.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.101+0000 7fe6d0600640 1 -- 172.21.15.27:0/1018839360 shutdown_connections 2024-01-15T18:07:09.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.101+0000 7fe6d0600640 1 -- 172.21.15.27:0/1018839360 wait complete. 2024-01-15T18:07:09.130 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:07:09.131 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:07:09.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.265+0000 7f7d4f400640 1 Processor -- start 2024-01-15T18:07:09.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.265+0000 7f7d4f400640 1 -- start start 2024-01-15T18:07:09.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d4f400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f7d480715f0 0x7f7d48164a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:09.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d4f400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4815a610 con 0x7f7d4815b100 2024-01-15T18:07:09.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d4f400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4815e3c0 con 0x7f7d4815ad20 2024-01-15T18:07:09.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d4f400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4815e6a0 con 0x7f7d480715f0 2024-01-15T18:07:09.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d47e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7d4815b100 0x7f7d48070ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57616/0 (socket says 172.21.15.27:57616) 2024-01-15T18:07:09.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d47e00640 1 -- 172.21.15.27:0/4155620711 learned_addr learned my addr 172.21.15.27:0/4155620711 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:09.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d4d400640 1 --2- 172.21.15.27:0/4155620711 >> v2:172.21.15.93:3300/0 conn(0x7f7d480715f0 0x7f7d48164a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:09.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d47400640 1 -- 172.21.15.27:0/4155620711 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480273286 0 0) 0x7f7d4815a610 con 0x7f7d4815b100 2024-01-15T18:07:09.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d47400640 1 -- 172.21.15.27:0/4155620711 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d28003530 con 0x7f7d4815b100 2024-01-15T18:07:09.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d47400640 1 -- 172.21.15.27:0/4155620711 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63561205 0 0) 0x7f7d4815e3c0 con 0x7f7d4815ad20 2024-01-15T18:07:09.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d47400640 1 -- 172.21.15.27:0/4155620711 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d4815a610 con 0x7f7d4815ad20 2024-01-15T18:07:09.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d47400640 1 -- 172.21.15.27:0/4155620711 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1358897206 0 0) 0x7f7d28003530 con 0x7f7d4815b100 2024-01-15T18:07:09.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d47400640 1 -- 172.21.15.27:0/4155620711 >> v2:172.21.15.93:3300/0 conn(0x7f7d480715f0 msgr2=0x7f7d48164a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:09.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d47400640 1 --2- 172.21.15.27:0/4155620711 >> v2:172.21.15.93:3300/0 conn(0x7f7d480715f0 0x7f7d48164a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d47400640 1 -- 172.21.15.27:0/4155620711 >> v1:172.21.15.93:6789/0 conn(0x7f7d4815ad20 legacy=0x7f7d4815d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:09.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d47400640 1 -- 172.21.15.27:0/4155620711 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4815b6e0 con 0x7f7d4815b100 2024-01-15T18:07:09.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d47400640 1 -- 172.21.15.27:0/4155620711 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f7d30001680 con 0x7f7d4815b100 2024-01-15T18:07:09.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d47400640 1 -- 172.21.15.27:0/4155620711 >> v1:172.21.15.27:6789/0 conn(0x7f7d4815b100 legacy=0x7f7d48070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:09.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d47400640 1 --2- 172.21.15.27:0/4155620711 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7d28003bd0 0x7f7d28003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:09.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d47400640 1 --2- 172.21.15.27:0/4155620711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d280047d0 0x7f7d28004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:09.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d47400640 1 --2- 172.21.15.27:0/4155620711 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7d28005250 0x7f7d2800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:09.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d47400640 1 -- 172.21.15.27:0/4155620711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4815e6a0 con 0x7f7d280047d0 2024-01-15T18:07:09.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d47400640 1 -- 172.21.15.27:0/4155620711 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2800c160 con 0x7f7d28005250 2024-01-15T18:07:09.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d47400640 1 -- 172.21.15.27:0/4155620711 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2800c320 con 0x7f7d28003bd0 2024-01-15T18:07:09.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d47e00640 1 --2- 172.21.15.27:0/4155620711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d280047d0 0x7f7d28004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:09.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d4ca00640 1 --2- 172.21.15.27:0/4155620711 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7d28003bd0 0x7f7d28003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:09.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d4d400640 1 --2- 172.21.15.27:0/4155620711 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7d28005250 0x7f7d2800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:09.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d47e00640 1 -- 172.21.15.27:0/4155620711 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7d28003bd0 msgr2=0x7f7d28003fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:09.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d47e00640 1 --2- 172.21.15.27:0/4155620711 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7d28003bd0 0x7f7d28003fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d47e00640 1 -- 172.21.15.27:0/4155620711 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7d28005250 msgr2=0x7f7d2800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:09.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d47e00640 1 --2- 172.21.15.27:0/4155620711 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7d28005250 0x7f7d2800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d47e00640 1 -- 172.21.15.27:0/4155620711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d28003530 con 0x7f7d280047d0 2024-01-15T18:07:09.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.269+0000 7f7d4d400640 1 -- 172.21.15.27:0/4155620711 reap_dead start 2024-01-15T18:07:09.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.273+0000 7f7d47e00640 1 --2- 172.21.15.27:0/4155620711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d280047d0 0x7f7d28004c50 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f7d30007fa0 tx=0x7f7d3001cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ce63b8aeb071f8a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:09.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.273+0000 7f7d47400640 1 -- 172.21.15.27:0/4155620711 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d30005ae0 con 0x7f7d280047d0 2024-01-15T18:07:09.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.273+0000 7f7d47400640 1 -- 172.21.15.27:0/4155620711 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d30005c40 con 0x7f7d280047d0 2024-01-15T18:07:09.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.273+0000 7f7d47400640 1 -- 172.21.15.27:0/4155620711 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d30007290 con 0x7f7d280047d0 2024-01-15T18:07:09.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.273+0000 7f7d4f400640 1 -- 172.21.15.27:0/4155620711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d280047d0 msgr2=0x7f7d28004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:09.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.273+0000 7f7d4f400640 1 --2- 172.21.15.27:0/4155620711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d280047d0 0x7f7d28004c50 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f7d30007fa0 tx=0x7f7d3001cab0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.273+0000 7f7d4f400640 1 -- 172.21.15.27:0/4155620711 shutdown_connections 2024-01-15T18:07:09.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.273+0000 7f7d4f400640 1 --2- 172.21.15.27:0/4155620711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d280047d0 0x7f7d28004c50 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.273+0000 7f7d4f400640 1 -- 172.21.15.27:0/4155620711 >> 172.21.15.27:0/4155620711 conn(0x7f7d480cb700 msgr2=0x7f7d480cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:09.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.273+0000 7f7d4f400640 1 -- 172.21.15.27:0/4155620711 shutdown_connections 2024-01-15T18:07:09.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.273+0000 7f7d4f400640 1 -- 172.21.15.27:0/4155620711 wait complete. 2024-01-15T18:07:09.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.277+0000 7f7d4f400640 1 Processor -- start 2024-01-15T18:07:09.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.277+0000 7f7d4f400640 1 -- start start 2024-01-15T18:07:09.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.277+0000 7f7d4f400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7d280047d0 0x7f7d480ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:09.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.277+0000 7f7d4f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d481561d0 0x7f7d480aa1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:09.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.277+0000 7f7d4f400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7d48156650 0x7f7d480aa6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:09.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.277+0000 7f7d4f400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d480a8470 con 0x7f7d481561d0 2024-01-15T18:07:09.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.277+0000 7f7d4f400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d480a9260 con 0x7f7d280047d0 2024-01-15T18:07:09.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.277+0000 7f7d4f400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d480a8650 con 0x7f7d48156650 2024-01-15T18:07:09.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.277+0000 7f7d47e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d481561d0 0x7f7d480aa1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:09.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.277+0000 7f7d47e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d481561d0 0x7f7d480aa1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34748/0 (socket says 172.21.15.27:34748) 2024-01-15T18:07:09.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.277+0000 7f7d47e00640 1 -- 172.21.15.27:0/356101990 learned_addr learned my addr 172.21.15.27:0/356101990 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:09.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.277+0000 7f7d4d400640 1 --2- 172.21.15.27:0/356101990 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7d48156650 0x7f7d480aa6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:09.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.277+0000 7f7d4ca00640 1 --2- 172.21.15.27:0/356101990 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7d280047d0 0x7f7d480ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:09.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.277+0000 7f7d47e00640 1 -- 172.21.15.27:0/356101990 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7d48156650 msgr2=0x7f7d480aa6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:09.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.277+0000 7f7d47e00640 1 --2- 172.21.15.27:0/356101990 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7d48156650 0x7f7d480aa6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.277+0000 7f7d47e00640 1 -- 172.21.15.27:0/356101990 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7d280047d0 msgr2=0x7f7d480ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:09.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.277+0000 7f7d47e00640 1 --2- 172.21.15.27:0/356101990 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7d280047d0 0x7f7d480ca2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.277+0000 7f7d47e00640 1 -- 172.21.15.27:0/356101990 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d480a8930 con 0x7f7d481561d0 2024-01-15T18:07:09.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.277+0000 7f7d47e00640 1 --2- 172.21.15.27:0/356101990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d481561d0 0x7f7d480aa1a0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f7d3001c8e0 tx=0x7f7d28005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:09.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.277+0000 7f7d44c00640 1 -- 172.21.15.27:0/356101990 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d300b4070 con 0x7f7d481561d0 2024-01-15T18:07:09.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.277+0000 7f7d44c00640 1 -- 172.21.15.27:0/356101990 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d30005e70 con 0x7f7d481561d0 2024-01-15T18:07:09.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.277+0000 7f7d44c00640 1 -- 172.21.15.27:0/356101990 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d30001680 con 0x7f7d481561d0 2024-01-15T18:07:09.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.277+0000 7f7d4f400640 1 -- 172.21.15.27:0/356101990 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d480a8c10 con 0x7f7d481561d0 2024-01-15T18:07:09.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.277+0000 7f7d4f400640 1 -- 172.21.15.27:0/356101990 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d48155e90 con 0x7f7d481561d0 2024-01-15T18:07:09.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.281+0000 7f7d4f400640 1 -- 172.21.15.27:0/356101990 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d480a93f0 con 0x7f7d481561d0 2024-01-15T18:07:09.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.281+0000 7f7d44c00640 1 -- 172.21.15.27:0/356101990 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f7d30007040 con 0x7f7d481561d0 2024-01-15T18:07:09.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.281+0000 7f7d44c00640 1 --2- 172.21.15.27:0/356101990 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7d20066b60 0x7f7d20069020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:09.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.281+0000 7f7d44c00640 1 -- 172.21.15.27:0/356101990 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f7d3011d740 con 0x7f7d481561d0 2024-01-15T18:07:09.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.281+0000 7f7d4ca00640 1 --2- 172.21.15.27:0/356101990 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7d20066b60 0x7f7d20069020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:09.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.281+0000 7f7d4ca00640 1 --2- 172.21.15.27:0/356101990 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7d20066b60 0x7f7d20069020 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7d48157e50 tx=0x7f7d38009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:09.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.285+0000 7f7d44c00640 1 -- 172.21.15.27:0/356101990 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7d300ea8f0 con 0x7f7d481561d0 2024-01-15T18:07:09.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.445+0000 7f7d4f400640 1 -- 172.21.15.27:0/356101990 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7d4804e850 con 0x7f7d481561d0 2024-01-15T18:07:09.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.445+0000 7f7d44c00640 1 -- 172.21.15.27:0/356101990 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7d300bf0a0 con 0x7f7d481561d0 2024-01-15T18:07:09.451 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:07:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.445+0000 7f7d4f400640 1 -- 172.21.15.27:0/356101990 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7d20066b60 msgr2=0x7f7d20069020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.445+0000 7f7d4f400640 1 --2- 172.21.15.27:0/356101990 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7d20066b60 0x7f7d20069020 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7d48157e50 tx=0x7f7d38009040 comp rx=0 tx=0).stop 2024-01-15T18:07:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.445+0000 7f7d4f400640 1 -- 172.21.15.27:0/356101990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d481561d0 msgr2=0x7f7d480aa1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.445+0000 7f7d4f400640 1 --2- 172.21.15.27:0/356101990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d481561d0 0x7f7d480aa1a0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f7d3001c8e0 tx=0x7f7d28005800 comp rx=0 tx=0).stop 2024-01-15T18:07:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.445+0000 7f7d4f400640 1 -- 172.21.15.27:0/356101990 shutdown_connections 2024-01-15T18:07:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.445+0000 7f7d4f400640 1 --2- 172.21.15.27:0/356101990 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7d20066b60 0x7f7d20069020 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.445+0000 7f7d4f400640 1 --2- 172.21.15.27:0/356101990 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7d48156650 0x7f7d480aa6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.445+0000 7f7d4f400640 1 --2- 172.21.15.27:0/356101990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d481561d0 0x7f7d480aa1a0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.445+0000 7f7d4f400640 1 --2- 172.21.15.27:0/356101990 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7d280047d0 0x7f7d480ca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.445+0000 7f7d4f400640 1 -- 172.21.15.27:0/356101990 >> 172.21.15.27:0/356101990 conn(0x7f7d480cb700 msgr2=0x7f7d480ce980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.445+0000 7f7d4f400640 1 -- 172.21.15.27:0/356101990 shutdown_connections 2024-01-15T18:07:09.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.445+0000 7f7d4f400640 1 -- 172.21.15.27:0/356101990 wait complete. 2024-01-15T18:07:09.474 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:07:09.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.609+0000 7fc529200640 1 Processor -- start 2024-01-15T18:07:09.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.609+0000 7fc529200640 1 -- start start 2024-01-15T18:07:09.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.609+0000 7fc529200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fc52415d2a0 0x7fc5240cfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:09.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.609+0000 7fc529200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc52415d680 0x7fc5240cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:09.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.609+0000 7fc529200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5240b3cd0 con 0x7fc5240cdc40 2024-01-15T18:07:09.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.609+0000 7fc529200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5240d0260 con 0x7fc52415d680 2024-01-15T18:07:09.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.609+0000 7fc529200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5240cb670 con 0x7fc52415d2a0 2024-01-15T18:07:09.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.609+0000 7fc521400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc52415d680 0x7fc5240cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:09.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.609+0000 7fc522800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc5240cdc40 0x7fc52406b810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57620/0 (socket says 172.21.15.27:57620) 2024-01-15T18:07:09.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.609+0000 7fc521400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc52415d680 0x7fc5240cd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34760/0 (socket says 172.21.15.27:34760) 2024-01-15T18:07:09.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.609+0000 7fc522800640 1 -- 172.21.15.27:0/2088422427 learned_addr learned my addr 172.21.15.27:0/2088422427 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:09.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.609+0000 7fc521e00640 1 --2- 172.21.15.27:0/2088422427 >> v2:172.21.15.93:3300/0 conn(0x7fc52415d2a0 0x7fc5240cfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.609+0000 7fc521400640 1 -- 172.21.15.27:0/2088422427 >> v2:172.21.15.93:3300/0 conn(0x7fc52415d2a0 msgr2=0x7fc5240cfbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.609+0000 7fc521400640 1 --2- 172.21.15.27:0/2088422427 >> v2:172.21.15.93:3300/0 conn(0x7fc52415d2a0 0x7fc5240cfbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.609+0000 7fc521400640 1 -- 172.21.15.27:0/2088422427 >> v1:172.21.15.27:6789/0 conn(0x7fc5240cdc40 legacy=0x7fc52406b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.609+0000 7fc521400640 1 -- 172.21.15.27:0/2088422427 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5240cb950 con 0x7fc52415d680 2024-01-15T18:07:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.609+0000 7fc520a00640 1 -- 172.21.15.27:0/2088422427 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713261663 0 0) 0x7fc5240b3cd0 con 0x7fc5240cdc40 2024-01-15T18:07:09.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.613+0000 7fc521400640 1 --2- 172.21.15.27:0/2088422427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc52415d680 0x7fc5240cd700 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc524071670 tx=0x7fc50c01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d30ff733ae20385f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:09.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.613+0000 7fc520a00640 1 -- 172.21.15.27:0/2088422427 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c0ad680 con 0x7fc52415d680 2024-01-15T18:07:09.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.613+0000 7fc520a00640 1 -- 172.21.15.27:0/2088422427 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc50c0ad7e0 con 0x7fc52415d680 2024-01-15T18:07:09.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.613+0000 7fc520a00640 1 -- 172.21.15.27:0/2088422427 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c0adb70 con 0x7fc52415d680 2024-01-15T18:07:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.613+0000 7fc529200640 1 -- 172.21.15.27:0/2088422427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc52415d680 msgr2=0x7fc5240cd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.613+0000 7fc529200640 1 --2- 172.21.15.27:0/2088422427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc52415d680 0x7fc5240cd700 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc524071670 tx=0x7fc50c01c3a0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.613+0000 7fc529200640 1 -- 172.21.15.27:0/2088422427 shutdown_connections 2024-01-15T18:07:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.613+0000 7fc529200640 1 --2- 172.21.15.27:0/2088422427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc52415d680 0x7fc5240cd700 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.613+0000 7fc529200640 1 --2- 172.21.15.27:0/2088422427 >> v2:172.21.15.93:3300/0 conn(0x7fc52415d2a0 0x7fc5240cfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.613+0000 7fc529200640 1 -- 172.21.15.27:0/2088422427 >> 172.21.15.27:0/2088422427 conn(0x7fc5240d0e80 msgr2=0x7fc5240d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.613+0000 7fc529200640 1 -- 172.21.15.27:0/2088422427 shutdown_connections 2024-01-15T18:07:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.613+0000 7fc529200640 1 -- 172.21.15.27:0/2088422427 wait complete. 2024-01-15T18:07:09.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.617+0000 7fc529200640 1 Processor -- start 2024-01-15T18:07:09.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.617+0000 7fc529200640 1 -- start start 2024-01-15T18:07:09.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.617+0000 7fc529200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5240cdc40 0x7fc5240c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:09.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.617+0000 7fc529200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc52415d2a0 0x7fc5240c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:09.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.617+0000 7fc529200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc52415d680 0x7fc5240c9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:09.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.617+0000 7fc529200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5241557a0 con 0x7fc5240cdc40 2024-01-15T18:07:09.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.617+0000 7fc529200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc524158be0 con 0x7fc52415d2a0 2024-01-15T18:07:09.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.617+0000 7fc529200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc524158900 con 0x7fc52415d680 2024-01-15T18:07:09.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.617+0000 7fc521e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5240cdc40 0x7fc5240c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:09.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.617+0000 7fc521e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5240cdc40 0x7fc5240c9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34766/0 (socket says 172.21.15.27:34766) 2024-01-15T18:07:09.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.617+0000 7fc521e00640 1 -- 172.21.15.27:0/2741509932 learned_addr learned my addr 172.21.15.27:0/2741509932 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:09.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.617+0000 7fc521400640 1 --2- 172.21.15.27:0/2741509932 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc52415d2a0 0x7fc5240c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:09.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.617+0000 7fc522800640 1 --2- 172.21.15.27:0/2741509932 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc52415d680 0x7fc5240c9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:09.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.617+0000 7fc521e00640 1 -- 172.21.15.27:0/2741509932 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc52415d680 msgr2=0x7fc5240c9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:09.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.617+0000 7fc521e00640 1 --2- 172.21.15.27:0/2741509932 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc52415d680 0x7fc5240c9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.617+0000 7fc521e00640 1 -- 172.21.15.27:0/2741509932 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc52415d2a0 msgr2=0x7fc5240c95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:09.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.617+0000 7fc521e00640 1 --2- 172.21.15.27:0/2741509932 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc52415d2a0 0x7fc5240c95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.617+0000 7fc521e00640 1 -- 172.21.15.27:0/2741509932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc50c009420 con 0x7fc5240cdc40 2024-01-15T18:07:09.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.617+0000 7fc521e00640 1 --2- 172.21.15.27:0/2741509932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5240cdc40 0x7fc5240c9080 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc51400d800 tx=0x7fc51400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:09.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.617+0000 7fc51a000640 1 -- 172.21.15.27:0/2741509932 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51400b6c0 con 0x7fc5240cdc40 2024-01-15T18:07:09.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.617+0000 7fc51a000640 1 -- 172.21.15.27:0/2741509932 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc51400b820 con 0x7fc5240cdc40 2024-01-15T18:07:09.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.617+0000 7fc529200640 1 -- 172.21.15.27:0/2741509932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5240c85a0 con 0x7fc5240cdc40 2024-01-15T18:07:09.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.617+0000 7fc51a000640 1 -- 172.21.15.27:0/2741509932 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51400bb90 con 0x7fc5240cdc40 2024-01-15T18:07:09.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.617+0000 7fc529200640 1 -- 172.21.15.27:0/2741509932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5240c8b60 con 0x7fc5240cdc40 2024-01-15T18:07:09.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.621+0000 7fc529200640 1 -- 172.21.15.27:0/2741509932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc52404e850 con 0x7fc5240cdc40 2024-01-15T18:07:09.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.621+0000 7fc51a000640 1 -- 172.21.15.27:0/2741509932 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7fc51400bcf0 con 0x7fc5240cdc40 2024-01-15T18:07:09.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.621+0000 7fc51a000640 1 --2- 172.21.15.27:0/2741509932 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc4fc066b60 0x7fc4fc069020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:09.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.621+0000 7fc51a000640 1 -- 172.21.15.27:0/2741509932 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fc514084990 con 0x7fc5240cdc40 2024-01-15T18:07:09.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.621+0000 7fc521400640 1 --2- 172.21.15.27:0/2741509932 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc4fc066b60 0x7fc4fc069020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:09.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.621+0000 7fc521400640 1 --2- 172.21.15.27:0/2741509932 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc4fc066b60 0x7fc4fc069020 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc50c0af000 tx=0x7fc50c0a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:09.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.625+0000 7fc51a000640 1 -- 172.21.15.27:0/2741509932 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc514051bc0 con 0x7fc5240cdc40 2024-01-15T18:07:09.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.797+0000 7fc529200640 1 -- 172.21.15.27:0/2741509932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc5241605d0 con 0x7fc5240cdc40 2024-01-15T18:07:09.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.797+0000 7fc51a000640 1 -- 172.21.15.27:0/2741509932 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v37) v1 ==== 80+0+122 (secure 0 0 0) 0x7fc5140551e0 con 0x7fc5240cdc40 2024-01-15T18:07:09.803 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6834/2187944510 2024-01-16T18:07:03.967765+0000 2024-01-15T18:07:09.804 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6832/2187944510 2024-01-16T18:07:03.967765+0000 2024-01-15T18:07:09.807 INFO:teuthology.orchestra.run.smithi027.stderr:listed 2 entries 2024-01-15T18:07:09.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.797+0000 7fc529200640 1 -- 172.21.15.27:0/2741509932 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc4fc066b60 msgr2=0x7fc4fc069020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:09.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.797+0000 7fc529200640 1 --2- 172.21.15.27:0/2741509932 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc4fc066b60 0x7fc4fc069020 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc50c0af000 tx=0x7fc50c0a7000 comp rx=0 tx=0).stop 2024-01-15T18:07:09.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.797+0000 7fc529200640 1 -- 172.21.15.27:0/2741509932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5240cdc40 msgr2=0x7fc5240c9080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:09.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.797+0000 7fc529200640 1 --2- 172.21.15.27:0/2741509932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5240cdc40 0x7fc5240c9080 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc51400d800 tx=0x7fc51400dcc0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.797+0000 7fc529200640 1 -- 172.21.15.27:0/2741509932 shutdown_connections 2024-01-15T18:07:09.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.797+0000 7fc529200640 1 --2- 172.21.15.27:0/2741509932 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc4fc066b60 0x7fc4fc069020 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.797+0000 7fc529200640 1 --2- 172.21.15.27:0/2741509932 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc52415d680 0x7fc5240c9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.797+0000 7fc529200640 1 --2- 172.21.15.27:0/2741509932 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc52415d2a0 0x7fc5240c95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.797+0000 7fc529200640 1 --2- 172.21.15.27:0/2741509932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5240cdc40 0x7fc5240c9080 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:09.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.797+0000 7fc529200640 1 -- 172.21.15.27:0/2741509932 >> 172.21.15.27:0/2741509932 conn(0x7fc5240d0e80 msgr2=0x7fc524159a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:09.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.797+0000 7fc529200640 1 -- 172.21.15.27:0/2741509932 shutdown_connections 2024-01-15T18:07:09.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:09.797+0000 7fc529200640 1 -- 172.21.15.27:0/2741509932 wait complete. 2024-01-15T18:07:09.825 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi093.front.sepia.ceph.com... 2024-01-15T18:07:09.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:09.826 DEBUG:teuthology.orchestra.run.smithi093:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-15T18:07:09.846 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:ceph-fuse[33287]: fuse finished with error 0 and tester_r 0 2024-01-15T18:07:09.849 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T18:07:09.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:09.849 DEBUG:teuthology.orchestra.run.smithi093:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-15T18:07:10.017 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:07:10.017 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:07:10.026 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354580k,nr_inodes=4088645,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9249 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:07:10.026 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T18:07:10.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:10.026 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-15T18:07:10.077 INFO:teuthology.orchestra.run.smithi027.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-15T18:07:10.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:07:10.079 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:07:10.079 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:07:10.087 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354484k,nr_inodes=4088621,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8403 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:07:10.088 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T18:07:10.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:10.088 DEBUG:teuthology.orchestra.run.smithi093:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-15T18:07:10.137 INFO:teuthology.orchestra.run.smithi093.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-15T18:07:10.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:07:10.138 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:07:10.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.273+0000 7f3eafc00640 1 Processor -- start 2024-01-15T18:07:10.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.273+0000 7f3eafc00640 1 -- start start 2024-01-15T18:07:10.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.273+0000 7f3eafc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ea80b2390 0x7f3ea80cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:10.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.273+0000 7f3eafc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3ea80cdf90 0x7f3ea80b0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:10.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.273+0000 7f3eafc00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea80b2770 con 0x7f3ea806bb80 2024-01-15T18:07:10.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.273+0000 7f3eafc00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ea806c060 con 0x7f3ea80cdf90 2024-01-15T18:07:10.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.273+0000 7f3eafc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea806c340 con 0x7f3ea80b2390 2024-01-15T18:07:10.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.273+0000 7f3eadc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3ea80cdf90 0x7f3ea80b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:10.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.273+0000 7f3eadc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3ea80cdf90 0x7f3ea80b0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34782/0 (socket says 172.21.15.27:34782) 2024-01-15T18:07:10.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.273+0000 7f3eadc00640 1 -- 172.21.15.27:0/408009268 learned_addr learned my addr 172.21.15.27:0/408009268 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:10.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.273+0000 7f3ead200640 1 --2- 172.21.15.27:0/408009268 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ea80b2390 0x7f3ea80cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:10.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.273+0000 7f3eadc00640 1 -- 172.21.15.27:0/408009268 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ea80b2390 msgr2=0x7f3ea80cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:10.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.273+0000 7f3eadc00640 1 --2- 172.21.15.27:0/408009268 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ea80b2390 0x7f3ea80cf780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:10.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.273+0000 7f3eadc00640 1 -- 172.21.15.27:0/408009268 >> v1:172.21.15.93:6789/0 conn(0x7f3ea806bb80 legacy=0x7f3ea80cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:10.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.273+0000 7f3eadc00640 1 -- 172.21.15.27:0/408009268 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea806c620 con 0x7f3ea80cdf90 2024-01-15T18:07:10.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.273+0000 7f3ea7400640 1 -- 172.21.15.27:0/408009268 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367022717 0 0) 0x7f3ea80b2770 con 0x7f3ea806bb80 2024-01-15T18:07:10.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.277+0000 7f3eadc00640 1 --2- 172.21.15.27:0/408009268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ea80cdf90 0x7f3ea80b0880 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0009e50 tx=0x7f3ea001c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6fc30f131f52752 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:10.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.277+0000 7f3ea7400640 1 -- 172.21.15.27:0/408009268 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea00a58a0 con 0x7f3ea80cdf90 2024-01-15T18:07:10.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.277+0000 7f3ea7400640 1 -- 172.21.15.27:0/408009268 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea00a5a00 con 0x7f3ea80cdf90 2024-01-15T18:07:10.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.277+0000 7f3ea7400640 1 -- 172.21.15.27:0/408009268 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea00a5d90 con 0x7f3ea80cdf90 2024-01-15T18:07:10.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.277+0000 7f3eafc00640 1 -- 172.21.15.27:0/408009268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ea80cdf90 msgr2=0x7f3ea80b0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:10.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.277+0000 7f3eafc00640 1 --2- 172.21.15.27:0/408009268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ea80cdf90 0x7f3ea80b0880 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0009e50 tx=0x7f3ea001c9e0 comp rx=0 tx=0).stop 2024-01-15T18:07:10.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.277+0000 7f3eafc00640 1 -- 172.21.15.27:0/408009268 shutdown_connections 2024-01-15T18:07:10.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.277+0000 7f3eafc00640 1 --2- 172.21.15.27:0/408009268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ea80cdf90 0x7f3ea80b0880 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:10.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.277+0000 7f3eafc00640 1 --2- 172.21.15.27:0/408009268 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ea80b2390 0x7f3ea80cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:10.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.277+0000 7f3eafc00640 1 -- 172.21.15.27:0/408009268 >> 172.21.15.27:0/408009268 conn(0x7f3ea80b2a10 msgr2=0x7f3ea80b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:10.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.277+0000 7f3eafc00640 1 -- 172.21.15.27:0/408009268 shutdown_connections 2024-01-15T18:07:10.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.277+0000 7f3eafc00640 1 -- 172.21.15.27:0/408009268 wait complete. 2024-01-15T18:07:10.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.281+0000 7f3eafc00640 1 Processor -- start 2024-01-15T18:07:10.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.281+0000 7f3eafc00640 1 -- start start 2024-01-15T18:07:10.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.281+0000 7f3eafc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ea806bb80 0x7f3ea815ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:10.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.281+0000 7f3eafc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ea80b2390 0x7f3ea814beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:10.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.281+0000 7f3eafc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ea80cdf90 0x7f3ea814b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:10.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.281+0000 7f3eafc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea814bcf0 con 0x7f3ea80b2390 2024-01-15T18:07:10.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.281+0000 7f3eafc00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea814d1c0 con 0x7f3ea806bb80 2024-01-15T18:07:10.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.281+0000 7f3eafc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea814c5b0 con 0x7f3ea80cdf90 2024-01-15T18:07:10.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.281+0000 7f3ea7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ea80b2390 0x7f3ea814beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:10.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.281+0000 7f3ea7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ea80b2390 0x7f3ea814beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34794/0 (socket says 172.21.15.27:34794) 2024-01-15T18:07:10.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.281+0000 7f3ea7e00640 1 -- 172.21.15.27:0/1251618916 learned_addr learned my addr 172.21.15.27:0/1251618916 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:10.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.281+0000 7f3ead200640 1 --2- 172.21.15.27:0/1251618916 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ea806bb80 0x7f3ea815ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:10.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.281+0000 7f3eadc00640 1 --2- 172.21.15.27:0/1251618916 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ea80cdf90 0x7f3ea814b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:10.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.281+0000 7f3ea7e00640 1 -- 172.21.15.27:0/1251618916 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ea80cdf90 msgr2=0x7f3ea814b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:10.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.281+0000 7f3ea7e00640 1 --2- 172.21.15.27:0/1251618916 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ea80cdf90 0x7f3ea814b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:10.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.281+0000 7f3ea7e00640 1 -- 172.21.15.27:0/1251618916 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ea806bb80 msgr2=0x7f3ea815ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:10.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.281+0000 7f3ea7e00640 1 --2- 172.21.15.27:0/1251618916 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ea806bb80 0x7f3ea815ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:10.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.281+0000 7f3ea7e00640 1 -- 172.21.15.27:0/1251618916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea0009b30 con 0x7f3ea80b2390 2024-01-15T18:07:10.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.281+0000 7f3ead200640 1 --2- 172.21.15.27:0/1251618916 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ea806bb80 0x7f3ea815ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:07:10.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.281+0000 7f3ea7e00640 1 --2- 172.21.15.27:0/1251618916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ea80b2390 0x7f3ea814beb0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f3e90001ec0 tx=0x7f3e9000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:10.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.281+0000 7f3ea4c00640 1 -- 172.21.15.27:0/1251618916 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e900054e0 con 0x7f3ea80b2390 2024-01-15T18:07:10.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.281+0000 7f3ea4c00640 1 -- 172.21.15.27:0/1251618916 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e90003600 con 0x7f3ea80b2390 2024-01-15T18:07:10.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.281+0000 7f3eafc00640 1 -- 172.21.15.27:0/1251618916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea814c890 con 0x7f3ea80b2390 2024-01-15T18:07:10.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.281+0000 7f3ea4c00640 1 -- 172.21.15.27:0/1251618916 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e900039b0 con 0x7f3ea80b2390 2024-01-15T18:07:10.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.281+0000 7f3eafc00640 1 -- 172.21.15.27:0/1251618916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea814ce50 con 0x7f3ea80b2390 2024-01-15T18:07:10.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.285+0000 7f3eafc00640 1 -- 172.21.15.27:0/1251618916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea80d07c0 con 0x7f3ea80b2390 2024-01-15T18:07:10.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.285+0000 7f3ea4c00640 1 -- 172.21.15.27:0/1251618916 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f3e90003b10 con 0x7f3ea80b2390 2024-01-15T18:07:10.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.285+0000 7f3ea4c00640 1 --2- 172.21.15.27:0/1251618916 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e80066b60 0x7f3e80069020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:10.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.285+0000 7f3ea4c00640 1 -- 172.21.15.27:0/1251618916 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f3e90083bc0 con 0x7f3ea80b2390 2024-01-15T18:07:10.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.285+0000 7f3ead200640 1 --2- 172.21.15.27:0/1251618916 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e80066b60 0x7f3e80069020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:10.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.285+0000 7f3ead200640 1 --2- 172.21.15.27:0/1251618916 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e80066b60 0x7f3e80069020 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3e98009eb0 tx=0x7f3e98009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:10.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.289+0000 7f3ea4c00640 1 -- 172.21.15.27:0/1251618916 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3e90050d00 con 0x7f3ea80b2390 2024-01-15T18:07:10.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.485+0000 7f3eafc00640 1 -- 172.21.15.27:0/1251618916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ea804e850 con 0x7f3ea80b2390 2024-01-15T18:07:10.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.485+0000 7f3ea4c00640 1 -- 172.21.15.27:0/1251618916 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f3e90017030 con 0x7f3ea80b2390 2024-01-15T18:07:10.491 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:07:10.491 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4910,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4204677709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4204677709},{"type":"v1","addr":"172.21.15.27:6835","nonce":4204677709}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4913,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/1766105879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":1766105879},{"type":"v1","addr":"172.21.15.93:6836","nonce":1766105879}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/2799226725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":2799226725},{"type":"v1","addr":"172.21.15.93:6837","nonce":2799226725}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5688,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1337712144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1337712144},{"type":"v1","addr":"172.21.15.27:6834","nonce":1337712144}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[]} 2024-01-15T18:07:10.492 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 20 2024-01-15T18:07:10.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.485+0000 7f3eafc00640 1 -- 172.21.15.27:0/1251618916 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e80066b60 msgr2=0x7f3e80069020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:10.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.485+0000 7f3eafc00640 1 --2- 172.21.15.27:0/1251618916 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e80066b60 0x7f3e80069020 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3e98009eb0 tx=0x7f3e98009190 comp rx=0 tx=0).stop 2024-01-15T18:07:10.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.485+0000 7f3eafc00640 1 -- 172.21.15.27:0/1251618916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ea80b2390 msgr2=0x7f3ea814beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:10.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.485+0000 7f3eafc00640 1 --2- 172.21.15.27:0/1251618916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ea80b2390 0x7f3ea814beb0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f3e90001ec0 tx=0x7f3e9000c8e0 comp rx=0 tx=0).stop 2024-01-15T18:07:10.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.485+0000 7f3eafc00640 1 -- 172.21.15.27:0/1251618916 shutdown_connections 2024-01-15T18:07:10.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.485+0000 7f3eafc00640 1 --2- 172.21.15.27:0/1251618916 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e80066b60 0x7f3e80069020 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:10.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.485+0000 7f3eafc00640 1 --2- 172.21.15.27:0/1251618916 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ea80cdf90 0x7f3ea814b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:10.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.485+0000 7f3eafc00640 1 --2- 172.21.15.27:0/1251618916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ea80b2390 0x7f3ea814beb0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:10.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.485+0000 7f3eafc00640 1 --2- 172.21.15.27:0/1251618916 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ea806bb80 0x7f3ea815ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:10.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.485+0000 7f3eafc00640 1 -- 172.21.15.27:0/1251618916 >> 172.21.15.27:0/1251618916 conn(0x7f3ea80b2a10 msgr2=0x7f3ea815d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:10.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.485+0000 7f3eafc00640 1 -- 172.21.15.27:0/1251618916 shutdown_connections 2024-01-15T18:07:10.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.485+0000 7f3eafc00640 1 -- 172.21.15.27:0/1251618916 wait complete. 2024-01-15T18:07:10.513 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-15T18:07:10.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.649+0000 7f32e6e00640 1 Processor -- start 2024-01-15T18:07:10.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.649+0000 7f32e6e00640 1 -- start start 2024-01-15T18:07:10.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.649+0000 7f32e6e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f32e0158900 0x7f32e015f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:10.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.649+0000 7f32e6e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32e015bf00 0x7f32e00cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:10.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.649+0000 7f32e6e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e00b1cb0 con 0x7f32e00cf110 2024-01-15T18:07:10.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.649+0000 7f32e6e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32e00cf540 con 0x7f32e0158900 2024-01-15T18:07:10.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.649+0000 7f32e6e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00b1f00 con 0x7f32e015bf00 2024-01-15T18:07:10.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.649+0000 7f32e4e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f32e00cf110 0x7f32e00701e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57622/0 (socket says 172.21.15.27:57622) 2024-01-15T18:07:10.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.649+0000 7f32e4e00640 1 -- 172.21.15.27:0/3836326232 learned_addr learned my addr 172.21.15.27:0/3836326232 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:10.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.649+0000 7f32df400640 1 --2- 172.21.15.27:0/3836326232 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32e015bf00 0x7f32e00cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:10.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.649+0000 7f32dfe00640 1 --2- 172.21.15.27:0/3836326232 >> v2:172.21.15.93:3300/0 conn(0x7f32e0158900 0x7f32e015f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:10.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.653+0000 7f32dea00640 1 -- 172.21.15.27:0/3836326232 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810151007 0 0) 0x7f32e00b1cb0 con 0x7f32e00cf110 2024-01-15T18:07:10.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.653+0000 7f32dea00640 1 -- 172.21.15.27:0/3836326232 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32c0003530 con 0x7f32e00cf110 2024-01-15T18:07:10.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.653+0000 7f32df400640 1 -- 172.21.15.27:0/3836326232 >> v2:172.21.15.93:3300/0 conn(0x7f32e0158900 msgr2=0x7f32e015f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:10.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.653+0000 7f32df400640 1 --2- 172.21.15.27:0/3836326232 >> v2:172.21.15.93:3300/0 conn(0x7f32e0158900 0x7f32e015f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:10.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.653+0000 7f32dea00640 1 -- 172.21.15.27:0/3836326232 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1491130181 0 0) 0x7f32c0003530 con 0x7f32e00cf110 2024-01-15T18:07:10.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.653+0000 7f32df400640 1 -- 172.21.15.27:0/3836326232 >> v1:172.21.15.27:6789/0 conn(0x7f32e00cf110 legacy=0x7f32e00701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:10.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.653+0000 7f32df400640 1 -- 172.21.15.27:0/3836326232 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e00b21e0 con 0x7f32e015bf00 2024-01-15T18:07:10.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.653+0000 7f32df400640 1 --2- 172.21.15.27:0/3836326232 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32e015bf00 0x7f32e00cebd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f32c80097c0 tx=0x7f32c801c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f876223945ad221e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:10.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.653+0000 7f32dea00640 1 -- 172.21.15.27:0/3836326232 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d8001a00 con 0x7f32e015bf00 2024-01-15T18:07:10.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.653+0000 7f32dea00640 1 -- 172.21.15.27:0/3836326232 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c80a7050 con 0x7f32e015bf00 2024-01-15T18:07:10.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.653+0000 7f32dea00640 1 -- 172.21.15.27:0/3836326232 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c80ab620 con 0x7f32e015bf00 2024-01-15T18:07:10.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.657+0000 7f32e6e00640 1 -- 172.21.15.27:0/3836326232 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32e015bf00 msgr2=0x7f32e00cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:10.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.657+0000 7f32e6e00640 1 --2- 172.21.15.27:0/3836326232 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32e015bf00 0x7f32e00cebd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f32c80097c0 tx=0x7f32c801c3a0 comp rx=0 tx=0).stop 2024-01-15T18:07:10.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.657+0000 7f32e6e00640 1 -- 172.21.15.27:0/3836326232 shutdown_connections 2024-01-15T18:07:10.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.657+0000 7f32e6e00640 1 --2- 172.21.15.27:0/3836326232 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32e015bf00 0x7f32e00cebd0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:10.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.657+0000 7f32e6e00640 1 --2- 172.21.15.27:0/3836326232 >> v2:172.21.15.93:3300/0 conn(0x7f32e0158900 0x7f32e015f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:10.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.657+0000 7f32e6e00640 1 -- 172.21.15.27:0/3836326232 >> 172.21.15.27:0/3836326232 conn(0x7f32e00cb700 msgr2=0x7f32e00cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:10.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.657+0000 7f32e6e00640 1 -- 172.21.15.27:0/3836326232 shutdown_connections 2024-01-15T18:07:10.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.657+0000 7f32e6e00640 1 -- 172.21.15.27:0/3836326232 wait complete. 2024-01-15T18:07:10.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.657+0000 7f32e6e00640 1 Processor -- start 2024-01-15T18:07:10.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.657+0000 7f32e6e00640 1 -- start start 2024-01-15T18:07:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.657+0000 7f32e6e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32e00cf110 0x7f32e0146ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.657+0000 7f32e6e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32e0158900 0x7f32e0155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.657+0000 7f32e6e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32e015bf00 0x7f32e01564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.657+0000 7f32e6e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e0157030 con 0x7f32e00cf110 2024-01-15T18:07:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.657+0000 7f32e6e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e015a550 con 0x7f32e0158900 2024-01-15T18:07:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.657+0000 7f32e6e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e0156d50 con 0x7f32e015bf00 2024-01-15T18:07:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.657+0000 7f32dfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32e00cf110 0x7f32e0146ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:10.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.657+0000 7f32dfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32e00cf110 0x7f32e0146ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34798/0 (socket says 172.21.15.27:34798) 2024-01-15T18:07:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.657+0000 7f32dfe00640 1 -- 172.21.15.27:0/3191016043 learned_addr learned my addr 172.21.15.27:0/3191016043 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.657+0000 7f32df400640 1 --2- 172.21.15.27:0/3191016043 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32e0158900 0x7f32e0155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.657+0000 7f32e4e00640 1 --2- 172.21.15.27:0/3191016043 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32e015bf00 0x7f32e01564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.661+0000 7f32dfe00640 1 -- 172.21.15.27:0/3191016043 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32e015bf00 msgr2=0x7f32e01564c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.661+0000 7f32dfe00640 1 --2- 172.21.15.27:0/3191016043 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32e015bf00 0x7f32e01564c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.661+0000 7f32dfe00640 1 -- 172.21.15.27:0/3191016043 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32e0158900 msgr2=0x7f32e0155f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.661+0000 7f32dfe00640 1 --2- 172.21.15.27:0/3191016043 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32e0158900 0x7f32e0155f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.661+0000 7f32dfe00640 1 -- 172.21.15.27:0/3191016043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c8009420 con 0x7f32e00cf110 2024-01-15T18:07:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.661+0000 7f32df400640 1 --2- 172.21.15.27:0/3191016043 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32e0158900 0x7f32e0155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:07:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.661+0000 7f32dfe00640 1 --2- 172.21.15.27:0/3191016043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32e00cf110 0x7f32e0146ee0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f32d000e930 tx=0x7f32d000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.661+0000 7f32d7e00640 1 -- 172.21.15.27:0/3191016043 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d0009690 con 0x7f32e00cf110 2024-01-15T18:07:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.661+0000 7f32d7e00640 1 -- 172.21.15.27:0/3191016043 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d000ca00 con 0x7f32e00cf110 2024-01-15T18:07:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.661+0000 7f32d7e00640 1 -- 172.21.15.27:0/3191016043 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d000cd70 con 0x7f32e00cf110 2024-01-15T18:07:10.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.661+0000 7f32e6e00640 1 -- 172.21.15.27:0/3191016043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32e015a760 con 0x7f32e00cf110 2024-01-15T18:07:10.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.661+0000 7f32e6e00640 1 -- 172.21.15.27:0/3191016043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e015ad20 con 0x7f32e00cf110 2024-01-15T18:07:10.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.661+0000 7f32e6e00640 1 -- 172.21.15.27:0/3191016043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32e004e850 con 0x7f32e00cf110 2024-01-15T18:07:10.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.661+0000 7f32d7e00640 1 -- 172.21.15.27:0/3191016043 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f32d0013470 con 0x7f32e00cf110 2024-01-15T18:07:10.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.661+0000 7f32d7e00640 1 --2- 172.21.15.27:0/3191016043 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f32b8066b30 0x7f32b8068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:10.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.661+0000 7f32d7e00640 1 -- 172.21.15.27:0/3191016043 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f32d00847a0 con 0x7f32e00cf110 2024-01-15T18:07:10.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.661+0000 7f32df400640 1 --2- 172.21.15.27:0/3191016043 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f32b8066b30 0x7f32b8068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:10.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.665+0000 7f32df400640 1 --2- 172.21.15.27:0/3191016043 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f32b8066b30 0x7f32b8068ff0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f32c80ad000 tx=0x7f32c80a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:10.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.665+0000 7f32d7e00640 1 -- 172.21.15.27:0/3191016043 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f32d0051ad0 con 0x7f32e00cf110 2024-01-15T18:07:10.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.825+0000 7f32e6e00640 1 -- 172.21.15.27:0/3191016043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7f32e006d540 con 0x7f32e00cf110 2024-01-15T18:07:10.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.829+0000 7f32d7e00640 1 -- 172.21.15.27:0/3191016043 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v12) v1 ==== 120+0+0 (secure 0 0 0) 0x7f32d00550f0 con 0x7f32e00cf110 2024-01-15T18:07:10.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.829+0000 7f32e6e00640 1 -- 172.21.15.27:0/3191016043 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f32b8066b30 msgr2=0x7f32b8068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:10.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.829+0000 7f32e6e00640 1 --2- 172.21.15.27:0/3191016043 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f32b8066b30 0x7f32b8068ff0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f32c80ad000 tx=0x7f32c80a6000 comp rx=0 tx=0).stop 2024-01-15T18:07:10.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.829+0000 7f32e6e00640 1 -- 172.21.15.27:0/3191016043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32e00cf110 msgr2=0x7f32e0146ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:10.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.829+0000 7f32e6e00640 1 --2- 172.21.15.27:0/3191016043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32e00cf110 0x7f32e0146ee0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f32d000e930 tx=0x7f32d000edf0 comp rx=0 tx=0).stop 2024-01-15T18:07:10.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.829+0000 7f32e6e00640 1 -- 172.21.15.27:0/3191016043 shutdown_connections 2024-01-15T18:07:10.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.829+0000 7f32e6e00640 1 --2- 172.21.15.27:0/3191016043 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f32b8066b30 0x7f32b8068ff0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:10.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.829+0000 7f32e6e00640 1 --2- 172.21.15.27:0/3191016043 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32e015bf00 0x7f32e01564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:10.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.829+0000 7f32e6e00640 1 --2- 172.21.15.27:0/3191016043 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32e0158900 0x7f32e0155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:10.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.829+0000 7f32e6e00640 1 --2- 172.21.15.27:0/3191016043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32e00cf110 0x7f32e0146ee0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:10.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.829+0000 7f32e6e00640 1 -- 172.21.15.27:0/3191016043 >> 172.21.15.27:0/3191016043 conn(0x7f32e00cb700 msgr2=0x7f32e00b3830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:10.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.829+0000 7f32e6e00640 1 -- 172.21.15.27:0/3191016043 shutdown_connections 2024-01-15T18:07:10.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.829+0000 7f32e6e00640 1 -- 172.21.15.27:0/3191016043 wait complete. 2024-01-15T18:07:10.858 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-15T18:07:11.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.993+0000 7f1c33a00640 1 Processor -- start 2024-01-15T18:07:11.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.993+0000 7f1c33a00640 1 -- start start 2024-01-15T18:07:11.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.997+0000 7f1c33a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1c2c070d50 0x7f1c2c0aec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:11.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.997+0000 7f1c33a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c2c1597c0 0x7f1c2c06de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:11.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.997+0000 7f1c33a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2c06d100 con 0x7f1c2c0ae780 2024-01-15T18:07:11.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.997+0000 7f1c33a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c2c06b2e0 con 0x7f1c2c070d50 2024-01-15T18:07:11.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.997+0000 7f1c33a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c06d350 con 0x7f1c2c1597c0 2024-01-15T18:07:11.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.997+0000 7f1c31000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1c2c070d50 0x7f1c2c0aec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:11.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.997+0000 7f1c31a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1c2c0ae780 0x7f1c2c071130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57634/0 (socket says 172.21.15.27:57634) 2024-01-15T18:07:11.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.997+0000 7f1c31a00640 1 -- 172.21.15.27:0/3025170755 learned_addr learned my addr 172.21.15.27:0/3025170755 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:11.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.997+0000 7f1c2be00640 1 --2- 172.21.15.27:0/3025170755 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c2c1597c0 0x7f1c2c06de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:11.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.997+0000 7f1c31000640 1 -- 172.21.15.27:0/3025170755 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c2c1597c0 msgr2=0x7f1c2c06de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:11.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.997+0000 7f1c31000640 1 --2- 172.21.15.27:0/3025170755 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c2c1597c0 0x7f1c2c06de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:11.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.997+0000 7f1c31000640 1 -- 172.21.15.27:0/3025170755 >> v1:172.21.15.27:6789/0 conn(0x7f1c2c0ae780 legacy=0x7f1c2c071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:11.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:10.997+0000 7f1c31000640 1 -- 172.21.15.27:0/3025170755 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2c06d630 con 0x7f1c2c070d50 2024-01-15T18:07:11.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.001+0000 7f1c31000640 1 --2- 172.21.15.27:0/3025170755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c2c070d50 0x7f1c2c0aec40 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c0097c0 tx=0x7f1c1c01c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccd452095b5df5d7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:11.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.001+0000 7f1c2b400640 1 -- 172.21.15.27:0/3025170755 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c0ab600 con 0x7f1c2c070d50 2024-01-15T18:07:11.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.001+0000 7f1c2b400640 1 -- 172.21.15.27:0/3025170755 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c1c0a7050 con 0x7f1c2c070d50 2024-01-15T18:07:11.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.001+0000 7f1c2b400640 1 -- 172.21.15.27:0/3025170755 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c0ab980 con 0x7f1c2c070d50 2024-01-15T18:07:11.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.001+0000 7f1c33a00640 1 -- 172.21.15.27:0/3025170755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c2c070d50 msgr2=0x7f1c2c0aec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:11.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.001+0000 7f1c33a00640 1 --2- 172.21.15.27:0/3025170755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c2c070d50 0x7f1c2c0aec40 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c0097c0 tx=0x7f1c1c01c330 comp rx=0 tx=0).stop 2024-01-15T18:07:11.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.001+0000 7f1c33a00640 1 -- 172.21.15.27:0/3025170755 shutdown_connections 2024-01-15T18:07:11.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.001+0000 7f1c33a00640 1 --2- 172.21.15.27:0/3025170755 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c2c1597c0 0x7f1c2c06de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:11.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.001+0000 7f1c33a00640 1 --2- 172.21.15.27:0/3025170755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c2c070d50 0x7f1c2c0aec40 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:11.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.001+0000 7f1c33a00640 1 -- 172.21.15.27:0/3025170755 >> 172.21.15.27:0/3025170755 conn(0x7f1c2c0b1620 msgr2=0x7f1c2c0b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:11.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.001+0000 7f1c33a00640 1 -- 172.21.15.27:0/3025170755 shutdown_connections 2024-01-15T18:07:11.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.001+0000 7f1c33a00640 1 -- 172.21.15.27:0/3025170755 wait complete. 2024-01-15T18:07:11.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.001+0000 7f1c33a00640 1 Processor -- start 2024-01-15T18:07:11.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.005+0000 7f1c33a00640 1 -- start start 2024-01-15T18:07:11.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.005+0000 7f1c33a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c2c070d50 0x7f1c2c163d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:11.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.005+0000 7f1c33a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c2c0ae780 0x7f1c2c153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:11.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.005+0000 7f1c33a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1c2c1597c0 0x7f1c2c0add60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:11.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.005+0000 7f1c33a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c154950 con 0x7f1c2c070d50 2024-01-15T18:07:11.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.005+0000 7f1c33a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c1f58e0 con 0x7f1c2c1597c0 2024-01-15T18:07:11.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.005+0000 7f1c33a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c0ae2a0 con 0x7f1c2c0ae780 2024-01-15T18:07:11.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.005+0000 7f1c31000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c2c070d50 0x7f1c2c163d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:11.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.005+0000 7f1c2be00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c2c0ae780 0x7f1c2c153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:11.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.005+0000 7f1c2be00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c2c0ae780 0x7f1c2c153e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:41534/0 (socket says 172.21.15.27:41534) 2024-01-15T18:07:11.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.005+0000 7f1c31000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c2c070d50 0x7f1c2c163d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34804/0 (socket says 172.21.15.27:34804) 2024-01-15T18:07:11.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.005+0000 7f1c2be00640 1 -- 172.21.15.27:0/3683167212 learned_addr learned my addr 172.21.15.27:0/3683167212 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:11.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.005+0000 7f1c31a00640 1 --2- 172.21.15.27:0/3683167212 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1c2c1597c0 0x7f1c2c0add60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:11.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.005+0000 7f1c2be00640 1 -- 172.21.15.27:0/3683167212 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1c2c1597c0 msgr2=0x7f1c2c0add60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:11.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.005+0000 7f1c2be00640 1 --2- 172.21.15.27:0/3683167212 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1c2c1597c0 0x7f1c2c0add60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:11.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.005+0000 7f1c2be00640 1 -- 172.21.15.27:0/3683167212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c2c070d50 msgr2=0x7f1c2c163d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:11.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.005+0000 7f1c2be00640 1 --2- 172.21.15.27:0/3683167212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c2c070d50 0x7f1c2c163d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:11.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.005+0000 7f1c2be00640 1 -- 172.21.15.27:0/3683167212 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c1c009420 con 0x7f1c2c0ae780 2024-01-15T18:07:11.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.005+0000 7f1c2be00640 1 --2- 172.21.15.27:0/3683167212 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c2c0ae780 0x7f1c2c153e40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400e870 tx=0x7f1c1400ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:11.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.005+0000 7f1c28c00640 1 -- 172.21.15.27:0/3683167212 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c14009510 con 0x7f1c2c0ae780 2024-01-15T18:07:11.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.005+0000 7f1c28c00640 1 -- 172.21.15.27:0/3683167212 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c1400eec0 con 0x7f1c2c0ae780 2024-01-15T18:07:11.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.005+0000 7f1c33a00640 1 -- 172.21.15.27:0/3683167212 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c2c154b60 con 0x7f1c2c0ae780 2024-01-15T18:07:11.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.005+0000 7f1c28c00640 1 -- 172.21.15.27:0/3683167212 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c1400cb40 con 0x7f1c2c0ae780 2024-01-15T18:07:11.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.005+0000 7f1c33a00640 1 -- 172.21.15.27:0/3683167212 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c2c155120 con 0x7f1c2c0ae780 2024-01-15T18:07:11.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.009+0000 7f1c33a00640 1 -- 172.21.15.27:0/3683167212 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c2c04e850 con 0x7f1c2c0ae780 2024-01-15T18:07:11.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.009+0000 7f1c28c00640 1 -- 172.21.15.27:0/3683167212 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f1c1400cca0 con 0x7f1c2c0ae780 2024-01-15T18:07:11.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.009+0000 7f1c28c00640 1 --2- 172.21.15.27:0/3683167212 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1c04066b30 0x7f1c04068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:11.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.009+0000 7f1c28c00640 1 -- 172.21.15.27:0/3683167212 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f1c140846a0 con 0x7f1c2c0ae780 2024-01-15T18:07:11.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.009+0000 7f1c31000640 1 --2- 172.21.15.27:0/3683167212 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1c04066b30 0x7f1c04068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:11.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.009+0000 7f1c31000640 1 --2- 172.21.15.27:0/3683167212 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1c04066b30 0x7f1c04068ff0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c0ad000 tx=0x7f1c1c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:11.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.013+0000 7f1c28c00640 1 -- 172.21.15.27:0/3683167212 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1c14051870 con 0x7f1c2c0ae780 2024-01-15T18:07:11.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.173+0000 7f1c33a00640 1 -- 172.21.15.27:0/3683167212 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f1c2c066e50 con 0x7f1c2c0ae780 2024-01-15T18:07:11.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.181+0000 7f1c28c00640 1 -- 172.21.15.27:0/3683167212 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1c14054e90 con 0x7f1c2c0ae780 2024-01-15T18:07:11.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.181+0000 7f1c33a00640 1 -- 172.21.15.27:0/3683167212 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1c04066b30 msgr2=0x7f1c04068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:11.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.181+0000 7f1c33a00640 1 --2- 172.21.15.27:0/3683167212 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1c04066b30 0x7f1c04068ff0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c0ad000 tx=0x7f1c1c008040 comp rx=0 tx=0).stop 2024-01-15T18:07:11.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.181+0000 7f1c33a00640 1 -- 172.21.15.27:0/3683167212 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c2c0ae780 msgr2=0x7f1c2c153e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:11.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.181+0000 7f1c33a00640 1 --2- 172.21.15.27:0/3683167212 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c2c0ae780 0x7f1c2c153e40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1c1400e870 tx=0x7f1c1400ec70 comp rx=0 tx=0).stop 2024-01-15T18:07:11.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.181+0000 7f1c33a00640 1 -- 172.21.15.27:0/3683167212 shutdown_connections 2024-01-15T18:07:11.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.181+0000 7f1c33a00640 1 --2- 172.21.15.27:0/3683167212 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1c04066b30 0x7f1c04068ff0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:11.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.181+0000 7f1c33a00640 1 --2- 172.21.15.27:0/3683167212 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1c2c1597c0 0x7f1c2c0add60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:11.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.181+0000 7f1c33a00640 1 --2- 172.21.15.27:0/3683167212 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1c2c0ae780 0x7f1c2c153e40 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:11.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.181+0000 7f1c33a00640 1 --2- 172.21.15.27:0/3683167212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c2c070d50 0x7f1c2c163d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:11.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.185+0000 7f1c33a00640 1 -- 172.21.15.27:0/3683167212 >> 172.21.15.27:0/3683167212 conn(0x7f1c2c0b1620 msgr2=0x7f1c2c0cda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:11.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.185+0000 7f1c33a00640 1 -- 172.21.15.27:0/3683167212 shutdown_connections 2024-01-15T18:07:11.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.185+0000 7f1c33a00640 1 -- 172.21.15.27:0/3683167212 wait complete. 2024-01-15T18:07:11.211 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone' 2024-01-15T18:07:11.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.345+0000 7fc76ba00640 1 Processor -- start 2024-01-15T18:07:11.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.345+0000 7fc76ba00640 1 -- start start 2024-01-15T18:07:11.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.345+0000 7fc76ba00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7640c5d90 0x7fc7640cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:11.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.345+0000 7fc76ba00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fc7640ceb20 0x7fc7640cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:11.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.345+0000 7fc76ba00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7640d1a70 con 0x7fc76415e7d0 2024-01-15T18:07:11.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.345+0000 7fc76ba00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7640ccdd0 con 0x7fc7640ceb20 2024-01-15T18:07:11.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.345+0000 7fc76ba00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc76415ebb0 con 0x7fc7640c5d90 2024-01-15T18:07:11.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.345+0000 7fc769000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc76415e7d0 0x7fc7640cd730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57644/0 (socket says 172.21.15.27:57644) 2024-01-15T18:07:11.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.345+0000 7fc769000640 1 -- 172.21.15.27:0/2837841488 learned_addr learned my addr 172.21.15.27:0/2837841488 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:11.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.349+0000 7fc769a00640 1 --2- 172.21.15.27:0/2837841488 >> v2:172.21.15.93:3300/0 conn(0x7fc7640ceb20 0x7fc7640cefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:11.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.349+0000 7fc763e00640 1 --2- 172.21.15.27:0/2837841488 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7640c5d90 0x7fc7640cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:11.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.349+0000 7fc763400640 1 -- 172.21.15.27:0/2837841488 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2933772420 0 0) 0x7fc7640d1a70 con 0x7fc76415e7d0 2024-01-15T18:07:11.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.349+0000 7fc763400640 1 -- 172.21.15.27:0/2837841488 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc744003530 con 0x7fc76415e7d0 2024-01-15T18:07:11.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.349+0000 7fc769a00640 1 -- 172.21.15.27:0/2837841488 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7640c5d90 msgr2=0x7fc7640cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:11.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.349+0000 7fc769a00640 1 --2- 172.21.15.27:0/2837841488 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7640c5d90 0x7fc7640cde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:11.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.349+0000 7fc769a00640 1 -- 172.21.15.27:0/2837841488 >> v1:172.21.15.27:6789/0 conn(0x7fc76415e7d0 legacy=0x7fc7640cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:11.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.349+0000 7fc769a00640 1 -- 172.21.15.27:0/2837841488 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7640d1cc0 con 0x7fc7640ceb20 2024-01-15T18:07:11.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.349+0000 7fc769a00640 1 --2- 172.21.15.27:0/2837841488 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7640ceb20 0x7fc7640cefb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc75c009d90 tx=0x7fc75c01c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf77089541cfcd4d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:11.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.349+0000 7fc763400640 1 -- 172.21.15.27:0/2837841488 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75c0a59e0 con 0x7fc7640ceb20 2024-01-15T18:07:11.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.349+0000 7fc763400640 1 -- 172.21.15.27:0/2837841488 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc75c0a5b40 con 0x7fc7640ceb20 2024-01-15T18:07:11.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.349+0000 7fc763400640 1 -- 172.21.15.27:0/2837841488 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75c0ad440 con 0x7fc7640ceb20 2024-01-15T18:07:11.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.353+0000 7fc76ba00640 1 -- 172.21.15.27:0/2837841488 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7640ceb20 msgr2=0x7fc7640cefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:11.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.353+0000 7fc76ba00640 1 --2- 172.21.15.27:0/2837841488 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7640ceb20 0x7fc7640cefb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc75c009d90 tx=0x7fc75c01c920 comp rx=0 tx=0).stop 2024-01-15T18:07:11.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.353+0000 7fc76ba00640 1 -- 172.21.15.27:0/2837841488 shutdown_connections 2024-01-15T18:07:11.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.353+0000 7fc76ba00640 1 --2- 172.21.15.27:0/2837841488 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7640c5d90 0x7fc7640cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:11.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.353+0000 7fc76ba00640 1 --2- 172.21.15.27:0/2837841488 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7640ceb20 0x7fc7640cefb0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:11.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.353+0000 7fc76ba00640 1 -- 172.21.15.27:0/2837841488 >> 172.21.15.27:0/2837841488 conn(0x7fc7640d0f50 msgr2=0x7fc7640d1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:11.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.353+0000 7fc76ba00640 1 -- 172.21.15.27:0/2837841488 shutdown_connections 2024-01-15T18:07:11.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.353+0000 7fc76ba00640 1 -- 172.21.15.27:0/2837841488 wait complete. 2024-01-15T18:07:11.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.353+0000 7fc76ba00640 1 Processor -- start 2024-01-15T18:07:11.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.353+0000 7fc76ba00640 1 -- start start 2024-01-15T18:07:11.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.353+0000 7fc76ba00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7640c5d90 0x7fc76406bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:11.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.353+0000 7fc76ba00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7640ceb20 0x7fc76415a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:11.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.353+0000 7fc76ba00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc76415e7d0 0x7fc76414ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:11.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.353+0000 7fc76ba00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76406d0a0 con 0x7fc76415e7d0 2024-01-15T18:07:11.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.353+0000 7fc76ba00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76406c800 con 0x7fc7640ceb20 2024-01-15T18:07:11.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.353+0000 7fc76ba00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc76406cae0 con 0x7fc7640c5d90 2024-01-15T18:07:11.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.353+0000 7fc769a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc76415e7d0 0x7fc76414ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:11.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.353+0000 7fc769000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7640c5d90 0x7fc76406bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:11.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.353+0000 7fc763e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7640ceb20 0x7fc76415a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:11.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.353+0000 7fc769a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc76415e7d0 0x7fc76414ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34806/0 (socket says 172.21.15.27:34806) 2024-01-15T18:07:11.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.353+0000 7fc769a00640 1 -- 172.21.15.27:0/3986269165 learned_addr learned my addr 172.21.15.27:0/3986269165 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:11.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.357+0000 7fc769000640 1 -- 172.21.15.27:0/3986269165 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7640ceb20 msgr2=0x7fc76415a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:11.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.357+0000 7fc769000640 1 --2- 172.21.15.27:0/3986269165 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7640ceb20 0x7fc76415a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:11.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.357+0000 7fc769000640 1 -- 172.21.15.27:0/3986269165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc76415e7d0 msgr2=0x7fc76414ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:11.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.357+0000 7fc769000640 1 --2- 172.21.15.27:0/3986269165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc76415e7d0 0x7fc76414ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:11.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.357+0000 7fc769000640 1 -- 172.21.15.27:0/3986269165 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc75c009a70 con 0x7fc7640c5d90 2024-01-15T18:07:11.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.357+0000 7fc769000640 1 --2- 172.21.15.27:0/3986269165 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7640c5d90 0x7fc76406bfd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc75400c400 tx=0x7fc75400c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:11.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.357+0000 7fc760c00640 1 -- 172.21.15.27:0/3986269165 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc754005cf0 con 0x7fc7640c5d90 2024-01-15T18:07:11.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.357+0000 7fc760c00640 1 -- 172.21.15.27:0/3986269165 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc754005e50 con 0x7fc7640c5d90 2024-01-15T18:07:11.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.357+0000 7fc760c00640 1 -- 172.21.15.27:0/3986269165 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7540072d0 con 0x7fc7640c5d90 2024-01-15T18:07:11.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.357+0000 7fc76ba00640 1 -- 172.21.15.27:0/3986269165 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc76406cdc0 con 0x7fc7640c5d90 2024-01-15T18:07:11.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.357+0000 7fc76ba00640 1 -- 172.21.15.27:0/3986269165 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc76406c510 con 0x7fc7640c5d90 2024-01-15T18:07:11.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.357+0000 7fc76ba00640 1 -- 172.21.15.27:0/3986269165 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc76404e850 con 0x7fc7640c5d90 2024-01-15T18:07:11.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.357+0000 7fc760c00640 1 -- 172.21.15.27:0/3986269165 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7fc754007430 con 0x7fc7640c5d90 2024-01-15T18:07:11.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.357+0000 7fc760c00640 1 --2- 172.21.15.27:0/3986269165 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc73c066b30 0x7fc73c068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:11.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.357+0000 7fc760c00640 1 -- 172.21.15.27:0/3986269165 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fc754083ab0 con 0x7fc7640c5d90 2024-01-15T18:07:11.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.357+0000 7fc763e00640 1 --2- 172.21.15.27:0/3986269165 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc73c066b30 0x7fc73c068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:11.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.361+0000 7fc763e00640 1 --2- 172.21.15.27:0/3986269165 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc73c066b30 0x7fc73c068ff0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc7640b2630 tx=0x7fc74c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:11.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.361+0000 7fc760c00640 1 -- 172.21.15.27:0/3986269165 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc754050d50 con 0x7fc7640c5d90 2024-01-15T18:07:11.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:11.577+0000 7fc76ba00640 1 -- 172.21.15.27:0/3986269165 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]} v 0) v1 -- 0x7fc7640b6a70 con 0x7fc7640c5d90 2024-01-15T18:07:12.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.001+0000 7fc760c00640 1 -- 172.21.15.27:0/3986269165 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]}]=0 v132) v1 ==== 152+0+0 (secure 0 0 0) 0x7fc754054370 con 0x7fc7640c5d90 2024-01-15T18:07:12.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.005+0000 7fc76ba00640 1 -- 172.21.15.27:0/3986269165 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc73c066b30 msgr2=0x7fc73c068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:12.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.005+0000 7fc76ba00640 1 --2- 172.21.15.27:0/3986269165 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc73c066b30 0x7fc73c068ff0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc7640b2630 tx=0x7fc74c009190 comp rx=0 tx=0).stop 2024-01-15T18:07:12.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.005+0000 7fc76ba00640 1 -- 172.21.15.27:0/3986269165 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7640c5d90 msgr2=0x7fc76406bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:12.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.005+0000 7fc76ba00640 1 --2- 172.21.15.27:0/3986269165 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7640c5d90 0x7fc76406bfd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc75400c400 tx=0x7fc75400c8c0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.005+0000 7fc76ba00640 1 -- 172.21.15.27:0/3986269165 shutdown_connections 2024-01-15T18:07:12.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.005+0000 7fc76ba00640 1 --2- 172.21.15.27:0/3986269165 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc73c066b30 0x7fc73c068ff0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.005+0000 7fc76ba00640 1 --2- 172.21.15.27:0/3986269165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc76415e7d0 0x7fc76414ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.005+0000 7fc76ba00640 1 --2- 172.21.15.27:0/3986269165 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7640ceb20 0x7fc76415a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.005+0000 7fc76ba00640 1 --2- 172.21.15.27:0/3986269165 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7640c5d90 0x7fc76406bfd0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.005+0000 7fc76ba00640 1 -- 172.21.15.27:0/3986269165 >> 172.21.15.27:0/3986269165 conn(0x7fc7640d0f50 msgr2=0x7fc7640c7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:12.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.005+0000 7fc76ba00640 1 -- 172.21.15.27:0/3986269165 shutdown_connections 2024-01-15T18:07:12.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.005+0000 7fc76ba00640 1 -- 172.21.15.27:0/3986269165 wait complete. 2024-01-15T18:07:12.034 INFO:tasks.cephfs_test_runner:Test that when one clone is in progress, one progress bar is printed ... ok 2024-01-15T18:07:12.034 INFO:tasks.cephfs_test_runner:Starting test: test_progress_bar_for_4_clones (tasks.cephfs.test_volumes.TestCloneProgress) X = number of clones issued. 2024-01-15T18:07:12.034 INFO:tasks.cephfs_test_runner:test_progress_bar_for_4_clones (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T18:07:12.034 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:07:12.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.169+0000 7f7ec8400640 1 Processor -- start 2024-01-15T18:07:12.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.169+0000 7f7ec8400640 1 -- start start 2024-01-15T18:07:12.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.169+0000 7f7ec8400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7ec006d430 0x7f7ec01644e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:12.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.169+0000 7f7ec8400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec0159a50 con 0x7f7ec0159bf0 2024-01-15T18:07:12.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.169+0000 7f7ec8400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec006d980 con 0x7f7ec0071e40 2024-01-15T18:07:12.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.169+0000 7f7ec8400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ec00cce00 con 0x7f7ec006d430 2024-01-15T18:07:12.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.169+0000 7f7ec5a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7ec0159bf0 0x7f7ec0159fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57646/0 (socket says 172.21.15.27:57646) 2024-01-15T18:07:12.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.169+0000 7f7ec5a00640 1 -- 172.21.15.27:0/2817009260 learned_addr learned my addr 172.21.15.27:0/2817009260 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:12.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.169+0000 7f7ec6400640 1 --2- 172.21.15.27:0/2817009260 >> v2:172.21.15.27:3300/0 conn(0x7f7ec006d430 0x7f7ec01644e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:12.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.173+0000 7f7ec6400640 1 -- 172.21.15.27:0/2817009260 >> v1:172.21.15.93:6789/0 conn(0x7f7ec0071e40 legacy=0x7f7ec00c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:12.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.173+0000 7f7ec6400640 1 -- 172.21.15.27:0/2817009260 >> v1:172.21.15.27:6789/0 conn(0x7f7ec0159bf0 legacy=0x7f7ec0159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:12.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.173+0000 7f7ec6400640 1 -- 172.21.15.27:0/2817009260 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec006dc60 con 0x7f7ec006d430 2024-01-15T18:07:12.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.173+0000 7f7ec6400640 1 --2- 172.21.15.27:0/2817009260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec006d430 0x7f7ec01644e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8009d90 tx=0x7f7eb801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d9068b631546ab4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:12.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.173+0000 7f7ebfe00640 1 -- 172.21.15.27:0/2817009260 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb80af600 con 0x7f7ec006d430 2024-01-15T18:07:12.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.173+0000 7f7ebfe00640 1 -- 172.21.15.27:0/2817009260 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb80ab050 con 0x7f7ec006d430 2024-01-15T18:07:12.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.173+0000 7f7ebfe00640 1 -- 172.21.15.27:0/2817009260 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb80af980 con 0x7f7ec006d430 2024-01-15T18:07:12.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.173+0000 7f7ec8400640 1 -- 172.21.15.27:0/2817009260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec006d430 msgr2=0x7f7ec01644e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:12.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.173+0000 7f7ec8400640 1 --2- 172.21.15.27:0/2817009260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec006d430 0x7f7ec01644e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8009d90 tx=0x7f7eb801f130 comp rx=0 tx=0).stop 2024-01-15T18:07:12.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.177+0000 7f7ec8400640 1 -- 172.21.15.27:0/2817009260 shutdown_connections 2024-01-15T18:07:12.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.177+0000 7f7ec8400640 1 --2- 172.21.15.27:0/2817009260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec006d430 0x7f7ec01644e0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.177+0000 7f7ec8400640 1 -- 172.21.15.27:0/2817009260 >> 172.21.15.27:0/2817009260 conn(0x7f7ec00cec60 msgr2=0x7f7ec00cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:12.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.177+0000 7f7ec8400640 1 -- 172.21.15.27:0/2817009260 shutdown_connections 2024-01-15T18:07:12.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.177+0000 7f7ec8400640 1 -- 172.21.15.27:0/2817009260 wait complete. 2024-01-15T18:07:12.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.177+0000 7f7ec8400640 1 Processor -- start 2024-01-15T18:07:12.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.177+0000 7f7ec8400640 1 -- start start 2024-01-15T18:07:12.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.177+0000 7f7ec8400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ec006d430 0x7f7ec00c3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:12.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.177+0000 7f7ec8400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ec0159bf0 0x7f7ec01f6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:12.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.177+0000 7f7ec8400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec01570d0 0x7f7ec0157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:12.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.177+0000 7f7ec8400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec0157ac0 con 0x7f7ec01570d0 2024-01-15T18:07:12.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.177+0000 7f7ec8400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec00c1de0 con 0x7f7ec006d430 2024-01-15T18:07:12.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.177+0000 7f7ec8400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec01402e0 con 0x7f7ec0159bf0 2024-01-15T18:07:12.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.177+0000 7f7ec6400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec01570d0 0x7f7ec0157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:12.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.177+0000 7f7ec6400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec01570d0 0x7f7ec0157580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34814/0 (socket says 172.21.15.27:34814) 2024-01-15T18:07:12.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.177+0000 7f7ec6400640 1 -- 172.21.15.27:0/2709858431 learned_addr learned my addr 172.21.15.27:0/2709858431 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:12.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.177+0000 7f7ec5000640 1 --2- 172.21.15.27:0/2709858431 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ec0159bf0 0x7f7ec01f6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:12.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.177+0000 7f7ec5a00640 1 --2- 172.21.15.27:0/2709858431 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ec006d430 0x7f7ec00c3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:12.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.177+0000 7f7ec6400640 1 -- 172.21.15.27:0/2709858431 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ec0159bf0 msgr2=0x7f7ec01f6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:12.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.177+0000 7f7ec6400640 1 --2- 172.21.15.27:0/2709858431 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ec0159bf0 0x7f7ec01f6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.177+0000 7f7ec6400640 1 -- 172.21.15.27:0/2709858431 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ec006d430 msgr2=0x7f7ec00c3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:12.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.177+0000 7f7ec6400640 1 --2- 172.21.15.27:0/2709858431 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ec006d430 0x7f7ec00c3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.177+0000 7f7ec6400640 1 -- 172.21.15.27:0/2709858431 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb8009a70 con 0x7f7ec01570d0 2024-01-15T18:07:12.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.177+0000 7f7ec6400640 1 --2- 172.21.15.27:0/2709858431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec01570d0 0x7f7ec0157580 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f7eb80b2040 tx=0x7f7eb801bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:12.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.181+0000 7f7ebd600640 1 -- 172.21.15.27:0/2709858431 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb80b9610 con 0x7f7ec01570d0 2024-01-15T18:07:12.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.181+0000 7f7ebd600640 1 -- 172.21.15.27:0/2709858431 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb80b9770 con 0x7f7ec01570d0 2024-01-15T18:07:12.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.181+0000 7f7ec8400640 1 -- 172.21.15.27:0/2709858431 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec01f7460 con 0x7f7ec01570d0 2024-01-15T18:07:12.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.181+0000 7f7ebd600640 1 -- 172.21.15.27:0/2709858431 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb80b9b20 con 0x7f7ec01570d0 2024-01-15T18:07:12.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.181+0000 7f7ec8400640 1 -- 172.21.15.27:0/2709858431 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec0140810 con 0x7f7ec01570d0 2024-01-15T18:07:12.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.181+0000 7f7ec8400640 1 -- 172.21.15.27:0/2709858431 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec004e850 con 0x7f7ec01570d0 2024-01-15T18:07:12.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.181+0000 7f7ebd600640 1 -- 172.21.15.27:0/2709858431 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f7eb80b6050 con 0x7f7ec01570d0 2024-01-15T18:07:12.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.181+0000 7f7ebd600640 1 --2- 172.21.15.27:0/2709858431 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7e98066b30 0x7f7e98068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:12.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.181+0000 7f7ebd600640 1 -- 172.21.15.27:0/2709858431 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f7eb8122ab0 con 0x7f7ec01570d0 2024-01-15T18:07:12.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.181+0000 7f7ec5a00640 1 --2- 172.21.15.27:0/2709858431 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7e98066b30 0x7f7e98068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:12.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.185+0000 7f7ec5a00640 1 --2- 172.21.15.27:0/2709858431 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7e98066b30 0x7f7e98068ff0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7eb0005890 tx=0x7f7eb0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:12.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.185+0000 7f7ebd600640 1 -- 172.21.15.27:0/2709858431 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7eb80efd00 con 0x7f7ec01570d0 2024-01-15T18:07:12.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.345+0000 7f7ec8400640 1 -- 172.21.15.27:0/2709858431 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ec006c270 con 0x7f7ec01570d0 2024-01-15T18:07:12.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.345+0000 7f7ebd600640 1 -- 172.21.15.27:0/2709858431 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+12933 (secure 0 0 0) 0x7f7eb80b4020 con 0x7f7ec01570d0 2024-01-15T18:07:12.352 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:07:12.352 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":37,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:07:06.979289+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6834/2187944510":"2024-01-16T18:07:03.967765+0000","172.21.15.27:6832/2187944510":"2024-01-16T18:07:03.967765+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:07:12.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.345+0000 7f7ec8400640 1 -- 172.21.15.27:0/2709858431 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7e98066b30 msgr2=0x7f7e98068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:12.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.345+0000 7f7ec8400640 1 --2- 172.21.15.27:0/2709858431 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7e98066b30 0x7f7e98068ff0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7eb0005890 tx=0x7f7eb0009040 comp rx=0 tx=0).stop 2024-01-15T18:07:12.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.345+0000 7f7ec8400640 1 -- 172.21.15.27:0/2709858431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec01570d0 msgr2=0x7f7ec0157580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:12.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.345+0000 7f7ec8400640 1 --2- 172.21.15.27:0/2709858431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec01570d0 0x7f7ec0157580 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f7eb80b2040 tx=0x7f7eb801bed0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.349+0000 7f7ec8400640 1 -- 172.21.15.27:0/2709858431 shutdown_connections 2024-01-15T18:07:12.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.349+0000 7f7ec8400640 1 --2- 172.21.15.27:0/2709858431 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7e98066b30 0x7f7e98068ff0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.349+0000 7f7ec8400640 1 --2- 172.21.15.27:0/2709858431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec01570d0 0x7f7ec0157580 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.349+0000 7f7ec8400640 1 --2- 172.21.15.27:0/2709858431 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ec0159bf0 0x7f7ec01f6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.349+0000 7f7ec8400640 1 --2- 172.21.15.27:0/2709858431 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ec006d430 0x7f7ec00c3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.349+0000 7f7ec8400640 1 -- 172.21.15.27:0/2709858431 >> 172.21.15.27:0/2709858431 conn(0x7f7ec00cec60 msgr2=0x7f7ec00ad9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:12.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.349+0000 7f7ec8400640 1 -- 172.21.15.27:0/2709858431 shutdown_connections 2024-01-15T18:07:12.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.349+0000 7f7ec8400640 1 -- 172.21.15.27:0/2709858431 wait complete. 2024-01-15T18:07:12.375 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:07:12.375 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:07:12.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.509+0000 7fc372800640 1 Processor -- start 2024-01-15T18:07:12.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.513+0000 7fc372800640 1 -- start start 2024-01-15T18:07:12.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.513+0000 7fc372800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc36c0cf450 0x7fc36c0d1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:12.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.513+0000 7fc372800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fc36c0b0650 0x7fc36c0b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:12.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.513+0000 7fc372800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc36c0cefa0 con 0x7fc36c0cba60 2024-01-15T18:07:12.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.513+0000 7fc372800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc36c0ce240 con 0x7fc36c0cf450 2024-01-15T18:07:12.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.513+0000 7fc372800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc36c0ccd70 con 0x7fc36c0b0650 2024-01-15T18:07:12.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.513+0000 7fc36b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc36c0cf450 0x7fc36c0d1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:12.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.513+0000 7fc36b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc36c0cf450 0x7fc36c0d1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34828/0 (socket says 172.21.15.27:34828) 2024-01-15T18:07:12.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.513+0000 7fc36aa00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc36c0cba60 0x7fc36c0cf830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57662/0 (socket says 172.21.15.27:57662) 2024-01-15T18:07:12.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.513+0000 7fc36b400640 1 -- 172.21.15.27:0/1247473651 learned_addr learned my addr 172.21.15.27:0/1247473651 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:12.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.513+0000 7fc36be00640 1 --2- 172.21.15.27:0/1247473651 >> v2:172.21.15.93:3300/0 conn(0x7fc36c0b0650 0x7fc36c0b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:12.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.513+0000 7fc36b400640 1 -- 172.21.15.27:0/1247473651 >> v2:172.21.15.93:3300/0 conn(0x7fc36c0b0650 msgr2=0x7fc36c0b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:12.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.513+0000 7fc36b400640 1 --2- 172.21.15.27:0/1247473651 >> v2:172.21.15.93:3300/0 conn(0x7fc36c0b0650 0x7fc36c0b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.513+0000 7fc36b400640 1 -- 172.21.15.27:0/1247473651 >> v1:172.21.15.27:6789/0 conn(0x7fc36c0cba60 legacy=0x7fc36c0cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:12.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.513+0000 7fc36b400640 1 -- 172.21.15.27:0/1247473651 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36c0cc1f0 con 0x7fc36c0cf450 2024-01-15T18:07:12.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.517+0000 7fc36b400640 1 --2- 172.21.15.27:0/1247473651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc36c0cf450 0x7fc36c0d1a40 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fc36c0717e0 tx=0x7fc35c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3296c89cc83f0b93 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:12.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.517+0000 7fc36a000640 1 -- 172.21.15.27:0/1247473651 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c0ab5f0 con 0x7fc36c0cf450 2024-01-15T18:07:12.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.517+0000 7fc36a000640 1 -- 172.21.15.27:0/1247473651 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35c0ab750 con 0x7fc36c0cf450 2024-01-15T18:07:12.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.517+0000 7fc36a000640 1 -- 172.21.15.27:0/1247473651 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c0abae0 con 0x7fc36c0cf450 2024-01-15T18:07:12.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.517+0000 7fc372800640 1 -- 172.21.15.27:0/1247473651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc36c0cf450 msgr2=0x7fc36c0d1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:12.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.517+0000 7fc372800640 1 --2- 172.21.15.27:0/1247473651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc36c0cf450 0x7fc36c0d1a40 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fc36c0717e0 tx=0x7fc35c01c500 comp rx=0 tx=0).stop 2024-01-15T18:07:12.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.517+0000 7fc372800640 1 -- 172.21.15.27:0/1247473651 shutdown_connections 2024-01-15T18:07:12.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.517+0000 7fc372800640 1 --2- 172.21.15.27:0/1247473651 >> v2:172.21.15.93:3300/0 conn(0x7fc36c0b0650 0x7fc36c0b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.517+0000 7fc372800640 1 --2- 172.21.15.27:0/1247473651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc36c0cf450 0x7fc36c0d1a40 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.517+0000 7fc372800640 1 -- 172.21.15.27:0/1247473651 >> 172.21.15.27:0/1247473651 conn(0x7fc36c0d0e70 msgr2=0x7fc36c0d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:12.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.517+0000 7fc372800640 1 -- 172.21.15.27:0/1247473651 shutdown_connections 2024-01-15T18:07:12.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.517+0000 7fc372800640 1 -- 172.21.15.27:0/1247473651 wait complete. 2024-01-15T18:07:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.517+0000 7fc372800640 1 Processor -- start 2024-01-15T18:07:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.521+0000 7fc372800640 1 -- start start 2024-01-15T18:07:12.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.521+0000 7fc372800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc36c0b0650 0x7fc36c1586e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:12.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.521+0000 7fc372800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc36c0cba60 0x7fc36c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:12.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.521+0000 7fc372800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc36c0cf450 0x7fc36c06b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:12.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.521+0000 7fc372800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c158c20 con 0x7fc36c0b0650 2024-01-15T18:07:12.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.521+0000 7fc372800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c0b5070 con 0x7fc36c0cf450 2024-01-15T18:07:12.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.521+0000 7fc372800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c0ae060 con 0x7fc36c0cba60 2024-01-15T18:07:12.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.521+0000 7fc36b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc36c0b0650 0x7fc36c1586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:12.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.521+0000 7fc36b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc36c0b0650 0x7fc36c1586e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34838/0 (socket says 172.21.15.27:34838) 2024-01-15T18:07:12.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.521+0000 7fc36b400640 1 -- 172.21.15.27:0/1560905940 learned_addr learned my addr 172.21.15.27:0/1560905940 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:12.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.521+0000 7fc36be00640 1 --2- 172.21.15.27:0/1560905940 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc36c0cf450 0x7fc36c06b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:12.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.521+0000 7fc36aa00640 1 --2- 172.21.15.27:0/1560905940 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc36c0cba60 0x7fc36c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:12.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.521+0000 7fc36b400640 1 -- 172.21.15.27:0/1560905940 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc36c0cba60 msgr2=0x7fc36c15ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:12.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.521+0000 7fc36b400640 1 --2- 172.21.15.27:0/1560905940 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc36c0cba60 0x7fc36c15ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.521+0000 7fc36b400640 1 -- 172.21.15.27:0/1560905940 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc36c0cf450 msgr2=0x7fc36c06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:12.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.521+0000 7fc36b400640 1 --2- 172.21.15.27:0/1560905940 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc36c0cf450 0x7fc36c06b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.521+0000 7fc36b400640 1 -- 172.21.15.27:0/1560905940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35c0094d0 con 0x7fc36c0b0650 2024-01-15T18:07:12.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.521+0000 7fc36b400640 1 --2- 172.21.15.27:0/1560905940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc36c0b0650 0x7fc36c1586e0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fc35c019110 tx=0x7fc35c01ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:12.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.521+0000 7fc363400640 1 -- 172.21.15.27:0/1560905940 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c0be5f0 con 0x7fc36c0b0650 2024-01-15T18:07:12.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.521+0000 7fc363400640 1 -- 172.21.15.27:0/1560905940 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35c0be750 con 0x7fc36c0b0650 2024-01-15T18:07:12.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.521+0000 7fc363400640 1 -- 172.21.15.27:0/1560905940 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35c0beb00 con 0x7fc36c0b0650 2024-01-15T18:07:12.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.521+0000 7fc372800640 1 -- 172.21.15.27:0/1560905940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc36c0ae340 con 0x7fc36c0b0650 2024-01-15T18:07:12.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.521+0000 7fc372800640 1 -- 172.21.15.27:0/1560905940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc36c0ae900 con 0x7fc36c0b0650 2024-01-15T18:07:12.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.521+0000 7fc372800640 1 -- 172.21.15.27:0/1560905940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc36c0b51b0 con 0x7fc36c0b0650 2024-01-15T18:07:12.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.525+0000 7fc363400640 1 -- 172.21.15.27:0/1560905940 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7fc35c0b43f0 con 0x7fc36c0b0650 2024-01-15T18:07:12.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.525+0000 7fc363400640 1 --2- 172.21.15.27:0/1560905940 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc344066b60 0x7fc344069020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:12.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.525+0000 7fc363400640 1 -- 172.21.15.27:0/1560905940 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fc35c0a8050 con 0x7fc36c0b0650 2024-01-15T18:07:12.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.525+0000 7fc36aa00640 1 --2- 172.21.15.27:0/1560905940 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc344066b60 0x7fc344069020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:12.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.525+0000 7fc36aa00640 1 --2- 172.21.15.27:0/1560905940 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc344066b60 0x7fc344069020 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc36c1f9750 tx=0x7fc354007c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:12.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.529+0000 7fc363400640 1 -- 172.21.15.27:0/1560905940 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc35c0ec1e0 con 0x7fc36c0b0650 2024-01-15T18:07:12.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.685+0000 7fc372800640 1 -- 172.21.15.27:0/1560905940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc36c04e850 con 0x7fc36c0b0650 2024-01-15T18:07:12.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.685+0000 7fc363400640 1 -- 172.21.15.27:0/1560905940 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc35c0ef800 con 0x7fc36c0b0650 2024-01-15T18:07:12.694 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:07:12.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.689+0000 7fc372800640 1 -- 172.21.15.27:0/1560905940 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc344066b60 msgr2=0x7fc344069020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:12.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.689+0000 7fc372800640 1 --2- 172.21.15.27:0/1560905940 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc344066b60 0x7fc344069020 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc36c1f9750 tx=0x7fc354007c90 comp rx=0 tx=0).stop 2024-01-15T18:07:12.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.689+0000 7fc372800640 1 -- 172.21.15.27:0/1560905940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc36c0b0650 msgr2=0x7fc36c1586e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:12.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.689+0000 7fc372800640 1 --2- 172.21.15.27:0/1560905940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc36c0b0650 0x7fc36c1586e0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fc35c019110 tx=0x7fc35c01ad40 comp rx=0 tx=0).stop 2024-01-15T18:07:12.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.689+0000 7fc372800640 1 -- 172.21.15.27:0/1560905940 shutdown_connections 2024-01-15T18:07:12.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.689+0000 7fc372800640 1 --2- 172.21.15.27:0/1560905940 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc344066b60 0x7fc344069020 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.689+0000 7fc372800640 1 --2- 172.21.15.27:0/1560905940 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc36c0cf450 0x7fc36c06b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.689+0000 7fc372800640 1 --2- 172.21.15.27:0/1560905940 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc36c0cba60 0x7fc36c15ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.689+0000 7fc372800640 1 --2- 172.21.15.27:0/1560905940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc36c0b0650 0x7fc36c1586e0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.689+0000 7fc372800640 1 -- 172.21.15.27:0/1560905940 >> 172.21.15.27:0/1560905940 conn(0x7fc36c0d0e70 msgr2=0x7fc36c06c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:12.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.689+0000 7fc372800640 1 -- 172.21.15.27:0/1560905940 shutdown_connections 2024-01-15T18:07:12.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.689+0000 7fc372800640 1 -- 172.21.15.27:0/1560905940 wait complete. 2024-01-15T18:07:12.716 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones' 2024-01-15T18:07:12.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.849+0000 7f8bbd200640 1 Processor -- start 2024-01-15T18:07:12.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.849+0000 7f8bbd200640 1 -- start start 2024-01-15T18:07:12.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.849+0000 7f8bbd200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8bb806db80 0x7f8bb806df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:12.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.849+0000 7f8bbd200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f8bb80cdd10 0x7f8bb8070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:12.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.849+0000 7f8bbd200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bb80ce920 con 0x7f8bb815e730 2024-01-15T18:07:12.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.849+0000 7f8bbd200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bb80cdb30 con 0x7f8bb806db80 2024-01-15T18:07:12.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.849+0000 7f8bbd200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bb806e4a0 con 0x7f8bb80cdd10 2024-01-15T18:07:12.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.849+0000 7f8bb5e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8bb806db80 0x7f8bb806df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:12.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.849+0000 7f8bb5400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8bb815e730 0x7f8bb80cd420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57674/0 (socket says 172.21.15.27:57674) 2024-01-15T18:07:12.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.849+0000 7f8bb5400640 1 -- 172.21.15.27:0/497686923 learned_addr learned my addr 172.21.15.27:0/497686923 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:12.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.849+0000 7f8bb6800640 1 --2- 172.21.15.27:0/497686923 >> v2:172.21.15.93:3300/0 conn(0x7f8bb80cdd10 0x7f8bb8070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:12.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.853+0000 7f8bb5e00640 1 -- 172.21.15.27:0/497686923 >> v2:172.21.15.93:3300/0 conn(0x7f8bb80cdd10 msgr2=0x7f8bb8070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:12.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.853+0000 7f8bb5e00640 1 --2- 172.21.15.27:0/497686923 >> v2:172.21.15.93:3300/0 conn(0x7f8bb80cdd10 0x7f8bb8070c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.853+0000 7f8bb5e00640 1 -- 172.21.15.27:0/497686923 >> v1:172.21.15.27:6789/0 conn(0x7f8bb815e730 legacy=0x7f8bb80cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:12.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.853+0000 7f8bb5e00640 1 -- 172.21.15.27:0/497686923 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb80ce640 con 0x7f8bb806db80 2024-01-15T18:07:12.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.853+0000 7f8bb4a00640 1 -- 172.21.15.27:0/497686923 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141168018 0 0) 0x7f8bb80ce920 con 0x7f8bb815e730 2024-01-15T18:07:12.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.853+0000 7f8bb5e00640 1 --2- 172.21.15.27:0/497686923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb806db80 0x7f8bb806df60 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f8bb815d9b0 tx=0x7f8ba801c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eee9371ae6a08601 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:12.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.853+0000 7f8bb4a00640 1 -- 172.21.15.27:0/497686923 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba80ab600 con 0x7f8bb806db80 2024-01-15T18:07:12.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.853+0000 7f8bb4a00640 1 -- 172.21.15.27:0/497686923 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba80ab760 con 0x7f8bb806db80 2024-01-15T18:07:12.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.853+0000 7f8bb4a00640 1 -- 172.21.15.27:0/497686923 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba80abaf0 con 0x7f8bb806db80 2024-01-15T18:07:12.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.853+0000 7f8bbd200640 1 -- 172.21.15.27:0/497686923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb806db80 msgr2=0x7f8bb806df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:12.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.853+0000 7f8bbd200640 1 --2- 172.21.15.27:0/497686923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb806db80 0x7f8bb806df60 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f8bb815d9b0 tx=0x7f8ba801c430 comp rx=0 tx=0).stop 2024-01-15T18:07:12.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.857+0000 7f8bbd200640 1 -- 172.21.15.27:0/497686923 shutdown_connections 2024-01-15T18:07:12.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.857+0000 7f8bbd200640 1 --2- 172.21.15.27:0/497686923 >> v2:172.21.15.93:3300/0 conn(0x7f8bb80cdd10 0x7f8bb8070c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.857+0000 7f8bbd200640 1 --2- 172.21.15.27:0/497686923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb806db80 0x7f8bb806df60 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.857+0000 7f8bbd200640 1 -- 172.21.15.27:0/497686923 >> 172.21.15.27:0/497686923 conn(0x7f8bb80cec60 msgr2=0x7f8bb80cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:12.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.857+0000 7f8bbd200640 1 -- 172.21.15.27:0/497686923 shutdown_connections 2024-01-15T18:07:12.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.857+0000 7f8bbd200640 1 -- 172.21.15.27:0/497686923 wait complete. 2024-01-15T18:07:12.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.857+0000 7f8bbd200640 1 Processor -- start 2024-01-15T18:07:12.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.857+0000 7f8bbd200640 1 -- start start 2024-01-15T18:07:12.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.857+0000 7f8bbd200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8bb806db80 0x7f8bb815cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:12.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.857+0000 7f8bbd200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8bb80cdd10 0x7f8bb81f9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:12.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.857+0000 7f8bbd200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb815e730 0x7f8bb81f9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:12.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.857+0000 7f8bbd200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb80b3ba0 con 0x7f8bb815e730 2024-01-15T18:07:12.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.857+0000 7f8bbd200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb815b790 con 0x7f8bb806db80 2024-01-15T18:07:12.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.857+0000 7f8bbd200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb815ba70 con 0x7f8bb80cdd10 2024-01-15T18:07:12.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.857+0000 7f8bb6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb815e730 0x7f8bb81f9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:12.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.857+0000 7f8bb6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb815e730 0x7f8bb81f9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34856/0 (socket says 172.21.15.27:34856) 2024-01-15T18:07:12.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.857+0000 7f8bb6800640 1 -- 172.21.15.27:0/536047291 learned_addr learned my addr 172.21.15.27:0/536047291 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:12.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.857+0000 7f8bb5e00640 1 --2- 172.21.15.27:0/536047291 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8bb806db80 0x7f8bb815cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:12.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.857+0000 7f8bb5400640 1 --2- 172.21.15.27:0/536047291 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8bb80cdd10 0x7f8bb81f9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:12.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.857+0000 7f8bb6800640 1 -- 172.21.15.27:0/536047291 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8bb80cdd10 msgr2=0x7f8bb81f9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:12.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.857+0000 7f8bb6800640 1 --2- 172.21.15.27:0/536047291 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8bb80cdd10 0x7f8bb81f9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.857+0000 7f8bb6800640 1 -- 172.21.15.27:0/536047291 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8bb806db80 msgr2=0x7f8bb815cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:12.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.857+0000 7f8bb6800640 1 --2- 172.21.15.27:0/536047291 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8bb806db80 0x7f8bb815cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:12.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.857+0000 7f8bb6800640 1 -- 172.21.15.27:0/536047291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba8009420 con 0x7f8bb815e730 2024-01-15T18:07:12.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.861+0000 7f8bb6800640 1 --2- 172.21.15.27:0/536047291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb815e730 0x7f8bb81f9650 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f8bb000efc0 tx=0x7f8bb000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.861+0000 7f8bae000640 1 -- 172.21.15.27:0/536047291 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb0014480 con 0x7f8bb815e730 2024-01-15T18:07:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.861+0000 7f8bae000640 1 -- 172.21.15.27:0/536047291 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb000f040 con 0x7f8bb815e730 2024-01-15T18:07:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.861+0000 7f8bbd200640 1 -- 172.21.15.27:0/536047291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bb815bd50 con 0x7f8bb815e730 2024-01-15T18:07:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.861+0000 7f8bae000640 1 -- 172.21.15.27:0/536047291 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb0014820 con 0x7f8bb815e730 2024-01-15T18:07:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.861+0000 7f8bbd200640 1 -- 172.21.15.27:0/536047291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bb815c310 con 0x7f8bb815e730 2024-01-15T18:07:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.861+0000 7f8bbd200640 1 -- 172.21.15.27:0/536047291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bb81516f0 con 0x7f8bb815e730 2024-01-15T18:07:12.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.861+0000 7f8bae000640 1 -- 172.21.15.27:0/536047291 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f8bb0014980 con 0x7f8bb815e730 2024-01-15T18:07:12.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.861+0000 7f8bae000640 1 --2- 172.21.15.27:0/536047291 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8b90066b30 0x7f8b90068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:12.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.861+0000 7f8bae000640 1 -- 172.21.15.27:0/536047291 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f8bb00853e0 con 0x7f8bb815e730 2024-01-15T18:07:12.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.861+0000 7f8bb5e00640 1 --2- 172.21.15.27:0/536047291 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8b90066b30 0x7f8b90068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:12.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.865+0000 7f8bb5e00640 1 --2- 172.21.15.27:0/536047291 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8b90066b30 0x7f8b90068ff0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8ba80af000 tx=0x7f8ba80a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:12.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:12.865+0000 7f8bae000640 1 -- 172.21.15.27:0/536047291 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8bb0052530 con 0x7f8bb815e730 2024-01-15T18:07:13.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:13.081+0000 7f8bbd200640 1 -- 172.21.15.27:0/536047291 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones"]} v 0) v1 -- 0x7f8bb804e850 con 0x7f8bb815e730 2024-01-15T18:07:14.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.009+0000 7f8bae000640 1 -- 172.21.15.27:0/536047291 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones"]}]=0 v134) v1 ==== 156+0+0 (secure 0 0 0) 0x7f8bb0055b50 con 0x7f8bb815e730 2024-01-15T18:07:14.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.013+0000 7f8bbd200640 1 -- 172.21.15.27:0/536047291 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8b90066b30 msgr2=0x7f8b90068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:14.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.013+0000 7f8bbd200640 1 --2- 172.21.15.27:0/536047291 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8b90066b30 0x7f8b90068ff0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8ba80af000 tx=0x7f8ba80a7000 comp rx=0 tx=0).stop 2024-01-15T18:07:14.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.013+0000 7f8bbd200640 1 -- 172.21.15.27:0/536047291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb815e730 msgr2=0x7f8bb81f9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:14.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.013+0000 7f8bbd200640 1 --2- 172.21.15.27:0/536047291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb815e730 0x7f8bb81f9650 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f8bb000efc0 tx=0x7f8bb000c490 comp rx=0 tx=0).stop 2024-01-15T18:07:14.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.013+0000 7f8bbd200640 1 -- 172.21.15.27:0/536047291 shutdown_connections 2024-01-15T18:07:14.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.013+0000 7f8bbd200640 1 --2- 172.21.15.27:0/536047291 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8b90066b30 0x7f8b90068ff0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:14.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.013+0000 7f8bbd200640 1 --2- 172.21.15.27:0/536047291 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8bb815e730 0x7f8bb81f9650 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:14.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.013+0000 7f8bbd200640 1 --2- 172.21.15.27:0/536047291 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8bb80cdd10 0x7f8bb81f9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:14.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.013+0000 7f8bbd200640 1 --2- 172.21.15.27:0/536047291 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8bb806db80 0x7f8bb815cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:14.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.013+0000 7f8bbd200640 1 -- 172.21.15.27:0/536047291 >> 172.21.15.27:0/536047291 conn(0x7f8bb80cec60 msgr2=0x7f8bb8164610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:14.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.013+0000 7f8bbd200640 1 -- 172.21.15.27:0/536047291 shutdown_connections 2024-01-15T18:07:14.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.013+0000 7f8bbd200640 1 -- 172.21.15.27:0/536047291 wait complete. 2024-01-15T18:07:14.043 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:07:14.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.185+0000 7f8031600640 1 Processor -- start 2024-01-15T18:07:14.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.185+0000 7f8031600640 1 -- start start 2024-01-15T18:07:14.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.185+0000 7f8031600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f802c15e240 0x7f802c160d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:14.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.185+0000 7f8031600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f802c15ddb0 con 0x7f802c070e40 2024-01-15T18:07:14.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.185+0000 7f8031600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f802c15e6b0 con 0x7f802c161240 2024-01-15T18:07:14.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.185+0000 7f8031600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f802c0b2760 con 0x7f802c15e240 2024-01-15T18:07:14.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.185+0000 7f802aa00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f802c070e40 0x7f802c071220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59310/0 (socket says 172.21.15.27:59310) 2024-01-15T18:07:14.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.185+0000 7f802aa00640 1 -- 172.21.15.27:0/92319209 learned_addr learned my addr 172.21.15.27:0/92319209 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:14.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.185+0000 7f802a000640 1 --2- 172.21.15.27:0/92319209 >> v2:172.21.15.93:3300/0 conn(0x7f802c15e240 0x7f802c160d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:14.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.185+0000 7f8029600640 1 -- 172.21.15.27:0/92319209 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449784057 0 0) 0x7f802c15ddb0 con 0x7f802c070e40 2024-01-15T18:07:14.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.185+0000 7f8029600640 1 -- 172.21.15.27:0/92319209 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f800c003530 con 0x7f802c070e40 2024-01-15T18:07:14.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.185+0000 7f802a000640 1 -- 172.21.15.27:0/92319209 >> v1:172.21.15.93:6789/0 conn(0x7f802c161240 legacy=0x7f802c06ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:14.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.185+0000 7f802a000640 1 -- 172.21.15.27:0/92319209 >> v1:172.21.15.27:6789/0 conn(0x7f802c070e40 legacy=0x7f802c071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:14.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.185+0000 7f802a000640 1 -- 172.21.15.27:0/92319209 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f802c161850 con 0x7f802c15e240 2024-01-15T18:07:14.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.189+0000 7f802a000640 1 --2- 172.21.15.27:0/92319209 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f802c15e240 0x7f802c160d00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f80140097c0 tx=0x7f801401c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba2703b2c62e76e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:14.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.189+0000 7f8029600640 1 -- 172.21.15.27:0/92319209 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80140ab5f0 con 0x7f802c15e240 2024-01-15T18:07:14.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.189+0000 7f8029600640 1 -- 172.21.15.27:0/92319209 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80140ab750 con 0x7f802c15e240 2024-01-15T18:07:14.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.189+0000 7f8029600640 1 -- 172.21.15.27:0/92319209 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80140abac0 con 0x7f802c15e240 2024-01-15T18:07:14.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.189+0000 7f8031600640 1 -- 172.21.15.27:0/92319209 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f802c15e240 msgr2=0x7f802c160d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:14.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.189+0000 7f8031600640 1 --2- 172.21.15.27:0/92319209 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f802c15e240 0x7f802c160d00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f80140097c0 tx=0x7f801401c3f0 comp rx=0 tx=0).stop 2024-01-15T18:07:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.189+0000 7f8031600640 1 -- 172.21.15.27:0/92319209 shutdown_connections 2024-01-15T18:07:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.189+0000 7f8031600640 1 --2- 172.21.15.27:0/92319209 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f802c15e240 0x7f802c160d00 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.189+0000 7f8031600640 1 -- 172.21.15.27:0/92319209 >> 172.21.15.27:0/92319209 conn(0x7f802c0b15b0 msgr2=0x7f802c0b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.189+0000 7f8031600640 1 -- 172.21.15.27:0/92319209 shutdown_connections 2024-01-15T18:07:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.189+0000 7f8031600640 1 -- 172.21.15.27:0/92319209 wait complete. 2024-01-15T18:07:14.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.193+0000 7f8031600640 1 Processor -- start 2024-01-15T18:07:14.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.193+0000 7f8031600640 1 -- start start 2024-01-15T18:07:14.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.193+0000 7f8031600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f802c070e40 0x7f802c13e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:14.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.193+0000 7f8031600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f802c15e240 0x7f802c13ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:14.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.193+0000 7f8031600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f802c161240 0x7f802c156220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:14.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.193+0000 7f8031600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802c0ac880 con 0x7f802c070e40 2024-01-15T18:07:14.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.193+0000 7f8031600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802c140510 con 0x7f802c161240 2024-01-15T18:07:14.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.193+0000 7f8031600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f802c13f0a0 con 0x7f802c15e240 2024-01-15T18:07:14.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.193+0000 7f802aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f802c070e40 0x7f802c13e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:14.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.193+0000 7f802aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f802c070e40 0x7f802c13e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32952/0 (socket says 172.21.15.27:32952) 2024-01-15T18:07:14.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.193+0000 7f802aa00640 1 -- 172.21.15.27:0/177728123 learned_addr learned my addr 172.21.15.27:0/177728123 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:14.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.193+0000 7f802a000640 1 --2- 172.21.15.27:0/177728123 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f802c15e240 0x7f802c13ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:14.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.193+0000 7f802b400640 1 --2- 172.21.15.27:0/177728123 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f802c161240 0x7f802c156220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:14.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.193+0000 7f802aa00640 1 -- 172.21.15.27:0/177728123 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f802c15e240 msgr2=0x7f802c13ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:14.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.193+0000 7f802aa00640 1 --2- 172.21.15.27:0/177728123 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f802c15e240 0x7f802c13ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:14.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.193+0000 7f802aa00640 1 -- 172.21.15.27:0/177728123 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f802c161240 msgr2=0x7f802c156220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:14.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.193+0000 7f802aa00640 1 --2- 172.21.15.27:0/177728123 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f802c161240 0x7f802c156220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:14.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.193+0000 7f802aa00640 1 -- 172.21.15.27:0/177728123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8014009420 con 0x7f802c070e40 2024-01-15T18:07:14.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.193+0000 7f802aa00640 1 --2- 172.21.15.27:0/177728123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f802c070e40 0x7f802c13e9e0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f801c00c500 tx=0x7f801c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:14.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.193+0000 7f8022a00640 1 -- 172.21.15.27:0/177728123 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801c005cf0 con 0x7f802c070e40 2024-01-15T18:07:14.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.193+0000 7f8022a00640 1 -- 172.21.15.27:0/177728123 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f801c005e50 con 0x7f802c070e40 2024-01-15T18:07:14.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.193+0000 7f8022a00640 1 -- 172.21.15.27:0/177728123 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801c0072f0 con 0x7f802c070e40 2024-01-15T18:07:14.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.193+0000 7f8031600640 1 -- 172.21.15.27:0/177728123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f802c13f380 con 0x7f802c070e40 2024-01-15T18:07:14.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.193+0000 7f8031600640 1 -- 172.21.15.27:0/177728123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f802c0acad0 con 0x7f802c070e40 2024-01-15T18:07:14.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.197+0000 7f8031600640 1 -- 172.21.15.27:0/177728123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f802c073b90 con 0x7f802c070e40 2024-01-15T18:07:14.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.197+0000 7f8022a00640 1 -- 172.21.15.27:0/177728123 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f801c007450 con 0x7f802c070e40 2024-01-15T18:07:14.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.197+0000 7f8022a00640 1 --2- 172.21.15.27:0/177728123 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8004066b30 0x7f8004068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:14.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.197+0000 7f8022a00640 1 -- 172.21.15.27:0/177728123 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f801c083b80 con 0x7f802c070e40 2024-01-15T18:07:14.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.197+0000 7f802a000640 1 --2- 172.21.15.27:0/177728123 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8004066b30 0x7f8004068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:14.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.197+0000 7f802a000640 1 --2- 172.21.15.27:0/177728123 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8004066b30 0x7f8004068ff0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f80140af000 tx=0x7f80140a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:14.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.201+0000 7f8022a00640 1 -- 172.21.15.27:0/177728123 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f801c050de0 con 0x7f802c070e40 2024-01-15T18:07:14.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.357+0000 7f8031600640 1 -- 172.21.15.27:0/177728123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f802c0accc0 con 0x7f802c070e40 2024-01-15T18:07:14.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.361+0000 7f8022a00640 1 -- 172.21.15.27:0/177728123 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f801c083e60 con 0x7f802c070e40 2024-01-15T18:07:14.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.365+0000 7f8031600640 1 -- 172.21.15.27:0/177728123 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8004066b30 msgr2=0x7f8004068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:14.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.365+0000 7f8031600640 1 --2- 172.21.15.27:0/177728123 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8004066b30 0x7f8004068ff0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f80140af000 tx=0x7f80140a7000 comp rx=0 tx=0).stop 2024-01-15T18:07:14.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.365+0000 7f8031600640 1 -- 172.21.15.27:0/177728123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f802c070e40 msgr2=0x7f802c13e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:14.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.365+0000 7f8031600640 1 --2- 172.21.15.27:0/177728123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f802c070e40 0x7f802c13e9e0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f801c00c500 tx=0x7f801c00c9c0 comp rx=0 tx=0).stop 2024-01-15T18:07:14.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.365+0000 7f8031600640 1 -- 172.21.15.27:0/177728123 shutdown_connections 2024-01-15T18:07:14.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.365+0000 7f8031600640 1 --2- 172.21.15.27:0/177728123 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8004066b30 0x7f8004068ff0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:14.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.365+0000 7f8031600640 1 --2- 172.21.15.27:0/177728123 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f802c161240 0x7f802c156220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:14.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.365+0000 7f8031600640 1 --2- 172.21.15.27:0/177728123 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f802c15e240 0x7f802c13ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:14.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.365+0000 7f8031600640 1 --2- 172.21.15.27:0/177728123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f802c070e40 0x7f802c13e9e0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:14.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.365+0000 7f8031600640 1 -- 172.21.15.27:0/177728123 >> 172.21.15.27:0/177728123 conn(0x7f802c0b15b0 msgr2=0x7f802c06d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:14.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.365+0000 7f8031600640 1 -- 172.21.15.27:0/177728123 shutdown_connections 2024-01-15T18:07:14.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.365+0000 7f8031600640 1 -- 172.21.15.27:0/177728123 wait complete. 2024-01-15T18:07:14.392 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T18:07:14.392 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:07:14.493 DEBUG:teuthology.orchestra.run.smithi093:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:07:14.561 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T18:07:14.561 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:07:14.561 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:07:14.570 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354580k,nr_inodes=4088645,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9249 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:07:14.570 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:07:14.570 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:07:14.579 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354484k,nr_inodes=4088621,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8403 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:07:14.579 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:07:14.579 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:07:14.622 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354580k,nr_inodes=4088645,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9249 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:07:14.622 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:07:14.622 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:07:14.630 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354484k,nr_inodes=4088621,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8403 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:07:14.630 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-15T18:07:14.631 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:07:14.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.797+0000 7eff29200640 1 Processor -- start 2024-01-15T18:07:14.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.797+0000 7eff29200640 1 -- start start 2024-01-15T18:07:14.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.801+0000 7eff29200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7eff2406db80 0x7eff2406df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:14.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.801+0000 7eff29200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff240ce920 con 0x7eff2415e730 2024-01-15T18:07:14.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.801+0000 7eff29200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff2406e4a0 con 0x7eff240cdd10 2024-01-15T18:07:14.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.801+0000 7eff29200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff240cdb30 con 0x7eff2406db80 2024-01-15T18:07:14.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.801+0000 7eff21e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7eff2406db80 0x7eff2406df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:14.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.801+0000 7eff21e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7eff2406db80 0x7eff2406df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32962/0 (socket says 172.21.15.27:32962) 2024-01-15T18:07:14.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.801+0000 7eff21e00640 1 -- 172.21.15.27:0/931322857 learned_addr learned my addr 172.21.15.27:0/931322857 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:14.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.801+0000 7eff21400640 1 --1- 172.21.15.27:0/931322857 >> v1:172.21.15.27:6789/0 conn(0x7eff2415e730 0x7eff240cd420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59324/0 (socket says 172.21.15.27:59324) 2024-01-15T18:07:14.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.801+0000 7eff21e00640 1 -- 172.21.15.27:0/931322857 >> v1:172.21.15.93:6789/0 conn(0x7eff240cdd10 legacy=0x7eff24070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:14.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.801+0000 7eff21e00640 1 -- 172.21.15.27:0/931322857 >> v1:172.21.15.27:6789/0 conn(0x7eff2415e730 legacy=0x7eff240cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:14.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.801+0000 7eff21e00640 1 -- 172.21.15.27:0/931322857 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff240ce640 con 0x7eff2406db80 2024-01-15T18:07:14.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.805+0000 7eff21e00640 1 --2- 172.21.15.27:0/931322857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff2406db80 0x7eff2406df60 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7eff14009870 tx=0x7eff1401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3f65c7f768eb0bb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:14.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.805+0000 7eff20a00640 1 -- 172.21.15.27:0/931322857 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff140aa430 con 0x7eff2406db80 2024-01-15T18:07:14.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.805+0000 7eff20a00640 1 -- 172.21.15.27:0/931322857 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff140aa590 con 0x7eff2406db80 2024-01-15T18:07:14.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.805+0000 7eff20a00640 1 -- 172.21.15.27:0/931322857 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff140aa920 con 0x7eff2406db80 2024-01-15T18:07:14.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.805+0000 7eff29200640 1 -- 172.21.15.27:0/931322857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff2406db80 msgr2=0x7eff2406df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:14.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.805+0000 7eff29200640 1 --2- 172.21.15.27:0/931322857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff2406db80 0x7eff2406df60 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7eff14009870 tx=0x7eff1401c4d0 comp rx=0 tx=0).stop 2024-01-15T18:07:14.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.805+0000 7eff29200640 1 -- 172.21.15.27:0/931322857 shutdown_connections 2024-01-15T18:07:14.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.805+0000 7eff29200640 1 --2- 172.21.15.27:0/931322857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff2406db80 0x7eff2406df60 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:14.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.805+0000 7eff29200640 1 -- 172.21.15.27:0/931322857 >> 172.21.15.27:0/931322857 conn(0x7eff240cec60 msgr2=0x7eff240cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:14.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.805+0000 7eff29200640 1 -- 172.21.15.27:0/931322857 shutdown_connections 2024-01-15T18:07:14.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.805+0000 7eff29200640 1 -- 172.21.15.27:0/931322857 wait complete. 2024-01-15T18:07:14.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.805+0000 7eff29200640 1 Processor -- start 2024-01-15T18:07:14.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.805+0000 7eff29200640 1 -- start start 2024-01-15T18:07:14.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.809+0000 7eff29200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7eff2406db80 0x7eff240c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:14.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.809+0000 7eff29200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff240cdd10 0x7eff240c78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:14.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.809+0000 7eff29200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7eff2415e730 0x7eff24153be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:14.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.809+0000 7eff29200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff240c83c0 con 0x7eff240cdd10 2024-01-15T18:07:14.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.809+0000 7eff29200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff240c86a0 con 0x7eff2415e730 2024-01-15T18:07:14.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.809+0000 7eff29200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff240c8980 con 0x7eff2406db80 2024-01-15T18:07:14.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.809+0000 7eff21400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff240cdd10 0x7eff240c78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:14.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.809+0000 7eff21400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff240cdd10 0x7eff240c78c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32966/0 (socket says 172.21.15.27:32966) 2024-01-15T18:07:14.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.809+0000 7eff21400640 1 -- 172.21.15.27:0/2403947559 learned_addr learned my addr 172.21.15.27:0/2403947559 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:14.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.809+0000 7eff21e00640 1 --2- 172.21.15.27:0/2403947559 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7eff2406db80 0x7eff240c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:14.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.809+0000 7eff22800640 1 --2- 172.21.15.27:0/2403947559 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7eff2415e730 0x7eff24153be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:14.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.809+0000 7eff21400640 1 -- 172.21.15.27:0/2403947559 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7eff2406db80 msgr2=0x7eff240c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:14.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.809+0000 7eff21400640 1 --2- 172.21.15.27:0/2403947559 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7eff2406db80 0x7eff240c4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:14.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.809+0000 7eff21400640 1 -- 172.21.15.27:0/2403947559 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7eff2415e730 msgr2=0x7eff24153be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:14.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.809+0000 7eff21400640 1 --2- 172.21.15.27:0/2403947559 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7eff2415e730 0x7eff24153be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:14.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.809+0000 7eff21400640 1 -- 172.21.15.27:0/2403947559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff140094d0 con 0x7eff240cdd10 2024-01-15T18:07:14.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.809+0000 7eff21400640 1 --2- 172.21.15.27:0/2403947559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff240cdd10 0x7eff240c78c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7eff0c00c490 tx=0x7eff0c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:14.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.809+0000 7eff1a000640 1 -- 172.21.15.27:0/2403947559 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff0c007480 con 0x7eff240cdd10 2024-01-15T18:07:14.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.809+0000 7eff1a000640 1 -- 172.21.15.27:0/2403947559 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff0c0075e0 con 0x7eff240cdd10 2024-01-15T18:07:14.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.809+0000 7eff1a000640 1 -- 172.21.15.27:0/2403947559 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff0c004a10 con 0x7eff240cdd10 2024-01-15T18:07:14.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.809+0000 7eff29200640 1 -- 172.21.15.27:0/2403947559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff241536c0 con 0x7eff240cdd10 2024-01-15T18:07:14.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.809+0000 7eff29200640 1 -- 172.21.15.27:0/2403947559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff240c8c60 con 0x7eff240cdd10 2024-01-15T18:07:14.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.809+0000 7eff29200640 1 -- 172.21.15.27:0/2403947559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff24153520 con 0x7eff240cdd10 2024-01-15T18:07:14.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.813+0000 7eff1a000640 1 -- 172.21.15.27:0/2403947559 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7eff0c004b70 con 0x7eff240cdd10 2024-01-15T18:07:14.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.813+0000 7eff1a000640 1 --2- 172.21.15.27:0/2403947559 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efefc066b30 0x7efefc068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:14.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.813+0000 7eff1a000640 1 -- 172.21.15.27:0/2403947559 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4971+0+0 (secure 0 0 0) 0x7eff0c083810 con 0x7eff240cdd10 2024-01-15T18:07:14.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.813+0000 7eff21e00640 1 --2- 172.21.15.27:0/2403947559 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efefc066b30 0x7efefc068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:14.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.813+0000 7eff21e00640 1 --2- 172.21.15.27:0/2403947559 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efefc066b30 0x7efefc068ff0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7eff140ae000 tx=0x7eff140a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:14.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:14.817+0000 7eff1a000640 1 -- 172.21.15.27:0/2403947559 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7eff24153520 con 0x7eff240cdd10 2024-01-15T18:07:15.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.005+0000 7eff29200640 1 -- 172.21.15.27:0/2403947559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff24153520 con 0x7eff240cdd10 2024-01-15T18:07:15.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.005+0000 7eff1a000640 1 -- 172.21.15.27:0/2403947559 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+3699 (secure 0 0 0) 0x7eff24153520 con 0x7eff240cdd10 2024-01-15T18:07:15.013 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:07:15.013 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4910,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4204677709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4204677709},{"type":"v1","addr":"172.21.15.27:6835","nonce":4204677709}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4913,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/1766105879","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":1766105879},{"type":"v1","addr":"172.21.15.93:6836","nonce":1766105879}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/2799226725","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":2799226725},{"type":"v1","addr":"172.21.15.93:6837","nonce":2799226725}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5688,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1337712144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1337712144},{"type":"v1","addr":"172.21.15.27:6834","nonce":1337712144}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[]} 2024-01-15T18:07:15.013 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 20 2024-01-15T18:07:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.009+0000 7eff29200640 1 -- 172.21.15.27:0/2403947559 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efefc066b30 msgr2=0x7efefc068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.009+0000 7eff29200640 1 --2- 172.21.15.27:0/2403947559 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efefc066b30 0x7efefc068ff0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7eff140ae000 tx=0x7eff140a7000 comp rx=0 tx=0).stop 2024-01-15T18:07:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.009+0000 7eff29200640 1 -- 172.21.15.27:0/2403947559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff240cdd10 msgr2=0x7eff240c78c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.009+0000 7eff29200640 1 --2- 172.21.15.27:0/2403947559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff240cdd10 0x7eff240c78c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7eff0c00c490 tx=0x7eff0c00c950 comp rx=0 tx=0).stop 2024-01-15T18:07:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.009+0000 7eff29200640 1 -- 172.21.15.27:0/2403947559 shutdown_connections 2024-01-15T18:07:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.009+0000 7eff29200640 1 --2- 172.21.15.27:0/2403947559 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efefc066b30 0x7efefc068ff0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.009+0000 7eff29200640 1 --2- 172.21.15.27:0/2403947559 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7eff2415e730 0x7eff24153be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:15.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.009+0000 7eff29200640 1 --2- 172.21.15.27:0/2403947559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff240cdd10 0x7eff240c78c0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:15.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.009+0000 7eff29200640 1 --2- 172.21.15.27:0/2403947559 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7eff2406db80 0x7eff240c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:15.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.009+0000 7eff29200640 1 -- 172.21.15.27:0/2403947559 >> 172.21.15.27:0/2403947559 conn(0x7eff240cec60 msgr2=0x7eff240ae860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:15.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.009+0000 7eff29200640 1 -- 172.21.15.27:0/2403947559 shutdown_connections 2024-01-15T18:07:15.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.009+0000 7eff29200640 1 -- 172.21.15.27:0/2403947559 wait complete. 2024-01-15T18:07:15.036 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-15T18:07:15.037 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-15T18:07:15.037 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-15T18:07:15.037 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:07:15.037 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-15T18:07:15.037 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-15T18:07:15.037 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-15T18:07:15.037 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:07:15.037 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-15T18:07:15.038 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-15T18:07:15.038 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-15T18:07:15.038 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:07:15.038 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-15T18:07:15.038 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-15T18:07:15.038 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-15T18:07:15.038 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:07:15.052 INFO:tasks.ceph.mds.a:Stopped 2024-01-15T18:07:15.053 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-15T18:07:15.053 INFO:tasks.ceph.mds.c:Stopped 2024-01-15T18:07:15.053 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-15T18:07:15.054 INFO:tasks.ceph.mds.d:Stopped 2024-01-15T18:07:15.055 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-15T18:07:15.055 INFO:tasks.ceph.mds.b:Stopped 2024-01-15T18:07:15.055 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-15T18:07:15.100 INFO:tasks.ceph.mds.a:Started 2024-01-15T18:07:15.101 DEBUG:teuthology.parallel:result is None 2024-01-15T18:07:15.103 INFO:tasks.ceph.mds.c:Started 2024-01-15T18:07:15.103 DEBUG:teuthology.parallel:result is None 2024-01-15T18:07:15.105 INFO:tasks.ceph.mds.d:Started 2024-01-15T18:07:15.105 DEBUG:teuthology.parallel:result is None 2024-01-15T18:07:15.107 INFO:tasks.ceph.mds.b:Started 2024-01-15T18:07:15.107 DEBUG:teuthology.parallel:result is None 2024-01-15T18:07:15.107 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-15T18:07:15.187 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-15T18:07:15.187 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-15T18:07:15.198 INFO:tasks.ceph.mds.b.smithi093.stdout:starting mds.b at 2024-01-15T18:07:15.199 INFO:tasks.ceph.mds.d.smithi093.stdout:starting mds.d at 2024-01-15T18:07:15.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.249+0000 7fb560000640 1 Processor -- start 2024-01-15T18:07:15.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.249+0000 7fb560000640 1 -- start start 2024-01-15T18:07:15.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.249+0000 7fb560000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb55815d2a0 0x7fb5580cfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:15.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.249+0000 7fb560000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5580b3cd0 con 0x7fb55815d680 2024-01-15T18:07:15.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.249+0000 7fb560000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5580d0260 con 0x7fb5580cde10 2024-01-15T18:07:15.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.249+0000 7fb560000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5580cb670 con 0x7fb55815d2a0 2024-01-15T18:07:15.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.249+0000 7fb55d600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb55815d2a0 0x7fb5580cfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:15.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.249+0000 7fb55cc00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb55815d680 0x7fb5580cd700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59330/0 (socket says 172.21.15.27:59330) 2024-01-15T18:07:15.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.249+0000 7fb55cc00640 1 -- 172.21.15.27:0/4131885718 learned_addr learned my addr 172.21.15.27:0/4131885718 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:15.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.249+0000 7fb55d600640 1 -- 172.21.15.27:0/4131885718 >> v1:172.21.15.93:6789/0 conn(0x7fb5580cde10 legacy=0x7fb5580cc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:15.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.249+0000 7fb55d600640 1 -- 172.21.15.27:0/4131885718 >> v1:172.21.15.27:6789/0 conn(0x7fb55815d680 legacy=0x7fb5580cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:15.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.249+0000 7fb55d600640 1 -- 172.21.15.27:0/4131885718 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5580cb950 con 0x7fb55815d2a0 2024-01-15T18:07:15.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.253+0000 7fb55d600640 1 --2- 172.21.15.27:0/4131885718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb55815d2a0 0x7fb5580cfbd0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fb5480097c0 tx=0x7fb54801c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf3f32d79f4c0bcd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:15.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.253+0000 7fb557e00640 1 -- 172.21.15.27:0/4131885718 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5480ad660 con 0x7fb55815d2a0 2024-01-15T18:07:15.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.253+0000 7fb557e00640 1 -- 172.21.15.27:0/4131885718 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5480ad7c0 con 0x7fb55815d2a0 2024-01-15T18:07:15.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.253+0000 7fb557e00640 1 -- 172.21.15.27:0/4131885718 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5480adb50 con 0x7fb55815d2a0 2024-01-15T18:07:15.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.253+0000 7fb560000640 1 -- 172.21.15.27:0/4131885718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb55815d2a0 msgr2=0x7fb5580cfbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:15.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.253+0000 7fb560000640 1 --2- 172.21.15.27:0/4131885718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb55815d2a0 0x7fb5580cfbd0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fb5480097c0 tx=0x7fb54801c380 comp rx=0 tx=0).stop 2024-01-15T18:07:15.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.253+0000 7fb560000640 1 -- 172.21.15.27:0/4131885718 shutdown_connections 2024-01-15T18:07:15.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.253+0000 7fb560000640 1 --2- 172.21.15.27:0/4131885718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb55815d2a0 0x7fb5580cfbd0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:15.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.253+0000 7fb560000640 1 -- 172.21.15.27:0/4131885718 >> 172.21.15.27:0/4131885718 conn(0x7fb5580d0e80 msgr2=0x7fb5580d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:15.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.253+0000 7fb560000640 1 -- 172.21.15.27:0/4131885718 shutdown_connections 2024-01-15T18:07:15.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.253+0000 7fb560000640 1 -- 172.21.15.27:0/4131885718 wait complete. 2024-01-15T18:07:15.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.257+0000 7fb560000640 1 Processor -- start 2024-01-15T18:07:15.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.257+0000 7fb560000640 1 -- start start 2024-01-15T18:07:15.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.257+0000 7fb560000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb5580cde10 0x7fb55814b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:15.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.257+0000 7fb560000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb55815d2a0 0x7fb558150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:15.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.257+0000 7fb560000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb55815d680 0x7fb5581506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:15.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.257+0000 7fb560000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55815da90 con 0x7fb55815d680 2024-01-15T18:07:15.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.257+0000 7fb560000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55815e610 con 0x7fb55815d2a0 2024-01-15T18:07:15.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.257+0000 7fb560000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb55815dce0 con 0x7fb5580cde10 2024-01-15T18:07:15.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.257+0000 7fb55e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb55815d680 0x7fb5581506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:15.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.257+0000 7fb55e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb55815d680 0x7fb5581506a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33014/0 (socket says 172.21.15.27:33014) 2024-01-15T18:07:15.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.257+0000 7fb55e000640 1 -- 172.21.15.27:0/429550240 learned_addr learned my addr 172.21.15.27:0/429550240 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:15.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.257+0000 7fb55d600640 1 --2- 172.21.15.27:0/429550240 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb5580cde10 0x7fb55814b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:15.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.257+0000 7fb55cc00640 1 --2- 172.21.15.27:0/429550240 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb55815d2a0 0x7fb558150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:15.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.257+0000 7fb55e000640 1 -- 172.21.15.27:0/429550240 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb5580cde10 msgr2=0x7fb55814b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:15.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.257+0000 7fb55e000640 1 --2- 172.21.15.27:0/429550240 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb5580cde10 0x7fb55814b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:15.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.257+0000 7fb55e000640 1 -- 172.21.15.27:0/429550240 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb55815d2a0 msgr2=0x7fb558150160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:15.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.257+0000 7fb55e000640 1 --2- 172.21.15.27:0/429550240 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb55815d2a0 0x7fb558150160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:15.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.257+0000 7fb55e000640 1 -- 172.21.15.27:0/429550240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb548009420 con 0x7fb55815d680 2024-01-15T18:07:15.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.257+0000 7fb55e000640 1 --2- 172.21.15.27:0/429550240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb55815d680 0x7fb5581506a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fb55000c780 tx=0x7fb55000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:15.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.257+0000 7fb555600640 1 -- 172.21.15.27:0/429550240 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb550004b70 con 0x7fb55815d680 2024-01-15T18:07:15.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.257+0000 7fb555600640 1 -- 172.21.15.27:0/429550240 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55000ce90 con 0x7fb55815d680 2024-01-15T18:07:15.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.257+0000 7fb560000640 1 -- 172.21.15.27:0/429550240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb55815dfc0 con 0x7fb55815d680 2024-01-15T18:07:15.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.257+0000 7fb555600640 1 -- 172.21.15.27:0/429550240 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55000a850 con 0x7fb55815d680 2024-01-15T18:07:15.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.257+0000 7fb560000640 1 -- 172.21.15.27:0/429550240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb55814b8f0 con 0x7fb55815d680 2024-01-15T18:07:15.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.257+0000 7fb560000640 1 -- 172.21.15.27:0/429550240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb55815e780 con 0x7fb55815d680 2024-01-15T18:07:15.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.261+0000 7fb555600640 1 -- 172.21.15.27:0/429550240 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7fb550006b60 con 0x7fb55815d680 2024-01-15T18:07:15.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.261+0000 7fb555600640 1 --2- 172.21.15.27:0/429550240 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb530066b30 0x7fb530068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:15.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.261+0000 7fb555600640 1 -- 172.21.15.27:0/429550240 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fb550084c70 con 0x7fb55815d680 2024-01-15T18:07:15.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.261+0000 7fb55d600640 1 --2- 172.21.15.27:0/429550240 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb530066b30 0x7fb530068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:15.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.261+0000 7fb55d600640 1 --2- 172.21.15.27:0/429550240 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb530066b30 0x7fb530068ff0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb5480af000 tx=0x7fb5480a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:15.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.265+0000 7fb555600640 1 -- 172.21.15.27:0/429550240 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb550051fc0 con 0x7fb55815d680 2024-01-15T18:07:15.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:15.437+0000 7fb560000640 1 -- 172.21.15.27:0/429550240 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fb55804e850 con 0x7fb55815d680 2024-01-15T18:07:16.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.029+0000 7fb555600640 1 -- 172.21.15.27:0/429550240 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v39) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb5500555e0 con 0x7fb55815d680 2024-01-15T18:07:16.036 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-01-15T18:07:16.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.029+0000 7fb560000640 1 -- 172.21.15.27:0/429550240 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb530066b30 msgr2=0x7fb530068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:16.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.029+0000 7fb560000640 1 --2- 172.21.15.27:0/429550240 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb530066b30 0x7fb530068ff0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb5480af000 tx=0x7fb5480a7000 comp rx=0 tx=0).stop 2024-01-15T18:07:16.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.029+0000 7fb560000640 1 -- 172.21.15.27:0/429550240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb55815d680 msgr2=0x7fb5581506a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:16.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.029+0000 7fb560000640 1 --2- 172.21.15.27:0/429550240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb55815d680 0x7fb5581506a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fb55000c780 tx=0x7fb55000cc40 comp rx=0 tx=0).stop 2024-01-15T18:07:16.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.029+0000 7fb560000640 1 -- 172.21.15.27:0/429550240 shutdown_connections 2024-01-15T18:07:16.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.029+0000 7fb560000640 1 --2- 172.21.15.27:0/429550240 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb530066b30 0x7fb530068ff0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.029+0000 7fb560000640 1 --2- 172.21.15.27:0/429550240 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb55815d680 0x7fb5581506a0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.029+0000 7fb560000640 1 --2- 172.21.15.27:0/429550240 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb55815d2a0 0x7fb558150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.029+0000 7fb560000640 1 --2- 172.21.15.27:0/429550240 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb5580cde10 0x7fb55814b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.029+0000 7fb560000640 1 -- 172.21.15.27:0/429550240 >> 172.21.15.27:0/429550240 conn(0x7fb5580d0e80 msgr2=0x7fb5580cf910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:16.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.033+0000 7fb560000640 1 -- 172.21.15.27:0/429550240 shutdown_connections 2024-01-15T18:07:16.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.033+0000 7fb560000640 1 -- 172.21.15.27:0/429550240 wait complete. 2024-01-15T18:07:16.062 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-15T18:07:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.197+0000 7f66cd800640 1 Processor -- start 2024-01-15T18:07:16.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.197+0000 7f66cd800640 1 -- start start 2024-01-15T18:07:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.197+0000 7f66cd800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f66c806a4f0 0x7f66c80d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.197+0000 7f66cd800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f66c80ca3f0 0x7f66c806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.197+0000 7f66cd800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66c815b470 0x7f66c815b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.197+0000 7f66cd800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66c806ece0 con 0x7f66c80ca3f0 2024-01-15T18:07:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.197+0000 7f66cd800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66c815be40 con 0x7f66c806a4f0 2024-01-15T18:07:16.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.197+0000 7f66cd800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66c806e6a0 con 0x7f66c815b470 2024-01-15T18:07:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.197+0000 7f66c6000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f66c80ca3f0 0x7f66c806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.197+0000 7f66c6000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f66c80ca3f0 0x7f66c806a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33022/0 (socket says 172.21.15.27:33022) 2024-01-15T18:07:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.197+0000 7f66c6000640 1 -- 172.21.15.27:0/1184480030 learned_addr learned my addr 172.21.15.27:0/1184480030 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.197+0000 7f66c7400640 1 --2- 172.21.15.27:0/1184480030 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66c815b470 0x7f66c815b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.197+0000 7f66c6a00640 1 --2- 172.21.15.27:0/1184480030 >> v2:172.21.15.93:3300/0 conn(0x7f66c806a4f0 0x7f66c80d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.197+0000 7f66c6000640 1 -- 172.21.15.27:0/1184480030 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66c815b470 msgr2=0x7f66c815b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.197+0000 7f66c6000640 1 --2- 172.21.15.27:0/1184480030 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66c815b470 0x7f66c815b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.197+0000 7f66c6000640 1 -- 172.21.15.27:0/1184480030 >> v2:172.21.15.93:3300/0 conn(0x7f66c806a4f0 msgr2=0x7f66c80d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.197+0000 7f66c6000640 1 --2- 172.21.15.27:0/1184480030 >> v2:172.21.15.93:3300/0 conn(0x7f66c806a4f0 0x7f66c80d08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.197+0000 7f66c6000640 1 -- 172.21.15.27:0/1184480030 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c806ae10 con 0x7f66c80ca3f0 2024-01-15T18:07:16.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.201+0000 7f66c6000640 1 --2- 172.21.15.27:0/1184480030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66c80ca3f0 0x7f66c806a8d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f66c806cd20 tx=0x7f66b001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb98fcb3e38cebd5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:16.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.201+0000 7f66c5600640 1 -- 172.21.15.27:0/1184480030 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b00ab600 con 0x7f66c80ca3f0 2024-01-15T18:07:16.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.201+0000 7f66c5600640 1 -- 172.21.15.27:0/1184480030 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b00a7050 con 0x7f66c80ca3f0 2024-01-15T18:07:16.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.201+0000 7f66c5600640 1 -- 172.21.15.27:0/1184480030 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b00ab980 con 0x7f66c80ca3f0 2024-01-15T18:07:16.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.201+0000 7f66cd800640 1 -- 172.21.15.27:0/1184480030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66c80ca3f0 msgr2=0x7f66c806a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:16.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.201+0000 7f66cd800640 1 --2- 172.21.15.27:0/1184480030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66c80ca3f0 0x7f66c806a8d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f66c806cd20 tx=0x7f66b001c470 comp rx=0 tx=0).stop 2024-01-15T18:07:16.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.201+0000 7f66cd800640 1 -- 172.21.15.27:0/1184480030 shutdown_connections 2024-01-15T18:07:16.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.201+0000 7f66cd800640 1 --2- 172.21.15.27:0/1184480030 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66c815b470 0x7f66c815b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.201+0000 7f66cd800640 1 --2- 172.21.15.27:0/1184480030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66c80ca3f0 0x7f66c806a8d0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.201+0000 7f66cd800640 1 --2- 172.21.15.27:0/1184480030 >> v2:172.21.15.93:3300/0 conn(0x7f66c806a4f0 0x7f66c80d08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.201+0000 7f66cd800640 1 -- 172.21.15.27:0/1184480030 >> 172.21.15.27:0/1184480030 conn(0x7f66c80cfda0 msgr2=0x7f66c80d01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:16.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.201+0000 7f66cd800640 1 -- 172.21.15.27:0/1184480030 shutdown_connections 2024-01-15T18:07:16.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.201+0000 7f66cd800640 1 -- 172.21.15.27:0/1184480030 wait complete. 2024-01-15T18:07:16.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66cd800640 1 Processor -- start 2024-01-15T18:07:16.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66cd800640 1 -- start start 2024-01-15T18:07:16.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66cd800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66c806a4f0 0x7f66c8156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:16.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66cd800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66c80ca3f0 0x7f66c80cf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:16.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66cd800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66c815b470 0x7f66c80b02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:16.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66cd800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c81585b0 con 0x7f66c806a4f0 2024-01-15T18:07:16.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66cd800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c8158890 con 0x7f66c815b470 2024-01-15T18:07:16.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66cd800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66c8164230 con 0x7f66c80ca3f0 2024-01-15T18:07:16.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66c6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66c806a4f0 0x7f66c8156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:16.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66c6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66c806a4f0 0x7f66c8156770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33028/0 (socket says 172.21.15.27:33028) 2024-01-15T18:07:16.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66c6a00640 1 -- 172.21.15.27:0/837677706 learned_addr learned my addr 172.21.15.27:0/837677706 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:16.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66c6000640 1 --2- 172.21.15.27:0/837677706 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66c80ca3f0 0x7f66c80cf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:16.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66c7400640 1 --2- 172.21.15.27:0/837677706 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66c815b470 0x7f66c80b02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:16.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66c6a00640 1 -- 172.21.15.27:0/837677706 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66c80ca3f0 msgr2=0x7f66c80cf1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:16.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66c6a00640 1 --2- 172.21.15.27:0/837677706 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66c80ca3f0 0x7f66c80cf1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66c6a00640 1 -- 172.21.15.27:0/837677706 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66c815b470 msgr2=0x7f66c80b02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:16.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66c6a00640 1 --2- 172.21.15.27:0/837677706 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66c815b470 0x7f66c80b02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66c6a00640 1 -- 172.21.15.27:0/837677706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b00094d0 con 0x7f66c806a4f0 2024-01-15T18:07:16.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66c7400640 1 --2- 172.21.15.27:0/837677706 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66c815b470 0x7f66c80b02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:07:16.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66c6000640 1 --2- 172.21.15.27:0/837677706 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66c80ca3f0 0x7f66c80cf1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:07:16.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66c6a00640 1 --2- 172.21.15.27:0/837677706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66c806a4f0 0x7f66c8156770 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f66b800d800 tx=0x7f66b800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:16.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66bea00640 1 -- 172.21.15.27:0/837677706 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b80096a0 con 0x7f66c806a4f0 2024-01-15T18:07:16.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66bea00640 1 -- 172.21.15.27:0/837677706 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b8009800 con 0x7f66c806a4f0 2024-01-15T18:07:16.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66bea00640 1 -- 172.21.15.27:0/837677706 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b8009990 con 0x7f66c806a4f0 2024-01-15T18:07:16.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66cd800640 1 -- 172.21.15.27:0/837677706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66c8164510 con 0x7f66c806a4f0 2024-01-15T18:07:16.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.205+0000 7f66cd800640 1 -- 172.21.15.27:0/837677706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66c8156cb0 con 0x7f66c806a4f0 2024-01-15T18:07:16.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.209+0000 7f66cd800640 1 -- 172.21.15.27:0/837677706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66c804e850 con 0x7f66c806a4f0 2024-01-15T18:07:16.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.209+0000 7f66bea00640 1 -- 172.21.15.27:0/837677706 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f66b8009af0 con 0x7f66c806a4f0 2024-01-15T18:07:16.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.209+0000 7f66bea00640 1 --2- 172.21.15.27:0/837677706 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f66a0066b60 0x7f66a0069020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:16.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.209+0000 7f66bea00640 1 -- 172.21.15.27:0/837677706 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f66b8084750 con 0x7f66c806a4f0 2024-01-15T18:07:16.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.209+0000 7f66c6000640 1 --2- 172.21.15.27:0/837677706 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f66a0066b60 0x7f66a0069020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:16.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.209+0000 7f66c6000640 1 --2- 172.21.15.27:0/837677706 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f66a0066b60 0x7f66a0069020 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f66b00ad000 tx=0x7f66b0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:16.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.213+0000 7f66bea00640 1 -- 172.21.15.27:0/837677706 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f66b8051ad0 con 0x7f66c806a4f0 2024-01-15T18:07:16.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.437+0000 7f66cd800640 1 -- 172.21.15.27:0/837677706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f66c8072540 con 0x7f66c806a4f0 2024-01-15T18:07:16.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.437+0000 7f66bea00640 1 -- 172.21.15.27:0/837677706 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v9) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f66b800b3a0 con 0x7f66c806a4f0 2024-01-15T18:07:16.445 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:07:16.445 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:07:16.445 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-01-15T18:07:16.445 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:07:16.445 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-01-15T18:07:16.445 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlbMZIGBAAhSU4y2DGJd3MH0v6v3LsDg==", 2024-01-15T18:07:16.445 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:07:16.445 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:07:16.445 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:07:16.445 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:07:16.445 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:07:16.445 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:16.446 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:07:16.446 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:07:16.446 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-01-15T18:07:16.446 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlC7B9JBAAsXmBjV4KwPZejOmowXO2Zg==", 2024-01-15T18:07:16.446 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:07:16.446 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:07:16.446 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:07:16.446 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:07:16.446 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:07:16.446 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:16.446 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:07:16.446 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:07:16.446 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-01-15T18:07:16.446 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6Vla6a3HxAA/AHc6FSTccj0JGwiR4Vl7g==", 2024-01-15T18:07:16.446 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:07:16.447 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:07:16.447 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:07:16.447 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:07:16.447 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:07:16.447 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:16.447 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:07:16.447 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:07:16.447 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-01-15T18:07:16.447 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6Vl9nqzKxAAx2aDA8UwidAUrU9DBxGldA==", 2024-01-15T18:07:16.447 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:07:16.447 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:07:16.447 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:07:16.447 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:07:16.447 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:07:16.448 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:16.448 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:07:16.448 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:07:16.448 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-01-15T18:07:16.448 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCSc6Vly4TACRAAYBmMIcsZsRp4CZ5a3xCWgA==", 2024-01-15T18:07:16.448 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:07:16.448 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:07:16.448 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:07:16.448 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:07:16.448 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:16.448 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:07:16.448 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:07:16.448 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-01-15T18:07:16.448 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCVc6VlW/HEERAAOBgPz+eGpHwNXDe8fkYo7w==", 2024-01-15T18:07:16.449 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:07:16.449 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:07:16.449 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:07:16.449 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:07:16.449 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:16.449 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:07:16.449 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:07:16.449 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-01-15T18:07:16.449 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCYc6Vl+Ec4GBAAWRe/6u7u4RXApd57H8ihng==", 2024-01-15T18:07:16.449 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:07:16.449 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:07:16.449 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:07:16.449 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:07:16.449 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:16.449 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:07:16.450 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:07:16.450 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-01-15T18:07:16.450 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCbc6VlNdMPHxAAHzJvzLltsTl1wrEnI/iyrQ==", 2024-01-15T18:07:16.450 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:07:16.450 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:07:16.450 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:07:16.450 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:07:16.450 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:16.450 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:07:16.450 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:07:16.450 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-01-15T18:07:16.450 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQChc6Vlg8eVKBAAwZWAncCE2Lra+CxmSIxZkA==", 2024-01-15T18:07:16.450 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:07:16.450 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:07:16.450 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:07:16.451 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:07:16.451 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:16.451 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:07:16.451 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:07:16.451 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-01-15T18:07:16.451 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCkc6VlIXDOLxAAFISn6ILqq/aNFK07RxNnaw==", 2024-01-15T18:07:16.451 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:07:16.451 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:07:16.451 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:07:16.451 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:07:16.451 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:16.451 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:07:16.451 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:07:16.451 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-01-15T18:07:16.452 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCnc6Vl9Ru2NRAAl996VBDudw23SGiFPusIcw==", 2024-01-15T18:07:16.452 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:07:16.452 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:07:16.452 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:07:16.452 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:07:16.452 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:16.452 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:07:16.452 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:07:16.452 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-01-15T18:07:16.452 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCrc6VlHoAsARAAIoPMridLsIcpU3I9Hr5XLQ==", 2024-01-15T18:07:16.452 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:07:16.452 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:07:16.452 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:07:16.452 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:07:16.453 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:16.453 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:07:16.453 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:07:16.453 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-01-15T18:07:16.453 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlH+DCLxAA1sKVut+3WjA/B+aYJ6/Jeg==", 2024-01-15T18:07:16.453 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:07:16.453 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:07:16.453 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-01-15T18:07:16.453 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-15T18:07:16.453 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:16.453 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:07:16.453 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:07:16.453 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-01-15T18:07:16.453 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlaRBaORAAijn3UJLHbWm8ol2NiKNSaw==", 2024-01-15T18:07:16.453 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:07:16.454 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:07:16.454 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-15T18:07:16.454 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-15T18:07:16.454 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-15T18:07:16.454 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:16.454 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:07:16.454 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:07:16.454 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-01-15T18:07:16.454 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlDbLQNRAA82Lwz+pKY+3nHPAr1DichA==", 2024-01-15T18:07:16.454 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:07:16.454 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:07:16.454 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-15T18:07:16.454 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-15T18:07:16.454 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-15T18:07:16.455 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:16.455 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:07:16.455 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:07:16.455 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-01-15T18:07:16.455 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCPc6VlOyDaAxAADea0K1OEDTfrzsM7VEkixg==", 2024-01-15T18:07:16.455 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:07:16.455 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:07:16.455 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-15T18:07:16.455 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-15T18:07:16.455 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-15T18:07:16.455 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:16.455 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:07:16.455 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:07:16.455 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-01-15T18:07:16.455 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNc6VlK70BNxAAgYXkW/pMlzy8GZoZhX9NMA==", 2024-01-15T18:07:16.456 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:07:16.456 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-15T18:07:16.456 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:07:16.456 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:07:16.456 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:07:16.456 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:16.456 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:07:16.456 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:07:16.456 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-01-15T18:07:16.456 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlQbinExAAWtUdgkUNR3cCKV+ytcbUuQ==", 2024-01-15T18:07:16.456 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:07:16.456 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-15T18:07:16.456 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-15T18:07:16.457 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:07:16.457 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:16.457 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:07:16.457 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:07:16.457 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-01-15T18:07:16.457 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlzafrDxAAVVEBuX8BDT3cB9NfD2xTdQ==", 2024-01-15T18:07:16.457 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:07:16.457 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-15T18:07:16.457 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-15T18:07:16.457 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:07:16.457 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:16.457 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:16.457 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-01-15T18:07:16.457 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:07:16.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.441+0000 7f66cd800640 1 -- 172.21.15.27:0/837677706 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f66a0066b60 msgr2=0x7f66a0069020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:16.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.441+0000 7f66cd800640 1 --2- 172.21.15.27:0/837677706 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f66a0066b60 0x7f66a0069020 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f66b00ad000 tx=0x7f66b0008040 comp rx=0 tx=0).stop 2024-01-15T18:07:16.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.441+0000 7f66cd800640 1 -- 172.21.15.27:0/837677706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66c806a4f0 msgr2=0x7f66c8156770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:16.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.441+0000 7f66cd800640 1 --2- 172.21.15.27:0/837677706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66c806a4f0 0x7f66c8156770 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f66b800d800 tx=0x7f66b800dcc0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.441+0000 7f66cd800640 1 -- 172.21.15.27:0/837677706 shutdown_connections 2024-01-15T18:07:16.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.441+0000 7f66cd800640 1 --2- 172.21.15.27:0/837677706 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f66a0066b60 0x7f66a0069020 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.441+0000 7f66cd800640 1 --2- 172.21.15.27:0/837677706 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f66c815b470 0x7f66c80b02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.441+0000 7f66cd800640 1 --2- 172.21.15.27:0/837677706 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66c80ca3f0 0x7f66c80cf1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.441+0000 7f66cd800640 1 --2- 172.21.15.27:0/837677706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66c806a4f0 0x7f66c8156770 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.441+0000 7f66cd800640 1 -- 172.21.15.27:0/837677706 >> 172.21.15.27:0/837677706 conn(0x7f66c80cfda0 msgr2=0x7f66c80afac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:16.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.441+0000 7f66cd800640 1 -- 172.21.15.27:0/837677706 shutdown_connections 2024-01-15T18:07:16.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.441+0000 7f66cd800640 1 -- 172.21.15.27:0/837677706 wait complete. 2024-01-15T18:07:16.467 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:07:16.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.601+0000 7f1cee000640 1 Processor -- start 2024-01-15T18:07:16.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.601+0000 7f1cee000640 1 -- start start 2024-01-15T18:07:16.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.601+0000 7f1cee000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f1ce806bb80 0x7f1ce80cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:16.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.601+0000 7f1cee000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce80b2770 con 0x7f1ce80b2390 2024-01-15T18:07:16.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.601+0000 7f1cee000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce806c060 con 0x7f1ce80cddc0 2024-01-15T18:07:16.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.601+0000 7f1cee000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ce80ce270 con 0x7f1ce806bb80 2024-01-15T18:07:16.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.601+0000 7f1ce7400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1ce80b2390 0x7f1ce80cf780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59334/0 (socket says 172.21.15.27:59334) 2024-01-15T18:07:16.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.601+0000 7f1ce7400640 1 -- 172.21.15.27:0/3247212177 learned_addr learned my addr 172.21.15.27:0/3247212177 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:16.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.601+0000 7f1ce6a00640 1 --2- 172.21.15.27:0/3247212177 >> v2:172.21.15.93:3300/0 conn(0x7f1ce806bb80 0x7f1ce80cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:16.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.601+0000 7f1ce6000640 1 -- 172.21.15.27:0/3247212177 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3918939723 0 0) 0x7f1ce80b2770 con 0x7f1ce80b2390 2024-01-15T18:07:16.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.601+0000 7f1ce6000640 1 -- 172.21.15.27:0/3247212177 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cc8003530 con 0x7f1ce80b2390 2024-01-15T18:07:16.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.601+0000 7f1ce6000640 1 -- 172.21.15.27:0/3247212177 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776795261 0 0) 0x7f1ce806c060 con 0x7f1ce80cddc0 2024-01-15T18:07:16.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.601+0000 7f1ce6000640 1 -- 172.21.15.27:0/3247212177 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce80b2770 con 0x7f1ce80cddc0 2024-01-15T18:07:16.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.601+0000 7f1ce6a00640 1 -- 172.21.15.27:0/3247212177 >> v1:172.21.15.93:6789/0 conn(0x7f1ce80cddc0 legacy=0x7f1ce80b0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:16.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.601+0000 7f1ce6a00640 1 -- 172.21.15.27:0/3247212177 >> v1:172.21.15.27:6789/0 conn(0x7f1ce80b2390 legacy=0x7f1ce80cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:16.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.601+0000 7f1ce6a00640 1 -- 172.21.15.27:0/3247212177 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce806c340 con 0x7f1ce806bb80 2024-01-15T18:07:16.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.605+0000 7f1ce6a00640 1 --2- 172.21.15.27:0/3247212177 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ce806bb80 0x7f1ce80cd880 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1cd00097c0 tx=0x7f1cd001c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4680b4023d525d7e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:16.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.605+0000 7f1ce6000640 1 -- 172.21.15.27:0/3247212177 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd00ab650 con 0x7f1ce806bb80 2024-01-15T18:07:16.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.605+0000 7f1ce6000640 1 -- 172.21.15.27:0/3247212177 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cd00ab7b0 con 0x7f1ce806bb80 2024-01-15T18:07:16.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.605+0000 7f1ce6000640 1 -- 172.21.15.27:0/3247212177 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd00abb40 con 0x7f1ce806bb80 2024-01-15T18:07:16.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.605+0000 7f1cee000640 1 -- 172.21.15.27:0/3247212177 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ce806bb80 msgr2=0x7f1ce80cd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:16.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.605+0000 7f1cee000640 1 --2- 172.21.15.27:0/3247212177 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ce806bb80 0x7f1ce80cd880 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1cd00097c0 tx=0x7f1cd001c450 comp rx=0 tx=0).stop 2024-01-15T18:07:16.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.605+0000 7f1cee000640 1 -- 172.21.15.27:0/3247212177 shutdown_connections 2024-01-15T18:07:16.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.605+0000 7f1cee000640 1 --2- 172.21.15.27:0/3247212177 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ce806bb80 0x7f1ce80cd880 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.605+0000 7f1cee000640 1 -- 172.21.15.27:0/3247212177 >> 172.21.15.27:0/3247212177 conn(0x7f1ce80b2a10 msgr2=0x7f1ce80b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:16.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.605+0000 7f1cee000640 1 -- 172.21.15.27:0/3247212177 shutdown_connections 2024-01-15T18:07:16.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.605+0000 7f1cee000640 1 -- 172.21.15.27:0/3247212177 wait complete. 2024-01-15T18:07:16.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.609+0000 7f1cee000640 1 Processor -- start 2024-01-15T18:07:16.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.609+0000 7f1cee000640 1 -- start start 2024-01-15T18:07:16.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.609+0000 7f1cee000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ce806bb80 0x7f1ce8151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:16.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.609+0000 7f1cee000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ce80b2390 0x7f1ce8158db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:16.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.609+0000 7f1cee000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ce80cddc0 0x7f1ce806f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:16.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.609+0000 7f1cee000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce80c8680 con 0x7f1ce80cddc0 2024-01-15T18:07:16.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.609+0000 7f1cee000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce81592f0 con 0x7f1ce806bb80 2024-01-15T18:07:16.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.609+0000 7f1cee000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce8159590 con 0x7f1ce80b2390 2024-01-15T18:07:16.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.609+0000 7f1ce7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ce80cddc0 0x7f1ce806f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:16.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.609+0000 7f1ce7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ce80cddc0 0x7f1ce806f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33032/0 (socket says 172.21.15.27:33032) 2024-01-15T18:07:16.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.609+0000 7f1ce7e00640 1 -- 172.21.15.27:0/1940553059 learned_addr learned my addr 172.21.15.27:0/1940553059 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:16.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.609+0000 7f1ce6a00640 1 --2- 172.21.15.27:0/1940553059 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ce80b2390 0x7f1ce8158db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:16.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.609+0000 7f1ce7400640 1 --2- 172.21.15.27:0/1940553059 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ce806bb80 0x7f1ce8151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:16.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.609+0000 7f1ce7e00640 1 -- 172.21.15.27:0/1940553059 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ce80b2390 msgr2=0x7f1ce8158db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:16.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.609+0000 7f1ce7e00640 1 --2- 172.21.15.27:0/1940553059 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ce80b2390 0x7f1ce8158db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.609+0000 7f1ce7e00640 1 -- 172.21.15.27:0/1940553059 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ce806bb80 msgr2=0x7f1ce8151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:16.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.609+0000 7f1ce7e00640 1 --2- 172.21.15.27:0/1940553059 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ce806bb80 0x7f1ce8151ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.609+0000 7f1ce7e00640 1 -- 172.21.15.27:0/1940553059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd0009420 con 0x7f1ce80cddc0 2024-01-15T18:07:16.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.609+0000 7f1ce7e00640 1 --2- 172.21.15.27:0/1940553059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ce80cddc0 0x7f1ce806f7a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1ce000c730 tx=0x7f1ce000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:16.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.609+0000 7f1cdf400640 1 -- 172.21.15.27:0/1940553059 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0003b80 con 0x7f1ce80cddc0 2024-01-15T18:07:16.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.613+0000 7f1cdf400640 1 -- 172.21.15.27:0/1940553059 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce0003ce0 con 0x7f1ce80cddc0 2024-01-15T18:07:16.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.613+0000 7f1cdf400640 1 -- 172.21.15.27:0/1940553059 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce000aa90 con 0x7f1ce80cddc0 2024-01-15T18:07:16.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.613+0000 7f1cee000640 1 -- 172.21.15.27:0/1940553059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce80c8030 con 0x7f1ce80cddc0 2024-01-15T18:07:16.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.613+0000 7f1cee000640 1 -- 172.21.15.27:0/1940553059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce8159870 con 0x7f1ce80cddc0 2024-01-15T18:07:16.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.613+0000 7f1cee000640 1 -- 172.21.15.27:0/1940553059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce804e850 con 0x7f1ce80cddc0 2024-01-15T18:07:16.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.613+0000 7f1cdf400640 1 -- 172.21.15.27:0/1940553059 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f1ce000abf0 con 0x7f1ce80cddc0 2024-01-15T18:07:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.617+0000 7f1cdf400640 1 --2- 172.21.15.27:0/1940553059 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1cc0066b30 0x7f1cc0068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.617+0000 7f1cdf400640 1 -- 172.21.15.27:0/1940553059 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f1ce0084bc0 con 0x7f1ce80cddc0 2024-01-15T18:07:16.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.617+0000 7f1ce7400640 1 --2- 172.21.15.27:0/1940553059 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1cc0066b30 0x7f1cc0068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:16.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.617+0000 7f1ce7400640 1 --2- 172.21.15.27:0/1940553059 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1cc0066b30 0x7f1cc0068ff0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1cd80011f0 tx=0x7f1cd8007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:16.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.617+0000 7f1cdf400640 1 -- 172.21.15.27:0/1940553059 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1ce0051ff0 con 0x7f1ce80cddc0 2024-01-15T18:07:16.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.777+0000 7f1cee000640 1 -- 172.21.15.27:0/1940553059 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ce80af560 con 0x7f1ce80cddc0 2024-01-15T18:07:16.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.777+0000 7f1cdf400640 1 -- 172.21.15.27:0/1940553059 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+12804 (secure 0 0 0) 0x7f1ce0055610 con 0x7f1ce80cddc0 2024-01-15T18:07:16.784 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:07:16.784 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":41,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:07:16.041642+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:07:16.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.777+0000 7f1cee000640 1 -- 172.21.15.27:0/1940553059 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1cc0066b30 msgr2=0x7f1cc0068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:16.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.777+0000 7f1cee000640 1 --2- 172.21.15.27:0/1940553059 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1cc0066b30 0x7f1cc0068ff0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1cd80011f0 tx=0x7f1cd8007d40 comp rx=0 tx=0).stop 2024-01-15T18:07:16.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.777+0000 7f1cee000640 1 -- 172.21.15.27:0/1940553059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ce80cddc0 msgr2=0x7f1ce806f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:16.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.777+0000 7f1cee000640 1 --2- 172.21.15.27:0/1940553059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ce80cddc0 0x7f1ce806f7a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1ce000c730 tx=0x7f1ce000cbf0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.781+0000 7f1cee000640 1 -- 172.21.15.27:0/1940553059 shutdown_connections 2024-01-15T18:07:16.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.781+0000 7f1cee000640 1 --2- 172.21.15.27:0/1940553059 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1cc0066b30 0x7f1cc0068ff0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.781+0000 7f1cee000640 1 --2- 172.21.15.27:0/1940553059 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ce80cddc0 0x7f1ce806f7a0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.781+0000 7f1cee000640 1 --2- 172.21.15.27:0/1940553059 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ce80b2390 0x7f1ce8158db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.781+0000 7f1cee000640 1 --2- 172.21.15.27:0/1940553059 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ce806bb80 0x7f1ce8151ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.781+0000 7f1cee000640 1 -- 172.21.15.27:0/1940553059 >> 172.21.15.27:0/1940553059 conn(0x7f1ce80b2a10 msgr2=0x7f1ce80cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:16.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.781+0000 7f1cee000640 1 -- 172.21.15.27:0/1940553059 shutdown_connections 2024-01-15T18:07:16.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.781+0000 7f1cee000640 1 -- 172.21.15.27:0/1940553059 wait complete. 2024-01-15T18:07:16.807 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:07:16.807 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:07:16.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.945+0000 7fabc1800640 1 Processor -- start 2024-01-15T18:07:16.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.945+0000 7fabc1800640 1 -- start start 2024-01-15T18:07:16.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.945+0000 7fabc1800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fabbc0715f0 0x7fabbc164a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:16.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.945+0000 7fabc1800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabbc15a610 con 0x7fabbc15b100 2024-01-15T18:07:16.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.945+0000 7fabc1800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabbc15e3c0 con 0x7fabbc15ad20 2024-01-15T18:07:16.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.945+0000 7fabc1800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabbc15e6a0 con 0x7fabbc0715f0 2024-01-15T18:07:16.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.945+0000 7fabba000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fabbc15b100 0x7fabbc070ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59342/0 (socket says 172.21.15.27:59342) 2024-01-15T18:07:16.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.945+0000 7fabba000640 1 -- 172.21.15.27:0/2763461304 learned_addr learned my addr 172.21.15.27:0/2763461304 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:16.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.945+0000 7fabbb400640 1 --2- 172.21.15.27:0/2763461304 >> v2:172.21.15.93:3300/0 conn(0x7fabbc0715f0 0x7fabbc164a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:16.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.945+0000 7fabb9600640 1 -- 172.21.15.27:0/2763461304 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260218516 0 0) 0x7fabbc15a610 con 0x7fabbc15b100 2024-01-15T18:07:16.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.945+0000 7fabb9600640 1 -- 172.21.15.27:0/2763461304 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab9c003530 con 0x7fabbc15b100 2024-01-15T18:07:16.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.945+0000 7fabbb400640 1 -- 172.21.15.27:0/2763461304 >> v1:172.21.15.93:6789/0 conn(0x7fabbc15ad20 legacy=0x7fabbc15d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:16.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.945+0000 7fabbb400640 1 -- 172.21.15.27:0/2763461304 >> v1:172.21.15.27:6789/0 conn(0x7fabbc15b100 legacy=0x7fabbc070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:16.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.945+0000 7fabbb400640 1 -- 172.21.15.27:0/2763461304 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabbc15b6e0 con 0x7fabbc0715f0 2024-01-15T18:07:16.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.949+0000 7fabbb400640 1 --2- 172.21.15.27:0/2763461304 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fabbc0715f0 0x7fabbc164a60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fabb4009dc0 tx=0x7fabb401c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee570bcc7036d53a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:16.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.949+0000 7fabb9600640 1 -- 172.21.15.27:0/2763461304 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb40a5850 con 0x7fabbc0715f0 2024-01-15T18:07:16.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.949+0000 7fabb9600640 1 -- 172.21.15.27:0/2763461304 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb40a59b0 con 0x7fabbc0715f0 2024-01-15T18:07:16.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.949+0000 7fabb9600640 1 -- 172.21.15.27:0/2763461304 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb40a5d40 con 0x7fabbc0715f0 2024-01-15T18:07:16.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.949+0000 7fabc1800640 1 -- 172.21.15.27:0/2763461304 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fabbc0715f0 msgr2=0x7fabbc164a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:16.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.949+0000 7fabc1800640 1 --2- 172.21.15.27:0/2763461304 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fabbc0715f0 0x7fabbc164a60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fabb4009dc0 tx=0x7fabb401c870 comp rx=0 tx=0).stop 2024-01-15T18:07:16.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.949+0000 7fabc1800640 1 -- 172.21.15.27:0/2763461304 shutdown_connections 2024-01-15T18:07:16.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.949+0000 7fabc1800640 1 --2- 172.21.15.27:0/2763461304 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fabbc0715f0 0x7fabbc164a60 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.949+0000 7fabc1800640 1 -- 172.21.15.27:0/2763461304 >> 172.21.15.27:0/2763461304 conn(0x7fabbc0cb700 msgr2=0x7fabbc0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:16.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.953+0000 7fabc1800640 1 -- 172.21.15.27:0/2763461304 shutdown_connections 2024-01-15T18:07:16.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.953+0000 7fabc1800640 1 -- 172.21.15.27:0/2763461304 wait complete. 2024-01-15T18:07:16.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.953+0000 7fabc1800640 1 Processor -- start 2024-01-15T18:07:16.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.953+0000 7fabc1800640 1 -- start start 2024-01-15T18:07:16.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.953+0000 7fabc1800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fabbc0715f0 0x7fabbc0b38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:16.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.953+0000 7fabc1800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabbc15ad20 0x7fabbc151d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:16.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.953+0000 7fabc1800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fabbc15b100 0x7fabbc152240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:16.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.953+0000 7fabc1800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabbc1538f0 con 0x7fabbc15ad20 2024-01-15T18:07:16.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.953+0000 7fabc1800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabbc152a20 con 0x7fabbc15b100 2024-01-15T18:07:16.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.953+0000 7fabc1800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabbc152c70 con 0x7fabbc0715f0 2024-01-15T18:07:16.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.953+0000 7fabba000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabbc15ad20 0x7fabbc151d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:16.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.953+0000 7fabba000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabbc15ad20 0x7fabbc151d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33046/0 (socket says 172.21.15.27:33046) 2024-01-15T18:07:16.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.953+0000 7fabba000640 1 -- 172.21.15.27:0/963779357 learned_addr learned my addr 172.21.15.27:0/963779357 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:16.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.953+0000 7fabbb400640 1 --2- 172.21.15.27:0/963779357 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fabbc15b100 0x7fabbc152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:16.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.953+0000 7fabbaa00640 1 --2- 172.21.15.27:0/963779357 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fabbc0715f0 0x7fabbc0b38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:16.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.953+0000 7fabba000640 1 -- 172.21.15.27:0/963779357 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fabbc0715f0 msgr2=0x7fabbc0b38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:16.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.953+0000 7fabba000640 1 --2- 172.21.15.27:0/963779357 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fabbc0715f0 0x7fabbc0b38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.953+0000 7fabba000640 1 -- 172.21.15.27:0/963779357 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fabbc15b100 msgr2=0x7fabbc152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:16.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.953+0000 7fabba000640 1 --2- 172.21.15.27:0/963779357 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fabbc15b100 0x7fabbc152240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.953+0000 7fabba000640 1 -- 172.21.15.27:0/963779357 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb4009a70 con 0x7fabbc15ad20 2024-01-15T18:07:16.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.957+0000 7fabba000640 1 --2- 172.21.15.27:0/963779357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabbc15ad20 0x7fabbc151d00 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7faba400c420 tx=0x7faba400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:16.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.957+0000 7fabb2a00640 1 -- 172.21.15.27:0/963779357 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba4005cf0 con 0x7fabbc15ad20 2024-01-15T18:07:16.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.957+0000 7fabb2a00640 1 -- 172.21.15.27:0/963779357 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba4005e50 con 0x7fabbc15ad20 2024-01-15T18:07:16.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.957+0000 7fabb2a00640 1 -- 172.21.15.27:0/963779357 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba40072d0 con 0x7fabbc15ad20 2024-01-15T18:07:16.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.957+0000 7fabc1800640 1 -- 172.21.15.27:0/963779357 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabbc152f50 con 0x7fabbc15ad20 2024-01-15T18:07:16.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.957+0000 7fabc1800640 1 -- 172.21.15.27:0/963779357 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabbc153510 con 0x7fabbc15ad20 2024-01-15T18:07:16.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.957+0000 7fabc1800640 1 -- 172.21.15.27:0/963779357 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabbc153a30 con 0x7fabbc15ad20 2024-01-15T18:07:16.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.957+0000 7fabb2a00640 1 -- 172.21.15.27:0/963779357 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7faba4007430 con 0x7fabbc15ad20 2024-01-15T18:07:16.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.957+0000 7fabb2a00640 1 --2- 172.21.15.27:0/963779357 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fab94066b30 0x7fab94068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:16.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.957+0000 7fabb2a00640 1 -- 172.21.15.27:0/963779357 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4885+0+0 (secure 0 0 0) 0x7faba40839e0 con 0x7fabbc15ad20 2024-01-15T18:07:16.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.957+0000 7fabbaa00640 1 --2- 172.21.15.27:0/963779357 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fab94066b30 0x7fab94068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:16.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.961+0000 7fabbaa00640 1 --2- 172.21.15.27:0/963779357 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fab94066b30 0x7fab94068ff0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fabbc14c390 tx=0x7fabac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:16.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:16.961+0000 7fabb2a00640 1 -- 172.21.15.27:0/963779357 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fabbc153a30 con 0x7fabbc15ad20 2024-01-15T18:07:17.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.125+0000 7fabc1800640 1 -- 172.21.15.27:0/963779357 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fabbc153a30 con 0x7fabbc15ad20 2024-01-15T18:07:17.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.125+0000 7fabb2a00640 1 -- 172.21.15.27:0/963779357 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7fabbc153a30 con 0x7fabbc15ad20 2024-01-15T18:07:17.132 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:07:17.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.125+0000 7fabc1800640 1 -- 172.21.15.27:0/963779357 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fab94066b30 msgr2=0x7fab94068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:17.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.125+0000 7fabc1800640 1 --2- 172.21.15.27:0/963779357 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fab94066b30 0x7fab94068ff0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fabbc14c390 tx=0x7fabac009040 comp rx=0 tx=0).stop 2024-01-15T18:07:17.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.125+0000 7fabc1800640 1 -- 172.21.15.27:0/963779357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabbc15ad20 msgr2=0x7fabbc151d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:17.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.125+0000 7fabc1800640 1 --2- 172.21.15.27:0/963779357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabbc15ad20 0x7fabbc151d00 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7faba400c420 tx=0x7faba400c8e0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.125+0000 7fabc1800640 1 -- 172.21.15.27:0/963779357 shutdown_connections 2024-01-15T18:07:17.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.125+0000 7fabc1800640 1 --2- 172.21.15.27:0/963779357 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fab94066b30 0x7fab94068ff0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.125+0000 7fabc1800640 1 --2- 172.21.15.27:0/963779357 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fabbc15b100 0x7fabbc152240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.125+0000 7fabc1800640 1 --2- 172.21.15.27:0/963779357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabbc15ad20 0x7fabbc151d00 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.125+0000 7fabc1800640 1 --2- 172.21.15.27:0/963779357 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fabbc0715f0 0x7fabbc0b38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.125+0000 7fabc1800640 1 -- 172.21.15.27:0/963779357 >> 172.21.15.27:0/963779357 conn(0x7fabbc0cb700 msgr2=0x7fabbc0b4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:17.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.125+0000 7fabc1800640 1 -- 172.21.15.27:0/963779357 shutdown_connections 2024-01-15T18:07:17.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.125+0000 7fabc1800640 1 -- 172.21.15.27:0/963779357 wait complete. 2024-01-15T18:07:17.156 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-15T18:07:17.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.289+0000 7fd58d200640 1 Processor -- start 2024-01-15T18:07:17.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.289+0000 7fd58d200640 1 -- start start 2024-01-15T18:07:17.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.293+0000 7fd58d200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd58806db80 0x7fd58806df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:17.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.293+0000 7fd58d200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd58815e730 0x7fd5880cd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:17.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.293+0000 7fd58d200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5880ce920 con 0x7fd5880cdd10 2024-01-15T18:07:17.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.293+0000 7fd58d200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd58806e4a0 con 0x7fd58815e730 2024-01-15T18:07:17.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.293+0000 7fd58d200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5880ce640 con 0x7fd58806db80 2024-01-15T18:07:17.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.293+0000 7fd585400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd58815e730 0x7fd5880cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:17.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.293+0000 7fd586800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd5880cdd10 0x7fd588070c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59354/0 (socket says 172.21.15.27:59354) 2024-01-15T18:07:17.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.293+0000 7fd586800640 1 -- 172.21.15.27:0/3916949198 learned_addr learned my addr 172.21.15.27:0/3916949198 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:17.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.293+0000 7fd585e00640 1 --2- 172.21.15.27:0/3916949198 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd58806db80 0x7fd58806df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:17.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.293+0000 7fd584a00640 1 -- 172.21.15.27:0/3916949198 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719969777 0 0) 0x7fd5880ce920 con 0x7fd5880cdd10 2024-01-15T18:07:17.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.293+0000 7fd584a00640 1 -- 172.21.15.27:0/3916949198 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd568003530 con 0x7fd5880cdd10 2024-01-15T18:07:17.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.293+0000 7fd585400640 1 -- 172.21.15.27:0/3916949198 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd58806db80 msgr2=0x7fd58806df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:17.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.293+0000 7fd585400640 1 --2- 172.21.15.27:0/3916949198 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd58806db80 0x7fd58806df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.293+0000 7fd585400640 1 -- 172.21.15.27:0/3916949198 >> v1:172.21.15.27:6789/0 conn(0x7fd5880cdd10 legacy=0x7fd588070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:17.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.293+0000 7fd585400640 1 -- 172.21.15.27:0/3916949198 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5880cc940 con 0x7fd58815e730 2024-01-15T18:07:17.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.297+0000 7fd585400640 1 --2- 172.21.15.27:0/3916949198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd58815e730 0x7fd5880cd420 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd5700097c0 tx=0x7fd57001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cb054c9065f535 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:17.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.297+0000 7fd584a00640 1 -- 172.21.15.27:0/3916949198 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5700ab650 con 0x7fd58815e730 2024-01-15T18:07:17.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.297+0000 7fd584a00640 1 -- 172.21.15.27:0/3916949198 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5700ab7b0 con 0x7fd58815e730 2024-01-15T18:07:17.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.297+0000 7fd584a00640 1 -- 172.21.15.27:0/3916949198 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5700abb20 con 0x7fd58815e730 2024-01-15T18:07:17.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.297+0000 7fd58d200640 1 -- 172.21.15.27:0/3916949198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd58815e730 msgr2=0x7fd5880cd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:17.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.297+0000 7fd58d200640 1 --2- 172.21.15.27:0/3916949198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd58815e730 0x7fd5880cd420 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd5700097c0 tx=0x7fd57001c450 comp rx=0 tx=0).stop 2024-01-15T18:07:17.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.297+0000 7fd58d200640 1 -- 172.21.15.27:0/3916949198 shutdown_connections 2024-01-15T18:07:17.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.297+0000 7fd58d200640 1 --2- 172.21.15.27:0/3916949198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd58815e730 0x7fd5880cd420 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.297+0000 7fd58d200640 1 --2- 172.21.15.27:0/3916949198 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd58806db80 0x7fd58806df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.297+0000 7fd58d200640 1 -- 172.21.15.27:0/3916949198 >> 172.21.15.27:0/3916949198 conn(0x7fd5880cec60 msgr2=0x7fd5880cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:17.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.297+0000 7fd58d200640 1 -- 172.21.15.27:0/3916949198 shutdown_connections 2024-01-15T18:07:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.297+0000 7fd58d200640 1 -- 172.21.15.27:0/3916949198 wait complete. 2024-01-15T18:07:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.297+0000 7fd58d200640 1 Processor -- start 2024-01-15T18:07:17.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.297+0000 7fd58d200640 1 -- start start 2024-01-15T18:07:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.301+0000 7fd58d200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd58806db80 0x7fd5880cbb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.301+0000 7fd58d200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd5880cdd10 0x7fd588152e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.301+0000 7fd58d200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd58815e730 0x7fd588166e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:17.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.301+0000 7fd58d200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5881553a0 con 0x7fd58815e730 2024-01-15T18:07:17.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.301+0000 7fd58d200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58815d2d0 con 0x7fd58806db80 2024-01-15T18:07:17.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.301+0000 7fd58d200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd58815d5b0 con 0x7fd5880cdd10 2024-01-15T18:07:17.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.301+0000 7fd586800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd58815e730 0x7fd588166e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:17.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.301+0000 7fd586800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd58815e730 0x7fd588166e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33054/0 (socket says 172.21.15.27:33054) 2024-01-15T18:07:17.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.301+0000 7fd586800640 1 -- 172.21.15.27:0/1046103168 learned_addr learned my addr 172.21.15.27:0/1046103168 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:17.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.301+0000 7fd585400640 1 --2- 172.21.15.27:0/1046103168 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd5880cdd10 0x7fd588152e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:17.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.301+0000 7fd585e00640 1 --2- 172.21.15.27:0/1046103168 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd58806db80 0x7fd5880cbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:17.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.301+0000 7fd586800640 1 -- 172.21.15.27:0/1046103168 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd5880cdd10 msgr2=0x7fd588152e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:17.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.301+0000 7fd586800640 1 --2- 172.21.15.27:0/1046103168 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd5880cdd10 0x7fd588152e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.301+0000 7fd586800640 1 -- 172.21.15.27:0/1046103168 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd58806db80 msgr2=0x7fd5880cbb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:17.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.301+0000 7fd586800640 1 --2- 172.21.15.27:0/1046103168 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd58806db80 0x7fd5880cbb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.301+0000 7fd586800640 1 -- 172.21.15.27:0/1046103168 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd570009420 con 0x7fd58815e730 2024-01-15T18:07:17.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.301+0000 7fd586800640 1 --2- 172.21.15.27:0/1046103168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd58815e730 0x7fd588166e40 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fd58000cb40 tx=0x7fd58000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.301+0000 7fd57e000640 1 -- 172.21.15.27:0/1046103168 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58000aca0 con 0x7fd58815e730 2024-01-15T18:07:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.301+0000 7fd57e000640 1 -- 172.21.15.27:0/1046103168 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58000ae00 con 0x7fd58815e730 2024-01-15T18:07:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.301+0000 7fd58d200640 1 -- 172.21.15.27:0/1046103168 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5881555f0 con 0x7fd58815e730 2024-01-15T18:07:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.301+0000 7fd57e000640 1 -- 172.21.15.27:0/1046103168 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd580012620 con 0x7fd58815e730 2024-01-15T18:07:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.301+0000 7fd58d200640 1 -- 172.21.15.27:0/1046103168 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd588153390 con 0x7fd58815e730 2024-01-15T18:07:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.301+0000 7fd58d200640 1 -- 172.21.15.27:0/1046103168 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd58804e850 con 0x7fd58815e730 2024-01-15T18:07:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.305+0000 7fd57e000640 1 -- 172.21.15.27:0/1046103168 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7fd580007070 con 0x7fd58815e730 2024-01-15T18:07:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.305+0000 7fd57e000640 1 --2- 172.21.15.27:0/1046103168 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd560066b30 0x7fd560068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.305+0000 7fd57e000640 1 -- 172.21.15.27:0/1046103168 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fd580084610 con 0x7fd58815e730 2024-01-15T18:07:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.305+0000 7fd585e00640 1 --2- 172.21.15.27:0/1046103168 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd560066b30 0x7fd560068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:17.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.305+0000 7fd585e00640 1 --2- 172.21.15.27:0/1046103168 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd560066b30 0x7fd560068ff0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd578009b60 tx=0x7fd578009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:17.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.309+0000 7fd57e000640 1 -- 172.21.15.27:0/1046103168 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd5800517b0 con 0x7fd58815e730 2024-01-15T18:07:17.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.469+0000 7fd58d200640 1 -- 172.21.15.27:0/1046103168 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd58815cc50 con 0x7fd58815e730 2024-01-15T18:07:17.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.469+0000 7fd57e000640 1 -- 172.21.15.27:0/1046103168 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v41) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd580054dd0 con 0x7fd58815e730 2024-01-15T18:07:17.476 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:07:17.476 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:07:17.477 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:07:17.477 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-01-15T18:07:17.477 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-01-15T18:07:17.477 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:17.477 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:07:17.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.469+0000 7fd58d200640 1 -- 172.21.15.27:0/1046103168 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd560066b30 msgr2=0x7fd560068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:17.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.469+0000 7fd58d200640 1 --2- 172.21.15.27:0/1046103168 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd560066b30 0x7fd560068ff0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd578009b60 tx=0x7fd578009190 comp rx=0 tx=0).stop 2024-01-15T18:07:17.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.469+0000 7fd58d200640 1 -- 172.21.15.27:0/1046103168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd58815e730 msgr2=0x7fd588166e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:17.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.469+0000 7fd58d200640 1 --2- 172.21.15.27:0/1046103168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd58815e730 0x7fd588166e40 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fd58000cb40 tx=0x7fd58000a060 comp rx=0 tx=0).stop 2024-01-15T18:07:17.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.473+0000 7fd58d200640 1 -- 172.21.15.27:0/1046103168 shutdown_connections 2024-01-15T18:07:17.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.473+0000 7fd58d200640 1 --2- 172.21.15.27:0/1046103168 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd560066b30 0x7fd560068ff0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.473+0000 7fd58d200640 1 --2- 172.21.15.27:0/1046103168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd58815e730 0x7fd588166e40 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.473+0000 7fd58d200640 1 --2- 172.21.15.27:0/1046103168 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd5880cdd10 0x7fd588152e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.473+0000 7fd58d200640 1 --2- 172.21.15.27:0/1046103168 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd58806db80 0x7fd5880cbb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.473+0000 7fd58d200640 1 -- 172.21.15.27:0/1046103168 >> 172.21.15.27:0/1046103168 conn(0x7fd5880cec60 msgr2=0x7fd5880cf040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:17.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.473+0000 7fd58d200640 1 -- 172.21.15.27:0/1046103168 shutdown_connections 2024-01-15T18:07:17.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.473+0000 7fd58d200640 1 -- 172.21.15.27:0/1046103168 wait complete. 2024-01-15T18:07:17.498 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-15T18:07:17.498 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-15T18:07:17.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.637+0000 7eff5ac00640 1 Processor -- start 2024-01-15T18:07:17.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.637+0000 7eff5ac00640 1 -- start start 2024-01-15T18:07:17.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.637+0000 7eff5ac00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7eff5415ad20 0x7eff5415b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:17.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.637+0000 7eff5ac00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7eff540715f0 0x7eff5415dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:17.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.637+0000 7eff5ac00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff5415a610 con 0x7eff5415d900 2024-01-15T18:07:17.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.637+0000 7eff5ac00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff5415e3c0 con 0x7eff540715f0 2024-01-15T18:07:17.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.637+0000 7eff5ac00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff5415e6a0 con 0x7eff5415ad20 2024-01-15T18:07:17.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.637+0000 7eff58c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7eff540715f0 0x7eff5415dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:17.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.637+0000 7eff53400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7eff5415d900 0x7eff54070ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59368/0 (socket says 172.21.15.27:59368) 2024-01-15T18:07:17.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.637+0000 7eff53400640 1 -- 172.21.15.27:0/2811962922 learned_addr learned my addr 172.21.15.27:0/2811962922 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:17.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.637+0000 7eff53e00640 1 --2- 172.21.15.27:0/2811962922 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7eff5415ad20 0x7eff5415b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:17.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.637+0000 7eff58c00640 1 -- 172.21.15.27:0/2811962922 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7eff5415ad20 msgr2=0x7eff5415b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:17.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.637+0000 7eff58c00640 1 --2- 172.21.15.27:0/2811962922 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7eff5415ad20 0x7eff5415b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.637+0000 7eff58c00640 1 -- 172.21.15.27:0/2811962922 >> v1:172.21.15.27:6789/0 conn(0x7eff5415d900 legacy=0x7eff54070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:17.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.637+0000 7eff58c00640 1 -- 172.21.15.27:0/2811962922 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5415b6e0 con 0x7eff540715f0 2024-01-15T18:07:17.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.637+0000 7eff53e00640 1 --2- 172.21.15.27:0/2811962922 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7eff5415ad20 0x7eff5415b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:07:17.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.641+0000 7eff58c00640 1 --2- 172.21.15.27:0/2811962922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff540715f0 0x7eff5415dd80 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7eff4c009d90 tx=0x7eff4c01c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=179f27fbc577dcba server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:17.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.641+0000 7eff52a00640 1 -- 172.21.15.27:0/2811962922 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4c0a5a00 con 0x7eff540715f0 2024-01-15T18:07:17.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.641+0000 7eff52a00640 1 -- 172.21.15.27:0/2811962922 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff4c0a5b60 con 0x7eff540715f0 2024-01-15T18:07:17.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.641+0000 7eff52a00640 1 -- 172.21.15.27:0/2811962922 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4c0ad440 con 0x7eff540715f0 2024-01-15T18:07:17.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.641+0000 7eff5ac00640 1 -- 172.21.15.27:0/2811962922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff540715f0 msgr2=0x7eff5415dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:17.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.641+0000 7eff5ac00640 1 --2- 172.21.15.27:0/2811962922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff540715f0 0x7eff5415dd80 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7eff4c009d90 tx=0x7eff4c01c920 comp rx=0 tx=0).stop 2024-01-15T18:07:17.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.641+0000 7eff5ac00640 1 -- 172.21.15.27:0/2811962922 shutdown_connections 2024-01-15T18:07:17.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.641+0000 7eff5ac00640 1 --2- 172.21.15.27:0/2811962922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff540715f0 0x7eff5415dd80 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.641+0000 7eff5ac00640 1 --2- 172.21.15.27:0/2811962922 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7eff5415ad20 0x7eff5415b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.641+0000 7eff5ac00640 1 -- 172.21.15.27:0/2811962922 >> 172.21.15.27:0/2811962922 conn(0x7eff540cb700 msgr2=0x7eff540cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:17.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.641+0000 7eff5ac00640 1 -- 172.21.15.27:0/2811962922 shutdown_connections 2024-01-15T18:07:17.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.641+0000 7eff5ac00640 1 -- 172.21.15.27:0/2811962922 wait complete. 2024-01-15T18:07:17.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.645+0000 7eff5ac00640 1 Processor -- start 2024-01-15T18:07:17.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.645+0000 7eff5ac00640 1 -- start start 2024-01-15T18:07:17.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.645+0000 7eff5ac00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff540715f0 0x7eff54160970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:17.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.645+0000 7eff5ac00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7eff5415ad20 0x7eff54163680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:17.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.645+0000 7eff5ac00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7eff5415d900 0x7eff5414d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:17.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.645+0000 7eff5ac00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5413d2a0 con 0x7eff540715f0 2024-01-15T18:07:17.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.645+0000 7eff5ac00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5413d580 con 0x7eff5415d900 2024-01-15T18:07:17.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.645+0000 7eff5ac00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff5413d860 con 0x7eff5415ad20 2024-01-15T18:07:17.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.645+0000 7eff53e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff540715f0 0x7eff54160970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:17.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.645+0000 7eff53e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff540715f0 0x7eff54160970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33070/0 (socket says 172.21.15.27:33070) 2024-01-15T18:07:17.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.645+0000 7eff53e00640 1 -- 172.21.15.27:0/3031132631 learned_addr learned my addr 172.21.15.27:0/3031132631 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:17.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.645+0000 7eff53400640 1 --2- 172.21.15.27:0/3031132631 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7eff5415ad20 0x7eff54163680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:17.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.645+0000 7eff58c00640 1 --2- 172.21.15.27:0/3031132631 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7eff5415d900 0x7eff5414d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:17.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.645+0000 7eff53e00640 1 -- 172.21.15.27:0/3031132631 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7eff5415ad20 msgr2=0x7eff54163680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.645+0000 7eff53e00640 1 --2- 172.21.15.27:0/3031132631 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7eff5415ad20 0x7eff54163680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.645+0000 7eff53e00640 1 -- 172.21.15.27:0/3031132631 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7eff5415d900 msgr2=0x7eff5414d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.645+0000 7eff53e00640 1 --2- 172.21.15.27:0/3031132631 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7eff5415d900 0x7eff5414d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.645+0000 7eff53e00640 1 -- 172.21.15.27:0/3031132631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff4c009a70 con 0x7eff540715f0 2024-01-15T18:07:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.645+0000 7eff53e00640 1 --2- 172.21.15.27:0/3031132631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff540715f0 0x7eff54160970 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7eff4400e810 tx=0x7eff4400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.645+0000 7eff4be00640 1 -- 172.21.15.27:0/3031132631 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4400c8d0 con 0x7eff540715f0 2024-01-15T18:07:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.645+0000 7eff4be00640 1 -- 172.21.15.27:0/3031132631 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff4400ca30 con 0x7eff540715f0 2024-01-15T18:07:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.645+0000 7eff4be00640 1 -- 172.21.15.27:0/3031132631 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4400cde0 con 0x7eff540715f0 2024-01-15T18:07:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.645+0000 7eff5ac00640 1 -- 172.21.15.27:0/3031132631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff5413db40 con 0x7eff540715f0 2024-01-15T18:07:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.645+0000 7eff5ac00640 1 -- 172.21.15.27:0/3031132631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff5413ca60 con 0x7eff540715f0 2024-01-15T18:07:17.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.645+0000 7eff5ac00640 1 -- 172.21.15.27:0/3031132631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff5404e850 con 0x7eff540715f0 2024-01-15T18:07:17.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.649+0000 7eff4be00640 1 -- 172.21.15.27:0/3031132631 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7eff44013410 con 0x7eff540715f0 2024-01-15T18:07:17.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.649+0000 7eff4be00640 1 --2- 172.21.15.27:0/3031132631 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7eff2c066b30 0x7eff2c068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:17.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.649+0000 7eff4be00640 1 -- 172.21.15.27:0/3031132631 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4885+0+0 (secure 0 0 0) 0x7eff440843f0 con 0x7eff540715f0 2024-01-15T18:07:17.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.649+0000 7eff53400640 1 --2- 172.21.15.27:0/3031132631 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7eff2c066b30 0x7eff2c068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:17.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.649+0000 7eff53400640 1 --2- 172.21.15.27:0/3031132631 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7eff2c066b30 0x7eff2c068ff0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7eff5415b390 tx=0x7eff3c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:17.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.653+0000 7eff4be00640 1 -- 172.21.15.27:0/3031132631 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7eff44051590 con 0x7eff540715f0 2024-01-15T18:07:17.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:17.813+0000 7eff5ac00640 1 -- 172.21.15.27:0/3031132631 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7eff54152390 con 0x7eff540715f0 2024-01-15T18:07:18.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.045+0000 7eff4be00640 1 -- 172.21.15.27:0/3031132631 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v42) v1 ==== 138+0+0 (secure 0 0 0) 0x7eff44054bb0 con 0x7eff540715f0 2024-01-15T18:07:18.052 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-01-15T18:07:18.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.049+0000 7eff5ac00640 1 -- 172.21.15.27:0/3031132631 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7eff2c066b30 msgr2=0x7eff2c068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:18.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.049+0000 7eff5ac00640 1 --2- 172.21.15.27:0/3031132631 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7eff2c066b30 0x7eff2c068ff0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7eff5415b390 tx=0x7eff3c009040 comp rx=0 tx=0).stop 2024-01-15T18:07:18.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.049+0000 7eff5ac00640 1 -- 172.21.15.27:0/3031132631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff540715f0 msgr2=0x7eff54160970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:18.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.049+0000 7eff5ac00640 1 --2- 172.21.15.27:0/3031132631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff540715f0 0x7eff54160970 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7eff4400e810 tx=0x7eff4400ecd0 comp rx=0 tx=0).stop 2024-01-15T18:07:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.049+0000 7eff5ac00640 1 -- 172.21.15.27:0/3031132631 shutdown_connections 2024-01-15T18:07:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.049+0000 7eff5ac00640 1 --2- 172.21.15.27:0/3031132631 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7eff2c066b30 0x7eff2c068ff0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.049+0000 7eff5ac00640 1 --2- 172.21.15.27:0/3031132631 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7eff5415d900 0x7eff5414d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.049+0000 7eff5ac00640 1 --2- 172.21.15.27:0/3031132631 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7eff5415ad20 0x7eff54163680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.049+0000 7eff5ac00640 1 --2- 172.21.15.27:0/3031132631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff540715f0 0x7eff54160970 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.049+0000 7eff5ac00640 1 -- 172.21.15.27:0/3031132631 >> 172.21.15.27:0/3031132631 conn(0x7eff540cb700 msgr2=0x7eff54151460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.049+0000 7eff5ac00640 1 -- 172.21.15.27:0/3031132631 shutdown_connections 2024-01-15T18:07:18.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.049+0000 7eff5ac00640 1 -- 172.21.15.27:0/3031132631 wait complete. 2024-01-15T18:07:18.081 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-15T18:07:18.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.221+0000 7f9ce9c00640 1 Processor -- start 2024-01-15T18:07:18.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.221+0000 7f9ce9c00640 1 -- start start 2024-01-15T18:07:18.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.221+0000 7f9ce9c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f9ce415b100 0x7f9ce4070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:18.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.221+0000 7f9ce9c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce4071860 con 0x7f9ce4071420 2024-01-15T18:07:18.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.221+0000 7f9ce9c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce415a610 con 0x7f9ce415ad20 2024-01-15T18:07:18.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.221+0000 7f9ce9c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ce415e3c0 con 0x7f9ce415b100 2024-01-15T18:07:18.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.225+0000 7f9ce3400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9ce4071420 0x7f9ce415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59380/0 (socket says 172.21.15.27:59380) 2024-01-15T18:07:18.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.225+0000 7f9ce3400640 1 -- 172.21.15.27:0/3997796583 learned_addr learned my addr 172.21.15.27:0/3997796583 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:18.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.225+0000 7f9ce2000640 1 --2- 172.21.15.27:0/3997796583 >> v2:172.21.15.93:3300/0 conn(0x7f9ce415b100 0x7f9ce4070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:18.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.225+0000 7f9ce1600640 1 -- 172.21.15.27:0/3997796583 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3572827955 0 0) 0x7f9ce4071860 con 0x7f9ce4071420 2024-01-15T18:07:18.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.225+0000 7f9ce1600640 1 -- 172.21.15.27:0/3997796583 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cc4003530 con 0x7f9ce4071420 2024-01-15T18:07:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.225+0000 7f9ce2000640 1 -- 172.21.15.27:0/3997796583 >> v1:172.21.15.93:6789/0 conn(0x7f9ce415ad20 legacy=0x7f9ce415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.225+0000 7f9ce2000640 1 -- 172.21.15.27:0/3997796583 >> v1:172.21.15.27:6789/0 conn(0x7f9ce4071420 legacy=0x7f9ce415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.225+0000 7f9ce1600640 1 -- 172.21.15.27:0/3997796583 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3850375236 0 0) 0x7f9cc4003530 con 0x7f9ce4071420 2024-01-15T18:07:18.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.225+0000 7f9ce2000640 1 -- 172.21.15.27:0/3997796583 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce415e6a0 con 0x7f9ce415b100 2024-01-15T18:07:18.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.229+0000 7f9ce2000640 1 --2- 172.21.15.27:0/3997796583 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9ce415b100 0x7f9ce4070ee0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc0097f0 tx=0x7f9ccc01c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2149bf8e5c57ae7d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:18.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.229+0000 7f9ce1600640 1 -- 172.21.15.27:0/3997796583 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc001a00 con 0x7f9ce415b100 2024-01-15T18:07:18.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.229+0000 7f9ce1600640 1 -- 172.21.15.27:0/3997796583 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ccc0a7050 con 0x7f9ce415b100 2024-01-15T18:07:18.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.229+0000 7f9ce1600640 1 -- 172.21.15.27:0/3997796583 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc0ab600 con 0x7f9ce415b100 2024-01-15T18:07:18.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.229+0000 7f9ce9c00640 1 -- 172.21.15.27:0/3997796583 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9ce415b100 msgr2=0x7f9ce4070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:18.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.229+0000 7f9ce9c00640 1 --2- 172.21.15.27:0/3997796583 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9ce415b100 0x7f9ce4070ee0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc0097f0 tx=0x7f9ccc01c2e0 comp rx=0 tx=0).stop 2024-01-15T18:07:18.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.229+0000 7f9ce9c00640 1 -- 172.21.15.27:0/3997796583 shutdown_connections 2024-01-15T18:07:18.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.229+0000 7f9ce9c00640 1 --2- 172.21.15.27:0/3997796583 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9ce415b100 0x7f9ce4070ee0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:18.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.229+0000 7f9ce9c00640 1 -- 172.21.15.27:0/3997796583 >> 172.21.15.27:0/3997796583 conn(0x7f9ce40cb700 msgr2=0x7f9ce40cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:18.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.229+0000 7f9ce9c00640 1 -- 172.21.15.27:0/3997796583 shutdown_connections 2024-01-15T18:07:18.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.229+0000 7f9ce9c00640 1 -- 172.21.15.27:0/3997796583 wait complete. 2024-01-15T18:07:18.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.229+0000 7f9ce9c00640 1 Processor -- start 2024-01-15T18:07:18.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.229+0000 7f9ce9c00640 1 -- start start 2024-01-15T18:07:18.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.233+0000 7f9ce9c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce4071420 0x7f9ce41479b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:18.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.233+0000 7f9ce9c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9ce415ad20 0x7f9ce41549f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:18.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.233+0000 7f9ce9c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9ce415b100 0x7f9ce41364a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.233+0000 7f9ce9c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce4147f90 con 0x7f9ce4071420 2024-01-15T18:07:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.233+0000 7f9ce9c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce4148270 con 0x7f9ce415b100 2024-01-15T18:07:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.233+0000 7f9ce9c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce4148550 con 0x7f9ce415ad20 2024-01-15T18:07:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.233+0000 7f9ce2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce4071420 0x7f9ce41479b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.233+0000 7f9ce2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce4071420 0x7f9ce41479b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33082/0 (socket says 172.21.15.27:33082) 2024-01-15T18:07:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.233+0000 7f9ce2a00640 1 -- 172.21.15.27:0/216152891 learned_addr learned my addr 172.21.15.27:0/216152891 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.233+0000 7f9ce3400640 1 --2- 172.21.15.27:0/216152891 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9ce415b100 0x7f9ce41364a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.233+0000 7f9ce2000640 1 --2- 172.21.15.27:0/216152891 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9ce415ad20 0x7f9ce41549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.233+0000 7f9ce2a00640 1 -- 172.21.15.27:0/216152891 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9ce415ad20 msgr2=0x7f9ce41549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.233+0000 7f9ce2a00640 1 --2- 172.21.15.27:0/216152891 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9ce415ad20 0x7f9ce41549f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.233+0000 7f9ce2a00640 1 -- 172.21.15.27:0/216152891 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9ce415b100 msgr2=0x7f9ce41364a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.233+0000 7f9ce2a00640 1 --2- 172.21.15.27:0/216152891 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9ce415b100 0x7f9ce41364a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.233+0000 7f9ce2a00640 1 -- 172.21.15.27:0/216152891 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ccc009420 con 0x7f9ce4071420 2024-01-15T18:07:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.233+0000 7f9ce2a00640 1 --2- 172.21.15.27:0/216152891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce4071420 0x7f9ce41479b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f9cd400c500 tx=0x7f9cd400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:18.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.233+0000 7f9cdaa00640 1 -- 172.21.15.27:0/216152891 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd4005220 con 0x7f9ce4071420 2024-01-15T18:07:18.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.233+0000 7f9cdaa00640 1 -- 172.21.15.27:0/216152891 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd4005380 con 0x7f9ce4071420 2024-01-15T18:07:18.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.233+0000 7f9cdaa00640 1 -- 172.21.15.27:0/216152891 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd4007270 con 0x7f9ce4071420 2024-01-15T18:07:18.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.233+0000 7f9ce9c00640 1 -- 172.21.15.27:0/216152891 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce4148830 con 0x7f9ce4071420 2024-01-15T18:07:18.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.233+0000 7f9ce9c00640 1 -- 172.21.15.27:0/216152891 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce4154f30 con 0x7f9ce4071420 2024-01-15T18:07:18.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.233+0000 7f9ce9c00640 1 -- 172.21.15.27:0/216152891 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce40719f0 con 0x7f9ce4071420 2024-01-15T18:07:18.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.237+0000 7f9cdaa00640 1 -- 172.21.15.27:0/216152891 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f9cd40073d0 con 0x7f9ce4071420 2024-01-15T18:07:18.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.237+0000 7f9cdaa00640 1 --2- 172.21.15.27:0/216152891 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9cbc066b30 0x7f9cbc068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:18.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.237+0000 7f9cdaa00640 1 -- 172.21.15.27:0/216152891 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5255+0+0 (secure 0 0 0) 0x7f9cd40840a0 con 0x7f9ce4071420 2024-01-15T18:07:18.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.237+0000 7f9ce2000640 1 --2- 172.21.15.27:0/216152891 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9cbc066b30 0x7f9cbc068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:18.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.237+0000 7f9ce2000640 1 --2- 172.21.15.27:0/216152891 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9cbc066b30 0x7f9cbc068ff0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc0ad000 tx=0x7f9ccc0a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:18.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.241+0000 7f9cdaa00640 1 -- 172.21.15.27:0/216152891 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9cd40510b0 con 0x7f9ce4071420 2024-01-15T18:07:18.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:18.401+0000 7f9ce9c00640 1 -- 172.21.15.27:0/216152891 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f9ce404e850 con 0x7f9ce4071420 2024-01-15T18:07:19.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.053+0000 7f9cdaa00640 1 -- 172.21.15.27:0/216152891 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v43) v1 ==== 170+0+0 (secure 0 0 0) 0x7f9cd40546d0 con 0x7f9ce4071420 2024-01-15T18:07:19.059 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-01-15T18:07:19.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.057+0000 7f9ce9c00640 1 -- 172.21.15.27:0/216152891 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9cbc066b30 msgr2=0x7f9cbc068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:19.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.057+0000 7f9ce9c00640 1 --2- 172.21.15.27:0/216152891 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9cbc066b30 0x7f9cbc068ff0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc0ad000 tx=0x7f9ccc0a6000 comp rx=0 tx=0).stop 2024-01-15T18:07:19.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.057+0000 7f9ce9c00640 1 -- 172.21.15.27:0/216152891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce4071420 msgr2=0x7f9ce41479b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:19.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.057+0000 7f9ce9c00640 1 --2- 172.21.15.27:0/216152891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce4071420 0x7f9ce41479b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f9cd400c500 tx=0x7f9cd400c9c0 comp rx=0 tx=0).stop 2024-01-15T18:07:19.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.057+0000 7f9ce9c00640 1 -- 172.21.15.27:0/216152891 shutdown_connections 2024-01-15T18:07:19.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.057+0000 7f9ce9c00640 1 --2- 172.21.15.27:0/216152891 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9cbc066b30 0x7f9cbc068ff0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:19.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.057+0000 7f9ce9c00640 1 --2- 172.21.15.27:0/216152891 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9ce415b100 0x7f9ce41364a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:19.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.057+0000 7f9ce9c00640 1 --2- 172.21.15.27:0/216152891 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9ce415ad20 0x7f9ce41549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:19.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.057+0000 7f9ce9c00640 1 --2- 172.21.15.27:0/216152891 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce4071420 0x7f9ce41479b0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:19.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.057+0000 7f9ce9c00640 1 -- 172.21.15.27:0/216152891 >> 172.21.15.27:0/216152891 conn(0x7f9ce40cb700 msgr2=0x7f9ce40cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:19.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.057+0000 7f9ce9c00640 1 -- 172.21.15.27:0/216152891 shutdown_connections 2024-01-15T18:07:19.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.057+0000 7f9ce9c00640 1 -- 172.21.15.27:0/216152891 wait complete. 2024-01-15T18:07:19.109 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-15T18:07:19.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.249+0000 7f56f8800640 1 Processor -- start 2024-01-15T18:07:19.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.249+0000 7f56f8800640 1 -- start start 2024-01-15T18:07:19.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.249+0000 7f56f8800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f56f00cf2e0 0x7f56f00701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:19.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.249+0000 7f56f8800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56f00b1cb0 con 0x7f56f015f550 2024-01-15T18:07:19.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.249+0000 7f56f8800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56f015fa00 con 0x7f56f0158900 2024-01-15T18:07:19.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.249+0000 7f56f8800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56f00b1f00 con 0x7f56f00cf2e0 2024-01-15T18:07:19.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.249+0000 7f56f5400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f56f015f550 0x7f56f00cebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59388/0 (socket says 172.21.15.27:59388) 2024-01-15T18:07:19.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.249+0000 7f56f5400640 1 -- 172.21.15.27:0/2177758391 learned_addr learned my addr 172.21.15.27:0/2177758391 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:19.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.249+0000 7f56f6800640 1 --2- 172.21.15.27:0/2177758391 >> v2:172.21.15.93:3300/0 conn(0x7f56f00cf2e0 0x7f56f00701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:19.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.249+0000 7f56f4a00640 1 -- 172.21.15.27:0/2177758391 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2083081268 0 0) 0x7f56f00b1cb0 con 0x7f56f015f550 2024-01-15T18:07:19.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.249+0000 7f56f4a00640 1 -- 172.21.15.27:0/2177758391 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56d0003530 con 0x7f56f015f550 2024-01-15T18:07:19.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.249+0000 7f56f4a00640 1 -- 172.21.15.27:0/2177758391 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786520060 0 0) 0x7f56f015fa00 con 0x7f56f0158900 2024-01-15T18:07:19.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.249+0000 7f56f4a00640 1 -- 172.21.15.27:0/2177758391 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56f00b1cb0 con 0x7f56f0158900 2024-01-15T18:07:19.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.249+0000 7f56f6800640 1 -- 172.21.15.27:0/2177758391 >> v1:172.21.15.93:6789/0 conn(0x7f56f0158900 legacy=0x7f56f015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:19.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.249+0000 7f56f6800640 1 -- 172.21.15.27:0/2177758391 >> v1:172.21.15.27:6789/0 conn(0x7f56f015f550 legacy=0x7f56f00cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:19.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.249+0000 7f56f6800640 1 -- 172.21.15.27:0/2177758391 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f00b21e0 con 0x7f56f00cf2e0 2024-01-15T18:07:19.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.253+0000 7f56f6800640 1 --2- 172.21.15.27:0/2177758391 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56f00cf2e0 0x7f56f00701e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f56e8009d90 tx=0x7f56e801c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fdbb745444bab454 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:19.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.253+0000 7f56f4a00640 1 -- 172.21.15.27:0/2177758391 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f0070f30 con 0x7f56f00cf2e0 2024-01-15T18:07:19.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.253+0000 7f56f4a00640 1 -- 172.21.15.27:0/2177758391 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e80a59e0 con 0x7f56f00cf2e0 2024-01-15T18:07:19.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.253+0000 7f56f4a00640 1 -- 172.21.15.27:0/2177758391 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e80a5d70 con 0x7f56f00cf2e0 2024-01-15T18:07:19.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.253+0000 7f56f8800640 1 -- 172.21.15.27:0/2177758391 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56f00cf2e0 msgr2=0x7f56f00701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:19.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.253+0000 7f56f8800640 1 --2- 172.21.15.27:0/2177758391 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56f00cf2e0 0x7f56f00701e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f56e8009d90 tx=0x7f56e801c920 comp rx=0 tx=0).stop 2024-01-15T18:07:19.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.253+0000 7f56f8800640 1 -- 172.21.15.27:0/2177758391 shutdown_connections 2024-01-15T18:07:19.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.253+0000 7f56f8800640 1 --2- 172.21.15.27:0/2177758391 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56f00cf2e0 0x7f56f00701e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:19.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.253+0000 7f56f8800640 1 -- 172.21.15.27:0/2177758391 >> 172.21.15.27:0/2177758391 conn(0x7f56f00cb700 msgr2=0x7f56f00cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:19.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.253+0000 7f56f8800640 1 -- 172.21.15.27:0/2177758391 shutdown_connections 2024-01-15T18:07:19.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.253+0000 7f56f8800640 1 -- 172.21.15.27:0/2177758391 wait complete. 2024-01-15T18:07:19.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.257+0000 7f56f8800640 1 Processor -- start 2024-01-15T18:07:19.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.257+0000 7f56f8800640 1 -- start start 2024-01-15T18:07:19.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.257+0000 7f56f8800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56f00cf2e0 0x7f56f014ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:19.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.257+0000 7f56f8800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56f0158900 0x7f56f01534b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:19.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.257+0000 7f56f8800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f015f550 0x7f56f01539f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:19.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.257+0000 7f56f8800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f00cc810 con 0x7f56f015f550 2024-01-15T18:07:19.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.257+0000 7f56f8800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f00ce1e0 con 0x7f56f00cf2e0 2024-01-15T18:07:19.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.257+0000 7f56f8800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f00ce480 con 0x7f56f0158900 2024-01-15T18:07:19.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.257+0000 7f56f6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f015f550 0x7f56f01539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:19.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.257+0000 7f56f6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f015f550 0x7f56f01539f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33092/0 (socket says 172.21.15.27:33092) 2024-01-15T18:07:19.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.257+0000 7f56f6800640 1 -- 172.21.15.27:0/345444689 learned_addr learned my addr 172.21.15.27:0/345444689 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:19.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.257+0000 7f56f5e00640 1 --2- 172.21.15.27:0/345444689 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56f00cf2e0 0x7f56f014ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:19.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.257+0000 7f56f5400640 1 --2- 172.21.15.27:0/345444689 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56f0158900 0x7f56f01534b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:19.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.257+0000 7f56f6800640 1 -- 172.21.15.27:0/345444689 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56f0158900 msgr2=0x7f56f01534b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:19.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.257+0000 7f56f6800640 1 --2- 172.21.15.27:0/345444689 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56f0158900 0x7f56f01534b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:19.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.257+0000 7f56f6800640 1 -- 172.21.15.27:0/345444689 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56f00cf2e0 msgr2=0x7f56f014ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:19.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.257+0000 7f56f6800640 1 --2- 172.21.15.27:0/345444689 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56f00cf2e0 0x7f56f014ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:19.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.257+0000 7f56f6800640 1 -- 172.21.15.27:0/345444689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e8009a70 con 0x7f56f015f550 2024-01-15T18:07:19.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.257+0000 7f56f6800640 1 --2- 172.21.15.27:0/345444689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f015f550 0x7f56f01539f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f56e801a550 tx=0x7f56e80196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:19.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.257+0000 7f56ee000640 1 -- 172.21.15.27:0/345444689 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e80b9860 con 0x7f56f015f550 2024-01-15T18:07:19.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.257+0000 7f56ee000640 1 -- 172.21.15.27:0/345444689 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e80b99c0 con 0x7f56f015f550 2024-01-15T18:07:19.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.257+0000 7f56f8800640 1 -- 172.21.15.27:0/345444689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56f00ccd40 con 0x7f56f015f550 2024-01-15T18:07:19.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.257+0000 7f56ee000640 1 -- 172.21.15.27:0/345444689 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e80b9d30 con 0x7f56f015f550 2024-01-15T18:07:19.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.257+0000 7f56f8800640 1 -- 172.21.15.27:0/345444689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56f00cca60 con 0x7f56f015f550 2024-01-15T18:07:19.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.261+0000 7f56f8800640 1 -- 172.21.15.27:0/345444689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56f00aca50 con 0x7f56f015f550 2024-01-15T18:07:19.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.261+0000 7f56ee000640 1 -- 172.21.15.27:0/345444689 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f56e80ad440 con 0x7f56f015f550 2024-01-15T18:07:19.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.261+0000 7f56ee000640 1 --2- 172.21.15.27:0/345444689 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f56c8066b30 0x7f56c8068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:19.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.261+0000 7f56ee000640 1 -- 172.21.15.27:0/345444689 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5637+0+0 (secure 0 0 0) 0x7f56e811fed0 con 0x7f56f015f550 2024-01-15T18:07:19.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.261+0000 7f56f5e00640 1 --2- 172.21.15.27:0/345444689 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f56c8066b30 0x7f56c8068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:19.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.265+0000 7f56f5e00640 1 --2- 172.21.15.27:0/345444689 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f56c8066b30 0x7f56c8068ff0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f56e00011f0 tx=0x7f56e0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:19.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.265+0000 7f56ee000640 1 -- 172.21.15.27:0/345444689 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f56e80ecf10 con 0x7f56f015f550 2024-01-15T18:07:19.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:19.453+0000 7f56f8800640 1 -- 172.21.15.27:0/345444689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f56f004e850 con 0x7f56f015f550 2024-01-15T18:07:20.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.056+0000 7f56ee000640 1 -- 172.21.15.27:0/345444689 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v23) v1 ==== 172+0+0 (secure 0 0 0) 0x7f56e80f0530 con 0x7f56f015f550 2024-01-15T18:07:20.063 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 6 and data pool 7 2024-01-15T18:07:20.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.056+0000 7f56f8800640 1 -- 172.21.15.27:0/345444689 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f56c8066b30 msgr2=0x7f56c8068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:20.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.056+0000 7f56f8800640 1 --2- 172.21.15.27:0/345444689 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f56c8066b30 0x7f56c8068ff0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f56e00011f0 tx=0x7f56e0009040 comp rx=0 tx=0).stop 2024-01-15T18:07:20.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.056+0000 7f56f8800640 1 -- 172.21.15.27:0/345444689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f015f550 msgr2=0x7f56f01539f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:20.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.056+0000 7f56f8800640 1 --2- 172.21.15.27:0/345444689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f015f550 0x7f56f01539f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f56e801a550 tx=0x7f56e80196c0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.056+0000 7f56f8800640 1 -- 172.21.15.27:0/345444689 shutdown_connections 2024-01-15T18:07:20.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.056+0000 7f56f8800640 1 --2- 172.21.15.27:0/345444689 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f56c8066b30 0x7f56c8068ff0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.056+0000 7f56f8800640 1 --2- 172.21.15.27:0/345444689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f015f550 0x7f56f01539f0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.056+0000 7f56f8800640 1 --2- 172.21.15.27:0/345444689 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56f0158900 0x7f56f01534b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.056+0000 7f56f8800640 1 --2- 172.21.15.27:0/345444689 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56f00cf2e0 0x7f56f014ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.056+0000 7f56f8800640 1 -- 172.21.15.27:0/345444689 >> 172.21.15.27:0/345444689 conn(0x7f56f00cb700 msgr2=0x7f56f00ae770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:20.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.060+0000 7f56f8800640 1 -- 172.21.15.27:0/345444689 shutdown_connections 2024-01-15T18:07:20.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.060+0000 7f56f8800640 1 -- 172.21.15.27:0/345444689 wait complete. 2024-01-15T18:07:20.121 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:07:20.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.256+0000 7fc7b8000640 1 Processor -- start 2024-01-15T18:07:20.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.256+0000 7fc7b8000640 1 -- start start 2024-01-15T18:07:20.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.256+0000 7fc7b8000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7b015e730 0x7fc7b00cd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:20.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.256+0000 7fc7b8000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b00ce920 con 0x7fc7b006db80 2024-01-15T18:07:20.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.256+0000 7fc7b8000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b00ce640 con 0x7fc7b00cdd10 2024-01-15T18:07:20.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.256+0000 7fc7b8000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b00cc940 con 0x7fc7b015e730 2024-01-15T18:07:20.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.260+0000 7fc7b5600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc7b006db80 0x7fc7b006df60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59402/0 (socket says 172.21.15.27:59402) 2024-01-15T18:07:20.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.260+0000 7fc7b5600640 1 -- 172.21.15.27:0/494038456 learned_addr learned my addr 172.21.15.27:0/494038456 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:20.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.260+0000 7fc7b4c00640 1 --2- 172.21.15.27:0/494038456 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7b015e730 0x7fc7b00cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:20.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.260+0000 7fc7afe00640 1 -- 172.21.15.27:0/494038456 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458346250 0 0) 0x7fc7b00ce920 con 0x7fc7b006db80 2024-01-15T18:07:20.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.260+0000 7fc7afe00640 1 -- 172.21.15.27:0/494038456 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc790003530 con 0x7fc7b006db80 2024-01-15T18:07:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.260+0000 7fc7afe00640 1 -- 172.21.15.27:0/494038456 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2882720833 0 0) 0x7fc7b00ce640 con 0x7fc7b00cdd10 2024-01-15T18:07:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.260+0000 7fc7afe00640 1 -- 172.21.15.27:0/494038456 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7b00ce920 con 0x7fc7b00cdd10 2024-01-15T18:07:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.260+0000 7fc7b4c00640 1 -- 172.21.15.27:0/494038456 >> v1:172.21.15.93:6789/0 conn(0x7fc7b00cdd10 legacy=0x7fc7b0070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.260+0000 7fc7b4c00640 1 -- 172.21.15.27:0/494038456 >> v1:172.21.15.27:6789/0 conn(0x7fc7b006db80 legacy=0x7fc7b006df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.260+0000 7fc7b4c00640 1 -- 172.21.15.27:0/494038456 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b00ccc20 con 0x7fc7b015e730 2024-01-15T18:07:20.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.260+0000 7fc7b4c00640 1 --2- 172.21.15.27:0/494038456 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7b015e730 0x7fc7b00cd420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc7980097f0 tx=0x7fc79801c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58dd37126321c499 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:20.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.260+0000 7fc7afe00640 1 -- 172.21.15.27:0/494038456 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7980aa610 con 0x7fc7b015e730 2024-01-15T18:07:20.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.260+0000 7fc7afe00640 1 -- 172.21.15.27:0/494038456 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7980a6040 con 0x7fc7b015e730 2024-01-15T18:07:20.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.260+0000 7fc7afe00640 1 -- 172.21.15.27:0/494038456 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7980aa970 con 0x7fc7b015e730 2024-01-15T18:07:20.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.264+0000 7fc7b8000640 1 -- 172.21.15.27:0/494038456 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7b015e730 msgr2=0x7fc7b00cd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:20.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.264+0000 7fc7b8000640 1 --2- 172.21.15.27:0/494038456 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7b015e730 0x7fc7b00cd420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc7980097f0 tx=0x7fc79801c490 comp rx=0 tx=0).stop 2024-01-15T18:07:20.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.264+0000 7fc7b8000640 1 -- 172.21.15.27:0/494038456 shutdown_connections 2024-01-15T18:07:20.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.264+0000 7fc7b8000640 1 --2- 172.21.15.27:0/494038456 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7b015e730 0x7fc7b00cd420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.264+0000 7fc7b8000640 1 -- 172.21.15.27:0/494038456 >> 172.21.15.27:0/494038456 conn(0x7fc7b00cec60 msgr2=0x7fc7b00cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:20.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.264+0000 7fc7b8000640 1 -- 172.21.15.27:0/494038456 shutdown_connections 2024-01-15T18:07:20.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.264+0000 7fc7b8000640 1 -- 172.21.15.27:0/494038456 wait complete. 2024-01-15T18:07:20.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.264+0000 7fc7b8000640 1 Processor -- start 2024-01-15T18:07:20.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.264+0000 7fc7b8000640 1 -- start start 2024-01-15T18:07:20.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.264+0000 7fc7b8000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7b006db80 0x7fc7b0155b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:20.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.264+0000 7fc7b8000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7b00cdd10 0x7fc7b0150390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:20.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.264+0000 7fc7b8000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7b015e730 0x7fc7b0154010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:20.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.264+0000 7fc7b8000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b01508d0 con 0x7fc7b015e730 2024-01-15T18:07:20.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.264+0000 7fc7b8000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b0154a60 con 0x7fc7b006db80 2024-01-15T18:07:20.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.264+0000 7fc7b8000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b0154d40 con 0x7fc7b00cdd10 2024-01-15T18:07:20.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.264+0000 7fc7b6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7b015e730 0x7fc7b0154010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:20.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.268+0000 7fc7b4c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7b00cdd10 0x7fc7b0150390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:20.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.268+0000 7fc7b5600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7b006db80 0x7fc7b0155b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:20.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.268+0000 7fc7b6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7b015e730 0x7fc7b0154010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33098/0 (socket says 172.21.15.27:33098) 2024-01-15T18:07:20.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.268+0000 7fc7b6000640 1 -- 172.21.15.27:0/930675383 learned_addr learned my addr 172.21.15.27:0/930675383 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:20.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.268+0000 7fc7b6000640 1 -- 172.21.15.27:0/930675383 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7b00cdd10 msgr2=0x7fc7b0150390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:20.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.268+0000 7fc7b6000640 1 --2- 172.21.15.27:0/930675383 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7b00cdd10 0x7fc7b0150390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.268+0000 7fc7b6000640 1 -- 172.21.15.27:0/930675383 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7b006db80 msgr2=0x7fc7b0155b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:20.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.268+0000 7fc7b6000640 1 --2- 172.21.15.27:0/930675383 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7b006db80 0x7fc7b0155b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.268+0000 7fc7b6000640 1 -- 172.21.15.27:0/930675383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc798009420 con 0x7fc7b015e730 2024-01-15T18:07:20.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.268+0000 7fc7b6000640 1 --2- 172.21.15.27:0/930675383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7b015e730 0x7fc7b0154010 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fc7a800c790 tx=0x7fc7a800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:20.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.268+0000 7fc7ad600640 1 -- 172.21.15.27:0/930675383 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a800cea0 con 0x7fc7b015e730 2024-01-15T18:07:20.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.268+0000 7fc7ad600640 1 -- 172.21.15.27:0/930675383 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a8003b80 con 0x7fc7b015e730 2024-01-15T18:07:20.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.268+0000 7fc7ad600640 1 -- 172.21.15.27:0/930675383 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a800a850 con 0x7fc7b015e730 2024-01-15T18:07:20.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.268+0000 7fc7b8000640 1 -- 172.21.15.27:0/930675383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b0155020 con 0x7fc7b015e730 2024-01-15T18:07:20.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.268+0000 7fc7b8000640 1 -- 172.21.15.27:0/930675383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b01555e0 con 0x7fc7b015e730 2024-01-15T18:07:20.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.268+0000 7fc7b8000640 1 -- 172.21.15.27:0/930675383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b01557d0 con 0x7fc7b015e730 2024-01-15T18:07:20.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.268+0000 7fc7ad600640 1 -- 172.21.15.27:0/930675383 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7fc7a800a9b0 con 0x7fc7b015e730 2024-01-15T18:07:20.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.272+0000 7fc7ad600640 1 --2- 172.21.15.27:0/930675383 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc788066b30 0x7fc788068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:20.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.272+0000 7fc7ad600640 1 -- 172.21.15.27:0/930675383 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc7a8085090 con 0x7fc7b015e730 2024-01-15T18:07:20.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.272+0000 7fc7b5600640 1 --2- 172.21.15.27:0/930675383 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc788066b30 0x7fc788068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:20.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.272+0000 7fc7b5600640 1 --2- 172.21.15.27:0/930675383 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc788066b30 0x7fc788068ff0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0007fd0 tx=0x7fc7a0007d60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:20.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.272+0000 7fc7ad600640 1 -- 172.21.15.27:0/930675383 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc7a8052070 con 0x7fc7b015e730 2024-01-15T18:07:20.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.432+0000 7fc7b8000640 1 -- 172.21.15.27:0/930675383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7b004e850 con 0x7fc7b015e730 2024-01-15T18:07:20.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.432+0000 7fc7ad600640 1 -- 172.21.15.27:0/930675383 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16473 (secure 0 0 0) 0x7fc7a8055690 con 0x7fc7b015e730 2024-01-15T18:07:20.442 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:07:20.442 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":44,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:07:19.460044+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:07:17.862115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-15T18:07:18.453645+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:07:20.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.436+0000 7fc7b8000640 1 -- 172.21.15.27:0/930675383 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc788066b30 msgr2=0x7fc788068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:20.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.436+0000 7fc7b8000640 1 --2- 172.21.15.27:0/930675383 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc788066b30 0x7fc788068ff0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0007fd0 tx=0x7fc7a0007d60 comp rx=0 tx=0).stop 2024-01-15T18:07:20.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.436+0000 7fc7b8000640 1 -- 172.21.15.27:0/930675383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7b015e730 msgr2=0x7fc7b0154010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:20.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.436+0000 7fc7b8000640 1 --2- 172.21.15.27:0/930675383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7b015e730 0x7fc7b0154010 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fc7a800c790 tx=0x7fc7a800cc50 comp rx=0 tx=0).stop 2024-01-15T18:07:20.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.436+0000 7fc7b8000640 1 -- 172.21.15.27:0/930675383 shutdown_connections 2024-01-15T18:07:20.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.436+0000 7fc7b8000640 1 --2- 172.21.15.27:0/930675383 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc788066b30 0x7fc788068ff0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.436+0000 7fc7b8000640 1 --2- 172.21.15.27:0/930675383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7b015e730 0x7fc7b0154010 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.436+0000 7fc7b8000640 1 --2- 172.21.15.27:0/930675383 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7b00cdd10 0x7fc7b0150390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.436+0000 7fc7b8000640 1 --2- 172.21.15.27:0/930675383 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7b006db80 0x7fc7b0155b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.436+0000 7fc7b8000640 1 -- 172.21.15.27:0/930675383 >> 172.21.15.27:0/930675383 conn(0x7fc7b00cec60 msgr2=0x7fc7b00c9f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:20.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.436+0000 7fc7b8000640 1 -- 172.21.15.27:0/930675383 shutdown_connections 2024-01-15T18:07:20.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.436+0000 7fc7b8000640 1 -- 172.21.15.27:0/930675383 wait complete. 2024-01-15T18:07:20.463 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:07:20.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.596+0000 7f677e800640 1 Processor -- start 2024-01-15T18:07:20.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.596+0000 7f677e800640 1 -- start start 2024-01-15T18:07:20.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.596+0000 7f677e800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6778071e40 0x7f67780c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:20.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.596+0000 7f677e800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f677806d430 0x7f67780cd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:20.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.596+0000 7f677e800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6778159a50 con 0x7f6778159bf0 2024-01-15T18:07:20.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.596+0000 7f677e800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67780cce00 con 0x7f677806d430 2024-01-15T18:07:20.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.596+0000 7f677e800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f677806d980 con 0x7f6778071e40 2024-01-15T18:07:20.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.600+0000 7f6777400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6778159bf0 0x7f6778159fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59414/0 (socket says 172.21.15.27:59414) 2024-01-15T18:07:20.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.600+0000 7f6777400640 1 -- 172.21.15.27:0/1356922633 learned_addr learned my addr 172.21.15.27:0/1356922633 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:20.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.600+0000 7f6777e00640 1 --2- 172.21.15.27:0/1356922633 >> v2:172.21.15.93:3300/0 conn(0x7f677806d430 0x7f67780cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:20.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.600+0000 7f6776a00640 1 --2- 172.21.15.27:0/1356922633 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6778071e40 0x7f67780c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:20.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.600+0000 7f6776000640 1 -- 172.21.15.27:0/1356922633 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2511737508 0 0) 0x7f6768001120 con 0x7f6778159bf0 2024-01-15T18:07:20.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.600+0000 7f6776000640 1 -- 172.21.15.27:0/1356922633 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6758003530 con 0x7f6778159bf0 2024-01-15T18:07:20.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.600+0000 7f6776a00640 1 -- 172.21.15.27:0/1356922633 >> v2:172.21.15.93:3300/0 conn(0x7f677806d430 msgr2=0x7f67780cd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:20.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.600+0000 7f6776a00640 1 --2- 172.21.15.27:0/1356922633 >> v2:172.21.15.93:3300/0 conn(0x7f677806d430 0x7f67780cd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.600+0000 7f6776a00640 1 -- 172.21.15.27:0/1356922633 >> v1:172.21.15.27:6789/0 conn(0x7f6778159bf0 legacy=0x7f6778159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:20.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.600+0000 7f6776a00640 1 -- 172.21.15.27:0/1356922633 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f677806dc60 con 0x7f6778071e40 2024-01-15T18:07:20.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.600+0000 7f6776a00640 1 --2- 172.21.15.27:0/1356922633 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6778071e40 0x7f67780c9be0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f67600097f0 tx=0x7f676001ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d6a547f89fb63d5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:20.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.604+0000 7f6776000640 1 -- 172.21.15.27:0/1356922633 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67600a8970 con 0x7f6778071e40 2024-01-15T18:07:20.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.604+0000 7f6776000640 1 -- 172.21.15.27:0/1356922633 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67600a8ad0 con 0x7f6778071e40 2024-01-15T18:07:20.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.604+0000 7f6776000640 1 -- 172.21.15.27:0/1356922633 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67600a8e60 con 0x7f6778071e40 2024-01-15T18:07:20.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.604+0000 7f677e800640 1 -- 172.21.15.27:0/1356922633 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6778071e40 msgr2=0x7f67780c9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:20.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.604+0000 7f677e800640 1 --2- 172.21.15.27:0/1356922633 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6778071e40 0x7f67780c9be0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f67600097f0 tx=0x7f676001ec70 comp rx=0 tx=0).stop 2024-01-15T18:07:20.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.604+0000 7f677e800640 1 -- 172.21.15.27:0/1356922633 shutdown_connections 2024-01-15T18:07:20.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.604+0000 7f677e800640 1 --2- 172.21.15.27:0/1356922633 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6778071e40 0x7f67780c9be0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.604+0000 7f677e800640 1 --2- 172.21.15.27:0/1356922633 >> v2:172.21.15.93:3300/0 conn(0x7f677806d430 0x7f67780cd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.604+0000 7f677e800640 1 -- 172.21.15.27:0/1356922633 >> 172.21.15.27:0/1356922633 conn(0x7f67780cec60 msgr2=0x7f67780cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:20.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.604+0000 7f677e800640 1 -- 172.21.15.27:0/1356922633 shutdown_connections 2024-01-15T18:07:20.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.604+0000 7f677e800640 1 -- 172.21.15.27:0/1356922633 wait complete. 2024-01-15T18:07:20.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.604+0000 7f677e800640 1 Processor -- start 2024-01-15T18:07:20.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.604+0000 7f677e800640 1 -- start start 2024-01-15T18:07:20.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.604+0000 7f677e800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f677806d430 0x7f67781f3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:20.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.608+0000 7f677e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6778159bf0 0x7f6778149f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:20.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.608+0000 7f677e800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f67781f6370 0x7f6778158b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:20.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.608+0000 7f677e800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67781f3d80 con 0x7f6778159bf0 2024-01-15T18:07:20.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.608+0000 7f677e800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67781f2d80 con 0x7f677806d430 2024-01-15T18:07:20.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.608+0000 7f677e800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67781f6df0 con 0x7f67781f6370 2024-01-15T18:07:20.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.608+0000 7f6776a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6778159bf0 0x7f6778149f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:20.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.608+0000 7f6776a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6778159bf0 0x7f6778149f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33100/0 (socket says 172.21.15.27:33100) 2024-01-15T18:07:20.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.608+0000 7f6776a00640 1 -- 172.21.15.27:0/1553110257 learned_addr learned my addr 172.21.15.27:0/1553110257 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:20.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.608+0000 7f6777400640 1 --2- 172.21.15.27:0/1553110257 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f677806d430 0x7f67781f3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:20.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.608+0000 7f6777e00640 1 --2- 172.21.15.27:0/1553110257 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f67781f6370 0x7f6778158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:20.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.608+0000 7f6776a00640 1 -- 172.21.15.27:0/1553110257 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f67781f6370 msgr2=0x7f6778158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:20.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.608+0000 7f6776a00640 1 --2- 172.21.15.27:0/1553110257 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f67781f6370 0x7f6778158b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.608+0000 7f6776a00640 1 -- 172.21.15.27:0/1553110257 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f677806d430 msgr2=0x7f67781f3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:20.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.608+0000 7f6776a00640 1 --2- 172.21.15.27:0/1553110257 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f677806d430 0x7f67781f3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.608+0000 7f6776a00640 1 -- 172.21.15.27:0/1553110257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6760009420 con 0x7f6778159bf0 2024-01-15T18:07:20.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.608+0000 7f6776a00640 1 --2- 172.21.15.27:0/1553110257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6778159bf0 0x7f6778149f70 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f67600b3040 tx=0x7f676001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:20.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.608+0000 7f676f400640 1 -- 172.21.15.27:0/1553110257 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67600b0d40 con 0x7f6778159bf0 2024-01-15T18:07:20.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.608+0000 7f676f400640 1 -- 172.21.15.27:0/1553110257 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67600b9430 con 0x7f6778159bf0 2024-01-15T18:07:20.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.608+0000 7f676f400640 1 -- 172.21.15.27:0/1553110257 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67600b0ea0 con 0x7f6778159bf0 2024-01-15T18:07:20.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.608+0000 7f677e800640 1 -- 172.21.15.27:0/1553110257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67781f70d0 con 0x7f6778159bf0 2024-01-15T18:07:20.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.608+0000 7f677e800640 1 -- 172.21.15.27:0/1553110257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67781f3320 con 0x7f6778159bf0 2024-01-15T18:07:20.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.608+0000 7f677e800640 1 -- 172.21.15.27:0/1553110257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67781f3510 con 0x7f6778159bf0 2024-01-15T18:07:20.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.608+0000 7f676f400640 1 -- 172.21.15.27:0/1553110257 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f67600b7040 con 0x7f6778159bf0 2024-01-15T18:07:20.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.612+0000 7f676f400640 1 --2- 172.21.15.27:0/1553110257 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6750066b30 0x7f6750068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:20.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.612+0000 7f676f400640 1 -- 172.21.15.27:0/1553110257 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f67601222c0 con 0x7f6778159bf0 2024-01-15T18:07:20.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.612+0000 7f6777400640 1 --2- 172.21.15.27:0/1553110257 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6750066b30 0x7f6750068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:20.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.612+0000 7f6777400640 1 --2- 172.21.15.27:0/1553110257 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6750066b30 0x7f6750068ff0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6768001280 tx=0x7f6768007bc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:20.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.616+0000 7f676f400640 1 -- 172.21.15.27:0/1553110257 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f67600ef190 con 0x7f6778159bf0 2024-01-15T18:07:20.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.772+0000 7f677e800640 1 -- 172.21.15.27:0/1553110257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f677804e850 con 0x7f6778159bf0 2024-01-15T18:07:20.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.772+0000 7f676f400640 1 -- 172.21.15.27:0/1553110257 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16473 (secure 0 0 0) 0x7f67600f27b0 con 0x7f6778159bf0 2024-01-15T18:07:20.782 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:07:20.782 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":44,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:07:19.460044+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:07:17.862115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-15T18:07:18.453645+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:07:20.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.776+0000 7f677e800640 1 -- 172.21.15.27:0/1553110257 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6750066b30 msgr2=0x7f6750068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:20.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.776+0000 7f677e800640 1 --2- 172.21.15.27:0/1553110257 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6750066b30 0x7f6750068ff0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6768001280 tx=0x7f6768007bc0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.776+0000 7f677e800640 1 -- 172.21.15.27:0/1553110257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6778159bf0 msgr2=0x7f6778149f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:20.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.776+0000 7f677e800640 1 --2- 172.21.15.27:0/1553110257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6778159bf0 0x7f6778149f70 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f67600b3040 tx=0x7f676001ba10 comp rx=0 tx=0).stop 2024-01-15T18:07:20.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.776+0000 7f677e800640 1 -- 172.21.15.27:0/1553110257 shutdown_connections 2024-01-15T18:07:20.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.776+0000 7f677e800640 1 --2- 172.21.15.27:0/1553110257 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6750066b30 0x7f6750068ff0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.776+0000 7f677e800640 1 --2- 172.21.15.27:0/1553110257 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f67781f6370 0x7f6778158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.776+0000 7f677e800640 1 --2- 172.21.15.27:0/1553110257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6778159bf0 0x7f6778149f70 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.776+0000 7f677e800640 1 --2- 172.21.15.27:0/1553110257 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f677806d430 0x7f67781f3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.776+0000 7f677e800640 1 -- 172.21.15.27:0/1553110257 >> 172.21.15.27:0/1553110257 conn(0x7f67780cec60 msgr2=0x7f6778070980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:20.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.776+0000 7f677e800640 1 -- 172.21.15.27:0/1553110257 shutdown_connections 2024-01-15T18:07:20.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.776+0000 7f677e800640 1 -- 172.21.15.27:0/1553110257 wait complete. 2024-01-15T18:07:20.805 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-15T18:07:20.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.940+0000 7f5f84e00640 1 Processor -- start 2024-01-15T18:07:20.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.940+0000 7f5f84e00640 1 -- start start 2024-01-15T18:07:20.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.944+0000 7f5f84e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5f8006c430 0x7f5f80070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:20.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.944+0000 7f5f84e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5f800ce760 0x7f5f800b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:20.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.944+0000 7f5f84e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f800ccfb0 con 0x7f5f800b1db0 2024-01-15T18:07:20.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.944+0000 7f5f84e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f800cd290 con 0x7f5f800ce760 2024-01-15T18:07:20.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.944+0000 7f5f84e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f800cd570 con 0x7f5f8006c430 2024-01-15T18:07:20.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.944+0000 7f5f7d400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5f800b1db0 0x7f5f800ce050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59416/0 (socket says 172.21.15.27:59416) 2024-01-15T18:07:20.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.944+0000 7f5f7d400640 1 -- 172.21.15.27:0/1234631904 learned_addr learned my addr 172.21.15.27:0/1234631904 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:20.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.944+0000 7f5f7e800640 1 --2- 172.21.15.27:0/1234631904 >> v2:172.21.15.27:3300/0 conn(0x7f5f800ce760 0x7f5f800b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:20.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.944+0000 7f5f7de00640 1 --2- 172.21.15.27:0/1234631904 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5f8006c430 0x7f5f80070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:20.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.944+0000 7f5f7ca00640 1 -- 172.21.15.27:0/1234631904 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563231608 0 0) 0x7f5f800ccfb0 con 0x7f5f800b1db0 2024-01-15T18:07:20.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.944+0000 7f5f7ca00640 1 -- 172.21.15.27:0/1234631904 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f60003530 con 0x7f5f800b1db0 2024-01-15T18:07:20.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.944+0000 7f5f7e800640 1 -- 172.21.15.27:0/1234631904 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5f8006c430 msgr2=0x7f5f80070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:20.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.944+0000 7f5f7e800640 1 --2- 172.21.15.27:0/1234631904 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5f8006c430 0x7f5f80070ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.944+0000 7f5f7e800640 1 -- 172.21.15.27:0/1234631904 >> v1:172.21.15.27:6789/0 conn(0x7f5f800b1db0 legacy=0x7f5f800ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:20.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.944+0000 7f5f7e800640 1 -- 172.21.15.27:0/1234631904 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f800cd850 con 0x7f5f800ce760 2024-01-15T18:07:20.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.948+0000 7f5f7e800640 1 --2- 172.21.15.27:0/1234631904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f800ce760 0x7f5f800b2210 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f5f7800aea0 tx=0x7f5f7801d9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c50dfb171dad948 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:20.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.948+0000 7f5f7ca00640 1 -- 172.21.15.27:0/1234631904 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7800a2d0 con 0x7f5f800ce760 2024-01-15T18:07:20.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.948+0000 7f5f7ca00640 1 -- 172.21.15.27:0/1234631904 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f780a6730 con 0x7f5f800ce760 2024-01-15T18:07:20.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.948+0000 7f5f7ca00640 1 -- 172.21.15.27:0/1234631904 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f780a6ac0 con 0x7f5f800ce760 2024-01-15T18:07:20.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.948+0000 7f5f84e00640 1 -- 172.21.15.27:0/1234631904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f800ce760 msgr2=0x7f5f800b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:20.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.948+0000 7f5f84e00640 1 --2- 172.21.15.27:0/1234631904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f800ce760 0x7f5f800b2210 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f5f7800aea0 tx=0x7f5f7801d9c0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.948+0000 7f5f84e00640 1 -- 172.21.15.27:0/1234631904 shutdown_connections 2024-01-15T18:07:20.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.948+0000 7f5f84e00640 1 --2- 172.21.15.27:0/1234631904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f800ce760 0x7f5f800b2210 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.948+0000 7f5f84e00640 1 --2- 172.21.15.27:0/1234631904 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5f8006c430 0x7f5f80070ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.948+0000 7f5f84e00640 1 -- 172.21.15.27:0/1234631904 >> 172.21.15.27:0/1234631904 conn(0x7f5f800cb700 msgr2=0x7f5f800cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:20.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.948+0000 7f5f84e00640 1 -- 172.21.15.27:0/1234631904 shutdown_connections 2024-01-15T18:07:20.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.948+0000 7f5f84e00640 1 -- 172.21.15.27:0/1234631904 wait complete. 2024-01-15T18:07:20.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.948+0000 7f5f84e00640 1 Processor -- start 2024-01-15T18:07:20.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.948+0000 7f5f84e00640 1 -- start start 2024-01-15T18:07:20.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.952+0000 7f5f84e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5f8006c430 0x7f5f8015c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:20.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.952+0000 7f5f84e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5f800b1db0 0x7f5f801421e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:20.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.952+0000 7f5f84e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f800ce760 0x7f5f801f9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:20.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.952+0000 7f5f84e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f80142ac0 con 0x7f5f800ce760 2024-01-15T18:07:20.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.952+0000 7f5f84e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f80142da0 con 0x7f5f8006c430 2024-01-15T18:07:20.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.952+0000 7f5f84e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f80143080 con 0x7f5f800b1db0 2024-01-15T18:07:20.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.952+0000 7f5f7e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f800ce760 0x7f5f801f9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:20.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.952+0000 7f5f7e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f800ce760 0x7f5f801f9bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33106/0 (socket says 172.21.15.27:33106) 2024-01-15T18:07:20.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.952+0000 7f5f7e800640 1 -- 172.21.15.27:0/124842054 learned_addr learned my addr 172.21.15.27:0/124842054 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:20.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.952+0000 7f5f7de00640 1 --2- 172.21.15.27:0/124842054 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5f8006c430 0x7f5f8015c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:20.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.952+0000 7f5f7d400640 1 --2- 172.21.15.27:0/124842054 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5f800b1db0 0x7f5f801421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:20.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.952+0000 7f5f7e800640 1 -- 172.21.15.27:0/124842054 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5f800b1db0 msgr2=0x7f5f801421e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:20.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.952+0000 7f5f7e800640 1 --2- 172.21.15.27:0/124842054 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5f800b1db0 0x7f5f801421e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.952+0000 7f5f7e800640 1 -- 172.21.15.27:0/124842054 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5f8006c430 msgr2=0x7f5f8015c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:20.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.952+0000 7f5f7e800640 1 --2- 172.21.15.27:0/124842054 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5f8006c430 0x7f5f8015c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.952+0000 7f5f7e800640 1 -- 172.21.15.27:0/124842054 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f7800ab80 con 0x7f5f800ce760 2024-01-15T18:07:20.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.952+0000 7f5f7d400640 1 --2- 172.21.15.27:0/124842054 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5f800b1db0 0x7f5f801421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:07:20.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.952+0000 7f5f7e800640 1 --2- 172.21.15.27:0/124842054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f800ce760 0x7f5f801f9bb0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f5f7801b5f0 tx=0x7f5f7801a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:20.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.952+0000 7f5f76000640 1 -- 172.21.15.27:0/124842054 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f780b7680 con 0x7f5f800ce760 2024-01-15T18:07:20.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.952+0000 7f5f76000640 1 -- 172.21.15.27:0/124842054 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f780b77e0 con 0x7f5f800ce760 2024-01-15T18:07:20.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.952+0000 7f5f84e00640 1 -- 172.21.15.27:0/124842054 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f800712f0 con 0x7f5f800ce760 2024-01-15T18:07:20.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.952+0000 7f5f76000640 1 -- 172.21.15.27:0/124842054 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f780b7b90 con 0x7f5f800ce760 2024-01-15T18:07:20.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.952+0000 7f5f84e00640 1 -- 172.21.15.27:0/124842054 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f801f9690 con 0x7f5f800ce760 2024-01-15T18:07:20.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.952+0000 7f5f84e00640 1 -- 172.21.15.27:0/124842054 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f801f9880 con 0x7f5f800ce760 2024-01-15T18:07:20.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.956+0000 7f5f76000640 1 -- 172.21.15.27:0/124842054 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f5f780b7cf0 con 0x7f5f800ce760 2024-01-15T18:07:20.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.956+0000 7f5f76000640 1 --2- 172.21.15.27:0/124842054 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5f58066b30 0x7f5f58068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:20.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.956+0000 7f5f76000640 1 -- 172.21.15.27:0/124842054 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f5f7811f9d0 con 0x7f5f800ce760 2024-01-15T18:07:20.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.956+0000 7f5f7de00640 1 --2- 172.21.15.27:0/124842054 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5f58066b30 0x7f5f58068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:20.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.956+0000 7f5f7de00640 1 --2- 172.21.15.27:0/124842054 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5f58066b30 0x7f5f58068ff0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5f70009b60 tx=0x7f5f70009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:20.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:20.960+0000 7f5f76000640 1 -- 172.21.15.27:0/124842054 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5f780ecb00 con 0x7f5f800ce760 2024-01-15T18:07:21.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:21.156+0000 7f5f84e00640 1 -- 172.21.15.27:0/124842054 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f5f8004e850 con 0x7f5f800ce760 2024-01-15T18:07:22.046 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-01-15T18:07:22.040+0000 7f6714c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-15T18:07:22.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.084+0000 7f5f76000640 1 -- 172.21.15.27:0/124842054 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v26) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5f7800a560 con 0x7f5f800ce760 2024-01-15T18:07:22.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.088+0000 7f5f84e00640 1 -- 172.21.15.27:0/124842054 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5f58066b30 msgr2=0x7f5f58068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:22.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.088+0000 7f5f84e00640 1 --2- 172.21.15.27:0/124842054 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5f58066b30 0x7f5f58068ff0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5f70009b60 tx=0x7f5f70009190 comp rx=0 tx=0).stop 2024-01-15T18:07:22.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.088+0000 7f5f84e00640 1 -- 172.21.15.27:0/124842054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f800ce760 msgr2=0x7f5f801f9bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:22.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.088+0000 7f5f84e00640 1 --2- 172.21.15.27:0/124842054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f800ce760 0x7f5f801f9bb0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f5f7801b5f0 tx=0x7f5f7801a760 comp rx=0 tx=0).stop 2024-01-15T18:07:22.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.088+0000 7f5f84e00640 1 -- 172.21.15.27:0/124842054 shutdown_connections 2024-01-15T18:07:22.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.088+0000 7f5f84e00640 1 --2- 172.21.15.27:0/124842054 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5f58066b30 0x7f5f58068ff0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:22.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.088+0000 7f5f84e00640 1 --2- 172.21.15.27:0/124842054 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f800ce760 0x7f5f801f9bb0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:22.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.088+0000 7f5f84e00640 1 --2- 172.21.15.27:0/124842054 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5f800b1db0 0x7f5f801421e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:22.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.088+0000 7f5f84e00640 1 --2- 172.21.15.27:0/124842054 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5f8006c430 0x7f5f8015c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:22.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.088+0000 7f5f84e00640 1 -- 172.21.15.27:0/124842054 >> 172.21.15.27:0/124842054 conn(0x7f5f800cb700 msgr2=0x7f5f800b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:22.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.088+0000 7f5f84e00640 1 -- 172.21.15.27:0/124842054 shutdown_connections 2024-01-15T18:07:22.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.088+0000 7f5f84e00640 1 -- 172.21.15.27:0/124842054 wait complete. 2024-01-15T18:07:22.120 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-15T18:07:22.120 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-15T18:07:22.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.260+0000 7f47f7800640 1 Processor -- start 2024-01-15T18:07:22.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.260+0000 7f47f7800640 1 -- start start 2024-01-15T18:07:22.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.264+0000 7f47f7800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f47f015b4b0 0x7f47f00b1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:22.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.264+0000 7f47f7800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f47f00cec60 0x7f47f00b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:22.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.264+0000 7f47f7800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f00d19a0 con 0x7f47f00cf460 2024-01-15T18:07:22.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.264+0000 7f47f7800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47f00d1c80 con 0x7f47f00cec60 2024-01-15T18:07:22.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.264+0000 7f47f7800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f00b07c0 con 0x7f47f015b4b0 2024-01-15T18:07:22.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.264+0000 7f47f5800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f47f00cec60 0x7f47f00b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:22.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.264+0000 7f47f5800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f47f00cec60 0x7f47f00b1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33122/0 (socket says 172.21.15.27:33122) 2024-01-15T18:07:22.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.264+0000 7f47f5800640 1 -- 172.21.15.27:0/3748041942 learned_addr learned my addr 172.21.15.27:0/3748041942 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:22.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.264+0000 7f47efe00640 1 --2- 172.21.15.27:0/3748041942 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f47f015b4b0 0x7f47f00b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:22.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.264+0000 7f47f5800640 1 -- 172.21.15.27:0/3748041942 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f47f015b4b0 msgr2=0x7f47f00b1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:22.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.264+0000 7f47f5800640 1 --2- 172.21.15.27:0/3748041942 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f47f015b4b0 0x7f47f00b1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:22.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.264+0000 7f47f5800640 1 -- 172.21.15.27:0/3748041942 >> v1:172.21.15.93:6789/0 conn(0x7f47f00cf460 legacy=0x7f47f00cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:22.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.264+0000 7f47f5800640 1 -- 172.21.15.27:0/3748041942 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f00b0aa0 con 0x7f47f00cec60 2024-01-15T18:07:22.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.268+0000 7f47f5800640 1 --2- 172.21.15.27:0/3748041942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47f00cec60 0x7f47f00b1ac0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f47e8009e50 tx=0x7f47e801ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daf9d93c26663218 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:22.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.268+0000 7f47ef400640 1 -- 172.21.15.27:0/3748041942 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f00b2810 con 0x7f47f00cec60 2024-01-15T18:07:22.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.268+0000 7f47ef400640 1 -- 172.21.15.27:0/3748041942 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e80a58e0 con 0x7f47f00cec60 2024-01-15T18:07:22.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.268+0000 7f47ef400640 1 -- 172.21.15.27:0/3748041942 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e80a5c70 con 0x7f47f00cec60 2024-01-15T18:07:22.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.268+0000 7f47f7800640 1 -- 172.21.15.27:0/3748041942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47f00cec60 msgr2=0x7f47f00b1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:22.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.268+0000 7f47f7800640 1 --2- 172.21.15.27:0/3748041942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47f00cec60 0x7f47f00b1ac0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f47e8009e50 tx=0x7f47e801ca10 comp rx=0 tx=0).stop 2024-01-15T18:07:22.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.268+0000 7f47f7800640 1 -- 172.21.15.27:0/3748041942 shutdown_connections 2024-01-15T18:07:22.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.268+0000 7f47f7800640 1 --2- 172.21.15.27:0/3748041942 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47f00cec60 0x7f47f00b1ac0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:22.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.268+0000 7f47f7800640 1 --2- 172.21.15.27:0/3748041942 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f47f015b4b0 0x7f47f00b1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:22.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.268+0000 7f47f7800640 1 -- 172.21.15.27:0/3748041942 >> 172.21.15.27:0/3748041942 conn(0x7f47f00d0e80 msgr2=0x7f47f00d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:22.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.268+0000 7f47f7800640 1 -- 172.21.15.27:0/3748041942 shutdown_connections 2024-01-15T18:07:22.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.268+0000 7f47f7800640 1 -- 172.21.15.27:0/3748041942 wait complete. 2024-01-15T18:07:22.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.268+0000 7f47f7800640 1 Processor -- start 2024-01-15T18:07:22.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.268+0000 7f47f7800640 1 -- start start 2024-01-15T18:07:22.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.272+0000 7f47f7800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f47f00cec60 0x7f47f00c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:22.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.272+0000 7f47f7800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f47f00cf460 0x7f47f014b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:22.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.272+0000 7f47f7800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47f015b4b0 0x7f47f014b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:22.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.272+0000 7f47f7800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f00c52a0 con 0x7f47f015b4b0 2024-01-15T18:07:22.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.272+0000 7f47f7800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f014bbd0 con 0x7f47f00cec60 2024-01-15T18:07:22.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.272+0000 7f47f7800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f014d4a0 con 0x7f47f00cf460 2024-01-15T18:07:22.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.272+0000 7f47f5800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47f015b4b0 0x7f47f014b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:22.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.272+0000 7f47f5800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47f015b4b0 0x7f47f014b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33132/0 (socket says 172.21.15.27:33132) 2024-01-15T18:07:22.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.272+0000 7f47f5800640 1 -- 172.21.15.27:0/4023846407 learned_addr learned my addr 172.21.15.27:0/4023846407 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:22.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.272+0000 7f47efe00640 1 --2- 172.21.15.27:0/4023846407 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f47f00cf460 0x7f47f014b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:22.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.272+0000 7f47f4e00640 1 --2- 172.21.15.27:0/4023846407 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f47f00cec60 0x7f47f00c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:22.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.272+0000 7f47f5800640 1 -- 172.21.15.27:0/4023846407 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f47f00cf460 msgr2=0x7f47f014b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:22.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.272+0000 7f47f5800640 1 --2- 172.21.15.27:0/4023846407 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f47f00cf460 0x7f47f014b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:22.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.272+0000 7f47f5800640 1 -- 172.21.15.27:0/4023846407 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f47f00cec60 msgr2=0x7f47f00c81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:22.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.272+0000 7f47f5800640 1 --2- 172.21.15.27:0/4023846407 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f47f00cec60 0x7f47f00c81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:22.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.272+0000 7f47f5800640 1 -- 172.21.15.27:0/4023846407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e8009b30 con 0x7f47f015b4b0 2024-01-15T18:07:22.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.272+0000 7f47f5800640 1 --2- 172.21.15.27:0/4023846407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47f015b4b0 0x7f47f014b690 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f47e80af040 tx=0x7f47e80197b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:22.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.272+0000 7f47ecc00640 1 -- 172.21.15.27:0/4023846407 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e80b8670 con 0x7f47f015b4b0 2024-01-15T18:07:22.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.272+0000 7f47ecc00640 1 -- 172.21.15.27:0/4023846407 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e80b87d0 con 0x7f47f015b4b0 2024-01-15T18:07:22.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.272+0000 7f47f7800640 1 -- 172.21.15.27:0/4023846407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f014d780 con 0x7f47f015b4b0 2024-01-15T18:07:22.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.272+0000 7f47ecc00640 1 -- 172.21.15.27:0/4023846407 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e80b8b80 con 0x7f47f015b4b0 2024-01-15T18:07:22.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.272+0000 7f47f7800640 1 -- 172.21.15.27:0/4023846407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f00c4ec0 con 0x7f47f015b4b0 2024-01-15T18:07:22.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.272+0000 7f47f7800640 1 -- 172.21.15.27:0/4023846407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f00c53e0 con 0x7f47f015b4b0 2024-01-15T18:07:22.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.272+0000 7f47ecc00640 1 -- 172.21.15.27:0/4023846407 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f47e80b8ce0 con 0x7f47f015b4b0 2024-01-15T18:07:22.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.276+0000 7f47ecc00640 1 --2- 172.21.15.27:0/4023846407 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f47c8066b30 0x7f47c8068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:22.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.276+0000 7f47ecc00640 1 -- 172.21.15.27:0/4023846407 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f47e811fe00 con 0x7f47f015b4b0 2024-01-15T18:07:22.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.276+0000 7f47f4e00640 1 --2- 172.21.15.27:0/4023846407 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f47c8066b30 0x7f47c8068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:22.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.276+0000 7f47f4e00640 1 --2- 172.21.15.27:0/4023846407 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f47c8066b30 0x7f47c8068ff0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f47e0007fd0 tx=0x7f47e0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:22.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.280+0000 7f47ecc00640 1 -- 172.21.15.27:0/4023846407 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f47f00c53e0 con 0x7f47f015b4b0 2024-01-15T18:07:22.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:22.472+0000 7f47f7800640 1 -- 172.21.15.27:0/4023846407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f47f00c53e0 con 0x7f47f015b4b0 2024-01-15T18:07:23.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.096+0000 7f47ecc00640 1 -- 172.21.15.27:0/4023846407 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v27) v1 ==== 122+0+0 (secure 0 0 0) 0x7f47f00c53e0 con 0x7f47f015b4b0 2024-01-15T18:07:23.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.096+0000 7f47f7800640 1 -- 172.21.15.27:0/4023846407 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f47c8066b30 msgr2=0x7f47c8068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:23.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.096+0000 7f47f7800640 1 --2- 172.21.15.27:0/4023846407 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f47c8066b30 0x7f47c8068ff0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f47e0007fd0 tx=0x7f47e0009040 comp rx=0 tx=0).stop 2024-01-15T18:07:23.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.096+0000 7f47f7800640 1 -- 172.21.15.27:0/4023846407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47f015b4b0 msgr2=0x7f47f014b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:23.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.096+0000 7f47f7800640 1 --2- 172.21.15.27:0/4023846407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47f015b4b0 0x7f47f014b690 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f47e80af040 tx=0x7f47e80197b0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.096+0000 7f47f7800640 1 -- 172.21.15.27:0/4023846407 shutdown_connections 2024-01-15T18:07:23.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.096+0000 7f47f7800640 1 --2- 172.21.15.27:0/4023846407 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f47c8066b30 0x7f47c8068ff0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.096+0000 7f47f7800640 1 --2- 172.21.15.27:0/4023846407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47f015b4b0 0x7f47f014b690 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.096+0000 7f47f7800640 1 --2- 172.21.15.27:0/4023846407 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f47f00cf460 0x7f47f014b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.096+0000 7f47f7800640 1 --2- 172.21.15.27:0/4023846407 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f47f00cec60 0x7f47f00c81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.096+0000 7f47f7800640 1 -- 172.21.15.27:0/4023846407 >> 172.21.15.27:0/4023846407 conn(0x7f47f00d0e80 msgr2=0x7f47f00ada70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:23.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.096+0000 7f47f7800640 1 -- 172.21.15.27:0/4023846407 shutdown_connections 2024-01-15T18:07:23.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.096+0000 7f47f7800640 1 -- 172.21.15.27:0/4023846407 wait complete. 2024-01-15T18:07:23.123 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-15T18:07:23.123 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:07:23.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.260+0000 7f1d59000640 1 Processor -- start 2024-01-15T18:07:23.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.260+0000 7f1d59000640 1 -- start start 2024-01-15T18:07:23.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.260+0000 7f1d59000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f1d54071e40 0x7f1d540c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:23.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.260+0000 7f1d59000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1d5406d430 0x7f1d540cd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:23.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.260+0000 7f1d59000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d54159a50 con 0x7f1d54159bf0 2024-01-15T18:07:23.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.260+0000 7f1d59000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d540cce00 con 0x7f1d5406d430 2024-01-15T18:07:23.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.260+0000 7f1d59000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d5406d980 con 0x7f1d54071e40 2024-01-15T18:07:23.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.264+0000 7f1d52800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1d5406d430 0x7f1d540cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:23.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.264+0000 7f1d52800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1d5406d430 0x7f1d540cd250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33138/0 (socket says 172.21.15.27:33138) 2024-01-15T18:07:23.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.264+0000 7f1d52800640 1 -- 172.21.15.27:0/1434355768 learned_addr learned my addr 172.21.15.27:0/1434355768 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:23.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.264+0000 7f1d51400640 1 --2- 172.21.15.27:0/1434355768 >> v2:172.21.15.93:3300/0 conn(0x7f1d54071e40 0x7f1d540c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:23.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.264+0000 7f1d52800640 1 -- 172.21.15.27:0/1434355768 >> v2:172.21.15.93:3300/0 conn(0x7f1d54071e40 msgr2=0x7f1d540c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:23.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.264+0000 7f1d52800640 1 --2- 172.21.15.27:0/1434355768 >> v2:172.21.15.93:3300/0 conn(0x7f1d54071e40 0x7f1d540c9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.264+0000 7f1d52800640 1 -- 172.21.15.27:0/1434355768 >> v1:172.21.15.93:6789/0 conn(0x7f1d54159bf0 legacy=0x7f1d54159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:23.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.264+0000 7f1d52800640 1 -- 172.21.15.27:0/1434355768 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d5406dc60 con 0x7f1d5406d430 2024-01-15T18:07:23.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.264+0000 7f1d52800640 1 --2- 172.21.15.27:0/1434355768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d5406d430 0x7f1d540cd250 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f1d540cfb10 tx=0x7f1d4c01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5a4c4ddd26149af server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:23.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.264+0000 7f1d50a00640 1 -- 172.21.15.27:0/1434355768 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4c0af400 con 0x7f1d5406d430 2024-01-15T18:07:23.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.264+0000 7f1d50a00640 1 -- 172.21.15.27:0/1434355768 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d4c0ab050 con 0x7f1d5406d430 2024-01-15T18:07:23.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.264+0000 7f1d50a00640 1 -- 172.21.15.27:0/1434355768 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4c0af780 con 0x7f1d5406d430 2024-01-15T18:07:23.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.268+0000 7f1d59000640 1 -- 172.21.15.27:0/1434355768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d5406d430 msgr2=0x7f1d540cd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:23.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.268+0000 7f1d59000640 1 --2- 172.21.15.27:0/1434355768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d5406d430 0x7f1d540cd250 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f1d540cfb10 tx=0x7f1d4c01f1c0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.268+0000 7f1d59000640 1 -- 172.21.15.27:0/1434355768 shutdown_connections 2024-01-15T18:07:23.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.268+0000 7f1d59000640 1 --2- 172.21.15.27:0/1434355768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d5406d430 0x7f1d540cd250 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.268+0000 7f1d59000640 1 --2- 172.21.15.27:0/1434355768 >> v2:172.21.15.93:3300/0 conn(0x7f1d54071e40 0x7f1d540c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.268+0000 7f1d59000640 1 -- 172.21.15.27:0/1434355768 >> 172.21.15.27:0/1434355768 conn(0x7f1d540cec60 msgr2=0x7f1d540cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:23.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.268+0000 7f1d59000640 1 -- 172.21.15.27:0/1434355768 shutdown_connections 2024-01-15T18:07:23.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.268+0000 7f1d59000640 1 -- 172.21.15.27:0/1434355768 wait complete. 2024-01-15T18:07:23.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.268+0000 7f1d59000640 1 Processor -- start 2024-01-15T18:07:23.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.268+0000 7f1d59000640 1 -- start start 2024-01-15T18:07:23.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.268+0000 7f1d59000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1d5406d430 0x7f1d540c1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:23.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.268+0000 7f1d59000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d54159bf0 0x7f1d540c3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:23.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.268+0000 7f1d59000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1d5415c740 0x7f1d540700a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:23.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.268+0000 7f1d59000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d540c19c0 con 0x7f1d54159bf0 2024-01-15T18:07:23.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.268+0000 7f1d59000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5415cbd0 con 0x7f1d5415c740 2024-01-15T18:07:23.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.268+0000 7f1d59000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d540c62a0 con 0x7f1d5406d430 2024-01-15T18:07:23.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.272+0000 7f1d51400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d54159bf0 0x7f1d540c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:23.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.272+0000 7f1d51e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1d5406d430 0x7f1d540c1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:23.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.272+0000 7f1d51400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d54159bf0 0x7f1d540c3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33146/0 (socket says 172.21.15.27:33146) 2024-01-15T18:07:23.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.272+0000 7f1d51400640 1 -- 172.21.15.27:0/4118950607 learned_addr learned my addr 172.21.15.27:0/4118950607 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:23.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.272+0000 7f1d52800640 1 --2- 172.21.15.27:0/4118950607 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1d5415c740 0x7f1d540700a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:23.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.272+0000 7f1d51400640 1 -- 172.21.15.27:0/4118950607 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1d5406d430 msgr2=0x7f1d540c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:23.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.272+0000 7f1d51400640 1 --2- 172.21.15.27:0/4118950607 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1d5406d430 0x7f1d540c1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.272+0000 7f1d51400640 1 -- 172.21.15.27:0/4118950607 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1d5415c740 msgr2=0x7f1d540700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:23.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.272+0000 7f1d51400640 1 --2- 172.21.15.27:0/4118950607 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1d5415c740 0x7f1d540700a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.272+0000 7f1d51400640 1 -- 172.21.15.27:0/4118950607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d4c009b30 con 0x7f1d54159bf0 2024-01-15T18:07:23.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.272+0000 7f1d51400640 1 --2- 172.21.15.27:0/4118950607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d54159bf0 0x7f1d540c3660 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c00e930 tx=0x7f1d3c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:23.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.272+0000 7f1d4a000640 1 -- 172.21.15.27:0/4118950607 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c009690 con 0x7f1d54159bf0 2024-01-15T18:07:23.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.272+0000 7f1d4a000640 1 -- 172.21.15.27:0/4118950607 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d3c00ca00 con 0x7f1d54159bf0 2024-01-15T18:07:23.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.272+0000 7f1d59000640 1 -- 172.21.15.27:0/4118950607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d540c6580 con 0x7f1d54159bf0 2024-01-15T18:07:23.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.272+0000 7f1d4a000640 1 -- 172.21.15.27:0/4118950607 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c00cd70 con 0x7f1d54159bf0 2024-01-15T18:07:23.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.272+0000 7f1d59000640 1 -- 172.21.15.27:0/4118950607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d540c6b40 con 0x7f1d54159bf0 2024-01-15T18:07:23.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.272+0000 7f1d59000640 1 -- 172.21.15.27:0/4118950607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d5404e850 con 0x7f1d54159bf0 2024-01-15T18:07:23.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.272+0000 7f1d4a000640 1 -- 172.21.15.27:0/4118950607 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f1d3c013470 con 0x7f1d54159bf0 2024-01-15T18:07:23.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.276+0000 7f1d4a000640 1 --2- 172.21.15.27:0/4118950607 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1d2c066b30 0x7f1d2c068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:23.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.276+0000 7f1d4a000640 1 -- 172.21.15.27:0/4118950607 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f1d3c084bf0 con 0x7f1d54159bf0 2024-01-15T18:07:23.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.276+0000 7f1d51e00640 1 --2- 172.21.15.27:0/4118950607 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1d2c066b30 0x7f1d2c068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:23.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.276+0000 7f1d51e00640 1 --2- 172.21.15.27:0/4118950607 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1d2c066b30 0x7f1d2c068ff0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1d44007fd0 tx=0x7f1d44009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:23.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.280+0000 7f1d4a000640 1 -- 172.21.15.27:0/4118950607 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1d3c051b00 con 0x7f1d54159bf0 2024-01-15T18:07:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.468+0000 7f1d59000640 1 -- 172.21.15.27:0/4118950607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d540c5fe0 con 0x7f1d54159bf0 2024-01-15T18:07:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.468+0000 7f1d4a000640 1 -- 172.21.15.27:0/4118950607 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4910 (secure 0 0 0) 0x7f1d3c055120 con 0x7f1d54159bf0 2024-01-15T18:07:23.474 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:07:23.474 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5306,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/1548117999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":1548117999},{"type":"v1","addr":"172.21.15.93:6836","nonce":1548117999}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5719,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/260141697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":260141697},{"type":"v1","addr":"172.21.15.93:6837","nonce":260141697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5802,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/2030664596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2030664596},{"type":"v1","addr":"172.21.15.27:6834","nonce":2030664596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:07:19.460378+0000","modified":"2024-01-15T18:07:22.477523+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5808},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5808":{"gid":5808,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/1005016790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1005016790},{"type":"v1","addr":"172.21.15.27:6835","nonce":1005016790}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-01-15T18:07:23.474 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 27 2024-01-15T18:07:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.472+0000 7f1d59000640 1 -- 172.21.15.27:0/4118950607 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1d2c066b30 msgr2=0x7f1d2c068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.472+0000 7f1d59000640 1 --2- 172.21.15.27:0/4118950607 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1d2c066b30 0x7f1d2c068ff0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1d44007fd0 tx=0x7f1d44009040 comp rx=0 tx=0).stop 2024-01-15T18:07:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.472+0000 7f1d59000640 1 -- 172.21.15.27:0/4118950607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d54159bf0 msgr2=0x7f1d540c3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.472+0000 7f1d59000640 1 --2- 172.21.15.27:0/4118950607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d54159bf0 0x7f1d540c3660 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c00e930 tx=0x7f1d3c00edf0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.472+0000 7f1d59000640 1 -- 172.21.15.27:0/4118950607 shutdown_connections 2024-01-15T18:07:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.472+0000 7f1d59000640 1 --2- 172.21.15.27:0/4118950607 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1d2c066b30 0x7f1d2c068ff0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.472+0000 7f1d59000640 1 --2- 172.21.15.27:0/4118950607 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1d5415c740 0x7f1d540700a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.472+0000 7f1d59000640 1 --2- 172.21.15.27:0/4118950607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d54159bf0 0x7f1d540c3660 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.472+0000 7f1d59000640 1 --2- 172.21.15.27:0/4118950607 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1d5406d430 0x7f1d540c1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.472+0000 7f1d59000640 1 -- 172.21.15.27:0/4118950607 >> 172.21.15.27:0/4118950607 conn(0x7f1d540cec60 msgr2=0x7f1d540ce050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.472+0000 7f1d59000640 1 -- 172.21.15.27:0/4118950607 shutdown_connections 2024-01-15T18:07:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.472+0000 7f1d59000640 1 -- 172.21.15.27:0/4118950607 wait complete. 2024-01-15T18:07:23.496 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:07:23.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.632+0000 7efe62600640 1 Processor -- start 2024-01-15T18:07:23.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.632+0000 7efe62600640 1 -- start start 2024-01-15T18:07:23.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.632+0000 7efe62600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7efe5c0c5d90 0x7efe5c0cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:23.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.632+0000 7efe62600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe5c0ceb20 0x7efe5c0cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:23.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.632+0000 7efe62600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5c0d1a70 con 0x7efe5c15e7d0 2024-01-15T18:07:23.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.632+0000 7efe62600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe5c0ccdd0 con 0x7efe5c0c5d90 2024-01-15T18:07:23.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.632+0000 7efe62600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe5c15ebb0 con 0x7efe5c0ceb20 2024-01-15T18:07:23.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.636+0000 7efe5b400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7efe5c15e7d0 0x7efe5c0cd730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45182/0 (socket says 172.21.15.27:45182) 2024-01-15T18:07:23.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.636+0000 7efe5b400640 1 -- 172.21.15.27:0/3749475356 learned_addr learned my addr 172.21.15.27:0/3749475356 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:23.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.636+0000 7efe5aa00640 1 --2- 172.21.15.27:0/3749475356 >> v2:172.21.15.93:3300/0 conn(0x7efe5c0c5d90 0x7efe5c0cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:23.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.636+0000 7efe5be00640 1 --2- 172.21.15.27:0/3749475356 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe5c0ceb20 0x7efe5c0cefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:23.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.636+0000 7efe5a000640 1 -- 172.21.15.27:0/3749475356 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204669561 0 0) 0x7efe5c0d1a70 con 0x7efe5c15e7d0 2024-01-15T18:07:23.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.636+0000 7efe5a000640 1 -- 172.21.15.27:0/3749475356 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe3c003530 con 0x7efe5c15e7d0 2024-01-15T18:07:23.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.636+0000 7efe5aa00640 1 -- 172.21.15.27:0/3749475356 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe5c0ceb20 msgr2=0x7efe5c0cefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:23.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.636+0000 7efe5aa00640 1 --2- 172.21.15.27:0/3749475356 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe5c0ceb20 0x7efe5c0cefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.636+0000 7efe5aa00640 1 -- 172.21.15.27:0/3749475356 >> v1:172.21.15.27:6789/0 conn(0x7efe5c15e7d0 legacy=0x7efe5c0cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:23.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.636+0000 7efe5aa00640 1 -- 172.21.15.27:0/3749475356 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe5c0d1cc0 con 0x7efe5c0c5d90 2024-01-15T18:07:23.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.636+0000 7efe5aa00640 1 --2- 172.21.15.27:0/3749475356 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efe5c0c5d90 0x7efe5c0cde40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7efe440097c0 tx=0x7efe4401c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da06a135ea6cabbd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:23.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.640+0000 7efe5a000640 1 -- 172.21.15.27:0/3749475356 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe440ab650 con 0x7efe5c0c5d90 2024-01-15T18:07:23.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.640+0000 7efe5a000640 1 -- 172.21.15.27:0/3749475356 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe440ab7b0 con 0x7efe5c0c5d90 2024-01-15T18:07:23.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.640+0000 7efe5a000640 1 -- 172.21.15.27:0/3749475356 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe440abb40 con 0x7efe5c0c5d90 2024-01-15T18:07:23.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.640+0000 7efe62600640 1 -- 172.21.15.27:0/3749475356 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efe5c0c5d90 msgr2=0x7efe5c0cde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:23.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.640+0000 7efe62600640 1 --2- 172.21.15.27:0/3749475356 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efe5c0c5d90 0x7efe5c0cde40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7efe440097c0 tx=0x7efe4401c450 comp rx=0 tx=0).stop 2024-01-15T18:07:23.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.640+0000 7efe62600640 1 -- 172.21.15.27:0/3749475356 shutdown_connections 2024-01-15T18:07:23.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.640+0000 7efe62600640 1 --2- 172.21.15.27:0/3749475356 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe5c0ceb20 0x7efe5c0cefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.640+0000 7efe62600640 1 --2- 172.21.15.27:0/3749475356 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efe5c0c5d90 0x7efe5c0cde40 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.640+0000 7efe62600640 1 -- 172.21.15.27:0/3749475356 >> 172.21.15.27:0/3749475356 conn(0x7efe5c0d0f50 msgr2=0x7efe5c0d1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:23.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.640+0000 7efe62600640 1 -- 172.21.15.27:0/3749475356 shutdown_connections 2024-01-15T18:07:23.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.640+0000 7efe62600640 1 -- 172.21.15.27:0/3749475356 wait complete. 2024-01-15T18:07:23.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.640+0000 7efe62600640 1 Processor -- start 2024-01-15T18:07:23.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.640+0000 7efe62600640 1 -- start start 2024-01-15T18:07:23.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.644+0000 7efe62600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efe5c0c5d90 0x7efe5c09ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:23.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.644+0000 7efe62600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe5c0ceb20 0x7efe5c15c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:23.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.644+0000 7efe62600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe5c15e7d0 0x7efe5c09e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:23.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.644+0000 7efe62600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe5c15c600 con 0x7efe5c15e7d0 2024-01-15T18:07:23.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.644+0000 7efe62600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe5c09c740 con 0x7efe5c0c5d90 2024-01-15T18:07:23.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.644+0000 7efe62600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe5c157430 con 0x7efe5c0ceb20 2024-01-15T18:07:23.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.644+0000 7efe5be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe5c15e7d0 0x7efe5c09e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:23.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.644+0000 7efe5be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe5c15e7d0 0x7efe5c09e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50672/0 (socket says 172.21.15.27:50672) 2024-01-15T18:07:23.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.644+0000 7efe5be00640 1 -- 172.21.15.27:0/3763114649 learned_addr learned my addr 172.21.15.27:0/3763114649 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:23.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.644+0000 7efe5b400640 1 --2- 172.21.15.27:0/3763114649 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efe5c0c5d90 0x7efe5c09ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:23.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.644+0000 7efe5aa00640 1 --2- 172.21.15.27:0/3763114649 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe5c0ceb20 0x7efe5c15c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:23.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.644+0000 7efe5be00640 1 -- 172.21.15.27:0/3763114649 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe5c0ceb20 msgr2=0x7efe5c15c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:23.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.644+0000 7efe5be00640 1 --2- 172.21.15.27:0/3763114649 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe5c0ceb20 0x7efe5c15c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.644+0000 7efe5be00640 1 -- 172.21.15.27:0/3763114649 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efe5c0c5d90 msgr2=0x7efe5c09ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:23.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.644+0000 7efe5be00640 1 --2- 172.21.15.27:0/3763114649 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efe5c0c5d90 0x7efe5c09ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.644+0000 7efe5be00640 1 -- 172.21.15.27:0/3763114649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe44009420 con 0x7efe5c15e7d0 2024-01-15T18:07:23.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.644+0000 7efe5be00640 1 --2- 172.21.15.27:0/3763114649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe5c15e7d0 0x7efe5c09e650 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7efe5400efc0 tx=0x7efe5400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:23.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.644+0000 7efe53400640 1 -- 172.21.15.27:0/3763114649 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe54009ba0 con 0x7efe5c15e7d0 2024-01-15T18:07:23.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.644+0000 7efe53400640 1 -- 172.21.15.27:0/3763114649 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe5400f040 con 0x7efe5c15e7d0 2024-01-15T18:07:23.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.644+0000 7efe62600640 1 -- 172.21.15.27:0/3763114649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe5c09c970 con 0x7efe5c15e7d0 2024-01-15T18:07:23.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.644+0000 7efe53400640 1 -- 172.21.15.27:0/3763114649 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe540146c0 con 0x7efe5c15e7d0 2024-01-15T18:07:23.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.644+0000 7efe62600640 1 -- 172.21.15.27:0/3763114649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe5c156e70 con 0x7efe5c15e7d0 2024-01-15T18:07:23.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.644+0000 7efe62600640 1 -- 172.21.15.27:0/3763114649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe5c04e850 con 0x7efe5c15e7d0 2024-01-15T18:07:23.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.644+0000 7efe53400640 1 -- 172.21.15.27:0/3763114649 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7efe54014820 con 0x7efe5c15e7d0 2024-01-15T18:07:23.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.648+0000 7efe53400640 1 --2- 172.21.15.27:0/3763114649 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efe34066b30 0x7efe34068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:23.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.648+0000 7efe53400640 1 -- 172.21.15.27:0/3763114649 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5737+0+0 (secure 0 0 0) 0x7efe54085300 con 0x7efe5c15e7d0 2024-01-15T18:07:23.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.648+0000 7efe5b400640 1 --2- 172.21.15.27:0/3763114649 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efe34066b30 0x7efe34068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:23.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.648+0000 7efe5b400640 1 --2- 172.21.15.27:0/3763114649 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efe34066b30 0x7efe34068ff0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7efe4c0011f0 tx=0x7efe4c007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:23.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.652+0000 7efe53400640 1 -- 172.21.15.27:0/3763114649 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7efe54052170 con 0x7efe5c15e7d0 2024-01-15T18:07:23.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.808+0000 7efe62600640 1 -- 172.21.15.27:0/3763114649 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe5c09cb00 con 0x7efe5c15e7d0 2024-01-15T18:07:23.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.808+0000 7efe53400640 1 -- 172.21.15.27:0/3763114649 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16460 (secure 0 0 0) 0x7efe54055790 con 0x7efe5c15e7d0 2024-01-15T18:07:23.813 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:07:23.813 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":45,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:07:21.073706+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:07:17.862115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-15T18:07:18.453645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:07:23.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.812+0000 7efe62600640 1 -- 172.21.15.27:0/3763114649 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efe34066b30 msgr2=0x7efe34068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:23.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.812+0000 7efe62600640 1 --2- 172.21.15.27:0/3763114649 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efe34066b30 0x7efe34068ff0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7efe4c0011f0 tx=0x7efe4c007d40 comp rx=0 tx=0).stop 2024-01-15T18:07:23.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.812+0000 7efe62600640 1 -- 172.21.15.27:0/3763114649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe5c15e7d0 msgr2=0x7efe5c09e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:23.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.812+0000 7efe62600640 1 --2- 172.21.15.27:0/3763114649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe5c15e7d0 0x7efe5c09e650 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7efe5400efc0 tx=0x7efe5400c490 comp rx=0 tx=0).stop 2024-01-15T18:07:23.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.812+0000 7efe62600640 1 -- 172.21.15.27:0/3763114649 shutdown_connections 2024-01-15T18:07:23.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.812+0000 7efe62600640 1 --2- 172.21.15.27:0/3763114649 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efe34066b30 0x7efe34068ff0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.812+0000 7efe62600640 1 --2- 172.21.15.27:0/3763114649 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe5c15e7d0 0x7efe5c09e650 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.812+0000 7efe62600640 1 --2- 172.21.15.27:0/3763114649 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe5c0ceb20 0x7efe5c15c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.812+0000 7efe62600640 1 --2- 172.21.15.27:0/3763114649 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efe5c0c5d90 0x7efe5c09ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.812+0000 7efe62600640 1 -- 172.21.15.27:0/3763114649 >> 172.21.15.27:0/3763114649 conn(0x7efe5c0d0f50 msgr2=0x7efe5c0ca030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:23.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.812+0000 7efe62600640 1 -- 172.21.15.27:0/3763114649 shutdown_connections 2024-01-15T18:07:23.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.812+0000 7efe62600640 1 -- 172.21.15.27:0/3763114649 wait complete. 2024-01-15T18:07:23.836 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-15T18:07:23.836 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:07:23.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.972+0000 7fdbda000640 1 Processor -- start 2024-01-15T18:07:23.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.972+0000 7fdbda000640 1 -- start start 2024-01-15T18:07:23.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.972+0000 7fdbda000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdbd415d0a0 0x7fdbd406c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:23.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.972+0000 7fdbda000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fdbd40c94c0 0x7fdbd40c9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:23.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.972+0000 7fdbda000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd40c9e50 con 0x7fdbd40ce710 2024-01-15T18:07:23.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.972+0000 7fdbda000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbd40ad180 con 0x7fdbd415d0a0 2024-01-15T18:07:23.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.972+0000 7fdbda000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbd40ac890 con 0x7fdbd40c94c0 2024-01-15T18:07:23.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.976+0000 7fdbd3400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdbd415d0a0 0x7fdbd406c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:23.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.976+0000 7fdbd3400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdbd415d0a0 0x7fdbd406c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50684/0 (socket says 172.21.15.27:50684) 2024-01-15T18:07:23.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.976+0000 7fdbd3400640 1 -- 172.21.15.27:0/2159399869 learned_addr learned my addr 172.21.15.27:0/2159399869 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:23.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.976+0000 7fdbd3e00640 1 --2- 172.21.15.27:0/2159399869 >> v2:172.21.15.93:3300/0 conn(0x7fdbd40c94c0 0x7fdbd40c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:23.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.976+0000 7fdbd3400640 1 -- 172.21.15.27:0/2159399869 >> v2:172.21.15.93:3300/0 conn(0x7fdbd40c94c0 msgr2=0x7fdbd40c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:23.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.976+0000 7fdbd3400640 1 --2- 172.21.15.27:0/2159399869 >> v2:172.21.15.93:3300/0 conn(0x7fdbd40c94c0 0x7fdbd40c9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.976+0000 7fdbd3400640 1 -- 172.21.15.27:0/2159399869 >> v1:172.21.15.93:6789/0 conn(0x7fdbd40ce710 legacy=0x7fdbd40cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:23.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.976+0000 7fdbd3400640 1 -- 172.21.15.27:0/2159399869 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd40cda90 con 0x7fdbd415d0a0 2024-01-15T18:07:23.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.976+0000 7fdbd3e00640 1 --2- 172.21.15.27:0/2159399869 >> v2:172.21.15.93:3300/0 conn(0x7fdbd40c94c0 0x7fdbd40c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:07:23.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.976+0000 7fdbd3400640 1 --2- 172.21.15.27:0/2159399869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbd415d0a0 0x7fdbd406c1e0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fdbd40cdf40 tx=0x7fdbc401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fa6e7b4eed1f19f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:23.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.976+0000 7fdbd2000640 1 -- 172.21.15.27:0/2159399869 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc40aa430 con 0x7fdbd415d0a0 2024-01-15T18:07:23.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.976+0000 7fdbd2000640 1 -- 172.21.15.27:0/2159399869 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc40aa590 con 0x7fdbd415d0a0 2024-01-15T18:07:23.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.976+0000 7fdbd2000640 1 -- 172.21.15.27:0/2159399869 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc40aa920 con 0x7fdbd415d0a0 2024-01-15T18:07:23.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.980+0000 7fdbda000640 1 -- 172.21.15.27:0/2159399869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbd415d0a0 msgr2=0x7fdbd406c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:23.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.980+0000 7fdbda000640 1 --2- 172.21.15.27:0/2159399869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbd415d0a0 0x7fdbd406c1e0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fdbd40cdf40 tx=0x7fdbc401c4d0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.980+0000 7fdbda000640 1 -- 172.21.15.27:0/2159399869 shutdown_connections 2024-01-15T18:07:23.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.980+0000 7fdbda000640 1 --2- 172.21.15.27:0/2159399869 >> v2:172.21.15.93:3300/0 conn(0x7fdbd40c94c0 0x7fdbd40c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.980+0000 7fdbda000640 1 --2- 172.21.15.27:0/2159399869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbd415d0a0 0x7fdbd406c1e0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.980+0000 7fdbda000640 1 -- 172.21.15.27:0/2159399869 >> 172.21.15.27:0/2159399869 conn(0x7fdbd40cec60 msgr2=0x7fdbd40cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:23.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.980+0000 7fdbda000640 1 -- 172.21.15.27:0/2159399869 shutdown_connections 2024-01-15T18:07:23.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.980+0000 7fdbda000640 1 -- 172.21.15.27:0/2159399869 wait complete. 2024-01-15T18:07:23.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.980+0000 7fdbda000640 1 Processor -- start 2024-01-15T18:07:23.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.980+0000 7fdbda000640 1 -- start start 2024-01-15T18:07:23.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.980+0000 7fdbda000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbd40c94c0 0x7fdbd41525e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:23.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.980+0000 7fdbda000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdbd40ce710 0x7fdbd414eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:23.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.980+0000 7fdbda000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdbd415d0a0 0x7fdbd4167250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:23.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.980+0000 7fdbda000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd414bc30 con 0x7fdbd40c94c0 2024-01-15T18:07:23.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.980+0000 7fdbda000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd414bf10 con 0x7fdbd40ce710 2024-01-15T18:07:23.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.980+0000 7fdbda000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd414c1f0 con 0x7fdbd415d0a0 2024-01-15T18:07:23.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.980+0000 7fdbd3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbd40c94c0 0x7fdbd41525e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:23.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.980+0000 7fdbd3e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdbd415d0a0 0x7fdbd4167250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:23.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.980+0000 7fdbd3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbd40c94c0 0x7fdbd41525e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50700/0 (socket says 172.21.15.27:50700) 2024-01-15T18:07:23.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.980+0000 7fdbd3400640 1 -- 172.21.15.27:0/1018809823 learned_addr learned my addr 172.21.15.27:0/1018809823 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:23.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.980+0000 7fdbd2a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdbd40ce710 0x7fdbd414eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:23.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.984+0000 7fdbd3400640 1 -- 172.21.15.27:0/1018809823 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdbd415d0a0 msgr2=0x7fdbd4167250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:23.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.984+0000 7fdbd3400640 1 --2- 172.21.15.27:0/1018809823 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdbd415d0a0 0x7fdbd4167250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.984+0000 7fdbd3400640 1 -- 172.21.15.27:0/1018809823 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdbd40ce710 msgr2=0x7fdbd414eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:23.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.984+0000 7fdbd3400640 1 --2- 172.21.15.27:0/1018809823 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdbd40ce710 0x7fdbd414eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:23.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.984+0000 7fdbd3400640 1 -- 172.21.15.27:0/1018809823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc40094d0 con 0x7fdbd40c94c0 2024-01-15T18:07:23.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.984+0000 7fdbd3400640 1 --2- 172.21.15.27:0/1018809823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbd40c94c0 0x7fdbd41525e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fdbc40ac040 tx=0x7fdbc401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:23.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.984+0000 7fdbcb400640 1 -- 172.21.15.27:0/1018809823 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc40b3800 con 0x7fdbd40c94c0 2024-01-15T18:07:23.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.984+0000 7fdbcb400640 1 -- 172.21.15.27:0/1018809823 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc40b3980 con 0x7fdbd40c94c0 2024-01-15T18:07:23.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.984+0000 7fdbcb400640 1 -- 172.21.15.27:0/1018809823 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc40b3d30 con 0x7fdbd40c94c0 2024-01-15T18:07:23.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.984+0000 7fdbda000640 1 -- 172.21.15.27:0/1018809823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbd41519a0 con 0x7fdbd40c94c0 2024-01-15T18:07:23.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.984+0000 7fdbda000640 1 -- 172.21.15.27:0/1018809823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbd4152b20 con 0x7fdbd40c94c0 2024-01-15T18:07:23.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.984+0000 7fdbda000640 1 -- 172.21.15.27:0/1018809823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbd404e850 con 0x7fdbd40c94c0 2024-01-15T18:07:23.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.984+0000 7fdbcb400640 1 -- 172.21.15.27:0/1018809823 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7fdbc40c6440 con 0x7fdbd40c94c0 2024-01-15T18:07:23.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.988+0000 7fdbcb400640 1 --2- 172.21.15.27:0/1018809823 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdbac066b30 0x7fdbac068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:23.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.988+0000 7fdbcb400640 1 -- 172.21.15.27:0/1018809823 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fdbc411f340 con 0x7fdbd40c94c0 2024-01-15T18:07:23.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.988+0000 7fdbd2a00640 1 --2- 172.21.15.27:0/1018809823 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdbac066b30 0x7fdbac068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:23.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.988+0000 7fdbd2a00640 1 --2- 172.21.15.27:0/1018809823 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdbac066b30 0x7fdbac068ff0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fdbd40c8b30 tx=0x7fdbbc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:23.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:23.992+0000 7fdbcb400640 1 -- 172.21.15.27:0/1018809823 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fdbc40ec230 con 0x7fdbd40c94c0 2024-01-15T18:07:24.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.144+0000 7fdbda000640 1 -- 172.21.15.27:0/1018809823 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdbd406c5f0 con 0x7fdbac066b30 2024-01-15T18:07:24.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.156+0000 7fdbcb400640 1 -- 172.21.15.27:0/1018809823 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287094 (secure 0 0 0) 0x7fdbd406c5f0 con 0x7fdbac066b30 2024-01-15T18:07:24.163 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:07:24.165 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:07:24.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.160+0000 7fdbda000640 1 -- 172.21.15.27:0/1018809823 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdbac066b30 msgr2=0x7fdbac068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:24.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.160+0000 7fdbda000640 1 --2- 172.21.15.27:0/1018809823 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdbac066b30 0x7fdbac068ff0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fdbd40c8b30 tx=0x7fdbbc009040 comp rx=0 tx=0).stop 2024-01-15T18:07:24.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.160+0000 7fdbda000640 1 -- 172.21.15.27:0/1018809823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbd40c94c0 msgr2=0x7fdbd41525e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:24.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.160+0000 7fdbda000640 1 --2- 172.21.15.27:0/1018809823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbd40c94c0 0x7fdbd41525e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fdbc40ac040 tx=0x7fdbc401ad10 comp rx=0 tx=0).stop 2024-01-15T18:07:24.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.160+0000 7fdbda000640 1 -- 172.21.15.27:0/1018809823 shutdown_connections 2024-01-15T18:07:24.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.160+0000 7fdbda000640 1 --2- 172.21.15.27:0/1018809823 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdbac066b30 0x7fdbac068ff0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.160+0000 7fdbda000640 1 --2- 172.21.15.27:0/1018809823 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdbd415d0a0 0x7fdbd4167250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.160+0000 7fdbda000640 1 --2- 172.21.15.27:0/1018809823 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdbd40ce710 0x7fdbd414eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.160+0000 7fdbda000640 1 --2- 172.21.15.27:0/1018809823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdbd40c94c0 0x7fdbd41525e0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.160+0000 7fdbda000640 1 -- 172.21.15.27:0/1018809823 >> 172.21.15.27:0/1018809823 conn(0x7fdbd40cec60 msgr2=0x7fdbd40733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:24.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.160+0000 7fdbda000640 1 -- 172.21.15.27:0/1018809823 shutdown_connections 2024-01-15T18:07:24.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.160+0000 7fdbda000640 1 -- 172.21.15.27:0/1018809823 wait complete. 2024-01-15T18:07:24.184 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2024-01-15T18:07:22.170818+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592708,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":187,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":47,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254540,"kb_used_data":5056,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720180,"statfs":{"total":773094113280,"available":772833464320,"internally_reserved":0,"allocated":5177344,"data_stored":2774458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2340,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.119895"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.094792+0000","last_change":"2024-01-15T18:07:20.088581+0000","last_active":"2024-01-15T18:07:20.094792+0000","last_peered":"2024-01-15T18:07:20.094792+0000","last_clean":"2024-01-15T18:07:20.094792+0000","last_became_active":"2024-01-15T18:07:20.088247+0000","last_became_peered":"2024-01-15T18:07:20.088247+0000","last_unstale":"2024-01-15T18:07:20.094792+0000","last_undegraded":"2024-01-15T18:07:20.094792+0000","last_fullsized":"2024-01-15T18:07:20.094792+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:04:14.022333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.067098+0000","last_change":"2024-01-15T18:07:19.067098+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.067098+0000","last_undegraded":"2024-01-15T18:07:19.067098+0000","last_fullsized":"2024-01-15T18:07:19.067098+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.099935+0000","last_change":"2024-01-15T18:07:20.094383+0000","last_active":"2024-01-15T18:07:20.099935+0000","last_peered":"2024-01-15T18:07:20.099935+0000","last_clean":"2024-01-15T18:07:20.099935+0000","last_became_active":"2024-01-15T18:07:20.094114+0000","last_became_peered":"2024-01-15T18:07:20.094114+0000","last_unstale":"2024-01-15T18:07:20.099935+0000","last_undegraded":"2024-01-15T18:07:20.099935+0000","last_fullsized":"2024-01-15T18:07:20.099935+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:07:17.293929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.068614+0000","last_change":"2024-01-15T18:07:19.068614+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.068614+0000","last_undegraded":"2024-01-15T18:07:19.068614+0000","last_fullsized":"2024-01-15T18:07:19.068614+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.065311+0000","last_change":"2024-01-15T18:07:19.065311+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.065311+0000","last_undegraded":"2024-01-15T18:07:19.065311+0000","last_fullsized":"2024-01-15T18:07:19.065311+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100427+0000","last_change":"2024-01-15T18:07:20.090463+0000","last_active":"2024-01-15T18:07:20.100427+0000","last_peered":"2024-01-15T18:07:20.100427+0000","last_clean":"2024-01-15T18:07:20.100427+0000","last_became_active":"2024-01-15T18:07:20.090196+0000","last_became_peered":"2024-01-15T18:07:20.090196+0000","last_unstale":"2024-01-15T18:07:20.100427+0000","last_undegraded":"2024-01-15T18:07:20.100427+0000","last_fullsized":"2024-01-15T18:07:20.100427+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:03:21.190309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.106586+0000","last_change":"2024-01-15T18:07:20.102595+0000","last_active":"2024-01-15T18:07:20.106586+0000","last_peered":"2024-01-15T18:07:20.106586+0000","last_clean":"2024-01-15T18:07:20.106586+0000","last_became_active":"2024-01-15T18:07:20.102222+0000","last_became_peered":"2024-01-15T18:07:20.102222+0000","last_unstale":"2024-01-15T18:07:20.106586+0000","last_undegraded":"2024-01-15T18:07:20.106586+0000","last_fullsized":"2024-01-15T18:07:20.106586+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:51:04.157158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.074897+0000","last_change":"2024-01-15T18:07:19.074897+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.074897+0000","last_undegraded":"2024-01-15T18:07:19.074897+0000","last_fullsized":"2024-01-15T18:07:19.074897+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.077258+0000","last_change":"2024-01-15T18:07:19.077258+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.077258+0000","last_undegraded":"2024-01-15T18:07:19.077258+0000","last_fullsized":"2024-01-15T18:07:19.077258+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.070141+0000","last_change":"2024-01-15T18:07:19.070141+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.070141+0000","last_undegraded":"2024-01-15T18:07:19.070141+0000","last_fullsized":"2024-01-15T18:07:19.070141+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.074684+0000","last_change":"2024-01-15T18:07:19.074684+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.074684+0000","last_undegraded":"2024-01-15T18:07:19.074684+0000","last_fullsized":"2024-01-15T18:07:19.074684+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.072234+0000","last_change":"2024-01-15T18:07:19.072234+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.072234+0000","last_undegraded":"2024-01-15T18:07:19.072234+0000","last_fullsized":"2024-01-15T18:07:19.072234+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.106908+0000","last_change":"2024-01-15T18:07:20.093608+0000","last_active":"2024-01-15T18:07:20.106908+0000","last_peered":"2024-01-15T18:07:20.106908+0000","last_clean":"2024-01-15T18:07:20.106908+0000","last_became_active":"2024-01-15T18:07:20.091735+0000","last_became_peered":"2024-01-15T18:07:20.091735+0000","last_unstale":"2024-01-15T18:07:20.106908+0000","last_undegraded":"2024-01-15T18:07:20.106908+0000","last_fullsized":"2024-01-15T18:07:20.106908+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:34.221280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.068436+0000","last_change":"2024-01-15T18:07:19.068436+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.068436+0000","last_undegraded":"2024-01-15T18:07:19.068436+0000","last_fullsized":"2024-01-15T18:07:19.068436+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.105807+0000","last_change":"2024-01-15T18:07:20.096359+0000","last_active":"2024-01-15T18:07:20.105807+0000","last_peered":"2024-01-15T18:07:20.105807+0000","last_clean":"2024-01-15T18:07:20.105807+0000","last_became_active":"2024-01-15T18:07:20.096018+0000","last_became_peered":"2024-01-15T18:07:20.096018+0000","last_unstale":"2024-01-15T18:07:20.105807+0000","last_undegraded":"2024-01-15T18:07:20.105807+0000","last_fullsized":"2024-01-15T18:07:20.105807+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:35:28.043293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100722+0000","last_change":"2024-01-15T18:07:20.097218+0000","last_active":"2024-01-15T18:07:20.100722+0000","last_peered":"2024-01-15T18:07:20.100722+0000","last_clean":"2024-01-15T18:07:20.100722+0000","last_became_active":"2024-01-15T18:07:20.096805+0000","last_became_peered":"2024-01-15T18:07:20.096805+0000","last_unstale":"2024-01-15T18:07:20.100722+0000","last_undegraded":"2024-01-15T18:07:20.100722+0000","last_fullsized":"2024-01-15T18:07:20.100722+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:34.221280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097051+0000","last_change":"2024-01-15T18:07:20.084911+0000","last_active":"2024-01-15T18:07:20.097051+0000","last_peered":"2024-01-15T18:07:20.097051+0000","last_clean":"2024-01-15T18:07:20.097051+0000","last_became_active":"2024-01-15T18:07:20.084594+0000","last_became_peered":"2024-01-15T18:07:20.084594+0000","last_unstale":"2024-01-15T18:07:20.097051+0000","last_undegraded":"2024-01-15T18:07:20.097051+0000","last_fullsized":"2024-01-15T18:07:20.097051+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:38:53.126797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100540+0000","last_change":"2024-01-15T18:07:20.097350+0000","last_active":"2024-01-15T18:07:20.100540+0000","last_peered":"2024-01-15T18:07:20.100540+0000","last_clean":"2024-01-15T18:07:20.100540+0000","last_became_active":"2024-01-15T18:07:20.095556+0000","last_became_peered":"2024-01-15T18:07:20.095556+0000","last_unstale":"2024-01-15T18:07:20.100540+0000","last_undegraded":"2024-01-15T18:07:20.100540+0000","last_fullsized":"2024-01-15T18:07:20.100540+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:01:01.613225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.103149+0000","last_change":"2024-01-15T18:07:20.098939+0000","last_active":"2024-01-15T18:07:20.103149+0000","last_peered":"2024-01-15T18:07:20.103149+0000","last_clean":"2024-01-15T18:07:20.103149+0000","last_became_active":"2024-01-15T18:07:20.098598+0000","last_became_peered":"2024-01-15T18:07:20.098598+0000","last_unstale":"2024-01-15T18:07:20.103149+0000","last_undegraded":"2024-01-15T18:07:20.103149+0000","last_fullsized":"2024-01-15T18:07:20.103149+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:26.049225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.107947+0000","last_change":"2024-01-15T18:07:20.095741+0000","last_active":"2024-01-15T18:07:20.107947+0000","last_peered":"2024-01-15T18:07:20.107947+0000","last_clean":"2024-01-15T18:07:20.107947+0000","last_became_active":"2024-01-15T18:07:20.095417+0000","last_became_peered":"2024-01-15T18:07:20.095417+0000","last_unstale":"2024-01-15T18:07:20.107947+0000","last_undegraded":"2024-01-15T18:07:20.107947+0000","last_fullsized":"2024-01-15T18:07:20.107947+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:32:43.186714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.105539+0000","last_change":"2024-01-15T18:07:20.094321+0000","last_active":"2024-01-15T18:07:20.105539+0000","last_peered":"2024-01-15T18:07:20.105539+0000","last_clean":"2024-01-15T18:07:20.105539+0000","last_became_active":"2024-01-15T18:07:20.094011+0000","last_became_peered":"2024-01-15T18:07:20.094011+0000","last_unstale":"2024-01-15T18:07:20.105539+0000","last_undegraded":"2024-01-15T18:07:20.105539+0000","last_fullsized":"2024-01-15T18:07:20.105539+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:48:24.482600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.067198+0000","last_change":"2024-01-15T18:07:19.067198+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.067198+0000","last_undegraded":"2024-01-15T18:07:19.067198+0000","last_fullsized":"2024-01-15T18:07:19.067198+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098234+0000","last_change":"2024-01-15T18:07:20.087728+0000","last_active":"2024-01-15T18:07:20.098234+0000","last_peered":"2024-01-15T18:07:20.098234+0000","last_clean":"2024-01-15T18:07:20.098234+0000","last_became_active":"2024-01-15T18:07:20.087118+0000","last_became_peered":"2024-01-15T18:07:20.087118+0000","last_unstale":"2024-01-15T18:07:20.098234+0000","last_undegraded":"2024-01-15T18:07:20.098234+0000","last_fullsized":"2024-01-15T18:07:20.098234+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:32:43.186714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100694+0000","last_change":"2024-01-15T18:07:20.092361+0000","last_active":"2024-01-15T18:07:20.100694+0000","last_peered":"2024-01-15T18:07:20.100694+0000","last_clean":"2024-01-15T18:07:20.100694+0000","last_became_active":"2024-01-15T18:07:20.091927+0000","last_became_peered":"2024-01-15T18:07:20.091927+0000","last_unstale":"2024-01-15T18:07:20.100694+0000","last_undegraded":"2024-01-15T18:07:20.100694+0000","last_fullsized":"2024-01-15T18:07:20.100694+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:55:19.615935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.106512+0000","last_change":"2024-01-15T18:07:20.102013+0000","last_active":"2024-01-15T18:07:20.106512+0000","last_peered":"2024-01-15T18:07:20.106512+0000","last_clean":"2024-01-15T18:07:20.106512+0000","last_became_active":"2024-01-15T18:07:20.101697+0000","last_became_peered":"2024-01-15T18:07:20.101697+0000","last_unstale":"2024-01-15T18:07:20.106512+0000","last_undegraded":"2024-01-15T18:07:20.106512+0000","last_fullsized":"2024-01-15T18:07:20.106512+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:32:14.924018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.103301+0000","last_change":"2024-01-15T18:07:20.086187+0000","last_active":"2024-01-15T18:07:20.103301+0000","last_peered":"2024-01-15T18:07:20.103301+0000","last_clean":"2024-01-15T18:07:20.103301+0000","last_became_active":"2024-01-15T18:07:20.085891+0000","last_became_peered":"2024-01-15T18:07:20.085891+0000","last_unstale":"2024-01-15T18:07:20.103301+0000","last_undegraded":"2024-01-15T18:07:20.103301+0000","last_fullsized":"2024-01-15T18:07:20.103301+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:54:09.185168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097020+0000","last_change":"2024-01-15T18:07:20.086602+0000","last_active":"2024-01-15T18:07:20.097020+0000","last_peered":"2024-01-15T18:07:20.097020+0000","last_clean":"2024-01-15T18:07:20.097020+0000","last_became_active":"2024-01-15T18:07:20.086238+0000","last_became_peered":"2024-01-15T18:07:20.086238+0000","last_unstale":"2024-01-15T18:07:20.097020+0000","last_undegraded":"2024-01-15T18:07:20.097020+0000","last_fullsized":"2024-01-15T18:07:20.097020+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:20:04.640555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.071723+0000","last_change":"2024-01-15T18:07:19.071723+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.071723+0000","last_undegraded":"2024-01-15T18:07:19.071723+0000","last_fullsized":"2024-01-15T18:07:19.071723+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.103004+0000","last_change":"2024-01-15T18:07:20.097835+0000","last_active":"2024-01-15T18:07:20.103004+0000","last_peered":"2024-01-15T18:07:20.103004+0000","last_clean":"2024-01-15T18:07:20.103004+0000","last_became_active":"2024-01-15T18:07:20.097477+0000","last_became_peered":"2024-01-15T18:07:20.097477+0000","last_unstale":"2024-01-15T18:07:20.103004+0000","last_undegraded":"2024-01-15T18:07:20.103004+0000","last_fullsized":"2024-01-15T18:07:20.103004+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:36:51.036352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097114+0000","last_change":"2024-01-15T18:07:20.083756+0000","last_active":"2024-01-15T18:07:20.097114+0000","last_peered":"2024-01-15T18:07:20.097114+0000","last_clean":"2024-01-15T18:07:20.097114+0000","last_became_active":"2024-01-15T18:07:20.079072+0000","last_became_peered":"2024-01-15T18:07:20.079072+0000","last_unstale":"2024-01-15T18:07:20.097114+0000","last_undegraded":"2024-01-15T18:07:20.097114+0000","last_fullsized":"2024-01-15T18:07:20.097114+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:55:26.194143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.105788+0000","last_change":"2024-01-15T18:07:20.096208+0000","last_active":"2024-01-15T18:07:20.105788+0000","last_peered":"2024-01-15T18:07:20.105788+0000","last_clean":"2024-01-15T18:07:20.105788+0000","last_became_active":"2024-01-15T18:07:20.095900+0000","last_became_peered":"2024-01-15T18:07:20.095900+0000","last_unstale":"2024-01-15T18:07:20.105788+0000","last_undegraded":"2024-01-15T18:07:20.105788+0000","last_fullsized":"2024-01-15T18:07:20.105788+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:50:12.413904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.106884+0000","last_change":"2024-01-15T18:07:20.091218+0000","last_active":"2024-01-15T18:07:20.106884+0000","last_peered":"2024-01-15T18:07:20.106884+0000","last_clean":"2024-01-15T18:07:20.106884+0000","last_became_active":"2024-01-15T18:07:20.090896+0000","last_became_peered":"2024-01-15T18:07:20.090896+0000","last_unstale":"2024-01-15T18:07:20.106884+0000","last_undegraded":"2024-01-15T18:07:20.106884+0000","last_fullsized":"2024-01-15T18:07:20.106884+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:54:56.923677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098011+0000","last_change":"2024-01-15T18:07:20.087031+0000","last_active":"2024-01-15T18:07:20.098011+0000","last_peered":"2024-01-15T18:07:20.098011+0000","last_clean":"2024-01-15T18:07:20.098011+0000","last_became_active":"2024-01-15T18:07:20.086158+0000","last_became_peered":"2024-01-15T18:07:20.086158+0000","last_unstale":"2024-01-15T18:07:20.098011+0000","last_undegraded":"2024-01-15T18:07:20.098011+0000","last_fullsized":"2024-01-15T18:07:20.098011+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:55:26.194143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098945+0000","last_change":"2024-01-15T18:07:19.085120+0000","last_active":"2024-01-15T18:07:20.098945+0000","last_peered":"2024-01-15T18:07:20.098945+0000","last_clean":"2024-01-15T18:07:20.098945+0000","last_became_active":"2024-01-15T18:07:19.084785+0000","last_became_peered":"2024-01-15T18:07:19.084785+0000","last_unstale":"2024-01-15T18:07:20.098945+0000","last_undegraded":"2024-01-15T18:07:20.098945+0000","last_fullsized":"2024-01-15T18:07:20.098945+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:50:51.007855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100675+0000","last_change":"2024-01-15T18:07:20.094786+0000","last_active":"2024-01-15T18:07:20.100675+0000","last_peered":"2024-01-15T18:07:20.100675+0000","last_clean":"2024-01-15T18:07:20.100675+0000","last_became_active":"2024-01-15T18:07:20.094523+0000","last_became_peered":"2024-01-15T18:07:20.094523+0000","last_unstale":"2024-01-15T18:07:20.100675+0000","last_undegraded":"2024-01-15T18:07:20.100675+0000","last_fullsized":"2024-01-15T18:07:20.100675+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:59:01.979409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.096749+0000","last_change":"2024-01-15T18:07:19.084713+0000","last_active":"2024-01-15T18:07:20.096749+0000","last_peered":"2024-01-15T18:07:20.096749+0000","last_clean":"2024-01-15T18:07:20.096749+0000","last_became_active":"2024-01-15T18:07:19.084414+0000","last_became_peered":"2024-01-15T18:07:19.084414+0000","last_unstale":"2024-01-15T18:07:20.096749+0000","last_undegraded":"2024-01-15T18:07:20.096749+0000","last_fullsized":"2024-01-15T18:07:20.096749+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:20.019967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.068126+0000","last_change":"2024-01-15T18:07:19.068126+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.068126+0000","last_undegraded":"2024-01-15T18:07:19.068126+0000","last_fullsized":"2024-01-15T18:07:19.068126+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100965+0000","last_change":"2024-01-15T18:07:19.077287+0000","last_active":"2024-01-15T18:07:20.100965+0000","last_peered":"2024-01-15T18:07:20.100965+0000","last_clean":"2024-01-15T18:07:20.100965+0000","last_became_active":"2024-01-15T18:07:19.075652+0000","last_became_peered":"2024-01-15T18:07:19.075652+0000","last_unstale":"2024-01-15T18:07:20.100965+0000","last_undegraded":"2024-01-15T18:07:20.100965+0000","last_fullsized":"2024-01-15T18:07:20.100965+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:57:56.264139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098517+0000","last_change":"2024-01-15T18:07:20.087874+0000","last_active":"2024-01-15T18:07:20.098517+0000","last_peered":"2024-01-15T18:07:20.098517+0000","last_clean":"2024-01-15T18:07:20.098517+0000","last_became_active":"2024-01-15T18:07:20.087579+0000","last_became_peered":"2024-01-15T18:07:20.087579+0000","last_unstale":"2024-01-15T18:07:20.098517+0000","last_undegraded":"2024-01-15T18:07:20.098517+0000","last_fullsized":"2024-01-15T18:07:20.098517+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:20:04.640555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098218+0000","last_change":"2024-01-15T18:07:19.085001+0000","last_active":"2024-01-15T18:07:20.098218+0000","last_peered":"2024-01-15T18:07:20.098218+0000","last_clean":"2024-01-15T18:07:20.098218+0000","last_became_active":"2024-01-15T18:07:19.084651+0000","last_became_peered":"2024-01-15T18:07:19.084651+0000","last_unstale":"2024-01-15T18:07:20.098218+0000","last_undegraded":"2024-01-15T18:07:20.098218+0000","last_fullsized":"2024-01-15T18:07:20.098218+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:13:24.552254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097208+0000","last_change":"2024-01-15T18:07:20.087693+0000","last_active":"2024-01-15T18:07:20.097208+0000","last_peered":"2024-01-15T18:07:20.097208+0000","last_clean":"2024-01-15T18:07:20.097208+0000","last_became_active":"2024-01-15T18:07:20.087416+0000","last_became_peered":"2024-01-15T18:07:20.087416+0000","last_unstale":"2024-01-15T18:07:20.097208+0000","last_undegraded":"2024-01-15T18:07:20.097208+0000","last_fullsized":"2024-01-15T18:07:20.097208+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:02.825993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097250+0000","last_change":"2024-01-15T18:07:19.086152+0000","last_active":"2024-01-15T18:07:20.097250+0000","last_peered":"2024-01-15T18:07:20.097250+0000","last_clean":"2024-01-15T18:07:20.097250+0000","last_became_active":"2024-01-15T18:07:19.082058+0000","last_became_peered":"2024-01-15T18:07:19.082058+0000","last_unstale":"2024-01-15T18:07:20.097250+0000","last_undegraded":"2024-01-15T18:07:20.097250+0000","last_fullsized":"2024-01-15T18:07:20.097250+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:06:20.719015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.074401+0000","last_change":"2024-01-15T18:07:19.074401+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.074401+0000","last_undegraded":"2024-01-15T18:07:19.074401+0000","last_fullsized":"2024-01-15T18:07:19.074401+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097139+0000","last_change":"2024-01-15T18:07:19.076932+0000","last_active":"2024-01-15T18:07:20.097139+0000","last_peered":"2024-01-15T18:07:20.097139+0000","last_clean":"2024-01-15T18:07:20.097139+0000","last_became_active":"2024-01-15T18:07:19.076629+0000","last_became_peered":"2024-01-15T18:07:19.076629+0000","last_unstale":"2024-01-15T18:07:20.097139+0000","last_undegraded":"2024-01-15T18:07:20.097139+0000","last_fullsized":"2024-01-15T18:07:20.097139+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:24:59.066289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.069432+0000","last_change":"2024-01-15T18:07:19.069432+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.069432+0000","last_undegraded":"2024-01-15T18:07:19.069432+0000","last_fullsized":"2024-01-15T18:07:19.069432+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.084299+0000","last_change":"2024-01-15T18:07:19.079711+0000","last_active":"2024-01-15T18:07:19.084299+0000","last_peered":"2024-01-15T18:07:19.084299+0000","last_clean":"2024-01-15T18:07:19.084299+0000","last_became_active":"2024-01-15T18:07:19.079336+0000","last_became_peered":"2024-01-15T18:07:19.079336+0000","last_unstale":"2024-01-15T18:07:19.084299+0000","last_undegraded":"2024-01-15T18:07:19.084299+0000","last_fullsized":"2024-01-15T18:07:19.084299+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:49:49.489371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098255+0000","last_change":"2024-01-15T18:07:20.087900+0000","last_active":"2024-01-15T18:07:20.098255+0000","last_peered":"2024-01-15T18:07:20.098255+0000","last_clean":"2024-01-15T18:07:20.098255+0000","last_became_active":"2024-01-15T18:07:20.087463+0000","last_became_peered":"2024-01-15T18:07:20.087463+0000","last_unstale":"2024-01-15T18:07:20.098255+0000","last_undegraded":"2024-01-15T18:07:20.098255+0000","last_fullsized":"2024-01-15T18:07:20.098255+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:54:09.185168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.082790+0000","last_change":"2024-01-15T18:07:19.079615+0000","last_active":"2024-01-15T18:07:19.082790+0000","last_peered":"2024-01-15T18:07:19.082790+0000","last_clean":"2024-01-15T18:07:19.082790+0000","last_became_active":"2024-01-15T18:07:19.079242+0000","last_became_peered":"2024-01-15T18:07:19.079242+0000","last_unstale":"2024-01-15T18:07:19.082790+0000","last_undegraded":"2024-01-15T18:07:19.082790+0000","last_fullsized":"2024-01-15T18:07:19.082790+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:47.335570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100426+0000","last_change":"2024-01-15T18:07:20.095844+0000","last_active":"2024-01-15T18:07:20.100426+0000","last_peered":"2024-01-15T18:07:20.100426+0000","last_clean":"2024-01-15T18:07:20.100426+0000","last_became_active":"2024-01-15T18:07:20.095110+0000","last_became_peered":"2024-01-15T18:07:20.095110+0000","last_unstale":"2024-01-15T18:07:20.100426+0000","last_undegraded":"2024-01-15T18:07:20.100426+0000","last_fullsized":"2024-01-15T18:07:20.100426+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:57:57.266821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.070487+0000","last_change":"2024-01-15T18:07:19.076938+0000","last_active":"2024-01-15T18:07:20.070487+0000","last_peered":"2024-01-15T18:07:20.070487+0000","last_clean":"2024-01-15T18:07:20.070487+0000","last_became_active":"2024-01-15T18:07:19.076605+0000","last_became_peered":"2024-01-15T18:07:19.076605+0000","last_unstale":"2024-01-15T18:07:20.070487+0000","last_undegraded":"2024-01-15T18:07:20.070487+0000","last_fullsized":"2024-01-15T18:07:20.070487+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:32.749313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.095519+0000","last_change":"2024-01-15T18:07:20.091131+0000","last_active":"2024-01-15T18:07:20.095519+0000","last_peered":"2024-01-15T18:07:20.095519+0000","last_clean":"2024-01-15T18:07:20.095519+0000","last_became_active":"2024-01-15T18:07:20.090683+0000","last_became_peered":"2024-01-15T18:07:20.090683+0000","last_unstale":"2024-01-15T18:07:20.095519+0000","last_undegraded":"2024-01-15T18:07:20.095519+0000","last_fullsized":"2024-01-15T18:07:20.095519+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:36:14.191387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.086659+0000","last_change":"2024-01-15T18:07:19.085014+0000","last_active":"2024-01-15T18:07:19.086659+0000","last_peered":"2024-01-15T18:07:19.086659+0000","last_clean":"2024-01-15T18:07:19.086659+0000","last_became_active":"2024-01-15T18:07:19.084470+0000","last_became_peered":"2024-01-15T18:07:19.084470+0000","last_unstale":"2024-01-15T18:07:19.086659+0000","last_undegraded":"2024-01-15T18:07:19.086659+0000","last_fullsized":"2024-01-15T18:07:19.086659+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:43:21.743406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.114219+0000","last_change":"2024-01-15T18:07:20.108821+0000","last_active":"2024-01-15T18:07:20.114219+0000","last_peered":"2024-01-15T18:07:20.114219+0000","last_clean":"2024-01-15T18:07:20.114219+0000","last_became_active":"2024-01-15T18:07:20.108464+0000","last_became_peered":"2024-01-15T18:07:20.108464+0000","last_unstale":"2024-01-15T18:07:20.114219+0000","last_undegraded":"2024-01-15T18:07:20.114219+0000","last_fullsized":"2024-01-15T18:07:20.114219+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:23:28.608885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"44'3","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.099259+0000","last_change":"2024-01-15T18:07:19.088202+0000","last_active":"2024-01-15T18:07:20.099259+0000","last_peered":"2024-01-15T18:07:20.099259+0000","last_clean":"2024-01-15T18:07:20.099259+0000","last_became_active":"2024-01-15T18:07:19.087900+0000","last_became_peered":"2024-01-15T18:07:19.087900+0000","last_unstale":"2024-01-15T18:07:20.099259+0000","last_undegraded":"2024-01-15T18:07:20.099259+0000","last_fullsized":"2024-01-15T18:07:20.099259+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:10:42.345570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.106599+0000","last_change":"2024-01-15T18:07:20.102897+0000","last_active":"2024-01-15T18:07:20.106599+0000","last_peered":"2024-01-15T18:07:20.106599+0000","last_clean":"2024-01-15T18:07:20.106599+0000","last_became_active":"2024-01-15T18:07:20.102579+0000","last_became_peered":"2024-01-15T18:07:20.102579+0000","last_unstale":"2024-01-15T18:07:20.106599+0000","last_undegraded":"2024-01-15T18:07:20.106599+0000","last_fullsized":"2024-01-15T18:07:20.106599+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:50:12.413904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":17,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.112054+0000","last_change":"2024-01-15T18:07:19.079571+0000","last_active":"2024-01-15T18:07:20.112054+0000","last_peered":"2024-01-15T18:07:20.112054+0000","last_clean":"2024-01-15T18:07:20.112054+0000","last_became_active":"2024-01-15T18:07:19.079308+0000","last_became_peered":"2024-01-15T18:07:19.079308+0000","last_unstale":"2024-01-15T18:07:20.112054+0000","last_undegraded":"2024-01-15T18:07:20.112054+0000","last_fullsized":"2024-01-15T18:07:20.112054+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:25.046543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.101579+0000","last_change":"2024-01-15T18:07:20.097611+0000","last_active":"2024-01-15T18:07:20.101579+0000","last_peered":"2024-01-15T18:07:20.101579+0000","last_clean":"2024-01-15T18:07:20.101579+0000","last_became_active":"2024-01-15T18:07:20.097202+0000","last_became_peered":"2024-01-15T18:07:20.097202+0000","last_unstale":"2024-01-15T18:07:20.101579+0000","last_undegraded":"2024-01-15T18:07:20.101579+0000","last_fullsized":"2024-01-15T18:07:20.101579+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:52:34.679712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.104292+0000","last_change":"2024-01-15T18:07:19.084662+0000","last_active":"2024-01-15T18:07:20.104292+0000","last_peered":"2024-01-15T18:07:20.104292+0000","last_clean":"2024-01-15T18:07:20.104292+0000","last_became_active":"2024-01-15T18:07:19.084348+0000","last_became_peered":"2024-01-15T18:07:19.084348+0000","last_unstale":"2024-01-15T18:07:20.104292+0000","last_undegraded":"2024-01-15T18:07:20.104292+0000","last_fullsized":"2024-01-15T18:07:20.104292+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:07:49.351274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.067641+0000","last_change":"2024-01-15T18:07:19.067641+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.067641+0000","last_undegraded":"2024-01-15T18:07:19.067641+0000","last_fullsized":"2024-01-15T18:07:19.067641+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.076930+0000","last_change":"2024-01-15T18:07:19.084512+0000","last_active":"2024-01-15T18:07:20.076930+0000","last_peered":"2024-01-15T18:07:20.076930+0000","last_clean":"2024-01-15T18:07:20.076930+0000","last_became_active":"2024-01-15T18:07:19.084169+0000","last_became_peered":"2024-01-15T18:07:19.084169+0000","last_unstale":"2024-01-15T18:07:20.076930+0000","last_undegraded":"2024-01-15T18:07:20.076930+0000","last_fullsized":"2024-01-15T18:07:20.076930+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:35:27.040611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.109334+0000","last_change":"2024-01-15T18:07:20.105484+0000","last_active":"2024-01-15T18:07:20.109334+0000","last_peered":"2024-01-15T18:07:20.109334+0000","last_clean":"2024-01-15T18:07:20.109334+0000","last_became_active":"2024-01-15T18:07:20.105160+0000","last_became_peered":"2024-01-15T18:07:20.105160+0000","last_unstale":"2024-01-15T18:07:20.109334+0000","last_undegraded":"2024-01-15T18:07:20.109334+0000","last_fullsized":"2024-01-15T18:07:20.109334+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:04:14.022333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.073804+0000","last_change":"2024-01-15T18:07:19.083510+0000","last_active":"2024-01-15T18:07:20.073804+0000","last_peered":"2024-01-15T18:07:20.073804+0000","last_clean":"2024-01-15T18:07:20.073804+0000","last_became_active":"2024-01-15T18:07:19.083134+0000","last_became_peered":"2024-01-15T18:07:19.083134+0000","last_unstale":"2024-01-15T18:07:20.073804+0000","last_undegraded":"2024-01-15T18:07:20.073804+0000","last_fullsized":"2024-01-15T18:07:20.073804+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:10:42.345570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.103258+0000","last_change":"2024-01-15T18:07:20.101397+0000","last_active":"2024-01-15T18:07:20.103258+0000","last_peered":"2024-01-15T18:07:20.103258+0000","last_clean":"2024-01-15T18:07:20.103258+0000","last_became_active":"2024-01-15T18:07:20.101106+0000","last_became_peered":"2024-01-15T18:07:20.101106+0000","last_unstale":"2024-01-15T18:07:20.103258+0000","last_undegraded":"2024-01-15T18:07:20.103258+0000","last_fullsized":"2024-01-15T18:07:20.103258+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:48:24.482600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.088352+0000","last_change":"2024-01-15T18:07:19.086583+0000","last_active":"2024-01-15T18:07:19.088352+0000","last_peered":"2024-01-15T18:07:19.088352+0000","last_clean":"2024-01-15T18:07:19.088352+0000","last_became_active":"2024-01-15T18:07:19.086227+0000","last_became_peered":"2024-01-15T18:07:19.086227+0000","last_unstale":"2024-01-15T18:07:19.088352+0000","last_undegraded":"2024-01-15T18:07:19.088352+0000","last_fullsized":"2024-01-15T18:07:19.088352+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:24:53.993680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.102535+0000","last_change":"2024-01-15T18:07:20.075833+0000","last_active":"2024-01-15T18:07:20.102535+0000","last_peered":"2024-01-15T18:07:20.102535+0000","last_clean":"2024-01-15T18:07:20.102535+0000","last_became_active":"2024-01-15T18:07:20.075521+0000","last_became_peered":"2024-01-15T18:07:20.075521+0000","last_unstale":"2024-01-15T18:07:20.102535+0000","last_undegraded":"2024-01-15T18:07:20.102535+0000","last_fullsized":"2024-01-15T18:07:20.102535+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:52:34.679712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.107096+0000","last_change":"2024-01-15T18:07:19.087887+0000","last_active":"2024-01-15T18:07:20.107096+0000","last_peered":"2024-01-15T18:07:20.107096+0000","last_clean":"2024-01-15T18:07:20.107096+0000","last_became_active":"2024-01-15T18:07:19.087610+0000","last_became_peered":"2024-01-15T18:07:19.087610+0000","last_unstale":"2024-01-15T18:07:20.107096+0000","last_undegraded":"2024-01-15T18:07:20.107096+0000","last_fullsized":"2024-01-15T18:07:20.107096+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:29:39.445420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.068689+0000","last_change":"2024-01-15T18:07:19.068689+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.068689+0000","last_undegraded":"2024-01-15T18:07:19.068689+0000","last_fullsized":"2024-01-15T18:07:19.068689+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.101388+0000","last_change":"2024-01-15T18:07:19.074682+0000","last_active":"2024-01-15T18:07:20.101388+0000","last_peered":"2024-01-15T18:07:20.101388+0000","last_clean":"2024-01-15T18:07:20.101388+0000","last_became_active":"2024-01-15T18:07:19.074397+0000","last_became_peered":"2024-01-15T18:07:19.074397+0000","last_unstale":"2024-01-15T18:07:20.101388+0000","last_undegraded":"2024-01-15T18:07:20.101388+0000","last_fullsized":"2024-01-15T18:07:20.101388+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:37:09.401497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100562+0000","last_change":"2024-01-15T18:07:20.093962+0000","last_active":"2024-01-15T18:07:20.100562+0000","last_peered":"2024-01-15T18:07:20.100562+0000","last_clean":"2024-01-15T18:07:20.100562+0000","last_became_active":"2024-01-15T18:07:20.093638+0000","last_became_peered":"2024-01-15T18:07:20.093638+0000","last_unstale":"2024-01-15T18:07:20.100562+0000","last_undegraded":"2024-01-15T18:07:20.100562+0000","last_fullsized":"2024-01-15T18:07:20.100562+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:30:43.231075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.086984+0000","last_change":"2024-01-15T18:07:19.082541+0000","last_active":"2024-01-15T18:07:19.086984+0000","last_peered":"2024-01-15T18:07:19.086984+0000","last_clean":"2024-01-15T18:07:19.086984+0000","last_became_active":"2024-01-15T18:07:19.082192+0000","last_became_peered":"2024-01-15T18:07:19.082192+0000","last_unstale":"2024-01-15T18:07:19.086984+0000","last_undegraded":"2024-01-15T18:07:19.086984+0000","last_fullsized":"2024-01-15T18:07:19.086984+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:10:51.637250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.103168+0000","last_change":"2024-01-15T18:07:20.096891+0000","last_active":"2024-01-15T18:07:20.103168+0000","last_peered":"2024-01-15T18:07:20.103168+0000","last_clean":"2024-01-15T18:07:20.103168+0000","last_became_active":"2024-01-15T18:07:20.096533+0000","last_became_peered":"2024-01-15T18:07:20.096533+0000","last_unstale":"2024-01-15T18:07:20.103168+0000","last_undegraded":"2024-01-15T18:07:20.103168+0000","last_fullsized":"2024-01-15T18:07:20.103168+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:14:41.208287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.075980+0000","last_change":"2024-01-15T18:07:19.080295+0000","last_active":"2024-01-15T18:07:20.075980+0000","last_peered":"2024-01-15T18:07:20.075980+0000","last_clean":"2024-01-15T18:07:20.075980+0000","last_became_active":"2024-01-15T18:07:19.080024+0000","last_became_peered":"2024-01-15T18:07:19.080024+0000","last_unstale":"2024-01-15T18:07:20.075980+0000","last_undegraded":"2024-01-15T18:07:20.075980+0000","last_fullsized":"2024-01-15T18:07:20.075980+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:36:48.732986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.112236+0000","last_change":"2024-01-15T18:07:20.105757+0000","last_active":"2024-01-15T18:07:20.112236+0000","last_peered":"2024-01-15T18:07:20.112236+0000","last_clean":"2024-01-15T18:07:20.112236+0000","last_became_active":"2024-01-15T18:07:20.105385+0000","last_became_peered":"2024-01-15T18:07:20.105385+0000","last_unstale":"2024-01-15T18:07:20.112236+0000","last_undegraded":"2024-01-15T18:07:20.112236+0000","last_fullsized":"2024-01-15T18:07:20.112236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:51:04.157158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097287+0000","last_change":"2024-01-15T18:07:19.082297+0000","last_active":"2024-01-15T18:07:20.097287+0000","last_peered":"2024-01-15T18:07:20.097287+0000","last_clean":"2024-01-15T18:07:20.097287+0000","last_became_active":"2024-01-15T18:07:19.081971+0000","last_became_peered":"2024-01-15T18:07:19.081971+0000","last_unstale":"2024-01-15T18:07:20.097287+0000","last_undegraded":"2024-01-15T18:07:20.097287+0000","last_fullsized":"2024-01-15T18:07:20.097287+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:23:41.478738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098551+0000","last_change":"2024-01-15T18:07:20.091112+0000","last_active":"2024-01-15T18:07:20.098551+0000","last_peered":"2024-01-15T18:07:20.098551+0000","last_clean":"2024-01-15T18:07:20.098551+0000","last_became_active":"2024-01-15T18:07:20.090802+0000","last_became_peered":"2024-01-15T18:07:20.090802+0000","last_unstale":"2024-01-15T18:07:20.098551+0000","last_undegraded":"2024-01-15T18:07:20.098551+0000","last_fullsized":"2024-01-15T18:07:20.098551+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:01:01.613225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.082669+0000","last_change":"2024-01-15T18:07:19.077896+0000","last_active":"2024-01-15T18:07:19.082669+0000","last_peered":"2024-01-15T18:07:19.082669+0000","last_clean":"2024-01-15T18:07:19.082669+0000","last_became_active":"2024-01-15T18:07:19.077439+0000","last_became_peered":"2024-01-15T18:07:19.077439+0000","last_unstale":"2024-01-15T18:07:19.082669+0000","last_undegraded":"2024-01-15T18:07:19.082669+0000","last_fullsized":"2024-01-15T18:07:19.082669+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:09:06.512495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.104003+0000","last_change":"2024-01-15T18:07:20.086589+0000","last_active":"2024-01-15T18:07:20.104003+0000","last_peered":"2024-01-15T18:07:20.104003+0000","last_clean":"2024-01-15T18:07:20.104003+0000","last_became_active":"2024-01-15T18:07:20.086312+0000","last_became_peered":"2024-01-15T18:07:20.086312+0000","last_unstale":"2024-01-15T18:07:20.104003+0000","last_undegraded":"2024-01-15T18:07:20.104003+0000","last_fullsized":"2024-01-15T18:07:20.104003+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:15:59.320681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.083833+0000","last_change":"2024-01-15T18:07:19.082155+0000","last_active":"2024-01-15T18:07:19.083833+0000","last_peered":"2024-01-15T18:07:19.083833+0000","last_clean":"2024-01-15T18:07:19.083833+0000","last_became_active":"2024-01-15T18:07:19.081855+0000","last_became_peered":"2024-01-15T18:07:19.081855+0000","last_unstale":"2024-01-15T18:07:19.083833+0000","last_undegraded":"2024-01-15T18:07:19.083833+0000","last_fullsized":"2024-01-15T18:07:19.083833+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:07.014633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.067698+0000","last_change":"2024-01-15T18:07:19.067698+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.067698+0000","last_undegraded":"2024-01-15T18:07:19.067698+0000","last_fullsized":"2024-01-15T18:07:19.067698+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.076645+0000","last_change":"2024-01-15T18:07:19.085582+0000","last_active":"2024-01-15T18:07:20.076645+0000","last_peered":"2024-01-15T18:07:20.076645+0000","last_clean":"2024-01-15T18:07:20.076645+0000","last_became_active":"2024-01-15T18:07:19.083796+0000","last_became_peered":"2024-01-15T18:07:19.083796+0000","last_unstale":"2024-01-15T18:07:20.076645+0000","last_undegraded":"2024-01-15T18:07:20.076645+0000","last_fullsized":"2024-01-15T18:07:20.076645+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:03:20.187627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.106501+0000","last_change":"2024-01-15T18:07:20.100547+0000","last_active":"2024-01-15T18:07:20.106501+0000","last_peered":"2024-01-15T18:07:20.106501+0000","last_clean":"2024-01-15T18:07:20.106501+0000","last_became_active":"2024-01-15T18:07:20.100201+0000","last_became_peered":"2024-01-15T18:07:20.100201+0000","last_unstale":"2024-01-15T18:07:20.106501+0000","last_undegraded":"2024-01-15T18:07:20.106501+0000","last_fullsized":"2024-01-15T18:07:20.106501+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:29:40.448102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.072838+0000","last_change":"2024-01-15T18:07:19.078965+0000","last_active":"2024-01-15T18:07:20.072838+0000","last_peered":"2024-01-15T18:07:20.072838+0000","last_clean":"2024-01-15T18:07:20.072838+0000","last_became_active":"2024-01-15T18:07:19.078692+0000","last_became_peered":"2024-01-15T18:07:19.078692+0000","last_unstale":"2024-01-15T18:07:20.072838+0000","last_undegraded":"2024-01-15T18:07:20.072838+0000","last_fullsized":"2024-01-15T18:07:20.072838+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:13:24.552254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.067320+0000","last_change":"2024-01-15T18:07:19.067320+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.067320+0000","last_undegraded":"2024-01-15T18:07:19.067320+0000","last_fullsized":"2024-01-15T18:07:19.067320+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.077286+0000","last_change":"2024-01-15T18:07:19.088174+0000","last_active":"2024-01-15T18:07:20.077286+0000","last_peered":"2024-01-15T18:07:20.077286+0000","last_clean":"2024-01-15T18:07:20.077286+0000","last_became_active":"2024-01-15T18:07:19.087881+0000","last_became_peered":"2024-01-15T18:07:19.087881+0000","last_unstale":"2024-01-15T18:07:20.077286+0000","last_undegraded":"2024-01-15T18:07:20.077286+0000","last_fullsized":"2024-01-15T18:07:20.077286+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:36:48.732986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.108471+0000","last_change":"2024-01-15T18:07:20.104788+0000","last_active":"2024-01-15T18:07:20.108471+0000","last_peered":"2024-01-15T18:07:20.108471+0000","last_clean":"2024-01-15T18:07:20.108471+0000","last_became_active":"2024-01-15T18:07:20.104475+0000","last_became_peered":"2024-01-15T18:07:20.104475+0000","last_unstale":"2024-01-15T18:07:20.108471+0000","last_undegraded":"2024-01-15T18:07:20.108471+0000","last_fullsized":"2024-01-15T18:07:20.108471+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:02:26.277338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.071516+0000","last_change":"2024-01-15T18:07:19.080590+0000","last_active":"2024-01-15T18:07:20.071516+0000","last_peered":"2024-01-15T18:07:20.071516+0000","last_clean":"2024-01-15T18:07:20.071516+0000","last_became_active":"2024-01-15T18:07:19.080276+0000","last_became_peered":"2024-01-15T18:07:19.080276+0000","last_unstale":"2024-01-15T18:07:20.071516+0000","last_undegraded":"2024-01-15T18:07:20.071516+0000","last_fullsized":"2024-01-15T18:07:20.071516+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:24:59.066289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100402+0000","last_change":"2024-01-15T18:07:20.090127+0000","last_active":"2024-01-15T18:07:20.100402+0000","last_peered":"2024-01-15T18:07:20.100402+0000","last_clean":"2024-01-15T18:07:20.100402+0000","last_became_active":"2024-01-15T18:07:20.089727+0000","last_became_peered":"2024-01-15T18:07:20.089727+0000","last_unstale":"2024-01-15T18:07:20.100402+0000","last_undegraded":"2024-01-15T18:07:20.100402+0000","last_fullsized":"2024-01-15T18:07:20.100402+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:36:14.191387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.077270+0000","last_change":"2024-01-15T18:07:19.088037+0000","last_active":"2024-01-15T18:07:20.077270+0000","last_peered":"2024-01-15T18:07:20.077270+0000","last_clean":"2024-01-15T18:07:20.077270+0000","last_became_active":"2024-01-15T18:07:19.087721+0000","last_became_peered":"2024-01-15T18:07:19.087721+0000","last_unstale":"2024-01-15T18:07:20.077270+0000","last_undegraded":"2024-01-15T18:07:20.077270+0000","last_fullsized":"2024-01-15T18:07:20.077270+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:23:27.606202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.111942+0000","last_change":"2024-01-15T18:07:20.105653+0000","last_active":"2024-01-15T18:07:20.111942+0000","last_peered":"2024-01-15T18:07:20.111942+0000","last_clean":"2024-01-15T18:07:20.111942+0000","last_became_active":"2024-01-15T18:07:20.105348+0000","last_became_peered":"2024-01-15T18:07:20.105348+0000","last_unstale":"2024-01-15T18:07:20.111942+0000","last_undegraded":"2024-01-15T18:07:20.111942+0000","last_fullsized":"2024-01-15T18:07:20.111942+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:02:26.277338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.068819+0000","last_change":"2024-01-15T18:07:19.076027+0000","last_active":"2024-01-15T18:07:20.068819+0000","last_peered":"2024-01-15T18:07:20.068819+0000","last_clean":"2024-01-15T18:07:20.068819+0000","last_became_active":"2024-01-15T18:07:19.075644+0000","last_became_peered":"2024-01-15T18:07:19.075644+0000","last_unstale":"2024-01-15T18:07:20.068819+0000","last_undegraded":"2024-01-15T18:07:20.068819+0000","last_fullsized":"2024-01-15T18:07:20.068819+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:07:57.062677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.064957+0000","last_change":"2024-01-15T18:07:19.064957+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.064957+0000","last_undegraded":"2024-01-15T18:07:19.064957+0000","last_fullsized":"2024-01-15T18:07:19.064957+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.072048+0000","last_change":"2024-01-15T18:07:19.081215+0000","last_active":"2024-01-15T18:07:20.072048+0000","last_peered":"2024-01-15T18:07:20.072048+0000","last_clean":"2024-01-15T18:07:20.072048+0000","last_became_active":"2024-01-15T18:07:19.080901+0000","last_became_peered":"2024-01-15T18:07:19.080901+0000","last_unstale":"2024-01-15T18:07:20.072048+0000","last_undegraded":"2024-01-15T18:07:20.072048+0000","last_fullsized":"2024-01-15T18:07:20.072048+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:38:52.124114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.094487+0000","last_change":"2024-01-15T18:07:20.081122+0000","last_active":"2024-01-15T18:07:20.094487+0000","last_peered":"2024-01-15T18:07:20.094487+0000","last_clean":"2024-01-15T18:07:20.094487+0000","last_became_active":"2024-01-15T18:07:20.080752+0000","last_became_peered":"2024-01-15T18:07:20.080752+0000","last_unstale":"2024-01-15T18:07:20.094487+0000","last_undegraded":"2024-01-15T18:07:20.094487+0000","last_fullsized":"2024-01-15T18:07:20.094487+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:08:56.987120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.087492+0000","last_change":"2024-01-15T18:07:19.085696+0000","last_active":"2024-01-15T18:07:19.087492+0000","last_peered":"2024-01-15T18:07:19.087492+0000","last_clean":"2024-01-15T18:07:19.087492+0000","last_became_active":"2024-01-15T18:07:19.085467+0000","last_became_peered":"2024-01-15T18:07:19.085467+0000","last_unstale":"2024-01-15T18:07:19.087492+0000","last_undegraded":"2024-01-15T18:07:19.087492+0000","last_fullsized":"2024-01-15T18:07:19.087492+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:26:07.301619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.068029+0000","last_change":"2024-01-15T18:07:19.068029+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.068029+0000","last_undegraded":"2024-01-15T18:07:19.068029+0000","last_fullsized":"2024-01-15T18:07:19.068029+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.086899+0000","last_change":"2024-01-15T18:07:19.081361+0000","last_active":"2024-01-15T18:07:19.086899+0000","last_peered":"2024-01-15T18:07:19.086899+0000","last_clean":"2024-01-15T18:07:19.086899+0000","last_became_active":"2024-01-15T18:07:19.081076+0000","last_became_peered":"2024-01-15T18:07:19.081076+0000","last_unstale":"2024-01-15T18:07:19.086899+0000","last_undegraded":"2024-01-15T18:07:19.086899+0000","last_fullsized":"2024-01-15T18:07:19.086899+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:34:07.305579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":183,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.074017+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:07:19.074017+0000","last_peered":"2024-01-15T18:07:19.074017+0000","last_clean":"2024-01-15T18:07:19.074017+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:07:19.074017+0000","last_undegraded":"2024-01-15T18:07:19.074017+0000","last_fullsized":"2024-01-15T18:07:19.074017+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2340,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509474,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32248,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339592,"statfs":{"total":96636764160,"available":96603742208,"internally_reserved":0,"allocated":1089536,"data_stored":789175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.628,"15min":0.625},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.67200000000000004,"15min":0.67700000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.78200000000000003}]},{"osd":1,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.67800000000000005,"15min":0.67100000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.69399999999999995,"15min":0.69799999999999995},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.90400000000000003,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.56000000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64800000000000002,"15min":0.64700000000000002},"min":{"1min":0.46700000000000003,"5min":0.38,"15min":0.38},"max":{"1min":0.84199999999999997,"5min":1.012,"15min":1.012},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.71299999999999997,"15min":0.71199999999999997},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.0980000000000001,"5min":1.175,"15min":1.175},"last":0.68799999999999994}]},{"osd":3,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.66900000000000004,"15min":0.66500000000000004},"min":{"1min":0.502,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.88200000000000001,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0569999999999999,"5min":1.103,"15min":1.103},"last":0.74099999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.60799999999999998,"15min":0.61599999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.69199999999999995,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.65700000000000003,"15min":0.66200000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.622}]},{"osd":5,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.69499999999999995,"15min":0.69899999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.63,"5min":0.70999999999999996,"15min":0.72299999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.89500000000000002,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.70699999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.78800000000000003,"15min":0.80000000000000004},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.89500000000000002,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72499999999999998,"15min":0.73099999999999998},"min":{"1min":0.52500000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.76600000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509474,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31668,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340172,"statfs":{"total":96636764160,"available":96604336128,"internally_reserved":0,"allocated":495616,"data_stored":198807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.432,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.67200000000000004,"15min":0.68899999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84899999999999998,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.755}]},{"osd":1,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.70299999999999996,"15min":0.72399999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.82699999999999996,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.74399999999999999,"15min":0.754},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":0.59099999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.82099999999999995,"15min":0.84399999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.825,"5min":2.8700000000000001,"15min":2.8700000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.755,"15min":0.78300000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.82399999999999995,"5min":1.857,"15min":1.857},"last":0.78600000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.77400000000000002,"15min":0.80100000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.92100000000000004,"5min":2.012,"15min":2.012},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.79900000000000004,"15min":0.82499999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":2.532,"15min":2.532},"last":0.66200000000000003}]},{"osd":4,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68300000000000005,"15min":0.69299999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.726,"5min":1.726,"15min":1.726},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.69399999999999995,"15min":0.71799999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.753,"5min":1.145,"15min":1.145},"last":0.81799999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.64500000000000002,"15min":0.65800000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.81299999999999994,"5min":1.121,"15min":1.121},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.77400000000000002,"15min":0.78100000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"last":0.91400000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.80100000000000005,"15min":0.82499999999999996},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.96499999999999997,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.874},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.78000000000000003,"15min":0.80200000000000005},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.081,"5min":1.165,"15min":1.165},"last":0.80500000000000005}]}]},{"osd":4,"up_from":15,"seq":64424509474,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31688,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340152,"statfs":{"total":96636764160,"available":96604315648,"internally_reserved":0,"allocated":520192,"data_stored":207222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.628,"15min":0.63},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.63200000000000001,"15min":0.64100000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.60299999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64200000000000002,"15min":0.64900000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.77300000000000002,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.628,"5min":0.64000000000000001,"15min":0.64200000000000002},"min":{"1min":0.40899999999999997,"5min":0.377,"15min":0.377},"max":{"1min":0.88,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.66200000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.626,"15min":0.628},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.69099999999999995,"15min":0.69499999999999995},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.81999999999999995,"5min":0.879,"15min":0.879},"last":0.80600000000000005}]},{"osd":3,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68899999999999995,"15min":0.68999999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.96699999999999997,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.72499999999999998,"15min":0.72599999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.93200000000000005,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.79000000000000004}]},{"osd":5,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40200000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65100000000000002,"15min":0.65000000000000002},"min":{"1min":0.40699999999999997,"5min":0.377,"15min":0.377},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.71699999999999997}]},{"osd":6,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.68200000000000005,"15min":0.68000000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.85099999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.46899999999999997,"5min":0.436,"15min":0.436},"max":{"1min":1.0069999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.81799999999999995}]}]},{"osd":3,"up_from":15,"seq":64424509473,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":520192,"data_stored":206248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.28499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.72399999999999998,"5min":0.752,"15min":0.752},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.499,"15min":0.47999999999999998},"min":{"1min":0.34000000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":2.6509999999999998,"5min":2.6509999999999998,"15min":2.6509999999999998},"last":0.38500000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70999999999999996,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.498},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.53700000000000003,"15min":0.53600000000000003},"min":{"1min":0.42499999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83399999999999996,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.52500000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.55800000000000005,"15min":0.56000000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.58599999999999997,"15min":0.57199999999999995},"min":{"1min":0.42899999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.56399999999999995}]},{"osd":4,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.79400000000000004,"15min":0.80300000000000005},"min":{"1min":0.50800000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.119,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.876},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.77300000000000002,"15min":0.77900000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.97899999999999998,"5min":1.341,"15min":1.341},"last":0.65600000000000003}]},{"osd":5,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.59199999999999997,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.72399999999999998,"15min":0.72199999999999998},"min":{"1min":0.53300000000000003,"5min":0.498,"15min":0.498},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":0.73299999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.78300000000000003,"15min":0.78500000000000003},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.77600000000000002,"15min":0.76600000000000001},"min":{"1min":0.628,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":2.0880000000000001,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":0.80200000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.78800000000000003,"15min":0.78100000000000003},"min":{"1min":0.58999999999999997,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.81299999999999994,"15min":0.81999999999999995},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.1259999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.78600000000000003}]}]},{"osd":2,"up_from":15,"seq":64424509473,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31668,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340172,"statfs":{"total":96636764160,"available":96604336128,"internally_reserved":0,"allocated":495616,"data_stored":198807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.45500000000000002,"15min":0.46100000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.75,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.441,"5min":0.46100000000000002,"15min":0.46500000000000002},"min":{"1min":0.29899999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.64700000000000002,"5min":0.753,"15min":0.753},"last":0.46400000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54300000000000004,"15min":0.54500000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.67600000000000005,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.314,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.65100000000000002,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.436}]},{"osd":3,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.57799999999999996,"15min":0.57299999999999995},"min":{"1min":0.375,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.55600000000000005,"15min":0.55300000000000005},"min":{"1min":0.40500000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71499999999999997,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.42099999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.75700000000000001,"15min":0.76700000000000002},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.038,"5min":1.466,"15min":1.466},"last":0.626}]},{"osd":5,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.79400000000000004,"15min":0.79700000000000004},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.609,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.747,"5min":0.78300000000000003,"15min":0.78900000000000003},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.64600000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.76800000000000002,"15min":0.77800000000000002},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0489999999999999,"5min":1.516,"15min":1.516},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.78600000000000003,"15min":0.79600000000000004},"min":{"1min":0.52800000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0820000000000001,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.56100000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.79500000000000004,"15min":0.80100000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.1659999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.81799999999999995,"15min":0.82599999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.69299999999999995}]}]},{"osd":1,"up_from":15,"seq":64424509475,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31668,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340172,"statfs":{"total":96636764160,"available":96604336128,"internally_reserved":0,"allocated":495616,"data_stored":198807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.49099999999999999,"15min":0.504},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.627,"5min":1.847,"15min":1.847},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.443,"5min":0.47899999999999998,"15min":0.48499999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.63500000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.54300000000000004}]},{"osd":2,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.53400000000000003,"15min":0.54400000000000004},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.77500000000000002,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.624},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.70299999999999996,"15min":0.73299999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.90100000000000002,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.46700000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.63100000000000001,"15min":0.64500000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80700000000000005,"5min":1.871,"15min":1.871},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.58099999999999996,"15min":0.59099999999999997},"min":{"1min":0.35299999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.72599999999999998,"5min":1.917,"15min":1.917},"last":0.496}]},{"osd":4,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.70299999999999996,"15min":0.69599999999999995},"min":{"1min":0.54800000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.161,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.77100000000000002,"15min":0.77700000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.147,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.73799999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.76900000000000002,"15min":0.77100000000000002},"min":{"1min":0.47299999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.214,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.53100000000000003,"5min":0.505,"15min":0.505},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.97899999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.753,"15min":0.755},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.1359999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.81599999999999995,"15min":0.82199999999999995},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.95699999999999996}]},{"osd":7,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78700000000000003,"15min":0.78500000000000003},"min":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.79900000000000004,"15min":0.80300000000000005},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.095,"5min":1.347,"15min":1.347},"last":0.90500000000000003}]}]},{"osd":5,"up_from":14,"seq":60129542178,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":483328,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":0.871,"15min":0.77500000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":21.385000000000002,"5min":21.385000000000002,"15min":21.385000000000002},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.202,"5min":0.878,"15min":0.82399999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":8.2029999999999994,"5min":8.2029999999999994,"15min":8.2029999999999994},"last":0.73599999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66400000000000003,"15min":0.67200000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":1.4670000000000001,"5min":0.84099999999999997,"15min":0.73699999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":21.5,"5min":21.5,"15min":21.5},"last":0.80100000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":0.81599999999999995,"15min":0.745},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":8.2439999999999998,"5min":8.2439999999999998,"15min":8.2439999999999998},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.748,"15min":0.77000000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.85099999999999998,"5min":1.663,"15min":1.663},"last":0.71999999999999997}]},{"osd":3,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":0.95299999999999996,"15min":0.85699999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":21.457000000000001,"5min":21.457000000000001,"15min":21.457000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.1919999999999999,"5min":0.92100000000000004,"15min":0.876},"min":{"1min":0.50900000000000001,"5min":0.502,"15min":0.502},"max":{"1min":8.2750000000000004,"5min":8.2750000000000004,"15min":8.2750000000000004},"last":0.70499999999999996}]},{"osd":4,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":0.82599999999999996,"15min":0.76600000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":9.6679999999999993,"5min":9.6679999999999993,"15min":9.6679999999999993},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.71299999999999997,"15min":0.73499999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.76300000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.60099999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":0.877,"15min":0.78400000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":21.535,"5min":21.535,"15min":21.535},"last":0.54100000000000004},{"interface":"front","average":{"1min":1.1870000000000001,"5min":0.88300000000000001,"15min":0.83199999999999996},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":8.298,"5min":8.298,"15min":8.298},"last":0.66700000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":0.83999999999999997,"15min":0.81399999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":9.2240000000000002,"5min":9.2240000000000002,"15min":9.2240000000000002},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.5249999999999999,"5min":0.97099999999999997,"15min":0.879},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":21.574000000000002,"5min":21.574000000000002,"15min":21.574000000000002},"last":0.748}]}]},{"osd":0,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32240,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339600,"statfs":{"total":96636764160,"available":96603750400,"internally_reserved":0,"allocated":1077248,"data_stored":782880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.46300000000000002,"15min":0.46500000000000002},"min":{"1min":0.28999999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.746,"5min":1.462,"15min":1.462},"last":0.501},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55400000000000005,"15min":0.55200000000000005},"min":{"1min":0.34399999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.995,"5min":1.373,"15min":1.373},"last":0.39000000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.371,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.89600000000000002,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.441},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.48699999999999999,"15min":0.48099999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.252,"5min":1.609,"15min":1.609},"last":0.54500000000000004}]},{"osd":3,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.371,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.321,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60899999999999999,"15min":0.60999999999999999},"min":{"1min":0.45000000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.94599999999999995,"5min":1.325,"15min":1.325},"last":0.52700000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.76700000000000002,"15min":0.77700000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.96099999999999997,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.80900000000000005,"15min":0.82599999999999996},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.93300000000000005,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.71099999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.78500000000000003,"15min":0.79500000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0109999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.83099999999999996,"15min":0.84699999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":0.63300000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.79400000000000004,"15min":0.79800000000000004},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.107,"5min":1.764,"15min":1.764},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.83099999999999996,"15min":0.84699999999999998},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.97499999999999998,"5min":1.806,"15min":1.806},"last":0.72699999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.82199999999999995,"15min":0.82999999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.04,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.84899999999999998,"15min":0.85999999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.0429999999999999,"5min":1.722,"15min":1.722},"last":0.67100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:07:24.190 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:07:24.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.360+0000 7f56f5c00640 1 Processor -- start 2024-01-15T18:07:24.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.360+0000 7f56f5c00640 1 -- start start 2024-01-15T18:07:24.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.364+0000 7f56f5c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f56f0160b10 0x7f56f00afe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:24.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.364+0000 7f56f5c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f56f00cd7a0 0x7f56f00cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:24.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.364+0000 7f56f5c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56f00ce160 0x7f56f006b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:24.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.364+0000 7f56f5c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56f00ce5f0 con 0x7f56f0160b10 2024-01-15T18:07:24.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.364+0000 7f56f5c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56f00d05b0 con 0x7f56f00cd7a0 2024-01-15T18:07:24.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.364+0000 7f56f5c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f00ce7d0 con 0x7f56f00ce160 2024-01-15T18:07:24.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.364+0000 7f56eea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f56f0160b10 0x7f56f00afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:24.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.364+0000 7f56eea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f56f0160b10 0x7f56f00afe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50706/0 (socket says 172.21.15.27:50706) 2024-01-15T18:07:24.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.364+0000 7f56eea00640 1 -- 172.21.15.27:0/1899297963 learned_addr learned my addr 172.21.15.27:0/1899297963 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:24.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.364+0000 7f56ee000640 1 --2- 172.21.15.27:0/1899297963 >> v2:172.21.15.93:3300/0 conn(0x7f56f00cd7a0 0x7f56f00cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:24.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.364+0000 7f56ef400640 1 --2- 172.21.15.27:0/1899297963 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56f00ce160 0x7f56f006b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:24.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.364+0000 7f56eea00640 1 -- 172.21.15.27:0/1899297963 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56f00ce160 msgr2=0x7f56f006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:24.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.364+0000 7f56eea00640 1 --2- 172.21.15.27:0/1899297963 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56f00ce160 0x7f56f006b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.364+0000 7f56eea00640 1 -- 172.21.15.27:0/1899297963 >> v2:172.21.15.93:3300/0 conn(0x7f56f00cd7a0 msgr2=0x7f56f00cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:24.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.364+0000 7f56eea00640 1 --2- 172.21.15.27:0/1899297963 >> v2:172.21.15.93:3300/0 conn(0x7f56f00cd7a0 0x7f56f00cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.364+0000 7f56eea00640 1 -- 172.21.15.27:0/1899297963 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f00cfc80 con 0x7f56f0160b10 2024-01-15T18:07:24.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.364+0000 7f56ee000640 1 --2- 172.21.15.27:0/1899297963 >> v2:172.21.15.93:3300/0 conn(0x7f56f00cd7a0 0x7f56f00cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:07:24.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.368+0000 7f56eea00640 1 --2- 172.21.15.27:0/1899297963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f0160b10 0x7f56f00afe70 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f56f0067d20 tx=0x7f56e001c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f1719930ec5a9bf server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:24.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.368+0000 7f56ed600640 1 -- 172.21.15.27:0/1899297963 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e00ab600 con 0x7f56f0160b10 2024-01-15T18:07:24.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.368+0000 7f56ed600640 1 -- 172.21.15.27:0/1899297963 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e00a7050 con 0x7f56f0160b10 2024-01-15T18:07:24.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.368+0000 7f56ed600640 1 -- 172.21.15.27:0/1899297963 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e00ab980 con 0x7f56f0160b10 2024-01-15T18:07:24.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.368+0000 7f56f5c00640 1 -- 172.21.15.27:0/1899297963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f0160b10 msgr2=0x7f56f00afe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:24.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.368+0000 7f56f5c00640 1 --2- 172.21.15.27:0/1899297963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f0160b10 0x7f56f00afe70 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f56f0067d20 tx=0x7f56e001c180 comp rx=0 tx=0).stop 2024-01-15T18:07:24.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.368+0000 7f56f5c00640 1 -- 172.21.15.27:0/1899297963 shutdown_connections 2024-01-15T18:07:24.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.368+0000 7f56f5c00640 1 --2- 172.21.15.27:0/1899297963 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56f00ce160 0x7f56f006b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.368+0000 7f56f5c00640 1 --2- 172.21.15.27:0/1899297963 >> v2:172.21.15.93:3300/0 conn(0x7f56f00cd7a0 0x7f56f00cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.368+0000 7f56f5c00640 1 --2- 172.21.15.27:0/1899297963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f0160b10 0x7f56f00afe70 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.368+0000 7f56f5c00640 1 -- 172.21.15.27:0/1899297963 >> 172.21.15.27:0/1899297963 conn(0x7f56f00d0f20 msgr2=0x7f56f00d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:24.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.368+0000 7f56f5c00640 1 -- 172.21.15.27:0/1899297963 shutdown_connections 2024-01-15T18:07:24.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.368+0000 7f56f5c00640 1 -- 172.21.15.27:0/1899297963 wait complete. 2024-01-15T18:07:24.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.368+0000 7f56f5c00640 1 Processor -- start 2024-01-15T18:07:24.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.368+0000 7f56f5c00640 1 -- start start 2024-01-15T18:07:24.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.372+0000 7f56f5c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56f00cd7a0 0x7f56f0147150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:24.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.372+0000 7f56f5c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f00ce160 0x7f56f0155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:24.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.372+0000 7f56f5c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56f0160b10 0x7f56f00c5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:24.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.372+0000 7f56f5c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f00c7ff0 con 0x7f56f00ce160 2024-01-15T18:07:24.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.372+0000 7f56f5c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f00c8240 con 0x7f56f00cd7a0 2024-01-15T18:07:24.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.372+0000 7f56f5c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f00c8520 con 0x7f56f0160b10 2024-01-15T18:07:24.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.372+0000 7f56ee000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f00ce160 0x7f56f0155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:24.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.372+0000 7f56ee000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f00ce160 0x7f56f0155e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50718/0 (socket says 172.21.15.27:50718) 2024-01-15T18:07:24.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.372+0000 7f56ee000640 1 -- 172.21.15.27:0/4172357728 learned_addr learned my addr 172.21.15.27:0/4172357728 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:24.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.372+0000 7f56eea00640 1 --2- 172.21.15.27:0/4172357728 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56f00cd7a0 0x7f56f0147150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:24.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.372+0000 7f56ef400640 1 --2- 172.21.15.27:0/4172357728 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56f0160b10 0x7f56f00c5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:24.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.372+0000 7f56ee000640 1 -- 172.21.15.27:0/4172357728 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56f0160b10 msgr2=0x7f56f00c5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:24.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.372+0000 7f56ee000640 1 --2- 172.21.15.27:0/4172357728 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56f0160b10 0x7f56f00c5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.372+0000 7f56ee000640 1 -- 172.21.15.27:0/4172357728 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56f00cd7a0 msgr2=0x7f56f0147150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:24.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.372+0000 7f56ee000640 1 --2- 172.21.15.27:0/4172357728 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56f00cd7a0 0x7f56f0147150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.372+0000 7f56ee000640 1 -- 172.21.15.27:0/4172357728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e00094d0 con 0x7f56f00ce160 2024-01-15T18:07:24.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.372+0000 7f56ee000640 1 --2- 172.21.15.27:0/4172357728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f00ce160 0x7f56f0155e40 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f56d800e810 tx=0x7f56d800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:24.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.372+0000 7f56e6a00640 1 -- 172.21.15.27:0/4172357728 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d8004030 con 0x7f56f00ce160 2024-01-15T18:07:24.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.372+0000 7f56e6a00640 1 -- 172.21.15.27:0/4172357728 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d8004190 con 0x7f56f00ce160 2024-01-15T18:07:24.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.372+0000 7f56f5c00640 1 -- 172.21.15.27:0/4172357728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56f00c8800 con 0x7f56f00ce160 2024-01-15T18:07:24.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.372+0000 7f56e6a00640 1 -- 172.21.15.27:0/4172357728 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d8004520 con 0x7f56f00ce160 2024-01-15T18:07:24.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.372+0000 7f56f5c00640 1 -- 172.21.15.27:0/4172357728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56f015d210 con 0x7f56f00ce160 2024-01-15T18:07:24.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.372+0000 7f56f5c00640 1 -- 172.21.15.27:0/4172357728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56f004e850 con 0x7f56f00ce160 2024-01-15T18:07:24.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.376+0000 7f56e6a00640 1 -- 172.21.15.27:0/4172357728 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f56d800c6d0 con 0x7f56f00ce160 2024-01-15T18:07:24.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.376+0000 7f56e6a00640 1 --2- 172.21.15.27:0/4172357728 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f56c8066b30 0x7f56c8068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:24.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.376+0000 7f56e6a00640 1 -- 172.21.15.27:0/4172357728 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f56d80844d0 con 0x7f56f00ce160 2024-01-15T18:07:24.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.376+0000 7f56eea00640 1 --2- 172.21.15.27:0/4172357728 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f56c8066b30 0x7f56c8068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:24.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.376+0000 7f56eea00640 1 --2- 172.21.15.27:0/4172357728 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f56c8066b30 0x7f56c8068ff0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f56e00ad000 tx=0x7f56e00a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:24.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.380+0000 7f56e6a00640 1 -- 172.21.15.27:0/4172357728 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f56d8051320 con 0x7f56f00ce160 2024-01-15T18:07:24.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.532+0000 7f56f5c00640 1 -- 172.21.15.27:0/4172357728 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f56f00d0cc0 con 0x7f56c8066b30 2024-01-15T18:07:24.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.540+0000 7f56e6a00640 1 -- 172.21.15.27:0/4172357728 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287094 (secure 0 0 0) 0x7f56e00ab580 con 0x7f56c8066b30 2024-01-15T18:07:24.549 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:07:24.550 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:07:24.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.544+0000 7f56f5c00640 1 -- 172.21.15.27:0/4172357728 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f56c8066b30 msgr2=0x7f56c8068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:24.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.544+0000 7f56f5c00640 1 --2- 172.21.15.27:0/4172357728 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f56c8066b30 0x7f56c8068ff0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f56e00ad000 tx=0x7f56e00a6000 comp rx=0 tx=0).stop 2024-01-15T18:07:24.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.544+0000 7f56f5c00640 1 -- 172.21.15.27:0/4172357728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f00ce160 msgr2=0x7f56f0155e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:24.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.544+0000 7f56f5c00640 1 --2- 172.21.15.27:0/4172357728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f00ce160 0x7f56f0155e40 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f56d800e810 tx=0x7f56d800ecd0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.548+0000 7f56f5c00640 1 -- 172.21.15.27:0/4172357728 shutdown_connections 2024-01-15T18:07:24.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.548+0000 7f56f5c00640 1 --2- 172.21.15.27:0/4172357728 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f56c8066b30 0x7f56c8068ff0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.548+0000 7f56f5c00640 1 --2- 172.21.15.27:0/4172357728 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56f0160b10 0x7f56f00c5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.548+0000 7f56f5c00640 1 --2- 172.21.15.27:0/4172357728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f00ce160 0x7f56f0155e40 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.548+0000 7f56f5c00640 1 --2- 172.21.15.27:0/4172357728 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56f00cd7a0 0x7f56f0147150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.548+0000 7f56f5c00640 1 -- 172.21.15.27:0/4172357728 >> 172.21.15.27:0/4172357728 conn(0x7f56f00d0f20 msgr2=0x7f56f00d1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:24.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.548+0000 7f56f5c00640 1 -- 172.21.15.27:0/4172357728 shutdown_connections 2024-01-15T18:07:24.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.548+0000 7f56f5c00640 1 -- 172.21.15.27:0/4172357728 wait complete. 2024-01-15T18:07:24.571 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-01-15T18:07:24.171279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592708,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":187,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":47,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254540,"kb_used_data":5056,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720180,"statfs":{"total":773094113280,"available":772833464320,"internally_reserved":0,"allocated":5177344,"data_stored":2774458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2340,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001241"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.094792+0000","last_change":"2024-01-15T18:07:20.088581+0000","last_active":"2024-01-15T18:07:20.094792+0000","last_peered":"2024-01-15T18:07:20.094792+0000","last_clean":"2024-01-15T18:07:20.094792+0000","last_became_active":"2024-01-15T18:07:20.088247+0000","last_became_peered":"2024-01-15T18:07:20.088247+0000","last_unstale":"2024-01-15T18:07:20.094792+0000","last_undegraded":"2024-01-15T18:07:20.094792+0000","last_fullsized":"2024-01-15T18:07:20.094792+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:04:14.022333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.067098+0000","last_change":"2024-01-15T18:07:19.067098+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.067098+0000","last_undegraded":"2024-01-15T18:07:19.067098+0000","last_fullsized":"2024-01-15T18:07:19.067098+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.099935+0000","last_change":"2024-01-15T18:07:20.094383+0000","last_active":"2024-01-15T18:07:20.099935+0000","last_peered":"2024-01-15T18:07:20.099935+0000","last_clean":"2024-01-15T18:07:20.099935+0000","last_became_active":"2024-01-15T18:07:20.094114+0000","last_became_peered":"2024-01-15T18:07:20.094114+0000","last_unstale":"2024-01-15T18:07:20.099935+0000","last_undegraded":"2024-01-15T18:07:20.099935+0000","last_fullsized":"2024-01-15T18:07:20.099935+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:07:17.293929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.068614+0000","last_change":"2024-01-15T18:07:19.068614+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.068614+0000","last_undegraded":"2024-01-15T18:07:19.068614+0000","last_fullsized":"2024-01-15T18:07:19.068614+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.065311+0000","last_change":"2024-01-15T18:07:19.065311+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.065311+0000","last_undegraded":"2024-01-15T18:07:19.065311+0000","last_fullsized":"2024-01-15T18:07:19.065311+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100427+0000","last_change":"2024-01-15T18:07:20.090463+0000","last_active":"2024-01-15T18:07:20.100427+0000","last_peered":"2024-01-15T18:07:20.100427+0000","last_clean":"2024-01-15T18:07:20.100427+0000","last_became_active":"2024-01-15T18:07:20.090196+0000","last_became_peered":"2024-01-15T18:07:20.090196+0000","last_unstale":"2024-01-15T18:07:20.100427+0000","last_undegraded":"2024-01-15T18:07:20.100427+0000","last_fullsized":"2024-01-15T18:07:20.100427+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:03:21.190309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.106586+0000","last_change":"2024-01-15T18:07:20.102595+0000","last_active":"2024-01-15T18:07:20.106586+0000","last_peered":"2024-01-15T18:07:20.106586+0000","last_clean":"2024-01-15T18:07:20.106586+0000","last_became_active":"2024-01-15T18:07:20.102222+0000","last_became_peered":"2024-01-15T18:07:20.102222+0000","last_unstale":"2024-01-15T18:07:20.106586+0000","last_undegraded":"2024-01-15T18:07:20.106586+0000","last_fullsized":"2024-01-15T18:07:20.106586+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:51:04.157158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.074897+0000","last_change":"2024-01-15T18:07:19.074897+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.074897+0000","last_undegraded":"2024-01-15T18:07:19.074897+0000","last_fullsized":"2024-01-15T18:07:19.074897+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.077258+0000","last_change":"2024-01-15T18:07:19.077258+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.077258+0000","last_undegraded":"2024-01-15T18:07:19.077258+0000","last_fullsized":"2024-01-15T18:07:19.077258+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.070141+0000","last_change":"2024-01-15T18:07:19.070141+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.070141+0000","last_undegraded":"2024-01-15T18:07:19.070141+0000","last_fullsized":"2024-01-15T18:07:19.070141+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.074684+0000","last_change":"2024-01-15T18:07:19.074684+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.074684+0000","last_undegraded":"2024-01-15T18:07:19.074684+0000","last_fullsized":"2024-01-15T18:07:19.074684+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.072234+0000","last_change":"2024-01-15T18:07:19.072234+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.072234+0000","last_undegraded":"2024-01-15T18:07:19.072234+0000","last_fullsized":"2024-01-15T18:07:19.072234+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.106908+0000","last_change":"2024-01-15T18:07:20.093608+0000","last_active":"2024-01-15T18:07:20.106908+0000","last_peered":"2024-01-15T18:07:20.106908+0000","last_clean":"2024-01-15T18:07:20.106908+0000","last_became_active":"2024-01-15T18:07:20.091735+0000","last_became_peered":"2024-01-15T18:07:20.091735+0000","last_unstale":"2024-01-15T18:07:20.106908+0000","last_undegraded":"2024-01-15T18:07:20.106908+0000","last_fullsized":"2024-01-15T18:07:20.106908+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:34.221280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.068436+0000","last_change":"2024-01-15T18:07:19.068436+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.068436+0000","last_undegraded":"2024-01-15T18:07:19.068436+0000","last_fullsized":"2024-01-15T18:07:19.068436+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.105807+0000","last_change":"2024-01-15T18:07:20.096359+0000","last_active":"2024-01-15T18:07:20.105807+0000","last_peered":"2024-01-15T18:07:20.105807+0000","last_clean":"2024-01-15T18:07:20.105807+0000","last_became_active":"2024-01-15T18:07:20.096018+0000","last_became_peered":"2024-01-15T18:07:20.096018+0000","last_unstale":"2024-01-15T18:07:20.105807+0000","last_undegraded":"2024-01-15T18:07:20.105807+0000","last_fullsized":"2024-01-15T18:07:20.105807+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:35:28.043293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100722+0000","last_change":"2024-01-15T18:07:20.097218+0000","last_active":"2024-01-15T18:07:20.100722+0000","last_peered":"2024-01-15T18:07:20.100722+0000","last_clean":"2024-01-15T18:07:20.100722+0000","last_became_active":"2024-01-15T18:07:20.096805+0000","last_became_peered":"2024-01-15T18:07:20.096805+0000","last_unstale":"2024-01-15T18:07:20.100722+0000","last_undegraded":"2024-01-15T18:07:20.100722+0000","last_fullsized":"2024-01-15T18:07:20.100722+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:34.221280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097051+0000","last_change":"2024-01-15T18:07:20.084911+0000","last_active":"2024-01-15T18:07:20.097051+0000","last_peered":"2024-01-15T18:07:20.097051+0000","last_clean":"2024-01-15T18:07:20.097051+0000","last_became_active":"2024-01-15T18:07:20.084594+0000","last_became_peered":"2024-01-15T18:07:20.084594+0000","last_unstale":"2024-01-15T18:07:20.097051+0000","last_undegraded":"2024-01-15T18:07:20.097051+0000","last_fullsized":"2024-01-15T18:07:20.097051+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:38:53.126797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100540+0000","last_change":"2024-01-15T18:07:20.097350+0000","last_active":"2024-01-15T18:07:20.100540+0000","last_peered":"2024-01-15T18:07:20.100540+0000","last_clean":"2024-01-15T18:07:20.100540+0000","last_became_active":"2024-01-15T18:07:20.095556+0000","last_became_peered":"2024-01-15T18:07:20.095556+0000","last_unstale":"2024-01-15T18:07:20.100540+0000","last_undegraded":"2024-01-15T18:07:20.100540+0000","last_fullsized":"2024-01-15T18:07:20.100540+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:01:01.613225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.103149+0000","last_change":"2024-01-15T18:07:20.098939+0000","last_active":"2024-01-15T18:07:20.103149+0000","last_peered":"2024-01-15T18:07:20.103149+0000","last_clean":"2024-01-15T18:07:20.103149+0000","last_became_active":"2024-01-15T18:07:20.098598+0000","last_became_peered":"2024-01-15T18:07:20.098598+0000","last_unstale":"2024-01-15T18:07:20.103149+0000","last_undegraded":"2024-01-15T18:07:20.103149+0000","last_fullsized":"2024-01-15T18:07:20.103149+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:26.049225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.107947+0000","last_change":"2024-01-15T18:07:20.095741+0000","last_active":"2024-01-15T18:07:20.107947+0000","last_peered":"2024-01-15T18:07:20.107947+0000","last_clean":"2024-01-15T18:07:20.107947+0000","last_became_active":"2024-01-15T18:07:20.095417+0000","last_became_peered":"2024-01-15T18:07:20.095417+0000","last_unstale":"2024-01-15T18:07:20.107947+0000","last_undegraded":"2024-01-15T18:07:20.107947+0000","last_fullsized":"2024-01-15T18:07:20.107947+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:32:43.186714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.105539+0000","last_change":"2024-01-15T18:07:20.094321+0000","last_active":"2024-01-15T18:07:20.105539+0000","last_peered":"2024-01-15T18:07:20.105539+0000","last_clean":"2024-01-15T18:07:20.105539+0000","last_became_active":"2024-01-15T18:07:20.094011+0000","last_became_peered":"2024-01-15T18:07:20.094011+0000","last_unstale":"2024-01-15T18:07:20.105539+0000","last_undegraded":"2024-01-15T18:07:20.105539+0000","last_fullsized":"2024-01-15T18:07:20.105539+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:48:24.482600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.067198+0000","last_change":"2024-01-15T18:07:19.067198+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.067198+0000","last_undegraded":"2024-01-15T18:07:19.067198+0000","last_fullsized":"2024-01-15T18:07:19.067198+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098234+0000","last_change":"2024-01-15T18:07:20.087728+0000","last_active":"2024-01-15T18:07:20.098234+0000","last_peered":"2024-01-15T18:07:20.098234+0000","last_clean":"2024-01-15T18:07:20.098234+0000","last_became_active":"2024-01-15T18:07:20.087118+0000","last_became_peered":"2024-01-15T18:07:20.087118+0000","last_unstale":"2024-01-15T18:07:20.098234+0000","last_undegraded":"2024-01-15T18:07:20.098234+0000","last_fullsized":"2024-01-15T18:07:20.098234+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:32:43.186714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100694+0000","last_change":"2024-01-15T18:07:20.092361+0000","last_active":"2024-01-15T18:07:20.100694+0000","last_peered":"2024-01-15T18:07:20.100694+0000","last_clean":"2024-01-15T18:07:20.100694+0000","last_became_active":"2024-01-15T18:07:20.091927+0000","last_became_peered":"2024-01-15T18:07:20.091927+0000","last_unstale":"2024-01-15T18:07:20.100694+0000","last_undegraded":"2024-01-15T18:07:20.100694+0000","last_fullsized":"2024-01-15T18:07:20.100694+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:55:19.615935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.106512+0000","last_change":"2024-01-15T18:07:20.102013+0000","last_active":"2024-01-15T18:07:20.106512+0000","last_peered":"2024-01-15T18:07:20.106512+0000","last_clean":"2024-01-15T18:07:20.106512+0000","last_became_active":"2024-01-15T18:07:20.101697+0000","last_became_peered":"2024-01-15T18:07:20.101697+0000","last_unstale":"2024-01-15T18:07:20.106512+0000","last_undegraded":"2024-01-15T18:07:20.106512+0000","last_fullsized":"2024-01-15T18:07:20.106512+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:32:14.924018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.103301+0000","last_change":"2024-01-15T18:07:20.086187+0000","last_active":"2024-01-15T18:07:20.103301+0000","last_peered":"2024-01-15T18:07:20.103301+0000","last_clean":"2024-01-15T18:07:20.103301+0000","last_became_active":"2024-01-15T18:07:20.085891+0000","last_became_peered":"2024-01-15T18:07:20.085891+0000","last_unstale":"2024-01-15T18:07:20.103301+0000","last_undegraded":"2024-01-15T18:07:20.103301+0000","last_fullsized":"2024-01-15T18:07:20.103301+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:54:09.185168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097020+0000","last_change":"2024-01-15T18:07:20.086602+0000","last_active":"2024-01-15T18:07:20.097020+0000","last_peered":"2024-01-15T18:07:20.097020+0000","last_clean":"2024-01-15T18:07:20.097020+0000","last_became_active":"2024-01-15T18:07:20.086238+0000","last_became_peered":"2024-01-15T18:07:20.086238+0000","last_unstale":"2024-01-15T18:07:20.097020+0000","last_undegraded":"2024-01-15T18:07:20.097020+0000","last_fullsized":"2024-01-15T18:07:20.097020+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:20:04.640555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.071723+0000","last_change":"2024-01-15T18:07:19.071723+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.071723+0000","last_undegraded":"2024-01-15T18:07:19.071723+0000","last_fullsized":"2024-01-15T18:07:19.071723+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.103004+0000","last_change":"2024-01-15T18:07:20.097835+0000","last_active":"2024-01-15T18:07:20.103004+0000","last_peered":"2024-01-15T18:07:20.103004+0000","last_clean":"2024-01-15T18:07:20.103004+0000","last_became_active":"2024-01-15T18:07:20.097477+0000","last_became_peered":"2024-01-15T18:07:20.097477+0000","last_unstale":"2024-01-15T18:07:20.103004+0000","last_undegraded":"2024-01-15T18:07:20.103004+0000","last_fullsized":"2024-01-15T18:07:20.103004+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:36:51.036352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097114+0000","last_change":"2024-01-15T18:07:20.083756+0000","last_active":"2024-01-15T18:07:20.097114+0000","last_peered":"2024-01-15T18:07:20.097114+0000","last_clean":"2024-01-15T18:07:20.097114+0000","last_became_active":"2024-01-15T18:07:20.079072+0000","last_became_peered":"2024-01-15T18:07:20.079072+0000","last_unstale":"2024-01-15T18:07:20.097114+0000","last_undegraded":"2024-01-15T18:07:20.097114+0000","last_fullsized":"2024-01-15T18:07:20.097114+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:55:26.194143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.105788+0000","last_change":"2024-01-15T18:07:20.096208+0000","last_active":"2024-01-15T18:07:20.105788+0000","last_peered":"2024-01-15T18:07:20.105788+0000","last_clean":"2024-01-15T18:07:20.105788+0000","last_became_active":"2024-01-15T18:07:20.095900+0000","last_became_peered":"2024-01-15T18:07:20.095900+0000","last_unstale":"2024-01-15T18:07:20.105788+0000","last_undegraded":"2024-01-15T18:07:20.105788+0000","last_fullsized":"2024-01-15T18:07:20.105788+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:50:12.413904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.106884+0000","last_change":"2024-01-15T18:07:20.091218+0000","last_active":"2024-01-15T18:07:20.106884+0000","last_peered":"2024-01-15T18:07:20.106884+0000","last_clean":"2024-01-15T18:07:20.106884+0000","last_became_active":"2024-01-15T18:07:20.090896+0000","last_became_peered":"2024-01-15T18:07:20.090896+0000","last_unstale":"2024-01-15T18:07:20.106884+0000","last_undegraded":"2024-01-15T18:07:20.106884+0000","last_fullsized":"2024-01-15T18:07:20.106884+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:54:56.923677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098011+0000","last_change":"2024-01-15T18:07:20.087031+0000","last_active":"2024-01-15T18:07:20.098011+0000","last_peered":"2024-01-15T18:07:20.098011+0000","last_clean":"2024-01-15T18:07:20.098011+0000","last_became_active":"2024-01-15T18:07:20.086158+0000","last_became_peered":"2024-01-15T18:07:20.086158+0000","last_unstale":"2024-01-15T18:07:20.098011+0000","last_undegraded":"2024-01-15T18:07:20.098011+0000","last_fullsized":"2024-01-15T18:07:20.098011+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:55:26.194143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098945+0000","last_change":"2024-01-15T18:07:19.085120+0000","last_active":"2024-01-15T18:07:20.098945+0000","last_peered":"2024-01-15T18:07:20.098945+0000","last_clean":"2024-01-15T18:07:20.098945+0000","last_became_active":"2024-01-15T18:07:19.084785+0000","last_became_peered":"2024-01-15T18:07:19.084785+0000","last_unstale":"2024-01-15T18:07:20.098945+0000","last_undegraded":"2024-01-15T18:07:20.098945+0000","last_fullsized":"2024-01-15T18:07:20.098945+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:50:51.007855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100675+0000","last_change":"2024-01-15T18:07:20.094786+0000","last_active":"2024-01-15T18:07:20.100675+0000","last_peered":"2024-01-15T18:07:20.100675+0000","last_clean":"2024-01-15T18:07:20.100675+0000","last_became_active":"2024-01-15T18:07:20.094523+0000","last_became_peered":"2024-01-15T18:07:20.094523+0000","last_unstale":"2024-01-15T18:07:20.100675+0000","last_undegraded":"2024-01-15T18:07:20.100675+0000","last_fullsized":"2024-01-15T18:07:20.100675+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:59:01.979409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.096749+0000","last_change":"2024-01-15T18:07:19.084713+0000","last_active":"2024-01-15T18:07:20.096749+0000","last_peered":"2024-01-15T18:07:20.096749+0000","last_clean":"2024-01-15T18:07:20.096749+0000","last_became_active":"2024-01-15T18:07:19.084414+0000","last_became_peered":"2024-01-15T18:07:19.084414+0000","last_unstale":"2024-01-15T18:07:20.096749+0000","last_undegraded":"2024-01-15T18:07:20.096749+0000","last_fullsized":"2024-01-15T18:07:20.096749+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:20.019967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.068126+0000","last_change":"2024-01-15T18:07:19.068126+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.068126+0000","last_undegraded":"2024-01-15T18:07:19.068126+0000","last_fullsized":"2024-01-15T18:07:19.068126+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100965+0000","last_change":"2024-01-15T18:07:19.077287+0000","last_active":"2024-01-15T18:07:20.100965+0000","last_peered":"2024-01-15T18:07:20.100965+0000","last_clean":"2024-01-15T18:07:20.100965+0000","last_became_active":"2024-01-15T18:07:19.075652+0000","last_became_peered":"2024-01-15T18:07:19.075652+0000","last_unstale":"2024-01-15T18:07:20.100965+0000","last_undegraded":"2024-01-15T18:07:20.100965+0000","last_fullsized":"2024-01-15T18:07:20.100965+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:57:56.264139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098517+0000","last_change":"2024-01-15T18:07:20.087874+0000","last_active":"2024-01-15T18:07:20.098517+0000","last_peered":"2024-01-15T18:07:20.098517+0000","last_clean":"2024-01-15T18:07:20.098517+0000","last_became_active":"2024-01-15T18:07:20.087579+0000","last_became_peered":"2024-01-15T18:07:20.087579+0000","last_unstale":"2024-01-15T18:07:20.098517+0000","last_undegraded":"2024-01-15T18:07:20.098517+0000","last_fullsized":"2024-01-15T18:07:20.098517+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:20:04.640555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098218+0000","last_change":"2024-01-15T18:07:19.085001+0000","last_active":"2024-01-15T18:07:20.098218+0000","last_peered":"2024-01-15T18:07:20.098218+0000","last_clean":"2024-01-15T18:07:20.098218+0000","last_became_active":"2024-01-15T18:07:19.084651+0000","last_became_peered":"2024-01-15T18:07:19.084651+0000","last_unstale":"2024-01-15T18:07:20.098218+0000","last_undegraded":"2024-01-15T18:07:20.098218+0000","last_fullsized":"2024-01-15T18:07:20.098218+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:13:24.552254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097208+0000","last_change":"2024-01-15T18:07:20.087693+0000","last_active":"2024-01-15T18:07:20.097208+0000","last_peered":"2024-01-15T18:07:20.097208+0000","last_clean":"2024-01-15T18:07:20.097208+0000","last_became_active":"2024-01-15T18:07:20.087416+0000","last_became_peered":"2024-01-15T18:07:20.087416+0000","last_unstale":"2024-01-15T18:07:20.097208+0000","last_undegraded":"2024-01-15T18:07:20.097208+0000","last_fullsized":"2024-01-15T18:07:20.097208+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:02.825993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097250+0000","last_change":"2024-01-15T18:07:19.086152+0000","last_active":"2024-01-15T18:07:20.097250+0000","last_peered":"2024-01-15T18:07:20.097250+0000","last_clean":"2024-01-15T18:07:20.097250+0000","last_became_active":"2024-01-15T18:07:19.082058+0000","last_became_peered":"2024-01-15T18:07:19.082058+0000","last_unstale":"2024-01-15T18:07:20.097250+0000","last_undegraded":"2024-01-15T18:07:20.097250+0000","last_fullsized":"2024-01-15T18:07:20.097250+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:06:20.719015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.074401+0000","last_change":"2024-01-15T18:07:19.074401+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.074401+0000","last_undegraded":"2024-01-15T18:07:19.074401+0000","last_fullsized":"2024-01-15T18:07:19.074401+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097139+0000","last_change":"2024-01-15T18:07:19.076932+0000","last_active":"2024-01-15T18:07:20.097139+0000","last_peered":"2024-01-15T18:07:20.097139+0000","last_clean":"2024-01-15T18:07:20.097139+0000","last_became_active":"2024-01-15T18:07:19.076629+0000","last_became_peered":"2024-01-15T18:07:19.076629+0000","last_unstale":"2024-01-15T18:07:20.097139+0000","last_undegraded":"2024-01-15T18:07:20.097139+0000","last_fullsized":"2024-01-15T18:07:20.097139+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:24:59.066289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.069432+0000","last_change":"2024-01-15T18:07:19.069432+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.069432+0000","last_undegraded":"2024-01-15T18:07:19.069432+0000","last_fullsized":"2024-01-15T18:07:19.069432+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.084299+0000","last_change":"2024-01-15T18:07:19.079711+0000","last_active":"2024-01-15T18:07:19.084299+0000","last_peered":"2024-01-15T18:07:19.084299+0000","last_clean":"2024-01-15T18:07:19.084299+0000","last_became_active":"2024-01-15T18:07:19.079336+0000","last_became_peered":"2024-01-15T18:07:19.079336+0000","last_unstale":"2024-01-15T18:07:19.084299+0000","last_undegraded":"2024-01-15T18:07:19.084299+0000","last_fullsized":"2024-01-15T18:07:19.084299+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:49:49.489371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098255+0000","last_change":"2024-01-15T18:07:20.087900+0000","last_active":"2024-01-15T18:07:20.098255+0000","last_peered":"2024-01-15T18:07:20.098255+0000","last_clean":"2024-01-15T18:07:20.098255+0000","last_became_active":"2024-01-15T18:07:20.087463+0000","last_became_peered":"2024-01-15T18:07:20.087463+0000","last_unstale":"2024-01-15T18:07:20.098255+0000","last_undegraded":"2024-01-15T18:07:20.098255+0000","last_fullsized":"2024-01-15T18:07:20.098255+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:54:09.185168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.082790+0000","last_change":"2024-01-15T18:07:19.079615+0000","last_active":"2024-01-15T18:07:19.082790+0000","last_peered":"2024-01-15T18:07:19.082790+0000","last_clean":"2024-01-15T18:07:19.082790+0000","last_became_active":"2024-01-15T18:07:19.079242+0000","last_became_peered":"2024-01-15T18:07:19.079242+0000","last_unstale":"2024-01-15T18:07:19.082790+0000","last_undegraded":"2024-01-15T18:07:19.082790+0000","last_fullsized":"2024-01-15T18:07:19.082790+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:47.335570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100426+0000","last_change":"2024-01-15T18:07:20.095844+0000","last_active":"2024-01-15T18:07:20.100426+0000","last_peered":"2024-01-15T18:07:20.100426+0000","last_clean":"2024-01-15T18:07:20.100426+0000","last_became_active":"2024-01-15T18:07:20.095110+0000","last_became_peered":"2024-01-15T18:07:20.095110+0000","last_unstale":"2024-01-15T18:07:20.100426+0000","last_undegraded":"2024-01-15T18:07:20.100426+0000","last_fullsized":"2024-01-15T18:07:20.100426+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:57:57.266821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.070487+0000","last_change":"2024-01-15T18:07:19.076938+0000","last_active":"2024-01-15T18:07:20.070487+0000","last_peered":"2024-01-15T18:07:20.070487+0000","last_clean":"2024-01-15T18:07:20.070487+0000","last_became_active":"2024-01-15T18:07:19.076605+0000","last_became_peered":"2024-01-15T18:07:19.076605+0000","last_unstale":"2024-01-15T18:07:20.070487+0000","last_undegraded":"2024-01-15T18:07:20.070487+0000","last_fullsized":"2024-01-15T18:07:20.070487+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:32.749313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.095519+0000","last_change":"2024-01-15T18:07:20.091131+0000","last_active":"2024-01-15T18:07:20.095519+0000","last_peered":"2024-01-15T18:07:20.095519+0000","last_clean":"2024-01-15T18:07:20.095519+0000","last_became_active":"2024-01-15T18:07:20.090683+0000","last_became_peered":"2024-01-15T18:07:20.090683+0000","last_unstale":"2024-01-15T18:07:20.095519+0000","last_undegraded":"2024-01-15T18:07:20.095519+0000","last_fullsized":"2024-01-15T18:07:20.095519+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:36:14.191387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.086659+0000","last_change":"2024-01-15T18:07:19.085014+0000","last_active":"2024-01-15T18:07:19.086659+0000","last_peered":"2024-01-15T18:07:19.086659+0000","last_clean":"2024-01-15T18:07:19.086659+0000","last_became_active":"2024-01-15T18:07:19.084470+0000","last_became_peered":"2024-01-15T18:07:19.084470+0000","last_unstale":"2024-01-15T18:07:19.086659+0000","last_undegraded":"2024-01-15T18:07:19.086659+0000","last_fullsized":"2024-01-15T18:07:19.086659+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:43:21.743406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.114219+0000","last_change":"2024-01-15T18:07:20.108821+0000","last_active":"2024-01-15T18:07:20.114219+0000","last_peered":"2024-01-15T18:07:20.114219+0000","last_clean":"2024-01-15T18:07:20.114219+0000","last_became_active":"2024-01-15T18:07:20.108464+0000","last_became_peered":"2024-01-15T18:07:20.108464+0000","last_unstale":"2024-01-15T18:07:20.114219+0000","last_undegraded":"2024-01-15T18:07:20.114219+0000","last_fullsized":"2024-01-15T18:07:20.114219+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:23:28.608885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"44'3","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.099259+0000","last_change":"2024-01-15T18:07:19.088202+0000","last_active":"2024-01-15T18:07:20.099259+0000","last_peered":"2024-01-15T18:07:20.099259+0000","last_clean":"2024-01-15T18:07:20.099259+0000","last_became_active":"2024-01-15T18:07:19.087900+0000","last_became_peered":"2024-01-15T18:07:19.087900+0000","last_unstale":"2024-01-15T18:07:20.099259+0000","last_undegraded":"2024-01-15T18:07:20.099259+0000","last_fullsized":"2024-01-15T18:07:20.099259+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:10:42.345570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.106599+0000","last_change":"2024-01-15T18:07:20.102897+0000","last_active":"2024-01-15T18:07:20.106599+0000","last_peered":"2024-01-15T18:07:20.106599+0000","last_clean":"2024-01-15T18:07:20.106599+0000","last_became_active":"2024-01-15T18:07:20.102579+0000","last_became_peered":"2024-01-15T18:07:20.102579+0000","last_unstale":"2024-01-15T18:07:20.106599+0000","last_undegraded":"2024-01-15T18:07:20.106599+0000","last_fullsized":"2024-01-15T18:07:20.106599+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:50:12.413904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":17,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.112054+0000","last_change":"2024-01-15T18:07:19.079571+0000","last_active":"2024-01-15T18:07:20.112054+0000","last_peered":"2024-01-15T18:07:20.112054+0000","last_clean":"2024-01-15T18:07:20.112054+0000","last_became_active":"2024-01-15T18:07:19.079308+0000","last_became_peered":"2024-01-15T18:07:19.079308+0000","last_unstale":"2024-01-15T18:07:20.112054+0000","last_undegraded":"2024-01-15T18:07:20.112054+0000","last_fullsized":"2024-01-15T18:07:20.112054+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:25.046543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.101579+0000","last_change":"2024-01-15T18:07:20.097611+0000","last_active":"2024-01-15T18:07:20.101579+0000","last_peered":"2024-01-15T18:07:20.101579+0000","last_clean":"2024-01-15T18:07:20.101579+0000","last_became_active":"2024-01-15T18:07:20.097202+0000","last_became_peered":"2024-01-15T18:07:20.097202+0000","last_unstale":"2024-01-15T18:07:20.101579+0000","last_undegraded":"2024-01-15T18:07:20.101579+0000","last_fullsized":"2024-01-15T18:07:20.101579+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:52:34.679712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.104292+0000","last_change":"2024-01-15T18:07:19.084662+0000","last_active":"2024-01-15T18:07:20.104292+0000","last_peered":"2024-01-15T18:07:20.104292+0000","last_clean":"2024-01-15T18:07:20.104292+0000","last_became_active":"2024-01-15T18:07:19.084348+0000","last_became_peered":"2024-01-15T18:07:19.084348+0000","last_unstale":"2024-01-15T18:07:20.104292+0000","last_undegraded":"2024-01-15T18:07:20.104292+0000","last_fullsized":"2024-01-15T18:07:20.104292+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:07:49.351274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.067641+0000","last_change":"2024-01-15T18:07:19.067641+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.067641+0000","last_undegraded":"2024-01-15T18:07:19.067641+0000","last_fullsized":"2024-01-15T18:07:19.067641+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.076930+0000","last_change":"2024-01-15T18:07:19.084512+0000","last_active":"2024-01-15T18:07:20.076930+0000","last_peered":"2024-01-15T18:07:20.076930+0000","last_clean":"2024-01-15T18:07:20.076930+0000","last_became_active":"2024-01-15T18:07:19.084169+0000","last_became_peered":"2024-01-15T18:07:19.084169+0000","last_unstale":"2024-01-15T18:07:20.076930+0000","last_undegraded":"2024-01-15T18:07:20.076930+0000","last_fullsized":"2024-01-15T18:07:20.076930+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:35:27.040611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.109334+0000","last_change":"2024-01-15T18:07:20.105484+0000","last_active":"2024-01-15T18:07:20.109334+0000","last_peered":"2024-01-15T18:07:20.109334+0000","last_clean":"2024-01-15T18:07:20.109334+0000","last_became_active":"2024-01-15T18:07:20.105160+0000","last_became_peered":"2024-01-15T18:07:20.105160+0000","last_unstale":"2024-01-15T18:07:20.109334+0000","last_undegraded":"2024-01-15T18:07:20.109334+0000","last_fullsized":"2024-01-15T18:07:20.109334+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:04:14.022333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.073804+0000","last_change":"2024-01-15T18:07:19.083510+0000","last_active":"2024-01-15T18:07:20.073804+0000","last_peered":"2024-01-15T18:07:20.073804+0000","last_clean":"2024-01-15T18:07:20.073804+0000","last_became_active":"2024-01-15T18:07:19.083134+0000","last_became_peered":"2024-01-15T18:07:19.083134+0000","last_unstale":"2024-01-15T18:07:20.073804+0000","last_undegraded":"2024-01-15T18:07:20.073804+0000","last_fullsized":"2024-01-15T18:07:20.073804+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:10:42.345570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.103258+0000","last_change":"2024-01-15T18:07:20.101397+0000","last_active":"2024-01-15T18:07:20.103258+0000","last_peered":"2024-01-15T18:07:20.103258+0000","last_clean":"2024-01-15T18:07:20.103258+0000","last_became_active":"2024-01-15T18:07:20.101106+0000","last_became_peered":"2024-01-15T18:07:20.101106+0000","last_unstale":"2024-01-15T18:07:20.103258+0000","last_undegraded":"2024-01-15T18:07:20.103258+0000","last_fullsized":"2024-01-15T18:07:20.103258+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:48:24.482600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.088352+0000","last_change":"2024-01-15T18:07:19.086583+0000","last_active":"2024-01-15T18:07:19.088352+0000","last_peered":"2024-01-15T18:07:19.088352+0000","last_clean":"2024-01-15T18:07:19.088352+0000","last_became_active":"2024-01-15T18:07:19.086227+0000","last_became_peered":"2024-01-15T18:07:19.086227+0000","last_unstale":"2024-01-15T18:07:19.088352+0000","last_undegraded":"2024-01-15T18:07:19.088352+0000","last_fullsized":"2024-01-15T18:07:19.088352+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:24:53.993680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.102535+0000","last_change":"2024-01-15T18:07:20.075833+0000","last_active":"2024-01-15T18:07:20.102535+0000","last_peered":"2024-01-15T18:07:20.102535+0000","last_clean":"2024-01-15T18:07:20.102535+0000","last_became_active":"2024-01-15T18:07:20.075521+0000","last_became_peered":"2024-01-15T18:07:20.075521+0000","last_unstale":"2024-01-15T18:07:20.102535+0000","last_undegraded":"2024-01-15T18:07:20.102535+0000","last_fullsized":"2024-01-15T18:07:20.102535+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:52:34.679712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.107096+0000","last_change":"2024-01-15T18:07:19.087887+0000","last_active":"2024-01-15T18:07:20.107096+0000","last_peered":"2024-01-15T18:07:20.107096+0000","last_clean":"2024-01-15T18:07:20.107096+0000","last_became_active":"2024-01-15T18:07:19.087610+0000","last_became_peered":"2024-01-15T18:07:19.087610+0000","last_unstale":"2024-01-15T18:07:20.107096+0000","last_undegraded":"2024-01-15T18:07:20.107096+0000","last_fullsized":"2024-01-15T18:07:20.107096+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:29:39.445420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.068689+0000","last_change":"2024-01-15T18:07:19.068689+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.068689+0000","last_undegraded":"2024-01-15T18:07:19.068689+0000","last_fullsized":"2024-01-15T18:07:19.068689+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.101388+0000","last_change":"2024-01-15T18:07:19.074682+0000","last_active":"2024-01-15T18:07:20.101388+0000","last_peered":"2024-01-15T18:07:20.101388+0000","last_clean":"2024-01-15T18:07:20.101388+0000","last_became_active":"2024-01-15T18:07:19.074397+0000","last_became_peered":"2024-01-15T18:07:19.074397+0000","last_unstale":"2024-01-15T18:07:20.101388+0000","last_undegraded":"2024-01-15T18:07:20.101388+0000","last_fullsized":"2024-01-15T18:07:20.101388+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:37:09.401497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100562+0000","last_change":"2024-01-15T18:07:20.093962+0000","last_active":"2024-01-15T18:07:20.100562+0000","last_peered":"2024-01-15T18:07:20.100562+0000","last_clean":"2024-01-15T18:07:20.100562+0000","last_became_active":"2024-01-15T18:07:20.093638+0000","last_became_peered":"2024-01-15T18:07:20.093638+0000","last_unstale":"2024-01-15T18:07:20.100562+0000","last_undegraded":"2024-01-15T18:07:20.100562+0000","last_fullsized":"2024-01-15T18:07:20.100562+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:30:43.231075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.086984+0000","last_change":"2024-01-15T18:07:19.082541+0000","last_active":"2024-01-15T18:07:19.086984+0000","last_peered":"2024-01-15T18:07:19.086984+0000","last_clean":"2024-01-15T18:07:19.086984+0000","last_became_active":"2024-01-15T18:07:19.082192+0000","last_became_peered":"2024-01-15T18:07:19.082192+0000","last_unstale":"2024-01-15T18:07:19.086984+0000","last_undegraded":"2024-01-15T18:07:19.086984+0000","last_fullsized":"2024-01-15T18:07:19.086984+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:10:51.637250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.103168+0000","last_change":"2024-01-15T18:07:20.096891+0000","last_active":"2024-01-15T18:07:20.103168+0000","last_peered":"2024-01-15T18:07:20.103168+0000","last_clean":"2024-01-15T18:07:20.103168+0000","last_became_active":"2024-01-15T18:07:20.096533+0000","last_became_peered":"2024-01-15T18:07:20.096533+0000","last_unstale":"2024-01-15T18:07:20.103168+0000","last_undegraded":"2024-01-15T18:07:20.103168+0000","last_fullsized":"2024-01-15T18:07:20.103168+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:14:41.208287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.075980+0000","last_change":"2024-01-15T18:07:19.080295+0000","last_active":"2024-01-15T18:07:20.075980+0000","last_peered":"2024-01-15T18:07:20.075980+0000","last_clean":"2024-01-15T18:07:20.075980+0000","last_became_active":"2024-01-15T18:07:19.080024+0000","last_became_peered":"2024-01-15T18:07:19.080024+0000","last_unstale":"2024-01-15T18:07:20.075980+0000","last_undegraded":"2024-01-15T18:07:20.075980+0000","last_fullsized":"2024-01-15T18:07:20.075980+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:36:48.732986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.112236+0000","last_change":"2024-01-15T18:07:20.105757+0000","last_active":"2024-01-15T18:07:20.112236+0000","last_peered":"2024-01-15T18:07:20.112236+0000","last_clean":"2024-01-15T18:07:20.112236+0000","last_became_active":"2024-01-15T18:07:20.105385+0000","last_became_peered":"2024-01-15T18:07:20.105385+0000","last_unstale":"2024-01-15T18:07:20.112236+0000","last_undegraded":"2024-01-15T18:07:20.112236+0000","last_fullsized":"2024-01-15T18:07:20.112236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:51:04.157158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097287+0000","last_change":"2024-01-15T18:07:19.082297+0000","last_active":"2024-01-15T18:07:20.097287+0000","last_peered":"2024-01-15T18:07:20.097287+0000","last_clean":"2024-01-15T18:07:20.097287+0000","last_became_active":"2024-01-15T18:07:19.081971+0000","last_became_peered":"2024-01-15T18:07:19.081971+0000","last_unstale":"2024-01-15T18:07:20.097287+0000","last_undegraded":"2024-01-15T18:07:20.097287+0000","last_fullsized":"2024-01-15T18:07:20.097287+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:23:41.478738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098551+0000","last_change":"2024-01-15T18:07:20.091112+0000","last_active":"2024-01-15T18:07:20.098551+0000","last_peered":"2024-01-15T18:07:20.098551+0000","last_clean":"2024-01-15T18:07:20.098551+0000","last_became_active":"2024-01-15T18:07:20.090802+0000","last_became_peered":"2024-01-15T18:07:20.090802+0000","last_unstale":"2024-01-15T18:07:20.098551+0000","last_undegraded":"2024-01-15T18:07:20.098551+0000","last_fullsized":"2024-01-15T18:07:20.098551+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:01:01.613225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.082669+0000","last_change":"2024-01-15T18:07:19.077896+0000","last_active":"2024-01-15T18:07:19.082669+0000","last_peered":"2024-01-15T18:07:19.082669+0000","last_clean":"2024-01-15T18:07:19.082669+0000","last_became_active":"2024-01-15T18:07:19.077439+0000","last_became_peered":"2024-01-15T18:07:19.077439+0000","last_unstale":"2024-01-15T18:07:19.082669+0000","last_undegraded":"2024-01-15T18:07:19.082669+0000","last_fullsized":"2024-01-15T18:07:19.082669+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:09:06.512495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.104003+0000","last_change":"2024-01-15T18:07:20.086589+0000","last_active":"2024-01-15T18:07:20.104003+0000","last_peered":"2024-01-15T18:07:20.104003+0000","last_clean":"2024-01-15T18:07:20.104003+0000","last_became_active":"2024-01-15T18:07:20.086312+0000","last_became_peered":"2024-01-15T18:07:20.086312+0000","last_unstale":"2024-01-15T18:07:20.104003+0000","last_undegraded":"2024-01-15T18:07:20.104003+0000","last_fullsized":"2024-01-15T18:07:20.104003+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:15:59.320681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.083833+0000","last_change":"2024-01-15T18:07:19.082155+0000","last_active":"2024-01-15T18:07:19.083833+0000","last_peered":"2024-01-15T18:07:19.083833+0000","last_clean":"2024-01-15T18:07:19.083833+0000","last_became_active":"2024-01-15T18:07:19.081855+0000","last_became_peered":"2024-01-15T18:07:19.081855+0000","last_unstale":"2024-01-15T18:07:19.083833+0000","last_undegraded":"2024-01-15T18:07:19.083833+0000","last_fullsized":"2024-01-15T18:07:19.083833+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:07.014633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.067698+0000","last_change":"2024-01-15T18:07:19.067698+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.067698+0000","last_undegraded":"2024-01-15T18:07:19.067698+0000","last_fullsized":"2024-01-15T18:07:19.067698+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.076645+0000","last_change":"2024-01-15T18:07:19.085582+0000","last_active":"2024-01-15T18:07:20.076645+0000","last_peered":"2024-01-15T18:07:20.076645+0000","last_clean":"2024-01-15T18:07:20.076645+0000","last_became_active":"2024-01-15T18:07:19.083796+0000","last_became_peered":"2024-01-15T18:07:19.083796+0000","last_unstale":"2024-01-15T18:07:20.076645+0000","last_undegraded":"2024-01-15T18:07:20.076645+0000","last_fullsized":"2024-01-15T18:07:20.076645+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:03:20.187627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.106501+0000","last_change":"2024-01-15T18:07:20.100547+0000","last_active":"2024-01-15T18:07:20.106501+0000","last_peered":"2024-01-15T18:07:20.106501+0000","last_clean":"2024-01-15T18:07:20.106501+0000","last_became_active":"2024-01-15T18:07:20.100201+0000","last_became_peered":"2024-01-15T18:07:20.100201+0000","last_unstale":"2024-01-15T18:07:20.106501+0000","last_undegraded":"2024-01-15T18:07:20.106501+0000","last_fullsized":"2024-01-15T18:07:20.106501+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:29:40.448102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.072838+0000","last_change":"2024-01-15T18:07:19.078965+0000","last_active":"2024-01-15T18:07:20.072838+0000","last_peered":"2024-01-15T18:07:20.072838+0000","last_clean":"2024-01-15T18:07:20.072838+0000","last_became_active":"2024-01-15T18:07:19.078692+0000","last_became_peered":"2024-01-15T18:07:19.078692+0000","last_unstale":"2024-01-15T18:07:20.072838+0000","last_undegraded":"2024-01-15T18:07:20.072838+0000","last_fullsized":"2024-01-15T18:07:20.072838+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:13:24.552254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.067320+0000","last_change":"2024-01-15T18:07:19.067320+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.067320+0000","last_undegraded":"2024-01-15T18:07:19.067320+0000","last_fullsized":"2024-01-15T18:07:19.067320+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.077286+0000","last_change":"2024-01-15T18:07:19.088174+0000","last_active":"2024-01-15T18:07:20.077286+0000","last_peered":"2024-01-15T18:07:20.077286+0000","last_clean":"2024-01-15T18:07:20.077286+0000","last_became_active":"2024-01-15T18:07:19.087881+0000","last_became_peered":"2024-01-15T18:07:19.087881+0000","last_unstale":"2024-01-15T18:07:20.077286+0000","last_undegraded":"2024-01-15T18:07:20.077286+0000","last_fullsized":"2024-01-15T18:07:20.077286+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:36:48.732986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.108471+0000","last_change":"2024-01-15T18:07:20.104788+0000","last_active":"2024-01-15T18:07:20.108471+0000","last_peered":"2024-01-15T18:07:20.108471+0000","last_clean":"2024-01-15T18:07:20.108471+0000","last_became_active":"2024-01-15T18:07:20.104475+0000","last_became_peered":"2024-01-15T18:07:20.104475+0000","last_unstale":"2024-01-15T18:07:20.108471+0000","last_undegraded":"2024-01-15T18:07:20.108471+0000","last_fullsized":"2024-01-15T18:07:20.108471+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:02:26.277338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.071516+0000","last_change":"2024-01-15T18:07:19.080590+0000","last_active":"2024-01-15T18:07:20.071516+0000","last_peered":"2024-01-15T18:07:20.071516+0000","last_clean":"2024-01-15T18:07:20.071516+0000","last_became_active":"2024-01-15T18:07:19.080276+0000","last_became_peered":"2024-01-15T18:07:19.080276+0000","last_unstale":"2024-01-15T18:07:20.071516+0000","last_undegraded":"2024-01-15T18:07:20.071516+0000","last_fullsized":"2024-01-15T18:07:20.071516+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:24:59.066289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100402+0000","last_change":"2024-01-15T18:07:20.090127+0000","last_active":"2024-01-15T18:07:20.100402+0000","last_peered":"2024-01-15T18:07:20.100402+0000","last_clean":"2024-01-15T18:07:20.100402+0000","last_became_active":"2024-01-15T18:07:20.089727+0000","last_became_peered":"2024-01-15T18:07:20.089727+0000","last_unstale":"2024-01-15T18:07:20.100402+0000","last_undegraded":"2024-01-15T18:07:20.100402+0000","last_fullsized":"2024-01-15T18:07:20.100402+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:36:14.191387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.077270+0000","last_change":"2024-01-15T18:07:19.088037+0000","last_active":"2024-01-15T18:07:20.077270+0000","last_peered":"2024-01-15T18:07:20.077270+0000","last_clean":"2024-01-15T18:07:20.077270+0000","last_became_active":"2024-01-15T18:07:19.087721+0000","last_became_peered":"2024-01-15T18:07:19.087721+0000","last_unstale":"2024-01-15T18:07:20.077270+0000","last_undegraded":"2024-01-15T18:07:20.077270+0000","last_fullsized":"2024-01-15T18:07:20.077270+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:23:27.606202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.111942+0000","last_change":"2024-01-15T18:07:20.105653+0000","last_active":"2024-01-15T18:07:20.111942+0000","last_peered":"2024-01-15T18:07:20.111942+0000","last_clean":"2024-01-15T18:07:20.111942+0000","last_became_active":"2024-01-15T18:07:20.105348+0000","last_became_peered":"2024-01-15T18:07:20.105348+0000","last_unstale":"2024-01-15T18:07:20.111942+0000","last_undegraded":"2024-01-15T18:07:20.111942+0000","last_fullsized":"2024-01-15T18:07:20.111942+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:02:26.277338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.068819+0000","last_change":"2024-01-15T18:07:19.076027+0000","last_active":"2024-01-15T18:07:20.068819+0000","last_peered":"2024-01-15T18:07:20.068819+0000","last_clean":"2024-01-15T18:07:20.068819+0000","last_became_active":"2024-01-15T18:07:19.075644+0000","last_became_peered":"2024-01-15T18:07:19.075644+0000","last_unstale":"2024-01-15T18:07:20.068819+0000","last_undegraded":"2024-01-15T18:07:20.068819+0000","last_fullsized":"2024-01-15T18:07:20.068819+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:07:57.062677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.064957+0000","last_change":"2024-01-15T18:07:19.064957+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.064957+0000","last_undegraded":"2024-01-15T18:07:19.064957+0000","last_fullsized":"2024-01-15T18:07:19.064957+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.072048+0000","last_change":"2024-01-15T18:07:19.081215+0000","last_active":"2024-01-15T18:07:20.072048+0000","last_peered":"2024-01-15T18:07:20.072048+0000","last_clean":"2024-01-15T18:07:20.072048+0000","last_became_active":"2024-01-15T18:07:19.080901+0000","last_became_peered":"2024-01-15T18:07:19.080901+0000","last_unstale":"2024-01-15T18:07:20.072048+0000","last_undegraded":"2024-01-15T18:07:20.072048+0000","last_fullsized":"2024-01-15T18:07:20.072048+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:38:52.124114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.094487+0000","last_change":"2024-01-15T18:07:20.081122+0000","last_active":"2024-01-15T18:07:20.094487+0000","last_peered":"2024-01-15T18:07:20.094487+0000","last_clean":"2024-01-15T18:07:20.094487+0000","last_became_active":"2024-01-15T18:07:20.080752+0000","last_became_peered":"2024-01-15T18:07:20.080752+0000","last_unstale":"2024-01-15T18:07:20.094487+0000","last_undegraded":"2024-01-15T18:07:20.094487+0000","last_fullsized":"2024-01-15T18:07:20.094487+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:08:56.987120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.087492+0000","last_change":"2024-01-15T18:07:19.085696+0000","last_active":"2024-01-15T18:07:19.087492+0000","last_peered":"2024-01-15T18:07:19.087492+0000","last_clean":"2024-01-15T18:07:19.087492+0000","last_became_active":"2024-01-15T18:07:19.085467+0000","last_became_peered":"2024-01-15T18:07:19.085467+0000","last_unstale":"2024-01-15T18:07:19.087492+0000","last_undegraded":"2024-01-15T18:07:19.087492+0000","last_fullsized":"2024-01-15T18:07:19.087492+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:26:07.301619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.068029+0000","last_change":"2024-01-15T18:07:19.068029+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.068029+0000","last_undegraded":"2024-01-15T18:07:19.068029+0000","last_fullsized":"2024-01-15T18:07:19.068029+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.086899+0000","last_change":"2024-01-15T18:07:19.081361+0000","last_active":"2024-01-15T18:07:19.086899+0000","last_peered":"2024-01-15T18:07:19.086899+0000","last_clean":"2024-01-15T18:07:19.086899+0000","last_became_active":"2024-01-15T18:07:19.081076+0000","last_became_peered":"2024-01-15T18:07:19.081076+0000","last_unstale":"2024-01-15T18:07:19.086899+0000","last_undegraded":"2024-01-15T18:07:19.086899+0000","last_fullsized":"2024-01-15T18:07:19.086899+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:34:07.305579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":183,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.074017+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:07:19.074017+0000","last_peered":"2024-01-15T18:07:19.074017+0000","last_clean":"2024-01-15T18:07:19.074017+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:07:19.074017+0000","last_undegraded":"2024-01-15T18:07:19.074017+0000","last_fullsized":"2024-01-15T18:07:19.074017+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2340,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509474,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32248,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339592,"statfs":{"total":96636764160,"available":96603742208,"internally_reserved":0,"allocated":1089536,"data_stored":789175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.628,"15min":0.625},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.67200000000000004,"15min":0.67700000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.78200000000000003}]},{"osd":1,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.67800000000000005,"15min":0.67100000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.69399999999999995,"15min":0.69799999999999995},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.90400000000000003,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.56000000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64800000000000002,"15min":0.64700000000000002},"min":{"1min":0.46700000000000003,"5min":0.38,"15min":0.38},"max":{"1min":0.84199999999999997,"5min":1.012,"15min":1.012},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.71299999999999997,"15min":0.71199999999999997},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.0980000000000001,"5min":1.175,"15min":1.175},"last":0.68799999999999994}]},{"osd":3,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.66900000000000004,"15min":0.66500000000000004},"min":{"1min":0.502,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.88200000000000001,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0569999999999999,"5min":1.103,"15min":1.103},"last":0.74099999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.60799999999999998,"15min":0.61599999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.69199999999999995,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.65700000000000003,"15min":0.66200000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.622}]},{"osd":5,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.69499999999999995,"15min":0.69899999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.63,"5min":0.70999999999999996,"15min":0.72299999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.89500000000000002,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.70699999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.78800000000000003,"15min":0.80000000000000004},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.89500000000000002,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72499999999999998,"15min":0.73099999999999998},"min":{"1min":0.52500000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.76600000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509474,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31668,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340172,"statfs":{"total":96636764160,"available":96604336128,"internally_reserved":0,"allocated":495616,"data_stored":198807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.432,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.67200000000000004,"15min":0.68899999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84899999999999998,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.755}]},{"osd":1,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.70299999999999996,"15min":0.72399999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.82699999999999996,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.74399999999999999,"15min":0.754},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":0.59099999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.82099999999999995,"15min":0.84399999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.825,"5min":2.8700000000000001,"15min":2.8700000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.755,"15min":0.78300000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.82399999999999995,"5min":1.857,"15min":1.857},"last":0.78600000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.77400000000000002,"15min":0.80100000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.92100000000000004,"5min":2.012,"15min":2.012},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.79900000000000004,"15min":0.82499999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":2.532,"15min":2.532},"last":0.66200000000000003}]},{"osd":4,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68300000000000005,"15min":0.69299999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.726,"5min":1.726,"15min":1.726},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.69399999999999995,"15min":0.71799999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.753,"5min":1.145,"15min":1.145},"last":0.81799999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.64500000000000002,"15min":0.65800000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.81299999999999994,"5min":1.121,"15min":1.121},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.77400000000000002,"15min":0.78100000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"last":0.91400000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.80100000000000005,"15min":0.82499999999999996},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.96499999999999997,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.874},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.78000000000000003,"15min":0.80200000000000005},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.081,"5min":1.165,"15min":1.165},"last":0.80500000000000005}]}]},{"osd":4,"up_from":15,"seq":64424509474,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31688,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340152,"statfs":{"total":96636764160,"available":96604315648,"internally_reserved":0,"allocated":520192,"data_stored":207222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.628,"15min":0.63},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.63200000000000001,"15min":0.64100000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.60299999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64200000000000002,"15min":0.64900000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.77300000000000002,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.628,"5min":0.64000000000000001,"15min":0.64200000000000002},"min":{"1min":0.40899999999999997,"5min":0.377,"15min":0.377},"max":{"1min":0.88,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.66200000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.626,"15min":0.628},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.69099999999999995,"15min":0.69499999999999995},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.81999999999999995,"5min":0.879,"15min":0.879},"last":0.80600000000000005}]},{"osd":3,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68899999999999995,"15min":0.68999999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.96699999999999997,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.72499999999999998,"15min":0.72599999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.93200000000000005,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.79000000000000004}]},{"osd":5,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40200000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65100000000000002,"15min":0.65000000000000002},"min":{"1min":0.40699999999999997,"5min":0.377,"15min":0.377},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.71699999999999997}]},{"osd":6,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.68200000000000005,"15min":0.68000000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.85099999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.46899999999999997,"5min":0.436,"15min":0.436},"max":{"1min":1.0069999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.81799999999999995}]}]},{"osd":3,"up_from":15,"seq":64424509473,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":520192,"data_stored":206248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.28499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.72399999999999998,"5min":0.752,"15min":0.752},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.499,"15min":0.47999999999999998},"min":{"1min":0.34000000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":2.6509999999999998,"5min":2.6509999999999998,"15min":2.6509999999999998},"last":0.38500000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70999999999999996,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.498},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.53700000000000003,"15min":0.53600000000000003},"min":{"1min":0.42499999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83399999999999996,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.52500000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.55800000000000005,"15min":0.56000000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.58599999999999997,"15min":0.57199999999999995},"min":{"1min":0.42899999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.56399999999999995}]},{"osd":4,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.79400000000000004,"15min":0.80300000000000005},"min":{"1min":0.50800000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.119,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.876},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.77300000000000002,"15min":0.77900000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.97899999999999998,"5min":1.341,"15min":1.341},"last":0.65600000000000003}]},{"osd":5,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.59199999999999997,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.72399999999999998,"15min":0.72199999999999998},"min":{"1min":0.53300000000000003,"5min":0.498,"15min":0.498},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":0.73299999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.78300000000000003,"15min":0.78500000000000003},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.77600000000000002,"15min":0.76600000000000001},"min":{"1min":0.628,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":2.0880000000000001,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":0.80200000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.78800000000000003,"15min":0.78100000000000003},"min":{"1min":0.58999999999999997,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.81299999999999994,"15min":0.81999999999999995},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.1259999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.78600000000000003}]}]},{"osd":2,"up_from":15,"seq":64424509473,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31668,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340172,"statfs":{"total":96636764160,"available":96604336128,"internally_reserved":0,"allocated":495616,"data_stored":198807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.45500000000000002,"15min":0.46100000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.75,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.441,"5min":0.46100000000000002,"15min":0.46500000000000002},"min":{"1min":0.29899999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.64700000000000002,"5min":0.753,"15min":0.753},"last":0.46400000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54300000000000004,"15min":0.54500000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.67600000000000005,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.314,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.65100000000000002,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.436}]},{"osd":3,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.57799999999999996,"15min":0.57299999999999995},"min":{"1min":0.375,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.55600000000000005,"15min":0.55300000000000005},"min":{"1min":0.40500000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71499999999999997,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.42099999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.75700000000000001,"15min":0.76700000000000002},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.038,"5min":1.466,"15min":1.466},"last":0.626}]},{"osd":5,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.79400000000000004,"15min":0.79700000000000004},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.609,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.747,"5min":0.78300000000000003,"15min":0.78900000000000003},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.64600000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.76800000000000002,"15min":0.77800000000000002},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0489999999999999,"5min":1.516,"15min":1.516},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.78600000000000003,"15min":0.79600000000000004},"min":{"1min":0.52800000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0820000000000001,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.56100000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.79500000000000004,"15min":0.80100000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.1659999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.81799999999999995,"15min":0.82599999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.69299999999999995}]}]},{"osd":1,"up_from":15,"seq":64424509475,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31668,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340172,"statfs":{"total":96636764160,"available":96604336128,"internally_reserved":0,"allocated":495616,"data_stored":198807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.49099999999999999,"15min":0.504},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.627,"5min":1.847,"15min":1.847},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.443,"5min":0.47899999999999998,"15min":0.48499999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.63500000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.54300000000000004}]},{"osd":2,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.53400000000000003,"15min":0.54400000000000004},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.77500000000000002,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.624},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.70299999999999996,"15min":0.73299999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.90100000000000002,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.46700000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.63100000000000001,"15min":0.64500000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80700000000000005,"5min":1.871,"15min":1.871},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.58099999999999996,"15min":0.59099999999999997},"min":{"1min":0.35299999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.72599999999999998,"5min":1.917,"15min":1.917},"last":0.496}]},{"osd":4,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.70299999999999996,"15min":0.69599999999999995},"min":{"1min":0.54800000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.161,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.77100000000000002,"15min":0.77700000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.147,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.73799999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.76900000000000002,"15min":0.77100000000000002},"min":{"1min":0.47299999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.214,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.53100000000000003,"5min":0.505,"15min":0.505},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.97899999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.753,"15min":0.755},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.1359999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.81599999999999995,"15min":0.82199999999999995},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.95699999999999996}]},{"osd":7,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78700000000000003,"15min":0.78500000000000003},"min":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.79900000000000004,"15min":0.80300000000000005},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.095,"5min":1.347,"15min":1.347},"last":0.90500000000000003}]}]},{"osd":5,"up_from":14,"seq":60129542178,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":483328,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":0.871,"15min":0.77500000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":21.385000000000002,"5min":21.385000000000002,"15min":21.385000000000002},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.202,"5min":0.878,"15min":0.82399999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":8.2029999999999994,"5min":8.2029999999999994,"15min":8.2029999999999994},"last":0.73599999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66400000000000003,"15min":0.67200000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":1.4670000000000001,"5min":0.84099999999999997,"15min":0.73699999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":21.5,"5min":21.5,"15min":21.5},"last":0.80100000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":0.81599999999999995,"15min":0.745},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":8.2439999999999998,"5min":8.2439999999999998,"15min":8.2439999999999998},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.748,"15min":0.77000000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.85099999999999998,"5min":1.663,"15min":1.663},"last":0.71999999999999997}]},{"osd":3,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":0.95299999999999996,"15min":0.85699999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":21.457000000000001,"5min":21.457000000000001,"15min":21.457000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.1919999999999999,"5min":0.92100000000000004,"15min":0.876},"min":{"1min":0.50900000000000001,"5min":0.502,"15min":0.502},"max":{"1min":8.2750000000000004,"5min":8.2750000000000004,"15min":8.2750000000000004},"last":0.70499999999999996}]},{"osd":4,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":0.82599999999999996,"15min":0.76600000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":9.6679999999999993,"5min":9.6679999999999993,"15min":9.6679999999999993},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.71299999999999997,"15min":0.73499999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.76300000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.60099999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":0.877,"15min":0.78400000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":21.535,"5min":21.535,"15min":21.535},"last":0.54100000000000004},{"interface":"front","average":{"1min":1.1870000000000001,"5min":0.88300000000000001,"15min":0.83199999999999996},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":8.298,"5min":8.298,"15min":8.298},"last":0.66700000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":0.83999999999999997,"15min":0.81399999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":9.2240000000000002,"5min":9.2240000000000002,"15min":9.2240000000000002},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.5249999999999999,"5min":0.97099999999999997,"15min":0.879},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":21.574000000000002,"5min":21.574000000000002,"15min":21.574000000000002},"last":0.748}]}]},{"osd":0,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32240,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339600,"statfs":{"total":96636764160,"available":96603750400,"internally_reserved":0,"allocated":1077248,"data_stored":782880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.46300000000000002,"15min":0.46500000000000002},"min":{"1min":0.28999999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.746,"5min":1.462,"15min":1.462},"last":0.501},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55400000000000005,"15min":0.55200000000000005},"min":{"1min":0.34399999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.995,"5min":1.373,"15min":1.373},"last":0.39000000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.371,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.89600000000000002,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.441},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.48699999999999999,"15min":0.48099999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.252,"5min":1.609,"15min":1.609},"last":0.54500000000000004}]},{"osd":3,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.371,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.321,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60899999999999999,"15min":0.60999999999999999},"min":{"1min":0.45000000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.94599999999999995,"5min":1.325,"15min":1.325},"last":0.52700000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.76700000000000002,"15min":0.77700000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.96099999999999997,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.80900000000000005,"15min":0.82599999999999996},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.93300000000000005,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.71099999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.78500000000000003,"15min":0.79500000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0109999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.83099999999999996,"15min":0.84699999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":0.63300000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.79400000000000004,"15min":0.79800000000000004},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.107,"5min":1.764,"15min":1.764},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.83099999999999996,"15min":0.84699999999999998},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.97499999999999998,"5min":1.806,"15min":1.806},"last":0.72699999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.82199999999999995,"15min":0.82999999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.04,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.84899999999999998,"15min":0.85999999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.0429999999999999,"5min":1.722,"15min":1.722},"last":0.67100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:07:24.576 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:07:24.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.752+0000 7f191da00640 1 Processor -- start 2024-01-15T18:07:24.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.756+0000 7f191da00640 1 -- start start 2024-01-15T18:07:24.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.756+0000 7f191da00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19180b2390 0x7f19180cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:24.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.756+0000 7f191da00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f191806bb80 0x7f19180cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:24.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.756+0000 7f191da00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19180b2770 con 0x7f19180cfcc0 2024-01-15T18:07:24.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.756+0000 7f191da00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f191806c060 con 0x7f191806bb80 2024-01-15T18:07:24.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.756+0000 7f191da00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f191806c340 con 0x7f19180b2390 2024-01-15T18:07:24.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.756+0000 7f1916000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f191806bb80 0x7f19180cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:24.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.756+0000 7f1917400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f19180cfcc0 0x7f1918161120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45184/0 (socket says 172.21.15.27:45184) 2024-01-15T18:07:24.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.756+0000 7f1917400640 1 -- 172.21.15.27:0/4213649443 learned_addr learned my addr 172.21.15.27:0/4213649443 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:24.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.756+0000 7f1916a00640 1 --2- 172.21.15.27:0/4213649443 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19180b2390 0x7f19180cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:24.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.756+0000 7f1916000640 1 -- 172.21.15.27:0/4213649443 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19180b2390 msgr2=0x7f19180cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:24.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.756+0000 7f1916000640 1 --2- 172.21.15.27:0/4213649443 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19180b2390 0x7f19180cf780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.756+0000 7f1916000640 1 -- 172.21.15.27:0/4213649443 >> v1:172.21.15.27:6789/0 conn(0x7f19180cfcc0 legacy=0x7f1918161120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:24.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.756+0000 7f1916000640 1 -- 172.21.15.27:0/4213649443 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f191806c620 con 0x7f191806bb80 2024-01-15T18:07:24.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.756+0000 7f1915600640 1 -- 172.21.15.27:0/4213649443 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115762819 0 0) 0x7f19180b2770 con 0x7f19180cfcc0 2024-01-15T18:07:24.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.760+0000 7f1916000640 1 --2- 172.21.15.27:0/4213649443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191806bb80 0x7f19180cd880 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f19000097c0 tx=0x7f190001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cca4a67068c7106c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:24.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.760+0000 7f1915600640 1 -- 172.21.15.27:0/4213649443 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19000ab650 con 0x7f191806bb80 2024-01-15T18:07:24.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.760+0000 7f1915600640 1 -- 172.21.15.27:0/4213649443 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19000ab7b0 con 0x7f191806bb80 2024-01-15T18:07:24.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.760+0000 7f1915600640 1 -- 172.21.15.27:0/4213649443 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19000abb40 con 0x7f191806bb80 2024-01-15T18:07:24.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.760+0000 7f191da00640 1 -- 172.21.15.27:0/4213649443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191806bb80 msgr2=0x7f19180cd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:24.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.760+0000 7f191da00640 1 --2- 172.21.15.27:0/4213649443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191806bb80 0x7f19180cd880 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f19000097c0 tx=0x7f190001c450 comp rx=0 tx=0).stop 2024-01-15T18:07:24.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.760+0000 7f191da00640 1 -- 172.21.15.27:0/4213649443 shutdown_connections 2024-01-15T18:07:24.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.760+0000 7f191da00640 1 --2- 172.21.15.27:0/4213649443 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191806bb80 0x7f19180cd880 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.760+0000 7f191da00640 1 --2- 172.21.15.27:0/4213649443 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19180b2390 0x7f19180cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.760+0000 7f191da00640 1 -- 172.21.15.27:0/4213649443 >> 172.21.15.27:0/4213649443 conn(0x7f19180b2a10 msgr2=0x7f19180b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:24.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.760+0000 7f191da00640 1 -- 172.21.15.27:0/4213649443 shutdown_connections 2024-01-15T18:07:24.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.760+0000 7f191da00640 1 -- 172.21.15.27:0/4213649443 wait complete. 2024-01-15T18:07:24.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.760+0000 7f191da00640 1 Processor -- start 2024-01-15T18:07:24.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f191da00640 1 -- start start 2024-01-15T18:07:24.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f191da00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191806bb80 0x7f19180c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:24.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f191da00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19180b2390 0x7f191815d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:24.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f191da00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f19180cfcc0 0x7f191815d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:24.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f191da00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f191815bfd0 con 0x7f191806bb80 2024-01-15T18:07:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f191da00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f191815c270 con 0x7f19180cfcc0 2024-01-15T18:07:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f191da00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f191815c550 con 0x7f19180b2390 2024-01-15T18:07:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f1916a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191806bb80 0x7f19180c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f1916a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191806bb80 0x7f19180c7f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50734/0 (socket says 172.21.15.27:50734) 2024-01-15T18:07:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f1916a00640 1 -- 172.21.15.27:0/1563628653 learned_addr learned my addr 172.21.15.27:0/1563628653 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f1916000640 1 --2- 172.21.15.27:0/1563628653 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19180b2390 0x7f191815d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f1917400640 1 --2- 172.21.15.27:0/1563628653 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f19180cfcc0 0x7f191815d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f1916a00640 1 -- 172.21.15.27:0/1563628653 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19180b2390 msgr2=0x7f191815d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f1916a00640 1 --2- 172.21.15.27:0/1563628653 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19180b2390 0x7f191815d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f1916a00640 1 -- 172.21.15.27:0/1563628653 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f19180cfcc0 msgr2=0x7f191815d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f1916a00640 1 --2- 172.21.15.27:0/1563628653 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f19180cfcc0 0x7f191815d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f1916a00640 1 -- 172.21.15.27:0/1563628653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1900009420 con 0x7f191806bb80 2024-01-15T18:07:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f1916000640 1 --2- 172.21.15.27:0/1563628653 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19180b2390 0x7f191815d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:07:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f1917400640 1 --2- 172.21.15.27:0/1563628653 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f19180cfcc0 0x7f191815d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:07:24.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f1916a00640 1 --2- 172.21.15.27:0/1563628653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191806bb80 0x7f19180c7f90 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f190800e870 tx=0x7f190800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:24.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f190ea00640 1 -- 172.21.15.27:0/1563628653 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1908009510 con 0x7f191806bb80 2024-01-15T18:07:24.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f190ea00640 1 -- 172.21.15.27:0/1563628653 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f190800eec0 con 0x7f191806bb80 2024-01-15T18:07:24.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f190ea00640 1 -- 172.21.15.27:0/1563628653 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190800cb40 con 0x7f191806bb80 2024-01-15T18:07:24.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f191da00640 1 -- 172.21.15.27:0/1563628653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f191815c830 con 0x7f191806bb80 2024-01-15T18:07:24.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f191da00640 1 -- 172.21.15.27:0/1563628653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19181561a0 con 0x7f191806bb80 2024-01-15T18:07:24.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.764+0000 7f191da00640 1 -- 172.21.15.27:0/1563628653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19180d07c0 con 0x7f191806bb80 2024-01-15T18:07:24.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.768+0000 7f190ea00640 1 -- 172.21.15.27:0/1563628653 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f190800cca0 con 0x7f191806bb80 2024-01-15T18:07:24.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.768+0000 7f190ea00640 1 --2- 172.21.15.27:0/1563628653 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f18f0066b30 0x7f18f0068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:24.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.768+0000 7f190ea00640 1 -- 172.21.15.27:0/1563628653 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f1908084920 con 0x7f191806bb80 2024-01-15T18:07:24.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.768+0000 7f1916000640 1 --2- 172.21.15.27:0/1563628653 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f18f0066b30 0x7f18f0068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:24.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.768+0000 7f1916000640 1 --2- 172.21.15.27:0/1563628653 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f18f0066b30 0x7f18f0068ff0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f19000af000 tx=0x7f1900008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:24.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.772+0000 7f190ea00640 1 -- 172.21.15.27:0/1563628653 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1908051870 con 0x7f191806bb80 2024-01-15T18:07:24.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.924+0000 7f191da00640 1 -- 172.21.15.27:0/1563628653 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f191815c3e0 con 0x7f18f0066b30 2024-01-15T18:07:24.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.932+0000 7f190ea00640 1 -- 172.21.15.27:0/1563628653 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287094 (secure 0 0 0) 0x7f191815c3e0 con 0x7f18f0066b30 2024-01-15T18:07:24.939 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:07:24.941 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:07:24.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.936+0000 7f191da00640 1 -- 172.21.15.27:0/1563628653 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f18f0066b30 msgr2=0x7f18f0068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:24.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.936+0000 7f191da00640 1 --2- 172.21.15.27:0/1563628653 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f18f0066b30 0x7f18f0068ff0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f19000af000 tx=0x7f1900008040 comp rx=0 tx=0).stop 2024-01-15T18:07:24.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.936+0000 7f191da00640 1 -- 172.21.15.27:0/1563628653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191806bb80 msgr2=0x7f19180c7f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:24.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.936+0000 7f191da00640 1 --2- 172.21.15.27:0/1563628653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191806bb80 0x7f19180c7f90 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f190800e870 tx=0x7f190800ec70 comp rx=0 tx=0).stop 2024-01-15T18:07:24.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.936+0000 7f191da00640 1 -- 172.21.15.27:0/1563628653 shutdown_connections 2024-01-15T18:07:24.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.936+0000 7f191da00640 1 --2- 172.21.15.27:0/1563628653 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f18f0066b30 0x7f18f0068ff0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.936+0000 7f191da00640 1 --2- 172.21.15.27:0/1563628653 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f19180cfcc0 0x7f191815d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.936+0000 7f191da00640 1 --2- 172.21.15.27:0/1563628653 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19180b2390 0x7f191815d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.936+0000 7f191da00640 1 --2- 172.21.15.27:0/1563628653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191806bb80 0x7f19180c7f90 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.936+0000 7f191da00640 1 -- 172.21.15.27:0/1563628653 >> 172.21.15.27:0/1563628653 conn(0x7f19180b2a10 msgr2=0x7f191815a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:24.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.936+0000 7f191da00640 1 -- 172.21.15.27:0/1563628653 shutdown_connections 2024-01-15T18:07:24.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:24.936+0000 7f191da00640 1 -- 172.21.15.27:0/1563628653 wait complete. 2024-01-15T18:07:24.960 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-01-15T18:07:24.171279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592708,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":187,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":47,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254540,"kb_used_data":5056,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720180,"statfs":{"total":773094113280,"available":772833464320,"internally_reserved":0,"allocated":5177344,"data_stored":2774458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2340,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001241"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.094792+0000","last_change":"2024-01-15T18:07:20.088581+0000","last_active":"2024-01-15T18:07:20.094792+0000","last_peered":"2024-01-15T18:07:20.094792+0000","last_clean":"2024-01-15T18:07:20.094792+0000","last_became_active":"2024-01-15T18:07:20.088247+0000","last_became_peered":"2024-01-15T18:07:20.088247+0000","last_unstale":"2024-01-15T18:07:20.094792+0000","last_undegraded":"2024-01-15T18:07:20.094792+0000","last_fullsized":"2024-01-15T18:07:20.094792+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:04:14.022333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.067098+0000","last_change":"2024-01-15T18:07:19.067098+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.067098+0000","last_undegraded":"2024-01-15T18:07:19.067098+0000","last_fullsized":"2024-01-15T18:07:19.067098+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.099935+0000","last_change":"2024-01-15T18:07:20.094383+0000","last_active":"2024-01-15T18:07:20.099935+0000","last_peered":"2024-01-15T18:07:20.099935+0000","last_clean":"2024-01-15T18:07:20.099935+0000","last_became_active":"2024-01-15T18:07:20.094114+0000","last_became_peered":"2024-01-15T18:07:20.094114+0000","last_unstale":"2024-01-15T18:07:20.099935+0000","last_undegraded":"2024-01-15T18:07:20.099935+0000","last_fullsized":"2024-01-15T18:07:20.099935+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:07:17.293929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.068614+0000","last_change":"2024-01-15T18:07:19.068614+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.068614+0000","last_undegraded":"2024-01-15T18:07:19.068614+0000","last_fullsized":"2024-01-15T18:07:19.068614+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.065311+0000","last_change":"2024-01-15T18:07:19.065311+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.065311+0000","last_undegraded":"2024-01-15T18:07:19.065311+0000","last_fullsized":"2024-01-15T18:07:19.065311+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100427+0000","last_change":"2024-01-15T18:07:20.090463+0000","last_active":"2024-01-15T18:07:20.100427+0000","last_peered":"2024-01-15T18:07:20.100427+0000","last_clean":"2024-01-15T18:07:20.100427+0000","last_became_active":"2024-01-15T18:07:20.090196+0000","last_became_peered":"2024-01-15T18:07:20.090196+0000","last_unstale":"2024-01-15T18:07:20.100427+0000","last_undegraded":"2024-01-15T18:07:20.100427+0000","last_fullsized":"2024-01-15T18:07:20.100427+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:03:21.190309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.106586+0000","last_change":"2024-01-15T18:07:20.102595+0000","last_active":"2024-01-15T18:07:20.106586+0000","last_peered":"2024-01-15T18:07:20.106586+0000","last_clean":"2024-01-15T18:07:20.106586+0000","last_became_active":"2024-01-15T18:07:20.102222+0000","last_became_peered":"2024-01-15T18:07:20.102222+0000","last_unstale":"2024-01-15T18:07:20.106586+0000","last_undegraded":"2024-01-15T18:07:20.106586+0000","last_fullsized":"2024-01-15T18:07:20.106586+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:51:04.157158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.074897+0000","last_change":"2024-01-15T18:07:19.074897+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.074897+0000","last_undegraded":"2024-01-15T18:07:19.074897+0000","last_fullsized":"2024-01-15T18:07:19.074897+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.077258+0000","last_change":"2024-01-15T18:07:19.077258+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.077258+0000","last_undegraded":"2024-01-15T18:07:19.077258+0000","last_fullsized":"2024-01-15T18:07:19.077258+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.070141+0000","last_change":"2024-01-15T18:07:19.070141+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.070141+0000","last_undegraded":"2024-01-15T18:07:19.070141+0000","last_fullsized":"2024-01-15T18:07:19.070141+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.074684+0000","last_change":"2024-01-15T18:07:19.074684+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.074684+0000","last_undegraded":"2024-01-15T18:07:19.074684+0000","last_fullsized":"2024-01-15T18:07:19.074684+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.072234+0000","last_change":"2024-01-15T18:07:19.072234+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.072234+0000","last_undegraded":"2024-01-15T18:07:19.072234+0000","last_fullsized":"2024-01-15T18:07:19.072234+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.106908+0000","last_change":"2024-01-15T18:07:20.093608+0000","last_active":"2024-01-15T18:07:20.106908+0000","last_peered":"2024-01-15T18:07:20.106908+0000","last_clean":"2024-01-15T18:07:20.106908+0000","last_became_active":"2024-01-15T18:07:20.091735+0000","last_became_peered":"2024-01-15T18:07:20.091735+0000","last_unstale":"2024-01-15T18:07:20.106908+0000","last_undegraded":"2024-01-15T18:07:20.106908+0000","last_fullsized":"2024-01-15T18:07:20.106908+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:34.221280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.068436+0000","last_change":"2024-01-15T18:07:19.068436+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.068436+0000","last_undegraded":"2024-01-15T18:07:19.068436+0000","last_fullsized":"2024-01-15T18:07:19.068436+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.105807+0000","last_change":"2024-01-15T18:07:20.096359+0000","last_active":"2024-01-15T18:07:20.105807+0000","last_peered":"2024-01-15T18:07:20.105807+0000","last_clean":"2024-01-15T18:07:20.105807+0000","last_became_active":"2024-01-15T18:07:20.096018+0000","last_became_peered":"2024-01-15T18:07:20.096018+0000","last_unstale":"2024-01-15T18:07:20.105807+0000","last_undegraded":"2024-01-15T18:07:20.105807+0000","last_fullsized":"2024-01-15T18:07:20.105807+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:35:28.043293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100722+0000","last_change":"2024-01-15T18:07:20.097218+0000","last_active":"2024-01-15T18:07:20.100722+0000","last_peered":"2024-01-15T18:07:20.100722+0000","last_clean":"2024-01-15T18:07:20.100722+0000","last_became_active":"2024-01-15T18:07:20.096805+0000","last_became_peered":"2024-01-15T18:07:20.096805+0000","last_unstale":"2024-01-15T18:07:20.100722+0000","last_undegraded":"2024-01-15T18:07:20.100722+0000","last_fullsized":"2024-01-15T18:07:20.100722+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:34.221280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097051+0000","last_change":"2024-01-15T18:07:20.084911+0000","last_active":"2024-01-15T18:07:20.097051+0000","last_peered":"2024-01-15T18:07:20.097051+0000","last_clean":"2024-01-15T18:07:20.097051+0000","last_became_active":"2024-01-15T18:07:20.084594+0000","last_became_peered":"2024-01-15T18:07:20.084594+0000","last_unstale":"2024-01-15T18:07:20.097051+0000","last_undegraded":"2024-01-15T18:07:20.097051+0000","last_fullsized":"2024-01-15T18:07:20.097051+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:38:53.126797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100540+0000","last_change":"2024-01-15T18:07:20.097350+0000","last_active":"2024-01-15T18:07:20.100540+0000","last_peered":"2024-01-15T18:07:20.100540+0000","last_clean":"2024-01-15T18:07:20.100540+0000","last_became_active":"2024-01-15T18:07:20.095556+0000","last_became_peered":"2024-01-15T18:07:20.095556+0000","last_unstale":"2024-01-15T18:07:20.100540+0000","last_undegraded":"2024-01-15T18:07:20.100540+0000","last_fullsized":"2024-01-15T18:07:20.100540+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:01:01.613225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.103149+0000","last_change":"2024-01-15T18:07:20.098939+0000","last_active":"2024-01-15T18:07:20.103149+0000","last_peered":"2024-01-15T18:07:20.103149+0000","last_clean":"2024-01-15T18:07:20.103149+0000","last_became_active":"2024-01-15T18:07:20.098598+0000","last_became_peered":"2024-01-15T18:07:20.098598+0000","last_unstale":"2024-01-15T18:07:20.103149+0000","last_undegraded":"2024-01-15T18:07:20.103149+0000","last_fullsized":"2024-01-15T18:07:20.103149+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:26.049225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.107947+0000","last_change":"2024-01-15T18:07:20.095741+0000","last_active":"2024-01-15T18:07:20.107947+0000","last_peered":"2024-01-15T18:07:20.107947+0000","last_clean":"2024-01-15T18:07:20.107947+0000","last_became_active":"2024-01-15T18:07:20.095417+0000","last_became_peered":"2024-01-15T18:07:20.095417+0000","last_unstale":"2024-01-15T18:07:20.107947+0000","last_undegraded":"2024-01-15T18:07:20.107947+0000","last_fullsized":"2024-01-15T18:07:20.107947+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:32:43.186714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.105539+0000","last_change":"2024-01-15T18:07:20.094321+0000","last_active":"2024-01-15T18:07:20.105539+0000","last_peered":"2024-01-15T18:07:20.105539+0000","last_clean":"2024-01-15T18:07:20.105539+0000","last_became_active":"2024-01-15T18:07:20.094011+0000","last_became_peered":"2024-01-15T18:07:20.094011+0000","last_unstale":"2024-01-15T18:07:20.105539+0000","last_undegraded":"2024-01-15T18:07:20.105539+0000","last_fullsized":"2024-01-15T18:07:20.105539+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:48:24.482600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.067198+0000","last_change":"2024-01-15T18:07:19.067198+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.067198+0000","last_undegraded":"2024-01-15T18:07:19.067198+0000","last_fullsized":"2024-01-15T18:07:19.067198+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098234+0000","last_change":"2024-01-15T18:07:20.087728+0000","last_active":"2024-01-15T18:07:20.098234+0000","last_peered":"2024-01-15T18:07:20.098234+0000","last_clean":"2024-01-15T18:07:20.098234+0000","last_became_active":"2024-01-15T18:07:20.087118+0000","last_became_peered":"2024-01-15T18:07:20.087118+0000","last_unstale":"2024-01-15T18:07:20.098234+0000","last_undegraded":"2024-01-15T18:07:20.098234+0000","last_fullsized":"2024-01-15T18:07:20.098234+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:32:43.186714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100694+0000","last_change":"2024-01-15T18:07:20.092361+0000","last_active":"2024-01-15T18:07:20.100694+0000","last_peered":"2024-01-15T18:07:20.100694+0000","last_clean":"2024-01-15T18:07:20.100694+0000","last_became_active":"2024-01-15T18:07:20.091927+0000","last_became_peered":"2024-01-15T18:07:20.091927+0000","last_unstale":"2024-01-15T18:07:20.100694+0000","last_undegraded":"2024-01-15T18:07:20.100694+0000","last_fullsized":"2024-01-15T18:07:20.100694+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:55:19.615935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.106512+0000","last_change":"2024-01-15T18:07:20.102013+0000","last_active":"2024-01-15T18:07:20.106512+0000","last_peered":"2024-01-15T18:07:20.106512+0000","last_clean":"2024-01-15T18:07:20.106512+0000","last_became_active":"2024-01-15T18:07:20.101697+0000","last_became_peered":"2024-01-15T18:07:20.101697+0000","last_unstale":"2024-01-15T18:07:20.106512+0000","last_undegraded":"2024-01-15T18:07:20.106512+0000","last_fullsized":"2024-01-15T18:07:20.106512+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:32:14.924018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.103301+0000","last_change":"2024-01-15T18:07:20.086187+0000","last_active":"2024-01-15T18:07:20.103301+0000","last_peered":"2024-01-15T18:07:20.103301+0000","last_clean":"2024-01-15T18:07:20.103301+0000","last_became_active":"2024-01-15T18:07:20.085891+0000","last_became_peered":"2024-01-15T18:07:20.085891+0000","last_unstale":"2024-01-15T18:07:20.103301+0000","last_undegraded":"2024-01-15T18:07:20.103301+0000","last_fullsized":"2024-01-15T18:07:20.103301+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:54:09.185168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097020+0000","last_change":"2024-01-15T18:07:20.086602+0000","last_active":"2024-01-15T18:07:20.097020+0000","last_peered":"2024-01-15T18:07:20.097020+0000","last_clean":"2024-01-15T18:07:20.097020+0000","last_became_active":"2024-01-15T18:07:20.086238+0000","last_became_peered":"2024-01-15T18:07:20.086238+0000","last_unstale":"2024-01-15T18:07:20.097020+0000","last_undegraded":"2024-01-15T18:07:20.097020+0000","last_fullsized":"2024-01-15T18:07:20.097020+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:20:04.640555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.071723+0000","last_change":"2024-01-15T18:07:19.071723+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.071723+0000","last_undegraded":"2024-01-15T18:07:19.071723+0000","last_fullsized":"2024-01-15T18:07:19.071723+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.103004+0000","last_change":"2024-01-15T18:07:20.097835+0000","last_active":"2024-01-15T18:07:20.103004+0000","last_peered":"2024-01-15T18:07:20.103004+0000","last_clean":"2024-01-15T18:07:20.103004+0000","last_became_active":"2024-01-15T18:07:20.097477+0000","last_became_peered":"2024-01-15T18:07:20.097477+0000","last_unstale":"2024-01-15T18:07:20.103004+0000","last_undegraded":"2024-01-15T18:07:20.103004+0000","last_fullsized":"2024-01-15T18:07:20.103004+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:36:51.036352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097114+0000","last_change":"2024-01-15T18:07:20.083756+0000","last_active":"2024-01-15T18:07:20.097114+0000","last_peered":"2024-01-15T18:07:20.097114+0000","last_clean":"2024-01-15T18:07:20.097114+0000","last_became_active":"2024-01-15T18:07:20.079072+0000","last_became_peered":"2024-01-15T18:07:20.079072+0000","last_unstale":"2024-01-15T18:07:20.097114+0000","last_undegraded":"2024-01-15T18:07:20.097114+0000","last_fullsized":"2024-01-15T18:07:20.097114+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:55:26.194143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.105788+0000","last_change":"2024-01-15T18:07:20.096208+0000","last_active":"2024-01-15T18:07:20.105788+0000","last_peered":"2024-01-15T18:07:20.105788+0000","last_clean":"2024-01-15T18:07:20.105788+0000","last_became_active":"2024-01-15T18:07:20.095900+0000","last_became_peered":"2024-01-15T18:07:20.095900+0000","last_unstale":"2024-01-15T18:07:20.105788+0000","last_undegraded":"2024-01-15T18:07:20.105788+0000","last_fullsized":"2024-01-15T18:07:20.105788+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:50:12.413904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.106884+0000","last_change":"2024-01-15T18:07:20.091218+0000","last_active":"2024-01-15T18:07:20.106884+0000","last_peered":"2024-01-15T18:07:20.106884+0000","last_clean":"2024-01-15T18:07:20.106884+0000","last_became_active":"2024-01-15T18:07:20.090896+0000","last_became_peered":"2024-01-15T18:07:20.090896+0000","last_unstale":"2024-01-15T18:07:20.106884+0000","last_undegraded":"2024-01-15T18:07:20.106884+0000","last_fullsized":"2024-01-15T18:07:20.106884+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:54:56.923677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098011+0000","last_change":"2024-01-15T18:07:20.087031+0000","last_active":"2024-01-15T18:07:20.098011+0000","last_peered":"2024-01-15T18:07:20.098011+0000","last_clean":"2024-01-15T18:07:20.098011+0000","last_became_active":"2024-01-15T18:07:20.086158+0000","last_became_peered":"2024-01-15T18:07:20.086158+0000","last_unstale":"2024-01-15T18:07:20.098011+0000","last_undegraded":"2024-01-15T18:07:20.098011+0000","last_fullsized":"2024-01-15T18:07:20.098011+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:55:26.194143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098945+0000","last_change":"2024-01-15T18:07:19.085120+0000","last_active":"2024-01-15T18:07:20.098945+0000","last_peered":"2024-01-15T18:07:20.098945+0000","last_clean":"2024-01-15T18:07:20.098945+0000","last_became_active":"2024-01-15T18:07:19.084785+0000","last_became_peered":"2024-01-15T18:07:19.084785+0000","last_unstale":"2024-01-15T18:07:20.098945+0000","last_undegraded":"2024-01-15T18:07:20.098945+0000","last_fullsized":"2024-01-15T18:07:20.098945+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:50:51.007855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100675+0000","last_change":"2024-01-15T18:07:20.094786+0000","last_active":"2024-01-15T18:07:20.100675+0000","last_peered":"2024-01-15T18:07:20.100675+0000","last_clean":"2024-01-15T18:07:20.100675+0000","last_became_active":"2024-01-15T18:07:20.094523+0000","last_became_peered":"2024-01-15T18:07:20.094523+0000","last_unstale":"2024-01-15T18:07:20.100675+0000","last_undegraded":"2024-01-15T18:07:20.100675+0000","last_fullsized":"2024-01-15T18:07:20.100675+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:59:01.979409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.096749+0000","last_change":"2024-01-15T18:07:19.084713+0000","last_active":"2024-01-15T18:07:20.096749+0000","last_peered":"2024-01-15T18:07:20.096749+0000","last_clean":"2024-01-15T18:07:20.096749+0000","last_became_active":"2024-01-15T18:07:19.084414+0000","last_became_peered":"2024-01-15T18:07:19.084414+0000","last_unstale":"2024-01-15T18:07:20.096749+0000","last_undegraded":"2024-01-15T18:07:20.096749+0000","last_fullsized":"2024-01-15T18:07:20.096749+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:20.019967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.068126+0000","last_change":"2024-01-15T18:07:19.068126+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.068126+0000","last_undegraded":"2024-01-15T18:07:19.068126+0000","last_fullsized":"2024-01-15T18:07:19.068126+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100965+0000","last_change":"2024-01-15T18:07:19.077287+0000","last_active":"2024-01-15T18:07:20.100965+0000","last_peered":"2024-01-15T18:07:20.100965+0000","last_clean":"2024-01-15T18:07:20.100965+0000","last_became_active":"2024-01-15T18:07:19.075652+0000","last_became_peered":"2024-01-15T18:07:19.075652+0000","last_unstale":"2024-01-15T18:07:20.100965+0000","last_undegraded":"2024-01-15T18:07:20.100965+0000","last_fullsized":"2024-01-15T18:07:20.100965+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:57:56.264139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098517+0000","last_change":"2024-01-15T18:07:20.087874+0000","last_active":"2024-01-15T18:07:20.098517+0000","last_peered":"2024-01-15T18:07:20.098517+0000","last_clean":"2024-01-15T18:07:20.098517+0000","last_became_active":"2024-01-15T18:07:20.087579+0000","last_became_peered":"2024-01-15T18:07:20.087579+0000","last_unstale":"2024-01-15T18:07:20.098517+0000","last_undegraded":"2024-01-15T18:07:20.098517+0000","last_fullsized":"2024-01-15T18:07:20.098517+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:20:04.640555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098218+0000","last_change":"2024-01-15T18:07:19.085001+0000","last_active":"2024-01-15T18:07:20.098218+0000","last_peered":"2024-01-15T18:07:20.098218+0000","last_clean":"2024-01-15T18:07:20.098218+0000","last_became_active":"2024-01-15T18:07:19.084651+0000","last_became_peered":"2024-01-15T18:07:19.084651+0000","last_unstale":"2024-01-15T18:07:20.098218+0000","last_undegraded":"2024-01-15T18:07:20.098218+0000","last_fullsized":"2024-01-15T18:07:20.098218+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:13:24.552254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097208+0000","last_change":"2024-01-15T18:07:20.087693+0000","last_active":"2024-01-15T18:07:20.097208+0000","last_peered":"2024-01-15T18:07:20.097208+0000","last_clean":"2024-01-15T18:07:20.097208+0000","last_became_active":"2024-01-15T18:07:20.087416+0000","last_became_peered":"2024-01-15T18:07:20.087416+0000","last_unstale":"2024-01-15T18:07:20.097208+0000","last_undegraded":"2024-01-15T18:07:20.097208+0000","last_fullsized":"2024-01-15T18:07:20.097208+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:02.825993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097250+0000","last_change":"2024-01-15T18:07:19.086152+0000","last_active":"2024-01-15T18:07:20.097250+0000","last_peered":"2024-01-15T18:07:20.097250+0000","last_clean":"2024-01-15T18:07:20.097250+0000","last_became_active":"2024-01-15T18:07:19.082058+0000","last_became_peered":"2024-01-15T18:07:19.082058+0000","last_unstale":"2024-01-15T18:07:20.097250+0000","last_undegraded":"2024-01-15T18:07:20.097250+0000","last_fullsized":"2024-01-15T18:07:20.097250+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:06:20.719015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.074401+0000","last_change":"2024-01-15T18:07:19.074401+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.074401+0000","last_undegraded":"2024-01-15T18:07:19.074401+0000","last_fullsized":"2024-01-15T18:07:19.074401+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097139+0000","last_change":"2024-01-15T18:07:19.076932+0000","last_active":"2024-01-15T18:07:20.097139+0000","last_peered":"2024-01-15T18:07:20.097139+0000","last_clean":"2024-01-15T18:07:20.097139+0000","last_became_active":"2024-01-15T18:07:19.076629+0000","last_became_peered":"2024-01-15T18:07:19.076629+0000","last_unstale":"2024-01-15T18:07:20.097139+0000","last_undegraded":"2024-01-15T18:07:20.097139+0000","last_fullsized":"2024-01-15T18:07:20.097139+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:24:59.066289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.069432+0000","last_change":"2024-01-15T18:07:19.069432+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.069432+0000","last_undegraded":"2024-01-15T18:07:19.069432+0000","last_fullsized":"2024-01-15T18:07:19.069432+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.084299+0000","last_change":"2024-01-15T18:07:19.079711+0000","last_active":"2024-01-15T18:07:19.084299+0000","last_peered":"2024-01-15T18:07:19.084299+0000","last_clean":"2024-01-15T18:07:19.084299+0000","last_became_active":"2024-01-15T18:07:19.079336+0000","last_became_peered":"2024-01-15T18:07:19.079336+0000","last_unstale":"2024-01-15T18:07:19.084299+0000","last_undegraded":"2024-01-15T18:07:19.084299+0000","last_fullsized":"2024-01-15T18:07:19.084299+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:49:49.489371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098255+0000","last_change":"2024-01-15T18:07:20.087900+0000","last_active":"2024-01-15T18:07:20.098255+0000","last_peered":"2024-01-15T18:07:20.098255+0000","last_clean":"2024-01-15T18:07:20.098255+0000","last_became_active":"2024-01-15T18:07:20.087463+0000","last_became_peered":"2024-01-15T18:07:20.087463+0000","last_unstale":"2024-01-15T18:07:20.098255+0000","last_undegraded":"2024-01-15T18:07:20.098255+0000","last_fullsized":"2024-01-15T18:07:20.098255+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:54:09.185168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.082790+0000","last_change":"2024-01-15T18:07:19.079615+0000","last_active":"2024-01-15T18:07:19.082790+0000","last_peered":"2024-01-15T18:07:19.082790+0000","last_clean":"2024-01-15T18:07:19.082790+0000","last_became_active":"2024-01-15T18:07:19.079242+0000","last_became_peered":"2024-01-15T18:07:19.079242+0000","last_unstale":"2024-01-15T18:07:19.082790+0000","last_undegraded":"2024-01-15T18:07:19.082790+0000","last_fullsized":"2024-01-15T18:07:19.082790+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:47.335570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100426+0000","last_change":"2024-01-15T18:07:20.095844+0000","last_active":"2024-01-15T18:07:20.100426+0000","last_peered":"2024-01-15T18:07:20.100426+0000","last_clean":"2024-01-15T18:07:20.100426+0000","last_became_active":"2024-01-15T18:07:20.095110+0000","last_became_peered":"2024-01-15T18:07:20.095110+0000","last_unstale":"2024-01-15T18:07:20.100426+0000","last_undegraded":"2024-01-15T18:07:20.100426+0000","last_fullsized":"2024-01-15T18:07:20.100426+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:57:57.266821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.070487+0000","last_change":"2024-01-15T18:07:19.076938+0000","last_active":"2024-01-15T18:07:20.070487+0000","last_peered":"2024-01-15T18:07:20.070487+0000","last_clean":"2024-01-15T18:07:20.070487+0000","last_became_active":"2024-01-15T18:07:19.076605+0000","last_became_peered":"2024-01-15T18:07:19.076605+0000","last_unstale":"2024-01-15T18:07:20.070487+0000","last_undegraded":"2024-01-15T18:07:20.070487+0000","last_fullsized":"2024-01-15T18:07:20.070487+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:32.749313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.095519+0000","last_change":"2024-01-15T18:07:20.091131+0000","last_active":"2024-01-15T18:07:20.095519+0000","last_peered":"2024-01-15T18:07:20.095519+0000","last_clean":"2024-01-15T18:07:20.095519+0000","last_became_active":"2024-01-15T18:07:20.090683+0000","last_became_peered":"2024-01-15T18:07:20.090683+0000","last_unstale":"2024-01-15T18:07:20.095519+0000","last_undegraded":"2024-01-15T18:07:20.095519+0000","last_fullsized":"2024-01-15T18:07:20.095519+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:36:14.191387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.086659+0000","last_change":"2024-01-15T18:07:19.085014+0000","last_active":"2024-01-15T18:07:19.086659+0000","last_peered":"2024-01-15T18:07:19.086659+0000","last_clean":"2024-01-15T18:07:19.086659+0000","last_became_active":"2024-01-15T18:07:19.084470+0000","last_became_peered":"2024-01-15T18:07:19.084470+0000","last_unstale":"2024-01-15T18:07:19.086659+0000","last_undegraded":"2024-01-15T18:07:19.086659+0000","last_fullsized":"2024-01-15T18:07:19.086659+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:43:21.743406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.114219+0000","last_change":"2024-01-15T18:07:20.108821+0000","last_active":"2024-01-15T18:07:20.114219+0000","last_peered":"2024-01-15T18:07:20.114219+0000","last_clean":"2024-01-15T18:07:20.114219+0000","last_became_active":"2024-01-15T18:07:20.108464+0000","last_became_peered":"2024-01-15T18:07:20.108464+0000","last_unstale":"2024-01-15T18:07:20.114219+0000","last_undegraded":"2024-01-15T18:07:20.114219+0000","last_fullsized":"2024-01-15T18:07:20.114219+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:23:28.608885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"44'3","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.099259+0000","last_change":"2024-01-15T18:07:19.088202+0000","last_active":"2024-01-15T18:07:20.099259+0000","last_peered":"2024-01-15T18:07:20.099259+0000","last_clean":"2024-01-15T18:07:20.099259+0000","last_became_active":"2024-01-15T18:07:19.087900+0000","last_became_peered":"2024-01-15T18:07:19.087900+0000","last_unstale":"2024-01-15T18:07:20.099259+0000","last_undegraded":"2024-01-15T18:07:20.099259+0000","last_fullsized":"2024-01-15T18:07:20.099259+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:10:42.345570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.106599+0000","last_change":"2024-01-15T18:07:20.102897+0000","last_active":"2024-01-15T18:07:20.106599+0000","last_peered":"2024-01-15T18:07:20.106599+0000","last_clean":"2024-01-15T18:07:20.106599+0000","last_became_active":"2024-01-15T18:07:20.102579+0000","last_became_peered":"2024-01-15T18:07:20.102579+0000","last_unstale":"2024-01-15T18:07:20.106599+0000","last_undegraded":"2024-01-15T18:07:20.106599+0000","last_fullsized":"2024-01-15T18:07:20.106599+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:50:12.413904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":17,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.112054+0000","last_change":"2024-01-15T18:07:19.079571+0000","last_active":"2024-01-15T18:07:20.112054+0000","last_peered":"2024-01-15T18:07:20.112054+0000","last_clean":"2024-01-15T18:07:20.112054+0000","last_became_active":"2024-01-15T18:07:19.079308+0000","last_became_peered":"2024-01-15T18:07:19.079308+0000","last_unstale":"2024-01-15T18:07:20.112054+0000","last_undegraded":"2024-01-15T18:07:20.112054+0000","last_fullsized":"2024-01-15T18:07:20.112054+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:25.046543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.101579+0000","last_change":"2024-01-15T18:07:20.097611+0000","last_active":"2024-01-15T18:07:20.101579+0000","last_peered":"2024-01-15T18:07:20.101579+0000","last_clean":"2024-01-15T18:07:20.101579+0000","last_became_active":"2024-01-15T18:07:20.097202+0000","last_became_peered":"2024-01-15T18:07:20.097202+0000","last_unstale":"2024-01-15T18:07:20.101579+0000","last_undegraded":"2024-01-15T18:07:20.101579+0000","last_fullsized":"2024-01-15T18:07:20.101579+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:52:34.679712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.104292+0000","last_change":"2024-01-15T18:07:19.084662+0000","last_active":"2024-01-15T18:07:20.104292+0000","last_peered":"2024-01-15T18:07:20.104292+0000","last_clean":"2024-01-15T18:07:20.104292+0000","last_became_active":"2024-01-15T18:07:19.084348+0000","last_became_peered":"2024-01-15T18:07:19.084348+0000","last_unstale":"2024-01-15T18:07:20.104292+0000","last_undegraded":"2024-01-15T18:07:20.104292+0000","last_fullsized":"2024-01-15T18:07:20.104292+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:07:49.351274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.067641+0000","last_change":"2024-01-15T18:07:19.067641+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.067641+0000","last_undegraded":"2024-01-15T18:07:19.067641+0000","last_fullsized":"2024-01-15T18:07:19.067641+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.076930+0000","last_change":"2024-01-15T18:07:19.084512+0000","last_active":"2024-01-15T18:07:20.076930+0000","last_peered":"2024-01-15T18:07:20.076930+0000","last_clean":"2024-01-15T18:07:20.076930+0000","last_became_active":"2024-01-15T18:07:19.084169+0000","last_became_peered":"2024-01-15T18:07:19.084169+0000","last_unstale":"2024-01-15T18:07:20.076930+0000","last_undegraded":"2024-01-15T18:07:20.076930+0000","last_fullsized":"2024-01-15T18:07:20.076930+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:35:27.040611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.109334+0000","last_change":"2024-01-15T18:07:20.105484+0000","last_active":"2024-01-15T18:07:20.109334+0000","last_peered":"2024-01-15T18:07:20.109334+0000","last_clean":"2024-01-15T18:07:20.109334+0000","last_became_active":"2024-01-15T18:07:20.105160+0000","last_became_peered":"2024-01-15T18:07:20.105160+0000","last_unstale":"2024-01-15T18:07:20.109334+0000","last_undegraded":"2024-01-15T18:07:20.109334+0000","last_fullsized":"2024-01-15T18:07:20.109334+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:04:14.022333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.073804+0000","last_change":"2024-01-15T18:07:19.083510+0000","last_active":"2024-01-15T18:07:20.073804+0000","last_peered":"2024-01-15T18:07:20.073804+0000","last_clean":"2024-01-15T18:07:20.073804+0000","last_became_active":"2024-01-15T18:07:19.083134+0000","last_became_peered":"2024-01-15T18:07:19.083134+0000","last_unstale":"2024-01-15T18:07:20.073804+0000","last_undegraded":"2024-01-15T18:07:20.073804+0000","last_fullsized":"2024-01-15T18:07:20.073804+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:10:42.345570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.103258+0000","last_change":"2024-01-15T18:07:20.101397+0000","last_active":"2024-01-15T18:07:20.103258+0000","last_peered":"2024-01-15T18:07:20.103258+0000","last_clean":"2024-01-15T18:07:20.103258+0000","last_became_active":"2024-01-15T18:07:20.101106+0000","last_became_peered":"2024-01-15T18:07:20.101106+0000","last_unstale":"2024-01-15T18:07:20.103258+0000","last_undegraded":"2024-01-15T18:07:20.103258+0000","last_fullsized":"2024-01-15T18:07:20.103258+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:48:24.482600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.088352+0000","last_change":"2024-01-15T18:07:19.086583+0000","last_active":"2024-01-15T18:07:19.088352+0000","last_peered":"2024-01-15T18:07:19.088352+0000","last_clean":"2024-01-15T18:07:19.088352+0000","last_became_active":"2024-01-15T18:07:19.086227+0000","last_became_peered":"2024-01-15T18:07:19.086227+0000","last_unstale":"2024-01-15T18:07:19.088352+0000","last_undegraded":"2024-01-15T18:07:19.088352+0000","last_fullsized":"2024-01-15T18:07:19.088352+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:24:53.993680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.102535+0000","last_change":"2024-01-15T18:07:20.075833+0000","last_active":"2024-01-15T18:07:20.102535+0000","last_peered":"2024-01-15T18:07:20.102535+0000","last_clean":"2024-01-15T18:07:20.102535+0000","last_became_active":"2024-01-15T18:07:20.075521+0000","last_became_peered":"2024-01-15T18:07:20.075521+0000","last_unstale":"2024-01-15T18:07:20.102535+0000","last_undegraded":"2024-01-15T18:07:20.102535+0000","last_fullsized":"2024-01-15T18:07:20.102535+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:52:34.679712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.107096+0000","last_change":"2024-01-15T18:07:19.087887+0000","last_active":"2024-01-15T18:07:20.107096+0000","last_peered":"2024-01-15T18:07:20.107096+0000","last_clean":"2024-01-15T18:07:20.107096+0000","last_became_active":"2024-01-15T18:07:19.087610+0000","last_became_peered":"2024-01-15T18:07:19.087610+0000","last_unstale":"2024-01-15T18:07:20.107096+0000","last_undegraded":"2024-01-15T18:07:20.107096+0000","last_fullsized":"2024-01-15T18:07:20.107096+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:29:39.445420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.068689+0000","last_change":"2024-01-15T18:07:19.068689+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.068689+0000","last_undegraded":"2024-01-15T18:07:19.068689+0000","last_fullsized":"2024-01-15T18:07:19.068689+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.101388+0000","last_change":"2024-01-15T18:07:19.074682+0000","last_active":"2024-01-15T18:07:20.101388+0000","last_peered":"2024-01-15T18:07:20.101388+0000","last_clean":"2024-01-15T18:07:20.101388+0000","last_became_active":"2024-01-15T18:07:19.074397+0000","last_became_peered":"2024-01-15T18:07:19.074397+0000","last_unstale":"2024-01-15T18:07:20.101388+0000","last_undegraded":"2024-01-15T18:07:20.101388+0000","last_fullsized":"2024-01-15T18:07:20.101388+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:37:09.401497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100562+0000","last_change":"2024-01-15T18:07:20.093962+0000","last_active":"2024-01-15T18:07:20.100562+0000","last_peered":"2024-01-15T18:07:20.100562+0000","last_clean":"2024-01-15T18:07:20.100562+0000","last_became_active":"2024-01-15T18:07:20.093638+0000","last_became_peered":"2024-01-15T18:07:20.093638+0000","last_unstale":"2024-01-15T18:07:20.100562+0000","last_undegraded":"2024-01-15T18:07:20.100562+0000","last_fullsized":"2024-01-15T18:07:20.100562+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:30:43.231075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.086984+0000","last_change":"2024-01-15T18:07:19.082541+0000","last_active":"2024-01-15T18:07:19.086984+0000","last_peered":"2024-01-15T18:07:19.086984+0000","last_clean":"2024-01-15T18:07:19.086984+0000","last_became_active":"2024-01-15T18:07:19.082192+0000","last_became_peered":"2024-01-15T18:07:19.082192+0000","last_unstale":"2024-01-15T18:07:19.086984+0000","last_undegraded":"2024-01-15T18:07:19.086984+0000","last_fullsized":"2024-01-15T18:07:19.086984+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:10:51.637250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.103168+0000","last_change":"2024-01-15T18:07:20.096891+0000","last_active":"2024-01-15T18:07:20.103168+0000","last_peered":"2024-01-15T18:07:20.103168+0000","last_clean":"2024-01-15T18:07:20.103168+0000","last_became_active":"2024-01-15T18:07:20.096533+0000","last_became_peered":"2024-01-15T18:07:20.096533+0000","last_unstale":"2024-01-15T18:07:20.103168+0000","last_undegraded":"2024-01-15T18:07:20.103168+0000","last_fullsized":"2024-01-15T18:07:20.103168+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:14:41.208287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.075980+0000","last_change":"2024-01-15T18:07:19.080295+0000","last_active":"2024-01-15T18:07:20.075980+0000","last_peered":"2024-01-15T18:07:20.075980+0000","last_clean":"2024-01-15T18:07:20.075980+0000","last_became_active":"2024-01-15T18:07:19.080024+0000","last_became_peered":"2024-01-15T18:07:19.080024+0000","last_unstale":"2024-01-15T18:07:20.075980+0000","last_undegraded":"2024-01-15T18:07:20.075980+0000","last_fullsized":"2024-01-15T18:07:20.075980+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:36:48.732986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.112236+0000","last_change":"2024-01-15T18:07:20.105757+0000","last_active":"2024-01-15T18:07:20.112236+0000","last_peered":"2024-01-15T18:07:20.112236+0000","last_clean":"2024-01-15T18:07:20.112236+0000","last_became_active":"2024-01-15T18:07:20.105385+0000","last_became_peered":"2024-01-15T18:07:20.105385+0000","last_unstale":"2024-01-15T18:07:20.112236+0000","last_undegraded":"2024-01-15T18:07:20.112236+0000","last_fullsized":"2024-01-15T18:07:20.112236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:51:04.157158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097287+0000","last_change":"2024-01-15T18:07:19.082297+0000","last_active":"2024-01-15T18:07:20.097287+0000","last_peered":"2024-01-15T18:07:20.097287+0000","last_clean":"2024-01-15T18:07:20.097287+0000","last_became_active":"2024-01-15T18:07:19.081971+0000","last_became_peered":"2024-01-15T18:07:19.081971+0000","last_unstale":"2024-01-15T18:07:20.097287+0000","last_undegraded":"2024-01-15T18:07:20.097287+0000","last_fullsized":"2024-01-15T18:07:20.097287+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:23:41.478738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098551+0000","last_change":"2024-01-15T18:07:20.091112+0000","last_active":"2024-01-15T18:07:20.098551+0000","last_peered":"2024-01-15T18:07:20.098551+0000","last_clean":"2024-01-15T18:07:20.098551+0000","last_became_active":"2024-01-15T18:07:20.090802+0000","last_became_peered":"2024-01-15T18:07:20.090802+0000","last_unstale":"2024-01-15T18:07:20.098551+0000","last_undegraded":"2024-01-15T18:07:20.098551+0000","last_fullsized":"2024-01-15T18:07:20.098551+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:01:01.613225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.082669+0000","last_change":"2024-01-15T18:07:19.077896+0000","last_active":"2024-01-15T18:07:19.082669+0000","last_peered":"2024-01-15T18:07:19.082669+0000","last_clean":"2024-01-15T18:07:19.082669+0000","last_became_active":"2024-01-15T18:07:19.077439+0000","last_became_peered":"2024-01-15T18:07:19.077439+0000","last_unstale":"2024-01-15T18:07:19.082669+0000","last_undegraded":"2024-01-15T18:07:19.082669+0000","last_fullsized":"2024-01-15T18:07:19.082669+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:09:06.512495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.104003+0000","last_change":"2024-01-15T18:07:20.086589+0000","last_active":"2024-01-15T18:07:20.104003+0000","last_peered":"2024-01-15T18:07:20.104003+0000","last_clean":"2024-01-15T18:07:20.104003+0000","last_became_active":"2024-01-15T18:07:20.086312+0000","last_became_peered":"2024-01-15T18:07:20.086312+0000","last_unstale":"2024-01-15T18:07:20.104003+0000","last_undegraded":"2024-01-15T18:07:20.104003+0000","last_fullsized":"2024-01-15T18:07:20.104003+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:15:59.320681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.083833+0000","last_change":"2024-01-15T18:07:19.082155+0000","last_active":"2024-01-15T18:07:19.083833+0000","last_peered":"2024-01-15T18:07:19.083833+0000","last_clean":"2024-01-15T18:07:19.083833+0000","last_became_active":"2024-01-15T18:07:19.081855+0000","last_became_peered":"2024-01-15T18:07:19.081855+0000","last_unstale":"2024-01-15T18:07:19.083833+0000","last_undegraded":"2024-01-15T18:07:19.083833+0000","last_fullsized":"2024-01-15T18:07:19.083833+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:07.014633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.067698+0000","last_change":"2024-01-15T18:07:19.067698+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.067698+0000","last_undegraded":"2024-01-15T18:07:19.067698+0000","last_fullsized":"2024-01-15T18:07:19.067698+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.076645+0000","last_change":"2024-01-15T18:07:19.085582+0000","last_active":"2024-01-15T18:07:20.076645+0000","last_peered":"2024-01-15T18:07:20.076645+0000","last_clean":"2024-01-15T18:07:20.076645+0000","last_became_active":"2024-01-15T18:07:19.083796+0000","last_became_peered":"2024-01-15T18:07:19.083796+0000","last_unstale":"2024-01-15T18:07:20.076645+0000","last_undegraded":"2024-01-15T18:07:20.076645+0000","last_fullsized":"2024-01-15T18:07:20.076645+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:03:20.187627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.106501+0000","last_change":"2024-01-15T18:07:20.100547+0000","last_active":"2024-01-15T18:07:20.106501+0000","last_peered":"2024-01-15T18:07:20.106501+0000","last_clean":"2024-01-15T18:07:20.106501+0000","last_became_active":"2024-01-15T18:07:20.100201+0000","last_became_peered":"2024-01-15T18:07:20.100201+0000","last_unstale":"2024-01-15T18:07:20.106501+0000","last_undegraded":"2024-01-15T18:07:20.106501+0000","last_fullsized":"2024-01-15T18:07:20.106501+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:29:40.448102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.072838+0000","last_change":"2024-01-15T18:07:19.078965+0000","last_active":"2024-01-15T18:07:20.072838+0000","last_peered":"2024-01-15T18:07:20.072838+0000","last_clean":"2024-01-15T18:07:20.072838+0000","last_became_active":"2024-01-15T18:07:19.078692+0000","last_became_peered":"2024-01-15T18:07:19.078692+0000","last_unstale":"2024-01-15T18:07:20.072838+0000","last_undegraded":"2024-01-15T18:07:20.072838+0000","last_fullsized":"2024-01-15T18:07:20.072838+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:13:24.552254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.067320+0000","last_change":"2024-01-15T18:07:19.067320+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.067320+0000","last_undegraded":"2024-01-15T18:07:19.067320+0000","last_fullsized":"2024-01-15T18:07:19.067320+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.077286+0000","last_change":"2024-01-15T18:07:19.088174+0000","last_active":"2024-01-15T18:07:20.077286+0000","last_peered":"2024-01-15T18:07:20.077286+0000","last_clean":"2024-01-15T18:07:20.077286+0000","last_became_active":"2024-01-15T18:07:19.087881+0000","last_became_peered":"2024-01-15T18:07:19.087881+0000","last_unstale":"2024-01-15T18:07:20.077286+0000","last_undegraded":"2024-01-15T18:07:20.077286+0000","last_fullsized":"2024-01-15T18:07:20.077286+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:36:48.732986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.108471+0000","last_change":"2024-01-15T18:07:20.104788+0000","last_active":"2024-01-15T18:07:20.108471+0000","last_peered":"2024-01-15T18:07:20.108471+0000","last_clean":"2024-01-15T18:07:20.108471+0000","last_became_active":"2024-01-15T18:07:20.104475+0000","last_became_peered":"2024-01-15T18:07:20.104475+0000","last_unstale":"2024-01-15T18:07:20.108471+0000","last_undegraded":"2024-01-15T18:07:20.108471+0000","last_fullsized":"2024-01-15T18:07:20.108471+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:02:26.277338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.071516+0000","last_change":"2024-01-15T18:07:19.080590+0000","last_active":"2024-01-15T18:07:20.071516+0000","last_peered":"2024-01-15T18:07:20.071516+0000","last_clean":"2024-01-15T18:07:20.071516+0000","last_became_active":"2024-01-15T18:07:19.080276+0000","last_became_peered":"2024-01-15T18:07:19.080276+0000","last_unstale":"2024-01-15T18:07:20.071516+0000","last_undegraded":"2024-01-15T18:07:20.071516+0000","last_fullsized":"2024-01-15T18:07:20.071516+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:24:59.066289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100402+0000","last_change":"2024-01-15T18:07:20.090127+0000","last_active":"2024-01-15T18:07:20.100402+0000","last_peered":"2024-01-15T18:07:20.100402+0000","last_clean":"2024-01-15T18:07:20.100402+0000","last_became_active":"2024-01-15T18:07:20.089727+0000","last_became_peered":"2024-01-15T18:07:20.089727+0000","last_unstale":"2024-01-15T18:07:20.100402+0000","last_undegraded":"2024-01-15T18:07:20.100402+0000","last_fullsized":"2024-01-15T18:07:20.100402+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:36:14.191387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.077270+0000","last_change":"2024-01-15T18:07:19.088037+0000","last_active":"2024-01-15T18:07:20.077270+0000","last_peered":"2024-01-15T18:07:20.077270+0000","last_clean":"2024-01-15T18:07:20.077270+0000","last_became_active":"2024-01-15T18:07:19.087721+0000","last_became_peered":"2024-01-15T18:07:19.087721+0000","last_unstale":"2024-01-15T18:07:20.077270+0000","last_undegraded":"2024-01-15T18:07:20.077270+0000","last_fullsized":"2024-01-15T18:07:20.077270+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:23:27.606202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.111942+0000","last_change":"2024-01-15T18:07:20.105653+0000","last_active":"2024-01-15T18:07:20.111942+0000","last_peered":"2024-01-15T18:07:20.111942+0000","last_clean":"2024-01-15T18:07:20.111942+0000","last_became_active":"2024-01-15T18:07:20.105348+0000","last_became_peered":"2024-01-15T18:07:20.105348+0000","last_unstale":"2024-01-15T18:07:20.111942+0000","last_undegraded":"2024-01-15T18:07:20.111942+0000","last_fullsized":"2024-01-15T18:07:20.111942+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:02:26.277338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.068819+0000","last_change":"2024-01-15T18:07:19.076027+0000","last_active":"2024-01-15T18:07:20.068819+0000","last_peered":"2024-01-15T18:07:20.068819+0000","last_clean":"2024-01-15T18:07:20.068819+0000","last_became_active":"2024-01-15T18:07:19.075644+0000","last_became_peered":"2024-01-15T18:07:19.075644+0000","last_unstale":"2024-01-15T18:07:20.068819+0000","last_undegraded":"2024-01-15T18:07:20.068819+0000","last_fullsized":"2024-01-15T18:07:20.068819+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:07:57.062677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.064957+0000","last_change":"2024-01-15T18:07:19.064957+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.064957+0000","last_undegraded":"2024-01-15T18:07:19.064957+0000","last_fullsized":"2024-01-15T18:07:19.064957+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.072048+0000","last_change":"2024-01-15T18:07:19.081215+0000","last_active":"2024-01-15T18:07:20.072048+0000","last_peered":"2024-01-15T18:07:20.072048+0000","last_clean":"2024-01-15T18:07:20.072048+0000","last_became_active":"2024-01-15T18:07:19.080901+0000","last_became_peered":"2024-01-15T18:07:19.080901+0000","last_unstale":"2024-01-15T18:07:20.072048+0000","last_undegraded":"2024-01-15T18:07:20.072048+0000","last_fullsized":"2024-01-15T18:07:20.072048+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:38:52.124114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.094487+0000","last_change":"2024-01-15T18:07:20.081122+0000","last_active":"2024-01-15T18:07:20.094487+0000","last_peered":"2024-01-15T18:07:20.094487+0000","last_clean":"2024-01-15T18:07:20.094487+0000","last_became_active":"2024-01-15T18:07:20.080752+0000","last_became_peered":"2024-01-15T18:07:20.080752+0000","last_unstale":"2024-01-15T18:07:20.094487+0000","last_undegraded":"2024-01-15T18:07:20.094487+0000","last_fullsized":"2024-01-15T18:07:20.094487+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:08:56.987120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.087492+0000","last_change":"2024-01-15T18:07:19.085696+0000","last_active":"2024-01-15T18:07:19.087492+0000","last_peered":"2024-01-15T18:07:19.087492+0000","last_clean":"2024-01-15T18:07:19.087492+0000","last_became_active":"2024-01-15T18:07:19.085467+0000","last_became_peered":"2024-01-15T18:07:19.085467+0000","last_unstale":"2024-01-15T18:07:19.087492+0000","last_undegraded":"2024-01-15T18:07:19.087492+0000","last_fullsized":"2024-01-15T18:07:19.087492+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:26:07.301619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.068029+0000","last_change":"2024-01-15T18:07:19.068029+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.068029+0000","last_undegraded":"2024-01-15T18:07:19.068029+0000","last_fullsized":"2024-01-15T18:07:19.068029+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.086899+0000","last_change":"2024-01-15T18:07:19.081361+0000","last_active":"2024-01-15T18:07:19.086899+0000","last_peered":"2024-01-15T18:07:19.086899+0000","last_clean":"2024-01-15T18:07:19.086899+0000","last_became_active":"2024-01-15T18:07:19.081076+0000","last_became_peered":"2024-01-15T18:07:19.081076+0000","last_unstale":"2024-01-15T18:07:19.086899+0000","last_undegraded":"2024-01-15T18:07:19.086899+0000","last_fullsized":"2024-01-15T18:07:19.086899+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:34:07.305579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":183,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.074017+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:07:19.074017+0000","last_peered":"2024-01-15T18:07:19.074017+0000","last_clean":"2024-01-15T18:07:19.074017+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:07:19.074017+0000","last_undegraded":"2024-01-15T18:07:19.074017+0000","last_fullsized":"2024-01-15T18:07:19.074017+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2340,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509474,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32248,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339592,"statfs":{"total":96636764160,"available":96603742208,"internally_reserved":0,"allocated":1089536,"data_stored":789175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.628,"15min":0.625},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.67200000000000004,"15min":0.67700000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.78200000000000003}]},{"osd":1,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.67800000000000005,"15min":0.67100000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.69399999999999995,"15min":0.69799999999999995},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.90400000000000003,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.56000000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64800000000000002,"15min":0.64700000000000002},"min":{"1min":0.46700000000000003,"5min":0.38,"15min":0.38},"max":{"1min":0.84199999999999997,"5min":1.012,"15min":1.012},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.71299999999999997,"15min":0.71199999999999997},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.0980000000000001,"5min":1.175,"15min":1.175},"last":0.68799999999999994}]},{"osd":3,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.66900000000000004,"15min":0.66500000000000004},"min":{"1min":0.502,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.88200000000000001,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0569999999999999,"5min":1.103,"15min":1.103},"last":0.74099999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.60799999999999998,"15min":0.61599999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.69199999999999995,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.65700000000000003,"15min":0.66200000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.622}]},{"osd":5,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.69499999999999995,"15min":0.69899999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.63,"5min":0.70999999999999996,"15min":0.72299999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.89500000000000002,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.70699999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.78800000000000003,"15min":0.80000000000000004},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.89500000000000002,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72499999999999998,"15min":0.73099999999999998},"min":{"1min":0.52500000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.76600000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509474,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31668,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340172,"statfs":{"total":96636764160,"available":96604336128,"internally_reserved":0,"allocated":495616,"data_stored":198807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.432,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.67200000000000004,"15min":0.68899999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84899999999999998,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.755}]},{"osd":1,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.70299999999999996,"15min":0.72399999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.82699999999999996,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.74399999999999999,"15min":0.754},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":0.59099999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.82099999999999995,"15min":0.84399999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.825,"5min":2.8700000000000001,"15min":2.8700000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.755,"15min":0.78300000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.82399999999999995,"5min":1.857,"15min":1.857},"last":0.78600000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.77400000000000002,"15min":0.80100000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.92100000000000004,"5min":2.012,"15min":2.012},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.79900000000000004,"15min":0.82499999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":2.532,"15min":2.532},"last":0.66200000000000003}]},{"osd":4,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68300000000000005,"15min":0.69299999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.726,"5min":1.726,"15min":1.726},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.69399999999999995,"15min":0.71799999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.753,"5min":1.145,"15min":1.145},"last":0.81799999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.64500000000000002,"15min":0.65800000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.81299999999999994,"5min":1.121,"15min":1.121},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.77400000000000002,"15min":0.78100000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"last":0.91400000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.80100000000000005,"15min":0.82499999999999996},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.96499999999999997,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.874},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.78000000000000003,"15min":0.80200000000000005},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.081,"5min":1.165,"15min":1.165},"last":0.80500000000000005}]}]},{"osd":4,"up_from":15,"seq":64424509474,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31688,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340152,"statfs":{"total":96636764160,"available":96604315648,"internally_reserved":0,"allocated":520192,"data_stored":207222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.628,"15min":0.63},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.63200000000000001,"15min":0.64100000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.60299999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64200000000000002,"15min":0.64900000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.77300000000000002,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.628,"5min":0.64000000000000001,"15min":0.64200000000000002},"min":{"1min":0.40899999999999997,"5min":0.377,"15min":0.377},"max":{"1min":0.88,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.66200000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.626,"15min":0.628},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.69099999999999995,"15min":0.69499999999999995},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.81999999999999995,"5min":0.879,"15min":0.879},"last":0.80600000000000005}]},{"osd":3,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68899999999999995,"15min":0.68999999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.96699999999999997,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.72499999999999998,"15min":0.72599999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.93200000000000005,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.79000000000000004}]},{"osd":5,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40200000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65100000000000002,"15min":0.65000000000000002},"min":{"1min":0.40699999999999997,"5min":0.377,"15min":0.377},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.71699999999999997}]},{"osd":6,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.68200000000000005,"15min":0.68000000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.85099999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.46899999999999997,"5min":0.436,"15min":0.436},"max":{"1min":1.0069999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.81799999999999995}]}]},{"osd":3,"up_from":15,"seq":64424509473,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":520192,"data_stored":206248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.28499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.72399999999999998,"5min":0.752,"15min":0.752},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.499,"15min":0.47999999999999998},"min":{"1min":0.34000000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":2.6509999999999998,"5min":2.6509999999999998,"15min":2.6509999999999998},"last":0.38500000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70999999999999996,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.498},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.53700000000000003,"15min":0.53600000000000003},"min":{"1min":0.42499999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83399999999999996,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.52500000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.55800000000000005,"15min":0.56000000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.58599999999999997,"15min":0.57199999999999995},"min":{"1min":0.42899999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.56399999999999995}]},{"osd":4,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.79400000000000004,"15min":0.80300000000000005},"min":{"1min":0.50800000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.119,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.876},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.77300000000000002,"15min":0.77900000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.97899999999999998,"5min":1.341,"15min":1.341},"last":0.65600000000000003}]},{"osd":5,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.59199999999999997,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.72399999999999998,"15min":0.72199999999999998},"min":{"1min":0.53300000000000003,"5min":0.498,"15min":0.498},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":0.73299999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.78300000000000003,"15min":0.78500000000000003},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.77600000000000002,"15min":0.76600000000000001},"min":{"1min":0.628,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":2.0880000000000001,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":0.80200000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.78800000000000003,"15min":0.78100000000000003},"min":{"1min":0.58999999999999997,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.81299999999999994,"15min":0.81999999999999995},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.1259999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.78600000000000003}]}]},{"osd":2,"up_from":15,"seq":64424509473,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31668,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340172,"statfs":{"total":96636764160,"available":96604336128,"internally_reserved":0,"allocated":495616,"data_stored":198807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.45500000000000002,"15min":0.46100000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.75,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.441,"5min":0.46100000000000002,"15min":0.46500000000000002},"min":{"1min":0.29899999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.64700000000000002,"5min":0.753,"15min":0.753},"last":0.46400000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54300000000000004,"15min":0.54500000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.67600000000000005,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.314,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.65100000000000002,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.436}]},{"osd":3,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.57799999999999996,"15min":0.57299999999999995},"min":{"1min":0.375,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.55600000000000005,"15min":0.55300000000000005},"min":{"1min":0.40500000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71499999999999997,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.42099999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.75700000000000001,"15min":0.76700000000000002},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.038,"5min":1.466,"15min":1.466},"last":0.626}]},{"osd":5,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.79400000000000004,"15min":0.79700000000000004},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.609,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.747,"5min":0.78300000000000003,"15min":0.78900000000000003},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.64600000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.76800000000000002,"15min":0.77800000000000002},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0489999999999999,"5min":1.516,"15min":1.516},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.78600000000000003,"15min":0.79600000000000004},"min":{"1min":0.52800000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0820000000000001,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.56100000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.79500000000000004,"15min":0.80100000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.1659999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.81799999999999995,"15min":0.82599999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.69299999999999995}]}]},{"osd":1,"up_from":15,"seq":64424509475,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31668,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340172,"statfs":{"total":96636764160,"available":96604336128,"internally_reserved":0,"allocated":495616,"data_stored":198807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.49099999999999999,"15min":0.504},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.627,"5min":1.847,"15min":1.847},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.443,"5min":0.47899999999999998,"15min":0.48499999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.63500000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.54300000000000004}]},{"osd":2,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.53400000000000003,"15min":0.54400000000000004},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.77500000000000002,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.624},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.70299999999999996,"15min":0.73299999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.90100000000000002,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.46700000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.63100000000000001,"15min":0.64500000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80700000000000005,"5min":1.871,"15min":1.871},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.58099999999999996,"15min":0.59099999999999997},"min":{"1min":0.35299999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.72599999999999998,"5min":1.917,"15min":1.917},"last":0.496}]},{"osd":4,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.70299999999999996,"15min":0.69599999999999995},"min":{"1min":0.54800000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.161,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.77100000000000002,"15min":0.77700000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.147,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.73799999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.76900000000000002,"15min":0.77100000000000002},"min":{"1min":0.47299999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.214,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.53100000000000003,"5min":0.505,"15min":0.505},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.97899999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.753,"15min":0.755},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.1359999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.81599999999999995,"15min":0.82199999999999995},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.95699999999999996}]},{"osd":7,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78700000000000003,"15min":0.78500000000000003},"min":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.79900000000000004,"15min":0.80300000000000005},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.095,"5min":1.347,"15min":1.347},"last":0.90500000000000003}]}]},{"osd":5,"up_from":14,"seq":60129542178,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":483328,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":0.871,"15min":0.77500000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":21.385000000000002,"5min":21.385000000000002,"15min":21.385000000000002},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.202,"5min":0.878,"15min":0.82399999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":8.2029999999999994,"5min":8.2029999999999994,"15min":8.2029999999999994},"last":0.73599999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66400000000000003,"15min":0.67200000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":1.4670000000000001,"5min":0.84099999999999997,"15min":0.73699999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":21.5,"5min":21.5,"15min":21.5},"last":0.80100000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":0.81599999999999995,"15min":0.745},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":8.2439999999999998,"5min":8.2439999999999998,"15min":8.2439999999999998},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.748,"15min":0.77000000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.85099999999999998,"5min":1.663,"15min":1.663},"last":0.71999999999999997}]},{"osd":3,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":0.95299999999999996,"15min":0.85699999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":21.457000000000001,"5min":21.457000000000001,"15min":21.457000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.1919999999999999,"5min":0.92100000000000004,"15min":0.876},"min":{"1min":0.50900000000000001,"5min":0.502,"15min":0.502},"max":{"1min":8.2750000000000004,"5min":8.2750000000000004,"15min":8.2750000000000004},"last":0.70499999999999996}]},{"osd":4,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":0.82599999999999996,"15min":0.76600000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":9.6679999999999993,"5min":9.6679999999999993,"15min":9.6679999999999993},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.71299999999999997,"15min":0.73499999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.76300000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.60099999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":0.877,"15min":0.78400000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":21.535,"5min":21.535,"15min":21.535},"last":0.54100000000000004},{"interface":"front","average":{"1min":1.1870000000000001,"5min":0.88300000000000001,"15min":0.83199999999999996},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":8.298,"5min":8.298,"15min":8.298},"last":0.66700000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":0.83999999999999997,"15min":0.81399999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":9.2240000000000002,"5min":9.2240000000000002,"15min":9.2240000000000002},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.5249999999999999,"5min":0.97099999999999997,"15min":0.879},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":21.574000000000002,"5min":21.574000000000002,"15min":21.574000000000002},"last":0.748}]}]},{"osd":0,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32240,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339600,"statfs":{"total":96636764160,"available":96603750400,"internally_reserved":0,"allocated":1077248,"data_stored":782880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.46300000000000002,"15min":0.46500000000000002},"min":{"1min":0.28999999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.746,"5min":1.462,"15min":1.462},"last":0.501},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55400000000000005,"15min":0.55200000000000005},"min":{"1min":0.34399999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.995,"5min":1.373,"15min":1.373},"last":0.39000000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.371,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.89600000000000002,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.441},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.48699999999999999,"15min":0.48099999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.252,"5min":1.609,"15min":1.609},"last":0.54500000000000004}]},{"osd":3,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.371,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.321,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60899999999999999,"15min":0.60999999999999999},"min":{"1min":0.45000000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.94599999999999995,"5min":1.325,"15min":1.325},"last":0.52700000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.76700000000000002,"15min":0.77700000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.96099999999999997,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.80900000000000005,"15min":0.82599999999999996},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.93300000000000005,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.71099999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.78500000000000003,"15min":0.79500000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0109999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.83099999999999996,"15min":0.84699999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":0.63300000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.79400000000000004,"15min":0.79800000000000004},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.107,"5min":1.764,"15min":1.764},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.83099999999999996,"15min":0.84699999999999998},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.97499999999999998,"5min":1.806,"15min":1.806},"last":0.72699999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.82199999999999995,"15min":0.82999999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.04,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.84899999999999998,"15min":0.85999999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.0429999999999999,"5min":1.722,"15min":1.722},"last":0.67100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:07:24.965 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3e is not active+clean 2024-01-15T18:07:24.965 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:07:19.067098+0000', 'last_change': '2024-01-15T18:07:19.067098+0000', 'last_active': '2024-01-15T18:07:19.050323+0000', 'last_peered': '2024-01-15T18:07:19.050323+0000', 'last_clean': '2024-01-15T18:07:19.050323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:07:19.067098+0000', 'last_undegraded': '2024-01-15T18:07:19.067098+0000', 'last_fullsized': '2024-01-15T18:07:19.067098+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_clean_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T18:07:24.965 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3c is not active+clean 2024-01-15T18:07:24.965 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:07:19.068614+0000', 'last_change': '2024-01-15T18:07:19.068614+0000', 'last_active': '2024-01-15T18:07:19.050323+0000', 'last_peered': '2024-01-15T18:07:19.050323+0000', 'last_clean': '2024-01-15T18:07:19.050323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:07:19.068614+0000', 'last_undegraded': '2024-01-15T18:07:19.068614+0000', 'last_fullsized': '2024-01-15T18:07:19.068614+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_clean_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:07:24.965 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3b is not active+clean 2024-01-15T18:07:24.965 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:07:19.065311+0000', 'last_change': '2024-01-15T18:07:19.065311+0000', 'last_active': '2024-01-15T18:07:19.050323+0000', 'last_peered': '2024-01-15T18:07:19.050323+0000', 'last_clean': '2024-01-15T18:07:19.050323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:07:19.065311+0000', 'last_undegraded': '2024-01-15T18:07:19.065311+0000', 'last_fullsized': '2024-01-15T18:07:19.065311+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_clean_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T18:07:24.965 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.38 is not active+clean 2024-01-15T18:07:24.966 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:07:19.074897+0000', 'last_change': '2024-01-15T18:07:19.074897+0000', 'last_active': '2024-01-15T18:07:19.050323+0000', 'last_peered': '2024-01-15T18:07:19.050323+0000', 'last_clean': '2024-01-15T18:07:19.050323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:07:19.074897+0000', 'last_undegraded': '2024-01-15T18:07:19.074897+0000', 'last_fullsized': '2024-01-15T18:07:19.074897+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_clean_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:07:24.966 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.37 is not active+clean 2024-01-15T18:07:24.966 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:07:19.077258+0000', 'last_change': '2024-01-15T18:07:19.077258+0000', 'last_active': '2024-01-15T18:07:19.050323+0000', 'last_peered': '2024-01-15T18:07:19.050323+0000', 'last_clean': '2024-01-15T18:07:19.050323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:07:19.077258+0000', 'last_undegraded': '2024-01-15T18:07:19.077258+0000', 'last_fullsized': '2024-01-15T18:07:19.077258+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_clean_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:07:24.966 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.36 is not active+clean 2024-01-15T18:07:24.966 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:07:19.070141+0000', 'last_change': '2024-01-15T18:07:19.070141+0000', 'last_active': '2024-01-15T18:07:19.050323+0000', 'last_peered': '2024-01-15T18:07:19.050323+0000', 'last_clean': '2024-01-15T18:07:19.050323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:07:19.070141+0000', 'last_undegraded': '2024-01-15T18:07:19.070141+0000', 'last_fullsized': '2024-01-15T18:07:19.070141+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_clean_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:07:24.966 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.35 is not active+clean 2024-01-15T18:07:24.966 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:07:19.074684+0000', 'last_change': '2024-01-15T18:07:19.074684+0000', 'last_active': '2024-01-15T18:07:19.050323+0000', 'last_peered': '2024-01-15T18:07:19.050323+0000', 'last_clean': '2024-01-15T18:07:19.050323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:07:19.074684+0000', 'last_undegraded': '2024-01-15T18:07:19.074684+0000', 'last_fullsized': '2024-01-15T18:07:19.074684+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_clean_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:07:24.966 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.34 is not active+clean 2024-01-15T18:07:24.966 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:07:19.072234+0000', 'last_change': '2024-01-15T18:07:19.072234+0000', 'last_active': '2024-01-15T18:07:19.050323+0000', 'last_peered': '2024-01-15T18:07:19.050323+0000', 'last_clean': '2024-01-15T18:07:19.050323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:07:19.072234+0000', 'last_undegraded': '2024-01-15T18:07:19.072234+0000', 'last_fullsized': '2024-01-15T18:07:19.072234+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_clean_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T18:07:24.966 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.32 is not active+clean 2024-01-15T18:07:24.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:07:19.068436+0000', 'last_change': '2024-01-15T18:07:19.068436+0000', 'last_active': '2024-01-15T18:07:19.050323+0000', 'last_peered': '2024-01-15T18:07:19.050323+0000', 'last_clean': '2024-01-15T18:07:19.050323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:07:19.068436+0000', 'last_undegraded': '2024-01-15T18:07:19.068436+0000', 'last_fullsized': '2024-01-15T18:07:19.068436+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_clean_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:07:24.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2a is not active+clean 2024-01-15T18:07:24.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:07:19.067198+0000', 'last_change': '2024-01-15T18:07:19.067198+0000', 'last_active': '2024-01-15T18:07:19.050323+0000', 'last_peered': '2024-01-15T18:07:19.050323+0000', 'last_clean': '2024-01-15T18:07:19.050323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:07:19.067198+0000', 'last_undegraded': '2024-01-15T18:07:19.067198+0000', 'last_fullsized': '2024-01-15T18:07:19.067198+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_clean_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:07:24.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.24 is not active+clean 2024-01-15T18:07:24.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:07:19.071723+0000', 'last_change': '2024-01-15T18:07:19.071723+0000', 'last_active': '2024-01-15T18:07:19.050323+0000', 'last_peered': '2024-01-15T18:07:19.050323+0000', 'last_clean': '2024-01-15T18:07:19.050323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:07:19.071723+0000', 'last_undegraded': '2024-01-15T18:07:19.071723+0000', 'last_fullsized': '2024-01-15T18:07:19.071723+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_clean_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T18:07:24.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1c is not active+clean 2024-01-15T18:07:24.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:07:19.068126+0000', 'last_change': '2024-01-15T18:07:19.068126+0000', 'last_active': '2024-01-15T18:07:19.050323+0000', 'last_peered': '2024-01-15T18:07:19.050323+0000', 'last_clean': '2024-01-15T18:07:19.050323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:07:19.068126+0000', 'last_undegraded': '2024-01-15T18:07:19.068126+0000', 'last_fullsized': '2024-01-15T18:07:19.068126+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_clean_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T18:07:24.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1b is not active+clean 2024-01-15T18:07:24.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:07:19.074401+0000', 'last_change': '2024-01-15T18:07:19.074401+0000', 'last_active': '2024-01-15T18:07:19.050323+0000', 'last_peered': '2024-01-15T18:07:19.050323+0000', 'last_clean': '2024-01-15T18:07:19.050323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:07:19.074401+0000', 'last_undegraded': '2024-01-15T18:07:19.074401+0000', 'last_fullsized': '2024-01-15T18:07:19.074401+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_clean_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:07:24.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.18 is not active+clean 2024-01-15T18:07:24.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:07:19.069432+0000', 'last_change': '2024-01-15T18:07:19.069432+0000', 'last_active': '2024-01-15T18:07:19.050323+0000', 'last_peered': '2024-01-15T18:07:19.050323+0000', 'last_clean': '2024-01-15T18:07:19.050323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:07:19.069432+0000', 'last_undegraded': '2024-01-15T18:07:19.069432+0000', 'last_fullsized': '2024-01-15T18:07:19.069432+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_clean_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:07:24.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.13 is not active+clean 2024-01-15T18:07:24.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:07:19.067641+0000', 'last_change': '2024-01-15T18:07:19.067641+0000', 'last_active': '2024-01-15T18:07:19.050323+0000', 'last_peered': '2024-01-15T18:07:19.050323+0000', 'last_clean': '2024-01-15T18:07:19.050323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:07:19.067641+0000', 'last_undegraded': '2024-01-15T18:07:19.067641+0000', 'last_fullsized': '2024-01-15T18:07:19.067641+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_clean_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:07:24.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.f is not active+clean 2024-01-15T18:07:24.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:07:19.068689+0000', 'last_change': '2024-01-15T18:07:19.068689+0000', 'last_active': '2024-01-15T18:07:19.050323+0000', 'last_peered': '2024-01-15T18:07:19.050323+0000', 'last_clean': '2024-01-15T18:07:19.050323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:07:19.068689+0000', 'last_undegraded': '2024-01-15T18:07:19.068689+0000', 'last_fullsized': '2024-01-15T18:07:19.068689+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_clean_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:07:24.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.9 is not active+clean 2024-01-15T18:07:24.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:07:19.067698+0000', 'last_change': '2024-01-15T18:07:19.067698+0000', 'last_active': '2024-01-15T18:07:19.050323+0000', 'last_peered': '2024-01-15T18:07:19.050323+0000', 'last_clean': '2024-01-15T18:07:19.050323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:07:19.067698+0000', 'last_undegraded': '2024-01-15T18:07:19.067698+0000', 'last_fullsized': '2024-01-15T18:07:19.067698+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_clean_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:07:24.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.4 is not active+clean 2024-01-15T18:07:24.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:07:19.067320+0000', 'last_change': '2024-01-15T18:07:19.067320+0000', 'last_active': '2024-01-15T18:07:19.050323+0000', 'last_peered': '2024-01-15T18:07:19.050323+0000', 'last_clean': '2024-01-15T18:07:19.050323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:07:19.067320+0000', 'last_undegraded': '2024-01-15T18:07:19.067320+0000', 'last_fullsized': '2024-01-15T18:07:19.067320+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_clean_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:07:24.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.0 is not active+clean 2024-01-15T18:07:24.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:07:19.064957+0000', 'last_change': '2024-01-15T18:07:19.064957+0000', 'last_active': '2024-01-15T18:07:19.050323+0000', 'last_peered': '2024-01-15T18:07:19.050323+0000', 'last_clean': '2024-01-15T18:07:19.050323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:07:19.064957+0000', 'last_undegraded': '2024-01-15T18:07:19.064957+0000', 'last_fullsized': '2024-01-15T18:07:19.064957+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_clean_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T18:07:24.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.6 is not active+clean 2024-01-15T18:07:24.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:07:19.068029+0000', 'last_change': '2024-01-15T18:07:19.068029+0000', 'last_active': '2024-01-15T18:07:19.050323+0000', 'last_peered': '2024-01-15T18:07:19.050323+0000', 'last_clean': '2024-01-15T18:07:19.050323+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:07:19.068029+0000', 'last_undegraded': '2024-01-15T18:07:19.068029+0000', 'last_fullsized': '2024-01-15T18:07:19.068029+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'last_clean_scrub_stamp': '2024-01-15T18:07:19.050323+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:07:24.969 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-15T18:07:25.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.140+0000 7fc1c8a00640 1 Processor -- start 2024-01-15T18:07:25.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.140+0000 7fc1c8a00640 1 -- start start 2024-01-15T18:07:25.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.140+0000 7fc1c8a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc1c0160b10 0x7fc1c00afe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:25.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.140+0000 7fc1c8a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fc1c00cd7a0 0x7fc1c00cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:25.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.140+0000 7fc1c8a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c00d05b0 con 0x7fc1c00ce160 2024-01-15T18:07:25.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.140+0000 7fc1c8a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c00ce5f0 con 0x7fc1c00cd7a0 2024-01-15T18:07:25.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.140+0000 7fc1c8a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c00ce7d0 con 0x7fc1c0160b10 2024-01-15T18:07:25.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.140+0000 7fc1c6a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc1c00ce160 0x7fc1c006b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45194/0 (socket says 172.21.15.27:45194) 2024-01-15T18:07:25.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.140+0000 7fc1c6a00640 1 -- 172.21.15.27:0/2273710053 learned_addr learned my addr 172.21.15.27:0/2273710053 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:25.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c6000640 1 --2- 172.21.15.27:0/2273710053 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc1c0160b10 0x7fc1c00afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:25.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c5600640 1 --2- 172.21.15.27:0/2273710053 >> v2:172.21.15.93:3300/0 conn(0x7fc1c00cd7a0 0x7fc1c00cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:25.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c4c00640 1 -- 172.21.15.27:0/2273710053 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 904561009 0 0) 0x7fc1c00d05b0 con 0x7fc1c00ce160 2024-01-15T18:07:25.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c4c00640 1 -- 172.21.15.27:0/2273710053 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1a0003530 con 0x7fc1c00ce160 2024-01-15T18:07:25.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c4c00640 1 -- 172.21.15.27:0/2273710053 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3603618425 0 0) 0x7fc1a0003530 con 0x7fc1c00ce160 2024-01-15T18:07:25.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c4c00640 1 -- 172.21.15.27:0/2273710053 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc1c0160b10 msgr2=0x7fc1c00afe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:25.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c4c00640 1 --2- 172.21.15.27:0/2273710053 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc1c0160b10 0x7fc1c00afe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:25.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c4c00640 1 -- 172.21.15.27:0/2273710053 >> v2:172.21.15.93:3300/0 conn(0x7fc1c00cd7a0 msgr2=0x7fc1c00cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:25.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c4c00640 1 --2- 172.21.15.27:0/2273710053 >> v2:172.21.15.93:3300/0 conn(0x7fc1c00cd7a0 0x7fc1c00cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:25.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c4c00640 1 -- 172.21.15.27:0/2273710053 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c00cfc80 con 0x7fc1c00ce160 2024-01-15T18:07:25.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c4c00640 1 -- 172.21.15.27:0/2273710053 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fc1b8001a00 con 0x7fc1c00ce160 2024-01-15T18:07:25.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c4c00640 1 -- 172.21.15.27:0/2273710053 >> v1:172.21.15.27:6789/0 conn(0x7fc1c00ce160 legacy=0x7fc1c006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:25.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c4c00640 1 --2- 172.21.15.27:0/2273710053 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc1c0160b10 0x7fc1a00039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:25.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c4c00640 1 --2- 172.21.15.27:0/2273710053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1a0003f70 0x7fc1a00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:25.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c4c00640 1 --2- 172.21.15.27:0/2273710053 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc1a0004d10 0x7fc1a0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:25.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c4c00640 1 -- 172.21.15.27:0/2273710053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c00ce5f0 con 0x7fc1a0003f70 2024-01-15T18:07:25.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c4c00640 1 -- 172.21.15.27:0/2273710053 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c00ce7d0 con 0x7fc1a0004d10 2024-01-15T18:07:25.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c4c00640 1 -- 172.21.15.27:0/2273710053 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a000ba40 con 0x7fc1c0160b10 2024-01-15T18:07:25.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c5600640 1 --2- 172.21.15.27:0/2273710053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1a0003f70 0x7fc1a00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:25.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c6000640 1 --2- 172.21.15.27:0/2273710053 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc1c0160b10 0x7fc1a00039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:25.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c6a00640 1 --2- 172.21.15.27:0/2273710053 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc1a0004d10 0x7fc1a0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:25.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c5600640 1 -- 172.21.15.27:0/2273710053 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc1c0160b10 msgr2=0x7fc1a00039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:25.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c5600640 1 --2- 172.21.15.27:0/2273710053 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc1c0160b10 0x7fc1a00039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:25.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c5600640 1 -- 172.21.15.27:0/2273710053 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc1a0004d10 msgr2=0x7fc1a0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:25.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c5600640 1 --2- 172.21.15.27:0/2273710053 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc1a0004d10 0x7fc1a0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:25.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c5600640 1 -- 172.21.15.27:0/2273710053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a0003530 con 0x7fc1a0003f70 2024-01-15T18:07:25.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c6000640 1 --2- 172.21.15.27:0/2273710053 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc1c0160b10 0x7fc1a00039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:07:25.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.144+0000 7fc1c6a00640 1 --2- 172.21.15.27:0/2273710053 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc1a0004d10 0x7fc1a0005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:07:25.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.148+0000 7fc1c5600640 1 --2- 172.21.15.27:0/2273710053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1a0003f70 0x7fc1a00047d0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8009ae0 tx=0x7fc1a801ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a7746616f80028c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:25.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.148+0000 7fc1c4c00640 1 -- 172.21.15.27:0/2273710053 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a80a5510 con 0x7fc1a0003f70 2024-01-15T18:07:25.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.148+0000 7fc1c4c00640 1 -- 172.21.15.27:0/2273710053 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a80a5670 con 0x7fc1a0003f70 2024-01-15T18:07:25.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.148+0000 7fc1c4c00640 1 -- 172.21.15.27:0/2273710053 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a80a5a20 con 0x7fc1a0003f70 2024-01-15T18:07:25.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.148+0000 7fc1c8a00640 1 -- 172.21.15.27:0/2273710053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1a0003f70 msgr2=0x7fc1a00047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:25.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.148+0000 7fc1c8a00640 1 --2- 172.21.15.27:0/2273710053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1a0003f70 0x7fc1a00047d0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fc1a8009ae0 tx=0x7fc1a801ca60 comp rx=0 tx=0).stop 2024-01-15T18:07:25.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.148+0000 7fc1c6a00640 1 -- 172.21.15.27:0/2273710053 reap_dead start 2024-01-15T18:07:25.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.148+0000 7fc1c8a00640 1 -- 172.21.15.27:0/2273710053 shutdown_connections 2024-01-15T18:07:25.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.148+0000 7fc1c8a00640 1 -- 172.21.15.27:0/2273710053 >> 172.21.15.27:0/2273710053 conn(0x7fc1c00d0f20 msgr2=0x7fc1c00d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:25.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.148+0000 7fc1c8a00640 1 -- 172.21.15.27:0/2273710053 shutdown_connections 2024-01-15T18:07:25.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.148+0000 7fc1c8a00640 1 -- 172.21.15.27:0/2273710053 wait complete. 2024-01-15T18:07:25.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.148+0000 7fc1c8a00640 1 Processor -- start 2024-01-15T18:07:25.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.148+0000 7fc1c8a00640 1 -- start start 2024-01-15T18:07:25.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.152+0000 7fc1c8a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc1a0003f70 0x7fc1c00c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:25.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.152+0000 7fc1c8a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc1c00c8b20 0x7fc1c01549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:25.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.152+0000 7fc1c8a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1c01665b0 0x7fc1c0166a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:25.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.152+0000 7fc1c8a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c015e7c0 con 0x7fc1c01665b0 2024-01-15T18:07:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.152+0000 7fc1c8a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c0157aa0 con 0x7fc1c00c8b20 2024-01-15T18:07:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.152+0000 7fc1c8a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c0166f70 con 0x7fc1a0003f70 2024-01-15T18:07:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.152+0000 7fc1c6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1c01665b0 0x7fc1c0166a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.152+0000 7fc1c6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1c01665b0 0x7fc1c0166a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50752/0 (socket says 172.21.15.27:50752) 2024-01-15T18:07:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.152+0000 7fc1c6a00640 1 -- 172.21.15.27:0/753657163 learned_addr learned my addr 172.21.15.27:0/753657163 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.152+0000 7fc1c6000640 1 --2- 172.21.15.27:0/753657163 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc1a0003f70 0x7fc1c00c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.152+0000 7fc1c5600640 1 --2- 172.21.15.27:0/753657163 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc1c00c8b20 0x7fc1c01549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.152+0000 7fc1c6a00640 1 -- 172.21.15.27:0/753657163 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc1a0003f70 msgr2=0x7fc1c00c7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.152+0000 7fc1c6a00640 1 --2- 172.21.15.27:0/753657163 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc1a0003f70 0x7fc1c00c7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.152+0000 7fc1c6a00640 1 -- 172.21.15.27:0/753657163 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc1c00c8b20 msgr2=0x7fc1c01549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.152+0000 7fc1c6a00640 1 --2- 172.21.15.27:0/753657163 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc1c00c8b20 0x7fc1c01549c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.152+0000 7fc1c6a00640 1 -- 172.21.15.27:0/753657163 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c00c8fa0 con 0x7fc1c01665b0 2024-01-15T18:07:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.152+0000 7fc1c6a00640 1 --2- 172.21.15.27:0/753657163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1c01665b0 0x7fc1c0166a30 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc1c00671e0 tx=0x7fc1b800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:25.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.152+0000 7fc1be000640 1 -- 172.21.15.27:0/753657163 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8007ae0 con 0x7fc1c01665b0 2024-01-15T18:07:25.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.152+0000 7fc1be000640 1 -- 172.21.15.27:0/753657163 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b800ed40 con 0x7fc1c01665b0 2024-01-15T18:07:25.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.152+0000 7fc1c8a00640 1 -- 172.21.15.27:0/753657163 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c015e140 con 0x7fc1c01665b0 2024-01-15T18:07:25.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.152+0000 7fc1be000640 1 -- 172.21.15.27:0/753657163 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b80146c0 con 0x7fc1c01665b0 2024-01-15T18:07:25.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.152+0000 7fc1c8a00640 1 -- 172.21.15.27:0/753657163 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c00c9500 con 0x7fc1c01665b0 2024-01-15T18:07:25.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.152+0000 7fc1c8a00640 1 -- 172.21.15.27:0/753657163 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c015e900 con 0x7fc1c01665b0 2024-01-15T18:07:25.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.156+0000 7fc1be000640 1 -- 172.21.15.27:0/753657163 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7fc1b8014820 con 0x7fc1c01665b0 2024-01-15T18:07:25.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.156+0000 7fc1be000640 1 --2- 172.21.15.27:0/753657163 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc198066b60 0x7fc198069020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:25.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.156+0000 7fc1be000640 1 -- 172.21.15.27:0/753657163 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc1b8084500 con 0x7fc1c01665b0 2024-01-15T18:07:25.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.156+0000 7fc1c6000640 1 --2- 172.21.15.27:0/753657163 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc198066b60 0x7fc198069020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:25.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.156+0000 7fc1c6000640 1 --2- 172.21.15.27:0/753657163 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc198066b60 0x7fc198069020 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0009a80 tx=0x7fc1b0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:25.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.160+0000 7fc1be000640 1 -- 172.21.15.27:0/753657163 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc1c015e900 con 0x7fc1c01665b0 2024-01-15T18:07:25.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.376+0000 7fc1c8a00640 1 -- 172.21.15.27:0/753657163 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fc1c015e900 con 0x7fc1c01665b0 2024-01-15T18:07:25.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.376+0000 7fc1be000640 1 -- 172.21.15.27:0/753657163 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1098 (secure 0 0 0) 0x7fc1c015e900 con 0x7fc1c01665b0 2024-01-15T18:07:25.380 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:07:25.380 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":172,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":45,"num_osds":8,"num_up_osds":8,"osd_up_since":1705341884,"num_in_osds":8,"osd_in_since":1705341875,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":77},{"state_name":"creating+peering","count":20}],"num_pgs":97,"num_pools":3,"num_objects":17,"data_bytes":592708,"bytes_used":260648960,"bytes_avail":772833464320,"bytes_total":773094113280,"inactive_pgs_ratio":0.20618556439876556,"write_bytes_sec":2218,"read_op_per_sec":0,"write_op_per_sec":4},"fsmap":{"epoch":27,"id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"a","status":"up:active","gid":5808}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-01-15T18:07:04.167195+0000","services":{}},"progress_events":{}} 2024-01-15T18:07:25.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.376+0000 7fc1c8a00640 1 -- 172.21.15.27:0/753657163 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc198066b60 msgr2=0x7fc198069020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:25.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.376+0000 7fc1c8a00640 1 --2- 172.21.15.27:0/753657163 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc198066b60 0x7fc198069020 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0009a80 tx=0x7fc1b0008040 comp rx=0 tx=0).stop 2024-01-15T18:07:25.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.376+0000 7fc1c8a00640 1 -- 172.21.15.27:0/753657163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1c01665b0 msgr2=0x7fc1c0166a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:25.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.376+0000 7fc1c8a00640 1 --2- 172.21.15.27:0/753657163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1c01665b0 0x7fc1c0166a30 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc1c00671e0 tx=0x7fc1b800e530 comp rx=0 tx=0).stop 2024-01-15T18:07:25.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.376+0000 7fc1c8a00640 1 -- 172.21.15.27:0/753657163 shutdown_connections 2024-01-15T18:07:25.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.376+0000 7fc1c8a00640 1 --2- 172.21.15.27:0/753657163 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc198066b60 0x7fc198069020 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:25.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.376+0000 7fc1c8a00640 1 --2- 172.21.15.27:0/753657163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1c01665b0 0x7fc1c0166a30 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:25.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.376+0000 7fc1c8a00640 1 --2- 172.21.15.27:0/753657163 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc1c00c8b20 0x7fc1c01549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:25.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.376+0000 7fc1c8a00640 1 --2- 172.21.15.27:0/753657163 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc1a0003f70 0x7fc1c00c7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:25.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.376+0000 7fc1c8a00640 1 -- 172.21.15.27:0/753657163 >> 172.21.15.27:0/753657163 conn(0x7fc1c00d0f20 msgr2=0x7fc1c00733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:25.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.376+0000 7fc1c8a00640 1 -- 172.21.15.27:0/753657163 shutdown_connections 2024-01-15T18:07:25.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.376+0000 7fc1c8a00640 1 -- 172.21.15.27:0/753657163 wait complete. 2024-01-15T18:07:25.402 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-01-15T18:07:25.402 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:07:25.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.540+0000 7ff0c8800640 1 Processor -- start 2024-01-15T18:07:25.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.540+0000 7ff0c8800640 1 -- start start 2024-01-15T18:07:25.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.540+0000 7ff0c8800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff0c00b2390 0x7ff0c00cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:25.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.540+0000 7ff0c8800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0c006bb80 0x7ff0c00cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:25.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.540+0000 7ff0c8800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c00b2770 con 0x7ff0c00cfcc0 2024-01-15T18:07:25.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.540+0000 7ff0c8800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0c006c060 con 0x7ff0c00b2390 2024-01-15T18:07:25.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.540+0000 7ff0c8800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c006c340 con 0x7ff0c006bb80 2024-01-15T18:07:25.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.540+0000 7ff0c5e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff0c00b2390 0x7ff0c00cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:25.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.540+0000 7ff0c6800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff0c00cfcc0 0x7ff0c0161120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45210/0 (socket says 172.21.15.27:45210) 2024-01-15T18:07:25.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.540+0000 7ff0c6800640 1 -- 172.21.15.27:0/1678410653 learned_addr learned my addr 172.21.15.27:0/1678410653 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:25.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.540+0000 7ff0c5400640 1 --2- 172.21.15.27:0/1678410653 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0c006bb80 0x7ff0c00cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:25.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.540+0000 7ff0c5e00640 1 -- 172.21.15.27:0/1678410653 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0c006bb80 msgr2=0x7ff0c00cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:25.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.540+0000 7ff0c5e00640 1 --2- 172.21.15.27:0/1678410653 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0c006bb80 0x7ff0c00cd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:25.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.540+0000 7ff0c5e00640 1 -- 172.21.15.27:0/1678410653 >> v1:172.21.15.27:6789/0 conn(0x7ff0c00cfcc0 legacy=0x7ff0c0161120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:25.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.540+0000 7ff0c5e00640 1 -- 172.21.15.27:0/1678410653 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c006c620 con 0x7ff0c00b2390 2024-01-15T18:07:25.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.540+0000 7ff0c4a00640 1 -- 172.21.15.27:0/1678410653 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1775376407 0 0) 0x7ff0c00b2770 con 0x7ff0c00cfcc0 2024-01-15T18:07:25.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.544+0000 7ff0c5e00640 1 --2- 172.21.15.27:0/1678410653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0c00b2390 0x7ff0c00cf780 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff0b00097c0 tx=0x7ff0b001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7b8f94a030c5b28 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:25.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.544+0000 7ff0c4a00640 1 -- 172.21.15.27:0/1678410653 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b00ab650 con 0x7ff0c00b2390 2024-01-15T18:07:25.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.544+0000 7ff0c4a00640 1 -- 172.21.15.27:0/1678410653 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b00ab7b0 con 0x7ff0c00b2390 2024-01-15T18:07:25.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.544+0000 7ff0c4a00640 1 -- 172.21.15.27:0/1678410653 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b00abb40 con 0x7ff0c00b2390 2024-01-15T18:07:25.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.544+0000 7ff0c8800640 1 -- 172.21.15.27:0/1678410653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0c00b2390 msgr2=0x7ff0c00cf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:25.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.544+0000 7ff0c8800640 1 --2- 172.21.15.27:0/1678410653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0c00b2390 0x7ff0c00cf780 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff0b00097c0 tx=0x7ff0b001c450 comp rx=0 tx=0).stop 2024-01-15T18:07:25.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.544+0000 7ff0c8800640 1 -- 172.21.15.27:0/1678410653 shutdown_connections 2024-01-15T18:07:25.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.544+0000 7ff0c8800640 1 --2- 172.21.15.27:0/1678410653 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0c006bb80 0x7ff0c00cd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:25.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.544+0000 7ff0c8800640 1 --2- 172.21.15.27:0/1678410653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0c00b2390 0x7ff0c00cf780 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:25.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.544+0000 7ff0c8800640 1 -- 172.21.15.27:0/1678410653 >> 172.21.15.27:0/1678410653 conn(0x7ff0c00b2a10 msgr2=0x7ff0c00b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:25.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.544+0000 7ff0c8800640 1 -- 172.21.15.27:0/1678410653 shutdown_connections 2024-01-15T18:07:25.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.544+0000 7ff0c8800640 1 -- 172.21.15.27:0/1678410653 wait complete. 2024-01-15T18:07:25.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.548+0000 7ff0c8800640 1 Processor -- start 2024-01-15T18:07:25.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.548+0000 7ff0c8800640 1 -- start start 2024-01-15T18:07:25.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.548+0000 7ff0c8800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0c006bb80 0x7ff0c0158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:25.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.548+0000 7ff0c8800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0c00b2390 0x7ff0c00ca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:25.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.548+0000 7ff0c8800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0c00cfcc0 0x7ff0c00c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:25.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.548+0000 7ff0c8800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c0159560 con 0x7ff0c006bb80 2024-01-15T18:07:25.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.548+0000 7ff0c8800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c0158fe0 con 0x7ff0c00cfcc0 2024-01-15T18:07:25.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.548+0000 7ff0c8800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c00caef0 con 0x7ff0c00b2390 2024-01-15T18:07:25.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.548+0000 7ff0c5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0c006bb80 0x7ff0c0158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:25.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.548+0000 7ff0c5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0c006bb80 0x7ff0c0158aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50782/0 (socket says 172.21.15.27:50782) 2024-01-15T18:07:25.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.548+0000 7ff0c5e00640 1 -- 172.21.15.27:0/3655036882 learned_addr learned my addr 172.21.15.27:0/3655036882 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:25.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.548+0000 7ff0c5400640 1 --2- 172.21.15.27:0/3655036882 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0c00b2390 0x7ff0c00ca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:25.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.548+0000 7ff0c6800640 1 --2- 172.21.15.27:0/3655036882 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0c00cfcc0 0x7ff0c00c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:25.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.548+0000 7ff0c5e00640 1 -- 172.21.15.27:0/3655036882 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0c00b2390 msgr2=0x7ff0c00ca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:25.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.548+0000 7ff0c5e00640 1 --2- 172.21.15.27:0/3655036882 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0c00b2390 0x7ff0c00ca9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:25.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.548+0000 7ff0c5e00640 1 -- 172.21.15.27:0/3655036882 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0c00cfcc0 msgr2=0x7ff0c00c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:25.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.548+0000 7ff0c5e00640 1 --2- 172.21.15.27:0/3655036882 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0c00cfcc0 0x7ff0c00c5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:25.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.548+0000 7ff0c5e00640 1 -- 172.21.15.27:0/3655036882 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b0009420 con 0x7ff0c006bb80 2024-01-15T18:07:25.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.548+0000 7ff0c5e00640 1 --2- 172.21.15.27:0/3655036882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0c006bb80 0x7ff0c0158aa0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff0b00ad040 tx=0x7ff0b001ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:25.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.548+0000 7ff0be000640 1 -- 172.21.15.27:0/3655036882 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0004220 con 0x7ff0c006bb80 2024-01-15T18:07:25.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.548+0000 7ff0be000640 1 -- 172.21.15.27:0/3655036882 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b00b9070 con 0x7ff0c006bb80 2024-01-15T18:07:25.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.548+0000 7ff0be000640 1 -- 172.21.15.27:0/3655036882 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0003620 con 0x7ff0c006bb80 2024-01-15T18:07:25.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.548+0000 7ff0c8800640 1 -- 172.21.15.27:0/3655036882 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c00c9910 con 0x7ff0c006bb80 2024-01-15T18:07:25.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.548+0000 7ff0c8800640 1 -- 172.21.15.27:0/3655036882 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c00c9ed0 con 0x7ff0c006bb80 2024-01-15T18:07:25.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.552+0000 7ff0c8800640 1 -- 172.21.15.27:0/3655036882 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c00d07c0 con 0x7ff0c006bb80 2024-01-15T18:07:25.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.552+0000 7ff0be000640 1 -- 172.21.15.27:0/3655036882 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7ff0b0003780 con 0x7ff0c006bb80 2024-01-15T18:07:25.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.552+0000 7ff0be000640 1 --2- 172.21.15.27:0/3655036882 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff098066b30 0x7ff098068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:25.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.552+0000 7ff0be000640 1 -- 172.21.15.27:0/3655036882 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5737+0+0 (secure 0 0 0) 0x7ff0b00bb070 con 0x7ff0c006bb80 2024-01-15T18:07:25.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.552+0000 7ff0c5400640 1 --2- 172.21.15.27:0/3655036882 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff098066b30 0x7ff098068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:25.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.552+0000 7ff0c5400640 1 --2- 172.21.15.27:0/3655036882 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff098066b30 0x7ff098068ff0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff0c009ef60 tx=0x7ff0a8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:25.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.556+0000 7ff0be000640 1 -- 172.21.15.27:0/3655036882 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff0b00ec190 con 0x7ff0c006bb80 2024-01-15T18:07:25.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.708+0000 7ff0c8800640 1 -- 172.21.15.27:0/3655036882 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff0c00ce1d0 con 0x7ff098066b30 2024-01-15T18:07:25.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.716+0000 7ff0be000640 1 -- 172.21.15.27:0/3655036882 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287094 (secure 0 0 0) 0x7ff0a80118e0 con 0x7ff098066b30 2024-01-15T18:07:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:07:25.726 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:07:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.720+0000 7ff0c8800640 1 -- 172.21.15.27:0/3655036882 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff098066b30 msgr2=0x7ff098068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.720+0000 7ff0c8800640 1 --2- 172.21.15.27:0/3655036882 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff098066b30 0x7ff098068ff0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff0c009ef60 tx=0x7ff0a8008040 comp rx=0 tx=0).stop 2024-01-15T18:07:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.720+0000 7ff0c8800640 1 -- 172.21.15.27:0/3655036882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0c006bb80 msgr2=0x7ff0c0158aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.720+0000 7ff0c8800640 1 --2- 172.21.15.27:0/3655036882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0c006bb80 0x7ff0c0158aa0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff0b00ad040 tx=0x7ff0b001ac90 comp rx=0 tx=0).stop 2024-01-15T18:07:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.720+0000 7ff0c8800640 1 -- 172.21.15.27:0/3655036882 shutdown_connections 2024-01-15T18:07:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.720+0000 7ff0c8800640 1 --2- 172.21.15.27:0/3655036882 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff098066b30 0x7ff098068ff0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.720+0000 7ff0c8800640 1 --2- 172.21.15.27:0/3655036882 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0c00cfcc0 0x7ff0c00c5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.720+0000 7ff0c8800640 1 --2- 172.21.15.27:0/3655036882 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0c00b2390 0x7ff0c00ca9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.720+0000 7ff0c8800640 1 --2- 172.21.15.27:0/3655036882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0c006bb80 0x7ff0c0158aa0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.720+0000 7ff0c8800640 1 -- 172.21.15.27:0/3655036882 >> 172.21.15.27:0/3655036882 conn(0x7ff0c00b2a10 msgr2=0x7ff0c00b3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.720+0000 7ff0c8800640 1 -- 172.21.15.27:0/3655036882 shutdown_connections 2024-01-15T18:07:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:25.724+0000 7ff0c8800640 1 -- 172.21.15.27:0/3655036882 wait complete. 2024-01-15T18:07:25.747 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-01-15T18:07:24.171279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592708,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":187,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":47,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254540,"kb_used_data":5056,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720180,"statfs":{"total":773094113280,"available":772833464320,"internally_reserved":0,"allocated":5177344,"data_stored":2774458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2340,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001241"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.094792+0000","last_change":"2024-01-15T18:07:20.088581+0000","last_active":"2024-01-15T18:07:20.094792+0000","last_peered":"2024-01-15T18:07:20.094792+0000","last_clean":"2024-01-15T18:07:20.094792+0000","last_became_active":"2024-01-15T18:07:20.088247+0000","last_became_peered":"2024-01-15T18:07:20.088247+0000","last_unstale":"2024-01-15T18:07:20.094792+0000","last_undegraded":"2024-01-15T18:07:20.094792+0000","last_fullsized":"2024-01-15T18:07:20.094792+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:04:14.022333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.067098+0000","last_change":"2024-01-15T18:07:19.067098+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.067098+0000","last_undegraded":"2024-01-15T18:07:19.067098+0000","last_fullsized":"2024-01-15T18:07:19.067098+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.099935+0000","last_change":"2024-01-15T18:07:20.094383+0000","last_active":"2024-01-15T18:07:20.099935+0000","last_peered":"2024-01-15T18:07:20.099935+0000","last_clean":"2024-01-15T18:07:20.099935+0000","last_became_active":"2024-01-15T18:07:20.094114+0000","last_became_peered":"2024-01-15T18:07:20.094114+0000","last_unstale":"2024-01-15T18:07:20.099935+0000","last_undegraded":"2024-01-15T18:07:20.099935+0000","last_fullsized":"2024-01-15T18:07:20.099935+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:07:17.293929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.068614+0000","last_change":"2024-01-15T18:07:19.068614+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.068614+0000","last_undegraded":"2024-01-15T18:07:19.068614+0000","last_fullsized":"2024-01-15T18:07:19.068614+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.065311+0000","last_change":"2024-01-15T18:07:19.065311+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.065311+0000","last_undegraded":"2024-01-15T18:07:19.065311+0000","last_fullsized":"2024-01-15T18:07:19.065311+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100427+0000","last_change":"2024-01-15T18:07:20.090463+0000","last_active":"2024-01-15T18:07:20.100427+0000","last_peered":"2024-01-15T18:07:20.100427+0000","last_clean":"2024-01-15T18:07:20.100427+0000","last_became_active":"2024-01-15T18:07:20.090196+0000","last_became_peered":"2024-01-15T18:07:20.090196+0000","last_unstale":"2024-01-15T18:07:20.100427+0000","last_undegraded":"2024-01-15T18:07:20.100427+0000","last_fullsized":"2024-01-15T18:07:20.100427+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:03:21.190309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.106586+0000","last_change":"2024-01-15T18:07:20.102595+0000","last_active":"2024-01-15T18:07:20.106586+0000","last_peered":"2024-01-15T18:07:20.106586+0000","last_clean":"2024-01-15T18:07:20.106586+0000","last_became_active":"2024-01-15T18:07:20.102222+0000","last_became_peered":"2024-01-15T18:07:20.102222+0000","last_unstale":"2024-01-15T18:07:20.106586+0000","last_undegraded":"2024-01-15T18:07:20.106586+0000","last_fullsized":"2024-01-15T18:07:20.106586+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:51:04.157158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.074897+0000","last_change":"2024-01-15T18:07:19.074897+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.074897+0000","last_undegraded":"2024-01-15T18:07:19.074897+0000","last_fullsized":"2024-01-15T18:07:19.074897+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.077258+0000","last_change":"2024-01-15T18:07:19.077258+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.077258+0000","last_undegraded":"2024-01-15T18:07:19.077258+0000","last_fullsized":"2024-01-15T18:07:19.077258+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.070141+0000","last_change":"2024-01-15T18:07:19.070141+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.070141+0000","last_undegraded":"2024-01-15T18:07:19.070141+0000","last_fullsized":"2024-01-15T18:07:19.070141+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.074684+0000","last_change":"2024-01-15T18:07:19.074684+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.074684+0000","last_undegraded":"2024-01-15T18:07:19.074684+0000","last_fullsized":"2024-01-15T18:07:19.074684+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.072234+0000","last_change":"2024-01-15T18:07:19.072234+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.072234+0000","last_undegraded":"2024-01-15T18:07:19.072234+0000","last_fullsized":"2024-01-15T18:07:19.072234+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.106908+0000","last_change":"2024-01-15T18:07:20.093608+0000","last_active":"2024-01-15T18:07:20.106908+0000","last_peered":"2024-01-15T18:07:20.106908+0000","last_clean":"2024-01-15T18:07:20.106908+0000","last_became_active":"2024-01-15T18:07:20.091735+0000","last_became_peered":"2024-01-15T18:07:20.091735+0000","last_unstale":"2024-01-15T18:07:20.106908+0000","last_undegraded":"2024-01-15T18:07:20.106908+0000","last_fullsized":"2024-01-15T18:07:20.106908+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:34.221280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.068436+0000","last_change":"2024-01-15T18:07:19.068436+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.068436+0000","last_undegraded":"2024-01-15T18:07:19.068436+0000","last_fullsized":"2024-01-15T18:07:19.068436+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.105807+0000","last_change":"2024-01-15T18:07:20.096359+0000","last_active":"2024-01-15T18:07:20.105807+0000","last_peered":"2024-01-15T18:07:20.105807+0000","last_clean":"2024-01-15T18:07:20.105807+0000","last_became_active":"2024-01-15T18:07:20.096018+0000","last_became_peered":"2024-01-15T18:07:20.096018+0000","last_unstale":"2024-01-15T18:07:20.105807+0000","last_undegraded":"2024-01-15T18:07:20.105807+0000","last_fullsized":"2024-01-15T18:07:20.105807+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:35:28.043293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100722+0000","last_change":"2024-01-15T18:07:20.097218+0000","last_active":"2024-01-15T18:07:20.100722+0000","last_peered":"2024-01-15T18:07:20.100722+0000","last_clean":"2024-01-15T18:07:20.100722+0000","last_became_active":"2024-01-15T18:07:20.096805+0000","last_became_peered":"2024-01-15T18:07:20.096805+0000","last_unstale":"2024-01-15T18:07:20.100722+0000","last_undegraded":"2024-01-15T18:07:20.100722+0000","last_fullsized":"2024-01-15T18:07:20.100722+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:34.221280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097051+0000","last_change":"2024-01-15T18:07:20.084911+0000","last_active":"2024-01-15T18:07:20.097051+0000","last_peered":"2024-01-15T18:07:20.097051+0000","last_clean":"2024-01-15T18:07:20.097051+0000","last_became_active":"2024-01-15T18:07:20.084594+0000","last_became_peered":"2024-01-15T18:07:20.084594+0000","last_unstale":"2024-01-15T18:07:20.097051+0000","last_undegraded":"2024-01-15T18:07:20.097051+0000","last_fullsized":"2024-01-15T18:07:20.097051+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:38:53.126797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100540+0000","last_change":"2024-01-15T18:07:20.097350+0000","last_active":"2024-01-15T18:07:20.100540+0000","last_peered":"2024-01-15T18:07:20.100540+0000","last_clean":"2024-01-15T18:07:20.100540+0000","last_became_active":"2024-01-15T18:07:20.095556+0000","last_became_peered":"2024-01-15T18:07:20.095556+0000","last_unstale":"2024-01-15T18:07:20.100540+0000","last_undegraded":"2024-01-15T18:07:20.100540+0000","last_fullsized":"2024-01-15T18:07:20.100540+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:01:01.613225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.103149+0000","last_change":"2024-01-15T18:07:20.098939+0000","last_active":"2024-01-15T18:07:20.103149+0000","last_peered":"2024-01-15T18:07:20.103149+0000","last_clean":"2024-01-15T18:07:20.103149+0000","last_became_active":"2024-01-15T18:07:20.098598+0000","last_became_peered":"2024-01-15T18:07:20.098598+0000","last_unstale":"2024-01-15T18:07:20.103149+0000","last_undegraded":"2024-01-15T18:07:20.103149+0000","last_fullsized":"2024-01-15T18:07:20.103149+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:26.049225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.107947+0000","last_change":"2024-01-15T18:07:20.095741+0000","last_active":"2024-01-15T18:07:20.107947+0000","last_peered":"2024-01-15T18:07:20.107947+0000","last_clean":"2024-01-15T18:07:20.107947+0000","last_became_active":"2024-01-15T18:07:20.095417+0000","last_became_peered":"2024-01-15T18:07:20.095417+0000","last_unstale":"2024-01-15T18:07:20.107947+0000","last_undegraded":"2024-01-15T18:07:20.107947+0000","last_fullsized":"2024-01-15T18:07:20.107947+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:32:43.186714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.105539+0000","last_change":"2024-01-15T18:07:20.094321+0000","last_active":"2024-01-15T18:07:20.105539+0000","last_peered":"2024-01-15T18:07:20.105539+0000","last_clean":"2024-01-15T18:07:20.105539+0000","last_became_active":"2024-01-15T18:07:20.094011+0000","last_became_peered":"2024-01-15T18:07:20.094011+0000","last_unstale":"2024-01-15T18:07:20.105539+0000","last_undegraded":"2024-01-15T18:07:20.105539+0000","last_fullsized":"2024-01-15T18:07:20.105539+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:48:24.482600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.067198+0000","last_change":"2024-01-15T18:07:19.067198+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.067198+0000","last_undegraded":"2024-01-15T18:07:19.067198+0000","last_fullsized":"2024-01-15T18:07:19.067198+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098234+0000","last_change":"2024-01-15T18:07:20.087728+0000","last_active":"2024-01-15T18:07:20.098234+0000","last_peered":"2024-01-15T18:07:20.098234+0000","last_clean":"2024-01-15T18:07:20.098234+0000","last_became_active":"2024-01-15T18:07:20.087118+0000","last_became_peered":"2024-01-15T18:07:20.087118+0000","last_unstale":"2024-01-15T18:07:20.098234+0000","last_undegraded":"2024-01-15T18:07:20.098234+0000","last_fullsized":"2024-01-15T18:07:20.098234+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:32:43.186714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100694+0000","last_change":"2024-01-15T18:07:20.092361+0000","last_active":"2024-01-15T18:07:20.100694+0000","last_peered":"2024-01-15T18:07:20.100694+0000","last_clean":"2024-01-15T18:07:20.100694+0000","last_became_active":"2024-01-15T18:07:20.091927+0000","last_became_peered":"2024-01-15T18:07:20.091927+0000","last_unstale":"2024-01-15T18:07:20.100694+0000","last_undegraded":"2024-01-15T18:07:20.100694+0000","last_fullsized":"2024-01-15T18:07:20.100694+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:55:19.615935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.106512+0000","last_change":"2024-01-15T18:07:20.102013+0000","last_active":"2024-01-15T18:07:20.106512+0000","last_peered":"2024-01-15T18:07:20.106512+0000","last_clean":"2024-01-15T18:07:20.106512+0000","last_became_active":"2024-01-15T18:07:20.101697+0000","last_became_peered":"2024-01-15T18:07:20.101697+0000","last_unstale":"2024-01-15T18:07:20.106512+0000","last_undegraded":"2024-01-15T18:07:20.106512+0000","last_fullsized":"2024-01-15T18:07:20.106512+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:32:14.924018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.103301+0000","last_change":"2024-01-15T18:07:20.086187+0000","last_active":"2024-01-15T18:07:20.103301+0000","last_peered":"2024-01-15T18:07:20.103301+0000","last_clean":"2024-01-15T18:07:20.103301+0000","last_became_active":"2024-01-15T18:07:20.085891+0000","last_became_peered":"2024-01-15T18:07:20.085891+0000","last_unstale":"2024-01-15T18:07:20.103301+0000","last_undegraded":"2024-01-15T18:07:20.103301+0000","last_fullsized":"2024-01-15T18:07:20.103301+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:54:09.185168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097020+0000","last_change":"2024-01-15T18:07:20.086602+0000","last_active":"2024-01-15T18:07:20.097020+0000","last_peered":"2024-01-15T18:07:20.097020+0000","last_clean":"2024-01-15T18:07:20.097020+0000","last_became_active":"2024-01-15T18:07:20.086238+0000","last_became_peered":"2024-01-15T18:07:20.086238+0000","last_unstale":"2024-01-15T18:07:20.097020+0000","last_undegraded":"2024-01-15T18:07:20.097020+0000","last_fullsized":"2024-01-15T18:07:20.097020+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:20:04.640555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.071723+0000","last_change":"2024-01-15T18:07:19.071723+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.071723+0000","last_undegraded":"2024-01-15T18:07:19.071723+0000","last_fullsized":"2024-01-15T18:07:19.071723+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.103004+0000","last_change":"2024-01-15T18:07:20.097835+0000","last_active":"2024-01-15T18:07:20.103004+0000","last_peered":"2024-01-15T18:07:20.103004+0000","last_clean":"2024-01-15T18:07:20.103004+0000","last_became_active":"2024-01-15T18:07:20.097477+0000","last_became_peered":"2024-01-15T18:07:20.097477+0000","last_unstale":"2024-01-15T18:07:20.103004+0000","last_undegraded":"2024-01-15T18:07:20.103004+0000","last_fullsized":"2024-01-15T18:07:20.103004+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:36:51.036352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097114+0000","last_change":"2024-01-15T18:07:20.083756+0000","last_active":"2024-01-15T18:07:20.097114+0000","last_peered":"2024-01-15T18:07:20.097114+0000","last_clean":"2024-01-15T18:07:20.097114+0000","last_became_active":"2024-01-15T18:07:20.079072+0000","last_became_peered":"2024-01-15T18:07:20.079072+0000","last_unstale":"2024-01-15T18:07:20.097114+0000","last_undegraded":"2024-01-15T18:07:20.097114+0000","last_fullsized":"2024-01-15T18:07:20.097114+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:55:26.194143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.105788+0000","last_change":"2024-01-15T18:07:20.096208+0000","last_active":"2024-01-15T18:07:20.105788+0000","last_peered":"2024-01-15T18:07:20.105788+0000","last_clean":"2024-01-15T18:07:20.105788+0000","last_became_active":"2024-01-15T18:07:20.095900+0000","last_became_peered":"2024-01-15T18:07:20.095900+0000","last_unstale":"2024-01-15T18:07:20.105788+0000","last_undegraded":"2024-01-15T18:07:20.105788+0000","last_fullsized":"2024-01-15T18:07:20.105788+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:50:12.413904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.106884+0000","last_change":"2024-01-15T18:07:20.091218+0000","last_active":"2024-01-15T18:07:20.106884+0000","last_peered":"2024-01-15T18:07:20.106884+0000","last_clean":"2024-01-15T18:07:20.106884+0000","last_became_active":"2024-01-15T18:07:20.090896+0000","last_became_peered":"2024-01-15T18:07:20.090896+0000","last_unstale":"2024-01-15T18:07:20.106884+0000","last_undegraded":"2024-01-15T18:07:20.106884+0000","last_fullsized":"2024-01-15T18:07:20.106884+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:54:56.923677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098011+0000","last_change":"2024-01-15T18:07:20.087031+0000","last_active":"2024-01-15T18:07:20.098011+0000","last_peered":"2024-01-15T18:07:20.098011+0000","last_clean":"2024-01-15T18:07:20.098011+0000","last_became_active":"2024-01-15T18:07:20.086158+0000","last_became_peered":"2024-01-15T18:07:20.086158+0000","last_unstale":"2024-01-15T18:07:20.098011+0000","last_undegraded":"2024-01-15T18:07:20.098011+0000","last_fullsized":"2024-01-15T18:07:20.098011+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:55:26.194143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098945+0000","last_change":"2024-01-15T18:07:19.085120+0000","last_active":"2024-01-15T18:07:20.098945+0000","last_peered":"2024-01-15T18:07:20.098945+0000","last_clean":"2024-01-15T18:07:20.098945+0000","last_became_active":"2024-01-15T18:07:19.084785+0000","last_became_peered":"2024-01-15T18:07:19.084785+0000","last_unstale":"2024-01-15T18:07:20.098945+0000","last_undegraded":"2024-01-15T18:07:20.098945+0000","last_fullsized":"2024-01-15T18:07:20.098945+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:50:51.007855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100675+0000","last_change":"2024-01-15T18:07:20.094786+0000","last_active":"2024-01-15T18:07:20.100675+0000","last_peered":"2024-01-15T18:07:20.100675+0000","last_clean":"2024-01-15T18:07:20.100675+0000","last_became_active":"2024-01-15T18:07:20.094523+0000","last_became_peered":"2024-01-15T18:07:20.094523+0000","last_unstale":"2024-01-15T18:07:20.100675+0000","last_undegraded":"2024-01-15T18:07:20.100675+0000","last_fullsized":"2024-01-15T18:07:20.100675+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:59:01.979409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.096749+0000","last_change":"2024-01-15T18:07:19.084713+0000","last_active":"2024-01-15T18:07:20.096749+0000","last_peered":"2024-01-15T18:07:20.096749+0000","last_clean":"2024-01-15T18:07:20.096749+0000","last_became_active":"2024-01-15T18:07:19.084414+0000","last_became_peered":"2024-01-15T18:07:19.084414+0000","last_unstale":"2024-01-15T18:07:20.096749+0000","last_undegraded":"2024-01-15T18:07:20.096749+0000","last_fullsized":"2024-01-15T18:07:20.096749+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:20.019967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.068126+0000","last_change":"2024-01-15T18:07:19.068126+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.068126+0000","last_undegraded":"2024-01-15T18:07:19.068126+0000","last_fullsized":"2024-01-15T18:07:19.068126+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100965+0000","last_change":"2024-01-15T18:07:19.077287+0000","last_active":"2024-01-15T18:07:20.100965+0000","last_peered":"2024-01-15T18:07:20.100965+0000","last_clean":"2024-01-15T18:07:20.100965+0000","last_became_active":"2024-01-15T18:07:19.075652+0000","last_became_peered":"2024-01-15T18:07:19.075652+0000","last_unstale":"2024-01-15T18:07:20.100965+0000","last_undegraded":"2024-01-15T18:07:20.100965+0000","last_fullsized":"2024-01-15T18:07:20.100965+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:57:56.264139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098517+0000","last_change":"2024-01-15T18:07:20.087874+0000","last_active":"2024-01-15T18:07:20.098517+0000","last_peered":"2024-01-15T18:07:20.098517+0000","last_clean":"2024-01-15T18:07:20.098517+0000","last_became_active":"2024-01-15T18:07:20.087579+0000","last_became_peered":"2024-01-15T18:07:20.087579+0000","last_unstale":"2024-01-15T18:07:20.098517+0000","last_undegraded":"2024-01-15T18:07:20.098517+0000","last_fullsized":"2024-01-15T18:07:20.098517+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:20:04.640555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098218+0000","last_change":"2024-01-15T18:07:19.085001+0000","last_active":"2024-01-15T18:07:20.098218+0000","last_peered":"2024-01-15T18:07:20.098218+0000","last_clean":"2024-01-15T18:07:20.098218+0000","last_became_active":"2024-01-15T18:07:19.084651+0000","last_became_peered":"2024-01-15T18:07:19.084651+0000","last_unstale":"2024-01-15T18:07:20.098218+0000","last_undegraded":"2024-01-15T18:07:20.098218+0000","last_fullsized":"2024-01-15T18:07:20.098218+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:13:24.552254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097208+0000","last_change":"2024-01-15T18:07:20.087693+0000","last_active":"2024-01-15T18:07:20.097208+0000","last_peered":"2024-01-15T18:07:20.097208+0000","last_clean":"2024-01-15T18:07:20.097208+0000","last_became_active":"2024-01-15T18:07:20.087416+0000","last_became_peered":"2024-01-15T18:07:20.087416+0000","last_unstale":"2024-01-15T18:07:20.097208+0000","last_undegraded":"2024-01-15T18:07:20.097208+0000","last_fullsized":"2024-01-15T18:07:20.097208+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:02.825993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097250+0000","last_change":"2024-01-15T18:07:19.086152+0000","last_active":"2024-01-15T18:07:20.097250+0000","last_peered":"2024-01-15T18:07:20.097250+0000","last_clean":"2024-01-15T18:07:20.097250+0000","last_became_active":"2024-01-15T18:07:19.082058+0000","last_became_peered":"2024-01-15T18:07:19.082058+0000","last_unstale":"2024-01-15T18:07:20.097250+0000","last_undegraded":"2024-01-15T18:07:20.097250+0000","last_fullsized":"2024-01-15T18:07:20.097250+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:06:20.719015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.074401+0000","last_change":"2024-01-15T18:07:19.074401+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.074401+0000","last_undegraded":"2024-01-15T18:07:19.074401+0000","last_fullsized":"2024-01-15T18:07:19.074401+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097139+0000","last_change":"2024-01-15T18:07:19.076932+0000","last_active":"2024-01-15T18:07:20.097139+0000","last_peered":"2024-01-15T18:07:20.097139+0000","last_clean":"2024-01-15T18:07:20.097139+0000","last_became_active":"2024-01-15T18:07:19.076629+0000","last_became_peered":"2024-01-15T18:07:19.076629+0000","last_unstale":"2024-01-15T18:07:20.097139+0000","last_undegraded":"2024-01-15T18:07:20.097139+0000","last_fullsized":"2024-01-15T18:07:20.097139+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:24:59.066289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.069432+0000","last_change":"2024-01-15T18:07:19.069432+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.069432+0000","last_undegraded":"2024-01-15T18:07:19.069432+0000","last_fullsized":"2024-01-15T18:07:19.069432+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.084299+0000","last_change":"2024-01-15T18:07:19.079711+0000","last_active":"2024-01-15T18:07:19.084299+0000","last_peered":"2024-01-15T18:07:19.084299+0000","last_clean":"2024-01-15T18:07:19.084299+0000","last_became_active":"2024-01-15T18:07:19.079336+0000","last_became_peered":"2024-01-15T18:07:19.079336+0000","last_unstale":"2024-01-15T18:07:19.084299+0000","last_undegraded":"2024-01-15T18:07:19.084299+0000","last_fullsized":"2024-01-15T18:07:19.084299+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:49:49.489371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098255+0000","last_change":"2024-01-15T18:07:20.087900+0000","last_active":"2024-01-15T18:07:20.098255+0000","last_peered":"2024-01-15T18:07:20.098255+0000","last_clean":"2024-01-15T18:07:20.098255+0000","last_became_active":"2024-01-15T18:07:20.087463+0000","last_became_peered":"2024-01-15T18:07:20.087463+0000","last_unstale":"2024-01-15T18:07:20.098255+0000","last_undegraded":"2024-01-15T18:07:20.098255+0000","last_fullsized":"2024-01-15T18:07:20.098255+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:54:09.185168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.082790+0000","last_change":"2024-01-15T18:07:19.079615+0000","last_active":"2024-01-15T18:07:19.082790+0000","last_peered":"2024-01-15T18:07:19.082790+0000","last_clean":"2024-01-15T18:07:19.082790+0000","last_became_active":"2024-01-15T18:07:19.079242+0000","last_became_peered":"2024-01-15T18:07:19.079242+0000","last_unstale":"2024-01-15T18:07:19.082790+0000","last_undegraded":"2024-01-15T18:07:19.082790+0000","last_fullsized":"2024-01-15T18:07:19.082790+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:47.335570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100426+0000","last_change":"2024-01-15T18:07:20.095844+0000","last_active":"2024-01-15T18:07:20.100426+0000","last_peered":"2024-01-15T18:07:20.100426+0000","last_clean":"2024-01-15T18:07:20.100426+0000","last_became_active":"2024-01-15T18:07:20.095110+0000","last_became_peered":"2024-01-15T18:07:20.095110+0000","last_unstale":"2024-01-15T18:07:20.100426+0000","last_undegraded":"2024-01-15T18:07:20.100426+0000","last_fullsized":"2024-01-15T18:07:20.100426+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:57:57.266821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.070487+0000","last_change":"2024-01-15T18:07:19.076938+0000","last_active":"2024-01-15T18:07:20.070487+0000","last_peered":"2024-01-15T18:07:20.070487+0000","last_clean":"2024-01-15T18:07:20.070487+0000","last_became_active":"2024-01-15T18:07:19.076605+0000","last_became_peered":"2024-01-15T18:07:19.076605+0000","last_unstale":"2024-01-15T18:07:20.070487+0000","last_undegraded":"2024-01-15T18:07:20.070487+0000","last_fullsized":"2024-01-15T18:07:20.070487+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:32.749313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.095519+0000","last_change":"2024-01-15T18:07:20.091131+0000","last_active":"2024-01-15T18:07:20.095519+0000","last_peered":"2024-01-15T18:07:20.095519+0000","last_clean":"2024-01-15T18:07:20.095519+0000","last_became_active":"2024-01-15T18:07:20.090683+0000","last_became_peered":"2024-01-15T18:07:20.090683+0000","last_unstale":"2024-01-15T18:07:20.095519+0000","last_undegraded":"2024-01-15T18:07:20.095519+0000","last_fullsized":"2024-01-15T18:07:20.095519+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:36:14.191387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.086659+0000","last_change":"2024-01-15T18:07:19.085014+0000","last_active":"2024-01-15T18:07:19.086659+0000","last_peered":"2024-01-15T18:07:19.086659+0000","last_clean":"2024-01-15T18:07:19.086659+0000","last_became_active":"2024-01-15T18:07:19.084470+0000","last_became_peered":"2024-01-15T18:07:19.084470+0000","last_unstale":"2024-01-15T18:07:19.086659+0000","last_undegraded":"2024-01-15T18:07:19.086659+0000","last_fullsized":"2024-01-15T18:07:19.086659+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:43:21.743406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.114219+0000","last_change":"2024-01-15T18:07:20.108821+0000","last_active":"2024-01-15T18:07:20.114219+0000","last_peered":"2024-01-15T18:07:20.114219+0000","last_clean":"2024-01-15T18:07:20.114219+0000","last_became_active":"2024-01-15T18:07:20.108464+0000","last_became_peered":"2024-01-15T18:07:20.108464+0000","last_unstale":"2024-01-15T18:07:20.114219+0000","last_undegraded":"2024-01-15T18:07:20.114219+0000","last_fullsized":"2024-01-15T18:07:20.114219+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:23:28.608885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"44'3","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.099259+0000","last_change":"2024-01-15T18:07:19.088202+0000","last_active":"2024-01-15T18:07:20.099259+0000","last_peered":"2024-01-15T18:07:20.099259+0000","last_clean":"2024-01-15T18:07:20.099259+0000","last_became_active":"2024-01-15T18:07:19.087900+0000","last_became_peered":"2024-01-15T18:07:19.087900+0000","last_unstale":"2024-01-15T18:07:20.099259+0000","last_undegraded":"2024-01-15T18:07:20.099259+0000","last_fullsized":"2024-01-15T18:07:20.099259+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:10:42.345570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.106599+0000","last_change":"2024-01-15T18:07:20.102897+0000","last_active":"2024-01-15T18:07:20.106599+0000","last_peered":"2024-01-15T18:07:20.106599+0000","last_clean":"2024-01-15T18:07:20.106599+0000","last_became_active":"2024-01-15T18:07:20.102579+0000","last_became_peered":"2024-01-15T18:07:20.102579+0000","last_unstale":"2024-01-15T18:07:20.106599+0000","last_undegraded":"2024-01-15T18:07:20.106599+0000","last_fullsized":"2024-01-15T18:07:20.106599+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:50:12.413904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":17,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.112054+0000","last_change":"2024-01-15T18:07:19.079571+0000","last_active":"2024-01-15T18:07:20.112054+0000","last_peered":"2024-01-15T18:07:20.112054+0000","last_clean":"2024-01-15T18:07:20.112054+0000","last_became_active":"2024-01-15T18:07:19.079308+0000","last_became_peered":"2024-01-15T18:07:19.079308+0000","last_unstale":"2024-01-15T18:07:20.112054+0000","last_undegraded":"2024-01-15T18:07:20.112054+0000","last_fullsized":"2024-01-15T18:07:20.112054+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:25.046543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.101579+0000","last_change":"2024-01-15T18:07:20.097611+0000","last_active":"2024-01-15T18:07:20.101579+0000","last_peered":"2024-01-15T18:07:20.101579+0000","last_clean":"2024-01-15T18:07:20.101579+0000","last_became_active":"2024-01-15T18:07:20.097202+0000","last_became_peered":"2024-01-15T18:07:20.097202+0000","last_unstale":"2024-01-15T18:07:20.101579+0000","last_undegraded":"2024-01-15T18:07:20.101579+0000","last_fullsized":"2024-01-15T18:07:20.101579+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:52:34.679712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.104292+0000","last_change":"2024-01-15T18:07:19.084662+0000","last_active":"2024-01-15T18:07:20.104292+0000","last_peered":"2024-01-15T18:07:20.104292+0000","last_clean":"2024-01-15T18:07:20.104292+0000","last_became_active":"2024-01-15T18:07:19.084348+0000","last_became_peered":"2024-01-15T18:07:19.084348+0000","last_unstale":"2024-01-15T18:07:20.104292+0000","last_undegraded":"2024-01-15T18:07:20.104292+0000","last_fullsized":"2024-01-15T18:07:20.104292+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:07:49.351274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.067641+0000","last_change":"2024-01-15T18:07:19.067641+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.067641+0000","last_undegraded":"2024-01-15T18:07:19.067641+0000","last_fullsized":"2024-01-15T18:07:19.067641+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.076930+0000","last_change":"2024-01-15T18:07:19.084512+0000","last_active":"2024-01-15T18:07:20.076930+0000","last_peered":"2024-01-15T18:07:20.076930+0000","last_clean":"2024-01-15T18:07:20.076930+0000","last_became_active":"2024-01-15T18:07:19.084169+0000","last_became_peered":"2024-01-15T18:07:19.084169+0000","last_unstale":"2024-01-15T18:07:20.076930+0000","last_undegraded":"2024-01-15T18:07:20.076930+0000","last_fullsized":"2024-01-15T18:07:20.076930+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:35:27.040611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.109334+0000","last_change":"2024-01-15T18:07:20.105484+0000","last_active":"2024-01-15T18:07:20.109334+0000","last_peered":"2024-01-15T18:07:20.109334+0000","last_clean":"2024-01-15T18:07:20.109334+0000","last_became_active":"2024-01-15T18:07:20.105160+0000","last_became_peered":"2024-01-15T18:07:20.105160+0000","last_unstale":"2024-01-15T18:07:20.109334+0000","last_undegraded":"2024-01-15T18:07:20.109334+0000","last_fullsized":"2024-01-15T18:07:20.109334+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:04:14.022333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.073804+0000","last_change":"2024-01-15T18:07:19.083510+0000","last_active":"2024-01-15T18:07:20.073804+0000","last_peered":"2024-01-15T18:07:20.073804+0000","last_clean":"2024-01-15T18:07:20.073804+0000","last_became_active":"2024-01-15T18:07:19.083134+0000","last_became_peered":"2024-01-15T18:07:19.083134+0000","last_unstale":"2024-01-15T18:07:20.073804+0000","last_undegraded":"2024-01-15T18:07:20.073804+0000","last_fullsized":"2024-01-15T18:07:20.073804+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:10:42.345570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.103258+0000","last_change":"2024-01-15T18:07:20.101397+0000","last_active":"2024-01-15T18:07:20.103258+0000","last_peered":"2024-01-15T18:07:20.103258+0000","last_clean":"2024-01-15T18:07:20.103258+0000","last_became_active":"2024-01-15T18:07:20.101106+0000","last_became_peered":"2024-01-15T18:07:20.101106+0000","last_unstale":"2024-01-15T18:07:20.103258+0000","last_undegraded":"2024-01-15T18:07:20.103258+0000","last_fullsized":"2024-01-15T18:07:20.103258+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:48:24.482600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.088352+0000","last_change":"2024-01-15T18:07:19.086583+0000","last_active":"2024-01-15T18:07:19.088352+0000","last_peered":"2024-01-15T18:07:19.088352+0000","last_clean":"2024-01-15T18:07:19.088352+0000","last_became_active":"2024-01-15T18:07:19.086227+0000","last_became_peered":"2024-01-15T18:07:19.086227+0000","last_unstale":"2024-01-15T18:07:19.088352+0000","last_undegraded":"2024-01-15T18:07:19.088352+0000","last_fullsized":"2024-01-15T18:07:19.088352+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:24:53.993680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.102535+0000","last_change":"2024-01-15T18:07:20.075833+0000","last_active":"2024-01-15T18:07:20.102535+0000","last_peered":"2024-01-15T18:07:20.102535+0000","last_clean":"2024-01-15T18:07:20.102535+0000","last_became_active":"2024-01-15T18:07:20.075521+0000","last_became_peered":"2024-01-15T18:07:20.075521+0000","last_unstale":"2024-01-15T18:07:20.102535+0000","last_undegraded":"2024-01-15T18:07:20.102535+0000","last_fullsized":"2024-01-15T18:07:20.102535+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:52:34.679712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.107096+0000","last_change":"2024-01-15T18:07:19.087887+0000","last_active":"2024-01-15T18:07:20.107096+0000","last_peered":"2024-01-15T18:07:20.107096+0000","last_clean":"2024-01-15T18:07:20.107096+0000","last_became_active":"2024-01-15T18:07:19.087610+0000","last_became_peered":"2024-01-15T18:07:19.087610+0000","last_unstale":"2024-01-15T18:07:20.107096+0000","last_undegraded":"2024-01-15T18:07:20.107096+0000","last_fullsized":"2024-01-15T18:07:20.107096+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:29:39.445420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.068689+0000","last_change":"2024-01-15T18:07:19.068689+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.068689+0000","last_undegraded":"2024-01-15T18:07:19.068689+0000","last_fullsized":"2024-01-15T18:07:19.068689+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.101388+0000","last_change":"2024-01-15T18:07:19.074682+0000","last_active":"2024-01-15T18:07:20.101388+0000","last_peered":"2024-01-15T18:07:20.101388+0000","last_clean":"2024-01-15T18:07:20.101388+0000","last_became_active":"2024-01-15T18:07:19.074397+0000","last_became_peered":"2024-01-15T18:07:19.074397+0000","last_unstale":"2024-01-15T18:07:20.101388+0000","last_undegraded":"2024-01-15T18:07:20.101388+0000","last_fullsized":"2024-01-15T18:07:20.101388+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:37:09.401497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100562+0000","last_change":"2024-01-15T18:07:20.093962+0000","last_active":"2024-01-15T18:07:20.100562+0000","last_peered":"2024-01-15T18:07:20.100562+0000","last_clean":"2024-01-15T18:07:20.100562+0000","last_became_active":"2024-01-15T18:07:20.093638+0000","last_became_peered":"2024-01-15T18:07:20.093638+0000","last_unstale":"2024-01-15T18:07:20.100562+0000","last_undegraded":"2024-01-15T18:07:20.100562+0000","last_fullsized":"2024-01-15T18:07:20.100562+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:30:43.231075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.086984+0000","last_change":"2024-01-15T18:07:19.082541+0000","last_active":"2024-01-15T18:07:19.086984+0000","last_peered":"2024-01-15T18:07:19.086984+0000","last_clean":"2024-01-15T18:07:19.086984+0000","last_became_active":"2024-01-15T18:07:19.082192+0000","last_became_peered":"2024-01-15T18:07:19.082192+0000","last_unstale":"2024-01-15T18:07:19.086984+0000","last_undegraded":"2024-01-15T18:07:19.086984+0000","last_fullsized":"2024-01-15T18:07:19.086984+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:10:51.637250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.103168+0000","last_change":"2024-01-15T18:07:20.096891+0000","last_active":"2024-01-15T18:07:20.103168+0000","last_peered":"2024-01-15T18:07:20.103168+0000","last_clean":"2024-01-15T18:07:20.103168+0000","last_became_active":"2024-01-15T18:07:20.096533+0000","last_became_peered":"2024-01-15T18:07:20.096533+0000","last_unstale":"2024-01-15T18:07:20.103168+0000","last_undegraded":"2024-01-15T18:07:20.103168+0000","last_fullsized":"2024-01-15T18:07:20.103168+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:14:41.208287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.075980+0000","last_change":"2024-01-15T18:07:19.080295+0000","last_active":"2024-01-15T18:07:20.075980+0000","last_peered":"2024-01-15T18:07:20.075980+0000","last_clean":"2024-01-15T18:07:20.075980+0000","last_became_active":"2024-01-15T18:07:19.080024+0000","last_became_peered":"2024-01-15T18:07:19.080024+0000","last_unstale":"2024-01-15T18:07:20.075980+0000","last_undegraded":"2024-01-15T18:07:20.075980+0000","last_fullsized":"2024-01-15T18:07:20.075980+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:36:48.732986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.112236+0000","last_change":"2024-01-15T18:07:20.105757+0000","last_active":"2024-01-15T18:07:20.112236+0000","last_peered":"2024-01-15T18:07:20.112236+0000","last_clean":"2024-01-15T18:07:20.112236+0000","last_became_active":"2024-01-15T18:07:20.105385+0000","last_became_peered":"2024-01-15T18:07:20.105385+0000","last_unstale":"2024-01-15T18:07:20.112236+0000","last_undegraded":"2024-01-15T18:07:20.112236+0000","last_fullsized":"2024-01-15T18:07:20.112236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:51:04.157158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.097287+0000","last_change":"2024-01-15T18:07:19.082297+0000","last_active":"2024-01-15T18:07:20.097287+0000","last_peered":"2024-01-15T18:07:20.097287+0000","last_clean":"2024-01-15T18:07:20.097287+0000","last_became_active":"2024-01-15T18:07:19.081971+0000","last_became_peered":"2024-01-15T18:07:19.081971+0000","last_unstale":"2024-01-15T18:07:20.097287+0000","last_undegraded":"2024-01-15T18:07:20.097287+0000","last_fullsized":"2024-01-15T18:07:20.097287+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:23:41.478738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.098551+0000","last_change":"2024-01-15T18:07:20.091112+0000","last_active":"2024-01-15T18:07:20.098551+0000","last_peered":"2024-01-15T18:07:20.098551+0000","last_clean":"2024-01-15T18:07:20.098551+0000","last_became_active":"2024-01-15T18:07:20.090802+0000","last_became_peered":"2024-01-15T18:07:20.090802+0000","last_unstale":"2024-01-15T18:07:20.098551+0000","last_undegraded":"2024-01-15T18:07:20.098551+0000","last_fullsized":"2024-01-15T18:07:20.098551+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:01:01.613225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.082669+0000","last_change":"2024-01-15T18:07:19.077896+0000","last_active":"2024-01-15T18:07:19.082669+0000","last_peered":"2024-01-15T18:07:19.082669+0000","last_clean":"2024-01-15T18:07:19.082669+0000","last_became_active":"2024-01-15T18:07:19.077439+0000","last_became_peered":"2024-01-15T18:07:19.077439+0000","last_unstale":"2024-01-15T18:07:19.082669+0000","last_undegraded":"2024-01-15T18:07:19.082669+0000","last_fullsized":"2024-01-15T18:07:19.082669+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:09:06.512495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.104003+0000","last_change":"2024-01-15T18:07:20.086589+0000","last_active":"2024-01-15T18:07:20.104003+0000","last_peered":"2024-01-15T18:07:20.104003+0000","last_clean":"2024-01-15T18:07:20.104003+0000","last_became_active":"2024-01-15T18:07:20.086312+0000","last_became_peered":"2024-01-15T18:07:20.086312+0000","last_unstale":"2024-01-15T18:07:20.104003+0000","last_undegraded":"2024-01-15T18:07:20.104003+0000","last_fullsized":"2024-01-15T18:07:20.104003+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:15:59.320681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.083833+0000","last_change":"2024-01-15T18:07:19.082155+0000","last_active":"2024-01-15T18:07:19.083833+0000","last_peered":"2024-01-15T18:07:19.083833+0000","last_clean":"2024-01-15T18:07:19.083833+0000","last_became_active":"2024-01-15T18:07:19.081855+0000","last_became_peered":"2024-01-15T18:07:19.081855+0000","last_unstale":"2024-01-15T18:07:19.083833+0000","last_undegraded":"2024-01-15T18:07:19.083833+0000","last_fullsized":"2024-01-15T18:07:19.083833+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:07.014633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.067698+0000","last_change":"2024-01-15T18:07:19.067698+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.067698+0000","last_undegraded":"2024-01-15T18:07:19.067698+0000","last_fullsized":"2024-01-15T18:07:19.067698+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.076645+0000","last_change":"2024-01-15T18:07:19.085582+0000","last_active":"2024-01-15T18:07:20.076645+0000","last_peered":"2024-01-15T18:07:20.076645+0000","last_clean":"2024-01-15T18:07:20.076645+0000","last_became_active":"2024-01-15T18:07:19.083796+0000","last_became_peered":"2024-01-15T18:07:19.083796+0000","last_unstale":"2024-01-15T18:07:20.076645+0000","last_undegraded":"2024-01-15T18:07:20.076645+0000","last_fullsized":"2024-01-15T18:07:20.076645+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:03:20.187627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.106501+0000","last_change":"2024-01-15T18:07:20.100547+0000","last_active":"2024-01-15T18:07:20.106501+0000","last_peered":"2024-01-15T18:07:20.106501+0000","last_clean":"2024-01-15T18:07:20.106501+0000","last_became_active":"2024-01-15T18:07:20.100201+0000","last_became_peered":"2024-01-15T18:07:20.100201+0000","last_unstale":"2024-01-15T18:07:20.106501+0000","last_undegraded":"2024-01-15T18:07:20.106501+0000","last_fullsized":"2024-01-15T18:07:20.106501+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:29:40.448102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.072838+0000","last_change":"2024-01-15T18:07:19.078965+0000","last_active":"2024-01-15T18:07:20.072838+0000","last_peered":"2024-01-15T18:07:20.072838+0000","last_clean":"2024-01-15T18:07:20.072838+0000","last_became_active":"2024-01-15T18:07:19.078692+0000","last_became_peered":"2024-01-15T18:07:19.078692+0000","last_unstale":"2024-01-15T18:07:20.072838+0000","last_undegraded":"2024-01-15T18:07:20.072838+0000","last_fullsized":"2024-01-15T18:07:20.072838+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:13:24.552254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.067320+0000","last_change":"2024-01-15T18:07:19.067320+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.067320+0000","last_undegraded":"2024-01-15T18:07:19.067320+0000","last_fullsized":"2024-01-15T18:07:19.067320+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.077286+0000","last_change":"2024-01-15T18:07:19.088174+0000","last_active":"2024-01-15T18:07:20.077286+0000","last_peered":"2024-01-15T18:07:20.077286+0000","last_clean":"2024-01-15T18:07:20.077286+0000","last_became_active":"2024-01-15T18:07:19.087881+0000","last_became_peered":"2024-01-15T18:07:19.087881+0000","last_unstale":"2024-01-15T18:07:20.077286+0000","last_undegraded":"2024-01-15T18:07:20.077286+0000","last_fullsized":"2024-01-15T18:07:20.077286+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:36:48.732986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.108471+0000","last_change":"2024-01-15T18:07:20.104788+0000","last_active":"2024-01-15T18:07:20.108471+0000","last_peered":"2024-01-15T18:07:20.108471+0000","last_clean":"2024-01-15T18:07:20.108471+0000","last_became_active":"2024-01-15T18:07:20.104475+0000","last_became_peered":"2024-01-15T18:07:20.104475+0000","last_unstale":"2024-01-15T18:07:20.108471+0000","last_undegraded":"2024-01-15T18:07:20.108471+0000","last_fullsized":"2024-01-15T18:07:20.108471+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:02:26.277338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.071516+0000","last_change":"2024-01-15T18:07:19.080590+0000","last_active":"2024-01-15T18:07:20.071516+0000","last_peered":"2024-01-15T18:07:20.071516+0000","last_clean":"2024-01-15T18:07:20.071516+0000","last_became_active":"2024-01-15T18:07:19.080276+0000","last_became_peered":"2024-01-15T18:07:19.080276+0000","last_unstale":"2024-01-15T18:07:20.071516+0000","last_undegraded":"2024-01-15T18:07:20.071516+0000","last_fullsized":"2024-01-15T18:07:20.071516+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:24:59.066289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100402+0000","last_change":"2024-01-15T18:07:20.090127+0000","last_active":"2024-01-15T18:07:20.100402+0000","last_peered":"2024-01-15T18:07:20.100402+0000","last_clean":"2024-01-15T18:07:20.100402+0000","last_became_active":"2024-01-15T18:07:20.089727+0000","last_became_peered":"2024-01-15T18:07:20.089727+0000","last_unstale":"2024-01-15T18:07:20.100402+0000","last_undegraded":"2024-01-15T18:07:20.100402+0000","last_fullsized":"2024-01-15T18:07:20.100402+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:36:14.191387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.077270+0000","last_change":"2024-01-15T18:07:19.088037+0000","last_active":"2024-01-15T18:07:20.077270+0000","last_peered":"2024-01-15T18:07:20.077270+0000","last_clean":"2024-01-15T18:07:20.077270+0000","last_became_active":"2024-01-15T18:07:19.087721+0000","last_became_peered":"2024-01-15T18:07:19.087721+0000","last_unstale":"2024-01-15T18:07:20.077270+0000","last_undegraded":"2024-01-15T18:07:20.077270+0000","last_fullsized":"2024-01-15T18:07:20.077270+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:23:27.606202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.111942+0000","last_change":"2024-01-15T18:07:20.105653+0000","last_active":"2024-01-15T18:07:20.111942+0000","last_peered":"2024-01-15T18:07:20.111942+0000","last_clean":"2024-01-15T18:07:20.111942+0000","last_became_active":"2024-01-15T18:07:20.105348+0000","last_became_peered":"2024-01-15T18:07:20.105348+0000","last_unstale":"2024-01-15T18:07:20.111942+0000","last_undegraded":"2024-01-15T18:07:20.111942+0000","last_fullsized":"2024-01-15T18:07:20.111942+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:02:26.277338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.068819+0000","last_change":"2024-01-15T18:07:19.076027+0000","last_active":"2024-01-15T18:07:20.068819+0000","last_peered":"2024-01-15T18:07:20.068819+0000","last_clean":"2024-01-15T18:07:20.068819+0000","last_became_active":"2024-01-15T18:07:19.075644+0000","last_became_peered":"2024-01-15T18:07:19.075644+0000","last_unstale":"2024-01-15T18:07:20.068819+0000","last_undegraded":"2024-01-15T18:07:20.068819+0000","last_fullsized":"2024-01-15T18:07:20.068819+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:07:57.062677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.064957+0000","last_change":"2024-01-15T18:07:19.064957+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.064957+0000","last_undegraded":"2024-01-15T18:07:19.064957+0000","last_fullsized":"2024-01-15T18:07:19.064957+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.072048+0000","last_change":"2024-01-15T18:07:19.081215+0000","last_active":"2024-01-15T18:07:20.072048+0000","last_peered":"2024-01-15T18:07:20.072048+0000","last_clean":"2024-01-15T18:07:20.072048+0000","last_became_active":"2024-01-15T18:07:19.080901+0000","last_became_peered":"2024-01-15T18:07:19.080901+0000","last_unstale":"2024-01-15T18:07:20.072048+0000","last_undegraded":"2024-01-15T18:07:20.072048+0000","last_fullsized":"2024-01-15T18:07:20.072048+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:38:52.124114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.094487+0000","last_change":"2024-01-15T18:07:20.081122+0000","last_active":"2024-01-15T18:07:20.094487+0000","last_peered":"2024-01-15T18:07:20.094487+0000","last_clean":"2024-01-15T18:07:20.094487+0000","last_became_active":"2024-01-15T18:07:20.080752+0000","last_became_peered":"2024-01-15T18:07:20.080752+0000","last_unstale":"2024-01-15T18:07:20.094487+0000","last_undegraded":"2024-01-15T18:07:20.094487+0000","last_fullsized":"2024-01-15T18:07:20.094487+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:08:56.987120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.087492+0000","last_change":"2024-01-15T18:07:19.085696+0000","last_active":"2024-01-15T18:07:19.087492+0000","last_peered":"2024-01-15T18:07:19.087492+0000","last_clean":"2024-01-15T18:07:19.087492+0000","last_became_active":"2024-01-15T18:07:19.085467+0000","last_became_peered":"2024-01-15T18:07:19.085467+0000","last_unstale":"2024-01-15T18:07:19.087492+0000","last_undegraded":"2024-01-15T18:07:19.087492+0000","last_fullsized":"2024-01-15T18:07:19.087492+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:26:07.301619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-01-15T18:07:19.068029+0000","last_change":"2024-01-15T18:07:19.068029+0000","last_active":"2024-01-15T18:07:19.050323+0000","last_peered":"2024-01-15T18:07:19.050323+0000","last_clean":"2024-01-15T18:07:19.050323+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:07:19.068029+0000","last_undegraded":"2024-01-15T18:07:19.068029+0000","last_fullsized":"2024-01-15T18:07:19.068029+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.086899+0000","last_change":"2024-01-15T18:07:19.081361+0000","last_active":"2024-01-15T18:07:19.086899+0000","last_peered":"2024-01-15T18:07:19.086899+0000","last_clean":"2024-01-15T18:07:19.086899+0000","last_became_active":"2024-01-15T18:07:19.081076+0000","last_became_peered":"2024-01-15T18:07:19.081076+0000","last_unstale":"2024-01-15T18:07:19.086899+0000","last_undegraded":"2024-01-15T18:07:19.086899+0000","last_fullsized":"2024-01-15T18:07:19.086899+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:34:07.305579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":183,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:07:19.074017+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:07:19.074017+0000","last_peered":"2024-01-15T18:07:19.074017+0000","last_clean":"2024-01-15T18:07:19.074017+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:07:19.074017+0000","last_undegraded":"2024-01-15T18:07:19.074017+0000","last_fullsized":"2024-01-15T18:07:19.074017+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2340,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509474,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32248,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339592,"statfs":{"total":96636764160,"available":96603742208,"internally_reserved":0,"allocated":1089536,"data_stored":789175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.628,"15min":0.625},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.67200000000000004,"15min":0.67700000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.78200000000000003}]},{"osd":1,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.67800000000000005,"15min":0.67100000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.69399999999999995,"15min":0.69799999999999995},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.90400000000000003,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.56000000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64800000000000002,"15min":0.64700000000000002},"min":{"1min":0.46700000000000003,"5min":0.38,"15min":0.38},"max":{"1min":0.84199999999999997,"5min":1.012,"15min":1.012},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.71299999999999997,"15min":0.71199999999999997},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.0980000000000001,"5min":1.175,"15min":1.175},"last":0.68799999999999994}]},{"osd":3,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.66900000000000004,"15min":0.66500000000000004},"min":{"1min":0.502,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.88200000000000001,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0569999999999999,"5min":1.103,"15min":1.103},"last":0.74099999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.60799999999999998,"15min":0.61599999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.69199999999999995,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.65700000000000003,"15min":0.66200000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.622}]},{"osd":5,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.69499999999999995,"15min":0.69899999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.63,"5min":0.70999999999999996,"15min":0.72299999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.89500000000000002,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.70699999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.78800000000000003,"15min":0.80000000000000004},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.89500000000000002,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72499999999999998,"15min":0.73099999999999998},"min":{"1min":0.52500000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.76600000000000001}]}]},{"osd":6,"up_from":15,"seq":64424509474,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31668,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340172,"statfs":{"total":96636764160,"available":96604336128,"internally_reserved":0,"allocated":495616,"data_stored":198807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.432,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.67200000000000004,"15min":0.68899999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84899999999999998,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.755}]},{"osd":1,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.70299999999999996,"15min":0.72399999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.82699999999999996,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.74399999999999999,"15min":0.754},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":0.59099999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.82099999999999995,"15min":0.84399999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.825,"5min":2.8700000000000001,"15min":2.8700000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.755,"15min":0.78300000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.82399999999999995,"5min":1.857,"15min":1.857},"last":0.78600000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.77400000000000002,"15min":0.80100000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.92100000000000004,"5min":2.012,"15min":2.012},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.79900000000000004,"15min":0.82499999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":2.532,"15min":2.532},"last":0.66200000000000003}]},{"osd":4,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68300000000000005,"15min":0.69299999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.726,"5min":1.726,"15min":1.726},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.69399999999999995,"15min":0.71799999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.753,"5min":1.145,"15min":1.145},"last":0.81799999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.64500000000000002,"15min":0.65800000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.81299999999999994,"5min":1.121,"15min":1.121},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.77400000000000002,"15min":0.78100000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"last":0.91400000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.80100000000000005,"15min":0.82499999999999996},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.96499999999999997,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.874},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.78000000000000003,"15min":0.80200000000000005},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.081,"5min":1.165,"15min":1.165},"last":0.80500000000000005}]}]},{"osd":4,"up_from":15,"seq":64424509474,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31688,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340152,"statfs":{"total":96636764160,"available":96604315648,"internally_reserved":0,"allocated":520192,"data_stored":207222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.628,"15min":0.63},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.63200000000000001,"15min":0.64100000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.60299999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64200000000000002,"15min":0.64900000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.77300000000000002,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.628,"5min":0.64000000000000001,"15min":0.64200000000000002},"min":{"1min":0.40899999999999997,"5min":0.377,"15min":0.377},"max":{"1min":0.88,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.66200000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.626,"15min":0.628},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.69099999999999995,"15min":0.69499999999999995},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.81999999999999995,"5min":0.879,"15min":0.879},"last":0.80600000000000005}]},{"osd":3,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68899999999999995,"15min":0.68999999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.96699999999999997,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.72499999999999998,"15min":0.72599999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.93200000000000005,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.79000000000000004}]},{"osd":5,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40200000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65100000000000002,"15min":0.65000000000000002},"min":{"1min":0.40699999999999997,"5min":0.377,"15min":0.377},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.71699999999999997}]},{"osd":6,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.68200000000000005,"15min":0.68000000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.85099999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.46899999999999997,"5min":0.436,"15min":0.436},"max":{"1min":1.0069999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.81799999999999995}]}]},{"osd":3,"up_from":15,"seq":64424509473,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":520192,"data_stored":206248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.28499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.72399999999999998,"5min":0.752,"15min":0.752},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.499,"15min":0.47999999999999998},"min":{"1min":0.34000000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":2.6509999999999998,"5min":2.6509999999999998,"15min":2.6509999999999998},"last":0.38500000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70999999999999996,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.498},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.53700000000000003,"15min":0.53600000000000003},"min":{"1min":0.42499999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83399999999999996,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.52500000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.55800000000000005,"15min":0.56000000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.58599999999999997,"15min":0.57199999999999995},"min":{"1min":0.42899999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.56399999999999995}]},{"osd":4,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.79400000000000004,"15min":0.80300000000000005},"min":{"1min":0.50800000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.119,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.876},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.77300000000000002,"15min":0.77900000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.97899999999999998,"5min":1.341,"15min":1.341},"last":0.65600000000000003}]},{"osd":5,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.59199999999999997,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.72399999999999998,"15min":0.72199999999999998},"min":{"1min":0.53300000000000003,"5min":0.498,"15min":0.498},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":0.73299999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.78300000000000003,"15min":0.78500000000000003},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.77600000000000002,"15min":0.76600000000000001},"min":{"1min":0.628,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":2.0880000000000001,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":0.80200000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.78800000000000003,"15min":0.78100000000000003},"min":{"1min":0.58999999999999997,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.81299999999999994,"15min":0.81999999999999995},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.1259999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.78600000000000003}]}]},{"osd":2,"up_from":15,"seq":64424509473,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31668,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340172,"statfs":{"total":96636764160,"available":96604336128,"internally_reserved":0,"allocated":495616,"data_stored":198807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.45500000000000002,"15min":0.46100000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.75,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.441,"5min":0.46100000000000002,"15min":0.46500000000000002},"min":{"1min":0.29899999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.64700000000000002,"5min":0.753,"15min":0.753},"last":0.46400000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54300000000000004,"15min":0.54500000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.67600000000000005,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.314,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.65100000000000002,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.436}]},{"osd":3,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.57799999999999996,"15min":0.57299999999999995},"min":{"1min":0.375,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.55600000000000005,"15min":0.55300000000000005},"min":{"1min":0.40500000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71499999999999997,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.42099999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.75700000000000001,"15min":0.76700000000000002},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.038,"5min":1.466,"15min":1.466},"last":0.626}]},{"osd":5,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.79400000000000004,"15min":0.79700000000000004},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.609,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.747,"5min":0.78300000000000003,"15min":0.78900000000000003},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.64600000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.76800000000000002,"15min":0.77800000000000002},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0489999999999999,"5min":1.516,"15min":1.516},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.78600000000000003,"15min":0.79600000000000004},"min":{"1min":0.52800000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0820000000000001,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.56100000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.79500000000000004,"15min":0.80100000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.1659999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.81799999999999995,"15min":0.82599999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.69299999999999995}]}]},{"osd":1,"up_from":15,"seq":64424509475,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31668,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340172,"statfs":{"total":96636764160,"available":96604336128,"internally_reserved":0,"allocated":495616,"data_stored":198807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.49099999999999999,"15min":0.504},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.627,"5min":1.847,"15min":1.847},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.443,"5min":0.47899999999999998,"15min":0.48499999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.63500000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.54300000000000004}]},{"osd":2,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.53400000000000003,"15min":0.54400000000000004},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.77500000000000002,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.624},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.70299999999999996,"15min":0.73299999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.90100000000000002,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.46700000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.63100000000000001,"15min":0.64500000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80700000000000005,"5min":1.871,"15min":1.871},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.58099999999999996,"15min":0.59099999999999997},"min":{"1min":0.35299999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.72599999999999998,"5min":1.917,"15min":1.917},"last":0.496}]},{"osd":4,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.70299999999999996,"15min":0.69599999999999995},"min":{"1min":0.54800000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.161,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.77100000000000002,"15min":0.77700000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.147,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.73799999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.76900000000000002,"15min":0.77100000000000002},"min":{"1min":0.47299999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.214,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.53100000000000003,"5min":0.505,"15min":0.505},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.97899999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.753,"15min":0.755},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.1359999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.81599999999999995,"15min":0.82199999999999995},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.95699999999999996}]},{"osd":7,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78700000000000003,"15min":0.78500000000000003},"min":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.79900000000000004,"15min":0.80300000000000005},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.095,"5min":1.347,"15min":1.347},"last":0.90500000000000003}]}]},{"osd":5,"up_from":14,"seq":60129542178,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":483328,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":0.871,"15min":0.77500000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":21.385000000000002,"5min":21.385000000000002,"15min":21.385000000000002},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.202,"5min":0.878,"15min":0.82399999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":8.2029999999999994,"5min":8.2029999999999994,"15min":8.2029999999999994},"last":0.73599999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66400000000000003,"15min":0.67200000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":1.4670000000000001,"5min":0.84099999999999997,"15min":0.73699999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":21.5,"5min":21.5,"15min":21.5},"last":0.80100000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":0.81599999999999995,"15min":0.745},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":8.2439999999999998,"5min":8.2439999999999998,"15min":8.2439999999999998},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.748,"15min":0.77000000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.85099999999999998,"5min":1.663,"15min":1.663},"last":0.71999999999999997}]},{"osd":3,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":0.95299999999999996,"15min":0.85699999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":21.457000000000001,"5min":21.457000000000001,"15min":21.457000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.1919999999999999,"5min":0.92100000000000004,"15min":0.876},"min":{"1min":0.50900000000000001,"5min":0.502,"15min":0.502},"max":{"1min":8.2750000000000004,"5min":8.2750000000000004,"15min":8.2750000000000004},"last":0.70499999999999996}]},{"osd":4,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":0.82599999999999996,"15min":0.76600000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":9.6679999999999993,"5min":9.6679999999999993,"15min":9.6679999999999993},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.71299999999999997,"15min":0.73499999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.76300000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.60099999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":0.877,"15min":0.78400000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":21.535,"5min":21.535,"15min":21.535},"last":0.54100000000000004},{"interface":"front","average":{"1min":1.1870000000000001,"5min":0.88300000000000001,"15min":0.83199999999999996},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":8.298,"5min":8.298,"15min":8.298},"last":0.66700000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":0.83999999999999997,"15min":0.81399999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":9.2240000000000002,"5min":9.2240000000000002,"15min":9.2240000000000002},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.5249999999999999,"5min":0.97099999999999997,"15min":0.879},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":21.574000000000002,"5min":21.574000000000002,"15min":21.574000000000002},"last":0.748}]}]},{"osd":0,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32240,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339600,"statfs":{"total":96636764160,"available":96603750400,"internally_reserved":0,"allocated":1077248,"data_stored":782880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.46300000000000002,"15min":0.46500000000000002},"min":{"1min":0.28999999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.746,"5min":1.462,"15min":1.462},"last":0.501},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55400000000000005,"15min":0.55200000000000005},"min":{"1min":0.34399999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.995,"5min":1.373,"15min":1.373},"last":0.39000000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.371,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.89600000000000002,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.441},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.48699999999999999,"15min":0.48099999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.252,"5min":1.609,"15min":1.609},"last":0.54500000000000004}]},{"osd":3,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.371,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.321,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60899999999999999,"15min":0.60999999999999999},"min":{"1min":0.45000000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.94599999999999995,"5min":1.325,"15min":1.325},"last":0.52700000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.76700000000000002,"15min":0.77700000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.96099999999999997,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.80900000000000005,"15min":0.82599999999999996},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.93300000000000005,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.71099999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.78500000000000003,"15min":0.79500000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0109999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.83099999999999996,"15min":0.84699999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":0.63300000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.79400000000000004,"15min":0.79800000000000004},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.107,"5min":1.764,"15min":1.764},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.83099999999999996,"15min":0.84699999999999998},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.97499999999999998,"5min":1.806,"15min":1.806},"last":0.72699999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.82199999999999995,"15min":0.82999999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.04,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.84899999999999998,"15min":0.85999999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.0429999999999999,"5min":1.722,"15min":1.722},"last":0.67100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:07:26.280 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:07:26.284 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:07:28.753 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:07:28.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.900+0000 7fc72f000640 1 Processor -- start 2024-01-15T18:07:28.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.900+0000 7fc72f000640 1 -- start start 2024-01-15T18:07:28.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc72f000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7280701a0 0x7fc728070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:28.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc72f000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc72806c4a0 con 0x7fc72815c130 2024-01-15T18:07:28.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc72f000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc728070ac0 con 0x7fc7280c92b0 2024-01-15T18:07:28.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc72f000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc72815c660 con 0x7fc7280701a0 2024-01-15T18:07:28.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc72d000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc72815c130 0x7fc72815d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45220/0 (socket says 172.21.15.27:45220) 2024-01-15T18:07:28.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc72d000640 1 -- 172.21.15.27:0/766674245 learned_addr learned my addr 172.21.15.27:0/766674245 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:28.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc727e00640 1 --2- 172.21.15.27:0/766674245 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7280701a0 0x7fc728070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:28.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc726a00640 1 -- 172.21.15.27:0/766674245 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1457962059 0 0) 0x7fc72806c4a0 con 0x7fc72815c130 2024-01-15T18:07:28.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc726a00640 1 -- 172.21.15.27:0/766674245 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc708003530 con 0x7fc72815c130 2024-01-15T18:07:28.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc726a00640 1 -- 172.21.15.27:0/766674245 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3238049385 0 0) 0x7fc708003530 con 0x7fc72815c130 2024-01-15T18:07:28.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc726a00640 1 -- 172.21.15.27:0/766674245 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7280701a0 msgr2=0x7fc728070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:28.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc726a00640 1 --2- 172.21.15.27:0/766674245 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7280701a0 0x7fc728070580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:28.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc726a00640 1 -- 172.21.15.27:0/766674245 >> v1:172.21.15.93:6789/0 conn(0x7fc7280c92b0 legacy=0x7fc72806dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:28.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc726a00640 1 -- 172.21.15.27:0/766674245 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72815c940 con 0x7fc72815c130 2024-01-15T18:07:28.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc726a00640 1 -- 172.21.15.27:0/766674245 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fc720001a00 con 0x7fc72815c130 2024-01-15T18:07:28.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc726a00640 1 -- 172.21.15.27:0/766674245 >> v1:172.21.15.27:6789/0 conn(0x7fc72815c130 legacy=0x7fc72815d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:28.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc726a00640 1 --2- 172.21.15.27:0/766674245 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc708003a20 0x7fc708003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:28.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc726a00640 1 --2- 172.21.15.27:0/766674245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7080047d0 0x7fc708004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:28.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc726a00640 1 --2- 172.21.15.27:0/766674245 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7280701a0 0x7fc708005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:28.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc726a00640 1 -- 172.21.15.27:0/766674245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72815c660 con 0x7fc7080047d0 2024-01-15T18:07:28.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc726a00640 1 -- 172.21.15.27:0/766674245 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70800ba40 con 0x7fc708003a20 2024-01-15T18:07:28.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc726a00640 1 -- 172.21.15.27:0/766674245 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc70800bbd0 con 0x7fc7280701a0 2024-01-15T18:07:28.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc727400640 1 --2- 172.21.15.27:0/766674245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7080047d0 0x7fc708004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:28.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc727e00640 1 --2- 172.21.15.27:0/766674245 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc708003a20 0x7fc708003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:28.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc72d000640 1 --2- 172.21.15.27:0/766674245 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7280701a0 0x7fc708005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:28.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc727400640 1 -- 172.21.15.27:0/766674245 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7280701a0 msgr2=0x7fc708005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:28.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc727400640 1 --2- 172.21.15.27:0/766674245 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7280701a0 0x7fc708005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:28.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc727400640 1 -- 172.21.15.27:0/766674245 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc708003a20 msgr2=0x7fc708003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:28.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc727400640 1 --2- 172.21.15.27:0/766674245 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc708003a20 0x7fc708003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:28.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.904+0000 7fc727400640 1 -- 172.21.15.27:0/766674245 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc708003530 con 0x7fc7080047d0 2024-01-15T18:07:28.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.908+0000 7fc727400640 1 --2- 172.21.15.27:0/766674245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7080047d0 0x7fc708004c50 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fc710001860 tx=0x7fc71001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba482e7248a4d0de server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:28.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.908+0000 7fc726a00640 1 -- 172.21.15.27:0/766674245 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7100ab600 con 0x7fc7080047d0 2024-01-15T18:07:28.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.908+0000 7fc726a00640 1 -- 172.21.15.27:0/766674245 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7100a7050 con 0x7fc7080047d0 2024-01-15T18:07:28.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.908+0000 7fc726a00640 1 -- 172.21.15.27:0/766674245 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7100ab9a0 con 0x7fc7080047d0 2024-01-15T18:07:28.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.908+0000 7fc72f000640 1 -- 172.21.15.27:0/766674245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7080047d0 msgr2=0x7fc708004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:28.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.908+0000 7fc72f000640 1 --2- 172.21.15.27:0/766674245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7080047d0 0x7fc708004c50 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fc710001860 tx=0x7fc71001c450 comp rx=0 tx=0).stop 2024-01-15T18:07:28.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.908+0000 7fc72d000640 1 -- 172.21.15.27:0/766674245 reap_dead start 2024-01-15T18:07:28.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.908+0000 7fc72f000640 1 -- 172.21.15.27:0/766674245 shutdown_connections 2024-01-15T18:07:28.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.908+0000 7fc72f000640 1 -- 172.21.15.27:0/766674245 >> 172.21.15.27:0/766674245 conn(0x7fc7280cec60 msgr2=0x7fc7280cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:28.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.908+0000 7fc72f000640 1 -- 172.21.15.27:0/766674245 shutdown_connections 2024-01-15T18:07:28.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.908+0000 7fc72f000640 1 -- 172.21.15.27:0/766674245 wait complete. 2024-01-15T18:07:28.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.912+0000 7fc72f000640 1 Processor -- start 2024-01-15T18:07:28.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.912+0000 7fc72f000640 1 -- start start 2024-01-15T18:07:28.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.912+0000 7fc72f000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7080047d0 0x7fc7281512a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:28.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.912+0000 7fc72f000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7280cc830 0x7fc7280ce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:28.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.912+0000 7fc72f000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc728153ca0 0x7fc728154130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:28.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.912+0000 7fc72f000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc728154670 con 0x7fc728153ca0 2024-01-15T18:07:28.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.912+0000 7fc72f000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7280cd8d0 con 0x7fc7280cc830 2024-01-15T18:07:28.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.912+0000 7fc72f000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7280cccc0 con 0x7fc7080047d0 2024-01-15T18:07:28.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.912+0000 7fc72d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc728153ca0 0x7fc728154130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:28.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.912+0000 7fc72d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc728153ca0 0x7fc728154130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50790/0 (socket says 172.21.15.27:50790) 2024-01-15T18:07:28.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.912+0000 7fc72d000640 1 -- 172.21.15.27:0/1240770979 learned_addr learned my addr 172.21.15.27:0/1240770979 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:28.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.912+0000 7fc727e00640 1 --2- 172.21.15.27:0/1240770979 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7080047d0 0x7fc7281512a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:28.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.912+0000 7fc727400640 1 --2- 172.21.15.27:0/1240770979 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7280cc830 0x7fc7280ce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:28.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.912+0000 7fc72d000640 1 -- 172.21.15.27:0/1240770979 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7080047d0 msgr2=0x7fc7281512a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:28.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.912+0000 7fc72d000640 1 --2- 172.21.15.27:0/1240770979 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7080047d0 0x7fc7281512a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:28.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.912+0000 7fc72d000640 1 -- 172.21.15.27:0/1240770979 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7280cc830 msgr2=0x7fc7280ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:28.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.912+0000 7fc72d000640 1 --2- 172.21.15.27:0/1240770979 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7280cc830 0x7fc7280ce1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:28.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.912+0000 7fc72d000640 1 -- 172.21.15.27:0/1240770979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7280ccfa0 con 0x7fc728153ca0 2024-01-15T18:07:28.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.912+0000 7fc72d000640 1 --2- 172.21.15.27:0/1240770979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc728153ca0 0x7fc728154130 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fc720002240 tx=0x7fc720007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:28.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.912+0000 7fc71fe00640 1 -- 172.21.15.27:0/1240770979 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72000abc0 con 0x7fc728153ca0 2024-01-15T18:07:28.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.912+0000 7fc71fe00640 1 -- 172.21.15.27:0/1240770979 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72000ad40 con 0x7fc728153ca0 2024-01-15T18:07:28.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.912+0000 7fc72f000640 1 -- 172.21.15.27:0/1240770979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7280cd280 con 0x7fc728153ca0 2024-01-15T18:07:28.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.912+0000 7fc71fe00640 1 -- 172.21.15.27:0/1240770979 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc720013630 con 0x7fc728153ca0 2024-01-15T18:07:28.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.912+0000 7fc72f000640 1 -- 172.21.15.27:0/1240770979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7280ce6e0 con 0x7fc728153ca0 2024-01-15T18:07:28.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.916+0000 7fc72f000640 1 -- 172.21.15.27:0/1240770979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc72806d990 con 0x7fc728153ca0 2024-01-15T18:07:28.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.916+0000 7fc71fe00640 1 -- 172.21.15.27:0/1240770979 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7fc720013790 con 0x7fc728153ca0 2024-01-15T18:07:28.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.916+0000 7fc71fe00640 1 --2- 172.21.15.27:0/1240770979 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc700066b30 0x7fc700068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:28.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.916+0000 7fc71fe00640 1 -- 172.21.15.27:0/1240770979 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc720084e90 con 0x7fc728153ca0 2024-01-15T18:07:28.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.916+0000 7fc727e00640 1 --2- 172.21.15.27:0/1240770979 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc700066b30 0x7fc700068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:28.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.916+0000 7fc727e00640 1 --2- 172.21.15.27:0/1240770979 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc700066b30 0x7fc700068ff0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fc718009b60 tx=0x7fc718009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:28.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:28.920+0000 7fc71fe00640 1 -- 172.21.15.27:0/1240770979 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc720051d50 con 0x7fc728153ca0 2024-01-15T18:07:29.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.072+0000 7fc72f000640 1 -- 172.21.15.27:0/1240770979 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc7280ce8d0 con 0x7fc700066b30 2024-01-15T18:07:29.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.084+0000 7fc71fe00640 1 -- 172.21.15.27:0/1240770979 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288804 (secure 0 0 0) 0x7fc7280ce8d0 con 0x7fc700066b30 2024-01-15T18:07:29.089 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:07:29.091 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:07:29.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.084+0000 7fc72f000640 1 -- 172.21.15.27:0/1240770979 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc700066b30 msgr2=0x7fc700068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:29.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.084+0000 7fc72f000640 1 --2- 172.21.15.27:0/1240770979 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc700066b30 0x7fc700068ff0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fc718009b60 tx=0x7fc718009190 comp rx=0 tx=0).stop 2024-01-15T18:07:29.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.084+0000 7fc72f000640 1 -- 172.21.15.27:0/1240770979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc728153ca0 msgr2=0x7fc728154130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:29.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.084+0000 7fc72f000640 1 --2- 172.21.15.27:0/1240770979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc728153ca0 0x7fc728154130 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fc720002240 tx=0x7fc720007b10 comp rx=0 tx=0).stop 2024-01-15T18:07:29.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.084+0000 7fc72f000640 1 -- 172.21.15.27:0/1240770979 shutdown_connections 2024-01-15T18:07:29.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.084+0000 7fc72f000640 1 --2- 172.21.15.27:0/1240770979 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc700066b30 0x7fc700068ff0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.084+0000 7fc72f000640 1 --2- 172.21.15.27:0/1240770979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc728153ca0 0x7fc728154130 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.084+0000 7fc72f000640 1 --2- 172.21.15.27:0/1240770979 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7280cc830 0x7fc7280ce1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.084+0000 7fc72f000640 1 --2- 172.21.15.27:0/1240770979 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7080047d0 0x7fc7281512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.084+0000 7fc72f000640 1 -- 172.21.15.27:0/1240770979 >> 172.21.15.27:0/1240770979 conn(0x7fc7280cec60 msgr2=0x7fc728158120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:29.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.088+0000 7fc72f000640 1 -- 172.21.15.27:0/1240770979 shutdown_connections 2024-01-15T18:07:29.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.088+0000 7fc72f000640 1 -- 172.21.15.27:0/1240770979 wait complete. 2024-01-15T18:07:29.111 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-01-15T18:07:28.172559+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":188,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254756,"kb_used_data":5288,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754719964,"statfs":{"total":773094113280,"available":772833243136,"internally_reserved":0,"allocated":5414912,"data_stored":2881060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.106940"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.369702+0000","last_change":"2024-01-15T18:07:20.088581+0000","last_active":"2024-01-15T18:07:21.369702+0000","last_peered":"2024-01-15T18:07:21.369702+0000","last_clean":"2024-01-15T18:07:21.369702+0000","last_became_active":"2024-01-15T18:07:20.088247+0000","last_became_peered":"2024-01-15T18:07:20.088247+0000","last_unstale":"2024-01-15T18:07:21.369702+0000","last_undegraded":"2024-01-15T18:07:21.369702+0000","last_fullsized":"2024-01-15T18:07:21.369702+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:52:29.947335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.370008+0000","last_change":"2024-01-15T18:07:20.099853+0000","last_active":"2024-01-15T18:07:21.370008+0000","last_peered":"2024-01-15T18:07:21.370008+0000","last_clean":"2024-01-15T18:07:21.370008+0000","last_became_active":"2024-01-15T18:07:20.099509+0000","last_became_peered":"2024-01-15T18:07:20.099509+0000","last_unstale":"2024-01-15T18:07:21.370008+0000","last_undegraded":"2024-01-15T18:07:21.370008+0000","last_fullsized":"2024-01-15T18:07:21.370008+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:29:49.488018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.083430+0000","last_change":"2024-01-15T18:07:20.094383+0000","last_active":"2024-01-15T18:07:21.083430+0000","last_peered":"2024-01-15T18:07:21.083430+0000","last_clean":"2024-01-15T18:07:21.083430+0000","last_became_active":"2024-01-15T18:07:20.094114+0000","last_became_peered":"2024-01-15T18:07:20.094114+0000","last_unstale":"2024-01-15T18:07:21.083430+0000","last_undegraded":"2024-01-15T18:07:21.083430+0000","last_fullsized":"2024-01-15T18:07:21.083430+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:50:12.413904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.369217+0000","last_change":"2024-01-15T18:07:20.103700+0000","last_active":"2024-01-15T18:07:21.369217+0000","last_peered":"2024-01-15T18:07:21.369217+0000","last_clean":"2024-01-15T18:07:21.369217+0000","last_became_active":"2024-01-15T18:07:20.102908+0000","last_became_peered":"2024-01-15T18:07:20.102908+0000","last_unstale":"2024-01-15T18:07:21.369217+0000","last_undegraded":"2024-01-15T18:07:21.369217+0000","last_fullsized":"2024-01-15T18:07:21.369217+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:07:19.733185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.369482+0000","last_change":"2024-01-15T18:07:20.096341+0000","last_active":"2024-01-15T18:07:21.369482+0000","last_peered":"2024-01-15T18:07:21.369482+0000","last_clean":"2024-01-15T18:07:21.369482+0000","last_became_active":"2024-01-15T18:07:20.095909+0000","last_became_peered":"2024-01-15T18:07:20.095909+0000","last_unstale":"2024-01-15T18:07:21.369482+0000","last_undegraded":"2024-01-15T18:07:21.369482+0000","last_fullsized":"2024-01-15T18:07:21.369482+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:52:17.415198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.368835+0000","last_change":"2024-01-15T18:07:20.090463+0000","last_active":"2024-01-15T18:07:21.368835+0000","last_peered":"2024-01-15T18:07:21.368835+0000","last_clean":"2024-01-15T18:07:21.368835+0000","last_became_active":"2024-01-15T18:07:20.090196+0000","last_became_peered":"2024-01-15T18:07:20.090196+0000","last_unstale":"2024-01-15T18:07:21.368835+0000","last_undegraded":"2024-01-15T18:07:21.368835+0000","last_fullsized":"2024-01-15T18:07:21.368835+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:52:29.947335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.369250+0000","last_change":"2024-01-15T18:07:20.102595+0000","last_active":"2024-01-15T18:07:21.369250+0000","last_peered":"2024-01-15T18:07:21.369250+0000","last_clean":"2024-01-15T18:07:21.369250+0000","last_became_active":"2024-01-15T18:07:20.102222+0000","last_became_peered":"2024-01-15T18:07:20.102222+0000","last_unstale":"2024-01-15T18:07:21.369250+0000","last_undegraded":"2024-01-15T18:07:21.369250+0000","last_fullsized":"2024-01-15T18:07:21.369250+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:07:50.353957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.369089+0000","last_change":"2024-01-15T18:07:20.097584+0000","last_active":"2024-01-15T18:07:21.369089+0000","last_peered":"2024-01-15T18:07:21.369089+0000","last_clean":"2024-01-15T18:07:21.369089+0000","last_became_active":"2024-01-15T18:07:20.097130+0000","last_became_peered":"2024-01-15T18:07:20.097130+0000","last_unstale":"2024-01-15T18:07:21.369089+0000","last_undegraded":"2024-01-15T18:07:21.369089+0000","last_fullsized":"2024-01-15T18:07:21.369089+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:34:08.308262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.370191+0000","last_change":"2024-01-15T18:07:20.103245+0000","last_active":"2024-01-15T18:07:21.370191+0000","last_peered":"2024-01-15T18:07:21.370191+0000","last_clean":"2024-01-15T18:07:21.370191+0000","last_became_active":"2024-01-15T18:07:20.102939+0000","last_became_peered":"2024-01-15T18:07:20.102939+0000","last_unstale":"2024-01-15T18:07:21.370191+0000","last_undegraded":"2024-01-15T18:07:21.370191+0000","last_fullsized":"2024-01-15T18:07:21.370191+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:25:11.028288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.370226+0000","last_change":"2024-01-15T18:07:20.103767+0000","last_active":"2024-01-15T18:07:21.370226+0000","last_peered":"2024-01-15T18:07:21.370226+0000","last_clean":"2024-01-15T18:07:21.370226+0000","last_became_active":"2024-01-15T18:07:20.103489+0000","last_became_peered":"2024-01-15T18:07:20.103489+0000","last_unstale":"2024-01-15T18:07:21.370226+0000","last_undegraded":"2024-01-15T18:07:21.370226+0000","last_fullsized":"2024-01-15T18:07:21.370226+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:41:43.179569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.370069+0000","last_change":"2024-01-15T18:07:20.103477+0000","last_active":"2024-01-15T18:07:21.370069+0000","last_peered":"2024-01-15T18:07:21.370069+0000","last_clean":"2024-01-15T18:07:21.370069+0000","last_became_active":"2024-01-15T18:07:20.103191+0000","last_became_peered":"2024-01-15T18:07:20.103191+0000","last_unstale":"2024-01-15T18:07:21.370069+0000","last_undegraded":"2024-01-15T18:07:21.370069+0000","last_fullsized":"2024-01-15T18:07:21.370069+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:04:22.318161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.369192+0000","last_change":"2024-01-15T18:07:20.105062+0000","last_active":"2024-01-15T18:07:21.369192+0000","last_peered":"2024-01-15T18:07:21.369192+0000","last_clean":"2024-01-15T18:07:21.369192+0000","last_became_active":"2024-01-15T18:07:20.104776+0000","last_became_peered":"2024-01-15T18:07:20.104776+0000","last_unstale":"2024-01-15T18:07:21.369192+0000","last_undegraded":"2024-01-15T18:07:21.369192+0000","last_fullsized":"2024-01-15T18:07:21.369192+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:39:06.434931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.082474+0000","last_change":"2024-01-15T18:07:20.093608+0000","last_active":"2024-01-15T18:07:21.082474+0000","last_peered":"2024-01-15T18:07:21.082474+0000","last_clean":"2024-01-15T18:07:21.082474+0000","last_became_active":"2024-01-15T18:07:20.091735+0000","last_became_peered":"2024-01-15T18:07:20.091735+0000","last_unstale":"2024-01-15T18:07:21.082474+0000","last_undegraded":"2024-01-15T18:07:21.082474+0000","last_fullsized":"2024-01-15T18:07:21.082474+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:30:43.231075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.369697+0000","last_change":"2024-01-15T18:07:20.102709+0000","last_active":"2024-01-15T18:07:21.369697+0000","last_peered":"2024-01-15T18:07:21.369697+0000","last_clean":"2024-01-15T18:07:21.369697+0000","last_became_active":"2024-01-15T18:07:20.102268+0000","last_became_peered":"2024-01-15T18:07:20.102268+0000","last_unstale":"2024-01-15T18:07:21.369697+0000","last_undegraded":"2024-01-15T18:07:21.369697+0000","last_fullsized":"2024-01-15T18:07:21.369697+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:31:53.030232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.368779+0000","last_change":"2024-01-15T18:07:20.096359+0000","last_active":"2024-01-15T18:07:21.368779+0000","last_peered":"2024-01-15T18:07:21.368779+0000","last_clean":"2024-01-15T18:07:21.368779+0000","last_became_active":"2024-01-15T18:07:20.096018+0000","last_became_peered":"2024-01-15T18:07:20.096018+0000","last_unstale":"2024-01-15T18:07:21.368779+0000","last_undegraded":"2024-01-15T18:07:21.368779+0000","last_fullsized":"2024-01-15T18:07:21.368779+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:02:26.257091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.084264+0000","last_change":"2024-01-15T18:07:20.097218+0000","last_active":"2024-01-15T18:07:21.084264+0000","last_peered":"2024-01-15T18:07:21.084264+0000","last_clean":"2024-01-15T18:07:21.084264+0000","last_became_active":"2024-01-15T18:07:20.096805+0000","last_became_peered":"2024-01-15T18:07:20.096805+0000","last_unstale":"2024-01-15T18:07:21.084264+0000","last_undegraded":"2024-01-15T18:07:21.084264+0000","last_fullsized":"2024-01-15T18:07:21.084264+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:12:43.126962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.084009+0000","last_change":"2024-01-15T18:07:20.084911+0000","last_active":"2024-01-15T18:07:21.084009+0000","last_peered":"2024-01-15T18:07:21.084009+0000","last_clean":"2024-01-15T18:07:21.084009+0000","last_became_active":"2024-01-15T18:07:20.084594+0000","last_became_peered":"2024-01-15T18:07:20.084594+0000","last_unstale":"2024-01-15T18:07:21.084009+0000","last_undegraded":"2024-01-15T18:07:21.084009+0000","last_fullsized":"2024-01-15T18:07:21.084009+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:13:25.554937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.083693+0000","last_change":"2024-01-15T18:07:20.097350+0000","last_active":"2024-01-15T18:07:21.083693+0000","last_peered":"2024-01-15T18:07:21.083693+0000","last_clean":"2024-01-15T18:07:21.083693+0000","last_became_active":"2024-01-15T18:07:20.095556+0000","last_became_peered":"2024-01-15T18:07:20.095556+0000","last_unstale":"2024-01-15T18:07:21.083693+0000","last_undegraded":"2024-01-15T18:07:21.083693+0000","last_fullsized":"2024-01-15T18:07:21.083693+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:03:21.190309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.085072+0000","last_change":"2024-01-15T18:07:20.098939+0000","last_active":"2024-01-15T18:07:21.085072+0000","last_peered":"2024-01-15T18:07:21.085072+0000","last_clean":"2024-01-15T18:07:21.085072+0000","last_became_active":"2024-01-15T18:07:20.098598+0000","last_became_peered":"2024-01-15T18:07:20.098598+0000","last_unstale":"2024-01-15T18:07:21.085072+0000","last_undegraded":"2024-01-15T18:07:21.085072+0000","last_fullsized":"2024-01-15T18:07:21.085072+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:55:26.194143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.082669+0000","last_change":"2024-01-15T18:07:20.095741+0000","last_active":"2024-01-15T18:07:21.082669+0000","last_peered":"2024-01-15T18:07:21.082669+0000","last_clean":"2024-01-15T18:07:21.082669+0000","last_became_active":"2024-01-15T18:07:20.095417+0000","last_became_peered":"2024-01-15T18:07:20.095417+0000","last_unstale":"2024-01-15T18:07:21.082669+0000","last_undegraded":"2024-01-15T18:07:21.082669+0000","last_fullsized":"2024-01-15T18:07:21.082669+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:36:51.036352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.369764+0000","last_change":"2024-01-15T18:07:20.094321+0000","last_active":"2024-01-15T18:07:21.369764+0000","last_peered":"2024-01-15T18:07:21.369764+0000","last_clean":"2024-01-15T18:07:21.369764+0000","last_became_active":"2024-01-15T18:07:20.094011+0000","last_became_peered":"2024-01-15T18:07:20.094011+0000","last_unstale":"2024-01-15T18:07:21.369764+0000","last_undegraded":"2024-01-15T18:07:21.369764+0000","last_fullsized":"2024-01-15T18:07:21.369764+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:11:12.404313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.369813+0000","last_change":"2024-01-15T18:07:20.095694+0000","last_active":"2024-01-15T18:07:21.369813+0000","last_peered":"2024-01-15T18:07:21.369813+0000","last_clean":"2024-01-15T18:07:21.369813+0000","last_became_active":"2024-01-15T18:07:20.095406+0000","last_became_peered":"2024-01-15T18:07:20.095406+0000","last_unstale":"2024-01-15T18:07:21.369813+0000","last_undegraded":"2024-01-15T18:07:21.369813+0000","last_fullsized":"2024-01-15T18:07:21.369813+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:49:07.332523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.085010+0000","last_change":"2024-01-15T18:07:20.087728+0000","last_active":"2024-01-15T18:07:21.085010+0000","last_peered":"2024-01-15T18:07:21.085010+0000","last_clean":"2024-01-15T18:07:21.085010+0000","last_became_active":"2024-01-15T18:07:20.087118+0000","last_became_peered":"2024-01-15T18:07:20.087118+0000","last_unstale":"2024-01-15T18:07:21.085010+0000","last_undegraded":"2024-01-15T18:07:21.085010+0000","last_fullsized":"2024-01-15T18:07:21.085010+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:20:04.640555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.084354+0000","last_change":"2024-01-15T18:07:20.092361+0000","last_active":"2024-01-15T18:07:21.084354+0000","last_peered":"2024-01-15T18:07:21.084354+0000","last_clean":"2024-01-15T18:07:21.084354+0000","last_became_active":"2024-01-15T18:07:20.091927+0000","last_became_peered":"2024-01-15T18:07:20.091927+0000","last_unstale":"2024-01-15T18:07:21.084354+0000","last_undegraded":"2024-01-15T18:07:21.084354+0000","last_fullsized":"2024-01-15T18:07:21.084354+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:42:37.049164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.085144+0000","last_change":"2024-01-15T18:07:20.102013+0000","last_active":"2024-01-15T18:07:21.085144+0000","last_peered":"2024-01-15T18:07:21.085144+0000","last_clean":"2024-01-15T18:07:21.085144+0000","last_became_active":"2024-01-15T18:07:20.101697+0000","last_became_peered":"2024-01-15T18:07:20.101697+0000","last_unstale":"2024-01-15T18:07:21.085144+0000","last_undegraded":"2024-01-15T18:07:21.085144+0000","last_fullsized":"2024-01-15T18:07:21.085144+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:01:01.613225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.082536+0000","last_change":"2024-01-15T18:07:20.086187+0000","last_active":"2024-01-15T18:07:21.082536+0000","last_peered":"2024-01-15T18:07:21.082536+0000","last_clean":"2024-01-15T18:07:21.082536+0000","last_became_active":"2024-01-15T18:07:20.085891+0000","last_became_peered":"2024-01-15T18:07:20.085891+0000","last_unstale":"2024-01-15T18:07:21.082536+0000","last_undegraded":"2024-01-15T18:07:21.082536+0000","last_fullsized":"2024-01-15T18:07:21.082536+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:59:01.979409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.083631+0000","last_change":"2024-01-15T18:07:20.086602+0000","last_active":"2024-01-15T18:07:21.083631+0000","last_peered":"2024-01-15T18:07:21.083631+0000","last_clean":"2024-01-15T18:07:21.083631+0000","last_became_active":"2024-01-15T18:07:20.086238+0000","last_became_peered":"2024-01-15T18:07:20.086238+0000","last_unstale":"2024-01-15T18:07:21.083631+0000","last_undegraded":"2024-01-15T18:07:21.083631+0000","last_fullsized":"2024-01-15T18:07:21.083631+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:51:04.157158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.369637+0000","last_change":"2024-01-15T18:07:20.102679+0000","last_active":"2024-01-15T18:07:21.369637+0000","last_peered":"2024-01-15T18:07:21.369637+0000","last_clean":"2024-01-15T18:07:21.369637+0000","last_became_active":"2024-01-15T18:07:20.102377+0000","last_became_peered":"2024-01-15T18:07:20.102377+0000","last_unstale":"2024-01-15T18:07:21.369637+0000","last_undegraded":"2024-01-15T18:07:21.369637+0000","last_fullsized":"2024-01-15T18:07:21.369637+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:12:07.169579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.085484+0000","last_change":"2024-01-15T18:07:20.097835+0000","last_active":"2024-01-15T18:07:21.085484+0000","last_peered":"2024-01-15T18:07:21.085484+0000","last_clean":"2024-01-15T18:07:21.085484+0000","last_became_active":"2024-01-15T18:07:20.097477+0000","last_became_peered":"2024-01-15T18:07:20.097477+0000","last_unstale":"2024-01-15T18:07:21.085484+0000","last_undegraded":"2024-01-15T18:07:21.085484+0000","last_fullsized":"2024-01-15T18:07:21.085484+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:38:53.126797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.083332+0000","last_change":"2024-01-15T18:07:20.083756+0000","last_active":"2024-01-15T18:07:21.083332+0000","last_peered":"2024-01-15T18:07:21.083332+0000","last_clean":"2024-01-15T18:07:21.083332+0000","last_became_active":"2024-01-15T18:07:20.079072+0000","last_became_peered":"2024-01-15T18:07:20.079072+0000","last_unstale":"2024-01-15T18:07:21.083332+0000","last_undegraded":"2024-01-15T18:07:21.083332+0000","last_fullsized":"2024-01-15T18:07:21.083332+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:24:48.277513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.368840+0000","last_change":"2024-01-15T18:07:20.096208+0000","last_active":"2024-01-15T18:07:21.368840+0000","last_peered":"2024-01-15T18:07:21.368840+0000","last_clean":"2024-01-15T18:07:21.368840+0000","last_became_active":"2024-01-15T18:07:20.095900+0000","last_became_peered":"2024-01-15T18:07:20.095900+0000","last_unstale":"2024-01-15T18:07:21.368840+0000","last_undegraded":"2024-01-15T18:07:21.368840+0000","last_fullsized":"2024-01-15T18:07:21.368840+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:20:34.187795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.082595+0000","last_change":"2024-01-15T18:07:20.091218+0000","last_active":"2024-01-15T18:07:21.082595+0000","last_peered":"2024-01-15T18:07:21.082595+0000","last_clean":"2024-01-15T18:07:21.082595+0000","last_became_active":"2024-01-15T18:07:20.090896+0000","last_became_peered":"2024-01-15T18:07:20.090896+0000","last_unstale":"2024-01-15T18:07:21.082595+0000","last_undegraded":"2024-01-15T18:07:21.082595+0000","last_fullsized":"2024-01-15T18:07:21.082595+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:32:14.924018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.369026+0000","last_change":"2024-01-15T18:07:20.087031+0000","last_active":"2024-01-15T18:07:21.369026+0000","last_peered":"2024-01-15T18:07:21.369026+0000","last_clean":"2024-01-15T18:07:21.369026+0000","last_became_active":"2024-01-15T18:07:20.086158+0000","last_became_peered":"2024-01-15T18:07:20.086158+0000","last_unstale":"2024-01-15T18:07:21.369026+0000","last_undegraded":"2024-01-15T18:07:21.369026+0000","last_fullsized":"2024-01-15T18:07:21.369026+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:21.022650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.193777+0000","last_change":"2024-01-15T18:07:19.085120+0000","last_active":"2024-01-15T18:07:20.193777+0000","last_peered":"2024-01-15T18:07:20.193777+0000","last_clean":"2024-01-15T18:07:20.193777+0000","last_became_active":"2024-01-15T18:07:19.084785+0000","last_became_peered":"2024-01-15T18:07:19.084785+0000","last_unstale":"2024-01-15T18:07:20.193777+0000","last_undegraded":"2024-01-15T18:07:20.193777+0000","last_fullsized":"2024-01-15T18:07:20.193777+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:50:51.007855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.085279+0000","last_change":"2024-01-15T18:07:20.094786+0000","last_active":"2024-01-15T18:07:21.085279+0000","last_peered":"2024-01-15T18:07:21.085279+0000","last_clean":"2024-01-15T18:07:21.085279+0000","last_became_active":"2024-01-15T18:07:20.094523+0000","last_became_peered":"2024-01-15T18:07:20.094523+0000","last_unstale":"2024-01-15T18:07:21.085279+0000","last_undegraded":"2024-01-15T18:07:21.085279+0000","last_fullsized":"2024-01-15T18:07:21.085279+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:57:57.266821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.188166+0000","last_change":"2024-01-15T18:07:19.084713+0000","last_active":"2024-01-15T18:07:20.188166+0000","last_peered":"2024-01-15T18:07:20.188166+0000","last_clean":"2024-01-15T18:07:20.188166+0000","last_became_active":"2024-01-15T18:07:19.084414+0000","last_became_peered":"2024-01-15T18:07:19.084414+0000","last_unstale":"2024-01-15T18:07:20.188166+0000","last_undegraded":"2024-01-15T18:07:20.188166+0000","last_fullsized":"2024-01-15T18:07:20.188166+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:20.019967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.369633+0000","last_change":"2024-01-15T18:07:20.090063+0000","last_active":"2024-01-15T18:07:21.369633+0000","last_peered":"2024-01-15T18:07:21.369633+0000","last_clean":"2024-01-15T18:07:21.369633+0000","last_became_active":"2024-01-15T18:07:20.089742+0000","last_became_peered":"2024-01-15T18:07:20.089742+0000","last_unstale":"2024-01-15T18:07:21.369633+0000","last_undegraded":"2024-01-15T18:07:21.369633+0000","last_fullsized":"2024-01-15T18:07:21.369633+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:29:57.598440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.185828+0000","last_change":"2024-01-15T18:07:19.077287+0000","last_active":"2024-01-15T18:07:20.185828+0000","last_peered":"2024-01-15T18:07:20.185828+0000","last_clean":"2024-01-15T18:07:20.185828+0000","last_became_active":"2024-01-15T18:07:19.075652+0000","last_became_peered":"2024-01-15T18:07:19.075652+0000","last_unstale":"2024-01-15T18:07:20.185828+0000","last_undegraded":"2024-01-15T18:07:20.185828+0000","last_fullsized":"2024-01-15T18:07:20.185828+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:57:56.264139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.369369+0000","last_change":"2024-01-15T18:07:20.087874+0000","last_active":"2024-01-15T18:07:21.369369+0000","last_peered":"2024-01-15T18:07:21.369369+0000","last_clean":"2024-01-15T18:07:21.369369+0000","last_became_active":"2024-01-15T18:07:20.087579+0000","last_became_peered":"2024-01-15T18:07:20.087579+0000","last_unstale":"2024-01-15T18:07:21.369369+0000","last_undegraded":"2024-01-15T18:07:21.369369+0000","last_fullsized":"2024-01-15T18:07:21.369369+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:37:10.404180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"44'2","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.177476+0000","last_change":"2024-01-15T18:07:19.085001+0000","last_active":"2024-01-15T18:07:20.177476+0000","last_peered":"2024-01-15T18:07:20.177476+0000","last_clean":"2024-01-15T18:07:20.177476+0000","last_became_active":"2024-01-15T18:07:19.084651+0000","last_became_peered":"2024-01-15T18:07:19.084651+0000","last_unstale":"2024-01-15T18:07:20.177476+0000","last_undegraded":"2024-01-15T18:07:20.177476+0000","last_fullsized":"2024-01-15T18:07:20.177476+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:13:24.552254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.083337+0000","last_change":"2024-01-15T18:07:20.087693+0000","last_active":"2024-01-15T18:07:21.083337+0000","last_peered":"2024-01-15T18:07:21.083337+0000","last_clean":"2024-01-15T18:07:21.083337+0000","last_became_active":"2024-01-15T18:07:20.087416+0000","last_became_peered":"2024-01-15T18:07:20.087416+0000","last_unstale":"2024-01-15T18:07:21.083337+0000","last_undegraded":"2024-01-15T18:07:21.083337+0000","last_fullsized":"2024-01-15T18:07:21.083337+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:04:14.022333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.193723+0000","last_change":"2024-01-15T18:07:19.086152+0000","last_active":"2024-01-15T18:07:20.193723+0000","last_peered":"2024-01-15T18:07:20.193723+0000","last_clean":"2024-01-15T18:07:20.193723+0000","last_became_active":"2024-01-15T18:07:19.082058+0000","last_became_peered":"2024-01-15T18:07:19.082058+0000","last_unstale":"2024-01-15T18:07:20.193723+0000","last_undegraded":"2024-01-15T18:07:20.193723+0000","last_fullsized":"2024-01-15T18:07:20.193723+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:06:20.719015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.369429+0000","last_change":"2024-01-15T18:07:20.106087+0000","last_active":"2024-01-15T18:07:21.369429+0000","last_peered":"2024-01-15T18:07:21.369429+0000","last_clean":"2024-01-15T18:07:21.369429+0000","last_became_active":"2024-01-15T18:07:20.105780+0000","last_became_peered":"2024-01-15T18:07:20.105780+0000","last_unstale":"2024-01-15T18:07:21.369429+0000","last_undegraded":"2024-01-15T18:07:21.369429+0000","last_fullsized":"2024-01-15T18:07:21.369429+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:30:18.078564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.188073+0000","last_change":"2024-01-15T18:07:19.076932+0000","last_active":"2024-01-15T18:07:20.188073+0000","last_peered":"2024-01-15T18:07:20.188073+0000","last_clean":"2024-01-15T18:07:20.188073+0000","last_became_active":"2024-01-15T18:07:19.076629+0000","last_became_peered":"2024-01-15T18:07:19.076629+0000","last_unstale":"2024-01-15T18:07:20.188073+0000","last_undegraded":"2024-01-15T18:07:20.188073+0000","last_fullsized":"2024-01-15T18:07:20.188073+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:24:59.066289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.369228+0000","last_change":"2024-01-15T18:07:20.087712+0000","last_active":"2024-01-15T18:07:21.369228+0000","last_peered":"2024-01-15T18:07:21.369228+0000","last_clean":"2024-01-15T18:07:21.369228+0000","last_became_active":"2024-01-15T18:07:20.087463+0000","last_became_peered":"2024-01-15T18:07:20.087463+0000","last_unstale":"2024-01-15T18:07:21.369228+0000","last_undegraded":"2024-01-15T18:07:21.369228+0000","last_fullsized":"2024-01-15T18:07:21.369228+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:06:12.518951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.071777+0000","last_change":"2024-01-15T18:07:19.079711+0000","last_active":"2024-01-15T18:07:20.071777+0000","last_peered":"2024-01-15T18:07:20.071777+0000","last_clean":"2024-01-15T18:07:20.071777+0000","last_became_active":"2024-01-15T18:07:19.079336+0000","last_became_peered":"2024-01-15T18:07:19.079336+0000","last_unstale":"2024-01-15T18:07:20.071777+0000","last_undegraded":"2024-01-15T18:07:20.071777+0000","last_fullsized":"2024-01-15T18:07:20.071777+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:04:21.315479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.084837+0000","last_change":"2024-01-15T18:07:20.087900+0000","last_active":"2024-01-15T18:07:21.084837+0000","last_peered":"2024-01-15T18:07:21.084837+0000","last_clean":"2024-01-15T18:07:21.084837+0000","last_became_active":"2024-01-15T18:07:20.087463+0000","last_became_peered":"2024-01-15T18:07:20.087463+0000","last_unstale":"2024-01-15T18:07:21.084837+0000","last_undegraded":"2024-01-15T18:07:21.084837+0000","last_fullsized":"2024-01-15T18:07:21.084837+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:07:17.293929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.080669+0000","last_change":"2024-01-15T18:07:19.079615+0000","last_active":"2024-01-15T18:07:20.080669+0000","last_peered":"2024-01-15T18:07:20.080669+0000","last_clean":"2024-01-15T18:07:20.080669+0000","last_became_active":"2024-01-15T18:07:19.079242+0000","last_became_peered":"2024-01-15T18:07:19.079242+0000","last_unstale":"2024-01-15T18:07:20.080669+0000","last_undegraded":"2024-01-15T18:07:20.080669+0000","last_fullsized":"2024-01-15T18:07:20.080669+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:17:57.220699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.083762+0000","last_change":"2024-01-15T18:07:20.095844+0000","last_active":"2024-01-15T18:07:21.083762+0000","last_peered":"2024-01-15T18:07:21.083762+0000","last_clean":"2024-01-15T18:07:21.083762+0000","last_became_active":"2024-01-15T18:07:20.095110+0000","last_became_peered":"2024-01-15T18:07:20.095110+0000","last_unstale":"2024-01-15T18:07:21.083762+0000","last_undegraded":"2024-01-15T18:07:21.083762+0000","last_fullsized":"2024-01-15T18:07:21.083762+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:35:28.043293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.188196+0000","last_change":"2024-01-15T18:07:19.076938+0000","last_active":"2024-01-15T18:07:20.188196+0000","last_peered":"2024-01-15T18:07:20.188196+0000","last_clean":"2024-01-15T18:07:20.188196+0000","last_became_active":"2024-01-15T18:07:19.076605+0000","last_became_peered":"2024-01-15T18:07:19.076605+0000","last_unstale":"2024-01-15T18:07:20.188196+0000","last_undegraded":"2024-01-15T18:07:20.188196+0000","last_fullsized":"2024-01-15T18:07:20.188196+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:32.749313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.369679+0000","last_change":"2024-01-15T18:07:20.091131+0000","last_active":"2024-01-15T18:07:21.369679+0000","last_peered":"2024-01-15T18:07:21.369679+0000","last_clean":"2024-01-15T18:07:21.369679+0000","last_became_active":"2024-01-15T18:07:20.090683+0000","last_became_peered":"2024-01-15T18:07:20.090683+0000","last_unstale":"2024-01-15T18:07:21.369679+0000","last_undegraded":"2024-01-15T18:07:21.369679+0000","last_fullsized":"2024-01-15T18:07:21.369679+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:02:26.257091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.086420+0000","last_change":"2024-01-15T18:07:19.085014+0000","last_active":"2024-01-15T18:07:20.086420+0000","last_peered":"2024-01-15T18:07:20.086420+0000","last_clean":"2024-01-15T18:07:20.086420+0000","last_became_active":"2024-01-15T18:07:19.084470+0000","last_became_peered":"2024-01-15T18:07:19.084470+0000","last_unstale":"2024-01-15T18:07:20.086420+0000","last_undegraded":"2024-01-15T18:07:20.086420+0000","last_fullsized":"2024-01-15T18:07:20.086420+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:10:51.637250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.370018+0000","last_change":"2024-01-15T18:07:20.108821+0000","last_active":"2024-01-15T18:07:21.370018+0000","last_peered":"2024-01-15T18:07:21.370018+0000","last_clean":"2024-01-15T18:07:21.370018+0000","last_became_active":"2024-01-15T18:07:20.108464+0000","last_became_peered":"2024-01-15T18:07:20.108464+0000","last_unstale":"2024-01-15T18:07:21.370018+0000","last_undegraded":"2024-01-15T18:07:21.370018+0000","last_fullsized":"2024-01-15T18:07:21.370018+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:08:14.419559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"44'3","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.177529+0000","last_change":"2024-01-15T18:07:19.088202+0000","last_active":"2024-01-15T18:07:20.177529+0000","last_peered":"2024-01-15T18:07:20.177529+0000","last_clean":"2024-01-15T18:07:20.177529+0000","last_became_active":"2024-01-15T18:07:19.087900+0000","last_became_peered":"2024-01-15T18:07:19.087900+0000","last_unstale":"2024-01-15T18:07:20.177529+0000","last_undegraded":"2024-01-15T18:07:20.177529+0000","last_fullsized":"2024-01-15T18:07:20.177529+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:10:42.345570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.369518+0000","last_change":"2024-01-15T18:07:20.102897+0000","last_active":"2024-01-15T18:07:21.369518+0000","last_peered":"2024-01-15T18:07:21.369518+0000","last_clean":"2024-01-15T18:07:21.369518+0000","last_became_active":"2024-01-15T18:07:20.102579+0000","last_became_peered":"2024-01-15T18:07:20.102579+0000","last_unstale":"2024-01-15T18:07:21.369518+0000","last_undegraded":"2024-01-15T18:07:21.369518+0000","last_fullsized":"2024-01-15T18:07:21.369518+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:50:52.010538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":18,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.180051+0000","last_change":"2024-01-15T18:07:19.079571+0000","last_active":"2024-01-15T18:07:20.180051+0000","last_peered":"2024-01-15T18:07:20.180051+0000","last_clean":"2024-01-15T18:07:20.180051+0000","last_became_active":"2024-01-15T18:07:19.079308+0000","last_became_peered":"2024-01-15T18:07:19.079308+0000","last_unstale":"2024-01-15T18:07:20.180051+0000","last_undegraded":"2024-01-15T18:07:20.180051+0000","last_fullsized":"2024-01-15T18:07:20.180051+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:25.046543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.084566+0000","last_change":"2024-01-15T18:07:20.097611+0000","last_active":"2024-01-15T18:07:21.084566+0000","last_peered":"2024-01-15T18:07:21.084566+0000","last_clean":"2024-01-15T18:07:21.084566+0000","last_became_active":"2024-01-15T18:07:20.097202+0000","last_became_peered":"2024-01-15T18:07:20.097202+0000","last_unstale":"2024-01-15T18:07:21.084566+0000","last_undegraded":"2024-01-15T18:07:21.084566+0000","last_fullsized":"2024-01-15T18:07:21.084566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:51:49.339183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.193733+0000","last_change":"2024-01-15T18:07:19.084662+0000","last_active":"2024-01-15T18:07:20.193733+0000","last_peered":"2024-01-15T18:07:20.193733+0000","last_clean":"2024-01-15T18:07:20.193733+0000","last_became_active":"2024-01-15T18:07:19.084348+0000","last_became_peered":"2024-01-15T18:07:19.084348+0000","last_unstale":"2024-01-15T18:07:20.193733+0000","last_undegraded":"2024-01-15T18:07:20.193733+0000","last_fullsized":"2024-01-15T18:07:20.193733+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:07:49.351274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.369450+0000","last_change":"2024-01-15T18:07:20.087611+0000","last_active":"2024-01-15T18:07:21.369450+0000","last_peered":"2024-01-15T18:07:21.369450+0000","last_clean":"2024-01-15T18:07:21.369450+0000","last_became_active":"2024-01-15T18:07:20.087328+0000","last_became_peered":"2024-01-15T18:07:20.087328+0000","last_unstale":"2024-01-15T18:07:21.369450+0000","last_undegraded":"2024-01-15T18:07:21.369450+0000","last_fullsized":"2024-01-15T18:07:21.369450+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:56:51.167704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.177441+0000","last_change":"2024-01-15T18:07:19.084512+0000","last_active":"2024-01-15T18:07:20.177441+0000","last_peered":"2024-01-15T18:07:20.177441+0000","last_clean":"2024-01-15T18:07:20.177441+0000","last_became_active":"2024-01-15T18:07:19.084169+0000","last_became_peered":"2024-01-15T18:07:19.084169+0000","last_unstale":"2024-01-15T18:07:20.177441+0000","last_undegraded":"2024-01-15T18:07:20.177441+0000","last_fullsized":"2024-01-15T18:07:20.177441+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:35:27.040611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.368754+0000","last_change":"2024-01-15T18:07:20.105484+0000","last_active":"2024-01-15T18:07:21.368754+0000","last_peered":"2024-01-15T18:07:21.368754+0000","last_clean":"2024-01-15T18:07:21.368754+0000","last_became_active":"2024-01-15T18:07:20.105160+0000","last_became_peered":"2024-01-15T18:07:20.105160+0000","last_unstale":"2024-01-15T18:07:21.368754+0000","last_undegraded":"2024-01-15T18:07:21.368754+0000","last_fullsized":"2024-01-15T18:07:21.368754+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:07:48.597563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.193672+0000","last_change":"2024-01-15T18:07:19.083510+0000","last_active":"2024-01-15T18:07:20.193672+0000","last_peered":"2024-01-15T18:07:20.193672+0000","last_clean":"2024-01-15T18:07:20.193672+0000","last_became_active":"2024-01-15T18:07:19.083134+0000","last_became_peered":"2024-01-15T18:07:19.083134+0000","last_unstale":"2024-01-15T18:07:20.193672+0000","last_undegraded":"2024-01-15T18:07:20.193672+0000","last_fullsized":"2024-01-15T18:07:20.193672+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:10:42.345570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.369317+0000","last_change":"2024-01-15T18:07:20.101397+0000","last_active":"2024-01-15T18:07:21.369317+0000","last_peered":"2024-01-15T18:07:21.369317+0000","last_clean":"2024-01-15T18:07:21.369317+0000","last_became_active":"2024-01-15T18:07:20.101106+0000","last_became_peered":"2024-01-15T18:07:20.101106+0000","last_unstale":"2024-01-15T18:07:21.369317+0000","last_undegraded":"2024-01-15T18:07:21.369317+0000","last_fullsized":"2024-01-15T18:07:21.369317+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:23:42.481421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"44'8","reported_seq":20,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.107111+0000","last_change":"2024-01-15T18:07:19.086583+0000","last_active":"2024-01-15T18:07:20.107111+0000","last_peered":"2024-01-15T18:07:20.107111+0000","last_clean":"2024-01-15T18:07:20.107111+0000","last_became_active":"2024-01-15T18:07:19.086227+0000","last_became_peered":"2024-01-15T18:07:19.086227+0000","last_unstale":"2024-01-15T18:07:20.107111+0000","last_undegraded":"2024-01-15T18:07:20.107111+0000","last_fullsized":"2024-01-15T18:07:20.107111+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:24:33.463261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.082678+0000","last_change":"2024-01-15T18:07:20.075833+0000","last_active":"2024-01-15T18:07:21.082678+0000","last_peered":"2024-01-15T18:07:21.082678+0000","last_clean":"2024-01-15T18:07:21.082678+0000","last_became_active":"2024-01-15T18:07:20.075521+0000","last_became_peered":"2024-01-15T18:07:20.075521+0000","last_unstale":"2024-01-15T18:07:21.082678+0000","last_undegraded":"2024-01-15T18:07:21.082678+0000","last_fullsized":"2024-01-15T18:07:21.082678+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:12:43.126962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.180091+0000","last_change":"2024-01-15T18:07:19.087887+0000","last_active":"2024-01-15T18:07:20.180091+0000","last_peered":"2024-01-15T18:07:20.180091+0000","last_clean":"2024-01-15T18:07:20.180091+0000","last_became_active":"2024-01-15T18:07:19.087610+0000","last_became_peered":"2024-01-15T18:07:19.087610+0000","last_unstale":"2024-01-15T18:07:20.180091+0000","last_undegraded":"2024-01-15T18:07:20.180091+0000","last_fullsized":"2024-01-15T18:07:20.180091+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:29:39.445420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.370330+0000","last_change":"2024-01-15T18:07:20.107410+0000","last_active":"2024-01-15T18:07:21.370330+0000","last_peered":"2024-01-15T18:07:21.370330+0000","last_clean":"2024-01-15T18:07:21.370330+0000","last_became_active":"2024-01-15T18:07:20.106966+0000","last_became_peered":"2024-01-15T18:07:20.106966+0000","last_unstale":"2024-01-15T18:07:21.370330+0000","last_undegraded":"2024-01-15T18:07:21.370330+0000","last_fullsized":"2024-01-15T18:07:21.370330+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:39:06.434931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.188177+0000","last_change":"2024-01-15T18:07:19.074682+0000","last_active":"2024-01-15T18:07:20.188177+0000","last_peered":"2024-01-15T18:07:20.188177+0000","last_clean":"2024-01-15T18:07:20.188177+0000","last_became_active":"2024-01-15T18:07:19.074397+0000","last_became_peered":"2024-01-15T18:07:19.074397+0000","last_unstale":"2024-01-15T18:07:20.188177+0000","last_undegraded":"2024-01-15T18:07:20.188177+0000","last_fullsized":"2024-01-15T18:07:20.188177+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:37:09.401497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.084874+0000","last_change":"2024-01-15T18:07:20.093962+0000","last_active":"2024-01-15T18:07:21.084874+0000","last_peered":"2024-01-15T18:07:21.084874+0000","last_clean":"2024-01-15T18:07:21.084874+0000","last_became_active":"2024-01-15T18:07:20.093638+0000","last_became_peered":"2024-01-15T18:07:20.093638+0000","last_unstale":"2024-01-15T18:07:21.084874+0000","last_undegraded":"2024-01-15T18:07:21.084874+0000","last_fullsized":"2024-01-15T18:07:21.084874+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:02.825993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"44'1","reported_seq":13,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.101597+0000","last_change":"2024-01-15T18:07:19.082541+0000","last_active":"2024-01-15T18:07:20.101597+0000","last_peered":"2024-01-15T18:07:20.101597+0000","last_clean":"2024-01-15T18:07:20.101597+0000","last_became_active":"2024-01-15T18:07:19.082192+0000","last_became_peered":"2024-01-15T18:07:19.082192+0000","last_unstale":"2024-01-15T18:07:20.101597+0000","last_undegraded":"2024-01-15T18:07:20.101597+0000","last_fullsized":"2024-01-15T18:07:20.101597+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:25:10.025606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.084976+0000","last_change":"2024-01-15T18:07:20.096891+0000","last_active":"2024-01-15T18:07:21.084976+0000","last_peered":"2024-01-15T18:07:21.084976+0000","last_clean":"2024-01-15T18:07:21.084976+0000","last_became_active":"2024-01-15T18:07:20.096533+0000","last_became_peered":"2024-01-15T18:07:20.096533+0000","last_unstale":"2024-01-15T18:07:21.084976+0000","last_undegraded":"2024-01-15T18:07:21.084976+0000","last_fullsized":"2024-01-15T18:07:21.084976+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:08:56.987120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.193760+0000","last_change":"2024-01-15T18:07:19.080295+0000","last_active":"2024-01-15T18:07:20.193760+0000","last_peered":"2024-01-15T18:07:20.193760+0000","last_clean":"2024-01-15T18:07:20.193760+0000","last_became_active":"2024-01-15T18:07:19.080024+0000","last_became_peered":"2024-01-15T18:07:19.080024+0000","last_unstale":"2024-01-15T18:07:20.193760+0000","last_undegraded":"2024-01-15T18:07:20.193760+0000","last_fullsized":"2024-01-15T18:07:20.193760+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:36:48.732986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.369693+0000","last_change":"2024-01-15T18:07:20.105757+0000","last_active":"2024-01-15T18:07:21.369693+0000","last_peered":"2024-01-15T18:07:21.369693+0000","last_clean":"2024-01-15T18:07:21.369693+0000","last_became_active":"2024-01-15T18:07:20.105385+0000","last_became_peered":"2024-01-15T18:07:20.105385+0000","last_unstale":"2024-01-15T18:07:21.369693+0000","last_undegraded":"2024-01-15T18:07:21.369693+0000","last_fullsized":"2024-01-15T18:07:21.369693+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:50:33.400931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.193743+0000","last_change":"2024-01-15T18:07:19.082297+0000","last_active":"2024-01-15T18:07:20.193743+0000","last_peered":"2024-01-15T18:07:20.193743+0000","last_clean":"2024-01-15T18:07:20.193743+0000","last_became_active":"2024-01-15T18:07:19.081971+0000","last_became_peered":"2024-01-15T18:07:19.081971+0000","last_unstale":"2024-01-15T18:07:20.193743+0000","last_undegraded":"2024-01-15T18:07:20.193743+0000","last_fullsized":"2024-01-15T18:07:20.193743+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:23:41.478738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.368973+0000","last_change":"2024-01-15T18:07:20.091112+0000","last_active":"2024-01-15T18:07:21.368973+0000","last_peered":"2024-01-15T18:07:21.368973+0000","last_clean":"2024-01-15T18:07:21.368973+0000","last_became_active":"2024-01-15T18:07:20.090802+0000","last_became_peered":"2024-01-15T18:07:20.090802+0000","last_unstale":"2024-01-15T18:07:21.368973+0000","last_undegraded":"2024-01-15T18:07:21.368973+0000","last_fullsized":"2024-01-15T18:07:21.368973+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:07:58.065360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.071434+0000","last_change":"2024-01-15T18:07:19.077896+0000","last_active":"2024-01-15T18:07:20.071434+0000","last_peered":"2024-01-15T18:07:20.071434+0000","last_clean":"2024-01-15T18:07:20.071434+0000","last_became_active":"2024-01-15T18:07:19.077439+0000","last_became_peered":"2024-01-15T18:07:19.077439+0000","last_unstale":"2024-01-15T18:07:20.071434+0000","last_undegraded":"2024-01-15T18:07:20.071434+0000","last_fullsized":"2024-01-15T18:07:20.071434+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:14:26.111042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.082389+0000","last_change":"2024-01-15T18:07:20.086589+0000","last_active":"2024-01-15T18:07:21.082389+0000","last_peered":"2024-01-15T18:07:21.082389+0000","last_clean":"2024-01-15T18:07:21.082389+0000","last_became_active":"2024-01-15T18:07:20.086312+0000","last_became_peered":"2024-01-15T18:07:20.086312+0000","last_unstale":"2024-01-15T18:07:21.082389+0000","last_undegraded":"2024-01-15T18:07:21.082389+0000","last_fullsized":"2024-01-15T18:07:21.082389+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:55:19.615935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.083478+0000","last_change":"2024-01-15T18:07:19.082155+0000","last_active":"2024-01-15T18:07:20.083478+0000","last_peered":"2024-01-15T18:07:20.083478+0000","last_clean":"2024-01-15T18:07:20.083478+0000","last_became_active":"2024-01-15T18:07:19.081855+0000","last_became_peered":"2024-01-15T18:07:19.081855+0000","last_unstale":"2024-01-15T18:07:20.083478+0000","last_undegraded":"2024-01-15T18:07:20.083478+0000","last_fullsized":"2024-01-15T18:07:20.083478+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:09:06.512495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.369407+0000","last_change":"2024-01-15T18:07:20.105923+0000","last_active":"2024-01-15T18:07:21.369407+0000","last_peered":"2024-01-15T18:07:21.369407+0000","last_clean":"2024-01-15T18:07:21.369407+0000","last_became_active":"2024-01-15T18:07:20.105617+0000","last_became_peered":"2024-01-15T18:07:20.105617+0000","last_unstale":"2024-01-15T18:07:21.369407+0000","last_undegraded":"2024-01-15T18:07:21.369407+0000","last_fullsized":"2024-01-15T18:07:21.369407+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:53:04.063260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.177403+0000","last_change":"2024-01-15T18:07:19.085582+0000","last_active":"2024-01-15T18:07:20.177403+0000","last_peered":"2024-01-15T18:07:20.177403+0000","last_clean":"2024-01-15T18:07:20.177403+0000","last_became_active":"2024-01-15T18:07:19.083796+0000","last_became_peered":"2024-01-15T18:07:19.083796+0000","last_unstale":"2024-01-15T18:07:20.177403+0000","last_undegraded":"2024-01-15T18:07:20.177403+0000","last_fullsized":"2024-01-15T18:07:20.177403+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:03:20.187627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.085371+0000","last_change":"2024-01-15T18:07:20.100547+0000","last_active":"2024-01-15T18:07:21.085371+0000","last_peered":"2024-01-15T18:07:21.085371+0000","last_clean":"2024-01-15T18:07:21.085371+0000","last_became_active":"2024-01-15T18:07:20.100201+0000","last_became_peered":"2024-01-15T18:07:20.100201+0000","last_unstale":"2024-01-15T18:07:21.085371+0000","last_undegraded":"2024-01-15T18:07:21.085371+0000","last_fullsized":"2024-01-15T18:07:21.085371+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:00.068972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.193768+0000","last_change":"2024-01-15T18:07:19.078965+0000","last_active":"2024-01-15T18:07:20.193768+0000","last_peered":"2024-01-15T18:07:20.193768+0000","last_clean":"2024-01-15T18:07:20.193768+0000","last_became_active":"2024-01-15T18:07:19.078692+0000","last_became_peered":"2024-01-15T18:07:19.078692+0000","last_unstale":"2024-01-15T18:07:20.193768+0000","last_undegraded":"2024-01-15T18:07:20.193768+0000","last_fullsized":"2024-01-15T18:07:20.193768+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:13:24.552254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.369986+0000","last_change":"2024-01-15T18:07:20.094476+0000","last_active":"2024-01-15T18:07:21.369986+0000","last_peered":"2024-01-15T18:07:21.369986+0000","last_clean":"2024-01-15T18:07:21.369986+0000","last_became_active":"2024-01-15T18:07:20.090742+0000","last_became_peered":"2024-01-15T18:07:20.090742+0000","last_unstale":"2024-01-15T18:07:21.369986+0000","last_undegraded":"2024-01-15T18:07:21.369986+0000","last_fullsized":"2024-01-15T18:07:21.369986+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:14:27.113724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.177540+0000","last_change":"2024-01-15T18:07:19.088174+0000","last_active":"2024-01-15T18:07:20.177540+0000","last_peered":"2024-01-15T18:07:20.177540+0000","last_clean":"2024-01-15T18:07:20.177540+0000","last_became_active":"2024-01-15T18:07:19.087881+0000","last_became_peered":"2024-01-15T18:07:19.087881+0000","last_unstale":"2024-01-15T18:07:20.177540+0000","last_undegraded":"2024-01-15T18:07:20.177540+0000","last_fullsized":"2024-01-15T18:07:20.177540+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:36:48.732986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.369679+0000","last_change":"2024-01-15T18:07:20.104788+0000","last_active":"2024-01-15T18:07:21.369679+0000","last_peered":"2024-01-15T18:07:21.369679+0000","last_clean":"2024-01-15T18:07:21.369679+0000","last_became_active":"2024-01-15T18:07:20.104475+0000","last_became_peered":"2024-01-15T18:07:20.104475+0000","last_unstale":"2024-01-15T18:07:21.369679+0000","last_undegraded":"2024-01-15T18:07:21.369679+0000","last_fullsized":"2024-01-15T18:07:21.369679+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:07:48.597563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.185788+0000","last_change":"2024-01-15T18:07:19.080590+0000","last_active":"2024-01-15T18:07:20.185788+0000","last_peered":"2024-01-15T18:07:20.185788+0000","last_clean":"2024-01-15T18:07:20.185788+0000","last_became_active":"2024-01-15T18:07:19.080276+0000","last_became_peered":"2024-01-15T18:07:19.080276+0000","last_unstale":"2024-01-15T18:07:20.185788+0000","last_undegraded":"2024-01-15T18:07:20.185788+0000","last_fullsized":"2024-01-15T18:07:20.185788+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:24:59.066289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.369713+0000","last_change":"2024-01-15T18:07:20.090127+0000","last_active":"2024-01-15T18:07:21.369713+0000","last_peered":"2024-01-15T18:07:21.369713+0000","last_clean":"2024-01-15T18:07:21.369713+0000","last_became_active":"2024-01-15T18:07:20.089727+0000","last_became_peered":"2024-01-15T18:07:20.089727+0000","last_unstale":"2024-01-15T18:07:21.369713+0000","last_undegraded":"2024-01-15T18:07:21.369713+0000","last_fullsized":"2024-01-15T18:07:21.369713+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:46:12.510216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.177459+0000","last_change":"2024-01-15T18:07:19.088037+0000","last_active":"2024-01-15T18:07:20.177459+0000","last_peered":"2024-01-15T18:07:20.177459+0000","last_clean":"2024-01-15T18:07:20.177459+0000","last_became_active":"2024-01-15T18:07:19.087721+0000","last_became_peered":"2024-01-15T18:07:19.087721+0000","last_unstale":"2024-01-15T18:07:20.177459+0000","last_undegraded":"2024-01-15T18:07:20.177459+0000","last_fullsized":"2024-01-15T18:07:20.177459+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:23:27.606202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.369770+0000","last_change":"2024-01-15T18:07:20.105653+0000","last_active":"2024-01-15T18:07:21.369770+0000","last_peered":"2024-01-15T18:07:21.369770+0000","last_clean":"2024-01-15T18:07:21.369770+0000","last_became_active":"2024-01-15T18:07:20.105348+0000","last_became_peered":"2024-01-15T18:07:20.105348+0000","last_unstale":"2024-01-15T18:07:21.369770+0000","last_undegraded":"2024-01-15T18:07:21.369770+0000","last_fullsized":"2024-01-15T18:07:21.369770+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:17:46.586670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.188083+0000","last_change":"2024-01-15T18:07:19.076027+0000","last_active":"2024-01-15T18:07:20.188083+0000","last_peered":"2024-01-15T18:07:20.188083+0000","last_clean":"2024-01-15T18:07:20.188083+0000","last_became_active":"2024-01-15T18:07:19.075644+0000","last_became_peered":"2024-01-15T18:07:19.075644+0000","last_unstale":"2024-01-15T18:07:20.188083+0000","last_undegraded":"2024-01-15T18:07:20.188083+0000","last_fullsized":"2024-01-15T18:07:20.188083+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:07:57.062677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.368857+0000","last_change":"2024-01-15T18:07:20.088258+0000","last_active":"2024-01-15T18:07:21.368857+0000","last_peered":"2024-01-15T18:07:21.368857+0000","last_clean":"2024-01-15T18:07:21.368857+0000","last_became_active":"2024-01-15T18:07:20.087932+0000","last_became_peered":"2024-01-15T18:07:20.087932+0000","last_unstale":"2024-01-15T18:07:21.368857+0000","last_undegraded":"2024-01-15T18:07:21.368857+0000","last_fullsized":"2024-01-15T18:07:21.368857+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:26:07.172780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.185750+0000","last_change":"2024-01-15T18:07:19.081215+0000","last_active":"2024-01-15T18:07:20.185750+0000","last_peered":"2024-01-15T18:07:20.185750+0000","last_clean":"2024-01-15T18:07:20.185750+0000","last_became_active":"2024-01-15T18:07:19.080901+0000","last_became_peered":"2024-01-15T18:07:19.080901+0000","last_unstale":"2024-01-15T18:07:20.185750+0000","last_undegraded":"2024-01-15T18:07:20.185750+0000","last_fullsized":"2024-01-15T18:07:20.185750+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:38:52.124114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.083415+0000","last_change":"2024-01-15T18:07:20.081122+0000","last_active":"2024-01-15T18:07:21.083415+0000","last_peered":"2024-01-15T18:07:21.083415+0000","last_clean":"2024-01-15T18:07:21.083415+0000","last_became_active":"2024-01-15T18:07:20.080752+0000","last_became_peered":"2024-01-15T18:07:20.080752+0000","last_unstale":"2024-01-15T18:07:21.083415+0000","last_undegraded":"2024-01-15T18:07:21.083415+0000","last_fullsized":"2024-01-15T18:07:21.083415+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:36:14.191387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.081001+0000","last_change":"2024-01-15T18:07:19.085696+0000","last_active":"2024-01-15T18:07:20.081001+0000","last_peered":"2024-01-15T18:07:20.081001+0000","last_clean":"2024-01-15T18:07:20.081001+0000","last_became_active":"2024-01-15T18:07:19.085467+0000","last_became_peered":"2024-01-15T18:07:19.085467+0000","last_unstale":"2024-01-15T18:07:20.081001+0000","last_undegraded":"2024-01-15T18:07:20.081001+0000","last_fullsized":"2024-01-15T18:07:20.081001+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:17:23.043077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-15T18:07:21.370310+0000","last_change":"2024-01-15T18:07:20.092107+0000","last_active":"2024-01-15T18:07:21.370310+0000","last_peered":"2024-01-15T18:07:21.370310+0000","last_clean":"2024-01-15T18:07:21.370310+0000","last_became_active":"2024-01-15T18:07:20.091789+0000","last_became_peered":"2024-01-15T18:07:20.091789+0000","last_unstale":"2024-01-15T18:07:21.370310+0000","last_undegraded":"2024-01-15T18:07:21.370310+0000","last_fullsized":"2024-01-15T18:07:21.370310+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:19.050323+0000","last_clean_scrub_stamp":"2024-01-15T18:07:19.050323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:26:07.172780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"44'2","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.100732+0000","last_change":"2024-01-15T18:07:19.081361+0000","last_active":"2024-01-15T18:07:20.100732+0000","last_peered":"2024-01-15T18:07:20.100732+0000","last_clean":"2024-01-15T18:07:20.100732+0000","last_became_active":"2024-01-15T18:07:19.081076+0000","last_became_peered":"2024-01-15T18:07:19.081076+0000","last_unstale":"2024-01-15T18:07:20.100732+0000","last_undegraded":"2024-01-15T18:07:20.100732+0000","last_fullsized":"2024-01-15T18:07:20.100732+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:07:18.047640+0000","last_clean_scrub_stamp":"2024-01-15T18:07:18.047640+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:41:42.176887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":184,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-01-15T18:07:20.076735+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:07:20.076735+0000","last_peered":"2024-01-15T18:07:20.076735+0000","last_clean":"2024-01-15T18:07:20.076735+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:07:20.076735+0000","last_undegraded":"2024-01-15T18:07:20.076735+0000","last_fullsized":"2024-01-15T18:07:20.076735+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509475,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1118208,"data_stored":802396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.628,"15min":0.625},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.505},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.67200000000000004,"15min":0.67700000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.52900000000000003}]},{"osd":1,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.67800000000000005,"15min":0.67100000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.69399999999999995,"15min":0.69799999999999995},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.90400000000000003,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.64700000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64800000000000002,"15min":0.64700000000000002},"min":{"1min":0.46700000000000003,"5min":0.38,"15min":0.38},"max":{"1min":0.84199999999999997,"5min":1.012,"15min":1.012},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.71299999999999997,"15min":0.71199999999999997},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.0980000000000001,"5min":1.175,"15min":1.175},"last":0.55900000000000005}]},{"osd":3,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.66900000000000004,"15min":0.66500000000000004},"min":{"1min":0.502,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.88200000000000001,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0569999999999999,"5min":1.103,"15min":1.103},"last":0.67000000000000004}]},{"osd":4,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.60799999999999998,"15min":0.61599999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.69199999999999995,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.65700000000000003,"15min":0.66200000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.63100000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.69499999999999995,"15min":0.69899999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.75},{"interface":"front","average":{"1min":0.63,"5min":0.70999999999999996,"15min":0.72299999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.89500000000000002,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.76600000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:06:48 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.78800000000000003,"15min":0.80000000000000004},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.89500000000000002,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72499999999999998,"15min":0.73099999999999998},"min":{"1min":0.52500000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.78000000000000003}]}]},{"osd":6,"up_from":15,"seq":64424509475,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31700,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":528384,"data_stored":212610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.432,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.67200000000000004,"15min":0.68899999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84899999999999998,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.91500000000000004}]},{"osd":1,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.70299999999999996,"15min":0.72399999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.82699999999999996,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.74399999999999999,"15min":0.754},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":0.89100000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.82099999999999995,"15min":0.84399999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.825,"5min":2.8700000000000001,"15min":2.8700000000000001},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.755,"15min":0.78300000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.82399999999999995,"5min":1.857,"15min":1.857},"last":0.78700000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.77400000000000002,"15min":0.80100000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.92100000000000004,"5min":2.012,"15min":2.012},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.79900000000000004,"15min":0.82499999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":2.532,"15min":2.532},"last":0.65800000000000003}]},{"osd":4,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.68300000000000005,"15min":0.69299999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.726,"5min":1.726,"15min":1.726},"last":0.629},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.69399999999999995,"15min":0.71799999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.753,"5min":1.145,"15min":1.145},"last":0.68700000000000006}]},{"osd":5,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.64500000000000002,"15min":0.65800000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.81299999999999994,"5min":1.121,"15min":1.121},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.77400000000000002,"15min":0.78100000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.7609999999999999,"5min":1.7609999999999999,"15min":1.7609999999999999},"last":0.753}]},{"osd":7,"last update":"Mon Jan 15 18:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.80100000000000005,"15min":0.82499999999999996},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.96499999999999997,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.78000000000000003,"15min":0.80200000000000005},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.081,"5min":1.165,"15min":1.165},"last":0.86299999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509475,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31704,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":532480,"data_stored":213550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.628,"15min":0.63},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.496},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.63200000000000001,"15min":0.64100000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.57599999999999996}]},{"osd":1,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.64200000000000002,"15min":0.64900000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.77300000000000002,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.64000000000000001,"15min":0.64200000000000002},"min":{"1min":0.40899999999999997,"5min":0.377,"15min":0.377},"max":{"1min":0.88,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.70099999999999996}]},{"osd":2,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.626,"15min":0.628},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.69099999999999995,"15min":0.69499999999999995},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.81999999999999995,"5min":0.879,"15min":0.879},"last":0.81200000000000006}]},{"osd":3,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68899999999999995,"15min":0.68999999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.96699999999999997,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.72499999999999998,"15min":0.72599999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.93200000000000005,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.79000000000000004}]},{"osd":5,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40200000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65100000000000002,"15min":0.65000000000000002},"min":{"1min":0.40699999999999997,"5min":0.377,"15min":0.377},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.753}]},{"osd":6,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.68200000000000005,"15min":0.68000000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.73699999999999999,"15min":0.73499999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.86099999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.46899999999999997,"5min":0.436,"15min":0.436},"max":{"1min":1.0069999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.83899999999999997}]}]},{"osd":3,"up_from":15,"seq":64424509474,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31700,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":532480,"data_stored":212576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.28499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.72399999999999998,"5min":0.752,"15min":0.752},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.499,"15min":0.47999999999999998},"min":{"1min":0.34000000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":2.6509999999999998,"5min":2.6509999999999998,"15min":2.6509999999999998},"last":0.52200000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70999999999999996,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.53700000000000003,"15min":0.53600000000000003},"min":{"1min":0.42499999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83399999999999996,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.46800000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.55800000000000005,"15min":0.56000000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.58599999999999997,"15min":0.57199999999999995},"min":{"1min":0.42899999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.43099999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.79400000000000004,"15min":0.80300000000000005},"min":{"1min":0.50800000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.119,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.77300000000000002,"15min":0.77900000000000003},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.97899999999999998,"5min":1.341,"15min":1.341},"last":0.86499999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.59199999999999997,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.72399999999999998,"15min":0.72199999999999998},"min":{"1min":0.53300000000000003,"5min":0.498,"15min":0.498},"max":{"1min":1.415,"5min":1.415,"15min":1.415},"last":0.754}]},{"osd":6,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.78300000000000003,"15min":0.78500000000000003},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.77600000000000002,"15min":0.76600000000000001},"min":{"1min":0.628,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":2.0880000000000001,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":0.92200000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.78800000000000003,"15min":0.78100000000000003},"min":{"1min":0.58999999999999997,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.9550000000000001,"5min":1.9550000000000001,"15min":1.9550000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.81299999999999994,"15min":0.81999999999999995},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.1259999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.83199999999999996}]}]},{"osd":2,"up_from":15,"seq":64424509474,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":536576,"data_stored":213572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.45500000000000002,"15min":0.46100000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.75,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.441,"5min":0.46100000000000002,"15min":0.46500000000000002},"min":{"1min":0.29899999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.64700000000000002,"5min":0.753,"15min":0.753},"last":0.34499999999999997}]},{"osd":1,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54300000000000004,"15min":0.54500000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.67600000000000005,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.314,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.65100000000000002,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.42899999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.57799999999999996,"15min":0.57299999999999995},"min":{"1min":0.375,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.55600000000000005,"15min":0.55300000000000005},"min":{"1min":0.40500000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71499999999999997,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.496}]},{"osd":4,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.75700000000000001,"15min":0.76700000000000002},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.038,"5min":1.466,"15min":1.466},"last":0.65400000000000003}]},{"osd":5,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.79400000000000004,"15min":0.79700000000000004},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.609,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.747,"5min":0.78300000000000003,"15min":0.78900000000000003},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.69799999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.76800000000000002,"15min":0.77800000000000002},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0489999999999999,"5min":1.516,"15min":1.516},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.78600000000000003,"15min":0.79600000000000004},"min":{"1min":0.52800000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0820000000000001,"5min":1.7849999999999999,"15min":1.7849999999999999},"last":0.63400000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.79500000000000004,"15min":0.80100000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.1659999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.81799999999999995,"15min":0.82599999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.59899999999999998}]}]},{"osd":1,"up_from":15,"seq":64424509476,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31692,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340148,"statfs":{"total":96636764160,"available":96604311552,"internally_reserved":0,"allocated":524288,"data_stored":212008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.49099999999999999,"15min":0.504},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.627,"5min":1.847,"15min":1.847},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.443,"5min":0.47899999999999998,"15min":0.48499999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.63500000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.26200000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.53400000000000003,"15min":0.54400000000000004},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.77500000000000002,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.70299999999999996,"15min":0.73299999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.90100000000000002,"5min":4.2450000000000001,"15min":4.2450000000000001},"last":0.33000000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.63100000000000001,"15min":0.64500000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80700000000000005,"5min":1.871,"15min":1.871},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.58099999999999996,"15min":0.59099999999999997},"min":{"1min":0.35299999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.72599999999999998,"5min":1.917,"15min":1.917},"last":0.35899999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.70299999999999996,"15min":0.69599999999999995},"min":{"1min":0.54800000000000004,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.161,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.751},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.77100000000000002,"15min":0.77700000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.147,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.71599999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.76900000000000002,"15min":0.77100000000000002},"min":{"1min":0.47299999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.214,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.53100000000000003,"5min":0.505,"15min":0.505},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.73399999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.753,"15min":0.755},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.1359999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.81599999999999995,"15min":0.82199999999999995},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.77100000000000002}]},{"osd":7,"last update":"Mon Jan 15 18:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78700000000000003,"15min":0.78500000000000003},"min":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.79900000000000004,"15min":0.80300000000000005},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.095,"5min":1.347,"15min":1.347},"last":0.85199999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542179,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31700,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":524288,"data_stored":212008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":0.871,"15min":0.77500000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":21.385000000000002,"5min":21.385000000000002,"15min":21.385000000000002},"last":0.63500000000000001},{"interface":"front","average":{"1min":1.202,"5min":0.878,"15min":0.82399999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":8.2029999999999994,"5min":8.2029999999999994,"15min":8.2029999999999994},"last":0.79200000000000004}]},{"osd":1,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66400000000000003,"15min":0.67200000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":1.4670000000000001,"5min":0.84099999999999997,"15min":0.73699999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":21.5,"5min":21.5,"15min":21.5},"last":0.53600000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":0.81599999999999995,"15min":0.745},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":8.2439999999999998,"5min":8.2439999999999998,"15min":8.2439999999999998},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.748,"15min":0.77000000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.85099999999999998,"5min":1.663,"15min":1.663},"last":0.748}]},{"osd":3,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":0.95299999999999996,"15min":0.85699999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":21.457000000000001,"5min":21.457000000000001,"15min":21.457000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.1919999999999999,"5min":0.92100000000000004,"15min":0.876},"min":{"1min":0.50900000000000001,"5min":0.502,"15min":0.502},"max":{"1min":8.2750000000000004,"5min":8.2750000000000004,"15min":8.2750000000000004},"last":0.59999999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":0.82599999999999996,"15min":0.76600000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":9.6679999999999993,"5min":9.6679999999999993,"15min":9.6679999999999993},"last":0.96599999999999997},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.71299999999999997,"15min":0.73499999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.76300000000000001,"5min":1.2290000000000001,"15min":1.2290000000000001},"last":0.85599999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":0.877,"15min":0.78400000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":21.535,"5min":21.535,"15min":21.535},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.1870000000000001,"5min":0.88300000000000001,"15min":0.83199999999999996},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":8.298,"5min":8.298,"15min":8.298},"last":0.72099999999999997}]},{"osd":7,"last update":"Mon Jan 15 18:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":0.83999999999999997,"15min":0.81399999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":9.2240000000000002,"5min":9.2240000000000002,"15min":9.2240000000000002},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.5249999999999999,"5min":0.97099999999999997,"15min":0.879},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":21.574000000000002,"5min":21.574000000000002,"15min":21.574000000000002},"last":0.98099999999999998}]}]},{"osd":0,"up_from":14,"seq":60129542179,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1118208,"data_stored":802340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.46300000000000002,"15min":0.46500000000000002},"min":{"1min":0.28999999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.746,"5min":1.462,"15min":1.462},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55400000000000005,"15min":0.55200000000000005},"min":{"1min":0.34399999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.995,"5min":1.373,"15min":1.373},"last":0.52100000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.371,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.89600000000000002,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.48699999999999999,"15min":0.48099999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.252,"5min":1.609,"15min":1.609},"last":0.45800000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.371,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.321,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60899999999999999,"15min":0.60999999999999999},"min":{"1min":0.45000000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.94599999999999995,"5min":1.325,"15min":1.325},"last":0.54400000000000004}]},{"osd":4,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.76700000000000002,"15min":0.77700000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.96099999999999997,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.80900000000000005,"15min":0.82599999999999996},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.93300000000000005,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.65200000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.78500000000000003,"15min":0.79500000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.0109999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.83099999999999996,"15min":0.84699999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":0.69899999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.79400000000000004,"15min":0.79800000000000004},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.107,"5min":1.764,"15min":1.764},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.83099999999999996,"15min":0.84699999999999998},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.97499999999999998,"5min":1.806,"15min":1.806},"last":0.67900000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.82199999999999995,"15min":0.82999999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.04,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.84899999999999998,"15min":0.85999999999999999},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.0429999999999999,"5min":1.722,"15min":1.722},"last":0.74299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:07:29.116 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-15T18:07:29.116 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-15T18:07:29.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.296+0000 7fcb3ae00640 1 Processor -- start 2024-01-15T18:07:29.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.296+0000 7fcb3ae00640 1 -- start start 2024-01-15T18:07:29.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.300+0000 7fcb3ae00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcb3406c430 0x7fcb34070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:29.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.300+0000 7fcb3ae00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fcb340b1db0 0x7fcb340b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:29.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.300+0000 7fcb3ae00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb340ccfb0 con 0x7fcb340ce050 2024-01-15T18:07:29.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.300+0000 7fcb3ae00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb340cd290 con 0x7fcb3406c430 2024-01-15T18:07:29.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.300+0000 7fcb3ae00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb340cd570 con 0x7fcb340b1db0 2024-01-15T18:07:29.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.300+0000 7fcb33e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcb3406c430 0x7fcb34070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:29.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.300+0000 7fcb33e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcb3406c430 0x7fcb34070ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50804/0 (socket says 172.21.15.27:50804) 2024-01-15T18:07:29.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.300+0000 7fcb33e00640 1 -- 172.21.15.27:0/4107771056 learned_addr learned my addr 172.21.15.27:0/4107771056 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:29.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.300+0000 7fcb33400640 1 --2- 172.21.15.27:0/4107771056 >> v2:172.21.15.93:3300/0 conn(0x7fcb340b1db0 0x7fcb340b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:29.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.300+0000 7fcb33e00640 1 -- 172.21.15.27:0/4107771056 >> v2:172.21.15.93:3300/0 conn(0x7fcb340b1db0 msgr2=0x7fcb340b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:29.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.300+0000 7fcb33e00640 1 --2- 172.21.15.27:0/4107771056 >> v2:172.21.15.93:3300/0 conn(0x7fcb340b1db0 0x7fcb340b2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.300+0000 7fcb33e00640 1 -- 172.21.15.27:0/4107771056 >> v1:172.21.15.93:6789/0 conn(0x7fcb340ce050 legacy=0x7fcb3406c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:29.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.300+0000 7fcb33e00640 1 -- 172.21.15.27:0/4107771056 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb340cd850 con 0x7fcb3406c430 2024-01-15T18:07:29.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.300+0000 7fcb33e00640 1 --2- 172.21.15.27:0/4107771056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb3406c430 0x7fcb34070ba0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fcb24009870 tx=0x7fcb2401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c935d2b6b48ffc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:29.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.304+0000 7fcb32a00640 1 -- 172.21.15.27:0/4107771056 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb240ab600 con 0x7fcb3406c430 2024-01-15T18:07:29.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.304+0000 7fcb32a00640 1 -- 172.21.15.27:0/4107771056 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb240a7050 con 0x7fcb3406c430 2024-01-15T18:07:29.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.304+0000 7fcb32a00640 1 -- 172.21.15.27:0/4107771056 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb240ab980 con 0x7fcb3406c430 2024-01-15T18:07:29.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.304+0000 7fcb3ae00640 1 -- 172.21.15.27:0/4107771056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb3406c430 msgr2=0x7fcb34070ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:29.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.304+0000 7fcb3ae00640 1 --2- 172.21.15.27:0/4107771056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb3406c430 0x7fcb34070ba0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fcb24009870 tx=0x7fcb2401c400 comp rx=0 tx=0).stop 2024-01-15T18:07:29.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.304+0000 7fcb3ae00640 1 -- 172.21.15.27:0/4107771056 shutdown_connections 2024-01-15T18:07:29.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.304+0000 7fcb3ae00640 1 --2- 172.21.15.27:0/4107771056 >> v2:172.21.15.93:3300/0 conn(0x7fcb340b1db0 0x7fcb340b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.304+0000 7fcb3ae00640 1 --2- 172.21.15.27:0/4107771056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb3406c430 0x7fcb34070ba0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.304+0000 7fcb3ae00640 1 -- 172.21.15.27:0/4107771056 >> 172.21.15.27:0/4107771056 conn(0x7fcb340cb700 msgr2=0x7fcb340cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:29.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.304+0000 7fcb3ae00640 1 -- 172.21.15.27:0/4107771056 shutdown_connections 2024-01-15T18:07:29.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.304+0000 7fcb3ae00640 1 -- 172.21.15.27:0/4107771056 wait complete. 2024-01-15T18:07:29.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.304+0000 7fcb3ae00640 1 Processor -- start 2024-01-15T18:07:29.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.304+0000 7fcb3ae00640 1 -- start start 2024-01-15T18:07:29.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.304+0000 7fcb3ae00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcb3406c430 0x7fcb3414e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:29.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.308+0000 7fcb3ae00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb340b1db0 0x7fcb3415af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:29.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.308+0000 7fcb3ae00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcb340ce050 0x7fcb341527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:29.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.308+0000 7fcb3ae00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb34159f20 con 0x7fcb340b1db0 2024-01-15T18:07:29.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.308+0000 7fcb3ae00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3415a170 con 0x7fcb3406c430 2024-01-15T18:07:29.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.308+0000 7fcb3ae00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3415a450 con 0x7fcb340ce050 2024-01-15T18:07:29.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.308+0000 7fcb33400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb340b1db0 0x7fcb3415af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:29.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.308+0000 7fcb33400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb340b1db0 0x7fcb3415af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50816/0 (socket says 172.21.15.27:50816) 2024-01-15T18:07:29.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.308+0000 7fcb33400640 1 -- 172.21.15.27:0/1938132494 learned_addr learned my addr 172.21.15.27:0/1938132494 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:29.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.308+0000 7fcb33e00640 1 --2- 172.21.15.27:0/1938132494 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcb3406c430 0x7fcb3414e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:29.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.308+0000 7fcb38e00640 1 --2- 172.21.15.27:0/1938132494 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcb340ce050 0x7fcb341527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:29.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.308+0000 7fcb33400640 1 -- 172.21.15.27:0/1938132494 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcb340ce050 msgr2=0x7fcb341527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:29.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.308+0000 7fcb33400640 1 --2- 172.21.15.27:0/1938132494 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcb340ce050 0x7fcb341527b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.308+0000 7fcb33400640 1 -- 172.21.15.27:0/1938132494 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcb3406c430 msgr2=0x7fcb3414e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:29.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.308+0000 7fcb33400640 1 --2- 172.21.15.27:0/1938132494 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcb3406c430 0x7fcb3414e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.308+0000 7fcb33400640 1 -- 172.21.15.27:0/1938132494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb240094d0 con 0x7fcb340b1db0 2024-01-15T18:07:29.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.308+0000 7fcb33400640 1 --2- 172.21.15.27:0/1938132494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb340b1db0 0x7fcb3415af30 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c00e930 tx=0x7fcb1c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:29.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.308+0000 7fcb2be00640 1 -- 172.21.15.27:0/1938132494 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c009690 con 0x7fcb340b1db0 2024-01-15T18:07:29.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.308+0000 7fcb2be00640 1 -- 172.21.15.27:0/1938132494 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb1c00ca00 con 0x7fcb340b1db0 2024-01-15T18:07:29.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.308+0000 7fcb2be00640 1 -- 172.21.15.27:0/1938132494 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c00cd70 con 0x7fcb340b1db0 2024-01-15T18:07:29.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.308+0000 7fcb3ae00640 1 -- 172.21.15.27:0/1938132494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb3415a730 con 0x7fcb340b1db0 2024-01-15T18:07:29.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.308+0000 7fcb3ae00640 1 -- 172.21.15.27:0/1938132494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb3415b470 con 0x7fcb340b1db0 2024-01-15T18:07:29.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.308+0000 7fcb3ae00640 1 -- 172.21.15.27:0/1938132494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb3404e850 con 0x7fcb340b1db0 2024-01-15T18:07:29.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.308+0000 7fcb2be00640 1 -- 172.21.15.27:0/1938132494 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7fcb1c013470 con 0x7fcb340b1db0 2024-01-15T18:07:29.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.312+0000 7fcb2be00640 1 --2- 172.21.15.27:0/1938132494 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcb0c066b30 0x7fcb0c068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:29.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.312+0000 7fcb2be00640 1 -- 172.21.15.27:0/1938132494 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fcb1c0849c0 con 0x7fcb340b1db0 2024-01-15T18:07:29.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.312+0000 7fcb33e00640 1 --2- 172.21.15.27:0/1938132494 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcb0c066b30 0x7fcb0c068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:29.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.312+0000 7fcb33e00640 1 --2- 172.21.15.27:0/1938132494 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcb0c066b30 0x7fcb0c068ff0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fcb240ad000 tx=0x7fcb240a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:29.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.316+0000 7fcb2be00640 1 -- 172.21.15.27:0/1938132494 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fcb1c051a80 con 0x7fcb340b1db0 2024-01-15T18:07:29.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.528+0000 7fcb3ae00640 1 -- 172.21.15.27:0/1938132494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fcb3415ced0 con 0x7fcb340b1db0 2024-01-15T18:07:29.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.532+0000 7fcb2be00640 1 -- 172.21.15.27:0/1938132494 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v10) v1 ==== 190+0+147 (secure 0 0 0) 0x7fcb1c0210b0 con 0x7fcb340b1db0 2024-01-15T18:07:29.535 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-01-15T18:07:29.535 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOc6VlH+DCLxAA1sKVut+3WjA/B+aYJ6/Jeg== 2024-01-15T18:07:29.535 INFO:teuthology.orchestra.run.smithi027.stdout: caps mds = "allow" 2024-01-15T18:07:29.535 INFO:teuthology.orchestra.run.smithi027.stdout: caps mon = "allow r" 2024-01-15T18:07:29.535 INFO:teuthology.orchestra.run.smithi027.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-15T18:07:29.535 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-01-15T18:07:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.532+0000 7fcb3ae00640 1 -- 172.21.15.27:0/1938132494 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcb0c066b30 msgr2=0x7fcb0c068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.532+0000 7fcb3ae00640 1 --2- 172.21.15.27:0/1938132494 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcb0c066b30 0x7fcb0c068ff0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fcb240ad000 tx=0x7fcb240a6000 comp rx=0 tx=0).stop 2024-01-15T18:07:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.532+0000 7fcb3ae00640 1 -- 172.21.15.27:0/1938132494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb340b1db0 msgr2=0x7fcb3415af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.532+0000 7fcb3ae00640 1 --2- 172.21.15.27:0/1938132494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb340b1db0 0x7fcb3415af30 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c00e930 tx=0x7fcb1c00edf0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.532+0000 7fcb3ae00640 1 -- 172.21.15.27:0/1938132494 shutdown_connections 2024-01-15T18:07:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.532+0000 7fcb3ae00640 1 --2- 172.21.15.27:0/1938132494 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcb0c066b30 0x7fcb0c068ff0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.532+0000 7fcb3ae00640 1 --2- 172.21.15.27:0/1938132494 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcb340ce050 0x7fcb341527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.532+0000 7fcb3ae00640 1 --2- 172.21.15.27:0/1938132494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb340b1db0 0x7fcb3415af30 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.532+0000 7fcb3ae00640 1 --2- 172.21.15.27:0/1938132494 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcb3406c430 0x7fcb3414e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.532+0000 7fcb3ae00640 1 -- 172.21.15.27:0/1938132494 >> 172.21.15.27:0/1938132494 conn(0x7fcb340cb700 msgr2=0x7fcb340ae920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:29.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.532+0000 7fcb3ae00640 1 -- 172.21.15.27:0/1938132494 shutdown_connections 2024-01-15T18:07:29.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.532+0000 7fcb3ae00640 1 -- 172.21.15.27:0/1938132494 wait complete. 2024-01-15T18:07:29.557 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:07:29.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.692+0000 7f6865e00640 1 Processor -- start 2024-01-15T18:07:29.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.692+0000 7f6865e00640 1 -- start start 2024-01-15T18:07:29.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.696+0000 7f6865e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f686015d0a0 0x7f686006c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:29.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.696+0000 7f6865e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68600ad180 con 0x7f68600c94c0 2024-01-15T18:07:29.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.696+0000 7f6865e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68600ac890 con 0x7f68600ce710 2024-01-15T18:07:29.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.696+0000 7f6865e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68600cda90 con 0x7f686015d0a0 2024-01-15T18:07:29.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.696+0000 7f685ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f686015d0a0 0x7f686006c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:29.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.696+0000 7f685f400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f68600c94c0 0x7f68600ad610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45222/0 (socket says 172.21.15.27:45222) 2024-01-15T18:07:29.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.696+0000 7f685f400640 1 -- 172.21.15.27:0/3064446897 learned_addr learned my addr 172.21.15.27:0/3064446897 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:29.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.696+0000 7f685ea00640 1 -- 172.21.15.27:0/3064446897 >> v1:172.21.15.93:6789/0 conn(0x7f68600ce710 legacy=0x7f68600cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:29.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.696+0000 7f685ea00640 1 -- 172.21.15.27:0/3064446897 >> v1:172.21.15.27:6789/0 conn(0x7f68600c94c0 legacy=0x7f68600ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:29.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.696+0000 7f685ea00640 1 -- 172.21.15.27:0/3064446897 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68600acb30 con 0x7f686015d0a0 2024-01-15T18:07:29.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.700+0000 7f685ea00640 1 --2- 172.21.15.27:0/3064446897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f686015d0a0 0x7f686006c1e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f68500097c0 tx=0x7f685001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=448923fefd11319c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:29.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.700+0000 7f685d600640 1 -- 172.21.15.27:0/3064446897 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68500ab600 con 0x7f686015d0a0 2024-01-15T18:07:29.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.700+0000 7f685d600640 1 -- 172.21.15.27:0/3064446897 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68500a7050 con 0x7f686015d0a0 2024-01-15T18:07:29.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.700+0000 7f685d600640 1 -- 172.21.15.27:0/3064446897 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68500ab980 con 0x7f686015d0a0 2024-01-15T18:07:29.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.700+0000 7f6865e00640 1 -- 172.21.15.27:0/3064446897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f686015d0a0 msgr2=0x7f686006c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:29.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.700+0000 7f6865e00640 1 --2- 172.21.15.27:0/3064446897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f686015d0a0 0x7f686006c1e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f68500097c0 tx=0x7f685001c410 comp rx=0 tx=0).stop 2024-01-15T18:07:29.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.700+0000 7f6865e00640 1 -- 172.21.15.27:0/3064446897 shutdown_connections 2024-01-15T18:07:29.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.700+0000 7f6865e00640 1 --2- 172.21.15.27:0/3064446897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f686015d0a0 0x7f686006c1e0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.700+0000 7f6865e00640 1 -- 172.21.15.27:0/3064446897 >> 172.21.15.27:0/3064446897 conn(0x7f68600cec60 msgr2=0x7f68600cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:29.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.700+0000 7f6865e00640 1 -- 172.21.15.27:0/3064446897 shutdown_connections 2024-01-15T18:07:29.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.700+0000 7f6865e00640 1 -- 172.21.15.27:0/3064446897 wait complete. 2024-01-15T18:07:29.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.700+0000 7f6865e00640 1 Processor -- start 2024-01-15T18:07:29.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.700+0000 7f6865e00640 1 -- start start 2024-01-15T18:07:29.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.700+0000 7f6865e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f68600c94c0 0x7f68601677c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:29.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.704+0000 7f6865e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68600ce710 0x7f6860145130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:29.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.704+0000 7f6865e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f686015d0a0 0x7f68601431a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:29.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.704+0000 7f6865e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6860142100 con 0x7f68600ce710 2024-01-15T18:07:29.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.704+0000 7f6865e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68601423e0 con 0x7f686015d0a0 2024-01-15T18:07:29.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.704+0000 7f6865e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68601426c0 con 0x7f68600c94c0 2024-01-15T18:07:29.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.704+0000 7f685e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68600ce710 0x7f6860145130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:29.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.704+0000 7f685e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68600ce710 0x7f6860145130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50828/0 (socket says 172.21.15.27:50828) 2024-01-15T18:07:29.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.704+0000 7f685e000640 1 -- 172.21.15.27:0/4106873985 learned_addr learned my addr 172.21.15.27:0/4106873985 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:29.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.704+0000 7f685e000640 1 -- 172.21.15.27:0/4106873985 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f68600c94c0 msgr2=0x7f68601677c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:29.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.704+0000 7f685e000640 1 --2- 172.21.15.27:0/4106873985 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f68600c94c0 0x7f68601677c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.704+0000 7f685e000640 1 -- 172.21.15.27:0/4106873985 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f686015d0a0 msgr2=0x7f68601431a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.704+0000 7f685f400640 1 --2- 172.21.15.27:0/4106873985 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f686015d0a0 0x7f68601431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.704+0000 7f685e000640 1 --2- 172.21.15.27:0/4106873985 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f686015d0a0 0x7f68601431a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.704+0000 7f685e000640 1 -- 172.21.15.27:0/4106873985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6850009420 con 0x7f68600ce710 2024-01-15T18:07:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.704+0000 7f685f400640 1 --2- 172.21.15.27:0/4106873985 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f686015d0a0 0x7f68601431a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:07:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.704+0000 7f685e000640 1 --2- 172.21.15.27:0/4106873985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68600ce710 0x7f6860145130 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f684800b410 tx=0x7f684800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.704+0000 7f6857400640 1 -- 172.21.15.27:0/4106873985 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6848004020 con 0x7f68600ce710 2024-01-15T18:07:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.704+0000 7f6857400640 1 -- 172.21.15.27:0/4106873985 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6848004180 con 0x7f68600ce710 2024-01-15T18:07:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.704+0000 7f6865e00640 1 -- 172.21.15.27:0/4106873985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68601429a0 con 0x7f68600ce710 2024-01-15T18:07:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.704+0000 7f6857400640 1 -- 172.21.15.27:0/4106873985 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6848011630 con 0x7f68600ce710 2024-01-15T18:07:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.704+0000 7f6865e00640 1 -- 172.21.15.27:0/4106873985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6860167d00 con 0x7f68600ce710 2024-01-15T18:07:29.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.704+0000 7f6865e00640 1 -- 172.21.15.27:0/4106873985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f686004e850 con 0x7f68600ce710 2024-01-15T18:07:29.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.704+0000 7f6857400640 1 -- 172.21.15.27:0/4106873985 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f6848011790 con 0x7f68600ce710 2024-01-15T18:07:29.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.708+0000 7f6857400640 1 --2- 172.21.15.27:0/4106873985 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6838066b30 0x7f6838068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:29.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.708+0000 7f6857400640 1 -- 172.21.15.27:0/4106873985 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f68480846c0 con 0x7f68600ce710 2024-01-15T18:07:29.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.708+0000 7f685ea00640 1 --2- 172.21.15.27:0/4106873985 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6838066b30 0x7f6838068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:29.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.708+0000 7f685ea00640 1 --2- 172.21.15.27:0/4106873985 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6838066b30 0x7f6838068ff0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f68500ad000 tx=0x7f6850004740 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:29.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.712+0000 7f6857400640 1 -- 172.21.15.27:0/4106873985 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f68480514e0 con 0x7f68600ce710 2024-01-15T18:07:29.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.900+0000 7f6865e00640 1 -- 172.21.15.27:0/4106873985 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6860072160 con 0x7f68600ce710 2024-01-15T18:07:29.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.900+0000 7f6857400640 1 -- 172.21.15.27:0/4106873985 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4910 (secure 0 0 0) 0x7f6848054b00 con 0x7f68600ce710 2024-01-15T18:07:29.906 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:07:29.906 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":27,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5306,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/1548117999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":1548117999},{"type":"v1","addr":"172.21.15.93:6836","nonce":1548117999}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5719,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/260141697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":260141697},{"type":"v1","addr":"172.21.15.93:6837","nonce":260141697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5802,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/2030664596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2030664596},{"type":"v1","addr":"172.21.15.27:6834","nonce":2030664596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:07:19.460378+0000","modified":"2024-01-15T18:07:22.477523+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5808},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5808":{"gid":5808,"name":"a","rank":0,"incarnation":23,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/1005016790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1005016790},{"type":"v1","addr":"172.21.15.27:6835","nonce":1005016790}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-01-15T18:07:29.906 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 27 2024-01-15T18:07:29.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.904+0000 7f6865e00640 1 -- 172.21.15.27:0/4106873985 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6838066b30 msgr2=0x7f6838068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:29.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.904+0000 7f6865e00640 1 --2- 172.21.15.27:0/4106873985 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6838066b30 0x7f6838068ff0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f68500ad000 tx=0x7f6850004740 comp rx=0 tx=0).stop 2024-01-15T18:07:29.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.904+0000 7f6865e00640 1 -- 172.21.15.27:0/4106873985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68600ce710 msgr2=0x7f6860145130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:29.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.904+0000 7f6865e00640 1 --2- 172.21.15.27:0/4106873985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68600ce710 0x7f6860145130 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f684800b410 tx=0x7f684800b8d0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.904+0000 7f6865e00640 1 -- 172.21.15.27:0/4106873985 shutdown_connections 2024-01-15T18:07:29.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.904+0000 7f6865e00640 1 --2- 172.21.15.27:0/4106873985 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6838066b30 0x7f6838068ff0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.904+0000 7f6865e00640 1 --2- 172.21.15.27:0/4106873985 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f686015d0a0 0x7f68601431a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.904+0000 7f6865e00640 1 --2- 172.21.15.27:0/4106873985 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68600ce710 0x7f6860145130 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.904+0000 7f6865e00640 1 --2- 172.21.15.27:0/4106873985 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f68600c94c0 0x7f68601677c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.904+0000 7f6865e00640 1 -- 172.21.15.27:0/4106873985 >> 172.21.15.27:0/4106873985 conn(0x7f68600cec60 msgr2=0x7f6860070180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:29.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.904+0000 7f6865e00640 1 -- 172.21.15.27:0/4106873985 shutdown_connections 2024-01-15T18:07:29.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:29.904+0000 7f6865e00640 1 -- 172.21.15.27:0/4106873985 wait complete. 2024-01-15T18:07:30.330 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-15T18:07:19.460378+0000', 'modified': '2024-01-15T18:07:22.477523+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5808}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5808': {'gid': 5808, 'name': 'a', 'rank': 0, 'incarnation': 23, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6835/1005016790', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 1005016790}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 1005016790}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-15T18:07:30.330 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-15T18:07:30.331 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-15T18:07:30.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.468+0000 7fb119000640 1 Processor -- start 2024-01-15T18:07:30.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.468+0000 7fb119000640 1 -- start start 2024-01-15T18:07:30.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.472+0000 7fb119000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb11415d680 0x7fb1140cfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.472+0000 7fb119000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1140b3cd0 con 0x7fb11415d2a0 2024-01-15T18:07:30.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.472+0000 7fb119000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1140d0260 con 0x7fb1140cde10 2024-01-15T18:07:30.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.472+0000 7fb119000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1140cb670 con 0x7fb11415d680 2024-01-15T18:07:30.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.472+0000 7fb111e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb11415d2a0 0x7fb1140cd700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45224/0 (socket says 172.21.15.27:45224) 2024-01-15T18:07:30.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.472+0000 7fb111e00640 1 -- 172.21.15.27:0/1786627014 learned_addr learned my addr 172.21.15.27:0/1786627014 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:30.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.472+0000 7fb111400640 1 --2- 172.21.15.27:0/1786627014 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb11415d680 0x7fb1140cfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.472+0000 7fb110a00640 1 -- 172.21.15.27:0/1786627014 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3187386975 0 0) 0x7fb1140b3cd0 con 0x7fb11415d2a0 2024-01-15T18:07:30.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.472+0000 7fb110a00640 1 -- 172.21.15.27:0/1786627014 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0f4003530 con 0x7fb11415d2a0 2024-01-15T18:07:30.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.472+0000 7fb110a00640 1 -- 172.21.15.27:0/1786627014 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592813618 0 0) 0x7fb1140d0260 con 0x7fb1140cde10 2024-01-15T18:07:30.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.472+0000 7fb110a00640 1 -- 172.21.15.27:0/1786627014 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1140b3cd0 con 0x7fb1140cde10 2024-01-15T18:07:30.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.472+0000 7fb110a00640 1 -- 172.21.15.27:0/1786627014 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 885844045 0 0) 0x7fb0f4003530 con 0x7fb11415d2a0 2024-01-15T18:07:30.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.472+0000 7fb111400640 1 -- 172.21.15.27:0/1786627014 >> v1:172.21.15.93:6789/0 conn(0x7fb1140cde10 legacy=0x7fb1140cc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:30.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.472+0000 7fb111400640 1 -- 172.21.15.27:0/1786627014 >> v1:172.21.15.27:6789/0 conn(0x7fb11415d2a0 legacy=0x7fb1140cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:30.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.472+0000 7fb111400640 1 -- 172.21.15.27:0/1786627014 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1140cb950 con 0x7fb11415d680 2024-01-15T18:07:30.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.476+0000 7fb111400640 1 --2- 172.21.15.27:0/1786627014 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb11415d680 0x7fb1140cfbd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc0097f0 tx=0x7fb0fc01c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81cd6a6722d5c776 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:30.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.476+0000 7fb110a00640 1 -- 172.21.15.27:0/1786627014 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb104001600 con 0x7fb11415d680 2024-01-15T18:07:30.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.476+0000 7fb110a00640 1 -- 172.21.15.27:0/1786627014 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0fc0a7050 con 0x7fb11415d680 2024-01-15T18:07:30.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.476+0000 7fb110a00640 1 -- 172.21.15.27:0/1786627014 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc0ab620 con 0x7fb11415d680 2024-01-15T18:07:30.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.476+0000 7fb119000640 1 -- 172.21.15.27:0/1786627014 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb11415d680 msgr2=0x7fb1140cfbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:30.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.476+0000 7fb119000640 1 --2- 172.21.15.27:0/1786627014 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb11415d680 0x7fb1140cfbd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc0097f0 tx=0x7fb0fc01c320 comp rx=0 tx=0).stop 2024-01-15T18:07:30.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.476+0000 7fb119000640 1 -- 172.21.15.27:0/1786627014 shutdown_connections 2024-01-15T18:07:30.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.476+0000 7fb119000640 1 --2- 172.21.15.27:0/1786627014 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb11415d680 0x7fb1140cfbd0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.476+0000 7fb119000640 1 -- 172.21.15.27:0/1786627014 >> 172.21.15.27:0/1786627014 conn(0x7fb1140d0e80 msgr2=0x7fb1140d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:30.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.476+0000 7fb119000640 1 -- 172.21.15.27:0/1786627014 shutdown_connections 2024-01-15T18:07:30.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.476+0000 7fb119000640 1 -- 172.21.15.27:0/1786627014 wait complete. 2024-01-15T18:07:30.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.476+0000 7fb119000640 1 Processor -- start 2024-01-15T18:07:30.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.480+0000 7fb119000640 1 -- start start 2024-01-15T18:07:30.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.480+0000 7fb119000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1140cde10 0x7fb1140c4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.480+0000 7fb119000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb11415d2a0 0x7fb11415c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.480+0000 7fb119000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb11415d680 0x7fb11415c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.480+0000 7fb119000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1141f9ce0 con 0x7fb1140cde10 2024-01-15T18:07:30.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.480+0000 7fb119000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb114156ff0 con 0x7fb11415d2a0 2024-01-15T18:07:30.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.480+0000 7fb119000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1141572d0 con 0x7fb11415d680 2024-01-15T18:07:30.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.480+0000 7fb111e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1140cde10 0x7fb1140c4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.480+0000 7fb111e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1140cde10 0x7fb1140c4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50844/0 (socket says 172.21.15.27:50844) 2024-01-15T18:07:30.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.480+0000 7fb111e00640 1 -- 172.21.15.27:0/2146983123 learned_addr learned my addr 172.21.15.27:0/2146983123 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:30.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.480+0000 7fb111400640 1 --2- 172.21.15.27:0/2146983123 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb11415d2a0 0x7fb11415c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.480+0000 7fb112800640 1 --2- 172.21.15.27:0/2146983123 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb11415d680 0x7fb11415c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.480+0000 7fb111e00640 1 -- 172.21.15.27:0/2146983123 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb11415d680 msgr2=0x7fb11415c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:30.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.480+0000 7fb111e00640 1 --2- 172.21.15.27:0/2146983123 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb11415d680 0x7fb11415c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.480+0000 7fb111e00640 1 -- 172.21.15.27:0/2146983123 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb11415d2a0 msgr2=0x7fb11415c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:30.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.480+0000 7fb111e00640 1 --2- 172.21.15.27:0/2146983123 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb11415d2a0 0x7fb11415c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.480+0000 7fb111e00640 1 -- 172.21.15.27:0/2146983123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0fc009420 con 0x7fb1140cde10 2024-01-15T18:07:30.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.480+0000 7fb111e00640 1 --2- 172.21.15.27:0/2146983123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1140cde10 0x7fb1140c4990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fb10400c550 tx=0x7fb10400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:30.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.480+0000 7fb10a000640 1 -- 172.21.15.27:0/2146983123 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb104005cf0 con 0x7fb1140cde10 2024-01-15T18:07:30.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.480+0000 7fb10a000640 1 -- 172.21.15.27:0/2146983123 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb104005e70 con 0x7fb1140cde10 2024-01-15T18:07:30.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.480+0000 7fb10a000640 1 -- 172.21.15.27:0/2146983123 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb104007290 con 0x7fb1140cde10 2024-01-15T18:07:30.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.480+0000 7fb119000640 1 -- 172.21.15.27:0/2146983123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1141575b0 con 0x7fb1140cde10 2024-01-15T18:07:30.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.480+0000 7fb119000640 1 -- 172.21.15.27:0/2146983123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1140c4ed0 con 0x7fb1140cde10 2024-01-15T18:07:30.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.484+0000 7fb10a000640 1 -- 172.21.15.27:0/2146983123 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7fb1040073f0 con 0x7fb1140cde10 2024-01-15T18:07:30.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.484+0000 7fb10a000640 1 --2- 172.21.15.27:0/2146983123 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb0ec066b30 0x7fb0ec068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.484+0000 7fb10a000640 1 -- 172.21.15.27:0/2146983123 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fb1040844a0 con 0x7fb1140cde10 2024-01-15T18:07:30.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.484+0000 7fb11c374000 1 Processor -- start 2024-01-15T18:07:30.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.484+0000 7fb11c374000 1 -- start start 2024-01-15T18:07:30.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.484+0000 7fb111400640 1 --2- 172.21.15.27:0/2146983123 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb0ec066b30 0x7fb0ec068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.484+0000 7fb11c374000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561ff4ec0c90 0x561ff4ec1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.484+0000 7fb11c374000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4ec15b0 0x561ff4e76de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.484+0000 7fb11c374000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x561ff4e77320 0x561ff4e77700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.484+0000 7fb11c374000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ff4ce1130 con 0x561ff4ec15b0 2024-01-15T18:07:30.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.484+0000 7fb11c374000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ff4cce9a0 con 0x561ff4ec0c90 2024-01-15T18:07:30.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.484+0000 7fb11c374000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x561ff4ccbed0 con 0x561ff4e77320 2024-01-15T18:07:30.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.484+0000 7fb111400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4ec15b0 0x561ff4e76de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.484+0000 7fb112800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x561ff4e77320 0x561ff4e77700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.484+0000 7fb111400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4ec15b0 0x561ff4e76de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50856/0 (socket says 172.21.15.27:50856) 2024-01-15T18:07:30.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.484+0000 7fb111400640 1 -- 172.21.15.27:0/1496380070 learned_addr learned my addr 172.21.15.27:0/1496380070 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:30.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.484+0000 7fb111e00640 1 --2- 172.21.15.27:0/1496380070 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561ff4ec0c90 0x561ff4ec1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.484+0000 7fb111400640 1 --2- 172.21.15.27:0/2146983123 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb0ec066b30 0x7fb0ec068ff0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc0ad000 tx=0x7fb0fc0a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:30.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.484+0000 7fb111400640 1 -- 172.21.15.27:0/1496380070 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x561ff4e77320 msgr2=0x561ff4e77700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:30.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.484+0000 7fb111400640 1 --2- 172.21.15.27:0/1496380070 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x561ff4e77320 0x561ff4e77700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.484+0000 7fb111400640 1 -- 172.21.15.27:0/1496380070 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561ff4ec0c90 msgr2=0x561ff4ec1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:30.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.484+0000 7fb111400640 1 --2- 172.21.15.27:0/1496380070 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561ff4ec0c90 0x561ff4ec1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.484+0000 7fb111400640 1 -- 172.21.15.27:0/1496380070 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ff49bf890 con 0x561ff4ec15b0 2024-01-15T18:07:30.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.484+0000 7fb111400640 1 --2- 172.21.15.27:0/1496380070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4ec15b0 0x561ff4e76de0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc009740 tx=0x7fb0fc0a8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=519086c288bdd5fc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:30.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb102a00640 1 -- 172.21.15.27:0/1496380070 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc0abc30 con 0x561ff4ec15b0 2024-01-15T18:07:30.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb102a00640 1 -- 172.21.15.27:0/1496380070 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0fc0c4070 con 0x561ff4ec15b0 2024-01-15T18:07:30.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb102a00640 1 -- 172.21.15.27:0/1496380070 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc0b5440 con 0x561ff4ec15b0 2024-01-15T18:07:30.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb11c374000 1 -- 172.21.15.27:0/1496380070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4ec15b0 msgr2=0x561ff4e76de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:30.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb11c374000 1 --2- 172.21.15.27:0/1496380070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4ec15b0 0x561ff4e76de0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc009740 tx=0x7fb0fc0a8000 comp rx=0 tx=0).stop 2024-01-15T18:07:30.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb11c374000 1 -- 172.21.15.27:0/1496380070 shutdown_connections 2024-01-15T18:07:30.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb11c374000 1 --2- 172.21.15.27:0/1496380070 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x561ff4e77320 0x561ff4e77700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb11c374000 1 --2- 172.21.15.27:0/1496380070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4ec15b0 0x561ff4e76de0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb11c374000 1 --2- 172.21.15.27:0/1496380070 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561ff4ec0c90 0x561ff4ec1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb11c374000 1 -- 172.21.15.27:0/1496380070 >> 172.21.15.27:0/1496380070 conn(0x561ff4e732d0 msgr2=0x561ff4e756c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:30.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb11c374000 1 -- 172.21.15.27:0/1496380070 shutdown_connections 2024-01-15T18:07:30.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb11c374000 1 -- 172.21.15.27:0/1496380070 wait complete. 2024-01-15T18:07:30.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb11c374000 1 -- start start 2024-01-15T18:07:30.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb11c374000 1 Processor -- start 2024-01-15T18:07:30.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb11c374000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4e77320 0x561ff4e7bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb11c374000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561ff4ec0c90 0x561ff4e7c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb11c374000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x561ff4ec15b0 0x561ff4e8f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb11c374000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ff4cc77c0 con 0x561ff4e77320 2024-01-15T18:07:30.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb11c374000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ff4c7fbf0 con 0x561ff4ec0c90 2024-01-15T18:07:30.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb11c374000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x561ff4c85a90 con 0x561ff4ec15b0 2024-01-15T18:07:30.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb11c374000 10 client.0 _ll_register_callbacks cb 0x561ff4e623c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:07:30.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb111e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4e77320 0x561ff4e7bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb111e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4e77320 0x561ff4e7bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50862/0 (socket says 172.21.15.27:50862) 2024-01-15T18:07:30.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb111e00640 1 -- 172.21.15.27:0/2449311979 learned_addr learned my addr 172.21.15.27:0/2449311979 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:30.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb111400640 1 --2- 172.21.15.27:0/2449311979 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561ff4ec0c90 0x561ff4e7c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb112800640 1 --2- 172.21.15.27:0/2449311979 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x561ff4ec15b0 0x561ff4e8f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb111e00640 1 -- 172.21.15.27:0/2449311979 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x561ff4ec15b0 msgr2=0x561ff4e8f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:30.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb111e00640 1 --2- 172.21.15.27:0/2449311979 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x561ff4ec15b0 0x561ff4e8f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb111e00640 1 -- 172.21.15.27:0/2449311979 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561ff4ec0c90 msgr2=0x561ff4e7c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:30.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb111e00640 1 --2- 172.21.15.27:0/2449311979 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561ff4ec0c90 0x561ff4e7c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb111e00640 1 -- 172.21.15.27:0/2449311979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1140cb950 con 0x561ff4e77320 2024-01-15T18:07:30.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.488+0000 7fb111400640 1 --2- 172.21.15.27:0/2449311979 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561ff4ec0c90 0x561ff4e7c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:07:30.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb111e00640 1 --2- 172.21.15.27:0/2449311979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4e77320 0x561ff4e7bd40 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fb104054cc0 tx=0x7fb104059220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bac0207ecc2d2e76 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:30.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb0fb400640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-15T18:07:30.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb0fb400640 1 -- 172.21.15.27:0/2449311979 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb104059ea0 con 0x561ff4e77320 2024-01-15T18:07:30.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb0fb400640 1 -- 172.21.15.27:0/2449311979 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fb1040840b0 con 0x561ff4e77320 2024-01-15T18:07:30.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb11c374000 1 -- 172.21.15.27:0/2449311979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561ff4c89470 con 0x561ff4e77320 2024-01-15T18:07:30.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb0fb400640 1 -- 172.21.15.27:0/2449311979 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb1040823b0 con 0x561ff4e77320 2024-01-15T18:07:30.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb11c374000 10 client.5979 fetch_fsmap learned FSMap version 27 2024-01-15T18:07:30.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb11c374000 1 -- 172.21.15.27:0/2449311979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) v3 -- 0x561ff4c88190 con 0x561ff4e77320 2024-01-15T18:07:30.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb0fb400640 1 -- 172.21.15.27:0/2449311979 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561ff4c89470 con 0x561ff4e77320 2024-01-15T18:07:30.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb0fb400640 1 -- 172.21.15.27:0/2449311979 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fb104019020 con 0x561ff4e77320 2024-01-15T18:07:30.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb11c374000 10 client.5979 fetch_fsmap finished waiting for FSMap version 27 2024-01-15T18:07:30.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb11c374000 10 client.5979 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-15T18:07:30.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb11c374000 20 client.5979 populate_metadata read hostname 'smithi027' 2024-01-15T18:07:30.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb11c374000 1 --2- 172.21.15.27:0/2449311979 >> [v2:172.21.15.27:6833/1005016790,v1:172.21.15.27:6835/1005016790] conn(0x561ff4e90320 0x561ff4ed2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb11c374000 4 client.5979 mds_command: new command op to 5808 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-15T18:07:30.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb11c374000 1 -- 172.21.15.27:0/2449311979 --> [v2:172.21.15.27:6833/1005016790,v1:172.21.15.27:6835/1005016790] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561ff4c8cb50 con 0x561ff4e90320 2024-01-15T18:07:30.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb111400640 1 --2- 172.21.15.27:0/2449311979 >> [v2:172.21.15.27:6833/1005016790,v1:172.21.15.27:6835/1005016790] conn(0x561ff4e90320 0x561ff4ed2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb111400640 1 --2- 172.21.15.27:0/2449311979 >> [v2:172.21.15.27:6833/1005016790,v1:172.21.15.27:6835/1005016790] conn(0x561ff4e90320 0x561ff4ed2f50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22732692e209136 server_cookie=d7a1bb8b17bbc4da in_seq=0 out_seq=0 2024-01-15T18:07:30.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb0fb400640 10 client.5979 ms_handle_connect on v2:172.21.15.27:6833/1005016790 2024-01-15T18:07:30.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb0fb400640 1 -- 172.21.15.27:0/2449311979 <== mds.0 v2:172.21.15.27:6833/1005016790 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561ff4c8cb50 con 0x561ff4e90320 2024-01-15T18:07:30.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb0fb400640 10 client.5979 handle_command_reply: tid=0 2024-01-15T18:07:30.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb11c374000 1 client.5979 shutdown 2024-01-15T18:07:30.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb11c374000 1 -- 172.21.15.27:0/2449311979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4e77320 msgr2=0x561ff4e7bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:30.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb11c374000 1 --2- 172.21.15.27:0/2449311979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4e77320 0x561ff4e7bd40 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fb104054cc0 tx=0x7fb104059220 comp rx=0 tx=0).stop 2024-01-15T18:07:30.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb11c374000 1 -- 172.21.15.27:0/2449311979 shutdown_connections 2024-01-15T18:07:30.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb11c374000 1 --2- 172.21.15.27:0/2449311979 >> [v2:172.21.15.27:6833/1005016790,v1:172.21.15.27:6835/1005016790] conn(0x561ff4e90320 0x561ff4ed2f50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb11c374000 1 --2- 172.21.15.27:0/2449311979 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x561ff4ec15b0 0x561ff4e8f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb11c374000 1 --2- 172.21.15.27:0/2449311979 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561ff4ec0c90 0x561ff4e7c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb0fb400640 0 client.5979 ms_handle_reset on v2:172.21.15.27:6833/1005016790 2024-01-15T18:07:30.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb11c374000 1 --2- 172.21.15.27:0/2449311979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4e77320 0x561ff4e7bd40 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb11c374000 1 -- 172.21.15.27:0/2449311979 >> 172.21.15.27:0/2449311979 conn(0x561ff4e732d0 msgr2=0x561ff4e756c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:30.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb11c374000 1 -- 172.21.15.27:0/2449311979 shutdown_connections 2024-01-15T18:07:30.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb11c374000 1 -- 172.21.15.27:0/2449311979 wait complete. 2024-01-15T18:07:30.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.492+0000 7fb11c374000 20 client.5979 trim_cache size 0 max 16384 2024-01-15T18:07:30.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.508+0000 7fb11c374000 1 Processor -- start 2024-01-15T18:07:30.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.508+0000 7fb11c374000 1 -- start start 2024-01-15T18:07:30.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb11c374000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x561ff4e732d0 0x561ff4ebeb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb11c374000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561ff4e77320 0x561ff4ebf070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb11c374000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4ec0c90 0x561ff4e77700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb11c374000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ff4980940 con 0x561ff4ec0c90 2024-01-15T18:07:30.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb11c374000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ff4e8f8c0 con 0x561ff4e77320 2024-01-15T18:07:30.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb11c374000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x561ff4e7e890 con 0x561ff4e732d0 2024-01-15T18:07:30.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb112800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4ec0c90 0x561ff4e77700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb112800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4ec0c90 0x561ff4e77700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50872/0 (socket says 172.21.15.27:50872) 2024-01-15T18:07:30.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb112800640 1 -- 172.21.15.27:0/3156747166 learned_addr learned my addr 172.21.15.27:0/3156747166 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:30.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb111e00640 1 --2- 172.21.15.27:0/3156747166 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x561ff4e732d0 0x561ff4ebeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb111400640 1 --2- 172.21.15.27:0/3156747166 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561ff4e77320 0x561ff4ebf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb112800640 1 -- 172.21.15.27:0/3156747166 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x561ff4e732d0 msgr2=0x561ff4ebeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:30.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb112800640 1 --2- 172.21.15.27:0/3156747166 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x561ff4e732d0 0x561ff4ebeb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb112800640 1 -- 172.21.15.27:0/3156747166 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561ff4e77320 msgr2=0x561ff4ebf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:30.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb112800640 1 --2- 172.21.15.27:0/3156747166 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561ff4e77320 0x561ff4ebf070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb112800640 1 -- 172.21.15.27:0/3156747166 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ff4c6e490 con 0x561ff4ec0c90 2024-01-15T18:07:30.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb112800640 1 --2- 172.21.15.27:0/3156747166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4ec0c90 0x561ff4e77700 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fb10c003c20 tx=0x7fb10c005420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b9dc043b4fcbda3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:30.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb103400640 1 -- 172.21.15.27:0/3156747166 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c001bf0 con 0x561ff4ec0c90 2024-01-15T18:07:30.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb103400640 1 -- 172.21.15.27:0/3156747166 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10c001d50 con 0x561ff4ec0c90 2024-01-15T18:07:30.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb103400640 1 -- 172.21.15.27:0/3156747166 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c0020e0 con 0x561ff4ec0c90 2024-01-15T18:07:30.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb11c374000 1 -- 172.21.15.27:0/3156747166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4ec0c90 msgr2=0x561ff4e77700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:30.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb11c374000 1 --2- 172.21.15.27:0/3156747166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4ec0c90 0x561ff4e77700 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fb10c003c20 tx=0x7fb10c005420 comp rx=0 tx=0).stop 2024-01-15T18:07:30.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb11c374000 1 -- 172.21.15.27:0/3156747166 shutdown_connections 2024-01-15T18:07:30.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb11c374000 1 --2- 172.21.15.27:0/3156747166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4ec0c90 0x561ff4e77700 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb11c374000 1 --2- 172.21.15.27:0/3156747166 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561ff4e77320 0x561ff4ebf070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb11c374000 1 --2- 172.21.15.27:0/3156747166 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x561ff4e732d0 0x561ff4ebeb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb11c374000 1 -- 172.21.15.27:0/3156747166 >> 172.21.15.27:0/3156747166 conn(0x561ff4e7bf10 msgr2=0x561ff4e756c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:30.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb11c374000 1 -- 172.21.15.27:0/3156747166 shutdown_connections 2024-01-15T18:07:30.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb11c374000 1 -- 172.21.15.27:0/3156747166 wait complete. 2024-01-15T18:07:30.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb11c374000 1 -- start start 2024-01-15T18:07:30.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.512+0000 7fb11c374000 1 Processor -- start 2024-01-15T18:07:30.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x561ff4e732d0 0x561ff4ed3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4e77320 0x561ff4ed44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561ff4ec0c90 0x561ff4ee71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ff4ca1e30 con 0x561ff4e77320 2024-01-15T18:07:30.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ff4ccb7c0 con 0x561ff4ec0c90 2024-01-15T18:07:30.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x561ff4e72c70 con 0x561ff4e732d0 2024-01-15T18:07:30.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 10 client.0 _ll_register_callbacks cb 0x561ff4d15c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:07:30.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb111400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4e77320 0x561ff4ed44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb111400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4e77320 0x561ff4ed44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50878/0 (socket says 172.21.15.27:50878) 2024-01-15T18:07:30.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb111400640 1 -- 172.21.15.27:0/1975805449 learned_addr learned my addr 172.21.15.27:0/1975805449 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:30.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb111e00640 1 --2- 172.21.15.27:0/1975805449 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x561ff4e732d0 0x561ff4ed3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb112800640 1 --2- 172.21.15.27:0/1975805449 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561ff4ec0c90 0x561ff4ee71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb111400640 1 -- 172.21.15.27:0/1975805449 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x561ff4e732d0 msgr2=0x561ff4ed3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:30.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb111400640 1 --2- 172.21.15.27:0/1975805449 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x561ff4e732d0 0x561ff4ed3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb111400640 1 -- 172.21.15.27:0/1975805449 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561ff4ec0c90 msgr2=0x561ff4ee71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:30.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb111400640 1 --2- 172.21.15.27:0/1975805449 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561ff4ec0c90 0x561ff4ee71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb111400640 1 -- 172.21.15.27:0/1975805449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb10c00f040 con 0x561ff4e77320 2024-01-15T18:07:30.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb111400640 1 --2- 172.21.15.27:0/1975805449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4e77320 0x561ff4ed44e0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc009740 tx=0x7fb0fc01ab60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87ee9f6e66950c4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:30.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb0fb400640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-15T18:07:30.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb0fb400640 1 -- 172.21.15.27:0/1975805449 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc0ab970 con 0x561ff4e77320 2024-01-15T18:07:30.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb0fb400640 1 -- 172.21.15.27:0/1975805449 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fb0fc0abbf0 con 0x561ff4e77320 2024-01-15T18:07:30.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 1 -- 172.21.15.27:0/1975805449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561ff4e7b5c0 con 0x561ff4e77320 2024-01-15T18:07:30.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb0fb400640 1 -- 172.21.15.27:0/1975805449 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0fc0d6110 con 0x561ff4e77320 2024-01-15T18:07:30.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 10 client.5985 fetch_fsmap learned FSMap version 27 2024-01-15T18:07:30.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 1 -- 172.21.15.27:0/1975805449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) v3 -- 0x561ff49bfcc0 con 0x561ff4e77320 2024-01-15T18:07:30.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb0fb400640 1 -- 172.21.15.27:0/1975805449 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561ff4e7b5c0 con 0x561ff4e77320 2024-01-15T18:07:30.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb0fb400640 1 -- 172.21.15.27:0/1975805449 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 27) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fb0fc0d0040 con 0x561ff4e77320 2024-01-15T18:07:30.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 10 client.5985 fetch_fsmap finished waiting for FSMap version 27 2024-01-15T18:07:30.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 10 client.5985 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-15T18:07:30.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 20 client.5985 populate_metadata read hostname 'smithi027' 2024-01-15T18:07:30.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 1 --2- 172.21.15.27:0/1975805449 >> [v2:172.21.15.27:6833/1005016790,v1:172.21.15.27:6835/1005016790] conn(0x561ff4ec15b0 0x561ff4f32660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 4 client.5985 mds_command: new command op to 5808 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-15T18:07:30.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 1 -- 172.21.15.27:0/1975805449 --> [v2:172.21.15.27:6833/1005016790,v1:172.21.15.27:6835/1005016790] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561ff4cc6110 con 0x561ff4ec15b0 2024-01-15T18:07:30.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb111e00640 1 --2- 172.21.15.27:0/1975805449 >> [v2:172.21.15.27:6833/1005016790,v1:172.21.15.27:6835/1005016790] conn(0x561ff4ec15b0 0x561ff4f32660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb0fb400640 10 client.5985 ms_handle_connect on v2:172.21.15.27:6833/1005016790 2024-01-15T18:07:30.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb111e00640 1 --2- 172.21.15.27:0/1975805449 >> [v2:172.21.15.27:6833/1005016790,v1:172.21.15.27:6835/1005016790] conn(0x561ff4ec15b0 0x561ff4f32660 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41befba73bfc7d4f server_cookie=28fb76250625c308 in_seq=0 out_seq=0 2024-01-15T18:07:30.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb0fb400640 1 -- 172.21.15.27:0/1975805449 <== mds.0 v2:172.21.15.27:6833/1005016790 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x561ff4cc6110 con 0x561ff4ec15b0 2024-01-15T18:07:30.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb0fb400640 10 client.5985 handle_command_reply: tid=0 2024-01-15T18:07:30.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 1 client.5985 shutdown 2024-01-15T18:07:30.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 1 -- 172.21.15.27:0/1975805449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4e77320 msgr2=0x561ff4ed44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:30.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 1 --2- 172.21.15.27:0/1975805449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4e77320 0x561ff4ed44e0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc009740 tx=0x7fb0fc01ab60 comp rx=0 tx=0).stop 2024-01-15T18:07:30.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 1 -- 172.21.15.27:0/1975805449 shutdown_connections 2024-01-15T18:07:30.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 1 --2- 172.21.15.27:0/1975805449 >> [v2:172.21.15.27:6833/1005016790,v1:172.21.15.27:6835/1005016790] conn(0x561ff4ec15b0 0x561ff4f32660 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 1 --2- 172.21.15.27:0/1975805449 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x561ff4ec0c90 0x561ff4ee71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 1 --2- 172.21.15.27:0/1975805449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x561ff4e77320 0x561ff4ed44e0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 1 --2- 172.21.15.27:0/1975805449 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x561ff4e732d0 0x561ff4ed3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb0fb400640 0 client.5985 ms_handle_reset on v2:172.21.15.27:6833/1005016790 2024-01-15T18:07:30.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 1 -- 172.21.15.27:0/1975805449 >> 172.21.15.27:0/1975805449 conn(0x561ff4e7bf10 msgr2=0x561ff4e75f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:30.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 1 -- 172.21.15.27:0/1975805449 shutdown_connections 2024-01-15T18:07:30.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.516+0000 7fb11c374000 1 -- 172.21.15.27:0/1975805449 wait complete. 2024-01-15T18:07:30.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.520+0000 7fb11c374000 20 client.5985 trim_cache size 0 max 16384 2024-01-15T18:07:30.526 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:07:30.526 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:07:30.526 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-01-15T18:07:30.526 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 5808, 2024-01-15T18:07:30.526 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-01-15T18:07:30.526 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-01-15T18:07:30.526 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-01-15T18:07:30.526 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 10.458192459999999, 2024-01-15T18:07:30.527 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 27, 2024-01-15T18:07:30.527 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 45, 2024-01-15T18:07:30.527 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 45, 2024-01-15T18:07:30.527 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 15.328620001000001 2024-01-15T18:07:30.527 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:07:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.520+0000 7fb119000640 1 -- 172.21.15.27:0/2146983123 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb0ec066b30 msgr2=0x7fb0ec068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.520+0000 7fb119000640 1 --2- 172.21.15.27:0/2146983123 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb0ec066b30 0x7fb0ec068ff0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc0ad000 tx=0x7fb0fc0a6000 comp rx=0 tx=0).stop 2024-01-15T18:07:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.520+0000 7fb119000640 1 -- 172.21.15.27:0/2146983123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1140cde10 msgr2=0x7fb1140c4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.520+0000 7fb119000640 1 --2- 172.21.15.27:0/2146983123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1140cde10 0x7fb1140c4990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fb10400c550 tx=0x7fb10400ca10 comp rx=0 tx=0).stop 2024-01-15T18:07:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.520+0000 7fb119000640 1 -- 172.21.15.27:0/2146983123 shutdown_connections 2024-01-15T18:07:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.520+0000 7fb119000640 1 --2- 172.21.15.27:0/2146983123 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb0ec066b30 0x7fb0ec068ff0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.520+0000 7fb119000640 1 --2- 172.21.15.27:0/2146983123 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb11415d680 0x7fb11415c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.520+0000 7fb119000640 1 --2- 172.21.15.27:0/2146983123 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb11415d2a0 0x7fb11415c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.520+0000 7fb119000640 1 --2- 172.21.15.27:0/2146983123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1140cde10 0x7fb1140c4990 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.520+0000 7fb119000640 1 -- 172.21.15.27:0/2146983123 >> 172.21.15.27:0/2146983123 conn(0x7fb1140d0e80 msgr2=0x7fb11406fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.520+0000 7fb119000640 1 -- 172.21.15.27:0/2146983123 shutdown_connections 2024-01-15T18:07:30.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.520+0000 7fb119000640 1 -- 172.21.15.27:0/2146983123 wait complete. 2024-01-15T18:07:30.543 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:07:30.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.680+0000 7f8ed3800640 1 Processor -- start 2024-01-15T18:07:30.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.680+0000 7f8ed3800640 1 -- start start 2024-01-15T18:07:30.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.680+0000 7f8ed3800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f8ecc15d1d0 0x7f8ecc0d1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.680+0000 7f8ed3800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8ecc0ce790 0x7f8ecc0cec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.680+0000 7f8ed3800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8ecc0cfb00 0x7f8ecc0cff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.680+0000 7f8ed3800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ecc0d0490 con 0x7f8ecc0ce790 2024-01-15T18:07:30.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.680+0000 7f8ed3800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ecc0d3e00 con 0x7f8ecc15d1d0 2024-01-15T18:07:30.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.680+0000 7f8ed3800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc0d0d60 con 0x7f8ecc0cfb00 2024-01-15T18:07:30.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.680+0000 7f8ecbe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8ecc0ce790 0x7f8ecc0cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.680+0000 7f8ecbe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8ecc0ce790 0x7f8ecc0cec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50880/0 (socket says 172.21.15.27:50880) 2024-01-15T18:07:30.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.680+0000 7f8ecbe00640 1 -- 172.21.15.27:0/1400346538 learned_addr learned my addr 172.21.15.27:0/1400346538 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:30.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.680+0000 7f8ed1800640 1 --2- 172.21.15.27:0/1400346538 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8ecc0cfb00 0x7f8ecc0cff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.680+0000 7f8ed0e00640 1 --2- 172.21.15.27:0/1400346538 >> v2:172.21.15.93:3300/0 conn(0x7f8ecc15d1d0 0x7f8ecc0d1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.680+0000 7f8ecbe00640 1 -- 172.21.15.27:0/1400346538 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8ecc0cfb00 msgr2=0x7f8ecc0cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:30.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.680+0000 7f8ecbe00640 1 --2- 172.21.15.27:0/1400346538 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8ecc0cfb00 0x7f8ecc0cff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.680+0000 7f8ecbe00640 1 -- 172.21.15.27:0/1400346538 >> v2:172.21.15.93:3300/0 conn(0x7f8ecc15d1d0 msgr2=0x7f8ecc0d1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:30.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.680+0000 7f8ecbe00640 1 --2- 172.21.15.27:0/1400346538 >> v2:172.21.15.93:3300/0 conn(0x7f8ecc15d1d0 0x7f8ecc0d1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.680+0000 7f8ecbe00640 1 -- 172.21.15.27:0/1400346538 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ecc0d1040 con 0x7f8ecc0ce790 2024-01-15T18:07:30.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.684+0000 7f8ecbe00640 1 --2- 172.21.15.27:0/1400346538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ecc0ce790 0x7f8ecc0cec00 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc08d1f0 tx=0x7f8eb401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7879ac47965626fd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:30.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.684+0000 7f8ecb400640 1 -- 172.21.15.27:0/1400346538 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb40aa620 con 0x7f8ecc0ce790 2024-01-15T18:07:30.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.684+0000 7f8ecb400640 1 -- 172.21.15.27:0/1400346538 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb40aa780 con 0x7f8ecc0ce790 2024-01-15T18:07:30.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.684+0000 7f8ecb400640 1 -- 172.21.15.27:0/1400346538 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb40aab10 con 0x7f8ecc0ce790 2024-01-15T18:07:30.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.684+0000 7f8ed3800640 1 -- 172.21.15.27:0/1400346538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ecc0ce790 msgr2=0x7f8ecc0cec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:30.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.684+0000 7f8ed3800640 1 --2- 172.21.15.27:0/1400346538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ecc0ce790 0x7f8ecc0cec00 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc08d1f0 tx=0x7f8eb401c500 comp rx=0 tx=0).stop 2024-01-15T18:07:30.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.684+0000 7f8ed3800640 1 -- 172.21.15.27:0/1400346538 shutdown_connections 2024-01-15T18:07:30.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.684+0000 7f8ed3800640 1 --2- 172.21.15.27:0/1400346538 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8ecc0cfb00 0x7f8ecc0cff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.684+0000 7f8ed3800640 1 --2- 172.21.15.27:0/1400346538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ecc0ce790 0x7f8ecc0cec00 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.684+0000 7f8ed3800640 1 --2- 172.21.15.27:0/1400346538 >> v2:172.21.15.93:3300/0 conn(0x7f8ecc15d1d0 0x7f8ecc0d1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.684+0000 7f8ed3800640 1 -- 172.21.15.27:0/1400346538 >> 172.21.15.27:0/1400346538 conn(0x7f8ecc09c440 msgr2=0x7f8ecc0d30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:30.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.684+0000 7f8ed3800640 1 -- 172.21.15.27:0/1400346538 shutdown_connections 2024-01-15T18:07:30.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.684+0000 7f8ed3800640 1 -- 172.21.15.27:0/1400346538 wait complete. 2024-01-15T18:07:30.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.688+0000 7f8ed3800640 1 Processor -- start 2024-01-15T18:07:30.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.688+0000 7f8ed3800640 1 -- start start 2024-01-15T18:07:30.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.688+0000 7f8ed3800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ecc0ce790 0x7f8ecc1517f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.688+0000 7f8ed3800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8ecc0cfb00 0x7f8ecc158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.688+0000 7f8ed3800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8ecc15d1d0 0x7f8ecc158980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.688+0000 7f8ed3800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc0c87c0 con 0x7f8ecc0ce790 2024-01-15T18:07:30.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.688+0000 7f8ed3800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc1525a0 con 0x7f8ecc0cfb00 2024-01-15T18:07:30.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.688+0000 7f8ed3800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc152880 con 0x7f8ecc15d1d0 2024-01-15T18:07:30.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.688+0000 7f8ed0e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ecc0ce790 0x7f8ecc1517f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.688+0000 7f8ed0e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ecc0ce790 0x7f8ecc1517f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50886/0 (socket says 172.21.15.27:50886) 2024-01-15T18:07:30.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.688+0000 7f8ed0e00640 1 -- 172.21.15.27:0/1114498265 learned_addr learned my addr 172.21.15.27:0/1114498265 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:30.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.688+0000 7f8ecbe00640 1 --2- 172.21.15.27:0/1114498265 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8ecc0cfb00 0x7f8ecc158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.688+0000 7f8ed1800640 1 --2- 172.21.15.27:0/1114498265 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8ecc15d1d0 0x7f8ecc158980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.688+0000 7f8ed0e00640 1 -- 172.21.15.27:0/1114498265 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8ecc15d1d0 msgr2=0x7f8ecc158980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:30.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.688+0000 7f8ed0e00640 1 --2- 172.21.15.27:0/1114498265 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8ecc15d1d0 0x7f8ecc158980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.688+0000 7f8ed0e00640 1 -- 172.21.15.27:0/1114498265 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8ecc0cfb00 msgr2=0x7f8ecc158440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:30.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.688+0000 7f8ed0e00640 1 --2- 172.21.15.27:0/1114498265 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8ecc0cfb00 0x7f8ecc158440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.688+0000 7f8ed0e00640 1 -- 172.21.15.27:0/1114498265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb40094d0 con 0x7f8ecc0ce790 2024-01-15T18:07:30.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.688+0000 7f8ecbe00640 1 --2- 172.21.15.27:0/1114498265 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8ecc0cfb00 0x7f8ecc158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:07:30.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.688+0000 7f8ed0e00640 1 --2- 172.21.15.27:0/1114498265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ecc0ce790 0x7f8ecc1517f0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc00d5f0 tx=0x7f8ebc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:30.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.688+0000 7f8ec8c00640 1 -- 172.21.15.27:0/1114498265 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc0125f0 con 0x7f8ecc0ce790 2024-01-15T18:07:30.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.688+0000 7f8ec8c00640 1 -- 172.21.15.27:0/1114498265 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ebc012750 con 0x7f8ecc0ce790 2024-01-15T18:07:30.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.688+0000 7f8ed3800640 1 -- 172.21.15.27:0/1114498265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ecc152b60 con 0x7f8ecc0ce790 2024-01-15T18:07:30.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.688+0000 7f8ec8c00640 1 -- 172.21.15.27:0/1114498265 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc012ae0 con 0x7f8ecc0ce790 2024-01-15T18:07:30.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.688+0000 7f8ed3800640 1 -- 172.21.15.27:0/1114498265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ecc153120 con 0x7f8ecc0ce790 2024-01-15T18:07:30.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.692+0000 7f8ed3800640 1 -- 172.21.15.27:0/1114498265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ecc067a10 con 0x7f8ecc0ce790 2024-01-15T18:07:30.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.692+0000 7f8ec8c00640 1 -- 172.21.15.27:0/1114498265 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f8ebc012c40 con 0x7f8ecc0ce790 2024-01-15T18:07:30.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.692+0000 7f8ec8c00640 1 --2- 172.21.15.27:0/1114498265 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8ea4066b30 0x7f8ea4068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:30.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.692+0000 7f8ec8c00640 1 -- 172.21.15.27:0/1114498265 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f8ebc084cb0 con 0x7f8ecc0ce790 2024-01-15T18:07:30.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.692+0000 7f8ecbe00640 1 --2- 172.21.15.27:0/1114498265 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8ea4066b30 0x7f8ea4068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:30.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.696+0000 7f8ecbe00640 1 --2- 172.21.15.27:0/1114498265 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8ea4066b30 0x7f8ea4068ff0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8eb40ae000 tx=0x7f8eb40b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:30.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.696+0000 7f8ec8c00640 1 -- 172.21.15.27:0/1114498265 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8ebc051b80 con 0x7f8ecc0ce790 2024-01-15T18:07:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.892+0000 7f8ed3800640 1 -- 172.21.15.27:0/1114498265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8ecc04e850 con 0x7f8ecc0ce790 2024-01-15T18:07:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.892+0000 7f8ec8c00640 1 -- 172.21.15.27:0/1114498265 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8ebc007380 con 0x7f8ecc0ce790 2024-01-15T18:07:30.896 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:07:30.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.892+0000 7f8ed3800640 1 -- 172.21.15.27:0/1114498265 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8ea4066b30 msgr2=0x7f8ea4068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.892+0000 7f8ed3800640 1 --2- 172.21.15.27:0/1114498265 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8ea4066b30 0x7f8ea4068ff0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8eb40ae000 tx=0x7f8eb40b5040 comp rx=0 tx=0).stop 2024-01-15T18:07:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.892+0000 7f8ed3800640 1 -- 172.21.15.27:0/1114498265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ecc0ce790 msgr2=0x7f8ecc1517f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.892+0000 7f8ed3800640 1 --2- 172.21.15.27:0/1114498265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ecc0ce790 0x7f8ecc1517f0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc00d5f0 tx=0x7f8ebc00dab0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.892+0000 7f8ed3800640 1 -- 172.21.15.27:0/1114498265 shutdown_connections 2024-01-15T18:07:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.892+0000 7f8ed3800640 1 --2- 172.21.15.27:0/1114498265 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8ea4066b30 0x7f8ea4068ff0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.892+0000 7f8ed3800640 1 --2- 172.21.15.27:0/1114498265 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8ecc15d1d0 0x7f8ecc158980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.892+0000 7f8ed3800640 1 --2- 172.21.15.27:0/1114498265 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8ecc0cfb00 0x7f8ecc158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.892+0000 7f8ed3800640 1 --2- 172.21.15.27:0/1114498265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8ecc0ce790 0x7f8ecc1517f0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.892+0000 7f8ed3800640 1 -- 172.21.15.27:0/1114498265 >> 172.21.15.27:0/1114498265 conn(0x7f8ecc09c440 msgr2=0x7f8ecc0b6ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.892+0000 7f8ed3800640 1 -- 172.21.15.27:0/1114498265 shutdown_connections 2024-01-15T18:07:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:30.892+0000 7f8ed3800640 1 -- 172.21.15.27:0/1114498265 wait complete. 2024-01-15T18:07:30.917 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:07:30.917 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:07:30.917 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-01-15T18:07:30.918 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-15T18:07:30.918 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-15T18:07:30.918 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:07:30.918 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:07:30.918 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:07:30.918 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:07:30.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:30.918 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-01-15T18:07:30.927 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:07:30.927 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:07:30.927 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:07:30.927 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:07:30.927 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-15T18:07:30.927 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:07:30.927 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:07:30.928 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:07:30.928 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-01-15T18:07:30.928 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:07:30.928 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:07:30.928 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-01-15T18:07:30.928 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:07:30.928 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-01-15T18:07:30.928 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-01-15T18:07:30.928 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-01-15T18:07:30.928 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42308sec preferred_lft 42308sec 2024-01-15T18:07:30.928 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-01-15T18:07:30.928 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:07:30.928 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:07:30.928 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-01-15T18:07:30.929 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-01-15T18:07:30.929 INFO:teuthology.orchestra.run.smithi027.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-01-15T18:07:30.929 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 02:42:65:20:14:72 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:07:30.929 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-01-15T18:07:30.929 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:07:30.929 INFO:teuthology.orchestra.run.smithi027.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:07:30.929 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-01-15T18:07:30.929 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:07:30.929 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:07:30.929 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-01-15T18:07:30.929 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:07:30.929 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:07:30.929 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:07:30.929 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-01-15T18:07:30.929 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:07:30.930 INFO:teuthology.orchestra.run.smithi027.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:07:30.930 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:07:30.930 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-01-15T18:07:30.930 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:07:30.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:30.931 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:07:30.992 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-01-15T18:07:30.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:30.994 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-01-15T18:07:31.051 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-01-15T18:07:31.051 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:07:31.051 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-01-15T18:07:31.051 INFO:teuthology.orchestra.run.smithi027.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-01-15T18:07:31.051 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-01-15T18:07:31.051 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:07:31.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:31.052 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-15T18:07:31.052 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-15T18:07:31.052 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-15T18:07:31.052 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-15T18:07:31.052 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-15T18:07:31.052 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-15T18:07:31.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:31.144 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-15T18:07:31.153 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-15T18:07:31.153 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-15T18:07:31.153 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:07:31.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:31.153 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-01-15T18:07:31.218 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-15T18:07:31.218 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:07:31.219 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-15T18:07:31.283 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:07:31.283 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:07:31.284 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:07:31.334 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-01-15T18:07:31.398 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:07:31.451 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-01-15T18:07:31.452 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-01-15T18:07:31.452 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-01-15T18:07:31.452 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-01-15T18:07:31.452 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-01-15T18:07:31.452 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-01-15T18:07:31.452 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-01-15T18:07:31.452 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-01-15T18:07:31.452 INFO:teuthology.orchestra.run.smithi027.stdout:/run/lock 2024-01-15T18:07:31.452 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-01-15T18:07:31.452 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-01-15T18:07:31.452 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-01-15T18:07:31.452 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:07:31.452 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-01-15T18:07:31.453 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-01-15T18:07:31.453 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-01-15T18:07:31.453 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-01-15T18:07:31.453 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/fs/nfsd 2024-01-15T18:07:31.453 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-01-15T18:07:31.453 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-01-15T18:07:31.453 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:07:31.453 INFO:teuthology.orchestra.run.smithi027.stdout:/run/rpc_pipefs 2024-01-15T18:07:31.453 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-01-15T18:07:31.453 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-01-15T18:07:31.453 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:07:31.453 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:07:31.453 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:07:31.454 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:07:31.454 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-01-15T18:07:31.454 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:07:31.454 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:07:31.454 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:07:31.454 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:07:31.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:31.455 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-01-15T18:07:31.506 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:07:31.506 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-15T18:07:31.553 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-01-15T18:07:31.574 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:07:31.623 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-01-15T18:07:31.624 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-01-15T18:07:31.624 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-01-15T18:07:31.624 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-01-15T18:07:31.624 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-01-15T18:07:31.624 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-01-15T18:07:31.624 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-01-15T18:07:31.624 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-01-15T18:07:31.624 INFO:teuthology.orchestra.run.smithi027.stdout:/run/lock 2024-01-15T18:07:31.624 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-01-15T18:07:31.624 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-01-15T18:07:31.624 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-01-15T18:07:31.625 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:07:31.625 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-01-15T18:07:31.625 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-01-15T18:07:31.625 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-01-15T18:07:31.625 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-01-15T18:07:31.625 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/fs/nfsd 2024-01-15T18:07:31.625 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-01-15T18:07:31.625 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-01-15T18:07:31.625 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:07:31.625 INFO:teuthology.orchestra.run.smithi027.stdout:/run/rpc_pipefs 2024-01-15T18:07:31.625 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-01-15T18:07:31.625 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-01-15T18:07:31.625 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:07:31.625 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:07:31.626 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:07:31.626 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:07:31.626 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-01-15T18:07:31.626 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:07:31.626 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:07:31.626 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:07:31.626 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:07:31.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:31.627 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-01-15T18:07:31.649 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:2024-01-15T18:07:31.644+0000 7fbfc85d0580 -1 init, newargv = 0x5596f85eac40 newargc=15 2024-01-15T18:07:31.649 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[38897]: starting ceph client 2024-01-15T18:07:31.657 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[38897]: starting fuse 2024-01-15T18:07:31.657 INFO:teuthology.orchestra.run.smithi027.stdout:43 2024-01-15T18:07:31.658 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-01-15T18:07:31.658 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:07:31.658 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-01-15T18:07:31.658 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-01-15T18:07:31.658 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-01-15T18:07:31.658 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-01-15T18:07:31.658 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:07:31.658 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-01-15T18:07:31.658 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-15T18:07:31.658 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-01-15T18:07:31.658 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-15T18:07:31.659 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:07:31.659 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-01-15T18:07:31.659 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-01-15T18:07:31.659 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-01-15T18:07:31.659 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-01-15T18:07:31.659 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:07:31.659 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-01-15T18:07:31.659 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:07:31.659 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:07:31.659 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:07:31.659 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-01-15T18:07:31.659 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-01-15T18:07:31.659 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-01-15T18:07:31.659 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:07:31.659 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:07:31.659 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.0")) 2024-01-15T18:07:31.660 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-01-15T18:07:31.780 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.0.38897.asok 2024-01-15T18:07:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:07:31.791 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.38897.asok 2024-01-15T18:07:31.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:31.791 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.38897.asok status 2024-01-15T18:07:31.965 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:07:31.966 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-01-15T18:07:31.966 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:07:31.966 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:07:31.966 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "0", 2024-01-15T18:07:31.966 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-01-15T18:07:31.966 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-15T18:07:31.966 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "38897", 2024-01-15T18:07:31.966 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-01-15T18:07:31.966 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:07:31.966 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-01-15T18:07:31.966 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-01-15T18:07:31.966 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 5994, 2024-01-15T18:07:31.966 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-01-15T18:07:31.966 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-01-15T18:07:31.967 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-01-15T18:07:31.967 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 5994 2024-01-15T18:07:31.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:07:31.967 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-01-15T18:07:31.967 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-01-15T18:07:31.967 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:07:31.967 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4105098866 2024-01-15T18:07:31.967 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:31.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:07:31.967 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-01-15T18:07:31.967 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-01-15T18:07:31.967 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:07:31.967 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 4105098866 2024-01-15T18:07:31.967 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:07:31.968 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.5994 192.168.0.1:0/4105098866", 2024-01-15T18:07:31.968 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.1:0/4105098866", 2024-01-15T18:07:31.968 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-01-15T18:07:31.968 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 27, 2024-01-15T18:07:31.968 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 45, 2024-01-15T18:07:31.968 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:07:31.968 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-01-15T18:07:31.968 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-01-15T18:07:31.968 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:07:31.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:31.982 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-01-15T18:07:31.982 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-15T18:07:31.991 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-01-15T18:07:31.992 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-15T18:07:31.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:31.992 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:07:32.055 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:07:32.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.240+0000 7f7fb1200640 1 Processor -- start 2024-01-15T18:07:32.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.240+0000 7f7fb1200640 1 -- start start 2024-01-15T18:07:32.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.240+0000 7f7fb1200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7fac15e7d0 0x7f7fac0cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:32.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.240+0000 7f7fb1200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f7fac0cedc0 0x7f7fac0cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:32.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.240+0000 7f7fb1200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fac0b38b0 con 0x7f7fac0ce2e0 2024-01-15T18:07:32.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.240+0000 7f7fb1200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fac0cd550 con 0x7f7fac15e7d0 2024-01-15T18:07:32.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.240+0000 7f7fb1200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fac0cd270 con 0x7f7fac0cedc0 2024-01-15T18:07:32.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.240+0000 7f7fa9e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7fac15e7d0 0x7f7fac0cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:32.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.240+0000 7f7fa9e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7fac15e7d0 0x7f7fac0cf2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50900/0 (socket says 172.21.15.27:50900) 2024-01-15T18:07:32.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.240+0000 7f7fa9e00640 1 -- 172.21.15.27:0/1050737961 learned_addr learned my addr 172.21.15.27:0/1050737961 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:32.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.240+0000 7f7fa9400640 1 --2- 172.21.15.27:0/1050737961 >> v2:172.21.15.93:3300/0 conn(0x7f7fac0cedc0 0x7f7fac0cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:32.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.240+0000 7f7fa9e00640 1 -- 172.21.15.27:0/1050737961 >> v2:172.21.15.93:3300/0 conn(0x7f7fac0cedc0 msgr2=0x7f7fac0cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:32.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.240+0000 7f7fa9e00640 1 --2- 172.21.15.27:0/1050737961 >> v2:172.21.15.93:3300/0 conn(0x7f7fac0cedc0 0x7f7fac0cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:32.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.240+0000 7f7fa9e00640 1 -- 172.21.15.27:0/1050737961 >> v1:172.21.15.93:6789/0 conn(0x7f7fac0ce2e0 legacy=0x7f7fac0cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:32.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.240+0000 7f7fa9e00640 1 -- 172.21.15.27:0/1050737961 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fac0b35d0 con 0x7f7fac15e7d0 2024-01-15T18:07:32.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.244+0000 7f7fa9e00640 1 --2- 172.21.15.27:0/1050737961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fac15e7d0 0x7f7fac0cf2a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0718a0 tx=0x7f7f9c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd4c7ef35d161c40 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:32.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.244+0000 7f7fa8a00640 1 -- 172.21.15.27:0/1050737961 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c0ad660 con 0x7f7fac15e7d0 2024-01-15T18:07:32.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.244+0000 7f7fa8a00640 1 -- 172.21.15.27:0/1050737961 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f9c0ad7c0 con 0x7f7fac15e7d0 2024-01-15T18:07:32.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.244+0000 7f7fa8a00640 1 -- 172.21.15.27:0/1050737961 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c0adb50 con 0x7f7fac15e7d0 2024-01-15T18:07:32.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.244+0000 7f7fb1200640 1 -- 172.21.15.27:0/1050737961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fac15e7d0 msgr2=0x7f7fac0cf2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:32.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.244+0000 7f7fb1200640 1 --2- 172.21.15.27:0/1050737961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fac15e7d0 0x7f7fac0cf2a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0718a0 tx=0x7f7f9c01c470 comp rx=0 tx=0).stop 2024-01-15T18:07:32.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.244+0000 7f7fb1200640 1 -- 172.21.15.27:0/1050737961 shutdown_connections 2024-01-15T18:07:32.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.244+0000 7f7fb1200640 1 --2- 172.21.15.27:0/1050737961 >> v2:172.21.15.93:3300/0 conn(0x7f7fac0cedc0 0x7f7fac0cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:32.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.244+0000 7f7fb1200640 1 --2- 172.21.15.27:0/1050737961 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fac15e7d0 0x7f7fac0cf2a0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:32.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.244+0000 7f7fb1200640 1 -- 172.21.15.27:0/1050737961 >> 172.21.15.27:0/1050737961 conn(0x7f7fac0b2a10 msgr2=0x7f7fac0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:32.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.244+0000 7f7fb1200640 1 -- 172.21.15.27:0/1050737961 shutdown_connections 2024-01-15T18:07:32.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.244+0000 7f7fb1200640 1 -- 172.21.15.27:0/1050737961 wait complete. 2024-01-15T18:07:32.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.248+0000 7f7fb1200640 1 Processor -- start 2024-01-15T18:07:32.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.248+0000 7f7fb1200640 1 -- start start 2024-01-15T18:07:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.248+0000 7f7fb1200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7fac0ce2e0 0x7f7fac15a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.248+0000 7f7fb1200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7fac0cedc0 0x7f7fac0ca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.248+0000 7f7fb1200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fac15e7d0 0x7f7fac0b5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.248+0000 7f7fb1200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac0af3d0 con 0x7f7fac15e7d0 2024-01-15T18:07:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.248+0000 7f7fb1200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac0aeda0 con 0x7f7fac0ce2e0 2024-01-15T18:07:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.248+0000 7f7fb1200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac0b3ad0 con 0x7f7fac0cedc0 2024-01-15T18:07:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.248+0000 7f7faa800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fac15e7d0 0x7f7fac0b5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:32.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.248+0000 7f7fa9e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7fac0ce2e0 0x7f7fac15a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:32.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.248+0000 7f7faa800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fac15e7d0 0x7f7fac0b5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50916/0 (socket says 172.21.15.27:50916) 2024-01-15T18:07:32.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.248+0000 7f7faa800640 1 -- 172.21.15.27:0/4263791738 learned_addr learned my addr 172.21.15.27:0/4263791738 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:32.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.248+0000 7f7fa9400640 1 --2- 172.21.15.27:0/4263791738 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7fac0cedc0 0x7f7fac0ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:32.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.248+0000 7f7faa800640 1 -- 172.21.15.27:0/4263791738 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7fac0cedc0 msgr2=0x7f7fac0ca7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:32.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.248+0000 7f7faa800640 1 --2- 172.21.15.27:0/4263791738 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7fac0cedc0 0x7f7fac0ca7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:32.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.248+0000 7f7faa800640 1 -- 172.21.15.27:0/4263791738 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7fac0ce2e0 msgr2=0x7f7fac15a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:32.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.248+0000 7f7faa800640 1 --2- 172.21.15.27:0/4263791738 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7fac0ce2e0 0x7f7fac15a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:32.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.248+0000 7f7faa800640 1 -- 172.21.15.27:0/4263791738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9c0094d0 con 0x7f7fac15e7d0 2024-01-15T18:07:32.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.248+0000 7f7faa800640 1 --2- 172.21.15.27:0/4263791738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fac15e7d0 0x7f7fac0b5610 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400c730 tx=0x7f7fa400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:32.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.248+0000 7f7fa2000640 1 -- 172.21.15.27:0/4263791738 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4003b80 con 0x7f7fac15e7d0 2024-01-15T18:07:32.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.248+0000 7f7fa2000640 1 -- 172.21.15.27:0/4263791738 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa4003ce0 con 0x7f7fac15e7d0 2024-01-15T18:07:32.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.248+0000 7f7fb1200640 1 -- 172.21.15.27:0/4263791738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fac0b3db0 con 0x7f7fac15e7d0 2024-01-15T18:07:32.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.248+0000 7f7fa2000640 1 -- 172.21.15.27:0/4263791738 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa400aa30 con 0x7f7fac15e7d0 2024-01-15T18:07:32.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.248+0000 7f7fb1200640 1 -- 172.21.15.27:0/4263791738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fac15acb0 con 0x7f7fac15e7d0 2024-01-15T18:07:32.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.248+0000 7f7fb1200640 1 -- 172.21.15.27:0/4263791738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fac0af580 con 0x7f7fac15e7d0 2024-01-15T18:07:32.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.252+0000 7f7fa2000640 1 -- 172.21.15.27:0/4263791738 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f7fa400ab90 con 0x7f7fac15e7d0 2024-01-15T18:07:32.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.252+0000 7f7fa2000640 1 --2- 172.21.15.27:0/4263791738 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7f84066b30 0x7f7f84068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:32.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.252+0000 7f7fa2000640 1 -- 172.21.15.27:0/4263791738 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f7fa40845b0 con 0x7f7fac15e7d0 2024-01-15T18:07:32.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.252+0000 7f7fa9e00640 1 --2- 172.21.15.27:0/4263791738 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7f84066b30 0x7f7f84068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:32.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.252+0000 7f7fa9e00640 1 --2- 172.21.15.27:0/4263791738 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7f84066b30 0x7f7f84068ff0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c0af000 tx=0x7f7f9c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:32.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.256+0000 7f7fa2000640 1 -- 172.21.15.27:0/4263791738 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7fac0af580 con 0x7f7fac15e7d0 2024-01-15T18:07:32.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.452+0000 7f7fb1200640 1 -- 172.21.15.27:0/4263791738 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7fac04e850 con 0x7f7fac15e7d0 2024-01-15T18:07:32.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.452+0000 7f7fa2000640 1 -- 172.21.15.27:0/4263791738 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7fa4051470 con 0x7f7fac15e7d0 2024-01-15T18:07:32.455 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:07:32.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.452+0000 7f7fb1200640 1 -- 172.21.15.27:0/4263791738 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7f84066b30 msgr2=0x7f7f84068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:32.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.452+0000 7f7fb1200640 1 --2- 172.21.15.27:0/4263791738 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7f84066b30 0x7f7f84068ff0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c0af000 tx=0x7f7f9c008040 comp rx=0 tx=0).stop 2024-01-15T18:07:32.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.452+0000 7f7fb1200640 1 -- 172.21.15.27:0/4263791738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fac15e7d0 msgr2=0x7f7fac0b5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:32.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.452+0000 7f7fb1200640 1 --2- 172.21.15.27:0/4263791738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fac15e7d0 0x7f7fac0b5610 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400c730 tx=0x7f7fa400cbf0 comp rx=0 tx=0).stop 2024-01-15T18:07:32.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.452+0000 7f7fb1200640 1 -- 172.21.15.27:0/4263791738 shutdown_connections 2024-01-15T18:07:32.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.452+0000 7f7fb1200640 1 --2- 172.21.15.27:0/4263791738 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7f84066b30 0x7f7f84068ff0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:32.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.452+0000 7f7fb1200640 1 --2- 172.21.15.27:0/4263791738 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fac15e7d0 0x7f7fac0b5610 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:32.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.452+0000 7f7fb1200640 1 --2- 172.21.15.27:0/4263791738 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7fac0cedc0 0x7f7fac0ca7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:32.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.452+0000 7f7fb1200640 1 --2- 172.21.15.27:0/4263791738 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7fac0ce2e0 0x7f7fac15a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:32.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.452+0000 7f7fb1200640 1 -- 172.21.15.27:0/4263791738 >> 172.21.15.27:0/4263791738 conn(0x7f7fac0b2a10 msgr2=0x7f7fac06b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:32.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.452+0000 7f7fb1200640 1 -- 172.21.15.27:0/4263791738 shutdown_connections 2024-01-15T18:07:32.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:32.452+0000 7f7fb1200640 1 -- 172.21.15.27:0/4263791738 wait complete. 2024-01-15T18:07:32.479 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:07:32.479 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:07:32.479 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi093.front.sepia.ceph.com 2024-01-15T18:07:32.479 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-15T18:07:32.480 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-15T18:07:32.480 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:07:32.480 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:07:32.480 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:07:32.480 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:07:32.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:32.480 DEBUG:teuthology.orchestra.run.smithi093:> ip addr 2024-01-15T18:07:32.490 INFO:teuthology.orchestra.run.smithi093.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:07:32.490 INFO:teuthology.orchestra.run.smithi093.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:07:32.490 INFO:teuthology.orchestra.run.smithi093.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:07:32.490 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:07:32.490 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 ::1/128 scope host 2024-01-15T18:07:32.490 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:07:32.490 INFO:teuthology.orchestra.run.smithi093.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:07:32.490 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:d8:da:1c brd ff:ff:ff:ff:ff:ff 2024-01-15T18:07:32.490 INFO:teuthology.orchestra.run.smithi093.stdout: altname enp7s0f0 2024-01-15T18:07:32.490 INFO:teuthology.orchestra.run.smithi093.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:07:32.490 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:d8:da:1d brd ff:ff:ff:ff:ff:ff 2024-01-15T18:07:32.491 INFO:teuthology.orchestra.run.smithi093.stdout: altname enp7s0f1 2024-01-15T18:07:32.491 INFO:teuthology.orchestra.run.smithi093.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:07:32.491 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:8f:d6:34 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:07:32.491 INFO:teuthology.orchestra.run.smithi093.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:07:32.491 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:8f:d6:35 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:07:32.491 INFO:teuthology.orchestra.run.smithi093.stdout: inet 172.21.15.93/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-01-15T18:07:32.491 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft 42286sec preferred_lft 42286sec 2024-01-15T18:07:32.491 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::ec4:7aff:fe8f:d635/64 scope link 2024-01-15T18:07:32.491 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:07:32.491 INFO:teuthology.orchestra.run.smithi093.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-01-15T18:07:32.491 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 02:42:2a:61:fc:7d brd ff:ff:ff:ff:ff:ff 2024-01-15T18:07:32.491 INFO:teuthology.orchestra.run.smithi093.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-01-15T18:07:32.491 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:07:32.491 INFO:teuthology.orchestra.run.smithi093.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:07:32.491 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-01-15T18:07:32.492 INFO:teuthology.orchestra.run.smithi093.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:07:32.492 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:07:32.492 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-01-15T18:07:32.492 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:07:32.492 INFO:teuthology.orchestra.run.smithi093.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:07:32.492 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:07:32.492 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-01-15T18:07:32.492 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:07:32.492 INFO:teuthology.orchestra.run.smithi093.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:07:32.492 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:07:32.492 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-01-15T18:07:32.492 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:07:32.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:32.493 DEBUG:teuthology.orchestra.run.smithi093:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:07:32.553 INFO:teuthology.orchestra.run.smithi093.stdout:1 2024-01-15T18:07:32.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:32.556 DEBUG:teuthology.orchestra.run.smithi093:> route 2024-01-15T18:07:32.609 INFO:teuthology.orchestra.run.smithi093.stdout:Kernel IP routing table 2024-01-15T18:07:32.609 INFO:teuthology.orchestra.run.smithi093.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:07:32.609 INFO:teuthology.orchestra.run.smithi093.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-01-15T18:07:32.609 INFO:teuthology.orchestra.run.smithi093.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T18:07:32.609 INFO:teuthology.orchestra.run.smithi093.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-01-15T18:07:32.609 INFO:teuthology.orchestra.run.smithi093.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T18:07:32.610 INFO:teuthology.orchestra.run.smithi093.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:07:32.610 INFO:teuthology.orchestra.run.smithi093.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:07:32.610 INFO:teuthology.orchestra.run.smithi093.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:07:32.610 INFO:teuthology.orchestra.run.smithi093.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:07:32.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:32.610 DEBUG:teuthology.orchestra.run.smithi093:> (cd / && exec bash -c ' 2024-01-15T18:07:32.610 DEBUG:teuthology.orchestra.run.smithi093:> set -e 2024-01-15T18:07:32.610 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T18:07:32.610 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T18:07:32.610 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T18:07:32.610 DEBUG:teuthology.orchestra.run.smithi093:> ') 2024-01-15T18:07:32.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:32.702 DEBUG:teuthology.orchestra.run.smithi093:> ip netns list 2024-01-15T18:07:32.711 INFO:teuthology.orchestra.run.smithi093.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-15T18:07:32.711 INFO:teuthology.orchestra.run.smithi093.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-15T18:07:32.711 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:07:32.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:32.711 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip link set brx.0 up 2024-01-15T18:07:32.773 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-15T18:07:32.774 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:07:32.774 DEBUG:teuthology.orchestra.run.smithi093:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-15T18:07:32.831 INFO:teuthology.orchestra.run.smithi093.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:07:32.831 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:07:32.831 DEBUG:teuthology.orchestra.run.smithi093:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:07:32.882 DEBUG:teuthology.orchestra.run.smithi093:> sudo modprobe fuse 2024-01-15T18:07:32.946 DEBUG:teuthology.orchestra.run.smithi093:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:07:32.999 INFO:teuthology.orchestra.run.smithi093.stdout:/sys 2024-01-15T18:07:33.000 INFO:teuthology.orchestra.run.smithi093.stdout:/proc 2024-01-15T18:07:33.000 INFO:teuthology.orchestra.run.smithi093.stdout:/dev 2024-01-15T18:07:33.000 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/pts 2024-01-15T18:07:33.000 INFO:teuthology.orchestra.run.smithi093.stdout:/run 2024-01-15T18:07:33.000 INFO:teuthology.orchestra.run.smithi093.stdout:/ 2024-01-15T18:07:33.000 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/security 2024-01-15T18:07:33.000 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/shm 2024-01-15T18:07:33.000 INFO:teuthology.orchestra.run.smithi093.stdout:/run/lock 2024-01-15T18:07:33.000 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/cgroup 2024-01-15T18:07:33.000 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/pstore 2024-01-15T18:07:33.000 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/bpf 2024-01-15T18:07:33.000 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:07:33.000 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/hugepages 2024-01-15T18:07:33.001 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/mqueue 2024-01-15T18:07:33.001 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/debug 2024-01-15T18:07:33.001 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/tracing 2024-01-15T18:07:33.001 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/fs/nfsd 2024-01-15T18:07:33.001 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/fuse/connections 2024-01-15T18:07:33.001 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/config 2024-01-15T18:07:33.001 INFO:teuthology.orchestra.run.smithi093.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:07:33.001 INFO:teuthology.orchestra.run.smithi093.stdout:/run/rpc_pipefs 2024-01-15T18:07:33.001 INFO:teuthology.orchestra.run.smithi093.stdout:/run/user/1000 2024-01-15T18:07:33.001 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph 2024-01-15T18:07:33.001 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:07:33.001 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:07:33.001 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:07:33.001 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:07:33.002 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns 2024-01-15T18:07:33.002 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:07:33.002 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:07:33.002 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:07:33.002 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:07:33.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:33.003 DEBUG:teuthology.orchestra.run.smithi093:> ls /sys/fs/fuse/connections 2024-01-15T18:07:33.054 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:07:33.054 DEBUG:teuthology.orchestra.run.smithi093:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-15T18:07:33.101 DEBUG:teuthology.orchestra.run.smithi093:> sudo modprobe fuse 2024-01-15T18:07:33.122 DEBUG:teuthology.orchestra.run.smithi093:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:07:33.179 INFO:teuthology.orchestra.run.smithi093.stdout:/sys 2024-01-15T18:07:33.179 INFO:teuthology.orchestra.run.smithi093.stdout:/proc 2024-01-15T18:07:33.179 INFO:teuthology.orchestra.run.smithi093.stdout:/dev 2024-01-15T18:07:33.179 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/pts 2024-01-15T18:07:33.179 INFO:teuthology.orchestra.run.smithi093.stdout:/run 2024-01-15T18:07:33.179 INFO:teuthology.orchestra.run.smithi093.stdout:/ 2024-01-15T18:07:33.179 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/security 2024-01-15T18:07:33.180 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/shm 2024-01-15T18:07:33.180 INFO:teuthology.orchestra.run.smithi093.stdout:/run/lock 2024-01-15T18:07:33.180 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/cgroup 2024-01-15T18:07:33.180 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/pstore 2024-01-15T18:07:33.180 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/bpf 2024-01-15T18:07:33.180 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:07:33.180 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/hugepages 2024-01-15T18:07:33.180 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/mqueue 2024-01-15T18:07:33.180 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/debug 2024-01-15T18:07:33.180 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/tracing 2024-01-15T18:07:33.180 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/fs/nfsd 2024-01-15T18:07:33.180 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/fuse/connections 2024-01-15T18:07:33.180 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/config 2024-01-15T18:07:33.181 INFO:teuthology.orchestra.run.smithi093.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:07:33.181 INFO:teuthology.orchestra.run.smithi093.stdout:/run/rpc_pipefs 2024-01-15T18:07:33.181 INFO:teuthology.orchestra.run.smithi093.stdout:/run/user/1000 2024-01-15T18:07:33.181 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph 2024-01-15T18:07:33.181 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:07:33.181 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:07:33.181 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:07:33.181 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:07:33.181 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns 2024-01-15T18:07:33.181 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:07:33.181 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:07:33.181 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:07:33.181 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:07:33.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:33.182 DEBUG:teuthology.orchestra.run.smithi093:> ls /sys/fs/fuse/connections 2024-01-15T18:07:33.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:2024-01-15T18:07:33.191+0000 7f135f6ec580 -1 init, newargv = 0x56292ec5a780 newargc=15 2024-01-15T18:07:33.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:ceph-fuse[33503]: starting ceph client 2024-01-15T18:07:33.204 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:ceph-fuse[33503]: starting fuse 2024-01-15T18:07:33.205 INFO:teuthology.orchestra.run.smithi093.stdout:43 2024-01-15T18:07:33.205 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-01-15T18:07:33.205 DEBUG:teuthology.orchestra.run.smithi093:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:07:33.205 DEBUG:teuthology.orchestra.run.smithi093:> import glob 2024-01-15T18:07:33.205 DEBUG:teuthology.orchestra.run.smithi093:> import re 2024-01-15T18:07:33.205 DEBUG:teuthology.orchestra.run.smithi093:> import os 2024-01-15T18:07:33.205 DEBUG:teuthology.orchestra.run.smithi093:> import subprocess 2024-01-15T18:07:33.205 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:07:33.206 DEBUG:teuthology.orchestra.run.smithi093:> def _find_admin_socket(client_name): 2024-01-15T18:07:33.206 DEBUG:teuthology.orchestra.run.smithi093:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-15T18:07:33.206 DEBUG:teuthology.orchestra.run.smithi093:> files = glob.glob(asok_path) 2024-01-15T18:07:33.206 DEBUG:teuthology.orchestra.run.smithi093:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-15T18:07:33.206 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:07:33.206 DEBUG:teuthology.orchestra.run.smithi093:> # Given a non-glob path, it better be there 2024-01-15T18:07:33.206 DEBUG:teuthology.orchestra.run.smithi093:> if "*" not in asok_path: 2024-01-15T18:07:33.206 DEBUG:teuthology.orchestra.run.smithi093:> assert(len(files) == 1) 2024-01-15T18:07:33.206 DEBUG:teuthology.orchestra.run.smithi093:> return files[0] 2024-01-15T18:07:33.206 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:07:33.206 DEBUG:teuthology.orchestra.run.smithi093:> for f in files: 2024-01-15T18:07:33.206 DEBUG:teuthology.orchestra.run.smithi093:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:07:33.206 DEBUG:teuthology.orchestra.run.smithi093:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:07:33.206 DEBUG:teuthology.orchestra.run.smithi093:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:07:33.206 DEBUG:teuthology.orchestra.run.smithi093:> contents = proc_f.read() 2024-01-15T18:07:33.206 DEBUG:teuthology.orchestra.run.smithi093:> if mountpoint in contents: 2024-01-15T18:07:33.207 DEBUG:teuthology.orchestra.run.smithi093:> return f 2024-01-15T18:07:33.207 DEBUG:teuthology.orchestra.run.smithi093:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:07:33.207 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:07:33.207 DEBUG:teuthology.orchestra.run.smithi093:> print(_find_admin_socket("client.1")) 2024-01-15T18:07:33.207 DEBUG:teuthology.orchestra.run.smithi093:> ' 2024-01-15T18:07:33.330 INFO:teuthology.orchestra.run.smithi093.stdout:/var/run/ceph/ceph-client.1.33503.asok 2024-01-15T18:07:33.333 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:07:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:07:33.341 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.33503.asok 2024-01-15T18:07:33.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:33.342 DEBUG:teuthology.orchestra.run.smithi093:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.33503.asok status 2024-01-15T18:07:33.512 INFO:teuthology.orchestra.run.smithi093.stdout:{ 2024-01-15T18:07:33.512 INFO:teuthology.orchestra.run.smithi093.stdout: "metadata": { 2024-01-15T18:07:33.512 INFO:teuthology.orchestra.run.smithi093.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:07:33.512 INFO:teuthology.orchestra.run.smithi093.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:07:33.512 INFO:teuthology.orchestra.run.smithi093.stdout: "entity_id": "1", 2024-01-15T18:07:33.512 INFO:teuthology.orchestra.run.smithi093.stdout: "hostname": "smithi093", 2024-01-15T18:07:33.512 INFO:teuthology.orchestra.run.smithi093.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-15T18:07:33.512 INFO:teuthology.orchestra.run.smithi093.stdout: "pid": "33503", 2024-01-15T18:07:33.512 INFO:teuthology.orchestra.run.smithi093.stdout: "root": "/" 2024-01-15T18:07:33.513 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:07:33.513 INFO:teuthology.orchestra.run.smithi093.stdout: "dentry_count": 0, 2024-01-15T18:07:33.513 INFO:teuthology.orchestra.run.smithi093.stdout: "dentry_pinned_count": 0, 2024-01-15T18:07:33.513 INFO:teuthology.orchestra.run.smithi093.stdout: "id": 5444, 2024-01-15T18:07:33.513 INFO:teuthology.orchestra.run.smithi093.stdout: "inst": { 2024-01-15T18:07:33.513 INFO:teuthology.orchestra.run.smithi093.stdout: "name": { 2024-01-15T18:07:33.513 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "client", 2024-01-15T18:07:33.513 INFO:teuthology.orchestra.run.smithi093.stdout: "num": 5444 2024-01-15T18:07:33.513 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:07:33.513 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": { 2024-01-15T18:07:33.513 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "v1", 2024-01-15T18:07:33.513 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:07:33.513 INFO:teuthology.orchestra.run.smithi093.stdout: "nonce": 2144101533 2024-01-15T18:07:33.513 INFO:teuthology.orchestra.run.smithi093.stdout: } 2024-01-15T18:07:33.514 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:07:33.514 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": { 2024-01-15T18:07:33.514 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "v1", 2024-01-15T18:07:33.514 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:07:33.514 INFO:teuthology.orchestra.run.smithi093.stdout: "nonce": 2144101533 2024-01-15T18:07:33.514 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:07:33.514 INFO:teuthology.orchestra.run.smithi093.stdout: "inst_str": "client.5444 192.168.0.1:0/2144101533", 2024-01-15T18:07:33.514 INFO:teuthology.orchestra.run.smithi093.stdout: "addr_str": "192.168.0.1:0/2144101533", 2024-01-15T18:07:33.514 INFO:teuthology.orchestra.run.smithi093.stdout: "inode_count": 1, 2024-01-15T18:07:33.514 INFO:teuthology.orchestra.run.smithi093.stdout: "mds_epoch": 27, 2024-01-15T18:07:33.514 INFO:teuthology.orchestra.run.smithi093.stdout: "osd_epoch": 45, 2024-01-15T18:07:33.514 INFO:teuthology.orchestra.run.smithi093.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:07:33.514 INFO:teuthology.orchestra.run.smithi093.stdout: "blocklisted": false, 2024-01-15T18:07:33.514 INFO:teuthology.orchestra.run.smithi093.stdout: "fs_name": "cephfs" 2024-01-15T18:07:33.514 INFO:teuthology.orchestra.run.smithi093.stdout:} 2024-01-15T18:07:33.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:33.527 DEBUG:teuthology.orchestra.run.smithi093:> stat --file-system '--printf=%T 2024-01-15T18:07:33.527 DEBUG:teuthology.orchestra.run.smithi093:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-15T18:07:33.536 INFO:teuthology.orchestra.run.smithi093.stdout:fuseblk 2024-01-15T18:07:33.536 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-15T18:07:33.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:33.536 DEBUG:teuthology.orchestra.run.smithi093:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:07:33.596 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-15T18:07:33.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.748+0000 7f81a0c00640 1 Processor -- start 2024-01-15T18:07:33.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.748+0000 7f81a0c00640 1 -- start start 2024-01-15T18:07:33.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.748+0000 7f81a0c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f819c15b350 0x7f819c0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.748+0000 7f81a0c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f819c0cfec0 con 0x7f819c09a5b0 2024-01-15T18:07:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.748+0000 7f81a0c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f819c0ccd50 con 0x7f819c099a10 2024-01-15T18:07:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.748+0000 7f81a0c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f819c0cebc0 con 0x7f819c15b350 2024-01-15T18:07:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.748+0000 7f8199e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f819c15b350 0x7f819c0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.748+0000 7f8199400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f819c09a5b0 0x7f819c0cc500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35662/0 (socket says 172.21.15.27:35662) 2024-01-15T18:07:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.748+0000 7f8199400640 1 -- 172.21.15.27:0/3025854364 learned_addr learned my addr 172.21.15.27:0/3025854364 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:33.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.752+0000 7f8199e00640 1 -- 172.21.15.27:0/3025854364 >> v1:172.21.15.93:6789/0 conn(0x7f819c099a10 legacy=0x7f819c099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:33.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.752+0000 7f8199e00640 1 -- 172.21.15.27:0/3025854364 >> v1:172.21.15.27:6789/0 conn(0x7f819c09a5b0 legacy=0x7f819c0cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:33.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.752+0000 7f8199e00640 1 -- 172.21.15.27:0/3025854364 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f819c0ceea0 con 0x7f819c15b350 2024-01-15T18:07:33.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.752+0000 7f8199e00640 1 --2- 172.21.15.27:0/3025854364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f819c15b350 0x7f819c0cf980 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f818c0097c0 tx=0x7f818c01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2508bf832e880e27 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:33.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.752+0000 7f8198a00640 1 -- 172.21.15.27:0/3025854364 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818c0ac620 con 0x7f819c15b350 2024-01-15T18:07:33.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.752+0000 7f8198a00640 1 -- 172.21.15.27:0/3025854364 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f818c0ac780 con 0x7f819c15b350 2024-01-15T18:07:33.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.752+0000 7f8198a00640 1 -- 172.21.15.27:0/3025854364 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818c0acb10 con 0x7f819c15b350 2024-01-15T18:07:33.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.752+0000 7f81a0c00640 1 -- 172.21.15.27:0/3025854364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f819c15b350 msgr2=0x7f819c0cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:33.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.752+0000 7f81a0c00640 1 --2- 172.21.15.27:0/3025854364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f819c15b350 0x7f819c0cf980 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f818c0097c0 tx=0x7f818c01c3c0 comp rx=0 tx=0).stop 2024-01-15T18:07:33.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.756+0000 7f81a0c00640 1 -- 172.21.15.27:0/3025854364 shutdown_connections 2024-01-15T18:07:33.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.756+0000 7f81a0c00640 1 --2- 172.21.15.27:0/3025854364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f819c15b350 0x7f819c0cf980 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:33.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.756+0000 7f81a0c00640 1 -- 172.21.15.27:0/3025854364 >> 172.21.15.27:0/3025854364 conn(0x7f819c0b2a80 msgr2=0x7f819c0b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:33.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.756+0000 7f81a0c00640 1 -- 172.21.15.27:0/3025854364 shutdown_connections 2024-01-15T18:07:33.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.756+0000 7f81a0c00640 1 -- 172.21.15.27:0/3025854364 wait complete. 2024-01-15T18:07:33.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.756+0000 7f81a0c00640 1 Processor -- start 2024-01-15T18:07:33.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.756+0000 7f81a0c00640 1 -- start start 2024-01-15T18:07:33.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.756+0000 7f81a0c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f819c099a10 0x7f819c06d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:33.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.756+0000 7f81a0c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f819c09a5b0 0x7f819c1570b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:33.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.756+0000 7f81a0c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f819c15b350 0x7f819c1575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:33.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.756+0000 7f81a0c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c06d5b0 con 0x7f819c09a5b0 2024-01-15T18:07:33.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.756+0000 7f81a0c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c1536e0 con 0x7f819c15b350 2024-01-15T18:07:33.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.756+0000 7f81a0c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f819c1539c0 con 0x7f819c099a10 2024-01-15T18:07:33.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.756+0000 7f8199400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f819c09a5b0 0x7f819c1570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:33.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.756+0000 7f8199400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f819c09a5b0 0x7f819c1570b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51098/0 (socket says 172.21.15.27:51098) 2024-01-15T18:07:33.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.756+0000 7f8199400640 1 -- 172.21.15.27:0/3370773586 learned_addr learned my addr 172.21.15.27:0/3370773586 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:33.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.756+0000 7f8199e00640 1 --2- 172.21.15.27:0/3370773586 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f819c099a10 0x7f819c06d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:33.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.756+0000 7f819a800640 1 --2- 172.21.15.27:0/3370773586 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f819c15b350 0x7f819c1575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:33.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.756+0000 7f8199400640 1 -- 172.21.15.27:0/3370773586 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f819c099a10 msgr2=0x7f819c06d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:33.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.756+0000 7f8199400640 1 --2- 172.21.15.27:0/3370773586 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f819c099a10 0x7f819c06d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:33.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.756+0000 7f8199400640 1 -- 172.21.15.27:0/3370773586 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f819c15b350 msgr2=0x7f819c1575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:33.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.756+0000 7f8199400640 1 --2- 172.21.15.27:0/3370773586 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f819c15b350 0x7f819c1575f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:33.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.756+0000 7f8199400640 1 -- 172.21.15.27:0/3370773586 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f818c009420 con 0x7f819c09a5b0 2024-01-15T18:07:33.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.760+0000 7f8199400640 1 --2- 172.21.15.27:0/3370773586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f819c09a5b0 0x7f819c1570b0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f818400c500 tx=0x7f818400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:33.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.760+0000 7f8192000640 1 -- 172.21.15.27:0/3370773586 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8184005cf0 con 0x7f819c09a5b0 2024-01-15T18:07:33.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.760+0000 7f8192000640 1 -- 172.21.15.27:0/3370773586 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8184005e50 con 0x7f819c09a5b0 2024-01-15T18:07:33.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.760+0000 7f81a0c00640 1 -- 172.21.15.27:0/3370773586 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f819c153ca0 con 0x7f819c09a5b0 2024-01-15T18:07:33.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.760+0000 7f8192000640 1 -- 172.21.15.27:0/3370773586 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81840072f0 con 0x7f819c09a5b0 2024-01-15T18:07:33.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.760+0000 7f81a0c00640 1 -- 172.21.15.27:0/3370773586 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f819c154260 con 0x7f819c09a5b0 2024-01-15T18:07:33.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.760+0000 7f81a0c00640 1 -- 172.21.15.27:0/3370773586 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f819c154450 con 0x7f819c09a5b0 2024-01-15T18:07:33.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.760+0000 7f8192000640 1 -- 172.21.15.27:0/3370773586 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f8184007450 con 0x7f819c09a5b0 2024-01-15T18:07:33.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.760+0000 7f8192000640 1 --2- 172.21.15.27:0/3370773586 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8174066b60 0x7f8174069020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:33.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.760+0000 7f8192000640 1 -- 172.21.15.27:0/3370773586 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f8184083e20 con 0x7f819c09a5b0 2024-01-15T18:07:33.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.764+0000 7f8199e00640 1 --2- 172.21.15.27:0/3370773586 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8174066b60 0x7f8174069020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:33.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.764+0000 7f8199e00640 1 --2- 172.21.15.27:0/3370773586 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8174066b60 0x7f8174069020 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f818c0ae000 tx=0x7f818c0a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:33.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.768+0000 7f8192000640 1 -- 172.21.15.27:0/3370773586 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8184050d00 con 0x7f819c09a5b0 2024-01-15T18:07:33.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:33.956+0000 7f81a0c00640 1 -- 172.21.15.27:0/3370773586 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f819c04e850 con 0x7f819c09a5b0 2024-01-15T18:07:34.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:34.844+0000 7f8192000640 1 -- 172.21.15.27:0/3370773586 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v28) v1 ==== 136+0+0 (secure 0 0 0) 0x7f8184054320 con 0x7f819c09a5b0 2024-01-15T18:07:34.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:34.848+0000 7f81a0c00640 1 -- 172.21.15.27:0/3370773586 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8174066b60 msgr2=0x7f8174069020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:34.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:34.848+0000 7f81a0c00640 1 --2- 172.21.15.27:0/3370773586 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8174066b60 0x7f8174069020 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f818c0ae000 tx=0x7f818c0a7000 comp rx=0 tx=0).stop 2024-01-15T18:07:34.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:34.848+0000 7f81a0c00640 1 -- 172.21.15.27:0/3370773586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f819c09a5b0 msgr2=0x7f819c1570b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:34.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:34.848+0000 7f81a0c00640 1 --2- 172.21.15.27:0/3370773586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f819c09a5b0 0x7f819c1570b0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f818400c500 tx=0x7f818400c9c0 comp rx=0 tx=0).stop 2024-01-15T18:07:34.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:34.848+0000 7f81a0c00640 1 -- 172.21.15.27:0/3370773586 shutdown_connections 2024-01-15T18:07:34.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:34.848+0000 7f81a0c00640 1 --2- 172.21.15.27:0/3370773586 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8174066b60 0x7f8174069020 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:34.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:34.848+0000 7f81a0c00640 1 --2- 172.21.15.27:0/3370773586 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f819c15b350 0x7f819c1575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:34.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:34.848+0000 7f81a0c00640 1 --2- 172.21.15.27:0/3370773586 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f819c09a5b0 0x7f819c1570b0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:34.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:34.848+0000 7f81a0c00640 1 --2- 172.21.15.27:0/3370773586 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f819c099a10 0x7f819c06d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:34.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:34.848+0000 7f81a0c00640 1 -- 172.21.15.27:0/3370773586 >> 172.21.15.27:0/3370773586 conn(0x7f819c0b2a80 msgr2=0x7f819c0ccf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:34.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:34.848+0000 7f81a0c00640 1 -- 172.21.15.27:0/3370773586 shutdown_connections 2024-01-15T18:07:34.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:34.848+0000 7f81a0c00640 1 -- 172.21.15.27:0/3370773586 wait complete. 2024-01-15T18:07:34.877 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:07:35.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.020+0000 7f8deb400640 1 Processor -- start 2024-01-15T18:07:35.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.020+0000 7f8deb400640 1 -- start start 2024-01-15T18:07:35.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.020+0000 7f8deb400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8de409a120 0x7f8de40b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.020+0000 7f8deb400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de40b1aa0 con 0x7f8de415d7c0 2024-01-15T18:07:35.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.020+0000 7f8deb400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de40b25a0 con 0x7f8de40b1d90 2024-01-15T18:07:35.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.020+0000 7f8deb400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8de40b1300 con 0x7f8de409a120 2024-01-15T18:07:35.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.020+0000 7f8de3e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8de409a120 0x7f8de40b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.020+0000 7f8de8a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8de415d7c0 0x7f8de4099a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35674/0 (socket says 172.21.15.27:35674) 2024-01-15T18:07:35.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.020+0000 7f8de3e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8de409a120 0x7f8de40b35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51112/0 (socket says 172.21.15.27:51112) 2024-01-15T18:07:35.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.020+0000 7f8de8a00640 1 -- 172.21.15.27:0/128429083 learned_addr learned my addr 172.21.15.27:0/128429083 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:35.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.020+0000 7f8de3e00640 1 -- 172.21.15.27:0/128429083 >> v1:172.21.15.93:6789/0 conn(0x7f8de40b1d90 legacy=0x7f8de40cf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:35.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.020+0000 7f8de3e00640 1 -- 172.21.15.27:0/128429083 >> v1:172.21.15.27:6789/0 conn(0x7f8de415d7c0 legacy=0x7f8de4099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.020+0000 7f8de3e00640 1 -- 172.21.15.27:0/128429083 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de40b14e0 con 0x7f8de409a120 2024-01-15T18:07:35.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.020+0000 7f8de3400640 1 -- 172.21.15.27:0/128429083 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755115476 0 0) 0x7f8de40b1aa0 con 0x7f8de415d7c0 2024-01-15T18:07:35.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.024+0000 7f8de3e00640 1 --2- 172.21.15.27:0/128429083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8de409a120 0x7f8de40b35a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc0097c0 tx=0x7f8dcc01c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d313c6c7a272f52b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:35.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.024+0000 7f8de3400640 1 -- 172.21.15.27:0/128429083 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc0ab600 con 0x7f8de409a120 2024-01-15T18:07:35.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.024+0000 7f8de3400640 1 -- 172.21.15.27:0/128429083 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dcc0a7050 con 0x7f8de409a120 2024-01-15T18:07:35.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.024+0000 7f8de3400640 1 -- 172.21.15.27:0/128429083 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc0ab980 con 0x7f8de409a120 2024-01-15T18:07:35.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.024+0000 7f8deb400640 1 -- 172.21.15.27:0/128429083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8de409a120 msgr2=0x7f8de40b35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.024+0000 7f8deb400640 1 --2- 172.21.15.27:0/128429083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8de409a120 0x7f8de40b35a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc0097c0 tx=0x7f8dcc01c3b0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.028+0000 7f8deb400640 1 -- 172.21.15.27:0/128429083 shutdown_connections 2024-01-15T18:07:35.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.028+0000 7f8deb400640 1 --2- 172.21.15.27:0/128429083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8de409a120 0x7f8de40b35a0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.028+0000 7f8deb400640 1 -- 172.21.15.27:0/128429083 >> 172.21.15.27:0/128429083 conn(0x7f8de40b2a80 msgr2=0x7f8de40b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:35.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.028+0000 7f8deb400640 1 -- 172.21.15.27:0/128429083 shutdown_connections 2024-01-15T18:07:35.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.028+0000 7f8deb400640 1 -- 172.21.15.27:0/128429083 wait complete. 2024-01-15T18:07:35.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.028+0000 7f8deb400640 1 Processor -- start 2024-01-15T18:07:35.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.028+0000 7f8deb400640 1 -- start start 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.028+0000 7f8deb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8de409a120 0x7f8de40b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.028+0000 7f8deb400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8de40b1d90 0x7f8de415c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.028+0000 7f8deb400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8de415d7c0 0x7f8de41556d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.028+0000 7f8deb400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de415cdc0 con 0x7f8de409a120 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.028+0000 7f8deb400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de40b6100 con 0x7f8de40b1d90 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.028+0000 7f8deb400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8de40b54f0 con 0x7f8de415d7c0 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.028+0000 7f8de8a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8de409a120 0x7f8de40b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.028+0000 7f8de8a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8de409a120 0x7f8de40b4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51124/0 (socket says 172.21.15.27:51124) 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.028+0000 7f8de8a00640 1 -- 172.21.15.27:0/2621048781 learned_addr learned my addr 172.21.15.27:0/2621048781 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.028+0000 7f8de9400640 1 --2- 172.21.15.27:0/2621048781 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8de415d7c0 0x7f8de41556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.028+0000 7f8de3e00640 1 --2- 172.21.15.27:0/2621048781 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8de40b1d90 0x7f8de415c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.028+0000 7f8de8a00640 1 -- 172.21.15.27:0/2621048781 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8de415d7c0 msgr2=0x7f8de41556d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.028+0000 7f8de8a00640 1 --2- 172.21.15.27:0/2621048781 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8de415d7c0 0x7f8de41556d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.028+0000 7f8de8a00640 1 -- 172.21.15.27:0/2621048781 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8de40b1d90 msgr2=0x7f8de415c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.028+0000 7f8de8a00640 1 --2- 172.21.15.27:0/2621048781 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8de40b1d90 0x7f8de415c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.028+0000 7f8de8a00640 1 -- 172.21.15.27:0/2621048781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dcc009420 con 0x7f8de409a120 2024-01-15T18:07:35.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.032+0000 7f8de8a00640 1 --2- 172.21.15.27:0/2621048781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8de409a120 0x7f8de40b4df0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f8dd40011f0 tx=0x7f8dd400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:35.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.032+0000 7f8de0c00640 1 -- 172.21.15.27:0/2621048781 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd40074e0 con 0x7f8de409a120 2024-01-15T18:07:35.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.032+0000 7f8de0c00640 1 -- 172.21.15.27:0/2621048781 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd4007640 con 0x7f8de409a120 2024-01-15T18:07:35.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.032+0000 7f8deb400640 1 -- 172.21.15.27:0/2621048781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de40b57d0 con 0x7f8de409a120 2024-01-15T18:07:35.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.032+0000 7f8de0c00640 1 -- 172.21.15.27:0/2621048781 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd4004a10 con 0x7f8de409a120 2024-01-15T18:07:35.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.032+0000 7f8deb400640 1 -- 172.21.15.27:0/2621048781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de40b5d90 con 0x7f8de409a120 2024-01-15T18:07:35.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.032+0000 7f8deb400640 1 -- 172.21.15.27:0/2621048781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de40cafa0 con 0x7f8de409a120 2024-01-15T18:07:35.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.032+0000 7f8de0c00640 1 -- 172.21.15.27:0/2621048781 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f8dd4004b70 con 0x7f8de409a120 2024-01-15T18:07:35.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.032+0000 7f8de0c00640 1 --2- 172.21.15.27:0/2621048781 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8dbc066b60 0x7f8dbc069020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.032+0000 7f8de0c00640 1 -- 172.21.15.27:0/2621048781 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f8dd40838e0 con 0x7f8de409a120 2024-01-15T18:07:35.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.032+0000 7f8de3e00640 1 --2- 172.21.15.27:0/2621048781 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8dbc066b60 0x7f8dbc069020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.036+0000 7f8de3e00640 1 --2- 172.21.15.27:0/2621048781 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8dbc066b60 0x7f8dbc069020 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc0ad000 tx=0x7f8dcc0a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:35.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.036+0000 7f8de0c00640 1 -- 172.21.15.27:0/2621048781 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8dd40508e0 con 0x7f8de409a120 2024-01-15T18:07:35.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.220+0000 7f8deb400640 1 -- 172.21.15.27:0/2621048781 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f8de40d0bf0 con 0x7f8dbc066b60 2024-01-15T18:07:35.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.224+0000 7f8de0c00640 1 -- 172.21.15.27:0/2621048781 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f8de40d0bf0 con 0x7f8dbc066b60 2024-01-15T18:07:35.227 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:07:35.227 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:07:35.227 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-15T18:07:35.227 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:07:35.228 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:07:35.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.224+0000 7f8deb400640 1 -- 172.21.15.27:0/2621048781 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8dbc066b60 msgr2=0x7f8dbc069020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.224+0000 7f8deb400640 1 --2- 172.21.15.27:0/2621048781 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8dbc066b60 0x7f8dbc069020 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc0ad000 tx=0x7f8dcc0a6000 comp rx=0 tx=0).stop 2024-01-15T18:07:35.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.224+0000 7f8deb400640 1 -- 172.21.15.27:0/2621048781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8de409a120 msgr2=0x7f8de40b4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.224+0000 7f8deb400640 1 --2- 172.21.15.27:0/2621048781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8de409a120 0x7f8de40b4df0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f8dd40011f0 tx=0x7f8dd400c9b0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.224+0000 7f8deb400640 1 -- 172.21.15.27:0/2621048781 shutdown_connections 2024-01-15T18:07:35.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.224+0000 7f8deb400640 1 --2- 172.21.15.27:0/2621048781 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8dbc066b60 0x7f8dbc069020 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.224+0000 7f8deb400640 1 --2- 172.21.15.27:0/2621048781 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8de415d7c0 0x7f8de41556d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.224+0000 7f8deb400640 1 --2- 172.21.15.27:0/2621048781 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8de40b1d90 0x7f8de415c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.224+0000 7f8deb400640 1 --2- 172.21.15.27:0/2621048781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8de409a120 0x7f8de40b4df0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.224+0000 7f8deb400640 1 -- 172.21.15.27:0/2621048781 >> 172.21.15.27:0/2621048781 conn(0x7f8de40b2a80 msgr2=0x7f8de406e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:35.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.224+0000 7f8deb400640 1 -- 172.21.15.27:0/2621048781 shutdown_connections 2024-01-15T18:07:35.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.224+0000 7f8deb400640 1 -- 172.21.15.27:0/2621048781 wait complete. 2024-01-15T18:07:35.250 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:07:35.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.388+0000 7fde32000640 1 Processor -- start 2024-01-15T18:07:35.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.388+0000 7fde32000640 1 -- start start 2024-01-15T18:07:35.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.392+0000 7fde32000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fde2c15d190 0x7fde2c0cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.392+0000 7fde32000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fde2c06c520 0x7fde2c0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.392+0000 7fde32000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde2c0b0d70 con 0x7fde2c15d570 2024-01-15T18:07:35.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.392+0000 7fde32000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde2c0d0540 con 0x7fde2c15d190 2024-01-15T18:07:35.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.392+0000 7fde32000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c0b0440 con 0x7fde2c06c520 2024-01-15T18:07:35.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.392+0000 7fde2be00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fde2c06c520 0x7fde2c0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.392+0000 7fde2b400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fde2c15d190 0x7fde2c0cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.392+0000 7fde2be00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fde2c06c520 0x7fde2c0cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:37590/0 (socket says 172.21.15.27:37590) 2024-01-15T18:07:35.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.392+0000 7fde2be00640 1 -- 172.21.15.27:0/3550703796 learned_addr learned my addr 172.21.15.27:0/3550703796 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:35.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.392+0000 7fde2aa00640 1 --1- 172.21.15.27:0/3550703796 >> v1:172.21.15.93:6789/0 conn(0x7fde2c15d570 0x7fde2c06be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:37086/0 (socket says 172.21.15.27:37086) 2024-01-15T18:07:35.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.392+0000 7fde2be00640 1 -- 172.21.15.27:0/3550703796 >> v2:172.21.15.93:3300/0 conn(0x7fde2c15d190 msgr2=0x7fde2c0cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:35.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.392+0000 7fde2be00640 1 --2- 172.21.15.27:0/3550703796 >> v2:172.21.15.93:3300/0 conn(0x7fde2c15d190 0x7fde2c0cfcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.392+0000 7fde2be00640 1 -- 172.21.15.27:0/3550703796 >> v1:172.21.15.93:6789/0 conn(0x7fde2c15d570 legacy=0x7fde2c06be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.392+0000 7fde2be00640 1 -- 172.21.15.27:0/3550703796 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde2c0b0720 con 0x7fde2c06c520 2024-01-15T18:07:35.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.392+0000 7fde2b400640 1 --2- 172.21.15.27:0/3550703796 >> v2:172.21.15.93:3300/0 conn(0x7fde2c15d190 0x7fde2c0cfcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:07:35.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.396+0000 7fde2be00640 1 --2- 172.21.15.27:0/3550703796 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fde2c06c520 0x7fde2c0cd5e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fde24009e50 tx=0x7fde2401c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b31d42063f412e4c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:35.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.396+0000 7fde2a000640 1 -- 172.21.15.27:0/3550703796 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde240a5870 con 0x7fde2c06c520 2024-01-15T18:07:35.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.396+0000 7fde2a000640 1 -- 172.21.15.27:0/3550703796 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde240a59d0 con 0x7fde2c06c520 2024-01-15T18:07:35.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.396+0000 7fde2a000640 1 -- 172.21.15.27:0/3550703796 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde240a5d60 con 0x7fde2c06c520 2024-01-15T18:07:35.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.396+0000 7fde32000640 1 -- 172.21.15.27:0/3550703796 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fde2c06c520 msgr2=0x7fde2c0cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.396+0000 7fde32000640 1 --2- 172.21.15.27:0/3550703796 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fde2c06c520 0x7fde2c0cd5e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fde24009e50 tx=0x7fde2401c9e0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.396+0000 7fde32000640 1 -- 172.21.15.27:0/3550703796 shutdown_connections 2024-01-15T18:07:35.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.396+0000 7fde32000640 1 --2- 172.21.15.27:0/3550703796 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fde2c06c520 0x7fde2c0cd5e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.396+0000 7fde32000640 1 --2- 172.21.15.27:0/3550703796 >> v2:172.21.15.93:3300/0 conn(0x7fde2c15d190 0x7fde2c0cfcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.396+0000 7fde32000640 1 -- 172.21.15.27:0/3550703796 >> 172.21.15.27:0/3550703796 conn(0x7fde2c0d0e70 msgr2=0x7fde2c0d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:35.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.396+0000 7fde32000640 1 -- 172.21.15.27:0/3550703796 shutdown_connections 2024-01-15T18:07:35.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.396+0000 7fde32000640 1 -- 172.21.15.27:0/3550703796 wait complete. 2024-01-15T18:07:35.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.396+0000 7fde32000640 1 Processor -- start 2024-01-15T18:07:35.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.396+0000 7fde32000640 1 -- start start 2024-01-15T18:07:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.400+0000 7fde32000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fde2c06c520 0x7fde2c1544a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.400+0000 7fde32000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fde2c15d190 0x7fde2c158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.400+0000 7fde32000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde2c15d570 0x7fde2c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.400+0000 7fde32000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c1549e0 con 0x7fde2c15d570 2024-01-15T18:07:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.400+0000 7fde32000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c15a480 con 0x7fde2c15d190 2024-01-15T18:07:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.400+0000 7fde32000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c1571e0 con 0x7fde2c06c520 2024-01-15T18:07:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.400+0000 7fde2be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde2c15d570 0x7fde2c06ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.400+0000 7fde2be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde2c15d570 0x7fde2c06ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51140/0 (socket says 172.21.15.27:51140) 2024-01-15T18:07:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.400+0000 7fde2be00640 1 -- 172.21.15.27:0/3684039150 learned_addr learned my addr 172.21.15.27:0/3684039150 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:35.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.400+0000 7fde2b400640 1 --2- 172.21.15.27:0/3684039150 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fde2c06c520 0x7fde2c1544a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.400+0000 7fde2aa00640 1 --2- 172.21.15.27:0/3684039150 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fde2c15d190 0x7fde2c158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.400+0000 7fde2be00640 1 -- 172.21.15.27:0/3684039150 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fde2c06c520 msgr2=0x7fde2c1544a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.400+0000 7fde2be00640 1 --2- 172.21.15.27:0/3684039150 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fde2c06c520 0x7fde2c1544a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.400+0000 7fde2be00640 1 -- 172.21.15.27:0/3684039150 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fde2c15d190 msgr2=0x7fde2c158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.400+0000 7fde2be00640 1 --2- 172.21.15.27:0/3684039150 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fde2c15d190 0x7fde2c158750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.400+0000 7fde2be00640 1 -- 172.21.15.27:0/3684039150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1c009420 con 0x7fde2c15d570 2024-01-15T18:07:35.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.400+0000 7fde2be00640 1 --2- 172.21.15.27:0/3684039150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde2c15d570 0x7fde2c06ea50 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fde2401a610 tx=0x7fde24019780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:35.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.400+0000 7fde23400640 1 -- 172.21.15.27:0/3684039150 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde240b8670 con 0x7fde2c15d570 2024-01-15T18:07:35.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.400+0000 7fde23400640 1 -- 172.21.15.27:0/3684039150 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde240b87d0 con 0x7fde2c15d570 2024-01-15T18:07:35.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.400+0000 7fde23400640 1 -- 172.21.15.27:0/3684039150 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde240b8b80 con 0x7fde2c15d570 2024-01-15T18:07:35.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.400+0000 7fde32000640 1 -- 172.21.15.27:0/3684039150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde24009b30 con 0x7fde2c15d570 2024-01-15T18:07:35.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.400+0000 7fde32000640 1 -- 172.21.15.27:0/3684039150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde2c1577a0 con 0x7fde2c15d570 2024-01-15T18:07:35.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.400+0000 7fde32000640 1 -- 172.21.15.27:0/3684039150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde2c04e850 con 0x7fde2c15d570 2024-01-15T18:07:35.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.400+0000 7fde23400640 1 -- 172.21.15.27:0/3684039150 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7fde240b8ce0 con 0x7fde2c15d570 2024-01-15T18:07:35.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.404+0000 7fde23400640 1 --2- 172.21.15.27:0/3684039150 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fde04066b30 0x7fde04068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.404+0000 7fde23400640 1 -- 172.21.15.27:0/3684039150 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fde2411fda0 con 0x7fde2c15d570 2024-01-15T18:07:35.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.404+0000 7fde2b400640 1 --2- 172.21.15.27:0/3684039150 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fde04066b30 0x7fde04068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.404+0000 7fde2b400640 1 --2- 172.21.15.27:0/3684039150 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fde04066b30 0x7fde04068ff0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fde2c0b0380 tx=0x7fde1c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:35.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.408+0000 7fde23400640 1 -- 172.21.15.27:0/3684039150 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fde240ece50 con 0x7fde2c15d570 2024-01-15T18:07:35.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.564+0000 7fde32000640 1 -- 172.21.15.27:0/3684039150 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fde2c04e850 con 0x7fde2c15d570 2024-01-15T18:07:35.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.564+0000 7fde23400640 1 -- 172.21.15.27:0/3684039150 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7fde240f0470 con 0x7fde2c15d570 2024-01-15T18:07:35.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.568+0000 7fde32000640 1 -- 172.21.15.27:0/3684039150 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fde04066b30 msgr2=0x7fde04068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.568+0000 7fde32000640 1 --2- 172.21.15.27:0/3684039150 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fde04066b30 0x7fde04068ff0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fde2c0b0380 tx=0x7fde1c009190 comp rx=0 tx=0).stop 2024-01-15T18:07:35.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.568+0000 7fde32000640 1 -- 172.21.15.27:0/3684039150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde2c15d570 msgr2=0x7fde2c06ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.568+0000 7fde32000640 1 --2- 172.21.15.27:0/3684039150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde2c15d570 0x7fde2c06ea50 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fde2401a610 tx=0x7fde24019780 comp rx=0 tx=0).stop 2024-01-15T18:07:35.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.568+0000 7fde32000640 1 -- 172.21.15.27:0/3684039150 shutdown_connections 2024-01-15T18:07:35.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.568+0000 7fde32000640 1 --2- 172.21.15.27:0/3684039150 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fde04066b30 0x7fde04068ff0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.568+0000 7fde32000640 1 --2- 172.21.15.27:0/3684039150 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde2c15d570 0x7fde2c06ea50 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.568+0000 7fde32000640 1 --2- 172.21.15.27:0/3684039150 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fde2c15d190 0x7fde2c158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.568+0000 7fde32000640 1 --2- 172.21.15.27:0/3684039150 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fde2c06c520 0x7fde2c1544a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.568+0000 7fde32000640 1 -- 172.21.15.27:0/3684039150 >> 172.21.15.27:0/3684039150 conn(0x7fde2c0d0e70 msgr2=0x7fde2c0b1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:35.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.568+0000 7fde32000640 1 -- 172.21.15.27:0/3684039150 shutdown_connections 2024-01-15T18:07:35.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.568+0000 7fde32000640 1 -- 172.21.15.27:0/3684039150 wait complete. 2024-01-15T18:07:35.592 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-15T18:07:35.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.736+0000 7f211ce00640 1 Processor -- start 2024-01-15T18:07:35.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.736+0000 7f211ce00640 1 -- start start 2024-01-15T18:07:35.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.736+0000 7f211ce00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2118158900 0x7f211815f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.736+0000 7f211ce00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f211815bf00 0x7f21180cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.736+0000 7f211ce00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21180b1cb0 con 0x7f21180cf110 2024-01-15T18:07:35.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.736+0000 7f211ce00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21180cf540 con 0x7f2118158900 2024-01-15T18:07:35.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.736+0000 7f211ce00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21180b1f00 con 0x7f211815bf00 2024-01-15T18:07:35.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.740+0000 7f2115e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2118158900 0x7f211815f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.740+0000 7f2115e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2118158900 0x7f211815f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51142/0 (socket says 172.21.15.27:51142) 2024-01-15T18:07:35.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.740+0000 7f2115e00640 1 -- 172.21.15.27:0/536286308 learned_addr learned my addr 172.21.15.27:0/536286308 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:35.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.740+0000 7f2115400640 1 --2- 172.21.15.27:0/536286308 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f211815bf00 0x7f21180cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.740+0000 7f2115e00640 1 -- 172.21.15.27:0/536286308 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f211815bf00 msgr2=0x7f21180cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:35.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.740+0000 7f2115e00640 1 --2- 172.21.15.27:0/536286308 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f211815bf00 0x7f21180cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.740+0000 7f2115e00640 1 -- 172.21.15.27:0/536286308 >> v1:172.21.15.93:6789/0 conn(0x7f21180cf110 legacy=0x7f21180701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.740+0000 7f2115e00640 1 -- 172.21.15.27:0/536286308 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21180b21e0 con 0x7f2118158900 2024-01-15T18:07:35.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.740+0000 7f2115e00640 1 --2- 172.21.15.27:0/536286308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2118158900 0x7f211815f550 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f2108009870 tx=0x7f210801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=454929d9f791f4af server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:35.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.740+0000 7f2114a00640 1 -- 172.21.15.27:0/536286308 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21080a7070 con 0x7f2118158900 2024-01-15T18:07:35.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.740+0000 7f2114a00640 1 -- 172.21.15.27:0/536286308 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21080ab430 con 0x7f2118158900 2024-01-15T18:07:35.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.740+0000 7f2114a00640 1 -- 172.21.15.27:0/536286308 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21080ab7c0 con 0x7f2118158900 2024-01-15T18:07:35.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.744+0000 7f211ce00640 1 -- 172.21.15.27:0/536286308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2118158900 msgr2=0x7f211815f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.744+0000 7f211ce00640 1 --2- 172.21.15.27:0/536286308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2118158900 0x7f211815f550 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f2108009870 tx=0x7f210801c540 comp rx=0 tx=0).stop 2024-01-15T18:07:35.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.744+0000 7f211ce00640 1 -- 172.21.15.27:0/536286308 shutdown_connections 2024-01-15T18:07:35.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.744+0000 7f211ce00640 1 --2- 172.21.15.27:0/536286308 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f211815bf00 0x7f21180cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.744+0000 7f211ce00640 1 --2- 172.21.15.27:0/536286308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2118158900 0x7f211815f550 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.744+0000 7f211ce00640 1 -- 172.21.15.27:0/536286308 >> 172.21.15.27:0/536286308 conn(0x7f21180cb700 msgr2=0x7f21180cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:35.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.744+0000 7f211ce00640 1 -- 172.21.15.27:0/536286308 shutdown_connections 2024-01-15T18:07:35.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.744+0000 7f211ce00640 1 -- 172.21.15.27:0/536286308 wait complete. 2024-01-15T18:07:35.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.744+0000 7f211ce00640 1 Processor -- start 2024-01-15T18:07:35.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.744+0000 7f211ce00640 1 -- start start 2024-01-15T18:07:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.744+0000 7f211ce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21180cf110 0x7f2118152520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.744+0000 7f211ce00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2118158900 0x7f211814e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.744+0000 7f211ce00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f211815bf00 0x7f211814e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.744+0000 7f211ce00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2118159220 con 0x7f21180cf110 2024-01-15T18:07:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.744+0000 7f211ce00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21180cd9c0 con 0x7f2118158900 2024-01-15T18:07:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.744+0000 7f211ce00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21180cdca0 con 0x7f211815bf00 2024-01-15T18:07:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.744+0000 7f2115e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21180cf110 0x7f2118152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.744+0000 7f2115e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21180cf110 0x7f2118152520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51158/0 (socket says 172.21.15.27:51158) 2024-01-15T18:07:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.744+0000 7f2115e00640 1 -- 172.21.15.27:0/24680807 learned_addr learned my addr 172.21.15.27:0/24680807 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.744+0000 7f2115400640 1 --2- 172.21.15.27:0/24680807 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2118158900 0x7f211814e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.744+0000 7f2116800640 1 --2- 172.21.15.27:0/24680807 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f211815bf00 0x7f211814e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.748+0000 7f2115e00640 1 -- 172.21.15.27:0/24680807 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f211815bf00 msgr2=0x7f211814e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.748+0000 7f2115e00640 1 --2- 172.21.15.27:0/24680807 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f211815bf00 0x7f211814e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.748+0000 7f2115e00640 1 -- 172.21.15.27:0/24680807 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2118158900 msgr2=0x7f211814e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.748+0000 7f2115e00640 1 --2- 172.21.15.27:0/24680807 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2118158900 0x7f211814e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.748+0000 7f2115e00640 1 -- 172.21.15.27:0/24680807 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21080094d0 con 0x7f21180cf110 2024-01-15T18:07:35.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.748+0000 7f2115e00640 1 --2- 172.21.15.27:0/24680807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21180cf110 0x7f2118152520 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f21080ad040 tx=0x7f21080192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:35.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.748+0000 7f210e000640 1 -- 172.21.15.27:0/24680807 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21080bc070 con 0x7f21180cf110 2024-01-15T18:07:35.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.748+0000 7f210e000640 1 -- 172.21.15.27:0/24680807 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21080b4420 con 0x7f21180cf110 2024-01-15T18:07:35.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.748+0000 7f211ce00640 1 -- 172.21.15.27:0/24680807 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f211815c490 con 0x7f21180cf110 2024-01-15T18:07:35.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.748+0000 7f210e000640 1 -- 172.21.15.27:0/24680807 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21080b47d0 con 0x7f21180cf110 2024-01-15T18:07:35.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.748+0000 7f211ce00640 1 -- 172.21.15.27:0/24680807 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21180ce260 con 0x7f21180cf110 2024-01-15T18:07:35.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.748+0000 7f211ce00640 1 -- 172.21.15.27:0/24680807 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21180b4ee0 con 0x7f21180cf110 2024-01-15T18:07:35.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.748+0000 7f210e000640 1 -- 172.21.15.27:0/24680807 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f21080b4930 con 0x7f21180cf110 2024-01-15T18:07:35.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.748+0000 7f210e000640 1 --2- 172.21.15.27:0/24680807 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f20f0066b30 0x7f20f0068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.748+0000 7f210e000640 1 -- 172.21.15.27:0/24680807 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f210811e110 con 0x7f21180cf110 2024-01-15T18:07:35.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.752+0000 7f2115400640 1 --2- 172.21.15.27:0/24680807 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f20f0066b30 0x7f20f0068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.752+0000 7f2115400640 1 --2- 172.21.15.27:0/24680807 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f20f0066b30 0x7f20f0068ff0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2118153f10 tx=0x7f2100009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:35.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.752+0000 7f210e000640 1 -- 172.21.15.27:0/24680807 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f21080eb000 con 0x7f21180cf110 2024-01-15T18:07:35.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.936+0000 7f211ce00640 1 -- 172.21.15.27:0/24680807 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f2118070d00 con 0x7f20f0066b30 2024-01-15T18:07:35.945 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:07:35.939+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:07:35.945 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:07:35.939+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:07:35.945 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:07:35.939+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:07:35.946 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:07:35.939+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:07:35.946 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:07:35.939+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:07:35.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.972+0000 7f210e000640 1 -- 172.21.15.27:0/24680807 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2100004020 con 0x7f20f0066b30 2024-01-15T18:07:35.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.976+0000 7f211ce00640 1 -- 172.21.15.27:0/24680807 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f20f0066b30 msgr2=0x7f20f0068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.976+0000 7f211ce00640 1 --2- 172.21.15.27:0/24680807 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f20f0066b30 0x7f20f0068ff0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2118153f10 tx=0x7f2100009190 comp rx=0 tx=0).stop 2024-01-15T18:07:35.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.976+0000 7f211ce00640 1 -- 172.21.15.27:0/24680807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21180cf110 msgr2=0x7f2118152520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.976+0000 7f211ce00640 1 --2- 172.21.15.27:0/24680807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21180cf110 0x7f2118152520 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f21080ad040 tx=0x7f21080192e0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.976+0000 7f211ce00640 1 -- 172.21.15.27:0/24680807 shutdown_connections 2024-01-15T18:07:35.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.976+0000 7f211ce00640 1 --2- 172.21.15.27:0/24680807 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f20f0066b30 0x7f20f0068ff0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.976+0000 7f211ce00640 1 --2- 172.21.15.27:0/24680807 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f211815bf00 0x7f211814e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.976+0000 7f211ce00640 1 --2- 172.21.15.27:0/24680807 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2118158900 0x7f211814e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.976+0000 7f211ce00640 1 --2- 172.21.15.27:0/24680807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21180cf110 0x7f2118152520 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.976+0000 7f211ce00640 1 -- 172.21.15.27:0/24680807 >> 172.21.15.27:0/24680807 conn(0x7f21180cb700 msgr2=0x7f21180700e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:35.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.976+0000 7f211ce00640 1 -- 172.21.15.27:0/24680807 shutdown_connections 2024-01-15T18:07:35.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:35.976+0000 7f211ce00640 1 -- 172.21.15.27:0/24680807 wait complete. 2024-01-15T18:07:36.000 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:07:36.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.144+0000 7f3821400640 1 Processor -- start 2024-01-15T18:07:36.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.144+0000 7f3821400640 1 -- start start 2024-01-15T18:07:36.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.144+0000 7f3821400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f381c158900 0x7f381c15f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:36.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.144+0000 7f3821400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f381c0cf2e0 0x7f381c0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:36.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.144+0000 7f3821400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f381c0b1cb0 con 0x7f381c15bf00 2024-01-15T18:07:36.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.144+0000 7f3821400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f381c0b1f00 con 0x7f381c158900 2024-01-15T18:07:36.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.144+0000 7f3821400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f381c0b21e0 con 0x7f381c0cf2e0 2024-01-15T18:07:36.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.144+0000 7f381b200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f381c0cf2e0 0x7f381c0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:36.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.144+0000 7f381a800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f381c158900 0x7f381c15f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:36.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.144+0000 7f3819e00640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7f381c15bf00 0x7f381c0cebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:37110/0 (socket says 172.21.15.27:37110) 2024-01-15T18:07:36.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.144+0000 7f3819e00640 1 -- 172.21.15.27:0/1556805078 learned_addr learned my addr 172.21.15.27:0/1556805078 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:36.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.148+0000 7f3819400640 1 -- 172.21.15.27:0/1556805078 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551778320 0 0) 0x7f381c0b1cb0 con 0x7f381c15bf00 2024-01-15T18:07:36.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.148+0000 7f3819400640 1 -- 172.21.15.27:0/1556805078 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37fc003530 con 0x7f381c15bf00 2024-01-15T18:07:36.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.148+0000 7f381b200640 1 -- 172.21.15.27:0/1556805078 >> v2:172.21.15.93:3300/0 conn(0x7f381c158900 msgr2=0x7f381c15f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:36.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.148+0000 7f381b200640 1 --2- 172.21.15.27:0/1556805078 >> v2:172.21.15.93:3300/0 conn(0x7f381c158900 0x7f381c15f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:36.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.148+0000 7f381b200640 1 -- 172.21.15.27:0/1556805078 >> v1:172.21.15.93:6789/0 conn(0x7f381c15bf00 legacy=0x7f381c0cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:36.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.148+0000 7f381b200640 1 -- 172.21.15.27:0/1556805078 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381c15ea70 con 0x7f381c0cf2e0 2024-01-15T18:07:36.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.148+0000 7f381a800640 1 --2- 172.21.15.27:0/1556805078 >> v2:172.21.15.93:3300/0 conn(0x7f381c158900 0x7f381c15f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:07:36.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.148+0000 7f381b200640 1 --2- 172.21.15.27:0/1556805078 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f381c0cf2e0 0x7f381c0701e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3814009d90 tx=0x7f381401c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb917fe48c84a952 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:36.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.148+0000 7f3819400640 1 -- 172.21.15.27:0/1556805078 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381c070f30 con 0x7f381c0cf2e0 2024-01-15T18:07:36.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.148+0000 7f3819400640 1 -- 172.21.15.27:0/1556805078 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38140a5900 con 0x7f381c0cf2e0 2024-01-15T18:07:36.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.148+0000 7f3819400640 1 -- 172.21.15.27:0/1556805078 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38140a5c90 con 0x7f381c0cf2e0 2024-01-15T18:07:36.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.152+0000 7f3821400640 1 -- 172.21.15.27:0/1556805078 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f381c0cf2e0 msgr2=0x7f381c0701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:36.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.152+0000 7f3821400640 1 --2- 172.21.15.27:0/1556805078 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f381c0cf2e0 0x7f381c0701e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3814009d90 tx=0x7f381401c950 comp rx=0 tx=0).stop 2024-01-15T18:07:36.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.152+0000 7f3821400640 1 -- 172.21.15.27:0/1556805078 shutdown_connections 2024-01-15T18:07:36.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.152+0000 7f3821400640 1 --2- 172.21.15.27:0/1556805078 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f381c0cf2e0 0x7f381c0701e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:36.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.152+0000 7f3821400640 1 --2- 172.21.15.27:0/1556805078 >> v2:172.21.15.93:3300/0 conn(0x7f381c158900 0x7f381c15f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:36.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.152+0000 7f3821400640 1 -- 172.21.15.27:0/1556805078 >> 172.21.15.27:0/1556805078 conn(0x7f381c0cb700 msgr2=0x7f381c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:36.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.152+0000 7f3821400640 1 -- 172.21.15.27:0/1556805078 shutdown_connections 2024-01-15T18:07:36.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.152+0000 7f3821400640 1 -- 172.21.15.27:0/1556805078 wait complete. 2024-01-15T18:07:36.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.152+0000 7f3821400640 1 Processor -- start 2024-01-15T18:07:36.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.152+0000 7f3821400640 1 -- start start 2024-01-15T18:07:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.152+0000 7f3821400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f381c0cf2e0 0x7f381c150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.152+0000 7f3821400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f381c158900 0x7f381c143710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.152+0000 7f3821400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f381c15bf00 0x7f381c157000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.152+0000 7f3821400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381c143c50 con 0x7f381c0cf2e0 2024-01-15T18:07:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.152+0000 7f3821400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381c0a8e80 con 0x7f381c158900 2024-01-15T18:07:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.152+0000 7f3821400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f381c0a9160 con 0x7f381c15bf00 2024-01-15T18:07:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.152+0000 7f381a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f381c0cf2e0 0x7f381c150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.152+0000 7f381a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f381c0cf2e0 0x7f381c150ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51168/0 (socket says 172.21.15.27:51168) 2024-01-15T18:07:36.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.152+0000 7f381a800640 1 -- 172.21.15.27:0/526982382 learned_addr learned my addr 172.21.15.27:0/526982382 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:36.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.152+0000 7f381b200640 1 --2- 172.21.15.27:0/526982382 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f381c15bf00 0x7f381c157000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:36.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.152+0000 7f3819e00640 1 --2- 172.21.15.27:0/526982382 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f381c158900 0x7f381c143710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.156+0000 7f381a800640 1 -- 172.21.15.27:0/526982382 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f381c15bf00 msgr2=0x7f381c157000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.156+0000 7f381a800640 1 --2- 172.21.15.27:0/526982382 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f381c15bf00 0x7f381c157000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.156+0000 7f381a800640 1 -- 172.21.15.27:0/526982382 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f381c158900 msgr2=0x7f381c143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.156+0000 7f381a800640 1 --2- 172.21.15.27:0/526982382 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f381c158900 0x7f381c143710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.156+0000 7f381a800640 1 -- 172.21.15.27:0/526982382 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f380c009420 con 0x7f381c0cf2e0 2024-01-15T18:07:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.156+0000 7f3819e00640 1 --2- 172.21.15.27:0/526982382 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f381c158900 0x7f381c143710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:07:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.156+0000 7f381a800640 1 --2- 172.21.15.27:0/526982382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f381c0cf2e0 0x7f381c150ad0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f380c00ebe0 tx=0x7f380c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.156+0000 7f3812a00640 1 -- 172.21.15.27:0/526982382 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380c00eee0 con 0x7f381c0cf2e0 2024-01-15T18:07:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.156+0000 7f3812a00640 1 -- 172.21.15.27:0/526982382 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380c00cc30 con 0x7f381c0cf2e0 2024-01-15T18:07:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.156+0000 7f3812a00640 1 -- 172.21.15.27:0/526982382 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380c013470 con 0x7f381c0cf2e0 2024-01-15T18:07:36.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.156+0000 7f3821400640 1 -- 172.21.15.27:0/526982382 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3814009a70 con 0x7f381c0cf2e0 2024-01-15T18:07:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.156+0000 7f3821400640 1 -- 172.21.15.27:0/526982382 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f381c0a9720 con 0x7f381c0cf2e0 2024-01-15T18:07:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.156+0000 7f3821400640 1 -- 172.21.15.27:0/526982382 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f381c04e850 con 0x7f381c0cf2e0 2024-01-15T18:07:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.156+0000 7f3812a00640 1 -- 172.21.15.27:0/526982382 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 86364+0+0 (secure 0 0 0) 0x7f380c0135d0 con 0x7f381c0cf2e0 2024-01-15T18:07:36.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.156+0000 7f3812a00640 1 --2- 172.21.15.27:0/526982382 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f37f4066b30 0x7f37f4068ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:36.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.156+0000 7f3812a00640 1 -- 172.21.15.27:0/526982382 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f380c084a40 con 0x7f381c0cf2e0 2024-01-15T18:07:36.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.160+0000 7f3819e00640 1 --2- 172.21.15.27:0/526982382 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f37f4066b30 0x7f37f4068ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:36.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.160+0000 7f3819e00640 1 --2- 172.21.15.27:0/526982382 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f37f4066b30 0x7f37f4068ff0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f381c0ab4c0 tx=0x7f3804009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:36.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.160+0000 7f3812a00640 1 -- 172.21.15.27:0/526982382 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f380c086e60 con 0x7f381c0cf2e0 2024-01-15T18:07:36.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.344+0000 7f3821400640 1 -- 172.21.15.27:0/526982382 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f381c06e170 con 0x7f37f4066b30 2024-01-15T18:07:36.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.352+0000 7f3812a00640 1 -- 172.21.15.27:0/526982382 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f38040013e0 con 0x7f37f4066b30 2024-01-15T18:07:36.355 INFO:teuthology.orchestra.run.smithi027.stdout:/volumes/_nogroup/sv1/5d7c5818-d701-4425-8eb7-540607ec283f 2024-01-15T18:07:36.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.352+0000 7f3821400640 1 -- 172.21.15.27:0/526982382 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f37f4066b30 msgr2=0x7f37f4068ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:36.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.352+0000 7f3821400640 1 --2- 172.21.15.27:0/526982382 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f37f4066b30 0x7f37f4068ff0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f381c0ab4c0 tx=0x7f3804009040 comp rx=0 tx=0).stop 2024-01-15T18:07:36.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.352+0000 7f3821400640 1 -- 172.21.15.27:0/526982382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f381c0cf2e0 msgr2=0x7f381c150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:36.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.352+0000 7f3821400640 1 --2- 172.21.15.27:0/526982382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f381c0cf2e0 0x7f381c150ad0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f380c00ebe0 tx=0x7f380c00c520 comp rx=0 tx=0).stop 2024-01-15T18:07:36.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.352+0000 7f3821400640 1 -- 172.21.15.27:0/526982382 shutdown_connections 2024-01-15T18:07:36.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.352+0000 7f3821400640 1 --2- 172.21.15.27:0/526982382 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f37f4066b30 0x7f37f4068ff0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:36.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.352+0000 7f3821400640 1 --2- 172.21.15.27:0/526982382 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f381c15bf00 0x7f381c157000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:36.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.352+0000 7f3821400640 1 --2- 172.21.15.27:0/526982382 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f381c158900 0x7f381c143710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:36.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.352+0000 7f3821400640 1 --2- 172.21.15.27:0/526982382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f381c0cf2e0 0x7f381c150ad0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:36.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.352+0000 7f3821400640 1 -- 172.21.15.27:0/526982382 >> 172.21.15.27:0/526982382 conn(0x7f381c0cb700 msgr2=0x7f381c071820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:36.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.352+0000 7f3821400640 1 -- 172.21.15.27:0/526982382 shutdown_connections 2024-01-15T18:07:36.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:36.352+0000 7f3821400640 1 -- 172.21.15.27:0/526982382 wait complete. 2024-01-15T18:07:36.378 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/5d7c5818-d701-4425-8eb7-540607ec283f 2024-01-15T18:07:36.378 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:07:36.378 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:07:36.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:36.387 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/5d7c5818-d701-4425-8eb7-540607ec283f/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:07:43.565 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records in 2024-01-15T18:07:43.565 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records out 2024-01-15T18:07:43.566 INFO:teuthology.orchestra.run.smithi027.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.10896 s, 151 MB/s 2024-01-15T18:07:43.567 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:07:43.567 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:07:43.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:43.575 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/5d7c5818-d701-4425-8eb7-540607ec283f/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:07:50.461 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records in 2024-01-15T18:07:50.461 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records out 2024-01-15T18:07:50.461 INFO:teuthology.orchestra.run.smithi027.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.83155 s, 157 MB/s 2024-01-15T18:07:50.463 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:07:50.463 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:07:50.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:50.472 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/5d7c5818-d701-4425-8eb7-540607ec283f/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:07:56.337 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:07:56.341 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:07:57.357 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records in 2024-01-15T18:07:57.357 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records out 2024-01-15T18:07:57.357 INFO:teuthology.orchestra.run.smithi027.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.826 s, 157 MB/s 2024-01-15T18:07:57.358 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 10 2024-01-15T18:07:57.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.495+0000 7fbd13600640 1 Processor -- start 2024-01-15T18:07:57.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.495+0000 7fbd13600640 1 -- start start 2024-01-15T18:07:57.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.499+0000 7fbd13600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbd0c15d190 0x7fbd0c0cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:57.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.499+0000 7fbd13600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbd0c06c520 0x7fbd0c0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:57.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.499+0000 7fbd13600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd0c0b0d70 con 0x7fbd0c15d570 2024-01-15T18:07:57.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.499+0000 7fbd13600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd0c0d0540 con 0x7fbd0c15d190 2024-01-15T18:07:57.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.499+0000 7fbd13600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c0b0440 con 0x7fbd0c06c520 2024-01-15T18:07:57.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.499+0000 7fbd10c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbd0c15d190 0x7fbd0c0cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:57.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.499+0000 7fbd10c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbd0c15d190 0x7fbd0c0cfcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60036/0 (socket says 172.21.15.27:60036) 2024-01-15T18:07:57.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.499+0000 7fbd10c00640 1 -- 172.21.15.27:0/3487411313 learned_addr learned my addr 172.21.15.27:0/3487411313 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:57.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.499+0000 7fbd11600640 1 --2- 172.21.15.27:0/3487411313 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbd0c06c520 0x7fbd0c0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:57.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.499+0000 7fbd10c00640 1 -- 172.21.15.27:0/3487411313 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbd0c06c520 msgr2=0x7fbd0c0cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:57.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.499+0000 7fbd10c00640 1 --2- 172.21.15.27:0/3487411313 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbd0c06c520 0x7fbd0c0cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:57.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.499+0000 7fbd10c00640 1 -- 172.21.15.27:0/3487411313 >> v1:172.21.15.93:6789/0 conn(0x7fbd0c15d570 legacy=0x7fbd0c06be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:57.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.499+0000 7fbd10c00640 1 -- 172.21.15.27:0/3487411313 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd0c0b0720 con 0x7fbd0c15d190 2024-01-15T18:07:57.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.499+0000 7fbd10c00640 1 --2- 172.21.15.27:0/3487411313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c15d190 0x7fbd0c0cfcf0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc009870 tx=0x7fbcfc01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c50b11551f14dc2d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:57.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.503+0000 7fbd0b400640 1 -- 172.21.15.27:0/3487411313 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc0aa430 con 0x7fbd0c15d190 2024-01-15T18:07:57.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.503+0000 7fbd0b400640 1 -- 172.21.15.27:0/3487411313 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcfc0aa590 con 0x7fbd0c15d190 2024-01-15T18:07:57.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.503+0000 7fbd0b400640 1 -- 172.21.15.27:0/3487411313 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc0aa920 con 0x7fbd0c15d190 2024-01-15T18:07:57.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.503+0000 7fbd13600640 1 -- 172.21.15.27:0/3487411313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c15d190 msgr2=0x7fbd0c0cfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:57.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.503+0000 7fbd13600640 1 --2- 172.21.15.27:0/3487411313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c15d190 0x7fbd0c0cfcf0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc009870 tx=0x7fbcfc01c510 comp rx=0 tx=0).stop 2024-01-15T18:07:57.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.503+0000 7fbd13600640 1 -- 172.21.15.27:0/3487411313 shutdown_connections 2024-01-15T18:07:57.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.503+0000 7fbd13600640 1 --2- 172.21.15.27:0/3487411313 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbd0c06c520 0x7fbd0c0cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:57.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.503+0000 7fbd13600640 1 --2- 172.21.15.27:0/3487411313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c15d190 0x7fbd0c0cfcf0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:57.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.503+0000 7fbd13600640 1 -- 172.21.15.27:0/3487411313 >> 172.21.15.27:0/3487411313 conn(0x7fbd0c0d0e70 msgr2=0x7fbd0c0d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:57.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.503+0000 7fbd13600640 1 -- 172.21.15.27:0/3487411313 shutdown_connections 2024-01-15T18:07:57.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.503+0000 7fbd13600640 1 -- 172.21.15.27:0/3487411313 wait complete. 2024-01-15T18:07:57.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.503+0000 7fbd13600640 1 Processor -- start 2024-01-15T18:07:57.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.503+0000 7fbd13600640 1 -- start start 2024-01-15T18:07:57.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.503+0000 7fbd13600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbd0c06c520 0x7fbd0c0b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:57.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.503+0000 7fbd13600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbd0c15d190 0x7fbd0c148300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:57.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.503+0000 7fbd13600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c15d570 0x7fbd0c14ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:57.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.503+0000 7fbd13600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c14fdc0 con 0x7fbd0c15d570 2024-01-15T18:07:57.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.503+0000 7fbd13600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c0b4200 con 0x7fbd0c06c520 2024-01-15T18:07:57.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.503+0000 7fbd13600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0c0b69f0 con 0x7fbd0c15d190 2024-01-15T18:07:57.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.507+0000 7fbd11600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c15d570 0x7fbd0c14ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:57.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.507+0000 7fbd11600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c15d570 0x7fbd0c14ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60052/0 (socket says 172.21.15.27:60052) 2024-01-15T18:07:57.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.507+0000 7fbd11600640 1 -- 172.21.15.27:0/3690890902 learned_addr learned my addr 172.21.15.27:0/3690890902 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:57.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.507+0000 7fbd10c00640 1 --2- 172.21.15.27:0/3690890902 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbd0c06c520 0x7fbd0c0b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:57.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.507+0000 7fbd0be00640 1 --2- 172.21.15.27:0/3690890902 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbd0c15d190 0x7fbd0c148300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:57.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.507+0000 7fbd11600640 1 -- 172.21.15.27:0/3690890902 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbd0c15d190 msgr2=0x7fbd0c148300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:57.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.507+0000 7fbd11600640 1 --2- 172.21.15.27:0/3690890902 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbd0c15d190 0x7fbd0c148300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:57.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.507+0000 7fbd11600640 1 -- 172.21.15.27:0/3690890902 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbd0c06c520 msgr2=0x7fbd0c0b3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:57.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.507+0000 7fbd11600640 1 --2- 172.21.15.27:0/3690890902 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbd0c06c520 0x7fbd0c0b3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:57.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.507+0000 7fbd11600640 1 -- 172.21.15.27:0/3690890902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcfc0094d0 con 0x7fbd0c15d570 2024-01-15T18:07:57.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.507+0000 7fbd10c00640 1 --2- 172.21.15.27:0/3690890902 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbd0c06c520 0x7fbd0c0b3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:07:57.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.507+0000 7fbd11600640 1 --2- 172.21.15.27:0/3690890902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c15d570 0x7fbd0c14ad20 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fbd04007de0 tx=0x7fbd0400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:57.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.507+0000 7fbd08c00640 1 -- 172.21.15.27:0/3690890902 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd04014620 con 0x7fbd0c15d570 2024-01-15T18:07:57.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.507+0000 7fbd08c00640 1 -- 172.21.15.27:0/3690890902 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd0400ee90 con 0x7fbd0c15d570 2024-01-15T18:07:57.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.507+0000 7fbd13600640 1 -- 172.21.15.27:0/3690890902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd0c150060 con 0x7fbd0c15d570 2024-01-15T18:07:57.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.507+0000 7fbd08c00640 1 -- 172.21.15.27:0/3690890902 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd040149e0 con 0x7fbd0c15d570 2024-01-15T18:07:57.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.507+0000 7fbd13600640 1 -- 172.21.15.27:0/3690890902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd0c0b6cd0 con 0x7fbd0c15d570 2024-01-15T18:07:57.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.507+0000 7fbd13600640 1 -- 172.21.15.27:0/3690890902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd0c04e850 con 0x7fbd0c15d570 2024-01-15T18:07:57.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.507+0000 7fbd08c00640 1 -- 172.21.15.27:0/3690890902 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fbd04014b40 con 0x7fbd0c15d570 2024-01-15T18:07:57.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.511+0000 7fbd08c00640 1 --2- 172.21.15.27:0/3690890902 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbce4066bc0 0x7fbce4069080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:57.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.511+0000 7fbd08c00640 1 -- 172.21.15.27:0/3690890902 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fbd04085180 con 0x7fbd0c15d570 2024-01-15T18:07:57.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.511+0000 7fbd10c00640 1 --2- 172.21.15.27:0/3690890902 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbce4066bc0 0x7fbce4069080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:57.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.511+0000 7fbd10c00640 1 --2- 172.21.15.27:0/3690890902 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbce4066bc0 0x7fbce4069080 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc0ae000 tx=0x7fbcfc0b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:57.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.515+0000 7fbd08c00640 1 -- 172.21.15.27:0/3690890902 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbd04052110 con 0x7fbd0c15d570 2024-01-15T18:07:57.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.671+0000 7fbd13600640 1 -- 172.21.15.27:0/3690890902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7fbd0c150510 con 0x7fbd0c15d570 2024-01-15T18:07:57.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.675+0000 7fbd08c00640 1 -- 172.21.15.27:0/3690890902 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v15)=0 v15) v1 ==== 131+0+0 (secure 0 0 0) 0x7fbd04055730 con 0x7fbd0c15d570 2024-01-15T18:07:57.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.675+0000 7fbd13600640 1 -- 172.21.15.27:0/3690890902 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbce4066bc0 msgr2=0x7fbce4069080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:57.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.675+0000 7fbd13600640 1 --2- 172.21.15.27:0/3690890902 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbce4066bc0 0x7fbce4069080 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc0ae000 tx=0x7fbcfc0b5040 comp rx=0 tx=0).stop 2024-01-15T18:07:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.675+0000 7fbd13600640 1 -- 172.21.15.27:0/3690890902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c15d570 msgr2=0x7fbd0c14ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.675+0000 7fbd13600640 1 --2- 172.21.15.27:0/3690890902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c15d570 0x7fbd0c14ad20 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fbd04007de0 tx=0x7fbd0400e4b0 comp rx=0 tx=0).stop 2024-01-15T18:07:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.675+0000 7fbd13600640 1 -- 172.21.15.27:0/3690890902 shutdown_connections 2024-01-15T18:07:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.675+0000 7fbd13600640 1 --2- 172.21.15.27:0/3690890902 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbce4066bc0 0x7fbce4069080 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.675+0000 7fbd13600640 1 --2- 172.21.15.27:0/3690890902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd0c15d570 0x7fbd0c14ad20 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.675+0000 7fbd13600640 1 --2- 172.21.15.27:0/3690890902 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbd0c15d190 0x7fbd0c148300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.675+0000 7fbd13600640 1 --2- 172.21.15.27:0/3690890902 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbd0c06c520 0x7fbd0c0b3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.675+0000 7fbd13600640 1 -- 172.21.15.27:0/3690890902 >> 172.21.15.27:0/3690890902 conn(0x7fbd0c0d0e70 msgr2=0x7fbd0c0ae3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.675+0000 7fbd13600640 1 -- 172.21.15.27:0/3690890902 shutdown_connections 2024-01-15T18:07:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.675+0000 7fbd13600640 1 -- 172.21.15.27:0/3690890902 wait complete. 2024-01-15T18:07:57.701 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 2 2024-01-15T18:07:57.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.839+0000 7f8ab3e00640 1 Processor -- start 2024-01-15T18:07:57.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.839+0000 7f8ab3e00640 1 -- start start 2024-01-15T18:07:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.843+0000 7f8ab3e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8aac0d1a40 0x7f8aac0cba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.843+0000 7f8ab3e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f8aac0b0650 0x7f8aac0b0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:57.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.843+0000 7f8ab3e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aac0cbfa0 con 0x7f8aac0cf450 2024-01-15T18:07:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.843+0000 7f8ab3e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aac0cefa0 con 0x7f8aac0b0650 2024-01-15T18:07:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.843+0000 7f8ab3e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac0ce240 con 0x7f8aac0d1a40 2024-01-15T18:07:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.843+0000 7f8ab1400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8aac0cf450 0x7f8aac0cf830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35026/0 (socket says 172.21.15.27:35026) 2024-01-15T18:07:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.843+0000 7f8ab1400640 1 -- 172.21.15.27:0/2443783566 learned_addr learned my addr 172.21.15.27:0/2443783566 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.843+0000 7f8ab1e00640 1 --2- 172.21.15.27:0/2443783566 >> v2:172.21.15.93:3300/0 conn(0x7f8aac0b0650 0x7f8aac0b0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:57.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.843+0000 7f8ab0a00640 1 --2- 172.21.15.27:0/2443783566 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8aac0d1a40 0x7f8aac0cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.843+0000 7f8aabe00640 1 -- 172.21.15.27:0/2443783566 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3427064738 0 0) 0x7f8aac0cbfa0 con 0x7f8aac0cf450 2024-01-15T18:07:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.843+0000 7f8ab1e00640 1 -- 172.21.15.27:0/2443783566 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8aac0d1a40 msgr2=0x7f8aac0cba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.843+0000 7f8ab1e00640 1 --2- 172.21.15.27:0/2443783566 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8aac0d1a40 0x7f8aac0cba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.843+0000 7f8ab1e00640 1 -- 172.21.15.27:0/2443783566 >> v1:172.21.15.27:6789/0 conn(0x7f8aac0cf450 legacy=0x7f8aac0cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.843+0000 7f8ab1e00640 1 -- 172.21.15.27:0/2443783566 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aac0ccd70 con 0x7f8aac0b0650 2024-01-15T18:07:57.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.843+0000 7f8ab0a00640 1 --2- 172.21.15.27:0/2443783566 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8aac0d1a40 0x7f8aac0cba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:07:57.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.847+0000 7f8ab1e00640 1 --2- 172.21.15.27:0/2443783566 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8aac0b0650 0x7f8aac0b0ae0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4009d90 tx=0x7f8aa401c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73b0ed94fab9ecb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:57.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.847+0000 7f8aabe00640 1 -- 172.21.15.27:0/2443783566 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa40a5810 con 0x7f8aac0b0650 2024-01-15T18:07:57.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.847+0000 7f8aabe00640 1 -- 172.21.15.27:0/2443783566 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa40a5970 con 0x7f8aac0b0650 2024-01-15T18:07:57.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.847+0000 7f8aabe00640 1 -- 172.21.15.27:0/2443783566 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa40a5d00 con 0x7f8aac0b0650 2024-01-15T18:07:57.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.847+0000 7f8ab3e00640 1 -- 172.21.15.27:0/2443783566 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8aac0b0650 msgr2=0x7f8aac0b0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:57.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.847+0000 7f8ab3e00640 1 --2- 172.21.15.27:0/2443783566 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8aac0b0650 0x7f8aac0b0ae0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4009d90 tx=0x7f8aa401c950 comp rx=0 tx=0).stop 2024-01-15T18:07:57.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.847+0000 7f8ab3e00640 1 -- 172.21.15.27:0/2443783566 shutdown_connections 2024-01-15T18:07:57.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.847+0000 7f8ab3e00640 1 --2- 172.21.15.27:0/2443783566 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8aac0d1a40 0x7f8aac0cba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.847+0000 7f8ab3e00640 1 --2- 172.21.15.27:0/2443783566 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8aac0b0650 0x7f8aac0b0ae0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.847+0000 7f8ab3e00640 1 -- 172.21.15.27:0/2443783566 >> 172.21.15.27:0/2443783566 conn(0x7f8aac0d0e70 msgr2=0x7f8aac0d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.847+0000 7f8ab3e00640 1 -- 172.21.15.27:0/2443783566 shutdown_connections 2024-01-15T18:07:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.847+0000 7f8ab3e00640 1 -- 172.21.15.27:0/2443783566 wait complete. 2024-01-15T18:07:57.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.847+0000 7f8ab3e00640 1 Processor -- start 2024-01-15T18:07:57.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8ab3e00640 1 -- start start 2024-01-15T18:07:57.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8ab3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8aac0b0650 0x7f8aac158670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8ab3e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8aac0cf450 0x7f8aac15d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8ab3e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8aac0d1a40 0x7f8aac15d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8ab3e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac0c8be0 con 0x7f8aac0b0650 2024-01-15T18:07:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8ab3e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac0cae60 con 0x7f8aac0cf450 2024-01-15T18:07:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8ab3e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac164b90 con 0x7f8aac0d1a40 2024-01-15T18:07:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8ab1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8aac0b0650 0x7f8aac158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8ab0a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8aac0cf450 0x7f8aac15d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8ab0a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8aac0cf450 0x7f8aac15d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:56766/0 (socket says 172.21.15.27:56766) 2024-01-15T18:07:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8ab0a00640 1 -- 172.21.15.27:0/3163501096 learned_addr learned my addr 172.21.15.27:0/3163501096 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8ab1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8aac0b0650 0x7f8aac158670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60060/0 (socket says 172.21.15.27:60060) 2024-01-15T18:07:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8ab1e00640 1 --2- 172.21.15.27:0/3163501096 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8aac0d1a40 0x7f8aac15d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8ab1400640 1 -- 172.21.15.27:0/3163501096 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8aac0d1a40 msgr2=0x7f8aac15d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8ab1400640 1 --2- 172.21.15.27:0/3163501096 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8aac0d1a40 0x7f8aac15d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8ab1400640 1 -- 172.21.15.27:0/3163501096 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8aac0cf450 msgr2=0x7f8aac15d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8ab1400640 1 --2- 172.21.15.27:0/3163501096 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8aac0cf450 0x7f8aac15d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8ab1400640 1 -- 172.21.15.27:0/3163501096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a94009420 con 0x7f8aac0b0650 2024-01-15T18:07:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8ab0a00640 1 --2- 172.21.15.27:0/3163501096 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8aac0cf450 0x7f8aac15d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:07:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8ab1400640 1 --2- 172.21.15.27:0/3163501096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8aac0b0650 0x7f8aac158670 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c00c410 tx=0x7f8a9c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8aa9600640 1 -- 172.21.15.27:0/3163501096 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c0074e0 con 0x7f8aac0b0650 2024-01-15T18:07:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8aa9600640 1 -- 172.21.15.27:0/3163501096 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9c007640 con 0x7f8aac0b0650 2024-01-15T18:07:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8ab3e00640 1 -- 172.21.15.27:0/3163501096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aa4009a70 con 0x7f8aac0b0650 2024-01-15T18:07:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8aa9600640 1 -- 172.21.15.27:0/3163501096 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c004a30 con 0x7f8aac0b0650 2024-01-15T18:07:57.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8ab3e00640 1 -- 172.21.15.27:0/3163501096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aac0ca660 con 0x7f8aac0b0650 2024-01-15T18:07:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.851+0000 7f8ab3e00640 1 -- 172.21.15.27:0/3163501096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aac04e850 con 0x7f8aac0b0650 2024-01-15T18:07:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.855+0000 7f8aa9600640 1 -- 172.21.15.27:0/3163501096 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f8a9c004b90 con 0x7f8aac0b0650 2024-01-15T18:07:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.855+0000 7f8aa9600640 1 --2- 172.21.15.27:0/3163501096 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8a84066bc0 0x7f8a84069080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.855+0000 7f8aa9600640 1 -- 172.21.15.27:0/3163501096 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8a9c084490 con 0x7f8aac0b0650 2024-01-15T18:07:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.855+0000 7f8ab0a00640 1 --2- 172.21.15.27:0/3163501096 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8a84066bc0 0x7f8a84069080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.855+0000 7f8ab0a00640 1 --2- 172.21.15.27:0/3163501096 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8a84066bc0 0x7f8a84069080 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8a94009fd0 tx=0x7f8a94009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:57.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:57.859+0000 7f8aa9600640 1 -- 172.21.15.27:0/3163501096 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8a9c0514e0 con 0x7f8aac0b0650 2024-01-15T18:07:58.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.019+0000 7f8ab3e00640 1 -- 172.21.15.27:0/3163501096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f8aac0b6bf0 con 0x7f8aac0b0650 2024-01-15T18:07:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.023+0000 7f8aa9600640 1 -- 172.21.15.27:0/3163501096 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v16)=0 v16) v1 ==== 135+0+0 (secure 0 0 0) 0x7f8a9c054b00 con 0x7f8aac0b0650 2024-01-15T18:07:58.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.023+0000 7f8ab3e00640 1 -- 172.21.15.27:0/3163501096 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8a84066bc0 msgr2=0x7f8a84069080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:58.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.023+0000 7f8ab3e00640 1 --2- 172.21.15.27:0/3163501096 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8a84066bc0 0x7f8a84069080 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8a94009fd0 tx=0x7f8a94009190 comp rx=0 tx=0).stop 2024-01-15T18:07:58.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.023+0000 7f8ab3e00640 1 -- 172.21.15.27:0/3163501096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8aac0b0650 msgr2=0x7f8aac158670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:58.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.023+0000 7f8ab3e00640 1 --2- 172.21.15.27:0/3163501096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8aac0b0650 0x7f8aac158670 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c00c410 tx=0x7f8a9c00c8d0 comp rx=0 tx=0).stop 2024-01-15T18:07:58.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.023+0000 7f8ab3e00640 1 -- 172.21.15.27:0/3163501096 shutdown_connections 2024-01-15T18:07:58.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.023+0000 7f8ab3e00640 1 --2- 172.21.15.27:0/3163501096 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8a84066bc0 0x7f8a84069080 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.023+0000 7f8ab3e00640 1 --2- 172.21.15.27:0/3163501096 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8aac0d1a40 0x7f8aac15d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.023+0000 7f8ab3e00640 1 --2- 172.21.15.27:0/3163501096 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8aac0cf450 0x7f8aac15d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.023+0000 7f8ab3e00640 1 --2- 172.21.15.27:0/3163501096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8aac0b0650 0x7f8aac158670 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.023+0000 7f8ab3e00640 1 -- 172.21.15.27:0/3163501096 >> 172.21.15.27:0/3163501096 conn(0x7f8aac0d0e70 msgr2=0x7f8aac15e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.023+0000 7f8ab3e00640 1 -- 172.21.15.27:0/3163501096 shutdown_connections 2024-01-15T18:07:58.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.023+0000 7f8ab3e00640 1 -- 172.21.15.27:0/3163501096 wait complete. 2024-01-15T18:07:58.049 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-15T18:07:58.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.191+0000 7fdd2bc00640 1 Processor -- start 2024-01-15T18:07:58.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.191+0000 7fdd2bc00640 1 -- start start 2024-01-15T18:07:58.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.191+0000 7fdd2bc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdd2409a5b0 0x7fdd240cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:58.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.191+0000 7fdd2bc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd24099a10 0x7fdd24099e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:58.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.191+0000 7fdd2bc00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd2409a3a0 con 0x7fdd2415b350 2024-01-15T18:07:58.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.191+0000 7fdd2bc00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd240ccd50 con 0x7fdd2409a5b0 2024-01-15T18:07:58.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.191+0000 7fdd2bc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd240cebc0 con 0x7fdd24099a10 2024-01-15T18:07:58.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.191+0000 7fdd23e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdd2409a5b0 0x7fdd240cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:58.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.191+0000 7fdd29200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdd2415b350 0x7fdd240cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35040/0 (socket says 172.21.15.27:35040) 2024-01-15T18:07:58.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.191+0000 7fdd29200640 1 -- 172.21.15.27:0/2620505945 learned_addr learned my addr 172.21.15.27:0/2620505945 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:07:58.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.191+0000 7fdd23e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdd2409a5b0 0x7fdd240cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60062/0 (socket says 172.21.15.27:60062) 2024-01-15T18:07:58.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.191+0000 7fdd29c00640 1 --2- 172.21.15.27:0/2620505945 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd24099a10 0x7fdd24099e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:58.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.191+0000 7fdd23e00640 1 -- 172.21.15.27:0/2620505945 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd24099a10 msgr2=0x7fdd24099e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:58.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.191+0000 7fdd23e00640 1 --2- 172.21.15.27:0/2620505945 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd24099a10 0x7fdd24099e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:58.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.191+0000 7fdd23e00640 1 -- 172.21.15.27:0/2620505945 >> v1:172.21.15.27:6789/0 conn(0x7fdd2415b350 legacy=0x7fdd240cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:58.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.191+0000 7fdd23e00640 1 -- 172.21.15.27:0/2620505945 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd240ceea0 con 0x7fdd2409a5b0 2024-01-15T18:07:58.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.195+0000 7fdd23e00640 1 --2- 172.21.15.27:0/2620505945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd2409a5b0 0x7fdd240cc500 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c0097c0 tx=0x7fdd0c01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44bc2510255b33ba server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:58.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.195+0000 7fdd23400640 1 -- 172.21.15.27:0/2620505945 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c0ac620 con 0x7fdd2409a5b0 2024-01-15T18:07:58.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.195+0000 7fdd23400640 1 -- 172.21.15.27:0/2620505945 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd0c0ac780 con 0x7fdd2409a5b0 2024-01-15T18:07:58.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.195+0000 7fdd23400640 1 -- 172.21.15.27:0/2620505945 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c0acb10 con 0x7fdd2409a5b0 2024-01-15T18:07:58.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.195+0000 7fdd2bc00640 1 -- 172.21.15.27:0/2620505945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd2409a5b0 msgr2=0x7fdd240cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:58.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.195+0000 7fdd2bc00640 1 --2- 172.21.15.27:0/2620505945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd2409a5b0 0x7fdd240cc500 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c0097c0 tx=0x7fdd0c01c3c0 comp rx=0 tx=0).stop 2024-01-15T18:07:58.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.195+0000 7fdd2bc00640 1 -- 172.21.15.27:0/2620505945 shutdown_connections 2024-01-15T18:07:58.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.195+0000 7fdd2bc00640 1 --2- 172.21.15.27:0/2620505945 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd24099a10 0x7fdd24099e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:58.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.195+0000 7fdd2bc00640 1 --2- 172.21.15.27:0/2620505945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd2409a5b0 0x7fdd240cc500 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:58.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.195+0000 7fdd2bc00640 1 -- 172.21.15.27:0/2620505945 >> 172.21.15.27:0/2620505945 conn(0x7fdd240b2a80 msgr2=0x7fdd240b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:58.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.195+0000 7fdd2bc00640 1 -- 172.21.15.27:0/2620505945 shutdown_connections 2024-01-15T18:07:58.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.195+0000 7fdd2bc00640 1 -- 172.21.15.27:0/2620505945 wait complete. 2024-01-15T18:07:58.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.199+0000 7fdd2bc00640 1 Processor -- start 2024-01-15T18:07:58.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.199+0000 7fdd2bc00640 1 -- start start 2024-01-15T18:07:58.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.199+0000 7fdd2bc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd24099a10 0x7fdd24153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:58.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.199+0000 7fdd2bc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd2409a5b0 0x7fdd24154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:58.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.199+0000 7fdd2bc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd2415b350 0x7fdd24154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:58.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.199+0000 7fdd2bc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd240c8790 con 0x7fdd2409a5b0 2024-01-15T18:07:58.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.199+0000 7fdd2bc00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd240ca690 con 0x7fdd2415b350 2024-01-15T18:07:58.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.199+0000 7fdd2bc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd240cb380 con 0x7fdd24099a10 2024-01-15T18:07:58.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.199+0000 7fdd23e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd2409a5b0 0x7fdd24154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:58.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.199+0000 7fdd23e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd2409a5b0 0x7fdd24154520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60068/0 (socket says 172.21.15.27:60068) 2024-01-15T18:07:58.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.199+0000 7fdd23e00640 1 -- 172.21.15.27:0/1232798374 learned_addr learned my addr 172.21.15.27:0/1232798374 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:58.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.199+0000 7fdd29c00640 1 --2- 172.21.15.27:0/1232798374 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd2415b350 0x7fdd24154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:58.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.199+0000 7fdd29200640 1 --2- 172.21.15.27:0/1232798374 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd24099a10 0x7fdd24153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:58.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.199+0000 7fdd23e00640 1 -- 172.21.15.27:0/1232798374 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd24099a10 msgr2=0x7fdd24153fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:58.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.199+0000 7fdd23e00640 1 --2- 172.21.15.27:0/1232798374 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd24099a10 0x7fdd24153fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:58.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.199+0000 7fdd23e00640 1 -- 172.21.15.27:0/1232798374 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd2415b350 msgr2=0x7fdd24154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:58.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.199+0000 7fdd23e00640 1 --2- 172.21.15.27:0/1232798374 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd2415b350 0x7fdd24154a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:58.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.199+0000 7fdd23e00640 1 -- 172.21.15.27:0/1232798374 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd0c009420 con 0x7fdd2409a5b0 2024-01-15T18:07:58.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.199+0000 7fdd23e00640 1 --2- 172.21.15.27:0/1232798374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd2409a5b0 0x7fdd24154520 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c0af040 tx=0x7fdd0c019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:58.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.199+0000 7fdd20c00640 1 -- 172.21.15.27:0/1232798374 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c0b5410 con 0x7fdd2409a5b0 2024-01-15T18:07:58.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.199+0000 7fdd20c00640 1 -- 172.21.15.27:0/1232798374 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd0c0b5570 con 0x7fdd2409a5b0 2024-01-15T18:07:58.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.199+0000 7fdd20c00640 1 -- 172.21.15.27:0/1232798374 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c0b5920 con 0x7fdd2409a5b0 2024-01-15T18:07:58.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.199+0000 7fdd2bc00640 1 -- 172.21.15.27:0/1232798374 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd240c84b0 con 0x7fdd2409a5b0 2024-01-15T18:07:58.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.203+0000 7fdd2bc00640 1 -- 172.21.15.27:0/1232798374 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd240cab80 con 0x7fdd2409a5b0 2024-01-15T18:07:58.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.203+0000 7fdd2bc00640 1 -- 172.21.15.27:0/1232798374 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd240d0720 con 0x7fdd2409a5b0 2024-01-15T18:07:58.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.203+0000 7fdd20c00640 1 -- 172.21.15.27:0/1232798374 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) v1 ==== 86415+0+0 (secure 0 0 0) 0x7fdd0c0b3040 con 0x7fdd2409a5b0 2024-01-15T18:07:58.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.203+0000 7fdd20c00640 1 --2- 172.21.15.27:0/1232798374 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdcfc066bc0 0x7fdcfc069080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:58.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.203+0000 7fdd20c00640 1 -- 172.21.15.27:0/1232798374 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fdd0c11ebd0 con 0x7fdd2409a5b0 2024-01-15T18:07:58.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.203+0000 7fdd29200640 1 --2- 172.21.15.27:0/1232798374 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdcfc066bc0 0x7fdcfc069080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:58.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.203+0000 7fdd29200640 1 --2- 172.21.15.27:0/1232798374 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdcfc066bc0 0x7fdcfc069080 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fdd14005890 tx=0x7fdd14009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:58.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.207+0000 7fdd20c00640 1 -- 172.21.15.27:0/1232798374 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fdd0c0eb8d0 con 0x7fdd2409a5b0 2024-01-15T18:07:58.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.395+0000 7fdd2bc00640 1 -- 172.21.15.27:0/1232798374 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd2404e850 con 0x7fdcfc066bc0 2024-01-15T18:07:58.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.407+0000 7fdd20c00640 1 -- 172.21.15.27:0/1232798374 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdd2404e850 con 0x7fdcfc066bc0 2024-01-15T18:07:58.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.411+0000 7fdd2bc00640 1 -- 172.21.15.27:0/1232798374 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdcfc066bc0 msgr2=0x7fdcfc069080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:58.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.411+0000 7fdd2bc00640 1 --2- 172.21.15.27:0/1232798374 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdcfc066bc0 0x7fdcfc069080 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fdd14005890 tx=0x7fdd14009040 comp rx=0 tx=0).stop 2024-01-15T18:07:58.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.411+0000 7fdd2bc00640 1 -- 172.21.15.27:0/1232798374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd2409a5b0 msgr2=0x7fdd24154520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:58.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.411+0000 7fdd2bc00640 1 --2- 172.21.15.27:0/1232798374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd2409a5b0 0x7fdd24154520 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c0af040 tx=0x7fdd0c019160 comp rx=0 tx=0).stop 2024-01-15T18:07:58.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.411+0000 7fdd2bc00640 1 -- 172.21.15.27:0/1232798374 shutdown_connections 2024-01-15T18:07:58.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.411+0000 7fdd2bc00640 1 --2- 172.21.15.27:0/1232798374 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdcfc066bc0 0x7fdcfc069080 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:58.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.411+0000 7fdd2bc00640 1 --2- 172.21.15.27:0/1232798374 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd2415b350 0x7fdd24154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:58.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.411+0000 7fdd2bc00640 1 --2- 172.21.15.27:0/1232798374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd2409a5b0 0x7fdd24154520 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:58.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.411+0000 7fdd2bc00640 1 --2- 172.21.15.27:0/1232798374 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd24099a10 0x7fdd24153fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:58.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.411+0000 7fdd2bc00640 1 -- 172.21.15.27:0/1232798374 >> 172.21.15.27:0/1232798374 conn(0x7fdd240b2a80 msgr2=0x7fdd240cb9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:58.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.411+0000 7fdd2bc00640 1 -- 172.21.15.27:0/1232798374 shutdown_connections 2024-01-15T18:07:58.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.411+0000 7fdd2bc00640 1 -- 172.21.15.27:0/1232798374 wait complete. 2024-01-15T18:07:58.437 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:07:58.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.575+0000 7ffaffe00640 1 Processor -- start 2024-01-15T18:07:58.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.575+0000 7ffaffe00640 1 -- start start 2024-01-15T18:07:58.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.575+0000 7ffaffe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffaf806c430 0x7ffaf8070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:58.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.575+0000 7ffaffe00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ffaf80b1db0 0x7ffaf80b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:58.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.579+0000 7ffaffe00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf80ccfb0 con 0x7ffaf80ce050 2024-01-15T18:07:58.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.579+0000 7ffaffe00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf80cd290 con 0x7ffaf80b1db0 2024-01-15T18:07:58.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.579+0000 7ffaffe00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf80cd570 con 0x7ffaf806c430 2024-01-15T18:07:58.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.579+0000 7ffafd400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffaf806c430 0x7ffaf8070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:58.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.579+0000 7ffafd400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffaf806c430 0x7ffaf8070ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:40530/0 (socket says 172.21.15.27:40530) 2024-01-15T18:07:58.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.579+0000 7ffafd400640 1 -- 172.21.15.27:0/25379037 learned_addr learned my addr 172.21.15.27:0/25379037 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:58.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.579+0000 7ffafde00640 1 --1- 172.21.15.27:0/25379037 >> v1:172.21.15.93:6789/0 conn(0x7ffaf80ce050 0x7ffaf806c810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:60280/0 (socket says 172.21.15.27:60280) 2024-01-15T18:07:58.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.579+0000 7ffafca00640 1 --2- 172.21.15.27:0/25379037 >> v2:172.21.15.93:3300/0 conn(0x7ffaf80b1db0 0x7ffaf80b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:58.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.579+0000 7ffafd400640 1 -- 172.21.15.27:0/25379037 >> v2:172.21.15.93:3300/0 conn(0x7ffaf80b1db0 msgr2=0x7ffaf80b2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:58.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.579+0000 7ffafd400640 1 --2- 172.21.15.27:0/25379037 >> v2:172.21.15.93:3300/0 conn(0x7ffaf80b1db0 0x7ffaf80b2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:58.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.579+0000 7ffafd400640 1 -- 172.21.15.27:0/25379037 >> v1:172.21.15.93:6789/0 conn(0x7ffaf80ce050 legacy=0x7ffaf806c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:58.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.579+0000 7ffaf7e00640 1 -- 172.21.15.27:0/25379037 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2664399024 0 0) 0x7ffaf80ccfb0 con 0x7ffaf80ce050 2024-01-15T18:07:58.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.579+0000 7ffafd400640 1 -- 172.21.15.27:0/25379037 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf80cd850 con 0x7ffaf806c430 2024-01-15T18:07:58.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.579+0000 7ffafca00640 1 --2- 172.21.15.27:0/25379037 >> v2:172.21.15.93:3300/0 conn(0x7ffaf80b1db0 0x7ffaf80b2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:07:58.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.579+0000 7ffafd400640 1 --2- 172.21.15.27:0/25379037 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffaf806c430 0x7ffaf8070ba0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffae8009870 tx=0x7ffae801c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38749b107d0f017a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:58.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.583+0000 7ffaf7e00640 1 -- 172.21.15.27:0/25379037 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae80ab600 con 0x7ffaf806c430 2024-01-15T18:07:58.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.583+0000 7ffaf7e00640 1 -- 172.21.15.27:0/25379037 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae80a7050 con 0x7ffaf806c430 2024-01-15T18:07:58.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.583+0000 7ffaf7e00640 1 -- 172.21.15.27:0/25379037 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae80ab980 con 0x7ffaf806c430 2024-01-15T18:07:58.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.583+0000 7ffaffe00640 1 -- 172.21.15.27:0/25379037 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffaf806c430 msgr2=0x7ffaf8070ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:58.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.583+0000 7ffaffe00640 1 --2- 172.21.15.27:0/25379037 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffaf806c430 0x7ffaf8070ba0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffae8009870 tx=0x7ffae801c4b0 comp rx=0 tx=0).stop 2024-01-15T18:07:58.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.583+0000 7ffaffe00640 1 -- 172.21.15.27:0/25379037 shutdown_connections 2024-01-15T18:07:58.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.583+0000 7ffaffe00640 1 --2- 172.21.15.27:0/25379037 >> v2:172.21.15.93:3300/0 conn(0x7ffaf80b1db0 0x7ffaf80b2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:58.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.583+0000 7ffaffe00640 1 --2- 172.21.15.27:0/25379037 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffaf806c430 0x7ffaf8070ba0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:58.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.583+0000 7ffaffe00640 1 -- 172.21.15.27:0/25379037 >> 172.21.15.27:0/25379037 conn(0x7ffaf80cb700 msgr2=0x7ffaf80cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:58.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.583+0000 7ffaffe00640 1 -- 172.21.15.27:0/25379037 shutdown_connections 2024-01-15T18:07:58.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.583+0000 7ffaffe00640 1 -- 172.21.15.27:0/25379037 wait complete. 2024-01-15T18:07:58.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.583+0000 7ffaffe00640 1 Processor -- start 2024-01-15T18:07:58.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.583+0000 7ffaffe00640 1 -- start start 2024-01-15T18:07:58.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.583+0000 7ffaffe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffaf806c430 0x7ffaf815bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:58.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.583+0000 7ffaffe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffaf80b1db0 0x7ffaf81414e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:58.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.583+0000 7ffaffe00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ffaf80ce050 0x7ffaf80a8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:58.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.583+0000 7ffaffe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf815b8e0 con 0x7ffaf806c430 2024-01-15T18:07:58.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.583+0000 7ffaffe00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf815c240 con 0x7ffaf80ce050 2024-01-15T18:07:58.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.583+0000 7ffaffe00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf815ac60 con 0x7ffaf80b1db0 2024-01-15T18:07:58.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.587+0000 7ffafd400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffaf806c430 0x7ffaf815bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:58.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.587+0000 7ffafd400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffaf806c430 0x7ffaf815bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60070/0 (socket says 172.21.15.27:60070) 2024-01-15T18:07:58.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.587+0000 7ffafd400640 1 -- 172.21.15.27:0/3498115091 learned_addr learned my addr 172.21.15.27:0/3498115091 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:07:58.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.587+0000 7ffafca00640 1 --2- 172.21.15.27:0/3498115091 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffaf80b1db0 0x7ffaf81414e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:58.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.587+0000 7ffafde00640 1 --2- 172.21.15.27:0/3498115091 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ffaf80ce050 0x7ffaf80a8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:58.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.587+0000 7ffafd400640 1 -- 172.21.15.27:0/3498115091 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffaf80b1db0 msgr2=0x7ffaf81414e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:58.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.587+0000 7ffafd400640 1 --2- 172.21.15.27:0/3498115091 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffaf80b1db0 0x7ffaf81414e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:58.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.587+0000 7ffafd400640 1 -- 172.21.15.27:0/3498115091 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ffaf80ce050 msgr2=0x7ffaf80a8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:58.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.587+0000 7ffafd400640 1 --2- 172.21.15.27:0/3498115091 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ffaf80ce050 0x7ffaf80a8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:58.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.587+0000 7ffafd400640 1 -- 172.21.15.27:0/3498115091 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae000ba70 con 0x7ffaf806c430 2024-01-15T18:07:58.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.587+0000 7ffafd400640 1 --2- 172.21.15.27:0/3498115091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffaf806c430 0x7ffaf815bd00 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ffae80ae040 tx=0x7ffae8019250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:58.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.587+0000 7ffaf5600640 1 -- 172.21.15.27:0/3498115091 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae80b4440 con 0x7ffaf806c430 2024-01-15T18:07:58.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.587+0000 7ffaf5600640 1 -- 172.21.15.27:0/3498115091 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae80b45a0 con 0x7ffaf806c430 2024-01-15T18:07:58.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.587+0000 7ffaffe00640 1 -- 172.21.15.27:0/3498115091 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae80094d0 con 0x7ffaf806c430 2024-01-15T18:07:58.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.587+0000 7ffaf5600640 1 -- 172.21.15.27:0/3498115091 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae80b4950 con 0x7ffaf806c430 2024-01-15T18:07:58.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.587+0000 7ffaffe00640 1 -- 172.21.15.27:0/3498115091 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf815b220 con 0x7ffaf806c430 2024-01-15T18:07:58.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.587+0000 7ffaffe00640 1 -- 172.21.15.27:0/3498115091 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf80cb500 con 0x7ffaf806c430 2024-01-15T18:07:58.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.587+0000 7ffaf5600640 1 -- 172.21.15.27:0/3498115091 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) v1 ==== 86415+0+0 (secure 0 0 0) 0x7ffae80b2070 con 0x7ffaf806c430 2024-01-15T18:07:58.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.587+0000 7ffaf5600640 1 --2- 172.21.15.27:0/3498115091 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ffad0066bf0 0x7ffad00690b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:58.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.591+0000 7ffaf5600640 1 -- 172.21.15.27:0/3498115091 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ffae811e5c0 con 0x7ffaf806c430 2024-01-15T18:07:58.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.591+0000 7ffafca00640 1 --2- 172.21.15.27:0/3498115091 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ffad0066bf0 0x7ffad00690b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:58.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.591+0000 7ffafca00640 1 --2- 172.21.15.27:0/3498115091 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ffad0066bf0 0x7ffad00690b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ffae000bfd0 tx=0x7ffae000b750 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:58.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.591+0000 7ffaf5600640 1 -- 172.21.15.27:0/3498115091 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ffae80eb440 con 0x7ffaf806c430 2024-01-15T18:07:58.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.779+0000 7ffaffe00640 1 -- 172.21.15.27:0/3498115091 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7ffaf80af820 con 0x7ffad0066bf0 2024-01-15T18:07:58.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.783+0000 7ffaf5600640 1 -- 172.21.15.27:0/3498115091 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7ffaf80af820 con 0x7ffad0066bf0 2024-01-15T18:07:58.789 INFO:teuthology.orchestra.run.smithi027.stdout:/volumes/_nogroup/sv1/5d7c5818-d701-4425-8eb7-540607ec283f 2024-01-15T18:07:58.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.787+0000 7ffaffe00640 1 -- 172.21.15.27:0/3498115091 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ffad0066bf0 msgr2=0x7ffad00690b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:58.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.787+0000 7ffaffe00640 1 --2- 172.21.15.27:0/3498115091 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ffad0066bf0 0x7ffad00690b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ffae000bfd0 tx=0x7ffae000b750 comp rx=0 tx=0).stop 2024-01-15T18:07:58.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.787+0000 7ffaffe00640 1 -- 172.21.15.27:0/3498115091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffaf806c430 msgr2=0x7ffaf815bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:58.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.787+0000 7ffaffe00640 1 --2- 172.21.15.27:0/3498115091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffaf806c430 0x7ffaf815bd00 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ffae80ae040 tx=0x7ffae8019250 comp rx=0 tx=0).stop 2024-01-15T18:07:58.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.787+0000 7ffaffe00640 1 -- 172.21.15.27:0/3498115091 shutdown_connections 2024-01-15T18:07:58.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.787+0000 7ffaffe00640 1 --2- 172.21.15.27:0/3498115091 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ffad0066bf0 0x7ffad00690b0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:58.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.787+0000 7ffaffe00640 1 --2- 172.21.15.27:0/3498115091 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ffaf80ce050 0x7ffaf80a8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:58.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.787+0000 7ffaffe00640 1 --2- 172.21.15.27:0/3498115091 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffaf80b1db0 0x7ffaf81414e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:58.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.787+0000 7ffaffe00640 1 --2- 172.21.15.27:0/3498115091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffaf806c430 0x7ffaf815bd00 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:58.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.787+0000 7ffaffe00640 1 -- 172.21.15.27:0/3498115091 >> 172.21.15.27:0/3498115091 conn(0x7ffaf80cb700 msgr2=0x7ffaf80cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:58.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.787+0000 7ffaffe00640 1 -- 172.21.15.27:0/3498115091 shutdown_connections 2024-01-15T18:07:58.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:07:58.787+0000 7ffaffe00640 1 -- 172.21.15.27:0/3498115091 wait complete. 2024-01-15T18:07:58.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:07:58.811 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/5d7c5818-d701-4425-8eb7-540607ec283f) 2024-01-15T18:07:58.823 INFO:teuthology.orchestra.run.smithi027.stdout:# file: volumes/_nogroup/sv1/5d7c5818-d701-4425-8eb7-540607ec283f 2024-01-15T18:07:58.823 INFO:teuthology.orchestra.run.smithi027.stdout:ceph.dir.rbytes="3138256896" 2024-01-15T18:07:58.824 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:07:58.824 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-15T18:08:00.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:08:00.825 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/5d7c5818-d701-4425-8eb7-540607ec283f) 2024-01-15T18:08:00.838 INFO:teuthology.orchestra.run.smithi027.stdout:# file: volumes/_nogroup/sv1/5d7c5818-d701-4425-8eb7-540607ec283f 2024-01-15T18:08:00.838 INFO:teuthology.orchestra.run.smithi027.stdout:ceph.dir.rbytes="3221225472" 2024-01-15T18:08:00.838 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:08:00.839 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-15T18:08:00.839 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone63891 2024-01-15T18:08:01.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.023+0000 7f4715800640 1 Processor -- start 2024-01-15T18:08:01.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.023+0000 7f4715800640 1 -- start start 2024-01-15T18:08:01.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.023+0000 7f4715800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f471006c430 0x7f4710070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:01.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.023+0000 7f4715800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f47100b1db0 0x7f47100b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:01.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.023+0000 7f4715800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f47100ce050 0x7f47100ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:01.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.023+0000 7f4715800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47100cea20 con 0x7f47100b1db0 2024-01-15T18:08:01.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.023+0000 7f4715800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47100ccfb0 con 0x7f47100ce050 2024-01-15T18:08:01.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.023+0000 7f4715800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47100cd290 con 0x7f471006c430 2024-01-15T18:08:01.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.023+0000 7f470e000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f47100b1db0 0x7f47100b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:01.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.027+0000 7f470e000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f47100b1db0 0x7f47100b2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60084/0 (socket says 172.21.15.27:60084) 2024-01-15T18:08:01.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.027+0000 7f470e000640 1 -- 172.21.15.27:0/389554001 learned_addr learned my addr 172.21.15.27:0/389554001 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:01.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.027+0000 7f470f400640 1 --2- 172.21.15.27:0/389554001 >> v2:172.21.15.93:3300/0 conn(0x7f47100ce050 0x7f47100ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:01.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.027+0000 7f470ea00640 1 --2- 172.21.15.27:0/389554001 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f471006c430 0x7f4710070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:01.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.027+0000 7f470e000640 1 -- 172.21.15.27:0/389554001 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f471006c430 msgr2=0x7f4710070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:01.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.027+0000 7f470e000640 1 --2- 172.21.15.27:0/389554001 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f471006c430 0x7f4710070ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.027+0000 7f470e000640 1 -- 172.21.15.27:0/389554001 >> v2:172.21.15.93:3300/0 conn(0x7f47100ce050 msgr2=0x7f47100ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:01.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.027+0000 7f470e000640 1 --2- 172.21.15.27:0/389554001 >> v2:172.21.15.93:3300/0 conn(0x7f47100ce050 0x7f47100ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.027+0000 7f470e000640 1 -- 172.21.15.27:0/389554001 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47100cd570 con 0x7f47100b1db0 2024-01-15T18:08:01.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.027+0000 7f470f400640 1 --2- 172.21.15.27:0/389554001 >> v2:172.21.15.93:3300/0 conn(0x7f47100ce050 0x7f47100ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:08:01.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.027+0000 7f470e000640 1 --2- 172.21.15.27:0/389554001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47100b1db0 0x7f47100b2210 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f4710072330 tx=0x7f46f801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c42a83ff75d94010 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:01.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.027+0000 7f470d600640 1 -- 172.21.15.27:0/389554001 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f80ab600 con 0x7f47100b1db0 2024-01-15T18:08:01.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.027+0000 7f470d600640 1 -- 172.21.15.27:0/389554001 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f80a7050 con 0x7f47100b1db0 2024-01-15T18:08:01.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.027+0000 7f470d600640 1 -- 172.21.15.27:0/389554001 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f80ab980 con 0x7f47100b1db0 2024-01-15T18:08:01.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.031+0000 7f4715800640 1 -- 172.21.15.27:0/389554001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47100b1db0 msgr2=0x7f47100b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:01.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.031+0000 7f4715800640 1 --2- 172.21.15.27:0/389554001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47100b1db0 0x7f47100b2210 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f4710072330 tx=0x7f46f801c470 comp rx=0 tx=0).stop 2024-01-15T18:08:01.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.031+0000 7f4715800640 1 -- 172.21.15.27:0/389554001 shutdown_connections 2024-01-15T18:08:01.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.031+0000 7f4715800640 1 --2- 172.21.15.27:0/389554001 >> v2:172.21.15.93:3300/0 conn(0x7f47100ce050 0x7f47100ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.031+0000 7f4715800640 1 --2- 172.21.15.27:0/389554001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47100b1db0 0x7f47100b2210 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.031+0000 7f4715800640 1 --2- 172.21.15.27:0/389554001 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f471006c430 0x7f4710070ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.031+0000 7f4715800640 1 -- 172.21.15.27:0/389554001 >> 172.21.15.27:0/389554001 conn(0x7f47100cb700 msgr2=0x7f47100cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:01.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.031+0000 7f4715800640 1 -- 172.21.15.27:0/389554001 shutdown_connections 2024-01-15T18:08:01.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.031+0000 7f4715800640 1 -- 172.21.15.27:0/389554001 wait complete. 2024-01-15T18:08:01.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.031+0000 7f4715800640 1 Processor -- start 2024-01-15T18:08:01.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.031+0000 7f4715800640 1 -- start start 2024-01-15T18:08:01.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.035+0000 7f4715800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471006c430 0x7f471015c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:01.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.035+0000 7f4715800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f47100b1db0 0x7f471013c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:01.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.035+0000 7f4715800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f47100ce050 0x7f471013c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:01.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.035+0000 7f4715800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471014e4a0 con 0x7f471006c430 2024-01-15T18:08:01.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.035+0000 7f4715800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4710150260 con 0x7f47100b1db0 2024-01-15T18:08:01.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.035+0000 7f4715800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f471014ff80 con 0x7f47100ce050 2024-01-15T18:08:01.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.035+0000 7f470ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471006c430 0x7f471015c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:01.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.035+0000 7f470ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471006c430 0x7f471015c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60088/0 (socket says 172.21.15.27:60088) 2024-01-15T18:08:01.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.035+0000 7f470ea00640 1 -- 172.21.15.27:0/2418733923 learned_addr learned my addr 172.21.15.27:0/2418733923 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:01.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.035+0000 7f470e000640 1 --2- 172.21.15.27:0/2418733923 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f47100b1db0 0x7f471013c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:01.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.035+0000 7f470f400640 1 --2- 172.21.15.27:0/2418733923 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f47100ce050 0x7f471013c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:01.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.035+0000 7f470ea00640 1 -- 172.21.15.27:0/2418733923 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f47100ce050 msgr2=0x7f471013c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:01.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.035+0000 7f470ea00640 1 --2- 172.21.15.27:0/2418733923 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f47100ce050 0x7f471013c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.035+0000 7f470ea00640 1 -- 172.21.15.27:0/2418733923 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f47100b1db0 msgr2=0x7f471013c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:01.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.035+0000 7f470ea00640 1 --2- 172.21.15.27:0/2418733923 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f47100b1db0 0x7f471013c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.035+0000 7f470ea00640 1 -- 172.21.15.27:0/2418733923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f80094d0 con 0x7f471006c430 2024-01-15T18:08:01.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.035+0000 7f470ea00640 1 --2- 172.21.15.27:0/2418733923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471006c430 0x7f471015c6c0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f470000d800 tx=0x7f470000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:01.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.035+0000 7f4706a00640 1 -- 172.21.15.27:0/2418733923 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470000b6c0 con 0x7f471006c430 2024-01-15T18:08:01.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.035+0000 7f4706a00640 1 -- 172.21.15.27:0/2418733923 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470000b820 con 0x7f471006c430 2024-01-15T18:08:01.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.035+0000 7f4715800640 1 -- 172.21.15.27:0/2418733923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f471013a100 con 0x7f471006c430 2024-01-15T18:08:01.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.035+0000 7f4706a00640 1 -- 172.21.15.27:0/2418733923 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470000bb90 con 0x7f471006c430 2024-01-15T18:08:01.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.035+0000 7f4715800640 1 -- 172.21.15.27:0/2418733923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f471013a6c0 con 0x7f471006c430 2024-01-15T18:08:01.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.035+0000 7f4715800640 1 -- 172.21.15.27:0/2418733923 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f471004e850 con 0x7f471006c430 2024-01-15T18:08:01.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.039+0000 7f4706a00640 1 -- 172.21.15.27:0/2418733923 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) v1 ==== 86415+0+0 (secure 0 0 0) 0x7f470000bcf0 con 0x7f471006c430 2024-01-15T18:08:01.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.039+0000 7f4706a00640 1 --2- 172.21.15.27:0/2418733923 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f46e8066bc0 0x7f46e8069080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:01.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.039+0000 7f4706a00640 1 -- 172.21.15.27:0/2418733923 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4700084d70 con 0x7f471006c430 2024-01-15T18:08:01.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.039+0000 7f470e000640 1 --2- 172.21.15.27:0/2418733923 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f46e8066bc0 0x7f46e8069080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:01.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.039+0000 7f470e000640 1 --2- 172.21.15.27:0/2418733923 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f46e8066bc0 0x7f46e8069080 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f46f80ad000 tx=0x7f46f80a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:01.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.043+0000 7f4706a00640 1 -- 172.21.15.27:0/2418733923 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4700051bf0 con 0x7f471006c430 2024-01-15T18:08:01.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.231+0000 7f4715800640 1 -- 172.21.15.27:0/2418733923 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone63891", "target": ["mon-mgr", ""]}) v1 -- 0x7f471006c810 con 0x7f46e8066bc0 2024-01-15T18:08:01.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.287+0000 7f4706a00640 1 -- 172.21.15.27:0/2418733923 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f46f80ab640 con 0x7f46e8066bc0 2024-01-15T18:08:01.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.291+0000 7f4715800640 1 -- 172.21.15.27:0/2418733923 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f46e8066bc0 msgr2=0x7f46e8069080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:01.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.291+0000 7f4715800640 1 --2- 172.21.15.27:0/2418733923 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f46e8066bc0 0x7f46e8069080 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f46f80ad000 tx=0x7f46f80a6000 comp rx=0 tx=0).stop 2024-01-15T18:08:01.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.291+0000 7f4715800640 1 -- 172.21.15.27:0/2418733923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471006c430 msgr2=0x7f471015c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:01.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.291+0000 7f4715800640 1 --2- 172.21.15.27:0/2418733923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471006c430 0x7f471015c6c0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f470000d800 tx=0x7f470000dcc0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.291+0000 7f4715800640 1 -- 172.21.15.27:0/2418733923 shutdown_connections 2024-01-15T18:08:01.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.291+0000 7f4715800640 1 --2- 172.21.15.27:0/2418733923 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f46e8066bc0 0x7f46e8069080 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.291+0000 7f4715800640 1 --2- 172.21.15.27:0/2418733923 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f47100ce050 0x7f471013c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.291+0000 7f4715800640 1 --2- 172.21.15.27:0/2418733923 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f47100b1db0 0x7f471013c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.291+0000 7f4715800640 1 --2- 172.21.15.27:0/2418733923 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f471006c430 0x7f471015c6c0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.291+0000 7f4715800640 1 -- 172.21.15.27:0/2418733923 >> 172.21.15.27:0/2418733923 conn(0x7f47100cb700 msgr2=0x7f4710154430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:01.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.291+0000 7f4715800640 1 -- 172.21.15.27:0/2418733923 shutdown_connections 2024-01-15T18:08:01.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.291+0000 7f4715800640 1 -- 172.21.15.27:0/2418733923 wait complete. 2024-01-15T18:08:01.298 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:01.294+0000 7f59f9c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:01.299 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:01.294+0000 7f59f9c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:01.299 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:01.294+0000 7f59f9c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:01.299 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:01.294+0000 7f59f9c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:01.299 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:01.294+0000 7f59f9c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:01.318 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone63892 2024-01-15T18:08:01.319 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:01.310+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:01.319 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:01.310+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:01.319 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:01.310+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:01.319 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:01.310+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:01.319 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:01.310+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:01.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.459+0000 7efc65a00640 1 Processor -- start 2024-01-15T18:08:01.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.459+0000 7efc65a00640 1 -- start start 2024-01-15T18:08:01.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.463+0000 7efc65a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efc60158900 0x7efc6015f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:01.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.463+0000 7efc65a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7efc6015bf00 0x7efc600cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:01.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.463+0000 7efc65a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc600b1cb0 con 0x7efc600cf110 2024-01-15T18:08:01.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.463+0000 7efc65a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc600cf580 con 0x7efc60158900 2024-01-15T18:08:01.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.463+0000 7efc65a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc600b1f00 con 0x7efc6015bf00 2024-01-15T18:08:01.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.463+0000 7efc5ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efc60158900 0x7efc6015f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:01.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.463+0000 7efc5ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efc60158900 0x7efc6015f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60094/0 (socket says 172.21.15.27:60094) 2024-01-15T18:08:01.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.463+0000 7efc5ea00640 1 -- 172.21.15.27:0/2758598981 learned_addr learned my addr 172.21.15.27:0/2758598981 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:01.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.463+0000 7efc5f400640 1 --1- 172.21.15.27:0/2758598981 >> v1:172.21.15.27:6789/0 conn(0x7efc600cf110 0x7efc600701e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35048/0 (socket says 172.21.15.27:35048) 2024-01-15T18:08:01.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.463+0000 7efc5e000640 1 --2- 172.21.15.27:0/2758598981 >> v2:172.21.15.93:3300/0 conn(0x7efc6015bf00 0x7efc600cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:01.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.463+0000 7efc5ea00640 1 -- 172.21.15.27:0/2758598981 >> v2:172.21.15.93:3300/0 conn(0x7efc6015bf00 msgr2=0x7efc600cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:01.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.463+0000 7efc5ea00640 1 --2- 172.21.15.27:0/2758598981 >> v2:172.21.15.93:3300/0 conn(0x7efc6015bf00 0x7efc600cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.463+0000 7efc5ea00640 1 -- 172.21.15.27:0/2758598981 >> v1:172.21.15.27:6789/0 conn(0x7efc600cf110 legacy=0x7efc600701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:01.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.463+0000 7efc5ea00640 1 -- 172.21.15.27:0/2758598981 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc600b21e0 con 0x7efc60158900 2024-01-15T18:08:01.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.463+0000 7efc5e000640 1 --2- 172.21.15.27:0/2758598981 >> v2:172.21.15.93:3300/0 conn(0x7efc6015bf00 0x7efc600cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:08:01.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.463+0000 7efc5d600640 1 -- 172.21.15.27:0/2758598981 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2568228557 0 0) 0x7efc600b1cb0 con 0x7efc600cf110 2024-01-15T18:08:01.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.467+0000 7efc5ea00640 1 --2- 172.21.15.27:0/2758598981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc60158900 0x7efc6015f550 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7efc50009870 tx=0x7efc5001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab72e42e59fc7dff server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:01.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.467+0000 7efc5d600640 1 -- 172.21.15.27:0/2758598981 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc500ab5f0 con 0x7efc60158900 2024-01-15T18:08:01.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.467+0000 7efc5d600640 1 -- 172.21.15.27:0/2758598981 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc500ab750 con 0x7efc60158900 2024-01-15T18:08:01.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.467+0000 7efc5d600640 1 -- 172.21.15.27:0/2758598981 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc500abae0 con 0x7efc60158900 2024-01-15T18:08:01.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.467+0000 7efc65a00640 1 -- 172.21.15.27:0/2758598981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc60158900 msgr2=0x7efc6015f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:01.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.467+0000 7efc65a00640 1 --2- 172.21.15.27:0/2758598981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc60158900 0x7efc6015f550 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7efc50009870 tx=0x7efc5001c500 comp rx=0 tx=0).stop 2024-01-15T18:08:01.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.467+0000 7efc65a00640 1 -- 172.21.15.27:0/2758598981 shutdown_connections 2024-01-15T18:08:01.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.467+0000 7efc65a00640 1 --2- 172.21.15.27:0/2758598981 >> v2:172.21.15.93:3300/0 conn(0x7efc6015bf00 0x7efc600cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.467+0000 7efc65a00640 1 --2- 172.21.15.27:0/2758598981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc60158900 0x7efc6015f550 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.467+0000 7efc65a00640 1 -- 172.21.15.27:0/2758598981 >> 172.21.15.27:0/2758598981 conn(0x7efc600cb700 msgr2=0x7efc600cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:01.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.467+0000 7efc65a00640 1 -- 172.21.15.27:0/2758598981 shutdown_connections 2024-01-15T18:08:01.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.467+0000 7efc65a00640 1 -- 172.21.15.27:0/2758598981 wait complete. 2024-01-15T18:08:01.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.471+0000 7efc65a00640 1 Processor -- start 2024-01-15T18:08:01.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.471+0000 7efc65a00640 1 -- start start 2024-01-15T18:08:01.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.471+0000 7efc65a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efc600cf110 0x7efc600adf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:01.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.471+0000 7efc65a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc60158900 0x7efc6014f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:01.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.471+0000 7efc65a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efc6015bf00 0x7efc6014e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:01.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.471+0000 7efc65a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc60150780 con 0x7efc60158900 2024-01-15T18:08:01.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.471+0000 7efc65a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc60156b20 con 0x7efc600cf110 2024-01-15T18:08:01.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.471+0000 7efc65a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc60150270 con 0x7efc6015bf00 2024-01-15T18:08:01.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.471+0000 7efc5e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc60158900 0x7efc6014f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:01.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.471+0000 7efc5e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc60158900 0x7efc6014f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60098/0 (socket says 172.21.15.27:60098) 2024-01-15T18:08:01.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.471+0000 7efc5e000640 1 -- 172.21.15.27:0/1242489764 learned_addr learned my addr 172.21.15.27:0/1242489764 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:01.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.471+0000 7efc5ea00640 1 --2- 172.21.15.27:0/1242489764 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efc600cf110 0x7efc600adf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:01.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.471+0000 7efc5f400640 1 --2- 172.21.15.27:0/1242489764 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efc6015bf00 0x7efc6014e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:01.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.471+0000 7efc5e000640 1 -- 172.21.15.27:0/1242489764 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efc6015bf00 msgr2=0x7efc6014e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:01.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.471+0000 7efc5e000640 1 --2- 172.21.15.27:0/1242489764 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efc6015bf00 0x7efc6014e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.471+0000 7efc5e000640 1 -- 172.21.15.27:0/1242489764 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efc600cf110 msgr2=0x7efc600adf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:01.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.471+0000 7efc5e000640 1 --2- 172.21.15.27:0/1242489764 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efc600cf110 0x7efc600adf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.471+0000 7efc5e000640 1 -- 172.21.15.27:0/1242489764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc500094d0 con 0x7efc60158900 2024-01-15T18:08:01.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.471+0000 7efc5f400640 1 --2- 172.21.15.27:0/1242489764 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efc6015bf00 0x7efc6014e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:08:01.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.471+0000 7efc5e000640 1 --2- 172.21.15.27:0/1242489764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc60158900 0x7efc6014f900 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7efc4800c820 tx=0x7efc4800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:01.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.471+0000 7efc56a00640 1 -- 172.21.15.27:0/1242489764 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc48007a30 con 0x7efc60158900 2024-01-15T18:08:01.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.471+0000 7efc56a00640 1 -- 172.21.15.27:0/1242489764 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc48007b90 con 0x7efc60158900 2024-01-15T18:08:01.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.471+0000 7efc56a00640 1 -- 172.21.15.27:0/1242489764 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4800ce10 con 0x7efc60158900 2024-01-15T18:08:01.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.471+0000 7efc65a00640 1 -- 172.21.15.27:0/1242489764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc6015c490 con 0x7efc60158900 2024-01-15T18:08:01.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.471+0000 7efc65a00640 1 -- 172.21.15.27:0/1242489764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc601564d0 con 0x7efc60158900 2024-01-15T18:08:01.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.475+0000 7efc65a00640 1 -- 172.21.15.27:0/1242489764 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc60156cd0 con 0x7efc60158900 2024-01-15T18:08:01.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.475+0000 7efc56a00640 1 -- 172.21.15.27:0/1242489764 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) v1 ==== 86415+0+0 (secure 0 0 0) 0x7efc480113f0 con 0x7efc60158900 2024-01-15T18:08:01.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.475+0000 7efc56a00640 1 --2- 172.21.15.27:0/1242489764 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efc38066bc0 0x7efc38069080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:01.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.475+0000 7efc56a00640 1 -- 172.21.15.27:0/1242489764 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7efc480849d0 con 0x7efc60158900 2024-01-15T18:08:01.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.475+0000 7efc5ea00640 1 --2- 172.21.15.27:0/1242489764 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efc38066bc0 0x7efc38069080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:01.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.475+0000 7efc5ea00640 1 --2- 172.21.15.27:0/1242489764 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efc38066bc0 0x7efc38069080 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7efc500af000 tx=0x7efc500a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:01.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.479+0000 7efc56a00640 1 -- 172.21.15.27:0/1242489764 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7efc60156cd0 con 0x7efc60158900 2024-01-15T18:08:01.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.671+0000 7efc65a00640 1 -- 172.21.15.27:0/1242489764 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone63892", "target": ["mon-mgr", ""]}) v1 -- 0x7efc600b2e40 con 0x7efc38066bc0 2024-01-15T18:08:01.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.739+0000 7efc56a00640 1 -- 172.21.15.27:0/1242489764 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7efc600b2e40 con 0x7efc38066bc0 2024-01-15T18:08:01.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.739+0000 7efc65a00640 1 -- 172.21.15.27:0/1242489764 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efc38066bc0 msgr2=0x7efc38069080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:01.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.739+0000 7efc65a00640 1 --2- 172.21.15.27:0/1242489764 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efc38066bc0 0x7efc38069080 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7efc500af000 tx=0x7efc500a7000 comp rx=0 tx=0).stop 2024-01-15T18:08:01.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.739+0000 7efc65a00640 1 -- 172.21.15.27:0/1242489764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc60158900 msgr2=0x7efc6014f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:01.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.739+0000 7efc65a00640 1 --2- 172.21.15.27:0/1242489764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc60158900 0x7efc6014f900 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7efc4800c820 tx=0x7efc4800cce0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.739+0000 7efc65a00640 1 -- 172.21.15.27:0/1242489764 shutdown_connections 2024-01-15T18:08:01.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.739+0000 7efc65a00640 1 --2- 172.21.15.27:0/1242489764 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efc38066bc0 0x7efc38069080 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.739+0000 7efc65a00640 1 --2- 172.21.15.27:0/1242489764 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efc6015bf00 0x7efc6014e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.739+0000 7efc65a00640 1 --2- 172.21.15.27:0/1242489764 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc60158900 0x7efc6014f900 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.739+0000 7efc65a00640 1 --2- 172.21.15.27:0/1242489764 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efc600cf110 0x7efc600adf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.739+0000 7efc65a00640 1 -- 172.21.15.27:0/1242489764 >> 172.21.15.27:0/1242489764 conn(0x7efc600cb700 msgr2=0x7efc6015b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:01.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.739+0000 7efc65a00640 1 -- 172.21.15.27:0/1242489764 shutdown_connections 2024-01-15T18:08:01.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.739+0000 7efc65a00640 1 -- 172.21.15.27:0/1242489764 wait complete. 2024-01-15T18:08:01.768 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone63893 2024-01-15T18:08:01.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.911+0000 7feb52a00640 1 Processor -- start 2024-01-15T18:08:01.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.911+0000 7feb52a00640 1 -- start start 2024-01-15T18:08:01.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.911+0000 7feb52a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7feb4c0cf2e0 0x7feb4c0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:01.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.911+0000 7feb52a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb4c0b1cb0 con 0x7feb4c15f550 2024-01-15T18:08:01.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.911+0000 7feb52a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb4c15fa00 con 0x7feb4c158900 2024-01-15T18:08:01.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.911+0000 7feb52a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb4c0b1f00 con 0x7feb4c0cf2e0 2024-01-15T18:08:01.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.911+0000 7feb50a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7feb4c0cf2e0 0x7feb4c0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:01.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.911+0000 7feb4b400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7feb4c15f550 0x7feb4c0cebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35056/0 (socket says 172.21.15.27:35056) 2024-01-15T18:08:01.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.911+0000 7feb4b400640 1 -- 172.21.15.27:0/1685138519 learned_addr learned my addr 172.21.15.27:0/1685138519 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:01.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.915+0000 7feb50a00640 1 -- 172.21.15.27:0/1685138519 >> v1:172.21.15.93:6789/0 conn(0x7feb4c158900 legacy=0x7feb4c15bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:01.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.915+0000 7feb50a00640 1 -- 172.21.15.27:0/1685138519 >> v1:172.21.15.27:6789/0 conn(0x7feb4c15f550 legacy=0x7feb4c0cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:01.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.915+0000 7feb4aa00640 1 -- 172.21.15.27:0/1685138519 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1472222308 0 0) 0x7feb4c0b1cb0 con 0x7feb4c15f550 2024-01-15T18:08:01.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.915+0000 7feb50a00640 1 -- 172.21.15.27:0/1685138519 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb4c0b21e0 con 0x7feb4c0cf2e0 2024-01-15T18:08:01.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.915+0000 7feb50a00640 1 --2- 172.21.15.27:0/1685138519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb4c0cf2e0 0x7feb4c0701e0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7feb44009d90 tx=0x7feb4401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3e7b6d7f38f1e59 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:01.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.915+0000 7feb4aa00640 1 -- 172.21.15.27:0/1685138519 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4c070f30 con 0x7feb4c0cf2e0 2024-01-15T18:08:01.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.915+0000 7feb4aa00640 1 -- 172.21.15.27:0/1685138519 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb440a59a0 con 0x7feb4c0cf2e0 2024-01-15T18:08:01.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.915+0000 7feb4aa00640 1 -- 172.21.15.27:0/1685138519 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb440a5d30 con 0x7feb4c0cf2e0 2024-01-15T18:08:01.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.915+0000 7feb52a00640 1 -- 172.21.15.27:0/1685138519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb4c0cf2e0 msgr2=0x7feb4c0701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:01.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.915+0000 7feb52a00640 1 --2- 172.21.15.27:0/1685138519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb4c0cf2e0 0x7feb4c0701e0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7feb44009d90 tx=0x7feb4401c8e0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.919+0000 7feb52a00640 1 -- 172.21.15.27:0/1685138519 shutdown_connections 2024-01-15T18:08:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.919+0000 7feb52a00640 1 --2- 172.21.15.27:0/1685138519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb4c0cf2e0 0x7feb4c0701e0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.919+0000 7feb52a00640 1 -- 172.21.15.27:0/1685138519 >> 172.21.15.27:0/1685138519 conn(0x7feb4c0cb700 msgr2=0x7feb4c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.919+0000 7feb52a00640 1 -- 172.21.15.27:0/1685138519 shutdown_connections 2024-01-15T18:08:01.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.919+0000 7feb52a00640 1 -- 172.21.15.27:0/1685138519 wait complete. 2024-01-15T18:08:01.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.919+0000 7feb52a00640 1 Processor -- start 2024-01-15T18:08:01.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.919+0000 7feb52a00640 1 -- start start 2024-01-15T18:08:01.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.919+0000 7feb52a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feb4c0cf2e0 0x7feb4c15dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:01.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.919+0000 7feb52a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb4c158900 0x7feb4c14d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:01.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.919+0000 7feb52a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feb4c15f550 0x7feb4c15ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:01.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.919+0000 7feb52a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c150830 con 0x7feb4c158900 2024-01-15T18:08:01.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.919+0000 7feb52a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c150270 con 0x7feb4c0cf2e0 2024-01-15T18:08:01.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.919+0000 7feb52a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c150550 con 0x7feb4c15f550 2024-01-15T18:08:01.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.919+0000 7feb4b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb4c158900 0x7feb4c14d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:01.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.919+0000 7feb4b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb4c158900 0x7feb4c14d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60128/0 (socket says 172.21.15.27:60128) 2024-01-15T18:08:01.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.919+0000 7feb4b400640 1 -- 172.21.15.27:0/2225228789 learned_addr learned my addr 172.21.15.27:0/2225228789 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:01.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.919+0000 7feb4be00640 1 --2- 172.21.15.27:0/2225228789 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feb4c0cf2e0 0x7feb4c15dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:01.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.919+0000 7feb50a00640 1 --2- 172.21.15.27:0/2225228789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feb4c15f550 0x7feb4c15ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:01.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.919+0000 7feb4b400640 1 -- 172.21.15.27:0/2225228789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feb4c15f550 msgr2=0x7feb4c15ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:01.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.919+0000 7feb4b400640 1 --2- 172.21.15.27:0/2225228789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feb4c15f550 0x7feb4c15ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.919+0000 7feb4b400640 1 -- 172.21.15.27:0/2225228789 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feb4c0cf2e0 msgr2=0x7feb4c15dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:01.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.919+0000 7feb4b400640 1 --2- 172.21.15.27:0/2225228789 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feb4c0cf2e0 0x7feb4c15dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:01.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.919+0000 7feb4b400640 1 -- 172.21.15.27:0/2225228789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb44009a70 con 0x7feb4c158900 2024-01-15T18:08:01.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.923+0000 7feb4b400640 1 --2- 172.21.15.27:0/2225228789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb4c158900 0x7feb4c14d7b0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7feb340011f0 tx=0x7feb3400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:01.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.923+0000 7feb43e00640 1 -- 172.21.15.27:0/2225228789 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb340074e0 con 0x7feb4c158900 2024-01-15T18:08:01.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.923+0000 7feb43e00640 1 -- 172.21.15.27:0/2225228789 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb34007640 con 0x7feb4c158900 2024-01-15T18:08:01.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.923+0000 7feb43e00640 1 -- 172.21.15.27:0/2225228789 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb34004a10 con 0x7feb4c158900 2024-01-15T18:08:01.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.923+0000 7feb52a00640 1 -- 172.21.15.27:0/2225228789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb4c150c90 con 0x7feb4c158900 2024-01-15T18:08:01.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.923+0000 7feb52a00640 1 -- 172.21.15.27:0/2225228789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb4c151250 con 0x7feb4c158900 2024-01-15T18:08:01.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.923+0000 7feb52a00640 1 -- 172.21.15.27:0/2225228789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb4c15c9d0 con 0x7feb4c158900 2024-01-15T18:08:01.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.923+0000 7feb43e00640 1 -- 172.21.15.27:0/2225228789 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) v1 ==== 86415+0+0 (secure 0 0 0) 0x7feb34004b70 con 0x7feb4c158900 2024-01-15T18:08:01.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.923+0000 7feb43e00640 1 --2- 172.21.15.27:0/2225228789 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7feb24066bc0 0x7feb24069080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:01.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.923+0000 7feb43e00640 1 -- 172.21.15.27:0/2225228789 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7feb34083a60 con 0x7feb4c158900 2024-01-15T18:08:01.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.927+0000 7feb4be00640 1 --2- 172.21.15.27:0/2225228789 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7feb24066bc0 0x7feb24069080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:01.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.927+0000 7feb4be00640 1 --2- 172.21.15.27:0/2225228789 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7feb24066bc0 0x7feb24069080 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7feb3c007fd0 tx=0x7feb3c007e40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:01.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:01.927+0000 7feb43e00640 1 -- 172.21.15.27:0/2225228789 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7feb34050a30 con 0x7feb4c158900 2024-01-15T18:08:02.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:02.115+0000 7feb52a00640 1 -- 172.21.15.27:0/2225228789 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone63893", "target": ["mon-mgr", ""]}) v1 -- 0x7feb4c06e170 con 0x7feb24066bc0 2024-01-15T18:08:03.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:03.047+0000 7feb43e00640 1 -- 172.21.15.27:0/2225228789 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 10) v1 ==== 86517+0+0 (secure 0 0 0) 0x7feb34050e00 con 0x7feb4c158900 2024-01-15T18:08:05.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.279+0000 7feb43e00640 1 -- 172.21.15.27:0/2225228789 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7feb3c004c00 con 0x7feb24066bc0 2024-01-15T18:08:05.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.283+0000 7feb52a00640 1 -- 172.21.15.27:0/2225228789 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7feb24066bc0 msgr2=0x7feb24069080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:05.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.283+0000 7feb52a00640 1 --2- 172.21.15.27:0/2225228789 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7feb24066bc0 0x7feb24069080 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7feb3c007fd0 tx=0x7feb3c007e40 comp rx=0 tx=0).stop 2024-01-15T18:08:05.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.283+0000 7feb52a00640 1 -- 172.21.15.27:0/2225228789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb4c158900 msgr2=0x7feb4c14d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:05.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.283+0000 7feb52a00640 1 --2- 172.21.15.27:0/2225228789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb4c158900 0x7feb4c14d7b0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7feb340011f0 tx=0x7feb3400c9b0 comp rx=0 tx=0).stop 2024-01-15T18:08:05.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.283+0000 7feb52a00640 1 -- 172.21.15.27:0/2225228789 shutdown_connections 2024-01-15T18:08:05.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.283+0000 7feb52a00640 1 --2- 172.21.15.27:0/2225228789 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7feb24066bc0 0x7feb24069080 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:05.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.283+0000 7feb52a00640 1 --2- 172.21.15.27:0/2225228789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feb4c15f550 0x7feb4c15ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:05.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.283+0000 7feb52a00640 1 --2- 172.21.15.27:0/2225228789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb4c158900 0x7feb4c14d7b0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:05.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.283+0000 7feb52a00640 1 --2- 172.21.15.27:0/2225228789 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feb4c0cf2e0 0x7feb4c15dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:05.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.283+0000 7feb52a00640 1 -- 172.21.15.27:0/2225228789 >> 172.21.15.27:0/2225228789 conn(0x7feb4c0cb700 msgr2=0x7feb4c0b4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:05.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.283+0000 7feb52a00640 1 -- 172.21.15.27:0/2225228789 shutdown_connections 2024-01-15T18:08:05.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.283+0000 7feb52a00640 1 -- 172.21.15.27:0/2225228789 wait complete. 2024-01-15T18:08:05.313 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone63894 2024-01-15T18:08:05.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.459+0000 7f017b400640 1 Processor -- start 2024-01-15T18:08:05.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.459+0000 7f017b400640 1 -- start start 2024-01-15T18:08:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.459+0000 7f017b400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f017415b100 0x7f0174070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.459+0000 7f017b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0174071420 0x7f017415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.459+0000 7f017b400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f017415a610 con 0x7f017415ad20 2024-01-15T18:08:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.459+0000 7f017b400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01740718b0 con 0x7f0174071420 2024-01-15T18:08:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.459+0000 7f017b400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f017415e3c0 con 0x7f017415b100 2024-01-15T18:08:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.459+0000 7f0179400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0174071420 0x7f017415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.459+0000 7f0179400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0174071420 0x7f017415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41324/0 (socket says 172.21.15.27:41324) 2024-01-15T18:08:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.459+0000 7f0179400640 1 -- 172.21.15.27:0/583324364 learned_addr learned my addr 172.21.15.27:0/583324364 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.459+0000 7f0173e00640 1 --2- 172.21.15.27:0/583324364 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f017415b100 0x7f0174070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.459+0000 7f0179400640 1 -- 172.21.15.27:0/583324364 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f017415b100 msgr2=0x7f0174070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:05.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.459+0000 7f0179400640 1 --2- 172.21.15.27:0/583324364 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f017415b100 0x7f0174070ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:05.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.459+0000 7f0179400640 1 -- 172.21.15.27:0/583324364 >> v1:172.21.15.93:6789/0 conn(0x7f017415ad20 legacy=0x7f017415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:05.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.459+0000 7f0179400640 1 -- 172.21.15.27:0/583324364 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017415e6a0 con 0x7f0174071420 2024-01-15T18:08:05.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.463+0000 7f0179400640 1 --2- 172.21.15.27:0/583324364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0174071420 0x7f017415bee0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f016c009e50 tx=0x7f016c01c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6642670f8a2be89 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:05.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.463+0000 7f0173400640 1 -- 172.21.15.27:0/583324364 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c0a57c0 con 0x7f0174071420 2024-01-15T18:08:05.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.463+0000 7f0173400640 1 -- 172.21.15.27:0/583324364 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f016c0a5920 con 0x7f0174071420 2024-01-15T18:08:05.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.463+0000 7f0173400640 1 -- 172.21.15.27:0/583324364 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c0a5cb0 con 0x7f0174071420 2024-01-15T18:08:05.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.463+0000 7f017b400640 1 -- 172.21.15.27:0/583324364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0174071420 msgr2=0x7f017415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:05.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.463+0000 7f017b400640 1 --2- 172.21.15.27:0/583324364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0174071420 0x7f017415bee0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f016c009e50 tx=0x7f016c01c9e0 comp rx=0 tx=0).stop 2024-01-15T18:08:05.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.463+0000 7f017b400640 1 -- 172.21.15.27:0/583324364 shutdown_connections 2024-01-15T18:08:05.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.463+0000 7f017b400640 1 --2- 172.21.15.27:0/583324364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0174071420 0x7f017415bee0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:05.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.463+0000 7f017b400640 1 --2- 172.21.15.27:0/583324364 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f017415b100 0x7f0174070ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:05.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.463+0000 7f017b400640 1 -- 172.21.15.27:0/583324364 >> 172.21.15.27:0/583324364 conn(0x7f01740cb700 msgr2=0x7f01740cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:05.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.463+0000 7f017b400640 1 -- 172.21.15.27:0/583324364 shutdown_connections 2024-01-15T18:08:05.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.463+0000 7f017b400640 1 -- 172.21.15.27:0/583324364 wait complete. 2024-01-15T18:08:05.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.467+0000 7f017b400640 1 Processor -- start 2024-01-15T18:08:05.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.467+0000 7f017b400640 1 -- start start 2024-01-15T18:08:05.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.467+0000 7f017b400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0174071420 0x7f017413e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:05.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.467+0000 7f017b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f017415ad20 0x7f01740c3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:05.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.467+0000 7f017b400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f017415b100 0x7f01740c4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:05.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.467+0000 7f017b400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017413ee40 con 0x7f017415ad20 2024-01-15T18:08:05.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.467+0000 7f017b400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01741435b0 con 0x7f017415b100 2024-01-15T18:08:05.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.467+0000 7f017b400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0174143890 con 0x7f0174071420 2024-01-15T18:08:05.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.467+0000 7f0173e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f017415ad20 0x7f01740c3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:05.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.467+0000 7f0173e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f017415ad20 0x7f01740c3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41330/0 (socket says 172.21.15.27:41330) 2024-01-15T18:08:05.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.467+0000 7f0173e00640 1 -- 172.21.15.27:0/3158520913 learned_addr learned my addr 172.21.15.27:0/3158520913 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:05.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.467+0000 7f0178a00640 1 --2- 172.21.15.27:0/3158520913 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0174071420 0x7f017413e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:05.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.467+0000 7f0179400640 1 --2- 172.21.15.27:0/3158520913 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f017415b100 0x7f01740c4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:05.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.467+0000 7f0173e00640 1 -- 172.21.15.27:0/3158520913 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0174071420 msgr2=0x7f017413e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:05.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.467+0000 7f0173e00640 1 --2- 172.21.15.27:0/3158520913 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0174071420 0x7f017413e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:05.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.467+0000 7f0173e00640 1 -- 172.21.15.27:0/3158520913 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f017415b100 msgr2=0x7f01740c4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:05.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.467+0000 7f0173e00640 1 --2- 172.21.15.27:0/3158520913 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f017415b100 0x7f01740c4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:05.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.467+0000 7f0173e00640 1 -- 172.21.15.27:0/3158520913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016c009b30 con 0x7f017415ad20 2024-01-15T18:08:05.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.467+0000 7f0173e00640 1 --2- 172.21.15.27:0/3158520913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f017415ad20 0x7f01740c3c20 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f015c00d5f0 tx=0x7f015c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:05.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.467+0000 7f0170c00640 1 -- 172.21.15.27:0/3158520913 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015c0125f0 con 0x7f017415ad20 2024-01-15T18:08:05.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.467+0000 7f0170c00640 1 -- 172.21.15.27:0/3158520913 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f015c012750 con 0x7f017415ad20 2024-01-15T18:08:05.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.467+0000 7f017b400640 1 -- 172.21.15.27:0/3158520913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0174143b70 con 0x7f017415ad20 2024-01-15T18:08:05.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.467+0000 7f0170c00640 1 -- 172.21.15.27:0/3158520913 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015c012b00 con 0x7f017415ad20 2024-01-15T18:08:05.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.467+0000 7f017b400640 1 -- 172.21.15.27:0/3158520913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0174144130 con 0x7f017415ad20 2024-01-15T18:08:05.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.471+0000 7f017b400640 1 -- 172.21.15.27:0/3158520913 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f017404e850 con 0x7f017415ad20 2024-01-15T18:08:05.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.471+0000 7f0170c00640 1 -- 172.21.15.27:0/3158520913 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 10) v1 ==== 86517+0+0 (secure 0 0 0) 0x7f015c012c60 con 0x7f017415ad20 2024-01-15T18:08:05.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.471+0000 7f0170c00640 1 --2- 172.21.15.27:0/3158520913 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f014c066ce0 0x7f014c0691a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:05.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.471+0000 7f0170c00640 1 -- 172.21.15.27:0/3158520913 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f015c085040 con 0x7f017415ad20 2024-01-15T18:08:05.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.471+0000 7f0178a00640 1 --2- 172.21.15.27:0/3158520913 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f014c066ce0 0x7f014c0691a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:05.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.471+0000 7f0178a00640 1 --2- 172.21.15.27:0/3158520913 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f014c066ce0 0x7f014c0691a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f0164007fd0 tx=0x7f0164009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:05.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.475+0000 7f0170c00640 1 -- 172.21.15.27:0/3158520913 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f015c051cb0 con 0x7f017415ad20 2024-01-15T18:08:05.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:05.663+0000 7f017b400640 1 -- 172.21.15.27:0/3158520913 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone63894", "target": ["mon-mgr", ""]}) v1 -- 0x7f0174067840 con 0x7f014c066ce0 2024-01-15T18:08:10.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.283+0000 7f0170c00640 1 -- 172.21.15.27:0/3158520913 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0174067840 con 0x7f014c066ce0 2024-01-15T18:08:10.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.287+0000 7f017b400640 1 -- 172.21.15.27:0/3158520913 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f014c066ce0 msgr2=0x7f014c0691a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:10.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.287+0000 7f017b400640 1 --2- 172.21.15.27:0/3158520913 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f014c066ce0 0x7f014c0691a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f0164007fd0 tx=0x7f0164009040 comp rx=0 tx=0).stop 2024-01-15T18:08:10.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.287+0000 7f017b400640 1 -- 172.21.15.27:0/3158520913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f017415ad20 msgr2=0x7f01740c3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:10.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.287+0000 7f017b400640 1 --2- 172.21.15.27:0/3158520913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f017415ad20 0x7f01740c3c20 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f015c00d5f0 tx=0x7f015c00dab0 comp rx=0 tx=0).stop 2024-01-15T18:08:10.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.287+0000 7f017b400640 1 -- 172.21.15.27:0/3158520913 shutdown_connections 2024-01-15T18:08:10.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.287+0000 7f017b400640 1 --2- 172.21.15.27:0/3158520913 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f014c066ce0 0x7f014c0691a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:10.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.287+0000 7f017b400640 1 --2- 172.21.15.27:0/3158520913 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f017415b100 0x7f01740c4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:10.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.287+0000 7f017b400640 1 --2- 172.21.15.27:0/3158520913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f017415ad20 0x7f01740c3c20 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:10.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.287+0000 7f017b400640 1 --2- 172.21.15.27:0/3158520913 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0174071420 0x7f017413e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:10.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.287+0000 7f017b400640 1 -- 172.21.15.27:0/3158520913 >> 172.21.15.27:0/3158520913 conn(0x7f01740cb700 msgr2=0x7f017406cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:10.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.287+0000 7f017b400640 1 -- 172.21.15.27:0/3158520913 shutdown_connections 2024-01-15T18:08:10.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.287+0000 7f017b400640 1 -- 172.21.15.27:0/3158520913 wait complete. 2024-01-15T18:08:10.316 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-15T18:08:10.316 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone63891 2024-01-15T18:08:10.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.455+0000 7fc365400640 1 Processor -- start 2024-01-15T18:08:10.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.455+0000 7fc365400640 1 -- start start 2024-01-15T18:08:10.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.455+0000 7fc365400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc36015f550 0x7fc3600cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:10.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.455+0000 7fc365400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc3600cf110 0x7fc3600701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:10.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.455+0000 7fc365400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3600b1cb0 con 0x7fc360158900 2024-01-15T18:08:10.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.455+0000 7fc365400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc36015fa00 con 0x7fc3600cf110 2024-01-15T18:08:10.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.455+0000 7fc365400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3600cf540 con 0x7fc36015f550 2024-01-15T18:08:10.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.459+0000 7fc35f200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc3600cf110 0x7fc3600701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:10.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.459+0000 7fc35f200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc3600cf110 0x7fc3600701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41342/0 (socket says 172.21.15.27:41342) 2024-01-15T18:08:10.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.459+0000 7fc35f200640 1 -- 172.21.15.27:0/317396489 learned_addr learned my addr 172.21.15.27:0/317396489 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:10.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.459+0000 7fc35de00640 1 --2- 172.21.15.27:0/317396489 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc36015f550 0x7fc3600cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:10.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.459+0000 7fc35f200640 1 -- 172.21.15.27:0/317396489 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc36015f550 msgr2=0x7fc3600cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:10.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.459+0000 7fc35f200640 1 --2- 172.21.15.27:0/317396489 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc36015f550 0x7fc3600cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:10.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.459+0000 7fc35f200640 1 -- 172.21.15.27:0/317396489 >> v1:172.21.15.93:6789/0 conn(0x7fc360158900 legacy=0x7fc36015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:10.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.459+0000 7fc35f200640 1 -- 172.21.15.27:0/317396489 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3600b1f00 con 0x7fc3600cf110 2024-01-15T18:08:10.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.459+0000 7fc35f200640 1 --2- 172.21.15.27:0/317396489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3600cf110 0x7fc3600701e0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc358009e50 tx=0x7fc35801c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b64a871c54b16bea server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:10.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.459+0000 7fc35d400640 1 -- 172.21.15.27:0/317396489 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc360070f30 con 0x7fc3600cf110 2024-01-15T18:08:10.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.459+0000 7fc35d400640 1 -- 172.21.15.27:0/317396489 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3580a58a0 con 0x7fc3600cf110 2024-01-15T18:08:10.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.459+0000 7fc35d400640 1 -- 172.21.15.27:0/317396489 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3580a5c30 con 0x7fc3600cf110 2024-01-15T18:08:10.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.463+0000 7fc365400640 1 -- 172.21.15.27:0/317396489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3600cf110 msgr2=0x7fc3600701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:10.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.463+0000 7fc365400640 1 --2- 172.21.15.27:0/317396489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3600cf110 0x7fc3600701e0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc358009e50 tx=0x7fc35801c9e0 comp rx=0 tx=0).stop 2024-01-15T18:08:10.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.463+0000 7fc365400640 1 -- 172.21.15.27:0/317396489 shutdown_connections 2024-01-15T18:08:10.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.463+0000 7fc365400640 1 --2- 172.21.15.27:0/317396489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3600cf110 0x7fc3600701e0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:10.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.463+0000 7fc365400640 1 --2- 172.21.15.27:0/317396489 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc36015f550 0x7fc3600cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:10.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.463+0000 7fc365400640 1 -- 172.21.15.27:0/317396489 >> 172.21.15.27:0/317396489 conn(0x7fc3600cb700 msgr2=0x7fc3600cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:10.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.463+0000 7fc365400640 1 -- 172.21.15.27:0/317396489 shutdown_connections 2024-01-15T18:08:10.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.463+0000 7fc365400640 1 -- 172.21.15.27:0/317396489 wait complete. 2024-01-15T18:08:10.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.463+0000 7fc365400640 1 Processor -- start 2024-01-15T18:08:10.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.463+0000 7fc365400640 1 -- start start 2024-01-15T18:08:10.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.463+0000 7fc365400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc3600cf110 0x7fc360159be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:10.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.463+0000 7fc365400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc360158900 0x7fc360140540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.463+0000 7fc365400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc36015f550 0x7fc3600b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.463+0000 7fc365400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36015b140 con 0x7fc36015f550 2024-01-15T18:08:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.463+0000 7fc365400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36015a4c0 con 0x7fc3600cf110 2024-01-15T18:08:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.463+0000 7fc365400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc36015a7a0 con 0x7fc360158900 2024-01-15T18:08:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.463+0000 7fc35f200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc36015f550 0x7fc3600b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.463+0000 7fc35f200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc36015f550 0x7fc3600b37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41352/0 (socket says 172.21.15.27:41352) 2024-01-15T18:08:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.463+0000 7fc35f200640 1 -- 172.21.15.27:0/2292656453 learned_addr learned my addr 172.21.15.27:0/2292656453 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.463+0000 7fc35de00640 1 --2- 172.21.15.27:0/2292656453 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc360158900 0x7fc360140540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.463+0000 7fc35e800640 1 --2- 172.21.15.27:0/2292656453 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc3600cf110 0x7fc360159be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.467+0000 7fc35f200640 1 -- 172.21.15.27:0/2292656453 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc360158900 msgr2=0x7fc360140540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.467+0000 7fc35f200640 1 --2- 172.21.15.27:0/2292656453 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc360158900 0x7fc360140540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:10.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.467+0000 7fc35f200640 1 -- 172.21.15.27:0/2292656453 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc3600cf110 msgr2=0x7fc360159be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:10.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.467+0000 7fc35f200640 1 --2- 172.21.15.27:0/2292656453 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc3600cf110 0x7fc360159be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:10.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.467+0000 7fc35f200640 1 -- 172.21.15.27:0/2292656453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc358009b30 con 0x7fc36015f550 2024-01-15T18:08:10.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.467+0000 7fc35e800640 1 --2- 172.21.15.27:0/2292656453 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc3600cf110 0x7fc360159be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:08:10.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.467+0000 7fc35f200640 1 --2- 172.21.15.27:0/2292656453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc36015f550 0x7fc3600b37f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc3580af040 tx=0x7fc358019780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:10.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.467+0000 7fc356a00640 1 -- 172.21.15.27:0/2292656453 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3580b8670 con 0x7fc36015f550 2024-01-15T18:08:10.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.467+0000 7fc356a00640 1 -- 172.21.15.27:0/2292656453 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3580b87d0 con 0x7fc36015f550 2024-01-15T18:08:10.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.467+0000 7fc365400640 1 -- 172.21.15.27:0/2292656453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc36015aa80 con 0x7fc36015f550 2024-01-15T18:08:10.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.467+0000 7fc356a00640 1 -- 172.21.15.27:0/2292656453 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3580b8b60 con 0x7fc36015f550 2024-01-15T18:08:10.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.467+0000 7fc365400640 1 -- 172.21.15.27:0/2292656453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc36015a120 con 0x7fc36015f550 2024-01-15T18:08:10.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.467+0000 7fc365400640 1 -- 172.21.15.27:0/2292656453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc360155460 con 0x7fc36015f550 2024-01-15T18:08:10.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.467+0000 7fc356a00640 1 -- 172.21.15.27:0/2292656453 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 10) v1 ==== 86517+0+0 (secure 0 0 0) 0x7fc3580b8cc0 con 0x7fc36015f550 2024-01-15T18:08:10.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.467+0000 7fc356a00640 1 --2- 172.21.15.27:0/2292656453 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc338066ce0 0x7fc3380691a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:10.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.467+0000 7fc356a00640 1 -- 172.21.15.27:0/2292656453 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc3581201b0 con 0x7fc36015f550 2024-01-15T18:08:10.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.467+0000 7fc35e800640 1 --2- 172.21.15.27:0/2292656453 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc338066ce0 0x7fc3380691a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:10.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.471+0000 7fc35e800640 1 --2- 172.21.15.27:0/2292656453 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc338066ce0 0x7fc3380691a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc350000f70 tx=0x7fc350009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:10.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.475+0000 7fc356a00640 1 -- 172.21.15.27:0/2292656453 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc3580ecf20 con 0x7fc36015f550 2024-01-15T18:08:10.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:10.663+0000 7fc365400640 1 -- 172.21.15.27:0/2292656453 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone63891", "target": ["mon-mgr", ""]}) v1 -- 0x7fc36006ced0 con 0x7fc338066ce0 2024-01-15T18:08:15.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:15.211+0000 7fc356a00640 1 -- 172.21.15.27:0/2292656453 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7fc36006ced0 con 0x7fc338066ce0 2024-01-15T18:08:15.215 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:08:15.215 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:08:15.215 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "pending", 2024-01-15T18:08:15.215 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:08:15.215 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:08:15.215 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:08:15.216 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:08:15.216 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:15.216 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:15.216 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:08:15.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:15.211+0000 7fc365400640 1 -- 172.21.15.27:0/2292656453 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc338066ce0 msgr2=0x7fc3380691a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:15.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:15.211+0000 7fc365400640 1 --2- 172.21.15.27:0/2292656453 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc338066ce0 0x7fc3380691a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc350000f70 tx=0x7fc350009040 comp rx=0 tx=0).stop 2024-01-15T18:08:15.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:15.211+0000 7fc365400640 1 -- 172.21.15.27:0/2292656453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc36015f550 msgr2=0x7fc3600b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:15.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:15.211+0000 7fc365400640 1 --2- 172.21.15.27:0/2292656453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc36015f550 0x7fc3600b37f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc3580af040 tx=0x7fc358019780 comp rx=0 tx=0).stop 2024-01-15T18:08:15.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:15.211+0000 7fc365400640 1 -- 172.21.15.27:0/2292656453 shutdown_connections 2024-01-15T18:08:15.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:15.211+0000 7fc365400640 1 --2- 172.21.15.27:0/2292656453 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc338066ce0 0x7fc3380691a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:15.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:15.211+0000 7fc365400640 1 --2- 172.21.15.27:0/2292656453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc36015f550 0x7fc3600b37f0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:15.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:15.211+0000 7fc365400640 1 --2- 172.21.15.27:0/2292656453 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc360158900 0x7fc360140540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:15.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:15.211+0000 7fc365400640 1 --2- 172.21.15.27:0/2292656453 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc3600cf110 0x7fc360159be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:15.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:15.211+0000 7fc365400640 1 -- 172.21.15.27:0/2292656453 >> 172.21.15.27:0/2292656453 conn(0x7fc3600cb700 msgr2=0x7fc3600cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:15.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:15.215+0000 7fc365400640 1 -- 172.21.15.27:0/2292656453 shutdown_connections 2024-01-15T18:08:15.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:15.215+0000 7fc365400640 1 -- 172.21.15.27:0/2292656453 wait complete. 2024-01-15T18:08:15.246 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-15T18:08:16.246 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone63891 2024-01-15T18:08:16.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.395+0000 7f97ddc00640 1 Processor -- start 2024-01-15T18:08:16.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.395+0000 7f97ddc00640 1 -- start start 2024-01-15T18:08:16.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.399+0000 7f97ddc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f97d806c430 0x7f97d8070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:16.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.399+0000 7f97ddc00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f97d80b1db0 0x7f97d80b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:16.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.399+0000 7f97ddc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f97d80ce050 0x7f97d80ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:16.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.399+0000 7f97ddc00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97d80cea20 con 0x7f97d806c430 2024-01-15T18:08:16.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.399+0000 7f97ddc00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97d80ccfb0 con 0x7f97d80b1db0 2024-01-15T18:08:16.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.399+0000 7f97ddc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97d80cd290 con 0x7f97d80ce050 2024-01-15T18:08:16.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.399+0000 7f97d6a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f97d806c430 0x7f97d8070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:16.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.399+0000 7f97d6a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f97d806c430 0x7f97d8070ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37180/0 (socket says 172.21.15.27:37180) 2024-01-15T18:08:16.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.399+0000 7f97d6a00640 1 -- 172.21.15.27:0/1966386667 learned_addr learned my addr 172.21.15.27:0/1966386667 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:16.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.399+0000 7f97d6a00640 1 -- 172.21.15.27:0/1966386667 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f97d80ce050 msgr2=0x7f97d80ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:16.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.399+0000 7f97d6a00640 1 --2- 172.21.15.27:0/1966386667 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f97d80ce050 0x7f97d80ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:16.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.399+0000 7f97d6a00640 1 -- 172.21.15.27:0/1966386667 >> v2:172.21.15.93:3300/0 conn(0x7f97d80b1db0 msgr2=0x7f97d80b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:16.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.399+0000 7f97d6000640 1 --2- 172.21.15.27:0/1966386667 >> v2:172.21.15.93:3300/0 conn(0x7f97d80b1db0 0x7f97d80b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:16.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.399+0000 7f97d6a00640 1 --2- 172.21.15.27:0/1966386667 >> v2:172.21.15.93:3300/0 conn(0x7f97d80b1db0 0x7f97d80b2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:16.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.399+0000 7f97d6a00640 1 -- 172.21.15.27:0/1966386667 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d80cd570 con 0x7f97d806c430 2024-01-15T18:08:16.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.399+0000 7f97d6000640 1 --2- 172.21.15.27:0/1966386667 >> v2:172.21.15.93:3300/0 conn(0x7f97d80b1db0 0x7f97d80b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:08:16.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.403+0000 7f97d6a00640 1 --2- 172.21.15.27:0/1966386667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d806c430 0x7f97d8070ba0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f97c8009870 tx=0x7f97c801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a23e43e3c2bfbddf server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:16.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.403+0000 7f97d5600640 1 -- 172.21.15.27:0/1966386667 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c80ab600 con 0x7f97d806c430 2024-01-15T18:08:16.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.403+0000 7f97d5600640 1 -- 172.21.15.27:0/1966386667 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c80ab760 con 0x7f97d806c430 2024-01-15T18:08:16.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.403+0000 7f97d5600640 1 -- 172.21.15.27:0/1966386667 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c80abaf0 con 0x7f97d806c430 2024-01-15T18:08:16.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.403+0000 7f97ddc00640 1 -- 172.21.15.27:0/1966386667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d806c430 msgr2=0x7f97d8070ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.403+0000 7f97ddc00640 1 --2- 172.21.15.27:0/1966386667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d806c430 0x7f97d8070ba0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f97c8009870 tx=0x7f97c801c440 comp rx=0 tx=0).stop 2024-01-15T18:08:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.403+0000 7f97ddc00640 1 -- 172.21.15.27:0/1966386667 shutdown_connections 2024-01-15T18:08:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.403+0000 7f97ddc00640 1 --2- 172.21.15.27:0/1966386667 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f97d80ce050 0x7f97d80ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.403+0000 7f97ddc00640 1 --2- 172.21.15.27:0/1966386667 >> v2:172.21.15.93:3300/0 conn(0x7f97d80b1db0 0x7f97d80b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.403+0000 7f97ddc00640 1 --2- 172.21.15.27:0/1966386667 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d806c430 0x7f97d8070ba0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.403+0000 7f97ddc00640 1 -- 172.21.15.27:0/1966386667 >> 172.21.15.27:0/1966386667 conn(0x7f97d80cb700 msgr2=0x7f97d80cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.403+0000 7f97ddc00640 1 -- 172.21.15.27:0/1966386667 shutdown_connections 2024-01-15T18:08:16.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.403+0000 7f97ddc00640 1 -- 172.21.15.27:0/1966386667 wait complete. 2024-01-15T18:08:16.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.403+0000 7f97ddc00640 1 Processor -- start 2024-01-15T18:08:16.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.403+0000 7f97ddc00640 1 -- start start 2024-01-15T18:08:16.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.407+0000 7f97ddc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d806c430 0x7f97d815afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:16.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.407+0000 7f97ddc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f97d80b1db0 0x7f97d8147c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:16.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.407+0000 7f97ddc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f97d80ce050 0x7f97d8145da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:16.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.407+0000 7f97ddc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d815c2c0 con 0x7f97d806c430 2024-01-15T18:08:16.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.407+0000 7f97ddc00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97d8148160 con 0x7f97d80ce050 2024-01-15T18:08:16.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.407+0000 7f97ddc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97d815b640 con 0x7f97d80b1db0 2024-01-15T18:08:16.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.407+0000 7f97d6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d806c430 0x7f97d815afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:16.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.407+0000 7f97d6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d806c430 0x7f97d815afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37184/0 (socket says 172.21.15.27:37184) 2024-01-15T18:08:16.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.407+0000 7f97d6a00640 1 -- 172.21.15.27:0/3269542341 learned_addr learned my addr 172.21.15.27:0/3269542341 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:16.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.407+0000 7f97d6000640 1 --2- 172.21.15.27:0/3269542341 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f97d80b1db0 0x7f97d8147c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:16.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.407+0000 7f97d7400640 1 --2- 172.21.15.27:0/3269542341 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f97d80ce050 0x7f97d8145da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:16.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.407+0000 7f97d6a00640 1 -- 172.21.15.27:0/3269542341 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f97d80b1db0 msgr2=0x7f97d8147c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:16.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.407+0000 7f97d6a00640 1 --2- 172.21.15.27:0/3269542341 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f97d80b1db0 0x7f97d8147c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:16.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.407+0000 7f97d6a00640 1 -- 172.21.15.27:0/3269542341 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f97d80ce050 msgr2=0x7f97d8145da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:16.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.407+0000 7f97d6a00640 1 --2- 172.21.15.27:0/3269542341 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f97d80ce050 0x7f97d8145da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:16.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.407+0000 7f97d6a00640 1 -- 172.21.15.27:0/3269542341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c80094d0 con 0x7f97d806c430 2024-01-15T18:08:16.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.407+0000 7f97d6a00640 1 --2- 172.21.15.27:0/3269542341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d806c430 0x7f97d815afd0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f97c801cce0 tx=0x7f97c801ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:16.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.407+0000 7f97cea00640 1 -- 172.21.15.27:0/3269542341 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c80b4800 con 0x7f97d806c430 2024-01-15T18:08:16.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.407+0000 7f97cea00640 1 -- 172.21.15.27:0/3269542341 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c80b4980 con 0x7f97d806c430 2024-01-15T18:08:16.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.407+0000 7f97ddc00640 1 -- 172.21.15.27:0/3269542341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97d815b920 con 0x7f97d806c430 2024-01-15T18:08:16.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.407+0000 7f97cea00640 1 -- 172.21.15.27:0/3269542341 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c80b4d30 con 0x7f97d806c430 2024-01-15T18:08:16.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.407+0000 7f97ddc00640 1 -- 172.21.15.27:0/3269542341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97d815bee0 con 0x7f97d806c430 2024-01-15T18:08:16.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.407+0000 7f97ddc00640 1 -- 172.21.15.27:0/3269542341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97d804e850 con 0x7f97d806c430 2024-01-15T18:08:16.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.407+0000 7f97cea00640 1 -- 172.21.15.27:0/3269542341 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 10) v1 ==== 86517+0+0 (secure 0 0 0) 0x7f97c80c7440 con 0x7f97d806c430 2024-01-15T18:08:16.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.411+0000 7f97cea00640 1 --2- 172.21.15.27:0/3269542341 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f97b0066ce0 0x7f97b00691a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:16.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.411+0000 7f97cea00640 1 -- 172.21.15.27:0/3269542341 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f97c811f750 con 0x7f97d806c430 2024-01-15T18:08:16.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.411+0000 7f97d6000640 1 --2- 172.21.15.27:0/3269542341 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f97b0066ce0 0x7f97b00691a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:16.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.411+0000 7f97d6000640 1 --2- 172.21.15.27:0/3269542341 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f97b0066ce0 0x7f97b00691a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f97d806df70 tx=0x7f97c0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:16.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.415+0000 7f97cea00640 1 -- 172.21.15.27:0/3269542341 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f97c80ec3a0 con 0x7f97d806c430 2024-01-15T18:08:16.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.603+0000 7f97ddc00640 1 -- 172.21.15.27:0/3269542341 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone63891", "target": ["mon-mgr", ""]}) v1 -- 0x7f97d806d020 con 0x7f97b0066ce0 2024-01-15T18:08:16.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.615+0000 7f97cea00640 1 -- 172.21.15.27:0/3269542341 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f97c000c040 con 0x7f97b0066ce0 2024-01-15T18:08:16.619 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:08:16.619 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:08:16.620 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:08:16.620 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:08:16.620 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:08:16.620 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:08:16.620 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:08:16.620 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:16.620 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:08:16.620 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 0.0, 2024-01-15T18:08:16.620 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "0 B/3 GB", 2024-01-15T18:08:16.620 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "0/3", 2024-01-15T18:08:16.620 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:08:16.620 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:08:16.620 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:16.620 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:16.621 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:08:16.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.615+0000 7f97ddc00640 1 -- 172.21.15.27:0/3269542341 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f97b0066ce0 msgr2=0x7f97b00691a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:16.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.615+0000 7f97ddc00640 1 --2- 172.21.15.27:0/3269542341 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f97b0066ce0 0x7f97b00691a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f97d806df70 tx=0x7f97c0008040 comp rx=0 tx=0).stop 2024-01-15T18:08:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.615+0000 7f97ddc00640 1 -- 172.21.15.27:0/3269542341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d806c430 msgr2=0x7f97d815afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.615+0000 7f97ddc00640 1 --2- 172.21.15.27:0/3269542341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d806c430 0x7f97d815afd0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f97c801cce0 tx=0x7f97c801ac80 comp rx=0 tx=0).stop 2024-01-15T18:08:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.615+0000 7f97ddc00640 1 -- 172.21.15.27:0/3269542341 shutdown_connections 2024-01-15T18:08:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.615+0000 7f97ddc00640 1 --2- 172.21.15.27:0/3269542341 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f97b0066ce0 0x7f97b00691a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.615+0000 7f97ddc00640 1 --2- 172.21.15.27:0/3269542341 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f97d80ce050 0x7f97d8145da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.615+0000 7f97ddc00640 1 --2- 172.21.15.27:0/3269542341 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f97d80b1db0 0x7f97d8147c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.615+0000 7f97ddc00640 1 --2- 172.21.15.27:0/3269542341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97d806c430 0x7f97d815afd0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.615+0000 7f97ddc00640 1 -- 172.21.15.27:0/3269542341 >> 172.21.15.27:0/3269542341 conn(0x7f97d80cb700 msgr2=0x7f97d80aea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.615+0000 7f97ddc00640 1 -- 172.21.15.27:0/3269542341 shutdown_connections 2024-01-15T18:08:16.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.615+0000 7f97ddc00640 1 -- 172.21.15.27:0/3269542341 wait complete. 2024-01-15T18:08:16.641 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:08:16.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.775+0000 7f5d24800640 1 Processor -- start 2024-01-15T18:08:16.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.775+0000 7f5d24800640 1 -- start start 2024-01-15T18:08:16.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.775+0000 7f5d24800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5d1c159b60 0x7f5d1c15ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:16.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.775+0000 7f5d24800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d1c06eef0 con 0x7f5d1c0b5360 2024-01-15T18:08:16.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.775+0000 7f5d24800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d1c0b5b60 con 0x7f5d1c06a4f0 2024-01-15T18:08:16.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.775+0000 7f5d24800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1c15e410 con 0x7f5d1c159b60 2024-01-15T18:08:16.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.775+0000 7f5d23200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5d1c0b5360 0x7f5d1c0ad640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44506/0 (socket says 172.21.15.27:44506) 2024-01-15T18:08:16.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.775+0000 7f5d23200640 1 -- 172.21.15.27:0/544358856 learned_addr learned my addr 172.21.15.27:0/544358856 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:16.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.775+0000 7f5d21400640 1 --2- 172.21.15.27:0/544358856 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5d1c159b60 0x7f5d1c15ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:16.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.775+0000 7f5d20a00640 1 -- 172.21.15.27:0/544358856 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 38412056 0 0) 0x7f5d1c06eef0 con 0x7f5d1c0b5360 2024-01-15T18:08:16.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.779+0000 7f5d20a00640 1 -- 172.21.15.27:0/544358856 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cfc003530 con 0x7f5d1c0b5360 2024-01-15T18:08:16.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.779+0000 7f5d20a00640 1 -- 172.21.15.27:0/544358856 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2786762368 0 0) 0x7f5d1c0b5b60 con 0x7f5d1c06a4f0 2024-01-15T18:08:16.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.779+0000 7f5d20a00640 1 -- 172.21.15.27:0/544358856 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d1c06eef0 con 0x7f5d1c06a4f0 2024-01-15T18:08:16.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.779+0000 7f5d21400640 1 -- 172.21.15.27:0/544358856 >> v1:172.21.15.93:6789/0 conn(0x7f5d1c06a4f0 legacy=0x7f5d1c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:16.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.779+0000 7f5d21400640 1 -- 172.21.15.27:0/544358856 >> v1:172.21.15.27:6789/0 conn(0x7f5d1c0b5360 legacy=0x7f5d1c0ad640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:16.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.779+0000 7f5d21400640 1 -- 172.21.15.27:0/544358856 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d1c0b3700 con 0x7f5d1c159b60 2024-01-15T18:08:16.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.779+0000 7f5d21400640 1 --2- 172.21.15.27:0/544358856 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5d1c159b60 0x7f5d1c15ded0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5d0400a990 tx=0x7f5d0401d860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38168a4fbf4ba7f5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:16.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.779+0000 7f5d20a00640 1 -- 172.21.15.27:0/544358856 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d040a6890 con 0x7f5d1c159b60 2024-01-15T18:08:16.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.779+0000 7f5d20a00640 1 -- 172.21.15.27:0/544358856 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d040a69f0 con 0x7f5d1c159b60 2024-01-15T18:08:16.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.779+0000 7f5d20a00640 1 -- 172.21.15.27:0/544358856 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d040a6d60 con 0x7f5d1c159b60 2024-01-15T18:08:16.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.783+0000 7f5d24800640 1 -- 172.21.15.27:0/544358856 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5d1c159b60 msgr2=0x7f5d1c15ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:16.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.783+0000 7f5d24800640 1 --2- 172.21.15.27:0/544358856 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5d1c159b60 0x7f5d1c15ded0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5d0400a990 tx=0x7f5d0401d860 comp rx=0 tx=0).stop 2024-01-15T18:08:16.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.783+0000 7f5d24800640 1 -- 172.21.15.27:0/544358856 shutdown_connections 2024-01-15T18:08:16.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.783+0000 7f5d24800640 1 --2- 172.21.15.27:0/544358856 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5d1c159b60 0x7f5d1c15ded0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:16.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.783+0000 7f5d24800640 1 -- 172.21.15.27:0/544358856 >> 172.21.15.27:0/544358856 conn(0x7f5d1c0b2360 msgr2=0x7f5d1c0b2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:16.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.783+0000 7f5d24800640 1 -- 172.21.15.27:0/544358856 shutdown_connections 2024-01-15T18:08:16.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.783+0000 7f5d24800640 1 -- 172.21.15.27:0/544358856 wait complete. 2024-01-15T18:08:16.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.783+0000 7f5d24800640 1 Processor -- start 2024-01-15T18:08:16.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.783+0000 7f5d24800640 1 -- start start 2024-01-15T18:08:16.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.783+0000 7f5d24800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5d1c06a4f0 0x7f5d1c159f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:16.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.783+0000 7f5d24800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d1c0b5360 0x7f5d1c141ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:16.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.783+0000 7f5d24800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5d1c159b60 0x7f5d1c13eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:16.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.783+0000 7f5d24800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1c15c610 con 0x7f5d1c0b5360 2024-01-15T18:08:16.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.783+0000 7f5d24800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1c1423e0 con 0x7f5d1c159b60 2024-01-15T18:08:16.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.783+0000 7f5d24800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1c15a520 con 0x7f5d1c06a4f0 2024-01-15T18:08:16.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.783+0000 7f5d21400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d1c0b5360 0x7f5d1c141ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:16.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.783+0000 7f5d21400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d1c0b5360 0x7f5d1c141ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37188/0 (socket says 172.21.15.27:37188) 2024-01-15T18:08:16.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.783+0000 7f5d21400640 1 -- 172.21.15.27:0/2575101729 learned_addr learned my addr 172.21.15.27:0/2575101729 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:16.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.787+0000 7f5d23200640 1 --2- 172.21.15.27:0/2575101729 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5d1c159b60 0x7f5d1c13eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:16.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.787+0000 7f5d21e00640 1 --2- 172.21.15.27:0/2575101729 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5d1c06a4f0 0x7f5d1c159f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:16.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.787+0000 7f5d21400640 1 -- 172.21.15.27:0/2575101729 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5d1c06a4f0 msgr2=0x7f5d1c159f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:16.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.787+0000 7f5d21400640 1 --2- 172.21.15.27:0/2575101729 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5d1c06a4f0 0x7f5d1c159f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:16.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.787+0000 7f5d21400640 1 -- 172.21.15.27:0/2575101729 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5d1c159b60 msgr2=0x7f5d1c13eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:16.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.787+0000 7f5d21400640 1 --2- 172.21.15.27:0/2575101729 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5d1c159b60 0x7f5d1c13eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:16.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.787+0000 7f5d21400640 1 -- 172.21.15.27:0/2575101729 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d0400a5c0 con 0x7f5d1c0b5360 2024-01-15T18:08:16.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.787+0000 7f5d21400640 1 --2- 172.21.15.27:0/2575101729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d1c0b5360 0x7f5d1c141ea0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f5d0401a470 tx=0x7f5d0401a600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:16.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.787+0000 7f5d1a000640 1 -- 172.21.15.27:0/2575101729 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d04004450 con 0x7f5d1c0b5360 2024-01-15T18:08:16.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.787+0000 7f5d1a000640 1 -- 172.21.15.27:0/2575101729 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d040a6810 con 0x7f5d1c0b5360 2024-01-15T18:08:16.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.787+0000 7f5d1a000640 1 -- 172.21.15.27:0/2575101729 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d04003820 con 0x7f5d1c0b5360 2024-01-15T18:08:16.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.787+0000 7f5d24800640 1 -- 172.21.15.27:0/2575101729 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d1c15bf50 con 0x7f5d1c0b5360 2024-01-15T18:08:16.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.787+0000 7f5d24800640 1 -- 172.21.15.27:0/2575101729 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d1c13f080 con 0x7f5d1c0b5360 2024-01-15T18:08:16.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.787+0000 7f5d24800640 1 -- 172.21.15.27:0/2575101729 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d1c073e10 con 0x7f5d1c0b5360 2024-01-15T18:08:16.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.787+0000 7f5d1a000640 1 -- 172.21.15.27:0/2575101729 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 10) v1 ==== 86517+0+0 (secure 0 0 0) 0x7f5d040b5040 con 0x7f5d1c0b5360 2024-01-15T18:08:16.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.787+0000 7f5d1a000640 1 --2- 172.21.15.27:0/2575101729 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5cf4066ce0 0x7f5cf40691a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:16.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.787+0000 7f5d1a000640 1 -- 172.21.15.27:0/2575101729 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5d0411f030 con 0x7f5d1c0b5360 2024-01-15T18:08:16.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.787+0000 7f5d21e00640 1 --2- 172.21.15.27:0/2575101729 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5cf4066ce0 0x7f5cf40691a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:16.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.791+0000 7f5d21e00640 1 --2- 172.21.15.27:0/2575101729 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5cf4066ce0 0x7f5cf40691a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c0011f0 tx=0x7f5d0c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:16.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:16.791+0000 7f5d1a000640 1 -- 172.21.15.27:0/2575101729 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5d1c073e10 con 0x7f5d1c0b5360 2024-01-15T18:08:17.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.003+0000 7f5d24800640 1 -- 172.21.15.27:0/2575101729 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5d1c073e10 con 0x7f5d1c0b5360 2024-01-15T18:08:17.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.003+0000 7f5d1a000640 1 -- 172.21.15.27:0/2575101729 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1852 (secure 0 0 0) 0x7f5d1c073e10 con 0x7f5d1c0b5360 2024-01-15T18:08:17.009 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:08:17.010 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:08:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:08:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:08:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:08:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:08:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:08:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:08:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:08:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:08:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:08:17.010 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:08:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:08:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:08:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:08:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:08:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:08:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:08:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 224, 2024-01-15T18:08:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:08:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:08:17.011 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:08:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:08:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:08:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 47, 2024-01-15T18:08:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:08:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:08:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:08:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:08:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:08:17.012 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:08:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:08:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:08:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:08:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:08:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:08:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:08:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:08:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:08:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 799, 2024-01-15T18:08:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 3222137317, 2024-01-15T18:08:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 6704136192, 2024-01-15T18:08:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 766389977088, 2024-01-15T18:08:17.013 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:08:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 341, 2024-01-15T18:08:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 13564, 2024-01-15T18:08:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 0, 2024-01-15T18:08:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 3 2024-01-15T18:08:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:08:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 28, 2024-01-15T18:08:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 3, 2024-01-15T18:08:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:08:17.014 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:08:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:08:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:08:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:08:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 3, 2024-01-15T18:08:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:08:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "a", 2024-01-15T18:08:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:08:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 5808 2024-01-15T18:08:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:08:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:08:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:08:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:08:17.015 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:08:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:08:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:08:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:08:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:08:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:08:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:08:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:08:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 8, 2024-01-15T18:08:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:07:04.167195+0000", 2024-01-15T18:08:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:08:17.016 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:17.017 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": {} 2024-01-15T18:08:17.017 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:08:17.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.007+0000 7f5d24800640 1 -- 172.21.15.27:0/2575101729 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5cf4066ce0 msgr2=0x7f5cf40691a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:17.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.007+0000 7f5d24800640 1 --2- 172.21.15.27:0/2575101729 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5cf4066ce0 0x7f5cf40691a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c0011f0 tx=0x7f5d0c009040 comp rx=0 tx=0).stop 2024-01-15T18:08:17.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.007+0000 7f5d24800640 1 -- 172.21.15.27:0/2575101729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d1c0b5360 msgr2=0x7f5d1c141ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:17.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.007+0000 7f5d24800640 1 --2- 172.21.15.27:0/2575101729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d1c0b5360 0x7f5d1c141ea0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f5d0401a470 tx=0x7f5d0401a600 comp rx=0 tx=0).stop 2024-01-15T18:08:17.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.007+0000 7f5d24800640 1 -- 172.21.15.27:0/2575101729 shutdown_connections 2024-01-15T18:08:17.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.007+0000 7f5d24800640 1 --2- 172.21.15.27:0/2575101729 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5cf4066ce0 0x7f5cf40691a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:17.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.007+0000 7f5d24800640 1 --2- 172.21.15.27:0/2575101729 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5d1c159b60 0x7f5d1c13eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:17.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.007+0000 7f5d24800640 1 --2- 172.21.15.27:0/2575101729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d1c0b5360 0x7f5d1c141ea0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:17.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.007+0000 7f5d24800640 1 --2- 172.21.15.27:0/2575101729 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5d1c06a4f0 0x7f5d1c159f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:17.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.007+0000 7f5d24800640 1 -- 172.21.15.27:0/2575101729 >> 172.21.15.27:0/2575101729 conn(0x7f5d1c0b2360 msgr2=0x7f5d1c0b0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:17.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.007+0000 7f5d24800640 1 -- 172.21.15.27:0/2575101729 shutdown_connections 2024-01-15T18:08:17.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.007+0000 7f5d24800640 1 -- 172.21.15.27:0/2575101729 wait complete. 2024-01-15T18:08:17.031 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:08:17.031 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone63891 2024-01-15T18:08:17.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.171+0000 7f2baf400640 1 Processor -- start 2024-01-15T18:08:17.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.171+0000 7f2baf400640 1 -- start start 2024-01-15T18:08:17.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.171+0000 7f2baf400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2ba815d190 0x7f2ba80cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:17.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.171+0000 7f2baf400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba80b0d70 con 0x7f2ba815d570 2024-01-15T18:08:17.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.171+0000 7f2baf400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba80d0540 con 0x7f2ba806c520 2024-01-15T18:08:17.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.171+0000 7f2baf400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba80b0440 con 0x7f2ba815d190 2024-01-15T18:08:17.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2ba7e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2ba815d570 0x7f2ba806be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44516/0 (socket says 172.21.15.27:44516) 2024-01-15T18:08:17.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2ba7e00640 1 -- 172.21.15.27:0/2693251269 learned_addr learned my addr 172.21.15.27:0/2693251269 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:17.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2baca00640 1 --2- 172.21.15.27:0/2693251269 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2ba815d190 0x7f2ba80cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:17.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2ba7400640 1 -- 172.21.15.27:0/2693251269 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216963070 0 0) 0x7f2ba80b0d70 con 0x7f2ba815d570 2024-01-15T18:08:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2ba7400640 1 -- 172.21.15.27:0/2693251269 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b88003530 con 0x7f2ba815d570 2024-01-15T18:08:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2ba7400640 1 -- 172.21.15.27:0/2693251269 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167368078 0 0) 0x7f2ba80d0540 con 0x7f2ba806c520 2024-01-15T18:08:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2ba7400640 1 -- 172.21.15.27:0/2693251269 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ba80b0d70 con 0x7f2ba806c520 2024-01-15T18:08:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2ba7400640 1 -- 172.21.15.27:0/2693251269 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3532889703 0 0) 0x7f2b88003530 con 0x7f2ba815d570 2024-01-15T18:08:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2ba7400640 1 -- 172.21.15.27:0/2693251269 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2ba815d190 msgr2=0x7f2ba80cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2ba7400640 1 --2- 172.21.15.27:0/2693251269 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2ba815d190 0x7f2ba80cfcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2ba7400640 1 -- 172.21.15.27:0/2693251269 >> v1:172.21.15.93:6789/0 conn(0x7f2ba806c520 legacy=0x7f2ba80cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2ba7400640 1 -- 172.21.15.27:0/2693251269 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba80b0720 con 0x7f2ba815d570 2024-01-15T18:08:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2ba7400640 1 -- 172.21.15.27:0/2693251269 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f2b90001680 con 0x7f2ba815d570 2024-01-15T18:08:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2ba7400640 1 -- 172.21.15.27:0/2693251269 >> v1:172.21.15.27:6789/0 conn(0x7f2ba815d570 legacy=0x7f2ba806be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2ba7400640 1 --2- 172.21.15.27:0/2693251269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b88003bb0 0x7f2b88003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:17.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2ba7400640 1 --2- 172.21.15.27:0/2693251269 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2ba815d190 0x7f2b880047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:17.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2ba7400640 1 --2- 172.21.15.27:0/2693251269 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2b88004d10 0x7f2b88005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:17.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2ba7400640 1 -- 172.21.15.27:0/2693251269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba80b0440 con 0x7f2b88003bb0 2024-01-15T18:08:17.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2ba7400640 1 -- 172.21.15.27:0/2693251269 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8800ba70 con 0x7f2b88004d10 2024-01-15T18:08:17.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2ba7400640 1 -- 172.21.15.27:0/2693251269 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8800bc30 con 0x7f2ba815d190 2024-01-15T18:08:17.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2baca00640 1 --2- 172.21.15.27:0/2693251269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b88003bb0 0x7f2b88003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:17.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2ba7e00640 1 --2- 172.21.15.27:0/2693251269 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2ba815d190 0x7f2b880047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:17.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2bad400640 1 --2- 172.21.15.27:0/2693251269 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2b88004d10 0x7f2b88005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:17.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2baca00640 1 -- 172.21.15.27:0/2693251269 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2ba815d190 msgr2=0x7f2b880047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:17.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2baca00640 1 --2- 172.21.15.27:0/2693251269 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2ba815d190 0x7f2b880047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:17.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2baca00640 1 -- 172.21.15.27:0/2693251269 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2b88004d10 msgr2=0x7f2b88005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:17.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2baca00640 1 --2- 172.21.15.27:0/2693251269 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2b88004d10 0x7f2b88005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:17.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.175+0000 7f2baca00640 1 -- 172.21.15.27:0/2693251269 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b88003530 con 0x7f2b88003bb0 2024-01-15T18:08:17.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.179+0000 7f2baca00640 1 --2- 172.21.15.27:0/2693251269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b88003bb0 0x7f2b88003fb0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f2b98009ab0 tx=0x7f2b9801ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c74507876cd1e08f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:17.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.179+0000 7f2ba7400640 1 -- 172.21.15.27:0/2693251269 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b980a55f0 con 0x7f2b88003bb0 2024-01-15T18:08:17.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.179+0000 7f2ba7400640 1 -- 172.21.15.27:0/2693251269 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b980a5750 con 0x7f2b88003bb0 2024-01-15T18:08:17.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.179+0000 7f2ba7400640 1 -- 172.21.15.27:0/2693251269 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b980a5b00 con 0x7f2b88003bb0 2024-01-15T18:08:17.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.179+0000 7f2baf400640 1 -- 172.21.15.27:0/2693251269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b88003bb0 msgr2=0x7f2b88003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:17.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.179+0000 7f2baf400640 1 --2- 172.21.15.27:0/2693251269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b88003bb0 0x7f2b88003fb0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f2b98009ab0 tx=0x7f2b9801ca00 comp rx=0 tx=0).stop 2024-01-15T18:08:17.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.179+0000 7f2bad400640 1 -- 172.21.15.27:0/2693251269 reap_dead start 2024-01-15T18:08:17.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.179+0000 7f2baf400640 1 -- 172.21.15.27:0/2693251269 shutdown_connections 2024-01-15T18:08:17.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.179+0000 7f2baf400640 1 -- 172.21.15.27:0/2693251269 >> 172.21.15.27:0/2693251269 conn(0x7f2ba80d0e70 msgr2=0x7f2ba80d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:17.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.179+0000 7f2baf400640 1 -- 172.21.15.27:0/2693251269 shutdown_connections 2024-01-15T18:08:17.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.179+0000 7f2baf400640 1 -- 172.21.15.27:0/2693251269 wait complete. 2024-01-15T18:08:17.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.179+0000 7f2baf400640 1 Processor -- start 2024-01-15T18:08:17.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.179+0000 7f2baf400640 1 -- start start 2024-01-15T18:08:17.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.183+0000 7f2baf400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2b88003bb0 0x7f2ba80c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:17.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.183+0000 7f2baf400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba81563a0 0x7f2ba806fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:17.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.183+0000 7f2baf400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2ba80b1340 0x7f2ba81557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:17.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.183+0000 7f2baf400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba8070360 con 0x7f2ba81563a0 2024-01-15T18:08:17.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.183+0000 7f2baf400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba80b69a0 con 0x7f2b88003bb0 2024-01-15T18:08:17.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.183+0000 7f2baf400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba80b5dd0 con 0x7f2ba80b1340 2024-01-15T18:08:17.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.183+0000 7f2ba7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba81563a0 0x7f2ba806fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:17.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.183+0000 7f2ba7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba81563a0 0x7f2ba806fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37204/0 (socket says 172.21.15.27:37204) 2024-01-15T18:08:17.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.183+0000 7f2ba7e00640 1 -- 172.21.15.27:0/1100753677 learned_addr learned my addr 172.21.15.27:0/1100753677 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:17.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.183+0000 7f2bad400640 1 --2- 172.21.15.27:0/1100753677 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2ba80b1340 0x7f2ba81557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:17.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.183+0000 7f2baca00640 1 --2- 172.21.15.27:0/1100753677 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2b88003bb0 0x7f2ba80c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:17.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.183+0000 7f2ba7e00640 1 -- 172.21.15.27:0/1100753677 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2ba80b1340 msgr2=0x7f2ba81557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:17.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.183+0000 7f2ba7e00640 1 --2- 172.21.15.27:0/1100753677 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2ba80b1340 0x7f2ba81557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:17.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.183+0000 7f2ba7e00640 1 -- 172.21.15.27:0/1100753677 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2b88003bb0 msgr2=0x7f2ba80c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:17.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.183+0000 7f2ba7e00640 1 --2- 172.21.15.27:0/1100753677 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2b88003bb0 0x7f2ba80c48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:17.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.183+0000 7f2ba7e00640 1 -- 172.21.15.27:0/1100753677 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba80b6070 con 0x7f2ba81563a0 2024-01-15T18:08:17.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.183+0000 7f2ba7e00640 1 --2- 172.21.15.27:0/1100753677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba81563a0 0x7f2ba806fb10 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f2b90000fc0 tx=0x7f2b9000ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:17.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.183+0000 7f2ba4c00640 1 -- 172.21.15.27:0/1100753677 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9000ee60 con 0x7f2ba81563a0 2024-01-15T18:08:17.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.183+0000 7f2ba4c00640 1 -- 172.21.15.27:0/1100753677 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b9000c870 con 0x7f2ba81563a0 2024-01-15T18:08:17.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.183+0000 7f2baf400640 1 -- 172.21.15.27:0/1100753677 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba80b6350 con 0x7f2ba81563a0 2024-01-15T18:08:17.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.183+0000 7f2ba4c00640 1 -- 172.21.15.27:0/1100753677 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9000cc20 con 0x7f2ba81563a0 2024-01-15T18:08:17.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.183+0000 7f2baf400640 1 -- 172.21.15.27:0/1100753677 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba8070050 con 0x7f2ba81563a0 2024-01-15T18:08:17.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.183+0000 7f2baf400640 1 -- 172.21.15.27:0/1100753677 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ba804e850 con 0x7f2ba81563a0 2024-01-15T18:08:17.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.183+0000 7f2ba4c00640 1 -- 172.21.15.27:0/1100753677 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 10) v1 ==== 86517+0+0 (secure 0 0 0) 0x7f2b9001d460 con 0x7f2ba81563a0 2024-01-15T18:08:17.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.187+0000 7f2ba4c00640 1 --2- 172.21.15.27:0/1100753677 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2b80066ce0 0x7f2b800691a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:17.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.187+0000 7f2ba4c00640 1 -- 172.21.15.27:0/1100753677 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2b90084980 con 0x7f2ba81563a0 2024-01-15T18:08:17.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.187+0000 7f2baca00640 1 --2- 172.21.15.27:0/1100753677 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2b80066ce0 0x7f2b800691a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:17.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.187+0000 7f2baca00640 1 --2- 172.21.15.27:0/1100753677 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2b80066ce0 0x7f2b800691a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2b980af000 tx=0x7f2b980046d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:17.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.191+0000 7f2ba4c00640 1 -- 172.21.15.27:0/1100753677 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2b90051680 con 0x7f2ba81563a0 2024-01-15T18:08:17.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.415+0000 7f2baf400640 1 -- 172.21.15.27:0/1100753677 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone63891", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ba80ce420 con 0x7f2b80066ce0 2024-01-15T18:08:17.443 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:08:17.443 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:08:17.443 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:08:17.443 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:08:17.443 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:08:17.443 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:08:17.443 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:08:17.443 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:17.444 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:08:17.444 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 0.0, 2024-01-15T18:08:17.444 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "0 B/3 GB", 2024-01-15T18:08:17.444 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "0/3", 2024-01-15T18:08:17.444 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:08:17.444 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:08:17.444 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:17.444 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:17.444 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:08:17.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.435+0000 7f2ba4c00640 1 -- 172.21.15.27:0/1100753677 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f2ba80ce420 con 0x7f2b80066ce0 2024-01-15T18:08:17.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.443+0000 7f2baf400640 1 -- 172.21.15.27:0/1100753677 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2b80066ce0 msgr2=0x7f2b800691a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:17.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.443+0000 7f2baf400640 1 --2- 172.21.15.27:0/1100753677 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2b80066ce0 0x7f2b800691a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2b980af000 tx=0x7f2b980046d0 comp rx=0 tx=0).stop 2024-01-15T18:08:17.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.443+0000 7f2baf400640 1 -- 172.21.15.27:0/1100753677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba81563a0 msgr2=0x7f2ba806fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:17.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.443+0000 7f2baf400640 1 --2- 172.21.15.27:0/1100753677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba81563a0 0x7f2ba806fb10 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f2b90000fc0 tx=0x7f2b9000ec10 comp rx=0 tx=0).stop 2024-01-15T18:08:17.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.443+0000 7f2baf400640 1 -- 172.21.15.27:0/1100753677 shutdown_connections 2024-01-15T18:08:17.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.443+0000 7f2baf400640 1 --2- 172.21.15.27:0/1100753677 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2b80066ce0 0x7f2b800691a0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:17.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.443+0000 7f2baf400640 1 --2- 172.21.15.27:0/1100753677 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2ba80b1340 0x7f2ba81557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:17.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.443+0000 7f2baf400640 1 --2- 172.21.15.27:0/1100753677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ba81563a0 0x7f2ba806fb10 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:17.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.443+0000 7f2baf400640 1 --2- 172.21.15.27:0/1100753677 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2b88003bb0 0x7f2ba80c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:17.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.443+0000 7f2baf400640 1 -- 172.21.15.27:0/1100753677 >> 172.21.15.27:0/1100753677 conn(0x7f2ba80d0e70 msgr2=0x7f2ba806f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:17.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.443+0000 7f2baf400640 1 -- 172.21.15.27:0/1100753677 shutdown_connections 2024-01-15T18:08:17.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:17.443+0000 7f2baf400640 1 -- 172.21.15.27:0/1100753677 wait complete. 2024-01-15T18:08:17.514 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:08:18.515 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone63891 2024-01-15T18:08:18.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.779+0000 7f2c9c800640 1 Processor -- start 2024-01-15T18:08:18.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.779+0000 7f2c9c800640 1 -- start start 2024-01-15T18:08:18.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.779+0000 7f2c9c800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f2c94072360 0x7f2c940b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:18.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.783+0000 7f2c9c800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2c940cf570 0x7f2c940b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:18.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.783+0000 7f2c9c800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c940b5980 con 0x7f2c9415d5c0 2024-01-15T18:08:18.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.783+0000 7f2c9c800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c940ce7b0 con 0x7f2c940cf570 2024-01-15T18:08:18.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.783+0000 7f2c9c800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c940cea90 con 0x7f2c94072360 2024-01-15T18:08:18.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.783+0000 7f2c9b200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2c940cf570 0x7f2c940b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:18.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.783+0000 7f2c9b200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2c940cf570 0x7f2c940b5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37206/0 (socket says 172.21.15.27:37206) 2024-01-15T18:08:18.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.783+0000 7f2c9b200640 1 -- 172.21.15.27:0/1697683256 learned_addr learned my addr 172.21.15.27:0/1697683256 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:18.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.783+0000 7f2c99400640 1 --2- 172.21.15.27:0/1697683256 >> v2:172.21.15.93:3300/0 conn(0x7f2c94072360 0x7f2c940b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:18.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.783+0000 7f2c9b200640 1 -- 172.21.15.27:0/1697683256 >> v2:172.21.15.93:3300/0 conn(0x7f2c94072360 msgr2=0x7f2c940b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:18.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.783+0000 7f2c9b200640 1 --2- 172.21.15.27:0/1697683256 >> v2:172.21.15.93:3300/0 conn(0x7f2c94072360 0x7f2c940b4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:18.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.783+0000 7f2c9b200640 1 -- 172.21.15.27:0/1697683256 >> v1:172.21.15.93:6789/0 conn(0x7f2c9415d5c0 legacy=0x7f2c94071c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:18.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.783+0000 7f2c9b200640 1 -- 172.21.15.27:0/1697683256 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c940ced70 con 0x7f2c940cf570 2024-01-15T18:08:18.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.787+0000 7f2c9b200640 1 --2- 172.21.15.27:0/1697683256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c940cf570 0x7f2c940b5050 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f2c940ce440 tx=0x7f2c8c01c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e9e507c7cf3ac9b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:18.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.787+0000 7f2c98a00640 1 -- 172.21.15.27:0/1697683256 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c0a56d0 con 0x7f2c940cf570 2024-01-15T18:08:18.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.787+0000 7f2c98a00640 1 -- 172.21.15.27:0/1697683256 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c8c0a5830 con 0x7f2c940cf570 2024-01-15T18:08:18.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.787+0000 7f2c98a00640 1 -- 172.21.15.27:0/1697683256 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c0a5bc0 con 0x7f2c940cf570 2024-01-15T18:08:18.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.787+0000 7f2c9c800640 1 -- 172.21.15.27:0/1697683256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c940cf570 msgr2=0x7f2c940b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:18.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.787+0000 7f2c9c800640 1 --2- 172.21.15.27:0/1697683256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c940cf570 0x7f2c940b5050 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f2c940ce440 tx=0x7f2c8c01c920 comp rx=0 tx=0).stop 2024-01-15T18:08:18.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.787+0000 7f2c9c800640 1 -- 172.21.15.27:0/1697683256 shutdown_connections 2024-01-15T18:08:18.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.787+0000 7f2c9c800640 1 --2- 172.21.15.27:0/1697683256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c940cf570 0x7f2c940b5050 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:18.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.787+0000 7f2c9c800640 1 --2- 172.21.15.27:0/1697683256 >> v2:172.21.15.93:3300/0 conn(0x7f2c94072360 0x7f2c940b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:18.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.787+0000 7f2c9c800640 1 -- 172.21.15.27:0/1697683256 >> 172.21.15.27:0/1697683256 conn(0x7f2c94099e10 msgr2=0x7f2c940b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:18.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.787+0000 7f2c9c800640 1 -- 172.21.15.27:0/1697683256 shutdown_connections 2024-01-15T18:08:18.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.787+0000 7f2c9c800640 1 -- 172.21.15.27:0/1697683256 wait complete. 2024-01-15T18:08:18.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.791+0000 7f2c9c800640 1 Processor -- start 2024-01-15T18:08:18.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.791+0000 7f2c9c800640 1 -- start start 2024-01-15T18:08:18.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.795+0000 7f2c9c800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c94072360 0x7f2c94156240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:18.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.795+0000 7f2c9c800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c940cf570 0x7f2c94156780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:18.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.795+0000 7f2c9c800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c9415d5c0 0x7f2c940ad5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:18.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.795+0000 7f2c9c800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c94155ab0 con 0x7f2c9415d5c0 2024-01-15T18:08:18.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.795+0000 7f2c9c800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c94150290 con 0x7f2c94072360 2024-01-15T18:08:18.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.795+0000 7f2c9c800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c94156db0 con 0x7f2c940cf570 2024-01-15T18:08:18.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.795+0000 7f2c9b200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c9415d5c0 0x7f2c940ad5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:18.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.795+0000 7f2c9b200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c9415d5c0 0x7f2c940ad5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37222/0 (socket says 172.21.15.27:37222) 2024-01-15T18:08:18.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.795+0000 7f2c9b200640 1 -- 172.21.15.27:0/1806624604 learned_addr learned my addr 172.21.15.27:0/1806624604 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:18.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.795+0000 7f2c99e00640 1 --2- 172.21.15.27:0/1806624604 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c94072360 0x7f2c94156240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:18.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.795+0000 7f2c99400640 1 --2- 172.21.15.27:0/1806624604 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c940cf570 0x7f2c94156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:18.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.795+0000 7f2c9b200640 1 -- 172.21.15.27:0/1806624604 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c940cf570 msgr2=0x7f2c94156780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:18.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.795+0000 7f2c9b200640 1 --2- 172.21.15.27:0/1806624604 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c940cf570 0x7f2c94156780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:18.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.795+0000 7f2c9b200640 1 -- 172.21.15.27:0/1806624604 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c94072360 msgr2=0x7f2c94156240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:18.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.795+0000 7f2c9b200640 1 --2- 172.21.15.27:0/1806624604 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c94072360 0x7f2c94156240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:18.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.795+0000 7f2c9b200640 1 -- 172.21.15.27:0/1806624604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c8c009b30 con 0x7f2c9415d5c0 2024-01-15T18:08:18.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.795+0000 7f2c99400640 1 --2- 172.21.15.27:0/1806624604 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c940cf570 0x7f2c94156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:08:18.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.795+0000 7f2c9b200640 1 --2- 172.21.15.27:0/1806624604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c9415d5c0 0x7f2c940ad5d0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c01a550 tx=0x7f2c8c0196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:18.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.795+0000 7f2c92000640 1 -- 172.21.15.27:0/1806624604 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c0b8670 con 0x7f2c9415d5c0 2024-01-15T18:08:18.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.795+0000 7f2c92000640 1 -- 172.21.15.27:0/1806624604 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c8c0b87d0 con 0x7f2c9415d5c0 2024-01-15T18:08:18.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.795+0000 7f2c92000640 1 -- 172.21.15.27:0/1806624604 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c0b8b80 con 0x7f2c9415d5c0 2024-01-15T18:08:18.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.795+0000 7f2c9c800640 1 -- 172.21.15.27:0/1806624604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c94155110 con 0x7f2c9415d5c0 2024-01-15T18:08:18.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.795+0000 7f2c9c800640 1 -- 172.21.15.27:0/1806624604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c941556d0 con 0x7f2c9415d5c0 2024-01-15T18:08:18.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.799+0000 7f2c9c800640 1 -- 172.21.15.27:0/1806624604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c94155bf0 con 0x7f2c9415d5c0 2024-01-15T18:08:18.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.799+0000 7f2c92000640 1 -- 172.21.15.27:0/1806624604 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 10) v1 ==== 86517+0+0 (secure 0 0 0) 0x7f2c8c0b8ce0 con 0x7f2c9415d5c0 2024-01-15T18:08:18.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.803+0000 7f2c92000640 1 --2- 172.21.15.27:0/1806624604 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2c6c066ce0 0x7f2c6c0691a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:18.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.803+0000 7f2c92000640 1 -- 172.21.15.27:0/1806624604 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2c8c120220 con 0x7f2c9415d5c0 2024-01-15T18:08:18.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.807+0000 7f2c92000640 1 -- 172.21.15.27:0/1806624604 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2c94155bf0 con 0x7f2c9415d5c0 2024-01-15T18:08:18.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.831+0000 7f2c99e00640 1 --2- 172.21.15.27:0/1806624604 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2c6c066ce0 0x7f2c6c0691a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:18.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:18.855+0000 7f2c99e00640 1 --2- 172.21.15.27:0/1806624604 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2c6c066ce0 0x7f2c6c0691a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c00a750 tx=0x7f2c7c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:19.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:19.099+0000 7f2c9c800640 1 -- 172.21.15.27:0/1806624604 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone63891", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c94066ff0 con 0x7f2c6c066ce0 2024-01-15T18:08:19.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:19.139+0000 7f2c92000640 1 -- 172.21.15.27:0/1806624604 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f2c94066ff0 con 0x7f2c6c066ce0 2024-01-15T18:08:19.146 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:08:19.146 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:08:19.146 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:08:19.146 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:08:19.146 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:08:19.146 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:08:19.146 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:08:19.146 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:19.146 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:08:19.146 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 0.0, 2024-01-15T18:08:19.146 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "0 B/3 GB", 2024-01-15T18:08:19.146 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "0/3", 2024-01-15T18:08:19.147 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:08:19.147 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:08:19.147 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:19.147 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:19.147 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:08:19.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:19.143+0000 7f2c9c800640 1 -- 172.21.15.27:0/1806624604 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2c6c066ce0 msgr2=0x7f2c6c0691a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:19.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:19.143+0000 7f2c9c800640 1 --2- 172.21.15.27:0/1806624604 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2c6c066ce0 0x7f2c6c0691a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c00a750 tx=0x7f2c7c008040 comp rx=0 tx=0).stop 2024-01-15T18:08:19.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:19.143+0000 7f2c9c800640 1 -- 172.21.15.27:0/1806624604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c9415d5c0 msgr2=0x7f2c940ad5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:19.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:19.143+0000 7f2c9c800640 1 --2- 172.21.15.27:0/1806624604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c9415d5c0 0x7f2c940ad5d0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c01a550 tx=0x7f2c8c0196c0 comp rx=0 tx=0).stop 2024-01-15T18:08:19.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:19.143+0000 7f2c9c800640 1 -- 172.21.15.27:0/1806624604 shutdown_connections 2024-01-15T18:08:19.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:19.143+0000 7f2c9c800640 1 --2- 172.21.15.27:0/1806624604 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2c6c066ce0 0x7f2c6c0691a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:19.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:19.143+0000 7f2c9c800640 1 --2- 172.21.15.27:0/1806624604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c9415d5c0 0x7f2c940ad5d0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:19.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:19.143+0000 7f2c9c800640 1 --2- 172.21.15.27:0/1806624604 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c940cf570 0x7f2c94156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:19.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:19.143+0000 7f2c9c800640 1 --2- 172.21.15.27:0/1806624604 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c94072360 0x7f2c94156240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:19.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:19.143+0000 7f2c9c800640 1 -- 172.21.15.27:0/1806624604 >> 172.21.15.27:0/1806624604 conn(0x7f2c94099e10 msgr2=0x7f2c940b22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:19.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:19.143+0000 7f2c9c800640 1 -- 172.21.15.27:0/1806624604 shutdown_connections 2024-01-15T18:08:19.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:19.143+0000 7f2c9c800640 1 -- 172.21.15.27:0/1806624604 wait complete. 2024-01-15T18:08:19.200 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:08:20.201 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone63891 2024-01-15T18:08:20.228 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:20.222+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:20.228 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:20.222+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:20.228 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:20.222+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:20.229 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:20.222+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:20.229 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:20.222+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:20.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.355+0000 7f2f61400640 1 Processor -- start 2024-01-15T18:08:20.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.355+0000 7f2f61400640 1 -- start start 2024-01-15T18:08:20.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.359+0000 7f2f61400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f2f5c161a90 0x7f2f5c161f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:20.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.359+0000 7f2f61400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c06c5d0 con 0x7f2f5c0cdb20 2024-01-15T18:08:20.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.359+0000 7f2f61400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c06d4a0 con 0x7f2f5c0ce810 2024-01-15T18:08:20.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.359+0000 7f2f61400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f5c162b00 con 0x7f2f5c161a90 2024-01-15T18:08:20.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.359+0000 7f2f5a800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2f5c0cdb20 0x7f2f5c06cd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44520/0 (socket says 172.21.15.27:44520) 2024-01-15T18:08:20.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.359+0000 7f2f5a800640 1 -- 172.21.15.27:0/1774220614 learned_addr learned my addr 172.21.15.27:0/1774220614 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:20.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.359+0000 7f2f59e00640 1 --2- 172.21.15.27:0/1774220614 >> v2:172.21.15.93:3300/0 conn(0x7f2f5c161a90 0x7f2f5c161f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:20.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.359+0000 7f2f59400640 1 -- 172.21.15.27:0/1774220614 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1814230482 0 0) 0x7f2f5c06c5d0 con 0x7f2f5c0cdb20 2024-01-15T18:08:20.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.359+0000 7f2f59400640 1 -- 172.21.15.27:0/1774220614 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f3c003530 con 0x7f2f5c0cdb20 2024-01-15T18:08:20.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.359+0000 7f2f59400640 1 -- 172.21.15.27:0/1774220614 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 11631211 0 0) 0x7f2f5c06d4a0 con 0x7f2f5c0ce810 2024-01-15T18:08:20.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.359+0000 7f2f59400640 1 -- 172.21.15.27:0/1774220614 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f5c06c5d0 con 0x7f2f5c0ce810 2024-01-15T18:08:20.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.359+0000 7f2f59e00640 1 -- 172.21.15.27:0/1774220614 >> v1:172.21.15.93:6789/0 conn(0x7f2f5c0ce810 legacy=0x7f2f5c0cc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:20.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.359+0000 7f2f59e00640 1 -- 172.21.15.27:0/1774220614 >> v1:172.21.15.27:6789/0 conn(0x7f2f5c0cdb20 legacy=0x7f2f5c06cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:20.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.359+0000 7f2f59e00640 1 -- 172.21.15.27:0/1774220614 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c162540 con 0x7f2f5c161a90 2024-01-15T18:08:20.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.363+0000 7f2f59e00640 1 --2- 172.21.15.27:0/1774220614 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2f5c161a90 0x7f2f5c161f20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2f440097c0 tx=0x7f2f4401c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0bae02056b87f6e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:20.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.363+0000 7f2f59400640 1 -- 172.21.15.27:0/1774220614 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f440ab650 con 0x7f2f5c161a90 2024-01-15T18:08:20.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.363+0000 7f2f59400640 1 -- 172.21.15.27:0/1774220614 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f440ab7b0 con 0x7f2f5c161a90 2024-01-15T18:08:20.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.363+0000 7f2f59400640 1 -- 172.21.15.27:0/1774220614 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f440abb40 con 0x7f2f5c161a90 2024-01-15T18:08:20.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.363+0000 7f2f61400640 1 -- 172.21.15.27:0/1774220614 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2f5c161a90 msgr2=0x7f2f5c161f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:20.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.363+0000 7f2f61400640 1 --2- 172.21.15.27:0/1774220614 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2f5c161a90 0x7f2f5c161f20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2f440097c0 tx=0x7f2f4401c450 comp rx=0 tx=0).stop 2024-01-15T18:08:20.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.363+0000 7f2f61400640 1 -- 172.21.15.27:0/1774220614 shutdown_connections 2024-01-15T18:08:20.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.363+0000 7f2f61400640 1 --2- 172.21.15.27:0/1774220614 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2f5c161a90 0x7f2f5c161f20 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:20.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.363+0000 7f2f61400640 1 -- 172.21.15.27:0/1774220614 >> 172.21.15.27:0/1774220614 conn(0x7f2f5c0cb700 msgr2=0x7f2f5c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:20.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.363+0000 7f2f61400640 1 -- 172.21.15.27:0/1774220614 shutdown_connections 2024-01-15T18:08:20.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.363+0000 7f2f61400640 1 -- 172.21.15.27:0/1774220614 wait complete. 2024-01-15T18:08:20.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.363+0000 7f2f61400640 1 Processor -- start 2024-01-15T18:08:20.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.363+0000 7f2f61400640 1 -- start start 2024-01-15T18:08:20.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.367+0000 7f2f61400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f5c0cdb20 0x7f2f5c146580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:20.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.367+0000 7f2f61400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2f5c0ce810 0x7f2f5c163720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:20.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.367+0000 7f2f61400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2f5c161a90 0x7f2f5c1fe210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:20.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.367+0000 7f2f61400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5c14a650 con 0x7f2f5c0cdb20 2024-01-15T18:08:20.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.367+0000 7f2f61400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5c1fac00 con 0x7f2f5c0ce810 2024-01-15T18:08:20.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.367+0000 7f2f61400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5c1faee0 con 0x7f2f5c161a90 2024-01-15T18:08:20.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.367+0000 7f2f5a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f5c0cdb20 0x7f2f5c146580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:20.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.367+0000 7f2f5a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f5c0cdb20 0x7f2f5c146580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37234/0 (socket says 172.21.15.27:37234) 2024-01-15T18:08:20.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.367+0000 7f2f5a800640 1 -- 172.21.15.27:0/232068194 learned_addr learned my addr 172.21.15.27:0/232068194 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:20.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.367+0000 7f2f59e00640 1 --2- 172.21.15.27:0/232068194 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2f5c0ce810 0x7f2f5c163720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:20.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.367+0000 7f2f5b200640 1 --2- 172.21.15.27:0/232068194 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2f5c161a90 0x7f2f5c1fe210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:20.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.367+0000 7f2f5a800640 1 -- 172.21.15.27:0/232068194 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2f5c161a90 msgr2=0x7f2f5c1fe210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:20.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.367+0000 7f2f5a800640 1 --2- 172.21.15.27:0/232068194 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2f5c161a90 0x7f2f5c1fe210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:20.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.367+0000 7f2f5a800640 1 -- 172.21.15.27:0/232068194 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2f5c0ce810 msgr2=0x7f2f5c163720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:20.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.367+0000 7f2f5a800640 1 --2- 172.21.15.27:0/232068194 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2f5c0ce810 0x7f2f5c163720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:20.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.367+0000 7f2f5a800640 1 -- 172.21.15.27:0/232068194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f44009420 con 0x7f2f5c0cdb20 2024-01-15T18:08:20.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.367+0000 7f2f5a800640 1 --2- 172.21.15.27:0/232068194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f5c0cdb20 0x7f2f5c146580 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c00c4d0 tx=0x7f2f4c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:20.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.367+0000 7f2f52a00640 1 -- 172.21.15.27:0/232068194 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c005cf0 con 0x7f2f5c0cdb20 2024-01-15T18:08:20.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.367+0000 7f2f52a00640 1 -- 172.21.15.27:0/232068194 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f4c005e50 con 0x7f2f5c0cdb20 2024-01-15T18:08:20.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.367+0000 7f2f61400640 1 -- 172.21.15.27:0/232068194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f5c1fb1c0 con 0x7f2f5c0cdb20 2024-01-15T18:08:20.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.367+0000 7f2f52a00640 1 -- 172.21.15.27:0/232068194 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c007270 con 0x7f2f5c0cdb20 2024-01-15T18:08:20.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.367+0000 7f2f61400640 1 -- 172.21.15.27:0/232068194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f5c146ac0 con 0x7f2f5c0cdb20 2024-01-15T18:08:20.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.367+0000 7f2f61400640 1 -- 172.21.15.27:0/232068194 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f5c1fdf30 con 0x7f2f5c0cdb20 2024-01-15T18:08:20.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.371+0000 7f2f52a00640 1 -- 172.21.15.27:0/232068194 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 10) v1 ==== 86517+0+0 (secure 0 0 0) 0x7f2f4c0073d0 con 0x7f2f5c0cdb20 2024-01-15T18:08:20.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.371+0000 7f2f52a00640 1 --2- 172.21.15.27:0/232068194 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2f34066ce0 0x7f2f340691a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:20.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.371+0000 7f2f52a00640 1 -- 172.21.15.27:0/232068194 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2f4c084000 con 0x7f2f5c0cdb20 2024-01-15T18:08:20.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.371+0000 7f2f59e00640 1 --2- 172.21.15.27:0/232068194 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2f34066ce0 0x7f2f340691a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:20.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.371+0000 7f2f59e00640 1 --2- 172.21.15.27:0/232068194 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2f34066ce0 0x7f2f340691a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2f440af000 tx=0x7f2f440b6040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:20.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.375+0000 7f2f52a00640 1 -- 172.21.15.27:0/232068194 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2f5c1fdf30 con 0x7f2f5c0cdb20 2024-01-15T18:08:20.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.563+0000 7f2f61400640 1 -- 172.21.15.27:0/232068194 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone63891", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f5c06e390 con 0x7f2f34066ce0 2024-01-15T18:08:20.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.567+0000 7f2f52a00640 1 -- 172.21.15.27:0/232068194 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f2f440ab630 con 0x7f2f34066ce0 2024-01-15T18:08:20.573 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:08:20.574 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:08:20.574 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:08:20.574 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:08:20.574 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:08:20.574 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:08:20.574 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:08:20.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:20.574 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:08:20.574 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 33.33333333333333, 2024-01-15T18:08:20.574 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "1 GB/3 GB", 2024-01-15T18:08:20.574 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "1/3", 2024-01-15T18:08:20.574 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:08:20.574 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:08:20.575 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:20.575 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:20.575 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:08:20.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.571+0000 7f2f61400640 1 -- 172.21.15.27:0/232068194 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2f34066ce0 msgr2=0x7f2f340691a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:20.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.571+0000 7f2f61400640 1 --2- 172.21.15.27:0/232068194 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2f34066ce0 0x7f2f340691a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2f440af000 tx=0x7f2f440b6040 comp rx=0 tx=0).stop 2024-01-15T18:08:20.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.571+0000 7f2f61400640 1 -- 172.21.15.27:0/232068194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f5c0cdb20 msgr2=0x7f2f5c146580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:20.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.571+0000 7f2f61400640 1 --2- 172.21.15.27:0/232068194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f5c0cdb20 0x7f2f5c146580 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c00c4d0 tx=0x7f2f4c00c990 comp rx=0 tx=0).stop 2024-01-15T18:08:20.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.571+0000 7f2f61400640 1 -- 172.21.15.27:0/232068194 shutdown_connections 2024-01-15T18:08:20.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.571+0000 7f2f61400640 1 --2- 172.21.15.27:0/232068194 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2f34066ce0 0x7f2f340691a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:20.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.571+0000 7f2f61400640 1 --2- 172.21.15.27:0/232068194 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2f5c161a90 0x7f2f5c1fe210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:20.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.571+0000 7f2f61400640 1 --2- 172.21.15.27:0/232068194 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2f5c0ce810 0x7f2f5c163720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:20.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.571+0000 7f2f61400640 1 --2- 172.21.15.27:0/232068194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f5c0cdb20 0x7f2f5c146580 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:20.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.571+0000 7f2f61400640 1 -- 172.21.15.27:0/232068194 >> 172.21.15.27:0/232068194 conn(0x7f2f5c0cb700 msgr2=0x7f2f5c0709c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:20.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.571+0000 7f2f61400640 1 -- 172.21.15.27:0/232068194 shutdown_connections 2024-01-15T18:08:20.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:20.571+0000 7f2f61400640 1 -- 172.21.15.27:0/232068194 wait complete. 2024-01-15T18:08:20.595 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:08:21.595 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone63891 2024-01-15T18:08:21.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.747+0000 7f0c68e00640 1 Processor -- start 2024-01-15T18:08:21.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.747+0000 7f0c68e00640 1 -- start start 2024-01-15T18:08:21.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.747+0000 7f0c68e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0c640cf2e0 0x7f0c640701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:21.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.747+0000 7f0c68e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c640b1cb0 con 0x7f0c64158900 2024-01-15T18:08:21.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.747+0000 7f0c68e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c6415fa00 con 0x7f0c6415f550 2024-01-15T18:08:21.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.747+0000 7f0c68e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c640b1f00 con 0x7f0c640cf2e0 2024-01-15T18:08:21.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.747+0000 7f0c61e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0c64158900 0x7f0c6415bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44534/0 (socket says 172.21.15.27:44534) 2024-01-15T18:08:21.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.747+0000 7f0c61e00640 1 -- 172.21.15.27:0/92613551 learned_addr learned my addr 172.21.15.27:0/92613551 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:21.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.747+0000 7f0c62800640 1 --2- 172.21.15.27:0/92613551 >> v2:172.21.15.93:3300/0 conn(0x7f0c640cf2e0 0x7f0c640701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:21.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.747+0000 7f0c60a00640 1 -- 172.21.15.27:0/92613551 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101865393 0 0) 0x7f0c640b1cb0 con 0x7f0c64158900 2024-01-15T18:08:21.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.747+0000 7f0c60a00640 1 -- 172.21.15.27:0/92613551 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c44003530 con 0x7f0c64158900 2024-01-15T18:08:21.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.747+0000 7f0c60a00640 1 -- 172.21.15.27:0/92613551 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 434541796 0 0) 0x7f0c44003530 con 0x7f0c64158900 2024-01-15T18:08:21.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.747+0000 7f0c60a00640 1 -- 172.21.15.27:0/92613551 >> v2:172.21.15.93:3300/0 conn(0x7f0c640cf2e0 msgr2=0x7f0c640701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:21.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.747+0000 7f0c60a00640 1 --2- 172.21.15.27:0/92613551 >> v2:172.21.15.93:3300/0 conn(0x7f0c640cf2e0 0x7f0c640701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:21.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.747+0000 7f0c60a00640 1 -- 172.21.15.27:0/92613551 >> v1:172.21.15.93:6789/0 conn(0x7f0c6415f550 legacy=0x7f0c640cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:21.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.747+0000 7f0c60a00640 1 -- 172.21.15.27:0/92613551 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c640b21e0 con 0x7f0c64158900 2024-01-15T18:08:21.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.747+0000 7f0c60a00640 1 -- 172.21.15.27:0/92613551 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f0c54001680 con 0x7f0c64158900 2024-01-15T18:08:21.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.747+0000 7f0c60a00640 1 -- 172.21.15.27:0/92613551 >> v1:172.21.15.27:6789/0 conn(0x7f0c64158900 legacy=0x7f0c6415bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:21.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.747+0000 7f0c60a00640 1 --2- 172.21.15.27:0/92613551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c440039f0 0x7f0c44003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:21.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.747+0000 7f0c60a00640 1 --2- 172.21.15.27:0/92613551 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0c440047d0 0x7f0c44004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:21.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.747+0000 7f0c60a00640 1 --2- 172.21.15.27:0/92613551 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0c44005190 0x7f0c4400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:21.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.747+0000 7f0c60a00640 1 -- 172.21.15.27:0/92613551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c640b1f00 con 0x7f0c440039f0 2024-01-15T18:08:21.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.747+0000 7f0c60a00640 1 -- 172.21.15.27:0/92613551 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4400c0c0 con 0x7f0c44005190 2024-01-15T18:08:21.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.747+0000 7f0c60a00640 1 -- 172.21.15.27:0/92613551 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4400c280 con 0x7f0c440047d0 2024-01-15T18:08:21.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.751+0000 7f0c61e00640 1 --2- 172.21.15.27:0/92613551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c440039f0 0x7f0c44003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:21.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.751+0000 7f0c61400640 1 --2- 172.21.15.27:0/92613551 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0c440047d0 0x7f0c44004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:21.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.751+0000 7f0c62800640 1 --2- 172.21.15.27:0/92613551 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0c44005190 0x7f0c4400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:21.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.751+0000 7f0c61e00640 1 -- 172.21.15.27:0/92613551 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0c440047d0 msgr2=0x7f0c44004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:21.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.751+0000 7f0c61e00640 1 --2- 172.21.15.27:0/92613551 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0c440047d0 0x7f0c44004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:21.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.751+0000 7f0c61e00640 1 -- 172.21.15.27:0/92613551 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0c44005190 msgr2=0x7f0c4400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:21.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.751+0000 7f0c61e00640 1 --2- 172.21.15.27:0/92613551 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0c44005190 0x7f0c4400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:21.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.751+0000 7f0c61e00640 1 -- 172.21.15.27:0/92613551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c44003530 con 0x7f0c440039f0 2024-01-15T18:08:21.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.751+0000 7f0c62800640 1 -- 172.21.15.27:0/92613551 reap_dead start 2024-01-15T18:08:21.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.751+0000 7f0c61e00640 1 --2- 172.21.15.27:0/92613551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c440039f0 0x7f0c44003e10 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f0c5400c240 tx=0x7f0c5401cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54c02a2ed5239c1e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:21.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.751+0000 7f0c60a00640 1 -- 172.21.15.27:0/92613551 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c54005ae0 con 0x7f0c440039f0 2024-01-15T18:08:21.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.751+0000 7f0c60a00640 1 -- 172.21.15.27:0/92613551 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c54005c40 con 0x7f0c440039f0 2024-01-15T18:08:21.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.751+0000 7f0c60a00640 1 -- 172.21.15.27:0/92613551 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c540072b0 con 0x7f0c440039f0 2024-01-15T18:08:21.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.755+0000 7f0c68e00640 1 -- 172.21.15.27:0/92613551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c440039f0 msgr2=0x7f0c44003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:21.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.755+0000 7f0c68e00640 1 --2- 172.21.15.27:0/92613551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c440039f0 0x7f0c44003e10 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f0c5400c240 tx=0x7f0c5401cb90 comp rx=0 tx=0).stop 2024-01-15T18:08:21.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.755+0000 7f0c68e00640 1 -- 172.21.15.27:0/92613551 shutdown_connections 2024-01-15T18:08:21.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.755+0000 7f0c68e00640 1 --2- 172.21.15.27:0/92613551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c440039f0 0x7f0c44003e10 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:21.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.755+0000 7f0c68e00640 1 -- 172.21.15.27:0/92613551 >> 172.21.15.27:0/92613551 conn(0x7f0c640cb700 msgr2=0x7f0c640cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:21.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.755+0000 7f0c68e00640 1 -- 172.21.15.27:0/92613551 shutdown_connections 2024-01-15T18:08:21.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.755+0000 7f0c68e00640 1 -- 172.21.15.27:0/92613551 wait complete. 2024-01-15T18:08:21.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.755+0000 7f0c68e00640 1 Processor -- start 2024-01-15T18:08:21.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.755+0000 7f0c68e00640 1 -- start start 2024-01-15T18:08:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.755+0000 7f0c68e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c440039f0 0x7f0c64144740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.755+0000 7f0c68e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0c641569a0 0x7f0c64156e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.755+0000 7f0c68e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0c64157370 0x7f0c640ad100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.755+0000 7f0c68e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c64144c80 con 0x7f0c440039f0 2024-01-15T18:08:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.755+0000 7f0c68e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c64155330 con 0x7f0c641569a0 2024-01-15T18:08:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.755+0000 7f0c68e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c64155610 con 0x7f0c64157370 2024-01-15T18:08:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.755+0000 7f0c61e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c440039f0 0x7f0c64144740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:21.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.755+0000 7f0c61e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c440039f0 0x7f0c64144740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37250/0 (socket says 172.21.15.27:37250) 2024-01-15T18:08:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.755+0000 7f0c61e00640 1 -- 172.21.15.27:0/1141968449 learned_addr learned my addr 172.21.15.27:0/1141968449 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.759+0000 7f0c61400640 1 --2- 172.21.15.27:0/1141968449 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0c641569a0 0x7f0c64156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:21.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.759+0000 7f0c62800640 1 --2- 172.21.15.27:0/1141968449 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0c64157370 0x7f0c640ad100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.759+0000 7f0c61e00640 1 -- 172.21.15.27:0/1141968449 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0c64157370 msgr2=0x7f0c640ad100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.759+0000 7f0c61e00640 1 --2- 172.21.15.27:0/1141968449 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0c64157370 0x7f0c640ad100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.759+0000 7f0c61e00640 1 -- 172.21.15.27:0/1141968449 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0c641569a0 msgr2=0x7f0c64156e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.759+0000 7f0c61e00640 1 --2- 172.21.15.27:0/1141968449 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0c641569a0 0x7f0c64156e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.759+0000 7f0c61e00640 1 -- 172.21.15.27:0/1141968449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c641558f0 con 0x7f0c440039f0 2024-01-15T18:08:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.759+0000 7f0c61e00640 1 --2- 172.21.15.27:0/1141968449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c440039f0 0x7f0c64144740 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f0c5401c9c0 tx=0x7f0c44005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:21.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.759+0000 7f0c5a000640 1 -- 172.21.15.27:0/1141968449 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c540b4070 con 0x7f0c440039f0 2024-01-15T18:08:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.759+0000 7f0c5a000640 1 -- 172.21.15.27:0/1141968449 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c54001540 con 0x7f0c440039f0 2024-01-15T18:08:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.759+0000 7f0c68e00640 1 -- 172.21.15.27:0/1141968449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c64155bd0 con 0x7f0c440039f0 2024-01-15T18:08:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.759+0000 7f0c5a000640 1 -- 172.21.15.27:0/1141968449 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c54007290 con 0x7f0c440039f0 2024-01-15T18:08:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.759+0000 7f0c68e00640 1 -- 172.21.15.27:0/1141968449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c640ad640 con 0x7f0c440039f0 2024-01-15T18:08:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.759+0000 7f0c68e00640 1 -- 172.21.15.27:0/1141968449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c64155750 con 0x7f0c440039f0 2024-01-15T18:08:21.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.759+0000 7f0c5a000640 1 -- 172.21.15.27:0/1141968449 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 10) v1 ==== 86517+0+0 (secure 0 0 0) 0x7f0c540073f0 con 0x7f0c440039f0 2024-01-15T18:08:21.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.759+0000 7f0c5a000640 1 --2- 172.21.15.27:0/1141968449 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c3c066d10 0x7f0c3c0691d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:21.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.763+0000 7f0c5a000640 1 -- 172.21.15.27:0/1141968449 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0c5411e8e0 con 0x7f0c440039f0 2024-01-15T18:08:21.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.763+0000 7f0c61400640 1 --2- 172.21.15.27:0/1141968449 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c3c066d10 0x7f0c3c0691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:21.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.763+0000 7f0c61400640 1 --2- 172.21.15.27:0/1141968449 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c3c066d10 0x7f0c3c0691d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0c640adc50 tx=0x7f0c4c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:21.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.763+0000 7f0c5a000640 1 -- 172.21.15.27:0/1141968449 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0c64155750 con 0x7f0c440039f0 2024-01-15T18:08:21.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.955+0000 7f0c68e00640 1 -- 172.21.15.27:0/1141968449 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone63891", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c6415c310 con 0x7f0c3c066d10 2024-01-15T18:08:21.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.963+0000 7f0c5a000640 1 -- 172.21.15.27:0/1141968449 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f0c4c0015b0 con 0x7f0c3c066d10 2024-01-15T18:08:21.968 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:08:21.968 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:08:21.968 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:08:21.968 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:08:21.968 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:08:21.968 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:08:21.968 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:08:21.969 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:21.969 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:08:21.969 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 33.33333333333333, 2024-01-15T18:08:21.969 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "1 GB/3 GB", 2024-01-15T18:08:21.969 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "1/3", 2024-01-15T18:08:21.969 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:08:21.969 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:08:21.969 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:21.969 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:21.969 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:08:21.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.963+0000 7f0c68e00640 1 -- 172.21.15.27:0/1141968449 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c3c066d10 msgr2=0x7f0c3c0691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:21.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.963+0000 7f0c68e00640 1 --2- 172.21.15.27:0/1141968449 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c3c066d10 0x7f0c3c0691d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0c640adc50 tx=0x7f0c4c009040 comp rx=0 tx=0).stop 2024-01-15T18:08:21.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.963+0000 7f0c68e00640 1 -- 172.21.15.27:0/1141968449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c440039f0 msgr2=0x7f0c64144740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:21.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.963+0000 7f0c68e00640 1 --2- 172.21.15.27:0/1141968449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c440039f0 0x7f0c64144740 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f0c5401c9c0 tx=0x7f0c44005800 comp rx=0 tx=0).stop 2024-01-15T18:08:21.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.963+0000 7f0c68e00640 1 -- 172.21.15.27:0/1141968449 shutdown_connections 2024-01-15T18:08:21.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.963+0000 7f0c68e00640 1 --2- 172.21.15.27:0/1141968449 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c3c066d10 0x7f0c3c0691d0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:21.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.963+0000 7f0c68e00640 1 --2- 172.21.15.27:0/1141968449 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0c64157370 0x7f0c640ad100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:21.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.963+0000 7f0c68e00640 1 --2- 172.21.15.27:0/1141968449 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0c641569a0 0x7f0c64156e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:21.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.963+0000 7f0c68e00640 1 --2- 172.21.15.27:0/1141968449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c440039f0 0x7f0c64144740 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:21.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.963+0000 7f0c68e00640 1 -- 172.21.15.27:0/1141968449 >> 172.21.15.27:0/1141968449 conn(0x7f0c640cb700 msgr2=0x7f0c6406c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:21.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.967+0000 7f0c68e00640 1 -- 172.21.15.27:0/1141968449 shutdown_connections 2024-01-15T18:08:21.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:21.967+0000 7f0c68e00640 1 -- 172.21.15.27:0/1141968449 wait complete. 2024-01-15T18:08:21.990 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:08:22.991 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone63891 2024-01-15T18:08:23.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.243+0000 7f6ba0000640 1 Processor -- start 2024-01-15T18:08:23.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.243+0000 7f6ba0000640 1 -- start start 2024-01-15T18:08:23.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.247+0000 7f6ba0000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f6b981468d0 0x7f6b980ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:23.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.247+0000 7f6ba0000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b980ad920 con 0x7f6b980cef30 2024-01-15T18:08:23.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.247+0000 7f6ba0000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b980ac880 con 0x7f6b980cd070 2024-01-15T18:08:23.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.247+0000 7f6ba0000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b980cd4c0 con 0x7f6b981468d0 2024-01-15T18:08:23.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.247+0000 7f6b9cc00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6b980cef30 0x7f6b980cc960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44544/0 (socket says 172.21.15.27:44544) 2024-01-15T18:08:23.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.247+0000 7f6b9cc00640 1 -- 172.21.15.27:0/4236250070 learned_addr learned my addr 172.21.15.27:0/4236250070 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:23.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.247+0000 7f6b9d600640 1 --2- 172.21.15.27:0/4236250070 >> v2:172.21.15.93:3300/0 conn(0x7f6b981468d0 0x7f6b980ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:23.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.251+0000 7f6b97e00640 1 -- 172.21.15.27:0/4236250070 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599940811 0 0) 0x7f6b88001120 con 0x7f6b980cef30 2024-01-15T18:08:23.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.251+0000 7f6b97e00640 1 -- 172.21.15.27:0/4236250070 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b78003530 con 0x7f6b980cef30 2024-01-15T18:08:23.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.251+0000 7f6b9d600640 1 -- 172.21.15.27:0/4236250070 >> v1:172.21.15.93:6789/0 conn(0x7f6b980cd070 legacy=0x7f6b9806c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:23.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.251+0000 7f6b9d600640 1 -- 172.21.15.27:0/4236250070 >> v1:172.21.15.27:6789/0 conn(0x7f6b980cef30 legacy=0x7f6b980cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:23.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.251+0000 7f6b9d600640 1 -- 172.21.15.27:0/4236250070 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b980acb60 con 0x7f6b981468d0 2024-01-15T18:08:23.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.255+0000 7f6b9d600640 1 --2- 172.21.15.27:0/4236250070 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6b981468d0 0x7f6b980ce9f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6b90009d90 tx=0x7f6b9001c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=175dd55b454dd892 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:23.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.259+0000 7f6b97e00640 1 -- 172.21.15.27:0/4236250070 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b900a5a20 con 0x7f6b981468d0 2024-01-15T18:08:23.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.259+0000 7f6b97e00640 1 -- 172.21.15.27:0/4236250070 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b900a5b80 con 0x7f6b981468d0 2024-01-15T18:08:23.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.259+0000 7f6b97e00640 1 -- 172.21.15.27:0/4236250070 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b900ad440 con 0x7f6b981468d0 2024-01-15T18:08:23.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.259+0000 7f6ba0000640 1 -- 172.21.15.27:0/4236250070 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6b981468d0 msgr2=0x7f6b980ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:23.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.259+0000 7f6ba0000640 1 --2- 172.21.15.27:0/4236250070 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6b981468d0 0x7f6b980ce9f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6b90009d90 tx=0x7f6b9001c960 comp rx=0 tx=0).stop 2024-01-15T18:08:23.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.259+0000 7f6ba0000640 1 -- 172.21.15.27:0/4236250070 shutdown_connections 2024-01-15T18:08:23.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.259+0000 7f6ba0000640 1 --2- 172.21.15.27:0/4236250070 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6b981468d0 0x7f6b980ce9f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:23.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.259+0000 7f6ba0000640 1 -- 172.21.15.27:0/4236250070 >> 172.21.15.27:0/4236250070 conn(0x7f6b980cb700 msgr2=0x7f6b980cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:23.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.259+0000 7f6ba0000640 1 -- 172.21.15.27:0/4236250070 shutdown_connections 2024-01-15T18:08:23.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.259+0000 7f6ba0000640 1 -- 172.21.15.27:0/4236250070 wait complete. 2024-01-15T18:08:23.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.263+0000 7f6ba0000640 1 Processor -- start 2024-01-15T18:08:23.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.263+0000 7f6ba0000640 1 -- start start 2024-01-15T18:08:23.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.263+0000 7f6ba0000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6b980cd070 0x7f6b980b2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:23.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.263+0000 7f6ba0000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b980cef30 0x7f6b9813b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:23.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.263+0000 7f6ba0000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6b981468d0 0x7f6b9815fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:23.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.263+0000 7f6ba0000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9813b840 con 0x7f6b980cef30 2024-01-15T18:08:23.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.263+0000 7f6ba0000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9813ba70 con 0x7f6b980cd070 2024-01-15T18:08:23.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.263+0000 7f6ba0000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9813bd50 con 0x7f6b981468d0 2024-01-15T18:08:23.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.263+0000 7f6b9cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b980cef30 0x7f6b9813b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:23.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.263+0000 7f6b9cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b980cef30 0x7f6b9813b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37252/0 (socket says 172.21.15.27:37252) 2024-01-15T18:08:23.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.263+0000 7f6b9cc00640 1 -- 172.21.15.27:0/2187557359 learned_addr learned my addr 172.21.15.27:0/2187557359 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:23.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.267+0000 7f6b9e000640 1 --2- 172.21.15.27:0/2187557359 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6b981468d0 0x7f6b9815fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:23.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.267+0000 7f6b9cc00640 1 -- 172.21.15.27:0/2187557359 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6b981468d0 msgr2=0x7f6b9815fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:23.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.267+0000 7f6b9cc00640 1 --2- 172.21.15.27:0/2187557359 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6b981468d0 0x7f6b9815fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:23.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.267+0000 7f6b9cc00640 1 -- 172.21.15.27:0/2187557359 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6b980cd070 msgr2=0x7f6b980b2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:23.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.267+0000 7f6b9cc00640 1 --2- 172.21.15.27:0/2187557359 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6b980cd070 0x7f6b980b2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:23.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.267+0000 7f6b9cc00640 1 -- 172.21.15.27:0/2187557359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b90009a70 con 0x7f6b980cef30 2024-01-15T18:08:23.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.267+0000 7f6b9cc00640 1 --2- 172.21.15.27:0/2187557359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b980cef30 0x7f6b9813b300 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6b8800c4d0 tx=0x7f6b8800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:23.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.267+0000 7f6b95600640 1 -- 172.21.15.27:0/2187557359 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b88005cf0 con 0x7f6b980cef30 2024-01-15T18:08:23.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.267+0000 7f6b95600640 1 -- 172.21.15.27:0/2187557359 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b88005e50 con 0x7f6b980cef30 2024-01-15T18:08:23.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.267+0000 7f6b95600640 1 -- 172.21.15.27:0/2187557359 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b88007290 con 0x7f6b980cef30 2024-01-15T18:08:23.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.267+0000 7f6ba0000640 1 -- 172.21.15.27:0/2187557359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b9813c030 con 0x7f6b980cef30 2024-01-15T18:08:23.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.267+0000 7f6ba0000640 1 -- 172.21.15.27:0/2187557359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b9813c5f0 con 0x7f6b980cef30 2024-01-15T18:08:23.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.271+0000 7f6ba0000640 1 -- 172.21.15.27:0/2187557359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b9813c810 con 0x7f6b980cef30 2024-01-15T18:08:23.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.271+0000 7f6b95600640 1 -- 172.21.15.27:0/2187557359 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 11) v1 ==== 86568+0+0 (secure 0 0 0) 0x7f6b880073f0 con 0x7f6b980cef30 2024-01-15T18:08:23.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.275+0000 7f6b95600640 1 --2- 172.21.15.27:0/2187557359 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6b70066dc0 0x7f6b70069280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:23.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.303+0000 7f6b95600640 1 -- 172.21.15.27:0/2187557359 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6b88055060 con 0x7f6b980cef30 2024-01-15T18:08:23.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.315+0000 7f6b95600640 1 -- 172.21.15.27:0/2187557359 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6b88050ca0 con 0x7f6b980cef30 2024-01-15T18:08:23.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.319+0000 7f6b9d600640 1 --2- 172.21.15.27:0/2187557359 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6b70066dc0 0x7f6b70069280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.359+0000 7f6b9d600640 1 --2- 172.21.15.27:0/2187557359 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6b70066dc0 0x7f6b70069280 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6b900af000 tx=0x7f6b90004ac0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:23.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.651+0000 7f6ba0000640 1 -- 172.21.15.27:0/2187557359 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone63891", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b9813bec0 con 0x7f6b70066dc0 2024-01-15T18:08:23.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.679+0000 7f6b95600640 1 -- 172.21.15.27:0/2187557359 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f6b9813bec0 con 0x7f6b70066dc0 2024-01-15T18:08:23.684 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:08:23.685 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:08:23.685 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:08:23.685 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:08:23.685 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:08:23.685 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:08:23.685 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:08:23.685 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:23.685 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:08:23.685 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 33.33333333333333, 2024-01-15T18:08:23.685 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "1 GB/3 GB", 2024-01-15T18:08:23.685 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "1/3", 2024-01-15T18:08:23.685 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:08:23.685 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:08:23.686 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:23.686 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:23.686 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:08:23.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.687+0000 7f6ba0000640 1 -- 172.21.15.27:0/2187557359 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6b70066dc0 msgr2=0x7f6b70069280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:23.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.687+0000 7f6ba0000640 1 --2- 172.21.15.27:0/2187557359 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6b70066dc0 0x7f6b70069280 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6b900af000 tx=0x7f6b90004ac0 comp rx=0 tx=0).stop 2024-01-15T18:08:23.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.687+0000 7f6ba0000640 1 -- 172.21.15.27:0/2187557359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b980cef30 msgr2=0x7f6b9813b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:23.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.687+0000 7f6ba0000640 1 --2- 172.21.15.27:0/2187557359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b980cef30 0x7f6b9813b300 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6b8800c4d0 tx=0x7f6b8800c990 comp rx=0 tx=0).stop 2024-01-15T18:08:23.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.687+0000 7f6ba0000640 1 -- 172.21.15.27:0/2187557359 shutdown_connections 2024-01-15T18:08:23.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.687+0000 7f6ba0000640 1 --2- 172.21.15.27:0/2187557359 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6b70066dc0 0x7f6b70069280 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:23.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.687+0000 7f6ba0000640 1 --2- 172.21.15.27:0/2187557359 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6b981468d0 0x7f6b9815fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:23.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.687+0000 7f6ba0000640 1 --2- 172.21.15.27:0/2187557359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b980cef30 0x7f6b9813b300 secure :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6b8800c4d0 tx=0x7f6b8800c990 comp rx=0 tx=0).stop 2024-01-15T18:08:23.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.687+0000 7f6ba0000640 1 --2- 172.21.15.27:0/2187557359 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6b980cd070 0x7f6b980b2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:23.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.687+0000 7f6ba0000640 1 -- 172.21.15.27:0/2187557359 >> 172.21.15.27:0/2187557359 conn(0x7f6b980cb700 msgr2=0x7f6b980cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:23.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.687+0000 7f6ba0000640 1 -- 172.21.15.27:0/2187557359 shutdown_connections 2024-01-15T18:08:23.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:23.687+0000 7f6ba0000640 1 -- 172.21.15.27:0/2187557359 wait complete. 2024-01-15T18:08:23.737 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:08:24.738 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone63891 2024-01-15T18:08:24.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:24.995+0000 7f5193200640 1 Processor -- start 2024-01-15T18:08:25.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:24.995+0000 7f5193200640 1 -- start start 2024-01-15T18:08:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:24.995+0000 7f5193200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f518c161a90 0x7f518c161f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:24.995+0000 7f5193200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f518c0ce810 0x7f518c0ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:24.995+0000 7f5193200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f518c06c5d0 con 0x7f518c0cdb20 2024-01-15T18:08:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:24.995+0000 7f5193200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f518c0cf1e0 con 0x7f518c161a90 2024-01-15T18:08:25.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:24.995+0000 7f5193200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f518c162b00 con 0x7f518c0ce810 2024-01-15T18:08:25.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:24.999+0000 7f518b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f518c161a90 0x7f518c161f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:25.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:24.999+0000 7f518be00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f518c0cdb20 0x7f518c06cd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35548/0 (socket says 172.21.15.27:35548) 2024-01-15T18:08:25.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:24.999+0000 7f518be00640 1 -- 172.21.15.27:0/3089362828 learned_addr learned my addr 172.21.15.27:0/3089362828 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:25.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:24.999+0000 7f5191200640 1 --2- 172.21.15.27:0/3089362828 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f518c0ce810 0x7f518c0ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:25.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:24.999+0000 7f518b400640 1 -- 172.21.15.27:0/3089362828 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f518c0ce810 msgr2=0x7f518c0ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:25.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:24.999+0000 7f518b400640 1 --2- 172.21.15.27:0/3089362828 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f518c0ce810 0x7f518c0ceca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:25.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:24.999+0000 7f518b400640 1 -- 172.21.15.27:0/3089362828 >> v1:172.21.15.27:6789/0 conn(0x7f518c0cdb20 legacy=0x7f518c06cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:25.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:24.999+0000 7f518b400640 1 -- 172.21.15.27:0/3089362828 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f518c06d4a0 con 0x7f518c161a90 2024-01-15T18:08:25.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.003+0000 7f518b400640 1 --2- 172.21.15.27:0/3089362828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f518c161a90 0x7f518c161f20 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f517c0097f0 tx=0x7f517c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=964ea7ea32abef02 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:25.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.003+0000 7f518aa00640 1 -- 172.21.15.27:0/3089362828 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517c0ab650 con 0x7f518c161a90 2024-01-15T18:08:25.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.003+0000 7f518aa00640 1 -- 172.21.15.27:0/3089362828 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f517c0ab7b0 con 0x7f518c161a90 2024-01-15T18:08:25.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.003+0000 7f518aa00640 1 -- 172.21.15.27:0/3089362828 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517c0abb40 con 0x7f518c161a90 2024-01-15T18:08:25.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.003+0000 7f5193200640 1 -- 172.21.15.27:0/3089362828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f518c161a90 msgr2=0x7f518c161f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:25.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.003+0000 7f5193200640 1 --2- 172.21.15.27:0/3089362828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f518c161a90 0x7f518c161f20 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f517c0097f0 tx=0x7f517c01c450 comp rx=0 tx=0).stop 2024-01-15T18:08:25.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.007+0000 7f5193200640 1 -- 172.21.15.27:0/3089362828 shutdown_connections 2024-01-15T18:08:25.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.007+0000 7f5193200640 1 --2- 172.21.15.27:0/3089362828 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f518c0ce810 0x7f518c0ceca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:25.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.007+0000 7f5193200640 1 --2- 172.21.15.27:0/3089362828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f518c161a90 0x7f518c161f20 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:25.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.007+0000 7f5193200640 1 -- 172.21.15.27:0/3089362828 >> 172.21.15.27:0/3089362828 conn(0x7f518c0cb700 msgr2=0x7f518c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:25.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.007+0000 7f5193200640 1 -- 172.21.15.27:0/3089362828 shutdown_connections 2024-01-15T18:08:25.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.007+0000 7f5193200640 1 -- 172.21.15.27:0/3089362828 wait complete. 2024-01-15T18:08:25.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.007+0000 7f5193200640 1 Processor -- start 2024-01-15T18:08:25.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.015+0000 7f5193200640 1 -- start start 2024-01-15T18:08:25.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.019+0000 7f5193200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f518c0cdb20 0x7f518c13de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:25.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.019+0000 7f5193200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f518c0ce810 0x7f518c14e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:25.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.019+0000 7f5193200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f518c161a90 0x7f518c0b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:25.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.019+0000 7f5193200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518c0b3480 con 0x7f518c0ce810 2024-01-15T18:08:25.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.019+0000 7f5193200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518c151b30 con 0x7f518c0cdb20 2024-01-15T18:08:25.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.019+0000 7f5193200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f518c14ec60 con 0x7f518c161a90 2024-01-15T18:08:25.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.019+0000 7f518b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f518c0ce810 0x7f518c14e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:25.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.019+0000 7f518b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f518c0ce810 0x7f518c14e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56530/0 (socket says 172.21.15.27:56530) 2024-01-15T18:08:25.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.019+0000 7f518b400640 1 -- 172.21.15.27:0/1014140128 learned_addr learned my addr 172.21.15.27:0/1014140128 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:25.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.019+0000 7f5191200640 1 --2- 172.21.15.27:0/1014140128 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f518c161a90 0x7f518c0b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:25.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.019+0000 7f518b400640 1 -- 172.21.15.27:0/1014140128 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f518c161a90 msgr2=0x7f518c0b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:25.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.019+0000 7f518b400640 1 --2- 172.21.15.27:0/1014140128 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f518c161a90 0x7f518c0b3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:25.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.019+0000 7f518b400640 1 -- 172.21.15.27:0/1014140128 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f518c0cdb20 msgr2=0x7f518c13de40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-15T18:08:25.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.019+0000 7f518b400640 1 --2- 172.21.15.27:0/1014140128 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f518c0cdb20 0x7f518c13de40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:25.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.019+0000 7f518b400640 1 -- 172.21.15.27:0/1014140128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517c009420 con 0x7f518c0ce810 2024-01-15T18:08:25.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.019+0000 7f518b400640 1 --2- 172.21.15.27:0/1014140128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f518c0ce810 0x7f518c14e720 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f517c0097f0 tx=0x7f517c0191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:25.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.023+0000 7f5183e00640 1 -- 172.21.15.27:0/1014140128 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517c0abe70 con 0x7f518c0ce810 2024-01-15T18:08:25.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.023+0000 7f5183e00640 1 -- 172.21.15.27:0/1014140128 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f517c0bf450 con 0x7f518c0ce810 2024-01-15T18:08:25.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.023+0000 7f5183e00640 1 -- 172.21.15.27:0/1014140128 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517c0abe70 con 0x7f518c0ce810 2024-01-15T18:08:25.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.023+0000 7f5193200640 1 -- 172.21.15.27:0/1014140128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f518c15b9c0 con 0x7f518c0ce810 2024-01-15T18:08:25.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.023+0000 7f5193200640 1 -- 172.21.15.27:0/1014140128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f518c151850 con 0x7f518c0ce810 2024-01-15T18:08:25.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.027+0000 7f5193200640 1 -- 172.21.15.27:0/1014140128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f518c04e850 con 0x7f518c0ce810 2024-01-15T18:08:25.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.027+0000 7f5183e00640 1 -- 172.21.15.27:0/1014140128 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 11) v1 ==== 86568+0+0 (secure 0 0 0) 0x7f517c0bf5c0 con 0x7f518c0ce810 2024-01-15T18:08:25.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.031+0000 7f5183e00640 1 --2- 172.21.15.27:0/1014140128 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5164066dd0 0x7f5164069290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:25.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.031+0000 7f518be00640 1 --2- 172.21.15.27:0/1014140128 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5164066dd0 0x7f5164069290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:25.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.031+0000 7f5183e00640 1 -- 172.21.15.27:0/1014140128 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f517c11f210 con 0x7f518c0ce810 2024-01-15T18:08:25.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.031+0000 7f518be00640 1 --2- 172.21.15.27:0/1014140128 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5164066dd0 0x7f5164069290 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5174001590 tx=0x7f5174007a50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:25.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.039+0000 7f5183e00640 1 -- 172.21.15.27:0/1014140128 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f517c0ebdc0 con 0x7f518c0ce810 2024-01-15T18:08:25.298 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:25.290+0000 7f59f8800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:25.298 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:25.290+0000 7f59f8800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:25.298 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:25.290+0000 7f59f8800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:25.298 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:25.290+0000 7f59f8800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:25.298 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:25.290+0000 7f59f8800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:25.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.359+0000 7f5193200640 1 -- 172.21.15.27:0/1014140128 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone63891", "target": ["mon-mgr", ""]}) v1 -- 0x7f518c06e390 con 0x7f5164066dd0 2024-01-15T18:08:25.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.719+0000 7f5183e00640 1 -- 172.21.15.27:0/1014140128 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f5174001250 con 0x7f5164066dd0 2024-01-15T18:08:25.724 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:08:25.724 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:08:25.724 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:08:25.724 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:08:25.724 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:08:25.724 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:08:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:08:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:08:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 33.33333333333333, 2024-01-15T18:08:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "1 GB/3 GB", 2024-01-15T18:08:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "1/3", 2024-01-15T18:08:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:08:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:08:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:25.725 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:25.725 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:08:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.723+0000 7f5193200640 1 -- 172.21.15.27:0/1014140128 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5164066dd0 msgr2=0x7f5164069290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.723+0000 7f5193200640 1 --2- 172.21.15.27:0/1014140128 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5164066dd0 0x7f5164069290 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5174001590 tx=0x7f5174007a50 comp rx=0 tx=0).stop 2024-01-15T18:08:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.723+0000 7f5193200640 1 -- 172.21.15.27:0/1014140128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f518c0ce810 msgr2=0x7f518c14e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.723+0000 7f5193200640 1 --2- 172.21.15.27:0/1014140128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f518c0ce810 0x7f518c14e720 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f517c0097f0 tx=0x7f517c0191f0 comp rx=0 tx=0).stop 2024-01-15T18:08:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.723+0000 7f5193200640 1 -- 172.21.15.27:0/1014140128 shutdown_connections 2024-01-15T18:08:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.723+0000 7f5193200640 1 --2- 172.21.15.27:0/1014140128 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5164066dd0 0x7f5164069290 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.723+0000 7f5193200640 1 --2- 172.21.15.27:0/1014140128 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f518c161a90 0x7f518c0b3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.723+0000 7f5193200640 1 --2- 172.21.15.27:0/1014140128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f518c0ce810 0x7f518c14e720 secure :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f517c0097f0 tx=0x7f517c0191f0 comp rx=0 tx=0).stop 2024-01-15T18:08:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.723+0000 7f5193200640 1 --2- 172.21.15.27:0/1014140128 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f518c0cdb20 0x7f518c13de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.723+0000 7f5193200640 1 -- 172.21.15.27:0/1014140128 >> 172.21.15.27:0/1014140128 conn(0x7f518c0cb700 msgr2=0x7f518c0aef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:25.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.723+0000 7f5193200640 1 -- 172.21.15.27:0/1014140128 shutdown_connections 2024-01-15T18:08:25.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:25.723+0000 7f5193200640 1 -- 172.21.15.27:0/1014140128 wait complete. 2024-01-15T18:08:25.790 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:08:26.527 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:08:26.531 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:08:26.791 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone63891 2024-01-15T18:08:26.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.931+0000 7fd768000640 1 Processor -- start 2024-01-15T18:08:26.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.931+0000 7fd768000640 1 -- start start 2024-01-15T18:08:26.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.931+0000 7fd768000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd76015b4b0 0x7fd7600b1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:26.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.931+0000 7fd768000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fd7600cec60 0x7fd7600b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:26.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.931+0000 7fd768000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7600d19a0 con 0x7fd7600cf460 2024-01-15T18:08:26.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.931+0000 7fd768000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7600d1c80 con 0x7fd76015b4b0 2024-01-15T18:08:26.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.931+0000 7fd768000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7600b07c0 con 0x7fd7600cec60 2024-01-15T18:08:26.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.931+0000 7fd764c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd76015b4b0 0x7fd7600b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:26.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.931+0000 7fd765600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd7600cf460 0x7fd7600cf840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35556/0 (socket says 172.21.15.27:35556) 2024-01-15T18:08:26.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.931+0000 7fd765600640 1 -- 172.21.15.27:0/3554965658 learned_addr learned my addr 172.21.15.27:0/3554965658 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:26.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.931+0000 7fd764c00640 1 --2- 172.21.15.27:0/3554965658 >> v2:172.21.15.27:3300/0 conn(0x7fd76015b4b0 0x7fd7600b1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56532/0 (socket says 172.21.15.27:56532) 2024-01-15T18:08:26.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.935+0000 7fd766000640 1 --2- 172.21.15.27:0/3554965658 >> v2:172.21.15.93:3300/0 conn(0x7fd7600cec60 0x7fd7600b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:26.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.935+0000 7fd764c00640 1 -- 172.21.15.27:0/3554965658 >> v2:172.21.15.93:3300/0 conn(0x7fd7600cec60 msgr2=0x7fd7600b1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:26.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.935+0000 7fd764c00640 1 --2- 172.21.15.27:0/3554965658 >> v2:172.21.15.93:3300/0 conn(0x7fd7600cec60 0x7fd7600b1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:26.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.935+0000 7fd764c00640 1 -- 172.21.15.27:0/3554965658 >> v1:172.21.15.27:6789/0 conn(0x7fd7600cf460 legacy=0x7fd7600cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:26.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.935+0000 7fd764c00640 1 -- 172.21.15.27:0/3554965658 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7600b0aa0 con 0x7fd76015b4b0 2024-01-15T18:08:26.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.935+0000 7fd764c00640 1 --2- 172.21.15.27:0/3554965658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd76015b4b0 0x7fd7600b1580 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fd76006af90 tx=0x7fd74801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fc5cc2766a59c29 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:26.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.935+0000 7fd75fe00640 1 -- 172.21.15.27:0/3554965658 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7600b2810 con 0x7fd76015b4b0 2024-01-15T18:08:26.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.935+0000 7fd75fe00640 1 -- 172.21.15.27:0/3554965658 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7480ab660 con 0x7fd76015b4b0 2024-01-15T18:08:26.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.935+0000 7fd75fe00640 1 -- 172.21.15.27:0/3554965658 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7480ab9f0 con 0x7fd76015b4b0 2024-01-15T18:08:26.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd768000640 1 -- 172.21.15.27:0/3554965658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd76015b4b0 msgr2=0x7fd7600b1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:26.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd768000640 1 --2- 172.21.15.27:0/3554965658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd76015b4b0 0x7fd7600b1580 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fd76006af90 tx=0x7fd74801c460 comp rx=0 tx=0).stop 2024-01-15T18:08:26.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd768000640 1 -- 172.21.15.27:0/3554965658 shutdown_connections 2024-01-15T18:08:26.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd768000640 1 --2- 172.21.15.27:0/3554965658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd76015b4b0 0x7fd7600b1580 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:26.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd768000640 1 --2- 172.21.15.27:0/3554965658 >> v2:172.21.15.93:3300/0 conn(0x7fd7600cec60 0x7fd7600b1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:26.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd768000640 1 -- 172.21.15.27:0/3554965658 >> 172.21.15.27:0/3554965658 conn(0x7fd7600d0e80 msgr2=0x7fd7600d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:26.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd768000640 1 -- 172.21.15.27:0/3554965658 shutdown_connections 2024-01-15T18:08:26.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd768000640 1 -- 172.21.15.27:0/3554965658 wait complete. 2024-01-15T18:08:26.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd768000640 1 Processor -- start 2024-01-15T18:08:26.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd768000640 1 -- start start 2024-01-15T18:08:26.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd768000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd7600cec60 0x7fd76014ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:26.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd768000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7600cf460 0x7fd760150530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:26.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd768000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd76015b4b0 0x7fd76006f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:26.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd768000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76015b9c0 con 0x7fd7600cf460 2024-01-15T18:08:26.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd768000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7600ca400 con 0x7fd7600cec60 2024-01-15T18:08:26.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd768000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7600ca6e0 con 0x7fd76015b4b0 2024-01-15T18:08:26.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd764c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7600cf460 0x7fd760150530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:26.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd764c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7600cf460 0x7fd760150530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56548/0 (socket says 172.21.15.27:56548) 2024-01-15T18:08:26.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd764c00640 1 -- 172.21.15.27:0/2396873184 learned_addr learned my addr 172.21.15.27:0/2396873184 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:26.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd766000640 1 --2- 172.21.15.27:0/2396873184 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd76015b4b0 0x7fd76006f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:26.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd765600640 1 --2- 172.21.15.27:0/2396873184 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd7600cec60 0x7fd76014ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:26.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd764c00640 1 -- 172.21.15.27:0/2396873184 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd76015b4b0 msgr2=0x7fd76006f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:26.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd764c00640 1 --2- 172.21.15.27:0/2396873184 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd76015b4b0 0x7fd76006f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:26.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd764c00640 1 -- 172.21.15.27:0/2396873184 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd7600cec60 msgr2=0x7fd76014ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:26.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd764c00640 1 --2- 172.21.15.27:0/2396873184 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd7600cec60 0x7fd76014ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:26.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.939+0000 7fd764c00640 1 -- 172.21.15.27:0/2396873184 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd748009420 con 0x7fd7600cf460 2024-01-15T18:08:26.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.943+0000 7fd764c00640 1 --2- 172.21.15.27:0/2396873184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7600cf460 0x7fd760150530 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fd748019070 tx=0x7fd74801aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:26.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.943+0000 7fd75d600640 1 -- 172.21.15.27:0/2396873184 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7480b5810 con 0x7fd7600cf460 2024-01-15T18:08:26.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.943+0000 7fd75d600640 1 -- 172.21.15.27:0/2396873184 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7480b5970 con 0x7fd7600cf460 2024-01-15T18:08:26.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.943+0000 7fd768000640 1 -- 172.21.15.27:0/2396873184 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7600ca9c0 con 0x7fd7600cf460 2024-01-15T18:08:26.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.943+0000 7fd75d600640 1 -- 172.21.15.27:0/2396873184 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7480b5d20 con 0x7fd7600cf460 2024-01-15T18:08:26.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.943+0000 7fd768000640 1 -- 172.21.15.27:0/2396873184 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd76014f8f0 con 0x7fd7600cf460 2024-01-15T18:08:26.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.943+0000 7fd768000640 1 -- 172.21.15.27:0/2396873184 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd76015bb00 con 0x7fd7600cf460 2024-01-15T18:08:26.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.943+0000 7fd75d600640 1 -- 172.21.15.27:0/2396873184 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fd7480a9070 con 0x7fd7600cf460 2024-01-15T18:08:26.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.943+0000 7fd75d600640 1 --2- 172.21.15.27:0/2396873184 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd738066e30 0x7fd7380692f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:26.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.943+0000 7fd75d600640 1 -- 172.21.15.27:0/2396873184 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd74811f6d0 con 0x7fd7600cf460 2024-01-15T18:08:26.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.943+0000 7fd765600640 1 --2- 172.21.15.27:0/2396873184 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd738066e30 0x7fd7380692f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:26.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.947+0000 7fd765600640 1 --2- 172.21.15.27:0/2396873184 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd738066e30 0x7fd7380692f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd750005890 tx=0x7fd750009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:26.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:26.947+0000 7fd75d600640 1 -- 172.21.15.27:0/2396873184 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd76015bb00 con 0x7fd7600cf460 2024-01-15T18:08:27.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:27.139+0000 7fd768000640 1 -- 172.21.15.27:0/2396873184 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone63891", "target": ["mon-mgr", ""]}) v1 -- 0x7fd76015bb00 con 0x7fd738066e30 2024-01-15T18:08:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:27.143+0000 7fd75d600640 1 -- 172.21.15.27:0/2396873184 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fd76015bb00 con 0x7fd738066e30 2024-01-15T18:08:27.149 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:08:27.149 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:08:27.149 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:08:27.149 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:08:27.149 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:08:27.149 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:08:27.149 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:08:27.149 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:27.149 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:08:27.150 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 66.66666666666666, 2024-01-15T18:08:27.150 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T18:08:27.150 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "2/3", 2024-01-15T18:08:27.150 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:08:27.150 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:08:27.150 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:27.150 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:27.150 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:08:27.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:27.147+0000 7fd768000640 1 -- 172.21.15.27:0/2396873184 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd738066e30 msgr2=0x7fd7380692f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:27.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:27.147+0000 7fd768000640 1 --2- 172.21.15.27:0/2396873184 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd738066e30 0x7fd7380692f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd750005890 tx=0x7fd750009040 comp rx=0 tx=0).stop 2024-01-15T18:08:27.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:27.147+0000 7fd768000640 1 -- 172.21.15.27:0/2396873184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7600cf460 msgr2=0x7fd760150530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:27.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:27.147+0000 7fd768000640 1 --2- 172.21.15.27:0/2396873184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7600cf460 0x7fd760150530 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fd748019070 tx=0x7fd74801aca0 comp rx=0 tx=0).stop 2024-01-15T18:08:27.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:27.147+0000 7fd768000640 1 -- 172.21.15.27:0/2396873184 shutdown_connections 2024-01-15T18:08:27.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:27.147+0000 7fd768000640 1 --2- 172.21.15.27:0/2396873184 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd738066e30 0x7fd7380692f0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:27.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:27.147+0000 7fd768000640 1 --2- 172.21.15.27:0/2396873184 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd76015b4b0 0x7fd76006f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:27.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:27.147+0000 7fd768000640 1 --2- 172.21.15.27:0/2396873184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7600cf460 0x7fd760150530 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:27.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:27.147+0000 7fd768000640 1 --2- 172.21.15.27:0/2396873184 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd7600cec60 0x7fd76014ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:27.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:27.147+0000 7fd768000640 1 -- 172.21.15.27:0/2396873184 >> 172.21.15.27:0/2396873184 conn(0x7fd7600d0e80 msgr2=0x7fd76006c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:27.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:27.147+0000 7fd768000640 1 -- 172.21.15.27:0/2396873184 shutdown_connections 2024-01-15T18:08:27.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:27.147+0000 7fd768000640 1 -- 172.21.15.27:0/2396873184 wait complete. 2024-01-15T18:08:27.172 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:08:28.173 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone63891 2024-01-15T18:08:28.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.363+0000 7f95cda00640 1 Processor -- start 2024-01-15T18:08:28.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.363+0000 7f95cda00640 1 -- start start 2024-01-15T18:08:28.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.363+0000 7f95cda00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f95c81617f0 0x7f95c806e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:28.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.363+0000 7f95cda00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f95c806dad0 0x7f95c8071c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:28.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.363+0000 7f95cda00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c8072620 con 0x7f95c8072190 2024-01-15T18:08:28.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.363+0000 7f95cda00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95c8161bd0 con 0x7f95c806dad0 2024-01-15T18:08:28.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.363+0000 7f95cda00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95c806ea50 con 0x7f95c81617f0 2024-01-15T18:08:28.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.363+0000 7f95c6000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f95c806dad0 0x7f95c8071c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:28.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.363+0000 7f95c6000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f95c806dad0 0x7f95c8071c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56564/0 (socket says 172.21.15.27:56564) 2024-01-15T18:08:28.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.363+0000 7f95c6000640 1 -- 172.21.15.27:0/2440146838 learned_addr learned my addr 172.21.15.27:0/2440146838 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:28.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.363+0000 7f95c6a00640 1 --2- 172.21.15.27:0/2440146838 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f95c81617f0 0x7f95c806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:28.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.363+0000 7f95c6000640 1 -- 172.21.15.27:0/2440146838 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f95c81617f0 msgr2=0x7f95c806e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:28.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.363+0000 7f95c6000640 1 --2- 172.21.15.27:0/2440146838 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f95c81617f0 0x7f95c806e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:28.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.363+0000 7f95c6000640 1 -- 172.21.15.27:0/2440146838 >> v1:172.21.15.93:6789/0 conn(0x7f95c8072190 legacy=0x7f95c8165250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:28.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.363+0000 7f95c6000640 1 -- 172.21.15.27:0/2440146838 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c80cf680 con 0x7f95c806dad0 2024-01-15T18:08:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.367+0000 7f95c6000640 1 --2- 172.21.15.27:0/2440146838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95c806dad0 0x7f95c8071c50 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f95b0009870 tx=0x7f95b001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1f9e7b053cf0e45 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.367+0000 7f95c5600640 1 -- 172.21.15.27:0/2440146838 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b00ab660 con 0x7f95c806dad0 2024-01-15T18:08:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.367+0000 7f95c5600640 1 -- 172.21.15.27:0/2440146838 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b00ab7c0 con 0x7f95c806dad0 2024-01-15T18:08:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.367+0000 7f95c5600640 1 -- 172.21.15.27:0/2440146838 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b00abb50 con 0x7f95c806dad0 2024-01-15T18:08:28.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.367+0000 7f95cda00640 1 -- 172.21.15.27:0/2440146838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95c806dad0 msgr2=0x7f95c8071c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:28.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.367+0000 7f95cda00640 1 --2- 172.21.15.27:0/2440146838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95c806dad0 0x7f95c8071c50 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f95b0009870 tx=0x7f95b001c540 comp rx=0 tx=0).stop 2024-01-15T18:08:28.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.371+0000 7f95cda00640 1 -- 172.21.15.27:0/2440146838 shutdown_connections 2024-01-15T18:08:28.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.371+0000 7f95cda00640 1 --2- 172.21.15.27:0/2440146838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95c806dad0 0x7f95c8071c50 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:28.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.371+0000 7f95cda00640 1 --2- 172.21.15.27:0/2440146838 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f95c81617f0 0x7f95c806e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:28.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.371+0000 7f95cda00640 1 -- 172.21.15.27:0/2440146838 >> 172.21.15.27:0/2440146838 conn(0x7f95c8099e10 msgr2=0x7f95c80b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:28.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.371+0000 7f95cda00640 1 -- 172.21.15.27:0/2440146838 shutdown_connections 2024-01-15T18:08:28.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.371+0000 7f95cda00640 1 -- 172.21.15.27:0/2440146838 wait complete. 2024-01-15T18:08:28.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.371+0000 7f95cda00640 1 Processor -- start 2024-01-15T18:08:28.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.371+0000 7f95cda00640 1 -- start start 2024-01-15T18:08:28.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.371+0000 7f95cda00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f95c806dad0 0x7f95c806be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:28.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.371+0000 7f95cda00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95c8072190 0x7f95c815c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:28.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.371+0000 7f95cda00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95c81617f0 0x7f95c815aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:28.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.371+0000 7f95cda00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c806c370 con 0x7f95c81617f0 2024-01-15T18:08:28.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.371+0000 7f95cda00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c806ad90 con 0x7f95c8072190 2024-01-15T18:08:28.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.371+0000 7f95cda00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95c806b070 con 0x7f95c806dad0 2024-01-15T18:08:28.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.371+0000 7f95c7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95c81617f0 0x7f95c815aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:28.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.371+0000 7f95c7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95c81617f0 0x7f95c815aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56578/0 (socket says 172.21.15.27:56578) 2024-01-15T18:08:28.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.371+0000 7f95c7e00640 1 -- 172.21.15.27:0/3522939190 learned_addr learned my addr 172.21.15.27:0/3522939190 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.375+0000 7f95c7e00640 1 -- 172.21.15.27:0/3522939190 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f95c806dad0 msgr2=0x7f95c806be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.375+0000 7f95c7e00640 1 --2- 172.21.15.27:0/3522939190 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f95c806dad0 0x7f95c806be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.375+0000 7f95c7e00640 1 -- 172.21.15.27:0/3522939190 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95c8072190 msgr2=0x7f95c815c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.375+0000 7f95c7e00640 1 --2- 172.21.15.27:0/3522939190 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95c8072190 0x7f95c815c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.375+0000 7f95c7e00640 1 -- 172.21.15.27:0/3522939190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b00094d0 con 0x7f95c81617f0 2024-01-15T18:08:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.375+0000 7f95c7e00640 1 --2- 172.21.15.27:0/3522939190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95c81617f0 0x7f95c815aef0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f95c000c770 tx=0x7f95c000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.375+0000 7f95bea00640 1 -- 172.21.15.27:0/3522939190 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c0003b80 con 0x7f95c81617f0 2024-01-15T18:08:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.375+0000 7f95bea00640 1 -- 172.21.15.27:0/3522939190 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95c0003ce0 con 0x7f95c81617f0 2024-01-15T18:08:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.375+0000 7f95bea00640 1 -- 172.21.15.27:0/3522939190 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c000aa90 con 0x7f95c81617f0 2024-01-15T18:08:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.375+0000 7f95cda00640 1 -- 172.21.15.27:0/3522939190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95c806b350 con 0x7f95c81617f0 2024-01-15T18:08:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.375+0000 7f95cda00640 1 -- 172.21.15.27:0/3522939190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c806b910 con 0x7f95c81617f0 2024-01-15T18:08:28.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.375+0000 7f95cda00640 1 -- 172.21.15.27:0/3522939190 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95c806bb00 con 0x7f95c81617f0 2024-01-15T18:08:28.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.375+0000 7f95bea00640 1 -- 172.21.15.27:0/3522939190 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f95c000abf0 con 0x7f95c81617f0 2024-01-15T18:08:28.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.379+0000 7f95bea00640 1 --2- 172.21.15.27:0/3522939190 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f95a0066e00 0x7f95a00692c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:28.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.379+0000 7f95bea00640 1 -- 172.21.15.27:0/3522939190 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f95c00849c0 con 0x7f95c81617f0 2024-01-15T18:08:28.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.379+0000 7f95c6a00640 1 --2- 172.21.15.27:0/3522939190 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f95a0066e00 0x7f95a00692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:28.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.383+0000 7f95c6a00640 1 --2- 172.21.15.27:0/3522939190 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f95a0066e00 0x7f95a00692c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f95b8006aa0 tx=0x7f95b8006a30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:28.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.387+0000 7f95bea00640 1 -- 172.21.15.27:0/3522939190 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f95c0051550 con 0x7f95c81617f0 2024-01-15T18:08:28.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.739+0000 7f95cda00640 1 -- 172.21.15.27:0/3522939190 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone63891", "target": ["mon-mgr", ""]}) v1 -- 0x7f95c80b49e0 con 0x7f95a0066e00 2024-01-15T18:08:28.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.767+0000 7f95bea00640 1 -- 172.21.15.27:0/3522939190 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f95b80096e0 con 0x7f95a0066e00 2024-01-15T18:08:28.773 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:08:28.773 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:08:28.774 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:08:28.774 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:08:28.774 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:08:28.774 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:08:28.774 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:08:28.774 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:28.774 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:08:28.774 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 66.66666666666666, 2024-01-15T18:08:28.774 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T18:08:28.774 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "2/3", 2024-01-15T18:08:28.774 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:08:28.774 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:08:28.774 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:28.775 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:28.775 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:08:28.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.771+0000 7f95cda00640 1 -- 172.21.15.27:0/3522939190 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f95a0066e00 msgr2=0x7f95a00692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:28.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.771+0000 7f95cda00640 1 --2- 172.21.15.27:0/3522939190 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f95a0066e00 0x7f95a00692c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f95b8006aa0 tx=0x7f95b8006a30 comp rx=0 tx=0).stop 2024-01-15T18:08:28.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.771+0000 7f95cda00640 1 -- 172.21.15.27:0/3522939190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95c81617f0 msgr2=0x7f95c815aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:28.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.771+0000 7f95cda00640 1 --2- 172.21.15.27:0/3522939190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95c81617f0 0x7f95c815aef0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f95c000c770 tx=0x7f95c000cc30 comp rx=0 tx=0).stop 2024-01-15T18:08:28.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.771+0000 7f95cda00640 1 -- 172.21.15.27:0/3522939190 shutdown_connections 2024-01-15T18:08:28.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.771+0000 7f95cda00640 1 --2- 172.21.15.27:0/3522939190 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f95a0066e00 0x7f95a00692c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:28.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.771+0000 7f95cda00640 1 --2- 172.21.15.27:0/3522939190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95c81617f0 0x7f95c815aef0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:28.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.771+0000 7f95cda00640 1 --2- 172.21.15.27:0/3522939190 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95c8072190 0x7f95c815c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:28.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.771+0000 7f95cda00640 1 --2- 172.21.15.27:0/3522939190 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f95c806dad0 0x7f95c806be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:28.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.771+0000 7f95cda00640 1 -- 172.21.15.27:0/3522939190 >> 172.21.15.27:0/3522939190 conn(0x7f95c8099e10 msgr2=0x7f95c80af5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:28.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.775+0000 7f95cda00640 1 -- 172.21.15.27:0/3522939190 shutdown_connections 2024-01-15T18:08:28.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:28.775+0000 7f95cda00640 1 -- 172.21.15.27:0/3522939190 wait complete. 2024-01-15T18:08:28.821 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:08:29.823 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone63891 2024-01-15T18:08:30.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.103+0000 7f0586e00640 1 Processor -- start 2024-01-15T18:08:30.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.103+0000 7f0586e00640 1 -- start start 2024-01-15T18:08:30.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.103+0000 7f0586e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f058015d0d0 0x7f05800b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:30.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.103+0000 7f0586e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f05800cfb20 0x7f05800b4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:30.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.103+0000 7f0586e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05800d0400 con 0x7f05800cff40 2024-01-15T18:08:30.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.103+0000 7f0586e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05800721c0 con 0x7f058015d0d0 2024-01-15T18:08:30.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.103+0000 7f0586e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0580071940 con 0x7f05800cfb20 2024-01-15T18:08:30.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.107+0000 7f057fe00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f058015d0d0 0x7f05800b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:30.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.107+0000 7f0584e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f05800cff40 0x7f0580161410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35562/0 (socket says 172.21.15.27:35562) 2024-01-15T18:08:30.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.107+0000 7f0584e00640 1 -- 172.21.15.27:0/1939647082 learned_addr learned my addr 172.21.15.27:0/1939647082 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:30.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.107+0000 7f057ea00640 1 -- 172.21.15.27:0/1939647082 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376571978 0 0) 0x7f05800d0400 con 0x7f05800cff40 2024-01-15T18:08:30.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.107+0000 7f057ea00640 1 -- 172.21.15.27:0/1939647082 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0560003530 con 0x7f05800cff40 2024-01-15T18:08:30.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.107+0000 7f057f400640 1 --2- 172.21.15.27:0/1939647082 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f05800cfb20 0x7f05800b4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:30.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.107+0000 7f057fe00640 1 -- 172.21.15.27:0/1939647082 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f05800cfb20 msgr2=0x7f05800b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:30.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.107+0000 7f057fe00640 1 --2- 172.21.15.27:0/1939647082 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f05800cfb20 0x7f05800b4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:30.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.107+0000 7f057fe00640 1 -- 172.21.15.27:0/1939647082 >> v1:172.21.15.27:6789/0 conn(0x7f05800cff40 legacy=0x7f0580161410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:30.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.107+0000 7f057fe00640 1 -- 172.21.15.27:0/1939647082 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0580161b20 con 0x7f058015d0d0 2024-01-15T18:08:30.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.107+0000 7f057f400640 1 --2- 172.21.15.27:0/1939647082 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f05800cfb20 0x7f05800b4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:08:30.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.111+0000 7f057fe00640 1 --2- 172.21.15.27:0/1939647082 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f058015d0d0 0x7f05800b2fa0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f05700097c0 tx=0x7f057001c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df212da679f6dafe server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:30.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.111+0000 7f057ea00640 1 -- 172.21.15.27:0/1939647082 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05700ac620 con 0x7f058015d0d0 2024-01-15T18:08:30.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.111+0000 7f057ea00640 1 -- 172.21.15.27:0/1939647082 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05700ac780 con 0x7f058015d0d0 2024-01-15T18:08:30.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.111+0000 7f057ea00640 1 -- 172.21.15.27:0/1939647082 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05700acb10 con 0x7f058015d0d0 2024-01-15T18:08:30.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.115+0000 7f0586e00640 1 -- 172.21.15.27:0/1939647082 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f058015d0d0 msgr2=0x7f05800b2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:30.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.115+0000 7f0586e00640 1 --2- 172.21.15.27:0/1939647082 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f058015d0d0 0x7f05800b2fa0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f05700097c0 tx=0x7f057001c3c0 comp rx=0 tx=0).stop 2024-01-15T18:08:30.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.115+0000 7f0586e00640 1 -- 172.21.15.27:0/1939647082 shutdown_connections 2024-01-15T18:08:30.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.115+0000 7f0586e00640 1 --2- 172.21.15.27:0/1939647082 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f05800cfb20 0x7f05800b4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:30.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.115+0000 7f0586e00640 1 --2- 172.21.15.27:0/1939647082 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f058015d0d0 0x7f05800b2fa0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:30.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.115+0000 7f0586e00640 1 -- 172.21.15.27:0/1939647082 >> 172.21.15.27:0/1939647082 conn(0x7f058006b600 msgr2=0x7f058006ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:30.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.115+0000 7f0586e00640 1 -- 172.21.15.27:0/1939647082 shutdown_connections 2024-01-15T18:08:30.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.115+0000 7f0586e00640 1 -- 172.21.15.27:0/1939647082 wait complete. 2024-01-15T18:08:30.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.119+0000 7f0586e00640 1 Processor -- start 2024-01-15T18:08:30.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.119+0000 7f0586e00640 1 -- start start 2024-01-15T18:08:30.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.123+0000 7f0586e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f05800cfb20 0x7f0580155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:30.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.123+0000 7f0586e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05800cff40 0x7f05800b2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:30.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.123+0000 7f0586e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f058015d0d0 0x7f05800b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:30.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.123+0000 7f0586e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05801556d0 con 0x7f05800cff40 2024-01-15T18:08:30.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.123+0000 7f0586e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05800b30d0 con 0x7f058015d0d0 2024-01-15T18:08:30.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.123+0000 7f0586e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05800b1390 con 0x7f05800cfb20 2024-01-15T18:08:30.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.127+0000 7f057f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05800cff40 0x7f05800b2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:30.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.127+0000 7f057f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05800cff40 0x7f05800b2b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56590/0 (socket says 172.21.15.27:56590) 2024-01-15T18:08:30.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.127+0000 7f057f400640 1 -- 172.21.15.27:0/3925719062 learned_addr learned my addr 172.21.15.27:0/3925719062 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:30.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.127+0000 7f057fe00640 1 --2- 172.21.15.27:0/3925719062 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f05800cfb20 0x7f0580155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:30.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.127+0000 7f057f400640 1 -- 172.21.15.27:0/3925719062 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f05800cfb20 msgr2=0x7f0580155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:30.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.127+0000 7f057f400640 1 --2- 172.21.15.27:0/3925719062 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f05800cfb20 0x7f0580155190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:30.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.127+0000 7f057f400640 1 -- 172.21.15.27:0/3925719062 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f058015d0d0 msgr2=0x7f05800b0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:30.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.127+0000 7f057f400640 1 --2- 172.21.15.27:0/3925719062 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f058015d0d0 0x7f05800b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:30.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.127+0000 7f057f400640 1 -- 172.21.15.27:0/3925719062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0570009420 con 0x7f05800cff40 2024-01-15T18:08:30.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.127+0000 7f057f400640 1 --2- 172.21.15.27:0/3925719062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05800cff40 0x7f05800b2b90 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f056800c7f0 tx=0x7f056800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:30.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.127+0000 7f0577e00640 1 -- 172.21.15.27:0/3925719062 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0568007820 con 0x7f05800cff40 2024-01-15T18:08:30.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.127+0000 7f0577e00640 1 -- 172.21.15.27:0/3925719062 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0568007980 con 0x7f05800cff40 2024-01-15T18:08:30.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.127+0000 7f0577e00640 1 -- 172.21.15.27:0/3925719062 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0568007d30 con 0x7f05800cff40 2024-01-15T18:08:30.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.127+0000 7f0586e00640 1 -- 172.21.15.27:0/3925719062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05800b1af0 con 0x7f05800cff40 2024-01-15T18:08:30.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.131+0000 7f0586e00640 1 -- 172.21.15.27:0/3925719062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05800b20b0 con 0x7f05800cff40 2024-01-15T18:08:30.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.131+0000 7f0577e00640 1 -- 172.21.15.27:0/3925719062 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f05680113f0 con 0x7f05800cff40 2024-01-15T18:08:30.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.135+0000 7f0586e00640 1 -- 172.21.15.27:0/3925719062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f058004e850 con 0x7f05800cff40 2024-01-15T18:08:30.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.139+0000 7f0577e00640 1 --2- 172.21.15.27:0/3925719062 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0558066e50 0x7f0558069310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:30.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.139+0000 7f0577e00640 1 -- 172.21.15.27:0/3925719062 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0568084fc0 con 0x7f05800cff40 2024-01-15T18:08:30.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.151+0000 7f0577e00640 1 -- 172.21.15.27:0/3925719062 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0568051b40 con 0x7f05800cff40 2024-01-15T18:08:30.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.163+0000 7f057fe00640 1 --2- 172.21.15.27:0/3925719062 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0558066e50 0x7f0558069310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:30.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.167+0000 7f057fe00640 1 --2- 172.21.15.27:0/3925719062 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0558066e50 0x7f0558069310 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f05700ae000 tx=0x7f0570008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:30.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.479+0000 7f0586e00640 1 -- 172.21.15.27:0/3925719062 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone63891", "target": ["mon-mgr", ""]}) v1 -- 0x7f05800b4dc0 con 0x7f0558066e50 2024-01-15T18:08:30.558 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:08:30.558 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:08:30.558 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:08:30.558 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:08:30.558 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:08:30.558 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:08:30.559 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:08:30.559 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:30.559 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:08:30.559 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 66.66666666666666, 2024-01-15T18:08:30.559 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T18:08:30.559 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "2/3", 2024-01-15T18:08:30.559 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:08:30.559 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:08:30.559 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:30.559 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:30.559 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:08:30.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.551+0000 7f0577e00640 1 -- 172.21.15.27:0/3925719062 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f05700ac8f0 con 0x7f0558066e50 2024-01-15T18:08:30.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.555+0000 7f0586e00640 1 -- 172.21.15.27:0/3925719062 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0558066e50 msgr2=0x7f0558069310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:30.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.555+0000 7f0586e00640 1 --2- 172.21.15.27:0/3925719062 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0558066e50 0x7f0558069310 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f05700ae000 tx=0x7f0570008040 comp rx=0 tx=0).stop 2024-01-15T18:08:30.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.555+0000 7f0586e00640 1 -- 172.21.15.27:0/3925719062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05800cff40 msgr2=0x7f05800b2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.555+0000 7f0586e00640 1 --2- 172.21.15.27:0/3925719062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05800cff40 0x7f05800b2b90 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f056800c7f0 tx=0x7f056800ccb0 comp rx=0 tx=0).stop 2024-01-15T18:08:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.555+0000 7f0586e00640 1 -- 172.21.15.27:0/3925719062 shutdown_connections 2024-01-15T18:08:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.555+0000 7f0586e00640 1 --2- 172.21.15.27:0/3925719062 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0558066e50 0x7f0558069310 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.555+0000 7f0586e00640 1 --2- 172.21.15.27:0/3925719062 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f058015d0d0 0x7f05800b0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.555+0000 7f0586e00640 1 --2- 172.21.15.27:0/3925719062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05800cff40 0x7f05800b2b90 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.555+0000 7f0586e00640 1 --2- 172.21.15.27:0/3925719062 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f05800cfb20 0x7f0580155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.555+0000 7f0586e00640 1 -- 172.21.15.27:0/3925719062 >> 172.21.15.27:0/3925719062 conn(0x7f058006b600 msgr2=0x7f05801604e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.555+0000 7f0586e00640 1 -- 172.21.15.27:0/3925719062 shutdown_connections 2024-01-15T18:08:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:30.555+0000 7f0586e00640 1 -- 172.21.15.27:0/3925719062 wait complete. 2024-01-15T18:08:30.619 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:08:31.620 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone63891 2024-01-15T18:08:31.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.875+0000 7f4d34600640 1 Processor -- start 2024-01-15T18:08:31.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.875+0000 7f4d34600640 1 -- start start 2024-01-15T18:08:31.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.875+0000 7f4d34600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f4d2c15b370 0x7f4d2c0d19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:31.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.875+0000 7f4d34600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4d2c09a120 0x7f4d2c0b0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:31.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.875+0000 7f4d34600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2c0cc520 con 0x7f4d2c0b04b0 2024-01-15T18:08:31.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.875+0000 7f4d34600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d2c15b750 con 0x7f4d2c09a120 2024-01-15T18:08:31.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.875+0000 7f4d34600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d2c0cc790 con 0x7f4d2c15b370 2024-01-15T18:08:31.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.875+0000 7f4d31200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4d2c0b04b0 0x7f4d2c099a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35578/0 (socket says 172.21.15.27:35578) 2024-01-15T18:08:31.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.875+0000 7f4d31200640 1 -- 172.21.15.27:0/2779489345 learned_addr learned my addr 172.21.15.27:0/2779489345 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:31.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.875+0000 7f4d32600640 1 --2- 172.21.15.27:0/2779489345 >> v2:172.21.15.27:3300/0 conn(0x7f4d2c09a120 0x7f4d2c0b0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:31.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.879+0000 7f4d2be00640 1 -- 172.21.15.27:0/2779489345 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1365666572 0 0) 0x7f4d2c0cc520 con 0x7f4d2c0b04b0 2024-01-15T18:08:31.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.879+0000 7f4d2be00640 1 -- 172.21.15.27:0/2779489345 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d0c003530 con 0x7f4d2c0b04b0 2024-01-15T18:08:31.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.879+0000 7f4d32600640 1 -- 172.21.15.27:0/2779489345 >> v2:172.21.15.93:3300/0 conn(0x7f4d2c15b370 msgr2=0x7f4d2c0d19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:31.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.879+0000 7f4d32600640 1 --2- 172.21.15.27:0/2779489345 >> v2:172.21.15.93:3300/0 conn(0x7f4d2c15b370 0x7f4d2c0d19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:31.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.879+0000 7f4d32600640 1 -- 172.21.15.27:0/2779489345 >> v1:172.21.15.27:6789/0 conn(0x7f4d2c0b04b0 legacy=0x7f4d2c099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:31.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.879+0000 7f4d32600640 1 -- 172.21.15.27:0/2779489345 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d2c0cca70 con 0x7f4d2c09a120 2024-01-15T18:08:31.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.887+0000 7f4d32600640 1 --2- 172.21.15.27:0/2779489345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d2c09a120 0x7f4d2c0b0b50 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c072200 tx=0x7f4d2401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bafceefd7b8fc34 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:31.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.887+0000 7f4d2be00640 1 -- 172.21.15.27:0/2779489345 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d240a58e0 con 0x7f4d2c09a120 2024-01-15T18:08:31.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.887+0000 7f4d2be00640 1 -- 172.21.15.27:0/2779489345 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d240a5a40 con 0x7f4d2c09a120 2024-01-15T18:08:31.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.887+0000 7f4d2be00640 1 -- 172.21.15.27:0/2779489345 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d240a5db0 con 0x7f4d2c09a120 2024-01-15T18:08:31.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.887+0000 7f4d34600640 1 -- 172.21.15.27:0/2779489345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d2c09a120 msgr2=0x7f4d2c0b0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:31.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.887+0000 7f4d34600640 1 --2- 172.21.15.27:0/2779489345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d2c09a120 0x7f4d2c0b0b50 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c072200 tx=0x7f4d2401c950 comp rx=0 tx=0).stop 2024-01-15T18:08:31.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.887+0000 7f4d34600640 1 -- 172.21.15.27:0/2779489345 shutdown_connections 2024-01-15T18:08:31.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.887+0000 7f4d34600640 1 --2- 172.21.15.27:0/2779489345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d2c09a120 0x7f4d2c0b0b50 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:31.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.887+0000 7f4d34600640 1 --2- 172.21.15.27:0/2779489345 >> v2:172.21.15.93:3300/0 conn(0x7f4d2c15b370 0x7f4d2c0d19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:31.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.887+0000 7f4d34600640 1 -- 172.21.15.27:0/2779489345 >> 172.21.15.27:0/2779489345 conn(0x7f4d2c0d0ed0 msgr2=0x7f4d2c0d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:31.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.887+0000 7f4d34600640 1 -- 172.21.15.27:0/2779489345 shutdown_connections 2024-01-15T18:08:31.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.887+0000 7f4d34600640 1 -- 172.21.15.27:0/2779489345 wait complete. 2024-01-15T18:08:31.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.891+0000 7f4d34600640 1 Processor -- start 2024-01-15T18:08:31.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.891+0000 7f4d34600640 1 -- start start 2024-01-15T18:08:31.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.895+0000 7f4d34600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d2c09a120 0x7f4d2c1f9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:31.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.895+0000 7f4d34600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4d2c0b04b0 0x7f4d2c0b55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:31.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.895+0000 7f4d34600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4d2c15b370 0x7f4d2c0af430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:31.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.895+0000 7f4d34600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2c1f9ca0 con 0x7f4d2c09a120 2024-01-15T18:08:31.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.895+0000 7f4d34600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2c15cd90 con 0x7f4d2c0b04b0 2024-01-15T18:08:31.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.895+0000 7f4d34600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2c15d070 con 0x7f4d2c15b370 2024-01-15T18:08:31.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.895+0000 7f4d31c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d2c09a120 0x7f4d2c1f9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:31.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.895+0000 7f4d31c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d2c09a120 0x7f4d2c1f9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56608/0 (socket says 172.21.15.27:56608) 2024-01-15T18:08:31.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.895+0000 7f4d31c00640 1 -- 172.21.15.27:0/2227654071 learned_addr learned my addr 172.21.15.27:0/2227654071 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:31.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.895+0000 7f4d31c00640 1 -- 172.21.15.27:0/2227654071 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4d2c15b370 msgr2=0x7f4d2c0af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:31.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.895+0000 7f4d31c00640 1 --2- 172.21.15.27:0/2227654071 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4d2c15b370 0x7f4d2c0af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:31.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.895+0000 7f4d31c00640 1 -- 172.21.15.27:0/2227654071 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4d2c0b04b0 msgr2=0x7f4d2c0b55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:31.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.895+0000 7f4d31c00640 1 --2- 172.21.15.27:0/2227654071 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4d2c0b04b0 0x7f4d2c0b55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:31.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.895+0000 7f4d31c00640 1 -- 172.21.15.27:0/2227654071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d24009a70 con 0x7f4d2c09a120 2024-01-15T18:08:31.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.895+0000 7f4d31c00640 1 --2- 172.21.15.27:0/2227654071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d2c09a120 0x7f4d2c1f9760 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c00b410 tx=0x7f4d1c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:31.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.899+0000 7f4d29600640 1 -- 172.21.15.27:0/2227654071 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c011830 con 0x7f4d2c09a120 2024-01-15T18:08:31.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.899+0000 7f4d29600640 1 -- 172.21.15.27:0/2227654071 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d1c011990 con 0x7f4d2c09a120 2024-01-15T18:08:31.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.899+0000 7f4d29600640 1 -- 172.21.15.27:0/2227654071 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c011d20 con 0x7f4d2c09a120 2024-01-15T18:08:31.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.899+0000 7f4d34600640 1 -- 172.21.15.27:0/2227654071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d2c15d350 con 0x7f4d2c09a120 2024-01-15T18:08:31.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.899+0000 7f4d34600640 1 -- 172.21.15.27:0/2227654071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d2c15d910 con 0x7f4d2c09a120 2024-01-15T18:08:31.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.903+0000 7f4d34600640 1 -- 172.21.15.27:0/2227654071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cfc005260 con 0x7f4d2c09a120 2024-01-15T18:08:31.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.903+0000 7f4d29600640 1 -- 172.21.15.27:0/2227654071 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f4d1c01a400 con 0x7f4d2c09a120 2024-01-15T18:08:31.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.903+0000 7f4d29600640 1 --2- 172.21.15.27:0/2227654071 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4d04066ed0 0x7f4d04069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:31.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.903+0000 7f4d29600640 1 -- 172.21.15.27:0/2227654071 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4d1c084d00 con 0x7f4d2c09a120 2024-01-15T18:08:31.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.911+0000 7f4d29600640 1 -- 172.21.15.27:0/2227654071 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4d1c051980 con 0x7f4d2c09a120 2024-01-15T18:08:31.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.919+0000 7f4d31200640 1 --2- 172.21.15.27:0/2227654071 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4d04066ed0 0x7f4d04069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:31.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:31.931+0000 7f4d31200640 1 --2- 172.21.15.27:0/2227654071 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4d04066ed0 0x7f4d04069390 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4d14007fd0 tx=0x7f4d14007940 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:32.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:32.271+0000 7f4d34600640 1 -- 172.21.15.27:0/2227654071 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone63891", "target": ["mon-mgr", ""]}) v1 -- 0x7f4cfc000bd0 con 0x7f4d04066ed0 2024-01-15T18:08:32.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:32.351+0000 7f4d29600640 1 -- 172.21.15.27:0/2227654071 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f4cfc000bd0 con 0x7f4d04066ed0 2024-01-15T18:08:32.357 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:08:32.358 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:08:32.358 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:08:32.358 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:08:32.358 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:08:32.358 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:08:32.358 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:08:32.358 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:32.358 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:08:32.358 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 66.66666666666666, 2024-01-15T18:08:32.358 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T18:08:32.358 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "2/3", 2024-01-15T18:08:32.358 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:08:32.358 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:08:32.359 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:32.359 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:32.359 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:08:32.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:32.355+0000 7f4d34600640 1 -- 172.21.15.27:0/2227654071 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4d04066ed0 msgr2=0x7f4d04069390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:32.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:32.355+0000 7f4d34600640 1 --2- 172.21.15.27:0/2227654071 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4d04066ed0 0x7f4d04069390 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4d14007fd0 tx=0x7f4d14007940 comp rx=0 tx=0).stop 2024-01-15T18:08:32.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:32.355+0000 7f4d34600640 1 -- 172.21.15.27:0/2227654071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d2c09a120 msgr2=0x7f4d2c1f9760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:32.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:32.355+0000 7f4d34600640 1 --2- 172.21.15.27:0/2227654071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d2c09a120 0x7f4d2c1f9760 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c00b410 tx=0x7f4d1c00b8d0 comp rx=0 tx=0).stop 2024-01-15T18:08:32.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:32.355+0000 7f4d34600640 1 -- 172.21.15.27:0/2227654071 shutdown_connections 2024-01-15T18:08:32.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:32.355+0000 7f4d34600640 1 --2- 172.21.15.27:0/2227654071 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4d04066ed0 0x7f4d04069390 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:32.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:32.355+0000 7f4d34600640 1 --2- 172.21.15.27:0/2227654071 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4d2c15b370 0x7f4d2c0af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:32.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:32.355+0000 7f4d34600640 1 --2- 172.21.15.27:0/2227654071 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4d2c0b04b0 0x7f4d2c0b55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:32.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:32.355+0000 7f4d34600640 1 --2- 172.21.15.27:0/2227654071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d2c09a120 0x7f4d2c1f9760 secure :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c00b410 tx=0x7f4d1c00b8d0 comp rx=0 tx=0).stop 2024-01-15T18:08:32.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:32.355+0000 7f4d34600640 1 -- 172.21.15.27:0/2227654071 >> 172.21.15.27:0/2227654071 conn(0x7f4d2c0d0ed0 msgr2=0x7f4d2c099a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:32.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:32.355+0000 7f4d34600640 1 -- 172.21.15.27:0/2227654071 shutdown_connections 2024-01-15T18:08:32.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:32.355+0000 7f4d34600640 1 -- 172.21.15.27:0/2227654071 wait complete. 2024-01-15T18:08:32.404 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:08:33.405 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone63891 2024-01-15T18:08:33.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.663+0000 7fd9b3200640 1 Processor -- start 2024-01-15T18:08:33.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.663+0000 7fd9b3200640 1 -- start start 2024-01-15T18:08:33.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.663+0000 7fd9b3200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fd9ac161840 0x7fd9ac162700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:33.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.663+0000 7fd9b3200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd9ac0ac5a0 0x7fd9ac06c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:33.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.663+0000 7fd9b3200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9ac0b2e40 con 0x7fd9ac0b32d0 2024-01-15T18:08:33.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.663+0000 7fd9b3200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9ac162c40 con 0x7fd9ac0ac5a0 2024-01-15T18:08:33.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.663+0000 7fd9b3200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9ac0b2230 con 0x7fd9ac161840 2024-01-15T18:08:33.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.667+0000 7fd9b1200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd9ac0ac5a0 0x7fd9ac06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:33.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.667+0000 7fd9ab400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd9ac0b32d0 0x7fd9ac0abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50638/0 (socket says 172.21.15.27:50638) 2024-01-15T18:08:33.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.667+0000 7fd9ab400640 1 -- 172.21.15.27:0/623396871 learned_addr learned my addr 172.21.15.27:0/623396871 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:33.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.667+0000 7fd9b1200640 1 -- 172.21.15.27:0/623396871 >> v2:172.21.15.93:3300/0 conn(0x7fd9ac161840 msgr2=0x7fd9ac162700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:33.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.667+0000 7fd9b1200640 1 --2- 172.21.15.27:0/623396871 >> v2:172.21.15.93:3300/0 conn(0x7fd9ac161840 0x7fd9ac162700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:33.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.667+0000 7fd9b1200640 1 -- 172.21.15.27:0/623396871 >> v1:172.21.15.27:6789/0 conn(0x7fd9ac0b32d0 legacy=0x7fd9ac0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:33.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.667+0000 7fd9aaa00640 1 -- 172.21.15.27:0/623396871 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466440140 0 0) 0x7fd9ac0b2e40 con 0x7fd9ac0b32d0 2024-01-15T18:08:33.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.667+0000 7fd9b1200640 1 -- 172.21.15.27:0/623396871 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ac0b2510 con 0x7fd9ac0ac5a0 2024-01-15T18:08:33.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.675+0000 7fd9b1200640 1 --2- 172.21.15.27:0/623396871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9ac0ac5a0 0x7fd9ac06c470 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac0b4080 tx=0x7fd9a401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50bab3b35fee25cb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:33.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.675+0000 7fd9aaa00640 1 -- 172.21.15.27:0/623396871 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a40a59a0 con 0x7fd9ac0ac5a0 2024-01-15T18:08:33.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.675+0000 7fd9aaa00640 1 -- 172.21.15.27:0/623396871 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a40a5b00 con 0x7fd9ac0ac5a0 2024-01-15T18:08:33.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.675+0000 7fd9aaa00640 1 -- 172.21.15.27:0/623396871 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a40a5e90 con 0x7fd9ac0ac5a0 2024-01-15T18:08:33.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.679+0000 7fd9b3200640 1 -- 172.21.15.27:0/623396871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9ac0ac5a0 msgr2=0x7fd9ac06c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:33.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.679+0000 7fd9b3200640 1 --2- 172.21.15.27:0/623396871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9ac0ac5a0 0x7fd9ac06c470 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac0b4080 tx=0x7fd9a401c8e0 comp rx=0 tx=0).stop 2024-01-15T18:08:33.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.679+0000 7fd9b3200640 1 -- 172.21.15.27:0/623396871 shutdown_connections 2024-01-15T18:08:33.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.679+0000 7fd9b3200640 1 --2- 172.21.15.27:0/623396871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9ac0ac5a0 0x7fd9ac06c470 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:33.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.679+0000 7fd9b3200640 1 --2- 172.21.15.27:0/623396871 >> v2:172.21.15.93:3300/0 conn(0x7fd9ac161840 0x7fd9ac162700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:33.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.679+0000 7fd9b3200640 1 -- 172.21.15.27:0/623396871 >> 172.21.15.27:0/623396871 conn(0x7fd9ac0b0bb0 msgr2=0x7fd9ac0b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:33.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.683+0000 7fd9b3200640 1 -- 172.21.15.27:0/623396871 shutdown_connections 2024-01-15T18:08:33.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.683+0000 7fd9b3200640 1 -- 172.21.15.27:0/623396871 wait complete. 2024-01-15T18:08:33.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.683+0000 7fd9b3200640 1 Processor -- start 2024-01-15T18:08:33.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.687+0000 7fd9b3200640 1 -- start start 2024-01-15T18:08:33.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.691+0000 7fd9b3200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9ac0b32d0 0x7fd9ac150e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:33.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.691+0000 7fd9b3200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd9ac161840 0x7fd9ac0cd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:33.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.691+0000 7fd9b3200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd9ac1517d0 0x7fd9ac163dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:33.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.691+0000 7fd9b3200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac1529f0 con 0x7fd9ac0b32d0 2024-01-15T18:08:33.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.691+0000 7fd9b3200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac151d70 con 0x7fd9ac1517d0 2024-01-15T18:08:33.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.691+0000 7fd9b3200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac152050 con 0x7fd9ac161840 2024-01-15T18:08:33.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.691+0000 7fd9abe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9ac0b32d0 0x7fd9ac150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:33.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.691+0000 7fd9abe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9ac0b32d0 0x7fd9ac150e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39866/0 (socket says 172.21.15.27:39866) 2024-01-15T18:08:33.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.691+0000 7fd9abe00640 1 -- 172.21.15.27:0/3890650349 learned_addr learned my addr 172.21.15.27:0/3890650349 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:33.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.695+0000 7fd9abe00640 1 -- 172.21.15.27:0/3890650349 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd9ac161840 msgr2=0x7fd9ac0cd950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-15T18:08:33.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.695+0000 7fd9abe00640 1 --2- 172.21.15.27:0/3890650349 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd9ac161840 0x7fd9ac0cd950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:33.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.695+0000 7fd9abe00640 1 -- 172.21.15.27:0/3890650349 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd9ac1517d0 msgr2=0x7fd9ac163dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-15T18:08:33.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.695+0000 7fd9abe00640 1 --2- 172.21.15.27:0/3890650349 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd9ac1517d0 0x7fd9ac163dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:33.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.695+0000 7fd9abe00640 1 -- 172.21.15.27:0/3890650349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a4009a70 con 0x7fd9ac0b32d0 2024-01-15T18:08:33.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.695+0000 7fd9abe00640 1 --2- 172.21.15.27:0/3890650349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9ac0b32d0 0x7fd9ac150e20 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fd99c00b410 tx=0x7fd99c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:33.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.695+0000 7fd9a3e00640 1 -- 172.21.15.27:0/3890650349 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99c011830 con 0x7fd9ac0b32d0 2024-01-15T18:08:33.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.695+0000 7fd9b3200640 1 -- 172.21.15.27:0/3890650349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9ac152330 con 0x7fd9ac0b32d0 2024-01-15T18:08:33.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.695+0000 7fd9b3200640 1 -- 172.21.15.27:0/3890650349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9ac151360 con 0x7fd9ac0b32d0 2024-01-15T18:08:33.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.695+0000 7fd9a3e00640 1 -- 172.21.15.27:0/3890650349 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd99c011990 con 0x7fd9ac0b32d0 2024-01-15T18:08:33.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.695+0000 7fd9a3e00640 1 -- 172.21.15.27:0/3890650349 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99c011830 con 0x7fd9ac0b32d0 2024-01-15T18:08:33.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.699+0000 7fd9b3200640 1 -- 172.21.15.27:0/3890650349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd97c005260 con 0x7fd9ac0b32d0 2024-01-15T18:08:33.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.703+0000 7fd9a3e00640 1 -- 172.21.15.27:0/3890650349 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fd99c011b00 con 0x7fd9ac0b32d0 2024-01-15T18:08:33.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.703+0000 7fd9a3e00640 1 --2- 172.21.15.27:0/3890650349 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd984066e60 0x7fd984069320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:33.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.703+0000 7fd9a3e00640 1 -- 172.21.15.27:0/3890650349 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd99c084aa0 con 0x7fd9ac0b32d0 2024-01-15T18:08:33.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.707+0000 7fd9ab400640 1 --2- 172.21.15.27:0/3890650349 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd984066e60 0x7fd984069320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:33.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.707+0000 7fd9ab400640 1 --2- 172.21.15.27:0/3890650349 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd984066e60 0x7fd984069320 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd994007fd0 tx=0x7fd9940078c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:33.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:33.727+0000 7fd9a3e00640 1 -- 172.21.15.27:0/3890650349 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd99c0515f0 con 0x7fd9ac0b32d0 2024-01-15T18:08:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:34.051+0000 7fd9b3200640 1 -- 172.21.15.27:0/3890650349 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone63891", "target": ["mon-mgr", ""]}) v1 -- 0x7fd97c000bd0 con 0x7fd984066e60 2024-01-15T18:08:35.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.158+0000 7fd9a3e00640 1 -- 172.21.15.27:0/3890650349 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fd97c000bd0 con 0x7fd984066e60 2024-01-15T18:08:35.166 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:08:35.166 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:08:35.166 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "complete" 2024-01-15T18:08:35.167 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:35.167 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:08:35.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.162+0000 7fd9b3200640 1 -- 172.21.15.27:0/3890650349 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd984066e60 msgr2=0x7fd984069320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:35.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.162+0000 7fd9b3200640 1 --2- 172.21.15.27:0/3890650349 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd984066e60 0x7fd984069320 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd994007fd0 tx=0x7fd9940078c0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.162+0000 7fd9b3200640 1 -- 172.21.15.27:0/3890650349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9ac0b32d0 msgr2=0x7fd9ac150e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:35.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.162+0000 7fd9b3200640 1 --2- 172.21.15.27:0/3890650349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9ac0b32d0 0x7fd9ac150e20 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fd99c00b410 tx=0x7fd99c00b8d0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.162+0000 7fd9b3200640 1 -- 172.21.15.27:0/3890650349 shutdown_connections 2024-01-15T18:08:35.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.162+0000 7fd9b3200640 1 --2- 172.21.15.27:0/3890650349 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd984066e60 0x7fd984069320 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.162+0000 7fd9b3200640 1 --2- 172.21.15.27:0/3890650349 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd9ac1517d0 0x7fd9ac163dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.162+0000 7fd9b3200640 1 --2- 172.21.15.27:0/3890650349 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd9ac161840 0x7fd9ac0cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.162+0000 7fd9b3200640 1 --2- 172.21.15.27:0/3890650349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9ac0b32d0 0x7fd9ac150e20 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.162+0000 7fd9b3200640 1 -- 172.21.15.27:0/3890650349 >> 172.21.15.27:0/3890650349 conn(0x7fd9ac0b0bb0 msgr2=0x7fd9ac0aeb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:35.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.162+0000 7fd9b3200640 1 -- 172.21.15.27:0/3890650349 shutdown_connections 2024-01-15T18:08:35.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.162+0000 7fd9b3200640 1 -- 172.21.15.27:0/3890650349 wait complete. 2024-01-15T18:08:35.192 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:08:35.192 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone63892 2024-01-15T18:08:35.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.338+0000 7f2173000640 1 Processor -- start 2024-01-15T18:08:35.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.338+0000 7f2173000640 1 -- start start 2024-01-15T18:08:35.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.338+0000 7f2173000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f216c158900 0x7f216c15f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:35.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.338+0000 7f2173000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f216c0cf2e0 0x7f216c0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:35.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.338+0000 7f2173000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f216c0b1cb0 con 0x7f216c15bf00 2024-01-15T18:08:35.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.338+0000 7f2173000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f216c0b1f00 con 0x7f216c0cf2e0 2024-01-15T18:08:35.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.338+0000 7f2173000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f216c0b21e0 con 0x7f216c158900 2024-01-15T18:08:35.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.338+0000 7f216be00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f216c158900 0x7f216c15f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:35.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.338+0000 7f216be00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f216c158900 0x7f216c15f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:38788/0 (socket says 172.21.15.27:38788) 2024-01-15T18:08:35.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.338+0000 7f216be00640 1 -- 172.21.15.27:0/332451744 learned_addr learned my addr 172.21.15.27:0/332451744 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:35.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.338+0000 7f2171000640 1 --2- 172.21.15.27:0/332451744 >> v2:172.21.15.93:3300/0 conn(0x7f216c0cf2e0 0x7f216c0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:35.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.338+0000 7f216be00640 1 -- 172.21.15.27:0/332451744 >> v2:172.21.15.93:3300/0 conn(0x7f216c0cf2e0 msgr2=0x7f216c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:35.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.338+0000 7f216be00640 1 --2- 172.21.15.27:0/332451744 >> v2:172.21.15.93:3300/0 conn(0x7f216c0cf2e0 0x7f216c0701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.338+0000 7f216be00640 1 -- 172.21.15.27:0/332451744 >> v1:172.21.15.93:6789/0 conn(0x7f216c15bf00 legacy=0x7f216c0cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:35.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.338+0000 7f216be00640 1 -- 172.21.15.27:0/332451744 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216c15ea70 con 0x7f216c158900 2024-01-15T18:08:35.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.342+0000 7f216be00640 1 --2- 172.21.15.27:0/332451744 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f216c158900 0x7f216c15f550 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f215c009870 tx=0x7f215c01c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1925237545e2c29 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:35.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.342+0000 7f216aa00640 1 -- 172.21.15.27:0/332451744 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215c0a7070 con 0x7f216c158900 2024-01-15T18:08:35.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.342+0000 7f216aa00640 1 -- 172.21.15.27:0/332451744 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f215c0ab430 con 0x7f216c158900 2024-01-15T18:08:35.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.342+0000 7f216aa00640 1 -- 172.21.15.27:0/332451744 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c070f30 con 0x7f216c158900 2024-01-15T18:08:35.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.342+0000 7f2173000640 1 -- 172.21.15.27:0/332451744 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f216c158900 msgr2=0x7f216c15f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:35.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.342+0000 7f2173000640 1 --2- 172.21.15.27:0/332451744 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f216c158900 0x7f216c15f550 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f215c009870 tx=0x7f215c01c540 comp rx=0 tx=0).stop 2024-01-15T18:08:35.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.342+0000 7f2173000640 1 -- 172.21.15.27:0/332451744 shutdown_connections 2024-01-15T18:08:35.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.342+0000 7f2173000640 1 --2- 172.21.15.27:0/332451744 >> v2:172.21.15.93:3300/0 conn(0x7f216c0cf2e0 0x7f216c0701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.342+0000 7f2173000640 1 --2- 172.21.15.27:0/332451744 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f216c158900 0x7f216c15f550 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.342+0000 7f2173000640 1 -- 172.21.15.27:0/332451744 >> 172.21.15.27:0/332451744 conn(0x7f216c0cb700 msgr2=0x7f216c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:35.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.346+0000 7f2173000640 1 -- 172.21.15.27:0/332451744 shutdown_connections 2024-01-15T18:08:35.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.346+0000 7f2173000640 1 -- 172.21.15.27:0/332451744 wait complete. 2024-01-15T18:08:35.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.346+0000 7f2173000640 1 Processor -- start 2024-01-15T18:08:35.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.346+0000 7f2173000640 1 -- start start 2024-01-15T18:08:35.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.346+0000 7f2173000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f216c0cf2e0 0x7f216c0b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:35.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.346+0000 7f2173000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f216c158900 0x7f216c157d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:35.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.346+0000 7f2173000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f216c15bf00 0x7f216c13e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:35.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.346+0000 7f2173000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216c158240 con 0x7f216c158900 2024-01-15T18:08:35.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.346+0000 7f2173000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216c0af130 con 0x7f216c0cf2e0 2024-01-15T18:08:35.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.346+0000 7f2173000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f216c0af410 con 0x7f216c15bf00 2024-01-15T18:08:35.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.346+0000 7f216b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f216c158900 0x7f216c157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:35.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.346+0000 7f216b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f216c158900 0x7f216c157d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39870/0 (socket says 172.21.15.27:39870) 2024-01-15T18:08:35.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.346+0000 7f216b400640 1 -- 172.21.15.27:0/3631331440 learned_addr learned my addr 172.21.15.27:0/3631331440 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:35.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.346+0000 7f216be00640 1 --2- 172.21.15.27:0/3631331440 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f216c0cf2e0 0x7f216c0b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:35.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.346+0000 7f2171000640 1 --2- 172.21.15.27:0/3631331440 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f216c15bf00 0x7f216c13e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:35.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.346+0000 7f216b400640 1 -- 172.21.15.27:0/3631331440 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f216c15bf00 msgr2=0x7f216c13e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:35.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.346+0000 7f216b400640 1 --2- 172.21.15.27:0/3631331440 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f216c15bf00 0x7f216c13e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.346+0000 7f216b400640 1 -- 172.21.15.27:0/3631331440 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f216c0cf2e0 msgr2=0x7f216c0b01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:35.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.346+0000 7f216b400640 1 --2- 172.21.15.27:0/3631331440 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f216c0cf2e0 0x7f216c0b01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.346+0000 7f216b400640 1 -- 172.21.15.27:0/3631331440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f215c0094d0 con 0x7f216c158900 2024-01-15T18:08:35.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.350+0000 7f216b400640 1 --2- 172.21.15.27:0/3631331440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f216c158900 0x7f216c157d00 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f215400c500 tx=0x7f215400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:35.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.350+0000 7f2163e00640 1 -- 172.21.15.27:0/3631331440 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2154005220 con 0x7f216c158900 2024-01-15T18:08:35.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.350+0000 7f2163e00640 1 -- 172.21.15.27:0/3631331440 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2154005380 con 0x7f216c158900 2024-01-15T18:08:35.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.350+0000 7f2173000640 1 -- 172.21.15.27:0/3631331440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f216c0af6f0 con 0x7f216c158900 2024-01-15T18:08:35.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.350+0000 7f2163e00640 1 -- 172.21.15.27:0/3631331440 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2154007270 con 0x7f216c158900 2024-01-15T18:08:35.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.350+0000 7f2173000640 1 -- 172.21.15.27:0/3631331440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f216c0afcb0 con 0x7f216c158900 2024-01-15T18:08:35.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.350+0000 7f2173000640 1 -- 172.21.15.27:0/3631331440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f216c0afed0 con 0x7f216c158900 2024-01-15T18:08:35.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.350+0000 7f2163e00640 1 -- 172.21.15.27:0/3631331440 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f21540073d0 con 0x7f216c158900 2024-01-15T18:08:35.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.350+0000 7f2163e00640 1 --2- 172.21.15.27:0/3631331440 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2144066e00 0x7f21440692c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:35.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.350+0000 7f2163e00640 1 -- 172.21.15.27:0/3631331440 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2154084580 con 0x7f216c158900 2024-01-15T18:08:35.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.354+0000 7f216be00640 1 --2- 172.21.15.27:0/3631331440 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2144066e00 0x7f21440692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:35.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.354+0000 7f216be00640 1 --2- 172.21.15.27:0/3631331440 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2144066e00 0x7f21440692c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f215c0af000 tx=0x7f215c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:35.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.354+0000 7f2163e00640 1 -- 172.21.15.27:0/3631331440 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2154051120 con 0x7f216c158900 2024-01-15T18:08:35.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.542+0000 7f2173000640 1 -- 172.21.15.27:0/3631331440 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone63892", "target": ["mon-mgr", ""]}) v1 -- 0x7f216c067560 con 0x7f2144066e00 2024-01-15T18:08:35.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.554+0000 7f2163e00640 1 -- 172.21.15.27:0/3631331440 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f216c067560 con 0x7f2144066e00 2024-01-15T18:08:35.561 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:08:35.561 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:08:35.561 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "complete" 2024-01-15T18:08:35.561 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:35.561 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:08:35.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.558+0000 7f2173000640 1 -- 172.21.15.27:0/3631331440 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2144066e00 msgr2=0x7f21440692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:35.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.558+0000 7f2173000640 1 --2- 172.21.15.27:0/3631331440 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2144066e00 0x7f21440692c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f215c0af000 tx=0x7f215c008040 comp rx=0 tx=0).stop 2024-01-15T18:08:35.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.558+0000 7f2173000640 1 -- 172.21.15.27:0/3631331440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f216c158900 msgr2=0x7f216c157d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:35.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.558+0000 7f2173000640 1 --2- 172.21.15.27:0/3631331440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f216c158900 0x7f216c157d00 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f215400c500 tx=0x7f215400c9c0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.558+0000 7f2173000640 1 -- 172.21.15.27:0/3631331440 shutdown_connections 2024-01-15T18:08:35.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.558+0000 7f2173000640 1 --2- 172.21.15.27:0/3631331440 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2144066e00 0x7f21440692c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.558+0000 7f2173000640 1 --2- 172.21.15.27:0/3631331440 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f216c15bf00 0x7f216c13e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.558+0000 7f2173000640 1 --2- 172.21.15.27:0/3631331440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f216c158900 0x7f216c157d00 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.558+0000 7f2173000640 1 --2- 172.21.15.27:0/3631331440 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f216c0cf2e0 0x7f216c0b01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.558+0000 7f2173000640 1 -- 172.21.15.27:0/3631331440 >> 172.21.15.27:0/3631331440 conn(0x7f216c0cb700 msgr2=0x7f216c0b0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:35.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.558+0000 7f2173000640 1 -- 172.21.15.27:0/3631331440 shutdown_connections 2024-01-15T18:08:35.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.558+0000 7f2173000640 1 -- 172.21.15.27:0/3631331440 wait complete. 2024-01-15T18:08:35.584 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:08:35.584 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone63893 2024-01-15T18:08:35.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.766+0000 7ff286400640 1 Processor -- start 2024-01-15T18:08:35.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.766+0000 7ff286400640 1 -- start start 2024-01-15T18:08:35.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.770+0000 7ff286400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff28015b370 0x7ff2800d19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:35.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.770+0000 7ff286400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff28009a120 0x7ff2800b0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:35.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.770+0000 7ff27ea00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff2800b04b0 0x7ff280099a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50648/0 (socket says 172.21.15.27:50648) 2024-01-15T18:08:35.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.770+0000 7ff27ea00640 1 -- 172.21.15.27:0/3047796489 learned_addr learned my addr 172.21.15.27:0/3047796489 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:35.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.770+0000 7ff27f400640 1 --2- 172.21.15.27:0/3047796489 >> v2:172.21.15.27:3300/0 conn(0x7ff28015b370 0x7ff2800d19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:35.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.770+0000 7ff286400640 1 -- 172.21.15.27:0/3047796489 --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2800cc520 con 0x7ff2800b04b0 2024-01-15T18:08:35.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.770+0000 7ff286400640 1 -- 172.21.15.27:0/3047796489 --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff28015b750 con 0x7ff28015b370 2024-01-15T18:08:35.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.770+0000 7ff286400640 1 -- 172.21.15.27:0/3047796489 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2800cc790 con 0x7ff28009a120 2024-01-15T18:08:35.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.770+0000 7ff27f400640 1 -- 172.21.15.27:0/3047796489 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff28009a120 msgr2=0x7ff2800b0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:35.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.770+0000 7ff27f400640 1 --2- 172.21.15.27:0/3047796489 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff28009a120 0x7ff2800b0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.774+0000 7ff27f400640 1 -- 172.21.15.27:0/3047796489 >> v1:172.21.15.27:6789/0 conn(0x7ff2800b04b0 legacy=0x7ff280099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:35.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.774+0000 7ff27f400640 1 -- 172.21.15.27:0/3047796489 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2800cca70 con 0x7ff28015b370 2024-01-15T18:08:35.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.778+0000 7ff27f400640 1 --2- 172.21.15.27:0/3047796489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff28015b370 0x7ff2800d19f0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff2700097c0 tx=0x7ff27001c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b490e1cc556fb105 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:35.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.778+0000 7ff27e000640 1 -- 172.21.15.27:0/3047796489 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2700ab600 con 0x7ff28015b370 2024-01-15T18:08:35.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.778+0000 7ff27e000640 1 -- 172.21.15.27:0/3047796489 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2700a7050 con 0x7ff28015b370 2024-01-15T18:08:35.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.778+0000 7ff27e000640 1 -- 172.21.15.27:0/3047796489 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2700ab980 con 0x7ff28015b370 2024-01-15T18:08:35.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.778+0000 7ff286400640 1 -- 172.21.15.27:0/3047796489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff28015b370 msgr2=0x7ff2800d19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:35.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.778+0000 7ff286400640 1 --2- 172.21.15.27:0/3047796489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff28015b370 0x7ff2800d19f0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff2700097c0 tx=0x7ff27001c330 comp rx=0 tx=0).stop 2024-01-15T18:08:35.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.778+0000 7ff286400640 1 -- 172.21.15.27:0/3047796489 shutdown_connections 2024-01-15T18:08:35.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.778+0000 7ff286400640 1 --2- 172.21.15.27:0/3047796489 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff28009a120 0x7ff2800b0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.778+0000 7ff286400640 1 --2- 172.21.15.27:0/3047796489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff28015b370 0x7ff2800d19f0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.778+0000 7ff286400640 1 -- 172.21.15.27:0/3047796489 >> 172.21.15.27:0/3047796489 conn(0x7ff2800d0ed0 msgr2=0x7ff2800d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:35.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.778+0000 7ff286400640 1 -- 172.21.15.27:0/3047796489 shutdown_connections 2024-01-15T18:08:35.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.778+0000 7ff286400640 1 -- 172.21.15.27:0/3047796489 wait complete. 2024-01-15T18:08:35.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.782+0000 7ff286400640 1 Processor -- start 2024-01-15T18:08:35.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.782+0000 7ff286400640 1 -- start start 2024-01-15T18:08:35.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.782+0000 7ff286400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff28009a120 0x7ff2800ca220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:35.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.782+0000 7ff286400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2800b04b0 0x7ff28014bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:35.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.782+0000 7ff286400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff28015b370 0x7ff280157d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:35.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.782+0000 7ff286400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2800cab20 con 0x7ff2800b04b0 2024-01-15T18:08:35.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.782+0000 7ff286400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2800cae00 con 0x7ff28015b370 2024-01-15T18:08:35.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.782+0000 7ff286400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff280157260 con 0x7ff28009a120 2024-01-15T18:08:35.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.782+0000 7ff27ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2800b04b0 0x7ff28014bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:35.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.782+0000 7ff27ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2800b04b0 0x7ff28014bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39902/0 (socket says 172.21.15.27:39902) 2024-01-15T18:08:35.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.782+0000 7ff27ea00640 1 -- 172.21.15.27:0/1549744921 learned_addr learned my addr 172.21.15.27:0/1549744921 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:35.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.782+0000 7ff27f400640 1 --2- 172.21.15.27:0/1549744921 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff28009a120 0x7ff2800ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:35.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.782+0000 7ff27fe00640 1 --2- 172.21.15.27:0/1549744921 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff28015b370 0x7ff280157d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:35.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.782+0000 7ff27ea00640 1 -- 172.21.15.27:0/1549744921 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff28009a120 msgr2=0x7ff2800ca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:35.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.782+0000 7ff27ea00640 1 --2- 172.21.15.27:0/1549744921 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff28009a120 0x7ff2800ca220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.782+0000 7ff27ea00640 1 -- 172.21.15.27:0/1549744921 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff28015b370 msgr2=0x7ff280157d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:35.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.782+0000 7ff27ea00640 1 --2- 172.21.15.27:0/1549744921 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff28015b370 0x7ff280157d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.782+0000 7ff27ea00640 1 -- 172.21.15.27:0/1549744921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff270009420 con 0x7ff2800b04b0 2024-01-15T18:08:35.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.786+0000 7ff27ea00640 1 --2- 172.21.15.27:0/1549744921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2800b04b0 0x7ff28014bd60 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7ff268009ef0 tx=0x7ff26800c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:35.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.786+0000 7ff277400640 1 -- 172.21.15.27:0/1549744921 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff268007af0 con 0x7ff2800b04b0 2024-01-15T18:08:35.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.786+0000 7ff277400640 1 -- 172.21.15.27:0/1549744921 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff268007c50 con 0x7ff2800b04b0 2024-01-15T18:08:35.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.786+0000 7ff286400640 1 -- 172.21.15.27:0/1549744921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff28009a6f0 con 0x7ff2800b04b0 2024-01-15T18:08:35.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.786+0000 7ff277400640 1 -- 172.21.15.27:0/1549744921 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26800ce10 con 0x7ff2800b04b0 2024-01-15T18:08:35.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.786+0000 7ff286400640 1 -- 172.21.15.27:0/1549744921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff280157820 con 0x7ff2800b04b0 2024-01-15T18:08:35.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.786+0000 7ff286400640 1 -- 172.21.15.27:0/1549744921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff280157a40 con 0x7ff2800b04b0 2024-01-15T18:08:35.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.790+0000 7ff277400640 1 -- 172.21.15.27:0/1549744921 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) v1 ==== 86619+0+0 (secure 0 0 0) 0x7ff268009070 con 0x7ff2800b04b0 2024-01-15T18:08:35.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.790+0000 7ff277400640 1 --2- 172.21.15.27:0/1549744921 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff258066e00 0x7ff2580692c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:35.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.794+0000 7ff27f400640 1 --2- 172.21.15.27:0/1549744921 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff258066e00 0x7ff2580692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:35.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.798+0000 7ff277400640 1 -- 172.21.15.27:0/1549744921 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff2680571a0 con 0x7ff2800b04b0 2024-01-15T18:08:35.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.802+0000 7ff27f400640 1 --2- 172.21.15.27:0/1549744921 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff258066e00 0x7ff2580692c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff2700ad000 tx=0x7ff270008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:35.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:35.810+0000 7ff277400640 1 -- 172.21.15.27:0/1549744921 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff2680bb280 con 0x7ff2800b04b0 2024-01-15T18:08:36.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:36.118+0000 7ff286400640 1 -- 172.21.15.27:0/1549744921 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone63893", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2801573d0 con 0x7ff258066e00 2024-01-15T18:08:36.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:36.158+0000 7ff277400640 1 -- 172.21.15.27:0/1549744921 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7ff2801573d0 con 0x7ff258066e00 2024-01-15T18:08:36.163 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:08:36.163 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:08:36.163 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:08:36.163 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:08:36.163 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:08:36.163 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:08:36.163 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:08:36.163 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:36.163 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:08:36.163 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 66.66666666666666, 2024-01-15T18:08:36.164 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T18:08:36.164 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "2/3", 2024-01-15T18:08:36.164 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:08:36.164 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:08:36.164 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:36.164 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:36.164 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:08:36.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:36.162+0000 7ff286400640 1 -- 172.21.15.27:0/1549744921 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff258066e00 msgr2=0x7ff2580692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:36.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:36.162+0000 7ff286400640 1 --2- 172.21.15.27:0/1549744921 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff258066e00 0x7ff2580692c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff2700ad000 tx=0x7ff270008040 comp rx=0 tx=0).stop 2024-01-15T18:08:36.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:36.162+0000 7ff286400640 1 -- 172.21.15.27:0/1549744921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2800b04b0 msgr2=0x7ff28014bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:36.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:36.162+0000 7ff286400640 1 --2- 172.21.15.27:0/1549744921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2800b04b0 0x7ff28014bd60 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7ff268009ef0 tx=0x7ff26800c430 comp rx=0 tx=0).stop 2024-01-15T18:08:36.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:36.162+0000 7ff286400640 1 -- 172.21.15.27:0/1549744921 shutdown_connections 2024-01-15T18:08:36.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:36.162+0000 7ff286400640 1 --2- 172.21.15.27:0/1549744921 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff258066e00 0x7ff2580692c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:36.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:36.162+0000 7ff286400640 1 --2- 172.21.15.27:0/1549744921 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff28015b370 0x7ff280157d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:36.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:36.162+0000 7ff286400640 1 --2- 172.21.15.27:0/1549744921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2800b04b0 0x7ff28014bd60 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:36.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:36.162+0000 7ff286400640 1 --2- 172.21.15.27:0/1549744921 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff28009a120 0x7ff2800ca220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:36.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:36.162+0000 7ff286400640 1 -- 172.21.15.27:0/1549744921 >> 172.21.15.27:0/1549744921 conn(0x7ff2800d0ed0 msgr2=0x7ff2800ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:36.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:36.162+0000 7ff286400640 1 -- 172.21.15.27:0/1549744921 shutdown_connections 2024-01-15T18:08:36.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:36.162+0000 7ff286400640 1 -- 172.21.15.27:0/1549744921 wait complete. 2024-01-15T18:08:36.201 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:08:37.202 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone63893 2024-01-15T18:08:37.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.446+0000 7fd5cea00640 1 Processor -- start 2024-01-15T18:08:37.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.446+0000 7fd5cea00640 1 -- start start 2024-01-15T18:08:37.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.446+0000 7fd5cea00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fd5c80cda40 0x7fd5c806e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:37.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.446+0000 7fd5cea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd5c80cfb10 0x7fd5c80cc930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:37.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.446+0000 7fd5cea00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5c80d1df0 con 0x7fd5c80b0270 2024-01-15T18:08:37.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.446+0000 7fd5cea00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5c80b00a0 con 0x7fd5c80cfb10 2024-01-15T18:08:37.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.446+0000 7fd5cea00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5c815b600 con 0x7fd5c80cda40 2024-01-15T18:08:37.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.450+0000 7fd5ccc00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd5c80b0270 0x7fd5c806e8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50662/0 (socket says 172.21.15.27:50662) 2024-01-15T18:08:37.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.450+0000 7fd5ccc00640 1 -- 172.21.15.27:0/122024565 learned_addr learned my addr 172.21.15.27:0/122024565 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:37.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.450+0000 7fd5cd600640 1 --2- 172.21.15.27:0/122024565 >> v2:172.21.15.93:3300/0 conn(0x7fd5c80cda40 0x7fd5c806e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:37.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.450+0000 7fd5ce000640 1 --2- 172.21.15.27:0/122024565 >> v2:172.21.15.27:3300/0 conn(0x7fd5c80cfb10 0x7fd5c80cc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:37.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.450+0000 7fd5cd600640 1 -- 172.21.15.27:0/122024565 >> v2:172.21.15.27:3300/0 conn(0x7fd5c80cfb10 msgr2=0x7fd5c80cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:37.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.450+0000 7fd5cd600640 1 --2- 172.21.15.27:0/122024565 >> v2:172.21.15.27:3300/0 conn(0x7fd5c80cfb10 0x7fd5c80cc930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:37.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.450+0000 7fd5cd600640 1 -- 172.21.15.27:0/122024565 >> v1:172.21.15.27:6789/0 conn(0x7fd5c80b0270 legacy=0x7fd5c806e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:37.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.450+0000 7fd5cd600640 1 -- 172.21.15.27:0/122024565 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c80cec50 con 0x7fd5c80cda40 2024-01-15T18:08:37.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.470+0000 7fd5cd600640 1 --2- 172.21.15.27:0/122024565 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5c80cda40 0x7fd5c806e390 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd5c80ce510 tx=0x7fd5c806f7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1708afe69a862568 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:37.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.474+0000 7fd5c7e00640 1 -- 172.21.15.27:0/122024565 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b80ab600 con 0x7fd5c80cda40 2024-01-15T18:08:37.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.474+0000 7fd5c7e00640 1 -- 172.21.15.27:0/122024565 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b80a7050 con 0x7fd5c80cda40 2024-01-15T18:08:37.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.474+0000 7fd5c7e00640 1 -- 172.21.15.27:0/122024565 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b80ab980 con 0x7fd5c80cda40 2024-01-15T18:08:37.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.474+0000 7fd5cea00640 1 -- 172.21.15.27:0/122024565 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5c80cda40 msgr2=0x7fd5c806e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:37.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.474+0000 7fd5cea00640 1 --2- 172.21.15.27:0/122024565 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5c80cda40 0x7fd5c806e390 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd5c80ce510 tx=0x7fd5c806f7f0 comp rx=0 tx=0).stop 2024-01-15T18:08:37.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.474+0000 7fd5cea00640 1 -- 172.21.15.27:0/122024565 shutdown_connections 2024-01-15T18:08:37.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.474+0000 7fd5cea00640 1 --2- 172.21.15.27:0/122024565 >> v2:172.21.15.27:3300/0 conn(0x7fd5c80cfb10 0x7fd5c80cc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:37.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.474+0000 7fd5cea00640 1 --2- 172.21.15.27:0/122024565 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5c80cda40 0x7fd5c806e390 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:37.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.474+0000 7fd5cea00640 1 -- 172.21.15.27:0/122024565 >> 172.21.15.27:0/122024565 conn(0x7fd5c80d0e50 msgr2=0x7fd5c80d1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:37.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.474+0000 7fd5cea00640 1 -- 172.21.15.27:0/122024565 shutdown_connections 2024-01-15T18:08:37.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.474+0000 7fd5cea00640 1 -- 172.21.15.27:0/122024565 wait complete. 2024-01-15T18:08:37.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.478+0000 7fd5cea00640 1 Processor -- start 2024-01-15T18:08:37.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.478+0000 7fd5cea00640 1 -- start start 2024-01-15T18:08:37.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.478+0000 7fd5cea00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5c80b0270 0x7fd5c814b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:37.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.478+0000 7fd5cea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5c80cda40 0x7fd5c80aedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:37.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.482+0000 7fd5cea00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd5c80cfb10 0x7fd5c8157b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.482+0000 7fd5cea00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c80ae920 con 0x7fd5c80cda40 2024-01-15T18:08:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.482+0000 7fd5cea00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c814fb80 con 0x7fd5c80b0270 2024-01-15T18:08:37.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.482+0000 7fd5cea00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c814fe60 con 0x7fd5c80cfb10 2024-01-15T18:08:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.482+0000 7fd5ce000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd5c80cfb10 0x7fd5c8157b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.482+0000 7fd5cd600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5c80b0270 0x7fd5c814b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.482+0000 7fd5ce000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd5c80cfb10 0x7fd5c8157b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:38814/0 (socket says 172.21.15.27:38814) 2024-01-15T18:08:37.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.482+0000 7fd5ce000640 1 -- 172.21.15.27:0/470088135 learned_addr learned my addr 172.21.15.27:0/470088135 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:37.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.482+0000 7fd5ccc00640 1 --2- 172.21.15.27:0/470088135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5c80cda40 0x7fd5c80aedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:37.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.482+0000 7fd5ce000640 1 -- 172.21.15.27:0/470088135 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5c80b0270 msgr2=0x7fd5c814b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:37.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.482+0000 7fd5ce000640 1 --2- 172.21.15.27:0/470088135 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5c80b0270 0x7fd5c814b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:37.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.482+0000 7fd5ce000640 1 -- 172.21.15.27:0/470088135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5c80cda40 msgr2=0x7fd5c80aedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:37.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.482+0000 7fd5ce000640 1 --2- 172.21.15.27:0/470088135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5c80cda40 0x7fd5c80aedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:37.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.482+0000 7fd5ce000640 1 -- 172.21.15.27:0/470088135 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b8009420 con 0x7fd5c80cfb10 2024-01-15T18:08:37.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.482+0000 7fd5ce000640 1 --2- 172.21.15.27:0/470088135 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd5c80cfb10 0x7fd5c8157b70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0009fd0 tx=0x7fd5c000e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:37.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.486+0000 7fd5c5600640 1 -- 172.21.15.27:0/470088135 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c0014420 con 0x7fd5c80cfb10 2024-01-15T18:08:37.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.486+0000 7fd5c5600640 1 -- 172.21.15.27:0/470088135 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c0014580 con 0x7fd5c80cfb10 2024-01-15T18:08:37.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.486+0000 7fd5c5600640 1 -- 172.21.15.27:0/470088135 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c0014910 con 0x7fd5c80cfb10 2024-01-15T18:08:37.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.486+0000 7fd5cea00640 1 -- 172.21.15.27:0/470088135 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5c80ae5b0 con 0x7fd5c80cfb10 2024-01-15T18:08:37.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.486+0000 7fd5cea00640 1 -- 172.21.15.27:0/470088135 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5c80b5050 con 0x7fd5c80cfb10 2024-01-15T18:08:37.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.486+0000 7fd5cea00640 1 -- 172.21.15.27:0/470088135 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5c80aead0 con 0x7fd5c80cfb10 2024-01-15T18:08:37.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.490+0000 7fd5c5600640 1 -- 172.21.15.27:0/470088135 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 12) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fd5c0014a70 con 0x7fd5c80cfb10 2024-01-15T18:08:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.490+0000 7fd5c5600640 1 --2- 172.21.15.27:0/470088135 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd5a0066e50 0x7fd5a0069310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.490+0000 7fd5c5600640 1 -- 172.21.15.27:0/470088135 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd5c0085490 con 0x7fd5c80cfb10 2024-01-15T18:08:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.490+0000 7fd5cd600640 1 --2- 172.21.15.27:0/470088135 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd5a0066e50 0x7fd5a0069310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:37.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.494+0000 7fd5cd600640 1 --2- 172.21.15.27:0/470088135 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd5a0066e50 0x7fd5a0069310 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd5c80afcd0 tx=0x7fd5b8004740 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:37.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.498+0000 7fd5c5600640 1 -- 172.21.15.27:0/470088135 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd5c00521b0 con 0x7fd5c80cfb10 2024-01-15T18:08:37.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.858+0000 7fd5cea00640 1 -- 172.21.15.27:0/470088135 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone63893", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5c80b5330 con 0x7fd5a0066e50 2024-01-15T18:08:37.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.886+0000 7fd5c5600640 1 -- 172.21.15.27:0/470088135 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fd5c80b5330 con 0x7fd5a0066e50 2024-01-15T18:08:37.893 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:08:37.893 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:08:37.893 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:08:37.894 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:08:37.894 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:08:37.894 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:08:37.894 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:08:37.894 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:37.894 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:08:37.894 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 66.66666666666666, 2024-01-15T18:08:37.894 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T18:08:37.894 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "2/3", 2024-01-15T18:08:37.894 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:08:37.894 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:08:37.894 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:37.894 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:37.894 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:08:37.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.890+0000 7fd5cea00640 1 -- 172.21.15.27:0/470088135 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd5a0066e50 msgr2=0x7fd5a0069310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:37.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.890+0000 7fd5cea00640 1 --2- 172.21.15.27:0/470088135 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd5a0066e50 0x7fd5a0069310 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd5c80afcd0 tx=0x7fd5b8004740 comp rx=0 tx=0).stop 2024-01-15T18:08:37.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.890+0000 7fd5cea00640 1 -- 172.21.15.27:0/470088135 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd5c80cfb10 msgr2=0x7fd5c8157b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:37.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.890+0000 7fd5cea00640 1 --2- 172.21.15.27:0/470088135 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd5c80cfb10 0x7fd5c8157b70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0009fd0 tx=0x7fd5c000e4b0 comp rx=0 tx=0).stop 2024-01-15T18:08:37.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.890+0000 7fd5cea00640 1 -- 172.21.15.27:0/470088135 shutdown_connections 2024-01-15T18:08:37.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.890+0000 7fd5cea00640 1 --2- 172.21.15.27:0/470088135 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd5a0066e50 0x7fd5a0069310 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:37.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.890+0000 7fd5cea00640 1 --2- 172.21.15.27:0/470088135 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd5c80cfb10 0x7fd5c8157b70 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:37.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.890+0000 7fd5cea00640 1 --2- 172.21.15.27:0/470088135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5c80cda40 0x7fd5c80aedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:37.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.890+0000 7fd5cea00640 1 --2- 172.21.15.27:0/470088135 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5c80b0270 0x7fd5c814b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:37.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.890+0000 7fd5cea00640 1 -- 172.21.15.27:0/470088135 >> 172.21.15.27:0/470088135 conn(0x7fd5c80d0e50 msgr2=0x7fd5c80b1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:37.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.890+0000 7fd5cea00640 1 -- 172.21.15.27:0/470088135 shutdown_connections 2024-01-15T18:08:37.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:37.890+0000 7fd5cea00640 1 -- 172.21.15.27:0/470088135 wait complete. 2024-01-15T18:08:37.933 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:08:38.933 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone63893 2024-01-15T18:08:39.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.086+0000 7f5c0ec00640 1 Processor -- start 2024-01-15T18:08:39.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.086+0000 7f5c0ec00640 1 -- start start 2024-01-15T18:08:39.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.090+0000 7f5c0ec00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c080ce680 0x7f5c080b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:39.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.090+0000 7f5c0ec00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c08072a40 con 0x7f5c080cd7d0 2024-01-15T18:08:39.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.090+0000 7f5c0ec00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c0806f030 con 0x7f5c080cd3f0 2024-01-15T18:08:39.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.090+0000 7f5c0ec00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c080b43f0 con 0x7f5c080ce680 2024-01-15T18:08:39.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.090+0000 7f5c0d600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c080ce680 0x7f5c080b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:39.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.090+0000 7f5c0d600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c080ce680 0x7f5c080b4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39944/0 (socket says 172.21.15.27:39944) 2024-01-15T18:08:39.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.090+0000 7f5c0d600640 1 -- 172.21.15.27:0/3761240756 learned_addr learned my addr 172.21.15.27:0/3761240756 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:39.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.090+0000 7f5c07400640 1 --1- 172.21.15.27:0/3761240756 >> v1:172.21.15.27:6789/0 conn(0x7f5c080cd7d0 0x7f5c0815a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50676/0 (socket says 172.21.15.27:50676) 2024-01-15T18:08:39.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.090+0000 7f5c07e00640 1 --1- 172.21.15.27:0/3761240756 >> v1:172.21.15.93:6789/0 conn(0x7f5c080cd3f0 0x7f5c0806d9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:46744/0 (socket says 172.21.15.27:46744) 2024-01-15T18:08:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.090+0000 7f5c0d600640 1 -- 172.21.15.27:0/3761240756 >> v1:172.21.15.93:6789/0 conn(0x7f5c080cd3f0 legacy=0x7f5c0806d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.090+0000 7f5c0d600640 1 -- 172.21.15.27:0/3761240756 >> v1:172.21.15.27:6789/0 conn(0x7f5c080cd7d0 legacy=0x7f5c0815a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:39.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.090+0000 7f5c0d600640 1 -- 172.21.15.27:0/3761240756 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c080b46d0 con 0x7f5c080ce680 2024-01-15T18:08:39.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.090+0000 7f5c0d600640 1 --2- 172.21.15.27:0/3761240756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c080ce680 0x7f5c080b4ed0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f5c0000af60 tx=0x7f5c0001dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=802bbeceef4b2234 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:39.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.090+0000 7f5c06a00640 1 -- 172.21.15.27:0/3761240756 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0000a2d0 con 0x7f5c080ce680 2024-01-15T18:08:39.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.094+0000 7f5c06a00640 1 -- 172.21.15.27:0/3761240756 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c000a6960 con 0x7f5c080ce680 2024-01-15T18:08:39.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.094+0000 7f5c06a00640 1 -- 172.21.15.27:0/3761240756 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c000a6cf0 con 0x7f5c080ce680 2024-01-15T18:08:39.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.094+0000 7f5c0ec00640 1 -- 172.21.15.27:0/3761240756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c080ce680 msgr2=0x7f5c080b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:39.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.094+0000 7f5c0ec00640 1 --2- 172.21.15.27:0/3761240756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c080ce680 0x7f5c080b4ed0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f5c0000af60 tx=0x7f5c0001dae0 comp rx=0 tx=0).stop 2024-01-15T18:08:39.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.094+0000 7f5c0ec00640 1 -- 172.21.15.27:0/3761240756 shutdown_connections 2024-01-15T18:08:39.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.094+0000 7f5c0ec00640 1 --2- 172.21.15.27:0/3761240756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c080ce680 0x7f5c080b4ed0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:39.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.094+0000 7f5c0ec00640 1 -- 172.21.15.27:0/3761240756 >> 172.21.15.27:0/3761240756 conn(0x7f5c08099e10 msgr2=0x7f5c080b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:39.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.094+0000 7f5c0ec00640 1 -- 172.21.15.27:0/3761240756 shutdown_connections 2024-01-15T18:08:39.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.094+0000 7f5c0ec00640 1 -- 172.21.15.27:0/3761240756 wait complete. 2024-01-15T18:08:39.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.094+0000 7f5c0ec00640 1 Processor -- start 2024-01-15T18:08:39.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.094+0000 7f5c0ec00640 1 -- start start 2024-01-15T18:08:39.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.094+0000 7f5c0ec00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c080cd3f0 0x7f5c0814e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:39.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.094+0000 7f5c0ec00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c080cd7d0 0x7f5c0809c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:39.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.094+0000 7f5c0ec00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5c080ce680 0x7f5c0806a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:39.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.094+0000 7f5c0ec00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0806b490 con 0x7f5c080cd3f0 2024-01-15T18:08:39.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.094+0000 7f5c0ec00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0806aed0 con 0x7f5c080ce680 2024-01-15T18:08:39.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.094+0000 7f5c0ec00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c0806b1b0 con 0x7f5c080cd7d0 2024-01-15T18:08:39.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.094+0000 7f5c07e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c080cd3f0 0x7f5c0814e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:39.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.098+0000 7f5c07e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c080cd3f0 0x7f5c0814e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39958/0 (socket says 172.21.15.27:39958) 2024-01-15T18:08:39.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.098+0000 7f5c07400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c080cd7d0 0x7f5c0809c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:39.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.098+0000 7f5c07e00640 1 -- 172.21.15.27:0/2723114640 learned_addr learned my addr 172.21.15.27:0/2723114640 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:39.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.098+0000 7f5c0d600640 1 --2- 172.21.15.27:0/2723114640 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5c080ce680 0x7f5c0806a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:39.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.098+0000 7f5c0d600640 1 -- 172.21.15.27:0/2723114640 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c080cd7d0 msgr2=0x7f5c0809c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:39.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.098+0000 7f5c0d600640 1 --2- 172.21.15.27:0/2723114640 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c080cd7d0 0x7f5c0809c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:39.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.098+0000 7f5c0d600640 1 -- 172.21.15.27:0/2723114640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c080cd3f0 msgr2=0x7f5c0814e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:39.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.098+0000 7f5c0d600640 1 --2- 172.21.15.27:0/2723114640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c080cd3f0 0x7f5c0814e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:39.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.098+0000 7f5c0d600640 1 -- 172.21.15.27:0/2723114640 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c0000ac40 con 0x7f5c080ce680 2024-01-15T18:08:39.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.098+0000 7f5c07e00640 1 --2- 172.21.15.27:0/2723114640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c080cd3f0 0x7f5c0814e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:08:39.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.098+0000 7f5c0d600640 1 --2- 172.21.15.27:0/2723114640 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5c080ce680 0x7f5c0806a4f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5c000ac000 tx=0x7f5c0001a880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:39.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.098+0000 7f5bffe00640 1 -- 172.21.15.27:0/2723114640 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c000be070 con 0x7f5c080ce680 2024-01-15T18:08:39.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.098+0000 7f5bffe00640 1 -- 172.21.15.27:0/2723114640 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c00008500 con 0x7f5c080ce680 2024-01-15T18:08:39.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.098+0000 7f5bffe00640 1 -- 172.21.15.27:0/2723114640 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c000088b0 con 0x7f5c080ce680 2024-01-15T18:08:39.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.098+0000 7f5c0ec00640 1 -- 172.21.15.27:0/2723114640 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c08150ea0 con 0x7f5c080ce680 2024-01-15T18:08:39.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.098+0000 7f5c0ec00640 1 -- 172.21.15.27:0/2723114640 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c0809caf0 con 0x7f5c080ce680 2024-01-15T18:08:39.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.098+0000 7f5c0ec00640 1 -- 172.21.15.27:0/2723114640 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c0815be20 con 0x7f5c080ce680 2024-01-15T18:08:39.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.098+0000 7f5bffe00640 1 -- 172.21.15.27:0/2723114640 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 12) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f5c00008a10 con 0x7f5c080ce680 2024-01-15T18:08:39.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.102+0000 7f5bffe00640 1 --2- 172.21.15.27:0/2723114640 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5be0066e00 0x7f5be00692c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:39.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.102+0000 7f5bffe00640 1 -- 172.21.15.27:0/2723114640 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5c0011fe60 con 0x7f5c080ce680 2024-01-15T18:08:39.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.102+0000 7f5c07e00640 1 --2- 172.21.15.27:0/2723114640 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5be0066e00 0x7f5be00692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:39.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.102+0000 7f5c07e00640 1 --2- 172.21.15.27:0/2723114640 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5be0066e00 0x7f5be00692c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8000f70 tx=0x7f5bf8007f10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:39.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.106+0000 7f5bffe00640 1 -- 172.21.15.27:0/2723114640 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5c000ecaf0 con 0x7f5c080ce680 2024-01-15T18:08:39.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:39.298+0000 7f5c0ec00640 1 -- 172.21.15.27:0/2723114640 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone63893", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c080ceb10 con 0x7f5be0066e00 2024-01-15T18:08:40.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.002+0000 7f5bffe00640 1 -- 172.21.15.27:0/2723114640 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f5bf80076e0 con 0x7f5be0066e00 2024-01-15T18:08:40.009 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:08:40.009 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:08:40.009 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "complete" 2024-01-15T18:08:40.009 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:40.009 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:08:40.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.006+0000 7f5c0ec00640 1 -- 172.21.15.27:0/2723114640 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5be0066e00 msgr2=0x7f5be00692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:40.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.006+0000 7f5c0ec00640 1 --2- 172.21.15.27:0/2723114640 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5be0066e00 0x7f5be00692c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8000f70 tx=0x7f5bf8007f10 comp rx=0 tx=0).stop 2024-01-15T18:08:40.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.006+0000 7f5c0ec00640 1 -- 172.21.15.27:0/2723114640 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5c080ce680 msgr2=0x7f5c0806a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:40.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.006+0000 7f5c0ec00640 1 --2- 172.21.15.27:0/2723114640 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5c080ce680 0x7f5c0806a4f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5c000ac000 tx=0x7f5c0001a880 comp rx=0 tx=0).stop 2024-01-15T18:08:40.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.006+0000 7f5c0ec00640 1 -- 172.21.15.27:0/2723114640 shutdown_connections 2024-01-15T18:08:40.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.006+0000 7f5c0ec00640 1 --2- 172.21.15.27:0/2723114640 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5be0066e00 0x7f5be00692c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:40.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.006+0000 7f5c0ec00640 1 --2- 172.21.15.27:0/2723114640 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5c080ce680 0x7f5c0806a4f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:40.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.006+0000 7f5c0ec00640 1 --2- 172.21.15.27:0/2723114640 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5c080cd7d0 0x7f5c0809c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:40.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.006+0000 7f5c0ec00640 1 --2- 172.21.15.27:0/2723114640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c080cd3f0 0x7f5c0814e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:40.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.006+0000 7f5c0ec00640 1 -- 172.21.15.27:0/2723114640 >> 172.21.15.27:0/2723114640 conn(0x7f5c08099e10 msgr2=0x7f5c080cfb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:40.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.006+0000 7f5c0ec00640 1 -- 172.21.15.27:0/2723114640 shutdown_connections 2024-01-15T18:08:40.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.006+0000 7f5c0ec00640 1 -- 172.21.15.27:0/2723114640 wait complete. 2024-01-15T18:08:40.035 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:08:40.035 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone63894 2024-01-15T18:08:40.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.182+0000 7f7415e00640 1 Processor -- start 2024-01-15T18:08:40.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.182+0000 7f7415e00640 1 -- start start 2024-01-15T18:08:40.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.182+0000 7f7415e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f741015b100 0x7f7410070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:40.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.182+0000 7f7415e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7410071420 0x7f741015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:40.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.182+0000 7f7415e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f741015a610 con 0x7f741015ad20 2024-01-15T18:08:40.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.182+0000 7f7415e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74100718b0 con 0x7f7410071420 2024-01-15T18:08:40.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.182+0000 7f7415e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f741015e3c0 con 0x7f741015b100 2024-01-15T18:08:40.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.182+0000 7f740ea00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f741015ad20 0x7f741015d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50684/0 (socket says 172.21.15.27:50684) 2024-01-15T18:08:40.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.182+0000 7f740ea00640 1 -- 172.21.15.27:0/2264082808 learned_addr learned my addr 172.21.15.27:0/2264082808 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:40.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.182+0000 7f740f400640 1 --2- 172.21.15.27:0/2264082808 >> v2:172.21.15.27:3300/0 conn(0x7f7410071420 0x7f741015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:40.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.182+0000 7f740e000640 1 --2- 172.21.15.27:0/2264082808 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f741015b100 0x7f7410070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:40.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.182+0000 7f740f400640 1 -- 172.21.15.27:0/2264082808 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f741015b100 msgr2=0x7f7410070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:40.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.182+0000 7f740f400640 1 --2- 172.21.15.27:0/2264082808 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f741015b100 0x7f7410070ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:40.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.182+0000 7f740f400640 1 -- 172.21.15.27:0/2264082808 >> v1:172.21.15.27:6789/0 conn(0x7f741015ad20 legacy=0x7f741015d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:40.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.182+0000 7f740f400640 1 -- 172.21.15.27:0/2264082808 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f741015e6a0 con 0x7f7410071420 2024-01-15T18:08:40.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.186+0000 7f740f400640 1 --2- 172.21.15.27:0/2264082808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7410071420 0x7f741015bee0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f7408009d90 tx=0x7f740801c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcea84085d5df70d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:40.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.186+0000 7f740d600640 1 -- 172.21.15.27:0/2264082808 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74080a58f0 con 0x7f7410071420 2024-01-15T18:08:40.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.186+0000 7f740d600640 1 -- 172.21.15.27:0/2264082808 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74080a5a50 con 0x7f7410071420 2024-01-15T18:08:40.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.186+0000 7f740d600640 1 -- 172.21.15.27:0/2264082808 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74080a5de0 con 0x7f7410071420 2024-01-15T18:08:40.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.186+0000 7f7415e00640 1 -- 172.21.15.27:0/2264082808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7410071420 msgr2=0x7f741015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:40.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.186+0000 7f7415e00640 1 --2- 172.21.15.27:0/2264082808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7410071420 0x7f741015bee0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f7408009d90 tx=0x7f740801c920 comp rx=0 tx=0).stop 2024-01-15T18:08:40.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.186+0000 7f7415e00640 1 -- 172.21.15.27:0/2264082808 shutdown_connections 2024-01-15T18:08:40.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.186+0000 7f7415e00640 1 --2- 172.21.15.27:0/2264082808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7410071420 0x7f741015bee0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:40.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.186+0000 7f7415e00640 1 --2- 172.21.15.27:0/2264082808 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f741015b100 0x7f7410070ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:40.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.186+0000 7f7415e00640 1 -- 172.21.15.27:0/2264082808 >> 172.21.15.27:0/2264082808 conn(0x7f74100cb700 msgr2=0x7f74100cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:40.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.186+0000 7f7415e00640 1 -- 172.21.15.27:0/2264082808 shutdown_connections 2024-01-15T18:08:40.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.186+0000 7f7415e00640 1 -- 172.21.15.27:0/2264082808 wait complete. 2024-01-15T18:08:40.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.190+0000 7f7415e00640 1 Processor -- start 2024-01-15T18:08:40.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.190+0000 7f7415e00640 1 -- start start 2024-01-15T18:08:40.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.190+0000 7f7415e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7410071420 0x7f7410145c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:40.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.190+0000 7f7415e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f741015ad20 0x7f74100b2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:40.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.190+0000 7f7415e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f741015b100 0x7f74100ceb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:40.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.190+0000 7f7415e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741014aaa0 con 0x7f741015ad20 2024-01-15T18:08:40.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.190+0000 7f7415e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7410152010 con 0x7f741015b100 2024-01-15T18:08:40.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.190+0000 7f7415e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74101522f0 con 0x7f7410071420 2024-01-15T18:08:40.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.190+0000 7f740e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f741015ad20 0x7f74100b2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:40.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.190+0000 7f740e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f741015ad20 0x7f74100b2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39978/0 (socket says 172.21.15.27:39978) 2024-01-15T18:08:40.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.190+0000 7f740e000640 1 -- 172.21.15.27:0/2895365419 learned_addr learned my addr 172.21.15.27:0/2895365419 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:40.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.190+0000 7f740ea00640 1 --2- 172.21.15.27:0/2895365419 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7410071420 0x7f7410145c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:40.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.190+0000 7f740f400640 1 --2- 172.21.15.27:0/2895365419 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f741015b100 0x7f74100ceb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:40.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.190+0000 7f740e000640 1 -- 172.21.15.27:0/2895365419 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7410071420 msgr2=0x7f7410145c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:40.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.190+0000 7f740e000640 1 --2- 172.21.15.27:0/2895365419 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7410071420 0x7f7410145c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:40.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.190+0000 7f740e000640 1 -- 172.21.15.27:0/2895365419 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f741015b100 msgr2=0x7f74100ceb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:40.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.190+0000 7f740e000640 1 --2- 172.21.15.27:0/2895365419 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f741015b100 0x7f74100ceb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:40.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.190+0000 7f740e000640 1 -- 172.21.15.27:0/2895365419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7408009a70 con 0x7f741015ad20 2024-01-15T18:08:40.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.190+0000 7f740e000640 1 --2- 172.21.15.27:0/2895365419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f741015ad20 0x7f74100b2610 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f73f800d5f0 tx=0x7f73f800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:40.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.190+0000 7f7407400640 1 -- 172.21.15.27:0/2895365419 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f80125f0 con 0x7f741015ad20 2024-01-15T18:08:40.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.194+0000 7f7407400640 1 -- 172.21.15.27:0/2895365419 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f8012750 con 0x7f741015ad20 2024-01-15T18:08:40.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.194+0000 7f7407400640 1 -- 172.21.15.27:0/2895365419 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f8012ae0 con 0x7f741015ad20 2024-01-15T18:08:40.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.194+0000 7f7415e00640 1 -- 172.21.15.27:0/2895365419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74101525d0 con 0x7f741015ad20 2024-01-15T18:08:40.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.194+0000 7f7415e00640 1 -- 172.21.15.27:0/2895365419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74100b2b50 con 0x7f741015ad20 2024-01-15T18:08:40.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.194+0000 7f7415e00640 1 -- 172.21.15.27:0/2895365419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f741004e850 con 0x7f741015ad20 2024-01-15T18:08:40.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.194+0000 7f7407400640 1 -- 172.21.15.27:0/2895365419 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f73f8012c40 con 0x7f741015ad20 2024-01-15T18:08:40.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.198+0000 7f7407400640 1 --2- 172.21.15.27:0/2895365419 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f73e8066e00 0x7f73e80692c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:40.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.198+0000 7f7407400640 1 -- 172.21.15.27:0/2895365419 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f73f8085060 con 0x7f741015ad20 2024-01-15T18:08:40.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.198+0000 7f740ea00640 1 --2- 172.21.15.27:0/2895365419 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f73e8066e00 0x7f73e80692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:40.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.198+0000 7f740ea00640 1 --2- 172.21.15.27:0/2895365419 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f73e8066e00 0x7f73e80692c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7400005890 tx=0x7f7400009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:40.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.202+0000 7f7407400640 1 -- 172.21.15.27:0/2895365419 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f73f8051c60 con 0x7f741015ad20 2024-01-15T18:08:40.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.462+0000 7f7415e00640 1 -- 172.21.15.27:0/2895365419 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone63894", "target": ["mon-mgr", ""]}) v1 -- 0x7f74101515b0 con 0x7f73e8066e00 2024-01-15T18:08:40.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.482+0000 7f7407400640 1 -- 172.21.15.27:0/2895365419 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f74101515b0 con 0x7f73e8066e00 2024-01-15T18:08:40.489 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:08:40.489 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:08:40.489 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:08:40.489 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:08:40.489 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:08:40.489 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:08:40.489 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:08:40.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:40.489 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:08:40.489 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 66.66666666666666, 2024-01-15T18:08:40.489 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T18:08:40.489 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "2/3", 2024-01-15T18:08:40.489 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:08:40.490 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:08:40.490 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:40.490 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:40.490 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:08:40.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.486+0000 7f7415e00640 1 -- 172.21.15.27:0/2895365419 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f73e8066e00 msgr2=0x7f73e80692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:40.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.486+0000 7f7415e00640 1 --2- 172.21.15.27:0/2895365419 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f73e8066e00 0x7f73e80692c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7400005890 tx=0x7f7400009040 comp rx=0 tx=0).stop 2024-01-15T18:08:40.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.486+0000 7f7415e00640 1 -- 172.21.15.27:0/2895365419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f741015ad20 msgr2=0x7f74100b2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:40.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.486+0000 7f7415e00640 1 --2- 172.21.15.27:0/2895365419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f741015ad20 0x7f74100b2610 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f73f800d5f0 tx=0x7f73f800dab0 comp rx=0 tx=0).stop 2024-01-15T18:08:40.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.486+0000 7f7415e00640 1 -- 172.21.15.27:0/2895365419 shutdown_connections 2024-01-15T18:08:40.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.486+0000 7f7415e00640 1 --2- 172.21.15.27:0/2895365419 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f73e8066e00 0x7f73e80692c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:40.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.486+0000 7f7415e00640 1 --2- 172.21.15.27:0/2895365419 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f741015b100 0x7f74100ceb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:40.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.486+0000 7f7415e00640 1 --2- 172.21.15.27:0/2895365419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f741015ad20 0x7f74100b2610 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:40.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.486+0000 7f7415e00640 1 --2- 172.21.15.27:0/2895365419 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7410071420 0x7f7410145c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:40.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.486+0000 7f7415e00640 1 -- 172.21.15.27:0/2895365419 >> 172.21.15.27:0/2895365419 conn(0x7f74100cb700 msgr2=0x7f7410150ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:40.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.486+0000 7f7415e00640 1 -- 172.21.15.27:0/2895365419 shutdown_connections 2024-01-15T18:08:40.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:40.486+0000 7f7415e00640 1 -- 172.21.15.27:0/2895365419 wait complete. 2024-01-15T18:08:40.520 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:08:41.521 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone63894 2024-01-15T18:08:41.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.726+0000 7fe1ce800640 1 Processor -- start 2024-01-15T18:08:41.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.730+0000 7fe1ce800640 1 -- start start 2024-01-15T18:08:41.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.730+0000 7fe1ce800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe1c80b2780 0x7fe1c8161d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:41.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.730+0000 7fe1ce800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fe1c80b2b60 0x7fe1c806dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:41.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.730+0000 7fe1ce800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1c80aee00 con 0x7fe1c8162240 2024-01-15T18:08:41.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.730+0000 7fe1ce800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c806a4f0 con 0x7fe1c80b2780 2024-01-15T18:08:41.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.730+0000 7fe1ce800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c806aa70 con 0x7fe1c80b2b60 2024-01-15T18:08:41.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.730+0000 7fe1c7e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe1c8162240 0x7fe1c806e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50688/0 (socket says 172.21.15.27:50688) 2024-01-15T18:08:41.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.730+0000 7fe1c6a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fe1c80b2b60 0x7fe1c806dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:41.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.730+0000 7fe1c7400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe1c80b2780 0x7fe1c8161d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:41.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.730+0000 7fe1c6a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fe1c80b2b60 0x7fe1c806dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:46994/0 (socket says 172.21.15.27:46994) 2024-01-15T18:08:41.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.730+0000 7fe1c7e00640 1 -- 172.21.15.27:0/1296655067 learned_addr learned my addr 172.21.15.27:0/1296655067 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:41.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.734+0000 7fe1c6a00640 1 -- 172.21.15.27:0/1296655067 >> v2:172.21.15.27:3300/0 conn(0x7fe1c80b2780 msgr2=0x7fe1c8161d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:41.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.734+0000 7fe1c6a00640 1 --2- 172.21.15.27:0/1296655067 >> v2:172.21.15.27:3300/0 conn(0x7fe1c80b2780 0x7fe1c8161d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:41.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.734+0000 7fe1c6000640 1 -- 172.21.15.27:0/1296655067 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338195732 0 0) 0x7fe1c80aee00 con 0x7fe1c8162240 2024-01-15T18:08:41.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.734+0000 7fe1c6a00640 1 -- 172.21.15.27:0/1296655067 >> v1:172.21.15.27:6789/0 conn(0x7fe1c8162240 legacy=0x7fe1c806e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:41.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.734+0000 7fe1c6a00640 1 -- 172.21.15.27:0/1296655067 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c8162810 con 0x7fe1c80b2b60 2024-01-15T18:08:41.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.738+0000 7fe1c6a00640 1 --2- 172.21.15.27:0/1296655067 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe1c80b2b60 0x7fe1c806dec0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe1c80665d0 tx=0x7fe1b001c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94a00de4dd629731 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:41.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.738+0000 7fe1c6000640 1 -- 172.21.15.27:0/1296655067 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b00ac620 con 0x7fe1c80b2b60 2024-01-15T18:08:41.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.738+0000 7fe1c6000640 1 -- 172.21.15.27:0/1296655067 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b00ac780 con 0x7fe1c80b2b60 2024-01-15T18:08:41.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.738+0000 7fe1c6000640 1 -- 172.21.15.27:0/1296655067 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b00acb10 con 0x7fe1c80b2b60 2024-01-15T18:08:41.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.742+0000 7fe1ce800640 1 -- 172.21.15.27:0/1296655067 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe1c80b2b60 msgr2=0x7fe1c806dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:41.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.742+0000 7fe1ce800640 1 --2- 172.21.15.27:0/1296655067 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe1c80b2b60 0x7fe1c806dec0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe1c80665d0 tx=0x7fe1b001c3a0 comp rx=0 tx=0).stop 2024-01-15T18:08:41.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.742+0000 7fe1ce800640 1 -- 172.21.15.27:0/1296655067 shutdown_connections 2024-01-15T18:08:41.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.742+0000 7fe1ce800640 1 --2- 172.21.15.27:0/1296655067 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe1c80b2b60 0x7fe1c806dec0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:41.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.742+0000 7fe1ce800640 1 --2- 172.21.15.27:0/1296655067 >> v2:172.21.15.27:3300/0 conn(0x7fe1c80b2780 0x7fe1c8161d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:41.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.742+0000 7fe1ce800640 1 -- 172.21.15.27:0/1296655067 >> 172.21.15.27:0/1296655067 conn(0x7fe1c80cf530 msgr2=0x7fe1c80cf940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:41.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.742+0000 7fe1ce800640 1 -- 172.21.15.27:0/1296655067 shutdown_connections 2024-01-15T18:08:41.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.742+0000 7fe1ce800640 1 -- 172.21.15.27:0/1296655067 wait complete. 2024-01-15T18:08:41.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.746+0000 7fe1ce800640 1 Processor -- start 2024-01-15T18:08:41.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.746+0000 7fe1ce800640 1 -- start start 2024-01-15T18:08:41.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.750+0000 7fe1ce800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1c80b2780 0x7fe1c8155ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:41.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.754+0000 7fe1ce800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe1c80b2b60 0x7fe1c8153780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:41.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.754+0000 7fe1ce800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe1c8162240 0x7fe1c806dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:41.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.754+0000 7fe1ce800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c8156000 con 0x7fe1c80b2780 2024-01-15T18:08:41.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.754+0000 7fe1ce800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c8156a30 con 0x7fe1c80b2b60 2024-01-15T18:08:41.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.754+0000 7fe1ce800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c815b8a0 con 0x7fe1c8162240 2024-01-15T18:08:41.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.754+0000 7fe1c7400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1c80b2780 0x7fe1c8155ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:41.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.754+0000 7fe1c7400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1c80b2780 0x7fe1c8155ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40002/0 (socket says 172.21.15.27:40002) 2024-01-15T18:08:41.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.754+0000 7fe1c7400640 1 -- 172.21.15.27:0/2502133611 learned_addr learned my addr 172.21.15.27:0/2502133611 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:41.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.754+0000 7fe1c7e00640 1 --2- 172.21.15.27:0/2502133611 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe1c8162240 0x7fe1c806dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:41.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.754+0000 7fe1c6a00640 1 --2- 172.21.15.27:0/2502133611 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe1c80b2b60 0x7fe1c8153780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:41.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.754+0000 7fe1c7e00640 1 -- 172.21.15.27:0/2502133611 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe1c80b2b60 msgr2=0x7fe1c8153780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:41.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.754+0000 7fe1c7e00640 1 --2- 172.21.15.27:0/2502133611 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe1c80b2b60 0x7fe1c8153780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:41.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.754+0000 7fe1c7e00640 1 -- 172.21.15.27:0/2502133611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1c80b2780 msgr2=0x7fe1c8155ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:41.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.754+0000 7fe1c7e00640 1 --2- 172.21.15.27:0/2502133611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1c80b2780 0x7fe1c8155ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:41.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.754+0000 7fe1c7e00640 1 -- 172.21.15.27:0/2502133611 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b0009420 con 0x7fe1c8162240 2024-01-15T18:08:41.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.754+0000 7fe1c7400640 1 --2- 172.21.15.27:0/2502133611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1c80b2780 0x7fe1c8155ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:08:41.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.754+0000 7fe1c7e00640 1 --2- 172.21.15.27:0/2502133611 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe1c8162240 0x7fe1c806dd70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000c780 tx=0x7fe1c000cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:41.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.758+0000 7fe1bf400640 1 -- 172.21.15.27:0/2502133611 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c000ce30 con 0x7fe1c8162240 2024-01-15T18:08:41.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.758+0000 7fe1ce800640 1 -- 172.21.15.27:0/2502133611 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c8205c40 con 0x7fe1c8162240 2024-01-15T18:08:41.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.758+0000 7fe1bf400640 1 -- 172.21.15.27:0/2502133611 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c000a640 con 0x7fe1c8162240 2024-01-15T18:08:41.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.758+0000 7fe1ce800640 1 -- 172.21.15.27:0/2502133611 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c8206200 con 0x7fe1c8162240 2024-01-15T18:08:41.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.758+0000 7fe1bf400640 1 -- 172.21.15.27:0/2502133611 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c000ce30 con 0x7fe1c8162240 2024-01-15T18:08:41.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.758+0000 7fe1ce800640 1 -- 172.21.15.27:0/2502133611 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe198005260 con 0x7fe1c8162240 2024-01-15T18:08:41.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.758+0000 7fe1bf400640 1 -- 172.21.15.27:0/2502133611 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 12) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fe1c000a7b0 con 0x7fe1c8162240 2024-01-15T18:08:41.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.762+0000 7fe1bf400640 1 --2- 172.21.15.27:0/2502133611 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe1a0066e90 0x7fe1a0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:41.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.762+0000 7fe1bf400640 1 -- 172.21.15.27:0/2502133611 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe1c0085de0 con 0x7fe1c8162240 2024-01-15T18:08:41.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.762+0000 7fe1c7400640 1 --2- 172.21.15.27:0/2502133611 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe1a0066e90 0x7fe1a0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:41.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.762+0000 7fe1c7400640 1 --2- 172.21.15.27:0/2502133611 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe1a0066e90 0x7fe1a0069350 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8011310 tx=0x7fe1b8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:41.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:41.766+0000 7fe1bf400640 1 -- 172.21.15.27:0/2502133611 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe1c0089060 con 0x7fe1c8162240 2024-01-15T18:08:42.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:42.030+0000 7fe1ce800640 1 -- 172.21.15.27:0/2502133611 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone63894", "target": ["mon-mgr", ""]}) v1 -- 0x7fe198000bd0 con 0x7fe1a0066e90 2024-01-15T18:08:42.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:42.042+0000 7fe1bf400640 1 -- 172.21.15.27:0/2502133611 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fe198000bd0 con 0x7fe1a0066e90 2024-01-15T18:08:42.050 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:08:42.050 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:08:42.050 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:08:42.050 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:08:42.050 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:08:42.050 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:08:42.051 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:08:42.051 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:42.051 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:08:42.051 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 66.66666666666666, 2024-01-15T18:08:42.051 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T18:08:42.051 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "2/3", 2024-01-15T18:08:42.051 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:08:42.051 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:08:42.051 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:42.051 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:42.051 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:08:42.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:42.046+0000 7fe1ce800640 1 -- 172.21.15.27:0/2502133611 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe1a0066e90 msgr2=0x7fe1a0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:42.046+0000 7fe1ce800640 1 --2- 172.21.15.27:0/2502133611 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe1a0066e90 0x7fe1a0069350 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8011310 tx=0x7fe1b8009190 comp rx=0 tx=0).stop 2024-01-15T18:08:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:42.046+0000 7fe1ce800640 1 -- 172.21.15.27:0/2502133611 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe1c8162240 msgr2=0x7fe1c806dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:42.046+0000 7fe1ce800640 1 --2- 172.21.15.27:0/2502133611 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe1c8162240 0x7fe1c806dd70 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000c780 tx=0x7fe1c000cc40 comp rx=0 tx=0).stop 2024-01-15T18:08:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:42.046+0000 7fe1ce800640 1 -- 172.21.15.27:0/2502133611 shutdown_connections 2024-01-15T18:08:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:42.046+0000 7fe1ce800640 1 --2- 172.21.15.27:0/2502133611 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe1a0066e90 0x7fe1a0069350 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:42.046+0000 7fe1ce800640 1 --2- 172.21.15.27:0/2502133611 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe1c8162240 0x7fe1c806dd70 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:42.046+0000 7fe1ce800640 1 --2- 172.21.15.27:0/2502133611 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe1c80b2b60 0x7fe1c8153780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:42.046+0000 7fe1ce800640 1 --2- 172.21.15.27:0/2502133611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1c80b2780 0x7fe1c8155ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:42.046+0000 7fe1ce800640 1 -- 172.21.15.27:0/2502133611 >> 172.21.15.27:0/2502133611 conn(0x7fe1c80cf530 msgr2=0x7fe1c80cb710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:42.046+0000 7fe1ce800640 1 -- 172.21.15.27:0/2502133611 shutdown_connections 2024-01-15T18:08:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:42.046+0000 7fe1ce800640 1 -- 172.21.15.27:0/2502133611 wait complete. 2024-01-15T18:08:42.075 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:08:43.075 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone63894 2024-01-15T18:08:43.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.226+0000 7f4816400640 1 Processor -- start 2024-01-15T18:08:43.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.226+0000 7f4816400640 1 -- start start 2024-01-15T18:08:43.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.226+0000 7f4816400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f48100cf460 0x7f48100cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:43.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.226+0000 7f4816400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f481015b4b0 0x7f48100cba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:43.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.226+0000 7f4816400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f48100cec60 0x7f48100cbf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:43.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.226+0000 7f4816400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48100cfd80 con 0x7f48100cf460 2024-01-15T18:08:43.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.226+0000 7f4816400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48100d19a0 con 0x7f481015b4b0 2024-01-15T18:08:43.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.226+0000 7f4816400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48100d1c80 con 0x7f48100cec60 2024-01-15T18:08:43.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.226+0000 7f480f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f48100cf460 0x7f48100cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:43.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.226+0000 7f480f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f48100cf460 0x7f48100cf840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40012/0 (socket says 172.21.15.27:40012) 2024-01-15T18:08:43.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.226+0000 7f480f400640 1 -- 172.21.15.27:0/551996483 learned_addr learned my addr 172.21.15.27:0/551996483 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:43.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.226+0000 7f480ea00640 1 --2- 172.21.15.27:0/551996483 >> v2:172.21.15.93:3300/0 conn(0x7f481015b4b0 0x7f48100cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:43.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.226+0000 7f480fe00640 1 --2- 172.21.15.27:0/551996483 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f48100cec60 0x7f48100cbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:43.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.226+0000 7f480f400640 1 -- 172.21.15.27:0/551996483 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f48100cec60 msgr2=0x7f48100cbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:43.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.226+0000 7f480f400640 1 --2- 172.21.15.27:0/551996483 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f48100cec60 0x7f48100cbf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:43.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.226+0000 7f480f400640 1 -- 172.21.15.27:0/551996483 >> v2:172.21.15.93:3300/0 conn(0x7f481015b4b0 msgr2=0x7f48100cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:43.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.226+0000 7f480f400640 1 --2- 172.21.15.27:0/551996483 >> v2:172.21.15.93:3300/0 conn(0x7f481015b4b0 0x7f48100cba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:43.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.226+0000 7f480f400640 1 -- 172.21.15.27:0/551996483 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48100b07c0 con 0x7f48100cf460 2024-01-15T18:08:43.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.230+0000 7f480f400640 1 --2- 172.21.15.27:0/551996483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48100cf460 0x7f48100cf840 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f48100717b0 tx=0x7f480001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fe35b04af4c5138 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:43.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.230+0000 7f480e000640 1 -- 172.21.15.27:0/551996483 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48000a7070 con 0x7f48100cf460 2024-01-15T18:08:43.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.230+0000 7f480e000640 1 -- 172.21.15.27:0/551996483 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48000ab430 con 0x7f48100cf460 2024-01-15T18:08:43.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.230+0000 7f480e000640 1 -- 172.21.15.27:0/551996483 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48000ab7c0 con 0x7f48100cf460 2024-01-15T18:08:43.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.230+0000 7f4816400640 1 -- 172.21.15.27:0/551996483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48100cf460 msgr2=0x7f48100cf840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:43.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.230+0000 7f4816400640 1 --2- 172.21.15.27:0/551996483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48100cf460 0x7f48100cf840 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f48100717b0 tx=0x7f480001c470 comp rx=0 tx=0).stop 2024-01-15T18:08:43.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.230+0000 7f4816400640 1 -- 172.21.15.27:0/551996483 shutdown_connections 2024-01-15T18:08:43.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.230+0000 7f4816400640 1 --2- 172.21.15.27:0/551996483 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f48100cec60 0x7f48100cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:43.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.230+0000 7f4816400640 1 --2- 172.21.15.27:0/551996483 >> v2:172.21.15.93:3300/0 conn(0x7f481015b4b0 0x7f48100cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:43.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.230+0000 7f4816400640 1 --2- 172.21.15.27:0/551996483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48100cf460 0x7f48100cf840 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:43.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.230+0000 7f4816400640 1 -- 172.21.15.27:0/551996483 >> 172.21.15.27:0/551996483 conn(0x7f48100d0e80 msgr2=0x7f48100d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:43.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.234+0000 7f4816400640 1 -- 172.21.15.27:0/551996483 shutdown_connections 2024-01-15T18:08:43.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.234+0000 7f4816400640 1 -- 172.21.15.27:0/551996483 wait complete. 2024-01-15T18:08:43.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.234+0000 7f4816400640 1 Processor -- start 2024-01-15T18:08:43.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.234+0000 7f4816400640 1 -- start start 2024-01-15T18:08:43.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.234+0000 7f4816400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48100cec60 0x7f48100c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:43.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.234+0000 7f4816400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f48100cf460 0x7f48100c6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:43.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.234+0000 7f4816400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f481015b4b0 0x7f48101fa200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:43.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.234+0000 7f4816400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48100cb2c0 con 0x7f481015b4b0 2024-01-15T18:08:43.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.234+0000 7f4816400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48100cad00 con 0x7f48100cec60 2024-01-15T18:08:43.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.234+0000 7f4816400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48100cafe0 con 0x7f48100cf460 2024-01-15T18:08:43.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.234+0000 7f480fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f481015b4b0 0x7f48101fa200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:43.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.234+0000 7f480fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f481015b4b0 0x7f48101fa200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40024/0 (socket says 172.21.15.27:40024) 2024-01-15T18:08:43.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.234+0000 7f480fe00640 1 -- 172.21.15.27:0/2628595056 learned_addr learned my addr 172.21.15.27:0/2628595056 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:43.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.234+0000 7f480ea00640 1 --2- 172.21.15.27:0/2628595056 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f48100cf460 0x7f48100c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:43.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.234+0000 7f480f400640 1 --2- 172.21.15.27:0/2628595056 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48100cec60 0x7f48100c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:43.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.234+0000 7f480fe00640 1 -- 172.21.15.27:0/2628595056 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f48100cf460 msgr2=0x7f48100c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:43.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.234+0000 7f480fe00640 1 --2- 172.21.15.27:0/2628595056 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f48100cf460 0x7f48100c6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:43.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.234+0000 7f480fe00640 1 -- 172.21.15.27:0/2628595056 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48100cec60 msgr2=0x7f48100c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:43.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.234+0000 7f480fe00640 1 --2- 172.21.15.27:0/2628595056 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48100cec60 0x7f48100c6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:43.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.234+0000 7f480fe00640 1 -- 172.21.15.27:0/2628595056 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48000094d0 con 0x7f481015b4b0 2024-01-15T18:08:43.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.238+0000 7f480fe00640 1 --2- 172.21.15.27:0/2628595056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f481015b4b0 0x7f48101fa200 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f480800eed0 tx=0x7f480800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:43.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.238+0000 7f4807400640 1 -- 172.21.15.27:0/2628595056 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4808010040 con 0x7f481015b4b0 2024-01-15T18:08:43.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.238+0000 7f4807400640 1 -- 172.21.15.27:0/2628595056 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4808014400 con 0x7f481015b4b0 2024-01-15T18:08:43.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.238+0000 7f4807400640 1 -- 172.21.15.27:0/2628595056 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4808009ea0 con 0x7f481015b4b0 2024-01-15T18:08:43.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.238+0000 7f4816400640 1 -- 172.21.15.27:0/2628595056 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f481015bbf0 con 0x7f481015b4b0 2024-01-15T18:08:43.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.238+0000 7f4816400640 1 -- 172.21.15.27:0/2628595056 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4810147600 con 0x7f481015b4b0 2024-01-15T18:08:43.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.238+0000 7f4816400640 1 -- 172.21.15.27:0/2628595056 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f481015c110 con 0x7f481015b4b0 2024-01-15T18:08:43.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.238+0000 7f4807400640 1 -- 172.21.15.27:0/2628595056 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f48080147b0 con 0x7f481015b4b0 2024-01-15T18:08:43.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.238+0000 7f4807400640 1 --2- 172.21.15.27:0/2628595056 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f47e8066e00 0x7f47e80692c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:43.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.238+0000 7f4807400640 1 -- 172.21.15.27:0/2628595056 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4808085690 con 0x7f481015b4b0 2024-01-15T18:08:43.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.238+0000 7f480f400640 1 --2- 172.21.15.27:0/2628595056 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f47e8066e00 0x7f47e80692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:43.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.242+0000 7f480f400640 1 --2- 172.21.15.27:0/2628595056 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f47e8066e00 0x7f47e80692c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f48000af000 tx=0x7f4800008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:43.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.242+0000 7f4807400640 1 -- 172.21.15.27:0/2628595056 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4808052210 con 0x7f481015b4b0 2024-01-15T18:08:43.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:43.434+0000 7f4816400640 1 -- 172.21.15.27:0/2628595056 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone63894", "target": ["mon-mgr", ""]}) v1 -- 0x7f48100cb150 con 0x7f47e8066e00 2024-01-15T18:08:45.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.202+0000 7f4807400640 1 -- 172.21.15.27:0/2628595056 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f48100cb150 con 0x7f47e8066e00 2024-01-15T18:08:45.211 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:08:45.211 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:08:45.211 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "complete" 2024-01-15T18:08:45.211 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:45.211 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:08:45.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.206+0000 7f4816400640 1 -- 172.21.15.27:0/2628595056 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f47e8066e00 msgr2=0x7f47e80692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:45.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.206+0000 7f4816400640 1 --2- 172.21.15.27:0/2628595056 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f47e8066e00 0x7f47e80692c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f48000af000 tx=0x7f4800008040 comp rx=0 tx=0).stop 2024-01-15T18:08:45.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.206+0000 7f4816400640 1 -- 172.21.15.27:0/2628595056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f481015b4b0 msgr2=0x7f48101fa200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:45.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.206+0000 7f4816400640 1 --2- 172.21.15.27:0/2628595056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f481015b4b0 0x7f48101fa200 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f480800eed0 tx=0x7f480800b480 comp rx=0 tx=0).stop 2024-01-15T18:08:45.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.206+0000 7f4816400640 1 -- 172.21.15.27:0/2628595056 shutdown_connections 2024-01-15T18:08:45.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.206+0000 7f4816400640 1 --2- 172.21.15.27:0/2628595056 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f47e8066e00 0x7f47e80692c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.206+0000 7f4816400640 1 --2- 172.21.15.27:0/2628595056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f481015b4b0 0x7f48101fa200 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.206+0000 7f4816400640 1 --2- 172.21.15.27:0/2628595056 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f48100cf460 0x7f48100c6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.206+0000 7f4816400640 1 --2- 172.21.15.27:0/2628595056 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48100cec60 0x7f48100c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.206+0000 7f4816400640 1 -- 172.21.15.27:0/2628595056 >> 172.21.15.27:0/2628595056 conn(0x7f48100d0e80 msgr2=0x7f48100d0010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:45.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.206+0000 7f4816400640 1 -- 172.21.15.27:0/2628595056 shutdown_connections 2024-01-15T18:08:45.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.206+0000 7f4816400640 1 -- 172.21.15.27:0/2628595056 wait complete. 2024-01-15T18:08:45.238 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:08:45.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.378+0000 7f8573a00640 1 Processor -- start 2024-01-15T18:08:45.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.378+0000 7f8573a00640 1 -- start start 2024-01-15T18:08:45.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.378+0000 7f8573a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f856c15ac50 0x7f856c0b2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:45.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.378+0000 7f8573a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f856c0b2ba0 0x7f856c071020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:45.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.378+0000 7f8573a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f856c071560 0x7f856c06c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:45.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.378+0000 7f8573a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f856c070b90 con 0x7f856c0b2ba0 2024-01-15T18:08:45.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.378+0000 7f8573a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f856c072020 con 0x7f856c071560 2024-01-15T18:08:45.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.378+0000 7f8573a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856c072300 con 0x7f856c15ac50 2024-01-15T18:08:45.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.382+0000 7f856be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f856c0b2ba0 0x7f856c071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:45.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.382+0000 7f856be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f856c0b2ba0 0x7f856c071020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38728/0 (socket says 172.21.15.27:38728) 2024-01-15T18:08:45.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.382+0000 7f856be00640 1 -- 172.21.15.27:0/551211180 learned_addr learned my addr 172.21.15.27:0/551211180 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:45.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.382+0000 7f8571000640 1 --2- 172.21.15.27:0/551211180 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f856c15ac50 0x7f856c0b2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:45.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.382+0000 7f8571a00640 1 --2- 172.21.15.27:0/551211180 >> v2:172.21.15.93:3300/0 conn(0x7f856c071560 0x7f856c06c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:45.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.382+0000 7f856be00640 1 -- 172.21.15.27:0/551211180 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f856c15ac50 msgr2=0x7f856c0b2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:45.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.382+0000 7f856be00640 1 --2- 172.21.15.27:0/551211180 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f856c15ac50 0x7f856c0b2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.382+0000 7f856be00640 1 -- 172.21.15.27:0/551211180 >> v2:172.21.15.93:3300/0 conn(0x7f856c071560 msgr2=0x7f856c06c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:45.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.382+0000 7f856be00640 1 --2- 172.21.15.27:0/551211180 >> v2:172.21.15.93:3300/0 conn(0x7f856c071560 0x7f856c06c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.382+0000 7f856be00640 1 -- 172.21.15.27:0/551211180 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856c070820 con 0x7f856c0b2ba0 2024-01-15T18:08:45.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.382+0000 7f856be00640 1 --2- 172.21.15.27:0/551211180 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c0b2ba0 0x7f856c071020 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f856c0b1400 tx=0x7f855401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd9f2e56c4bdc070 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:45.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.382+0000 7f856b400640 1 -- 172.21.15.27:0/551211180 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85540ac630 con 0x7f856c0b2ba0 2024-01-15T18:08:45.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.382+0000 7f856b400640 1 -- 172.21.15.27:0/551211180 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85540ac790 con 0x7f856c0b2ba0 2024-01-15T18:08:45.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.382+0000 7f856b400640 1 -- 172.21.15.27:0/551211180 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85540acb20 con 0x7f856c0b2ba0 2024-01-15T18:08:45.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.386+0000 7f8573a00640 1 -- 172.21.15.27:0/551211180 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c0b2ba0 msgr2=0x7f856c071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:45.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.386+0000 7f8573a00640 1 --2- 172.21.15.27:0/551211180 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c0b2ba0 0x7f856c071020 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f856c0b1400 tx=0x7f855401c500 comp rx=0 tx=0).stop 2024-01-15T18:08:45.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.386+0000 7f8573a00640 1 -- 172.21.15.27:0/551211180 shutdown_connections 2024-01-15T18:08:45.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.386+0000 7f8573a00640 1 --2- 172.21.15.27:0/551211180 >> v2:172.21.15.93:3300/0 conn(0x7f856c071560 0x7f856c06c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.386+0000 7f8573a00640 1 --2- 172.21.15.27:0/551211180 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c0b2ba0 0x7f856c071020 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.386+0000 7f8573a00640 1 --2- 172.21.15.27:0/551211180 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f856c15ac50 0x7f856c0b2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.386+0000 7f8573a00640 1 -- 172.21.15.27:0/551211180 >> 172.21.15.27:0/551211180 conn(0x7f856c0b0850 msgr2=0x7f856c0b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:45.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.386+0000 7f8573a00640 1 -- 172.21.15.27:0/551211180 shutdown_connections 2024-01-15T18:08:45.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.386+0000 7f8573a00640 1 -- 172.21.15.27:0/551211180 wait complete. 2024-01-15T18:08:45.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.386+0000 7f8573a00640 1 Processor -- start 2024-01-15T18:08:45.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.386+0000 7f8573a00640 1 -- start start 2024-01-15T18:08:45.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.386+0000 7f8573a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f856c071560 0x7f856c0ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:45.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.386+0000 7f8573a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c0b2ba0 0x7f856c0ada10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:45.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.386+0000 7f8573a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f856c15ac50 0x7f856c14aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:45.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.386+0000 7f8573a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c156760 con 0x7f856c0b2ba0 2024-01-15T18:08:45.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.386+0000 7f8573a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c14b820 con 0x7f856c15ac50 2024-01-15T18:08:45.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.386+0000 7f8573a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856c14bb00 con 0x7f856c071560 2024-01-15T18:08:45.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.386+0000 7f856be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c0b2ba0 0x7f856c0ada10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:45.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.386+0000 7f8571a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f856c15ac50 0x7f856c14aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:45.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.386+0000 7f856be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c0b2ba0 0x7f856c0ada10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38734/0 (socket says 172.21.15.27:38734) 2024-01-15T18:08:45.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.386+0000 7f856be00640 1 -- 172.21.15.27:0/3293740571 learned_addr learned my addr 172.21.15.27:0/3293740571 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:45.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.386+0000 7f8571a00640 1 --2- 172.21.15.27:0/3293740571 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f856c15ac50 0x7f856c14aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:36788/0 (socket says 172.21.15.27:36788) 2024-01-15T18:08:45.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.386+0000 7f8571000640 1 --2- 172.21.15.27:0/3293740571 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f856c071560 0x7f856c0ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:45.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.390+0000 7f856be00640 1 -- 172.21.15.27:0/3293740571 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f856c071560 msgr2=0x7f856c0ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:45.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.390+0000 7f856be00640 1 --2- 172.21.15.27:0/3293740571 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f856c071560 0x7f856c0ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.390+0000 7f856be00640 1 -- 172.21.15.27:0/3293740571 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f856c15ac50 msgr2=0x7f856c14aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:45.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.390+0000 7f856be00640 1 --2- 172.21.15.27:0/3293740571 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f856c15ac50 0x7f856c14aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.390+0000 7f856be00640 1 -- 172.21.15.27:0/3293740571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85540094d0 con 0x7f856c0b2ba0 2024-01-15T18:08:45.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.390+0000 7f856be00640 1 --2- 172.21.15.27:0/3293740571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c0b2ba0 0x7f856c0ada10 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f855401abb0 tx=0x7f85540192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:45.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.390+0000 7f8568c00640 1 -- 172.21.15.27:0/3293740571 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85540b7620 con 0x7f856c0b2ba0 2024-01-15T18:08:45.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.390+0000 7f8568c00640 1 -- 172.21.15.27:0/3293740571 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85540b77a0 con 0x7f856c0b2ba0 2024-01-15T18:08:45.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.390+0000 7f8573a00640 1 -- 172.21.15.27:0/3293740571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f856c156110 con 0x7f856c0b2ba0 2024-01-15T18:08:45.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.390+0000 7f8568c00640 1 -- 172.21.15.27:0/3293740571 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85540b7b50 con 0x7f856c0b2ba0 2024-01-15T18:08:45.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.390+0000 7f8573a00640 1 -- 172.21.15.27:0/3293740571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f856c146ef0 con 0x7f856c0b2ba0 2024-01-15T18:08:45.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.390+0000 7f8573a00640 1 -- 172.21.15.27:0/3293740571 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f856c04e850 con 0x7f856c0b2ba0 2024-01-15T18:08:45.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.390+0000 7f8568c00640 1 -- 172.21.15.27:0/3293740571 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) v1 ==== 86619+0+0 (secure 0 0 0) 0x7f85540b3040 con 0x7f856c0b2ba0 2024-01-15T18:08:45.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.394+0000 7f8568c00640 1 --2- 172.21.15.27:0/3293740571 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8544066e00 0x7f85440692c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:45.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.394+0000 7f8568c00640 1 -- 172.21.15.27:0/3293740571 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f855411e6d0 con 0x7f856c0b2ba0 2024-01-15T18:08:45.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.394+0000 7f8571000640 1 --2- 172.21.15.27:0/3293740571 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8544066e00 0x7f85440692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:45.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.394+0000 7f8571000640 1 --2- 172.21.15.27:0/3293740571 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8544066e00 0x7f85440692c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f855c0079c0 tx=0x7f855c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:45.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.394+0000 7f8568c00640 1 -- 172.21.15.27:0/3293740571 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f85540eb390 con 0x7f856c0b2ba0 2024-01-15T18:08:45.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.574+0000 7f8573a00640 1 -- 172.21.15.27:0/3293740571 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f856c067c00 con 0x7f8544066e00 2024-01-15T18:08:45.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.574+0000 7f8568c00640 1 -- 172.21.15.27:0/3293740571 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f856c067c00 con 0x7f8544066e00 2024-01-15T18:08:45.581 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:08:45.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:08:45.581 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-15T18:08:45.582 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:45.582 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:08:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.578+0000 7f8573a00640 1 -- 172.21.15.27:0/3293740571 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8544066e00 msgr2=0x7f85440692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.578+0000 7f8573a00640 1 --2- 172.21.15.27:0/3293740571 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8544066e00 0x7f85440692c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f855c0079c0 tx=0x7f855c008040 comp rx=0 tx=0).stop 2024-01-15T18:08:45.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.578+0000 7f8573a00640 1 -- 172.21.15.27:0/3293740571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c0b2ba0 msgr2=0x7f856c0ada10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:45.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.578+0000 7f8573a00640 1 --2- 172.21.15.27:0/3293740571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c0b2ba0 0x7f856c0ada10 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f855401abb0 tx=0x7f85540192a0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.578+0000 7f8573a00640 1 -- 172.21.15.27:0/3293740571 shutdown_connections 2024-01-15T18:08:45.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.578+0000 7f8573a00640 1 --2- 172.21.15.27:0/3293740571 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8544066e00 0x7f85440692c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.578+0000 7f8573a00640 1 --2- 172.21.15.27:0/3293740571 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f856c15ac50 0x7f856c14aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.578+0000 7f8573a00640 1 --2- 172.21.15.27:0/3293740571 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c0b2ba0 0x7f856c0ada10 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.578+0000 7f8573a00640 1 --2- 172.21.15.27:0/3293740571 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f856c071560 0x7f856c0ad4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.578+0000 7f8573a00640 1 -- 172.21.15.27:0/3293740571 >> 172.21.15.27:0/3293740571 conn(0x7f856c0b0850 msgr2=0x7f856c0b1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:45.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.578+0000 7f8573a00640 1 -- 172.21.15.27:0/3293740571 shutdown_connections 2024-01-15T18:08:45.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.578+0000 7f8573a00640 1 -- 172.21.15.27:0/3293740571 wait complete. 2024-01-15T18:08:45.604 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-15T18:08:45.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.742+0000 7fd87aa00640 1 Processor -- start 2024-01-15T18:08:45.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.742+0000 7fd87aa00640 1 -- start start 2024-01-15T18:08:45.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.742+0000 7fd87aa00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fd8741468d0 0x7fd8740ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:45.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.742+0000 7fd87aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd8740cef30 0x7fd8740cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:45.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.742+0000 7fd87aa00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8740ccea0 0x7fd874160cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:45.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.742+0000 7fd87aa00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8740ad920 con 0x7fd8740cef30 2024-01-15T18:08:45.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.742+0000 7fd87aa00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8740ac880 con 0x7fd8741468d0 2024-01-15T18:08:45.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.742+0000 7fd87aa00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8740acb60 con 0x7fd8740ccea0 2024-01-15T18:08:45.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.742+0000 7fd873400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd8740cef30 0x7fd8740cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:45.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.742+0000 7fd873400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd8740cef30 0x7fd8740cc960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38736/0 (socket says 172.21.15.27:38736) 2024-01-15T18:08:45.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.742+0000 7fd873400640 1 -- 172.21.15.27:0/4276633759 learned_addr learned my addr 172.21.15.27:0/4276633759 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:45.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.742+0000 7fd878a00640 1 --2- 172.21.15.27:0/4276633759 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8740ccea0 0x7fd874160cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:45.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.742+0000 7fd873e00640 1 --2- 172.21.15.27:0/4276633759 >> v2:172.21.15.93:3300/0 conn(0x7fd8741468d0 0x7fd8740ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:45.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.742+0000 7fd873400640 1 -- 172.21.15.27:0/4276633759 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8740ccea0 msgr2=0x7fd874160cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:45.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.742+0000 7fd873400640 1 --2- 172.21.15.27:0/4276633759 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8740ccea0 0x7fd874160cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.742+0000 7fd873400640 1 -- 172.21.15.27:0/4276633759 >> v2:172.21.15.93:3300/0 conn(0x7fd8741468d0 msgr2=0x7fd8740ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:45.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.742+0000 7fd873400640 1 --2- 172.21.15.27:0/4276633759 >> v2:172.21.15.93:3300/0 conn(0x7fd8741468d0 0x7fd8740ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.742+0000 7fd873400640 1 -- 172.21.15.27:0/4276633759 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8740ace40 con 0x7fd8740cef30 2024-01-15T18:08:45.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.742+0000 7fd878a00640 1 --2- 172.21.15.27:0/4276633759 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8740ccea0 0x7fd874160cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:08:45.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.746+0000 7fd873400640 1 --2- 172.21.15.27:0/4276633759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8740cef30 0x7fd8740cc960 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fd8740cc650 tx=0x7fd85c01c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29134287ba9d3f19 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:45.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.746+0000 7fd872a00640 1 -- 172.21.15.27:0/4276633759 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0ab600 con 0x7fd8740cef30 2024-01-15T18:08:45.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.746+0000 7fd872a00640 1 -- 172.21.15.27:0/4276633759 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85c0a7050 con 0x7fd8740cef30 2024-01-15T18:08:45.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.746+0000 7fd872a00640 1 -- 172.21.15.27:0/4276633759 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0ab980 con 0x7fd8740cef30 2024-01-15T18:08:45.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.746+0000 7fd87aa00640 1 -- 172.21.15.27:0/4276633759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8740cef30 msgr2=0x7fd8740cc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:45.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.746+0000 7fd87aa00640 1 --2- 172.21.15.27:0/4276633759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8740cef30 0x7fd8740cc960 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fd8740cc650 tx=0x7fd85c01c210 comp rx=0 tx=0).stop 2024-01-15T18:08:45.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.746+0000 7fd87aa00640 1 -- 172.21.15.27:0/4276633759 shutdown_connections 2024-01-15T18:08:45.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.746+0000 7fd87aa00640 1 --2- 172.21.15.27:0/4276633759 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8740ccea0 0x7fd874160cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.746+0000 7fd87aa00640 1 --2- 172.21.15.27:0/4276633759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8740cef30 0x7fd8740cc960 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.746+0000 7fd87aa00640 1 --2- 172.21.15.27:0/4276633759 >> v2:172.21.15.93:3300/0 conn(0x7fd8741468d0 0x7fd8740ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.746+0000 7fd87aa00640 1 -- 172.21.15.27:0/4276633759 >> 172.21.15.27:0/4276633759 conn(0x7fd8740cb700 msgr2=0x7fd8740cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:45.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.746+0000 7fd87aa00640 1 -- 172.21.15.27:0/4276633759 shutdown_connections 2024-01-15T18:08:45.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.746+0000 7fd87aa00640 1 -- 172.21.15.27:0/4276633759 wait complete. 2024-01-15T18:08:45.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.750+0000 7fd87aa00640 1 Processor -- start 2024-01-15T18:08:45.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.750+0000 7fd87aa00640 1 -- start start 2024-01-15T18:08:45.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.750+0000 7fd87aa00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd8740ccea0 0x7fd8741586f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:45.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.750+0000 7fd87aa00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8740cef30 0x7fd87413fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:45.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.750+0000 7fd87aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8741468d0 0x7fd87413d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:45.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.750+0000 7fd87aa00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd874070e70 con 0x7fd8741468d0 2024-01-15T18:08:45.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.750+0000 7fd87aa00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd874071710 con 0x7fd8740ccea0 2024-01-15T18:08:45.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.750+0000 7fd87aa00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd87413e3e0 con 0x7fd8740cef30 2024-01-15T18:08:45.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.750+0000 7fd878a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8741468d0 0x7fd87413d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:45.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.750+0000 7fd878a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8741468d0 0x7fd87413d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38746/0 (socket says 172.21.15.27:38746) 2024-01-15T18:08:45.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.750+0000 7fd878a00640 1 -- 172.21.15.27:0/2913984542 learned_addr learned my addr 172.21.15.27:0/2913984542 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:45.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.750+0000 7fd873400640 1 --2- 172.21.15.27:0/2913984542 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8740cef30 0x7fd87413fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:45.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.750+0000 7fd873e00640 1 --2- 172.21.15.27:0/2913984542 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd8740ccea0 0x7fd8741586f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:45.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.750+0000 7fd878a00640 1 -- 172.21.15.27:0/2913984542 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8740cef30 msgr2=0x7fd87413fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:45.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.750+0000 7fd878a00640 1 --2- 172.21.15.27:0/2913984542 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8740cef30 0x7fd87413fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.750+0000 7fd878a00640 1 -- 172.21.15.27:0/2913984542 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd8740ccea0 msgr2=0x7fd8741586f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:45.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.750+0000 7fd878a00640 1 --2- 172.21.15.27:0/2913984542 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd8740ccea0 0x7fd8741586f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.750+0000 7fd878a00640 1 -- 172.21.15.27:0/2913984542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd85c0094d0 con 0x7fd8741468d0 2024-01-15T18:08:45.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.750+0000 7fd878a00640 1 --2- 172.21.15.27:0/2913984542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8741468d0 0x7fd87413d3d0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd86c007e70 tx=0x7fd86c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:45.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.750+0000 7fd86be00640 1 -- 172.21.15.27:0/2913984542 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c003ba0 con 0x7fd8741468d0 2024-01-15T18:08:45.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.750+0000 7fd86be00640 1 -- 172.21.15.27:0/2913984542 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86c00f040 con 0x7fd8741468d0 2024-01-15T18:08:45.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.750+0000 7fd86be00640 1 -- 172.21.15.27:0/2913984542 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c00de80 con 0x7fd8741468d0 2024-01-15T18:08:45.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.750+0000 7fd87aa00640 1 -- 172.21.15.27:0/2913984542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8740710c0 con 0x7fd8741468d0 2024-01-15T18:08:45.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.750+0000 7fd87aa00640 1 -- 172.21.15.27:0/2913984542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd874071ba0 con 0x7fd8741468d0 2024-01-15T18:08:45.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.754+0000 7fd87aa00640 1 -- 172.21.15.27:0/2913984542 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd874071880 con 0x7fd8741468d0 2024-01-15T18:08:45.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.754+0000 7fd86be00640 1 -- 172.21.15.27:0/2913984542 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fd86c004030 con 0x7fd8741468d0 2024-01-15T18:08:45.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.754+0000 7fd86be00640 1 --2- 172.21.15.27:0/2913984542 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd84c066e30 0x7fd84c0692f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:45.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.754+0000 7fd86be00640 1 -- 172.21.15.27:0/2913984542 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd86c084f80 con 0x7fd8741468d0 2024-01-15T18:08:45.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.754+0000 7fd873e00640 1 --2- 172.21.15.27:0/2913984542 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd84c066e30 0x7fd84c0692f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:45.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.754+0000 7fd873e00640 1 --2- 172.21.15.27:0/2913984542 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd84c066e30 0x7fd84c0692f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd8640079e0 tx=0x7fd864008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:45.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.758+0000 7fd86be00640 1 -- 172.21.15.27:0/2913984542 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd86c051b00 con 0x7fd8741468d0 2024-01-15T18:08:45.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.938+0000 7fd87aa00640 1 -- 172.21.15.27:0/2913984542 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd874146cb0 con 0x7fd84c066e30 2024-01-15T18:08:45.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.946+0000 7fd86be00640 1 -- 172.21.15.27:0/2913984542 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+234 (secure 0 0 0) 0x7fd8640108d0 con 0x7fd84c066e30 2024-01-15T18:08:45.951 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:08:45.951 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:08:45.951 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:08:45.951 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone63893" 2024-01-15T18:08:45.951 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:45.951 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:08:45.951 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sv1" 2024-01-15T18:08:45.951 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:45.951 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:08:45.951 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone63892" 2024-01-15T18:08:45.951 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:45.952 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:08:45.952 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone63891" 2024-01-15T18:08:45.952 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:08:45.952 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:08:45.952 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone63894" 2024-01-15T18:08:45.952 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:45.952 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:08:45.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.946+0000 7fd87aa00640 1 -- 172.21.15.27:0/2913984542 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd84c066e30 msgr2=0x7fd84c0692f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:45.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.946+0000 7fd87aa00640 1 --2- 172.21.15.27:0/2913984542 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd84c066e30 0x7fd84c0692f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd8640079e0 tx=0x7fd864008040 comp rx=0 tx=0).stop 2024-01-15T18:08:45.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.946+0000 7fd87aa00640 1 -- 172.21.15.27:0/2913984542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8741468d0 msgr2=0x7fd87413d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:45.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.946+0000 7fd87aa00640 1 --2- 172.21.15.27:0/2913984542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8741468d0 0x7fd87413d3d0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd86c007e70 tx=0x7fd86c00d4a0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.946+0000 7fd87aa00640 1 -- 172.21.15.27:0/2913984542 shutdown_connections 2024-01-15T18:08:45.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.946+0000 7fd87aa00640 1 --2- 172.21.15.27:0/2913984542 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd84c066e30 0x7fd84c0692f0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.946+0000 7fd87aa00640 1 --2- 172.21.15.27:0/2913984542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8741468d0 0x7fd87413d3d0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.946+0000 7fd87aa00640 1 --2- 172.21.15.27:0/2913984542 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8740cef30 0x7fd87413fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.946+0000 7fd87aa00640 1 --2- 172.21.15.27:0/2913984542 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd8740ccea0 0x7fd8741586f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:45.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.946+0000 7fd87aa00640 1 -- 172.21.15.27:0/2913984542 >> 172.21.15.27:0/2913984542 conn(0x7fd8740cb700 msgr2=0x7fd8740b12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:45.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.946+0000 7fd87aa00640 1 -- 172.21.15.27:0/2913984542 shutdown_connections 2024-01-15T18:08:45.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:45.946+0000 7fd87aa00640 1 -- 172.21.15.27:0/2913984542 wait complete. 2024-01-15T18:08:45.974 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone63893 2024-01-15T18:08:46.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.114+0000 7fa104c00640 1 Processor -- start 2024-01-15T18:08:46.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.114+0000 7fa104c00640 1 -- start start 2024-01-15T18:08:46.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.114+0000 7fa104c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa10006c520 0x7fa1000cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:46.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.114+0000 7fa104c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1000b0d70 con 0x7fa10015d190 2024-01-15T18:08:46.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.114+0000 7fa104c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1000d0540 con 0x7fa10015d570 2024-01-15T18:08:46.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.114+0000 7fa104c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1000b0440 con 0x7fa10006c520 2024-01-15T18:08:46.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.114+0000 7fa0fde00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa10015d190 0x7fa1000cfcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55160/0 (socket says 172.21.15.27:55160) 2024-01-15T18:08:46.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.114+0000 7fa0fde00640 1 -- 172.21.15.27:0/1113021590 learned_addr learned my addr 172.21.15.27:0/1113021590 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:46.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.114+0000 7fa0fe800640 1 --2- 172.21.15.27:0/1113021590 >> v2:172.21.15.93:3300/0 conn(0x7fa10006c520 0x7fa1000cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:46.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.114+0000 7fa0fca00640 1 -- 172.21.15.27:0/1113021590 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908812983 0 0) 0x7fa1000b0d70 con 0x7fa10015d190 2024-01-15T18:08:46.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.114+0000 7fa0fca00640 1 -- 172.21.15.27:0/1113021590 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0e0003530 con 0x7fa10015d190 2024-01-15T18:08:46.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.114+0000 7fa0fca00640 1 -- 172.21.15.27:0/1113021590 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210443684 0 0) 0x7fa1000d0540 con 0x7fa10015d570 2024-01-15T18:08:46.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.114+0000 7fa0fca00640 1 -- 172.21.15.27:0/1113021590 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1000b0d70 con 0x7fa10015d570 2024-01-15T18:08:46.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.114+0000 7fa0fca00640 1 -- 172.21.15.27:0/1113021590 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 179409486 0 0) 0x7fa0e0003530 con 0x7fa10015d190 2024-01-15T18:08:46.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.114+0000 7fa0fe800640 1 -- 172.21.15.27:0/1113021590 >> v1:172.21.15.93:6789/0 conn(0x7fa10015d570 legacy=0x7fa10006be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:46.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.114+0000 7fa0fe800640 1 -- 172.21.15.27:0/1113021590 >> v1:172.21.15.27:6789/0 conn(0x7fa10015d190 legacy=0x7fa1000cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:46.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.114+0000 7fa0fe800640 1 -- 172.21.15.27:0/1113021590 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1000b0720 con 0x7fa10006c520 2024-01-15T18:08:46.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.118+0000 7fa0fe800640 1 --2- 172.21.15.27:0/1113021590 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa10006c520 0x7fa1000cd5e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8009d90 tx=0x7fa0f801c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5d2ab257c7d04bf server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:46.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.118+0000 7fa0fca00640 1 -- 172.21.15.27:0/1113021590 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0001680 con 0x7fa10006c520 2024-01-15T18:08:46.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.118+0000 7fa0fca00640 1 -- 172.21.15.27:0/1113021590 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f80a56a0 con 0x7fa10006c520 2024-01-15T18:08:46.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.118+0000 7fa0fca00640 1 -- 172.21.15.27:0/1113021590 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f80a5a30 con 0x7fa10006c520 2024-01-15T18:08:46.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.118+0000 7fa104c00640 1 -- 172.21.15.27:0/1113021590 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa10006c520 msgr2=0x7fa1000cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:46.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.118+0000 7fa104c00640 1 --2- 172.21.15.27:0/1113021590 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa10006c520 0x7fa1000cd5e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8009d90 tx=0x7fa0f801c840 comp rx=0 tx=0).stop 2024-01-15T18:08:46.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.118+0000 7fa104c00640 1 -- 172.21.15.27:0/1113021590 shutdown_connections 2024-01-15T18:08:46.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.118+0000 7fa104c00640 1 --2- 172.21.15.27:0/1113021590 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa10006c520 0x7fa1000cd5e0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:46.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.118+0000 7fa104c00640 1 -- 172.21.15.27:0/1113021590 >> 172.21.15.27:0/1113021590 conn(0x7fa1000d0e70 msgr2=0x7fa1000d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:46.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.118+0000 7fa104c00640 1 -- 172.21.15.27:0/1113021590 shutdown_connections 2024-01-15T18:08:46.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.118+0000 7fa104c00640 1 -- 172.21.15.27:0/1113021590 wait complete. 2024-01-15T18:08:46.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.122+0000 7fa104c00640 1 Processor -- start 2024-01-15T18:08:46.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.122+0000 7fa104c00640 1 -- start start 2024-01-15T18:08:46.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.122+0000 7fa104c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa10006c520 0x7fa100158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:46.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.122+0000 7fa104c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa10015d190 0x7fa10015b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:46.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.122+0000 7fa104c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa10015d570 0x7fa100155070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:46.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.122+0000 7fa104c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10015ab80 con 0x7fa10006c520 2024-01-15T18:08:46.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.122+0000 7fa104c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10015ae60 con 0x7fa10015d190 2024-01-15T18:08:46.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.122+0000 7fa104c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa10015b140 con 0x7fa10015d570 2024-01-15T18:08:46.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.122+0000 7fa0fde00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa10006c520 0x7fa100158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:46.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.122+0000 7fa0fde00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa10006c520 0x7fa100158160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38758/0 (socket says 172.21.15.27:38758) 2024-01-15T18:08:46.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.122+0000 7fa0fde00640 1 -- 172.21.15.27:0/3410947578 learned_addr learned my addr 172.21.15.27:0/3410947578 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:46.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.122+0000 7fa0fe800640 1 --2- 172.21.15.27:0/3410947578 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa10015d570 0x7fa100155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:46.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.122+0000 7fa0fd400640 1 --2- 172.21.15.27:0/3410947578 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa10015d190 0x7fa10015b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:46.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.122+0000 7fa0fde00640 1 -- 172.21.15.27:0/3410947578 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa10015d570 msgr2=0x7fa100155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:46.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.122+0000 7fa0fde00640 1 --2- 172.21.15.27:0/3410947578 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa10015d570 0x7fa100155070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:46.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.122+0000 7fa0fde00640 1 -- 172.21.15.27:0/3410947578 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa10015d190 msgr2=0x7fa10015b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:46.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.122+0000 7fa0fde00640 1 --2- 172.21.15.27:0/3410947578 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa10015d190 0x7fa10015b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:46.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.122+0000 7fa0fde00640 1 -- 172.21.15.27:0/3410947578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f8009a70 con 0x7fa10006c520 2024-01-15T18:08:46.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.122+0000 7fa0fde00640 1 --2- 172.21.15.27:0/3410947578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa10006c520 0x7fa100158160 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fa0f000c750 tx=0x7fa0f000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:46.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.122+0000 7fa0f6000640 1 -- 172.21.15.27:0/3410947578 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0005af0 con 0x7fa10006c520 2024-01-15T18:08:46.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.122+0000 7fa0f6000640 1 -- 172.21.15.27:0/3410947578 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f0005c50 con 0x7fa10006c520 2024-01-15T18:08:46.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.122+0000 7fa104c00640 1 -- 172.21.15.27:0/3410947578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa10015b420 con 0x7fa10006c520 2024-01-15T18:08:46.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.122+0000 7fa0f6000640 1 -- 172.21.15.27:0/3410947578 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0007070 con 0x7fa10006c520 2024-01-15T18:08:46.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.122+0000 7fa104c00640 1 -- 172.21.15.27:0/3410947578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa10015be80 con 0x7fa10006c520 2024-01-15T18:08:46.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.126+0000 7fa104c00640 1 -- 172.21.15.27:0/3410947578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa10004e850 con 0x7fa10006c520 2024-01-15T18:08:46.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.126+0000 7fa0f6000640 1 -- 172.21.15.27:0/3410947578 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fa0f00071d0 con 0x7fa10006c520 2024-01-15T18:08:46.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.126+0000 7fa0f6000640 1 --2- 172.21.15.27:0/3410947578 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0d8066e00 0x7fa0d80692c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:46.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.126+0000 7fa0f6000640 1 -- 172.21.15.27:0/3410947578 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa0f0084a40 con 0x7fa10006c520 2024-01-15T18:08:46.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.126+0000 7fa0fd400640 1 --2- 172.21.15.27:0/3410947578 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0d8066e00 0x7fa0d80692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:46.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.126+0000 7fa0fd400640 1 --2- 172.21.15.27:0/3410947578 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0d8066e00 0x7fa0d80692c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa1000cec90 tx=0x7fa0e8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:46.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.130+0000 7fa0f6000640 1 -- 172.21.15.27:0/3410947578 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa0f0051660 con 0x7fa10006c520 2024-01-15T18:08:46.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.310+0000 7fa104c00640 1 -- 172.21.15.27:0/3410947578 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone63893", "target": ["mon-mgr", ""]}) v1 -- 0x7fa10015acc0 con 0x7fa0d8066e00 2024-01-15T18:08:46.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.330+0000 7fa0f6000640 1 -- 172.21.15.27:0/3410947578 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa10015acc0 con 0x7fa0d8066e00 2024-01-15T18:08:46.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.330+0000 7fa104c00640 1 -- 172.21.15.27:0/3410947578 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0d8066e00 msgr2=0x7fa0d80692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:46.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.330+0000 7fa104c00640 1 --2- 172.21.15.27:0/3410947578 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0d8066e00 0x7fa0d80692c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa1000cec90 tx=0x7fa0e8009040 comp rx=0 tx=0).stop 2024-01-15T18:08:46.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.330+0000 7fa104c00640 1 -- 172.21.15.27:0/3410947578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa10006c520 msgr2=0x7fa100158160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:46.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.330+0000 7fa104c00640 1 --2- 172.21.15.27:0/3410947578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa10006c520 0x7fa100158160 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fa0f000c750 tx=0x7fa0f000cc10 comp rx=0 tx=0).stop 2024-01-15T18:08:46.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.330+0000 7fa104c00640 1 -- 172.21.15.27:0/3410947578 shutdown_connections 2024-01-15T18:08:46.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.330+0000 7fa104c00640 1 --2- 172.21.15.27:0/3410947578 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0d8066e00 0x7fa0d80692c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:46.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.330+0000 7fa104c00640 1 --2- 172.21.15.27:0/3410947578 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa10015d570 0x7fa100155070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:46.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.330+0000 7fa104c00640 1 --2- 172.21.15.27:0/3410947578 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa10015d190 0x7fa10015b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:46.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.330+0000 7fa104c00640 1 --2- 172.21.15.27:0/3410947578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa10006c520 0x7fa100158160 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:46.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.330+0000 7fa104c00640 1 -- 172.21.15.27:0/3410947578 >> 172.21.15.27:0/3410947578 conn(0x7fa1000d0e70 msgr2=0x7fa1000adf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:46.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.330+0000 7fa104c00640 1 -- 172.21.15.27:0/3410947578 shutdown_connections 2024-01-15T18:08:46.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.330+0000 7fa104c00640 1 -- 172.21.15.27:0/3410947578 wait complete. 2024-01-15T18:08:46.339 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:46.334+0000 7f59f4c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:46.340 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:46.334+0000 7f59f4c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:46.340 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:46.334+0000 7f59f4c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:46.340 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:46.334+0000 7f59f4c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:46.340 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:46.334+0000 7f59f4c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:46.356 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:46.350+0000 7f59f6000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:46.356 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:46.350+0000 7f59f6000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:46.356 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:46.350+0000 7f59f6000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:46.356 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:46.350+0000 7f59f6000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:46.356 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:08:46.350+0000 7f59f6000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:46.360 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-15T18:08:46.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.502+0000 7fac11e00640 1 Processor -- start 2024-01-15T18:08:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.502+0000 7fac11e00640 1 -- start start 2024-01-15T18:08:46.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.506+0000 7fac11e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac0c099a10 0x7fac0c099e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.506+0000 7fac11e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac0c09a3a0 con 0x7fac0c15b350 2024-01-15T18:08:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.506+0000 7fac11e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac0c0ccd50 con 0x7fac0c09a5b0 2024-01-15T18:08:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.506+0000 7fac11e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c0cebc0 con 0x7fac0c099a10 2024-01-15T18:08:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.506+0000 7fac0aa00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fac0c15b350 0x7fac0c0cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55164/0 (socket says 172.21.15.27:55164) 2024-01-15T18:08:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.506+0000 7fac0aa00640 1 -- 172.21.15.27:0/2499657091 learned_addr learned my addr 172.21.15.27:0/2499657091 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:46.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.506+0000 7fac0b400640 1 --2- 172.21.15.27:0/2499657091 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac0c099a10 0x7fac0c099e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:46.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.506+0000 7fac09600640 1 -- 172.21.15.27:0/2499657091 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426569599 0 0) 0x7fabfc001120 con 0x7fac0c15b350 2024-01-15T18:08:46.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.506+0000 7fac09600640 1 -- 172.21.15.27:0/2499657091 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabec003530 con 0x7fac0c15b350 2024-01-15T18:08:46.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.506+0000 7fac09600640 1 -- 172.21.15.27:0/2499657091 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21205774 0 0) 0x7fac0c0ccd50 con 0x7fac0c09a5b0 2024-01-15T18:08:46.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.506+0000 7fac09600640 1 -- 172.21.15.27:0/2499657091 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabfc001120 con 0x7fac0c09a5b0 2024-01-15T18:08:46.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.506+0000 7fac0b400640 1 -- 172.21.15.27:0/2499657091 >> v1:172.21.15.93:6789/0 conn(0x7fac0c09a5b0 legacy=0x7fac0c0cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:46.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.506+0000 7fac0b400640 1 -- 172.21.15.27:0/2499657091 >> v1:172.21.15.27:6789/0 conn(0x7fac0c15b350 legacy=0x7fac0c0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:46.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.506+0000 7fac0b400640 1 -- 172.21.15.27:0/2499657091 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0c0ceea0 con 0x7fac0c099a10 2024-01-15T18:08:46.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.506+0000 7fac0b400640 1 --2- 172.21.15.27:0/2499657091 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac0c099a10 0x7fac0c099e60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fac04009dc0 tx=0x7fac0401c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f5fb06e68f2e375 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:46.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.510+0000 7fac09600640 1 -- 172.21.15.27:0/2499657091 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac040a5990 con 0x7fac0c099a10 2024-01-15T18:08:46.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.510+0000 7fac09600640 1 -- 172.21.15.27:0/2499657091 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac040a5af0 con 0x7fac0c099a10 2024-01-15T18:08:46.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.510+0000 7fac09600640 1 -- 172.21.15.27:0/2499657091 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac040a5e80 con 0x7fac0c099a10 2024-01-15T18:08:46.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.510+0000 7fac11e00640 1 -- 172.21.15.27:0/2499657091 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac0c099a10 msgr2=0x7fac0c099e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:46.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.510+0000 7fac11e00640 1 --2- 172.21.15.27:0/2499657091 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac0c099a10 0x7fac0c099e60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fac04009dc0 tx=0x7fac0401c8b0 comp rx=0 tx=0).stop 2024-01-15T18:08:46.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.510+0000 7fac11e00640 1 -- 172.21.15.27:0/2499657091 shutdown_connections 2024-01-15T18:08:46.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.510+0000 7fac11e00640 1 --2- 172.21.15.27:0/2499657091 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac0c099a10 0x7fac0c099e60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:46.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.510+0000 7fac11e00640 1 -- 172.21.15.27:0/2499657091 >> 172.21.15.27:0/2499657091 conn(0x7fac0c0b2a80 msgr2=0x7fac0c0b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:46.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.510+0000 7fac11e00640 1 -- 172.21.15.27:0/2499657091 shutdown_connections 2024-01-15T18:08:46.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.510+0000 7fac11e00640 1 -- 172.21.15.27:0/2499657091 wait complete. 2024-01-15T18:08:46.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.510+0000 7fac11e00640 1 Processor -- start 2024-01-15T18:08:46.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.510+0000 7fac11e00640 1 -- start start 2024-01-15T18:08:46.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.510+0000 7fac11e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fac0c099a10 0x7fac0c14c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:46.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.514+0000 7fac11e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac0c09a5b0 0x7fac0c15ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:46.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.514+0000 7fac11e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac0c15b350 0x7fac0c15b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:46.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.514+0000 7fac11e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c15da40 con 0x7fac0c09a5b0 2024-01-15T18:08:46.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.514+0000 7fac11e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c15c1f0 con 0x7fac0c099a10 2024-01-15T18:08:46.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.514+0000 7fac11e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c15c4d0 con 0x7fac0c15b350 2024-01-15T18:08:46.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.514+0000 7fac0a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac0c09a5b0 0x7fac0c15ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:46.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.514+0000 7fac0aa00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fac0c099a10 0x7fac0c14c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:46.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.514+0000 7fac0a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac0c09a5b0 0x7fac0c15ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38770/0 (socket says 172.21.15.27:38770) 2024-01-15T18:08:46.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.514+0000 7fac0a000640 1 -- 172.21.15.27:0/714934329 learned_addr learned my addr 172.21.15.27:0/714934329 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:46.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.514+0000 7fac0b400640 1 --2- 172.21.15.27:0/714934329 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac0c15b350 0x7fac0c15b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:46.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.514+0000 7fac0a000640 1 -- 172.21.15.27:0/714934329 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac0c15b350 msgr2=0x7fac0c15b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:46.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.514+0000 7fac0a000640 1 --2- 172.21.15.27:0/714934329 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac0c15b350 0x7fac0c15b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:46.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.514+0000 7fac0a000640 1 -- 172.21.15.27:0/714934329 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fac0c099a10 msgr2=0x7fac0c14c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:46.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.514+0000 7fac0a000640 1 --2- 172.21.15.27:0/714934329 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fac0c099a10 0x7fac0c14c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:46.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.514+0000 7fac0a000640 1 -- 172.21.15.27:0/714934329 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac04009a70 con 0x7fac0c09a5b0 2024-01-15T18:08:46.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.514+0000 7fac0a000640 1 --2- 172.21.15.27:0/714934329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac0c09a5b0 0x7fac0c15ded0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fabf400c490 tx=0x7fabf400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:46.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.514+0000 7fac03400640 1 -- 172.21.15.27:0/714934329 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf40052d0 con 0x7fac0c09a5b0 2024-01-15T18:08:46.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.514+0000 7fac03400640 1 -- 172.21.15.27:0/714934329 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf4005430 con 0x7fac0c09a5b0 2024-01-15T18:08:46.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.514+0000 7fac03400640 1 -- 172.21.15.27:0/714934329 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf4012870 con 0x7fac0c09a5b0 2024-01-15T18:08:46.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.514+0000 7fac11e00640 1 -- 172.21.15.27:0/714934329 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac0c15d6d0 con 0x7fac0c09a5b0 2024-01-15T18:08:46.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.514+0000 7fac11e00640 1 -- 172.21.15.27:0/714934329 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac0c15e410 con 0x7fac0c09a5b0 2024-01-15T18:08:46.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.514+0000 7fac11e00640 1 -- 172.21.15.27:0/714934329 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac0c15dbb0 con 0x7fac0c09a5b0 2024-01-15T18:08:46.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.514+0000 7fac03400640 1 -- 172.21.15.27:0/714934329 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) v1 ==== 86619+0+0 (secure 0 0 0) 0x7fabf40129d0 con 0x7fac0c09a5b0 2024-01-15T18:08:46.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.518+0000 7fac03400640 1 --2- 172.21.15.27:0/714934329 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fabe4066e00 0x7fabe40692c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:46.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.518+0000 7fac03400640 1 -- 172.21.15.27:0/714934329 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fabf40846f0 con 0x7fac0c09a5b0 2024-01-15T18:08:46.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.518+0000 7fac0aa00640 1 --2- 172.21.15.27:0/714934329 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fabe4066e00 0x7fabe40692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:46.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.518+0000 7fac0aa00640 1 --2- 172.21.15.27:0/714934329 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fabe4066e00 0x7fabe40692c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fabfc007fd0 tx=0x7fabfc007d60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:46.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.522+0000 7fac03400640 1 -- 172.21.15.27:0/714934329 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fabf4051330 con 0x7fac0c09a5b0 2024-01-15T18:08:46.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:46.702+0000 7fac11e00640 1 -- 172.21.15.27:0/714934329 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fac0c0c4840 con 0x7fabe4066e00 2024-01-15T18:08:48.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:48.962+0000 7fac03400640 1 -- 172.21.15.27:0/714934329 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 13) v1 ==== 86721+0+0 (secure 0 0 0) 0x7fabf4051700 con 0x7fac0c09a5b0 2024-01-15T18:08:50.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.202+0000 7fac03400640 1 -- 172.21.15.27:0/714934329 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fac0c0c4840 con 0x7fabe4066e00 2024-01-15T18:08:50.208 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:08:50.208 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:08:50.208 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:08:50.209 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ss1" 2024-01-15T18:08:50.209 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:08:50.209 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:08:50.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.202+0000 7fac11e00640 1 -- 172.21.15.27:0/714934329 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fabe4066e00 msgr2=0x7fabe40692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:50.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.202+0000 7fac11e00640 1 --2- 172.21.15.27:0/714934329 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fabe4066e00 0x7fabe40692c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fabfc007fd0 tx=0x7fabfc007d60 comp rx=0 tx=0).stop 2024-01-15T18:08:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.202+0000 7fac11e00640 1 -- 172.21.15.27:0/714934329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac0c09a5b0 msgr2=0x7fac0c15ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.202+0000 7fac11e00640 1 --2- 172.21.15.27:0/714934329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac0c09a5b0 0x7fac0c15ded0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fabf400c490 tx=0x7fabf400c950 comp rx=0 tx=0).stop 2024-01-15T18:08:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.202+0000 7fac11e00640 1 -- 172.21.15.27:0/714934329 shutdown_connections 2024-01-15T18:08:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.202+0000 7fac11e00640 1 --2- 172.21.15.27:0/714934329 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fabe4066e00 0x7fabe40692c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.202+0000 7fac11e00640 1 --2- 172.21.15.27:0/714934329 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac0c15b350 0x7fac0c15b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.202+0000 7fac11e00640 1 --2- 172.21.15.27:0/714934329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac0c09a5b0 0x7fac0c15ded0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.202+0000 7fac11e00640 1 --2- 172.21.15.27:0/714934329 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fac0c099a10 0x7fac0c14c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.206+0000 7fac11e00640 1 -- 172.21.15.27:0/714934329 >> 172.21.15.27:0/714934329 conn(0x7fac0c0b2a80 msgr2=0x7fac0c159b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.206+0000 7fac11e00640 1 -- 172.21.15.27:0/714934329 shutdown_connections 2024-01-15T18:08:50.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.206+0000 7fac11e00640 1 -- 172.21.15.27:0/714934329 wait complete. 2024-01-15T18:08:50.233 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:08:50.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.378+0000 7f9bbca00640 1 Processor -- start 2024-01-15T18:08:50.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.378+0000 7f9bbca00640 1 -- start start 2024-01-15T18:08:50.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.378+0000 7f9bbca00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9bb40cf450 0x7f9bb40d1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:50.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.378+0000 7f9bbca00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f9bb40b0650 0x7f9bb40b0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:50.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.378+0000 7f9bbca00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bb40cefa0 con 0x7f9bb40cba60 2024-01-15T18:08:50.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.378+0000 7f9bbca00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bb40ce240 con 0x7f9bb40b0650 2024-01-15T18:08:50.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.378+0000 7f9bbca00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb40ccd70 con 0x7f9bb40cf450 2024-01-15T18:08:50.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.378+0000 7f9bb9600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9bb40cba60 0x7f9bb40cf830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55172/0 (socket says 172.21.15.27:55172) 2024-01-15T18:08:50.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.378+0000 7f9bb9600640 1 -- 172.21.15.27:0/1363761841 learned_addr learned my addr 172.21.15.27:0/1363761841 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:50.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.378+0000 7f9bbaa00640 1 --2- 172.21.15.27:0/1363761841 >> v2:172.21.15.93:3300/0 conn(0x7f9bb40b0650 0x7f9bb40b0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:50.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.378+0000 7f9bba000640 1 --2- 172.21.15.27:0/1363761841 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9bb40cf450 0x7f9bb40d1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:50.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.378+0000 7f9bb8c00640 1 -- 172.21.15.27:0/1363761841 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805582983 0 0) 0x7f9bb40cefa0 con 0x7f9bb40cba60 2024-01-15T18:08:50.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.378+0000 7f9bb8c00640 1 -- 172.21.15.27:0/1363761841 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b94003530 con 0x7f9bb40cba60 2024-01-15T18:08:50.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.378+0000 7f9bbaa00640 1 -- 172.21.15.27:0/1363761841 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9bb40cf450 msgr2=0x7f9bb40d1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:50.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.378+0000 7f9bbaa00640 1 --2- 172.21.15.27:0/1363761841 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9bb40cf450 0x7f9bb40d1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.378+0000 7f9bbaa00640 1 -- 172.21.15.27:0/1363761841 >> v1:172.21.15.27:6789/0 conn(0x7f9bb40cba60 legacy=0x7f9bb40cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:50.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.378+0000 7f9bbaa00640 1 -- 172.21.15.27:0/1363761841 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb40cc1f0 con 0x7f9bb40b0650 2024-01-15T18:08:50.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.378+0000 7f9bba000640 1 --2- 172.21.15.27:0/1363761841 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9bb40cf450 0x7f9bb40d1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:08:50.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.382+0000 7f9bbaa00640 1 --2- 172.21.15.27:0/1363761841 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9bb40b0650 0x7f9bb40b0ae0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9bac009d90 tx=0x7f9bac01c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d50bed22afdf49b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:50.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.382+0000 7f9bb8c00640 1 -- 172.21.15.27:0/1363761841 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bac0a5900 con 0x7f9bb40b0650 2024-01-15T18:08:50.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.382+0000 7f9bb8c00640 1 -- 172.21.15.27:0/1363761841 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bac0a5a60 con 0x7f9bb40b0650 2024-01-15T18:08:50.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.382+0000 7f9bb8c00640 1 -- 172.21.15.27:0/1363761841 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bac0a5df0 con 0x7f9bb40b0650 2024-01-15T18:08:50.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.382+0000 7f9bbca00640 1 -- 172.21.15.27:0/1363761841 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9bb40b0650 msgr2=0x7f9bb40b0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:50.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.382+0000 7f9bbca00640 1 --2- 172.21.15.27:0/1363761841 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9bb40b0650 0x7f9bb40b0ae0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9bac009d90 tx=0x7f9bac01c950 comp rx=0 tx=0).stop 2024-01-15T18:08:50.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.382+0000 7f9bbca00640 1 -- 172.21.15.27:0/1363761841 shutdown_connections 2024-01-15T18:08:50.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.382+0000 7f9bbca00640 1 --2- 172.21.15.27:0/1363761841 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9bb40b0650 0x7f9bb40b0ae0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.382+0000 7f9bbca00640 1 --2- 172.21.15.27:0/1363761841 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9bb40cf450 0x7f9bb40d1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.382+0000 7f9bbca00640 1 -- 172.21.15.27:0/1363761841 >> 172.21.15.27:0/1363761841 conn(0x7f9bb40d0e70 msgr2=0x7f9bb40d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:50.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.382+0000 7f9bbca00640 1 -- 172.21.15.27:0/1363761841 shutdown_connections 2024-01-15T18:08:50.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.382+0000 7f9bbca00640 1 -- 172.21.15.27:0/1363761841 wait complete. 2024-01-15T18:08:50.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.386+0000 7f9bbca00640 1 Processor -- start 2024-01-15T18:08:50.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.386+0000 7f9bbca00640 1 -- start start 2024-01-15T18:08:50.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.386+0000 7f9bbca00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9bb40b0650 0x7f9bb414d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:50.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.386+0000 7f9bbca00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9bb40cba60 0x7f9bb40c9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:50.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.386+0000 7f9bbca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bb40cf450 0x7f9bb40b5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:50.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.386+0000 7f9bbca00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb40c8f30 con 0x7f9bb40cf450 2024-01-15T18:08:50.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.386+0000 7f9bbca00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb40c82b0 con 0x7f9bb40cba60 2024-01-15T18:08:50.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.386+0000 7f9bbca00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb40c8590 con 0x7f9bb40b0650 2024-01-15T18:08:50.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.386+0000 7f9bbaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bb40cf450 0x7f9bb40b5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:50.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.386+0000 7f9bbaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bb40cf450 0x7f9bb40b5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38784/0 (socket says 172.21.15.27:38784) 2024-01-15T18:08:50.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.386+0000 7f9bbaa00640 1 -- 172.21.15.27:0/1572956057 learned_addr learned my addr 172.21.15.27:0/1572956057 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:50.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.386+0000 7f9bba000640 1 --2- 172.21.15.27:0/1572956057 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9bb40b0650 0x7f9bb414d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:50.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.386+0000 7f9bb9600640 1 --2- 172.21.15.27:0/1572956057 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9bb40cba60 0x7f9bb40c9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:50.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.386+0000 7f9bbaa00640 1 -- 172.21.15.27:0/1572956057 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9bb40b0650 msgr2=0x7f9bb414d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:50.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.386+0000 7f9bbaa00640 1 --2- 172.21.15.27:0/1572956057 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9bb40b0650 0x7f9bb414d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.386+0000 7f9bbaa00640 1 -- 172.21.15.27:0/1572956057 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9bb40cba60 msgr2=0x7f9bb40c9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:50.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.386+0000 7f9bbaa00640 1 --2- 172.21.15.27:0/1572956057 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9bb40cba60 0x7f9bb40c9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.386+0000 7f9bbaa00640 1 -- 172.21.15.27:0/1572956057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba4009420 con 0x7f9bb40cf450 2024-01-15T18:08:50.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.386+0000 7f9bbaa00640 1 --2- 172.21.15.27:0/1572956057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bb40cf450 0x7f9bb40b5ac0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f9bac01a580 tx=0x7f9bac0196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:50.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.386+0000 7f9bb2000640 1 -- 172.21.15.27:0/1572956057 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bac0ad850 con 0x7f9bb40cf450 2024-01-15T18:08:50.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.386+0000 7f9bb2000640 1 -- 172.21.15.27:0/1572956057 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bac0ad9b0 con 0x7f9bb40cf450 2024-01-15T18:08:50.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.386+0000 7f9bb2000640 1 -- 172.21.15.27:0/1572956057 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bac0add60 con 0x7f9bb40cf450 2024-01-15T18:08:50.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.386+0000 7f9bbca00640 1 -- 172.21.15.27:0/1572956057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bac009a70 con 0x7f9bb40cf450 2024-01-15T18:08:50.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.386+0000 7f9bbca00640 1 -- 172.21.15.27:0/1572956057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb40cbfc0 con 0x7f9bb40cf450 2024-01-15T18:08:50.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.390+0000 7f9bbca00640 1 -- 172.21.15.27:0/1572956057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bb40c9070 con 0x7f9bb40cf450 2024-01-15T18:08:50.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.390+0000 7f9bb2000640 1 -- 172.21.15.27:0/1572956057 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 86721+0+0 (secure 0 0 0) 0x7f9bac0b4070 con 0x7f9bb40cf450 2024-01-15T18:08:50.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.390+0000 7f9bb2000640 1 --2- 172.21.15.27:0/1572956057 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9b8c066f20 0x7f9b8c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:50.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.390+0000 7f9bb2000640 1 -- 172.21.15.27:0/1572956057 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9bac11fc80 con 0x7f9bb40cf450 2024-01-15T18:08:50.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.390+0000 7f9bba000640 1 --2- 172.21.15.27:0/1572956057 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9b8c066f20 0x7f9b8c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:50.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.390+0000 7f9bba000640 1 --2- 172.21.15.27:0/1572956057 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9b8c066f20 0x7f9b8c0693e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9bb41f95d0 tx=0x7f9ba4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:50.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.394+0000 7f9bb2000640 1 -- 172.21.15.27:0/1572956057 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9bac0ec770 con 0x7f9bb40cf450 2024-01-15T18:08:50.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.586+0000 7f9bbca00640 1 -- 172.21.15.27:0/1572956057 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9bb40cbe40 con 0x7f9b8c066f20 2024-01-15T18:08:50.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.614+0000 7f9bb2000640 1 -- 172.21.15.27:0/1572956057 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9ba4009960 con 0x7f9b8c066f20 2024-01-15T18:08:50.620 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:08:50.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.614+0000 7f9bbca00640 1 -- 172.21.15.27:0/1572956057 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9b8c066f20 msgr2=0x7f9b8c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:50.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.614+0000 7f9bbca00640 1 --2- 172.21.15.27:0/1572956057 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9b8c066f20 0x7f9b8c0693e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9bb41f95d0 tx=0x7f9ba4009190 comp rx=0 tx=0).stop 2024-01-15T18:08:50.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.614+0000 7f9bbca00640 1 -- 172.21.15.27:0/1572956057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bb40cf450 msgr2=0x7f9bb40b5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:50.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.614+0000 7f9bbca00640 1 --2- 172.21.15.27:0/1572956057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bb40cf450 0x7f9bb40b5ac0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f9bac01a580 tx=0x7f9bac0196f0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.614+0000 7f9bbca00640 1 -- 172.21.15.27:0/1572956057 shutdown_connections 2024-01-15T18:08:50.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.614+0000 7f9bbca00640 1 --2- 172.21.15.27:0/1572956057 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9b8c066f20 0x7f9b8c0693e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.614+0000 7f9bbca00640 1 --2- 172.21.15.27:0/1572956057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bb40cf450 0x7f9bb40b5ac0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.614+0000 7f9bbca00640 1 --2- 172.21.15.27:0/1572956057 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9bb40cba60 0x7f9bb40c9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.614+0000 7f9bbca00640 1 --2- 172.21.15.27:0/1572956057 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9bb40b0650 0x7f9bb414d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.614+0000 7f9bbca00640 1 -- 172.21.15.27:0/1572956057 >> 172.21.15.27:0/1572956057 conn(0x7f9bb40d0e70 msgr2=0x7f9bb40d1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:50.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.614+0000 7f9bbca00640 1 -- 172.21.15.27:0/1572956057 shutdown_connections 2024-01-15T18:08:50.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.614+0000 7f9bbca00640 1 -- 172.21.15.27:0/1572956057 wait complete. 2024-01-15T18:08:50.642 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-15T18:08:50.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.778+0000 7f6429000640 1 Processor -- start 2024-01-15T18:08:50.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.778+0000 7f6429000640 1 -- start start 2024-01-15T18:08:50.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.782+0000 7f6429000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f642415d570 0x7f642406be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:50.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.782+0000 7f6429000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64240b0d70 con 0x7f642415d190 2024-01-15T18:08:50.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.782+0000 7f6429000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64240d0540 con 0x7f642406b810 2024-01-15T18:08:50.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.782+0000 7f6429000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64240b0440 con 0x7f642415d570 2024-01-15T18:08:50.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.782+0000 7f6421400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f642415d570 0x7f642406be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:50.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.782+0000 7f6421e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f642415d190 0x7f64240cfcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55180/0 (socket says 172.21.15.27:55180) 2024-01-15T18:08:50.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.782+0000 7f6421e00640 1 -- 172.21.15.27:0/3285216851 learned_addr learned my addr 172.21.15.27:0/3285216851 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:50.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.782+0000 7f6421400640 1 -- 172.21.15.27:0/3285216851 >> v1:172.21.15.93:6789/0 conn(0x7f642406b810 legacy=0x7f64240cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:50.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.782+0000 7f6420a00640 1 -- 172.21.15.27:0/3285216851 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443597824 0 0) 0x7f64240b0d70 con 0x7f642415d190 2024-01-15T18:08:50.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.782+0000 7f6421400640 1 -- 172.21.15.27:0/3285216851 >> v1:172.21.15.27:6789/0 conn(0x7f642415d190 legacy=0x7f64240cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:50.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.782+0000 7f6421400640 1 -- 172.21.15.27:0/3285216851 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64240b0720 con 0x7f642415d570 2024-01-15T18:08:50.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.786+0000 7f6421400640 1 --2- 172.21.15.27:0/3285216851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f642415d570 0x7f642406be10 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f640c0097c0 tx=0x7f640c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec40a6815fe8adc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:50.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.786+0000 7f6420a00640 1 -- 172.21.15.27:0/3285216851 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c0ab670 con 0x7f642415d570 2024-01-15T18:08:50.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.786+0000 7f6420a00640 1 -- 172.21.15.27:0/3285216851 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640c0ab7d0 con 0x7f642415d570 2024-01-15T18:08:50.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.786+0000 7f6420a00640 1 -- 172.21.15.27:0/3285216851 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c0abb60 con 0x7f642415d570 2024-01-15T18:08:50.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.786+0000 7f6429000640 1 -- 172.21.15.27:0/3285216851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f642415d570 msgr2=0x7f642406be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:50.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.786+0000 7f6429000640 1 --2- 172.21.15.27:0/3285216851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f642415d570 0x7f642406be10 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f640c0097c0 tx=0x7f640c01c450 comp rx=0 tx=0).stop 2024-01-15T18:08:50.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.786+0000 7f6429000640 1 -- 172.21.15.27:0/3285216851 shutdown_connections 2024-01-15T18:08:50.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.786+0000 7f6429000640 1 --2- 172.21.15.27:0/3285216851 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f642415d570 0x7f642406be10 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.786+0000 7f6429000640 1 -- 172.21.15.27:0/3285216851 >> 172.21.15.27:0/3285216851 conn(0x7f64240d0e70 msgr2=0x7f64240d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:50.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.786+0000 7f6429000640 1 -- 172.21.15.27:0/3285216851 shutdown_connections 2024-01-15T18:08:50.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.786+0000 7f6429000640 1 -- 172.21.15.27:0/3285216851 wait complete. 2024-01-15T18:08:50.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.786+0000 7f6429000640 1 Processor -- start 2024-01-15T18:08:50.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.786+0000 7f6429000640 1 -- start start 2024-01-15T18:08:50.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.790+0000 7f6429000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f642406b810 0x7f64241476b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:50.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.790+0000 7f6429000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f642415d190 0x7f642414d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:50.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.790+0000 7f6429000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f642415d570 0x7f6424156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:50.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.790+0000 7f6429000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6424147bf0 con 0x7f642415d190 2024-01-15T18:08:50.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.790+0000 7f6429000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642414ddb0 con 0x7f642406b810 2024-01-15T18:08:50.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.790+0000 7f6429000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642414e090 con 0x7f642415d570 2024-01-15T18:08:50.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.790+0000 7f6421400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f642415d190 0x7f642414d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:50.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.790+0000 7f6421400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f642415d190 0x7f642414d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38798/0 (socket says 172.21.15.27:38798) 2024-01-15T18:08:50.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.790+0000 7f6421400640 1 -- 172.21.15.27:0/2390457684 learned_addr learned my addr 172.21.15.27:0/2390457684 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:50.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.790+0000 7f6422800640 1 --2- 172.21.15.27:0/2390457684 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f642415d570 0x7f6424156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:50.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.790+0000 7f6421e00640 1 --2- 172.21.15.27:0/2390457684 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f642406b810 0x7f64241476b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:50.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.790+0000 7f6421400640 1 -- 172.21.15.27:0/2390457684 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f642415d570 msgr2=0x7f6424156650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:50.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.790+0000 7f6421400640 1 --2- 172.21.15.27:0/2390457684 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f642415d570 0x7f6424156650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.790+0000 7f6421400640 1 -- 172.21.15.27:0/2390457684 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f642406b810 msgr2=0x7f64241476b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:50.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.790+0000 7f6421400640 1 --2- 172.21.15.27:0/2390457684 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f642406b810 0x7f64241476b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.790+0000 7f6421400640 1 -- 172.21.15.27:0/2390457684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640c009420 con 0x7f642415d190 2024-01-15T18:08:50.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.790+0000 7f6421400640 1 --2- 172.21.15.27:0/2390457684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f642415d190 0x7f642414d4a0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f640c019060 tx=0x7f640c0191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:50.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.790+0000 7f641a000640 1 -- 172.21.15.27:0/2390457684 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c0bf450 con 0x7f642415d190 2024-01-15T18:08:50.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.790+0000 7f641a000640 1 -- 172.21.15.27:0/2390457684 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640c0bf5b0 con 0x7f642415d190 2024-01-15T18:08:50.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.790+0000 7f641a000640 1 -- 172.21.15.27:0/2390457684 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c0bf940 con 0x7f642415d190 2024-01-15T18:08:50.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.790+0000 7f6429000640 1 -- 172.21.15.27:0/2390457684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642414e370 con 0x7f642415d190 2024-01-15T18:08:50.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.790+0000 7f6429000640 1 -- 172.21.15.27:0/2390457684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642414d9e0 con 0x7f642415d190 2024-01-15T18:08:50.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.790+0000 7f6429000640 1 -- 172.21.15.27:0/2390457684 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f642404e850 con 0x7f642415d190 2024-01-15T18:08:50.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.790+0000 7f641a000640 1 -- 172.21.15.27:0/2390457684 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 86721+0+0 (secure 0 0 0) 0x7f640c0bfaa0 con 0x7f642415d190 2024-01-15T18:08:50.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.794+0000 7f641a000640 1 --2- 172.21.15.27:0/2390457684 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f63fc066f20 0x7f63fc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:50.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.794+0000 7f641a000640 1 -- 172.21.15.27:0/2390457684 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f640c11fc60 con 0x7f642415d190 2024-01-15T18:08:50.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.794+0000 7f6421e00640 1 --2- 172.21.15.27:0/2390457684 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f63fc066f20 0x7f63fc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:50.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.794+0000 7f6421e00640 1 --2- 172.21.15.27:0/2390457684 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f63fc066f20 0x7f63fc0693e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6414000f70 tx=0x7f6414007240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:50.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.798+0000 7f641a000640 1 -- 172.21.15.27:0/2390457684 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f640c0ec530 con 0x7f642415d190 2024-01-15T18:08:50.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.978+0000 7f6429000640 1 -- 172.21.15.27:0/2390457684 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6424154e00 con 0x7f63fc066f20 2024-01-15T18:08:51.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.998+0000 7f641a000640 1 -- 172.21.15.27:0/2390457684 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6424154e00 con 0x7f63fc066f20 2024-01-15T18:08:51.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.998+0000 7f6429000640 1 -- 172.21.15.27:0/2390457684 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f63fc066f20 msgr2=0x7f63fc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:51.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.998+0000 7f6429000640 1 --2- 172.21.15.27:0/2390457684 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f63fc066f20 0x7f63fc0693e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6414000f70 tx=0x7f6414007240 comp rx=0 tx=0).stop 2024-01-15T18:08:51.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.998+0000 7f6429000640 1 -- 172.21.15.27:0/2390457684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f642415d190 msgr2=0x7f642414d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:51.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.998+0000 7f6429000640 1 --2- 172.21.15.27:0/2390457684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f642415d190 0x7f642414d4a0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f640c019060 tx=0x7f640c0191f0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.998+0000 7f6429000640 1 -- 172.21.15.27:0/2390457684 shutdown_connections 2024-01-15T18:08:51.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.998+0000 7f6429000640 1 --2- 172.21.15.27:0/2390457684 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f63fc066f20 0x7f63fc0693e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.998+0000 7f6429000640 1 --2- 172.21.15.27:0/2390457684 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f642415d570 0x7f6424156650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.998+0000 7f6429000640 1 --2- 172.21.15.27:0/2390457684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f642415d190 0x7f642414d4a0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.998+0000 7f6429000640 1 --2- 172.21.15.27:0/2390457684 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f642406b810 0x7f64241476b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.998+0000 7f6429000640 1 -- 172.21.15.27:0/2390457684 >> 172.21.15.27:0/2390457684 conn(0x7f64240d0e70 msgr2=0x7f64240b2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:51.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.998+0000 7f6429000640 1 -- 172.21.15.27:0/2390457684 shutdown_connections 2024-01-15T18:08:51.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:50.998+0000 7f6429000640 1 -- 172.21.15.27:0/2390457684 wait complete. 2024-01-15T18:08:51.027 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone63892 2024-01-15T18:08:51.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.170+0000 7ff363200640 1 Processor -- start 2024-01-15T18:08:51.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.170+0000 7ff363200640 1 -- start start 2024-01-15T18:08:51.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.170+0000 7ff363200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff35c0cf460 0x7ff35c0cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.170+0000 7ff363200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff35c0b2590 0x7ff35c0cc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.170+0000 7ff363200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35c0cfd80 con 0x7ff35c15b4b0 2024-01-15T18:08:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.170+0000 7ff363200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff35c0d19a0 con 0x7ff35c0cf460 2024-01-15T18:08:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.170+0000 7ff363200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c0d1c80 con 0x7ff35c0b2590 2024-01-15T18:08:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.170+0000 7ff35be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff35c0cf460 0x7ff35c0cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.170+0000 7ff35be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff35c0cf460 0x7ff35c0cf840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38802/0 (socket says 172.21.15.27:38802) 2024-01-15T18:08:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.170+0000 7ff35be00640 1 -- 172.21.15.27:0/3709574886 learned_addr learned my addr 172.21.15.27:0/3709574886 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.170+0000 7ff361200640 1 --2- 172.21.15.27:0/3709574886 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff35c0b2590 0x7ff35c0cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.170+0000 7ff35be00640 1 -- 172.21.15.27:0/3709574886 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff35c0b2590 msgr2=0x7ff35c0cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.170+0000 7ff35be00640 1 --2- 172.21.15.27:0/3709574886 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff35c0b2590 0x7ff35c0cc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.170+0000 7ff35be00640 1 -- 172.21.15.27:0/3709574886 >> v1:172.21.15.93:6789/0 conn(0x7ff35c15b4b0 legacy=0x7ff35c0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.170+0000 7ff35be00640 1 -- 172.21.15.27:0/3709574886 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35c0b07c0 con 0x7ff35c0cf460 2024-01-15T18:08:51.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.170+0000 7ff361200640 1 --2- 172.21.15.27:0/3709574886 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff35c0b2590 0x7ff35c0cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:08:51.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.174+0000 7ff35be00640 1 --2- 172.21.15.27:0/3709574886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff35c0cf460 0x7ff35c0cf840 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7ff34c009870 tx=0x7ff34c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d725681e551b217e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:51.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.174+0000 7ff35aa00640 1 -- 172.21.15.27:0/3709574886 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c0aa620 con 0x7ff35c0cf460 2024-01-15T18:08:51.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.174+0000 7ff35aa00640 1 -- 172.21.15.27:0/3709574886 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34c0aa780 con 0x7ff35c0cf460 2024-01-15T18:08:51.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.174+0000 7ff35aa00640 1 -- 172.21.15.27:0/3709574886 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c0aab10 con 0x7ff35c0cf460 2024-01-15T18:08:51.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.174+0000 7ff363200640 1 -- 172.21.15.27:0/3709574886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff35c0cf460 msgr2=0x7ff35c0cf840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:51.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.174+0000 7ff363200640 1 --2- 172.21.15.27:0/3709574886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff35c0cf460 0x7ff35c0cf840 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7ff34c009870 tx=0x7ff34c01c500 comp rx=0 tx=0).stop 2024-01-15T18:08:51.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.174+0000 7ff363200640 1 -- 172.21.15.27:0/3709574886 shutdown_connections 2024-01-15T18:08:51.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.174+0000 7ff363200640 1 --2- 172.21.15.27:0/3709574886 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff35c0b2590 0x7ff35c0cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.174+0000 7ff363200640 1 --2- 172.21.15.27:0/3709574886 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff35c0cf460 0x7ff35c0cf840 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.174+0000 7ff363200640 1 -- 172.21.15.27:0/3709574886 >> 172.21.15.27:0/3709574886 conn(0x7ff35c0d0e80 msgr2=0x7ff35c0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:51.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.174+0000 7ff363200640 1 -- 172.21.15.27:0/3709574886 shutdown_connections 2024-01-15T18:08:51.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.174+0000 7ff363200640 1 -- 172.21.15.27:0/3709574886 wait complete. 2024-01-15T18:08:51.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.178+0000 7ff363200640 1 Processor -- start 2024-01-15T18:08:51.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.178+0000 7ff363200640 1 -- start start 2024-01-15T18:08:51.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.178+0000 7ff363200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff35c0b2590 0x7ff35c149e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:51.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.178+0000 7ff363200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff35c0cf460 0x7ff35c153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:51.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.178+0000 7ff363200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff35c15b4b0 0x7ff35c154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:51.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.178+0000 7ff363200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c14a390 con 0x7ff35c0b2590 2024-01-15T18:08:51.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.178+0000 7ff363200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c157dd0 con 0x7ff35c15b4b0 2024-01-15T18:08:51.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.178+0000 7ff363200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff35c1580b0 con 0x7ff35c0cf460 2024-01-15T18:08:51.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.178+0000 7ff35be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff35c0b2590 0x7ff35c149e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:51.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.178+0000 7ff35be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff35c0b2590 0x7ff35c149e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38814/0 (socket says 172.21.15.27:38814) 2024-01-15T18:08:51.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.178+0000 7ff35be00640 1 -- 172.21.15.27:0/2929509457 learned_addr learned my addr 172.21.15.27:0/2929509457 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:51.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.178+0000 7ff35b400640 1 --2- 172.21.15.27:0/2929509457 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff35c0cf460 0x7ff35c153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:51.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.178+0000 7ff361200640 1 --2- 172.21.15.27:0/2929509457 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff35c15b4b0 0x7ff35c154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:51.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.178+0000 7ff35be00640 1 -- 172.21.15.27:0/2929509457 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff35c0cf460 msgr2=0x7ff35c153e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:51.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.178+0000 7ff35be00640 1 --2- 172.21.15.27:0/2929509457 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff35c0cf460 0x7ff35c153e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.178+0000 7ff35be00640 1 -- 172.21.15.27:0/2929509457 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff35c15b4b0 msgr2=0x7ff35c154380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:51.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.178+0000 7ff35be00640 1 --2- 172.21.15.27:0/2929509457 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff35c15b4b0 0x7ff35c154380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.178+0000 7ff35be00640 1 -- 172.21.15.27:0/2929509457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34c0094d0 con 0x7ff35c0b2590 2024-01-15T18:08:51.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.178+0000 7ff361200640 1 --2- 172.21.15.27:0/2929509457 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff35c15b4b0 0x7ff35c154380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:08:51.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.178+0000 7ff35be00640 1 --2- 172.21.15.27:0/2929509457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff35c0b2590 0x7ff35c149e50 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0ac040 tx=0x7ff34c01ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:51.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.182+0000 7ff353e00640 1 -- 172.21.15.27:0/2929509457 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c0bd5f0 con 0x7ff35c0b2590 2024-01-15T18:08:51.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.182+0000 7ff353e00640 1 -- 172.21.15.27:0/2929509457 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34c0bd770 con 0x7ff35c0b2590 2024-01-15T18:08:51.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.182+0000 7ff353e00640 1 -- 172.21.15.27:0/2929509457 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c0bdb20 con 0x7ff35c0b2590 2024-01-15T18:08:51.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.182+0000 7ff363200640 1 -- 172.21.15.27:0/2929509457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff35c158390 con 0x7ff35c0b2590 2024-01-15T18:08:51.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.182+0000 7ff363200640 1 -- 172.21.15.27:0/2929509457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff35c158950 con 0x7ff35c0b2590 2024-01-15T18:08:51.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.182+0000 7ff363200640 1 -- 172.21.15.27:0/2929509457 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff35c04e850 con 0x7ff35c0b2590 2024-01-15T18:08:51.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.182+0000 7ff353e00640 1 -- 172.21.15.27:0/2929509457 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 86721+0+0 (secure 0 0 0) 0x7ff34c0b33f0 con 0x7ff35c0b2590 2024-01-15T18:08:51.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.182+0000 7ff353e00640 1 --2- 172.21.15.27:0/2929509457 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff334066f20 0x7ff3340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:51.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.182+0000 7ff353e00640 1 -- 172.21.15.27:0/2929509457 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff34c11fac0 con 0x7ff35c0b2590 2024-01-15T18:08:51.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.182+0000 7ff35b400640 1 --2- 172.21.15.27:0/2929509457 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff334066f20 0x7ff3340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:51.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.186+0000 7ff35b400640 1 --2- 172.21.15.27:0/2929509457 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff334066f20 0x7ff3340693e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff35c0cde70 tx=0x7ff344008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:51.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.186+0000 7ff353e00640 1 -- 172.21.15.27:0/2929509457 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff34c0ec420 con 0x7ff35c0b2590 2024-01-15T18:08:51.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.370+0000 7ff363200640 1 -- 172.21.15.27:0/2929509457 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone63892", "target": ["mon-mgr", ""]}) v1 -- 0x7ff35c1581f0 con 0x7ff334066f20 2024-01-15T18:08:51.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.386+0000 7ff353e00640 1 -- 172.21.15.27:0/2929509457 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff35c1581f0 con 0x7ff334066f20 2024-01-15T18:08:51.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.390+0000 7ff363200640 1 -- 172.21.15.27:0/2929509457 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff334066f20 msgr2=0x7ff3340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:51.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.390+0000 7ff363200640 1 --2- 172.21.15.27:0/2929509457 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff334066f20 0x7ff3340693e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff35c0cde70 tx=0x7ff344008040 comp rx=0 tx=0).stop 2024-01-15T18:08:51.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.390+0000 7ff363200640 1 -- 172.21.15.27:0/2929509457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff35c0b2590 msgr2=0x7ff35c149e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:51.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.390+0000 7ff363200640 1 --2- 172.21.15.27:0/2929509457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff35c0b2590 0x7ff35c149e50 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0ac040 tx=0x7ff34c01ad40 comp rx=0 tx=0).stop 2024-01-15T18:08:51.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.390+0000 7ff363200640 1 -- 172.21.15.27:0/2929509457 shutdown_connections 2024-01-15T18:08:51.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.390+0000 7ff363200640 1 --2- 172.21.15.27:0/2929509457 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff334066f20 0x7ff3340693e0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.390+0000 7ff363200640 1 --2- 172.21.15.27:0/2929509457 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff35c15b4b0 0x7ff35c154380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.390+0000 7ff363200640 1 --2- 172.21.15.27:0/2929509457 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff35c0cf460 0x7ff35c153e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.390+0000 7ff363200640 1 --2- 172.21.15.27:0/2929509457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff35c0b2590 0x7ff35c149e50 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.390+0000 7ff363200640 1 -- 172.21.15.27:0/2929509457 >> 172.21.15.27:0/2929509457 conn(0x7ff35c0d0e80 msgr2=0x7ff35c1567b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:51.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.390+0000 7ff363200640 1 -- 172.21.15.27:0/2929509457 shutdown_connections 2024-01-15T18:08:51.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.390+0000 7ff363200640 1 -- 172.21.15.27:0/2929509457 wait complete. 2024-01-15T18:08:51.419 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone63891 2024-01-15T18:08:51.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.558+0000 7f171fc00640 1 Processor -- start 2024-01-15T18:08:51.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.558+0000 7f171fc00640 1 -- start start 2024-01-15T18:08:51.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.562+0000 7f171fc00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f17180cf460 0x7f17180cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:51.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.562+0000 7f171fc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f171815b4b0 0x7f17180cba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:51.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.562+0000 7f171fc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17180cec60 0x7f17180cbf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:51.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.562+0000 7f171fc00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17180cfd80 con 0x7f171815b4b0 2024-01-15T18:08:51.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.562+0000 7f171fc00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17180d19a0 con 0x7f17180cf460 2024-01-15T18:08:51.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.562+0000 7f171fc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17180d1c80 con 0x7f17180cec60 2024-01-15T18:08:51.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.562+0000 7f1717e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f171815b4b0 0x7f17180cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:51.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.562+0000 7f1717e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f171815b4b0 0x7f17180cba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38826/0 (socket says 172.21.15.27:38826) 2024-01-15T18:08:51.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.562+0000 7f1717e00640 1 -- 172.21.15.27:0/3291989567 learned_addr learned my addr 172.21.15.27:0/3291989567 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:51.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.562+0000 7f171dc00640 1 --2- 172.21.15.27:0/3291989567 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17180cec60 0x7f17180cbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:51.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.562+0000 7f1717e00640 1 -- 172.21.15.27:0/3291989567 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17180cec60 msgr2=0x7f17180cbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:51.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.562+0000 7f1717e00640 1 --2- 172.21.15.27:0/3291989567 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17180cec60 0x7f17180cbf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.562+0000 7f1717e00640 1 -- 172.21.15.27:0/3291989567 >> v2:172.21.15.93:3300/0 conn(0x7f17180cf460 msgr2=0x7f17180cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:51.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.562+0000 7f171d200640 1 --2- 172.21.15.27:0/3291989567 >> v2:172.21.15.93:3300/0 conn(0x7f17180cf460 0x7f17180cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:51.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.562+0000 7f1717e00640 1 --2- 172.21.15.27:0/3291989567 >> v2:172.21.15.93:3300/0 conn(0x7f17180cf460 0x7f17180cf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.562+0000 7f1717e00640 1 -- 172.21.15.27:0/3291989567 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17180b07c0 con 0x7f171815b4b0 2024-01-15T18:08:51.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.562+0000 7f171dc00640 1 --2- 172.21.15.27:0/3291989567 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17180cec60 0x7f17180cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:08:51.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.562+0000 7f171d200640 1 --2- 172.21.15.27:0/3291989567 >> v2:172.21.15.93:3300/0 conn(0x7f17180cf460 0x7f17180cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:08:51.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.566+0000 7f1717e00640 1 --2- 172.21.15.27:0/3291989567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f171815b4b0 0x7f17180cba20 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f17180681f0 tx=0x7f170001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27316d079f15062b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:51.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.566+0000 7f1717400640 1 -- 172.21.15.27:0/3291989567 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17000a7070 con 0x7f171815b4b0 2024-01-15T18:08:51.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.566+0000 7f1717400640 1 -- 172.21.15.27:0/3291989567 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17000ab430 con 0x7f171815b4b0 2024-01-15T18:08:51.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.566+0000 7f1717400640 1 -- 172.21.15.27:0/3291989567 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17000ab7c0 con 0x7f171815b4b0 2024-01-15T18:08:51.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.566+0000 7f171fc00640 1 -- 172.21.15.27:0/3291989567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f171815b4b0 msgr2=0x7f17180cba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:51.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.566+0000 7f171fc00640 1 --2- 172.21.15.27:0/3291989567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f171815b4b0 0x7f17180cba20 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f17180681f0 tx=0x7f170001c470 comp rx=0 tx=0).stop 2024-01-15T18:08:51.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.566+0000 7f171fc00640 1 -- 172.21.15.27:0/3291989567 shutdown_connections 2024-01-15T18:08:51.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.566+0000 7f171fc00640 1 --2- 172.21.15.27:0/3291989567 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17180cec60 0x7f17180cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.566+0000 7f171fc00640 1 --2- 172.21.15.27:0/3291989567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f171815b4b0 0x7f17180cba20 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.566+0000 7f171fc00640 1 --2- 172.21.15.27:0/3291989567 >> v2:172.21.15.93:3300/0 conn(0x7f17180cf460 0x7f17180cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.566+0000 7f171fc00640 1 -- 172.21.15.27:0/3291989567 >> 172.21.15.27:0/3291989567 conn(0x7f17180d0e80 msgr2=0x7f17180d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:51.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.566+0000 7f171fc00640 1 -- 172.21.15.27:0/3291989567 shutdown_connections 2024-01-15T18:08:51.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.566+0000 7f171fc00640 1 -- 172.21.15.27:0/3291989567 wait complete. 2024-01-15T18:08:51.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.566+0000 7f171fc00640 1 Processor -- start 2024-01-15T18:08:51.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.566+0000 7f171fc00640 1 -- start start 2024-01-15T18:08:51.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.570+0000 7f171fc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17180cec60 0x7f1718153cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:51.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.570+0000 7f171fc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17180cf460 0x7f1718153530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.570+0000 7f171fc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f171815b4b0 0x7f17180c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.570+0000 7f171fc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171806b9d0 con 0x7f17180cf460 2024-01-15T18:08:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.570+0000 7f171fc00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171806b6a0 con 0x7f171815b4b0 2024-01-15T18:08:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.570+0000 7f171fc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f171815d790 con 0x7f17180cec60 2024-01-15T18:08:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.570+0000 7f1717e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17180cf460 0x7f1718153530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.570+0000 7f1717e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17180cf460 0x7f1718153530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38834/0 (socket says 172.21.15.27:38834) 2024-01-15T18:08:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.570+0000 7f1717e00640 1 -- 172.21.15.27:0/4084709300 learned_addr learned my addr 172.21.15.27:0/4084709300 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.570+0000 7f171d200640 1 --2- 172.21.15.27:0/4084709300 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17180cec60 0x7f1718153cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.570+0000 7f171dc00640 1 --2- 172.21.15.27:0/4084709300 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f171815b4b0 0x7f17180c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.570+0000 7f1717e00640 1 -- 172.21.15.27:0/4084709300 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17180cec60 msgr2=0x7f1718153cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.570+0000 7f1717e00640 1 --2- 172.21.15.27:0/4084709300 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17180cec60 0x7f1718153cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.570+0000 7f1717e00640 1 -- 172.21.15.27:0/4084709300 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f171815b4b0 msgr2=0x7f17180c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.570+0000 7f1717e00640 1 --2- 172.21.15.27:0/4084709300 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f171815b4b0 0x7f17180c7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.570+0000 7f1717e00640 1 -- 172.21.15.27:0/4084709300 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17000094d0 con 0x7f17180cf460 2024-01-15T18:08:51.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.570+0000 7f1717e00640 1 --2- 172.21.15.27:0/4084709300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17180cf460 0x7f1718153530 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f170001ab20 tx=0x7f1700019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:51.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.570+0000 7f1714c00640 1 -- 172.21.15.27:0/4084709300 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17000ba070 con 0x7f17180cf460 2024-01-15T18:08:51.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.570+0000 7f1714c00640 1 -- 172.21.15.27:0/4084709300 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17000b4630 con 0x7f17180cf460 2024-01-15T18:08:51.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.570+0000 7f1714c00640 1 -- 172.21.15.27:0/4084709300 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17000b49e0 con 0x7f17180cf460 2024-01-15T18:08:51.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.570+0000 7f171fc00640 1 -- 172.21.15.27:0/4084709300 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f171815d1d0 con 0x7f17180cf460 2024-01-15T18:08:51.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.570+0000 7f171fc00640 1 -- 172.21.15.27:0/4084709300 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1718153a70 con 0x7f17180cf460 2024-01-15T18:08:51.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.570+0000 7f171fc00640 1 -- 172.21.15.27:0/4084709300 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f171806bb40 con 0x7f17180cf460 2024-01-15T18:08:51.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.574+0000 7f1714c00640 1 -- 172.21.15.27:0/4084709300 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 86721+0+0 (secure 0 0 0) 0x7f17000b4b40 con 0x7f17180cf460 2024-01-15T18:08:51.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.574+0000 7f1714c00640 1 --2- 172.21.15.27:0/4084709300 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f16f0066f20 0x7f16f00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:51.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.574+0000 7f1714c00640 1 -- 172.21.15.27:0/4084709300 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f170011ea90 con 0x7f17180cf460 2024-01-15T18:08:51.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.574+0000 7f171d200640 1 --2- 172.21.15.27:0/4084709300 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f16f0066f20 0x7f16f00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:51.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.574+0000 7f171d200640 1 --2- 172.21.15.27:0/4084709300 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f16f0066f20 0x7f16f00693e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f17180af180 tx=0x7f1708008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:51.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.578+0000 7f1714c00640 1 -- 172.21.15.27:0/4084709300 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f17000eb330 con 0x7f17180cf460 2024-01-15T18:08:51.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.758+0000 7f171fc00640 1 -- 172.21.15.27:0/4084709300 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone63891", "target": ["mon-mgr", ""]}) v1 -- 0x7f17180cc8b0 con 0x7f16f0066f20 2024-01-15T18:08:51.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.774+0000 7f1714c00640 1 -- 172.21.15.27:0/4084709300 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f170800c040 con 0x7f16f0066f20 2024-01-15T18:08:51.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.778+0000 7f171fc00640 1 -- 172.21.15.27:0/4084709300 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f16f0066f20 msgr2=0x7f16f00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:51.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.778+0000 7f171fc00640 1 --2- 172.21.15.27:0/4084709300 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f16f0066f20 0x7f16f00693e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f17180af180 tx=0x7f1708008040 comp rx=0 tx=0).stop 2024-01-15T18:08:51.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.778+0000 7f171fc00640 1 -- 172.21.15.27:0/4084709300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17180cf460 msgr2=0x7f1718153530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:51.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.778+0000 7f171fc00640 1 --2- 172.21.15.27:0/4084709300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17180cf460 0x7f1718153530 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f170001ab20 tx=0x7f1700019210 comp rx=0 tx=0).stop 2024-01-15T18:08:51.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.778+0000 7f171fc00640 1 -- 172.21.15.27:0/4084709300 shutdown_connections 2024-01-15T18:08:51.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.778+0000 7f171fc00640 1 --2- 172.21.15.27:0/4084709300 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f16f0066f20 0x7f16f00693e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.778+0000 7f171fc00640 1 --2- 172.21.15.27:0/4084709300 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f171815b4b0 0x7f17180c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.778+0000 7f171fc00640 1 --2- 172.21.15.27:0/4084709300 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17180cf460 0x7f1718153530 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.778+0000 7f171fc00640 1 --2- 172.21.15.27:0/4084709300 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17180cec60 0x7f1718153cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.778+0000 7f171fc00640 1 -- 172.21.15.27:0/4084709300 >> 172.21.15.27:0/4084709300 conn(0x7f17180d0e80 msgr2=0x7f171815a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:51.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.778+0000 7f171fc00640 1 -- 172.21.15.27:0/4084709300 shutdown_connections 2024-01-15T18:08:51.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.778+0000 7f171fc00640 1 -- 172.21.15.27:0/4084709300 wait complete. 2024-01-15T18:08:51.805 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone63894 2024-01-15T18:08:51.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.942+0000 7f2f39c00640 1 Processor -- start 2024-01-15T18:08:51.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.946+0000 7f2f39c00640 1 -- start start 2024-01-15T18:08:51.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.946+0000 7f2f39c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2f3406c520 0x7f2f340cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:51.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.946+0000 7f2f39c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f340b0d70 con 0x7f2f3415d570 2024-01-15T18:08:51.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.946+0000 7f2f39c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f340d0540 con 0x7f2f3415d190 2024-01-15T18:08:51.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.946+0000 7f2f39c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f340b0440 con 0x7f2f3406c520 2024-01-15T18:08:51.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.946+0000 7f2f33400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2f3406c520 0x7f2f340cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:51.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.946+0000 7f2f32000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2f3415d570 0x7f2f3406be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55182/0 (socket says 172.21.15.27:55182) 2024-01-15T18:08:51.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.946+0000 7f2f32000640 1 -- 172.21.15.27:0/1932331765 learned_addr learned my addr 172.21.15.27:0/1932331765 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:51.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.946+0000 7f2f33400640 1 -- 172.21.15.27:0/1932331765 >> v1:172.21.15.93:6789/0 conn(0x7f2f3415d190 legacy=0x7f2f340cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:51.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.946+0000 7f2f33400640 1 -- 172.21.15.27:0/1932331765 >> v1:172.21.15.27:6789/0 conn(0x7f2f3415d570 legacy=0x7f2f3406be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:51.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.946+0000 7f2f33400640 1 -- 172.21.15.27:0/1932331765 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f340b0720 con 0x7f2f3406c520 2024-01-15T18:08:51.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.950+0000 7f2f33400640 1 --2- 172.21.15.27:0/1932331765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f3406c520 0x7f2f340cd5e0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c009d90 tx=0x7f2f2c01c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5f3fc5872ac0981 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:51.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.950+0000 7f2f31600640 1 -- 172.21.15.27:0/1932331765 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c0a5a00 con 0x7f2f3406c520 2024-01-15T18:08:51.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.950+0000 7f2f31600640 1 -- 172.21.15.27:0/1932331765 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2c0a5b60 con 0x7f2f3406c520 2024-01-15T18:08:51.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.950+0000 7f2f31600640 1 -- 172.21.15.27:0/1932331765 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c0ad440 con 0x7f2f3406c520 2024-01-15T18:08:51.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.950+0000 7f2f39c00640 1 -- 172.21.15.27:0/1932331765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f3406c520 msgr2=0x7f2f340cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:51.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.950+0000 7f2f39c00640 1 --2- 172.21.15.27:0/1932331765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f3406c520 0x7f2f340cd5e0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c009d90 tx=0x7f2f2c01c920 comp rx=0 tx=0).stop 2024-01-15T18:08:51.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.950+0000 7f2f39c00640 1 -- 172.21.15.27:0/1932331765 shutdown_connections 2024-01-15T18:08:51.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.950+0000 7f2f39c00640 1 --2- 172.21.15.27:0/1932331765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f3406c520 0x7f2f340cd5e0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.950+0000 7f2f39c00640 1 -- 172.21.15.27:0/1932331765 >> 172.21.15.27:0/1932331765 conn(0x7f2f340d0e70 msgr2=0x7f2f340d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:51.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.950+0000 7f2f39c00640 1 -- 172.21.15.27:0/1932331765 shutdown_connections 2024-01-15T18:08:51.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.950+0000 7f2f39c00640 1 -- 172.21.15.27:0/1932331765 wait complete. 2024-01-15T18:08:51.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.950+0000 7f2f39c00640 1 Processor -- start 2024-01-15T18:08:51.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.954+0000 7f2f39c00640 1 -- start start 2024-01-15T18:08:51.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.954+0000 7f2f39c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2f3406c520 0x7f2f3414e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:51.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.954+0000 7f2f39c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f3415d190 0x7f2f340b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:51.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.954+0000 7f2f39c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2f3415d570 0x7f2f340c5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:51.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.954+0000 7f2f39c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f340c4ad0 con 0x7f2f3415d190 2024-01-15T18:08:51.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.954+0000 7f2f39c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f340c4db0 con 0x7f2f3415d570 2024-01-15T18:08:51.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.954+0000 7f2f39c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f340c5090 con 0x7f2f3406c520 2024-01-15T18:08:51.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.954+0000 7f2f32000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f3415d190 0x7f2f340b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:51.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.954+0000 7f2f32a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2f3406c520 0x7f2f3414e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:51.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.954+0000 7f2f32000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f3415d190 0x7f2f340b61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38854/0 (socket says 172.21.15.27:38854) 2024-01-15T18:08:51.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.954+0000 7f2f32000640 1 -- 172.21.15.27:0/1016748299 learned_addr learned my addr 172.21.15.27:0/1016748299 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:51.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.954+0000 7f2f33400640 1 --2- 172.21.15.27:0/1016748299 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2f3415d570 0x7f2f340c5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:51.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.954+0000 7f2f32000640 1 -- 172.21.15.27:0/1016748299 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2f3406c520 msgr2=0x7f2f3414e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:51.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.954+0000 7f2f32000640 1 --2- 172.21.15.27:0/1016748299 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2f3406c520 0x7f2f3414e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.954+0000 7f2f32000640 1 -- 172.21.15.27:0/1016748299 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2f3415d570 msgr2=0x7f2f340c5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:51.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.954+0000 7f2f32000640 1 --2- 172.21.15.27:0/1016748299 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2f3415d570 0x7f2f340c5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:51.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.954+0000 7f2f32000640 1 -- 172.21.15.27:0/1016748299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2c009a70 con 0x7f2f3415d190 2024-01-15T18:08:51.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.954+0000 7f2f32a00640 1 --2- 172.21.15.27:0/1016748299 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2f3406c520 0x7f2f3414e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:08:51.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.954+0000 7f2f32000640 1 --2- 172.21.15.27:0/1016748299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f3415d190 0x7f2f340b61f0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c00c400 tx=0x7f2f1c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:51.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.954+0000 7f2f2aa00640 1 -- 172.21.15.27:0/1016748299 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c005cf0 con 0x7f2f3415d190 2024-01-15T18:08:51.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.954+0000 7f2f2aa00640 1 -- 172.21.15.27:0/1016748299 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f1c005e50 con 0x7f2f3415d190 2024-01-15T18:08:51.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.954+0000 7f2f39c00640 1 -- 172.21.15.27:0/1016748299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f340c5370 con 0x7f2f3415d190 2024-01-15T18:08:51.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.954+0000 7f2f2aa00640 1 -- 172.21.15.27:0/1016748299 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c0072d0 con 0x7f2f3415d190 2024-01-15T18:08:51.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.954+0000 7f2f39c00640 1 -- 172.21.15.27:0/1016748299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f3414ec40 con 0x7f2f3415d190 2024-01-15T18:08:51.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.954+0000 7f2f39c00640 1 -- 172.21.15.27:0/1016748299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f340c55b0 con 0x7f2f3415d190 2024-01-15T18:08:51.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.958+0000 7f2f2aa00640 1 -- 172.21.15.27:0/1016748299 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 86721+0+0 (secure 0 0 0) 0x7f2f1c007430 con 0x7f2f3415d190 2024-01-15T18:08:51.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.958+0000 7f2f2aa00640 1 --2- 172.21.15.27:0/1016748299 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2f0c066f20 0x7f2f0c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:51.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.958+0000 7f2f2aa00640 1 -- 172.21.15.27:0/1016748299 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2f1c084ab0 con 0x7f2f3415d190 2024-01-15T18:08:51.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.958+0000 7f2f32a00640 1 --2- 172.21.15.27:0/1016748299 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2f0c066f20 0x7f2f0c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:51.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.958+0000 7f2f32a00640 1 --2- 172.21.15.27:0/1016748299 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2f0c066f20 0x7f2f0c0693e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2f24007fd0 tx=0x7f2f24009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:51.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:51.962+0000 7f2f2aa00640 1 -- 172.21.15.27:0/1016748299 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2f1c0514d0 con 0x7f2f3415d190 2024-01-15T18:08:52.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.146+0000 7f2f39c00640 1 -- 172.21.15.27:0/1016748299 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone63894", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f340c4f20 con 0x7f2f0c066f20 2024-01-15T18:08:52.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.162+0000 7f2f2aa00640 1 -- 172.21.15.27:0/1016748299 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2f340c4f20 con 0x7f2f0c066f20 2024-01-15T18:08:52.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.162+0000 7f2f39c00640 1 -- 172.21.15.27:0/1016748299 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2f0c066f20 msgr2=0x7f2f0c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:52.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.162+0000 7f2f39c00640 1 --2- 172.21.15.27:0/1016748299 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2f0c066f20 0x7f2f0c0693e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2f24007fd0 tx=0x7f2f24009040 comp rx=0 tx=0).stop 2024-01-15T18:08:52.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.162+0000 7f2f39c00640 1 -- 172.21.15.27:0/1016748299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f3415d190 msgr2=0x7f2f340b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:52.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.162+0000 7f2f39c00640 1 --2- 172.21.15.27:0/1016748299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f3415d190 0x7f2f340b61f0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c00c400 tx=0x7f2f1c00c8c0 comp rx=0 tx=0).stop 2024-01-15T18:08:52.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.162+0000 7f2f39c00640 1 -- 172.21.15.27:0/1016748299 shutdown_connections 2024-01-15T18:08:52.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.162+0000 7f2f39c00640 1 --2- 172.21.15.27:0/1016748299 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2f0c066f20 0x7f2f0c0693e0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:52.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.162+0000 7f2f39c00640 1 --2- 172.21.15.27:0/1016748299 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2f3415d570 0x7f2f340c5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:52.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.162+0000 7f2f39c00640 1 --2- 172.21.15.27:0/1016748299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f3415d190 0x7f2f340b61f0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:52.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.162+0000 7f2f39c00640 1 --2- 172.21.15.27:0/1016748299 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2f3406c520 0x7f2f3414e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:52.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.162+0000 7f2f39c00640 1 -- 172.21.15.27:0/1016748299 >> 172.21.15.27:0/1016748299 conn(0x7f2f340d0e70 msgr2=0x7f2f340adae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:52.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.162+0000 7f2f39c00640 1 -- 172.21.15.27:0/1016748299 shutdown_connections 2024-01-15T18:08:52.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.162+0000 7f2f39c00640 1 -- 172.21.15.27:0/1016748299 wait complete. 2024-01-15T18:08:52.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:08:52.191 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-15T18:08:52.207 INFO:teuthology.orchestra.run.smithi027.stdout:2 2024-01-15T18:08:52.208 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-15T18:08:52.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.386+0000 7f1dd7a00640 1 Processor -- start 2024-01-15T18:08:52.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.386+0000 7f1dd7a00640 1 -- start start 2024-01-15T18:08:52.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.390+0000 7f1dd7a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1dd0071550 0x7f1dd006a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:52.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.390+0000 7f1dd7a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1dd006bc50 0x7f1dd015d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:52.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.390+0000 7f1dd7a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dd0161020 con 0x7f1dd006b050 2024-01-15T18:08:52.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.390+0000 7f1dd7a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dd0161300 con 0x7f1dd006bc50 2024-01-15T18:08:52.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.390+0000 7f1dd7a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd01615e0 con 0x7f1dd0071550 2024-01-15T18:08:52.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.390+0000 7f1dd5a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1dd006bc50 0x7f1dd015d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:52.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.390+0000 7f1dd5a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1dd006bc50 0x7f1dd015d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38864/0 (socket says 172.21.15.27:38864) 2024-01-15T18:08:52.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.390+0000 7f1dd5a00640 1 -- 172.21.15.27:0/331567239 learned_addr learned my addr 172.21.15.27:0/331567239 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:52.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.390+0000 7f1dcfe00640 1 --2- 172.21.15.27:0/331567239 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1dd0071550 0x7f1dd006a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:52.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.390+0000 7f1dd5a00640 1 -- 172.21.15.27:0/331567239 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1dd0071550 msgr2=0x7f1dd006a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:52.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.390+0000 7f1dd5a00640 1 --2- 172.21.15.27:0/331567239 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1dd0071550 0x7f1dd006a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:52.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.390+0000 7f1dd5a00640 1 -- 172.21.15.27:0/331567239 >> v1:172.21.15.93:6789/0 conn(0x7f1dd006b050 legacy=0x7f1dd0070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:52.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.390+0000 7f1dd5a00640 1 -- 172.21.15.27:0/331567239 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd01618c0 con 0x7f1dd006bc50 2024-01-15T18:08:52.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.394+0000 7f1dd5a00640 1 --2- 172.21.15.27:0/331567239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dd006bc50 0x7f1dd015d9f0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8009e50 tx=0x7f1dc801c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc3f193eb42c5e17 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:52.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.394+0000 7f1dcf400640 1 -- 172.21.15.27:0/331567239 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc80a5820 con 0x7f1dd006bc50 2024-01-15T18:08:52.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.394+0000 7f1dcf400640 1 -- 172.21.15.27:0/331567239 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc80a5980 con 0x7f1dd006bc50 2024-01-15T18:08:52.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.394+0000 7f1dcf400640 1 -- 172.21.15.27:0/331567239 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc80a5d10 con 0x7f1dd006bc50 2024-01-15T18:08:52.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.394+0000 7f1dd7a00640 1 -- 172.21.15.27:0/331567239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dd006bc50 msgr2=0x7f1dd015d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:52.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.394+0000 7f1dd7a00640 1 --2- 172.21.15.27:0/331567239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dd006bc50 0x7f1dd015d9f0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8009e50 tx=0x7f1dc801c980 comp rx=0 tx=0).stop 2024-01-15T18:08:52.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.394+0000 7f1dd7a00640 1 -- 172.21.15.27:0/331567239 shutdown_connections 2024-01-15T18:08:52.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.394+0000 7f1dd7a00640 1 --2- 172.21.15.27:0/331567239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dd006bc50 0x7f1dd015d9f0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:52.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.394+0000 7f1dd7a00640 1 --2- 172.21.15.27:0/331567239 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1dd0071550 0x7f1dd006a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:52.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.394+0000 7f1dd7a00640 1 -- 172.21.15.27:0/331567239 >> 172.21.15.27:0/331567239 conn(0x7f1dd00b15b0 msgr2=0x7f1dd00b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:52.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.394+0000 7f1dd7a00640 1 -- 172.21.15.27:0/331567239 shutdown_connections 2024-01-15T18:08:52.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.394+0000 7f1dd7a00640 1 -- 172.21.15.27:0/331567239 wait complete. 2024-01-15T18:08:52.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.394+0000 7f1dd7a00640 1 Processor -- start 2024-01-15T18:08:52.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.394+0000 7f1dd7a00640 1 -- start start 2024-01-15T18:08:52.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.398+0000 7f1dd7a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dd006b050 0x7f1dd014eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:52.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.398+0000 7f1dd7a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1dd006bc50 0x7f1dd0150f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:52.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.398+0000 7f1dd7a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1dd0071550 0x7f1dd0151470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:52.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.398+0000 7f1dd7a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd00cd5e0 con 0x7f1dd006b050 2024-01-15T18:08:52.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.398+0000 7f1dd7a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd014f140 con 0x7f1dd006bc50 2024-01-15T18:08:52.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.398+0000 7f1dd7a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd00cd830 con 0x7f1dd0071550 2024-01-15T18:08:52.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.398+0000 7f1dd5000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dd006b050 0x7f1dd014eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:52.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.398+0000 7f1dcfe00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1dd006bc50 0x7f1dd0150f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:52.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.398+0000 7f1dd5000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dd006b050 0x7f1dd014eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38868/0 (socket says 172.21.15.27:38868) 2024-01-15T18:08:52.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.398+0000 7f1dd5000640 1 -- 172.21.15.27:0/1311550045 learned_addr learned my addr 172.21.15.27:0/1311550045 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:52.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.398+0000 7f1dd5a00640 1 --2- 172.21.15.27:0/1311550045 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1dd0071550 0x7f1dd0151470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:52.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.398+0000 7f1dd5000640 1 -- 172.21.15.27:0/1311550045 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1dd0071550 msgr2=0x7f1dd0151470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:52.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.398+0000 7f1dd5000640 1 --2- 172.21.15.27:0/1311550045 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1dd0071550 0x7f1dd0151470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:52.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.398+0000 7f1dd5000640 1 -- 172.21.15.27:0/1311550045 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1dd006bc50 msgr2=0x7f1dd0150f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:52.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.398+0000 7f1dd5000640 1 --2- 172.21.15.27:0/1311550045 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1dd006bc50 0x7f1dd0150f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:52.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.398+0000 7f1dd5000640 1 -- 172.21.15.27:0/1311550045 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc8009b30 con 0x7f1dd006b050 2024-01-15T18:08:52.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.398+0000 7f1dd5a00640 1 --2- 172.21.15.27:0/1311550045 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1dd0071550 0x7f1dd0151470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:08:52.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.398+0000 7f1dcfe00640 1 --2- 172.21.15.27:0/1311550045 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1dd006bc50 0x7f1dd0150f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:08:52.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.398+0000 7f1dd5000640 1 --2- 172.21.15.27:0/1311550045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dd006b050 0x7f1dd014eb60 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f1dc000c3c0 tx=0x7f1dc000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:52.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.398+0000 7f1dccc00640 1 -- 172.21.15.27:0/1311550045 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc00052d0 con 0x7f1dd006b050 2024-01-15T18:08:52.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.398+0000 7f1dccc00640 1 -- 172.21.15.27:0/1311550045 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc0005430 con 0x7f1dd006b050 2024-01-15T18:08:52.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.398+0000 7f1dccc00640 1 -- 172.21.15.27:0/1311550045 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc0003840 con 0x7f1dd006b050 2024-01-15T18:08:52.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.398+0000 7f1dd7a00640 1 -- 172.21.15.27:0/1311550045 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dd00cdb10 con 0x7f1dd006b050 2024-01-15T18:08:52.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.398+0000 7f1dd7a00640 1 -- 172.21.15.27:0/1311550045 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dd0151b40 con 0x7f1dd006b050 2024-01-15T18:08:52.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.398+0000 7f1dd7a00640 1 -- 172.21.15.27:0/1311550045 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dd004e850 con 0x7f1dd006b050 2024-01-15T18:08:52.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.402+0000 7f1dccc00640 1 -- 172.21.15.27:0/1311550045 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 86721+0+0 (secure 0 0 0) 0x7f1dc00039a0 con 0x7f1dd006b050 2024-01-15T18:08:52.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.402+0000 7f1dccc00640 1 --2- 172.21.15.27:0/1311550045 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1da8066f50 0x7f1da8069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:52.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.402+0000 7f1dccc00640 1 -- 172.21.15.27:0/1311550045 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1dc00843c0 con 0x7f1dd006b050 2024-01-15T18:08:52.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.402+0000 7f1dcfe00640 1 --2- 172.21.15.27:0/1311550045 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1da8066f50 0x7f1da8069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:52.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.402+0000 7f1dcfe00640 1 --2- 172.21.15.27:0/1311550045 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1da8066f50 0x7f1da8069410 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1dd00b2ad0 tx=0x7f1db8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:52.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.406+0000 7f1dccc00640 1 -- 172.21.15.27:0/1311550045 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1dc0050da0 con 0x7f1dd006b050 2024-01-15T18:08:52.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:52.582+0000 7f1dd7a00640 1 -- 172.21.15.27:0/1311550045 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7f1dd015d9f0 con 0x7f1da8066f50 2024-01-15T18:08:53.006 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-01-15T18:08:52.994+0000 7f053dc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-15T18:08:53.107 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-15T18:08:55.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:55.022+0000 7f1dccc00640 1 -- 172.21.15.27:0/1311550045 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f1dc0051170 con 0x7f1dd006b050 2024-01-15T18:08:56.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.034+0000 7f1dccc00640 1 -- 172.21.15.27:0/1311550045 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f1dd015d9f0 con 0x7f1da8066f50 2024-01-15T18:08:56.042 INFO:teuthology.orchestra.run.smithi027.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-15T18:08:56.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.038+0000 7f1dd7a00640 1 -- 172.21.15.27:0/1311550045 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1da8066f50 msgr2=0x7f1da8069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:56.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.038+0000 7f1dd7a00640 1 --2- 172.21.15.27:0/1311550045 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1da8066f50 0x7f1da8069410 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1dd00b2ad0 tx=0x7f1db8008040 comp rx=0 tx=0).stop 2024-01-15T18:08:56.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.038+0000 7f1dd7a00640 1 -- 172.21.15.27:0/1311550045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dd006b050 msgr2=0x7f1dd014eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:56.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.038+0000 7f1dd7a00640 1 --2- 172.21.15.27:0/1311550045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dd006b050 0x7f1dd014eb60 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f1dc000c3c0 tx=0x7f1dc000c880 comp rx=0 tx=0).stop 2024-01-15T18:08:56.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.038+0000 7f1dd7a00640 1 -- 172.21.15.27:0/1311550045 shutdown_connections 2024-01-15T18:08:56.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.038+0000 7f1dd7a00640 1 --2- 172.21.15.27:0/1311550045 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1da8066f50 0x7f1da8069410 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:56.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.038+0000 7f1dd7a00640 1 --2- 172.21.15.27:0/1311550045 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1dd0071550 0x7f1dd0151470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:56.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.038+0000 7f1dd7a00640 1 --2- 172.21.15.27:0/1311550045 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1dd006bc50 0x7f1dd0150f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:56.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.038+0000 7f1dd7a00640 1 --2- 172.21.15.27:0/1311550045 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1dd006b050 0x7f1dd014eb60 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:56.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.038+0000 7f1dd7a00640 1 -- 172.21.15.27:0/1311550045 >> 172.21.15.27:0/1311550045 conn(0x7f1dd00b15b0 msgr2=0x7f1dd014d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:56.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.038+0000 7f1dd7a00640 1 -- 172.21.15.27:0/1311550045 shutdown_connections 2024-01-15T18:08:56.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.038+0000 7f1dd7a00640 1 -- 172.21.15.27:0/1311550045 wait complete. 2024-01-15T18:08:56.085 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T18:08:56.086 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:08:56.183 DEBUG:teuthology.orchestra.run.smithi093:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:08:56.259 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T18:08:56.259 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:08:56.259 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:08:56.274 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:08:56.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.486+0000 7f44ad800640 1 Processor -- start 2024-01-15T18:08:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.486+0000 7f44ad800640 1 -- start start 2024-01-15T18:08:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.490+0000 7f44ad800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f44a81594b0 0x7f44a80b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.490+0000 7f44ad800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44a80b2fe0 0x7f44a80ae110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.490+0000 7f44ad800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44a80adc80 con 0x7f44a80ae650 2024-01-15T18:08:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.490+0000 7f44ad800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44a80ad350 con 0x7f44a81594b0 2024-01-15T18:08:56.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.490+0000 7f44ad800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44a80ad630 con 0x7f44a80b2fe0 2024-01-15T18:08:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.490+0000 7f44a6a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f44a81594b0 0x7f44a80b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.490+0000 7f44a6a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f44a81594b0 0x7f44a80b2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59632/0 (socket says 172.21.15.27:59632) 2024-01-15T18:08:56.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.490+0000 7f44a6a00640 1 -- 172.21.15.27:0/959642235 learned_addr learned my addr 172.21.15.27:0/959642235 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:56.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.494+0000 7f44a6000640 1 --2- 172.21.15.27:0/959642235 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44a80b2fe0 0x7f44a80ae110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:56.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.494+0000 7f44a6a00640 1 -- 172.21.15.27:0/959642235 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44a80b2fe0 msgr2=0x7f44a80ae110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:56.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.494+0000 7f44a6a00640 1 --2- 172.21.15.27:0/959642235 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44a80b2fe0 0x7f44a80ae110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:56.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.494+0000 7f44a6a00640 1 -- 172.21.15.27:0/959642235 >> v1:172.21.15.27:6789/0 conn(0x7f44a80ae650 legacy=0x7f44a806c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:56.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.494+0000 7f44a6a00640 1 -- 172.21.15.27:0/959642235 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a80ad910 con 0x7f44a81594b0 2024-01-15T18:08:56.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.498+0000 7f44a6a00640 1 --2- 172.21.15.27:0/959642235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a81594b0 0x7f44a80b2aa0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f4490007360 tx=0x7f449001d420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82511d0c2d51daaf server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:56.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.498+0000 7f44a5600640 1 -- 172.21.15.27:0/959642235 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4490008be0 con 0x7f44a81594b0 2024-01-15T18:08:56.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.498+0000 7f44a5600640 1 -- 172.21.15.27:0/959642235 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4490008d40 con 0x7f44a81594b0 2024-01-15T18:08:56.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.498+0000 7f44a5600640 1 -- 172.21.15.27:0/959642235 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449000a2f0 con 0x7f44a81594b0 2024-01-15T18:08:56.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.502+0000 7f44ad800640 1 -- 172.21.15.27:0/959642235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a81594b0 msgr2=0x7f44a80b2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:56.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.502+0000 7f44ad800640 1 --2- 172.21.15.27:0/959642235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a81594b0 0x7f44a80b2aa0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f4490007360 tx=0x7f449001d420 comp rx=0 tx=0).stop 2024-01-15T18:08:56.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.506+0000 7f44ad800640 1 -- 172.21.15.27:0/959642235 shutdown_connections 2024-01-15T18:08:56.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.506+0000 7f44ad800640 1 --2- 172.21.15.27:0/959642235 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44a80b2fe0 0x7f44a80ae110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:56.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.506+0000 7f44ad800640 1 --2- 172.21.15.27:0/959642235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a81594b0 0x7f44a80b2aa0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:56.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.506+0000 7f44ad800640 1 -- 172.21.15.27:0/959642235 >> 172.21.15.27:0/959642235 conn(0x7f44a8070aa0 msgr2=0x7f44a8071950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:56.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.506+0000 7f44ad800640 1 -- 172.21.15.27:0/959642235 shutdown_connections 2024-01-15T18:08:56.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.506+0000 7f44ad800640 1 -- 172.21.15.27:0/959642235 wait complete. 2024-01-15T18:08:56.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.510+0000 7f44ad800640 1 Processor -- start 2024-01-15T18:08:56.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.510+0000 7f44ad800640 1 -- start start 2024-01-15T18:08:56.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.510+0000 7f44ad800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44a80ae650 0x7f44a8156c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:56.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.510+0000 7f44ad800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a80b2fe0 0x7f44a8154670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:56.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.510+0000 7f44ad800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44a81594b0 0x7f44a8206480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:56.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.514+0000 7f44ad800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a8099690 con 0x7f44a80b2fe0 2024-01-15T18:08:56.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.514+0000 7f44ad800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a81578f0 con 0x7f44a80ae650 2024-01-15T18:08:56.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.514+0000 7f44ad800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44a809d160 con 0x7f44a81594b0 2024-01-15T18:08:56.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.514+0000 7f44a7400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44a81594b0 0x7f44a8206480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:56.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.514+0000 7f44a7400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44a81594b0 0x7f44a8206480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:49210/0 (socket says 172.21.15.27:49210) 2024-01-15T18:08:56.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.514+0000 7f44a7400640 1 -- 172.21.15.27:0/3035535727 learned_addr learned my addr 172.21.15.27:0/3035535727 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:56.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.514+0000 7f44a6a00640 1 --2- 172.21.15.27:0/3035535727 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44a80ae650 0x7f44a8156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:56.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.514+0000 7f44a6000640 1 --2- 172.21.15.27:0/3035535727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a80b2fe0 0x7f44a8154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:56.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.514+0000 7f44a7400640 1 -- 172.21.15.27:0/3035535727 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44a80ae650 msgr2=0x7f44a8156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:56.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.514+0000 7f44a7400640 1 --2- 172.21.15.27:0/3035535727 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44a80ae650 0x7f44a8156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:56.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.514+0000 7f44a7400640 1 -- 172.21.15.27:0/3035535727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a80b2fe0 msgr2=0x7f44a8154670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:56.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.514+0000 7f44a7400640 1 --2- 172.21.15.27:0/3035535727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a80b2fe0 0x7f44a8154670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:56.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.514+0000 7f44a7400640 1 -- 172.21.15.27:0/3035535727 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4490007040 con 0x7f44a81594b0 2024-01-15T18:08:56.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.518+0000 7f44a7400640 1 --2- 172.21.15.27:0/3035535727 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44a81594b0 0x7f44a8206480 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f44a000b970 tx=0x7f44a000be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:56.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.518+0000 7f449ea00640 1 -- 172.21.15.27:0/3035535727 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a000cb40 con 0x7f44a81594b0 2024-01-15T18:08:56.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.518+0000 7f449ea00640 1 -- 172.21.15.27:0/3035535727 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a000cca0 con 0x7f44a81594b0 2024-01-15T18:08:56.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.518+0000 7f449ea00640 1 -- 172.21.15.27:0/3035535727 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a0014680 con 0x7f44a81594b0 2024-01-15T18:08:56.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.518+0000 7f44ad800640 1 -- 172.21.15.27:0/3035535727 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a8157330 con 0x7f44a81594b0 2024-01-15T18:08:56.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.518+0000 7f44ad800640 1 -- 172.21.15.27:0/3035535727 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a80992b0 con 0x7f44a81594b0 2024-01-15T18:08:56.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.522+0000 7f4497e00640 1 -- 172.21.15.27:0/3035535727 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44a80997d0 con 0x7f44a81594b0 2024-01-15T18:08:56.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.522+0000 7f449ea00640 1 -- 172.21.15.27:0/3035535727 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f44a0003980 con 0x7f44a81594b0 2024-01-15T18:08:56.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.526+0000 7f449ea00640 1 --2- 172.21.15.27:0/3035535727 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4480066f60 0x7f4480069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:56.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.526+0000 7f449ea00640 1 -- 172.21.15.27:0/3035535727 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f44a00848e0 con 0x7f44a81594b0 2024-01-15T18:08:56.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.526+0000 7f44a6a00640 1 --2- 172.21.15.27:0/3035535727 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4480066f60 0x7f4480069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:56.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.534+0000 7f44a6a00640 1 --2- 172.21.15.27:0/3035535727 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4480066f60 0x7f4480069420 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f44900af000 tx=0x7f44900b6040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:56.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.534+0000 7f449ea00640 1 -- 172.21.15.27:0/3035535727 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f44a0051710 con 0x7f44a81594b0 2024-01-15T18:08:56.611 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:08:56.615 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:08:56.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.906+0000 7f4497e00640 1 -- 172.21.15.27:0/3035535727 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44a80997d0 con 0x7f44a81594b0 2024-01-15T18:08:56.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.910+0000 7f449ea00640 1 -- 172.21.15.27:0/3035535727 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+12933 (secure 0 0 0) 0x7f44a80997d0 con 0x7f44a81594b0 2024-01-15T18:08:56.916 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:08:56.920 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":50,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:08:56.019879+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6835/1005016790":"2024-01-16T18:08:52.999612+0000","172.21.15.27:6833/1005016790":"2024-01-16T18:08:52.999612+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:08:56.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.918+0000 7f4497e00640 1 -- 172.21.15.27:0/3035535727 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4480066f60 msgr2=0x7f4480069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:56.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.918+0000 7f4497e00640 1 --2- 172.21.15.27:0/3035535727 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4480066f60 0x7f4480069420 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f44900af000 tx=0x7f44900b6040 comp rx=0 tx=0).stop 2024-01-15T18:08:56.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.922+0000 7f4497e00640 1 -- 172.21.15.27:0/3035535727 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44a81594b0 msgr2=0x7f44a8206480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:56.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.922+0000 7f4497e00640 1 --2- 172.21.15.27:0/3035535727 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44a81594b0 0x7f44a8206480 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f44a000b970 tx=0x7f44a000be30 comp rx=0 tx=0).stop 2024-01-15T18:08:56.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.922+0000 7f4497e00640 1 -- 172.21.15.27:0/3035535727 shutdown_connections 2024-01-15T18:08:56.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.922+0000 7f4497e00640 1 --2- 172.21.15.27:0/3035535727 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4480066f60 0x7f4480069420 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:56.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.922+0000 7f4497e00640 1 --2- 172.21.15.27:0/3035535727 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44a81594b0 0x7f44a8206480 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:56.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.922+0000 7f4497e00640 1 --2- 172.21.15.27:0/3035535727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44a80b2fe0 0x7f44a8154670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:56.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.922+0000 7f4497e00640 1 --2- 172.21.15.27:0/3035535727 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44a80ae650 0x7f44a8156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:56.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.922+0000 7f4497e00640 1 -- 172.21.15.27:0/3035535727 >> 172.21.15.27:0/3035535727 conn(0x7f44a8070aa0 msgr2=0x7f44a80ce340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:56.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.922+0000 7f4497e00640 1 -- 172.21.15.27:0/3035535727 shutdown_connections 2024-01-15T18:08:56.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:56.922+0000 7f4497e00640 1 -- 172.21.15.27:0/3035535727 wait complete. 2024-01-15T18:08:56.971 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:08:56.971 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:08:57.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.210+0000 7fe88f600640 1 Processor -- start 2024-01-15T18:08:57.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.210+0000 7fe88f600640 1 -- start start 2024-01-15T18:08:57.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.214+0000 7fe88f600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fe888161840 0x7fe888162700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:57.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.214+0000 7fe88f600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8880ac5a0 0x7fe88806c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:57.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.214+0000 7fe88f600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8880b2e40 con 0x7fe8880b32d0 2024-01-15T18:08:57.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.214+0000 7fe88f600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe888162c40 con 0x7fe888161840 2024-01-15T18:08:57.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.214+0000 7fe88f600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8880b2230 con 0x7fe8880ac5a0 2024-01-15T18:08:57.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.214+0000 7fe887e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe8880b32d0 0x7fe8880abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48442/0 (socket says 172.21.15.27:48442) 2024-01-15T18:08:57.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.214+0000 7fe887e00640 1 -- 172.21.15.27:0/2723290613 learned_addr learned my addr 172.21.15.27:0/2723290613 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:57.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.214+0000 7fe88cc00640 1 --2- 172.21.15.27:0/2723290613 >> v2:172.21.15.93:3300/0 conn(0x7fe888161840 0x7fe888162700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:57.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.214+0000 7fe88d600640 1 --2- 172.21.15.27:0/2723290613 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8880ac5a0 0x7fe88806c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:57.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.214+0000 7fe887400640 1 -- 172.21.15.27:0/2723290613 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1860569843 0 0) 0x7fe8880b2e40 con 0x7fe8880b32d0 2024-01-15T18:08:57.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.214+0000 7fe887400640 1 -- 172.21.15.27:0/2723290613 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe868003530 con 0x7fe8880b32d0 2024-01-15T18:08:57.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.218+0000 7fe887400640 1 -- 172.21.15.27:0/2723290613 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1884458727 0 0) 0x7fe868003530 con 0x7fe8880b32d0 2024-01-15T18:08:57.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.218+0000 7fe887400640 1 -- 172.21.15.27:0/2723290613 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8880ac5a0 msgr2=0x7fe88806c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:57.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.218+0000 7fe887400640 1 --2- 172.21.15.27:0/2723290613 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe8880ac5a0 0x7fe88806c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:57.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.218+0000 7fe887400640 1 -- 172.21.15.27:0/2723290613 >> v2:172.21.15.93:3300/0 conn(0x7fe888161840 msgr2=0x7fe888162700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:57.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.218+0000 7fe887400640 1 --2- 172.21.15.27:0/2723290613 >> v2:172.21.15.93:3300/0 conn(0x7fe888161840 0x7fe888162700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:57.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.218+0000 7fe887400640 1 -- 172.21.15.27:0/2723290613 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8880b2510 con 0x7fe8880b32d0 2024-01-15T18:08:57.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.218+0000 7fe887400640 1 -- 172.21.15.27:0/2723290613 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fe870001680 con 0x7fe8880b32d0 2024-01-15T18:08:57.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.218+0000 7fe887400640 1 -- 172.21.15.27:0/2723290613 >> v1:172.21.15.27:6789/0 conn(0x7fe8880b32d0 legacy=0x7fe8880abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:57.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.218+0000 7fe887400640 1 --2- 172.21.15.27:0/2723290613 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe868003980 0x7fe868003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:57.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.218+0000 7fe887400640 1 --2- 172.21.15.27:0/2723290613 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe868004370 0x7fe868004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:57.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.218+0000 7fe887400640 1 --2- 172.21.15.27:0/2723290613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe868004ec0 0x7fe86800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:57.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.218+0000 7fe887400640 1 -- 172.21.15.27:0/2723290613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe888162c40 con 0x7fe868004ec0 2024-01-15T18:08:57.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.218+0000 7fe887400640 1 -- 172.21.15.27:0/2723290613 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8880b2230 con 0x7fe868004370 2024-01-15T18:08:57.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.218+0000 7fe887400640 1 -- 172.21.15.27:0/2723290613 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe86800bed0 con 0x7fe868003980 2024-01-15T18:08:57.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.218+0000 7fe887e00640 1 --2- 172.21.15.27:0/2723290613 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe868004370 0x7fe868004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:57.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.218+0000 7fe887e00640 1 -- 172.21.15.27:0/2723290613 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe868003980 msgr2=0x7fe868003da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-15T18:08:57.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.218+0000 7fe887e00640 1 --2- 172.21.15.27:0/2723290613 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe868003980 0x7fe868003da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:57.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.218+0000 7fe887e00640 1 -- 172.21.15.27:0/2723290613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe868004ec0 msgr2=0x7fe86800b990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-15T18:08:57.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.218+0000 7fe887e00640 1 --2- 172.21.15.27:0/2723290613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe868004ec0 0x7fe86800b990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:57.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.218+0000 7fe887e00640 1 -- 172.21.15.27:0/2723290613 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8880b2510 con 0x7fe868004370 2024-01-15T18:08:57.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.222+0000 7fe887e00640 1 --2- 172.21.15.27:0/2723290613 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe868004370 0x7fe868004850 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe87000c310 tx=0x7fe87001cd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ce19a208b16f73c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:57.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.222+0000 7fe887400640 1 -- 172.21.15.27:0/2723290613 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8700a5910 con 0x7fe868004370 2024-01-15T18:08:57.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.222+0000 7fe887400640 1 -- 172.21.15.27:0/2723290613 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8700a5a70 con 0x7fe868004370 2024-01-15T18:08:57.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.222+0000 7fe887400640 1 -- 172.21.15.27:0/2723290613 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8700a5e60 con 0x7fe868004370 2024-01-15T18:08:57.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.226+0000 7fe88f600640 1 -- 172.21.15.27:0/2723290613 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe868004370 msgr2=0x7fe868004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:57.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.226+0000 7fe88f600640 1 --2- 172.21.15.27:0/2723290613 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe868004370 0x7fe868004850 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe87000c310 tx=0x7fe87001cd80 comp rx=0 tx=0).stop 2024-01-15T18:08:57.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.226+0000 7fe88d600640 1 -- 172.21.15.27:0/2723290613 reap_dead start 2024-01-15T18:08:57.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.226+0000 7fe88f600640 1 -- 172.21.15.27:0/2723290613 shutdown_connections 2024-01-15T18:08:57.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.226+0000 7fe88f600640 1 -- 172.21.15.27:0/2723290613 >> 172.21.15.27:0/2723290613 conn(0x7fe8880b0bb0 msgr2=0x7fe8880b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:57.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.226+0000 7fe88f600640 1 -- 172.21.15.27:0/2723290613 shutdown_connections 2024-01-15T18:08:57.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.226+0000 7fe88f600640 1 -- 172.21.15.27:0/2723290613 wait complete. 2024-01-15T18:08:57.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.230+0000 7fe88f600640 1 Processor -- start 2024-01-15T18:08:57.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.230+0000 7fe88f600640 1 -- start start 2024-01-15T18:08:57.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.234+0000 7fe88f600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe868004370 0x7fe88814f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:57.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.234+0000 7fe88f600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe88814e930 0x7fe8881638d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:57.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.234+0000 7fe88f600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe88814a770 0x7fe88814ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:57.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.234+0000 7fe88cc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe868004370 0x7fe88814f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:57.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.234+0000 7fe88cc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe868004370 0x7fe88814f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:49234/0 (socket says 172.21.15.27:49234) 2024-01-15T18:08:57.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.234+0000 7fe887e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe88814e930 0x7fe8881638d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:57.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.234+0000 7fe887e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe88814e930 0x7fe8881638d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:36578/0 (socket says 172.21.15.27:36578) 2024-01-15T18:08:57.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.234+0000 7fe88d600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe88814a770 0x7fe88814ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:57.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.234+0000 7fe88d600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe88814a770 0x7fe88814ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59642/0 (socket says 172.21.15.27:59642) 2024-01-15T18:08:57.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.234+0000 7fe88cc00640 1 -- 172.21.15.27:0/3195670491 learned_addr learned my addr 172.21.15.27:0/3195670491 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:57.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.234+0000 7fe88f600640 1 -- 172.21.15.27:0/3195670491 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8881548e0 con 0x7fe88814a770 2024-01-15T18:08:57.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.234+0000 7fe88f600640 1 -- 172.21.15.27:0/3195670491 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88814b160 con 0x7fe88814e930 2024-01-15T18:08:57.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.234+0000 7fe88f600640 1 -- 172.21.15.27:0/3195670491 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8881542b0 con 0x7fe868004370 2024-01-15T18:08:57.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.238+0000 7fe887e00640 1 -- 172.21.15.27:0/3195670491 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe868004370 msgr2=0x7fe88814f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:57.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.238+0000 7fe887e00640 1 --2- 172.21.15.27:0/3195670491 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe868004370 0x7fe88814f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:57.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.238+0000 7fe887e00640 1 -- 172.21.15.27:0/3195670491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe88814a770 msgr2=0x7fe88814ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:57.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.238+0000 7fe887e00640 1 --2- 172.21.15.27:0/3195670491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe88814a770 0x7fe88814ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:57.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.238+0000 7fe887e00640 1 -- 172.21.15.27:0/3195670491 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe88814fad0 con 0x7fe88814e930 2024-01-15T18:08:57.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.238+0000 7fe887e00640 1 --2- 172.21.15.27:0/3195670491 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe88814e930 0x7fe8881638d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe868005730 tx=0x7fe868005800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:57.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.238+0000 7fe884c00640 1 -- 172.21.15.27:0/3195670491 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe870004cc0 con 0x7fe88814e930 2024-01-15T18:08:57.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.238+0000 7fe884c00640 1 -- 172.21.15.27:0/3195670491 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe870004e20 con 0x7fe88814e930 2024-01-15T18:08:57.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.238+0000 7fe884c00640 1 -- 172.21.15.27:0/3195670491 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe870007310 con 0x7fe88814e930 2024-01-15T18:08:57.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.238+0000 7fe88f600640 1 -- 172.21.15.27:0/3195670491 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe888099a10 con 0x7fe88814e930 2024-01-15T18:08:57.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.238+0000 7fe88f600640 1 -- 172.21.15.27:0/3195670491 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe888151300 con 0x7fe88814e930 2024-01-15T18:08:57.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.238+0000 7fe88f600640 1 -- 172.21.15.27:0/3195670491 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8880b1b10 con 0x7fe88814e930 2024-01-15T18:08:57.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.242+0000 7fe884c00640 1 -- 172.21.15.27:0/3195670491 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7fe8700b5070 con 0x7fe88814e930 2024-01-15T18:08:57.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.242+0000 7fe884c00640 1 --2- 172.21.15.27:0/3195670491 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe860066ee0 0x7fe8600693a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:57.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.242+0000 7fe884c00640 1 -- 172.21.15.27:0/3195670491 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fe87011de90 con 0x7fe88814e930 2024-01-15T18:08:57.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.242+0000 7fe88cc00640 1 --2- 172.21.15.27:0/3195670491 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe860066ee0 0x7fe8600693a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:57.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.242+0000 7fe88cc00640 1 --2- 172.21.15.27:0/3195670491 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe860066ee0 0x7fe8600693a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe878009be0 tx=0x7fe878009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:57.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.246+0000 7fe884c00640 1 -- 172.21.15.27:0/3195670491 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe8700eacc0 con 0x7fe88814e930 2024-01-15T18:08:57.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.426+0000 7fe88f600640 1 -- 172.21.15.27:0/3195670491 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe888154a20 con 0x7fe88814e930 2024-01-15T18:08:57.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.430+0000 7fe884c00640 1 -- 172.21.15.27:0/3195670491 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe888154a20 con 0x7fe88814e930 2024-01-15T18:08:57.434 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:08:57.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.430+0000 7fe88f600640 1 -- 172.21.15.27:0/3195670491 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe860066ee0 msgr2=0x7fe8600693a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:57.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.430+0000 7fe88f600640 1 --2- 172.21.15.27:0/3195670491 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe860066ee0 0x7fe8600693a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe878009be0 tx=0x7fe878009190 comp rx=0 tx=0).stop 2024-01-15T18:08:57.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.430+0000 7fe88f600640 1 -- 172.21.15.27:0/3195670491 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe88814e930 msgr2=0x7fe8881638d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:57.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.430+0000 7fe88f600640 1 --2- 172.21.15.27:0/3195670491 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe88814e930 0x7fe8881638d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe868005730 tx=0x7fe868005800 comp rx=0 tx=0).stop 2024-01-15T18:08:57.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.430+0000 7fe88f600640 1 -- 172.21.15.27:0/3195670491 shutdown_connections 2024-01-15T18:08:57.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.430+0000 7fe88f600640 1 --2- 172.21.15.27:0/3195670491 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe860066ee0 0x7fe8600693a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:57.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.430+0000 7fe88f600640 1 --2- 172.21.15.27:0/3195670491 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe88814a770 0x7fe88814ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:57.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.430+0000 7fe88f600640 1 --2- 172.21.15.27:0/3195670491 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe88814e930 0x7fe8881638d0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:57.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.430+0000 7fe88f600640 1 --2- 172.21.15.27:0/3195670491 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe868004370 0x7fe88814f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:57.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.430+0000 7fe88f600640 1 -- 172.21.15.27:0/3195670491 >> 172.21.15.27:0/3195670491 conn(0x7fe8880b0bb0 msgr2=0x7fe8880708b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:57.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.430+0000 7fe88f600640 1 -- 172.21.15.27:0/3195670491 shutdown_connections 2024-01-15T18:08:57.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.430+0000 7fe88f600640 1 -- 172.21.15.27:0/3195670491 wait complete. 2024-01-15T18:08:57.459 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:08:57.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.594+0000 7f6257200640 1 Processor -- start 2024-01-15T18:08:57.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.594+0000 7f6257200640 1 -- start start 2024-01-15T18:08:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.594+0000 7f6257200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f625015b290 0x7f62500d19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.594+0000 7f6257200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f625015b670 con 0x7f62500cde30 2024-01-15T18:08:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.594+0000 7f6257200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62500cc470 con 0x7f62500cfbc0 2024-01-15T18:08:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.594+0000 7f6257200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62500ccd10 con 0x7f625015b290 2024-01-15T18:08:57.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.594+0000 7f624fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f625015b290 0x7f62500d19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:57.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.594+0000 7f6255200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f62500cde30 0x7f62500b0a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48456/0 (socket says 172.21.15.27:48456) 2024-01-15T18:08:57.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.594+0000 7f6255200640 1 -- 172.21.15.27:0/2202313514 learned_addr learned my addr 172.21.15.27:0/2202313514 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:57.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.594+0000 7f624fe00640 1 -- 172.21.15.27:0/2202313514 >> v1:172.21.15.93:6789/0 conn(0x7f62500cfbc0 legacy=0x7f62500cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:57.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.594+0000 7f624fe00640 1 -- 172.21.15.27:0/2202313514 >> v1:172.21.15.27:6789/0 conn(0x7f62500cde30 legacy=0x7f62500b0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:57.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.594+0000 7f624fe00640 1 -- 172.21.15.27:0/2202313514 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62500ce630 con 0x7f625015b290 2024-01-15T18:08:57.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.598+0000 7f624fe00640 1 --2- 172.21.15.27:0/2202313514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f625015b290 0x7f62500d19b0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f62400097c0 tx=0x7f624001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d540b7ddc2de0c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:57.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.598+0000 7f624ea00640 1 -- 172.21.15.27:0/2202313514 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62400ab650 con 0x7f625015b290 2024-01-15T18:08:57.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.598+0000 7f624ea00640 1 -- 172.21.15.27:0/2202313514 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62400ab7b0 con 0x7f625015b290 2024-01-15T18:08:57.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.598+0000 7f624ea00640 1 -- 172.21.15.27:0/2202313514 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62400abb40 con 0x7f625015b290 2024-01-15T18:08:57.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.598+0000 7f6257200640 1 -- 172.21.15.27:0/2202313514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f625015b290 msgr2=0x7f62500d19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:57.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.598+0000 7f6257200640 1 --2- 172.21.15.27:0/2202313514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f625015b290 0x7f62500d19b0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f62400097c0 tx=0x7f624001c450 comp rx=0 tx=0).stop 2024-01-15T18:08:57.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.602+0000 7f6257200640 1 -- 172.21.15.27:0/2202313514 shutdown_connections 2024-01-15T18:08:57.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.602+0000 7f6257200640 1 --2- 172.21.15.27:0/2202313514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f625015b290 0x7f62500d19b0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:57.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.602+0000 7f6257200640 1 -- 172.21.15.27:0/2202313514 >> 172.21.15.27:0/2202313514 conn(0x7f62500d0e90 msgr2=0x7f62500d12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:57.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.602+0000 7f6257200640 1 -- 172.21.15.27:0/2202313514 shutdown_connections 2024-01-15T18:08:57.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.602+0000 7f6257200640 1 -- 172.21.15.27:0/2202313514 wait complete. 2024-01-15T18:08:57.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.602+0000 7f6257200640 1 Processor -- start 2024-01-15T18:08:57.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.602+0000 7f6257200640 1 -- start start 2024-01-15T18:08:57.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.602+0000 7f6257200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f62500cde30 0x7f6250151b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:57.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.602+0000 7f6257200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62500cfbc0 0x7f625006d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:57.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.602+0000 7f6257200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f625015b290 0x7f62501f5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:57.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.602+0000 7f6257200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62501520b0 con 0x7f62500cfbc0 2024-01-15T18:08:57.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.602+0000 7f6257200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625006bfb0 con 0x7f625015b290 2024-01-15T18:08:57.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.602+0000 7f6257200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f625006c290 con 0x7f62500cde30 2024-01-15T18:08:57.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.602+0000 7f624f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62500cfbc0 0x7f625006d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:57.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.602+0000 7f624f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62500cfbc0 0x7f625006d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59662/0 (socket says 172.21.15.27:59662) 2024-01-15T18:08:57.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.602+0000 7f624f400640 1 -- 172.21.15.27:0/2262652887 learned_addr learned my addr 172.21.15.27:0/2262652887 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:57.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.602+0000 7f624fe00640 1 --2- 172.21.15.27:0/2262652887 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f62500cde30 0x7f6250151b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:57.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.602+0000 7f6255200640 1 --2- 172.21.15.27:0/2262652887 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f625015b290 0x7f62501f5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:57.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.602+0000 7f624f400640 1 -- 172.21.15.27:0/2262652887 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f62500cde30 msgr2=0x7f6250151b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:57.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.602+0000 7f624f400640 1 --2- 172.21.15.27:0/2262652887 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f62500cde30 0x7f6250151b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:57.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.602+0000 7f624f400640 1 -- 172.21.15.27:0/2262652887 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f625015b290 msgr2=0x7f62501f5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:57.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.602+0000 7f624f400640 1 --2- 172.21.15.27:0/2262652887 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f625015b290 0x7f62501f5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:57.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.602+0000 7f624f400640 1 -- 172.21.15.27:0/2262652887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6240009420 con 0x7f62500cfbc0 2024-01-15T18:08:57.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.606+0000 7f624f400640 1 --2- 172.21.15.27:0/2262652887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62500cfbc0 0x7f625006d050 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f623800c4d0 tx=0x7f623800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:57.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.606+0000 7f6247e00640 1 -- 172.21.15.27:0/2262652887 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62380052d0 con 0x7f62500cfbc0 2024-01-15T18:08:57.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.606+0000 7f6247e00640 1 -- 172.21.15.27:0/2262652887 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6238005430 con 0x7f62500cfbc0 2024-01-15T18:08:57.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.606+0000 7f6247e00640 1 -- 172.21.15.27:0/2262652887 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6238012830 con 0x7f62500cfbc0 2024-01-15T18:08:57.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.606+0000 7f6257200640 1 -- 172.21.15.27:0/2262652887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f625006c570 con 0x7f62500cfbc0 2024-01-15T18:08:57.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.606+0000 7f6257200640 1 -- 172.21.15.27:0/2262652887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f625006cb30 con 0x7f62500cfbc0 2024-01-15T18:08:57.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.606+0000 7f6257200640 1 -- 172.21.15.27:0/2262652887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f625004e850 con 0x7f62500cfbc0 2024-01-15T18:08:57.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.606+0000 7f6247e00640 1 -- 172.21.15.27:0/2262652887 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f6238012990 con 0x7f62500cfbc0 2024-01-15T18:08:57.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.606+0000 7f6247e00640 1 --2- 172.21.15.27:0/2262652887 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6228066ec0 0x7f6228069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:57.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.606+0000 7f6247e00640 1 -- 172.21.15.27:0/2262652887 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f6238084570 con 0x7f62500cfbc0 2024-01-15T18:08:57.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.606+0000 7f624fe00640 1 --2- 172.21.15.27:0/2262652887 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6228066ec0 0x7f6228069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:57.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.610+0000 7f624fe00640 1 --2- 172.21.15.27:0/2262652887 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6228066ec0 0x7f6228069380 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f62400af000 tx=0x7f62400a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:57.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.614+0000 7f6247e00640 1 -- 172.21.15.27:0/2262652887 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6238051290 con 0x7f62500cfbc0 2024-01-15T18:08:57.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.782+0000 7f6257200640 1 -- 172.21.15.27:0/2262652887 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6250073ab0 con 0x7f62500cfbc0 2024-01-15T18:08:57.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.782+0000 7f6247e00640 1 -- 172.21.15.27:0/2262652887 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v50) v1 ==== 80+0+122 (secure 0 0 0) 0x7f62380548b0 con 0x7f62500cfbc0 2024-01-15T18:08:57.789 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6835/1005016790 2024-01-16T18:08:52.999612+0000 2024-01-15T18:08:57.789 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6833/1005016790 2024-01-16T18:08:52.999612+0000 2024-01-15T18:08:57.789 INFO:teuthology.orchestra.run.smithi027.stderr:listed 2 entries 2024-01-15T18:08:57.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.786+0000 7f6257200640 1 -- 172.21.15.27:0/2262652887 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6228066ec0 msgr2=0x7f6228069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:57.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.786+0000 7f6257200640 1 --2- 172.21.15.27:0/2262652887 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6228066ec0 0x7f6228069380 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f62400af000 tx=0x7f62400a7000 comp rx=0 tx=0).stop 2024-01-15T18:08:57.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.786+0000 7f6257200640 1 -- 172.21.15.27:0/2262652887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62500cfbc0 msgr2=0x7f625006d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:57.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.786+0000 7f6257200640 1 --2- 172.21.15.27:0/2262652887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62500cfbc0 0x7f625006d050 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f623800c4d0 tx=0x7f623800c990 comp rx=0 tx=0).stop 2024-01-15T18:08:57.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.786+0000 7f6257200640 1 -- 172.21.15.27:0/2262652887 shutdown_connections 2024-01-15T18:08:57.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.786+0000 7f6257200640 1 --2- 172.21.15.27:0/2262652887 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6228066ec0 0x7f6228069380 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:57.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.786+0000 7f6257200640 1 --2- 172.21.15.27:0/2262652887 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f625015b290 0x7f62501f5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:57.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.786+0000 7f6257200640 1 --2- 172.21.15.27:0/2262652887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62500cfbc0 0x7f625006d050 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:57.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.786+0000 7f6257200640 1 --2- 172.21.15.27:0/2262652887 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f62500cde30 0x7f6250151b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:57.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.786+0000 7f6257200640 1 -- 172.21.15.27:0/2262652887 >> 172.21.15.27:0/2262652887 conn(0x7f62500d0e90 msgr2=0x7f625015de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:57.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.786+0000 7f6257200640 1 -- 172.21.15.27:0/2262652887 shutdown_connections 2024-01-15T18:08:57.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:57.786+0000 7f6257200640 1 -- 172.21.15.27:0/2262652887 wait complete. 2024-01-15T18:08:57.812 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-01-15T18:08:57.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:08:57.812 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-15T18:08:57.833 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T18:08:57.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:08:57.833 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-15T18:08:57.840 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[38897]: fuse finished with error 0 and tester_r 0 2024-01-15T18:08:57.968 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:08:57.968 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:08:57.977 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:08:58.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.146+0000 7f9b8ae00640 1 Processor -- start 2024-01-15T18:08:58.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.146+0000 7f9b8ae00640 1 -- start start 2024-01-15T18:08:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.146+0000 7f9b8ae00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9b840cf9b0 0x7f9b840b10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.146+0000 7f9b8ae00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b840cedc0 0x7f9b840b1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.146+0000 7f9b8ae00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b840b38b0 con 0x7f9b8415e7d0 2024-01-15T18:08:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.146+0000 7f9b8ae00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b840cd550 con 0x7f9b840cf9b0 2024-01-15T18:08:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.146+0000 7f9b8ae00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b840cd270 con 0x7f9b840cedc0 2024-01-15T18:08:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.146+0000 7f9b83400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9b840cf9b0 0x7f9b840b10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.146+0000 7f9b83e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9b8415e7d0 0x7f9b840cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48470/0 (socket says 172.21.15.27:48470) 2024-01-15T18:08:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.146+0000 7f9b83e00640 1 -- 172.21.15.27:0/2866755387 learned_addr learned my addr 172.21.15.27:0/2866755387 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.146+0000 7f9b88e00640 1 --2- 172.21.15.27:0/2866755387 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b840cedc0 0x7f9b840b1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.146+0000 7f9b83400640 1 -- 172.21.15.27:0/2866755387 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b840cedc0 msgr2=0x7f9b840b1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:58.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.146+0000 7f9b83400640 1 --2- 172.21.15.27:0/2866755387 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b840cedc0 0x7f9b840b1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.146+0000 7f9b83400640 1 -- 172.21.15.27:0/2866755387 >> v1:172.21.15.27:6789/0 conn(0x7f9b8415e7d0 legacy=0x7f9b840cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:58.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.146+0000 7f9b83400640 1 -- 172.21.15.27:0/2866755387 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b840b35d0 con 0x7f9b840cf9b0 2024-01-15T18:08:58.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.150+0000 7f9b83400640 1 --2- 172.21.15.27:0/2866755387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b840cf9b0 0x7f9b840b10c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c0097c0 tx=0x7f9b6c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=871fd852dae9157e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:58.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.150+0000 7f9b82a00640 1 -- 172.21.15.27:0/2866755387 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c0ab600 con 0x7f9b840cf9b0 2024-01-15T18:08:58.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.150+0000 7f9b82a00640 1 -- 172.21.15.27:0/2866755387 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b6c0a7050 con 0x7f9b840cf9b0 2024-01-15T18:08:58.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.150+0000 7f9b82a00640 1 -- 172.21.15.27:0/2866755387 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c0ab980 con 0x7f9b840cf9b0 2024-01-15T18:08:58.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.150+0000 7f9b8ae00640 1 -- 172.21.15.27:0/2866755387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b840cf9b0 msgr2=0x7f9b840b10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:58.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.150+0000 7f9b8ae00640 1 --2- 172.21.15.27:0/2866755387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b840cf9b0 0x7f9b840b10c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c0097c0 tx=0x7f9b6c01c480 comp rx=0 tx=0).stop 2024-01-15T18:08:58.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.154+0000 7f9b8ae00640 1 -- 172.21.15.27:0/2866755387 shutdown_connections 2024-01-15T18:08:58.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.154+0000 7f9b8ae00640 1 --2- 172.21.15.27:0/2866755387 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b840cedc0 0x7f9b840b1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.154+0000 7f9b8ae00640 1 --2- 172.21.15.27:0/2866755387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b840cf9b0 0x7f9b840b10c0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.154+0000 7f9b8ae00640 1 -- 172.21.15.27:0/2866755387 >> 172.21.15.27:0/2866755387 conn(0x7f9b840b2a10 msgr2=0x7f9b840b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:58.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.154+0000 7f9b8ae00640 1 -- 172.21.15.27:0/2866755387 shutdown_connections 2024-01-15T18:08:58.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.154+0000 7f9b8ae00640 1 -- 172.21.15.27:0/2866755387 wait complete. 2024-01-15T18:08:58.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.154+0000 7f9b8ae00640 1 Processor -- start 2024-01-15T18:08:58.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.154+0000 7f9b8ae00640 1 -- start start 2024-01-15T18:08:58.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.154+0000 7f9b8ae00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9b840cedc0 0x7f9b840b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:58.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.154+0000 7f9b8ae00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b840cf9b0 0x7f9b841592d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:58.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.154+0000 7f9b8ae00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b8415e7d0 0x7f9b84151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:58.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.154+0000 7f9b8ae00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b840b40c0 con 0x7f9b840cf9b0 2024-01-15T18:08:58.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.154+0000 7f9b8ae00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b84159810 con 0x7f9b840cedc0 2024-01-15T18:08:58.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.154+0000 7f9b8ae00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b840adeb0 con 0x7f9b8415e7d0 2024-01-15T18:08:58.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.154+0000 7f9b83400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b840cf9b0 0x7f9b841592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:58.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.154+0000 7f9b83400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b840cf9b0 0x7f9b841592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59682/0 (socket says 172.21.15.27:59682) 2024-01-15T18:08:58.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.154+0000 7f9b83400640 1 -- 172.21.15.27:0/1200969312 learned_addr learned my addr 172.21.15.27:0/1200969312 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:58.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.154+0000 7f9b88e00640 1 --2- 172.21.15.27:0/1200969312 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b8415e7d0 0x7f9b84151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:58.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.154+0000 7f9b83e00640 1 --2- 172.21.15.27:0/1200969312 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9b840cedc0 0x7f9b840b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:58.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.154+0000 7f9b83400640 1 -- 172.21.15.27:0/1200969312 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b8415e7d0 msgr2=0x7f9b84151fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:58.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.154+0000 7f9b83400640 1 --2- 172.21.15.27:0/1200969312 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b8415e7d0 0x7f9b84151fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.154+0000 7f9b83400640 1 -- 172.21.15.27:0/1200969312 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9b840cedc0 msgr2=0x7f9b840b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:58.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.154+0000 7f9b83400640 1 --2- 172.21.15.27:0/1200969312 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9b840cedc0 0x7f9b840b3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.154+0000 7f9b83400640 1 -- 172.21.15.27:0/1200969312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b6c009420 con 0x7f9b840cf9b0 2024-01-15T18:08:58.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.158+0000 7f9b83400640 1 --2- 172.21.15.27:0/1200969312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b840cf9b0 0x7f9b841592d0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c01a0b0 tx=0x7f9b6c019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:58.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.158+0000 7f9b7be00640 1 -- 172.21.15.27:0/1200969312 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c0b4840 con 0x7f9b840cf9b0 2024-01-15T18:08:58.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.158+0000 7f9b7be00640 1 -- 172.21.15.27:0/1200969312 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b6c0b49c0 con 0x7f9b840cf9b0 2024-01-15T18:08:58.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.158+0000 7f9b8ae00640 1 -- 172.21.15.27:0/1200969312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b8415a2e0 con 0x7f9b840cf9b0 2024-01-15T18:08:58.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.158+0000 7f9b7be00640 1 -- 172.21.15.27:0/1200969312 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c0b4d70 con 0x7f9b840cf9b0 2024-01-15T18:08:58.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.158+0000 7f9b8ae00640 1 -- 172.21.15.27:0/1200969312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b84159d20 con 0x7f9b840cf9b0 2024-01-15T18:08:58.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.158+0000 7f9b8ae00640 1 -- 172.21.15.27:0/1200969312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b840ae020 con 0x7f9b840cf9b0 2024-01-15T18:08:58.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.158+0000 7f9b7be00640 1 -- 172.21.15.27:0/1200969312 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f9b6c0b2040 con 0x7f9b840cf9b0 2024-01-15T18:08:58.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.162+0000 7f9b7be00640 1 --2- 172.21.15.27:0/1200969312 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9b5c066e90 0x7f9b5c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:58.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.162+0000 7f9b7be00640 1 -- 172.21.15.27:0/1200969312 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f9b6c11ef20 con 0x7f9b840cf9b0 2024-01-15T18:08:58.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.162+0000 7f9b83e00640 1 --2- 172.21.15.27:0/1200969312 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9b5c066e90 0x7f9b5c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:58.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.162+0000 7f9b83e00640 1 --2- 172.21.15.27:0/1200969312 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9b5c066e90 0x7f9b5c069350 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9b74005890 tx=0x7f9b74009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:58.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.166+0000 7f9b7be00640 1 -- 172.21.15.27:0/1200969312 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9b6c0ebbc0 con 0x7f9b840cf9b0 2024-01-15T18:08:58.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.322+0000 7f9b8ae00640 1 -- 172.21.15.27:0/1200969312 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b8404e850 con 0x7f9b840cf9b0 2024-01-15T18:08:58.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.326+0000 7f9b7be00640 1 -- 172.21.15.27:0/1200969312 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+12933 (secure 0 0 0) 0x7f9b6c0b0090 con 0x7f9b840cf9b0 2024-01-15T18:08:58.331 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:08:58.331 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":50,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:08:56.019879+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6835/1005016790":"2024-01-16T18:08:52.999612+0000","172.21.15.27:6833/1005016790":"2024-01-16T18:08:52.999612+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:08:58.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.326+0000 7f9b8ae00640 1 -- 172.21.15.27:0/1200969312 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9b5c066e90 msgr2=0x7f9b5c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:58.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.326+0000 7f9b8ae00640 1 --2- 172.21.15.27:0/1200969312 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9b5c066e90 0x7f9b5c069350 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9b74005890 tx=0x7f9b74009040 comp rx=0 tx=0).stop 2024-01-15T18:08:58.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.326+0000 7f9b8ae00640 1 -- 172.21.15.27:0/1200969312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b840cf9b0 msgr2=0x7f9b841592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:58.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.326+0000 7f9b8ae00640 1 --2- 172.21.15.27:0/1200969312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b840cf9b0 0x7f9b841592d0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c01a0b0 tx=0x7f9b6c019220 comp rx=0 tx=0).stop 2024-01-15T18:08:58.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.326+0000 7f9b8ae00640 1 -- 172.21.15.27:0/1200969312 shutdown_connections 2024-01-15T18:08:58.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.326+0000 7f9b8ae00640 1 --2- 172.21.15.27:0/1200969312 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9b5c066e90 0x7f9b5c069350 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.326+0000 7f9b8ae00640 1 --2- 172.21.15.27:0/1200969312 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b8415e7d0 0x7f9b84151fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.326+0000 7f9b8ae00640 1 --2- 172.21.15.27:0/1200969312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b840cf9b0 0x7f9b841592d0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.326+0000 7f9b8ae00640 1 --2- 172.21.15.27:0/1200969312 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9b840cedc0 0x7f9b840b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.326+0000 7f9b8ae00640 1 -- 172.21.15.27:0/1200969312 >> 172.21.15.27:0/1200969312 conn(0x7f9b840b2a10 msgr2=0x7f9b8406fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:58.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.326+0000 7f9b8ae00640 1 -- 172.21.15.27:0/1200969312 shutdown_connections 2024-01-15T18:08:58.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.326+0000 7f9b8ae00640 1 -- 172.21.15.27:0/1200969312 wait complete. 2024-01-15T18:08:58.354 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:08:58.354 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:08:58.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.490+0000 7f4594200640 1 Processor -- start 2024-01-15T18:08:58.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.490+0000 7f4594200640 1 -- start start 2024-01-15T18:08:58.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.494+0000 7f4594200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f458c15d5d0 0x7f458c0d3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:58.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.494+0000 7f4594200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f458c0cf890 0x7f458c0cfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:58.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.494+0000 7f4594200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f458c0cde00 0x7f458c0ce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:58.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.494+0000 7f4594200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f458c0cf5b0 con 0x7f458c0cf890 2024-01-15T18:08:58.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.494+0000 7f4594200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f458c0d0210 con 0x7f458c0cde00 2024-01-15T18:08:58.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.494+0000 7f4594200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f458c0cf2d0 con 0x7f458c15d5d0 2024-01-15T18:08:58.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.494+0000 7f4590e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f458c0cf890 0x7f458c0cfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:58.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.494+0000 7f4590e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f458c0cf890 0x7f458c0cfcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59684/0 (socket says 172.21.15.27:59684) 2024-01-15T18:08:58.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.494+0000 7f4590e00640 1 -- 172.21.15.27:0/1429870904 learned_addr learned my addr 172.21.15.27:0/1429870904 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:58.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.494+0000 7f4591800640 1 --2- 172.21.15.27:0/1429870904 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f458c15d5d0 0x7f458c0d3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:58.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.494+0000 7f4592200640 1 --2- 172.21.15.27:0/1429870904 >> v2:172.21.15.93:3300/0 conn(0x7f458c0cde00 0x7f458c0ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:58.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.494+0000 7f4590e00640 1 -- 172.21.15.27:0/1429870904 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f458c15d5d0 msgr2=0x7f458c0d3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:58.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.494+0000 7f4590e00640 1 --2- 172.21.15.27:0/1429870904 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f458c15d5d0 0x7f458c0d3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.494+0000 7f4590e00640 1 -- 172.21.15.27:0/1429870904 >> v2:172.21.15.93:3300/0 conn(0x7f458c0cde00 msgr2=0x7f458c0ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:58.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.494+0000 7f4590e00640 1 --2- 172.21.15.27:0/1429870904 >> v2:172.21.15.93:3300/0 conn(0x7f458c0cde00 0x7f458c0ce200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.494+0000 7f4590e00640 1 -- 172.21.15.27:0/1429870904 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f458c0b2610 con 0x7f458c0cf890 2024-01-15T18:08:58.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.494+0000 7f4591800640 1 --2- 172.21.15.27:0/1429870904 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f458c15d5d0 0x7f458c0d3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:08:58.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.498+0000 7f4590e00640 1 --2- 172.21.15.27:0/1429870904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f458c0cf890 0x7f458c0cfcd0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f458c08d570 tx=0x7f457401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b45009d1144dec8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:58.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.498+0000 7f458be00640 1 -- 172.21.15.27:0/1429870904 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45740aa620 con 0x7f458c0cf890 2024-01-15T18:08:58.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.498+0000 7f458be00640 1 -- 172.21.15.27:0/1429870904 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45740aa780 con 0x7f458c0cf890 2024-01-15T18:08:58.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.498+0000 7f458be00640 1 -- 172.21.15.27:0/1429870904 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45740aab10 con 0x7f458c0cf890 2024-01-15T18:08:58.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.498+0000 7f4594200640 1 -- 172.21.15.27:0/1429870904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f458c0cf890 msgr2=0x7f458c0cfcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:58.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.498+0000 7f4594200640 1 --2- 172.21.15.27:0/1429870904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f458c0cf890 0x7f458c0cfcd0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f458c08d570 tx=0x7f457401c500 comp rx=0 tx=0).stop 2024-01-15T18:08:58.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.498+0000 7f4594200640 1 -- 172.21.15.27:0/1429870904 shutdown_connections 2024-01-15T18:08:58.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.498+0000 7f4594200640 1 --2- 172.21.15.27:0/1429870904 >> v2:172.21.15.93:3300/0 conn(0x7f458c0cde00 0x7f458c0ce200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.498+0000 7f4594200640 1 --2- 172.21.15.27:0/1429870904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f458c0cf890 0x7f458c0cfcd0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.498+0000 7f4594200640 1 --2- 172.21.15.27:0/1429870904 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f458c15d5d0 0x7f458c0d3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.498+0000 7f4594200640 1 -- 172.21.15.27:0/1429870904 >> 172.21.15.27:0/1429870904 conn(0x7f458c0d3010 msgr2=0x7f458c0d3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:58.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.498+0000 7f4594200640 1 -- 172.21.15.27:0/1429870904 shutdown_connections 2024-01-15T18:08:58.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.498+0000 7f4594200640 1 -- 172.21.15.27:0/1429870904 wait complete. 2024-01-15T18:08:58.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.498+0000 7f4594200640 1 Processor -- start 2024-01-15T18:08:58.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.498+0000 7f4594200640 1 -- start start 2024-01-15T18:08:58.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.502+0000 7f4594200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f458c0cde00 0x7f458c151e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:58.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.502+0000 7f4594200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f458c0cf890 0x7f458c154020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:58.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.502+0000 7f4594200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f458c15d5d0 0x7f458c15ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:58.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.502+0000 7f4594200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458c154560 con 0x7f458c15d5d0 2024-01-15T18:08:58.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.502+0000 7f4594200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458c15f2c0 con 0x7f458c0cde00 2024-01-15T18:08:58.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.502+0000 7f4594200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f458c154750 con 0x7f458c0cf890 2024-01-15T18:08:58.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.502+0000 7f4592200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f458c15d5d0 0x7f458c15ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:58.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.502+0000 7f4592200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f458c15d5d0 0x7f458c15ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59686/0 (socket says 172.21.15.27:59686) 2024-01-15T18:08:58.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.502+0000 7f4592200640 1 -- 172.21.15.27:0/2821311458 learned_addr learned my addr 172.21.15.27:0/2821311458 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:58.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.502+0000 7f4590e00640 1 --2- 172.21.15.27:0/2821311458 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f458c0cf890 0x7f458c154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:58.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.502+0000 7f4591800640 1 --2- 172.21.15.27:0/2821311458 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f458c0cde00 0x7f458c151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:58.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.502+0000 7f4592200640 1 -- 172.21.15.27:0/2821311458 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f458c0cf890 msgr2=0x7f458c154020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:58.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.502+0000 7f4592200640 1 --2- 172.21.15.27:0/2821311458 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f458c0cf890 0x7f458c154020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.502+0000 7f4592200640 1 -- 172.21.15.27:0/2821311458 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f458c0cde00 msgr2=0x7f458c151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:58.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.502+0000 7f4592200640 1 --2- 172.21.15.27:0/2821311458 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f458c0cde00 0x7f458c151e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.502+0000 7f4592200640 1 -- 172.21.15.27:0/2821311458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45740094d0 con 0x7f458c15d5d0 2024-01-15T18:08:58.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.502+0000 7f4592200640 1 --2- 172.21.15.27:0/2821311458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f458c15d5d0 0x7f458c15ed80 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f4584007e90 tx=0x7f458400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:58.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.502+0000 7f4589600640 1 -- 172.21.15.27:0/2821311458 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4584014420 con 0x7f458c15d5d0 2024-01-15T18:08:58.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.502+0000 7f4589600640 1 -- 172.21.15.27:0/2821311458 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45840145a0 con 0x7f458c15d5d0 2024-01-15T18:08:58.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.502+0000 7f4594200640 1 -- 172.21.15.27:0/2821311458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f458c154a30 con 0x7f458c15d5d0 2024-01-15T18:08:58.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.502+0000 7f4589600640 1 -- 172.21.15.27:0/2821311458 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4584014950 con 0x7f458c15d5d0 2024-01-15T18:08:58.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.502+0000 7f4594200640 1 -- 172.21.15.27:0/2821311458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f458c154ff0 con 0x7f458c15d5d0 2024-01-15T18:08:58.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.502+0000 7f4594200640 1 -- 172.21.15.27:0/2821311458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f458c04e850 con 0x7f458c15d5d0 2024-01-15T18:08:58.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.502+0000 7f4589600640 1 -- 172.21.15.27:0/2821311458 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f4584014ab0 con 0x7f458c15d5d0 2024-01-15T18:08:58.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.506+0000 7f4589600640 1 --2- 172.21.15.27:0/2821311458 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4564066e90 0x7f4564069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:58.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.506+0000 7f4589600640 1 -- 172.21.15.27:0/2821311458 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f4584085340 con 0x7f458c15d5d0 2024-01-15T18:08:58.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.506+0000 7f4591800640 1 --2- 172.21.15.27:0/2821311458 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4564066e90 0x7f4564069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:58.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.506+0000 7f4591800640 1 --2- 172.21.15.27:0/2821311458 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4564066e90 0x7f4564069350 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f457c0045f0 tx=0x7f457c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:58.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.510+0000 7f4589600640 1 -- 172.21.15.27:0/2821311458 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f45840520f0 con 0x7f458c15d5d0 2024-01-15T18:08:58.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.670+0000 7f4594200640 1 -- 172.21.15.27:0/2821311458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f458c0b09b0 con 0x7f458c15d5d0 2024-01-15T18:08:58.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.670+0000 7f4589600640 1 -- 172.21.15.27:0/2821311458 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4584055710 con 0x7f458c15d5d0 2024-01-15T18:08:58.675 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:08:58.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.670+0000 7f4594200640 1 -- 172.21.15.27:0/2821311458 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4564066e90 msgr2=0x7f4564069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:58.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.670+0000 7f4594200640 1 --2- 172.21.15.27:0/2821311458 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4564066e90 0x7f4564069350 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f457c0045f0 tx=0x7f457c008040 comp rx=0 tx=0).stop 2024-01-15T18:08:58.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.670+0000 7f4594200640 1 -- 172.21.15.27:0/2821311458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f458c15d5d0 msgr2=0x7f458c15ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:58.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.670+0000 7f4594200640 1 --2- 172.21.15.27:0/2821311458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f458c15d5d0 0x7f458c15ed80 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f4584007e90 tx=0x7f458400e4b0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.670+0000 7f4594200640 1 -- 172.21.15.27:0/2821311458 shutdown_connections 2024-01-15T18:08:58.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.670+0000 7f4594200640 1 --2- 172.21.15.27:0/2821311458 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4564066e90 0x7f4564069350 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.670+0000 7f4594200640 1 --2- 172.21.15.27:0/2821311458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f458c15d5d0 0x7f458c15ed80 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.670+0000 7f4594200640 1 --2- 172.21.15.27:0/2821311458 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f458c0cf890 0x7f458c154020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.670+0000 7f4594200640 1 --2- 172.21.15.27:0/2821311458 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f458c0cde00 0x7f458c151e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.670+0000 7f4594200640 1 -- 172.21.15.27:0/2821311458 >> 172.21.15.27:0/2821311458 conn(0x7f458c0d3010 msgr2=0x7f458c0afdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:58.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.670+0000 7f4594200640 1 -- 172.21.15.27:0/2821311458 shutdown_connections 2024-01-15T18:08:58.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.670+0000 7f4594200640 1 -- 172.21.15.27:0/2821311458 wait complete. 2024-01-15T18:08:58.699 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:08:58.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.834+0000 7f36e7400640 1 Processor -- start 2024-01-15T18:08:58.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.834+0000 7f36e7400640 1 -- start start 2024-01-15T18:08:58.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.838+0000 7f36e7400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f36e015b060 0x7f36e015a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:58.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.838+0000 7f36e7400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f36e00719f0 0x7f36e015bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:58.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.838+0000 7f36e7400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e015dfa0 con 0x7f36e015d9e0 2024-01-15T18:08:58.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.838+0000 7f36e7400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36e015b440 con 0x7f36e015b060 2024-01-15T18:08:58.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.838+0000 7f36e7400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36e015e280 con 0x7f36e00719f0 2024-01-15T18:08:58.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.838+0000 7f36e5400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f36e00719f0 0x7f36e015bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:58.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.838+0000 7f36dfe00640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7f36e015d9e0 0x7f36e00712e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:52816/0 (socket says 172.21.15.27:52816) 2024-01-15T18:08:58.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.838+0000 7f36e4a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f36e015b060 0x7f36e015a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:58.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.838+0000 7f36dfe00640 1 -- 172.21.15.27:0/1912670128 learned_addr learned my addr 172.21.15.27:0/1912670128 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:58.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.838+0000 7f36df400640 1 -- 172.21.15.27:0/1912670128 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069819803 0 0) 0x7f36e015dfa0 con 0x7f36e015d9e0 2024-01-15T18:08:58.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.838+0000 7f36df400640 1 -- 172.21.15.27:0/1912670128 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36c0005740 con 0x7f36e015d9e0 2024-01-15T18:08:58.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.838+0000 7f36e4a00640 1 -- 172.21.15.27:0/1912670128 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f36e00719f0 msgr2=0x7f36e015bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:58.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.838+0000 7f36e4a00640 1 --2- 172.21.15.27:0/1912670128 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f36e00719f0 0x7f36e015bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.838+0000 7f36e4a00640 1 -- 172.21.15.27:0/1912670128 >> v1:172.21.15.93:6789/0 conn(0x7f36e015d9e0 legacy=0x7f36e00712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:58.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.838+0000 7f36e4a00640 1 -- 172.21.15.27:0/1912670128 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e015e560 con 0x7f36e015b060 2024-01-15T18:08:58.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.838+0000 7f36e5400640 1 --2- 172.21.15.27:0/1912670128 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f36e00719f0 0x7f36e015bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:08:58.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.842+0000 7f36e4a00640 1 --2- 172.21.15.27:0/1912670128 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f36e015b060 0x7f36e015a3f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f36d00097c0 tx=0x7f36d001c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2729200b5d739e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.842+0000 7f36df400640 1 -- 172.21.15.27:0/1912670128 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d00ac620 con 0x7f36e015b060 2024-01-15T18:08:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.842+0000 7f36df400640 1 -- 172.21.15.27:0/1912670128 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36d00ac780 con 0x7f36e015b060 2024-01-15T18:08:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.842+0000 7f36df400640 1 -- 172.21.15.27:0/1912670128 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d00acb10 con 0x7f36e015b060 2024-01-15T18:08:58.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.842+0000 7f36e7400640 1 -- 172.21.15.27:0/1912670128 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f36e015b060 msgr2=0x7f36e015a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.842+0000 7f36e7400640 1 --2- 172.21.15.27:0/1912670128 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f36e015b060 0x7f36e015a3f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f36d00097c0 tx=0x7f36d001c3c0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.842+0000 7f36e7400640 1 -- 172.21.15.27:0/1912670128 shutdown_connections 2024-01-15T18:08:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.842+0000 7f36e7400640 1 --2- 172.21.15.27:0/1912670128 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f36e00719f0 0x7f36e015bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.842+0000 7f36e7400640 1 --2- 172.21.15.27:0/1912670128 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f36e015b060 0x7f36e015a3f0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.842+0000 7f36e7400640 1 -- 172.21.15.27:0/1912670128 >> 172.21.15.27:0/1912670128 conn(0x7f36e00b0850 msgr2=0x7f36e00b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:58.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.842+0000 7f36e7400640 1 -- 172.21.15.27:0/1912670128 shutdown_connections 2024-01-15T18:08:58.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.842+0000 7f36e7400640 1 -- 172.21.15.27:0/1912670128 wait complete. 2024-01-15T18:08:58.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.842+0000 7f36e7400640 1 Processor -- start 2024-01-15T18:08:58.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.842+0000 7f36e7400640 1 -- start start 2024-01-15T18:08:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.846+0000 7f36e7400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f36e00719f0 0x7f36e0137890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.846+0000 7f36e7400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36e015b060 0x7f36e00b1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.846+0000 7f36e7400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f36e015d9e0 0x7f36e00b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.846+0000 7f36e7400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e00ce200 con 0x7f36e015b060 2024-01-15T18:08:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.846+0000 7f36e7400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e00cdc20 con 0x7f36e015d9e0 2024-01-15T18:08:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.846+0000 7f36e7400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36e0137dd0 con 0x7f36e00719f0 2024-01-15T18:08:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.846+0000 7f36dfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36e015b060 0x7f36e00b1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.846+0000 7f36dfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36e015b060 0x7f36e00b1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59698/0 (socket says 172.21.15.27:59698) 2024-01-15T18:08:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.846+0000 7f36dfe00640 1 -- 172.21.15.27:0/3386664134 learned_addr learned my addr 172.21.15.27:0/3386664134 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.846+0000 7f36e4a00640 1 --2- 172.21.15.27:0/3386664134 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f36e00719f0 0x7f36e0137890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.846+0000 7f36e5400640 1 --2- 172.21.15.27:0/3386664134 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f36e015d9e0 0x7f36e00b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.846+0000 7f36dfe00640 1 -- 172.21.15.27:0/3386664134 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f36e00719f0 msgr2=0x7f36e0137890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:58.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.846+0000 7f36dfe00640 1 --2- 172.21.15.27:0/3386664134 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f36e00719f0 0x7f36e0137890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.846+0000 7f36dfe00640 1 -- 172.21.15.27:0/3386664134 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f36e015d9e0 msgr2=0x7f36e00b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:58.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.846+0000 7f36dfe00640 1 --2- 172.21.15.27:0/3386664134 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f36e015d9e0 0x7f36e00b1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:58.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.846+0000 7f36dfe00640 1 -- 172.21.15.27:0/3386664134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d8009a70 con 0x7f36e015b060 2024-01-15T18:08:58.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.846+0000 7f36dfe00640 1 --2- 172.21.15.27:0/3386664134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36e015b060 0x7f36e00b1790 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f36c800c450 tx=0x7f36c800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.846+0000 7f36dcc00640 1 -- 172.21.15.27:0/3386664134 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c8005cf0 con 0x7f36e015b060 2024-01-15T18:08:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.846+0000 7f36dcc00640 1 -- 172.21.15.27:0/3386664134 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c8005e50 con 0x7f36e015b060 2024-01-15T18:08:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.846+0000 7f36dcc00640 1 -- 172.21.15.27:0/3386664134 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c80072d0 con 0x7f36e015b060 2024-01-15T18:08:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.846+0000 7f36e7400640 1 -- 172.21.15.27:0/3386664134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d0009420 con 0x7f36e015b060 2024-01-15T18:08:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.846+0000 7f36e7400640 1 -- 172.21.15.27:0/3386664134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e00cde90 con 0x7f36e015b060 2024-01-15T18:08:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.846+0000 7f36e7400640 1 -- 172.21.15.27:0/3386664134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36e00cda80 con 0x7f36e015b060 2024-01-15T18:08:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.850+0000 7f36dcc00640 1 -- 172.21.15.27:0/3386664134 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f36c8007430 con 0x7f36e015b060 2024-01-15T18:08:58.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.850+0000 7f36dcc00640 1 --2- 172.21.15.27:0/3386664134 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f36b8066e90 0x7f36b8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:58.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.850+0000 7f36dcc00640 1 -- 172.21.15.27:0/3386664134 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f36c8083fb0 con 0x7f36e015b060 2024-01-15T18:08:58.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.850+0000 7f36e4a00640 1 --2- 172.21.15.27:0/3386664134 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f36b8066e90 0x7f36b8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:58.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.850+0000 7f36e4a00640 1 --2- 172.21.15.27:0/3386664134 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f36b8066e90 0x7f36b8069350 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f36d00ae000 tx=0x7f36d0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:58.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:58.854+0000 7f36dcc00640 1 -- 172.21.15.27:0/3386664134 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f36e00cda80 con 0x7f36e015b060 2024-01-15T18:08:59.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.022+0000 7f36e7400640 1 -- 172.21.15.27:0/3386664134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f36e00ce3b0 con 0x7f36e015b060 2024-01-15T18:08:59.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.022+0000 7f36dcc00640 1 -- 172.21.15.27:0/3386664134 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v50) v1 ==== 80+0+122 (secure 0 0 0) 0x7f36e00ce3b0 con 0x7f36e015b060 2024-01-15T18:08:59.028 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6835/1005016790 2024-01-16T18:08:52.999612+0000 2024-01-15T18:08:59.028 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6833/1005016790 2024-01-16T18:08:52.999612+0000 2024-01-15T18:08:59.029 INFO:teuthology.orchestra.run.smithi027.stderr:listed 2 entries 2024-01-15T18:08:59.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.022+0000 7f36e7400640 1 -- 172.21.15.27:0/3386664134 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f36b8066e90 msgr2=0x7f36b8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.022+0000 7f36e7400640 1 --2- 172.21.15.27:0/3386664134 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f36b8066e90 0x7f36b8069350 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f36d00ae000 tx=0x7f36d0008040 comp rx=0 tx=0).stop 2024-01-15T18:08:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.022+0000 7f36e7400640 1 -- 172.21.15.27:0/3386664134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36e015b060 msgr2=0x7f36e00b1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.022+0000 7f36e7400640 1 --2- 172.21.15.27:0/3386664134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36e015b060 0x7f36e00b1790 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f36c800c450 tx=0x7f36c800c850 comp rx=0 tx=0).stop 2024-01-15T18:08:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.022+0000 7f36e7400640 1 -- 172.21.15.27:0/3386664134 shutdown_connections 2024-01-15T18:08:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.022+0000 7f36e7400640 1 --2- 172.21.15.27:0/3386664134 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f36b8066e90 0x7f36b8069350 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.022+0000 7f36e7400640 1 --2- 172.21.15.27:0/3386664134 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f36e015d9e0 0x7f36e00b1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.022+0000 7f36e7400640 1 --2- 172.21.15.27:0/3386664134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36e015b060 0x7f36e00b1790 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.022+0000 7f36e7400640 1 --2- 172.21.15.27:0/3386664134 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f36e00719f0 0x7f36e0137890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.022+0000 7f36e7400640 1 -- 172.21.15.27:0/3386664134 >> 172.21.15.27:0/3386664134 conn(0x7f36e00b0850 msgr2=0x7f36e00b0c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.026+0000 7f36e7400640 1 -- 172.21.15.27:0/3386664134 shutdown_connections 2024-01-15T18:08:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.026+0000 7f36e7400640 1 -- 172.21.15.27:0/3386664134 wait complete. 2024-01-15T18:08:59.051 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi093.front.sepia.ceph.com... 2024-01-15T18:08:59.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:08:59.051 DEBUG:teuthology.orchestra.run.smithi093:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-15T18:08:59.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:ceph-fuse[33503]: fuse finished with error 0 and tester_r 0 2024-01-15T18:08:59.075 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T18:08:59.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:08:59.075 DEBUG:teuthology.orchestra.run.smithi093:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-15T18:08:59.245 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:08:59.245 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:08:59.254 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354580k,nr_inodes=4088645,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9249 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:08:59.255 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T18:08:59.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:08:59.255 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-15T18:08:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-15T18:08:59.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:08:59.306 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:08:59.306 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:08:59.315 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354484k,nr_inodes=4088621,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8403 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:08:59.315 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T18:08:59.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:08:59.316 DEBUG:teuthology.orchestra.run.smithi093:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-15T18:08:59.366 INFO:teuthology.orchestra.run.smithi093.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-15T18:08:59.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:08:59.367 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:08:59.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.502+0000 7f555d600640 1 Processor -- start 2024-01-15T18:08:59.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.502+0000 7f555d600640 1 -- start start 2024-01-15T18:08:59.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.502+0000 7f555d600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f55580cf080 0x7f55580cf460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:59.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.502+0000 7f555d600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55580d1ec0 con 0x7f55580b24a0 2024-01-15T18:08:59.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.502+0000 7f555d600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55580ae160 con 0x7f55580b0130 2024-01-15T18:08:59.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.502+0000 7f555d600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55580b16a0 con 0x7f55580cf080 2024-01-15T18:08:59.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.502+0000 7f5556a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f55580cf080 0x7f55580cf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:59.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.502+0000 7f5557400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f55580b24a0 0x7f55580b0f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48486/0 (socket says 172.21.15.27:48486) 2024-01-15T18:08:59.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.502+0000 7f5557400640 1 -- 172.21.15.27:0/2049246712 learned_addr learned my addr 172.21.15.27:0/2049246712 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:59.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.502+0000 7f5556a00640 1 -- 172.21.15.27:0/2049246712 >> v1:172.21.15.93:6789/0 conn(0x7f55580b0130 legacy=0x7f55580b0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:59.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.502+0000 7f5556a00640 1 -- 172.21.15.27:0/2049246712 >> v1:172.21.15.27:6789/0 conn(0x7f55580b24a0 legacy=0x7f55580b0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:59.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.506+0000 7f5556a00640 1 -- 172.21.15.27:0/2049246712 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55580d1be0 con 0x7f55580cf080 2024-01-15T18:08:59.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.506+0000 7f5555600640 1 -- 172.21.15.27:0/2049246712 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894742418 0 0) 0x7f55580d1ec0 con 0x7f55580b24a0 2024-01-15T18:08:59.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.506+0000 7f5556a00640 1 --2- 172.21.15.27:0/2049246712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55580cf080 0x7f55580cf460 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f55480097c0 tx=0x7f554801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61b4939ba5bdf09 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:59.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.506+0000 7f5555600640 1 -- 172.21.15.27:0/2049246712 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55480a7070 con 0x7f55580cf080 2024-01-15T18:08:59.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.506+0000 7f5555600640 1 -- 172.21.15.27:0/2049246712 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55480ab430 con 0x7f55580cf080 2024-01-15T18:08:59.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.506+0000 7f5555600640 1 -- 172.21.15.27:0/2049246712 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55480ab7c0 con 0x7f55580cf080 2024-01-15T18:08:59.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.506+0000 7f555d600640 1 -- 172.21.15.27:0/2049246712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55580cf080 msgr2=0x7f55580cf460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:59.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.506+0000 7f555d600640 1 --2- 172.21.15.27:0/2049246712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55580cf080 0x7f55580cf460 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f55480097c0 tx=0x7f554801c410 comp rx=0 tx=0).stop 2024-01-15T18:08:59.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.506+0000 7f555d600640 1 -- 172.21.15.27:0/2049246712 shutdown_connections 2024-01-15T18:08:59.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.506+0000 7f555d600640 1 --2- 172.21.15.27:0/2049246712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55580cf080 0x7f55580cf460 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:59.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.506+0000 7f555d600640 1 -- 172.21.15.27:0/2049246712 >> 172.21.15.27:0/2049246712 conn(0x7f55580d0f20 msgr2=0x7f55580d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:59.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.510+0000 7f555d600640 1 -- 172.21.15.27:0/2049246712 shutdown_connections 2024-01-15T18:08:59.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.510+0000 7f555d600640 1 -- 172.21.15.27:0/2049246712 wait complete. 2024-01-15T18:08:59.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.510+0000 7f555d600640 1 Processor -- start 2024-01-15T18:08:59.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.510+0000 7f555d600640 1 -- start start 2024-01-15T18:08:59.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.510+0000 7f555d600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f55580b0130 0x7f555814de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:59.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.510+0000 7f555d600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55580b24a0 0x7f5558155b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:59.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.510+0000 7f555d600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55580cf080 0x7f5558156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:59.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.510+0000 7f555d600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555814e3c0 con 0x7f55580b24a0 2024-01-15T18:08:59.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.510+0000 7f555d600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55580cd160 con 0x7f55580cf080 2024-01-15T18:08:59.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.510+0000 7f555d600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55580cd400 con 0x7f55580b0130 2024-01-15T18:08:59.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.510+0000 7f5556000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55580b24a0 0x7f5558155b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:59.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.510+0000 7f5556000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55580b24a0 0x7f5558155b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59710/0 (socket says 172.21.15.27:59710) 2024-01-15T18:08:59.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.510+0000 7f5556000640 1 -- 172.21.15.27:0/1120598290 learned_addr learned my addr 172.21.15.27:0/1120598290 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:59.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.510+0000 7f5557400640 1 --2- 172.21.15.27:0/1120598290 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55580cf080 0x7f5558156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:59.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.510+0000 7f5556a00640 1 --2- 172.21.15.27:0/1120598290 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f55580b0130 0x7f555814de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:59.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.510+0000 7f5556000640 1 -- 172.21.15.27:0/1120598290 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f55580b0130 msgr2=0x7f555814de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:59.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.510+0000 7f5556000640 1 --2- 172.21.15.27:0/1120598290 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f55580b0130 0x7f555814de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:59.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.510+0000 7f5556000640 1 -- 172.21.15.27:0/1120598290 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55580cf080 msgr2=0x7f5558156040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:59.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.510+0000 7f5556000640 1 --2- 172.21.15.27:0/1120598290 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55580cf080 0x7f5558156040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:59.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.510+0000 7f5556000640 1 -- 172.21.15.27:0/1120598290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5548009420 con 0x7f55580b24a0 2024-01-15T18:08:59.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.510+0000 7f5557400640 1 --2- 172.21.15.27:0/1120598290 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55580cf080 0x7f5558156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:08:59.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.510+0000 7f5556a00640 1 --2- 172.21.15.27:0/1120598290 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f55580b0130 0x7f555814de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:08:59.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.510+0000 7f5556000640 1 --2- 172.21.15.27:0/1120598290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55580b24a0 0x7f5558155b00 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f554000c410 tx=0x7f554000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:59.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.514+0000 7f554ea00640 1 -- 172.21.15.27:0/1120598290 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5540016070 con 0x7f55580b24a0 2024-01-15T18:08:59.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.514+0000 7f554ea00640 1 -- 172.21.15.27:0/1120598290 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5540005020 con 0x7f55580b24a0 2024-01-15T18:08:59.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.514+0000 7f554ea00640 1 -- 172.21.15.27:0/1120598290 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5540005410 con 0x7f55580b24a0 2024-01-15T18:08:59.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.514+0000 7f555d600640 1 -- 172.21.15.27:0/1120598290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55580cfbb0 con 0x7f55580b24a0 2024-01-15T18:08:59.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.514+0000 7f555d600640 1 -- 172.21.15.27:0/1120598290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55580c5500 con 0x7f55580b24a0 2024-01-15T18:08:59.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.514+0000 7f555d600640 1 -- 172.21.15.27:0/1120598290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55580c56f0 con 0x7f55580b24a0 2024-01-15T18:08:59.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.514+0000 7f554ea00640 1 -- 172.21.15.27:0/1120598290 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f55400067e0 con 0x7f55580b24a0 2024-01-15T18:08:59.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.514+0000 7f554ea00640 1 --2- 172.21.15.27:0/1120598290 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5530066e90 0x7f5530069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:59.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.514+0000 7f554ea00640 1 -- 172.21.15.27:0/1120598290 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f55400842d0 con 0x7f55580b24a0 2024-01-15T18:08:59.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.518+0000 7f5556a00640 1 --2- 172.21.15.27:0/1120598290 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5530066e90 0x7f5530069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:59.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.518+0000 7f5556a00640 1 --2- 172.21.15.27:0/1120598290 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5530066e90 0x7f5530069350 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f55480af000 tx=0x7f55480a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:59.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.518+0000 7f554ea00640 1 -- 172.21.15.27:0/1120598290 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5540051010 con 0x7f55580b24a0 2024-01-15T18:08:59.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.706+0000 7f555d600640 1 -- 172.21.15.27:0/1120598290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f555804e850 con 0x7f55580b24a0 2024-01-15T18:08:59.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.710+0000 7f554ea00640 1 -- 172.21.15.27:0/1120598290 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+3696 (secure 0 0 0) 0x7f5540003830 con 0x7f55580b24a0 2024-01-15T18:08:59.715 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:08:59.715 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":30,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5306,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/1548117999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":1548117999},{"type":"v1","addr":"172.21.15.93:6836","nonce":1548117999}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5719,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/260141697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":260141697},{"type":"v1","addr":"172.21.15.93:6837","nonce":260141697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5802,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/2030664596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2030664596},{"type":"v1","addr":"172.21.15.27:6834","nonce":2030664596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4247767797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4247767797},{"type":"v1","addr":"172.21.15.27:6835","nonce":4247767797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[]} 2024-01-15T18:08:59.715 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 30 2024-01-15T18:08:59.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.710+0000 7f555d600640 1 -- 172.21.15.27:0/1120598290 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5530066e90 msgr2=0x7f5530069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:59.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.710+0000 7f555d600640 1 --2- 172.21.15.27:0/1120598290 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5530066e90 0x7f5530069350 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f55480af000 tx=0x7f55480a7000 comp rx=0 tx=0).stop 2024-01-15T18:08:59.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.710+0000 7f555d600640 1 -- 172.21.15.27:0/1120598290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55580b24a0 msgr2=0x7f5558155b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:59.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.710+0000 7f555d600640 1 --2- 172.21.15.27:0/1120598290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55580b24a0 0x7f5558155b00 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f554000c410 tx=0x7f554000c8d0 comp rx=0 tx=0).stop 2024-01-15T18:08:59.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.710+0000 7f555d600640 1 -- 172.21.15.27:0/1120598290 shutdown_connections 2024-01-15T18:08:59.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.710+0000 7f555d600640 1 --2- 172.21.15.27:0/1120598290 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5530066e90 0x7f5530069350 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:59.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.710+0000 7f555d600640 1 --2- 172.21.15.27:0/1120598290 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55580cf080 0x7f5558156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:59.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.710+0000 7f555d600640 1 --2- 172.21.15.27:0/1120598290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55580b24a0 0x7f5558155b00 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:59.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.710+0000 7f555d600640 1 --2- 172.21.15.27:0/1120598290 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f55580b0130 0x7f555814de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:59.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.710+0000 7f555d600640 1 -- 172.21.15.27:0/1120598290 >> 172.21.15.27:0/1120598290 conn(0x7f55580d0f20 msgr2=0x7f5558157220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:59.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.710+0000 7f555d600640 1 -- 172.21.15.27:0/1120598290 shutdown_connections 2024-01-15T18:08:59.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.710+0000 7f555d600640 1 -- 172.21.15.27:0/1120598290 wait complete. 2024-01-15T18:08:59.737 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-15T18:08:59.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.874+0000 7f5ef6c00640 1 Processor -- start 2024-01-15T18:08:59.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.874+0000 7f5ef6c00640 1 -- start start 2024-01-15T18:08:59.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.874+0000 7f5ef6c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5ef015d570 0x7f5ef006be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:59.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.874+0000 7f5ef6c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ef006b810 0x7f5ef006c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:59.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.874+0000 7f5ef6c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ef00b0d70 con 0x7f5ef015d190 2024-01-15T18:08:59.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.874+0000 7f5ef6c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef00d0540 con 0x7f5ef006b810 2024-01-15T18:08:59.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.874+0000 7f5ef6c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef00b0440 con 0x7f5ef015d570 2024-01-15T18:08:59.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.874+0000 7f5eefe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5ef015d190 0x7f5ef00cfcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48490/0 (socket says 172.21.15.27:48490) 2024-01-15T18:08:59.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.874+0000 7f5eefe00640 1 -- 172.21.15.27:0/879712330 learned_addr learned my addr 172.21.15.27:0/879712330 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:08:59.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.874+0000 7f5ef4c00640 1 --2- 172.21.15.27:0/879712330 >> v2:172.21.15.27:3300/0 conn(0x7f5ef006b810 0x7f5ef006c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:59.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.874+0000 7f5eef400640 1 --2- 172.21.15.27:0/879712330 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5ef015d570 0x7f5ef006be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:59.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.874+0000 7f5eeea00640 1 -- 172.21.15.27:0/879712330 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129832238 0 0) 0x7f5ef00b0d70 con 0x7f5ef015d190 2024-01-15T18:08:59.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.874+0000 7f5eeea00640 1 -- 172.21.15.27:0/879712330 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ed0003530 con 0x7f5ef015d190 2024-01-15T18:08:59.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.878+0000 7f5ef4c00640 1 -- 172.21.15.27:0/879712330 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5ef015d570 msgr2=0x7f5ef006be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:59.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.878+0000 7f5ef4c00640 1 --2- 172.21.15.27:0/879712330 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5ef015d570 0x7f5ef006be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:59.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.878+0000 7f5ef4c00640 1 -- 172.21.15.27:0/879712330 >> v1:172.21.15.27:6789/0 conn(0x7f5ef015d190 legacy=0x7f5ef00cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:59.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.878+0000 7f5ef4c00640 1 -- 172.21.15.27:0/879712330 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef00b0720 con 0x7f5ef006b810 2024-01-15T18:08:59.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.878+0000 7f5ef4c00640 1 --2- 172.21.15.27:0/879712330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ef006b810 0x7f5ef006c350 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8009d90 tx=0x7f5ee801c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c77f11fbb10891c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:59.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.878+0000 7f5eeea00640 1 -- 172.21.15.27:0/879712330 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee80a5a00 con 0x7f5ef006b810 2024-01-15T18:08:59.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.878+0000 7f5eeea00640 1 -- 172.21.15.27:0/879712330 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee80a5b60 con 0x7f5ef006b810 2024-01-15T18:08:59.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.878+0000 7f5eeea00640 1 -- 172.21.15.27:0/879712330 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee80ad440 con 0x7f5ef006b810 2024-01-15T18:08:59.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.878+0000 7f5ef6c00640 1 -- 172.21.15.27:0/879712330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ef006b810 msgr2=0x7f5ef006c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:59.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.878+0000 7f5ef6c00640 1 --2- 172.21.15.27:0/879712330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ef006b810 0x7f5ef006c350 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8009d90 tx=0x7f5ee801c920 comp rx=0 tx=0).stop 2024-01-15T18:08:59.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.882+0000 7f5ef6c00640 1 -- 172.21.15.27:0/879712330 shutdown_connections 2024-01-15T18:08:59.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.882+0000 7f5ef6c00640 1 --2- 172.21.15.27:0/879712330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ef006b810 0x7f5ef006c350 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:59.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.882+0000 7f5ef6c00640 1 --2- 172.21.15.27:0/879712330 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5ef015d570 0x7f5ef006be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:59.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.882+0000 7f5ef6c00640 1 -- 172.21.15.27:0/879712330 >> 172.21.15.27:0/879712330 conn(0x7f5ef00d0e70 msgr2=0x7f5ef00d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:59.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.882+0000 7f5ef6c00640 1 -- 172.21.15.27:0/879712330 shutdown_connections 2024-01-15T18:08:59.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.882+0000 7f5ef6c00640 1 -- 172.21.15.27:0/879712330 wait complete. 2024-01-15T18:08:59.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.882+0000 7f5ef6c00640 1 Processor -- start 2024-01-15T18:08:59.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.882+0000 7f5ef6c00640 1 -- start start 2024-01-15T18:08:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.882+0000 7f5ef6c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5ef006b810 0x7f5ef00c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.882+0000 7f5ef6c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ef015d190 0x7f5ef00b65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.882+0000 7f5ef6c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5ef015d570 0x7f5ef0157c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.882+0000 7f5ef6c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef00adeb0 con 0x7f5ef015d190 2024-01-15T18:08:59.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.882+0000 7f5ef6c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef00b6b10 con 0x7f5ef015d570 2024-01-15T18:08:59.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.882+0000 7f5ef6c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef00adb80 con 0x7f5ef006b810 2024-01-15T18:08:59.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.882+0000 7f5eef400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ef015d190 0x7f5ef00b65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:59.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.882+0000 7f5eef400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ef015d190 0x7f5ef00b65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59718/0 (socket says 172.21.15.27:59718) 2024-01-15T18:08:59.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.882+0000 7f5eef400640 1 -- 172.21.15.27:0/2083314441 learned_addr learned my addr 172.21.15.27:0/2083314441 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:08:59.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.882+0000 7f5ef4c00640 1 --2- 172.21.15.27:0/2083314441 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5ef015d570 0x7f5ef0157c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:59.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.882+0000 7f5eefe00640 1 --2- 172.21.15.27:0/2083314441 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5ef006b810 0x7f5ef00c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:59.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.882+0000 7f5eef400640 1 -- 172.21.15.27:0/2083314441 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5ef006b810 msgr2=0x7f5ef00c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:59.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.882+0000 7f5eef400640 1 --2- 172.21.15.27:0/2083314441 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5ef006b810 0x7f5ef00c48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:59.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.882+0000 7f5eef400640 1 -- 172.21.15.27:0/2083314441 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5ef015d570 msgr2=0x7f5ef0157c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:59.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.882+0000 7f5eef400640 1 --2- 172.21.15.27:0/2083314441 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5ef015d570 0x7f5ef0157c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:59.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.882+0000 7f5eef400640 1 -- 172.21.15.27:0/2083314441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee8009a70 con 0x7f5ef015d190 2024-01-15T18:08:59.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.882+0000 7f5eef400640 1 --2- 172.21.15.27:0/2083314441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ef015d190 0x7f5ef00b65d0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f5ed800e870 tx=0x7f5ed800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:59.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.886+0000 7f5ee7e00640 1 -- 172.21.15.27:0/2083314441 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed8009510 con 0x7f5ef015d190 2024-01-15T18:08:59.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.886+0000 7f5ee7e00640 1 -- 172.21.15.27:0/2083314441 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ed800eec0 con 0x7f5ef015d190 2024-01-15T18:08:59.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.886+0000 7f5ee7e00640 1 -- 172.21.15.27:0/2083314441 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed800cb40 con 0x7f5ef015d190 2024-01-15T18:08:59.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.886+0000 7f5ef6c00640 1 -- 172.21.15.27:0/2083314441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef00c7070 con 0x7f5ef015d190 2024-01-15T18:08:59.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.886+0000 7f5ef6c00640 1 -- 172.21.15.27:0/2083314441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef00c7630 con 0x7f5ef015d190 2024-01-15T18:08:59.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.886+0000 7f5ef6c00640 1 -- 172.21.15.27:0/2083314441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef004e850 con 0x7f5ef015d190 2024-01-15T18:08:59.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.886+0000 7f5ee7e00640 1 -- 172.21.15.27:0/2083314441 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f5ed800cca0 con 0x7f5ef015d190 2024-01-15T18:08:59.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.886+0000 7f5ee7e00640 1 --2- 172.21.15.27:0/2083314441 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5ec8066ec0 0x7f5ec8069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:59.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.886+0000 7f5ee7e00640 1 -- 172.21.15.27:0/2083314441 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f5ed8084bd0 con 0x7f5ef015d190 2024-01-15T18:08:59.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.886+0000 7f5eefe00640 1 --2- 172.21.15.27:0/2083314441 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5ec8066ec0 0x7f5ec8069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:59.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.890+0000 7f5eefe00640 1 --2- 172.21.15.27:0/2083314441 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5ec8066ec0 0x7f5ec8069380 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5ee00011f0 tx=0x7f5ee0007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:59.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:08:59.890+0000 7f5ee7e00640 1 -- 172.21.15.27:0/2083314441 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5ed80519c0 con 0x7f5ef015d190 2024-01-15T18:09:00.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.090+0000 7f5ef6c00640 1 -- 172.21.15.27:0/2083314441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7f5ef00ce420 con 0x7f5ef015d190 2024-01-15T18:09:00.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.094+0000 7f5ee7e00640 1 -- 172.21.15.27:0/2083314441 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v17) v1 ==== 120+0+0 (secure 0 0 0) 0x7f5ed8054fe0 con 0x7f5ef015d190 2024-01-15T18:09:00.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.098+0000 7f5ef6c00640 1 -- 172.21.15.27:0/2083314441 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5ec8066ec0 msgr2=0x7f5ec8069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:00.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.098+0000 7f5ef6c00640 1 --2- 172.21.15.27:0/2083314441 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5ec8066ec0 0x7f5ec8069380 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5ee00011f0 tx=0x7f5ee0007d40 comp rx=0 tx=0).stop 2024-01-15T18:09:00.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.098+0000 7f5ef6c00640 1 -- 172.21.15.27:0/2083314441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ef015d190 msgr2=0x7f5ef00b65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:00.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.098+0000 7f5ef6c00640 1 --2- 172.21.15.27:0/2083314441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ef015d190 0x7f5ef00b65d0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f5ed800e870 tx=0x7f5ed800ec70 comp rx=0 tx=0).stop 2024-01-15T18:09:00.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.098+0000 7f5ef6c00640 1 -- 172.21.15.27:0/2083314441 shutdown_connections 2024-01-15T18:09:00.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.098+0000 7f5ef6c00640 1 --2- 172.21.15.27:0/2083314441 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5ec8066ec0 0x7f5ec8069380 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.098+0000 7f5ef6c00640 1 --2- 172.21.15.27:0/2083314441 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5ef015d570 0x7f5ef0157c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.098+0000 7f5ef6c00640 1 --2- 172.21.15.27:0/2083314441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ef015d190 0x7f5ef00b65d0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.098+0000 7f5ef6c00640 1 --2- 172.21.15.27:0/2083314441 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5ef006b810 0x7f5ef00c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.098+0000 7f5ef6c00640 1 -- 172.21.15.27:0/2083314441 >> 172.21.15.27:0/2083314441 conn(0x7f5ef00d0e70 msgr2=0x7f5ef00cfc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:00.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.098+0000 7f5ef6c00640 1 -- 172.21.15.27:0/2083314441 shutdown_connections 2024-01-15T18:09:00.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.098+0000 7f5ef6c00640 1 -- 172.21.15.27:0/2083314441 wait complete. 2024-01-15T18:09:00.125 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-15T18:09:00.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.262+0000 7f85bda00640 1 Processor -- start 2024-01-15T18:09:00.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.262+0000 7f85bda00640 1 -- start start 2024-01-15T18:09:00.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.266+0000 7f85bda00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85b80cf460 0x7f85b80cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:00.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.266+0000 7f85bda00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f85b815b4b0 0x7f85b80cba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:00.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.266+0000 7f85bda00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f85b80cec60 0x7f85b80cbf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:00.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.266+0000 7f85bda00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85b80cfd80 con 0x7f85b815b4b0 2024-01-15T18:09:00.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.266+0000 7f85bda00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85b80d19a0 con 0x7f85b80cec60 2024-01-15T18:09:00.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.266+0000 7f85bda00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b80d1c80 con 0x7f85b80cf460 2024-01-15T18:09:00.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.266+0000 7f85b6000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f85b815b4b0 0x7f85b80cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:00.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.266+0000 7f85b6000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f85b815b4b0 0x7f85b80cba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59720/0 (socket says 172.21.15.27:59720) 2024-01-15T18:09:00.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.266+0000 7f85b6000640 1 -- 172.21.15.27:0/1571954646 learned_addr learned my addr 172.21.15.27:0/1571954646 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:00.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.266+0000 7f85b7400640 1 --2- 172.21.15.27:0/1571954646 >> v2:172.21.15.93:3300/0 conn(0x7f85b80cec60 0x7f85b80cbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:00.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.266+0000 7f85b6a00640 1 --2- 172.21.15.27:0/1571954646 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85b80cf460 0x7f85b80cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:00.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.266+0000 7f85b6000640 1 -- 172.21.15.27:0/1571954646 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85b80cf460 msgr2=0x7f85b80cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:00.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.266+0000 7f85b6000640 1 --2- 172.21.15.27:0/1571954646 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85b80cf460 0x7f85b80cf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.266+0000 7f85b6000640 1 -- 172.21.15.27:0/1571954646 >> v2:172.21.15.93:3300/0 conn(0x7f85b80cec60 msgr2=0x7f85b80cbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:00.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.266+0000 7f85b6000640 1 --2- 172.21.15.27:0/1571954646 >> v2:172.21.15.93:3300/0 conn(0x7f85b80cec60 0x7f85b80cbf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.266+0000 7f85b6000640 1 -- 172.21.15.27:0/1571954646 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b80b07c0 con 0x7f85b815b4b0 2024-01-15T18:09:00.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.266+0000 7f85b7400640 1 --2- 172.21.15.27:0/1571954646 >> v2:172.21.15.93:3300/0 conn(0x7f85b80cec60 0x7f85b80cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:00.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.266+0000 7f85b6a00640 1 --2- 172.21.15.27:0/1571954646 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85b80cf460 0x7f85b80cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:00.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.270+0000 7f85b6000640 1 --2- 172.21.15.27:0/1571954646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85b815b4b0 0x7f85b80cba20 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f85b8067f30 tx=0x7f85a001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efd3d5cb39a675d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:00.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.270+0000 7f85b5600640 1 -- 172.21.15.27:0/1571954646 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a00aa620 con 0x7f85b815b4b0 2024-01-15T18:09:00.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.270+0000 7f85b5600640 1 -- 172.21.15.27:0/1571954646 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a00aa780 con 0x7f85b815b4b0 2024-01-15T18:09:00.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.270+0000 7f85b5600640 1 -- 172.21.15.27:0/1571954646 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a00aab10 con 0x7f85b815b4b0 2024-01-15T18:09:00.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.270+0000 7f85bda00640 1 -- 172.21.15.27:0/1571954646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85b815b4b0 msgr2=0x7f85b80cba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:00.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.270+0000 7f85bda00640 1 --2- 172.21.15.27:0/1571954646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85b815b4b0 0x7f85b80cba20 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f85b8067f30 tx=0x7f85a001c500 comp rx=0 tx=0).stop 2024-01-15T18:09:00.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.270+0000 7f85bda00640 1 -- 172.21.15.27:0/1571954646 shutdown_connections 2024-01-15T18:09:00.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.270+0000 7f85bda00640 1 --2- 172.21.15.27:0/1571954646 >> v2:172.21.15.93:3300/0 conn(0x7f85b80cec60 0x7f85b80cbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.270+0000 7f85bda00640 1 --2- 172.21.15.27:0/1571954646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85b815b4b0 0x7f85b80cba20 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.270+0000 7f85bda00640 1 --2- 172.21.15.27:0/1571954646 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85b80cf460 0x7f85b80cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.270+0000 7f85bda00640 1 -- 172.21.15.27:0/1571954646 >> 172.21.15.27:0/1571954646 conn(0x7f85b80d0e80 msgr2=0x7f85b80d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:00.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.270+0000 7f85bda00640 1 -- 172.21.15.27:0/1571954646 shutdown_connections 2024-01-15T18:09:00.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.270+0000 7f85bda00640 1 -- 172.21.15.27:0/1571954646 wait complete. 2024-01-15T18:09:00.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.270+0000 7f85bda00640 1 Processor -- start 2024-01-15T18:09:00.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.270+0000 7f85bda00640 1 -- start start 2024-01-15T18:09:00.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.274+0000 7f85bda00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85b80cec60 0x7f85b81526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:00.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.274+0000 7f85bda00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85b80cf460 0x7f85b8157ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:00.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.274+0000 7f85bda00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f85b815b4b0 0x7f85b815d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:00.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.274+0000 7f85bda00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b8152c10 con 0x7f85b80cec60 2024-01-15T18:09:00.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.274+0000 7f85bda00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b806cc30 con 0x7f85b815b4b0 2024-01-15T18:09:00.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.274+0000 7f85bda00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b806cf10 con 0x7f85b80cf460 2024-01-15T18:09:00.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.274+0000 7f85b6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85b80cec60 0x7f85b81526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:00.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.274+0000 7f85b6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85b80cec60 0x7f85b81526d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59732/0 (socket says 172.21.15.27:59732) 2024-01-15T18:09:00.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.274+0000 7f85b6a00640 1 -- 172.21.15.27:0/2075245532 learned_addr learned my addr 172.21.15.27:0/2075245532 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:00.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.274+0000 7f85b6000640 1 --2- 172.21.15.27:0/2075245532 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85b80cf460 0x7f85b8157ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:00.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.274+0000 7f85b7400640 1 --2- 172.21.15.27:0/2075245532 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f85b815b4b0 0x7f85b815d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:00.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.274+0000 7f85b6a00640 1 -- 172.21.15.27:0/2075245532 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85b80cf460 msgr2=0x7f85b8157ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:00.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.274+0000 7f85b6a00640 1 --2- 172.21.15.27:0/2075245532 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85b80cf460 0x7f85b8157ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.274+0000 7f85b6a00640 1 -- 172.21.15.27:0/2075245532 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f85b815b4b0 msgr2=0x7f85b815d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:00.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.274+0000 7f85b6a00640 1 --2- 172.21.15.27:0/2075245532 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f85b815b4b0 0x7f85b815d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.274+0000 7f85b6a00640 1 -- 172.21.15.27:0/2075245532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a00094d0 con 0x7f85b80cec60 2024-01-15T18:09:00.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.274+0000 7f85b7400640 1 --2- 172.21.15.27:0/2075245532 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f85b815b4b0 0x7f85b815d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:09:00.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.274+0000 7f85b6a00640 1 --2- 172.21.15.27:0/2075245532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85b80cec60 0x7f85b81526d0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f85a800c7c0 tx=0x7f85a800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:00.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.274+0000 7f85aea00640 1 -- 172.21.15.27:0/2075245532 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a8007a30 con 0x7f85b80cec60 2024-01-15T18:09:00.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.274+0000 7f85aea00640 1 -- 172.21.15.27:0/2075245532 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a8007b90 con 0x7f85b80cec60 2024-01-15T18:09:00.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.274+0000 7f85aea00640 1 -- 172.21.15.27:0/2075245532 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a800ce70 con 0x7f85b80cec60 2024-01-15T18:09:00.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.274+0000 7f85bda00640 1 -- 172.21.15.27:0/2075245532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b806d1f0 con 0x7f85b80cec60 2024-01-15T18:09:00.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.274+0000 7f85bda00640 1 -- 172.21.15.27:0/2075245532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b8150850 con 0x7f85b80cec60 2024-01-15T18:09:00.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.274+0000 7f85bda00640 1 -- 172.21.15.27:0/2075245532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b8150a40 con 0x7f85b80cec60 2024-01-15T18:09:00.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.278+0000 7f85aea00640 1 -- 172.21.15.27:0/2075245532 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f85a80113f0 con 0x7f85b80cec60 2024-01-15T18:09:00.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.278+0000 7f85aea00640 1 --2- 172.21.15.27:0/2075245532 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8590066e90 0x7f8590069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:00.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.278+0000 7f85aea00640 1 -- 172.21.15.27:0/2075245532 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f85a8085050 con 0x7f85b80cec60 2024-01-15T18:09:00.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.278+0000 7f85b6000640 1 --2- 172.21.15.27:0/2075245532 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8590066e90 0x7f8590069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:00.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.278+0000 7f85b6000640 1 --2- 172.21.15.27:0/2075245532 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8590066e90 0x7f8590069350 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f85a00ae000 tx=0x7f85a00b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:00.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.282+0000 7f85aea00640 1 -- 172.21.15.27:0/2075245532 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f85a8051cf0 con 0x7f85b80cec60 2024-01-15T18:09:00.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.442+0000 7f85bda00640 1 -- 172.21.15.27:0/2075245532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f85b804e850 con 0x7f85b80cec60 2024-01-15T18:09:00.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.446+0000 7f85aea00640 1 -- 172.21.15.27:0/2075245532 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v18) v1 ==== 104+0+0 (secure 0 0 0) 0x7f85a8055310 con 0x7f85b80cec60 2024-01-15T18:09:00.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.450+0000 7f85bda00640 1 -- 172.21.15.27:0/2075245532 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8590066e90 msgr2=0x7f8590069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:00.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.450+0000 7f85bda00640 1 --2- 172.21.15.27:0/2075245532 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8590066e90 0x7f8590069350 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f85a00ae000 tx=0x7f85a00b5040 comp rx=0 tx=0).stop 2024-01-15T18:09:00.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.450+0000 7f85bda00640 1 -- 172.21.15.27:0/2075245532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85b80cec60 msgr2=0x7f85b81526d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:00.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.450+0000 7f85bda00640 1 --2- 172.21.15.27:0/2075245532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85b80cec60 0x7f85b81526d0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f85a800c7c0 tx=0x7f85a800cc80 comp rx=0 tx=0).stop 2024-01-15T18:09:00.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.450+0000 7f85bda00640 1 -- 172.21.15.27:0/2075245532 shutdown_connections 2024-01-15T18:09:00.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.450+0000 7f85bda00640 1 --2- 172.21.15.27:0/2075245532 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8590066e90 0x7f8590069350 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.450+0000 7f85bda00640 1 --2- 172.21.15.27:0/2075245532 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f85b815b4b0 0x7f85b815d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.450+0000 7f85bda00640 1 --2- 172.21.15.27:0/2075245532 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85b80cf460 0x7f85b8157ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.450+0000 7f85bda00640 1 --2- 172.21.15.27:0/2075245532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85b80cec60 0x7f85b81526d0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.450+0000 7f85bda00640 1 -- 172.21.15.27:0/2075245532 >> 172.21.15.27:0/2075245532 conn(0x7f85b80d0e80 msgr2=0x7f85b8156300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:00.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.450+0000 7f85bda00640 1 -- 172.21.15.27:0/2075245532 shutdown_connections 2024-01-15T18:09:00.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.450+0000 7f85bda00640 1 -- 172.21.15.27:0/2075245532 wait complete. 2024-01-15T18:09:00.476 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2024-01-15T18:09:00.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.614+0000 7f74d3200640 1 Processor -- start 2024-01-15T18:09:00.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.614+0000 7f74d3200640 1 -- start start 2024-01-15T18:09:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.614+0000 7f74d3200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f74cc15d190 0x7f74cc0cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.614+0000 7f74d3200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f74cc15d570 0x7f74cc06be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.614+0000 7f74d3200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f74cc06b810 0x7f74cc06c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.614+0000 7f74d3200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74cc0b0d70 con 0x7f74cc15d190 2024-01-15T18:09:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.614+0000 7f74d3200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74cc0d0540 con 0x7f74cc15d570 2024-01-15T18:09:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.614+0000 7f74d3200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc0b0440 con 0x7f74cc06b810 2024-01-15T18:09:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.614+0000 7f74cbe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f74cc15d190 0x7f74cc0cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.614+0000 7f74cbe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f74cc15d190 0x7f74cc0cfcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59736/0 (socket says 172.21.15.27:59736) 2024-01-15T18:09:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.614+0000 7f74cbe00640 1 -- 172.21.15.27:0/1952857913 learned_addr learned my addr 172.21.15.27:0/1952857913 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.614+0000 7f74d1200640 1 --2- 172.21.15.27:0/1952857913 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f74cc06b810 0x7f74cc06c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.614+0000 7f74cb400640 1 --2- 172.21.15.27:0/1952857913 >> v2:172.21.15.93:3300/0 conn(0x7f74cc15d570 0x7f74cc06be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.614+0000 7f74cbe00640 1 -- 172.21.15.27:0/1952857913 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f74cc06b810 msgr2=0x7f74cc06c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.614+0000 7f74cbe00640 1 --2- 172.21.15.27:0/1952857913 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f74cc06b810 0x7f74cc06c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.614+0000 7f74cbe00640 1 -- 172.21.15.27:0/1952857913 >> v2:172.21.15.93:3300/0 conn(0x7f74cc15d570 msgr2=0x7f74cc06be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:00.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.614+0000 7f74cbe00640 1 --2- 172.21.15.27:0/1952857913 >> v2:172.21.15.93:3300/0 conn(0x7f74cc15d570 0x7f74cc06be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.614+0000 7f74cbe00640 1 -- 172.21.15.27:0/1952857913 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74cc0b0720 con 0x7f74cc15d190 2024-01-15T18:09:00.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.618+0000 7f74cbe00640 1 --2- 172.21.15.27:0/1952857913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74cc15d190 0x7f74cc0cfcf0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f74cc0713a0 tx=0x7f74bc01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac7cd7b43e7a3426 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:00.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.618+0000 7f74caa00640 1 -- 172.21.15.27:0/1952857913 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74bc0ab650 con 0x7f74cc15d190 2024-01-15T18:09:00.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.618+0000 7f74caa00640 1 -- 172.21.15.27:0/1952857913 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74bc0ab7b0 con 0x7f74cc15d190 2024-01-15T18:09:00.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.618+0000 7f74caa00640 1 -- 172.21.15.27:0/1952857913 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74bc0abb40 con 0x7f74cc15d190 2024-01-15T18:09:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.618+0000 7f74d3200640 1 -- 172.21.15.27:0/1952857913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74cc15d190 msgr2=0x7f74cc0cfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.618+0000 7f74d3200640 1 --2- 172.21.15.27:0/1952857913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74cc15d190 0x7f74cc0cfcf0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f74cc0713a0 tx=0x7f74bc01c4d0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.618+0000 7f74d3200640 1 -- 172.21.15.27:0/1952857913 shutdown_connections 2024-01-15T18:09:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.618+0000 7f74d3200640 1 --2- 172.21.15.27:0/1952857913 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f74cc06b810 0x7f74cc06c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.618+0000 7f74d3200640 1 --2- 172.21.15.27:0/1952857913 >> v2:172.21.15.93:3300/0 conn(0x7f74cc15d570 0x7f74cc06be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.618+0000 7f74d3200640 1 --2- 172.21.15.27:0/1952857913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74cc15d190 0x7f74cc0cfcf0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.618+0000 7f74d3200640 1 -- 172.21.15.27:0/1952857913 >> 172.21.15.27:0/1952857913 conn(0x7f74cc0d0e70 msgr2=0x7f74cc0d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.618+0000 7f74d3200640 1 -- 172.21.15.27:0/1952857913 shutdown_connections 2024-01-15T18:09:00.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.618+0000 7f74d3200640 1 -- 172.21.15.27:0/1952857913 wait complete. 2024-01-15T18:09:00.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.622+0000 7f74d3200640 1 Processor -- start 2024-01-15T18:09:00.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.622+0000 7f74d3200640 1 -- start start 2024-01-15T18:09:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.622+0000 7f74d3200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f74cc06b810 0x7f74cc15c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.622+0000 7f74d3200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74cc15d190 0x7f74cc1603a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.622+0000 7f74d3200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f74cc15d570 0x7f74cc15e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.622+0000 7f74d3200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc15ff10 con 0x7f74cc15d190 2024-01-15T18:09:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.622+0000 7f74d3200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc15f300 con 0x7f74cc15d570 2024-01-15T18:09:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.622+0000 7f74d3200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc15f5e0 con 0x7f74cc06b810 2024-01-15T18:09:00.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.622+0000 7f74cb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74cc15d190 0x7f74cc1603a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.622+0000 7f74cb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74cc15d190 0x7f74cc1603a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59744/0 (socket says 172.21.15.27:59744) 2024-01-15T18:09:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.622+0000 7f74cb400640 1 -- 172.21.15.27:0/807444726 learned_addr learned my addr 172.21.15.27:0/807444726 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.622+0000 7f74d1200640 1 --2- 172.21.15.27:0/807444726 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f74cc15d570 0x7f74cc15e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.622+0000 7f74cbe00640 1 --2- 172.21.15.27:0/807444726 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f74cc06b810 0x7f74cc15c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.622+0000 7f74cb400640 1 -- 172.21.15.27:0/807444726 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f74cc06b810 msgr2=0x7f74cc15c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.622+0000 7f74cb400640 1 --2- 172.21.15.27:0/807444726 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f74cc06b810 0x7f74cc15c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.622+0000 7f74cb400640 1 -- 172.21.15.27:0/807444726 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f74cc15d570 msgr2=0x7f74cc15e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.622+0000 7f74cb400640 1 --2- 172.21.15.27:0/807444726 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f74cc15d570 0x7f74cc15e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.622+0000 7f74cb400640 1 -- 172.21.15.27:0/807444726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74bc0094d0 con 0x7f74cc15d190 2024-01-15T18:09:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.622+0000 7f74cb400640 1 --2- 172.21.15.27:0/807444726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74cc15d190 0x7f74cc1603a0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f74b400e870 tx=0x7f74b400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.622+0000 7f74c3e00640 1 -- 172.21.15.27:0/807444726 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b4009510 con 0x7f74cc15d190 2024-01-15T18:09:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.622+0000 7f74c3e00640 1 -- 172.21.15.27:0/807444726 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b400eec0 con 0x7f74cc15d190 2024-01-15T18:09:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.622+0000 7f74d3200640 1 -- 172.21.15.27:0/807444726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74cc15f8c0 con 0x7f74cc15d190 2024-01-15T18:09:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.622+0000 7f74c3e00640 1 -- 172.21.15.27:0/807444726 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b400cb40 con 0x7f74cc15d190 2024-01-15T18:09:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.622+0000 7f74d3200640 1 -- 172.21.15.27:0/807444726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74cc1608e0 con 0x7f74cc15d190 2024-01-15T18:09:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.622+0000 7f74d3200640 1 -- 172.21.15.27:0/807444726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74cc1600c0 con 0x7f74cc15d190 2024-01-15T18:09:00.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.626+0000 7f74c3e00640 1 -- 172.21.15.27:0/807444726 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f74b400cca0 con 0x7f74cc15d190 2024-01-15T18:09:00.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.626+0000 7f74c3e00640 1 --2- 172.21.15.27:0/807444726 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f74a4066e90 0x7f74a4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.626+0000 7f74c3e00640 1 -- 172.21.15.27:0/807444726 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f74b40852c0 con 0x7f74cc15d190 2024-01-15T18:09:00.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.626+0000 7f74cbe00640 1 --2- 172.21.15.27:0/807444726 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f74a4066e90 0x7f74a4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.626+0000 7f74cbe00640 1 --2- 172.21.15.27:0/807444726 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f74a4066e90 0x7f74a4069350 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f74bc0af000 tx=0x7f74bc0a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:00.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.630+0000 7f74c3e00640 1 -- 172.21.15.27:0/807444726 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f74b4051fb0 con 0x7f74cc15d190 2024-01-15T18:09:00.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.786+0000 7f74d3200640 1 -- 172.21.15.27:0/807444726 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7f74cc04e850 con 0x7f74cc15d190 2024-01-15T18:09:00.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.790+0000 7f74c3e00640 1 -- 172.21.15.27:0/807444726 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v19) v1 ==== 115+0+0 (secure 0 0 0) 0x7f74b40555d0 con 0x7f74cc15d190 2024-01-15T18:09:00.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.794+0000 7f74d3200640 1 -- 172.21.15.27:0/807444726 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f74a4066e90 msgr2=0x7f74a4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:00.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.794+0000 7f74d3200640 1 --2- 172.21.15.27:0/807444726 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f74a4066e90 0x7f74a4069350 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f74bc0af000 tx=0x7f74bc0a7000 comp rx=0 tx=0).stop 2024-01-15T18:09:00.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.794+0000 7f74d3200640 1 -- 172.21.15.27:0/807444726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74cc15d190 msgr2=0x7f74cc1603a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:00.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.794+0000 7f74d3200640 1 --2- 172.21.15.27:0/807444726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74cc15d190 0x7f74cc1603a0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f74b400e870 tx=0x7f74b400ec70 comp rx=0 tx=0).stop 2024-01-15T18:09:00.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.794+0000 7f74d3200640 1 -- 172.21.15.27:0/807444726 shutdown_connections 2024-01-15T18:09:00.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.794+0000 7f74d3200640 1 --2- 172.21.15.27:0/807444726 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f74a4066e90 0x7f74a4069350 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.794+0000 7f74d3200640 1 --2- 172.21.15.27:0/807444726 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f74cc15d570 0x7f74cc15e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.794+0000 7f74d3200640 1 --2- 172.21.15.27:0/807444726 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74cc15d190 0x7f74cc1603a0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.794+0000 7f74d3200640 1 --2- 172.21.15.27:0/807444726 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f74cc06b810 0x7f74cc15c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.794+0000 7f74d3200640 1 -- 172.21.15.27:0/807444726 >> 172.21.15.27:0/807444726 conn(0x7f74cc0d0e70 msgr2=0x7f74cc0afaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:00.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.794+0000 7f74d3200640 1 -- 172.21.15.27:0/807444726 shutdown_connections 2024-01-15T18:09:00.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.794+0000 7f74d3200640 1 -- 172.21.15.27:0/807444726 wait complete. 2024-01-15T18:09:00.821 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones' 2024-01-15T18:09:00.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.954+0000 7fd930800640 1 Processor -- start 2024-01-15T18:09:00.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.954+0000 7fd930800640 1 -- start start 2024-01-15T18:09:00.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.958+0000 7fd930800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fd92815d2c0 0x7fd9280c62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:00.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.958+0000 7fd930800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd9280c6800 0x7fd92806a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:00.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.958+0000 7fd930800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd92806aa30 0x7fd9280712b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:00.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.958+0000 7fd930800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd92806aec0 con 0x7fd92806aa30 2024-01-15T18:09:00.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.958+0000 7fd930800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9280ce600 con 0x7fd92815d2c0 2024-01-15T18:09:00.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.958+0000 7fd930800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9280c5d00 con 0x7fd9280c6800 2024-01-15T18:09:00.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.958+0000 7fd92e800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd92806aa30 0x7fd9280712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:00.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.958+0000 7fd92e800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd92806aa30 0x7fd9280712b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59748/0 (socket says 172.21.15.27:59748) 2024-01-15T18:09:00.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.958+0000 7fd92e800640 1 -- 172.21.15.27:0/2894883452 learned_addr learned my addr 172.21.15.27:0/2894883452 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:00.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.958+0000 7fd92de00640 1 --2- 172.21.15.27:0/2894883452 >> v2:172.21.15.93:3300/0 conn(0x7fd92815d2c0 0x7fd9280c62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:00.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.958+0000 7fd92d400640 1 --2- 172.21.15.27:0/2894883452 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd9280c6800 0x7fd92806a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:00.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.958+0000 7fd92e800640 1 -- 172.21.15.27:0/2894883452 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd9280c6800 msgr2=0x7fd92806a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:00.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.958+0000 7fd92e800640 1 --2- 172.21.15.27:0/2894883452 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd9280c6800 0x7fd92806a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.958+0000 7fd92e800640 1 -- 172.21.15.27:0/2894883452 >> v2:172.21.15.93:3300/0 conn(0x7fd92815d2c0 msgr2=0x7fd9280c62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:00.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.958+0000 7fd92e800640 1 --2- 172.21.15.27:0/2894883452 >> v2:172.21.15.93:3300/0 conn(0x7fd92815d2c0 0x7fd9280c62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.958+0000 7fd92e800640 1 -- 172.21.15.27:0/2894883452 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92806fc90 con 0x7fd92806aa30 2024-01-15T18:09:00.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.958+0000 7fd92de00640 1 --2- 172.21.15.27:0/2894883452 >> v2:172.21.15.93:3300/0 conn(0x7fd92815d2c0 0x7fd9280c62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:00.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.958+0000 7fd92d400640 1 --2- 172.21.15.27:0/2894883452 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd9280c6800 0x7fd92806a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:00.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.958+0000 7fd92e800640 1 --2- 172.21.15.27:0/2894883452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd92806aa30 0x7fd9280712b0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fd928070fa0 tx=0x7fd92001c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9526020a900a5b11 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:00.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.958+0000 7fd92ca00640 1 -- 172.21.15.27:0/2894883452 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9200a6480 con 0x7fd92806aa30 2024-01-15T18:09:00.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.958+0000 7fd92ca00640 1 -- 172.21.15.27:0/2894883452 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9200a65e0 con 0x7fd92806aa30 2024-01-15T18:09:00.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.958+0000 7fd92ca00640 1 -- 172.21.15.27:0/2894883452 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9200a6970 con 0x7fd92806aa30 2024-01-15T18:09:00.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.962+0000 7fd930800640 1 -- 172.21.15.27:0/2894883452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd92806aa30 msgr2=0x7fd9280712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:00.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.962+0000 7fd930800640 1 --2- 172.21.15.27:0/2894883452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd92806aa30 0x7fd9280712b0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fd928070fa0 tx=0x7fd92001c6e0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.962+0000 7fd930800640 1 -- 172.21.15.27:0/2894883452 shutdown_connections 2024-01-15T18:09:00.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.962+0000 7fd930800640 1 --2- 172.21.15.27:0/2894883452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd92806aa30 0x7fd9280712b0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.962+0000 7fd930800640 1 --2- 172.21.15.27:0/2894883452 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd9280c6800 0x7fd92806a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.962+0000 7fd930800640 1 --2- 172.21.15.27:0/2894883452 >> v2:172.21.15.93:3300/0 conn(0x7fd92815d2c0 0x7fd9280c62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.962+0000 7fd930800640 1 -- 172.21.15.27:0/2894883452 >> 172.21.15.27:0/2894883452 conn(0x7fd9280cfd60 msgr2=0x7fd9280d0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:00.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.962+0000 7fd930800640 1 -- 172.21.15.27:0/2894883452 shutdown_connections 2024-01-15T18:09:00.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.962+0000 7fd930800640 1 -- 172.21.15.27:0/2894883452 wait complete. 2024-01-15T18:09:00.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.962+0000 7fd930800640 1 Processor -- start 2024-01-15T18:09:00.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.962+0000 7fd930800640 1 -- start start 2024-01-15T18:09:00.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.962+0000 7fd930800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd92806aa30 0x7fd9280b4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:00.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.962+0000 7fd930800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd9280c6800 0x7fd92815a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:00.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.962+0000 7fd930800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd92815d2c0 0x7fd92815c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:00.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.962+0000 7fd930800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92815ab50 con 0x7fd92815d2c0 2024-01-15T18:09:00.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.962+0000 7fd930800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92815ad60 con 0x7fd9280c6800 2024-01-15T18:09:00.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.962+0000 7fd930800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd92815b040 con 0x7fd92806aa30 2024-01-15T18:09:00.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.966+0000 7fd92e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd92815d2c0 0x7fd92815c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.966+0000 7fd92de00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd92806aa30 0x7fd9280b4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.966+0000 7fd92de00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd92806aa30 0x7fd9280b4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:49366/0 (socket says 172.21.15.27:49366) 2024-01-15T18:09:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.966+0000 7fd92d400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd9280c6800 0x7fd92815a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.966+0000 7fd92de00640 1 -- 172.21.15.27:0/3050903030 learned_addr learned my addr 172.21.15.27:0/3050903030 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.966+0000 7fd92de00640 1 -- 172.21.15.27:0/3050903030 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd9280c6800 msgr2=0x7fd92815a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.966+0000 7fd92de00640 1 --2- 172.21.15.27:0/3050903030 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd9280c6800 0x7fd92815a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.966+0000 7fd92de00640 1 -- 172.21.15.27:0/3050903030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd92815d2c0 msgr2=0x7fd92815c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.966+0000 7fd92de00640 1 --2- 172.21.15.27:0/3050903030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd92815d2c0 0x7fd92815c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.966+0000 7fd92de00640 1 -- 172.21.15.27:0/3050903030 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd920009b30 con 0x7fd92806aa30 2024-01-15T18:09:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.966+0000 7fd92e800640 1 --2- 172.21.15.27:0/3050903030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd92815d2c0 0x7fd92815c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:09:00.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.966+0000 7fd92de00640 1 --2- 172.21.15.27:0/3050903030 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd92806aa30 0x7fd9280b4d20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd91800c7f0 tx=0x7fd91800ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:00.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.966+0000 7fd926000640 1 -- 172.21.15.27:0/3050903030 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd918007820 con 0x7fd92806aa30 2024-01-15T18:09:00.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.966+0000 7fd926000640 1 -- 172.21.15.27:0/3050903030 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd918007980 con 0x7fd92806aa30 2024-01-15T18:09:00.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.966+0000 7fd926000640 1 -- 172.21.15.27:0/3050903030 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd918007d30 con 0x7fd92806aa30 2024-01-15T18:09:00.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.966+0000 7fd930800640 1 -- 172.21.15.27:0/3050903030 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd92815b320 con 0x7fd92806aa30 2024-01-15T18:09:00.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.966+0000 7fd930800640 1 -- 172.21.15.27:0/3050903030 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd92815b8e0 con 0x7fd92806aa30 2024-01-15T18:09:00.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.966+0000 7fd930800640 1 -- 172.21.15.27:0/3050903030 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd928067c00 con 0x7fd92806aa30 2024-01-15T18:09:00.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.966+0000 7fd926000640 1 -- 172.21.15.27:0/3050903030 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7fd9180113f0 con 0x7fd92806aa30 2024-01-15T18:09:00.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.970+0000 7fd926000640 1 --2- 172.21.15.27:0/3050903030 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd900066e90 0x7fd900069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:00.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.970+0000 7fd926000640 1 -- 172.21.15.27:0/3050903030 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fd918084ec0 con 0x7fd92806aa30 2024-01-15T18:09:00.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.970+0000 7fd92d400640 1 --2- 172.21.15.27:0/3050903030 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd900066e90 0x7fd900069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:00.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.970+0000 7fd92d400640 1 --2- 172.21.15.27:0/3050903030 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd900066e90 0x7fd900069350 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd92815c270 tx=0x7fd910008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:00.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:00.974+0000 7fd926000640 1 -- 172.21.15.27:0/3050903030 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd918051b10 con 0x7fd92806aa30 2024-01-15T18:09:01.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:01.186+0000 7fd930800640 1 -- 172.21.15.27:0/3050903030 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones"]} v 0) v1 -- 0x7fd92804e850 con 0x7fd92806aa30 2024-01-15T18:09:02.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.066+0000 7fd926000640 1 -- 172.21.15.27:0/3050903030 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones"]}]=0 v219) v1 ==== 153+0+0 (secure 0 0 0) 0x7fd91801f0e0 con 0x7fd92806aa30 2024-01-15T18:09:02.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.070+0000 7fd930800640 1 -- 172.21.15.27:0/3050903030 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd900066e90 msgr2=0x7fd900069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:02.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.070+0000 7fd930800640 1 --2- 172.21.15.27:0/3050903030 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd900066e90 0x7fd900069350 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd92815c270 tx=0x7fd910008040 comp rx=0 tx=0).stop 2024-01-15T18:09:02.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.070+0000 7fd930800640 1 -- 172.21.15.27:0/3050903030 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd92806aa30 msgr2=0x7fd9280b4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:02.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.070+0000 7fd930800640 1 --2- 172.21.15.27:0/3050903030 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd92806aa30 0x7fd9280b4d20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd91800c7f0 tx=0x7fd91800ccb0 comp rx=0 tx=0).stop 2024-01-15T18:09:02.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.070+0000 7fd930800640 1 -- 172.21.15.27:0/3050903030 shutdown_connections 2024-01-15T18:09:02.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.070+0000 7fd930800640 1 --2- 172.21.15.27:0/3050903030 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd900066e90 0x7fd900069350 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:02.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.070+0000 7fd930800640 1 --2- 172.21.15.27:0/3050903030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd92815d2c0 0x7fd92815c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:02.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.070+0000 7fd930800640 1 --2- 172.21.15.27:0/3050903030 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd9280c6800 0x7fd92815a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:02.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.070+0000 7fd930800640 1 --2- 172.21.15.27:0/3050903030 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd92806aa30 0x7fd9280b4d20 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:02.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.070+0000 7fd930800640 1 -- 172.21.15.27:0/3050903030 >> 172.21.15.27:0/3050903030 conn(0x7fd9280cfd60 msgr2=0x7fd9280afec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:02.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.070+0000 7fd930800640 1 -- 172.21.15.27:0/3050903030 shutdown_connections 2024-01-15T18:09:02.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.070+0000 7fd930800640 1 -- 172.21.15.27:0/3050903030 wait complete. 2024-01-15T18:09:02.100 INFO:tasks.cephfs_test_runner:X = number of clones issued. ... ok 2024-01-15T18:09:02.101 INFO:tasks.cephfs_test_runner:Starting test: test_progress_bars_for_7_clones (tasks.cephfs.test_volumes.TestCloneProgress) X = number of clones issued. 2024-01-15T18:09:02.101 INFO:tasks.cephfs_test_runner:test_progress_bars_for_7_clones (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T18:09:02.102 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:09:02.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.242+0000 7fd982e00640 1 Processor -- start 2024-01-15T18:09:02.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.242+0000 7fd982e00640 1 -- start start 2024-01-15T18:09:02.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.242+0000 7fd982e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd97c0b2390 0x7fd97c0cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:02.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.242+0000 7fd982e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97c0b2770 con 0x7fd97c0cdf90 2024-01-15T18:09:02.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.242+0000 7fd982e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97c06c060 con 0x7fd97c06bb80 2024-01-15T18:09:02.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.242+0000 7fd982e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd97c06c340 con 0x7fd97c0b2390 2024-01-15T18:09:02.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.242+0000 7fd980e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd97c0cdf90 0x7fd97c0b0880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48496/0 (socket says 172.21.15.27:48496) 2024-01-15T18:09:02.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.242+0000 7fd980e00640 1 -- 172.21.15.27:0/711782393 learned_addr learned my addr 172.21.15.27:0/711782393 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:02.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.242+0000 7fd97be00640 1 --2- 172.21.15.27:0/711782393 >> v2:172.21.15.27:3300/0 conn(0x7fd97c0b2390 0x7fd97c0cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:02.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.242+0000 7fd97be00640 1 -- 172.21.15.27:0/711782393 >> v1:172.21.15.93:6789/0 conn(0x7fd97c06bb80 legacy=0x7fd97c0cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:02.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.242+0000 7fd97be00640 1 -- 172.21.15.27:0/711782393 >> v1:172.21.15.27:6789/0 conn(0x7fd97c0cdf90 legacy=0x7fd97c0b0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:02.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.242+0000 7fd97be00640 1 -- 172.21.15.27:0/711782393 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd97c06c620 con 0x7fd97c0b2390 2024-01-15T18:09:02.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.246+0000 7fd97be00640 1 --2- 172.21.15.27:0/711782393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd97c0b2390 0x7fd97c0cf780 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fd96c0097c0 tx=0x7fd96c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa1b34ec37f9386 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:02.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.246+0000 7fd97aa00640 1 -- 172.21.15.27:0/711782393 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c0ab650 con 0x7fd97c0b2390 2024-01-15T18:09:02.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.246+0000 7fd97aa00640 1 -- 172.21.15.27:0/711782393 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96c0ab7b0 con 0x7fd97c0b2390 2024-01-15T18:09:02.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.246+0000 7fd97aa00640 1 -- 172.21.15.27:0/711782393 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c0abb40 con 0x7fd97c0b2390 2024-01-15T18:09:02.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.246+0000 7fd982e00640 1 -- 172.21.15.27:0/711782393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd97c0b2390 msgr2=0x7fd97c0cf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:02.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.246+0000 7fd982e00640 1 --2- 172.21.15.27:0/711782393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd97c0b2390 0x7fd97c0cf780 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fd96c0097c0 tx=0x7fd96c01c450 comp rx=0 tx=0).stop 2024-01-15T18:09:02.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.246+0000 7fd982e00640 1 -- 172.21.15.27:0/711782393 shutdown_connections 2024-01-15T18:09:02.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.246+0000 7fd982e00640 1 --2- 172.21.15.27:0/711782393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd97c0b2390 0x7fd97c0cf780 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:02.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.246+0000 7fd982e00640 1 -- 172.21.15.27:0/711782393 >> 172.21.15.27:0/711782393 conn(0x7fd97c0b2a10 msgr2=0x7fd97c0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:02.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.246+0000 7fd982e00640 1 -- 172.21.15.27:0/711782393 shutdown_connections 2024-01-15T18:09:02.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.246+0000 7fd982e00640 1 -- 172.21.15.27:0/711782393 wait complete. 2024-01-15T18:09:02.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.250+0000 7fd982e00640 1 Processor -- start 2024-01-15T18:09:02.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.250+0000 7fd982e00640 1 -- start start 2024-01-15T18:09:02.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.250+0000 7fd982e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd97c06bb80 0x7fd97c160940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:02.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.250+0000 7fd982e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd97c0b2390 0x7fd97c0adcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:02.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.250+0000 7fd982e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd97c0cdf90 0x7fd97c0ae230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:02.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.250+0000 7fd982e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c14cbc0 con 0x7fd97c06bb80 2024-01-15T18:09:02.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.250+0000 7fd982e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c0c8cd0 con 0x7fd97c0b2390 2024-01-15T18:09:02.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.250+0000 7fd982e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c160e80 con 0x7fd97c0cdf90 2024-01-15T18:09:02.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.250+0000 7fd97be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd97c06bb80 0x7fd97c160940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:02.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.250+0000 7fd97be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd97c06bb80 0x7fd97c160940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59766/0 (socket says 172.21.15.27:59766) 2024-01-15T18:09:02.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.250+0000 7fd97be00640 1 -- 172.21.15.27:0/1899032944 learned_addr learned my addr 172.21.15.27:0/1899032944 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:02.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.250+0000 7fd97b400640 1 --2- 172.21.15.27:0/1899032944 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd97c0b2390 0x7fd97c0adcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:02.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.250+0000 7fd980e00640 1 --2- 172.21.15.27:0/1899032944 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd97c0cdf90 0x7fd97c0ae230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:02.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.250+0000 7fd97be00640 1 -- 172.21.15.27:0/1899032944 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd97c0cdf90 msgr2=0x7fd97c0ae230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:02.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.250+0000 7fd97be00640 1 --2- 172.21.15.27:0/1899032944 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd97c0cdf90 0x7fd97c0ae230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:02.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.250+0000 7fd97be00640 1 -- 172.21.15.27:0/1899032944 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd97c0b2390 msgr2=0x7fd97c0adcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:02.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.250+0000 7fd97be00640 1 --2- 172.21.15.27:0/1899032944 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd97c0b2390 0x7fd97c0adcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:02.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.250+0000 7fd97be00640 1 -- 172.21.15.27:0/1899032944 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96c009420 con 0x7fd97c06bb80 2024-01-15T18:09:02.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.250+0000 7fd97be00640 1 --2- 172.21.15.27:0/1899032944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd97c06bb80 0x7fd97c160940 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fd96c0ad040 tx=0x7fd96c01ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:02.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.250+0000 7fd973e00640 1 -- 172.21.15.27:0/1899032944 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c0b5810 con 0x7fd97c06bb80 2024-01-15T18:09:02.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.250+0000 7fd973e00640 1 -- 172.21.15.27:0/1899032944 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96c0b5990 con 0x7fd97c06bb80 2024-01-15T18:09:02.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.250+0000 7fd973e00640 1 -- 172.21.15.27:0/1899032944 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c0b5d40 con 0x7fd97c06bb80 2024-01-15T18:09:02.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.250+0000 7fd982e00640 1 -- 172.21.15.27:0/1899032944 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd97c0c80c0 con 0x7fd97c06bb80 2024-01-15T18:09:02.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.250+0000 7fd982e00640 1 -- 172.21.15.27:0/1899032944 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd97c0c8680 con 0x7fd97c06bb80 2024-01-15T18:09:02.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.254+0000 7fd982e00640 1 -- 172.21.15.27:0/1899032944 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd97c0cf310 con 0x7fd97c06bb80 2024-01-15T18:09:02.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.254+0000 7fd973e00640 1 -- 172.21.15.27:0/1899032944 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7fd96c0bf410 con 0x7fd97c06bb80 2024-01-15T18:09:02.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.254+0000 7fd973e00640 1 --2- 172.21.15.27:0/1899032944 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd954066e90 0x7fd954069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:02.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.254+0000 7fd973e00640 1 -- 172.21.15.27:0/1899032944 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fd96c11fab0 con 0x7fd97c06bb80 2024-01-15T18:09:02.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.254+0000 7fd97b400640 1 --2- 172.21.15.27:0/1899032944 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd954066e90 0x7fd954069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:02.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.254+0000 7fd97b400640 1 --2- 172.21.15.27:0/1899032944 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd954066e90 0x7fd954069350 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd97c0cb160 tx=0x7fd964009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:02.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.258+0000 7fd973e00640 1 -- 172.21.15.27:0/1899032944 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd96c0ec910 con 0x7fd97c06bb80 2024-01-15T18:09:02.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.418+0000 7fd982e00640 1 -- 172.21.15.27:0/1899032944 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd97c04e850 con 0x7fd97c06bb80 2024-01-15T18:09:02.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.418+0000 7fd973e00640 1 -- 172.21.15.27:0/1899032944 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+12933 (secure 0 0 0) 0x7fd96c0eff30 con 0x7fd97c06bb80 2024-01-15T18:09:02.424 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:09:02.424 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":50,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:08:56.019879+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6835/1005016790":"2024-01-16T18:08:52.999612+0000","172.21.15.27:6833/1005016790":"2024-01-16T18:08:52.999612+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:09:02.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.418+0000 7fd982e00640 1 -- 172.21.15.27:0/1899032944 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd954066e90 msgr2=0x7fd954069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:02.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.418+0000 7fd982e00640 1 --2- 172.21.15.27:0/1899032944 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd954066e90 0x7fd954069350 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd97c0cb160 tx=0x7fd964009040 comp rx=0 tx=0).stop 2024-01-15T18:09:02.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.418+0000 7fd982e00640 1 -- 172.21.15.27:0/1899032944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd97c06bb80 msgr2=0x7fd97c160940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:02.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.418+0000 7fd982e00640 1 --2- 172.21.15.27:0/1899032944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd97c06bb80 0x7fd97c160940 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fd96c0ad040 tx=0x7fd96c01ac90 comp rx=0 tx=0).stop 2024-01-15T18:09:02.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.418+0000 7fd982e00640 1 -- 172.21.15.27:0/1899032944 shutdown_connections 2024-01-15T18:09:02.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.418+0000 7fd982e00640 1 --2- 172.21.15.27:0/1899032944 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd954066e90 0x7fd954069350 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:02.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.418+0000 7fd982e00640 1 --2- 172.21.15.27:0/1899032944 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd97c0cdf90 0x7fd97c0ae230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:02.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.418+0000 7fd982e00640 1 --2- 172.21.15.27:0/1899032944 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd97c0b2390 0x7fd97c0adcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:02.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.418+0000 7fd982e00640 1 --2- 172.21.15.27:0/1899032944 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd97c06bb80 0x7fd97c160940 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:02.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.418+0000 7fd982e00640 1 -- 172.21.15.27:0/1899032944 >> 172.21.15.27:0/1899032944 conn(0x7fd97c0b2a10 msgr2=0x7fd97c0cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:02.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.418+0000 7fd982e00640 1 -- 172.21.15.27:0/1899032944 shutdown_connections 2024-01-15T18:09:02.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.418+0000 7fd982e00640 1 -- 172.21.15.27:0/1899032944 wait complete. 2024-01-15T18:09:02.447 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:09:02.447 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:09:02.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.586+0000 7ff6d7e00640 1 Processor -- start 2024-01-15T18:09:02.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.586+0000 7ff6d7e00640 1 -- start start 2024-01-15T18:09:02.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.586+0000 7ff6d7e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff6d0158900 0x7ff6d015f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:02.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.586+0000 7ff6d7e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d00b1cb0 con 0x7ff6d015bf00 2024-01-15T18:09:02.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.586+0000 7ff6d7e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d00b1f00 con 0x7ff6d00cf2e0 2024-01-15T18:09:02.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.586+0000 7ff6d7e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6d00b21e0 con 0x7ff6d0158900 2024-01-15T18:09:02.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.586+0000 7ff6d4a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff6d015bf00 0x7ff6d00cebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48510/0 (socket says 172.21.15.27:48510) 2024-01-15T18:09:02.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.586+0000 7ff6d4a00640 1 -- 172.21.15.27:0/174255483 learned_addr learned my addr 172.21.15.27:0/174255483 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:02.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.586+0000 7ff6d5400640 1 --2- 172.21.15.27:0/174255483 >> v2:172.21.15.93:3300/0 conn(0x7ff6d0158900 0x7ff6d015f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:02.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.586+0000 7ff6cfe00640 1 -- 172.21.15.27:0/174255483 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4259114029 0 0) 0x7ff6d00b1cb0 con 0x7ff6d015bf00 2024-01-15T18:09:02.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.586+0000 7ff6cfe00640 1 -- 172.21.15.27:0/174255483 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6b0003530 con 0x7ff6d015bf00 2024-01-15T18:09:02.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.586+0000 7ff6d5400640 1 -- 172.21.15.27:0/174255483 >> v1:172.21.15.93:6789/0 conn(0x7ff6d00cf2e0 legacy=0x7ff6d00701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:02.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.586+0000 7ff6d5400640 1 -- 172.21.15.27:0/174255483 >> v1:172.21.15.27:6789/0 conn(0x7ff6d015bf00 legacy=0x7ff6d00cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:02.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.586+0000 7ff6d5400640 1 -- 172.21.15.27:0/174255483 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d015ea70 con 0x7ff6d0158900 2024-01-15T18:09:02.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.590+0000 7ff6d5400640 1 --2- 172.21.15.27:0/174255483 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff6d0158900 0x7ff6d015f550 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00097c0 tx=0x7ff6c001c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32cb6aa7c66183f7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:02.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.590+0000 7ff6cfe00640 1 -- 172.21.15.27:0/174255483 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c00ab600 con 0x7ff6d0158900 2024-01-15T18:09:02.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.590+0000 7ff6cfe00640 1 -- 172.21.15.27:0/174255483 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c00a7050 con 0x7ff6d0158900 2024-01-15T18:09:02.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.590+0000 7ff6cfe00640 1 -- 172.21.15.27:0/174255483 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c00ab980 con 0x7ff6d0158900 2024-01-15T18:09:02.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.590+0000 7ff6d7e00640 1 -- 172.21.15.27:0/174255483 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff6d0158900 msgr2=0x7ff6d015f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:02.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.590+0000 7ff6d7e00640 1 --2- 172.21.15.27:0/174255483 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff6d0158900 0x7ff6d015f550 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00097c0 tx=0x7ff6c001c480 comp rx=0 tx=0).stop 2024-01-15T18:09:02.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.590+0000 7ff6d7e00640 1 -- 172.21.15.27:0/174255483 shutdown_connections 2024-01-15T18:09:02.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.590+0000 7ff6d7e00640 1 --2- 172.21.15.27:0/174255483 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff6d0158900 0x7ff6d015f550 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:02.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.590+0000 7ff6d7e00640 1 -- 172.21.15.27:0/174255483 >> 172.21.15.27:0/174255483 conn(0x7ff6d00cb700 msgr2=0x7ff6d00cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:02.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.590+0000 7ff6d7e00640 1 -- 172.21.15.27:0/174255483 shutdown_connections 2024-01-15T18:09:02.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.590+0000 7ff6d7e00640 1 -- 172.21.15.27:0/174255483 wait complete. 2024-01-15T18:09:02.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.594+0000 7ff6d7e00640 1 Processor -- start 2024-01-15T18:09:02.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.594+0000 7ff6d7e00640 1 -- start start 2024-01-15T18:09:02.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.594+0000 7ff6d7e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff6d00cf2e0 0x7ff6d014f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:02.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.594+0000 7ff6d7e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff6d0158900 0x7ff6d00af650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:02.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.594+0000 7ff6d7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d015bf00 0x7ff6d014c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:02.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.594+0000 7ff6d7e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d0152cd0 con 0x7ff6d015bf00 2024-01-15T18:09:02.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.594+0000 7ff6d7e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d0153190 con 0x7ff6d00cf2e0 2024-01-15T18:09:02.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.594+0000 7ff6d7e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d014cce0 con 0x7ff6d0158900 2024-01-15T18:09:02.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.594+0000 7ff6d5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d015bf00 0x7ff6d014c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:02.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.594+0000 7ff6d5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d015bf00 0x7ff6d014c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59774/0 (socket says 172.21.15.27:59774) 2024-01-15T18:09:02.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.594+0000 7ff6d5e00640 1 -- 172.21.15.27:0/1761823330 learned_addr learned my addr 172.21.15.27:0/1761823330 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:02.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.594+0000 7ff6d5400640 1 --2- 172.21.15.27:0/1761823330 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff6d00cf2e0 0x7ff6d014f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:02.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.594+0000 7ff6d4a00640 1 --2- 172.21.15.27:0/1761823330 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff6d0158900 0x7ff6d00af650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:02.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.594+0000 7ff6d5e00640 1 -- 172.21.15.27:0/1761823330 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff6d0158900 msgr2=0x7ff6d00af650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:02.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.594+0000 7ff6d5e00640 1 --2- 172.21.15.27:0/1761823330 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff6d0158900 0x7ff6d00af650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:02.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.594+0000 7ff6d5e00640 1 -- 172.21.15.27:0/1761823330 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff6d00cf2e0 msgr2=0x7ff6d014f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:02.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.594+0000 7ff6d5e00640 1 --2- 172.21.15.27:0/1761823330 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff6d00cf2e0 0x7ff6d014f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:02.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.594+0000 7ff6d5e00640 1 -- 172.21.15.27:0/1761823330 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c0009420 con 0x7ff6d015bf00 2024-01-15T18:09:02.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.594+0000 7ff6d5e00640 1 --2- 172.21.15.27:0/1761823330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d015bf00 0x7ff6d014c7a0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800c7d0 tx=0x7ff6c800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:02.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.594+0000 7ff6cd600640 1 -- 172.21.15.27:0/1761823330 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c8003b80 con 0x7ff6d015bf00 2024-01-15T18:09:02.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.594+0000 7ff6cd600640 1 -- 172.21.15.27:0/1761823330 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c8003ce0 con 0x7ff6d015bf00 2024-01-15T18:09:02.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.594+0000 7ff6d7e00640 1 -- 172.21.15.27:0/1761823330 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d0152eb0 con 0x7ff6d015bf00 2024-01-15T18:09:02.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.594+0000 7ff6cd600640 1 -- 172.21.15.27:0/1761823330 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c800aa30 con 0x7ff6d015bf00 2024-01-15T18:09:02.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.594+0000 7ff6d7e00640 1 -- 172.21.15.27:0/1761823330 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d01515d0 con 0x7ff6d015bf00 2024-01-15T18:09:02.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.598+0000 7ff6d7e00640 1 -- 172.21.15.27:0/1761823330 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d004e850 con 0x7ff6d015bf00 2024-01-15T18:09:02.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.598+0000 7ff6cd600640 1 -- 172.21.15.27:0/1761823330 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7ff6c800ab90 con 0x7ff6d015bf00 2024-01-15T18:09:02.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.598+0000 7ff6cd600640 1 --2- 172.21.15.27:0/1761823330 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff6a8066ec0 0x7ff6a8069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:02.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.598+0000 7ff6cd600640 1 -- 172.21.15.27:0/1761823330 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4971+0+0 (secure 0 0 0) 0x7ff6c8084860 con 0x7ff6d015bf00 2024-01-15T18:09:02.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.598+0000 7ff6d5400640 1 --2- 172.21.15.27:0/1761823330 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff6a8066ec0 0x7ff6a8069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:02.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.598+0000 7ff6d5400640 1 --2- 172.21.15.27:0/1761823330 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff6a8066ec0 0x7ff6a8069380 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00ad000 tx=0x7ff6c00a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:02.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.602+0000 7ff6cd600640 1 -- 172.21.15.27:0/1761823330 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff6c8051610 con 0x7ff6d015bf00 2024-01-15T18:09:02.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.762+0000 7ff6d7e00640 1 -- 172.21.15.27:0/1761823330 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff6d00b0720 con 0x7ff6d015bf00 2024-01-15T18:09:02.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.762+0000 7ff6cd600640 1 -- 172.21.15.27:0/1761823330 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff6c8054c30 con 0x7ff6d015bf00 2024-01-15T18:09:02.767 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:09:02.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.762+0000 7ff6d7e00640 1 -- 172.21.15.27:0/1761823330 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff6a8066ec0 msgr2=0x7ff6a8069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:02.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.762+0000 7ff6d7e00640 1 --2- 172.21.15.27:0/1761823330 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff6a8066ec0 0x7ff6a8069380 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00ad000 tx=0x7ff6c00a6000 comp rx=0 tx=0).stop 2024-01-15T18:09:02.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.762+0000 7ff6d7e00640 1 -- 172.21.15.27:0/1761823330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d015bf00 msgr2=0x7ff6d014c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:02.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.762+0000 7ff6d7e00640 1 --2- 172.21.15.27:0/1761823330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d015bf00 0x7ff6d014c7a0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7ff6c800c7d0 tx=0x7ff6c800cc90 comp rx=0 tx=0).stop 2024-01-15T18:09:02.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.762+0000 7ff6d7e00640 1 -- 172.21.15.27:0/1761823330 shutdown_connections 2024-01-15T18:09:02.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.762+0000 7ff6d7e00640 1 --2- 172.21.15.27:0/1761823330 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff6a8066ec0 0x7ff6a8069380 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:02.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.762+0000 7ff6d7e00640 1 --2- 172.21.15.27:0/1761823330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff6d015bf00 0x7ff6d014c7a0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:02.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.762+0000 7ff6d7e00640 1 --2- 172.21.15.27:0/1761823330 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff6d0158900 0x7ff6d00af650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:02.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.762+0000 7ff6d7e00640 1 --2- 172.21.15.27:0/1761823330 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff6d00cf2e0 0x7ff6d014f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:02.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.762+0000 7ff6d7e00640 1 -- 172.21.15.27:0/1761823330 >> 172.21.15.27:0/1761823330 conn(0x7ff6d00cb700 msgr2=0x7ff6d00cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:02.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.762+0000 7ff6d7e00640 1 -- 172.21.15.27:0/1761823330 shutdown_connections 2024-01-15T18:09:02.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.762+0000 7ff6d7e00640 1 -- 172.21.15.27:0/1761823330 wait complete. 2024-01-15T18:09:02.790 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones' 2024-01-15T18:09:02.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.926+0000 7f6d64000640 1 Processor -- start 2024-01-15T18:09:02.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.930+0000 7f6d64000640 1 -- start start 2024-01-15T18:09:02.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.930+0000 7f6d64000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6d5c06db80 0x7f6d5c06df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:02.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.930+0000 7f6d64000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f6d5c15e730 0x7f6d5c0cd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:02.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.930+0000 7f6d64000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d5c0ce920 con 0x7f6d5c0cdd10 2024-01-15T18:09:02.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.930+0000 7f6d64000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d5c06e4a0 con 0x7f6d5c06db80 2024-01-15T18:09:02.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.930+0000 7f6d64000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d5c0ce640 con 0x7f6d5c15e730 2024-01-15T18:09:02.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.930+0000 7f6d61600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6d5c06db80 0x7f6d5c06df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:02.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.930+0000 7f6d62000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6d5c0cdd10 0x7f6d5c070c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48516/0 (socket says 172.21.15.27:48516) 2024-01-15T18:09:02.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.930+0000 7f6d62000640 1 -- 172.21.15.27:0/1848035279 learned_addr learned my addr 172.21.15.27:0/1848035279 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:02.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.930+0000 7f6d60c00640 1 --2- 172.21.15.27:0/1848035279 >> v2:172.21.15.93:3300/0 conn(0x7f6d5c15e730 0x7f6d5c0cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:02.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.930+0000 7f6d61600640 1 -- 172.21.15.27:0/1848035279 >> v2:172.21.15.93:3300/0 conn(0x7f6d5c15e730 msgr2=0x7f6d5c0cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:02.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.930+0000 7f6d61600640 1 --2- 172.21.15.27:0/1848035279 >> v2:172.21.15.93:3300/0 conn(0x7f6d5c15e730 0x7f6d5c0cd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:02.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.930+0000 7f6d61600640 1 -- 172.21.15.27:0/1848035279 >> v1:172.21.15.27:6789/0 conn(0x7f6d5c0cdd10 legacy=0x7f6d5c070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:02.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.930+0000 7f6d61600640 1 -- 172.21.15.27:0/1848035279 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d5c0cc940 con 0x7f6d5c06db80 2024-01-15T18:09:02.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.930+0000 7f6d60c00640 1 --2- 172.21.15.27:0/1848035279 >> v2:172.21.15.93:3300/0 conn(0x7f6d5c15e730 0x7f6d5c0cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:09:02.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.934+0000 7f6d61600640 1 --2- 172.21.15.27:0/1848035279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d5c06db80 0x7f6d5c06df60 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c0cfd00 tx=0x7f6d4c01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=824318b6fd0d1b0f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:02.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.934+0000 7f6d5be00640 1 -- 172.21.15.27:0/1848035279 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c0a7070 con 0x7f6d5c06db80 2024-01-15T18:09:02.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.934+0000 7f6d5be00640 1 -- 172.21.15.27:0/1848035279 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4c0ab430 con 0x7f6d5c06db80 2024-01-15T18:09:02.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.934+0000 7f6d5be00640 1 -- 172.21.15.27:0/1848035279 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c0ab7c0 con 0x7f6d5c06db80 2024-01-15T18:09:02.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.934+0000 7f6d64000640 1 -- 172.21.15.27:0/1848035279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d5c06db80 msgr2=0x7f6d5c06df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:02.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.934+0000 7f6d64000640 1 --2- 172.21.15.27:0/1848035279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d5c06db80 0x7f6d5c06df60 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c0cfd00 tx=0x7f6d4c01c430 comp rx=0 tx=0).stop 2024-01-15T18:09:02.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.934+0000 7f6d64000640 1 -- 172.21.15.27:0/1848035279 shutdown_connections 2024-01-15T18:09:02.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.934+0000 7f6d64000640 1 --2- 172.21.15.27:0/1848035279 >> v2:172.21.15.93:3300/0 conn(0x7f6d5c15e730 0x7f6d5c0cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:02.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.934+0000 7f6d64000640 1 --2- 172.21.15.27:0/1848035279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d5c06db80 0x7f6d5c06df60 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:02.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.934+0000 7f6d64000640 1 -- 172.21.15.27:0/1848035279 >> 172.21.15.27:0/1848035279 conn(0x7f6d5c0cec60 msgr2=0x7f6d5c0cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:02.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.934+0000 7f6d64000640 1 -- 172.21.15.27:0/1848035279 shutdown_connections 2024-01-15T18:09:02.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.934+0000 7f6d64000640 1 -- 172.21.15.27:0/1848035279 wait complete. 2024-01-15T18:09:02.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.934+0000 7f6d64000640 1 Processor -- start 2024-01-15T18:09:02.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.938+0000 7f6d64000640 1 -- start start 2024-01-15T18:09:02.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.938+0000 7f6d64000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d5c06db80 0x7f6d5c152120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:02.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.938+0000 7f6d64000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6d5c0cdd10 0x7f6d5c162fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:02.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.938+0000 7f6d64000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6d5c15e730 0x7f6d5c1634f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:02.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.938+0000 7f6d64000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5c152660 con 0x7f6d5c06db80 2024-01-15T18:09:02.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.938+0000 7f6d64000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5c163ae0 con 0x7f6d5c0cdd10 2024-01-15T18:09:02.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.938+0000 7f6d64000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5c163dc0 con 0x7f6d5c15e730 2024-01-15T18:09:02.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.938+0000 7f6d61600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d5c06db80 0x7f6d5c152120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:02.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.938+0000 7f6d61600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d5c06db80 0x7f6d5c152120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59794/0 (socket says 172.21.15.27:59794) 2024-01-15T18:09:02.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.938+0000 7f6d61600640 1 -- 172.21.15.27:0/1702441643 learned_addr learned my addr 172.21.15.27:0/1702441643 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:02.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.938+0000 7f6d60c00640 1 --2- 172.21.15.27:0/1702441643 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6d5c0cdd10 0x7f6d5c162fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:02.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.938+0000 7f6d62000640 1 --2- 172.21.15.27:0/1702441643 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6d5c15e730 0x7f6d5c1634f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:02.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.938+0000 7f6d61600640 1 -- 172.21.15.27:0/1702441643 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6d5c15e730 msgr2=0x7f6d5c1634f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:02.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.938+0000 7f6d61600640 1 --2- 172.21.15.27:0/1702441643 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6d5c15e730 0x7f6d5c1634f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:02.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.938+0000 7f6d61600640 1 -- 172.21.15.27:0/1702441643 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6d5c0cdd10 msgr2=0x7f6d5c162fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:02.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.938+0000 7f6d61600640 1 --2- 172.21.15.27:0/1702441643 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6d5c0cdd10 0x7f6d5c162fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:02.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.938+0000 7f6d61600640 1 -- 172.21.15.27:0/1702441643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4c009420 con 0x7f6d5c06db80 2024-01-15T18:09:02.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.938+0000 7f6d61600640 1 --2- 172.21.15.27:0/1702441643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d5c06db80 0x7f6d5c152120 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c019040 tx=0x7f6d4c01ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:02.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.938+0000 7f6d59600640 1 -- 172.21.15.27:0/1702441643 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c0b5810 con 0x7f6d5c06db80 2024-01-15T18:09:02.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.938+0000 7f6d59600640 1 -- 172.21.15.27:0/1702441643 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4c0b5990 con 0x7f6d5c06db80 2024-01-15T18:09:02.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.938+0000 7f6d59600640 1 -- 172.21.15.27:0/1702441643 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c0b5d40 con 0x7f6d5c06db80 2024-01-15T18:09:02.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.938+0000 7f6d64000640 1 -- 172.21.15.27:0/1702441643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d5c1640a0 con 0x7f6d5c06db80 2024-01-15T18:09:02.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.938+0000 7f6d64000640 1 -- 172.21.15.27:0/1702441643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d5c164660 con 0x7f6d5c06db80 2024-01-15T18:09:02.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.938+0000 7f6d64000640 1 -- 172.21.15.27:0/1702441643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d5c04e850 con 0x7f6d5c06db80 2024-01-15T18:09:02.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.942+0000 7f6d59600640 1 -- 172.21.15.27:0/1702441643 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f6d4c0b4390 con 0x7f6d5c06db80 2024-01-15T18:09:02.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.942+0000 7f6d59600640 1 --2- 172.21.15.27:0/1702441643 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6d34066e90 0x7f6d34069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:02.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.942+0000 7f6d59600640 1 -- 172.21.15.27:0/1702441643 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f6d4c11f3c0 con 0x7f6d5c06db80 2024-01-15T18:09:02.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.942+0000 7f6d60c00640 1 --2- 172.21.15.27:0/1702441643 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6d34066e90 0x7f6d34069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:02.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.942+0000 7f6d60c00640 1 --2- 172.21.15.27:0/1702441643 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6d34066e90 0x7f6d34069350 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c0cb280 tx=0x7f6d44008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:02.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:02.946+0000 7f6d59600640 1 -- 172.21.15.27:0/1702441643 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6d4c0ec0f0 con 0x7f6d5c06db80 2024-01-15T18:09:03.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:03.162+0000 7f6d64000640 1 -- 172.21.15.27:0/1702441643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones"]} v 0) v1 -- 0x7f6d5c06c800 con 0x7f6d5c06db80 2024-01-15T18:09:04.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.074+0000 7f6d59600640 1 -- 172.21.15.27:0/1702441643 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones"]}]=0 v221) v1 ==== 157+0+0 (secure 0 0 0) 0x7f6d4c006e20 con 0x7f6d5c06db80 2024-01-15T18:09:04.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.074+0000 7f6d64000640 1 -- 172.21.15.27:0/1702441643 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6d34066e90 msgr2=0x7f6d34069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:04.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.074+0000 7f6d64000640 1 --2- 172.21.15.27:0/1702441643 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6d34066e90 0x7f6d34069350 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c0cb280 tx=0x7f6d44008040 comp rx=0 tx=0).stop 2024-01-15T18:09:04.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.074+0000 7f6d64000640 1 -- 172.21.15.27:0/1702441643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d5c06db80 msgr2=0x7f6d5c152120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:04.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.074+0000 7f6d64000640 1 --2- 172.21.15.27:0/1702441643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d5c06db80 0x7f6d5c152120 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c019040 tx=0x7f6d4c01ac70 comp rx=0 tx=0).stop 2024-01-15T18:09:04.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.074+0000 7f6d64000640 1 -- 172.21.15.27:0/1702441643 shutdown_connections 2024-01-15T18:09:04.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.074+0000 7f6d64000640 1 --2- 172.21.15.27:0/1702441643 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6d34066e90 0x7f6d34069350 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:04.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.074+0000 7f6d64000640 1 --2- 172.21.15.27:0/1702441643 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6d5c15e730 0x7f6d5c1634f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:04.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.074+0000 7f6d64000640 1 --2- 172.21.15.27:0/1702441643 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6d5c0cdd10 0x7f6d5c162fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:04.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.074+0000 7f6d64000640 1 --2- 172.21.15.27:0/1702441643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d5c06db80 0x7f6d5c152120 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:04.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.074+0000 7f6d64000640 1 -- 172.21.15.27:0/1702441643 >> 172.21.15.27:0/1702441643 conn(0x7f6d5c0cec60 msgr2=0x7f6d5c0b0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:04.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.074+0000 7f6d64000640 1 -- 172.21.15.27:0/1702441643 shutdown_connections 2024-01-15T18:09:04.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.074+0000 7f6d64000640 1 -- 172.21.15.27:0/1702441643 wait complete. 2024-01-15T18:09:04.106 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:09:04.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.246+0000 7f3d80c00640 1 Processor -- start 2024-01-15T18:09:04.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.246+0000 7f3d80c00640 1 -- start start 2024-01-15T18:09:04.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d80c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3d7c15ad20 0x7f3d7c15b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:04.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d80c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d7c15a610 con 0x7f3d7c0715f0 2024-01-15T18:09:04.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d80c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d7c15e3c0 con 0x7f3d7c15d900 2024-01-15T18:09:04.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d80c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c15e6a0 con 0x7f3d7c15ad20 2024-01-15T18:09:04.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d7a800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3d7c0715f0 0x7f3d7c15bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53974/0 (socket says 172.21.15.27:53974) 2024-01-15T18:09:04.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d7a800640 1 -- 172.21.15.27:0/1854526708 learned_addr learned my addr 172.21.15.27:0/1854526708 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:04.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d79e00640 1 --2- 172.21.15.27:0/1854526708 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3d7c15ad20 0x7f3d7c15b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:04.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d78a00640 1 -- 172.21.15.27:0/1854526708 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2914253707 0 0) 0x7f3d7c15a610 con 0x7f3d7c0715f0 2024-01-15T18:09:04.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d78a00640 1 -- 172.21.15.27:0/1854526708 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d5c003530 con 0x7f3d7c0715f0 2024-01-15T18:09:04.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d78a00640 1 -- 172.21.15.27:0/1854526708 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308819665 0 0) 0x7f3d7c15e3c0 con 0x7f3d7c15d900 2024-01-15T18:09:04.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d78a00640 1 -- 172.21.15.27:0/1854526708 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d7c15a610 con 0x7f3d7c15d900 2024-01-15T18:09:04.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d78a00640 1 -- 172.21.15.27:0/1854526708 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1302009752 0 0) 0x7f3d5c003530 con 0x7f3d7c0715f0 2024-01-15T18:09:04.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d78a00640 1 -- 172.21.15.27:0/1854526708 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3d7c15ad20 msgr2=0x7f3d7c15b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:04.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d78a00640 1 --2- 172.21.15.27:0/1854526708 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3d7c15ad20 0x7f3d7c15b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d78a00640 1 -- 172.21.15.27:0/1854526708 >> v1:172.21.15.93:6789/0 conn(0x7f3d7c15d900 legacy=0x7f3d7c070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d78a00640 1 -- 172.21.15.27:0/1854526708 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7c15b6e0 con 0x7f3d7c0715f0 2024-01-15T18:09:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d78a00640 1 -- 172.21.15.27:0/1854526708 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f3d74001a00 con 0x7f3d7c0715f0 2024-01-15T18:09:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d78a00640 1 -- 172.21.15.27:0/1854526708 >> v1:172.21.15.27:6789/0 conn(0x7f3d7c0715f0 legacy=0x7f3d7c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d78a00640 1 --2- 172.21.15.27:0/1854526708 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3d7c15ad20 0x7f3d5c003bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d78a00640 1 --2- 172.21.15.27:0/1854526708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d5c004160 0x7f3d5c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d78a00640 1 --2- 172.21.15.27:0/1854526708 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3d5c004d10 0x7f3d5c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d78a00640 1 -- 172.21.15.27:0/1854526708 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c15e6a0 con 0x7f3d5c004160 2024-01-15T18:09:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d78a00640 1 -- 172.21.15.27:0/1854526708 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5c00bb20 con 0x7f3d5c004d10 2024-01-15T18:09:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d78a00640 1 -- 172.21.15.27:0/1854526708 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5c00bce0 con 0x7f3d7c15ad20 2024-01-15T18:09:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d79400640 1 --2- 172.21.15.27:0/1854526708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d5c004160 0x7f3d5c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d7a800640 1 --2- 172.21.15.27:0/1854526708 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3d5c004d10 0x7f3d5c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d79e00640 1 --2- 172.21.15.27:0/1854526708 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3d7c15ad20 0x7f3d5c003bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:04.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d79400640 1 -- 172.21.15.27:0/1854526708 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3d7c15ad20 msgr2=0x7f3d5c003bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:04.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d79400640 1 --2- 172.21.15.27:0/1854526708 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3d7c15ad20 0x7f3d5c003bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:04.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d79400640 1 -- 172.21.15.27:0/1854526708 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3d5c004d10 msgr2=0x7f3d5c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:04.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d79400640 1 --2- 172.21.15.27:0/1854526708 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3d5c004d10 0x7f3d5c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:04.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.250+0000 7f3d79400640 1 -- 172.21.15.27:0/1854526708 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d5c003530 con 0x7f3d5c004160 2024-01-15T18:09:04.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.254+0000 7f3d79400640 1 --2- 172.21.15.27:0/1854526708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d5c004160 0x7f3d5c0047d0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f3d640018d0 tx=0x7f3d6401ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbbc6bf6afba6ec1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:04.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.254+0000 7f3d78a00640 1 -- 172.21.15.27:0/1854526708 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d640052d0 con 0x7f3d5c004160 2024-01-15T18:09:04.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.254+0000 7f3d78a00640 1 -- 172.21.15.27:0/1854526708 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d64005450 con 0x7f3d5c004160 2024-01-15T18:09:04.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.254+0000 7f3d78a00640 1 -- 172.21.15.27:0/1854526708 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d640a8930 con 0x7f3d5c004160 2024-01-15T18:09:04.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.254+0000 7f3d80c00640 1 -- 172.21.15.27:0/1854526708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d5c004160 msgr2=0x7f3d5c0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:04.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.254+0000 7f3d80c00640 1 --2- 172.21.15.27:0/1854526708 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d5c004160 0x7f3d5c0047d0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f3d640018d0 tx=0x7f3d6401ecd0 comp rx=0 tx=0).stop 2024-01-15T18:09:04.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.254+0000 7f3d7a800640 1 -- 172.21.15.27:0/1854526708 reap_dead start 2024-01-15T18:09:04.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.254+0000 7f3d80c00640 1 -- 172.21.15.27:0/1854526708 shutdown_connections 2024-01-15T18:09:04.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.254+0000 7f3d80c00640 1 -- 172.21.15.27:0/1854526708 >> 172.21.15.27:0/1854526708 conn(0x7f3d7c0cb700 msgr2=0x7f3d7c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:04.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.254+0000 7f3d80c00640 1 -- 172.21.15.27:0/1854526708 shutdown_connections 2024-01-15T18:09:04.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.254+0000 7f3d80c00640 1 -- 172.21.15.27:0/1854526708 wait complete. 2024-01-15T18:09:04.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.258+0000 7f3d80c00640 1 Processor -- start 2024-01-15T18:09:04.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.258+0000 7f3d80c00640 1 -- start start 2024-01-15T18:09:04.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.258+0000 7f3d80c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d5c004160 0x7f3d7c143830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:04.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.258+0000 7f3d80c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3d7c1552c0 0x7f3d7c1569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:04.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.258+0000 7f3d80c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3d7c1eec00 0x7f3d7c1ef080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:04.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.258+0000 7f3d80c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c0ce6e0 con 0x7f3d5c004160 2024-01-15T18:09:04.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.258+0000 7f3d80c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c155720 con 0x7f3d7c1eec00 2024-01-15T18:09:04.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.258+0000 7f3d80c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7c14e960 con 0x7f3d7c1552c0 2024-01-15T18:09:04.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.258+0000 7f3d79e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d5c004160 0x7f3d7c143830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:04.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.258+0000 7f3d79e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d5c004160 0x7f3d7c143830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43622/0 (socket says 172.21.15.27:43622) 2024-01-15T18:09:04.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.258+0000 7f3d79e00640 1 -- 172.21.15.27:0/2971160611 learned_addr learned my addr 172.21.15.27:0/2971160611 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:04.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.258+0000 7f3d7a800640 1 --2- 172.21.15.27:0/2971160611 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3d7c1eec00 0x7f3d7c1ef080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:04.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.258+0000 7f3d79400640 1 --2- 172.21.15.27:0/2971160611 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3d7c1552c0 0x7f3d7c1569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:04.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.258+0000 7f3d79e00640 1 -- 172.21.15.27:0/2971160611 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3d7c1552c0 msgr2=0x7f3d7c1569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:04.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.258+0000 7f3d79e00640 1 --2- 172.21.15.27:0/2971160611 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3d7c1552c0 0x7f3d7c1569e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:04.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.258+0000 7f3d79e00640 1 -- 172.21.15.27:0/2971160611 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3d7c1eec00 msgr2=0x7f3d7c1ef080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:04.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.258+0000 7f3d79e00640 1 --2- 172.21.15.27:0/2971160611 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3d7c1eec00 0x7f3d7c1ef080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:04.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.258+0000 7f3d79e00640 1 -- 172.21.15.27:0/2971160611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7c155940 con 0x7f3d5c004160 2024-01-15T18:09:04.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.258+0000 7f3d79400640 1 --2- 172.21.15.27:0/2971160611 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3d7c1552c0 0x7f3d7c1569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:09:04.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.258+0000 7f3d79e00640 1 --2- 172.21.15.27:0/2971160611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d5c004160 0x7f3d7c143830 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c011a00 tx=0x7f3d6c011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:04.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.258+0000 7f3d72000640 1 -- 172.21.15.27:0/2971160611 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c00faf0 con 0x7f3d5c004160 2024-01-15T18:09:04.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.258+0000 7f3d72000640 1 -- 172.21.15.27:0/2971160611 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d6c00fc50 con 0x7f3d5c004160 2024-01-15T18:09:04.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.258+0000 7f3d80c00640 1 -- 172.21.15.27:0/2971160611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d7c155c20 con 0x7f3d5c004160 2024-01-15T18:09:04.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.258+0000 7f3d72000640 1 -- 172.21.15.27:0/2971160611 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c014440 con 0x7f3d5c004160 2024-01-15T18:09:04.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.258+0000 7f3d80c00640 1 -- 172.21.15.27:0/2971160611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d7c1561e0 con 0x7f3d5c004160 2024-01-15T18:09:04.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.262+0000 7f3d80c00640 1 -- 172.21.15.27:0/2971160611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d7c04e850 con 0x7f3d5c004160 2024-01-15T18:09:04.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.262+0000 7f3d72000640 1 -- 172.21.15.27:0/2971160611 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f3d6c0145a0 con 0x7f3d5c004160 2024-01-15T18:09:04.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.262+0000 7f3d72000640 1 --2- 172.21.15.27:0/2971160611 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3d54066ec0 0x7f3d54069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:04.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.262+0000 7f3d72000640 1 -- 172.21.15.27:0/2971160611 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f3d6c0852c0 con 0x7f3d5c004160 2024-01-15T18:09:04.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.262+0000 7f3d79400640 1 --2- 172.21.15.27:0/2971160611 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3d54066ec0 0x7f3d54069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:04.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.262+0000 7f3d79400640 1 --2- 172.21.15.27:0/2971160611 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3d54066ec0 0x7f3d54069380 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3d640018d0 tx=0x7f3d640aa070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:04.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.266+0000 7f3d72000640 1 -- 172.21.15.27:0/2971160611 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3d6c051fa0 con 0x7f3d5c004160 2024-01-15T18:09:04.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.422+0000 7f3d80c00640 1 -- 172.21.15.27:0/2971160611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f3d7c152dd0 con 0x7f3d5c004160 2024-01-15T18:09:04.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.430+0000 7f3d72000640 1 -- 172.21.15.27:0/2971160611 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3d6c0220c0 con 0x7f3d5c004160 2024-01-15T18:09:04.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.430+0000 7f3d80c00640 1 -- 172.21.15.27:0/2971160611 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3d54066ec0 msgr2=0x7f3d54069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:04.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.430+0000 7f3d80c00640 1 --2- 172.21.15.27:0/2971160611 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3d54066ec0 0x7f3d54069380 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3d640018d0 tx=0x7f3d640aa070 comp rx=0 tx=0).stop 2024-01-15T18:09:04.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.430+0000 7f3d80c00640 1 -- 172.21.15.27:0/2971160611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d5c004160 msgr2=0x7f3d7c143830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:04.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.430+0000 7f3d80c00640 1 --2- 172.21.15.27:0/2971160611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d5c004160 0x7f3d7c143830 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c011a00 tx=0x7f3d6c011ec0 comp rx=0 tx=0).stop 2024-01-15T18:09:04.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.430+0000 7f3d80c00640 1 -- 172.21.15.27:0/2971160611 shutdown_connections 2024-01-15T18:09:04.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.430+0000 7f3d80c00640 1 --2- 172.21.15.27:0/2971160611 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3d54066ec0 0x7f3d54069380 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:04.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.430+0000 7f3d80c00640 1 --2- 172.21.15.27:0/2971160611 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3d7c1eec00 0x7f3d7c1ef080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:04.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.430+0000 7f3d80c00640 1 --2- 172.21.15.27:0/2971160611 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3d7c1552c0 0x7f3d7c1569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:04.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.430+0000 7f3d80c00640 1 --2- 172.21.15.27:0/2971160611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d5c004160 0x7f3d7c143830 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:04.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.430+0000 7f3d80c00640 1 -- 172.21.15.27:0/2971160611 >> 172.21.15.27:0/2971160611 conn(0x7f3d7c0cb700 msgr2=0x7f3d7c1535d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:04.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.430+0000 7f3d80c00640 1 -- 172.21.15.27:0/2971160611 shutdown_connections 2024-01-15T18:09:04.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.430+0000 7f3d80c00640 1 -- 172.21.15.27:0/2971160611 wait complete. 2024-01-15T18:09:04.458 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T18:09:04.459 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:09:04.509 DEBUG:teuthology.orchestra.run.smithi093:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:09:04.609 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T18:09:04.609 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:09:04.609 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:09:04.618 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354580k,nr_inodes=4088645,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9249 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:09:04.618 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:09:04.618 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:09:04.627 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354484k,nr_inodes=4088621,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8403 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:09:04.627 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:09:04.627 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:09:04.670 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354580k,nr_inodes=4088645,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9249 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:09:04.670 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:09:04.671 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:09:04.680 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354484k,nr_inodes=4088621,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8403 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:09:04.680 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-15T18:09:04.680 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:09:04.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.846+0000 7f491fe00640 1 Processor -- start 2024-01-15T18:09:04.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.850+0000 7f491fe00640 1 -- start start 2024-01-15T18:09:04.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.850+0000 7f491fe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f491815e730 0x7f49180cd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:04.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.850+0000 7f491fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f49180cdd10 0x7f4918070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:04.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.850+0000 7f491fe00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49180ce920 con 0x7f491806db80 2024-01-15T18:09:04.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.850+0000 7f491fe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49180ce640 con 0x7f49180cdd10 2024-01-15T18:09:04.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.850+0000 7f491fe00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49180cc940 con 0x7f491815e730 2024-01-15T18:09:04.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.850+0000 7f491de00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f49180cdd10 0x7f4918070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:04.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.850+0000 7f491d400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f491806db80 0x7f491806df60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53978/0 (socket says 172.21.15.27:53978) 2024-01-15T18:09:04.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.850+0000 7f491d400640 1 -- 172.21.15.27:0/1358923485 learned_addr learned my addr 172.21.15.27:0/1358923485 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:04.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.850+0000 7f491ca00640 1 --2- 172.21.15.27:0/1358923485 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f491815e730 0x7f49180cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:04.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.850+0000 7f491de00640 1 -- 172.21.15.27:0/1358923485 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f491815e730 msgr2=0x7f49180cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:04.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.850+0000 7f491de00640 1 --2- 172.21.15.27:0/1358923485 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f491815e730 0x7f49180cd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:04.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.850+0000 7f491de00640 1 -- 172.21.15.27:0/1358923485 >> v1:172.21.15.27:6789/0 conn(0x7f491806db80 legacy=0x7f491806df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:04.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.850+0000 7f491de00640 1 -- 172.21.15.27:0/1358923485 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49180ccc20 con 0x7f49180cdd10 2024-01-15T18:09:04.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.854+0000 7f491de00640 1 --2- 172.21.15.27:0/1358923485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49180cdd10 0x7f4918070c20 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f4910009d90 tx=0x7f491001c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=870f96d084b09fda server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:04.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.854+0000 7f4917e00640 1 -- 172.21.15.27:0/1358923485 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49100a5900 con 0x7f49180cdd10 2024-01-15T18:09:04.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.854+0000 7f4917e00640 1 -- 172.21.15.27:0/1358923485 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49100a5a60 con 0x7f49180cdd10 2024-01-15T18:09:04.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.854+0000 7f4917e00640 1 -- 172.21.15.27:0/1358923485 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49100a5df0 con 0x7f49180cdd10 2024-01-15T18:09:04.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.854+0000 7f491fe00640 1 -- 172.21.15.27:0/1358923485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49180cdd10 msgr2=0x7f4918070c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:04.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.854+0000 7f491fe00640 1 --2- 172.21.15.27:0/1358923485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49180cdd10 0x7f4918070c20 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f4910009d90 tx=0x7f491001c8f0 comp rx=0 tx=0).stop 2024-01-15T18:09:04.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.854+0000 7f491fe00640 1 -- 172.21.15.27:0/1358923485 shutdown_connections 2024-01-15T18:09:04.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.854+0000 7f491fe00640 1 --2- 172.21.15.27:0/1358923485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49180cdd10 0x7f4918070c20 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:04.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.854+0000 7f491fe00640 1 --2- 172.21.15.27:0/1358923485 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f491815e730 0x7f49180cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:04.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.854+0000 7f491fe00640 1 -- 172.21.15.27:0/1358923485 >> 172.21.15.27:0/1358923485 conn(0x7f49180cec60 msgr2=0x7f49180cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:04.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.854+0000 7f491fe00640 1 -- 172.21.15.27:0/1358923485 shutdown_connections 2024-01-15T18:09:04.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.854+0000 7f491fe00640 1 -- 172.21.15.27:0/1358923485 wait complete. 2024-01-15T18:09:04.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.854+0000 7f491fe00640 1 Processor -- start 2024-01-15T18:09:04.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f491fe00640 1 -- start start 2024-01-15T18:09:04.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f491fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f491806db80 0x7f49180c84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:04.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f491fe00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49180cdd10 0x7f49180b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:04.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f491fe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f491815e730 0x7f491814e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:04.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f491fe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49180b1bc0 con 0x7f491806db80 2024-01-15T18:09:04.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f491fe00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49180c5d30 con 0x7f49180cdd10 2024-01-15T18:09:04.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f491fe00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49180b4090 con 0x7f491815e730 2024-01-15T18:09:04.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f491d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f491806db80 0x7f49180c84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:04.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f491d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f491806db80 0x7f49180c84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43644/0 (socket says 172.21.15.27:43644) 2024-01-15T18:09:04.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f491d400640 1 -- 172.21.15.27:0/4222938156 learned_addr learned my addr 172.21.15.27:0/4222938156 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:04.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f491de00640 1 --2- 172.21.15.27:0/4222938156 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f491815e730 0x7f491814e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:04.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f491ca00640 1 --2- 172.21.15.27:0/4222938156 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49180cdd10 0x7f49180b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:04.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f491d400640 1 -- 172.21.15.27:0/4222938156 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f491815e730 msgr2=0x7f491814e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:04.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f491d400640 1 --2- 172.21.15.27:0/4222938156 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f491815e730 0x7f491814e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:04.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f491d400640 1 -- 172.21.15.27:0/4222938156 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49180cdd10 msgr2=0x7f49180b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:04.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f491d400640 1 --2- 172.21.15.27:0/4222938156 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49180cdd10 0x7f49180b3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:04.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f491d400640 1 -- 172.21.15.27:0/4222938156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4910009a70 con 0x7f491806db80 2024-01-15T18:09:04.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f491de00640 1 --2- 172.21.15.27:0/4222938156 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f491815e730 0x7f491814e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:09:04.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f491ca00640 1 --2- 172.21.15.27:0/4222938156 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49180cdd10 0x7f49180b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:09:04.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f491d400640 1 --2- 172.21.15.27:0/4222938156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f491806db80 0x7f49180c84f0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f490800c500 tx=0x7f490800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:04.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f4915600640 1 -- 172.21.15.27:0/4222938156 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4908005cf0 con 0x7f491806db80 2024-01-15T18:09:04.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f4915600640 1 -- 172.21.15.27:0/4222938156 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4908005e50 con 0x7f491806db80 2024-01-15T18:09:04.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f491fe00640 1 -- 172.21.15.27:0/4222938156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49180b42f0 con 0x7f491806db80 2024-01-15T18:09:04.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f4915600640 1 -- 172.21.15.27:0/4222938156 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49080072f0 con 0x7f491806db80 2024-01-15T18:09:04.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f491fe00640 1 -- 172.21.15.27:0/4222938156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49180c5f80 con 0x7f491806db80 2024-01-15T18:09:04.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.858+0000 7f491fe00640 1 -- 172.21.15.27:0/4222938156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f491804e850 con 0x7f491806db80 2024-01-15T18:09:04.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.862+0000 7f4915600640 1 -- 172.21.15.27:0/4222938156 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f4908007450 con 0x7f491806db80 2024-01-15T18:09:04.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.862+0000 7f4915600640 1 --2- 172.21.15.27:0/4222938156 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f48f0066e90 0x7f48f0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:04.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.862+0000 7f4915600640 1 -- 172.21.15.27:0/4222938156 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f4908084050 con 0x7f491806db80 2024-01-15T18:09:04.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.862+0000 7f491ca00640 1 --2- 172.21.15.27:0/4222938156 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f48f0066e90 0x7f48f0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:04.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.862+0000 7f491ca00640 1 --2- 172.21.15.27:0/4222938156 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f48f0066e90 0x7f48f0069350 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f49180b4c90 tx=0x7f4900008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:04.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:04.866+0000 7f4915600640 1 -- 172.21.15.27:0/4222938156 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4908050dc0 con 0x7f491806db80 2024-01-15T18:09:05.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.058+0000 7f491fe00640 1 -- 172.21.15.27:0/4222938156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f491806c800 con 0x7f491806db80 2024-01-15T18:09:05.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.058+0000 7f4915600640 1 -- 172.21.15.27:0/4222938156 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+3696 (secure 0 0 0) 0x7f4908017090 con 0x7f491806db80 2024-01-15T18:09:05.067 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:09:05.067 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":30,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5306,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/1548117999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":1548117999},{"type":"v1","addr":"172.21.15.93:6836","nonce":1548117999}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5719,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/260141697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":260141697},{"type":"v1","addr":"172.21.15.93:6837","nonce":260141697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5802,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/2030664596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2030664596},{"type":"v1","addr":"172.21.15.27:6834","nonce":2030664596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":6351,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4247767797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4247767797},{"type":"v1","addr":"172.21.15.27:6835","nonce":4247767797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[]} 2024-01-15T18:09:05.068 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 30 2024-01-15T18:09:05.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.058+0000 7f491fe00640 1 -- 172.21.15.27:0/4222938156 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f48f0066e90 msgr2=0x7f48f0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:05.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.058+0000 7f491fe00640 1 --2- 172.21.15.27:0/4222938156 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f48f0066e90 0x7f48f0069350 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f49180b4c90 tx=0x7f4900008040 comp rx=0 tx=0).stop 2024-01-15T18:09:05.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.058+0000 7f491fe00640 1 -- 172.21.15.27:0/4222938156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f491806db80 msgr2=0x7f49180c84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:05.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.058+0000 7f491fe00640 1 --2- 172.21.15.27:0/4222938156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f491806db80 0x7f49180c84f0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f490800c500 tx=0x7f490800c9c0 comp rx=0 tx=0).stop 2024-01-15T18:09:05.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.058+0000 7f491fe00640 1 -- 172.21.15.27:0/4222938156 shutdown_connections 2024-01-15T18:09:05.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.058+0000 7f491fe00640 1 --2- 172.21.15.27:0/4222938156 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f48f0066e90 0x7f48f0069350 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:05.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.058+0000 7f491fe00640 1 --2- 172.21.15.27:0/4222938156 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f491815e730 0x7f491814e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:05.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.058+0000 7f491fe00640 1 --2- 172.21.15.27:0/4222938156 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49180cdd10 0x7f49180b3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:05.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.058+0000 7f491fe00640 1 --2- 172.21.15.27:0/4222938156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f491806db80 0x7f49180c84f0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:05.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.058+0000 7f491fe00640 1 -- 172.21.15.27:0/4222938156 >> 172.21.15.27:0/4222938156 conn(0x7f49180cec60 msgr2=0x7f49180af1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:05.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.062+0000 7f491fe00640 1 -- 172.21.15.27:0/4222938156 shutdown_connections 2024-01-15T18:09:05.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.062+0000 7f491fe00640 1 -- 172.21.15.27:0/4222938156 wait complete. 2024-01-15T18:09:05.086 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-15T18:09:05.086 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-15T18:09:05.086 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-15T18:09:05.086 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:09:05.087 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-15T18:09:05.087 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-15T18:09:05.087 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-15T18:09:05.087 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:09:05.087 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-15T18:09:05.087 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-15T18:09:05.087 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-15T18:09:05.088 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:09:05.088 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-15T18:09:05.088 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-15T18:09:05.088 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-15T18:09:05.088 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:09:05.101 INFO:tasks.ceph.mds.a:Stopped 2024-01-15T18:09:05.101 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-15T18:09:05.102 INFO:tasks.ceph.mds.c:Stopped 2024-01-15T18:09:05.102 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-15T18:09:05.103 INFO:tasks.ceph.mds.b:Stopped 2024-01-15T18:09:05.103 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-15T18:09:05.104 INFO:tasks.ceph.mds.d:Stopped 2024-01-15T18:09:05.104 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-15T18:09:05.145 INFO:tasks.ceph.mds.a:Started 2024-01-15T18:09:05.145 DEBUG:teuthology.parallel:result is None 2024-01-15T18:09:05.147 INFO:tasks.ceph.mds.c:Started 2024-01-15T18:09:05.147 DEBUG:teuthology.parallel:result is None 2024-01-15T18:09:05.148 INFO:tasks.ceph.mds.b:Started 2024-01-15T18:09:05.148 DEBUG:teuthology.parallel:result is None 2024-01-15T18:09:05.150 INFO:tasks.ceph.mds.d:Started 2024-01-15T18:09:05.150 DEBUG:teuthology.parallel:result is None 2024-01-15T18:09:05.151 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-15T18:09:05.233 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-15T18:09:05.233 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-15T18:09:05.239 INFO:tasks.ceph.mds.b.smithi093.stdout:starting mds.b at 2024-01-15T18:09:05.239 INFO:tasks.ceph.mds.d.smithi093.stdout:starting mds.d at 2024-01-15T18:09:05.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.294+0000 7fcc23400640 1 Processor -- start 2024-01-15T18:09:05.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.294+0000 7fcc23400640 1 -- start start 2024-01-15T18:09:05.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.294+0000 7fcc23400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc1c0cdef0 0x7fcc1c0cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:05.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.294+0000 7fcc23400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fcc1c0b19e0 0x7fcc1c0cb990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:05.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.294+0000 7fcc23400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1c0ce7d0 con 0x7fcc1c15b180 2024-01-15T18:09:05.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.294+0000 7fcc23400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc1c0d1e20 con 0x7fcc1c0cdef0 2024-01-15T18:09:05.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.294+0000 7fcc23400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc1c0ceed0 con 0x7fcc1c0b19e0 2024-01-15T18:09:05.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.294+0000 7fcc1be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc1c0cdef0 0x7fcc1c0cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:05.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.294+0000 7fcc1be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcc1c0cdef0 0x7fcc1c0cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43688/0 (socket says 172.21.15.27:43688) 2024-01-15T18:09:05.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.294+0000 7fcc1be00640 1 -- 172.21.15.27:0/1790345337 learned_addr learned my addr 172.21.15.27:0/1790345337 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:05.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.294+0000 7fcc21400640 1 --2- 172.21.15.27:0/1790345337 >> v2:172.21.15.93:3300/0 conn(0x7fcc1c0b19e0 0x7fcc1c0cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:05.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.294+0000 7fcc1be00640 1 -- 172.21.15.27:0/1790345337 >> v2:172.21.15.93:3300/0 conn(0x7fcc1c0b19e0 msgr2=0x7fcc1c0cb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:05.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.294+0000 7fcc1be00640 1 --2- 172.21.15.27:0/1790345337 >> v2:172.21.15.93:3300/0 conn(0x7fcc1c0b19e0 0x7fcc1c0cb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:05.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.294+0000 7fcc1be00640 1 -- 172.21.15.27:0/1790345337 >> v1:172.21.15.27:6789/0 conn(0x7fcc1c15b180 legacy=0x7fcc1c0cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:05.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.294+0000 7fcc1be00640 1 -- 172.21.15.27:0/1790345337 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1c0d1b40 con 0x7fcc1c0cdef0 2024-01-15T18:09:05.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.298+0000 7fcc1be00640 1 --2- 172.21.15.27:0/1790345337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc1c0cdef0 0x7fcc1c0cfa90 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c06e1e0 tx=0x7fcc0401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6695e43adb1f99e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:05.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.298+0000 7fcc1b400640 1 -- 172.21.15.27:0/1790345337 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc040aa430 con 0x7fcc1c0cdef0 2024-01-15T18:09:05.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.298+0000 7fcc1b400640 1 -- 172.21.15.27:0/1790345337 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc040aa590 con 0x7fcc1c0cdef0 2024-01-15T18:09:05.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.298+0000 7fcc1b400640 1 -- 172.21.15.27:0/1790345337 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc040aa920 con 0x7fcc1c0cdef0 2024-01-15T18:09:05.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.298+0000 7fcc23400640 1 -- 172.21.15.27:0/1790345337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc1c0cdef0 msgr2=0x7fcc1c0cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:05.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.298+0000 7fcc23400640 1 --2- 172.21.15.27:0/1790345337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc1c0cdef0 0x7fcc1c0cfa90 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c06e1e0 tx=0x7fcc0401c4d0 comp rx=0 tx=0).stop 2024-01-15T18:09:05.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.298+0000 7fcc23400640 1 -- 172.21.15.27:0/1790345337 shutdown_connections 2024-01-15T18:09:05.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.298+0000 7fcc23400640 1 --2- 172.21.15.27:0/1790345337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc1c0cdef0 0x7fcc1c0cfa90 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:05.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.298+0000 7fcc23400640 1 --2- 172.21.15.27:0/1790345337 >> v2:172.21.15.93:3300/0 conn(0x7fcc1c0b19e0 0x7fcc1c0cb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:05.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.298+0000 7fcc23400640 1 -- 172.21.15.27:0/1790345337 >> 172.21.15.27:0/1790345337 conn(0x7fcc1c0d0e80 msgr2=0x7fcc1c0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:05.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.298+0000 7fcc23400640 1 -- 172.21.15.27:0/1790345337 shutdown_connections 2024-01-15T18:09:05.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.298+0000 7fcc23400640 1 -- 172.21.15.27:0/1790345337 wait complete. 2024-01-15T18:09:05.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.302+0000 7fcc23400640 1 Processor -- start 2024-01-15T18:09:05.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.302+0000 7fcc23400640 1 -- start start 2024-01-15T18:09:05.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.302+0000 7fcc23400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcc1c0b19e0 0x7fcc1c1528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:05.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.302+0000 7fcc23400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc1c0cdef0 0x7fcc1c0c6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:05.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.302+0000 7fcc23400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcc1c15b180 0x7fcc1c0c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:05.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.302+0000 7fcc23400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c158a10 con 0x7fcc1c0cdef0 2024-01-15T18:09:05.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.302+0000 7fcc23400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c1580e0 con 0x7fcc1c0b19e0 2024-01-15T18:09:05.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.302+0000 7fcc23400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c1583c0 con 0x7fcc1c15b180 2024-01-15T18:09:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.302+0000 7fcc1be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc1c0cdef0 0x7fcc1c0c6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.302+0000 7fcc1be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc1c0cdef0 0x7fcc1c0c6660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43698/0 (socket says 172.21.15.27:43698) 2024-01-15T18:09:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.302+0000 7fcc1be00640 1 -- 172.21.15.27:0/1307704594 learned_addr learned my addr 172.21.15.27:0/1307704594 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.302+0000 7fcc20a00640 1 --2- 172.21.15.27:0/1307704594 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcc1c0b19e0 0x7fcc1c1528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.302+0000 7fcc21400640 1 --2- 172.21.15.27:0/1307704594 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcc1c15b180 0x7fcc1c0c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.302+0000 7fcc1be00640 1 -- 172.21.15.27:0/1307704594 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcc1c15b180 msgr2=0x7fcc1c0c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.302+0000 7fcc1be00640 1 --2- 172.21.15.27:0/1307704594 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcc1c15b180 0x7fcc1c0c6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.302+0000 7fcc1be00640 1 -- 172.21.15.27:0/1307704594 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcc1c0b19e0 msgr2=0x7fcc1c1528f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.302+0000 7fcc1be00640 1 --2- 172.21.15.27:0/1307704594 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcc1c0b19e0 0x7fcc1c1528f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.302+0000 7fcc1be00640 1 -- 172.21.15.27:0/1307704594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc040094d0 con 0x7fcc1c0cdef0 2024-01-15T18:09:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.302+0000 7fcc1be00640 1 --2- 172.21.15.27:0/1307704594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc1c0cdef0 0x7fcc1c0c6660 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fcc040190e0 tx=0x7fcc0401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.306+0000 7fcc18c00640 1 -- 172.21.15.27:0/1307704594 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc040aa430 con 0x7fcc1c0cdef0 2024-01-15T18:09:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.306+0000 7fcc18c00640 1 -- 172.21.15.27:0/1307704594 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc040b3820 con 0x7fcc1c0cdef0 2024-01-15T18:09:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.306+0000 7fcc18c00640 1 -- 172.21.15.27:0/1307704594 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc040b3bd0 con 0x7fcc1c0cdef0 2024-01-15T18:09:05.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.306+0000 7fcc23400640 1 -- 172.21.15.27:0/1307704594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc1c1586a0 con 0x7fcc1c0cdef0 2024-01-15T18:09:05.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.306+0000 7fcc23400640 1 -- 172.21.15.27:0/1307704594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc1c153000 con 0x7fcc1c0cdef0 2024-01-15T18:09:05.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.306+0000 7fcc23400640 1 -- 172.21.15.27:0/1307704594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc1c158b80 con 0x7fcc1c0cdef0 2024-01-15T18:09:05.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.306+0000 7fcc18c00640 1 -- 172.21.15.27:0/1307704594 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7fcc040b3d30 con 0x7fcc1c0cdef0 2024-01-15T18:09:05.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.306+0000 7fcc18c00640 1 --2- 172.21.15.27:0/1307704594 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcbf4066ec0 0x7fcbf4069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.306+0000 7fcc18c00640 1 -- 172.21.15.27:0/1307704594 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fcc0411ece0 con 0x7fcc1c0cdef0 2024-01-15T18:09:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.306+0000 7fcc20a00640 1 --2- 172.21.15.27:0/1307704594 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcbf4066ec0 0x7fcbf4069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.310+0000 7fcc20a00640 1 --2- 172.21.15.27:0/1307704594 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcbf4066ec0 0x7fcbf4069380 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c007fd0 tx=0x7fcc0c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:05.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.310+0000 7fcc18c00640 1 -- 172.21.15.27:0/1307704594 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fcc040eb8f0 con 0x7fcc1c0cdef0 2024-01-15T18:09:05.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:05.478+0000 7fcc23400640 1 -- 172.21.15.27:0/1307704594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fcc1c04e850 con 0x7fcc1c0cdef0 2024-01-15T18:09:06.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.090+0000 7fcc18c00640 1 -- 172.21.15.27:0/1307704594 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v52) v1 ==== 97+0+0 (secure 0 0 0) 0x7fcc040c3020 con 0x7fcc1c0cdef0 2024-01-15T18:09:06.098 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-01-15T18:09:06.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.094+0000 7fcc23400640 1 -- 172.21.15.27:0/1307704594 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcbf4066ec0 msgr2=0x7fcbf4069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:06.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.094+0000 7fcc23400640 1 --2- 172.21.15.27:0/1307704594 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcbf4066ec0 0x7fcbf4069380 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c007fd0 tx=0x7fcc0c009040 comp rx=0 tx=0).stop 2024-01-15T18:09:06.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.094+0000 7fcc23400640 1 -- 172.21.15.27:0/1307704594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc1c0cdef0 msgr2=0x7fcc1c0c6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:06.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.094+0000 7fcc23400640 1 --2- 172.21.15.27:0/1307704594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc1c0cdef0 0x7fcc1c0c6660 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fcc040190e0 tx=0x7fcc0401ad10 comp rx=0 tx=0).stop 2024-01-15T18:09:06.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.094+0000 7fcc23400640 1 -- 172.21.15.27:0/1307704594 shutdown_connections 2024-01-15T18:09:06.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.094+0000 7fcc23400640 1 --2- 172.21.15.27:0/1307704594 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcbf4066ec0 0x7fcbf4069380 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.094+0000 7fcc23400640 1 --2- 172.21.15.27:0/1307704594 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcc1c15b180 0x7fcc1c0c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.094+0000 7fcc23400640 1 --2- 172.21.15.27:0/1307704594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcc1c0cdef0 0x7fcc1c0c6660 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.094+0000 7fcc23400640 1 --2- 172.21.15.27:0/1307704594 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcc1c0b19e0 0x7fcc1c1528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.094+0000 7fcc23400640 1 -- 172.21.15.27:0/1307704594 >> 172.21.15.27:0/1307704594 conn(0x7fcc1c0d0e80 msgr2=0x7fcc1c154090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:06.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.094+0000 7fcc23400640 1 -- 172.21.15.27:0/1307704594 shutdown_connections 2024-01-15T18:09:06.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.094+0000 7fcc23400640 1 -- 172.21.15.27:0/1307704594 wait complete. 2024-01-15T18:09:06.125 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-15T18:09:06.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.262+0000 7f3911a00640 1 Processor -- start 2024-01-15T18:09:06.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.262+0000 7f3911a00640 1 -- start start 2024-01-15T18:09:06.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.266+0000 7f3911a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f390c0701a0 0x7f390c070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:06.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.266+0000 7f3911a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f390c0c92b0 0x7f390c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:06.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.266+0000 7f3911a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f390c06e240 0x7f390c15d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:06.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.266+0000 7f3911a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f390c06c4a0 con 0x7f390c06e240 2024-01-15T18:09:06.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.266+0000 7f3911a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f390c070ac0 con 0x7f390c0701a0 2024-01-15T18:09:06.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.266+0000 7f3911a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f390c15c660 con 0x7f390c0c92b0 2024-01-15T18:09:06.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.266+0000 7f390b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f390c06e240 0x7f390c15d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:06.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.266+0000 7f390b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f390c06e240 0x7f390c15d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43708/0 (socket says 172.21.15.27:43708) 2024-01-15T18:09:06.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.266+0000 7f390b400640 1 -- 172.21.15.27:0/1494728097 learned_addr learned my addr 172.21.15.27:0/1494728097 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:06.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.266+0000 7f390a000640 1 --2- 172.21.15.27:0/1494728097 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f390c0c92b0 0x7f390c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:06.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.266+0000 7f390aa00640 1 --2- 172.21.15.27:0/1494728097 >> v2:172.21.15.93:3300/0 conn(0x7f390c0701a0 0x7f390c070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:06.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.266+0000 7f390b400640 1 -- 172.21.15.27:0/1494728097 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f390c0c92b0 msgr2=0x7f390c06dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:06.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.266+0000 7f390b400640 1 --2- 172.21.15.27:0/1494728097 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f390c0c92b0 0x7f390c06dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.266+0000 7f390b400640 1 -- 172.21.15.27:0/1494728097 >> v2:172.21.15.93:3300/0 conn(0x7f390c0701a0 msgr2=0x7f390c070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:06.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.266+0000 7f390b400640 1 --2- 172.21.15.27:0/1494728097 >> v2:172.21.15.93:3300/0 conn(0x7f390c0701a0 0x7f390c070580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.266+0000 7f390b400640 1 -- 172.21.15.27:0/1494728097 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f390c15c940 con 0x7f390c06e240 2024-01-15T18:09:06.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.266+0000 7f390aa00640 1 --2- 172.21.15.27:0/1494728097 >> v2:172.21.15.93:3300/0 conn(0x7f390c0701a0 0x7f390c070580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:06.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.270+0000 7f390b400640 1 --2- 172.21.15.27:0/1494728097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f390c06e240 0x7f390c15d9e0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f390c0681f0 tx=0x7f390401c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f4a25221c6fd5a2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.270+0000 7f3909600640 1 -- 172.21.15.27:0/1494728097 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39040a5820 con 0x7f390c06e240 2024-01-15T18:09:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.270+0000 7f3909600640 1 -- 172.21.15.27:0/1494728097 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39040a5980 con 0x7f390c06e240 2024-01-15T18:09:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.270+0000 7f3909600640 1 -- 172.21.15.27:0/1494728097 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39040a5d10 con 0x7f390c06e240 2024-01-15T18:09:06.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.270+0000 7f3911a00640 1 -- 172.21.15.27:0/1494728097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f390c06e240 msgr2=0x7f390c15d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:06.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.270+0000 7f3911a00640 1 --2- 172.21.15.27:0/1494728097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f390c06e240 0x7f390c15d9e0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f390c0681f0 tx=0x7f390401c9a0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.270+0000 7f3911a00640 1 -- 172.21.15.27:0/1494728097 shutdown_connections 2024-01-15T18:09:06.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.270+0000 7f3911a00640 1 --2- 172.21.15.27:0/1494728097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f390c06e240 0x7f390c15d9e0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.270+0000 7f3911a00640 1 --2- 172.21.15.27:0/1494728097 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f390c0c92b0 0x7f390c06dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.270+0000 7f3911a00640 1 --2- 172.21.15.27:0/1494728097 >> v2:172.21.15.93:3300/0 conn(0x7f390c0701a0 0x7f390c070580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.270+0000 7f3911a00640 1 -- 172.21.15.27:0/1494728097 >> 172.21.15.27:0/1494728097 conn(0x7f390c0cec60 msgr2=0x7f390c0cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:06.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.270+0000 7f3911a00640 1 -- 172.21.15.27:0/1494728097 shutdown_connections 2024-01-15T18:09:06.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.270+0000 7f3911a00640 1 -- 172.21.15.27:0/1494728097 wait complete. 2024-01-15T18:09:06.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.270+0000 7f3911a00640 1 Processor -- start 2024-01-15T18:09:06.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.270+0000 7f3911a00640 1 -- start start 2024-01-15T18:09:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.274+0000 7f3911a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f390c06e240 0x7f390c1576a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.274+0000 7f3911a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f390c0701a0 0x7f390c155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.274+0000 7f3911a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f390c0c92b0 0x7f390c150670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.274+0000 7f3911a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c154d50 con 0x7f390c0c92b0 2024-01-15T18:09:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.274+0000 7f3911a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c155030 con 0x7f390c0701a0 2024-01-15T18:09:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.274+0000 7f3911a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f390c155310 con 0x7f390c06e240 2024-01-15T18:09:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.274+0000 7f390b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f390c0c92b0 0x7f390c150670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.274+0000 7f390b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f390c0c92b0 0x7f390c150670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43722/0 (socket says 172.21.15.27:43722) 2024-01-15T18:09:06.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.274+0000 7f390b400640 1 -- 172.21.15.27:0/1084016873 learned_addr learned my addr 172.21.15.27:0/1084016873 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.274+0000 7f390aa00640 1 --2- 172.21.15.27:0/1084016873 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f390c06e240 0x7f390c1576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.274+0000 7f390a000640 1 --2- 172.21.15.27:0/1084016873 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f390c0701a0 0x7f390c155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.274+0000 7f390b400640 1 -- 172.21.15.27:0/1084016873 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f390c06e240 msgr2=0x7f390c1576a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.274+0000 7f390b400640 1 --2- 172.21.15.27:0/1084016873 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f390c06e240 0x7f390c1576a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.274+0000 7f390b400640 1 -- 172.21.15.27:0/1084016873 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f390c0701a0 msgr2=0x7f390c155df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.274+0000 7f390b400640 1 --2- 172.21.15.27:0/1084016873 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f390c0701a0 0x7f390c155df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.274+0000 7f390b400640 1 -- 172.21.15.27:0/1084016873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3904009b30 con 0x7f390c0c92b0 2024-01-15T18:09:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.274+0000 7f390b400640 1 --2- 172.21.15.27:0/1084016873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f390c0c92b0 0x7f390c150670 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f39040af040 tx=0x7f3904019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.274+0000 7f3902a00640 1 -- 172.21.15.27:0/1084016873 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39040b8670 con 0x7f390c0c92b0 2024-01-15T18:09:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.274+0000 7f3902a00640 1 -- 172.21.15.27:0/1084016873 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39040b87d0 con 0x7f390c0c92b0 2024-01-15T18:09:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.274+0000 7f3902a00640 1 -- 172.21.15.27:0/1084016873 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39040b8b80 con 0x7f390c0c92b0 2024-01-15T18:09:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.274+0000 7f3911a00640 1 -- 172.21.15.27:0/1084016873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390c1555f0 con 0x7f390c0c92b0 2024-01-15T18:09:06.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.274+0000 7f3911a00640 1 -- 172.21.15.27:0/1084016873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390c156330 con 0x7f390c0c92b0 2024-01-15T18:09:06.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.274+0000 7f3911a00640 1 -- 172.21.15.27:0/1084016873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f390c04e850 con 0x7f390c0c92b0 2024-01-15T18:09:06.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.274+0000 7f3902a00640 1 -- 172.21.15.27:0/1084016873 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f39040b8ce0 con 0x7f390c0c92b0 2024-01-15T18:09:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.278+0000 7f3902a00640 1 --2- 172.21.15.27:0/1084016873 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f38e4066e90 0x7f38e4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.278+0000 7f3902a00640 1 -- 172.21.15.27:0/1084016873 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f39041200c0 con 0x7f390c0c92b0 2024-01-15T18:09:06.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.278+0000 7f390aa00640 1 --2- 172.21.15.27:0/1084016873 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f38e4066e90 0x7f38e4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:06.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.278+0000 7f390aa00640 1 --2- 172.21.15.27:0/1084016873 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f38e4066e90 0x7f38e4069350 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00ab60 tx=0x7f38fc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:06.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.282+0000 7f3902a00640 1 -- 172.21.15.27:0/1084016873 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f39040ecfe0 con 0x7f390c0c92b0 2024-01-15T18:09:06.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.490+0000 7f3911a00640 1 -- 172.21.15.27:0/1084016873 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f390c155450 con 0x7f390c0c92b0 2024-01-15T18:09:06.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.494+0000 7f3902a00640 1 -- 172.21.15.27:0/1084016873 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v10) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f390c155450 con 0x7f390c0c92b0 2024-01-15T18:09:06.498 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:09:06.498 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:09:06.499 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-01-15T18:09:06.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:09:06.499 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-01-15T18:09:06.499 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlbMZIGBAAhSU4y2DGJd3MH0v6v3LsDg==", 2024-01-15T18:09:06.499 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:09:06.499 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:09:06.499 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:09:06.499 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:09:06.499 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:09:06.499 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:09:06.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:09:06.499 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:09:06.499 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-01-15T18:09:06.500 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlC7B9JBAAsXmBjV4KwPZejOmowXO2Zg==", 2024-01-15T18:09:06.500 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:09:06.500 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:09:06.500 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:09:06.500 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:09:06.500 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:09:06.500 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:09:06.500 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:09:06.500 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:09:06.500 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-01-15T18:09:06.500 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6Vla6a3HxAA/AHc6FSTccj0JGwiR4Vl7g==", 2024-01-15T18:09:06.500 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:09:06.500 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:09:06.500 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:09:06.500 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:09:06.501 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:09:06.501 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:09:06.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:09:06.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:09:06.501 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-01-15T18:09:06.501 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6Vl9nqzKxAAx2aDA8UwidAUrU9DBxGldA==", 2024-01-15T18:09:06.501 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:09:06.501 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:09:06.501 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:09:06.501 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:09:06.501 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:09:06.501 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:09:06.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:09:06.501 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:09:06.501 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-01-15T18:09:06.502 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCSc6Vly4TACRAAYBmMIcsZsRp4CZ5a3xCWgA==", 2024-01-15T18:09:06.502 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:09:06.502 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:09:06.502 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:09:06.502 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:09:06.502 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:09:06.502 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:09:06.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:09:06.502 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-01-15T18:09:06.502 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCVc6VlW/HEERAAOBgPz+eGpHwNXDe8fkYo7w==", 2024-01-15T18:09:06.502 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:09:06.502 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:09:06.502 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:09:06.502 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:09:06.502 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:09:06.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:09:06.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:09:06.503 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-01-15T18:09:06.503 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCYc6Vl+Ec4GBAAWRe/6u7u4RXApd57H8ihng==", 2024-01-15T18:09:06.503 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:09:06.503 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:09:06.503 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:09:06.503 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:09:06.503 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:09:06.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:09:06.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:09:06.503 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-01-15T18:09:06.503 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCbc6VlNdMPHxAAHzJvzLltsTl1wrEnI/iyrQ==", 2024-01-15T18:09:06.503 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:09:06.503 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:09:06.504 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:09:06.504 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:09:06.504 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:09:06.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:09:06.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:09:06.504 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-01-15T18:09:06.504 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQChc6Vlg8eVKBAAwZWAncCE2Lra+CxmSIxZkA==", 2024-01-15T18:09:06.504 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:09:06.504 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:09:06.504 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:09:06.504 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:09:06.504 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:09:06.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:09:06.504 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:09:06.505 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-01-15T18:09:06.505 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCkc6VlIXDOLxAAFISn6ILqq/aNFK07RxNnaw==", 2024-01-15T18:09:06.505 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:09:06.505 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:09:06.505 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:09:06.505 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:09:06.505 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:09:06.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:09:06.505 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:09:06.505 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-01-15T18:09:06.505 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCnc6Vl9Ru2NRAAl996VBDudw23SGiFPusIcw==", 2024-01-15T18:09:06.505 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:09:06.505 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:09:06.505 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:09:06.505 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:09:06.505 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:09:06.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:09:06.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:09:06.506 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-01-15T18:09:06.506 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCrc6VlHoAsARAAIoPMridLsIcpU3I9Hr5XLQ==", 2024-01-15T18:09:06.506 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:09:06.506 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:09:06.506 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:09:06.506 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:09:06.506 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:09:06.506 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:09:06.506 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:09:06.506 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-01-15T18:09:06.506 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlH+DCLxAA1sKVut+3WjA/B+aYJ6/Jeg==", 2024-01-15T18:09:06.506 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:09:06.507 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:09:06.507 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-01-15T18:09:06.507 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-15T18:09:06.507 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:09:06.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:09:06.507 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:09:06.507 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-01-15T18:09:06.507 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlaRBaORAAijn3UJLHbWm8ol2NiKNSaw==", 2024-01-15T18:09:06.507 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:09:06.507 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:09:06.507 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-15T18:09:06.507 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-15T18:09:06.507 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-15T18:09:06.507 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:09:06.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:09:06.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:09:06.508 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-01-15T18:09:06.508 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlDbLQNRAA82Lwz+pKY+3nHPAr1DichA==", 2024-01-15T18:09:06.508 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:09:06.508 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:09:06.508 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-15T18:09:06.508 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-15T18:09:06.508 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-15T18:09:06.508 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:09:06.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:09:06.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:09:06.508 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-01-15T18:09:06.509 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCPc6VlOyDaAxAADea0K1OEDTfrzsM7VEkixg==", 2024-01-15T18:09:06.509 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:09:06.509 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:09:06.509 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-15T18:09:06.509 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-15T18:09:06.509 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-15T18:09:06.509 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:09:06.509 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:09:06.509 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:09:06.509 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-01-15T18:09:06.509 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNc6VlK70BNxAAgYXkW/pMlzy8GZoZhX9NMA==", 2024-01-15T18:09:06.509 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:09:06.509 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-15T18:09:06.509 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:09:06.509 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:09:06.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:09:06.510 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:09:06.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:09:06.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:09:06.510 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-01-15T18:09:06.510 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlQbinExAAWtUdgkUNR3cCKV+ytcbUuQ==", 2024-01-15T18:09:06.510 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:09:06.510 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-15T18:09:06.510 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-15T18:09:06.510 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:09:06.510 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:09:06.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:09:06.510 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:09:06.510 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-01-15T18:09:06.510 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlzafrDxAAVVEBuX8BDT3cB9NfD2xTdQ==", 2024-01-15T18:09:06.511 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:09:06.511 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-15T18:09:06.511 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-15T18:09:06.511 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:09:06.511 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:09:06.511 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:09:06.511 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-01-15T18:09:06.511 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:09:06.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.494+0000 7f3911a00640 1 -- 172.21.15.27:0/1084016873 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f38e4066e90 msgr2=0x7f38e4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:06.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.494+0000 7f3911a00640 1 --2- 172.21.15.27:0/1084016873 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f38e4066e90 0x7f38e4069350 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00ab60 tx=0x7f38fc008040 comp rx=0 tx=0).stop 2024-01-15T18:09:06.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.494+0000 7f3911a00640 1 -- 172.21.15.27:0/1084016873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f390c0c92b0 msgr2=0x7f390c150670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:06.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.494+0000 7f3911a00640 1 --2- 172.21.15.27:0/1084016873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f390c0c92b0 0x7f390c150670 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f39040af040 tx=0x7f3904019740 comp rx=0 tx=0).stop 2024-01-15T18:09:06.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.494+0000 7f3911a00640 1 -- 172.21.15.27:0/1084016873 shutdown_connections 2024-01-15T18:09:06.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.494+0000 7f3911a00640 1 --2- 172.21.15.27:0/1084016873 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f38e4066e90 0x7f38e4069350 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.494+0000 7f3911a00640 1 --2- 172.21.15.27:0/1084016873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f390c0c92b0 0x7f390c150670 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.494+0000 7f3911a00640 1 --2- 172.21.15.27:0/1084016873 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f390c0701a0 0x7f390c155df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.494+0000 7f3911a00640 1 --2- 172.21.15.27:0/1084016873 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f390c06e240 0x7f390c1576a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.494+0000 7f3911a00640 1 -- 172.21.15.27:0/1084016873 >> 172.21.15.27:0/1084016873 conn(0x7f390c0cec60 msgr2=0x7f390c0c7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:06.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.494+0000 7f3911a00640 1 -- 172.21.15.27:0/1084016873 shutdown_connections 2024-01-15T18:09:06.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.494+0000 7f3911a00640 1 -- 172.21.15.27:0/1084016873 wait complete. 2024-01-15T18:09:06.520 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:09:06.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.654+0000 7ff9a5800640 1 Processor -- start 2024-01-15T18:09:06.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.654+0000 7ff9a5800640 1 -- start start 2024-01-15T18:09:06.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.654+0000 7ff9a5800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff9a006bb80 0x7ff9a00cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:06.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.654+0000 7ff9a5800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff9a00cddc0 0x7ff9a00b0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:06.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.654+0000 7ff9a5800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a00b2770 con 0x7ff9a00b2390 2024-01-15T18:09:06.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.654+0000 7ff9a5800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9a00ce270 con 0x7ff9a00cddc0 2024-01-15T18:09:06.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.654+0000 7ff9a5800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9a006c060 con 0x7ff9a006bb80 2024-01-15T18:09:06.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.658+0000 7ff99f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff9a00cddc0 0x7ff9a00b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:06.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.658+0000 7ff99f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff9a00cddc0 0x7ff9a00b0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43730/0 (socket says 172.21.15.27:43730) 2024-01-15T18:09:06.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.658+0000 7ff99f400640 1 -- 172.21.15.27:0/283865336 learned_addr learned my addr 172.21.15.27:0/283865336 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:06.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.658+0000 7ff99ea00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff9a00b2390 0x7ff9a00cf780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54000/0 (socket says 172.21.15.27:54000) 2024-01-15T18:09:06.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.658+0000 7ff99e000640 1 --2- 172.21.15.27:0/283865336 >> v2:172.21.15.93:3300/0 conn(0x7ff9a006bb80 0x7ff9a00cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:06.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.658+0000 7ff99f400640 1 -- 172.21.15.27:0/283865336 >> v2:172.21.15.93:3300/0 conn(0x7ff9a006bb80 msgr2=0x7ff9a00cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:06.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.658+0000 7ff99f400640 1 --2- 172.21.15.27:0/283865336 >> v2:172.21.15.93:3300/0 conn(0x7ff9a006bb80 0x7ff9a00cd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.658+0000 7ff99f400640 1 -- 172.21.15.27:0/283865336 >> v1:172.21.15.27:6789/0 conn(0x7ff9a00b2390 legacy=0x7ff9a00cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:06.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.658+0000 7ff99f400640 1 -- 172.21.15.27:0/283865336 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a006c340 con 0x7ff9a00cddc0 2024-01-15T18:09:06.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.658+0000 7ff99d600640 1 -- 172.21.15.27:0/283865336 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786266189 0 0) 0x7ff9a00b2770 con 0x7ff9a00b2390 2024-01-15T18:09:06.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.658+0000 7ff99f400640 1 --2- 172.21.15.27:0/283865336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00cddc0 0x7ff9a00b0880 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0070ea0 tx=0x7ff99801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b3d43f4d9aef6a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.658+0000 7ff99d600640 1 -- 172.21.15.27:0/283865336 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9980a5860 con 0x7ff9a00cddc0 2024-01-15T18:09:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.658+0000 7ff99d600640 1 -- 172.21.15.27:0/283865336 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9980a59c0 con 0x7ff9a00cddc0 2024-01-15T18:09:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.658+0000 7ff99d600640 1 -- 172.21.15.27:0/283865336 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9980a5d50 con 0x7ff9a00cddc0 2024-01-15T18:09:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.662+0000 7ff9a5800640 1 -- 172.21.15.27:0/283865336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00cddc0 msgr2=0x7ff9a00b0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.662+0000 7ff9a5800640 1 --2- 172.21.15.27:0/283865336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00cddc0 0x7ff9a00b0880 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0070ea0 tx=0x7ff99801c9a0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.662+0000 7ff9a5800640 1 -- 172.21.15.27:0/283865336 shutdown_connections 2024-01-15T18:09:06.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.662+0000 7ff9a5800640 1 --2- 172.21.15.27:0/283865336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00cddc0 0x7ff9a00b0880 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.662+0000 7ff9a5800640 1 --2- 172.21.15.27:0/283865336 >> v2:172.21.15.93:3300/0 conn(0x7ff9a006bb80 0x7ff9a00cd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.662+0000 7ff9a5800640 1 -- 172.21.15.27:0/283865336 >> 172.21.15.27:0/283865336 conn(0x7ff9a00b2a10 msgr2=0x7ff9a00b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:06.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.662+0000 7ff9a5800640 1 -- 172.21.15.27:0/283865336 shutdown_connections 2024-01-15T18:09:06.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.662+0000 7ff9a5800640 1 -- 172.21.15.27:0/283865336 wait complete. 2024-01-15T18:09:06.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.662+0000 7ff9a5800640 1 Processor -- start 2024-01-15T18:09:06.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.662+0000 7ff9a5800640 1 -- start start 2024-01-15T18:09:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.662+0000 7ff9a5800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9a006bb80 0x7ff9a00c5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.662+0000 7ff9a5800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff9a00b2390 0x7ff9a00c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.662+0000 7ff9a5800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00cddc0 0x7ff9a0153370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.662+0000 7ff9a5800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a00c5840 con 0x7ff9a00cddc0 2024-01-15T18:09:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.662+0000 7ff9a5800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a00c61e0 con 0x7ff9a00b2390 2024-01-15T18:09:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.662+0000 7ff9a5800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a0153cb0 con 0x7ff9a006bb80 2024-01-15T18:09:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.662+0000 7ff99f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00cddc0 0x7ff9a0153370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.662+0000 7ff99f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00cddc0 0x7ff9a0153370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43744/0 (socket says 172.21.15.27:43744) 2024-01-15T18:09:06.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.662+0000 7ff99f400640 1 -- 172.21.15.27:0/289157462 learned_addr learned my addr 172.21.15.27:0/289157462 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:06.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.662+0000 7ff99e000640 1 --2- 172.21.15.27:0/289157462 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff9a00b2390 0x7ff9a00c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:06.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.662+0000 7ff99ea00640 1 --2- 172.21.15.27:0/289157462 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9a006bb80 0x7ff9a00c5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:06.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.666+0000 7ff99f400640 1 -- 172.21.15.27:0/289157462 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9a006bb80 msgr2=0x7ff9a00c5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:06.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.666+0000 7ff99f400640 1 --2- 172.21.15.27:0/289157462 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9a006bb80 0x7ff9a00c5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.666+0000 7ff99f400640 1 -- 172.21.15.27:0/289157462 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff9a00b2390 msgr2=0x7ff9a00c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:06.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.666+0000 7ff99f400640 1 --2- 172.21.15.27:0/289157462 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff9a00b2390 0x7ff9a00c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.666+0000 7ff99f400640 1 -- 172.21.15.27:0/289157462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff998009b30 con 0x7ff9a00cddc0 2024-01-15T18:09:06.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.666+0000 7ff99f400640 1 --2- 172.21.15.27:0/289157462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00cddc0 0x7ff9a0153370 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7ff9980195b0 tx=0x7ff998019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:06.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.666+0000 7ff996a00640 1 -- 172.21.15.27:0/289157462 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9980b8670 con 0x7ff9a00cddc0 2024-01-15T18:09:06.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.666+0000 7ff996a00640 1 -- 172.21.15.27:0/289157462 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9980b8830 con 0x7ff9a00cddc0 2024-01-15T18:09:06.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.666+0000 7ff9a5800640 1 -- 172.21.15.27:0/289157462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a01538b0 con 0x7ff9a00cddc0 2024-01-15T18:09:06.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.666+0000 7ff996a00640 1 -- 172.21.15.27:0/289157462 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9980b8be0 con 0x7ff9a00cddc0 2024-01-15T18:09:06.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.666+0000 7ff9a5800640 1 -- 172.21.15.27:0/289157462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a00b6540 con 0x7ff9a00cddc0 2024-01-15T18:09:06.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.666+0000 7ff9a5800640 1 -- 172.21.15.27:0/289157462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9a00d07c0 con 0x7ff9a00cddc0 2024-01-15T18:09:06.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.666+0000 7ff996a00640 1 -- 172.21.15.27:0/289157462 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7ff9980b8d40 con 0x7ff9a00cddc0 2024-01-15T18:09:06.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.666+0000 7ff996a00640 1 --2- 172.21.15.27:0/289157462 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff978066e90 0x7ff978069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:06.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.666+0000 7ff996a00640 1 -- 172.21.15.27:0/289157462 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4885+0+0 (secure 0 0 0) 0x7ff9981201b0 con 0x7ff9a00cddc0 2024-01-15T18:09:06.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.670+0000 7ff99ea00640 1 --2- 172.21.15.27:0/289157462 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff978066e90 0x7ff978069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:06.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.670+0000 7ff99ea00640 1 --2- 172.21.15.27:0/289157462 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff978066e90 0x7ff978069350 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff990000f70 tx=0x7ff990007b80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:06.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.670+0000 7ff996a00640 1 -- 172.21.15.27:0/289157462 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff9980ecf90 con 0x7ff9a00cddc0 2024-01-15T18:09:06.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.830+0000 7ff9a5800640 1 -- 172.21.15.27:0/289157462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9a0153a80 con 0x7ff9a00cddc0 2024-01-15T18:09:06.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.830+0000 7ff996a00640 1 -- 172.21.15.27:0/289157462 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+12804 (secure 0 0 0) 0x7ff9a0153a80 con 0x7ff9a00cddc0 2024-01-15T18:09:06.838 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:09:06.838 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":54,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:09:06.103981+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:09:06.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.834+0000 7ff9a5800640 1 -- 172.21.15.27:0/289157462 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff978066e90 msgr2=0x7ff978069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:06.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.834+0000 7ff9a5800640 1 --2- 172.21.15.27:0/289157462 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff978066e90 0x7ff978069350 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff990000f70 tx=0x7ff990007b80 comp rx=0 tx=0).stop 2024-01-15T18:09:06.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.834+0000 7ff9a5800640 1 -- 172.21.15.27:0/289157462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00cddc0 msgr2=0x7ff9a0153370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:06.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.834+0000 7ff9a5800640 1 --2- 172.21.15.27:0/289157462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00cddc0 0x7ff9a0153370 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7ff9980195b0 tx=0x7ff998019740 comp rx=0 tx=0).stop 2024-01-15T18:09:06.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.834+0000 7ff9a5800640 1 -- 172.21.15.27:0/289157462 shutdown_connections 2024-01-15T18:09:06.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.834+0000 7ff9a5800640 1 --2- 172.21.15.27:0/289157462 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff978066e90 0x7ff978069350 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.834+0000 7ff9a5800640 1 --2- 172.21.15.27:0/289157462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00cddc0 0x7ff9a0153370 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.834+0000 7ff9a5800640 1 --2- 172.21.15.27:0/289157462 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff9a00b2390 0x7ff9a00c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.834+0000 7ff9a5800640 1 --2- 172.21.15.27:0/289157462 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9a006bb80 0x7ff9a00c5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:06.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.834+0000 7ff9a5800640 1 -- 172.21.15.27:0/289157462 >> 172.21.15.27:0/289157462 conn(0x7ff9a00b2a10 msgr2=0x7ff9a00ccd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:06.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.834+0000 7ff9a5800640 1 -- 172.21.15.27:0/289157462 shutdown_connections 2024-01-15T18:09:06.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.834+0000 7ff9a5800640 1 -- 172.21.15.27:0/289157462 wait complete. 2024-01-15T18:09:06.860 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:09:06.860 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:09:07.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.998+0000 7fbb39600640 1 Processor -- start 2024-01-15T18:09:07.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.998+0000 7fbb39600640 1 -- start start 2024-01-15T18:09:07.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.998+0000 7fbb39600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbb3415b4b0 0x7fbb340b1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:07.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.998+0000 7fbb39600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbb340cec60 0x7fbb340b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:07.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.998+0000 7fbb39600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb340d19a0 con 0x7fbb340cf460 2024-01-15T18:09:07.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.998+0000 7fbb39600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb340d1c80 con 0x7fbb340cec60 2024-01-15T18:09:07.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.998+0000 7fbb39600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb340b07c0 con 0x7fbb3415b4b0 2024-01-15T18:09:07.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.998+0000 7fbb33400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbb340cec60 0x7fbb340b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:07.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.998+0000 7fbb33400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbb340cec60 0x7fbb340b1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43746/0 (socket says 172.21.15.27:43746) 2024-01-15T18:09:07.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:06.998+0000 7fbb33400640 1 -- 172.21.15.27:0/153129189 learned_addr learned my addr 172.21.15.27:0/153129189 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:07.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.002+0000 7fbb32000640 1 --2- 172.21.15.27:0/153129189 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbb3415b4b0 0x7fbb340b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:07.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.002+0000 7fbb33400640 1 -- 172.21.15.27:0/153129189 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbb3415b4b0 msgr2=0x7fbb340b1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:07.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.002+0000 7fbb33400640 1 --2- 172.21.15.27:0/153129189 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbb3415b4b0 0x7fbb340b1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.002+0000 7fbb33400640 1 -- 172.21.15.27:0/153129189 >> v1:172.21.15.27:6789/0 conn(0x7fbb340cf460 legacy=0x7fbb340cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:07.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.002+0000 7fbb33400640 1 -- 172.21.15.27:0/153129189 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb340b0aa0 con 0x7fbb340cec60 2024-01-15T18:09:07.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.002+0000 7fbb32000640 1 --2- 172.21.15.27:0/153129189 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbb3415b4b0 0x7fbb340b1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:09:07.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.002+0000 7fbb33400640 1 --2- 172.21.15.27:0/153129189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb340cec60 0x7fbb340b1ac0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c009e50 tx=0x7fbb2c01ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d61706c5ab264a7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:07.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.002+0000 7fbb31600640 1 -- 172.21.15.27:0/153129189 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb340b2810 con 0x7fbb340cec60 2024-01-15T18:09:07.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.002+0000 7fbb31600640 1 -- 172.21.15.27:0/153129189 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb2c0a58e0 con 0x7fbb340cec60 2024-01-15T18:09:07.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.002+0000 7fbb31600640 1 -- 172.21.15.27:0/153129189 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c0a5c70 con 0x7fbb340cec60 2024-01-15T18:09:07.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb39600640 1 -- 172.21.15.27:0/153129189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb340cec60 msgr2=0x7fbb340b1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:07.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb39600640 1 --2- 172.21.15.27:0/153129189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb340cec60 0x7fbb340b1ac0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c009e50 tx=0x7fbb2c01ca10 comp rx=0 tx=0).stop 2024-01-15T18:09:07.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb39600640 1 -- 172.21.15.27:0/153129189 shutdown_connections 2024-01-15T18:09:07.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb39600640 1 --2- 172.21.15.27:0/153129189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb340cec60 0x7fbb340b1ac0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb39600640 1 --2- 172.21.15.27:0/153129189 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbb3415b4b0 0x7fbb340b1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb39600640 1 -- 172.21.15.27:0/153129189 >> 172.21.15.27:0/153129189 conn(0x7fbb340d0e80 msgr2=0x7fbb340d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:07.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb39600640 1 -- 172.21.15.27:0/153129189 shutdown_connections 2024-01-15T18:09:07.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb39600640 1 -- 172.21.15.27:0/153129189 wait complete. 2024-01-15T18:09:07.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb39600640 1 Processor -- start 2024-01-15T18:09:07.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb39600640 1 -- start start 2024-01-15T18:09:07.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb39600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbb340cec60 0x7fbb34153cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:07.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb39600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb340cf460 0x7fbb34152e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:07.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb39600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbb3415b4b0 0x7fbb3415dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:07.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb39600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb341f6930 con 0x7fbb340cf460 2024-01-15T18:09:07.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb39600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb34153370 con 0x7fbb340cec60 2024-01-15T18:09:07.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb39600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3415e5a0 con 0x7fbb3415b4b0 2024-01-15T18:09:07.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb32000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb340cf460 0x7fbb34152e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:07.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb32000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb340cf460 0x7fbb34152e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43762/0 (socket says 172.21.15.27:43762) 2024-01-15T18:09:07.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb32000640 1 -- 172.21.15.27:0/3852784328 learned_addr learned my addr 172.21.15.27:0/3852784328 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:07.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb33400640 1 --2- 172.21.15.27:0/3852784328 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbb3415b4b0 0x7fbb3415dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:07.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb32a00640 1 --2- 172.21.15.27:0/3852784328 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbb340cec60 0x7fbb34153cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:07.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb32000640 1 -- 172.21.15.27:0/3852784328 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbb3415b4b0 msgr2=0x7fbb3415dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:07.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb32000640 1 --2- 172.21.15.27:0/3852784328 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbb3415b4b0 0x7fbb3415dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb32000640 1 -- 172.21.15.27:0/3852784328 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbb340cec60 msgr2=0x7fbb34153cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:07.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb32000640 1 --2- 172.21.15.27:0/3852784328 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbb340cec60 0x7fbb34153cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb32000640 1 -- 172.21.15.27:0/3852784328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb2c009b30 con 0x7fbb340cf460 2024-01-15T18:09:07.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.006+0000 7fbb32a00640 1 --2- 172.21.15.27:0/3852784328 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbb340cec60 0x7fbb34153cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:07.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.010+0000 7fbb32000640 1 --2- 172.21.15.27:0/3852784328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb340cf460 0x7fbb34152e30 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c00e810 tx=0x7fbb1c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:07.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.010+0000 7fbb2aa00640 1 -- 172.21.15.27:0/3852784328 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c00c8d0 con 0x7fbb340cf460 2024-01-15T18:09:07.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.010+0000 7fbb39600640 1 -- 172.21.15.27:0/3852784328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb341f6000 con 0x7fbb340cf460 2024-01-15T18:09:07.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.010+0000 7fbb2aa00640 1 -- 172.21.15.27:0/3852784328 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb1c00ca30 con 0x7fbb340cf460 2024-01-15T18:09:07.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.010+0000 7fbb2aa00640 1 -- 172.21.15.27:0/3852784328 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c00cdc0 con 0x7fbb340cf460 2024-01-15T18:09:07.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.010+0000 7fbb39600640 1 -- 172.21.15.27:0/3852784328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb341f65c0 con 0x7fbb340cf460 2024-01-15T18:09:07.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.010+0000 7fbb39600640 1 -- 172.21.15.27:0/3852784328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb341f6aa0 con 0x7fbb340cf460 2024-01-15T18:09:07.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.010+0000 7fbb2aa00640 1 -- 172.21.15.27:0/3852784328 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7fbb1c013410 con 0x7fbb340cf460 2024-01-15T18:09:07.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.010+0000 7fbb2aa00640 1 --2- 172.21.15.27:0/3852784328 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbb0c066e90 0x7fbb0c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:07.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.010+0000 7fbb2aa00640 1 -- 172.21.15.27:0/3852784328 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fbb1c084ac0 con 0x7fbb340cf460 2024-01-15T18:09:07.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.010+0000 7fbb32a00640 1 --2- 172.21.15.27:0/3852784328 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbb0c066e90 0x7fbb0c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:07.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.014+0000 7fbb32a00640 1 --2- 172.21.15.27:0/3852784328 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbb0c066e90 0x7fbb0c069350 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbb24000f70 tx=0x7fbb24009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:07.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.014+0000 7fbb2aa00640 1 -- 172.21.15.27:0/3852784328 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbb1c051630 con 0x7fbb340cf460 2024-01-15T18:09:07.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.178+0000 7fbb39600640 1 -- 172.21.15.27:0/3852784328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbb3404e850 con 0x7fbb340cf460 2024-01-15T18:09:07.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.178+0000 7fbb2aa00640 1 -- 172.21.15.27:0/3852784328 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbb1c009480 con 0x7fbb340cf460 2024-01-15T18:09:07.183 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:09:07.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.178+0000 7fbb39600640 1 -- 172.21.15.27:0/3852784328 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbb0c066e90 msgr2=0x7fbb0c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:07.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.178+0000 7fbb39600640 1 --2- 172.21.15.27:0/3852784328 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbb0c066e90 0x7fbb0c069350 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbb24000f70 tx=0x7fbb24009040 comp rx=0 tx=0).stop 2024-01-15T18:09:07.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.178+0000 7fbb39600640 1 -- 172.21.15.27:0/3852784328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb340cf460 msgr2=0x7fbb34152e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:07.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.178+0000 7fbb39600640 1 --2- 172.21.15.27:0/3852784328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb340cf460 0x7fbb34152e30 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c00e810 tx=0x7fbb1c00ecd0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.178+0000 7fbb39600640 1 -- 172.21.15.27:0/3852784328 shutdown_connections 2024-01-15T18:09:07.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.178+0000 7fbb39600640 1 --2- 172.21.15.27:0/3852784328 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbb0c066e90 0x7fbb0c069350 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.178+0000 7fbb39600640 1 --2- 172.21.15.27:0/3852784328 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbb3415b4b0 0x7fbb3415dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.178+0000 7fbb39600640 1 --2- 172.21.15.27:0/3852784328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb340cf460 0x7fbb34152e30 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.178+0000 7fbb39600640 1 --2- 172.21.15.27:0/3852784328 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbb340cec60 0x7fbb34153cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.178+0000 7fbb39600640 1 -- 172.21.15.27:0/3852784328 >> 172.21.15.27:0/3852784328 conn(0x7fbb340d0e80 msgr2=0x7fbb3406c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:07.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.178+0000 7fbb39600640 1 -- 172.21.15.27:0/3852784328 shutdown_connections 2024-01-15T18:09:07.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.178+0000 7fbb39600640 1 -- 172.21.15.27:0/3852784328 wait complete. 2024-01-15T18:09:07.205 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-15T18:09:07.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.338+0000 7fc06ee00640 1 Processor -- start 2024-01-15T18:09:07.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.338+0000 7fc06ee00640 1 -- start start 2024-01-15T18:09:07.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc06ee00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc068160b10 0x7fc0680afe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:07.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc06ee00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fc0680ce330 0x7fc0680cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:07.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc06ee00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0680d05b0 con 0x7fc0680cd7a0 2024-01-15T18:09:07.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc06ee00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0680ce7d0 con 0x7fc0680ce330 2024-01-15T18:09:07.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc06ee00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0680cfc80 con 0x7fc068160b10 2024-01-15T18:09:07.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc067400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc0680cd7a0 0x7fc0680cdc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54016/0 (socket says 172.21.15.27:54016) 2024-01-15T18:09:07.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc067400640 1 -- 172.21.15.27:0/4179817522 learned_addr learned my addr 172.21.15.27:0/4179817522 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:07.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc067e00640 1 --2- 172.21.15.27:0/4179817522 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc068160b10 0x7fc0680afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:07.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc06ce00640 1 --2- 172.21.15.27:0/4179817522 >> v2:172.21.15.93:3300/0 conn(0x7fc0680ce330 0x7fc0680cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:07.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc066a00640 1 -- 172.21.15.27:0/4179817522 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726704858 0 0) 0x7fc0680d05b0 con 0x7fc0680cd7a0 2024-01-15T18:09:07.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc066a00640 1 -- 172.21.15.27:0/4179817522 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc048003530 con 0x7fc0680cd7a0 2024-01-15T18:09:07.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc066a00640 1 -- 172.21.15.27:0/4179817522 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 197027995 0 0) 0x7fc048003530 con 0x7fc0680cd7a0 2024-01-15T18:09:07.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc066a00640 1 -- 172.21.15.27:0/4179817522 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc068160b10 msgr2=0x7fc0680afe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:07.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc066a00640 1 --2- 172.21.15.27:0/4179817522 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc068160b10 0x7fc0680afe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc066a00640 1 -- 172.21.15.27:0/4179817522 >> v2:172.21.15.93:3300/0 conn(0x7fc0680ce330 msgr2=0x7fc0680cb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:07.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc066a00640 1 --2- 172.21.15.27:0/4179817522 >> v2:172.21.15.93:3300/0 conn(0x7fc0680ce330 0x7fc0680cb7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc066a00640 1 -- 172.21.15.27:0/4179817522 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0680cff60 con 0x7fc0680cd7a0 2024-01-15T18:09:07.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc066a00640 1 -- 172.21.15.27:0/4179817522 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fc050001680 con 0x7fc0680cd7a0 2024-01-15T18:09:07.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc066a00640 1 -- 172.21.15.27:0/4179817522 >> v1:172.21.15.27:6789/0 conn(0x7fc0680cd7a0 legacy=0x7fc0680cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:07.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc066a00640 1 --2- 172.21.15.27:0/4179817522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0480039f0 0x7fc048003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:07.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc066a00640 1 --2- 172.21.15.27:0/4179817522 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc0480047d0 0x7fc048004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:07.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc066a00640 1 --2- 172.21.15.27:0/4179817522 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc068160b10 0x7fc048005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:07.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc066a00640 1 -- 172.21.15.27:0/4179817522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0680ce7d0 con 0x7fc0480039f0 2024-01-15T18:09:07.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc066a00640 1 -- 172.21.15.27:0/4179817522 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0680cfc80 con 0x7fc0480047d0 2024-01-15T18:09:07.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc066a00640 1 -- 172.21.15.27:0/4179817522 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc04800ba10 con 0x7fc068160b10 2024-01-15T18:09:07.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc067e00640 1 --2- 172.21.15.27:0/4179817522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0480039f0 0x7fc048003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:07.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc067400640 1 --2- 172.21.15.27:0/4179817522 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc0480047d0 0x7fc048004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:07.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc06ce00640 1 --2- 172.21.15.27:0/4179817522 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc068160b10 0x7fc048005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:07.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc067e00640 1 -- 172.21.15.27:0/4179817522 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc068160b10 msgr2=0x7fc048005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:07.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc067e00640 1 --2- 172.21.15.27:0/4179817522 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc068160b10 0x7fc048005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc067e00640 1 -- 172.21.15.27:0/4179817522 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc0480047d0 msgr2=0x7fc048004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:07.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc067e00640 1 --2- 172.21.15.27:0/4179817522 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc0480047d0 0x7fc048004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc067e00640 1 -- 172.21.15.27:0/4179817522 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc048003530 con 0x7fc0480039f0 2024-01-15T18:09:07.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.342+0000 7fc067400640 1 --2- 172.21.15.27:0/4179817522 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc0480047d0 0x7fc048004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:09:07.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.346+0000 7fc067e00640 1 --2- 172.21.15.27:0/4179817522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0480039f0 0x7fc048003e10 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fc058009ae0 tx=0x7fc05801ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d964c5a86849cbb0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:07.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.346+0000 7fc066a00640 1 -- 172.21.15.27:0/4179817522 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0580a6430 con 0x7fc0480039f0 2024-01-15T18:09:07.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.346+0000 7fc066a00640 1 -- 172.21.15.27:0/4179817522 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0580a6590 con 0x7fc0480039f0 2024-01-15T18:09:07.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.346+0000 7fc066a00640 1 -- 172.21.15.27:0/4179817522 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0580a6940 con 0x7fc0480039f0 2024-01-15T18:09:07.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.346+0000 7fc06ee00640 1 -- 172.21.15.27:0/4179817522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0480039f0 msgr2=0x7fc048003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:07.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.346+0000 7fc06ee00640 1 --2- 172.21.15.27:0/4179817522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0480039f0 0x7fc048003e10 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fc058009ae0 tx=0x7fc05801ca20 comp rx=0 tx=0).stop 2024-01-15T18:09:07.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.346+0000 7fc06ce00640 1 -- 172.21.15.27:0/4179817522 reap_dead start 2024-01-15T18:09:07.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.346+0000 7fc06ee00640 1 -- 172.21.15.27:0/4179817522 shutdown_connections 2024-01-15T18:09:07.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.346+0000 7fc06ee00640 1 -- 172.21.15.27:0/4179817522 >> 172.21.15.27:0/4179817522 conn(0x7fc0680d0f20 msgr2=0x7fc0680d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:07.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.346+0000 7fc06ee00640 1 -- 172.21.15.27:0/4179817522 shutdown_connections 2024-01-15T18:09:07.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.346+0000 7fc06ee00640 1 -- 172.21.15.27:0/4179817522 wait complete. 2024-01-15T18:09:07.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.350+0000 7fc06ee00640 1 Processor -- start 2024-01-15T18:09:07.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.350+0000 7fc06ee00640 1 -- start start 2024-01-15T18:09:07.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.350+0000 7fc06ee00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc0480039f0 0x7fc068153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:07.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.350+0000 7fc06ee00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc068153820 0x7fc06815f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:07.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.350+0000 7fc06ee00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0680c79f0 0x7fc0680c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:07.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.350+0000 7fc06ee00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0680c97b0 con 0x7fc0680c79f0 2024-01-15T18:09:07.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.350+0000 7fc06ee00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0680c8ba0 con 0x7fc0480039f0 2024-01-15T18:09:07.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.350+0000 7fc06ee00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0680c8e80 con 0x7fc068153820 2024-01-15T18:09:07.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.350+0000 7fc06ce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0680c79f0 0x7fc0680c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:07.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.350+0000 7fc06ce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0680c79f0 0x7fc0680c9c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43778/0 (socket says 172.21.15.27:43778) 2024-01-15T18:09:07.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.350+0000 7fc06ce00640 1 -- 172.21.15.27:0/296391688 learned_addr learned my addr 172.21.15.27:0/296391688 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:07.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.350+0000 7fc067400640 1 --2- 172.21.15.27:0/296391688 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc068153820 0x7fc06815f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:07.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.350+0000 7fc067e00640 1 --2- 172.21.15.27:0/296391688 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc0480039f0 0x7fc068153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:07.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.350+0000 7fc06ce00640 1 -- 172.21.15.27:0/296391688 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc068153820 msgr2=0x7fc06815f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:07.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.350+0000 7fc06ce00640 1 --2- 172.21.15.27:0/296391688 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc068153820 0x7fc06815f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.350+0000 7fc06ce00640 1 -- 172.21.15.27:0/296391688 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc0480039f0 msgr2=0x7fc068153fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:07.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.350+0000 7fc06ce00640 1 --2- 172.21.15.27:0/296391688 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc0480039f0 0x7fc068153fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.350+0000 7fc06ce00640 1 -- 172.21.15.27:0/296391688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0680c9160 con 0x7fc0680c79f0 2024-01-15T18:09:07.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.350+0000 7fc06ce00640 1 --2- 172.21.15.27:0/296391688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0680c79f0 0x7fc0680c9c40 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fc068071aa0 tx=0x7fc0600115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:07.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.350+0000 7fc05fe00640 1 -- 172.21.15.27:0/296391688 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06000e070 con 0x7fc0680c79f0 2024-01-15T18:09:07.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.350+0000 7fc05fe00640 1 -- 172.21.15.27:0/296391688 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc060011ed0 con 0x7fc0680c79f0 2024-01-15T18:09:07.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.350+0000 7fc06ee00640 1 -- 172.21.15.27:0/296391688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0680c9440 con 0x7fc0680c79f0 2024-01-15T18:09:07.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.350+0000 7fc05fe00640 1 -- 172.21.15.27:0/296391688 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0600039a0 con 0x7fc0680c79f0 2024-01-15T18:09:07.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.350+0000 7fc06ee00640 1 -- 172.21.15.27:0/296391688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0680ca180 con 0x7fc0680c79f0 2024-01-15T18:09:07.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.354+0000 7fc06ee00640 1 -- 172.21.15.27:0/296391688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0680c9960 con 0x7fc0680c79f0 2024-01-15T18:09:07.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.354+0000 7fc05fe00640 1 -- 172.21.15.27:0/296391688 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7fc060003b00 con 0x7fc0680c79f0 2024-01-15T18:09:07.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.354+0000 7fc05fe00640 1 --2- 172.21.15.27:0/296391688 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc040066e90 0x7fc040069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:07.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.354+0000 7fc05fe00640 1 -- 172.21.15.27:0/296391688 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fc060085370 con 0x7fc0680c79f0 2024-01-15T18:09:07.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.354+0000 7fc067e00640 1 --2- 172.21.15.27:0/296391688 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc040066e90 0x7fc040069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:07.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.354+0000 7fc067e00640 1 --2- 172.21.15.27:0/296391688 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc040066e90 0x7fc040069350 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc0580ad000 tx=0x7fc0580b2040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:07.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.358+0000 7fc05fe00640 1 -- 172.21.15.27:0/296391688 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc0680c9960 con 0x7fc0680c79f0 2024-01-15T18:09:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.518+0000 7fc06ee00640 1 -- 172.21.15.27:0/296391688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc06804e850 con 0x7fc0680c79f0 2024-01-15T18:09:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.518+0000 7fc05fe00640 1 -- 172.21.15.27:0/296391688 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v54) v1 ==== 84+0+65 (secure 0 0 0) 0x7fc060052070 con 0x7fc0680c79f0 2024-01-15T18:09:07.523 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:09:07.524 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:09:07.524 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:09:07.524 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-01-15T18:09:07.524 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-01-15T18:09:07.524 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:09:07.524 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:09:07.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.518+0000 7fc06ee00640 1 -- 172.21.15.27:0/296391688 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc040066e90 msgr2=0x7fc040069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:07.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.518+0000 7fc06ee00640 1 --2- 172.21.15.27:0/296391688 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc040066e90 0x7fc040069350 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc0580ad000 tx=0x7fc0580b2040 comp rx=0 tx=0).stop 2024-01-15T18:09:07.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.518+0000 7fc06ee00640 1 -- 172.21.15.27:0/296391688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0680c79f0 msgr2=0x7fc0680c9c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:07.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.518+0000 7fc06ee00640 1 --2- 172.21.15.27:0/296391688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0680c79f0 0x7fc0680c9c40 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fc068071aa0 tx=0x7fc0600115a0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.518+0000 7fc06ee00640 1 -- 172.21.15.27:0/296391688 shutdown_connections 2024-01-15T18:09:07.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.518+0000 7fc06ee00640 1 --2- 172.21.15.27:0/296391688 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc040066e90 0x7fc040069350 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.518+0000 7fc06ee00640 1 --2- 172.21.15.27:0/296391688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0680c79f0 0x7fc0680c9c40 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.518+0000 7fc06ee00640 1 --2- 172.21.15.27:0/296391688 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc068153820 0x7fc06815f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.518+0000 7fc06ee00640 1 --2- 172.21.15.27:0/296391688 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc0480039f0 0x7fc068153fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.518+0000 7fc06ee00640 1 -- 172.21.15.27:0/296391688 >> 172.21.15.27:0/296391688 conn(0x7fc0680d0f20 msgr2=0x7fc0680ae090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:07.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.518+0000 7fc06ee00640 1 -- 172.21.15.27:0/296391688 shutdown_connections 2024-01-15T18:09:07.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.518+0000 7fc06ee00640 1 -- 172.21.15.27:0/296391688 wait complete. 2024-01-15T18:09:07.545 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-15T18:09:07.545 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-15T18:09:07.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.686+0000 7f3558200640 1 Processor -- start 2024-01-15T18:09:07.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.686+0000 7f3558200640 1 -- start start 2024-01-15T18:09:07.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.686+0000 7f3558200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f35500b2360 0x7f355006c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:07.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.686+0000 7f3558200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3550161640 0x7f35500cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:07.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.686+0000 7f3558200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35500b2740 con 0x7f35500cf3c0 2024-01-15T18:09:07.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.686+0000 7f3558200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35500cf850 con 0x7f35500b2360 2024-01-15T18:09:07.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.686+0000 7f3558200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35500cfb30 con 0x7f3550161640 2024-01-15T18:09:07.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.686+0000 7f3555800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f35500b2360 0x7f355006c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:07.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.686+0000 7f3556200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3550161640 0x7f35500cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:07.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.686+0000 7f3554e00640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7f35500cf3c0 0x7f3550160f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:59836/0 (socket says 172.21.15.27:59836) 2024-01-15T18:09:07.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.686+0000 7f3554e00640 1 -- 172.21.15.27:0/1820576672 learned_addr learned my addr 172.21.15.27:0/1820576672 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:07.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.686+0000 7f354fe00640 1 -- 172.21.15.27:0/1820576672 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744295008 0 0) 0x7f35500b2740 con 0x7f35500cf3c0 2024-01-15T18:09:07.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.686+0000 7f354fe00640 1 -- 172.21.15.27:0/1820576672 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3530003530 con 0x7f35500cf3c0 2024-01-15T18:09:07.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.686+0000 7f3556200640 1 -- 172.21.15.27:0/1820576672 >> v2:172.21.15.93:3300/0 conn(0x7f35500b2360 msgr2=0x7f355006c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:07.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.686+0000 7f3556200640 1 --2- 172.21.15.27:0/1820576672 >> v2:172.21.15.93:3300/0 conn(0x7f35500b2360 0x7f355006c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.686+0000 7f3556200640 1 -- 172.21.15.27:0/1820576672 >> v1:172.21.15.93:6789/0 conn(0x7f35500cf3c0 legacy=0x7f3550160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:07.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.686+0000 7f3556200640 1 -- 172.21.15.27:0/1820576672 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35500cfe10 con 0x7f3550161640 2024-01-15T18:09:07.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.686+0000 7f3555800640 1 --2- 172.21.15.27:0/1820576672 >> v2:172.21.15.93:3300/0 conn(0x7f35500b2360 0x7f355006c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:09:07.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.690+0000 7f3556200640 1 --2- 172.21.15.27:0/1820576672 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3550161640 0x7f35500cb4c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3548009d90 tx=0x7f354801c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6d47b3294793cd3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:07.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.690+0000 7f354fe00640 1 -- 172.21.15.27:0/1820576672 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35480a5900 con 0x7f3550161640 2024-01-15T18:09:07.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.690+0000 7f354fe00640 1 -- 172.21.15.27:0/1820576672 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35480a5a60 con 0x7f3550161640 2024-01-15T18:09:07.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.690+0000 7f354fe00640 1 -- 172.21.15.27:0/1820576672 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35480a5dd0 con 0x7f3550161640 2024-01-15T18:09:07.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.690+0000 7f3558200640 1 -- 172.21.15.27:0/1820576672 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3550161640 msgr2=0x7f35500cb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:07.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.690+0000 7f3558200640 1 --2- 172.21.15.27:0/1820576672 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3550161640 0x7f35500cb4c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3548009d90 tx=0x7f354801c950 comp rx=0 tx=0).stop 2024-01-15T18:09:07.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.690+0000 7f3558200640 1 -- 172.21.15.27:0/1820576672 shutdown_connections 2024-01-15T18:09:07.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.690+0000 7f3558200640 1 --2- 172.21.15.27:0/1820576672 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3550161640 0x7f35500cb4c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.690+0000 7f3558200640 1 --2- 172.21.15.27:0/1820576672 >> v2:172.21.15.93:3300/0 conn(0x7f35500b2360 0x7f355006c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.690+0000 7f3558200640 1 -- 172.21.15.27:0/1820576672 >> 172.21.15.27:0/1820576672 conn(0x7f35500d0e70 msgr2=0x7f35500d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:07.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.694+0000 7f3558200640 1 -- 172.21.15.27:0/1820576672 shutdown_connections 2024-01-15T18:09:07.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.694+0000 7f3558200640 1 -- 172.21.15.27:0/1820576672 wait complete. 2024-01-15T18:09:07.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.694+0000 7f3558200640 1 Processor -- start 2024-01-15T18:09:07.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.694+0000 7f3558200640 1 -- start start 2024-01-15T18:09:07.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.694+0000 7f3558200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35500b2360 0x7f355014f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:07.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.694+0000 7f3558200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3550161640 0x7f355015e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:07.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.694+0000 7f3558200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35500c48d0 0x7f355015ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:07.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.694+0000 7f3558200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35500c4e90 con 0x7f35500b2360 2024-01-15T18:09:07.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.694+0000 7f3558200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35500c5170 con 0x7f3550161640 2024-01-15T18:09:07.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.694+0000 7f3558200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35500c5450 con 0x7f35500c48d0 2024-01-15T18:09:07.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.694+0000 7f3555800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35500b2360 0x7f355014f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:07.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.694+0000 7f3555800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35500b2360 0x7f355014f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43788/0 (socket says 172.21.15.27:43788) 2024-01-15T18:09:07.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.694+0000 7f3555800640 1 -- 172.21.15.27:0/3254861057 learned_addr learned my addr 172.21.15.27:0/3254861057 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:07.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.694+0000 7f3556200640 1 --2- 172.21.15.27:0/3254861057 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35500c48d0 0x7f355015ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:07.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.694+0000 7f3554e00640 1 --2- 172.21.15.27:0/3254861057 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3550161640 0x7f355015e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:07.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.694+0000 7f3555800640 1 -- 172.21.15.27:0/3254861057 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35500c48d0 msgr2=0x7f355015ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:07.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.694+0000 7f3555800640 1 --2- 172.21.15.27:0/3254861057 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35500c48d0 0x7f355015ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.694+0000 7f3555800640 1 -- 172.21.15.27:0/3254861057 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3550161640 msgr2=0x7f355015e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:07.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.694+0000 7f3555800640 1 --2- 172.21.15.27:0/3254861057 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3550161640 0x7f355015e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:07.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.694+0000 7f3555800640 1 -- 172.21.15.27:0/3254861057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3540009420 con 0x7f35500b2360 2024-01-15T18:09:07.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.694+0000 7f3554e00640 1 --2- 172.21.15.27:0/3254861057 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3550161640 0x7f355015e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:09:07.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.694+0000 7f3555800640 1 --2- 172.21.15.27:0/3254861057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35500b2360 0x7f355014f350 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f354000e980 tx=0x7f354000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:07.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.694+0000 7f354d600640 1 -- 172.21.15.27:0/3254861057 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354000ca00 con 0x7f35500b2360 2024-01-15T18:09:07.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.698+0000 7f354d600640 1 -- 172.21.15.27:0/3254861057 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354000cb60 con 0x7f35500b2360 2024-01-15T18:09:07.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.698+0000 7f3558200640 1 -- 172.21.15.27:0/3254861057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3548009a70 con 0x7f35500b2360 2024-01-15T18:09:07.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.698+0000 7f354d600640 1 -- 172.21.15.27:0/3254861057 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3540013470 con 0x7f35500b2360 2024-01-15T18:09:07.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.698+0000 7f3558200640 1 -- 172.21.15.27:0/3254861057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35500c5a10 con 0x7f35500b2360 2024-01-15T18:09:07.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.698+0000 7f3558200640 1 -- 172.21.15.27:0/3254861057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f355004e850 con 0x7f35500b2360 2024-01-15T18:09:07.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.698+0000 7f354d600640 1 -- 172.21.15.27:0/3254861057 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f35400135d0 con 0x7f35500b2360 2024-01-15T18:09:07.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.698+0000 7f354d600640 1 --2- 172.21.15.27:0/3254861057 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3528066ec0 0x7f3528069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:07.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.698+0000 7f354d600640 1 -- 172.21.15.27:0/3254861057 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f3540084d40 con 0x7f35500b2360 2024-01-15T18:09:07.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.698+0000 7f3554e00640 1 --2- 172.21.15.27:0/3254861057 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3528066ec0 0x7f3528069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:07.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.702+0000 7f3554e00640 1 --2- 172.21.15.27:0/3254861057 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3528066ec0 0x7f3528069380 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f35500cac70 tx=0x7f3538009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:07.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.702+0000 7f354d600640 1 -- 172.21.15.27:0/3254861057 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3540051ab0 con 0x7f35500b2360 2024-01-15T18:09:07.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:07.862+0000 7f3558200640 1 -- 172.21.15.27:0/3254861057 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f3550161d10 con 0x7f35500b2360 2024-01-15T18:09:08.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.106+0000 7f354d600640 1 -- 172.21.15.27:0/3254861057 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v55) v1 ==== 138+0+0 (secure 0 0 0) 0x7f35400550d0 con 0x7f35500b2360 2024-01-15T18:09:08.112 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-01-15T18:09:08.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.106+0000 7f3558200640 1 -- 172.21.15.27:0/3254861057 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3528066ec0 msgr2=0x7f3528069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:08.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.106+0000 7f3558200640 1 --2- 172.21.15.27:0/3254861057 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3528066ec0 0x7f3528069380 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f35500cac70 tx=0x7f3538009040 comp rx=0 tx=0).stop 2024-01-15T18:09:08.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.106+0000 7f3558200640 1 -- 172.21.15.27:0/3254861057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35500b2360 msgr2=0x7f355014f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:08.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.106+0000 7f3558200640 1 --2- 172.21.15.27:0/3254861057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35500b2360 0x7f355014f350 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f354000e980 tx=0x7f354000ee40 comp rx=0 tx=0).stop 2024-01-15T18:09:08.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.106+0000 7f3558200640 1 -- 172.21.15.27:0/3254861057 shutdown_connections 2024-01-15T18:09:08.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.106+0000 7f3558200640 1 --2- 172.21.15.27:0/3254861057 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3528066ec0 0x7f3528069380 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:08.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.106+0000 7f3558200640 1 --2- 172.21.15.27:0/3254861057 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35500c48d0 0x7f355015ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:08.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.106+0000 7f3558200640 1 --2- 172.21.15.27:0/3254861057 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3550161640 0x7f355015e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:08.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.106+0000 7f3558200640 1 --2- 172.21.15.27:0/3254861057 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35500b2360 0x7f355014f350 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:08.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.106+0000 7f3558200640 1 -- 172.21.15.27:0/3254861057 >> 172.21.15.27:0/3254861057 conn(0x7f35500d0e70 msgr2=0x7f355006fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:08.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.110+0000 7f3558200640 1 -- 172.21.15.27:0/3254861057 shutdown_connections 2024-01-15T18:09:08.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.110+0000 7f3558200640 1 -- 172.21.15.27:0/3254861057 wait complete. 2024-01-15T18:09:08.139 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-15T18:09:08.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.282+0000 7f6933400640 1 Processor -- start 2024-01-15T18:09:08.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.282+0000 7f6933400640 1 -- start start 2024-01-15T18:09:08.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.282+0000 7f6933400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f692c15f550 0x7f692c0cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:08.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.282+0000 7f6933400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692c0b1cb0 con 0x7f692c0cf110 2024-01-15T18:09:08.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.282+0000 7f6933400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692c15fa00 con 0x7f692c158900 2024-01-15T18:09:08.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.282+0000 7f6933400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f692c0cf540 con 0x7f692c15f550 2024-01-15T18:09:08.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.282+0000 7f6931400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f692c0cf110 0x7f692c0701e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54018/0 (socket says 172.21.15.27:54018) 2024-01-15T18:09:08.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.282+0000 7f6931400640 1 -- 172.21.15.27:0/4263059259 learned_addr learned my addr 172.21.15.27:0/4263059259 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:08.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.282+0000 7f692be00640 1 --2- 172.21.15.27:0/4263059259 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f692c15f550 0x7f692c0cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:08.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.282+0000 7f692b400640 1 -- 172.21.15.27:0/4263059259 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610454028 0 0) 0x7f692c0b1cb0 con 0x7f692c0cf110 2024-01-15T18:09:08.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.282+0000 7f692b400640 1 -- 172.21.15.27:0/4263059259 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f690c003530 con 0x7f692c0cf110 2024-01-15T18:09:08.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.282+0000 7f692b400640 1 -- 172.21.15.27:0/4263059259 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512470679 0 0) 0x7f692c15fa00 con 0x7f692c158900 2024-01-15T18:09:08.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.282+0000 7f692b400640 1 -- 172.21.15.27:0/4263059259 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f692c0b1cb0 con 0x7f692c158900 2024-01-15T18:09:08.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.282+0000 7f692be00640 1 -- 172.21.15.27:0/4263059259 >> v1:172.21.15.93:6789/0 conn(0x7f692c158900 legacy=0x7f692c15bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:08.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.282+0000 7f692be00640 1 -- 172.21.15.27:0/4263059259 >> v1:172.21.15.27:6789/0 conn(0x7f692c0cf110 legacy=0x7f692c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:08.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.282+0000 7f692be00640 1 -- 172.21.15.27:0/4263059259 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692c0b1f00 con 0x7f692c15f550 2024-01-15T18:09:08.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.286+0000 7f692be00640 1 --2- 172.21.15.27:0/4263059259 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f692c15f550 0x7f692c0cebd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f69140097f0 tx=0x7f691401c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5eca15a2cfbc20e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:08.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.286+0000 7f692b400640 1 -- 172.21.15.27:0/4263059259 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69140aa5f0 con 0x7f692c15f550 2024-01-15T18:09:08.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.286+0000 7f692b400640 1 -- 172.21.15.27:0/4263059259 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69140aa750 con 0x7f692c15f550 2024-01-15T18:09:08.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.286+0000 7f692b400640 1 -- 172.21.15.27:0/4263059259 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69140aaac0 con 0x7f692c15f550 2024-01-15T18:09:08.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.286+0000 7f6933400640 1 -- 172.21.15.27:0/4263059259 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f692c15f550 msgr2=0x7f692c0cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:08.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.286+0000 7f6933400640 1 --2- 172.21.15.27:0/4263059259 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f692c15f550 0x7f692c0cebd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f69140097f0 tx=0x7f691401c490 comp rx=0 tx=0).stop 2024-01-15T18:09:08.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.286+0000 7f6933400640 1 -- 172.21.15.27:0/4263059259 shutdown_connections 2024-01-15T18:09:08.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.286+0000 7f6933400640 1 --2- 172.21.15.27:0/4263059259 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f692c15f550 0x7f692c0cebd0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:08.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.286+0000 7f6933400640 1 -- 172.21.15.27:0/4263059259 >> 172.21.15.27:0/4263059259 conn(0x7f692c0cb700 msgr2=0x7f692c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:08.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.286+0000 7f6933400640 1 -- 172.21.15.27:0/4263059259 shutdown_connections 2024-01-15T18:09:08.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.286+0000 7f6933400640 1 -- 172.21.15.27:0/4263059259 wait complete. 2024-01-15T18:09:08.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.290+0000 7f6933400640 1 Processor -- start 2024-01-15T18:09:08.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.290+0000 7f6933400640 1 -- start start 2024-01-15T18:09:08.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.290+0000 7f6933400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f692c0cf110 0x7f692c152690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:08.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.290+0000 7f6933400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f692c158900 0x7f692c14e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:08.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.290+0000 7f6933400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f692c15f550 0x7f692c15cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:08.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.290+0000 7f6933400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692c1582b0 con 0x7f692c158900 2024-01-15T18:09:08.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.290+0000 7f6933400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692c14f960 con 0x7f692c0cf110 2024-01-15T18:09:08.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.290+0000 7f6933400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f692c157b40 con 0x7f692c15f550 2024-01-15T18:09:08.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.290+0000 7f692be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f692c158900 0x7f692c14e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:08.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.290+0000 7f692be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f692c158900 0x7f692c14e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43798/0 (socket says 172.21.15.27:43798) 2024-01-15T18:09:08.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.290+0000 7f692be00640 1 -- 172.21.15.27:0/4055248060 learned_addr learned my addr 172.21.15.27:0/4055248060 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:08.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.290+0000 7f6931400640 1 --2- 172.21.15.27:0/4055248060 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f692c15f550 0x7f692c15cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:08.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.290+0000 7f6930a00640 1 --2- 172.21.15.27:0/4055248060 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f692c0cf110 0x7f692c152690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:08.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.290+0000 7f692be00640 1 -- 172.21.15.27:0/4055248060 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f692c15f550 msgr2=0x7f692c15cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:08.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.290+0000 7f692be00640 1 --2- 172.21.15.27:0/4055248060 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f692c15f550 0x7f692c15cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:08.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.290+0000 7f692be00640 1 -- 172.21.15.27:0/4055248060 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f692c0cf110 msgr2=0x7f692c152690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:08.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.290+0000 7f692be00640 1 --2- 172.21.15.27:0/4055248060 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f692c0cf110 0x7f692c152690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:08.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.290+0000 7f692be00640 1 -- 172.21.15.27:0/4055248060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6914009420 con 0x7f692c158900 2024-01-15T18:09:08.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.290+0000 7f6930a00640 1 --2- 172.21.15.27:0/4055248060 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f692c0cf110 0x7f692c152690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:09:08.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.290+0000 7f692be00640 1 --2- 172.21.15.27:0/4055248060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f692c158900 0x7f692c14e4b0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f691401a0c0 tx=0x7f6914019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:08.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.290+0000 7f6928c00640 1 -- 172.21.15.27:0/4055248060 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69140aacd0 con 0x7f692c158900 2024-01-15T18:09:08.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.294+0000 7f6933400640 1 -- 172.21.15.27:0/4055248060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f692c14ef10 con 0x7f692c158900 2024-01-15T18:09:08.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.294+0000 7f6928c00640 1 -- 172.21.15.27:0/4055248060 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69140aae30 con 0x7f692c158900 2024-01-15T18:09:08.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.294+0000 7f6928c00640 1 -- 172.21.15.27:0/4055248060 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69140bd610 con 0x7f692c158900 2024-01-15T18:09:08.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.294+0000 7f6933400640 1 -- 172.21.15.27:0/4055248060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f692c14ec30 con 0x7f692c158900 2024-01-15T18:09:08.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.294+0000 7f6933400640 1 -- 172.21.15.27:0/4055248060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f692c04e850 con 0x7f692c158900 2024-01-15T18:09:08.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.294+0000 7f6928c00640 1 -- 172.21.15.27:0/4055248060 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f69140bd770 con 0x7f692c158900 2024-01-15T18:09:08.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.294+0000 7f6928c00640 1 --2- 172.21.15.27:0/4055248060 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6904066e90 0x7f6904069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:08.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.294+0000 7f6928c00640 1 -- 172.21.15.27:0/4055248060 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5255+0+0 (secure 0 0 0) 0x7f69140a7070 con 0x7f692c158900 2024-01-15T18:09:08.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.294+0000 7f6930a00640 1 --2- 172.21.15.27:0/4055248060 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6904066e90 0x7f6904069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:08.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.298+0000 7f6930a00640 1 --2- 172.21.15.27:0/4055248060 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6904066e90 0x7f6904069350 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f692c14fed0 tx=0x7f691c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:08.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.298+0000 7f6928c00640 1 -- 172.21.15.27:0/4055248060 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f69140ebaa0 con 0x7f692c158900 2024-01-15T18:09:08.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:08.458+0000 7f6933400640 1 -- 172.21.15.27:0/4055248060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f692c06d540 con 0x7f692c158900 2024-01-15T18:09:09.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.110+0000 7f6928c00640 1 -- 172.21.15.27:0/4055248060 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v56) v1 ==== 170+0+0 (secure 0 0 0) 0x7f69140c20d0 con 0x7f692c158900 2024-01-15T18:09:09.115 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-01-15T18:09:09.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.110+0000 7f6933400640 1 -- 172.21.15.27:0/4055248060 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6904066e90 msgr2=0x7f6904069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:09.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.110+0000 7f6933400640 1 --2- 172.21.15.27:0/4055248060 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6904066e90 0x7f6904069350 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f692c14fed0 tx=0x7f691c009040 comp rx=0 tx=0).stop 2024-01-15T18:09:09.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.110+0000 7f6933400640 1 -- 172.21.15.27:0/4055248060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f692c158900 msgr2=0x7f692c14e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:09.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.110+0000 7f6933400640 1 --2- 172.21.15.27:0/4055248060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f692c158900 0x7f692c14e4b0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f691401a0c0 tx=0x7f6914019230 comp rx=0 tx=0).stop 2024-01-15T18:09:09.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.114+0000 7f6933400640 1 -- 172.21.15.27:0/4055248060 shutdown_connections 2024-01-15T18:09:09.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.114+0000 7f6933400640 1 --2- 172.21.15.27:0/4055248060 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6904066e90 0x7f6904069350 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:09.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.114+0000 7f6933400640 1 --2- 172.21.15.27:0/4055248060 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f692c15f550 0x7f692c15cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:09.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.114+0000 7f6933400640 1 --2- 172.21.15.27:0/4055248060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f692c158900 0x7f692c14e4b0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:09.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.114+0000 7f6933400640 1 --2- 172.21.15.27:0/4055248060 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f692c0cf110 0x7f692c152690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:09.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.114+0000 7f6933400640 1 -- 172.21.15.27:0/4055248060 >> 172.21.15.27:0/4055248060 conn(0x7f692c0cb700 msgr2=0x7f692c06cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:09.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.114+0000 7f6933400640 1 -- 172.21.15.27:0/4055248060 shutdown_connections 2024-01-15T18:09:09.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.114+0000 7f6933400640 1 -- 172.21.15.27:0/4055248060 wait complete. 2024-01-15T18:09:09.160 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-15T18:09:09.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.298+0000 7fa51f400640 1 Processor -- start 2024-01-15T18:09:09.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.298+0000 7fa51f400640 1 -- start start 2024-01-15T18:09:09.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.302+0000 7fa51f400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa5180b2360 0x7fa51806c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:09.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.302+0000 7fa51f400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa5180cf3c0 0x7fa518160f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:09.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.302+0000 7fa51f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa51806ca00 0x7fa518161470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:09.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.302+0000 7fa51f400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5180b2740 con 0x7fa51806ca00 2024-01-15T18:09:09.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.302+0000 7fa51f400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5180cf850 con 0x7fa5180cf3c0 2024-01-15T18:09:09.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.302+0000 7fa51f400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5180cfb30 con 0x7fa5180b2360 2024-01-15T18:09:09.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.302+0000 7fa51d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa51806ca00 0x7fa518161470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:09.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.302+0000 7fa51d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa51806ca00 0x7fa518161470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43800/0 (socket says 172.21.15.27:43800) 2024-01-15T18:09:09.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.302+0000 7fa51d400640 1 -- 172.21.15.27:0/2158113469 learned_addr learned my addr 172.21.15.27:0/2158113469 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:09.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.302+0000 7fa517e00640 1 --2- 172.21.15.27:0/2158113469 >> v2:172.21.15.93:3300/0 conn(0x7fa5180cf3c0 0x7fa518160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:09.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.302+0000 7fa51ca00640 1 --2- 172.21.15.27:0/2158113469 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa5180b2360 0x7fa51806c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:09.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.302+0000 7fa51d400640 1 -- 172.21.15.27:0/2158113469 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa5180b2360 msgr2=0x7fa51806c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:09.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.302+0000 7fa51d400640 1 --2- 172.21.15.27:0/2158113469 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa5180b2360 0x7fa51806c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:09.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.302+0000 7fa51d400640 1 -- 172.21.15.27:0/2158113469 >> v2:172.21.15.93:3300/0 conn(0x7fa5180cf3c0 msgr2=0x7fa518160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:09.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.302+0000 7fa51d400640 1 --2- 172.21.15.27:0/2158113469 >> v2:172.21.15.93:3300/0 conn(0x7fa5180cf3c0 0x7fa518160f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:09.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.302+0000 7fa51d400640 1 -- 172.21.15.27:0/2158113469 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5180cfe10 con 0x7fa51806ca00 2024-01-15T18:09:09.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.302+0000 7fa51ca00640 1 --2- 172.21.15.27:0/2158113469 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa5180b2360 0x7fa51806c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:09.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.302+0000 7fa517e00640 1 --2- 172.21.15.27:0/2158113469 >> v2:172.21.15.93:3300/0 conn(0x7fa5180cf3c0 0x7fa518160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:09.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.306+0000 7fa51d400640 1 --2- 172.21.15.27:0/2158113469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa51806ca00 0x7fa518161470 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fa518071470 tx=0x7fa51001c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3aa6dff85914605 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:09.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.306+0000 7fa517400640 1 -- 172.21.15.27:0/2158113469 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5100a5720 con 0x7fa51806ca00 2024-01-15T18:09:09.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.306+0000 7fa517400640 1 -- 172.21.15.27:0/2158113469 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5100a5880 con 0x7fa51806ca00 2024-01-15T18:09:09.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.306+0000 7fa517400640 1 -- 172.21.15.27:0/2158113469 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5100a5c10 con 0x7fa51806ca00 2024-01-15T18:09:09.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.306+0000 7fa51f400640 1 -- 172.21.15.27:0/2158113469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa51806ca00 msgr2=0x7fa518161470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:09.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.306+0000 7fa51f400640 1 --2- 172.21.15.27:0/2158113469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa51806ca00 0x7fa518161470 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fa518071470 tx=0x7fa51001c9b0 comp rx=0 tx=0).stop 2024-01-15T18:09:09.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.306+0000 7fa51f400640 1 -- 172.21.15.27:0/2158113469 shutdown_connections 2024-01-15T18:09:09.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.306+0000 7fa51f400640 1 --2- 172.21.15.27:0/2158113469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa51806ca00 0x7fa518161470 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:09.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.306+0000 7fa51f400640 1 --2- 172.21.15.27:0/2158113469 >> v2:172.21.15.93:3300/0 conn(0x7fa5180cf3c0 0x7fa518160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:09.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.306+0000 7fa51f400640 1 --2- 172.21.15.27:0/2158113469 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa5180b2360 0x7fa51806c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:09.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.306+0000 7fa51f400640 1 -- 172.21.15.27:0/2158113469 >> 172.21.15.27:0/2158113469 conn(0x7fa5180d0e70 msgr2=0x7fa5180d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:09.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.306+0000 7fa51f400640 1 -- 172.21.15.27:0/2158113469 shutdown_connections 2024-01-15T18:09:09.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.306+0000 7fa51f400640 1 -- 172.21.15.27:0/2158113469 wait complete. 2024-01-15T18:09:09.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.306+0000 7fa51f400640 1 Processor -- start 2024-01-15T18:09:09.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.306+0000 7fa51f400640 1 -- start start 2024-01-15T18:09:09.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.310+0000 7fa51f400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa51806ca00 0x7fa518151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:09.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.310+0000 7fa51f400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5180b2360 0x7fa518156cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:09.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.310+0000 7fa51f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa518158eb0 0x7fa518153f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:09.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.310+0000 7fa51f400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5181524e0 con 0x7fa518158eb0 2024-01-15T18:09:09.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.310+0000 7fa51f400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5181527c0 con 0x7fa5180b2360 2024-01-15T18:09:09.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.310+0000 7fa51f400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa518152aa0 con 0x7fa51806ca00 2024-01-15T18:09:09.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.310+0000 7fa51d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa518158eb0 0x7fa518153f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:09.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.310+0000 7fa51d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa518158eb0 0x7fa518153f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43816/0 (socket says 172.21.15.27:43816) 2024-01-15T18:09:09.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.310+0000 7fa51d400640 1 -- 172.21.15.27:0/1740256084 learned_addr learned my addr 172.21.15.27:0/1740256084 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:09.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.310+0000 7fa51ca00640 1 --2- 172.21.15.27:0/1740256084 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa51806ca00 0x7fa518151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:09.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.310+0000 7fa517e00640 1 --2- 172.21.15.27:0/1740256084 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5180b2360 0x7fa518156cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:09.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.310+0000 7fa51d400640 1 -- 172.21.15.27:0/1740256084 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa51806ca00 msgr2=0x7fa518151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:09.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.310+0000 7fa51d400640 1 --2- 172.21.15.27:0/1740256084 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa51806ca00 0x7fa518151f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:09.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.310+0000 7fa51d400640 1 -- 172.21.15.27:0/1740256084 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5180b2360 msgr2=0x7fa518156cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:09.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.310+0000 7fa51d400640 1 --2- 172.21.15.27:0/1740256084 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5180b2360 0x7fa518156cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:09.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.310+0000 7fa51d400640 1 -- 172.21.15.27:0/1740256084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa510009b30 con 0x7fa518158eb0 2024-01-15T18:09:09.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.310+0000 7fa517e00640 1 --2- 172.21.15.27:0/1740256084 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5180b2360 0x7fa518156cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:09:09.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.310+0000 7fa51d400640 1 --2- 172.21.15.27:0/1740256084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa518158eb0 0x7fa518153f10 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fa5100af040 tx=0x7fa510019750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:09.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.310+0000 7fa514c00640 1 -- 172.21.15.27:0/1740256084 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5100b8670 con 0x7fa518158eb0 2024-01-15T18:09:09.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.310+0000 7fa514c00640 1 -- 172.21.15.27:0/1740256084 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5100b87d0 con 0x7fa518158eb0 2024-01-15T18:09:09.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.310+0000 7fa514c00640 1 -- 172.21.15.27:0/1740256084 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5100b8b80 con 0x7fa518158eb0 2024-01-15T18:09:09.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.310+0000 7fa51f400640 1 -- 172.21.15.27:0/1740256084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa518152d80 con 0x7fa518158eb0 2024-01-15T18:09:09.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.310+0000 7fa51f400640 1 -- 172.21.15.27:0/1740256084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa518157200 con 0x7fa518158eb0 2024-01-15T18:09:09.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.310+0000 7fa51f400640 1 -- 172.21.15.27:0/1740256084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5181532a0 con 0x7fa518158eb0 2024-01-15T18:09:09.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.314+0000 7fa514c00640 1 -- 172.21.15.27:0/1740256084 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7fa5100b8ce0 con 0x7fa518158eb0 2024-01-15T18:09:09.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.314+0000 7fa514c00640 1 --2- 172.21.15.27:0/1740256084 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa4f0066e90 0x7fa4f0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:09.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.314+0000 7fa514c00640 1 -- 172.21.15.27:0/1740256084 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5637+0+0 (secure 0 0 0) 0x7fa510120e00 con 0x7fa518158eb0 2024-01-15T18:09:09.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.314+0000 7fa51ca00640 1 --2- 172.21.15.27:0/1740256084 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa4f0066e90 0x7fa4f0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:09.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.314+0000 7fa51ca00640 1 --2- 172.21.15.27:0/1740256084 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa4f0066e90 0x7fa4f0069350 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa508006b20 tx=0x7fa508008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:09.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.318+0000 7fa514c00640 1 -- 172.21.15.27:0/1740256084 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa5100ed9a0 con 0x7fa518158eb0 2024-01-15T18:09:09.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:09.506+0000 7fa51f400640 1 -- 172.21.15.27:0/1740256084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa5181532a0 con 0x7fa518158eb0 2024-01-15T18:09:10.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.138+0000 7fa514c00640 1 -- 172.21.15.27:0/1740256084 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v33) v1 ==== 172+0+0 (secure 0 0 0) 0x7fa5181532a0 con 0x7fa518158eb0 2024-01-15T18:09:10.150 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 8 and data pool 9 2024-01-15T18:09:10.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.150+0000 7fa51f400640 1 -- 172.21.15.27:0/1740256084 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa4f0066e90 msgr2=0x7fa4f0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:10.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.150+0000 7fa51f400640 1 --2- 172.21.15.27:0/1740256084 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa4f0066e90 0x7fa4f0069350 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa508006b20 tx=0x7fa508008040 comp rx=0 tx=0).stop 2024-01-15T18:09:10.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.150+0000 7fa51f400640 1 -- 172.21.15.27:0/1740256084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa518158eb0 msgr2=0x7fa518153f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:10.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.150+0000 7fa51f400640 1 --2- 172.21.15.27:0/1740256084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa518158eb0 0x7fa518153f10 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fa5100af040 tx=0x7fa510019750 comp rx=0 tx=0).stop 2024-01-15T18:09:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.154+0000 7fa51f400640 1 -- 172.21.15.27:0/1740256084 shutdown_connections 2024-01-15T18:09:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.154+0000 7fa51f400640 1 --2- 172.21.15.27:0/1740256084 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa4f0066e90 0x7fa4f0069350 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.154+0000 7fa51f400640 1 --2- 172.21.15.27:0/1740256084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa518158eb0 0x7fa518153f10 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.154+0000 7fa51f400640 1 --2- 172.21.15.27:0/1740256084 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5180b2360 0x7fa518156cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.154+0000 7fa51f400640 1 --2- 172.21.15.27:0/1740256084 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa51806ca00 0x7fa518151f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.154+0000 7fa51f400640 1 -- 172.21.15.27:0/1740256084 >> 172.21.15.27:0/1740256084 conn(0x7fa5180d0e70 msgr2=0x7fa5180cbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:10.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.162+0000 7fa51f400640 1 -- 172.21.15.27:0/1740256084 shutdown_connections 2024-01-15T18:09:10.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.162+0000 7fa51f400640 1 -- 172.21.15.27:0/1740256084 wait complete. 2024-01-15T18:09:10.198 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:09:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.334+0000 7fdd7b400640 1 Processor -- start 2024-01-15T18:09:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.334+0000 7fdd7b400640 1 -- start start 2024-01-15T18:09:10.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.334+0000 7fdd7b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdd740c92b0 0x7fdd7415d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:10.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.334+0000 7fdd7b400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd7406c4a0 con 0x7fdd740701a0 2024-01-15T18:09:10.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.334+0000 7fdd7b400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd7415c660 con 0x7fdd7415c130 2024-01-15T18:09:10.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.334+0000 7fdd7b400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd7415c940 con 0x7fdd740c92b0 2024-01-15T18:09:10.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.334+0000 7fdd79400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdd740c92b0 0x7fdd7415d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:10.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.334+0000 7fdd79400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdd740c92b0 0x7fdd7415d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43818/0 (socket says 172.21.15.27:43818) 2024-01-15T18:09:10.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.334+0000 7fdd79400640 1 -- 172.21.15.27:0/659435223 learned_addr learned my addr 172.21.15.27:0/659435223 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:10.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.334+0000 7fdd78a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdd740701a0 0x7fdd74070580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54034/0 (socket says 172.21.15.27:54034) 2024-01-15T18:09:10.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.338+0000 7fdd79400640 1 -- 172.21.15.27:0/659435223 >> v1:172.21.15.93:6789/0 conn(0x7fdd7415c130 legacy=0x7fdd7406dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:10.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.338+0000 7fdd79400640 1 -- 172.21.15.27:0/659435223 >> v1:172.21.15.27:6789/0 conn(0x7fdd740701a0 legacy=0x7fdd74070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:10.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.338+0000 7fdd79400640 1 -- 172.21.15.27:0/659435223 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd7415cc20 con 0x7fdd740c92b0 2024-01-15T18:09:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.338+0000 7fdd79400640 1 --2- 172.21.15.27:0/659435223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740c92b0 0x7fdd7415d9e0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c009e50 tx=0x7fdd6c01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1263bbee5a3172d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.338+0000 7fdd73400640 1 -- 172.21.15.27:0/659435223 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c0a5810 con 0x7fdd740c92b0 2024-01-15T18:09:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.338+0000 7fdd73400640 1 -- 172.21.15.27:0/659435223 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd6c0a5970 con 0x7fdd740c92b0 2024-01-15T18:09:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.338+0000 7fdd73400640 1 -- 172.21.15.27:0/659435223 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c0a5d00 con 0x7fdd740c92b0 2024-01-15T18:09:10.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.338+0000 7fdd7b400640 1 -- 172.21.15.27:0/659435223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740c92b0 msgr2=0x7fdd7415d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:10.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.338+0000 7fdd7b400640 1 --2- 172.21.15.27:0/659435223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740c92b0 0x7fdd7415d9e0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c009e50 tx=0x7fdd6c01c950 comp rx=0 tx=0).stop 2024-01-15T18:09:10.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.342+0000 7fdd7b400640 1 -- 172.21.15.27:0/659435223 shutdown_connections 2024-01-15T18:09:10.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.342+0000 7fdd7b400640 1 --2- 172.21.15.27:0/659435223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740c92b0 0x7fdd7415d9e0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:10.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.342+0000 7fdd7b400640 1 -- 172.21.15.27:0/659435223 >> 172.21.15.27:0/659435223 conn(0x7fdd740cec60 msgr2=0x7fdd740cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:10.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.342+0000 7fdd7b400640 1 -- 172.21.15.27:0/659435223 shutdown_connections 2024-01-15T18:09:10.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.342+0000 7fdd7b400640 1 -- 172.21.15.27:0/659435223 wait complete. 2024-01-15T18:09:10.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.342+0000 7fdd7b400640 1 Processor -- start 2024-01-15T18:09:10.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.342+0000 7fdd7b400640 1 -- start start 2024-01-15T18:09:10.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.342+0000 7fdd7b400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd740701a0 0x7fdd740cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:10.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.342+0000 7fdd7b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740c92b0 0x7fdd740ce140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:10.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.342+0000 7fdd7b400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd7415c130 0x7fdd740cc830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:10.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.342+0000 7fdd7b400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd74158290 con 0x7fdd740c92b0 2024-01-15T18:09:10.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.342+0000 7fdd7b400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd741590d0 con 0x7fdd740701a0 2024-01-15T18:09:10.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.342+0000 7fdd7b400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd741584c0 con 0x7fdd7415c130 2024-01-15T18:09:10.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.342+0000 7fdd73e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740c92b0 0x7fdd740ce140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:10.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.342+0000 7fdd73e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740c92b0 0x7fdd740ce140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43824/0 (socket says 172.21.15.27:43824) 2024-01-15T18:09:10.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.342+0000 7fdd73e00640 1 -- 172.21.15.27:0/4026215351 learned_addr learned my addr 172.21.15.27:0/4026215351 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:10.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.342+0000 7fdd79400640 1 --2- 172.21.15.27:0/4026215351 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd7415c130 0x7fdd740cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:10.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.342+0000 7fdd78a00640 1 --2- 172.21.15.27:0/4026215351 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd740701a0 0x7fdd740cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:10.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.342+0000 7fdd73e00640 1 -- 172.21.15.27:0/4026215351 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd7415c130 msgr2=0x7fdd740cc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:10.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.342+0000 7fdd73e00640 1 --2- 172.21.15.27:0/4026215351 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd7415c130 0x7fdd740cc830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:10.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.342+0000 7fdd73e00640 1 -- 172.21.15.27:0/4026215351 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd740701a0 msgr2=0x7fdd740cdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:10.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.342+0000 7fdd73e00640 1 --2- 172.21.15.27:0/4026215351 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd740701a0 0x7fdd740cdc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:10.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.342+0000 7fdd73e00640 1 -- 172.21.15.27:0/4026215351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd6c009b30 con 0x7fdd740c92b0 2024-01-15T18:09:10.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.346+0000 7fdd73e00640 1 --2- 172.21.15.27:0/4026215351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740c92b0 0x7fdd740ce140 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c00c340 tx=0x7fdd5c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:10.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.346+0000 7fdd70c00640 1 -- 172.21.15.27:0/4026215351 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c0052d0 con 0x7fdd740c92b0 2024-01-15T18:09:10.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.346+0000 7fdd7b400640 1 -- 172.21.15.27:0/4026215351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd741587a0 con 0x7fdd740c92b0 2024-01-15T18:09:10.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.346+0000 7fdd70c00640 1 -- 172.21.15.27:0/4026215351 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5c005430 con 0x7fdd740c92b0 2024-01-15T18:09:10.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.346+0000 7fdd70c00640 1 -- 172.21.15.27:0/4026215351 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c007260 con 0x7fdd740c92b0 2024-01-15T18:09:10.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.346+0000 7fdd7b400640 1 -- 172.21.15.27:0/4026215351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd74158d60 con 0x7fdd740c92b0 2024-01-15T18:09:10.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.346+0000 7fdd7b400640 1 -- 172.21.15.27:0/4026215351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd740af470 con 0x7fdd740c92b0 2024-01-15T18:09:10.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.346+0000 7fdd70c00640 1 -- 172.21.15.27:0/4026215351 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7fdd5c0073c0 con 0x7fdd740c92b0 2024-01-15T18:09:10.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.346+0000 7fdd70c00640 1 --2- 172.21.15.27:0/4026215351 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd4c066ec0 0x7fdd4c069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:10.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.346+0000 7fdd70c00640 1 -- 172.21.15.27:0/4026215351 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fdd5c0841f0 con 0x7fdd740c92b0 2024-01-15T18:09:10.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.346+0000 7fdd78a00640 1 --2- 172.21.15.27:0/4026215351 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd4c066ec0 0x7fdd4c069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:10.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.350+0000 7fdd78a00640 1 --2- 172.21.15.27:0/4026215351 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd4c066ec0 0x7fdd4c069380 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fdd64000f70 tx=0x7fdd64007e50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:10.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.350+0000 7fdd70c00640 1 -- 172.21.15.27:0/4026215351 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fdd5c050b90 con 0x7fdd740c92b0 2024-01-15T18:09:10.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.510+0000 7fdd7b400640 1 -- 172.21.15.27:0/4026215351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd7404e850 con 0x7fdd740c92b0 2024-01-15T18:09:10.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.514+0000 7fdd70c00640 1 -- 172.21.15.27:0/4026215351 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16525 (secure 0 0 0) 0x7fdd5c0541b0 con 0x7fdd740c92b0 2024-01-15T18:09:10.518 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:09:10.519 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":57,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:09:09.513142+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:09:07.910453+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-15T18:09:08.507842+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:09:10.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.514+0000 7fdd7b400640 1 -- 172.21.15.27:0/4026215351 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd4c066ec0 msgr2=0x7fdd4c069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:10.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.514+0000 7fdd7b400640 1 --2- 172.21.15.27:0/4026215351 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd4c066ec0 0x7fdd4c069380 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fdd64000f70 tx=0x7fdd64007e50 comp rx=0 tx=0).stop 2024-01-15T18:09:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.514+0000 7fdd7b400640 1 -- 172.21.15.27:0/4026215351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740c92b0 msgr2=0x7fdd740ce140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.514+0000 7fdd7b400640 1 --2- 172.21.15.27:0/4026215351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740c92b0 0x7fdd740ce140 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c00c340 tx=0x7fdd5c00c800 comp rx=0 tx=0).stop 2024-01-15T18:09:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.514+0000 7fdd7b400640 1 -- 172.21.15.27:0/4026215351 shutdown_connections 2024-01-15T18:09:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.514+0000 7fdd7b400640 1 --2- 172.21.15.27:0/4026215351 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd4c066ec0 0x7fdd4c069380 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.514+0000 7fdd7b400640 1 --2- 172.21.15.27:0/4026215351 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd7415c130 0x7fdd740cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.514+0000 7fdd7b400640 1 --2- 172.21.15.27:0/4026215351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740c92b0 0x7fdd740ce140 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.514+0000 7fdd7b400640 1 --2- 172.21.15.27:0/4026215351 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd740701a0 0x7fdd740cdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.514+0000 7fdd7b400640 1 -- 172.21.15.27:0/4026215351 >> 172.21.15.27:0/4026215351 conn(0x7fdd740cec60 msgr2=0x7fdd74071c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.514+0000 7fdd7b400640 1 -- 172.21.15.27:0/4026215351 shutdown_connections 2024-01-15T18:09:10.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.514+0000 7fdd7b400640 1 -- 172.21.15.27:0/4026215351 wait complete. 2024-01-15T18:09:10.541 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:09:10.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.674+0000 7f3bc4800640 1 Processor -- start 2024-01-15T18:09:10.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.674+0000 7f3bc4800640 1 -- start start 2024-01-15T18:09:10.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.678+0000 7f3bc4800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3bbc15e7d0 0x7f3bbc0cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.678+0000 7f3bc4800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f3bbc0cedc0 0x7f3bbc0cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.678+0000 7f3bc4800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bbc0b38b0 con 0x7f3bbc0ce2e0 2024-01-15T18:09:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.678+0000 7f3bc4800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bbc0cd550 con 0x7f3bbc15e7d0 2024-01-15T18:09:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.678+0000 7f3bc4800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bbc0cd270 con 0x7f3bbc0cedc0 2024-01-15T18:09:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.678+0000 7f3bc1e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3bbc15e7d0 0x7f3bbc0cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.678+0000 7f3bc2800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3bbc0ce2e0 0x7f3bbc0cba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54036/0 (socket says 172.21.15.27:54036) 2024-01-15T18:09:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.678+0000 7f3bc1e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3bbc15e7d0 0x7f3bbc0cf2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43832/0 (socket says 172.21.15.27:43832) 2024-01-15T18:09:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.678+0000 7f3bc2800640 1 -- 172.21.15.27:0/528486353 learned_addr learned my addr 172.21.15.27:0/528486353 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.678+0000 7f3bc1400640 1 --2- 172.21.15.27:0/528486353 >> v2:172.21.15.93:3300/0 conn(0x7f3bbc0cedc0 0x7f3bbc0cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.678+0000 7f3bc1e00640 1 -- 172.21.15.27:0/528486353 >> v2:172.21.15.93:3300/0 conn(0x7f3bbc0cedc0 msgr2=0x7f3bbc0cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.678+0000 7f3bc1e00640 1 --2- 172.21.15.27:0/528486353 >> v2:172.21.15.93:3300/0 conn(0x7f3bbc0cedc0 0x7f3bbc0cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.678+0000 7f3bc1e00640 1 -- 172.21.15.27:0/528486353 >> v1:172.21.15.27:6789/0 conn(0x7f3bbc0ce2e0 legacy=0x7f3bbc0cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:10.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.678+0000 7f3bc1e00640 1 -- 172.21.15.27:0/528486353 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bbc0b35d0 con 0x7f3bbc15e7d0 2024-01-15T18:09:10.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.682+0000 7f3bc1e00640 1 --2- 172.21.15.27:0/528486353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bbc15e7d0 0x7f3bbc0cf2a0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc0718a0 tx=0x7f3bac01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=607af26bcd98c0c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:10.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.682+0000 7f3bc0a00640 1 -- 172.21.15.27:0/528486353 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac0ab600 con 0x7f3bbc15e7d0 2024-01-15T18:09:10.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.682+0000 7f3bc0a00640 1 -- 172.21.15.27:0/528486353 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bac0a7050 con 0x7f3bbc15e7d0 2024-01-15T18:09:10.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.682+0000 7f3bc0a00640 1 -- 172.21.15.27:0/528486353 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac0ab980 con 0x7f3bbc15e7d0 2024-01-15T18:09:10.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.682+0000 7f3bc4800640 1 -- 172.21.15.27:0/528486353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bbc15e7d0 msgr2=0x7f3bbc0cf2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:10.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.682+0000 7f3bc4800640 1 --2- 172.21.15.27:0/528486353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bbc15e7d0 0x7f3bbc0cf2a0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc0718a0 tx=0x7f3bac01c3d0 comp rx=0 tx=0).stop 2024-01-15T18:09:10.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.682+0000 7f3bc4800640 1 -- 172.21.15.27:0/528486353 shutdown_connections 2024-01-15T18:09:10.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.682+0000 7f3bc4800640 1 --2- 172.21.15.27:0/528486353 >> v2:172.21.15.93:3300/0 conn(0x7f3bbc0cedc0 0x7f3bbc0cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:10.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.682+0000 7f3bc4800640 1 --2- 172.21.15.27:0/528486353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bbc15e7d0 0x7f3bbc0cf2a0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:10.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.682+0000 7f3bc4800640 1 -- 172.21.15.27:0/528486353 >> 172.21.15.27:0/528486353 conn(0x7f3bbc0b2a10 msgr2=0x7f3bbc0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:10.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.682+0000 7f3bc4800640 1 -- 172.21.15.27:0/528486353 shutdown_connections 2024-01-15T18:09:10.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.682+0000 7f3bc4800640 1 -- 172.21.15.27:0/528486353 wait complete. 2024-01-15T18:09:10.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.682+0000 7f3bc4800640 1 Processor -- start 2024-01-15T18:09:10.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.682+0000 7f3bc4800640 1 -- start start 2024-01-15T18:09:10.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.686+0000 7f3bc4800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3bbc0ce2e0 0x7f3bbc153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:10.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.686+0000 7f3bc4800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bbc0cedc0 0x7f3bbc06c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:10.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.686+0000 7f3bc4800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3bbc15e7d0 0x7f3bbc0aed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:10.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.686+0000 7f3bc4800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc06d4a0 con 0x7f3bbc0cedc0 2024-01-15T18:09:10.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.686+0000 7f3bc4800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc0c8c50 con 0x7f3bbc0ce2e0 2024-01-15T18:09:10.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.686+0000 7f3bc4800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc06b9e0 con 0x7f3bbc15e7d0 2024-01-15T18:09:10.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.686+0000 7f3bc1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bbc0cedc0 0x7f3bbc06c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:10.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.686+0000 7f3bc1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bbc0cedc0 0x7f3bbc06c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43838/0 (socket says 172.21.15.27:43838) 2024-01-15T18:09:10.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.686+0000 7f3bc1400640 1 -- 172.21.15.27:0/3222537881 learned_addr learned my addr 172.21.15.27:0/3222537881 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:10.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.686+0000 7f3bc1e00640 1 --2- 172.21.15.27:0/3222537881 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3bbc0ce2e0 0x7f3bbc153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:10.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.686+0000 7f3bc2800640 1 --2- 172.21.15.27:0/3222537881 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3bbc15e7d0 0x7f3bbc0aed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:10.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.686+0000 7f3bc1400640 1 -- 172.21.15.27:0/3222537881 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3bbc15e7d0 msgr2=0x7f3bbc0aed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:10.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.686+0000 7f3bc1400640 1 --2- 172.21.15.27:0/3222537881 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3bbc15e7d0 0x7f3bbc0aed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:10.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.686+0000 7f3bc1400640 1 -- 172.21.15.27:0/3222537881 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3bbc0ce2e0 msgr2=0x7f3bbc153fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:10.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.686+0000 7f3bc1400640 1 --2- 172.21.15.27:0/3222537881 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3bbc0ce2e0 0x7f3bbc153fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:10.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.686+0000 7f3bc1400640 1 -- 172.21.15.27:0/3222537881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bac009420 con 0x7f3bbc0cedc0 2024-01-15T18:09:10.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.686+0000 7f3bc1e00640 1 --2- 172.21.15.27:0/3222537881 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3bbc0ce2e0 0x7f3bbc153fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:09:10.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.686+0000 7f3bc1400640 1 --2- 172.21.15.27:0/3222537881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bbc0cedc0 0x7f3bbc06c560 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f3ba400d5f0 tx=0x7f3ba400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:10.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.686+0000 7f3bba000640 1 -- 172.21.15.27:0/3222537881 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba40125f0 con 0x7f3bbc0cedc0 2024-01-15T18:09:10.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.686+0000 7f3bba000640 1 -- 172.21.15.27:0/3222537881 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba4012750 con 0x7f3bbc0cedc0 2024-01-15T18:09:10.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.686+0000 7f3bc4800640 1 -- 172.21.15.27:0/3222537881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bbc06bcc0 con 0x7f3bbc0cedc0 2024-01-15T18:09:10.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.686+0000 7f3bba000640 1 -- 172.21.15.27:0/3222537881 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4012ae0 con 0x7f3bbc0cedc0 2024-01-15T18:09:10.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.686+0000 7f3bc4800640 1 -- 172.21.15.27:0/3222537881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bbc06c280 con 0x7f3bbc0cedc0 2024-01-15T18:09:10.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.686+0000 7f3bc4800640 1 -- 172.21.15.27:0/3222537881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bbc04e850 con 0x7f3bbc0cedc0 2024-01-15T18:09:10.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.690+0000 7f3bba000640 1 -- 172.21.15.27:0/3222537881 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f3ba4012c40 con 0x7f3bbc0cedc0 2024-01-15T18:09:10.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.690+0000 7f3bba000640 1 --2- 172.21.15.27:0/3222537881 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3b94066e90 0x7f3b94069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:10.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.690+0000 7f3bba000640 1 -- 172.21.15.27:0/3222537881 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f3ba4085440 con 0x7f3bbc0cedc0 2024-01-15T18:09:10.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.690+0000 7f3bc1e00640 1 --2- 172.21.15.27:0/3222537881 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3b94066e90 0x7f3b94069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:10.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.690+0000 7f3bc1e00640 1 --2- 172.21.15.27:0/3222537881 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3b94066e90 0x7f3b94069350 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3bac0ad000 tx=0x7f3bac008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:10.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.694+0000 7f3bba000640 1 -- 172.21.15.27:0/3222537881 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3ba4051d70 con 0x7f3bbc0cedc0 2024-01-15T18:09:10.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.850+0000 7f3bc4800640 1 -- 172.21.15.27:0/3222537881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3bbc067070 con 0x7f3bbc0cedc0 2024-01-15T18:09:10.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.850+0000 7f3bba000640 1 -- 172.21.15.27:0/3222537881 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+16525 (secure 0 0 0) 0x7f3ba4007380 con 0x7f3bbc0cedc0 2024-01-15T18:09:10.856 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:09:10.857 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":57,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:09:09.513142+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:09:07.910453+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-15T18:09:08.507842+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:09:10.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.854+0000 7f3bc4800640 1 -- 172.21.15.27:0/3222537881 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3b94066e90 msgr2=0x7f3b94069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:10.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.854+0000 7f3bc4800640 1 --2- 172.21.15.27:0/3222537881 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3b94066e90 0x7f3b94069350 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3bac0ad000 tx=0x7f3bac008040 comp rx=0 tx=0).stop 2024-01-15T18:09:10.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.854+0000 7f3bc4800640 1 -- 172.21.15.27:0/3222537881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bbc0cedc0 msgr2=0x7f3bbc06c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:10.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.854+0000 7f3bc4800640 1 --2- 172.21.15.27:0/3222537881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bbc0cedc0 0x7f3bbc06c560 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f3ba400d5f0 tx=0x7f3ba400dab0 comp rx=0 tx=0).stop 2024-01-15T18:09:10.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.854+0000 7f3bc4800640 1 -- 172.21.15.27:0/3222537881 shutdown_connections 2024-01-15T18:09:10.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.854+0000 7f3bc4800640 1 --2- 172.21.15.27:0/3222537881 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3b94066e90 0x7f3b94069350 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:10.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.854+0000 7f3bc4800640 1 --2- 172.21.15.27:0/3222537881 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3bbc15e7d0 0x7f3bbc0aed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:10.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.854+0000 7f3bc4800640 1 --2- 172.21.15.27:0/3222537881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3bbc0cedc0 0x7f3bbc06c560 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:10.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.854+0000 7f3bc4800640 1 --2- 172.21.15.27:0/3222537881 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3bbc0ce2e0 0x7f3bbc153fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:10.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.854+0000 7f3bc4800640 1 -- 172.21.15.27:0/3222537881 >> 172.21.15.27:0/3222537881 conn(0x7f3bbc0b2a10 msgr2=0x7f3bbc15b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:10.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.854+0000 7f3bc4800640 1 -- 172.21.15.27:0/3222537881 shutdown_connections 2024-01-15T18:09:10.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:10.854+0000 7f3bc4800640 1 -- 172.21.15.27:0/3222537881 wait complete. 2024-01-15T18:09:10.880 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-15T18:09:11.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.018+0000 7f55df200640 1 Processor -- start 2024-01-15T18:09:11.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.018+0000 7f55df200640 1 -- start start 2024-01-15T18:09:11.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.018+0000 7f55df200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f55d8099a10 0x7f55d8099e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:11.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.018+0000 7f55df200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55d809a360 con 0x7f55d815b350 2024-01-15T18:09:11.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.018+0000 7f55df200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55d80ccd50 con 0x7f55d809a5b0 2024-01-15T18:09:11.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.018+0000 7f55df200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55d80cebc0 con 0x7f55d8099a10 2024-01-15T18:09:11.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.018+0000 7f55d7e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f55d815b350 0x7f55d80cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54050/0 (socket says 172.21.15.27:54050) 2024-01-15T18:09:11.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.018+0000 7f55d7e00640 1 -- 172.21.15.27:0/3356106834 learned_addr learned my addr 172.21.15.27:0/3356106834 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:11.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.018+0000 7f55dd200640 1 --2- 172.21.15.27:0/3356106834 >> v2:172.21.15.93:3300/0 conn(0x7f55d8099a10 0x7f55d8099e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:11.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.022+0000 7f55d6a00640 1 -- 172.21.15.27:0/3356106834 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 316232138 0 0) 0x7f55d809a360 con 0x7f55d815b350 2024-01-15T18:09:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.022+0000 7f55d6a00640 1 -- 172.21.15.27:0/3356106834 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55b8003530 con 0x7f55d815b350 2024-01-15T18:09:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.022+0000 7f55d6a00640 1 -- 172.21.15.27:0/3356106834 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1605781498 0 0) 0x7f55d80ccd50 con 0x7f55d809a5b0 2024-01-15T18:09:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.022+0000 7f55d6a00640 1 -- 172.21.15.27:0/3356106834 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55d809a360 con 0x7f55d809a5b0 2024-01-15T18:09:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.022+0000 7f55dd200640 1 -- 172.21.15.27:0/3356106834 >> v1:172.21.15.93:6789/0 conn(0x7f55d809a5b0 legacy=0x7f55d80cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.022+0000 7f55dd200640 1 -- 172.21.15.27:0/3356106834 >> v1:172.21.15.27:6789/0 conn(0x7f55d815b350 legacy=0x7f55d80cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:11.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.022+0000 7f55dd200640 1 -- 172.21.15.27:0/3356106834 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d80ceea0 con 0x7f55d8099a10 2024-01-15T18:09:11.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.022+0000 7f55dd200640 1 --2- 172.21.15.27:0/3356106834 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55d8099a10 0x7f55d8099e20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f55d0009d90 tx=0x7f55d001c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0fecd3decb07e9b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:11.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.022+0000 7f55d6a00640 1 -- 172.21.15.27:0/3356106834 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d00a5970 con 0x7f55d8099a10 2024-01-15T18:09:11.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.022+0000 7f55d6a00640 1 -- 172.21.15.27:0/3356106834 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55d00a5ad0 con 0x7f55d8099a10 2024-01-15T18:09:11.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.022+0000 7f55d6a00640 1 -- 172.21.15.27:0/3356106834 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d00a5e60 con 0x7f55d8099a10 2024-01-15T18:09:11.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.026+0000 7f55df200640 1 -- 172.21.15.27:0/3356106834 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55d8099a10 msgr2=0x7f55d8099e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:11.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.026+0000 7f55df200640 1 --2- 172.21.15.27:0/3356106834 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55d8099a10 0x7f55d8099e20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f55d0009d90 tx=0x7f55d001c890 comp rx=0 tx=0).stop 2024-01-15T18:09:11.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.026+0000 7f55df200640 1 -- 172.21.15.27:0/3356106834 shutdown_connections 2024-01-15T18:09:11.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.026+0000 7f55df200640 1 --2- 172.21.15.27:0/3356106834 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55d8099a10 0x7f55d8099e20 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:11.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.026+0000 7f55df200640 1 -- 172.21.15.27:0/3356106834 >> 172.21.15.27:0/3356106834 conn(0x7f55d80b2a80 msgr2=0x7f55d80b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:11.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.026+0000 7f55df200640 1 -- 172.21.15.27:0/3356106834 shutdown_connections 2024-01-15T18:09:11.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.026+0000 7f55df200640 1 -- 172.21.15.27:0/3356106834 wait complete. 2024-01-15T18:09:11.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.026+0000 7f55df200640 1 Processor -- start 2024-01-15T18:09:11.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.026+0000 7f55df200640 1 -- start start 2024-01-15T18:09:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.026+0000 7f55df200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55d8099a10 0x7f55d80b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.026+0000 7f55df200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55d809a5b0 0x7f55d806eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.026+0000 7f55df200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f55d815b350 0x7f55d8152f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:11.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.026+0000 7f55df200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d80b57b0 con 0x7f55d8099a10 2024-01-15T18:09:11.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.026+0000 7f55df200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d80b4ba0 con 0x7f55d809a5b0 2024-01-15T18:09:11.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.026+0000 7f55df200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d80b4e80 con 0x7f55d815b350 2024-01-15T18:09:11.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.026+0000 7f55d7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55d8099a10 0x7f55d80b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:11.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.026+0000 7f55d7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55d8099a10 0x7f55d80b5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43842/0 (socket says 172.21.15.27:43842) 2024-01-15T18:09:11.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.026+0000 7f55d7e00640 1 -- 172.21.15.27:0/2773352870 learned_addr learned my addr 172.21.15.27:0/2773352870 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:11.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.026+0000 7f55dd200640 1 --2- 172.21.15.27:0/2773352870 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f55d815b350 0x7f55d8152f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:11.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.026+0000 7f55d7400640 1 --2- 172.21.15.27:0/2773352870 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55d809a5b0 0x7f55d806eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:11.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.026+0000 7f55d7e00640 1 -- 172.21.15.27:0/2773352870 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f55d815b350 msgr2=0x7f55d8152f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:11.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.026+0000 7f55d7e00640 1 --2- 172.21.15.27:0/2773352870 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f55d815b350 0x7f55d8152f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:11.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.026+0000 7f55d7e00640 1 -- 172.21.15.27:0/2773352870 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55d809a5b0 msgr2=0x7f55d806eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:11.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.026+0000 7f55d7e00640 1 --2- 172.21.15.27:0/2773352870 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55d809a5b0 0x7f55d806eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:11.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.026+0000 7f55d7e00640 1 -- 172.21.15.27:0/2773352870 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d0009a70 con 0x7f55d8099a10 2024-01-15T18:09:11.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.030+0000 7f55d7e00640 1 --2- 172.21.15.27:0/2773352870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55d8099a10 0x7f55d80b5c40 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f55c800c420 tx=0x7f55c800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:11.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.030+0000 7f55cfe00640 1 -- 172.21.15.27:0/2773352870 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c8005cf0 con 0x7f55d8099a10 2024-01-15T18:09:11.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.030+0000 7f55cfe00640 1 -- 172.21.15.27:0/2773352870 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c8005e50 con 0x7f55d8099a10 2024-01-15T18:09:11.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.030+0000 7f55cfe00640 1 -- 172.21.15.27:0/2773352870 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c8007310 con 0x7f55d8099a10 2024-01-15T18:09:11.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.030+0000 7f55df200640 1 -- 172.21.15.27:0/2773352870 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55d80b5160 con 0x7f55d8099a10 2024-01-15T18:09:11.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.030+0000 7f55df200640 1 -- 172.21.15.27:0/2773352870 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55d815d8c0 con 0x7f55d8099a10 2024-01-15T18:09:11.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.030+0000 7f55df200640 1 -- 172.21.15.27:0/2773352870 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55d804e850 con 0x7f55d8099a10 2024-01-15T18:09:11.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.030+0000 7f55cfe00640 1 -- 172.21.15.27:0/2773352870 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f55c8007470 con 0x7f55d8099a10 2024-01-15T18:09:11.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.030+0000 7f55cfe00640 1 --2- 172.21.15.27:0/2773352870 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f55b0066ec0 0x7f55b0069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:11.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.030+0000 7f55cfe00640 1 -- 172.21.15.27:0/2773352870 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f55c8084370 con 0x7f55d8099a10 2024-01-15T18:09:11.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.030+0000 7f55d7400640 1 --2- 172.21.15.27:0/2773352870 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f55b0066ec0 0x7f55b0069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:11.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.034+0000 7f55d7400640 1 --2- 172.21.15.27:0/2773352870 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f55b0066ec0 0x7f55b0069380 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f55d81f9670 tx=0x7f55c0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.034+0000 7f55cfe00640 1 -- 172.21.15.27:0/2773352870 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f55c8050e20 con 0x7f55d8099a10 2024-01-15T18:09:11.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:11.226+0000 7f55df200640 1 -- 172.21.15.27:0/2773352870 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f55d80b6ed0 con 0x7f55d8099a10 2024-01-15T18:09:12.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.154+0000 7f55cfe00640 1 -- 172.21.15.27:0/2773352870 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v36) v1 ==== 118+0+0 (secure 0 0 0) 0x7f55c8054440 con 0x7f55d8099a10 2024-01-15T18:09:12.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.158+0000 7f55df200640 1 -- 172.21.15.27:0/2773352870 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f55b0066ec0 msgr2=0x7f55b0069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:12.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.158+0000 7f55df200640 1 --2- 172.21.15.27:0/2773352870 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f55b0066ec0 0x7f55b0069380 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f55d81f9670 tx=0x7f55c0009040 comp rx=0 tx=0).stop 2024-01-15T18:09:12.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.158+0000 7f55df200640 1 -- 172.21.15.27:0/2773352870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55d8099a10 msgr2=0x7f55d80b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:12.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.158+0000 7f55df200640 1 --2- 172.21.15.27:0/2773352870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55d8099a10 0x7f55d80b5c40 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f55c800c420 tx=0x7f55c800c8e0 comp rx=0 tx=0).stop 2024-01-15T18:09:12.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.158+0000 7f55df200640 1 -- 172.21.15.27:0/2773352870 shutdown_connections 2024-01-15T18:09:12.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.158+0000 7f55df200640 1 --2- 172.21.15.27:0/2773352870 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f55b0066ec0 0x7f55b0069380 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:12.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.158+0000 7f55df200640 1 --2- 172.21.15.27:0/2773352870 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f55d815b350 0x7f55d8152f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:12.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.158+0000 7f55df200640 1 --2- 172.21.15.27:0/2773352870 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55d809a5b0 0x7f55d806eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:12.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.158+0000 7f55df200640 1 --2- 172.21.15.27:0/2773352870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55d8099a10 0x7f55d80b5c40 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:12.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.158+0000 7f55df200640 1 -- 172.21.15.27:0/2773352870 >> 172.21.15.27:0/2773352870 conn(0x7f55d80b2a80 msgr2=0x7f55d80cf980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:12.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.158+0000 7f55df200640 1 -- 172.21.15.27:0/2773352870 shutdown_connections 2024-01-15T18:09:12.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.158+0000 7f55df200640 1 -- 172.21.15.27:0/2773352870 wait complete. 2024-01-15T18:09:12.185 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-15T18:09:12.185 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-15T18:09:12.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.326+0000 7f0604200640 1 Processor -- start 2024-01-15T18:09:12.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.326+0000 7f0604200640 1 -- start start 2024-01-15T18:09:12.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.326+0000 7f0604200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f05fc15b100 0x7f05fc070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:12.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.326+0000 7f0604200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05fc071860 con 0x7f05fc071420 2024-01-15T18:09:12.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.326+0000 7f0604200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05fc15a610 con 0x7f05fc15ad20 2024-01-15T18:09:12.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.326+0000 7f0604200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05fc15e3c0 con 0x7f05fc15b100 2024-01-15T18:09:12.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.326+0000 7f0600e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f05fc15b100 0x7f05fc070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:12.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.326+0000 7f0600e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f05fc15b100 0x7f05fc070ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43858/0 (socket says 172.21.15.27:43858) 2024-01-15T18:09:12.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.326+0000 7f0600e00640 1 -- 172.21.15.27:0/1292192756 learned_addr learned my addr 172.21.15.27:0/1292192756 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:12.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.326+0000 7f0600e00640 1 -- 172.21.15.27:0/1292192756 >> v1:172.21.15.93:6789/0 conn(0x7f05fc15ad20 legacy=0x7f05fc15d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:12.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.326+0000 7f0600e00640 1 -- 172.21.15.27:0/1292192756 >> v1:172.21.15.27:6789/0 conn(0x7f05fc071420 legacy=0x7f05fc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:12.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.326+0000 7f0600e00640 1 -- 172.21.15.27:0/1292192756 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05fc15e6a0 con 0x7f05fc15b100 2024-01-15T18:09:12.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.326+0000 7f05fbe00640 1 -- 172.21.15.27:0/1292192756 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249470674 0 0) 0x7f05fc071860 con 0x7f05fc071420 2024-01-15T18:09:12.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.330+0000 7f0600e00640 1 --2- 172.21.15.27:0/1292192756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05fc15b100 0x7f05fc070ee0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f05e40098a0 tx=0x7f05e401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa67df138930c5e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:12.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.330+0000 7f05fbe00640 1 -- 172.21.15.27:0/1292192756 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e40ab660 con 0x7f05fc15b100 2024-01-15T18:09:12.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.330+0000 7f05fbe00640 1 -- 172.21.15.27:0/1292192756 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e40ab7c0 con 0x7f05fc15b100 2024-01-15T18:09:12.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.330+0000 7f05fbe00640 1 -- 172.21.15.27:0/1292192756 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e40abb50 con 0x7f05fc15b100 2024-01-15T18:09:12.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.330+0000 7f0604200640 1 -- 172.21.15.27:0/1292192756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05fc15b100 msgr2=0x7f05fc070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:12.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.330+0000 7f0604200640 1 --2- 172.21.15.27:0/1292192756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05fc15b100 0x7f05fc070ee0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f05e40098a0 tx=0x7f05e401c460 comp rx=0 tx=0).stop 2024-01-15T18:09:12.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.330+0000 7f0604200640 1 -- 172.21.15.27:0/1292192756 shutdown_connections 2024-01-15T18:09:12.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.330+0000 7f0604200640 1 --2- 172.21.15.27:0/1292192756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05fc15b100 0x7f05fc070ee0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:12.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.330+0000 7f0604200640 1 -- 172.21.15.27:0/1292192756 >> 172.21.15.27:0/1292192756 conn(0x7f05fc0cb700 msgr2=0x7f05fc0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:12.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.330+0000 7f0604200640 1 -- 172.21.15.27:0/1292192756 shutdown_connections 2024-01-15T18:09:12.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.330+0000 7f0604200640 1 -- 172.21.15.27:0/1292192756 wait complete. 2024-01-15T18:09:12.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.334+0000 7f0604200640 1 Processor -- start 2024-01-15T18:09:12.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.334+0000 7f0604200640 1 -- start start 2024-01-15T18:09:12.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.334+0000 7f0604200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05fc071420 0x7f05fc1552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:12.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.334+0000 7f0604200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f05fc15ad20 0x7f05fc158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:12.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.334+0000 7f0604200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05fc15b100 0x7f05fc152d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:12.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.334+0000 7f0604200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc137150 con 0x7f05fc071420 2024-01-15T18:09:12.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.334+0000 7f0604200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc14b4f0 con 0x7f05fc15b100 2024-01-15T18:09:12.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.334+0000 7f0604200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc14b7d0 con 0x7f05fc15ad20 2024-01-15T18:09:12.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.334+0000 7f0601800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05fc071420 0x7f05fc1552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:12.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.334+0000 7f0601800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05fc071420 0x7f05fc1552f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43864/0 (socket says 172.21.15.27:43864) 2024-01-15T18:09:12.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.334+0000 7f0601800640 1 -- 172.21.15.27:0/4251722643 learned_addr learned my addr 172.21.15.27:0/4251722643 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:12.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.334+0000 7f0602200640 1 --2- 172.21.15.27:0/4251722643 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05fc15b100 0x7f05fc152d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:12.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.334+0000 7f0600e00640 1 --2- 172.21.15.27:0/4251722643 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f05fc15ad20 0x7f05fc158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:12.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.334+0000 7f0601800640 1 -- 172.21.15.27:0/4251722643 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f05fc15ad20 msgr2=0x7f05fc158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:12.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.334+0000 7f0601800640 1 --2- 172.21.15.27:0/4251722643 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f05fc15ad20 0x7f05fc158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:12.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.334+0000 7f0601800640 1 -- 172.21.15.27:0/4251722643 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05fc15b100 msgr2=0x7f05fc152d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:12.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.334+0000 7f0601800640 1 --2- 172.21.15.27:0/4251722643 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05fc15b100 0x7f05fc152d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:12.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.334+0000 7f0601800640 1 -- 172.21.15.27:0/4251722643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e40094d0 con 0x7f05fc071420 2024-01-15T18:09:12.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.334+0000 7f0601800640 1 --2- 172.21.15.27:0/4251722643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05fc071420 0x7f05fc1552f0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f05ec00c500 tx=0x7f05ec00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:12.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.334+0000 7f05f9600640 1 -- 172.21.15.27:0/4251722643 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ec005220 con 0x7f05fc071420 2024-01-15T18:09:12.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.334+0000 7f05f9600640 1 -- 172.21.15.27:0/4251722643 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05ec005380 con 0x7f05fc071420 2024-01-15T18:09:12.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.334+0000 7f0604200640 1 -- 172.21.15.27:0/4251722643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05fc14bab0 con 0x7f05fc071420 2024-01-15T18:09:12.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.334+0000 7f05f9600640 1 -- 172.21.15.27:0/4251722643 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ec007270 con 0x7f05fc071420 2024-01-15T18:09:12.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.334+0000 7f0604200640 1 -- 172.21.15.27:0/4251722643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05fc14c070 con 0x7f05fc071420 2024-01-15T18:09:12.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.338+0000 7f0604200640 1 -- 172.21.15.27:0/4251722643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05fc04e850 con 0x7f05fc071420 2024-01-15T18:09:12.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.338+0000 7f05f9600640 1 -- 172.21.15.27:0/4251722643 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f05ec0073d0 con 0x7f05fc071420 2024-01-15T18:09:12.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.338+0000 7f05f9600640 1 --2- 172.21.15.27:0/4251722643 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05d4066e90 0x7f05d4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:12.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.338+0000 7f05f9600640 1 -- 172.21.15.27:0/4251722643 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f05ec0846b0 con 0x7f05fc071420 2024-01-15T18:09:12.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.338+0000 7f0600e00640 1 --2- 172.21.15.27:0/4251722643 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05d4066e90 0x7f05d4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:12.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.338+0000 7f0600e00640 1 --2- 172.21.15.27:0/4251722643 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05d4066e90 0x7f05d4069350 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f05e40af000 tx=0x7f05e40a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:12.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.342+0000 7f05f9600640 1 -- 172.21.15.27:0/4251722643 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f05ec0510e0 con 0x7f05fc071420 2024-01-15T18:09:12.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:12.538+0000 7f0604200640 1 -- 172.21.15.27:0/4251722643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f05fc0b1b00 con 0x7f05fc071420 2024-01-15T18:09:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.162+0000 7f05f9600640 1 -- 172.21.15.27:0/4251722643 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v37) v1 ==== 122+0+0 (secure 0 0 0) 0x7f05ec054700 con 0x7f05fc071420 2024-01-15T18:09:13.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.166+0000 7f0604200640 1 -- 172.21.15.27:0/4251722643 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05d4066e90 msgr2=0x7f05d4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:13.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.166+0000 7f0604200640 1 --2- 172.21.15.27:0/4251722643 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05d4066e90 0x7f05d4069350 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f05e40af000 tx=0x7f05e40a7000 comp rx=0 tx=0).stop 2024-01-15T18:09:13.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.166+0000 7f0604200640 1 -- 172.21.15.27:0/4251722643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05fc071420 msgr2=0x7f05fc1552f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:13.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.166+0000 7f0604200640 1 --2- 172.21.15.27:0/4251722643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05fc071420 0x7f05fc1552f0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f05ec00c500 tx=0x7f05ec00c9c0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.166+0000 7f0604200640 1 -- 172.21.15.27:0/4251722643 shutdown_connections 2024-01-15T18:09:13.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.166+0000 7f0604200640 1 --2- 172.21.15.27:0/4251722643 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05d4066e90 0x7f05d4069350 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.166+0000 7f0604200640 1 --2- 172.21.15.27:0/4251722643 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f05fc15b100 0x7f05fc152d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.166+0000 7f0604200640 1 --2- 172.21.15.27:0/4251722643 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f05fc15ad20 0x7f05fc158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.166+0000 7f0604200640 1 --2- 172.21.15.27:0/4251722643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05fc071420 0x7f05fc1552f0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.166+0000 7f0604200640 1 -- 172.21.15.27:0/4251722643 >> 172.21.15.27:0/4251722643 conn(0x7f05fc0cb700 msgr2=0x7f05fc14db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:13.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.166+0000 7f0604200640 1 -- 172.21.15.27:0/4251722643 shutdown_connections 2024-01-15T18:09:13.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.166+0000 7f0604200640 1 -- 172.21.15.27:0/4251722643 wait complete. 2024-01-15T18:09:13.192 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-15T18:09:13.193 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:09:13.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.330+0000 7f2ca3200640 1 Processor -- start 2024-01-15T18:09:13.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.330+0000 7f2ca3200640 1 -- start start 2024-01-15T18:09:13.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.330+0000 7f2ca3200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2c9c15e830 0x7f2c9c0ceaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:13.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.330+0000 7f2ca3200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c9c0d1ab0 0x7f2c9c0cf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:13.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.330+0000 7f2ca3200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f2c9c0cf570 0x7f2c9c0cd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.330+0000 7f2ca3200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c9c0c4ba0 con 0x7f2c9c15e830 2024-01-15T18:09:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.330+0000 7f2ca3200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c9c0cc5a0 con 0x7f2c9c0cf570 2024-01-15T18:09:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.330+0000 7f2ca3200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c0cc840 con 0x7f2c9c0d1ab0 2024-01-15T18:09:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.334+0000 7f2c9be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2c9c15e830 0x7f2c9c0ceaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.334+0000 7f2c9be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2c9c15e830 0x7f2c9c0ceaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43872/0 (socket says 172.21.15.27:43872) 2024-01-15T18:09:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.334+0000 7f2c9be00640 1 -- 172.21.15.27:0/3345563500 learned_addr learned my addr 172.21.15.27:0/3345563500 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.334+0000 7f2ca1200640 1 --2- 172.21.15.27:0/3345563500 >> v2:172.21.15.93:3300/0 conn(0x7f2c9c0cf570 0x7f2c9c0cd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.334+0000 7f2c9b400640 1 --2- 172.21.15.27:0/3345563500 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c9c0d1ab0 0x7f2c9c0cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.334+0000 7f2c9be00640 1 -- 172.21.15.27:0/3345563500 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c9c0d1ab0 msgr2=0x7f2c9c0cf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.334+0000 7f2c9be00640 1 --2- 172.21.15.27:0/3345563500 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c9c0d1ab0 0x7f2c9c0cf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.334+0000 7f2c9be00640 1 -- 172.21.15.27:0/3345563500 >> v2:172.21.15.93:3300/0 conn(0x7f2c9c0cf570 msgr2=0x7f2c9c0cd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.334+0000 7f2c9be00640 1 --2- 172.21.15.27:0/3345563500 >> v2:172.21.15.93:3300/0 conn(0x7f2c9c0cf570 0x7f2c9c0cd7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.334+0000 7f2c9be00640 1 -- 172.21.15.27:0/3345563500 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9c0ccb20 con 0x7f2c9c15e830 2024-01-15T18:09:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.334+0000 7f2ca1200640 1 --2- 172.21.15.27:0/3345563500 >> v2:172.21.15.93:3300/0 conn(0x7f2c9c0cf570 0x7f2c9c0cd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:09:13.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.334+0000 7f2c9be00640 1 --2- 172.21.15.27:0/3345563500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c9c15e830 0x7f2c9c0ceaf0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c06e270 tx=0x7f2c8c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab4b6a851b88772 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:13.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.334+0000 7f2c9aa00640 1 -- 172.21.15.27:0/3345563500 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c0ab600 con 0x7f2c9c15e830 2024-01-15T18:09:13.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.334+0000 7f2c9aa00640 1 -- 172.21.15.27:0/3345563500 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c8c0a7050 con 0x7f2c9c15e830 2024-01-15T18:09:13.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.334+0000 7f2c9aa00640 1 -- 172.21.15.27:0/3345563500 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c0ab980 con 0x7f2c9c15e830 2024-01-15T18:09:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.338+0000 7f2ca3200640 1 -- 172.21.15.27:0/3345563500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c9c15e830 msgr2=0x7f2c9c0ceaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.338+0000 7f2ca3200640 1 --2- 172.21.15.27:0/3345563500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c9c15e830 0x7f2c9c0ceaf0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c06e270 tx=0x7f2c8c01c500 comp rx=0 tx=0).stop 2024-01-15T18:09:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.338+0000 7f2ca3200640 1 -- 172.21.15.27:0/3345563500 shutdown_connections 2024-01-15T18:09:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.338+0000 7f2ca3200640 1 --2- 172.21.15.27:0/3345563500 >> v2:172.21.15.93:3300/0 conn(0x7f2c9c0cf570 0x7f2c9c0cd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.338+0000 7f2ca3200640 1 --2- 172.21.15.27:0/3345563500 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c9c0d1ab0 0x7f2c9c0cf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.338+0000 7f2ca3200640 1 --2- 172.21.15.27:0/3345563500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c9c15e830 0x7f2c9c0ceaf0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.338+0000 7f2ca3200640 1 -- 172.21.15.27:0/3345563500 >> 172.21.15.27:0/3345563500 conn(0x7f2c9c0d0f20 msgr2=0x7f2c9c0d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.338+0000 7f2ca3200640 1 -- 172.21.15.27:0/3345563500 shutdown_connections 2024-01-15T18:09:13.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.338+0000 7f2ca3200640 1 -- 172.21.15.27:0/3345563500 wait complete. 2024-01-15T18:09:13.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.338+0000 7f2ca3200640 1 Processor -- start 2024-01-15T18:09:13.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.338+0000 7f2ca3200640 1 -- start start 2024-01-15T18:09:13.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.338+0000 7f2ca3200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c9c0cf570 0x7f2c9c148e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:13.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.338+0000 7f2ca3200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c9c0d1ab0 0x7f2c9c1514c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:13.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.338+0000 7f2ca3200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c9c15e830 0x7f2c9c151a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:13.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.338+0000 7f2ca3200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c14fe70 con 0x7f2c9c0cf570 2024-01-15T18:09:13.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.338+0000 7f2ca3200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c14aef0 con 0x7f2c9c15e830 2024-01-15T18:09:13.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.338+0000 7f2ca3200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c151f40 con 0x7f2c9c0d1ab0 2024-01-15T18:09:13.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.338+0000 7f2c9be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c9c0cf570 0x7f2c9c148e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:13.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.338+0000 7f2c9be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c9c0cf570 0x7f2c9c148e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43878/0 (socket says 172.21.15.27:43878) 2024-01-15T18:09:13.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.342+0000 7f2c9be00640 1 -- 172.21.15.27:0/1616233010 learned_addr learned my addr 172.21.15.27:0/1616233010 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:13.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.342+0000 7f2ca1200640 1 --2- 172.21.15.27:0/1616233010 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c9c15e830 0x7f2c9c151a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:13.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.342+0000 7f2c9b400640 1 --2- 172.21.15.27:0/1616233010 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c9c0d1ab0 0x7f2c9c1514c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:13.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.342+0000 7f2c9be00640 1 -- 172.21.15.27:0/1616233010 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c9c0d1ab0 msgr2=0x7f2c9c1514c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:13.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.342+0000 7f2c9be00640 1 --2- 172.21.15.27:0/1616233010 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c9c0d1ab0 0x7f2c9c1514c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.342+0000 7f2c9be00640 1 -- 172.21.15.27:0/1616233010 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c9c15e830 msgr2=0x7f2c9c151a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:13.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.342+0000 7f2c9be00640 1 --2- 172.21.15.27:0/1616233010 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c9c15e830 0x7f2c9c151a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.342+0000 7f2c9be00640 1 -- 172.21.15.27:0/1616233010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c8c0094d0 con 0x7f2c9c0cf570 2024-01-15T18:09:13.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.342+0000 7f2ca1200640 1 --2- 172.21.15.27:0/1616233010 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c9c15e830 0x7f2c9c151a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:13.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.342+0000 7f2c9be00640 1 --2- 172.21.15.27:0/1616233010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c9c0cf570 0x7f2c9c148e30 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c0ae040 tx=0x7f2c8c01ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:13.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.342+0000 7f2c93e00640 1 -- 172.21.15.27:0/1616233010 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c0b6830 con 0x7f2c9c0cf570 2024-01-15T18:09:13.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.342+0000 7f2c93e00640 1 -- 172.21.15.27:0/1616233010 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c8c0abe90 con 0x7f2c9c0cf570 2024-01-15T18:09:13.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.342+0000 7f2ca3200640 1 -- 172.21.15.27:0/1616233010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c9c0ce890 con 0x7f2c9c0cf570 2024-01-15T18:09:13.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.342+0000 7f2c93e00640 1 -- 172.21.15.27:0/1616233010 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c0b6bf0 con 0x7f2c9c0cf570 2024-01-15T18:09:13.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.342+0000 7f2ca3200640 1 -- 172.21.15.27:0/1616233010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c9c14a900 con 0x7f2c9c0cf570 2024-01-15T18:09:13.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.342+0000 7f2ca3200640 1 -- 172.21.15.27:0/1616233010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c9c14ffb0 con 0x7f2c9c0cf570 2024-01-15T18:09:13.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.342+0000 7f2c93e00640 1 -- 172.21.15.27:0/1616233010 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f2c8c0b2040 con 0x7f2c9c0cf570 2024-01-15T18:09:13.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.346+0000 7f2c93e00640 1 --2- 172.21.15.27:0/1616233010 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2c74066e90 0x7f2c74069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:13.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.346+0000 7f2c93e00640 1 -- 172.21.15.27:0/1616233010 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f2c8c0a4e40 con 0x7f2c9c0cf570 2024-01-15T18:09:13.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.346+0000 7f2c9b400640 1 --2- 172.21.15.27:0/1616233010 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2c74066e90 0x7f2c74069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:13.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.346+0000 7f2c9b400640 1 --2- 172.21.15.27:0/1616233010 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2c74066e90 0x7f2c74069350 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c15cb90 tx=0x7f2c84008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:13.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.350+0000 7f2c93e00640 1 -- 172.21.15.27:0/1616233010 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2c9c14ffb0 con 0x7f2c9c0cf570 2024-01-15T18:09:13.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.538+0000 7f2ca3200640 1 -- 172.21.15.27:0/1616233010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c9c14ffb0 con 0x7f2c9c0cf570 2024-01-15T18:09:13.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.538+0000 7f2c93e00640 1 -- 172.21.15.27:0/1616233010 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+4910 (secure 0 0 0) 0x7f2c9c14ffb0 con 0x7f2c9c0cf570 2024-01-15T18:09:13.545 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:09:13.545 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":37,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1939851998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1939851998},{"type":"v1","addr":"172.21.15.27:6835","nonce":1939851998}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":5759,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/255207804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":255207804},{"type":"v1","addr":"172.21.15.93:6837","nonce":255207804}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":6244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/1075225596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":1075225596},{"type":"v1","addr":"172.21.15.93:6836","nonce":1075225596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:09:09.513597+0000","modified":"2024-01-15T18:09:12.543751+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6480},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6480":{"gid":6480,"name":"c","rank":0,"incarnation":33,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/1969845074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1969845074},{"type":"v1","addr":"172.21.15.27:6834","nonce":1969845074}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-01-15T18:09:13.545 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 37 2024-01-15T18:09:13.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.542+0000 7f2ca3200640 1 -- 172.21.15.27:0/1616233010 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2c74066e90 msgr2=0x7f2c74069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:13.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.542+0000 7f2ca3200640 1 --2- 172.21.15.27:0/1616233010 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2c74066e90 0x7f2c74069350 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c15cb90 tx=0x7f2c84008040 comp rx=0 tx=0).stop 2024-01-15T18:09:13.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.542+0000 7f2ca3200640 1 -- 172.21.15.27:0/1616233010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c9c0cf570 msgr2=0x7f2c9c148e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:13.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.542+0000 7f2ca3200640 1 --2- 172.21.15.27:0/1616233010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c9c0cf570 0x7f2c9c148e30 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c0ae040 tx=0x7f2c8c01ad40 comp rx=0 tx=0).stop 2024-01-15T18:09:13.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.542+0000 7f2ca3200640 1 -- 172.21.15.27:0/1616233010 shutdown_connections 2024-01-15T18:09:13.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.542+0000 7f2ca3200640 1 --2- 172.21.15.27:0/1616233010 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2c74066e90 0x7f2c74069350 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.542+0000 7f2ca3200640 1 --2- 172.21.15.27:0/1616233010 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c9c15e830 0x7f2c9c151a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.542+0000 7f2ca3200640 1 --2- 172.21.15.27:0/1616233010 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c9c0d1ab0 0x7f2c9c1514c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.542+0000 7f2ca3200640 1 --2- 172.21.15.27:0/1616233010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c9c0cf570 0x7f2c9c148e30 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.542+0000 7f2ca3200640 1 -- 172.21.15.27:0/1616233010 >> 172.21.15.27:0/1616233010 conn(0x7f2c9c0d0f20 msgr2=0x7f2c9c0c72a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:13.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.542+0000 7f2ca3200640 1 -- 172.21.15.27:0/1616233010 shutdown_connections 2024-01-15T18:09:13.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.542+0000 7f2ca3200640 1 -- 172.21.15.27:0/1616233010 wait complete. 2024-01-15T18:09:13.568 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:09:13.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.702+0000 7f1921e00640 1 Processor -- start 2024-01-15T18:09:13.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.702+0000 7f1921e00640 1 -- start start 2024-01-15T18:09:13.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.702+0000 7f1921e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f191c0b0130 0x7f191c15f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:13.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.702+0000 7f1921e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f191c0b24a0 0x7f191c15f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:13.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.702+0000 7f1921e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f191c0d1ec0 con 0x7f191c0cf080 2024-01-15T18:09:13.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.702+0000 7f1921e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f191c0ae160 con 0x7f191c0b0130 2024-01-15T18:09:13.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.702+0000 7f1921e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f191c0cfb70 con 0x7f191c0b24a0 2024-01-15T18:09:13.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.702+0000 7f191a000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f191c0b0130 0x7f191c15f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:13.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.702+0000 7f191a000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f191c0b0130 0x7f191c15f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34310/0 (socket says 172.21.15.27:34310) 2024-01-15T18:09:13.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.702+0000 7f191a000640 1 -- 172.21.15.27:0/949421419 learned_addr learned my addr 172.21.15.27:0/949421419 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:13.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.706+0000 7f191b400640 1 --2- 172.21.15.27:0/949421419 >> v2:172.21.15.93:3300/0 conn(0x7f191c0b24a0 0x7f191c15f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:13.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.706+0000 7f191a000640 1 -- 172.21.15.27:0/949421419 >> v2:172.21.15.93:3300/0 conn(0x7f191c0b24a0 msgr2=0x7f191c15f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:13.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.706+0000 7f191a000640 1 --2- 172.21.15.27:0/949421419 >> v2:172.21.15.93:3300/0 conn(0x7f191c0b24a0 0x7f191c15f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.706+0000 7f191a000640 1 -- 172.21.15.27:0/949421419 >> v1:172.21.15.27:6789/0 conn(0x7f191c0cf080 legacy=0x7f191c0cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:13.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.706+0000 7f191a000640 1 -- 172.21.15.27:0/949421419 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f191c0d1be0 con 0x7f191c0b0130 2024-01-15T18:09:13.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.706+0000 7f1919600640 1 -- 172.21.15.27:0/949421419 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 815434591 0 0) 0x7f191c0d1ec0 con 0x7f191c0cf080 2024-01-15T18:09:13.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.706+0000 7f191a000640 1 --2- 172.21.15.27:0/949421419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191c0b0130 0x7f191c15f300 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f191c072870 tx=0x7f190401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54c9d3ebc9ca63ca server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:13.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.706+0000 7f1919600640 1 -- 172.21.15.27:0/949421419 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19040aa650 con 0x7f191c0b0130 2024-01-15T18:09:13.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.706+0000 7f1919600640 1 -- 172.21.15.27:0/949421419 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19040aa7b0 con 0x7f191c0b0130 2024-01-15T18:09:13.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.706+0000 7f1919600640 1 -- 172.21.15.27:0/949421419 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19040aab40 con 0x7f191c0b0130 2024-01-15T18:09:13.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.710+0000 7f1921e00640 1 -- 172.21.15.27:0/949421419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191c0b0130 msgr2=0x7f191c15f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:13.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.710+0000 7f1921e00640 1 --2- 172.21.15.27:0/949421419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191c0b0130 0x7f191c15f300 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f191c072870 tx=0x7f190401c4d0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.710+0000 7f1921e00640 1 -- 172.21.15.27:0/949421419 shutdown_connections 2024-01-15T18:09:13.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.710+0000 7f1921e00640 1 --2- 172.21.15.27:0/949421419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191c0b0130 0x7f191c15f300 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.710+0000 7f1921e00640 1 --2- 172.21.15.27:0/949421419 >> v2:172.21.15.93:3300/0 conn(0x7f191c0b24a0 0x7f191c15f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.710+0000 7f1921e00640 1 -- 172.21.15.27:0/949421419 >> 172.21.15.27:0/949421419 conn(0x7f191c0d0f20 msgr2=0x7f191c0d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:13.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.710+0000 7f1921e00640 1 -- 172.21.15.27:0/949421419 shutdown_connections 2024-01-15T18:09:13.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.710+0000 7f1921e00640 1 -- 172.21.15.27:0/949421419 wait complete. 2024-01-15T18:09:13.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.710+0000 7f1921e00640 1 Processor -- start 2024-01-15T18:09:13.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.710+0000 7f1921e00640 1 -- start start 2024-01-15T18:09:13.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.710+0000 7f1921e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f191c0b0130 0x7f191c1520a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:13.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.710+0000 7f1921e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f191c0b24a0 0x7f191c153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:13.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.710+0000 7f1921e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191c0cf080 0x7f191c1576b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:13.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.710+0000 7f1921e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f191c151f30 con 0x7f191c0cf080 2024-01-15T18:09:13.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.710+0000 7f1921e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f191c154420 con 0x7f191c0b0130 2024-01-15T18:09:13.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.710+0000 7f1921e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f191c0c57f0 con 0x7f191c0b24a0 2024-01-15T18:09:13.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.710+0000 7f191b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191c0cf080 0x7f191c1576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:13.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.710+0000 7f191b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191c0cf080 0x7f191c1576b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34322/0 (socket says 172.21.15.27:34322) 2024-01-15T18:09:13.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.710+0000 7f191b400640 1 -- 172.21.15.27:0/3106883392 learned_addr learned my addr 172.21.15.27:0/3106883392 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:13.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.710+0000 7f191a000640 1 --2- 172.21.15.27:0/3106883392 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f191c0b24a0 0x7f191c153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:13.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.710+0000 7f191aa00640 1 --2- 172.21.15.27:0/3106883392 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f191c0b0130 0x7f191c1520a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:13.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.714+0000 7f191b400640 1 -- 172.21.15.27:0/3106883392 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f191c0b24a0 msgr2=0x7f191c153ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:13.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.714+0000 7f191b400640 1 --2- 172.21.15.27:0/3106883392 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f191c0b24a0 0x7f191c153ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.714+0000 7f191b400640 1 -- 172.21.15.27:0/3106883392 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f191c0b0130 msgr2=0x7f191c1520a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:13.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.714+0000 7f191b400640 1 --2- 172.21.15.27:0/3106883392 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f191c0b0130 0x7f191c1520a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.714+0000 7f191b400640 1 -- 172.21.15.27:0/3106883392 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19040094d0 con 0x7f191c0cf080 2024-01-15T18:09:13.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.714+0000 7f191b400640 1 --2- 172.21.15.27:0/3106883392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191c0cf080 0x7f191c1576b0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f1914007e70 tx=0x7f191400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:13.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.714+0000 7f1913400640 1 -- 172.21.15.27:0/3106883392 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1914014620 con 0x7f191c0cf080 2024-01-15T18:09:13.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.714+0000 7f1913400640 1 -- 172.21.15.27:0/3106883392 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19140147a0 con 0x7f191c0cf080 2024-01-15T18:09:13.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.714+0000 7f1921e00640 1 -- 172.21.15.27:0/3106883392 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f191c0c5ad0 con 0x7f191c0cf080 2024-01-15T18:09:13.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.714+0000 7f1913400640 1 -- 172.21.15.27:0/3106883392 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1914014b50 con 0x7f191c0cf080 2024-01-15T18:09:13.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.714+0000 7f1921e00640 1 -- 172.21.15.27:0/3106883392 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f191c0c6090 con 0x7f191c0cf080 2024-01-15T18:09:13.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.714+0000 7f1921e00640 1 -- 172.21.15.27:0/3106883392 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f191c04e850 con 0x7f191c0cf080 2024-01-15T18:09:13.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.714+0000 7f1913400640 1 -- 172.21.15.27:0/3106883392 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f1914014cb0 con 0x7f191c0cf080 2024-01-15T18:09:13.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.718+0000 7f1913400640 1 --2- 172.21.15.27:0/3106883392 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f18f4066e90 0x7f18f4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:13.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.718+0000 7f1913400640 1 -- 172.21.15.27:0/3106883392 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f19140856b0 con 0x7f191c0cf080 2024-01-15T18:09:13.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.718+0000 7f191aa00640 1 --2- 172.21.15.27:0/3106883392 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f18f4066e90 0x7f18f4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:13.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.718+0000 7f191aa00640 1 --2- 172.21.15.27:0/3106883392 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f18f4066e90 0x7f18f4069350 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f190c007fd0 tx=0x7f190c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:13.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.722+0000 7f1913400640 1 -- 172.21.15.27:0/3106883392 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f19140521b0 con 0x7f191c0cf080 2024-01-15T18:09:13.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.878+0000 7f1921e00640 1 -- 172.21.15.27:0/3106883392 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f191c0cf600 con 0x7f191c0cf080 2024-01-15T18:09:13.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.878+0000 7f1913400640 1 -- 172.21.15.27:0/3106883392 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16512 (secure 0 0 0) 0x7f19140557d0 con 0x7f191c0cf080 2024-01-15T18:09:13.887 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:09:13.887 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":58,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:09:11.143366+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:09:07.910453+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-15T18:09:08.507842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:09:13.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.882+0000 7f1921e00640 1 -- 172.21.15.27:0/3106883392 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f18f4066e90 msgr2=0x7f18f4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:13.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.882+0000 7f1921e00640 1 --2- 172.21.15.27:0/3106883392 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f18f4066e90 0x7f18f4069350 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f190c007fd0 tx=0x7f190c009040 comp rx=0 tx=0).stop 2024-01-15T18:09:13.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.882+0000 7f1921e00640 1 -- 172.21.15.27:0/3106883392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191c0cf080 msgr2=0x7f191c1576b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:13.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.882+0000 7f1921e00640 1 --2- 172.21.15.27:0/3106883392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191c0cf080 0x7f191c1576b0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f1914007e70 tx=0x7f191400e4b0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.882+0000 7f1921e00640 1 -- 172.21.15.27:0/3106883392 shutdown_connections 2024-01-15T18:09:13.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.882+0000 7f1921e00640 1 --2- 172.21.15.27:0/3106883392 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f18f4066e90 0x7f18f4069350 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.882+0000 7f1921e00640 1 --2- 172.21.15.27:0/3106883392 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f191c0cf080 0x7f191c1576b0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.882+0000 7f1921e00640 1 --2- 172.21.15.27:0/3106883392 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f191c0b24a0 0x7f191c153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.882+0000 7f1921e00640 1 --2- 172.21.15.27:0/3106883392 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f191c0b0130 0x7f191c1520a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.882+0000 7f1921e00640 1 -- 172.21.15.27:0/3106883392 >> 172.21.15.27:0/3106883392 conn(0x7f191c0d0f20 msgr2=0x7f191c06e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:13.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.882+0000 7f1921e00640 1 -- 172.21.15.27:0/3106883392 shutdown_connections 2024-01-15T18:09:13.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:13.882+0000 7f1921e00640 1 -- 172.21.15.27:0/3106883392 wait complete. 2024-01-15T18:09:13.909 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-15T18:09:13.909 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:09:14.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.042+0000 7fdfb4a00640 1 Processor -- start 2024-01-15T18:09:14.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.046+0000 7fdfb4a00640 1 -- start start 2024-01-15T18:09:14.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.046+0000 7fdfb4a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdfac06db80 0x7fdfac06df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:14.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.046+0000 7fdfb4a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fdfac15e730 0x7fdfac0cd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:14.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.046+0000 7fdfb4a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdfac0cdd10 0x7fdfac070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:14.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.046+0000 7fdfb4a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfac0ce920 con 0x7fdfac0cdd10 2024-01-15T18:09:14.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.046+0000 7fdfb4a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfac06e4a0 con 0x7fdfac15e730 2024-01-15T18:09:14.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.046+0000 7fdfb4a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfac0ce640 con 0x7fdfac06db80 2024-01-15T18:09:14.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.046+0000 7fdfb2a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdfac0cdd10 0x7fdfac070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:14.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.046+0000 7fdfb2a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdfac0cdd10 0x7fdfac070c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34332/0 (socket says 172.21.15.27:34332) 2024-01-15T18:09:14.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.046+0000 7fdfb2a00640 1 -- 172.21.15.27:0/2241496867 learned_addr learned my addr 172.21.15.27:0/2241496867 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:14.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.046+0000 7fdfb1600640 1 --2- 172.21.15.27:0/2241496867 >> v2:172.21.15.93:3300/0 conn(0x7fdfac15e730 0x7fdfac0cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:14.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.046+0000 7fdfb2000640 1 --2- 172.21.15.27:0/2241496867 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdfac06db80 0x7fdfac06df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:14.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.046+0000 7fdfb2a00640 1 -- 172.21.15.27:0/2241496867 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdfac06db80 msgr2=0x7fdfac06df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:14.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.046+0000 7fdfb2a00640 1 --2- 172.21.15.27:0/2241496867 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdfac06db80 0x7fdfac06df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.046+0000 7fdfb2a00640 1 -- 172.21.15.27:0/2241496867 >> v2:172.21.15.93:3300/0 conn(0x7fdfac15e730 msgr2=0x7fdfac0cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:14.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.046+0000 7fdfb2a00640 1 --2- 172.21.15.27:0/2241496867 >> v2:172.21.15.93:3300/0 conn(0x7fdfac15e730 0x7fdfac0cd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.046+0000 7fdfb2a00640 1 -- 172.21.15.27:0/2241496867 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfac0cc940 con 0x7fdfac0cdd10 2024-01-15T18:09:14.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.050+0000 7fdfb2a00640 1 --2- 172.21.15.27:0/2241496867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfac0cdd10 0x7fdfac070c20 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fdfac067dd0 tx=0x7fdfa401eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bb5e87af88d78d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:14.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.050+0000 7fdfb0c00640 1 -- 172.21.15.27:0/2241496867 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa400b2d0 con 0x7fdfac0cdd10 2024-01-15T18:09:14.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.050+0000 7fdfb0c00640 1 -- 172.21.15.27:0/2241496867 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa400b430 con 0x7fdfac0cdd10 2024-01-15T18:09:14.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.050+0000 7fdfb0c00640 1 -- 172.21.15.27:0/2241496867 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa40a7c50 con 0x7fdfac0cdd10 2024-01-15T18:09:14.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.050+0000 7fdfb4a00640 1 -- 172.21.15.27:0/2241496867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfac0cdd10 msgr2=0x7fdfac070c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:14.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.050+0000 7fdfb4a00640 1 --2- 172.21.15.27:0/2241496867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfac0cdd10 0x7fdfac070c20 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fdfac067dd0 tx=0x7fdfa401eba0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.050+0000 7fdfb4a00640 1 -- 172.21.15.27:0/2241496867 shutdown_connections 2024-01-15T18:09:14.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.050+0000 7fdfb4a00640 1 --2- 172.21.15.27:0/2241496867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfac0cdd10 0x7fdfac070c20 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.050+0000 7fdfb4a00640 1 --2- 172.21.15.27:0/2241496867 >> v2:172.21.15.93:3300/0 conn(0x7fdfac15e730 0x7fdfac0cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.050+0000 7fdfb4a00640 1 --2- 172.21.15.27:0/2241496867 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdfac06db80 0x7fdfac06df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.050+0000 7fdfb4a00640 1 -- 172.21.15.27:0/2241496867 >> 172.21.15.27:0/2241496867 conn(0x7fdfac0cec60 msgr2=0x7fdfac0cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:14.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.050+0000 7fdfb4a00640 1 -- 172.21.15.27:0/2241496867 shutdown_connections 2024-01-15T18:09:14.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.050+0000 7fdfb4a00640 1 -- 172.21.15.27:0/2241496867 wait complete. 2024-01-15T18:09:14.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.050+0000 7fdfb4a00640 1 Processor -- start 2024-01-15T18:09:14.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.054+0000 7fdfb4a00640 1 -- start start 2024-01-15T18:09:14.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.054+0000 7fdfb4a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdfac06db80 0x7fdfac0c6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:14.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.054+0000 7fdfb4a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdfac0cdd10 0x7fdfac0c8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:14.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.054+0000 7fdfb4a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfac15e730 0x7fdfac15c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:14.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.054+0000 7fdfb4a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfac156f00 con 0x7fdfac15e730 2024-01-15T18:09:14.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.054+0000 7fdfb4a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfac0c4f80 con 0x7fdfac0cdd10 2024-01-15T18:09:14.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.054+0000 7fdfb4a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfac0c9000 con 0x7fdfac06db80 2024-01-15T18:09:14.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.054+0000 7fdfb2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfac15e730 0x7fdfac15c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:14.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.054+0000 7fdfb2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfac15e730 0x7fdfac15c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34342/0 (socket says 172.21.15.27:34342) 2024-01-15T18:09:14.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.054+0000 7fdfb2a00640 1 -- 172.21.15.27:0/2361062258 learned_addr learned my addr 172.21.15.27:0/2361062258 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:14.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.054+0000 7fdfb1600640 1 --2- 172.21.15.27:0/2361062258 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdfac0cdd10 0x7fdfac0c8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:14.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.054+0000 7fdfb2000640 1 --2- 172.21.15.27:0/2361062258 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdfac06db80 0x7fdfac0c6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:14.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.054+0000 7fdfb2a00640 1 -- 172.21.15.27:0/2361062258 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdfac06db80 msgr2=0x7fdfac0c6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:14.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.054+0000 7fdfb2a00640 1 --2- 172.21.15.27:0/2361062258 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdfac06db80 0x7fdfac0c6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.054+0000 7fdfb2a00640 1 -- 172.21.15.27:0/2361062258 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdfac0cdd10 msgr2=0x7fdfac0c8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:14.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.054+0000 7fdfb2a00640 1 --2- 172.21.15.27:0/2361062258 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdfac0cdd10 0x7fdfac0c8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.054+0000 7fdfb2a00640 1 -- 172.21.15.27:0/2361062258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa400bd40 con 0x7fdfac15e730 2024-01-15T18:09:14.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.054+0000 7fdfb2a00640 1 --2- 172.21.15.27:0/2361062258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfac15e730 0x7fdfac15c7c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fdfa401d250 tx=0x7fdfa401b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:14.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.054+0000 7fdfaa000640 1 -- 172.21.15.27:0/2361062258 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa4009800 con 0x7fdfac15e730 2024-01-15T18:09:14.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.054+0000 7fdfaa000640 1 -- 172.21.15.27:0/2361062258 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa40099a0 con 0x7fdfac15e730 2024-01-15T18:09:14.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.054+0000 7fdfaa000640 1 -- 172.21.15.27:0/2361062258 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa4009d50 con 0x7fdfac15e730 2024-01-15T18:09:14.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.054+0000 7fdfb4a00640 1 -- 172.21.15.27:0/2361062258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfac1471c0 con 0x7fdfac15e730 2024-01-15T18:09:14.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.054+0000 7fdfb4a00640 1 -- 172.21.15.27:0/2361062258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfac0c7fe0 con 0x7fdfac15e730 2024-01-15T18:09:14.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.054+0000 7fdfb4a00640 1 -- 172.21.15.27:0/2361062258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfac0c7e40 con 0x7fdfac15e730 2024-01-15T18:09:14.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.058+0000 7fdfaa000640 1 -- 172.21.15.27:0/2361062258 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7fdfa40c3460 con 0x7fdfac15e730 2024-01-15T18:09:14.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.058+0000 7fdfaa000640 1 --2- 172.21.15.27:0/2361062258 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdf84066e90 0x7fdf84069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:14.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.058+0000 7fdfaa000640 1 -- 172.21.15.27:0/2361062258 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fdfa4120600 con 0x7fdfac15e730 2024-01-15T18:09:14.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.058+0000 7fdfb2000640 1 --2- 172.21.15.27:0/2361062258 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdf84066e90 0x7fdf84069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:14.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.058+0000 7fdfb2000640 1 --2- 172.21.15.27:0/2361062258 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdf84066e90 0x7fdf84069350 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c007a60 tx=0x7fdf9c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:14.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.062+0000 7fdfaa000640 1 -- 172.21.15.27:0/2361062258 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fdfac0c7e40 con 0x7fdfac15e730 2024-01-15T18:09:14.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.218+0000 7fdfb4a00640 1 -- 172.21.15.27:0/2361062258 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdfac06ca70 con 0x7fdf84066e90 2024-01-15T18:09:14.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.226+0000 7fdfaa000640 1 -- 172.21.15.27:0/2361062258 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287472 (secure 0 0 0) 0x7fdf9c010990 con 0x7fdf84066e90 2024-01-15T18:09:14.236 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:09:14.239 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:09:14.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.230+0000 7fdfb4a00640 1 -- 172.21.15.27:0/2361062258 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdf84066e90 msgr2=0x7fdf84069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:14.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.230+0000 7fdfb4a00640 1 --2- 172.21.15.27:0/2361062258 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdf84066e90 0x7fdf84069350 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c007a60 tx=0x7fdf9c008040 comp rx=0 tx=0).stop 2024-01-15T18:09:14.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.230+0000 7fdfb4a00640 1 -- 172.21.15.27:0/2361062258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfac15e730 msgr2=0x7fdfac15c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:14.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.230+0000 7fdfb4a00640 1 --2- 172.21.15.27:0/2361062258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfac15e730 0x7fdfac15c7c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fdfa401d250 tx=0x7fdfa401b940 comp rx=0 tx=0).stop 2024-01-15T18:09:14.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.230+0000 7fdfb4a00640 1 -- 172.21.15.27:0/2361062258 shutdown_connections 2024-01-15T18:09:14.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.230+0000 7fdfb4a00640 1 --2- 172.21.15.27:0/2361062258 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdf84066e90 0x7fdf84069350 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.230+0000 7fdfb4a00640 1 --2- 172.21.15.27:0/2361062258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfac15e730 0x7fdfac15c7c0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.230+0000 7fdfb4a00640 1 --2- 172.21.15.27:0/2361062258 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdfac0cdd10 0x7fdfac0c8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.230+0000 7fdfb4a00640 1 --2- 172.21.15.27:0/2361062258 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdfac06db80 0x7fdfac0c6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.230+0000 7fdfb4a00640 1 -- 172.21.15.27:0/2361062258 >> 172.21.15.27:0/2361062258 conn(0x7fdfac0cec60 msgr2=0x7fdfac162fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:14.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.230+0000 7fdfb4a00640 1 -- 172.21.15.27:0/2361062258 shutdown_connections 2024-01-15T18:09:14.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.230+0000 7fdfb4a00640 1 -- 172.21.15.27:0/2361062258 wait complete. 2024-01-15T18:09:14.257 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-01-15T18:09:14.206380+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592196,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":193,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":289120,"kb_used_data":6384,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754685600,"statfs":{"total":773094113280,"available":772798054400,"internally_reserved":0,"allocated":6537216,"data_stored":3391187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":289394224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1828,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001226"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.174696+0000","last_change":"2024-01-15T18:09:10.164426+0000","last_active":"2024-01-15T18:09:10.174696+0000","last_peered":"2024-01-15T18:09:10.174696+0000","last_clean":"2024-01-15T18:09:10.174696+0000","last_became_active":"2024-01-15T18:09:10.164120+0000","last_became_peered":"2024-01-15T18:09:10.164120+0000","last_unstale":"2024-01-15T18:09:10.174696+0000","last_undegraded":"2024-01-15T18:09:10.174696+0000","last_fullsized":"2024-01-15T18:09:10.174696+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:13:02.463974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.156544+0000","last_change":"2024-01-15T18:09:10.149693+0000","last_active":"2024-01-15T18:09:10.156544+0000","last_peered":"2024-01-15T18:09:10.156544+0000","last_clean":"2024-01-15T18:09:10.156544+0000","last_became_active":"2024-01-15T18:09:10.149380+0000","last_became_peered":"2024-01-15T18:09:10.149380+0000","last_unstale":"2024-01-15T18:09:10.156544+0000","last_undegraded":"2024-01-15T18:09:10.156544+0000","last_fullsized":"2024-01-15T18:09:10.156544+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:14:10.432447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.176732+0000","last_change":"2024-01-15T18:09:10.172281+0000","last_active":"2024-01-15T18:09:10.176732+0000","last_peered":"2024-01-15T18:09:10.176732+0000","last_clean":"2024-01-15T18:09:10.176732+0000","last_became_active":"2024-01-15T18:09:10.171520+0000","last_became_peered":"2024-01-15T18:09:10.171520+0000","last_unstale":"2024-01-15T18:09:10.176732+0000","last_undegraded":"2024-01-15T18:09:10.176732+0000","last_fullsized":"2024-01-15T18:09:10.176732+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:05.433977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155356+0000","last_change":"2024-01-15T18:09:10.154186+0000","last_active":"2024-01-15T18:09:11.155356+0000","last_peered":"2024-01-15T18:09:11.155356+0000","last_clean":"2024-01-15T18:09:11.155356+0000","last_became_active":"2024-01-15T18:09:10.153961+0000","last_became_peered":"2024-01-15T18:09:10.153961+0000","last_unstale":"2024-01-15T18:09:11.155356+0000","last_undegraded":"2024-01-15T18:09:11.155356+0000","last_fullsized":"2024-01-15T18:09:11.155356+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:44:11.424905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163531+0000","last_change":"2024-01-15T18:09:10.149056+0000","last_active":"2024-01-15T18:09:10.163531+0000","last_peered":"2024-01-15T18:09:10.163531+0000","last_clean":"2024-01-15T18:09:10.163531+0000","last_became_active":"2024-01-15T18:09:10.148756+0000","last_became_peered":"2024-01-15T18:09:10.148756+0000","last_unstale":"2024-01-15T18:09:10.163531+0000","last_undegraded":"2024-01-15T18:09:10.163531+0000","last_fullsized":"2024-01-15T18:09:10.163531+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:20.006996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.127717+0000","last_change":"2024-01-15T18:09:09.127717+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.127717+0000","last_undegraded":"2024-01-15T18:09:09.127717+0000","last_fullsized":"2024-01-15T18:09:09.127717+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.149739+0000","last_change":"2024-01-15T18:09:10.147574+0000","last_active":"2024-01-15T18:09:10.149739+0000","last_peered":"2024-01-15T18:09:10.149739+0000","last_clean":"2024-01-15T18:09:10.149739+0000","last_became_active":"2024-01-15T18:09:10.147288+0000","last_became_peered":"2024-01-15T18:09:10.147288+0000","last_unstale":"2024-01-15T18:09:10.149739+0000","last_undegraded":"2024-01-15T18:09:10.149739+0000","last_fullsized":"2024-01-15T18:09:10.149739+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:29:52.274001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.160773+0000","last_change":"2024-01-15T18:09:10.149706+0000","last_active":"2024-01-15T18:09:10.160773+0000","last_peered":"2024-01-15T18:09:10.160773+0000","last_clean":"2024-01-15T18:09:10.160773+0000","last_became_active":"2024-01-15T18:09:10.149353+0000","last_became_peered":"2024-01-15T18:09:10.149353+0000","last_unstale":"2024-01-15T18:09:10.160773+0000","last_undegraded":"2024-01-15T18:09:10.160773+0000","last_fullsized":"2024-01-15T18:09:10.160773+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:22:35.521714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.166664+0000","last_change":"2024-01-15T18:09:10.162835+0000","last_active":"2024-01-15T18:09:10.166664+0000","last_peered":"2024-01-15T18:09:10.166664+0000","last_clean":"2024-01-15T18:09:10.166664+0000","last_became_active":"2024-01-15T18:09:10.162459+0000","last_became_peered":"2024-01-15T18:09:10.162459+0000","last_unstale":"2024-01-15T18:09:10.166664+0000","last_undegraded":"2024-01-15T18:09:10.166664+0000","last_fullsized":"2024-01-15T18:09:10.166664+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:26.052280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.164826+0000","last_change":"2024-01-15T18:09:10.151168+0000","last_active":"2024-01-15T18:09:10.164826+0000","last_peered":"2024-01-15T18:09:10.164826+0000","last_clean":"2024-01-15T18:09:10.164826+0000","last_became_active":"2024-01-15T18:09:10.150812+0000","last_became_peered":"2024-01-15T18:09:10.150812+0000","last_unstale":"2024-01-15T18:09:10.164826+0000","last_undegraded":"2024-01-15T18:09:10.164826+0000","last_fullsized":"2024-01-15T18:09:10.164826+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:09:59.563201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.157867+0000","last_change":"2024-01-15T18:09:10.148209+0000","last_active":"2024-01-15T18:09:10.157867+0000","last_peered":"2024-01-15T18:09:10.157867+0000","last_clean":"2024-01-15T18:09:10.157867+0000","last_became_active":"2024-01-15T18:09:10.147897+0000","last_became_peered":"2024-01-15T18:09:10.147897+0000","last_unstale":"2024-01-15T18:09:10.157867+0000","last_undegraded":"2024-01-15T18:09:10.157867+0000","last_fullsized":"2024-01-15T18:09:10.157867+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:53.437883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155548+0000","last_change":"2024-01-15T18:09:10.153400+0000","last_active":"2024-01-15T18:09:11.155548+0000","last_peered":"2024-01-15T18:09:11.155548+0000","last_clean":"2024-01-15T18:09:11.155548+0000","last_became_active":"2024-01-15T18:09:10.152222+0000","last_became_peered":"2024-01-15T18:09:10.152222+0000","last_unstale":"2024-01-15T18:09:11.155548+0000","last_undegraded":"2024-01-15T18:09:11.155548+0000","last_fullsized":"2024-01-15T18:09:11.155548+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:03.126844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.165807+0000","last_change":"2024-01-15T18:09:10.160925+0000","last_active":"2024-01-15T18:09:10.165807+0000","last_peered":"2024-01-15T18:09:10.165807+0000","last_clean":"2024-01-15T18:09:10.165807+0000","last_became_active":"2024-01-15T18:09:10.160620+0000","last_became_peered":"2024-01-15T18:09:10.160620+0000","last_unstale":"2024-01-15T18:09:10.165807+0000","last_undegraded":"2024-01-15T18:09:10.165807+0000","last_fullsized":"2024-01-15T18:09:10.165807+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:40:15.227589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.166438+0000","last_change":"2024-01-15T18:09:10.154922+0000","last_active":"2024-01-15T18:09:10.166438+0000","last_peered":"2024-01-15T18:09:10.166438+0000","last_clean":"2024-01-15T18:09:10.166438+0000","last_became_active":"2024-01-15T18:09:10.154594+0000","last_became_peered":"2024-01-15T18:09:10.154594+0000","last_unstale":"2024-01-15T18:09:10.166438+0000","last_undegraded":"2024-01-15T18:09:10.166438+0000","last_fullsized":"2024-01-15T18:09:10.166438+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:55:44.009610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.148569+0000","last_change":"2024-01-15T18:09:10.143163+0000","last_active":"2024-01-15T18:09:10.148569+0000","last_peered":"2024-01-15T18:09:10.148569+0000","last_clean":"2024-01-15T18:09:10.148569+0000","last_became_active":"2024-01-15T18:09:10.142874+0000","last_became_peered":"2024-01-15T18:09:10.142874+0000","last_unstale":"2024-01-15T18:09:10.148569+0000","last_undegraded":"2024-01-15T18:09:10.148569+0000","last_fullsized":"2024-01-15T18:09:10.148569+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:02:56.938951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.127271+0000","last_change":"2024-01-15T18:09:09.127271+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.127271+0000","last_undegraded":"2024-01-15T18:09:09.127271+0000","last_fullsized":"2024-01-15T18:09:09.127271+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163654+0000","last_change":"2024-01-15T18:09:10.156152+0000","last_active":"2024-01-15T18:09:10.163654+0000","last_peered":"2024-01-15T18:09:10.163654+0000","last_clean":"2024-01-15T18:09:10.163654+0000","last_became_active":"2024-01-15T18:09:10.155849+0000","last_became_peered":"2024-01-15T18:09:10.155849+0000","last_unstale":"2024-01-15T18:09:10.163654+0000","last_undegraded":"2024-01-15T18:09:10.163654+0000","last_fullsized":"2024-01-15T18:09:10.163654+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:53.373020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.177069+0000","last_change":"2024-01-15T18:09:10.168066+0000","last_active":"2024-01-15T18:09:10.177069+0000","last_peered":"2024-01-15T18:09:10.177069+0000","last_clean":"2024-01-15T18:09:10.177069+0000","last_became_active":"2024-01-15T18:09:10.167691+0000","last_became_peered":"2024-01-15T18:09:10.167691+0000","last_unstale":"2024-01-15T18:09:10.177069+0000","last_undegraded":"2024-01-15T18:09:10.177069+0000","last_fullsized":"2024-01-15T18:09:10.177069+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:58:26.855862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.174836+0000","last_change":"2024-01-15T18:09:10.163790+0000","last_active":"2024-01-15T18:09:10.174836+0000","last_peered":"2024-01-15T18:09:10.174836+0000","last_clean":"2024-01-15T18:09:10.174836+0000","last_became_active":"2024-01-15T18:09:10.163509+0000","last_became_peered":"2024-01-15T18:09:10.163509+0000","last_unstale":"2024-01-15T18:09:10.174836+0000","last_undegraded":"2024-01-15T18:09:10.174836+0000","last_fullsized":"2024-01-15T18:09:10.174836+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:22:24.247456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173893+0000","last_change":"2024-01-15T18:09:10.161978+0000","last_active":"2024-01-15T18:09:10.173893+0000","last_peered":"2024-01-15T18:09:10.173893+0000","last_clean":"2024-01-15T18:09:10.173893+0000","last_became_active":"2024-01-15T18:09:10.161717+0000","last_became_peered":"2024-01-15T18:09:10.161717+0000","last_unstale":"2024-01-15T18:09:10.173893+0000","last_undegraded":"2024-01-15T18:09:10.173893+0000","last_fullsized":"2024-01-15T18:09:10.173893+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:05.433977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163131+0000","last_change":"2024-01-15T18:09:10.159447+0000","last_active":"2024-01-15T18:09:10.163131+0000","last_peered":"2024-01-15T18:09:10.163131+0000","last_clean":"2024-01-15T18:09:10.163131+0000","last_became_active":"2024-01-15T18:09:10.159055+0000","last_became_peered":"2024-01-15T18:09:10.159055+0000","last_unstale":"2024-01-15T18:09:10.163131+0000","last_undegraded":"2024-01-15T18:09:10.163131+0000","last_fullsized":"2024-01-15T18:09:10.163131+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:54:09.462802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173806+0000","last_change":"2024-01-15T18:09:10.157352+0000","last_active":"2024-01-15T18:09:10.173806+0000","last_peered":"2024-01-15T18:09:10.173806+0000","last_clean":"2024-01-15T18:09:10.173806+0000","last_became_active":"2024-01-15T18:09:10.155991+0000","last_became_peered":"2024-01-15T18:09:10.155991+0000","last_unstale":"2024-01-15T18:09:10.173806+0000","last_undegraded":"2024-01-15T18:09:10.173806+0000","last_fullsized":"2024-01-15T18:09:10.173806+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:25:19.060578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163476+0000","last_change":"2024-01-15T18:09:10.160338+0000","last_active":"2024-01-15T18:09:10.163476+0000","last_peered":"2024-01-15T18:09:10.163476+0000","last_clean":"2024-01-15T18:09:10.163476+0000","last_became_active":"2024-01-15T18:09:10.159325+0000","last_became_peered":"2024-01-15T18:09:10.159325+0000","last_unstale":"2024-01-15T18:09:10.163476+0000","last_undegraded":"2024-01-15T18:09:10.163476+0000","last_fullsized":"2024-01-15T18:09:10.163476+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:46:36.631105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.168684+0000","last_change":"2024-01-15T18:09:10.167327+0000","last_active":"2024-01-15T18:09:10.168684+0000","last_peered":"2024-01-15T18:09:10.168684+0000","last_clean":"2024-01-15T18:09:10.168684+0000","last_became_active":"2024-01-15T18:09:10.167008+0000","last_became_peered":"2024-01-15T18:09:10.167008+0000","last_unstale":"2024-01-15T18:09:10.168684+0000","last_undegraded":"2024-01-15T18:09:10.168684+0000","last_fullsized":"2024-01-15T18:09:10.168684+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:18:13.449428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.174731+0000","last_change":"2024-01-15T18:09:10.165962+0000","last_active":"2024-01-15T18:09:10.174731+0000","last_peered":"2024-01-15T18:09:10.174731+0000","last_clean":"2024-01-15T18:09:10.174731+0000","last_became_active":"2024-01-15T18:09:10.165689+0000","last_became_peered":"2024-01-15T18:09:10.165689+0000","last_unstale":"2024-01-15T18:09:10.174731+0000","last_undegraded":"2024-01-15T18:09:10.174731+0000","last_fullsized":"2024-01-15T18:09:10.174731+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:48:02.569878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155565+0000","last_change":"2024-01-15T18:09:10.147816+0000","last_active":"2024-01-15T18:09:11.155565+0000","last_peered":"2024-01-15T18:09:11.155565+0000","last_clean":"2024-01-15T18:09:11.155565+0000","last_became_active":"2024-01-15T18:09:10.147487+0000","last_became_peered":"2024-01-15T18:09:10.147487+0000","last_unstale":"2024-01-15T18:09:11.155565+0000","last_undegraded":"2024-01-15T18:09:11.155565+0000","last_fullsized":"2024-01-15T18:09:11.155565+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:40:51.237238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.133285+0000","last_change":"2024-01-15T18:09:09.133285+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.133285+0000","last_undegraded":"2024-01-15T18:09:09.133285+0000","last_fullsized":"2024-01-15T18:09:09.133285+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.156169+0000","last_change":"2024-01-15T18:09:10.149120+0000","last_active":"2024-01-15T18:09:10.156169+0000","last_peered":"2024-01-15T18:09:10.156169+0000","last_clean":"2024-01-15T18:09:10.156169+0000","last_became_active":"2024-01-15T18:09:10.148675+0000","last_became_peered":"2024-01-15T18:09:10.148675+0000","last_unstale":"2024-01-15T18:09:10.156169+0000","last_undegraded":"2024-01-15T18:09:10.156169+0000","last_fullsized":"2024-01-15T18:09:10.156169+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:52:23.460592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173871+0000","last_change":"2024-01-15T18:09:10.161188+0000","last_active":"2024-01-15T18:09:10.173871+0000","last_peered":"2024-01-15T18:09:10.173871+0000","last_clean":"2024-01-15T18:09:10.173871+0000","last_became_active":"2024-01-15T18:09:10.151645+0000","last_became_peered":"2024-01-15T18:09:10.151645+0000","last_unstale":"2024-01-15T18:09:10.173871+0000","last_undegraded":"2024-01-15T18:09:10.173871+0000","last_fullsized":"2024-01-15T18:09:10.173871+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:07:18.177097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.176602+0000","last_change":"2024-01-15T18:09:10.167895+0000","last_active":"2024-01-15T18:09:10.176602+0000","last_peered":"2024-01-15T18:09:10.176602+0000","last_clean":"2024-01-15T18:09:10.176602+0000","last_became_active":"2024-01-15T18:09:10.167639+0000","last_became_peered":"2024-01-15T18:09:10.167639+0000","last_unstale":"2024-01-15T18:09:10.176602+0000","last_undegraded":"2024-01-15T18:09:10.176602+0000","last_fullsized":"2024-01-15T18:09:10.176602+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:18:53.201764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.161004+0000","last_change":"2024-01-15T18:09:10.148253+0000","last_active":"2024-01-15T18:09:10.161004+0000","last_peered":"2024-01-15T18:09:10.161004+0000","last_clean":"2024-01-15T18:09:10.161004+0000","last_became_active":"2024-01-15T18:09:10.147941+0000","last_became_peered":"2024-01-15T18:09:10.147941+0000","last_unstale":"2024-01-15T18:09:10.161004+0000","last_undegraded":"2024-01-15T18:09:10.161004+0000","last_fullsized":"2024-01-15T18:09:10.161004+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:04:16.316753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.126719+0000","last_change":"2024-01-15T18:09:09.126719+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.126719+0000","last_undegraded":"2024-01-15T18:09:09.126719+0000","last_fullsized":"2024-01-15T18:09:09.126719+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.138025+0000","last_change":"2024-01-15T18:09:09.138025+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.138025+0000","last_undegraded":"2024-01-15T18:09:09.138025+0000","last_fullsized":"2024-01-15T18:09:09.138025+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.155581+0000","last_change":"2024-01-15T18:09:09.152802+0000","last_active":"2024-01-15T18:09:09.155581+0000","last_peered":"2024-01-15T18:09:09.155581+0000","last_clean":"2024-01-15T18:09:09.155581+0000","last_became_active":"2024-01-15T18:09:09.152520+0000","last_became_peered":"2024-01-15T18:09:09.152520+0000","last_unstale":"2024-01-15T18:09:09.155581+0000","last_undegraded":"2024-01-15T18:09:09.155581+0000","last_fullsized":"2024-01-15T18:09:09.155581+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:04.586121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.170556+0000","last_change":"2024-01-15T18:09:10.161734+0000","last_active":"2024-01-15T18:09:10.170556+0000","last_peered":"2024-01-15T18:09:10.170556+0000","last_clean":"2024-01-15T18:09:10.170556+0000","last_became_active":"2024-01-15T18:09:10.161466+0000","last_became_peered":"2024-01-15T18:09:10.161466+0000","last_unstale":"2024-01-15T18:09:10.170556+0000","last_undegraded":"2024-01-15T18:09:10.170556+0000","last_fullsized":"2024-01-15T18:09:10.170556+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:58:24.569904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"57'1","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.177728+0000","last_change":"2024-01-15T18:09:09.125625+0000","last_active":"2024-01-15T18:09:10.177728+0000","last_peered":"2024-01-15T18:09:10.177728+0000","last_clean":"2024-01-15T18:09:10.177728+0000","last_became_active":"2024-01-15T18:09:09.125373+0000","last_became_peered":"2024-01-15T18:09:09.125373+0000","last_unstale":"2024-01-15T18:09:10.177728+0000","last_undegraded":"2024-01-15T18:09:10.177728+0000","last_fullsized":"2024-01-15T18:09:10.177728+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:14:59.883458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.130696+0000","last_change":"2024-01-15T18:09:09.130696+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.130696+0000","last_undegraded":"2024-01-15T18:09:09.130696+0000","last_fullsized":"2024-01-15T18:09:09.130696+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.155587+0000","last_change":"2024-01-15T18:09:09.153007+0000","last_active":"2024-01-15T18:09:09.155587+0000","last_peered":"2024-01-15T18:09:09.155587+0000","last_clean":"2024-01-15T18:09:09.155587+0000","last_became_active":"2024-01-15T18:09:09.152405+0000","last_became_peered":"2024-01-15T18:09:09.152405+0000","last_unstale":"2024-01-15T18:09:09.155587+0000","last_undegraded":"2024-01-15T18:09:09.155587+0000","last_fullsized":"2024-01-15T18:09:09.155587+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:53:06.843401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.132775+0000","last_change":"2024-01-15T18:09:09.132775+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.132775+0000","last_undegraded":"2024-01-15T18:09:09.132775+0000","last_fullsized":"2024-01-15T18:09:09.132775+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.176876+0000","last_change":"2024-01-15T18:09:09.138209+0000","last_active":"2024-01-15T18:09:10.176876+0000","last_peered":"2024-01-15T18:09:10.176876+0000","last_clean":"2024-01-15T18:09:10.176876+0000","last_became_active":"2024-01-15T18:09:09.137921+0000","last_became_peered":"2024-01-15T18:09:09.137921+0000","last_unstale":"2024-01-15T18:09:10.176876+0000","last_undegraded":"2024-01-15T18:09:10.176876+0000","last_fullsized":"2024-01-15T18:09:10.176876+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:54:56.016726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163688+0000","last_change":"2024-01-15T18:09:10.156211+0000","last_active":"2024-01-15T18:09:10.163688+0000","last_peered":"2024-01-15T18:09:10.163688+0000","last_clean":"2024-01-15T18:09:10.163688+0000","last_became_active":"2024-01-15T18:09:10.155914+0000","last_became_peered":"2024-01-15T18:09:10.155914+0000","last_unstale":"2024-01-15T18:09:10.163688+0000","last_undegraded":"2024-01-15T18:09:10.163688+0000","last_fullsized":"2024-01-15T18:09:10.163688+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:15:38.206084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"57'1","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.172500+0000","last_change":"2024-01-15T18:09:09.139963+0000","last_active":"2024-01-15T18:09:10.172500+0000","last_peered":"2024-01-15T18:09:10.172500+0000","last_clean":"2024-01-15T18:09:10.172500+0000","last_became_active":"2024-01-15T18:09:09.139603+0000","last_became_peered":"2024-01-15T18:09:09.139603+0000","last_unstale":"2024-01-15T18:09:10.172500+0000","last_undegraded":"2024-01-15T18:09:10.172500+0000","last_fullsized":"2024-01-15T18:09:10.172500+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:14:01.036199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.174128+0000","last_change":"2024-01-15T18:09:10.165154+0000","last_active":"2024-01-15T18:09:10.174128+0000","last_peered":"2024-01-15T18:09:10.174128+0000","last_clean":"2024-01-15T18:09:10.174128+0000","last_became_active":"2024-01-15T18:09:10.164900+0000","last_became_peered":"2024-01-15T18:09:10.164900+0000","last_unstale":"2024-01-15T18:09:10.174128+0000","last_undegraded":"2024-01-15T18:09:10.174128+0000","last_fullsized":"2024-01-15T18:09:10.174128+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:57:26.029056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.153437+0000","last_change":"2024-01-15T18:09:09.147874+0000","last_active":"2024-01-15T18:09:09.153437+0000","last_peered":"2024-01-15T18:09:09.153437+0000","last_clean":"2024-01-15T18:09:09.153437+0000","last_became_active":"2024-01-15T18:09:09.147666+0000","last_became_peered":"2024-01-15T18:09:09.147666+0000","last_unstale":"2024-01-15T18:09:09.153437+0000","last_undegraded":"2024-01-15T18:09:09.153437+0000","last_fullsized":"2024-01-15T18:09:09.153437+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:44:37.388258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163308+0000","last_change":"2024-01-15T18:09:10.154886+0000","last_active":"2024-01-15T18:09:10.163308+0000","last_peered":"2024-01-15T18:09:10.163308+0000","last_clean":"2024-01-15T18:09:10.163308+0000","last_became_active":"2024-01-15T18:09:10.154594+0000","last_became_peered":"2024-01-15T18:09:10.154594+0000","last_unstale":"2024-01-15T18:09:10.163308+0000","last_undegraded":"2024-01-15T18:09:10.163308+0000","last_fullsized":"2024-01-15T18:09:10.163308+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:34:09.142491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.137679+0000","last_change":"2024-01-15T18:09:09.140948+0000","last_active":"2024-01-15T18:09:10.137679+0000","last_peered":"2024-01-15T18:09:10.137679+0000","last_clean":"2024-01-15T18:09:10.137679+0000","last_became_active":"2024-01-15T18:09:09.140010+0000","last_became_peered":"2024-01-15T18:09:09.140010+0000","last_unstale":"2024-01-15T18:09:10.137679+0000","last_undegraded":"2024-01-15T18:09:10.137679+0000","last_fullsized":"2024-01-15T18:09:10.137679+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:39:25.682654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173959+0000","last_change":"2024-01-15T18:09:10.160588+0000","last_active":"2024-01-15T18:09:10.173959+0000","last_peered":"2024-01-15T18:09:10.173959+0000","last_clean":"2024-01-15T18:09:10.173959+0000","last_became_active":"2024-01-15T18:09:10.157197+0000","last_became_peered":"2024-01-15T18:09:10.157197+0000","last_unstale":"2024-01-15T18:09:10.173959+0000","last_undegraded":"2024-01-15T18:09:10.173959+0000","last_fullsized":"2024-01-15T18:09:10.173959+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:27:59.969234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.138903+0000","last_change":"2024-01-15T18:09:09.131170+0000","last_active":"2024-01-15T18:09:10.138903+0000","last_peered":"2024-01-15T18:09:10.138903+0000","last_clean":"2024-01-15T18:09:10.138903+0000","last_became_active":"2024-01-15T18:09:09.130789+0000","last_became_peered":"2024-01-15T18:09:09.130789+0000","last_unstale":"2024-01-15T18:09:10.138903+0000","last_undegraded":"2024-01-15T18:09:10.138903+0000","last_fullsized":"2024-01-15T18:09:10.138903+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:13:26.979622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.166623+0000","last_change":"2024-01-15T18:09:10.154127+0000","last_active":"2024-01-15T18:09:10.166623+0000","last_peered":"2024-01-15T18:09:10.166623+0000","last_clean":"2024-01-15T18:09:10.166623+0000","last_became_active":"2024-01-15T18:09:10.153699+0000","last_became_peered":"2024-01-15T18:09:10.153699+0000","last_unstale":"2024-01-15T18:09:10.166623+0000","last_undegraded":"2024-01-15T18:09:10.166623+0000","last_fullsized":"2024-01-15T18:09:10.166623+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:37:29.585671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.151155+0000","last_change":"2024-01-15T18:09:09.145070+0000","last_active":"2024-01-15T18:09:10.151155+0000","last_peered":"2024-01-15T18:09:10.151155+0000","last_clean":"2024-01-15T18:09:10.151155+0000","last_became_active":"2024-01-15T18:09:09.144808+0000","last_became_peered":"2024-01-15T18:09:09.144808+0000","last_unstale":"2024-01-15T18:09:10.151155+0000","last_undegraded":"2024-01-15T18:09:10.151155+0000","last_fullsized":"2024-01-15T18:09:10.151155+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:07:17.174892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.178747+0000","last_change":"2024-01-15T18:09:10.173359+0000","last_active":"2024-01-15T18:09:10.178747+0000","last_peered":"2024-01-15T18:09:10.178747+0000","last_clean":"2024-01-15T18:09:10.178747+0000","last_became_active":"2024-01-15T18:09:10.173020+0000","last_became_peered":"2024-01-15T18:09:10.173020+0000","last_unstale":"2024-01-15T18:09:10.178747+0000","last_undegraded":"2024-01-15T18:09:10.178747+0000","last_fullsized":"2024-01-15T18:09:10.178747+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:22:35.521714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.140713+0000","last_change":"2024-01-15T18:09:09.142076+0000","last_active":"2024-01-15T18:09:10.140713+0000","last_peered":"2024-01-15T18:09:10.140713+0000","last_clean":"2024-01-15T18:09:10.140713+0000","last_became_active":"2024-01-15T18:09:09.141747+0000","last_became_peered":"2024-01-15T18:09:09.141747+0000","last_unstale":"2024-01-15T18:09:10.140713+0000","last_undegraded":"2024-01-15T18:09:10.140713+0000","last_fullsized":"2024-01-15T18:09:10.140713+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:02.592720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.165189+0000","last_change":"2024-01-15T18:09:10.150403+0000","last_active":"2024-01-15T18:09:10.165189+0000","last_peered":"2024-01-15T18:09:10.165189+0000","last_clean":"2024-01-15T18:09:10.165189+0000","last_became_active":"2024-01-15T18:09:10.150113+0000","last_became_peered":"2024-01-15T18:09:10.150113+0000","last_unstale":"2024-01-15T18:09:10.165189+0000","last_undegraded":"2024-01-15T18:09:10.165189+0000","last_fullsized":"2024-01-15T18:09:10.165189+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:45:59.978009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"57'3","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.180373+0000","last_change":"2024-01-15T18:09:09.142251+0000","last_active":"2024-01-15T18:09:10.180373+0000","last_peered":"2024-01-15T18:09:10.180373+0000","last_clean":"2024-01-15T18:09:10.180373+0000","last_became_active":"2024-01-15T18:09:09.141961+0000","last_became_peered":"2024-01-15T18:09:09.141961+0000","last_unstale":"2024-01-15T18:09:10.180373+0000","last_undegraded":"2024-01-15T18:09:10.180373+0000","last_fullsized":"2024-01-15T18:09:10.180373+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:56:33.327612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.176749+0000","last_change":"2024-01-15T18:09:10.172402+0000","last_active":"2024-01-15T18:09:10.176749+0000","last_peered":"2024-01-15T18:09:10.176749+0000","last_clean":"2024-01-15T18:09:10.176749+0000","last_became_active":"2024-01-15T18:09:10.171813+0000","last_became_peered":"2024-01-15T18:09:10.171813+0000","last_unstale":"2024-01-15T18:09:10.176749+0000","last_undegraded":"2024-01-15T18:09:10.176749+0000","last_fullsized":"2024-01-15T18:09:10.176749+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:15:38.206084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"57'6","reported_seq":17,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.180712+0000","last_change":"2024-01-15T18:09:09.147769+0000","last_active":"2024-01-15T18:09:10.180712+0000","last_peered":"2024-01-15T18:09:10.180712+0000","last_clean":"2024-01-15T18:09:10.180712+0000","last_became_active":"2024-01-15T18:09:09.147356+0000","last_became_peered":"2024-01-15T18:09:09.147356+0000","last_unstale":"2024-01-15T18:09:10.180712+0000","last_undegraded":"2024-01-15T18:09:10.180712+0000","last_fullsized":"2024-01-15T18:09:10.180712+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:18:52.199559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155288+0000","last_change":"2024-01-15T18:09:10.154188+0000","last_active":"2024-01-15T18:09:11.155288+0000","last_peered":"2024-01-15T18:09:11.155288+0000","last_clean":"2024-01-15T18:09:11.155288+0000","last_became_active":"2024-01-15T18:09:10.153881+0000","last_became_peered":"2024-01-15T18:09:10.153881+0000","last_unstale":"2024-01-15T18:09:11.155288+0000","last_undegraded":"2024-01-15T18:09:11.155288+0000","last_fullsized":"2024-01-15T18:09:11.155288+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:57:04.284081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"57'2","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.180721+0000","last_change":"2024-01-15T18:09:09.141986+0000","last_active":"2024-01-15T18:09:10.180721+0000","last_peered":"2024-01-15T18:09:10.180721+0000","last_clean":"2024-01-15T18:09:10.180721+0000","last_became_active":"2024-01-15T18:09:09.141626+0000","last_became_peered":"2024-01-15T18:09:09.141626+0000","last_unstale":"2024-01-15T18:09:10.180721+0000","last_undegraded":"2024-01-15T18:09:10.180721+0000","last_fullsized":"2024-01-15T18:09:10.180721+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:44:08.042896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.176175+0000","last_change":"2024-01-15T18:09:10.163997+0000","last_active":"2024-01-15T18:09:10.176175+0000","last_peered":"2024-01-15T18:09:10.176175+0000","last_clean":"2024-01-15T18:09:10.176175+0000","last_became_active":"2024-01-15T18:09:10.163758+0000","last_became_peered":"2024-01-15T18:09:10.163758+0000","last_unstale":"2024-01-15T18:09:10.176175+0000","last_undegraded":"2024-01-15T18:09:10.176175+0000","last_fullsized":"2024-01-15T18:09:10.176175+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:02:59.936727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.144084+0000","last_change":"2024-01-15T18:09:09.142040+0000","last_active":"2024-01-15T18:09:10.144084+0000","last_peered":"2024-01-15T18:09:10.144084+0000","last_clean":"2024-01-15T18:09:10.144084+0000","last_became_active":"2024-01-15T18:09:09.141713+0000","last_became_peered":"2024-01-15T18:09:09.141713+0000","last_unstale":"2024-01-15T18:09:10.144084+0000","last_undegraded":"2024-01-15T18:09:10.144084+0000","last_fullsized":"2024-01-15T18:09:10.144084+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:17:15.118577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.166494+0000","last_change":"2024-01-15T18:09:10.156574+0000","last_active":"2024-01-15T18:09:10.166494+0000","last_peered":"2024-01-15T18:09:10.166494+0000","last_clean":"2024-01-15T18:09:10.166494+0000","last_became_active":"2024-01-15T18:09:10.156225+0000","last_became_peered":"2024-01-15T18:09:10.156225+0000","last_unstale":"2024-01-15T18:09:10.166494+0000","last_undegraded":"2024-01-15T18:09:10.166494+0000","last_fullsized":"2024-01-15T18:09:10.166494+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:15:39.864433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.137769+0000","last_change":"2024-01-15T18:09:09.135271+0000","last_active":"2024-01-15T18:09:10.137769+0000","last_peered":"2024-01-15T18:09:10.137769+0000","last_clean":"2024-01-15T18:09:10.137769+0000","last_became_active":"2024-01-15T18:09:09.135011+0000","last_became_peered":"2024-01-15T18:09:09.135011+0000","last_unstale":"2024-01-15T18:09:10.137769+0000","last_undegraded":"2024-01-15T18:09:10.137769+0000","last_fullsized":"2024-01-15T18:09:10.137769+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:29:02.803967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163455+0000","last_change":"2024-01-15T18:09:10.160727+0000","last_active":"2024-01-15T18:09:10.163455+0000","last_peered":"2024-01-15T18:09:10.163455+0000","last_clean":"2024-01-15T18:09:10.163455+0000","last_became_active":"2024-01-15T18:09:10.160132+0000","last_became_peered":"2024-01-15T18:09:10.160132+0000","last_unstale":"2024-01-15T18:09:10.163455+0000","last_undegraded":"2024-01-15T18:09:10.163455+0000","last_fullsized":"2024-01-15T18:09:10.163455+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:27:59.969234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"57'8","reported_seq":19,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.177242+0000","last_change":"2024-01-15T18:09:09.137932+0000","last_active":"2024-01-15T18:09:10.177242+0000","last_peered":"2024-01-15T18:09:10.177242+0000","last_clean":"2024-01-15T18:09:10.177242+0000","last_became_active":"2024-01-15T18:09:09.137626+0000","last_became_peered":"2024-01-15T18:09:09.137626+0000","last_unstale":"2024-01-15T18:09:10.177242+0000","last_undegraded":"2024-01-15T18:09:10.177242+0000","last_fullsized":"2024-01-15T18:09:10.177242+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:14:01.036199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.177049+0000","last_change":"2024-01-15T18:09:10.165126+0000","last_active":"2024-01-15T18:09:10.177049+0000","last_peered":"2024-01-15T18:09:10.177049+0000","last_clean":"2024-01-15T18:09:10.177049+0000","last_became_active":"2024-01-15T18:09:10.164811+0000","last_became_peered":"2024-01-15T18:09:10.164811+0000","last_unstale":"2024-01-15T18:09:10.177049+0000","last_undegraded":"2024-01-15T18:09:10.177049+0000","last_fullsized":"2024-01-15T18:09:10.177049+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:53:41.639339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"57'1","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.180173+0000","last_change":"2024-01-15T18:09:09.137481+0000","last_active":"2024-01-15T18:09:10.180173+0000","last_peered":"2024-01-15T18:09:10.180173+0000","last_clean":"2024-01-15T18:09:10.180173+0000","last_became_active":"2024-01-15T18:09:09.137205+0000","last_became_peered":"2024-01-15T18:09:09.137205+0000","last_unstale":"2024-01-15T18:09:10.180173+0000","last_undegraded":"2024-01-15T18:09:10.180173+0000","last_fullsized":"2024-01-15T18:09:10.180173+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:16:45.705931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155989+0000","last_change":"2024-01-15T18:09:10.166022+0000","last_active":"2024-01-15T18:09:11.155989+0000","last_peered":"2024-01-15T18:09:11.155989+0000","last_clean":"2024-01-15T18:09:11.155989+0000","last_became_active":"2024-01-15T18:09:10.165752+0000","last_became_peered":"2024-01-15T18:09:10.165752+0000","last_unstale":"2024-01-15T18:09:11.155989+0000","last_undegraded":"2024-01-15T18:09:11.155989+0000","last_fullsized":"2024-01-15T18:09:11.155989+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:45:32.407323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.155562+0000","last_change":"2024-01-15T18:09:09.152740+0000","last_active":"2024-01-15T18:09:09.155562+0000","last_peered":"2024-01-15T18:09:09.155562+0000","last_clean":"2024-01-15T18:09:09.155562+0000","last_became_active":"2024-01-15T18:09:09.152418+0000","last_became_peered":"2024-01-15T18:09:09.152418+0000","last_unstale":"2024-01-15T18:09:09.155562+0000","last_undegraded":"2024-01-15T18:09:09.155562+0000","last_fullsized":"2024-01-15T18:09:09.155562+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:56:49.381443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.166880+0000","last_change":"2024-01-15T18:09:10.155205+0000","last_active":"2024-01-15T18:09:10.166880+0000","last_peered":"2024-01-15T18:09:10.166880+0000","last_clean":"2024-01-15T18:09:10.166880+0000","last_became_active":"2024-01-15T18:09:10.154910+0000","last_became_peered":"2024-01-15T18:09:10.154910+0000","last_unstale":"2024-01-15T18:09:10.166880+0000","last_undegraded":"2024-01-15T18:09:10.166880+0000","last_fullsized":"2024-01-15T18:09:10.166880+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:56:34.329818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.177022+0000","last_change":"2024-01-15T18:09:09.141114+0000","last_active":"2024-01-15T18:09:10.177022+0000","last_peered":"2024-01-15T18:09:10.177022+0000","last_clean":"2024-01-15T18:09:10.177022+0000","last_became_active":"2024-01-15T18:09:09.140786+0000","last_became_peered":"2024-01-15T18:09:09.140786+0000","last_unstale":"2024-01-15T18:09:10.177022+0000","last_undegraded":"2024-01-15T18:09:10.177022+0000","last_fullsized":"2024-01-15T18:09:10.177022+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:37:27.921887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.176767+0000","last_change":"2024-01-15T18:09:10.172606+0000","last_active":"2024-01-15T18:09:10.176767+0000","last_peered":"2024-01-15T18:09:10.176767+0000","last_clean":"2024-01-15T18:09:10.176767+0000","last_became_active":"2024-01-15T18:09:10.172090+0000","last_became_peered":"2024-01-15T18:09:10.172090+0000","last_unstale":"2024-01-15T18:09:10.176767+0000","last_undegraded":"2024-01-15T18:09:10.176767+0000","last_fullsized":"2024-01-15T18:09:10.176767+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:25:19.060578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.153526+0000","last_change":"2024-01-15T18:09:09.148967+0000","last_active":"2024-01-15T18:09:09.153526+0000","last_peered":"2024-01-15T18:09:09.153526+0000","last_clean":"2024-01-15T18:09:09.153526+0000","last_became_active":"2024-01-15T18:09:09.148670+0000","last_became_peered":"2024-01-15T18:09:09.148670+0000","last_unstale":"2024-01-15T18:09:09.153526+0000","last_undegraded":"2024-01-15T18:09:09.153526+0000","last_fullsized":"2024-01-15T18:09:09.153526+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:04:47.742716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155444+0000","last_change":"2024-01-15T18:09:10.151767+0000","last_active":"2024-01-15T18:09:11.155444+0000","last_peered":"2024-01-15T18:09:11.155444+0000","last_clean":"2024-01-15T18:09:11.155444+0000","last_became_active":"2024-01-15T18:09:10.151462+0000","last_became_peered":"2024-01-15T18:09:10.151462+0000","last_unstale":"2024-01-15T18:09:11.155444+0000","last_undegraded":"2024-01-15T18:09:11.155444+0000","last_fullsized":"2024-01-15T18:09:11.155444+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:07:18.547932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"57'2","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.174259+0000","last_change":"2024-01-15T18:09:09.141213+0000","last_active":"2024-01-15T18:09:10.174259+0000","last_peered":"2024-01-15T18:09:10.174259+0000","last_clean":"2024-01-15T18:09:10.174259+0000","last_became_active":"2024-01-15T18:09:09.139957+0000","last_became_peered":"2024-01-15T18:09:09.139957+0000","last_unstale":"2024-01-15T18:09:10.174259+0000","last_undegraded":"2024-01-15T18:09:10.174259+0000","last_fullsized":"2024-01-15T18:09:10.174259+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:42:47.132801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.127537+0000","last_change":"2024-01-15T18:09:09.127537+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.127537+0000","last_undegraded":"2024-01-15T18:09:09.127537+0000","last_fullsized":"2024-01-15T18:09:09.127537+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.133917+0000","last_change":"2024-01-15T18:09:09.136084+0000","last_active":"2024-01-15T18:09:10.133917+0000","last_peered":"2024-01-15T18:09:10.133917+0000","last_clean":"2024-01-15T18:09:10.133917+0000","last_became_active":"2024-01-15T18:09:09.135760+0000","last_became_peered":"2024-01-15T18:09:09.135760+0000","last_unstale":"2024-01-15T18:09:10.133917+0000","last_undegraded":"2024-01-15T18:09:10.133917+0000","last_fullsized":"2024-01-15T18:09:10.133917+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:37.003640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.128633+0000","last_change":"2024-01-15T18:09:09.128633+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.128633+0000","last_undegraded":"2024-01-15T18:09:09.128633+0000","last_fullsized":"2024-01-15T18:09:09.128633+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.137490+0000","last_change":"2024-01-15T18:09:09.136145+0000","last_active":"2024-01-15T18:09:10.137490+0000","last_peered":"2024-01-15T18:09:10.137490+0000","last_clean":"2024-01-15T18:09:10.137490+0000","last_became_active":"2024-01-15T18:09:09.135807+0000","last_became_peered":"2024-01-15T18:09:09.135807+0000","last_unstale":"2024-01-15T18:09:10.137490+0000","last_undegraded":"2024-01-15T18:09:10.137490+0000","last_fullsized":"2024-01-15T18:09:10.137490+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:35:26.455284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155791+0000","last_change":"2024-01-15T18:09:10.157661+0000","last_active":"2024-01-15T18:09:11.155791+0000","last_peered":"2024-01-15T18:09:11.155791+0000","last_clean":"2024-01-15T18:09:11.155791+0000","last_became_active":"2024-01-15T18:09:10.157399+0000","last_became_peered":"2024-01-15T18:09:10.157399+0000","last_unstale":"2024-01-15T18:09:11.155791+0000","last_undegraded":"2024-01-15T18:09:11.155791+0000","last_fullsized":"2024-01-15T18:09:11.155791+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:41:56.291009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.177996+0000","last_change":"2024-01-15T18:09:09.145612+0000","last_active":"2024-01-15T18:09:10.177996+0000","last_peered":"2024-01-15T18:09:10.177996+0000","last_clean":"2024-01-15T18:09:10.177996+0000","last_became_active":"2024-01-15T18:09:09.145279+0000","last_became_peered":"2024-01-15T18:09:09.145279+0000","last_unstale":"2024-01-15T18:09:10.177996+0000","last_undegraded":"2024-01-15T18:09:10.177996+0000","last_fullsized":"2024-01-15T18:09:10.177996+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:53.930232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.156102+0000","last_change":"2024-01-15T18:09:10.153156+0000","last_active":"2024-01-15T18:09:11.156102+0000","last_peered":"2024-01-15T18:09:11.156102+0000","last_clean":"2024-01-15T18:09:11.156102+0000","last_became_active":"2024-01-15T18:09:10.152783+0000","last_became_peered":"2024-01-15T18:09:10.152783+0000","last_unstale":"2024-01-15T18:09:11.156102+0000","last_undegraded":"2024-01-15T18:09:11.156102+0000","last_fullsized":"2024-01-15T18:09:11.156102+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:36:30.872154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.155546+0000","last_change":"2024-01-15T18:09:09.152741+0000","last_active":"2024-01-15T18:09:09.155546+0000","last_peered":"2024-01-15T18:09:09.155546+0000","last_clean":"2024-01-15T18:09:09.155546+0000","last_became_active":"2024-01-15T18:09:09.152470+0000","last_became_peered":"2024-01-15T18:09:09.152470+0000","last_unstale":"2024-01-15T18:09:09.155546+0000","last_undegraded":"2024-01-15T18:09:09.155546+0000","last_fullsized":"2024-01-15T18:09:09.155546+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:14:31.566501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.153880+0000","last_change":"2024-01-15T18:09:10.143039+0000","last_active":"2024-01-15T18:09:10.153880+0000","last_peered":"2024-01-15T18:09:10.153880+0000","last_clean":"2024-01-15T18:09:10.153880+0000","last_became_active":"2024-01-15T18:09:10.142814+0000","last_became_peered":"2024-01-15T18:09:10.142814+0000","last_unstale":"2024-01-15T18:09:10.153880+0000","last_undegraded":"2024-01-15T18:09:10.153880+0000","last_fullsized":"2024-01-15T18:09:10.153880+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:54:31.215934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.143973+0000","last_change":"2024-01-15T18:09:09.141870+0000","last_active":"2024-01-15T18:09:10.143973+0000","last_peered":"2024-01-15T18:09:10.143973+0000","last_clean":"2024-01-15T18:09:10.143973+0000","last_became_active":"2024-01-15T18:09:09.141523+0000","last_became_peered":"2024-01-15T18:09:09.141523+0000","last_unstale":"2024-01-15T18:09:10.143973+0000","last_undegraded":"2024-01-15T18:09:10.143973+0000","last_fullsized":"2024-01-15T18:09:10.143973+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:15:13.572542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.132306+0000","last_change":"2024-01-15T18:09:09.132306+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.132306+0000","last_undegraded":"2024-01-15T18:09:09.132306+0000","last_fullsized":"2024-01-15T18:09:09.132306+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.137795+0000","last_change":"2024-01-15T18:09:09.136593+0000","last_active":"2024-01-15T18:09:10.137795+0000","last_peered":"2024-01-15T18:09:10.137795+0000","last_clean":"2024-01-15T18:09:10.137795+0000","last_became_active":"2024-01-15T18:09:09.136246+0000","last_became_peered":"2024-01-15T18:09:09.136246+0000","last_unstale":"2024-01-15T18:09:10.137795+0000","last_undegraded":"2024-01-15T18:09:10.137795+0000","last_fullsized":"2024-01-15T18:09:10.137795+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:10:09.158484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.155341+0000","last_change":"2024-01-15T18:09:10.145733+0000","last_active":"2024-01-15T18:09:10.155341+0000","last_peered":"2024-01-15T18:09:10.155341+0000","last_clean":"2024-01-15T18:09:10.155341+0000","last_became_active":"2024-01-15T18:09:10.145479+0000","last_became_peered":"2024-01-15T18:09:10.145479+0000","last_unstale":"2024-01-15T18:09:10.155341+0000","last_undegraded":"2024-01-15T18:09:10.155341+0000","last_fullsized":"2024-01-15T18:09:10.155341+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:53:41.639339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.133917+0000","last_change":"2024-01-15T18:09:09.142681+0000","last_active":"2024-01-15T18:09:10.133917+0000","last_peered":"2024-01-15T18:09:10.133917+0000","last_clean":"2024-01-15T18:09:10.133917+0000","last_became_active":"2024-01-15T18:09:09.142357+0000","last_became_peered":"2024-01-15T18:09:09.142357+0000","last_unstale":"2024-01-15T18:09:10.133917+0000","last_undegraded":"2024-01-15T18:09:10.133917+0000","last_fullsized":"2024-01-15T18:09:10.133917+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:41:15.076278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173119+0000","last_change":"2024-01-15T18:09:10.162125+0000","last_active":"2024-01-15T18:09:10.173119+0000","last_peered":"2024-01-15T18:09:10.173119+0000","last_clean":"2024-01-15T18:09:10.173119+0000","last_became_active":"2024-01-15T18:09:10.161778+0000","last_became_peered":"2024-01-15T18:09:10.161778+0000","last_unstale":"2024-01-15T18:09:10.173119+0000","last_undegraded":"2024-01-15T18:09:10.173119+0000","last_fullsized":"2024-01-15T18:09:10.173119+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:07:31.195306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.139632+0000","last_change":"2024-01-15T18:09:09.135449+0000","last_active":"2024-01-15T18:09:10.139632+0000","last_peered":"2024-01-15T18:09:10.139632+0000","last_clean":"2024-01-15T18:09:10.139632+0000","last_became_active":"2024-01-15T18:09:09.135010+0000","last_became_peered":"2024-01-15T18:09:09.135010+0000","last_unstale":"2024-01-15T18:09:10.139632+0000","last_undegraded":"2024-01-15T18:09:10.139632+0000","last_fullsized":"2024-01-15T18:09:10.139632+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:54:08.460596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.170500+0000","last_change":"2024-01-15T18:09:10.158452+0000","last_active":"2024-01-15T18:09:10.170500+0000","last_peered":"2024-01-15T18:09:10.170500+0000","last_clean":"2024-01-15T18:09:10.170500+0000","last_became_active":"2024-01-15T18:09:10.158267+0000","last_became_peered":"2024-01-15T18:09:10.158267+0000","last_unstale":"2024-01-15T18:09:10.170500+0000","last_undegraded":"2024-01-15T18:09:10.170500+0000","last_fullsized":"2024-01-15T18:09:10.170500+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:26:45.212614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.133918+0000","last_change":"2024-01-15T18:09:09.146561+0000","last_active":"2024-01-15T18:09:10.133918+0000","last_peered":"2024-01-15T18:09:10.133918+0000","last_clean":"2024-01-15T18:09:10.133918+0000","last_became_active":"2024-01-15T18:09:09.146263+0000","last_became_peered":"2024-01-15T18:09:09.146263+0000","last_unstale":"2024-01-15T18:09:10.133918+0000","last_undegraded":"2024-01-15T18:09:10.133918+0000","last_fullsized":"2024-01-15T18:09:10.133918+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:00:48.078792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173947+0000","last_change":"2024-01-15T18:09:10.157427+0000","last_active":"2024-01-15T18:09:10.173947+0000","last_peered":"2024-01-15T18:09:10.173947+0000","last_clean":"2024-01-15T18:09:10.173947+0000","last_became_active":"2024-01-15T18:09:10.157146+0000","last_became_peered":"2024-01-15T18:09:10.157146+0000","last_unstale":"2024-01-15T18:09:10.173947+0000","last_undegraded":"2024-01-15T18:09:10.173947+0000","last_fullsized":"2024-01-15T18:09:10.173947+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:07:31.195306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.142185+0000","last_change":"2024-01-15T18:09:09.136141+0000","last_active":"2024-01-15T18:09:10.142185+0000","last_peered":"2024-01-15T18:09:10.142185+0000","last_clean":"2024-01-15T18:09:10.142185+0000","last_became_active":"2024-01-15T18:09:09.135853+0000","last_became_peered":"2024-01-15T18:09:09.135853+0000","last_unstale":"2024-01-15T18:09:10.142185+0000","last_undegraded":"2024-01-15T18:09:10.142185+0000","last_fullsized":"2024-01-15T18:09:10.142185+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:25:50.421687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173953+0000","last_change":"2024-01-15T18:09:10.161307+0000","last_active":"2024-01-15T18:09:10.173953+0000","last_peered":"2024-01-15T18:09:10.173953+0000","last_clean":"2024-01-15T18:09:10.173953+0000","last_became_active":"2024-01-15T18:09:10.160945+0000","last_became_peered":"2024-01-15T18:09:10.160945+0000","last_unstale":"2024-01-15T18:09:10.173953+0000","last_undegraded":"2024-01-15T18:09:10.173953+0000","last_fullsized":"2024-01-15T18:09:10.173953+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:26:05.237285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.144458+0000","last_change":"2024-01-15T18:09:09.142696+0000","last_active":"2024-01-15T18:09:10.144458+0000","last_peered":"2024-01-15T18:09:10.144458+0000","last_clean":"2024-01-15T18:09:10.144458+0000","last_became_active":"2024-01-15T18:09:09.142448+0000","last_became_peered":"2024-01-15T18:09:09.142448+0000","last_unstale":"2024-01-15T18:09:10.144458+0000","last_undegraded":"2024-01-15T18:09:10.144458+0000","last_fullsized":"2024-01-15T18:09:10.144458+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:40:45.529038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":194,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.122277+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:09:09.122277+0000","last_peered":"2024-01-15T18:09:09.122277+0000","last_clean":"2024-01-15T18:09:09.122277+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:09:09.122277+0000","last_undegraded":"2024-01-15T18:09:09.122277+0000","last_fullsized":"2024-01-15T18:09:09.122277+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":1828,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":2626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509496,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36572,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335268,"statfs":{"total":96636764160,"available":96599314432,"internally_reserved":0,"allocated":1257472,"data_stored":864881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":0.82899999999999996,"15min":0.69199999999999995},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":10.284000000000001,"5min":10.284000000000001,"15min":10.284000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":1.8440000000000001,"5min":0.89800000000000002,"15min":0.752},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":9.9610000000000003,"5min":9.9610000000000003,"15min":9.9610000000000003},"last":0.51800000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":0.76600000000000001,"15min":0.69999999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":7.0279999999999996,"5min":7.0279999999999996,"15min":7.0279999999999996},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.7230000000000001,"5min":0.89700000000000002,"15min":0.76500000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":9.9250000000000007,"5min":9.9250000000000007,"15min":9.9250000000000007},"last":0.73599999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":0.872,"15min":0.72199999999999998},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":10.119999999999999,"5min":10.119999999999999,"15min":10.119999999999999},"last":0.495},{"interface":"front","average":{"1min":1.242,"5min":0.82199999999999995,"15min":0.748},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":7.0789999999999997,"5min":7.0789999999999997,"15min":7.0789999999999997},"last":0.67400000000000004}]},{"osd":3,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.625,"5min":0.92500000000000004,"15min":0.751},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":9.3539999999999992,"5min":9.3539999999999992,"15min":9.3539999999999992},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.7070000000000001,"5min":0.93400000000000005,"15min":0.81999999999999995},"min":{"1min":0.45700000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":10.34,"5min":10.34,"15min":10.34},"last":0.58899999999999997}]},{"osd":4,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":2.4079999999999999,"5min":0.96499999999999997,"15min":0.73499999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":17.021000000000001,"5min":17.021000000000001,"15min":17.021000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.734,"5min":0.92700000000000005,"15min":0.752},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":14.871,"5min":14.871,"15min":14.871},"last":0.57299999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.837,"5min":0.91000000000000003,"15min":0.77000000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":19.959,"5min":19.959,"15min":19.959},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.9610000000000001,"5min":0.94399999999999995,"15min":0.80100000000000005},"min":{"1min":0.47999999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":11.053000000000001,"5min":11.053000000000001,"15min":11.053000000000001},"last":0.623}]},{"osd":6,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":3.0070000000000001,"5min":1.2170000000000001,"15min":0.94299999999999995},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":22.478999999999999,"5min":22.478999999999999,"15min":22.478999999999999},"last":0.78600000000000003},{"interface":"front","average":{"1min":2.6269999999999998,"5min":1.1739999999999999,"15min":0.88100000000000001},"min":{"1min":0.52800000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":21.509,"5min":21.509,"15min":21.509},"last":0.77200000000000002}]}]},{"osd":6,"up_from":15,"seq":64424509496,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35988,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335852,"statfs":{"total":96636764160,"available":96599912448,"internally_reserved":0,"allocated":663552,"data_stored":274513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":0.86299999999999999,"15min":0.77800000000000002},"min":{"1min":0.45200000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":9.3339999999999996,"5min":9.3339999999999996,"15min":9.3339999999999996},"last":0.623},{"interface":"front","average":{"1min":1.4850000000000001,"5min":0.83599999999999997,"15min":0.74399999999999999},"min":{"1min":0.47099999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":6.7759999999999998,"5min":6.7759999999999998,"15min":6.7759999999999998},"last":0.79800000000000004}]},{"osd":1,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":3.8119999999999998,"5min":1.329,"15min":0.93300000000000005},"min":{"1min":0.51400000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":53.451999999999998,"5min":53.451999999999998,"15min":53.451999999999998},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.3009999999999999,"5min":0.83599999999999997,"15min":0.78400000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":5.9900000000000002,"5min":5.9900000000000002,"15min":5.9900000000000002},"last":0.68400000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":0.84499999999999997,"15min":0.85199999999999998},"min":{"1min":0.46700000000000003,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":6.0380000000000003,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.2310000000000001,"5min":0.82799999999999996,"15min":0.80700000000000005},"min":{"1min":0.57099999999999995,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":4.6059999999999999,"5min":4.6059999999999999,"15min":4.6059999999999999},"last":0.66600000000000004}]},{"osd":3,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":0.84899999999999998,"15min":0.82599999999999996},"min":{"1min":0.51900000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":3.9689999999999999,"5min":3.9689999999999999,"15min":3.9689999999999999},"last":0.755},{"interface":"front","average":{"1min":3.6709999999999998,"5min":1.355,"15min":1.0109999999999999},"min":{"1min":0.45200000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":0.88}]},{"osd":4,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.6479999999999999,"5min":0.85499999999999998,"15min":0.751},"min":{"1min":0.48099999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":8.6259999999999994,"5min":8.6259999999999994,"15min":8.6259999999999994},"last":0.45900000000000002},{"interface":"front","average":{"1min":1.2809999999999999,"5min":0.79600000000000004,"15min":0.752},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":6.9589999999999996,"5min":6.9589999999999996,"15min":6.9589999999999996},"last":0.50900000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":4.2560000000000002,"5min":1.371,"15min":0.90000000000000002},"min":{"1min":0.44500000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":53.229999999999997,"5min":53.229999999999997,"15min":53.229999999999997},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.847,"5min":0.96799999999999997,"15min":0.84599999999999997},"min":{"1min":0.53900000000000003,"5min":0.435,"15min":0.435},"max":{"1min":10.029999999999999,"5min":10.029999999999999,"15min":10.029999999999999},"last":0.54500000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":3.1640000000000001,"5min":1.2589999999999999,"15min":0.97699999999999998},"min":{"1min":0.46200000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":45.070999999999998,"5min":45.070999999999998,"15min":45.070999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":4.5990000000000002,"5min":1.5369999999999999,"15min":1.054},"min":{"1min":0.48499999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":53.338000000000001,"5min":53.338000000000001,"15min":53.338000000000001},"last":0.70399999999999996}]}]},{"osd":4,"up_from":15,"seq":64424509496,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35992,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335848,"statfs":{"total":96636764160,"available":96599908352,"internally_reserved":0,"allocated":663552,"data_stored":274513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":0.91700000000000004,"15min":0.72599999999999998},"min":{"1min":0.434,"5min":0.38,"15min":0.38},"max":{"1min":23.318000000000001,"5min":23.318000000000001,"15min":23.318000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.7649999999999999,"5min":0.84999999999999998,"15min":0.71399999999999997},"min":{"1min":0.44800000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":10.210000000000001,"5min":10.210000000000001,"15min":10.210000000000001},"last":0.57299999999999995}]},{"osd":1,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.71099999999999997,"15min":0.67200000000000004},"min":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":5.016,"5min":5.016,"15min":5.016},"last":0.85099999999999998},{"interface":"front","average":{"1min":2.0990000000000002,"5min":0.93400000000000005,"15min":0.73999999999999999},"min":{"1min":0.45100000000000001,"5min":0.377,"15min":0.377},"max":{"1min":22.408999999999999,"5min":22.408999999999999,"15min":22.408999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":0.71799999999999997,"15min":0.65900000000000003},"min":{"1min":0.46700000000000003,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":5.1159999999999997,"5min":5.1159999999999997,"15min":5.1159999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":2.1419999999999999,"5min":1.036,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":22.338999999999999,"5min":22.338999999999999,"15min":22.338999999999999},"last":0.54900000000000004}]},{"osd":3,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.8380000000000001,"5min":0.91600000000000004,"15min":0.76600000000000001},"min":{"1min":0.42699999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":10.271000000000001,"5min":10.271000000000001,"15min":10.271000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.27,"5min":0.84099999999999997,"15min":0.76500000000000001},"min":{"1min":0.59899999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":8.0719999999999992,"5min":8.0719999999999992,"15min":8.0719999999999992},"last":0.749}]},{"osd":5,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":2.7879999999999998,"5min":1.081,"15min":0.80400000000000005},"min":{"1min":0.39000000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":31.617999999999999,"5min":31.617999999999999,"15min":31.617999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":2.4369999999999998,"5min":1.0009999999999999,"15min":0.76700000000000002},"min":{"1min":0.439,"5min":0.377,"15min":0.377},"max":{"1min":17.145,"5min":17.145,"15min":17.145},"last":0.86399999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":0.83499999999999996,"15min":0.73099999999999998},"min":{"1min":0.53900000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.577,"5min":5.577,"15min":5.577},"last":0.80300000000000005},{"interface":"front","average":{"1min":4.9329999999999998,"5min":1.5720000000000001,"15min":1.014},"min":{"1min":0.56299999999999994,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":28.872,"5min":28.872,"15min":28.872},"last":0.78200000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":3.726,"5min":1.3149999999999999,"15min":0.91700000000000004},"min":{"1min":0.48199999999999998,"5min":0.436,"15min":0.436},"max":{"1min":39.256,"5min":39.256,"15min":39.256},"last":0.82599999999999996},{"interface":"front","average":{"1min":2.069,"5min":1.0389999999999999,"15min":0.86699999999999999},"min":{"1min":0.51800000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":26.573,"5min":26.573,"15min":26.573},"last":0.88700000000000001}]}]},{"osd":3,"up_from":15,"seq":64424509495,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35992,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335848,"statfs":{"total":96636764160,"available":96599908352,"internally_reserved":0,"allocated":663552,"data_stored":274513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.45900000000000002,"15min":0.433},"min":{"1min":0.28899999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.641,"5min":2.641,"15min":2.641},"last":0.434},{"interface":"front","average":{"1min":1.4099999999999999,"5min":0.68300000000000005,"15min":0.54100000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":17.756,"5min":17.756,"15min":17.756},"last":0.33000000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.54100000000000004,"15min":0.52000000000000002},"min":{"1min":0.29799999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.0620000000000003,"5min":4.0620000000000003,"15min":4.0620000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.54500000000000004,"15min":0.53900000000000003},"min":{"1min":0.371,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.38}]},{"osd":2,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":0.69199999999999995,"15min":0.60499999999999998},"min":{"1min":0.42499999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":10.849,"5min":10.849,"15min":10.849},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.626,"15min":0.58499999999999996},"min":{"1min":0.40500000000000003,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.617,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":0.5}]},{"osd":4,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":1.756,"5min":1.004,"15min":0.873},"min":{"1min":0.56499999999999995,"5min":0.5,"15min":0.5},"max":{"1min":17.853999999999999,"5min":17.853999999999999,"15min":17.853999999999999},"last":0.63},{"interface":"front","average":{"1min":1.022,"5min":0.85099999999999998,"15min":0.80500000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":4.1420000000000003,"5min":5.0369999999999999,"15min":5.0369999999999999},"last":0.71399999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":0.82699999999999996,"15min":0.76900000000000002},"min":{"1min":0.51100000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":4.7320000000000002,"5min":4.7320000000000002,"15min":4.7320000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":2.145,"5min":1.004,"15min":0.81499999999999995},"min":{"1min":0.54900000000000004,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":17.937999999999999,"5min":17.937999999999999,"15min":17.937999999999999},"last":0.57399999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":2.125,"5min":1.046,"15min":0.873},"min":{"1min":0.52300000000000002,"5min":0.499,"15min":0.499},"max":{"1min":23.100999999999999,"5min":23.100999999999999,"15min":23.100999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":1.113,"5min":0.84599999999999997,"15min":0.79000000000000004},"min":{"1min":0.61899999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":5.7030000000000003,"5min":5.7030000000000003,"15min":5.7030000000000003},"last":0.59399999999999997}]},{"osd":7,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":0.95999999999999996,"15min":0.83799999999999997},"min":{"1min":0.54900000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":17.896999999999998,"5min":17.896999999999998,"15min":17.896999999999998},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.1020000000000001,"5min":0.93999999999999995,"15min":0.86199999999999999},"min":{"1min":0.60599999999999998,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":4.1920000000000002,"5min":10.705,"15min":10.705},"last":0.68300000000000005}]}]},{"osd":2,"up_from":15,"seq":64424509495,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36008,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335832,"statfs":{"total":96636764160,"available":96599891968,"internally_reserved":0,"allocated":684032,"data_stored":281372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.55500000000000005,"15min":0.49399999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":5.1849999999999996,"5min":5.1849999999999996,"15min":5.1849999999999996},"last":0.29899999999999999},{"interface":"front","average":{"1min":1.0640000000000001,"5min":0.57799999999999996,"15min":0.504},"min":{"1min":0.307,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":7.516,"5min":7.516,"15min":7.516},"last":0.36399999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.60799999999999998,"15min":0.56599999999999995},"min":{"1min":0.35499999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.2509999999999999,"5min":4.8079999999999998,"15min":4.8079999999999998},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.61599999999999999,"15min":0.50800000000000001},"min":{"1min":0.32500000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":8.9190000000000005,"5min":8.9190000000000005,"15min":8.9190000000000005},"last":0.32500000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":0.69799999999999995,"15min":0.61299999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":12.185,"5min":12.185,"15min":12.185},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.68600000000000005,"15min":0.59599999999999997},"min":{"1min":0.42899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":7.4580000000000002,"5min":7.4580000000000002,"15min":7.4580000000000002},"last":0.435}]},{"osd":4,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":0.91400000000000003,"15min":0.80100000000000005},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":8.6609999999999996,"5min":8.6609999999999996,"15min":8.6609999999999996},"last":0.68799999999999994},{"interface":"front","average":{"1min":2.3519999999999999,"5min":1.0820000000000001,"15min":0.875},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":19.146999999999998,"5min":19.146999999999998,"15min":19.146999999999998},"last":0.79000000000000004}]},{"osd":5,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":2.4929999999999999,"5min":1.1100000000000001,"15min":0.90200000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":31.710000000000001,"5min":31.710000000000001,"15min":31.710000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":2.7970000000000002,"5min":1.2150000000000001,"15min":0.93300000000000005},"min":{"1min":0.52900000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":22.539000000000001,"5min":22.539000000000001,"15min":22.539000000000001},"last":0.747}]},{"osd":6,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":0.92000000000000004,"15min":0.82899999999999996},"min":{"1min":0.51100000000000001,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":9.4540000000000006,"5min":9.4540000000000006,"15min":9.4540000000000006},"last":0.85699999999999998},{"interface":"front","average":{"1min":2.29,"5min":1.085,"15min":0.89500000000000002},"min":{"1min":0.54400000000000004,"5min":0.503,"15min":0.503},"max":{"1min":18.614999999999998,"5min":18.614999999999998,"15min":18.614999999999998},"last":0.873}]},{"osd":7,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":0.94599999999999995,"15min":0.85199999999999998},"min":{"1min":0.51200000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":11.432,"5min":11.432,"15min":11.432},"last":0.76000000000000001},{"interface":"front","average":{"1min":2.577,"5min":1.204,"15min":0.95499999999999996},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":17.013000000000002,"5min":17.013000000000002,"15min":17.013000000000002},"last":0.94799999999999995}]}]},{"osd":1,"up_from":15,"seq":64424509497,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35992,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335848,"statfs":{"total":96636764160,"available":96599908352,"internally_reserved":0,"allocated":663552,"data_stored":274513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":0.78400000000000003,"15min":0.60199999999999998},"min":{"1min":0.27300000000000002,"5min":0.254,"15min":0.254},"max":{"1min":9.8620000000000001,"5min":19.577000000000002,"15min":19.577000000000002},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.052,"15min":0.67600000000000005},"min":{"1min":0.28799999999999998,"5min":0.251,"15min":0.251},"max":{"1min":5.6639999999999997,"5min":48.277999999999999,"15min":48.277999999999999},"last":0.57599999999999996}]},{"osd":2,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":0.997,"15min":0.69799999999999995},"min":{"1min":0.312,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":6.4569999999999999,"5min":40.698,"15min":40.698},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.97,"5min":1.351,"15min":0.94899999999999995},"min":{"1min":0.375,"5min":0.317,"15min":0.317},"max":{"1min":18.827000000000002,"5min":48.316000000000003,"15min":48.316000000000003},"last":0.39500000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.1899999999999999,"15min":0.83199999999999996},"min":{"1min":0.373,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.915,"5min":48.234999999999999,"15min":48.234999999999999},"last":0.432},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.65700000000000003,"15min":0.61699999999999999},"min":{"1min":0.34699999999999998,"5min":0.311,"15min":0.311},"max":{"1min":4.1420000000000003,"5min":6.093,"15min":6.093},"last":0.55300000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":0.86899999999999999,"15min":0.751},"min":{"1min":0.48299999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":4.1950000000000003,"5min":5.9169999999999998,"15min":5.9169999999999998},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.175,"5min":1.0129999999999999,"15min":0.85799999999999998},"min":{"1min":0.46999999999999997,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":6.3789999999999996,"5min":19.829000000000001,"15min":19.829000000000001},"last":0.86099999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":1.339,"15min":0.96099999999999997},"min":{"1min":0.58399999999999996,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":22.594999999999999,"5min":22.594999999999999,"15min":22.594999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":2.0470000000000002,"5min":1.466,"15min":1.004},"min":{"1min":0.55900000000000005,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":11.768000000000001,"5min":48.198,"15min":48.198},"last":0.91500000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":3.0390000000000001,"5min":1.2649999999999999,"15min":0.92500000000000004},"min":{"1min":0.59199999999999997,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":35.112000000000002,"5min":35.112000000000002,"15min":35.112000000000002},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.516,"5min":1.123,"15min":0.92500000000000004},"min":{"1min":0.58399999999999996,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":8.3670000000000009,"5min":19.780999999999999,"15min":19.780999999999999},"last":0.73999999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.3959999999999999,"15min":0.98799999999999999},"min":{"1min":0.56999999999999995,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":9.2870000000000008,"5min":48.148000000000003,"15min":48.148000000000003},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.325,"5min":0.95499999999999996,"15min":0.85499999999999998},"min":{"1min":0.56799999999999995,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":4.2919999999999998,"5min":6.0549999999999997,"15min":6.0549999999999997},"last":0.89900000000000002}]}]},{"osd":5,"up_from":14,"seq":60129542201,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36020,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335820,"statfs":{"total":96636764160,"available":96599879680,"internally_reserved":0,"allocated":696320,"data_stored":288296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.85999999999999999,"15min":0.77200000000000002},"min":{"1min":0.41199999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.2640000000000002,"5min":21.385000000000002,"15min":21.385000000000002},"last":0.60599999999999998},{"interface":"front","average":{"1min":1.173,"5min":0.91900000000000004,"15min":0.83699999999999997},"min":{"1min":0.51100000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":8.6590000000000007,"5min":8.6590000000000007,"15min":8.6590000000000007},"last":0.54200000000000004}]},{"osd":1,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.70799999999999996,"15min":0.68700000000000006},"min":{"1min":0.42999999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":3.1110000000000002,"5min":3.1110000000000002,"15min":3.1110000000000002},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.89700000000000002,"15min":0.755},"min":{"1min":0.47299999999999998,"5min":0.315,"15min":0.315},"max":{"1min":4.2039999999999997,"5min":21.5,"15min":21.5},"last":0.78700000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":0.99199999999999999,"15min":0.80400000000000005},"min":{"1min":0.55700000000000005,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":8.7750000000000004,"5min":8.7750000000000004,"15min":8.7750000000000004},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.86599999999999999,"15min":0.81000000000000005},"min":{"1min":0.50800000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":7.9160000000000004,"5min":7.9160000000000004,"15min":7.9160000000000004},"last":0.77500000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.0009999999999999,"15min":0.873},"min":{"1min":0.53400000000000003,"5min":0.44,"15min":0.44},"max":{"1min":5.0650000000000004,"5min":21.457000000000001,"15min":21.457000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.694,"5min":1.083,"15min":0.93000000000000005},"min":{"1min":0.58699999999999997,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":8.7300000000000004,"5min":8.7300000000000004,"15min":8.7300000000000004},"last":0.80400000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":0.88100000000000001,"15min":0.78400000000000003},"min":{"1min":0.40699999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":8.8179999999999996,"5min":9.6679999999999993,"15min":9.6679999999999993},"last":0.73599999999999999},{"interface":"front","average":{"1min":2.0089999999999999,"5min":0.94899999999999995,"15min":0.81299999999999994},"min":{"1min":0.439,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":26.146999999999998,"5min":26.146999999999998,"15min":26.146999999999998},"last":0.69399999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":0.95499999999999996,"15min":0.81000000000000005},"min":{"1min":0.46300000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":6.6699999999999999,"5min":21.535,"15min":21.535},"last":0.70599999999999996},{"interface":"front","average":{"1min":2.649,"5min":1.2250000000000001,"15min":0.94599999999999995},"min":{"1min":0.46999999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":30.488,"5min":30.488,"15min":30.488},"last":0.84699999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":0.874,"15min":0.82599999999999996},"min":{"1min":0.51600000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":2.758,"5min":9.2240000000000002,"15min":9.2240000000000002},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.96699999999999997,"15min":0.877},"min":{"1min":0.44600000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.3520000000000003,"5min":21.574000000000002,"15min":21.574000000000002},"last":0.81899999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542200,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36556,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335284,"statfs":{"total":96636764160,"available":96599330816,"internally_reserved":0,"allocated":1245184,"data_stored":858586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.52600000000000002,"15min":0.48599999999999999},"min":{"1min":0.26100000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.2409999999999997,"5min":4.2409999999999997,"15min":4.2409999999999997},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55600000000000005,"15min":0.55300000000000005},"min":{"1min":0.35499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":2.0219999999999998,"5min":2.0219999999999998,"15min":2.0219999999999998},"last":0.53800000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.57599999999999996,"15min":0.57399999999999995},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.2330000000000001,"5min":2.2330000000000001,"15min":2.2330000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.53500000000000003,"15min":0.497},"min":{"1min":0.30099999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.6789999999999998,"5min":2.6789999999999998,"15min":2.6789999999999998},"last":0.56899999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.63900000000000001,"15min":0.59699999999999998},"min":{"1min":0.38900000000000001,"5min":0.308,"15min":0.308},"max":{"1min":4.274,"5min":4.274,"15min":4.274},"last":0.503},{"interface":"front","average":{"1min":1.0309999999999999,"5min":0.68200000000000005,"15min":0.63400000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":7.2309999999999999,"5min":7.2309999999999999,"15min":7.2309999999999999},"last":0.61799999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":2.339,"5min":1.0680000000000001,"15min":0.878},"min":{"1min":0.46600000000000003,"5min":0.38,"15min":0.38},"max":{"1min":22.814,"5min":22.814,"15min":22.814},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.516,"5min":0.92700000000000005,"15min":0.86499999999999999},"min":{"1min":0.54500000000000004,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":4.4800000000000004,"5min":4.4800000000000004,"15min":4.4800000000000004},"last":0.90800000000000003}]},{"osd":5,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":1.8600000000000001,"5min":0.97499999999999998,"15min":0.85799999999999998},"min":{"1min":0.495,"5min":0.441,"15min":0.441},"max":{"1min":12.428000000000001,"5min":12.428000000000001,"15min":12.428000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.915,"5min":1.018,"15min":0.91000000000000003},"min":{"1min":0.63800000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":13.064,"5min":13.064,"15min":13.064},"last":0.82799999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":2.8029999999999999,"5min":1.1699999999999999,"15min":0.92400000000000004},"min":{"1min":0.50800000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":31.059000000000001,"5min":31.059000000000001,"15min":31.059000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":3.6059999999999999,"5min":1.357,"15min":1.022},"min":{"1min":0.52000000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":40.731000000000002,"5min":40.731000000000002,"15min":40.731000000000002},"last":0.98399999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":1.657,"5min":0.96499999999999997,"15min":0.878},"min":{"1min":0.53800000000000003,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":14.032,"5min":14.032,"15min":14.032},"last":0.93200000000000005},{"interface":"front","average":{"1min":2.9569999999999999,"5min":1.254,"15min":0.995},"min":{"1min":0.58099999999999996,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":23.524000000000001,"5min":23.524000000000001,"15min":23.524000000000001},"last":0.878}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:09:14.263 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:09:14.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.434+0000 7efec8200640 1 Processor -- start 2024-01-15T18:09:14.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.434+0000 7efec8200640 1 -- start start 2024-01-15T18:09:14.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.434+0000 7efec8200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7efec00701a0 0x7efec0070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:14.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.434+0000 7efec8200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efec00c92b0 0x7efec006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:14.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.434+0000 7efec8200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec006c4a0 con 0x7efec006e240 2024-01-15T18:09:14.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.434+0000 7efec8200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efec0070ac0 con 0x7efec00701a0 2024-01-15T18:09:14.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.434+0000 7efec8200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efec015c660 con 0x7efec00c92b0 2024-01-15T18:09:14.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.434+0000 7efec5800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7efec00701a0 0x7efec0070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:14.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.434+0000 7efec6200640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7efec006e240 0x7efec015d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:49546/0 (socket says 172.21.15.27:49546) 2024-01-15T18:09:14.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.434+0000 7efec4e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efec00c92b0 0x7efec006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:14.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.434+0000 7efec6200640 1 -- 172.21.15.27:0/339782779 learned_addr learned my addr 172.21.15.27:0/339782779 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:14.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.434+0000 7efec5800640 1 -- 172.21.15.27:0/339782779 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efec00c92b0 msgr2=0x7efec006dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:14.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.434+0000 7efec5800640 1 --2- 172.21.15.27:0/339782779 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efec00c92b0 0x7efec006dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.434+0000 7efec5800640 1 -- 172.21.15.27:0/339782779 >> v1:172.21.15.93:6789/0 conn(0x7efec006e240 legacy=0x7efec015d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:14.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.434+0000 7efec5800640 1 -- 172.21.15.27:0/339782779 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec015c940 con 0x7efec00701a0 2024-01-15T18:09:14.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.438+0000 7efec5800640 1 --2- 172.21.15.27:0/339782779 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efec00701a0 0x7efec0070580 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7efeb00097c0 tx=0x7efeb001c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fd6cddfbf719579 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:14.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.438+0000 7efebfe00640 1 -- 172.21.15.27:0/339782779 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb00ab650 con 0x7efec00701a0 2024-01-15T18:09:14.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.438+0000 7efebfe00640 1 -- 172.21.15.27:0/339782779 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb00ab7b0 con 0x7efec00701a0 2024-01-15T18:09:14.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.438+0000 7efebfe00640 1 -- 172.21.15.27:0/339782779 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb00abb40 con 0x7efec00701a0 2024-01-15T18:09:14.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.438+0000 7efec8200640 1 -- 172.21.15.27:0/339782779 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efec00701a0 msgr2=0x7efec0070580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:14.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.438+0000 7efec8200640 1 --2- 172.21.15.27:0/339782779 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efec00701a0 0x7efec0070580 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7efeb00097c0 tx=0x7efeb001c450 comp rx=0 tx=0).stop 2024-01-15T18:09:14.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.438+0000 7efec8200640 1 -- 172.21.15.27:0/339782779 shutdown_connections 2024-01-15T18:09:14.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.438+0000 7efec8200640 1 --2- 172.21.15.27:0/339782779 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efec00c92b0 0x7efec006dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.438+0000 7efec8200640 1 --2- 172.21.15.27:0/339782779 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efec00701a0 0x7efec0070580 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.438+0000 7efec8200640 1 -- 172.21.15.27:0/339782779 >> 172.21.15.27:0/339782779 conn(0x7efec00cec60 msgr2=0x7efec00cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:14.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.438+0000 7efec8200640 1 -- 172.21.15.27:0/339782779 shutdown_connections 2024-01-15T18:09:14.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.438+0000 7efec8200640 1 -- 172.21.15.27:0/339782779 wait complete. 2024-01-15T18:09:14.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.442+0000 7efec8200640 1 Processor -- start 2024-01-15T18:09:14.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.442+0000 7efec8200640 1 -- start start 2024-01-15T18:09:14.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.442+0000 7efec8200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efec006e240 0x7efec00c14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:14.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.442+0000 7efec8200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efec00701a0 0x7efec00c1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:14.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.442+0000 7efec8200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efec00c92b0 0x7efec00c7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:14.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.442+0000 7efec8200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec00c1f50 con 0x7efec00c92b0 2024-01-15T18:09:14.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.442+0000 7efec8200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec00ac890 con 0x7efec006e240 2024-01-15T18:09:14.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.442+0000 7efec8200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efec00c5dd0 con 0x7efec00701a0 2024-01-15T18:09:14.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.442+0000 7efec6200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efec00c92b0 0x7efec00c7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:14.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.442+0000 7efec6200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efec00c92b0 0x7efec00c7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34358/0 (socket says 172.21.15.27:34358) 2024-01-15T18:09:14.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.442+0000 7efec6200640 1 -- 172.21.15.27:0/3011139016 learned_addr learned my addr 172.21.15.27:0/3011139016 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:14.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.442+0000 7efec5800640 1 --2- 172.21.15.27:0/3011139016 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efec006e240 0x7efec00c14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:14.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.442+0000 7efec4e00640 1 --2- 172.21.15.27:0/3011139016 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efec00701a0 0x7efec00c1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:14.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.442+0000 7efec6200640 1 -- 172.21.15.27:0/3011139016 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efec00701a0 msgr2=0x7efec00c1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:14.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.442+0000 7efec6200640 1 --2- 172.21.15.27:0/3011139016 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efec00701a0 0x7efec00c1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.442+0000 7efec6200640 1 -- 172.21.15.27:0/3011139016 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efec006e240 msgr2=0x7efec00c14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:14.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.442+0000 7efec6200640 1 --2- 172.21.15.27:0/3011139016 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efec006e240 0x7efec00c14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.442+0000 7efec6200640 1 -- 172.21.15.27:0/3011139016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb0009420 con 0x7efec00c92b0 2024-01-15T18:09:14.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.442+0000 7efec4e00640 1 --2- 172.21.15.27:0/3011139016 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efec00701a0 0x7efec00c1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:09:14.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.442+0000 7efec6200640 1 --2- 172.21.15.27:0/3011139016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efec00c92b0 0x7efec00c7d00 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7efeb800cb20 tx=0x7efeb800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:14.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.446+0000 7efebd600640 1 -- 172.21.15.27:0/3011139016 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb800aca0 con 0x7efec00c92b0 2024-01-15T18:09:14.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.446+0000 7efebd600640 1 -- 172.21.15.27:0/3011139016 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb800ae00 con 0x7efec00c92b0 2024-01-15T18:09:14.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.446+0000 7efec8200640 1 -- 172.21.15.27:0/3011139016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec00c6040 con 0x7efec00c92b0 2024-01-15T18:09:14.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.446+0000 7efebd600640 1 -- 172.21.15.27:0/3011139016 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb8012600 con 0x7efec00c92b0 2024-01-15T18:09:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.446+0000 7efec8200640 1 -- 172.21.15.27:0/3011139016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efec00acae0 con 0x7efec00c92b0 2024-01-15T18:09:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.446+0000 7efec8200640 1 -- 172.21.15.27:0/3011139016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efec00c6560 con 0x7efec00c92b0 2024-01-15T18:09:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.446+0000 7efebd600640 1 -- 172.21.15.27:0/3011139016 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7efeb8012760 con 0x7efec00c92b0 2024-01-15T18:09:14.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.446+0000 7efebd600640 1 --2- 172.21.15.27:0/3011139016 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efe98066e90 0x7efe98069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:14.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.446+0000 7efebd600640 1 -- 172.21.15.27:0/3011139016 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7efeb8084d60 con 0x7efec00c92b0 2024-01-15T18:09:14.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.446+0000 7efec5800640 1 --2- 172.21.15.27:0/3011139016 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efe98066e90 0x7efe98069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:14.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.450+0000 7efec5800640 1 --2- 172.21.15.27:0/3011139016 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efe98066e90 0x7efe98069350 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7efeb00af000 tx=0x7efeb00a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:14.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.450+0000 7efebd600640 1 -- 172.21.15.27:0/3011139016 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7efeb8051780 con 0x7efec00c92b0 2024-01-15T18:09:14.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.606+0000 7efec8200640 1 -- 172.21.15.27:0/3011139016 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efec015d4c0 con 0x7efe98066e90 2024-01-15T18:09:14.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.618+0000 7efebd600640 1 -- 172.21.15.27:0/3011139016 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287472 (secure 0 0 0) 0x7efeb00ab920 con 0x7efe98066e90 2024-01-15T18:09:14.625 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:09:14.627 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:09:14.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.618+0000 7efec8200640 1 -- 172.21.15.27:0/3011139016 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efe98066e90 msgr2=0x7efe98069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:14.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.618+0000 7efec8200640 1 --2- 172.21.15.27:0/3011139016 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efe98066e90 0x7efe98069350 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7efeb00af000 tx=0x7efeb00a7000 comp rx=0 tx=0).stop 2024-01-15T18:09:14.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.618+0000 7efec8200640 1 -- 172.21.15.27:0/3011139016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efec00c92b0 msgr2=0x7efec00c7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:14.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.618+0000 7efec8200640 1 --2- 172.21.15.27:0/3011139016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efec00c92b0 0x7efec00c7d00 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7efeb800cb20 tx=0x7efeb800a060 comp rx=0 tx=0).stop 2024-01-15T18:09:14.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.622+0000 7efec8200640 1 -- 172.21.15.27:0/3011139016 shutdown_connections 2024-01-15T18:09:14.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.622+0000 7efec8200640 1 --2- 172.21.15.27:0/3011139016 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efe98066e90 0x7efe98069350 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.622+0000 7efec8200640 1 --2- 172.21.15.27:0/3011139016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efec00c92b0 0x7efec00c7d00 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.622+0000 7efec8200640 1 --2- 172.21.15.27:0/3011139016 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efec00701a0 0x7efec00c1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.622+0000 7efec8200640 1 --2- 172.21.15.27:0/3011139016 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efec006e240 0x7efec00c14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.622+0000 7efec8200640 1 -- 172.21.15.27:0/3011139016 >> 172.21.15.27:0/3011139016 conn(0x7efec00cec60 msgr2=0x7efec0071a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:14.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.622+0000 7efec8200640 1 -- 172.21.15.27:0/3011139016 shutdown_connections 2024-01-15T18:09:14.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.622+0000 7efec8200640 1 -- 172.21.15.27:0/3011139016 wait complete. 2024-01-15T18:09:14.647 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-01-15T18:09:14.206380+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592196,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":193,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":289120,"kb_used_data":6384,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754685600,"statfs":{"total":773094113280,"available":772798054400,"internally_reserved":0,"allocated":6537216,"data_stored":3391187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":289394224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1828,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001226"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.174696+0000","last_change":"2024-01-15T18:09:10.164426+0000","last_active":"2024-01-15T18:09:10.174696+0000","last_peered":"2024-01-15T18:09:10.174696+0000","last_clean":"2024-01-15T18:09:10.174696+0000","last_became_active":"2024-01-15T18:09:10.164120+0000","last_became_peered":"2024-01-15T18:09:10.164120+0000","last_unstale":"2024-01-15T18:09:10.174696+0000","last_undegraded":"2024-01-15T18:09:10.174696+0000","last_fullsized":"2024-01-15T18:09:10.174696+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:13:02.463974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.156544+0000","last_change":"2024-01-15T18:09:10.149693+0000","last_active":"2024-01-15T18:09:10.156544+0000","last_peered":"2024-01-15T18:09:10.156544+0000","last_clean":"2024-01-15T18:09:10.156544+0000","last_became_active":"2024-01-15T18:09:10.149380+0000","last_became_peered":"2024-01-15T18:09:10.149380+0000","last_unstale":"2024-01-15T18:09:10.156544+0000","last_undegraded":"2024-01-15T18:09:10.156544+0000","last_fullsized":"2024-01-15T18:09:10.156544+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:14:10.432447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.176732+0000","last_change":"2024-01-15T18:09:10.172281+0000","last_active":"2024-01-15T18:09:10.176732+0000","last_peered":"2024-01-15T18:09:10.176732+0000","last_clean":"2024-01-15T18:09:10.176732+0000","last_became_active":"2024-01-15T18:09:10.171520+0000","last_became_peered":"2024-01-15T18:09:10.171520+0000","last_unstale":"2024-01-15T18:09:10.176732+0000","last_undegraded":"2024-01-15T18:09:10.176732+0000","last_fullsized":"2024-01-15T18:09:10.176732+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:05.433977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155356+0000","last_change":"2024-01-15T18:09:10.154186+0000","last_active":"2024-01-15T18:09:11.155356+0000","last_peered":"2024-01-15T18:09:11.155356+0000","last_clean":"2024-01-15T18:09:11.155356+0000","last_became_active":"2024-01-15T18:09:10.153961+0000","last_became_peered":"2024-01-15T18:09:10.153961+0000","last_unstale":"2024-01-15T18:09:11.155356+0000","last_undegraded":"2024-01-15T18:09:11.155356+0000","last_fullsized":"2024-01-15T18:09:11.155356+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:44:11.424905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163531+0000","last_change":"2024-01-15T18:09:10.149056+0000","last_active":"2024-01-15T18:09:10.163531+0000","last_peered":"2024-01-15T18:09:10.163531+0000","last_clean":"2024-01-15T18:09:10.163531+0000","last_became_active":"2024-01-15T18:09:10.148756+0000","last_became_peered":"2024-01-15T18:09:10.148756+0000","last_unstale":"2024-01-15T18:09:10.163531+0000","last_undegraded":"2024-01-15T18:09:10.163531+0000","last_fullsized":"2024-01-15T18:09:10.163531+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:20.006996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.127717+0000","last_change":"2024-01-15T18:09:09.127717+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.127717+0000","last_undegraded":"2024-01-15T18:09:09.127717+0000","last_fullsized":"2024-01-15T18:09:09.127717+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.149739+0000","last_change":"2024-01-15T18:09:10.147574+0000","last_active":"2024-01-15T18:09:10.149739+0000","last_peered":"2024-01-15T18:09:10.149739+0000","last_clean":"2024-01-15T18:09:10.149739+0000","last_became_active":"2024-01-15T18:09:10.147288+0000","last_became_peered":"2024-01-15T18:09:10.147288+0000","last_unstale":"2024-01-15T18:09:10.149739+0000","last_undegraded":"2024-01-15T18:09:10.149739+0000","last_fullsized":"2024-01-15T18:09:10.149739+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:29:52.274001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.160773+0000","last_change":"2024-01-15T18:09:10.149706+0000","last_active":"2024-01-15T18:09:10.160773+0000","last_peered":"2024-01-15T18:09:10.160773+0000","last_clean":"2024-01-15T18:09:10.160773+0000","last_became_active":"2024-01-15T18:09:10.149353+0000","last_became_peered":"2024-01-15T18:09:10.149353+0000","last_unstale":"2024-01-15T18:09:10.160773+0000","last_undegraded":"2024-01-15T18:09:10.160773+0000","last_fullsized":"2024-01-15T18:09:10.160773+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:22:35.521714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.166664+0000","last_change":"2024-01-15T18:09:10.162835+0000","last_active":"2024-01-15T18:09:10.166664+0000","last_peered":"2024-01-15T18:09:10.166664+0000","last_clean":"2024-01-15T18:09:10.166664+0000","last_became_active":"2024-01-15T18:09:10.162459+0000","last_became_peered":"2024-01-15T18:09:10.162459+0000","last_unstale":"2024-01-15T18:09:10.166664+0000","last_undegraded":"2024-01-15T18:09:10.166664+0000","last_fullsized":"2024-01-15T18:09:10.166664+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:26.052280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.164826+0000","last_change":"2024-01-15T18:09:10.151168+0000","last_active":"2024-01-15T18:09:10.164826+0000","last_peered":"2024-01-15T18:09:10.164826+0000","last_clean":"2024-01-15T18:09:10.164826+0000","last_became_active":"2024-01-15T18:09:10.150812+0000","last_became_peered":"2024-01-15T18:09:10.150812+0000","last_unstale":"2024-01-15T18:09:10.164826+0000","last_undegraded":"2024-01-15T18:09:10.164826+0000","last_fullsized":"2024-01-15T18:09:10.164826+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:09:59.563201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.157867+0000","last_change":"2024-01-15T18:09:10.148209+0000","last_active":"2024-01-15T18:09:10.157867+0000","last_peered":"2024-01-15T18:09:10.157867+0000","last_clean":"2024-01-15T18:09:10.157867+0000","last_became_active":"2024-01-15T18:09:10.147897+0000","last_became_peered":"2024-01-15T18:09:10.147897+0000","last_unstale":"2024-01-15T18:09:10.157867+0000","last_undegraded":"2024-01-15T18:09:10.157867+0000","last_fullsized":"2024-01-15T18:09:10.157867+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:53.437883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155548+0000","last_change":"2024-01-15T18:09:10.153400+0000","last_active":"2024-01-15T18:09:11.155548+0000","last_peered":"2024-01-15T18:09:11.155548+0000","last_clean":"2024-01-15T18:09:11.155548+0000","last_became_active":"2024-01-15T18:09:10.152222+0000","last_became_peered":"2024-01-15T18:09:10.152222+0000","last_unstale":"2024-01-15T18:09:11.155548+0000","last_undegraded":"2024-01-15T18:09:11.155548+0000","last_fullsized":"2024-01-15T18:09:11.155548+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:03.126844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.165807+0000","last_change":"2024-01-15T18:09:10.160925+0000","last_active":"2024-01-15T18:09:10.165807+0000","last_peered":"2024-01-15T18:09:10.165807+0000","last_clean":"2024-01-15T18:09:10.165807+0000","last_became_active":"2024-01-15T18:09:10.160620+0000","last_became_peered":"2024-01-15T18:09:10.160620+0000","last_unstale":"2024-01-15T18:09:10.165807+0000","last_undegraded":"2024-01-15T18:09:10.165807+0000","last_fullsized":"2024-01-15T18:09:10.165807+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:40:15.227589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.166438+0000","last_change":"2024-01-15T18:09:10.154922+0000","last_active":"2024-01-15T18:09:10.166438+0000","last_peered":"2024-01-15T18:09:10.166438+0000","last_clean":"2024-01-15T18:09:10.166438+0000","last_became_active":"2024-01-15T18:09:10.154594+0000","last_became_peered":"2024-01-15T18:09:10.154594+0000","last_unstale":"2024-01-15T18:09:10.166438+0000","last_undegraded":"2024-01-15T18:09:10.166438+0000","last_fullsized":"2024-01-15T18:09:10.166438+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:55:44.009610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.148569+0000","last_change":"2024-01-15T18:09:10.143163+0000","last_active":"2024-01-15T18:09:10.148569+0000","last_peered":"2024-01-15T18:09:10.148569+0000","last_clean":"2024-01-15T18:09:10.148569+0000","last_became_active":"2024-01-15T18:09:10.142874+0000","last_became_peered":"2024-01-15T18:09:10.142874+0000","last_unstale":"2024-01-15T18:09:10.148569+0000","last_undegraded":"2024-01-15T18:09:10.148569+0000","last_fullsized":"2024-01-15T18:09:10.148569+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:02:56.938951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.127271+0000","last_change":"2024-01-15T18:09:09.127271+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.127271+0000","last_undegraded":"2024-01-15T18:09:09.127271+0000","last_fullsized":"2024-01-15T18:09:09.127271+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163654+0000","last_change":"2024-01-15T18:09:10.156152+0000","last_active":"2024-01-15T18:09:10.163654+0000","last_peered":"2024-01-15T18:09:10.163654+0000","last_clean":"2024-01-15T18:09:10.163654+0000","last_became_active":"2024-01-15T18:09:10.155849+0000","last_became_peered":"2024-01-15T18:09:10.155849+0000","last_unstale":"2024-01-15T18:09:10.163654+0000","last_undegraded":"2024-01-15T18:09:10.163654+0000","last_fullsized":"2024-01-15T18:09:10.163654+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:53.373020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.177069+0000","last_change":"2024-01-15T18:09:10.168066+0000","last_active":"2024-01-15T18:09:10.177069+0000","last_peered":"2024-01-15T18:09:10.177069+0000","last_clean":"2024-01-15T18:09:10.177069+0000","last_became_active":"2024-01-15T18:09:10.167691+0000","last_became_peered":"2024-01-15T18:09:10.167691+0000","last_unstale":"2024-01-15T18:09:10.177069+0000","last_undegraded":"2024-01-15T18:09:10.177069+0000","last_fullsized":"2024-01-15T18:09:10.177069+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:58:26.855862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.174836+0000","last_change":"2024-01-15T18:09:10.163790+0000","last_active":"2024-01-15T18:09:10.174836+0000","last_peered":"2024-01-15T18:09:10.174836+0000","last_clean":"2024-01-15T18:09:10.174836+0000","last_became_active":"2024-01-15T18:09:10.163509+0000","last_became_peered":"2024-01-15T18:09:10.163509+0000","last_unstale":"2024-01-15T18:09:10.174836+0000","last_undegraded":"2024-01-15T18:09:10.174836+0000","last_fullsized":"2024-01-15T18:09:10.174836+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:22:24.247456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173893+0000","last_change":"2024-01-15T18:09:10.161978+0000","last_active":"2024-01-15T18:09:10.173893+0000","last_peered":"2024-01-15T18:09:10.173893+0000","last_clean":"2024-01-15T18:09:10.173893+0000","last_became_active":"2024-01-15T18:09:10.161717+0000","last_became_peered":"2024-01-15T18:09:10.161717+0000","last_unstale":"2024-01-15T18:09:10.173893+0000","last_undegraded":"2024-01-15T18:09:10.173893+0000","last_fullsized":"2024-01-15T18:09:10.173893+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:05.433977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163131+0000","last_change":"2024-01-15T18:09:10.159447+0000","last_active":"2024-01-15T18:09:10.163131+0000","last_peered":"2024-01-15T18:09:10.163131+0000","last_clean":"2024-01-15T18:09:10.163131+0000","last_became_active":"2024-01-15T18:09:10.159055+0000","last_became_peered":"2024-01-15T18:09:10.159055+0000","last_unstale":"2024-01-15T18:09:10.163131+0000","last_undegraded":"2024-01-15T18:09:10.163131+0000","last_fullsized":"2024-01-15T18:09:10.163131+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:54:09.462802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173806+0000","last_change":"2024-01-15T18:09:10.157352+0000","last_active":"2024-01-15T18:09:10.173806+0000","last_peered":"2024-01-15T18:09:10.173806+0000","last_clean":"2024-01-15T18:09:10.173806+0000","last_became_active":"2024-01-15T18:09:10.155991+0000","last_became_peered":"2024-01-15T18:09:10.155991+0000","last_unstale":"2024-01-15T18:09:10.173806+0000","last_undegraded":"2024-01-15T18:09:10.173806+0000","last_fullsized":"2024-01-15T18:09:10.173806+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:25:19.060578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163476+0000","last_change":"2024-01-15T18:09:10.160338+0000","last_active":"2024-01-15T18:09:10.163476+0000","last_peered":"2024-01-15T18:09:10.163476+0000","last_clean":"2024-01-15T18:09:10.163476+0000","last_became_active":"2024-01-15T18:09:10.159325+0000","last_became_peered":"2024-01-15T18:09:10.159325+0000","last_unstale":"2024-01-15T18:09:10.163476+0000","last_undegraded":"2024-01-15T18:09:10.163476+0000","last_fullsized":"2024-01-15T18:09:10.163476+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:46:36.631105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.168684+0000","last_change":"2024-01-15T18:09:10.167327+0000","last_active":"2024-01-15T18:09:10.168684+0000","last_peered":"2024-01-15T18:09:10.168684+0000","last_clean":"2024-01-15T18:09:10.168684+0000","last_became_active":"2024-01-15T18:09:10.167008+0000","last_became_peered":"2024-01-15T18:09:10.167008+0000","last_unstale":"2024-01-15T18:09:10.168684+0000","last_undegraded":"2024-01-15T18:09:10.168684+0000","last_fullsized":"2024-01-15T18:09:10.168684+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:18:13.449428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.174731+0000","last_change":"2024-01-15T18:09:10.165962+0000","last_active":"2024-01-15T18:09:10.174731+0000","last_peered":"2024-01-15T18:09:10.174731+0000","last_clean":"2024-01-15T18:09:10.174731+0000","last_became_active":"2024-01-15T18:09:10.165689+0000","last_became_peered":"2024-01-15T18:09:10.165689+0000","last_unstale":"2024-01-15T18:09:10.174731+0000","last_undegraded":"2024-01-15T18:09:10.174731+0000","last_fullsized":"2024-01-15T18:09:10.174731+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:48:02.569878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155565+0000","last_change":"2024-01-15T18:09:10.147816+0000","last_active":"2024-01-15T18:09:11.155565+0000","last_peered":"2024-01-15T18:09:11.155565+0000","last_clean":"2024-01-15T18:09:11.155565+0000","last_became_active":"2024-01-15T18:09:10.147487+0000","last_became_peered":"2024-01-15T18:09:10.147487+0000","last_unstale":"2024-01-15T18:09:11.155565+0000","last_undegraded":"2024-01-15T18:09:11.155565+0000","last_fullsized":"2024-01-15T18:09:11.155565+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:40:51.237238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.133285+0000","last_change":"2024-01-15T18:09:09.133285+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.133285+0000","last_undegraded":"2024-01-15T18:09:09.133285+0000","last_fullsized":"2024-01-15T18:09:09.133285+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.156169+0000","last_change":"2024-01-15T18:09:10.149120+0000","last_active":"2024-01-15T18:09:10.156169+0000","last_peered":"2024-01-15T18:09:10.156169+0000","last_clean":"2024-01-15T18:09:10.156169+0000","last_became_active":"2024-01-15T18:09:10.148675+0000","last_became_peered":"2024-01-15T18:09:10.148675+0000","last_unstale":"2024-01-15T18:09:10.156169+0000","last_undegraded":"2024-01-15T18:09:10.156169+0000","last_fullsized":"2024-01-15T18:09:10.156169+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:52:23.460592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173871+0000","last_change":"2024-01-15T18:09:10.161188+0000","last_active":"2024-01-15T18:09:10.173871+0000","last_peered":"2024-01-15T18:09:10.173871+0000","last_clean":"2024-01-15T18:09:10.173871+0000","last_became_active":"2024-01-15T18:09:10.151645+0000","last_became_peered":"2024-01-15T18:09:10.151645+0000","last_unstale":"2024-01-15T18:09:10.173871+0000","last_undegraded":"2024-01-15T18:09:10.173871+0000","last_fullsized":"2024-01-15T18:09:10.173871+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:07:18.177097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.176602+0000","last_change":"2024-01-15T18:09:10.167895+0000","last_active":"2024-01-15T18:09:10.176602+0000","last_peered":"2024-01-15T18:09:10.176602+0000","last_clean":"2024-01-15T18:09:10.176602+0000","last_became_active":"2024-01-15T18:09:10.167639+0000","last_became_peered":"2024-01-15T18:09:10.167639+0000","last_unstale":"2024-01-15T18:09:10.176602+0000","last_undegraded":"2024-01-15T18:09:10.176602+0000","last_fullsized":"2024-01-15T18:09:10.176602+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:18:53.201764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.161004+0000","last_change":"2024-01-15T18:09:10.148253+0000","last_active":"2024-01-15T18:09:10.161004+0000","last_peered":"2024-01-15T18:09:10.161004+0000","last_clean":"2024-01-15T18:09:10.161004+0000","last_became_active":"2024-01-15T18:09:10.147941+0000","last_became_peered":"2024-01-15T18:09:10.147941+0000","last_unstale":"2024-01-15T18:09:10.161004+0000","last_undegraded":"2024-01-15T18:09:10.161004+0000","last_fullsized":"2024-01-15T18:09:10.161004+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:04:16.316753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.126719+0000","last_change":"2024-01-15T18:09:09.126719+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.126719+0000","last_undegraded":"2024-01-15T18:09:09.126719+0000","last_fullsized":"2024-01-15T18:09:09.126719+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.138025+0000","last_change":"2024-01-15T18:09:09.138025+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.138025+0000","last_undegraded":"2024-01-15T18:09:09.138025+0000","last_fullsized":"2024-01-15T18:09:09.138025+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.155581+0000","last_change":"2024-01-15T18:09:09.152802+0000","last_active":"2024-01-15T18:09:09.155581+0000","last_peered":"2024-01-15T18:09:09.155581+0000","last_clean":"2024-01-15T18:09:09.155581+0000","last_became_active":"2024-01-15T18:09:09.152520+0000","last_became_peered":"2024-01-15T18:09:09.152520+0000","last_unstale":"2024-01-15T18:09:09.155581+0000","last_undegraded":"2024-01-15T18:09:09.155581+0000","last_fullsized":"2024-01-15T18:09:09.155581+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:04.586121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.170556+0000","last_change":"2024-01-15T18:09:10.161734+0000","last_active":"2024-01-15T18:09:10.170556+0000","last_peered":"2024-01-15T18:09:10.170556+0000","last_clean":"2024-01-15T18:09:10.170556+0000","last_became_active":"2024-01-15T18:09:10.161466+0000","last_became_peered":"2024-01-15T18:09:10.161466+0000","last_unstale":"2024-01-15T18:09:10.170556+0000","last_undegraded":"2024-01-15T18:09:10.170556+0000","last_fullsized":"2024-01-15T18:09:10.170556+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:58:24.569904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"57'1","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.177728+0000","last_change":"2024-01-15T18:09:09.125625+0000","last_active":"2024-01-15T18:09:10.177728+0000","last_peered":"2024-01-15T18:09:10.177728+0000","last_clean":"2024-01-15T18:09:10.177728+0000","last_became_active":"2024-01-15T18:09:09.125373+0000","last_became_peered":"2024-01-15T18:09:09.125373+0000","last_unstale":"2024-01-15T18:09:10.177728+0000","last_undegraded":"2024-01-15T18:09:10.177728+0000","last_fullsized":"2024-01-15T18:09:10.177728+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:14:59.883458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.130696+0000","last_change":"2024-01-15T18:09:09.130696+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.130696+0000","last_undegraded":"2024-01-15T18:09:09.130696+0000","last_fullsized":"2024-01-15T18:09:09.130696+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.155587+0000","last_change":"2024-01-15T18:09:09.153007+0000","last_active":"2024-01-15T18:09:09.155587+0000","last_peered":"2024-01-15T18:09:09.155587+0000","last_clean":"2024-01-15T18:09:09.155587+0000","last_became_active":"2024-01-15T18:09:09.152405+0000","last_became_peered":"2024-01-15T18:09:09.152405+0000","last_unstale":"2024-01-15T18:09:09.155587+0000","last_undegraded":"2024-01-15T18:09:09.155587+0000","last_fullsized":"2024-01-15T18:09:09.155587+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:53:06.843401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.132775+0000","last_change":"2024-01-15T18:09:09.132775+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.132775+0000","last_undegraded":"2024-01-15T18:09:09.132775+0000","last_fullsized":"2024-01-15T18:09:09.132775+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.176876+0000","last_change":"2024-01-15T18:09:09.138209+0000","last_active":"2024-01-15T18:09:10.176876+0000","last_peered":"2024-01-15T18:09:10.176876+0000","last_clean":"2024-01-15T18:09:10.176876+0000","last_became_active":"2024-01-15T18:09:09.137921+0000","last_became_peered":"2024-01-15T18:09:09.137921+0000","last_unstale":"2024-01-15T18:09:10.176876+0000","last_undegraded":"2024-01-15T18:09:10.176876+0000","last_fullsized":"2024-01-15T18:09:10.176876+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:54:56.016726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163688+0000","last_change":"2024-01-15T18:09:10.156211+0000","last_active":"2024-01-15T18:09:10.163688+0000","last_peered":"2024-01-15T18:09:10.163688+0000","last_clean":"2024-01-15T18:09:10.163688+0000","last_became_active":"2024-01-15T18:09:10.155914+0000","last_became_peered":"2024-01-15T18:09:10.155914+0000","last_unstale":"2024-01-15T18:09:10.163688+0000","last_undegraded":"2024-01-15T18:09:10.163688+0000","last_fullsized":"2024-01-15T18:09:10.163688+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:15:38.206084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"57'1","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.172500+0000","last_change":"2024-01-15T18:09:09.139963+0000","last_active":"2024-01-15T18:09:10.172500+0000","last_peered":"2024-01-15T18:09:10.172500+0000","last_clean":"2024-01-15T18:09:10.172500+0000","last_became_active":"2024-01-15T18:09:09.139603+0000","last_became_peered":"2024-01-15T18:09:09.139603+0000","last_unstale":"2024-01-15T18:09:10.172500+0000","last_undegraded":"2024-01-15T18:09:10.172500+0000","last_fullsized":"2024-01-15T18:09:10.172500+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:14:01.036199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.174128+0000","last_change":"2024-01-15T18:09:10.165154+0000","last_active":"2024-01-15T18:09:10.174128+0000","last_peered":"2024-01-15T18:09:10.174128+0000","last_clean":"2024-01-15T18:09:10.174128+0000","last_became_active":"2024-01-15T18:09:10.164900+0000","last_became_peered":"2024-01-15T18:09:10.164900+0000","last_unstale":"2024-01-15T18:09:10.174128+0000","last_undegraded":"2024-01-15T18:09:10.174128+0000","last_fullsized":"2024-01-15T18:09:10.174128+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:57:26.029056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.153437+0000","last_change":"2024-01-15T18:09:09.147874+0000","last_active":"2024-01-15T18:09:09.153437+0000","last_peered":"2024-01-15T18:09:09.153437+0000","last_clean":"2024-01-15T18:09:09.153437+0000","last_became_active":"2024-01-15T18:09:09.147666+0000","last_became_peered":"2024-01-15T18:09:09.147666+0000","last_unstale":"2024-01-15T18:09:09.153437+0000","last_undegraded":"2024-01-15T18:09:09.153437+0000","last_fullsized":"2024-01-15T18:09:09.153437+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:44:37.388258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163308+0000","last_change":"2024-01-15T18:09:10.154886+0000","last_active":"2024-01-15T18:09:10.163308+0000","last_peered":"2024-01-15T18:09:10.163308+0000","last_clean":"2024-01-15T18:09:10.163308+0000","last_became_active":"2024-01-15T18:09:10.154594+0000","last_became_peered":"2024-01-15T18:09:10.154594+0000","last_unstale":"2024-01-15T18:09:10.163308+0000","last_undegraded":"2024-01-15T18:09:10.163308+0000","last_fullsized":"2024-01-15T18:09:10.163308+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:34:09.142491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.137679+0000","last_change":"2024-01-15T18:09:09.140948+0000","last_active":"2024-01-15T18:09:10.137679+0000","last_peered":"2024-01-15T18:09:10.137679+0000","last_clean":"2024-01-15T18:09:10.137679+0000","last_became_active":"2024-01-15T18:09:09.140010+0000","last_became_peered":"2024-01-15T18:09:09.140010+0000","last_unstale":"2024-01-15T18:09:10.137679+0000","last_undegraded":"2024-01-15T18:09:10.137679+0000","last_fullsized":"2024-01-15T18:09:10.137679+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:39:25.682654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173959+0000","last_change":"2024-01-15T18:09:10.160588+0000","last_active":"2024-01-15T18:09:10.173959+0000","last_peered":"2024-01-15T18:09:10.173959+0000","last_clean":"2024-01-15T18:09:10.173959+0000","last_became_active":"2024-01-15T18:09:10.157197+0000","last_became_peered":"2024-01-15T18:09:10.157197+0000","last_unstale":"2024-01-15T18:09:10.173959+0000","last_undegraded":"2024-01-15T18:09:10.173959+0000","last_fullsized":"2024-01-15T18:09:10.173959+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:27:59.969234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.138903+0000","last_change":"2024-01-15T18:09:09.131170+0000","last_active":"2024-01-15T18:09:10.138903+0000","last_peered":"2024-01-15T18:09:10.138903+0000","last_clean":"2024-01-15T18:09:10.138903+0000","last_became_active":"2024-01-15T18:09:09.130789+0000","last_became_peered":"2024-01-15T18:09:09.130789+0000","last_unstale":"2024-01-15T18:09:10.138903+0000","last_undegraded":"2024-01-15T18:09:10.138903+0000","last_fullsized":"2024-01-15T18:09:10.138903+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:13:26.979622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.166623+0000","last_change":"2024-01-15T18:09:10.154127+0000","last_active":"2024-01-15T18:09:10.166623+0000","last_peered":"2024-01-15T18:09:10.166623+0000","last_clean":"2024-01-15T18:09:10.166623+0000","last_became_active":"2024-01-15T18:09:10.153699+0000","last_became_peered":"2024-01-15T18:09:10.153699+0000","last_unstale":"2024-01-15T18:09:10.166623+0000","last_undegraded":"2024-01-15T18:09:10.166623+0000","last_fullsized":"2024-01-15T18:09:10.166623+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:37:29.585671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.151155+0000","last_change":"2024-01-15T18:09:09.145070+0000","last_active":"2024-01-15T18:09:10.151155+0000","last_peered":"2024-01-15T18:09:10.151155+0000","last_clean":"2024-01-15T18:09:10.151155+0000","last_became_active":"2024-01-15T18:09:09.144808+0000","last_became_peered":"2024-01-15T18:09:09.144808+0000","last_unstale":"2024-01-15T18:09:10.151155+0000","last_undegraded":"2024-01-15T18:09:10.151155+0000","last_fullsized":"2024-01-15T18:09:10.151155+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:07:17.174892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.178747+0000","last_change":"2024-01-15T18:09:10.173359+0000","last_active":"2024-01-15T18:09:10.178747+0000","last_peered":"2024-01-15T18:09:10.178747+0000","last_clean":"2024-01-15T18:09:10.178747+0000","last_became_active":"2024-01-15T18:09:10.173020+0000","last_became_peered":"2024-01-15T18:09:10.173020+0000","last_unstale":"2024-01-15T18:09:10.178747+0000","last_undegraded":"2024-01-15T18:09:10.178747+0000","last_fullsized":"2024-01-15T18:09:10.178747+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:22:35.521714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.140713+0000","last_change":"2024-01-15T18:09:09.142076+0000","last_active":"2024-01-15T18:09:10.140713+0000","last_peered":"2024-01-15T18:09:10.140713+0000","last_clean":"2024-01-15T18:09:10.140713+0000","last_became_active":"2024-01-15T18:09:09.141747+0000","last_became_peered":"2024-01-15T18:09:09.141747+0000","last_unstale":"2024-01-15T18:09:10.140713+0000","last_undegraded":"2024-01-15T18:09:10.140713+0000","last_fullsized":"2024-01-15T18:09:10.140713+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:02.592720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.165189+0000","last_change":"2024-01-15T18:09:10.150403+0000","last_active":"2024-01-15T18:09:10.165189+0000","last_peered":"2024-01-15T18:09:10.165189+0000","last_clean":"2024-01-15T18:09:10.165189+0000","last_became_active":"2024-01-15T18:09:10.150113+0000","last_became_peered":"2024-01-15T18:09:10.150113+0000","last_unstale":"2024-01-15T18:09:10.165189+0000","last_undegraded":"2024-01-15T18:09:10.165189+0000","last_fullsized":"2024-01-15T18:09:10.165189+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:45:59.978009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"57'3","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.180373+0000","last_change":"2024-01-15T18:09:09.142251+0000","last_active":"2024-01-15T18:09:10.180373+0000","last_peered":"2024-01-15T18:09:10.180373+0000","last_clean":"2024-01-15T18:09:10.180373+0000","last_became_active":"2024-01-15T18:09:09.141961+0000","last_became_peered":"2024-01-15T18:09:09.141961+0000","last_unstale":"2024-01-15T18:09:10.180373+0000","last_undegraded":"2024-01-15T18:09:10.180373+0000","last_fullsized":"2024-01-15T18:09:10.180373+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:56:33.327612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.176749+0000","last_change":"2024-01-15T18:09:10.172402+0000","last_active":"2024-01-15T18:09:10.176749+0000","last_peered":"2024-01-15T18:09:10.176749+0000","last_clean":"2024-01-15T18:09:10.176749+0000","last_became_active":"2024-01-15T18:09:10.171813+0000","last_became_peered":"2024-01-15T18:09:10.171813+0000","last_unstale":"2024-01-15T18:09:10.176749+0000","last_undegraded":"2024-01-15T18:09:10.176749+0000","last_fullsized":"2024-01-15T18:09:10.176749+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:15:38.206084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"57'6","reported_seq":17,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.180712+0000","last_change":"2024-01-15T18:09:09.147769+0000","last_active":"2024-01-15T18:09:10.180712+0000","last_peered":"2024-01-15T18:09:10.180712+0000","last_clean":"2024-01-15T18:09:10.180712+0000","last_became_active":"2024-01-15T18:09:09.147356+0000","last_became_peered":"2024-01-15T18:09:09.147356+0000","last_unstale":"2024-01-15T18:09:10.180712+0000","last_undegraded":"2024-01-15T18:09:10.180712+0000","last_fullsized":"2024-01-15T18:09:10.180712+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:18:52.199559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155288+0000","last_change":"2024-01-15T18:09:10.154188+0000","last_active":"2024-01-15T18:09:11.155288+0000","last_peered":"2024-01-15T18:09:11.155288+0000","last_clean":"2024-01-15T18:09:11.155288+0000","last_became_active":"2024-01-15T18:09:10.153881+0000","last_became_peered":"2024-01-15T18:09:10.153881+0000","last_unstale":"2024-01-15T18:09:11.155288+0000","last_undegraded":"2024-01-15T18:09:11.155288+0000","last_fullsized":"2024-01-15T18:09:11.155288+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:57:04.284081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"57'2","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.180721+0000","last_change":"2024-01-15T18:09:09.141986+0000","last_active":"2024-01-15T18:09:10.180721+0000","last_peered":"2024-01-15T18:09:10.180721+0000","last_clean":"2024-01-15T18:09:10.180721+0000","last_became_active":"2024-01-15T18:09:09.141626+0000","last_became_peered":"2024-01-15T18:09:09.141626+0000","last_unstale":"2024-01-15T18:09:10.180721+0000","last_undegraded":"2024-01-15T18:09:10.180721+0000","last_fullsized":"2024-01-15T18:09:10.180721+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:44:08.042896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.176175+0000","last_change":"2024-01-15T18:09:10.163997+0000","last_active":"2024-01-15T18:09:10.176175+0000","last_peered":"2024-01-15T18:09:10.176175+0000","last_clean":"2024-01-15T18:09:10.176175+0000","last_became_active":"2024-01-15T18:09:10.163758+0000","last_became_peered":"2024-01-15T18:09:10.163758+0000","last_unstale":"2024-01-15T18:09:10.176175+0000","last_undegraded":"2024-01-15T18:09:10.176175+0000","last_fullsized":"2024-01-15T18:09:10.176175+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:02:59.936727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.144084+0000","last_change":"2024-01-15T18:09:09.142040+0000","last_active":"2024-01-15T18:09:10.144084+0000","last_peered":"2024-01-15T18:09:10.144084+0000","last_clean":"2024-01-15T18:09:10.144084+0000","last_became_active":"2024-01-15T18:09:09.141713+0000","last_became_peered":"2024-01-15T18:09:09.141713+0000","last_unstale":"2024-01-15T18:09:10.144084+0000","last_undegraded":"2024-01-15T18:09:10.144084+0000","last_fullsized":"2024-01-15T18:09:10.144084+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:17:15.118577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.166494+0000","last_change":"2024-01-15T18:09:10.156574+0000","last_active":"2024-01-15T18:09:10.166494+0000","last_peered":"2024-01-15T18:09:10.166494+0000","last_clean":"2024-01-15T18:09:10.166494+0000","last_became_active":"2024-01-15T18:09:10.156225+0000","last_became_peered":"2024-01-15T18:09:10.156225+0000","last_unstale":"2024-01-15T18:09:10.166494+0000","last_undegraded":"2024-01-15T18:09:10.166494+0000","last_fullsized":"2024-01-15T18:09:10.166494+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:15:39.864433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.137769+0000","last_change":"2024-01-15T18:09:09.135271+0000","last_active":"2024-01-15T18:09:10.137769+0000","last_peered":"2024-01-15T18:09:10.137769+0000","last_clean":"2024-01-15T18:09:10.137769+0000","last_became_active":"2024-01-15T18:09:09.135011+0000","last_became_peered":"2024-01-15T18:09:09.135011+0000","last_unstale":"2024-01-15T18:09:10.137769+0000","last_undegraded":"2024-01-15T18:09:10.137769+0000","last_fullsized":"2024-01-15T18:09:10.137769+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:29:02.803967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163455+0000","last_change":"2024-01-15T18:09:10.160727+0000","last_active":"2024-01-15T18:09:10.163455+0000","last_peered":"2024-01-15T18:09:10.163455+0000","last_clean":"2024-01-15T18:09:10.163455+0000","last_became_active":"2024-01-15T18:09:10.160132+0000","last_became_peered":"2024-01-15T18:09:10.160132+0000","last_unstale":"2024-01-15T18:09:10.163455+0000","last_undegraded":"2024-01-15T18:09:10.163455+0000","last_fullsized":"2024-01-15T18:09:10.163455+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:27:59.969234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"57'8","reported_seq":19,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.177242+0000","last_change":"2024-01-15T18:09:09.137932+0000","last_active":"2024-01-15T18:09:10.177242+0000","last_peered":"2024-01-15T18:09:10.177242+0000","last_clean":"2024-01-15T18:09:10.177242+0000","last_became_active":"2024-01-15T18:09:09.137626+0000","last_became_peered":"2024-01-15T18:09:09.137626+0000","last_unstale":"2024-01-15T18:09:10.177242+0000","last_undegraded":"2024-01-15T18:09:10.177242+0000","last_fullsized":"2024-01-15T18:09:10.177242+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:14:01.036199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.177049+0000","last_change":"2024-01-15T18:09:10.165126+0000","last_active":"2024-01-15T18:09:10.177049+0000","last_peered":"2024-01-15T18:09:10.177049+0000","last_clean":"2024-01-15T18:09:10.177049+0000","last_became_active":"2024-01-15T18:09:10.164811+0000","last_became_peered":"2024-01-15T18:09:10.164811+0000","last_unstale":"2024-01-15T18:09:10.177049+0000","last_undegraded":"2024-01-15T18:09:10.177049+0000","last_fullsized":"2024-01-15T18:09:10.177049+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:53:41.639339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"57'1","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.180173+0000","last_change":"2024-01-15T18:09:09.137481+0000","last_active":"2024-01-15T18:09:10.180173+0000","last_peered":"2024-01-15T18:09:10.180173+0000","last_clean":"2024-01-15T18:09:10.180173+0000","last_became_active":"2024-01-15T18:09:09.137205+0000","last_became_peered":"2024-01-15T18:09:09.137205+0000","last_unstale":"2024-01-15T18:09:10.180173+0000","last_undegraded":"2024-01-15T18:09:10.180173+0000","last_fullsized":"2024-01-15T18:09:10.180173+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:16:45.705931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155989+0000","last_change":"2024-01-15T18:09:10.166022+0000","last_active":"2024-01-15T18:09:11.155989+0000","last_peered":"2024-01-15T18:09:11.155989+0000","last_clean":"2024-01-15T18:09:11.155989+0000","last_became_active":"2024-01-15T18:09:10.165752+0000","last_became_peered":"2024-01-15T18:09:10.165752+0000","last_unstale":"2024-01-15T18:09:11.155989+0000","last_undegraded":"2024-01-15T18:09:11.155989+0000","last_fullsized":"2024-01-15T18:09:11.155989+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:45:32.407323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.155562+0000","last_change":"2024-01-15T18:09:09.152740+0000","last_active":"2024-01-15T18:09:09.155562+0000","last_peered":"2024-01-15T18:09:09.155562+0000","last_clean":"2024-01-15T18:09:09.155562+0000","last_became_active":"2024-01-15T18:09:09.152418+0000","last_became_peered":"2024-01-15T18:09:09.152418+0000","last_unstale":"2024-01-15T18:09:09.155562+0000","last_undegraded":"2024-01-15T18:09:09.155562+0000","last_fullsized":"2024-01-15T18:09:09.155562+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:56:49.381443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.166880+0000","last_change":"2024-01-15T18:09:10.155205+0000","last_active":"2024-01-15T18:09:10.166880+0000","last_peered":"2024-01-15T18:09:10.166880+0000","last_clean":"2024-01-15T18:09:10.166880+0000","last_became_active":"2024-01-15T18:09:10.154910+0000","last_became_peered":"2024-01-15T18:09:10.154910+0000","last_unstale":"2024-01-15T18:09:10.166880+0000","last_undegraded":"2024-01-15T18:09:10.166880+0000","last_fullsized":"2024-01-15T18:09:10.166880+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:56:34.329818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.177022+0000","last_change":"2024-01-15T18:09:09.141114+0000","last_active":"2024-01-15T18:09:10.177022+0000","last_peered":"2024-01-15T18:09:10.177022+0000","last_clean":"2024-01-15T18:09:10.177022+0000","last_became_active":"2024-01-15T18:09:09.140786+0000","last_became_peered":"2024-01-15T18:09:09.140786+0000","last_unstale":"2024-01-15T18:09:10.177022+0000","last_undegraded":"2024-01-15T18:09:10.177022+0000","last_fullsized":"2024-01-15T18:09:10.177022+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:37:27.921887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.176767+0000","last_change":"2024-01-15T18:09:10.172606+0000","last_active":"2024-01-15T18:09:10.176767+0000","last_peered":"2024-01-15T18:09:10.176767+0000","last_clean":"2024-01-15T18:09:10.176767+0000","last_became_active":"2024-01-15T18:09:10.172090+0000","last_became_peered":"2024-01-15T18:09:10.172090+0000","last_unstale":"2024-01-15T18:09:10.176767+0000","last_undegraded":"2024-01-15T18:09:10.176767+0000","last_fullsized":"2024-01-15T18:09:10.176767+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:25:19.060578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.153526+0000","last_change":"2024-01-15T18:09:09.148967+0000","last_active":"2024-01-15T18:09:09.153526+0000","last_peered":"2024-01-15T18:09:09.153526+0000","last_clean":"2024-01-15T18:09:09.153526+0000","last_became_active":"2024-01-15T18:09:09.148670+0000","last_became_peered":"2024-01-15T18:09:09.148670+0000","last_unstale":"2024-01-15T18:09:09.153526+0000","last_undegraded":"2024-01-15T18:09:09.153526+0000","last_fullsized":"2024-01-15T18:09:09.153526+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:04:47.742716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155444+0000","last_change":"2024-01-15T18:09:10.151767+0000","last_active":"2024-01-15T18:09:11.155444+0000","last_peered":"2024-01-15T18:09:11.155444+0000","last_clean":"2024-01-15T18:09:11.155444+0000","last_became_active":"2024-01-15T18:09:10.151462+0000","last_became_peered":"2024-01-15T18:09:10.151462+0000","last_unstale":"2024-01-15T18:09:11.155444+0000","last_undegraded":"2024-01-15T18:09:11.155444+0000","last_fullsized":"2024-01-15T18:09:11.155444+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:07:18.547932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"57'2","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.174259+0000","last_change":"2024-01-15T18:09:09.141213+0000","last_active":"2024-01-15T18:09:10.174259+0000","last_peered":"2024-01-15T18:09:10.174259+0000","last_clean":"2024-01-15T18:09:10.174259+0000","last_became_active":"2024-01-15T18:09:09.139957+0000","last_became_peered":"2024-01-15T18:09:09.139957+0000","last_unstale":"2024-01-15T18:09:10.174259+0000","last_undegraded":"2024-01-15T18:09:10.174259+0000","last_fullsized":"2024-01-15T18:09:10.174259+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:42:47.132801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.127537+0000","last_change":"2024-01-15T18:09:09.127537+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.127537+0000","last_undegraded":"2024-01-15T18:09:09.127537+0000","last_fullsized":"2024-01-15T18:09:09.127537+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.133917+0000","last_change":"2024-01-15T18:09:09.136084+0000","last_active":"2024-01-15T18:09:10.133917+0000","last_peered":"2024-01-15T18:09:10.133917+0000","last_clean":"2024-01-15T18:09:10.133917+0000","last_became_active":"2024-01-15T18:09:09.135760+0000","last_became_peered":"2024-01-15T18:09:09.135760+0000","last_unstale":"2024-01-15T18:09:10.133917+0000","last_undegraded":"2024-01-15T18:09:10.133917+0000","last_fullsized":"2024-01-15T18:09:10.133917+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:37.003640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.128633+0000","last_change":"2024-01-15T18:09:09.128633+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.128633+0000","last_undegraded":"2024-01-15T18:09:09.128633+0000","last_fullsized":"2024-01-15T18:09:09.128633+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.137490+0000","last_change":"2024-01-15T18:09:09.136145+0000","last_active":"2024-01-15T18:09:10.137490+0000","last_peered":"2024-01-15T18:09:10.137490+0000","last_clean":"2024-01-15T18:09:10.137490+0000","last_became_active":"2024-01-15T18:09:09.135807+0000","last_became_peered":"2024-01-15T18:09:09.135807+0000","last_unstale":"2024-01-15T18:09:10.137490+0000","last_undegraded":"2024-01-15T18:09:10.137490+0000","last_fullsized":"2024-01-15T18:09:10.137490+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:35:26.455284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155791+0000","last_change":"2024-01-15T18:09:10.157661+0000","last_active":"2024-01-15T18:09:11.155791+0000","last_peered":"2024-01-15T18:09:11.155791+0000","last_clean":"2024-01-15T18:09:11.155791+0000","last_became_active":"2024-01-15T18:09:10.157399+0000","last_became_peered":"2024-01-15T18:09:10.157399+0000","last_unstale":"2024-01-15T18:09:11.155791+0000","last_undegraded":"2024-01-15T18:09:11.155791+0000","last_fullsized":"2024-01-15T18:09:11.155791+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:41:56.291009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.177996+0000","last_change":"2024-01-15T18:09:09.145612+0000","last_active":"2024-01-15T18:09:10.177996+0000","last_peered":"2024-01-15T18:09:10.177996+0000","last_clean":"2024-01-15T18:09:10.177996+0000","last_became_active":"2024-01-15T18:09:09.145279+0000","last_became_peered":"2024-01-15T18:09:09.145279+0000","last_unstale":"2024-01-15T18:09:10.177996+0000","last_undegraded":"2024-01-15T18:09:10.177996+0000","last_fullsized":"2024-01-15T18:09:10.177996+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:53.930232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.156102+0000","last_change":"2024-01-15T18:09:10.153156+0000","last_active":"2024-01-15T18:09:11.156102+0000","last_peered":"2024-01-15T18:09:11.156102+0000","last_clean":"2024-01-15T18:09:11.156102+0000","last_became_active":"2024-01-15T18:09:10.152783+0000","last_became_peered":"2024-01-15T18:09:10.152783+0000","last_unstale":"2024-01-15T18:09:11.156102+0000","last_undegraded":"2024-01-15T18:09:11.156102+0000","last_fullsized":"2024-01-15T18:09:11.156102+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:36:30.872154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.155546+0000","last_change":"2024-01-15T18:09:09.152741+0000","last_active":"2024-01-15T18:09:09.155546+0000","last_peered":"2024-01-15T18:09:09.155546+0000","last_clean":"2024-01-15T18:09:09.155546+0000","last_became_active":"2024-01-15T18:09:09.152470+0000","last_became_peered":"2024-01-15T18:09:09.152470+0000","last_unstale":"2024-01-15T18:09:09.155546+0000","last_undegraded":"2024-01-15T18:09:09.155546+0000","last_fullsized":"2024-01-15T18:09:09.155546+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:14:31.566501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.153880+0000","last_change":"2024-01-15T18:09:10.143039+0000","last_active":"2024-01-15T18:09:10.153880+0000","last_peered":"2024-01-15T18:09:10.153880+0000","last_clean":"2024-01-15T18:09:10.153880+0000","last_became_active":"2024-01-15T18:09:10.142814+0000","last_became_peered":"2024-01-15T18:09:10.142814+0000","last_unstale":"2024-01-15T18:09:10.153880+0000","last_undegraded":"2024-01-15T18:09:10.153880+0000","last_fullsized":"2024-01-15T18:09:10.153880+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:54:31.215934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.143973+0000","last_change":"2024-01-15T18:09:09.141870+0000","last_active":"2024-01-15T18:09:10.143973+0000","last_peered":"2024-01-15T18:09:10.143973+0000","last_clean":"2024-01-15T18:09:10.143973+0000","last_became_active":"2024-01-15T18:09:09.141523+0000","last_became_peered":"2024-01-15T18:09:09.141523+0000","last_unstale":"2024-01-15T18:09:10.143973+0000","last_undegraded":"2024-01-15T18:09:10.143973+0000","last_fullsized":"2024-01-15T18:09:10.143973+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:15:13.572542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.132306+0000","last_change":"2024-01-15T18:09:09.132306+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.132306+0000","last_undegraded":"2024-01-15T18:09:09.132306+0000","last_fullsized":"2024-01-15T18:09:09.132306+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.137795+0000","last_change":"2024-01-15T18:09:09.136593+0000","last_active":"2024-01-15T18:09:10.137795+0000","last_peered":"2024-01-15T18:09:10.137795+0000","last_clean":"2024-01-15T18:09:10.137795+0000","last_became_active":"2024-01-15T18:09:09.136246+0000","last_became_peered":"2024-01-15T18:09:09.136246+0000","last_unstale":"2024-01-15T18:09:10.137795+0000","last_undegraded":"2024-01-15T18:09:10.137795+0000","last_fullsized":"2024-01-15T18:09:10.137795+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:10:09.158484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.155341+0000","last_change":"2024-01-15T18:09:10.145733+0000","last_active":"2024-01-15T18:09:10.155341+0000","last_peered":"2024-01-15T18:09:10.155341+0000","last_clean":"2024-01-15T18:09:10.155341+0000","last_became_active":"2024-01-15T18:09:10.145479+0000","last_became_peered":"2024-01-15T18:09:10.145479+0000","last_unstale":"2024-01-15T18:09:10.155341+0000","last_undegraded":"2024-01-15T18:09:10.155341+0000","last_fullsized":"2024-01-15T18:09:10.155341+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:53:41.639339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.133917+0000","last_change":"2024-01-15T18:09:09.142681+0000","last_active":"2024-01-15T18:09:10.133917+0000","last_peered":"2024-01-15T18:09:10.133917+0000","last_clean":"2024-01-15T18:09:10.133917+0000","last_became_active":"2024-01-15T18:09:09.142357+0000","last_became_peered":"2024-01-15T18:09:09.142357+0000","last_unstale":"2024-01-15T18:09:10.133917+0000","last_undegraded":"2024-01-15T18:09:10.133917+0000","last_fullsized":"2024-01-15T18:09:10.133917+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:41:15.076278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173119+0000","last_change":"2024-01-15T18:09:10.162125+0000","last_active":"2024-01-15T18:09:10.173119+0000","last_peered":"2024-01-15T18:09:10.173119+0000","last_clean":"2024-01-15T18:09:10.173119+0000","last_became_active":"2024-01-15T18:09:10.161778+0000","last_became_peered":"2024-01-15T18:09:10.161778+0000","last_unstale":"2024-01-15T18:09:10.173119+0000","last_undegraded":"2024-01-15T18:09:10.173119+0000","last_fullsized":"2024-01-15T18:09:10.173119+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:07:31.195306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.139632+0000","last_change":"2024-01-15T18:09:09.135449+0000","last_active":"2024-01-15T18:09:10.139632+0000","last_peered":"2024-01-15T18:09:10.139632+0000","last_clean":"2024-01-15T18:09:10.139632+0000","last_became_active":"2024-01-15T18:09:09.135010+0000","last_became_peered":"2024-01-15T18:09:09.135010+0000","last_unstale":"2024-01-15T18:09:10.139632+0000","last_undegraded":"2024-01-15T18:09:10.139632+0000","last_fullsized":"2024-01-15T18:09:10.139632+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:54:08.460596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.170500+0000","last_change":"2024-01-15T18:09:10.158452+0000","last_active":"2024-01-15T18:09:10.170500+0000","last_peered":"2024-01-15T18:09:10.170500+0000","last_clean":"2024-01-15T18:09:10.170500+0000","last_became_active":"2024-01-15T18:09:10.158267+0000","last_became_peered":"2024-01-15T18:09:10.158267+0000","last_unstale":"2024-01-15T18:09:10.170500+0000","last_undegraded":"2024-01-15T18:09:10.170500+0000","last_fullsized":"2024-01-15T18:09:10.170500+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:26:45.212614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.133918+0000","last_change":"2024-01-15T18:09:09.146561+0000","last_active":"2024-01-15T18:09:10.133918+0000","last_peered":"2024-01-15T18:09:10.133918+0000","last_clean":"2024-01-15T18:09:10.133918+0000","last_became_active":"2024-01-15T18:09:09.146263+0000","last_became_peered":"2024-01-15T18:09:09.146263+0000","last_unstale":"2024-01-15T18:09:10.133918+0000","last_undegraded":"2024-01-15T18:09:10.133918+0000","last_fullsized":"2024-01-15T18:09:10.133918+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:00:48.078792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173947+0000","last_change":"2024-01-15T18:09:10.157427+0000","last_active":"2024-01-15T18:09:10.173947+0000","last_peered":"2024-01-15T18:09:10.173947+0000","last_clean":"2024-01-15T18:09:10.173947+0000","last_became_active":"2024-01-15T18:09:10.157146+0000","last_became_peered":"2024-01-15T18:09:10.157146+0000","last_unstale":"2024-01-15T18:09:10.173947+0000","last_undegraded":"2024-01-15T18:09:10.173947+0000","last_fullsized":"2024-01-15T18:09:10.173947+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:07:31.195306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.142185+0000","last_change":"2024-01-15T18:09:09.136141+0000","last_active":"2024-01-15T18:09:10.142185+0000","last_peered":"2024-01-15T18:09:10.142185+0000","last_clean":"2024-01-15T18:09:10.142185+0000","last_became_active":"2024-01-15T18:09:09.135853+0000","last_became_peered":"2024-01-15T18:09:09.135853+0000","last_unstale":"2024-01-15T18:09:10.142185+0000","last_undegraded":"2024-01-15T18:09:10.142185+0000","last_fullsized":"2024-01-15T18:09:10.142185+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:25:50.421687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173953+0000","last_change":"2024-01-15T18:09:10.161307+0000","last_active":"2024-01-15T18:09:10.173953+0000","last_peered":"2024-01-15T18:09:10.173953+0000","last_clean":"2024-01-15T18:09:10.173953+0000","last_became_active":"2024-01-15T18:09:10.160945+0000","last_became_peered":"2024-01-15T18:09:10.160945+0000","last_unstale":"2024-01-15T18:09:10.173953+0000","last_undegraded":"2024-01-15T18:09:10.173953+0000","last_fullsized":"2024-01-15T18:09:10.173953+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:26:05.237285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.144458+0000","last_change":"2024-01-15T18:09:09.142696+0000","last_active":"2024-01-15T18:09:10.144458+0000","last_peered":"2024-01-15T18:09:10.144458+0000","last_clean":"2024-01-15T18:09:10.144458+0000","last_became_active":"2024-01-15T18:09:09.142448+0000","last_became_peered":"2024-01-15T18:09:09.142448+0000","last_unstale":"2024-01-15T18:09:10.144458+0000","last_undegraded":"2024-01-15T18:09:10.144458+0000","last_fullsized":"2024-01-15T18:09:10.144458+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:40:45.529038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":194,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.122277+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:09:09.122277+0000","last_peered":"2024-01-15T18:09:09.122277+0000","last_clean":"2024-01-15T18:09:09.122277+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:09:09.122277+0000","last_undegraded":"2024-01-15T18:09:09.122277+0000","last_fullsized":"2024-01-15T18:09:09.122277+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":1828,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":2626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509496,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36572,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335268,"statfs":{"total":96636764160,"available":96599314432,"internally_reserved":0,"allocated":1257472,"data_stored":864881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":0.82899999999999996,"15min":0.69199999999999995},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":10.284000000000001,"5min":10.284000000000001,"15min":10.284000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":1.8440000000000001,"5min":0.89800000000000002,"15min":0.752},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":9.9610000000000003,"5min":9.9610000000000003,"15min":9.9610000000000003},"last":0.51800000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":0.76600000000000001,"15min":0.69999999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":7.0279999999999996,"5min":7.0279999999999996,"15min":7.0279999999999996},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.7230000000000001,"5min":0.89700000000000002,"15min":0.76500000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":9.9250000000000007,"5min":9.9250000000000007,"15min":9.9250000000000007},"last":0.73599999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":0.872,"15min":0.72199999999999998},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":10.119999999999999,"5min":10.119999999999999,"15min":10.119999999999999},"last":0.495},{"interface":"front","average":{"1min":1.242,"5min":0.82199999999999995,"15min":0.748},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":7.0789999999999997,"5min":7.0789999999999997,"15min":7.0789999999999997},"last":0.67400000000000004}]},{"osd":3,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.625,"5min":0.92500000000000004,"15min":0.751},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":9.3539999999999992,"5min":9.3539999999999992,"15min":9.3539999999999992},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.7070000000000001,"5min":0.93400000000000005,"15min":0.81999999999999995},"min":{"1min":0.45700000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":10.34,"5min":10.34,"15min":10.34},"last":0.58899999999999997}]},{"osd":4,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":2.4079999999999999,"5min":0.96499999999999997,"15min":0.73499999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":17.021000000000001,"5min":17.021000000000001,"15min":17.021000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.734,"5min":0.92700000000000005,"15min":0.752},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":14.871,"5min":14.871,"15min":14.871},"last":0.57299999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.837,"5min":0.91000000000000003,"15min":0.77000000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":19.959,"5min":19.959,"15min":19.959},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.9610000000000001,"5min":0.94399999999999995,"15min":0.80100000000000005},"min":{"1min":0.47999999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":11.053000000000001,"5min":11.053000000000001,"15min":11.053000000000001},"last":0.623}]},{"osd":6,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":3.0070000000000001,"5min":1.2170000000000001,"15min":0.94299999999999995},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":22.478999999999999,"5min":22.478999999999999,"15min":22.478999999999999},"last":0.78600000000000003},{"interface":"front","average":{"1min":2.6269999999999998,"5min":1.1739999999999999,"15min":0.88100000000000001},"min":{"1min":0.52800000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":21.509,"5min":21.509,"15min":21.509},"last":0.77200000000000002}]}]},{"osd":6,"up_from":15,"seq":64424509496,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35988,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335852,"statfs":{"total":96636764160,"available":96599912448,"internally_reserved":0,"allocated":663552,"data_stored":274513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":0.86299999999999999,"15min":0.77800000000000002},"min":{"1min":0.45200000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":9.3339999999999996,"5min":9.3339999999999996,"15min":9.3339999999999996},"last":0.623},{"interface":"front","average":{"1min":1.4850000000000001,"5min":0.83599999999999997,"15min":0.74399999999999999},"min":{"1min":0.47099999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":6.7759999999999998,"5min":6.7759999999999998,"15min":6.7759999999999998},"last":0.79800000000000004}]},{"osd":1,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":3.8119999999999998,"5min":1.329,"15min":0.93300000000000005},"min":{"1min":0.51400000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":53.451999999999998,"5min":53.451999999999998,"15min":53.451999999999998},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.3009999999999999,"5min":0.83599999999999997,"15min":0.78400000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":5.9900000000000002,"5min":5.9900000000000002,"15min":5.9900000000000002},"last":0.68400000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":0.84499999999999997,"15min":0.85199999999999998},"min":{"1min":0.46700000000000003,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":6.0380000000000003,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.2310000000000001,"5min":0.82799999999999996,"15min":0.80700000000000005},"min":{"1min":0.57099999999999995,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":4.6059999999999999,"5min":4.6059999999999999,"15min":4.6059999999999999},"last":0.66600000000000004}]},{"osd":3,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":0.84899999999999998,"15min":0.82599999999999996},"min":{"1min":0.51900000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":3.9689999999999999,"5min":3.9689999999999999,"15min":3.9689999999999999},"last":0.755},{"interface":"front","average":{"1min":3.6709999999999998,"5min":1.355,"15min":1.0109999999999999},"min":{"1min":0.45200000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":0.88}]},{"osd":4,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.6479999999999999,"5min":0.85499999999999998,"15min":0.751},"min":{"1min":0.48099999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":8.6259999999999994,"5min":8.6259999999999994,"15min":8.6259999999999994},"last":0.45900000000000002},{"interface":"front","average":{"1min":1.2809999999999999,"5min":0.79600000000000004,"15min":0.752},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":6.9589999999999996,"5min":6.9589999999999996,"15min":6.9589999999999996},"last":0.50900000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":4.2560000000000002,"5min":1.371,"15min":0.90000000000000002},"min":{"1min":0.44500000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":53.229999999999997,"5min":53.229999999999997,"15min":53.229999999999997},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.847,"5min":0.96799999999999997,"15min":0.84599999999999997},"min":{"1min":0.53900000000000003,"5min":0.435,"15min":0.435},"max":{"1min":10.029999999999999,"5min":10.029999999999999,"15min":10.029999999999999},"last":0.54500000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":3.1640000000000001,"5min":1.2589999999999999,"15min":0.97699999999999998},"min":{"1min":0.46200000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":45.070999999999998,"5min":45.070999999999998,"15min":45.070999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":4.5990000000000002,"5min":1.5369999999999999,"15min":1.054},"min":{"1min":0.48499999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":53.338000000000001,"5min":53.338000000000001,"15min":53.338000000000001},"last":0.70399999999999996}]}]},{"osd":4,"up_from":15,"seq":64424509496,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35992,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335848,"statfs":{"total":96636764160,"available":96599908352,"internally_reserved":0,"allocated":663552,"data_stored":274513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":0.91700000000000004,"15min":0.72599999999999998},"min":{"1min":0.434,"5min":0.38,"15min":0.38},"max":{"1min":23.318000000000001,"5min":23.318000000000001,"15min":23.318000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.7649999999999999,"5min":0.84999999999999998,"15min":0.71399999999999997},"min":{"1min":0.44800000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":10.210000000000001,"5min":10.210000000000001,"15min":10.210000000000001},"last":0.57299999999999995}]},{"osd":1,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.71099999999999997,"15min":0.67200000000000004},"min":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":5.016,"5min":5.016,"15min":5.016},"last":0.85099999999999998},{"interface":"front","average":{"1min":2.0990000000000002,"5min":0.93400000000000005,"15min":0.73999999999999999},"min":{"1min":0.45100000000000001,"5min":0.377,"15min":0.377},"max":{"1min":22.408999999999999,"5min":22.408999999999999,"15min":22.408999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":0.71799999999999997,"15min":0.65900000000000003},"min":{"1min":0.46700000000000003,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":5.1159999999999997,"5min":5.1159999999999997,"15min":5.1159999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":2.1419999999999999,"5min":1.036,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":22.338999999999999,"5min":22.338999999999999,"15min":22.338999999999999},"last":0.54900000000000004}]},{"osd":3,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.8380000000000001,"5min":0.91600000000000004,"15min":0.76600000000000001},"min":{"1min":0.42699999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":10.271000000000001,"5min":10.271000000000001,"15min":10.271000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.27,"5min":0.84099999999999997,"15min":0.76500000000000001},"min":{"1min":0.59899999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":8.0719999999999992,"5min":8.0719999999999992,"15min":8.0719999999999992},"last":0.749}]},{"osd":5,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":2.7879999999999998,"5min":1.081,"15min":0.80400000000000005},"min":{"1min":0.39000000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":31.617999999999999,"5min":31.617999999999999,"15min":31.617999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":2.4369999999999998,"5min":1.0009999999999999,"15min":0.76700000000000002},"min":{"1min":0.439,"5min":0.377,"15min":0.377},"max":{"1min":17.145,"5min":17.145,"15min":17.145},"last":0.86399999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":0.83499999999999996,"15min":0.73099999999999998},"min":{"1min":0.53900000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.577,"5min":5.577,"15min":5.577},"last":0.80300000000000005},{"interface":"front","average":{"1min":4.9329999999999998,"5min":1.5720000000000001,"15min":1.014},"min":{"1min":0.56299999999999994,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":28.872,"5min":28.872,"15min":28.872},"last":0.78200000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":3.726,"5min":1.3149999999999999,"15min":0.91700000000000004},"min":{"1min":0.48199999999999998,"5min":0.436,"15min":0.436},"max":{"1min":39.256,"5min":39.256,"15min":39.256},"last":0.82599999999999996},{"interface":"front","average":{"1min":2.069,"5min":1.0389999999999999,"15min":0.86699999999999999},"min":{"1min":0.51800000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":26.573,"5min":26.573,"15min":26.573},"last":0.88700000000000001}]}]},{"osd":3,"up_from":15,"seq":64424509495,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35992,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335848,"statfs":{"total":96636764160,"available":96599908352,"internally_reserved":0,"allocated":663552,"data_stored":274513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.45900000000000002,"15min":0.433},"min":{"1min":0.28899999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.641,"5min":2.641,"15min":2.641},"last":0.434},{"interface":"front","average":{"1min":1.4099999999999999,"5min":0.68300000000000005,"15min":0.54100000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":17.756,"5min":17.756,"15min":17.756},"last":0.33000000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.54100000000000004,"15min":0.52000000000000002},"min":{"1min":0.29799999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.0620000000000003,"5min":4.0620000000000003,"15min":4.0620000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.54500000000000004,"15min":0.53900000000000003},"min":{"1min":0.371,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.38}]},{"osd":2,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":0.69199999999999995,"15min":0.60499999999999998},"min":{"1min":0.42499999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":10.849,"5min":10.849,"15min":10.849},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.626,"15min":0.58499999999999996},"min":{"1min":0.40500000000000003,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.617,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":0.5}]},{"osd":4,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":1.756,"5min":1.004,"15min":0.873},"min":{"1min":0.56499999999999995,"5min":0.5,"15min":0.5},"max":{"1min":17.853999999999999,"5min":17.853999999999999,"15min":17.853999999999999},"last":0.63},{"interface":"front","average":{"1min":1.022,"5min":0.85099999999999998,"15min":0.80500000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":4.1420000000000003,"5min":5.0369999999999999,"15min":5.0369999999999999},"last":0.71399999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":0.82699999999999996,"15min":0.76900000000000002},"min":{"1min":0.51100000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":4.7320000000000002,"5min":4.7320000000000002,"15min":4.7320000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":2.145,"5min":1.004,"15min":0.81499999999999995},"min":{"1min":0.54900000000000004,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":17.937999999999999,"5min":17.937999999999999,"15min":17.937999999999999},"last":0.57399999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":2.125,"5min":1.046,"15min":0.873},"min":{"1min":0.52300000000000002,"5min":0.499,"15min":0.499},"max":{"1min":23.100999999999999,"5min":23.100999999999999,"15min":23.100999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":1.113,"5min":0.84599999999999997,"15min":0.79000000000000004},"min":{"1min":0.61899999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":5.7030000000000003,"5min":5.7030000000000003,"15min":5.7030000000000003},"last":0.59399999999999997}]},{"osd":7,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":0.95999999999999996,"15min":0.83799999999999997},"min":{"1min":0.54900000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":17.896999999999998,"5min":17.896999999999998,"15min":17.896999999999998},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.1020000000000001,"5min":0.93999999999999995,"15min":0.86199999999999999},"min":{"1min":0.60599999999999998,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":4.1920000000000002,"5min":10.705,"15min":10.705},"last":0.68300000000000005}]}]},{"osd":2,"up_from":15,"seq":64424509495,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36008,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335832,"statfs":{"total":96636764160,"available":96599891968,"internally_reserved":0,"allocated":684032,"data_stored":281372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.55500000000000005,"15min":0.49399999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":5.1849999999999996,"5min":5.1849999999999996,"15min":5.1849999999999996},"last":0.29899999999999999},{"interface":"front","average":{"1min":1.0640000000000001,"5min":0.57799999999999996,"15min":0.504},"min":{"1min":0.307,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":7.516,"5min":7.516,"15min":7.516},"last":0.36399999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.60799999999999998,"15min":0.56599999999999995},"min":{"1min":0.35499999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.2509999999999999,"5min":4.8079999999999998,"15min":4.8079999999999998},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.61599999999999999,"15min":0.50800000000000001},"min":{"1min":0.32500000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":8.9190000000000005,"5min":8.9190000000000005,"15min":8.9190000000000005},"last":0.32500000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":0.69799999999999995,"15min":0.61299999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":12.185,"5min":12.185,"15min":12.185},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.68600000000000005,"15min":0.59599999999999997},"min":{"1min":0.42899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":7.4580000000000002,"5min":7.4580000000000002,"15min":7.4580000000000002},"last":0.435}]},{"osd":4,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":0.91400000000000003,"15min":0.80100000000000005},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":8.6609999999999996,"5min":8.6609999999999996,"15min":8.6609999999999996},"last":0.68799999999999994},{"interface":"front","average":{"1min":2.3519999999999999,"5min":1.0820000000000001,"15min":0.875},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":19.146999999999998,"5min":19.146999999999998,"15min":19.146999999999998},"last":0.79000000000000004}]},{"osd":5,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":2.4929999999999999,"5min":1.1100000000000001,"15min":0.90200000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":31.710000000000001,"5min":31.710000000000001,"15min":31.710000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":2.7970000000000002,"5min":1.2150000000000001,"15min":0.93300000000000005},"min":{"1min":0.52900000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":22.539000000000001,"5min":22.539000000000001,"15min":22.539000000000001},"last":0.747}]},{"osd":6,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":0.92000000000000004,"15min":0.82899999999999996},"min":{"1min":0.51100000000000001,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":9.4540000000000006,"5min":9.4540000000000006,"15min":9.4540000000000006},"last":0.85699999999999998},{"interface":"front","average":{"1min":2.29,"5min":1.085,"15min":0.89500000000000002},"min":{"1min":0.54400000000000004,"5min":0.503,"15min":0.503},"max":{"1min":18.614999999999998,"5min":18.614999999999998,"15min":18.614999999999998},"last":0.873}]},{"osd":7,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":0.94599999999999995,"15min":0.85199999999999998},"min":{"1min":0.51200000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":11.432,"5min":11.432,"15min":11.432},"last":0.76000000000000001},{"interface":"front","average":{"1min":2.577,"5min":1.204,"15min":0.95499999999999996},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":17.013000000000002,"5min":17.013000000000002,"15min":17.013000000000002},"last":0.94799999999999995}]}]},{"osd":1,"up_from":15,"seq":64424509497,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35992,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335848,"statfs":{"total":96636764160,"available":96599908352,"internally_reserved":0,"allocated":663552,"data_stored":274513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":0.78400000000000003,"15min":0.60199999999999998},"min":{"1min":0.27300000000000002,"5min":0.254,"15min":0.254},"max":{"1min":9.8620000000000001,"5min":19.577000000000002,"15min":19.577000000000002},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.052,"15min":0.67600000000000005},"min":{"1min":0.28799999999999998,"5min":0.251,"15min":0.251},"max":{"1min":5.6639999999999997,"5min":48.277999999999999,"15min":48.277999999999999},"last":0.57599999999999996}]},{"osd":2,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":0.997,"15min":0.69799999999999995},"min":{"1min":0.312,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":6.4569999999999999,"5min":40.698,"15min":40.698},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.97,"5min":1.351,"15min":0.94899999999999995},"min":{"1min":0.375,"5min":0.317,"15min":0.317},"max":{"1min":18.827000000000002,"5min":48.316000000000003,"15min":48.316000000000003},"last":0.39500000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.1899999999999999,"15min":0.83199999999999996},"min":{"1min":0.373,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.915,"5min":48.234999999999999,"15min":48.234999999999999},"last":0.432},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.65700000000000003,"15min":0.61699999999999999},"min":{"1min":0.34699999999999998,"5min":0.311,"15min":0.311},"max":{"1min":4.1420000000000003,"5min":6.093,"15min":6.093},"last":0.55300000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":0.86899999999999999,"15min":0.751},"min":{"1min":0.48299999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":4.1950000000000003,"5min":5.9169999999999998,"15min":5.9169999999999998},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.175,"5min":1.0129999999999999,"15min":0.85799999999999998},"min":{"1min":0.46999999999999997,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":6.3789999999999996,"5min":19.829000000000001,"15min":19.829000000000001},"last":0.86099999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":1.339,"15min":0.96099999999999997},"min":{"1min":0.58399999999999996,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":22.594999999999999,"5min":22.594999999999999,"15min":22.594999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":2.0470000000000002,"5min":1.466,"15min":1.004},"min":{"1min":0.55900000000000005,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":11.768000000000001,"5min":48.198,"15min":48.198},"last":0.91500000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":3.0390000000000001,"5min":1.2649999999999999,"15min":0.92500000000000004},"min":{"1min":0.59199999999999997,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":35.112000000000002,"5min":35.112000000000002,"15min":35.112000000000002},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.516,"5min":1.123,"15min":0.92500000000000004},"min":{"1min":0.58399999999999996,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":8.3670000000000009,"5min":19.780999999999999,"15min":19.780999999999999},"last":0.73999999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.3959999999999999,"15min":0.98799999999999999},"min":{"1min":0.56999999999999995,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":9.2870000000000008,"5min":48.148000000000003,"15min":48.148000000000003},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.325,"5min":0.95499999999999996,"15min":0.85499999999999998},"min":{"1min":0.56799999999999995,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":4.2919999999999998,"5min":6.0549999999999997,"15min":6.0549999999999997},"last":0.89900000000000002}]}]},{"osd":5,"up_from":14,"seq":60129542201,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36020,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335820,"statfs":{"total":96636764160,"available":96599879680,"internally_reserved":0,"allocated":696320,"data_stored":288296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.85999999999999999,"15min":0.77200000000000002},"min":{"1min":0.41199999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.2640000000000002,"5min":21.385000000000002,"15min":21.385000000000002},"last":0.60599999999999998},{"interface":"front","average":{"1min":1.173,"5min":0.91900000000000004,"15min":0.83699999999999997},"min":{"1min":0.51100000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":8.6590000000000007,"5min":8.6590000000000007,"15min":8.6590000000000007},"last":0.54200000000000004}]},{"osd":1,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.70799999999999996,"15min":0.68700000000000006},"min":{"1min":0.42999999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":3.1110000000000002,"5min":3.1110000000000002,"15min":3.1110000000000002},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.89700000000000002,"15min":0.755},"min":{"1min":0.47299999999999998,"5min":0.315,"15min":0.315},"max":{"1min":4.2039999999999997,"5min":21.5,"15min":21.5},"last":0.78700000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":0.99199999999999999,"15min":0.80400000000000005},"min":{"1min":0.55700000000000005,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":8.7750000000000004,"5min":8.7750000000000004,"15min":8.7750000000000004},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.86599999999999999,"15min":0.81000000000000005},"min":{"1min":0.50800000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":7.9160000000000004,"5min":7.9160000000000004,"15min":7.9160000000000004},"last":0.77500000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.0009999999999999,"15min":0.873},"min":{"1min":0.53400000000000003,"5min":0.44,"15min":0.44},"max":{"1min":5.0650000000000004,"5min":21.457000000000001,"15min":21.457000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.694,"5min":1.083,"15min":0.93000000000000005},"min":{"1min":0.58699999999999997,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":8.7300000000000004,"5min":8.7300000000000004,"15min":8.7300000000000004},"last":0.80400000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":0.88100000000000001,"15min":0.78400000000000003},"min":{"1min":0.40699999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":8.8179999999999996,"5min":9.6679999999999993,"15min":9.6679999999999993},"last":0.73599999999999999},{"interface":"front","average":{"1min":2.0089999999999999,"5min":0.94899999999999995,"15min":0.81299999999999994},"min":{"1min":0.439,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":26.146999999999998,"5min":26.146999999999998,"15min":26.146999999999998},"last":0.69399999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":0.95499999999999996,"15min":0.81000000000000005},"min":{"1min":0.46300000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":6.6699999999999999,"5min":21.535,"15min":21.535},"last":0.70599999999999996},{"interface":"front","average":{"1min":2.649,"5min":1.2250000000000001,"15min":0.94599999999999995},"min":{"1min":0.46999999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":30.488,"5min":30.488,"15min":30.488},"last":0.84699999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":0.874,"15min":0.82599999999999996},"min":{"1min":0.51600000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":2.758,"5min":9.2240000000000002,"15min":9.2240000000000002},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.96699999999999997,"15min":0.877},"min":{"1min":0.44600000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.3520000000000003,"5min":21.574000000000002,"15min":21.574000000000002},"last":0.81899999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542200,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36556,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335284,"statfs":{"total":96636764160,"available":96599330816,"internally_reserved":0,"allocated":1245184,"data_stored":858586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.52600000000000002,"15min":0.48599999999999999},"min":{"1min":0.26100000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.2409999999999997,"5min":4.2409999999999997,"15min":4.2409999999999997},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55600000000000005,"15min":0.55300000000000005},"min":{"1min":0.35499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":2.0219999999999998,"5min":2.0219999999999998,"15min":2.0219999999999998},"last":0.53800000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.57599999999999996,"15min":0.57399999999999995},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.2330000000000001,"5min":2.2330000000000001,"15min":2.2330000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.53500000000000003,"15min":0.497},"min":{"1min":0.30099999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.6789999999999998,"5min":2.6789999999999998,"15min":2.6789999999999998},"last":0.56899999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.63900000000000001,"15min":0.59699999999999998},"min":{"1min":0.38900000000000001,"5min":0.308,"15min":0.308},"max":{"1min":4.274,"5min":4.274,"15min":4.274},"last":0.503},{"interface":"front","average":{"1min":1.0309999999999999,"5min":0.68200000000000005,"15min":0.63400000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":7.2309999999999999,"5min":7.2309999999999999,"15min":7.2309999999999999},"last":0.61799999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":2.339,"5min":1.0680000000000001,"15min":0.878},"min":{"1min":0.46600000000000003,"5min":0.38,"15min":0.38},"max":{"1min":22.814,"5min":22.814,"15min":22.814},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.516,"5min":0.92700000000000005,"15min":0.86499999999999999},"min":{"1min":0.54500000000000004,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":4.4800000000000004,"5min":4.4800000000000004,"15min":4.4800000000000004},"last":0.90800000000000003}]},{"osd":5,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":1.8600000000000001,"5min":0.97499999999999998,"15min":0.85799999999999998},"min":{"1min":0.495,"5min":0.441,"15min":0.441},"max":{"1min":12.428000000000001,"5min":12.428000000000001,"15min":12.428000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.915,"5min":1.018,"15min":0.91000000000000003},"min":{"1min":0.63800000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":13.064,"5min":13.064,"15min":13.064},"last":0.82799999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":2.8029999999999999,"5min":1.1699999999999999,"15min":0.92400000000000004},"min":{"1min":0.50800000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":31.059000000000001,"5min":31.059000000000001,"15min":31.059000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":3.6059999999999999,"5min":1.357,"15min":1.022},"min":{"1min":0.52000000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":40.731000000000002,"5min":40.731000000000002,"15min":40.731000000000002},"last":0.98399999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":1.657,"5min":0.96499999999999997,"15min":0.878},"min":{"1min":0.53800000000000003,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":14.032,"5min":14.032,"15min":14.032},"last":0.93200000000000005},{"interface":"front","average":{"1min":2.9569999999999999,"5min":1.254,"15min":0.995},"min":{"1min":0.58099999999999996,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":23.524000000000001,"5min":23.524000000000001,"15min":23.524000000000001},"last":0.878}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:09:14.652 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:09:14.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.826+0000 7f9bff800640 1 Processor -- start 2024-01-15T18:09:14.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.826+0000 7f9bff800640 1 -- start start 2024-01-15T18:09:14.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.830+0000 7f9bff800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9bf80701a0 0x7f9bf8070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:14.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.830+0000 7f9bff800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f9bf80c92b0 0x7f9bf806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:14.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.830+0000 7f9bff800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf806c4a0 con 0x7f9bf806e240 2024-01-15T18:09:14.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.830+0000 7f9bff800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf8070ac0 con 0x7f9bf80701a0 2024-01-15T18:09:14.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.830+0000 7f9bff800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf815c660 con 0x7f9bf80c92b0 2024-01-15T18:09:14.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.830+0000 7f9bfce00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9bf80701a0 0x7f9bf8070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:14.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.830+0000 7f9bfd800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9bf806e240 0x7f9bf815d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49566/0 (socket says 172.21.15.27:49566) 2024-01-15T18:09:14.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.830+0000 7f9bfd800640 1 -- 172.21.15.27:0/2498502639 learned_addr learned my addr 172.21.15.27:0/2498502639 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:14.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.830+0000 7f9bf7e00640 1 --2- 172.21.15.27:0/2498502639 >> v2:172.21.15.93:3300/0 conn(0x7f9bf80c92b0 0x7f9bf806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:14.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.830+0000 7f9bfce00640 1 -- 172.21.15.27:0/2498502639 >> v2:172.21.15.93:3300/0 conn(0x7f9bf80c92b0 msgr2=0x7f9bf806dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:14.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.830+0000 7f9bfce00640 1 --2- 172.21.15.27:0/2498502639 >> v2:172.21.15.93:3300/0 conn(0x7f9bf80c92b0 0x7f9bf806dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.830+0000 7f9bfce00640 1 -- 172.21.15.27:0/2498502639 >> v1:172.21.15.27:6789/0 conn(0x7f9bf806e240 legacy=0x7f9bf815d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:14.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.830+0000 7f9bfce00640 1 -- 172.21.15.27:0/2498502639 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf815c940 con 0x7f9bf80701a0 2024-01-15T18:09:14.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.834+0000 7f9bfce00640 1 --2- 172.21.15.27:0/2498502639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bf80701a0 0x7f9bf8070580 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8067d80 tx=0x7f9be801c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f4371b1acae6c2e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:14.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.834+0000 7f9bf7400640 1 -- 172.21.15.27:0/2498502639 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be80a7070 con 0x7f9bf80701a0 2024-01-15T18:09:14.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.834+0000 7f9bf7400640 1 -- 172.21.15.27:0/2498502639 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be80ab430 con 0x7f9bf80701a0 2024-01-15T18:09:14.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.834+0000 7f9bf7400640 1 -- 172.21.15.27:0/2498502639 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be80ab7c0 con 0x7f9bf80701a0 2024-01-15T18:09:14.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.834+0000 7f9bff800640 1 -- 172.21.15.27:0/2498502639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bf80701a0 msgr2=0x7f9bf8070580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:14.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.834+0000 7f9bff800640 1 --2- 172.21.15.27:0/2498502639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bf80701a0 0x7f9bf8070580 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9bf8067d80 tx=0x7f9be801c430 comp rx=0 tx=0).stop 2024-01-15T18:09:14.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.834+0000 7f9bff800640 1 -- 172.21.15.27:0/2498502639 shutdown_connections 2024-01-15T18:09:14.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.834+0000 7f9bff800640 1 --2- 172.21.15.27:0/2498502639 >> v2:172.21.15.93:3300/0 conn(0x7f9bf80c92b0 0x7f9bf806dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.834+0000 7f9bff800640 1 --2- 172.21.15.27:0/2498502639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bf80701a0 0x7f9bf8070580 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.834+0000 7f9bff800640 1 -- 172.21.15.27:0/2498502639 >> 172.21.15.27:0/2498502639 conn(0x7f9bf80cec60 msgr2=0x7f9bf80cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:14.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.834+0000 7f9bff800640 1 -- 172.21.15.27:0/2498502639 shutdown_connections 2024-01-15T18:09:14.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.834+0000 7f9bff800640 1 -- 172.21.15.27:0/2498502639 wait complete. 2024-01-15T18:09:14.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.834+0000 7f9bff800640 1 Processor -- start 2024-01-15T18:09:14.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.838+0000 7f9bff800640 1 -- start start 2024-01-15T18:09:14.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.838+0000 7f9bff800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9bf806e240 0x7f9bf80c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:14.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.838+0000 7f9bff800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9bf80701a0 0x7f9bf80c2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:14.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.838+0000 7f9bff800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bf80c92b0 0x7f9bf80c2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:14.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.838+0000 7f9bff800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf80c30f0 con 0x7f9bf80c92b0 2024-01-15T18:09:14.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.838+0000 7f9bff800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf80c4400 con 0x7f9bf806e240 2024-01-15T18:09:14.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.838+0000 7f9bff800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf80c3360 con 0x7f9bf80701a0 2024-01-15T18:09:14.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.838+0000 7f9bfd800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bf80c92b0 0x7f9bf80c2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:14.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.838+0000 7f9bfd800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bf80c92b0 0x7f9bf80c2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34376/0 (socket says 172.21.15.27:34376) 2024-01-15T18:09:14.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.838+0000 7f9bfd800640 1 -- 172.21.15.27:0/76289685 learned_addr learned my addr 172.21.15.27:0/76289685 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:14.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.838+0000 7f9bfce00640 1 --2- 172.21.15.27:0/76289685 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9bf806e240 0x7f9bf80c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:14.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.838+0000 7f9bf7e00640 1 --2- 172.21.15.27:0/76289685 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9bf80701a0 0x7f9bf80c2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:14.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.838+0000 7f9bfd800640 1 -- 172.21.15.27:0/76289685 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9bf80701a0 msgr2=0x7f9bf80c2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:14.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.838+0000 7f9bfd800640 1 --2- 172.21.15.27:0/76289685 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9bf80701a0 0x7f9bf80c2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.838+0000 7f9bfd800640 1 -- 172.21.15.27:0/76289685 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9bf806e240 msgr2=0x7f9bf80c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:14.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.838+0000 7f9bfd800640 1 --2- 172.21.15.27:0/76289685 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9bf806e240 0x7f9bf80c7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.838+0000 7f9bfd800640 1 -- 172.21.15.27:0/76289685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be8009420 con 0x7f9bf80c92b0 2024-01-15T18:09:14.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.838+0000 7f9bfd800640 1 --2- 172.21.15.27:0/76289685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bf80c92b0 0x7f9bf80c2bb0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f9bf000cbc0 tx=0x7f9bf000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:14.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.838+0000 7f9bf4c00640 1 -- 172.21.15.27:0/76289685 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf000ad40 con 0x7f9bf80c92b0 2024-01-15T18:09:14.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.838+0000 7f9bf4c00640 1 -- 172.21.15.27:0/76289685 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bf0012400 con 0x7f9bf80c92b0 2024-01-15T18:09:14.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.838+0000 7f9bff800640 1 -- 172.21.15.27:0/76289685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf80c3640 con 0x7f9bf80c92b0 2024-01-15T18:09:14.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.838+0000 7f9bf4c00640 1 -- 172.21.15.27:0/76289685 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf000aea0 con 0x7f9bf80c92b0 2024-01-15T18:09:14.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.838+0000 7f9bff800640 1 -- 172.21.15.27:0/76289685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf80c3c00 con 0x7f9bf80c92b0 2024-01-15T18:09:14.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.838+0000 7f9bff800640 1 -- 172.21.15.27:0/76289685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf804e850 con 0x7f9bf80c92b0 2024-01-15T18:09:14.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.842+0000 7f9bf4c00640 1 -- 172.21.15.27:0/76289685 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f9bf0007070 con 0x7f9bf80c92b0 2024-01-15T18:09:14.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.842+0000 7f9bf4c00640 1 --2- 172.21.15.27:0/76289685 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9bd0066e90 0x7f9bd0069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:14.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.842+0000 7f9bf4c00640 1 -- 172.21.15.27:0/76289685 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f9bf0084de0 con 0x7f9bf80c92b0 2024-01-15T18:09:14.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.842+0000 7f9bfce00640 1 --2- 172.21.15.27:0/76289685 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9bd0066e90 0x7f9bd0069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:14.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.842+0000 7f9bfce00640 1 --2- 172.21.15.27:0/76289685 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9bd0066e90 0x7f9bd0069350 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f9be80af000 tx=0x7f9be80a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:14.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.846+0000 7f9bf4c00640 1 -- 172.21.15.27:0/76289685 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9bf008a020 con 0x7f9bf80c92b0 2024-01-15T18:09:15.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:14.998+0000 7f9bff800640 1 -- 172.21.15.27:0/76289685 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9bf806cfd0 con 0x7f9bd0066e90 2024-01-15T18:09:15.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.006+0000 7f9bf4c00640 1 -- 172.21.15.27:0/76289685 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287472 (secure 0 0 0) 0x7f9bf806cfd0 con 0x7f9bd0066e90 2024-01-15T18:09:15.015 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:09:15.016 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:09:15.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.010+0000 7f9bff800640 1 -- 172.21.15.27:0/76289685 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9bd0066e90 msgr2=0x7f9bd0069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:15.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.010+0000 7f9bff800640 1 --2- 172.21.15.27:0/76289685 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9bd0066e90 0x7f9bd0069350 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f9be80af000 tx=0x7f9be80a7000 comp rx=0 tx=0).stop 2024-01-15T18:09:15.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.010+0000 7f9bff800640 1 -- 172.21.15.27:0/76289685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bf80c92b0 msgr2=0x7f9bf80c2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:15.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.010+0000 7f9bff800640 1 --2- 172.21.15.27:0/76289685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bf80c92b0 0x7f9bf80c2bb0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f9bf000cbc0 tx=0x7f9bf000a430 comp rx=0 tx=0).stop 2024-01-15T18:09:15.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.010+0000 7f9bff800640 1 -- 172.21.15.27:0/76289685 shutdown_connections 2024-01-15T18:09:15.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.010+0000 7f9bff800640 1 --2- 172.21.15.27:0/76289685 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9bd0066e90 0x7f9bd0069350 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:15.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.010+0000 7f9bff800640 1 --2- 172.21.15.27:0/76289685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9bf80c92b0 0x7f9bf80c2bb0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:15.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.010+0000 7f9bff800640 1 --2- 172.21.15.27:0/76289685 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9bf80701a0 0x7f9bf80c2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:15.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.010+0000 7f9bff800640 1 --2- 172.21.15.27:0/76289685 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9bf806e240 0x7f9bf80c7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:15.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.010+0000 7f9bff800640 1 -- 172.21.15.27:0/76289685 >> 172.21.15.27:0/76289685 conn(0x7f9bf80cec60 msgr2=0x7f9bf8071af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:15.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.010+0000 7f9bff800640 1 -- 172.21.15.27:0/76289685 shutdown_connections 2024-01-15T18:09:15.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.010+0000 7f9bff800640 1 -- 172.21.15.27:0/76289685 wait complete. 2024-01-15T18:09:15.037 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-01-15T18:09:14.206380+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592196,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":193,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":289120,"kb_used_data":6384,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754685600,"statfs":{"total":773094113280,"available":772798054400,"internally_reserved":0,"allocated":6537216,"data_stored":3391187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":289394224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1828,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001226"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.174696+0000","last_change":"2024-01-15T18:09:10.164426+0000","last_active":"2024-01-15T18:09:10.174696+0000","last_peered":"2024-01-15T18:09:10.174696+0000","last_clean":"2024-01-15T18:09:10.174696+0000","last_became_active":"2024-01-15T18:09:10.164120+0000","last_became_peered":"2024-01-15T18:09:10.164120+0000","last_unstale":"2024-01-15T18:09:10.174696+0000","last_undegraded":"2024-01-15T18:09:10.174696+0000","last_fullsized":"2024-01-15T18:09:10.174696+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:13:02.463974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.156544+0000","last_change":"2024-01-15T18:09:10.149693+0000","last_active":"2024-01-15T18:09:10.156544+0000","last_peered":"2024-01-15T18:09:10.156544+0000","last_clean":"2024-01-15T18:09:10.156544+0000","last_became_active":"2024-01-15T18:09:10.149380+0000","last_became_peered":"2024-01-15T18:09:10.149380+0000","last_unstale":"2024-01-15T18:09:10.156544+0000","last_undegraded":"2024-01-15T18:09:10.156544+0000","last_fullsized":"2024-01-15T18:09:10.156544+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:14:10.432447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.176732+0000","last_change":"2024-01-15T18:09:10.172281+0000","last_active":"2024-01-15T18:09:10.176732+0000","last_peered":"2024-01-15T18:09:10.176732+0000","last_clean":"2024-01-15T18:09:10.176732+0000","last_became_active":"2024-01-15T18:09:10.171520+0000","last_became_peered":"2024-01-15T18:09:10.171520+0000","last_unstale":"2024-01-15T18:09:10.176732+0000","last_undegraded":"2024-01-15T18:09:10.176732+0000","last_fullsized":"2024-01-15T18:09:10.176732+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:05.433977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155356+0000","last_change":"2024-01-15T18:09:10.154186+0000","last_active":"2024-01-15T18:09:11.155356+0000","last_peered":"2024-01-15T18:09:11.155356+0000","last_clean":"2024-01-15T18:09:11.155356+0000","last_became_active":"2024-01-15T18:09:10.153961+0000","last_became_peered":"2024-01-15T18:09:10.153961+0000","last_unstale":"2024-01-15T18:09:11.155356+0000","last_undegraded":"2024-01-15T18:09:11.155356+0000","last_fullsized":"2024-01-15T18:09:11.155356+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:44:11.424905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163531+0000","last_change":"2024-01-15T18:09:10.149056+0000","last_active":"2024-01-15T18:09:10.163531+0000","last_peered":"2024-01-15T18:09:10.163531+0000","last_clean":"2024-01-15T18:09:10.163531+0000","last_became_active":"2024-01-15T18:09:10.148756+0000","last_became_peered":"2024-01-15T18:09:10.148756+0000","last_unstale":"2024-01-15T18:09:10.163531+0000","last_undegraded":"2024-01-15T18:09:10.163531+0000","last_fullsized":"2024-01-15T18:09:10.163531+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:20.006996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.127717+0000","last_change":"2024-01-15T18:09:09.127717+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.127717+0000","last_undegraded":"2024-01-15T18:09:09.127717+0000","last_fullsized":"2024-01-15T18:09:09.127717+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.149739+0000","last_change":"2024-01-15T18:09:10.147574+0000","last_active":"2024-01-15T18:09:10.149739+0000","last_peered":"2024-01-15T18:09:10.149739+0000","last_clean":"2024-01-15T18:09:10.149739+0000","last_became_active":"2024-01-15T18:09:10.147288+0000","last_became_peered":"2024-01-15T18:09:10.147288+0000","last_unstale":"2024-01-15T18:09:10.149739+0000","last_undegraded":"2024-01-15T18:09:10.149739+0000","last_fullsized":"2024-01-15T18:09:10.149739+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:29:52.274001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.160773+0000","last_change":"2024-01-15T18:09:10.149706+0000","last_active":"2024-01-15T18:09:10.160773+0000","last_peered":"2024-01-15T18:09:10.160773+0000","last_clean":"2024-01-15T18:09:10.160773+0000","last_became_active":"2024-01-15T18:09:10.149353+0000","last_became_peered":"2024-01-15T18:09:10.149353+0000","last_unstale":"2024-01-15T18:09:10.160773+0000","last_undegraded":"2024-01-15T18:09:10.160773+0000","last_fullsized":"2024-01-15T18:09:10.160773+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:22:35.521714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.166664+0000","last_change":"2024-01-15T18:09:10.162835+0000","last_active":"2024-01-15T18:09:10.166664+0000","last_peered":"2024-01-15T18:09:10.166664+0000","last_clean":"2024-01-15T18:09:10.166664+0000","last_became_active":"2024-01-15T18:09:10.162459+0000","last_became_peered":"2024-01-15T18:09:10.162459+0000","last_unstale":"2024-01-15T18:09:10.166664+0000","last_undegraded":"2024-01-15T18:09:10.166664+0000","last_fullsized":"2024-01-15T18:09:10.166664+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:26.052280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.164826+0000","last_change":"2024-01-15T18:09:10.151168+0000","last_active":"2024-01-15T18:09:10.164826+0000","last_peered":"2024-01-15T18:09:10.164826+0000","last_clean":"2024-01-15T18:09:10.164826+0000","last_became_active":"2024-01-15T18:09:10.150812+0000","last_became_peered":"2024-01-15T18:09:10.150812+0000","last_unstale":"2024-01-15T18:09:10.164826+0000","last_undegraded":"2024-01-15T18:09:10.164826+0000","last_fullsized":"2024-01-15T18:09:10.164826+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:09:59.563201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.157867+0000","last_change":"2024-01-15T18:09:10.148209+0000","last_active":"2024-01-15T18:09:10.157867+0000","last_peered":"2024-01-15T18:09:10.157867+0000","last_clean":"2024-01-15T18:09:10.157867+0000","last_became_active":"2024-01-15T18:09:10.147897+0000","last_became_peered":"2024-01-15T18:09:10.147897+0000","last_unstale":"2024-01-15T18:09:10.157867+0000","last_undegraded":"2024-01-15T18:09:10.157867+0000","last_fullsized":"2024-01-15T18:09:10.157867+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:53.437883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155548+0000","last_change":"2024-01-15T18:09:10.153400+0000","last_active":"2024-01-15T18:09:11.155548+0000","last_peered":"2024-01-15T18:09:11.155548+0000","last_clean":"2024-01-15T18:09:11.155548+0000","last_became_active":"2024-01-15T18:09:10.152222+0000","last_became_peered":"2024-01-15T18:09:10.152222+0000","last_unstale":"2024-01-15T18:09:11.155548+0000","last_undegraded":"2024-01-15T18:09:11.155548+0000","last_fullsized":"2024-01-15T18:09:11.155548+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:03.126844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.165807+0000","last_change":"2024-01-15T18:09:10.160925+0000","last_active":"2024-01-15T18:09:10.165807+0000","last_peered":"2024-01-15T18:09:10.165807+0000","last_clean":"2024-01-15T18:09:10.165807+0000","last_became_active":"2024-01-15T18:09:10.160620+0000","last_became_peered":"2024-01-15T18:09:10.160620+0000","last_unstale":"2024-01-15T18:09:10.165807+0000","last_undegraded":"2024-01-15T18:09:10.165807+0000","last_fullsized":"2024-01-15T18:09:10.165807+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:40:15.227589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.166438+0000","last_change":"2024-01-15T18:09:10.154922+0000","last_active":"2024-01-15T18:09:10.166438+0000","last_peered":"2024-01-15T18:09:10.166438+0000","last_clean":"2024-01-15T18:09:10.166438+0000","last_became_active":"2024-01-15T18:09:10.154594+0000","last_became_peered":"2024-01-15T18:09:10.154594+0000","last_unstale":"2024-01-15T18:09:10.166438+0000","last_undegraded":"2024-01-15T18:09:10.166438+0000","last_fullsized":"2024-01-15T18:09:10.166438+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:55:44.009610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.148569+0000","last_change":"2024-01-15T18:09:10.143163+0000","last_active":"2024-01-15T18:09:10.148569+0000","last_peered":"2024-01-15T18:09:10.148569+0000","last_clean":"2024-01-15T18:09:10.148569+0000","last_became_active":"2024-01-15T18:09:10.142874+0000","last_became_peered":"2024-01-15T18:09:10.142874+0000","last_unstale":"2024-01-15T18:09:10.148569+0000","last_undegraded":"2024-01-15T18:09:10.148569+0000","last_fullsized":"2024-01-15T18:09:10.148569+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:02:56.938951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.127271+0000","last_change":"2024-01-15T18:09:09.127271+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.127271+0000","last_undegraded":"2024-01-15T18:09:09.127271+0000","last_fullsized":"2024-01-15T18:09:09.127271+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163654+0000","last_change":"2024-01-15T18:09:10.156152+0000","last_active":"2024-01-15T18:09:10.163654+0000","last_peered":"2024-01-15T18:09:10.163654+0000","last_clean":"2024-01-15T18:09:10.163654+0000","last_became_active":"2024-01-15T18:09:10.155849+0000","last_became_peered":"2024-01-15T18:09:10.155849+0000","last_unstale":"2024-01-15T18:09:10.163654+0000","last_undegraded":"2024-01-15T18:09:10.163654+0000","last_fullsized":"2024-01-15T18:09:10.163654+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:53.373020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.177069+0000","last_change":"2024-01-15T18:09:10.168066+0000","last_active":"2024-01-15T18:09:10.177069+0000","last_peered":"2024-01-15T18:09:10.177069+0000","last_clean":"2024-01-15T18:09:10.177069+0000","last_became_active":"2024-01-15T18:09:10.167691+0000","last_became_peered":"2024-01-15T18:09:10.167691+0000","last_unstale":"2024-01-15T18:09:10.177069+0000","last_undegraded":"2024-01-15T18:09:10.177069+0000","last_fullsized":"2024-01-15T18:09:10.177069+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:58:26.855862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.174836+0000","last_change":"2024-01-15T18:09:10.163790+0000","last_active":"2024-01-15T18:09:10.174836+0000","last_peered":"2024-01-15T18:09:10.174836+0000","last_clean":"2024-01-15T18:09:10.174836+0000","last_became_active":"2024-01-15T18:09:10.163509+0000","last_became_peered":"2024-01-15T18:09:10.163509+0000","last_unstale":"2024-01-15T18:09:10.174836+0000","last_undegraded":"2024-01-15T18:09:10.174836+0000","last_fullsized":"2024-01-15T18:09:10.174836+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:22:24.247456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173893+0000","last_change":"2024-01-15T18:09:10.161978+0000","last_active":"2024-01-15T18:09:10.173893+0000","last_peered":"2024-01-15T18:09:10.173893+0000","last_clean":"2024-01-15T18:09:10.173893+0000","last_became_active":"2024-01-15T18:09:10.161717+0000","last_became_peered":"2024-01-15T18:09:10.161717+0000","last_unstale":"2024-01-15T18:09:10.173893+0000","last_undegraded":"2024-01-15T18:09:10.173893+0000","last_fullsized":"2024-01-15T18:09:10.173893+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:05.433977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163131+0000","last_change":"2024-01-15T18:09:10.159447+0000","last_active":"2024-01-15T18:09:10.163131+0000","last_peered":"2024-01-15T18:09:10.163131+0000","last_clean":"2024-01-15T18:09:10.163131+0000","last_became_active":"2024-01-15T18:09:10.159055+0000","last_became_peered":"2024-01-15T18:09:10.159055+0000","last_unstale":"2024-01-15T18:09:10.163131+0000","last_undegraded":"2024-01-15T18:09:10.163131+0000","last_fullsized":"2024-01-15T18:09:10.163131+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:54:09.462802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173806+0000","last_change":"2024-01-15T18:09:10.157352+0000","last_active":"2024-01-15T18:09:10.173806+0000","last_peered":"2024-01-15T18:09:10.173806+0000","last_clean":"2024-01-15T18:09:10.173806+0000","last_became_active":"2024-01-15T18:09:10.155991+0000","last_became_peered":"2024-01-15T18:09:10.155991+0000","last_unstale":"2024-01-15T18:09:10.173806+0000","last_undegraded":"2024-01-15T18:09:10.173806+0000","last_fullsized":"2024-01-15T18:09:10.173806+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:25:19.060578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163476+0000","last_change":"2024-01-15T18:09:10.160338+0000","last_active":"2024-01-15T18:09:10.163476+0000","last_peered":"2024-01-15T18:09:10.163476+0000","last_clean":"2024-01-15T18:09:10.163476+0000","last_became_active":"2024-01-15T18:09:10.159325+0000","last_became_peered":"2024-01-15T18:09:10.159325+0000","last_unstale":"2024-01-15T18:09:10.163476+0000","last_undegraded":"2024-01-15T18:09:10.163476+0000","last_fullsized":"2024-01-15T18:09:10.163476+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:46:36.631105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.168684+0000","last_change":"2024-01-15T18:09:10.167327+0000","last_active":"2024-01-15T18:09:10.168684+0000","last_peered":"2024-01-15T18:09:10.168684+0000","last_clean":"2024-01-15T18:09:10.168684+0000","last_became_active":"2024-01-15T18:09:10.167008+0000","last_became_peered":"2024-01-15T18:09:10.167008+0000","last_unstale":"2024-01-15T18:09:10.168684+0000","last_undegraded":"2024-01-15T18:09:10.168684+0000","last_fullsized":"2024-01-15T18:09:10.168684+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:18:13.449428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.174731+0000","last_change":"2024-01-15T18:09:10.165962+0000","last_active":"2024-01-15T18:09:10.174731+0000","last_peered":"2024-01-15T18:09:10.174731+0000","last_clean":"2024-01-15T18:09:10.174731+0000","last_became_active":"2024-01-15T18:09:10.165689+0000","last_became_peered":"2024-01-15T18:09:10.165689+0000","last_unstale":"2024-01-15T18:09:10.174731+0000","last_undegraded":"2024-01-15T18:09:10.174731+0000","last_fullsized":"2024-01-15T18:09:10.174731+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:48:02.569878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155565+0000","last_change":"2024-01-15T18:09:10.147816+0000","last_active":"2024-01-15T18:09:11.155565+0000","last_peered":"2024-01-15T18:09:11.155565+0000","last_clean":"2024-01-15T18:09:11.155565+0000","last_became_active":"2024-01-15T18:09:10.147487+0000","last_became_peered":"2024-01-15T18:09:10.147487+0000","last_unstale":"2024-01-15T18:09:11.155565+0000","last_undegraded":"2024-01-15T18:09:11.155565+0000","last_fullsized":"2024-01-15T18:09:11.155565+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:40:51.237238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.133285+0000","last_change":"2024-01-15T18:09:09.133285+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.133285+0000","last_undegraded":"2024-01-15T18:09:09.133285+0000","last_fullsized":"2024-01-15T18:09:09.133285+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.156169+0000","last_change":"2024-01-15T18:09:10.149120+0000","last_active":"2024-01-15T18:09:10.156169+0000","last_peered":"2024-01-15T18:09:10.156169+0000","last_clean":"2024-01-15T18:09:10.156169+0000","last_became_active":"2024-01-15T18:09:10.148675+0000","last_became_peered":"2024-01-15T18:09:10.148675+0000","last_unstale":"2024-01-15T18:09:10.156169+0000","last_undegraded":"2024-01-15T18:09:10.156169+0000","last_fullsized":"2024-01-15T18:09:10.156169+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:52:23.460592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173871+0000","last_change":"2024-01-15T18:09:10.161188+0000","last_active":"2024-01-15T18:09:10.173871+0000","last_peered":"2024-01-15T18:09:10.173871+0000","last_clean":"2024-01-15T18:09:10.173871+0000","last_became_active":"2024-01-15T18:09:10.151645+0000","last_became_peered":"2024-01-15T18:09:10.151645+0000","last_unstale":"2024-01-15T18:09:10.173871+0000","last_undegraded":"2024-01-15T18:09:10.173871+0000","last_fullsized":"2024-01-15T18:09:10.173871+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:07:18.177097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.176602+0000","last_change":"2024-01-15T18:09:10.167895+0000","last_active":"2024-01-15T18:09:10.176602+0000","last_peered":"2024-01-15T18:09:10.176602+0000","last_clean":"2024-01-15T18:09:10.176602+0000","last_became_active":"2024-01-15T18:09:10.167639+0000","last_became_peered":"2024-01-15T18:09:10.167639+0000","last_unstale":"2024-01-15T18:09:10.176602+0000","last_undegraded":"2024-01-15T18:09:10.176602+0000","last_fullsized":"2024-01-15T18:09:10.176602+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:18:53.201764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.161004+0000","last_change":"2024-01-15T18:09:10.148253+0000","last_active":"2024-01-15T18:09:10.161004+0000","last_peered":"2024-01-15T18:09:10.161004+0000","last_clean":"2024-01-15T18:09:10.161004+0000","last_became_active":"2024-01-15T18:09:10.147941+0000","last_became_peered":"2024-01-15T18:09:10.147941+0000","last_unstale":"2024-01-15T18:09:10.161004+0000","last_undegraded":"2024-01-15T18:09:10.161004+0000","last_fullsized":"2024-01-15T18:09:10.161004+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:04:16.316753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.126719+0000","last_change":"2024-01-15T18:09:09.126719+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.126719+0000","last_undegraded":"2024-01-15T18:09:09.126719+0000","last_fullsized":"2024-01-15T18:09:09.126719+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.138025+0000","last_change":"2024-01-15T18:09:09.138025+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.138025+0000","last_undegraded":"2024-01-15T18:09:09.138025+0000","last_fullsized":"2024-01-15T18:09:09.138025+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.155581+0000","last_change":"2024-01-15T18:09:09.152802+0000","last_active":"2024-01-15T18:09:09.155581+0000","last_peered":"2024-01-15T18:09:09.155581+0000","last_clean":"2024-01-15T18:09:09.155581+0000","last_became_active":"2024-01-15T18:09:09.152520+0000","last_became_peered":"2024-01-15T18:09:09.152520+0000","last_unstale":"2024-01-15T18:09:09.155581+0000","last_undegraded":"2024-01-15T18:09:09.155581+0000","last_fullsized":"2024-01-15T18:09:09.155581+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:04.586121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.170556+0000","last_change":"2024-01-15T18:09:10.161734+0000","last_active":"2024-01-15T18:09:10.170556+0000","last_peered":"2024-01-15T18:09:10.170556+0000","last_clean":"2024-01-15T18:09:10.170556+0000","last_became_active":"2024-01-15T18:09:10.161466+0000","last_became_peered":"2024-01-15T18:09:10.161466+0000","last_unstale":"2024-01-15T18:09:10.170556+0000","last_undegraded":"2024-01-15T18:09:10.170556+0000","last_fullsized":"2024-01-15T18:09:10.170556+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:58:24.569904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"57'1","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.177728+0000","last_change":"2024-01-15T18:09:09.125625+0000","last_active":"2024-01-15T18:09:10.177728+0000","last_peered":"2024-01-15T18:09:10.177728+0000","last_clean":"2024-01-15T18:09:10.177728+0000","last_became_active":"2024-01-15T18:09:09.125373+0000","last_became_peered":"2024-01-15T18:09:09.125373+0000","last_unstale":"2024-01-15T18:09:10.177728+0000","last_undegraded":"2024-01-15T18:09:10.177728+0000","last_fullsized":"2024-01-15T18:09:10.177728+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:14:59.883458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.130696+0000","last_change":"2024-01-15T18:09:09.130696+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.130696+0000","last_undegraded":"2024-01-15T18:09:09.130696+0000","last_fullsized":"2024-01-15T18:09:09.130696+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.155587+0000","last_change":"2024-01-15T18:09:09.153007+0000","last_active":"2024-01-15T18:09:09.155587+0000","last_peered":"2024-01-15T18:09:09.155587+0000","last_clean":"2024-01-15T18:09:09.155587+0000","last_became_active":"2024-01-15T18:09:09.152405+0000","last_became_peered":"2024-01-15T18:09:09.152405+0000","last_unstale":"2024-01-15T18:09:09.155587+0000","last_undegraded":"2024-01-15T18:09:09.155587+0000","last_fullsized":"2024-01-15T18:09:09.155587+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:53:06.843401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.132775+0000","last_change":"2024-01-15T18:09:09.132775+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.132775+0000","last_undegraded":"2024-01-15T18:09:09.132775+0000","last_fullsized":"2024-01-15T18:09:09.132775+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.176876+0000","last_change":"2024-01-15T18:09:09.138209+0000","last_active":"2024-01-15T18:09:10.176876+0000","last_peered":"2024-01-15T18:09:10.176876+0000","last_clean":"2024-01-15T18:09:10.176876+0000","last_became_active":"2024-01-15T18:09:09.137921+0000","last_became_peered":"2024-01-15T18:09:09.137921+0000","last_unstale":"2024-01-15T18:09:10.176876+0000","last_undegraded":"2024-01-15T18:09:10.176876+0000","last_fullsized":"2024-01-15T18:09:10.176876+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:54:56.016726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163688+0000","last_change":"2024-01-15T18:09:10.156211+0000","last_active":"2024-01-15T18:09:10.163688+0000","last_peered":"2024-01-15T18:09:10.163688+0000","last_clean":"2024-01-15T18:09:10.163688+0000","last_became_active":"2024-01-15T18:09:10.155914+0000","last_became_peered":"2024-01-15T18:09:10.155914+0000","last_unstale":"2024-01-15T18:09:10.163688+0000","last_undegraded":"2024-01-15T18:09:10.163688+0000","last_fullsized":"2024-01-15T18:09:10.163688+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:15:38.206084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"57'1","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.172500+0000","last_change":"2024-01-15T18:09:09.139963+0000","last_active":"2024-01-15T18:09:10.172500+0000","last_peered":"2024-01-15T18:09:10.172500+0000","last_clean":"2024-01-15T18:09:10.172500+0000","last_became_active":"2024-01-15T18:09:09.139603+0000","last_became_peered":"2024-01-15T18:09:09.139603+0000","last_unstale":"2024-01-15T18:09:10.172500+0000","last_undegraded":"2024-01-15T18:09:10.172500+0000","last_fullsized":"2024-01-15T18:09:10.172500+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:14:01.036199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.174128+0000","last_change":"2024-01-15T18:09:10.165154+0000","last_active":"2024-01-15T18:09:10.174128+0000","last_peered":"2024-01-15T18:09:10.174128+0000","last_clean":"2024-01-15T18:09:10.174128+0000","last_became_active":"2024-01-15T18:09:10.164900+0000","last_became_peered":"2024-01-15T18:09:10.164900+0000","last_unstale":"2024-01-15T18:09:10.174128+0000","last_undegraded":"2024-01-15T18:09:10.174128+0000","last_fullsized":"2024-01-15T18:09:10.174128+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:57:26.029056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.153437+0000","last_change":"2024-01-15T18:09:09.147874+0000","last_active":"2024-01-15T18:09:09.153437+0000","last_peered":"2024-01-15T18:09:09.153437+0000","last_clean":"2024-01-15T18:09:09.153437+0000","last_became_active":"2024-01-15T18:09:09.147666+0000","last_became_peered":"2024-01-15T18:09:09.147666+0000","last_unstale":"2024-01-15T18:09:09.153437+0000","last_undegraded":"2024-01-15T18:09:09.153437+0000","last_fullsized":"2024-01-15T18:09:09.153437+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:44:37.388258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163308+0000","last_change":"2024-01-15T18:09:10.154886+0000","last_active":"2024-01-15T18:09:10.163308+0000","last_peered":"2024-01-15T18:09:10.163308+0000","last_clean":"2024-01-15T18:09:10.163308+0000","last_became_active":"2024-01-15T18:09:10.154594+0000","last_became_peered":"2024-01-15T18:09:10.154594+0000","last_unstale":"2024-01-15T18:09:10.163308+0000","last_undegraded":"2024-01-15T18:09:10.163308+0000","last_fullsized":"2024-01-15T18:09:10.163308+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:34:09.142491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.137679+0000","last_change":"2024-01-15T18:09:09.140948+0000","last_active":"2024-01-15T18:09:10.137679+0000","last_peered":"2024-01-15T18:09:10.137679+0000","last_clean":"2024-01-15T18:09:10.137679+0000","last_became_active":"2024-01-15T18:09:09.140010+0000","last_became_peered":"2024-01-15T18:09:09.140010+0000","last_unstale":"2024-01-15T18:09:10.137679+0000","last_undegraded":"2024-01-15T18:09:10.137679+0000","last_fullsized":"2024-01-15T18:09:10.137679+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:39:25.682654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173959+0000","last_change":"2024-01-15T18:09:10.160588+0000","last_active":"2024-01-15T18:09:10.173959+0000","last_peered":"2024-01-15T18:09:10.173959+0000","last_clean":"2024-01-15T18:09:10.173959+0000","last_became_active":"2024-01-15T18:09:10.157197+0000","last_became_peered":"2024-01-15T18:09:10.157197+0000","last_unstale":"2024-01-15T18:09:10.173959+0000","last_undegraded":"2024-01-15T18:09:10.173959+0000","last_fullsized":"2024-01-15T18:09:10.173959+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:27:59.969234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.138903+0000","last_change":"2024-01-15T18:09:09.131170+0000","last_active":"2024-01-15T18:09:10.138903+0000","last_peered":"2024-01-15T18:09:10.138903+0000","last_clean":"2024-01-15T18:09:10.138903+0000","last_became_active":"2024-01-15T18:09:09.130789+0000","last_became_peered":"2024-01-15T18:09:09.130789+0000","last_unstale":"2024-01-15T18:09:10.138903+0000","last_undegraded":"2024-01-15T18:09:10.138903+0000","last_fullsized":"2024-01-15T18:09:10.138903+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:13:26.979622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.166623+0000","last_change":"2024-01-15T18:09:10.154127+0000","last_active":"2024-01-15T18:09:10.166623+0000","last_peered":"2024-01-15T18:09:10.166623+0000","last_clean":"2024-01-15T18:09:10.166623+0000","last_became_active":"2024-01-15T18:09:10.153699+0000","last_became_peered":"2024-01-15T18:09:10.153699+0000","last_unstale":"2024-01-15T18:09:10.166623+0000","last_undegraded":"2024-01-15T18:09:10.166623+0000","last_fullsized":"2024-01-15T18:09:10.166623+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:37:29.585671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.151155+0000","last_change":"2024-01-15T18:09:09.145070+0000","last_active":"2024-01-15T18:09:10.151155+0000","last_peered":"2024-01-15T18:09:10.151155+0000","last_clean":"2024-01-15T18:09:10.151155+0000","last_became_active":"2024-01-15T18:09:09.144808+0000","last_became_peered":"2024-01-15T18:09:09.144808+0000","last_unstale":"2024-01-15T18:09:10.151155+0000","last_undegraded":"2024-01-15T18:09:10.151155+0000","last_fullsized":"2024-01-15T18:09:10.151155+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:07:17.174892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.178747+0000","last_change":"2024-01-15T18:09:10.173359+0000","last_active":"2024-01-15T18:09:10.178747+0000","last_peered":"2024-01-15T18:09:10.178747+0000","last_clean":"2024-01-15T18:09:10.178747+0000","last_became_active":"2024-01-15T18:09:10.173020+0000","last_became_peered":"2024-01-15T18:09:10.173020+0000","last_unstale":"2024-01-15T18:09:10.178747+0000","last_undegraded":"2024-01-15T18:09:10.178747+0000","last_fullsized":"2024-01-15T18:09:10.178747+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:22:35.521714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.140713+0000","last_change":"2024-01-15T18:09:09.142076+0000","last_active":"2024-01-15T18:09:10.140713+0000","last_peered":"2024-01-15T18:09:10.140713+0000","last_clean":"2024-01-15T18:09:10.140713+0000","last_became_active":"2024-01-15T18:09:09.141747+0000","last_became_peered":"2024-01-15T18:09:09.141747+0000","last_unstale":"2024-01-15T18:09:10.140713+0000","last_undegraded":"2024-01-15T18:09:10.140713+0000","last_fullsized":"2024-01-15T18:09:10.140713+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:02.592720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.165189+0000","last_change":"2024-01-15T18:09:10.150403+0000","last_active":"2024-01-15T18:09:10.165189+0000","last_peered":"2024-01-15T18:09:10.165189+0000","last_clean":"2024-01-15T18:09:10.165189+0000","last_became_active":"2024-01-15T18:09:10.150113+0000","last_became_peered":"2024-01-15T18:09:10.150113+0000","last_unstale":"2024-01-15T18:09:10.165189+0000","last_undegraded":"2024-01-15T18:09:10.165189+0000","last_fullsized":"2024-01-15T18:09:10.165189+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:45:59.978009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"57'3","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.180373+0000","last_change":"2024-01-15T18:09:09.142251+0000","last_active":"2024-01-15T18:09:10.180373+0000","last_peered":"2024-01-15T18:09:10.180373+0000","last_clean":"2024-01-15T18:09:10.180373+0000","last_became_active":"2024-01-15T18:09:09.141961+0000","last_became_peered":"2024-01-15T18:09:09.141961+0000","last_unstale":"2024-01-15T18:09:10.180373+0000","last_undegraded":"2024-01-15T18:09:10.180373+0000","last_fullsized":"2024-01-15T18:09:10.180373+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:56:33.327612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.176749+0000","last_change":"2024-01-15T18:09:10.172402+0000","last_active":"2024-01-15T18:09:10.176749+0000","last_peered":"2024-01-15T18:09:10.176749+0000","last_clean":"2024-01-15T18:09:10.176749+0000","last_became_active":"2024-01-15T18:09:10.171813+0000","last_became_peered":"2024-01-15T18:09:10.171813+0000","last_unstale":"2024-01-15T18:09:10.176749+0000","last_undegraded":"2024-01-15T18:09:10.176749+0000","last_fullsized":"2024-01-15T18:09:10.176749+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:15:38.206084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"57'6","reported_seq":17,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.180712+0000","last_change":"2024-01-15T18:09:09.147769+0000","last_active":"2024-01-15T18:09:10.180712+0000","last_peered":"2024-01-15T18:09:10.180712+0000","last_clean":"2024-01-15T18:09:10.180712+0000","last_became_active":"2024-01-15T18:09:09.147356+0000","last_became_peered":"2024-01-15T18:09:09.147356+0000","last_unstale":"2024-01-15T18:09:10.180712+0000","last_undegraded":"2024-01-15T18:09:10.180712+0000","last_fullsized":"2024-01-15T18:09:10.180712+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:18:52.199559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155288+0000","last_change":"2024-01-15T18:09:10.154188+0000","last_active":"2024-01-15T18:09:11.155288+0000","last_peered":"2024-01-15T18:09:11.155288+0000","last_clean":"2024-01-15T18:09:11.155288+0000","last_became_active":"2024-01-15T18:09:10.153881+0000","last_became_peered":"2024-01-15T18:09:10.153881+0000","last_unstale":"2024-01-15T18:09:11.155288+0000","last_undegraded":"2024-01-15T18:09:11.155288+0000","last_fullsized":"2024-01-15T18:09:11.155288+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:57:04.284081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"57'2","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.180721+0000","last_change":"2024-01-15T18:09:09.141986+0000","last_active":"2024-01-15T18:09:10.180721+0000","last_peered":"2024-01-15T18:09:10.180721+0000","last_clean":"2024-01-15T18:09:10.180721+0000","last_became_active":"2024-01-15T18:09:09.141626+0000","last_became_peered":"2024-01-15T18:09:09.141626+0000","last_unstale":"2024-01-15T18:09:10.180721+0000","last_undegraded":"2024-01-15T18:09:10.180721+0000","last_fullsized":"2024-01-15T18:09:10.180721+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:44:08.042896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.176175+0000","last_change":"2024-01-15T18:09:10.163997+0000","last_active":"2024-01-15T18:09:10.176175+0000","last_peered":"2024-01-15T18:09:10.176175+0000","last_clean":"2024-01-15T18:09:10.176175+0000","last_became_active":"2024-01-15T18:09:10.163758+0000","last_became_peered":"2024-01-15T18:09:10.163758+0000","last_unstale":"2024-01-15T18:09:10.176175+0000","last_undegraded":"2024-01-15T18:09:10.176175+0000","last_fullsized":"2024-01-15T18:09:10.176175+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:02:59.936727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.144084+0000","last_change":"2024-01-15T18:09:09.142040+0000","last_active":"2024-01-15T18:09:10.144084+0000","last_peered":"2024-01-15T18:09:10.144084+0000","last_clean":"2024-01-15T18:09:10.144084+0000","last_became_active":"2024-01-15T18:09:09.141713+0000","last_became_peered":"2024-01-15T18:09:09.141713+0000","last_unstale":"2024-01-15T18:09:10.144084+0000","last_undegraded":"2024-01-15T18:09:10.144084+0000","last_fullsized":"2024-01-15T18:09:10.144084+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:17:15.118577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.166494+0000","last_change":"2024-01-15T18:09:10.156574+0000","last_active":"2024-01-15T18:09:10.166494+0000","last_peered":"2024-01-15T18:09:10.166494+0000","last_clean":"2024-01-15T18:09:10.166494+0000","last_became_active":"2024-01-15T18:09:10.156225+0000","last_became_peered":"2024-01-15T18:09:10.156225+0000","last_unstale":"2024-01-15T18:09:10.166494+0000","last_undegraded":"2024-01-15T18:09:10.166494+0000","last_fullsized":"2024-01-15T18:09:10.166494+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:15:39.864433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.137769+0000","last_change":"2024-01-15T18:09:09.135271+0000","last_active":"2024-01-15T18:09:10.137769+0000","last_peered":"2024-01-15T18:09:10.137769+0000","last_clean":"2024-01-15T18:09:10.137769+0000","last_became_active":"2024-01-15T18:09:09.135011+0000","last_became_peered":"2024-01-15T18:09:09.135011+0000","last_unstale":"2024-01-15T18:09:10.137769+0000","last_undegraded":"2024-01-15T18:09:10.137769+0000","last_fullsized":"2024-01-15T18:09:10.137769+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:29:02.803967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163455+0000","last_change":"2024-01-15T18:09:10.160727+0000","last_active":"2024-01-15T18:09:10.163455+0000","last_peered":"2024-01-15T18:09:10.163455+0000","last_clean":"2024-01-15T18:09:10.163455+0000","last_became_active":"2024-01-15T18:09:10.160132+0000","last_became_peered":"2024-01-15T18:09:10.160132+0000","last_unstale":"2024-01-15T18:09:10.163455+0000","last_undegraded":"2024-01-15T18:09:10.163455+0000","last_fullsized":"2024-01-15T18:09:10.163455+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:27:59.969234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"57'8","reported_seq":19,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.177242+0000","last_change":"2024-01-15T18:09:09.137932+0000","last_active":"2024-01-15T18:09:10.177242+0000","last_peered":"2024-01-15T18:09:10.177242+0000","last_clean":"2024-01-15T18:09:10.177242+0000","last_became_active":"2024-01-15T18:09:09.137626+0000","last_became_peered":"2024-01-15T18:09:09.137626+0000","last_unstale":"2024-01-15T18:09:10.177242+0000","last_undegraded":"2024-01-15T18:09:10.177242+0000","last_fullsized":"2024-01-15T18:09:10.177242+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:14:01.036199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.177049+0000","last_change":"2024-01-15T18:09:10.165126+0000","last_active":"2024-01-15T18:09:10.177049+0000","last_peered":"2024-01-15T18:09:10.177049+0000","last_clean":"2024-01-15T18:09:10.177049+0000","last_became_active":"2024-01-15T18:09:10.164811+0000","last_became_peered":"2024-01-15T18:09:10.164811+0000","last_unstale":"2024-01-15T18:09:10.177049+0000","last_undegraded":"2024-01-15T18:09:10.177049+0000","last_fullsized":"2024-01-15T18:09:10.177049+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:53:41.639339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"57'1","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.180173+0000","last_change":"2024-01-15T18:09:09.137481+0000","last_active":"2024-01-15T18:09:10.180173+0000","last_peered":"2024-01-15T18:09:10.180173+0000","last_clean":"2024-01-15T18:09:10.180173+0000","last_became_active":"2024-01-15T18:09:09.137205+0000","last_became_peered":"2024-01-15T18:09:09.137205+0000","last_unstale":"2024-01-15T18:09:10.180173+0000","last_undegraded":"2024-01-15T18:09:10.180173+0000","last_fullsized":"2024-01-15T18:09:10.180173+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:16:45.705931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155989+0000","last_change":"2024-01-15T18:09:10.166022+0000","last_active":"2024-01-15T18:09:11.155989+0000","last_peered":"2024-01-15T18:09:11.155989+0000","last_clean":"2024-01-15T18:09:11.155989+0000","last_became_active":"2024-01-15T18:09:10.165752+0000","last_became_peered":"2024-01-15T18:09:10.165752+0000","last_unstale":"2024-01-15T18:09:11.155989+0000","last_undegraded":"2024-01-15T18:09:11.155989+0000","last_fullsized":"2024-01-15T18:09:11.155989+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:45:32.407323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.155562+0000","last_change":"2024-01-15T18:09:09.152740+0000","last_active":"2024-01-15T18:09:09.155562+0000","last_peered":"2024-01-15T18:09:09.155562+0000","last_clean":"2024-01-15T18:09:09.155562+0000","last_became_active":"2024-01-15T18:09:09.152418+0000","last_became_peered":"2024-01-15T18:09:09.152418+0000","last_unstale":"2024-01-15T18:09:09.155562+0000","last_undegraded":"2024-01-15T18:09:09.155562+0000","last_fullsized":"2024-01-15T18:09:09.155562+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:56:49.381443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.166880+0000","last_change":"2024-01-15T18:09:10.155205+0000","last_active":"2024-01-15T18:09:10.166880+0000","last_peered":"2024-01-15T18:09:10.166880+0000","last_clean":"2024-01-15T18:09:10.166880+0000","last_became_active":"2024-01-15T18:09:10.154910+0000","last_became_peered":"2024-01-15T18:09:10.154910+0000","last_unstale":"2024-01-15T18:09:10.166880+0000","last_undegraded":"2024-01-15T18:09:10.166880+0000","last_fullsized":"2024-01-15T18:09:10.166880+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:56:34.329818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.177022+0000","last_change":"2024-01-15T18:09:09.141114+0000","last_active":"2024-01-15T18:09:10.177022+0000","last_peered":"2024-01-15T18:09:10.177022+0000","last_clean":"2024-01-15T18:09:10.177022+0000","last_became_active":"2024-01-15T18:09:09.140786+0000","last_became_peered":"2024-01-15T18:09:09.140786+0000","last_unstale":"2024-01-15T18:09:10.177022+0000","last_undegraded":"2024-01-15T18:09:10.177022+0000","last_fullsized":"2024-01-15T18:09:10.177022+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:37:27.921887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.176767+0000","last_change":"2024-01-15T18:09:10.172606+0000","last_active":"2024-01-15T18:09:10.176767+0000","last_peered":"2024-01-15T18:09:10.176767+0000","last_clean":"2024-01-15T18:09:10.176767+0000","last_became_active":"2024-01-15T18:09:10.172090+0000","last_became_peered":"2024-01-15T18:09:10.172090+0000","last_unstale":"2024-01-15T18:09:10.176767+0000","last_undegraded":"2024-01-15T18:09:10.176767+0000","last_fullsized":"2024-01-15T18:09:10.176767+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:25:19.060578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.153526+0000","last_change":"2024-01-15T18:09:09.148967+0000","last_active":"2024-01-15T18:09:09.153526+0000","last_peered":"2024-01-15T18:09:09.153526+0000","last_clean":"2024-01-15T18:09:09.153526+0000","last_became_active":"2024-01-15T18:09:09.148670+0000","last_became_peered":"2024-01-15T18:09:09.148670+0000","last_unstale":"2024-01-15T18:09:09.153526+0000","last_undegraded":"2024-01-15T18:09:09.153526+0000","last_fullsized":"2024-01-15T18:09:09.153526+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:04:47.742716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155444+0000","last_change":"2024-01-15T18:09:10.151767+0000","last_active":"2024-01-15T18:09:11.155444+0000","last_peered":"2024-01-15T18:09:11.155444+0000","last_clean":"2024-01-15T18:09:11.155444+0000","last_became_active":"2024-01-15T18:09:10.151462+0000","last_became_peered":"2024-01-15T18:09:10.151462+0000","last_unstale":"2024-01-15T18:09:11.155444+0000","last_undegraded":"2024-01-15T18:09:11.155444+0000","last_fullsized":"2024-01-15T18:09:11.155444+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:07:18.547932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"57'2","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.174259+0000","last_change":"2024-01-15T18:09:09.141213+0000","last_active":"2024-01-15T18:09:10.174259+0000","last_peered":"2024-01-15T18:09:10.174259+0000","last_clean":"2024-01-15T18:09:10.174259+0000","last_became_active":"2024-01-15T18:09:09.139957+0000","last_became_peered":"2024-01-15T18:09:09.139957+0000","last_unstale":"2024-01-15T18:09:10.174259+0000","last_undegraded":"2024-01-15T18:09:10.174259+0000","last_fullsized":"2024-01-15T18:09:10.174259+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:42:47.132801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.127537+0000","last_change":"2024-01-15T18:09:09.127537+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.127537+0000","last_undegraded":"2024-01-15T18:09:09.127537+0000","last_fullsized":"2024-01-15T18:09:09.127537+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.133917+0000","last_change":"2024-01-15T18:09:09.136084+0000","last_active":"2024-01-15T18:09:10.133917+0000","last_peered":"2024-01-15T18:09:10.133917+0000","last_clean":"2024-01-15T18:09:10.133917+0000","last_became_active":"2024-01-15T18:09:09.135760+0000","last_became_peered":"2024-01-15T18:09:09.135760+0000","last_unstale":"2024-01-15T18:09:10.133917+0000","last_undegraded":"2024-01-15T18:09:10.133917+0000","last_fullsized":"2024-01-15T18:09:10.133917+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:37.003640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.128633+0000","last_change":"2024-01-15T18:09:09.128633+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.128633+0000","last_undegraded":"2024-01-15T18:09:09.128633+0000","last_fullsized":"2024-01-15T18:09:09.128633+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.137490+0000","last_change":"2024-01-15T18:09:09.136145+0000","last_active":"2024-01-15T18:09:10.137490+0000","last_peered":"2024-01-15T18:09:10.137490+0000","last_clean":"2024-01-15T18:09:10.137490+0000","last_became_active":"2024-01-15T18:09:09.135807+0000","last_became_peered":"2024-01-15T18:09:09.135807+0000","last_unstale":"2024-01-15T18:09:10.137490+0000","last_undegraded":"2024-01-15T18:09:10.137490+0000","last_fullsized":"2024-01-15T18:09:10.137490+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:35:26.455284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155791+0000","last_change":"2024-01-15T18:09:10.157661+0000","last_active":"2024-01-15T18:09:11.155791+0000","last_peered":"2024-01-15T18:09:11.155791+0000","last_clean":"2024-01-15T18:09:11.155791+0000","last_became_active":"2024-01-15T18:09:10.157399+0000","last_became_peered":"2024-01-15T18:09:10.157399+0000","last_unstale":"2024-01-15T18:09:11.155791+0000","last_undegraded":"2024-01-15T18:09:11.155791+0000","last_fullsized":"2024-01-15T18:09:11.155791+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:41:56.291009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.177996+0000","last_change":"2024-01-15T18:09:09.145612+0000","last_active":"2024-01-15T18:09:10.177996+0000","last_peered":"2024-01-15T18:09:10.177996+0000","last_clean":"2024-01-15T18:09:10.177996+0000","last_became_active":"2024-01-15T18:09:09.145279+0000","last_became_peered":"2024-01-15T18:09:09.145279+0000","last_unstale":"2024-01-15T18:09:10.177996+0000","last_undegraded":"2024-01-15T18:09:10.177996+0000","last_fullsized":"2024-01-15T18:09:10.177996+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:53.930232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.156102+0000","last_change":"2024-01-15T18:09:10.153156+0000","last_active":"2024-01-15T18:09:11.156102+0000","last_peered":"2024-01-15T18:09:11.156102+0000","last_clean":"2024-01-15T18:09:11.156102+0000","last_became_active":"2024-01-15T18:09:10.152783+0000","last_became_peered":"2024-01-15T18:09:10.152783+0000","last_unstale":"2024-01-15T18:09:11.156102+0000","last_undegraded":"2024-01-15T18:09:11.156102+0000","last_fullsized":"2024-01-15T18:09:11.156102+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:36:30.872154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.155546+0000","last_change":"2024-01-15T18:09:09.152741+0000","last_active":"2024-01-15T18:09:09.155546+0000","last_peered":"2024-01-15T18:09:09.155546+0000","last_clean":"2024-01-15T18:09:09.155546+0000","last_became_active":"2024-01-15T18:09:09.152470+0000","last_became_peered":"2024-01-15T18:09:09.152470+0000","last_unstale":"2024-01-15T18:09:09.155546+0000","last_undegraded":"2024-01-15T18:09:09.155546+0000","last_fullsized":"2024-01-15T18:09:09.155546+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:14:31.566501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.153880+0000","last_change":"2024-01-15T18:09:10.143039+0000","last_active":"2024-01-15T18:09:10.153880+0000","last_peered":"2024-01-15T18:09:10.153880+0000","last_clean":"2024-01-15T18:09:10.153880+0000","last_became_active":"2024-01-15T18:09:10.142814+0000","last_became_peered":"2024-01-15T18:09:10.142814+0000","last_unstale":"2024-01-15T18:09:10.153880+0000","last_undegraded":"2024-01-15T18:09:10.153880+0000","last_fullsized":"2024-01-15T18:09:10.153880+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:54:31.215934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.143973+0000","last_change":"2024-01-15T18:09:09.141870+0000","last_active":"2024-01-15T18:09:10.143973+0000","last_peered":"2024-01-15T18:09:10.143973+0000","last_clean":"2024-01-15T18:09:10.143973+0000","last_became_active":"2024-01-15T18:09:09.141523+0000","last_became_peered":"2024-01-15T18:09:09.141523+0000","last_unstale":"2024-01-15T18:09:10.143973+0000","last_undegraded":"2024-01-15T18:09:10.143973+0000","last_fullsized":"2024-01-15T18:09:10.143973+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:15:13.572542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.132306+0000","last_change":"2024-01-15T18:09:09.132306+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.132306+0000","last_undegraded":"2024-01-15T18:09:09.132306+0000","last_fullsized":"2024-01-15T18:09:09.132306+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.137795+0000","last_change":"2024-01-15T18:09:09.136593+0000","last_active":"2024-01-15T18:09:10.137795+0000","last_peered":"2024-01-15T18:09:10.137795+0000","last_clean":"2024-01-15T18:09:10.137795+0000","last_became_active":"2024-01-15T18:09:09.136246+0000","last_became_peered":"2024-01-15T18:09:09.136246+0000","last_unstale":"2024-01-15T18:09:10.137795+0000","last_undegraded":"2024-01-15T18:09:10.137795+0000","last_fullsized":"2024-01-15T18:09:10.137795+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:10:09.158484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.155341+0000","last_change":"2024-01-15T18:09:10.145733+0000","last_active":"2024-01-15T18:09:10.155341+0000","last_peered":"2024-01-15T18:09:10.155341+0000","last_clean":"2024-01-15T18:09:10.155341+0000","last_became_active":"2024-01-15T18:09:10.145479+0000","last_became_peered":"2024-01-15T18:09:10.145479+0000","last_unstale":"2024-01-15T18:09:10.155341+0000","last_undegraded":"2024-01-15T18:09:10.155341+0000","last_fullsized":"2024-01-15T18:09:10.155341+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:53:41.639339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.133917+0000","last_change":"2024-01-15T18:09:09.142681+0000","last_active":"2024-01-15T18:09:10.133917+0000","last_peered":"2024-01-15T18:09:10.133917+0000","last_clean":"2024-01-15T18:09:10.133917+0000","last_became_active":"2024-01-15T18:09:09.142357+0000","last_became_peered":"2024-01-15T18:09:09.142357+0000","last_unstale":"2024-01-15T18:09:10.133917+0000","last_undegraded":"2024-01-15T18:09:10.133917+0000","last_fullsized":"2024-01-15T18:09:10.133917+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:41:15.076278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173119+0000","last_change":"2024-01-15T18:09:10.162125+0000","last_active":"2024-01-15T18:09:10.173119+0000","last_peered":"2024-01-15T18:09:10.173119+0000","last_clean":"2024-01-15T18:09:10.173119+0000","last_became_active":"2024-01-15T18:09:10.161778+0000","last_became_peered":"2024-01-15T18:09:10.161778+0000","last_unstale":"2024-01-15T18:09:10.173119+0000","last_undegraded":"2024-01-15T18:09:10.173119+0000","last_fullsized":"2024-01-15T18:09:10.173119+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:07:31.195306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.139632+0000","last_change":"2024-01-15T18:09:09.135449+0000","last_active":"2024-01-15T18:09:10.139632+0000","last_peered":"2024-01-15T18:09:10.139632+0000","last_clean":"2024-01-15T18:09:10.139632+0000","last_became_active":"2024-01-15T18:09:09.135010+0000","last_became_peered":"2024-01-15T18:09:09.135010+0000","last_unstale":"2024-01-15T18:09:10.139632+0000","last_undegraded":"2024-01-15T18:09:10.139632+0000","last_fullsized":"2024-01-15T18:09:10.139632+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:54:08.460596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.170500+0000","last_change":"2024-01-15T18:09:10.158452+0000","last_active":"2024-01-15T18:09:10.170500+0000","last_peered":"2024-01-15T18:09:10.170500+0000","last_clean":"2024-01-15T18:09:10.170500+0000","last_became_active":"2024-01-15T18:09:10.158267+0000","last_became_peered":"2024-01-15T18:09:10.158267+0000","last_unstale":"2024-01-15T18:09:10.170500+0000","last_undegraded":"2024-01-15T18:09:10.170500+0000","last_fullsized":"2024-01-15T18:09:10.170500+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:26:45.212614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.133918+0000","last_change":"2024-01-15T18:09:09.146561+0000","last_active":"2024-01-15T18:09:10.133918+0000","last_peered":"2024-01-15T18:09:10.133918+0000","last_clean":"2024-01-15T18:09:10.133918+0000","last_became_active":"2024-01-15T18:09:09.146263+0000","last_became_peered":"2024-01-15T18:09:09.146263+0000","last_unstale":"2024-01-15T18:09:10.133918+0000","last_undegraded":"2024-01-15T18:09:10.133918+0000","last_fullsized":"2024-01-15T18:09:10.133918+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:00:48.078792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173947+0000","last_change":"2024-01-15T18:09:10.157427+0000","last_active":"2024-01-15T18:09:10.173947+0000","last_peered":"2024-01-15T18:09:10.173947+0000","last_clean":"2024-01-15T18:09:10.173947+0000","last_became_active":"2024-01-15T18:09:10.157146+0000","last_became_peered":"2024-01-15T18:09:10.157146+0000","last_unstale":"2024-01-15T18:09:10.173947+0000","last_undegraded":"2024-01-15T18:09:10.173947+0000","last_fullsized":"2024-01-15T18:09:10.173947+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:07:31.195306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.142185+0000","last_change":"2024-01-15T18:09:09.136141+0000","last_active":"2024-01-15T18:09:10.142185+0000","last_peered":"2024-01-15T18:09:10.142185+0000","last_clean":"2024-01-15T18:09:10.142185+0000","last_became_active":"2024-01-15T18:09:09.135853+0000","last_became_peered":"2024-01-15T18:09:09.135853+0000","last_unstale":"2024-01-15T18:09:10.142185+0000","last_undegraded":"2024-01-15T18:09:10.142185+0000","last_fullsized":"2024-01-15T18:09:10.142185+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:25:50.421687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173953+0000","last_change":"2024-01-15T18:09:10.161307+0000","last_active":"2024-01-15T18:09:10.173953+0000","last_peered":"2024-01-15T18:09:10.173953+0000","last_clean":"2024-01-15T18:09:10.173953+0000","last_became_active":"2024-01-15T18:09:10.160945+0000","last_became_peered":"2024-01-15T18:09:10.160945+0000","last_unstale":"2024-01-15T18:09:10.173953+0000","last_undegraded":"2024-01-15T18:09:10.173953+0000","last_fullsized":"2024-01-15T18:09:10.173953+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:26:05.237285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.144458+0000","last_change":"2024-01-15T18:09:09.142696+0000","last_active":"2024-01-15T18:09:10.144458+0000","last_peered":"2024-01-15T18:09:10.144458+0000","last_clean":"2024-01-15T18:09:10.144458+0000","last_became_active":"2024-01-15T18:09:09.142448+0000","last_became_peered":"2024-01-15T18:09:09.142448+0000","last_unstale":"2024-01-15T18:09:10.144458+0000","last_undegraded":"2024-01-15T18:09:10.144458+0000","last_fullsized":"2024-01-15T18:09:10.144458+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:40:45.529038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":194,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.122277+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:09:09.122277+0000","last_peered":"2024-01-15T18:09:09.122277+0000","last_clean":"2024-01-15T18:09:09.122277+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:09:09.122277+0000","last_undegraded":"2024-01-15T18:09:09.122277+0000","last_fullsized":"2024-01-15T18:09:09.122277+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":1828,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":2626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509496,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36572,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335268,"statfs":{"total":96636764160,"available":96599314432,"internally_reserved":0,"allocated":1257472,"data_stored":864881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":0.82899999999999996,"15min":0.69199999999999995},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":10.284000000000001,"5min":10.284000000000001,"15min":10.284000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":1.8440000000000001,"5min":0.89800000000000002,"15min":0.752},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":9.9610000000000003,"5min":9.9610000000000003,"15min":9.9610000000000003},"last":0.51800000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":0.76600000000000001,"15min":0.69999999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":7.0279999999999996,"5min":7.0279999999999996,"15min":7.0279999999999996},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.7230000000000001,"5min":0.89700000000000002,"15min":0.76500000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":9.9250000000000007,"5min":9.9250000000000007,"15min":9.9250000000000007},"last":0.73599999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":0.872,"15min":0.72199999999999998},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":10.119999999999999,"5min":10.119999999999999,"15min":10.119999999999999},"last":0.495},{"interface":"front","average":{"1min":1.242,"5min":0.82199999999999995,"15min":0.748},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":7.0789999999999997,"5min":7.0789999999999997,"15min":7.0789999999999997},"last":0.67400000000000004}]},{"osd":3,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.625,"5min":0.92500000000000004,"15min":0.751},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":9.3539999999999992,"5min":9.3539999999999992,"15min":9.3539999999999992},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.7070000000000001,"5min":0.93400000000000005,"15min":0.81999999999999995},"min":{"1min":0.45700000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":10.34,"5min":10.34,"15min":10.34},"last":0.58899999999999997}]},{"osd":4,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":2.4079999999999999,"5min":0.96499999999999997,"15min":0.73499999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":17.021000000000001,"5min":17.021000000000001,"15min":17.021000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.734,"5min":0.92700000000000005,"15min":0.752},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":14.871,"5min":14.871,"15min":14.871},"last":0.57299999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.837,"5min":0.91000000000000003,"15min":0.77000000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":19.959,"5min":19.959,"15min":19.959},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.9610000000000001,"5min":0.94399999999999995,"15min":0.80100000000000005},"min":{"1min":0.47999999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":11.053000000000001,"5min":11.053000000000001,"15min":11.053000000000001},"last":0.623}]},{"osd":6,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":3.0070000000000001,"5min":1.2170000000000001,"15min":0.94299999999999995},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":22.478999999999999,"5min":22.478999999999999,"15min":22.478999999999999},"last":0.78600000000000003},{"interface":"front","average":{"1min":2.6269999999999998,"5min":1.1739999999999999,"15min":0.88100000000000001},"min":{"1min":0.52800000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":21.509,"5min":21.509,"15min":21.509},"last":0.77200000000000002}]}]},{"osd":6,"up_from":15,"seq":64424509496,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35988,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335852,"statfs":{"total":96636764160,"available":96599912448,"internally_reserved":0,"allocated":663552,"data_stored":274513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":0.86299999999999999,"15min":0.77800000000000002},"min":{"1min":0.45200000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":9.3339999999999996,"5min":9.3339999999999996,"15min":9.3339999999999996},"last":0.623},{"interface":"front","average":{"1min":1.4850000000000001,"5min":0.83599999999999997,"15min":0.74399999999999999},"min":{"1min":0.47099999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":6.7759999999999998,"5min":6.7759999999999998,"15min":6.7759999999999998},"last":0.79800000000000004}]},{"osd":1,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":3.8119999999999998,"5min":1.329,"15min":0.93300000000000005},"min":{"1min":0.51400000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":53.451999999999998,"5min":53.451999999999998,"15min":53.451999999999998},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.3009999999999999,"5min":0.83599999999999997,"15min":0.78400000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":5.9900000000000002,"5min":5.9900000000000002,"15min":5.9900000000000002},"last":0.68400000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":0.84499999999999997,"15min":0.85199999999999998},"min":{"1min":0.46700000000000003,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":6.0380000000000003,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.2310000000000001,"5min":0.82799999999999996,"15min":0.80700000000000005},"min":{"1min":0.57099999999999995,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":4.6059999999999999,"5min":4.6059999999999999,"15min":4.6059999999999999},"last":0.66600000000000004}]},{"osd":3,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":0.84899999999999998,"15min":0.82599999999999996},"min":{"1min":0.51900000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":3.9689999999999999,"5min":3.9689999999999999,"15min":3.9689999999999999},"last":0.755},{"interface":"front","average":{"1min":3.6709999999999998,"5min":1.355,"15min":1.0109999999999999},"min":{"1min":0.45200000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":0.88}]},{"osd":4,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.6479999999999999,"5min":0.85499999999999998,"15min":0.751},"min":{"1min":0.48099999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":8.6259999999999994,"5min":8.6259999999999994,"15min":8.6259999999999994},"last":0.45900000000000002},{"interface":"front","average":{"1min":1.2809999999999999,"5min":0.79600000000000004,"15min":0.752},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":6.9589999999999996,"5min":6.9589999999999996,"15min":6.9589999999999996},"last":0.50900000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":4.2560000000000002,"5min":1.371,"15min":0.90000000000000002},"min":{"1min":0.44500000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":53.229999999999997,"5min":53.229999999999997,"15min":53.229999999999997},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.847,"5min":0.96799999999999997,"15min":0.84599999999999997},"min":{"1min":0.53900000000000003,"5min":0.435,"15min":0.435},"max":{"1min":10.029999999999999,"5min":10.029999999999999,"15min":10.029999999999999},"last":0.54500000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":3.1640000000000001,"5min":1.2589999999999999,"15min":0.97699999999999998},"min":{"1min":0.46200000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":45.070999999999998,"5min":45.070999999999998,"15min":45.070999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":4.5990000000000002,"5min":1.5369999999999999,"15min":1.054},"min":{"1min":0.48499999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":53.338000000000001,"5min":53.338000000000001,"15min":53.338000000000001},"last":0.70399999999999996}]}]},{"osd":4,"up_from":15,"seq":64424509496,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35992,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335848,"statfs":{"total":96636764160,"available":96599908352,"internally_reserved":0,"allocated":663552,"data_stored":274513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":0.91700000000000004,"15min":0.72599999999999998},"min":{"1min":0.434,"5min":0.38,"15min":0.38},"max":{"1min":23.318000000000001,"5min":23.318000000000001,"15min":23.318000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.7649999999999999,"5min":0.84999999999999998,"15min":0.71399999999999997},"min":{"1min":0.44800000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":10.210000000000001,"5min":10.210000000000001,"15min":10.210000000000001},"last":0.57299999999999995}]},{"osd":1,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.71099999999999997,"15min":0.67200000000000004},"min":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":5.016,"5min":5.016,"15min":5.016},"last":0.85099999999999998},{"interface":"front","average":{"1min":2.0990000000000002,"5min":0.93400000000000005,"15min":0.73999999999999999},"min":{"1min":0.45100000000000001,"5min":0.377,"15min":0.377},"max":{"1min":22.408999999999999,"5min":22.408999999999999,"15min":22.408999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":0.71799999999999997,"15min":0.65900000000000003},"min":{"1min":0.46700000000000003,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":5.1159999999999997,"5min":5.1159999999999997,"15min":5.1159999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":2.1419999999999999,"5min":1.036,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":22.338999999999999,"5min":22.338999999999999,"15min":22.338999999999999},"last":0.54900000000000004}]},{"osd":3,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.8380000000000001,"5min":0.91600000000000004,"15min":0.76600000000000001},"min":{"1min":0.42699999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":10.271000000000001,"5min":10.271000000000001,"15min":10.271000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.27,"5min":0.84099999999999997,"15min":0.76500000000000001},"min":{"1min":0.59899999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":8.0719999999999992,"5min":8.0719999999999992,"15min":8.0719999999999992},"last":0.749}]},{"osd":5,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":2.7879999999999998,"5min":1.081,"15min":0.80400000000000005},"min":{"1min":0.39000000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":31.617999999999999,"5min":31.617999999999999,"15min":31.617999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":2.4369999999999998,"5min":1.0009999999999999,"15min":0.76700000000000002},"min":{"1min":0.439,"5min":0.377,"15min":0.377},"max":{"1min":17.145,"5min":17.145,"15min":17.145},"last":0.86399999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":0.83499999999999996,"15min":0.73099999999999998},"min":{"1min":0.53900000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.577,"5min":5.577,"15min":5.577},"last":0.80300000000000005},{"interface":"front","average":{"1min":4.9329999999999998,"5min":1.5720000000000001,"15min":1.014},"min":{"1min":0.56299999999999994,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":28.872,"5min":28.872,"15min":28.872},"last":0.78200000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":3.726,"5min":1.3149999999999999,"15min":0.91700000000000004},"min":{"1min":0.48199999999999998,"5min":0.436,"15min":0.436},"max":{"1min":39.256,"5min":39.256,"15min":39.256},"last":0.82599999999999996},{"interface":"front","average":{"1min":2.069,"5min":1.0389999999999999,"15min":0.86699999999999999},"min":{"1min":0.51800000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":26.573,"5min":26.573,"15min":26.573},"last":0.88700000000000001}]}]},{"osd":3,"up_from":15,"seq":64424509495,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35992,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335848,"statfs":{"total":96636764160,"available":96599908352,"internally_reserved":0,"allocated":663552,"data_stored":274513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.45900000000000002,"15min":0.433},"min":{"1min":0.28899999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.641,"5min":2.641,"15min":2.641},"last":0.434},{"interface":"front","average":{"1min":1.4099999999999999,"5min":0.68300000000000005,"15min":0.54100000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":17.756,"5min":17.756,"15min":17.756},"last":0.33000000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.54100000000000004,"15min":0.52000000000000002},"min":{"1min":0.29799999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.0620000000000003,"5min":4.0620000000000003,"15min":4.0620000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.54500000000000004,"15min":0.53900000000000003},"min":{"1min":0.371,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.38}]},{"osd":2,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":0.69199999999999995,"15min":0.60499999999999998},"min":{"1min":0.42499999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":10.849,"5min":10.849,"15min":10.849},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.626,"15min":0.58499999999999996},"min":{"1min":0.40500000000000003,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.617,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":0.5}]},{"osd":4,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":1.756,"5min":1.004,"15min":0.873},"min":{"1min":0.56499999999999995,"5min":0.5,"15min":0.5},"max":{"1min":17.853999999999999,"5min":17.853999999999999,"15min":17.853999999999999},"last":0.63},{"interface":"front","average":{"1min":1.022,"5min":0.85099999999999998,"15min":0.80500000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":4.1420000000000003,"5min":5.0369999999999999,"15min":5.0369999999999999},"last":0.71399999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":0.82699999999999996,"15min":0.76900000000000002},"min":{"1min":0.51100000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":4.7320000000000002,"5min":4.7320000000000002,"15min":4.7320000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":2.145,"5min":1.004,"15min":0.81499999999999995},"min":{"1min":0.54900000000000004,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":17.937999999999999,"5min":17.937999999999999,"15min":17.937999999999999},"last":0.57399999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":2.125,"5min":1.046,"15min":0.873},"min":{"1min":0.52300000000000002,"5min":0.499,"15min":0.499},"max":{"1min":23.100999999999999,"5min":23.100999999999999,"15min":23.100999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":1.113,"5min":0.84599999999999997,"15min":0.79000000000000004},"min":{"1min":0.61899999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":5.7030000000000003,"5min":5.7030000000000003,"15min":5.7030000000000003},"last":0.59399999999999997}]},{"osd":7,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":0.95999999999999996,"15min":0.83799999999999997},"min":{"1min":0.54900000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":17.896999999999998,"5min":17.896999999999998,"15min":17.896999999999998},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.1020000000000001,"5min":0.93999999999999995,"15min":0.86199999999999999},"min":{"1min":0.60599999999999998,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":4.1920000000000002,"5min":10.705,"15min":10.705},"last":0.68300000000000005}]}]},{"osd":2,"up_from":15,"seq":64424509495,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36008,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335832,"statfs":{"total":96636764160,"available":96599891968,"internally_reserved":0,"allocated":684032,"data_stored":281372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.55500000000000005,"15min":0.49399999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":5.1849999999999996,"5min":5.1849999999999996,"15min":5.1849999999999996},"last":0.29899999999999999},{"interface":"front","average":{"1min":1.0640000000000001,"5min":0.57799999999999996,"15min":0.504},"min":{"1min":0.307,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":7.516,"5min":7.516,"15min":7.516},"last":0.36399999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.60799999999999998,"15min":0.56599999999999995},"min":{"1min":0.35499999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.2509999999999999,"5min":4.8079999999999998,"15min":4.8079999999999998},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.61599999999999999,"15min":0.50800000000000001},"min":{"1min":0.32500000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":8.9190000000000005,"5min":8.9190000000000005,"15min":8.9190000000000005},"last":0.32500000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":0.69799999999999995,"15min":0.61299999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":12.185,"5min":12.185,"15min":12.185},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.68600000000000005,"15min":0.59599999999999997},"min":{"1min":0.42899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":7.4580000000000002,"5min":7.4580000000000002,"15min":7.4580000000000002},"last":0.435}]},{"osd":4,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":0.91400000000000003,"15min":0.80100000000000005},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":8.6609999999999996,"5min":8.6609999999999996,"15min":8.6609999999999996},"last":0.68799999999999994},{"interface":"front","average":{"1min":2.3519999999999999,"5min":1.0820000000000001,"15min":0.875},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":19.146999999999998,"5min":19.146999999999998,"15min":19.146999999999998},"last":0.79000000000000004}]},{"osd":5,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":2.4929999999999999,"5min":1.1100000000000001,"15min":0.90200000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":31.710000000000001,"5min":31.710000000000001,"15min":31.710000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":2.7970000000000002,"5min":1.2150000000000001,"15min":0.93300000000000005},"min":{"1min":0.52900000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":22.539000000000001,"5min":22.539000000000001,"15min":22.539000000000001},"last":0.747}]},{"osd":6,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":0.92000000000000004,"15min":0.82899999999999996},"min":{"1min":0.51100000000000001,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":9.4540000000000006,"5min":9.4540000000000006,"15min":9.4540000000000006},"last":0.85699999999999998},{"interface":"front","average":{"1min":2.29,"5min":1.085,"15min":0.89500000000000002},"min":{"1min":0.54400000000000004,"5min":0.503,"15min":0.503},"max":{"1min":18.614999999999998,"5min":18.614999999999998,"15min":18.614999999999998},"last":0.873}]},{"osd":7,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":0.94599999999999995,"15min":0.85199999999999998},"min":{"1min":0.51200000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":11.432,"5min":11.432,"15min":11.432},"last":0.76000000000000001},{"interface":"front","average":{"1min":2.577,"5min":1.204,"15min":0.95499999999999996},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":17.013000000000002,"5min":17.013000000000002,"15min":17.013000000000002},"last":0.94799999999999995}]}]},{"osd":1,"up_from":15,"seq":64424509497,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35992,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335848,"statfs":{"total":96636764160,"available":96599908352,"internally_reserved":0,"allocated":663552,"data_stored":274513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":0.78400000000000003,"15min":0.60199999999999998},"min":{"1min":0.27300000000000002,"5min":0.254,"15min":0.254},"max":{"1min":9.8620000000000001,"5min":19.577000000000002,"15min":19.577000000000002},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.052,"15min":0.67600000000000005},"min":{"1min":0.28799999999999998,"5min":0.251,"15min":0.251},"max":{"1min":5.6639999999999997,"5min":48.277999999999999,"15min":48.277999999999999},"last":0.57599999999999996}]},{"osd":2,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":0.997,"15min":0.69799999999999995},"min":{"1min":0.312,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":6.4569999999999999,"5min":40.698,"15min":40.698},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.97,"5min":1.351,"15min":0.94899999999999995},"min":{"1min":0.375,"5min":0.317,"15min":0.317},"max":{"1min":18.827000000000002,"5min":48.316000000000003,"15min":48.316000000000003},"last":0.39500000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.1899999999999999,"15min":0.83199999999999996},"min":{"1min":0.373,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.915,"5min":48.234999999999999,"15min":48.234999999999999},"last":0.432},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.65700000000000003,"15min":0.61699999999999999},"min":{"1min":0.34699999999999998,"5min":0.311,"15min":0.311},"max":{"1min":4.1420000000000003,"5min":6.093,"15min":6.093},"last":0.55300000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":0.86899999999999999,"15min":0.751},"min":{"1min":0.48299999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":4.1950000000000003,"5min":5.9169999999999998,"15min":5.9169999999999998},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.175,"5min":1.0129999999999999,"15min":0.85799999999999998},"min":{"1min":0.46999999999999997,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":6.3789999999999996,"5min":19.829000000000001,"15min":19.829000000000001},"last":0.86099999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":1.339,"15min":0.96099999999999997},"min":{"1min":0.58399999999999996,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":22.594999999999999,"5min":22.594999999999999,"15min":22.594999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":2.0470000000000002,"5min":1.466,"15min":1.004},"min":{"1min":0.55900000000000005,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":11.768000000000001,"5min":48.198,"15min":48.198},"last":0.91500000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":3.0390000000000001,"5min":1.2649999999999999,"15min":0.92500000000000004},"min":{"1min":0.59199999999999997,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":35.112000000000002,"5min":35.112000000000002,"15min":35.112000000000002},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.516,"5min":1.123,"15min":0.92500000000000004},"min":{"1min":0.58399999999999996,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":8.3670000000000009,"5min":19.780999999999999,"15min":19.780999999999999},"last":0.73999999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.3959999999999999,"15min":0.98799999999999999},"min":{"1min":0.56999999999999995,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":9.2870000000000008,"5min":48.148000000000003,"15min":48.148000000000003},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.325,"5min":0.95499999999999996,"15min":0.85499999999999998},"min":{"1min":0.56799999999999995,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":4.2919999999999998,"5min":6.0549999999999997,"15min":6.0549999999999997},"last":0.89900000000000002}]}]},{"osd":5,"up_from":14,"seq":60129542201,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36020,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335820,"statfs":{"total":96636764160,"available":96599879680,"internally_reserved":0,"allocated":696320,"data_stored":288296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.85999999999999999,"15min":0.77200000000000002},"min":{"1min":0.41199999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.2640000000000002,"5min":21.385000000000002,"15min":21.385000000000002},"last":0.60599999999999998},{"interface":"front","average":{"1min":1.173,"5min":0.91900000000000004,"15min":0.83699999999999997},"min":{"1min":0.51100000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":8.6590000000000007,"5min":8.6590000000000007,"15min":8.6590000000000007},"last":0.54200000000000004}]},{"osd":1,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.70799999999999996,"15min":0.68700000000000006},"min":{"1min":0.42999999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":3.1110000000000002,"5min":3.1110000000000002,"15min":3.1110000000000002},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.89700000000000002,"15min":0.755},"min":{"1min":0.47299999999999998,"5min":0.315,"15min":0.315},"max":{"1min":4.2039999999999997,"5min":21.5,"15min":21.5},"last":0.78700000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":0.99199999999999999,"15min":0.80400000000000005},"min":{"1min":0.55700000000000005,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":8.7750000000000004,"5min":8.7750000000000004,"15min":8.7750000000000004},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.86599999999999999,"15min":0.81000000000000005},"min":{"1min":0.50800000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":7.9160000000000004,"5min":7.9160000000000004,"15min":7.9160000000000004},"last":0.77500000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.0009999999999999,"15min":0.873},"min":{"1min":0.53400000000000003,"5min":0.44,"15min":0.44},"max":{"1min":5.0650000000000004,"5min":21.457000000000001,"15min":21.457000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.694,"5min":1.083,"15min":0.93000000000000005},"min":{"1min":0.58699999999999997,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":8.7300000000000004,"5min":8.7300000000000004,"15min":8.7300000000000004},"last":0.80400000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":0.88100000000000001,"15min":0.78400000000000003},"min":{"1min":0.40699999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":8.8179999999999996,"5min":9.6679999999999993,"15min":9.6679999999999993},"last":0.73599999999999999},{"interface":"front","average":{"1min":2.0089999999999999,"5min":0.94899999999999995,"15min":0.81299999999999994},"min":{"1min":0.439,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":26.146999999999998,"5min":26.146999999999998,"15min":26.146999999999998},"last":0.69399999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":0.95499999999999996,"15min":0.81000000000000005},"min":{"1min":0.46300000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":6.6699999999999999,"5min":21.535,"15min":21.535},"last":0.70599999999999996},{"interface":"front","average":{"1min":2.649,"5min":1.2250000000000001,"15min":0.94599999999999995},"min":{"1min":0.46999999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":30.488,"5min":30.488,"15min":30.488},"last":0.84699999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":0.874,"15min":0.82599999999999996},"min":{"1min":0.51600000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":2.758,"5min":9.2240000000000002,"15min":9.2240000000000002},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.96699999999999997,"15min":0.877},"min":{"1min":0.44600000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.3520000000000003,"5min":21.574000000000002,"15min":21.574000000000002},"last":0.81899999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542200,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36556,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335284,"statfs":{"total":96636764160,"available":96599330816,"internally_reserved":0,"allocated":1245184,"data_stored":858586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.52600000000000002,"15min":0.48599999999999999},"min":{"1min":0.26100000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.2409999999999997,"5min":4.2409999999999997,"15min":4.2409999999999997},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55600000000000005,"15min":0.55300000000000005},"min":{"1min":0.35499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":2.0219999999999998,"5min":2.0219999999999998,"15min":2.0219999999999998},"last":0.53800000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.57599999999999996,"15min":0.57399999999999995},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.2330000000000001,"5min":2.2330000000000001,"15min":2.2330000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.53500000000000003,"15min":0.497},"min":{"1min":0.30099999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.6789999999999998,"5min":2.6789999999999998,"15min":2.6789999999999998},"last":0.56899999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.63900000000000001,"15min":0.59699999999999998},"min":{"1min":0.38900000000000001,"5min":0.308,"15min":0.308},"max":{"1min":4.274,"5min":4.274,"15min":4.274},"last":0.503},{"interface":"front","average":{"1min":1.0309999999999999,"5min":0.68200000000000005,"15min":0.63400000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":7.2309999999999999,"5min":7.2309999999999999,"15min":7.2309999999999999},"last":0.61799999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":2.339,"5min":1.0680000000000001,"15min":0.878},"min":{"1min":0.46600000000000003,"5min":0.38,"15min":0.38},"max":{"1min":22.814,"5min":22.814,"15min":22.814},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.516,"5min":0.92700000000000005,"15min":0.86499999999999999},"min":{"1min":0.54500000000000004,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":4.4800000000000004,"5min":4.4800000000000004,"15min":4.4800000000000004},"last":0.90800000000000003}]},{"osd":5,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":1.8600000000000001,"5min":0.97499999999999998,"15min":0.85799999999999998},"min":{"1min":0.495,"5min":0.441,"15min":0.441},"max":{"1min":12.428000000000001,"5min":12.428000000000001,"15min":12.428000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.915,"5min":1.018,"15min":0.91000000000000003},"min":{"1min":0.63800000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":13.064,"5min":13.064,"15min":13.064},"last":0.82799999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":2.8029999999999999,"5min":1.1699999999999999,"15min":0.92400000000000004},"min":{"1min":0.50800000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":31.059000000000001,"5min":31.059000000000001,"15min":31.059000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":3.6059999999999999,"5min":1.357,"15min":1.022},"min":{"1min":0.52000000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":40.731000000000002,"5min":40.731000000000002,"15min":40.731000000000002},"last":0.98399999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":1.657,"5min":0.96499999999999997,"15min":0.878},"min":{"1min":0.53800000000000003,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":14.032,"5min":14.032,"15min":14.032},"last":0.93200000000000005},{"interface":"front","average":{"1min":2.9569999999999999,"5min":1.254,"15min":0.995},"min":{"1min":0.58099999999999996,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":23.524000000000001,"5min":23.524000000000001,"15min":23.524000000000001},"last":0.878}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:09:15.042 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3a is not active+clean 2024-01-15T18:09:15.042 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:09:09.127717+0000', 'last_change': '2024-01-15T18:09:09.127717+0000', 'last_active': '2024-01-15T18:09:09.109985+0000', 'last_peered': '2024-01-15T18:09:09.109985+0000', 'last_clean': '2024-01-15T18:09:09.109985+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:09:09.127717+0000', 'last_undegraded': '2024-01-15T18:09:09.127717+0000', 'last_fullsized': '2024-01-15T18:09:09.127717+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'last_clean_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:09:15.042 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.30 is not active+clean 2024-01-15T18:09:15.042 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:09:09.127271+0000', 'last_change': '2024-01-15T18:09:09.127271+0000', 'last_active': '2024-01-15T18:09:09.109985+0000', 'last_peered': '2024-01-15T18:09:09.109985+0000', 'last_clean': '2024-01-15T18:09:09.109985+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:09:09.127271+0000', 'last_undegraded': '2024-01-15T18:09:09.127271+0000', 'last_fullsized': '2024-01-15T18:09:09.127271+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'last_clean_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:09:15.042 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.25 is not active+clean 2024-01-15T18:09:15.042 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:09:09.133285+0000', 'last_change': '2024-01-15T18:09:09.133285+0000', 'last_active': '2024-01-15T18:09:09.109985+0000', 'last_peered': '2024-01-15T18:09:09.109985+0000', 'last_clean': '2024-01-15T18:09:09.109985+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:09:09.133285+0000', 'last_undegraded': '2024-01-15T18:09:09.133285+0000', 'last_fullsized': '2024-01-15T18:09:09.133285+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'last_clean_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:09:15.043 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.20 is not active+clean 2024-01-15T18:09:15.043 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:09:09.126719+0000', 'last_change': '2024-01-15T18:09:09.126719+0000', 'last_active': '2024-01-15T18:09:09.109985+0000', 'last_peered': '2024-01-15T18:09:09.109985+0000', 'last_clean': '2024-01-15T18:09:09.109985+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:09:09.126719+0000', 'last_undegraded': '2024-01-15T18:09:09.126719+0000', 'last_fullsized': '2024-01-15T18:09:09.126719+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'last_clean_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:09:15.043 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1e is not active+clean 2024-01-15T18:09:15.043 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:09:09.138025+0000', 'last_change': '2024-01-15T18:09:09.138025+0000', 'last_active': '2024-01-15T18:09:09.109985+0000', 'last_peered': '2024-01-15T18:09:09.109985+0000', 'last_clean': '2024-01-15T18:09:09.109985+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:09:09.138025+0000', 'last_undegraded': '2024-01-15T18:09:09.138025+0000', 'last_fullsized': '2024-01-15T18:09:09.138025+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'last_clean_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:09:15.043 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1c is not active+clean 2024-01-15T18:09:15.043 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:09:09.130696+0000', 'last_change': '2024-01-15T18:09:09.130696+0000', 'last_active': '2024-01-15T18:09:09.109985+0000', 'last_peered': '2024-01-15T18:09:09.109985+0000', 'last_clean': '2024-01-15T18:09:09.109985+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:09:09.130696+0000', 'last_undegraded': '2024-01-15T18:09:09.130696+0000', 'last_fullsized': '2024-01-15T18:09:09.130696+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'last_clean_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:09:15.043 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1d is not active+clean 2024-01-15T18:09:15.043 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:09:09.132775+0000', 'last_change': '2024-01-15T18:09:09.132775+0000', 'last_active': '2024-01-15T18:09:09.109985+0000', 'last_peered': '2024-01-15T18:09:09.109985+0000', 'last_clean': '2024-01-15T18:09:09.109985+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:09:09.132775+0000', 'last_undegraded': '2024-01-15T18:09:09.132775+0000', 'last_fullsized': '2024-01-15T18:09:09.132775+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'last_clean_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:09:15.043 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.b is not active+clean 2024-01-15T18:09:15.043 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:09:09.127537+0000', 'last_change': '2024-01-15T18:09:09.127537+0000', 'last_active': '2024-01-15T18:09:09.109985+0000', 'last_peered': '2024-01-15T18:09:09.109985+0000', 'last_clean': '2024-01-15T18:09:09.109985+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:09:09.127537+0000', 'last_undegraded': '2024-01-15T18:09:09.127537+0000', 'last_fullsized': '2024-01-15T18:09:09.127537+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'last_clean_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:09:15.044 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.9 is not active+clean 2024-01-15T18:09:15.044 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:09:09.128633+0000', 'last_change': '2024-01-15T18:09:09.128633+0000', 'last_active': '2024-01-15T18:09:09.109985+0000', 'last_peered': '2024-01-15T18:09:09.109985+0000', 'last_clean': '2024-01-15T18:09:09.109985+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:09:09.128633+0000', 'last_undegraded': '2024-01-15T18:09:09.128633+0000', 'last_fullsized': '2024-01-15T18:09:09.128633+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'last_clean_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:09:15.044 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.5 is not active+clean 2024-01-15T18:09:15.044 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 56, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:09:09.132306+0000', 'last_change': '2024-01-15T18:09:09.132306+0000', 'last_active': '2024-01-15T18:09:09.109985+0000', 'last_peered': '2024-01-15T18:09:09.109985+0000', 'last_clean': '2024-01-15T18:09:09.109985+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:09:09.132306+0000', 'last_undegraded': '2024-01-15T18:09:09.132306+0000', 'last_fullsized': '2024-01-15T18:09:09.132306+0000', 'mapping_epoch': 56, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 56, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'last_clean_scrub_stamp': '2024-01-15T18:09:09.109985+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:09:15.044 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-15T18:09:15.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.214+0000 7ffab9600640 1 Processor -- start 2024-01-15T18:09:15.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.214+0000 7ffab9600640 1 -- start start 2024-01-15T18:09:15.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.214+0000 7ffab9600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffab415e7d0 0x7ffab40cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:15.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.214+0000 7ffab9600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab40b38b0 con 0x7ffab40b20d0 2024-01-15T18:09:15.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.214+0000 7ffab9600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab40cd550 con 0x7ffab40cedc0 2024-01-15T18:09:15.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.214+0000 7ffab9600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffab40cd270 con 0x7ffab415e7d0 2024-01-15T18:09:15.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.214+0000 7ffab3400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ffab40b20d0 0x7ffab40cc1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49574/0 (socket says 172.21.15.27:49574) 2024-01-15T18:09:15.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.214+0000 7ffab3400640 1 -- 172.21.15.27:0/1348770772 learned_addr learned my addr 172.21.15.27:0/1348770772 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:15.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.214+0000 7ffab2a00640 1 --2- 172.21.15.27:0/1348770772 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffab415e7d0 0x7ffab40cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:15.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.214+0000 7ffab1600640 1 -- 172.21.15.27:0/1348770772 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991227861 0 0) 0x7ffab40b38b0 con 0x7ffab40b20d0 2024-01-15T18:09:15.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.214+0000 7ffab1600640 1 -- 172.21.15.27:0/1348770772 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa94003530 con 0x7ffab40b20d0 2024-01-15T18:09:15.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.214+0000 7ffab1600640 1 -- 172.21.15.27:0/1348770772 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2473383010 0 0) 0x7ffab40cd550 con 0x7ffab40cedc0 2024-01-15T18:09:15.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.214+0000 7ffab1600640 1 -- 172.21.15.27:0/1348770772 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffab40b38b0 con 0x7ffab40cedc0 2024-01-15T18:09:15.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.214+0000 7ffab2a00640 1 -- 172.21.15.27:0/1348770772 >> v1:172.21.15.93:6789/0 conn(0x7ffab40cedc0 legacy=0x7ffab40cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:15.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.214+0000 7ffab2a00640 1 -- 172.21.15.27:0/1348770772 >> v1:172.21.15.27:6789/0 conn(0x7ffab40b20d0 legacy=0x7ffab40cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:15.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.214+0000 7ffab2a00640 1 -- 172.21.15.27:0/1348770772 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab40b35d0 con 0x7ffab415e7d0 2024-01-15T18:09:15.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.218+0000 7ffab2a00640 1 --2- 172.21.15.27:0/1348770772 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffab415e7d0 0x7ffab40cf2a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ffaa40097f0 tx=0x7ffaa401c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43f88824d383545e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:15.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.218+0000 7ffab1600640 1 -- 172.21.15.27:0/1348770772 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa40ac410 con 0x7ffab415e7d0 2024-01-15T18:09:15.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.218+0000 7ffab1600640 1 -- 172.21.15.27:0/1348770772 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa40ac570 con 0x7ffab415e7d0 2024-01-15T18:09:15.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.218+0000 7ffab1600640 1 -- 172.21.15.27:0/1348770772 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa40ac900 con 0x7ffab415e7d0 2024-01-15T18:09:15.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.218+0000 7ffab9600640 1 -- 172.21.15.27:0/1348770772 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffab415e7d0 msgr2=0x7ffab40cf2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:15.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.218+0000 7ffab9600640 1 --2- 172.21.15.27:0/1348770772 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffab415e7d0 0x7ffab40cf2a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ffaa40097f0 tx=0x7ffaa401c4c0 comp rx=0 tx=0).stop 2024-01-15T18:09:15.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.218+0000 7ffab9600640 1 -- 172.21.15.27:0/1348770772 shutdown_connections 2024-01-15T18:09:15.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.218+0000 7ffab9600640 1 --2- 172.21.15.27:0/1348770772 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffab415e7d0 0x7ffab40cf2a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:15.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.218+0000 7ffab9600640 1 -- 172.21.15.27:0/1348770772 >> 172.21.15.27:0/1348770772 conn(0x7ffab40b2a10 msgr2=0x7ffab40b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:15.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.218+0000 7ffab9600640 1 -- 172.21.15.27:0/1348770772 shutdown_connections 2024-01-15T18:09:15.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.218+0000 7ffab9600640 1 -- 172.21.15.27:0/1348770772 wait complete. 2024-01-15T18:09:15.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.222+0000 7ffab9600640 1 Processor -- start 2024-01-15T18:09:15.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.222+0000 7ffab9600640 1 -- start start 2024-01-15T18:09:15.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.222+0000 7ffab9600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab40b20d0 0x7ffab406ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:15.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.222+0000 7ffab9600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ffab40cedc0 0x7ffab41528b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:15.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.222+0000 7ffab9600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffab415e7d0 0x7ffab40b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:15.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.222+0000 7ffab9600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab406d630 con 0x7ffab40b20d0 2024-01-15T18:09:15.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.222+0000 7ffab9600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab40c5170 con 0x7ffab40cedc0 2024-01-15T18:09:15.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.222+0000 7ffab9600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffab406d350 con 0x7ffab415e7d0 2024-01-15T18:09:15.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.222+0000 7ffab2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab40b20d0 0x7ffab406ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:15.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.222+0000 7ffab2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab40b20d0 0x7ffab406ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34392/0 (socket says 172.21.15.27:34392) 2024-01-15T18:09:15.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.222+0000 7ffab2a00640 1 -- 172.21.15.27:0/977247758 learned_addr learned my addr 172.21.15.27:0/977247758 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:15.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.222+0000 7ffab2000640 1 --2- 172.21.15.27:0/977247758 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ffab40cedc0 0x7ffab41528b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:15.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.222+0000 7ffab3400640 1 --2- 172.21.15.27:0/977247758 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffab415e7d0 0x7ffab40b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:15.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.222+0000 7ffab2a00640 1 -- 172.21.15.27:0/977247758 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffab415e7d0 msgr2=0x7ffab40b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:15.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.222+0000 7ffab2a00640 1 --2- 172.21.15.27:0/977247758 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffab415e7d0 0x7ffab40b47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:15.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.222+0000 7ffab2a00640 1 -- 172.21.15.27:0/977247758 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ffab40cedc0 msgr2=0x7ffab41528b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:15.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.222+0000 7ffab2a00640 1 --2- 172.21.15.27:0/977247758 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ffab40cedc0 0x7ffab41528b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:15.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.222+0000 7ffab2a00640 1 -- 172.21.15.27:0/977247758 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa4009420 con 0x7ffab40b20d0 2024-01-15T18:09:15.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.222+0000 7ffab3400640 1 --2- 172.21.15.27:0/977247758 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffab415e7d0 0x7ffab40b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:15.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.222+0000 7ffab2a00640 1 --2- 172.21.15.27:0/977247758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab40b20d0 0x7ffab406ca70 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4009740 tx=0x7ffaa4019260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:15.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.222+0000 7ffaaaa00640 1 -- 172.21.15.27:0/977247758 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa4006e20 con 0x7ffab40b20d0 2024-01-15T18:09:15.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.222+0000 7ffaaaa00640 1 -- 172.21.15.27:0/977247758 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa40ace00 con 0x7ffab40b20d0 2024-01-15T18:09:15.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.222+0000 7ffaaaa00640 1 -- 172.21.15.27:0/977247758 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa40b5850 con 0x7ffab40b20d0 2024-01-15T18:09:15.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.222+0000 7ffab9600640 1 -- 172.21.15.27:0/977247758 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffab40c4b20 con 0x7ffab40b20d0 2024-01-15T18:09:15.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.222+0000 7ffab9600640 1 -- 172.21.15.27:0/977247758 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffab406cfb0 con 0x7ffab40b20d0 2024-01-15T18:09:15.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.226+0000 7ffab9600640 1 -- 172.21.15.27:0/977247758 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffab404e850 con 0x7ffab40b20d0 2024-01-15T18:09:15.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.226+0000 7ffaaaa00640 1 -- 172.21.15.27:0/977247758 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7ffaa40b3040 con 0x7ffab40b20d0 2024-01-15T18:09:15.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.226+0000 7ffaaaa00640 1 --2- 172.21.15.27:0/977247758 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ffa8c066e90 0x7ffa8c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:15.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.226+0000 7ffaaaa00640 1 -- 172.21.15.27:0/977247758 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7ffaa411efb0 con 0x7ffab40b20d0 2024-01-15T18:09:15.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.226+0000 7ffab2000640 1 --2- 172.21.15.27:0/977247758 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ffa8c066e90 0x7ffa8c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:15.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.226+0000 7ffab2000640 1 --2- 172.21.15.27:0/977247758 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ffa8c066e90 0x7ffa8c069350 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ffab41538e0 tx=0x7ffa9c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:15.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.230+0000 7ffaaaa00640 1 -- 172.21.15.27:0/977247758 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ffaa40eb8a0 con 0x7ffab40b20d0 2024-01-15T18:09:15.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.446+0000 7ffab9600640 1 -- 172.21.15.27:0/977247758 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7ffab415d220 con 0x7ffab40b20d0 2024-01-15T18:09:15.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.446+0000 7ffaaaa00640 1 -- 172.21.15.27:0/977247758 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1099 (secure 0 0 0) 0x7ffaa40eeec0 con 0x7ffab40b20d0 2024-01-15T18:09:15.453 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:09:15.453 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":282,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":58,"num_osds":8,"num_up_osds":8,"osd_up_since":1705341884,"num_in_osds":8,"osd_in_since":1705341875,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":87},{"state_name":"creating+peering","count":10}],"num_pgs":97,"num_pools":3,"num_objects":19,"data_bytes":592196,"bytes_used":296058880,"bytes_avail":772798054400,"bytes_total":773094113280,"inactive_pgs_ratio":0.10309278219938278,"write_bytes_sec":2218,"read_op_per_sec":0,"write_op_per_sec":5},"fsmap":{"epoch":37,"id":4,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":4,"rank":0,"name":"c","status":"up:active","gid":6480}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-01-15T18:08:28.191482+0000","services":{}},"progress_events":{}} 2024-01-15T18:09:15.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.450+0000 7ffab9600640 1 -- 172.21.15.27:0/977247758 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ffa8c066e90 msgr2=0x7ffa8c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:15.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.450+0000 7ffab9600640 1 --2- 172.21.15.27:0/977247758 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ffa8c066e90 0x7ffa8c069350 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ffab41538e0 tx=0x7ffa9c009040 comp rx=0 tx=0).stop 2024-01-15T18:09:15.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.450+0000 7ffab9600640 1 -- 172.21.15.27:0/977247758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab40b20d0 msgr2=0x7ffab406ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:15.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.450+0000 7ffab9600640 1 --2- 172.21.15.27:0/977247758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab40b20d0 0x7ffab406ca70 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4009740 tx=0x7ffaa4019260 comp rx=0 tx=0).stop 2024-01-15T18:09:15.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.450+0000 7ffab9600640 1 -- 172.21.15.27:0/977247758 shutdown_connections 2024-01-15T18:09:15.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.450+0000 7ffab9600640 1 --2- 172.21.15.27:0/977247758 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ffa8c066e90 0x7ffa8c069350 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:15.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.450+0000 7ffab9600640 1 --2- 172.21.15.27:0/977247758 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ffab415e7d0 0x7ffab40b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:15.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.450+0000 7ffab9600640 1 --2- 172.21.15.27:0/977247758 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ffab40cedc0 0x7ffab41528b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:15.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.450+0000 7ffab9600640 1 --2- 172.21.15.27:0/977247758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffab40b20d0 0x7ffab406ca70 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:15.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.450+0000 7ffab9600640 1 -- 172.21.15.27:0/977247758 >> 172.21.15.27:0/977247758 conn(0x7ffab40b2a10 msgr2=0x7ffab415da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:15.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.450+0000 7ffab9600640 1 -- 172.21.15.27:0/977247758 shutdown_connections 2024-01-15T18:09:15.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.450+0000 7ffab9600640 1 -- 172.21.15.27:0/977247758 wait complete. 2024-01-15T18:09:15.476 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-01-15T18:09:15.476 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:09:15.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.610+0000 7fad29000640 1 Processor -- start 2024-01-15T18:09:15.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.610+0000 7fad29000640 1 -- start start 2024-01-15T18:09:15.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad29000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fad24159bf0 0x7fad24159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:15.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad29000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fad2406d430 0x7fad240cd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:15.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad29000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad24159a50 con 0x7fad24071e40 2024-01-15T18:09:15.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad29000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad240cce00 con 0x7fad24159bf0 2024-01-15T18:09:15.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad29000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2415a510 con 0x7fad2406d430 2024-01-15T18:09:15.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad21400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fad24071e40 0x7fad240c9be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49590/0 (socket says 172.21.15.27:49590) 2024-01-15T18:09:15.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad21400640 1 -- 172.21.15.27:0/321261465 learned_addr learned my addr 172.21.15.27:0/321261465 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:15.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad22800640 1 --2- 172.21.15.27:0/321261465 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fad2406d430 0x7fad240cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:15.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad21e00640 1 --2- 172.21.15.27:0/321261465 >> v2:172.21.15.93:3300/0 conn(0x7fad24159bf0 0x7fad24159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:15.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad20a00640 1 -- 172.21.15.27:0/321261465 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 976535221 0 0) 0x7fad0c001120 con 0x7fad24071e40 2024-01-15T18:09:15.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad20a00640 1 -- 172.21.15.27:0/321261465 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad04003530 con 0x7fad24071e40 2024-01-15T18:09:15.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad20a00640 1 -- 172.21.15.27:0/321261465 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1048448850 0 0) 0x7fad04003530 con 0x7fad24071e40 2024-01-15T18:09:15.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad20a00640 1 -- 172.21.15.27:0/321261465 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fad2406d430 msgr2=0x7fad240cd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:15.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad20a00640 1 --2- 172.21.15.27:0/321261465 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fad2406d430 0x7fad240cd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:15.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad20a00640 1 -- 172.21.15.27:0/321261465 >> v2:172.21.15.93:3300/0 conn(0x7fad24159bf0 msgr2=0x7fad24159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:15.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad20a00640 1 --2- 172.21.15.27:0/321261465 >> v2:172.21.15.93:3300/0 conn(0x7fad24159bf0 0x7fad24159fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:15.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad20a00640 1 -- 172.21.15.27:0/321261465 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2406d980 con 0x7fad24071e40 2024-01-15T18:09:15.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad20a00640 1 -- 172.21.15.27:0/321261465 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fad0c001810 con 0x7fad24071e40 2024-01-15T18:09:15.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad20a00640 1 -- 172.21.15.27:0/321261465 >> v1:172.21.15.27:6789/0 conn(0x7fad24071e40 legacy=0x7fad240c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:15.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad20a00640 1 --2- 172.21.15.27:0/321261465 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fad040039f0 0x7fad04003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:15.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad20a00640 1 --2- 172.21.15.27:0/321261465 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fad2406d430 0x7fad040047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:15.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad20a00640 1 --2- 172.21.15.27:0/321261465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad04004d10 0x7fad04005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:15.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad20a00640 1 -- 172.21.15.27:0/321261465 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad240cce00 con 0x7fad04004d10 2024-01-15T18:09:15.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad20a00640 1 -- 172.21.15.27:0/321261465 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2415a510 con 0x7fad040039f0 2024-01-15T18:09:15.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad20a00640 1 -- 172.21.15.27:0/321261465 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad0400ba40 con 0x7fad2406d430 2024-01-15T18:09:15.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad21e00640 1 --2- 172.21.15.27:0/321261465 >> v2:172.21.15.93:3300/0 conn(0x7fad24159bf0 0x7fad24159fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:09:15.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad22800640 1 --2- 172.21.15.27:0/321261465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad04004d10 0x7fad04005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:15.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad21400640 1 --2- 172.21.15.27:0/321261465 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fad2406d430 0x7fad040047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:15.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad21e00640 1 --2- 172.21.15.27:0/321261465 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fad040039f0 0x7fad04003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:15.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad22800640 1 -- 172.21.15.27:0/321261465 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fad2406d430 msgr2=0x7fad040047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:15.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad22800640 1 --2- 172.21.15.27:0/321261465 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fad2406d430 0x7fad040047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:15.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad22800640 1 -- 172.21.15.27:0/321261465 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fad040039f0 msgr2=0x7fad04003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:15.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad22800640 1 --2- 172.21.15.27:0/321261465 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fad040039f0 0x7fad04003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:15.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.614+0000 7fad22800640 1 -- 172.21.15.27:0/321261465 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad04003530 con 0x7fad04004d10 2024-01-15T18:09:15.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.618+0000 7fad22800640 1 --2- 172.21.15.27:0/321261465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad04004d10 0x7fad04005190 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fad1c009fc0 tx=0x7fad1c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e501b5f212add63 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:15.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.618+0000 7fad20a00640 1 -- 172.21.15.27:0/321261465 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c0a5c70 con 0x7fad04004d10 2024-01-15T18:09:15.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.618+0000 7fad20a00640 1 -- 172.21.15.27:0/321261465 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1c00f040 con 0x7fad04004d10 2024-01-15T18:09:15.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.618+0000 7fad20a00640 1 -- 172.21.15.27:0/321261465 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c0ad440 con 0x7fad04004d10 2024-01-15T18:09:15.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.618+0000 7fad29000640 1 -- 172.21.15.27:0/321261465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad04004d10 msgr2=0x7fad04005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:15.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.618+0000 7fad29000640 1 --2- 172.21.15.27:0/321261465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad04004d10 0x7fad04005190 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fad1c009fc0 tx=0x7fad1c01cc60 comp rx=0 tx=0).stop 2024-01-15T18:09:15.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.618+0000 7fad22800640 1 -- 172.21.15.27:0/321261465 reap_dead start 2024-01-15T18:09:15.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.618+0000 7fad29000640 1 -- 172.21.15.27:0/321261465 shutdown_connections 2024-01-15T18:09:15.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.618+0000 7fad29000640 1 -- 172.21.15.27:0/321261465 >> 172.21.15.27:0/321261465 conn(0x7fad240cec60 msgr2=0x7fad240cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:15.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.618+0000 7fad29000640 1 -- 172.21.15.27:0/321261465 shutdown_connections 2024-01-15T18:09:15.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.618+0000 7fad29000640 1 -- 172.21.15.27:0/321261465 wait complete. 2024-01-15T18:09:15.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.622+0000 7fad29000640 1 Processor -- start 2024-01-15T18:09:15.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.622+0000 7fad29000640 1 -- start start 2024-01-15T18:09:15.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.622+0000 7fad29000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad04004d10 0x7fad240cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:15.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.622+0000 7fad29000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fad241f6a80 0x7fad241f6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:15.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.622+0000 7fad29000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fad241490e0 0x7fad24149560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:15.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.622+0000 7fad29000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad24166600 con 0x7fad04004d10 2024-01-15T18:09:15.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.622+0000 7fad29000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad24149aa0 con 0x7fad241f6a80 2024-01-15T18:09:15.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.622+0000 7fad29000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad241f7540 con 0x7fad241490e0 2024-01-15T18:09:15.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.622+0000 7fad21e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad04004d10 0x7fad240cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:15.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.622+0000 7fad21e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad04004d10 0x7fad240cd3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34410/0 (socket says 172.21.15.27:34410) 2024-01-15T18:09:15.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.622+0000 7fad21e00640 1 -- 172.21.15.27:0/459819569 learned_addr learned my addr 172.21.15.27:0/459819569 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:15.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.622+0000 7fad22800640 1 --2- 172.21.15.27:0/459819569 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fad241490e0 0x7fad24149560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:15.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.622+0000 7fad21400640 1 --2- 172.21.15.27:0/459819569 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fad241f6a80 0x7fad241f6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:15.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.622+0000 7fad21e00640 1 -- 172.21.15.27:0/459819569 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fad241490e0 msgr2=0x7fad24149560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:15.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.622+0000 7fad21e00640 1 --2- 172.21.15.27:0/459819569 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fad241490e0 0x7fad24149560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:15.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.622+0000 7fad21e00640 1 -- 172.21.15.27:0/459819569 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fad241f6a80 msgr2=0x7fad241f6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:15.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.622+0000 7fad21e00640 1 --2- 172.21.15.27:0/459819569 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fad241f6a80 0x7fad241f6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:15.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.622+0000 7fad21e00640 1 -- 172.21.15.27:0/459819569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad241f7820 con 0x7fad04004d10 2024-01-15T18:09:15.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.622+0000 7fad21e00640 1 --2- 172.21.15.27:0/459819569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad04004d10 0x7fad240cd3b0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fad14011cd0 tx=0x7fad1400f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:15.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.622+0000 7fad1a000640 1 -- 172.21.15.27:0/459819569 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1400fd20 con 0x7fad04004d10 2024-01-15T18:09:15.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.622+0000 7fad1a000640 1 -- 172.21.15.27:0/459819569 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1400fe80 con 0x7fad04004d10 2024-01-15T18:09:15.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.622+0000 7fad29000640 1 -- 172.21.15.27:0/459819569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad24166850 con 0x7fad04004d10 2024-01-15T18:09:15.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.622+0000 7fad1a000640 1 -- 172.21.15.27:0/459819569 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad14004270 con 0x7fad04004d10 2024-01-15T18:09:15.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.622+0000 7fad29000640 1 -- 172.21.15.27:0/459819569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad2414c410 con 0x7fad04004d10 2024-01-15T18:09:15.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.626+0000 7fad29000640 1 -- 172.21.15.27:0/459819569 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad2404e850 con 0x7fad04004d10 2024-01-15T18:09:15.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.626+0000 7fad1a000640 1 -- 172.21.15.27:0/459819569 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7fad140043d0 con 0x7fad04004d10 2024-01-15T18:09:15.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.626+0000 7fad1a000640 1 --2- 172.21.15.27:0/459819569 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7facfc066e90 0x7facfc069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:15.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.626+0000 7fad1a000640 1 -- 172.21.15.27:0/459819569 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fad14085650 con 0x7fad04004d10 2024-01-15T18:09:15.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.626+0000 7fad21400640 1 --2- 172.21.15.27:0/459819569 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7facfc066e90 0x7facfc069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:15.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.626+0000 7fad21400640 1 --2- 172.21.15.27:0/459819569 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7facfc066e90 0x7facfc069350 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fad241525e0 tx=0x7fad0c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:15.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.630+0000 7fad1a000640 1 -- 172.21.15.27:0/459819569 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fad14052080 con 0x7fad04004d10 2024-01-15T18:09:15.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.786+0000 7fad29000640 1 -- 172.21.15.27:0/459819569 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fad2406e220 con 0x7facfc066e90 2024-01-15T18:09:15.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.794+0000 7fad1a000640 1 -- 172.21.15.27:0/459819569 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287472 (secure 0 0 0) 0x7fad0c0076b0 con 0x7facfc066e90 2024-01-15T18:09:15.802 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:09:15.804 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:09:15.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.798+0000 7fad29000640 1 -- 172.21.15.27:0/459819569 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7facfc066e90 msgr2=0x7facfc069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:15.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.798+0000 7fad29000640 1 --2- 172.21.15.27:0/459819569 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7facfc066e90 0x7facfc069350 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fad241525e0 tx=0x7fad0c009040 comp rx=0 tx=0).stop 2024-01-15T18:09:15.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.798+0000 7fad29000640 1 -- 172.21.15.27:0/459819569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad04004d10 msgr2=0x7fad240cd3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:15.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.798+0000 7fad29000640 1 --2- 172.21.15.27:0/459819569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad04004d10 0x7fad240cd3b0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fad14011cd0 tx=0x7fad1400f610 comp rx=0 tx=0).stop 2024-01-15T18:09:15.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.798+0000 7fad29000640 1 -- 172.21.15.27:0/459819569 shutdown_connections 2024-01-15T18:09:15.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.798+0000 7fad29000640 1 --2- 172.21.15.27:0/459819569 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7facfc066e90 0x7facfc069350 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:15.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.798+0000 7fad29000640 1 --2- 172.21.15.27:0/459819569 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fad241490e0 0x7fad24149560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:15.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.798+0000 7fad29000640 1 --2- 172.21.15.27:0/459819569 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fad241f6a80 0x7fad241f6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:15.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.798+0000 7fad29000640 1 --2- 172.21.15.27:0/459819569 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad04004d10 0x7fad240cd3b0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:15.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.798+0000 7fad29000640 1 -- 172.21.15.27:0/459819569 >> 172.21.15.27:0/459819569 conn(0x7fad240cec60 msgr2=0x7fad2406c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:15.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.798+0000 7fad29000640 1 -- 172.21.15.27:0/459819569 shutdown_connections 2024-01-15T18:09:15.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:15.798+0000 7fad29000640 1 -- 172.21.15.27:0/459819569 wait complete. 2024-01-15T18:09:15.824 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-01-15T18:09:14.206380+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592196,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":193,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":289120,"kb_used_data":6384,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754685600,"statfs":{"total":773094113280,"available":772798054400,"internally_reserved":0,"allocated":6537216,"data_stored":3391187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":289394224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1828,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001226"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.174696+0000","last_change":"2024-01-15T18:09:10.164426+0000","last_active":"2024-01-15T18:09:10.174696+0000","last_peered":"2024-01-15T18:09:10.174696+0000","last_clean":"2024-01-15T18:09:10.174696+0000","last_became_active":"2024-01-15T18:09:10.164120+0000","last_became_peered":"2024-01-15T18:09:10.164120+0000","last_unstale":"2024-01-15T18:09:10.174696+0000","last_undegraded":"2024-01-15T18:09:10.174696+0000","last_fullsized":"2024-01-15T18:09:10.174696+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:13:02.463974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.156544+0000","last_change":"2024-01-15T18:09:10.149693+0000","last_active":"2024-01-15T18:09:10.156544+0000","last_peered":"2024-01-15T18:09:10.156544+0000","last_clean":"2024-01-15T18:09:10.156544+0000","last_became_active":"2024-01-15T18:09:10.149380+0000","last_became_peered":"2024-01-15T18:09:10.149380+0000","last_unstale":"2024-01-15T18:09:10.156544+0000","last_undegraded":"2024-01-15T18:09:10.156544+0000","last_fullsized":"2024-01-15T18:09:10.156544+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:14:10.432447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.176732+0000","last_change":"2024-01-15T18:09:10.172281+0000","last_active":"2024-01-15T18:09:10.176732+0000","last_peered":"2024-01-15T18:09:10.176732+0000","last_clean":"2024-01-15T18:09:10.176732+0000","last_became_active":"2024-01-15T18:09:10.171520+0000","last_became_peered":"2024-01-15T18:09:10.171520+0000","last_unstale":"2024-01-15T18:09:10.176732+0000","last_undegraded":"2024-01-15T18:09:10.176732+0000","last_fullsized":"2024-01-15T18:09:10.176732+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:05.433977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155356+0000","last_change":"2024-01-15T18:09:10.154186+0000","last_active":"2024-01-15T18:09:11.155356+0000","last_peered":"2024-01-15T18:09:11.155356+0000","last_clean":"2024-01-15T18:09:11.155356+0000","last_became_active":"2024-01-15T18:09:10.153961+0000","last_became_peered":"2024-01-15T18:09:10.153961+0000","last_unstale":"2024-01-15T18:09:11.155356+0000","last_undegraded":"2024-01-15T18:09:11.155356+0000","last_fullsized":"2024-01-15T18:09:11.155356+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:44:11.424905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163531+0000","last_change":"2024-01-15T18:09:10.149056+0000","last_active":"2024-01-15T18:09:10.163531+0000","last_peered":"2024-01-15T18:09:10.163531+0000","last_clean":"2024-01-15T18:09:10.163531+0000","last_became_active":"2024-01-15T18:09:10.148756+0000","last_became_peered":"2024-01-15T18:09:10.148756+0000","last_unstale":"2024-01-15T18:09:10.163531+0000","last_undegraded":"2024-01-15T18:09:10.163531+0000","last_fullsized":"2024-01-15T18:09:10.163531+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:20.006996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.127717+0000","last_change":"2024-01-15T18:09:09.127717+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.127717+0000","last_undegraded":"2024-01-15T18:09:09.127717+0000","last_fullsized":"2024-01-15T18:09:09.127717+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.149739+0000","last_change":"2024-01-15T18:09:10.147574+0000","last_active":"2024-01-15T18:09:10.149739+0000","last_peered":"2024-01-15T18:09:10.149739+0000","last_clean":"2024-01-15T18:09:10.149739+0000","last_became_active":"2024-01-15T18:09:10.147288+0000","last_became_peered":"2024-01-15T18:09:10.147288+0000","last_unstale":"2024-01-15T18:09:10.149739+0000","last_undegraded":"2024-01-15T18:09:10.149739+0000","last_fullsized":"2024-01-15T18:09:10.149739+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:29:52.274001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.160773+0000","last_change":"2024-01-15T18:09:10.149706+0000","last_active":"2024-01-15T18:09:10.160773+0000","last_peered":"2024-01-15T18:09:10.160773+0000","last_clean":"2024-01-15T18:09:10.160773+0000","last_became_active":"2024-01-15T18:09:10.149353+0000","last_became_peered":"2024-01-15T18:09:10.149353+0000","last_unstale":"2024-01-15T18:09:10.160773+0000","last_undegraded":"2024-01-15T18:09:10.160773+0000","last_fullsized":"2024-01-15T18:09:10.160773+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:22:35.521714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.166664+0000","last_change":"2024-01-15T18:09:10.162835+0000","last_active":"2024-01-15T18:09:10.166664+0000","last_peered":"2024-01-15T18:09:10.166664+0000","last_clean":"2024-01-15T18:09:10.166664+0000","last_became_active":"2024-01-15T18:09:10.162459+0000","last_became_peered":"2024-01-15T18:09:10.162459+0000","last_unstale":"2024-01-15T18:09:10.166664+0000","last_undegraded":"2024-01-15T18:09:10.166664+0000","last_fullsized":"2024-01-15T18:09:10.166664+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:26.052280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.164826+0000","last_change":"2024-01-15T18:09:10.151168+0000","last_active":"2024-01-15T18:09:10.164826+0000","last_peered":"2024-01-15T18:09:10.164826+0000","last_clean":"2024-01-15T18:09:10.164826+0000","last_became_active":"2024-01-15T18:09:10.150812+0000","last_became_peered":"2024-01-15T18:09:10.150812+0000","last_unstale":"2024-01-15T18:09:10.164826+0000","last_undegraded":"2024-01-15T18:09:10.164826+0000","last_fullsized":"2024-01-15T18:09:10.164826+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:09:59.563201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.157867+0000","last_change":"2024-01-15T18:09:10.148209+0000","last_active":"2024-01-15T18:09:10.157867+0000","last_peered":"2024-01-15T18:09:10.157867+0000","last_clean":"2024-01-15T18:09:10.157867+0000","last_became_active":"2024-01-15T18:09:10.147897+0000","last_became_peered":"2024-01-15T18:09:10.147897+0000","last_unstale":"2024-01-15T18:09:10.157867+0000","last_undegraded":"2024-01-15T18:09:10.157867+0000","last_fullsized":"2024-01-15T18:09:10.157867+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:53.437883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155548+0000","last_change":"2024-01-15T18:09:10.153400+0000","last_active":"2024-01-15T18:09:11.155548+0000","last_peered":"2024-01-15T18:09:11.155548+0000","last_clean":"2024-01-15T18:09:11.155548+0000","last_became_active":"2024-01-15T18:09:10.152222+0000","last_became_peered":"2024-01-15T18:09:10.152222+0000","last_unstale":"2024-01-15T18:09:11.155548+0000","last_undegraded":"2024-01-15T18:09:11.155548+0000","last_fullsized":"2024-01-15T18:09:11.155548+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:03.126844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.165807+0000","last_change":"2024-01-15T18:09:10.160925+0000","last_active":"2024-01-15T18:09:10.165807+0000","last_peered":"2024-01-15T18:09:10.165807+0000","last_clean":"2024-01-15T18:09:10.165807+0000","last_became_active":"2024-01-15T18:09:10.160620+0000","last_became_peered":"2024-01-15T18:09:10.160620+0000","last_unstale":"2024-01-15T18:09:10.165807+0000","last_undegraded":"2024-01-15T18:09:10.165807+0000","last_fullsized":"2024-01-15T18:09:10.165807+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:40:15.227589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.166438+0000","last_change":"2024-01-15T18:09:10.154922+0000","last_active":"2024-01-15T18:09:10.166438+0000","last_peered":"2024-01-15T18:09:10.166438+0000","last_clean":"2024-01-15T18:09:10.166438+0000","last_became_active":"2024-01-15T18:09:10.154594+0000","last_became_peered":"2024-01-15T18:09:10.154594+0000","last_unstale":"2024-01-15T18:09:10.166438+0000","last_undegraded":"2024-01-15T18:09:10.166438+0000","last_fullsized":"2024-01-15T18:09:10.166438+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:55:44.009610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.148569+0000","last_change":"2024-01-15T18:09:10.143163+0000","last_active":"2024-01-15T18:09:10.148569+0000","last_peered":"2024-01-15T18:09:10.148569+0000","last_clean":"2024-01-15T18:09:10.148569+0000","last_became_active":"2024-01-15T18:09:10.142874+0000","last_became_peered":"2024-01-15T18:09:10.142874+0000","last_unstale":"2024-01-15T18:09:10.148569+0000","last_undegraded":"2024-01-15T18:09:10.148569+0000","last_fullsized":"2024-01-15T18:09:10.148569+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:02:56.938951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.127271+0000","last_change":"2024-01-15T18:09:09.127271+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.127271+0000","last_undegraded":"2024-01-15T18:09:09.127271+0000","last_fullsized":"2024-01-15T18:09:09.127271+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163654+0000","last_change":"2024-01-15T18:09:10.156152+0000","last_active":"2024-01-15T18:09:10.163654+0000","last_peered":"2024-01-15T18:09:10.163654+0000","last_clean":"2024-01-15T18:09:10.163654+0000","last_became_active":"2024-01-15T18:09:10.155849+0000","last_became_peered":"2024-01-15T18:09:10.155849+0000","last_unstale":"2024-01-15T18:09:10.163654+0000","last_undegraded":"2024-01-15T18:09:10.163654+0000","last_fullsized":"2024-01-15T18:09:10.163654+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:53.373020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.177069+0000","last_change":"2024-01-15T18:09:10.168066+0000","last_active":"2024-01-15T18:09:10.177069+0000","last_peered":"2024-01-15T18:09:10.177069+0000","last_clean":"2024-01-15T18:09:10.177069+0000","last_became_active":"2024-01-15T18:09:10.167691+0000","last_became_peered":"2024-01-15T18:09:10.167691+0000","last_unstale":"2024-01-15T18:09:10.177069+0000","last_undegraded":"2024-01-15T18:09:10.177069+0000","last_fullsized":"2024-01-15T18:09:10.177069+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:58:26.855862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.174836+0000","last_change":"2024-01-15T18:09:10.163790+0000","last_active":"2024-01-15T18:09:10.174836+0000","last_peered":"2024-01-15T18:09:10.174836+0000","last_clean":"2024-01-15T18:09:10.174836+0000","last_became_active":"2024-01-15T18:09:10.163509+0000","last_became_peered":"2024-01-15T18:09:10.163509+0000","last_unstale":"2024-01-15T18:09:10.174836+0000","last_undegraded":"2024-01-15T18:09:10.174836+0000","last_fullsized":"2024-01-15T18:09:10.174836+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:22:24.247456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173893+0000","last_change":"2024-01-15T18:09:10.161978+0000","last_active":"2024-01-15T18:09:10.173893+0000","last_peered":"2024-01-15T18:09:10.173893+0000","last_clean":"2024-01-15T18:09:10.173893+0000","last_became_active":"2024-01-15T18:09:10.161717+0000","last_became_peered":"2024-01-15T18:09:10.161717+0000","last_unstale":"2024-01-15T18:09:10.173893+0000","last_undegraded":"2024-01-15T18:09:10.173893+0000","last_fullsized":"2024-01-15T18:09:10.173893+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:05.433977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163131+0000","last_change":"2024-01-15T18:09:10.159447+0000","last_active":"2024-01-15T18:09:10.163131+0000","last_peered":"2024-01-15T18:09:10.163131+0000","last_clean":"2024-01-15T18:09:10.163131+0000","last_became_active":"2024-01-15T18:09:10.159055+0000","last_became_peered":"2024-01-15T18:09:10.159055+0000","last_unstale":"2024-01-15T18:09:10.163131+0000","last_undegraded":"2024-01-15T18:09:10.163131+0000","last_fullsized":"2024-01-15T18:09:10.163131+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:54:09.462802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173806+0000","last_change":"2024-01-15T18:09:10.157352+0000","last_active":"2024-01-15T18:09:10.173806+0000","last_peered":"2024-01-15T18:09:10.173806+0000","last_clean":"2024-01-15T18:09:10.173806+0000","last_became_active":"2024-01-15T18:09:10.155991+0000","last_became_peered":"2024-01-15T18:09:10.155991+0000","last_unstale":"2024-01-15T18:09:10.173806+0000","last_undegraded":"2024-01-15T18:09:10.173806+0000","last_fullsized":"2024-01-15T18:09:10.173806+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:25:19.060578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163476+0000","last_change":"2024-01-15T18:09:10.160338+0000","last_active":"2024-01-15T18:09:10.163476+0000","last_peered":"2024-01-15T18:09:10.163476+0000","last_clean":"2024-01-15T18:09:10.163476+0000","last_became_active":"2024-01-15T18:09:10.159325+0000","last_became_peered":"2024-01-15T18:09:10.159325+0000","last_unstale":"2024-01-15T18:09:10.163476+0000","last_undegraded":"2024-01-15T18:09:10.163476+0000","last_fullsized":"2024-01-15T18:09:10.163476+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:46:36.631105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.168684+0000","last_change":"2024-01-15T18:09:10.167327+0000","last_active":"2024-01-15T18:09:10.168684+0000","last_peered":"2024-01-15T18:09:10.168684+0000","last_clean":"2024-01-15T18:09:10.168684+0000","last_became_active":"2024-01-15T18:09:10.167008+0000","last_became_peered":"2024-01-15T18:09:10.167008+0000","last_unstale":"2024-01-15T18:09:10.168684+0000","last_undegraded":"2024-01-15T18:09:10.168684+0000","last_fullsized":"2024-01-15T18:09:10.168684+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:18:13.449428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.174731+0000","last_change":"2024-01-15T18:09:10.165962+0000","last_active":"2024-01-15T18:09:10.174731+0000","last_peered":"2024-01-15T18:09:10.174731+0000","last_clean":"2024-01-15T18:09:10.174731+0000","last_became_active":"2024-01-15T18:09:10.165689+0000","last_became_peered":"2024-01-15T18:09:10.165689+0000","last_unstale":"2024-01-15T18:09:10.174731+0000","last_undegraded":"2024-01-15T18:09:10.174731+0000","last_fullsized":"2024-01-15T18:09:10.174731+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:48:02.569878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155565+0000","last_change":"2024-01-15T18:09:10.147816+0000","last_active":"2024-01-15T18:09:11.155565+0000","last_peered":"2024-01-15T18:09:11.155565+0000","last_clean":"2024-01-15T18:09:11.155565+0000","last_became_active":"2024-01-15T18:09:10.147487+0000","last_became_peered":"2024-01-15T18:09:10.147487+0000","last_unstale":"2024-01-15T18:09:11.155565+0000","last_undegraded":"2024-01-15T18:09:11.155565+0000","last_fullsized":"2024-01-15T18:09:11.155565+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:40:51.237238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.133285+0000","last_change":"2024-01-15T18:09:09.133285+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.133285+0000","last_undegraded":"2024-01-15T18:09:09.133285+0000","last_fullsized":"2024-01-15T18:09:09.133285+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.156169+0000","last_change":"2024-01-15T18:09:10.149120+0000","last_active":"2024-01-15T18:09:10.156169+0000","last_peered":"2024-01-15T18:09:10.156169+0000","last_clean":"2024-01-15T18:09:10.156169+0000","last_became_active":"2024-01-15T18:09:10.148675+0000","last_became_peered":"2024-01-15T18:09:10.148675+0000","last_unstale":"2024-01-15T18:09:10.156169+0000","last_undegraded":"2024-01-15T18:09:10.156169+0000","last_fullsized":"2024-01-15T18:09:10.156169+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:52:23.460592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173871+0000","last_change":"2024-01-15T18:09:10.161188+0000","last_active":"2024-01-15T18:09:10.173871+0000","last_peered":"2024-01-15T18:09:10.173871+0000","last_clean":"2024-01-15T18:09:10.173871+0000","last_became_active":"2024-01-15T18:09:10.151645+0000","last_became_peered":"2024-01-15T18:09:10.151645+0000","last_unstale":"2024-01-15T18:09:10.173871+0000","last_undegraded":"2024-01-15T18:09:10.173871+0000","last_fullsized":"2024-01-15T18:09:10.173871+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:07:18.177097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.176602+0000","last_change":"2024-01-15T18:09:10.167895+0000","last_active":"2024-01-15T18:09:10.176602+0000","last_peered":"2024-01-15T18:09:10.176602+0000","last_clean":"2024-01-15T18:09:10.176602+0000","last_became_active":"2024-01-15T18:09:10.167639+0000","last_became_peered":"2024-01-15T18:09:10.167639+0000","last_unstale":"2024-01-15T18:09:10.176602+0000","last_undegraded":"2024-01-15T18:09:10.176602+0000","last_fullsized":"2024-01-15T18:09:10.176602+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:18:53.201764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.161004+0000","last_change":"2024-01-15T18:09:10.148253+0000","last_active":"2024-01-15T18:09:10.161004+0000","last_peered":"2024-01-15T18:09:10.161004+0000","last_clean":"2024-01-15T18:09:10.161004+0000","last_became_active":"2024-01-15T18:09:10.147941+0000","last_became_peered":"2024-01-15T18:09:10.147941+0000","last_unstale":"2024-01-15T18:09:10.161004+0000","last_undegraded":"2024-01-15T18:09:10.161004+0000","last_fullsized":"2024-01-15T18:09:10.161004+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:04:16.316753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.126719+0000","last_change":"2024-01-15T18:09:09.126719+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.126719+0000","last_undegraded":"2024-01-15T18:09:09.126719+0000","last_fullsized":"2024-01-15T18:09:09.126719+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.138025+0000","last_change":"2024-01-15T18:09:09.138025+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.138025+0000","last_undegraded":"2024-01-15T18:09:09.138025+0000","last_fullsized":"2024-01-15T18:09:09.138025+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.155581+0000","last_change":"2024-01-15T18:09:09.152802+0000","last_active":"2024-01-15T18:09:09.155581+0000","last_peered":"2024-01-15T18:09:09.155581+0000","last_clean":"2024-01-15T18:09:09.155581+0000","last_became_active":"2024-01-15T18:09:09.152520+0000","last_became_peered":"2024-01-15T18:09:09.152520+0000","last_unstale":"2024-01-15T18:09:09.155581+0000","last_undegraded":"2024-01-15T18:09:09.155581+0000","last_fullsized":"2024-01-15T18:09:09.155581+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:04.586121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.170556+0000","last_change":"2024-01-15T18:09:10.161734+0000","last_active":"2024-01-15T18:09:10.170556+0000","last_peered":"2024-01-15T18:09:10.170556+0000","last_clean":"2024-01-15T18:09:10.170556+0000","last_became_active":"2024-01-15T18:09:10.161466+0000","last_became_peered":"2024-01-15T18:09:10.161466+0000","last_unstale":"2024-01-15T18:09:10.170556+0000","last_undegraded":"2024-01-15T18:09:10.170556+0000","last_fullsized":"2024-01-15T18:09:10.170556+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:58:24.569904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"57'1","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.177728+0000","last_change":"2024-01-15T18:09:09.125625+0000","last_active":"2024-01-15T18:09:10.177728+0000","last_peered":"2024-01-15T18:09:10.177728+0000","last_clean":"2024-01-15T18:09:10.177728+0000","last_became_active":"2024-01-15T18:09:09.125373+0000","last_became_peered":"2024-01-15T18:09:09.125373+0000","last_unstale":"2024-01-15T18:09:10.177728+0000","last_undegraded":"2024-01-15T18:09:10.177728+0000","last_fullsized":"2024-01-15T18:09:10.177728+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:14:59.883458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.130696+0000","last_change":"2024-01-15T18:09:09.130696+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.130696+0000","last_undegraded":"2024-01-15T18:09:09.130696+0000","last_fullsized":"2024-01-15T18:09:09.130696+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.155587+0000","last_change":"2024-01-15T18:09:09.153007+0000","last_active":"2024-01-15T18:09:09.155587+0000","last_peered":"2024-01-15T18:09:09.155587+0000","last_clean":"2024-01-15T18:09:09.155587+0000","last_became_active":"2024-01-15T18:09:09.152405+0000","last_became_peered":"2024-01-15T18:09:09.152405+0000","last_unstale":"2024-01-15T18:09:09.155587+0000","last_undegraded":"2024-01-15T18:09:09.155587+0000","last_fullsized":"2024-01-15T18:09:09.155587+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:53:06.843401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.132775+0000","last_change":"2024-01-15T18:09:09.132775+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.132775+0000","last_undegraded":"2024-01-15T18:09:09.132775+0000","last_fullsized":"2024-01-15T18:09:09.132775+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.176876+0000","last_change":"2024-01-15T18:09:09.138209+0000","last_active":"2024-01-15T18:09:10.176876+0000","last_peered":"2024-01-15T18:09:10.176876+0000","last_clean":"2024-01-15T18:09:10.176876+0000","last_became_active":"2024-01-15T18:09:09.137921+0000","last_became_peered":"2024-01-15T18:09:09.137921+0000","last_unstale":"2024-01-15T18:09:10.176876+0000","last_undegraded":"2024-01-15T18:09:10.176876+0000","last_fullsized":"2024-01-15T18:09:10.176876+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:54:56.016726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163688+0000","last_change":"2024-01-15T18:09:10.156211+0000","last_active":"2024-01-15T18:09:10.163688+0000","last_peered":"2024-01-15T18:09:10.163688+0000","last_clean":"2024-01-15T18:09:10.163688+0000","last_became_active":"2024-01-15T18:09:10.155914+0000","last_became_peered":"2024-01-15T18:09:10.155914+0000","last_unstale":"2024-01-15T18:09:10.163688+0000","last_undegraded":"2024-01-15T18:09:10.163688+0000","last_fullsized":"2024-01-15T18:09:10.163688+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:15:38.206084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"57'1","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.172500+0000","last_change":"2024-01-15T18:09:09.139963+0000","last_active":"2024-01-15T18:09:10.172500+0000","last_peered":"2024-01-15T18:09:10.172500+0000","last_clean":"2024-01-15T18:09:10.172500+0000","last_became_active":"2024-01-15T18:09:09.139603+0000","last_became_peered":"2024-01-15T18:09:09.139603+0000","last_unstale":"2024-01-15T18:09:10.172500+0000","last_undegraded":"2024-01-15T18:09:10.172500+0000","last_fullsized":"2024-01-15T18:09:10.172500+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:14:01.036199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.174128+0000","last_change":"2024-01-15T18:09:10.165154+0000","last_active":"2024-01-15T18:09:10.174128+0000","last_peered":"2024-01-15T18:09:10.174128+0000","last_clean":"2024-01-15T18:09:10.174128+0000","last_became_active":"2024-01-15T18:09:10.164900+0000","last_became_peered":"2024-01-15T18:09:10.164900+0000","last_unstale":"2024-01-15T18:09:10.174128+0000","last_undegraded":"2024-01-15T18:09:10.174128+0000","last_fullsized":"2024-01-15T18:09:10.174128+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:57:26.029056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.153437+0000","last_change":"2024-01-15T18:09:09.147874+0000","last_active":"2024-01-15T18:09:09.153437+0000","last_peered":"2024-01-15T18:09:09.153437+0000","last_clean":"2024-01-15T18:09:09.153437+0000","last_became_active":"2024-01-15T18:09:09.147666+0000","last_became_peered":"2024-01-15T18:09:09.147666+0000","last_unstale":"2024-01-15T18:09:09.153437+0000","last_undegraded":"2024-01-15T18:09:09.153437+0000","last_fullsized":"2024-01-15T18:09:09.153437+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:44:37.388258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163308+0000","last_change":"2024-01-15T18:09:10.154886+0000","last_active":"2024-01-15T18:09:10.163308+0000","last_peered":"2024-01-15T18:09:10.163308+0000","last_clean":"2024-01-15T18:09:10.163308+0000","last_became_active":"2024-01-15T18:09:10.154594+0000","last_became_peered":"2024-01-15T18:09:10.154594+0000","last_unstale":"2024-01-15T18:09:10.163308+0000","last_undegraded":"2024-01-15T18:09:10.163308+0000","last_fullsized":"2024-01-15T18:09:10.163308+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:34:09.142491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.137679+0000","last_change":"2024-01-15T18:09:09.140948+0000","last_active":"2024-01-15T18:09:10.137679+0000","last_peered":"2024-01-15T18:09:10.137679+0000","last_clean":"2024-01-15T18:09:10.137679+0000","last_became_active":"2024-01-15T18:09:09.140010+0000","last_became_peered":"2024-01-15T18:09:09.140010+0000","last_unstale":"2024-01-15T18:09:10.137679+0000","last_undegraded":"2024-01-15T18:09:10.137679+0000","last_fullsized":"2024-01-15T18:09:10.137679+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:39:25.682654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173959+0000","last_change":"2024-01-15T18:09:10.160588+0000","last_active":"2024-01-15T18:09:10.173959+0000","last_peered":"2024-01-15T18:09:10.173959+0000","last_clean":"2024-01-15T18:09:10.173959+0000","last_became_active":"2024-01-15T18:09:10.157197+0000","last_became_peered":"2024-01-15T18:09:10.157197+0000","last_unstale":"2024-01-15T18:09:10.173959+0000","last_undegraded":"2024-01-15T18:09:10.173959+0000","last_fullsized":"2024-01-15T18:09:10.173959+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:27:59.969234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.138903+0000","last_change":"2024-01-15T18:09:09.131170+0000","last_active":"2024-01-15T18:09:10.138903+0000","last_peered":"2024-01-15T18:09:10.138903+0000","last_clean":"2024-01-15T18:09:10.138903+0000","last_became_active":"2024-01-15T18:09:09.130789+0000","last_became_peered":"2024-01-15T18:09:09.130789+0000","last_unstale":"2024-01-15T18:09:10.138903+0000","last_undegraded":"2024-01-15T18:09:10.138903+0000","last_fullsized":"2024-01-15T18:09:10.138903+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:13:26.979622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.166623+0000","last_change":"2024-01-15T18:09:10.154127+0000","last_active":"2024-01-15T18:09:10.166623+0000","last_peered":"2024-01-15T18:09:10.166623+0000","last_clean":"2024-01-15T18:09:10.166623+0000","last_became_active":"2024-01-15T18:09:10.153699+0000","last_became_peered":"2024-01-15T18:09:10.153699+0000","last_unstale":"2024-01-15T18:09:10.166623+0000","last_undegraded":"2024-01-15T18:09:10.166623+0000","last_fullsized":"2024-01-15T18:09:10.166623+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:37:29.585671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.151155+0000","last_change":"2024-01-15T18:09:09.145070+0000","last_active":"2024-01-15T18:09:10.151155+0000","last_peered":"2024-01-15T18:09:10.151155+0000","last_clean":"2024-01-15T18:09:10.151155+0000","last_became_active":"2024-01-15T18:09:09.144808+0000","last_became_peered":"2024-01-15T18:09:09.144808+0000","last_unstale":"2024-01-15T18:09:10.151155+0000","last_undegraded":"2024-01-15T18:09:10.151155+0000","last_fullsized":"2024-01-15T18:09:10.151155+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:07:17.174892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.178747+0000","last_change":"2024-01-15T18:09:10.173359+0000","last_active":"2024-01-15T18:09:10.178747+0000","last_peered":"2024-01-15T18:09:10.178747+0000","last_clean":"2024-01-15T18:09:10.178747+0000","last_became_active":"2024-01-15T18:09:10.173020+0000","last_became_peered":"2024-01-15T18:09:10.173020+0000","last_unstale":"2024-01-15T18:09:10.178747+0000","last_undegraded":"2024-01-15T18:09:10.178747+0000","last_fullsized":"2024-01-15T18:09:10.178747+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:22:35.521714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.140713+0000","last_change":"2024-01-15T18:09:09.142076+0000","last_active":"2024-01-15T18:09:10.140713+0000","last_peered":"2024-01-15T18:09:10.140713+0000","last_clean":"2024-01-15T18:09:10.140713+0000","last_became_active":"2024-01-15T18:09:09.141747+0000","last_became_peered":"2024-01-15T18:09:09.141747+0000","last_unstale":"2024-01-15T18:09:10.140713+0000","last_undegraded":"2024-01-15T18:09:10.140713+0000","last_fullsized":"2024-01-15T18:09:10.140713+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:02.592720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.165189+0000","last_change":"2024-01-15T18:09:10.150403+0000","last_active":"2024-01-15T18:09:10.165189+0000","last_peered":"2024-01-15T18:09:10.165189+0000","last_clean":"2024-01-15T18:09:10.165189+0000","last_became_active":"2024-01-15T18:09:10.150113+0000","last_became_peered":"2024-01-15T18:09:10.150113+0000","last_unstale":"2024-01-15T18:09:10.165189+0000","last_undegraded":"2024-01-15T18:09:10.165189+0000","last_fullsized":"2024-01-15T18:09:10.165189+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:45:59.978009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"57'3","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.180373+0000","last_change":"2024-01-15T18:09:09.142251+0000","last_active":"2024-01-15T18:09:10.180373+0000","last_peered":"2024-01-15T18:09:10.180373+0000","last_clean":"2024-01-15T18:09:10.180373+0000","last_became_active":"2024-01-15T18:09:09.141961+0000","last_became_peered":"2024-01-15T18:09:09.141961+0000","last_unstale":"2024-01-15T18:09:10.180373+0000","last_undegraded":"2024-01-15T18:09:10.180373+0000","last_fullsized":"2024-01-15T18:09:10.180373+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:56:33.327612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.176749+0000","last_change":"2024-01-15T18:09:10.172402+0000","last_active":"2024-01-15T18:09:10.176749+0000","last_peered":"2024-01-15T18:09:10.176749+0000","last_clean":"2024-01-15T18:09:10.176749+0000","last_became_active":"2024-01-15T18:09:10.171813+0000","last_became_peered":"2024-01-15T18:09:10.171813+0000","last_unstale":"2024-01-15T18:09:10.176749+0000","last_undegraded":"2024-01-15T18:09:10.176749+0000","last_fullsized":"2024-01-15T18:09:10.176749+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:15:38.206084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"57'6","reported_seq":17,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.180712+0000","last_change":"2024-01-15T18:09:09.147769+0000","last_active":"2024-01-15T18:09:10.180712+0000","last_peered":"2024-01-15T18:09:10.180712+0000","last_clean":"2024-01-15T18:09:10.180712+0000","last_became_active":"2024-01-15T18:09:09.147356+0000","last_became_peered":"2024-01-15T18:09:09.147356+0000","last_unstale":"2024-01-15T18:09:10.180712+0000","last_undegraded":"2024-01-15T18:09:10.180712+0000","last_fullsized":"2024-01-15T18:09:10.180712+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:18:52.199559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155288+0000","last_change":"2024-01-15T18:09:10.154188+0000","last_active":"2024-01-15T18:09:11.155288+0000","last_peered":"2024-01-15T18:09:11.155288+0000","last_clean":"2024-01-15T18:09:11.155288+0000","last_became_active":"2024-01-15T18:09:10.153881+0000","last_became_peered":"2024-01-15T18:09:10.153881+0000","last_unstale":"2024-01-15T18:09:11.155288+0000","last_undegraded":"2024-01-15T18:09:11.155288+0000","last_fullsized":"2024-01-15T18:09:11.155288+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:57:04.284081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"57'2","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.180721+0000","last_change":"2024-01-15T18:09:09.141986+0000","last_active":"2024-01-15T18:09:10.180721+0000","last_peered":"2024-01-15T18:09:10.180721+0000","last_clean":"2024-01-15T18:09:10.180721+0000","last_became_active":"2024-01-15T18:09:09.141626+0000","last_became_peered":"2024-01-15T18:09:09.141626+0000","last_unstale":"2024-01-15T18:09:10.180721+0000","last_undegraded":"2024-01-15T18:09:10.180721+0000","last_fullsized":"2024-01-15T18:09:10.180721+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:44:08.042896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.176175+0000","last_change":"2024-01-15T18:09:10.163997+0000","last_active":"2024-01-15T18:09:10.176175+0000","last_peered":"2024-01-15T18:09:10.176175+0000","last_clean":"2024-01-15T18:09:10.176175+0000","last_became_active":"2024-01-15T18:09:10.163758+0000","last_became_peered":"2024-01-15T18:09:10.163758+0000","last_unstale":"2024-01-15T18:09:10.176175+0000","last_undegraded":"2024-01-15T18:09:10.176175+0000","last_fullsized":"2024-01-15T18:09:10.176175+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:02:59.936727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.144084+0000","last_change":"2024-01-15T18:09:09.142040+0000","last_active":"2024-01-15T18:09:10.144084+0000","last_peered":"2024-01-15T18:09:10.144084+0000","last_clean":"2024-01-15T18:09:10.144084+0000","last_became_active":"2024-01-15T18:09:09.141713+0000","last_became_peered":"2024-01-15T18:09:09.141713+0000","last_unstale":"2024-01-15T18:09:10.144084+0000","last_undegraded":"2024-01-15T18:09:10.144084+0000","last_fullsized":"2024-01-15T18:09:10.144084+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:17:15.118577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.166494+0000","last_change":"2024-01-15T18:09:10.156574+0000","last_active":"2024-01-15T18:09:10.166494+0000","last_peered":"2024-01-15T18:09:10.166494+0000","last_clean":"2024-01-15T18:09:10.166494+0000","last_became_active":"2024-01-15T18:09:10.156225+0000","last_became_peered":"2024-01-15T18:09:10.156225+0000","last_unstale":"2024-01-15T18:09:10.166494+0000","last_undegraded":"2024-01-15T18:09:10.166494+0000","last_fullsized":"2024-01-15T18:09:10.166494+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:15:39.864433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.137769+0000","last_change":"2024-01-15T18:09:09.135271+0000","last_active":"2024-01-15T18:09:10.137769+0000","last_peered":"2024-01-15T18:09:10.137769+0000","last_clean":"2024-01-15T18:09:10.137769+0000","last_became_active":"2024-01-15T18:09:09.135011+0000","last_became_peered":"2024-01-15T18:09:09.135011+0000","last_unstale":"2024-01-15T18:09:10.137769+0000","last_undegraded":"2024-01-15T18:09:10.137769+0000","last_fullsized":"2024-01-15T18:09:10.137769+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:29:02.803967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.163455+0000","last_change":"2024-01-15T18:09:10.160727+0000","last_active":"2024-01-15T18:09:10.163455+0000","last_peered":"2024-01-15T18:09:10.163455+0000","last_clean":"2024-01-15T18:09:10.163455+0000","last_became_active":"2024-01-15T18:09:10.160132+0000","last_became_peered":"2024-01-15T18:09:10.160132+0000","last_unstale":"2024-01-15T18:09:10.163455+0000","last_undegraded":"2024-01-15T18:09:10.163455+0000","last_fullsized":"2024-01-15T18:09:10.163455+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:27:59.969234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"57'8","reported_seq":19,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.177242+0000","last_change":"2024-01-15T18:09:09.137932+0000","last_active":"2024-01-15T18:09:10.177242+0000","last_peered":"2024-01-15T18:09:10.177242+0000","last_clean":"2024-01-15T18:09:10.177242+0000","last_became_active":"2024-01-15T18:09:09.137626+0000","last_became_peered":"2024-01-15T18:09:09.137626+0000","last_unstale":"2024-01-15T18:09:10.177242+0000","last_undegraded":"2024-01-15T18:09:10.177242+0000","last_fullsized":"2024-01-15T18:09:10.177242+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:14:01.036199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.177049+0000","last_change":"2024-01-15T18:09:10.165126+0000","last_active":"2024-01-15T18:09:10.177049+0000","last_peered":"2024-01-15T18:09:10.177049+0000","last_clean":"2024-01-15T18:09:10.177049+0000","last_became_active":"2024-01-15T18:09:10.164811+0000","last_became_peered":"2024-01-15T18:09:10.164811+0000","last_unstale":"2024-01-15T18:09:10.177049+0000","last_undegraded":"2024-01-15T18:09:10.177049+0000","last_fullsized":"2024-01-15T18:09:10.177049+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:53:41.639339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"57'1","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.180173+0000","last_change":"2024-01-15T18:09:09.137481+0000","last_active":"2024-01-15T18:09:10.180173+0000","last_peered":"2024-01-15T18:09:10.180173+0000","last_clean":"2024-01-15T18:09:10.180173+0000","last_became_active":"2024-01-15T18:09:09.137205+0000","last_became_peered":"2024-01-15T18:09:09.137205+0000","last_unstale":"2024-01-15T18:09:10.180173+0000","last_undegraded":"2024-01-15T18:09:10.180173+0000","last_fullsized":"2024-01-15T18:09:10.180173+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:16:45.705931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155989+0000","last_change":"2024-01-15T18:09:10.166022+0000","last_active":"2024-01-15T18:09:11.155989+0000","last_peered":"2024-01-15T18:09:11.155989+0000","last_clean":"2024-01-15T18:09:11.155989+0000","last_became_active":"2024-01-15T18:09:10.165752+0000","last_became_peered":"2024-01-15T18:09:10.165752+0000","last_unstale":"2024-01-15T18:09:11.155989+0000","last_undegraded":"2024-01-15T18:09:11.155989+0000","last_fullsized":"2024-01-15T18:09:11.155989+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:45:32.407323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.155562+0000","last_change":"2024-01-15T18:09:09.152740+0000","last_active":"2024-01-15T18:09:09.155562+0000","last_peered":"2024-01-15T18:09:09.155562+0000","last_clean":"2024-01-15T18:09:09.155562+0000","last_became_active":"2024-01-15T18:09:09.152418+0000","last_became_peered":"2024-01-15T18:09:09.152418+0000","last_unstale":"2024-01-15T18:09:09.155562+0000","last_undegraded":"2024-01-15T18:09:09.155562+0000","last_fullsized":"2024-01-15T18:09:09.155562+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:56:49.381443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.166880+0000","last_change":"2024-01-15T18:09:10.155205+0000","last_active":"2024-01-15T18:09:10.166880+0000","last_peered":"2024-01-15T18:09:10.166880+0000","last_clean":"2024-01-15T18:09:10.166880+0000","last_became_active":"2024-01-15T18:09:10.154910+0000","last_became_peered":"2024-01-15T18:09:10.154910+0000","last_unstale":"2024-01-15T18:09:10.166880+0000","last_undegraded":"2024-01-15T18:09:10.166880+0000","last_fullsized":"2024-01-15T18:09:10.166880+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:56:34.329818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.177022+0000","last_change":"2024-01-15T18:09:09.141114+0000","last_active":"2024-01-15T18:09:10.177022+0000","last_peered":"2024-01-15T18:09:10.177022+0000","last_clean":"2024-01-15T18:09:10.177022+0000","last_became_active":"2024-01-15T18:09:09.140786+0000","last_became_peered":"2024-01-15T18:09:09.140786+0000","last_unstale":"2024-01-15T18:09:10.177022+0000","last_undegraded":"2024-01-15T18:09:10.177022+0000","last_fullsized":"2024-01-15T18:09:10.177022+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:37:27.921887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.176767+0000","last_change":"2024-01-15T18:09:10.172606+0000","last_active":"2024-01-15T18:09:10.176767+0000","last_peered":"2024-01-15T18:09:10.176767+0000","last_clean":"2024-01-15T18:09:10.176767+0000","last_became_active":"2024-01-15T18:09:10.172090+0000","last_became_peered":"2024-01-15T18:09:10.172090+0000","last_unstale":"2024-01-15T18:09:10.176767+0000","last_undegraded":"2024-01-15T18:09:10.176767+0000","last_fullsized":"2024-01-15T18:09:10.176767+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:25:19.060578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.153526+0000","last_change":"2024-01-15T18:09:09.148967+0000","last_active":"2024-01-15T18:09:09.153526+0000","last_peered":"2024-01-15T18:09:09.153526+0000","last_clean":"2024-01-15T18:09:09.153526+0000","last_became_active":"2024-01-15T18:09:09.148670+0000","last_became_peered":"2024-01-15T18:09:09.148670+0000","last_unstale":"2024-01-15T18:09:09.153526+0000","last_undegraded":"2024-01-15T18:09:09.153526+0000","last_fullsized":"2024-01-15T18:09:09.153526+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:04:47.742716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155444+0000","last_change":"2024-01-15T18:09:10.151767+0000","last_active":"2024-01-15T18:09:11.155444+0000","last_peered":"2024-01-15T18:09:11.155444+0000","last_clean":"2024-01-15T18:09:11.155444+0000","last_became_active":"2024-01-15T18:09:10.151462+0000","last_became_peered":"2024-01-15T18:09:10.151462+0000","last_unstale":"2024-01-15T18:09:11.155444+0000","last_undegraded":"2024-01-15T18:09:11.155444+0000","last_fullsized":"2024-01-15T18:09:11.155444+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:07:18.547932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"57'2","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.174259+0000","last_change":"2024-01-15T18:09:09.141213+0000","last_active":"2024-01-15T18:09:10.174259+0000","last_peered":"2024-01-15T18:09:10.174259+0000","last_clean":"2024-01-15T18:09:10.174259+0000","last_became_active":"2024-01-15T18:09:09.139957+0000","last_became_peered":"2024-01-15T18:09:09.139957+0000","last_unstale":"2024-01-15T18:09:10.174259+0000","last_undegraded":"2024-01-15T18:09:10.174259+0000","last_fullsized":"2024-01-15T18:09:10.174259+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:42:47.132801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.127537+0000","last_change":"2024-01-15T18:09:09.127537+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.127537+0000","last_undegraded":"2024-01-15T18:09:09.127537+0000","last_fullsized":"2024-01-15T18:09:09.127537+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.133917+0000","last_change":"2024-01-15T18:09:09.136084+0000","last_active":"2024-01-15T18:09:10.133917+0000","last_peered":"2024-01-15T18:09:10.133917+0000","last_clean":"2024-01-15T18:09:10.133917+0000","last_became_active":"2024-01-15T18:09:09.135760+0000","last_became_peered":"2024-01-15T18:09:09.135760+0000","last_unstale":"2024-01-15T18:09:10.133917+0000","last_undegraded":"2024-01-15T18:09:10.133917+0000","last_fullsized":"2024-01-15T18:09:10.133917+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:37.003640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.128633+0000","last_change":"2024-01-15T18:09:09.128633+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.128633+0000","last_undegraded":"2024-01-15T18:09:09.128633+0000","last_fullsized":"2024-01-15T18:09:09.128633+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.137490+0000","last_change":"2024-01-15T18:09:09.136145+0000","last_active":"2024-01-15T18:09:10.137490+0000","last_peered":"2024-01-15T18:09:10.137490+0000","last_clean":"2024-01-15T18:09:10.137490+0000","last_became_active":"2024-01-15T18:09:09.135807+0000","last_became_peered":"2024-01-15T18:09:09.135807+0000","last_unstale":"2024-01-15T18:09:10.137490+0000","last_undegraded":"2024-01-15T18:09:10.137490+0000","last_fullsized":"2024-01-15T18:09:10.137490+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:35:26.455284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155791+0000","last_change":"2024-01-15T18:09:10.157661+0000","last_active":"2024-01-15T18:09:11.155791+0000","last_peered":"2024-01-15T18:09:11.155791+0000","last_clean":"2024-01-15T18:09:11.155791+0000","last_became_active":"2024-01-15T18:09:10.157399+0000","last_became_peered":"2024-01-15T18:09:10.157399+0000","last_unstale":"2024-01-15T18:09:11.155791+0000","last_undegraded":"2024-01-15T18:09:11.155791+0000","last_fullsized":"2024-01-15T18:09:11.155791+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:41:56.291009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.177996+0000","last_change":"2024-01-15T18:09:09.145612+0000","last_active":"2024-01-15T18:09:10.177996+0000","last_peered":"2024-01-15T18:09:10.177996+0000","last_clean":"2024-01-15T18:09:10.177996+0000","last_became_active":"2024-01-15T18:09:09.145279+0000","last_became_peered":"2024-01-15T18:09:09.145279+0000","last_unstale":"2024-01-15T18:09:10.177996+0000","last_undegraded":"2024-01-15T18:09:10.177996+0000","last_fullsized":"2024-01-15T18:09:10.177996+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:53.930232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.156102+0000","last_change":"2024-01-15T18:09:10.153156+0000","last_active":"2024-01-15T18:09:11.156102+0000","last_peered":"2024-01-15T18:09:11.156102+0000","last_clean":"2024-01-15T18:09:11.156102+0000","last_became_active":"2024-01-15T18:09:10.152783+0000","last_became_peered":"2024-01-15T18:09:10.152783+0000","last_unstale":"2024-01-15T18:09:11.156102+0000","last_undegraded":"2024-01-15T18:09:11.156102+0000","last_fullsized":"2024-01-15T18:09:11.156102+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:36:30.872154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.155546+0000","last_change":"2024-01-15T18:09:09.152741+0000","last_active":"2024-01-15T18:09:09.155546+0000","last_peered":"2024-01-15T18:09:09.155546+0000","last_clean":"2024-01-15T18:09:09.155546+0000","last_became_active":"2024-01-15T18:09:09.152470+0000","last_became_peered":"2024-01-15T18:09:09.152470+0000","last_unstale":"2024-01-15T18:09:09.155546+0000","last_undegraded":"2024-01-15T18:09:09.155546+0000","last_fullsized":"2024-01-15T18:09:09.155546+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:14:31.566501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.153880+0000","last_change":"2024-01-15T18:09:10.143039+0000","last_active":"2024-01-15T18:09:10.153880+0000","last_peered":"2024-01-15T18:09:10.153880+0000","last_clean":"2024-01-15T18:09:10.153880+0000","last_became_active":"2024-01-15T18:09:10.142814+0000","last_became_peered":"2024-01-15T18:09:10.142814+0000","last_unstale":"2024-01-15T18:09:10.153880+0000","last_undegraded":"2024-01-15T18:09:10.153880+0000","last_fullsized":"2024-01-15T18:09:10.153880+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:54:31.215934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.143973+0000","last_change":"2024-01-15T18:09:09.141870+0000","last_active":"2024-01-15T18:09:10.143973+0000","last_peered":"2024-01-15T18:09:10.143973+0000","last_clean":"2024-01-15T18:09:10.143973+0000","last_became_active":"2024-01-15T18:09:09.141523+0000","last_became_peered":"2024-01-15T18:09:09.141523+0000","last_unstale":"2024-01-15T18:09:10.143973+0000","last_undegraded":"2024-01-15T18:09:10.143973+0000","last_fullsized":"2024-01-15T18:09:10.143973+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:15:13.572542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":56,"state":"creating+peering","last_fresh":"2024-01-15T18:09:09.132306+0000","last_change":"2024-01-15T18:09:09.132306+0000","last_active":"2024-01-15T18:09:09.109985+0000","last_peered":"2024-01-15T18:09:09.109985+0000","last_clean":"2024-01-15T18:09:09.109985+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:09:09.132306+0000","last_undegraded":"2024-01-15T18:09:09.132306+0000","last_fullsized":"2024-01-15T18:09:09.132306+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.137795+0000","last_change":"2024-01-15T18:09:09.136593+0000","last_active":"2024-01-15T18:09:10.137795+0000","last_peered":"2024-01-15T18:09:10.137795+0000","last_clean":"2024-01-15T18:09:10.137795+0000","last_became_active":"2024-01-15T18:09:09.136246+0000","last_became_peered":"2024-01-15T18:09:09.136246+0000","last_unstale":"2024-01-15T18:09:10.137795+0000","last_undegraded":"2024-01-15T18:09:10.137795+0000","last_fullsized":"2024-01-15T18:09:10.137795+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:10:09.158484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.155341+0000","last_change":"2024-01-15T18:09:10.145733+0000","last_active":"2024-01-15T18:09:10.155341+0000","last_peered":"2024-01-15T18:09:10.155341+0000","last_clean":"2024-01-15T18:09:10.155341+0000","last_became_active":"2024-01-15T18:09:10.145479+0000","last_became_peered":"2024-01-15T18:09:10.145479+0000","last_unstale":"2024-01-15T18:09:10.155341+0000","last_undegraded":"2024-01-15T18:09:10.155341+0000","last_fullsized":"2024-01-15T18:09:10.155341+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:53:41.639339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.133917+0000","last_change":"2024-01-15T18:09:09.142681+0000","last_active":"2024-01-15T18:09:10.133917+0000","last_peered":"2024-01-15T18:09:10.133917+0000","last_clean":"2024-01-15T18:09:10.133917+0000","last_became_active":"2024-01-15T18:09:09.142357+0000","last_became_peered":"2024-01-15T18:09:09.142357+0000","last_unstale":"2024-01-15T18:09:10.133917+0000","last_undegraded":"2024-01-15T18:09:10.133917+0000","last_fullsized":"2024-01-15T18:09:10.133917+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:41:15.076278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173119+0000","last_change":"2024-01-15T18:09:10.162125+0000","last_active":"2024-01-15T18:09:10.173119+0000","last_peered":"2024-01-15T18:09:10.173119+0000","last_clean":"2024-01-15T18:09:10.173119+0000","last_became_active":"2024-01-15T18:09:10.161778+0000","last_became_peered":"2024-01-15T18:09:10.161778+0000","last_unstale":"2024-01-15T18:09:10.173119+0000","last_undegraded":"2024-01-15T18:09:10.173119+0000","last_fullsized":"2024-01-15T18:09:10.173119+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:07:31.195306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.139632+0000","last_change":"2024-01-15T18:09:09.135449+0000","last_active":"2024-01-15T18:09:10.139632+0000","last_peered":"2024-01-15T18:09:10.139632+0000","last_clean":"2024-01-15T18:09:10.139632+0000","last_became_active":"2024-01-15T18:09:09.135010+0000","last_became_peered":"2024-01-15T18:09:09.135010+0000","last_unstale":"2024-01-15T18:09:10.139632+0000","last_undegraded":"2024-01-15T18:09:10.139632+0000","last_fullsized":"2024-01-15T18:09:10.139632+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:54:08.460596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.170500+0000","last_change":"2024-01-15T18:09:10.158452+0000","last_active":"2024-01-15T18:09:10.170500+0000","last_peered":"2024-01-15T18:09:10.170500+0000","last_clean":"2024-01-15T18:09:10.170500+0000","last_became_active":"2024-01-15T18:09:10.158267+0000","last_became_peered":"2024-01-15T18:09:10.158267+0000","last_unstale":"2024-01-15T18:09:10.170500+0000","last_undegraded":"2024-01-15T18:09:10.170500+0000","last_fullsized":"2024-01-15T18:09:10.170500+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:26:45.212614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.133918+0000","last_change":"2024-01-15T18:09:09.146561+0000","last_active":"2024-01-15T18:09:10.133918+0000","last_peered":"2024-01-15T18:09:10.133918+0000","last_clean":"2024-01-15T18:09:10.133918+0000","last_became_active":"2024-01-15T18:09:09.146263+0000","last_became_peered":"2024-01-15T18:09:09.146263+0000","last_unstale":"2024-01-15T18:09:10.133918+0000","last_undegraded":"2024-01-15T18:09:10.133918+0000","last_fullsized":"2024-01-15T18:09:10.133918+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:00:48.078792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173947+0000","last_change":"2024-01-15T18:09:10.157427+0000","last_active":"2024-01-15T18:09:10.173947+0000","last_peered":"2024-01-15T18:09:10.173947+0000","last_clean":"2024-01-15T18:09:10.173947+0000","last_became_active":"2024-01-15T18:09:10.157146+0000","last_became_peered":"2024-01-15T18:09:10.157146+0000","last_unstale":"2024-01-15T18:09:10.173947+0000","last_undegraded":"2024-01-15T18:09:10.173947+0000","last_fullsized":"2024-01-15T18:09:10.173947+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:07:31.195306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.142185+0000","last_change":"2024-01-15T18:09:09.136141+0000","last_active":"2024-01-15T18:09:10.142185+0000","last_peered":"2024-01-15T18:09:10.142185+0000","last_clean":"2024-01-15T18:09:10.142185+0000","last_became_active":"2024-01-15T18:09:09.135853+0000","last_became_peered":"2024-01-15T18:09:09.135853+0000","last_unstale":"2024-01-15T18:09:10.142185+0000","last_undegraded":"2024-01-15T18:09:10.142185+0000","last_fullsized":"2024-01-15T18:09:10.142185+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:25:50.421687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.173953+0000","last_change":"2024-01-15T18:09:10.161307+0000","last_active":"2024-01-15T18:09:10.173953+0000","last_peered":"2024-01-15T18:09:10.173953+0000","last_clean":"2024-01-15T18:09:10.173953+0000","last_became_active":"2024-01-15T18:09:10.160945+0000","last_became_peered":"2024-01-15T18:09:10.160945+0000","last_unstale":"2024-01-15T18:09:10.173953+0000","last_undegraded":"2024-01-15T18:09:10.173953+0000","last_fullsized":"2024-01-15T18:09:10.173953+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:26:05.237285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.144458+0000","last_change":"2024-01-15T18:09:09.142696+0000","last_active":"2024-01-15T18:09:10.144458+0000","last_peered":"2024-01-15T18:09:10.144458+0000","last_clean":"2024-01-15T18:09:10.144458+0000","last_became_active":"2024-01-15T18:09:09.142448+0000","last_became_peered":"2024-01-15T18:09:09.142448+0000","last_unstale":"2024-01-15T18:09:10.144458+0000","last_undegraded":"2024-01-15T18:09:10.144458+0000","last_fullsized":"2024-01-15T18:09:10.144458+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:40:45.529038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":194,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-01-15T18:09:09.122277+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:09:09.122277+0000","last_peered":"2024-01-15T18:09:09.122277+0000","last_clean":"2024-01-15T18:09:09.122277+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:09:09.122277+0000","last_undegraded":"2024-01-15T18:09:09.122277+0000","last_fullsized":"2024-01-15T18:09:09.122277+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":1828,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":2626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509496,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36572,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335268,"statfs":{"total":96636764160,"available":96599314432,"internally_reserved":0,"allocated":1257472,"data_stored":864881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":0.82899999999999996,"15min":0.69199999999999995},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":10.284000000000001,"5min":10.284000000000001,"15min":10.284000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":1.8440000000000001,"5min":0.89800000000000002,"15min":0.752},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":9.9610000000000003,"5min":9.9610000000000003,"15min":9.9610000000000003},"last":0.51800000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":0.76600000000000001,"15min":0.69999999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":7.0279999999999996,"5min":7.0279999999999996,"15min":7.0279999999999996},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.7230000000000001,"5min":0.89700000000000002,"15min":0.76500000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":9.9250000000000007,"5min":9.9250000000000007,"15min":9.9250000000000007},"last":0.73599999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":0.872,"15min":0.72199999999999998},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":10.119999999999999,"5min":10.119999999999999,"15min":10.119999999999999},"last":0.495},{"interface":"front","average":{"1min":1.242,"5min":0.82199999999999995,"15min":0.748},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":7.0789999999999997,"5min":7.0789999999999997,"15min":7.0789999999999997},"last":0.67400000000000004}]},{"osd":3,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.625,"5min":0.92500000000000004,"15min":0.751},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":9.3539999999999992,"5min":9.3539999999999992,"15min":9.3539999999999992},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.7070000000000001,"5min":0.93400000000000005,"15min":0.81999999999999995},"min":{"1min":0.45700000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":10.34,"5min":10.34,"15min":10.34},"last":0.58899999999999997}]},{"osd":4,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":2.4079999999999999,"5min":0.96499999999999997,"15min":0.73499999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":17.021000000000001,"5min":17.021000000000001,"15min":17.021000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.734,"5min":0.92700000000000005,"15min":0.752},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":14.871,"5min":14.871,"15min":14.871},"last":0.57299999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.837,"5min":0.91000000000000003,"15min":0.77000000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":19.959,"5min":19.959,"15min":19.959},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.9610000000000001,"5min":0.94399999999999995,"15min":0.80100000000000005},"min":{"1min":0.47999999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":11.053000000000001,"5min":11.053000000000001,"15min":11.053000000000001},"last":0.623}]},{"osd":6,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":3.0070000000000001,"5min":1.2170000000000001,"15min":0.94299999999999995},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":22.478999999999999,"5min":22.478999999999999,"15min":22.478999999999999},"last":0.78600000000000003},{"interface":"front","average":{"1min":2.6269999999999998,"5min":1.1739999999999999,"15min":0.88100000000000001},"min":{"1min":0.52800000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":21.509,"5min":21.509,"15min":21.509},"last":0.77200000000000002}]}]},{"osd":6,"up_from":15,"seq":64424509496,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35988,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335852,"statfs":{"total":96636764160,"available":96599912448,"internally_reserved":0,"allocated":663552,"data_stored":274513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":0.86299999999999999,"15min":0.77800000000000002},"min":{"1min":0.45200000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":9.3339999999999996,"5min":9.3339999999999996,"15min":9.3339999999999996},"last":0.623},{"interface":"front","average":{"1min":1.4850000000000001,"5min":0.83599999999999997,"15min":0.74399999999999999},"min":{"1min":0.47099999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":6.7759999999999998,"5min":6.7759999999999998,"15min":6.7759999999999998},"last":0.79800000000000004}]},{"osd":1,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":3.8119999999999998,"5min":1.329,"15min":0.93300000000000005},"min":{"1min":0.51400000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":53.451999999999998,"5min":53.451999999999998,"15min":53.451999999999998},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.3009999999999999,"5min":0.83599999999999997,"15min":0.78400000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":5.9900000000000002,"5min":5.9900000000000002,"15min":5.9900000000000002},"last":0.68400000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":0.84499999999999997,"15min":0.85199999999999998},"min":{"1min":0.46700000000000003,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":6.0380000000000003,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.2310000000000001,"5min":0.82799999999999996,"15min":0.80700000000000005},"min":{"1min":0.57099999999999995,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":4.6059999999999999,"5min":4.6059999999999999,"15min":4.6059999999999999},"last":0.66600000000000004}]},{"osd":3,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":0.84899999999999998,"15min":0.82599999999999996},"min":{"1min":0.51900000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":3.9689999999999999,"5min":3.9689999999999999,"15min":3.9689999999999999},"last":0.755},{"interface":"front","average":{"1min":3.6709999999999998,"5min":1.355,"15min":1.0109999999999999},"min":{"1min":0.45200000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":0.88}]},{"osd":4,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.6479999999999999,"5min":0.85499999999999998,"15min":0.751},"min":{"1min":0.48099999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":8.6259999999999994,"5min":8.6259999999999994,"15min":8.6259999999999994},"last":0.45900000000000002},{"interface":"front","average":{"1min":1.2809999999999999,"5min":0.79600000000000004,"15min":0.752},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":6.9589999999999996,"5min":6.9589999999999996,"15min":6.9589999999999996},"last":0.50900000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":4.2560000000000002,"5min":1.371,"15min":0.90000000000000002},"min":{"1min":0.44500000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":53.229999999999997,"5min":53.229999999999997,"15min":53.229999999999997},"last":0.85199999999999998},{"interface":"front","average":{"1min":1.847,"5min":0.96799999999999997,"15min":0.84599999999999997},"min":{"1min":0.53900000000000003,"5min":0.435,"15min":0.435},"max":{"1min":10.029999999999999,"5min":10.029999999999999,"15min":10.029999999999999},"last":0.54500000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":3.1640000000000001,"5min":1.2589999999999999,"15min":0.97699999999999998},"min":{"1min":0.46200000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":45.070999999999998,"5min":45.070999999999998,"15min":45.070999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":4.5990000000000002,"5min":1.5369999999999999,"15min":1.054},"min":{"1min":0.48499999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":53.338000000000001,"5min":53.338000000000001,"15min":53.338000000000001},"last":0.70399999999999996}]}]},{"osd":4,"up_from":15,"seq":64424509496,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35992,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335848,"statfs":{"total":96636764160,"available":96599908352,"internally_reserved":0,"allocated":663552,"data_stored":274513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":0.91700000000000004,"15min":0.72599999999999998},"min":{"1min":0.434,"5min":0.38,"15min":0.38},"max":{"1min":23.318000000000001,"5min":23.318000000000001,"15min":23.318000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.7649999999999999,"5min":0.84999999999999998,"15min":0.71399999999999997},"min":{"1min":0.44800000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":10.210000000000001,"5min":10.210000000000001,"15min":10.210000000000001},"last":0.57299999999999995}]},{"osd":1,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.71099999999999997,"15min":0.67200000000000004},"min":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":5.016,"5min":5.016,"15min":5.016},"last":0.85099999999999998},{"interface":"front","average":{"1min":2.0990000000000002,"5min":0.93400000000000005,"15min":0.73999999999999999},"min":{"1min":0.45100000000000001,"5min":0.377,"15min":0.377},"max":{"1min":22.408999999999999,"5min":22.408999999999999,"15min":22.408999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":0.71799999999999997,"15min":0.65900000000000003},"min":{"1min":0.46700000000000003,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":5.1159999999999997,"5min":5.1159999999999997,"15min":5.1159999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":2.1419999999999999,"5min":1.036,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":22.338999999999999,"5min":22.338999999999999,"15min":22.338999999999999},"last":0.54900000000000004}]},{"osd":3,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.8380000000000001,"5min":0.91600000000000004,"15min":0.76600000000000001},"min":{"1min":0.42699999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":10.271000000000001,"5min":10.271000000000001,"15min":10.271000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.27,"5min":0.84099999999999997,"15min":0.76500000000000001},"min":{"1min":0.59899999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":8.0719999999999992,"5min":8.0719999999999992,"15min":8.0719999999999992},"last":0.749}]},{"osd":5,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":2.7879999999999998,"5min":1.081,"15min":0.80400000000000005},"min":{"1min":0.39000000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":31.617999999999999,"5min":31.617999999999999,"15min":31.617999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":2.4369999999999998,"5min":1.0009999999999999,"15min":0.76700000000000002},"min":{"1min":0.439,"5min":0.377,"15min":0.377},"max":{"1min":17.145,"5min":17.145,"15min":17.145},"last":0.86399999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":0.83499999999999996,"15min":0.73099999999999998},"min":{"1min":0.53900000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.577,"5min":5.577,"15min":5.577},"last":0.80300000000000005},{"interface":"front","average":{"1min":4.9329999999999998,"5min":1.5720000000000001,"15min":1.014},"min":{"1min":0.56299999999999994,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":28.872,"5min":28.872,"15min":28.872},"last":0.78200000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":3.726,"5min":1.3149999999999999,"15min":0.91700000000000004},"min":{"1min":0.48199999999999998,"5min":0.436,"15min":0.436},"max":{"1min":39.256,"5min":39.256,"15min":39.256},"last":0.82599999999999996},{"interface":"front","average":{"1min":2.069,"5min":1.0389999999999999,"15min":0.86699999999999999},"min":{"1min":0.51800000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":26.573,"5min":26.573,"15min":26.573},"last":0.88700000000000001}]}]},{"osd":3,"up_from":15,"seq":64424509495,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35992,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335848,"statfs":{"total":96636764160,"available":96599908352,"internally_reserved":0,"allocated":663552,"data_stored":274513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.45900000000000002,"15min":0.433},"min":{"1min":0.28899999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.641,"5min":2.641,"15min":2.641},"last":0.434},{"interface":"front","average":{"1min":1.4099999999999999,"5min":0.68300000000000005,"15min":0.54100000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":17.756,"5min":17.756,"15min":17.756},"last":0.33000000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.54100000000000004,"15min":0.52000000000000002},"min":{"1min":0.29799999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.0620000000000003,"5min":4.0620000000000003,"15min":4.0620000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.54500000000000004,"15min":0.53900000000000003},"min":{"1min":0.371,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.38}]},{"osd":2,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":0.69199999999999995,"15min":0.60499999999999998},"min":{"1min":0.42499999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":10.849,"5min":10.849,"15min":10.849},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.626,"15min":0.58499999999999996},"min":{"1min":0.40500000000000003,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.617,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":0.5}]},{"osd":4,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":1.756,"5min":1.004,"15min":0.873},"min":{"1min":0.56499999999999995,"5min":0.5,"15min":0.5},"max":{"1min":17.853999999999999,"5min":17.853999999999999,"15min":17.853999999999999},"last":0.63},{"interface":"front","average":{"1min":1.022,"5min":0.85099999999999998,"15min":0.80500000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":4.1420000000000003,"5min":5.0369999999999999,"15min":5.0369999999999999},"last":0.71399999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":0.82699999999999996,"15min":0.76900000000000002},"min":{"1min":0.51100000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":4.7320000000000002,"5min":4.7320000000000002,"15min":4.7320000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":2.145,"5min":1.004,"15min":0.81499999999999995},"min":{"1min":0.54900000000000004,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":17.937999999999999,"5min":17.937999999999999,"15min":17.937999999999999},"last":0.57399999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":2.125,"5min":1.046,"15min":0.873},"min":{"1min":0.52300000000000002,"5min":0.499,"15min":0.499},"max":{"1min":23.100999999999999,"5min":23.100999999999999,"15min":23.100999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":1.113,"5min":0.84599999999999997,"15min":0.79000000000000004},"min":{"1min":0.61899999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":5.7030000000000003,"5min":5.7030000000000003,"15min":5.7030000000000003},"last":0.59399999999999997}]},{"osd":7,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":0.95999999999999996,"15min":0.83799999999999997},"min":{"1min":0.54900000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":17.896999999999998,"5min":17.896999999999998,"15min":17.896999999999998},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.1020000000000001,"5min":0.93999999999999995,"15min":0.86199999999999999},"min":{"1min":0.60599999999999998,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":4.1920000000000002,"5min":10.705,"15min":10.705},"last":0.68300000000000005}]}]},{"osd":2,"up_from":15,"seq":64424509495,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36008,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335832,"statfs":{"total":96636764160,"available":96599891968,"internally_reserved":0,"allocated":684032,"data_stored":281372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.55500000000000005,"15min":0.49399999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":5.1849999999999996,"5min":5.1849999999999996,"15min":5.1849999999999996},"last":0.29899999999999999},{"interface":"front","average":{"1min":1.0640000000000001,"5min":0.57799999999999996,"15min":0.504},"min":{"1min":0.307,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":7.516,"5min":7.516,"15min":7.516},"last":0.36399999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.60799999999999998,"15min":0.56599999999999995},"min":{"1min":0.35499999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.2509999999999999,"5min":4.8079999999999998,"15min":4.8079999999999998},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.61599999999999999,"15min":0.50800000000000001},"min":{"1min":0.32500000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":8.9190000000000005,"5min":8.9190000000000005,"15min":8.9190000000000005},"last":0.32500000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":0.69799999999999995,"15min":0.61299999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":12.185,"5min":12.185,"15min":12.185},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.68600000000000005,"15min":0.59599999999999997},"min":{"1min":0.42899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":7.4580000000000002,"5min":7.4580000000000002,"15min":7.4580000000000002},"last":0.435}]},{"osd":4,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":0.91400000000000003,"15min":0.80100000000000005},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":8.6609999999999996,"5min":8.6609999999999996,"15min":8.6609999999999996},"last":0.68799999999999994},{"interface":"front","average":{"1min":2.3519999999999999,"5min":1.0820000000000001,"15min":0.875},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":19.146999999999998,"5min":19.146999999999998,"15min":19.146999999999998},"last":0.79000000000000004}]},{"osd":5,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":2.4929999999999999,"5min":1.1100000000000001,"15min":0.90200000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":31.710000000000001,"5min":31.710000000000001,"15min":31.710000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":2.7970000000000002,"5min":1.2150000000000001,"15min":0.93300000000000005},"min":{"1min":0.52900000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":22.539000000000001,"5min":22.539000000000001,"15min":22.539000000000001},"last":0.747}]},{"osd":6,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":0.92000000000000004,"15min":0.82899999999999996},"min":{"1min":0.51100000000000001,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":9.4540000000000006,"5min":9.4540000000000006,"15min":9.4540000000000006},"last":0.85699999999999998},{"interface":"front","average":{"1min":2.29,"5min":1.085,"15min":0.89500000000000002},"min":{"1min":0.54400000000000004,"5min":0.503,"15min":0.503},"max":{"1min":18.614999999999998,"5min":18.614999999999998,"15min":18.614999999999998},"last":0.873}]},{"osd":7,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":0.94599999999999995,"15min":0.85199999999999998},"min":{"1min":0.51200000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":11.432,"5min":11.432,"15min":11.432},"last":0.76000000000000001},{"interface":"front","average":{"1min":2.577,"5min":1.204,"15min":0.95499999999999996},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":17.013000000000002,"5min":17.013000000000002,"15min":17.013000000000002},"last":0.94799999999999995}]}]},{"osd":1,"up_from":15,"seq":64424509497,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35992,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335848,"statfs":{"total":96636764160,"available":96599908352,"internally_reserved":0,"allocated":663552,"data_stored":274513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":0.78400000000000003,"15min":0.60199999999999998},"min":{"1min":0.27300000000000002,"5min":0.254,"15min":0.254},"max":{"1min":9.8620000000000001,"5min":19.577000000000002,"15min":19.577000000000002},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.052,"15min":0.67600000000000005},"min":{"1min":0.28799999999999998,"5min":0.251,"15min":0.251},"max":{"1min":5.6639999999999997,"5min":48.277999999999999,"15min":48.277999999999999},"last":0.57599999999999996}]},{"osd":2,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":0.997,"15min":0.69799999999999995},"min":{"1min":0.312,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":6.4569999999999999,"5min":40.698,"15min":40.698},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.97,"5min":1.351,"15min":0.94899999999999995},"min":{"1min":0.375,"5min":0.317,"15min":0.317},"max":{"1min":18.827000000000002,"5min":48.316000000000003,"15min":48.316000000000003},"last":0.39500000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.1899999999999999,"15min":0.83199999999999996},"min":{"1min":0.373,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.915,"5min":48.234999999999999,"15min":48.234999999999999},"last":0.432},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.65700000000000003,"15min":0.61699999999999999},"min":{"1min":0.34699999999999998,"5min":0.311,"15min":0.311},"max":{"1min":4.1420000000000003,"5min":6.093,"15min":6.093},"last":0.55300000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":0.86899999999999999,"15min":0.751},"min":{"1min":0.48299999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":4.1950000000000003,"5min":5.9169999999999998,"15min":5.9169999999999998},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.175,"5min":1.0129999999999999,"15min":0.85799999999999998},"min":{"1min":0.46999999999999997,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":6.3789999999999996,"5min":19.829000000000001,"15min":19.829000000000001},"last":0.86099999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":1.339,"15min":0.96099999999999997},"min":{"1min":0.58399999999999996,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":22.594999999999999,"5min":22.594999999999999,"15min":22.594999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":2.0470000000000002,"5min":1.466,"15min":1.004},"min":{"1min":0.55900000000000005,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":11.768000000000001,"5min":48.198,"15min":48.198},"last":0.91500000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":3.0390000000000001,"5min":1.2649999999999999,"15min":0.92500000000000004},"min":{"1min":0.59199999999999997,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":35.112000000000002,"5min":35.112000000000002,"15min":35.112000000000002},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.516,"5min":1.123,"15min":0.92500000000000004},"min":{"1min":0.58399999999999996,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":8.3670000000000009,"5min":19.780999999999999,"15min":19.780999999999999},"last":0.73999999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.3959999999999999,"15min":0.98799999999999999},"min":{"1min":0.56999999999999995,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":9.2870000000000008,"5min":48.148000000000003,"15min":48.148000000000003},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.325,"5min":0.95499999999999996,"15min":0.85499999999999998},"min":{"1min":0.56799999999999995,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":4.2919999999999998,"5min":6.0549999999999997,"15min":6.0549999999999997},"last":0.89900000000000002}]}]},{"osd":5,"up_from":14,"seq":60129542201,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36020,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335820,"statfs":{"total":96636764160,"available":96599879680,"internally_reserved":0,"allocated":696320,"data_stored":288296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.85999999999999999,"15min":0.77200000000000002},"min":{"1min":0.41199999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.2640000000000002,"5min":21.385000000000002,"15min":21.385000000000002},"last":0.60599999999999998},{"interface":"front","average":{"1min":1.173,"5min":0.91900000000000004,"15min":0.83699999999999997},"min":{"1min":0.51100000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":8.6590000000000007,"5min":8.6590000000000007,"15min":8.6590000000000007},"last":0.54200000000000004}]},{"osd":1,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.70799999999999996,"15min":0.68700000000000006},"min":{"1min":0.42999999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":3.1110000000000002,"5min":3.1110000000000002,"15min":3.1110000000000002},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.89700000000000002,"15min":0.755},"min":{"1min":0.47299999999999998,"5min":0.315,"15min":0.315},"max":{"1min":4.2039999999999997,"5min":21.5,"15min":21.5},"last":0.78700000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":0.99199999999999999,"15min":0.80400000000000005},"min":{"1min":0.55700000000000005,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":8.7750000000000004,"5min":8.7750000000000004,"15min":8.7750000000000004},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.86599999999999999,"15min":0.81000000000000005},"min":{"1min":0.50800000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":7.9160000000000004,"5min":7.9160000000000004,"15min":7.9160000000000004},"last":0.77500000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.0009999999999999,"15min":0.873},"min":{"1min":0.53400000000000003,"5min":0.44,"15min":0.44},"max":{"1min":5.0650000000000004,"5min":21.457000000000001,"15min":21.457000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.694,"5min":1.083,"15min":0.93000000000000005},"min":{"1min":0.58699999999999997,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":8.7300000000000004,"5min":8.7300000000000004,"15min":8.7300000000000004},"last":0.80400000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":0.88100000000000001,"15min":0.78400000000000003},"min":{"1min":0.40699999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":8.8179999999999996,"5min":9.6679999999999993,"15min":9.6679999999999993},"last":0.73599999999999999},{"interface":"front","average":{"1min":2.0089999999999999,"5min":0.94899999999999995,"15min":0.81299999999999994},"min":{"1min":0.439,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":26.146999999999998,"5min":26.146999999999998,"15min":26.146999999999998},"last":0.69399999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":0.95499999999999996,"15min":0.81000000000000005},"min":{"1min":0.46300000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":6.6699999999999999,"5min":21.535,"15min":21.535},"last":0.70599999999999996},{"interface":"front","average":{"1min":2.649,"5min":1.2250000000000001,"15min":0.94599999999999995},"min":{"1min":0.46999999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":30.488,"5min":30.488,"15min":30.488},"last":0.84699999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":0.874,"15min":0.82599999999999996},"min":{"1min":0.51600000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":2.758,"5min":9.2240000000000002,"15min":9.2240000000000002},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.96699999999999997,"15min":0.877},"min":{"1min":0.44600000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.3520000000000003,"5min":21.574000000000002,"15min":21.574000000000002},"last":0.81899999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542200,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36556,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335284,"statfs":{"total":96636764160,"available":96599330816,"internally_reserved":0,"allocated":1245184,"data_stored":858586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.52600000000000002,"15min":0.48599999999999999},"min":{"1min":0.26100000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.2409999999999997,"5min":4.2409999999999997,"15min":4.2409999999999997},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55600000000000005,"15min":0.55300000000000005},"min":{"1min":0.35499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":2.0219999999999998,"5min":2.0219999999999998,"15min":2.0219999999999998},"last":0.53800000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.57599999999999996,"15min":0.57399999999999995},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.2330000000000001,"5min":2.2330000000000001,"15min":2.2330000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.53500000000000003,"15min":0.497},"min":{"1min":0.30099999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.6789999999999998,"5min":2.6789999999999998,"15min":2.6789999999999998},"last":0.56899999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.63900000000000001,"15min":0.59699999999999998},"min":{"1min":0.38900000000000001,"5min":0.308,"15min":0.308},"max":{"1min":4.274,"5min":4.274,"15min":4.274},"last":0.503},{"interface":"front","average":{"1min":1.0309999999999999,"5min":0.68200000000000005,"15min":0.63400000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":7.2309999999999999,"5min":7.2309999999999999,"15min":7.2309999999999999},"last":0.61799999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":2.339,"5min":1.0680000000000001,"15min":0.878},"min":{"1min":0.46600000000000003,"5min":0.38,"15min":0.38},"max":{"1min":22.814,"5min":22.814,"15min":22.814},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.516,"5min":0.92700000000000005,"15min":0.86499999999999999},"min":{"1min":0.54500000000000004,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":4.4800000000000004,"5min":4.4800000000000004,"15min":4.4800000000000004},"last":0.90800000000000003}]},{"osd":5,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":1.8600000000000001,"5min":0.97499999999999998,"15min":0.85799999999999998},"min":{"1min":0.495,"5min":0.441,"15min":0.441},"max":{"1min":12.428000000000001,"5min":12.428000000000001,"15min":12.428000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.915,"5min":1.018,"15min":0.91000000000000003},"min":{"1min":0.63800000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":13.064,"5min":13.064,"15min":13.064},"last":0.82799999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":2.8029999999999999,"5min":1.1699999999999999,"15min":0.92400000000000004},"min":{"1min":0.50800000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":31.059000000000001,"5min":31.059000000000001,"15min":31.059000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":3.6059999999999999,"5min":1.357,"15min":1.022},"min":{"1min":0.52000000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":40.731000000000002,"5min":40.731000000000002,"15min":40.731000000000002},"last":0.98399999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":1.657,"5min":0.96499999999999997,"15min":0.878},"min":{"1min":0.53800000000000003,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":14.032,"5min":14.032,"15min":14.032},"last":0.93200000000000005},{"interface":"front","average":{"1min":2.9569999999999999,"5min":1.254,"15min":0.995},"min":{"1min":0.58099999999999996,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":23.524000000000001,"5min":23.524000000000001,"15min":23.524000000000001},"last":0.878}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:09:16.098 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-01-15T18:09:16.090+0000 7fda52c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-15T18:09:18.829 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:09:18.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.978+0000 7f0ebf200640 1 Processor -- start 2024-01-15T18:09:18.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.978+0000 7f0ebf200640 1 -- start start 2024-01-15T18:09:18.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.978+0000 7f0ebf200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0eb806bb80 0x7f0eb80cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:18.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.978+0000 7f0ebf200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0eb80cddc0 0x7f0eb80b0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:18.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.978+0000 7f0ebf200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb80b2770 con 0x7f0eb80b2390 2024-01-15T18:09:18.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.978+0000 7f0ebf200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eb80ce270 con 0x7f0eb80cddc0 2024-01-15T18:09:18.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.978+0000 7f0ebf200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb806c060 con 0x7f0eb806bb80 2024-01-15T18:09:18.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.978+0000 7f0ebd200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0eb80cddc0 0x7f0eb80b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:18.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.978+0000 7f0eb7400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0eb806bb80 0x7f0eb80cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:18.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.978+0000 7f0ebd200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0eb80cddc0 0x7f0eb80b0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:59490/0 (socket says 172.21.15.27:59490) 2024-01-15T18:09:18.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.978+0000 7f0ebd200640 1 -- 172.21.15.27:0/3894688486 learned_addr learned my addr 172.21.15.27:0/3894688486 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:18.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.978+0000 7f0ebd200640 1 -- 172.21.15.27:0/3894688486 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0eb806bb80 msgr2=0x7f0eb80cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:18.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.978+0000 7f0ebd200640 1 --2- 172.21.15.27:0/3894688486 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0eb806bb80 0x7f0eb80cd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:18.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.978+0000 7f0ebd200640 1 -- 172.21.15.27:0/3894688486 >> v1:172.21.15.93:6789/0 conn(0x7f0eb80b2390 legacy=0x7f0eb80cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:18.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.978+0000 7f0ebd200640 1 -- 172.21.15.27:0/3894688486 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb806c340 con 0x7f0eb80cddc0 2024-01-15T18:09:18.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.982+0000 7f0ebd200640 1 --2- 172.21.15.27:0/3894688486 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0eb80cddc0 0x7f0eb80b0880 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0009e50 tx=0x7f0eb001c9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5618a2557e6ed2d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:18.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.982+0000 7f0eb6a00640 1 -- 172.21.15.27:0/3894688486 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb00a58a0 con 0x7f0eb80cddc0 2024-01-15T18:09:18.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.982+0000 7f0eb6a00640 1 -- 172.21.15.27:0/3894688486 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb00a5a00 con 0x7f0eb80cddc0 2024-01-15T18:09:18.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.982+0000 7f0eb6a00640 1 -- 172.21.15.27:0/3894688486 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb00a5d90 con 0x7f0eb80cddc0 2024-01-15T18:09:18.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.982+0000 7f0ebf200640 1 -- 172.21.15.27:0/3894688486 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0eb80cddc0 msgr2=0x7f0eb80b0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:18.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.982+0000 7f0ebf200640 1 --2- 172.21.15.27:0/3894688486 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0eb80cddc0 0x7f0eb80b0880 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0009e50 tx=0x7f0eb001c9e0 comp rx=0 tx=0).stop 2024-01-15T18:09:18.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.982+0000 7f0ebf200640 1 -- 172.21.15.27:0/3894688486 shutdown_connections 2024-01-15T18:09:18.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.982+0000 7f0ebf200640 1 --2- 172.21.15.27:0/3894688486 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0eb80cddc0 0x7f0eb80b0880 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:18.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.982+0000 7f0ebf200640 1 --2- 172.21.15.27:0/3894688486 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0eb806bb80 0x7f0eb80cd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:18.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.982+0000 7f0ebf200640 1 -- 172.21.15.27:0/3894688486 >> 172.21.15.27:0/3894688486 conn(0x7f0eb80b2a10 msgr2=0x7f0eb80b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:18.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.982+0000 7f0ebf200640 1 -- 172.21.15.27:0/3894688486 shutdown_connections 2024-01-15T18:09:18.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.982+0000 7f0ebf200640 1 -- 172.21.15.27:0/3894688486 wait complete. 2024-01-15T18:09:18.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.986+0000 7f0ebf200640 1 Processor -- start 2024-01-15T18:09:18.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.986+0000 7f0ebf200640 1 -- start start 2024-01-15T18:09:18.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.986+0000 7f0ebf200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0eb806bb80 0x7f0eb8158310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:18.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.986+0000 7f0ebf200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0eb80b2390 0x7f0eb80c6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:18.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.986+0000 7f0ebf200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0eb80cddc0 0x7f0eb80c49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:18.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.986+0000 7f0ebf200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb8150ce0 con 0x7f0eb80b2390 2024-01-15T18:09:18.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.986+0000 7f0ebf200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb80c7000 con 0x7f0eb806bb80 2024-01-15T18:09:18.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.986+0000 7f0ebf200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb80c8550 con 0x7f0eb80cddc0 2024-01-15T18:09:18.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.986+0000 7f0eb7400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0eb80b2390 0x7f0eb80c6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:18.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.986+0000 7f0eb7400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0eb80b2390 0x7f0eb80c6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34414/0 (socket says 172.21.15.27:34414) 2024-01-15T18:09:18.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.986+0000 7f0eb7400640 1 -- 172.21.15.27:0/2247342047 learned_addr learned my addr 172.21.15.27:0/2247342047 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:18.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.986+0000 7f0eb7400640 1 -- 172.21.15.27:0/2247342047 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0eb80cddc0 msgr2=0x7f0eb80c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:18.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.986+0000 7f0eb7400640 1 --2- 172.21.15.27:0/2247342047 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0eb80cddc0 0x7f0eb80c49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:18.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.986+0000 7f0eb7400640 1 -- 172.21.15.27:0/2247342047 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0eb806bb80 msgr2=0x7f0eb8158310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:18.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.986+0000 7f0eb7400640 1 --2- 172.21.15.27:0/2247342047 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0eb806bb80 0x7f0eb8158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:18.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.986+0000 7f0eb7400640 1 -- 172.21.15.27:0/2247342047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb0009b30 con 0x7f0eb80b2390 2024-01-15T18:09:18.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.986+0000 7f0eb7400640 1 --2- 172.21.15.27:0/2247342047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0eb80b2390 0x7f0eb80c6ac0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f0ea000e930 tx=0x7f0ea000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:18.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.986+0000 7f0eafe00640 1 -- 172.21.15.27:0/2247342047 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea0009690 con 0x7f0eb80b2390 2024-01-15T18:09:18.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.986+0000 7f0eafe00640 1 -- 172.21.15.27:0/2247342047 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea000ca00 con 0x7f0eb80b2390 2024-01-15T18:09:18.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.986+0000 7f0ebf200640 1 -- 172.21.15.27:0/2247342047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb80c8830 con 0x7f0eb80b2390 2024-01-15T18:09:18.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.986+0000 7f0eafe00640 1 -- 172.21.15.27:0/2247342047 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea000cd70 con 0x7f0eb80b2390 2024-01-15T18:09:18.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.986+0000 7f0ebf200640 1 -- 172.21.15.27:0/2247342047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb80c8df0 con 0x7f0eb80b2390 2024-01-15T18:09:18.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.990+0000 7f0ebf200640 1 -- 172.21.15.27:0/2247342047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eb80d07c0 con 0x7f0eb80b2390 2024-01-15T18:09:18.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.990+0000 7f0eafe00640 1 -- 172.21.15.27:0/2247342047 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f0ea0013470 con 0x7f0eb80b2390 2024-01-15T18:09:18.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.990+0000 7f0eafe00640 1 --2- 172.21.15.27:0/2247342047 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0e90066e90 0x7f0e90069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:18.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.990+0000 7f0eafe00640 1 -- 172.21.15.27:0/2247342047 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f0ea0085230 con 0x7f0eb80b2390 2024-01-15T18:09:18.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.990+0000 7f0eb7e00640 1 --2- 172.21.15.27:0/2247342047 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0e90066e90 0x7f0e90069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:18.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.990+0000 7f0eb7e00640 1 --2- 172.21.15.27:0/2247342047 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0e90066e90 0x7f0e90069350 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8000f70 tx=0x7f0ea80079c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:19.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:18.994+0000 7f0eafe00640 1 -- 172.21.15.27:0/2247342047 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0ea0051be0 con 0x7f0eb80b2390 2024-01-15T18:09:19.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.150+0000 7f0ebf200640 1 -- 172.21.15.27:0/2247342047 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0eb80b1b50 con 0x7f0e90066e90 2024-01-15T18:09:19.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.158+0000 7f0eafe00640 1 -- 172.21.15.27:0/2247342047 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288428 (secure 0 0 0) 0x7f0eb80b1b50 con 0x7f0e90066e90 2024-01-15T18:09:19.166 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:09:19.167 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:09:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.162+0000 7f0ebf200640 1 -- 172.21.15.27:0/2247342047 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0e90066e90 msgr2=0x7f0e90069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.162+0000 7f0ebf200640 1 --2- 172.21.15.27:0/2247342047 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0e90066e90 0x7f0e90069350 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8000f70 tx=0x7f0ea80079c0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.162+0000 7f0ebf200640 1 -- 172.21.15.27:0/2247342047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0eb80b2390 msgr2=0x7f0eb80c6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.162+0000 7f0ebf200640 1 --2- 172.21.15.27:0/2247342047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0eb80b2390 0x7f0eb80c6ac0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f0ea000e930 tx=0x7f0ea000edf0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.162+0000 7f0ebf200640 1 -- 172.21.15.27:0/2247342047 shutdown_connections 2024-01-15T18:09:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.162+0000 7f0ebf200640 1 --2- 172.21.15.27:0/2247342047 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0e90066e90 0x7f0e90069350 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.162+0000 7f0ebf200640 1 --2- 172.21.15.27:0/2247342047 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0eb80cddc0 0x7f0eb80c49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.162+0000 7f0ebf200640 1 --2- 172.21.15.27:0/2247342047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0eb80b2390 0x7f0eb80c6ac0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.162+0000 7f0ebf200640 1 --2- 172.21.15.27:0/2247342047 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0eb806bb80 0x7f0eb8158310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.162+0000 7f0ebf200640 1 -- 172.21.15.27:0/2247342047 >> 172.21.15.27:0/2247342047 conn(0x7f0eb80b2a10 msgr2=0x7f0eb806d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.162+0000 7f0ebf200640 1 -- 172.21.15.27:0/2247342047 shutdown_connections 2024-01-15T18:09:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.162+0000 7f0ebf200640 1 -- 172.21.15.27:0/2247342047 wait complete. 2024-01-15T18:09:19.187 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-01-15T18:09:18.207623+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":289340,"kb_used_data":6600,"kb_used_omap":12,"kb_used_meta":282611,"kb_avail":754685380,"statfs":{"total":773094113280,"available":772797829120,"internally_reserved":0,"allocated":6758400,"data_stored":3486708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":289394224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.075734"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.028238+0000","last_change":"2024-01-15T18:09:10.164426+0000","last_active":"2024-01-15T18:09:12.028238+0000","last_peered":"2024-01-15T18:09:12.028238+0000","last_clean":"2024-01-15T18:09:12.028238+0000","last_became_active":"2024-01-15T18:09:10.164120+0000","last_became_peered":"2024-01-15T18:09:10.164120+0000","last_unstale":"2024-01-15T18:09:12.028238+0000","last_undegraded":"2024-01-15T18:09:12.028238+0000","last_fullsized":"2024-01-15T18:09:12.028238+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:01:56.180664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.543294+0000","last_change":"2024-01-15T18:09:10.149693+0000","last_active":"2024-01-15T18:09:11.543294+0000","last_peered":"2024-01-15T18:09:11.543294+0000","last_clean":"2024-01-15T18:09:11.543294+0000","last_became_active":"2024-01-15T18:09:10.149380+0000","last_became_peered":"2024-01-15T18:09:10.149380+0000","last_unstale":"2024-01-15T18:09:11.543294+0000","last_undegraded":"2024-01-15T18:09:11.543294+0000","last_fullsized":"2024-01-15T18:09:11.543294+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:01:37.209089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.043822+0000","last_change":"2024-01-15T18:09:10.172281+0000","last_active":"2024-01-15T18:09:12.043822+0000","last_peered":"2024-01-15T18:09:12.043822+0000","last_clean":"2024-01-15T18:09:12.043822+0000","last_became_active":"2024-01-15T18:09:10.171520+0000","last_became_peered":"2024-01-15T18:09:10.171520+0000","last_unstale":"2024-01-15T18:09:12.043822+0000","last_undegraded":"2024-01-15T18:09:12.043822+0000","last_fullsized":"2024-01-15T18:09:12.043822+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:53.373020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155356+0000","last_change":"2024-01-15T18:09:10.154186+0000","last_active":"2024-01-15T18:09:11.155356+0000","last_peered":"2024-01-15T18:09:11.155356+0000","last_clean":"2024-01-15T18:09:11.155356+0000","last_became_active":"2024-01-15T18:09:10.153961+0000","last_became_peered":"2024-01-15T18:09:10.153961+0000","last_unstale":"2024-01-15T18:09:11.155356+0000","last_undegraded":"2024-01-15T18:09:11.155356+0000","last_fullsized":"2024-01-15T18:09:11.155356+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:44:11.424905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.027944+0000","last_change":"2024-01-15T18:09:10.149056+0000","last_active":"2024-01-15T18:09:12.027944+0000","last_peered":"2024-01-15T18:09:12.027944+0000","last_clean":"2024-01-15T18:09:12.027944+0000","last_became_active":"2024-01-15T18:09:10.148756+0000","last_became_peered":"2024-01-15T18:09:10.148756+0000","last_unstale":"2024-01-15T18:09:12.027944+0000","last_undegraded":"2024-01-15T18:09:12.027944+0000","last_fullsized":"2024-01-15T18:09:12.027944+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:13:41.479011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.043391+0000","last_change":"2024-01-15T18:09:10.163323+0000","last_active":"2024-01-15T18:09:12.043391+0000","last_peered":"2024-01-15T18:09:12.043391+0000","last_clean":"2024-01-15T18:09:12.043391+0000","last_became_active":"2024-01-15T18:09:10.163090+0000","last_became_peered":"2024-01-15T18:09:10.163090+0000","last_unstale":"2024-01-15T18:09:12.043391+0000","last_undegraded":"2024-01-15T18:09:12.043391+0000","last_fullsized":"2024-01-15T18:09:12.043391+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:37:28.924092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.154981+0000","last_change":"2024-01-15T18:09:10.147574+0000","last_active":"2024-01-15T18:09:11.154981+0000","last_peered":"2024-01-15T18:09:11.154981+0000","last_clean":"2024-01-15T18:09:11.154981+0000","last_became_active":"2024-01-15T18:09:10.147288+0000","last_became_peered":"2024-01-15T18:09:10.147288+0000","last_unstale":"2024-01-15T18:09:11.154981+0000","last_undegraded":"2024-01-15T18:09:11.154981+0000","last_fullsized":"2024-01-15T18:09:11.154981+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:58:26.855862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.542816+0000","last_change":"2024-01-15T18:09:10.149706+0000","last_active":"2024-01-15T18:09:11.542816+0000","last_peered":"2024-01-15T18:09:11.542816+0000","last_clean":"2024-01-15T18:09:11.542816+0000","last_became_active":"2024-01-15T18:09:10.149353+0000","last_became_peered":"2024-01-15T18:09:10.149353+0000","last_unstale":"2024-01-15T18:09:11.542816+0000","last_undegraded":"2024-01-15T18:09:11.542816+0000","last_fullsized":"2024-01-15T18:09:11.542816+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:12:26.323498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155719+0000","last_change":"2024-01-15T18:09:10.162835+0000","last_active":"2024-01-15T18:09:11.155719+0000","last_peered":"2024-01-15T18:09:11.155719+0000","last_clean":"2024-01-15T18:09:11.155719+0000","last_became_active":"2024-01-15T18:09:10.162459+0000","last_became_peered":"2024-01-15T18:09:10.162459+0000","last_unstale":"2024-01-15T18:09:11.155719+0000","last_undegraded":"2024-01-15T18:09:11.155719+0000","last_fullsized":"2024-01-15T18:09:11.155719+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:07:31.195306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.156091+0000","last_change":"2024-01-15T18:09:10.151168+0000","last_active":"2024-01-15T18:09:11.156091+0000","last_peered":"2024-01-15T18:09:11.156091+0000","last_clean":"2024-01-15T18:09:11.156091+0000","last_became_active":"2024-01-15T18:09:10.150812+0000","last_became_peered":"2024-01-15T18:09:10.150812+0000","last_unstale":"2024-01-15T18:09:11.156091+0000","last_undegraded":"2024-01-15T18:09:11.156091+0000","last_fullsized":"2024-01-15T18:09:11.156091+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:15:39.864433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.156230+0000","last_change":"2024-01-15T18:09:10.148209+0000","last_active":"2024-01-15T18:09:11.156230+0000","last_peered":"2024-01-15T18:09:11.156230+0000","last_clean":"2024-01-15T18:09:11.156230+0000","last_became_active":"2024-01-15T18:09:10.147897+0000","last_became_peered":"2024-01-15T18:09:10.147897+0000","last_unstale":"2024-01-15T18:09:11.156230+0000","last_undegraded":"2024-01-15T18:09:11.156230+0000","last_fullsized":"2024-01-15T18:09:11.156230+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:13:27.981827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155548+0000","last_change":"2024-01-15T18:09:10.153400+0000","last_active":"2024-01-15T18:09:11.155548+0000","last_peered":"2024-01-15T18:09:11.155548+0000","last_clean":"2024-01-15T18:09:11.155548+0000","last_became_active":"2024-01-15T18:09:10.152222+0000","last_became_peered":"2024-01-15T18:09:10.152222+0000","last_unstale":"2024-01-15T18:09:11.155548+0000","last_undegraded":"2024-01-15T18:09:11.155548+0000","last_fullsized":"2024-01-15T18:09:11.155548+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:03.126844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.044660+0000","last_change":"2024-01-15T18:09:10.160925+0000","last_active":"2024-01-15T18:09:12.044660+0000","last_peered":"2024-01-15T18:09:12.044660+0000","last_clean":"2024-01-15T18:09:12.044660+0000","last_became_active":"2024-01-15T18:09:10.160620+0000","last_became_peered":"2024-01-15T18:09:10.160620+0000","last_unstale":"2024-01-15T18:09:12.044660+0000","last_undegraded":"2024-01-15T18:09:12.044660+0000","last_fullsized":"2024-01-15T18:09:12.044660+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:41:16.078483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155367+0000","last_change":"2024-01-15T18:09:10.154922+0000","last_active":"2024-01-15T18:09:11.155367+0000","last_peered":"2024-01-15T18:09:11.155367+0000","last_clean":"2024-01-15T18:09:11.155367+0000","last_became_active":"2024-01-15T18:09:10.154594+0000","last_became_peered":"2024-01-15T18:09:10.154594+0000","last_unstale":"2024-01-15T18:09:11.155367+0000","last_undegraded":"2024-01-15T18:09:11.155367+0000","last_fullsized":"2024-01-15T18:09:11.155367+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:27:59.969234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155240+0000","last_change":"2024-01-15T18:09:10.143163+0000","last_active":"2024-01-15T18:09:11.155240+0000","last_peered":"2024-01-15T18:09:11.155240+0000","last_clean":"2024-01-15T18:09:11.155240+0000","last_became_active":"2024-01-15T18:09:10.142874+0000","last_became_peered":"2024-01-15T18:09:10.142874+0000","last_unstale":"2024-01-15T18:09:11.155240+0000","last_undegraded":"2024-01-15T18:09:11.155240+0000","last_fullsized":"2024-01-15T18:09:11.155240+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:22:35.521714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.043326+0000","last_change":"2024-01-15T18:09:10.168650+0000","last_active":"2024-01-15T18:09:12.043326+0000","last_peered":"2024-01-15T18:09:12.043326+0000","last_clean":"2024-01-15T18:09:12.043326+0000","last_became_active":"2024-01-15T18:09:10.168331+0000","last_became_peered":"2024-01-15T18:09:10.168331+0000","last_unstale":"2024-01-15T18:09:12.043326+0000","last_undegraded":"2024-01-15T18:09:12.043326+0000","last_fullsized":"2024-01-15T18:09:12.043326+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:10:10.160689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.543293+0000","last_change":"2024-01-15T18:09:10.156152+0000","last_active":"2024-01-15T18:09:11.543293+0000","last_peered":"2024-01-15T18:09:11.543293+0000","last_clean":"2024-01-15T18:09:11.543293+0000","last_became_active":"2024-01-15T18:09:10.155849+0000","last_became_peered":"2024-01-15T18:09:10.155849+0000","last_unstale":"2024-01-15T18:09:11.543293+0000","last_undegraded":"2024-01-15T18:09:11.543293+0000","last_fullsized":"2024-01-15T18:09:11.543293+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:47:45.103470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.043838+0000","last_change":"2024-01-15T18:09:10.168066+0000","last_active":"2024-01-15T18:09:12.043838+0000","last_peered":"2024-01-15T18:09:12.043838+0000","last_clean":"2024-01-15T18:09:12.043838+0000","last_became_active":"2024-01-15T18:09:10.167691+0000","last_became_peered":"2024-01-15T18:09:10.167691+0000","last_unstale":"2024-01-15T18:09:12.043838+0000","last_undegraded":"2024-01-15T18:09:12.043838+0000","last_fullsized":"2024-01-15T18:09:12.043838+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:03.594926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.028127+0000","last_change":"2024-01-15T18:09:10.163790+0000","last_active":"2024-01-15T18:09:12.028127+0000","last_peered":"2024-01-15T18:09:12.028127+0000","last_clean":"2024-01-15T18:09:12.028127+0000","last_became_active":"2024-01-15T18:09:10.163509+0000","last_became_peered":"2024-01-15T18:09:10.163509+0000","last_unstale":"2024-01-15T18:09:12.028127+0000","last_undegraded":"2024-01-15T18:09:12.028127+0000","last_fullsized":"2024-01-15T18:09:12.028127+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:49:28.000188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.543143+0000","last_change":"2024-01-15T18:09:10.161978+0000","last_active":"2024-01-15T18:09:11.543143+0000","last_peered":"2024-01-15T18:09:11.543143+0000","last_clean":"2024-01-15T18:09:11.543143+0000","last_became_active":"2024-01-15T18:09:10.161717+0000","last_became_peered":"2024-01-15T18:09:10.161717+0000","last_unstale":"2024-01-15T18:09:11.543143+0000","last_undegraded":"2024-01-15T18:09:11.543143+0000","last_fullsized":"2024-01-15T18:09:11.543143+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:48:10.797360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.043649+0000","last_change":"2024-01-15T18:09:10.159447+0000","last_active":"2024-01-15T18:09:12.043649+0000","last_peered":"2024-01-15T18:09:12.043649+0000","last_clean":"2024-01-15T18:09:12.043649+0000","last_became_active":"2024-01-15T18:09:10.159055+0000","last_became_peered":"2024-01-15T18:09:10.159055+0000","last_unstale":"2024-01-15T18:09:12.043649+0000","last_undegraded":"2024-01-15T18:09:12.043649+0000","last_fullsized":"2024-01-15T18:09:12.043649+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:42:48.135006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.542856+0000","last_change":"2024-01-15T18:09:10.157352+0000","last_active":"2024-01-15T18:09:11.542856+0000","last_peered":"2024-01-15T18:09:11.542856+0000","last_clean":"2024-01-15T18:09:11.542856+0000","last_became_active":"2024-01-15T18:09:10.155991+0000","last_became_peered":"2024-01-15T18:09:10.155991+0000","last_unstale":"2024-01-15T18:09:11.542856+0000","last_undegraded":"2024-01-15T18:09:11.542856+0000","last_fullsized":"2024-01-15T18:09:11.542856+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:05:50.347839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.043462+0000","last_change":"2024-01-15T18:09:10.160338+0000","last_active":"2024-01-15T18:09:12.043462+0000","last_peered":"2024-01-15T18:09:12.043462+0000","last_clean":"2024-01-15T18:09:12.043462+0000","last_became_active":"2024-01-15T18:09:10.159325+0000","last_became_peered":"2024-01-15T18:09:10.159325+0000","last_unstale":"2024-01-15T18:09:12.043462+0000","last_undegraded":"2024-01-15T18:09:12.043462+0000","last_fullsized":"2024-01-15T18:09:12.043462+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:57:26.029056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.043837+0000","last_change":"2024-01-15T18:09:10.167327+0000","last_active":"2024-01-15T18:09:12.043837+0000","last_peered":"2024-01-15T18:09:12.043837+0000","last_clean":"2024-01-15T18:09:12.043837+0000","last_became_active":"2024-01-15T18:09:10.167008+0000","last_became_peered":"2024-01-15T18:09:10.167008+0000","last_unstale":"2024-01-15T18:09:12.043837+0000","last_undegraded":"2024-01-15T18:09:12.043837+0000","last_fullsized":"2024-01-15T18:09:12.043837+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:38.005845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.028276+0000","last_change":"2024-01-15T18:09:10.165962+0000","last_active":"2024-01-15T18:09:12.028276+0000","last_peered":"2024-01-15T18:09:12.028276+0000","last_clean":"2024-01-15T18:09:12.028276+0000","last_became_active":"2024-01-15T18:09:10.165689+0000","last_became_peered":"2024-01-15T18:09:10.165689+0000","last_unstale":"2024-01-15T18:09:12.028276+0000","last_undegraded":"2024-01-15T18:09:12.028276+0000","last_fullsized":"2024-01-15T18:09:12.028276+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:16:46.421655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155565+0000","last_change":"2024-01-15T18:09:10.147816+0000","last_active":"2024-01-15T18:09:11.155565+0000","last_peered":"2024-01-15T18:09:11.155565+0000","last_clean":"2024-01-15T18:09:11.155565+0000","last_became_active":"2024-01-15T18:09:10.147487+0000","last_became_peered":"2024-01-15T18:09:10.147487+0000","last_unstale":"2024-01-15T18:09:11.155565+0000","last_undegraded":"2024-01-15T18:09:11.155565+0000","last_fullsized":"2024-01-15T18:09:11.155565+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:40:51.237238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.043730+0000","last_change":"2024-01-15T18:09:10.167371+0000","last_active":"2024-01-15T18:09:12.043730+0000","last_peered":"2024-01-15T18:09:12.043730+0000","last_clean":"2024-01-15T18:09:12.043730+0000","last_became_active":"2024-01-15T18:09:10.167043+0000","last_became_peered":"2024-01-15T18:09:10.167043+0000","last_unstale":"2024-01-15T18:09:12.043730+0000","last_undegraded":"2024-01-15T18:09:12.043730+0000","last_fullsized":"2024-01-15T18:09:12.043730+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:41:56.291009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.028113+0000","last_change":"2024-01-15T18:09:10.149120+0000","last_active":"2024-01-15T18:09:12.028113+0000","last_peered":"2024-01-15T18:09:12.028113+0000","last_clean":"2024-01-15T18:09:12.028113+0000","last_became_active":"2024-01-15T18:09:10.148675+0000","last_became_peered":"2024-01-15T18:09:10.148675+0000","last_unstale":"2024-01-15T18:09:12.028113+0000","last_undegraded":"2024-01-15T18:09:12.028113+0000","last_fullsized":"2024-01-15T18:09:12.028113+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:28:06.451548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.543296+0000","last_change":"2024-01-15T18:09:10.161188+0000","last_active":"2024-01-15T18:09:11.543296+0000","last_peered":"2024-01-15T18:09:11.543296+0000","last_clean":"2024-01-15T18:09:11.543296+0000","last_became_active":"2024-01-15T18:09:10.151645+0000","last_became_peered":"2024-01-15T18:09:10.151645+0000","last_unstale":"2024-01-15T18:09:11.543296+0000","last_undegraded":"2024-01-15T18:09:11.543296+0000","last_fullsized":"2024-01-15T18:09:11.543296+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:34:36.902366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.043637+0000","last_change":"2024-01-15T18:09:10.167895+0000","last_active":"2024-01-15T18:09:12.043637+0000","last_peered":"2024-01-15T18:09:12.043637+0000","last_clean":"2024-01-15T18:09:12.043637+0000","last_became_active":"2024-01-15T18:09:10.167639+0000","last_became_peered":"2024-01-15T18:09:10.167639+0000","last_unstale":"2024-01-15T18:09:12.043637+0000","last_undegraded":"2024-01-15T18:09:12.043637+0000","last_fullsized":"2024-01-15T18:09:12.043637+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:44:09.045102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.027910+0000","last_change":"2024-01-15T18:09:10.148253+0000","last_active":"2024-01-15T18:09:12.027910+0000","last_peered":"2024-01-15T18:09:12.027910+0000","last_clean":"2024-01-15T18:09:12.027910+0000","last_became_active":"2024-01-15T18:09:10.147941+0000","last_became_peered":"2024-01-15T18:09:10.147941+0000","last_unstale":"2024-01-15T18:09:12.027910+0000","last_undegraded":"2024-01-15T18:09:12.027910+0000","last_fullsized":"2024-01-15T18:09:12.027910+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:19:36.646332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.043422+0000","last_change":"2024-01-15T18:09:10.174575+0000","last_active":"2024-01-15T18:09:12.043422+0000","last_peered":"2024-01-15T18:09:12.043422+0000","last_clean":"2024-01-15T18:09:12.043422+0000","last_became_active":"2024-01-15T18:09:10.174278+0000","last_became_peered":"2024-01-15T18:09:10.174278+0000","last_unstale":"2024-01-15T18:09:12.043422+0000","last_undegraded":"2024-01-15T18:09:12.043422+0000","last_fullsized":"2024-01-15T18:09:12.043422+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:57:04.284081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.043843+0000","last_change":"2024-01-15T18:09:10.160188+0000","last_active":"2024-01-15T18:09:12.043843+0000","last_peered":"2024-01-15T18:09:12.043843+0000","last_clean":"2024-01-15T18:09:12.043843+0000","last_became_active":"2024-01-15T18:09:10.159846+0000","last_became_peered":"2024-01-15T18:09:10.159846+0000","last_unstale":"2024-01-15T18:09:12.043843+0000","last_undegraded":"2024-01-15T18:09:12.043843+0000","last_fullsized":"2024-01-15T18:09:12.043843+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:45:32.407323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"57'4","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.175122+0000","last_change":"2024-01-15T18:09:09.152802+0000","last_active":"2024-01-15T18:09:10.175122+0000","last_peered":"2024-01-15T18:09:10.175122+0000","last_clean":"2024-01-15T18:09:10.175122+0000","last_became_active":"2024-01-15T18:09:09.152520+0000","last_became_peered":"2024-01-15T18:09:09.152520+0000","last_unstale":"2024-01-15T18:09:10.175122+0000","last_undegraded":"2024-01-15T18:09:10.175122+0000","last_fullsized":"2024-01-15T18:09:10.175122+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:54:56.016726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.156148+0000","last_change":"2024-01-15T18:09:10.161734+0000","last_active":"2024-01-15T18:09:11.156148+0000","last_peered":"2024-01-15T18:09:11.156148+0000","last_clean":"2024-01-15T18:09:11.156148+0000","last_became_active":"2024-01-15T18:09:10.161466+0000","last_became_peered":"2024-01-15T18:09:10.161466+0000","last_unstale":"2024-01-15T18:09:11.156148+0000","last_undegraded":"2024-01-15T18:09:11.156148+0000","last_fullsized":"2024-01-15T18:09:11.156148+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:37:29.585671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.214126+0000","last_change":"2024-01-15T18:09:09.125625+0000","last_active":"2024-01-15T18:09:10.214126+0000","last_peered":"2024-01-15T18:09:10.214126+0000","last_clean":"2024-01-15T18:09:10.214126+0000","last_became_active":"2024-01-15T18:09:09.125373+0000","last_became_peered":"2024-01-15T18:09:09.125373+0000","last_unstale":"2024-01-15T18:09:10.214126+0000","last_undegraded":"2024-01-15T18:09:10.214126+0000","last_fullsized":"2024-01-15T18:09:10.214126+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:14:59.883458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.043579+0000","last_change":"2024-01-15T18:09:10.168487+0000","last_active":"2024-01-15T18:09:12.043579+0000","last_peered":"2024-01-15T18:09:12.043579+0000","last_clean":"2024-01-15T18:09:12.043579+0000","last_became_active":"2024-01-15T18:09:10.167828+0000","last_became_peered":"2024-01-15T18:09:10.167828+0000","last_unstale":"2024-01-15T18:09:12.043579+0000","last_undegraded":"2024-01-15T18:09:12.043579+0000","last_fullsized":"2024-01-15T18:09:12.043579+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:07:18.547932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.180345+0000","last_change":"2024-01-15T18:09:09.153007+0000","last_active":"2024-01-15T18:09:10.180345+0000","last_peered":"2024-01-15T18:09:10.180345+0000","last_clean":"2024-01-15T18:09:10.180345+0000","last_became_active":"2024-01-15T18:09:09.152405+0000","last_became_peered":"2024-01-15T18:09:09.152405+0000","last_unstale":"2024-01-15T18:09:10.180345+0000","last_undegraded":"2024-01-15T18:09:10.180345+0000","last_fullsized":"2024-01-15T18:09:10.180345+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:36:46.142785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.043694+0000","last_change":"2024-01-15T18:09:10.160090+0000","last_active":"2024-01-15T18:09:12.043694+0000","last_peered":"2024-01-15T18:09:12.043694+0000","last_clean":"2024-01-15T18:09:12.043694+0000","last_became_active":"2024-01-15T18:09:10.159741+0000","last_became_peered":"2024-01-15T18:09:10.159741+0000","last_unstale":"2024-01-15T18:09:12.043694+0000","last_undegraded":"2024-01-15T18:09:12.043694+0000","last_fullsized":"2024-01-15T18:09:12.043694+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:40:51.237238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.176876+0000","last_change":"2024-01-15T18:09:09.138209+0000","last_active":"2024-01-15T18:09:10.176876+0000","last_peered":"2024-01-15T18:09:10.176876+0000","last_clean":"2024-01-15T18:09:10.176876+0000","last_became_active":"2024-01-15T18:09:09.137921+0000","last_became_peered":"2024-01-15T18:09:09.137921+0000","last_unstale":"2024-01-15T18:09:10.176876+0000","last_undegraded":"2024-01-15T18:09:10.176876+0000","last_fullsized":"2024-01-15T18:09:10.176876+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:54:56.016726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.542887+0000","last_change":"2024-01-15T18:09:10.156211+0000","last_active":"2024-01-15T18:09:11.542887+0000","last_peered":"2024-01-15T18:09:11.542887+0000","last_clean":"2024-01-15T18:09:11.542887+0000","last_became_active":"2024-01-15T18:09:10.155914+0000","last_became_peered":"2024-01-15T18:09:10.155914+0000","last_unstale":"2024-01-15T18:09:11.542887+0000","last_undegraded":"2024-01-15T18:09:11.542887+0000","last_fullsized":"2024-01-15T18:09:11.542887+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:48:43.265793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.240194+0000","last_change":"2024-01-15T18:09:09.139963+0000","last_active":"2024-01-15T18:09:10.240194+0000","last_peered":"2024-01-15T18:09:10.240194+0000","last_clean":"2024-01-15T18:09:10.240194+0000","last_became_active":"2024-01-15T18:09:09.139603+0000","last_became_peered":"2024-01-15T18:09:09.139603+0000","last_unstale":"2024-01-15T18:09:10.240194+0000","last_undegraded":"2024-01-15T18:09:10.240194+0000","last_fullsized":"2024-01-15T18:09:10.240194+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:14:01.036199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.543133+0000","last_change":"2024-01-15T18:09:10.165154+0000","last_active":"2024-01-15T18:09:11.543133+0000","last_peered":"2024-01-15T18:09:11.543133+0000","last_clean":"2024-01-15T18:09:11.543133+0000","last_became_active":"2024-01-15T18:09:10.164900+0000","last_became_peered":"2024-01-15T18:09:10.164900+0000","last_unstale":"2024-01-15T18:09:11.543133+0000","last_undegraded":"2024-01-15T18:09:11.543133+0000","last_fullsized":"2024-01-15T18:09:11.543133+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:58:27.357593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.175916+0000","last_change":"2024-01-15T18:09:09.147874+0000","last_active":"2024-01-15T18:09:10.175916+0000","last_peered":"2024-01-15T18:09:10.175916+0000","last_clean":"2024-01-15T18:09:10.175916+0000","last_became_active":"2024-01-15T18:09:09.147666+0000","last_became_peered":"2024-01-15T18:09:09.147666+0000","last_unstale":"2024-01-15T18:09:10.175916+0000","last_undegraded":"2024-01-15T18:09:10.175916+0000","last_fullsized":"2024-01-15T18:09:10.175916+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:00:53.442320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.043994+0000","last_change":"2024-01-15T18:09:10.154886+0000","last_active":"2024-01-15T18:09:12.043994+0000","last_peered":"2024-01-15T18:09:12.043994+0000","last_clean":"2024-01-15T18:09:12.043994+0000","last_became_active":"2024-01-15T18:09:10.154594+0000","last_became_peered":"2024-01-15T18:09:10.154594+0000","last_unstale":"2024-01-15T18:09:12.043994+0000","last_undegraded":"2024-01-15T18:09:12.043994+0000","last_fullsized":"2024-01-15T18:09:12.043994+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:00:49.080998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.137679+0000","last_change":"2024-01-15T18:09:09.140948+0000","last_active":"2024-01-15T18:09:10.137679+0000","last_peered":"2024-01-15T18:09:10.137679+0000","last_clean":"2024-01-15T18:09:10.137679+0000","last_became_active":"2024-01-15T18:09:09.140010+0000","last_became_peered":"2024-01-15T18:09:09.140010+0000","last_unstale":"2024-01-15T18:09:10.137679+0000","last_undegraded":"2024-01-15T18:09:10.137679+0000","last_fullsized":"2024-01-15T18:09:10.137679+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:39:25.682654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.542882+0000","last_change":"2024-01-15T18:09:10.160588+0000","last_active":"2024-01-15T18:09:11.542882+0000","last_peered":"2024-01-15T18:09:11.542882+0000","last_clean":"2024-01-15T18:09:11.542882+0000","last_became_active":"2024-01-15T18:09:10.157197+0000","last_became_peered":"2024-01-15T18:09:10.157197+0000","last_unstale":"2024-01-15T18:09:11.542882+0000","last_undegraded":"2024-01-15T18:09:11.542882+0000","last_fullsized":"2024-01-15T18:09:11.542882+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:36:54.863597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.240093+0000","last_change":"2024-01-15T18:09:09.131170+0000","last_active":"2024-01-15T18:09:10.240093+0000","last_peered":"2024-01-15T18:09:10.240093+0000","last_clean":"2024-01-15T18:09:10.240093+0000","last_became_active":"2024-01-15T18:09:09.130789+0000","last_became_peered":"2024-01-15T18:09:09.130789+0000","last_unstale":"2024-01-15T18:09:10.240093+0000","last_undegraded":"2024-01-15T18:09:10.240093+0000","last_fullsized":"2024-01-15T18:09:10.240093+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:13:26.979622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155742+0000","last_change":"2024-01-15T18:09:10.154127+0000","last_active":"2024-01-15T18:09:11.155742+0000","last_peered":"2024-01-15T18:09:11.155742+0000","last_clean":"2024-01-15T18:09:11.155742+0000","last_became_active":"2024-01-15T18:09:10.153699+0000","last_became_peered":"2024-01-15T18:09:10.153699+0000","last_unstale":"2024-01-15T18:09:11.155742+0000","last_undegraded":"2024-01-15T18:09:11.155742+0000","last_fullsized":"2024-01-15T18:09:11.155742+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:53:41.639339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.222249+0000","last_change":"2024-01-15T18:09:09.145070+0000","last_active":"2024-01-15T18:09:10.222249+0000","last_peered":"2024-01-15T18:09:10.222249+0000","last_clean":"2024-01-15T18:09:10.222249+0000","last_became_active":"2024-01-15T18:09:09.144808+0000","last_became_peered":"2024-01-15T18:09:09.144808+0000","last_unstale":"2024-01-15T18:09:10.222249+0000","last_undegraded":"2024-01-15T18:09:10.222249+0000","last_fullsized":"2024-01-15T18:09:10.222249+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:07:17.174892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.044276+0000","last_change":"2024-01-15T18:09:10.173359+0000","last_active":"2024-01-15T18:09:12.044276+0000","last_peered":"2024-01-15T18:09:12.044276+0000","last_clean":"2024-01-15T18:09:12.044276+0000","last_became_active":"2024-01-15T18:09:10.173020+0000","last_became_peered":"2024-01-15T18:09:10.173020+0000","last_unstale":"2024-01-15T18:09:12.044276+0000","last_undegraded":"2024-01-15T18:09:12.044276+0000","last_fullsized":"2024-01-15T18:09:12.044276+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:37:29.872153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.214135+0000","last_change":"2024-01-15T18:09:09.142076+0000","last_active":"2024-01-15T18:09:10.214135+0000","last_peered":"2024-01-15T18:09:10.214135+0000","last_clean":"2024-01-15T18:09:10.214135+0000","last_became_active":"2024-01-15T18:09:09.141747+0000","last_became_peered":"2024-01-15T18:09:09.141747+0000","last_unstale":"2024-01-15T18:09:10.214135+0000","last_undegraded":"2024-01-15T18:09:10.214135+0000","last_fullsized":"2024-01-15T18:09:10.214135+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:02.592720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155819+0000","last_change":"2024-01-15T18:09:10.150403+0000","last_active":"2024-01-15T18:09:11.155819+0000","last_peered":"2024-01-15T18:09:11.155819+0000","last_clean":"2024-01-15T18:09:11.155819+0000","last_became_active":"2024-01-15T18:09:10.150113+0000","last_became_peered":"2024-01-15T18:09:10.150113+0000","last_unstale":"2024-01-15T18:09:11.155819+0000","last_undegraded":"2024-01-15T18:09:11.155819+0000","last_fullsized":"2024-01-15T18:09:11.155819+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:29:52.274001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"57'3","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.222212+0000","last_change":"2024-01-15T18:09:09.142251+0000","last_active":"2024-01-15T18:09:10.222212+0000","last_peered":"2024-01-15T18:09:10.222212+0000","last_clean":"2024-01-15T18:09:10.222212+0000","last_became_active":"2024-01-15T18:09:09.141961+0000","last_became_peered":"2024-01-15T18:09:09.141961+0000","last_unstale":"2024-01-15T18:09:10.222212+0000","last_undegraded":"2024-01-15T18:09:10.222212+0000","last_fullsized":"2024-01-15T18:09:10.222212+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:56:33.327612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.044068+0000","last_change":"2024-01-15T18:09:10.172402+0000","last_active":"2024-01-15T18:09:12.044068+0000","last_peered":"2024-01-15T18:09:12.044068+0000","last_clean":"2024-01-15T18:09:12.044068+0000","last_became_active":"2024-01-15T18:09:10.171813+0000","last_became_peered":"2024-01-15T18:09:10.171813+0000","last_unstale":"2024-01-15T18:09:12.044068+0000","last_undegraded":"2024-01-15T18:09:12.044068+0000","last_fullsized":"2024-01-15T18:09:12.044068+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:26:05.237285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"57'6","reported_seq":18,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.240212+0000","last_change":"2024-01-15T18:09:09.147769+0000","last_active":"2024-01-15T18:09:10.240212+0000","last_peered":"2024-01-15T18:09:10.240212+0000","last_clean":"2024-01-15T18:09:10.240212+0000","last_became_active":"2024-01-15T18:09:09.147356+0000","last_became_peered":"2024-01-15T18:09:09.147356+0000","last_unstale":"2024-01-15T18:09:10.240212+0000","last_undegraded":"2024-01-15T18:09:10.240212+0000","last_fullsized":"2024-01-15T18:09:10.240212+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:18:52.199559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155288+0000","last_change":"2024-01-15T18:09:10.154188+0000","last_active":"2024-01-15T18:09:11.155288+0000","last_peered":"2024-01-15T18:09:11.155288+0000","last_clean":"2024-01-15T18:09:11.155288+0000","last_became_active":"2024-01-15T18:09:10.153881+0000","last_became_peered":"2024-01-15T18:09:10.153881+0000","last_unstale":"2024-01-15T18:09:11.155288+0000","last_undegraded":"2024-01-15T18:09:11.155288+0000","last_fullsized":"2024-01-15T18:09:11.155288+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:57:04.284081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.214074+0000","last_change":"2024-01-15T18:09:09.141986+0000","last_active":"2024-01-15T18:09:10.214074+0000","last_peered":"2024-01-15T18:09:10.214074+0000","last_clean":"2024-01-15T18:09:10.214074+0000","last_became_active":"2024-01-15T18:09:09.141626+0000","last_became_peered":"2024-01-15T18:09:09.141626+0000","last_unstale":"2024-01-15T18:09:10.214074+0000","last_undegraded":"2024-01-15T18:09:10.214074+0000","last_fullsized":"2024-01-15T18:09:10.214074+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:44:08.042896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.043830+0000","last_change":"2024-01-15T18:09:10.163997+0000","last_active":"2024-01-15T18:09:12.043830+0000","last_peered":"2024-01-15T18:09:12.043830+0000","last_clean":"2024-01-15T18:09:12.043830+0000","last_became_active":"2024-01-15T18:09:10.163758+0000","last_became_peered":"2024-01-15T18:09:10.163758+0000","last_unstale":"2024-01-15T18:09:12.043830+0000","last_undegraded":"2024-01-15T18:09:12.043830+0000","last_fullsized":"2024-01-15T18:09:12.043830+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:15:00.885663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.349932+0000","last_change":"2024-01-15T18:09:09.142040+0000","last_active":"2024-01-15T18:09:10.349932+0000","last_peered":"2024-01-15T18:09:10.349932+0000","last_clean":"2024-01-15T18:09:10.349932+0000","last_became_active":"2024-01-15T18:09:09.141713+0000","last_became_peered":"2024-01-15T18:09:09.141713+0000","last_unstale":"2024-01-15T18:09:10.349932+0000","last_undegraded":"2024-01-15T18:09:10.349932+0000","last_fullsized":"2024-01-15T18:09:10.349932+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:17:15.118577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.154765+0000","last_change":"2024-01-15T18:09:10.156574+0000","last_active":"2024-01-15T18:09:11.154765+0000","last_peered":"2024-01-15T18:09:11.154765+0000","last_clean":"2024-01-15T18:09:11.154765+0000","last_became_active":"2024-01-15T18:09:10.156225+0000","last_became_peered":"2024-01-15T18:09:10.156225+0000","last_unstale":"2024-01-15T18:09:11.154765+0000","last_undegraded":"2024-01-15T18:09:11.154765+0000","last_fullsized":"2024-01-15T18:09:11.154765+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:02:59.936727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.137769+0000","last_change":"2024-01-15T18:09:09.135271+0000","last_active":"2024-01-15T18:09:10.137769+0000","last_peered":"2024-01-15T18:09:10.137769+0000","last_clean":"2024-01-15T18:09:10.137769+0000","last_became_active":"2024-01-15T18:09:09.135011+0000","last_became_peered":"2024-01-15T18:09:09.135011+0000","last_unstale":"2024-01-15T18:09:10.137769+0000","last_undegraded":"2024-01-15T18:09:10.137769+0000","last_fullsized":"2024-01-15T18:09:10.137769+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:29:02.803967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.043560+0000","last_change":"2024-01-15T18:09:10.160727+0000","last_active":"2024-01-15T18:09:12.043560+0000","last_peered":"2024-01-15T18:09:12.043560+0000","last_clean":"2024-01-15T18:09:12.043560+0000","last_became_active":"2024-01-15T18:09:10.160132+0000","last_became_peered":"2024-01-15T18:09:10.160132+0000","last_unstale":"2024-01-15T18:09:12.043560+0000","last_undegraded":"2024-01-15T18:09:12.043560+0000","last_fullsized":"2024-01-15T18:09:12.043560+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:14:10.432447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"57'8","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.222099+0000","last_change":"2024-01-15T18:09:09.137932+0000","last_active":"2024-01-15T18:09:10.222099+0000","last_peered":"2024-01-15T18:09:10.222099+0000","last_clean":"2024-01-15T18:09:10.222099+0000","last_became_active":"2024-01-15T18:09:09.137626+0000","last_became_peered":"2024-01-15T18:09:09.137626+0000","last_unstale":"2024-01-15T18:09:10.222099+0000","last_undegraded":"2024-01-15T18:09:10.222099+0000","last_fullsized":"2024-01-15T18:09:10.222099+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:14:01.036199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.044095+0000","last_change":"2024-01-15T18:09:10.165126+0000","last_active":"2024-01-15T18:09:12.044095+0000","last_peered":"2024-01-15T18:09:12.044095+0000","last_clean":"2024-01-15T18:09:12.044095+0000","last_became_active":"2024-01-15T18:09:10.164811+0000","last_became_peered":"2024-01-15T18:09:10.164811+0000","last_unstale":"2024-01-15T18:09:12.044095+0000","last_undegraded":"2024-01-15T18:09:12.044095+0000","last_fullsized":"2024-01-15T18:09:12.044095+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:25:51.423892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.214152+0000","last_change":"2024-01-15T18:09:09.137481+0000","last_active":"2024-01-15T18:09:10.214152+0000","last_peered":"2024-01-15T18:09:10.214152+0000","last_clean":"2024-01-15T18:09:10.214152+0000","last_became_active":"2024-01-15T18:09:09.137205+0000","last_became_peered":"2024-01-15T18:09:09.137205+0000","last_unstale":"2024-01-15T18:09:10.214152+0000","last_undegraded":"2024-01-15T18:09:10.214152+0000","last_fullsized":"2024-01-15T18:09:10.214152+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:16:45.705931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155989+0000","last_change":"2024-01-15T18:09:10.166022+0000","last_active":"2024-01-15T18:09:11.155989+0000","last_peered":"2024-01-15T18:09:11.155989+0000","last_clean":"2024-01-15T18:09:11.155989+0000","last_became_active":"2024-01-15T18:09:10.165752+0000","last_became_peered":"2024-01-15T18:09:10.165752+0000","last_unstale":"2024-01-15T18:09:11.155989+0000","last_undegraded":"2024-01-15T18:09:11.155989+0000","last_fullsized":"2024-01-15T18:09:11.155989+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:45:32.407323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.175606+0000","last_change":"2024-01-15T18:09:09.152740+0000","last_active":"2024-01-15T18:09:10.175606+0000","last_peered":"2024-01-15T18:09:10.175606+0000","last_clean":"2024-01-15T18:09:10.175606+0000","last_became_active":"2024-01-15T18:09:09.152418+0000","last_became_peered":"2024-01-15T18:09:09.152418+0000","last_unstale":"2024-01-15T18:09:10.175606+0000","last_undegraded":"2024-01-15T18:09:10.175606+0000","last_fullsized":"2024-01-15T18:09:10.175606+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:15.421387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.543194+0000","last_change":"2024-01-15T18:09:10.155205+0000","last_active":"2024-01-15T18:09:11.543194+0000","last_peered":"2024-01-15T18:09:11.543194+0000","last_clean":"2024-01-15T18:09:11.543194+0000","last_became_active":"2024-01-15T18:09:10.154910+0000","last_became_peered":"2024-01-15T18:09:10.154910+0000","last_unstale":"2024-01-15T18:09:11.543194+0000","last_undegraded":"2024-01-15T18:09:11.543194+0000","last_fullsized":"2024-01-15T18:09:11.543194+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:15:46.043116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.177022+0000","last_change":"2024-01-15T18:09:09.141114+0000","last_active":"2024-01-15T18:09:10.177022+0000","last_peered":"2024-01-15T18:09:10.177022+0000","last_clean":"2024-01-15T18:09:10.177022+0000","last_became_active":"2024-01-15T18:09:09.140786+0000","last_became_peered":"2024-01-15T18:09:09.140786+0000","last_unstale":"2024-01-15T18:09:10.177022+0000","last_undegraded":"2024-01-15T18:09:10.177022+0000","last_fullsized":"2024-01-15T18:09:10.177022+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:37:27.921887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.044408+0000","last_change":"2024-01-15T18:09:10.172606+0000","last_active":"2024-01-15T18:09:12.044408+0000","last_peered":"2024-01-15T18:09:12.044408+0000","last_clean":"2024-01-15T18:09:12.044408+0000","last_became_active":"2024-01-15T18:09:10.172090+0000","last_became_peered":"2024-01-15T18:09:10.172090+0000","last_unstale":"2024-01-15T18:09:12.044408+0000","last_undegraded":"2024-01-15T18:09:12.044408+0000","last_fullsized":"2024-01-15T18:09:12.044408+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:32:35.991734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.152694+0000","last_change":"2024-01-15T18:09:09.148967+0000","last_active":"2024-01-15T18:09:10.152694+0000","last_peered":"2024-01-15T18:09:10.152694+0000","last_clean":"2024-01-15T18:09:10.152694+0000","last_became_active":"2024-01-15T18:09:09.148670+0000","last_became_peered":"2024-01-15T18:09:09.148670+0000","last_unstale":"2024-01-15T18:09:10.152694+0000","last_undegraded":"2024-01-15T18:09:10.152694+0000","last_fullsized":"2024-01-15T18:09:10.152694+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:48:58.170043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155444+0000","last_change":"2024-01-15T18:09:10.151767+0000","last_active":"2024-01-15T18:09:11.155444+0000","last_peered":"2024-01-15T18:09:11.155444+0000","last_clean":"2024-01-15T18:09:11.155444+0000","last_became_active":"2024-01-15T18:09:10.151462+0000","last_became_peered":"2024-01-15T18:09:10.151462+0000","last_unstale":"2024-01-15T18:09:11.155444+0000","last_undegraded":"2024-01-15T18:09:11.155444+0000","last_fullsized":"2024-01-15T18:09:11.155444+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:07:18.547932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.214091+0000","last_change":"2024-01-15T18:09:09.141213+0000","last_active":"2024-01-15T18:09:10.214091+0000","last_peered":"2024-01-15T18:09:10.214091+0000","last_clean":"2024-01-15T18:09:10.214091+0000","last_became_active":"2024-01-15T18:09:09.139957+0000","last_became_peered":"2024-01-15T18:09:09.139957+0000","last_unstale":"2024-01-15T18:09:10.214091+0000","last_undegraded":"2024-01-15T18:09:10.214091+0000","last_fullsized":"2024-01-15T18:09:10.214091+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:42:47.132801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.043692+0000","last_change":"2024-01-15T18:09:10.169239+0000","last_active":"2024-01-15T18:09:12.043692+0000","last_peered":"2024-01-15T18:09:12.043692+0000","last_clean":"2024-01-15T18:09:12.043692+0000","last_became_active":"2024-01-15T18:09:10.168825+0000","last_became_peered":"2024-01-15T18:09:10.168825+0000","last_unstale":"2024-01-15T18:09:12.043692+0000","last_undegraded":"2024-01-15T18:09:12.043692+0000","last_fullsized":"2024-01-15T18:09:12.043692+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:03.126844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.200044+0000","last_change":"2024-01-15T18:09:09.136084+0000","last_active":"2024-01-15T18:09:10.200044+0000","last_peered":"2024-01-15T18:09:10.200044+0000","last_clean":"2024-01-15T18:09:10.200044+0000","last_became_active":"2024-01-15T18:09:09.135760+0000","last_became_peered":"2024-01-15T18:09:09.135760+0000","last_unstale":"2024-01-15T18:09:10.200044+0000","last_undegraded":"2024-01-15T18:09:10.200044+0000","last_fullsized":"2024-01-15T18:09:10.200044+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:37.003640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.043507+0000","last_change":"2024-01-15T18:09:10.159195+0000","last_active":"2024-01-15T18:09:12.043507+0000","last_peered":"2024-01-15T18:09:12.043507+0000","last_clean":"2024-01-15T18:09:12.043507+0000","last_became_active":"2024-01-15T18:09:10.158906+0000","last_became_peered":"2024-01-15T18:09:10.158906+0000","last_unstale":"2024-01-15T18:09:12.043507+0000","last_undegraded":"2024-01-15T18:09:12.043507+0000","last_fullsized":"2024-01-15T18:09:12.043507+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:44:11.424905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.137490+0000","last_change":"2024-01-15T18:09:09.136145+0000","last_active":"2024-01-15T18:09:10.137490+0000","last_peered":"2024-01-15T18:09:10.137490+0000","last_clean":"2024-01-15T18:09:10.137490+0000","last_became_active":"2024-01-15T18:09:09.135807+0000","last_became_peered":"2024-01-15T18:09:09.135807+0000","last_unstale":"2024-01-15T18:09:10.137490+0000","last_undegraded":"2024-01-15T18:09:10.137490+0000","last_fullsized":"2024-01-15T18:09:10.137490+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:35:26.455284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155791+0000","last_change":"2024-01-15T18:09:10.157661+0000","last_active":"2024-01-15T18:09:11.155791+0000","last_peered":"2024-01-15T18:09:11.155791+0000","last_clean":"2024-01-15T18:09:11.155791+0000","last_became_active":"2024-01-15T18:09:10.157399+0000","last_became_peered":"2024-01-15T18:09:10.157399+0000","last_unstale":"2024-01-15T18:09:11.155791+0000","last_undegraded":"2024-01-15T18:09:11.155791+0000","last_fullsized":"2024-01-15T18:09:11.155791+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:41:56.291009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.200109+0000","last_change":"2024-01-15T18:09:09.145612+0000","last_active":"2024-01-15T18:09:10.200109+0000","last_peered":"2024-01-15T18:09:10.200109+0000","last_clean":"2024-01-15T18:09:10.200109+0000","last_became_active":"2024-01-15T18:09:09.145279+0000","last_became_peered":"2024-01-15T18:09:09.145279+0000","last_unstale":"2024-01-15T18:09:10.200109+0000","last_undegraded":"2024-01-15T18:09:10.200109+0000","last_fullsized":"2024-01-15T18:09:10.200109+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:53.930232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.156102+0000","last_change":"2024-01-15T18:09:10.153156+0000","last_active":"2024-01-15T18:09:11.156102+0000","last_peered":"2024-01-15T18:09:11.156102+0000","last_clean":"2024-01-15T18:09:11.156102+0000","last_became_active":"2024-01-15T18:09:10.152783+0000","last_became_peered":"2024-01-15T18:09:10.152783+0000","last_unstale":"2024-01-15T18:09:11.156102+0000","last_undegraded":"2024-01-15T18:09:11.156102+0000","last_fullsized":"2024-01-15T18:09:11.156102+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:36:30.872154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.153686+0000","last_change":"2024-01-15T18:09:09.152741+0000","last_active":"2024-01-15T18:09:10.153686+0000","last_peered":"2024-01-15T18:09:10.153686+0000","last_clean":"2024-01-15T18:09:10.153686+0000","last_became_active":"2024-01-15T18:09:09.152470+0000","last_became_peered":"2024-01-15T18:09:09.152470+0000","last_unstale":"2024-01-15T18:09:10.153686+0000","last_undegraded":"2024-01-15T18:09:10.153686+0000","last_fullsized":"2024-01-15T18:09:10.153686+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:35:26.455284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.156127+0000","last_change":"2024-01-15T18:09:10.143039+0000","last_active":"2024-01-15T18:09:11.156127+0000","last_peered":"2024-01-15T18:09:11.156127+0000","last_clean":"2024-01-15T18:09:11.156127+0000","last_became_active":"2024-01-15T18:09:10.142814+0000","last_became_peered":"2024-01-15T18:09:10.142814+0000","last_unstale":"2024-01-15T18:09:11.156127+0000","last_undegraded":"2024-01-15T18:09:11.156127+0000","last_fullsized":"2024-01-15T18:09:11.156127+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:55:44.009610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.349971+0000","last_change":"2024-01-15T18:09:09.141870+0000","last_active":"2024-01-15T18:09:10.349971+0000","last_peered":"2024-01-15T18:09:10.349971+0000","last_clean":"2024-01-15T18:09:10.349971+0000","last_became_active":"2024-01-15T18:09:09.141523+0000","last_became_peered":"2024-01-15T18:09:09.141523+0000","last_unstale":"2024-01-15T18:09:10.349971+0000","last_undegraded":"2024-01-15T18:09:10.349971+0000","last_fullsized":"2024-01-15T18:09:10.349971+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:15:13.572542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.043975+0000","last_change":"2024-01-15T18:09:10.159462+0000","last_active":"2024-01-15T18:09:12.043975+0000","last_peered":"2024-01-15T18:09:12.043975+0000","last_clean":"2024-01-15T18:09:12.043975+0000","last_became_active":"2024-01-15T18:09:10.159186+0000","last_became_peered":"2024-01-15T18:09:10.159186+0000","last_unstale":"2024-01-15T18:09:12.043975+0000","last_undegraded":"2024-01-15T18:09:12.043975+0000","last_fullsized":"2024-01-15T18:09:12.043975+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:36:30.872154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.137795+0000","last_change":"2024-01-15T18:09:09.136593+0000","last_active":"2024-01-15T18:09:10.137795+0000","last_peered":"2024-01-15T18:09:10.137795+0000","last_clean":"2024-01-15T18:09:10.137795+0000","last_became_active":"2024-01-15T18:09:09.136246+0000","last_became_peered":"2024-01-15T18:09:09.136246+0000","last_unstale":"2024-01-15T18:09:10.137795+0000","last_undegraded":"2024-01-15T18:09:10.137795+0000","last_fullsized":"2024-01-15T18:09:10.137795+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:10:09.158484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.542992+0000","last_change":"2024-01-15T18:09:10.145733+0000","last_active":"2024-01-15T18:09:11.542992+0000","last_peered":"2024-01-15T18:09:11.542992+0000","last_clean":"2024-01-15T18:09:11.542992+0000","last_became_active":"2024-01-15T18:09:10.145479+0000","last_became_peered":"2024-01-15T18:09:10.145479+0000","last_unstale":"2024-01-15T18:09:11.542992+0000","last_undegraded":"2024-01-15T18:09:11.542992+0000","last_fullsized":"2024-01-15T18:09:11.542992+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:21:55.216394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.200053+0000","last_change":"2024-01-15T18:09:09.142681+0000","last_active":"2024-01-15T18:09:10.200053+0000","last_peered":"2024-01-15T18:09:10.200053+0000","last_clean":"2024-01-15T18:09:10.200053+0000","last_became_active":"2024-01-15T18:09:09.142357+0000","last_became_peered":"2024-01-15T18:09:09.142357+0000","last_unstale":"2024-01-15T18:09:10.200053+0000","last_undegraded":"2024-01-15T18:09:10.200053+0000","last_fullsized":"2024-01-15T18:09:10.200053+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:41:15.076278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:12.043957+0000","last_change":"2024-01-15T18:09:10.162125+0000","last_active":"2024-01-15T18:09:12.043957+0000","last_peered":"2024-01-15T18:09:12.043957+0000","last_clean":"2024-01-15T18:09:12.043957+0000","last_became_active":"2024-01-15T18:09:10.161778+0000","last_became_peered":"2024-01-15T18:09:10.161778+0000","last_unstale":"2024-01-15T18:09:12.043957+0000","last_undegraded":"2024-01-15T18:09:12.043957+0000","last_fullsized":"2024-01-15T18:09:12.043957+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:16:46.708136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.240177+0000","last_change":"2024-01-15T18:09:09.135449+0000","last_active":"2024-01-15T18:09:10.240177+0000","last_peered":"2024-01-15T18:09:10.240177+0000","last_clean":"2024-01-15T18:09:10.240177+0000","last_became_active":"2024-01-15T18:09:09.135010+0000","last_became_peered":"2024-01-15T18:09:09.135010+0000","last_unstale":"2024-01-15T18:09:10.240177+0000","last_undegraded":"2024-01-15T18:09:10.240177+0000","last_fullsized":"2024-01-15T18:09:10.240177+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:54:08.460596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.155565+0000","last_change":"2024-01-15T18:09:10.158452+0000","last_active":"2024-01-15T18:09:11.155565+0000","last_peered":"2024-01-15T18:09:11.155565+0000","last_clean":"2024-01-15T18:09:11.155565+0000","last_became_active":"2024-01-15T18:09:10.158267+0000","last_became_peered":"2024-01-15T18:09:10.158267+0000","last_unstale":"2024-01-15T18:09:11.155565+0000","last_undegraded":"2024-01-15T18:09:11.155565+0000","last_fullsized":"2024-01-15T18:09:11.155565+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:02:56.938951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.200034+0000","last_change":"2024-01-15T18:09:09.146561+0000","last_active":"2024-01-15T18:09:10.200034+0000","last_peered":"2024-01-15T18:09:10.200034+0000","last_clean":"2024-01-15T18:09:10.200034+0000","last_became_active":"2024-01-15T18:09:09.146263+0000","last_became_peered":"2024-01-15T18:09:09.146263+0000","last_unstale":"2024-01-15T18:09:10.200034+0000","last_undegraded":"2024-01-15T18:09:10.200034+0000","last_fullsized":"2024-01-15T18:09:10.200034+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:00:48.078792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.542929+0000","last_change":"2024-01-15T18:09:10.157427+0000","last_active":"2024-01-15T18:09:11.542929+0000","last_peered":"2024-01-15T18:09:11.542929+0000","last_clean":"2024-01-15T18:09:11.542929+0000","last_became_active":"2024-01-15T18:09:10.157146+0000","last_became_peered":"2024-01-15T18:09:10.157146+0000","last_unstale":"2024-01-15T18:09:11.542929+0000","last_undegraded":"2024-01-15T18:09:11.542929+0000","last_fullsized":"2024-01-15T18:09:11.542929+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:10:43.276239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.214010+0000","last_change":"2024-01-15T18:09:09.136141+0000","last_active":"2024-01-15T18:09:10.214010+0000","last_peered":"2024-01-15T18:09:10.214010+0000","last_clean":"2024-01-15T18:09:10.214010+0000","last_became_active":"2024-01-15T18:09:09.135853+0000","last_became_peered":"2024-01-15T18:09:09.135853+0000","last_unstale":"2024-01-15T18:09:10.214010+0000","last_undegraded":"2024-01-15T18:09:10.214010+0000","last_fullsized":"2024-01-15T18:09:10.214010+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:25:50.421687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":11,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-15T18:09:11.542818+0000","last_change":"2024-01-15T18:09:10.161307+0000","last_active":"2024-01-15T18:09:11.542818+0000","last_peered":"2024-01-15T18:09:11.542818+0000","last_clean":"2024-01-15T18:09:11.542818+0000","last_became_active":"2024-01-15T18:09:10.160945+0000","last_became_peered":"2024-01-15T18:09:10.160945+0000","last_unstale":"2024-01-15T18:09:11.542818+0000","last_undegraded":"2024-01-15T18:09:11.542818+0000","last_fullsized":"2024-01-15T18:09:11.542818+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:09.109985+0000","last_clean_scrub_stamp":"2024-01-15T18:09:09.109985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:32:35.991734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.349915+0000","last_change":"2024-01-15T18:09:09.142696+0000","last_active":"2024-01-15T18:09:10.349915+0000","last_peered":"2024-01-15T18:09:10.349915+0000","last_clean":"2024-01-15T18:09:10.349915+0000","last_became_active":"2024-01-15T18:09:09.142448+0000","last_became_peered":"2024-01-15T18:09:09.142448+0000","last_unstale":"2024-01-15T18:09:10.349915+0000","last_undegraded":"2024-01-15T18:09:10.349915+0000","last_fullsized":"2024-01-15T18:09:10.349915+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:09:08.107779+0000","last_clean_scrub_stamp":"2024-01-15T18:09:08.107779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:40:45.529038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":195,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-15T18:09:10.147481+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:09:10.147481+0000","last_peered":"2024-01-15T18:09:10.147481+0000","last_clean":"2024-01-15T18:09:10.147481+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:09:10.147481+0000","last_undegraded":"2024-01-15T18:09:10.147481+0000","last_fullsized":"2024-01-15T18:09:10.147481+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509497,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36604,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335236,"statfs":{"total":96636764160,"available":96599281664,"internally_reserved":0,"allocated":1290240,"data_stored":878172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":0.82899999999999996,"15min":0.69199999999999995},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":10.284000000000001,"5min":10.284000000000001,"15min":10.284000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.8440000000000001,"5min":0.89800000000000002,"15min":0.752},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":9.9610000000000003,"5min":9.9610000000000003,"15min":9.9610000000000003},"last":0.54500000000000004}]},{"osd":1,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":0.76600000000000001,"15min":0.69999999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":7.0279999999999996,"5min":7.0279999999999996,"15min":7.0279999999999996},"last":0.63},{"interface":"front","average":{"1min":1.7230000000000001,"5min":0.89700000000000002,"15min":0.76500000000000001},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":9.9250000000000007,"5min":9.9250000000000007,"15min":9.9250000000000007},"last":0.60799999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":0.872,"15min":0.72199999999999998},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":10.119999999999999,"5min":10.119999999999999,"15min":10.119999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":1.242,"5min":0.82199999999999995,"15min":0.748},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":7.0789999999999997,"5min":7.0789999999999997,"15min":7.0789999999999997},"last":0.48099999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.625,"5min":0.92500000000000004,"15min":0.751},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":9.3539999999999992,"5min":9.3539999999999992,"15min":9.3539999999999992},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.7070000000000001,"5min":0.93400000000000005,"15min":0.81999999999999995},"min":{"1min":0.45700000000000002,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":10.34,"5min":10.34,"15min":10.34},"last":0.53100000000000003}]},{"osd":4,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":2.4079999999999999,"5min":0.96499999999999997,"15min":0.73499999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":17.021000000000001,"5min":17.021000000000001,"15min":17.021000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":1.734,"5min":0.92700000000000005,"15min":0.752},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":14.871,"5min":14.871,"15min":14.871},"last":0.57299999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":1.837,"5min":0.91000000000000003,"15min":0.77000000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":19.959,"5min":19.959,"15min":19.959},"last":0.747},{"interface":"front","average":{"1min":1.9610000000000001,"5min":0.94399999999999995,"15min":0.80100000000000005},"min":{"1min":0.47999999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":11.053000000000001,"5min":11.053000000000001,"15min":11.053000000000001},"last":0.79500000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:08:55 2024","interfaces":[{"interface":"back","average":{"1min":3.0070000000000001,"5min":1.2170000000000001,"15min":0.94299999999999995},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":22.478999999999999,"5min":22.478999999999999,"15min":22.478999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":2.6269999999999998,"5min":1.1739999999999999,"15min":0.88100000000000001},"min":{"1min":0.52800000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":21.509,"5min":21.509,"15min":21.509},"last":0.86399999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509497,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36012,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335828,"statfs":{"total":96636764160,"available":96599887872,"internally_reserved":0,"allocated":688128,"data_stored":287644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":0.86299999999999999,"15min":0.77800000000000002},"min":{"1min":0.45200000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":9.3339999999999996,"5min":9.3339999999999996,"15min":9.3339999999999996},"last":0.60799999999999998},{"interface":"front","average":{"1min":1.4850000000000001,"5min":0.83599999999999997,"15min":0.74399999999999999},"min":{"1min":0.47099999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":6.7759999999999998,"5min":6.7759999999999998,"15min":6.7759999999999998},"last":0.46200000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":3.8119999999999998,"5min":1.329,"15min":0.93300000000000005},"min":{"1min":0.51400000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":53.451999999999998,"5min":53.451999999999998,"15min":53.451999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":1.3009999999999999,"5min":0.83599999999999997,"15min":0.78400000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":5.9900000000000002,"5min":5.9900000000000002,"15min":5.9900000000000002},"last":0.66100000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":0.84499999999999997,"15min":0.85199999999999998},"min":{"1min":0.46700000000000003,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":6.0380000000000003,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.2310000000000001,"5min":0.82799999999999996,"15min":0.80700000000000005},"min":{"1min":0.57099999999999995,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":4.6059999999999999,"5min":4.6059999999999999,"15min":4.6059999999999999},"last":0.624}]},{"osd":3,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":0.84899999999999998,"15min":0.82599999999999996},"min":{"1min":0.51900000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":3.9689999999999999,"5min":3.9689999999999999,"15min":3.9689999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":3.6709999999999998,"5min":1.355,"15min":1.0109999999999999},"min":{"1min":0.45200000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":53.398000000000003,"5min":53.398000000000003,"15min":53.398000000000003},"last":0.63500000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":1.6479999999999999,"5min":0.85499999999999998,"15min":0.751},"min":{"1min":0.48099999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":8.6259999999999994,"5min":8.6259999999999994,"15min":8.6259999999999994},"last":0.52500000000000002},{"interface":"front","average":{"1min":1.2809999999999999,"5min":0.79600000000000004,"15min":0.752},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":6.9589999999999996,"5min":6.9589999999999996,"15min":6.9589999999999996},"last":0.41099999999999998}]},{"osd":5,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":4.2560000000000002,"5min":1.371,"15min":0.90000000000000002},"min":{"1min":0.44500000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":53.229999999999997,"5min":53.229999999999997,"15min":53.229999999999997},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.847,"5min":0.96799999999999997,"15min":0.84599999999999997},"min":{"1min":0.53900000000000003,"5min":0.435,"15min":0.435},"max":{"1min":10.029999999999999,"5min":10.029999999999999,"15min":10.029999999999999},"last":0.72899999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:08:51 2024","interfaces":[{"interface":"back","average":{"1min":3.1640000000000001,"5min":1.2589999999999999,"15min":0.97699999999999998},"min":{"1min":0.46200000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":45.070999999999998,"5min":45.070999999999998,"15min":45.070999999999998},"last":0.58999999999999997},{"interface":"front","average":{"1min":4.5990000000000002,"5min":1.5369999999999999,"15min":1.054},"min":{"1min":0.48499999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":53.338000000000001,"5min":53.338000000000001,"15min":53.338000000000001},"last":0.68400000000000005}]}]},{"osd":4,"up_from":15,"seq":64424509497,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36028,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335812,"statfs":{"total":96636764160,"available":96599871488,"internally_reserved":0,"allocated":700416,"data_stored":289256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":0.91700000000000004,"15min":0.72599999999999998},"min":{"1min":0.434,"5min":0.38,"15min":0.38},"max":{"1min":23.318000000000001,"5min":23.318000000000001,"15min":23.318000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.7649999999999999,"5min":0.84999999999999998,"15min":0.71399999999999997},"min":{"1min":0.44800000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":10.210000000000001,"5min":10.210000000000001,"15min":10.210000000000001},"last":0.35099999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.71099999999999997,"15min":0.67200000000000004},"min":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":5.016,"5min":5.016,"15min":5.016},"last":0.59099999999999997},{"interface":"front","average":{"1min":2.0990000000000002,"5min":0.93400000000000005,"15min":0.73999999999999999},"min":{"1min":0.45100000000000001,"5min":0.377,"15min":0.377},"max":{"1min":22.408999999999999,"5min":22.408999999999999,"15min":22.408999999999999},"last":0.53800000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":0.71799999999999997,"15min":0.65900000000000003},"min":{"1min":0.46700000000000003,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":5.1159999999999997,"5min":5.1159999999999997,"15min":5.1159999999999997},"last":0.51900000000000002},{"interface":"front","average":{"1min":2.1419999999999999,"5min":1.036,"15min":0.81000000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":22.338999999999999,"5min":22.338999999999999,"15min":22.338999999999999},"last":0.46100000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.8380000000000001,"5min":0.91600000000000004,"15min":0.76600000000000001},"min":{"1min":0.42699999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":10.271000000000001,"5min":10.271000000000001,"15min":10.271000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":1.27,"5min":0.84099999999999997,"15min":0.76500000000000001},"min":{"1min":0.59899999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":8.0719999999999992,"5min":8.0719999999999992,"15min":8.0719999999999992},"last":0.63700000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":2.7879999999999998,"5min":1.081,"15min":0.80400000000000005},"min":{"1min":0.39000000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":31.617999999999999,"5min":31.617999999999999,"15min":31.617999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":2.4369999999999998,"5min":1.0009999999999999,"15min":0.76700000000000002},"min":{"1min":0.439,"5min":0.377,"15min":0.377},"max":{"1min":17.145,"5min":17.145,"15min":17.145},"last":0.439}]},{"osd":6,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":0.83499999999999996,"15min":0.73099999999999998},"min":{"1min":0.53900000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.577,"5min":5.577,"15min":5.577},"last":0.47899999999999998},{"interface":"front","average":{"1min":4.9329999999999998,"5min":1.5720000000000001,"15min":1.014},"min":{"1min":0.56299999999999994,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":28.872,"5min":28.872,"15min":28.872},"last":0.57399999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":3.726,"5min":1.3149999999999999,"15min":0.91700000000000004},"min":{"1min":0.48199999999999998,"5min":0.436,"15min":0.436},"max":{"1min":39.256,"5min":39.256,"15min":39.256},"last":0.497},{"interface":"front","average":{"1min":2.069,"5min":1.0389999999999999,"15min":0.86699999999999999},"min":{"1min":0.51800000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":26.573,"5min":26.573,"15min":26.573},"last":0.55100000000000005}]}]},{"osd":3,"up_from":15,"seq":64424509496,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36024,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335816,"statfs":{"total":96636764160,"available":96599875584,"internally_reserved":0,"allocated":696320,"data_stored":287700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.45900000000000002,"15min":0.433},"min":{"1min":0.28899999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.641,"5min":2.641,"15min":2.641},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.4099999999999999,"5min":0.68300000000000005,"15min":0.54100000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":17.756,"5min":17.756,"15min":17.756},"last":0.442}]},{"osd":1,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.54100000000000004,"15min":0.52000000000000002},"min":{"1min":0.29799999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.0620000000000003,"5min":4.0620000000000003,"15min":4.0620000000000003},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.54500000000000004,"15min":0.53900000000000003},"min":{"1min":0.371,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.58799999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":0.69199999999999995,"15min":0.60499999999999998},"min":{"1min":0.42499999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":10.849,"5min":10.849,"15min":10.849},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.626,"15min":0.58499999999999996},"min":{"1min":0.40500000000000003,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.617,"5min":2.7589999999999999,"15min":2.7589999999999999},"last":0.63700000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":1.756,"5min":1.004,"15min":0.873},"min":{"1min":0.56499999999999995,"5min":0.5,"15min":0.5},"max":{"1min":17.853999999999999,"5min":17.853999999999999,"15min":17.853999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.022,"5min":0.85099999999999998,"15min":0.80500000000000005},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":4.1420000000000003,"5min":5.0369999999999999,"15min":5.0369999999999999},"last":0.84099999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":0.82699999999999996,"15min":0.76900000000000002},"min":{"1min":0.51100000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":4.7320000000000002,"5min":4.7320000000000002,"15min":4.7320000000000002},"last":0.85199999999999998},{"interface":"front","average":{"1min":2.145,"5min":1.004,"15min":0.81499999999999995},"min":{"1min":0.54900000000000004,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":17.937999999999999,"5min":17.937999999999999,"15min":17.937999999999999},"last":0.83199999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":2.125,"5min":1.046,"15min":0.873},"min":{"1min":0.52300000000000002,"5min":0.499,"15min":0.499},"max":{"1min":23.100999999999999,"5min":23.100999999999999,"15min":23.100999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":1.113,"5min":0.84599999999999997,"15min":0.79000000000000004},"min":{"1min":0.61899999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":5.7030000000000003,"5min":5.7030000000000003,"15min":5.7030000000000003},"last":0.80900000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:08:54 2024","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":0.95999999999999996,"15min":0.83799999999999997},"min":{"1min":0.54900000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":17.896999999999998,"5min":17.896999999999998,"15min":17.896999999999998},"last":0.872},{"interface":"front","average":{"1min":1.1020000000000001,"5min":0.93999999999999995,"15min":0.86199999999999999},"min":{"1min":0.60599999999999998,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":4.1920000000000002,"5min":10.705,"15min":10.705},"last":0.76600000000000001}]}]},{"osd":2,"up_from":15,"seq":64424509496,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36024,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335816,"statfs":{"total":96636764160,"available":96599875584,"internally_reserved":0,"allocated":696320,"data_stored":287700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.55500000000000005,"15min":0.49399999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":5.1849999999999996,"5min":5.1849999999999996,"15min":5.1849999999999996},"last":0.51700000000000002},{"interface":"front","average":{"1min":1.0640000000000001,"5min":0.57799999999999996,"15min":0.504},"min":{"1min":0.307,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":7.516,"5min":7.516,"15min":7.516},"last":0.39600000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.60799999999999998,"15min":0.56599999999999995},"min":{"1min":0.35499999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.2509999999999999,"5min":4.8079999999999998,"15min":4.8079999999999998},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.61599999999999999,"15min":0.50800000000000001},"min":{"1min":0.32500000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":8.9190000000000005,"5min":8.9190000000000005,"15min":8.9190000000000005},"last":0.47399999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":0.69799999999999995,"15min":0.61299999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":12.185,"5min":12.185,"15min":12.185},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.68600000000000005,"15min":0.59599999999999997},"min":{"1min":0.42899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":7.4580000000000002,"5min":7.4580000000000002,"15min":7.4580000000000002},"last":0.48799999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":0.91400000000000003,"15min":0.80100000000000005},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":8.6609999999999996,"5min":8.6609999999999996,"15min":8.6609999999999996},"last":0.58799999999999997},{"interface":"front","average":{"1min":2.3519999999999999,"5min":1.0820000000000001,"15min":0.875},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":19.146999999999998,"5min":19.146999999999998,"15min":19.146999999999998},"last":0.55100000000000005}]},{"osd":5,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":2.4929999999999999,"5min":1.1100000000000001,"15min":0.90200000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":31.710000000000001,"5min":31.710000000000001,"15min":31.710000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":2.7970000000000002,"5min":1.2150000000000001,"15min":0.93300000000000005},"min":{"1min":0.52900000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":22.539000000000001,"5min":22.539000000000001,"15min":22.539000000000001},"last":0.69099999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":0.92000000000000004,"15min":0.82899999999999996},"min":{"1min":0.51100000000000001,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":9.4540000000000006,"5min":9.4540000000000006,"15min":9.4540000000000006},"last":0.753},{"interface":"front","average":{"1min":2.29,"5min":1.085,"15min":0.89500000000000002},"min":{"1min":0.54400000000000004,"5min":0.503,"15min":0.503},"max":{"1min":18.614999999999998,"5min":18.614999999999998,"15min":18.614999999999998},"last":0.78800000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:08:48 2024","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":0.94599999999999995,"15min":0.85199999999999998},"min":{"1min":0.51200000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":11.432,"5min":11.432,"15min":11.432},"last":0.71799999999999997},{"interface":"front","average":{"1min":2.577,"5min":1.204,"15min":0.95499999999999996},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":17.013000000000002,"5min":17.013000000000002,"15min":17.013000000000002},"last":0.65500000000000003}]}]},{"osd":1,"up_from":15,"seq":64424509498,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36020,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335820,"statfs":{"total":96636764160,"available":96599879680,"internally_reserved":0,"allocated":692224,"data_stored":287734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":0.78400000000000003,"15min":0.60199999999999998},"min":{"1min":0.27300000000000002,"5min":0.254,"15min":0.254},"max":{"1min":9.8620000000000001,"5min":19.577000000000002,"15min":19.577000000000002},"last":0.38500000000000001},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.052,"15min":0.67600000000000005},"min":{"1min":0.28799999999999998,"5min":0.251,"15min":0.251},"max":{"1min":5.6639999999999997,"5min":48.277999999999999,"15min":48.277999999999999},"last":0.28999999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":0.997,"15min":0.69799999999999995},"min":{"1min":0.312,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":6.4569999999999999,"5min":40.698,"15min":40.698},"last":0.442},{"interface":"front","average":{"1min":1.97,"5min":1.351,"15min":0.94899999999999995},"min":{"1min":0.375,"5min":0.317,"15min":0.317},"max":{"1min":18.827000000000002,"5min":48.316000000000003,"15min":48.316000000000003},"last":0.33700000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.1899999999999999,"15min":0.83199999999999996},"min":{"1min":0.373,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.915,"5min":48.234999999999999,"15min":48.234999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.65700000000000003,"15min":0.61699999999999999},"min":{"1min":0.34699999999999998,"5min":0.311,"15min":0.311},"max":{"1min":4.1420000000000003,"5min":6.093,"15min":6.093},"last":0.42999999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":0.86899999999999999,"15min":0.751},"min":{"1min":0.48299999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":4.1950000000000003,"5min":5.9169999999999998,"15min":5.9169999999999998},"last":0.72099999999999997},{"interface":"front","average":{"1min":1.175,"5min":1.0129999999999999,"15min":0.85799999999999998},"min":{"1min":0.46999999999999997,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":6.3789999999999996,"5min":19.829000000000001,"15min":19.829000000000001},"last":0.55300000000000005}]},{"osd":5,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":1.339,"15min":0.96099999999999997},"min":{"1min":0.58399999999999996,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":22.594999999999999,"5min":22.594999999999999,"15min":22.594999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":2.0470000000000002,"5min":1.466,"15min":1.004},"min":{"1min":0.55900000000000005,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":11.768000000000001,"5min":48.198,"15min":48.198},"last":0.57199999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":3.0390000000000001,"5min":1.2649999999999999,"15min":0.92500000000000004},"min":{"1min":0.59199999999999997,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":35.112000000000002,"5min":35.112000000000002,"15min":35.112000000000002},"last":0.58299999999999996},{"interface":"front","average":{"1min":1.516,"5min":1.123,"15min":0.92500000000000004},"min":{"1min":0.58399999999999996,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":8.3670000000000009,"5min":19.780999999999999,"15min":19.780999999999999},"last":0.66600000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.3959999999999999,"15min":0.98799999999999999},"min":{"1min":0.56999999999999995,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":9.2870000000000008,"5min":48.148000000000003,"15min":48.148000000000003},"last":0.60899999999999999},{"interface":"front","average":{"1min":1.325,"5min":0.95499999999999996,"15min":0.85499999999999998},"min":{"1min":0.56799999999999995,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":4.2919999999999998,"5min":6.0549999999999997,"15min":6.0549999999999997},"last":0.625}]}]},{"osd":5,"up_from":14,"seq":60129542201,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36020,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335820,"statfs":{"total":96636764160,"available":96599879680,"internally_reserved":0,"allocated":696320,"data_stored":288296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.85999999999999999,"15min":0.77200000000000002},"min":{"1min":0.41199999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.2640000000000002,"5min":21.385000000000002,"15min":21.385000000000002},"last":0.60599999999999998},{"interface":"front","average":{"1min":1.173,"5min":0.91900000000000004,"15min":0.83699999999999997},"min":{"1min":0.51100000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":8.6590000000000007,"5min":8.6590000000000007,"15min":8.6590000000000007},"last":0.54200000000000004}]},{"osd":1,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.70799999999999996,"15min":0.68700000000000006},"min":{"1min":0.42999999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":3.1110000000000002,"5min":3.1110000000000002,"15min":3.1110000000000002},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.89700000000000002,"15min":0.755},"min":{"1min":0.47299999999999998,"5min":0.315,"15min":0.315},"max":{"1min":4.2039999999999997,"5min":21.5,"15min":21.5},"last":0.78700000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":0.99199999999999999,"15min":0.80400000000000005},"min":{"1min":0.55700000000000005,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":8.7750000000000004,"5min":8.7750000000000004,"15min":8.7750000000000004},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.1819999999999999,"5min":0.86599999999999999,"15min":0.81000000000000005},"min":{"1min":0.50800000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":7.9160000000000004,"5min":7.9160000000000004,"15min":7.9160000000000004},"last":0.77500000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.0009999999999999,"15min":0.873},"min":{"1min":0.53400000000000003,"5min":0.44,"15min":0.44},"max":{"1min":5.0650000000000004,"5min":21.457000000000001,"15min":21.457000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.694,"5min":1.083,"15min":0.93000000000000005},"min":{"1min":0.58699999999999997,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":8.7300000000000004,"5min":8.7300000000000004,"15min":8.7300000000000004},"last":0.80400000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":0.88100000000000001,"15min":0.78400000000000003},"min":{"1min":0.40699999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":8.8179999999999996,"5min":9.6679999999999993,"15min":9.6679999999999993},"last":0.73599999999999999},{"interface":"front","average":{"1min":2.0089999999999999,"5min":0.94899999999999995,"15min":0.81299999999999994},"min":{"1min":0.439,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":26.146999999999998,"5min":26.146999999999998,"15min":26.146999999999998},"last":0.69399999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":0.95499999999999996,"15min":0.81000000000000005},"min":{"1min":0.46300000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":6.6699999999999999,"5min":21.535,"15min":21.535},"last":0.70599999999999996},{"interface":"front","average":{"1min":2.649,"5min":1.2250000000000001,"15min":0.94599999999999995},"min":{"1min":0.46999999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":30.488,"5min":30.488,"15min":30.488},"last":0.84699999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":0.874,"15min":0.82599999999999996},"min":{"1min":0.51600000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":2.758,"5min":9.2240000000000002,"15min":9.2240000000000002},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.96699999999999997,"15min":0.877},"min":{"1min":0.44600000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":4.3520000000000003,"5min":21.574000000000002,"15min":21.574000000000002},"last":0.81899999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542201,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36608,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335232,"statfs":{"total":96636764160,"available":96599277568,"internally_reserved":0,"allocated":1298432,"data_stored":880206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.52600000000000002,"15min":0.48599999999999999},"min":{"1min":0.26100000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.2409999999999997,"5min":4.2409999999999997,"15min":4.2409999999999997},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55600000000000005,"15min":0.55300000000000005},"min":{"1min":0.35499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":2.0219999999999998,"5min":2.0219999999999998,"15min":2.0219999999999998},"last":0.48299999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.57599999999999996,"15min":0.57399999999999995},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.2330000000000001,"5min":2.2330000000000001,"15min":2.2330000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.53500000000000003,"15min":0.497},"min":{"1min":0.30099999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.6789999999999998,"5min":2.6789999999999998,"15min":2.6789999999999998},"last":0.54100000000000004}]},{"osd":3,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.63900000000000001,"15min":0.59699999999999998},"min":{"1min":0.38900000000000001,"5min":0.308,"15min":0.308},"max":{"1min":4.274,"5min":4.274,"15min":4.274},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.0309999999999999,"5min":0.68200000000000005,"15min":0.63400000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":7.2309999999999999,"5min":7.2309999999999999,"15min":7.2309999999999999},"last":0.621}]},{"osd":4,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":2.339,"5min":1.0680000000000001,"15min":0.878},"min":{"1min":0.46600000000000003,"5min":0.38,"15min":0.38},"max":{"1min":22.814,"5min":22.814,"15min":22.814},"last":0.63500000000000001},{"interface":"front","average":{"1min":1.516,"5min":0.92700000000000005,"15min":0.86499999999999999},"min":{"1min":0.54500000000000004,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":4.4800000000000004,"5min":4.4800000000000004,"15min":4.4800000000000004},"last":0.55700000000000005}]},{"osd":5,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":1.8600000000000001,"5min":0.97499999999999998,"15min":0.85799999999999998},"min":{"1min":0.495,"5min":0.441,"15min":0.441},"max":{"1min":12.428000000000001,"5min":12.428000000000001,"15min":12.428000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":1.915,"5min":1.018,"15min":0.91000000000000003},"min":{"1min":0.63800000000000001,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":13.064,"5min":13.064,"15min":13.064},"last":0.60199999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":2.8029999999999999,"5min":1.1699999999999999,"15min":0.92400000000000004},"min":{"1min":0.50800000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":31.059000000000001,"5min":31.059000000000001,"15min":31.059000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":3.6059999999999999,"5min":1.357,"15min":1.022},"min":{"1min":0.52000000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":40.731000000000002,"5min":40.731000000000002,"15min":40.731000000000002},"last":0.73999999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:08:52 2024","interfaces":[{"interface":"back","average":{"1min":1.657,"5min":0.96499999999999997,"15min":0.878},"min":{"1min":0.53800000000000003,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":14.032,"5min":14.032,"15min":14.032},"last":0.70799999999999996},{"interface":"front","average":{"1min":2.9569999999999999,"5min":1.254,"15min":0.995},"min":{"1min":0.58099999999999996,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":23.524000000000001,"5min":23.524000000000001,"15min":23.524000000000001},"last":0.68999999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:09:19.192 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-15T18:09:19.192 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-15T18:09:19.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.370+0000 7f3eb4a00640 1 Processor -- start 2024-01-15T18:09:19.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.370+0000 7f3eb4a00640 1 -- start start 2024-01-15T18:09:19.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.370+0000 7f3eb4a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3eac15f550 0x7f3eac0cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:19.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.370+0000 7f3eb4a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f3eac0cf110 0x7f3eac0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:19.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.370+0000 7f3eb4a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eac0b1cb0 con 0x7f3eac158900 2024-01-15T18:09:19.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.370+0000 7f3eb4a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eac15fa00 con 0x7f3eac0cf110 2024-01-15T18:09:19.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.370+0000 7f3eb4a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eac0cf540 con 0x7f3eac15f550 2024-01-15T18:09:19.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.374+0000 7f3eb1600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3eac15f550 0x7f3eac0cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:19.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.374+0000 7f3eb2a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f3eac0cf110 0x7f3eac0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:19.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.374+0000 7f3eb2a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f3eac0cf110 0x7f3eac0701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:59514/0 (socket says 172.21.15.27:59514) 2024-01-15T18:09:19.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.374+0000 7f3eb2a00640 1 -- 172.21.15.27:0/3495241013 learned_addr learned my addr 172.21.15.27:0/3495241013 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:19.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.374+0000 7f3eb1600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3eac15f550 0x7f3eac0cebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:45334/0 (socket says 172.21.15.27:45334) 2024-01-15T18:09:19.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.374+0000 7f3eb1600640 1 -- 172.21.15.27:0/3495241013 >> v2:172.21.15.93:3300/0 conn(0x7f3eac0cf110 msgr2=0x7f3eac0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:19.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.374+0000 7f3eb1600640 1 --2- 172.21.15.27:0/3495241013 >> v2:172.21.15.93:3300/0 conn(0x7f3eac0cf110 0x7f3eac0701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.374+0000 7f3eb1600640 1 -- 172.21.15.27:0/3495241013 >> v1:172.21.15.93:6789/0 conn(0x7f3eac158900 legacy=0x7f3eac15bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:19.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.374+0000 7f3eb0c00640 1 -- 172.21.15.27:0/3495241013 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3752623448 0 0) 0x7f3eac0b1cb0 con 0x7f3eac158900 2024-01-15T18:09:19.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.374+0000 7f3eb1600640 1 -- 172.21.15.27:0/3495241013 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eac0b1f00 con 0x7f3eac15f550 2024-01-15T18:09:19.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.374+0000 7f3eb2a00640 1 --2- 172.21.15.27:0/3495241013 >> v2:172.21.15.93:3300/0 conn(0x7f3eac0cf110 0x7f3eac0701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:09:19.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.378+0000 7f3eb1600640 1 --2- 172.21.15.27:0/3495241013 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3eac15f550 0x7f3eac0cebd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3e940097c0 tx=0x7f3e9401c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5122a4e72e673998 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:19.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.378+0000 7f3eb0c00640 1 -- 172.21.15.27:0/3495241013 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eac070f30 con 0x7f3eac15f550 2024-01-15T18:09:19.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.378+0000 7f3eb0c00640 1 -- 172.21.15.27:0/3495241013 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e940ab650 con 0x7f3eac15f550 2024-01-15T18:09:19.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.378+0000 7f3eb0c00640 1 -- 172.21.15.27:0/3495241013 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e940ab9e0 con 0x7f3eac15f550 2024-01-15T18:09:19.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.378+0000 7f3eb4a00640 1 -- 172.21.15.27:0/3495241013 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3eac15f550 msgr2=0x7f3eac0cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:19.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.378+0000 7f3eb4a00640 1 --2- 172.21.15.27:0/3495241013 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3eac15f550 0x7f3eac0cebd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3e940097c0 tx=0x7f3e9401c450 comp rx=0 tx=0).stop 2024-01-15T18:09:19.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.378+0000 7f3eb4a00640 1 -- 172.21.15.27:0/3495241013 shutdown_connections 2024-01-15T18:09:19.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.378+0000 7f3eb4a00640 1 --2- 172.21.15.27:0/3495241013 >> v2:172.21.15.93:3300/0 conn(0x7f3eac0cf110 0x7f3eac0701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.378+0000 7f3eb4a00640 1 --2- 172.21.15.27:0/3495241013 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3eac15f550 0x7f3eac0cebd0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.378+0000 7f3eb4a00640 1 -- 172.21.15.27:0/3495241013 >> 172.21.15.27:0/3495241013 conn(0x7f3eac0cb700 msgr2=0x7f3eac0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:19.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.378+0000 7f3eb4a00640 1 -- 172.21.15.27:0/3495241013 shutdown_connections 2024-01-15T18:09:19.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.378+0000 7f3eb4a00640 1 -- 172.21.15.27:0/3495241013 wait complete. 2024-01-15T18:09:19.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.378+0000 7f3eb4a00640 1 Processor -- start 2024-01-15T18:09:19.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.378+0000 7f3eb4a00640 1 -- start start 2024-01-15T18:09:19.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.382+0000 7f3eb4a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3eac0cf110 0x7f3eac153d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:19.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.382+0000 7f3eb4a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eac158900 0x7f3eac14eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:19.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.382+0000 7f3eb4a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3eac15f550 0x7f3eac0af0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:19.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.382+0000 7f3eb4a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eac14f020 con 0x7f3eac158900 2024-01-15T18:09:19.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.382+0000 7f3eb4a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eac15c350 con 0x7f3eac0cf110 2024-01-15T18:09:19.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.382+0000 7f3eb4a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eac15cbf0 con 0x7f3eac15f550 2024-01-15T18:09:19.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.382+0000 7f3eb1600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eac158900 0x7f3eac14eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:19.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.382+0000 7f3eb1600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eac158900 0x7f3eac14eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34416/0 (socket says 172.21.15.27:34416) 2024-01-15T18:09:19.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.382+0000 7f3eb1600640 1 -- 172.21.15.27:0/1572638232 learned_addr learned my addr 172.21.15.27:0/1572638232 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:19.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.382+0000 7f3eb2a00640 1 --2- 172.21.15.27:0/1572638232 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3eac15f550 0x7f3eac0af0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:19.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.382+0000 7f3eb2000640 1 --2- 172.21.15.27:0/1572638232 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3eac0cf110 0x7f3eac153d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:19.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.382+0000 7f3eb1600640 1 -- 172.21.15.27:0/1572638232 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3eac15f550 msgr2=0x7f3eac0af0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:19.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.382+0000 7f3eb1600640 1 --2- 172.21.15.27:0/1572638232 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3eac15f550 0x7f3eac0af0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.382+0000 7f3eb1600640 1 -- 172.21.15.27:0/1572638232 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3eac0cf110 msgr2=0x7f3eac153d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:19.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.382+0000 7f3eb1600640 1 --2- 172.21.15.27:0/1572638232 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3eac0cf110 0x7f3eac153d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.382+0000 7f3eb1600640 1 -- 172.21.15.27:0/1572638232 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea4009b30 con 0x7f3eac158900 2024-01-15T18:09:19.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.382+0000 7f3eb1600640 1 --2- 172.21.15.27:0/1572638232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eac158900 0x7f3eac14eae0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f3e9401a080 tx=0x7f3e940191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:19.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.382+0000 7f3eaa000640 1 -- 172.21.15.27:0/1572638232 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e940bf450 con 0x7f3eac158900 2024-01-15T18:09:19.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.382+0000 7f3eaa000640 1 -- 172.21.15.27:0/1572638232 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e940bf5b0 con 0x7f3eac158900 2024-01-15T18:09:19.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.382+0000 7f3eaa000640 1 -- 172.21.15.27:0/1572638232 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e940bf940 con 0x7f3eac158900 2024-01-15T18:09:19.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.382+0000 7f3eb4a00640 1 -- 172.21.15.27:0/1572638232 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e94009420 con 0x7f3eac158900 2024-01-15T18:09:19.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.382+0000 7f3eb4a00640 1 -- 172.21.15.27:0/1572638232 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eac15c5a0 con 0x7f3eac158900 2024-01-15T18:09:19.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.382+0000 7f3eb4a00640 1 -- 172.21.15.27:0/1572638232 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eac15cd80 con 0x7f3eac158900 2024-01-15T18:09:19.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.382+0000 7f3eaa000640 1 -- 172.21.15.27:0/1572638232 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f3e940b43f0 con 0x7f3eac158900 2024-01-15T18:09:19.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.386+0000 7f3eaa000640 1 --2- 172.21.15.27:0/1572638232 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e84066e90 0x7f3e84069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:19.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.386+0000 7f3eaa000640 1 -- 172.21.15.27:0/1572638232 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f3e9411f340 con 0x7f3eac158900 2024-01-15T18:09:19.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.386+0000 7f3eb2000640 1 --2- 172.21.15.27:0/1572638232 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e84066e90 0x7f3e84069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:19.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.386+0000 7f3eb2000640 1 --2- 172.21.15.27:0/1572638232 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e84066e90 0x7f3e84069350 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c000f70 tx=0x7f3e9c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:19.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.390+0000 7f3eaa000640 1 -- 172.21.15.27:0/1572638232 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3e940ebdf0 con 0x7f3eac158900 2024-01-15T18:09:19.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.598+0000 7f3eb4a00640 1 -- 172.21.15.27:0/1572638232 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f3eac04e850 con 0x7f3eac158900 2024-01-15T18:09:19.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.602+0000 7f3eaa000640 1 -- 172.21.15.27:0/1572638232 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7f3e940ef410 con 0x7f3eac158900 2024-01-15T18:09:19.608 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-01-15T18:09:19.608 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOc6VlH+DCLxAA1sKVut+3WjA/B+aYJ6/Jeg== 2024-01-15T18:09:19.608 INFO:teuthology.orchestra.run.smithi027.stdout: caps mds = "allow" 2024-01-15T18:09:19.608 INFO:teuthology.orchestra.run.smithi027.stdout: caps mon = "allow r" 2024-01-15T18:09:19.608 INFO:teuthology.orchestra.run.smithi027.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-15T18:09:19.609 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-01-15T18:09:19.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.602+0000 7f3eb4a00640 1 -- 172.21.15.27:0/1572638232 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e84066e90 msgr2=0x7f3e84069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:19.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.602+0000 7f3eb4a00640 1 --2- 172.21.15.27:0/1572638232 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e84066e90 0x7f3e84069350 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c000f70 tx=0x7f3e9c009040 comp rx=0 tx=0).stop 2024-01-15T18:09:19.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.602+0000 7f3eb4a00640 1 -- 172.21.15.27:0/1572638232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eac158900 msgr2=0x7f3eac14eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:19.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.602+0000 7f3eb4a00640 1 --2- 172.21.15.27:0/1572638232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eac158900 0x7f3eac14eae0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f3e9401a080 tx=0x7f3e940191f0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.606+0000 7f3eb4a00640 1 -- 172.21.15.27:0/1572638232 shutdown_connections 2024-01-15T18:09:19.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.606+0000 7f3eb4a00640 1 --2- 172.21.15.27:0/1572638232 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e84066e90 0x7f3e84069350 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.606+0000 7f3eb4a00640 1 --2- 172.21.15.27:0/1572638232 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3eac15f550 0x7f3eac0af0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.606+0000 7f3eb4a00640 1 --2- 172.21.15.27:0/1572638232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eac158900 0x7f3eac14eae0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.606+0000 7f3eb4a00640 1 --2- 172.21.15.27:0/1572638232 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3eac0cf110 0x7f3eac153d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.606+0000 7f3eb4a00640 1 -- 172.21.15.27:0/1572638232 >> 172.21.15.27:0/1572638232 conn(0x7f3eac0cb700 msgr2=0x7f3eac0b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:19.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.606+0000 7f3eb4a00640 1 -- 172.21.15.27:0/1572638232 shutdown_connections 2024-01-15T18:09:19.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.606+0000 7f3eb4a00640 1 -- 172.21.15.27:0/1572638232 wait complete. 2024-01-15T18:09:19.630 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:09:19.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c90200640 1 Processor -- start 2024-01-15T18:09:19.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c90200640 1 -- start start 2024-01-15T18:09:19.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c90200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f8c8815e7d0 0x7f8c880cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:19.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c90200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8c880cedc0 0x7f8c880cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:19.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c90200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c880b38b0 con 0x7f8c880ce2e0 2024-01-15T18:09:19.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c90200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c880cd550 con 0x7f8c8815e7d0 2024-01-15T18:09:19.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c90200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c880cd270 con 0x7f8c880cedc0 2024-01-15T18:09:19.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c8e200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8c880ce2e0 0x7f8c880cba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49592/0 (socket says 172.21.15.27:49592) 2024-01-15T18:09:19.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c8e200640 1 -- 172.21.15.27:0/1723146665 learned_addr learned my addr 172.21.15.27:0/1723146665 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:19.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c8d800640 1 --2- 172.21.15.27:0/1723146665 >> v2:172.21.15.93:3300/0 conn(0x7f8c8815e7d0 0x7f8c880cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:19.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c8ce00640 1 --2- 172.21.15.27:0/1723146665 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8c880cedc0 0x7f8c880cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:19.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c87e00640 1 -- 172.21.15.27:0/1723146665 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247172634 0 0) 0x7f8c880b38b0 con 0x7f8c880ce2e0 2024-01-15T18:09:19.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c87e00640 1 -- 172.21.15.27:0/1723146665 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c68003530 con 0x7f8c880ce2e0 2024-01-15T18:09:19.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c87e00640 1 -- 172.21.15.27:0/1723146665 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3731135859 0 0) 0x7f8c68003530 con 0x7f8c880ce2e0 2024-01-15T18:09:19.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c87e00640 1 -- 172.21.15.27:0/1723146665 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8c880cedc0 msgr2=0x7f8c880cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:19.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c87e00640 1 --2- 172.21.15.27:0/1723146665 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8c880cedc0 0x7f8c880cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c87e00640 1 -- 172.21.15.27:0/1723146665 >> v2:172.21.15.93:3300/0 conn(0x7f8c8815e7d0 msgr2=0x7f8c880cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:19.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c87e00640 1 --2- 172.21.15.27:0/1723146665 >> v2:172.21.15.93:3300/0 conn(0x7f8c8815e7d0 0x7f8c880cf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c87e00640 1 -- 172.21.15.27:0/1723146665 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c880b35d0 con 0x7f8c880ce2e0 2024-01-15T18:09:19.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c87e00640 1 -- 172.21.15.27:0/1723146665 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f8c80001a00 con 0x7f8c880ce2e0 2024-01-15T18:09:19.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c87e00640 1 -- 172.21.15.27:0/1723146665 >> v1:172.21.15.27:6789/0 conn(0x7f8c880ce2e0 legacy=0x7f8c880cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:19.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c87e00640 1 --2- 172.21.15.27:0/1723146665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c680039f0 0x7f8c68003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:19.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c87e00640 1 --2- 172.21.15.27:0/1723146665 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8c880cedc0 0x7f8c680047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:19.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c87e00640 1 --2- 172.21.15.27:0/1723146665 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8c68004d10 0x7f8c68005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:19.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c87e00640 1 -- 172.21.15.27:0/1723146665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c880cd550 con 0x7f8c680039f0 2024-01-15T18:09:19.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c87e00640 1 -- 172.21.15.27:0/1723146665 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c880cd270 con 0x7f8c68004d10 2024-01-15T18:09:19.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.766+0000 7f8c87e00640 1 -- 172.21.15.27:0/1723146665 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6800b990 con 0x7f8c880cedc0 2024-01-15T18:09:19.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.770+0000 7f8c8d800640 1 --2- 172.21.15.27:0/1723146665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c680039f0 0x7f8c68003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:19.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.770+0000 7f8c8e200640 1 --2- 172.21.15.27:0/1723146665 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8c68004d10 0x7f8c68005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:19.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.770+0000 7f8c8ce00640 1 --2- 172.21.15.27:0/1723146665 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8c880cedc0 0x7f8c680047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:19.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.770+0000 7f8c8d800640 1 -- 172.21.15.27:0/1723146665 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8c880cedc0 msgr2=0x7f8c680047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:19.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.770+0000 7f8c8d800640 1 --2- 172.21.15.27:0/1723146665 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8c880cedc0 0x7f8c680047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.770+0000 7f8c8d800640 1 -- 172.21.15.27:0/1723146665 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8c68004d10 msgr2=0x7f8c68005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:19.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.770+0000 7f8c8d800640 1 --2- 172.21.15.27:0/1723146665 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8c68004d10 0x7f8c68005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.770+0000 7f8c8d800640 1 -- 172.21.15.27:0/1723146665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c68003530 con 0x7f8c680039f0 2024-01-15T18:09:19.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.770+0000 7f8c8d800640 1 --2- 172.21.15.27:0/1723146665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c680039f0 0x7f8c68003e10 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f8c78009ae0 tx=0x7f8c7801ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffb086d63e0a95ea server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:19.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.770+0000 7f8c87e00640 1 -- 172.21.15.27:0/1723146665 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c780a5550 con 0x7f8c680039f0 2024-01-15T18:09:19.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.770+0000 7f8c87e00640 1 -- 172.21.15.27:0/1723146665 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c780a56b0 con 0x7f8c680039f0 2024-01-15T18:09:19.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.770+0000 7f8c87e00640 1 -- 172.21.15.27:0/1723146665 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c780a5a60 con 0x7f8c680039f0 2024-01-15T18:09:19.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.774+0000 7f8c90200640 1 -- 172.21.15.27:0/1723146665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c680039f0 msgr2=0x7f8c68003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:19.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.774+0000 7f8c90200640 1 --2- 172.21.15.27:0/1723146665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c680039f0 0x7f8c68003e10 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f8c78009ae0 tx=0x7f8c7801ca90 comp rx=0 tx=0).stop 2024-01-15T18:09:19.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.774+0000 7f8c8e200640 1 -- 172.21.15.27:0/1723146665 reap_dead start 2024-01-15T18:09:19.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.774+0000 7f8c90200640 1 -- 172.21.15.27:0/1723146665 shutdown_connections 2024-01-15T18:09:19.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.774+0000 7f8c90200640 1 -- 172.21.15.27:0/1723146665 >> 172.21.15.27:0/1723146665 conn(0x7f8c880b2a10 msgr2=0x7f8c880b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:19.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.774+0000 7f8c90200640 1 -- 172.21.15.27:0/1723146665 shutdown_connections 2024-01-15T18:09:19.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.774+0000 7f8c90200640 1 -- 172.21.15.27:0/1723146665 wait complete. 2024-01-15T18:09:19.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.774+0000 7f8c90200640 1 Processor -- start 2024-01-15T18:09:19.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.774+0000 7f8c90200640 1 -- start start 2024-01-15T18:09:19.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.774+0000 7f8c90200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8c680039f0 0x7f8c8815cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:19.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.774+0000 7f8c90200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8c880afa40 0x7f8c880afec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:19.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.774+0000 7f8c90200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c88154cf0 0x7f8c881551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:19.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.774+0000 7f8c90200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c880b0400 con 0x7f8c88154cf0 2024-01-15T18:09:19.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.774+0000 7f8c90200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c880cae00 con 0x7f8c880afa40 2024-01-15T18:09:19.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.774+0000 7f8c90200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c880ca0a0 con 0x7f8c680039f0 2024-01-15T18:09:19.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.774+0000 7f8c8e200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c88154cf0 0x7f8c881551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:19.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.774+0000 7f8c8e200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c88154cf0 0x7f8c881551a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34434/0 (socket says 172.21.15.27:34434) 2024-01-15T18:09:19.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.774+0000 7f8c8e200640 1 -- 172.21.15.27:0/3519021241 learned_addr learned my addr 172.21.15.27:0/3519021241 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:19.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.774+0000 7f8c8d800640 1 --2- 172.21.15.27:0/3519021241 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8c680039f0 0x7f8c8815cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:19.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.774+0000 7f8c8ce00640 1 --2- 172.21.15.27:0/3519021241 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8c880afa40 0x7f8c880afec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:19.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.778+0000 7f8c8e200640 1 -- 172.21.15.27:0/3519021241 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8c680039f0 msgr2=0x7f8c8815cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:19.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.778+0000 7f8c8e200640 1 --2- 172.21.15.27:0/3519021241 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8c680039f0 0x7f8c8815cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.778+0000 7f8c8e200640 1 -- 172.21.15.27:0/3519021241 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8c880afa40 msgr2=0x7f8c880afec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:19.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.778+0000 7f8c8e200640 1 --2- 172.21.15.27:0/3519021241 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8c880afa40 0x7f8c880afec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.778+0000 7f8c8e200640 1 -- 172.21.15.27:0/3519021241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8815d290 con 0x7f8c88154cf0 2024-01-15T18:09:19.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.778+0000 7f8c8e200640 1 --2- 172.21.15.27:0/3519021241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c88154cf0 0x7f8c881551a0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f8c88071ab0 tx=0x7f8c80007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:19.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.778+0000 7f8c85600640 1 -- 172.21.15.27:0/3519021241 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8000ab40 con 0x7f8c88154cf0 2024-01-15T18:09:19.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.778+0000 7f8c85600640 1 -- 172.21.15.27:0/3519021241 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c8000acc0 con 0x7f8c88154cf0 2024-01-15T18:09:19.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.778+0000 7f8c85600640 1 -- 172.21.15.27:0/3519021241 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c80013630 con 0x7f8c88154cf0 2024-01-15T18:09:19.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.778+0000 7f8c90200640 1 -- 172.21.15.27:0/3519021241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c880ca320 con 0x7f8c88154cf0 2024-01-15T18:09:19.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.778+0000 7f8c90200640 1 -- 172.21.15.27:0/3519021241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c880ca8e0 con 0x7f8c88154cf0 2024-01-15T18:09:19.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.778+0000 7f8c90200640 1 -- 172.21.15.27:0/3519021241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c8804e850 con 0x7f8c88154cf0 2024-01-15T18:09:19.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.778+0000 7f8c85600640 1 -- 172.21.15.27:0/3519021241 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f8c80013790 con 0x7f8c88154cf0 2024-01-15T18:09:19.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.778+0000 7f8c85600640 1 --2- 172.21.15.27:0/3519021241 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8c60066ec0 0x7f8c60069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:19.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.778+0000 7f8c85600640 1 -- 172.21.15.27:0/3519021241 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f8c800853d0 con 0x7f8c88154cf0 2024-01-15T18:09:19.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.782+0000 7f8c8d800640 1 --2- 172.21.15.27:0/3519021241 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8c60066ec0 0x7f8c60069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:19.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.782+0000 7f8c8d800640 1 --2- 172.21.15.27:0/3519021241 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8c60066ec0 0x7f8c60069380 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8c780af000 tx=0x7f8c780046d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:19.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.786+0000 7f8c85600640 1 -- 172.21.15.27:0/3519021241 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8c80051dd0 con 0x7f8c88154cf0 2024-01-15T18:09:19.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.974+0000 7f8c90200640 1 -- 172.21.15.27:0/3519021241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c8815b970 con 0x7f8c88154cf0 2024-01-15T18:09:19.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.974+0000 7f8c85600640 1 -- 172.21.15.27:0/3519021241 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+4910 (secure 0 0 0) 0x7f8c800553f0 con 0x7f8c88154cf0 2024-01-15T18:09:19.980 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:09:19.980 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":37,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1939851998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1939851998},{"type":"v1","addr":"172.21.15.27:6835","nonce":1939851998}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":5759,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/255207804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":255207804},{"type":"v1","addr":"172.21.15.93:6837","nonce":255207804}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":6244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/1075225596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":1075225596},{"type":"v1","addr":"172.21.15.93:6836","nonce":1075225596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:09:09.513597+0000","modified":"2024-01-15T18:09:12.543751+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6480},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6480":{"gid":6480,"name":"c","rank":0,"incarnation":33,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6834/1969845074","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1969845074},{"type":"v1","addr":"172.21.15.27:6834","nonce":1969845074}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-01-15T18:09:19.980 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 37 2024-01-15T18:09:19.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.974+0000 7f8c90200640 1 -- 172.21.15.27:0/3519021241 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8c60066ec0 msgr2=0x7f8c60069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:19.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.974+0000 7f8c90200640 1 --2- 172.21.15.27:0/3519021241 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8c60066ec0 0x7f8c60069380 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8c780af000 tx=0x7f8c780046d0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.974+0000 7f8c90200640 1 -- 172.21.15.27:0/3519021241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c88154cf0 msgr2=0x7f8c881551a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:19.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.974+0000 7f8c90200640 1 --2- 172.21.15.27:0/3519021241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c88154cf0 0x7f8c881551a0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f8c88071ab0 tx=0x7f8c80007b10 comp rx=0 tx=0).stop 2024-01-15T18:09:19.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.978+0000 7f8c90200640 1 -- 172.21.15.27:0/3519021241 shutdown_connections 2024-01-15T18:09:19.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.978+0000 7f8c90200640 1 --2- 172.21.15.27:0/3519021241 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8c60066ec0 0x7f8c60069380 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.978+0000 7f8c90200640 1 --2- 172.21.15.27:0/3519021241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c88154cf0 0x7f8c881551a0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.978+0000 7f8c90200640 1 --2- 172.21.15.27:0/3519021241 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8c880afa40 0x7f8c880afec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.978+0000 7f8c90200640 1 --2- 172.21.15.27:0/3519021241 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8c680039f0 0x7f8c8815cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.978+0000 7f8c90200640 1 -- 172.21.15.27:0/3519021241 >> 172.21.15.27:0/3519021241 conn(0x7f8c880b2a10 msgr2=0x7f8c88165400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:19.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.978+0000 7f8c90200640 1 -- 172.21.15.27:0/3519021241 shutdown_connections 2024-01-15T18:09:19.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:19.978+0000 7f8c90200640 1 -- 172.21.15.27:0/3519021241 wait complete. 2024-01-15T18:09:20.405 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 37, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-15T18:09:09.513597+0000', 'modified': '2024-01-15T18:09:12.543751+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6480}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6480': {'gid': 6480, 'name': 'c', 'rank': 0, 'incarnation': 33, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6834/1969845074', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6832', 'nonce': 1969845074}, {'type': 'v1', 'addr': '172.21.15.27:6834', 'nonce': 1969845074}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-15T18:09:20.405 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-15T18:09:20.406 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-15T18:09:20.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.542+0000 7f86ba600640 1 Processor -- start 2024-01-15T18:09:20.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.542+0000 7f86ba600640 1 -- start start 2024-01-15T18:09:20.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.546+0000 7f86ba600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f86b415d7c0 0x7f86b40b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.546+0000 7f86ba600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f86b40b1d90 0x7f86b4099a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.546+0000 7f86ba600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f86b406bc70 0x7f86b4099f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.546+0000 7f86ba600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86b40b1300 con 0x7f86b406bc70 2024-01-15T18:09:20.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.546+0000 7f86ba600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86b40b1aa0 con 0x7f86b40b1d90 2024-01-15T18:09:20.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.546+0000 7f86ba600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86b40b25a0 con 0x7f86b415d7c0 2024-01-15T18:09:20.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.546+0000 7f86b3e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f86b406bc70 0x7f86b4099f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.546+0000 7f86b3e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f86b406bc70 0x7f86b4099f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34440/0 (socket says 172.21.15.27:34440) 2024-01-15T18:09:20.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.546+0000 7f86b3e00640 1 -- 172.21.15.27:0/651526670 learned_addr learned my addr 172.21.15.27:0/651526670 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:20.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.546+0000 7f86b3400640 1 --2- 172.21.15.27:0/651526670 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f86b415d7c0 0x7f86b40b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.546+0000 7f86b2a00640 1 --2- 172.21.15.27:0/651526670 >> v2:172.21.15.93:3300/0 conn(0x7f86b40b1d90 0x7f86b4099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.546+0000 7f86b3e00640 1 -- 172.21.15.27:0/651526670 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f86b415d7c0 msgr2=0x7f86b40b35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:20.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.546+0000 7f86b3e00640 1 --2- 172.21.15.27:0/651526670 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f86b415d7c0 0x7f86b40b35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.546+0000 7f86b3e00640 1 -- 172.21.15.27:0/651526670 >> v2:172.21.15.93:3300/0 conn(0x7f86b40b1d90 msgr2=0x7f86b4099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:20.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.546+0000 7f86b3e00640 1 --2- 172.21.15.27:0/651526670 >> v2:172.21.15.93:3300/0 conn(0x7f86b40b1d90 0x7f86b4099a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.546+0000 7f86b3e00640 1 -- 172.21.15.27:0/651526670 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b40b14e0 con 0x7f86b406bc70 2024-01-15T18:09:20.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.546+0000 7f86b3e00640 1 --2- 172.21.15.27:0/651526670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86b406bc70 0x7f86b4099f50 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f86b4072050 tx=0x7f86ac01c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8ada0d59b80786d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:20.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.550+0000 7f86b2000640 1 -- 172.21.15.27:0/651526670 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ac0a5550 con 0x7f86b406bc70 2024-01-15T18:09:20.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.550+0000 7f86b2000640 1 -- 172.21.15.27:0/651526670 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86ac0a56b0 con 0x7f86b406bc70 2024-01-15T18:09:20.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.550+0000 7f86b2000640 1 -- 172.21.15.27:0/651526670 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ac0a5a40 con 0x7f86b406bc70 2024-01-15T18:09:20.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.550+0000 7f86ba600640 1 -- 172.21.15.27:0/651526670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86b406bc70 msgr2=0x7f86b4099f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:20.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.550+0000 7f86ba600640 1 --2- 172.21.15.27:0/651526670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86b406bc70 0x7f86b4099f50 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f86b4072050 tx=0x7f86ac01c8b0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.550+0000 7f86ba600640 1 -- 172.21.15.27:0/651526670 shutdown_connections 2024-01-15T18:09:20.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.550+0000 7f86ba600640 1 --2- 172.21.15.27:0/651526670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86b406bc70 0x7f86b4099f50 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.550+0000 7f86ba600640 1 --2- 172.21.15.27:0/651526670 >> v2:172.21.15.93:3300/0 conn(0x7f86b40b1d90 0x7f86b4099a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.550+0000 7f86ba600640 1 --2- 172.21.15.27:0/651526670 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f86b415d7c0 0x7f86b40b35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.550+0000 7f86ba600640 1 -- 172.21.15.27:0/651526670 >> 172.21.15.27:0/651526670 conn(0x7f86b40b2a80 msgr2=0x7f86b40b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:20.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.550+0000 7f86ba600640 1 -- 172.21.15.27:0/651526670 shutdown_connections 2024-01-15T18:09:20.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.550+0000 7f86ba600640 1 -- 172.21.15.27:0/651526670 wait complete. 2024-01-15T18:09:20.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.550+0000 7f86ba600640 1 Processor -- start 2024-01-15T18:09:20.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.550+0000 7f86ba600640 1 -- start start 2024-01-15T18:09:20.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.550+0000 7f86ba600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f86b406bc70 0x7f86b4151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.550+0000 7f86ba600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f86b40b1d90 0x7f86b4154c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.550+0000 7f86ba600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86b415d7c0 0x7f86b40cd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.550+0000 7f86ba600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b415b3d0 con 0x7f86b415d7c0 2024-01-15T18:09:20.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.550+0000 7f86ba600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b415adc0 con 0x7f86b406bc70 2024-01-15T18:09:20.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.550+0000 7f86ba600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86b4159470 con 0x7f86b40b1d90 2024-01-15T18:09:20.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.554+0000 7f86b3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86b415d7c0 0x7f86b40cd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.554+0000 7f86b3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86b415d7c0 0x7f86b40cd9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34452/0 (socket says 172.21.15.27:34452) 2024-01-15T18:09:20.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.554+0000 7f86b3e00640 1 -- 172.21.15.27:0/3968450725 learned_addr learned my addr 172.21.15.27:0/3968450725 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:20.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.554+0000 7f86b3400640 1 --2- 172.21.15.27:0/3968450725 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f86b406bc70 0x7f86b4151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.554+0000 7f86b2a00640 1 --2- 172.21.15.27:0/3968450725 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f86b40b1d90 0x7f86b4154c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.554+0000 7f86b3e00640 1 -- 172.21.15.27:0/3968450725 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f86b40b1d90 msgr2=0x7f86b4154c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:20.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.554+0000 7f86b3e00640 1 --2- 172.21.15.27:0/3968450725 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f86b40b1d90 0x7f86b4154c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.554+0000 7f86b3e00640 1 -- 172.21.15.27:0/3968450725 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f86b406bc70 msgr2=0x7f86b4151ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:20.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.554+0000 7f86b3e00640 1 --2- 172.21.15.27:0/3968450725 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f86b406bc70 0x7f86b4151ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.554+0000 7f86b3e00640 1 -- 172.21.15.27:0/3968450725 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ac009b30 con 0x7f86b415d7c0 2024-01-15T18:09:20.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.554+0000 7f86b3e00640 1 --2- 172.21.15.27:0/3968450725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86b415d7c0 0x7f86b40cd9d0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f86ac01af60 tx=0x7f86ac019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:20.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.554+0000 7f86ab400640 1 -- 172.21.15.27:0/3968450725 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ac0a5550 con 0x7f86b415d7c0 2024-01-15T18:09:20.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.554+0000 7f86ab400640 1 -- 172.21.15.27:0/3968450725 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86ac0b8670 con 0x7f86b415d7c0 2024-01-15T18:09:20.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.554+0000 7f86ab400640 1 -- 172.21.15.27:0/3968450725 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ac0b8a20 con 0x7f86b415d7c0 2024-01-15T18:09:20.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.554+0000 7f86ba600640 1 -- 172.21.15.27:0/3968450725 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86b4159750 con 0x7f86b415d7c0 2024-01-15T18:09:20.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.554+0000 7f86ba600640 1 -- 172.21.15.27:0/3968450725 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86b41551b0 con 0x7f86b415d7c0 2024-01-15T18:09:20.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.554+0000 7f86ab400640 1 -- 172.21.15.27:0/3968450725 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f86ac0b8b80 con 0x7f86b415d7c0 2024-01-15T18:09:20.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86ab400640 1 --2- 172.21.15.27:0/3968450725 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f868c066e90 0x7f868c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86ab400640 1 -- 172.21.15.27:0/3968450725 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f86ac11ffd0 con 0x7f86b415d7c0 2024-01-15T18:09:20.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86b3400640 1 --2- 172.21.15.27:0/3968450725 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f868c066e90 0x7f868c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86bd98a000 1 Processor -- start 2024-01-15T18:09:20.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86bd98a000 1 -- start start 2024-01-15T18:09:20.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86b3400640 1 --2- 172.21.15.27:0/3968450725 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f868c066e90 0x7f868c069350 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f86b415d1e0 tx=0x7f86a4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:20.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86bd98a000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec3dfc90 0x559bec3e0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86bd98a000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x559bec3e05b0 0x559bec395de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86bd98a000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x559bec396320 0x559bec396700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86bd98a000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bec200130 con 0x559bec3dfc90 2024-01-15T18:09:20.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86bd98a000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bec1ed9a0 con 0x559bec396320 2024-01-15T18:09:20.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86bd98a000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x559bec1eaed0 con 0x559bec3e05b0 2024-01-15T18:09:20.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86b3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec3dfc90 0x559bec3e0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86b3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec3dfc90 0x559bec3e0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34462/0 (socket says 172.21.15.27:34462) 2024-01-15T18:09:20.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86b3400640 1 -- 172.21.15.27:0/273373917 learned_addr learned my addr 172.21.15.27:0/273373917 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:20.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86b3e00640 1 --2- 172.21.15.27:0/273373917 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x559bec396320 0x559bec396700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86b2a00640 1 --2- 172.21.15.27:0/273373917 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x559bec3e05b0 0x559bec395de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86b3400640 1 -- 172.21.15.27:0/273373917 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x559bec3e05b0 msgr2=0x559bec395de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:20.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86b3400640 1 --2- 172.21.15.27:0/273373917 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x559bec3e05b0 0x559bec395de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86b3400640 1 -- 172.21.15.27:0/273373917 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x559bec396320 msgr2=0x559bec396700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:20.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86b3400640 1 --2- 172.21.15.27:0/273373917 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x559bec396320 0x559bec396700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86b3400640 1 -- 172.21.15.27:0/273373917 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559bebede890 con 0x559bec3dfc90 2024-01-15T18:09:20.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86b3400640 1 --2- 172.21.15.27:0/273373917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec3dfc90 0x559bec3e0070 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f86a4011d80 tx=0x7f86a4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c0b9fa60e0f7cff server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:20.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86a3e00640 1 -- 172.21.15.27:0/273373917 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a4014bd0 con 0x559bec3dfc90 2024-01-15T18:09:20.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86a3e00640 1 -- 172.21.15.27:0/273373917 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a4014d30 con 0x559bec3dfc90 2024-01-15T18:09:20.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.558+0000 7f86a3e00640 1 -- 172.21.15.27:0/273373917 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a4014ea0 con 0x559bec3dfc90 2024-01-15T18:09:20.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86bd98a000 1 -- 172.21.15.27:0/273373917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec3dfc90 msgr2=0x559bec3e0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:20.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86bd98a000 1 --2- 172.21.15.27:0/273373917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec3dfc90 0x559bec3e0070 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f86a4011d80 tx=0x7f86a4014000 comp rx=0 tx=0).stop 2024-01-15T18:09:20.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86bd98a000 1 -- 172.21.15.27:0/273373917 shutdown_connections 2024-01-15T18:09:20.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86bd98a000 1 --2- 172.21.15.27:0/273373917 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x559bec396320 0x559bec396700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86bd98a000 1 --2- 172.21.15.27:0/273373917 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x559bec3e05b0 0x559bec395de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86bd98a000 1 --2- 172.21.15.27:0/273373917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec3dfc90 0x559bec3e0070 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86bd98a000 1 -- 172.21.15.27:0/273373917 >> 172.21.15.27:0/273373917 conn(0x559bec3922d0 msgr2=0x559bec3946c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:20.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86bd98a000 1 -- 172.21.15.27:0/273373917 shutdown_connections 2024-01-15T18:09:20.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86bd98a000 1 -- 172.21.15.27:0/273373917 wait complete. 2024-01-15T18:09:20.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86bd98a000 1 -- start start 2024-01-15T18:09:20.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86bd98a000 1 Processor -- start 2024-01-15T18:09:20.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86bd98a000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x559bec396320 0x559bec3c4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86bd98a000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x559bec3dfc90 0x559bec3c4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86bd98a000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec3e05b0 0x559bec3c5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86bd98a000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bec1e67c0 con 0x559bec3e05b0 2024-01-15T18:09:20.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86bd98a000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bec19ebf0 con 0x559bec396320 2024-01-15T18:09:20.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86bd98a000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x559bec1a4a90 con 0x559bec3dfc90 2024-01-15T18:09:20.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86bd98a000 10 client.0 _ll_register_callbacks cb 0x559bec3813c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:09:20.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86b3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec3e05b0 0x559bec3c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86b3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec3e05b0 0x559bec3c5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34478/0 (socket says 172.21.15.27:34478) 2024-01-15T18:09:20.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86b3e00640 1 -- 172.21.15.27:0/4110770228 learned_addr learned my addr 172.21.15.27:0/4110770228 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:20.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86b3400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x559bec396320 0x559bec3c4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86b2a00640 1 --2- 172.21.15.27:0/4110770228 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x559bec3dfc90 0x559bec3c4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86b3e00640 1 -- 172.21.15.27:0/4110770228 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x559bec3dfc90 msgr2=0x559bec3c4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:20.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86b3e00640 1 --2- 172.21.15.27:0/4110770228 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x559bec3dfc90 0x559bec3c4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86b3e00640 1 -- 172.21.15.27:0/4110770228 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x559bec396320 msgr2=0x559bec3c4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:20.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86b3e00640 1 --2- 172.21.15.27:0/4110770228 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x559bec396320 0x559bec3c4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86b3e00640 1 -- 172.21.15.27:0/4110770228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a4011a60 con 0x559bec3e05b0 2024-01-15T18:09:20.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86b3e00640 1 --2- 172.21.15.27:0/4110770228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec3e05b0 0x559bec3c5530 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f86ac0f04d0 tx=0x7f86ac0095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b802a7618c6e109f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:20.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86a0c00640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-15T18:09:20.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86a0c00640 1 -- 172.21.15.27:0/4110770228 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ac11db00 con 0x559bec3e05b0 2024-01-15T18:09:20.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86a0c00640 1 -- 172.21.15.27:0/4110770228 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f86ac0f8a20 con 0x559bec3e05b0 2024-01-15T18:09:20.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.562+0000 7f86bd98a000 1 -- 172.21.15.27:0/4110770228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559bec1a8470 con 0x559bec3e05b0 2024-01-15T18:09:20.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86a0c00640 1 -- 172.21.15.27:0/4110770228 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f86ac0efcc0 con 0x559bec3e05b0 2024-01-15T18:09:20.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86bd98a000 10 client.6645 fetch_fsmap learned FSMap version 37 2024-01-15T18:09:20.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86bd98a000 1 -- 172.21.15.27:0/4110770228 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x559bec1a7190 con 0x559bec3e05b0 2024-01-15T18:09:20.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86a0c00640 1 -- 172.21.15.27:0/4110770228 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559bec1a8470 con 0x559bec3e05b0 2024-01-15T18:09:20.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86a0c00640 1 -- 172.21.15.27:0/4110770228 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 37) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f86ac005d20 con 0x559bec3e05b0 2024-01-15T18:09:20.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86bd98a000 10 client.6645 fetch_fsmap finished waiting for FSMap version 37 2024-01-15T18:09:20.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86bd98a000 10 client.6645 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-15T18:09:20.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86bd98a000 20 client.6645 populate_metadata read hostname 'smithi027' 2024-01-15T18:09:20.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86bd98a000 1 --2- 172.21.15.27:0/4110770228 >> [v2:172.21.15.27:6832/1969845074,v1:172.21.15.27:6834/1969845074] conn(0x559bec3c5f40 0x559bec3f1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86bd98a000 4 client.6645 mds_command: new command op to 6480 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-15T18:09:20.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86bd98a000 1 -- 172.21.15.27:0/4110770228 --> [v2:172.21.15.27:6832/1969845074,v1:172.21.15.27:6834/1969845074] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559bec1abb50 con 0x559bec3c5f40 2024-01-15T18:09:20.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86b3400640 1 --2- 172.21.15.27:0/4110770228 >> [v2:172.21.15.27:6832/1969845074,v1:172.21.15.27:6834/1969845074] conn(0x559bec3c5f40 0x559bec3f1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86b3400640 1 --2- 172.21.15.27:0/4110770228 >> [v2:172.21.15.27:6832/1969845074,v1:172.21.15.27:6834/1969845074] conn(0x559bec3c5f40 0x559bec3f1f50 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1214405d96fd0ae server_cookie=feae7eee696893d1 in_seq=0 out_seq=0 2024-01-15T18:09:20.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86a0c00640 10 client.6645 ms_handle_connect on v2:172.21.15.27:6832/1969845074 2024-01-15T18:09:20.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86a0c00640 1 -- 172.21.15.27:0/4110770228 <== mds.0 v2:172.21.15.27:6832/1969845074 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559bec1abb50 con 0x559bec3c5f40 2024-01-15T18:09:20.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86a0c00640 10 client.6645 handle_command_reply: tid=0 2024-01-15T18:09:20.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86bd98a000 1 client.6645 shutdown 2024-01-15T18:09:20.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86bd98a000 1 -- 172.21.15.27:0/4110770228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec3e05b0 msgr2=0x559bec3c5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:20.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86bd98a000 1 --2- 172.21.15.27:0/4110770228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec3e05b0 0x559bec3c5530 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f86ac0f04d0 tx=0x7f86ac0095d0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86bd98a000 1 -- 172.21.15.27:0/4110770228 shutdown_connections 2024-01-15T18:09:20.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86bd98a000 1 --2- 172.21.15.27:0/4110770228 >> [v2:172.21.15.27:6832/1969845074,v1:172.21.15.27:6834/1969845074] conn(0x559bec3c5f40 0x559bec3f1f50 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86bd98a000 1 --2- 172.21.15.27:0/4110770228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec3e05b0 0x559bec3c5530 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86bd98a000 1 --2- 172.21.15.27:0/4110770228 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x559bec3dfc90 0x559bec3c4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86bd98a000 1 --2- 172.21.15.27:0/4110770228 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x559bec396320 0x559bec3c4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86a0c00640 0 client.6645 ms_handle_reset on v2:172.21.15.27:6832/1969845074 2024-01-15T18:09:20.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86bd98a000 1 -- 172.21.15.27:0/4110770228 >> 172.21.15.27:0/4110770228 conn(0x559bec3922d0 msgr2=0x559bec3946c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:20.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86bd98a000 1 -- 172.21.15.27:0/4110770228 shutdown_connections 2024-01-15T18:09:20.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86bd98a000 1 -- 172.21.15.27:0/4110770228 wait complete. 2024-01-15T18:09:20.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.566+0000 7f86bd98a000 20 client.6645 trim_cache size 0 max 16384 2024-01-15T18:09:20.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.582+0000 7f86bd98a000 1 Processor -- start 2024-01-15T18:09:20.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.582+0000 7f86bd98a000 1 -- start start 2024-01-15T18:09:20.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.582+0000 7f86bd98a000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x559bec396320 0x559bec3de3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.582+0000 7f86bd98a000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x559bec3dfc90 0x559bec3de920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.582+0000 7f86bd98a000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec3e05b0 0x559bec396700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.582+0000 7f86bd98a000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bebe9f940 con 0x559bec3e05b0 2024-01-15T18:09:20.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.582+0000 7f86bd98a000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bec1e3bc0 con 0x559bec3dfc90 2024-01-15T18:09:20.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.582+0000 7f86bd98a000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x559bec39bfb0 con 0x559bec396320 2024-01-15T18:09:20.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86b3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec3e05b0 0x559bec396700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86b3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec3e05b0 0x559bec396700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34484/0 (socket says 172.21.15.27:34484) 2024-01-15T18:09:20.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86b3e00640 1 -- 172.21.15.27:0/2761731964 learned_addr learned my addr 172.21.15.27:0/2761731964 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:20.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86b3e00640 1 -- 172.21.15.27:0/2761731964 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x559bec396320 msgr2=0x559bec3de3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:20.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86b3e00640 1 --2- 172.21.15.27:0/2761731964 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x559bec396320 0x559bec3de3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86b3e00640 1 -- 172.21.15.27:0/2761731964 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x559bec3dfc90 msgr2=0x559bec3de920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:20.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86b2a00640 1 --2- 172.21.15.27:0/2761731964 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x559bec3dfc90 0x559bec3de920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86b3e00640 1 --2- 172.21.15.27:0/2761731964 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x559bec3dfc90 0x559bec3de920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86b3e00640 1 -- 172.21.15.27:0/2761731964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559bec3999e0 con 0x559bec3e05b0 2024-01-15T18:09:20.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86b2a00640 1 --2- 172.21.15.27:0/2761731964 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x559bec3dfc90 0x559bec3de920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:20.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86b3e00640 1 --2- 172.21.15.27:0/2761731964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec3e05b0 0x559bec396700 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f86ac0a5fd0 tx=0x7f86ac0ba000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98f322f8a12fef69 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:20.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86a8c00640 1 -- 172.21.15.27:0/2761731964 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ac0efcc0 con 0x559bec3e05b0 2024-01-15T18:09:20.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86a8c00640 1 -- 172.21.15.27:0/2761731964 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86ac0f0130 con 0x559bec3e05b0 2024-01-15T18:09:20.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86a8c00640 1 -- 172.21.15.27:0/2761731964 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ac0f8a20 con 0x559bec3e05b0 2024-01-15T18:09:20.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86bd98a000 1 -- 172.21.15.27:0/2761731964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec3e05b0 msgr2=0x559bec396700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:20.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86bd98a000 1 --2- 172.21.15.27:0/2761731964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec3e05b0 0x559bec396700 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f86ac0a5fd0 tx=0x7f86ac0ba000 comp rx=0 tx=0).stop 2024-01-15T18:09:20.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86bd98a000 1 -- 172.21.15.27:0/2761731964 shutdown_connections 2024-01-15T18:09:20.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86bd98a000 1 --2- 172.21.15.27:0/2761731964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec3e05b0 0x559bec396700 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86bd98a000 1 --2- 172.21.15.27:0/2761731964 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x559bec3dfc90 0x559bec3de920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86bd98a000 1 --2- 172.21.15.27:0/2761731964 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x559bec396320 0x559bec3de3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86bd98a000 1 -- 172.21.15.27:0/2761731964 >> 172.21.15.27:0/2761731964 conn(0x559bec3922d0 msgr2=0x559bec3946c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:20.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86bd98a000 1 -- 172.21.15.27:0/2761731964 shutdown_connections 2024-01-15T18:09:20.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86bd98a000 1 -- 172.21.15.27:0/2761731964 wait complete. 2024-01-15T18:09:20.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86bd98a000 1 -- start start 2024-01-15T18:09:20.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86bd98a000 1 Processor -- start 2024-01-15T18:09:20.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86bd98a000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec396320 0x559bec3f3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86bd98a000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x559bec3dfc90 0x559bec3fd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86bd98a000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x559bec3e05b0 0x559bec3fdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86bd98a000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bec1e9320 con 0x559bec396320 2024-01-15T18:09:20.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86bd98a000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x559bec228140 con 0x559bec3e05b0 2024-01-15T18:09:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86bd98a000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x559bec1e3bc0 con 0x559bec3dfc90 2024-01-15T18:09:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.586+0000 7f86bd98a000 10 client.0 _ll_register_callbacks cb 0x559bec234c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:09:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86b3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec396320 0x559bec3f3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86b3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec396320 0x559bec3f3390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34486/0 (socket says 172.21.15.27:34486) 2024-01-15T18:09:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86b3400640 1 -- 172.21.15.27:0/1924290079 learned_addr learned my addr 172.21.15.27:0/1924290079 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86b2a00640 1 --2- 172.21.15.27:0/1924290079 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x559bec3dfc90 0x559bec3fd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86b3e00640 1 --2- 172.21.15.27:0/1924290079 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x559bec3e05b0 0x559bec3fdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86b3400640 1 -- 172.21.15.27:0/1924290079 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x559bec3dfc90 msgr2=0x559bec3fd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86b3400640 1 --2- 172.21.15.27:0/1924290079 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x559bec3dfc90 0x559bec3fd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86b3400640 1 -- 172.21.15.27:0/1924290079 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x559bec3e05b0 msgr2=0x559bec3fdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86b3400640 1 --2- 172.21.15.27:0/1924290079 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x559bec3e05b0 0x559bec3fdf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86b3400640 1 -- 172.21.15.27:0/1924290079 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559bec1a7190 con 0x559bec396320 2024-01-15T18:09:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86b3e00640 1 --2- 172.21.15.27:0/1924290079 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x559bec3e05b0 0x559bec3fdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:09:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86b3400640 1 --2- 172.21.15.27:0/1924290079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec396320 0x559bec3f3390 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f86a4011690 tx=0x7f86a40046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31e64f38c71dc7f6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86a0c00640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-15T18:09:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86a0c00640 1 -- 172.21.15.27:0/1924290079 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a402e050 con 0x559bec396320 2024-01-15T18:09:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86a0c00640 1 -- 172.21.15.27:0/1924290079 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f86a401c040 con 0x559bec396320 2024-01-15T18:09:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86bd98a000 1 -- 172.21.15.27:0/1924290079 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559bec1e5110 con 0x559bec396320 2024-01-15T18:09:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86a0c00640 1 -- 172.21.15.27:0/1924290079 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f86a4004230 con 0x559bec396320 2024-01-15T18:09:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86bd98a000 10 client.6651 fetch_fsmap learned FSMap version 37 2024-01-15T18:09:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86bd98a000 1 -- 172.21.15.27:0/1924290079 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x559bec18d490 con 0x559bec396320 2024-01-15T18:09:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86a0c00640 1 -- 172.21.15.27:0/1924290079 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559bec1e5110 con 0x559bec396320 2024-01-15T18:09:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86a0c00640 1 -- 172.21.15.27:0/1924290079 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 37) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f86a4004390 con 0x559bec396320 2024-01-15T18:09:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86bd98a000 10 client.6651 fetch_fsmap finished waiting for FSMap version 37 2024-01-15T18:09:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86bd98a000 10 client.6651 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-15T18:09:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86bd98a000 20 client.6651 populate_metadata read hostname 'smithi027' 2024-01-15T18:09:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86bd98a000 1 --2- 172.21.15.27:0/1924290079 >> [v2:172.21.15.27:6832/1969845074,v1:172.21.15.27:6834/1969845074] conn(0x559bec446e40 0x559bec449230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86bd98a000 4 client.6651 mds_command: new command op to 6480 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-15T18:09:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86bd98a000 1 -- 172.21.15.27:0/1924290079 --> [v2:172.21.15.27:6832/1969845074,v1:172.21.15.27:6834/1969845074] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559bec2149a0 con 0x559bec446e40 2024-01-15T18:09:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86b2a00640 1 --2- 172.21.15.27:0/1924290079 >> [v2:172.21.15.27:6832/1969845074,v1:172.21.15.27:6834/1969845074] conn(0x559bec446e40 0x559bec449230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86b2a00640 1 --2- 172.21.15.27:0/1924290079 >> [v2:172.21.15.27:6832/1969845074,v1:172.21.15.27:6834/1969845074] conn(0x559bec446e40 0x559bec449230 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1015aa8dbcc7933 server_cookie=2011e16f6b879b47 in_seq=0 out_seq=0 2024-01-15T18:09:20.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86a0c00640 10 client.6651 ms_handle_connect on v2:172.21.15.27:6832/1969845074 2024-01-15T18:09:20.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86a0c00640 1 -- 172.21.15.27:0/1924290079 <== mds.0 v2:172.21.15.27:6832/1969845074 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+319 (crc 0 0 0) 0x559bec2149a0 con 0x559bec446e40 2024-01-15T18:09:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86a0c00640 10 client.6651 handle_command_reply: tid=0 2024-01-15T18:09:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86bd98a000 1 client.6651 shutdown 2024-01-15T18:09:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86bd98a000 1 -- 172.21.15.27:0/1924290079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec396320 msgr2=0x559bec3f3390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86bd98a000 1 --2- 172.21.15.27:0/1924290079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec396320 0x559bec3f3390 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f86a4011690 tx=0x7f86a40046d0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86bd98a000 1 -- 172.21.15.27:0/1924290079 shutdown_connections 2024-01-15T18:09:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86bd98a000 1 --2- 172.21.15.27:0/1924290079 >> [v2:172.21.15.27:6832/1969845074,v1:172.21.15.27:6834/1969845074] conn(0x559bec446e40 0x559bec449230 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86bd98a000 1 --2- 172.21.15.27:0/1924290079 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x559bec3e05b0 0x559bec3fdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86bd98a000 1 --2- 172.21.15.27:0/1924290079 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x559bec3dfc90 0x559bec3fd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86bd98a000 1 --2- 172.21.15.27:0/1924290079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x559bec396320 0x559bec3f3390 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86a0c00640 0 client.6651 ms_handle_reset on v2:172.21.15.27:6832/1969845074 2024-01-15T18:09:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86bd98a000 1 -- 172.21.15.27:0/1924290079 >> 172.21.15.27:0/1924290079 conn(0x559bec3922d0 msgr2=0x559bec394f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86bd98a000 1 -- 172.21.15.27:0/1924290079 shutdown_connections 2024-01-15T18:09:20.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.590+0000 7f86bd98a000 1 -- 172.21.15.27:0/1924290079 wait complete. 2024-01-15T18:09:20.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.594+0000 7f86bd98a000 20 client.6651 trim_cache size 0 max 16384 2024-01-15T18:09:20.601 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:09:20.601 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:09:20.601 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-01-15T18:09:20.601 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 6480, 2024-01-15T18:09:20.601 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-01-15T18:09:20.601 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-01-15T18:09:20.601 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-01-15T18:09:20.601 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 10.450414511, 2024-01-15T18:09:20.601 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 37, 2024-01-15T18:09:20.601 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 58, 2024-01-15T18:09:20.601 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 58, 2024-01-15T18:09:20.602 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 15.360523785 2024-01-15T18:09:20.602 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:09:20.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.594+0000 7f86ba600640 1 -- 172.21.15.27:0/3968450725 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f868c066e90 msgr2=0x7f868c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:20.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.594+0000 7f86ba600640 1 --2- 172.21.15.27:0/3968450725 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f868c066e90 0x7f868c069350 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f86b415d1e0 tx=0x7f86a4009190 comp rx=0 tx=0).stop 2024-01-15T18:09:20.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.594+0000 7f86ba600640 1 -- 172.21.15.27:0/3968450725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86b415d7c0 msgr2=0x7f86b40cd9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:20.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.594+0000 7f86ba600640 1 --2- 172.21.15.27:0/3968450725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86b415d7c0 0x7f86b40cd9d0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f86ac01af60 tx=0x7f86ac019650 comp rx=0 tx=0).stop 2024-01-15T18:09:20.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.594+0000 7f86ba600640 1 -- 172.21.15.27:0/3968450725 shutdown_connections 2024-01-15T18:09:20.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.594+0000 7f86ba600640 1 --2- 172.21.15.27:0/3968450725 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f868c066e90 0x7f868c069350 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.594+0000 7f86ba600640 1 --2- 172.21.15.27:0/3968450725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86b415d7c0 0x7f86b40cd9d0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.594+0000 7f86ba600640 1 --2- 172.21.15.27:0/3968450725 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f86b40b1d90 0x7f86b4154c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.594+0000 7f86ba600640 1 --2- 172.21.15.27:0/3968450725 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f86b406bc70 0x7f86b4151ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.594+0000 7f86ba600640 1 -- 172.21.15.27:0/3968450725 >> 172.21.15.27:0/3968450725 conn(0x7f86b40b2a80 msgr2=0x7f86b406e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:20.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.594+0000 7f86ba600640 1 -- 172.21.15.27:0/3968450725 shutdown_connections 2024-01-15T18:09:20.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.594+0000 7f86ba600640 1 -- 172.21.15.27:0/3968450725 wait complete. 2024-01-15T18:09:20.619 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:09:20.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.754+0000 7f85f6c00640 1 Processor -- start 2024-01-15T18:09:20.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.754+0000 7f85f6c00640 1 -- start start 2024-01-15T18:09:20.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.754+0000 7f85f6c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f85f00afe70 0x7f85f00cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.754+0000 7f85f6c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85f00cb7b0 0x7f85f00cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.754+0000 7f85f6c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f00d05b0 con 0x7f85f0160b10 2024-01-15T18:09:20.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.754+0000 7f85f6c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85f00ce7d0 con 0x7f85f00afe70 2024-01-15T18:09:20.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.754+0000 7f85f6c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f00cfc80 con 0x7f85f00cb7b0 2024-01-15T18:09:20.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.754+0000 7f85ef400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f85f00afe70 0x7f85f00cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.754+0000 7f85ef400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f85f00afe70 0x7f85f00cdeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34498/0 (socket says 172.21.15.27:34498) 2024-01-15T18:09:20.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.754+0000 7f85ef400640 1 -- 172.21.15.27:0/2838322761 learned_addr learned my addr 172.21.15.27:0/2838322761 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:20.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.758+0000 7f85f4c00640 1 --2- 172.21.15.27:0/2838322761 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85f00cb7b0 0x7f85f00cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.758+0000 7f85ef400640 1 -- 172.21.15.27:0/2838322761 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85f00cb7b0 msgr2=0x7f85f00cbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:20.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.758+0000 7f85ef400640 1 --2- 172.21.15.27:0/2838322761 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85f00cb7b0 0x7f85f00cbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.758+0000 7f85ef400640 1 -- 172.21.15.27:0/2838322761 >> v1:172.21.15.93:6789/0 conn(0x7f85f0160b10 legacy=0x7f85f00cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:20.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.758+0000 7f85ef400640 1 -- 172.21.15.27:0/2838322761 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f00cff60 con 0x7f85f00afe70 2024-01-15T18:09:20.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.758+0000 7f85f4c00640 1 --2- 172.21.15.27:0/2838322761 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85f00cb7b0 0x7f85f00cbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:09:20.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.758+0000 7f85ef400640 1 --2- 172.21.15.27:0/2838322761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f00afe70 0x7f85f00cdeb0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f85d8009870 tx=0x7f85d801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8715e8474236a1d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:20.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.758+0000 7f85eea00640 1 -- 172.21.15.27:0/2838322761 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d80a6040 con 0x7f85f00afe70 2024-01-15T18:09:20.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.758+0000 7f85eea00640 1 -- 172.21.15.27:0/2838322761 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d80aa600 con 0x7f85f00afe70 2024-01-15T18:09:20.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.758+0000 7f85eea00640 1 -- 172.21.15.27:0/2838322761 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d80aa990 con 0x7f85f00afe70 2024-01-15T18:09:20.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.762+0000 7f85f6c00640 1 -- 172.21.15.27:0/2838322761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f00afe70 msgr2=0x7f85f00cdeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:20.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.762+0000 7f85f6c00640 1 --2- 172.21.15.27:0/2838322761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f00afe70 0x7f85f00cdeb0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f85d8009870 tx=0x7f85d801c480 comp rx=0 tx=0).stop 2024-01-15T18:09:20.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.762+0000 7f85f6c00640 1 -- 172.21.15.27:0/2838322761 shutdown_connections 2024-01-15T18:09:20.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.762+0000 7f85f6c00640 1 --2- 172.21.15.27:0/2838322761 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85f00cb7b0 0x7f85f00cbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.762+0000 7f85f6c00640 1 --2- 172.21.15.27:0/2838322761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f00afe70 0x7f85f00cdeb0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.762+0000 7f85f6c00640 1 -- 172.21.15.27:0/2838322761 >> 172.21.15.27:0/2838322761 conn(0x7f85f00d0f20 msgr2=0x7f85f00d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:20.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.762+0000 7f85f6c00640 1 -- 172.21.15.27:0/2838322761 shutdown_connections 2024-01-15T18:09:20.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.762+0000 7f85f6c00640 1 -- 172.21.15.27:0/2838322761 wait complete. 2024-01-15T18:09:20.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.762+0000 7f85f6c00640 1 Processor -- start 2024-01-15T18:09:20.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.762+0000 7f85f6c00640 1 -- start start 2024-01-15T18:09:20.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.762+0000 7f85f6c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f00afe70 0x7f85f014e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.762+0000 7f85f6c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85f00cb7b0 0x7f85f00cf280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.762+0000 7f85f6c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f85f0160b10 0x7f85f01509f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.762+0000 7f85f6c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f0150860 con 0x7f85f00afe70 2024-01-15T18:09:20.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.762+0000 7f85f6c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f0151d90 con 0x7f85f0160b10 2024-01-15T18:09:20.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.762+0000 7f85f6c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f00ca290 con 0x7f85f00cb7b0 2024-01-15T18:09:20.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.762+0000 7f85efe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f00afe70 0x7f85f014e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.762+0000 7f85efe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f00afe70 0x7f85f014e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34500/0 (socket says 172.21.15.27:34500) 2024-01-15T18:09:20.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.762+0000 7f85efe00640 1 -- 172.21.15.27:0/4262371633 learned_addr learned my addr 172.21.15.27:0/4262371633 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:20.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.762+0000 7f85ef400640 1 --2- 172.21.15.27:0/4262371633 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85f00cb7b0 0x7f85f00cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.766+0000 7f85f4c00640 1 --2- 172.21.15.27:0/4262371633 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f85f0160b10 0x7f85f01509f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.766+0000 7f85efe00640 1 -- 172.21.15.27:0/4262371633 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85f00cb7b0 msgr2=0x7f85f00cf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:20.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.766+0000 7f85efe00640 1 --2- 172.21.15.27:0/4262371633 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85f00cb7b0 0x7f85f00cf280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.766+0000 7f85efe00640 1 -- 172.21.15.27:0/4262371633 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f85f0160b10 msgr2=0x7f85f01509f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:20.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.766+0000 7f85efe00640 1 --2- 172.21.15.27:0/4262371633 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f85f0160b10 0x7f85f01509f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.766+0000 7f85efe00640 1 -- 172.21.15.27:0/4262371633 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d80094d0 con 0x7f85f00afe70 2024-01-15T18:09:20.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.766+0000 7f85f4c00640 1 --2- 172.21.15.27:0/4262371633 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f85f0160b10 0x7f85f01509f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:20.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.766+0000 7f85efe00640 1 --2- 172.21.15.27:0/4262371633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f00afe70 0x7f85f014e7a0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f85e000c500 tx=0x7f85e000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:20.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.766+0000 7f85e7e00640 1 -- 172.21.15.27:0/4262371633 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e0005220 con 0x7f85f00afe70 2024-01-15T18:09:20.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.766+0000 7f85e7e00640 1 -- 172.21.15.27:0/4262371633 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e0005380 con 0x7f85f00afe70 2024-01-15T18:09:20.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.766+0000 7f85e7e00640 1 -- 172.21.15.27:0/4262371633 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e0007270 con 0x7f85f00afe70 2024-01-15T18:09:20.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.766+0000 7f85f6c00640 1 -- 172.21.15.27:0/4262371633 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f00ca570 con 0x7f85f00afe70 2024-01-15T18:09:20.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.766+0000 7f85f6c00640 1 -- 172.21.15.27:0/4262371633 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f00cab30 con 0x7f85f00afe70 2024-01-15T18:09:20.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.766+0000 7f85f6c00640 1 -- 172.21.15.27:0/4262371633 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f004e850 con 0x7f85f00afe70 2024-01-15T18:09:20.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.766+0000 7f85e7e00640 1 -- 172.21.15.27:0/4262371633 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f85e00073d0 con 0x7f85f00afe70 2024-01-15T18:09:20.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.766+0000 7f85e7e00640 1 --2- 172.21.15.27:0/4262371633 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f85c8066e90 0x7f85c8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:20.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.766+0000 7f85e7e00640 1 -- 172.21.15.27:0/4262371633 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f85e0084860 con 0x7f85f00afe70 2024-01-15T18:09:20.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.770+0000 7f85ef400640 1 --2- 172.21.15.27:0/4262371633 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f85c8066e90 0x7f85c8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:20.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.770+0000 7f85ef400640 1 --2- 172.21.15.27:0/4262371633 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f85c8066e90 0x7f85c8069350 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f85d80ae000 tx=0x7f85d80a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:20.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.770+0000 7f85e7e00640 1 -- 172.21.15.27:0/4262371633 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f85e00511e0 con 0x7f85f00afe70 2024-01-15T18:09:20.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.962+0000 7f85f6c00640 1 -- 172.21.15.27:0/4262371633 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f85f00b6aa0 con 0x7f85f00afe70 2024-01-15T18:09:20.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.962+0000 7f85e7e00640 1 -- 172.21.15.27:0/4262371633 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v37) v1 ==== 53+0+73 (secure 0 0 0) 0x7f85e0054800 con 0x7f85f00afe70 2024-01-15T18:09:20.969 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:09:20.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.966+0000 7f85f6c00640 1 -- 172.21.15.27:0/4262371633 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f85c8066e90 msgr2=0x7f85c8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:20.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.966+0000 7f85f6c00640 1 --2- 172.21.15.27:0/4262371633 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f85c8066e90 0x7f85c8069350 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f85d80ae000 tx=0x7f85d80a7000 comp rx=0 tx=0).stop 2024-01-15T18:09:20.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.966+0000 7f85f6c00640 1 -- 172.21.15.27:0/4262371633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f00afe70 msgr2=0x7f85f014e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:20.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.966+0000 7f85f6c00640 1 --2- 172.21.15.27:0/4262371633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f00afe70 0x7f85f014e7a0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f85e000c500 tx=0x7f85e000c9c0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.966+0000 7f85f6c00640 1 -- 172.21.15.27:0/4262371633 shutdown_connections 2024-01-15T18:09:20.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.966+0000 7f85f6c00640 1 --2- 172.21.15.27:0/4262371633 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f85c8066e90 0x7f85c8069350 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.966+0000 7f85f6c00640 1 --2- 172.21.15.27:0/4262371633 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f85f0160b10 0x7f85f01509f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.966+0000 7f85f6c00640 1 --2- 172.21.15.27:0/4262371633 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85f00cb7b0 0x7f85f00cf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.966+0000 7f85f6c00640 1 --2- 172.21.15.27:0/4262371633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85f00afe70 0x7f85f014e7a0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:20.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.966+0000 7f85f6c00640 1 -- 172.21.15.27:0/4262371633 >> 172.21.15.27:0/4262371633 conn(0x7f85f00d0f20 msgr2=0x7f85f00af680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:20.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.966+0000 7f85f6c00640 1 -- 172.21.15.27:0/4262371633 shutdown_connections 2024-01-15T18:09:20.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:20.966+0000 7f85f6c00640 1 -- 172.21.15.27:0/4262371633 wait complete. 2024-01-15T18:09:20.990 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:09:20.990 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:09:20.991 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-01-15T18:09:20.991 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-15T18:09:20.991 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-15T18:09:20.991 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:09:20.991 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:09:20.991 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:09:20.991 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:09:20.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:20.991 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-01-15T18:09:21.001 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:09:21.001 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:09:21.001 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:09:21.001 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:09:21.001 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-15T18:09:21.002 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:09:21.002 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:09:21.002 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:09:21.002 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-01-15T18:09:21.002 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:09:21.002 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:09:21.002 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-01-15T18:09:21.002 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:09:21.002 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-01-15T18:09:21.002 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-01-15T18:09:21.002 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-01-15T18:09:21.002 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42198sec preferred_lft 42198sec 2024-01-15T18:09:21.002 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-01-15T18:09:21.003 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:09:21.003 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:09:21.003 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-01-15T18:09:21.003 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-01-15T18:09:21.003 INFO:teuthology.orchestra.run.smithi027.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-01-15T18:09:21.003 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 02:42:65:20:14:72 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:09:21.003 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-01-15T18:09:21.003 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:09:21.003 INFO:teuthology.orchestra.run.smithi027.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:09:21.003 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-01-15T18:09:21.003 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:09:21.003 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:09:21.003 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-01-15T18:09:21.003 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:09:21.004 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:09:21.004 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:09:21.004 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-01-15T18:09:21.004 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:09:21.004 INFO:teuthology.orchestra.run.smithi027.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:09:21.004 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:09:21.004 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-01-15T18:09:21.004 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:09:21.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:21.005 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:09:21.064 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-01-15T18:09:21.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:21.066 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-01-15T18:09:21.123 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-01-15T18:09:21.123 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:09:21.123 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-01-15T18:09:21.123 INFO:teuthology.orchestra.run.smithi027.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-01-15T18:09:21.123 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-01-15T18:09:21.123 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:09:21.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:21.124 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-15T18:09:21.124 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-15T18:09:21.124 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-15T18:09:21.124 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-15T18:09:21.124 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-15T18:09:21.124 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-15T18:09:21.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:21.216 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-15T18:09:21.224 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-15T18:09:21.224 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-15T18:09:21.225 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:09:21.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:21.225 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-01-15T18:09:21.289 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-15T18:09:21.289 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:09:21.290 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-15T18:09:21.341 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:09:21.342 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:09:21.342 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:09:21.397 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-01-15T18:09:21.458 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:09:21.511 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-01-15T18:09:21.511 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-01-15T18:09:21.511 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-01-15T18:09:21.511 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-01-15T18:09:21.511 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-01-15T18:09:21.511 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-01-15T18:09:21.512 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-01-15T18:09:21.512 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-01-15T18:09:21.512 INFO:teuthology.orchestra.run.smithi027.stdout:/run/lock 2024-01-15T18:09:21.512 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-01-15T18:09:21.512 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-01-15T18:09:21.512 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-01-15T18:09:21.512 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:09:21.512 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-01-15T18:09:21.512 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-01-15T18:09:21.512 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-01-15T18:09:21.512 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-01-15T18:09:21.512 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/fs/nfsd 2024-01-15T18:09:21.512 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-01-15T18:09:21.513 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-01-15T18:09:21.513 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:09:21.513 INFO:teuthology.orchestra.run.smithi027.stdout:/run/rpc_pipefs 2024-01-15T18:09:21.513 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-01-15T18:09:21.513 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-01-15T18:09:21.513 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:09:21.513 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:09:21.513 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:09:21.513 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:09:21.513 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-01-15T18:09:21.513 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:09:21.513 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:09:21.513 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:09:21.513 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:09:21.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:21.515 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-01-15T18:09:21.569 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:09:21.570 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-15T18:09:21.616 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-01-15T18:09:21.637 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:09:21.687 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-01-15T18:09:21.687 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-01-15T18:09:21.687 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-01-15T18:09:21.687 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-01-15T18:09:21.687 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-01-15T18:09:21.688 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-01-15T18:09:21.688 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-01-15T18:09:21.688 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-01-15T18:09:21.688 INFO:teuthology.orchestra.run.smithi027.stdout:/run/lock 2024-01-15T18:09:21.688 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-01-15T18:09:21.688 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-01-15T18:09:21.688 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-01-15T18:09:21.688 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:09:21.688 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-01-15T18:09:21.688 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-01-15T18:09:21.688 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-01-15T18:09:21.688 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-01-15T18:09:21.688 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/fs/nfsd 2024-01-15T18:09:21.689 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-01-15T18:09:21.689 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-01-15T18:09:21.689 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:09:21.689 INFO:teuthology.orchestra.run.smithi027.stdout:/run/rpc_pipefs 2024-01-15T18:09:21.689 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-01-15T18:09:21.689 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-01-15T18:09:21.689 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:09:21.689 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:09:21.689 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:09:21.689 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:09:21.689 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-01-15T18:09:21.689 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:09:21.689 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:09:21.689 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:09:21.690 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:09:21.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:21.691 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-01-15T18:09:21.712 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:2024-01-15T18:09:21.706+0000 7f33915b4580 -1 init, newargv = 0x55858354c430 newargc=15 2024-01-15T18:09:21.713 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[41334]: starting ceph client 2024-01-15T18:09:21.719 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[41334]: starting fuse 2024-01-15T18:09:21.721 INFO:teuthology.orchestra.run.smithi027.stdout:43 2024-01-15T18:09:21.721 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-01-15T18:09:21.721 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:09:21.721 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-01-15T18:09:21.721 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-01-15T18:09:21.721 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-01-15T18:09:21.721 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-01-15T18:09:21.721 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:09:21.721 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-01-15T18:09:21.722 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-15T18:09:21.722 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-01-15T18:09:21.722 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-15T18:09:21.722 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:09:21.722 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-01-15T18:09:21.722 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-01-15T18:09:21.722 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-01-15T18:09:21.722 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-01-15T18:09:21.722 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:09:21.722 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-01-15T18:09:21.722 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:09:21.722 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:09:21.722 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:09:21.722 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-01-15T18:09:21.722 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-01-15T18:09:21.723 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-01-15T18:09:21.723 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:09:21.723 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:09:21.723 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.0")) 2024-01-15T18:09:21.723 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-01-15T18:09:21.849 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.0.41334.asok 2024-01-15T18:09:21.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:09:21.860 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.41334.asok 2024-01-15T18:09:21.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:21.861 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.41334.asok status 2024-01-15T18:09:22.033 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:09:22.034 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-01-15T18:09:22.034 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:09:22.034 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:09:22.034 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "0", 2024-01-15T18:09:22.034 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-01-15T18:09:22.034 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-15T18:09:22.034 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "41334", 2024-01-15T18:09:22.034 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-01-15T18:09:22.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:09:22.034 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-01-15T18:09:22.034 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-01-15T18:09:22.034 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 6663, 2024-01-15T18:09:22.034 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-01-15T18:09:22.034 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-01-15T18:09:22.034 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-01-15T18:09:22.035 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 6663 2024-01-15T18:09:22.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:09:22.035 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-01-15T18:09:22.035 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-01-15T18:09:22.035 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:09:22.035 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3552411983 2024-01-15T18:09:22.035 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:09:22.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:09:22.035 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-01-15T18:09:22.035 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-01-15T18:09:22.035 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:09:22.035 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3552411983 2024-01-15T18:09:22.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:09:22.035 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.6663 192.168.0.1:0/3552411983", 2024-01-15T18:09:22.035 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.1:0/3552411983", 2024-01-15T18:09:22.035 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-01-15T18:09:22.036 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 37, 2024-01-15T18:09:22.036 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 58, 2024-01-15T18:09:22.036 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:09:22.036 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-01-15T18:09:22.036 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-01-15T18:09:22.036 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:09:22.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:22.049 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-01-15T18:09:22.049 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-15T18:09:22.058 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-01-15T18:09:22.058 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-15T18:09:22.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:22.059 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:09:22.125 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:09:22.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.306+0000 7fa9bcc00640 1 Processor -- start 2024-01-15T18:09:22.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.306+0000 7fa9bcc00640 1 -- start start 2024-01-15T18:09:22.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.310+0000 7fa9bcc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa9b8160af0 0x7fa9b806b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:22.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.310+0000 7fa9bcc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa9b80ae710 0x7fa9b80aebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:22.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.310+0000 7fa9bcc00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b80ae280 con 0x7fa9b80cfaa0 2024-01-15T18:09:22.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.310+0000 7fa9bcc00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9b80adc50 con 0x7fa9b8160af0 2024-01-15T18:09:22.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.310+0000 7fa9bcc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b80cff80 con 0x7fa9b80ae710 2024-01-15T18:09:22.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.310+0000 7fa9b5400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa9b80cfaa0 0x7fa9b806bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49596/0 (socket says 172.21.15.27:49596) 2024-01-15T18:09:22.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.310+0000 7fa9b5400640 1 -- 172.21.15.27:0/1739934919 learned_addr learned my addr 172.21.15.27:0/1739934919 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:22.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.310+0000 7fa9b5e00640 1 --2- 172.21.15.27:0/1739934919 >> v2:172.21.15.27:3300/0 conn(0x7fa9b8160af0 0x7fa9b806b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:22.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.310+0000 7fa9b6800640 1 --2- 172.21.15.27:0/1739934919 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa9b80ae710 0x7fa9b80aebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:22.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.310+0000 7fa9b4a00640 1 -- 172.21.15.27:0/1739934919 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1535173631 0 0) 0x7fa9b80ae280 con 0x7fa9b80cfaa0 2024-01-15T18:09:22.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.310+0000 7fa9b4a00640 1 -- 172.21.15.27:0/1739934919 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa998003530 con 0x7fa9b80cfaa0 2024-01-15T18:09:22.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.310+0000 7fa9b5e00640 1 -- 172.21.15.27:0/1739934919 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa9b80ae710 msgr2=0x7fa9b80aebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:22.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.310+0000 7fa9b5e00640 1 --2- 172.21.15.27:0/1739934919 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa9b80ae710 0x7fa9b80aebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:22.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.310+0000 7fa9b5e00640 1 -- 172.21.15.27:0/1739934919 >> v1:172.21.15.27:6789/0 conn(0x7fa9b80cfaa0 legacy=0x7fa9b806bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:22.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.310+0000 7fa9b5e00640 1 -- 172.21.15.27:0/1739934919 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b80d0260 con 0x7fa9b8160af0 2024-01-15T18:09:22.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.314+0000 7fa9b5e00640 1 --2- 172.21.15.27:0/1739934919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b8160af0 0x7fa9b806b600 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80097c0 tx=0x7fa9a801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e6a64bdbc4fc05e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:22.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.314+0000 7fa9b4a00640 1 -- 172.21.15.27:0/1739934919 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a80ab600 con 0x7fa9b8160af0 2024-01-15T18:09:22.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.314+0000 7fa9b4a00640 1 -- 172.21.15.27:0/1739934919 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a80a7050 con 0x7fa9b8160af0 2024-01-15T18:09:22.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.314+0000 7fa9b4a00640 1 -- 172.21.15.27:0/1739934919 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a80ab980 con 0x7fa9b8160af0 2024-01-15T18:09:22.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.314+0000 7fa9bcc00640 1 -- 172.21.15.27:0/1739934919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b8160af0 msgr2=0x7fa9b806b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:22.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.314+0000 7fa9bcc00640 1 --2- 172.21.15.27:0/1739934919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b8160af0 0x7fa9b806b600 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80097c0 tx=0x7fa9a801c480 comp rx=0 tx=0).stop 2024-01-15T18:09:22.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.314+0000 7fa9bcc00640 1 -- 172.21.15.27:0/1739934919 shutdown_connections 2024-01-15T18:09:22.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.314+0000 7fa9bcc00640 1 --2- 172.21.15.27:0/1739934919 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa9b80ae710 0x7fa9b80aebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:22.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.314+0000 7fa9bcc00640 1 --2- 172.21.15.27:0/1739934919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b8160af0 0x7fa9b806b600 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:22.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.314+0000 7fa9bcc00640 1 -- 172.21.15.27:0/1739934919 >> 172.21.15.27:0/1739934919 conn(0x7fa9b80d0f20 msgr2=0x7fa9b80d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:22.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.314+0000 7fa9bcc00640 1 -- 172.21.15.27:0/1739934919 shutdown_connections 2024-01-15T18:09:22.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.314+0000 7fa9bcc00640 1 -- 172.21.15.27:0/1739934919 wait complete. 2024-01-15T18:09:22.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.314+0000 7fa9bcc00640 1 Processor -- start 2024-01-15T18:09:22.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.314+0000 7fa9bcc00640 1 -- start start 2024-01-15T18:09:22.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.318+0000 7fa9bcc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa9b80ae710 0x7fa9b8146350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:22.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.318+0000 7fa9bcc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa9b80cfaa0 0x7fa9b81576e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:22.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.318+0000 7fa9bcc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b8160af0 0x7fa9b80b3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:22.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.318+0000 7fa9bcc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b8158ab0 con 0x7fa9b8160af0 2024-01-15T18:09:22.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.318+0000 7fa9bcc00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b8146890 con 0x7fa9b80cfaa0 2024-01-15T18:09:22.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.318+0000 7fa9bcc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b80b5360 con 0x7fa9b80ae710 2024-01-15T18:09:22.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.318+0000 7fa9b6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b8160af0 0x7fa9b80b3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:22.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.318+0000 7fa9b6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b8160af0 0x7fa9b80b3390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34516/0 (socket says 172.21.15.27:34516) 2024-01-15T18:09:22.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.318+0000 7fa9b6800640 1 -- 172.21.15.27:0/1627203126 learned_addr learned my addr 172.21.15.27:0/1627203126 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:22.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.318+0000 7fa9b5e00640 1 --2- 172.21.15.27:0/1627203126 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa9b80ae710 0x7fa9b8146350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:22.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.318+0000 7fa9b5400640 1 --2- 172.21.15.27:0/1627203126 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa9b80cfaa0 0x7fa9b81576e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:22.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.318+0000 7fa9b6800640 1 -- 172.21.15.27:0/1627203126 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa9b80ae710 msgr2=0x7fa9b8146350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:22.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.318+0000 7fa9b6800640 1 --2- 172.21.15.27:0/1627203126 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa9b80ae710 0x7fa9b8146350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:22.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.318+0000 7fa9b6800640 1 -- 172.21.15.27:0/1627203126 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa9b80cfaa0 msgr2=0x7fa9b81576e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:22.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.318+0000 7fa9b6800640 1 --2- 172.21.15.27:0/1627203126 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa9b80cfaa0 0x7fa9b81576e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:22.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.318+0000 7fa9b6800640 1 -- 172.21.15.27:0/1627203126 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a8009420 con 0x7fa9b8160af0 2024-01-15T18:09:22.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.318+0000 7fa9b6800640 1 --2- 172.21.15.27:0/1627203126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b8160af0 0x7fa9b80b3390 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fa9b000efc0 tx=0x7fa9b000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:22.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.318+0000 7fa9ae000640 1 -- 172.21.15.27:0/1627203126 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b0009ba0 con 0x7fa9b8160af0 2024-01-15T18:09:22.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.318+0000 7fa9ae000640 1 -- 172.21.15.27:0/1627203126 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9b000f040 con 0x7fa9b8160af0 2024-01-15T18:09:22.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.318+0000 7fa9bcc00640 1 -- 172.21.15.27:0/1627203126 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9b8157f30 con 0x7fa9b8160af0 2024-01-15T18:09:22.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.318+0000 7fa9ae000640 1 -- 172.21.15.27:0/1627203126 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b00146c0 con 0x7fa9b8160af0 2024-01-15T18:09:22.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.318+0000 7fa9bcc00640 1 -- 172.21.15.27:0/1627203126 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b81584f0 con 0x7fa9b8160af0 2024-01-15T18:09:22.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.318+0000 7fa9bcc00640 1 -- 172.21.15.27:0/1627203126 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9b80b5520 con 0x7fa9b8160af0 2024-01-15T18:09:22.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.322+0000 7fa9ae000640 1 -- 172.21.15.27:0/1627203126 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7fa9b0014820 con 0x7fa9b8160af0 2024-01-15T18:09:22.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.322+0000 7fa9ae000640 1 --2- 172.21.15.27:0/1627203126 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa990066ec0 0x7fa990069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:22.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.322+0000 7fa9ae000640 1 -- 172.21.15.27:0/1627203126 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fa9b0085a30 con 0x7fa9b8160af0 2024-01-15T18:09:22.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.322+0000 7fa9b5e00640 1 --2- 172.21.15.27:0/1627203126 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa990066ec0 0x7fa990069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:22.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.322+0000 7fa9b5e00640 1 --2- 172.21.15.27:0/1627203126 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa990066ec0 0x7fa990069380 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80ad000 tx=0x7fa9a80a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:22.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.326+0000 7fa9ae000640 1 -- 172.21.15.27:0/1627203126 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa9b0052320 con 0x7fa9b8160af0 2024-01-15T18:09:22.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.514+0000 7fa9bcc00640 1 -- 172.21.15.27:0/1627203126 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa9b804e850 con 0x7fa9b8160af0 2024-01-15T18:09:22.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.514+0000 7fa9ae000640 1 -- 172.21.15.27:0/1627203126 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v37) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa9b0018060 con 0x7fa9b8160af0 2024-01-15T18:09:22.521 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:09:22.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.518+0000 7fa9bcc00640 1 -- 172.21.15.27:0/1627203126 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa990066ec0 msgr2=0x7fa990069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:22.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.518+0000 7fa9bcc00640 1 --2- 172.21.15.27:0/1627203126 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa990066ec0 0x7fa990069380 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80ad000 tx=0x7fa9a80a6000 comp rx=0 tx=0).stop 2024-01-15T18:09:22.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.518+0000 7fa9bcc00640 1 -- 172.21.15.27:0/1627203126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b8160af0 msgr2=0x7fa9b80b3390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:22.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.518+0000 7fa9bcc00640 1 --2- 172.21.15.27:0/1627203126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b8160af0 0x7fa9b80b3390 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fa9b000efc0 tx=0x7fa9b000c490 comp rx=0 tx=0).stop 2024-01-15T18:09:22.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.518+0000 7fa9bcc00640 1 -- 172.21.15.27:0/1627203126 shutdown_connections 2024-01-15T18:09:22.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.518+0000 7fa9bcc00640 1 --2- 172.21.15.27:0/1627203126 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa990066ec0 0x7fa990069380 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:22.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.518+0000 7fa9bcc00640 1 --2- 172.21.15.27:0/1627203126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9b8160af0 0x7fa9b80b3390 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:22.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.518+0000 7fa9bcc00640 1 --2- 172.21.15.27:0/1627203126 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa9b80cfaa0 0x7fa9b81576e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:22.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.518+0000 7fa9bcc00640 1 --2- 172.21.15.27:0/1627203126 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa9b80ae710 0x7fa9b8146350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:22.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.518+0000 7fa9bcc00640 1 -- 172.21.15.27:0/1627203126 >> 172.21.15.27:0/1627203126 conn(0x7fa9b80d0f20 msgr2=0x7fa9b80b21a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:22.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.518+0000 7fa9bcc00640 1 -- 172.21.15.27:0/1627203126 shutdown_connections 2024-01-15T18:09:22.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:22.518+0000 7fa9bcc00640 1 -- 172.21.15.27:0/1627203126 wait complete. 2024-01-15T18:09:22.544 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:09:22.544 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:09:22.544 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi093.front.sepia.ceph.com 2024-01-15T18:09:22.544 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-15T18:09:22.544 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-15T18:09:22.544 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:09:22.544 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:09:22.545 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:09:22.545 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:09:22.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:22.545 DEBUG:teuthology.orchestra.run.smithi093:> ip addr 2024-01-15T18:09:22.555 INFO:teuthology.orchestra.run.smithi093.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:09:22.555 INFO:teuthology.orchestra.run.smithi093.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:09:22.555 INFO:teuthology.orchestra.run.smithi093.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:09:22.555 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:09:22.555 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 ::1/128 scope host 2024-01-15T18:09:22.556 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:09:22.556 INFO:teuthology.orchestra.run.smithi093.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:09:22.556 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:d8:da:1c brd ff:ff:ff:ff:ff:ff 2024-01-15T18:09:22.556 INFO:teuthology.orchestra.run.smithi093.stdout: altname enp7s0f0 2024-01-15T18:09:22.556 INFO:teuthology.orchestra.run.smithi093.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:09:22.556 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:d8:da:1d brd ff:ff:ff:ff:ff:ff 2024-01-15T18:09:22.556 INFO:teuthology.orchestra.run.smithi093.stdout: altname enp7s0f1 2024-01-15T18:09:22.556 INFO:teuthology.orchestra.run.smithi093.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:09:22.556 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:8f:d6:34 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:09:22.556 INFO:teuthology.orchestra.run.smithi093.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:09:22.556 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:8f:d6:35 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:09:22.556 INFO:teuthology.orchestra.run.smithi093.stdout: inet 172.21.15.93/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-01-15T18:09:22.556 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft 42176sec preferred_lft 42176sec 2024-01-15T18:09:22.557 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::ec4:7aff:fe8f:d635/64 scope link 2024-01-15T18:09:22.557 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:09:22.557 INFO:teuthology.orchestra.run.smithi093.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-01-15T18:09:22.557 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 02:42:2a:61:fc:7d brd ff:ff:ff:ff:ff:ff 2024-01-15T18:09:22.557 INFO:teuthology.orchestra.run.smithi093.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-01-15T18:09:22.557 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:09:22.557 INFO:teuthology.orchestra.run.smithi093.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:09:22.557 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-01-15T18:09:22.557 INFO:teuthology.orchestra.run.smithi093.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:09:22.557 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:09:22.557 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-01-15T18:09:22.557 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:09:22.557 INFO:teuthology.orchestra.run.smithi093.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:09:22.557 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:09:22.557 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-01-15T18:09:22.558 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:09:22.558 INFO:teuthology.orchestra.run.smithi093.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:09:22.558 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:09:22.558 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-01-15T18:09:22.558 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:09:22.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:22.559 DEBUG:teuthology.orchestra.run.smithi093:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:09:22.622 INFO:teuthology.orchestra.run.smithi093.stdout:1 2024-01-15T18:09:22.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:22.625 DEBUG:teuthology.orchestra.run.smithi093:> route 2024-01-15T18:09:22.677 INFO:teuthology.orchestra.run.smithi093.stdout:Kernel IP routing table 2024-01-15T18:09:22.677 INFO:teuthology.orchestra.run.smithi093.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:09:22.677 INFO:teuthology.orchestra.run.smithi093.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-01-15T18:09:22.677 INFO:teuthology.orchestra.run.smithi093.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T18:09:22.677 INFO:teuthology.orchestra.run.smithi093.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-01-15T18:09:22.678 INFO:teuthology.orchestra.run.smithi093.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T18:09:22.678 INFO:teuthology.orchestra.run.smithi093.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:09:22.678 INFO:teuthology.orchestra.run.smithi093.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:09:22.678 INFO:teuthology.orchestra.run.smithi093.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:09:22.678 INFO:teuthology.orchestra.run.smithi093.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:09:22.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:22.678 DEBUG:teuthology.orchestra.run.smithi093:> (cd / && exec bash -c ' 2024-01-15T18:09:22.678 DEBUG:teuthology.orchestra.run.smithi093:> set -e 2024-01-15T18:09:22.678 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T18:09:22.679 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T18:09:22.679 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T18:09:22.679 DEBUG:teuthology.orchestra.run.smithi093:> ') 2024-01-15T18:09:22.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:22.774 DEBUG:teuthology.orchestra.run.smithi093:> ip netns list 2024-01-15T18:09:22.784 INFO:teuthology.orchestra.run.smithi093.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-15T18:09:22.784 INFO:teuthology.orchestra.run.smithi093.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-15T18:09:22.784 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:09:22.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:22.785 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip link set brx.0 up 2024-01-15T18:09:22.846 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-15T18:09:22.846 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:09:22.846 DEBUG:teuthology.orchestra.run.smithi093:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-15T18:09:22.899 INFO:teuthology.orchestra.run.smithi093.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:09:22.900 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:09:22.900 DEBUG:teuthology.orchestra.run.smithi093:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:09:22.949 DEBUG:teuthology.orchestra.run.smithi093:> sudo modprobe fuse 2024-01-15T18:09:23.009 DEBUG:teuthology.orchestra.run.smithi093:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:09:23.062 INFO:teuthology.orchestra.run.smithi093.stdout:/sys 2024-01-15T18:09:23.062 INFO:teuthology.orchestra.run.smithi093.stdout:/proc 2024-01-15T18:09:23.063 INFO:teuthology.orchestra.run.smithi093.stdout:/dev 2024-01-15T18:09:23.063 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/pts 2024-01-15T18:09:23.063 INFO:teuthology.orchestra.run.smithi093.stdout:/run 2024-01-15T18:09:23.063 INFO:teuthology.orchestra.run.smithi093.stdout:/ 2024-01-15T18:09:23.063 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/security 2024-01-15T18:09:23.063 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/shm 2024-01-15T18:09:23.063 INFO:teuthology.orchestra.run.smithi093.stdout:/run/lock 2024-01-15T18:09:23.063 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/cgroup 2024-01-15T18:09:23.063 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/pstore 2024-01-15T18:09:23.063 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/bpf 2024-01-15T18:09:23.063 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:09:23.063 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/hugepages 2024-01-15T18:09:23.063 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/mqueue 2024-01-15T18:09:23.063 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/debug 2024-01-15T18:09:23.064 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/tracing 2024-01-15T18:09:23.064 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/fs/nfsd 2024-01-15T18:09:23.064 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/fuse/connections 2024-01-15T18:09:23.064 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/config 2024-01-15T18:09:23.064 INFO:teuthology.orchestra.run.smithi093.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:09:23.064 INFO:teuthology.orchestra.run.smithi093.stdout:/run/rpc_pipefs 2024-01-15T18:09:23.064 INFO:teuthology.orchestra.run.smithi093.stdout:/run/user/1000 2024-01-15T18:09:23.064 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph 2024-01-15T18:09:23.064 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:09:23.064 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:09:23.064 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:09:23.064 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:09:23.064 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns 2024-01-15T18:09:23.064 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:09:23.065 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:09:23.065 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:09:23.065 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:09:23.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:23.066 DEBUG:teuthology.orchestra.run.smithi093:> ls /sys/fs/fuse/connections 2024-01-15T18:09:23.117 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:09:23.117 DEBUG:teuthology.orchestra.run.smithi093:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-15T18:09:23.160 DEBUG:teuthology.orchestra.run.smithi093:> sudo modprobe fuse 2024-01-15T18:09:23.181 DEBUG:teuthology.orchestra.run.smithi093:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:09:23.239 INFO:teuthology.orchestra.run.smithi093.stdout:/sys 2024-01-15T18:09:23.239 INFO:teuthology.orchestra.run.smithi093.stdout:/proc 2024-01-15T18:09:23.239 INFO:teuthology.orchestra.run.smithi093.stdout:/dev 2024-01-15T18:09:23.239 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/pts 2024-01-15T18:09:23.239 INFO:teuthology.orchestra.run.smithi093.stdout:/run 2024-01-15T18:09:23.239 INFO:teuthology.orchestra.run.smithi093.stdout:/ 2024-01-15T18:09:23.239 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/security 2024-01-15T18:09:23.239 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/shm 2024-01-15T18:09:23.239 INFO:teuthology.orchestra.run.smithi093.stdout:/run/lock 2024-01-15T18:09:23.239 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/cgroup 2024-01-15T18:09:23.240 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/pstore 2024-01-15T18:09:23.240 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/bpf 2024-01-15T18:09:23.240 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:09:23.240 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/hugepages 2024-01-15T18:09:23.240 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/mqueue 2024-01-15T18:09:23.240 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/debug 2024-01-15T18:09:23.240 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/tracing 2024-01-15T18:09:23.240 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/fs/nfsd 2024-01-15T18:09:23.240 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/fuse/connections 2024-01-15T18:09:23.240 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/config 2024-01-15T18:09:23.240 INFO:teuthology.orchestra.run.smithi093.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:09:23.240 INFO:teuthology.orchestra.run.smithi093.stdout:/run/rpc_pipefs 2024-01-15T18:09:23.240 INFO:teuthology.orchestra.run.smithi093.stdout:/run/user/1000 2024-01-15T18:09:23.240 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph 2024-01-15T18:09:23.241 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:09:23.241 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:09:23.241 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:09:23.241 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:09:23.241 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns 2024-01-15T18:09:23.241 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:09:23.241 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:09:23.241 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:09:23.241 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:09:23.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:23.242 DEBUG:teuthology.orchestra.run.smithi093:> ls /sys/fs/fuse/connections 2024-01-15T18:09:23.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:2024-01-15T18:09:23.249+0000 7fcc68603580 -1 init, newargv = 0x562f7de730a0 newargc=15 2024-01-15T18:09:23.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:ceph-fuse[33741]: starting ceph client 2024-01-15T18:09:23.261 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:ceph-fuse[33741]: starting fuse 2024-01-15T18:09:23.263 INFO:teuthology.orchestra.run.smithi093.stdout:43 2024-01-15T18:09:23.263 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-01-15T18:09:23.263 DEBUG:teuthology.orchestra.run.smithi093:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:09:23.264 DEBUG:teuthology.orchestra.run.smithi093:> import glob 2024-01-15T18:09:23.264 DEBUG:teuthology.orchestra.run.smithi093:> import re 2024-01-15T18:09:23.264 DEBUG:teuthology.orchestra.run.smithi093:> import os 2024-01-15T18:09:23.264 DEBUG:teuthology.orchestra.run.smithi093:> import subprocess 2024-01-15T18:09:23.264 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:09:23.264 DEBUG:teuthology.orchestra.run.smithi093:> def _find_admin_socket(client_name): 2024-01-15T18:09:23.264 DEBUG:teuthology.orchestra.run.smithi093:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-15T18:09:23.264 DEBUG:teuthology.orchestra.run.smithi093:> files = glob.glob(asok_path) 2024-01-15T18:09:23.264 DEBUG:teuthology.orchestra.run.smithi093:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-15T18:09:23.264 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:09:23.264 DEBUG:teuthology.orchestra.run.smithi093:> # Given a non-glob path, it better be there 2024-01-15T18:09:23.264 DEBUG:teuthology.orchestra.run.smithi093:> if "*" not in asok_path: 2024-01-15T18:09:23.264 DEBUG:teuthology.orchestra.run.smithi093:> assert(len(files) == 1) 2024-01-15T18:09:23.264 DEBUG:teuthology.orchestra.run.smithi093:> return files[0] 2024-01-15T18:09:23.265 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:09:23.265 DEBUG:teuthology.orchestra.run.smithi093:> for f in files: 2024-01-15T18:09:23.265 DEBUG:teuthology.orchestra.run.smithi093:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:09:23.265 DEBUG:teuthology.orchestra.run.smithi093:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:09:23.265 DEBUG:teuthology.orchestra.run.smithi093:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:09:23.265 DEBUG:teuthology.orchestra.run.smithi093:> contents = proc_f.read() 2024-01-15T18:09:23.265 DEBUG:teuthology.orchestra.run.smithi093:> if mountpoint in contents: 2024-01-15T18:09:23.265 DEBUG:teuthology.orchestra.run.smithi093:> return f 2024-01-15T18:09:23.265 DEBUG:teuthology.orchestra.run.smithi093:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:09:23.265 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:09:23.265 DEBUG:teuthology.orchestra.run.smithi093:> print(_find_admin_socket("client.1")) 2024-01-15T18:09:23.265 DEBUG:teuthology.orchestra.run.smithi093:> ' 2024-01-15T18:09:23.388 INFO:teuthology.orchestra.run.smithi093.stdout:/var/run/ceph/ceph-client.1.33741.asok 2024-01-15T18:09:23.392 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:09:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:09:23.400 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.33741.asok 2024-01-15T18:09:23.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:23.400 DEBUG:teuthology.orchestra.run.smithi093:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.33741.asok status 2024-01-15T18:09:23.567 INFO:teuthology.orchestra.run.smithi093.stdout:{ 2024-01-15T18:09:23.568 INFO:teuthology.orchestra.run.smithi093.stdout: "metadata": { 2024-01-15T18:09:23.568 INFO:teuthology.orchestra.run.smithi093.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:09:23.568 INFO:teuthology.orchestra.run.smithi093.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:09:23.568 INFO:teuthology.orchestra.run.smithi093.stdout: "entity_id": "1", 2024-01-15T18:09:23.568 INFO:teuthology.orchestra.run.smithi093.stdout: "hostname": "smithi093", 2024-01-15T18:09:23.568 INFO:teuthology.orchestra.run.smithi093.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-15T18:09:23.568 INFO:teuthology.orchestra.run.smithi093.stdout: "pid": "33741", 2024-01-15T18:09:23.568 INFO:teuthology.orchestra.run.smithi093.stdout: "root": "/" 2024-01-15T18:09:23.568 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:09:23.568 INFO:teuthology.orchestra.run.smithi093.stdout: "dentry_count": 0, 2024-01-15T18:09:23.568 INFO:teuthology.orchestra.run.smithi093.stdout: "dentry_pinned_count": 0, 2024-01-15T18:09:23.568 INFO:teuthology.orchestra.run.smithi093.stdout: "id": 5891, 2024-01-15T18:09:23.569 INFO:teuthology.orchestra.run.smithi093.stdout: "inst": { 2024-01-15T18:09:23.569 INFO:teuthology.orchestra.run.smithi093.stdout: "name": { 2024-01-15T18:09:23.569 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "client", 2024-01-15T18:09:23.569 INFO:teuthology.orchestra.run.smithi093.stdout: "num": 5891 2024-01-15T18:09:23.569 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:09:23.569 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": { 2024-01-15T18:09:23.569 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "v1", 2024-01-15T18:09:23.569 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:09:23.569 INFO:teuthology.orchestra.run.smithi093.stdout: "nonce": 1158128417 2024-01-15T18:09:23.569 INFO:teuthology.orchestra.run.smithi093.stdout: } 2024-01-15T18:09:23.569 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:09:23.569 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": { 2024-01-15T18:09:23.569 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "v1", 2024-01-15T18:09:23.569 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:09:23.569 INFO:teuthology.orchestra.run.smithi093.stdout: "nonce": 1158128417 2024-01-15T18:09:23.570 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:09:23.570 INFO:teuthology.orchestra.run.smithi093.stdout: "inst_str": "client.5891 192.168.0.1:0/1158128417", 2024-01-15T18:09:23.570 INFO:teuthology.orchestra.run.smithi093.stdout: "addr_str": "192.168.0.1:0/1158128417", 2024-01-15T18:09:23.570 INFO:teuthology.orchestra.run.smithi093.stdout: "inode_count": 1, 2024-01-15T18:09:23.570 INFO:teuthology.orchestra.run.smithi093.stdout: "mds_epoch": 37, 2024-01-15T18:09:23.570 INFO:teuthology.orchestra.run.smithi093.stdout: "osd_epoch": 58, 2024-01-15T18:09:23.570 INFO:teuthology.orchestra.run.smithi093.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:09:23.570 INFO:teuthology.orchestra.run.smithi093.stdout: "blocklisted": false, 2024-01-15T18:09:23.570 INFO:teuthology.orchestra.run.smithi093.stdout: "fs_name": "cephfs" 2024-01-15T18:09:23.570 INFO:teuthology.orchestra.run.smithi093.stdout:} 2024-01-15T18:09:23.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:23.582 DEBUG:teuthology.orchestra.run.smithi093:> stat --file-system '--printf=%T 2024-01-15T18:09:23.582 DEBUG:teuthology.orchestra.run.smithi093:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-15T18:09:23.591 INFO:teuthology.orchestra.run.smithi093.stdout:fuseblk 2024-01-15T18:09:23.591 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-15T18:09:23.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:23.592 DEBUG:teuthology.orchestra.run.smithi093:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:09:23.652 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-15T18:09:23.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.802+0000 7fdda8e00640 1 Processor -- start 2024-01-15T18:09:23.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.802+0000 7fdda8e00640 1 -- start start 2024-01-15T18:09:23.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.806+0000 7fdda8e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdda415d570 0x7fdda406be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:23.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.806+0000 7fdda8e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fdda406b810 0x7fdda406c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:23.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.806+0000 7fdda8e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda40b0d70 con 0x7fdda415d190 2024-01-15T18:09:23.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.806+0000 7fdda8e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdda40d0540 con 0x7fdda406b810 2024-01-15T18:09:23.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.806+0000 7fdda8e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdda40b0440 con 0x7fdda415d570 2024-01-15T18:09:23.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.806+0000 7fdda1e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdda415d190 0x7fdda40cfcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43594/0 (socket says 172.21.15.27:43594) 2024-01-15T18:09:23.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.806+0000 7fdda1e00640 1 -- 172.21.15.27:0/112874296 learned_addr learned my addr 172.21.15.27:0/112874296 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:23.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.806+0000 7fdda1400640 1 --2- 172.21.15.27:0/112874296 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdda415d570 0x7fdda406be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:23.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.806+0000 7fdda2800640 1 --2- 172.21.15.27:0/112874296 >> v2:172.21.15.93:3300/0 conn(0x7fdda406b810 0x7fdda406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:23.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.806+0000 7fdda0a00640 1 -- 172.21.15.27:0/112874296 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065925083 0 0) 0x7fdda40b0d70 con 0x7fdda415d190 2024-01-15T18:09:23.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.806+0000 7fdda0a00640 1 -- 172.21.15.27:0/112874296 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd84003530 con 0x7fdda415d190 2024-01-15T18:09:23.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.806+0000 7fdda1400640 1 -- 172.21.15.27:0/112874296 >> v2:172.21.15.93:3300/0 conn(0x7fdda406b810 msgr2=0x7fdda406c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:23.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.806+0000 7fdda1400640 1 --2- 172.21.15.27:0/112874296 >> v2:172.21.15.93:3300/0 conn(0x7fdda406b810 0x7fdda406c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:23.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.806+0000 7fdda1400640 1 -- 172.21.15.27:0/112874296 >> v1:172.21.15.27:6789/0 conn(0x7fdda415d190 legacy=0x7fdda40cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:23.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.806+0000 7fdda1400640 1 -- 172.21.15.27:0/112874296 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda40b0720 con 0x7fdda415d570 2024-01-15T18:09:23.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.806+0000 7fdda2800640 1 --2- 172.21.15.27:0/112874296 >> v2:172.21.15.93:3300/0 conn(0x7fdda406b810 0x7fdda406c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:09:23.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.810+0000 7fdda1400640 1 --2- 172.21.15.27:0/112874296 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdda415d570 0x7fdda406be10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c0097c0 tx=0x7fdd8c01c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe4af3e9dc878e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:23.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.810+0000 7fdda0a00640 1 -- 172.21.15.27:0/112874296 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c0ab670 con 0x7fdda415d570 2024-01-15T18:09:23.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.810+0000 7fdda0a00640 1 -- 172.21.15.27:0/112874296 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd8c0ab7d0 con 0x7fdda415d570 2024-01-15T18:09:23.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.810+0000 7fdda0a00640 1 -- 172.21.15.27:0/112874296 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c0abb60 con 0x7fdda415d570 2024-01-15T18:09:23.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.810+0000 7fdda8e00640 1 -- 172.21.15.27:0/112874296 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdda415d570 msgr2=0x7fdda406be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:23.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.810+0000 7fdda8e00640 1 --2- 172.21.15.27:0/112874296 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdda415d570 0x7fdda406be10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c0097c0 tx=0x7fdd8c01c450 comp rx=0 tx=0).stop 2024-01-15T18:09:23.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.810+0000 7fdda8e00640 1 -- 172.21.15.27:0/112874296 shutdown_connections 2024-01-15T18:09:23.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.810+0000 7fdda8e00640 1 --2- 172.21.15.27:0/112874296 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdda415d570 0x7fdda406be10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:23.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.810+0000 7fdda8e00640 1 --2- 172.21.15.27:0/112874296 >> v2:172.21.15.93:3300/0 conn(0x7fdda406b810 0x7fdda406c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:23.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.810+0000 7fdda8e00640 1 -- 172.21.15.27:0/112874296 >> 172.21.15.27:0/112874296 conn(0x7fdda40d0e70 msgr2=0x7fdda40d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:23.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.810+0000 7fdda8e00640 1 -- 172.21.15.27:0/112874296 shutdown_connections 2024-01-15T18:09:23.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.810+0000 7fdda8e00640 1 -- 172.21.15.27:0/112874296 wait complete. 2024-01-15T18:09:23.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.814+0000 7fdda8e00640 1 Processor -- start 2024-01-15T18:09:23.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.814+0000 7fdda8e00640 1 -- start start 2024-01-15T18:09:23.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.814+0000 7fdda8e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdda406b810 0x7fdda4160fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:23.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.814+0000 7fdda8e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdda415d190 0x7fdda4158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:23.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.814+0000 7fdda8e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdda415d570 0x7fdda41589c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:23.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.814+0000 7fdda8e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda40b3df0 con 0x7fdda415d190 2024-01-15T18:09:23.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.814+0000 7fdda8e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda40b4090 con 0x7fdda406b810 2024-01-15T18:09:23.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.814+0000 7fdda8e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdda40b4370 con 0x7fdda415d570 2024-01-15T18:09:23.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.814+0000 7fdda1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdda415d190 0x7fdda4158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:23.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.814+0000 7fdda1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdda415d190 0x7fdda4158480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48234/0 (socket says 172.21.15.27:48234) 2024-01-15T18:09:23.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.814+0000 7fdda1400640 1 -- 172.21.15.27:0/4238592334 learned_addr learned my addr 172.21.15.27:0/4238592334 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:23.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.814+0000 7fdda2800640 1 --2- 172.21.15.27:0/4238592334 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdda415d570 0x7fdda41589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:23.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.814+0000 7fdda1e00640 1 --2- 172.21.15.27:0/4238592334 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdda406b810 0x7fdda4160fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:23.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.814+0000 7fdda1400640 1 -- 172.21.15.27:0/4238592334 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdda415d570 msgr2=0x7fdda41589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:23.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.814+0000 7fdda1400640 1 --2- 172.21.15.27:0/4238592334 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdda415d570 0x7fdda41589c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:23.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.814+0000 7fdda1400640 1 -- 172.21.15.27:0/4238592334 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdda406b810 msgr2=0x7fdda4160fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:23.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.814+0000 7fdda1400640 1 --2- 172.21.15.27:0/4238592334 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdda406b810 0x7fdda4160fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:23.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.814+0000 7fdda1400640 1 -- 172.21.15.27:0/4238592334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd9c009a70 con 0x7fdda415d190 2024-01-15T18:09:23.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.814+0000 7fdda1400640 1 --2- 172.21.15.27:0/4238592334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdda415d190 0x7fdda4158480 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c0ad040 tx=0x7fdd8c01ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:23.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.814+0000 7fdd9a000640 1 -- 172.21.15.27:0/4238592334 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c0b5810 con 0x7fdda415d190 2024-01-15T18:09:23.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.814+0000 7fdd9a000640 1 -- 172.21.15.27:0/4238592334 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd8c0b59b0 con 0x7fdda415d190 2024-01-15T18:09:23.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.814+0000 7fdd9a000640 1 -- 172.21.15.27:0/4238592334 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c0b5d60 con 0x7fdda415d190 2024-01-15T18:09:23.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.814+0000 7fdda8e00640 1 -- 172.21.15.27:0/4238592334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd8c009420 con 0x7fdda415d190 2024-01-15T18:09:23.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.814+0000 7fdda8e00640 1 -- 172.21.15.27:0/4238592334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdda40b4930 con 0x7fdda415d190 2024-01-15T18:09:23.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.818+0000 7fdda8e00640 1 -- 172.21.15.27:0/4238592334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdda40b4b20 con 0x7fdda415d190 2024-01-15T18:09:23.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.818+0000 7fdd9a000640 1 -- 172.21.15.27:0/4238592334 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7fdd8c0bf410 con 0x7fdda415d190 2024-01-15T18:09:23.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.818+0000 7fdd9a000640 1 --2- 172.21.15.27:0/4238592334 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd7c066e90 0x7fdd7c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:23.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.818+0000 7fdd9a000640 1 -- 172.21.15.27:0/4238592334 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fdd8c11fe40 con 0x7fdda415d190 2024-01-15T18:09:23.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.818+0000 7fdda1e00640 1 --2- 172.21.15.27:0/4238592334 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd7c066e90 0x7fdd7c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:23.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.818+0000 7fdda1e00640 1 --2- 172.21.15.27:0/4238592334 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd7c066e90 0x7fdd7c069350 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fdd940011f0 tx=0x7fdd94009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:23.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:23.822+0000 7fdd9a000640 1 -- 172.21.15.27:0/4238592334 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fdd8c0ec900 con 0x7fdda415d190 2024-01-15T18:09:24.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:24.014+0000 7fdda8e00640 1 -- 172.21.15.27:0/4238592334 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fdda404e850 con 0x7fdda415d190 2024-01-15T18:09:24.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:24.938+0000 7fdd9a000640 1 -- 172.21.15.27:0/4238592334 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v38) v1 ==== 136+0+0 (secure 0 0 0) 0x7fdd8c0eff20 con 0x7fdda415d190 2024-01-15T18:09:24.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:24.942+0000 7fdda8e00640 1 -- 172.21.15.27:0/4238592334 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd7c066e90 msgr2=0x7fdd7c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:24.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:24.942+0000 7fdda8e00640 1 --2- 172.21.15.27:0/4238592334 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd7c066e90 0x7fdd7c069350 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fdd940011f0 tx=0x7fdd94009040 comp rx=0 tx=0).stop 2024-01-15T18:09:24.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:24.942+0000 7fdda8e00640 1 -- 172.21.15.27:0/4238592334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdda415d190 msgr2=0x7fdda4158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:24.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:24.942+0000 7fdda8e00640 1 --2- 172.21.15.27:0/4238592334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdda415d190 0x7fdda4158480 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c0ad040 tx=0x7fdd8c01ac90 comp rx=0 tx=0).stop 2024-01-15T18:09:24.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:24.942+0000 7fdda8e00640 1 -- 172.21.15.27:0/4238592334 shutdown_connections 2024-01-15T18:09:24.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:24.942+0000 7fdda8e00640 1 --2- 172.21.15.27:0/4238592334 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd7c066e90 0x7fdd7c069350 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:24.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:24.942+0000 7fdda8e00640 1 --2- 172.21.15.27:0/4238592334 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdda415d570 0x7fdda41589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:24.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:24.942+0000 7fdda8e00640 1 --2- 172.21.15.27:0/4238592334 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdda415d190 0x7fdda4158480 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:24.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:24.942+0000 7fdda8e00640 1 --2- 172.21.15.27:0/4238592334 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdda406b810 0x7fdda4160fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:24.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:24.942+0000 7fdda8e00640 1 -- 172.21.15.27:0/4238592334 >> 172.21.15.27:0/4238592334 conn(0x7fdda40d0e70 msgr2=0x7fdda406e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:24.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:24.942+0000 7fdda8e00640 1 -- 172.21.15.27:0/4238592334 shutdown_connections 2024-01-15T18:09:24.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:24.942+0000 7fdda8e00640 1 -- 172.21.15.27:0/4238592334 wait complete. 2024-01-15T18:09:24.973 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:09:25.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.114+0000 7feb83800640 1 Processor -- start 2024-01-15T18:09:25.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.114+0000 7feb83800640 1 -- start start 2024-01-15T18:09:25.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.114+0000 7feb83800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7feb7c143da0 0x7feb7c06d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:25.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.118+0000 7feb83800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feb7c0cad40 0x7feb7c0cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:25.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.118+0000 7feb83800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb7c06c750 con 0x7feb7c06c200 2024-01-15T18:09:25.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.118+0000 7feb83800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb7c0cb710 con 0x7feb7c143da0 2024-01-15T18:09:25.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.118+0000 7feb83800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb7c144180 con 0x7feb7c0cad40 2024-01-15T18:09:25.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.118+0000 7feb80e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7feb7c143da0 0x7feb7c06d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:25.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.118+0000 7feb80e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7feb7c143da0 0x7feb7c06d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48248/0 (socket says 172.21.15.27:48248) 2024-01-15T18:09:25.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.118+0000 7feb80e00640 1 -- 172.21.15.27:0/3279842239 learned_addr learned my addr 172.21.15.27:0/3279842239 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:25.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.118+0000 7feb80e00640 1 -- 172.21.15.27:0/3279842239 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feb7c0cad40 msgr2=0x7feb7c0cb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:25.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.118+0000 7feb80e00640 1 --2- 172.21.15.27:0/3279842239 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feb7c0cad40 0x7feb7c0cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:25.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.118+0000 7feb80e00640 1 -- 172.21.15.27:0/3279842239 >> v1:172.21.15.93:6789/0 conn(0x7feb7c06c200 legacy=0x7feb7c0cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:25.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.118+0000 7feb80e00640 1 -- 172.21.15.27:0/3279842239 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb7c06ca30 con 0x7feb7c143da0 2024-01-15T18:09:25.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.118+0000 7feb80e00640 1 --2- 172.21.15.27:0/3279842239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7c143da0 0x7feb7c06d7f0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7feb6c009870 tx=0x7feb6c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95b8465713453f44 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:25.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.118+0000 7feb7b400640 1 -- 172.21.15.27:0/3279842239 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c0ac410 con 0x7feb7c143da0 2024-01-15T18:09:25.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.118+0000 7feb7b400640 1 -- 172.21.15.27:0/3279842239 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6c0ac570 con 0x7feb7c143da0 2024-01-15T18:09:25.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.118+0000 7feb7b400640 1 -- 172.21.15.27:0/3279842239 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c0ac900 con 0x7feb7c143da0 2024-01-15T18:09:25.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.122+0000 7feb83800640 1 -- 172.21.15.27:0/3279842239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7c143da0 msgr2=0x7feb7c06d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:25.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.122+0000 7feb83800640 1 --2- 172.21.15.27:0/3279842239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7c143da0 0x7feb7c06d7f0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7feb6c009870 tx=0x7feb6c01c500 comp rx=0 tx=0).stop 2024-01-15T18:09:25.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.122+0000 7feb83800640 1 -- 172.21.15.27:0/3279842239 shutdown_connections 2024-01-15T18:09:25.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.122+0000 7feb83800640 1 --2- 172.21.15.27:0/3279842239 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feb7c0cad40 0x7feb7c0cb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:25.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.122+0000 7feb83800640 1 --2- 172.21.15.27:0/3279842239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7c143da0 0x7feb7c06d7f0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:25.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.122+0000 7feb83800640 1 -- 172.21.15.27:0/3279842239 >> 172.21.15.27:0/3279842239 conn(0x7feb7c0b0850 msgr2=0x7feb7c0b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:25.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.122+0000 7feb83800640 1 -- 172.21.15.27:0/3279842239 shutdown_connections 2024-01-15T18:09:25.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.122+0000 7feb83800640 1 -- 172.21.15.27:0/3279842239 wait complete. 2024-01-15T18:09:25.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.122+0000 7feb83800640 1 Processor -- start 2024-01-15T18:09:25.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.122+0000 7feb83800640 1 -- start start 2024-01-15T18:09:25.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.122+0000 7feb83800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7c06c200 0x7feb7c15c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:25.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.122+0000 7feb83800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feb7c0cad40 0x7feb7c13e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:25.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.122+0000 7feb83800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feb7c143da0 0x7feb7c0ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:25.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.122+0000 7feb83800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7c140ca0 con 0x7feb7c06c200 2024-01-15T18:09:25.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.122+0000 7feb83800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7c0b3b10 con 0x7feb7c143da0 2024-01-15T18:09:25.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.126+0000 7feb83800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb7c140650 con 0x7feb7c0cad40 2024-01-15T18:09:25.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.126+0000 7feb80e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7c06c200 0x7feb7c15c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:25.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.126+0000 7feb80e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7c06c200 0x7feb7c15c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48260/0 (socket says 172.21.15.27:48260) 2024-01-15T18:09:25.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.126+0000 7feb80e00640 1 -- 172.21.15.27:0/365071503 learned_addr learned my addr 172.21.15.27:0/365071503 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:25.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.126+0000 7feb7be00640 1 --2- 172.21.15.27:0/365071503 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feb7c0cad40 0x7feb7c13e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:25.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.126+0000 7feb81800640 1 --2- 172.21.15.27:0/365071503 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feb7c143da0 0x7feb7c0ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:25.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.126+0000 7feb80e00640 1 -- 172.21.15.27:0/365071503 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feb7c0cad40 msgr2=0x7feb7c13e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:25.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.126+0000 7feb80e00640 1 --2- 172.21.15.27:0/365071503 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feb7c0cad40 0x7feb7c13e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:25.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.126+0000 7feb80e00640 1 -- 172.21.15.27:0/365071503 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feb7c143da0 msgr2=0x7feb7c0ae980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:25.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.126+0000 7feb80e00640 1 --2- 172.21.15.27:0/365071503 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feb7c143da0 0x7feb7c0ae980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:25.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.126+0000 7feb80e00640 1 -- 172.21.15.27:0/365071503 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb6c0094d0 con 0x7feb7c06c200 2024-01-15T18:09:25.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.126+0000 7feb81800640 1 --2- 172.21.15.27:0/365071503 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feb7c143da0 0x7feb7c0ae980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:09:25.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.126+0000 7feb80e00640 1 --2- 172.21.15.27:0/365071503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7c06c200 0x7feb7c15c130 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0af040 tx=0x7feb6c0192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:25.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.126+0000 7feb78c00640 1 -- 172.21.15.27:0/365071503 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c0ac410 con 0x7feb7c06c200 2024-01-15T18:09:25.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.126+0000 7feb78c00640 1 -- 172.21.15.27:0/365071503 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6c0b5630 con 0x7feb7c06c200 2024-01-15T18:09:25.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.126+0000 7feb78c00640 1 -- 172.21.15.27:0/365071503 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c0b59e0 con 0x7feb7c06c200 2024-01-15T18:09:25.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.126+0000 7feb83800640 1 -- 172.21.15.27:0/365071503 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb7c140930 con 0x7feb7c06c200 2024-01-15T18:09:25.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.126+0000 7feb83800640 1 -- 172.21.15.27:0/365071503 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb7c13ed70 con 0x7feb7c06c200 2024-01-15T18:09:25.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.126+0000 7feb83800640 1 -- 172.21.15.27:0/365071503 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb7c04e850 con 0x7feb7c06c200 2024-01-15T18:09:25.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.126+0000 7feb78c00640 1 -- 172.21.15.27:0/365071503 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7feb6c0b3070 con 0x7feb7c06c200 2024-01-15T18:09:25.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.130+0000 7feb78c00640 1 --2- 172.21.15.27:0/365071503 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7feb54066ec0 0x7feb54069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:25.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.130+0000 7feb78c00640 1 -- 172.21.15.27:0/365071503 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7feb6c11f580 con 0x7feb7c06c200 2024-01-15T18:09:25.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.130+0000 7feb7be00640 1 --2- 172.21.15.27:0/365071503 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7feb54066ec0 0x7feb54069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:25.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.130+0000 7feb7be00640 1 --2- 172.21.15.27:0/365071503 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7feb54066ec0 0x7feb54069380 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7feb7c0b0160 tx=0x7feb64007e40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:25.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.134+0000 7feb78c00640 1 -- 172.21.15.27:0/365071503 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7feb6c0ebde0 con 0x7feb7c06c200 2024-01-15T18:09:25.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.310+0000 7feb83800640 1 -- 172.21.15.27:0/365071503 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7feb7c06d7f0 con 0x7feb54066ec0 2024-01-15T18:09:25.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.314+0000 7feb78c00640 1 -- 172.21.15.27:0/365071503 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7feb7c06d7f0 con 0x7feb54066ec0 2024-01-15T18:09:25.319 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:09:25.319 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:09:25.319 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-15T18:09:25.319 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:09:25.319 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:09:25.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.314+0000 7feb83800640 1 -- 172.21.15.27:0/365071503 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7feb54066ec0 msgr2=0x7feb54069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:25.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.314+0000 7feb83800640 1 --2- 172.21.15.27:0/365071503 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7feb54066ec0 0x7feb54069380 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7feb7c0b0160 tx=0x7feb64007e40 comp rx=0 tx=0).stop 2024-01-15T18:09:25.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.314+0000 7feb83800640 1 -- 172.21.15.27:0/365071503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7c06c200 msgr2=0x7feb7c15c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:25.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.314+0000 7feb83800640 1 --2- 172.21.15.27:0/365071503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7c06c200 0x7feb7c15c130 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0af040 tx=0x7feb6c0192a0 comp rx=0 tx=0).stop 2024-01-15T18:09:25.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.314+0000 7feb83800640 1 -- 172.21.15.27:0/365071503 shutdown_connections 2024-01-15T18:09:25.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.314+0000 7feb83800640 1 --2- 172.21.15.27:0/365071503 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7feb54066ec0 0x7feb54069380 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:25.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.314+0000 7feb83800640 1 --2- 172.21.15.27:0/365071503 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feb7c143da0 0x7feb7c0ae980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:25.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.314+0000 7feb83800640 1 --2- 172.21.15.27:0/365071503 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feb7c0cad40 0x7feb7c13e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:25.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.314+0000 7feb83800640 1 --2- 172.21.15.27:0/365071503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb7c06c200 0x7feb7c15c130 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:25.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.314+0000 7feb83800640 1 -- 172.21.15.27:0/365071503 >> 172.21.15.27:0/365071503 conn(0x7feb7c0b0850 msgr2=0x7feb7c13db90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:25.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.314+0000 7feb83800640 1 -- 172.21.15.27:0/365071503 shutdown_connections 2024-01-15T18:09:25.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.314+0000 7feb83800640 1 -- 172.21.15.27:0/365071503 wait complete. 2024-01-15T18:09:25.340 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:09:25.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.478+0000 7f280d800640 1 Processor -- start 2024-01-15T18:09:25.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.478+0000 7f280d800640 1 -- start start 2024-01-15T18:09:25.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.478+0000 7f280d800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2808158900 0x7f280815f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:25.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.478+0000 7f280d800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28080b1cb0 con 0x7f28080cf2e0 2024-01-15T18:09:25.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.478+0000 7f280d800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28080b1f00 con 0x7f280815bf00 2024-01-15T18:09:25.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.478+0000 7f280d800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28080b21e0 con 0x7f2808158900 2024-01-15T18:09:25.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.478+0000 7f2807400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f28080cf2e0 0x7f28080701e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43598/0 (socket says 172.21.15.27:43598) 2024-01-15T18:09:25.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.478+0000 7f2806a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2808158900 0x7f280815f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:25.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.478+0000 7f2807400640 1 -- 172.21.15.27:0/1720639211 learned_addr learned my addr 172.21.15.27:0/1720639211 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:25.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.478+0000 7f2806a00640 1 -- 172.21.15.27:0/1720639211 >> v1:172.21.15.93:6789/0 conn(0x7f280815bf00 legacy=0x7f28080cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:25.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.478+0000 7f2806a00640 1 -- 172.21.15.27:0/1720639211 >> v1:172.21.15.27:6789/0 conn(0x7f28080cf2e0 legacy=0x7f28080701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:25.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.478+0000 7f2806a00640 1 -- 172.21.15.27:0/1720639211 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f280815ea70 con 0x7f2808158900 2024-01-15T18:09:25.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.482+0000 7f2806a00640 1 --2- 172.21.15.27:0/1720639211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808158900 0x7f280815f550 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f27f80097c0 tx=0x7f27f801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4927b4b976f67bc2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:25.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.482+0000 7f2805600640 1 -- 172.21.15.27:0/1720639211 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f80ab600 con 0x7f2808158900 2024-01-15T18:09:25.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.482+0000 7f2805600640 1 -- 172.21.15.27:0/1720639211 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f80a7050 con 0x7f2808158900 2024-01-15T18:09:25.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.482+0000 7f2805600640 1 -- 172.21.15.27:0/1720639211 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f80ab980 con 0x7f2808158900 2024-01-15T18:09:25.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.482+0000 7f280d800640 1 -- 172.21.15.27:0/1720639211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808158900 msgr2=0x7f280815f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:25.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.482+0000 7f280d800640 1 --2- 172.21.15.27:0/1720639211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808158900 0x7f280815f550 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f27f80097c0 tx=0x7f27f801c480 comp rx=0 tx=0).stop 2024-01-15T18:09:25.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.482+0000 7f280d800640 1 -- 172.21.15.27:0/1720639211 shutdown_connections 2024-01-15T18:09:25.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.482+0000 7f280d800640 1 --2- 172.21.15.27:0/1720639211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808158900 0x7f280815f550 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:25.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.482+0000 7f280d800640 1 -- 172.21.15.27:0/1720639211 >> 172.21.15.27:0/1720639211 conn(0x7f28080cb700 msgr2=0x7f28080cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:25.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.482+0000 7f280d800640 1 -- 172.21.15.27:0/1720639211 shutdown_connections 2024-01-15T18:09:25.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.482+0000 7f280d800640 1 -- 172.21.15.27:0/1720639211 wait complete. 2024-01-15T18:09:25.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.486+0000 7f280d800640 1 Processor -- start 2024-01-15T18:09:25.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.486+0000 7f280d800640 1 -- start start 2024-01-15T18:09:25.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.486+0000 7f280d800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28080cf2e0 0x7f2808137d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:25.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.486+0000 7f280d800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808158900 0x7f28080cd220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:25.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.486+0000 7f280d800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f280815bf00 0x7f28080cde10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:25.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.486+0000 7f280d800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2808138250 con 0x7f2808158900 2024-01-15T18:09:25.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.486+0000 7f280d800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28080cd760 con 0x7f28080cf2e0 2024-01-15T18:09:25.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.486+0000 7f280d800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f280813a2b0 con 0x7f280815bf00 2024-01-15T18:09:25.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.486+0000 7f2806000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808158900 0x7f28080cd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:25.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.486+0000 7f2806000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808158900 0x7f28080cd220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48276/0 (socket says 172.21.15.27:48276) 2024-01-15T18:09:25.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.486+0000 7f2806000640 1 -- 172.21.15.27:0/161024619 learned_addr learned my addr 172.21.15.27:0/161024619 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:25.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.486+0000 7f2806a00640 1 --2- 172.21.15.27:0/161024619 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28080cf2e0 0x7f2808137d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:25.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.486+0000 7f2807400640 1 --2- 172.21.15.27:0/161024619 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f280815bf00 0x7f28080cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:25.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.486+0000 7f2806000640 1 -- 172.21.15.27:0/161024619 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f280815bf00 msgr2=0x7f28080cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:25.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.486+0000 7f2806000640 1 --2- 172.21.15.27:0/161024619 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f280815bf00 0x7f28080cde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:25.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.486+0000 7f2806000640 1 -- 172.21.15.27:0/161024619 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28080cf2e0 msgr2=0x7f2808137d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:25.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.486+0000 7f2806000640 1 --2- 172.21.15.27:0/161024619 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28080cf2e0 0x7f2808137d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:25.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.486+0000 7f2806000640 1 -- 172.21.15.27:0/161024619 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f8009420 con 0x7f2808158900 2024-01-15T18:09:25.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.486+0000 7f2807400640 1 --2- 172.21.15.27:0/161024619 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f280815bf00 0x7f28080cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:25.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.486+0000 7f2806000640 1 --2- 172.21.15.27:0/161024619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808158900 0x7f28080cd220 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f27f000c3c0 tx=0x7f27f000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:25.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.486+0000 7f27fea00640 1 -- 172.21.15.27:0/161024619 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f00052d0 con 0x7f2808158900 2024-01-15T18:09:25.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.486+0000 7f27fea00640 1 -- 172.21.15.27:0/161024619 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f0005430 con 0x7f2808158900 2024-01-15T18:09:25.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.486+0000 7f280d800640 1 -- 172.21.15.27:0/161024619 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f280814ce30 con 0x7f2808158900 2024-01-15T18:09:25.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.486+0000 7f27fea00640 1 -- 172.21.15.27:0/161024619 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f0003840 con 0x7f2808158900 2024-01-15T18:09:25.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.486+0000 7f280d800640 1 -- 172.21.15.27:0/161024619 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2808159690 con 0x7f2808158900 2024-01-15T18:09:25.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.490+0000 7f280d800640 1 -- 172.21.15.27:0/161024619 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f280804e850 con 0x7f2808158900 2024-01-15T18:09:25.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.490+0000 7f27fea00640 1 -- 172.21.15.27:0/161024619 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f27f00039a0 con 0x7f2808158900 2024-01-15T18:09:25.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.490+0000 7f27fea00640 1 --2- 172.21.15.27:0/161024619 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f27e0066ec0 0x7f27e0069380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:25.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.490+0000 7f27fea00640 1 -- 172.21.15.27:0/161024619 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f27f00843c0 con 0x7f2808158900 2024-01-15T18:09:25.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.490+0000 7f2806a00640 1 --2- 172.21.15.27:0/161024619 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f27e0066ec0 0x7f27e0069380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:25.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.490+0000 7f2806a00640 1 --2- 172.21.15.27:0/161024619 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f27e0066ec0 0x7f27e0069380 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f27f80ad000 tx=0x7f27f80a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:25.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.494+0000 7f27fea00640 1 -- 172.21.15.27:0/161024619 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f27f0050dc0 con 0x7f2808158900 2024-01-15T18:09:25.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.650+0000 7f280d800640 1 -- 172.21.15.27:0/161024619 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f280806d540 con 0x7f2808158900 2024-01-15T18:09:25.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.650+0000 7f27fea00640 1 -- 172.21.15.27:0/161024619 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7f27f00543e0 con 0x7f2808158900 2024-01-15T18:09:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.654+0000 7f280d800640 1 -- 172.21.15.27:0/161024619 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f27e0066ec0 msgr2=0x7f27e0069380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:25.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.654+0000 7f280d800640 1 --2- 172.21.15.27:0/161024619 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f27e0066ec0 0x7f27e0069380 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f27f80ad000 tx=0x7f27f80a6000 comp rx=0 tx=0).stop 2024-01-15T18:09:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.654+0000 7f280d800640 1 -- 172.21.15.27:0/161024619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808158900 msgr2=0x7f28080cd220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.654+0000 7f280d800640 1 --2- 172.21.15.27:0/161024619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808158900 0x7f28080cd220 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f27f000c3c0 tx=0x7f27f000c880 comp rx=0 tx=0).stop 2024-01-15T18:09:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.654+0000 7f280d800640 1 -- 172.21.15.27:0/161024619 shutdown_connections 2024-01-15T18:09:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.654+0000 7f280d800640 1 --2- 172.21.15.27:0/161024619 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f27e0066ec0 0x7f27e0069380 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.654+0000 7f280d800640 1 --2- 172.21.15.27:0/161024619 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f280815bf00 0x7f28080cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.654+0000 7f280d800640 1 --2- 172.21.15.27:0/161024619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2808158900 0x7f28080cd220 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.654+0000 7f280d800640 1 --2- 172.21.15.27:0/161024619 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28080cf2e0 0x7f2808137d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.654+0000 7f280d800640 1 -- 172.21.15.27:0/161024619 >> 172.21.15.27:0/161024619 conn(0x7f28080cb700 msgr2=0x7f2808071090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.654+0000 7f280d800640 1 -- 172.21.15.27:0/161024619 shutdown_connections 2024-01-15T18:09:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.654+0000 7f280d800640 1 -- 172.21.15.27:0/161024619 wait complete. 2024-01-15T18:09:25.680 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-15T18:09:25.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.818+0000 7f5ba8e00640 1 Processor -- start 2024-01-15T18:09:25.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.818+0000 7f5ba8e00640 1 -- start start 2024-01-15T18:09:25.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.822+0000 7f5ba8e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5ba415ad20 0x7f5ba415b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:25.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.822+0000 7f5ba8e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f5ba415d900 0x7f5ba415dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:25.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.822+0000 7f5ba8e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ba415a610 con 0x7f5ba4070ee0 2024-01-15T18:09:25.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.822+0000 7f5ba8e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ba415e3c0 con 0x7f5ba415d900 2024-01-15T18:09:25.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.822+0000 7f5ba8e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba415e6a0 con 0x7f5ba415ad20 2024-01-15T18:09:25.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.822+0000 7f5ba1400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f5ba415d900 0x7f5ba415dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:25.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.822+0000 7f5ba1e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5ba415ad20 0x7f5ba415b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:25.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.822+0000 7f5ba2800640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7f5ba4070ee0 0x7f5ba406d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:45320/0 (socket says 172.21.15.27:45320) 2024-01-15T18:09:25.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.822+0000 7f5ba2800640 1 -- 172.21.15.27:0/1176135798 learned_addr learned my addr 172.21.15.27:0/1176135798 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:25.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.822+0000 7f5ba0a00640 1 -- 172.21.15.27:0/1176135798 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3411225623 0 0) 0x7f5ba415a610 con 0x7f5ba4070ee0 2024-01-15T18:09:25.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.822+0000 7f5ba0a00640 1 -- 172.21.15.27:0/1176135798 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b84003530 con 0x7f5ba4070ee0 2024-01-15T18:09:25.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.822+0000 7f5ba1400640 1 -- 172.21.15.27:0/1176135798 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5ba415ad20 msgr2=0x7f5ba415b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:25.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.822+0000 7f5ba1400640 1 --2- 172.21.15.27:0/1176135798 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5ba415ad20 0x7f5ba415b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:25.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.822+0000 7f5ba1400640 1 -- 172.21.15.27:0/1176135798 >> v1:172.21.15.93:6789/0 conn(0x7f5ba4070ee0 legacy=0x7f5ba406d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:25.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.822+0000 7f5ba1400640 1 -- 172.21.15.27:0/1176135798 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba415b6e0 con 0x7f5ba415d900 2024-01-15T18:09:25.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.822+0000 7f5ba1e00640 1 --2- 172.21.15.27:0/1176135798 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5ba415ad20 0x7f5ba415b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:09:25.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.826+0000 7f5ba1400640 1 --2- 172.21.15.27:0/1176135798 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5ba415d900 0x7f5ba415dd80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c0097c0 tx=0x7f5b8c01c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5836eb369a3e4256 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:25.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.826+0000 7f5ba0a00640 1 -- 172.21.15.27:0/1176135798 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c0ab670 con 0x7f5ba415d900 2024-01-15T18:09:25.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.826+0000 7f5ba0a00640 1 -- 172.21.15.27:0/1176135798 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b8c0ab7d0 con 0x7f5ba415d900 2024-01-15T18:09:25.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.826+0000 7f5ba0a00640 1 -- 172.21.15.27:0/1176135798 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c0abb60 con 0x7f5ba415d900 2024-01-15T18:09:25.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.826+0000 7f5ba8e00640 1 -- 172.21.15.27:0/1176135798 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5ba415d900 msgr2=0x7f5ba415dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:25.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.826+0000 7f5ba8e00640 1 --2- 172.21.15.27:0/1176135798 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5ba415d900 0x7f5ba415dd80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c0097c0 tx=0x7f5b8c01c450 comp rx=0 tx=0).stop 2024-01-15T18:09:25.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.826+0000 7f5ba8e00640 1 -- 172.21.15.27:0/1176135798 shutdown_connections 2024-01-15T18:09:25.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.826+0000 7f5ba8e00640 1 --2- 172.21.15.27:0/1176135798 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5ba415d900 0x7f5ba415dd80 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:25.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.826+0000 7f5ba8e00640 1 --2- 172.21.15.27:0/1176135798 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5ba415ad20 0x7f5ba415b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:25.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.826+0000 7f5ba8e00640 1 -- 172.21.15.27:0/1176135798 >> 172.21.15.27:0/1176135798 conn(0x7f5ba40cb700 msgr2=0x7f5ba40cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:25.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.826+0000 7f5ba8e00640 1 -- 172.21.15.27:0/1176135798 shutdown_connections 2024-01-15T18:09:25.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.826+0000 7f5ba8e00640 1 -- 172.21.15.27:0/1176135798 wait complete. 2024-01-15T18:09:25.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.826+0000 7f5ba8e00640 1 Processor -- start 2024-01-15T18:09:25.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.826+0000 7f5ba8e00640 1 -- start start 2024-01-15T18:09:25.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.830+0000 7f5ba8e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5ba4070ee0 0x7f5ba4141610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:25.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.830+0000 7f5ba8e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ba415ad20 0x7f5ba40ce800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:25.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.830+0000 7f5ba8e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5ba415d900 0x7f5ba41f73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:25.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.830+0000 7f5ba8e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba40cf580 con 0x7f5ba415ad20 2024-01-15T18:09:25.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.830+0000 7f5ba8e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba4155e80 con 0x7f5ba415d900 2024-01-15T18:09:25.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.830+0000 7f5ba8e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba40cf2a0 con 0x7f5ba4070ee0 2024-01-15T18:09:25.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.830+0000 7f5ba1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ba415ad20 0x7f5ba40ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:25.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.830+0000 7f5ba1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ba415ad20 0x7f5ba40ce800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48286/0 (socket says 172.21.15.27:48286) 2024-01-15T18:09:25.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.830+0000 7f5ba1400640 1 -- 172.21.15.27:0/4145735535 learned_addr learned my addr 172.21.15.27:0/4145735535 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:25.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.830+0000 7f5ba1e00640 1 --2- 172.21.15.27:0/4145735535 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5ba4070ee0 0x7f5ba4141610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:25.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.830+0000 7f5ba2800640 1 --2- 172.21.15.27:0/4145735535 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5ba415d900 0x7f5ba41f73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:25.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.830+0000 7f5ba1400640 1 -- 172.21.15.27:0/4145735535 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5ba4070ee0 msgr2=0x7f5ba4141610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:25.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.830+0000 7f5ba1400640 1 --2- 172.21.15.27:0/4145735535 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5ba4070ee0 0x7f5ba4141610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:25.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.830+0000 7f5ba1400640 1 -- 172.21.15.27:0/4145735535 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5ba415d900 msgr2=0x7f5ba41f73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:25.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.830+0000 7f5ba1400640 1 --2- 172.21.15.27:0/4145735535 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5ba415d900 0x7f5ba41f73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:25.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.830+0000 7f5ba1400640 1 -- 172.21.15.27:0/4145735535 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b94009420 con 0x7f5ba415ad20 2024-01-15T18:09:25.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.830+0000 7f5ba1400640 1 --2- 172.21.15.27:0/4145735535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ba415ad20 0x7f5ba40ce800 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c01a080 tx=0x7f5b8c0191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:25.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.830+0000 7f5b9a000640 1 -- 172.21.15.27:0/4145735535 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c0bf650 con 0x7f5ba415ad20 2024-01-15T18:09:25.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.830+0000 7f5b9a000640 1 -- 172.21.15.27:0/4145735535 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b8c0bf7d0 con 0x7f5ba415ad20 2024-01-15T18:09:25.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.830+0000 7f5b9a000640 1 -- 172.21.15.27:0/4145735535 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c0bfb80 con 0x7f5ba415ad20 2024-01-15T18:09:25.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.830+0000 7f5ba8e00640 1 -- 172.21.15.27:0/4145735535 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b8c009420 con 0x7f5ba415ad20 2024-01-15T18:09:25.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.830+0000 7f5ba8e00640 1 -- 172.21.15.27:0/4145735535 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ba4156400 con 0x7f5ba415ad20 2024-01-15T18:09:25.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.830+0000 7f5ba8e00640 1 -- 172.21.15.27:0/4145735535 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ba404e850 con 0x7f5ba415ad20 2024-01-15T18:09:25.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.834+0000 7f5b9a000640 1 -- 172.21.15.27:0/4145735535 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f5b8c0b43f0 con 0x7f5ba415ad20 2024-01-15T18:09:25.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.834+0000 7f5b9a000640 1 --2- 172.21.15.27:0/4145735535 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5b7c066e90 0x7f5b7c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:25.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.834+0000 7f5b9a000640 1 -- 172.21.15.27:0/4145735535 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f5b8c11fed0 con 0x7f5ba415ad20 2024-01-15T18:09:25.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.834+0000 7f5ba1e00640 1 --2- 172.21.15.27:0/4145735535 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5b7c066e90 0x7f5b7c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:25.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.834+0000 7f5ba1e00640 1 --2- 172.21.15.27:0/4145735535 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5b7c066e90 0x7f5b7c069350 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f5b94009cf0 tx=0x7f5b94009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:25.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:25.838+0000 7f5b9a000640 1 -- 172.21.15.27:0/4145735535 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5b8c0ec870 con 0x7f5ba415ad20 2024-01-15T18:09:26.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.018+0000 7f5ba8e00640 1 -- 172.21.15.27:0/4145735535 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ba415e050 con 0x7f5b7c066e90 2024-01-15T18:09:26.029 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:09:26.021+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:26.029 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:09:26.021+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:26.030 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:09:26.021+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:26.030 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:09:26.021+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:26.030 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:09:26.021+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:26.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.058+0000 7f5b9a000640 1 -- 172.21.15.27:0/4145735535 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5ba415e050 con 0x7f5b7c066e90 2024-01-15T18:09:26.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.062+0000 7f5ba8e00640 1 -- 172.21.15.27:0/4145735535 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5b7c066e90 msgr2=0x7f5b7c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:26.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.062+0000 7f5ba8e00640 1 --2- 172.21.15.27:0/4145735535 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5b7c066e90 0x7f5b7c069350 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f5b94009cf0 tx=0x7f5b94009190 comp rx=0 tx=0).stop 2024-01-15T18:09:26.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.062+0000 7f5ba8e00640 1 -- 172.21.15.27:0/4145735535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ba415ad20 msgr2=0x7f5ba40ce800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:26.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.062+0000 7f5ba8e00640 1 --2- 172.21.15.27:0/4145735535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ba415ad20 0x7f5ba40ce800 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c01a080 tx=0x7f5b8c0191f0 comp rx=0 tx=0).stop 2024-01-15T18:09:26.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.062+0000 7f5ba8e00640 1 -- 172.21.15.27:0/4145735535 shutdown_connections 2024-01-15T18:09:26.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.062+0000 7f5ba8e00640 1 --2- 172.21.15.27:0/4145735535 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5b7c066e90 0x7f5b7c069350 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:26.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.062+0000 7f5ba8e00640 1 --2- 172.21.15.27:0/4145735535 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5ba415d900 0x7f5ba41f73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:26.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.062+0000 7f5ba8e00640 1 --2- 172.21.15.27:0/4145735535 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ba415ad20 0x7f5ba40ce800 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:26.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.062+0000 7f5ba8e00640 1 --2- 172.21.15.27:0/4145735535 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5ba4070ee0 0x7f5ba4141610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:26.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.062+0000 7f5ba8e00640 1 -- 172.21.15.27:0/4145735535 >> 172.21.15.27:0/4145735535 conn(0x7f5ba40cb700 msgr2=0x7f5ba40cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:26.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.062+0000 7f5ba8e00640 1 -- 172.21.15.27:0/4145735535 shutdown_connections 2024-01-15T18:09:26.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.062+0000 7f5ba8e00640 1 -- 172.21.15.27:0/4145735535 wait complete. 2024-01-15T18:09:26.087 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:09:26.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.230+0000 7f33d4800640 1 Processor -- start 2024-01-15T18:09:26.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.230+0000 7f33d4800640 1 -- start start 2024-01-15T18:09:26.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.230+0000 7f33d4800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f33cc15f550 0x7f33cc0cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:26.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.230+0000 7f33d4800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f33cc0cf110 0x7f33cc0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:26.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.230+0000 7f33d4800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33cc0b1cb0 con 0x7f33cc158900 2024-01-15T18:09:26.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.230+0000 7f33d4800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33cc15fa00 con 0x7f33cc0cf110 2024-01-15T18:09:26.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.230+0000 7f33d4800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc0cf540 con 0x7f33cc15f550 2024-01-15T18:09:26.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.230+0000 7f33d2800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f33cc0cf110 0x7f33cc0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:26.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.230+0000 7f33d1e00640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7f33cc158900 0x7f33cc15bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:45336/0 (socket says 172.21.15.27:45336) 2024-01-15T18:09:26.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.230+0000 7f33d1e00640 1 -- 172.21.15.27:0/4171720789 learned_addr learned my addr 172.21.15.27:0/4171720789 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:26.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.230+0000 7f33d1400640 1 --2- 172.21.15.27:0/4171720789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f33cc15f550 0x7f33cc0cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:26.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.230+0000 7f33d0a00640 1 -- 172.21.15.27:0/4171720789 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2331969972 0 0) 0x7f33cc0b1cb0 con 0x7f33cc158900 2024-01-15T18:09:26.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.230+0000 7f33d0a00640 1 -- 172.21.15.27:0/4171720789 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33ac003530 con 0x7f33cc158900 2024-01-15T18:09:26.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.230+0000 7f33d1400640 1 -- 172.21.15.27:0/4171720789 >> v2:172.21.15.27:3300/0 conn(0x7f33cc0cf110 msgr2=0x7f33cc0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:26.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.230+0000 7f33d1400640 1 --2- 172.21.15.27:0/4171720789 >> v2:172.21.15.27:3300/0 conn(0x7f33cc0cf110 0x7f33cc0701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:26.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.230+0000 7f33d1400640 1 -- 172.21.15.27:0/4171720789 >> v1:172.21.15.93:6789/0 conn(0x7f33cc158900 legacy=0x7f33cc15bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:26.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.230+0000 7f33d1400640 1 -- 172.21.15.27:0/4171720789 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33cc0b1f00 con 0x7f33cc15f550 2024-01-15T18:09:26.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.230+0000 7f33d2800640 1 --2- 172.21.15.27:0/4171720789 >> v2:172.21.15.27:3300/0 conn(0x7f33cc0cf110 0x7f33cc0701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:09:26.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.234+0000 7f33d1400640 1 --2- 172.21.15.27:0/4171720789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f33cc15f550 0x7f33cc0cebd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f33b40097c0 tx=0x7f33b401c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24b4621bea591765 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:26.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.234+0000 7f33d0a00640 1 -- 172.21.15.27:0/4171720789 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33cc070f30 con 0x7f33cc15f550 2024-01-15T18:09:26.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.234+0000 7f33d0a00640 1 -- 172.21.15.27:0/4171720789 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b40ab650 con 0x7f33cc15f550 2024-01-15T18:09:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.234+0000 7f33d0a00640 1 -- 172.21.15.27:0/4171720789 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b40ab9c0 con 0x7f33cc15f550 2024-01-15T18:09:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.234+0000 7f33d4800640 1 -- 172.21.15.27:0/4171720789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f33cc15f550 msgr2=0x7f33cc0cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:26.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.234+0000 7f33d4800640 1 --2- 172.21.15.27:0/4171720789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f33cc15f550 0x7f33cc0cebd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f33b40097c0 tx=0x7f33b401c450 comp rx=0 tx=0).stop 2024-01-15T18:09:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.234+0000 7f33d4800640 1 -- 172.21.15.27:0/4171720789 shutdown_connections 2024-01-15T18:09:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.234+0000 7f33d4800640 1 --2- 172.21.15.27:0/4171720789 >> v2:172.21.15.27:3300/0 conn(0x7f33cc0cf110 0x7f33cc0701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.234+0000 7f33d4800640 1 --2- 172.21.15.27:0/4171720789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f33cc15f550 0x7f33cc0cebd0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.234+0000 7f33d4800640 1 -- 172.21.15.27:0/4171720789 >> 172.21.15.27:0/4171720789 conn(0x7f33cc0cb700 msgr2=0x7f33cc0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.238+0000 7f33d4800640 1 -- 172.21.15.27:0/4171720789 shutdown_connections 2024-01-15T18:09:26.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.238+0000 7f33d4800640 1 -- 172.21.15.27:0/4171720789 wait complete. 2024-01-15T18:09:26.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.238+0000 7f33d4800640 1 Processor -- start 2024-01-15T18:09:26.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.238+0000 7f33d4800640 1 -- start start 2024-01-15T18:09:26.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.238+0000 7f33d4800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f33cc0cf110 0x7f33cc150c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:26.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.238+0000 7f33d4800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33cc158900 0x7f33cc152c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:26.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.238+0000 7f33d4800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33cc15f550 0x7f33cc1531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:26.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.238+0000 7f33d4800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc0cdbe0 con 0x7f33cc158900 2024-01-15T18:09:26.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.238+0000 7f33d4800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc15bcf0 con 0x7f33cc15f550 2024-01-15T18:09:26.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.238+0000 7f33d4800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc153870 con 0x7f33cc0cf110 2024-01-15T18:09:26.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.238+0000 7f33d1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33cc158900 0x7f33cc152c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:26.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.238+0000 7f33d1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33cc158900 0x7f33cc152c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48302/0 (socket says 172.21.15.27:48302) 2024-01-15T18:09:26.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.238+0000 7f33d1400640 1 -- 172.21.15.27:0/1172490419 learned_addr learned my addr 172.21.15.27:0/1172490419 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:26.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.238+0000 7f33d1e00640 1 --2- 172.21.15.27:0/1172490419 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f33cc0cf110 0x7f33cc150c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:26.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.238+0000 7f33d2800640 1 --2- 172.21.15.27:0/1172490419 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33cc15f550 0x7f33cc1531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:26.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.238+0000 7f33d1400640 1 -- 172.21.15.27:0/1172490419 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f33cc0cf110 msgr2=0x7f33cc150c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:26.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.238+0000 7f33d1400640 1 --2- 172.21.15.27:0/1172490419 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f33cc0cf110 0x7f33cc150c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:26.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.238+0000 7f33d1400640 1 -- 172.21.15.27:0/1172490419 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33cc15f550 msgr2=0x7f33cc1531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:26.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.238+0000 7f33d1400640 1 --2- 172.21.15.27:0/1172490419 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33cc15f550 0x7f33cc1531c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:26.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.238+0000 7f33d1400640 1 -- 172.21.15.27:0/1172490419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c4009a70 con 0x7f33cc158900 2024-01-15T18:09:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.238+0000 7f33d1400640 1 --2- 172.21.15.27:0/1172490419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33cc158900 0x7f33cc152c80 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f33b401ccf0 tx=0x7f33b40191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.242+0000 7f33ca000640 1 -- 172.21.15.27:0/1172490419 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b40abcd0 con 0x7f33cc158900 2024-01-15T18:09:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.242+0000 7f33ca000640 1 -- 172.21.15.27:0/1172490419 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b40abe30 con 0x7f33cc158900 2024-01-15T18:09:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.242+0000 7f33ca000640 1 -- 172.21.15.27:0/1172490419 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b40be630 con 0x7f33cc158900 2024-01-15T18:09:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.242+0000 7f33d4800640 1 -- 172.21.15.27:0/1172490419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33b4009420 con 0x7f33cc158900 2024-01-15T18:09:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.242+0000 7f33d4800640 1 -- 172.21.15.27:0/1172490419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33cc153e30 con 0x7f33cc158900 2024-01-15T18:09:26.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.242+0000 7f33d4800640 1 -- 172.21.15.27:0/1172490419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33cc04e850 con 0x7f33cc158900 2024-01-15T18:09:26.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.242+0000 7f33ca000640 1 -- 172.21.15.27:0/1172490419 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 86670+0+0 (secure 0 0 0) 0x7f33b40b43f0 con 0x7f33cc158900 2024-01-15T18:09:26.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.242+0000 7f33ca000640 1 --2- 172.21.15.27:0/1172490419 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f33a4066e90 0x7f33a4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:26.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.242+0000 7f33ca000640 1 -- 172.21.15.27:0/1172490419 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f33b411efd0 con 0x7f33cc158900 2024-01-15T18:09:26.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.242+0000 7f33d1e00640 1 --2- 172.21.15.27:0/1172490419 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f33a4066e90 0x7f33a4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:26.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.246+0000 7f33d1e00640 1 --2- 172.21.15.27:0/1172490419 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f33a4066e90 0x7f33a4069350 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f33bc007fd0 tx=0x7f33bc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:26.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.246+0000 7f33ca000640 1 -- 172.21.15.27:0/1172490419 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f33b40eb950 con 0x7f33cc158900 2024-01-15T18:09:26.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.430+0000 7f33d4800640 1 -- 172.21.15.27:0/1172490419 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f33cc15bac0 con 0x7f33a4066e90 2024-01-15T18:09:26.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.434+0000 7f33ca000640 1 -- 172.21.15.27:0/1172490419 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f33cc15bac0 con 0x7f33a4066e90 2024-01-15T18:09:26.440 INFO:teuthology.orchestra.run.smithi027.stdout:/volumes/_nogroup/sv1/b5529c8c-34b5-49c9-ab51-a3ff8b9246fc 2024-01-15T18:09:26.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.438+0000 7f33d4800640 1 -- 172.21.15.27:0/1172490419 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f33a4066e90 msgr2=0x7f33a4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:26.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.438+0000 7f33d4800640 1 --2- 172.21.15.27:0/1172490419 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f33a4066e90 0x7f33a4069350 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f33bc007fd0 tx=0x7f33bc009040 comp rx=0 tx=0).stop 2024-01-15T18:09:26.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.438+0000 7f33d4800640 1 -- 172.21.15.27:0/1172490419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33cc158900 msgr2=0x7f33cc152c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:26.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.438+0000 7f33d4800640 1 --2- 172.21.15.27:0/1172490419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33cc158900 0x7f33cc152c80 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f33b401ccf0 tx=0x7f33b40191f0 comp rx=0 tx=0).stop 2024-01-15T18:09:26.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.438+0000 7f33d4800640 1 -- 172.21.15.27:0/1172490419 shutdown_connections 2024-01-15T18:09:26.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.438+0000 7f33d4800640 1 --2- 172.21.15.27:0/1172490419 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f33a4066e90 0x7f33a4069350 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:26.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.438+0000 7f33d4800640 1 --2- 172.21.15.27:0/1172490419 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33cc15f550 0x7f33cc1531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:26.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.438+0000 7f33d4800640 1 --2- 172.21.15.27:0/1172490419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33cc158900 0x7f33cc152c80 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:26.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.438+0000 7f33d4800640 1 --2- 172.21.15.27:0/1172490419 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f33cc0cf110 0x7f33cc150c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:26.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.438+0000 7f33d4800640 1 -- 172.21.15.27:0/1172490419 >> 172.21.15.27:0/1172490419 conn(0x7f33cc0cb700 msgr2=0x7f33cc0b1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:26.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.438+0000 7f33d4800640 1 -- 172.21.15.27:0/1172490419 shutdown_connections 2024-01-15T18:09:26.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:26.438+0000 7f33d4800640 1 -- 172.21.15.27:0/1172490419 wait complete. 2024-01-15T18:09:26.462 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/b5529c8c-34b5-49c9-ab51-a3ff8b9246fc 2024-01-15T18:09:26.463 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:09:26.463 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:09:26.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:26.472 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/b5529c8c-34b5-49c9-ab51-a3ff8b9246fc/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:09:26.706 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:09:26.710 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:09:33.587 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records in 2024-01-15T18:09:33.588 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records out 2024-01-15T18:09:33.588 INFO:teuthology.orchestra.run.smithi027.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.05303 s, 152 MB/s 2024-01-15T18:09:33.589 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:09:33.589 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:09:33.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:33.598 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/b5529c8c-34b5-49c9-ab51-a3ff8b9246fc/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:09:40.499 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records in 2024-01-15T18:09:40.500 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records out 2024-01-15T18:09:40.500 INFO:teuthology.orchestra.run.smithi027.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.84635 s, 157 MB/s 2024-01-15T18:09:40.501 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:09:40.501 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:09:40.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:40.510 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/b5529c8c-34b5-49c9-ab51-a3ff8b9246fc/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:09:47.379 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records in 2024-01-15T18:09:47.380 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records out 2024-01-15T18:09:47.380 INFO:teuthology.orchestra.run.smithi027.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.81315 s, 158 MB/s 2024-01-15T18:09:47.380 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 10 2024-01-15T18:09:47.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.517+0000 7f2217c00640 1 Processor -- start 2024-01-15T18:09:47.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.517+0000 7f2217c00640 1 -- start start 2024-01-15T18:09:47.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.517+0000 7f2217c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f221006c430 0x7f2210070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:47.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.517+0000 7f2217c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f22100b1db0 0x7f22100b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:47.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.517+0000 7f2217c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22100ccfb0 con 0x7f22100ce050 2024-01-15T18:09:47.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.517+0000 7f2217c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22100cd290 con 0x7f221006c430 2024-01-15T18:09:47.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.517+0000 7f2217c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22100cd570 con 0x7f22100b1db0 2024-01-15T18:09:47.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.521+0000 7f220fe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f22100b1db0 0x7f22100b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:47.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.521+0000 7f2215c00640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7f22100ce050 0x7f221006c810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:46664/0 (socket says 172.21.15.27:46664) 2024-01-15T18:09:47.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.521+0000 7f2215c00640 1 -- 172.21.15.27:0/2960360042 learned_addr learned my addr 172.21.15.27:0/2960360042 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:47.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.521+0000 7f2215200640 1 --2- 172.21.15.27:0/2960360042 >> v2:172.21.15.93:3300/0 conn(0x7f221006c430 0x7f2210070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:47.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.521+0000 7f220f400640 1 -- 172.21.15.27:0/2960360042 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131737838 0 0) 0x7f22100ccfb0 con 0x7f22100ce050 2024-01-15T18:09:47.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.521+0000 7f220f400640 1 -- 172.21.15.27:0/2960360042 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21f0003530 con 0x7f22100ce050 2024-01-15T18:09:47.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.521+0000 7f2215200640 1 -- 172.21.15.27:0/2960360042 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f22100b1db0 msgr2=0x7f22100b2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:47.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.521+0000 7f2215200640 1 --2- 172.21.15.27:0/2960360042 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f22100b1db0 0x7f22100b2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.521+0000 7f2215200640 1 -- 172.21.15.27:0/2960360042 >> v1:172.21.15.93:6789/0 conn(0x7f22100ce050 legacy=0x7f221006c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:47.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.521+0000 7f2215200640 1 -- 172.21.15.27:0/2960360042 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22100cd850 con 0x7f221006c430 2024-01-15T18:09:47.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.521+0000 7f220fe00640 1 --2- 172.21.15.27:0/2960360042 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f22100b1db0 0x7f22100b2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:09:47.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.521+0000 7f2215200640 1 --2- 172.21.15.27:0/2960360042 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f221006c430 0x7f2210070ba0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f22000097c0 tx=0x7f220001c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e140d8634b985f3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:47.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.521+0000 7f220f400640 1 -- 172.21.15.27:0/2960360042 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22000ab600 con 0x7f221006c430 2024-01-15T18:09:47.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.521+0000 7f220f400640 1 -- 172.21.15.27:0/2960360042 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22000a7050 con 0x7f221006c430 2024-01-15T18:09:47.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.521+0000 7f220f400640 1 -- 172.21.15.27:0/2960360042 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22000ab980 con 0x7f221006c430 2024-01-15T18:09:47.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.525+0000 7f2217c00640 1 -- 172.21.15.27:0/2960360042 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f221006c430 msgr2=0x7f2210070ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:47.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.525+0000 7f2217c00640 1 --2- 172.21.15.27:0/2960360042 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f221006c430 0x7f2210070ba0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f22000097c0 tx=0x7f220001c3f0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.525+0000 7f2217c00640 1 -- 172.21.15.27:0/2960360042 shutdown_connections 2024-01-15T18:09:47.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.525+0000 7f2217c00640 1 --2- 172.21.15.27:0/2960360042 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f22100b1db0 0x7f22100b2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.525+0000 7f2217c00640 1 --2- 172.21.15.27:0/2960360042 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f221006c430 0x7f2210070ba0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.525+0000 7f2217c00640 1 -- 172.21.15.27:0/2960360042 >> 172.21.15.27:0/2960360042 conn(0x7f22100cb700 msgr2=0x7f22100cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:47.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.525+0000 7f2217c00640 1 -- 172.21.15.27:0/2960360042 shutdown_connections 2024-01-15T18:09:47.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.525+0000 7f2217c00640 1 -- 172.21.15.27:0/2960360042 wait complete. 2024-01-15T18:09:47.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.525+0000 7f2217c00640 1 Processor -- start 2024-01-15T18:09:47.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.525+0000 7f2217c00640 1 -- start start 2024-01-15T18:09:47.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.529+0000 7f2217c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f221006c430 0x7f22101527f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:47.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.529+0000 7f2217c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22100b1db0 0x7f2210152d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:47.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.529+0000 7f2217c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f22100ce050 0x7f2210153270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:47.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.529+0000 7f2217c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2210159310 con 0x7f221006c430 2024-01-15T18:09:47.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.529+0000 7f2217c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2210158e80 con 0x7f22100b1db0 2024-01-15T18:09:47.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.529+0000 7f2217c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22101524b0 con 0x7f22100ce050 2024-01-15T18:09:47.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.529+0000 7f2215200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f221006c430 0x7f22101527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:47.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.529+0000 7f2215200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f221006c430 0x7f22101527f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35462/0 (socket says 172.21.15.27:35462) 2024-01-15T18:09:47.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.529+0000 7f2215200640 1 -- 172.21.15.27:0/175870605 learned_addr learned my addr 172.21.15.27:0/175870605 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:47.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.529+0000 7f2215c00640 1 --2- 172.21.15.27:0/175870605 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f22100ce050 0x7f2210153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:47.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.529+0000 7f220fe00640 1 --2- 172.21.15.27:0/175870605 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22100b1db0 0x7f2210152d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:47.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.529+0000 7f2215200640 1 -- 172.21.15.27:0/175870605 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f22100ce050 msgr2=0x7f2210153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:47.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.529+0000 7f2215200640 1 --2- 172.21.15.27:0/175870605 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f22100ce050 0x7f2210153270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.529+0000 7f2215200640 1 -- 172.21.15.27:0/175870605 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22100b1db0 msgr2=0x7f2210152d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:47.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.529+0000 7f2215200640 1 --2- 172.21.15.27:0/175870605 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22100b1db0 0x7f2210152d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.529+0000 7f2215200640 1 -- 172.21.15.27:0/175870605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f8009420 con 0x7f221006c430 2024-01-15T18:09:47.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.529+0000 7f220fe00640 1 --2- 172.21.15.27:0/175870605 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22100b1db0 0x7f2210152d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:09:47.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.529+0000 7f2215200640 1 --2- 172.21.15.27:0/175870605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f221006c430 0x7f22101527f0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f220001a020 tx=0x7f2200019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:47.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.529+0000 7f220cc00640 1 -- 172.21.15.27:0/175870605 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22000b4410 con 0x7f221006c430 2024-01-15T18:09:47.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.529+0000 7f220cc00640 1 -- 172.21.15.27:0/175870605 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22000b4590 con 0x7f221006c430 2024-01-15T18:09:47.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.529+0000 7f220cc00640 1 -- 172.21.15.27:0/175870605 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22000b4940 con 0x7f221006c430 2024-01-15T18:09:47.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.529+0000 7f2217c00640 1 -- 172.21.15.27:0/175870605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2200009420 con 0x7f221006c430 2024-01-15T18:09:47.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.529+0000 7f2217c00640 1 -- 172.21.15.27:0/175870605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2210158830 con 0x7f221006c430 2024-01-15T18:09:47.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.529+0000 7f2217c00640 1 -- 172.21.15.27:0/175870605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2210152640 con 0x7f221006c430 2024-01-15T18:09:47.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.529+0000 7f220cc00640 1 -- 172.21.15.27:0/175870605 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 15) v1 ==== 86721+0+0 (secure 0 0 0) 0x7f22000b2040 con 0x7f221006c430 2024-01-15T18:09:47.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.533+0000 7f220cc00640 1 --2- 172.21.15.27:0/175870605 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f21e8066f20 0x7f21e80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:47.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.533+0000 7f220cc00640 1 -- 172.21.15.27:0/175870605 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f220011ef50 con 0x7f221006c430 2024-01-15T18:09:47.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.533+0000 7f220fe00640 1 --2- 172.21.15.27:0/175870605 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f21e8066f20 0x7f21e80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:47.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.533+0000 7f220fe00640 1 --2- 172.21.15.27:0/175870605 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f21e8066f20 0x7f21e80693e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f2210071c50 tx=0x7f21f8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:47.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.537+0000 7f220cc00640 1 -- 172.21.15.27:0/175870605 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f22000eb820 con 0x7f221006c430 2024-01-15T18:09:47.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.697+0000 7f2217c00640 1 -- 172.21.15.27:0/175870605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7f221004e850 con 0x7f221006c430 2024-01-15T18:09:47.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.701+0000 7f220cc00640 1 -- 172.21.15.27:0/175870605 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v21)=0 v21) v1 ==== 131+0+0 (secure 0 0 0) 0x7f22000c20c0 con 0x7f221006c430 2024-01-15T18:09:47.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.705+0000 7f2217c00640 1 -- 172.21.15.27:0/175870605 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f21e8066f20 msgr2=0x7f21e80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:47.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.705+0000 7f2217c00640 1 --2- 172.21.15.27:0/175870605 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f21e8066f20 0x7f21e80693e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f2210071c50 tx=0x7f21f8008040 comp rx=0 tx=0).stop 2024-01-15T18:09:47.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.705+0000 7f2217c00640 1 -- 172.21.15.27:0/175870605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f221006c430 msgr2=0x7f22101527f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:47.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.705+0000 7f2217c00640 1 --2- 172.21.15.27:0/175870605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f221006c430 0x7f22101527f0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f220001a020 tx=0x7f2200019190 comp rx=0 tx=0).stop 2024-01-15T18:09:47.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.705+0000 7f2217c00640 1 -- 172.21.15.27:0/175870605 shutdown_connections 2024-01-15T18:09:47.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.705+0000 7f2217c00640 1 --2- 172.21.15.27:0/175870605 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f21e8066f20 0x7f21e80693e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.705+0000 7f2217c00640 1 --2- 172.21.15.27:0/175870605 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f22100ce050 0x7f2210153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.705+0000 7f2217c00640 1 --2- 172.21.15.27:0/175870605 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f22100b1db0 0x7f2210152d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.705+0000 7f2217c00640 1 --2- 172.21.15.27:0/175870605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f221006c430 0x7f22101527f0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.705+0000 7f2217c00640 1 -- 172.21.15.27:0/175870605 >> 172.21.15.27:0/175870605 conn(0x7f22100cb700 msgr2=0x7f22100adbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:47.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.705+0000 7f2217c00640 1 -- 172.21.15.27:0/175870605 shutdown_connections 2024-01-15T18:09:47.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.705+0000 7f2217c00640 1 -- 172.21.15.27:0/175870605 wait complete. 2024-01-15T18:09:47.730 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 2 2024-01-15T18:09:47.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f3154600640 1 Processor -- start 2024-01-15T18:09:47.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f3154600640 1 -- start start 2024-01-15T18:09:47.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f3154600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f314c15f550 0x7f314c0cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:47.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f3154600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f314c0b1cb0 con 0x7f314c158900 2024-01-15T18:09:47.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f3154600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f314c15fa00 con 0x7f314c0cf110 2024-01-15T18:09:47.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f3154600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f314c0cf540 con 0x7f314c15f550 2024-01-15T18:09:47.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f3151c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f314c158900 0x7f314c15bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34838/0 (socket says 172.21.15.27:34838) 2024-01-15T18:09:47.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f3151c00640 1 -- 172.21.15.27:0/935599166 learned_addr learned my addr 172.21.15.27:0/935599166 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:47.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f3151200640 1 --2- 172.21.15.27:0/935599166 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f314c15f550 0x7f314c0cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:47.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f314be00640 1 -- 172.21.15.27:0/935599166 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3409892245 0 0) 0x7f314c0b1cb0 con 0x7f314c158900 2024-01-15T18:09:47.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f314be00640 1 -- 172.21.15.27:0/935599166 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f312c003530 con 0x7f314c158900 2024-01-15T18:09:47.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f314be00640 1 -- 172.21.15.27:0/935599166 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 507692389 0 0) 0x7f312c003530 con 0x7f314c158900 2024-01-15T18:09:47.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f314be00640 1 -- 172.21.15.27:0/935599166 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f314c15f550 msgr2=0x7f314c0cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:47.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f314be00640 1 --2- 172.21.15.27:0/935599166 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f314c15f550 0x7f314c0cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f314be00640 1 -- 172.21.15.27:0/935599166 >> v1:172.21.15.93:6789/0 conn(0x7f314c0cf110 legacy=0x7f314c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:47.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f314be00640 1 -- 172.21.15.27:0/935599166 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f314c0b1f00 con 0x7f314c158900 2024-01-15T18:09:47.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f314be00640 1 -- 172.21.15.27:0/935599166 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f313c001680 con 0x7f314c158900 2024-01-15T18:09:47.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f314be00640 1 -- 172.21.15.27:0/935599166 >> v1:172.21.15.27:6789/0 conn(0x7f314c158900 legacy=0x7f314c15bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:47.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f314be00640 1 --2- 172.21.15.27:0/935599166 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f314c15f550 0x7f312c003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f314be00640 1 --2- 172.21.15.27:0/935599166 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f312c003fa0 0x7f312c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f314be00640 1 --2- 172.21.15.27:0/935599166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f312c004d10 0x7f312c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f314be00640 1 -- 172.21.15.27:0/935599166 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314c0cf540 con 0x7f312c004d10 2024-01-15T18:09:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f314be00640 1 -- 172.21.15.27:0/935599166 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312c00baf0 con 0x7f312c003fa0 2024-01-15T18:09:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f314be00640 1 -- 172.21.15.27:0/935599166 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f312c00bcb0 con 0x7f314c15f550 2024-01-15T18:09:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f3151200640 1 --2- 172.21.15.27:0/935599166 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f312c003fa0 0x7f312c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f3152600640 1 --2- 172.21.15.27:0/935599166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f312c004d10 0x7f312c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f3151c00640 1 --2- 172.21.15.27:0/935599166 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f314c15f550 0x7f312c003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f3151200640 1 -- 172.21.15.27:0/935599166 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f314c15f550 msgr2=0x7f312c003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f3151200640 1 --2- 172.21.15.27:0/935599166 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f314c15f550 0x7f312c003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f3151200640 1 -- 172.21.15.27:0/935599166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f312c004d10 msgr2=0x7f312c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f3151200640 1 --2- 172.21.15.27:0/935599166 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f312c004d10 0x7f312c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.869+0000 7f3151200640 1 -- 172.21.15.27:0/935599166 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312c003530 con 0x7f312c003fa0 2024-01-15T18:09:47.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.873+0000 7f3151200640 1 --2- 172.21.15.27:0/935599166 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f312c003fa0 0x7f312c0047d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3134009ae0 tx=0x7f313401c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc1dab138f213c49 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:47.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.873+0000 7f314be00640 1 -- 172.21.15.27:0/935599166 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31340a5580 con 0x7f312c003fa0 2024-01-15T18:09:47.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.873+0000 7f314be00640 1 -- 172.21.15.27:0/935599166 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31340a56e0 con 0x7f312c003fa0 2024-01-15T18:09:47.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.873+0000 7f314be00640 1 -- 172.21.15.27:0/935599166 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31340a5a90 con 0x7f312c003fa0 2024-01-15T18:09:47.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.873+0000 7f3154600640 1 -- 172.21.15.27:0/935599166 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f312c003fa0 msgr2=0x7f312c0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:47.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.873+0000 7f3154600640 1 --2- 172.21.15.27:0/935599166 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f312c003fa0 0x7f312c0047d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3134009ae0 tx=0x7f313401c990 comp rx=0 tx=0).stop 2024-01-15T18:09:47.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.873+0000 7f3152600640 1 -- 172.21.15.27:0/935599166 reap_dead start 2024-01-15T18:09:47.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.873+0000 7f3154600640 1 -- 172.21.15.27:0/935599166 shutdown_connections 2024-01-15T18:09:47.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.873+0000 7f3154600640 1 -- 172.21.15.27:0/935599166 >> 172.21.15.27:0/935599166 conn(0x7f314c0cb700 msgr2=0x7f314c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:47.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.877+0000 7f3154600640 1 -- 172.21.15.27:0/935599166 shutdown_connections 2024-01-15T18:09:47.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.877+0000 7f3154600640 1 -- 172.21.15.27:0/935599166 wait complete. 2024-01-15T18:09:47.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.877+0000 7f3154600640 1 Processor -- start 2024-01-15T18:09:47.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.877+0000 7f3154600640 1 -- start start 2024-01-15T18:09:47.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.877+0000 7f3154600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f312c003fa0 0x7f314c146580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:47.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.877+0000 7f3154600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f314c15d250 0x7f314c15d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:47.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.877+0000 7f3154600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f314c154ba0 0x7f314c155020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:47.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.877+0000 7f3154600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314c15dc10 con 0x7f314c15d250 2024-01-15T18:09:47.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.877+0000 7f3154600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314c155820 con 0x7f312c003fa0 2024-01-15T18:09:47.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.877+0000 7f3154600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f314c155560 con 0x7f314c154ba0 2024-01-15T18:09:47.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.877+0000 7f3151200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f314c15d250 0x7f314c15d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:47.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.877+0000 7f3151200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f314c15d250 0x7f314c15d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35474/0 (socket says 172.21.15.27:35474) 2024-01-15T18:09:47.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.877+0000 7f3151200640 1 -- 172.21.15.27:0/3778763201 learned_addr learned my addr 172.21.15.27:0/3778763201 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:47.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.877+0000 7f3151c00640 1 --2- 172.21.15.27:0/3778763201 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f312c003fa0 0x7f314c146580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:47.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.877+0000 7f3152600640 1 --2- 172.21.15.27:0/3778763201 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f314c154ba0 0x7f314c155020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:47.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.877+0000 7f3151200640 1 -- 172.21.15.27:0/3778763201 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f314c154ba0 msgr2=0x7f314c155020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:47.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.877+0000 7f3151200640 1 --2- 172.21.15.27:0/3778763201 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f314c154ba0 0x7f314c155020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.877+0000 7f3151200640 1 -- 172.21.15.27:0/3778763201 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f312c003fa0 msgr2=0x7f314c146580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:47.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.877+0000 7f3151200640 1 --2- 172.21.15.27:0/3778763201 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f312c003fa0 0x7f314c146580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.877+0000 7f3151200640 1 -- 172.21.15.27:0/3778763201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f314c0afa40 con 0x7f314c15d250 2024-01-15T18:09:47.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.877+0000 7f3151200640 1 --2- 172.21.15.27:0/3778763201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f314c15d250 0x7f314c15d6d0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f3134009ae0 tx=0x7f312c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:47.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.881+0000 7f3149600640 1 -- 172.21.15.27:0/3778763201 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3134004230 con 0x7f314c15d250 2024-01-15T18:09:47.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.881+0000 7f3149600640 1 -- 172.21.15.27:0/3778763201 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3134004390 con 0x7f314c15d250 2024-01-15T18:09:47.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.881+0000 7f3149600640 1 -- 172.21.15.27:0/3778763201 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3134003820 con 0x7f314c15d250 2024-01-15T18:09:47.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.881+0000 7f3154600640 1 -- 172.21.15.27:0/3778763201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f314c0afd20 con 0x7f314c15d250 2024-01-15T18:09:47.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.881+0000 7f3154600640 1 -- 172.21.15.27:0/3778763201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f314c0b02e0 con 0x7f314c15d250 2024-01-15T18:09:47.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.881+0000 7f3154600640 1 -- 172.21.15.27:0/3778763201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f314c04e850 con 0x7f314c15d250 2024-01-15T18:09:47.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.881+0000 7f3149600640 1 -- 172.21.15.27:0/3778763201 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 15) v1 ==== 86721+0+0 (secure 0 0 0) 0x7f31340b4040 con 0x7f314c15d250 2024-01-15T18:09:47.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.881+0000 7f3149600640 1 --2- 172.21.15.27:0/3778763201 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3124066f20 0x7f31240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:47.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.881+0000 7f3149600640 1 -- 172.21.15.27:0/3778763201 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f313411f5f0 con 0x7f314c15d250 2024-01-15T18:09:47.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.881+0000 7f3151c00640 1 --2- 172.21.15.27:0/3778763201 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3124066f20 0x7f31240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:47.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.885+0000 7f3151c00640 1 --2- 172.21.15.27:0/3778763201 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3124066f20 0x7f31240693e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f313c000c10 tx=0x7f313c001180 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:47.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:47.885+0000 7f3149600640 1 -- 172.21.15.27:0/3778763201 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f31340ebfd0 con 0x7f314c15d250 2024-01-15T18:09:48.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.045+0000 7f3154600640 1 -- 172.21.15.27:0/3778763201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f314c06d540 con 0x7f314c15d250 2024-01-15T18:09:48.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.049+0000 7f3149600640 1 -- 172.21.15.27:0/3778763201 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v22)=0 v22) v1 ==== 135+0+0 (secure 0 0 0) 0x7f31340ef5f0 con 0x7f314c15d250 2024-01-15T18:09:48.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.053+0000 7f3154600640 1 -- 172.21.15.27:0/3778763201 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3124066f20 msgr2=0x7f31240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:48.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.053+0000 7f3154600640 1 --2- 172.21.15.27:0/3778763201 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3124066f20 0x7f31240693e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f313c000c10 tx=0x7f313c001180 comp rx=0 tx=0).stop 2024-01-15T18:09:48.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.053+0000 7f3154600640 1 -- 172.21.15.27:0/3778763201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f314c15d250 msgr2=0x7f314c15d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:48.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.053+0000 7f3154600640 1 --2- 172.21.15.27:0/3778763201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f314c15d250 0x7f314c15d6d0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f3134009ae0 tx=0x7f312c005800 comp rx=0 tx=0).stop 2024-01-15T18:09:48.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.053+0000 7f3154600640 1 -- 172.21.15.27:0/3778763201 shutdown_connections 2024-01-15T18:09:48.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.053+0000 7f3154600640 1 --2- 172.21.15.27:0/3778763201 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3124066f20 0x7f31240693e0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.053+0000 7f3154600640 1 --2- 172.21.15.27:0/3778763201 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f314c154ba0 0x7f314c155020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.053+0000 7f3154600640 1 --2- 172.21.15.27:0/3778763201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f314c15d250 0x7f314c15d6d0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.053+0000 7f3154600640 1 --2- 172.21.15.27:0/3778763201 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f312c003fa0 0x7f314c146580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.053+0000 7f3154600640 1 -- 172.21.15.27:0/3778763201 >> 172.21.15.27:0/3778763201 conn(0x7f314c0cb700 msgr2=0x7f314c0ccaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:48.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.053+0000 7f3154600640 1 -- 172.21.15.27:0/3778763201 shutdown_connections 2024-01-15T18:09:48.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.053+0000 7f3154600640 1 -- 172.21.15.27:0/3778763201 wait complete. 2024-01-15T18:09:48.079 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-15T18:09:48.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.221+0000 7f1e91a00640 1 Processor -- start 2024-01-15T18:09:48.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.221+0000 7f1e91a00640 1 -- start start 2024-01-15T18:09:48.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.221+0000 7f1e91a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e8c158900 0x7f1e8c15f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:48.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.221+0000 7f1e91a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1e8c15bf00 0x7f1e8c0cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:48.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.221+0000 7f1e91a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e8c0b1cb0 con 0x7f1e8c0cf110 2024-01-15T18:09:48.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.221+0000 7f1e91a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e8c0cf510 con 0x7f1e8c15bf00 2024-01-15T18:09:48.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.221+0000 7f1e91a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c0b1f00 con 0x7f1e8c158900 2024-01-15T18:09:48.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.221+0000 7f1e8b400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1e8c0cf110 0x7f1e8c0701e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34842/0 (socket says 172.21.15.27:34842) 2024-01-15T18:09:48.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.221+0000 7f1e8a000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1e8c15bf00 0x7f1e8c0cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:48.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.221+0000 7f1e8b400640 1 -- 172.21.15.27:0/1751364576 learned_addr learned my addr 172.21.15.27:0/1751364576 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:48.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.221+0000 7f1e8aa00640 1 --2- 172.21.15.27:0/1751364576 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e8c158900 0x7f1e8c15f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:48.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.221+0000 7f1e8a000640 1 -- 172.21.15.27:0/1751364576 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e8c158900 msgr2=0x7f1e8c15f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:48.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.221+0000 7f1e8a000640 1 --2- 172.21.15.27:0/1751364576 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e8c158900 0x7f1e8c15f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.221+0000 7f1e8a000640 1 -- 172.21.15.27:0/1751364576 >> v1:172.21.15.27:6789/0 conn(0x7f1e8c0cf110 legacy=0x7f1e8c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:48.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.221+0000 7f1e8a000640 1 -- 172.21.15.27:0/1751364576 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e8c0b21e0 con 0x7f1e8c15bf00 2024-01-15T18:09:48.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.221+0000 7f1e8aa00640 1 --2- 172.21.15.27:0/1751364576 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e8c158900 0x7f1e8c15f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:09:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.225+0000 7f1e8a000640 1 --2- 172.21.15.27:0/1751364576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e8c15bf00 0x7f1e8c0cebd0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f1e740097c0 tx=0x7f1e7401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea07fa0e3b47343d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.225+0000 7f1e89600640 1 -- 172.21.15.27:0/1751364576 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e740ab600 con 0x7f1e8c15bf00 2024-01-15T18:09:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.225+0000 7f1e89600640 1 -- 172.21.15.27:0/1751364576 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e740a7050 con 0x7f1e8c15bf00 2024-01-15T18:09:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.225+0000 7f1e89600640 1 -- 172.21.15.27:0/1751364576 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e740ab980 con 0x7f1e8c15bf00 2024-01-15T18:09:48.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.225+0000 7f1e91a00640 1 -- 172.21.15.27:0/1751364576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e8c15bf00 msgr2=0x7f1e8c0cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:48.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.225+0000 7f1e91a00640 1 --2- 172.21.15.27:0/1751364576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e8c15bf00 0x7f1e8c0cebd0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f1e740097c0 tx=0x7f1e7401c480 comp rx=0 tx=0).stop 2024-01-15T18:09:48.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.225+0000 7f1e91a00640 1 -- 172.21.15.27:0/1751364576 shutdown_connections 2024-01-15T18:09:48.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.225+0000 7f1e91a00640 1 --2- 172.21.15.27:0/1751364576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e8c15bf00 0x7f1e8c0cebd0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.225+0000 7f1e91a00640 1 --2- 172.21.15.27:0/1751364576 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e8c158900 0x7f1e8c15f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.225+0000 7f1e91a00640 1 -- 172.21.15.27:0/1751364576 >> 172.21.15.27:0/1751364576 conn(0x7f1e8c0cb700 msgr2=0x7f1e8c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:48.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.225+0000 7f1e91a00640 1 -- 172.21.15.27:0/1751364576 shutdown_connections 2024-01-15T18:09:48.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.225+0000 7f1e91a00640 1 -- 172.21.15.27:0/1751364576 wait complete. 2024-01-15T18:09:48.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.229+0000 7f1e91a00640 1 Processor -- start 2024-01-15T18:09:48.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.229+0000 7f1e91a00640 1 -- start start 2024-01-15T18:09:48.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.229+0000 7f1e91a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1e8c0cf110 0x7f1e8c147290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:48.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.229+0000 7f1e91a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e8c158900 0x7f1e8c1507e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:48.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.229+0000 7f1e91a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e8c15bf00 0x7f1e8c0afba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:48.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.229+0000 7f1e91a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c15ad50 con 0x7f1e8c15bf00 2024-01-15T18:09:48.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.229+0000 7f1e91a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c156100 con 0x7f1e8c0cf110 2024-01-15T18:09:48.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.229+0000 7f1e91a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8c14f420 con 0x7f1e8c158900 2024-01-15T18:09:48.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.229+0000 7f1e8b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e8c15bf00 0x7f1e8c0afba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:48.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.229+0000 7f1e8b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e8c15bf00 0x7f1e8c0afba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35488/0 (socket says 172.21.15.27:35488) 2024-01-15T18:09:48.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.229+0000 7f1e8b400640 1 -- 172.21.15.27:0/420511293 learned_addr learned my addr 172.21.15.27:0/420511293 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:48.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.229+0000 7f1e8a000640 1 --2- 172.21.15.27:0/420511293 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e8c158900 0x7f1e8c1507e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:48.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.229+0000 7f1e8aa00640 1 --2- 172.21.15.27:0/420511293 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1e8c0cf110 0x7f1e8c147290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:48.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.229+0000 7f1e8b400640 1 -- 172.21.15.27:0/420511293 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e8c158900 msgr2=0x7f1e8c1507e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:48.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.229+0000 7f1e8b400640 1 --2- 172.21.15.27:0/420511293 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e8c158900 0x7f1e8c1507e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.229+0000 7f1e8b400640 1 -- 172.21.15.27:0/420511293 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1e8c0cf110 msgr2=0x7f1e8c147290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:48.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.229+0000 7f1e8b400640 1 --2- 172.21.15.27:0/420511293 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1e8c0cf110 0x7f1e8c147290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.229+0000 7f1e8b400640 1 -- 172.21.15.27:0/420511293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e74009420 con 0x7f1e8c15bf00 2024-01-15T18:09:48.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.229+0000 7f1e8b400640 1 --2- 172.21.15.27:0/420511293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e8c15bf00 0x7f1e8c0afba0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f1e8400cbc0 tx=0x7f1e8400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:48.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.229+0000 7f1e82a00640 1 -- 172.21.15.27:0/420511293 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8400ad40 con 0x7f1e8c15bf00 2024-01-15T18:09:48.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.229+0000 7f1e82a00640 1 -- 172.21.15.27:0/420511293 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e84012400 con 0x7f1e8c15bf00 2024-01-15T18:09:48.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.229+0000 7f1e91a00640 1 -- 172.21.15.27:0/420511293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e8c1557f0 con 0x7f1e8c15bf00 2024-01-15T18:09:48.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.229+0000 7f1e82a00640 1 -- 172.21.15.27:0/420511293 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8400aea0 con 0x7f1e8c15bf00 2024-01-15T18:09:48.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.229+0000 7f1e91a00640 1 -- 172.21.15.27:0/420511293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e8c14f140 con 0x7f1e8c15bf00 2024-01-15T18:09:48.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.233+0000 7f1e91a00640 1 -- 172.21.15.27:0/420511293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e8c067840 con 0x7f1e8c15bf00 2024-01-15T18:09:48.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.233+0000 7f1e82a00640 1 -- 172.21.15.27:0/420511293 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 15) v1 ==== 86721+0+0 (secure 0 0 0) 0x7f1e84007070 con 0x7f1e8c15bf00 2024-01-15T18:09:48.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.233+0000 7f1e82a00640 1 --2- 172.21.15.27:0/420511293 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1e64066f20 0x7f1e640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:48.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.233+0000 7f1e82a00640 1 -- 172.21.15.27:0/420511293 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f1e84084ea0 con 0x7f1e8c15bf00 2024-01-15T18:09:48.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.233+0000 7f1e8aa00640 1 --2- 172.21.15.27:0/420511293 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1e64066f20 0x7f1e640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:48.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.233+0000 7f1e8aa00640 1 --2- 172.21.15.27:0/420511293 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1e64066f20 0x7f1e640693e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c0afb30 tx=0x7f1e7c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:48.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.237+0000 7f1e82a00640 1 -- 172.21.15.27:0/420511293 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1e84088030 con 0x7f1e8c15bf00 2024-01-15T18:09:48.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.421+0000 7f1e91a00640 1 -- 172.21.15.27:0/420511293 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e8c06d320 con 0x7f1e64066f20 2024-01-15T18:09:48.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.437+0000 7f1e82a00640 1 -- 172.21.15.27:0/420511293 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1e8c06d320 con 0x7f1e64066f20 2024-01-15T18:09:48.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.437+0000 7f1e91a00640 1 -- 172.21.15.27:0/420511293 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1e64066f20 msgr2=0x7f1e640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:48.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.437+0000 7f1e91a00640 1 --2- 172.21.15.27:0/420511293 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1e64066f20 0x7f1e640693e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c0afb30 tx=0x7f1e7c008040 comp rx=0 tx=0).stop 2024-01-15T18:09:48.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.437+0000 7f1e91a00640 1 -- 172.21.15.27:0/420511293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e8c15bf00 msgr2=0x7f1e8c0afba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:48.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.437+0000 7f1e91a00640 1 --2- 172.21.15.27:0/420511293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e8c15bf00 0x7f1e8c0afba0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f1e8400cbc0 tx=0x7f1e8400a430 comp rx=0 tx=0).stop 2024-01-15T18:09:48.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.437+0000 7f1e91a00640 1 -- 172.21.15.27:0/420511293 shutdown_connections 2024-01-15T18:09:48.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.437+0000 7f1e91a00640 1 --2- 172.21.15.27:0/420511293 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1e64066f20 0x7f1e640693e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.437+0000 7f1e91a00640 1 --2- 172.21.15.27:0/420511293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e8c15bf00 0x7f1e8c0afba0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.437+0000 7f1e91a00640 1 --2- 172.21.15.27:0/420511293 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1e8c158900 0x7f1e8c1507e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.437+0000 7f1e91a00640 1 --2- 172.21.15.27:0/420511293 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1e8c0cf110 0x7f1e8c147290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.437+0000 7f1e91a00640 1 -- 172.21.15.27:0/420511293 >> 172.21.15.27:0/420511293 conn(0x7f1e8c0cb700 msgr2=0x7f1e8c073450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:48.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.437+0000 7f1e91a00640 1 -- 172.21.15.27:0/420511293 shutdown_connections 2024-01-15T18:09:48.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.437+0000 7f1e91a00640 1 -- 172.21.15.27:0/420511293 wait complete. 2024-01-15T18:09:48.463 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:09:48.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.601+0000 7f8572600640 1 Processor -- start 2024-01-15T18:09:48.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.601+0000 7f8572600640 1 -- start start 2024-01-15T18:09:48.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.601+0000 7f8572600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f856c15d190 0x7f856c0cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:48.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.601+0000 7f8572600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f856c06c520 0x7f856c0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:48.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.601+0000 7f8572600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f856c0b0d70 con 0x7f856c15d570 2024-01-15T18:09:48.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.601+0000 7f8572600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f856c0d0540 con 0x7f856c15d190 2024-01-15T18:09:48.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.601+0000 7f8572600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f856c0b0440 con 0x7f856c06c520 2024-01-15T18:09:48.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.601+0000 7f856aa00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f856c15d570 0x7f856c06be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34846/0 (socket says 172.21.15.27:34846) 2024-01-15T18:09:48.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.601+0000 7f856b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f856c15d190 0x7f856c0cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:48.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.601+0000 7f856aa00640 1 -- 172.21.15.27:0/92940090 learned_addr learned my addr 172.21.15.27:0/92940090 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:48.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.601+0000 7f856be00640 1 --2- 172.21.15.27:0/92940090 >> v2:172.21.15.93:3300/0 conn(0x7f856c06c520 0x7f856c0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:48.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.601+0000 7f856b400640 1 -- 172.21.15.27:0/92940090 >> v2:172.21.15.93:3300/0 conn(0x7f856c06c520 msgr2=0x7f856c0cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:48.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.601+0000 7f856b400640 1 --2- 172.21.15.27:0/92940090 >> v2:172.21.15.93:3300/0 conn(0x7f856c06c520 0x7f856c0cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.601+0000 7f856b400640 1 -- 172.21.15.27:0/92940090 >> v1:172.21.15.27:6789/0 conn(0x7f856c15d570 legacy=0x7f856c06be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:48.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.601+0000 7f856b400640 1 -- 172.21.15.27:0/92940090 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856c0b0720 con 0x7f856c15d190 2024-01-15T18:09:48.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.601+0000 7f856a000640 1 -- 172.21.15.27:0/92940090 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819882810 0 0) 0x7f856c0b0d70 con 0x7f856c15d570 2024-01-15T18:09:48.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.605+0000 7f856b400640 1 --2- 172.21.15.27:0/92940090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c15d190 0x7f856c0cfcf0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f856c0717e0 tx=0x7f855c01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a49c7dafa9984fb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:48.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.605+0000 7f856a000640 1 -- 172.21.15.27:0/92940090 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855c0ab5f0 con 0x7f856c15d190 2024-01-15T18:09:48.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.605+0000 7f856a000640 1 -- 172.21.15.27:0/92940090 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f855c0ab750 con 0x7f856c15d190 2024-01-15T18:09:48.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.605+0000 7f856a000640 1 -- 172.21.15.27:0/92940090 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855c0abae0 con 0x7f856c15d190 2024-01-15T18:09:48.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.605+0000 7f8572600640 1 -- 172.21.15.27:0/92940090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c15d190 msgr2=0x7f856c0cfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:48.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.605+0000 7f8572600640 1 --2- 172.21.15.27:0/92940090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c15d190 0x7f856c0cfcf0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f856c0717e0 tx=0x7f855c01c430 comp rx=0 tx=0).stop 2024-01-15T18:09:48.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.605+0000 7f8572600640 1 -- 172.21.15.27:0/92940090 shutdown_connections 2024-01-15T18:09:48.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.605+0000 7f8572600640 1 --2- 172.21.15.27:0/92940090 >> v2:172.21.15.93:3300/0 conn(0x7f856c06c520 0x7f856c0cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.605+0000 7f8572600640 1 --2- 172.21.15.27:0/92940090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c15d190 0x7f856c0cfcf0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.605+0000 7f8572600640 1 -- 172.21.15.27:0/92940090 >> 172.21.15.27:0/92940090 conn(0x7f856c0d0e70 msgr2=0x7f856c0d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:48.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.605+0000 7f8572600640 1 -- 172.21.15.27:0/92940090 shutdown_connections 2024-01-15T18:09:48.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.605+0000 7f8572600640 1 -- 172.21.15.27:0/92940090 wait complete. 2024-01-15T18:09:48.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.609+0000 7f8572600640 1 Processor -- start 2024-01-15T18:09:48.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.609+0000 7f8572600640 1 -- start start 2024-01-15T18:09:48.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.609+0000 7f8572600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f856c06c520 0x7f856c158850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:48.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.609+0000 7f8572600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c15d190 0x7f856c0b4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:48.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.609+0000 7f8572600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f856c15d570 0x7f856c14fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:48.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.609+0000 7f8572600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c149a00 con 0x7f856c15d190 2024-01-15T18:09:48.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.609+0000 7f8572600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0b50f0 con 0x7f856c06c520 2024-01-15T18:09:48.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.609+0000 7f8572600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856c14d7e0 con 0x7f856c15d570 2024-01-15T18:09:48.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.609+0000 7f856aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c15d190 0x7f856c0b4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:48.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.609+0000 7f856aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c15d190 0x7f856c0b4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35500/0 (socket says 172.21.15.27:35500) 2024-01-15T18:09:48.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.609+0000 7f856aa00640 1 -- 172.21.15.27:0/2299548849 learned_addr learned my addr 172.21.15.27:0/2299548849 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:48.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.609+0000 7f856be00640 1 --2- 172.21.15.27:0/2299548849 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f856c15d570 0x7f856c14fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:48.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.609+0000 7f856b400640 1 --2- 172.21.15.27:0/2299548849 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f856c06c520 0x7f856c158850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:48.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.609+0000 7f856aa00640 1 -- 172.21.15.27:0/2299548849 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f856c15d570 msgr2=0x7f856c14fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:48.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.609+0000 7f856aa00640 1 --2- 172.21.15.27:0/2299548849 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f856c15d570 0x7f856c14fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.609+0000 7f856aa00640 1 -- 172.21.15.27:0/2299548849 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f856c06c520 msgr2=0x7f856c158850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:48.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.609+0000 7f856aa00640 1 --2- 172.21.15.27:0/2299548849 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f856c06c520 0x7f856c158850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.609+0000 7f856aa00640 1 -- 172.21.15.27:0/2299548849 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f855c009420 con 0x7f856c15d190 2024-01-15T18:09:48.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.609+0000 7f856aa00640 1 --2- 172.21.15.27:0/2299548849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c15d190 0x7f856c0b4bb0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f855400c480 tx=0x7f855400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:48.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.609+0000 7f8563400640 1 -- 172.21.15.27:0/2299548849 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85540074e0 con 0x7f856c15d190 2024-01-15T18:09:48.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.609+0000 7f8563400640 1 -- 172.21.15.27:0/2299548849 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8554007660 con 0x7f856c15d190 2024-01-15T18:09:48.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.609+0000 7f8563400640 1 -- 172.21.15.27:0/2299548849 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8554004a30 con 0x7f856c15d190 2024-01-15T18:09:48.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.609+0000 7f8572600640 1 -- 172.21.15.27:0/2299548849 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f856c14dac0 con 0x7f856c15d190 2024-01-15T18:09:48.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.609+0000 7f8572600640 1 -- 172.21.15.27:0/2299548849 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f856c14e080 con 0x7f856c15d190 2024-01-15T18:09:48.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.613+0000 7f8572600640 1 -- 172.21.15.27:0/2299548849 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f856c073840 con 0x7f856c15d190 2024-01-15T18:09:48.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.613+0000 7f8563400640 1 -- 172.21.15.27:0/2299548849 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 15) v1 ==== 86721+0+0 (secure 0 0 0) 0x7f8554004b90 con 0x7f856c15d190 2024-01-15T18:09:48.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.613+0000 7f8563400640 1 --2- 172.21.15.27:0/2299548849 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8544066f20 0x7f85440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:48.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.613+0000 7f8563400640 1 -- 172.21.15.27:0/2299548849 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f85540840a0 con 0x7f856c15d190 2024-01-15T18:09:48.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.613+0000 7f856b400640 1 --2- 172.21.15.27:0/2299548849 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8544066f20 0x7f85440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:48.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.613+0000 7f856b400640 1 --2- 172.21.15.27:0/2299548849 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8544066f20 0x7f85440693e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f855c0af000 tx=0x7f855c0a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:48.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.617+0000 7f8563400640 1 -- 172.21.15.27:0/2299548849 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8554050ac0 con 0x7f856c15d190 2024-01-15T18:09:48.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.797+0000 7f8572600640 1 -- 172.21.15.27:0/2299548849 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f856c04e850 con 0x7f8544066f20 2024-01-15T18:09:48.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.801+0000 7f8563400640 1 -- 172.21.15.27:0/2299548849 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f856c04e850 con 0x7f8544066f20 2024-01-15T18:09:48.808 INFO:teuthology.orchestra.run.smithi027.stdout:/volumes/_nogroup/sv1/b5529c8c-34b5-49c9-ab51-a3ff8b9246fc 2024-01-15T18:09:48.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.805+0000 7f8572600640 1 -- 172.21.15.27:0/2299548849 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8544066f20 msgr2=0x7f85440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:48.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.805+0000 7f8572600640 1 --2- 172.21.15.27:0/2299548849 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8544066f20 0x7f85440693e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f855c0af000 tx=0x7f855c0a7000 comp rx=0 tx=0).stop 2024-01-15T18:09:48.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.805+0000 7f8572600640 1 -- 172.21.15.27:0/2299548849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c15d190 msgr2=0x7f856c0b4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:48.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.805+0000 7f8572600640 1 --2- 172.21.15.27:0/2299548849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c15d190 0x7f856c0b4bb0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f855400c480 tx=0x7f855400c940 comp rx=0 tx=0).stop 2024-01-15T18:09:48.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.805+0000 7f8572600640 1 -- 172.21.15.27:0/2299548849 shutdown_connections 2024-01-15T18:09:48.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.805+0000 7f8572600640 1 --2- 172.21.15.27:0/2299548849 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8544066f20 0x7f85440693e0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.805+0000 7f8572600640 1 --2- 172.21.15.27:0/2299548849 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f856c15d570 0x7f856c14fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.805+0000 7f8572600640 1 --2- 172.21.15.27:0/2299548849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856c15d190 0x7f856c0b4bb0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.805+0000 7f8572600640 1 --2- 172.21.15.27:0/2299548849 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f856c06c520 0x7f856c158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.805+0000 7f8572600640 1 -- 172.21.15.27:0/2299548849 >> 172.21.15.27:0/2299548849 conn(0x7f856c0d0e70 msgr2=0x7f856c0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:48.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.805+0000 7f8572600640 1 -- 172.21.15.27:0/2299548849 shutdown_connections 2024-01-15T18:09:48.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:48.805+0000 7f8572600640 1 -- 172.21.15.27:0/2299548849 wait complete. 2024-01-15T18:09:48.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:48.830 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/b5529c8c-34b5-49c9-ab51-a3ff8b9246fc) 2024-01-15T18:09:48.843 INFO:teuthology.orchestra.run.smithi027.stdout:# file: volumes/_nogroup/sv1/b5529c8c-34b5-49c9-ab51-a3ff8b9246fc 2024-01-15T18:09:48.843 INFO:teuthology.orchestra.run.smithi027.stdout:ceph.dir.rbytes="3138256896" 2024-01-15T18:09:48.843 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:09:48.844 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-15T18:09:50.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:09:50.846 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/b5529c8c-34b5-49c9-ab51-a3ff8b9246fc) 2024-01-15T18:09:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:# file: volumes/_nogroup/sv1/b5529c8c-34b5-49c9-ab51-a3ff8b9246fc 2024-01-15T18:09:50.860 INFO:teuthology.orchestra.run.smithi027.stdout:ceph.dir.rbytes="3221225472" 2024-01-15T18:09:50.860 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:09:50.861 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-15T18:09:50.861 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone91811 2024-01-15T18:09:51.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.057+0000 7f77c6400640 1 Processor -- start 2024-01-15T18:09:51.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.061+0000 7f77c6400640 1 -- start start 2024-01-15T18:09:51.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.061+0000 7f77c6400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f77c015d3b0 0x7f77c00cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:51.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.061+0000 7f77c6400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f77c006c3f0 0x7f77c00b1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:51.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.061+0000 7f77c6400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c00b04b0 con 0x7f77c006b600 2024-01-15T18:09:51.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.061+0000 7f77c6400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77c00d0160 con 0x7f77c006c3f0 2024-01-15T18:09:51.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.061+0000 7f77c6400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77c00b0720 con 0x7f77c015d3b0 2024-01-15T18:09:51.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.061+0000 7f77bea00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f77c006b600 0x7f77c006bce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34854/0 (socket says 172.21.15.27:34854) 2024-01-15T18:09:51.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.061+0000 7f77bea00640 1 -- 172.21.15.27:0/109669893 learned_addr learned my addr 172.21.15.27:0/109669893 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:51.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.061+0000 7f77bfe00640 1 --2- 172.21.15.27:0/109669893 >> v2:172.21.15.27:3300/0 conn(0x7f77c006c3f0 0x7f77c00b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:51.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.061+0000 7f77bf400640 1 --2- 172.21.15.27:0/109669893 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f77c015d3b0 0x7f77c00cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:51.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.061+0000 7f77bf400640 1 -- 172.21.15.27:0/109669893 >> v2:172.21.15.27:3300/0 conn(0x7f77c006c3f0 msgr2=0x7f77c00b1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:51.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.061+0000 7f77bf400640 1 --2- 172.21.15.27:0/109669893 >> v2:172.21.15.27:3300/0 conn(0x7f77c006c3f0 0x7f77c00b1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.061+0000 7f77bf400640 1 -- 172.21.15.27:0/109669893 >> v1:172.21.15.27:6789/0 conn(0x7f77c006b600 legacy=0x7f77c006bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:51.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.061+0000 7f77bf400640 1 -- 172.21.15.27:0/109669893 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c00b0a00 con 0x7f77c015d3b0 2024-01-15T18:09:51.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.065+0000 7f77bf400640 1 --2- 172.21.15.27:0/109669893 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f77c015d3b0 0x7f77c00cfc20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f77b00097c0 tx=0x7f77b001c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5351ade361990a53 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:51.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.065+0000 7f77be000640 1 -- 172.21.15.27:0/109669893 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b00ab5f0 con 0x7f77c015d3b0 2024-01-15T18:09:51.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.065+0000 7f77be000640 1 -- 172.21.15.27:0/109669893 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77b00ab750 con 0x7f77c015d3b0 2024-01-15T18:09:51.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.065+0000 7f77be000640 1 -- 172.21.15.27:0/109669893 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b00abae0 con 0x7f77c015d3b0 2024-01-15T18:09:51.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.065+0000 7f77c6400640 1 -- 172.21.15.27:0/109669893 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f77c015d3b0 msgr2=0x7f77c00cfc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:51.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.065+0000 7f77c6400640 1 --2- 172.21.15.27:0/109669893 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f77c015d3b0 0x7f77c00cfc20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f77b00097c0 tx=0x7f77b001c450 comp rx=0 tx=0).stop 2024-01-15T18:09:51.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.065+0000 7f77c6400640 1 -- 172.21.15.27:0/109669893 shutdown_connections 2024-01-15T18:09:51.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.065+0000 7f77c6400640 1 --2- 172.21.15.27:0/109669893 >> v2:172.21.15.27:3300/0 conn(0x7f77c006c3f0 0x7f77c00b1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.065+0000 7f77c6400640 1 --2- 172.21.15.27:0/109669893 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f77c015d3b0 0x7f77c00cfc20 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.065+0000 7f77c6400640 1 -- 172.21.15.27:0/109669893 >> 172.21.15.27:0/109669893 conn(0x7f77c00d0ed0 msgr2=0x7f77c00d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:51.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.065+0000 7f77c6400640 1 -- 172.21.15.27:0/109669893 shutdown_connections 2024-01-15T18:09:51.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.065+0000 7f77c6400640 1 -- 172.21.15.27:0/109669893 wait complete. 2024-01-15T18:09:51.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.069+0000 7f77c6400640 1 Processor -- start 2024-01-15T18:09:51.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.069+0000 7f77c6400640 1 -- start start 2024-01-15T18:09:51.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.069+0000 7f77c6400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f77c006b600 0x7f77c0154dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:51.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.069+0000 7f77c6400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77c006c3f0 0x7f77c014a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:51.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.069+0000 7f77c6400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f77c015d3b0 0x7f77c0159890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:51.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.069+0000 7f77c6400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c015e960 con 0x7f77c006c3f0 2024-01-15T18:09:51.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.069+0000 7f77c6400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c014c540 con 0x7f77c015d3b0 2024-01-15T18:09:51.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.069+0000 7f77c6400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77c0155300 con 0x7f77c006b600 2024-01-15T18:09:51.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.069+0000 7f77bea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77c006c3f0 0x7f77c014a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:51.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.069+0000 7f77bea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77c006c3f0 0x7f77c014a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35520/0 (socket says 172.21.15.27:35520) 2024-01-15T18:09:51.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.069+0000 7f77bea00640 1 -- 172.21.15.27:0/3984363261 learned_addr learned my addr 172.21.15.27:0/3984363261 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:51.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.069+0000 7f77bfe00640 1 --2- 172.21.15.27:0/3984363261 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f77c015d3b0 0x7f77c0159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:51.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.069+0000 7f77bf400640 1 --2- 172.21.15.27:0/3984363261 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f77c006b600 0x7f77c0154dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:51.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.069+0000 7f77bea00640 1 -- 172.21.15.27:0/3984363261 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f77c006b600 msgr2=0x7f77c0154dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:51.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.069+0000 7f77bea00640 1 --2- 172.21.15.27:0/3984363261 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f77c006b600 0x7f77c0154dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.069+0000 7f77bea00640 1 -- 172.21.15.27:0/3984363261 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f77c015d3b0 msgr2=0x7f77c0159890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:51.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.069+0000 7f77bea00640 1 --2- 172.21.15.27:0/3984363261 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f77c015d3b0 0x7f77c0159890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.069+0000 7f77bea00640 1 -- 172.21.15.27:0/3984363261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b0009420 con 0x7f77c006c3f0 2024-01-15T18:09:51.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.069+0000 7f77bea00640 1 --2- 172.21.15.27:0/3984363261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77c006c3f0 0x7f77c014a510 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f77a800c530 tx=0x7f77a800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:51.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.069+0000 7f77b7400640 1 -- 172.21.15.27:0/3984363261 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a8005cf0 con 0x7f77c006c3f0 2024-01-15T18:09:51.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.069+0000 7f77b7400640 1 -- 172.21.15.27:0/3984363261 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a8005e50 con 0x7f77c006c3f0 2024-01-15T18:09:51.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.069+0000 7f77c6400640 1 -- 172.21.15.27:0/3984363261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77c014bb30 con 0x7f77c006c3f0 2024-01-15T18:09:51.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.069+0000 7f77b7400640 1 -- 172.21.15.27:0/3984363261 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a8007270 con 0x7f77c006c3f0 2024-01-15T18:09:51.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.069+0000 7f77c6400640 1 -- 172.21.15.27:0/3984363261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77c015e5f0 con 0x7f77c006c3f0 2024-01-15T18:09:51.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.069+0000 7f77c6400640 1 -- 172.21.15.27:0/3984363261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77c006d550 con 0x7f77c006c3f0 2024-01-15T18:09:51.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.073+0000 7f77b7400640 1 -- 172.21.15.27:0/3984363261 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 15) v1 ==== 86721+0+0 (secure 0 0 0) 0x7f77a80073d0 con 0x7f77c006c3f0 2024-01-15T18:09:51.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.073+0000 7f77b7400640 1 --2- 172.21.15.27:0/3984363261 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7798066f50 0x7f7798069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:51.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.073+0000 7f77b7400640 1 -- 172.21.15.27:0/3984363261 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f77a8084410 con 0x7f77c006c3f0 2024-01-15T18:09:51.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.073+0000 7f77bf400640 1 --2- 172.21.15.27:0/3984363261 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7798066f50 0x7f7798069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.073+0000 7f77bf400640 1 --2- 172.21.15.27:0/3984363261 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7798066f50 0x7f7798069410 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f77b00af000 tx=0x7f77b00a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:51.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.077+0000 7f77b7400640 1 -- 172.21.15.27:0/3984363261 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f77a8050d60 con 0x7f77c006c3f0 2024-01-15T18:09:51.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.265+0000 7f77c6400640 1 -- 172.21.15.27:0/3984363261 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone91811", "target": ["mon-mgr", ""]}) v1 -- 0x7f77c015e470 con 0x7f7798066f50 2024-01-15T18:09:51.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.333+0000 7f77b7400640 1 -- 172.21.15.27:0/3984363261 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f77b00ab570 con 0x7f7798066f50 2024-01-15T18:09:51.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.333+0000 7f77c6400640 1 -- 172.21.15.27:0/3984363261 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7798066f50 msgr2=0x7f7798069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:51.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.333+0000 7f77c6400640 1 --2- 172.21.15.27:0/3984363261 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7798066f50 0x7f7798069410 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f77b00af000 tx=0x7f77b00a7000 comp rx=0 tx=0).stop 2024-01-15T18:09:51.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.333+0000 7f77c6400640 1 -- 172.21.15.27:0/3984363261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77c006c3f0 msgr2=0x7f77c014a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:51.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.333+0000 7f77c6400640 1 --2- 172.21.15.27:0/3984363261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77c006c3f0 0x7f77c014a510 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f77a800c530 tx=0x7f77a800c9f0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.333+0000 7f77c6400640 1 -- 172.21.15.27:0/3984363261 shutdown_connections 2024-01-15T18:09:51.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.333+0000 7f77c6400640 1 --2- 172.21.15.27:0/3984363261 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7798066f50 0x7f7798069410 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.333+0000 7f77c6400640 1 --2- 172.21.15.27:0/3984363261 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f77c015d3b0 0x7f77c0159890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.333+0000 7f77c6400640 1 --2- 172.21.15.27:0/3984363261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77c006c3f0 0x7f77c014a510 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.333+0000 7f77c6400640 1 --2- 172.21.15.27:0/3984363261 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f77c006b600 0x7f77c0154dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.333+0000 7f77c6400640 1 -- 172.21.15.27:0/3984363261 >> 172.21.15.27:0/3984363261 conn(0x7f77c00d0ed0 msgr2=0x7f77c015c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:51.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.333+0000 7f77c6400640 1 -- 172.21.15.27:0/3984363261 shutdown_connections 2024-01-15T18:09:51.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.333+0000 7f77c6400640 1 -- 172.21.15.27:0/3984363261 wait complete. 2024-01-15T18:09:51.342 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:09:51.337+0000 7f59f9c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:51.342 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:09:51.337+0000 7f59f9c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:51.342 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:09:51.337+0000 7f59f9c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:51.342 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:09:51.337+0000 7f59f9c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:51.342 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:09:51.337+0000 7f59f9c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:51.359 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:09:51.353+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:51.359 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:09:51.353+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:51.359 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:09:51.353+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:51.359 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:09:51.353+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:51.359 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:09:51.353+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:51.360 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone91812 2024-01-15T18:09:51.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.501+0000 7fb14c600640 1 Processor -- start 2024-01-15T18:09:51.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.501+0000 7fb14c600640 1 -- start start 2024-01-15T18:09:51.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.505+0000 7fb14c600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb14406c430 0x7fb144070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:51.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.505+0000 7fb14c600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fb1440b1db0 0x7fb1440b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:51.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.505+0000 7fb14c600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1440ccfb0 con 0x7fb1440ce050 2024-01-15T18:09:51.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.505+0000 7fb14c600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1440cd290 con 0x7fb14406c430 2024-01-15T18:09:51.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.505+0000 7fb14c600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1440cd570 con 0x7fb1440b1db0 2024-01-15T18:09:51.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.505+0000 7fb149c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb14406c430 0x7fb144070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:51.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.505+0000 7fb149c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb14406c430 0x7fb144070ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35532/0 (socket says 172.21.15.27:35532) 2024-01-15T18:09:51.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.505+0000 7fb149c00640 1 -- 172.21.15.27:0/3459125655 learned_addr learned my addr 172.21.15.27:0/3459125655 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:51.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.505+0000 7fb149200640 1 --2- 172.21.15.27:0/3459125655 >> v2:172.21.15.93:3300/0 conn(0x7fb1440b1db0 0x7fb1440b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:51.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.505+0000 7fb149c00640 1 -- 172.21.15.27:0/3459125655 >> v2:172.21.15.93:3300/0 conn(0x7fb1440b1db0 msgr2=0x7fb1440b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:51.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.505+0000 7fb149c00640 1 --2- 172.21.15.27:0/3459125655 >> v2:172.21.15.93:3300/0 conn(0x7fb1440b1db0 0x7fb1440b2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.505+0000 7fb149c00640 1 -- 172.21.15.27:0/3459125655 >> v1:172.21.15.93:6789/0 conn(0x7fb1440ce050 legacy=0x7fb14406c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:51.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.505+0000 7fb149c00640 1 -- 172.21.15.27:0/3459125655 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1440cd850 con 0x7fb14406c430 2024-01-15T18:09:51.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.505+0000 7fb149200640 1 --2- 172.21.15.27:0/3459125655 >> v2:172.21.15.93:3300/0 conn(0x7fb1440b1db0 0x7fb1440b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:09:51.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.509+0000 7fb149c00640 1 --2- 172.21.15.27:0/3459125655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb14406c430 0x7fb144070ba0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fb134009870 tx=0x7fb13401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51f75596424c1bae server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:51.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.509+0000 7fb143e00640 1 -- 172.21.15.27:0/3459125655 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb134004230 con 0x7fb14406c430 2024-01-15T18:09:51.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.509+0000 7fb143e00640 1 -- 172.21.15.27:0/3459125655 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1340a7050 con 0x7fb14406c430 2024-01-15T18:09:51.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.509+0000 7fb143e00640 1 -- 172.21.15.27:0/3459125655 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb134003620 con 0x7fb14406c430 2024-01-15T18:09:51.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.509+0000 7fb14c600640 1 -- 172.21.15.27:0/3459125655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb14406c430 msgr2=0x7fb144070ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:51.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.509+0000 7fb14c600640 1 --2- 172.21.15.27:0/3459125655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb14406c430 0x7fb144070ba0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fb134009870 tx=0x7fb13401c440 comp rx=0 tx=0).stop 2024-01-15T18:09:51.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.509+0000 7fb14c600640 1 -- 172.21.15.27:0/3459125655 shutdown_connections 2024-01-15T18:09:51.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.509+0000 7fb14c600640 1 --2- 172.21.15.27:0/3459125655 >> v2:172.21.15.93:3300/0 conn(0x7fb1440b1db0 0x7fb1440b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.509+0000 7fb14c600640 1 --2- 172.21.15.27:0/3459125655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb14406c430 0x7fb144070ba0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.509+0000 7fb14c600640 1 -- 172.21.15.27:0/3459125655 >> 172.21.15.27:0/3459125655 conn(0x7fb1440cb700 msgr2=0x7fb1440cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:51.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.509+0000 7fb14c600640 1 -- 172.21.15.27:0/3459125655 shutdown_connections 2024-01-15T18:09:51.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.509+0000 7fb14c600640 1 -- 172.21.15.27:0/3459125655 wait complete. 2024-01-15T18:09:51.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.509+0000 7fb14c600640 1 Processor -- start 2024-01-15T18:09:51.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.509+0000 7fb14c600640 1 -- start start 2024-01-15T18:09:51.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.513+0000 7fb14c600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb14406c430 0x7fb144148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:51.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.513+0000 7fb14c600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440b1db0 0x7fb14415b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:51.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.513+0000 7fb14c600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1440ce050 0x7fb144150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:51.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.513+0000 7fb14c600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1441fea10 con 0x7fb1440b1db0 2024-01-15T18:09:51.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.513+0000 7fb14c600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb144149430 con 0x7fb1440ce050 2024-01-15T18:09:51.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.513+0000 7fb14c600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1441fde00 con 0x7fb14406c430 2024-01-15T18:09:51.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.513+0000 7fb149c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb14406c430 0x7fb144148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:51.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.513+0000 7fb149200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440b1db0 0x7fb14415b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:51.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.513+0000 7fb149c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb14406c430 0x7fb144148ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:46394/0 (socket says 172.21.15.27:46394) 2024-01-15T18:09:51.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.513+0000 7fb149c00640 1 -- 172.21.15.27:0/3705054652 learned_addr learned my addr 172.21.15.27:0/3705054652 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:51.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.513+0000 7fb14a600640 1 --2- 172.21.15.27:0/3705054652 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1440ce050 0x7fb144150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:51.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.513+0000 7fb149c00640 1 -- 172.21.15.27:0/3705054652 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1440ce050 msgr2=0x7fb144150160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:51.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.513+0000 7fb149c00640 1 --2- 172.21.15.27:0/3705054652 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1440ce050 0x7fb144150160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.513+0000 7fb149c00640 1 -- 172.21.15.27:0/3705054652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440b1db0 msgr2=0x7fb14415b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:51.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.513+0000 7fb149c00640 1 --2- 172.21.15.27:0/3705054652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440b1db0 0x7fb14415b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.513+0000 7fb149c00640 1 -- 172.21.15.27:0/3705054652 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1340094d0 con 0x7fb14406c430 2024-01-15T18:09:51.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.513+0000 7fb149200640 1 --2- 172.21.15.27:0/3705054652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440b1db0 0x7fb14415b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:09:51.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.513+0000 7fb149c00640 1 --2- 172.21.15.27:0/3705054652 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb14406c430 0x7fb144148ef0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb134019050 tx=0x7fb1340191e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:51.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.513+0000 7fb141600640 1 -- 172.21.15.27:0/3705054652 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb134003c70 con 0x7fb14406c430 2024-01-15T18:09:51.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.513+0000 7fb141600640 1 -- 172.21.15.27:0/3705054652 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb134003dd0 con 0x7fb14406c430 2024-01-15T18:09:51.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.513+0000 7fb141600640 1 -- 172.21.15.27:0/3705054652 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1340b6660 con 0x7fb14406c430 2024-01-15T18:09:51.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.513+0000 7fb14c600640 1 -- 172.21.15.27:0/3705054652 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1441fe0e0 con 0x7fb14406c430 2024-01-15T18:09:51.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.513+0000 7fb14c600640 1 -- 172.21.15.27:0/3705054652 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1441fe6a0 con 0x7fb14406c430 2024-01-15T18:09:51.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.513+0000 7fb14c600640 1 -- 172.21.15.27:0/3705054652 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb14404e850 con 0x7fb14406c430 2024-01-15T18:09:51.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.517+0000 7fb141600640 1 -- 172.21.15.27:0/3705054652 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 15) v1 ==== 86721+0+0 (secure 0 0 0) 0x7fb1340b2040 con 0x7fb14406c430 2024-01-15T18:09:51.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.517+0000 7fb141600640 1 --2- 172.21.15.27:0/3705054652 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb11c066f50 0x7fb11c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:51.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.517+0000 7fb141600640 1 -- 172.21.15.27:0/3705054652 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fb13411f140 con 0x7fb14406c430 2024-01-15T18:09:51.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.517+0000 7fb149200640 1 --2- 172.21.15.27:0/3705054652 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb11c066f50 0x7fb11c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:51.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.517+0000 7fb149200640 1 --2- 172.21.15.27:0/3705054652 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb11c066f50 0x7fb11c069410 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb1440cb1b0 tx=0x7fb12c005c40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:51.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.521+0000 7fb141600640 1 -- 172.21.15.27:0/3705054652 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb1340ebc40 con 0x7fb14406c430 2024-01-15T18:09:51.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.709+0000 7fb14c600640 1 -- 172.21.15.27:0/3705054652 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone91812", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1441661b0 con 0x7fb11c066f50 2024-01-15T18:09:51.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.773+0000 7fb141600640 1 -- 172.21.15.27:0/3705054652 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb12c00d040 con 0x7fb11c066f50 2024-01-15T18:09:51.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.773+0000 7fb14c600640 1 -- 172.21.15.27:0/3705054652 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb11c066f50 msgr2=0x7fb11c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:51.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.773+0000 7fb14c600640 1 --2- 172.21.15.27:0/3705054652 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb11c066f50 0x7fb11c069410 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb1440cb1b0 tx=0x7fb12c005c40 comp rx=0 tx=0).stop 2024-01-15T18:09:51.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.773+0000 7fb14c600640 1 -- 172.21.15.27:0/3705054652 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb14406c430 msgr2=0x7fb144148ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:51.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.773+0000 7fb14c600640 1 --2- 172.21.15.27:0/3705054652 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb14406c430 0x7fb144148ef0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb134019050 tx=0x7fb1340191e0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.773+0000 7fb14c600640 1 -- 172.21.15.27:0/3705054652 shutdown_connections 2024-01-15T18:09:51.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.773+0000 7fb14c600640 1 --2- 172.21.15.27:0/3705054652 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb11c066f50 0x7fb11c069410 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.773+0000 7fb14c600640 1 --2- 172.21.15.27:0/3705054652 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1440ce050 0x7fb144150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.773+0000 7fb14c600640 1 --2- 172.21.15.27:0/3705054652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440b1db0 0x7fb14415b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.773+0000 7fb14c600640 1 --2- 172.21.15.27:0/3705054652 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb14406c430 0x7fb144148ef0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.773+0000 7fb14c600640 1 -- 172.21.15.27:0/3705054652 >> 172.21.15.27:0/3705054652 conn(0x7fb1440cb700 msgr2=0x7fb1440b3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:51.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.773+0000 7fb14c600640 1 -- 172.21.15.27:0/3705054652 shutdown_connections 2024-01-15T18:09:51.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.773+0000 7fb14c600640 1 -- 172.21.15.27:0/3705054652 wait complete. 2024-01-15T18:09:51.802 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone91813 2024-01-15T18:09:51.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.941+0000 7f54c4000640 1 Processor -- start 2024-01-15T18:09:51.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.941+0000 7f54c4000640 1 -- start start 2024-01-15T18:09:51.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.945+0000 7f54c4000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f54bc06b050 0x7f54bc06a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:51.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.945+0000 7f54c4000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f54bc06bc50 0x7f54bc070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:51.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.945+0000 7f54c4000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f54bc0b24b0 0x7f54bc071380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:51.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.945+0000 7f54c4000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54bc161020 con 0x7f54bc06b050 2024-01-15T18:09:51.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.945+0000 7f54c4000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54bc161300 con 0x7f54bc0b24b0 2024-01-15T18:09:51.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.945+0000 7f54c4000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc1615e0 con 0x7f54bc06bc50 2024-01-15T18:09:51.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.945+0000 7f54c1600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f54bc06b050 0x7f54bc06a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:51.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.945+0000 7f54c1600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f54bc06b050 0x7f54bc06a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35546/0 (socket says 172.21.15.27:35546) 2024-01-15T18:09:51.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.945+0000 7f54c1600640 1 -- 172.21.15.27:0/2721651102 learned_addr learned my addr 172.21.15.27:0/2721651102 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:51.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.945+0000 7f54c0c00640 1 --2- 172.21.15.27:0/2721651102 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f54bc06bc50 0x7f54bc070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:51.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.945+0000 7f54c2000640 1 --2- 172.21.15.27:0/2721651102 >> v2:172.21.15.93:3300/0 conn(0x7f54bc0b24b0 0x7f54bc071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:51.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.945+0000 7f54c1600640 1 -- 172.21.15.27:0/2721651102 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f54bc06bc50 msgr2=0x7f54bc070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:51.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.945+0000 7f54c1600640 1 --2- 172.21.15.27:0/2721651102 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f54bc06bc50 0x7f54bc070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.945+0000 7f54c1600640 1 -- 172.21.15.27:0/2721651102 >> v2:172.21.15.93:3300/0 conn(0x7f54bc0b24b0 msgr2=0x7f54bc071380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:51.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.945+0000 7f54c1600640 1 --2- 172.21.15.27:0/2721651102 >> v2:172.21.15.93:3300/0 conn(0x7f54bc0b24b0 0x7f54bc071380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.945+0000 7f54c1600640 1 -- 172.21.15.27:0/2721651102 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54bc1618c0 con 0x7f54bc06b050 2024-01-15T18:09:51.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.949+0000 7f54c1600640 1 --2- 172.21.15.27:0/2721651102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54bc06b050 0x7f54bc06a4f0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f54bc071bb0 tx=0x7f54ac01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ada840114b850ba server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:51.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.949+0000 7f54bbe00640 1 -- 172.21.15.27:0/2721651102 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac0ab610 con 0x7f54bc06b050 2024-01-15T18:09:51.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.949+0000 7f54bbe00640 1 -- 172.21.15.27:0/2721651102 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54ac0ab770 con 0x7f54bc06b050 2024-01-15T18:09:51.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.949+0000 7f54bbe00640 1 -- 172.21.15.27:0/2721651102 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac0abb00 con 0x7f54bc06b050 2024-01-15T18:09:51.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.949+0000 7f54c4000640 1 -- 172.21.15.27:0/2721651102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54bc06b050 msgr2=0x7f54bc06a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:51.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.949+0000 7f54c4000640 1 --2- 172.21.15.27:0/2721651102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54bc06b050 0x7f54bc06a4f0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f54bc071bb0 tx=0x7f54ac01c470 comp rx=0 tx=0).stop 2024-01-15T18:09:51.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.949+0000 7f54c4000640 1 -- 172.21.15.27:0/2721651102 shutdown_connections 2024-01-15T18:09:51.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.949+0000 7f54c4000640 1 --2- 172.21.15.27:0/2721651102 >> v2:172.21.15.93:3300/0 conn(0x7f54bc0b24b0 0x7f54bc071380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.949+0000 7f54c4000640 1 --2- 172.21.15.27:0/2721651102 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f54bc06bc50 0x7f54bc070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.949+0000 7f54c4000640 1 --2- 172.21.15.27:0/2721651102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54bc06b050 0x7f54bc06a4f0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.949+0000 7f54c4000640 1 -- 172.21.15.27:0/2721651102 >> 172.21.15.27:0/2721651102 conn(0x7f54bc0b15b0 msgr2=0x7f54bc0b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:51.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.949+0000 7f54c4000640 1 -- 172.21.15.27:0/2721651102 shutdown_connections 2024-01-15T18:09:51.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.949+0000 7f54c4000640 1 -- 172.21.15.27:0/2721651102 wait complete. 2024-01-15T18:09:51.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.949+0000 7f54c4000640 1 Processor -- start 2024-01-15T18:09:51.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.949+0000 7f54c4000640 1 -- start start 2024-01-15T18:09:51.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.953+0000 7f54c4000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54bc06b050 0x7f54bc141e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:51.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.953+0000 7f54c4000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f54bc06bc50 0x7f54bc151db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:51.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.953+0000 7f54c4000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54bc0b24b0 0x7f54bc153b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:51.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.953+0000 7f54c4000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc153380 con 0x7f54bc0b24b0 2024-01-15T18:09:51.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.953+0000 7f54c4000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc152770 con 0x7f54bc06b050 2024-01-15T18:09:51.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.953+0000 7f54c4000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc152a50 con 0x7f54bc06bc50 2024-01-15T18:09:51.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.953+0000 7f54c2000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54bc0b24b0 0x7f54bc153b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:51.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.953+0000 7f54c1600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54bc06b050 0x7f54bc141e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:51.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.953+0000 7f54c0c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f54bc06bc50 0x7f54bc151db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:51.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.953+0000 7f54c1600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54bc06b050 0x7f54bc141e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:43886/0 (socket says 172.21.15.27:43886) 2024-01-15T18:09:51.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.953+0000 7f54c1600640 1 -- 172.21.15.27:0/3306995953 learned_addr learned my addr 172.21.15.27:0/3306995953 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:51.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.953+0000 7f54c1600640 1 -- 172.21.15.27:0/3306995953 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f54bc06bc50 msgr2=0x7f54bc151db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:51.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.953+0000 7f54c1600640 1 --2- 172.21.15.27:0/3306995953 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f54bc06bc50 0x7f54bc151db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.953+0000 7f54c1600640 1 -- 172.21.15.27:0/3306995953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54bc0b24b0 msgr2=0x7f54bc153b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:51.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.953+0000 7f54c1600640 1 --2- 172.21.15.27:0/3306995953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54bc0b24b0 0x7f54bc153b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:51.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.953+0000 7f54c1600640 1 -- 172.21.15.27:0/3306995953 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54ac0094d0 con 0x7f54bc06b050 2024-01-15T18:09:51.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.953+0000 7f54c1600640 1 --2- 172.21.15.27:0/3306995953 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54bc06b050 0x7f54bc141e90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f54ac01cd10 tx=0x7f54ac01acb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:51.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.953+0000 7f54b9600640 1 -- 172.21.15.27:0/3306995953 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac0be5f0 con 0x7f54bc06b050 2024-01-15T18:09:51.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.953+0000 7f54b9600640 1 -- 172.21.15.27:0/3306995953 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54ac0be750 con 0x7f54bc06b050 2024-01-15T18:09:51.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.953+0000 7f54c4000640 1 -- 172.21.15.27:0/3306995953 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54bc152d30 con 0x7f54bc06b050 2024-01-15T18:09:51.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.953+0000 7f54b9600640 1 -- 172.21.15.27:0/3306995953 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac0beb00 con 0x7f54bc06b050 2024-01-15T18:09:51.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.953+0000 7f54c4000640 1 -- 172.21.15.27:0/3306995953 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54bc1423d0 con 0x7f54bc06b050 2024-01-15T18:09:51.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.953+0000 7f54c4000640 1 -- 172.21.15.27:0/3306995953 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54bc15d140 con 0x7f54bc06b050 2024-01-15T18:09:51.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.953+0000 7f54b9600640 1 -- 172.21.15.27:0/3306995953 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 15) v1 ==== 86721+0+0 (secure 0 0 0) 0x7f54ac0b43f0 con 0x7f54bc06b050 2024-01-15T18:09:51.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.957+0000 7f54b9600640 1 --2- 172.21.15.27:0/3306995953 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5494066f20 0x7f54940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:51.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.957+0000 7f54b9600640 1 -- 172.21.15.27:0/3306995953 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f54ac11fff0 con 0x7f54bc06b050 2024-01-15T18:09:51.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.957+0000 7f54c0c00640 1 --2- 172.21.15.27:0/3306995953 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5494066f20 0x7f54940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:51.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.957+0000 7f54c0c00640 1 --2- 172.21.15.27:0/3306995953 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5494066f20 0x7f54940693e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f54bc153780 tx=0x7f54a4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:51.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:51.961+0000 7f54b9600640 1 -- 172.21.15.27:0/3306995953 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f54ac0ec840 con 0x7f54bc06b050 2024-01-15T18:09:52.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:52.145+0000 7f54c4000640 1 -- 172.21.15.27:0/3306995953 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone91813", "target": ["mon-mgr", ""]}) v1 -- 0x7f54bc04e850 con 0x7f5494066f20 2024-01-15T18:09:53.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:53.077+0000 7f54b9600640 1 -- 172.21.15.27:0/3306995953 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mgrmap(e 16) v1 ==== 86823+0+0 (secure 0 0 0) 0x7f54ac0ecc10 con 0x7f54bc06b050 2024-01-15T18:09:55.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.309+0000 7f54b9600640 1 -- 172.21.15.27:0/3306995953 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f54bc04e850 con 0x7f5494066f20 2024-01-15T18:09:55.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.309+0000 7f54c4000640 1 -- 172.21.15.27:0/3306995953 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5494066f20 msgr2=0x7f54940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:55.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.309+0000 7f54c4000640 1 --2- 172.21.15.27:0/3306995953 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5494066f20 0x7f54940693e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f54bc153780 tx=0x7f54a4009190 comp rx=0 tx=0).stop 2024-01-15T18:09:55.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.309+0000 7f54c4000640 1 -- 172.21.15.27:0/3306995953 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54bc06b050 msgr2=0x7f54bc141e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:55.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.309+0000 7f54c4000640 1 --2- 172.21.15.27:0/3306995953 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54bc06b050 0x7f54bc141e90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f54ac01cd10 tx=0x7f54ac01acb0 comp rx=0 tx=0).stop 2024-01-15T18:09:55.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.313+0000 7f54c4000640 1 -- 172.21.15.27:0/3306995953 shutdown_connections 2024-01-15T18:09:55.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.313+0000 7f54c4000640 1 --2- 172.21.15.27:0/3306995953 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5494066f20 0x7f54940693e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:55.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.313+0000 7f54c4000640 1 --2- 172.21.15.27:0/3306995953 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54bc0b24b0 0x7f54bc153b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:55.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.313+0000 7f54c4000640 1 --2- 172.21.15.27:0/3306995953 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f54bc06bc50 0x7f54bc151db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:55.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.313+0000 7f54c4000640 1 --2- 172.21.15.27:0/3306995953 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f54bc06b050 0x7f54bc141e90 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:55.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.313+0000 7f54c4000640 1 -- 172.21.15.27:0/3306995953 >> 172.21.15.27:0/3306995953 conn(0x7f54bc0b15b0 msgr2=0x7f54bc15c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:55.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.313+0000 7f54c4000640 1 -- 172.21.15.27:0/3306995953 shutdown_connections 2024-01-15T18:09:55.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.313+0000 7f54c4000640 1 -- 172.21.15.27:0/3306995953 wait complete. 2024-01-15T18:09:55.342 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone91814 2024-01-15T18:09:55.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.485+0000 7fb574800640 1 Processor -- start 2024-01-15T18:09:55.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.485+0000 7fb574800640 1 -- start start 2024-01-15T18:09:55.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.485+0000 7fb574800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fb56c06b600 0x7fb56c06bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:55.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.485+0000 7fb574800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb56c0b04b0 con 0x7fb56c06c220 2024-01-15T18:09:55.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.485+0000 7fb574800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb56c0b0720 con 0x7fb56c15d3b0 2024-01-15T18:09:55.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.485+0000 7fb574800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb56c06c6b0 con 0x7fb56c06b600 2024-01-15T18:09:55.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.485+0000 7fb572800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb56c06c220 0x7fb56c0b1200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:33664/0 (socket says 172.21.15.27:33664) 2024-01-15T18:09:55.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.485+0000 7fb572800640 1 -- 172.21.15.27:0/4264901231 learned_addr learned my addr 172.21.15.27:0/4264901231 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:09:55.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.485+0000 7fb571400640 1 --2- 172.21.15.27:0/4264901231 >> v2:172.21.15.93:3300/0 conn(0x7fb56c06b600 0x7fb56c06bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:55.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.485+0000 7fb570a00640 1 -- 172.21.15.27:0/4264901231 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736180754 0 0) 0x7fb56c0b04b0 con 0x7fb56c06c220 2024-01-15T18:09:55.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.485+0000 7fb570a00640 1 -- 172.21.15.27:0/4264901231 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb54c003530 con 0x7fb56c06c220 2024-01-15T18:09:55.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.489+0000 7fb570a00640 1 -- 172.21.15.27:0/4264901231 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1348400579 0 0) 0x7fb56c0b0720 con 0x7fb56c15d3b0 2024-01-15T18:09:55.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.489+0000 7fb570a00640 1 -- 172.21.15.27:0/4264901231 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb56c0b04b0 con 0x7fb56c15d3b0 2024-01-15T18:09:55.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.489+0000 7fb570a00640 1 -- 172.21.15.27:0/4264901231 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 610074633 0 0) 0x7fb54c003530 con 0x7fb56c06c220 2024-01-15T18:09:55.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.489+0000 7fb571400640 1 -- 172.21.15.27:0/4264901231 >> v1:172.21.15.93:6789/0 conn(0x7fb56c15d3b0 legacy=0x7fb56c0cfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:55.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.489+0000 7fb571400640 1 -- 172.21.15.27:0/4264901231 >> v1:172.21.15.27:6789/0 conn(0x7fb56c06c220 legacy=0x7fb56c0b1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:55.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.489+0000 7fb571400640 1 -- 172.21.15.27:0/4264901231 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb56c0b0a00 con 0x7fb56c06b600 2024-01-15T18:09:55.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.489+0000 7fb571400640 1 --2- 172.21.15.27:0/4264901231 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb56c06b600 0x7fb56c06bce0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb5540097c0 tx=0x7fb55401c370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edf52017384ad1de server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:55.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.489+0000 7fb570a00640 1 -- 172.21.15.27:0/4264901231 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb564001a00 con 0x7fb56c06b600 2024-01-15T18:09:55.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.489+0000 7fb570a00640 1 -- 172.21.15.27:0/4264901231 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5540a7050 con 0x7fb56c06b600 2024-01-15T18:09:55.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.489+0000 7fb570a00640 1 -- 172.21.15.27:0/4264901231 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5540ab620 con 0x7fb56c06b600 2024-01-15T18:09:55.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.493+0000 7fb574800640 1 -- 172.21.15.27:0/4264901231 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb56c06b600 msgr2=0x7fb56c06bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:55.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.493+0000 7fb574800640 1 --2- 172.21.15.27:0/4264901231 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb56c06b600 0x7fb56c06bce0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb5540097c0 tx=0x7fb55401c370 comp rx=0 tx=0).stop 2024-01-15T18:09:55.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.493+0000 7fb574800640 1 -- 172.21.15.27:0/4264901231 shutdown_connections 2024-01-15T18:09:55.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.493+0000 7fb574800640 1 --2- 172.21.15.27:0/4264901231 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb56c06b600 0x7fb56c06bce0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:55.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.493+0000 7fb574800640 1 -- 172.21.15.27:0/4264901231 >> 172.21.15.27:0/4264901231 conn(0x7fb56c0d0ed0 msgr2=0x7fb56c0d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:55.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.493+0000 7fb574800640 1 -- 172.21.15.27:0/4264901231 shutdown_connections 2024-01-15T18:09:55.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.493+0000 7fb574800640 1 -- 172.21.15.27:0/4264901231 wait complete. 2024-01-15T18:09:55.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.493+0000 7fb574800640 1 Processor -- start 2024-01-15T18:09:55.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.493+0000 7fb574800640 1 -- start start 2024-01-15T18:09:55.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.493+0000 7fb574800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb56c06b600 0x7fb56c14f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:55.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.493+0000 7fb574800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb56c06c220 0x7fb56c1592c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:55.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.493+0000 7fb574800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb56c15d3b0 0x7fb56c14ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:55.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.493+0000 7fb574800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c159110 con 0x7fb56c06b600 2024-01-15T18:09:55.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.493+0000 7fb574800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c150510 con 0x7fb56c15d3b0 2024-01-15T18:09:55.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.493+0000 7fb574800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c14f900 con 0x7fb56c06c220 2024-01-15T18:09:55.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.493+0000 7fb571e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb56c06b600 0x7fb56c14f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:55.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.493+0000 7fb571e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb56c06b600 0x7fb56c14f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60580/0 (socket says 172.21.15.27:60580) 2024-01-15T18:09:55.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.493+0000 7fb571e00640 1 -- 172.21.15.27:0/3035450677 learned_addr learned my addr 172.21.15.27:0/3035450677 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:09:55.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.493+0000 7fb572800640 1 --2- 172.21.15.27:0/3035450677 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb56c15d3b0 0x7fb56c14ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:55.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.497+0000 7fb571400640 1 --2- 172.21.15.27:0/3035450677 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb56c06c220 0x7fb56c1592c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:55.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.497+0000 7fb571e00640 1 -- 172.21.15.27:0/3035450677 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb56c06c220 msgr2=0x7fb56c1592c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:55.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.497+0000 7fb571e00640 1 --2- 172.21.15.27:0/3035450677 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb56c06c220 0x7fb56c1592c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:55.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.497+0000 7fb571e00640 1 -- 172.21.15.27:0/3035450677 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb56c15d3b0 msgr2=0x7fb56c14ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:55.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.497+0000 7fb571e00640 1 --2- 172.21.15.27:0/3035450677 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb56c15d3b0 0x7fb56c14ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:55.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.497+0000 7fb571e00640 1 -- 172.21.15.27:0/3035450677 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb554009420 con 0x7fb56c06b600 2024-01-15T18:09:55.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.497+0000 7fb571e00640 1 --2- 172.21.15.27:0/3035450677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb56c06b600 0x7fb56c14f240 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fb55c00c4c0 tx=0x7fb55c00c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:55.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.497+0000 7fb56a000640 1 -- 172.21.15.27:0/3035450677 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c0052d0 con 0x7fb56c06b600 2024-01-15T18:09:55.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.497+0000 7fb56a000640 1 -- 172.21.15.27:0/3035450677 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55c005430 con 0x7fb56c06b600 2024-01-15T18:09:55.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.497+0000 7fb574800640 1 -- 172.21.15.27:0/3035450677 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb56c14fbe0 con 0x7fb56c06b600 2024-01-15T18:09:55.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.497+0000 7fb56a000640 1 -- 172.21.15.27:0/3035450677 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c012870 con 0x7fb56c06b600 2024-01-15T18:09:55.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.497+0000 7fb574800640 1 -- 172.21.15.27:0/3035450677 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb56c1501a0 con 0x7fb56c06b600 2024-01-15T18:09:55.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.497+0000 7fb574800640 1 -- 172.21.15.27:0/3035450677 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb56c04e850 con 0x7fb56c06b600 2024-01-15T18:09:55.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.497+0000 7fb56a000640 1 -- 172.21.15.27:0/3035450677 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 86823+0+0 (secure 0 0 0) 0x7fb55c0129d0 con 0x7fb56c06b600 2024-01-15T18:09:55.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.497+0000 7fb56a000640 1 --2- 172.21.15.27:0/3035450677 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb544067040 0x7fb544069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:55.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.501+0000 7fb56a000640 1 -- 172.21.15.27:0/3035450677 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fb55c0850b0 con 0x7fb56c06b600 2024-01-15T18:09:55.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.501+0000 7fb571400640 1 --2- 172.21.15.27:0/3035450677 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb544067040 0x7fb544069500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:55.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.501+0000 7fb571400640 1 --2- 172.21.15.27:0/3035450677 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb544067040 0x7fb544069500 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb5540ad000 tx=0x7fb5540a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:55.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.505+0000 7fb56a000640 1 -- 172.21.15.27:0/3035450677 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb55c082550 con 0x7fb56c06b600 2024-01-15T18:09:55.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:09:55.689+0000 7fb574800640 1 -- 172.21.15.27:0/3035450677 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone91814", "target": ["mon-mgr", ""]}) v1 -- 0x7fb56c150650 con 0x7fb544067040 2024-01-15T18:09:56.764 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:09:56.767 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:10:00.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.301+0000 7fb56a000640 1 -- 172.21.15.27:0/3035450677 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb554004200 con 0x7fb544067040 2024-01-15T18:10:00.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.305+0000 7fb574800640 1 -- 172.21.15.27:0/3035450677 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb544067040 msgr2=0x7fb544069500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:00.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.305+0000 7fb574800640 1 --2- 172.21.15.27:0/3035450677 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb544067040 0x7fb544069500 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb5540ad000 tx=0x7fb5540a6000 comp rx=0 tx=0).stop 2024-01-15T18:10:00.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.305+0000 7fb574800640 1 -- 172.21.15.27:0/3035450677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb56c06b600 msgr2=0x7fb56c14f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:00.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.305+0000 7fb574800640 1 --2- 172.21.15.27:0/3035450677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb56c06b600 0x7fb56c14f240 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fb55c00c4c0 tx=0x7fb55c00c980 comp rx=0 tx=0).stop 2024-01-15T18:10:00.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.305+0000 7fb574800640 1 -- 172.21.15.27:0/3035450677 shutdown_connections 2024-01-15T18:10:00.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.305+0000 7fb574800640 1 --2- 172.21.15.27:0/3035450677 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb544067040 0x7fb544069500 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:00.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.305+0000 7fb574800640 1 --2- 172.21.15.27:0/3035450677 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb56c15d3b0 0x7fb56c14ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:00.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.305+0000 7fb574800640 1 --2- 172.21.15.27:0/3035450677 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb56c06c220 0x7fb56c1592c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:00.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.305+0000 7fb574800640 1 --2- 172.21.15.27:0/3035450677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb56c06b600 0x7fb56c14f240 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:00.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.305+0000 7fb574800640 1 -- 172.21.15.27:0/3035450677 >> 172.21.15.27:0/3035450677 conn(0x7fb56c0d0ed0 msgr2=0x7fb56c0d12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:00.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.305+0000 7fb574800640 1 -- 172.21.15.27:0/3035450677 shutdown_connections 2024-01-15T18:10:00.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.305+0000 7fb574800640 1 -- 172.21.15.27:0/3035450677 wait complete. 2024-01-15T18:10:00.331 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone91815 2024-01-15T18:10:00.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.473+0000 7f1401a00640 1 Processor -- start 2024-01-15T18:10:00.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.473+0000 7f1401a00640 1 -- start start 2024-01-15T18:10:00.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.473+0000 7f1401a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f13fc1596d0 0x7f13fc0b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:00.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.473+0000 7f1401a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13fc0b3ee0 0x7f13fc06d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:00.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.473+0000 7f1401a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13fc0ccc90 con 0x7f13fc06d6b0 2024-01-15T18:10:00.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.473+0000 7f1401a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13fc0b2fa0 con 0x7f13fc1596d0 2024-01-15T18:10:00.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.473+0000 7f1401a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13fc06db40 con 0x7f13fc0b3ee0 2024-01-15T18:10:00.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.473+0000 7f13faa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f13fc1596d0 0x7f13fc0b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:00.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.473+0000 7f13faa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f13fc1596d0 0x7f13fc0b1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60592/0 (socket says 172.21.15.27:60592) 2024-01-15T18:10:00.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.473+0000 7f13faa00640 1 -- 172.21.15.27:0/1600491885 learned_addr learned my addr 172.21.15.27:0/1600491885 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:00.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.473+0000 7f13faa00640 1 -- 172.21.15.27:0/1600491885 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13fc0b3ee0 msgr2=0x7f13fc06d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:00.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.473+0000 7f13faa00640 1 --2- 172.21.15.27:0/1600491885 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13fc0b3ee0 0x7f13fc06d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:00.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.473+0000 7f13faa00640 1 -- 172.21.15.27:0/1600491885 >> v1:172.21.15.93:6789/0 conn(0x7f13fc06d6b0 legacy=0x7f13fc070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:00.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.473+0000 7f13faa00640 1 -- 172.21.15.27:0/1600491885 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13fc071850 con 0x7f13fc1596d0 2024-01-15T18:10:00.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.477+0000 7f13faa00640 1 --2- 172.21.15.27:0/1600491885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13fc1596d0 0x7f13fc0b1db0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f13ec009870 tx=0x7f13ec01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c11f4879d5247634 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:00.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.477+0000 7f13f9600640 1 -- 172.21.15.27:0/1600491885 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ec0ab600 con 0x7f13fc1596d0 2024-01-15T18:10:00.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.477+0000 7f13f9600640 1 -- 172.21.15.27:0/1600491885 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13ec0ab760 con 0x7f13fc1596d0 2024-01-15T18:10:00.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.477+0000 7f13f9600640 1 -- 172.21.15.27:0/1600491885 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ec0abaf0 con 0x7f13fc1596d0 2024-01-15T18:10:00.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.477+0000 7f1401a00640 1 -- 172.21.15.27:0/1600491885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13fc1596d0 msgr2=0x7f13fc0b1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:00.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.477+0000 7f1401a00640 1 --2- 172.21.15.27:0/1600491885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13fc1596d0 0x7f13fc0b1db0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f13ec009870 tx=0x7f13ec01c440 comp rx=0 tx=0).stop 2024-01-15T18:10:00.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.477+0000 7f1401a00640 1 -- 172.21.15.27:0/1600491885 shutdown_connections 2024-01-15T18:10:00.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.477+0000 7f1401a00640 1 --2- 172.21.15.27:0/1600491885 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13fc0b3ee0 0x7f13fc06d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:00.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.477+0000 7f1401a00640 1 --2- 172.21.15.27:0/1600491885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13fc1596d0 0x7f13fc0b1db0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:00.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.477+0000 7f1401a00640 1 -- 172.21.15.27:0/1600491885 >> 172.21.15.27:0/1600491885 conn(0x7f13fc06a8f0 msgr2=0x7f13fc06ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:00.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.477+0000 7f1401a00640 1 -- 172.21.15.27:0/1600491885 shutdown_connections 2024-01-15T18:10:00.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.477+0000 7f1401a00640 1 -- 172.21.15.27:0/1600491885 wait complete. 2024-01-15T18:10:00.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.481+0000 7f1401a00640 1 Processor -- start 2024-01-15T18:10:00.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.481+0000 7f1401a00640 1 -- start start 2024-01-15T18:10:00.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.481+0000 7f1401a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13fc06d6b0 0x7f13fc15b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:00.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.481+0000 7f1401a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13fc0b3ee0 0x7f13fc1384e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:00.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.481+0000 7f1401a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13fc1596d0 0x7f13fc1555f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:00.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.481+0000 7f1401a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13fc14ad30 con 0x7f13fc06d6b0 2024-01-15T18:10:00.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.481+0000 7f1401a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13fc14af80 con 0x7f13fc1596d0 2024-01-15T18:10:00.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.481+0000 7f1401a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13fc14b260 con 0x7f13fc0b3ee0 2024-01-15T18:10:00.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.481+0000 7f13faa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13fc06d6b0 0x7f13fc15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:00.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.481+0000 7f13faa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13fc06d6b0 0x7f13fc15b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60598/0 (socket says 172.21.15.27:60598) 2024-01-15T18:10:00.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.481+0000 7f13faa00640 1 -- 172.21.15.27:0/2414421290 learned_addr learned my addr 172.21.15.27:0/2414421290 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:00.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.481+0000 7f13fb400640 1 --2- 172.21.15.27:0/2414421290 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13fc1596d0 0x7f13fc1555f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:00.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.481+0000 7f13fa000640 1 --2- 172.21.15.27:0/2414421290 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13fc0b3ee0 0x7f13fc1384e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:00.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.481+0000 7f13faa00640 1 -- 172.21.15.27:0/2414421290 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13fc0b3ee0 msgr2=0x7f13fc1384e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:00.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.481+0000 7f13faa00640 1 --2- 172.21.15.27:0/2414421290 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13fc0b3ee0 0x7f13fc1384e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:00.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.481+0000 7f13faa00640 1 -- 172.21.15.27:0/2414421290 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13fc1596d0 msgr2=0x7f13fc1555f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:00.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.481+0000 7f13faa00640 1 --2- 172.21.15.27:0/2414421290 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13fc1596d0 0x7f13fc1555f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:00.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.481+0000 7f13faa00640 1 -- 172.21.15.27:0/2414421290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ec0094d0 con 0x7f13fc06d6b0 2024-01-15T18:10:00.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.481+0000 7f13faa00640 1 --2- 172.21.15.27:0/2414421290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13fc06d6b0 0x7f13fc15b690 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f13ec0ad040 tx=0x7f13ec01ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:00.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.481+0000 7f13f2a00640 1 -- 172.21.15.27:0/2414421290 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ec0b4800 con 0x7f13fc06d6b0 2024-01-15T18:10:00.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.481+0000 7f13f2a00640 1 -- 172.21.15.27:0/2414421290 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13ec0b4980 con 0x7f13fc06d6b0 2024-01-15T18:10:00.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.481+0000 7f13f2a00640 1 -- 172.21.15.27:0/2414421290 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ec0b4d30 con 0x7f13fc06d6b0 2024-01-15T18:10:00.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.481+0000 7f1401a00640 1 -- 172.21.15.27:0/2414421290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13fc14b540 con 0x7f13fc06d6b0 2024-01-15T18:10:00.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.481+0000 7f1401a00640 1 -- 172.21.15.27:0/2414421290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13fc14bb00 con 0x7f13fc06d6b0 2024-01-15T18:10:00.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.481+0000 7f1401a00640 1 -- 172.21.15.27:0/2414421290 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13fc04e850 con 0x7f13fc06d6b0 2024-01-15T18:10:00.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.485+0000 7f13f2a00640 1 -- 172.21.15.27:0/2414421290 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 86823+0+0 (secure 0 0 0) 0x7f13ec0b3380 con 0x7f13fc06d6b0 2024-01-15T18:10:00.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.485+0000 7f13f2a00640 1 --2- 172.21.15.27:0/2414421290 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f13d4067040 0x7f13d4069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:00.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.485+0000 7f13f2a00640 1 -- 172.21.15.27:0/2414421290 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f13ec11f990 con 0x7f13fc06d6b0 2024-01-15T18:10:00.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.485+0000 7f13fa000640 1 --2- 172.21.15.27:0/2414421290 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f13d4067040 0x7f13d4069500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:00.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.485+0000 7f13fa000640 1 --2- 172.21.15.27:0/2414421290 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f13d4067040 0x7f13d4069500 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f13fc14fe90 tx=0x7f13e4009110 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:00.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.489+0000 7f13f2a00640 1 -- 172.21.15.27:0/2414421290 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f13ec11d130 con 0x7f13fc06d6b0 2024-01-15T18:10:00.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:00.673+0000 7f1401a00640 1 -- 172.21.15.27:0/2414421290 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone91815", "target": ["mon-mgr", ""]}) v1 -- 0x7f13fc159ab0 con 0x7f13d4067040 2024-01-15T18:10:05.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.309+0000 7f13f2a00640 1 -- 172.21.15.27:0/2414421290 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f13e40095e0 con 0x7f13d4067040 2024-01-15T18:10:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.313+0000 7f1401a00640 1 -- 172.21.15.27:0/2414421290 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f13d4067040 msgr2=0x7f13d4069500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.313+0000 7f1401a00640 1 --2- 172.21.15.27:0/2414421290 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f13d4067040 0x7f13d4069500 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f13fc14fe90 tx=0x7f13e4009110 comp rx=0 tx=0).stop 2024-01-15T18:10:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.313+0000 7f1401a00640 1 -- 172.21.15.27:0/2414421290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13fc06d6b0 msgr2=0x7f13fc15b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.313+0000 7f1401a00640 1 --2- 172.21.15.27:0/2414421290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13fc06d6b0 0x7f13fc15b690 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f13ec0ad040 tx=0x7f13ec01ac80 comp rx=0 tx=0).stop 2024-01-15T18:10:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.313+0000 7f1401a00640 1 -- 172.21.15.27:0/2414421290 shutdown_connections 2024-01-15T18:10:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.313+0000 7f1401a00640 1 --2- 172.21.15.27:0/2414421290 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f13d4067040 0x7f13d4069500 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.313+0000 7f1401a00640 1 --2- 172.21.15.27:0/2414421290 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13fc1596d0 0x7f13fc1555f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:05.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.313+0000 7f1401a00640 1 --2- 172.21.15.27:0/2414421290 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13fc0b3ee0 0x7f13fc1384e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:05.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.313+0000 7f1401a00640 1 --2- 172.21.15.27:0/2414421290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13fc06d6b0 0x7f13fc15b690 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:05.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.313+0000 7f1401a00640 1 -- 172.21.15.27:0/2414421290 >> 172.21.15.27:0/2414421290 conn(0x7f13fc06a8f0 msgr2=0x7f13fc149c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:05.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.313+0000 7f1401a00640 1 -- 172.21.15.27:0/2414421290 shutdown_connections 2024-01-15T18:10:05.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.313+0000 7f1401a00640 1 -- 172.21.15.27:0/2414421290 wait complete. 2024-01-15T18:10:05.345 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone91816 2024-01-15T18:10:05.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.497+0000 7fa460600640 1 Processor -- start 2024-01-15T18:10:05.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.497+0000 7fa460600640 1 -- start start 2024-01-15T18:10:05.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.497+0000 7fa460600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa4581468d0 0x7fa4580ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:05.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.497+0000 7fa460600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4580cef30 0x7fa4580cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:05.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.497+0000 7fa460600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa4580ccea0 0x7fa458160cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:05.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.497+0000 7fa460600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4580ad920 con 0x7fa4580ccea0 2024-01-15T18:10:05.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.497+0000 7fa460600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4580ac880 con 0x7fa4581468d0 2024-01-15T18:10:05.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.497+0000 7fa460600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4580acb60 con 0x7fa4580cef30 2024-01-15T18:10:05.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.497+0000 7fa45e600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa4580ccea0 0x7fa458160cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:05.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.497+0000 7fa45e600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa4580ccea0 0x7fa458160cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55596/0 (socket says 172.21.15.27:55596) 2024-01-15T18:10:05.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.497+0000 7fa45e600640 1 -- 172.21.15.27:0/806434789 learned_addr learned my addr 172.21.15.27:0/806434789 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:05.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.497+0000 7fa45dc00640 1 --2- 172.21.15.27:0/806434789 >> v2:172.21.15.93:3300/0 conn(0x7fa4581468d0 0x7fa4580ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:05.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.497+0000 7fa45d200640 1 --2- 172.21.15.27:0/806434789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4580cef30 0x7fa4580cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:05.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.497+0000 7fa45e600640 1 -- 172.21.15.27:0/806434789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4580cef30 msgr2=0x7fa4580cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:05.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.497+0000 7fa45e600640 1 --2- 172.21.15.27:0/806434789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4580cef30 0x7fa4580cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:05.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.497+0000 7fa45e600640 1 -- 172.21.15.27:0/806434789 >> v2:172.21.15.93:3300/0 conn(0x7fa4581468d0 msgr2=0x7fa4580ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:05.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.497+0000 7fa45e600640 1 --2- 172.21.15.27:0/806434789 >> v2:172.21.15.93:3300/0 conn(0x7fa4581468d0 0x7fa4580ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:05.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.497+0000 7fa45e600640 1 -- 172.21.15.27:0/806434789 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4580ace40 con 0x7fa4580ccea0 2024-01-15T18:10:05.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.501+0000 7fa45e600640 1 --2- 172.21.15.27:0/806434789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4580ccea0 0x7fa458160cd0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fa450009e50 tx=0x7fa45001c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb19576a8d119164 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:05.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.501+0000 7fa457e00640 1 -- 172.21.15.27:0/806434789 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4500a5560 con 0x7fa4580ccea0 2024-01-15T18:10:05.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.501+0000 7fa457e00640 1 -- 172.21.15.27:0/806434789 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4500a56c0 con 0x7fa4580ccea0 2024-01-15T18:10:05.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.501+0000 7fa457e00640 1 -- 172.21.15.27:0/806434789 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4500a5a50 con 0x7fa4580ccea0 2024-01-15T18:10:05.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.501+0000 7fa460600640 1 -- 172.21.15.27:0/806434789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4580ccea0 msgr2=0x7fa458160cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:05.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.501+0000 7fa460600640 1 --2- 172.21.15.27:0/806434789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4580ccea0 0x7fa458160cd0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fa450009e50 tx=0x7fa45001c6e0 comp rx=0 tx=0).stop 2024-01-15T18:10:05.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.501+0000 7fa460600640 1 -- 172.21.15.27:0/806434789 shutdown_connections 2024-01-15T18:10:05.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.501+0000 7fa460600640 1 --2- 172.21.15.27:0/806434789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4580ccea0 0x7fa458160cd0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:05.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.501+0000 7fa460600640 1 --2- 172.21.15.27:0/806434789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4580cef30 0x7fa4580cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:05.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.501+0000 7fa460600640 1 --2- 172.21.15.27:0/806434789 >> v2:172.21.15.93:3300/0 conn(0x7fa4581468d0 0x7fa4580ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:05.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.501+0000 7fa460600640 1 -- 172.21.15.27:0/806434789 >> 172.21.15.27:0/806434789 conn(0x7fa4580cb700 msgr2=0x7fa4580cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:05.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.501+0000 7fa460600640 1 -- 172.21.15.27:0/806434789 shutdown_connections 2024-01-15T18:10:05.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.501+0000 7fa460600640 1 -- 172.21.15.27:0/806434789 wait complete. 2024-01-15T18:10:05.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.501+0000 7fa460600640 1 Processor -- start 2024-01-15T18:10:05.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa460600640 1 -- start start 2024-01-15T18:10:05.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa460600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4580ccea0 0x7fa45815e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:05.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa460600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4580cef30 0x7fa458159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:05.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa460600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa4581468d0 0x7fa45813b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:05.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa460600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4580704a0 con 0x7fa4580cef30 2024-01-15T18:10:05.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa460600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45815daa0 con 0x7fa4581468d0 2024-01-15T18:10:05.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa460600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa45815e7f0 con 0x7fa4580ccea0 2024-01-15T18:10:05.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa45d200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4580cef30 0x7fa458159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:05.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa45dc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4580ccea0 0x7fa45815e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:05.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa45e600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa4581468d0 0x7fa45813b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:05.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa45d200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4580cef30 0x7fa458159c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55610/0 (socket says 172.21.15.27:55610) 2024-01-15T18:10:05.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa45dc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4580ccea0 0x7fa45815e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:41428/0 (socket says 172.21.15.27:41428) 2024-01-15T18:10:05.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa45d200640 1 -- 172.21.15.27:0/1056545143 learned_addr learned my addr 172.21.15.27:0/1056545143 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:05.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa45e600640 1 -- 172.21.15.27:0/1056545143 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4580ccea0 msgr2=0x7fa45815e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:05.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa45e600640 1 --2- 172.21.15.27:0/1056545143 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4580ccea0 0x7fa45815e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:05.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa45e600640 1 -- 172.21.15.27:0/1056545143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4580cef30 msgr2=0x7fa458159c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:05.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa45e600640 1 --2- 172.21.15.27:0/1056545143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4580cef30 0x7fa458159c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:05.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa45e600640 1 -- 172.21.15.27:0/1056545143 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa450009b30 con 0x7fa4581468d0 2024-01-15T18:10:05.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa45dc00640 1 --2- 172.21.15.27:0/1056545143 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4580ccea0 0x7fa45815e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:10:05.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa45e600640 1 --2- 172.21.15.27:0/1056545143 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa4581468d0 0x7fa45813b450 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa4580cf9a0 tx=0x7fa450019480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:05.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa455600640 1 -- 172.21.15.27:0/1056545143 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4500a5560 con 0x7fa4581468d0 2024-01-15T18:10:05.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa455600640 1 -- 172.21.15.27:0/1056545143 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4500b8670 con 0x7fa4581468d0 2024-01-15T18:10:05.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa455600640 1 -- 172.21.15.27:0/1056545143 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4500b8a20 con 0x7fa4581468d0 2024-01-15T18:10:05.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa460600640 1 -- 172.21.15.27:0/1056545143 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa45815ac10 con 0x7fa4581468d0 2024-01-15T18:10:05.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa460600640 1 -- 172.21.15.27:0/1056545143 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa45815a930 con 0x7fa4581468d0 2024-01-15T18:10:05.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.505+0000 7fa460600640 1 -- 172.21.15.27:0/1056545143 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa45804e850 con 0x7fa4581468d0 2024-01-15T18:10:05.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.509+0000 7fa455600640 1 -- 172.21.15.27:0/1056545143 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 16) v1 ==== 86823+0+0 (secure 0 0 0) 0x7fa4500b8b80 con 0x7fa4581468d0 2024-01-15T18:10:05.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.509+0000 7fa455600640 1 --2- 172.21.15.27:0/1056545143 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa430067040 0x7fa430069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:05.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.509+0000 7fa455600640 1 -- 172.21.15.27:0/1056545143 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fa4501203d0 con 0x7fa4581468d0 2024-01-15T18:10:05.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.509+0000 7fa45dc00640 1 --2- 172.21.15.27:0/1056545143 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa430067040 0x7fa430069500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:05.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.509+0000 7fa45dc00640 1 --2- 172.21.15.27:0/1056545143 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa430067040 0x7fa430069500 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa448007d30 tx=0x7fa448008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:05.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.513+0000 7fa455600640 1 -- 172.21.15.27:0/1056545143 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa45011dd10 con 0x7fa4581468d0 2024-01-15T18:10:05.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.701+0000 7fa460600640 1 -- 172.21.15.27:0/1056545143 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone91816", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4580cf310 con 0x7fa430067040 2024-01-15T18:10:05.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.761+0000 7fa455600640 1 -- 172.21.15.27:0/1056545143 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa448007e80 con 0x7fa430067040 2024-01-15T18:10:05.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.765+0000 7fa460600640 1 -- 172.21.15.27:0/1056545143 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa430067040 msgr2=0x7fa430069500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:05.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.765+0000 7fa460600640 1 --2- 172.21.15.27:0/1056545143 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa430067040 0x7fa430069500 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa448007d30 tx=0x7fa448008040 comp rx=0 tx=0).stop 2024-01-15T18:10:05.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.765+0000 7fa460600640 1 -- 172.21.15.27:0/1056545143 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa4581468d0 msgr2=0x7fa45813b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:05.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.765+0000 7fa460600640 1 --2- 172.21.15.27:0/1056545143 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa4581468d0 0x7fa45813b450 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa4580cf9a0 tx=0x7fa450019480 comp rx=0 tx=0).stop 2024-01-15T18:10:05.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.765+0000 7fa460600640 1 -- 172.21.15.27:0/1056545143 shutdown_connections 2024-01-15T18:10:05.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.765+0000 7fa460600640 1 --2- 172.21.15.27:0/1056545143 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa430067040 0x7fa430069500 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:05.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.765+0000 7fa460600640 1 --2- 172.21.15.27:0/1056545143 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa4581468d0 0x7fa45813b450 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:05.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.765+0000 7fa460600640 1 --2- 172.21.15.27:0/1056545143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4580cef30 0x7fa458159c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:05.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.765+0000 7fa460600640 1 --2- 172.21.15.27:0/1056545143 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4580ccea0 0x7fa45815e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:05.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.765+0000 7fa460600640 1 -- 172.21.15.27:0/1056545143 >> 172.21.15.27:0/1056545143 conn(0x7fa4580cb700 msgr2=0x7fa4580ae860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:05.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.765+0000 7fa460600640 1 -- 172.21.15.27:0/1056545143 shutdown_connections 2024-01-15T18:10:05.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.765+0000 7fa460600640 1 -- 172.21.15.27:0/1056545143 wait complete. 2024-01-15T18:10:05.790 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone91817 2024-01-15T18:10:05.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.933+0000 7f6ad8200640 1 Processor -- start 2024-01-15T18:10:05.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.933+0000 7f6ad8200640 1 -- start start 2024-01-15T18:10:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.933+0000 7f6ad8200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6ad015b4b0 0x7f6ad00b1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.933+0000 7f6ad8200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad00d19a0 con 0x7f6ad00cf460 2024-01-15T18:10:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.933+0000 7f6ad8200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad00d1c80 con 0x7f6ad00cec60 2024-01-15T18:10:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.933+0000 7f6ad8200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ad00b07c0 con 0x7f6ad015b4b0 2024-01-15T18:10:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.933+0000 7f6ad4e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6ad015b4b0 0x7f6ad00b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.933+0000 7f6ad5800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6ad00cf460 0x7f6ad00cf840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45280/0 (socket says 172.21.15.27:45280) 2024-01-15T18:10:05.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.933+0000 7f6ad5800640 1 -- 172.21.15.27:0/2548883207 learned_addr learned my addr 172.21.15.27:0/2548883207 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:10:05.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.933+0000 7f6ad4e00640 1 -- 172.21.15.27:0/2548883207 >> v1:172.21.15.93:6789/0 conn(0x7f6ad00cec60 legacy=0x7f6ad00b1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:05.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.933+0000 7f6ad4e00640 1 -- 172.21.15.27:0/2548883207 >> v1:172.21.15.27:6789/0 conn(0x7f6ad00cf460 legacy=0x7f6ad00cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:05.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.933+0000 7f6ad4e00640 1 -- 172.21.15.27:0/2548883207 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad00b0aa0 con 0x7f6ad015b4b0 2024-01-15T18:10:05.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.933+0000 7f6acfe00640 1 -- 172.21.15.27:0/2548883207 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2818231427 0 0) 0x7f6ad00d19a0 con 0x7f6ad00cf460 2024-01-15T18:10:05.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.937+0000 7f6ad4e00640 1 --2- 172.21.15.27:0/2548883207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ad015b4b0 0x7f6ad00b1580 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f6ab80097c0 tx=0x7f6ab801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcb177b05b8eb943 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:05.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.937+0000 7f6acfe00640 1 -- 172.21.15.27:0/2548883207 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab80ab680 con 0x7f6ad015b4b0 2024-01-15T18:10:05.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.937+0000 7f6acfe00640 1 -- 172.21.15.27:0/2548883207 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab80ab7e0 con 0x7f6ad015b4b0 2024-01-15T18:10:05.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.937+0000 7f6acfe00640 1 -- 172.21.15.27:0/2548883207 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab80abb70 con 0x7f6ad015b4b0 2024-01-15T18:10:05.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.937+0000 7f6ad8200640 1 -- 172.21.15.27:0/2548883207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ad015b4b0 msgr2=0x7f6ad00b1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:05.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.937+0000 7f6ad8200640 1 --2- 172.21.15.27:0/2548883207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ad015b4b0 0x7f6ad00b1580 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f6ab80097c0 tx=0x7f6ab801c480 comp rx=0 tx=0).stop 2024-01-15T18:10:05.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.937+0000 7f6ad8200640 1 -- 172.21.15.27:0/2548883207 shutdown_connections 2024-01-15T18:10:05.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.937+0000 7f6ad8200640 1 --2- 172.21.15.27:0/2548883207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ad015b4b0 0x7f6ad00b1580 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:05.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.937+0000 7f6ad8200640 1 -- 172.21.15.27:0/2548883207 >> 172.21.15.27:0/2548883207 conn(0x7f6ad00d0e80 msgr2=0x7f6ad00d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:05.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.937+0000 7f6ad8200640 1 -- 172.21.15.27:0/2548883207 shutdown_connections 2024-01-15T18:10:05.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.937+0000 7f6ad8200640 1 -- 172.21.15.27:0/2548883207 wait complete. 2024-01-15T18:10:05.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6ad8200640 1 Processor -- start 2024-01-15T18:10:05.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6ad8200640 1 -- start start 2024-01-15T18:10:05.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6ad8200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6ad00cec60 0x7f6ad0149540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:05.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6ad8200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ad00cf460 0x7f6ad006c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:05.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6ad8200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6ad015b4b0 0x7f6ad00c6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:05.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6ad8200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad00c7f50 con 0x7f6ad00cf460 2024-01-15T18:10:05.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6ad8200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad00adc10 con 0x7f6ad00cec60 2024-01-15T18:10:05.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6ad8200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad00adef0 con 0x7f6ad015b4b0 2024-01-15T18:10:05.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6ad4e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ad00cf460 0x7f6ad006c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:05.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6ad5800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6ad00cec60 0x7f6ad0149540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:05.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6ad6200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6ad015b4b0 0x7f6ad00c6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:05.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6ad4e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ad00cf460 0x7f6ad006c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55632/0 (socket says 172.21.15.27:55632) 2024-01-15T18:10:05.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6ad5800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6ad00cec60 0x7f6ad0149540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:59630/0 (socket says 172.21.15.27:59630) 2024-01-15T18:10:05.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6ad4e00640 1 -- 172.21.15.27:0/3282782538 learned_addr learned my addr 172.21.15.27:0/3282782538 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:05.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6ad6200640 1 -- 172.21.15.27:0/3282782538 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6ad00cec60 msgr2=0x7f6ad0149540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:05.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6ad6200640 1 --2- 172.21.15.27:0/3282782538 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6ad00cec60 0x7f6ad0149540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:05.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6ad6200640 1 -- 172.21.15.27:0/3282782538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ad00cf460 msgr2=0x7f6ad006c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:05.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6ad6200640 1 --2- 172.21.15.27:0/3282782538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ad00cf460 0x7f6ad006c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:05.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6ad6200640 1 -- 172.21.15.27:0/3282782538 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab8009420 con 0x7f6ad015b4b0 2024-01-15T18:10:05.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6ad4e00640 1 --2- 172.21.15.27:0/3282782538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ad00cf460 0x7f6ad006c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:10:05.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6ad6200640 1 --2- 172.21.15.27:0/3282782538 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6ad015b4b0 0x7f6ad00c6560 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6ac800c700 tx=0x7f6ac800cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:05.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6acd600640 1 -- 172.21.15.27:0/3282782538 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8003b80 con 0x7f6ad015b4b0 2024-01-15T18:10:05.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6acd600640 1 -- 172.21.15.27:0/3282782538 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac8003ce0 con 0x7f6ad015b4b0 2024-01-15T18:10:05.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6ad8200640 1 -- 172.21.15.27:0/3282782538 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ad00ae1d0 con 0x7f6ad015b4b0 2024-01-15T18:10:05.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6acd600640 1 -- 172.21.15.27:0/3282782538 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8005310 con 0x7f6ad015b4b0 2024-01-15T18:10:05.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.941+0000 7f6ad8200640 1 -- 172.21.15.27:0/3282782538 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ad006c700 con 0x7f6ad015b4b0 2024-01-15T18:10:05.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.945+0000 7f6ad8200640 1 -- 172.21.15.27:0/3282782538 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ad006c920 con 0x7f6ad015b4b0 2024-01-15T18:10:05.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.945+0000 7f6acd600640 1 -- 172.21.15.27:0/3282782538 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 16) v1 ==== 86823+0+0 (secure 0 0 0) 0x7f6ac8005470 con 0x7f6ad015b4b0 2024-01-15T18:10:05.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.945+0000 7f6acd600640 1 --2- 172.21.15.27:0/3282782538 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6aa8067070 0x7f6aa8069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:05.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.945+0000 7f6acd600640 1 -- 172.21.15.27:0/3282782538 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f6ac80851c0 con 0x7f6ad015b4b0 2024-01-15T18:10:05.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.945+0000 7f6ad5800640 1 --2- 172.21.15.27:0/3282782538 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6aa8067070 0x7f6aa8069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:05.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.945+0000 7f6ad5800640 1 --2- 172.21.15.27:0/3282782538 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6aa8067070 0x7f6aa8069530 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0007a90 tx=0x7f6ac00079e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:05.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:05.949+0000 7f6acd600640 1 -- 172.21.15.27:0/3282782538 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6ac80b6280 con 0x7f6ad015b4b0 2024-01-15T18:10:06.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.137+0000 7f6ad8200640 1 -- 172.21.15.27:0/3282782538 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone91817", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ad004e850 con 0x7f6aa8067070 2024-01-15T18:10:06.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.205+0000 7f6acd600640 1 -- 172.21.15.27:0/3282782538 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6ad004e850 con 0x7f6aa8067070 2024-01-15T18:10:06.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.205+0000 7f6ad8200640 1 -- 172.21.15.27:0/3282782538 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6aa8067070 msgr2=0x7f6aa8069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:06.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.205+0000 7f6ad8200640 1 --2- 172.21.15.27:0/3282782538 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6aa8067070 0x7f6aa8069530 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0007a90 tx=0x7f6ac00079e0 comp rx=0 tx=0).stop 2024-01-15T18:10:06.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.205+0000 7f6ad8200640 1 -- 172.21.15.27:0/3282782538 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6ad015b4b0 msgr2=0x7f6ad00c6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:06.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.205+0000 7f6ad8200640 1 --2- 172.21.15.27:0/3282782538 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6ad015b4b0 0x7f6ad00c6560 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6ac800c700 tx=0x7f6ac800cbc0 comp rx=0 tx=0).stop 2024-01-15T18:10:06.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.205+0000 7f6ad8200640 1 -- 172.21.15.27:0/3282782538 shutdown_connections 2024-01-15T18:10:06.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.205+0000 7f6ad8200640 1 --2- 172.21.15.27:0/3282782538 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6aa8067070 0x7f6aa8069530 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:06.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.205+0000 7f6ad8200640 1 --2- 172.21.15.27:0/3282782538 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6ad015b4b0 0x7f6ad00c6560 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:06.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.205+0000 7f6ad8200640 1 --2- 172.21.15.27:0/3282782538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ad00cf460 0x7f6ad006c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:06.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.205+0000 7f6ad8200640 1 --2- 172.21.15.27:0/3282782538 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6ad00cec60 0x7f6ad0149540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:06.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.205+0000 7f6ad8200640 1 -- 172.21.15.27:0/3282782538 >> 172.21.15.27:0/3282782538 conn(0x7f6ad00d0e80 msgr2=0x7f6ad006d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:06.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.205+0000 7f6ad8200640 1 -- 172.21.15.27:0/3282782538 shutdown_connections 2024-01-15T18:10:06.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.205+0000 7f6ad8200640 1 -- 172.21.15.27:0/3282782538 wait complete. 2024-01-15T18:10:06.232 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-15T18:10:06.232 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone91811 2024-01-15T18:10:06.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.373+0000 7f9097400640 1 Processor -- start 2024-01-15T18:10:06.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.373+0000 7f9097400640 1 -- start start 2024-01-15T18:10:06.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.373+0000 7f9097400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f90900715f0 0x7f9090164a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:06.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.373+0000 7f9097400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909015a610 con 0x7f909015b100 2024-01-15T18:10:06.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.373+0000 7f9097400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909015e3c0 con 0x7f909015ad20 2024-01-15T18:10:06.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.373+0000 7f9097400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f909015e6a0 con 0x7f90900715f0 2024-01-15T18:10:06.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.373+0000 7f908fe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f909015b100 0x7f9090070ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45286/0 (socket says 172.21.15.27:45286) 2024-01-15T18:10:06.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.373+0000 7f9095400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f90900715f0 0x7f9090164a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:06.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.373+0000 7f908fe00640 1 -- 172.21.15.27:0/901714775 learned_addr learned my addr 172.21.15.27:0/901714775 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:10:06.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.373+0000 7f9095400640 1 -- 172.21.15.27:0/901714775 >> v1:172.21.15.93:6789/0 conn(0x7f909015ad20 legacy=0x7f909015d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:06.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.373+0000 7f9095400640 1 -- 172.21.15.27:0/901714775 >> v1:172.21.15.27:6789/0 conn(0x7f909015b100 legacy=0x7f9090070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:06.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.373+0000 7f9095400640 1 -- 172.21.15.27:0/901714775 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909015b6e0 con 0x7f90900715f0 2024-01-15T18:10:06.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.377+0000 7f9095400640 1 --2- 172.21.15.27:0/901714775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90900715f0 0x7f9090164a60 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f9088009dc0 tx=0x7f908801c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6c0a38ae7622fd4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:06.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.377+0000 7f908f400640 1 -- 172.21.15.27:0/901714775 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90880a5850 con 0x7f90900715f0 2024-01-15T18:10:06.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.377+0000 7f908f400640 1 -- 172.21.15.27:0/901714775 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90880a59b0 con 0x7f90900715f0 2024-01-15T18:10:06.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.377+0000 7f908f400640 1 -- 172.21.15.27:0/901714775 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90880a5d40 con 0x7f90900715f0 2024-01-15T18:10:06.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.377+0000 7f9097400640 1 -- 172.21.15.27:0/901714775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90900715f0 msgr2=0x7f9090164a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:06.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.377+0000 7f9097400640 1 --2- 172.21.15.27:0/901714775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90900715f0 0x7f9090164a60 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f9088009dc0 tx=0x7f908801c870 comp rx=0 tx=0).stop 2024-01-15T18:10:06.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.377+0000 7f9097400640 1 -- 172.21.15.27:0/901714775 shutdown_connections 2024-01-15T18:10:06.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.377+0000 7f9097400640 1 --2- 172.21.15.27:0/901714775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90900715f0 0x7f9090164a60 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:06.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.377+0000 7f9097400640 1 -- 172.21.15.27:0/901714775 >> 172.21.15.27:0/901714775 conn(0x7f90900cb700 msgr2=0x7f90900cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:06.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.377+0000 7f9097400640 1 -- 172.21.15.27:0/901714775 shutdown_connections 2024-01-15T18:10:06.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.377+0000 7f9097400640 1 -- 172.21.15.27:0/901714775 wait complete. 2024-01-15T18:10:06.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.381+0000 7f9097400640 1 Processor -- start 2024-01-15T18:10:06.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.381+0000 7f9097400640 1 -- start start 2024-01-15T18:10:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.381+0000 7f9097400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f90900715f0 0x7f909013c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.381+0000 7f9097400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909015ad20 0x7f9090151b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.381+0000 7f9097400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f909015b100 0x7f909014b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.381+0000 7f9097400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90900b3e50 con 0x7f909015ad20 2024-01-15T18:10:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.381+0000 7f9097400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90900b4130 con 0x7f90900715f0 2024-01-15T18:10:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.381+0000 7f9097400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90900b4410 con 0x7f909015b100 2024-01-15T18:10:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.381+0000 7f908fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909015ad20 0x7f9090151b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.381+0000 7f908fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909015ad20 0x7f9090151b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55646/0 (socket says 172.21.15.27:55646) 2024-01-15T18:10:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.381+0000 7f908fe00640 1 -- 172.21.15.27:0/3346348528 learned_addr learned my addr 172.21.15.27:0/3346348528 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.381+0000 7f9095400640 1 --2- 172.21.15.27:0/3346348528 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f909015b100 0x7f909014b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.381+0000 7f9094a00640 1 --2- 172.21.15.27:0/3346348528 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f90900715f0 0x7f909013c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:06.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.381+0000 7f908fe00640 1 -- 172.21.15.27:0/3346348528 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f909015b100 msgr2=0x7f909014b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:06.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.381+0000 7f908fe00640 1 --2- 172.21.15.27:0/3346348528 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f909015b100 0x7f909014b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:06.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.381+0000 7f908fe00640 1 -- 172.21.15.27:0/3346348528 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f90900715f0 msgr2=0x7f909013c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:06.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.381+0000 7f908fe00640 1 --2- 172.21.15.27:0/3346348528 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f90900715f0 0x7f909013c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:06.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.381+0000 7f908fe00640 1 -- 172.21.15.27:0/3346348528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9088009a70 con 0x7f909015ad20 2024-01-15T18:10:06.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.381+0000 7f908fe00640 1 --2- 172.21.15.27:0/3346348528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909015ad20 0x7f9090151b60 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f907800c400 tx=0x7f907800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:06.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.381+0000 7f908cc00640 1 -- 172.21.15.27:0/3346348528 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9078005cf0 con 0x7f909015ad20 2024-01-15T18:10:06.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.381+0000 7f908cc00640 1 -- 172.21.15.27:0/3346348528 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9078005e50 con 0x7f909015ad20 2024-01-15T18:10:06.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.381+0000 7f908cc00640 1 -- 172.21.15.27:0/3346348528 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90780072d0 con 0x7f909015ad20 2024-01-15T18:10:06.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.381+0000 7f9097400640 1 -- 172.21.15.27:0/3346348528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90900b2790 con 0x7f909015ad20 2024-01-15T18:10:06.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.381+0000 7f9097400640 1 -- 172.21.15.27:0/3346348528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9090149840 con 0x7f909015ad20 2024-01-15T18:10:06.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.381+0000 7f9097400640 1 -- 172.21.15.27:0/3346348528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90900b29d0 con 0x7f909015ad20 2024-01-15T18:10:06.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.385+0000 7f908cc00640 1 -- 172.21.15.27:0/3346348528 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 86823+0+0 (secure 0 0 0) 0x7f9078007430 con 0x7f909015ad20 2024-01-15T18:10:06.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.385+0000 7f908cc00640 1 --2- 172.21.15.27:0/3346348528 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9068067040 0x7f9068069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:06.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.385+0000 7f908cc00640 1 -- 172.21.15.27:0/3346348528 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f9078084a60 con 0x7f909015ad20 2024-01-15T18:10:06.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.385+0000 7f9094a00640 1 --2- 172.21.15.27:0/3346348528 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9068067040 0x7f9068069500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:06.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.385+0000 7f9094a00640 1 --2- 172.21.15.27:0/3346348528 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9068067040 0x7f9068069500 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9080007fd0 tx=0x7f9080009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:06.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.389+0000 7f908cc00640 1 -- 172.21.15.27:0/3346348528 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f90900b29d0 con 0x7f909015ad20 2024-01-15T18:10:06.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.577+0000 7f9097400640 1 -- 172.21.15.27:0/3346348528 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone91811", "target": ["mon-mgr", ""]}) v1 -- 0x7f9090072090 con 0x7f9068067040 2024-01-15T18:10:06.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.589+0000 7f908cc00640 1 -- 172.21.15.27:0/3346348528 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f9090072090 con 0x7f9068067040 2024-01-15T18:10:06.594 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:10:06.594 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:10:06.594 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:10:06.594 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:10:06.594 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:10:06.594 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:10:06.594 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:10:06.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:06.595 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:10:06.595 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 0.0, 2024-01-15T18:10:06.595 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "0 B/3 GB", 2024-01-15T18:10:06.595 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "0/3", 2024-01-15T18:10:06.595 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:10:06.595 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:10:06.595 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:06.595 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:06.595 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:10:06.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.589+0000 7f9097400640 1 -- 172.21.15.27:0/3346348528 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9068067040 msgr2=0x7f9068069500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:06.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.589+0000 7f9097400640 1 --2- 172.21.15.27:0/3346348528 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9068067040 0x7f9068069500 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9080007fd0 tx=0x7f9080009040 comp rx=0 tx=0).stop 2024-01-15T18:10:06.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.589+0000 7f9097400640 1 -- 172.21.15.27:0/3346348528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909015ad20 msgr2=0x7f9090151b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:06.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.589+0000 7f9097400640 1 --2- 172.21.15.27:0/3346348528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909015ad20 0x7f9090151b60 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f907800c400 tx=0x7f907800c8c0 comp rx=0 tx=0).stop 2024-01-15T18:10:06.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.589+0000 7f9097400640 1 -- 172.21.15.27:0/3346348528 shutdown_connections 2024-01-15T18:10:06.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.589+0000 7f9097400640 1 --2- 172.21.15.27:0/3346348528 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9068067040 0x7f9068069500 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:06.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.589+0000 7f9097400640 1 --2- 172.21.15.27:0/3346348528 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f909015b100 0x7f909014b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:06.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.589+0000 7f9097400640 1 --2- 172.21.15.27:0/3346348528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f909015ad20 0x7f9090151b60 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:06.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.589+0000 7f9097400640 1 --2- 172.21.15.27:0/3346348528 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f90900715f0 0x7f909013c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:06.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.589+0000 7f9097400640 1 -- 172.21.15.27:0/3346348528 >> 172.21.15.27:0/3346348528 conn(0x7f90900cb700 msgr2=0x7f90901509c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:06.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.589+0000 7f9097400640 1 -- 172.21.15.27:0/3346348528 shutdown_connections 2024-01-15T18:10:06.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.589+0000 7f9097400640 1 -- 172.21.15.27:0/3346348528 wait complete. 2024-01-15T18:10:06.616 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-15T18:10:06.616 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone91817 2024-01-15T18:10:06.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.753+0000 7f174ac00640 1 Processor -- start 2024-01-15T18:10:06.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.753+0000 7f174ac00640 1 -- start start 2024-01-15T18:10:06.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.753+0000 7f174ac00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f17440cca40 0x7f17440d19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:06.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.753+0000 7f174ac00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f174415b6b0 con 0x7f17440cef20 2024-01-15T18:10:06.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.753+0000 7f174ac00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17440cfdb0 con 0x7f17440ce2c0 2024-01-15T18:10:06.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.753+0000 7f174ac00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17440cb9e0 con 0x7f17440cca40 2024-01-15T18:10:06.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.757+0000 7f1743e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f17440cef20 0x7f17440cf300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45292/0 (socket says 172.21.15.27:45292) 2024-01-15T18:10:06.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.757+0000 7f1748c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f17440cca40 0x7f17440d19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:06.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.757+0000 7f1743e00640 1 -- 172.21.15.27:0/1663013986 learned_addr learned my addr 172.21.15.27:0/1663013986 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:10:06.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.757+0000 7f1748c00640 1 -- 172.21.15.27:0/1663013986 >> v1:172.21.15.93:6789/0 conn(0x7f17440ce2c0 legacy=0x7f17440b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:06.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.757+0000 7f1748c00640 1 -- 172.21.15.27:0/1663013986 >> v1:172.21.15.27:6789/0 conn(0x7f17440cef20 legacy=0x7f17440cf300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:06.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.757+0000 7f1742a00640 1 -- 172.21.15.27:0/1663013986 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1040047596 0 0) 0x7f174415b6b0 con 0x7f17440cef20 2024-01-15T18:10:06.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.757+0000 7f1748c00640 1 -- 172.21.15.27:0/1663013986 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17440cbc80 con 0x7f17440cca40 2024-01-15T18:10:06.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.757+0000 7f1748c00640 1 --2- 172.21.15.27:0/1663013986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17440cca40 0x7f17440d19f0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f173c009d90 tx=0x7f173c01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=161da6af10de13ec server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:06.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.757+0000 7f1742a00640 1 -- 172.21.15.27:0/1663013986 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c0a5900 con 0x7f17440cca40 2024-01-15T18:10:06.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.757+0000 7f1742a00640 1 -- 172.21.15.27:0/1663013986 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173c0a5a60 con 0x7f17440cca40 2024-01-15T18:10:06.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.757+0000 7f1742a00640 1 -- 172.21.15.27:0/1663013986 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c0a5df0 con 0x7f17440cca40 2024-01-15T18:10:06.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.761+0000 7f174ac00640 1 -- 172.21.15.27:0/1663013986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17440cca40 msgr2=0x7f17440d19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:06.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.761+0000 7f174ac00640 1 --2- 172.21.15.27:0/1663013986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17440cca40 0x7f17440d19f0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f173c009d90 tx=0x7f173c01c950 comp rx=0 tx=0).stop 2024-01-15T18:10:06.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.761+0000 7f174ac00640 1 -- 172.21.15.27:0/1663013986 shutdown_connections 2024-01-15T18:10:06.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.761+0000 7f174ac00640 1 --2- 172.21.15.27:0/1663013986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17440cca40 0x7f17440d19f0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:06.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.761+0000 7f174ac00640 1 -- 172.21.15.27:0/1663013986 >> 172.21.15.27:0/1663013986 conn(0x7f17440d0ed0 msgr2=0x7f17440d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:06.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.761+0000 7f174ac00640 1 -- 172.21.15.27:0/1663013986 shutdown_connections 2024-01-15T18:10:06.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.761+0000 7f174ac00640 1 -- 172.21.15.27:0/1663013986 wait complete. 2024-01-15T18:10:06.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.761+0000 7f174ac00640 1 Processor -- start 2024-01-15T18:10:06.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.761+0000 7f174ac00640 1 -- start start 2024-01-15T18:10:06.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.761+0000 7f174ac00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17440cca40 0x7f17440c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:06.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.761+0000 7f174ac00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17440ce2c0 0x7f1744165950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:06.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.761+0000 7f174ac00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17440cef20 0x7f1744165e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:06.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.761+0000 7f174ac00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17441663d0 con 0x7f17440cca40 2024-01-15T18:10:06.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.761+0000 7f174ac00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17441648b0 con 0x7f17440cef20 2024-01-15T18:10:06.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.761+0000 7f174ac00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1744164b90 con 0x7f17440ce2c0 2024-01-15T18:10:06.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.761+0000 7f1743e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17440cca40 0x7f17440c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:06.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.761+0000 7f1743e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17440cca40 0x7f17440c87d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55666/0 (socket says 172.21.15.27:55666) 2024-01-15T18:10:06.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.761+0000 7f1743e00640 1 -- 172.21.15.27:0/793815623 learned_addr learned my addr 172.21.15.27:0/793815623 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:06.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.765+0000 7f1748c00640 1 --2- 172.21.15.27:0/793815623 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17440cef20 0x7f1744165e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:06.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.765+0000 7f1743400640 1 --2- 172.21.15.27:0/793815623 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17440ce2c0 0x7f1744165950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:06.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.765+0000 7f1743e00640 1 -- 172.21.15.27:0/793815623 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17440ce2c0 msgr2=0x7f1744165950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:06.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.765+0000 7f1743e00640 1 --2- 172.21.15.27:0/793815623 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17440ce2c0 0x7f1744165950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:06.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.765+0000 7f1743e00640 1 -- 172.21.15.27:0/793815623 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17440cef20 msgr2=0x7f1744165e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:06.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.765+0000 7f1743e00640 1 --2- 172.21.15.27:0/793815623 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17440cef20 0x7f1744165e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:06.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.765+0000 7f1743e00640 1 -- 172.21.15.27:0/793815623 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f173c009a70 con 0x7f17440cca40 2024-01-15T18:10:06.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.765+0000 7f1748c00640 1 --2- 172.21.15.27:0/793815623 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17440cef20 0x7f1744165e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:10:06.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.765+0000 7f1743e00640 1 --2- 172.21.15.27:0/793815623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17440cca40 0x7f17440c87d0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f173400c410 tx=0x7f173400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:06.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.765+0000 7f173be00640 1 -- 172.21.15.27:0/793815623 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17340074e0 con 0x7f17440cca40 2024-01-15T18:10:06.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.765+0000 7f173be00640 1 -- 172.21.15.27:0/793815623 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1734007640 con 0x7f17440cca40 2024-01-15T18:10:06.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.765+0000 7f174ac00640 1 -- 172.21.15.27:0/793815623 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1744164e70 con 0x7f17440cca40 2024-01-15T18:10:06.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.765+0000 7f173be00640 1 -- 172.21.15.27:0/793815623 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1734004a30 con 0x7f17440cca40 2024-01-15T18:10:06.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.765+0000 7f174ac00640 1 -- 172.21.15.27:0/793815623 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1744165430 con 0x7f17440cca40 2024-01-15T18:10:06.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.765+0000 7f174ac00640 1 -- 172.21.15.27:0/793815623 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17440d09f0 con 0x7f17440cca40 2024-01-15T18:10:06.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.765+0000 7f173be00640 1 -- 172.21.15.27:0/793815623 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 86823+0+0 (secure 0 0 0) 0x7f1734004b90 con 0x7f17440cca40 2024-01-15T18:10:06.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.765+0000 7f173be00640 1 --2- 172.21.15.27:0/793815623 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f171c067070 0x7f171c069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:06.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.769+0000 7f173be00640 1 -- 172.21.15.27:0/793815623 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f1734085250 con 0x7f17440cca40 2024-01-15T18:10:06.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.769+0000 7f1743400640 1 --2- 172.21.15.27:0/793815623 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f171c067070 0x7f171c069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:06.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.769+0000 7f1743400640 1 --2- 172.21.15.27:0/793815623 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f171c067070 0x7f171c069530 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f17441f9770 tx=0x7f172c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:06.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.769+0000 7f173be00640 1 -- 172.21.15.27:0/793815623 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f17340826f0 con 0x7f17440cca40 2024-01-15T18:10:06.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.961+0000 7f174ac00640 1 -- 172.21.15.27:0/793815623 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone91817", "target": ["mon-mgr", ""]}) v1 -- 0x7f174404e850 con 0x7f171c067070 2024-01-15T18:10:06.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.969+0000 7f173be00640 1 -- 172.21.15.27:0/793815623 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f174404e850 con 0x7f171c067070 2024-01-15T18:10:06.974 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:10:06.974 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:10:06.974 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "pending", 2024-01-15T18:10:06.974 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:10:06.974 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:10:06.974 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:10:06.974 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:10:06.974 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:06.974 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:06.974 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:10:06.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.969+0000 7f174ac00640 1 -- 172.21.15.27:0/793815623 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f171c067070 msgr2=0x7f171c069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:06.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.969+0000 7f174ac00640 1 --2- 172.21.15.27:0/793815623 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f171c067070 0x7f171c069530 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f17441f9770 tx=0x7f172c009040 comp rx=0 tx=0).stop 2024-01-15T18:10:06.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.969+0000 7f174ac00640 1 -- 172.21.15.27:0/793815623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17440cca40 msgr2=0x7f17440c87d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:06.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.969+0000 7f174ac00640 1 --2- 172.21.15.27:0/793815623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17440cca40 0x7f17440c87d0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f173400c410 tx=0x7f173400c8d0 comp rx=0 tx=0).stop 2024-01-15T18:10:06.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.969+0000 7f174ac00640 1 -- 172.21.15.27:0/793815623 shutdown_connections 2024-01-15T18:10:06.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.969+0000 7f174ac00640 1 --2- 172.21.15.27:0/793815623 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f171c067070 0x7f171c069530 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:06.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.969+0000 7f174ac00640 1 --2- 172.21.15.27:0/793815623 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17440cef20 0x7f1744165e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:06.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.969+0000 7f174ac00640 1 --2- 172.21.15.27:0/793815623 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17440ce2c0 0x7f1744165950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:06.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.969+0000 7f174ac00640 1 --2- 172.21.15.27:0/793815623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17440cca40 0x7f17440c87d0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:06.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.969+0000 7f174ac00640 1 -- 172.21.15.27:0/793815623 >> 172.21.15.27:0/793815623 conn(0x7f17440d0ed0 msgr2=0x7f174406ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:06.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.969+0000 7f174ac00640 1 -- 172.21.15.27:0/793815623 shutdown_connections 2024-01-15T18:10:06.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:06.969+0000 7f174ac00640 1 -- 172.21.15.27:0/793815623 wait complete. 2024-01-15T18:10:06.996 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:10:07.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.129+0000 7fa361800640 1 Processor -- start 2024-01-15T18:10:07.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.133+0000 7fa361800640 1 -- start start 2024-01-15T18:10:07.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.133+0000 7fa361800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa35c0cdf90 0x7fa35c168450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:07.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.133+0000 7fa361800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35c0b2770 con 0x7fa35c0b2390 2024-01-15T18:10:07.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.133+0000 7fa361800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35c06c060 con 0x7fa35c06bb80 2024-01-15T18:10:07.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.133+0000 7fa361800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa35c06c340 con 0x7fa35c0cdf90 2024-01-15T18:10:07.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.133+0000 7fa35aa00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa35c0b2390 0x7fa35c0cf780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45308/0 (socket says 172.21.15.27:45308) 2024-01-15T18:10:07.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.133+0000 7fa35aa00640 1 -- 172.21.15.27:0/812217685 learned_addr learned my addr 172.21.15.27:0/812217685 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:10:07.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.133+0000 7fa35b400640 1 --2- 172.21.15.27:0/812217685 >> v2:172.21.15.93:3300/0 conn(0x7fa35c0cdf90 0x7fa35c168450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:07.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.133+0000 7fa359600640 1 -- 172.21.15.27:0/812217685 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 678377224 0 0) 0x7fa35c0b2770 con 0x7fa35c0b2390 2024-01-15T18:10:07.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.133+0000 7fa359600640 1 -- 172.21.15.27:0/812217685 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa33c003530 con 0x7fa35c0b2390 2024-01-15T18:10:07.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.133+0000 7fa359600640 1 -- 172.21.15.27:0/812217685 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1934484767 0 0) 0x7fa35c06c060 con 0x7fa35c06bb80 2024-01-15T18:10:07.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.133+0000 7fa359600640 1 -- 172.21.15.27:0/812217685 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa35c0b2770 con 0x7fa35c06bb80 2024-01-15T18:10:07.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.133+0000 7fa35b400640 1 -- 172.21.15.27:0/812217685 >> v1:172.21.15.93:6789/0 conn(0x7fa35c06bb80 legacy=0x7fa35c0cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:07.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.133+0000 7fa35b400640 1 -- 172.21.15.27:0/812217685 >> v1:172.21.15.27:6789/0 conn(0x7fa35c0b2390 legacy=0x7fa35c0cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:07.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.133+0000 7fa35b400640 1 -- 172.21.15.27:0/812217685 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35c06c620 con 0x7fa35c0cdf90 2024-01-15T18:10:07.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.137+0000 7fa35b400640 1 --2- 172.21.15.27:0/812217685 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa35c0cdf90 0x7fa35c168450 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa354009d90 tx=0x7fa35401c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38f0b34aa153f263 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:07.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.137+0000 7fa359600640 1 -- 172.21.15.27:0/812217685 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3540a59e0 con 0x7fa35c0cdf90 2024-01-15T18:10:07.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.137+0000 7fa359600640 1 -- 172.21.15.27:0/812217685 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3540a5b40 con 0x7fa35c0cdf90 2024-01-15T18:10:07.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.137+0000 7fa359600640 1 -- 172.21.15.27:0/812217685 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3540ad440 con 0x7fa35c0cdf90 2024-01-15T18:10:07.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.137+0000 7fa361800640 1 -- 172.21.15.27:0/812217685 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa35c0cdf90 msgr2=0x7fa35c168450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:07.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.137+0000 7fa361800640 1 --2- 172.21.15.27:0/812217685 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa35c0cdf90 0x7fa35c168450 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa354009d90 tx=0x7fa35401c920 comp rx=0 tx=0).stop 2024-01-15T18:10:07.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.137+0000 7fa361800640 1 -- 172.21.15.27:0/812217685 shutdown_connections 2024-01-15T18:10:07.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.137+0000 7fa361800640 1 --2- 172.21.15.27:0/812217685 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa35c0cdf90 0x7fa35c168450 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:07.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.137+0000 7fa361800640 1 -- 172.21.15.27:0/812217685 >> 172.21.15.27:0/812217685 conn(0x7fa35c0b2a10 msgr2=0x7fa35c0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:07.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.137+0000 7fa361800640 1 -- 172.21.15.27:0/812217685 shutdown_connections 2024-01-15T18:10:07.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.137+0000 7fa361800640 1 -- 172.21.15.27:0/812217685 wait complete. 2024-01-15T18:10:07.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa361800640 1 Processor -- start 2024-01-15T18:10:07.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa361800640 1 -- start start 2024-01-15T18:10:07.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa361800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa35c06bb80 0x7fa35c158050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:07.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa361800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa35c0b2390 0x7fa35c156860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:07.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa361800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa35c0cdf90 0x7fa35c14eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:07.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa361800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa35c155580 con 0x7fa35c0b2390 2024-01-15T18:10:07.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa361800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa35c156da0 con 0x7fa35c0cdf90 2024-01-15T18:10:07.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa361800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa35c14f400 con 0x7fa35c06bb80 2024-01-15T18:10:07.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa35a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa35c0b2390 0x7fa35c156860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:07.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa35b400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa35c0cdf90 0x7fa35c14eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:07.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa35a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa35c0b2390 0x7fa35c156860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55668/0 (socket says 172.21.15.27:55668) 2024-01-15T18:10:07.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa35a000640 1 -- 172.21.15.27:0/3771015984 learned_addr learned my addr 172.21.15.27:0/3771015984 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:07.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa35aa00640 1 --2- 172.21.15.27:0/3771015984 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa35c06bb80 0x7fa35c158050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:07.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa35a000640 1 -- 172.21.15.27:0/3771015984 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa35c06bb80 msgr2=0x7fa35c158050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:07.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa35a000640 1 --2- 172.21.15.27:0/3771015984 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa35c06bb80 0x7fa35c158050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:07.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa35a000640 1 -- 172.21.15.27:0/3771015984 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa35c0cdf90 msgr2=0x7fa35c14eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:07.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa35a000640 1 --2- 172.21.15.27:0/3771015984 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa35c0cdf90 0x7fa35c14eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:07.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa35a000640 1 -- 172.21.15.27:0/3771015984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa354009a70 con 0x7fa35c0b2390 2024-01-15T18:10:07.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa35b400640 1 --2- 172.21.15.27:0/3771015984 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa35c0cdf90 0x7fa35c14eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:10:07.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa35a000640 1 --2- 172.21.15.27:0/3771015984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa35c0b2390 0x7fa35c156860 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fa34400c4c0 tx=0x7fa34400c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:07.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa352a00640 1 -- 172.21.15.27:0/3771015984 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3440052d0 con 0x7fa35c0b2390 2024-01-15T18:10:07.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa352a00640 1 -- 172.21.15.27:0/3771015984 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa344005430 con 0x7fa35c0b2390 2024-01-15T18:10:07.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa352a00640 1 -- 172.21.15.27:0/3771015984 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa344012870 con 0x7fa35c0b2390 2024-01-15T18:10:07.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa361800640 1 -- 172.21.15.27:0/3771015984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa35c156fb0 con 0x7fa35c0b2390 2024-01-15T18:10:07.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa361800640 1 -- 172.21.15.27:0/3771015984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa35c157570 con 0x7fa35c0b2390 2024-01-15T18:10:07.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.141+0000 7fa361800640 1 -- 172.21.15.27:0/3771015984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa35c04e850 con 0x7fa35c0b2390 2024-01-15T18:10:07.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.145+0000 7fa352a00640 1 -- 172.21.15.27:0/3771015984 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 86823+0+0 (secure 0 0 0) 0x7fa3440129d0 con 0x7fa35c0b2390 2024-01-15T18:10:07.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.145+0000 7fa352a00640 1 --2- 172.21.15.27:0/3771015984 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa334067040 0x7fa334069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:07.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.145+0000 7fa352a00640 1 -- 172.21.15.27:0/3771015984 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fa344085010 con 0x7fa35c0b2390 2024-01-15T18:10:07.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.145+0000 7fa35aa00640 1 --2- 172.21.15.27:0/3771015984 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa334067040 0x7fa334069500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:07.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.145+0000 7fa35aa00640 1 --2- 172.21.15.27:0/3771015984 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa334067040 0x7fa334069500 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa34c0011f0 tx=0x7fa34c007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:07.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.149+0000 7fa352a00640 1 -- 172.21.15.27:0/3771015984 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa344082540 con 0x7fa35c0b2390 2024-01-15T18:10:07.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.361+0000 7fa361800640 1 -- 172.21.15.27:0/3771015984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa35c06afc0 con 0x7fa35c0b2390 2024-01-15T18:10:07.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.361+0000 7fa352a00640 1 -- 172.21.15.27:0/3771015984 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2337 (secure 0 0 0) 0x7fa344051430 con 0x7fa35c0b2390 2024-01-15T18:10:07.368 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:10:07.368 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:10:07.368 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:10:07.368 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:10:07.368 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:10:07.368 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:10:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:10:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:10:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:10:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:10:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:10:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:10:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:10:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:10:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:10:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:10:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 334, 2024-01-15T18:10:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:10:07.369 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:10:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:10:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:10:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:10:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 58, 2024-01-15T18:10:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:10:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:10:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:10:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:10:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:10:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:10:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:10:07.370 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:10:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:10:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:10:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:10:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:10:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 800, 2024-01-15T18:10:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 3222190772, 2024-01-15T18:10:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 6743728128, 2024-01-15T18:10:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 766350385152, 2024-01-15T18:10:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:10:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 511, 2024-01-15T18:10:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 18682, 2024-01-15T18:10:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 0, 2024-01-15T18:10:07.371 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 5 2024-01-15T18:10:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:10:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 38, 2024-01-15T18:10:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4, 2024-01-15T18:10:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:10:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:10:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:10:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:10:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 4, 2024-01-15T18:10:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:10:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:10:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:10:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 6480 2024-01-15T18:10:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:07.372 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:10:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:10:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:10:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:10:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:10:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:10:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:10:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:10:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:10:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 11, 2024-01-15T18:10:07.373 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:08:28.191482+0000", 2024-01-15T18:10:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:10:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: "268d93d2-2eea-412c-b804-608e15344ce7": { 2024-01-15T18:10:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "5 ongoing+pending cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-15T18:10:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0, 2024-01-15T18:10:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: "9a27fb80-350b-410b-a65e-91a770cccfdf": { 2024-01-15T18:10:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "2 ongoing cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-15T18:10:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0, 2024-01-15T18:10:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:07.374 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:07.374 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:10:07.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.365+0000 7fa361800640 1 -- 172.21.15.27:0/3771015984 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa334067040 msgr2=0x7fa334069500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:07.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.365+0000 7fa361800640 1 --2- 172.21.15.27:0/3771015984 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa334067040 0x7fa334069500 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa34c0011f0 tx=0x7fa34c007d40 comp rx=0 tx=0).stop 2024-01-15T18:10:07.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.365+0000 7fa361800640 1 -- 172.21.15.27:0/3771015984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa35c0b2390 msgr2=0x7fa35c156860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:07.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.365+0000 7fa361800640 1 --2- 172.21.15.27:0/3771015984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa35c0b2390 0x7fa35c156860 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fa34400c4c0 tx=0x7fa34400c980 comp rx=0 tx=0).stop 2024-01-15T18:10:07.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.365+0000 7fa361800640 1 -- 172.21.15.27:0/3771015984 shutdown_connections 2024-01-15T18:10:07.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.365+0000 7fa361800640 1 --2- 172.21.15.27:0/3771015984 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa334067040 0x7fa334069500 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:07.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.365+0000 7fa361800640 1 --2- 172.21.15.27:0/3771015984 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa35c0cdf90 0x7fa35c14eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:07.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.365+0000 7fa361800640 1 --2- 172.21.15.27:0/3771015984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa35c0b2390 0x7fa35c156860 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:07.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.365+0000 7fa361800640 1 --2- 172.21.15.27:0/3771015984 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa35c06bb80 0x7fa35c158050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:07.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.365+0000 7fa361800640 1 -- 172.21.15.27:0/3771015984 >> 172.21.15.27:0/3771015984 conn(0x7fa35c0b2a10 msgr2=0x7fa35c0addd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:07.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.365+0000 7fa361800640 1 -- 172.21.15.27:0/3771015984 shutdown_connections 2024-01-15T18:10:07.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:07.365+0000 7fa361800640 1 -- 172.21.15.27:0/3771015984 wait complete. 2024-01-15T18:10:08.391 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:10:08.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.657+0000 7fb3b6e00640 1 Processor -- start 2024-01-15T18:10:08.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.657+0000 7fb3b6e00640 1 -- start start 2024-01-15T18:10:08.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.661+0000 7fb3b6e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb3b00b39b0 0x7fb3b00711e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:08.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.661+0000 7fb3b6e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fb3b00ac000 0x7fb3b0071720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:08.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.661+0000 7fb3b6e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b013da40 con 0x7fb3b015a100 2024-01-15T18:10:08.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.661+0000 7fb3b6e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3b00ac850 con 0x7fb3b00ac000 2024-01-15T18:10:08.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.661+0000 7fb3b6e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b00ac570 con 0x7fb3b00b39b0 2024-01-15T18:10:08.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.661+0000 7fb3af400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb3b00b39b0 0x7fb3b00711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:08.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.661+0000 7fb3af400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb3b00b39b0 0x7fb3b00711e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:41474/0 (socket says 172.21.15.27:41474) 2024-01-15T18:10:08.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.661+0000 7fb3af400640 1 -- 172.21.15.27:0/2185258823 learned_addr learned my addr 172.21.15.27:0/2185258823 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:08.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.661+0000 7fb3af400640 1 -- 172.21.15.27:0/2185258823 >> v2:172.21.15.93:3300/0 conn(0x7fb3b00ac000 msgr2=0x7fb3b0071720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:08.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.661+0000 7fb3af400640 1 --2- 172.21.15.27:0/2185258823 >> v2:172.21.15.93:3300/0 conn(0x7fb3b00ac000 0x7fb3b0071720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:08.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.661+0000 7fb3af400640 1 -- 172.21.15.27:0/2185258823 >> v1:172.21.15.93:6789/0 conn(0x7fb3b015a100 legacy=0x7fb3b00b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:08.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.661+0000 7fb3af400640 1 -- 172.21.15.27:0/2185258823 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b0070700 con 0x7fb3b00b39b0 2024-01-15T18:10:08.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.669+0000 7fb3af400640 1 --2- 172.21.15.27:0/2185258823 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb3b00b39b0 0x7fb3b00711e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb398009870 tx=0x7fb39801c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3697b24f684e0a1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:08.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.669+0000 7fb3aea00640 1 -- 172.21.15.27:0/2185258823 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3980aa5f0 con 0x7fb3b00b39b0 2024-01-15T18:10:08.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.669+0000 7fb3aea00640 1 -- 172.21.15.27:0/2185258823 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3980aa750 con 0x7fb3b00b39b0 2024-01-15T18:10:08.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.669+0000 7fb3aea00640 1 -- 172.21.15.27:0/2185258823 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3980aaae0 con 0x7fb3b00b39b0 2024-01-15T18:10:08.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.673+0000 7fb3b6e00640 1 -- 172.21.15.27:0/2185258823 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb3b00b39b0 msgr2=0x7fb3b00711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:08.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.673+0000 7fb3b6e00640 1 --2- 172.21.15.27:0/2185258823 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb3b00b39b0 0x7fb3b00711e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb398009870 tx=0x7fb39801c4d0 comp rx=0 tx=0).stop 2024-01-15T18:10:08.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.673+0000 7fb3b6e00640 1 -- 172.21.15.27:0/2185258823 shutdown_connections 2024-01-15T18:10:08.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.673+0000 7fb3b6e00640 1 --2- 172.21.15.27:0/2185258823 >> v2:172.21.15.93:3300/0 conn(0x7fb3b00ac000 0x7fb3b0071720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:08.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.673+0000 7fb3b6e00640 1 --2- 172.21.15.27:0/2185258823 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb3b00b39b0 0x7fb3b00711e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:08.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.673+0000 7fb3b6e00640 1 -- 172.21.15.27:0/2185258823 >> 172.21.15.27:0/2185258823 conn(0x7fb3b00b0d90 msgr2=0x7fb3b00b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:08.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.673+0000 7fb3b6e00640 1 -- 172.21.15.27:0/2185258823 shutdown_connections 2024-01-15T18:10:08.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.673+0000 7fb3b6e00640 1 -- 172.21.15.27:0/2185258823 wait complete. 2024-01-15T18:10:08.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.673+0000 7fb3b6e00640 1 Processor -- start 2024-01-15T18:10:08.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.673+0000 7fb3b6e00640 1 -- start start 2024-01-15T18:10:08.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.673+0000 7fb3b6e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3b00ac000 0x7fb3b00cd450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:08.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.673+0000 7fb3b6e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb3b00b39b0 0x7fb3b015cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:08.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.673+0000 7fb3b6e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb3b015a100 0x7fb3b00cba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:08.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.673+0000 7fb3b6e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b00ce100 con 0x7fb3b00ac000 2024-01-15T18:10:08.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.673+0000 7fb3b6e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b00ce3e0 con 0x7fb3b00b39b0 2024-01-15T18:10:08.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.673+0000 7fb3b6e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b00ce6c0 con 0x7fb3b015a100 2024-01-15T18:10:08.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.677+0000 7fb3afe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3b00ac000 0x7fb3b00cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:08.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.677+0000 7fb3afe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3b00ac000 0x7fb3b00cd450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55680/0 (socket says 172.21.15.27:55680) 2024-01-15T18:10:08.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.677+0000 7fb3afe00640 1 -- 172.21.15.27:0/3743529641 learned_addr learned my addr 172.21.15.27:0/3743529641 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:08.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.677+0000 7fb3af400640 1 --2- 172.21.15.27:0/3743529641 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb3b00b39b0 0x7fb3b015cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:08.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.677+0000 7fb3afe00640 1 -- 172.21.15.27:0/3743529641 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb3b015a100 msgr2=0x7fb3b00cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:08.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.677+0000 7fb3afe00640 1 --2- 172.21.15.27:0/3743529641 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb3b015a100 0x7fb3b00cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:08.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.677+0000 7fb3afe00640 1 -- 172.21.15.27:0/3743529641 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb3b00b39b0 msgr2=0x7fb3b015cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:08.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.677+0000 7fb3afe00640 1 --2- 172.21.15.27:0/3743529641 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb3b00b39b0 0x7fb3b015cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:08.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.677+0000 7fb3afe00640 1 -- 172.21.15.27:0/3743529641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3980094d0 con 0x7fb3b00ac000 2024-01-15T18:10:08.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.677+0000 7fb3af400640 1 --2- 172.21.15.27:0/3743529641 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb3b00b39b0 0x7fb3b015cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:08.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.677+0000 7fb3afe00640 1 --2- 172.21.15.27:0/3743529641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3b00ac000 0x7fb3b00cd450 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fb3a000b3e0 tx=0x7fb3a000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:08.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.677+0000 7fb3a7e00640 1 -- 172.21.15.27:0/3743529641 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0011830 con 0x7fb3b00ac000 2024-01-15T18:10:08.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.677+0000 7fb3a7e00640 1 -- 172.21.15.27:0/3743529641 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a00119b0 con 0x7fb3b00ac000 2024-01-15T18:10:08.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.677+0000 7fb3a7e00640 1 -- 172.21.15.27:0/3743529641 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0011d60 con 0x7fb3b00ac000 2024-01-15T18:10:08.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.677+0000 7fb3b6e00640 1 -- 172.21.15.27:0/3743529641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b009a430 con 0x7fb3b00ac000 2024-01-15T18:10:08.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.681+0000 7fb3b6e00640 1 -- 172.21.15.27:0/3743529641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b009a090 con 0x7fb3b00ac000 2024-01-15T18:10:08.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.681+0000 7fb3b6e00640 1 -- 172.21.15.27:0/3743529641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b0067a10 con 0x7fb3b00ac000 2024-01-15T18:10:08.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.681+0000 7fb3a7e00640 1 -- 172.21.15.27:0/3743529641 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 86823+0+0 (secure 0 0 0) 0x7fb3a001a400 con 0x7fb3b00ac000 2024-01-15T18:10:08.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.681+0000 7fb3a7e00640 1 --2- 172.21.15.27:0/3743529641 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb388067040 0x7fb388069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:08.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.685+0000 7fb3af400640 1 --2- 172.21.15.27:0/3743529641 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb388067040 0x7fb388069500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:08.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.685+0000 7fb3a7e00640 1 -- 172.21.15.27:0/3743529641 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fb3a00550c0 con 0x7fb3b00ac000 2024-01-15T18:10:08.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.689+0000 7fb3af400640 1 --2- 172.21.15.27:0/3743529641 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb388067040 0x7fb388069500 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb3980ae000 tx=0x7fb398008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:08.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:08.697+0000 7fb3a7e00640 1 -- 172.21.15.27:0/3743529641 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb3a0081ec0 con 0x7fb3b00ac000 2024-01-15T18:10:09.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:09.069+0000 7fb3b6e00640 1 -- 172.21.15.27:0/3743529641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb3b0074820 con 0x7fb3b00ac000 2024-01-15T18:10:09.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:09.069+0000 7fb3a7e00640 1 -- 172.21.15.27:0/3743529641 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2337 (secure 0 0 0) 0x7fb3a0015070 con 0x7fb3b00ac000 2024-01-15T18:10:09.077 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:10:09.077 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:10:09.077 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:10:09.077 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:10:09.077 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:10:09.078 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:10:09.078 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:10:09.078 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:09.078 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:10:09.078 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:10:09.078 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:10:09.078 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:10:09.078 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:10:09.078 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:09.078 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:10:09.078 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:10:09.078 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:10:09.078 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:10:09.078 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:09.078 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 336, 2024-01-15T18:10:09.079 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:10:09.079 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:10:09.079 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:10:09.079 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:10:09.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:09.079 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:10:09.079 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 58, 2024-01-15T18:10:09.079 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:10:09.079 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:10:09.079 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:10:09.079 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:10:09.079 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:10:09.079 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:10:09.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:09.079 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:10:09.079 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:10:09.080 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:09.080 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:10:09.080 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:10:09.080 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:09.080 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:09.080 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:10:09.080 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:10:09.080 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 800, 2024-01-15T18:10:09.080 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 3222190772, 2024-01-15T18:10:09.080 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 6743728128, 2024-01-15T18:10:09.080 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 766350385152, 2024-01-15T18:10:09.080 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:10:09.080 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 511, 2024-01-15T18:10:09.080 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 18682, 2024-01-15T18:10:09.081 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 0, 2024-01-15T18:10:09.081 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 5 2024-01-15T18:10:09.081 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:09.081 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:10:09.081 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 38, 2024-01-15T18:10:09.081 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4, 2024-01-15T18:10:09.081 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:10:09.081 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:10:09.081 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:10:09.081 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:10:09.081 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:09.081 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 4, 2024-01-15T18:10:09.081 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:10:09.081 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:10:09.081 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:10:09.081 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 6480 2024-01-15T18:10:09.082 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:09.082 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:09.082 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:10:09.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:09.082 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:10:09.082 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:10:09.082 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:10:09.082 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:10:09.082 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:10:09.082 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:10:09.082 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:10:09.082 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:09.082 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:09.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:09.082 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:10:09.083 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 11, 2024-01-15T18:10:09.083 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:08:28.191482+0000", 2024-01-15T18:10:09.083 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:09.083 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:09.083 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:10:09.083 INFO:teuthology.orchestra.run.smithi027.stdout: "268d93d2-2eea-412c-b804-608e15344ce7": { 2024-01-15T18:10:09.083 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "5 ongoing+pending cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-15T18:10:09.083 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0, 2024-01-15T18:10:09.083 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:09.083 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:09.083 INFO:teuthology.orchestra.run.smithi027.stdout: "9a27fb80-350b-410b-a65e-91a770cccfdf": { 2024-01-15T18:10:09.083 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "2 ongoing cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-15T18:10:09.083 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0, 2024-01-15T18:10:09.083 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:09.084 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:09.084 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:09.084 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:10:09.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:09.073+0000 7fb3b6e00640 1 -- 172.21.15.27:0/3743529641 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb388067040 msgr2=0x7fb388069500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:09.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:09.073+0000 7fb3b6e00640 1 --2- 172.21.15.27:0/3743529641 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb388067040 0x7fb388069500 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb3980ae000 tx=0x7fb398008040 comp rx=0 tx=0).stop 2024-01-15T18:10:09.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:09.073+0000 7fb3b6e00640 1 -- 172.21.15.27:0/3743529641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3b00ac000 msgr2=0x7fb3b00cd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:09.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:09.073+0000 7fb3b6e00640 1 --2- 172.21.15.27:0/3743529641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3b00ac000 0x7fb3b00cd450 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fb3a000b3e0 tx=0x7fb3a000b8a0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:09.077+0000 7fb3b6e00640 1 -- 172.21.15.27:0/3743529641 shutdown_connections 2024-01-15T18:10:09.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:09.077+0000 7fb3b6e00640 1 --2- 172.21.15.27:0/3743529641 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb388067040 0x7fb388069500 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:09.077+0000 7fb3b6e00640 1 --2- 172.21.15.27:0/3743529641 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb3b015a100 0x7fb3b00cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:09.077+0000 7fb3b6e00640 1 --2- 172.21.15.27:0/3743529641 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb3b00b39b0 0x7fb3b015cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:09.077+0000 7fb3b6e00640 1 --2- 172.21.15.27:0/3743529641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb3b00ac000 0x7fb3b00cd450 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:09.077+0000 7fb3b6e00640 1 -- 172.21.15.27:0/3743529641 >> 172.21.15.27:0/3743529641 conn(0x7fb3b00b0d90 msgr2=0x7fb3b0166cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:09.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:09.077+0000 7fb3b6e00640 1 -- 172.21.15.27:0/3743529641 shutdown_connections 2024-01-15T18:10:09.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:09.077+0000 7fb3b6e00640 1 -- 172.21.15.27:0/3743529641 wait complete. 2024-01-15T18:10:10.265 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:10:10.261+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:10:10.265 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:10:10.261+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:10:10.266 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:10:10.261+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:10:10.266 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:10:10.261+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:10:10.266 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:10:10.261+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:10:11.129 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:10:11.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.269+0000 7ff813c00640 1 Processor -- start 2024-01-15T18:10:11.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.269+0000 7ff813c00640 1 -- start start 2024-01-15T18:10:11.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.269+0000 7ff813c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff80c15e7d0 0x7ff80c0cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:11.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.269+0000 7ff813c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff80c0cedc0 0x7ff80c0cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:11.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.269+0000 7ff813c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff80c0ce2e0 0x7ff80c0cba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:11.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.269+0000 7ff813c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff80c0b38b0 con 0x7ff80c0cedc0 2024-01-15T18:10:11.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.269+0000 7ff813c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff80c0cd550 con 0x7ff80c0ce2e0 2024-01-15T18:10:11.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.269+0000 7ff813c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c0cd270 con 0x7ff80c15e7d0 2024-01-15T18:10:11.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.269+0000 7ff80be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff80c0cedc0 0x7ff80c0cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:11.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.269+0000 7ff80be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff80c0cedc0 0x7ff80c0cf7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55694/0 (socket says 172.21.15.27:55694) 2024-01-15T18:10:11.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.269+0000 7ff80be00640 1 -- 172.21.15.27:0/609931604 learned_addr learned my addr 172.21.15.27:0/609931604 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:11.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.269+0000 7ff811200640 1 --2- 172.21.15.27:0/609931604 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff80c15e7d0 0x7ff80c0cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:11.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.269+0000 7ff811c00640 1 --2- 172.21.15.27:0/609931604 >> v2:172.21.15.93:3300/0 conn(0x7ff80c0ce2e0 0x7ff80c0cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:11.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.269+0000 7ff80be00640 1 -- 172.21.15.27:0/609931604 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff80c15e7d0 msgr2=0x7ff80c0cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:11.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.269+0000 7ff80be00640 1 --2- 172.21.15.27:0/609931604 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff80c15e7d0 0x7ff80c0cf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.269+0000 7ff80be00640 1 -- 172.21.15.27:0/609931604 >> v2:172.21.15.93:3300/0 conn(0x7ff80c0ce2e0 msgr2=0x7ff80c0cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:11.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.269+0000 7ff80be00640 1 --2- 172.21.15.27:0/609931604 >> v2:172.21.15.93:3300/0 conn(0x7ff80c0ce2e0 0x7ff80c0cba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.269+0000 7ff80be00640 1 -- 172.21.15.27:0/609931604 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80c0b35d0 con 0x7ff80c0cedc0 2024-01-15T18:10:11.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.269+0000 7ff811200640 1 --2- 172.21.15.27:0/609931604 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff80c15e7d0 0x7ff80c0cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:10:11.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.269+0000 7ff811c00640 1 --2- 172.21.15.27:0/609931604 >> v2:172.21.15.93:3300/0 conn(0x7ff80c0ce2e0 0x7ff80c0cba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:10:11.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.273+0000 7ff80be00640 1 --2- 172.21.15.27:0/609931604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff80c0cedc0 0x7ff80c0cf7e0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7ff80c071b60 tx=0x7ff7f401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56ff9e64f3f2d808 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:11.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.273+0000 7ff80b400640 1 -- 172.21.15.27:0/609931604 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f40ac410 con 0x7ff80c0cedc0 2024-01-15T18:10:11.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.273+0000 7ff80b400640 1 -- 172.21.15.27:0/609931604 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f40ac570 con 0x7ff80c0cedc0 2024-01-15T18:10:11.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.273+0000 7ff80b400640 1 -- 172.21.15.27:0/609931604 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f40ac900 con 0x7ff80c0cedc0 2024-01-15T18:10:11.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.273+0000 7ff813c00640 1 -- 172.21.15.27:0/609931604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff80c0cedc0 msgr2=0x7ff80c0cf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:11.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.273+0000 7ff813c00640 1 --2- 172.21.15.27:0/609931604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff80c0cedc0 0x7ff80c0cf7e0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7ff80c071b60 tx=0x7ff7f401c500 comp rx=0 tx=0).stop 2024-01-15T18:10:11.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.273+0000 7ff813c00640 1 -- 172.21.15.27:0/609931604 shutdown_connections 2024-01-15T18:10:11.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.273+0000 7ff813c00640 1 --2- 172.21.15.27:0/609931604 >> v2:172.21.15.93:3300/0 conn(0x7ff80c0ce2e0 0x7ff80c0cba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.273+0000 7ff813c00640 1 --2- 172.21.15.27:0/609931604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff80c0cedc0 0x7ff80c0cf7e0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.273+0000 7ff813c00640 1 --2- 172.21.15.27:0/609931604 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff80c15e7d0 0x7ff80c0cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.273+0000 7ff813c00640 1 -- 172.21.15.27:0/609931604 >> 172.21.15.27:0/609931604 conn(0x7ff80c0b2a10 msgr2=0x7ff80c0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:11.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.273+0000 7ff813c00640 1 -- 172.21.15.27:0/609931604 shutdown_connections 2024-01-15T18:10:11.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.273+0000 7ff813c00640 1 -- 172.21.15.27:0/609931604 wait complete. 2024-01-15T18:10:11.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.277+0000 7ff813c00640 1 Processor -- start 2024-01-15T18:10:11.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.277+0000 7ff813c00640 1 -- start start 2024-01-15T18:10:11.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.277+0000 7ff813c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff80c0ce2e0 0x7ff80c163f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:11.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.277+0000 7ff813c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff80c0cedc0 0x7ff80c154840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:11.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.277+0000 7ff813c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff80c15e7d0 0x7ff80c158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:11.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.277+0000 7ff813c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c165200 con 0x7ff80c0cedc0 2024-01-15T18:10:11.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.277+0000 7ff813c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c154d80 con 0x7ff80c15e7d0 2024-01-15T18:10:11.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.277+0000 7ff813c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c164580 con 0x7ff80c0ce2e0 2024-01-15T18:10:11.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.277+0000 7ff80be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff80c0cedc0 0x7ff80c154840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:11.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.277+0000 7ff80be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff80c0cedc0 0x7ff80c154840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55704/0 (socket says 172.21.15.27:55704) 2024-01-15T18:10:11.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.277+0000 7ff80be00640 1 -- 172.21.15.27:0/3452979131 learned_addr learned my addr 172.21.15.27:0/3452979131 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:11.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.277+0000 7ff811200640 1 --2- 172.21.15.27:0/3452979131 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff80c0ce2e0 0x7ff80c163f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:11.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.277+0000 7ff811c00640 1 --2- 172.21.15.27:0/3452979131 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff80c15e7d0 0x7ff80c158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:11.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.277+0000 7ff80be00640 1 -- 172.21.15.27:0/3452979131 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff80c0ce2e0 msgr2=0x7ff80c163f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:11.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.277+0000 7ff80be00640 1 --2- 172.21.15.27:0/3452979131 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff80c0ce2e0 0x7ff80c163f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.277+0000 7ff80be00640 1 -- 172.21.15.27:0/3452979131 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff80c15e7d0 msgr2=0x7ff80c158740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:11.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.277+0000 7ff80be00640 1 --2- 172.21.15.27:0/3452979131 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff80c15e7d0 0x7ff80c158740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.277+0000 7ff80be00640 1 -- 172.21.15.27:0/3452979131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f40094d0 con 0x7ff80c0cedc0 2024-01-15T18:10:11.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.277+0000 7ff80be00640 1 --2- 172.21.15.27:0/3452979131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff80c0cedc0 0x7ff80c154840 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7ff7f401abb0 tx=0x7ff7f40192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:11.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.277+0000 7ff808c00640 1 -- 172.21.15.27:0/3452979131 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f40ac410 con 0x7ff80c0cedc0 2024-01-15T18:10:11.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.277+0000 7ff808c00640 1 -- 172.21.15.27:0/3452979131 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f40b65f0 con 0x7ff80c0cedc0 2024-01-15T18:10:11.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.277+0000 7ff813c00640 1 -- 172.21.15.27:0/3452979131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff80c164860 con 0x7ff80c0cedc0 2024-01-15T18:10:11.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.277+0000 7ff808c00640 1 -- 172.21.15.27:0/3452979131 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f40b69c0 con 0x7ff80c0cedc0 2024-01-15T18:10:11.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.277+0000 7ff813c00640 1 -- 172.21.15.27:0/3452979131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff80c164e20 con 0x7ff80c0cedc0 2024-01-15T18:10:11.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.281+0000 7ff813c00640 1 -- 172.21.15.27:0/3452979131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff80c165340 con 0x7ff80c0cedc0 2024-01-15T18:10:11.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.281+0000 7ff808c00640 1 -- 172.21.15.27:0/3452979131 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 16) v1 ==== 86823+0+0 (secure 0 0 0) 0x7ff7f40b6b20 con 0x7ff80c0cedc0 2024-01-15T18:10:11.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.281+0000 7ff808c00640 1 --2- 172.21.15.27:0/3452979131 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff7e4067070 0x7ff7e4069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:11.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.281+0000 7ff808c00640 1 -- 172.21.15.27:0/3452979131 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7ff7f411d730 con 0x7ff80c0cedc0 2024-01-15T18:10:11.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.281+0000 7ff811200640 1 --2- 172.21.15.27:0/3452979131 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff7e4067070 0x7ff7e4069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:11.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.285+0000 7ff811200640 1 --2- 172.21.15.27:0/3452979131 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff7e4067070 0x7ff7e4069530 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc0045f0 tx=0x7ff7fc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:11.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.285+0000 7ff808c00640 1 -- 172.21.15.27:0/3452979131 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff80c165340 con 0x7ff80c0cedc0 2024-01-15T18:10:11.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.497+0000 7ff813c00640 1 -- 172.21.15.27:0/3452979131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff80c165340 con 0x7ff80c0cedc0 2024-01-15T18:10:11.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.497+0000 7ff808c00640 1 -- 172.21.15.27:0/3452979131 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2801 (secure 0 0 0) 0x7ff7f40b1b10 con 0x7ff80c0cedc0 2024-01-15T18:10:11.504 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:10:11.505 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:10:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:10:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:10:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:10:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:10:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:10:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:11.505 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:10:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:10:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:10:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:10:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:10:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:10:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:10:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:10:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:10:11.506 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 338, 2024-01-15T18:10:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:10:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:10:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:10:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:10:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:10:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 58, 2024-01-15T18:10:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:10:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:10:11.507 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:10:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:10:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:10:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:10:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:10:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:10:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:11.508 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:10:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:10:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:10:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:10:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 1056, 2024-01-15T18:10:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 4291738872, 2024-01-15T18:10:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 8536616960, 2024-01-15T18:10:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 764557496320, 2024-01-15T18:10:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:10:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 88728053, 2024-01-15T18:10:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 89087436, 2024-01-15T18:10:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 22, 2024-01-15T18:10:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 25 2024-01-15T18:10:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:10:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 38, 2024-01-15T18:10:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4, 2024-01-15T18:10:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:10:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:10:11.510 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:10:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:10:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 4, 2024-01-15T18:10:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:10:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:10:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:10:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 6480 2024-01-15T18:10:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:10:11.511 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:10:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:10:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:10:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:10:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:10:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:10:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:10:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:11.512 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:10:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 12, 2024-01-15T18:10:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:10:08.223580+0000", 2024-01-15T18:10:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "services": { 2024-01-15T18:10:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-01-15T18:10:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "daemons": { 2024-01-15T18:10:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "summary": "", 2024-01-15T18:10:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "c": { 2024-01-15T18:10:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "start_epoch": 0, 2024-01-15T18:10:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "start_stamp": "0.000000", 2024-01-15T18:10:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 0, 2024-01-15T18:10:11.513 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "(unrecognized address family 0)/0", 2024-01-15T18:10:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": {}, 2024-01-15T18:10:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: "task_status": {} 2024-01-15T18:10:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:10:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: "268d93d2-2eea-412c-b804-608e15344ce7": { 2024-01-15T18:10:11.514 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "5 ongoing+pending cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-15T18:10:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0, 2024-01-15T18:10:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: "9a27fb80-350b-410b-a65e-91a770cccfdf": { 2024-01-15T18:10:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "2 ongoing cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-15T18:10:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0, 2024-01-15T18:10:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:11.515 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:11.515 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:10:11.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.501+0000 7ff813c00640 1 -- 172.21.15.27:0/3452979131 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff7e4067070 msgr2=0x7ff7e4069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:11.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.501+0000 7ff813c00640 1 --2- 172.21.15.27:0/3452979131 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff7e4067070 0x7ff7e4069530 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc0045f0 tx=0x7ff7fc008040 comp rx=0 tx=0).stop 2024-01-15T18:10:11.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.501+0000 7ff813c00640 1 -- 172.21.15.27:0/3452979131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff80c0cedc0 msgr2=0x7ff80c154840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:11.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.501+0000 7ff813c00640 1 --2- 172.21.15.27:0/3452979131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff80c0cedc0 0x7ff80c154840 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7ff7f401abb0 tx=0x7ff7f40192a0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.501+0000 7ff813c00640 1 -- 172.21.15.27:0/3452979131 shutdown_connections 2024-01-15T18:10:11.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.501+0000 7ff813c00640 1 --2- 172.21.15.27:0/3452979131 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff7e4067070 0x7ff7e4069530 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.501+0000 7ff813c00640 1 --2- 172.21.15.27:0/3452979131 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff80c15e7d0 0x7ff80c158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.501+0000 7ff813c00640 1 --2- 172.21.15.27:0/3452979131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff80c0cedc0 0x7ff80c154840 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.501+0000 7ff813c00640 1 --2- 172.21.15.27:0/3452979131 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff80c0ce2e0 0x7ff80c163f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.501+0000 7ff813c00640 1 -- 172.21.15.27:0/3452979131 >> 172.21.15.27:0/3452979131 conn(0x7ff80c0b2a10 msgr2=0x7ff80c15cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:11.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.501+0000 7ff813c00640 1 -- 172.21.15.27:0/3452979131 shutdown_connections 2024-01-15T18:10:11.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:11.501+0000 7ff813c00640 1 -- 172.21.15.27:0/3452979131 wait complete. 2024-01-15T18:10:13.527 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:10:13.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.805+0000 7f30bb600640 1 Processor -- start 2024-01-15T18:10:13.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.805+0000 7f30bb600640 1 -- start start 2024-01-15T18:10:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.805+0000 7f30bb600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f30b40b2390 0x7f30b40cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.805+0000 7f30bb600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f30b406bb80 0x7f30b40cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.805+0000 7f30bb600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30b40cfcc0 0x7f30b40cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.805+0000 7f30bb600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30b40b2770 con 0x7f30b40b2390 2024-01-15T18:10:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.805+0000 7f30bb600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30b406c060 con 0x7f30b406bb80 2024-01-15T18:10:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.805+0000 7f30bb600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30b406c340 con 0x7f30b40cfcc0 2024-01-15T18:10:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.805+0000 7f30b8c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f30b40b2390 0x7f30b40cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.805+0000 7f30b8c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f30b40b2390 0x7f30b40cf780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39752/0 (socket says 172.21.15.27:39752) 2024-01-15T18:10:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.805+0000 7f30b8c00640 1 -- 172.21.15.27:0/3897451943 learned_addr learned my addr 172.21.15.27:0/3897451943 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.805+0000 7f30b8c00640 1 -- 172.21.15.27:0/3897451943 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30b40cfcc0 msgr2=0x7f30b40cddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.805+0000 7f30b8c00640 1 --2- 172.21.15.27:0/3897451943 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30b40cfcc0 0x7f30b40cddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.805+0000 7f30b8c00640 1 -- 172.21.15.27:0/3897451943 >> v2:172.21.15.93:3300/0 conn(0x7f30b406bb80 msgr2=0x7f30b40cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.805+0000 7f30b8c00640 1 --2- 172.21.15.27:0/3897451943 >> v2:172.21.15.93:3300/0 conn(0x7f30b406bb80 0x7f30b40cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.805+0000 7f30b8c00640 1 -- 172.21.15.27:0/3897451943 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b406c620 con 0x7f30b40b2390 2024-01-15T18:10:13.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.813+0000 7f30b8c00640 1 --2- 172.21.15.27:0/3897451943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b40b2390 0x7f30b40cf780 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f30b4070ea0 tx=0x7f30a401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0b050e4ec5fa5b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:13.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.813+0000 7f30b3400640 1 -- 172.21.15.27:0/3897451943 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a40aa5f0 con 0x7f30b40b2390 2024-01-15T18:10:13.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.813+0000 7f30b3400640 1 -- 172.21.15.27:0/3897451943 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30a40aa750 con 0x7f30b40b2390 2024-01-15T18:10:13.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.813+0000 7f30b3400640 1 -- 172.21.15.27:0/3897451943 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a40aaae0 con 0x7f30b40b2390 2024-01-15T18:10:13.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.813+0000 7f30bb600640 1 -- 172.21.15.27:0/3897451943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b40b2390 msgr2=0x7f30b40cf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:13.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.813+0000 7f30bb600640 1 --2- 172.21.15.27:0/3897451943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b40b2390 0x7f30b40cf780 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f30b4070ea0 tx=0x7f30a401c4d0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.813+0000 7f30bb600640 1 -- 172.21.15.27:0/3897451943 shutdown_connections 2024-01-15T18:10:13.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.813+0000 7f30bb600640 1 --2- 172.21.15.27:0/3897451943 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30b40cfcc0 0x7f30b40cddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.813+0000 7f30bb600640 1 --2- 172.21.15.27:0/3897451943 >> v2:172.21.15.93:3300/0 conn(0x7f30b406bb80 0x7f30b40cd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.813+0000 7f30bb600640 1 --2- 172.21.15.27:0/3897451943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b40b2390 0x7f30b40cf780 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.813+0000 7f30bb600640 1 -- 172.21.15.27:0/3897451943 >> 172.21.15.27:0/3897451943 conn(0x7f30b40b2a10 msgr2=0x7f30b40b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:13.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.813+0000 7f30bb600640 1 -- 172.21.15.27:0/3897451943 shutdown_connections 2024-01-15T18:10:13.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.813+0000 7f30bb600640 1 -- 172.21.15.27:0/3897451943 wait complete. 2024-01-15T18:10:13.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.817+0000 7f30bb600640 1 Processor -- start 2024-01-15T18:10:13.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.817+0000 7f30bb600640 1 -- start start 2024-01-15T18:10:13.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.821+0000 7f30bb600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30b406bb80 0x7f30b41f99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:13.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.821+0000 7f30bb600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b40b2390 0x7f30b415c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:13.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.821+0000 7f30bb600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30b40cfcc0 0x7f30b415cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.821+0000 7f30bb600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b40c5dc0 con 0x7f30b40b2390 2024-01-15T18:10:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.821+0000 7f30bb600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b41f9f10 con 0x7f30b40cfcc0 2024-01-15T18:10:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.821+0000 7f30bb600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30b41fa190 con 0x7f30b406bb80 2024-01-15T18:10:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.821+0000 7f30b8c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30b406bb80 0x7f30b41f99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.821+0000 7f30b8c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30b406bb80 0x7f30b41f99d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:59866/0 (socket says 172.21.15.27:59866) 2024-01-15T18:10:13.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.821+0000 7f30b8c00640 1 -- 172.21.15.27:0/183246283 learned_addr learned my addr 172.21.15.27:0/183246283 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:13.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.821+0000 7f30b3e00640 1 --2- 172.21.15.27:0/183246283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b40b2390 0x7f30b415c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:13.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.825+0000 7f30b8c00640 1 -- 172.21.15.27:0/183246283 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30b40cfcc0 msgr2=0x7f30b415cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:13.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.825+0000 7f30b8c00640 1 --2- 172.21.15.27:0/183246283 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30b40cfcc0 0x7f30b415cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.825+0000 7f30b8c00640 1 -- 172.21.15.27:0/183246283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b40b2390 msgr2=0x7f30b415c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:13.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.825+0000 7f30b8c00640 1 --2- 172.21.15.27:0/183246283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b40b2390 0x7f30b415c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.825+0000 7f30b8c00640 1 -- 172.21.15.27:0/183246283 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a40094d0 con 0x7f30b406bb80 2024-01-15T18:10:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.829+0000 7f30b8c00640 1 --2- 172.21.15.27:0/183246283 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30b406bb80 0x7f30b41f99d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f30a401ab80 tx=0x7f30a401ad10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.829+0000 7f30b0c00640 1 -- 172.21.15.27:0/183246283 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a40b3800 con 0x7f30b406bb80 2024-01-15T18:10:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.829+0000 7f30b0c00640 1 -- 172.21.15.27:0/183246283 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30a40b3980 con 0x7f30b406bb80 2024-01-15T18:10:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.829+0000 7f30b0c00640 1 -- 172.21.15.27:0/183246283 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a40b3d30 con 0x7f30b406bb80 2024-01-15T18:10:13.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.829+0000 7f30bb600640 1 -- 172.21.15.27:0/183246283 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30b41fa470 con 0x7f30b406bb80 2024-01-15T18:10:13.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.829+0000 7f30bb600640 1 -- 172.21.15.27:0/183246283 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30b41faa30 con 0x7f30b406bb80 2024-01-15T18:10:13.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.833+0000 7f30bb600640 1 -- 172.21.15.27:0/183246283 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30b4067a10 con 0x7f30b406bb80 2024-01-15T18:10:13.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.841+0000 7f30b0c00640 1 -- 172.21.15.27:0/183246283 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 17) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f30a40c6390 con 0x7f30b406bb80 2024-01-15T18:10:13.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.841+0000 7f30b0c00640 1 --2- 172.21.15.27:0/183246283 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f308c067120 0x7f308c0695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:13.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.841+0000 7f30b0c00640 1 -- 172.21.15.27:0/183246283 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f30a40a7070 con 0x7f30b406bb80 2024-01-15T18:10:13.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.849+0000 7f30b0c00640 1 -- 172.21.15.27:0/183246283 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f30a40ec390 con 0x7f30b406bb80 2024-01-15T18:10:13.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.853+0000 7f30b3e00640 1 --2- 172.21.15.27:0/183246283 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f308c067120 0x7f308c0695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:13.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:13.853+0000 7f30b3e00640 1 --2- 172.21.15.27:0/183246283 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f308c067120 0x7f308c0695e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f309c006fd0 tx=0x7f309c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:14.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:14.269+0000 7f30bb600640 1 -- 172.21.15.27:0/183246283 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f30b404e850 con 0x7f30b406bb80 2024-01-15T18:10:14.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:14.269+0000 7f30b0c00640 1 -- 172.21.15.27:0/183246283 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2875 (secure 0 0 0) 0x7f30a40ef9b0 con 0x7f30b406bb80 2024-01-15T18:10:14.274 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:10:14.274 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:10:14.274 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:10:14.274 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:10:14.274 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:10:14.275 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:10:14.275 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:10:14.275 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:14.275 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:10:14.275 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:10:14.275 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:10:14.275 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:10:14.275 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:10:14.275 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:14.275 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:10:14.275 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:10:14.275 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:10:14.275 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:10:14.275 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:14.276 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 341, 2024-01-15T18:10:14.276 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:10:14.276 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:10:14.276 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:10:14.276 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:10:14.276 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:14.276 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:10:14.276 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 58, 2024-01-15T18:10:14.276 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:10:14.276 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:10:14.276 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:10:14.276 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:10:14.276 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:10:14.276 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:10:14.277 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:14.277 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:10:14.277 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:10:14.277 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:14.277 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:10:14.277 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:10:14.277 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:14.277 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:14.277 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:10:14.277 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:10:14.277 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 1256, 2024-01-15T18:10:14.277 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 5122432506, 2024-01-15T18:10:14.277 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 9920974848, 2024-01-15T18:10:14.278 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 763173138432, 2024-01-15T18:10:14.278 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:10:14.278 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 163483109, 2024-01-15T18:10:14.278 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 158273665, 2024-01-15T18:10:14.278 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 40, 2024-01-15T18:10:14.278 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 46 2024-01-15T18:10:14.278 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:14.278 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:10:14.278 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 38, 2024-01-15T18:10:14.278 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4, 2024-01-15T18:10:14.278 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:10:14.278 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:10:14.278 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:10:14.278 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:10:14.279 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:14.279 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 4, 2024-01-15T18:10:14.279 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:10:14.279 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:10:14.279 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:10:14.279 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 6480 2024-01-15T18:10:14.279 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:14.279 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:14.279 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:10:14.279 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:14.279 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:10:14.279 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:10:14.279 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:10:14.279 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:10:14.280 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:10:14.280 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:10:14.280 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:10:14.280 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:14.280 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:14.280 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:14.280 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:10:14.280 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 12, 2024-01-15T18:10:14.280 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:10:08.223580+0000", 2024-01-15T18:10:14.280 INFO:teuthology.orchestra.run.smithi027.stdout: "services": { 2024-01-15T18:10:14.280 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-01-15T18:10:14.280 INFO:teuthology.orchestra.run.smithi027.stdout: "daemons": { 2024-01-15T18:10:14.280 INFO:teuthology.orchestra.run.smithi027.stdout: "summary": "", 2024-01-15T18:10:14.280 INFO:teuthology.orchestra.run.smithi027.stdout: "c": { 2024-01-15T18:10:14.280 INFO:teuthology.orchestra.run.smithi027.stdout: "start_epoch": 0, 2024-01-15T18:10:14.281 INFO:teuthology.orchestra.run.smithi027.stdout: "start_stamp": "0.000000", 2024-01-15T18:10:14.281 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 0, 2024-01-15T18:10:14.281 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "(unrecognized address family 0)/0", 2024-01-15T18:10:14.281 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": {}, 2024-01-15T18:10:14.281 INFO:teuthology.orchestra.run.smithi027.stdout: "task_status": {} 2024-01-15T18:10:14.281 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:14.281 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:14.281 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:14.281 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:14.281 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:14.281 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:10:14.281 INFO:teuthology.orchestra.run.smithi027.stdout: "268d93d2-2eea-412c-b804-608e15344ce7": { 2024-01-15T18:10:14.281 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (4s)\n [==..........................] (remaining: 46s)", 2024-01-15T18:10:14.281 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.095228567719459534, 2024-01-15T18:10:14.281 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:14.282 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:14.282 INFO:teuthology.orchestra.run.smithi027.stdout: "9a27fb80-350b-410b-a65e-91a770cccfdf": { 2024-01-15T18:10:14.282 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "2 ongoing cloning jobs; Avg progress = 33.33% (4s)\n [=========...................] (remaining: 9s)", 2024-01-15T18:10:14.282 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.33329999446868896, 2024-01-15T18:10:14.282 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:14.282 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:14.282 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:14.282 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:10:14.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:14.269+0000 7f30bb600640 1 -- 172.21.15.27:0/183246283 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f308c067120 msgr2=0x7f308c0695e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:14.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:14.269+0000 7f30bb600640 1 --2- 172.21.15.27:0/183246283 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f308c067120 0x7f308c0695e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f309c006fd0 tx=0x7f309c008040 comp rx=0 tx=0).stop 2024-01-15T18:10:14.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:14.269+0000 7f30bb600640 1 -- 172.21.15.27:0/183246283 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30b406bb80 msgr2=0x7f30b41f99d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:14.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:14.269+0000 7f30bb600640 1 --2- 172.21.15.27:0/183246283 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30b406bb80 0x7f30b41f99d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f30a401ab80 tx=0x7f30a401ad10 comp rx=0 tx=0).stop 2024-01-15T18:10:14.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:14.269+0000 7f30bb600640 1 -- 172.21.15.27:0/183246283 shutdown_connections 2024-01-15T18:10:14.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:14.269+0000 7f30bb600640 1 --2- 172.21.15.27:0/183246283 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f308c067120 0x7f308c0695e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:14.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:14.269+0000 7f30bb600640 1 --2- 172.21.15.27:0/183246283 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30b40cfcc0 0x7f30b415cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:14.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:14.269+0000 7f30bb600640 1 --2- 172.21.15.27:0/183246283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b40b2390 0x7f30b415c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:14.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:14.269+0000 7f30bb600640 1 --2- 172.21.15.27:0/183246283 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30b406bb80 0x7f30b41f99d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:14.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:14.269+0000 7f30bb600640 1 -- 172.21.15.27:0/183246283 >> 172.21.15.27:0/183246283 conn(0x7f30b40b2a10 msgr2=0x7f30b40cd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:14.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:14.273+0000 7f30bb600640 1 -- 172.21.15.27:0/183246283 shutdown_connections 2024-01-15T18:10:14.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:14.273+0000 7f30bb600640 1 -- 172.21.15.27:0/183246283 wait complete. 2024-01-15T18:10:15.287 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:10:15.281+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:10:15.287 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:10:15.281+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:10:15.287 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:10:15.281+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:10:15.287 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:10:15.281+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:10:15.287 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:10:15.281+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:10:16.304 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:10:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.525+0000 7fdb60000640 1 Processor -- start 2024-01-15T18:10:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.525+0000 7fdb60000640 1 -- start start 2024-01-15T18:10:16.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.525+0000 7fdb60000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdb58165830 0x7fdb5816c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:16.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.525+0000 7fdb60000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb580b2850 con 0x7fdb58161590 2024-01-15T18:10:16.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.525+0000 7fdb60000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb58161970 con 0x7fdb580ae800 2024-01-15T18:10:16.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.525+0000 7fdb60000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb580aec60 con 0x7fdb58165830 2024-01-15T18:10:16.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.525+0000 7fdb5d600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdb58161590 0x7fdb5806af00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46162/0 (socket says 172.21.15.27:46162) 2024-01-15T18:10:16.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.525+0000 7fdb5d600640 1 -- 172.21.15.27:0/878358869 learned_addr learned my addr 172.21.15.27:0/878358869 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:10:16.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.525+0000 7fdb5e000640 1 --2- 172.21.15.27:0/878358869 >> v2:172.21.15.27:3300/0 conn(0x7fdb58165830 0x7fdb5816c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:16.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.525+0000 7fdb57e00640 1 -- 172.21.15.27:0/878358869 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2205507350 0 0) 0x7fdb580b2850 con 0x7fdb58161590 2024-01-15T18:10:16.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.525+0000 7fdb57e00640 1 -- 172.21.15.27:0/878358869 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb38003530 con 0x7fdb58161590 2024-01-15T18:10:16.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.525+0000 7fdb5e000640 1 -- 172.21.15.27:0/878358869 >> v1:172.21.15.93:6789/0 conn(0x7fdb580ae800 legacy=0x7fdb58165120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:16.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.529+0000 7fdb57e00640 1 -- 172.21.15.27:0/878358869 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3076002364 0 0) 0x7fdb38003530 con 0x7fdb58161590 2024-01-15T18:10:16.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.529+0000 7fdb5e000640 1 -- 172.21.15.27:0/878358869 >> v1:172.21.15.27:6789/0 conn(0x7fdb58161590 legacy=0x7fdb5806af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:16.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.529+0000 7fdb5e000640 1 -- 172.21.15.27:0/878358869 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5806b6e0 con 0x7fdb58165830 2024-01-15T18:10:16.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.533+0000 7fdb5e000640 1 --2- 172.21.15.27:0/878358869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb58165830 0x7fdb5816c4b0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fdb50009d90 tx=0x7fdb5001c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60c26d16f607b77f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:16.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.533+0000 7fdb57e00640 1 -- 172.21.15.27:0/878358869 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb48001680 con 0x7fdb58165830 2024-01-15T18:10:16.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.533+0000 7fdb57e00640 1 -- 172.21.15.27:0/878358869 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb500a55f0 con 0x7fdb58165830 2024-01-15T18:10:16.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.533+0000 7fdb57e00640 1 -- 172.21.15.27:0/878358869 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb500a5980 con 0x7fdb58165830 2024-01-15T18:10:16.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.533+0000 7fdb60000640 1 -- 172.21.15.27:0/878358869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb58165830 msgr2=0x7fdb5816c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:16.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.533+0000 7fdb60000640 1 --2- 172.21.15.27:0/878358869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb58165830 0x7fdb5816c4b0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fdb50009d90 tx=0x7fdb5001c7b0 comp rx=0 tx=0).stop 2024-01-15T18:10:16.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.533+0000 7fdb60000640 1 -- 172.21.15.27:0/878358869 shutdown_connections 2024-01-15T18:10:16.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.533+0000 7fdb60000640 1 --2- 172.21.15.27:0/878358869 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb58165830 0x7fdb5816c4b0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:16.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.533+0000 7fdb60000640 1 -- 172.21.15.27:0/878358869 >> 172.21.15.27:0/878358869 conn(0x7fdb580cf520 msgr2=0x7fdb580cf930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:16.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.533+0000 7fdb60000640 1 -- 172.21.15.27:0/878358869 shutdown_connections 2024-01-15T18:10:16.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.533+0000 7fdb60000640 1 -- 172.21.15.27:0/878358869 wait complete. 2024-01-15T18:10:16.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.537+0000 7fdb60000640 1 Processor -- start 2024-01-15T18:10:16.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.537+0000 7fdb60000640 1 -- start start 2024-01-15T18:10:16.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.541+0000 7fdb60000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb580ae800 0x7fdb5814ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:16.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.541+0000 7fdb60000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb58161590 0x7fdb581639c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:16.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.541+0000 7fdb60000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdb58165830 0x7fdb58163f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:16.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.541+0000 7fdb60000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5814fdd0 con 0x7fdb580ae800 2024-01-15T18:10:16.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.541+0000 7fdb60000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb58158090 con 0x7fdb58165830 2024-01-15T18:10:16.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.541+0000 7fdb60000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb581590a0 con 0x7fdb58161590 2024-01-15T18:10:16.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.541+0000 7fdb5d600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb580ae800 0x7fdb5814ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:16.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.541+0000 7fdb5d600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb580ae800 0x7fdb5814ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39764/0 (socket says 172.21.15.27:39764) 2024-01-15T18:10:16.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.541+0000 7fdb5d600640 1 -- 172.21.15.27:0/60286247 learned_addr learned my addr 172.21.15.27:0/60286247 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:16.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.541+0000 7fdb5e000640 1 --2- 172.21.15.27:0/60286247 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdb58165830 0x7fdb58163f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:16.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.541+0000 7fdb5d600640 1 -- 172.21.15.27:0/60286247 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb58161590 msgr2=0x7fdb581639c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:16.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.541+0000 7fdb5cc00640 1 --2- 172.21.15.27:0/60286247 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb58161590 0x7fdb581639c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:16.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.541+0000 7fdb5d600640 1 --2- 172.21.15.27:0/60286247 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb58161590 0x7fdb581639c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:16.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.541+0000 7fdb5d600640 1 -- 172.21.15.27:0/60286247 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdb58165830 msgr2=0x7fdb58163f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:16.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.541+0000 7fdb5d600640 1 --2- 172.21.15.27:0/60286247 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdb58165830 0x7fdb58163f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:16.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.541+0000 7fdb5d600640 1 -- 172.21.15.27:0/60286247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb50009a70 con 0x7fdb580ae800 2024-01-15T18:10:16.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.541+0000 7fdb5cc00640 1 --2- 172.21.15.27:0/60286247 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb58161590 0x7fdb581639c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:16.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.541+0000 7fdb5d600640 1 --2- 172.21.15.27:0/60286247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb580ae800 0x7fdb5814ce70 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fdb4800c760 tx=0x7fdb4800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:16.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.541+0000 7fdb55600640 1 -- 172.21.15.27:0/60286247 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb48005ae0 con 0x7fdb580ae800 2024-01-15T18:10:16.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.541+0000 7fdb55600640 1 -- 172.21.15.27:0/60286247 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb48005c40 con 0x7fdb580ae800 2024-01-15T18:10:16.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.541+0000 7fdb55600640 1 -- 172.21.15.27:0/60286247 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb48007070 con 0x7fdb580ae800 2024-01-15T18:10:16.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.541+0000 7fdb60000640 1 -- 172.21.15.27:0/60286247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb581582e0 con 0x7fdb580ae800 2024-01-15T18:10:16.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.541+0000 7fdb60000640 1 -- 172.21.15.27:0/60286247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb581588a0 con 0x7fdb580ae800 2024-01-15T18:10:16.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.545+0000 7fdb60000640 1 -- 172.21.15.27:0/60286247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb5804e850 con 0x7fdb580ae800 2024-01-15T18:10:16.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.545+0000 7fdb55600640 1 -- 172.21.15.27:0/60286247 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7fdb480071d0 con 0x7fdb580ae800 2024-01-15T18:10:16.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.545+0000 7fdb55600640 1 --2- 172.21.15.27:0/60286247 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdb30067160 0x7fdb30069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:16.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.549+0000 7fdb55600640 1 -- 172.21.15.27:0/60286247 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fdb480826b0 con 0x7fdb580ae800 2024-01-15T18:10:16.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.549+0000 7fdb5cc00640 1 --2- 172.21.15.27:0/60286247 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdb30067160 0x7fdb30069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:16.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.549+0000 7fdb5cc00640 1 --2- 172.21.15.27:0/60286247 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdb30067160 0x7fdb30069620 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fdb5815fdc0 tx=0x7fdb40009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:16.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.553+0000 7fdb55600640 1 -- 172.21.15.27:0/60286247 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fdb48054c60 con 0x7fdb580ae800 2024-01-15T18:10:16.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.841+0000 7fdb60000640 1 -- 172.21.15.27:0/60286247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fdb58160d50 con 0x7fdb580ae800 2024-01-15T18:10:16.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.841+0000 7fdb55600640 1 -- 172.21.15.27:0/60286247 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2427 (secure 0 0 0) 0x7fdb48017090 con 0x7fdb580ae800 2024-01-15T18:10:16.849 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:10:16.849 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:10:16.849 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:10:16.849 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:10:16.849 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:10:16.849 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:10:16.849 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:10:16.849 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:16.849 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:10:16.849 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:10:16.849 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:10:16.849 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:10:16.850 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:10:16.850 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:16.850 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:10:16.850 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:10:16.850 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:10:16.850 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:10:16.850 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:16.850 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 343, 2024-01-15T18:10:16.850 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:10:16.850 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:10:16.850 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:10:16.850 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:10:16.850 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:16.850 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:10:16.850 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 58, 2024-01-15T18:10:16.851 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:10:16.851 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:10:16.851 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:10:16.851 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:10:16.851 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:10:16.851 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:10:16.851 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:16.851 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:10:16.851 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:10:16.851 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:16.851 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:10:16.851 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:10:16.851 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:16.851 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:16.851 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:10:16.852 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:10:16.852 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 1791, 2024-01-15T18:10:16.852 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 7366502886, 2024-01-15T18:10:16.852 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 14448672768, 2024-01-15T18:10:16.852 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 758645440512, 2024-01-15T18:10:16.852 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:10:16.852 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 360852335, 2024-01-15T18:10:16.852 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 345168342, 2024-01-15T18:10:16.852 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 87, 2024-01-15T18:10:16.852 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 93 2024-01-15T18:10:16.852 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:16.852 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:10:16.852 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 38, 2024-01-15T18:10:16.852 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4, 2024-01-15T18:10:16.853 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:10:16.853 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:10:16.853 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:10:16.853 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:10:16.853 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:16.853 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 4, 2024-01-15T18:10:16.853 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:10:16.853 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:10:16.853 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:10:16.853 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 6480 2024-01-15T18:10:16.853 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:16.853 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:16.853 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:10:16.853 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:16.853 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:10:16.854 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:10:16.854 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:10:16.854 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:10:16.854 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:10:16.854 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:10:16.854 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:10:16.854 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:16.854 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:16.854 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:16.854 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:10:16.854 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 13, 2024-01-15T18:10:16.854 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:10:16.226140+0000", 2024-01-15T18:10:16.854 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:16.854 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:16.855 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:10:16.855 INFO:teuthology.orchestra.run.smithi027.stdout: "268d93d2-2eea-412c-b804-608e15344ce7": { 2024-01-15T18:10:16.855 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 19.049% (10s)\n [=====.......................] (remaining: 43s)", 2024-01-15T18:10:16.855 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.19048571586608887, 2024-01-15T18:10:16.855 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:16.855 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:16.855 INFO:teuthology.orchestra.run.smithi027.stdout: "9a27fb80-350b-410b-a65e-91a770cccfdf": { 2024-01-15T18:10:16.855 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "4 ongoing cloning jobs; Avg progress = 33.335% (10s)\n [=========...................] (remaining: 20s)", 2024-01-15T18:10:16.855 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.33335000276565552, 2024-01-15T18:10:16.855 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:16.855 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:16.855 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:16.855 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:10:16.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.845+0000 7fdb60000640 1 -- 172.21.15.27:0/60286247 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdb30067160 msgr2=0x7fdb30069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:16.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.845+0000 7fdb60000640 1 --2- 172.21.15.27:0/60286247 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdb30067160 0x7fdb30069620 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fdb5815fdc0 tx=0x7fdb40009040 comp rx=0 tx=0).stop 2024-01-15T18:10:16.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.845+0000 7fdb60000640 1 -- 172.21.15.27:0/60286247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb580ae800 msgr2=0x7fdb5814ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:16.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.845+0000 7fdb60000640 1 --2- 172.21.15.27:0/60286247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb580ae800 0x7fdb5814ce70 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fdb4800c760 tx=0x7fdb4800cc20 comp rx=0 tx=0).stop 2024-01-15T18:10:16.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.845+0000 7fdb60000640 1 -- 172.21.15.27:0/60286247 shutdown_connections 2024-01-15T18:10:16.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.845+0000 7fdb60000640 1 --2- 172.21.15.27:0/60286247 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdb30067160 0x7fdb30069620 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:16.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.845+0000 7fdb60000640 1 --2- 172.21.15.27:0/60286247 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdb58165830 0x7fdb58163f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:16.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.845+0000 7fdb60000640 1 --2- 172.21.15.27:0/60286247 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb58161590 0x7fdb581639c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:16.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.845+0000 7fdb60000640 1 --2- 172.21.15.27:0/60286247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb580ae800 0x7fdb5814ce70 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:16.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.845+0000 7fdb60000640 1 -- 172.21.15.27:0/60286247 >> 172.21.15.27:0/60286247 conn(0x7fdb580cf520 msgr2=0x7fdb5806aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:16.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.845+0000 7fdb60000640 1 -- 172.21.15.27:0/60286247 shutdown_connections 2024-01-15T18:10:16.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:16.845+0000 7fdb60000640 1 -- 172.21.15.27:0/60286247 wait complete. 2024-01-15T18:10:18.872 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:10:19.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.145+0000 7f304d000640 1 Processor -- start 2024-01-15T18:10:19.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.149+0000 7f304d000640 1 -- start start 2024-01-15T18:10:19.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.149+0000 7f304d000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3048071a70 0x7f30480ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:19.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.149+0000 7f304d000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f304806d5d0 0x7f304806da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:19.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.149+0000 7f304d000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30480ad520 con 0x7f3048161930 2024-01-15T18:10:19.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.149+0000 7f304d000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f304806df80 con 0x7f304806d5d0 2024-01-15T18:10:19.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.149+0000 7f304d000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30480acc60 con 0x7f3048071a70 2024-01-15T18:10:19.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.149+0000 7f3046800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f304806d5d0 0x7f304806da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:19.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.149+0000 7f3046800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f304806d5d0 0x7f304806da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39780/0 (socket says 172.21.15.27:39780) 2024-01-15T18:10:19.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.149+0000 7f3046800640 1 -- 172.21.15.27:0/3376434962 learned_addr learned my addr 172.21.15.27:0/3376434962 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:19.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.149+0000 7f3045400640 1 --2- 172.21.15.27:0/3376434962 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3048071a70 0x7f30480ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:19.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.149+0000 7f3046800640 1 -- 172.21.15.27:0/3376434962 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3048071a70 msgr2=0x7f30480ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:19.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.149+0000 7f3046800640 1 --2- 172.21.15.27:0/3376434962 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3048071a70 0x7f30480ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:19.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.149+0000 7f3046800640 1 -- 172.21.15.27:0/3376434962 >> v1:172.21.15.93:6789/0 conn(0x7f3048161930 legacy=0x7f30480ad9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:19.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.149+0000 7f3046800640 1 -- 172.21.15.27:0/3376434962 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3048071f10 con 0x7f304806d5d0 2024-01-15T18:10:19.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.153+0000 7f3046800640 1 --2- 172.21.15.27:0/3376434962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f304806d5d0 0x7f304806da40 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f3040009e50 tx=0x7f304001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c29222b806058942 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:19.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.153+0000 7f3044a00640 1 -- 172.21.15.27:0/3376434962 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30400a5740 con 0x7f304806d5d0 2024-01-15T18:10:19.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.153+0000 7f3044a00640 1 -- 172.21.15.27:0/3376434962 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30400a58a0 con 0x7f304806d5d0 2024-01-15T18:10:19.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.153+0000 7f3044a00640 1 -- 172.21.15.27:0/3376434962 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30400a5c30 con 0x7f304806d5d0 2024-01-15T18:10:19.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.153+0000 7f304d000640 1 -- 172.21.15.27:0/3376434962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f304806d5d0 msgr2=0x7f304806da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:19.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.153+0000 7f304d000640 1 --2- 172.21.15.27:0/3376434962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f304806d5d0 0x7f304806da40 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f3040009e50 tx=0x7f304001c9d0 comp rx=0 tx=0).stop 2024-01-15T18:10:19.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.157+0000 7f304d000640 1 -- 172.21.15.27:0/3376434962 shutdown_connections 2024-01-15T18:10:19.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.157+0000 7f304d000640 1 --2- 172.21.15.27:0/3376434962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f304806d5d0 0x7f304806da40 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:19.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.157+0000 7f304d000640 1 --2- 172.21.15.27:0/3376434962 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3048071a70 0x7f30480ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:19.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.157+0000 7f304d000640 1 -- 172.21.15.27:0/3376434962 >> 172.21.15.27:0/3376434962 conn(0x7f30480b0d90 msgr2=0x7f30480b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:19.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.157+0000 7f304d000640 1 -- 172.21.15.27:0/3376434962 shutdown_connections 2024-01-15T18:10:19.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.157+0000 7f304d000640 1 -- 172.21.15.27:0/3376434962 wait complete. 2024-01-15T18:10:19.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.161+0000 7f304d000640 1 Processor -- start 2024-01-15T18:10:19.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.161+0000 7f304d000640 1 -- start start 2024-01-15T18:10:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.161+0000 7f304d000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f304806d5d0 0x7f3048153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.161+0000 7f304d000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3048071a70 0x7f30481528b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.161+0000 7f304d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3048161930 0x7f30481644d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.161+0000 7f304d000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3048154410 con 0x7f3048161930 2024-01-15T18:10:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.161+0000 7f304d000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3048154dd0 con 0x7f3048071a70 2024-01-15T18:10:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.161+0000 7f304d000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3048153800 con 0x7f304806d5d0 2024-01-15T18:10:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.161+0000 7f3046800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3048161930 0x7f30481644d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.161+0000 7f3046800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3048161930 0x7f30481644d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39784/0 (socket says 172.21.15.27:39784) 2024-01-15T18:10:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.161+0000 7f3046800640 1 -- 172.21.15.27:0/1176523056 learned_addr learned my addr 172.21.15.27:0/1176523056 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.161+0000 7f3045e00640 1 --2- 172.21.15.27:0/1176523056 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f304806d5d0 0x7f3048153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:19.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.161+0000 7f3046800640 1 -- 172.21.15.27:0/1176523056 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f304806d5d0 msgr2=0x7f3048153100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:19.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.165+0000 7f3046800640 1 --2- 172.21.15.27:0/1176523056 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f304806d5d0 0x7f3048153100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:19.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.165+0000 7f3046800640 1 -- 172.21.15.27:0/1176523056 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3048071a70 msgr2=0x7f30481528b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:19.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.165+0000 7f3046800640 1 --2- 172.21.15.27:0/1176523056 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3048071a70 0x7f30481528b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:19.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.165+0000 7f3046800640 1 -- 172.21.15.27:0/1176523056 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3040009b30 con 0x7f3048161930 2024-01-15T18:10:19.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.165+0000 7f3045e00640 1 --2- 172.21.15.27:0/1176523056 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f304806d5d0 0x7f3048153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:19.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.165+0000 7f3046800640 1 --2- 172.21.15.27:0/1176523056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3048161930 0x7f30481644d0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f30400af040 tx=0x7f3040019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:19.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.165+0000 7f303e000640 1 -- 172.21.15.27:0/1176523056 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30400b8670 con 0x7f3048161930 2024-01-15T18:10:19.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.165+0000 7f303e000640 1 -- 172.21.15.27:0/1176523056 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30400b87f0 con 0x7f3048161930 2024-01-15T18:10:19.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.165+0000 7f303e000640 1 -- 172.21.15.27:0/1176523056 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30400b8ba0 con 0x7f3048161930 2024-01-15T18:10:19.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.165+0000 7f304d000640 1 -- 172.21.15.27:0/1176523056 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3048153ae0 con 0x7f3048161930 2024-01-15T18:10:19.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.165+0000 7f304d000640 1 -- 172.21.15.27:0/1176523056 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30481540a0 con 0x7f3048161930 2024-01-15T18:10:19.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.165+0000 7f304d000640 1 -- 172.21.15.27:0/1176523056 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3048074820 con 0x7f3048161930 2024-01-15T18:10:19.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.169+0000 7f303e000640 1 -- 172.21.15.27:0/1176523056 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7f30400b8d00 con 0x7f3048161930 2024-01-15T18:10:19.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.169+0000 7f303e000640 1 --2- 172.21.15.27:0/1176523056 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3020067190 0x7f3020069650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:19.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.169+0000 7f303e000640 1 -- 172.21.15.27:0/1176523056 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f304011e060 con 0x7f3048161930 2024-01-15T18:10:19.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.185+0000 7f303e000640 1 -- 172.21.15.27:0/1176523056 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f30400ecc40 con 0x7f3048161930 2024-01-15T18:10:19.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.197+0000 7f3045e00640 1 --2- 172.21.15.27:0/1176523056 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3020067190 0x7f3020069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:19.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.201+0000 7f3045e00640 1 --2- 172.21.15.27:0/1176523056 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3020067190 0x7f3020069650 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f30480af5e0 tx=0x7f3038008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:19.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.677+0000 7f304d000640 1 -- 172.21.15.27:0/1176523056 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3048074820 con 0x7f3048161930 2024-01-15T18:10:19.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.677+0000 7f303e000640 1 -- 172.21.15.27:0/1176523056 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f30400bc990 con 0x7f3048161930 2024-01-15T18:10:19.684 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:10:19.684 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:10:19.684 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:10:19.684 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:10:19.684 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:10:19.684 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:10:19.684 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:10:19.684 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:19.685 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:10:19.685 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:10:19.685 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:10:19.685 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:10:19.685 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:10:19.685 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:19.685 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:10:19.685 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:10:19.685 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:10:19.685 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:10:19.685 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:19.685 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 346, 2024-01-15T18:10:19.685 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:10:19.685 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:10:19.686 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:10:19.686 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:10:19.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:19.686 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:10:19.686 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 58, 2024-01-15T18:10:19.686 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:10:19.686 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:10:19.686 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:10:19.686 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:10:19.686 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:10:19.686 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:10:19.686 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:19.686 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:10:19.686 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:10:19.687 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:19.687 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:10:19.687 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:10:19.687 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:19.687 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:19.687 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:10:19.687 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:10:19.687 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 1791, 2024-01-15T18:10:19.687 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 7366502886, 2024-01-15T18:10:19.687 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 14448672768, 2024-01-15T18:10:19.687 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 758645440512, 2024-01-15T18:10:19.687 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:10:19.687 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 360839413, 2024-01-15T18:10:19.687 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 345151543, 2024-01-15T18:10:19.688 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 87, 2024-01-15T18:10:19.688 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 92 2024-01-15T18:10:19.688 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:19.688 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:10:19.688 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 38, 2024-01-15T18:10:19.688 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4, 2024-01-15T18:10:19.688 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:10:19.688 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:10:19.688 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:10:19.688 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:10:19.688 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:19.688 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 4, 2024-01-15T18:10:19.688 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:10:19.688 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:10:19.689 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:10:19.689 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 6480 2024-01-15T18:10:19.689 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:19.689 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:19.689 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:10:19.689 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:19.689 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:10:19.689 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:10:19.689 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:10:19.689 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:10:19.689 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:10:19.689 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:10:19.689 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:10:19.689 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:19.690 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:19.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:19.690 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:10:19.690 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 13, 2024-01-15T18:10:19.690 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:10:16.226140+0000", 2024-01-15T18:10:19.690 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:19.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:19.690 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:10:19.690 INFO:teuthology.orchestra.run.smithi027.stdout: "268d93d2-2eea-412c-b804-608e15344ce7": { 2024-01-15T18:10:19.690 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 23.81% (11s)\n [======......................] (remaining: 36s)", 2024-01-15T18:10:19.690 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.23810000717639923, 2024-01-15T18:10:19.690 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:19.690 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:19.690 INFO:teuthology.orchestra.run.smithi027.stdout: "9a27fb80-350b-410b-a65e-91a770cccfdf": { 2024-01-15T18:10:19.690 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "4 ongoing cloning jobs; Avg progress = 41.668% (11s)\n [===========.................] (remaining: 15s)", 2024-01-15T18:10:19.691 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.41667500138282776, 2024-01-15T18:10:19.691 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:19.691 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:19.691 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:19.691 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:10:19.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.681+0000 7f304d000640 1 -- 172.21.15.27:0/1176523056 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3020067190 msgr2=0x7f3020069650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:19.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.681+0000 7f304d000640 1 --2- 172.21.15.27:0/1176523056 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3020067190 0x7f3020069650 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f30480af5e0 tx=0x7f3038008040 comp rx=0 tx=0).stop 2024-01-15T18:10:19.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.681+0000 7f304d000640 1 -- 172.21.15.27:0/1176523056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3048161930 msgr2=0x7f30481644d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:19.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.681+0000 7f304d000640 1 --2- 172.21.15.27:0/1176523056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3048161930 0x7f30481644d0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f30400af040 tx=0x7f3040019770 comp rx=0 tx=0).stop 2024-01-15T18:10:19.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.681+0000 7f304d000640 1 -- 172.21.15.27:0/1176523056 shutdown_connections 2024-01-15T18:10:19.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.681+0000 7f304d000640 1 --2- 172.21.15.27:0/1176523056 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3020067190 0x7f3020069650 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:19.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.681+0000 7f304d000640 1 --2- 172.21.15.27:0/1176523056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3048161930 0x7f30481644d0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:19.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.681+0000 7f304d000640 1 --2- 172.21.15.27:0/1176523056 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3048071a70 0x7f30481528b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:19.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.681+0000 7f304d000640 1 --2- 172.21.15.27:0/1176523056 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f304806d5d0 0x7f3048153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:19.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.681+0000 7f304d000640 1 -- 172.21.15.27:0/1176523056 >> 172.21.15.27:0/1176523056 conn(0x7f30480b0d90 msgr2=0x7f30480b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:19.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.681+0000 7f304d000640 1 -- 172.21.15.27:0/1176523056 shutdown_connections 2024-01-15T18:10:19.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:19.681+0000 7f304d000640 1 -- 172.21.15.27:0/1176523056 wait complete. 2024-01-15T18:10:21.724 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:10:22.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.013+0000 7fac5e000640 1 Processor -- start 2024-01-15T18:10:22.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.013+0000 7fac5e000640 1 -- start start 2024-01-15T18:10:22.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.017+0000 7fac5e000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac5815b460 0x7fac5815ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:22.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.017+0000 7fac5e000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fac5806d2b0 0x7fac5806d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:22.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.017+0000 7fac5e000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac580cee40 con 0x7fac580ce5c0 2024-01-15T18:10:22.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.017+0000 7fac5e000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac5815b840 con 0x7fac5806d2b0 2024-01-15T18:10:22.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.017+0000 7fac5e000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac580ac8d0 con 0x7fac5815b460 2024-01-15T18:10:22.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.017+0000 7fac57400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac5815b460 0x7fac5815ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.017+0000 7fac57400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac5815b460 0x7fac5815ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:59902/0 (socket says 172.21.15.27:59902) 2024-01-15T18:10:22.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.017+0000 7fac57400640 1 -- 172.21.15.27:0/3337639359 learned_addr learned my addr 172.21.15.27:0/3337639359 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:22.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.017+0000 7fac57400640 1 -- 172.21.15.27:0/3337639359 >> v2:172.21.15.93:3300/0 conn(0x7fac5806d2b0 msgr2=0x7fac5806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:22.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.017+0000 7fac57400640 1 --2- 172.21.15.27:0/3337639359 >> v2:172.21.15.93:3300/0 conn(0x7fac5806d2b0 0x7fac5806d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:22.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.017+0000 7fac57400640 1 -- 172.21.15.27:0/3337639359 >> v1:172.21.15.93:6789/0 conn(0x7fac580ce5c0 legacy=0x7fac5815f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:22.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.017+0000 7fac57400640 1 -- 172.21.15.27:0/3337639359 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac580ac3a0 con 0x7fac5815b460 2024-01-15T18:10:22.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.029+0000 7fac57400640 1 --2- 172.21.15.27:0/3337639359 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac5815b460 0x7fac5815ee80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fac48009870 tx=0x7fac4801c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f97fbf1afdbbe0dd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:22.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.033+0000 7fac56000640 1 -- 172.21.15.27:0/3337639359 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac480aa5f0 con 0x7fac5815b460 2024-01-15T18:10:22.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.033+0000 7fac56000640 1 -- 172.21.15.27:0/3337639359 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac480aa750 con 0x7fac5815b460 2024-01-15T18:10:22.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.033+0000 7fac56000640 1 -- 172.21.15.27:0/3337639359 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac480aaae0 con 0x7fac5815b460 2024-01-15T18:10:22.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.037+0000 7fac5e000640 1 -- 172.21.15.27:0/3337639359 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac5815b460 msgr2=0x7fac5815ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:22.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.037+0000 7fac5e000640 1 --2- 172.21.15.27:0/3337639359 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac5815b460 0x7fac5815ee80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fac48009870 tx=0x7fac4801c4d0 comp rx=0 tx=0).stop 2024-01-15T18:10:22.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.037+0000 7fac5e000640 1 -- 172.21.15.27:0/3337639359 shutdown_connections 2024-01-15T18:10:22.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.037+0000 7fac5e000640 1 --2- 172.21.15.27:0/3337639359 >> v2:172.21.15.93:3300/0 conn(0x7fac5806d2b0 0x7fac5806d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:22.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.037+0000 7fac5e000640 1 --2- 172.21.15.27:0/3337639359 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac5815b460 0x7fac5815ee80 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:22.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.037+0000 7fac5e000640 1 -- 172.21.15.27:0/3337639359 >> 172.21.15.27:0/3337639359 conn(0x7fac58070aa0 msgr2=0x7fac58071950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:22.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.037+0000 7fac5e000640 1 -- 172.21.15.27:0/3337639359 shutdown_connections 2024-01-15T18:10:22.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.037+0000 7fac5e000640 1 -- 172.21.15.27:0/3337639359 wait complete. 2024-01-15T18:10:22.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.037+0000 7fac5e000640 1 Processor -- start 2024-01-15T18:10:22.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.041+0000 7fac5e000640 1 -- start start 2024-01-15T18:10:22.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.041+0000 7fac5e000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac5806d2b0 0x7fac581fd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:22.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.041+0000 7fac5e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac580ce5c0 0x7fac58151680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:22.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.041+0000 7fac5e000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fac5815b460 0x7fac581fdb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:22.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.041+0000 7fac5e000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac581fea40 con 0x7fac580ce5c0 2024-01-15T18:10:22.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.041+0000 7fac5e000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac580b2220 con 0x7fac5815b460 2024-01-15T18:10:22.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.041+0000 7fac5e000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac580b26b0 con 0x7fac5806d2b0 2024-01-15T18:10:22.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.041+0000 7fac56a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac580ce5c0 0x7fac58151680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:22.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.041+0000 7fac56a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac580ce5c0 0x7fac58151680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39798/0 (socket says 172.21.15.27:39798) 2024-01-15T18:10:22.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.041+0000 7fac56a00640 1 -- 172.21.15.27:0/4115353129 learned_addr learned my addr 172.21.15.27:0/4115353129 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:22.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.041+0000 7fac57400640 1 --2- 172.21.15.27:0/4115353129 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac5806d2b0 0x7fac581fd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:22.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.041+0000 7fac57e00640 1 --2- 172.21.15.27:0/4115353129 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fac5815b460 0x7fac581fdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:22.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.041+0000 7fac56a00640 1 -- 172.21.15.27:0/4115353129 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac5806d2b0 msgr2=0x7fac581fd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:22.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.041+0000 7fac56a00640 1 --2- 172.21.15.27:0/4115353129 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac5806d2b0 0x7fac581fd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:22.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.041+0000 7fac56a00640 1 -- 172.21.15.27:0/4115353129 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fac5815b460 msgr2=0x7fac581fdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:22.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.041+0000 7fac56a00640 1 --2- 172.21.15.27:0/4115353129 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fac5815b460 0x7fac581fdb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:22.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.041+0000 7fac56a00640 1 -- 172.21.15.27:0/4115353129 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac480094d0 con 0x7fac580ce5c0 2024-01-15T18:10:22.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.045+0000 7fac56a00640 1 --2- 172.21.15.27:0/4115353129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac580ce5c0 0x7fac58151680 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fac4000c4c0 tx=0x7fac40001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:22.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.045+0000 7fac4f400640 1 -- 172.21.15.27:0/4115353129 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac40005480 con 0x7fac580ce5c0 2024-01-15T18:10:22.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.045+0000 7fac4f400640 1 -- 172.21.15.27:0/4115353129 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac40003600 con 0x7fac580ce5c0 2024-01-15T18:10:22.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.045+0000 7fac4f400640 1 -- 172.21.15.27:0/4115353129 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac40003990 con 0x7fac580ce5c0 2024-01-15T18:10:22.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.045+0000 7fac5e000640 1 -- 172.21.15.27:0/4115353129 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac581fe760 con 0x7fac580ce5c0 2024-01-15T18:10:22.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.045+0000 7fac5e000640 1 -- 172.21.15.27:0/4115353129 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5815ab80 con 0x7fac580ce5c0 2024-01-15T18:10:22.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.045+0000 7fac5e000640 1 -- 172.21.15.27:0/4115353129 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac58074820 con 0x7fac580ce5c0 2024-01-15T18:10:22.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.049+0000 7fac4f400640 1 -- 172.21.15.27:0/4115353129 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7fac40003af0 con 0x7fac580ce5c0 2024-01-15T18:10:22.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.049+0000 7fac4f400640 1 --2- 172.21.15.27:0/4115353129 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fac300671e0 0x7fac300696a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:22.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.049+0000 7fac4f400640 1 -- 172.21.15.27:0/4115353129 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fac40082450 con 0x7fac580ce5c0 2024-01-15T18:10:22.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.053+0000 7fac4f400640 1 -- 172.21.15.27:0/4115353129 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fac40050f10 con 0x7fac580ce5c0 2024-01-15T18:10:22.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.057+0000 7fac57400640 1 --2- 172.21.15.27:0/4115353129 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fac300671e0 0x7fac300696a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:22.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.065+0000 7fac57400640 1 --2- 172.21.15.27:0/4115353129 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fac300671e0 0x7fac300696a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fac480ae000 tx=0x7fac480a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:22.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.477+0000 7fac5e000640 1 -- 172.21.15.27:0/4115353129 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fac5806b270 con 0x7fac580ce5c0 2024-01-15T18:10:22.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.477+0000 7fac4f400640 1 -- 172.21.15.27:0/4115353129 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2428 (secure 0 0 0) 0x7fac40017070 con 0x7fac580ce5c0 2024-01-15T18:10:22.483 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:10:22.483 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:10:22.483 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:10:22.483 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:10:22.483 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:10:22.483 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:10:22.483 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:10:22.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:22.483 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:10:22.483 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:10:22.484 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:10:22.484 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:10:22.484 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:10:22.484 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:22.484 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:10:22.484 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:10:22.484 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:10:22.484 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:10:22.484 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:22.484 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 349, 2024-01-15T18:10:22.484 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:10:22.484 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:10:22.484 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:10:22.484 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:10:22.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:22.485 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:10:22.485 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 58, 2024-01-15T18:10:22.485 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:10:22.485 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:10:22.485 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:10:22.485 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:10:22.485 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:10:22.485 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:10:22.485 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:22.485 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:10:22.485 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:10:22.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:22.485 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:10:22.485 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:10:22.485 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:22.485 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:22.486 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:10:22.486 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:10:22.486 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 1992, 2024-01-15T18:10:22.486 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 8209557990, 2024-01-15T18:10:22.486 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 15967162368, 2024-01-15T18:10:22.486 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 757126950912, 2024-01-15T18:10:22.486 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:10:22.486 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 436071566, 2024-01-15T18:10:22.486 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 415487277, 2024-01-15T18:10:22.486 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 105, 2024-01-15T18:10:22.486 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 109 2024-01-15T18:10:22.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:22.486 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:10:22.486 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 38, 2024-01-15T18:10:22.486 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4, 2024-01-15T18:10:22.486 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:10:22.487 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:10:22.487 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:10:22.487 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:10:22.487 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:22.487 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 4, 2024-01-15T18:10:22.487 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:10:22.487 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:10:22.487 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:10:22.487 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 6480 2024-01-15T18:10:22.487 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:22.487 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:22.487 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:10:22.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:22.487 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:10:22.487 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:10:22.487 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:10:22.488 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:10:22.488 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:10:22.488 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:10:22.488 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:10:22.488 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:22.488 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:22.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:22.488 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:10:22.488 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 13, 2024-01-15T18:10:22.488 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:10:16.226140+0000", 2024-01-15T18:10:22.489 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:22.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:22.489 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:10:22.489 INFO:teuthology.orchestra.run.smithi027.stdout: "268d93d2-2eea-412c-b804-608e15344ce7": { 2024-01-15T18:10:22.489 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 23.81% (11s)\n [======......................] (remaining: 36s)", 2024-01-15T18:10:22.489 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.23810000717639923, 2024-01-15T18:10:22.489 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:22.489 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:22.489 INFO:teuthology.orchestra.run.smithi027.stdout: "9a27fb80-350b-410b-a65e-91a770cccfdf": { 2024-01-15T18:10:22.489 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "4 ongoing cloning jobs; Avg progress = 41.668% (11s)\n [===========.................] (remaining: 15s)", 2024-01-15T18:10:22.489 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.41667500138282776, 2024-01-15T18:10:22.489 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:22.489 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:22.489 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:22.489 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:10:22.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.481+0000 7fac5e000640 1 -- 172.21.15.27:0/4115353129 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fac300671e0 msgr2=0x7fac300696a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:22.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.481+0000 7fac5e000640 1 --2- 172.21.15.27:0/4115353129 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fac300671e0 0x7fac300696a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fac480ae000 tx=0x7fac480a7000 comp rx=0 tx=0).stop 2024-01-15T18:10:22.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.481+0000 7fac5e000640 1 -- 172.21.15.27:0/4115353129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac580ce5c0 msgr2=0x7fac58151680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:22.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.481+0000 7fac5e000640 1 --2- 172.21.15.27:0/4115353129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac580ce5c0 0x7fac58151680 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fac4000c4c0 tx=0x7fac40001e80 comp rx=0 tx=0).stop 2024-01-15T18:10:22.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.481+0000 7fac5e000640 1 -- 172.21.15.27:0/4115353129 shutdown_connections 2024-01-15T18:10:22.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.481+0000 7fac5e000640 1 --2- 172.21.15.27:0/4115353129 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fac300671e0 0x7fac300696a0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:22.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.481+0000 7fac5e000640 1 --2- 172.21.15.27:0/4115353129 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fac5815b460 0x7fac581fdb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:22.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.481+0000 7fac5e000640 1 --2- 172.21.15.27:0/4115353129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac580ce5c0 0x7fac58151680 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:22.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.481+0000 7fac5e000640 1 --2- 172.21.15.27:0/4115353129 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac5806d2b0 0x7fac581fd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:22.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.481+0000 7fac5e000640 1 -- 172.21.15.27:0/4115353129 >> 172.21.15.27:0/4115353129 conn(0x7fac58070aa0 msgr2=0x7fac5814b140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:22.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.481+0000 7fac5e000640 1 -- 172.21.15.27:0/4115353129 shutdown_connections 2024-01-15T18:10:22.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:22.481+0000 7fac5e000640 1 -- 172.21.15.27:0/4115353129 wait complete. 2024-01-15T18:10:24.531 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:10:24.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.721+0000 7ff647400640 1 Processor -- start 2024-01-15T18:10:24.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.721+0000 7ff647400640 1 -- start start 2024-01-15T18:10:24.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.725+0000 7ff647400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff64006a4f0 0x7ff64006a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:24.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.725+0000 7ff647400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff6400b3e00 0x7ff6400b4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:24.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.725+0000 7ff647400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6400b3040 con 0x7ff6401444c0 2024-01-15T18:10:24.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.725+0000 7ff647400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff64006aec0 con 0x7ff64006a4f0 2024-01-15T18:10:24.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.725+0000 7ff647400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6400b3320 con 0x7ff6400b3e00 2024-01-15T18:10:24.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.725+0000 7ff63fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff64006a4f0 0x7ff64006a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:24.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.725+0000 7ff63fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff64006a4f0 0x7ff64006a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56402/0 (socket says 172.21.15.27:56402) 2024-01-15T18:10:24.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.725+0000 7ff63fe00640 1 -- 172.21.15.27:0/2753878486 learned_addr learned my addr 172.21.15.27:0/2753878486 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:24.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.725+0000 7ff63fe00640 1 -- 172.21.15.27:0/2753878486 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff6400b3e00 msgr2=0x7ff6400b4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:24.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.725+0000 7ff63fe00640 1 --2- 172.21.15.27:0/2753878486 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff6400b3e00 0x7ff6400b4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:24.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.725+0000 7ff63fe00640 1 -- 172.21.15.27:0/2753878486 >> v1:172.21.15.93:6789/0 conn(0x7ff6401444c0 legacy=0x7ff6400cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:24.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.725+0000 7ff63fe00640 1 -- 172.21.15.27:0/2753878486 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6400b3600 con 0x7ff64006a4f0 2024-01-15T18:10:24.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.725+0000 7ff63fe00640 1 --2- 172.21.15.27:0/2753878486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff64006a4f0 0x7ff64006a980 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7ff628009870 tx=0x7ff62801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=904d09fda263fb56 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:24.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.729+0000 7ff63f400640 1 -- 172.21.15.27:0/2753878486 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6280ab600 con 0x7ff64006a4f0 2024-01-15T18:10:24.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.729+0000 7ff63f400640 1 -- 172.21.15.27:0/2753878486 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6280a7050 con 0x7ff64006a4f0 2024-01-15T18:10:24.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.729+0000 7ff63f400640 1 -- 172.21.15.27:0/2753878486 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6280ab980 con 0x7ff64006a4f0 2024-01-15T18:10:24.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.729+0000 7ff647400640 1 -- 172.21.15.27:0/2753878486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff64006a4f0 msgr2=0x7ff64006a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:24.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.729+0000 7ff647400640 1 --2- 172.21.15.27:0/2753878486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff64006a4f0 0x7ff64006a980 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7ff628009870 tx=0x7ff62801c440 comp rx=0 tx=0).stop 2024-01-15T18:10:24.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.729+0000 7ff647400640 1 -- 172.21.15.27:0/2753878486 shutdown_connections 2024-01-15T18:10:24.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.729+0000 7ff647400640 1 --2- 172.21.15.27:0/2753878486 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff6400b3e00 0x7ff6400b4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:24.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.729+0000 7ff647400640 1 --2- 172.21.15.27:0/2753878486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff64006a4f0 0x7ff64006a980 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:24.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.729+0000 7ff647400640 1 -- 172.21.15.27:0/2753878486 >> 172.21.15.27:0/2753878486 conn(0x7ff6400b1ba0 msgr2=0x7ff6400b1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:24.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.729+0000 7ff647400640 1 -- 172.21.15.27:0/2753878486 shutdown_connections 2024-01-15T18:10:24.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.729+0000 7ff647400640 1 -- 172.21.15.27:0/2753878486 wait complete. 2024-01-15T18:10:24.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.729+0000 7ff647400640 1 Processor -- start 2024-01-15T18:10:24.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.729+0000 7ff647400640 1 -- start start 2024-01-15T18:10:24.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.733+0000 7ff647400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff64006a4f0 0x7ff64006dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:24.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.733+0000 7ff647400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff6400b3e00 0x7ff64013a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:24.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.733+0000 7ff647400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff6401444c0 0x7ff6401640a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:24.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.733+0000 7ff647400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64015c6d0 con 0x7ff64006a4f0 2024-01-15T18:10:24.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.733+0000 7ff647400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64006e520 con 0x7ff6401444c0 2024-01-15T18:10:24.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.733+0000 7ff647400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff64015ba50 con 0x7ff6400b3e00 2024-01-15T18:10:24.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.733+0000 7ff644a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff64006a4f0 0x7ff64006dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:24.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.733+0000 7ff644a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff64006a4f0 0x7ff64006dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56412/0 (socket says 172.21.15.27:56412) 2024-01-15T18:10:24.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.733+0000 7ff644a00640 1 -- 172.21.15.27:0/882106936 learned_addr learned my addr 172.21.15.27:0/882106936 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:24.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.733+0000 7ff645400640 1 --2- 172.21.15.27:0/882106936 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff6401444c0 0x7ff6401640a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.733+0000 7ff63fe00640 1 --2- 172.21.15.27:0/882106936 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff6400b3e00 0x7ff64013a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.733+0000 7ff644a00640 1 -- 172.21.15.27:0/882106936 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff6400b3e00 msgr2=0x7ff64013a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.733+0000 7ff644a00640 1 --2- 172.21.15.27:0/882106936 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff6400b3e00 0x7ff64013a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.733+0000 7ff644a00640 1 -- 172.21.15.27:0/882106936 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff6401444c0 msgr2=0x7ff6401640a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.733+0000 7ff644a00640 1 --2- 172.21.15.27:0/882106936 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff6401444c0 0x7ff6401640a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.733+0000 7ff644a00640 1 -- 172.21.15.27:0/882106936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6280094d0 con 0x7ff64006a4f0 2024-01-15T18:10:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.733+0000 7ff645400640 1 --2- 172.21.15.27:0/882106936 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff6401444c0 0x7ff6401640a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:10:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.733+0000 7ff644a00640 1 --2- 172.21.15.27:0/882106936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff64006a4f0 0x7ff64006dfe0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7ff63000b3e0 tx=0x7ff63000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.733+0000 7ff63cc00640 1 -- 172.21.15.27:0/882106936 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff630011830 con 0x7ff64006a4f0 2024-01-15T18:10:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.733+0000 7ff63cc00640 1 -- 172.21.15.27:0/882106936 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6300119b0 con 0x7ff64006a4f0 2024-01-15T18:10:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.733+0000 7ff63cc00640 1 -- 172.21.15.27:0/882106936 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff630011d60 con 0x7ff64006a4f0 2024-01-15T18:10:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.733+0000 7ff647400640 1 -- 172.21.15.27:0/882106936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff64015bd30 con 0x7ff64006a4f0 2024-01-15T18:10:24.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.733+0000 7ff647400640 1 -- 172.21.15.27:0/882106936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff64015c2f0 con 0x7ff64006a4f0 2024-01-15T18:10:24.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.737+0000 7ff647400640 1 -- 172.21.15.27:0/882106936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff64004e850 con 0x7ff64006a4f0 2024-01-15T18:10:24.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.737+0000 7ff63cc00640 1 -- 172.21.15.27:0/882106936 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7ff63001a400 con 0x7ff64006a4f0 2024-01-15T18:10:24.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.737+0000 7ff63cc00640 1 --2- 172.21.15.27:0/882106936 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff618067160 0x7ff618069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:24.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.737+0000 7ff63cc00640 1 -- 172.21.15.27:0/882106936 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7ff630082fb0 con 0x7ff64006a4f0 2024-01-15T18:10:24.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.737+0000 7ff63fe00640 1 --2- 172.21.15.27:0/882106936 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff618067160 0x7ff618069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:24.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.741+0000 7ff63fe00640 1 --2- 172.21.15.27:0/882106936 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff618067160 0x7ff618069620 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff6280ad000 tx=0x7ff6280a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:24.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:24.745+0000 7ff63cc00640 1 -- 172.21.15.27:0/882106936 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff630051a70 con 0x7ff64006a4f0 2024-01-15T18:10:25.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:25.069+0000 7ff647400640 1 -- 172.21.15.27:0/882106936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff640067dc0 con 0x7ff64006a4f0 2024-01-15T18:10:25.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:25.069+0000 7ff63cc00640 1 -- 172.21.15.27:0/882106936 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2430 (secure 0 0 0) 0x7ff630015090 con 0x7ff64006a4f0 2024-01-15T18:10:25.078 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:10:25.078 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:10:25.078 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:10:25.078 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:10:25.078 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:10:25.079 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:10:25.079 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:10:25.079 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:25.079 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:10:25.079 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:10:25.079 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:10:25.079 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:10:25.079 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:10:25.079 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:25.079 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:10:25.079 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:10:25.079 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:10:25.079 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:10:25.079 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:25.080 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 352, 2024-01-15T18:10:25.080 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:10:25.080 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:10:25.080 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:10:25.080 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:10:25.080 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:25.080 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:10:25.080 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 58, 2024-01-15T18:10:25.080 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:10:25.080 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:10:25.080 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:10:25.080 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:10:25.080 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:10:25.080 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:10:25.080 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:25.081 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:10:25.081 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:10:25.081 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:25.081 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:10:25.081 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:10:25.081 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:25.081 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:25.081 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:10:25.081 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:10:25.081 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 2572, 2024-01-15T18:10:25.081 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 10642401689, 2024-01-15T18:10:25.081 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 20967976960, 2024-01-15T18:10:25.081 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 752126136320, 2024-01-15T18:10:25.081 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:10:25.082 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 546115349, 2024-01-15T18:10:25.082 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 529039029, 2024-01-15T18:10:25.082 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 130, 2024-01-15T18:10:25.082 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 139 2024-01-15T18:10:25.082 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:25.082 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:10:25.082 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 38, 2024-01-15T18:10:25.082 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4, 2024-01-15T18:10:25.082 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:10:25.082 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:10:25.082 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:10:25.082 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:10:25.082 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:25.082 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 4, 2024-01-15T18:10:25.083 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:10:25.083 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:10:25.083 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:10:25.083 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 6480 2024-01-15T18:10:25.083 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:25.083 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:25.083 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:10:25.083 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:25.083 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:10:25.083 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:10:25.083 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:10:25.083 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:10:25.083 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:10:25.083 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:10:25.083 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:10:25.084 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:25.084 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:25.084 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:25.084 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:10:25.084 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 13, 2024-01-15T18:10:25.084 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:10:16.226140+0000", 2024-01-15T18:10:25.084 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:25.084 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:25.084 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:10:25.084 INFO:teuthology.orchestra.run.smithi027.stdout: "268d93d2-2eea-412c-b804-608e15344ce7": { 2024-01-15T18:10:25.084 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 33.333% (16s)\n [=========...................] (remaining: 33s)", 2024-01-15T18:10:25.084 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.33332857489585876, 2024-01-15T18:10:25.084 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:25.085 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:25.085 INFO:teuthology.orchestra.run.smithi027.stdout: "9a27fb80-350b-410b-a65e-91a770cccfdf": { 2024-01-15T18:10:25.085 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "3 ongoing cloning jobs; Avg progress = 44.443% (16s)\n [============................] (remaining: 20s)", 2024-01-15T18:10:25.085 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.44443333148956299, 2024-01-15T18:10:25.085 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:25.085 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:25.085 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:25.085 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:10:25.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:25.073+0000 7ff647400640 1 -- 172.21.15.27:0/882106936 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff618067160 msgr2=0x7ff618069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:25.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:25.073+0000 7ff647400640 1 --2- 172.21.15.27:0/882106936 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff618067160 0x7ff618069620 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff6280ad000 tx=0x7ff6280a6000 comp rx=0 tx=0).stop 2024-01-15T18:10:25.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:25.073+0000 7ff647400640 1 -- 172.21.15.27:0/882106936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff64006a4f0 msgr2=0x7ff64006dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:25.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:25.073+0000 7ff647400640 1 --2- 172.21.15.27:0/882106936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff64006a4f0 0x7ff64006dfe0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7ff63000b3e0 tx=0x7ff63000b8a0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:25.073+0000 7ff647400640 1 -- 172.21.15.27:0/882106936 shutdown_connections 2024-01-15T18:10:25.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:25.073+0000 7ff647400640 1 --2- 172.21.15.27:0/882106936 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff618067160 0x7ff618069620 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:25.073+0000 7ff647400640 1 --2- 172.21.15.27:0/882106936 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff6401444c0 0x7ff6401640a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:25.073+0000 7ff647400640 1 --2- 172.21.15.27:0/882106936 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff6400b3e00 0x7ff64013a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:25.073+0000 7ff647400640 1 --2- 172.21.15.27:0/882106936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff64006a4f0 0x7ff64006dfe0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:25.073+0000 7ff647400640 1 -- 172.21.15.27:0/882106936 >> 172.21.15.27:0/882106936 conn(0x7ff6400b1ba0 msgr2=0x7ff6401436a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:25.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:25.073+0000 7ff647400640 1 -- 172.21.15.27:0/882106936 shutdown_connections 2024-01-15T18:10:25.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:25.073+0000 7ff647400640 1 -- 172.21.15.27:0/882106936 wait complete. 2024-01-15T18:10:26.826 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:10:26.836 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:10:27.131 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:10:27.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.393+0000 7f33cb600640 1 Processor -- start 2024-01-15T18:10:27.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.393+0000 7f33cb600640 1 -- start start 2024-01-15T18:10:27.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.393+0000 7f33cb600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f33c4071a70 0x7f33c40ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:27.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.393+0000 7f33cb600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c40ad520 con 0x7f33c4161930 2024-01-15T18:10:27.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.393+0000 7f33cb600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c40acc60 con 0x7f33c406d5d0 2024-01-15T18:10:27.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.393+0000 7f33cb600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c4071f10 con 0x7f33c4071a70 2024-01-15T18:10:27.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.393+0000 7f33c8c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f33c4161930 0x7f33c40ad9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58974/0 (socket says 172.21.15.27:58974) 2024-01-15T18:10:27.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.393+0000 7f33c8c00640 1 -- 172.21.15.27:0/3906242943 learned_addr learned my addr 172.21.15.27:0/3906242943 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:10:27.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.397+0000 7f33c3e00640 1 --2- 172.21.15.27:0/3906242943 >> v2:172.21.15.27:3300/0 conn(0x7f33c4071a70 0x7f33c40ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:27.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.397+0000 7f33c3e00640 1 -- 172.21.15.27:0/3906242943 >> v1:172.21.15.93:6789/0 conn(0x7f33c406d5d0 legacy=0x7f33c406d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:27.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.397+0000 7f33c3e00640 1 -- 172.21.15.27:0/3906242943 >> v1:172.21.15.27:6789/0 conn(0x7f33c4161930 legacy=0x7f33c40ad9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:27.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.397+0000 7f33c3e00640 1 -- 172.21.15.27:0/3906242943 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c40aced0 con 0x7f33c4071a70 2024-01-15T18:10:27.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.401+0000 7f33c3e00640 1 --2- 172.21.15.27:0/3906242943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33c4071a70 0x7f33c40ae6b0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f33ac0097c0 tx=0x7f33ac01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14f928080f82017a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:27.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.401+0000 7f33c3400640 1 -- 172.21.15.27:0/3906242943 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ac0a8050 con 0x7f33c4071a70 2024-01-15T18:10:27.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.401+0000 7f33c3400640 1 -- 172.21.15.27:0/3906242943 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ac0ac5f0 con 0x7f33c4071a70 2024-01-15T18:10:27.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.401+0000 7f33c3400640 1 -- 172.21.15.27:0/3906242943 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ac0ac980 con 0x7f33c4071a70 2024-01-15T18:10:27.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.401+0000 7f33cb600640 1 -- 172.21.15.27:0/3906242943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33c4071a70 msgr2=0x7f33c40ae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:27.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.401+0000 7f33cb600640 1 --2- 172.21.15.27:0/3906242943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33c4071a70 0x7f33c40ae6b0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f33ac0097c0 tx=0x7f33ac01c480 comp rx=0 tx=0).stop 2024-01-15T18:10:27.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.401+0000 7f33cb600640 1 -- 172.21.15.27:0/3906242943 shutdown_connections 2024-01-15T18:10:27.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.401+0000 7f33cb600640 1 --2- 172.21.15.27:0/3906242943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33c4071a70 0x7f33c40ae6b0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:27.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.401+0000 7f33cb600640 1 -- 172.21.15.27:0/3906242943 >> 172.21.15.27:0/3906242943 conn(0x7f33c40b0d90 msgr2=0x7f33c40b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:27.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.401+0000 7f33cb600640 1 -- 172.21.15.27:0/3906242943 shutdown_connections 2024-01-15T18:10:27.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.401+0000 7f33cb600640 1 -- 172.21.15.27:0/3906242943 wait complete. 2024-01-15T18:10:27.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33cb600640 1 Processor -- start 2024-01-15T18:10:27.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33cb600640 1 -- start start 2024-01-15T18:10:27.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33cb600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f33c406d5d0 0x7f33c41594e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:27.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33cb600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33c4071a70 0x7f33c415cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:27.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33cb600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33c4161930 0x7f33c4153f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:27.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33cb600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c4157980 con 0x7f33c4161930 2024-01-15T18:10:27.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33cb600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c4156d70 con 0x7f33c4071a70 2024-01-15T18:10:27.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33cb600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c4157050 con 0x7f33c406d5d0 2024-01-15T18:10:27.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33c9600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33c4161930 0x7f33c4153f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:27.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33c9600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33c4161930 0x7f33c4153f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56438/0 (socket says 172.21.15.27:56438) 2024-01-15T18:10:27.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33c9600640 1 -- 172.21.15.27:0/1886813072 learned_addr learned my addr 172.21.15.27:0/1886813072 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:27.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33c3e00640 1 --2- 172.21.15.27:0/1886813072 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33c4071a70 0x7f33c415cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:27.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33c8c00640 1 --2- 172.21.15.27:0/1886813072 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f33c406d5d0 0x7f33c41594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:27.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33c9600640 1 -- 172.21.15.27:0/1886813072 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f33c406d5d0 msgr2=0x7f33c41594e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:27.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33c9600640 1 --2- 172.21.15.27:0/1886813072 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f33c406d5d0 0x7f33c41594e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:27.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33c9600640 1 -- 172.21.15.27:0/1886813072 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33c4071a70 msgr2=0x7f33c415cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:27.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33c9600640 1 --2- 172.21.15.27:0/1886813072 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33c4071a70 0x7f33c415cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:27.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33c9600640 1 -- 172.21.15.27:0/1886813072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ac009420 con 0x7f33c4161930 2024-01-15T18:10:27.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33c8c00640 1 --2- 172.21.15.27:0/1886813072 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f33c406d5d0 0x7f33c41594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:10:27.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33c3e00640 1 --2- 172.21.15.27:0/1886813072 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33c4071a70 0x7f33c415cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:10:27.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33c9600640 1 --2- 172.21.15.27:0/1886813072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33c4161930 0x7f33c4153f60 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f33bc00c770 tx=0x7f33bc00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:27.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33c0c00640 1 -- 172.21.15.27:0/1886813072 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc003b80 con 0x7f33c4161930 2024-01-15T18:10:27.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33c0c00640 1 -- 172.21.15.27:0/1886813072 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33bc003ce0 con 0x7f33c4161930 2024-01-15T18:10:27.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33cb600640 1 -- 172.21.15.27:0/1886813072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c4157330 con 0x7f33c4161930 2024-01-15T18:10:27.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33c0c00640 1 -- 172.21.15.27:0/1886813072 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc00aa90 con 0x7f33c4161930 2024-01-15T18:10:27.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.405+0000 7f33cb600640 1 -- 172.21.15.27:0/1886813072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c4159a20 con 0x7f33c4161930 2024-01-15T18:10:27.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.409+0000 7f33cb600640 1 -- 172.21.15.27:0/1886813072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c4157af0 con 0x7f33c4161930 2024-01-15T18:10:27.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.409+0000 7f33c0c00640 1 -- 172.21.15.27:0/1886813072 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7f33bc00abf0 con 0x7f33c4161930 2024-01-15T18:10:27.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.409+0000 7f33c0c00640 1 --2- 172.21.15.27:0/1886813072 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f339c067160 0x7f339c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:27.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.409+0000 7f33c0c00640 1 -- 172.21.15.27:0/1886813072 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f33bc082ba0 con 0x7f33c4161930 2024-01-15T18:10:27.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.409+0000 7f33c8c00640 1 --2- 172.21.15.27:0/1886813072 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f339c067160 0x7f339c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:27.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.413+0000 7f33c8c00640 1 --2- 172.21.15.27:0/1886813072 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f339c067160 0x7f339c069620 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f33b4005890 tx=0x7f33b4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:27.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.413+0000 7f33c0c00640 1 -- 172.21.15.27:0/1886813072 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f33bc051710 con 0x7f33c4161930 2024-01-15T18:10:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.801+0000 7f33cb600640 1 -- 172.21.15.27:0/1886813072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f33c4074820 con 0x7f33c4161930 2024-01-15T18:10:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.801+0000 7f33c0c00640 1 -- 172.21.15.27:0/1886813072 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2411 (secure 0 0 0) 0x7f33bc054d30 con 0x7f33c4161930 2024-01-15T18:10:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:10:27.806 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:10:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:10:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:10:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:10:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:10:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:10:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:10:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:10:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:10:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:10:27.806 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:10:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:10:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:10:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:10:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:10:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 354, 2024-01-15T18:10:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:10:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:10:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:10:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:10:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:10:27.807 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 58, 2024-01-15T18:10:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:10:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:10:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:10:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:10:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:10:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:10:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:10:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:10:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:10:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:10:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:27.808 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:10:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:10:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 3175, 2024-01-15T18:10:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 13146582376, 2024-01-15T18:10:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 26422722560, 2024-01-15T18:10:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 746671390720, 2024-01-15T18:10:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:10:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 642177095, 2024-01-15T18:10:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 642916048, 2024-01-15T18:10:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 154, 2024-01-15T18:10:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 166 2024-01-15T18:10:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:10:27.809 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 38, 2024-01-15T18:10:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4, 2024-01-15T18:10:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:10:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:10:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:10:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:10:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 4, 2024-01-15T18:10:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:10:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:10:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:10:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 6480 2024-01-15T18:10:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:10:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:27.810 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:10:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:10:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:10:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:10:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:10:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:10:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:10:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:10:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 13, 2024-01-15T18:10:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:10:16.226140+0000", 2024-01-15T18:10:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:27.811 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:10:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: "268d93d2-2eea-412c-b804-608e15344ce7": { 2024-01-15T18:10:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 42.857% (17s)\n [============................] (remaining: 23s)", 2024-01-15T18:10:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.4285714328289032, 2024-01-15T18:10:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: "9a27fb80-350b-410b-a65e-91a770cccfdf": { 2024-01-15T18:10:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "2 ongoing cloning jobs; Avg progress = 50.0% (17s)\n [==============..............] (remaining: 17s)", 2024-01-15T18:10:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.5, 2024-01-15T18:10:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:27.812 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:27.812 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:10:27.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.801+0000 7f33cb600640 1 -- 172.21.15.27:0/1886813072 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f339c067160 msgr2=0x7f339c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:27.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.801+0000 7f33cb600640 1 --2- 172.21.15.27:0/1886813072 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f339c067160 0x7f339c069620 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f33b4005890 tx=0x7f33b4009040 comp rx=0 tx=0).stop 2024-01-15T18:10:27.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.801+0000 7f33cb600640 1 -- 172.21.15.27:0/1886813072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33c4161930 msgr2=0x7f33c4153f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:27.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.801+0000 7f33cb600640 1 --2- 172.21.15.27:0/1886813072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33c4161930 0x7f33c4153f60 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f33bc00c770 tx=0x7f33bc00cc30 comp rx=0 tx=0).stop 2024-01-15T18:10:27.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.801+0000 7f33cb600640 1 -- 172.21.15.27:0/1886813072 shutdown_connections 2024-01-15T18:10:27.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.801+0000 7f33cb600640 1 --2- 172.21.15.27:0/1886813072 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f339c067160 0x7f339c069620 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:27.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.801+0000 7f33cb600640 1 --2- 172.21.15.27:0/1886813072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33c4161930 0x7f33c4153f60 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:27.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.801+0000 7f33cb600640 1 --2- 172.21.15.27:0/1886813072 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f33c4071a70 0x7f33c415cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:27.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.801+0000 7f33cb600640 1 --2- 172.21.15.27:0/1886813072 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f33c406d5d0 0x7f33c41594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:27.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.801+0000 7f33cb600640 1 -- 172.21.15.27:0/1886813072 >> 172.21.15.27:0/1886813072 conn(0x7f33c40b0d90 msgr2=0x7f33c40733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:27.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.801+0000 7f33cb600640 1 -- 172.21.15.27:0/1886813072 shutdown_connections 2024-01-15T18:10:27.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:27.801+0000 7f33cb600640 1 -- 172.21.15.27:0/1886813072 wait complete. 2024-01-15T18:10:29.831 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:10:30.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.053+0000 7f211b800640 1 Processor -- start 2024-01-15T18:10:30.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.053+0000 7f211b800640 1 -- start start 2024-01-15T18:10:30.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f211b800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f2114158d50 0x7f21140b5390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f211b800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f21140ae4a0 0x7f211415b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f211b800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2114159130 con 0x7f2114071c70 2024-01-15T18:10:30.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f211b800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f211415c6b0 con 0x7f2114158d50 2024-01-15T18:10:30.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f211b800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21140b3230 con 0x7f21140ae4a0 2024-01-15T18:10:30.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f211a200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2114071c70 0x7f211406abb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58980/0 (socket says 172.21.15.27:58980) 2024-01-15T18:10:30.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f211a200640 1 -- 172.21.15.27:0/1986435455 learned_addr learned my addr 172.21.15.27:0/1986435455 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:10:30.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f2118e00640 1 --2- 172.21.15.27:0/1986435455 >> v2:172.21.15.93:3300/0 conn(0x7f2114158d50 0x7f21140b5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:30.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f2113400640 1 -- 172.21.15.27:0/1986435455 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1531058284 0 0) 0x7f2114159130 con 0x7f2114071c70 2024-01-15T18:10:30.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f2113400640 1 -- 172.21.15.27:0/1986435455 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20f4003530 con 0x7f2114071c70 2024-01-15T18:10:30.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f2113400640 1 -- 172.21.15.27:0/1986435455 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2890014289 0 0) 0x7f20f4003530 con 0x7f2114071c70 2024-01-15T18:10:30.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f2113400640 1 -- 172.21.15.27:0/1986435455 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f21140ae4a0 msgr2=0x7f211415b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:30.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f2113400640 1 --2- 172.21.15.27:0/1986435455 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f21140ae4a0 0x7f211415b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f2113400640 1 -- 172.21.15.27:0/1986435455 >> v2:172.21.15.93:3300/0 conn(0x7f2114158d50 msgr2=0x7f21140b5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:30.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f2113400640 1 --2- 172.21.15.27:0/1986435455 >> v2:172.21.15.93:3300/0 conn(0x7f2114158d50 0x7f21140b5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f2113400640 1 -- 172.21.15.27:0/1986435455 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f211415bbe0 con 0x7f2114071c70 2024-01-15T18:10:30.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f2113400640 1 -- 172.21.15.27:0/1986435455 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f210c001a00 con 0x7f2114071c70 2024-01-15T18:10:30.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f2113400640 1 -- 172.21.15.27:0/1986435455 >> v1:172.21.15.27:6789/0 conn(0x7f2114071c70 legacy=0x7f211406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:30.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f2113400640 1 --2- 172.21.15.27:0/1986435455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20f4003980 0x7f20f4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f2113400640 1 --2- 172.21.15.27:0/1986435455 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f21140ae4a0 0x7f20f40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f2113400640 1 --2- 172.21.15.27:0/1986435455 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20f4004d60 0x7f20f40051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f2113400640 1 -- 172.21.15.27:0/1986435455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211415c6b0 con 0x7f20f4003980 2024-01-15T18:10:30.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f2113400640 1 -- 172.21.15.27:0/1986435455 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21140b3230 con 0x7f20f4004d60 2024-01-15T18:10:30.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f2113400640 1 -- 172.21.15.27:0/1986435455 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20f400bbf0 con 0x7f21140ae4a0 2024-01-15T18:10:30.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f2118e00640 1 --2- 172.21.15.27:0/1986435455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20f4003980 0x7f20f4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:30.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f2118e00640 1 -- 172.21.15.27:0/1986435455 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f21140ae4a0 msgr2=0x7f20f40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:30.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f2118e00640 1 --2- 172.21.15.27:0/1986435455 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f21140ae4a0 0x7f20f40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f2118e00640 1 -- 172.21.15.27:0/1986435455 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20f4004d60 msgr2=0x7f20f40051e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T18:10:30.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f2118e00640 1 --2- 172.21.15.27:0/1986435455 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20f4004d60 0x7f20f40051e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.057+0000 7f2118e00640 1 -- 172.21.15.27:0/1986435455 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f4003530 con 0x7f20f4003980 2024-01-15T18:10:30.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.065+0000 7f2118e00640 1 --2- 172.21.15.27:0/1986435455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20f4003980 0x7f20f4003da0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f210400a990 tx=0x7f210401c5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee37aafed89bb133 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:30.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.065+0000 7f2113400640 1 -- 172.21.15.27:0/1986435455 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21040ab600 con 0x7f20f4003980 2024-01-15T18:10:30.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.065+0000 7f2113400640 1 -- 172.21.15.27:0/1986435455 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21040a7050 con 0x7f20f4003980 2024-01-15T18:10:30.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.065+0000 7f2113400640 1 -- 172.21.15.27:0/1986435455 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21040ab960 con 0x7f20f4003980 2024-01-15T18:10:30.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.065+0000 7f211b800640 1 -- 172.21.15.27:0/1986435455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20f4003980 msgr2=0x7f20f4003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:30.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.065+0000 7f211b800640 1 --2- 172.21.15.27:0/1986435455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20f4003980 0x7f20f4003da0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f210400a990 tx=0x7f210401c5b0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.065+0000 7f211a200640 1 -- 172.21.15.27:0/1986435455 reap_dead start 2024-01-15T18:10:30.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.065+0000 7f211b800640 1 -- 172.21.15.27:0/1986435455 shutdown_connections 2024-01-15T18:10:30.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.065+0000 7f211b800640 1 -- 172.21.15.27:0/1986435455 >> 172.21.15.27:0/1986435455 conn(0x7f21140b2360 msgr2=0x7f21140b2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:30.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.065+0000 7f211b800640 1 -- 172.21.15.27:0/1986435455 shutdown_connections 2024-01-15T18:10:30.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.065+0000 7f211b800640 1 -- 172.21.15.27:0/1986435455 wait complete. 2024-01-15T18:10:30.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.069+0000 7f211b800640 1 Processor -- start 2024-01-15T18:10:30.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.069+0000 7f211b800640 1 -- start start 2024-01-15T18:10:30.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.069+0000 7f211b800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20f4003980 0x7f21141573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.069+0000 7f211b800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f211415a950 0x7f21141f9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.069+0000 7f211b800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f21141fa080 0x7f21141fa500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.069+0000 7f211b800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2114152860 con 0x7f211415a950 2024-01-15T18:10:30.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.069+0000 7f211b800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211414ddc0 con 0x7f20f4003980 2024-01-15T18:10:30.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.069+0000 7f211b800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2114157aa0 con 0x7f21141fa080 2024-01-15T18:10:30.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.069+0000 7f2113e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f211415a950 0x7f21141f9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:30.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.069+0000 7f2118e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20f4003980 0x7f21141573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:30.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.069+0000 7f2113e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f211415a950 0x7f21141f9b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56456/0 (socket says 172.21.15.27:56456) 2024-01-15T18:10:30.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.069+0000 7f2113e00640 1 -- 172.21.15.27:0/2575249650 learned_addr learned my addr 172.21.15.27:0/2575249650 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:30.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.069+0000 7f211a200640 1 --2- 172.21.15.27:0/2575249650 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f21141fa080 0x7f21141fa500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:30.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.073+0000 7f2113e00640 1 -- 172.21.15.27:0/2575249650 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f21141fa080 msgr2=0x7f21141fa500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:30.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.073+0000 7f2113e00640 1 --2- 172.21.15.27:0/2575249650 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f21141fa080 0x7f21141fa500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.073+0000 7f2113e00640 1 -- 172.21.15.27:0/2575249650 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20f4003980 msgr2=0x7f21141573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:30.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.073+0000 7f2113e00640 1 --2- 172.21.15.27:0/2575249650 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20f4003980 0x7f21141573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.073+0000 7f2113e00640 1 -- 172.21.15.27:0/2575249650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2114157d80 con 0x7f211415a950 2024-01-15T18:10:30.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.073+0000 7f2113e00640 1 --2- 172.21.15.27:0/2575249650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f211415a950 0x7f21141f9b40 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f20fc00b4d0 tx=0x7f20fc00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.073+0000 7f2110c00640 1 -- 172.21.15.27:0/2575249650 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20fc010870 con 0x7f211415a950 2024-01-15T18:10:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.073+0000 7f2110c00640 1 -- 172.21.15.27:0/2575249650 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20fc0109d0 con 0x7f211415a950 2024-01-15T18:10:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.073+0000 7f211b800640 1 -- 172.21.15.27:0/2575249650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2114152a70 con 0x7f211415a950 2024-01-15T18:10:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.073+0000 7f2110c00640 1 -- 172.21.15.27:0/2575249650 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20fc010d60 con 0x7f211415a950 2024-01-15T18:10:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.073+0000 7f211b800640 1 -- 172.21.15.27:0/2575249650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2114159130 con 0x7f211415a950 2024-01-15T18:10:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.073+0000 7f211b800640 1 -- 172.21.15.27:0/2575249650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2114074820 con 0x7f211415a950 2024-01-15T18:10:30.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.073+0000 7f2110c00640 1 -- 172.21.15.27:0/2575249650 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7f20fc00f3c0 con 0x7f211415a950 2024-01-15T18:10:30.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.077+0000 7f2110c00640 1 --2- 172.21.15.27:0/2575249650 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f20ec067190 0x7f20ec069650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.077+0000 7f2110c00640 1 -- 172.21.15.27:0/2575249650 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f20fc0856b0 con 0x7f211415a950 2024-01-15T18:10:30.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.077+0000 7f2118e00640 1 --2- 172.21.15.27:0/2575249650 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f20ec067190 0x7f20ec069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:30.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.077+0000 7f2118e00640 1 --2- 172.21.15.27:0/2575249650 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f20ec067190 0x7f20ec069650 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f21040ad000 tx=0x7f21040046d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:30.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.081+0000 7f2110c00640 1 -- 172.21.15.27:0/2575249650 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f20fc0518d0 con 0x7f211415a950 2024-01-15T18:10:30.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.465+0000 7f211b800640 1 -- 172.21.15.27:0/2575249650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2114165ca0 con 0x7f211415a950 2024-01-15T18:10:30.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.469+0000 7f2110c00640 1 -- 172.21.15.27:0/2575249650 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2429 (secure 0 0 0) 0x7f20fc014990 con 0x7f211415a950 2024-01-15T18:10:30.473 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:10:30.473 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:10:30.473 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:10:30.473 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:10:30.474 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:10:30.474 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:10:30.474 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:10:30.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:30.474 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:10:30.474 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:10:30.474 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:10:30.474 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:10:30.474 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:10:30.474 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:30.474 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:10:30.474 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:10:30.474 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:10:30.474 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:10:30.474 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:30.475 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 357, 2024-01-15T18:10:30.475 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:10:30.475 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:10:30.475 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:10:30.475 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:10:30.475 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:30.475 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:10:30.475 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 58, 2024-01-15T18:10:30.475 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:10:30.475 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:10:30.475 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:10:30.475 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:10:30.475 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:10:30.475 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:10:30.475 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:30.476 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:10:30.476 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:10:30.476 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:30.476 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:10:30.476 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:10:30.476 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:30.476 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:30.476 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:10:30.476 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:10:30.476 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 3175, 2024-01-15T18:10:30.476 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 13146582376, 2024-01-15T18:10:30.476 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 26422722560, 2024-01-15T18:10:30.477 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 746671390720, 2024-01-15T18:10:30.477 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:10:30.477 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 473411148, 2024-01-15T18:10:30.477 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 481476490, 2024-01-15T18:10:30.477 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 113, 2024-01-15T18:10:30.477 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 124 2024-01-15T18:10:30.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:30.477 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:10:30.477 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 38, 2024-01-15T18:10:30.477 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4, 2024-01-15T18:10:30.477 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:10:30.477 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:10:30.477 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:10:30.477 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:10:30.478 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:30.478 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 4, 2024-01-15T18:10:30.478 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:10:30.478 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:10:30.478 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:10:30.478 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 6480 2024-01-15T18:10:30.478 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:30.478 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:30.478 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:10:30.478 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:30.478 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:10:30.478 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:10:30.478 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:10:30.479 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:10:30.479 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:10:30.479 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:10:30.479 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:10:30.479 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:30.479 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:30.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:30.479 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:10:30.479 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 13, 2024-01-15T18:10:30.479 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:10:16.226140+0000", 2024-01-15T18:10:30.479 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:30.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:30.479 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:10:30.479 INFO:teuthology.orchestra.run.smithi027.stdout: "268d93d2-2eea-412c-b804-608e15344ce7": { 2024-01-15T18:10:30.479 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 52.381% (22s)\n [==============..............] (remaining: 20s)", 2024-01-15T18:10:30.479 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.52381426095962524, 2024-01-15T18:10:30.480 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:30.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:30.480 INFO:teuthology.orchestra.run.smithi027.stdout: "9a27fb80-350b-410b-a65e-91a770cccfdf": { 2024-01-15T18:10:30.480 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "2 ongoing cloning jobs; Avg progress = 83.335% (22s)\n [=======================.....] (remaining: 4s)", 2024-01-15T18:10:30.480 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.83335000276565552, 2024-01-15T18:10:30.480 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:30.480 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:30.480 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:30.480 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:10:30.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.469+0000 7f211b800640 1 -- 172.21.15.27:0/2575249650 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f20ec067190 msgr2=0x7f20ec069650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:30.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.469+0000 7f211b800640 1 --2- 172.21.15.27:0/2575249650 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f20ec067190 0x7f20ec069650 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f21040ad000 tx=0x7f21040046d0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.469+0000 7f211b800640 1 -- 172.21.15.27:0/2575249650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f211415a950 msgr2=0x7f21141f9b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:30.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.469+0000 7f211b800640 1 --2- 172.21.15.27:0/2575249650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f211415a950 0x7f21141f9b40 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f20fc00b4d0 tx=0x7f20fc00b990 comp rx=0 tx=0).stop 2024-01-15T18:10:30.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.469+0000 7f211b800640 1 -- 172.21.15.27:0/2575249650 shutdown_connections 2024-01-15T18:10:30.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.469+0000 7f211b800640 1 --2- 172.21.15.27:0/2575249650 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f20ec067190 0x7f20ec069650 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.469+0000 7f211b800640 1 --2- 172.21.15.27:0/2575249650 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f21141fa080 0x7f21141fa500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.469+0000 7f211b800640 1 --2- 172.21.15.27:0/2575249650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f211415a950 0x7f21141f9b40 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.469+0000 7f211b800640 1 --2- 172.21.15.27:0/2575249650 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20f4003980 0x7f21141573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.469+0000 7f211b800640 1 -- 172.21.15.27:0/2575249650 >> 172.21.15.27:0/2575249650 conn(0x7f21140b2360 msgr2=0x7f21140b3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:30.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.473+0000 7f211b800640 1 -- 172.21.15.27:0/2575249650 shutdown_connections 2024-01-15T18:10:30.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:30.473+0000 7f211b800640 1 -- 172.21.15.27:0/2575249650 wait complete. 2024-01-15T18:10:32.518 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:10:32.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.665+0000 7faa7a000640 1 Processor -- start 2024-01-15T18:10:32.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.665+0000 7faa7a000640 1 -- start start 2024-01-15T18:10:32.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.665+0000 7faa7a000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7faa740c92b0 0x7faa7415d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:32.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.665+0000 7faa7a000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7406c4a0 con 0x7faa740701a0 2024-01-15T18:10:32.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.665+0000 7faa7a000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7415c660 con 0x7faa7415c130 2024-01-15T18:10:32.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.665+0000 7faa7a000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa7415c940 con 0x7faa740c92b0 2024-01-15T18:10:32.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.665+0000 7faa73400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7faa740701a0 0x7faa74070580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58994/0 (socket says 172.21.15.27:58994) 2024-01-15T18:10:32.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.665+0000 7faa73400640 1 -- 172.21.15.27:0/2571017071 learned_addr learned my addr 172.21.15.27:0/2571017071 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:10:32.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.665+0000 7faa73e00640 1 --2- 172.21.15.27:0/2571017071 >> v2:172.21.15.93:3300/0 conn(0x7faa740c92b0 0x7faa7415d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:32.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.665+0000 7faa72000640 1 -- 172.21.15.27:0/2571017071 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122171014 0 0) 0x7faa7406c4a0 con 0x7faa740701a0 2024-01-15T18:10:32.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.665+0000 7faa72000640 1 -- 172.21.15.27:0/2571017071 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa54003530 con 0x7faa740701a0 2024-01-15T18:10:32.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.669+0000 7faa72000640 1 -- 172.21.15.27:0/2571017071 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2211300452 0 0) 0x7faa54003530 con 0x7faa740701a0 2024-01-15T18:10:32.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.669+0000 7faa72000640 1 -- 172.21.15.27:0/2571017071 >> v2:172.21.15.93:3300/0 conn(0x7faa740c92b0 msgr2=0x7faa7415d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:32.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.669+0000 7faa72000640 1 --2- 172.21.15.27:0/2571017071 >> v2:172.21.15.93:3300/0 conn(0x7faa740c92b0 0x7faa7415d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:32.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.669+0000 7faa72000640 1 -- 172.21.15.27:0/2571017071 >> v1:172.21.15.93:6789/0 conn(0x7faa7415c130 legacy=0x7faa7406dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:32.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.669+0000 7faa72000640 1 -- 172.21.15.27:0/2571017071 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7415cc20 con 0x7faa740701a0 2024-01-15T18:10:32.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.669+0000 7faa72000640 1 -- 172.21.15.27:0/2571017071 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7faa64001680 con 0x7faa740701a0 2024-01-15T18:10:32.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.669+0000 7faa72000640 1 -- 172.21.15.27:0/2571017071 >> v1:172.21.15.27:6789/0 conn(0x7faa740701a0 legacy=0x7faa74070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:32.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.669+0000 7faa72000640 1 --2- 172.21.15.27:0/2571017071 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faa54003a40 0x7faa54003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:32.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.669+0000 7faa72000640 1 --2- 172.21.15.27:0/2571017071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa540047d0 0x7faa54004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:32.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.669+0000 7faa72000640 1 --2- 172.21.15.27:0/2571017071 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faa540052b0 0x7faa5400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:32.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.669+0000 7faa72000640 1 -- 172.21.15.27:0/2571017071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7415c940 con 0x7faa540047d0 2024-01-15T18:10:32.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.669+0000 7faa72000640 1 -- 172.21.15.27:0/2571017071 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa5400c180 con 0x7faa54003a40 2024-01-15T18:10:32.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.669+0000 7faa72000640 1 -- 172.21.15.27:0/2571017071 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa5400c340 con 0x7faa540052b0 2024-01-15T18:10:32.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.669+0000 7faa72a00640 1 --2- 172.21.15.27:0/2571017071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa540047d0 0x7faa54004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:32.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.669+0000 7faa73400640 1 --2- 172.21.15.27:0/2571017071 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faa54003a40 0x7faa54003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:32.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.669+0000 7faa73e00640 1 --2- 172.21.15.27:0/2571017071 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faa540052b0 0x7faa5400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:32.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.669+0000 7faa72a00640 1 -- 172.21.15.27:0/2571017071 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faa540052b0 msgr2=0x7faa5400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:32.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.669+0000 7faa72a00640 1 --2- 172.21.15.27:0/2571017071 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faa540052b0 0x7faa5400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:32.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.669+0000 7faa72a00640 1 -- 172.21.15.27:0/2571017071 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faa54003a40 msgr2=0x7faa54003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:32.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.669+0000 7faa72a00640 1 --2- 172.21.15.27:0/2571017071 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faa54003a40 0x7faa54003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:32.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.669+0000 7faa72a00640 1 -- 172.21.15.27:0/2571017071 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa54003530 con 0x7faa540047d0 2024-01-15T18:10:32.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.669+0000 7faa73e00640 1 -- 172.21.15.27:0/2571017071 reap_dead start 2024-01-15T18:10:32.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.669+0000 7faa73400640 1 --2- 172.21.15.27:0/2571017071 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faa54003a40 0x7faa54003e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:10:32.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.669+0000 7faa72a00640 1 --2- 172.21.15.27:0/2571017071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa540047d0 0x7faa54004c50 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7faa5c001750 tx=0x7faa5c01c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f46946c285cdbc9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:32.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.673+0000 7faa72000640 1 -- 172.21.15.27:0/2571017071 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c0052d0 con 0x7faa540047d0 2024-01-15T18:10:32.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.673+0000 7faa72000640 1 -- 172.21.15.27:0/2571017071 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa5c0a7050 con 0x7faa540047d0 2024-01-15T18:10:32.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.673+0000 7faa72000640 1 -- 172.21.15.27:0/2571017071 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c003600 con 0x7faa540047d0 2024-01-15T18:10:32.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.673+0000 7faa7a000640 1 -- 172.21.15.27:0/2571017071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa540047d0 msgr2=0x7faa54004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:32.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.673+0000 7faa7a000640 1 --2- 172.21.15.27:0/2571017071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa540047d0 0x7faa54004c50 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7faa5c001750 tx=0x7faa5c01c390 comp rx=0 tx=0).stop 2024-01-15T18:10:32.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.673+0000 7faa7a000640 1 -- 172.21.15.27:0/2571017071 shutdown_connections 2024-01-15T18:10:32.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.673+0000 7faa7a000640 1 --2- 172.21.15.27:0/2571017071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa540047d0 0x7faa54004c50 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:32.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.673+0000 7faa7a000640 1 -- 172.21.15.27:0/2571017071 >> 172.21.15.27:0/2571017071 conn(0x7faa740cec60 msgr2=0x7faa740cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:32.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.673+0000 7faa7a000640 1 -- 172.21.15.27:0/2571017071 shutdown_connections 2024-01-15T18:10:32.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.673+0000 7faa7a000640 1 -- 172.21.15.27:0/2571017071 wait complete. 2024-01-15T18:10:32.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.673+0000 7faa7a000640 1 Processor -- start 2024-01-15T18:10:32.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.673+0000 7faa7a000640 1 -- start start 2024-01-15T18:10:32.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.677+0000 7faa7a000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faa540047d0 0x7faa74155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:32.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.677+0000 7faa7a000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faa74150cc0 0x7faa741e2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:32.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.677+0000 7faa7a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa741e31d0 0x7faa740c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:32.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.677+0000 7faa7a000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa741f1dd0 con 0x7faa741e31d0 2024-01-15T18:10:32.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.677+0000 7faa7a000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa74155e90 con 0x7faa74150cc0 2024-01-15T18:10:32.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.677+0000 7faa7a000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa74151140 con 0x7faa540047d0 2024-01-15T18:10:32.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.677+0000 7faa73e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa741e31d0 0x7faa740c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:32.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.677+0000 7faa73e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa741e31d0 0x7faa740c7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56488/0 (socket says 172.21.15.27:56488) 2024-01-15T18:10:32.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.677+0000 7faa73e00640 1 -- 172.21.15.27:0/3845445230 learned_addr learned my addr 172.21.15.27:0/3845445230 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:32.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.677+0000 7faa72a00640 1 --2- 172.21.15.27:0/3845445230 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faa74150cc0 0x7faa741e2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:32.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.677+0000 7faa73400640 1 --2- 172.21.15.27:0/3845445230 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faa540047d0 0x7faa74155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:32.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.677+0000 7faa73e00640 1 -- 172.21.15.27:0/3845445230 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faa540047d0 msgr2=0x7faa74155350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:32.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.677+0000 7faa73e00640 1 --2- 172.21.15.27:0/3845445230 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faa540047d0 0x7faa74155350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:32.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.677+0000 7faa73e00640 1 -- 172.21.15.27:0/3845445230 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faa74150cc0 msgr2=0x7faa741e2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:32.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.677+0000 7faa73e00640 1 --2- 172.21.15.27:0/3845445230 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faa74150cc0 0x7faa741e2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:32.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.677+0000 7faa73e00640 1 -- 172.21.15.27:0/3845445230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa740c8a50 con 0x7faa741e31d0 2024-01-15T18:10:32.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.677+0000 7faa73400640 1 --2- 172.21.15.27:0/3845445230 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faa540047d0 0x7faa74155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:10:32.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.677+0000 7faa73e00640 1 --2- 172.21.15.27:0/3845445230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa741e31d0 0x7faa740c7dd0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7faa74072ad0 tx=0x7faa6c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:32.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.677+0000 7faa6b400640 1 -- 172.21.15.27:0/3845445230 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6c00e040 con 0x7faa741e31d0 2024-01-15T18:10:32.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.677+0000 7faa6b400640 1 -- 172.21.15.27:0/3845445230 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa6c00f040 con 0x7faa741e31d0 2024-01-15T18:10:32.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.677+0000 7faa6b400640 1 -- 172.21.15.27:0/3845445230 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6c015430 con 0x7faa741e31d0 2024-01-15T18:10:32.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.677+0000 7faa7a000640 1 -- 172.21.15.27:0/3845445230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa741542f0 con 0x7faa741e31d0 2024-01-15T18:10:32.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.677+0000 7faa7a000640 1 -- 172.21.15.27:0/3845445230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa74154870 con 0x7faa741e31d0 2024-01-15T18:10:32.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.677+0000 7faa7a000640 1 -- 172.21.15.27:0/3845445230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa7404e850 con 0x7faa741e31d0 2024-01-15T18:10:32.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.677+0000 7faa6b400640 1 -- 172.21.15.27:0/3845445230 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7faa6c015590 con 0x7faa741e31d0 2024-01-15T18:10:32.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.681+0000 7faa6b400640 1 --2- 172.21.15.27:0/3845445230 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faa4c067160 0x7faa4c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:32.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.681+0000 7faa6b400640 1 -- 172.21.15.27:0/3845445230 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7faa6c083440 con 0x7faa741e31d0 2024-01-15T18:10:32.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.681+0000 7faa73400640 1 --2- 172.21.15.27:0/3845445230 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faa4c067160 0x7faa4c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:32.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.681+0000 7faa73400640 1 --2- 172.21.15.27:0/3845445230 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faa4c067160 0x7faa4c069620 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7faa64007fd0 tx=0x7faa64009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:32.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.685+0000 7faa6b400640 1 -- 172.21.15.27:0/3845445230 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7faa6c051f00 con 0x7faa741e31d0 2024-01-15T18:10:32.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.897+0000 7faa7a000640 1 -- 172.21.15.27:0/3845445230 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7faa740c8a50 con 0x7faa741e31d0 2024-01-15T18:10:32.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.901+0000 7faa6b400640 1 -- 172.21.15.27:0/3845445230 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2429 (secure 0 0 0) 0x7faa740c8a50 con 0x7faa741e31d0 2024-01-15T18:10:32.905 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:10:32.906 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:10:32.906 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:10:32.906 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:10:32.906 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:10:32.906 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:10:32.906 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:10:32.906 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:32.906 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:10:32.906 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:10:32.906 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:10:32.906 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:10:32.906 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:10:32.906 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:32.906 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:10:32.906 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:10:32.907 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:10:32.907 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:10:32.907 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:32.907 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 359, 2024-01-15T18:10:32.907 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:10:32.907 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:10:32.907 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:10:32.907 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:10:32.907 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:32.907 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:10:32.907 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 58, 2024-01-15T18:10:32.907 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:10:32.907 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:10:32.907 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:10:32.908 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:10:32.908 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:10:32.908 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:10:32.908 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:32.908 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:10:32.908 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:10:32.908 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:32.908 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:10:32.908 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:10:32.908 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:32.908 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:32.908 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:10:32.908 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:10:32.908 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 3312, 2024-01-15T18:10:32.909 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 13725396050, 2024-01-15T18:10:32.909 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 27500859392, 2024-01-15T18:10:32.909 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 745593253888, 2024-01-15T18:10:32.909 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:10:32.909 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 523386997, 2024-01-15T18:10:32.909 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 529705784, 2024-01-15T18:10:32.909 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 125, 2024-01-15T18:10:32.909 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 136 2024-01-15T18:10:32.909 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:32.909 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:10:32.909 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 38, 2024-01-15T18:10:32.909 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4, 2024-01-15T18:10:32.909 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:10:32.909 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:10:32.909 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:10:32.910 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:10:32.910 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:32.910 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 4, 2024-01-15T18:10:32.910 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:10:32.910 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:10:32.910 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:10:32.910 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 6480 2024-01-15T18:10:32.910 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:32.910 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:32.910 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:10:32.910 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:32.910 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:10:32.910 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:10:32.910 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:10:32.911 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:10:32.911 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:10:32.911 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:10:32.911 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:10:32.911 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:32.911 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:32.911 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:32.911 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:10:32.911 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 13, 2024-01-15T18:10:32.911 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:10:16.226140+0000", 2024-01-15T18:10:32.911 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:32.911 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:32.911 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:10:32.911 INFO:teuthology.orchestra.run.smithi027.stdout: "268d93d2-2eea-412c-b804-608e15344ce7": { 2024-01-15T18:10:32.911 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 52.381% (22s)\n [==============..............] (remaining: 20s)", 2024-01-15T18:10:32.912 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.52381426095962524, 2024-01-15T18:10:32.912 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:32.912 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:32.912 INFO:teuthology.orchestra.run.smithi027.stdout: "9a27fb80-350b-410b-a65e-91a770cccfdf": { 2024-01-15T18:10:32.912 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "2 ongoing cloning jobs; Avg progress = 83.335% (22s)\n [=======================.....] (remaining: 4s)", 2024-01-15T18:10:32.912 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.83335000276565552, 2024-01-15T18:10:32.912 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:32.912 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:32.912 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:32.912 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:10:32.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.901+0000 7faa7a000640 1 -- 172.21.15.27:0/3845445230 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faa4c067160 msgr2=0x7faa4c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:32.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.901+0000 7faa7a000640 1 --2- 172.21.15.27:0/3845445230 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faa4c067160 0x7faa4c069620 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7faa64007fd0 tx=0x7faa64009040 comp rx=0 tx=0).stop 2024-01-15T18:10:32.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.901+0000 7faa7a000640 1 -- 172.21.15.27:0/3845445230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa741e31d0 msgr2=0x7faa740c7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:32.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.901+0000 7faa7a000640 1 --2- 172.21.15.27:0/3845445230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa741e31d0 0x7faa740c7dd0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7faa74072ad0 tx=0x7faa6c00f580 comp rx=0 tx=0).stop 2024-01-15T18:10:32.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.901+0000 7faa7a000640 1 -- 172.21.15.27:0/3845445230 shutdown_connections 2024-01-15T18:10:32.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.901+0000 7faa7a000640 1 --2- 172.21.15.27:0/3845445230 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faa4c067160 0x7faa4c069620 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:32.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.901+0000 7faa7a000640 1 --2- 172.21.15.27:0/3845445230 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa741e31d0 0x7faa740c7dd0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:32.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.901+0000 7faa7a000640 1 --2- 172.21.15.27:0/3845445230 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faa74150cc0 0x7faa741e2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:32.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.901+0000 7faa7a000640 1 --2- 172.21.15.27:0/3845445230 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faa540047d0 0x7faa74155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:32.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.901+0000 7faa7a000640 1 -- 172.21.15.27:0/3845445230 >> 172.21.15.27:0/3845445230 conn(0x7faa740cec60 msgr2=0x7faa740aea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:32.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.901+0000 7faa7a000640 1 -- 172.21.15.27:0/3845445230 shutdown_connections 2024-01-15T18:10:32.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:32.901+0000 7faa7a000640 1 -- 172.21.15.27:0/3845445230 wait complete. 2024-01-15T18:10:34.929 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:10:35.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.077+0000 7f70d5e00640 1 Processor -- start 2024-01-15T18:10:35.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.077+0000 7f70d5e00640 1 -- start start 2024-01-15T18:10:35.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.077+0000 7f70d5e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f70d0143b90 0x7f70d00cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:35.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.077+0000 7f70d5e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f70d00cf160 0x7f70d006d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:35.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.077+0000 7f70d5e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70d00ce080 con 0x7f70d006d5f0 2024-01-15T18:10:35.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.077+0000 7f70d5e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70d006da80 con 0x7f70d0143b90 2024-01-15T18:10:35.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.077+0000 7f70d5e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70d00acdc0 con 0x7f70d00cf160 2024-01-15T18:10:35.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.077+0000 7f70cf400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f70d006d5f0 0x7f70d0071330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46974/0 (socket says 172.21.15.27:46974) 2024-01-15T18:10:35.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.077+0000 7f70cf400640 1 -- 172.21.15.27:0/3648291861 learned_addr learned my addr 172.21.15.27:0/3648291861 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:10:35.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.077+0000 7f70cea00640 1 --2- 172.21.15.27:0/3648291861 >> v2:172.21.15.93:3300/0 conn(0x7f70d0143b90 0x7f70d00cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:35.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.077+0000 7f70ce000640 1 --2- 172.21.15.27:0/3648291861 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f70d00cf160 0x7f70d006d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:35.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.077+0000 7f70cd600640 1 -- 172.21.15.27:0/3648291861 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217866183 0 0) 0x7f70d00ce080 con 0x7f70d006d5f0 2024-01-15T18:10:35.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.077+0000 7f70cd600640 1 -- 172.21.15.27:0/3648291861 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70b0003530 con 0x7f70d006d5f0 2024-01-15T18:10:35.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.077+0000 7f70cea00640 1 -- 172.21.15.27:0/3648291861 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f70d00cf160 msgr2=0x7f70d006d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:35.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.077+0000 7f70cea00640 1 --2- 172.21.15.27:0/3648291861 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f70d00cf160 0x7f70d006d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.077+0000 7f70cea00640 1 -- 172.21.15.27:0/3648291861 >> v1:172.21.15.27:6789/0 conn(0x7f70d006d5f0 legacy=0x7f70d0071330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:35.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.077+0000 7f70cea00640 1 -- 172.21.15.27:0/3648291861 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d00ad380 con 0x7f70d0143b90 2024-01-15T18:10:35.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.081+0000 7f70cea00640 1 --2- 172.21.15.27:0/3648291861 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f70d0143b90 0x7f70d00cec20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f70c00097c0 tx=0x7f70c001ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae33271f07d0ebd6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:35.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.081+0000 7f70cd600640 1 -- 172.21.15.27:0/3648291861 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c00a8970 con 0x7f70d0143b90 2024-01-15T18:10:35.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.081+0000 7f70cd600640 1 -- 172.21.15.27:0/3648291861 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70c00a8ad0 con 0x7f70d0143b90 2024-01-15T18:10:35.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.081+0000 7f70cd600640 1 -- 172.21.15.27:0/3648291861 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c00a8e60 con 0x7f70d0143b90 2024-01-15T18:10:35.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.081+0000 7f70d5e00640 1 -- 172.21.15.27:0/3648291861 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f70d0143b90 msgr2=0x7f70d00cec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:35.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.081+0000 7f70d5e00640 1 --2- 172.21.15.27:0/3648291861 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f70d0143b90 0x7f70d00cec20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f70c00097c0 tx=0x7f70c001ec70 comp rx=0 tx=0).stop 2024-01-15T18:10:35.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.085+0000 7f70d5e00640 1 -- 172.21.15.27:0/3648291861 shutdown_connections 2024-01-15T18:10:35.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.085+0000 7f70d5e00640 1 --2- 172.21.15.27:0/3648291861 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f70d00cf160 0x7f70d006d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.085+0000 7f70d5e00640 1 --2- 172.21.15.27:0/3648291861 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f70d0143b90 0x7f70d00cec20 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.085+0000 7f70d5e00640 1 -- 172.21.15.27:0/3648291861 >> 172.21.15.27:0/3648291861 conn(0x7f70d00b1ae0 msgr2=0x7f70d00b1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:35.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.085+0000 7f70d5e00640 1 -- 172.21.15.27:0/3648291861 shutdown_connections 2024-01-15T18:10:35.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.085+0000 7f70d5e00640 1 -- 172.21.15.27:0/3648291861 wait complete. 2024-01-15T18:10:35.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.085+0000 7f70d5e00640 1 Processor -- start 2024-01-15T18:10:35.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.085+0000 7f70d5e00640 1 -- start start 2024-01-15T18:10:35.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.085+0000 7f70d5e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f70d006d5f0 0x7f70d0159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:35.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.085+0000 7f70d5e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f70d00cf160 0x7f70d01e5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:35.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.085+0000 7f70d5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70d0143b90 0x7f70d0132940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:35.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.085+0000 7f70d5e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d01e66a0 con 0x7f70d0143b90 2024-01-15T18:10:35.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.085+0000 7f70d5e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d013f760 con 0x7f70d00cf160 2024-01-15T18:10:35.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.085+0000 7f70d5e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70d0132e80 con 0x7f70d006d5f0 2024-01-15T18:10:35.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.085+0000 7f70cf400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70d0143b90 0x7f70d0132940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:35.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.085+0000 7f70cf400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70d0143b90 0x7f70d0132940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35754/0 (socket says 172.21.15.27:35754) 2024-01-15T18:10:35.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.085+0000 7f70cf400640 1 -- 172.21.15.27:0/4138636341 learned_addr learned my addr 172.21.15.27:0/4138636341 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:35.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.085+0000 7f70cea00640 1 --2- 172.21.15.27:0/4138636341 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f70d006d5f0 0x7f70d0159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:35.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.085+0000 7f70ce000640 1 --2- 172.21.15.27:0/4138636341 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f70d00cf160 0x7f70d01e5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:35.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.085+0000 7f70cf400640 1 -- 172.21.15.27:0/4138636341 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f70d006d5f0 msgr2=0x7f70d0159b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:35.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.085+0000 7f70cf400640 1 --2- 172.21.15.27:0/4138636341 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f70d006d5f0 0x7f70d0159b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.085+0000 7f70cf400640 1 -- 172.21.15.27:0/4138636341 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f70d00cf160 msgr2=0x7f70d01e5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:35.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.085+0000 7f70cf400640 1 --2- 172.21.15.27:0/4138636341 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f70d00cf160 0x7f70d01e5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.085+0000 7f70cf400640 1 -- 172.21.15.27:0/4138636341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c0009420 con 0x7f70d0143b90 2024-01-15T18:10:35.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.085+0000 7f70cea00640 1 --2- 172.21.15.27:0/4138636341 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f70d006d5f0 0x7f70d0159b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:10:35.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.089+0000 7f70cf400640 1 --2- 172.21.15.27:0/4138636341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70d0143b90 0x7f70d0132940 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f70c800cb40 tx=0x7f70c800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:35.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.089+0000 7f70c7400640 1 -- 172.21.15.27:0/4138636341 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c800aca0 con 0x7f70d0143b90 2024-01-15T18:10:35.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.089+0000 7f70c7400640 1 -- 172.21.15.27:0/4138636341 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70c800ae00 con 0x7f70d0143b90 2024-01-15T18:10:35.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.089+0000 7f70d5e00640 1 -- 172.21.15.27:0/4138636341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70d01304e0 con 0x7f70d0143b90 2024-01-15T18:10:35.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.089+0000 7f70c7400640 1 -- 172.21.15.27:0/4138636341 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c8012620 con 0x7f70d0143b90 2024-01-15T18:10:35.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.089+0000 7f70d5e00640 1 -- 172.21.15.27:0/4138636341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70d01e63c0 con 0x7f70d0143b90 2024-01-15T18:10:35.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.089+0000 7f70d5e00640 1 -- 172.21.15.27:0/4138636341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70d0067a10 con 0x7f70d0143b90 2024-01-15T18:10:35.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.089+0000 7f70c7400640 1 -- 172.21.15.27:0/4138636341 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7f70c8007070 con 0x7f70d0143b90 2024-01-15T18:10:35.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.093+0000 7f70c7400640 1 --2- 172.21.15.27:0/4138636341 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f70a8067160 0x7f70a8069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:35.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.093+0000 7f70c7400640 1 -- 172.21.15.27:0/4138636341 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f70c80856b0 con 0x7f70d0143b90 2024-01-15T18:10:35.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.093+0000 7f70cea00640 1 --2- 172.21.15.27:0/4138636341 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f70a8067160 0x7f70a8069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:35.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.093+0000 7f70cea00640 1 --2- 172.21.15.27:0/4138636341 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f70a8067160 0x7f70a8069620 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f70c00b2000 tx=0x7f70c0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:35.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.097+0000 7f70c7400640 1 -- 172.21.15.27:0/4138636341 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f70c80519e0 con 0x7f70d0143b90 2024-01-15T18:10:35.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.313+0000 7f70d5e00640 1 -- 172.21.15.27:0/4138636341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f70d004e850 con 0x7f70d0143b90 2024-01-15T18:10:35.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.313+0000 7f70c7400640 1 -- 172.21.15.27:0/4138636341 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2401 (secure 0 0 0) 0x7f70c8017090 con 0x7f70d0143b90 2024-01-15T18:10:35.321 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:10:35.321 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:10:35.321 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:10:35.321 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:10:35.321 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:10:35.321 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:10:35.321 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:10:35.321 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:35.321 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:10:35.321 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:10:35.321 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:10:35.321 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:10:35.322 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:10:35.322 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:35.322 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:10:35.322 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:10:35.322 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:10:35.322 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:10:35.322 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:35.322 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 362, 2024-01-15T18:10:35.322 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:10:35.322 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:10:35.322 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:10:35.322 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:10:35.322 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:35.322 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:10:35.323 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 58, 2024-01-15T18:10:35.323 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:10:35.323 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:10:35.323 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:10:35.323 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:10:35.323 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:10:35.323 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:10:35.323 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:35.323 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:10:35.323 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:10:35.323 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:35.323 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:10:35.323 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:10:35.323 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:35.323 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:35.323 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:10:35.324 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:10:35.324 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 3706, 2024-01-15T18:10:35.324 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 15390660781, 2024-01-15T18:10:35.324 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 31038537728, 2024-01-15T18:10:35.324 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 742055575552, 2024-01-15T18:10:35.324 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:10:35.324 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 380149829, 2024-01-15T18:10:35.324 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 395551385, 2024-01-15T18:10:35.324 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 91, 2024-01-15T18:10:35.324 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 102 2024-01-15T18:10:35.324 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:35.324 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:10:35.324 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 38, 2024-01-15T18:10:35.324 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4, 2024-01-15T18:10:35.325 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:10:35.325 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:10:35.325 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:10:35.325 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:10:35.325 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:35.325 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 4, 2024-01-15T18:10:35.325 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:10:35.325 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:10:35.325 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:10:35.325 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 6480 2024-01-15T18:10:35.325 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:35.325 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:35.325 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:10:35.325 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:35.325 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:10:35.326 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:10:35.326 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:10:35.326 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:10:35.326 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:10:35.326 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:10:35.326 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:10:35.326 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:35.326 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:35.326 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:35.326 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:10:35.326 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 13, 2024-01-15T18:10:35.326 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:10:16.226140+0000", 2024-01-15T18:10:35.326 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:35.326 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:35.326 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:10:35.326 INFO:teuthology.orchestra.run.smithi027.stdout: "268d93d2-2eea-412c-b804-608e15344ce7": { 2024-01-15T18:10:35.326 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 57.143% (28s)\n [================............] (remaining: 21s)", 2024-01-15T18:10:35.327 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.57142859697341919, 2024-01-15T18:10:35.327 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:35.327 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:35.327 INFO:teuthology.orchestra.run.smithi027.stdout: "9a27fb80-350b-410b-a65e-91a770cccfdf": { 2024-01-15T18:10:35.327 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "Subvolume \"subvol_clone91817\" has been 0.0% cloned (28s)\n [............................] ", 2024-01-15T18:10:35.327 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0, 2024-01-15T18:10:35.327 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:35.327 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:35.327 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:35.327 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:10:35.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.317+0000 7f70d5e00640 1 -- 172.21.15.27:0/4138636341 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f70a8067160 msgr2=0x7f70a8069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:35.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.317+0000 7f70d5e00640 1 --2- 172.21.15.27:0/4138636341 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f70a8067160 0x7f70a8069620 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f70c00b2000 tx=0x7f70c0008040 comp rx=0 tx=0).stop 2024-01-15T18:10:35.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.317+0000 7f70d5e00640 1 -- 172.21.15.27:0/4138636341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70d0143b90 msgr2=0x7f70d0132940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:35.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.317+0000 7f70d5e00640 1 --2- 172.21.15.27:0/4138636341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70d0143b90 0x7f70d0132940 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f70c800cb40 tx=0x7f70c800a060 comp rx=0 tx=0).stop 2024-01-15T18:10:35.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.317+0000 7f70d5e00640 1 -- 172.21.15.27:0/4138636341 shutdown_connections 2024-01-15T18:10:35.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.317+0000 7f70d5e00640 1 --2- 172.21.15.27:0/4138636341 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f70a8067160 0x7f70a8069620 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.317+0000 7f70d5e00640 1 --2- 172.21.15.27:0/4138636341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70d0143b90 0x7f70d0132940 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.317+0000 7f70d5e00640 1 --2- 172.21.15.27:0/4138636341 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f70d00cf160 0x7f70d01e5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.317+0000 7f70d5e00640 1 --2- 172.21.15.27:0/4138636341 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f70d006d5f0 0x7f70d0159b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.317+0000 7f70d5e00640 1 -- 172.21.15.27:0/4138636341 >> 172.21.15.27:0/4138636341 conn(0x7f70d00b1ae0 msgr2=0x7f70d00afa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:35.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.317+0000 7f70d5e00640 1 -- 172.21.15.27:0/4138636341 shutdown_connections 2024-01-15T18:10:35.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:35.317+0000 7f70d5e00640 1 -- 172.21.15.27:0/4138636341 wait complete. 2024-01-15T18:10:37.344 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:10:37.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.553+0000 7fd232800640 1 Processor -- start 2024-01-15T18:10:37.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.553+0000 7fd232800640 1 -- start start 2024-01-15T18:10:37.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd232800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fd22c0cdf90 0x7fd22c168450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:37.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd232800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22c0b2770 con 0x7fd22c06bb80 2024-01-15T18:10:37.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd232800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22c06c060 con 0x7fd22c0b2390 2024-01-15T18:10:37.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd232800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd22c06c340 con 0x7fd22c0cdf90 2024-01-15T18:10:37.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22aa00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd22c06bb80 0x7fd22c0cd880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46986/0 (socket says 172.21.15.27:46986) 2024-01-15T18:10:37.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22aa00640 1 -- 172.21.15.27:0/3966901934 learned_addr learned my addr 172.21.15.27:0/3966901934 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:10:37.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22be00640 1 --2- 172.21.15.27:0/3966901934 >> v2:172.21.15.93:3300/0 conn(0x7fd22c0cdf90 0x7fd22c168450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:37.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22a000640 1 -- 172.21.15.27:0/3966901934 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368031642 0 0) 0x7fd22c0b2770 con 0x7fd22c06bb80 2024-01-15T18:10:37.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22a000640 1 -- 172.21.15.27:0/3966901934 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd20c003530 con 0x7fd22c06bb80 2024-01-15T18:10:37.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22a000640 1 -- 172.21.15.27:0/3966901934 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 745523173 0 0) 0x7fd20c003530 con 0x7fd22c06bb80 2024-01-15T18:10:37.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22a000640 1 -- 172.21.15.27:0/3966901934 >> v2:172.21.15.93:3300/0 conn(0x7fd22c0cdf90 msgr2=0x7fd22c168450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:37.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22a000640 1 --2- 172.21.15.27:0/3966901934 >> v2:172.21.15.93:3300/0 conn(0x7fd22c0cdf90 0x7fd22c168450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:37.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22a000640 1 -- 172.21.15.27:0/3966901934 >> v1:172.21.15.93:6789/0 conn(0x7fd22c0b2390 legacy=0x7fd22c0cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:37.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22a000640 1 -- 172.21.15.27:0/3966901934 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22c06c620 con 0x7fd22c06bb80 2024-01-15T18:10:37.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22a000640 1 -- 172.21.15.27:0/3966901934 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fd214001680 con 0x7fd22c06bb80 2024-01-15T18:10:37.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22a000640 1 -- 172.21.15.27:0/3966901934 >> v1:172.21.15.27:6789/0 conn(0x7fd22c06bb80 legacy=0x7fd22c0cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:37.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22a000640 1 --2- 172.21.15.27:0/3966901934 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd20c003a40 0x7fd20c003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:37.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22a000640 1 --2- 172.21.15.27:0/3966901934 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd20c0047d0 0x7fd20c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:37.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22a000640 1 --2- 172.21.15.27:0/3966901934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd20c0051e0 0x7fd20c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:37.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22a000640 1 -- 172.21.15.27:0/3966901934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c06c340 con 0x7fd20c0051e0 2024-01-15T18:10:37.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22a000640 1 -- 172.21.15.27:0/3966901934 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd20c00c110 con 0x7fd20c0047d0 2024-01-15T18:10:37.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22a000640 1 -- 172.21.15.27:0/3966901934 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd20c00c2d0 con 0x7fd20c003a40 2024-01-15T18:10:37.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22be00640 1 --2- 172.21.15.27:0/3966901934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd20c0051e0 0x7fd20c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:37.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22aa00640 1 --2- 172.21.15.27:0/3966901934 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd20c0047d0 0x7fd20c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:37.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22b400640 1 --2- 172.21.15.27:0/3966901934 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd20c003a40 0x7fd20c003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:37.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22be00640 1 -- 172.21.15.27:0/3966901934 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd20c003a40 msgr2=0x7fd20c003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:37.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22be00640 1 --2- 172.21.15.27:0/3966901934 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd20c003a40 0x7fd20c003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:37.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22be00640 1 -- 172.21.15.27:0/3966901934 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd20c0047d0 msgr2=0x7fd20c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:37.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22be00640 1 --2- 172.21.15.27:0/3966901934 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd20c0047d0 0x7fd20c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:37.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.557+0000 7fd22be00640 1 -- 172.21.15.27:0/3966901934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd20c003530 con 0x7fd20c0051e0 2024-01-15T18:10:37.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.561+0000 7fd22be00640 1 -- 172.21.15.27:0/3966901934 reap_dead start 2024-01-15T18:10:37.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.561+0000 7fd22be00640 1 --2- 172.21.15.27:0/3966901934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd20c0051e0 0x7fd20c00b990 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fd22c071990 tx=0x7fd22401cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19c9b4a7abc08cab server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:37.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.561+0000 7fd22a000640 1 -- 172.21.15.27:0/3966901934 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2240a5c50 con 0x7fd20c0051e0 2024-01-15T18:10:37.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.561+0000 7fd22a000640 1 -- 172.21.15.27:0/3966901934 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd22400f040 con 0x7fd20c0051e0 2024-01-15T18:10:37.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.561+0000 7fd22a000640 1 -- 172.21.15.27:0/3966901934 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2240ad440 con 0x7fd20c0051e0 2024-01-15T18:10:37.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.565+0000 7fd232800640 1 -- 172.21.15.27:0/3966901934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd20c0051e0 msgr2=0x7fd20c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:37.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.565+0000 7fd232800640 1 --2- 172.21.15.27:0/3966901934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd20c0051e0 0x7fd20c00b990 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fd22c071990 tx=0x7fd22401cc40 comp rx=0 tx=0).stop 2024-01-15T18:10:37.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.565+0000 7fd232800640 1 -- 172.21.15.27:0/3966901934 shutdown_connections 2024-01-15T18:10:37.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.565+0000 7fd232800640 1 --2- 172.21.15.27:0/3966901934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd20c0051e0 0x7fd20c00b990 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:37.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.565+0000 7fd232800640 1 -- 172.21.15.27:0/3966901934 >> 172.21.15.27:0/3966901934 conn(0x7fd22c0b2a10 msgr2=0x7fd22c0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:37.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.565+0000 7fd232800640 1 -- 172.21.15.27:0/3966901934 shutdown_connections 2024-01-15T18:10:37.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.565+0000 7fd232800640 1 -- 172.21.15.27:0/3966901934 wait complete. 2024-01-15T18:10:37.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.569+0000 7fd232800640 1 Processor -- start 2024-01-15T18:10:37.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.569+0000 7fd232800640 1 -- start start 2024-01-15T18:10:37.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.573+0000 7fd232800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd20c0051e0 0x7fd22c0c7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:37.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.573+0000 7fd232800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd22c15c630 0x7fd22c203050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:37.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.573+0000 7fd232800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd22c203590 0x7fd22c203a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:37.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.573+0000 7fd232800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c0c99f0 con 0x7fd22c15c630 2024-01-15T18:10:37.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.573+0000 7fd232800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c0c9cd0 con 0x7fd22c203590 2024-01-15T18:10:37.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.573+0000 7fd232800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c0c9fb0 con 0x7fd20c0051e0 2024-01-15T18:10:37.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.573+0000 7fd22aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd22c15c630 0x7fd22c203050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:37.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.573+0000 7fd22aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd22c15c630 0x7fd22c203050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35786/0 (socket says 172.21.15.27:35786) 2024-01-15T18:10:37.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.573+0000 7fd22aa00640 1 -- 172.21.15.27:0/4008246622 learned_addr learned my addr 172.21.15.27:0/4008246622 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:37.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.573+0000 7fd22be00640 1 --2- 172.21.15.27:0/4008246622 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd22c203590 0x7fd22c203a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:37.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.573+0000 7fd22b400640 1 --2- 172.21.15.27:0/4008246622 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd20c0051e0 0x7fd22c0c7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:37.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.573+0000 7fd22aa00640 1 -- 172.21.15.27:0/4008246622 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd20c0051e0 msgr2=0x7fd22c0c7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:37.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.573+0000 7fd22aa00640 1 --2- 172.21.15.27:0/4008246622 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd20c0051e0 0x7fd22c0c7340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:37.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.573+0000 7fd22aa00640 1 -- 172.21.15.27:0/4008246622 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd22c203590 msgr2=0x7fd22c203a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:37.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.573+0000 7fd22aa00640 1 --2- 172.21.15.27:0/4008246622 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd22c203590 0x7fd22c203a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:37.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.573+0000 7fd22aa00640 1 -- 172.21.15.27:0/4008246622 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22c0ca290 con 0x7fd22c15c630 2024-01-15T18:10:37.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.573+0000 7fd22aa00640 1 --2- 172.21.15.27:0/4008246622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd22c15c630 0x7fd22c203050 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fd21400e570 tx=0x7fd21400ea30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:37.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.573+0000 7fd223400640 1 -- 172.21.15.27:0/4008246622 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd214012590 con 0x7fd22c15c630 2024-01-15T18:10:37.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.573+0000 7fd223400640 1 -- 172.21.15.27:0/4008246622 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2140126f0 con 0x7fd22c15c630 2024-01-15T18:10:37.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.573+0000 7fd223400640 1 -- 172.21.15.27:0/4008246622 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd214012a80 con 0x7fd22c15c630 2024-01-15T18:10:37.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.577+0000 7fd232800640 1 -- 172.21.15.27:0/4008246622 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd22c0ca570 con 0x7fd22c15c630 2024-01-15T18:10:37.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.577+0000 7fd232800640 1 -- 172.21.15.27:0/4008246622 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd22c0c7a50 con 0x7fd22c15c630 2024-01-15T18:10:37.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.577+0000 7fd232800640 1 -- 172.21.15.27:0/4008246622 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd22c04e850 con 0x7fd22c15c630 2024-01-15T18:10:37.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.577+0000 7fd223400640 1 -- 172.21.15.27:0/4008246622 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7fd214012be0 con 0x7fd22c15c630 2024-01-15T18:10:37.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.581+0000 7fd223400640 1 --2- 172.21.15.27:0/4008246622 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd2040671b0 0x7fd204069670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:37.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.581+0000 7fd223400640 1 -- 172.21.15.27:0/4008246622 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fd214082ad0 con 0x7fd22c15c630 2024-01-15T18:10:37.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.585+0000 7fd22b400640 1 --2- 172.21.15.27:0/4008246622 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd2040671b0 0x7fd204069670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:37.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.585+0000 7fd223400640 1 -- 172.21.15.27:0/4008246622 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd214051680 con 0x7fd22c15c630 2024-01-15T18:10:37.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.609+0000 7fd22b400640 1 --2- 172.21.15.27:0/4008246622 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd2040671b0 0x7fd204069670 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd21c001660 tx=0x7fd21c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:37.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.965+0000 7fd232800640 1 -- 172.21.15.27:0/4008246622 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd22c0afd00 con 0x7fd22c15c630 2024-01-15T18:10:37.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.965+0000 7fd223400640 1 -- 172.21.15.27:0/4008246622 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2392 (secure 0 0 0) 0x7fd214054ca0 con 0x7fd22c15c630 2024-01-15T18:10:37.972 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:10:37.972 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:10:37.972 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:10:37.972 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:10:37.973 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:10:37.973 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:10:37.973 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:10:37.973 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:37.973 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:10:37.973 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:10:37.973 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:10:37.973 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:10:37.973 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:10:37.973 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:37.973 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:10:37.973 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:10:37.973 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:10:37.973 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:10:37.974 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:37.974 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 364, 2024-01-15T18:10:37.974 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:10:37.974 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:10:37.974 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:10:37.974 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:10:37.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:37.974 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:10:37.974 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 58, 2024-01-15T18:10:37.974 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:10:37.974 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:10:37.974 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:10:37.974 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:10:37.974 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:10:37.974 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:10:37.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:37.975 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:10:37.975 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:10:37.975 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:37.975 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:10:37.975 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:10:37.975 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:37.975 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:37.975 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:10:37.975 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:10:37.975 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 3877, 2024-01-15T18:10:37.975 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 16108008922, 2024-01-15T18:10:37.975 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 32674566144, 2024-01-15T18:10:37.975 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 740419547136, 2024-01-15T18:10:37.976 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:10:37.976 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 409161675, 2024-01-15T18:10:37.976 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 421430464, 2024-01-15T18:10:37.976 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 99, 2024-01-15T18:10:37.976 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 112 2024-01-15T18:10:37.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:37.976 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:10:37.976 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 38, 2024-01-15T18:10:37.976 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4, 2024-01-15T18:10:37.976 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:10:37.976 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:10:37.976 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:10:37.976 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:10:37.976 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:37.977 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 4, 2024-01-15T18:10:37.977 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:10:37.977 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:10:37.977 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:10:37.977 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 6480 2024-01-15T18:10:37.977 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:37.977 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:37.977 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:10:37.977 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:37.977 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:10:37.977 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:10:37.977 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:10:37.977 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:10:37.977 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:10:37.977 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:10:37.977 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:10:37.978 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:37.978 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:37.978 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:37.978 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:10:37.978 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 13, 2024-01-15T18:10:37.978 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:10:16.226140+0000", 2024-01-15T18:10:37.978 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:37.978 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:37.978 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:10:37.978 INFO:teuthology.orchestra.run.smithi027.stdout: "268d93d2-2eea-412c-b804-608e15344ce7": { 2024-01-15T18:10:37.978 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 57.143% (29s)\n [================............] (remaining: 22s)", 2024-01-15T18:10:37.978 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.57142859697341919, 2024-01-15T18:10:37.978 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:37.978 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:37.978 INFO:teuthology.orchestra.run.smithi027.stdout: "9a27fb80-350b-410b-a65e-91a770cccfdf": { 2024-01-15T18:10:37.978 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "2 ongoing cloning jobs; Avg progress = 0.0% (29s)\n [............................] ", 2024-01-15T18:10:37.979 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0, 2024-01-15T18:10:37.979 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:37.979 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:37.979 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:37.979 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:10:37.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.969+0000 7fd232800640 1 -- 172.21.15.27:0/4008246622 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd2040671b0 msgr2=0x7fd204069670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:37.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.969+0000 7fd232800640 1 --2- 172.21.15.27:0/4008246622 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd2040671b0 0x7fd204069670 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd21c001660 tx=0x7fd21c009040 comp rx=0 tx=0).stop 2024-01-15T18:10:37.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.969+0000 7fd232800640 1 -- 172.21.15.27:0/4008246622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd22c15c630 msgr2=0x7fd22c203050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:37.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.969+0000 7fd232800640 1 --2- 172.21.15.27:0/4008246622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd22c15c630 0x7fd22c203050 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fd21400e570 tx=0x7fd21400ea30 comp rx=0 tx=0).stop 2024-01-15T18:10:37.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.969+0000 7fd232800640 1 -- 172.21.15.27:0/4008246622 shutdown_connections 2024-01-15T18:10:37.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.969+0000 7fd232800640 1 --2- 172.21.15.27:0/4008246622 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd2040671b0 0x7fd204069670 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:37.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.969+0000 7fd232800640 1 --2- 172.21.15.27:0/4008246622 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd22c203590 0x7fd22c203a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:37.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.969+0000 7fd232800640 1 --2- 172.21.15.27:0/4008246622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd22c15c630 0x7fd22c203050 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:37.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.969+0000 7fd232800640 1 --2- 172.21.15.27:0/4008246622 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd20c0051e0 0x7fd22c0c7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:37.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.969+0000 7fd232800640 1 -- 172.21.15.27:0/4008246622 >> 172.21.15.27:0/4008246622 conn(0x7fd22c0b2a10 msgr2=0x7fd22c0c3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:37.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.969+0000 7fd232800640 1 -- 172.21.15.27:0/4008246622 shutdown_connections 2024-01-15T18:10:37.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:37.969+0000 7fd232800640 1 -- 172.21.15.27:0/4008246622 wait complete. 2024-01-15T18:10:40.014 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:10:40.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.221+0000 7fbc46200640 1 Processor -- start 2024-01-15T18:10:40.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.221+0000 7fbc46200640 1 -- start start 2024-01-15T18:10:40.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.225+0000 7fbc46200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbc40158d50 0x7fbc400b5390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:40.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.225+0000 7fbc46200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc40159130 con 0x7fbc40072380 2024-01-15T18:10:40.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.225+0000 7fbc46200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4006a720 con 0x7fbc400ae4a0 2024-01-15T18:10:40.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.225+0000 7fbc46200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc4015c6b0 con 0x7fbc40158d50 2024-01-15T18:10:40.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.225+0000 7fbc3f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbc40158d50 0x7fbc400b5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:40.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.225+0000 7fbc3f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbc40158d50 0x7fbc400b5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35790/0 (socket says 172.21.15.27:35790) 2024-01-15T18:10:40.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.225+0000 7fbc3f400640 1 -- 172.21.15.27:0/736754728 learned_addr learned my addr 172.21.15.27:0/736754728 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:40.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.225+0000 7fbc3f400640 1 -- 172.21.15.27:0/736754728 >> v1:172.21.15.93:6789/0 conn(0x7fbc400ae4a0 legacy=0x7fbc40071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:40.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.225+0000 7fbc3f400640 1 -- 172.21.15.27:0/736754728 >> v1:172.21.15.27:6789/0 conn(0x7fbc40072380 legacy=0x7fbc4006abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:40.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.225+0000 7fbc3f400640 1 -- 172.21.15.27:0/736754728 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc400ae900 con 0x7fbc40158d50 2024-01-15T18:10:40.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.229+0000 7fbc3f400640 1 --2- 172.21.15.27:0/736754728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc40158d50 0x7fbc400b5390 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fbc30009870 tx=0x7fbc3001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1f00db0a6690069 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:40.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.233+0000 7fbc3e000640 1 -- 172.21.15.27:0/736754728 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc300ac620 con 0x7fbc40158d50 2024-01-15T18:10:40.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.233+0000 7fbc3e000640 1 -- 172.21.15.27:0/736754728 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc300ac780 con 0x7fbc40158d50 2024-01-15T18:10:40.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.233+0000 7fbc3e000640 1 -- 172.21.15.27:0/736754728 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc300acb10 con 0x7fbc40158d50 2024-01-15T18:10:40.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.233+0000 7fbc46200640 1 -- 172.21.15.27:0/736754728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc40158d50 msgr2=0x7fbc400b5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:40.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.233+0000 7fbc46200640 1 --2- 172.21.15.27:0/736754728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc40158d50 0x7fbc400b5390 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fbc30009870 tx=0x7fbc3001c500 comp rx=0 tx=0).stop 2024-01-15T18:10:40.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.233+0000 7fbc46200640 1 -- 172.21.15.27:0/736754728 shutdown_connections 2024-01-15T18:10:40.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.233+0000 7fbc46200640 1 --2- 172.21.15.27:0/736754728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc40158d50 0x7fbc400b5390 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.233+0000 7fbc46200640 1 -- 172.21.15.27:0/736754728 >> 172.21.15.27:0/736754728 conn(0x7fbc400b2360 msgr2=0x7fbc400b2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:40.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.233+0000 7fbc46200640 1 -- 172.21.15.27:0/736754728 shutdown_connections 2024-01-15T18:10:40.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.233+0000 7fbc46200640 1 -- 172.21.15.27:0/736754728 wait complete. 2024-01-15T18:10:40.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.237+0000 7fbc46200640 1 Processor -- start 2024-01-15T18:10:40.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.237+0000 7fbc46200640 1 -- start start 2024-01-15T18:10:40.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.241+0000 7fbc46200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc40072380 0x7fbc4013e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:40.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.241+0000 7fbc46200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc400ae4a0 0x7fbc40149f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:40.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.241+0000 7fbc46200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc40158d50 0x7fbc40142810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:40.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.241+0000 7fbc46200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4015d2c0 con 0x7fbc400ae4a0 2024-01-15T18:10:40.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.241+0000 7fbc46200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc401431b0 con 0x7fbc40072380 2024-01-15T18:10:40.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.241+0000 7fbc46200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4015c920 con 0x7fbc40158d50 2024-01-15T18:10:40.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.241+0000 7fbc3ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc400ae4a0 0x7fbc40149f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:40.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.241+0000 7fbc3ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc400ae4a0 0x7fbc40149f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35794/0 (socket says 172.21.15.27:35794) 2024-01-15T18:10:40.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.241+0000 7fbc3ea00640 1 -- 172.21.15.27:0/2781150753 learned_addr learned my addr 172.21.15.27:0/2781150753 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:40.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.241+0000 7fbc44c00640 1 --2- 172.21.15.27:0/2781150753 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc40158d50 0x7fbc40142810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:40.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.241+0000 7fbc3ea00640 1 -- 172.21.15.27:0/2781150753 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc40158d50 msgr2=0x7fbc40142810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:40.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.241+0000 7fbc3ea00640 1 --2- 172.21.15.27:0/2781150753 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc40158d50 0x7fbc40142810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.241+0000 7fbc3ea00640 1 -- 172.21.15.27:0/2781150753 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc40072380 msgr2=0x7fbc4013e010 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-15T18:10:40.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.241+0000 7fbc3ea00640 1 --2- 172.21.15.27:0/2781150753 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc40072380 0x7fbc4013e010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.241+0000 7fbc3ea00640 1 -- 172.21.15.27:0/2781150753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc300094d0 con 0x7fbc400ae4a0 2024-01-15T18:10:40.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.241+0000 7fbc3ea00640 1 --2- 172.21.15.27:0/2781150753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc400ae4a0 0x7fbc40149f20 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800c490 tx=0x7fbc28001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:40.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.241+0000 7fbc37400640 1 -- 172.21.15.27:0/2781150753 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc28003600 con 0x7fbc400ae4a0 2024-01-15T18:10:40.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.241+0000 7fbc46200640 1 -- 172.21.15.27:0/2781150753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc4015cc00 con 0x7fbc400ae4a0 2024-01-15T18:10:40.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.241+0000 7fbc37400640 1 -- 172.21.15.27:0/2781150753 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc28003780 con 0x7fbc400ae4a0 2024-01-15T18:10:40.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.241+0000 7fbc37400640 1 -- 172.21.15.27:0/2781150753 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc28003b70 con 0x7fbc400ae4a0 2024-01-15T18:10:40.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.241+0000 7fbc46200640 1 -- 172.21.15.27:0/2781150753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc4013e550 con 0x7fbc400ae4a0 2024-01-15T18:10:40.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.245+0000 7fbc46200640 1 -- 172.21.15.27:0/2781150753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc10005260 con 0x7fbc400ae4a0 2024-01-15T18:10:40.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.245+0000 7fbc37400640 1 -- 172.21.15.27:0/2781150753 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7fbc28003cd0 con 0x7fbc400ae4a0 2024-01-15T18:10:40.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.245+0000 7fbc37400640 1 --2- 172.21.15.27:0/2781150753 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbc18067160 0x7fbc18069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:40.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.245+0000 7fbc37400640 1 -- 172.21.15.27:0/2781150753 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fbc280823f0 con 0x7fbc400ae4a0 2024-01-15T18:10:40.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.249+0000 7fbc3f400640 1 --2- 172.21.15.27:0/2781150753 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbc18067160 0x7fbc18069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:40.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.249+0000 7fbc37400640 1 -- 172.21.15.27:0/2781150753 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbc28050f50 con 0x7fbc400ae4a0 2024-01-15T18:10:40.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.249+0000 7fbc3f400640 1 --2- 172.21.15.27:0/2781150753 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbc18067160 0x7fbc18069620 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fbc300ae000 tx=0x7fbc300a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:40.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.581+0000 7fbc46200640 1 -- 172.21.15.27:0/2781150753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbc100057e0 con 0x7fbc400ae4a0 2024-01-15T18:10:40.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.585+0000 7fbc37400640 1 -- 172.21.15.27:0/2781150753 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2427 (secure 0 0 0) 0x7fbc28017070 con 0x7fbc400ae4a0 2024-01-15T18:10:40.589 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:10:40.589 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:10:40.589 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:10:40.589 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:10:40.589 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:10:40.589 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:10:40.589 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:10:40.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:40.589 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:10:40.590 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:10:40.590 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:10:40.590 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:10:40.590 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:10:40.590 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:40.590 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:10:40.590 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:10:40.590 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:10:40.590 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:10:40.590 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:40.590 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 367, 2024-01-15T18:10:40.590 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:10:40.590 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:10:40.590 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:10:40.590 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:10:40.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:40.591 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:10:40.591 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 58, 2024-01-15T18:10:40.591 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:10:40.591 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:10:40.591 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:10:40.591 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:10:40.591 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:10:40.591 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:10:40.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:40.591 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:10:40.591 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:10:40.591 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:40.591 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:10:40.591 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:10:40.591 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:40.592 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:40.592 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:10:40.592 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:10:40.592 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 3877, 2024-01-15T18:10:40.592 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 16108008922, 2024-01-15T18:10:40.592 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 32674566144, 2024-01-15T18:10:40.592 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 740419547136, 2024-01-15T18:10:40.592 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:10:40.592 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 238994977, 2024-01-15T18:10:40.592 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 246705122, 2024-01-15T18:10:40.592 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 57, 2024-01-15T18:10:40.592 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 66 2024-01-15T18:10:40.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:40.592 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:10:40.593 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 38, 2024-01-15T18:10:40.593 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4, 2024-01-15T18:10:40.593 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:10:40.593 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:10:40.593 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:10:40.593 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:10:40.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:40.593 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 4, 2024-01-15T18:10:40.593 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:10:40.593 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:10:40.593 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:10:40.593 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 6480 2024-01-15T18:10:40.593 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:40.593 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:40.593 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:10:40.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:40.594 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:10:40.594 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:10:40.594 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:10:40.594 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:10:40.594 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:10:40.594 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:10:40.594 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:10:40.594 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:40.594 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:40.594 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:40.594 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:10:40.594 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 13, 2024-01-15T18:10:40.594 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:10:16.226140+0000", 2024-01-15T18:10:40.595 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:40.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:40.595 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:10:40.595 INFO:teuthology.orchestra.run.smithi027.stdout: "268d93d2-2eea-412c-b804-608e15344ce7": { 2024-01-15T18:10:40.595 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 61.904% (31s)\n [=================...........] (remaining: 19s)", 2024-01-15T18:10:40.595 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.61904287338256836, 2024-01-15T18:10:40.595 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:40.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:40.595 INFO:teuthology.orchestra.run.smithi027.stdout: "9a27fb80-350b-410b-a65e-91a770cccfdf": { 2024-01-15T18:10:40.595 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "2 ongoing cloning jobs; Avg progress = 16.665% (31s)\n [====........................] (remaining: 2m)", 2024-01-15T18:10:40.595 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.16664999723434448, 2024-01-15T18:10:40.595 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:40.595 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:40.595 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:40.595 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:10:40.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.585+0000 7fbc46200640 1 -- 172.21.15.27:0/2781150753 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbc18067160 msgr2=0x7fbc18069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:40.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.585+0000 7fbc46200640 1 --2- 172.21.15.27:0/2781150753 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbc18067160 0x7fbc18069620 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fbc300ae000 tx=0x7fbc300a7000 comp rx=0 tx=0).stop 2024-01-15T18:10:40.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.585+0000 7fbc46200640 1 -- 172.21.15.27:0/2781150753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc400ae4a0 msgr2=0x7fbc40149f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:40.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.585+0000 7fbc46200640 1 --2- 172.21.15.27:0/2781150753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc400ae4a0 0x7fbc40149f20 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800c490 tx=0x7fbc28001e80 comp rx=0 tx=0).stop 2024-01-15T18:10:40.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.585+0000 7fbc46200640 1 -- 172.21.15.27:0/2781150753 shutdown_connections 2024-01-15T18:10:40.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.585+0000 7fbc46200640 1 --2- 172.21.15.27:0/2781150753 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbc18067160 0x7fbc18069620 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.585+0000 7fbc46200640 1 --2- 172.21.15.27:0/2781150753 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc40158d50 0x7fbc40142810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.585+0000 7fbc46200640 1 --2- 172.21.15.27:0/2781150753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc400ae4a0 0x7fbc40149f20 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.585+0000 7fbc46200640 1 --2- 172.21.15.27:0/2781150753 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc40072380 0x7fbc4013e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.585+0000 7fbc46200640 1 -- 172.21.15.27:0/2781150753 >> 172.21.15.27:0/2781150753 conn(0x7fbc400b2360 msgr2=0x7fbc40071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:40.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.585+0000 7fbc46200640 1 -- 172.21.15.27:0/2781150753 shutdown_connections 2024-01-15T18:10:40.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:40.585+0000 7fbc46200640 1 -- 172.21.15.27:0/2781150753 wait complete. 2024-01-15T18:10:42.622 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:10:42.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.885+0000 7f1d1b600640 1 Processor -- start 2024-01-15T18:10:42.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.885+0000 7f1d1b600640 1 -- start start 2024-01-15T18:10:42.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.889+0000 7f1d1b600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f1d1415db90 0x7f1d140b4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:42.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.889+0000 7f1d1b600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1d140cca00 0x7f1d1406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:42.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.889+0000 7f1d1b600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d1415a460 con 0x7f1d1415a080 2024-01-15T18:10:42.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.889+0000 7f1d1b600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d1406e8c0 con 0x7f1d140cca00 2024-01-15T18:10:42.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.889+0000 7f1d1b600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d1406dcd0 con 0x7f1d1415db90 2024-01-15T18:10:42.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.889+0000 7f1d1a000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1d140cca00 0x7f1d1406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:42.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.889+0000 7f1d13e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f1d1415db90 0x7f1d140b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:42.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.889+0000 7f1d1a000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1d140cca00 0x7f1d1406ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35800/0 (socket says 172.21.15.27:35800) 2024-01-15T18:10:42.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.889+0000 7f1d1a000640 1 -- 172.21.15.27:0/1987877564 learned_addr learned my addr 172.21.15.27:0/1987877564 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:42.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.889+0000 7f1d13e00640 1 -- 172.21.15.27:0/1987877564 >> v2:172.21.15.27:3300/0 conn(0x7f1d140cca00 msgr2=0x7f1d1406ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:42.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.889+0000 7f1d13e00640 1 --2- 172.21.15.27:0/1987877564 >> v2:172.21.15.27:3300/0 conn(0x7f1d140cca00 0x7f1d1406ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:42.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.889+0000 7f1d13e00640 1 -- 172.21.15.27:0/1987877564 >> v1:172.21.15.93:6789/0 conn(0x7f1d1415a080 legacy=0x7f1d140b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:42.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.889+0000 7f1d13e00640 1 -- 172.21.15.27:0/1987877564 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d1406df20 con 0x7f1d1415db90 2024-01-15T18:10:42.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.893+0000 7f1d13e00640 1 --2- 172.21.15.27:0/1987877564 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1d1415db90 0x7f1d140b4110 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f1d14073d90 tx=0x7f1cfc01c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=581a85c20a1e510c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:42.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.893+0000 7f1d13400640 1 -- 172.21.15.27:0/1987877564 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc0ab660 con 0x7f1d1415db90 2024-01-15T18:10:42.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.893+0000 7f1d13400640 1 -- 172.21.15.27:0/1987877564 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cfc0ab7c0 con 0x7f1d1415db90 2024-01-15T18:10:42.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.893+0000 7f1d13400640 1 -- 172.21.15.27:0/1987877564 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc0abb50 con 0x7f1d1415db90 2024-01-15T18:10:42.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.897+0000 7f1d1b600640 1 -- 172.21.15.27:0/1987877564 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1d1415db90 msgr2=0x7f1d140b4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:42.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.897+0000 7f1d1b600640 1 --2- 172.21.15.27:0/1987877564 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1d1415db90 0x7f1d140b4110 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f1d14073d90 tx=0x7f1cfc01c460 comp rx=0 tx=0).stop 2024-01-15T18:10:42.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.897+0000 7f1d1b600640 1 -- 172.21.15.27:0/1987877564 shutdown_connections 2024-01-15T18:10:42.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.897+0000 7f1d1b600640 1 --2- 172.21.15.27:0/1987877564 >> v2:172.21.15.27:3300/0 conn(0x7f1d140cca00 0x7f1d1406ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:42.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.897+0000 7f1d1b600640 1 --2- 172.21.15.27:0/1987877564 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1d1415db90 0x7f1d140b4110 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:42.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.897+0000 7f1d1b600640 1 -- 172.21.15.27:0/1987877564 >> 172.21.15.27:0/1987877564 conn(0x7f1d140b2360 msgr2=0x7f1d140b2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:42.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.897+0000 7f1d1b600640 1 -- 172.21.15.27:0/1987877564 shutdown_connections 2024-01-15T18:10:42.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.897+0000 7f1d1b600640 1 -- 172.21.15.27:0/1987877564 wait complete. 2024-01-15T18:10:42.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.901+0000 7f1d1b600640 1 Processor -- start 2024-01-15T18:10:42.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.901+0000 7f1d1b600640 1 -- start start 2024-01-15T18:10:42.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.901+0000 7f1d1b600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1d140cca00 0x7f1d14147230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:42.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.901+0000 7f1d1b600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1d1415a080 0x7f1d141668b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:42.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.901+0000 7f1d1b600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d14166df0 0x7f1d140ce6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:42.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.901+0000 7f1d1b600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d141672a0 con 0x7f1d14166df0 2024-01-15T18:10:42.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.901+0000 7f1d1b600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1414e460 con 0x7f1d140cca00 2024-01-15T18:10:42.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.901+0000 7f1d1b600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d140cf490 con 0x7f1d1415a080 2024-01-15T18:10:42.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.901+0000 7f1d1a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d14166df0 0x7f1d140ce6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:42.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.901+0000 7f1d1a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d14166df0 0x7f1d140ce6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35802/0 (socket says 172.21.15.27:35802) 2024-01-15T18:10:42.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.901+0000 7f1d1a000640 1 -- 172.21.15.27:0/1496508021 learned_addr learned my addr 172.21.15.27:0/1496508021 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:42.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.901+0000 7f1d18c00640 1 --2- 172.21.15.27:0/1496508021 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1d140cca00 0x7f1d14147230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:42.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.905+0000 7f1d1a000640 1 -- 172.21.15.27:0/1496508021 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1d1415a080 msgr2=0x7f1d141668b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:42.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.905+0000 7f1d1a000640 1 --2- 172.21.15.27:0/1496508021 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1d1415a080 0x7f1d141668b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:42.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.905+0000 7f1d1a000640 1 -- 172.21.15.27:0/1496508021 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1d140cca00 msgr2=0x7f1d14147230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:42.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.905+0000 7f1d1a000640 1 --2- 172.21.15.27:0/1496508021 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1d140cca00 0x7f1d14147230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:42.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.905+0000 7f1d1a000640 1 -- 172.21.15.27:0/1496508021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cfc009420 con 0x7f1d14166df0 2024-01-15T18:10:42.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.905+0000 7f1d1a000640 1 --2- 172.21.15.27:0/1496508021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d14166df0 0x7f1d140ce6d0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c00efc0 tx=0x7f1d0c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:42.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.905+0000 7f1d10c00640 1 -- 172.21.15.27:0/1496508021 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c00cd20 con 0x7f1d14166df0 2024-01-15T18:10:42.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.905+0000 7f1d1b600640 1 -- 172.21.15.27:0/1496508021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d140cf770 con 0x7f1d14166df0 2024-01-15T18:10:42.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.905+0000 7f1d1b600640 1 -- 172.21.15.27:0/1496508021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d1414e990 con 0x7f1d14166df0 2024-01-15T18:10:42.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.905+0000 7f1d10c00640 1 -- 172.21.15.27:0/1496508021 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0c00f040 con 0x7f1d14166df0 2024-01-15T18:10:42.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.905+0000 7f1d10c00640 1 -- 172.21.15.27:0/1496508021 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c003be0 con 0x7f1d14166df0 2024-01-15T18:10:42.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.909+0000 7f1d0a000640 1 -- 172.21.15.27:0/1496508021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce4005260 con 0x7f1d14166df0 2024-01-15T18:10:42.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.909+0000 7f1d10c00640 1 -- 172.21.15.27:0/1496508021 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7f1d0c003d40 con 0x7f1d14166df0 2024-01-15T18:10:42.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.913+0000 7f1d10c00640 1 --2- 172.21.15.27:0/1496508021 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1cec067160 0x7f1cec069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:42.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.913+0000 7f1d10c00640 1 -- 172.21.15.27:0/1496508021 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f1d0c083460 con 0x7f1d14166df0 2024-01-15T18:10:42.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.917+0000 7f1d10c00640 1 -- 172.21.15.27:0/1496508021 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1d0c051f20 con 0x7f1d14166df0 2024-01-15T18:10:42.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.933+0000 7f1d18c00640 1 --2- 172.21.15.27:0/1496508021 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1cec067160 0x7f1cec069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:42.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:42.953+0000 7f1d18c00640 1 --2- 172.21.15.27:0/1496508021 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1cec067160 0x7f1cec069620 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1d04000f70 tx=0x7f1d04009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:43.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:43.381+0000 7f1d0a000640 1 -- 172.21.15.27:0/1496508021 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1ce40050d0 con 0x7f1d14166df0 2024-01-15T18:10:43.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:43.381+0000 7f1d10c00640 1 -- 172.21.15.27:0/1496508021 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2428 (secure 0 0 0) 0x7f1d0c055540 con 0x7f1d14166df0 2024-01-15T18:10:43.389 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:10:43.389 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:10:43.389 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:10:43.389 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:10:43.389 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:10:43.389 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:10:43.389 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:10:43.389 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:43.389 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:10:43.389 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:10:43.389 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:10:43.389 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:10:43.390 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:10:43.390 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:43.390 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:10:43.390 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:10:43.390 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:10:43.390 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:10:43.390 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:43.390 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 370, 2024-01-15T18:10:43.390 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:10:43.390 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:10:43.390 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:10:43.390 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:10:43.390 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:43.390 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:10:43.390 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 58, 2024-01-15T18:10:43.390 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:10:43.390 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:10:43.391 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:10:43.391 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:10:43.391 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:10:43.391 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:10:43.391 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:43.391 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:10:43.391 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:10:43.391 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:43.391 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:10:43.391 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:10:43.391 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:43.391 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:43.391 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:10:43.391 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:10:43.391 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 4503, 2024-01-15T18:10:43.391 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 18737942868, 2024-01-15T18:10:43.392 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 37747384320, 2024-01-15T18:10:43.392 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 735346728960, 2024-01-15T18:10:43.392 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:10:43.392 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 409524129, 2024-01-15T18:10:43.392 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 417593257, 2024-01-15T18:10:43.392 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 98, 2024-01-15T18:10:43.392 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 110 2024-01-15T18:10:43.392 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:43.392 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:10:43.392 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 38, 2024-01-15T18:10:43.392 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4, 2024-01-15T18:10:43.392 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:10:43.392 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:10:43.393 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:10:43.393 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:10:43.393 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:43.393 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 4, 2024-01-15T18:10:43.393 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:10:43.393 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:10:43.393 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:10:43.393 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 6480 2024-01-15T18:10:43.393 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:43.393 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:43.393 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:10:43.393 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:43.393 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:10:43.393 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:10:43.393 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:10:43.394 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:10:43.394 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:10:43.394 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:10:43.394 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:10:43.394 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:43.394 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:43.394 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:43.394 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:10:43.394 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 13, 2024-01-15T18:10:43.394 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:10:16.226140+0000", 2024-01-15T18:10:43.394 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:43.394 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:43.394 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:10:43.394 INFO:teuthology.orchestra.run.smithi027.stdout: "268d93d2-2eea-412c-b804-608e15344ce7": { 2024-01-15T18:10:43.394 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 71.429% (35s)\n [====================........] (remaining: 14s)", 2024-01-15T18:10:43.394 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.71428573131561279, 2024-01-15T18:10:43.395 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:43.395 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:43.395 INFO:teuthology.orchestra.run.smithi027.stdout: "9a27fb80-350b-410b-a65e-91a770cccfdf": { 2024-01-15T18:10:43.395 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "3 ongoing cloning jobs; Avg progress = 33.333% (35s)\n [=========...................] (remaining: 71s)", 2024-01-15T18:10:43.395 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.3333333432674408, 2024-01-15T18:10:43.395 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:43.395 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:43.395 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:43.395 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:10:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:43.385+0000 7f1d0a000640 1 -- 172.21.15.27:0/1496508021 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1cec067160 msgr2=0x7f1cec069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:43.385+0000 7f1d0a000640 1 --2- 172.21.15.27:0/1496508021 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1cec067160 0x7f1cec069620 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1d04000f70 tx=0x7f1d04009040 comp rx=0 tx=0).stop 2024-01-15T18:10:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:43.385+0000 7f1d0a000640 1 -- 172.21.15.27:0/1496508021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d14166df0 msgr2=0x7f1d140ce6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:43.385+0000 7f1d0a000640 1 --2- 172.21.15.27:0/1496508021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d14166df0 0x7f1d140ce6d0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c00efc0 tx=0x7f1d0c00c490 comp rx=0 tx=0).stop 2024-01-15T18:10:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:43.389+0000 7f1d0a000640 1 -- 172.21.15.27:0/1496508021 shutdown_connections 2024-01-15T18:10:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:43.389+0000 7f1d0a000640 1 --2- 172.21.15.27:0/1496508021 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1cec067160 0x7f1cec069620 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:43.389+0000 7f1d0a000640 1 --2- 172.21.15.27:0/1496508021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d14166df0 0x7f1d140ce6d0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:43.389+0000 7f1d0a000640 1 --2- 172.21.15.27:0/1496508021 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1d1415a080 0x7f1d141668b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:43.389+0000 7f1d0a000640 1 --2- 172.21.15.27:0/1496508021 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1d140cca00 0x7f1d14147230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:43.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:43.389+0000 7f1d0a000640 1 -- 172.21.15.27:0/1496508021 >> 172.21.15.27:0/1496508021 conn(0x7f1d140b2360 msgr2=0x7f1d140b1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:43.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:43.389+0000 7f1d0a000640 1 -- 172.21.15.27:0/1496508021 shutdown_connections 2024-01-15T18:10:43.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:43.389+0000 7f1d0a000640 1 -- 172.21.15.27:0/1496508021 wait complete. 2024-01-15T18:10:45.442 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:10:45.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.652+0000 7fc661200640 1 Processor -- start 2024-01-15T18:10:45.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.652+0000 7fc661200640 1 -- start start 2024-01-15T18:10:45.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.652+0000 7fc661200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fc65c071a70 0x7fc65c0ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:45.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.652+0000 7fc661200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc65c0ad520 con 0x7fc65c161930 2024-01-15T18:10:45.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.652+0000 7fc661200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc65c0acc60 con 0x7fc65c06d5d0 2024-01-15T18:10:45.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.652+0000 7fc661200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc65c071f10 con 0x7fc65c071a70 2024-01-15T18:10:45.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.652+0000 7fc659e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc65c161930 0x7fc65c0ad9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39130/0 (socket says 172.21.15.27:39130) 2024-01-15T18:10:45.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.656+0000 7fc659e00640 1 -- 172.21.15.27:0/2749138546 learned_addr learned my addr 172.21.15.27:0/2749138546 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:10:45.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.656+0000 7fc659400640 1 --2- 172.21.15.27:0/2749138546 >> v2:172.21.15.93:3300/0 conn(0x7fc65c071a70 0x7fc65c0ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:45.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.656+0000 7fc658a00640 1 -- 172.21.15.27:0/2749138546 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165326068 0 0) 0x7fc65c0ad520 con 0x7fc65c161930 2024-01-15T18:10:45.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.656+0000 7fc658a00640 1 -- 172.21.15.27:0/2749138546 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc63c003530 con 0x7fc65c161930 2024-01-15T18:10:45.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.656+0000 7fc658a00640 1 -- 172.21.15.27:0/2749138546 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2253128396 0 0) 0x7fc63c003530 con 0x7fc65c161930 2024-01-15T18:10:45.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.656+0000 7fc658a00640 1 -- 172.21.15.27:0/2749138546 >> v2:172.21.15.93:3300/0 conn(0x7fc65c071a70 msgr2=0x7fc65c0ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:45.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.656+0000 7fc658a00640 1 --2- 172.21.15.27:0/2749138546 >> v2:172.21.15.93:3300/0 conn(0x7fc65c071a70 0x7fc65c0ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:45.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.656+0000 7fc658a00640 1 -- 172.21.15.27:0/2749138546 >> v1:172.21.15.93:6789/0 conn(0x7fc65c06d5d0 legacy=0x7fc65c06d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:45.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.656+0000 7fc658a00640 1 -- 172.21.15.27:0/2749138546 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65c0aced0 con 0x7fc65c161930 2024-01-15T18:10:45.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.656+0000 7fc658a00640 1 -- 172.21.15.27:0/2749138546 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fc644001680 con 0x7fc65c161930 2024-01-15T18:10:45.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.656+0000 7fc658a00640 1 -- 172.21.15.27:0/2749138546 >> v1:172.21.15.27:6789/0 conn(0x7fc65c161930 legacy=0x7fc65c0ad9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:45.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.656+0000 7fc658a00640 1 --2- 172.21.15.27:0/2749138546 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc63c003980 0x7fc63c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:45.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.656+0000 7fc658a00640 1 --2- 172.21.15.27:0/2749138546 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc63c0047d0 0x7fc63c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:45.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.656+0000 7fc658a00640 1 --2- 172.21.15.27:0/2749138546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc63c005190 0x7fc63c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:45.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.656+0000 7fc658a00640 1 -- 172.21.15.27:0/2749138546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c071f10 con 0x7fc63c005190 2024-01-15T18:10:45.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.656+0000 7fc658a00640 1 -- 172.21.15.27:0/2749138546 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63c00c030 con 0x7fc63c0047d0 2024-01-15T18:10:45.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.656+0000 7fc658a00640 1 -- 172.21.15.27:0/2749138546 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc63c00c1f0 con 0x7fc63c003980 2024-01-15T18:10:45.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.656+0000 7fc65a800640 1 --2- 172.21.15.27:0/2749138546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc63c005190 0x7fc63c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:45.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.656+0000 7fc65a800640 1 -- 172.21.15.27:0/2749138546 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc63c003980 msgr2=0x7fc63c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:45.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.656+0000 7fc65a800640 1 --2- 172.21.15.27:0/2749138546 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc63c003980 0x7fc63c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:45.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.656+0000 7fc65a800640 1 -- 172.21.15.27:0/2749138546 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc63c0047d0 msgr2=0x7fc63c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:45.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.656+0000 7fc65a800640 1 --2- 172.21.15.27:0/2749138546 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc63c0047d0 0x7fc63c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:45.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.656+0000 7fc65a800640 1 -- 172.21.15.27:0/2749138546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63c003530 con 0x7fc63c005190 2024-01-15T18:10:45.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.660+0000 7fc65a800640 1 -- 172.21.15.27:0/2749138546 reap_dead start 2024-01-15T18:10:45.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.660+0000 7fc65a800640 1 --2- 172.21.15.27:0/2749138546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc63c005190 0x7fc63c00b990 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fc64c001780 tx=0x7fc64c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5015a5d83a4be62 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:45.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.660+0000 7fc658a00640 1 -- 172.21.15.27:0/2749138546 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c0052d0 con 0x7fc63c005190 2024-01-15T18:10:45.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.660+0000 7fc658a00640 1 -- 172.21.15.27:0/2749138546 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64c0a7050 con 0x7fc63c005190 2024-01-15T18:10:45.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.660+0000 7fc661200640 1 -- 172.21.15.27:0/2749138546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc63c005190 msgr2=0x7fc63c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:45.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.660+0000 7fc658a00640 1 -- 172.21.15.27:0/2749138546 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c003840 con 0x7fc63c005190 2024-01-15T18:10:45.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.660+0000 7fc661200640 1 --2- 172.21.15.27:0/2749138546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc63c005190 0x7fc63c00b990 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fc64c001780 tx=0x7fc64c01c400 comp rx=0 tx=0).stop 2024-01-15T18:10:45.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.660+0000 7fc661200640 1 -- 172.21.15.27:0/2749138546 shutdown_connections 2024-01-15T18:10:45.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.660+0000 7fc661200640 1 --2- 172.21.15.27:0/2749138546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc63c005190 0x7fc63c00b990 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:45.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.660+0000 7fc661200640 1 -- 172.21.15.27:0/2749138546 >> 172.21.15.27:0/2749138546 conn(0x7fc65c0b0d90 msgr2=0x7fc65c0b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:45.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.660+0000 7fc661200640 1 -- 172.21.15.27:0/2749138546 shutdown_connections 2024-01-15T18:10:45.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.660+0000 7fc661200640 1 -- 172.21.15.27:0/2749138546 wait complete. 2024-01-15T18:10:45.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.664+0000 7fc661200640 1 Processor -- start 2024-01-15T18:10:45.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.664+0000 7fc661200640 1 -- start start 2024-01-15T18:10:45.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.668+0000 7fc661200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc63c005190 0x7fc65c1f6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:45.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.668+0000 7fc661200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc65c163790 0x7fc65c163c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:45.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.668+0000 7fc661200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc65c0cd2d0 0x7fc65c0cd750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:45.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.668+0000 7fc661200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c164150 con 0x7fc63c005190 2024-01-15T18:10:45.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.668+0000 7fc661200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c0ce5c0 con 0x7fc65c0cd2d0 2024-01-15T18:10:45.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.668+0000 7fc661200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc65c15d840 con 0x7fc65c163790 2024-01-15T18:10:45.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.668+0000 7fc659e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc63c005190 0x7fc65c1f6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:45.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.668+0000 7fc659e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc63c005190 0x7fc65c1f6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49728/0 (socket says 172.21.15.27:49728) 2024-01-15T18:10:45.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.668+0000 7fc659e00640 1 -- 172.21.15.27:0/911003656 learned_addr learned my addr 172.21.15.27:0/911003656 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:45.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.668+0000 7fc659400640 1 --2- 172.21.15.27:0/911003656 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc65c163790 0x7fc65c163c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:45.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.668+0000 7fc659e00640 1 -- 172.21.15.27:0/911003656 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc65c163790 msgr2=0x7fc65c163c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:45.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.668+0000 7fc659e00640 1 --2- 172.21.15.27:0/911003656 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc65c163790 0x7fc65c163c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:45.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.668+0000 7fc659e00640 1 -- 172.21.15.27:0/911003656 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc65c0cd2d0 msgr2=0x7fc65c0cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:45.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.668+0000 7fc659e00640 1 --2- 172.21.15.27:0/911003656 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc65c0cd2d0 0x7fc65c0cd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:45.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.672+0000 7fc659e00640 1 -- 172.21.15.27:0/911003656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65c0ce000 con 0x7fc63c005190 2024-01-15T18:10:45.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.672+0000 7fc659e00640 1 --2- 172.21.15.27:0/911003656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc63c005190 0x7fc65c1f6ae0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fc64400c730 tx=0x7fc64400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:45.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.672+0000 7fc652000640 1 -- 172.21.15.27:0/911003656 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6440074e0 con 0x7fc63c005190 2024-01-15T18:10:45.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.672+0000 7fc652000640 1 -- 172.21.15.27:0/911003656 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc644007660 con 0x7fc63c005190 2024-01-15T18:10:45.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.672+0000 7fc652000640 1 -- 172.21.15.27:0/911003656 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc644004a30 con 0x7fc63c005190 2024-01-15T18:10:45.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.672+0000 7fc661200640 1 -- 172.21.15.27:0/911003656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc65c0ce2e0 con 0x7fc63c005190 2024-01-15T18:10:45.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.672+0000 7fc661200640 1 -- 172.21.15.27:0/911003656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc65c0cdc90 con 0x7fc63c005190 2024-01-15T18:10:45.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.676+0000 7fc661200640 1 -- 172.21.15.27:0/911003656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc65c074820 con 0x7fc63c005190 2024-01-15T18:10:45.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.680+0000 7fc652000640 1 -- 172.21.15.27:0/911003656 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7fc644004b90 con 0x7fc63c005190 2024-01-15T18:10:45.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.680+0000 7fc652000640 1 --2- 172.21.15.27:0/911003656 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc634067260 0x7fc634069720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:45.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.680+0000 7fc652000640 1 -- 172.21.15.27:0/911003656 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc644082700 con 0x7fc63c005190 2024-01-15T18:10:45.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.680+0000 7fc659400640 1 --2- 172.21.15.27:0/911003656 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc634067260 0x7fc634069720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:45.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.684+0000 7fc659400640 1 --2- 172.21.15.27:0/911003656 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc634067260 0x7fc634069720 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc654004b40 tx=0x7fc65400b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:45.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:45.688+0000 7fc652000640 1 -- 172.21.15.27:0/911003656 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc6440511c0 con 0x7fc63c005190 2024-01-15T18:10:46.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:46.000+0000 7fc661200640 1 -- 172.21.15.27:0/911003656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc65c06e340 con 0x7fc63c005190 2024-01-15T18:10:46.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:46.000+0000 7fc652000640 1 -- 172.21.15.27:0/911003656 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2427 (secure 0 0 0) 0x7fc644018020 con 0x7fc63c005190 2024-01-15T18:10:46.008 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:10:46.008 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:10:46.008 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:10:46.008 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:10:46.008 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:10:46.008 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:10:46.009 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:10:46.009 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:46.009 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:10:46.009 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:10:46.009 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:10:46.009 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:10:46.009 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:10:46.009 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:46.009 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:10:46.009 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:10:46.009 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:10:46.009 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:10:46.009 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:46.009 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 373, 2024-01-15T18:10:46.010 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:10:46.010 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:10:46.010 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:10:46.010 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:10:46.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:46.010 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:10:46.010 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 58, 2024-01-15T18:10:46.010 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:10:46.010 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:10:46.010 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:10:46.010 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:10:46.010 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:10:46.010 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:10:46.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:46.011 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:10:46.011 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:10:46.011 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:46.011 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:10:46.011 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:10:46.011 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:46.011 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:46.011 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:10:46.011 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:10:46.011 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 4603, 2024-01-15T18:10:46.011 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 19157373268, 2024-01-15T18:10:46.011 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 38481387520, 2024-01-15T18:10:46.011 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 734612725760, 2024-01-15T18:10:46.011 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:10:46.011 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 329104400, 2024-01-15T18:10:46.011 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 324583529, 2024-01-15T18:10:46.012 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 79, 2024-01-15T18:10:46.012 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 85 2024-01-15T18:10:46.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:46.012 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:10:46.012 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 38, 2024-01-15T18:10:46.012 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4, 2024-01-15T18:10:46.012 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:10:46.012 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:10:46.012 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:10:46.012 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:10:46.012 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:46.012 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 4, 2024-01-15T18:10:46.012 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:10:46.012 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:10:46.013 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:10:46.013 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 6480 2024-01-15T18:10:46.013 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:46.013 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:46.013 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:10:46.013 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:46.013 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:10:46.013 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:10:46.013 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:10:46.013 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:10:46.013 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:10:46.013 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:10:46.013 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:10:46.013 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:46.013 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:46.014 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:46.014 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:10:46.014 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 13, 2024-01-15T18:10:46.014 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:10:16.226140+0000", 2024-01-15T18:10:46.014 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:46.014 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:46.014 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:10:46.014 INFO:teuthology.orchestra.run.smithi027.stdout: "268d93d2-2eea-412c-b804-608e15344ce7": { 2024-01-15T18:10:46.014 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 71.429% (35s)\n [====================........] (remaining: 14s)", 2024-01-15T18:10:46.014 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.71428573131561279, 2024-01-15T18:10:46.014 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:46.014 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:46.014 INFO:teuthology.orchestra.run.smithi027.stdout: "9a27fb80-350b-410b-a65e-91a770cccfdf": { 2024-01-15T18:10:46.014 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "3 ongoing cloning jobs; Avg progress = 33.333% (35s)\n [=========...................] (remaining: 71s)", 2024-01-15T18:10:46.014 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.3333333432674408, 2024-01-15T18:10:46.014 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:46.015 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:46.015 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:46.015 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:10:46.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:46.004+0000 7fc661200640 1 -- 172.21.15.27:0/911003656 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc634067260 msgr2=0x7fc634069720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:46.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:46.004+0000 7fc661200640 1 --2- 172.21.15.27:0/911003656 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc634067260 0x7fc634069720 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc654004b40 tx=0x7fc65400b040 comp rx=0 tx=0).stop 2024-01-15T18:10:46.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:46.004+0000 7fc661200640 1 -- 172.21.15.27:0/911003656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc63c005190 msgr2=0x7fc65c1f6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:46.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:46.004+0000 7fc661200640 1 --2- 172.21.15.27:0/911003656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc63c005190 0x7fc65c1f6ae0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fc64400c730 tx=0x7fc64400cbf0 comp rx=0 tx=0).stop 2024-01-15T18:10:46.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:46.004+0000 7fc661200640 1 -- 172.21.15.27:0/911003656 shutdown_connections 2024-01-15T18:10:46.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:46.004+0000 7fc661200640 1 --2- 172.21.15.27:0/911003656 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc634067260 0x7fc634069720 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:46.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:46.004+0000 7fc661200640 1 --2- 172.21.15.27:0/911003656 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc65c0cd2d0 0x7fc65c0cd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:46.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:46.004+0000 7fc661200640 1 --2- 172.21.15.27:0/911003656 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc65c163790 0x7fc65c163c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:46.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:46.004+0000 7fc661200640 1 --2- 172.21.15.27:0/911003656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc63c005190 0x7fc65c1f6ae0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:46.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:46.004+0000 7fc661200640 1 -- 172.21.15.27:0/911003656 >> 172.21.15.27:0/911003656 conn(0x7fc65c0b0d90 msgr2=0x7fc65c0b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:46.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:46.004+0000 7fc661200640 1 -- 172.21.15.27:0/911003656 shutdown_connections 2024-01-15T18:10:46.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:46.004+0000 7fc661200640 1 -- 172.21.15.27:0/911003656 wait complete. 2024-01-15T18:10:48.032 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:10:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.304+0000 7fe359e00640 1 Processor -- start 2024-01-15T18:10:48.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.304+0000 7fe359e00640 1 -- start start 2024-01-15T18:10:48.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.308+0000 7fe359e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe3540cf330 0x7fe35406d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:48.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.308+0000 7fe359e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3540ce080 con 0x7fe354143b90 2024-01-15T18:10:48.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.308+0000 7fe359e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3540acdc0 con 0x7fe35406d5f0 2024-01-15T18:10:48.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.308+0000 7fe359e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35406da80 con 0x7fe3540cf330 2024-01-15T18:10:48.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.308+0000 7fe352a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe354143b90 0x7fe3540cec20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39144/0 (socket says 172.21.15.27:39144) 2024-01-15T18:10:48.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.308+0000 7fe352a00640 1 -- 172.21.15.27:0/1352225429 learned_addr learned my addr 172.21.15.27:0/1352225429 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:10:48.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.308+0000 7fe352000640 1 --2- 172.21.15.27:0/1352225429 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe3540cf330 0x7fe35406d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:48.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.312+0000 7fe351600640 1 -- 172.21.15.27:0/1352225429 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3573237484 0 0) 0x7fe3540ce080 con 0x7fe354143b90 2024-01-15T18:10:48.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.312+0000 7fe351600640 1 -- 172.21.15.27:0/1352225429 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe334003530 con 0x7fe354143b90 2024-01-15T18:10:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.312+0000 7fe351600640 1 -- 172.21.15.27:0/1352225429 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1490259001 0 0) 0x7fe334003530 con 0x7fe354143b90 2024-01-15T18:10:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.312+0000 7fe351600640 1 -- 172.21.15.27:0/1352225429 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe3540cf330 msgr2=0x7fe35406d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.312+0000 7fe351600640 1 --2- 172.21.15.27:0/1352225429 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe3540cf330 0x7fe35406d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.312+0000 7fe351600640 1 -- 172.21.15.27:0/1352225429 >> v1:172.21.15.93:6789/0 conn(0x7fe35406d5f0 legacy=0x7fe3540ad810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.312+0000 7fe351600640 1 -- 172.21.15.27:0/1352225429 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3540ad380 con 0x7fe354143b90 2024-01-15T18:10:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.312+0000 7fe351600640 1 -- 172.21.15.27:0/1352225429 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fe33c001680 con 0x7fe354143b90 2024-01-15T18:10:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.312+0000 7fe351600640 1 -- 172.21.15.27:0/1352225429 >> v1:172.21.15.27:6789/0 conn(0x7fe354143b90 legacy=0x7fe3540cec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.312+0000 7fe351600640 1 --2- 172.21.15.27:0/1352225429 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe334003a20 0x7fe334003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.312+0000 7fe351600640 1 --2- 172.21.15.27:0/1352225429 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe3540cf330 0x7fe3340047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.312+0000 7fe351600640 1 --2- 172.21.15.27:0/1352225429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe334004d10 0x7fe334005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.312+0000 7fe351600640 1 -- 172.21.15.27:0/1352225429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35406da80 con 0x7fe334004d10 2024-01-15T18:10:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.312+0000 7fe351600640 1 -- 172.21.15.27:0/1352225429 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33400bad0 con 0x7fe334003a20 2024-01-15T18:10:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.312+0000 7fe351600640 1 -- 172.21.15.27:0/1352225429 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe33400bc90 con 0x7fe3540cf330 2024-01-15T18:10:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.312+0000 7fe353400640 1 --2- 172.21.15.27:0/1352225429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe334004d10 0x7fe334005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:48.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.312+0000 7fe352000640 1 --2- 172.21.15.27:0/1352225429 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe3540cf330 0x7fe3340047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:48.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.312+0000 7fe353400640 1 -- 172.21.15.27:0/1352225429 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe3540cf330 msgr2=0x7fe3340047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:48.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.312+0000 7fe353400640 1 --2- 172.21.15.27:0/1352225429 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe3540cf330 0x7fe3340047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.312+0000 7fe353400640 1 -- 172.21.15.27:0/1352225429 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe334003a20 msgr2=0x7fe334003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:48.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.312+0000 7fe353400640 1 --2- 172.21.15.27:0/1352225429 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe334003a20 0x7fe334003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.312+0000 7fe353400640 1 -- 172.21.15.27:0/1352225429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3540ad380 con 0x7fe334004d10 2024-01-15T18:10:48.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.320+0000 7fe353400640 1 --2- 172.21.15.27:0/1352225429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe334004d10 0x7fe334005190 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fe34c001e40 tx=0x7fe34c020b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=545b244085e6d945 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.320+0000 7fe351600640 1 -- 172.21.15.27:0/1352225429 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c003b80 con 0x7fe334004d10 2024-01-15T18:10:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.320+0000 7fe351600640 1 -- 172.21.15.27:0/1352225429 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe34c003ce0 con 0x7fe334004d10 2024-01-15T18:10:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.320+0000 7fe351600640 1 -- 172.21.15.27:0/1352225429 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c0a9a90 con 0x7fe334004d10 2024-01-15T18:10:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.320+0000 7fe359e00640 1 -- 172.21.15.27:0/1352225429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe334004d10 msgr2=0x7fe334005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.320+0000 7fe359e00640 1 --2- 172.21.15.27:0/1352225429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe334004d10 0x7fe334005190 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fe34c001e40 tx=0x7fe34c020b50 comp rx=0 tx=0).stop 2024-01-15T18:10:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.320+0000 7fe353400640 1 -- 172.21.15.27:0/1352225429 reap_dead start 2024-01-15T18:10:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.320+0000 7fe359e00640 1 -- 172.21.15.27:0/1352225429 shutdown_connections 2024-01-15T18:10:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.320+0000 7fe359e00640 1 -- 172.21.15.27:0/1352225429 >> 172.21.15.27:0/1352225429 conn(0x7fe3540b1ae0 msgr2=0x7fe3540b1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:48.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.324+0000 7fe359e00640 1 -- 172.21.15.27:0/1352225429 shutdown_connections 2024-01-15T18:10:48.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.324+0000 7fe359e00640 1 -- 172.21.15.27:0/1352225429 wait complete. 2024-01-15T18:10:48.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.324+0000 7fe359e00640 1 Processor -- start 2024-01-15T18:10:48.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.324+0000 7fe359e00640 1 -- start start 2024-01-15T18:10:48.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.328+0000 7fe359e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe334004d10 0x7fe3541e0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:48.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.328+0000 7fe359e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe35409c570 0x7fe35409aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:48.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.328+0000 7fe359e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe35409afc0 0x7fe3540cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:48.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.328+0000 7fe359e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35409b4d0 con 0x7fe334004d10 2024-01-15T18:10:48.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.328+0000 7fe359e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35409b7b0 con 0x7fe35409c570 2024-01-15T18:10:48.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.328+0000 7fe359e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35409ba90 con 0x7fe35409afc0 2024-01-15T18:10:48.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.328+0000 7fe352a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe334004d10 0x7fe3541e0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:48.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.328+0000 7fe352a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe334004d10 0x7fe3541e0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49744/0 (socket says 172.21.15.27:49744) 2024-01-15T18:10:48.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.328+0000 7fe352a00640 1 -- 172.21.15.27:0/3227623233 learned_addr learned my addr 172.21.15.27:0/3227623233 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:48.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.328+0000 7fe352000640 1 --2- 172.21.15.27:0/3227623233 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe35409c570 0x7fe35409aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:48.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.328+0000 7fe352a00640 1 -- 172.21.15.27:0/3227623233 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe35409afc0 msgr2=0x7fe3540cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:48.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.328+0000 7fe352a00640 1 --2- 172.21.15.27:0/3227623233 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe35409afc0 0x7fe3540cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.328+0000 7fe352a00640 1 -- 172.21.15.27:0/3227623233 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe35409c570 msgr2=0x7fe35409aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:48.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.328+0000 7fe352a00640 1 --2- 172.21.15.27:0/3227623233 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe35409c570 0x7fe35409aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.328+0000 7fe352a00640 1 -- 172.21.15.27:0/3227623233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35409bd70 con 0x7fe334004d10 2024-01-15T18:10:48.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.328+0000 7fe352a00640 1 --2- 172.21.15.27:0/3227623233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe334004d10 0x7fe3541e0050 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fe33c001ec0 tx=0x7fe33c00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:48.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.328+0000 7fe34b400640 1 -- 172.21.15.27:0/3227623233 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33c0074e0 con 0x7fe334004d10 2024-01-15T18:10:48.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.328+0000 7fe359e00640 1 -- 172.21.15.27:0/3227623233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe35409c050 con 0x7fe334004d10 2024-01-15T18:10:48.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.328+0000 7fe359e00640 1 -- 172.21.15.27:0/3227623233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3540d0370 con 0x7fe334004d10 2024-01-15T18:10:48.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.328+0000 7fe34b400640 1 -- 172.21.15.27:0/3227623233 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe33c007640 con 0x7fe334004d10 2024-01-15T18:10:48.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.328+0000 7fe34b400640 1 -- 172.21.15.27:0/3227623233 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33c004a10 con 0x7fe334004d10 2024-01-15T18:10:48.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.332+0000 7fe359e00640 1 -- 172.21.15.27:0/3227623233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe324005260 con 0x7fe334004d10 2024-01-15T18:10:48.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.332+0000 7fe34b400640 1 -- 172.21.15.27:0/3227623233 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7fe33c004b70 con 0x7fe334004d10 2024-01-15T18:10:48.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.336+0000 7fe34b400640 1 --2- 172.21.15.27:0/3227623233 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe32c067160 0x7fe32c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:48.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.336+0000 7fe34b400640 1 -- 172.21.15.27:0/3227623233 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe33c082740 con 0x7fe334004d10 2024-01-15T18:10:48.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.344+0000 7fe34b400640 1 -- 172.21.15.27:0/3227623233 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe33c0512b0 con 0x7fe334004d10 2024-01-15T18:10:48.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.348+0000 7fe352000640 1 --2- 172.21.15.27:0/3227623233 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe32c067160 0x7fe32c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:48.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.352+0000 7fe352000640 1 --2- 172.21.15.27:0/3227623233 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe32c067160 0x7fe32c069620 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fe35409dd10 tx=0x7fe344008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:48.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.736+0000 7fe359e00640 1 -- 172.21.15.27:0/3227623233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe3240057e0 con 0x7fe334004d10 2024-01-15T18:10:48.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.736+0000 7fe34b400640 1 -- 172.21.15.27:0/3227623233 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2154 (secure 0 0 0) 0x7fe33c018090 con 0x7fe334004d10 2024-01-15T18:10:48.743 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:10:48.743 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:10:48.743 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:10:48.743 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:10:48.743 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:10:48.743 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:10:48.743 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:10:48.743 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:48.743 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:10:48.744 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:10:48.744 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:10:48.744 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:10:48.744 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:10:48.744 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:48.744 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:10:48.744 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:10:48.744 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:10:48.744 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:10:48.744 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:48.744 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 375, 2024-01-15T18:10:48.744 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:10:48.744 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:10:48.745 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:10:48.745 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:10:48.745 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:48.745 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:10:48.745 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 58, 2024-01-15T18:10:48.745 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:10:48.745 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:10:48.745 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:10:48.745 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:10:48.745 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:10:48.745 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:10:48.745 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:48.745 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:10:48.745 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:10:48.745 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:48.745 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:10:48.746 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:10:48.746 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:48.746 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:48.746 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:10:48.746 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:10:48.746 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 5273, 2024-01-15T18:10:48.746 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 21971867926, 2024-01-15T18:10:48.746 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 44123004928, 2024-01-15T18:10:48.746 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 728971108352, 2024-01-15T18:10:48.746 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:10:48.746 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 553708694, 2024-01-15T18:10:48.746 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 548150603, 2024-01-15T18:10:48.746 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 133, 2024-01-15T18:10:48.746 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 142 2024-01-15T18:10:48.746 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:48.747 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:10:48.747 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 38, 2024-01-15T18:10:48.747 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4, 2024-01-15T18:10:48.747 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:10:48.747 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:10:48.747 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:10:48.747 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:10:48.747 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:48.747 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 4, 2024-01-15T18:10:48.747 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:10:48.747 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:10:48.747 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:10:48.747 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 6480 2024-01-15T18:10:48.747 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:48.748 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:48.748 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:10:48.748 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:48.748 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:10:48.748 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:10:48.748 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:10:48.748 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:10:48.748 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:10:48.748 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:10:48.748 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:10:48.748 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:48.748 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:48.748 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:48.748 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:10:48.749 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 13, 2024-01-15T18:10:48.749 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:10:16.226140+0000", 2024-01-15T18:10:48.749 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:48.749 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:48.749 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:10:48.749 INFO:teuthology.orchestra.run.smithi027.stdout: "268d93d2-2eea-412c-b804-608e15344ce7": { 2024-01-15T18:10:48.749 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 85.714% (40s)\n [========================....] (remaining: 6s)", 2024-01-15T18:10:48.749 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.8571428656578064, 2024-01-15T18:10:48.749 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:48.749 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:48.749 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:48.749 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:10:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.740+0000 7fe359e00640 1 -- 172.21.15.27:0/3227623233 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe32c067160 msgr2=0x7fe32c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.740+0000 7fe359e00640 1 --2- 172.21.15.27:0/3227623233 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe32c067160 0x7fe32c069620 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fe35409dd10 tx=0x7fe344008040 comp rx=0 tx=0).stop 2024-01-15T18:10:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.740+0000 7fe359e00640 1 -- 172.21.15.27:0/3227623233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe334004d10 msgr2=0x7fe3541e0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.740+0000 7fe359e00640 1 --2- 172.21.15.27:0/3227623233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe334004d10 0x7fe3541e0050 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fe33c001ec0 tx=0x7fe33c00ccd0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.740+0000 7fe359e00640 1 -- 172.21.15.27:0/3227623233 shutdown_connections 2024-01-15T18:10:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.740+0000 7fe359e00640 1 --2- 172.21.15.27:0/3227623233 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe32c067160 0x7fe32c069620 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.740+0000 7fe359e00640 1 --2- 172.21.15.27:0/3227623233 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe35409afc0 0x7fe3540cfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.740+0000 7fe359e00640 1 --2- 172.21.15.27:0/3227623233 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe35409c570 0x7fe35409aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.740+0000 7fe359e00640 1 --2- 172.21.15.27:0/3227623233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe334004d10 0x7fe3541e0050 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.740+0000 7fe359e00640 1 -- 172.21.15.27:0/3227623233 >> 172.21.15.27:0/3227623233 conn(0x7fe3540b1ae0 msgr2=0x7fe3540ae600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.740+0000 7fe359e00640 1 -- 172.21.15.27:0/3227623233 shutdown_connections 2024-01-15T18:10:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:48.740+0000 7fe359e00640 1 -- 172.21.15.27:0/3227623233 wait complete. 2024-01-15T18:10:48.793 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:10:49.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.068+0000 7f316e800640 1 Processor -- start 2024-01-15T18:10:49.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.068+0000 7f316e800640 1 -- start start 2024-01-15T18:10:49.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.072+0000 7f316e800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f316815e7d0 0x7f31680cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:49.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.072+0000 7f316e800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f31680b20d0 0x7f31680cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:49.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.072+0000 7f316e800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31680b38b0 con 0x7f31680cedc0 2024-01-15T18:10:49.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.072+0000 7f316e800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31680cd550 con 0x7f31680b20d0 2024-01-15T18:10:49.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.072+0000 7f316e800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31680cd270 con 0x7f316815e7d0 2024-01-15T18:10:49.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.072+0000 7f3167e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f31680b20d0 0x7f31680cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:49.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.072+0000 7f3167e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f31680b20d0 0x7f31680cf7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49750/0 (socket says 172.21.15.27:49750) 2024-01-15T18:10:49.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.072+0000 7f3167e00640 1 -- 172.21.15.27:0/2895049987 learned_addr learned my addr 172.21.15.27:0/2895049987 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:49.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.072+0000 7f3167e00640 1 -- 172.21.15.27:0/2895049987 >> v2:172.21.15.93:3300/0 conn(0x7f316815e7d0 msgr2=0x7f31680cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:49.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.072+0000 7f3167e00640 1 --2- 172.21.15.27:0/2895049987 >> v2:172.21.15.93:3300/0 conn(0x7f316815e7d0 0x7f31680cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:49.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.072+0000 7f3167e00640 1 -- 172.21.15.27:0/2895049987 >> v1:172.21.15.93:6789/0 conn(0x7f31680cedc0 legacy=0x7f31680cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:49.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.072+0000 7f3167e00640 1 -- 172.21.15.27:0/2895049987 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31680b35d0 con 0x7f31680b20d0 2024-01-15T18:10:49.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.076+0000 7f3167e00640 1 --2- 172.21.15.27:0/2895049987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31680b20d0 0x7f31680cf7e0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f3168071b00 tx=0x7f316001c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faaa5c00a256ee8c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:49.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.076+0000 7f3166000640 1 -- 172.21.15.27:0/2895049987 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31600a5740 con 0x7f31680b20d0 2024-01-15T18:10:49.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.076+0000 7f3166000640 1 -- 172.21.15.27:0/2895049987 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31600a58a0 con 0x7f31680b20d0 2024-01-15T18:10:49.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.076+0000 7f3166000640 1 -- 172.21.15.27:0/2895049987 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31600a5c30 con 0x7f31680b20d0 2024-01-15T18:10:49.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.080+0000 7f316e800640 1 -- 172.21.15.27:0/2895049987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31680b20d0 msgr2=0x7f31680cf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:49.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.080+0000 7f316e800640 1 --2- 172.21.15.27:0/2895049987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31680b20d0 0x7f31680cf7e0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f3168071b00 tx=0x7f316001c990 comp rx=0 tx=0).stop 2024-01-15T18:10:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.080+0000 7f316e800640 1 -- 172.21.15.27:0/2895049987 shutdown_connections 2024-01-15T18:10:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.080+0000 7f316e800640 1 --2- 172.21.15.27:0/2895049987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31680b20d0 0x7f31680cf7e0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.080+0000 7f316e800640 1 --2- 172.21.15.27:0/2895049987 >> v2:172.21.15.93:3300/0 conn(0x7f316815e7d0 0x7f31680cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.080+0000 7f316e800640 1 -- 172.21.15.27:0/2895049987 >> 172.21.15.27:0/2895049987 conn(0x7f31680b2a10 msgr2=0x7f31680b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:49.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.080+0000 7f316e800640 1 -- 172.21.15.27:0/2895049987 shutdown_connections 2024-01-15T18:10:49.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.080+0000 7f316e800640 1 -- 172.21.15.27:0/2895049987 wait complete. 2024-01-15T18:10:49.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.084+0000 7f316e800640 1 Processor -- start 2024-01-15T18:10:49.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.084+0000 7f316e800640 1 -- start start 2024-01-15T18:10:49.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.084+0000 7f316e800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f31680b20d0 0x7f31681658c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:49.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.084+0000 7f316e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31680cedc0 0x7f31680b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:49.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.084+0000 7f316e800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f316815e7d0 0x7f31680adef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:49.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.084+0000 7f316e800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31680b5980 con 0x7f31680cedc0 2024-01-15T18:10:49.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.084+0000 7f316e800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31680b5c60 con 0x7f316815e7d0 2024-01-15T18:10:49.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.084+0000 7f316e800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31680b5f40 con 0x7f31680b20d0 2024-01-15T18:10:49.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.084+0000 7f3166a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31680cedc0 0x7f31680b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:49.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.084+0000 7f3166a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31680cedc0 0x7f31680b4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49754/0 (socket says 172.21.15.27:49754) 2024-01-15T18:10:49.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.084+0000 7f3166a00640 1 -- 172.21.15.27:0/4146889266 learned_addr learned my addr 172.21.15.27:0/4146889266 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:49.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.084+0000 7f3167e00640 1 --2- 172.21.15.27:0/4146889266 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f316815e7d0 0x7f31680adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:49.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.084+0000 7f3166a00640 1 -- 172.21.15.27:0/4146889266 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f31680b20d0 msgr2=0x7f31681658c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:49.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.084+0000 7f3166a00640 1 --2- 172.21.15.27:0/4146889266 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f31680b20d0 0x7f31681658c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:49.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.084+0000 7f3166a00640 1 -- 172.21.15.27:0/4146889266 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f316815e7d0 msgr2=0x7f31680adef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:49.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.084+0000 7f3166a00640 1 --2- 172.21.15.27:0/4146889266 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f316815e7d0 0x7f31680adef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:49.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.084+0000 7f3166a00640 1 -- 172.21.15.27:0/4146889266 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3160009b30 con 0x7f31680cedc0 2024-01-15T18:10:49.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.084+0000 7f3167e00640 1 --2- 172.21.15.27:0/4146889266 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f316815e7d0 0x7f31680adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:49.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.084+0000 7f3166a00640 1 --2- 172.21.15.27:0/4146889266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31680cedc0 0x7f31680b4f80 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f315000c490 tx=0x7f3150001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:49.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.088+0000 7f315f400640 1 -- 172.21.15.27:0/4146889266 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3150003600 con 0x7f31680cedc0 2024-01-15T18:10:49.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.088+0000 7f315f400640 1 -- 172.21.15.27:0/4146889266 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3150003780 con 0x7f31680cedc0 2024-01-15T18:10:49.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.088+0000 7f316e800640 1 -- 172.21.15.27:0/4146889266 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31680b6220 con 0x7f31680cedc0 2024-01-15T18:10:49.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.088+0000 7f315f400640 1 -- 172.21.15.27:0/4146889266 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3150003b70 con 0x7f31680cedc0 2024-01-15T18:10:49.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.088+0000 7f316e800640 1 -- 172.21.15.27:0/4146889266 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31680b54c0 con 0x7f31680cedc0 2024-01-15T18:10:49.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.088+0000 7f316e800640 1 -- 172.21.15.27:0/4146889266 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31680b5740 con 0x7f31680cedc0 2024-01-15T18:10:49.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.088+0000 7f315f400640 1 -- 172.21.15.27:0/4146889266 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7f3150003cd0 con 0x7f31680cedc0 2024-01-15T18:10:49.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.092+0000 7f315f400640 1 --2- 172.21.15.27:0/4146889266 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3140067160 0x7f3140069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:49.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.092+0000 7f315f400640 1 -- 172.21.15.27:0/4146889266 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f31500824f0 con 0x7f31680cedc0 2024-01-15T18:10:49.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.092+0000 7f3167400640 1 --2- 172.21.15.27:0/4146889266 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3140067160 0x7f3140069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:49.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.092+0000 7f3167400640 1 --2- 172.21.15.27:0/4146889266 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3140067160 0x7f3140069620 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3158009690 tx=0x7f3158006c00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:49.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.100+0000 7f315f400640 1 -- 172.21.15.27:0/4146889266 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f31680b5740 con 0x7f31680cedc0 2024-01-15T18:10:49.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.472+0000 7f316e800640 1 -- 172.21.15.27:0/4146889266 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f31680b5740 con 0x7f31680cedc0 2024-01-15T18:10:49.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.476+0000 7f315f400640 1 -- 172.21.15.27:0/4146889266 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2154 (secure 0 0 0) 0x7f31680b5740 con 0x7f31680cedc0 2024-01-15T18:10:49.482 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:10:49.482 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:10:49.482 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:10:49.482 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:10:49.483 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:10:49.483 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:10:49.483 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:10:49.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:49.483 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:10:49.483 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:10:49.483 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:10:49.483 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:10:49.483 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:10:49.483 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:49.483 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:10:49.483 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:10:49.483 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:10:49.483 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:10:49.483 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:49.483 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 376, 2024-01-15T18:10:49.484 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:10:49.484 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:10:49.484 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:10:49.484 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:10:49.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:49.484 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:10:49.484 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 58, 2024-01-15T18:10:49.484 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:10:49.484 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:10:49.484 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:10:49.484 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:10:49.484 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:10:49.484 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:10:49.484 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:49.485 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:10:49.485 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:10:49.485 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:49.485 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:10:49.485 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:10:49.485 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:49.485 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:49.485 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:10:49.485 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:10:49.485 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 5273, 2024-01-15T18:10:49.485 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 21971867926, 2024-01-15T18:10:49.485 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 44123004928, 2024-01-15T18:10:49.485 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 728971108352, 2024-01-15T18:10:49.485 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:10:49.485 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 495365519, 2024-01-15T18:10:49.485 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 488398578, 2024-01-15T18:10:49.486 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 118, 2024-01-15T18:10:49.486 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 123 2024-01-15T18:10:49.486 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:49.486 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:10:49.486 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 38, 2024-01-15T18:10:49.486 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4, 2024-01-15T18:10:49.486 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:10:49.486 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:10:49.486 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:10:49.486 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:10:49.486 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:49.486 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 4, 2024-01-15T18:10:49.486 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:10:49.486 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:10:49.486 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:10:49.486 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 6480 2024-01-15T18:10:49.487 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:49.487 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:49.487 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:10:49.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:49.487 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:10:49.487 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:10:49.487 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:10:49.487 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:10:49.487 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:10:49.487 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:10:49.487 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:10:49.487 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:49.487 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:49.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:49.487 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:10:49.487 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 13, 2024-01-15T18:10:49.488 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:10:16.226140+0000", 2024-01-15T18:10:49.488 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:49.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:49.488 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:10:49.488 INFO:teuthology.orchestra.run.smithi027.stdout: "268d93d2-2eea-412c-b804-608e15344ce7": { 2024-01-15T18:10:49.488 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 85.714% (40s)\n [========================....] (remaining: 6s)", 2024-01-15T18:10:49.488 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.8571428656578064, 2024-01-15T18:10:49.488 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:49.488 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:49.488 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:49.488 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:10:49.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.484+0000 7f316e800640 1 -- 172.21.15.27:0/4146889266 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3140067160 msgr2=0x7f3140069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:49.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.484+0000 7f316e800640 1 --2- 172.21.15.27:0/4146889266 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3140067160 0x7f3140069620 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3158009690 tx=0x7f3158006c00 comp rx=0 tx=0).stop 2024-01-15T18:10:49.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.484+0000 7f316e800640 1 -- 172.21.15.27:0/4146889266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31680cedc0 msgr2=0x7f31680b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:49.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.484+0000 7f316e800640 1 --2- 172.21.15.27:0/4146889266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31680cedc0 0x7f31680b4f80 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f315000c490 tx=0x7f3150001e80 comp rx=0 tx=0).stop 2024-01-15T18:10:49.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.488+0000 7f316e800640 1 -- 172.21.15.27:0/4146889266 shutdown_connections 2024-01-15T18:10:49.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.488+0000 7f316e800640 1 --2- 172.21.15.27:0/4146889266 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3140067160 0x7f3140069620 secure :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3158009690 tx=0x7f3158006c00 comp rx=0 tx=0).stop 2024-01-15T18:10:49.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.488+0000 7f316e800640 1 --2- 172.21.15.27:0/4146889266 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f316815e7d0 0x7f31680adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:49.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.488+0000 7f316e800640 1 --2- 172.21.15.27:0/4146889266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31680cedc0 0x7f31680b4f80 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:49.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.488+0000 7f316e800640 1 --2- 172.21.15.27:0/4146889266 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f31680b20d0 0x7f31681658c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:49.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.488+0000 7f316e800640 1 -- 172.21.15.27:0/4146889266 >> 172.21.15.27:0/4146889266 conn(0x7f31680b2a10 msgr2=0x7f316815d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:49.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.488+0000 7f316e800640 1 -- 172.21.15.27:0/4146889266 shutdown_connections 2024-01-15T18:10:49.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.488+0000 7f316e800640 1 -- 172.21.15.27:0/4146889266 wait complete. 2024-01-15T18:10:49.532 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:10:49.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.752+0000 7f0617a00640 1 Processor -- start 2024-01-15T18:10:49.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.752+0000 7f0617a00640 1 -- start start 2024-01-15T18:10:49.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.752+0000 7f0617a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06100cca00 0x7f0610164f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:49.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.752+0000 7f0617a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f061015a460 con 0x7f061015db90 2024-01-15T18:10:49.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.752+0000 7f0617a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f061006dcd0 con 0x7f061015a080 2024-01-15T18:10:49.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.752+0000 7f0617a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f061006e8c0 con 0x7f06100cca00 2024-01-15T18:10:49.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.752+0000 7f060fe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f061015db90 0x7f06100b4110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:39146/0 (socket says 172.21.15.27:39146) 2024-01-15T18:10:49.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.752+0000 7f060fe00640 1 -- 172.21.15.27:0/2251424948 learned_addr learned my addr 172.21.15.27:0/2251424948 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:10:49.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.752+0000 7f0616400640 1 --2- 172.21.15.27:0/2251424948 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06100cca00 0x7f0610164f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:49.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.752+0000 7f060f400640 1 -- 172.21.15.27:0/2251424948 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846010196 0 0) 0x7f061015a460 con 0x7f061015db90 2024-01-15T18:10:49.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.752+0000 7f060f400640 1 -- 172.21.15.27:0/2251424948 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05f0003530 con 0x7f061015db90 2024-01-15T18:10:49.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.752+0000 7f060f400640 1 -- 172.21.15.27:0/2251424948 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213710238 0 0) 0x7f061006dcd0 con 0x7f061015a080 2024-01-15T18:10:49.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.752+0000 7f060f400640 1 -- 172.21.15.27:0/2251424948 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f061015a460 con 0x7f061015a080 2024-01-15T18:10:49.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.752+0000 7f0616400640 1 -- 172.21.15.27:0/2251424948 >> v1:172.21.15.93:6789/0 conn(0x7f061015a080 legacy=0x7f06100b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:49.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.752+0000 7f0616400640 1 -- 172.21.15.27:0/2251424948 >> v1:172.21.15.27:6789/0 conn(0x7f061015db90 legacy=0x7f06100b4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:49.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.752+0000 7f0616400640 1 -- 172.21.15.27:0/2251424948 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f061006df20 con 0x7f06100cca00 2024-01-15T18:10:49.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.756+0000 7f0616400640 1 --2- 172.21.15.27:0/2251424948 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06100cca00 0x7f0610164f20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0608009dc0 tx=0x7f060801c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e18a136d59de1dfd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:49.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.756+0000 7f060f400640 1 -- 172.21.15.27:0/2251424948 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06080a5a30 con 0x7f06100cca00 2024-01-15T18:10:49.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.756+0000 7f060f400640 1 -- 172.21.15.27:0/2251424948 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06080a5b90 con 0x7f06100cca00 2024-01-15T18:10:49.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.756+0000 7f060f400640 1 -- 172.21.15.27:0/2251424948 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06080ad440 con 0x7f06100cca00 2024-01-15T18:10:49.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.760+0000 7f0617a00640 1 -- 172.21.15.27:0/2251424948 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06100cca00 msgr2=0x7f0610164f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:49.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.760+0000 7f0617a00640 1 --2- 172.21.15.27:0/2251424948 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06100cca00 0x7f0610164f20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0608009dc0 tx=0x7f060801c970 comp rx=0 tx=0).stop 2024-01-15T18:10:49.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.760+0000 7f0617a00640 1 -- 172.21.15.27:0/2251424948 shutdown_connections 2024-01-15T18:10:49.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.760+0000 7f0617a00640 1 --2- 172.21.15.27:0/2251424948 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06100cca00 0x7f0610164f20 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:49.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.760+0000 7f0617a00640 1 -- 172.21.15.27:0/2251424948 >> 172.21.15.27:0/2251424948 conn(0x7f06100b2360 msgr2=0x7f06100b2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:49.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.760+0000 7f0617a00640 1 -- 172.21.15.27:0/2251424948 shutdown_connections 2024-01-15T18:10:49.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.760+0000 7f0617a00640 1 -- 172.21.15.27:0/2251424948 wait complete. 2024-01-15T18:10:49.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.760+0000 7f0617a00640 1 Processor -- start 2024-01-15T18:10:49.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.760+0000 7f0617a00640 1 -- start start 2024-01-15T18:10:49.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.760+0000 7f0617a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06100cca00 0x7f06100cf730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:49.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.760+0000 7f0617a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f061015a080 0x7f0610154b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:49.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.760+0000 7f0617a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06100b0ba0 0x7f0610141870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:49.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.760+0000 7f0617a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06100cfd30 con 0x7f061015a080 2024-01-15T18:10:49.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.760+0000 7f0617a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06100cd040 con 0x7f06100cca00 2024-01-15T18:10:49.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.760+0000 7f0617a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06100cd320 con 0x7f06100b0ba0 2024-01-15T18:10:49.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.760+0000 7f060fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f061015a080 0x7f0610154b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:49.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.760+0000 7f060fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f061015a080 0x7f0610154b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49762/0 (socket says 172.21.15.27:49762) 2024-01-15T18:10:49.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.760+0000 7f060fe00640 1 -- 172.21.15.27:0/1975292800 learned_addr learned my addr 172.21.15.27:0/1975292800 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:49.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.760+0000 7f0615000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06100cca00 0x7f06100cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:49.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.760+0000 7f0616400640 1 --2- 172.21.15.27:0/1975292800 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06100b0ba0 0x7f0610141870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:49.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.764+0000 7f0615000640 1 -- 172.21.15.27:0/1975292800 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06100b0ba0 msgr2=0x7f0610141870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:49.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.764+0000 7f0615000640 1 --2- 172.21.15.27:0/1975292800 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06100b0ba0 0x7f0610141870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:49.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.764+0000 7f0615000640 1 -- 172.21.15.27:0/1975292800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f061015a080 msgr2=0x7f0610154b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:49.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.764+0000 7f0615000640 1 --2- 172.21.15.27:0/1975292800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f061015a080 0x7f0610154b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:49.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.764+0000 7f0615000640 1 -- 172.21.15.27:0/1975292800 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0608009a70 con 0x7f06100cca00 2024-01-15T18:10:49.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.764+0000 7f060fe00640 1 --2- 172.21.15.27:0/1975292800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f061015a080 0x7f0610154b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:10:49.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.764+0000 7f0615000640 1 --2- 172.21.15.27:0/1975292800 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06100cca00 0x7f06100cf730 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f060000c430 tx=0x7f060000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:49.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.764+0000 7f060cc00640 1 -- 172.21.15.27:0/1975292800 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06000052d0 con 0x7f06100cca00 2024-01-15T18:10:49.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.764+0000 7f060cc00640 1 -- 172.21.15.27:0/1975292800 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0600005430 con 0x7f06100cca00 2024-01-15T18:10:49.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.764+0000 7f060cc00640 1 -- 172.21.15.27:0/1975292800 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0600003840 con 0x7f06100cca00 2024-01-15T18:10:49.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.764+0000 7f0617a00640 1 -- 172.21.15.27:0/1975292800 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06100cd600 con 0x7f06100cca00 2024-01-15T18:10:49.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.764+0000 7f0617a00640 1 -- 172.21.15.27:0/1975292800 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06100b1030 con 0x7f06100cca00 2024-01-15T18:10:49.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.764+0000 7f0617a00640 1 -- 172.21.15.27:0/1975292800 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0610074820 con 0x7f06100cca00 2024-01-15T18:10:49.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.764+0000 7f060cc00640 1 -- 172.21.15.27:0/1975292800 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7f06000039a0 con 0x7f06100cca00 2024-01-15T18:10:49.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.768+0000 7f060cc00640 1 --2- 172.21.15.27:0/1975292800 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05e8067160 0x7f05e8069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:49.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.768+0000 7f060cc00640 1 -- 172.21.15.27:0/1975292800 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f0600082460 con 0x7f06100cca00 2024-01-15T18:10:49.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.768+0000 7f060fe00640 1 --2- 172.21.15.27:0/1975292800 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05e8067160 0x7f05e8069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:49.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.768+0000 7f060fe00640 1 --2- 172.21.15.27:0/1975292800 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05e8067160 0x7f05e8069620 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f06100afc90 tx=0x7f05f8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:49.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.772+0000 7f060cc00640 1 -- 172.21.15.27:0/1975292800 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0600050fd0 con 0x7f06100cca00 2024-01-15T18:10:49.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.992+0000 7f0617a00640 1 -- 172.21.15.27:0/1975292800 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f06100b1250 con 0x7f06100cca00 2024-01-15T18:10:49.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.992+0000 7f060cc00640 1 -- 172.21.15.27:0/1975292800 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2154 (secure 0 0 0) 0x7f0600017090 con 0x7f06100cca00 2024-01-15T18:10:49.998 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:10:49.998 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:10:49.998 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:10:49.998 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:10:49.998 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:10:49.998 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:10:49.999 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:10:49.999 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:49.999 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:10:49.999 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:10:49.999 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:10:49.999 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:10:49.999 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:10:49.999 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:49.999 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:10:49.999 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:10:49.999 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:10:49.999 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:10:49.999 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:49.999 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 376, 2024-01-15T18:10:49.999 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:10:50.000 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:10:50.000 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:10:50.000 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:10:50.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:50.000 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:10:50.000 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 58, 2024-01-15T18:10:50.000 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:10:50.000 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:10:50.000 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:10:50.000 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:10:50.000 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:10:50.000 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:10:50.000 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:50.001 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:10:50.001 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:10:50.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:50.001 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:10:50.001 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:10:50.001 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:50.001 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:50.001 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:10:50.001 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:10:50.001 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 5273, 2024-01-15T18:10:50.001 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 21971867926, 2024-01-15T18:10:50.001 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 44123004928, 2024-01-15T18:10:50.001 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 728971108352, 2024-01-15T18:10:50.001 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:10:50.002 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 495365519, 2024-01-15T18:10:50.002 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 488398578, 2024-01-15T18:10:50.002 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 118, 2024-01-15T18:10:50.002 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 123 2024-01-15T18:10:50.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:50.002 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:10:50.002 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 38, 2024-01-15T18:10:50.002 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4, 2024-01-15T18:10:50.002 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:10:50.002 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:10:50.002 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:10:50.002 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:10:50.002 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:50.002 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 4, 2024-01-15T18:10:50.003 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:10:50.003 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:10:50.003 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:10:50.003 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 6480 2024-01-15T18:10:50.003 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:50.003 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:50.003 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:10:50.003 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:50.003 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:10:50.003 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:10:50.003 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:10:50.003 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:10:50.003 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:10:50.003 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:10:50.003 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:10:50.004 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:50.004 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:50.004 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:50.004 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:10:50.004 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 13, 2024-01-15T18:10:50.004 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:10:16.226140+0000", 2024-01-15T18:10:50.004 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:50.004 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:50.004 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:10:50.004 INFO:teuthology.orchestra.run.smithi027.stdout: "268d93d2-2eea-412c-b804-608e15344ce7": { 2024-01-15T18:10:50.004 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 85.714% (40s)\n [========================....] (remaining: 6s)", 2024-01-15T18:10:50.004 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.8571428656578064, 2024-01-15T18:10:50.004 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:50.004 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:50.005 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:50.005 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:10:50.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.992+0000 7f0617a00640 1 -- 172.21.15.27:0/1975292800 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05e8067160 msgr2=0x7f05e8069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:50.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.992+0000 7f0617a00640 1 --2- 172.21.15.27:0/1975292800 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05e8067160 0x7f05e8069620 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f06100afc90 tx=0x7f05f8009040 comp rx=0 tx=0).stop 2024-01-15T18:10:50.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.992+0000 7f0617a00640 1 -- 172.21.15.27:0/1975292800 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06100cca00 msgr2=0x7f06100cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:50.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.992+0000 7f0617a00640 1 --2- 172.21.15.27:0/1975292800 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06100cca00 0x7f06100cf730 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f060000c430 tx=0x7f060000c8f0 comp rx=0 tx=0).stop 2024-01-15T18:10:50.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.992+0000 7f0617a00640 1 -- 172.21.15.27:0/1975292800 shutdown_connections 2024-01-15T18:10:50.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.996+0000 7f0617a00640 1 --2- 172.21.15.27:0/1975292800 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05e8067160 0x7f05e8069620 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:50.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.996+0000 7f0617a00640 1 --2- 172.21.15.27:0/1975292800 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06100b0ba0 0x7f0610141870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:50.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.996+0000 7f0617a00640 1 --2- 172.21.15.27:0/1975292800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f061015a080 0x7f0610154b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:50.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.996+0000 7f0617a00640 1 --2- 172.21.15.27:0/1975292800 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06100cca00 0x7f06100cf730 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:50.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.996+0000 7f0617a00640 1 -- 172.21.15.27:0/1975292800 >> 172.21.15.27:0/1975292800 conn(0x7f06100b2360 msgr2=0x7f061006ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:50.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.996+0000 7f0617a00640 1 -- 172.21.15.27:0/1975292800 shutdown_connections 2024-01-15T18:10:50.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:49.996+0000 7f0617a00640 1 -- 172.21.15.27:0/1975292800 wait complete. 2024-01-15T18:10:50.021 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:10:50.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.160+0000 7fe53b600640 1 Processor -- start 2024-01-15T18:10:50.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.160+0000 7fe53b600640 1 -- start start 2024-01-15T18:10:50.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.160+0000 7fe53b600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe53415d4b0 0x7fe53406e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:50.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.160+0000 7fe53b600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fe534161060 0x7fe5341614a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:50.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.160+0000 7fe53b600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5341619e0 con 0x7fe53406a4f0 2024-01-15T18:10:50.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.160+0000 7fe53b600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe53406e930 con 0x7fe534161060 2024-01-15T18:10:50.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.160+0000 7fe53b600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe534070e90 con 0x7fe53415d4b0 2024-01-15T18:10:50.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.160+0000 7fe533e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fe534161060 0x7fe5341614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:50.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.160+0000 7fe538c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe53415d4b0 0x7fe53406e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:50.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.160+0000 7fe533e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fe534161060 0x7fe5341614a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:50718/0 (socket says 172.21.15.27:50718) 2024-01-15T18:10:50.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.160+0000 7fe533e00640 1 -- 172.21.15.27:0/2950628412 learned_addr learned my addr 172.21.15.27:0/2950628412 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:50.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.164+0000 7fe538c00640 1 -- 172.21.15.27:0/2950628412 >> v2:172.21.15.93:3300/0 conn(0x7fe534161060 msgr2=0x7fe5341614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:50.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.164+0000 7fe538c00640 1 --2- 172.21.15.27:0/2950628412 >> v2:172.21.15.93:3300/0 conn(0x7fe534161060 0x7fe5341614a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:50.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.164+0000 7fe538c00640 1 -- 172.21.15.27:0/2950628412 >> v1:172.21.15.93:6789/0 conn(0x7fe53406a4f0 legacy=0x7fe53406a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:50.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.164+0000 7fe538c00640 1 -- 172.21.15.27:0/2950628412 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe534071170 con 0x7fe53415d4b0 2024-01-15T18:10:50.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.164+0000 7fe533e00640 1 --2- 172.21.15.27:0/2950628412 >> v2:172.21.15.93:3300/0 conn(0x7fe534161060 0x7fe5341614a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:10:50.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.164+0000 7fe533400640 1 -- 172.21.15.27:0/2950628412 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28659458 0 0) 0x7fe5341619e0 con 0x7fe53406a4f0 2024-01-15T18:10:50.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.164+0000 7fe538c00640 1 --2- 172.21.15.27:0/2950628412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe53415d4b0 0x7fe53406e3f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe5240097c0 tx=0x7fe52401c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=263cd414ae1c75f2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:50.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.164+0000 7fe533400640 1 -- 172.21.15.27:0/2950628412 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5240ab650 con 0x7fe53415d4b0 2024-01-15T18:10:50.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.164+0000 7fe533400640 1 -- 172.21.15.27:0/2950628412 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5240ab7b0 con 0x7fe53415d4b0 2024-01-15T18:10:50.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.164+0000 7fe533400640 1 -- 172.21.15.27:0/2950628412 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5240abb40 con 0x7fe53415d4b0 2024-01-15T18:10:50.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.168+0000 7fe53b600640 1 -- 172.21.15.27:0/2950628412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe53415d4b0 msgr2=0x7fe53406e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:50.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.168+0000 7fe53b600640 1 --2- 172.21.15.27:0/2950628412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe53415d4b0 0x7fe53406e3f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe5240097c0 tx=0x7fe52401c450 comp rx=0 tx=0).stop 2024-01-15T18:10:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.168+0000 7fe53b600640 1 -- 172.21.15.27:0/2950628412 shutdown_connections 2024-01-15T18:10:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.168+0000 7fe53b600640 1 --2- 172.21.15.27:0/2950628412 >> v2:172.21.15.93:3300/0 conn(0x7fe534161060 0x7fe5341614a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.168+0000 7fe53b600640 1 --2- 172.21.15.27:0/2950628412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe53415d4b0 0x7fe53406e3f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.168+0000 7fe53b600640 1 -- 172.21.15.27:0/2950628412 >> 172.21.15.27:0/2950628412 conn(0x7fe5340cf510 msgr2=0x7fe5340cf920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.168+0000 7fe53b600640 1 -- 172.21.15.27:0/2950628412 shutdown_connections 2024-01-15T18:10:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.168+0000 7fe53b600640 1 -- 172.21.15.27:0/2950628412 wait complete. 2024-01-15T18:10:50.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.168+0000 7fe53b600640 1 Processor -- start 2024-01-15T18:10:50.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.168+0000 7fe53b600640 1 -- start start 2024-01-15T18:10:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.168+0000 7fe53b600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe53406a4f0 0x7fe5340b2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.168+0000 7fe53b600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe53415d4b0 0x7fe53414f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.168+0000 7fe53b600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe534161060 0x7fe53413be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.168+0000 7fe53b600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5341519a0 con 0x7fe53406a4f0 2024-01-15T18:10:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.168+0000 7fe53b600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53414e470 con 0x7fe534161060 2024-01-15T18:10:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.168+0000 7fe53b600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe53414e750 con 0x7fe53415d4b0 2024-01-15T18:10:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.168+0000 7fe533e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe53415d4b0 0x7fe53414f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.168+0000 7fe533e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe53415d4b0 0x7fe53414f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:60022/0 (socket says 172.21.15.27:60022) 2024-01-15T18:10:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.168+0000 7fe533e00640 1 -- 172.21.15.27:0/3565388257 learned_addr learned my addr 172.21.15.27:0/3565388257 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.168+0000 7fe539600640 1 --2- 172.21.15.27:0/3565388257 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe534161060 0x7fe53413be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.168+0000 7fe538c00640 1 --2- 172.21.15.27:0/3565388257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe53406a4f0 0x7fe5340b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.172+0000 7fe533e00640 1 -- 172.21.15.27:0/3565388257 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe534161060 msgr2=0x7fe53413be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.172+0000 7fe533e00640 1 --2- 172.21.15.27:0/3565388257 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe534161060 0x7fe53413be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.172+0000 7fe533e00640 1 -- 172.21.15.27:0/3565388257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe53406a4f0 msgr2=0x7fe5340b2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.172+0000 7fe533e00640 1 --2- 172.21.15.27:0/3565388257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe53406a4f0 0x7fe5340b2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.172+0000 7fe533e00640 1 -- 172.21.15.27:0/3565388257 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51c0094d0 con 0x7fe53415d4b0 2024-01-15T18:10:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.172+0000 7fe533e00640 1 --2- 172.21.15.27:0/3565388257 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe53415d4b0 0x7fe53414f8c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe51c00e9d0 tx=0x7fe51c00ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.172+0000 7fe530c00640 1 -- 172.21.15.27:0/3565388257 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c00ca60 con 0x7fe53415d4b0 2024-01-15T18:10:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.172+0000 7fe530c00640 1 -- 172.21.15.27:0/3565388257 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51c00cbc0 con 0x7fe53415d4b0 2024-01-15T18:10:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.172+0000 7fe53b600640 1 -- 172.21.15.27:0/3565388257 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe524009420 con 0x7fe53415d4b0 2024-01-15T18:10:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.172+0000 7fe530c00640 1 -- 172.21.15.27:0/3565388257 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c013470 con 0x7fe53415d4b0 2024-01-15T18:10:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.172+0000 7fe53b600640 1 -- 172.21.15.27:0/3565388257 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe534151550 con 0x7fe53415d4b0 2024-01-15T18:10:50.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.172+0000 7fe53b600640 1 -- 172.21.15.27:0/3565388257 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe53415d890 con 0x7fe53415d4b0 2024-01-15T18:10:50.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.172+0000 7fe530c00640 1 -- 172.21.15.27:0/3565388257 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7fe51c0135d0 con 0x7fe53415d4b0 2024-01-15T18:10:50.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.172+0000 7fe530c00640 1 --2- 172.21.15.27:0/3565388257 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe50c067160 0x7fe50c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:50.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.172+0000 7fe530c00640 1 -- 172.21.15.27:0/3565388257 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe51c083110 con 0x7fe53415d4b0 2024-01-15T18:10:50.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.176+0000 7fe538c00640 1 --2- 172.21.15.27:0/3565388257 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe50c067160 0x7fe50c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.176+0000 7fe538c00640 1 --2- 172.21.15.27:0/3565388257 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe50c067160 0x7fe50c069620 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe5240af000 tx=0x7fe5240a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:50.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.176+0000 7fe530c00640 1 -- 172.21.15.27:0/3565388257 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe51c051c80 con 0x7fe53415d4b0 2024-01-15T18:10:50.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.392+0000 7fe53b600640 1 -- 172.21.15.27:0/3565388257 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe53404e850 con 0x7fe53415d4b0 2024-01-15T18:10:50.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.392+0000 7fe530c00640 1 -- 172.21.15.27:0/3565388257 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2420 (secure 0 0 0) 0x7fe51c017990 con 0x7fe53415d4b0 2024-01-15T18:10:50.400 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:10:50.400 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:10:50.400 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:10:50.400 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:10:50.400 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:10:50.400 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:10:50.400 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:10:50.400 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:50.400 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:10:50.400 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:10:50.400 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:10:50.401 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:10:50.401 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:10:50.401 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:50.401 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:10:50.401 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:10:50.401 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:10:50.401 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:10:50.401 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:50.401 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 377, 2024-01-15T18:10:50.401 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:10:50.401 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:10:50.401 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:10:50.401 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:10:50.401 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:50.401 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:10:50.401 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 58, 2024-01-15T18:10:50.402 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:10:50.402 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:10:50.402 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:10:50.402 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:10:50.402 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:10:50.402 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:10:50.402 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:50.402 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:10:50.402 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:10:50.402 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:50.402 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:10:50.402 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:10:50.402 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:50.402 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:50.402 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:10:50.403 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:10:50.403 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 5474, 2024-01-15T18:10:50.403 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 22806534777, 2024-01-15T18:10:50.403 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 45603500032, 2024-01-15T18:10:50.403 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 727490613248, 2024-01-15T18:10:50.403 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:10:50.403 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 562801562, 2024-01-15T18:10:50.403 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 557930805, 2024-01-15T18:10:50.403 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 134, 2024-01-15T18:10:50.403 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 140 2024-01-15T18:10:50.403 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:50.403 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:10:50.403 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 38, 2024-01-15T18:10:50.403 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4, 2024-01-15T18:10:50.403 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:10:50.404 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:10:50.404 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:10:50.404 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:10:50.404 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:50.404 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 4, 2024-01-15T18:10:50.404 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:10:50.404 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:10:50.404 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:10:50.404 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 6480 2024-01-15T18:10:50.404 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:50.404 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:50.404 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:10:50.404 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:50.404 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:10:50.404 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:10:50.405 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:10:50.405 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:10:50.405 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:10:50.405 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:10:50.405 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:10:50.405 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:50.405 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:50.405 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:50.405 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:10:50.405 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 13, 2024-01-15T18:10:50.405 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:10:16.226140+0000", 2024-01-15T18:10:50.405 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:50.405 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:50.405 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:10:50.405 INFO:teuthology.orchestra.run.smithi027.stdout: "268d93d2-2eea-412c-b804-608e15344ce7": { 2024-01-15T18:10:50.405 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 95.239% (44s)\n [==========================..] (remaining: 2s)", 2024-01-15T18:10:50.406 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.95238572359085083, 2024-01-15T18:10:50.406 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:50.406 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:50.406 INFO:teuthology.orchestra.run.smithi027.stdout: "9a27fb80-350b-410b-a65e-91a770cccfdf": { 2024-01-15T18:10:50.406 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "Subvolume \"subvol_clone91817\" has been 66.67% cloned (0s)\n [==================..........] ", 2024-01-15T18:10:50.406 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.66670000553131104, 2024-01-15T18:10:50.406 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:50.406 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:50.406 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:50.406 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:10:50.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.396+0000 7fe53b600640 1 -- 172.21.15.27:0/3565388257 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe50c067160 msgr2=0x7fe50c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:50.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.396+0000 7fe53b600640 1 --2- 172.21.15.27:0/3565388257 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe50c067160 0x7fe50c069620 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fe5240af000 tx=0x7fe5240a7000 comp rx=0 tx=0).stop 2024-01-15T18:10:50.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.396+0000 7fe53b600640 1 -- 172.21.15.27:0/3565388257 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe53415d4b0 msgr2=0x7fe53414f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:50.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.396+0000 7fe53b600640 1 --2- 172.21.15.27:0/3565388257 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe53415d4b0 0x7fe53414f8c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe51c00e9d0 tx=0x7fe51c00ee90 comp rx=0 tx=0).stop 2024-01-15T18:10:50.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.396+0000 7fe53b600640 1 -- 172.21.15.27:0/3565388257 shutdown_connections 2024-01-15T18:10:50.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.396+0000 7fe53b600640 1 --2- 172.21.15.27:0/3565388257 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe50c067160 0x7fe50c069620 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:50.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.396+0000 7fe53b600640 1 --2- 172.21.15.27:0/3565388257 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe534161060 0x7fe53413be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:50.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.396+0000 7fe53b600640 1 --2- 172.21.15.27:0/3565388257 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe53415d4b0 0x7fe53414f8c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:50.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.396+0000 7fe53b600640 1 --2- 172.21.15.27:0/3565388257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe53406a4f0 0x7fe5340b2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:50.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.396+0000 7fe53b600640 1 -- 172.21.15.27:0/3565388257 >> 172.21.15.27:0/3565388257 conn(0x7fe5340cf510 msgr2=0x7fe53406a940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:50.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.396+0000 7fe53b600640 1 -- 172.21.15.27:0/3565388257 shutdown_connections 2024-01-15T18:10:50.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:50.396+0000 7fe53b600640 1 -- 172.21.15.27:0/3565388257 wait complete. 2024-01-15T18:10:52.424 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:10:52.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.640+0000 7f0815400640 1 Processor -- start 2024-01-15T18:10:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.640+0000 7f0815400640 1 -- start start 2024-01-15T18:10:52.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.640+0000 7f0815400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0810145f40 0x7f0810146420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:52.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.640+0000 7f0815400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0810146960 0x7f08100712f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:52.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.640+0000 7f0815400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0810071830 0x7f081006a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:52.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.640+0000 7f0815400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0810071ce0 con 0x7f0810145f40 2024-01-15T18:10:52.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.640+0000 7f0815400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0810160d10 con 0x7f0810146960 2024-01-15T18:10:52.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.640+0000 7f0815400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0810161890 con 0x7f0810071830 2024-01-15T18:10:52.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.640+0000 7f080fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0810145f40 0x7f0810146420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:52.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.640+0000 7f080fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0810145f40 0x7f0810146420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49780/0 (socket says 172.21.15.27:49780) 2024-01-15T18:10:52.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.640+0000 7f080fe00640 1 -- 172.21.15.27:0/3084879595 learned_addr learned my addr 172.21.15.27:0/3084879595 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:52.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.644+0000 7f080f400640 1 --2- 172.21.15.27:0/3084879595 >> v2:172.21.15.93:3300/0 conn(0x7f0810146960 0x7f08100712f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:52.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.644+0000 7f080fe00640 1 -- 172.21.15.27:0/3084879595 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0810071830 msgr2=0x7f081006a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:52.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.644+0000 7f080fe00640 1 --2- 172.21.15.27:0/3084879595 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0810071830 0x7f081006a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:52.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.644+0000 7f080fe00640 1 -- 172.21.15.27:0/3084879595 >> v2:172.21.15.93:3300/0 conn(0x7f0810146960 msgr2=0x7f08100712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:52.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.644+0000 7f080fe00640 1 --2- 172.21.15.27:0/3084879595 >> v2:172.21.15.93:3300/0 conn(0x7f0810146960 0x7f08100712f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:52.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.644+0000 7f080fe00640 1 -- 172.21.15.27:0/3084879595 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0810160f60 con 0x7f0810145f40 2024-01-15T18:10:52.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.648+0000 7f080fe00640 1 --2- 172.21.15.27:0/3084879595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0810145f40 0x7f0810146420 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f0810146fc0 tx=0x7f080001c240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d7bf8bdf61f80d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:52.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.648+0000 7f080ea00640 1 -- 172.21.15.27:0/3084879595 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08000ab600 con 0x7f0810145f40 2024-01-15T18:10:52.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.648+0000 7f080ea00640 1 -- 172.21.15.27:0/3084879595 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08000a7050 con 0x7f0810145f40 2024-01-15T18:10:52.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.648+0000 7f080ea00640 1 -- 172.21.15.27:0/3084879595 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08000ab980 con 0x7f0810145f40 2024-01-15T18:10:52.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.648+0000 7f0815400640 1 -- 172.21.15.27:0/3084879595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0810145f40 msgr2=0x7f0810146420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:52.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.648+0000 7f0815400640 1 --2- 172.21.15.27:0/3084879595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0810145f40 0x7f0810146420 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f0810146fc0 tx=0x7f080001c240 comp rx=0 tx=0).stop 2024-01-15T18:10:52.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.652+0000 7f0815400640 1 -- 172.21.15.27:0/3084879595 shutdown_connections 2024-01-15T18:10:52.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.652+0000 7f0815400640 1 --2- 172.21.15.27:0/3084879595 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0810071830 0x7f081006a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:52.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.652+0000 7f0815400640 1 --2- 172.21.15.27:0/3084879595 >> v2:172.21.15.93:3300/0 conn(0x7f0810146960 0x7f08100712f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:52.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.652+0000 7f0815400640 1 --2- 172.21.15.27:0/3084879595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0810145f40 0x7f0810146420 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:52.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.652+0000 7f0815400640 1 -- 172.21.15.27:0/3084879595 >> 172.21.15.27:0/3084879595 conn(0x7f08100b1a00 msgr2=0x7f08100b1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:52.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.652+0000 7f0815400640 1 -- 172.21.15.27:0/3084879595 shutdown_connections 2024-01-15T18:10:52.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.652+0000 7f0815400640 1 -- 172.21.15.27:0/3084879595 wait complete. 2024-01-15T18:10:52.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.656+0000 7f0815400640 1 Processor -- start 2024-01-15T18:10:52.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.656+0000 7f0815400640 1 -- start start 2024-01-15T18:10:52.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.660+0000 7f0815400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0810071830 0x7f08101600d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:52.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.660+0000 7f0815400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0810145f40 0x7f0810099a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:52.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.660+0000 7f0815400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0810146960 0x7f0810137210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:52.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.660+0000 7f0815400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0810160610 con 0x7f0810146960 2024-01-15T18:10:52.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.660+0000 7f0815400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0810099f50 con 0x7f0810071830 2024-01-15T18:10:52.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.660+0000 7f0815400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f081015e4b0 con 0x7f0810145f40 2024-01-15T18:10:52.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.660+0000 7f080fe00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0810071830 0x7f08101600d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:52.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.660+0000 7f080fe00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0810071830 0x7f08101600d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:50752/0 (socket says 172.21.15.27:50752) 2024-01-15T18:10:52.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.660+0000 7f080fe00640 1 -- 172.21.15.27:0/1673376566 learned_addr learned my addr 172.21.15.27:0/1673376566 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:52.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.660+0000 7f080f400640 1 --2- 172.21.15.27:0/1673376566 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0810145f40 0x7f0810099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:52.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.660+0000 7f0814a00640 1 --2- 172.21.15.27:0/1673376566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0810146960 0x7f0810137210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:52.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.660+0000 7f080fe00640 1 -- 172.21.15.27:0/1673376566 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0810145f40 msgr2=0x7f0810099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:52.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.660+0000 7f080fe00640 1 --2- 172.21.15.27:0/1673376566 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0810145f40 0x7f0810099a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:52.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.664+0000 7f080fe00640 1 -- 172.21.15.27:0/1673376566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0810146960 msgr2=0x7f0810137210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:52.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.664+0000 7f080fe00640 1 --2- 172.21.15.27:0/1673376566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0810146960 0x7f0810137210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:52.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.664+0000 7f080fe00640 1 -- 172.21.15.27:0/1673376566 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08000094d0 con 0x7f0810071830 2024-01-15T18:10:52.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.664+0000 7f080fe00640 1 --2- 172.21.15.27:0/1673376566 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0810071830 0x7f08101600d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f080001cae0 tx=0x7f080001aa80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:52.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.664+0000 7f0807e00640 1 -- 172.21.15.27:0/1673376566 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08000b6420 con 0x7f0810071830 2024-01-15T18:10:52.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.664+0000 7f0807e00640 1 -- 172.21.15.27:0/1673376566 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08000b6580 con 0x7f0810071830 2024-01-15T18:10:52.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.664+0000 7f0807e00640 1 -- 172.21.15.27:0/1673376566 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08000b6910 con 0x7f0810071830 2024-01-15T18:10:52.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.668+0000 7f0815400640 1 -- 172.21.15.27:0/1673376566 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f081015d7f0 con 0x7f0810071830 2024-01-15T18:10:52.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.668+0000 7f0815400640 1 -- 172.21.15.27:0/1673376566 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f081015ddb0 con 0x7f0810071830 2024-01-15T18:10:52.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.672+0000 7f0815400640 1 -- 172.21.15.27:0/1673376566 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f081004e850 con 0x7f0810071830 2024-01-15T18:10:52.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.672+0000 7f0807e00640 1 -- 172.21.15.27:0/1673376566 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7f08000b2040 con 0x7f0810071830 2024-01-15T18:10:52.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.676+0000 7f0807e00640 1 --2- 172.21.15.27:0/1673376566 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f07e8067230 0x7f07e80696f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:52.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.676+0000 7f0807e00640 1 -- 172.21.15.27:0/1673376566 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f080011d230 con 0x7f0810071830 2024-01-15T18:10:52.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.676+0000 7f080f400640 1 --2- 172.21.15.27:0/1673376566 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f07e8067230 0x7f07e80696f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:52.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.676+0000 7f080f400640 1 --2- 172.21.15.27:0/1673376566 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f07e8067230 0x7f07e80696f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f081009dd70 tx=0x7f07f8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:52.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:52.680+0000 7f0807e00640 1 -- 172.21.15.27:0/1673376566 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f08000ebda0 con 0x7f0810071830 2024-01-15T18:10:53.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:53.056+0000 7f0815400640 1 -- 172.21.15.27:0/1673376566 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f08100b2830 con 0x7f0810071830 2024-01-15T18:10:53.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:53.060+0000 7f0807e00640 1 -- 172.21.15.27:0/1673376566 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2420 (secure 0 0 0) 0x7f08000b0990 con 0x7f0810071830 2024-01-15T18:10:53.065 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:10:53.065 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:10:53.065 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:10:53.065 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:10:53.065 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:10:53.065 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:10:53.065 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:10:53.065 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:53.065 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:10:53.066 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:10:53.066 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:10:53.066 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:10:53.066 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:10:53.066 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:53.066 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:10:53.066 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:10:53.066 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:10:53.066 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:10:53.066 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:53.066 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 380, 2024-01-15T18:10:53.066 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:10:53.066 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:10:53.066 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:10:53.066 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:10:53.066 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:53.067 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:10:53.067 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 58, 2024-01-15T18:10:53.067 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:10:53.067 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:10:53.067 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:10:53.067 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:10:53.067 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:10:53.067 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:10:53.067 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:53.067 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:10:53.067 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:10:53.067 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:53.067 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:10:53.067 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:10:53.067 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:53.068 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:53.068 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:10:53.068 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:10:53.068 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 5908, 2024-01-15T18:10:53.068 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 24627051424, 2024-01-15T18:10:53.068 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 49512849408, 2024-01-15T18:10:53.068 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 723581263872, 2024-01-15T18:10:53.068 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:10:53.068 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 657824235, 2024-01-15T18:10:53.068 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 652271674, 2024-01-15T18:10:53.068 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 157, 2024-01-15T18:10:53.068 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 167 2024-01-15T18:10:53.068 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:53.069 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:10:53.069 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 38, 2024-01-15T18:10:53.069 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4, 2024-01-15T18:10:53.069 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:10:53.069 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:10:53.069 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:10:53.069 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:10:53.069 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:53.069 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 4, 2024-01-15T18:10:53.069 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:10:53.069 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:10:53.069 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:10:53.069 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 6480 2024-01-15T18:10:53.069 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:53.070 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:53.070 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:10:53.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:53.070 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:10:53.070 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:10:53.070 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:10:53.070 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:10:53.070 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:10:53.070 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:10:53.070 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:10:53.070 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:53.070 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:53.070 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:53.070 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:10:53.070 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 13, 2024-01-15T18:10:53.071 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:10:16.226140+0000", 2024-01-15T18:10:53.071 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:53.071 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:53.071 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:10:53.071 INFO:teuthology.orchestra.run.smithi027.stdout: "268d93d2-2eea-412c-b804-608e15344ce7": { 2024-01-15T18:10:53.071 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 95.239% (44s)\n [==========================..] (remaining: 2s)", 2024-01-15T18:10:53.071 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.95238572359085083, 2024-01-15T18:10:53.071 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:53.071 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:53.071 INFO:teuthology.orchestra.run.smithi027.stdout: "9a27fb80-350b-410b-a65e-91a770cccfdf": { 2024-01-15T18:10:53.071 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "Subvolume \"subvol_clone91817\" has been 66.67% cloned (0s)\n [==================..........] ", 2024-01-15T18:10:53.071 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.66670000553131104, 2024-01-15T18:10:53.071 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:10:53.071 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:53.071 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:53.071 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:10:53.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:53.060+0000 7f0815400640 1 -- 172.21.15.27:0/1673376566 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f07e8067230 msgr2=0x7f07e80696f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:53.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:53.060+0000 7f0815400640 1 --2- 172.21.15.27:0/1673376566 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f07e8067230 0x7f07e80696f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f081009dd70 tx=0x7f07f8009190 comp rx=0 tx=0).stop 2024-01-15T18:10:53.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:53.060+0000 7f0815400640 1 -- 172.21.15.27:0/1673376566 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0810071830 msgr2=0x7f08101600d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:53.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:53.060+0000 7f0815400640 1 --2- 172.21.15.27:0/1673376566 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0810071830 0x7f08101600d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f080001cae0 tx=0x7f080001aa80 comp rx=0 tx=0).stop 2024-01-15T18:10:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:53.064+0000 7f0815400640 1 -- 172.21.15.27:0/1673376566 shutdown_connections 2024-01-15T18:10:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:53.064+0000 7f0815400640 1 --2- 172.21.15.27:0/1673376566 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f07e8067230 0x7f07e80696f0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:53.064+0000 7f0815400640 1 --2- 172.21.15.27:0/1673376566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0810146960 0x7f0810137210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:53.064+0000 7f0815400640 1 --2- 172.21.15.27:0/1673376566 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0810145f40 0x7f0810099a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:53.064+0000 7f0815400640 1 --2- 172.21.15.27:0/1673376566 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0810071830 0x7f08101600d0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:53.064+0000 7f0815400640 1 -- 172.21.15.27:0/1673376566 >> 172.21.15.27:0/1673376566 conn(0x7f08100b1a00 msgr2=0x7f08100b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:53.064+0000 7f0815400640 1 -- 172.21.15.27:0/1673376566 shutdown_connections 2024-01-15T18:10:53.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:53.064+0000 7f0815400640 1 -- 172.21.15.27:0/1673376566 wait complete. 2024-01-15T18:10:55.114 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:10:55.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.264+0000 7fe247000640 1 Processor -- start 2024-01-15T18:10:55.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.264+0000 7fe247000640 1 -- start start 2024-01-15T18:10:55.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.264+0000 7fe247000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fe2400b2390 0x7fe2400cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:55.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.264+0000 7fe247000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe24006bb80 0x7fe2400cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:55.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.264+0000 7fe247000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe2400cfcc0 0x7fe2400cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:55.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.264+0000 7fe247000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2400b2770 con 0x7fe24006bb80 2024-01-15T18:10:55.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.264+0000 7fe247000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe24006c060 con 0x7fe2400b2390 2024-01-15T18:10:55.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.264+0000 7fe247000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe24006c340 con 0x7fe2400cfcc0 2024-01-15T18:10:55.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.264+0000 7fe23f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe24006bb80 0x7fe2400cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:55.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.264+0000 7fe23f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe24006bb80 0x7fe2400cd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34342/0 (socket says 172.21.15.27:34342) 2024-01-15T18:10:55.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.264+0000 7fe23f400640 1 -- 172.21.15.27:0/3388169414 learned_addr learned my addr 172.21.15.27:0/3388169414 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:55.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.264+0000 7fe23fe00640 1 --2- 172.21.15.27:0/3388169414 >> v2:172.21.15.93:3300/0 conn(0x7fe2400b2390 0x7fe2400cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:55.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.264+0000 7fe245000640 1 --2- 172.21.15.27:0/3388169414 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe2400cfcc0 0x7fe2400cddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:55.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.264+0000 7fe23f400640 1 -- 172.21.15.27:0/3388169414 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe2400cfcc0 msgr2=0x7fe2400cddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:55.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.264+0000 7fe23f400640 1 --2- 172.21.15.27:0/3388169414 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe2400cfcc0 0x7fe2400cddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:55.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.264+0000 7fe23f400640 1 -- 172.21.15.27:0/3388169414 >> v2:172.21.15.93:3300/0 conn(0x7fe2400b2390 msgr2=0x7fe2400cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:55.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.264+0000 7fe23f400640 1 --2- 172.21.15.27:0/3388169414 >> v2:172.21.15.93:3300/0 conn(0x7fe2400b2390 0x7fe2400cf780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:55.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.264+0000 7fe23f400640 1 -- 172.21.15.27:0/3388169414 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe24006c620 con 0x7fe24006bb80 2024-01-15T18:10:55.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.264+0000 7fe23fe00640 1 --2- 172.21.15.27:0/3388169414 >> v2:172.21.15.93:3300/0 conn(0x7fe2400b2390 0x7fe2400cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:10:55.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.268+0000 7fe23f400640 1 --2- 172.21.15.27:0/3388169414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe24006bb80 0x7fe2400cd880 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fe240071b00 tx=0x7fe22801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b940bd092d754c76 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:55.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.268+0000 7fe23ea00640 1 -- 172.21.15.27:0/3388169414 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2280aa5f0 con 0x7fe24006bb80 2024-01-15T18:10:55.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.268+0000 7fe23ea00640 1 -- 172.21.15.27:0/3388169414 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2280aa750 con 0x7fe24006bb80 2024-01-15T18:10:55.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.268+0000 7fe23ea00640 1 -- 172.21.15.27:0/3388169414 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2280aaae0 con 0x7fe24006bb80 2024-01-15T18:10:55.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.268+0000 7fe247000640 1 -- 172.21.15.27:0/3388169414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe24006bb80 msgr2=0x7fe2400cd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:55.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.268+0000 7fe247000640 1 --2- 172.21.15.27:0/3388169414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe24006bb80 0x7fe2400cd880 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fe240071b00 tx=0x7fe22801c4d0 comp rx=0 tx=0).stop 2024-01-15T18:10:55.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.268+0000 7fe247000640 1 -- 172.21.15.27:0/3388169414 shutdown_connections 2024-01-15T18:10:55.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.268+0000 7fe247000640 1 --2- 172.21.15.27:0/3388169414 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe2400cfcc0 0x7fe2400cddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:55.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.268+0000 7fe247000640 1 --2- 172.21.15.27:0/3388169414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe24006bb80 0x7fe2400cd880 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:55.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.268+0000 7fe247000640 1 --2- 172.21.15.27:0/3388169414 >> v2:172.21.15.93:3300/0 conn(0x7fe2400b2390 0x7fe2400cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:55.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.268+0000 7fe247000640 1 -- 172.21.15.27:0/3388169414 >> 172.21.15.27:0/3388169414 conn(0x7fe2400b2a10 msgr2=0x7fe2400b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:55.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.268+0000 7fe247000640 1 -- 172.21.15.27:0/3388169414 shutdown_connections 2024-01-15T18:10:55.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.268+0000 7fe247000640 1 -- 172.21.15.27:0/3388169414 wait complete. 2024-01-15T18:10:55.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.268+0000 7fe247000640 1 Processor -- start 2024-01-15T18:10:55.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.272+0000 7fe247000640 1 -- start start 2024-01-15T18:10:55.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.272+0000 7fe247000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe24006bb80 0x7fe24009e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:55.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.272+0000 7fe247000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2400b2390 0x7fe24015a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:55.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.272+0000 7fe247000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe2400cfcc0 0x7fe2401584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:55.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.272+0000 7fe247000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24015a050 con 0x7fe2400b2390 2024-01-15T18:10:55.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.272+0000 7fe247000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2400b4510 con 0x7fe24006bb80 2024-01-15T18:10:55.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.272+0000 7fe247000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2400b3b00 con 0x7fe2400cfcc0 2024-01-15T18:10:55.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.272+0000 7fe23f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2400b2390 0x7fe24015a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:55.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.272+0000 7fe23f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2400b2390 0x7fe24015a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34344/0 (socket says 172.21.15.27:34344) 2024-01-15T18:10:55.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.272+0000 7fe23f400640 1 -- 172.21.15.27:0/3638708406 learned_addr learned my addr 172.21.15.27:0/3638708406 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:55.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.272+0000 7fe245000640 1 --2- 172.21.15.27:0/3638708406 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe2400cfcc0 0x7fe2401584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:55.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.272+0000 7fe23fe00640 1 --2- 172.21.15.27:0/3638708406 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe24006bb80 0x7fe24009e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:55.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.272+0000 7fe23f400640 1 -- 172.21.15.27:0/3638708406 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe2400cfcc0 msgr2=0x7fe2401584a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:55.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.272+0000 7fe23f400640 1 --2- 172.21.15.27:0/3638708406 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe2400cfcc0 0x7fe2401584a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:55.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.272+0000 7fe23f400640 1 -- 172.21.15.27:0/3638708406 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe24006bb80 msgr2=0x7fe24009e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:55.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.272+0000 7fe23f400640 1 --2- 172.21.15.27:0/3638708406 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe24006bb80 0x7fe24009e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:55.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.272+0000 7fe23f400640 1 -- 172.21.15.27:0/3638708406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2280094d0 con 0x7fe2400b2390 2024-01-15T18:10:55.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.272+0000 7fe23f400640 1 --2- 172.21.15.27:0/3638708406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2400b2390 0x7fe24015a1c0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fe2280ac040 tx=0x7fe22801ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:55.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.272+0000 7fe237e00640 1 -- 172.21.15.27:0/3638708406 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2280b3800 con 0x7fe2400b2390 2024-01-15T18:10:55.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.272+0000 7fe237e00640 1 -- 172.21.15.27:0/3638708406 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2280b3980 con 0x7fe2400b2390 2024-01-15T18:10:55.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.272+0000 7fe237e00640 1 -- 172.21.15.27:0/3638708406 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2280b3d30 con 0x7fe2400b2390 2024-01-15T18:10:55.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.272+0000 7fe247000640 1 -- 172.21.15.27:0/3638708406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2400b4720 con 0x7fe2400b2390 2024-01-15T18:10:55.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.272+0000 7fe247000640 1 -- 172.21.15.27:0/3638708406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2400b6f00 con 0x7fe2400b2390 2024-01-15T18:10:55.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.272+0000 7fe247000640 1 -- 172.21.15.27:0/3638708406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2400d07c0 con 0x7fe2400b2390 2024-01-15T18:10:55.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.276+0000 7fe237e00640 1 -- 172.21.15.27:0/3638708406 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7fe2280b2380 con 0x7fe2400b2390 2024-01-15T18:10:55.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.276+0000 7fe237e00640 1 --2- 172.21.15.27:0/3638708406 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe218067160 0x7fe218069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:55.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.276+0000 7fe237e00640 1 -- 172.21.15.27:0/3638708406 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe2280a7070 con 0x7fe2400b2390 2024-01-15T18:10:55.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.276+0000 7fe23fe00640 1 --2- 172.21.15.27:0/3638708406 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe218067160 0x7fe218069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:55.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.276+0000 7fe23fe00640 1 --2- 172.21.15.27:0/3638708406 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe218067160 0x7fe218069620 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe2300045f0 tx=0x7fe230008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:55.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.280+0000 7fe237e00640 1 -- 172.21.15.27:0/3638708406 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe2280ec0d0 con 0x7fe2400b2390 2024-01-15T18:10:55.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.492+0000 7fe247000640 1 -- 172.21.15.27:0/3638708406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe24004e850 con 0x7fe2400b2390 2024-01-15T18:10:55.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.492+0000 7fe237e00640 1 -- 172.21.15.27:0/3638708406 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1870 (secure 0 0 0) 0x7fe2280c3090 con 0x7fe2400b2390 2024-01-15T18:10:55.499 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:10:55.499 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:10:55.499 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:10:55.499 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:10:55.499 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:10:55.499 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:10:55.499 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:10:55.499 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:55.500 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:10:55.500 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:10:55.500 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:10:55.500 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:10:55.500 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:10:55.500 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:55.500 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:10:55.500 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:10:55.500 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:10:55.500 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:10:55.500 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:55.500 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 382, 2024-01-15T18:10:55.500 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:10:55.500 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:10:55.501 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:10:55.501 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:10:55.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:55.501 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:10:55.501 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 58, 2024-01-15T18:10:55.501 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:10:55.501 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:10:55.501 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:10:55.501 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:10:55.501 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:10:55.501 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:10:55.501 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:55.501 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:10:55.501 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:10:55.502 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:55.502 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:10:55.502 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:10:55.502 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:55.502 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:55.502 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:10:55.502 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:10:55.502 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 5970, 2024-01-15T18:10:55.502 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 24887098272, 2024-01-15T18:10:55.502 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 49932275712, 2024-01-15T18:10:55.502 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 723161837568, 2024-01-15T18:10:55.502 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:10:55.502 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 505148988, 2024-01-15T18:10:55.502 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 512163546, 2024-01-15T18:10:55.503 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 120, 2024-01-15T18:10:55.503 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 130 2024-01-15T18:10:55.503 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:55.503 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:10:55.503 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 38, 2024-01-15T18:10:55.503 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4, 2024-01-15T18:10:55.503 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:10:55.503 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:10:55.503 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:10:55.503 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:10:55.503 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:10:55.503 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 4, 2024-01-15T18:10:55.503 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:10:55.503 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:10:55.503 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:10:55.504 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 6480 2024-01-15T18:10:55.504 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:10:55.504 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:55.504 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:10:55.504 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:55.504 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:10:55.504 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:10:55.504 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:10:55.504 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:10:55.504 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:10:55.504 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:10:55.504 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:10:55.505 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:10:55.505 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:55.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:55.505 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:10:55.505 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 13, 2024-01-15T18:10:55.505 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:10:16.226140+0000", 2024-01-15T18:10:55.505 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:10:55.505 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:10:55.505 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": {} 2024-01-15T18:10:55.505 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:10:55.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.500+0000 7fe247000640 1 -- 172.21.15.27:0/3638708406 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe218067160 msgr2=0x7fe218069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:55.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.500+0000 7fe247000640 1 --2- 172.21.15.27:0/3638708406 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe218067160 0x7fe218069620 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe2300045f0 tx=0x7fe230008040 comp rx=0 tx=0).stop 2024-01-15T18:10:55.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.500+0000 7fe247000640 1 -- 172.21.15.27:0/3638708406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2400b2390 msgr2=0x7fe24015a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:55.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.500+0000 7fe247000640 1 --2- 172.21.15.27:0/3638708406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2400b2390 0x7fe24015a1c0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fe2280ac040 tx=0x7fe22801ad10 comp rx=0 tx=0).stop 2024-01-15T18:10:55.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.500+0000 7fe247000640 1 -- 172.21.15.27:0/3638708406 shutdown_connections 2024-01-15T18:10:55.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.500+0000 7fe247000640 1 --2- 172.21.15.27:0/3638708406 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe218067160 0x7fe218069620 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:55.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.500+0000 7fe247000640 1 --2- 172.21.15.27:0/3638708406 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe2400cfcc0 0x7fe2401584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:55.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.500+0000 7fe247000640 1 --2- 172.21.15.27:0/3638708406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2400b2390 0x7fe24015a1c0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:55.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.500+0000 7fe247000640 1 --2- 172.21.15.27:0/3638708406 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe24006bb80 0x7fe24009e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:55.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.500+0000 7fe247000640 1 -- 172.21.15.27:0/3638708406 >> 172.21.15.27:0/3638708406 conn(0x7fe2400b2a10 msgr2=0x7fe2400b10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:55.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.500+0000 7fe247000640 1 -- 172.21.15.27:0/3638708406 shutdown_connections 2024-01-15T18:10:55.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.500+0000 7fe247000640 1 -- 172.21.15.27:0/3638708406 wait complete. 2024-01-15T18:10:55.521 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:10:55.521 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone91811 2024-01-15T18:10:55.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.664+0000 7fbc1f000640 1 Processor -- start 2024-01-15T18:10:55.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.664+0000 7fbc1f000640 1 -- start start 2024-01-15T18:10:55.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.664+0000 7fbc1f000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc18158900 0x7fbc1815f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:55.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.664+0000 7fbc1f000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fbc180cf2e0 0x7fbc180701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:55.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.664+0000 7fbc1f000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc180b1cb0 con 0x7fbc1815bf00 2024-01-15T18:10:55.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.664+0000 7fbc1f000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc180b1f00 con 0x7fbc180cf2e0 2024-01-15T18:10:55.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.664+0000 7fbc1f000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc180b21e0 con 0x7fbc18158900 2024-01-15T18:10:55.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.664+0000 7fbc1d000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fbc180cf2e0 0x7fbc180701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:55.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.664+0000 7fbc17400640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7fbc1815bf00 0x7fbc180cebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:45454/0 (socket says 172.21.15.27:45454) 2024-01-15T18:10:55.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.664+0000 7fbc17400640 1 -- 172.21.15.27:0/4042206751 learned_addr learned my addr 172.21.15.27:0/4042206751 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:10:55.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.664+0000 7fbc17e00640 1 --2- 172.21.15.27:0/4042206751 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc18158900 0x7fbc1815f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:55.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.664+0000 7fbc1d000640 1 -- 172.21.15.27:0/4042206751 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc18158900 msgr2=0x7fbc1815f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:55.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.664+0000 7fbc1d000640 1 --2- 172.21.15.27:0/4042206751 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc18158900 0x7fbc1815f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:55.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.664+0000 7fbc1d000640 1 -- 172.21.15.27:0/4042206751 >> v1:172.21.15.93:6789/0 conn(0x7fbc1815bf00 legacy=0x7fbc180cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:55.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.664+0000 7fbc1d000640 1 -- 172.21.15.27:0/4042206751 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc1815ea70 con 0x7fbc180cf2e0 2024-01-15T18:10:55.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.668+0000 7fbc1d000640 1 --2- 172.21.15.27:0/4042206751 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc180cf2e0 0x7fbc180701e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbc10009d90 tx=0x7fbc1001c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff4777cb5cf99b50 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:55.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.668+0000 7fbc16a00640 1 -- 172.21.15.27:0/4042206751 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc18070f30 con 0x7fbc180cf2e0 2024-01-15T18:10:55.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.668+0000 7fbc16a00640 1 -- 172.21.15.27:0/4042206751 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc100a5900 con 0x7fbc180cf2e0 2024-01-15T18:10:55.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.668+0000 7fbc16a00640 1 -- 172.21.15.27:0/4042206751 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc100a5c90 con 0x7fbc180cf2e0 2024-01-15T18:10:55.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.668+0000 7fbc1f000640 1 -- 172.21.15.27:0/4042206751 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc180cf2e0 msgr2=0x7fbc180701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:55.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.668+0000 7fbc1f000640 1 --2- 172.21.15.27:0/4042206751 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc180cf2e0 0x7fbc180701e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbc10009d90 tx=0x7fbc1001c950 comp rx=0 tx=0).stop 2024-01-15T18:10:55.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.668+0000 7fbc1f000640 1 -- 172.21.15.27:0/4042206751 shutdown_connections 2024-01-15T18:10:55.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.668+0000 7fbc1f000640 1 --2- 172.21.15.27:0/4042206751 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc180cf2e0 0x7fbc180701e0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:55.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.668+0000 7fbc1f000640 1 --2- 172.21.15.27:0/4042206751 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc18158900 0x7fbc1815f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:55.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.668+0000 7fbc1f000640 1 -- 172.21.15.27:0/4042206751 >> 172.21.15.27:0/4042206751 conn(0x7fbc180cb700 msgr2=0x7fbc180cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:55.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.668+0000 7fbc1f000640 1 -- 172.21.15.27:0/4042206751 shutdown_connections 2024-01-15T18:10:55.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.668+0000 7fbc1f000640 1 -- 172.21.15.27:0/4042206751 wait complete. 2024-01-15T18:10:55.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.672+0000 7fbc1f000640 1 Processor -- start 2024-01-15T18:10:55.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.672+0000 7fbc1f000640 1 -- start start 2024-01-15T18:10:55.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.672+0000 7fbc1f000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc180cf2e0 0x7fbc1814e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:55.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.672+0000 7fbc1f000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc18158900 0x7fbc180ad290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:55.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.672+0000 7fbc1f000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc1815bf00 0x7fbc180cde60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:55.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.672+0000 7fbc1f000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc180ccb70 con 0x7fbc180cf2e0 2024-01-15T18:10:55.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.672+0000 7fbc1f000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc180ccdc0 con 0x7fbc18158900 2024-01-15T18:10:55.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.672+0000 7fbc1f000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc180cd0a0 con 0x7fbc1815bf00 2024-01-15T18:10:55.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.672+0000 7fbc17e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc180cf2e0 0x7fbc1814e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:55.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.672+0000 7fbc17e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc180cf2e0 0x7fbc1814e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34352/0 (socket says 172.21.15.27:34352) 2024-01-15T18:10:55.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.672+0000 7fbc17e00640 1 -- 172.21.15.27:0/3539098672 learned_addr learned my addr 172.21.15.27:0/3539098672 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:10:55.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.672+0000 7fbc17400640 1 --2- 172.21.15.27:0/3539098672 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc18158900 0x7fbc180ad290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:55.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.672+0000 7fbc1d000640 1 --2- 172.21.15.27:0/3539098672 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc1815bf00 0x7fbc180cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:55.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.672+0000 7fbc17e00640 1 -- 172.21.15.27:0/3539098672 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc1815bf00 msgr2=0x7fbc180cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:55.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.672+0000 7fbc17e00640 1 --2- 172.21.15.27:0/3539098672 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc1815bf00 0x7fbc180cde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:55.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.672+0000 7fbc17e00640 1 -- 172.21.15.27:0/3539098672 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc18158900 msgr2=0x7fbc180ad290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:55.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.672+0000 7fbc17e00640 1 --2- 172.21.15.27:0/3539098672 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc18158900 0x7fbc180ad290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:55.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.672+0000 7fbc17e00640 1 -- 172.21.15.27:0/3539098672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc10009a70 con 0x7fbc180cf2e0 2024-01-15T18:10:55.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.672+0000 7fbc17e00640 1 --2- 172.21.15.27:0/3539098672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc180cf2e0 0x7fbc1814e6e0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fbc0800e930 tx=0x7fbc0800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:55.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.672+0000 7fbc0fe00640 1 -- 172.21.15.27:0/3539098672 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc08009690 con 0x7fbc180cf2e0 2024-01-15T18:10:55.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.672+0000 7fbc0fe00640 1 -- 172.21.15.27:0/3539098672 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc0800ca00 con 0x7fbc180cf2e0 2024-01-15T18:10:55.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.672+0000 7fbc0fe00640 1 -- 172.21.15.27:0/3539098672 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0800cd70 con 0x7fbc180cf2e0 2024-01-15T18:10:55.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.672+0000 7fbc1f000640 1 -- 172.21.15.27:0/3539098672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc180cd380 con 0x7fbc180cf2e0 2024-01-15T18:10:55.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.672+0000 7fbc1f000640 1 -- 172.21.15.27:0/3539098672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc180cd940 con 0x7fbc180cf2e0 2024-01-15T18:10:55.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.676+0000 7fbc1f000640 1 -- 172.21.15.27:0/3539098672 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc180b5020 con 0x7fbc180cf2e0 2024-01-15T18:10:55.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.676+0000 7fbc0fe00640 1 -- 172.21.15.27:0/3539098672 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7fbc08013470 con 0x7fbc180cf2e0 2024-01-15T18:10:55.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.676+0000 7fbc0fe00640 1 --2- 172.21.15.27:0/3539098672 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbbf0067160 0x7fbbf0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:55.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.676+0000 7fbc0fe00640 1 -- 172.21.15.27:0/3539098672 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fbc08083180 con 0x7fbc180cf2e0 2024-01-15T18:10:55.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.676+0000 7fbc17400640 1 --2- 172.21.15.27:0/3539098672 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbbf0067160 0x7fbbf0069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:55.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.676+0000 7fbc17400640 1 --2- 172.21.15.27:0/3539098672 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbbf0067160 0x7fbbf0069620 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fbc00005890 tx=0x7fbc00009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:55.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.680+0000 7fbc0fe00640 1 -- 172.21.15.27:0/3539098672 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbc08051d00 con 0x7fbc180cf2e0 2024-01-15T18:10:55.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:10:55.872+0000 7fbc1f000640 1 -- 172.21.15.27:0/3539098672 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone91811", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc180cf990 con 0x7fbbf0067160 2024-01-15T18:10:56.910 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:10:56.914 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:11:00.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.252+0000 7fbc0fe00640 1 -- 172.21.15.27:0/3539098672 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fbc180cf990 con 0x7fbbf0067160 2024-01-15T18:11:00.256 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:11:00.256 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:11:00.256 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "complete" 2024-01-15T18:11:00.256 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:00.256 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:11:00.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.252+0000 7fbc1f000640 1 -- 172.21.15.27:0/3539098672 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbbf0067160 msgr2=0x7fbbf0069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:00.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.252+0000 7fbc1f000640 1 --2- 172.21.15.27:0/3539098672 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbbf0067160 0x7fbbf0069620 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fbc00005890 tx=0x7fbc00009040 comp rx=0 tx=0).stop 2024-01-15T18:11:00.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.252+0000 7fbc1f000640 1 -- 172.21.15.27:0/3539098672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc180cf2e0 msgr2=0x7fbc1814e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:00.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.252+0000 7fbc1f000640 1 --2- 172.21.15.27:0/3539098672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc180cf2e0 0x7fbc1814e6e0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fbc0800e930 tx=0x7fbc0800edf0 comp rx=0 tx=0).stop 2024-01-15T18:11:00.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.252+0000 7fbc1f000640 1 -- 172.21.15.27:0/3539098672 shutdown_connections 2024-01-15T18:11:00.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.252+0000 7fbc1f000640 1 --2- 172.21.15.27:0/3539098672 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbbf0067160 0x7fbbf0069620 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:00.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.252+0000 7fbc1f000640 1 --2- 172.21.15.27:0/3539098672 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc1815bf00 0x7fbc180cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:00.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.252+0000 7fbc1f000640 1 --2- 172.21.15.27:0/3539098672 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc18158900 0x7fbc180ad290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:00.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.252+0000 7fbc1f000640 1 --2- 172.21.15.27:0/3539098672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc180cf2e0 0x7fbc1814e6e0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:00.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.252+0000 7fbc1f000640 1 -- 172.21.15.27:0/3539098672 >> 172.21.15.27:0/3539098672 conn(0x7fbc180cb700 msgr2=0x7fbc18154fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:00.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.252+0000 7fbc1f000640 1 -- 172.21.15.27:0/3539098672 shutdown_connections 2024-01-15T18:11:00.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.252+0000 7fbc1f000640 1 -- 172.21.15.27:0/3539098672 wait complete. 2024-01-15T18:11:00.276 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:11:00.276 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone91812 2024-01-15T18:11:00.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.420+0000 7f913ec00640 1 Processor -- start 2024-01-15T18:11:00.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.420+0000 7f913ec00640 1 -- start start 2024-01-15T18:11:00.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.420+0000 7f913ec00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f913815ad20 0x7f913815b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:00.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.420+0000 7f913ec00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913815a610 con 0x7f91380715f0 2024-01-15T18:11:00.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.420+0000 7f913ec00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913815e3c0 con 0x7f913815d900 2024-01-15T18:11:00.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.420+0000 7f913ec00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913815e6a0 con 0x7f913815ad20 2024-01-15T18:11:00.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.420+0000 7f913cc00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f91380715f0 0x7f913815bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44654/0 (socket says 172.21.15.27:44654) 2024-01-15T18:11:00.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.420+0000 7f913cc00640 1 -- 172.21.15.27:0/128830111 learned_addr learned my addr 172.21.15.27:0/128830111 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:00.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.420+0000 7f9137e00640 1 --2- 172.21.15.27:0/128830111 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f913815ad20 0x7f913815b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:00.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.424+0000 7f9136a00640 1 -- 172.21.15.27:0/128830111 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600235772 0 0) 0x7f913815e3c0 con 0x7f913815d900 2024-01-15T18:11:00.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.424+0000 7f9136a00640 1 -- 172.21.15.27:0/128830111 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9118005740 con 0x7f913815d900 2024-01-15T18:11:00.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.424+0000 7f9136a00640 1 -- 172.21.15.27:0/128830111 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291395389 0 0) 0x7f913815a610 con 0x7f91380715f0 2024-01-15T18:11:00.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.424+0000 7f9136a00640 1 -- 172.21.15.27:0/128830111 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f913815e3c0 con 0x7f91380715f0 2024-01-15T18:11:00.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.424+0000 7f9137e00640 1 -- 172.21.15.27:0/128830111 >> v1:172.21.15.93:6789/0 conn(0x7f913815d900 legacy=0x7f9138070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:00.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.424+0000 7f9137e00640 1 -- 172.21.15.27:0/128830111 >> v1:172.21.15.27:6789/0 conn(0x7f91380715f0 legacy=0x7f913815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:00.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.424+0000 7f9137e00640 1 -- 172.21.15.27:0/128830111 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913815b6e0 con 0x7f913815ad20 2024-01-15T18:11:00.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.424+0000 7f9137e00640 1 --2- 172.21.15.27:0/128830111 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f913815ad20 0x7f913815b100 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f91280097f0 tx=0x7f912801c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6eed52e5823d7e6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:00.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.424+0000 7f9136a00640 1 -- 172.21.15.27:0/128830111 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91280aa610 con 0x7f913815ad20 2024-01-15T18:11:00.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.424+0000 7f9136a00640 1 -- 172.21.15.27:0/128830111 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91280a6040 con 0x7f913815ad20 2024-01-15T18:11:00.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.424+0000 7f9136a00640 1 -- 172.21.15.27:0/128830111 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91280aa990 con 0x7f913815ad20 2024-01-15T18:11:00.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.428+0000 7f913ec00640 1 -- 172.21.15.27:0/128830111 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f913815ad20 msgr2=0x7f913815b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:00.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.428+0000 7f913ec00640 1 --2- 172.21.15.27:0/128830111 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f913815ad20 0x7f913815b100 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f91280097f0 tx=0x7f912801c490 comp rx=0 tx=0).stop 2024-01-15T18:11:00.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.428+0000 7f913ec00640 1 -- 172.21.15.27:0/128830111 shutdown_connections 2024-01-15T18:11:00.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.428+0000 7f913ec00640 1 --2- 172.21.15.27:0/128830111 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f913815ad20 0x7f913815b100 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:00.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.428+0000 7f913ec00640 1 -- 172.21.15.27:0/128830111 >> 172.21.15.27:0/128830111 conn(0x7f91380cb700 msgr2=0x7f91380cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:00.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.428+0000 7f913ec00640 1 -- 172.21.15.27:0/128830111 shutdown_connections 2024-01-15T18:11:00.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.428+0000 7f913ec00640 1 -- 172.21.15.27:0/128830111 wait complete. 2024-01-15T18:11:00.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.428+0000 7f913ec00640 1 Processor -- start 2024-01-15T18:11:00.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.428+0000 7f913ec00640 1 -- start start 2024-01-15T18:11:00.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.428+0000 7f913ec00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f91380715f0 0x7f913814c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:00.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.428+0000 7f913ec00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f913815ad20 0x7f9138154820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:00.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.428+0000 7f913ec00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f913815d900 0x7f913814b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:00.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.428+0000 7f913ec00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91381647d0 con 0x7f913815ad20 2024-01-15T18:11:00.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.428+0000 7f913ec00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913814c1e0 con 0x7f91380715f0 2024-01-15T18:11:00.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.428+0000 7f913ec00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91381644f0 con 0x7f913815d900 2024-01-15T18:11:00.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.428+0000 7f9137400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f913815ad20 0x7f9138154820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:00.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.428+0000 7f9137400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f913815ad20 0x7f9138154820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34364/0 (socket says 172.21.15.27:34364) 2024-01-15T18:11:00.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.428+0000 7f9137400640 1 -- 172.21.15.27:0/3715423727 learned_addr learned my addr 172.21.15.27:0/3715423727 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:00.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.428+0000 7f913cc00640 1 --2- 172.21.15.27:0/3715423727 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f913815d900 0x7f913814b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:00.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.432+0000 7f9137e00640 1 --2- 172.21.15.27:0/3715423727 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f91380715f0 0x7f913814c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:00.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.432+0000 7f9137400640 1 -- 172.21.15.27:0/3715423727 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f913815d900 msgr2=0x7f913814b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:00.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.432+0000 7f9137400640 1 --2- 172.21.15.27:0/3715423727 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f913815d900 0x7f913814b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:00.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.432+0000 7f9137400640 1 -- 172.21.15.27:0/3715423727 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f91380715f0 msgr2=0x7f913814c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:00.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.432+0000 7f9137400640 1 --2- 172.21.15.27:0/3715423727 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f91380715f0 0x7f913814c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:00.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.432+0000 7f9137400640 1 -- 172.21.15.27:0/3715423727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9128009420 con 0x7f913815ad20 2024-01-15T18:11:00.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.432+0000 7f913cc00640 1 --2- 172.21.15.27:0/3715423727 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f913815d900 0x7f913814b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:11:00.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.432+0000 7f9137400640 1 --2- 172.21.15.27:0/3715423727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f913815ad20 0x7f9138154820 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f91200010f0 tx=0x7f912000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:00.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.432+0000 7f912fe00640 1 -- 172.21.15.27:0/3715423727 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91200052d0 con 0x7f913815ad20 2024-01-15T18:11:00.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.432+0000 7f912fe00640 1 -- 172.21.15.27:0/3715423727 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9120005430 con 0x7f913815ad20 2024-01-15T18:11:00.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.432+0000 7f912fe00640 1 -- 172.21.15.27:0/3715423727 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9120003820 con 0x7f913815ad20 2024-01-15T18:11:00.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.432+0000 7f913ec00640 1 -- 172.21.15.27:0/3715423727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91380cd380 con 0x7f913815ad20 2024-01-15T18:11:00.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.432+0000 7f913ec00640 1 -- 172.21.15.27:0/3715423727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91380cd940 con 0x7f913815ad20 2024-01-15T18:11:00.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.432+0000 7f913ec00640 1 -- 172.21.15.27:0/3715423727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91380cd1e0 con 0x7f913815ad20 2024-01-15T18:11:00.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.432+0000 7f912fe00640 1 -- 172.21.15.27:0/3715423727 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7f9120003980 con 0x7f913815ad20 2024-01-15T18:11:00.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.432+0000 7f912fe00640 1 --2- 172.21.15.27:0/3715423727 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9110067160 0x7f9110069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:00.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.432+0000 7f912fe00640 1 -- 172.21.15.27:0/3715423727 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f91200825b0 con 0x7f913815ad20 2024-01-15T18:11:00.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.436+0000 7f9137e00640 1 --2- 172.21.15.27:0/3715423727 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9110067160 0x7f9110069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:00.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.436+0000 7f9137e00640 1 --2- 172.21.15.27:0/3715423727 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9110067160 0x7f9110069620 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f91280ae000 tx=0x7f91280a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:00.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.440+0000 7f912fe00640 1 -- 172.21.15.27:0/3715423727 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f91380cd1e0 con 0x7f913815ad20 2024-01-15T18:11:00.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:00.628+0000 7f913ec00640 1 -- 172.21.15.27:0/3715423727 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone91812", "target": ["mon-mgr", ""]}) v1 -- 0x7f913806c430 con 0x7f9110067160 2024-01-15T18:11:05.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.256+0000 7f912fe00640 1 -- 172.21.15.27:0/3715423727 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f91280aa690 con 0x7f9110067160 2024-01-15T18:11:05.257 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:11:05.258 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:11:05.258 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "complete" 2024-01-15T18:11:05.258 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:05.258 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:11:05.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.256+0000 7f913ec00640 1 -- 172.21.15.27:0/3715423727 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9110067160 msgr2=0x7f9110069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:05.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.256+0000 7f913ec00640 1 --2- 172.21.15.27:0/3715423727 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9110067160 0x7f9110069620 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f91280ae000 tx=0x7f91280a7000 comp rx=0 tx=0).stop 2024-01-15T18:11:05.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.256+0000 7f913ec00640 1 -- 172.21.15.27:0/3715423727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f913815ad20 msgr2=0x7f9138154820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:05.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.256+0000 7f913ec00640 1 --2- 172.21.15.27:0/3715423727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f913815ad20 0x7f9138154820 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f91200010f0 tx=0x7f912000c8b0 comp rx=0 tx=0).stop 2024-01-15T18:11:05.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.256+0000 7f913ec00640 1 -- 172.21.15.27:0/3715423727 shutdown_connections 2024-01-15T18:11:05.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.256+0000 7f913ec00640 1 --2- 172.21.15.27:0/3715423727 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9110067160 0x7f9110069620 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:05.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.256+0000 7f913ec00640 1 --2- 172.21.15.27:0/3715423727 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f913815d900 0x7f913814b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:05.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.256+0000 7f913ec00640 1 --2- 172.21.15.27:0/3715423727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f913815ad20 0x7f9138154820 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:05.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.256+0000 7f913ec00640 1 --2- 172.21.15.27:0/3715423727 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f91380715f0 0x7f913814c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:05.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.256+0000 7f913ec00640 1 -- 172.21.15.27:0/3715423727 >> 172.21.15.27:0/3715423727 conn(0x7f91380cb700 msgr2=0x7f913806de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:05.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.256+0000 7f913ec00640 1 -- 172.21.15.27:0/3715423727 shutdown_connections 2024-01-15T18:11:05.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.260+0000 7f913ec00640 1 -- 172.21.15.27:0/3715423727 wait complete. 2024-01-15T18:11:05.285 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:11:05.285 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone91813 2024-01-15T18:11:05.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.432+0000 7faab9600640 1 Processor -- start 2024-01-15T18:11:05.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.432+0000 7faab9600640 1 -- start start 2024-01-15T18:11:05.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.436+0000 7faab9600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faab41468d0 0x7faab40ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:05.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.436+0000 7faab9600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7faab40cef30 0x7faab40cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:05.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.436+0000 7faab9600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab40ad920 con 0x7faab40ccea0 2024-01-15T18:11:05.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.436+0000 7faab9600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faab40ac880 con 0x7faab40cef30 2024-01-15T18:11:05.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.436+0000 7faab9600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab40acb60 con 0x7faab41468d0 2024-01-15T18:11:05.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.436+0000 7faab2000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7faab40cef30 0x7faab40cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:05.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.436+0000 7faab2a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faab41468d0 0x7faab40ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:05.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.436+0000 7faab2000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7faab40cef30 0x7faab40cc960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:41020/0 (socket says 172.21.15.27:41020) 2024-01-15T18:11:05.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.436+0000 7faab2000640 1 -- 172.21.15.27:0/4194801229 learned_addr learned my addr 172.21.15.27:0/4194801229 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:05.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.436+0000 7faab1600640 1 -- 172.21.15.27:0/4194801229 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186653706 0 0) 0x7faaac0014a0 con 0x7faab40ccea0 2024-01-15T18:11:05.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.436+0000 7faab2a00640 1 -- 172.21.15.27:0/4194801229 >> v2:172.21.15.93:3300/0 conn(0x7faab40cef30 msgr2=0x7faab40cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:05.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.436+0000 7faab2a00640 1 --2- 172.21.15.27:0/4194801229 >> v2:172.21.15.93:3300/0 conn(0x7faab40cef30 0x7faab40cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:05.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.436+0000 7faab2a00640 1 -- 172.21.15.27:0/4194801229 >> v1:172.21.15.93:6789/0 conn(0x7faab40ccea0 legacy=0x7faab4160cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:05.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.436+0000 7faab2a00640 1 -- 172.21.15.27:0/4194801229 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab40ace40 con 0x7faab41468d0 2024-01-15T18:11:05.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.440+0000 7faab2a00640 1 --2- 172.21.15.27:0/4194801229 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faab41468d0 0x7faab40ce9f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7faaa40097f0 tx=0x7faaa401c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a75c6d52acab8a35 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:05.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.440+0000 7faab1600640 1 -- 172.21.15.27:0/4194801229 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa40ab600 con 0x7faab41468d0 2024-01-15T18:11:05.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.440+0000 7faab1600640 1 -- 172.21.15.27:0/4194801229 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa40a7050 con 0x7faab41468d0 2024-01-15T18:11:05.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.440+0000 7faab1600640 1 -- 172.21.15.27:0/4194801229 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa40ab980 con 0x7faab41468d0 2024-01-15T18:11:05.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.440+0000 7faab9600640 1 -- 172.21.15.27:0/4194801229 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faab41468d0 msgr2=0x7faab40ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:05.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.440+0000 7faab9600640 1 --2- 172.21.15.27:0/4194801229 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faab41468d0 0x7faab40ce9f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7faaa40097f0 tx=0x7faaa401c450 comp rx=0 tx=0).stop 2024-01-15T18:11:05.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.440+0000 7faab9600640 1 -- 172.21.15.27:0/4194801229 shutdown_connections 2024-01-15T18:11:05.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.440+0000 7faab9600640 1 --2- 172.21.15.27:0/4194801229 >> v2:172.21.15.93:3300/0 conn(0x7faab40cef30 0x7faab40cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:05.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.440+0000 7faab9600640 1 --2- 172.21.15.27:0/4194801229 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faab41468d0 0x7faab40ce9f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:05.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.440+0000 7faab9600640 1 -- 172.21.15.27:0/4194801229 >> 172.21.15.27:0/4194801229 conn(0x7faab40cb700 msgr2=0x7faab40cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:05.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.440+0000 7faab9600640 1 -- 172.21.15.27:0/4194801229 shutdown_connections 2024-01-15T18:11:05.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.440+0000 7faab9600640 1 -- 172.21.15.27:0/4194801229 wait complete. 2024-01-15T18:11:05.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faab9600640 1 Processor -- start 2024-01-15T18:11:05.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faab9600640 1 -- start start 2024-01-15T18:11:05.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faab9600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faab40ccea0 0x7faab4137960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:05.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faab9600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faab40cef30 0x7faab4135610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:05.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faab9600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab41468d0 0x7faab415fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:05.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faab9600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab4159dd0 con 0x7faab41468d0 2024-01-15T18:11:05.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faab9600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab415ccd0 con 0x7faab40ccea0 2024-01-15T18:11:05.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faab9600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab4159750 con 0x7faab40cef30 2024-01-15T18:11:05.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faab3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab41468d0 0x7faab415fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:05.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faab2a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faab40ccea0 0x7faab4137960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:05.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faab3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab41468d0 0x7faab415fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56388/0 (socket says 172.21.15.27:56388) 2024-01-15T18:11:05.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faab3400640 1 -- 172.21.15.27:0/2999753234 learned_addr learned my addr 172.21.15.27:0/2999753234 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:05.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faab2000640 1 --2- 172.21.15.27:0/2999753234 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faab40cef30 0x7faab4135610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:05.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faab3400640 1 -- 172.21.15.27:0/2999753234 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faab40cef30 msgr2=0x7faab4135610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:05.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faab3400640 1 --2- 172.21.15.27:0/2999753234 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faab40cef30 0x7faab4135610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:05.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faab3400640 1 -- 172.21.15.27:0/2999753234 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faab40ccea0 msgr2=0x7faab4137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:05.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faab3400640 1 --2- 172.21.15.27:0/2999753234 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faab40ccea0 0x7faab4137960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:05.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faab3400640 1 -- 172.21.15.27:0/2999753234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa4009420 con 0x7faab41468d0 2024-01-15T18:11:05.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faab2a00640 1 --2- 172.21.15.27:0/2999753234 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faab40ccea0 0x7faab4137960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:11:05.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faab3400640 1 --2- 172.21.15.27:0/2999753234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab41468d0 0x7faab415fc00 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7faaac00c7c0 tx=0x7faaac00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:05.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faaaaa00640 1 -- 172.21.15.27:0/2999753234 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac003db0 con 0x7faab41468d0 2024-01-15T18:11:05.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faaaaa00640 1 -- 172.21.15.27:0/2999753234 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaac005070 con 0x7faab41468d0 2024-01-15T18:11:05.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faaaaa00640 1 -- 172.21.15.27:0/2999753234 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac0054a0 con 0x7faab41468d0 2024-01-15T18:11:05.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faab9600640 1 -- 172.21.15.27:0/2999753234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab40cf490 con 0x7faab41468d0 2024-01-15T18:11:05.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faab9600640 1 -- 172.21.15.27:0/2999753234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab4137f90 con 0x7faab41468d0 2024-01-15T18:11:05.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.444+0000 7faab9600640 1 -- 172.21.15.27:0/2999753234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab4159f10 con 0x7faab41468d0 2024-01-15T18:11:05.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.448+0000 7faaaaa00640 1 -- 172.21.15.27:0/2999753234 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7faaac005600 con 0x7faab41468d0 2024-01-15T18:11:05.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.448+0000 7faaaaa00640 1 --2- 172.21.15.27:0/2999753234 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faa8c067160 0x7faa8c069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:05.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.448+0000 7faaaaa00640 1 -- 172.21.15.27:0/2999753234 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7faaac082530 con 0x7faab41468d0 2024-01-15T18:11:05.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.448+0000 7faab2a00640 1 --2- 172.21.15.27:0/2999753234 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faa8c067160 0x7faa8c069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:05.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.448+0000 7faab2a00640 1 --2- 172.21.15.27:0/2999753234 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faa8c067160 0x7faa8c069620 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7faaa40ad000 tx=0x7faaa40a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:05.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.452+0000 7faaaaa00640 1 -- 172.21.15.27:0/2999753234 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7faab4159f10 con 0x7faab41468d0 2024-01-15T18:11:05.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:05.640+0000 7faab9600640 1 -- 172.21.15.27:0/2999753234 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone91813", "target": ["mon-mgr", ""]}) v1 -- 0x7faab406d950 con 0x7faa8c067160 2024-01-15T18:11:10.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.252+0000 7faaaaa00640 1 -- 172.21.15.27:0/2999753234 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7faaa40abaf0 con 0x7faa8c067160 2024-01-15T18:11:10.254 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:11:10.254 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:11:10.254 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "complete" 2024-01-15T18:11:10.254 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:10.254 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:11:10.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.252+0000 7faab9600640 1 -- 172.21.15.27:0/2999753234 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faa8c067160 msgr2=0x7faa8c069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:10.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.252+0000 7faab9600640 1 --2- 172.21.15.27:0/2999753234 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faa8c067160 0x7faa8c069620 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7faaa40ad000 tx=0x7faaa40a6000 comp rx=0 tx=0).stop 2024-01-15T18:11:10.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.252+0000 7faab9600640 1 -- 172.21.15.27:0/2999753234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab41468d0 msgr2=0x7faab415fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:10.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.252+0000 7faab9600640 1 --2- 172.21.15.27:0/2999753234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab41468d0 0x7faab415fc00 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7faaac00c7c0 tx=0x7faaac00cc80 comp rx=0 tx=0).stop 2024-01-15T18:11:10.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.252+0000 7faab9600640 1 -- 172.21.15.27:0/2999753234 shutdown_connections 2024-01-15T18:11:10.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.252+0000 7faab9600640 1 --2- 172.21.15.27:0/2999753234 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faa8c067160 0x7faa8c069620 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:10.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.252+0000 7faab9600640 1 --2- 172.21.15.27:0/2999753234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab41468d0 0x7faab415fc00 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:10.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.252+0000 7faab9600640 1 --2- 172.21.15.27:0/2999753234 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faab40cef30 0x7faab4135610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:10.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.252+0000 7faab9600640 1 --2- 172.21.15.27:0/2999753234 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faab40ccea0 0x7faab4137960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:10.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.252+0000 7faab9600640 1 -- 172.21.15.27:0/2999753234 >> 172.21.15.27:0/2999753234 conn(0x7faab40cb700 msgr2=0x7faab40cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:10.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.252+0000 7faab9600640 1 -- 172.21.15.27:0/2999753234 shutdown_connections 2024-01-15T18:11:10.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.256+0000 7faab9600640 1 -- 172.21.15.27:0/2999753234 wait complete. 2024-01-15T18:11:10.281 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:11:10.281 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone91814 2024-01-15T18:11:10.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.428+0000 7f2643a00640 1 Processor -- start 2024-01-15T18:11:10.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f2643a00640 1 -- start start 2024-01-15T18:11:10.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f2643a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f263c0b1db0 0x7f263c0b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:10.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f2643a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f263c0ce050 0x7f263c0ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:10.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f2643a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263c0712b0 con 0x7f263c06c430 2024-01-15T18:11:10.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f2643a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f263c0cea20 con 0x7f263c0b1db0 2024-01-15T18:11:10.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f2643a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f263c0ccfb0 con 0x7f263c0ce050 2024-01-15T18:11:10.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f2641000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f263c06c430 0x7f263c070ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47230/0 (socket says 172.21.15.27:47230) 2024-01-15T18:11:10.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f2641000640 1 -- 172.21.15.27:0/2883477762 learned_addr learned my addr 172.21.15.27:0/2883477762 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:10.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f263b400640 1 -- 172.21.15.27:0/2883477762 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3224951863 0 0) 0x7f263c0712b0 con 0x7f263c06c430 2024-01-15T18:11:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f2641a00640 1 --2- 172.21.15.27:0/2883477762 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f263c0ce050 0x7f263c0ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f263b400640 1 -- 172.21.15.27:0/2883477762 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f261c003530 con 0x7f263c06c430 2024-01-15T18:11:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f263be00640 1 --2- 172.21.15.27:0/2883477762 >> v2:172.21.15.93:3300/0 conn(0x7f263c0b1db0 0x7f263c0b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f263b400640 1 -- 172.21.15.27:0/2883477762 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1108442096 0 0) 0x7f261c003530 con 0x7f263c06c430 2024-01-15T18:11:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f263b400640 1 -- 172.21.15.27:0/2883477762 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f263c0ce050 msgr2=0x7f263c0ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f263b400640 1 --2- 172.21.15.27:0/2883477762 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f263c0ce050 0x7f263c0ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f263b400640 1 -- 172.21.15.27:0/2883477762 >> v2:172.21.15.93:3300/0 conn(0x7f263c0b1db0 msgr2=0x7f263c0b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f263b400640 1 --2- 172.21.15.27:0/2883477762 >> v2:172.21.15.93:3300/0 conn(0x7f263c0b1db0 0x7f263c0b2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f263b400640 1 -- 172.21.15.27:0/2883477762 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263c0cd290 con 0x7f263c06c430 2024-01-15T18:11:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f263b400640 1 -- 172.21.15.27:0/2883477762 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f262c001680 con 0x7f263c06c430 2024-01-15T18:11:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f263b400640 1 -- 172.21.15.27:0/2883477762 >> v1:172.21.15.27:6789/0 conn(0x7f263c06c430 legacy=0x7f263c070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f263b400640 1 --2- 172.21.15.27:0/2883477762 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f261c0039f0 0x7f261c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f263b400640 1 --2- 172.21.15.27:0/2883477762 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f263c0ce050 0x7f261c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:10.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f263b400640 1 --2- 172.21.15.27:0/2883477762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f261c004d10 0x7f261c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f263b400640 1 -- 172.21.15.27:0/2883477762 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263c0ccfb0 con 0x7f261c004d10 2024-01-15T18:11:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f263b400640 1 -- 172.21.15.27:0/2883477762 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f261c00b9c0 con 0x7f261c0039f0 2024-01-15T18:11:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f263b400640 1 -- 172.21.15.27:0/2883477762 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f261c00bb80 con 0x7f263c0ce050 2024-01-15T18:11:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f263be00640 1 --2- 172.21.15.27:0/2883477762 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f263c0ce050 0x7f261c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f2641a00640 1 --2- 172.21.15.27:0/2883477762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f261c004d10 0x7f261c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f2641000640 1 --2- 172.21.15.27:0/2883477762 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f261c0039f0 0x7f261c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f2641a00640 1 -- 172.21.15.27:0/2883477762 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f263c0ce050 msgr2=0x7f261c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f2641a00640 1 --2- 172.21.15.27:0/2883477762 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f263c0ce050 0x7f261c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f2641a00640 1 -- 172.21.15.27:0/2883477762 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f261c0039f0 msgr2=0x7f261c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f2641a00640 1 --2- 172.21.15.27:0/2883477762 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f261c0039f0 0x7f261c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:10.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.432+0000 7f2641a00640 1 -- 172.21.15.27:0/2883477762 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261c003530 con 0x7f261c004d10 2024-01-15T18:11:10.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.436+0000 7f2641a00640 1 --2- 172.21.15.27:0/2883477762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f261c004d10 0x7f261c005190 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f2634009f90 tx=0x7f263401cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=356f25d7527ef0e7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:10.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.436+0000 7f263b400640 1 -- 172.21.15.27:0/2883477762 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26340a5c30 con 0x7f261c004d10 2024-01-15T18:11:10.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.436+0000 7f263b400640 1 -- 172.21.15.27:0/2883477762 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263400f040 con 0x7f261c004d10 2024-01-15T18:11:10.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.436+0000 7f263b400640 1 -- 172.21.15.27:0/2883477762 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26340ad440 con 0x7f261c004d10 2024-01-15T18:11:10.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.436+0000 7f2643a00640 1 -- 172.21.15.27:0/2883477762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f261c004d10 msgr2=0x7f261c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:10.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.436+0000 7f2643a00640 1 --2- 172.21.15.27:0/2883477762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f261c004d10 0x7f261c005190 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f2634009f90 tx=0x7f263401cc20 comp rx=0 tx=0).stop 2024-01-15T18:11:10.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.436+0000 7f2641a00640 1 -- 172.21.15.27:0/2883477762 reap_dead start 2024-01-15T18:11:10.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.436+0000 7f2643a00640 1 -- 172.21.15.27:0/2883477762 shutdown_connections 2024-01-15T18:11:10.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.436+0000 7f2643a00640 1 -- 172.21.15.27:0/2883477762 >> 172.21.15.27:0/2883477762 conn(0x7f263c0cb700 msgr2=0x7f263c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:10.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.436+0000 7f2643a00640 1 -- 172.21.15.27:0/2883477762 shutdown_connections 2024-01-15T18:11:10.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.436+0000 7f2643a00640 1 -- 172.21.15.27:0/2883477762 wait complete. 2024-01-15T18:11:10.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.440+0000 7f2643a00640 1 Processor -- start 2024-01-15T18:11:10.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.440+0000 7f2643a00640 1 -- start start 2024-01-15T18:11:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.440+0000 7f2643a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f261c004d10 0x7f263c13e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.440+0000 7f2643a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f263c159cd0 0x7f263c1edfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.440+0000 7f2643a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f263c1e1e00 0x7f263c13d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.440+0000 7f2643a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263c13c0b0 con 0x7f261c004d10 2024-01-15T18:11:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.440+0000 7f2643a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263c15a150 con 0x7f263c159cd0 2024-01-15T18:11:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.440+0000 7f2643a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f263c13b4e0 con 0x7f263c1e1e00 2024-01-15T18:11:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.440+0000 7f2641000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f261c004d10 0x7f263c13e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.440+0000 7f2641000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f261c004d10 0x7f263c13e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56418/0 (socket says 172.21.15.27:56418) 2024-01-15T18:11:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.440+0000 7f2641000640 1 -- 172.21.15.27:0/2810064355 learned_addr learned my addr 172.21.15.27:0/2810064355 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.440+0000 7f263be00640 1 --2- 172.21.15.27:0/2810064355 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f263c159cd0 0x7f263c1edfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:10.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.440+0000 7f2641a00640 1 --2- 172.21.15.27:0/2810064355 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f263c1e1e00 0x7f263c13d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:10.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.440+0000 7f2641000640 1 -- 172.21.15.27:0/2810064355 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f263c1e1e00 msgr2=0x7f263c13d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:10.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.440+0000 7f2641000640 1 --2- 172.21.15.27:0/2810064355 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f263c1e1e00 0x7f263c13d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:10.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.440+0000 7f2641000640 1 -- 172.21.15.27:0/2810064355 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f263c159cd0 msgr2=0x7f263c1edfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:10.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.440+0000 7f2641000640 1 --2- 172.21.15.27:0/2810064355 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f263c159cd0 0x7f263c1edfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:10.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.440+0000 7f2641000640 1 -- 172.21.15.27:0/2810064355 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263c13b780 con 0x7f261c004d10 2024-01-15T18:11:10.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.440+0000 7f2641a00640 1 --2- 172.21.15.27:0/2810064355 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f263c1e1e00 0x7f263c13d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:11:10.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.440+0000 7f2641000640 1 --2- 172.21.15.27:0/2810064355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f261c004d10 0x7f263c13e000 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f262c012490 tx=0x7f262c012890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:10.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.440+0000 7f2638c00640 1 -- 172.21.15.27:0/2810064355 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c014750 con 0x7f261c004d10 2024-01-15T18:11:10.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.440+0000 7f2638c00640 1 -- 172.21.15.27:0/2810064355 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262c0148d0 con 0x7f261c004d10 2024-01-15T18:11:10.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.440+0000 7f2638c00640 1 -- 172.21.15.27:0/2810064355 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c014c80 con 0x7f261c004d10 2024-01-15T18:11:10.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.440+0000 7f2643a00640 1 -- 172.21.15.27:0/2810064355 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f263c13ba60 con 0x7f261c004d10 2024-01-15T18:11:10.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.440+0000 7f2643a00640 1 -- 172.21.15.27:0/2810064355 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f263c1ee4f0 con 0x7f261c004d10 2024-01-15T18:11:10.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.444+0000 7f2643a00640 1 -- 172.21.15.27:0/2810064355 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f263c04e850 con 0x7f261c004d10 2024-01-15T18:11:10.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.444+0000 7f2638c00640 1 -- 172.21.15.27:0/2810064355 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7f262c0153e0 con 0x7f261c004d10 2024-01-15T18:11:10.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.444+0000 7f2638c00640 1 --2- 172.21.15.27:0/2810064355 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2614067160 0x7f2614069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:10.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.444+0000 7f2638c00640 1 -- 172.21.15.27:0/2810064355 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f262c085730 con 0x7f261c004d10 2024-01-15T18:11:10.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.444+0000 7f263be00640 1 --2- 172.21.15.27:0/2810064355 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2614067160 0x7f2614069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:10.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.444+0000 7f263be00640 1 --2- 172.21.15.27:0/2810064355 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2614067160 0x7f2614069620 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f263c0706d0 tx=0x7f2624013040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:10.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.448+0000 7f2638c00640 1 -- 172.21.15.27:0/2810064355 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f262c0542b0 con 0x7f261c004d10 2024-01-15T18:11:10.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:10.636+0000 7f2643a00640 1 -- 172.21.15.27:0/2810064355 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone91814", "target": ["mon-mgr", ""]}) v1 -- 0x7f263c0cdb30 con 0x7f2614067160 2024-01-15T18:11:15.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.252+0000 7f2638c00640 1 -- 172.21.15.27:0/2810064355 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f2624017040 con 0x7f2614067160 2024-01-15T18:11:15.255 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:11:15.255 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:11:15.255 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "complete" 2024-01-15T18:11:15.255 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:15.255 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:11:15.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.256+0000 7f2643a00640 1 -- 172.21.15.27:0/2810064355 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2614067160 msgr2=0x7f2614069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:15.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.256+0000 7f2643a00640 1 --2- 172.21.15.27:0/2810064355 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2614067160 0x7f2614069620 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f263c0706d0 tx=0x7f2624013040 comp rx=0 tx=0).stop 2024-01-15T18:11:15.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.256+0000 7f2643a00640 1 -- 172.21.15.27:0/2810064355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f261c004d10 msgr2=0x7f263c13e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:15.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.256+0000 7f2643a00640 1 --2- 172.21.15.27:0/2810064355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f261c004d10 0x7f263c13e000 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f262c012490 tx=0x7f262c012890 comp rx=0 tx=0).stop 2024-01-15T18:11:15.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.256+0000 7f2643a00640 1 -- 172.21.15.27:0/2810064355 shutdown_connections 2024-01-15T18:11:15.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.256+0000 7f2643a00640 1 --2- 172.21.15.27:0/2810064355 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2614067160 0x7f2614069620 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:15.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.256+0000 7f2643a00640 1 --2- 172.21.15.27:0/2810064355 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f263c1e1e00 0x7f263c13d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:15.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.256+0000 7f2643a00640 1 --2- 172.21.15.27:0/2810064355 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f263c159cd0 0x7f263c1edfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:15.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.256+0000 7f2643a00640 1 --2- 172.21.15.27:0/2810064355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f261c004d10 0x7f263c13e000 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:15.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.256+0000 7f2643a00640 1 -- 172.21.15.27:0/2810064355 >> 172.21.15.27:0/2810064355 conn(0x7f263c0cb700 msgr2=0x7f263c1549e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:15.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.256+0000 7f2643a00640 1 -- 172.21.15.27:0/2810064355 shutdown_connections 2024-01-15T18:11:15.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.256+0000 7f2643a00640 1 -- 172.21.15.27:0/2810064355 wait complete. 2024-01-15T18:11:15.282 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:11:15.282 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone91815 2024-01-15T18:11:15.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.428+0000 7f2454200640 1 Processor -- start 2024-01-15T18:11:15.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.432+0000 7f2454200640 1 -- start start 2024-01-15T18:11:15.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.432+0000 7f2454200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f244c158900 0x7f244c15f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:15.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.432+0000 7f2454200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f244c15bf00 0x7f244c0cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:15.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.432+0000 7f2454200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f244c0b1cb0 con 0x7f244c0cf110 2024-01-15T18:11:15.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.432+0000 7f2454200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f244c0cf540 con 0x7f244c158900 2024-01-15T18:11:15.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.432+0000 7f2454200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f244c0b1f00 con 0x7f244c15bf00 2024-01-15T18:11:15.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.432+0000 7f2451800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f244c158900 0x7f244c15f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:15.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.432+0000 7f2452200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f244c0cf110 0x7f244c0701e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:33854/0 (socket says 172.21.15.27:33854) 2024-01-15T18:11:15.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.432+0000 7f2452200640 1 -- 172.21.15.27:0/2727566055 learned_addr learned my addr 172.21.15.27:0/2727566055 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:15.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.432+0000 7f2450e00640 1 --2- 172.21.15.27:0/2727566055 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f244c15bf00 0x7f244c0cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:15.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.432+0000 7f2451800640 1 -- 172.21.15.27:0/2727566055 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f244c15bf00 msgr2=0x7f244c0cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:15.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.432+0000 7f2451800640 1 --2- 172.21.15.27:0/2727566055 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f244c15bf00 0x7f244c0cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:15.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.432+0000 7f2451800640 1 -- 172.21.15.27:0/2727566055 >> v1:172.21.15.27:6789/0 conn(0x7f244c0cf110 legacy=0x7f244c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:15.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.432+0000 7f2451800640 1 -- 172.21.15.27:0/2727566055 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f244c0b21e0 con 0x7f244c158900 2024-01-15T18:11:15.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.436+0000 7f2451800640 1 --2- 172.21.15.27:0/2727566055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f244c158900 0x7f244c15f550 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f243c0097c0 tx=0x7f243c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=706d30b45f7e2d3e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:15.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.436+0000 7f244be00640 1 -- 172.21.15.27:0/2727566055 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f243c0ab600 con 0x7f244c158900 2024-01-15T18:11:15.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.436+0000 7f244be00640 1 -- 172.21.15.27:0/2727566055 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f243c0a7050 con 0x7f244c158900 2024-01-15T18:11:15.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.436+0000 7f244be00640 1 -- 172.21.15.27:0/2727566055 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f243c0ab980 con 0x7f244c158900 2024-01-15T18:11:15.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.436+0000 7f2454200640 1 -- 172.21.15.27:0/2727566055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f244c158900 msgr2=0x7f244c15f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:15.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.436+0000 7f2454200640 1 --2- 172.21.15.27:0/2727566055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f244c158900 0x7f244c15f550 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f243c0097c0 tx=0x7f243c01c480 comp rx=0 tx=0).stop 2024-01-15T18:11:15.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.436+0000 7f2454200640 1 -- 172.21.15.27:0/2727566055 shutdown_connections 2024-01-15T18:11:15.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.436+0000 7f2454200640 1 --2- 172.21.15.27:0/2727566055 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f244c15bf00 0x7f244c0cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:15.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.436+0000 7f2454200640 1 --2- 172.21.15.27:0/2727566055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f244c158900 0x7f244c15f550 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:15.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.436+0000 7f2454200640 1 -- 172.21.15.27:0/2727566055 >> 172.21.15.27:0/2727566055 conn(0x7f244c0cb700 msgr2=0x7f244c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:15.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.436+0000 7f2454200640 1 -- 172.21.15.27:0/2727566055 shutdown_connections 2024-01-15T18:11:15.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.436+0000 7f2454200640 1 -- 172.21.15.27:0/2727566055 wait complete. 2024-01-15T18:11:15.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.436+0000 7f2454200640 1 Processor -- start 2024-01-15T18:11:15.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.440+0000 7f2454200640 1 -- start start 2024-01-15T18:11:15.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.440+0000 7f2454200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f244c0cf110 0x7f244c151d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:15.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.440+0000 7f2454200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f244c158900 0x7f244c0cc810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:15.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.440+0000 7f2454200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f244c15bf00 0x7f244c0ccd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:15.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.440+0000 7f2454200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244c1536e0 con 0x7f244c158900 2024-01-15T18:11:15.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.440+0000 7f2454200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244c152260 con 0x7f244c15bf00 2024-01-15T18:11:15.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.440+0000 7f2454200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f244c153c70 con 0x7f244c0cf110 2024-01-15T18:11:15.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.440+0000 7f2450e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f244c158900 0x7f244c0cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:15.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.440+0000 7f2450e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f244c158900 0x7f244c0cc810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43266/0 (socket says 172.21.15.27:43266) 2024-01-15T18:11:15.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.440+0000 7f2450e00640 1 -- 172.21.15.27:0/1967147282 learned_addr learned my addr 172.21.15.27:0/1967147282 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:15.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.440+0000 7f2451800640 1 --2- 172.21.15.27:0/1967147282 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f244c0cf110 0x7f244c151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:15.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.440+0000 7f2452200640 1 --2- 172.21.15.27:0/1967147282 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f244c15bf00 0x7f244c0ccd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:15.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.440+0000 7f2450e00640 1 -- 172.21.15.27:0/1967147282 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f244c0cf110 msgr2=0x7f244c151d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:15.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.440+0000 7f2450e00640 1 --2- 172.21.15.27:0/1967147282 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f244c0cf110 0x7f244c151d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:15.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.440+0000 7f2450e00640 1 -- 172.21.15.27:0/1967147282 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f244c15bf00 msgr2=0x7f244c0ccd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:15.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.440+0000 7f2450e00640 1 --2- 172.21.15.27:0/1967147282 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f244c15bf00 0x7f244c0ccd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:15.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.440+0000 7f2450e00640 1 -- 172.21.15.27:0/1967147282 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f243c009420 con 0x7f244c158900 2024-01-15T18:11:15.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.440+0000 7f2450e00640 1 --2- 172.21.15.27:0/1967147282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f244c158900 0x7f244c0cc810 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f243400d620 tx=0x7f243400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:15.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.440+0000 7f2449600640 1 -- 172.21.15.27:0/1967147282 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f243400b620 con 0x7f244c158900 2024-01-15T18:11:15.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.440+0000 7f2449600640 1 -- 172.21.15.27:0/1967147282 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f243400b780 con 0x7f244c158900 2024-01-15T18:11:15.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.440+0000 7f2454200640 1 -- 172.21.15.27:0/1967147282 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f244c15c490 con 0x7f244c158900 2024-01-15T18:11:15.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.440+0000 7f2449600640 1 -- 172.21.15.27:0/1967147282 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f243400bb10 con 0x7f244c158900 2024-01-15T18:11:15.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.440+0000 7f2454200640 1 -- 172.21.15.27:0/1967147282 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f244c153300 con 0x7f244c158900 2024-01-15T18:11:15.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.440+0000 7f2454200640 1 -- 172.21.15.27:0/1967147282 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f244c153820 con 0x7f244c158900 2024-01-15T18:11:15.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.444+0000 7f2449600640 1 -- 172.21.15.27:0/1967147282 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7f243400bc70 con 0x7f244c158900 2024-01-15T18:11:15.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.444+0000 7f2449600640 1 --2- 172.21.15.27:0/1967147282 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2424067190 0x7f2424069650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:15.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.444+0000 7f2449600640 1 -- 172.21.15.27:0/1967147282 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f2434083330 con 0x7f244c158900 2024-01-15T18:11:15.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.444+0000 7f2451800640 1 --2- 172.21.15.27:0/1967147282 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2424067190 0x7f2424069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:15.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.444+0000 7f2451800640 1 --2- 172.21.15.27:0/1967147282 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2424067190 0x7f2424069650 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f243c0ad000 tx=0x7f243c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:15.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.448+0000 7f2449600640 1 -- 172.21.15.27:0/1967147282 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2434051df0 con 0x7f244c158900 2024-01-15T18:11:15.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:15.636+0000 7f2454200640 1 -- 172.21.15.27:0/1967147282 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone91815", "target": ["mon-mgr", ""]}) v1 -- 0x7f244c15c310 con 0x7f2424067190 2024-01-15T18:11:20.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.252+0000 7f2449600640 1 -- 172.21.15.27:0/1967147282 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f243c0bb0c0 con 0x7f2424067190 2024-01-15T18:11:20.254 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:11:20.255 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:11:20.255 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "complete" 2024-01-15T18:11:20.255 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:20.255 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:11:20.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.256+0000 7f2454200640 1 -- 172.21.15.27:0/1967147282 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2424067190 msgr2=0x7f2424069650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:20.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.256+0000 7f2454200640 1 --2- 172.21.15.27:0/1967147282 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2424067190 0x7f2424069650 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f243c0ad000 tx=0x7f243c008040 comp rx=0 tx=0).stop 2024-01-15T18:11:20.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.256+0000 7f2454200640 1 -- 172.21.15.27:0/1967147282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f244c158900 msgr2=0x7f244c0cc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:20.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.256+0000 7f2454200640 1 --2- 172.21.15.27:0/1967147282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f244c158900 0x7f244c0cc810 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f243400d620 tx=0x7f243400dae0 comp rx=0 tx=0).stop 2024-01-15T18:11:20.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.256+0000 7f2454200640 1 -- 172.21.15.27:0/1967147282 shutdown_connections 2024-01-15T18:11:20.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.256+0000 7f2454200640 1 --2- 172.21.15.27:0/1967147282 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2424067190 0x7f2424069650 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:20.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.256+0000 7f2454200640 1 --2- 172.21.15.27:0/1967147282 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f244c15bf00 0x7f244c0ccd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:20.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.256+0000 7f2454200640 1 --2- 172.21.15.27:0/1967147282 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f244c158900 0x7f244c0cc810 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:20.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.256+0000 7f2454200640 1 --2- 172.21.15.27:0/1967147282 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f244c0cf110 0x7f244c151d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:20.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.256+0000 7f2454200640 1 -- 172.21.15.27:0/1967147282 >> 172.21.15.27:0/1967147282 conn(0x7f244c0cb700 msgr2=0x7f244c0b0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:20.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.256+0000 7f2454200640 1 -- 172.21.15.27:0/1967147282 shutdown_connections 2024-01-15T18:11:20.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.256+0000 7f2454200640 1 -- 172.21.15.27:0/1967147282 wait complete. 2024-01-15T18:11:20.282 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:11:20.282 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone91816 2024-01-15T18:11:20.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.432+0000 7ff130200640 1 Processor -- start 2024-01-15T18:11:20.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.432+0000 7ff130200640 1 -- start start 2024-01-15T18:11:20.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.432+0000 7ff130200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff1280ce760 0x7ff1280b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:20.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.432+0000 7ff130200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1280712b0 con 0x7ff12806c430 2024-01-15T18:11:20.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.432+0000 7ff130200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1280ccfb0 con 0x7ff1280b1db0 2024-01-15T18:11:20.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.432+0000 7ff130200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1280cd290 con 0x7ff1280ce760 2024-01-15T18:11:20.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.432+0000 7ff12d800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff12806c430 0x7ff128070ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:33862/0 (socket says 172.21.15.27:33862) 2024-01-15T18:11:20.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.432+0000 7ff12d800640 1 -- 172.21.15.27:0/1841505266 learned_addr learned my addr 172.21.15.27:0/1841505266 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:20.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.432+0000 7ff12e200640 1 --2- 172.21.15.27:0/1841505266 >> v2:172.21.15.93:3300/0 conn(0x7ff1280ce760 0x7ff1280b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:20.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.432+0000 7ff127e00640 1 -- 172.21.15.27:0/1841505266 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713140325 0 0) 0x7ff1280712b0 con 0x7ff12806c430 2024-01-15T18:11:20.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.432+0000 7ff127e00640 1 -- 172.21.15.27:0/1841505266 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff108003530 con 0x7ff12806c430 2024-01-15T18:11:20.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.432+0000 7ff127e00640 1 -- 172.21.15.27:0/1841505266 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 166408568 0 0) 0x7ff108003530 con 0x7ff12806c430 2024-01-15T18:11:20.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.432+0000 7ff127e00640 1 -- 172.21.15.27:0/1841505266 >> v2:172.21.15.93:3300/0 conn(0x7ff1280ce760 msgr2=0x7ff1280b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:20.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.432+0000 7ff127e00640 1 --2- 172.21.15.27:0/1841505266 >> v2:172.21.15.93:3300/0 conn(0x7ff1280ce760 0x7ff1280b2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:20.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.432+0000 7ff127e00640 1 -- 172.21.15.27:0/1841505266 >> v1:172.21.15.93:6789/0 conn(0x7ff1280b1db0 legacy=0x7ff1280ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:20.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.432+0000 7ff127e00640 1 -- 172.21.15.27:0/1841505266 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1280cd570 con 0x7ff12806c430 2024-01-15T18:11:20.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.432+0000 7ff127e00640 1 -- 172.21.15.27:0/1841505266 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7ff118001680 con 0x7ff12806c430 2024-01-15T18:11:20.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.432+0000 7ff127e00640 1 -- 172.21.15.27:0/1841505266 >> v1:172.21.15.27:6789/0 conn(0x7ff12806c430 legacy=0x7ff128070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:20.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.432+0000 7ff127e00640 1 --2- 172.21.15.27:0/1841505266 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff1080039f0 0x7ff108003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:20.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.432+0000 7ff127e00640 1 --2- 172.21.15.27:0/1841505266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1080047d0 0x7ff108004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:20.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.432+0000 7ff127e00640 1 --2- 172.21.15.27:0/1841505266 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff108005200 0x7ff10800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:20.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.432+0000 7ff127e00640 1 -- 172.21.15.27:0/1841505266 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1280cd290 con 0x7ff1080047d0 2024-01-15T18:11:20.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.432+0000 7ff127e00640 1 -- 172.21.15.27:0/1841505266 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10800c160 con 0x7ff1080039f0 2024-01-15T18:11:20.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.432+0000 7ff127e00640 1 -- 172.21.15.27:0/1841505266 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff10800c320 con 0x7ff108005200 2024-01-15T18:11:20.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.436+0000 7ff12ce00640 1 --2- 172.21.15.27:0/1841505266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1080047d0 0x7ff108004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:20.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.436+0000 7ff12d800640 1 --2- 172.21.15.27:0/1841505266 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff1080039f0 0x7ff108003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:20.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.436+0000 7ff12e200640 1 --2- 172.21.15.27:0/1841505266 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff108005200 0x7ff10800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:20.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.436+0000 7ff12ce00640 1 -- 172.21.15.27:0/1841505266 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff108005200 msgr2=0x7ff10800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:20.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.436+0000 7ff12ce00640 1 --2- 172.21.15.27:0/1841505266 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff108005200 0x7ff10800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:20.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.436+0000 7ff12ce00640 1 -- 172.21.15.27:0/1841505266 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff1080039f0 msgr2=0x7ff108003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:20.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.436+0000 7ff12ce00640 1 --2- 172.21.15.27:0/1841505266 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff1080039f0 0x7ff108003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:20.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.436+0000 7ff12ce00640 1 -- 172.21.15.27:0/1841505266 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff108003530 con 0x7ff1080047d0 2024-01-15T18:11:20.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.436+0000 7ff12e200640 1 -- 172.21.15.27:0/1841505266 reap_dead start 2024-01-15T18:11:20.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.436+0000 7ff12ce00640 1 --2- 172.21.15.27:0/1841505266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1080047d0 0x7ff108004c50 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7ff110001780 tx=0x7ff11001c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d5b1ea6fd08add0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:20.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.436+0000 7ff127e00640 1 -- 172.21.15.27:0/1841505266 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1100052d0 con 0x7ff1080047d0 2024-01-15T18:11:20.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.436+0000 7ff127e00640 1 -- 172.21.15.27:0/1841505266 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1100a7050 con 0x7ff1080047d0 2024-01-15T18:11:20.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.436+0000 7ff127e00640 1 -- 172.21.15.27:0/1841505266 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff110003600 con 0x7ff1080047d0 2024-01-15T18:11:20.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.440+0000 7ff130200640 1 -- 172.21.15.27:0/1841505266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1080047d0 msgr2=0x7ff108004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:20.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.440+0000 7ff130200640 1 --2- 172.21.15.27:0/1841505266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1080047d0 0x7ff108004c50 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7ff110001780 tx=0x7ff11001c3b0 comp rx=0 tx=0).stop 2024-01-15T18:11:20.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.440+0000 7ff130200640 1 -- 172.21.15.27:0/1841505266 shutdown_connections 2024-01-15T18:11:20.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.440+0000 7ff130200640 1 --2- 172.21.15.27:0/1841505266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1080047d0 0x7ff108004c50 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:20.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.440+0000 7ff130200640 1 -- 172.21.15.27:0/1841505266 >> 172.21.15.27:0/1841505266 conn(0x7ff1280cb700 msgr2=0x7ff1280cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:20.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.440+0000 7ff130200640 1 -- 172.21.15.27:0/1841505266 shutdown_connections 2024-01-15T18:11:20.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.440+0000 7ff130200640 1 -- 172.21.15.27:0/1841505266 wait complete. 2024-01-15T18:11:20.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.440+0000 7ff130200640 1 Processor -- start 2024-01-15T18:11:20.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.440+0000 7ff130200640 1 -- start start 2024-01-15T18:11:20.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.440+0000 7ff130200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff1080047d0 0x7ff12806c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:20.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.440+0000 7ff130200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1281e9990 0x7ff1281e76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:20.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.440+0000 7ff130200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1281e40e0 0x7ff1281e8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:20.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.440+0000 7ff130200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1281e6c40 con 0x7ff1281e40e0 2024-01-15T18:11:20.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.440+0000 7ff130200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12806cea0 con 0x7ff1080047d0 2024-01-15T18:11:20.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.440+0000 7ff130200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff12806d180 con 0x7ff1281e9990 2024-01-15T18:11:20.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.440+0000 7ff12e200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1281e40e0 0x7ff1281e8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:20.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.440+0000 7ff12e200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1281e40e0 0x7ff1281e8400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43282/0 (socket says 172.21.15.27:43282) 2024-01-15T18:11:20.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.440+0000 7ff12e200640 1 -- 172.21.15.27:0/3329765992 learned_addr learned my addr 172.21.15.27:0/3329765992 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:20.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.440+0000 7ff12ce00640 1 --2- 172.21.15.27:0/3329765992 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1281e9990 0x7ff1281e76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:20.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.440+0000 7ff12d800640 1 --2- 172.21.15.27:0/3329765992 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff1080047d0 0x7ff12806c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:20.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.444+0000 7ff12e200640 1 -- 172.21.15.27:0/3329765992 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1281e9990 msgr2=0x7ff1281e76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:20.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.444+0000 7ff12e200640 1 --2- 172.21.15.27:0/3329765992 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1281e9990 0x7ff1281e76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:20.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.444+0000 7ff12e200640 1 -- 172.21.15.27:0/3329765992 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff1080047d0 msgr2=0x7ff12806c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:20.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.444+0000 7ff12e200640 1 --2- 172.21.15.27:0/3329765992 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff1080047d0 0x7ff12806c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:20.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.444+0000 7ff12e200640 1 -- 172.21.15.27:0/3329765992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1281e6ee0 con 0x7ff1281e40e0 2024-01-15T18:11:20.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.444+0000 7ff12e200640 1 --2- 172.21.15.27:0/3329765992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1281e40e0 0x7ff1281e8400 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7ff1280cc6a0 tx=0x7ff1200075c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:20.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.444+0000 7ff125600640 1 -- 172.21.15.27:0/3329765992 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff120003ba0 con 0x7ff1281e40e0 2024-01-15T18:11:20.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.444+0000 7ff125600640 1 -- 172.21.15.27:0/3329765992 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff120010040 con 0x7ff1281e40e0 2024-01-15T18:11:20.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.444+0000 7ff125600640 1 -- 172.21.15.27:0/3329765992 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff120007e30 con 0x7ff1281e40e0 2024-01-15T18:11:20.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.444+0000 7ff130200640 1 -- 172.21.15.27:0/3329765992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1281e71c0 con 0x7ff1281e40e0 2024-01-15T18:11:20.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.444+0000 7ff130200640 1 -- 172.21.15.27:0/3329765992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1281e7c20 con 0x7ff1281e40e0 2024-01-15T18:11:20.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.444+0000 7ff130200640 1 -- 172.21.15.27:0/3329765992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1281e9fe0 con 0x7ff1281e40e0 2024-01-15T18:11:20.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.444+0000 7ff125600640 1 -- 172.21.15.27:0/3329765992 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7ff120004030 con 0x7ff1281e40e0 2024-01-15T18:11:20.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.444+0000 7ff125600640 1 --2- 172.21.15.27:0/3329765992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff100067190 0x7ff100069650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:20.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.444+0000 7ff125600640 1 -- 172.21.15.27:0/3329765992 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7ff1200855d0 con 0x7ff1281e40e0 2024-01-15T18:11:20.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.448+0000 7ff12d800640 1 --2- 172.21.15.27:0/3329765992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff100067190 0x7ff100069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:20.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.448+0000 7ff12d800640 1 --2- 172.21.15.27:0/3329765992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff100067190 0x7ff100069650 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ff118001060 tx=0x7ff118001c00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:20.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.448+0000 7ff125600640 1 -- 172.21.15.27:0/3329765992 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff120051b60 con 0x7ff1281e40e0 2024-01-15T18:11:20.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:20.636+0000 7ff130200640 1 -- 172.21.15.27:0/3329765992 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone91816", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1280b25a0 con 0x7ff100067190 2024-01-15T18:11:25.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.252+0000 7ff125600640 1 -- 172.21.15.27:0/3329765992 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7ff118001560 con 0x7ff100067190 2024-01-15T18:11:25.254 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:11:25.254 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:11:25.254 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "complete" 2024-01-15T18:11:25.254 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:25.254 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:11:25.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.252+0000 7ff130200640 1 -- 172.21.15.27:0/3329765992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff100067190 msgr2=0x7ff100069650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:25.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.252+0000 7ff130200640 1 --2- 172.21.15.27:0/3329765992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff100067190 0x7ff100069650 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ff118001060 tx=0x7ff118001c00 comp rx=0 tx=0).stop 2024-01-15T18:11:25.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.252+0000 7ff130200640 1 -- 172.21.15.27:0/3329765992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1281e40e0 msgr2=0x7ff1281e8400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:25.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.252+0000 7ff130200640 1 --2- 172.21.15.27:0/3329765992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1281e40e0 0x7ff1281e8400 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7ff1280cc6a0 tx=0x7ff1200075c0 comp rx=0 tx=0).stop 2024-01-15T18:11:25.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.256+0000 7ff130200640 1 -- 172.21.15.27:0/3329765992 shutdown_connections 2024-01-15T18:11:25.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.256+0000 7ff130200640 1 --2- 172.21.15.27:0/3329765992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff100067190 0x7ff100069650 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:25.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.256+0000 7ff130200640 1 --2- 172.21.15.27:0/3329765992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1281e40e0 0x7ff1281e8400 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:25.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.256+0000 7ff130200640 1 --2- 172.21.15.27:0/3329765992 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff1281e9990 0x7ff1281e76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:25.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.256+0000 7ff130200640 1 --2- 172.21.15.27:0/3329765992 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff1080047d0 0x7ff12806c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:25.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.256+0000 7ff130200640 1 -- 172.21.15.27:0/3329765992 >> 172.21.15.27:0/3329765992 conn(0x7ff1280cb700 msgr2=0x7ff1280aeb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:25.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.256+0000 7ff130200640 1 -- 172.21.15.27:0/3329765992 shutdown_connections 2024-01-15T18:11:25.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.256+0000 7ff130200640 1 -- 172.21.15.27:0/3329765992 wait complete. 2024-01-15T18:11:25.282 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:11:25.282 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone91817 2024-01-15T18:11:25.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.428+0000 7fed69600640 1 Processor -- start 2024-01-15T18:11:25.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.428+0000 7fed69600640 1 -- start start 2024-01-15T18:11:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.432+0000 7fed69600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fed6415d570 0x7fed6406be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.432+0000 7fed69600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fed6406b810 0x7fed6406c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.432+0000 7fed69600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed640b0d70 con 0x7fed6415d190 2024-01-15T18:11:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.432+0000 7fed69600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed640d0540 con 0x7fed6406b810 2024-01-15T18:11:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.432+0000 7fed69600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed640b0440 con 0x7fed6415d570 2024-01-15T18:11:25.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.432+0000 7fed63400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fed6406b810 0x7fed6406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:25.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.432+0000 7fed62a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fed6415d190 0x7fed640cfcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55174/0 (socket says 172.21.15.27:55174) 2024-01-15T18:11:25.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.432+0000 7fed62a00640 1 -- 172.21.15.27:0/1043634770 learned_addr learned my addr 172.21.15.27:0/1043634770 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:25.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.432+0000 7fed62000640 1 --2- 172.21.15.27:0/1043634770 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fed6415d570 0x7fed6406be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:25.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.432+0000 7fed63400640 1 -- 172.21.15.27:0/1043634770 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fed6415d570 msgr2=0x7fed6406be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:25.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.432+0000 7fed63400640 1 --2- 172.21.15.27:0/1043634770 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fed6415d570 0x7fed6406be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:25.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.432+0000 7fed63400640 1 -- 172.21.15.27:0/1043634770 >> v1:172.21.15.27:6789/0 conn(0x7fed6415d190 legacy=0x7fed640cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:25.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.432+0000 7fed63400640 1 -- 172.21.15.27:0/1043634770 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed640b0720 con 0x7fed6406b810 2024-01-15T18:11:25.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.432+0000 7fed61600640 1 -- 172.21.15.27:0/1043634770 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097294967 0 0) 0x7fed640b0d70 con 0x7fed6415d190 2024-01-15T18:11:25.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.432+0000 7fed62000640 1 --2- 172.21.15.27:0/1043634770 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fed6415d570 0x7fed6406be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:11:25.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.436+0000 7fed63400640 1 --2- 172.21.15.27:0/1043634770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed6406b810 0x7fed6406c350 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fed5c009d90 tx=0x7fed5c01c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61374c6366146994 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:25.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.436+0000 7fed61600640 1 -- 172.21.15.27:0/1043634770 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c0a5a00 con 0x7fed6406b810 2024-01-15T18:11:25.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.436+0000 7fed61600640 1 -- 172.21.15.27:0/1043634770 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed5c0a5b60 con 0x7fed6406b810 2024-01-15T18:11:25.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.436+0000 7fed61600640 1 -- 172.21.15.27:0/1043634770 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c0ad440 con 0x7fed6406b810 2024-01-15T18:11:25.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.436+0000 7fed69600640 1 -- 172.21.15.27:0/1043634770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed6406b810 msgr2=0x7fed6406c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:25.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.436+0000 7fed69600640 1 --2- 172.21.15.27:0/1043634770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed6406b810 0x7fed6406c350 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fed5c009d90 tx=0x7fed5c01c920 comp rx=0 tx=0).stop 2024-01-15T18:11:25.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.436+0000 7fed69600640 1 -- 172.21.15.27:0/1043634770 shutdown_connections 2024-01-15T18:11:25.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.436+0000 7fed69600640 1 --2- 172.21.15.27:0/1043634770 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed6406b810 0x7fed6406c350 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:25.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.436+0000 7fed69600640 1 --2- 172.21.15.27:0/1043634770 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fed6415d570 0x7fed6406be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:25.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.436+0000 7fed69600640 1 -- 172.21.15.27:0/1043634770 >> 172.21.15.27:0/1043634770 conn(0x7fed640d0e70 msgr2=0x7fed640d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:25.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.436+0000 7fed69600640 1 -- 172.21.15.27:0/1043634770 shutdown_connections 2024-01-15T18:11:25.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.436+0000 7fed69600640 1 -- 172.21.15.27:0/1043634770 wait complete. 2024-01-15T18:11:25.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.436+0000 7fed69600640 1 Processor -- start 2024-01-15T18:11:25.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.436+0000 7fed69600640 1 -- start start 2024-01-15T18:11:25.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.440+0000 7fed69600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fed6406b810 0x7fed6415a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:25.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.440+0000 7fed69600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed6415d190 0x7fed6414eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:25.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.440+0000 7fed69600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fed6415d570 0x7fed6415af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:25.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.440+0000 7fed69600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed64159740 con 0x7fed6415d190 2024-01-15T18:11:25.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.440+0000 7fed69600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed64158970 con 0x7fed6415d570 2024-01-15T18:11:25.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.440+0000 7fed69600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed6414f080 con 0x7fed6406b810 2024-01-15T18:11:25.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.440+0000 7fed62000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed6415d190 0x7fed6414eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:25.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.440+0000 7fed62000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed6415d190 0x7fed6414eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39636/0 (socket says 172.21.15.27:39636) 2024-01-15T18:11:25.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.440+0000 7fed62000640 1 -- 172.21.15.27:0/1850621494 learned_addr learned my addr 172.21.15.27:0/1850621494 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:25.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.440+0000 7fed62a00640 1 --2- 172.21.15.27:0/1850621494 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fed6406b810 0x7fed6415a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:25.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.440+0000 7fed63400640 1 --2- 172.21.15.27:0/1850621494 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fed6415d570 0x7fed6415af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:25.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.440+0000 7fed62000640 1 -- 172.21.15.27:0/1850621494 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fed6406b810 msgr2=0x7fed6415a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:25.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.440+0000 7fed62000640 1 --2- 172.21.15.27:0/1850621494 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fed6406b810 0x7fed6415a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:25.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.440+0000 7fed62000640 1 -- 172.21.15.27:0/1850621494 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fed6415d570 msgr2=0x7fed6415af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:25.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.440+0000 7fed62000640 1 --2- 172.21.15.27:0/1850621494 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fed6415d570 0x7fed6415af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:25.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.440+0000 7fed62000640 1 -- 172.21.15.27:0/1850621494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5c009a70 con 0x7fed6415d190 2024-01-15T18:11:25.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.440+0000 7fed62000640 1 --2- 172.21.15.27:0/1850621494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed6415d190 0x7fed6414eb40 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fed4c00d830 tx=0x7fed4c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:25.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.440+0000 7fed5aa00640 1 -- 172.21.15.27:0/1850621494 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c00b6c0 con 0x7fed6415d190 2024-01-15T18:11:25.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.440+0000 7fed5aa00640 1 -- 172.21.15.27:0/1850621494 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4c00b820 con 0x7fed6415d190 2024-01-15T18:11:25.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.440+0000 7fed5aa00640 1 -- 172.21.15.27:0/1850621494 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c00bbd0 con 0x7fed6415d190 2024-01-15T18:11:25.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.440+0000 7fed69600640 1 -- 172.21.15.27:0/1850621494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed64159950 con 0x7fed6415d190 2024-01-15T18:11:25.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.440+0000 7fed69600640 1 -- 172.21.15.27:0/1850621494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed64159f10 con 0x7fed6415d190 2024-01-15T18:11:25.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.440+0000 7fed69600640 1 -- 172.21.15.27:0/1850621494 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed640b11e0 con 0x7fed6415d190 2024-01-15T18:11:25.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.440+0000 7fed5aa00640 1 -- 172.21.15.27:0/1850621494 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7fed4c01e460 con 0x7fed6415d190 2024-01-15T18:11:25.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.444+0000 7fed5aa00640 1 --2- 172.21.15.27:0/1850621494 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fed3c067190 0x7fed3c069650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:25.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.444+0000 7fed5aa00640 1 -- 172.21.15.27:0/1850621494 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fed4c083450 con 0x7fed6415d190 2024-01-15T18:11:25.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.444+0000 7fed62a00640 1 --2- 172.21.15.27:0/1850621494 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fed3c067190 0x7fed3c069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:25.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.444+0000 7fed62a00640 1 --2- 172.21.15.27:0/1850621494 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fed3c067190 0x7fed3c069650 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fed54007fd0 tx=0x7fed54007d60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:25.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.448+0000 7fed5aa00640 1 -- 172.21.15.27:0/1850621494 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fed640b11e0 con 0x7fed6415d190 2024-01-15T18:11:25.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:25.636+0000 7fed69600640 1 -- 172.21.15.27:0/1850621494 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone91817", "target": ["mon-mgr", ""]}) v1 -- 0x7fed640b11e0 con 0x7fed3c067190 2024-01-15T18:11:26.966 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:11:26.970 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:11:30.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.252+0000 7fed5aa00640 1 -- 172.21.15.27:0/1850621494 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fed640b11e0 con 0x7fed3c067190 2024-01-15T18:11:30.256 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:11:30.256 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:11:30.256 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "complete" 2024-01-15T18:11:30.256 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:30.256 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:11:30.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.256+0000 7fed69600640 1 -- 172.21.15.27:0/1850621494 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fed3c067190 msgr2=0x7fed3c069650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:30.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.256+0000 7fed69600640 1 --2- 172.21.15.27:0/1850621494 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fed3c067190 0x7fed3c069650 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fed54007fd0 tx=0x7fed54007d60 comp rx=0 tx=0).stop 2024-01-15T18:11:30.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.256+0000 7fed69600640 1 -- 172.21.15.27:0/1850621494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed6415d190 msgr2=0x7fed6414eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:30.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.256+0000 7fed69600640 1 --2- 172.21.15.27:0/1850621494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed6415d190 0x7fed6414eb40 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fed4c00d830 tx=0x7fed4c00dcf0 comp rx=0 tx=0).stop 2024-01-15T18:11:30.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.256+0000 7fed69600640 1 -- 172.21.15.27:0/1850621494 shutdown_connections 2024-01-15T18:11:30.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.256+0000 7fed69600640 1 --2- 172.21.15.27:0/1850621494 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fed3c067190 0x7fed3c069650 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:30.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.256+0000 7fed69600640 1 --2- 172.21.15.27:0/1850621494 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fed6415d570 0x7fed6415af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:30.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.256+0000 7fed69600640 1 --2- 172.21.15.27:0/1850621494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed6415d190 0x7fed6414eb40 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:30.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.256+0000 7fed69600640 1 --2- 172.21.15.27:0/1850621494 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fed6406b810 0x7fed6415a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:30.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.256+0000 7fed69600640 1 -- 172.21.15.27:0/1850621494 >> 172.21.15.27:0/1850621494 conn(0x7fed640d0e70 msgr2=0x7fed6406bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:30.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.256+0000 7fed69600640 1 -- 172.21.15.27:0/1850621494 shutdown_connections 2024-01-15T18:11:30.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.256+0000 7fed69600640 1 -- 172.21.15.27:0/1850621494 wait complete. 2024-01-15T18:11:30.283 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:11:30.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.428+0000 7fab0ec00640 1 Processor -- start 2024-01-15T18:11:30.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.428+0000 7fab0ec00640 1 -- start start 2024-01-15T18:11:30.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.428+0000 7fab0ec00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fab0815d2a0 0x7fab080cfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:30.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.432+0000 7fab0ec00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fab080cde10 0x7fab080cc150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:30.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.432+0000 7fab0ec00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab080b3cd0 con 0x7fab0815d680 2024-01-15T18:11:30.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.432+0000 7fab0ec00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab080d0260 con 0x7fab080cde10 2024-01-15T18:11:30.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.432+0000 7fab0ec00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab080cb670 con 0x7fab0815d2a0 2024-01-15T18:11:30.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.432+0000 7fab0cc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fab080cde10 0x7fab080cc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:30.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.432+0000 7fab07400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fab0815d680 0x7fab080cd700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55178/0 (socket says 172.21.15.27:55178) 2024-01-15T18:11:30.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.432+0000 7fab07400640 1 -- 172.21.15.27:0/3132499353 learned_addr learned my addr 172.21.15.27:0/3132499353 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:30.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.432+0000 7fab07e00640 1 --2- 172.21.15.27:0/3132499353 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fab0815d2a0 0x7fab080cfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:30.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.432+0000 7fab0cc00640 1 -- 172.21.15.27:0/3132499353 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fab0815d2a0 msgr2=0x7fab080cfbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:30.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.432+0000 7fab0cc00640 1 --2- 172.21.15.27:0/3132499353 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fab0815d2a0 0x7fab080cfbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:30.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.432+0000 7fab0cc00640 1 -- 172.21.15.27:0/3132499353 >> v1:172.21.15.27:6789/0 conn(0x7fab0815d680 legacy=0x7fab080cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:30.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.432+0000 7fab0cc00640 1 -- 172.21.15.27:0/3132499353 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab080cb950 con 0x7fab080cde10 2024-01-15T18:11:30.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.432+0000 7fab06a00640 1 -- 172.21.15.27:0/3132499353 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2181582377 0 0) 0x7fab080b3cd0 con 0x7fab0815d680 2024-01-15T18:11:30.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.432+0000 7fab07e00640 1 --2- 172.21.15.27:0/3132499353 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fab0815d2a0 0x7fab080cfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:11:30.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.432+0000 7fab0cc00640 1 --2- 172.21.15.27:0/3132499353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab080cde10 0x7fab080cc150 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fab00009d90 tx=0x7fab0001c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2db73b83faa459c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:30.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.432+0000 7fab06a00640 1 -- 172.21.15.27:0/3132499353 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab000a5930 con 0x7fab080cde10 2024-01-15T18:11:30.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.436+0000 7fab06a00640 1 -- 172.21.15.27:0/3132499353 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab000a5a90 con 0x7fab080cde10 2024-01-15T18:11:30.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.436+0000 7fab06a00640 1 -- 172.21.15.27:0/3132499353 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab000a5e20 con 0x7fab080cde10 2024-01-15T18:11:30.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.436+0000 7fab0ec00640 1 -- 172.21.15.27:0/3132499353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab080cde10 msgr2=0x7fab080cc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:30.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.436+0000 7fab0ec00640 1 --2- 172.21.15.27:0/3132499353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab080cde10 0x7fab080cc150 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fab00009d90 tx=0x7fab0001c890 comp rx=0 tx=0).stop 2024-01-15T18:11:30.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.436+0000 7fab0ec00640 1 -- 172.21.15.27:0/3132499353 shutdown_connections 2024-01-15T18:11:30.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.436+0000 7fab0ec00640 1 --2- 172.21.15.27:0/3132499353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab080cde10 0x7fab080cc150 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:30.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.436+0000 7fab0ec00640 1 --2- 172.21.15.27:0/3132499353 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fab0815d2a0 0x7fab080cfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:30.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.436+0000 7fab0ec00640 1 -- 172.21.15.27:0/3132499353 >> 172.21.15.27:0/3132499353 conn(0x7fab080d0e80 msgr2=0x7fab080d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:30.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.436+0000 7fab0ec00640 1 -- 172.21.15.27:0/3132499353 shutdown_connections 2024-01-15T18:11:30.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.436+0000 7fab0ec00640 1 -- 172.21.15.27:0/3132499353 wait complete. 2024-01-15T18:11:30.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.436+0000 7fab0ec00640 1 Processor -- start 2024-01-15T18:11:30.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.436+0000 7fab0ec00640 1 -- start start 2024-01-15T18:11:30.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.436+0000 7fab0ec00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab080cde10 0x7fab080c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:30.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.436+0000 7fab0ec00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fab0815d2a0 0x7fab080c8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:30.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.436+0000 7fab0ec00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab0815d680 0x7fab080c8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.436+0000 7fab0ec00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab080c92e0 con 0x7fab0815d680 2024-01-15T18:11:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.436+0000 7fab0ec00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab080c95c0 con 0x7fab080cde10 2024-01-15T18:11:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.436+0000 7fab0ec00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab080c98a0 con 0x7fab0815d2a0 2024-01-15T18:11:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.440+0000 7fab0cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab0815d680 0x7fab080c8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.440+0000 7fab0cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab0815d680 0x7fab080c8690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39644/0 (socket says 172.21.15.27:39644) 2024-01-15T18:11:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.440+0000 7fab0cc00640 1 -- 172.21.15.27:0/1310095753 learned_addr learned my addr 172.21.15.27:0/1310095753 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.440+0000 7fab07400640 1 --2- 172.21.15.27:0/1310095753 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fab0815d2a0 0x7fab080c8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.440+0000 7fab07e00640 1 --2- 172.21.15.27:0/1310095753 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab080cde10 0x7fab080c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.440+0000 7fab0cc00640 1 -- 172.21.15.27:0/1310095753 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fab0815d2a0 msgr2=0x7fab080c8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.440+0000 7fab0cc00640 1 --2- 172.21.15.27:0/1310095753 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fab0815d2a0 0x7fab080c8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.440+0000 7fab0cc00640 1 -- 172.21.15.27:0/1310095753 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab080cde10 msgr2=0x7fab080c72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.440+0000 7fab0cc00640 1 --2- 172.21.15.27:0/1310095753 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab080cde10 0x7fab080c72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.440+0000 7fab0cc00640 1 -- 172.21.15.27:0/1310095753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab00009a70 con 0x7fab0815d680 2024-01-15T18:11:30.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.440+0000 7fab0cc00640 1 --2- 172.21.15.27:0/1310095753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab0815d680 0x7fab080c8690 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fab0001af40 tx=0x7fab00019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:30.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.440+0000 7faaffe00640 1 -- 172.21.15.27:0/1310095753 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab000ade40 con 0x7fab0815d680 2024-01-15T18:11:30.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.440+0000 7faaffe00640 1 -- 172.21.15.27:0/1310095753 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab00009450 con 0x7fab0815d680 2024-01-15T18:11:30.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.440+0000 7faaffe00640 1 -- 172.21.15.27:0/1310095753 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab000b9670 con 0x7fab0815d680 2024-01-15T18:11:30.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.440+0000 7fab0ec00640 1 -- 172.21.15.27:0/1310095753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab08157130 con 0x7fab0815d680 2024-01-15T18:11:30.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.440+0000 7fab0ec00640 1 -- 172.21.15.27:0/1310095753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab08154d80 con 0x7fab0815d680 2024-01-15T18:11:30.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.440+0000 7fab0ec00640 1 -- 172.21.15.27:0/1310095753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab08156f90 con 0x7fab0815d680 2024-01-15T18:11:30.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.440+0000 7faaffe00640 1 -- 172.21.15.27:0/1310095753 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7fab000b97d0 con 0x7fab0815d680 2024-01-15T18:11:30.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.444+0000 7faaffe00640 1 --2- 172.21.15.27:0/1310095753 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faae0067160 0x7faae0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:30.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.444+0000 7faaffe00640 1 -- 172.21.15.27:0/1310095753 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fab0011e630 con 0x7fab0815d680 2024-01-15T18:11:30.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.444+0000 7fab07e00640 1 --2- 172.21.15.27:0/1310095753 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faae0067160 0x7faae0069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:30.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.444+0000 7fab07e00640 1 --2- 172.21.15.27:0/1310095753 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faae0067160 0x7faae0069620 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7faaf8006e90 tx=0x7faaf8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:30.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.448+0000 7faaffe00640 1 -- 172.21.15.27:0/1310095753 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fab08156f90 con 0x7fab0815d680 2024-01-15T18:11:30.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.624+0000 7fab0ec00640 1 -- 172.21.15.27:0/1310095753 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fab080c9730 con 0x7faae0067160 2024-01-15T18:11:30.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.628+0000 7faaffe00640 1 -- 172.21.15.27:0/1310095753 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fab080c9730 con 0x7faae0067160 2024-01-15T18:11:30.629 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:11:30.629 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:30.629 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-15T18:11:30.629 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:30.629 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:11:30.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.628+0000 7fab0ec00640 1 -- 172.21.15.27:0/1310095753 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faae0067160 msgr2=0x7faae0069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:30.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.628+0000 7fab0ec00640 1 --2- 172.21.15.27:0/1310095753 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faae0067160 0x7faae0069620 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7faaf8006e90 tx=0x7faaf8008040 comp rx=0 tx=0).stop 2024-01-15T18:11:30.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.628+0000 7fab0ec00640 1 -- 172.21.15.27:0/1310095753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab0815d680 msgr2=0x7fab080c8690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:30.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.628+0000 7fab0ec00640 1 --2- 172.21.15.27:0/1310095753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab0815d680 0x7fab080c8690 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fab0001af40 tx=0x7fab00019630 comp rx=0 tx=0).stop 2024-01-15T18:11:30.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.628+0000 7fab0ec00640 1 -- 172.21.15.27:0/1310095753 shutdown_connections 2024-01-15T18:11:30.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.628+0000 7fab0ec00640 1 --2- 172.21.15.27:0/1310095753 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faae0067160 0x7faae0069620 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:30.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.628+0000 7fab0ec00640 1 --2- 172.21.15.27:0/1310095753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab0815d680 0x7fab080c8690 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:30.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.628+0000 7fab0ec00640 1 --2- 172.21.15.27:0/1310095753 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fab0815d2a0 0x7fab080c8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:30.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.628+0000 7fab0ec00640 1 --2- 172.21.15.27:0/1310095753 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab080cde10 0x7fab080c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:30.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.628+0000 7fab0ec00640 1 -- 172.21.15.27:0/1310095753 >> 172.21.15.27:0/1310095753 conn(0x7fab080d0e80 msgr2=0x7fab0815c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:30.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.628+0000 7fab0ec00640 1 -- 172.21.15.27:0/1310095753 shutdown_connections 2024-01-15T18:11:30.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.628+0000 7fab0ec00640 1 -- 172.21.15.27:0/1310095753 wait complete. 2024-01-15T18:11:30.652 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-15T18:11:30.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.792+0000 7fd642400640 1 Processor -- start 2024-01-15T18:11:30.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.792+0000 7fd642400640 1 -- start start 2024-01-15T18:11:30.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.796+0000 7fd642400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd63c143690 0x7fd63c070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:30.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.796+0000 7fd642400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd63c15dab0 0x7fd63c15af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:30.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.796+0000 7fd642400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fd63c0b2fc0 0x7fd63c0b3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:30.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.796+0000 7fd642400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd63c15ea10 con 0x7fd63c15dab0 2024-01-15T18:11:30.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.796+0000 7fd642400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd63c15b480 con 0x7fd63c0b2fc0 2024-01-15T18:11:30.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.796+0000 7fd642400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c0b3970 con 0x7fd63c143690 2024-01-15T18:11:30.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.796+0000 7fd63aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd63c15dab0 0x7fd63c15af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:30.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.796+0000 7fd63aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd63c15dab0 0x7fd63c15af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39654/0 (socket says 172.21.15.27:39654) 2024-01-15T18:11:30.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.796+0000 7fd63aa00640 1 -- 172.21.15.27:0/2265394699 learned_addr learned my addr 172.21.15.27:0/2265394699 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:30.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.796+0000 7fd63b400640 1 --2- 172.21.15.27:0/2265394699 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd63c143690 0x7fd63c070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:30.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.796+0000 7fd63be00640 1 --2- 172.21.15.27:0/2265394699 >> v2:172.21.15.93:3300/0 conn(0x7fd63c0b2fc0 0x7fd63c0b3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:30.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.796+0000 7fd63aa00640 1 -- 172.21.15.27:0/2265394699 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd63c143690 msgr2=0x7fd63c070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:30.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.796+0000 7fd63aa00640 1 --2- 172.21.15.27:0/2265394699 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd63c143690 0x7fd63c070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:30.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.796+0000 7fd63aa00640 1 -- 172.21.15.27:0/2265394699 >> v2:172.21.15.93:3300/0 conn(0x7fd63c0b2fc0 msgr2=0x7fd63c0b3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:30.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.796+0000 7fd63aa00640 1 --2- 172.21.15.27:0/2265394699 >> v2:172.21.15.93:3300/0 conn(0x7fd63c0b2fc0 0x7fd63c0b3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:30.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.796+0000 7fd63aa00640 1 -- 172.21.15.27:0/2265394699 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd63c0cd520 con 0x7fd63c15dab0 2024-01-15T18:11:30.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.796+0000 7fd63be00640 1 --2- 172.21.15.27:0/2265394699 >> v2:172.21.15.93:3300/0 conn(0x7fd63c0b2fc0 0x7fd63c0b3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:11:30.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.800+0000 7fd63aa00640 1 --2- 172.21.15.27:0/2265394699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd63c15dab0 0x7fd63c15af40 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fd63c0b1e40 tx=0x7fd62401ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2396bd27ec5e8bb8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:30.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.800+0000 7fd63a000640 1 -- 172.21.15.27:0/2265394699 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6240a7b50 con 0x7fd63c15dab0 2024-01-15T18:11:30.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.800+0000 7fd63a000640 1 -- 172.21.15.27:0/2265394699 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6240a7cb0 con 0x7fd63c15dab0 2024-01-15T18:11:30.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.800+0000 7fd63a000640 1 -- 172.21.15.27:0/2265394699 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6240af660 con 0x7fd63c15dab0 2024-01-15T18:11:30.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.800+0000 7fd642400640 1 -- 172.21.15.27:0/2265394699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd63c15dab0 msgr2=0x7fd63c15af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:30.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.800+0000 7fd642400640 1 --2- 172.21.15.27:0/2265394699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd63c15dab0 0x7fd63c15af40 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fd63c0b1e40 tx=0x7fd62401ebd0 comp rx=0 tx=0).stop 2024-01-15T18:11:30.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.800+0000 7fd642400640 1 -- 172.21.15.27:0/2265394699 shutdown_connections 2024-01-15T18:11:30.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.800+0000 7fd642400640 1 --2- 172.21.15.27:0/2265394699 >> v2:172.21.15.93:3300/0 conn(0x7fd63c0b2fc0 0x7fd63c0b3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:30.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.800+0000 7fd642400640 1 --2- 172.21.15.27:0/2265394699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd63c15dab0 0x7fd63c15af40 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:30.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.800+0000 7fd642400640 1 --2- 172.21.15.27:0/2265394699 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd63c143690 0x7fd63c070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:30.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.800+0000 7fd642400640 1 -- 172.21.15.27:0/2265394699 >> 172.21.15.27:0/2265394699 conn(0x7fd63c06a8f0 msgr2=0x7fd63c06ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:30.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.800+0000 7fd642400640 1 -- 172.21.15.27:0/2265394699 shutdown_connections 2024-01-15T18:11:30.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.800+0000 7fd642400640 1 -- 172.21.15.27:0/2265394699 wait complete. 2024-01-15T18:11:30.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.800+0000 7fd642400640 1 Processor -- start 2024-01-15T18:11:30.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.800+0000 7fd642400640 1 -- start start 2024-01-15T18:11:30.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd642400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd63c0b2fc0 0x7fd63c139980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:30.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd642400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd63c143690 0x7fd63c13f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:30.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd642400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd63c15dab0 0x7fd63c15bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:30.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd642400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c13ca60 con 0x7fd63c143690 2024-01-15T18:11:30.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd642400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c13d300 con 0x7fd63c15dab0 2024-01-15T18:11:30.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd642400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c140420 con 0x7fd63c0b2fc0 2024-01-15T18:11:30.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd63aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd63c143690 0x7fd63c13f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:30.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd63aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd63c143690 0x7fd63c13f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39664/0 (socket says 172.21.15.27:39664) 2024-01-15T18:11:30.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd63be00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd63c15dab0 0x7fd63c15bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:30.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd63b400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd63c0b2fc0 0x7fd63c139980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:30.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd63aa00640 1 -- 172.21.15.27:0/1305353 learned_addr learned my addr 172.21.15.27:0/1305353 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:30.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd63aa00640 1 -- 172.21.15.27:0/1305353 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd63c0b2fc0 msgr2=0x7fd63c139980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:30.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd63aa00640 1 --2- 172.21.15.27:0/1305353 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd63c0b2fc0 0x7fd63c139980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:30.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd63aa00640 1 -- 172.21.15.27:0/1305353 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd63c15dab0 msgr2=0x7fd63c15bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:30.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd63aa00640 1 --2- 172.21.15.27:0/1305353 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd63c15dab0 0x7fd63c15bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:30.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd63aa00640 1 -- 172.21.15.27:0/1305353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd62400b770 con 0x7fd63c143690 2024-01-15T18:11:30.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd63b400640 1 --2- 172.21.15.27:0/1305353 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd63c0b2fc0 0x7fd63c139980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:11:30.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd63be00640 1 --2- 172.21.15.27:0/1305353 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd63c15dab0 0x7fd63c15bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:11:30.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd63aa00640 1 --2- 172.21.15.27:0/1305353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd63c143690 0x7fd63c13f9d0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fd62400b2c0 tx=0x7fd62401b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:30.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd633400640 1 -- 172.21.15.27:0/1305353 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6240bf070 con 0x7fd63c143690 2024-01-15T18:11:30.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd633400640 1 -- 172.21.15.27:0/1305353 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6240b9400 con 0x7fd63c143690 2024-01-15T18:11:30.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd633400640 1 -- 172.21.15.27:0/1305353 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd624006ee0 con 0x7fd63c143690 2024-01-15T18:11:30.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd642400640 1 -- 172.21.15.27:0/1305353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd63c13ccb0 con 0x7fd63c143690 2024-01-15T18:11:30.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd642400640 1 -- 172.21.15.27:0/1305353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd63c139ec0 con 0x7fd63c143690 2024-01-15T18:11:30.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd642400640 1 -- 172.21.15.27:0/1305353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd63c0b0890 con 0x7fd63c143690 2024-01-15T18:11:30.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.804+0000 7fd633400640 1 -- 172.21.15.27:0/1305353 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7fd6240b97b0 con 0x7fd63c143690 2024-01-15T18:11:30.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.808+0000 7fd633400640 1 --2- 172.21.15.27:0/1305353 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd614067160 0x7fd614069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:30.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.808+0000 7fd633400640 1 -- 172.21.15.27:0/1305353 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fd62411fc00 con 0x7fd63c143690 2024-01-15T18:11:30.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.808+0000 7fd63b400640 1 --2- 172.21.15.27:0/1305353 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd614067160 0x7fd614069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:30.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.808+0000 7fd63b400640 1 --2- 172.21.15.27:0/1305353 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd614067160 0x7fd614069620 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd63c13c9f0 tx=0x7fd62c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:30.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.812+0000 7fd633400640 1 -- 172.21.15.27:0/1305353 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd6240ee770 con 0x7fd63c143690 2024-01-15T18:11:30.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:30.988+0000 7fd642400640 1 -- 172.21.15.27:0/1305353 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd63c157270 con 0x7fd614067160 2024-01-15T18:11:31.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.000+0000 7fd633400640 1 -- 172.21.15.27:0/1305353 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+381 (secure 0 0 0) 0x7fd63c157270 con 0x7fd614067160 2024-01-15T18:11:31.001 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:11:31.001 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:11:31.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:31.001 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone91816" 2024-01-15T18:11:31.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:11:31.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:31.001 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sv1" 2024-01-15T18:11:31.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:11:31.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:31.001 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone91817" 2024-01-15T18:11:31.001 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:11:31.001 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:31.002 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone91814" 2024-01-15T18:11:31.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:11:31.002 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:31.002 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone91813" 2024-01-15T18:11:31.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:11:31.002 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:31.002 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone91812" 2024-01-15T18:11:31.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:11:31.002 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:31.002 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone91815" 2024-01-15T18:11:31.002 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:11:31.002 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:31.002 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone91811" 2024-01-15T18:11:31.002 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:31.003 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:11:31.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.000+0000 7fd642400640 1 -- 172.21.15.27:0/1305353 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd614067160 msgr2=0x7fd614069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:31.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.000+0000 7fd642400640 1 --2- 172.21.15.27:0/1305353 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd614067160 0x7fd614069620 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd63c13c9f0 tx=0x7fd62c008040 comp rx=0 tx=0).stop 2024-01-15T18:11:31.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.000+0000 7fd642400640 1 -- 172.21.15.27:0/1305353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd63c143690 msgr2=0x7fd63c13f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:31.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.000+0000 7fd642400640 1 --2- 172.21.15.27:0/1305353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd63c143690 0x7fd63c13f9d0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fd62400b2c0 tx=0x7fd62401b970 comp rx=0 tx=0).stop 2024-01-15T18:11:31.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.000+0000 7fd642400640 1 -- 172.21.15.27:0/1305353 shutdown_connections 2024-01-15T18:11:31.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.000+0000 7fd642400640 1 --2- 172.21.15.27:0/1305353 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd614067160 0x7fd614069620 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:31.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.000+0000 7fd642400640 1 --2- 172.21.15.27:0/1305353 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd63c15dab0 0x7fd63c15bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:31.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.000+0000 7fd642400640 1 --2- 172.21.15.27:0/1305353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd63c143690 0x7fd63c13f9d0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:31.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.000+0000 7fd642400640 1 --2- 172.21.15.27:0/1305353 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd63c0b2fc0 0x7fd63c139980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:31.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.000+0000 7fd642400640 1 -- 172.21.15.27:0/1305353 >> 172.21.15.27:0/1305353 conn(0x7fd63c06a8f0 msgr2=0x7fd63c072040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:31.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.000+0000 7fd642400640 1 -- 172.21.15.27:0/1305353 shutdown_connections 2024-01-15T18:11:31.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.000+0000 7fd642400640 1 -- 172.21.15.27:0/1305353 wait complete. 2024-01-15T18:11:31.023 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone91816 2024-01-15T18:11:31.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.168+0000 7f6fe9a00640 1 Processor -- start 2024-01-15T18:11:31.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.168+0000 7f6fe9a00640 1 -- start start 2024-01-15T18:11:31.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.168+0000 7f6fe9a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f6fe415f550 0x7f6fe40cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:31.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.168+0000 7f6fe9a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe40b1cb0 con 0x7f6fe4158900 2024-01-15T18:11:31.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.168+0000 7f6fe9a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fe415fa00 con 0x7f6fe40cf110 2024-01-15T18:11:31.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.168+0000 7f6fe9a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fe40cf580 con 0x7f6fe415f550 2024-01-15T18:11:31.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.168+0000 7f6fe2a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6fe4158900 0x7f6fe415bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55190/0 (socket says 172.21.15.27:55190) 2024-01-15T18:11:31.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.168+0000 7f6fe2a00640 1 -- 172.21.15.27:0/2451700808 learned_addr learned my addr 172.21.15.27:0/2451700808 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:31.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.168+0000 7f6fe2000640 1 --2- 172.21.15.27:0/2451700808 >> v2:172.21.15.93:3300/0 conn(0x7f6fe415f550 0x7f6fe40cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:31.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.168+0000 7f6fe1600640 1 -- 172.21.15.27:0/2451700808 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524890142 0 0) 0x7f6fe40b1cb0 con 0x7f6fe4158900 2024-01-15T18:11:31.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.168+0000 7f6fe1600640 1 -- 172.21.15.27:0/2451700808 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fc4003530 con 0x7f6fe4158900 2024-01-15T18:11:31.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.172+0000 7f6fe1600640 1 -- 172.21.15.27:0/2451700808 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1759699467 0 0) 0x7f6fc4003530 con 0x7f6fe4158900 2024-01-15T18:11:31.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.172+0000 7f6fe1600640 1 -- 172.21.15.27:0/2451700808 >> v2:172.21.15.93:3300/0 conn(0x7f6fe415f550 msgr2=0x7f6fe40cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:31.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.172+0000 7f6fe1600640 1 --2- 172.21.15.27:0/2451700808 >> v2:172.21.15.93:3300/0 conn(0x7f6fe415f550 0x7f6fe40cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:31.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.172+0000 7f6fe1600640 1 -- 172.21.15.27:0/2451700808 >> v1:172.21.15.93:6789/0 conn(0x7f6fe40cf110 legacy=0x7f6fe40701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:31.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.172+0000 7f6fe1600640 1 -- 172.21.15.27:0/2451700808 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe40b1f00 con 0x7f6fe4158900 2024-01-15T18:11:31.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.172+0000 7f6fe1600640 1 -- 172.21.15.27:0/2451700808 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f6fd4001680 con 0x7f6fe4158900 2024-01-15T18:11:31.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.172+0000 7f6fe1600640 1 -- 172.21.15.27:0/2451700808 >> v1:172.21.15.27:6789/0 conn(0x7f6fe4158900 legacy=0x7f6fe415bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:31.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.172+0000 7f6fe1600640 1 --2- 172.21.15.27:0/2451700808 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6fc4003a40 0x7f6fc4003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:31.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.172+0000 7f6fe1600640 1 --2- 172.21.15.27:0/2451700808 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6fc40047d0 0x7f6fc4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:31.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.172+0000 7f6fe1600640 1 --2- 172.21.15.27:0/2451700808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc40051e0 0x7f6fc400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:31.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.172+0000 7f6fe1600640 1 -- 172.21.15.27:0/2451700808 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe40cf580 con 0x7f6fc40051e0 2024-01-15T18:11:31.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.172+0000 7f6fe1600640 1 -- 172.21.15.27:0/2451700808 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc400c110 con 0x7f6fc4003a40 2024-01-15T18:11:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.172+0000 7f6fe1600640 1 -- 172.21.15.27:0/2451700808 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc400c2d0 con 0x7f6fc40047d0 2024-01-15T18:11:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.172+0000 7f6fe3400640 1 --2- 172.21.15.27:0/2451700808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc40051e0 0x7f6fc400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.172+0000 7f6fe2a00640 1 --2- 172.21.15.27:0/2451700808 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6fc4003a40 0x7f6fc4003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.172+0000 7f6fe2000640 1 --2- 172.21.15.27:0/2451700808 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6fc40047d0 0x7f6fc4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.172+0000 7f6fe2000640 1 -- 172.21.15.27:0/2451700808 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6fc4003a40 msgr2=0x7f6fc4003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.172+0000 7f6fe2000640 1 --2- 172.21.15.27:0/2451700808 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6fc4003a40 0x7f6fc4003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.172+0000 7f6fe2000640 1 -- 172.21.15.27:0/2451700808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc40051e0 msgr2=0x7f6fc400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.172+0000 7f6fe2000640 1 --2- 172.21.15.27:0/2451700808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc40051e0 0x7f6fc400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.172+0000 7f6fe2000640 1 -- 172.21.15.27:0/2451700808 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc4003530 con 0x7f6fc40047d0 2024-01-15T18:11:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.172+0000 7f6fe3400640 1 --2- 172.21.15.27:0/2451700808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc40051e0 0x7f6fc400b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:11:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.172+0000 7f6fe3400640 1 -- 172.21.15.27:0/2451700808 reap_dead start 2024-01-15T18:11:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.176+0000 7f6fe2000640 1 --2- 172.21.15.27:0/2451700808 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6fc40047d0 0x7f6fc4004c50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc009ae0 tx=0x7f6fcc01ca50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8f250ce956865cd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.176+0000 7f6fe1600640 1 -- 172.21.15.27:0/2451700808 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc0a5640 con 0x7f6fc40047d0 2024-01-15T18:11:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.176+0000 7f6fe1600640 1 -- 172.21.15.27:0/2451700808 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fcc0a57a0 con 0x7f6fc40047d0 2024-01-15T18:11:31.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.176+0000 7f6fe1600640 1 -- 172.21.15.27:0/2451700808 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc0a5b50 con 0x7f6fc40047d0 2024-01-15T18:11:31.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.176+0000 7f6fe9a00640 1 -- 172.21.15.27:0/2451700808 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6fc40047d0 msgr2=0x7f6fc4004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:31.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.176+0000 7f6fe9a00640 1 --2- 172.21.15.27:0/2451700808 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6fc40047d0 0x7f6fc4004c50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc009ae0 tx=0x7f6fcc01ca50 comp rx=0 tx=0).stop 2024-01-15T18:11:31.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.176+0000 7f6fe9a00640 1 -- 172.21.15.27:0/2451700808 shutdown_connections 2024-01-15T18:11:31.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.176+0000 7f6fe9a00640 1 --2- 172.21.15.27:0/2451700808 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6fc40047d0 0x7f6fc4004c50 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:31.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.176+0000 7f6fe9a00640 1 -- 172.21.15.27:0/2451700808 >> 172.21.15.27:0/2451700808 conn(0x7f6fe40cb700 msgr2=0x7f6fe40cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:31.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.176+0000 7f6fe9a00640 1 -- 172.21.15.27:0/2451700808 shutdown_connections 2024-01-15T18:11:31.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.176+0000 7f6fe9a00640 1 -- 172.21.15.27:0/2451700808 wait complete. 2024-01-15T18:11:31.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.176+0000 7f6fe9a00640 1 Processor -- start 2024-01-15T18:11:31.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.176+0000 7f6fe9a00640 1 -- start start 2024-01-15T18:11:31.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.180+0000 7f6fe9a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc40047d0 0x7f6fe41596f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:31.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.180+0000 7f6fe9a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6fe40adf60 0x7f6fe4148660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:31.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.180+0000 7f6fe9a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6fe40af780 0x7f6fe40afc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:31.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.180+0000 7f6fe9a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe40b0140 con 0x7f6fc40047d0 2024-01-15T18:11:31.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.180+0000 7f6fe9a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe4148ba0 con 0x7f6fe40af780 2024-01-15T18:11:31.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.180+0000 7f6fe9a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe40cd970 con 0x7f6fe40adf60 2024-01-15T18:11:31.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.180+0000 7f6fe2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc40047d0 0x7f6fe41596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:31.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.180+0000 7f6fe2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc40047d0 0x7f6fe41596f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39676/0 (socket says 172.21.15.27:39676) 2024-01-15T18:11:31.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.180+0000 7f6fe2a00640 1 -- 172.21.15.27:0/3099790973 learned_addr learned my addr 172.21.15.27:0/3099790973 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:31.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.180+0000 7f6fe2000640 1 --2- 172.21.15.27:0/3099790973 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6fe40adf60 0x7f6fe4148660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.180+0000 7f6fe3400640 1 --2- 172.21.15.27:0/3099790973 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6fe40af780 0x7f6fe40afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.180+0000 7f6fe2a00640 1 -- 172.21.15.27:0/3099790973 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6fe40adf60 msgr2=0x7f6fe4148660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.180+0000 7f6fe2a00640 1 --2- 172.21.15.27:0/3099790973 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6fe40adf60 0x7f6fe4148660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.180+0000 7f6fe2a00640 1 -- 172.21.15.27:0/3099790973 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6fe40af780 msgr2=0x7f6fe40afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.180+0000 7f6fe2a00640 1 --2- 172.21.15.27:0/3099790973 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6fe40af780 0x7f6fe40afc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.180+0000 7f6fe2a00640 1 -- 172.21.15.27:0/3099790973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe40cde00 con 0x7f6fc40047d0 2024-01-15T18:11:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.180+0000 7f6fe3400640 1 --2- 172.21.15.27:0/3099790973 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6fe40af780 0x7f6fe40afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:11:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.180+0000 7f6fe2a00640 1 --2- 172.21.15.27:0/3099790973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc40047d0 0x7f6fe41596f0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4001410 tx=0x7f6fd4007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.180+0000 7f6fdaa00640 1 -- 172.21.15.27:0/3099790973 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd4001670 con 0x7f6fc40047d0 2024-01-15T18:11:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.180+0000 7f6fdaa00640 1 -- 172.21.15.27:0/3099790973 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd400f960 con 0x7f6fc40047d0 2024-01-15T18:11:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.180+0000 7f6fe9a00640 1 -- 172.21.15.27:0/3099790973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe40ccd80 con 0x7f6fc40047d0 2024-01-15T18:11:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.180+0000 7f6fdaa00640 1 -- 172.21.15.27:0/3099790973 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd400fd50 con 0x7f6fc40047d0 2024-01-15T18:11:31.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.180+0000 7f6fe9a00640 1 -- 172.21.15.27:0/3099790973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe40cd320 con 0x7f6fc40047d0 2024-01-15T18:11:31.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.180+0000 7f6fe9a00640 1 -- 172.21.15.27:0/3099790973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe404e850 con 0x7f6fc40047d0 2024-01-15T18:11:31.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.184+0000 7f6fdaa00640 1 -- 172.21.15.27:0/3099790973 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7f6fd401d460 con 0x7f6fc40047d0 2024-01-15T18:11:31.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.184+0000 7f6fdaa00640 1 --2- 172.21.15.27:0/3099790973 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6fbc067160 0x7f6fbc069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:31.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.184+0000 7f6fdaa00640 1 -- 172.21.15.27:0/3099790973 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f6fd4082ef0 con 0x7f6fc40047d0 2024-01-15T18:11:31.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.184+0000 7f6fe2000640 1 --2- 172.21.15.27:0/3099790973 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6fbc067160 0x7f6fbc069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:31.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.184+0000 7f6fe2000640 1 --2- 172.21.15.27:0/3099790973 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6fbc067160 0x7f6fbc069620 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc0af000 tx=0x7f6fcc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:31.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.188+0000 7f6fdaa00640 1 -- 172.21.15.27:0/3099790973 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6fd4051a10 con 0x7f6fc40047d0 2024-01-15T18:11:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.368+0000 7f6fe9a00640 1 -- 172.21.15.27:0/3099790973 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone91816", "target": ["mon-mgr", ""]}) v1 -- 0x7f6fe415ed50 con 0x7f6fbc067160 2024-01-15T18:11:31.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.388+0000 7f6fdaa00640 1 -- 172.21.15.27:0/3099790973 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6fcc0a59d0 con 0x7f6fbc067160 2024-01-15T18:11:31.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.388+0000 7f6fe9a00640 1 -- 172.21.15.27:0/3099790973 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6fbc067160 msgr2=0x7f6fbc069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:31.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.388+0000 7f6fe9a00640 1 --2- 172.21.15.27:0/3099790973 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6fbc067160 0x7f6fbc069620 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc0af000 tx=0x7f6fcc008040 comp rx=0 tx=0).stop 2024-01-15T18:11:31.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.388+0000 7f6fe9a00640 1 -- 172.21.15.27:0/3099790973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc40047d0 msgr2=0x7f6fe41596f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:31.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.388+0000 7f6fe9a00640 1 --2- 172.21.15.27:0/3099790973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc40047d0 0x7f6fe41596f0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4001410 tx=0x7f6fd4007720 comp rx=0 tx=0).stop 2024-01-15T18:11:31.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.388+0000 7f6fe9a00640 1 -- 172.21.15.27:0/3099790973 shutdown_connections 2024-01-15T18:11:31.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.388+0000 7f6fe9a00640 1 --2- 172.21.15.27:0/3099790973 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6fbc067160 0x7f6fbc069620 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:31.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.388+0000 7f6fe9a00640 1 --2- 172.21.15.27:0/3099790973 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6fe40af780 0x7f6fe40afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:31.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.388+0000 7f6fe9a00640 1 --2- 172.21.15.27:0/3099790973 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6fe40adf60 0x7f6fe4148660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:31.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.388+0000 7f6fe9a00640 1 --2- 172.21.15.27:0/3099790973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fc40047d0 0x7f6fe41596f0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:31.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.388+0000 7f6fe9a00640 1 -- 172.21.15.27:0/3099790973 >> 172.21.15.27:0/3099790973 conn(0x7f6fe40cb700 msgr2=0x7f6fe4157210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:31.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.392+0000 7f6fe9a00640 1 -- 172.21.15.27:0/3099790973 shutdown_connections 2024-01-15T18:11:31.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.392+0000 7f6fe9a00640 1 -- 172.21.15.27:0/3099790973 wait complete. 2024-01-15T18:11:31.394 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:11:31.392+0000 7f59f4c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:11:31.394 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:11:31.392+0000 7f59f4c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:11:31.394 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:11:31.392+0000 7f59f4c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:11:31.394 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:11:31.392+0000 7f59f4c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:11:31.394 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:11:31.392+0000 7f59f4c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:11:31.413 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-15T18:11:31.414 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:11:31.412+0000 7f59f5600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:11:31.414 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:11:31.412+0000 7f59f5600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:11:31.414 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:11:31.412+0000 7f59f5600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:11:31.414 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:11:31.412+0000 7f59f5600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:11:31.414 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:11:31.412+0000 7f59f5600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:11:31.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.564+0000 7f2ef1e00640 1 Processor -- start 2024-01-15T18:11:31.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.564+0000 7f2ef1e00640 1 -- start start 2024-01-15T18:11:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.564+0000 7f2ef1e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2eec06c430 0x7f2eec070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.564+0000 7f2ef1e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2eec0ce760 0x7f2eec0b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.564+0000 7f2ef1e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eec0ccfb0 con 0x7f2eec0b1db0 2024-01-15T18:11:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.564+0000 7f2ef1e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eec0cd290 con 0x7f2eec06c430 2024-01-15T18:11:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.564+0000 7f2ef1e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eec0cd570 con 0x7f2eec0ce760 2024-01-15T18:11:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.564+0000 7f2eeaa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2eec06c430 0x7f2eec070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.564+0000 7f2eea000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2eec0b1db0 0x7f2eec0ce050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55202/0 (socket says 172.21.15.27:55202) 2024-01-15T18:11:31.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.564+0000 7f2eea000640 1 -- 172.21.15.27:0/1463405444 learned_addr learned my addr 172.21.15.27:0/1463405444 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:31.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.564+0000 7f2eeaa00640 1 -- 172.21.15.27:0/1463405444 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2eec0ce760 msgr2=0x7f2eec0b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:31.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.564+0000 7f2eeaa00640 1 --2- 172.21.15.27:0/1463405444 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2eec0ce760 0x7f2eec0b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:31.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.564+0000 7f2eeaa00640 1 -- 172.21.15.27:0/1463405444 >> v1:172.21.15.27:6789/0 conn(0x7f2eec0b1db0 legacy=0x7f2eec0ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:31.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.564+0000 7f2eeaa00640 1 -- 172.21.15.27:0/1463405444 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eec0cd850 con 0x7f2eec06c430 2024-01-15T18:11:31.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.568+0000 7f2eeaa00640 1 --2- 172.21.15.27:0/1463405444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec06c430 0x7f2eec070ba0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f2edc0097f0 tx=0x7f2edc01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b488e2be3b9bccf1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:31.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.568+0000 7f2ee9600640 1 -- 172.21.15.27:0/1463405444 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc0ad680 con 0x7f2eec06c430 2024-01-15T18:11:31.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.568+0000 7f2ee9600640 1 -- 172.21.15.27:0/1463405444 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2edc0ad7e0 con 0x7f2eec06c430 2024-01-15T18:11:31.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.568+0000 7f2ee9600640 1 -- 172.21.15.27:0/1463405444 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc0adb70 con 0x7f2eec06c430 2024-01-15T18:11:31.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.568+0000 7f2ef1e00640 1 -- 172.21.15.27:0/1463405444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec06c430 msgr2=0x7f2eec070ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:31.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.568+0000 7f2ef1e00640 1 --2- 172.21.15.27:0/1463405444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec06c430 0x7f2eec070ba0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f2edc0097f0 tx=0x7f2edc01c3e0 comp rx=0 tx=0).stop 2024-01-15T18:11:31.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.568+0000 7f2ef1e00640 1 -- 172.21.15.27:0/1463405444 shutdown_connections 2024-01-15T18:11:31.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.568+0000 7f2ef1e00640 1 --2- 172.21.15.27:0/1463405444 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2eec0ce760 0x7f2eec0b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:31.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.568+0000 7f2ef1e00640 1 --2- 172.21.15.27:0/1463405444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec06c430 0x7f2eec070ba0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:31.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.568+0000 7f2ef1e00640 1 -- 172.21.15.27:0/1463405444 >> 172.21.15.27:0/1463405444 conn(0x7f2eec0cb700 msgr2=0x7f2eec0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:31.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.568+0000 7f2ef1e00640 1 -- 172.21.15.27:0/1463405444 shutdown_connections 2024-01-15T18:11:31.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.568+0000 7f2ef1e00640 1 -- 172.21.15.27:0/1463405444 wait complete. 2024-01-15T18:11:31.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.572+0000 7f2ef1e00640 1 Processor -- start 2024-01-15T18:11:31.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.572+0000 7f2ef1e00640 1 -- start start 2024-01-15T18:11:31.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.572+0000 7f2ef1e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2eec06c430 0x7f2eec159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:31.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.572+0000 7f2ef1e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2eec0b1db0 0x7f2eec142990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.572+0000 7f2ef1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec0ce760 0x7f2eec14ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.572+0000 7f2ef1e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eec15a190 con 0x7f2eec0ce760 2024-01-15T18:11:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.572+0000 7f2ef1e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eec15a470 con 0x7f2eec06c430 2024-01-15T18:11:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.572+0000 7f2ef1e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eec15a750 con 0x7f2eec0b1db0 2024-01-15T18:11:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.572+0000 7f2eeb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec0ce760 0x7f2eec14ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.572+0000 7f2eeb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec0ce760 0x7f2eec14ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39698/0 (socket says 172.21.15.27:39698) 2024-01-15T18:11:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.572+0000 7f2eeb400640 1 -- 172.21.15.27:0/1083137087 learned_addr learned my addr 172.21.15.27:0/1083137087 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.572+0000 7f2eeb400640 1 -- 172.21.15.27:0/1083137087 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2eec0b1db0 msgr2=0x7f2eec142990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.572+0000 7f2eeb400640 1 --2- 172.21.15.27:0/1083137087 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2eec0b1db0 0x7f2eec142990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.572+0000 7f2eeb400640 1 -- 172.21.15.27:0/1083137087 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2eec06c430 msgr2=0x7f2eec159b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.572+0000 7f2eeb400640 1 --2- 172.21.15.27:0/1083137087 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2eec06c430 0x7f2eec159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:31.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.572+0000 7f2eeb400640 1 -- 172.21.15.27:0/1083137087 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2edc009420 con 0x7f2eec0ce760 2024-01-15T18:11:31.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.572+0000 7f2eeb400640 1 --2- 172.21.15.27:0/1083137087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec0ce760 0x7f2eec14ed00 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f2ee400b9e0 tx=0x7f2ee400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:31.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.572+0000 7f2ee3400640 1 -- 172.21.15.27:0/1083137087 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee400cb20 con 0x7f2eec0ce760 2024-01-15T18:11:31.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.572+0000 7f2ee3400640 1 -- 172.21.15.27:0/1083137087 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee400cca0 con 0x7f2eec0ce760 2024-01-15T18:11:31.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.572+0000 7f2ee3400640 1 -- 172.21.15.27:0/1083137087 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4014680 con 0x7f2eec0ce760 2024-01-15T18:11:31.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.572+0000 7f2ef1e00640 1 -- 172.21.15.27:0/1083137087 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eec0712f0 con 0x7f2eec0ce760 2024-01-15T18:11:31.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.572+0000 7f2ef1e00640 1 -- 172.21.15.27:0/1083137087 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eec14f240 con 0x7f2eec0ce760 2024-01-15T18:11:31.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.572+0000 7f2ef1e00640 1 -- 172.21.15.27:0/1083137087 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eec04e850 con 0x7f2eec0ce760 2024-01-15T18:11:31.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.576+0000 7f2ee3400640 1 -- 172.21.15.27:0/1083137087 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86925+0+0 (secure 0 0 0) 0x7f2ee401d3b0 con 0x7f2eec0ce760 2024-01-15T18:11:31.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.576+0000 7f2ee3400640 1 --2- 172.21.15.27:0/1083137087 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2ec4067190 0x7f2ec4069650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:31.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.576+0000 7f2ee3400640 1 -- 172.21.15.27:0/1083137087 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f2ee4082d40 con 0x7f2eec0ce760 2024-01-15T18:11:31.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.576+0000 7f2eeaa00640 1 --2- 172.21.15.27:0/1083137087 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2ec4067190 0x7f2ec4069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:31.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.576+0000 7f2eeaa00640 1 --2- 172.21.15.27:0/1083137087 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2ec4067190 0x7f2ec4069650 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f2edc0af000 tx=0x7f2edc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:31.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.580+0000 7f2ee3400640 1 -- 172.21.15.27:0/1083137087 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2ee4051800 con 0x7f2eec0ce760 2024-01-15T18:11:31.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:31.764+0000 7f2ef1e00640 1 -- 172.21.15.27:0/1083137087 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2eec15ced0 con 0x7f2ec4067190 2024-01-15T18:11:33.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:33.056+0000 7f2ee3400640 1 -- 172.21.15.27:0/1083137087 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 19) v1 ==== 87027+0+0 (secure 0 0 0) 0x7f2ee4051bd0 con 0x7f2eec0ce760 2024-01-15T18:11:35.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.252+0000 7f2ee3400640 1 -- 172.21.15.27:0/1083137087 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f2eec15ced0 con 0x7f2ec4067190 2024-01-15T18:11:35.255 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:11:35.255 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:11:35.255 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:35.255 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ss1" 2024-01-15T18:11:35.255 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:35.255 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:11:35.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.256+0000 7f2ef1e00640 1 -- 172.21.15.27:0/1083137087 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2ec4067190 msgr2=0x7f2ec4069650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:35.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.256+0000 7f2ef1e00640 1 --2- 172.21.15.27:0/1083137087 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2ec4067190 0x7f2ec4069650 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f2edc0af000 tx=0x7f2edc008040 comp rx=0 tx=0).stop 2024-01-15T18:11:35.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.256+0000 7f2ef1e00640 1 -- 172.21.15.27:0/1083137087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec0ce760 msgr2=0x7f2eec14ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:35.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.256+0000 7f2ef1e00640 1 --2- 172.21.15.27:0/1083137087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec0ce760 0x7f2eec14ed00 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f2ee400b9e0 tx=0x7f2ee400bea0 comp rx=0 tx=0).stop 2024-01-15T18:11:35.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.256+0000 7f2ef1e00640 1 -- 172.21.15.27:0/1083137087 shutdown_connections 2024-01-15T18:11:35.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.256+0000 7f2ef1e00640 1 --2- 172.21.15.27:0/1083137087 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2ec4067190 0x7f2ec4069650 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:35.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.256+0000 7f2ef1e00640 1 --2- 172.21.15.27:0/1083137087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2eec0ce760 0x7f2eec14ed00 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:35.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.256+0000 7f2ef1e00640 1 --2- 172.21.15.27:0/1083137087 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2eec0b1db0 0x7f2eec142990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:35.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.256+0000 7f2ef1e00640 1 --2- 172.21.15.27:0/1083137087 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2eec06c430 0x7f2eec159b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:35.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.256+0000 7f2ef1e00640 1 -- 172.21.15.27:0/1083137087 >> 172.21.15.27:0/1083137087 conn(0x7f2eec0cb700 msgr2=0x7f2eec1528a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:35.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.256+0000 7f2ef1e00640 1 -- 172.21.15.27:0/1083137087 shutdown_connections 2024-01-15T18:11:35.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.256+0000 7f2ef1e00640 1 -- 172.21.15.27:0/1083137087 wait complete. 2024-01-15T18:11:35.278 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:11:35.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.424+0000 7fb546a00640 1 Processor -- start 2024-01-15T18:11:35.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.424+0000 7fb546a00640 1 -- start start 2024-01-15T18:11:35.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.428+0000 7fb546a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb54015d570 0x7fb54006be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:35.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.428+0000 7fb546a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5400b0d70 con 0x7fb54015d190 2024-01-15T18:11:35.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.428+0000 7fb546a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5400d0540 con 0x7fb54006b810 2024-01-15T18:11:35.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.428+0000 7fb546a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5400b0440 con 0x7fb54015d570 2024-01-15T18:11:35.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.428+0000 7fb53fe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb54015d190 0x7fb5400cfcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53070/0 (socket says 172.21.15.27:53070) 2024-01-15T18:11:35.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.428+0000 7fb53fe00640 1 -- 172.21.15.27:0/2062126412 learned_addr learned my addr 172.21.15.27:0/2062126412 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:35.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.428+0000 7fb53f400640 1 --2- 172.21.15.27:0/2062126412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb54015d570 0x7fb54006be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:35.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.428+0000 7fb53ea00640 1 -- 172.21.15.27:0/2062126412 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3327729695 0 0) 0x7fb5400b0d70 con 0x7fb54015d190 2024-01-15T18:11:35.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.428+0000 7fb53ea00640 1 -- 172.21.15.27:0/2062126412 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb520003530 con 0x7fb54015d190 2024-01-15T18:11:35.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.428+0000 7fb53f400640 1 -- 172.21.15.27:0/2062126412 >> v1:172.21.15.93:6789/0 conn(0x7fb54006b810 legacy=0x7fb5400cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:35.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.428+0000 7fb53ea00640 1 -- 172.21.15.27:0/2062126412 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3388836264 0 0) 0x7fb5400d0540 con 0x7fb54006b810 2024-01-15T18:11:35.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.428+0000 7fb53f400640 1 -- 172.21.15.27:0/2062126412 >> v1:172.21.15.27:6789/0 conn(0x7fb54015d190 legacy=0x7fb5400cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:35.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.428+0000 7fb53f400640 1 -- 172.21.15.27:0/2062126412 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5400b0720 con 0x7fb54015d570 2024-01-15T18:11:35.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.432+0000 7fb53f400640 1 --2- 172.21.15.27:0/2062126412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb54015d570 0x7fb54006be10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb5280097f0 tx=0x7fb52801c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95bccddfe5ef87e3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:35.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.432+0000 7fb53ea00640 1 -- 172.21.15.27:0/2062126412 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5280aa610 con 0x7fb54015d570 2024-01-15T18:11:35.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.432+0000 7fb53ea00640 1 -- 172.21.15.27:0/2062126412 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5280a6040 con 0x7fb54015d570 2024-01-15T18:11:35.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.432+0000 7fb53ea00640 1 -- 172.21.15.27:0/2062126412 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5280aa970 con 0x7fb54015d570 2024-01-15T18:11:35.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.432+0000 7fb546a00640 1 -- 172.21.15.27:0/2062126412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb54015d570 msgr2=0x7fb54006be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:35.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.432+0000 7fb546a00640 1 --2- 172.21.15.27:0/2062126412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb54015d570 0x7fb54006be10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb5280097f0 tx=0x7fb52801c490 comp rx=0 tx=0).stop 2024-01-15T18:11:35.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.432+0000 7fb546a00640 1 -- 172.21.15.27:0/2062126412 shutdown_connections 2024-01-15T18:11:35.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.432+0000 7fb546a00640 1 --2- 172.21.15.27:0/2062126412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb54015d570 0x7fb54006be10 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:35.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.432+0000 7fb546a00640 1 -- 172.21.15.27:0/2062126412 >> 172.21.15.27:0/2062126412 conn(0x7fb5400d0e70 msgr2=0x7fb5400d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:35.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.432+0000 7fb546a00640 1 -- 172.21.15.27:0/2062126412 shutdown_connections 2024-01-15T18:11:35.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.432+0000 7fb546a00640 1 -- 172.21.15.27:0/2062126412 wait complete. 2024-01-15T18:11:35.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.432+0000 7fb546a00640 1 Processor -- start 2024-01-15T18:11:35.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.432+0000 7fb546a00640 1 -- start start 2024-01-15T18:11:35.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.436+0000 7fb546a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54006b810 0x7fb5400c4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:35.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.436+0000 7fb546a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb54015d190 0x7fb54014d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:35.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.436+0000 7fb546a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb54015d570 0x7fb54015c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:35.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.436+0000 7fb546a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54014e6a0 con 0x7fb54006b810 2024-01-15T18:11:35.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.436+0000 7fb546a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5400b6840 con 0x7fb54015d190 2024-01-15T18:11:35.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.436+0000 7fb546a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb54014e050 con 0x7fb54015d570 2024-01-15T18:11:35.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.436+0000 7fb53fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54006b810 0x7fb5400c4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:35.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.436+0000 7fb53fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54006b810 0x7fb5400c4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57368/0 (socket says 172.21.15.27:57368) 2024-01-15T18:11:35.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.436+0000 7fb53fe00640 1 -- 172.21.15.27:0/4251310402 learned_addr learned my addr 172.21.15.27:0/4251310402 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:35.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.436+0000 7fb53f400640 1 --2- 172.21.15.27:0/4251310402 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb54015d190 0x7fb54014d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:35.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.436+0000 7fb544a00640 1 --2- 172.21.15.27:0/4251310402 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb54015d570 0x7fb54015c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:35.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.436+0000 7fb53fe00640 1 -- 172.21.15.27:0/4251310402 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb54015d570 msgr2=0x7fb54015c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:35.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.436+0000 7fb53fe00640 1 --2- 172.21.15.27:0/4251310402 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb54015d570 0x7fb54015c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:35.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.436+0000 7fb53fe00640 1 -- 172.21.15.27:0/4251310402 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb54015d190 msgr2=0x7fb54014d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:35.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.436+0000 7fb53fe00640 1 --2- 172.21.15.27:0/4251310402 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb54015d190 0x7fb54014d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:35.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.436+0000 7fb53fe00640 1 -- 172.21.15.27:0/4251310402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb528009420 con 0x7fb54006b810 2024-01-15T18:11:35.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.436+0000 7fb53fe00640 1 --2- 172.21.15.27:0/4251310402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54006b810 0x7fb5400c4f70 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fb53000c450 tx=0x7fb53000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:35.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.436+0000 7fb537e00640 1 -- 172.21.15.27:0/4251310402 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb530005cf0 con 0x7fb54006b810 2024-01-15T18:11:35.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.436+0000 7fb537e00640 1 -- 172.21.15.27:0/4251310402 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb530005e50 con 0x7fb54006b810 2024-01-15T18:11:35.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.436+0000 7fb537e00640 1 -- 172.21.15.27:0/4251310402 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb530007290 con 0x7fb54006b810 2024-01-15T18:11:35.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.436+0000 7fb546a00640 1 -- 172.21.15.27:0/4251310402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb54014e330 con 0x7fb54006b810 2024-01-15T18:11:35.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.436+0000 7fb546a00640 1 -- 172.21.15.27:0/4251310402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb54015c8e0 con 0x7fb54006b810 2024-01-15T18:11:35.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.436+0000 7fb546a00640 1 -- 172.21.15.27:0/4251310402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb54014deb0 con 0x7fb54006b810 2024-01-15T18:11:35.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.436+0000 7fb537e00640 1 -- 172.21.15.27:0/4251310402 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 87027+0+0 (secure 0 0 0) 0x7fb5300073f0 con 0x7fb54006b810 2024-01-15T18:11:35.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.440+0000 7fb537e00640 1 --2- 172.21.15.27:0/4251310402 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb518067280 0x7fb518069740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:35.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.440+0000 7fb537e00640 1 -- 172.21.15.27:0/4251310402 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fb530082420 con 0x7fb54006b810 2024-01-15T18:11:35.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.440+0000 7fb53f400640 1 --2- 172.21.15.27:0/4251310402 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb518067280 0x7fb518069740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:35.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.440+0000 7fb53f400640 1 --2- 172.21.15.27:0/4251310402 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb518067280 0x7fb518069740 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb5280ae000 tx=0x7fb5280a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:35.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.444+0000 7fb537e00640 1 -- 172.21.15.27:0/4251310402 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb54014deb0 con 0x7fb54006b810 2024-01-15T18:11:35.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.632+0000 7fb546a00640 1 -- 172.21.15.27:0/4251310402 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb54014e1c0 con 0x7fb518067280 2024-01-15T18:11:35.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.660+0000 7fb537e00640 1 -- 172.21.15.27:0/4251310402 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb54014e1c0 con 0x7fb518067280 2024-01-15T18:11:35.662 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:11:35.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.660+0000 7fb546a00640 1 -- 172.21.15.27:0/4251310402 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb518067280 msgr2=0x7fb518069740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:35.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.660+0000 7fb546a00640 1 --2- 172.21.15.27:0/4251310402 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb518067280 0x7fb518069740 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb5280ae000 tx=0x7fb5280a7000 comp rx=0 tx=0).stop 2024-01-15T18:11:35.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.660+0000 7fb546a00640 1 -- 172.21.15.27:0/4251310402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54006b810 msgr2=0x7fb5400c4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:35.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.660+0000 7fb546a00640 1 --2- 172.21.15.27:0/4251310402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54006b810 0x7fb5400c4f70 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fb53000c450 tx=0x7fb53000c910 comp rx=0 tx=0).stop 2024-01-15T18:11:35.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.664+0000 7fb546a00640 1 -- 172.21.15.27:0/4251310402 shutdown_connections 2024-01-15T18:11:35.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.664+0000 7fb546a00640 1 --2- 172.21.15.27:0/4251310402 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb518067280 0x7fb518069740 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:35.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.664+0000 7fb546a00640 1 --2- 172.21.15.27:0/4251310402 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb54015d570 0x7fb54015c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:35.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.664+0000 7fb546a00640 1 --2- 172.21.15.27:0/4251310402 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb54015d190 0x7fb54014d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:35.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.664+0000 7fb546a00640 1 --2- 172.21.15.27:0/4251310402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb54006b810 0x7fb5400c4f70 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:35.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.664+0000 7fb546a00640 1 -- 172.21.15.27:0/4251310402 >> 172.21.15.27:0/4251310402 conn(0x7fb5400d0e70 msgr2=0x7fb54006fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:35.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.664+0000 7fb546a00640 1 -- 172.21.15.27:0/4251310402 shutdown_connections 2024-01-15T18:11:35.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.664+0000 7fb546a00640 1 -- 172.21.15.27:0/4251310402 wait complete. 2024-01-15T18:11:35.684 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-15T18:11:35.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.828+0000 7fcd5e200640 1 Processor -- start 2024-01-15T18:11:35.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.828+0000 7fcd5e200640 1 -- start start 2024-01-15T18:11:35.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.828+0000 7fcd5e200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd5806c3f0 0x7fcd58168250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:35.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.828+0000 7fcd5e200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd580b04b0 con 0x7fcd5806b600 2024-01-15T18:11:35.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.828+0000 7fcd5e200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd580b0720 con 0x7fcd5815d3b0 2024-01-15T18:11:35.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.828+0000 7fcd5e200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd580b0a00 con 0x7fcd5806c3f0 2024-01-15T18:11:35.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.828+0000 7fcd56a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcd5806b600 0x7fcd5806bce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53076/0 (socket says 172.21.15.27:53076) 2024-01-15T18:11:35.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.828+0000 7fcd56a00640 1 -- 172.21.15.27:0/385014216 learned_addr learned my addr 172.21.15.27:0/385014216 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:35.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.828+0000 7fcd57e00640 1 --2- 172.21.15.27:0/385014216 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd5806c3f0 0x7fcd58168250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:35.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.828+0000 7fcd56000640 1 -- 172.21.15.27:0/385014216 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254079585 0 0) 0x7fcd580b04b0 con 0x7fcd5806b600 2024-01-15T18:11:35.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.828+0000 7fcd56000640 1 -- 172.21.15.27:0/385014216 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd38003530 con 0x7fcd5806b600 2024-01-15T18:11:35.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.828+0000 7fcd56000640 1 -- 172.21.15.27:0/385014216 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2246853951 0 0) 0x7fcd580b0720 con 0x7fcd5815d3b0 2024-01-15T18:11:35.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.828+0000 7fcd56000640 1 -- 172.21.15.27:0/385014216 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd580b04b0 con 0x7fcd5815d3b0 2024-01-15T18:11:35.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.828+0000 7fcd57e00640 1 -- 172.21.15.27:0/385014216 >> v1:172.21.15.93:6789/0 conn(0x7fcd5815d3b0 legacy=0x7fcd580cfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:35.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.828+0000 7fcd57e00640 1 -- 172.21.15.27:0/385014216 >> v1:172.21.15.27:6789/0 conn(0x7fcd5806b600 legacy=0x7fcd5806bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:35.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.828+0000 7fcd57e00640 1 -- 172.21.15.27:0/385014216 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd580b0ce0 con 0x7fcd5806c3f0 2024-01-15T18:11:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.832+0000 7fcd57e00640 1 --2- 172.21.15.27:0/385014216 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd5806c3f0 0x7fcd58168250 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcd50009dc0 tx=0x7fcd5001c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55195d4c94147b80 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.832+0000 7fcd56000640 1 -- 172.21.15.27:0/385014216 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd500a59d0 con 0x7fcd5806c3f0 2024-01-15T18:11:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.832+0000 7fcd56000640 1 -- 172.21.15.27:0/385014216 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd500a5b30 con 0x7fcd5806c3f0 2024-01-15T18:11:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.832+0000 7fcd56000640 1 -- 172.21.15.27:0/385014216 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd500ad440 con 0x7fcd5806c3f0 2024-01-15T18:11:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.832+0000 7fcd5e200640 1 -- 172.21.15.27:0/385014216 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd5806c3f0 msgr2=0x7fcd58168250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.832+0000 7fcd5e200640 1 --2- 172.21.15.27:0/385014216 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd5806c3f0 0x7fcd58168250 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcd50009dc0 tx=0x7fcd5001c970 comp rx=0 tx=0).stop 2024-01-15T18:11:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.832+0000 7fcd5e200640 1 -- 172.21.15.27:0/385014216 shutdown_connections 2024-01-15T18:11:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.832+0000 7fcd5e200640 1 --2- 172.21.15.27:0/385014216 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd5806c3f0 0x7fcd58168250 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.832+0000 7fcd5e200640 1 -- 172.21.15.27:0/385014216 >> 172.21.15.27:0/385014216 conn(0x7fcd580d0ed0 msgr2=0x7fcd580d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.832+0000 7fcd5e200640 1 -- 172.21.15.27:0/385014216 shutdown_connections 2024-01-15T18:11:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.832+0000 7fcd5e200640 1 -- 172.21.15.27:0/385014216 wait complete. 2024-01-15T18:11:35.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.836+0000 7fcd5e200640 1 Processor -- start 2024-01-15T18:11:35.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.836+0000 7fcd5e200640 1 -- start start 2024-01-15T18:11:35.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.836+0000 7fcd5e200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcd5806b600 0x7fcd58157400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:35.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.836+0000 7fcd5e200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd5806c3f0 0x7fcd581558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:35.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.836+0000 7fcd5e200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd5815d3b0 0x7fcd5809d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:35.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.836+0000 7fcd5e200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd58156360 con 0x7fcd5815d3b0 2024-01-15T18:11:35.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.836+0000 7fcd5e200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd58156640 con 0x7fcd5806b600 2024-01-15T18:11:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.836+0000 7fcd5e200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd58156920 con 0x7fcd5806c3f0 2024-01-15T18:11:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.836+0000 7fcd57e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd5815d3b0 0x7fcd5809d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.836+0000 7fcd57e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd5815d3b0 0x7fcd5809d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57382/0 (socket says 172.21.15.27:57382) 2024-01-15T18:11:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.836+0000 7fcd57e00640 1 -- 172.21.15.27:0/56111127 learned_addr learned my addr 172.21.15.27:0/56111127 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.836+0000 7fcd56a00640 1 --2- 172.21.15.27:0/56111127 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd5806c3f0 0x7fcd581558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.836+0000 7fcd57400640 1 --2- 172.21.15.27:0/56111127 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcd5806b600 0x7fcd58157400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.836+0000 7fcd57e00640 1 -- 172.21.15.27:0/56111127 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd5806c3f0 msgr2=0x7fcd581558a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.836+0000 7fcd57e00640 1 --2- 172.21.15.27:0/56111127 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd5806c3f0 0x7fcd581558a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.836+0000 7fcd57e00640 1 -- 172.21.15.27:0/56111127 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcd5806b600 msgr2=0x7fcd58157400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.836+0000 7fcd57e00640 1 --2- 172.21.15.27:0/56111127 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcd5806b600 0x7fcd58157400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.836+0000 7fcd57e00640 1 -- 172.21.15.27:0/56111127 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd50009a70 con 0x7fcd5815d3b0 2024-01-15T18:11:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.836+0000 7fcd56a00640 1 --2- 172.21.15.27:0/56111127 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd5806c3f0 0x7fcd581558a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:11:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.836+0000 7fcd57e00640 1 --2- 172.21.15.27:0/56111127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd5815d3b0 0x7fcd5809d780 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fcd5001a5a0 tx=0x7fcd50019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.836+0000 7fcd4f400640 1 -- 172.21.15.27:0/56111127 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd500adc80 con 0x7fcd5815d3b0 2024-01-15T18:11:35.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.836+0000 7fcd4f400640 1 -- 172.21.15.27:0/56111127 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd500adde0 con 0x7fcd5815d3b0 2024-01-15T18:11:35.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.836+0000 7fcd5e200640 1 -- 172.21.15.27:0/56111127 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd58156c00 con 0x7fcd5815d3b0 2024-01-15T18:11:35.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.836+0000 7fcd4f400640 1 -- 172.21.15.27:0/56111127 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd500b9690 con 0x7fcd5815d3b0 2024-01-15T18:11:35.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.836+0000 7fcd5e200640 1 -- 172.21.15.27:0/56111127 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd581580a0 con 0x7fcd5815d3b0 2024-01-15T18:11:35.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.840+0000 7fcd5e200640 1 -- 172.21.15.27:0/56111127 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd5804e850 con 0x7fcd5815d3b0 2024-01-15T18:11:35.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.840+0000 7fcd4f400640 1 -- 172.21.15.27:0/56111127 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 87027+0+0 (secure 0 0 0) 0x7fcd500b97f0 con 0x7fcd5815d3b0 2024-01-15T18:11:35.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.840+0000 7fcd4f400640 1 --2- 172.21.15.27:0/56111127 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcd30067280 0x7fcd30069740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:35.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.840+0000 7fcd4f400640 1 -- 172.21.15.27:0/56111127 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fcd5011e620 con 0x7fcd5815d3b0 2024-01-15T18:11:35.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.840+0000 7fcd57400640 1 --2- 172.21.15.27:0/56111127 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcd30067280 0x7fcd30069740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:35.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.840+0000 7fcd57400640 1 --2- 172.21.15.27:0/56111127 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcd30067280 0x7fcd30069740 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fcd480011f0 tx=0x7fcd48009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:35.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:35.844+0000 7fcd4f400640 1 -- 172.21.15.27:0/56111127 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fcd500ed0b0 con 0x7fcd5815d3b0 2024-01-15T18:11:36.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.028+0000 7fcd5e200640 1 -- 172.21.15.27:0/56111127 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd581570b0 con 0x7fcd30067280 2024-01-15T18:11:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.044+0000 7fcd4f400640 1 -- 172.21.15.27:0/56111127 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fcd581570b0 con 0x7fcd30067280 2024-01-15T18:11:36.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.044+0000 7fcd5e200640 1 -- 172.21.15.27:0/56111127 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcd30067280 msgr2=0x7fcd30069740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:36.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.044+0000 7fcd5e200640 1 --2- 172.21.15.27:0/56111127 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcd30067280 0x7fcd30069740 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fcd480011f0 tx=0x7fcd48009040 comp rx=0 tx=0).stop 2024-01-15T18:11:36.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.044+0000 7fcd5e200640 1 -- 172.21.15.27:0/56111127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd5815d3b0 msgr2=0x7fcd5809d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:36.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.044+0000 7fcd5e200640 1 --2- 172.21.15.27:0/56111127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd5815d3b0 0x7fcd5809d780 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fcd5001a5a0 tx=0x7fcd50019710 comp rx=0 tx=0).stop 2024-01-15T18:11:36.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.044+0000 7fcd5e200640 1 -- 172.21.15.27:0/56111127 shutdown_connections 2024-01-15T18:11:36.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.044+0000 7fcd5e200640 1 --2- 172.21.15.27:0/56111127 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcd30067280 0x7fcd30069740 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:36.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.044+0000 7fcd5e200640 1 --2- 172.21.15.27:0/56111127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd5815d3b0 0x7fcd5809d780 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:36.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.044+0000 7fcd5e200640 1 --2- 172.21.15.27:0/56111127 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd5806c3f0 0x7fcd581558a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:36.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.044+0000 7fcd5e200640 1 --2- 172.21.15.27:0/56111127 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcd5806b600 0x7fcd58157400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:36.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.044+0000 7fcd5e200640 1 -- 172.21.15.27:0/56111127 >> 172.21.15.27:0/56111127 conn(0x7fcd580d0ed0 msgr2=0x7fcd580c6a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:36.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.044+0000 7fcd5e200640 1 -- 172.21.15.27:0/56111127 shutdown_connections 2024-01-15T18:11:36.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.044+0000 7fcd5e200640 1 -- 172.21.15.27:0/56111127 wait complete. 2024-01-15T18:11:36.071 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone91817 2024-01-15T18:11:36.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.216+0000 7f8f6bc00640 1 Processor -- start 2024-01-15T18:11:36.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.220+0000 7f8f6bc00640 1 -- start start 2024-01-15T18:11:36.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.220+0000 7f8f6bc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8f6415d680 0x7f8f640cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:36.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.220+0000 7f8f6bc00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f8f640ccf80 0x7f8f640cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:36.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.220+0000 7f8f6bc00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f640cb4d0 con 0x7f8f640cfc30 2024-01-15T18:11:36.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.220+0000 7f8f6bc00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f640d0120 con 0x7f8f6415d680 2024-01-15T18:11:36.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.220+0000 7f8f6bc00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f640d0400 con 0x7f8f640ccf80 2024-01-15T18:11:36.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.220+0000 7f8f69200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8f6415d680 0x7f8f640cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:36.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.220+0000 7f8f69200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8f6415d680 0x7f8f640cd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57386/0 (socket says 172.21.15.27:57386) 2024-01-15T18:11:36.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.220+0000 7f8f69200640 1 -- 172.21.15.27:0/828708704 learned_addr learned my addr 172.21.15.27:0/828708704 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:36.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.220+0000 7f8f69c00640 1 --2- 172.21.15.27:0/828708704 >> v2:172.21.15.93:3300/0 conn(0x7f8f640ccf80 0x7f8f640cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:36.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.220+0000 7f8f69200640 1 -- 172.21.15.27:0/828708704 >> v2:172.21.15.93:3300/0 conn(0x7f8f640ccf80 msgr2=0x7f8f640cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:36.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.220+0000 7f8f69200640 1 --2- 172.21.15.27:0/828708704 >> v2:172.21.15.93:3300/0 conn(0x7f8f640ccf80 0x7f8f640cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:36.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.220+0000 7f8f69200640 1 -- 172.21.15.27:0/828708704 >> v1:172.21.15.93:6789/0 conn(0x7f8f640cfc30 legacy=0x7f8f6406b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:36.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.220+0000 7f8f69200640 1 -- 172.21.15.27:0/828708704 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f640cb770 con 0x7f8f6415d680 2024-01-15T18:11:36.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.224+0000 7f8f69200640 1 --2- 172.21.15.27:0/828708704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f6415d680 0x7f8f640cd700 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f8f64067d20 tx=0x7f8f5401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f316f5ec047ea5b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:36.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.224+0000 7f8f63400640 1 -- 172.21.15.27:0/828708704 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f640ce850 con 0x7f8f6415d680 2024-01-15T18:11:36.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.224+0000 7f8f63400640 1 -- 172.21.15.27:0/828708704 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f540aa650 con 0x7f8f6415d680 2024-01-15T18:11:36.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.224+0000 7f8f63400640 1 -- 172.21.15.27:0/828708704 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f540aa9e0 con 0x7f8f6415d680 2024-01-15T18:11:36.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.224+0000 7f8f6bc00640 1 -- 172.21.15.27:0/828708704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f6415d680 msgr2=0x7f8f640cd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:36.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.224+0000 7f8f6bc00640 1 --2- 172.21.15.27:0/828708704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f6415d680 0x7f8f640cd700 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f8f64067d20 tx=0x7f8f5401c4d0 comp rx=0 tx=0).stop 2024-01-15T18:11:36.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.224+0000 7f8f6bc00640 1 -- 172.21.15.27:0/828708704 shutdown_connections 2024-01-15T18:11:36.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.224+0000 7f8f6bc00640 1 --2- 172.21.15.27:0/828708704 >> v2:172.21.15.93:3300/0 conn(0x7f8f640ccf80 0x7f8f640cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:36.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.224+0000 7f8f6bc00640 1 --2- 172.21.15.27:0/828708704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f6415d680 0x7f8f640cd700 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:36.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.224+0000 7f8f6bc00640 1 -- 172.21.15.27:0/828708704 >> 172.21.15.27:0/828708704 conn(0x7f8f640d0e80 msgr2=0x7f8f640d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:36.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.228+0000 7f8f6bc00640 1 -- 172.21.15.27:0/828708704 shutdown_connections 2024-01-15T18:11:36.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.228+0000 7f8f6bc00640 1 -- 172.21.15.27:0/828708704 wait complete. 2024-01-15T18:11:36.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.228+0000 7f8f6bc00640 1 Processor -- start 2024-01-15T18:11:36.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.228+0000 7f8f6bc00640 1 -- start start 2024-01-15T18:11:36.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.228+0000 7f8f6bc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f640ccf80 0x7f8f641516a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:36.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.228+0000 7f8f6bc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f640cfc30 0x7f8f640ae360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:36.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.228+0000 7f8f6bc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8f6415d680 0x7f8f640ae8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:36.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.228+0000 7f8f6bc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f640ca7f0 con 0x7f8f640ccf80 2024-01-15T18:11:36.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.228+0000 7f8f6bc00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f640cac80 con 0x7f8f640cfc30 2024-01-15T18:11:36.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.228+0000 7f8f6bc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f64153510 con 0x7f8f6415d680 2024-01-15T18:11:36.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.228+0000 7f8f69200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f640ccf80 0x7f8f641516a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:36.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.228+0000 7f8f63e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f640cfc30 0x7f8f640ae360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:36.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.228+0000 7f8f69200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f640ccf80 0x7f8f641516a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57402/0 (socket says 172.21.15.27:57402) 2024-01-15T18:11:36.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.228+0000 7f8f69200640 1 -- 172.21.15.27:0/839145876 learned_addr learned my addr 172.21.15.27:0/839145876 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:36.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.228+0000 7f8f69c00640 1 --2- 172.21.15.27:0/839145876 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8f6415d680 0x7f8f640ae8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:36.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.228+0000 7f8f69200640 1 -- 172.21.15.27:0/839145876 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8f6415d680 msgr2=0x7f8f640ae8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:36.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.228+0000 7f8f69200640 1 --2- 172.21.15.27:0/839145876 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8f6415d680 0x7f8f640ae8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:36.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.228+0000 7f8f69200640 1 -- 172.21.15.27:0/839145876 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f640cfc30 msgr2=0x7f8f640ae360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:36.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.228+0000 7f8f69200640 1 --2- 172.21.15.27:0/839145876 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f640cfc30 0x7f8f640ae360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:36.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.228+0000 7f8f69200640 1 -- 172.21.15.27:0/839145876 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f540094d0 con 0x7f8f640ccf80 2024-01-15T18:11:36.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.228+0000 7f8f63e00640 1 --2- 172.21.15.27:0/839145876 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f640cfc30 0x7f8f640ae360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:11:36.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.228+0000 7f8f69200640 1 --2- 172.21.15.27:0/839145876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f640ccf80 0x7f8f641516a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f8f540190e0 tx=0x7f8f5401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:36.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.232+0000 7f8f60c00640 1 -- 172.21.15.27:0/839145876 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f540b3600 con 0x7f8f640ccf80 2024-01-15T18:11:36.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.232+0000 7f8f60c00640 1 -- 172.21.15.27:0/839145876 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f540b3760 con 0x7f8f640ccf80 2024-01-15T18:11:36.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.232+0000 7f8f60c00640 1 -- 172.21.15.27:0/839145876 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f540b3b10 con 0x7f8f640ccf80 2024-01-15T18:11:36.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.232+0000 7f8f6bc00640 1 -- 172.21.15.27:0/839145876 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f640ca1a0 con 0x7f8f640ccf80 2024-01-15T18:11:36.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.232+0000 7f8f6bc00640 1 -- 172.21.15.27:0/839145876 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f64153210 con 0x7f8f640ccf80 2024-01-15T18:11:36.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.232+0000 7f8f6bc00640 1 -- 172.21.15.27:0/839145876 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f6404e850 con 0x7f8f640ccf80 2024-01-15T18:11:36.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.232+0000 7f8f60c00640 1 -- 172.21.15.27:0/839145876 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 87027+0+0 (secure 0 0 0) 0x7f8f540b3c70 con 0x7f8f640ccf80 2024-01-15T18:11:36.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.236+0000 7f8f60c00640 1 --2- 172.21.15.27:0/839145876 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8f3c0672b0 0x7f8f3c069770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:36.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.236+0000 7f8f60c00640 1 -- 172.21.15.27:0/839145876 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f8f5411d4d0 con 0x7f8f640ccf80 2024-01-15T18:11:36.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.236+0000 7f8f63e00640 1 --2- 172.21.15.27:0/839145876 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8f3c0672b0 0x7f8f3c069770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:36.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.236+0000 7f8f63e00640 1 --2- 172.21.15.27:0/839145876 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8f3c0672b0 0x7f8f3c069770 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8f64160410 tx=0x7f8f4c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:36.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.240+0000 7f8f60c00640 1 -- 172.21.15.27:0/839145876 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8f540ec010 con 0x7f8f640ccf80 2024-01-15T18:11:36.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.420+0000 7f8f6bc00640 1 -- 172.21.15.27:0/839145876 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone91817", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f640ca930 con 0x7f8f3c0672b0 2024-01-15T18:11:36.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.440+0000 7f8f60c00640 1 -- 172.21.15.27:0/839145876 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8f4c001900 con 0x7f8f3c0672b0 2024-01-15T18:11:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.440+0000 7f8f6bc00640 1 -- 172.21.15.27:0/839145876 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8f3c0672b0 msgr2=0x7f8f3c069770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.440+0000 7f8f6bc00640 1 --2- 172.21.15.27:0/839145876 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8f3c0672b0 0x7f8f3c069770 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8f64160410 tx=0x7f8f4c009040 comp rx=0 tx=0).stop 2024-01-15T18:11:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.440+0000 7f8f6bc00640 1 -- 172.21.15.27:0/839145876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f640ccf80 msgr2=0x7f8f641516a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.440+0000 7f8f6bc00640 1 --2- 172.21.15.27:0/839145876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f640ccf80 0x7f8f641516a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f8f540190e0 tx=0x7f8f5401ad10 comp rx=0 tx=0).stop 2024-01-15T18:11:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.440+0000 7f8f6bc00640 1 -- 172.21.15.27:0/839145876 shutdown_connections 2024-01-15T18:11:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.440+0000 7f8f6bc00640 1 --2- 172.21.15.27:0/839145876 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8f3c0672b0 0x7f8f3c069770 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.440+0000 7f8f6bc00640 1 --2- 172.21.15.27:0/839145876 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8f6415d680 0x7f8f640ae8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:36.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.440+0000 7f8f6bc00640 1 --2- 172.21.15.27:0/839145876 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f640cfc30 0x7f8f640ae360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:36.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.440+0000 7f8f6bc00640 1 --2- 172.21.15.27:0/839145876 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f640ccf80 0x7f8f641516a0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:36.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.440+0000 7f8f6bc00640 1 -- 172.21.15.27:0/839145876 >> 172.21.15.27:0/839145876 conn(0x7f8f640d0e80 msgr2=0x7f8f6406e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:36.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.440+0000 7f8f6bc00640 1 -- 172.21.15.27:0/839145876 shutdown_connections 2024-01-15T18:11:36.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.440+0000 7f8f6bc00640 1 -- 172.21.15.27:0/839145876 wait complete. 2024-01-15T18:11:36.465 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone91814 2024-01-15T18:11:36.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.612+0000 7fce73c00640 1 Processor -- start 2024-01-15T18:11:36.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.612+0000 7fce73c00640 1 -- start start 2024-01-15T18:11:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.612+0000 7fce73c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce6c0cf460 0x7fce6c0cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.612+0000 7fce73c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce6c0cfd80 con 0x7fce6c0b2590 2024-01-15T18:11:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.612+0000 7fce73c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce6c0d19a0 con 0x7fce6c15b4b0 2024-01-15T18:11:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.612+0000 7fce73c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c0d1c80 con 0x7fce6c0cf460 2024-01-15T18:11:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.612+0000 7fce71c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fce6c0b2590 0x7fce6c0cc130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53084/0 (socket says 172.21.15.27:53084) 2024-01-15T18:11:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.612+0000 7fce71c00640 1 -- 172.21.15.27:0/1439783143 learned_addr learned my addr 172.21.15.27:0/1439783143 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.612+0000 7fce71200640 1 --2- 172.21.15.27:0/1439783143 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce6c0cf460 0x7fce6c0cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:36.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.612+0000 7fce6b400640 1 -- 172.21.15.27:0/1439783143 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776869123 0 0) 0x7fce6c0cfd80 con 0x7fce6c0b2590 2024-01-15T18:11:36.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.612+0000 7fce6b400640 1 -- 172.21.15.27:0/1439783143 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce4c003530 con 0x7fce6c0b2590 2024-01-15T18:11:36.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.612+0000 7fce6b400640 1 -- 172.21.15.27:0/1439783143 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939233052 0 0) 0x7fce6c0d19a0 con 0x7fce6c15b4b0 2024-01-15T18:11:36.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.612+0000 7fce6b400640 1 -- 172.21.15.27:0/1439783143 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce6c0cfd80 con 0x7fce6c15b4b0 2024-01-15T18:11:36.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.612+0000 7fce71200640 1 -- 172.21.15.27:0/1439783143 >> v1:172.21.15.93:6789/0 conn(0x7fce6c15b4b0 legacy=0x7fce6c0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:36.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.612+0000 7fce71200640 1 -- 172.21.15.27:0/1439783143 >> v1:172.21.15.27:6789/0 conn(0x7fce6c0b2590 legacy=0x7fce6c0cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:36.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.612+0000 7fce71200640 1 -- 172.21.15.27:0/1439783143 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6c0b07c0 con 0x7fce6c0cf460 2024-01-15T18:11:36.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.616+0000 7fce71200640 1 --2- 172.21.15.27:0/1439783143 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce6c0cf460 0x7fce6c0cf840 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fce5c0097f0 tx=0x7fce5c01c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20bc5f245b239148 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:36.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.616+0000 7fce6b400640 1 -- 172.21.15.27:0/1439783143 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c0aa620 con 0x7fce6c0cf460 2024-01-15T18:11:36.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.616+0000 7fce6b400640 1 -- 172.21.15.27:0/1439783143 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5c0aa780 con 0x7fce6c0cf460 2024-01-15T18:11:36.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.616+0000 7fce6b400640 1 -- 172.21.15.27:0/1439783143 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c0aab10 con 0x7fce6c0cf460 2024-01-15T18:11:36.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.616+0000 7fce73c00640 1 -- 172.21.15.27:0/1439783143 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce6c0cf460 msgr2=0x7fce6c0cf840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:36.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.616+0000 7fce73c00640 1 --2- 172.21.15.27:0/1439783143 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce6c0cf460 0x7fce6c0cf840 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fce5c0097f0 tx=0x7fce5c01c4c0 comp rx=0 tx=0).stop 2024-01-15T18:11:36.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.616+0000 7fce73c00640 1 -- 172.21.15.27:0/1439783143 shutdown_connections 2024-01-15T18:11:36.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.616+0000 7fce73c00640 1 --2- 172.21.15.27:0/1439783143 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce6c0cf460 0x7fce6c0cf840 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:36.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.616+0000 7fce73c00640 1 -- 172.21.15.27:0/1439783143 >> 172.21.15.27:0/1439783143 conn(0x7fce6c0d0e80 msgr2=0x7fce6c0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:36.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.616+0000 7fce73c00640 1 -- 172.21.15.27:0/1439783143 shutdown_connections 2024-01-15T18:11:36.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.616+0000 7fce73c00640 1 -- 172.21.15.27:0/1439783143 wait complete. 2024-01-15T18:11:36.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.620+0000 7fce73c00640 1 Processor -- start 2024-01-15T18:11:36.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.620+0000 7fce73c00640 1 -- start start 2024-01-15T18:11:36.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.620+0000 7fce73c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce6c0b2590 0x7fce6c1542d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:36.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.620+0000 7fce73c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce6c0cf460 0x7fce6c0c8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:36.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.620+0000 7fce73c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce6c15b4b0 0x7fce6c156f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:36.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.620+0000 7fce73c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c14ba40 con 0x7fce6c0cf460 2024-01-15T18:11:36.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.620+0000 7fce73c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c0cadf0 con 0x7fce6c15b4b0 2024-01-15T18:11:36.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.620+0000 7fce73c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c0ca1e0 con 0x7fce6c0b2590 2024-01-15T18:11:36.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.620+0000 7fce6be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce6c0cf460 0x7fce6c0c8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:36.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.620+0000 7fce71200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce6c0b2590 0x7fce6c1542d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:36.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.620+0000 7fce6be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce6c0cf460 0x7fce6c0c8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57414/0 (socket says 172.21.15.27:57414) 2024-01-15T18:11:36.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.620+0000 7fce71c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce6c15b4b0 0x7fce6c156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:36.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.620+0000 7fce6be00640 1 -- 172.21.15.27:0/2461621759 learned_addr learned my addr 172.21.15.27:0/2461621759 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:36.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.620+0000 7fce71200640 1 -- 172.21.15.27:0/2461621759 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce6c15b4b0 msgr2=0x7fce6c156f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:36.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.620+0000 7fce71200640 1 --2- 172.21.15.27:0/2461621759 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce6c15b4b0 0x7fce6c156f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:36.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.620+0000 7fce71200640 1 -- 172.21.15.27:0/2461621759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce6c0cf460 msgr2=0x7fce6c0c8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:36.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.620+0000 7fce71200640 1 --2- 172.21.15.27:0/2461621759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce6c0cf460 0x7fce6c0c8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:36.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.620+0000 7fce71200640 1 -- 172.21.15.27:0/2461621759 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce5c009420 con 0x7fce6c0b2590 2024-01-15T18:11:36.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.620+0000 7fce6be00640 1 --2- 172.21.15.27:0/2461621759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce6c0cf460 0x7fce6c0c8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:11:36.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.620+0000 7fce71200640 1 --2- 172.21.15.27:0/2461621759 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce6c0b2590 0x7fce6c1542d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fce6c071d20 tx=0x7fce5c01ad00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:36.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.620+0000 7fce68c00640 1 -- 172.21.15.27:0/2461621759 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c0aadc0 con 0x7fce6c0b2590 2024-01-15T18:11:36.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.620+0000 7fce68c00640 1 -- 172.21.15.27:0/2461621759 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5c0be410 con 0x7fce6c0b2590 2024-01-15T18:11:36.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.620+0000 7fce68c00640 1 -- 172.21.15.27:0/2461621759 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c0be7c0 con 0x7fce6c0b2590 2024-01-15T18:11:36.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.620+0000 7fce73c00640 1 -- 172.21.15.27:0/2461621759 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce6c0ca4c0 con 0x7fce6c0b2590 2024-01-15T18:11:36.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.624+0000 7fce73c00640 1 -- 172.21.15.27:0/2461621759 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce6c0caa80 con 0x7fce6c0b2590 2024-01-15T18:11:36.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.624+0000 7fce73c00640 1 -- 172.21.15.27:0/2461621759 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce6c04e850 con 0x7fce6c0b2590 2024-01-15T18:11:36.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.624+0000 7fce68c00640 1 -- 172.21.15.27:0/2461621759 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 19) v1 ==== 87027+0+0 (secure 0 0 0) 0x7fce5c0be920 con 0x7fce6c0b2590 2024-01-15T18:11:36.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.624+0000 7fce68c00640 1 --2- 172.21.15.27:0/2461621759 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce44067280 0x7fce44069740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:36.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.624+0000 7fce68c00640 1 -- 172.21.15.27:0/2461621759 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fce5c11d150 con 0x7fce6c0b2590 2024-01-15T18:11:36.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.624+0000 7fce6be00640 1 --2- 172.21.15.27:0/2461621759 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce44067280 0x7fce44069740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:36.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.628+0000 7fce6be00640 1 --2- 172.21.15.27:0/2461621759 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce44067280 0x7fce44069740 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fce6c1553b0 tx=0x7fce54009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:36.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.628+0000 7fce68c00640 1 -- 172.21.15.27:0/2461621759 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fce5c0ebc90 con 0x7fce6c0b2590 2024-01-15T18:11:36.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.808+0000 7fce73c00640 1 -- 172.21.15.27:0/2461621759 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone91814", "target": ["mon-mgr", ""]}) v1 -- 0x7fce6c0cc840 con 0x7fce44067280 2024-01-15T18:11:36.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.828+0000 7fce68c00640 1 -- 172.21.15.27:0/2461621759 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fce54001910 con 0x7fce44067280 2024-01-15T18:11:36.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.832+0000 7fce73c00640 1 -- 172.21.15.27:0/2461621759 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce44067280 msgr2=0x7fce44069740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:36.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.832+0000 7fce73c00640 1 --2- 172.21.15.27:0/2461621759 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce44067280 0x7fce44069740 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fce6c1553b0 tx=0x7fce54009040 comp rx=0 tx=0).stop 2024-01-15T18:11:36.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.832+0000 7fce73c00640 1 -- 172.21.15.27:0/2461621759 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce6c0b2590 msgr2=0x7fce6c1542d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:36.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.832+0000 7fce73c00640 1 --2- 172.21.15.27:0/2461621759 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce6c0b2590 0x7fce6c1542d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fce6c071d20 tx=0x7fce5c01ad00 comp rx=0 tx=0).stop 2024-01-15T18:11:36.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.832+0000 7fce73c00640 1 -- 172.21.15.27:0/2461621759 shutdown_connections 2024-01-15T18:11:36.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.832+0000 7fce73c00640 1 --2- 172.21.15.27:0/2461621759 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce44067280 0x7fce44069740 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:36.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.832+0000 7fce73c00640 1 --2- 172.21.15.27:0/2461621759 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce6c15b4b0 0x7fce6c156f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:36.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.832+0000 7fce73c00640 1 --2- 172.21.15.27:0/2461621759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce6c0cf460 0x7fce6c0c8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:36.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.832+0000 7fce73c00640 1 --2- 172.21.15.27:0/2461621759 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce6c0b2590 0x7fce6c1542d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:36.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.832+0000 7fce73c00640 1 -- 172.21.15.27:0/2461621759 >> 172.21.15.27:0/2461621759 conn(0x7fce6c0d0e80 msgr2=0x7fce6c06bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:36.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.832+0000 7fce73c00640 1 -- 172.21.15.27:0/2461621759 shutdown_connections 2024-01-15T18:11:36.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:36.832+0000 7fce73c00640 1 -- 172.21.15.27:0/2461621759 wait complete. 2024-01-15T18:11:36.855 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone91813 2024-01-15T18:11:37.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.000+0000 7f492fc00640 1 Processor -- start 2024-01-15T18:11:37.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.000+0000 7f492fc00640 1 -- start start 2024-01-15T18:11:37.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.000+0000 7f492fc00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f492815b350 0x7f49280cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:37.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.000+0000 7f492fc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4928099a10 0x7f4928099e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:37.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.000+0000 7f492fc00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492809a3a0 con 0x7f492809a5b0 2024-01-15T18:11:37.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.000+0000 7f492fc00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49280cfec0 con 0x7f492815b350 2024-01-15T18:11:37.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.000+0000 7f492fc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49280ccd50 con 0x7f4928099a10 2024-01-15T18:11:37.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.000+0000 7f4927e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f492809a5b0 0x7f49280cc500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53092/0 (socket says 172.21.15.27:53092) 2024-01-15T18:11:37.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.000+0000 7f4927e00640 1 -- 172.21.15.27:0/585227901 learned_addr learned my addr 172.21.15.27:0/585227901 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:37.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.000+0000 7f492d200640 1 --2- 172.21.15.27:0/585227901 >> v2:172.21.15.93:3300/0 conn(0x7f492815b350 0x7f49280cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:37.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.000+0000 7f492dc00640 1 --2- 172.21.15.27:0/585227901 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4928099a10 0x7f4928099e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:37.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.000+0000 7f4927400640 1 -- 172.21.15.27:0/585227901 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249560734 0 0) 0x7f4910001120 con 0x7f492809a5b0 2024-01-15T18:11:37.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.000+0000 7f4927400640 1 -- 172.21.15.27:0/585227901 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4908003530 con 0x7f492809a5b0 2024-01-15T18:11:37.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.000+0000 7f492dc00640 1 -- 172.21.15.27:0/585227901 >> v2:172.21.15.93:3300/0 conn(0x7f492815b350 msgr2=0x7f49280cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:37.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.000+0000 7f492dc00640 1 --2- 172.21.15.27:0/585227901 >> v2:172.21.15.93:3300/0 conn(0x7f492815b350 0x7f49280cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.000+0000 7f492dc00640 1 -- 172.21.15.27:0/585227901 >> v1:172.21.15.27:6789/0 conn(0x7f492809a5b0 legacy=0x7f49280cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:37.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.000+0000 7f4927400640 1 -- 172.21.15.27:0/585227901 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3780647919 0 0) 0x7f4908003530 con 0x7f492809a5b0 2024-01-15T18:11:37.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.000+0000 7f492dc00640 1 -- 172.21.15.27:0/585227901 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49280cebc0 con 0x7f4928099a10 2024-01-15T18:11:37.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.000+0000 7f492d200640 1 --2- 172.21.15.27:0/585227901 >> v2:172.21.15.93:3300/0 conn(0x7f492815b350 0x7f49280cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:11:37.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.004+0000 7f492dc00640 1 --2- 172.21.15.27:0/585227901 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4928099a10 0x7f4928099e60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4920009d90 tx=0x7f492001c7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b2d6d93fa35bc7e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:37.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.004+0000 7f4927400640 1 -- 172.21.15.27:0/585227901 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4910001810 con 0x7f4928099a10 2024-01-15T18:11:37.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.004+0000 7f4927400640 1 -- 172.21.15.27:0/585227901 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49200a55b0 con 0x7f4928099a10 2024-01-15T18:11:37.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.004+0000 7f4927400640 1 -- 172.21.15.27:0/585227901 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49200a5940 con 0x7f4928099a10 2024-01-15T18:11:37.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.004+0000 7f492fc00640 1 -- 172.21.15.27:0/585227901 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4928099a10 msgr2=0x7f4928099e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:37.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.004+0000 7f492fc00640 1 --2- 172.21.15.27:0/585227901 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4928099a10 0x7f4928099e60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4920009d90 tx=0x7f492001c7b0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.004+0000 7f492fc00640 1 -- 172.21.15.27:0/585227901 shutdown_connections 2024-01-15T18:11:37.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.004+0000 7f492fc00640 1 --2- 172.21.15.27:0/585227901 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4928099a10 0x7f4928099e60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.004+0000 7f492fc00640 1 --2- 172.21.15.27:0/585227901 >> v2:172.21.15.93:3300/0 conn(0x7f492815b350 0x7f49280cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.004+0000 7f492fc00640 1 -- 172.21.15.27:0/585227901 >> 172.21.15.27:0/585227901 conn(0x7f49280b2a80 msgr2=0x7f49280b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:37.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.008+0000 7f492fc00640 1 -- 172.21.15.27:0/585227901 shutdown_connections 2024-01-15T18:11:37.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.008+0000 7f492fc00640 1 -- 172.21.15.27:0/585227901 wait complete. 2024-01-15T18:11:37.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.008+0000 7f492fc00640 1 Processor -- start 2024-01-15T18:11:37.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.008+0000 7f492fc00640 1 -- start start 2024-01-15T18:11:37.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.008+0000 7f492fc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4928099a10 0x7f492806c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:37.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.008+0000 7f492fc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f492809a5b0 0x7f492815c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:37.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.008+0000 7f492fc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f492815b350 0x7f492815c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:37.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.008+0000 7f492fc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492815b730 con 0x7f492815b350 2024-01-15T18:11:37.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.008+0000 7f492fc00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49280c68d0 con 0x7f4928099a10 2024-01-15T18:11:37.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.008+0000 7f492fc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49280c6bb0 con 0x7f492809a5b0 2024-01-15T18:11:37.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.008+0000 7f492dc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f492815b350 0x7f492815c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:37.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.008+0000 7f492dc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f492815b350 0x7f492815c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57416/0 (socket says 172.21.15.27:57416) 2024-01-15T18:11:37.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.008+0000 7f492dc00640 1 -- 172.21.15.27:0/389590831 learned_addr learned my addr 172.21.15.27:0/389590831 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:37.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.008+0000 7f492d200640 1 --2- 172.21.15.27:0/389590831 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4928099a10 0x7f492806c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:37.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.008+0000 7f4927e00640 1 --2- 172.21.15.27:0/389590831 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f492809a5b0 0x7f492815c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:37.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.008+0000 7f492dc00640 1 -- 172.21.15.27:0/389590831 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f492809a5b0 msgr2=0x7f492815c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:37.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.008+0000 7f492dc00640 1 --2- 172.21.15.27:0/389590831 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f492809a5b0 0x7f492815c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.008+0000 7f492dc00640 1 -- 172.21.15.27:0/389590831 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4928099a10 msgr2=0x7f492806c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:37.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.008+0000 7f492dc00640 1 --2- 172.21.15.27:0/389590831 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4928099a10 0x7f492806c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.008+0000 7f492dc00640 1 -- 172.21.15.27:0/389590831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4918009420 con 0x7f492815b350 2024-01-15T18:11:37.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.008+0000 7f492d200640 1 --2- 172.21.15.27:0/389590831 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4928099a10 0x7f492806c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:11:37.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.008+0000 7f492dc00640 1 --2- 172.21.15.27:0/389590831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f492815b350 0x7f492815c6c0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f49200b0040 tx=0x7f4920019550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:37.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.012+0000 7f4924c00640 1 -- 172.21.15.27:0/389590831 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49200b9650 con 0x7f492815b350 2024-01-15T18:11:37.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.012+0000 7f4924c00640 1 -- 172.21.15.27:0/389590831 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49200b97b0 con 0x7f492815b350 2024-01-15T18:11:37.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.012+0000 7f492fc00640 1 -- 172.21.15.27:0/389590831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4920009a70 con 0x7f492815b350 2024-01-15T18:11:37.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.012+0000 7f4924c00640 1 -- 172.21.15.27:0/389590831 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49200b9b60 con 0x7f492815b350 2024-01-15T18:11:37.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.012+0000 7f492fc00640 1 -- 172.21.15.27:0/389590831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f492815bc60 con 0x7f492815b350 2024-01-15T18:11:37.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.012+0000 7f492fc00640 1 -- 172.21.15.27:0/389590831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49280d0720 con 0x7f492815b350 2024-01-15T18:11:37.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.012+0000 7f4924c00640 1 -- 172.21.15.27:0/389590831 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 87027+0+0 (secure 0 0 0) 0x7f49200b9cc0 con 0x7f492815b350 2024-01-15T18:11:37.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.012+0000 7f4924c00640 1 --2- 172.21.15.27:0/389590831 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4900067280 0x7f4900069740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:37.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.012+0000 7f4924c00640 1 -- 172.21.15.27:0/389590831 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f49201206a0 con 0x7f492815b350 2024-01-15T18:11:37.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.012+0000 7f492d200640 1 --2- 172.21.15.27:0/389590831 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4900067280 0x7f4900069740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:37.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.016+0000 7f492d200640 1 --2- 172.21.15.27:0/389590831 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4900067280 0x7f4900069740 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f4918011310 tx=0x7f4918009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:37.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.016+0000 7f4924c00640 1 -- 172.21.15.27:0/389590831 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f49200ecd90 con 0x7f492815b350 2024-01-15T18:11:37.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.200+0000 7f492fc00640 1 -- 172.21.15.27:0/389590831 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone91813", "target": ["mon-mgr", ""]}) v1 -- 0x7f492804e850 con 0x7f4900067280 2024-01-15T18:11:37.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.216+0000 7f4924c00640 1 -- 172.21.15.27:0/389590831 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f492804e850 con 0x7f4900067280 2024-01-15T18:11:37.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.216+0000 7f492fc00640 1 -- 172.21.15.27:0/389590831 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4900067280 msgr2=0x7f4900069740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:37.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.216+0000 7f492fc00640 1 --2- 172.21.15.27:0/389590831 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4900067280 0x7f4900069740 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f4918011310 tx=0x7f4918009190 comp rx=0 tx=0).stop 2024-01-15T18:11:37.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.216+0000 7f492fc00640 1 -- 172.21.15.27:0/389590831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f492815b350 msgr2=0x7f492815c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:37.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.216+0000 7f492fc00640 1 --2- 172.21.15.27:0/389590831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f492815b350 0x7f492815c6c0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f49200b0040 tx=0x7f4920019550 comp rx=0 tx=0).stop 2024-01-15T18:11:37.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.216+0000 7f492fc00640 1 -- 172.21.15.27:0/389590831 shutdown_connections 2024-01-15T18:11:37.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.216+0000 7f492fc00640 1 --2- 172.21.15.27:0/389590831 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4900067280 0x7f4900069740 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.216+0000 7f492fc00640 1 --2- 172.21.15.27:0/389590831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f492815b350 0x7f492815c6c0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.216+0000 7f492fc00640 1 --2- 172.21.15.27:0/389590831 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f492809a5b0 0x7f492815c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.216+0000 7f492fc00640 1 --2- 172.21.15.27:0/389590831 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4928099a10 0x7f492806c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.216+0000 7f492fc00640 1 -- 172.21.15.27:0/389590831 >> 172.21.15.27:0/389590831 conn(0x7f49280b2a80 msgr2=0x7f49280cc500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:37.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.216+0000 7f492fc00640 1 -- 172.21.15.27:0/389590831 shutdown_connections 2024-01-15T18:11:37.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.216+0000 7f492fc00640 1 -- 172.21.15.27:0/389590831 wait complete. 2024-01-15T18:11:37.243 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone91812 2024-01-15T18:11:37.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.388+0000 7fc735600640 1 Processor -- start 2024-01-15T18:11:37.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.388+0000 7fc735600640 1 -- start start 2024-01-15T18:11:37.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.388+0000 7fc735600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc73015f550 0x7fc7300cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:37.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.388+0000 7fc735600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7300cf110 0x7fc7300701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:37.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.388+0000 7fc735600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7300b1cb0 con 0x7fc730158900 2024-01-15T18:11:37.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.388+0000 7fc735600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc73015fa00 con 0x7fc7300cf110 2024-01-15T18:11:37.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.388+0000 7fc735600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7300cf540 con 0x7fc73015f550 2024-01-15T18:11:37.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.388+0000 7fc72f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7300cf110 0x7fc7300701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:37.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.388+0000 7fc72f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7300cf110 0x7fc7300701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57424/0 (socket says 172.21.15.27:57424) 2024-01-15T18:11:37.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.388+0000 7fc72f400640 1 -- 172.21.15.27:0/3345308336 learned_addr learned my addr 172.21.15.27:0/3345308336 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:37.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.392+0000 7fc72e000640 1 --2- 172.21.15.27:0/3345308336 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc73015f550 0x7fc7300cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:37.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.392+0000 7fc72f400640 1 -- 172.21.15.27:0/3345308336 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc73015f550 msgr2=0x7fc7300cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:37.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.392+0000 7fc72f400640 1 --2- 172.21.15.27:0/3345308336 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc73015f550 0x7fc7300cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.392+0000 7fc72f400640 1 -- 172.21.15.27:0/3345308336 >> v1:172.21.15.93:6789/0 conn(0x7fc730158900 legacy=0x7fc73015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:37.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.392+0000 7fc72f400640 1 -- 172.21.15.27:0/3345308336 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7300b1f00 con 0x7fc7300cf110 2024-01-15T18:11:37.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.392+0000 7fc72f400640 1 --2- 172.21.15.27:0/3345308336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7300cf110 0x7fc7300701e0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fc728009e50 tx=0x7fc72801c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eaff97c0b894508 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:37.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.392+0000 7fc72d600640 1 -- 172.21.15.27:0/3345308336 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc730070f30 con 0x7fc7300cf110 2024-01-15T18:11:37.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.392+0000 7fc72d600640 1 -- 172.21.15.27:0/3345308336 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7280a58a0 con 0x7fc7300cf110 2024-01-15T18:11:37.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.392+0000 7fc72d600640 1 -- 172.21.15.27:0/3345308336 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7280a5c30 con 0x7fc7300cf110 2024-01-15T18:11:37.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.396+0000 7fc735600640 1 -- 172.21.15.27:0/3345308336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7300cf110 msgr2=0x7fc7300701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:37.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.396+0000 7fc735600640 1 --2- 172.21.15.27:0/3345308336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7300cf110 0x7fc7300701e0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fc728009e50 tx=0x7fc72801c9e0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.396+0000 7fc735600640 1 -- 172.21.15.27:0/3345308336 shutdown_connections 2024-01-15T18:11:37.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.396+0000 7fc735600640 1 --2- 172.21.15.27:0/3345308336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7300cf110 0x7fc7300701e0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.396+0000 7fc735600640 1 --2- 172.21.15.27:0/3345308336 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc73015f550 0x7fc7300cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.396+0000 7fc735600640 1 -- 172.21.15.27:0/3345308336 >> 172.21.15.27:0/3345308336 conn(0x7fc7300cb700 msgr2=0x7fc7300cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:37.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.396+0000 7fc735600640 1 -- 172.21.15.27:0/3345308336 shutdown_connections 2024-01-15T18:11:37.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.396+0000 7fc735600640 1 -- 172.21.15.27:0/3345308336 wait complete. 2024-01-15T18:11:37.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.396+0000 7fc735600640 1 Processor -- start 2024-01-15T18:11:37.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.396+0000 7fc735600640 1 -- start start 2024-01-15T18:11:37.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.396+0000 7fc735600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7300cf110 0x7fc7301518f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.396+0000 7fc735600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc730158900 0x7fc73014cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.396+0000 7fc735600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc73015f550 0x7fc730152f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.396+0000 7fc735600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73014d1a0 con 0x7fc730158900 2024-01-15T18:11:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.396+0000 7fc735600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc730151ed0 con 0x7fc73015f550 2024-01-15T18:11:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.396+0000 7fc735600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7301521b0 con 0x7fc7300cf110 2024-01-15T18:11:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.396+0000 7fc72e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc730158900 0x7fc73014cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.396+0000 7fc72e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc730158900 0x7fc73014cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57426/0 (socket says 172.21.15.27:57426) 2024-01-15T18:11:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.396+0000 7fc72e000640 1 -- 172.21.15.27:0/2718751532 learned_addr learned my addr 172.21.15.27:0/2718751532 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:37.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.396+0000 7fc72f400640 1 --2- 172.21.15.27:0/2718751532 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc73015f550 0x7fc730152f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:37.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.396+0000 7fc72ea00640 1 --2- 172.21.15.27:0/2718751532 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7300cf110 0x7fc7301518f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:37.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.400+0000 7fc72e000640 1 -- 172.21.15.27:0/2718751532 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7300cf110 msgr2=0x7fc7301518f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:37.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.400+0000 7fc72e000640 1 --2- 172.21.15.27:0/2718751532 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7300cf110 0x7fc7301518f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.400+0000 7fc72e000640 1 -- 172.21.15.27:0/2718751532 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc73015f550 msgr2=0x7fc730152f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:37.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.400+0000 7fc72e000640 1 --2- 172.21.15.27:0/2718751532 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc73015f550 0x7fc730152f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.400+0000 7fc72e000640 1 -- 172.21.15.27:0/2718751532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc728009b30 con 0x7fc730158900 2024-01-15T18:11:37.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.400+0000 7fc72e000640 1 --2- 172.21.15.27:0/2718751532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc730158900 0x7fc73014cc60 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fc71800e930 tx=0x7fc71800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:37.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.400+0000 7fc726a00640 1 -- 172.21.15.27:0/2718751532 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc718009690 con 0x7fc730158900 2024-01-15T18:11:37.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.400+0000 7fc726a00640 1 -- 172.21.15.27:0/2718751532 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71800ca00 con 0x7fc730158900 2024-01-15T18:11:37.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.400+0000 7fc735600640 1 -- 172.21.15.27:0/2718751532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc730152490 con 0x7fc730158900 2024-01-15T18:11:37.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.400+0000 7fc726a00640 1 -- 172.21.15.27:0/2718751532 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71800cd70 con 0x7fc730158900 2024-01-15T18:11:37.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.400+0000 7fc735600640 1 -- 172.21.15.27:0/2718751532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc730152a50 con 0x7fc730158900 2024-01-15T18:11:37.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.400+0000 7fc735600640 1 -- 172.21.15.27:0/2718751532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc73004e850 con 0x7fc730158900 2024-01-15T18:11:37.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.400+0000 7fc726a00640 1 -- 172.21.15.27:0/2718751532 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 87027+0+0 (secure 0 0 0) 0x7fc718013470 con 0x7fc730158900 2024-01-15T18:11:37.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.400+0000 7fc726a00640 1 --2- 172.21.15.27:0/2718751532 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc7080672b0 0x7fc708069770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:37.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.400+0000 7fc726a00640 1 -- 172.21.15.27:0/2718751532 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc718083090 con 0x7fc730158900 2024-01-15T18:11:37.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.404+0000 7fc72ea00640 1 --2- 172.21.15.27:0/2718751532 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc7080672b0 0x7fc708069770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:37.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.404+0000 7fc72ea00640 1 --2- 172.21.15.27:0/2718751532 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc7080672b0 0x7fc708069770 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc720007fd0 tx=0x7fc720009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:37.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.404+0000 7fc726a00640 1 -- 172.21.15.27:0/2718751532 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc718051c70 con 0x7fc730158900 2024-01-15T18:11:37.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.592+0000 7fc735600640 1 -- 172.21.15.27:0/2718751532 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone91812", "target": ["mon-mgr", ""]}) v1 -- 0x7fc73015f030 con 0x7fc7080672b0 2024-01-15T18:11:37.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.612+0000 7fc726a00640 1 -- 172.21.15.27:0/2718751532 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc720001450 con 0x7fc7080672b0 2024-01-15T18:11:37.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.612+0000 7fc735600640 1 -- 172.21.15.27:0/2718751532 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc7080672b0 msgr2=0x7fc708069770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:37.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.612+0000 7fc735600640 1 --2- 172.21.15.27:0/2718751532 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc7080672b0 0x7fc708069770 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc720007fd0 tx=0x7fc720009040 comp rx=0 tx=0).stop 2024-01-15T18:11:37.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.612+0000 7fc735600640 1 -- 172.21.15.27:0/2718751532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc730158900 msgr2=0x7fc73014cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:37.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.612+0000 7fc735600640 1 --2- 172.21.15.27:0/2718751532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc730158900 0x7fc73014cc60 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fc71800e930 tx=0x7fc71800edf0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.612+0000 7fc735600640 1 -- 172.21.15.27:0/2718751532 shutdown_connections 2024-01-15T18:11:37.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.612+0000 7fc735600640 1 --2- 172.21.15.27:0/2718751532 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc7080672b0 0x7fc708069770 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.612+0000 7fc735600640 1 --2- 172.21.15.27:0/2718751532 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc73015f550 0x7fc730152f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.612+0000 7fc735600640 1 --2- 172.21.15.27:0/2718751532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc730158900 0x7fc73014cc60 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.612+0000 7fc735600640 1 --2- 172.21.15.27:0/2718751532 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7300cf110 0x7fc7301518f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.612+0000 7fc735600640 1 -- 172.21.15.27:0/2718751532 >> 172.21.15.27:0/2718751532 conn(0x7fc7300cb700 msgr2=0x7fc7300cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:37.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.612+0000 7fc735600640 1 -- 172.21.15.27:0/2718751532 shutdown_connections 2024-01-15T18:11:37.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.612+0000 7fc735600640 1 -- 172.21.15.27:0/2718751532 wait complete. 2024-01-15T18:11:37.636 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone91815 2024-01-15T18:11:37.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.780+0000 7f801c600640 1 Processor -- start 2024-01-15T18:11:37.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.780+0000 7f801c600640 1 -- start start 2024-01-15T18:11:37.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.780+0000 7f801c600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f80140cfc30 0x7f80140cc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:37.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.780+0000 7f801c600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80140cb4d0 con 0x7f801415d680 2024-01-15T18:11:37.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.780+0000 7f801c600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80140d0120 con 0x7f80140cde10 2024-01-15T18:11:37.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.780+0000 7f801c600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80140d0400 con 0x7f80140cfc30 2024-01-15T18:11:37.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.780+0000 7f8019c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f801415d680 0x7f80140cd700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53106/0 (socket says 172.21.15.27:53106) 2024-01-15T18:11:37.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.780+0000 7f8019c00640 1 -- 172.21.15.27:0/873759298 learned_addr learned my addr 172.21.15.27:0/873759298 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:37.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.780+0000 7f801a600640 1 --2- 172.21.15.27:0/873759298 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f80140cfc30 0x7f80140cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:37.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.780+0000 7f8013e00640 1 -- 172.21.15.27:0/873759298 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3832957601 0 0) 0x7f80140cb4d0 con 0x7f801415d680 2024-01-15T18:11:37.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.780+0000 7f8013e00640 1 -- 172.21.15.27:0/873759298 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff4003530 con 0x7f801415d680 2024-01-15T18:11:37.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.780+0000 7f8013e00640 1 -- 172.21.15.27:0/873759298 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452265354 0 0) 0x7f80140d0120 con 0x7f80140cde10 2024-01-15T18:11:37.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.780+0000 7f8013e00640 1 -- 172.21.15.27:0/873759298 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80140cb4d0 con 0x7f80140cde10 2024-01-15T18:11:37.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.780+0000 7f8013e00640 1 -- 172.21.15.27:0/873759298 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1167146826 0 0) 0x7f7ff4003530 con 0x7f801415d680 2024-01-15T18:11:37.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.780+0000 7f8013e00640 1 -- 172.21.15.27:0/873759298 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f80140cfc30 msgr2=0x7f80140cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:37.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.780+0000 7f8013e00640 1 --2- 172.21.15.27:0/873759298 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f80140cfc30 0x7f80140cc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.780+0000 7f8013e00640 1 -- 172.21.15.27:0/873759298 >> v1:172.21.15.93:6789/0 conn(0x7f80140cde10 legacy=0x7f80140cbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.780+0000 7f8013e00640 1 -- 172.21.15.27:0/873759298 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80140cb770 con 0x7f801415d680 2024-01-15T18:11:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.780+0000 7f8013e00640 1 -- 172.21.15.27:0/873759298 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f8004001680 con 0x7f801415d680 2024-01-15T18:11:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.780+0000 7f8013e00640 1 -- 172.21.15.27:0/873759298 >> v1:172.21.15.27:6789/0 conn(0x7f801415d680 legacy=0x7f80140cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.784+0000 7f8013e00640 1 --2- 172.21.15.27:0/873759298 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ff4003bb0 0x7f7ff4003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.784+0000 7f8013e00640 1 --2- 172.21.15.27:0/873759298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ff40047d0 0x7f7ff4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.784+0000 7f8013e00640 1 --2- 172.21.15.27:0/873759298 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f80140cfc30 0x7f7ff40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.784+0000 7f8013e00640 1 -- 172.21.15.27:0/873759298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80140d0400 con 0x7f7ff40047d0 2024-01-15T18:11:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.784+0000 7f8013e00640 1 -- 172.21.15.27:0/873759298 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff400bac0 con 0x7f7ff4003bb0 2024-01-15T18:11:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.784+0000 7f8013e00640 1 -- 172.21.15.27:0/873759298 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff400bc80 con 0x7f80140cfc30 2024-01-15T18:11:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.784+0000 7f8019200640 1 --2- 172.21.15.27:0/873759298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ff40047d0 0x7f7ff4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.784+0000 7f8019c00640 1 --2- 172.21.15.27:0/873759298 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ff4003bb0 0x7f7ff4003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.784+0000 7f801a600640 1 --2- 172.21.15.27:0/873759298 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f80140cfc30 0x7f7ff40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.784+0000 7f8019200640 1 -- 172.21.15.27:0/873759298 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f80140cfc30 msgr2=0x7f7ff40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.784+0000 7f8019200640 1 --2- 172.21.15.27:0/873759298 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f80140cfc30 0x7f7ff40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.784+0000 7f8019200640 1 -- 172.21.15.27:0/873759298 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ff4003bb0 msgr2=0x7f7ff4003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:37.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.784+0000 7f8019200640 1 --2- 172.21.15.27:0/873759298 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ff4003bb0 0x7f7ff4003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.784+0000 7f8019200640 1 -- 172.21.15.27:0/873759298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff4003530 con 0x7f7ff40047d0 2024-01-15T18:11:37.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.784+0000 7f8019200640 1 --2- 172.21.15.27:0/873759298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ff40047d0 0x7f7ff4004c50 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc0018d0 tx=0x7f7ffc01ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa12738b1a7842d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:37.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.788+0000 7f8013e00640 1 -- 172.21.15.27:0/873759298 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc0052d0 con 0x7f7ff40047d0 2024-01-15T18:11:37.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.788+0000 7f8013e00640 1 -- 172.21.15.27:0/873759298 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ffc005450 con 0x7f7ff40047d0 2024-01-15T18:11:37.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.788+0000 7f8013e00640 1 -- 172.21.15.27:0/873759298 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc0a8930 con 0x7f7ff40047d0 2024-01-15T18:11:37.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.788+0000 7f801c600640 1 -- 172.21.15.27:0/873759298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ff40047d0 msgr2=0x7f7ff4004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:37.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.788+0000 7f801c600640 1 --2- 172.21.15.27:0/873759298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ff40047d0 0x7f7ff4004c50 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc0018d0 tx=0x7f7ffc01ecd0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.788+0000 7f801a600640 1 -- 172.21.15.27:0/873759298 reap_dead start 2024-01-15T18:11:37.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.788+0000 7f801c600640 1 -- 172.21.15.27:0/873759298 shutdown_connections 2024-01-15T18:11:37.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.788+0000 7f801c600640 1 -- 172.21.15.27:0/873759298 >> 172.21.15.27:0/873759298 conn(0x7f80140d0e80 msgr2=0x7f80140d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:37.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.788+0000 7f801c600640 1 -- 172.21.15.27:0/873759298 shutdown_connections 2024-01-15T18:11:37.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.788+0000 7f801c600640 1 -- 172.21.15.27:0/873759298 wait complete. 2024-01-15T18:11:37.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.788+0000 7f801c600640 1 Processor -- start 2024-01-15T18:11:37.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.788+0000 7f801c600640 1 -- start start 2024-01-15T18:11:37.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.788+0000 7f801c600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ff40047d0 0x7f80141f2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.792+0000 7f801c600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80140c8070 0x7f80140c84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.792+0000 7f801c600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f80140ca0d0 0x7f80140ca550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.792+0000 7f801c600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80140cae90 con 0x7f80140c8070 2024-01-15T18:11:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.792+0000 7f801c600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80140c8a30 con 0x7f7ff40047d0 2024-01-15T18:11:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.792+0000 7f801c600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80140c5e00 con 0x7f80140ca0d0 2024-01-15T18:11:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.792+0000 7f8019200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80140c8070 0x7f80140c84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.792+0000 7f8019c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ff40047d0 0x7f80141f2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.792+0000 7f8019200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80140c8070 0x7f80140c84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57446/0 (socket says 172.21.15.27:57446) 2024-01-15T18:11:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.792+0000 7f8019200640 1 -- 172.21.15.27:0/3429442916 learned_addr learned my addr 172.21.15.27:0/3429442916 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.792+0000 7f801a600640 1 --2- 172.21.15.27:0/3429442916 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f80140ca0d0 0x7f80140ca550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.792+0000 7f8019c00640 1 -- 172.21.15.27:0/3429442916 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f80140ca0d0 msgr2=0x7f80140ca550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.792+0000 7f8019c00640 1 --2- 172.21.15.27:0/3429442916 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f80140ca0d0 0x7f80140ca550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.792+0000 7f8019c00640 1 -- 172.21.15.27:0/3429442916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80140c8070 msgr2=0x7f80140c84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.792+0000 7f8019c00640 1 --2- 172.21.15.27:0/3429442916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80140c8070 0x7f80140c84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:37.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.792+0000 7f8019c00640 1 -- 172.21.15.27:0/3429442916 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80140c5510 con 0x7f7ff40047d0 2024-01-15T18:11:37.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.792+0000 7f8019200640 1 --2- 172.21.15.27:0/3429442916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80140c8070 0x7f80140c84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:11:37.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.792+0000 7f8019c00640 1 --2- 172.21.15.27:0/3429442916 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ff40047d0 0x7f80141f2fa0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8004004e30 tx=0x7f8004007430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:37.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.792+0000 7f8011600640 1 -- 172.21.15.27:0/3429442916 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8004007770 con 0x7f7ff40047d0 2024-01-15T18:11:37.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.792+0000 7f8011600640 1 -- 172.21.15.27:0/3429442916 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f800400f750 con 0x7f7ff40047d0 2024-01-15T18:11:37.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.792+0000 7f801c600640 1 -- 172.21.15.27:0/3429442916 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80140c57b0 con 0x7f7ff40047d0 2024-01-15T18:11:37.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.792+0000 7f8011600640 1 -- 172.21.15.27:0/3429442916 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800400fae0 con 0x7f7ff40047d0 2024-01-15T18:11:37.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.792+0000 7f801c600640 1 -- 172.21.15.27:0/3429442916 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80140caa90 con 0x7f7ff40047d0 2024-01-15T18:11:37.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.792+0000 7f801c600640 1 -- 172.21.15.27:0/3429442916 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f801404e850 con 0x7f7ff40047d0 2024-01-15T18:11:37.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.792+0000 7f8011600640 1 -- 172.21.15.27:0/3429442916 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 19) v1 ==== 87027+0+0 (secure 0 0 0) 0x7f800400fc40 con 0x7f7ff40047d0 2024-01-15T18:11:37.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.796+0000 7f8011600640 1 --2- 172.21.15.27:0/3429442916 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7fec0672b0 0x7f7fec069770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:37.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.796+0000 7f8011600640 1 -- 172.21.15.27:0/3429442916 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f80040827b0 con 0x7f7ff40047d0 2024-01-15T18:11:37.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.796+0000 7f8019200640 1 --2- 172.21.15.27:0/3429442916 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7fec0672b0 0x7f7fec069770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:37.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.796+0000 7f8019200640 1 --2- 172.21.15.27:0/3429442916 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7fec0672b0 0x7f7fec069770 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc000c10 tx=0x7f7ffc0aa070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:37.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.800+0000 7f8011600640 1 -- 172.21.15.27:0/3429442916 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f80040512f0 con 0x7f7ff40047d0 2024-01-15T18:11:37.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:37.984+0000 7f801c600640 1 -- 172.21.15.27:0/3429442916 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone91815", "target": ["mon-mgr", ""]}) v1 -- 0x7f80140d19a0 con 0x7f7fec0672b0 2024-01-15T18:11:38.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.000+0000 7f8011600640 1 -- 172.21.15.27:0/3429442916 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7ffc0a8aa0 con 0x7f7fec0672b0 2024-01-15T18:11:38.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.004+0000 7f801c600640 1 -- 172.21.15.27:0/3429442916 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7fec0672b0 msgr2=0x7f7fec069770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:38.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.004+0000 7f801c600640 1 --2- 172.21.15.27:0/3429442916 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7fec0672b0 0x7f7fec069770 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc000c10 tx=0x7f7ffc0aa070 comp rx=0 tx=0).stop 2024-01-15T18:11:38.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.004+0000 7f801c600640 1 -- 172.21.15.27:0/3429442916 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ff40047d0 msgr2=0x7f80141f2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:38.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.004+0000 7f801c600640 1 --2- 172.21.15.27:0/3429442916 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ff40047d0 0x7f80141f2fa0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8004004e30 tx=0x7f8004007430 comp rx=0 tx=0).stop 2024-01-15T18:11:38.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.004+0000 7f801c600640 1 -- 172.21.15.27:0/3429442916 shutdown_connections 2024-01-15T18:11:38.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.004+0000 7f801c600640 1 --2- 172.21.15.27:0/3429442916 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7fec0672b0 0x7f7fec069770 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:38.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.004+0000 7f801c600640 1 --2- 172.21.15.27:0/3429442916 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f80140ca0d0 0x7f80140ca550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:38.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.004+0000 7f801c600640 1 --2- 172.21.15.27:0/3429442916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f80140c8070 0x7f80140c84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:38.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.004+0000 7f801c600640 1 --2- 172.21.15.27:0/3429442916 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ff40047d0 0x7f80141f2fa0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:38.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.004+0000 7f801c600640 1 -- 172.21.15.27:0/3429442916 >> 172.21.15.27:0/3429442916 conn(0x7f80140d0e80 msgr2=0x7f801415c520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:38.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.004+0000 7f801c600640 1 -- 172.21.15.27:0/3429442916 shutdown_connections 2024-01-15T18:11:38.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.004+0000 7f801c600640 1 -- 172.21.15.27:0/3429442916 wait complete. 2024-01-15T18:11:38.029 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone91811 2024-01-15T18:11:38.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.176+0000 7fdde0c00640 1 Processor -- start 2024-01-15T18:11:38.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.176+0000 7fdde0c00640 1 -- start start 2024-01-15T18:11:38.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.176+0000 7fdde0c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdddc15b100 0x7fdddc070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:38.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.176+0000 7fdde0c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdddc071420 0x7fdddc15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:38.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.176+0000 7fdde0c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdddc15a610 con 0x7fdddc15ad20 2024-01-15T18:11:38.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.176+0000 7fdde0c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdddc0718b0 con 0x7fdddc15b100 2024-01-15T18:11:38.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.176+0000 7fdde0c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc15e3c0 con 0x7fdddc071420 2024-01-15T18:11:38.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.176+0000 7fddd9400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdddc15b100 0x7fdddc070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:38.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.176+0000 7fddd9400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdddc15b100 0x7fdddc070ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57458/0 (socket says 172.21.15.27:57458) 2024-01-15T18:11:38.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.176+0000 7fddd9400640 1 -- 172.21.15.27:0/951309109 learned_addr learned my addr 172.21.15.27:0/951309109 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:38.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.176+0000 7fddda800640 1 --2- 172.21.15.27:0/951309109 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdddc071420 0x7fdddc15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:38.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.176+0000 7fddd9400640 1 -- 172.21.15.27:0/951309109 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdddc071420 msgr2=0x7fdddc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:38.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.176+0000 7fddd9400640 1 --2- 172.21.15.27:0/951309109 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdddc071420 0x7fdddc15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:38.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.176+0000 7fddd9400640 1 -- 172.21.15.27:0/951309109 >> v1:172.21.15.93:6789/0 conn(0x7fdddc15ad20 legacy=0x7fdddc15d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:38.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.180+0000 7fddd9400640 1 -- 172.21.15.27:0/951309109 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc15e6a0 con 0x7fdddc15b100 2024-01-15T18:11:38.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.180+0000 7fddda800640 1 --2- 172.21.15.27:0/951309109 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdddc071420 0x7fdddc15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:11:38.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.180+0000 7fddd9400640 1 --2- 172.21.15.27:0/951309109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdddc15b100 0x7fdddc070ee0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fddc40098a0 tx=0x7fddc401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40d793508eb40fe1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:38.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.180+0000 7fddd8a00640 1 -- 172.21.15.27:0/951309109 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc40ab660 con 0x7fdddc15b100 2024-01-15T18:11:38.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.180+0000 7fddd8a00640 1 -- 172.21.15.27:0/951309109 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc40ab7c0 con 0x7fdddc15b100 2024-01-15T18:11:38.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.180+0000 7fddd8a00640 1 -- 172.21.15.27:0/951309109 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc40abb50 con 0x7fdddc15b100 2024-01-15T18:11:38.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.184+0000 7fdde0c00640 1 -- 172.21.15.27:0/951309109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdddc15b100 msgr2=0x7fdddc070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.184+0000 7fdde0c00640 1 --2- 172.21.15.27:0/951309109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdddc15b100 0x7fdddc070ee0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fddc40098a0 tx=0x7fddc401c460 comp rx=0 tx=0).stop 2024-01-15T18:11:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.184+0000 7fdde0c00640 1 -- 172.21.15.27:0/951309109 shutdown_connections 2024-01-15T18:11:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.184+0000 7fdde0c00640 1 --2- 172.21.15.27:0/951309109 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdddc071420 0x7fdddc15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.184+0000 7fdde0c00640 1 --2- 172.21.15.27:0/951309109 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdddc15b100 0x7fdddc070ee0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.184+0000 7fdde0c00640 1 -- 172.21.15.27:0/951309109 >> 172.21.15.27:0/951309109 conn(0x7fdddc0cb700 msgr2=0x7fdddc0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:38.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.184+0000 7fdde0c00640 1 -- 172.21.15.27:0/951309109 shutdown_connections 2024-01-15T18:11:38.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.184+0000 7fdde0c00640 1 -- 172.21.15.27:0/951309109 wait complete. 2024-01-15T18:11:38.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.184+0000 7fdde0c00640 1 Processor -- start 2024-01-15T18:11:38.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.184+0000 7fdde0c00640 1 -- start start 2024-01-15T18:11:38.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.184+0000 7fdde0c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdddc071420 0x7fdddc152b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:38.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.184+0000 7fdde0c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdddc15ad20 0x7fdddc163620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:38.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.184+0000 7fdde0c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdddc15b100 0x7fdddc0cea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:38.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.184+0000 7fdde0c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc0cd970 con 0x7fdddc15ad20 2024-01-15T18:11:38.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.184+0000 7fdde0c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc0cdc50 con 0x7fdddc071420 2024-01-15T18:11:38.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.184+0000 7fdde0c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc0cdf30 con 0x7fdddc15b100 2024-01-15T18:11:38.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.184+0000 7fddd9400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdddc15ad20 0x7fdddc163620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:38.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.184+0000 7fddd9400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdddc15ad20 0x7fdddc163620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57468/0 (socket says 172.21.15.27:57468) 2024-01-15T18:11:38.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.184+0000 7fddd9400640 1 -- 172.21.15.27:0/2699304834 learned_addr learned my addr 172.21.15.27:0/2699304834 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:38.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.184+0000 7fddda800640 1 --2- 172.21.15.27:0/2699304834 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdddc15b100 0x7fdddc0cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:38.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.184+0000 7fddd9e00640 1 --2- 172.21.15.27:0/2699304834 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdddc071420 0x7fdddc152b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:38.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.188+0000 7fddd9400640 1 -- 172.21.15.27:0/2699304834 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdddc15b100 msgr2=0x7fdddc0cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:38.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.188+0000 7fddd9400640 1 --2- 172.21.15.27:0/2699304834 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdddc15b100 0x7fdddc0cea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:38.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.188+0000 7fddd9400640 1 -- 172.21.15.27:0/2699304834 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdddc071420 msgr2=0x7fdddc152b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:38.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.188+0000 7fddd9400640 1 --2- 172.21.15.27:0/2699304834 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdddc071420 0x7fdddc152b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:38.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.188+0000 7fddd9400640 1 -- 172.21.15.27:0/2699304834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc40094d0 con 0x7fdddc15ad20 2024-01-15T18:11:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.188+0000 7fddd9400640 1 --2- 172.21.15.27:0/2699304834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdddc15ad20 0x7fdddc163620 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fddc40ad040 tx=0x7fddc4019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.188+0000 7fddd2000640 1 -- 172.21.15.27:0/2699304834 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc40b5810 con 0x7fdddc15ad20 2024-01-15T18:11:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.188+0000 7fddd2000640 1 -- 172.21.15.27:0/2699304834 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc40b5970 con 0x7fdddc15ad20 2024-01-15T18:11:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.188+0000 7fdde0c00640 1 -- 172.21.15.27:0/2699304834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdddc0ce210 con 0x7fdddc15ad20 2024-01-15T18:11:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.188+0000 7fddd2000640 1 -- 172.21.15.27:0/2699304834 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc40b5d20 con 0x7fdddc15ad20 2024-01-15T18:11:38.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.188+0000 7fdde0c00640 1 -- 172.21.15.27:0/2699304834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdddc14c8d0 con 0x7fdddc15ad20 2024-01-15T18:11:38.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.188+0000 7fdde0c00640 1 -- 172.21.15.27:0/2699304834 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdddc0ce070 con 0x7fdddc15ad20 2024-01-15T18:11:38.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.192+0000 7fddd2000640 1 -- 172.21.15.27:0/2699304834 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 87027+0+0 (secure 0 0 0) 0x7fddc40a9070 con 0x7fdddc15ad20 2024-01-15T18:11:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.192+0000 7fddd2000640 1 --2- 172.21.15.27:0/2699304834 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fddb4067280 0x7fddb4069740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:38.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.192+0000 7fddd2000640 1 -- 172.21.15.27:0/2699304834 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fddc411da30 con 0x7fdddc15ad20 2024-01-15T18:11:38.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.200+0000 7fddd9e00640 1 --2- 172.21.15.27:0/2699304834 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fddb4067280 0x7fddb4069740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:38.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.208+0000 7fddd2000640 1 -- 172.21.15.27:0/2699304834 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fdddc0ce070 con 0x7fdddc15ad20 2024-01-15T18:11:38.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.220+0000 7fddd9e00640 1 --2- 172.21.15.27:0/2699304834 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fddb4067280 0x7fddb4069740 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fddcc007fd0 tx=0x7fddcc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:38.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.576+0000 7fdde0c00640 1 -- 172.21.15.27:0/2699304834 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone91811", "target": ["mon-mgr", ""]}) v1 -- 0x7fdddc0cddc0 con 0x7fddb4067280 2024-01-15T18:11:38.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.608+0000 7fddd2000640 1 -- 172.21.15.27:0/2699304834 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdddc0cddc0 con 0x7fddb4067280 2024-01-15T18:11:38.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.616+0000 7fdde0c00640 1 -- 172.21.15.27:0/2699304834 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fddb4067280 msgr2=0x7fddb4069740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:38.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.616+0000 7fdde0c00640 1 --2- 172.21.15.27:0/2699304834 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fddb4067280 0x7fddb4069740 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fddcc007fd0 tx=0x7fddcc009040 comp rx=0 tx=0).stop 2024-01-15T18:11:38.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.616+0000 7fdde0c00640 1 -- 172.21.15.27:0/2699304834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdddc15ad20 msgr2=0x7fdddc163620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:38.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.616+0000 7fdde0c00640 1 --2- 172.21.15.27:0/2699304834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdddc15ad20 0x7fdddc163620 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fddc40ad040 tx=0x7fddc4019200 comp rx=0 tx=0).stop 2024-01-15T18:11:38.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.616+0000 7fdde0c00640 1 -- 172.21.15.27:0/2699304834 shutdown_connections 2024-01-15T18:11:38.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.616+0000 7fdde0c00640 1 --2- 172.21.15.27:0/2699304834 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fddb4067280 0x7fddb4069740 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:38.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.616+0000 7fdde0c00640 1 --2- 172.21.15.27:0/2699304834 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdddc15b100 0x7fdddc0cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:38.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.616+0000 7fdde0c00640 1 --2- 172.21.15.27:0/2699304834 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdddc15ad20 0x7fdddc163620 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:38.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.616+0000 7fdde0c00640 1 --2- 172.21.15.27:0/2699304834 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdddc071420 0x7fdddc152b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:38.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.616+0000 7fdde0c00640 1 -- 172.21.15.27:0/2699304834 >> 172.21.15.27:0/2699304834 conn(0x7fdddc0cb700 msgr2=0x7fdddc15d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:38.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.616+0000 7fdde0c00640 1 -- 172.21.15.27:0/2699304834 shutdown_connections 2024-01-15T18:11:38.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:38.616+0000 7fdde0c00640 1 -- 172.21.15.27:0/2699304834 wait complete. 2024-01-15T18:11:38.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:11:38.655 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-15T18:11:38.677 INFO:teuthology.orchestra.run.smithi027.stdout:3 2024-01-15T18:11:43.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:11:43.681 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-15T18:11:43.696 INFO:teuthology.orchestra.run.smithi027.stdout:2 2024-01-15T18:11:43.699 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-15T18:11:43.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.892+0000 7fb065200640 1 Processor -- start 2024-01-15T18:11:43.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.892+0000 7fb065200640 1 -- start start 2024-01-15T18:11:43.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.896+0000 7fb065200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fb06006c430 0x7fb060070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:43.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.896+0000 7fb065200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0600ce760 0x7fb0600b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:43.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.896+0000 7fb065200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0600ccfb0 con 0x7fb0600b1db0 2024-01-15T18:11:43.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.896+0000 7fb065200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0600cd290 con 0x7fb06006c430 2024-01-15T18:11:43.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.896+0000 7fb065200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0600cd570 con 0x7fb0600ce760 2024-01-15T18:11:43.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.896+0000 7fb05d400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb0600b1db0 0x7fb0600ce050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46270/0 (socket says 172.21.15.27:46270) 2024-01-15T18:11:43.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.896+0000 7fb05d400640 1 -- 172.21.15.27:0/3478212976 learned_addr learned my addr 172.21.15.27:0/3478212976 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:43.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.896+0000 7fb05de00640 1 --2- 172.21.15.27:0/3478212976 >> v2:172.21.15.93:3300/0 conn(0x7fb06006c430 0x7fb060070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:43.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.896+0000 7fb05e800640 1 --2- 172.21.15.27:0/3478212976 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0600ce760 0x7fb0600b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:43.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.896+0000 7fb05ca00640 1 -- 172.21.15.27:0/3478212976 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613117888 0 0) 0x7fb0600ccfb0 con 0x7fb0600b1db0 2024-01-15T18:11:43.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.896+0000 7fb05ca00640 1 -- 172.21.15.27:0/3478212976 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb040003530 con 0x7fb0600b1db0 2024-01-15T18:11:43.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.896+0000 7fb05de00640 1 -- 172.21.15.27:0/3478212976 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0600ce760 msgr2=0x7fb0600b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:43.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.896+0000 7fb05de00640 1 --2- 172.21.15.27:0/3478212976 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0600ce760 0x7fb0600b2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:43.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.896+0000 7fb05de00640 1 -- 172.21.15.27:0/3478212976 >> v1:172.21.15.27:6789/0 conn(0x7fb0600b1db0 legacy=0x7fb0600ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:43.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.896+0000 7fb05de00640 1 -- 172.21.15.27:0/3478212976 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0600cd850 con 0x7fb06006c430 2024-01-15T18:11:43.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.900+0000 7fb05de00640 1 --2- 172.21.15.27:0/3478212976 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb06006c430 0x7fb060070ba0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb0500097f0 tx=0x7fb05001c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=747417db11c5b600 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:43.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.900+0000 7fb05ca00640 1 -- 172.21.15.27:0/3478212976 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0500ad660 con 0x7fb06006c430 2024-01-15T18:11:43.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.900+0000 7fb05ca00640 1 -- 172.21.15.27:0/3478212976 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0500ad7c0 con 0x7fb06006c430 2024-01-15T18:11:43.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.900+0000 7fb05ca00640 1 -- 172.21.15.27:0/3478212976 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0500adb50 con 0x7fb06006c430 2024-01-15T18:11:43.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.900+0000 7fb065200640 1 -- 172.21.15.27:0/3478212976 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb06006c430 msgr2=0x7fb060070ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:43.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.900+0000 7fb065200640 1 --2- 172.21.15.27:0/3478212976 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb06006c430 0x7fb060070ba0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb0500097f0 tx=0x7fb05001c3c0 comp rx=0 tx=0).stop 2024-01-15T18:11:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.900+0000 7fb065200640 1 -- 172.21.15.27:0/3478212976 shutdown_connections 2024-01-15T18:11:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.900+0000 7fb065200640 1 --2- 172.21.15.27:0/3478212976 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0600ce760 0x7fb0600b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.900+0000 7fb065200640 1 --2- 172.21.15.27:0/3478212976 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb06006c430 0x7fb060070ba0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.900+0000 7fb065200640 1 -- 172.21.15.27:0/3478212976 >> 172.21.15.27:0/3478212976 conn(0x7fb0600cb700 msgr2=0x7fb0600cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.900+0000 7fb065200640 1 -- 172.21.15.27:0/3478212976 shutdown_connections 2024-01-15T18:11:43.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.900+0000 7fb065200640 1 -- 172.21.15.27:0/3478212976 wait complete. 2024-01-15T18:11:43.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb065200640 1 Processor -- start 2024-01-15T18:11:43.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb065200640 1 -- start start 2024-01-15T18:11:43.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb065200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb06006c430 0x7fb060144840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:43.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb065200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0600b1db0 0x7fb06014ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:43.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb065200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0600ce760 0x7fb060146800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:43.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb065200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06014e8f0 con 0x7fb06006c430 2024-01-15T18:11:43.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb065200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb060153bb0 con 0x7fb0600b1db0 2024-01-15T18:11:43.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb065200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb060153e90 con 0x7fb0600ce760 2024-01-15T18:11:43.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb05de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb06006c430 0x7fb060144840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:43.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb05de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb06006c430 0x7fb060144840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58052/0 (socket says 172.21.15.27:58052) 2024-01-15T18:11:43.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb05de00640 1 -- 172.21.15.27:0/127443583 learned_addr learned my addr 172.21.15.27:0/127443583 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:43.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb05e800640 1 --2- 172.21.15.27:0/127443583 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0600ce760 0x7fb060146800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:43.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb05d400640 1 --2- 172.21.15.27:0/127443583 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0600b1db0 0x7fb06014ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:43.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb05de00640 1 -- 172.21.15.27:0/127443583 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0600ce760 msgr2=0x7fb060146800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:43.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb05de00640 1 --2- 172.21.15.27:0/127443583 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0600ce760 0x7fb060146800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:43.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb05de00640 1 -- 172.21.15.27:0/127443583 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0600b1db0 msgr2=0x7fb06014ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:43.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb05de00640 1 --2- 172.21.15.27:0/127443583 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0600b1db0 0x7fb06014ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:43.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb05de00640 1 -- 172.21.15.27:0/127443583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb050009420 con 0x7fb06006c430 2024-01-15T18:11:43.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb05d400640 1 --2- 172.21.15.27:0/127443583 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0600b1db0 0x7fb06014ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:11:43.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb05de00640 1 --2- 172.21.15.27:0/127443583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb06006c430 0x7fb060144840 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fb0500b0040 tx=0x7fb050019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:43.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb056000640 1 -- 172.21.15.27:0/127443583 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0500b6610 con 0x7fb06006c430 2024-01-15T18:11:43.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb056000640 1 -- 172.21.15.27:0/127443583 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0500b6790 con 0x7fb06006c430 2024-01-15T18:11:43.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb056000640 1 -- 172.21.15.27:0/127443583 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0500b6b40 con 0x7fb06006c430 2024-01-15T18:11:43.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb065200640 1 -- 172.21.15.27:0/127443583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0600712f0 con 0x7fb06006c430 2024-01-15T18:11:43.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb065200640 1 -- 172.21.15.27:0/127443583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0601537d0 con 0x7fb06006c430 2024-01-15T18:11:43.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.904+0000 7fb065200640 1 -- 172.21.15.27:0/127443583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb060153fd0 con 0x7fb06006c430 2024-01-15T18:11:43.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.908+0000 7fb056000640 1 -- 172.21.15.27:0/127443583 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 20) v1 ==== 86925+0+0 (secure 0 0 0) 0x7fb0500b4040 con 0x7fb06006c430 2024-01-15T18:11:43.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.908+0000 7fb056000640 1 --2- 172.21.15.27:0/127443583 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb038067160 0x7fb038069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:43.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.908+0000 7fb056000640 1 -- 172.21.15.27:0/127443583 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fb05011f700 con 0x7fb06006c430 2024-01-15T18:11:43.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.908+0000 7fb05d400640 1 --2- 172.21.15.27:0/127443583 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb038067160 0x7fb038069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:43.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.908+0000 7fb05d400640 1 --2- 172.21.15.27:0/127443583 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb038067160 0x7fb038069620 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb0601f70c0 tx=0x7fb048009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:43.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:43.912+0000 7fb056000640 1 -- 172.21.15.27:0/127443583 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb060153fd0 con 0x7fb06006c430 2024-01-15T18:11:44.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:44.092+0000 7fb065200640 1 -- 172.21.15.27:0/127443583 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0600cafe0 con 0x7fb038067160 2024-01-15T18:11:44.135 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-01-15T18:11:44.132+0000 7f053dc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-15T18:11:44.225 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-15T18:11:45.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:45.156+0000 7fb056000640 1 -- 172.21.15.27:0/127443583 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7fb0500ebf00 con 0x7fb06006c430 2024-01-15T18:11:47.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.180+0000 7fb056000640 1 -- 172.21.15.27:0/127443583 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fb0600cafe0 con 0x7fb038067160 2024-01-15T18:11:47.195 INFO:teuthology.orchestra.run.smithi027.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-15T18:11:47.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.208+0000 7fb065200640 1 -- 172.21.15.27:0/127443583 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb038067160 msgr2=0x7fb038069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:47.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.208+0000 7fb065200640 1 --2- 172.21.15.27:0/127443583 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb038067160 0x7fb038069620 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fb0601f70c0 tx=0x7fb048009040 comp rx=0 tx=0).stop 2024-01-15T18:11:47.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.208+0000 7fb065200640 1 -- 172.21.15.27:0/127443583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb06006c430 msgr2=0x7fb060144840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:47.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.208+0000 7fb065200640 1 --2- 172.21.15.27:0/127443583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb06006c430 0x7fb060144840 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fb0500b0040 tx=0x7fb050019160 comp rx=0 tx=0).stop 2024-01-15T18:11:47.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.212+0000 7fb065200640 1 -- 172.21.15.27:0/127443583 shutdown_connections 2024-01-15T18:11:47.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.212+0000 7fb065200640 1 --2- 172.21.15.27:0/127443583 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb038067160 0x7fb038069620 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:47.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.212+0000 7fb065200640 1 --2- 172.21.15.27:0/127443583 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0600ce760 0x7fb060146800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:47.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.212+0000 7fb065200640 1 --2- 172.21.15.27:0/127443583 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0600b1db0 0x7fb06014ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:47.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.212+0000 7fb065200640 1 --2- 172.21.15.27:0/127443583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb06006c430 0x7fb060144840 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:47.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.212+0000 7fb065200640 1 -- 172.21.15.27:0/127443583 >> 172.21.15.27:0/127443583 conn(0x7fb0600cb700 msgr2=0x7fb06014a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:47.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.212+0000 7fb065200640 1 -- 172.21.15.27:0/127443583 shutdown_connections 2024-01-15T18:11:47.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.212+0000 7fb065200640 1 -- 172.21.15.27:0/127443583 wait complete. 2024-01-15T18:11:47.257 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T18:11:47.257 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:11:47.354 DEBUG:teuthology.orchestra.run.smithi093:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:11:47.426 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T18:11:47.427 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:11:47.427 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:11:47.437 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:11:47.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.648+0000 7f060f000640 1 Processor -- start 2024-01-15T18:11:47.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.648+0000 7f060f000640 1 -- start start 2024-01-15T18:11:47.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.652+0000 7f060f000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06080cf9b0 0x7f06080b10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:47.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.652+0000 7f060f000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06080b38b0 con 0x7f06080cedc0 2024-01-15T18:11:47.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.652+0000 7f060f000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06080cd550 con 0x7f060815e7d0 2024-01-15T18:11:47.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.652+0000 7f060f000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06080cd270 con 0x7f06080cf9b0 2024-01-15T18:11:47.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.652+0000 7f060d000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f06080cedc0 0x7f06080b1600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46284/0 (socket says 172.21.15.27:46284) 2024-01-15T18:11:47.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.652+0000 7f060d000640 1 -- 172.21.15.27:0/1243175911 learned_addr learned my addr 172.21.15.27:0/1243175911 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:47.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.652+0000 7f0607400640 1 --2- 172.21.15.27:0/1243175911 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06080cf9b0 0x7f06080b10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:47.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.652+0000 7f0606a00640 1 -- 172.21.15.27:0/1243175911 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1229645260 0 0) 0x7f06080b38b0 con 0x7f06080cedc0 2024-01-15T18:11:47.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.652+0000 7f0606a00640 1 -- 172.21.15.27:0/1243175911 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05e8003530 con 0x7f06080cedc0 2024-01-15T18:11:47.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.652+0000 7f0606a00640 1 -- 172.21.15.27:0/1243175911 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 467430958 0 0) 0x7f06080cd550 con 0x7f060815e7d0 2024-01-15T18:11:47.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.652+0000 7f0606a00640 1 -- 172.21.15.27:0/1243175911 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06080b38b0 con 0x7f060815e7d0 2024-01-15T18:11:47.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.652+0000 7f0607400640 1 -- 172.21.15.27:0/1243175911 >> v1:172.21.15.93:6789/0 conn(0x7f060815e7d0 legacy=0x7f06080cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:47.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.652+0000 7f0607400640 1 -- 172.21.15.27:0/1243175911 >> v1:172.21.15.27:6789/0 conn(0x7f06080cedc0 legacy=0x7f06080b1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:47.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.652+0000 7f0607400640 1 -- 172.21.15.27:0/1243175911 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06080b35d0 con 0x7f06080cf9b0 2024-01-15T18:11:47.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.656+0000 7f0607400640 1 --2- 172.21.15.27:0/1243175911 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06080cf9b0 0x7f06080b10c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f05f00097f0 tx=0x7f06080b2520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1749efc61111fac6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:47.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.656+0000 7f0606a00640 1 -- 172.21.15.27:0/1243175911 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f00ac410 con 0x7f06080cf9b0 2024-01-15T18:11:47.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.656+0000 7f0606a00640 1 -- 172.21.15.27:0/1243175911 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f00ac570 con 0x7f06080cf9b0 2024-01-15T18:11:47.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.656+0000 7f0606a00640 1 -- 172.21.15.27:0/1243175911 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f00ac900 con 0x7f06080cf9b0 2024-01-15T18:11:47.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.656+0000 7f060f000640 1 -- 172.21.15.27:0/1243175911 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06080cf9b0 msgr2=0x7f06080b10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:47.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.656+0000 7f060f000640 1 --2- 172.21.15.27:0/1243175911 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06080cf9b0 0x7f06080b10c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f05f00097f0 tx=0x7f06080b2520 comp rx=0 tx=0).stop 2024-01-15T18:11:47.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.656+0000 7f060f000640 1 -- 172.21.15.27:0/1243175911 shutdown_connections 2024-01-15T18:11:47.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.656+0000 7f060f000640 1 --2- 172.21.15.27:0/1243175911 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06080cf9b0 0x7f06080b10c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:47.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.656+0000 7f060f000640 1 -- 172.21.15.27:0/1243175911 >> 172.21.15.27:0/1243175911 conn(0x7f06080b2a10 msgr2=0x7f06080b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:47.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.660+0000 7f060f000640 1 -- 172.21.15.27:0/1243175911 shutdown_connections 2024-01-15T18:11:47.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.660+0000 7f060f000640 1 -- 172.21.15.27:0/1243175911 wait complete. 2024-01-15T18:11:47.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.660+0000 7f060f000640 1 Processor -- start 2024-01-15T18:11:47.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.660+0000 7f060f000640 1 -- start start 2024-01-15T18:11:47.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f060f000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06080cedc0 0x7f06080cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:47.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f060f000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06080cf9b0 0x7f0608149d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:47.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f060f000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f060815e7d0 0x7f06081495d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:47.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f060f000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0608149b10 con 0x7f060815e7d0 2024-01-15T18:11:47.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f060f000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060814a590 con 0x7f06080cedc0 2024-01-15T18:11:47.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f060f000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f060815d9b0 con 0x7f06080cf9b0 2024-01-15T18:11:47.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f060d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f060815e7d0 0x7f06081495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:47.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f060d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f060815e7d0 0x7f06081495d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58072/0 (socket says 172.21.15.27:58072) 2024-01-15T18:11:47.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f060d000640 1 -- 172.21.15.27:0/2095538288 learned_addr learned my addr 172.21.15.27:0/2095538288 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:47.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f0607e00640 1 --2- 172.21.15.27:0/2095538288 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06080cedc0 0x7f06080cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:47.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f0607400640 1 --2- 172.21.15.27:0/2095538288 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06080cf9b0 0x7f0608149d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:47.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f060d000640 1 -- 172.21.15.27:0/2095538288 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06080cf9b0 msgr2=0x7f0608149d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:47.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f060d000640 1 --2- 172.21.15.27:0/2095538288 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06080cf9b0 0x7f0608149d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:47.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f060d000640 1 -- 172.21.15.27:0/2095538288 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06080cedc0 msgr2=0x7f06080cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:47.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f060d000640 1 --2- 172.21.15.27:0/2095538288 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06080cedc0 0x7f06080cf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:47.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f060d000640 1 -- 172.21.15.27:0/2095538288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f0009420 con 0x7f060815e7d0 2024-01-15T18:11:47.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f0607e00640 1 --2- 172.21.15.27:0/2095538288 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06080cedc0 0x7f06080cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:11:47.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f0607400640 1 --2- 172.21.15.27:0/2095538288 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06080cf9b0 0x7f0608149d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:11:47.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f060d000640 1 --2- 172.21.15.27:0/2095538288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f060815e7d0 0x7f06081495d0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f060000caa0 tx=0x7f060000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:47.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f05ffe00640 1 -- 172.21.15.27:0/2095538288 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060000ac00 con 0x7f060815e7d0 2024-01-15T18:11:47.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f05ffe00640 1 -- 172.21.15.27:0/2095538288 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f060000ad80 con 0x7f060815e7d0 2024-01-15T18:11:47.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f060f000640 1 -- 172.21.15.27:0/2095538288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f060815dc90 con 0x7f060815e7d0 2024-01-15T18:11:47.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f05ffe00640 1 -- 172.21.15.27:0/2095538288 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0600012640 con 0x7f060815e7d0 2024-01-15T18:11:47.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f060f000640 1 -- 172.21.15.27:0/2095538288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f060806c490 con 0x7f060815e7d0 2024-01-15T18:11:47.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f060f000640 1 -- 172.21.15.27:0/2095538288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f060815deb0 con 0x7f060815e7d0 2024-01-15T18:11:47.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.664+0000 7f05ffe00640 1 -- 172.21.15.27:0/2095538288 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f0600007070 con 0x7f060815e7d0 2024-01-15T18:11:47.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.668+0000 7f05ffe00640 1 --2- 172.21.15.27:0/2095538288 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05e0067100 0x7f05e00695c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:47.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.668+0000 7f05ffe00640 1 -- 172.21.15.27:0/2095538288 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4999+0+0 (secure 0 0 0) 0x7f0600082990 con 0x7f060815e7d0 2024-01-15T18:11:47.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.668+0000 7f0607e00640 1 --2- 172.21.15.27:0/2095538288 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05e0067100 0x7f05e00695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:47.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.668+0000 7f0607e00640 1 --2- 172.21.15.27:0/2095538288 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05e0067100 0x7f05e00695c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f05f80011f0 tx=0x7f05f8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:47.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.672+0000 7f05ffe00640 1 -- 172.21.15.27:0/2095538288 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0600051880 con 0x7f060815e7d0 2024-01-15T18:11:47.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.840+0000 7f060f000640 1 -- 172.21.15.27:0/2095538288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f060806bfe0 con 0x7f060815e7d0 2024-01-15T18:11:47.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.840+0000 7f05ffe00640 1 -- 172.21.15.27:0/2095538288 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+12977 (secure 0 0 0) 0x7f0600017070 con 0x7f060815e7d0 2024-01-15T18:11:47.843 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:11:47.844 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":62,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:11:47.160522+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6834/1969845074":"2024-01-16T18:11:44.135277+0000","172.21.15.27:6832/1969845074":"2024-01-16T18:11:44.135277+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:11:47.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.844+0000 7f060f000640 1 -- 172.21.15.27:0/2095538288 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05e0067100 msgr2=0x7f05e00695c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:47.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.844+0000 7f060f000640 1 --2- 172.21.15.27:0/2095538288 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05e0067100 0x7f05e00695c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f05f80011f0 tx=0x7f05f8009040 comp rx=0 tx=0).stop 2024-01-15T18:11:47.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.844+0000 7f060f000640 1 -- 172.21.15.27:0/2095538288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f060815e7d0 msgr2=0x7f06081495d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:47.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.844+0000 7f060f000640 1 --2- 172.21.15.27:0/2095538288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f060815e7d0 0x7f06081495d0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f060000caa0 tx=0x7f060000cf60 comp rx=0 tx=0).stop 2024-01-15T18:11:47.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.844+0000 7f060f000640 1 -- 172.21.15.27:0/2095538288 shutdown_connections 2024-01-15T18:11:47.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.844+0000 7f060f000640 1 --2- 172.21.15.27:0/2095538288 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f05e0067100 0x7f05e00695c0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:47.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.844+0000 7f060f000640 1 --2- 172.21.15.27:0/2095538288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f060815e7d0 0x7f06081495d0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:47.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.844+0000 7f060f000640 1 --2- 172.21.15.27:0/2095538288 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f06080cf9b0 0x7f0608149d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:47.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.844+0000 7f060f000640 1 --2- 172.21.15.27:0/2095538288 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f06080cedc0 0x7f06080cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:47.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.844+0000 7f060f000640 1 -- 172.21.15.27:0/2095538288 >> 172.21.15.27:0/2095538288 conn(0x7f06080b2a10 msgr2=0x7f06080cc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:47.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.844+0000 7f060f000640 1 -- 172.21.15.27:0/2095538288 shutdown_connections 2024-01-15T18:11:47.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:47.844+0000 7f060f000640 1 -- 172.21.15.27:0/2095538288 wait complete. 2024-01-15T18:11:47.866 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:11:47.866 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:11:48.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.036+0000 7f13e7a00640 1 Processor -- start 2024-01-15T18:11:48.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.036+0000 7f13e7a00640 1 -- start start 2024-01-15T18:11:48.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.036+0000 7f13e7a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13e015b4b0 0x7f13e00b1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:48.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.036+0000 7f13e7a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f13e00cec60 0x7f13e00b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:48.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.036+0000 7f13e7a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e00d19a0 con 0x7f13e00cf460 2024-01-15T18:11:48.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.036+0000 7f13e7a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13e00d1c80 con 0x7f13e00cec60 2024-01-15T18:11:48.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.036+0000 7f13e7a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13e00b07c0 con 0x7f13e015b4b0 2024-01-15T18:11:48.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.036+0000 7f13e5000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f13e00cf460 0x7f13e00cf840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46292/0 (socket says 172.21.15.27:46292) 2024-01-15T18:11:48.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.036+0000 7f13e5000640 1 -- 172.21.15.27:0/2996171768 learned_addr learned my addr 172.21.15.27:0/2996171768 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:48.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.036+0000 7f13e5a00640 1 --2- 172.21.15.27:0/2996171768 >> v2:172.21.15.27:3300/0 conn(0x7f13e00cec60 0x7f13e00b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:48.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.036+0000 7f13dfe00640 1 --2- 172.21.15.27:0/2996171768 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13e015b4b0 0x7f13e00b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:48.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.036+0000 7f13df400640 1 -- 172.21.15.27:0/2996171768 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256464792 0 0) 0x7f13e00d19a0 con 0x7f13e00cf460 2024-01-15T18:11:48.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.036+0000 7f13df400640 1 -- 172.21.15.27:0/2996171768 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c0003530 con 0x7f13e00cf460 2024-01-15T18:11:48.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.036+0000 7f13e5a00640 1 -- 172.21.15.27:0/2996171768 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13e015b4b0 msgr2=0x7f13e00b1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:48.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.036+0000 7f13e5a00640 1 --2- 172.21.15.27:0/2996171768 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13e015b4b0 0x7f13e00b1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:48.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.036+0000 7f13e5a00640 1 -- 172.21.15.27:0/2996171768 >> v1:172.21.15.27:6789/0 conn(0x7f13e00cf460 legacy=0x7f13e00cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:48.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.036+0000 7f13e5a00640 1 -- 172.21.15.27:0/2996171768 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e00b0aa0 con 0x7f13e00cec60 2024-01-15T18:11:48.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.040+0000 7f13e5a00640 1 --2- 172.21.15.27:0/2996171768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13e00cec60 0x7f13e00b1ac0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f13d8009d90 tx=0x7f13d801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a9905e59e40d3d5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:48.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.040+0000 7f13df400640 1 -- 172.21.15.27:0/2996171768 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e00b2810 con 0x7f13e00cec60 2024-01-15T18:11:48.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.040+0000 7f13df400640 1 -- 172.21.15.27:0/2996171768 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d80a5a10 con 0x7f13e00cec60 2024-01-15T18:11:48.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.040+0000 7f13df400640 1 -- 172.21.15.27:0/2996171768 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d80a5da0 con 0x7f13e00cec60 2024-01-15T18:11:48.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.040+0000 7f13e7a00640 1 -- 172.21.15.27:0/2996171768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13e00cec60 msgr2=0x7f13e00b1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:48.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.040+0000 7f13e7a00640 1 --2- 172.21.15.27:0/2996171768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13e00cec60 0x7f13e00b1ac0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f13d8009d90 tx=0x7f13d801c950 comp rx=0 tx=0).stop 2024-01-15T18:11:48.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.040+0000 7f13e7a00640 1 -- 172.21.15.27:0/2996171768 shutdown_connections 2024-01-15T18:11:48.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.040+0000 7f13e7a00640 1 --2- 172.21.15.27:0/2996171768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13e00cec60 0x7f13e00b1ac0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:48.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.040+0000 7f13e7a00640 1 --2- 172.21.15.27:0/2996171768 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13e015b4b0 0x7f13e00b1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:48.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.040+0000 7f13e7a00640 1 -- 172.21.15.27:0/2996171768 >> 172.21.15.27:0/2996171768 conn(0x7f13e00d0e80 msgr2=0x7f13e00d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:48.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.040+0000 7f13e7a00640 1 -- 172.21.15.27:0/2996171768 shutdown_connections 2024-01-15T18:11:48.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.040+0000 7f13e7a00640 1 -- 172.21.15.27:0/2996171768 wait complete. 2024-01-15T18:11:48.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.044+0000 7f13e7a00640 1 Processor -- start 2024-01-15T18:11:48.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.044+0000 7f13e7a00640 1 -- start start 2024-01-15T18:11:48.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.044+0000 7f13e7a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13e00cec60 0x7f13e015a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:48.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.044+0000 7f13e7a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13e00cf460 0x7f13e014d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:48.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.044+0000 7f13e7a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13e015b4b0 0x7f13e0159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:48.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.044+0000 7f13e7a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e0159fc0 con 0x7f13e00cf460 2024-01-15T18:11:48.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.044+0000 7f13e7a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e014dcf0 con 0x7f13e015b4b0 2024-01-15T18:11:48.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.044+0000 7f13e7a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13e00c8b00 con 0x7f13e00cec60 2024-01-15T18:11:48.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.044+0000 7f13dfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13e00cf460 0x7f13e014d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:48.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.044+0000 7f13dfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13e00cf460 0x7f13e014d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58088/0 (socket says 172.21.15.27:58088) 2024-01-15T18:11:48.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.044+0000 7f13dfe00640 1 -- 172.21.15.27:0/2557553113 learned_addr learned my addr 172.21.15.27:0/2557553113 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:48.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.044+0000 7f13e5000640 1 --2- 172.21.15.27:0/2557553113 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13e00cec60 0x7f13e015a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:48.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.044+0000 7f13e5a00640 1 --2- 172.21.15.27:0/2557553113 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13e015b4b0 0x7f13e0159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:48.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.044+0000 7f13dfe00640 1 -- 172.21.15.27:0/2557553113 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13e00cec60 msgr2=0x7f13e015a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:48.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.044+0000 7f13dfe00640 1 --2- 172.21.15.27:0/2557553113 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13e00cec60 0x7f13e015a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:48.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.044+0000 7f13dfe00640 1 -- 172.21.15.27:0/2557553113 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13e015b4b0 msgr2=0x7f13e0159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:48.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.044+0000 7f13dfe00640 1 --2- 172.21.15.27:0/2557553113 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13e015b4b0 0x7f13e0159770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:48.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.044+0000 7f13dfe00640 1 -- 172.21.15.27:0/2557553113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d8009a70 con 0x7f13e00cf460 2024-01-15T18:11:48.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.044+0000 7f13dfe00640 1 --2- 172.21.15.27:0/2557553113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13e00cf460 0x7f13e014d7b0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f13c800e930 tx=0x7f13c800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:48.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.044+0000 7f13dcc00640 1 -- 172.21.15.27:0/2557553113 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c8009690 con 0x7f13e00cf460 2024-01-15T18:11:48.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.044+0000 7f13dcc00640 1 -- 172.21.15.27:0/2557553113 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c800ca00 con 0x7f13e00cf460 2024-01-15T18:11:48.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.044+0000 7f13e7a00640 1 -- 172.21.15.27:0/2557553113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e00c8da0 con 0x7f13e00cf460 2024-01-15T18:11:48.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.044+0000 7f13dcc00640 1 -- 172.21.15.27:0/2557553113 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c800cd70 con 0x7f13e00cf460 2024-01-15T18:11:48.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.044+0000 7f13e7a00640 1 -- 172.21.15.27:0/2557553113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e00c9360 con 0x7f13e00cf460 2024-01-15T18:11:48.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.048+0000 7f13e7a00640 1 -- 172.21.15.27:0/2557553113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13e004e850 con 0x7f13e00cf460 2024-01-15T18:11:48.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.048+0000 7f13dcc00640 1 -- 172.21.15.27:0/2557553113 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f13c8013470 con 0x7f13e00cf460 2024-01-15T18:11:48.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.048+0000 7f13dcc00640 1 --2- 172.21.15.27:0/2557553113 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f13b8067100 0x7f13b80695c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:48.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.048+0000 7f13dcc00640 1 -- 172.21.15.27:0/2557553113 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4999+0+0 (secure 0 0 0) 0x7f13c8082d20 con 0x7f13e00cf460 2024-01-15T18:11:48.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.048+0000 7f13e5000640 1 --2- 172.21.15.27:0/2557553113 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f13b8067100 0x7f13b80695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:48.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.052+0000 7f13e5000640 1 --2- 172.21.15.27:0/2557553113 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f13b8067100 0x7f13b80695c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f13d00011f0 tx=0x7f13d0007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:48.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.052+0000 7f13dcc00640 1 -- 172.21.15.27:0/2557553113 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f13c8051c40 con 0x7f13e00cf460 2024-01-15T18:11:48.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.224+0000 7f13e7a00640 1 -- 172.21.15.27:0/2557553113 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f13e00b6970 con 0x7f13e00cf460 2024-01-15T18:11:48.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.224+0000 7f13dcc00640 1 -- 172.21.15.27:0/2557553113 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v62) v1 ==== 93+0+10 (secure 0 0 0) 0x7f13c80210a0 con 0x7f13e00cf460 2024-01-15T18:11:48.228 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:11:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.228+0000 7f13e7a00640 1 -- 172.21.15.27:0/2557553113 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f13b8067100 msgr2=0x7f13b80695c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.228+0000 7f13e7a00640 1 --2- 172.21.15.27:0/2557553113 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f13b8067100 0x7f13b80695c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f13d00011f0 tx=0x7f13d0007d40 comp rx=0 tx=0).stop 2024-01-15T18:11:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.228+0000 7f13e7a00640 1 -- 172.21.15.27:0/2557553113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13e00cf460 msgr2=0x7f13e014d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.228+0000 7f13e7a00640 1 --2- 172.21.15.27:0/2557553113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13e00cf460 0x7f13e014d7b0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f13c800e930 tx=0x7f13c800edf0 comp rx=0 tx=0).stop 2024-01-15T18:11:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.228+0000 7f13e7a00640 1 -- 172.21.15.27:0/2557553113 shutdown_connections 2024-01-15T18:11:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.228+0000 7f13e7a00640 1 --2- 172.21.15.27:0/2557553113 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f13b8067100 0x7f13b80695c0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.228+0000 7f13e7a00640 1 --2- 172.21.15.27:0/2557553113 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13e015b4b0 0x7f13e0159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.228+0000 7f13e7a00640 1 --2- 172.21.15.27:0/2557553113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13e00cf460 0x7f13e014d7b0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.228+0000 7f13e7a00640 1 --2- 172.21.15.27:0/2557553113 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13e00cec60 0x7f13e015a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.228+0000 7f13e7a00640 1 -- 172.21.15.27:0/2557553113 >> 172.21.15.27:0/2557553113 conn(0x7f13e00d0e80 msgr2=0x7f13e00aea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.228+0000 7f13e7a00640 1 -- 172.21.15.27:0/2557553113 shutdown_connections 2024-01-15T18:11:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.228+0000 7f13e7a00640 1 -- 172.21.15.27:0/2557553113 wait complete. 2024-01-15T18:11:48.250 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:11:48.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.392+0000 7f3fd8a00640 1 Processor -- start 2024-01-15T18:11:48.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.392+0000 7f3fd8a00640 1 -- start start 2024-01-15T18:11:48.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.392+0000 7f3fd8a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3fd006c200 0x7f3fd006d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:48.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.392+0000 7f3fd8a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fd006c750 con 0x7f3fd00cad40 2024-01-15T18:11:48.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.392+0000 7f3fd8a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fd006ca30 con 0x7f3fd0143da0 2024-01-15T18:11:48.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.392+0000 7f3fd8a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd0144180 con 0x7f3fd006c200 2024-01-15T18:11:48.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.392+0000 7f3fd6a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3fd00cad40 0x7f3fd00701c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46308/0 (socket says 172.21.15.27:46308) 2024-01-15T18:11:48.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.392+0000 7f3fd6a00640 1 -- 172.21.15.27:0/1222750064 learned_addr learned my addr 172.21.15.27:0/1222750064 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:48.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.392+0000 7f3fd5600640 1 --2- 172.21.15.27:0/1222750064 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3fd006c200 0x7f3fd006d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:48.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.392+0000 7f3fd4c00640 1 -- 172.21.15.27:0/1222750064 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899045301 0 0) 0x7f3fd006c750 con 0x7f3fd00cad40 2024-01-15T18:11:48.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.392+0000 7f3fd4c00640 1 -- 172.21.15.27:0/1222750064 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fb0003530 con 0x7f3fd00cad40 2024-01-15T18:11:48.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.392+0000 7f3fd4c00640 1 -- 172.21.15.27:0/1222750064 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860978476 0 0) 0x7f3fd006ca30 con 0x7f3fd0143da0 2024-01-15T18:11:48.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.392+0000 7f3fd4c00640 1 -- 172.21.15.27:0/1222750064 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fd006c750 con 0x7f3fd0143da0 2024-01-15T18:11:48.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.392+0000 7f3fd5600640 1 -- 172.21.15.27:0/1222750064 >> v1:172.21.15.93:6789/0 conn(0x7f3fd0143da0 legacy=0x7f3fd00cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:48.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.392+0000 7f3fd5600640 1 -- 172.21.15.27:0/1222750064 >> v1:172.21.15.27:6789/0 conn(0x7f3fd00cad40 legacy=0x7f3fd00701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:48.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.392+0000 7f3fd5600640 1 -- 172.21.15.27:0/1222750064 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd006cd10 con 0x7f3fd006c200 2024-01-15T18:11:48.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.396+0000 7f3fd5600640 1 --2- 172.21.15.27:0/1222750064 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3fd006c200 0x7f3fd006d7f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3fb80097f0 tx=0x7f3fb801c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88633c4ab48be47a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:48.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.396+0000 7f3fd4c00640 1 -- 172.21.15.27:0/1222750064 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb80ac410 con 0x7f3fd006c200 2024-01-15T18:11:48.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.396+0000 7f3fd4c00640 1 -- 172.21.15.27:0/1222750064 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb80ac570 con 0x7f3fd006c200 2024-01-15T18:11:48.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.396+0000 7f3fd4c00640 1 -- 172.21.15.27:0/1222750064 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb80ac900 con 0x7f3fd006c200 2024-01-15T18:11:48.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.396+0000 7f3fd8a00640 1 -- 172.21.15.27:0/1222750064 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3fd006c200 msgr2=0x7f3fd006d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:48.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.396+0000 7f3fd8a00640 1 --2- 172.21.15.27:0/1222750064 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3fd006c200 0x7f3fd006d7f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3fb80097f0 tx=0x7f3fb801c4c0 comp rx=0 tx=0).stop 2024-01-15T18:11:48.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.396+0000 7f3fd8a00640 1 -- 172.21.15.27:0/1222750064 shutdown_connections 2024-01-15T18:11:48.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.396+0000 7f3fd8a00640 1 --2- 172.21.15.27:0/1222750064 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3fd006c200 0x7f3fd006d7f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:48.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.396+0000 7f3fd8a00640 1 -- 172.21.15.27:0/1222750064 >> 172.21.15.27:0/1222750064 conn(0x7f3fd00b0850 msgr2=0x7f3fd00b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:48.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.396+0000 7f3fd8a00640 1 -- 172.21.15.27:0/1222750064 shutdown_connections 2024-01-15T18:11:48.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.396+0000 7f3fd8a00640 1 -- 172.21.15.27:0/1222750064 wait complete. 2024-01-15T18:11:48.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.400+0000 7f3fd8a00640 1 Processor -- start 2024-01-15T18:11:48.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.400+0000 7f3fd8a00640 1 -- start start 2024-01-15T18:11:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.400+0000 7f3fd8a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fd006c200 0x7f3fd015d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.400+0000 7f3fd8a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3fd00cad40 0x7f3fd00adea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.400+0000 7f3fd8a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3fd0143da0 0x7f3fd00b2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.400+0000 7f3fd8a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd00ada10 con 0x7f3fd006c200 2024-01-15T18:11:48.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.400+0000 7f3fd8a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd015df00 con 0x7f3fd0143da0 2024-01-15T18:11:48.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.400+0000 7f3fd8a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd00ad3c0 con 0x7f3fd00cad40 2024-01-15T18:11:48.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.400+0000 7f3fd6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fd006c200 0x7f3fd015d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:48.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.400+0000 7f3fd6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fd006c200 0x7f3fd015d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58104/0 (socket says 172.21.15.27:58104) 2024-01-15T18:11:48.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.400+0000 7f3fd6000640 1 -- 172.21.15.27:0/95630451 learned_addr learned my addr 172.21.15.27:0/95630451 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:48.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.400+0000 7f3fd6a00640 1 --2- 172.21.15.27:0/95630451 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3fd0143da0 0x7f3fd00b2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:48.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.400+0000 7f3fd5600640 1 --2- 172.21.15.27:0/95630451 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3fd00cad40 0x7f3fd00adea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:48.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.400+0000 7f3fd6000640 1 -- 172.21.15.27:0/95630451 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3fd00cad40 msgr2=0x7f3fd00adea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:48.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.400+0000 7f3fd6000640 1 --2- 172.21.15.27:0/95630451 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3fd00cad40 0x7f3fd00adea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:48.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.400+0000 7f3fd6000640 1 -- 172.21.15.27:0/95630451 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3fd0143da0 msgr2=0x7f3fd00b2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:48.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.400+0000 7f3fd6000640 1 --2- 172.21.15.27:0/95630451 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3fd0143da0 0x7f3fd00b2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:48.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.400+0000 7f3fd6000640 1 -- 172.21.15.27:0/95630451 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb8009420 con 0x7f3fd006c200 2024-01-15T18:11:48.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.400+0000 7f3fd6a00640 1 --2- 172.21.15.27:0/95630451 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3fd0143da0 0x7f3fd00b2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:11:48.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.400+0000 7f3fd6000640 1 --2- 172.21.15.27:0/95630451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fd006c200 0x7f3fd015d4a0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f3fc000c430 tx=0x7f3fc000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:48.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.404+0000 7f3fce000640 1 -- 172.21.15.27:0/95630451 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc00052d0 con 0x7f3fd006c200 2024-01-15T18:11:48.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.404+0000 7f3fce000640 1 -- 172.21.15.27:0/95630451 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc0005450 con 0x7f3fd006c200 2024-01-15T18:11:48.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.404+0000 7f3fce000640 1 -- 172.21.15.27:0/95630451 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc0003840 con 0x7f3fd006c200 2024-01-15T18:11:48.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.404+0000 7f3fd8a00640 1 -- 172.21.15.27:0/95630451 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fd00ad6a0 con 0x7f3fd006c200 2024-01-15T18:11:48.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.404+0000 7f3fd8a00640 1 -- 172.21.15.27:0/95630451 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fd015dad0 con 0x7f3fd006c200 2024-01-15T18:11:48.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.404+0000 7f3fd8a00640 1 -- 172.21.15.27:0/95630451 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fd015dcc0 con 0x7f3fd006c200 2024-01-15T18:11:48.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.404+0000 7f3fce000640 1 -- 172.21.15.27:0/95630451 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f3fc00039a0 con 0x7f3fd006c200 2024-01-15T18:11:48.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.404+0000 7f3fce000640 1 --2- 172.21.15.27:0/95630451 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3fa80670d0 0x7f3fa8069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:48.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.404+0000 7f3fce000640 1 -- 172.21.15.27:0/95630451 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4999+0+0 (secure 0 0 0) 0x7f3fc0082080 con 0x7f3fd006c200 2024-01-15T18:11:48.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.404+0000 7f3fd5600640 1 --2- 172.21.15.27:0/95630451 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3fa80670d0 0x7f3fa8069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:48.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.408+0000 7f3fd5600640 1 --2- 172.21.15.27:0/95630451 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3fa80670d0 0x7f3fa8069590 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f3fb80ae000 tx=0x7f3fb8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:48.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.408+0000 7f3fce000640 1 -- 172.21.15.27:0/95630451 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3fd015dcc0 con 0x7f3fd006c200 2024-01-15T18:11:48.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.588+0000 7f3fd8a00640 1 -- 172.21.15.27:0/95630451 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3fd015dcc0 con 0x7f3fd006c200 2024-01-15T18:11:48.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.588+0000 7f3fce000640 1 -- 172.21.15.27:0/95630451 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v62) v1 ==== 80+0+122 (secure 0 0 0) 0x7f3fd015dcc0 con 0x7f3fd006c200 2024-01-15T18:11:48.589 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6834/1969845074 2024-01-16T18:11:44.135277+0000 2024-01-15T18:11:48.589 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6832/1969845074 2024-01-16T18:11:44.135277+0000 2024-01-15T18:11:48.590 INFO:teuthology.orchestra.run.smithi027.stderr:listed 2 entries 2024-01-15T18:11:48.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.588+0000 7f3fd8a00640 1 -- 172.21.15.27:0/95630451 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3fa80670d0 msgr2=0x7f3fa8069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:48.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.588+0000 7f3fd8a00640 1 --2- 172.21.15.27:0/95630451 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3fa80670d0 0x7f3fa8069590 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f3fb80ae000 tx=0x7f3fb8008040 comp rx=0 tx=0).stop 2024-01-15T18:11:48.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.588+0000 7f3fd8a00640 1 -- 172.21.15.27:0/95630451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fd006c200 msgr2=0x7f3fd015d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:48.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.588+0000 7f3fd8a00640 1 --2- 172.21.15.27:0/95630451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fd006c200 0x7f3fd015d4a0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f3fc000c430 tx=0x7f3fc000c8f0 comp rx=0 tx=0).stop 2024-01-15T18:11:48.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.588+0000 7f3fd8a00640 1 -- 172.21.15.27:0/95630451 shutdown_connections 2024-01-15T18:11:48.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.588+0000 7f3fd8a00640 1 --2- 172.21.15.27:0/95630451 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3fa80670d0 0x7f3fa8069590 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:48.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.588+0000 7f3fd8a00640 1 --2- 172.21.15.27:0/95630451 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3fd0143da0 0x7f3fd00b2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:48.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.588+0000 7f3fd8a00640 1 --2- 172.21.15.27:0/95630451 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3fd00cad40 0x7f3fd00adea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:48.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.588+0000 7f3fd8a00640 1 --2- 172.21.15.27:0/95630451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fd006c200 0x7f3fd015d4a0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:48.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.588+0000 7f3fd8a00640 1 -- 172.21.15.27:0/95630451 >> 172.21.15.27:0/95630451 conn(0x7f3fd00b0850 msgr2=0x7f3fd00712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:48.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.592+0000 7f3fd8a00640 1 -- 172.21.15.27:0/95630451 shutdown_connections 2024-01-15T18:11:48.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.592+0000 7f3fd8a00640 1 -- 172.21.15.27:0/95630451 wait complete. 2024-01-15T18:11:48.611 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-01-15T18:11:48.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:11:48.611 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-15T18:11:48.632 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T18:11:48.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:11:48.632 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-15T18:11:48.637 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[41334]: fuse finished with error 0 and tester_r 0 2024-01-15T18:11:48.785 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:11:48.785 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:11:48.795 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:11:48.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.932+0000 7f0ea8c00640 1 Processor -- start 2024-01-15T18:11:48.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.932+0000 7f0ea8c00640 1 -- start start 2024-01-15T18:11:48.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.936+0000 7f0ea8c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0ea415d2c0 0x7f0ea40c62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:48.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.936+0000 7f0ea8c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0ea406ac00 0x7f0ea406d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:48.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.936+0000 7f0ea8c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea40c5d00 con 0x7f0ea40c6800 2024-01-15T18:11:48.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.936+0000 7f0ea8c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ea40ce600 con 0x7f0ea406ac00 2024-01-15T18:11:48.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.936+0000 7f0ea8c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ea406fc90 con 0x7f0ea415d2c0 2024-01-15T18:11:48.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.936+0000 7f0ea2800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0ea406ac00 0x7f0ea406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:48.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.936+0000 7f0ea2800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0ea406ac00 0x7f0ea406d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58108/0 (socket says 172.21.15.27:58108) 2024-01-15T18:11:48.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.936+0000 7f0ea2800640 1 -- 172.21.15.27:0/3934318538 learned_addr learned my addr 172.21.15.27:0/3934318538 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:48.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.936+0000 7f0ea2800640 1 -- 172.21.15.27:0/3934318538 >> v2:172.21.15.93:3300/0 conn(0x7f0ea415d2c0 msgr2=0x7f0ea40c62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:48.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.936+0000 7f0ea1e00640 1 --2- 172.21.15.27:0/3934318538 >> v2:172.21.15.93:3300/0 conn(0x7f0ea415d2c0 0x7f0ea40c62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:48.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.936+0000 7f0ea2800640 1 --2- 172.21.15.27:0/3934318538 >> v2:172.21.15.93:3300/0 conn(0x7f0ea415d2c0 0x7f0ea40c62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:48.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.936+0000 7f0ea2800640 1 -- 172.21.15.27:0/3934318538 >> v1:172.21.15.93:6789/0 conn(0x7f0ea40c6800 legacy=0x7f0ea406a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:48.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.936+0000 7f0ea2800640 1 -- 172.21.15.27:0/3934318538 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea40c50f0 con 0x7f0ea406ac00 2024-01-15T18:11:48.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.936+0000 7f0ea1e00640 1 --2- 172.21.15.27:0/3934318538 >> v2:172.21.15.93:3300/0 conn(0x7f0ea415d2c0 0x7f0ea40c62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:11:48.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.940+0000 7f0ea2800640 1 --2- 172.21.15.27:0/3934318538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea406ac00 0x7f0ea406d390 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4070fa0 tx=0x7f0e9c01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b74d413304b4baf4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:48.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.940+0000 7f0ea0a00640 1 -- 172.21.15.27:0/3934318538 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c0a5820 con 0x7f0ea406ac00 2024-01-15T18:11:48.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.940+0000 7f0ea0a00640 1 -- 172.21.15.27:0/3934318538 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9c0a5980 con 0x7f0ea406ac00 2024-01-15T18:11:48.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.940+0000 7f0ea0a00640 1 -- 172.21.15.27:0/3934318538 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c0a5d10 con 0x7f0ea406ac00 2024-01-15T18:11:48.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.940+0000 7f0ea8c00640 1 -- 172.21.15.27:0/3934318538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea406ac00 msgr2=0x7f0ea406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:48.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.940+0000 7f0ea8c00640 1 --2- 172.21.15.27:0/3934318538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea406ac00 0x7f0ea406d390 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4070fa0 tx=0x7f0e9c01c960 comp rx=0 tx=0).stop 2024-01-15T18:11:48.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.940+0000 7f0ea8c00640 1 -- 172.21.15.27:0/3934318538 shutdown_connections 2024-01-15T18:11:48.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.940+0000 7f0ea8c00640 1 --2- 172.21.15.27:0/3934318538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea406ac00 0x7f0ea406d390 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:48.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.940+0000 7f0ea8c00640 1 --2- 172.21.15.27:0/3934318538 >> v2:172.21.15.93:3300/0 conn(0x7f0ea415d2c0 0x7f0ea40c62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:48.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.940+0000 7f0ea8c00640 1 -- 172.21.15.27:0/3934318538 >> 172.21.15.27:0/3934318538 conn(0x7f0ea40cfd60 msgr2=0x7f0ea40d0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:48.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.940+0000 7f0ea8c00640 1 -- 172.21.15.27:0/3934318538 shutdown_connections 2024-01-15T18:11:48.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.940+0000 7f0ea8c00640 1 -- 172.21.15.27:0/3934318538 wait complete. 2024-01-15T18:11:48.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.940+0000 7f0ea8c00640 1 Processor -- start 2024-01-15T18:11:48.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.944+0000 7f0ea8c00640 1 -- start start 2024-01-15T18:11:48.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.944+0000 7f0ea8c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ea406ac00 0x7f0ea40c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:48.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.944+0000 7f0ea8c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ea40c6800 0x7f0ea41509c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:48.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.944+0000 7f0ea8c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea415d2c0 0x7f0ea4158e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:48.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.944+0000 7f0ea8c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea41527c0 con 0x7f0ea415d2c0 2024-01-15T18:11:48.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.944+0000 7f0ea8c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea4152a30 con 0x7f0ea406ac00 2024-01-15T18:11:48.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.944+0000 7f0ea8c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea40c6e30 con 0x7f0ea40c6800 2024-01-15T18:11:48.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.944+0000 7f0ea1e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ea406ac00 0x7f0ea40c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:48.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.944+0000 7f0ea2800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea415d2c0 0x7f0ea4158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:48.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.944+0000 7f0ea1400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ea40c6800 0x7f0ea41509c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:48.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.944+0000 7f0ea1e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ea406ac00 0x7f0ea40c98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:46208/0 (socket says 172.21.15.27:46208) 2024-01-15T18:11:48.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.944+0000 7f0ea1e00640 1 -- 172.21.15.27:0/2288076644 learned_addr learned my addr 172.21.15.27:0/2288076644 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:48.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.944+0000 7f0ea1e00640 1 -- 172.21.15.27:0/2288076644 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ea40c6800 msgr2=0x7f0ea41509c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:48.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.944+0000 7f0ea1e00640 1 --2- 172.21.15.27:0/2288076644 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ea40c6800 0x7f0ea41509c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:48.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.944+0000 7f0ea1e00640 1 -- 172.21.15.27:0/2288076644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea415d2c0 msgr2=0x7f0ea4158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:48.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.944+0000 7f0ea1e00640 1 --2- 172.21.15.27:0/2288076644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea415d2c0 0x7f0ea4158e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:48.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.944+0000 7f0ea1e00640 1 -- 172.21.15.27:0/2288076644 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e9c009b30 con 0x7f0ea406ac00 2024-01-15T18:11:48.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.944+0000 7f0ea1400640 1 --2- 172.21.15.27:0/2288076644 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ea40c6800 0x7f0ea41509c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:11:48.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.944+0000 7f0ea1e00640 1 --2- 172.21.15.27:0/2288076644 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ea406ac00 0x7f0ea40c98e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400c7f0 tx=0x7f0e9400ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:48.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.944+0000 7f0e9a000640 1 -- 172.21.15.27:0/2288076644 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e94007820 con 0x7f0ea406ac00 2024-01-15T18:11:48.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.944+0000 7f0e9a000640 1 -- 172.21.15.27:0/2288076644 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e94007980 con 0x7f0ea406ac00 2024-01-15T18:11:48.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.944+0000 7f0ea8c00640 1 -- 172.21.15.27:0/2288076644 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ea40c7110 con 0x7f0ea406ac00 2024-01-15T18:11:48.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.944+0000 7f0e9a000640 1 -- 172.21.15.27:0/2288076644 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e94007d30 con 0x7f0ea406ac00 2024-01-15T18:11:48.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.944+0000 7f0ea8c00640 1 -- 172.21.15.27:0/2288076644 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ea4150f00 con 0x7f0ea406ac00 2024-01-15T18:11:48.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.944+0000 7f0ea8c00640 1 -- 172.21.15.27:0/2288076644 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ea4067c00 con 0x7f0ea406ac00 2024-01-15T18:11:48.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.948+0000 7f0e9a000640 1 -- 172.21.15.27:0/2288076644 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f0e940113f0 con 0x7f0ea406ac00 2024-01-15T18:11:48.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.948+0000 7f0e9a000640 1 --2- 172.21.15.27:0/2288076644 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0e7c0670d0 0x7f0e7c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:48.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.948+0000 7f0e9a000640 1 -- 172.21.15.27:0/2288076644 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4999+0+0 (secure 0 0 0) 0x7f0e94082d50 con 0x7f0ea406ac00 2024-01-15T18:11:48.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.948+0000 7f0ea1400640 1 --2- 172.21.15.27:0/2288076644 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0e7c0670d0 0x7f0e7c069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:48.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.948+0000 7f0ea1400640 1 --2- 172.21.15.27:0/2288076644 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0e7c0670d0 0x7f0e7c069590 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f0ea40cef20 tx=0x7f0e8c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:48.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:48.952+0000 7f0e9a000640 1 -- 172.21.15.27:0/2288076644 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0e94051c40 con 0x7f0ea406ac00 2024-01-15T18:11:49.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.108+0000 7f0ea8c00640 1 -- 172.21.15.27:0/2288076644 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ea406d730 con 0x7f0ea406ac00 2024-01-15T18:11:49.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.108+0000 7f0e9a000640 1 -- 172.21.15.27:0/2288076644 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+12977 (secure 0 0 0) 0x7f0ea406d730 con 0x7f0ea406ac00 2024-01-15T18:11:49.111 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:11:49.111 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":62,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:11:47.160522+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6834/1969845074":"2024-01-16T18:11:44.135277+0000","172.21.15.27:6832/1969845074":"2024-01-16T18:11:44.135277+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:11:49.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.112+0000 7f0ea8c00640 1 -- 172.21.15.27:0/2288076644 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0e7c0670d0 msgr2=0x7f0e7c069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:49.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.112+0000 7f0ea8c00640 1 --2- 172.21.15.27:0/2288076644 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0e7c0670d0 0x7f0e7c069590 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f0ea40cef20 tx=0x7f0e8c009040 comp rx=0 tx=0).stop 2024-01-15T18:11:49.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.112+0000 7f0ea8c00640 1 -- 172.21.15.27:0/2288076644 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ea406ac00 msgr2=0x7f0ea40c98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:49.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.112+0000 7f0ea8c00640 1 --2- 172.21.15.27:0/2288076644 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ea406ac00 0x7f0ea40c98e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400c7f0 tx=0x7f0e9400ccb0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.112+0000 7f0ea8c00640 1 -- 172.21.15.27:0/2288076644 shutdown_connections 2024-01-15T18:11:49.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.112+0000 7f0ea8c00640 1 --2- 172.21.15.27:0/2288076644 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0e7c0670d0 0x7f0e7c069590 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.112+0000 7f0ea8c00640 1 --2- 172.21.15.27:0/2288076644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea415d2c0 0x7f0ea4158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.112+0000 7f0ea8c00640 1 --2- 172.21.15.27:0/2288076644 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ea40c6800 0x7f0ea41509c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.112+0000 7f0ea8c00640 1 --2- 172.21.15.27:0/2288076644 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ea406ac00 0x7f0ea40c98e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.112+0000 7f0ea8c00640 1 -- 172.21.15.27:0/2288076644 >> 172.21.15.27:0/2288076644 conn(0x7f0ea40cfd60 msgr2=0x7f0ea40d0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:49.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.112+0000 7f0ea8c00640 1 -- 172.21.15.27:0/2288076644 shutdown_connections 2024-01-15T18:11:49.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.112+0000 7f0ea8c00640 1 -- 172.21.15.27:0/2288076644 wait complete. 2024-01-15T18:11:49.134 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:11:49.135 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:11:49.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.280+0000 7f2e9f600640 1 Processor -- start 2024-01-15T18:11:49.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.280+0000 7f2e9f600640 1 -- start start 2024-01-15T18:11:49.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e9f600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2e9815d570 0x7f2e9806be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:49.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e9f600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f2e9806b810 0x7f2e9806c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:49.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e9f600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e980b0d70 con 0x7f2e9815d190 2024-01-15T18:11:49.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e9f600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e980d0540 con 0x7f2e9806b810 2024-01-15T18:11:49.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e9f600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e980b0440 con 0x7f2e9815d570 2024-01-15T18:11:49.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e9cc00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2e9815d190 0x7f2e980cfcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46316/0 (socket says 172.21.15.27:46316) 2024-01-15T18:11:49.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e9cc00640 1 -- 172.21.15.27:0/1747532837 learned_addr learned my addr 172.21.15.27:0/1747532837 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:49.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e97e00640 1 --2- 172.21.15.27:0/1747532837 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2e9815d570 0x7f2e9806be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:49.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e9d600640 1 --2- 172.21.15.27:0/1747532837 >> v2:172.21.15.93:3300/0 conn(0x7f2e9806b810 0x7f2e9806c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:49.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e97400640 1 -- 172.21.15.27:0/1747532837 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 838618308 0 0) 0x7f2e980b0d70 con 0x7f2e9815d190 2024-01-15T18:11:49.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e97400640 1 -- 172.21.15.27:0/1747532837 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e78003530 con 0x7f2e9815d190 2024-01-15T18:11:49.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e97400640 1 -- 172.21.15.27:0/1747532837 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 786347005 0 0) 0x7f2e78003530 con 0x7f2e9815d190 2024-01-15T18:11:49.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e97400640 1 -- 172.21.15.27:0/1747532837 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2e9815d570 msgr2=0x7f2e9806be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:49.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e97400640 1 --2- 172.21.15.27:0/1747532837 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2e9815d570 0x7f2e9806be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e97400640 1 -- 172.21.15.27:0/1747532837 >> v2:172.21.15.93:3300/0 conn(0x7f2e9806b810 msgr2=0x7f2e9806c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:49.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e97400640 1 --2- 172.21.15.27:0/1747532837 >> v2:172.21.15.93:3300/0 conn(0x7f2e9806b810 0x7f2e9806c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e97400640 1 -- 172.21.15.27:0/1747532837 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e980b0720 con 0x7f2e9815d190 2024-01-15T18:11:49.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e97400640 1 -- 172.21.15.27:0/1747532837 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f2e88001680 con 0x7f2e9815d190 2024-01-15T18:11:49.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e97400640 1 -- 172.21.15.27:0/1747532837 >> v1:172.21.15.27:6789/0 conn(0x7f2e9815d190 legacy=0x7f2e980cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:49.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e97400640 1 --2- 172.21.15.27:0/1747532837 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2e780039f0 0x7f2e78003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:49.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e97400640 1 --2- 172.21.15.27:0/1747532837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e780047d0 0x7f2e78004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:49.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e97400640 1 --2- 172.21.15.27:0/1747532837 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2e78005190 0x7f2e7800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:49.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e97400640 1 -- 172.21.15.27:0/1747532837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e980d0540 con 0x7f2e780047d0 2024-01-15T18:11:49.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e97400640 1 -- 172.21.15.27:0/1747532837 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e980b0440 con 0x7f2e780039f0 2024-01-15T18:11:49.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e97400640 1 -- 172.21.15.27:0/1747532837 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7800bed0 con 0x7f2e78005190 2024-01-15T18:11:49.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e9d600640 1 --2- 172.21.15.27:0/1747532837 >> v2:172.21.15.93:3300/0 conn(0x7f2e9806b810 0x7f2e9806c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:11:49.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e97e00640 1 --2- 172.21.15.27:0/1747532837 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2e9815d570 0x7f2e9806be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:11:49.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e97e00640 1 --2- 172.21.15.27:0/1747532837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e780047d0 0x7f2e78004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:49.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e9cc00640 1 --2- 172.21.15.27:0/1747532837 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2e780039f0 0x7f2e78003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:49.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e9d600640 1 --2- 172.21.15.27:0/1747532837 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2e78005190 0x7f2e7800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:49.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e97e00640 1 -- 172.21.15.27:0/1747532837 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2e78005190 msgr2=0x7f2e7800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:49.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e97e00640 1 --2- 172.21.15.27:0/1747532837 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2e78005190 0x7f2e7800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e97e00640 1 -- 172.21.15.27:0/1747532837 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2e780039f0 msgr2=0x7f2e78003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:49.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e97e00640 1 --2- 172.21.15.27:0/1747532837 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2e780039f0 0x7f2e78003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.284+0000 7f2e97e00640 1 -- 172.21.15.27:0/1747532837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e78003530 con 0x7f2e780047d0 2024-01-15T18:11:49.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.288+0000 7f2e97e00640 1 --2- 172.21.15.27:0/1747532837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e780047d0 0x7f2e78004c50 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f2e80009b20 tx=0x7f2e8001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7f4bd1009813438 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:49.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.288+0000 7f2e97400640 1 -- 172.21.15.27:0/1747532837 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e80004030 con 0x7f2e780047d0 2024-01-15T18:11:49.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.288+0000 7f2e97400640 1 -- 172.21.15.27:0/1747532837 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e80004190 con 0x7f2e780047d0 2024-01-15T18:11:49.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.288+0000 7f2e97400640 1 -- 172.21.15.27:0/1747532837 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e80004520 con 0x7f2e780047d0 2024-01-15T18:11:49.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.288+0000 7f2e9f600640 1 -- 172.21.15.27:0/1747532837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e780047d0 msgr2=0x7f2e78004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:49.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.288+0000 7f2e9f600640 1 --2- 172.21.15.27:0/1747532837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e780047d0 0x7f2e78004c50 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f2e80009b20 tx=0x7f2e8001ca90 comp rx=0 tx=0).stop 2024-01-15T18:11:49.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.288+0000 7f2e9d600640 1 -- 172.21.15.27:0/1747532837 reap_dead start 2024-01-15T18:11:49.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.288+0000 7f2e9f600640 1 -- 172.21.15.27:0/1747532837 shutdown_connections 2024-01-15T18:11:49.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.288+0000 7f2e9f600640 1 -- 172.21.15.27:0/1747532837 >> 172.21.15.27:0/1747532837 conn(0x7f2e980d0e70 msgr2=0x7f2e980d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:49.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.288+0000 7f2e9f600640 1 -- 172.21.15.27:0/1747532837 shutdown_connections 2024-01-15T18:11:49.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.288+0000 7f2e9f600640 1 -- 172.21.15.27:0/1747532837 wait complete. 2024-01-15T18:11:49.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.292+0000 7f2e9f600640 1 Processor -- start 2024-01-15T18:11:49.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.292+0000 7f2e9f600640 1 -- start start 2024-01-15T18:11:49.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.292+0000 7f2e9f600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e780047d0 0x7f2e98151520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:49.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.292+0000 7f2e9f600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2e980c67d0 0x7f2e980b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:49.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.292+0000 7f2e9f600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2e980b1200 0x7f2e980b1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:49.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.292+0000 7f2e9f600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e980b5ba0 con 0x7f2e780047d0 2024-01-15T18:11:49.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.292+0000 7f2e9f600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e980b1bc0 con 0x7f2e980b1200 2024-01-15T18:11:49.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.292+0000 7f2e9f600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e980b4f20 con 0x7f2e980c67d0 2024-01-15T18:11:49.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.292+0000 7f2e9cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e780047d0 0x7f2e98151520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:49.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.292+0000 7f2e9cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e780047d0 0x7f2e98151520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58136/0 (socket says 172.21.15.27:58136) 2024-01-15T18:11:49.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.292+0000 7f2e9cc00640 1 -- 172.21.15.27:0/1855494712 learned_addr learned my addr 172.21.15.27:0/1855494712 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:49.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.292+0000 7f2e9d600640 1 --2- 172.21.15.27:0/1855494712 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2e980b1200 0x7f2e980b1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:49.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.292+0000 7f2e97e00640 1 --2- 172.21.15.27:0/1855494712 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2e980c67d0 0x7f2e980b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:49.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.292+0000 7f2e9cc00640 1 -- 172.21.15.27:0/1855494712 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2e980c67d0 msgr2=0x7f2e980b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:49.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.292+0000 7f2e9cc00640 1 --2- 172.21.15.27:0/1855494712 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2e980c67d0 0x7f2e980b46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.292+0000 7f2e9cc00640 1 -- 172.21.15.27:0/1855494712 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2e980b1200 msgr2=0x7f2e980b1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:49.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.292+0000 7f2e9cc00640 1 --2- 172.21.15.27:0/1855494712 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2e980b1200 0x7f2e980b1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.292+0000 7f2e9cc00640 1 -- 172.21.15.27:0/1855494712 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e980b5200 con 0x7f2e780047d0 2024-01-15T18:11:49.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.292+0000 7f2e9cc00640 1 --2- 172.21.15.27:0/1855494712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e780047d0 0x7f2e98151520 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f2e880122e0 tx=0x7f2e880127a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:49.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.292+0000 7f2e94c00640 1 -- 172.21.15.27:0/1855494712 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e88014750 con 0x7f2e780047d0 2024-01-15T18:11:49.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.292+0000 7f2e94c00640 1 -- 172.21.15.27:0/1855494712 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e880148b0 con 0x7f2e780047d0 2024-01-15T18:11:49.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.292+0000 7f2e9f600640 1 -- 172.21.15.27:0/1855494712 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e980b54e0 con 0x7f2e780047d0 2024-01-15T18:11:49.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.292+0000 7f2e94c00640 1 -- 172.21.15.27:0/1855494712 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e88014c60 con 0x7f2e780047d0 2024-01-15T18:11:49.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.292+0000 7f2e9f600640 1 -- 172.21.15.27:0/1855494712 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e980b4c10 con 0x7f2e780047d0 2024-01-15T18:11:49.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.292+0000 7f2e9f600640 1 -- 172.21.15.27:0/1855494712 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e980b5060 con 0x7f2e780047d0 2024-01-15T18:11:49.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.296+0000 7f2e94c00640 1 -- 172.21.15.27:0/1855494712 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f2e880153e0 con 0x7f2e780047d0 2024-01-15T18:11:49.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.296+0000 7f2e94c00640 1 --2- 172.21.15.27:0/1855494712 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2e700670d0 0x7f2e70069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:49.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.296+0000 7f2e94c00640 1 -- 172.21.15.27:0/1855494712 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4999+0+0 (secure 0 0 0) 0x7f2e880855f0 con 0x7f2e780047d0 2024-01-15T18:11:49.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.296+0000 7f2e97e00640 1 --2- 172.21.15.27:0/1855494712 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2e700670d0 0x7f2e70069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:49.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.296+0000 7f2e97e00640 1 --2- 172.21.15.27:0/1855494712 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2e700670d0 0x7f2e70069590 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2e80009b20 tx=0x7f2e800046d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:49.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.300+0000 7f2e94c00640 1 -- 172.21.15.27:0/1855494712 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2e980b5060 con 0x7f2e780047d0 2024-01-15T18:11:49.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.460+0000 7f2e9f600640 1 -- 172.21.15.27:0/1855494712 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2e980b5060 con 0x7f2e780047d0 2024-01-15T18:11:49.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.460+0000 7f2e94c00640 1 -- 172.21.15.27:0/1855494712 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v62) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2e980b5060 con 0x7f2e780047d0 2024-01-15T18:11:49.461 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:11:49.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.460+0000 7f2e9f600640 1 -- 172.21.15.27:0/1855494712 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2e700670d0 msgr2=0x7f2e70069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:49.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.460+0000 7f2e9f600640 1 --2- 172.21.15.27:0/1855494712 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2e700670d0 0x7f2e70069590 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2e80009b20 tx=0x7f2e800046d0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.460+0000 7f2e9f600640 1 -- 172.21.15.27:0/1855494712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e780047d0 msgr2=0x7f2e98151520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:49.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.460+0000 7f2e9f600640 1 --2- 172.21.15.27:0/1855494712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e780047d0 0x7f2e98151520 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f2e880122e0 tx=0x7f2e880127a0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.460+0000 7f2e9f600640 1 -- 172.21.15.27:0/1855494712 shutdown_connections 2024-01-15T18:11:49.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.460+0000 7f2e9f600640 1 --2- 172.21.15.27:0/1855494712 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2e700670d0 0x7f2e70069590 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.460+0000 7f2e9f600640 1 --2- 172.21.15.27:0/1855494712 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2e980b1200 0x7f2e980b1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.460+0000 7f2e9f600640 1 --2- 172.21.15.27:0/1855494712 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2e980c67d0 0x7f2e980b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.460+0000 7f2e9f600640 1 --2- 172.21.15.27:0/1855494712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e780047d0 0x7f2e98151520 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.460+0000 7f2e9f600640 1 -- 172.21.15.27:0/1855494712 >> 172.21.15.27:0/1855494712 conn(0x7f2e980d0e70 msgr2=0x7f2e9806f470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:49.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.460+0000 7f2e9f600640 1 -- 172.21.15.27:0/1855494712 shutdown_connections 2024-01-15T18:11:49.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.460+0000 7f2e9f600640 1 -- 172.21.15.27:0/1855494712 wait complete. 2024-01-15T18:11:49.485 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:11:49.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.624+0000 7f15e6800640 1 Processor -- start 2024-01-15T18:11:49.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.624+0000 7f15e6800640 1 -- start start 2024-01-15T18:11:49.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.628+0000 7f15e6800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f15e009a120 0x7f15e00cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:49.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.628+0000 7f15e6800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e009a700 con 0x7f15e015e7d0 2024-01-15T18:11:49.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.628+0000 7f15e6800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e00cd590 con 0x7f15e00b35a0 2024-01-15T18:11:49.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.628+0000 7f15e6800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15e00cd870 con 0x7f15e009a120 2024-01-15T18:11:49.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.628+0000 7f15df400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f15e015e7d0 0x7f15e0099a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46324/0 (socket says 172.21.15.27:46324) 2024-01-15T18:11:49.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.628+0000 7f15df400640 1 -- 172.21.15.27:0/3117398997 learned_addr learned my addr 172.21.15.27:0/3117398997 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:49.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.628+0000 7f15dea00640 1 --2- 172.21.15.27:0/3117398997 >> v2:172.21.15.93:3300/0 conn(0x7f15e009a120 0x7f15e00cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:49.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.628+0000 7f15de000640 1 -- 172.21.15.27:0/3117398997 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834874797 0 0) 0x7f15e009a700 con 0x7f15e015e7d0 2024-01-15T18:11:49.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.628+0000 7f15de000640 1 -- 172.21.15.27:0/3117398997 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15c0003530 con 0x7f15e015e7d0 2024-01-15T18:11:49.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.628+0000 7f15de000640 1 -- 172.21.15.27:0/3117398997 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1353097071 0 0) 0x7f15c0003530 con 0x7f15e015e7d0 2024-01-15T18:11:49.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.628+0000 7f15dea00640 1 -- 172.21.15.27:0/3117398997 >> v1:172.21.15.93:6789/0 conn(0x7f15e00b35a0 legacy=0x7f15e00cc530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:49.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.628+0000 7f15dea00640 1 -- 172.21.15.27:0/3117398997 >> v1:172.21.15.27:6789/0 conn(0x7f15e015e7d0 legacy=0x7f15e0099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:49.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.628+0000 7f15dea00640 1 -- 172.21.15.27:0/3117398997 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e00cfa00 con 0x7f15e009a120 2024-01-15T18:11:49.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.632+0000 7f15dea00640 1 --2- 172.21.15.27:0/3117398997 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f15e009a120 0x7f15e00cbff0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f15c80097c0 tx=0x7f15c801c370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c35ae3bbfad8058 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:49.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.632+0000 7f15de000640 1 -- 172.21.15.27:0/3117398997 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d0001680 con 0x7f15e009a120 2024-01-15T18:11:49.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.632+0000 7f15de000640 1 -- 172.21.15.27:0/3117398997 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c80ac440 con 0x7f15e009a120 2024-01-15T18:11:49.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.632+0000 7f15de000640 1 -- 172.21.15.27:0/3117398997 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c80ac7d0 con 0x7f15e009a120 2024-01-15T18:11:49.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.632+0000 7f15e6800640 1 -- 172.21.15.27:0/3117398997 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f15e009a120 msgr2=0x7f15e00cbff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:49.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.632+0000 7f15e6800640 1 --2- 172.21.15.27:0/3117398997 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f15e009a120 0x7f15e00cbff0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f15c80097c0 tx=0x7f15c801c370 comp rx=0 tx=0).stop 2024-01-15T18:11:49.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.632+0000 7f15e6800640 1 -- 172.21.15.27:0/3117398997 shutdown_connections 2024-01-15T18:11:49.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.632+0000 7f15e6800640 1 --2- 172.21.15.27:0/3117398997 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f15e009a120 0x7f15e00cbff0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.632+0000 7f15e6800640 1 -- 172.21.15.27:0/3117398997 >> 172.21.15.27:0/3117398997 conn(0x7f15e00b2a80 msgr2=0x7f15e00b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:49.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.632+0000 7f15e6800640 1 -- 172.21.15.27:0/3117398997 shutdown_connections 2024-01-15T18:11:49.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.632+0000 7f15e6800640 1 -- 172.21.15.27:0/3117398997 wait complete. 2024-01-15T18:11:49.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.632+0000 7f15e6800640 1 Processor -- start 2024-01-15T18:11:49.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.632+0000 7f15e6800640 1 -- start start 2024-01-15T18:11:49.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.636+0000 7f15e6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15e009a120 0x7f15e006bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.636+0000 7f15e6800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f15e00b35a0 0x7f15e00c7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.636+0000 7f15e6800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f15e015e7d0 0x7f15e00b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.636+0000 7f15e6800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e0157620 con 0x7f15e009a120 2024-01-15T18:11:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.636+0000 7f15e6800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e006c910 con 0x7f15e00b35a0 2024-01-15T18:11:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.636+0000 7f15e6800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e006cbf0 con 0x7f15e015e7d0 2024-01-15T18:11:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.636+0000 7f15df400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15e009a120 0x7f15e006bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.636+0000 7f15df400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15e009a120 0x7f15e006bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58138/0 (socket says 172.21.15.27:58138) 2024-01-15T18:11:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.636+0000 7f15df400640 1 -- 172.21.15.27:0/3032436440 learned_addr learned my addr 172.21.15.27:0/3032436440 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.636+0000 7f15dea00640 1 --2- 172.21.15.27:0/3032436440 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f15e00b35a0 0x7f15e00c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.636+0000 7f15dfe00640 1 --2- 172.21.15.27:0/3032436440 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f15e015e7d0 0x7f15e00b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.636+0000 7f15df400640 1 -- 172.21.15.27:0/3032436440 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f15e015e7d0 msgr2=0x7f15e00b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.636+0000 7f15df400640 1 --2- 172.21.15.27:0/3032436440 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f15e015e7d0 0x7f15e00b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.636+0000 7f15df400640 1 -- 172.21.15.27:0/3032436440 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f15e00b35a0 msgr2=0x7f15e00c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.636+0000 7f15df400640 1 --2- 172.21.15.27:0/3032436440 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f15e00b35a0 0x7f15e00c7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.636+0000 7f15df400640 1 -- 172.21.15.27:0/3032436440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c8009420 con 0x7f15e009a120 2024-01-15T18:11:49.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.636+0000 7f15df400640 1 --2- 172.21.15.27:0/3032436440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15e009a120 0x7f15e006bfc0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f15d000c750 tx=0x7f15d000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:49.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.636+0000 7f15d7400640 1 -- 172.21.15.27:0/3032436440 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d0005ae0 con 0x7f15e009a120 2024-01-15T18:11:49.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.636+0000 7f15d7400640 1 -- 172.21.15.27:0/3032436440 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d0005c40 con 0x7f15e009a120 2024-01-15T18:11:49.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.636+0000 7f15d7400640 1 -- 172.21.15.27:0/3032436440 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d00072b0 con 0x7f15e009a120 2024-01-15T18:11:49.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.636+0000 7f15e6800640 1 -- 172.21.15.27:0/3032436440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e006ced0 con 0x7f15e009a120 2024-01-15T18:11:49.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.636+0000 7f15e6800640 1 -- 172.21.15.27:0/3032436440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e006c500 con 0x7f15e009a120 2024-01-15T18:11:49.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.636+0000 7f15e6800640 1 -- 172.21.15.27:0/3032436440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e004e850 con 0x7f15e009a120 2024-01-15T18:11:49.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.640+0000 7f15d7400640 1 -- 172.21.15.27:0/3032436440 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f15d0007410 con 0x7f15e009a120 2024-01-15T18:11:49.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.640+0000 7f15d7400640 1 --2- 172.21.15.27:0/3032436440 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f15b80670d0 0x7f15b8069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:49.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.640+0000 7f15d7400640 1 -- 172.21.15.27:0/3032436440 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4999+0+0 (secure 0 0 0) 0x7f15d0082240 con 0x7f15e009a120 2024-01-15T18:11:49.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.640+0000 7f15dea00640 1 --2- 172.21.15.27:0/3032436440 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f15b80670d0 0x7f15b8069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:49.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.640+0000 7f15dea00640 1 --2- 172.21.15.27:0/3032436440 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f15b80670d0 0x7f15b8069590 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f15c80ae000 tx=0x7f15c80a6070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:49.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.644+0000 7f15d7400640 1 -- 172.21.15.27:0/3032436440 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f15d00510c0 con 0x7f15e009a120 2024-01-15T18:11:49.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.808+0000 7f15e6800640 1 -- 172.21.15.27:0/3032436440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f15e00cd0e0 con 0x7f15e009a120 2024-01-15T18:11:49.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.812+0000 7f15d7400640 1 -- 172.21.15.27:0/3032436440 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v62) v1 ==== 80+0+122 (secure 0 0 0) 0x7f15d0017090 con 0x7f15e009a120 2024-01-15T18:11:49.812 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6834/1969845074 2024-01-16T18:11:44.135277+0000 2024-01-15T18:11:49.812 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6832/1969845074 2024-01-16T18:11:44.135277+0000 2024-01-15T18:11:49.812 INFO:teuthology.orchestra.run.smithi027.stderr:listed 2 entries 2024-01-15T18:11:49.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.812+0000 7f15e6800640 1 -- 172.21.15.27:0/3032436440 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f15b80670d0 msgr2=0x7f15b8069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:49.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.812+0000 7f15e6800640 1 --2- 172.21.15.27:0/3032436440 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f15b80670d0 0x7f15b8069590 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f15c80ae000 tx=0x7f15c80a6070 comp rx=0 tx=0).stop 2024-01-15T18:11:49.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.812+0000 7f15e6800640 1 -- 172.21.15.27:0/3032436440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15e009a120 msgr2=0x7f15e006bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:49.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.812+0000 7f15e6800640 1 --2- 172.21.15.27:0/3032436440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15e009a120 0x7f15e006bfc0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f15d000c750 tx=0x7f15d000cc10 comp rx=0 tx=0).stop 2024-01-15T18:11:49.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.812+0000 7f15e6800640 1 -- 172.21.15.27:0/3032436440 shutdown_connections 2024-01-15T18:11:49.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.812+0000 7f15e6800640 1 --2- 172.21.15.27:0/3032436440 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f15b80670d0 0x7f15b8069590 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.812+0000 7f15e6800640 1 --2- 172.21.15.27:0/3032436440 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f15e015e7d0 0x7f15e00b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.812+0000 7f15e6800640 1 --2- 172.21.15.27:0/3032436440 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f15e00b35a0 0x7f15e00c7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.812+0000 7f15e6800640 1 --2- 172.21.15.27:0/3032436440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15e009a120 0x7f15e006bfc0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.812+0000 7f15e6800640 1 -- 172.21.15.27:0/3032436440 >> 172.21.15.27:0/3032436440 conn(0x7f15e00b2a80 msgr2=0x7f15e00b2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:49.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.812+0000 7f15e6800640 1 -- 172.21.15.27:0/3032436440 shutdown_connections 2024-01-15T18:11:49.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:49.812+0000 7f15e6800640 1 -- 172.21.15.27:0/3032436440 wait complete. 2024-01-15T18:11:49.834 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi093.front.sepia.ceph.com... 2024-01-15T18:11:49.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:11:49.835 DEBUG:teuthology.orchestra.run.smithi093:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-15T18:11:49.854 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:ceph-fuse[33741]: fuse finished with error 0 and tester_r 0 2024-01-15T18:11:49.856 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T18:11:49.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:11:49.856 DEBUG:teuthology.orchestra.run.smithi093:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-15T18:11:50.026 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:11:50.026 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:11:50.035 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354580k,nr_inodes=4088645,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9249 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:11:50.035 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T18:11:50.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:11:50.035 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-15T18:11:50.082 INFO:teuthology.orchestra.run.smithi027.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-15T18:11:50.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:11:50.084 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:11:50.084 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:11:50.092 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354484k,nr_inodes=4088621,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8403 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:11:50.093 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T18:11:50.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:11:50.093 DEBUG:teuthology.orchestra.run.smithi093:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-15T18:11:50.143 INFO:teuthology.orchestra.run.smithi093.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-15T18:11:50.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:11:50.144 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:11:50.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.284+0000 7fb605800640 1 Processor -- start 2024-01-15T18:11:50.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.284+0000 7fb605800640 1 -- start start 2024-01-15T18:11:50.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.284+0000 7fb605800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb60015e810 0x7fb6000afc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:50.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.288+0000 7fb605800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6000b18a0 con 0x7fb6000cf6c0 2024-01-15T18:11:50.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.288+0000 7fb605800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6000b27f0 con 0x7fb6000b2070 2024-01-15T18:11:50.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.288+0000 7fb605800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6000b2510 con 0x7fb60015e810 2024-01-15T18:11:50.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.288+0000 7fb5ff400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb6000cf6c0 0x7fb6000adc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46338/0 (socket says 172.21.15.27:46338) 2024-01-15T18:11:50.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.288+0000 7fb5ff400640 1 -- 172.21.15.27:0/1001315028 learned_addr learned my addr 172.21.15.27:0/1001315028 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:50.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.288+0000 7fb5fea00640 1 --2- 172.21.15.27:0/1001315028 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb60015e810 0x7fb6000afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:50.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.288+0000 7fb5fd600640 1 -- 172.21.15.27:0/1001315028 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4087088539 0 0) 0x7fb6000b18a0 con 0x7fb6000cf6c0 2024-01-15T18:11:50.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.288+0000 7fb5fd600640 1 -- 172.21.15.27:0/1001315028 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5e0003530 con 0x7fb6000cf6c0 2024-01-15T18:11:50.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.288+0000 7fb5fd600640 1 -- 172.21.15.27:0/1001315028 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194185052 0 0) 0x7fb6000b27f0 con 0x7fb6000b2070 2024-01-15T18:11:50.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.288+0000 7fb5fd600640 1 -- 172.21.15.27:0/1001315028 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6000b18a0 con 0x7fb6000b2070 2024-01-15T18:11:50.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.288+0000 7fb5fea00640 1 -- 172.21.15.27:0/1001315028 >> v1:172.21.15.93:6789/0 conn(0x7fb6000b2070 legacy=0x7fb6000cefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:50.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.288+0000 7fb5fea00640 1 -- 172.21.15.27:0/1001315028 >> v1:172.21.15.27:6789/0 conn(0x7fb6000cf6c0 legacy=0x7fb6000adc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:50.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.288+0000 7fb5fea00640 1 -- 172.21.15.27:0/1001315028 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6000b1250 con 0x7fb60015e810 2024-01-15T18:11:50.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.288+0000 7fb5fea00640 1 --2- 172.21.15.27:0/1001315028 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb60015e810 0x7fb6000afc90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb5f00097f0 tx=0x7fb5f001c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f81e4be939a3e2f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:50.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.292+0000 7fb5fd600640 1 -- 172.21.15.27:0/1001315028 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f00ab620 con 0x7fb60015e810 2024-01-15T18:11:50.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.292+0000 7fb5fd600640 1 -- 172.21.15.27:0/1001315028 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5f00ab780 con 0x7fb60015e810 2024-01-15T18:11:50.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.292+0000 7fb5fd600640 1 -- 172.21.15.27:0/1001315028 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f00abb10 con 0x7fb60015e810 2024-01-15T18:11:50.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.292+0000 7fb605800640 1 -- 172.21.15.27:0/1001315028 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb60015e810 msgr2=0x7fb6000afc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:50.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.292+0000 7fb605800640 1 --2- 172.21.15.27:0/1001315028 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb60015e810 0x7fb6000afc90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb5f00097f0 tx=0x7fb5f001c490 comp rx=0 tx=0).stop 2024-01-15T18:11:50.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.292+0000 7fb605800640 1 -- 172.21.15.27:0/1001315028 shutdown_connections 2024-01-15T18:11:50.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.292+0000 7fb605800640 1 --2- 172.21.15.27:0/1001315028 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb60015e810 0x7fb6000afc90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.292+0000 7fb605800640 1 -- 172.21.15.27:0/1001315028 >> 172.21.15.27:0/1001315028 conn(0x7fb6000d0f20 msgr2=0x7fb6000d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:50.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.292+0000 7fb605800640 1 -- 172.21.15.27:0/1001315028 shutdown_connections 2024-01-15T18:11:50.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.292+0000 7fb605800640 1 -- 172.21.15.27:0/1001315028 wait complete. 2024-01-15T18:11:50.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.292+0000 7fb605800640 1 Processor -- start 2024-01-15T18:11:50.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.292+0000 7fb605800640 1 -- start start 2024-01-15T18:11:50.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.292+0000 7fb605800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6000b2070 0x7fb600141e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:50.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.292+0000 7fb605800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6000cf6c0 0x7fb6001527f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:50.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.292+0000 7fb605800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb60015e810 0x7fb600152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:50.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.292+0000 7fb605800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60006b600 con 0x7fb6000b2070 2024-01-15T18:11:50.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.292+0000 7fb605800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60006c460 con 0x7fb60015e810 2024-01-15T18:11:50.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.292+0000 7fb605800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60006b850 con 0x7fb6000cf6c0 2024-01-15T18:11:50.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.296+0000 7fb5fea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6000b2070 0x7fb600141e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:50.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.296+0000 7fb5fea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6000b2070 0x7fb600141e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58146/0 (socket says 172.21.15.27:58146) 2024-01-15T18:11:50.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.296+0000 7fb5fea00640 1 -- 172.21.15.27:0/1973916509 learned_addr learned my addr 172.21.15.27:0/1973916509 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:50.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.296+0000 7fb5fe000640 1 --2- 172.21.15.27:0/1973916509 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6000cf6c0 0x7fb6001527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:50.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.296+0000 7fb5ff400640 1 --2- 172.21.15.27:0/1973916509 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb60015e810 0x7fb600152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:50.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.296+0000 7fb5fe000640 1 -- 172.21.15.27:0/1973916509 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb60015e810 msgr2=0x7fb600152fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:50.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.296+0000 7fb5fe000640 1 --2- 172.21.15.27:0/1973916509 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb60015e810 0x7fb600152fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.296+0000 7fb5fe000640 1 -- 172.21.15.27:0/1973916509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6000b2070 msgr2=0x7fb600141e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:50.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.296+0000 7fb5fe000640 1 --2- 172.21.15.27:0/1973916509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6000b2070 0x7fb600141e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.296+0000 7fb5fe000640 1 -- 172.21.15.27:0/1973916509 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f0009420 con 0x7fb6000cf6c0 2024-01-15T18:11:50.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.296+0000 7fb5fea00640 1 --2- 172.21.15.27:0/1973916509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6000b2070 0x7fb600141e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:11:50.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.296+0000 7fb5fe000640 1 --2- 172.21.15.27:0/1973916509 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6000cf6c0 0x7fb6001527f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb5e800c3c0 tx=0x7fb5e800c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:50.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.296+0000 7fb5f6a00640 1 -- 172.21.15.27:0/1973916509 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e80052d0 con 0x7fb6000cf6c0 2024-01-15T18:11:50.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.296+0000 7fb5f6a00640 1 -- 172.21.15.27:0/1973916509 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e8005430 con 0x7fb6000cf6c0 2024-01-15T18:11:50.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.296+0000 7fb5f6a00640 1 -- 172.21.15.27:0/1973916509 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e8003840 con 0x7fb6000cf6c0 2024-01-15T18:11:50.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.296+0000 7fb605800640 1 -- 172.21.15.27:0/1973916509 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb60006bb30 con 0x7fb6000cf6c0 2024-01-15T18:11:50.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.296+0000 7fb605800640 1 -- 172.21.15.27:0/1973916509 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb60006c0f0 con 0x7fb6000cf6c0 2024-01-15T18:11:50.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.296+0000 7fb605800640 1 -- 172.21.15.27:0/1973916509 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb60004e850 con 0x7fb6000cf6c0 2024-01-15T18:11:50.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.296+0000 7fb5f6a00640 1 -- 172.21.15.27:0/1973916509 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7fb5e80039a0 con 0x7fb6000cf6c0 2024-01-15T18:11:50.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.300+0000 7fb5f6a00640 1 --2- 172.21.15.27:0/1973916509 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb5d80670d0 0x7fb5d8069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:50.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.300+0000 7fb5f6a00640 1 -- 172.21.15.27:0/1973916509 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4999+0+0 (secure 0 0 0) 0x7fb5e8081f40 con 0x7fb6000cf6c0 2024-01-15T18:11:50.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.300+0000 7fb5fea00640 1 --2- 172.21.15.27:0/1973916509 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb5d80670d0 0x7fb5d8069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:50.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.300+0000 7fb5fea00640 1 --2- 172.21.15.27:0/1973916509 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb5d80670d0 0x7fb5d8069590 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb5f001c4d0 tx=0x7fb5f00a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:50.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.304+0000 7fb5f6a00640 1 -- 172.21.15.27:0/1973916509 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb5e8050e30 con 0x7fb6000cf6c0 2024-01-15T18:11:50.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.496+0000 7fb605800640 1 -- 172.21.15.27:0/1973916509 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb600072620 con 0x7fb6000cf6c0 2024-01-15T18:11:50.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.496+0000 7fb5f6a00640 1 -- 172.21.15.27:0/1973916509 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+3696 (secure 0 0 0) 0x7fb5e8017090 con 0x7fb6000cf6c0 2024-01-15T18:11:50.497 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:11:50.497 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":40,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1939851998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1939851998},{"type":"v1","addr":"172.21.15.27:6835","nonce":1939851998}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":5759,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/255207804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":255207804},{"type":"v1","addr":"172.21.15.93:6837","nonce":255207804}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":6244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/1075225596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":1075225596},{"type":"v1","addr":"172.21.15.93:6836","nonce":1075225596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":7101,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/4215930867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":4215930867},{"type":"v1","addr":"172.21.15.27:6834","nonce":4215930867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[]} 2024-01-15T18:11:50.497 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 40 2024-01-15T18:11:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.496+0000 7fb605800640 1 -- 172.21.15.27:0/1973916509 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb5d80670d0 msgr2=0x7fb5d8069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.496+0000 7fb605800640 1 --2- 172.21.15.27:0/1973916509 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb5d80670d0 0x7fb5d8069590 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb5f001c4d0 tx=0x7fb5f00a7000 comp rx=0 tx=0).stop 2024-01-15T18:11:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.496+0000 7fb605800640 1 -- 172.21.15.27:0/1973916509 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6000cf6c0 msgr2=0x7fb6001527f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.496+0000 7fb605800640 1 --2- 172.21.15.27:0/1973916509 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6000cf6c0 0x7fb6001527f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb5e800c3c0 tx=0x7fb5e800c880 comp rx=0 tx=0).stop 2024-01-15T18:11:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.496+0000 7fb605800640 1 -- 172.21.15.27:0/1973916509 shutdown_connections 2024-01-15T18:11:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.496+0000 7fb605800640 1 --2- 172.21.15.27:0/1973916509 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb5d80670d0 0x7fb5d8069590 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.496+0000 7fb605800640 1 --2- 172.21.15.27:0/1973916509 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb60015e810 0x7fb600152fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.496+0000 7fb605800640 1 --2- 172.21.15.27:0/1973916509 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6000cf6c0 0x7fb6001527f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.496+0000 7fb605800640 1 --2- 172.21.15.27:0/1973916509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6000b2070 0x7fb600141e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.496+0000 7fb605800640 1 -- 172.21.15.27:0/1973916509 >> 172.21.15.27:0/1973916509 conn(0x7fb6000d0f20 msgr2=0x7fb6000cd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.496+0000 7fb605800640 1 -- 172.21.15.27:0/1973916509 shutdown_connections 2024-01-15T18:11:50.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.496+0000 7fb605800640 1 -- 172.21.15.27:0/1973916509 wait complete. 2024-01-15T18:11:50.518 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-15T18:11:50.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.660+0000 7ff807200640 1 Processor -- start 2024-01-15T18:11:50.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.660+0000 7ff807200640 1 -- start start 2024-01-15T18:11:50.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.660+0000 7ff807200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff80015d680 0x7ff8000cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:50.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.660+0000 7ff807200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff8000cfc30 0x7ff8000cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:50.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.660+0000 7ff807200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8000cb4d0 con 0x7ff8000d19a0 2024-01-15T18:11:50.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.660+0000 7ff807200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8000d0120 con 0x7ff80015d680 2024-01-15T18:11:50.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.660+0000 7ff807200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8000d0400 con 0x7ff8000cfc30 2024-01-15T18:11:50.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.660+0000 7ff805200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff8000d19a0 0x7ff80006b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46340/0 (socket says 172.21.15.27:46340) 2024-01-15T18:11:50.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.660+0000 7ff805200640 1 -- 172.21.15.27:0/393584520 learned_addr learned my addr 172.21.15.27:0/393584520 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:50.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.660+0000 7ff7ff400640 1 --2- 172.21.15.27:0/393584520 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff8000cfc30 0x7ff8000cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:50.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.660+0000 7ff7ffe00640 1 --2- 172.21.15.27:0/393584520 >> v2:172.21.15.93:3300/0 conn(0x7ff80015d680 0x7ff8000cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:50.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.660+0000 7ff7fea00640 1 -- 172.21.15.27:0/393584520 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3447587645 0 0) 0x7ff8000cb4d0 con 0x7ff8000d19a0 2024-01-15T18:11:50.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.660+0000 7ff7fea00640 1 -- 172.21.15.27:0/393584520 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7e0003530 con 0x7ff8000d19a0 2024-01-15T18:11:50.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.660+0000 7ff7ff400640 1 -- 172.21.15.27:0/393584520 >> v2:172.21.15.93:3300/0 conn(0x7ff80015d680 msgr2=0x7ff8000cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:50.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.660+0000 7ff7ff400640 1 --2- 172.21.15.27:0/393584520 >> v2:172.21.15.93:3300/0 conn(0x7ff80015d680 0x7ff8000cd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.660+0000 7ff7ff400640 1 -- 172.21.15.27:0/393584520 >> v1:172.21.15.27:6789/0 conn(0x7ff8000d19a0 legacy=0x7ff80006b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:50.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.660+0000 7ff7ff400640 1 -- 172.21.15.27:0/393584520 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8000cb770 con 0x7ff8000cfc30 2024-01-15T18:11:50.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.660+0000 7ff7ffe00640 1 --2- 172.21.15.27:0/393584520 >> v2:172.21.15.93:3300/0 conn(0x7ff80015d680 0x7ff8000cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:11:50.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.664+0000 7ff7ff400640 1 --2- 172.21.15.27:0/393584520 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff8000cfc30 0x7ff8000cdc40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff7e80097c0 tx=0x7ff7e801c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c438caedc85eee2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:50.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.664+0000 7ff7fea00640 1 -- 172.21.15.27:0/393584520 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8000ce850 con 0x7ff8000cfc30 2024-01-15T18:11:50.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.664+0000 7ff7fea00640 1 -- 172.21.15.27:0/393584520 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7e80ab670 con 0x7ff8000cfc30 2024-01-15T18:11:50.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.664+0000 7ff7fea00640 1 -- 172.21.15.27:0/393584520 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7e80aba00 con 0x7ff8000cfc30 2024-01-15T18:11:50.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.664+0000 7ff807200640 1 -- 172.21.15.27:0/393584520 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff8000cfc30 msgr2=0x7ff8000cdc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:50.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.664+0000 7ff807200640 1 --2- 172.21.15.27:0/393584520 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff8000cfc30 0x7ff8000cdc40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff7e80097c0 tx=0x7ff7e801c450 comp rx=0 tx=0).stop 2024-01-15T18:11:50.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.668+0000 7ff807200640 1 -- 172.21.15.27:0/393584520 shutdown_connections 2024-01-15T18:11:50.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.668+0000 7ff807200640 1 --2- 172.21.15.27:0/393584520 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff8000cfc30 0x7ff8000cdc40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.668+0000 7ff807200640 1 --2- 172.21.15.27:0/393584520 >> v2:172.21.15.93:3300/0 conn(0x7ff80015d680 0x7ff8000cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.668+0000 7ff807200640 1 -- 172.21.15.27:0/393584520 >> 172.21.15.27:0/393584520 conn(0x7ff8000d0e80 msgr2=0x7ff8000d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:50.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.668+0000 7ff807200640 1 -- 172.21.15.27:0/393584520 shutdown_connections 2024-01-15T18:11:50.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.668+0000 7ff807200640 1 -- 172.21.15.27:0/393584520 wait complete. 2024-01-15T18:11:50.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.668+0000 7ff807200640 1 Processor -- start 2024-01-15T18:11:50.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.668+0000 7ff807200640 1 -- start start 2024-01-15T18:11:50.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.668+0000 7ff807200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8000cfc30 0x7ff800153f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:50.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.668+0000 7ff807200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff8000d19a0 0x7ff800158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:50.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.668+0000 7ff807200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff80015d680 0x7ff8001600c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:50.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.668+0000 7ff807200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff800153540 con 0x7ff8000cfc30 2024-01-15T18:11:50.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.668+0000 7ff807200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff800153ac0 con 0x7ff80015d680 2024-01-15T18:11:50.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.668+0000 7ff807200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff80015dad0 con 0x7ff8000d19a0 2024-01-15T18:11:50.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.668+0000 7ff7ffe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8000cfc30 0x7ff800153f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:50.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.668+0000 7ff7ffe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8000cfc30 0x7ff800153f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58160/0 (socket says 172.21.15.27:58160) 2024-01-15T18:11:50.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.668+0000 7ff7ffe00640 1 -- 172.21.15.27:0/3198729029 learned_addr learned my addr 172.21.15.27:0/3198729029 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:50.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.668+0000 7ff7ff400640 1 --2- 172.21.15.27:0/3198729029 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff8000d19a0 0x7ff800158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:50.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.668+0000 7ff805200640 1 --2- 172.21.15.27:0/3198729029 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff80015d680 0x7ff8001600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:50.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.668+0000 7ff7ffe00640 1 -- 172.21.15.27:0/3198729029 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff8000d19a0 msgr2=0x7ff800158080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:50.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.668+0000 7ff7ffe00640 1 --2- 172.21.15.27:0/3198729029 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff8000d19a0 0x7ff800158080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.668+0000 7ff7ffe00640 1 -- 172.21.15.27:0/3198729029 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff80015d680 msgr2=0x7ff8001600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:50.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.668+0000 7ff7ffe00640 1 --2- 172.21.15.27:0/3198729029 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff80015d680 0x7ff8001600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.668+0000 7ff7ffe00640 1 -- 172.21.15.27:0/3198729029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f0009420 con 0x7ff8000cfc30 2024-01-15T18:11:50.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.668+0000 7ff7ff400640 1 --2- 172.21.15.27:0/3198729029 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff8000d19a0 0x7ff800158080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:11:50.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.672+0000 7ff7ffe00640 1 --2- 172.21.15.27:0/3198729029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8000cfc30 0x7ff800153f50 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7ff7f000e980 tx=0x7ff7f000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:50.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.672+0000 7ff7f7e00640 1 -- 172.21.15.27:0/3198729029 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f000ca00 con 0x7ff8000cfc30 2024-01-15T18:11:50.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.672+0000 7ff7f7e00640 1 -- 172.21.15.27:0/3198729029 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f000cb60 con 0x7ff8000cfc30 2024-01-15T18:11:50.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.672+0000 7ff807200640 1 -- 172.21.15.27:0/3198729029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7e8009420 con 0x7ff8000cfc30 2024-01-15T18:11:50.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.672+0000 7ff7f7e00640 1 -- 172.21.15.27:0/3198729029 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f0013470 con 0x7ff8000cfc30 2024-01-15T18:11:50.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.672+0000 7ff807200640 1 -- 172.21.15.27:0/3198729029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff80015e050 con 0x7ff8000cfc30 2024-01-15T18:11:50.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.672+0000 7ff807200640 1 -- 172.21.15.27:0/3198729029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff80004e850 con 0x7ff8000cfc30 2024-01-15T18:11:50.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.672+0000 7ff7f7e00640 1 -- 172.21.15.27:0/3198729029 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7ff7f00135d0 con 0x7ff8000cfc30 2024-01-15T18:11:50.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.672+0000 7ff7f7e00640 1 --2- 172.21.15.27:0/3198729029 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff7d80670d0 0x7ff7d8069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:50.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.672+0000 7ff7f7e00640 1 -- 172.21.15.27:0/3198729029 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4999+0+0 (secure 0 0 0) 0x7ff7f0082d70 con 0x7ff8000cfc30 2024-01-15T18:11:50.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.672+0000 7ff7ff400640 1 --2- 172.21.15.27:0/3198729029 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff7d80670d0 0x7ff7d8069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:50.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.676+0000 7ff7ff400640 1 --2- 172.21.15.27:0/3198729029 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff7d80670d0 0x7ff7d8069590 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff7e80af000 tx=0x7ff7e80a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:50.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.676+0000 7ff7f7e00640 1 -- 172.21.15.27:0/3198729029 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff7f0051bb0 con 0x7ff8000cfc30 2024-01-15T18:11:50.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.836+0000 7ff807200640 1 -- 172.21.15.27:0/3198729029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7ff800067060 con 0x7ff8000cfc30 2024-01-15T18:11:50.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.840+0000 7ff7f7e00640 1 -- 172.21.15.27:0/3198729029 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v23) v1 ==== 120+0+0 (secure 0 0 0) 0x7ff7f00551d0 con 0x7ff8000cfc30 2024-01-15T18:11:50.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.840+0000 7ff807200640 1 -- 172.21.15.27:0/3198729029 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff7d80670d0 msgr2=0x7ff7d8069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:50.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.840+0000 7ff807200640 1 --2- 172.21.15.27:0/3198729029 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff7d80670d0 0x7ff7d8069590 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff7e80af000 tx=0x7ff7e80a7000 comp rx=0 tx=0).stop 2024-01-15T18:11:50.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.840+0000 7ff807200640 1 -- 172.21.15.27:0/3198729029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8000cfc30 msgr2=0x7ff800153f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:50.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.840+0000 7ff807200640 1 --2- 172.21.15.27:0/3198729029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8000cfc30 0x7ff800153f50 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7ff7f000e980 tx=0x7ff7f000ee40 comp rx=0 tx=0).stop 2024-01-15T18:11:50.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.840+0000 7ff807200640 1 -- 172.21.15.27:0/3198729029 shutdown_connections 2024-01-15T18:11:50.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.840+0000 7ff807200640 1 --2- 172.21.15.27:0/3198729029 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff7d80670d0 0x7ff7d8069590 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.840+0000 7ff807200640 1 --2- 172.21.15.27:0/3198729029 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff80015d680 0x7ff8001600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.840+0000 7ff807200640 1 --2- 172.21.15.27:0/3198729029 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff8000d19a0 0x7ff800158080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.840+0000 7ff807200640 1 --2- 172.21.15.27:0/3198729029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8000cfc30 0x7ff800153f50 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.840+0000 7ff807200640 1 -- 172.21.15.27:0/3198729029 >> 172.21.15.27:0/3198729029 conn(0x7ff8000d0e80 msgr2=0x7ff80015bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:50.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.840+0000 7ff807200640 1 -- 172.21.15.27:0/3198729029 shutdown_connections 2024-01-15T18:11:50.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:50.840+0000 7ff807200640 1 -- 172.21.15.27:0/3198729029 wait complete. 2024-01-15T18:11:50.863 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-15T18:11:51.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.004+0000 7f0f18000640 1 Processor -- start 2024-01-15T18:11:51.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.004+0000 7f0f18000640 1 -- start start 2024-01-15T18:11:51.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.008+0000 7f0f18000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0f1006ca40 0x7f0f1006ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:51.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.008+0000 7f0f18000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1015ddb0 con 0x7f0f1015e240 2024-01-15T18:11:51.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.008+0000 7f0f18000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1015e6b0 con 0x7f0f10070e40 2024-01-15T18:11:51.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.008+0000 7f0f18000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f100b2760 con 0x7f0f1006ca40 2024-01-15T18:11:51.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.008+0000 7f0f14c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0f1015e240 0x7f0f10160d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46350/0 (socket says 172.21.15.27:46350) 2024-01-15T18:11:51.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.008+0000 7f0f14c00640 1 -- 172.21.15.27:0/2653390426 learned_addr learned my addr 172.21.15.27:0/2653390426 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:51.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.008+0000 7f0f16000640 1 --2- 172.21.15.27:0/2653390426 >> v2:172.21.15.93:3300/0 conn(0x7f0f1006ca40 0x7f0f1006ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.008+0000 7f0f0fe00640 1 -- 172.21.15.27:0/2653390426 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3834187757 0 0) 0x7f0f1015ddb0 con 0x7f0f1015e240 2024-01-15T18:11:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.008+0000 7f0f0fe00640 1 -- 172.21.15.27:0/2653390426 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ef0003530 con 0x7f0f1015e240 2024-01-15T18:11:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.008+0000 7f0f16000640 1 -- 172.21.15.27:0/2653390426 >> v1:172.21.15.93:6789/0 conn(0x7f0f10070e40 legacy=0x7f0f10071220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.008+0000 7f0f16000640 1 -- 172.21.15.27:0/2653390426 >> v1:172.21.15.27:6789/0 conn(0x7f0f1015e240 legacy=0x7f0f10160d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:51.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.008+0000 7f0f16000640 1 -- 172.21.15.27:0/2653390426 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f10161850 con 0x7f0f1006ca40 2024-01-15T18:11:51.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.012+0000 7f0f16000640 1 --2- 172.21.15.27:0/2653390426 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f1006ca40 0x7f0f1006ce90 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0f08009d90 tx=0x7f0f0801c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e739ea17798e012 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:51.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.012+0000 7f0f0fe00640 1 -- 172.21.15.27:0/2653390426 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f080a5860 con 0x7f0f1006ca40 2024-01-15T18:11:51.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.012+0000 7f0f0fe00640 1 -- 172.21.15.27:0/2653390426 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f080a59c0 con 0x7f0f1006ca40 2024-01-15T18:11:51.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.012+0000 7f0f0fe00640 1 -- 172.21.15.27:0/2653390426 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f080a5d50 con 0x7f0f1006ca40 2024-01-15T18:11:51.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.012+0000 7f0f18000640 1 -- 172.21.15.27:0/2653390426 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f1006ca40 msgr2=0x7f0f1006ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:51.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.012+0000 7f0f18000640 1 --2- 172.21.15.27:0/2653390426 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f1006ca40 0x7f0f1006ce90 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0f08009d90 tx=0x7f0f0801c890 comp rx=0 tx=0).stop 2024-01-15T18:11:51.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.012+0000 7f0f18000640 1 -- 172.21.15.27:0/2653390426 shutdown_connections 2024-01-15T18:11:51.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.012+0000 7f0f18000640 1 --2- 172.21.15.27:0/2653390426 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f1006ca40 0x7f0f1006ce90 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:51.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.012+0000 7f0f18000640 1 -- 172.21.15.27:0/2653390426 >> 172.21.15.27:0/2653390426 conn(0x7f0f100b15b0 msgr2=0x7f0f100b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:51.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.012+0000 7f0f18000640 1 -- 172.21.15.27:0/2653390426 shutdown_connections 2024-01-15T18:11:51.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.012+0000 7f0f18000640 1 -- 172.21.15.27:0/2653390426 wait complete. 2024-01-15T18:11:51.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.012+0000 7f0f18000640 1 Processor -- start 2024-01-15T18:11:51.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.012+0000 7f0f18000640 1 -- start start 2024-01-15T18:11:51.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.016+0000 7f0f18000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f1006ca40 0x7f0f100ae850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:51.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.016+0000 7f0f18000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f10070e40 0x7f0f10155380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:51.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.016+0000 7f0f18000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0f1015e240 0x7f0f100aff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:51.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.016+0000 7f0f18000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f100ad990 con 0x7f0f1006ca40 2024-01-15T18:11:51.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.016+0000 7f0f18000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f100acd80 con 0x7f0f10070e40 2024-01-15T18:11:51.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.016+0000 7f0f18000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f100ad060 con 0x7f0f1015e240 2024-01-15T18:11:51.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.016+0000 7f0f15600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f1006ca40 0x7f0f100ae850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:51.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.016+0000 7f0f14c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f10070e40 0x7f0f10155380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:51.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.016+0000 7f0f15600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f1006ca40 0x7f0f100ae850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58174/0 (socket says 172.21.15.27:58174) 2024-01-15T18:11:51.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.016+0000 7f0f15600640 1 -- 172.21.15.27:0/934773709 learned_addr learned my addr 172.21.15.27:0/934773709 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:51.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.016+0000 7f0f16000640 1 --2- 172.21.15.27:0/934773709 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0f1015e240 0x7f0f100aff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.016+0000 7f0f14c00640 1 -- 172.21.15.27:0/934773709 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0f1015e240 msgr2=0x7f0f100aff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.016+0000 7f0f14c00640 1 --2- 172.21.15.27:0/934773709 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0f1015e240 0x7f0f100aff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.016+0000 7f0f14c00640 1 -- 172.21.15.27:0/934773709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f1006ca40 msgr2=0x7f0f100ae850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.016+0000 7f0f14c00640 1 --2- 172.21.15.27:0/934773709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f1006ca40 0x7f0f100ae850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.016+0000 7f0f14c00640 1 -- 172.21.15.27:0/934773709 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f08009a70 con 0x7f0f10070e40 2024-01-15T18:11:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.016+0000 7f0f15600640 1 --2- 172.21.15.27:0/934773709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f1006ca40 0x7f0f100ae850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:11:51.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.016+0000 7f0f14c00640 1 --2- 172.21.15.27:0/934773709 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f10070e40 0x7f0f10155380 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8001ec0 tx=0x7f0ef800c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.016+0000 7f0f0d600640 1 -- 172.21.15.27:0/934773709 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8005cf0 con 0x7f0f10070e40 2024-01-15T18:11:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.016+0000 7f0f0d600640 1 -- 172.21.15.27:0/934773709 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef8005e50 con 0x7f0f10070e40 2024-01-15T18:11:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.016+0000 7f0f0d600640 1 -- 172.21.15.27:0/934773709 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef80072d0 con 0x7f0f10070e40 2024-01-15T18:11:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.016+0000 7f0f18000640 1 -- 172.21.15.27:0/934773709 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f100ad340 con 0x7f0f10070e40 2024-01-15T18:11:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.016+0000 7f0f18000640 1 -- 172.21.15.27:0/934773709 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f101558c0 con 0x7f0f10070e40 2024-01-15T18:11:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.016+0000 7f0f18000640 1 -- 172.21.15.27:0/934773709 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f1013f1e0 con 0x7f0f10070e40 2024-01-15T18:11:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.016+0000 7f0f0d600640 1 -- 172.21.15.27:0/934773709 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f0ef8007430 con 0x7f0f10070e40 2024-01-15T18:11:51.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.020+0000 7f0f0d600640 1 --2- 172.21.15.27:0/934773709 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0ee80670d0 0x7f0ee8069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:51.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.020+0000 7f0f0d600640 1 -- 172.21.15.27:0/934773709 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4999+0+0 (secure 0 0 0) 0x7f0ef8081f50 con 0x7f0f10070e40 2024-01-15T18:11:51.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.020+0000 7f0f15600640 1 --2- 172.21.15.27:0/934773709 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0ee80670d0 0x7f0ee8069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:51.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.020+0000 7f0f15600640 1 --2- 172.21.15.27:0/934773709 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0ee80670d0 0x7f0ee8069590 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0f00001ca0 tx=0x7f0f00009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:51.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.024+0000 7f0f0d600640 1 -- 172.21.15.27:0/934773709 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0ef8050e40 con 0x7f0f10070e40 2024-01-15T18:11:51.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.180+0000 7f0f18000640 1 -- 172.21.15.27:0/934773709 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f0f1004e850 con 0x7f0f10070e40 2024-01-15T18:11:51.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.192+0000 7f0f0d600640 1 -- 172.21.15.27:0/934773709 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v24) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0ef8054460 con 0x7f0f10070e40 2024-01-15T18:11:51.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.192+0000 7f0f18000640 1 -- 172.21.15.27:0/934773709 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0ee80670d0 msgr2=0x7f0ee8069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:51.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.192+0000 7f0f18000640 1 --2- 172.21.15.27:0/934773709 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0ee80670d0 0x7f0ee8069590 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0f00001ca0 tx=0x7f0f00009040 comp rx=0 tx=0).stop 2024-01-15T18:11:51.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.192+0000 7f0f18000640 1 -- 172.21.15.27:0/934773709 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f10070e40 msgr2=0x7f0f10155380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:51.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.192+0000 7f0f18000640 1 --2- 172.21.15.27:0/934773709 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f10070e40 0x7f0f10155380 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8001ec0 tx=0x7f0ef800c820 comp rx=0 tx=0).stop 2024-01-15T18:11:51.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.192+0000 7f0f18000640 1 -- 172.21.15.27:0/934773709 shutdown_connections 2024-01-15T18:11:51.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.192+0000 7f0f18000640 1 --2- 172.21.15.27:0/934773709 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0ee80670d0 0x7f0ee8069590 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:51.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.192+0000 7f0f18000640 1 --2- 172.21.15.27:0/934773709 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0f1015e240 0x7f0f100aff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:51.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.192+0000 7f0f18000640 1 --2- 172.21.15.27:0/934773709 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f10070e40 0x7f0f10155380 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:51.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.192+0000 7f0f18000640 1 --2- 172.21.15.27:0/934773709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f1006ca40 0x7f0f100ae850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:51.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.192+0000 7f0f18000640 1 -- 172.21.15.27:0/934773709 >> 172.21.15.27:0/934773709 conn(0x7f0f100b15b0 msgr2=0x7f0f1015e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:51.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.192+0000 7f0f18000640 1 -- 172.21.15.27:0/934773709 shutdown_connections 2024-01-15T18:11:51.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.192+0000 7f0f18000640 1 -- 172.21.15.27:0/934773709 wait complete. 2024-01-15T18:11:51.215 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2024-01-15T18:11:51.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.356+0000 7f48f3a00640 1 Processor -- start 2024-01-15T18:11:51.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.356+0000 7f48f3a00640 1 -- start start 2024-01-15T18:11:51.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48f3a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f48ec15b100 0x7f48ec070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:51.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48f3a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ec071860 con 0x7f48ec071420 2024-01-15T18:11:51.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48f3a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ec15a610 con 0x7f48ec15ad20 2024-01-15T18:11:51.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48f3a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48ec15e3c0 con 0x7f48ec15b100 2024-01-15T18:11:51.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48f1a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f48ec071420 0x7f48ec15bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46366/0 (socket says 172.21.15.27:46366) 2024-01-15T18:11:51.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48f1a00640 1 -- 172.21.15.27:0/1460603800 learned_addr learned my addr 172.21.15.27:0/1460603800 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48ebe00640 1 --2- 172.21.15.27:0/1460603800 >> v2:172.21.15.93:3300/0 conn(0x7f48ec15b100 0x7f48ec070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48eb400640 1 -- 172.21.15.27:0/1460603800 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255053791 0 0) 0x7f48ec071860 con 0x7f48ec071420 2024-01-15T18:11:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48eb400640 1 -- 172.21.15.27:0/1460603800 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48cc003530 con 0x7f48ec071420 2024-01-15T18:11:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48eb400640 1 -- 172.21.15.27:0/1460603800 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2108797365 0 0) 0x7f48cc003530 con 0x7f48ec071420 2024-01-15T18:11:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48eb400640 1 -- 172.21.15.27:0/1460603800 >> v2:172.21.15.93:3300/0 conn(0x7f48ec15b100 msgr2=0x7f48ec070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48eb400640 1 --2- 172.21.15.27:0/1460603800 >> v2:172.21.15.93:3300/0 conn(0x7f48ec15b100 0x7f48ec070ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48eb400640 1 -- 172.21.15.27:0/1460603800 >> v1:172.21.15.93:6789/0 conn(0x7f48ec15ad20 legacy=0x7f48ec15d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48eb400640 1 -- 172.21.15.27:0/1460603800 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ec15e6a0 con 0x7f48ec071420 2024-01-15T18:11:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48eb400640 1 -- 172.21.15.27:0/1460603800 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f48e4001a00 con 0x7f48ec071420 2024-01-15T18:11:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48eb400640 1 -- 172.21.15.27:0/1460603800 >> v1:172.21.15.27:6789/0 conn(0x7f48ec071420 legacy=0x7f48ec15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48eb400640 1 --2- 172.21.15.27:0/1460603800 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48cc003a20 0x7f48cc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48eb400640 1 --2- 172.21.15.27:0/1460603800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48cc0047d0 0x7f48cc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48eb400640 1 --2- 172.21.15.27:0/1460603800 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f48cc005190 0x7f48cc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48eb400640 1 -- 172.21.15.27:0/1460603800 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ec15e3c0 con 0x7f48cc0047d0 2024-01-15T18:11:51.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48eb400640 1 -- 172.21.15.27:0/1460603800 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc00c0e0 con 0x7f48cc003a20 2024-01-15T18:11:51.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48eb400640 1 -- 172.21.15.27:0/1460603800 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48cc00c2a0 con 0x7f48cc005190 2024-01-15T18:11:51.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48ebe00640 1 --2- 172.21.15.27:0/1460603800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48cc0047d0 0x7f48cc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:51.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48f1000640 1 --2- 172.21.15.27:0/1460603800 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48cc003a20 0x7f48cc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:51.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48f1a00640 1 --2- 172.21.15.27:0/1460603800 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f48cc005190 0x7f48cc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:51.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48ebe00640 1 -- 172.21.15.27:0/1460603800 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f48cc005190 msgr2=0x7f48cc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:51.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48ebe00640 1 --2- 172.21.15.27:0/1460603800 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f48cc005190 0x7f48cc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:51.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48ebe00640 1 -- 172.21.15.27:0/1460603800 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48cc003a20 msgr2=0x7f48cc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:51.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48ebe00640 1 --2- 172.21.15.27:0/1460603800 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48cc003a20 0x7f48cc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:51.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48ebe00640 1 -- 172.21.15.27:0/1460603800 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48cc003530 con 0x7f48cc0047d0 2024-01-15T18:11:51.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.360+0000 7f48f1a00640 1 -- 172.21.15.27:0/1460603800 reap_dead start 2024-01-15T18:11:51.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.364+0000 7f48ebe00640 1 --2- 172.21.15.27:0/1460603800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48cc0047d0 0x7f48cc004c50 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f48d4009ae0 tx=0x7f48d401ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ec990a5ab4933a1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:51.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.364+0000 7f48eb400640 1 -- 172.21.15.27:0/1460603800 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d40a5660 con 0x7f48cc0047d0 2024-01-15T18:11:51.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.364+0000 7f48eb400640 1 -- 172.21.15.27:0/1460603800 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d40a57c0 con 0x7f48cc0047d0 2024-01-15T18:11:51.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.364+0000 7f48eb400640 1 -- 172.21.15.27:0/1460603800 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d40a5b70 con 0x7f48cc0047d0 2024-01-15T18:11:51.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.364+0000 7f48f3a00640 1 -- 172.21.15.27:0/1460603800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48cc0047d0 msgr2=0x7f48cc004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:51.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.364+0000 7f48f3a00640 1 --2- 172.21.15.27:0/1460603800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48cc0047d0 0x7f48cc004c50 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f48d4009ae0 tx=0x7f48d401ca70 comp rx=0 tx=0).stop 2024-01-15T18:11:51.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.364+0000 7f48f3a00640 1 -- 172.21.15.27:0/1460603800 shutdown_connections 2024-01-15T18:11:51.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.364+0000 7f48f3a00640 1 --2- 172.21.15.27:0/1460603800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48cc0047d0 0x7f48cc004c50 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:51.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.364+0000 7f48f3a00640 1 -- 172.21.15.27:0/1460603800 >> 172.21.15.27:0/1460603800 conn(0x7f48ec0cb700 msgr2=0x7f48ec0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:51.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.364+0000 7f48f3a00640 1 -- 172.21.15.27:0/1460603800 shutdown_connections 2024-01-15T18:11:51.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.364+0000 7f48f3a00640 1 -- 172.21.15.27:0/1460603800 wait complete. 2024-01-15T18:11:51.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.368+0000 7f48f3a00640 1 Processor -- start 2024-01-15T18:11:51.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.368+0000 7f48f3a00640 1 -- start start 2024-01-15T18:11:51.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.368+0000 7f48f3a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48cc0047d0 0x7f48ec13be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:51.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.368+0000 7f48f3a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f48ec13a460 0x7f48ec0cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:51.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.368+0000 7f48f3a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48ec13a8e0 0x7f48ec0ce460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:51.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.368+0000 7f48f3a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ec13b970 con 0x7f48ec13a8e0 2024-01-15T18:11:51.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.368+0000 7f48f3a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ec13ad60 con 0x7f48cc0047d0 2024-01-15T18:11:51.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.368+0000 7f48f3a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48ec13b040 con 0x7f48ec13a460 2024-01-15T18:11:51.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.368+0000 7f48f1a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48ec13a8e0 0x7f48ec0ce460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:51.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.368+0000 7f48f1a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48ec13a8e0 0x7f48ec0ce460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58190/0 (socket says 172.21.15.27:58190) 2024-01-15T18:11:51.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.368+0000 7f48f1a00640 1 -- 172.21.15.27:0/1211631840 learned_addr learned my addr 172.21.15.27:0/1211631840 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:51.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.368+0000 7f48f1000640 1 --2- 172.21.15.27:0/1211631840 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48cc0047d0 0x7f48ec13be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:51.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.368+0000 7f48ebe00640 1 --2- 172.21.15.27:0/1211631840 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f48ec13a460 0x7f48ec0cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:51.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.368+0000 7f48f1a00640 1 -- 172.21.15.27:0/1211631840 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f48ec13a460 msgr2=0x7f48ec0cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:51.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.368+0000 7f48f1a00640 1 --2- 172.21.15.27:0/1211631840 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f48ec13a460 0x7f48ec0cdf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:51.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.368+0000 7f48f1a00640 1 -- 172.21.15.27:0/1211631840 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48cc0047d0 msgr2=0x7f48ec13be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:51.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.368+0000 7f48f1a00640 1 --2- 172.21.15.27:0/1211631840 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48cc0047d0 0x7f48ec13be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:51.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.368+0000 7f48f1a00640 1 -- 172.21.15.27:0/1211631840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ec13b320 con 0x7f48ec13a8e0 2024-01-15T18:11:51.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.368+0000 7f48f1a00640 1 --2- 172.21.15.27:0/1211631840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48ec13a8e0 0x7f48ec0ce460 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f48ec0cf740 tx=0x7f48e4007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:51.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.368+0000 7f48e8c00640 1 -- 172.21.15.27:0/1211631840 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e400a9b0 con 0x7f48ec13a8e0 2024-01-15T18:11:51.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.368+0000 7f48e8c00640 1 -- 172.21.15.27:0/1211631840 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e400ab30 con 0x7f48ec13a8e0 2024-01-15T18:11:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.368+0000 7f48f3a00640 1 -- 172.21.15.27:0/1211631840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48ec13b600 con 0x7f48ec13a8e0 2024-01-15T18:11:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.368+0000 7f48e8c00640 1 -- 172.21.15.27:0/1211631840 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e40133f0 con 0x7f48ec13a8e0 2024-01-15T18:11:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.368+0000 7f48f3a00640 1 -- 172.21.15.27:0/1211631840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48ec13c9b0 con 0x7f48ec13a8e0 2024-01-15T18:11:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.368+0000 7f48f3a00640 1 -- 172.21.15.27:0/1211631840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48ec04e850 con 0x7f48ec13a8e0 2024-01-15T18:11:51.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.372+0000 7f48e8c00640 1 -- 172.21.15.27:0/1211631840 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f48e4013550 con 0x7f48ec13a8e0 2024-01-15T18:11:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.372+0000 7f48e8c00640 1 --2- 172.21.15.27:0/1211631840 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f48c40670d0 0x7f48c4069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.372+0000 7f48e8c00640 1 -- 172.21.15.27:0/1211631840 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4999+0+0 (secure 0 0 0) 0x7f48e4082f30 con 0x7f48ec13a8e0 2024-01-15T18:11:51.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.372+0000 7f48f1000640 1 --2- 172.21.15.27:0/1211631840 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f48c40670d0 0x7f48c4069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:51.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.372+0000 7f48f1000640 1 --2- 172.21.15.27:0/1211631840 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f48c40670d0 0x7f48c4069590 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f48dc005630 tx=0x7f48dc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:51.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.376+0000 7f48e8c00640 1 -- 172.21.15.27:0/1211631840 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f48e4051e20 con 0x7f48ec13a8e0 2024-01-15T18:11:51.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.532+0000 7f48f3a00640 1 -- 172.21.15.27:0/1211631840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7f48ec06d320 con 0x7f48ec13a8e0 2024-01-15T18:11:51.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.536+0000 7f48e8c00640 1 -- 172.21.15.27:0/1211631840 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v25) v1 ==== 115+0+0 (secure 0 0 0) 0x7f48e4055440 con 0x7f48ec13a8e0 2024-01-15T18:11:51.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.540+0000 7f48f3a00640 1 -- 172.21.15.27:0/1211631840 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f48c40670d0 msgr2=0x7f48c4069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:51.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.540+0000 7f48f3a00640 1 --2- 172.21.15.27:0/1211631840 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f48c40670d0 0x7f48c4069590 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f48dc005630 tx=0x7f48dc009040 comp rx=0 tx=0).stop 2024-01-15T18:11:51.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.540+0000 7f48f3a00640 1 -- 172.21.15.27:0/1211631840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48ec13a8e0 msgr2=0x7f48ec0ce460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:51.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.540+0000 7f48f3a00640 1 --2- 172.21.15.27:0/1211631840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48ec13a8e0 0x7f48ec0ce460 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f48ec0cf740 tx=0x7f48e4007a20 comp rx=0 tx=0).stop 2024-01-15T18:11:51.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.540+0000 7f48f3a00640 1 -- 172.21.15.27:0/1211631840 shutdown_connections 2024-01-15T18:11:51.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.540+0000 7f48f3a00640 1 --2- 172.21.15.27:0/1211631840 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f48c40670d0 0x7f48c4069590 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:51.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.540+0000 7f48f3a00640 1 --2- 172.21.15.27:0/1211631840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48ec13a8e0 0x7f48ec0ce460 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:51.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.540+0000 7f48f3a00640 1 --2- 172.21.15.27:0/1211631840 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f48ec13a460 0x7f48ec0cdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:51.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.540+0000 7f48f3a00640 1 --2- 172.21.15.27:0/1211631840 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48cc0047d0 0x7f48ec13be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:51.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.540+0000 7f48f3a00640 1 -- 172.21.15.27:0/1211631840 >> 172.21.15.27:0/1211631840 conn(0x7f48ec0cb700 msgr2=0x7f48ec06c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:51.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.540+0000 7f48f3a00640 1 -- 172.21.15.27:0/1211631840 shutdown_connections 2024-01-15T18:11:51.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.540+0000 7f48f3a00640 1 -- 172.21.15.27:0/1211631840 wait complete. 2024-01-15T18:11:51.560 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones' 2024-01-15T18:11:51.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.700+0000 7f1f64800640 1 Processor -- start 2024-01-15T18:11:51.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.700+0000 7f1f64800640 1 -- start start 2024-01-15T18:11:51.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.704+0000 7f1f64800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1f5c0701a0 0x7f1f5c070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.704+0000 7f1f64800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f1f5c15c130 0x7f1f5c15d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.704+0000 7f1f64800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f5c06c4a0 con 0x7f1f5c0c92b0 2024-01-15T18:11:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.704+0000 7f1f64800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f5c070ac0 con 0x7f1f5c0701a0 2024-01-15T18:11:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.704+0000 7f1f64800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f5c15c660 con 0x7f1f5c15c130 2024-01-15T18:11:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.704+0000 7f1f61e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1f5c0701a0 0x7f1f5c070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.704+0000 7f1f61400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1f5c0c92b0 0x7f1f5c06dd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46378/0 (socket says 172.21.15.27:46378) 2024-01-15T18:11:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.704+0000 7f1f61e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1f5c0701a0 0x7f1f5c070580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58204/0 (socket says 172.21.15.27:58204) 2024-01-15T18:11:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.704+0000 7f1f61400640 1 -- 172.21.15.27:0/3881276652 learned_addr learned my addr 172.21.15.27:0/3881276652 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.704+0000 7f1f62800640 1 --2- 172.21.15.27:0/3881276652 >> v2:172.21.15.93:3300/0 conn(0x7f1f5c15c130 0x7f1f5c15d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.704+0000 7f1f61e00640 1 -- 172.21.15.27:0/3881276652 >> v2:172.21.15.93:3300/0 conn(0x7f1f5c15c130 msgr2=0x7f1f5c15d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.704+0000 7f1f61e00640 1 --2- 172.21.15.27:0/3881276652 >> v2:172.21.15.93:3300/0 conn(0x7f1f5c15c130 0x7f1f5c15d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.704+0000 7f1f61e00640 1 -- 172.21.15.27:0/3881276652 >> v1:172.21.15.27:6789/0 conn(0x7f1f5c0c92b0 legacy=0x7f1f5c06dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.704+0000 7f1f61e00640 1 -- 172.21.15.27:0/3881276652 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f5c15c940 con 0x7f1f5c0701a0 2024-01-15T18:11:51.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.704+0000 7f1f62800640 1 --2- 172.21.15.27:0/3881276652 >> v2:172.21.15.93:3300/0 conn(0x7f1f5c15c130 0x7f1f5c15d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:11:51.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.704+0000 7f1f61e00640 1 --2- 172.21.15.27:0/3881276652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f5c0701a0 0x7f1f5c070580 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c06c050 tx=0x7f1f4c01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=162263577ec6d626 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:51.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.708+0000 7f1f60a00640 1 -- 172.21.15.27:0/3881276652 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c0a7070 con 0x7f1f5c0701a0 2024-01-15T18:11:51.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.708+0000 7f1f60a00640 1 -- 172.21.15.27:0/3881276652 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f4c0ab430 con 0x7f1f5c0701a0 2024-01-15T18:11:51.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.708+0000 7f1f60a00640 1 -- 172.21.15.27:0/3881276652 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c0ab7c0 con 0x7f1f5c0701a0 2024-01-15T18:11:51.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.708+0000 7f1f64800640 1 -- 172.21.15.27:0/3881276652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f5c0701a0 msgr2=0x7f1f5c070580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:51.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.708+0000 7f1f64800640 1 --2- 172.21.15.27:0/3881276652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f5c0701a0 0x7f1f5c070580 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c06c050 tx=0x7f1f4c01c430 comp rx=0 tx=0).stop 2024-01-15T18:11:51.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.708+0000 7f1f64800640 1 -- 172.21.15.27:0/3881276652 shutdown_connections 2024-01-15T18:11:51.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.708+0000 7f1f64800640 1 --2- 172.21.15.27:0/3881276652 >> v2:172.21.15.93:3300/0 conn(0x7f1f5c15c130 0x7f1f5c15d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:51.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.708+0000 7f1f64800640 1 --2- 172.21.15.27:0/3881276652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f5c0701a0 0x7f1f5c070580 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:51.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.708+0000 7f1f64800640 1 -- 172.21.15.27:0/3881276652 >> 172.21.15.27:0/3881276652 conn(0x7f1f5c0cec60 msgr2=0x7f1f5c0cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:51.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.708+0000 7f1f64800640 1 -- 172.21.15.27:0/3881276652 shutdown_connections 2024-01-15T18:11:51.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.708+0000 7f1f64800640 1 -- 172.21.15.27:0/3881276652 wait complete. 2024-01-15T18:11:51.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.708+0000 7f1f64800640 1 Processor -- start 2024-01-15T18:11:51.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.708+0000 7f1f64800640 1 -- start start 2024-01-15T18:11:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.712+0000 7f1f64800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1f5c0701a0 0x7f1f5c15a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.712+0000 7f1f64800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1f5c0c92b0 0x7f1f5c0ae210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.712+0000 7f1f64800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f5c15c130 0x7f1f5c0c43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.712+0000 7f1f64800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5c0c7670 con 0x7f1f5c15c130 2024-01-15T18:11:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.712+0000 7f1f64800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5c0c7950 con 0x7f1f5c0701a0 2024-01-15T18:11:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.712+0000 7f1f64800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5c0c7c30 con 0x7f1f5c0c92b0 2024-01-15T18:11:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.712+0000 7f1f62800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f5c15c130 0x7f1f5c0c43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:51.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.712+0000 7f1f62800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f5c15c130 0x7f1f5c0c43e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58214/0 (socket says 172.21.15.27:58214) 2024-01-15T18:11:51.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.712+0000 7f1f62800640 1 -- 172.21.15.27:0/804129080 learned_addr learned my addr 172.21.15.27:0/804129080 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:51.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.712+0000 7f1f61400640 1 --2- 172.21.15.27:0/804129080 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1f5c0c92b0 0x7f1f5c0ae210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:51.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.712+0000 7f1f61e00640 1 --2- 172.21.15.27:0/804129080 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1f5c0701a0 0x7f1f5c15a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:51.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.712+0000 7f1f62800640 1 -- 172.21.15.27:0/804129080 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1f5c0c92b0 msgr2=0x7f1f5c0ae210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:51.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.712+0000 7f1f62800640 1 --2- 172.21.15.27:0/804129080 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1f5c0c92b0 0x7f1f5c0ae210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:51.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.712+0000 7f1f62800640 1 -- 172.21.15.27:0/804129080 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1f5c0701a0 msgr2=0x7f1f5c15a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:51.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.712+0000 7f1f62800640 1 --2- 172.21.15.27:0/804129080 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1f5c0701a0 0x7f1f5c15a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:51.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.712+0000 7f1f62800640 1 -- 172.21.15.27:0/804129080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f4c009420 con 0x7f1f5c15c130 2024-01-15T18:11:51.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.712+0000 7f1f62800640 1 --2- 172.21.15.27:0/804129080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f5c15c130 0x7f1f5c0c43e0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f1f5400cd80 tx=0x7f1f5400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:51.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.712+0000 7f1f5a000640 1 -- 172.21.15.27:0/804129080 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f54003ba0 con 0x7f1f5c15c130 2024-01-15T18:11:51.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.712+0000 7f1f5a000640 1 -- 172.21.15.27:0/804129080 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f54003d20 con 0x7f1f5c15c130 2024-01-15T18:11:51.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.712+0000 7f1f5a000640 1 -- 172.21.15.27:0/804129080 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f54004260 con 0x7f1f5c15c130 2024-01-15T18:11:51.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.712+0000 7f1f64800640 1 -- 172.21.15.27:0/804129080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f5c0c7f10 con 0x7f1f5c15c130 2024-01-15T18:11:51.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.712+0000 7f1f64800640 1 -- 172.21.15.27:0/804129080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f5c15a7a0 con 0x7f1f5c15c130 2024-01-15T18:11:51.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.712+0000 7f1f64800640 1 -- 172.21.15.27:0/804129080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f5c04e850 con 0x7f1f5c15c130 2024-01-15T18:11:51.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.712+0000 7f1f5a000640 1 -- 172.21.15.27:0/804129080 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f1f5400b040 con 0x7f1f5c15c130 2024-01-15T18:11:51.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.716+0000 7f1f5a000640 1 --2- 172.21.15.27:0/804129080 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1f340670d0 0x7f1f34069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:51.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.716+0000 7f1f5a000640 1 -- 172.21.15.27:0/804129080 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4999+0+0 (secure 0 0 0) 0x7f1f54083230 con 0x7f1f5c15c130 2024-01-15T18:11:51.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.716+0000 7f1f61e00640 1 --2- 172.21.15.27:0/804129080 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1f340670d0 0x7f1f34069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:51.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.716+0000 7f1f61e00640 1 --2- 172.21.15.27:0/804129080 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1f340670d0 0x7f1f34069590 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c0af000 tx=0x7f1f4c0a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:51.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.720+0000 7f1f5a000640 1 -- 172.21.15.27:0/804129080 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1f54051fe0 con 0x7f1f5c15c130 2024-01-15T18:11:51.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:51.932+0000 7f1f64800640 1 -- 172.21.15.27:0/804129080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones"]} v 0) v1 -- 0x7f1f5c0c83c0 con 0x7f1f5c15c130 2024-01-15T18:11:52.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.184+0000 7f1f5a000640 1 -- 172.21.15.27:0/804129080 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones"]}]=0 v340) v1 ==== 154+0+0 (secure 0 0 0) 0x7f1f5401d070 con 0x7f1f5c15c130 2024-01-15T18:11:52.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.188+0000 7f1f64800640 1 -- 172.21.15.27:0/804129080 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1f340670d0 msgr2=0x7f1f34069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:52.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.188+0000 7f1f64800640 1 --2- 172.21.15.27:0/804129080 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1f340670d0 0x7f1f34069590 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c0af000 tx=0x7f1f4c0a7000 comp rx=0 tx=0).stop 2024-01-15T18:11:52.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.188+0000 7f1f64800640 1 -- 172.21.15.27:0/804129080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f5c15c130 msgr2=0x7f1f5c0c43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:52.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.188+0000 7f1f64800640 1 --2- 172.21.15.27:0/804129080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f5c15c130 0x7f1f5c0c43e0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f1f5400cd80 tx=0x7f1f5400a650 comp rx=0 tx=0).stop 2024-01-15T18:11:52.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.188+0000 7f1f64800640 1 -- 172.21.15.27:0/804129080 shutdown_connections 2024-01-15T18:11:52.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.188+0000 7f1f64800640 1 --2- 172.21.15.27:0/804129080 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1f340670d0 0x7f1f34069590 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.188+0000 7f1f64800640 1 --2- 172.21.15.27:0/804129080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f5c15c130 0x7f1f5c0c43e0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.188+0000 7f1f64800640 1 --2- 172.21.15.27:0/804129080 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1f5c0c92b0 0x7f1f5c0ae210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.188+0000 7f1f64800640 1 --2- 172.21.15.27:0/804129080 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1f5c0701a0 0x7f1f5c15a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.188+0000 7f1f64800640 1 -- 172.21.15.27:0/804129080 >> 172.21.15.27:0/804129080 conn(0x7f1f5c0cec60 msgr2=0x7f1f5c0cf040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:52.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.188+0000 7f1f64800640 1 -- 172.21.15.27:0/804129080 shutdown_connections 2024-01-15T18:11:52.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.188+0000 7f1f64800640 1 -- 172.21.15.27:0/804129080 wait complete. 2024-01-15T18:11:52.210 INFO:tasks.cephfs_test_runner:X = number of clones issued. ... ok 2024-01-15T18:11:52.211 INFO:tasks.cephfs_test_runner:Starting test: test_progress_drops_when_new_jobs_are_added (tasks.cephfs.test_volumes.TestCloneProgress) X = number of clones issued. 2024-01-15T18:11:52.211 INFO:tasks.cephfs_test_runner:test_progress_drops_when_new_jobs_are_added (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T18:11:52.211 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:11:52.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.352+0000 7f0d59e00640 1 Processor -- start 2024-01-15T18:11:52.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.352+0000 7f0d59e00640 1 -- start start 2024-01-15T18:11:52.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.352+0000 7f0d59e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0d5406db80 0x7f0d5406df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:52.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.352+0000 7f0d59e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0d5415e730 0x7f0d540cd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:52.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.352+0000 7f0d59e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d540ce920 con 0x7f0d540cdd10 2024-01-15T18:11:52.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.352+0000 7f0d59e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d5406e4a0 con 0x7f0d5415e730 2024-01-15T18:11:52.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.352+0000 7f0d59e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d540ce640 con 0x7f0d5406db80 2024-01-15T18:11:52.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.352+0000 7f0d52000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0d5415e730 0x7f0d540cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:52.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.352+0000 7f0d52000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0d5415e730 0x7f0d540cd420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58220/0 (socket says 172.21.15.27:58220) 2024-01-15T18:11:52.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.352+0000 7f0d52000640 1 -- 172.21.15.27:0/3252487460 learned_addr learned my addr 172.21.15.27:0/3252487460 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:52.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.352+0000 7f0d52a00640 1 --2- 172.21.15.27:0/3252487460 >> v2:172.21.15.93:3300/0 conn(0x7f0d5406db80 0x7f0d5406df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:52.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.356+0000 7f0d52000640 1 -- 172.21.15.27:0/3252487460 >> v2:172.21.15.93:3300/0 conn(0x7f0d5406db80 msgr2=0x7f0d5406df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:52.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.356+0000 7f0d52000640 1 --2- 172.21.15.27:0/3252487460 >> v2:172.21.15.93:3300/0 conn(0x7f0d5406db80 0x7f0d5406df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.356+0000 7f0d52000640 1 -- 172.21.15.27:0/3252487460 >> v1:172.21.15.93:6789/0 conn(0x7f0d540cdd10 legacy=0x7f0d54070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:52.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.356+0000 7f0d52000640 1 -- 172.21.15.27:0/3252487460 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d540cc940 con 0x7f0d5415e730 2024-01-15T18:11:52.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.356+0000 7f0d52000640 1 --2- 172.21.15.27:0/3252487460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d5415e730 0x7f0d540cd420 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f0d5406be30 tx=0x7f0d3c01eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8ddf4c3a46be7f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:52.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.356+0000 7f0d51600640 1 -- 172.21.15.27:0/3252487460 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c0a7b30 con 0x7f0d5415e730 2024-01-15T18:11:52.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.356+0000 7f0d51600640 1 -- 172.21.15.27:0/3252487460 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d3c0a7c90 con 0x7f0d5415e730 2024-01-15T18:11:52.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.356+0000 7f0d51600640 1 -- 172.21.15.27:0/3252487460 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c0af440 con 0x7f0d5415e730 2024-01-15T18:11:52.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.356+0000 7f0d59e00640 1 -- 172.21.15.27:0/3252487460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d5415e730 msgr2=0x7f0d540cd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:52.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.356+0000 7f0d59e00640 1 --2- 172.21.15.27:0/3252487460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d5415e730 0x7f0d540cd420 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f0d5406be30 tx=0x7f0d3c01eb60 comp rx=0 tx=0).stop 2024-01-15T18:11:52.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.360+0000 7f0d59e00640 1 -- 172.21.15.27:0/3252487460 shutdown_connections 2024-01-15T18:11:52.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.360+0000 7f0d59e00640 1 --2- 172.21.15.27:0/3252487460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d5415e730 0x7f0d540cd420 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.360+0000 7f0d59e00640 1 --2- 172.21.15.27:0/3252487460 >> v2:172.21.15.93:3300/0 conn(0x7f0d5406db80 0x7f0d5406df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.360+0000 7f0d59e00640 1 -- 172.21.15.27:0/3252487460 >> 172.21.15.27:0/3252487460 conn(0x7f0d540cec60 msgr2=0x7f0d540cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:52.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.360+0000 7f0d59e00640 1 -- 172.21.15.27:0/3252487460 shutdown_connections 2024-01-15T18:11:52.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.360+0000 7f0d59e00640 1 -- 172.21.15.27:0/3252487460 wait complete. 2024-01-15T18:11:52.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.360+0000 7f0d59e00640 1 Processor -- start 2024-01-15T18:11:52.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.360+0000 7f0d59e00640 1 -- start start 2024-01-15T18:11:52.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.360+0000 7f0d59e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d5406db80 0x7f0d54146810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:52.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.360+0000 7f0d59e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d540cdd10 0x7f0d540b4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.360+0000 7f0d59e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d5415e730 0x7f0d540c8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.360+0000 7f0d59e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d540c21d0 con 0x7f0d540cdd10 2024-01-15T18:11:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.360+0000 7f0d59e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d540b4aa0 con 0x7f0d5415e730 2024-01-15T18:11:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.360+0000 7f0d59e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d540b3500 con 0x7f0d5406db80 2024-01-15T18:11:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.360+0000 7f0d52000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d540cdd10 0x7f0d540b4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.360+0000 7f0d52000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d540cdd10 0x7f0d540b4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58226/0 (socket says 172.21.15.27:58226) 2024-01-15T18:11:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.360+0000 7f0d52000640 1 -- 172.21.15.27:0/3992962357 learned_addr learned my addr 172.21.15.27:0/3992962357 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.360+0000 7f0d52a00640 1 --2- 172.21.15.27:0/3992962357 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d5406db80 0x7f0d54146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.360+0000 7f0d53400640 1 --2- 172.21.15.27:0/3992962357 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d5415e730 0x7f0d540c8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.360+0000 7f0d52000640 1 -- 172.21.15.27:0/3992962357 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d5406db80 msgr2=0x7f0d54146810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.360+0000 7f0d52000640 1 --2- 172.21.15.27:0/3992962357 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d5406db80 0x7f0d54146810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.360+0000 7f0d52000640 1 -- 172.21.15.27:0/3992962357 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d5415e730 msgr2=0x7f0d540c8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:52.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.360+0000 7f0d52000640 1 --2- 172.21.15.27:0/3992962357 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d5415e730 0x7f0d540c8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.360+0000 7f0d52000640 1 -- 172.21.15.27:0/3992962357 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d3c00b770 con 0x7f0d540cdd10 2024-01-15T18:11:52.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.364+0000 7f0d52000640 1 --2- 172.21.15.27:0/3992962357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d540cdd10 0x7f0d540b4560 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c01c790 tx=0x7f0d3c01b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:52.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.364+0000 7f0d4b400640 1 -- 172.21.15.27:0/3992962357 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c0ba460 con 0x7f0d540cdd10 2024-01-15T18:11:52.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.364+0000 7f0d4b400640 1 -- 172.21.15.27:0/3992962357 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d3c0ba5e0 con 0x7f0d540cdd10 2024-01-15T18:11:52.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.364+0000 7f0d59e00640 1 -- 172.21.15.27:0/3992962357 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d540b37a0 con 0x7f0d540cdd10 2024-01-15T18:11:52.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.364+0000 7f0d4b400640 1 -- 172.21.15.27:0/3992962357 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c0ba990 con 0x7f0d540cdd10 2024-01-15T18:11:52.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.364+0000 7f0d59e00640 1 -- 172.21.15.27:0/3992962357 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d540b3d60 con 0x7f0d540cdd10 2024-01-15T18:11:52.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.364+0000 7f0d59e00640 1 -- 172.21.15.27:0/3992962357 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d5404e850 con 0x7f0d540cdd10 2024-01-15T18:11:52.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.364+0000 7f0d4b400640 1 -- 172.21.15.27:0/3992962357 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f0d3c0c33b0 con 0x7f0d540cdd10 2024-01-15T18:11:52.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.364+0000 7f0d4b400640 1 --2- 172.21.15.27:0/3992962357 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0d2c067100 0x7f0d2c0695c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:52.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.364+0000 7f0d4b400640 1 -- 172.21.15.27:0/3992962357 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4999+0+0 (secure 0 0 0) 0x7f0d3c11d6a0 con 0x7f0d540cdd10 2024-01-15T18:11:52.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.364+0000 7f0d52a00640 1 --2- 172.21.15.27:0/3992962357 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0d2c067100 0x7f0d2c0695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:52.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.368+0000 7f0d52a00640 1 --2- 172.21.15.27:0/3992962357 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0d2c067100 0x7f0d2c0695c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f0d44007a60 tx=0x7f0d44008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:52.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.368+0000 7f0d4b400640 1 -- 172.21.15.27:0/3992962357 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0d3c0ec590 con 0x7f0d540cdd10 2024-01-15T18:11:52.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.528+0000 7f0d59e00640 1 -- 172.21.15.27:0/3992962357 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d5406c800 con 0x7f0d540cdd10 2024-01-15T18:11:52.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.528+0000 7f0d4b400640 1 -- 172.21.15.27:0/3992962357 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+12977 (secure 0 0 0) 0x7f0d3c0bf090 con 0x7f0d540cdd10 2024-01-15T18:11:52.530 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:11:52.530 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":62,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:11:47.160522+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6834/1969845074":"2024-01-16T18:11:44.135277+0000","172.21.15.27:6832/1969845074":"2024-01-16T18:11:44.135277+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:11:52.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.528+0000 7f0d59e00640 1 -- 172.21.15.27:0/3992962357 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0d2c067100 msgr2=0x7f0d2c0695c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:52.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.528+0000 7f0d59e00640 1 --2- 172.21.15.27:0/3992962357 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0d2c067100 0x7f0d2c0695c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f0d44007a60 tx=0x7f0d44008040 comp rx=0 tx=0).stop 2024-01-15T18:11:52.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.528+0000 7f0d59e00640 1 -- 172.21.15.27:0/3992962357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d540cdd10 msgr2=0x7f0d540b4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:52.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.528+0000 7f0d59e00640 1 --2- 172.21.15.27:0/3992962357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d540cdd10 0x7f0d540b4560 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c01c790 tx=0x7f0d3c01b900 comp rx=0 tx=0).stop 2024-01-15T18:11:52.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.532+0000 7f0d59e00640 1 -- 172.21.15.27:0/3992962357 shutdown_connections 2024-01-15T18:11:52.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.532+0000 7f0d59e00640 1 --2- 172.21.15.27:0/3992962357 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0d2c067100 0x7f0d2c0695c0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.532+0000 7f0d59e00640 1 --2- 172.21.15.27:0/3992962357 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d5415e730 0x7f0d540c8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.532+0000 7f0d59e00640 1 --2- 172.21.15.27:0/3992962357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d540cdd10 0x7f0d540b4560 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.532+0000 7f0d59e00640 1 --2- 172.21.15.27:0/3992962357 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d5406db80 0x7f0d54146810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.532+0000 7f0d59e00640 1 -- 172.21.15.27:0/3992962357 >> 172.21.15.27:0/3992962357 conn(0x7f0d540cec60 msgr2=0x7f0d54163220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:52.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.532+0000 7f0d59e00640 1 -- 172.21.15.27:0/3992962357 shutdown_connections 2024-01-15T18:11:52.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.532+0000 7f0d59e00640 1 -- 172.21.15.27:0/3992962357 wait complete. 2024-01-15T18:11:52.551 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:11:52.552 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:11:52.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.696+0000 7f9232e00640 1 Processor -- start 2024-01-15T18:11:52.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.696+0000 7f9232e00640 1 -- start start 2024-01-15T18:11:52.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.696+0000 7f9232e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f922c0cc2c0 0x7f922c06a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:52.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.696+0000 7f9232e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f922c15bb90 0x7f922c0b4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:52.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.696+0000 7f9232e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f922c06ae90 con 0x7f922c06aa30 2024-01-15T18:11:52.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.696+0000 7f9232e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f922c15c020 con 0x7f922c15bb90 2024-01-15T18:11:52.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.696+0000 7f9232e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f922c0cc6a0 con 0x7f922c0cc2c0 2024-01-15T18:11:52.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.696+0000 7f922b400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f922c06aa30 0x7f922c15b480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46392/0 (socket says 172.21.15.27:46392) 2024-01-15T18:11:52.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.696+0000 7f922b400640 1 -- 172.21.15.27:0/694770393 learned_addr learned my addr 172.21.15.27:0/694770393 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:52.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.696+0000 7f9230e00640 1 --2- 172.21.15.27:0/694770393 >> v2:172.21.15.93:3300/0 conn(0x7f922c15bb90 0x7f922c0b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:52.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.696+0000 7f922be00640 1 --2- 172.21.15.27:0/694770393 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f922c0cc2c0 0x7f922c06a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:52.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.696+0000 7f922aa00640 1 -- 172.21.15.27:0/694770393 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169447467 0 0) 0x7f922c06ae90 con 0x7f922c06aa30 2024-01-15T18:11:52.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.696+0000 7f922aa00640 1 -- 172.21.15.27:0/694770393 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f920c003530 con 0x7f922c06aa30 2024-01-15T18:11:52.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.696+0000 7f922be00640 1 -- 172.21.15.27:0/694770393 >> v2:172.21.15.93:3300/0 conn(0x7f922c15bb90 msgr2=0x7f922c0b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:52.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.696+0000 7f922be00640 1 --2- 172.21.15.27:0/694770393 >> v2:172.21.15.93:3300/0 conn(0x7f922c15bb90 0x7f922c0b4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.696+0000 7f922be00640 1 -- 172.21.15.27:0/694770393 >> v1:172.21.15.27:6789/0 conn(0x7f922c06aa30 legacy=0x7f922c15b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:52.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.696+0000 7f922be00640 1 -- 172.21.15.27:0/694770393 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f922c13e230 con 0x7f922c0cc2c0 2024-01-15T18:11:52.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.700+0000 7f922be00640 1 --2- 172.21.15.27:0/694770393 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f922c0cc2c0 0x7f922c06a4f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f921c0097c0 tx=0x7f921c01c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a1dc8d9ca63f975 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:52.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.700+0000 7f922aa00640 1 -- 172.21.15.27:0/694770393 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c0ab600 con 0x7f922c0cc2c0 2024-01-15T18:11:52.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.700+0000 7f922aa00640 1 -- 172.21.15.27:0/694770393 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f921c0a7050 con 0x7f922c0cc2c0 2024-01-15T18:11:52.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.700+0000 7f922aa00640 1 -- 172.21.15.27:0/694770393 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c0ab980 con 0x7f922c0cc2c0 2024-01-15T18:11:52.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.700+0000 7f9232e00640 1 -- 172.21.15.27:0/694770393 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f922c0cc2c0 msgr2=0x7f922c06a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:52.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.700+0000 7f9232e00640 1 --2- 172.21.15.27:0/694770393 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f922c0cc2c0 0x7f922c06a4f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f921c0097c0 tx=0x7f921c01c480 comp rx=0 tx=0).stop 2024-01-15T18:11:52.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.700+0000 7f9232e00640 1 -- 172.21.15.27:0/694770393 shutdown_connections 2024-01-15T18:11:52.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.700+0000 7f9232e00640 1 --2- 172.21.15.27:0/694770393 >> v2:172.21.15.93:3300/0 conn(0x7f922c15bb90 0x7f922c0b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.700+0000 7f9232e00640 1 --2- 172.21.15.27:0/694770393 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f922c0cc2c0 0x7f922c06a4f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.700+0000 7f9232e00640 1 -- 172.21.15.27:0/694770393 >> 172.21.15.27:0/694770393 conn(0x7f922c0cce10 msgr2=0x7f922c0b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:52.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.700+0000 7f9232e00640 1 -- 172.21.15.27:0/694770393 shutdown_connections 2024-01-15T18:11:52.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.700+0000 7f9232e00640 1 -- 172.21.15.27:0/694770393 wait complete. 2024-01-15T18:11:52.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.704+0000 7f9232e00640 1 Processor -- start 2024-01-15T18:11:52.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.704+0000 7f9232e00640 1 -- start start 2024-01-15T18:11:52.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.704+0000 7f9232e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f922c06aa30 0x7f922c128d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:52.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.704+0000 7f9232e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f922c0cc2c0 0x7f922c165280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:52.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.704+0000 7f9232e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f922c15bb90 0x7f922c12c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:52.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.704+0000 7f9232e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922c129e80 con 0x7f922c0cc2c0 2024-01-15T18:11:52.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.704+0000 7f9232e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922c1295e0 con 0x7f922c06aa30 2024-01-15T18:11:52.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.704+0000 7f9232e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f922c1298c0 con 0x7f922c15bb90 2024-01-15T18:11:52.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.704+0000 7f922b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f922c0cc2c0 0x7f922c165280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:52.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.704+0000 7f922b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f922c0cc2c0 0x7f922c165280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58228/0 (socket says 172.21.15.27:58228) 2024-01-15T18:11:52.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.704+0000 7f922b400640 1 -- 172.21.15.27:0/2406691211 learned_addr learned my addr 172.21.15.27:0/2406691211 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:52.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.704+0000 7f9230e00640 1 --2- 172.21.15.27:0/2406691211 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f922c15bb90 0x7f922c12c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:52.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.704+0000 7f922be00640 1 --2- 172.21.15.27:0/2406691211 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f922c06aa30 0x7f922c128d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:52.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.704+0000 7f922b400640 1 -- 172.21.15.27:0/2406691211 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f922c15bb90 msgr2=0x7f922c12c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:52.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.704+0000 7f922b400640 1 --2- 172.21.15.27:0/2406691211 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f922c15bb90 0x7f922c12c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.704+0000 7f922b400640 1 -- 172.21.15.27:0/2406691211 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f922c06aa30 msgr2=0x7f922c128d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:52.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.704+0000 7f922b400640 1 --2- 172.21.15.27:0/2406691211 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f922c06aa30 0x7f922c128d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.704+0000 7f922b400640 1 -- 172.21.15.27:0/2406691211 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f921c009420 con 0x7f922c0cc2c0 2024-01-15T18:11:52.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.704+0000 7f922b400640 1 --2- 172.21.15.27:0/2406691211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f922c0cc2c0 0x7f922c165280 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f921400c420 tx=0x7f921400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:52.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.704+0000 7f9223e00640 1 -- 172.21.15.27:0/2406691211 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9214005cf0 con 0x7f922c0cc2c0 2024-01-15T18:11:52.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.704+0000 7f9223e00640 1 -- 172.21.15.27:0/2406691211 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9214005e50 con 0x7f922c0cc2c0 2024-01-15T18:11:52.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.704+0000 7f9232e00640 1 -- 172.21.15.27:0/2406691211 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f922c129ba0 con 0x7f922c0cc2c0 2024-01-15T18:11:52.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.704+0000 7f9223e00640 1 -- 172.21.15.27:0/2406691211 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92140072d0 con 0x7f922c0cc2c0 2024-01-15T18:11:52.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.708+0000 7f9232e00640 1 -- 172.21.15.27:0/2406691211 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f922c129260 con 0x7f922c0cc2c0 2024-01-15T18:11:52.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.708+0000 7f9232e00640 1 -- 172.21.15.27:0/2406691211 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f922c154330 con 0x7f922c0cc2c0 2024-01-15T18:11:52.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.708+0000 7f9223e00640 1 -- 172.21.15.27:0/2406691211 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f9214007430 con 0x7f922c0cc2c0 2024-01-15T18:11:52.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.708+0000 7f9223e00640 1 --2- 172.21.15.27:0/2406691211 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f92040670d0 0x7f9204069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:52.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.708+0000 7f9223e00640 1 -- 172.21.15.27:0/2406691211 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4999+0+0 (secure 0 0 0) 0x7f9214082080 con 0x7f922c0cc2c0 2024-01-15T18:11:52.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.708+0000 7f922be00640 1 --2- 172.21.15.27:0/2406691211 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f92040670d0 0x7f9204069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:52.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.708+0000 7f922be00640 1 --2- 172.21.15.27:0/2406691211 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f92040670d0 0x7f9204069590 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f921c0ad000 tx=0x7f921c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:52.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.712+0000 7f9223e00640 1 -- 172.21.15.27:0/2406691211 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f922c154330 con 0x7f922c0cc2c0 2024-01-15T18:11:52.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.872+0000 7f9232e00640 1 -- 172.21.15.27:0/2406691211 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f922c154330 con 0x7f922c0cc2c0 2024-01-15T18:11:52.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.872+0000 7f9223e00640 1 -- 172.21.15.27:0/2406691211 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v62) v1 ==== 93+0+10 (secure 0 0 0) 0x7f922c154330 con 0x7f922c0cc2c0 2024-01-15T18:11:52.875 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:11:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.876+0000 7f9232e00640 1 -- 172.21.15.27:0/2406691211 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f92040670d0 msgr2=0x7f9204069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.876+0000 7f9232e00640 1 --2- 172.21.15.27:0/2406691211 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f92040670d0 0x7f9204069590 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f921c0ad000 tx=0x7f921c008040 comp rx=0 tx=0).stop 2024-01-15T18:11:52.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.876+0000 7f9232e00640 1 -- 172.21.15.27:0/2406691211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f922c0cc2c0 msgr2=0x7f922c165280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.876+0000 7f9232e00640 1 --2- 172.21.15.27:0/2406691211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f922c0cc2c0 0x7f922c165280 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f921400c420 tx=0x7f921400c8e0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.876+0000 7f9232e00640 1 -- 172.21.15.27:0/2406691211 shutdown_connections 2024-01-15T18:11:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.876+0000 7f9232e00640 1 --2- 172.21.15.27:0/2406691211 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f92040670d0 0x7f9204069590 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.876+0000 7f9232e00640 1 --2- 172.21.15.27:0/2406691211 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f922c15bb90 0x7f922c12c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.876+0000 7f9232e00640 1 --2- 172.21.15.27:0/2406691211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f922c0cc2c0 0x7f922c165280 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.876+0000 7f9232e00640 1 --2- 172.21.15.27:0/2406691211 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f922c06aa30 0x7f922c128d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.876+0000 7f9232e00640 1 -- 172.21.15.27:0/2406691211 >> 172.21.15.27:0/2406691211 conn(0x7f922c0cce10 msgr2=0x7f922c15dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.876+0000 7f9232e00640 1 -- 172.21.15.27:0/2406691211 shutdown_connections 2024-01-15T18:11:52.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:52.876+0000 7f9232e00640 1 -- 172.21.15.27:0/2406691211 wait complete. 2024-01-15T18:11:52.898 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added' 2024-01-15T18:11:53.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.036+0000 7fda71e00640 1 Processor -- start 2024-01-15T18:11:53.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.036+0000 7fda71e00640 1 -- start start 2024-01-15T18:11:53.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.036+0000 7fda71e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fda6c0afe70 0x7fda6c0cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:53.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.036+0000 7fda71e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda6c0cb7b0 0x7fda6c0cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:53.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.040+0000 7fda71e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6c0d05b0 con 0x7fda6c160b10 2024-01-15T18:11:53.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.040+0000 7fda71e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda6c0ce7d0 con 0x7fda6c0afe70 2024-01-15T18:11:53.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.040+0000 7fda71e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c0cfc80 con 0x7fda6c0cb7b0 2024-01-15T18:11:53.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.040+0000 7fda6a000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fda6c0afe70 0x7fda6c0cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:53.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.040+0000 7fda6a000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fda6c0afe70 0x7fda6c0cdeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58242/0 (socket says 172.21.15.27:58242) 2024-01-15T18:11:53.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.040+0000 7fda6a000640 1 -- 172.21.15.27:0/3590303071 learned_addr learned my addr 172.21.15.27:0/3590303071 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:53.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.040+0000 7fda6b400640 1 --2- 172.21.15.27:0/3590303071 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda6c0cb7b0 0x7fda6c0cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:53.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.040+0000 7fda6a000640 1 -- 172.21.15.27:0/3590303071 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda6c0cb7b0 msgr2=0x7fda6c0cbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:53.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.040+0000 7fda6a000640 1 --2- 172.21.15.27:0/3590303071 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda6c0cb7b0 0x7fda6c0cbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:53.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.040+0000 7fda6a000640 1 -- 172.21.15.27:0/3590303071 >> v1:172.21.15.93:6789/0 conn(0x7fda6c160b10 legacy=0x7fda6c0cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:53.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.040+0000 7fda6a000640 1 -- 172.21.15.27:0/3590303071 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6c0cff60 con 0x7fda6c0afe70 2024-01-15T18:11:53.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.040+0000 7fda6a000640 1 --2- 172.21.15.27:0/3590303071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda6c0afe70 0x7fda6c0cdeb0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fda54009870 tx=0x7fda5401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86ebf91240ac8b24 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:53.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.040+0000 7fda69600640 1 -- 172.21.15.27:0/3590303071 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda540a6040 con 0x7fda6c0afe70 2024-01-15T18:11:53.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.040+0000 7fda69600640 1 -- 172.21.15.27:0/3590303071 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda540aa600 con 0x7fda6c0afe70 2024-01-15T18:11:53.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.040+0000 7fda69600640 1 -- 172.21.15.27:0/3590303071 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda540aa990 con 0x7fda6c0afe70 2024-01-15T18:11:53.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.044+0000 7fda71e00640 1 -- 172.21.15.27:0/3590303071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda6c0afe70 msgr2=0x7fda6c0cdeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:53.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.044+0000 7fda71e00640 1 --2- 172.21.15.27:0/3590303071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda6c0afe70 0x7fda6c0cdeb0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fda54009870 tx=0x7fda5401c480 comp rx=0 tx=0).stop 2024-01-15T18:11:53.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.044+0000 7fda71e00640 1 -- 172.21.15.27:0/3590303071 shutdown_connections 2024-01-15T18:11:53.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.044+0000 7fda71e00640 1 --2- 172.21.15.27:0/3590303071 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda6c0cb7b0 0x7fda6c0cbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:53.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.044+0000 7fda71e00640 1 --2- 172.21.15.27:0/3590303071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda6c0afe70 0x7fda6c0cdeb0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:53.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.044+0000 7fda71e00640 1 -- 172.21.15.27:0/3590303071 >> 172.21.15.27:0/3590303071 conn(0x7fda6c0d0f20 msgr2=0x7fda6c0d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:53.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.044+0000 7fda71e00640 1 -- 172.21.15.27:0/3590303071 shutdown_connections 2024-01-15T18:11:53.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.044+0000 7fda71e00640 1 -- 172.21.15.27:0/3590303071 wait complete. 2024-01-15T18:11:53.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.044+0000 7fda71e00640 1 Processor -- start 2024-01-15T18:11:53.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.044+0000 7fda71e00640 1 -- start start 2024-01-15T18:11:53.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.044+0000 7fda71e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda6c0afe70 0x7fda6c154020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:53.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.044+0000 7fda71e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda6c0cb7b0 0x7fda6c151660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:53.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.044+0000 7fda71e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda6c160b10 0x7fda6c06f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:53.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.044+0000 7fda71e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c154560 con 0x7fda6c0afe70 2024-01-15T18:11:53.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.044+0000 7fda71e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c151ba0 con 0x7fda6c0cb7b0 2024-01-15T18:11:53.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.044+0000 7fda71e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c154790 con 0x7fda6c160b10 2024-01-15T18:11:53.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.044+0000 7fda6aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda6c0afe70 0x7fda6c154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:53.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.048+0000 7fda6aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda6c0afe70 0x7fda6c154020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58250/0 (socket says 172.21.15.27:58250) 2024-01-15T18:11:53.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.048+0000 7fda6a000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda6c0cb7b0 0x7fda6c151660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:53.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.048+0000 7fda6aa00640 1 -- 172.21.15.27:0/1398198906 learned_addr learned my addr 172.21.15.27:0/1398198906 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:53.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.048+0000 7fda6b400640 1 --2- 172.21.15.27:0/1398198906 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda6c160b10 0x7fda6c06f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:53.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.048+0000 7fda6aa00640 1 -- 172.21.15.27:0/1398198906 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda6c160b10 msgr2=0x7fda6c06f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:53.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.048+0000 7fda6aa00640 1 --2- 172.21.15.27:0/1398198906 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda6c160b10 0x7fda6c06f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:53.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.048+0000 7fda6aa00640 1 -- 172.21.15.27:0/1398198906 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda6c0cb7b0 msgr2=0x7fda6c151660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:53.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.048+0000 7fda6aa00640 1 --2- 172.21.15.27:0/1398198906 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda6c0cb7b0 0x7fda6c151660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:53.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.048+0000 7fda6aa00640 1 -- 172.21.15.27:0/1398198906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda540094d0 con 0x7fda6c0afe70 2024-01-15T18:11:53.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.048+0000 7fda6aa00640 1 --2- 172.21.15.27:0/1398198906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda6c0afe70 0x7fda6c154020 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fda5c00c500 tx=0x7fda5c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:53.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.048+0000 7fda63400640 1 -- 172.21.15.27:0/1398198906 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5c005220 con 0x7fda6c0afe70 2024-01-15T18:11:53.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.048+0000 7fda63400640 1 -- 172.21.15.27:0/1398198906 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda5c005380 con 0x7fda6c0afe70 2024-01-15T18:11:53.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.048+0000 7fda63400640 1 -- 172.21.15.27:0/1398198906 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5c007270 con 0x7fda6c0afe70 2024-01-15T18:11:53.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.048+0000 7fda71e00640 1 -- 172.21.15.27:0/1398198906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda6c154a70 con 0x7fda6c0afe70 2024-01-15T18:11:53.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.048+0000 7fda71e00640 1 -- 172.21.15.27:0/1398198906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda6c155030 con 0x7fda6c0afe70 2024-01-15T18:11:53.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.048+0000 7fda71e00640 1 -- 172.21.15.27:0/1398198906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda6c04e850 con 0x7fda6c0afe70 2024-01-15T18:11:53.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.048+0000 7fda63400640 1 -- 172.21.15.27:0/1398198906 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7fda5c0073d0 con 0x7fda6c0afe70 2024-01-15T18:11:53.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.052+0000 7fda63400640 1 --2- 172.21.15.27:0/1398198906 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fda44067100 0x7fda440695c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:53.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.052+0000 7fda63400640 1 -- 172.21.15.27:0/1398198906 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4999+0+0 (secure 0 0 0) 0x7fda5c0822d0 con 0x7fda6c0afe70 2024-01-15T18:11:53.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.052+0000 7fda6a000640 1 --2- 172.21.15.27:0/1398198906 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fda44067100 0x7fda440695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:53.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.052+0000 7fda6a000640 1 --2- 172.21.15.27:0/1398198906 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fda44067100 0x7fda440695c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fda540ae000 tx=0x7fda540b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:53.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.056+0000 7fda63400640 1 -- 172.21.15.27:0/1398198906 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fda5c051140 con 0x7fda6c0afe70 2024-01-15T18:11:53.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:53.268+0000 7fda71e00640 1 -- 172.21.15.27:0/1398198906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added"]} v 0) v1 -- 0x7fda6c0736a0 con 0x7fda6c0afe70 2024-01-15T18:11:54.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.192+0000 7fda63400640 1 -- 172.21.15.27:0/1398198906 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added"]}]=0 v342) v1 ==== 169+0+0 (secure 0 0 0) 0x7fda5c054760 con 0x7fda6c0afe70 2024-01-15T18:11:54.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.196+0000 7fda71e00640 1 -- 172.21.15.27:0/1398198906 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fda44067100 msgr2=0x7fda440695c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:54.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.196+0000 7fda71e00640 1 --2- 172.21.15.27:0/1398198906 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fda44067100 0x7fda440695c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fda540ae000 tx=0x7fda540b5040 comp rx=0 tx=0).stop 2024-01-15T18:11:54.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.196+0000 7fda71e00640 1 -- 172.21.15.27:0/1398198906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda6c0afe70 msgr2=0x7fda6c154020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:54.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.196+0000 7fda71e00640 1 --2- 172.21.15.27:0/1398198906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda6c0afe70 0x7fda6c154020 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fda5c00c500 tx=0x7fda5c00c9c0 comp rx=0 tx=0).stop 2024-01-15T18:11:54.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.196+0000 7fda71e00640 1 -- 172.21.15.27:0/1398198906 shutdown_connections 2024-01-15T18:11:54.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.196+0000 7fda71e00640 1 --2- 172.21.15.27:0/1398198906 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fda44067100 0x7fda440695c0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:54.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.196+0000 7fda71e00640 1 --2- 172.21.15.27:0/1398198906 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fda6c160b10 0x7fda6c06f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:54.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.196+0000 7fda71e00640 1 --2- 172.21.15.27:0/1398198906 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fda6c0cb7b0 0x7fda6c151660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:54.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.196+0000 7fda71e00640 1 --2- 172.21.15.27:0/1398198906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda6c0afe70 0x7fda6c154020 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:54.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.196+0000 7fda71e00640 1 -- 172.21.15.27:0/1398198906 >> 172.21.15.27:0/1398198906 conn(0x7fda6c0d0f20 msgr2=0x7fda6c06e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:54.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.196+0000 7fda71e00640 1 -- 172.21.15.27:0/1398198906 shutdown_connections 2024-01-15T18:11:54.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.196+0000 7fda71e00640 1 -- 172.21.15.27:0/1398198906 wait complete. 2024-01-15T18:11:54.224 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:11:54.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.372+0000 7f4cc9e00640 1 Processor -- start 2024-01-15T18:11:54.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.372+0000 7f4cc9e00640 1 -- start start 2024-01-15T18:11:54.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.372+0000 7f4cc9e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4cc415d480 0x7f4cc40cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:54.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.372+0000 7f4cc9e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4cc406c740 0x7f4cc40d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:54.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.372+0000 7f4cc9e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f4cc4099a10 0x7f4cc4099ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:54.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.372+0000 7f4cc9e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cc409a400 con 0x7f4cc415d480 2024-01-15T18:11:54.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.372+0000 7f4cc9e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cc415d860 con 0x7f4cc4099a10 2024-01-15T18:11:54.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.372+0000 7f4cc9e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc406b6a0 con 0x7f4cc406c740 2024-01-15T18:11:54.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.372+0000 7f4cc2a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4cc415d480 0x7f4cc40cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:54.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.372+0000 7f4cc2a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4cc415d480 0x7f4cc40cfbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58398/0 (socket says 172.21.15.27:58398) 2024-01-15T18:11:54.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.372+0000 7f4cc2a00640 1 -- 172.21.15.27:0/3263084999 learned_addr learned my addr 172.21.15.27:0/3263084999 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:54.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.372+0000 7f4cc2000640 1 --2- 172.21.15.27:0/3263084999 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4cc406c740 0x7f4cc40d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:54.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.372+0000 7f4cc2a00640 1 -- 172.21.15.27:0/3263084999 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4cc406c740 msgr2=0x7f4cc40d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:54.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.372+0000 7f4cc2a00640 1 --2- 172.21.15.27:0/3263084999 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4cc406c740 0x7f4cc40d0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:54.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.372+0000 7f4cc3400640 1 --2- 172.21.15.27:0/3263084999 >> v2:172.21.15.93:3300/0 conn(0x7f4cc4099a10 0x7f4cc4099ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:54.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.372+0000 7f4cc2a00640 1 -- 172.21.15.27:0/3263084999 >> v2:172.21.15.93:3300/0 conn(0x7f4cc4099a10 msgr2=0x7f4cc4099ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:54.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.376+0000 7f4cc2a00640 1 --2- 172.21.15.27:0/3263084999 >> v2:172.21.15.93:3300/0 conn(0x7f4cc4099a10 0x7f4cc4099ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:54.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.376+0000 7f4cc2a00640 1 -- 172.21.15.27:0/3263084999 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc406b980 con 0x7f4cc415d480 2024-01-15T18:11:54.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.376+0000 7f4cc2000640 1 --2- 172.21.15.27:0/3263084999 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4cc406c740 0x7f4cc40d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:11:54.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.376+0000 7f4cc3400640 1 --2- 172.21.15.27:0/3263084999 >> v2:172.21.15.93:3300/0 conn(0x7f4cc4099a10 0x7f4cc4099ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:11:54.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.376+0000 7f4cc2a00640 1 --2- 172.21.15.27:0/3263084999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4cc415d480 0x7f4cc40cfbf0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f4cc40719a0 tx=0x7f4cb401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31197eb6bd36fb90 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:54.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.376+0000 7f4cc1600640 1 -- 172.21.15.27:0/3263084999 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb40aa5f0 con 0x7f4cc415d480 2024-01-15T18:11:54.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.376+0000 7f4cc1600640 1 -- 172.21.15.27:0/3263084999 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb40aa750 con 0x7f4cc415d480 2024-01-15T18:11:54.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.376+0000 7f4cc1600640 1 -- 172.21.15.27:0/3263084999 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb40aaae0 con 0x7f4cc415d480 2024-01-15T18:11:54.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.376+0000 7f4cc9e00640 1 -- 172.21.15.27:0/3263084999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4cc415d480 msgr2=0x7f4cc40cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:54.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.376+0000 7f4cc9e00640 1 --2- 172.21.15.27:0/3263084999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4cc415d480 0x7f4cc40cfbf0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f4cc40719a0 tx=0x7f4cb401c4d0 comp rx=0 tx=0).stop 2024-01-15T18:11:54.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.380+0000 7f4cc9e00640 1 -- 172.21.15.27:0/3263084999 shutdown_connections 2024-01-15T18:11:54.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.380+0000 7f4cc9e00640 1 --2- 172.21.15.27:0/3263084999 >> v2:172.21.15.93:3300/0 conn(0x7f4cc4099a10 0x7f4cc4099ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:54.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.380+0000 7f4cc9e00640 1 --2- 172.21.15.27:0/3263084999 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4cc406c740 0x7f4cc40d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:54.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.380+0000 7f4cc9e00640 1 --2- 172.21.15.27:0/3263084999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4cc415d480 0x7f4cc40cfbf0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:54.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.380+0000 7f4cc9e00640 1 -- 172.21.15.27:0/3263084999 >> 172.21.15.27:0/3263084999 conn(0x7f4cc40b2a80 msgr2=0x7f4cc40b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:54.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.380+0000 7f4cc9e00640 1 -- 172.21.15.27:0/3263084999 shutdown_connections 2024-01-15T18:11:54.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.380+0000 7f4cc9e00640 1 -- 172.21.15.27:0/3263084999 wait complete. 2024-01-15T18:11:54.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.380+0000 7f4cc9e00640 1 Processor -- start 2024-01-15T18:11:54.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.380+0000 7f4cc9e00640 1 -- start start 2024-01-15T18:11:54.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.380+0000 7f4cc9e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4cc406c740 0x7f4cc406eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:54.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.380+0000 7f4cc9e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4cc4099a10 0x7f4cc4151710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:54.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.380+0000 7f4cc9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4cc415d480 0x7f4cc414f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:54.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.380+0000 7f4cc9e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc414f460 con 0x7f4cc415d480 2024-01-15T18:11:54.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.380+0000 7f4cc9e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc40c7ac0 con 0x7f4cc4099a10 2024-01-15T18:11:54.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.380+0000 7f4cc9e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc4150500 con 0x7f4cc406c740 2024-01-15T18:11:54.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.380+0000 7f4cc3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4cc415d480 0x7f4cc414f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:54.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.380+0000 7f4cc3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4cc415d480 0x7f4cc414f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58406/0 (socket says 172.21.15.27:58406) 2024-01-15T18:11:54.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.380+0000 7f4cc3400640 1 -- 172.21.15.27:0/3190178763 learned_addr learned my addr 172.21.15.27:0/3190178763 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:54.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.380+0000 7f4cc2a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4cc406c740 0x7f4cc406eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:54.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.380+0000 7f4cc2000640 1 --2- 172.21.15.27:0/3190178763 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4cc4099a10 0x7f4cc4151710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:54.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.380+0000 7f4cc3400640 1 -- 172.21.15.27:0/3190178763 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4cc406c740 msgr2=0x7f4cc406eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:54.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.380+0000 7f4cc3400640 1 --2- 172.21.15.27:0/3190178763 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4cc406c740 0x7f4cc406eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:54.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.380+0000 7f4cc3400640 1 -- 172.21.15.27:0/3190178763 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4cc4099a10 msgr2=0x7f4cc4151710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:54.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.380+0000 7f4cc3400640 1 --2- 172.21.15.27:0/3190178763 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4cc4099a10 0x7f4cc4151710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:54.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.380+0000 7f4cc3400640 1 -- 172.21.15.27:0/3190178763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb40094d0 con 0x7f4cc415d480 2024-01-15T18:11:54.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.384+0000 7f4cc3400640 1 --2- 172.21.15.27:0/3190178763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4cc415d480 0x7f4cc414f880 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc00cef0 tx=0x7f4cbc00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:54.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.384+0000 7f4cbb400640 1 -- 172.21.15.27:0/3190178763 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cbc00d840 con 0x7f4cc415d480 2024-01-15T18:11:54.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.384+0000 7f4cbb400640 1 -- 172.21.15.27:0/3190178763 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cbc00d9c0 con 0x7f4cc415d480 2024-01-15T18:11:54.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.384+0000 7f4cc9e00640 1 -- 172.21.15.27:0/3190178763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cc40c77e0 con 0x7f4cc415d480 2024-01-15T18:11:54.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.384+0000 7f4cbb400640 1 -- 172.21.15.27:0/3190178763 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cbc00dd70 con 0x7f4cc415d480 2024-01-15T18:11:54.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.384+0000 7f4cc9e00640 1 -- 172.21.15.27:0/3190178763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cc414f080 con 0x7f4cc415d480 2024-01-15T18:11:54.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.384+0000 7f4cc9e00640 1 -- 172.21.15.27:0/3190178763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cc404e850 con 0x7f4cc415d480 2024-01-15T18:11:54.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.384+0000 7f4cbb400640 1 -- 172.21.15.27:0/3190178763 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f4cbc01e3b0 con 0x7f4cc415d480 2024-01-15T18:11:54.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.384+0000 7f4cbb400640 1 --2- 172.21.15.27:0/3190178763 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4c9c067100 0x7f4c9c0695c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:54.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.384+0000 7f4cbb400640 1 -- 172.21.15.27:0/3190178763 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4999+0+0 (secure 0 0 0) 0x7f4cbc0834c0 con 0x7f4cc415d480 2024-01-15T18:11:54.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.388+0000 7f4cc2a00640 1 --2- 172.21.15.27:0/3190178763 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4c9c067100 0x7f4c9c0695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:54.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.388+0000 7f4cc2a00640 1 --2- 172.21.15.27:0/3190178763 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4c9c067100 0x7f4c9c0695c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4cb40ae000 tx=0x7f4cb40b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:54.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.388+0000 7f4cbb400640 1 -- 172.21.15.27:0/3190178763 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4cbc052470 con 0x7f4cc415d480 2024-01-15T18:11:54.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.548+0000 7f4cc9e00640 1 -- 172.21.15.27:0/3190178763 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4cc40d09e0 con 0x7f4cc415d480 2024-01-15T18:11:54.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.552+0000 7f4cbb400640 1 -- 172.21.15.27:0/3190178763 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4cbc01a020 con 0x7f4cc415d480 2024-01-15T18:11:54.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.552+0000 7f4cc9e00640 1 -- 172.21.15.27:0/3190178763 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4c9c067100 msgr2=0x7f4c9c0695c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:54.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.552+0000 7f4cc9e00640 1 --2- 172.21.15.27:0/3190178763 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4c9c067100 0x7f4c9c0695c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4cb40ae000 tx=0x7f4cb40b5040 comp rx=0 tx=0).stop 2024-01-15T18:11:54.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.552+0000 7f4cc9e00640 1 -- 172.21.15.27:0/3190178763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4cc415d480 msgr2=0x7f4cc414f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:54.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.552+0000 7f4cc9e00640 1 --2- 172.21.15.27:0/3190178763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4cc415d480 0x7f4cc414f880 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc00cef0 tx=0x7f4cbc00a710 comp rx=0 tx=0).stop 2024-01-15T18:11:54.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.552+0000 7f4cc9e00640 1 -- 172.21.15.27:0/3190178763 shutdown_connections 2024-01-15T18:11:54.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.552+0000 7f4cc9e00640 1 --2- 172.21.15.27:0/3190178763 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4c9c067100 0x7f4c9c0695c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:54.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.552+0000 7f4cc9e00640 1 --2- 172.21.15.27:0/3190178763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4cc415d480 0x7f4cc414f880 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:54.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.552+0000 7f4cc9e00640 1 --2- 172.21.15.27:0/3190178763 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4cc4099a10 0x7f4cc4151710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:54.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.552+0000 7f4cc9e00640 1 --2- 172.21.15.27:0/3190178763 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4cc406c740 0x7f4cc406eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:54.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.552+0000 7f4cc9e00640 1 -- 172.21.15.27:0/3190178763 >> 172.21.15.27:0/3190178763 conn(0x7f4cc40b2a80 msgr2=0x7f4cc415dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:54.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.552+0000 7f4cc9e00640 1 -- 172.21.15.27:0/3190178763 shutdown_connections 2024-01-15T18:11:54.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.552+0000 7f4cc9e00640 1 -- 172.21.15.27:0/3190178763 wait complete. 2024-01-15T18:11:54.576 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T18:11:54.576 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:11:54.674 DEBUG:teuthology.orchestra.run.smithi093:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:11:54.727 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T18:11:54.727 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:11:54.727 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:11:54.736 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354580k,nr_inodes=4088645,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9249 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:11:54.736 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:11:54.736 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:11:54.745 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354484k,nr_inodes=4088621,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8403 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:11:54.745 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:11:54.745 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:11:54.787 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354580k,nr_inodes=4088645,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9249 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:11:54.787 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:11:54.787 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:11:54.800 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354484k,nr_inodes=4088621,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8403 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:11:54.800 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-15T18:11:54.800 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:11:54.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.968+0000 7fa50ac00640 1 Processor -- start 2024-01-15T18:11:54.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.968+0000 7fa50ac00640 1 -- start start 2024-01-15T18:11:54.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa50ac00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa50415e830 0x7fa5040ceaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:54.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa50ac00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5040c4ba0 con 0x7fa5040c5030 2024-01-15T18:11:54.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa50ac00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5040cc5a0 con 0x7fa5040d1ab0 2024-01-15T18:11:54.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa50ac00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5040cc840 con 0x7fa50415e830 2024-01-15T18:11:54.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa508c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa5040c5030 0x7fa5040c54a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48514/0 (socket says 172.21.15.27:48514) 2024-01-15T18:11:54.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa508c00640 1 -- 172.21.15.27:0/3475542986 learned_addr learned my addr 172.21.15.27:0/3475542986 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:54.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa503e00640 1 --2- 172.21.15.27:0/3475542986 >> v2:172.21.15.93:3300/0 conn(0x7fa50415e830 0x7fa5040ceaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:54.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa502a00640 1 -- 172.21.15.27:0/3475542986 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 650472142 0 0) 0x7fa5040c4ba0 con 0x7fa5040c5030 2024-01-15T18:11:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa502a00640 1 -- 172.21.15.27:0/3475542986 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4e4003530 con 0x7fa5040c5030 2024-01-15T18:11:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa502a00640 1 -- 172.21.15.27:0/3475542986 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1731347993 0 0) 0x7fa5040cc5a0 con 0x7fa5040d1ab0 2024-01-15T18:11:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa502a00640 1 -- 172.21.15.27:0/3475542986 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5040c4ba0 con 0x7fa5040d1ab0 2024-01-15T18:11:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa502a00640 1 -- 172.21.15.27:0/3475542986 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3261304720 0 0) 0x7fa4e4003530 con 0x7fa5040c5030 2024-01-15T18:11:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa502a00640 1 -- 172.21.15.27:0/3475542986 >> v2:172.21.15.93:3300/0 conn(0x7fa50415e830 msgr2=0x7fa5040ceaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa502a00640 1 --2- 172.21.15.27:0/3475542986 >> v2:172.21.15.93:3300/0 conn(0x7fa50415e830 0x7fa5040ceaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa502a00640 1 -- 172.21.15.27:0/3475542986 >> v1:172.21.15.93:6789/0 conn(0x7fa5040d1ab0 legacy=0x7fa5040cf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa502a00640 1 -- 172.21.15.27:0/3475542986 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5040ccb20 con 0x7fa5040c5030 2024-01-15T18:11:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa502a00640 1 -- 172.21.15.27:0/3475542986 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fa4fc001a00 con 0x7fa5040c5030 2024-01-15T18:11:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa502a00640 1 -- 172.21.15.27:0/3475542986 >> v1:172.21.15.27:6789/0 conn(0x7fa5040c5030 legacy=0x7fa5040c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa502a00640 1 --2- 172.21.15.27:0/3475542986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4e4003b80 0x7fa4e4003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa502a00640 1 --2- 172.21.15.27:0/3475542986 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4e40047d0 0x7fa4e4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa502a00640 1 --2- 172.21.15.27:0/3475542986 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa4e40051e0 0x7fa4e400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa502a00640 1 -- 172.21.15.27:0/3475542986 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5040cc840 con 0x7fa4e4003b80 2024-01-15T18:11:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa502a00640 1 -- 172.21.15.27:0/3475542986 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e400c130 con 0x7fa4e40051e0 2024-01-15T18:11:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa502a00640 1 -- 172.21.15.27:0/3475542986 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e400c2f0 con 0x7fa4e40047d0 2024-01-15T18:11:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa503e00640 1 --2- 172.21.15.27:0/3475542986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4e4003b80 0x7fa4e4003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa508c00640 1 --2- 172.21.15.27:0/3475542986 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa4e40051e0 0x7fa4e400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa503400640 1 --2- 172.21.15.27:0/3475542986 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4e40047d0 0x7fa4e4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa503e00640 1 -- 172.21.15.27:0/3475542986 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4e40047d0 msgr2=0x7fa4e4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa503e00640 1 --2- 172.21.15.27:0/3475542986 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4e40047d0 0x7fa4e4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:54.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa503e00640 1 -- 172.21.15.27:0/3475542986 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa4e40051e0 msgr2=0x7fa4e400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:54.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa503e00640 1 --2- 172.21.15.27:0/3475542986 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa4e40051e0 0x7fa4e400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:54.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa508c00640 1 -- 172.21.15.27:0/3475542986 reap_dead start 2024-01-15T18:11:54.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa503e00640 1 -- 172.21.15.27:0/3475542986 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e4003530 con 0x7fa4e4003b80 2024-01-15T18:11:54.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.972+0000 7fa503400640 1 --2- 172.21.15.27:0/3475542986 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4e40047d0 0x7fa4e4004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:11:54.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.976+0000 7fa503e00640 1 --2- 172.21.15.27:0/3475542986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4e4003b80 0x7fa4e4003f80 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fa4f4009ab0 tx=0x7fa4f401ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61280e0a94c51a70 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:54.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.976+0000 7fa502a00640 1 -- 172.21.15.27:0/3475542986 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f40aa480 con 0x7fa4e4003b80 2024-01-15T18:11:54.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.976+0000 7fa502a00640 1 -- 172.21.15.27:0/3475542986 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4f40aa5e0 con 0x7fa4e4003b80 2024-01-15T18:11:54.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.976+0000 7fa502a00640 1 -- 172.21.15.27:0/3475542986 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f40aa990 con 0x7fa4e4003b80 2024-01-15T18:11:54.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.976+0000 7fa50ac00640 1 -- 172.21.15.27:0/3475542986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4e4003b80 msgr2=0x7fa4e4003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.976+0000 7fa50ac00640 1 --2- 172.21.15.27:0/3475542986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4e4003b80 0x7fa4e4003f80 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fa4f4009ab0 tx=0x7fa4f401ca10 comp rx=0 tx=0).stop 2024-01-15T18:11:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.976+0000 7fa50ac00640 1 -- 172.21.15.27:0/3475542986 shutdown_connections 2024-01-15T18:11:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.976+0000 7fa50ac00640 1 --2- 172.21.15.27:0/3475542986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4e4003b80 0x7fa4e4003f80 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.976+0000 7fa50ac00640 1 -- 172.21.15.27:0/3475542986 >> 172.21.15.27:0/3475542986 conn(0x7fa5040d0f20 msgr2=0x7fa5040d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.976+0000 7fa50ac00640 1 -- 172.21.15.27:0/3475542986 shutdown_connections 2024-01-15T18:11:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.976+0000 7fa50ac00640 1 -- 172.21.15.27:0/3475542986 wait complete. 2024-01-15T18:11:54.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.980+0000 7fa50ac00640 1 Processor -- start 2024-01-15T18:11:54.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.980+0000 7fa50ac00640 1 -- start start 2024-01-15T18:11:54.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.980+0000 7fa50ac00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4e4003b80 0x7fa5041eb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:54.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.980+0000 7fa50ac00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5040c8920 0x7fa504155c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:54.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.980+0000 7fa50ac00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5040c8da0 0x7fa504156160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:54.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.980+0000 7fa50ac00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50414ae50 con 0x7fa5040c8920 2024-01-15T18:11:54.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.980+0000 7fa50ac00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa504156da0 con 0x7fa5040c8da0 2024-01-15T18:11:54.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.980+0000 7fa50ac00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5041566a0 con 0x7fa4e4003b80 2024-01-15T18:11:54.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.980+0000 7fa503400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5040c8920 0x7fa504155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:54.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.980+0000 7fa503400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5040c8920 0x7fa504155c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58424/0 (socket says 172.21.15.27:58424) 2024-01-15T18:11:54.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.980+0000 7fa503400640 1 -- 172.21.15.27:0/1491159560 learned_addr learned my addr 172.21.15.27:0/1491159560 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:54.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.980+0000 7fa503e00640 1 --2- 172.21.15.27:0/1491159560 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4e4003b80 0x7fa5041eb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:54.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.980+0000 7fa508c00640 1 --2- 172.21.15.27:0/1491159560 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5040c8da0 0x7fa504156160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:54.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.980+0000 7fa503400640 1 -- 172.21.15.27:0/1491159560 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4e4003b80 msgr2=0x7fa5041eb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:54.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.980+0000 7fa503400640 1 --2- 172.21.15.27:0/1491159560 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4e4003b80 0x7fa5041eb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:54.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.980+0000 7fa503400640 1 -- 172.21.15.27:0/1491159560 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5040c8da0 msgr2=0x7fa504156160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:54.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.980+0000 7fa503400640 1 --2- 172.21.15.27:0/1491159560 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5040c8da0 0x7fa504156160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:54.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.980+0000 7fa503400640 1 -- 172.21.15.27:0/1491159560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5040c77f0 con 0x7fa5040c8920 2024-01-15T18:11:54.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.980+0000 7fa503400640 1 --2- 172.21.15.27:0/1491159560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5040c8920 0x7fa504155c20 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec00d7a0 tx=0x7fa4ec00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:54.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.980+0000 7fa4fbe00640 1 -- 172.21.15.27:0/1491159560 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec007220 con 0x7fa5040c8920 2024-01-15T18:11:54.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.980+0000 7fa4fbe00640 1 -- 172.21.15.27:0/1491159560 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ec007380 con 0x7fa5040c8920 2024-01-15T18:11:54.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.980+0000 7fa4fbe00640 1 -- 172.21.15.27:0/1491159560 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec0067e0 con 0x7fa5040c8920 2024-01-15T18:11:54.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.980+0000 7fa50ac00640 1 -- 172.21.15.27:0/1491159560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5041f9050 con 0x7fa5040c8920 2024-01-15T18:11:54.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.980+0000 7fa50ac00640 1 -- 172.21.15.27:0/1491159560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5041f95f0 con 0x7fa5040c8920 2024-01-15T18:11:54.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.984+0000 7fa50ac00640 1 -- 172.21.15.27:0/1491159560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa50404e850 con 0x7fa5040c8920 2024-01-15T18:11:54.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.984+0000 7fa4fbe00640 1 -- 172.21.15.27:0/1491159560 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7fa4ec006940 con 0x7fa5040c8920 2024-01-15T18:11:54.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.984+0000 7fa4fbe00640 1 --2- 172.21.15.27:0/1491159560 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa4dc0670d0 0x7fa4dc069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:54.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.984+0000 7fa4fbe00640 1 -- 172.21.15.27:0/1491159560 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4999+0+0 (secure 0 0 0) 0x7fa4ec082850 con 0x7fa5040c8920 2024-01-15T18:11:54.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.984+0000 7fa503e00640 1 --2- 172.21.15.27:0/1491159560 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa4dc0670d0 0x7fa4dc069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:54.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.984+0000 7fa503e00640 1 --2- 172.21.15.27:0/1491159560 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa4dc0670d0 0x7fa4dc069590 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa4f40ae000 tx=0x7fa4f40046d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:54.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:54.988+0000 7fa4fbe00640 1 -- 172.21.15.27:0/1491159560 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa4ec051750 con 0x7fa5040c8920 2024-01-15T18:11:55.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.176+0000 7fa50ac00640 1 -- 172.21.15.27:0/1491159560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5040b2650 con 0x7fa5040c8920 2024-01-15T18:11:55.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.176+0000 7fa4fbe00640 1 -- 172.21.15.27:0/1491159560 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+3696 (secure 0 0 0) 0x7fa4ec01f070 con 0x7fa5040c8920 2024-01-15T18:11:55.178 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:11:55.178 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":40,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5753,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1939851998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1939851998},{"type":"v1","addr":"172.21.15.27:6835","nonce":1939851998}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":5759,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/255207804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":255207804},{"type":"v1","addr":"172.21.15.93:6837","nonce":255207804}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":6244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/1075225596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":1075225596},{"type":"v1","addr":"172.21.15.93:6836","nonce":1075225596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":32},{"gid":7101,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/4215930867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":4215930867},{"type":"v1","addr":"172.21.15.27:6834","nonce":4215930867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[]} 2024-01-15T18:11:55.178 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 40 2024-01-15T18:11:55.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.176+0000 7fa50ac00640 1 -- 172.21.15.27:0/1491159560 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa4dc0670d0 msgr2=0x7fa4dc069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:55.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.176+0000 7fa50ac00640 1 --2- 172.21.15.27:0/1491159560 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa4dc0670d0 0x7fa4dc069590 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa4f40ae000 tx=0x7fa4f40046d0 comp rx=0 tx=0).stop 2024-01-15T18:11:55.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.176+0000 7fa50ac00640 1 -- 172.21.15.27:0/1491159560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5040c8920 msgr2=0x7fa504155c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:55.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.176+0000 7fa50ac00640 1 --2- 172.21.15.27:0/1491159560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5040c8920 0x7fa504155c20 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec00d7a0 tx=0x7fa4ec00dc60 comp rx=0 tx=0).stop 2024-01-15T18:11:55.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.180+0000 7fa50ac00640 1 -- 172.21.15.27:0/1491159560 shutdown_connections 2024-01-15T18:11:55.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.180+0000 7fa50ac00640 1 --2- 172.21.15.27:0/1491159560 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa4dc0670d0 0x7fa4dc069590 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:55.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.180+0000 7fa50ac00640 1 --2- 172.21.15.27:0/1491159560 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5040c8da0 0x7fa504156160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:55.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.180+0000 7fa50ac00640 1 --2- 172.21.15.27:0/1491159560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5040c8920 0x7fa504155c20 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:55.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.180+0000 7fa50ac00640 1 --2- 172.21.15.27:0/1491159560 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4e4003b80 0x7fa5041eb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:55.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.180+0000 7fa50ac00640 1 -- 172.21.15.27:0/1491159560 >> 172.21.15.27:0/1491159560 conn(0x7fa5040d0f20 msgr2=0x7fa5040ce260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:55.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.180+0000 7fa50ac00640 1 -- 172.21.15.27:0/1491159560 shutdown_connections 2024-01-15T18:11:55.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.180+0000 7fa50ac00640 1 -- 172.21.15.27:0/1491159560 wait complete. 2024-01-15T18:11:55.201 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-15T18:11:55.201 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-15T18:11:55.202 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-15T18:11:55.202 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:11:55.202 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-15T18:11:55.202 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-15T18:11:55.202 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-15T18:11:55.202 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:11:55.203 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-15T18:11:55.203 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-15T18:11:55.203 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-15T18:11:55.203 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:11:55.203 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-15T18:11:55.203 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-15T18:11:55.204 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-15T18:11:55.204 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:11:55.215 INFO:tasks.ceph.mds.a:Stopped 2024-01-15T18:11:55.215 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-15T18:11:55.216 INFO:tasks.ceph.mds.c:Stopped 2024-01-15T18:11:55.216 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-15T18:11:55.220 INFO:tasks.ceph.mds.b:Stopped 2024-01-15T18:11:55.220 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-15T18:11:55.221 INFO:tasks.ceph.mds.d:Stopped 2024-01-15T18:11:55.221 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-15T18:11:55.261 INFO:tasks.ceph.mds.a:Started 2024-01-15T18:11:55.261 DEBUG:teuthology.parallel:result is None 2024-01-15T18:11:55.263 INFO:tasks.ceph.mds.c:Started 2024-01-15T18:11:55.264 DEBUG:teuthology.parallel:result is None 2024-01-15T18:11:55.266 INFO:tasks.ceph.mds.b:Started 2024-01-15T18:11:55.267 DEBUG:teuthology.parallel:result is None 2024-01-15T18:11:55.268 INFO:tasks.ceph.mds.d:Started 2024-01-15T18:11:55.268 DEBUG:teuthology.parallel:result is None 2024-01-15T18:11:55.269 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-15T18:11:55.348 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-15T18:11:55.350 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-15T18:11:55.359 INFO:tasks.ceph.mds.d.smithi093.stdout:starting mds.d at 2024-01-15T18:11:55.361 INFO:tasks.ceph.mds.b.smithi093.stdout:starting mds.b at 2024-01-15T18:11:55.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.416+0000 7fcda4000640 1 Processor -- start 2024-01-15T18:11:55.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.416+0000 7fcda4000640 1 -- start start 2024-01-15T18:11:55.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.416+0000 7fcda4000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd9c09a120 0x7fcd9c0cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:55.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.416+0000 7fcda4000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd9c09a700 con 0x7fcd9c0b35a0 2024-01-15T18:11:55.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.416+0000 7fcda4000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd9c0cd590 con 0x7fcd9c15e7d0 2024-01-15T18:11:55.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.416+0000 7fcda4000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0cd870 con 0x7fcd9c09a120 2024-01-15T18:11:55.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.416+0000 7fcda2000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcd9c0b35a0 0x7fcd9c0cc530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48534/0 (socket says 172.21.15.27:48534) 2024-01-15T18:11:55.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.416+0000 7fcda2000640 1 -- 172.21.15.27:0/1668199 learned_addr learned my addr 172.21.15.27:0/1668199 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:55.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.416+0000 7fcda0c00640 1 --2- 172.21.15.27:0/1668199 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd9c09a120 0x7fcd9c0cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:55.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.416+0000 7fcd9be00640 1 -- 172.21.15.27:0/1668199 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627415096 0 0) 0x7fcd9c09a700 con 0x7fcd9c0b35a0 2024-01-15T18:11:55.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.416+0000 7fcd9be00640 1 -- 172.21.15.27:0/1668199 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd7c003530 con 0x7fcd9c0b35a0 2024-01-15T18:11:55.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.416+0000 7fcd9be00640 1 -- 172.21.15.27:0/1668199 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4108346395 0 0) 0x7fcd9c0cd590 con 0x7fcd9c15e7d0 2024-01-15T18:11:55.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.416+0000 7fcd9be00640 1 -- 172.21.15.27:0/1668199 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd9c09a700 con 0x7fcd9c15e7d0 2024-01-15T18:11:55.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.416+0000 7fcda0c00640 1 -- 172.21.15.27:0/1668199 >> v1:172.21.15.93:6789/0 conn(0x7fcd9c15e7d0 legacy=0x7fcd9c099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:55.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.416+0000 7fcda0c00640 1 -- 172.21.15.27:0/1668199 >> v1:172.21.15.27:6789/0 conn(0x7fcd9c0b35a0 legacy=0x7fcd9c0cc530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:55.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.416+0000 7fcda0c00640 1 -- 172.21.15.27:0/1668199 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9c0cfa00 con 0x7fcd9c09a120 2024-01-15T18:11:55.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.420+0000 7fcda0c00640 1 --2- 172.21.15.27:0/1668199 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd9c09a120 0x7fcd9c0cbff0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcd840097f0 tx=0x7fcd8401c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a71c51564bd13d76 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:55.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.420+0000 7fcd9be00640 1 -- 172.21.15.27:0/1668199 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd840aa610 con 0x7fcd9c09a120 2024-01-15T18:11:55.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.420+0000 7fcd9be00640 1 -- 172.21.15.27:0/1668199 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd840a6040 con 0x7fcd9c09a120 2024-01-15T18:11:55.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.420+0000 7fcd9be00640 1 -- 172.21.15.27:0/1668199 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd840aa990 con 0x7fcd9c09a120 2024-01-15T18:11:55.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.420+0000 7fcda4000640 1 -- 172.21.15.27:0/1668199 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd9c09a120 msgr2=0x7fcd9c0cbff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:55.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.420+0000 7fcda4000640 1 --2- 172.21.15.27:0/1668199 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd9c09a120 0x7fcd9c0cbff0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcd840097f0 tx=0x7fcd8401c490 comp rx=0 tx=0).stop 2024-01-15T18:11:55.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.420+0000 7fcda4000640 1 -- 172.21.15.27:0/1668199 shutdown_connections 2024-01-15T18:11:55.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.420+0000 7fcda4000640 1 --2- 172.21.15.27:0/1668199 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd9c09a120 0x7fcd9c0cbff0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:55.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.420+0000 7fcda4000640 1 -- 172.21.15.27:0/1668199 >> 172.21.15.27:0/1668199 conn(0x7fcd9c0b2a80 msgr2=0x7fcd9c0b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:55.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.420+0000 7fcda4000640 1 -- 172.21.15.27:0/1668199 shutdown_connections 2024-01-15T18:11:55.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.420+0000 7fcda4000640 1 -- 172.21.15.27:0/1668199 wait complete. 2024-01-15T18:11:55.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.424+0000 7fcda4000640 1 Processor -- start 2024-01-15T18:11:55.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.424+0000 7fcda4000640 1 -- start start 2024-01-15T18:11:55.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.424+0000 7fcda4000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd9c09a120 0x7fcd9c0ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:55.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.424+0000 7fcda4000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcd9c0b35a0 0x7fcd9c0b5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:55.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.424+0000 7fcda4000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd9c15e7d0 0x7fcd9c159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:55.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.424+0000 7fcda4000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c06cd90 con 0x7fcd9c09a120 2024-01-15T18:11:55.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.424+0000 7fcda4000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0cae30 con 0x7fcd9c0b35a0 2024-01-15T18:11:55.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.424+0000 7fcda4000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c0b3da0 con 0x7fcd9c15e7d0 2024-01-15T18:11:55.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.424+0000 7fcda1600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd9c09a120 0x7fcd9c0ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:55.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.424+0000 7fcda1600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd9c09a120 0x7fcd9c0ca8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58466/0 (socket says 172.21.15.27:58466) 2024-01-15T18:11:55.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.424+0000 7fcda1600640 1 -- 172.21.15.27:0/2764402251 learned_addr learned my addr 172.21.15.27:0/2764402251 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:55.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.424+0000 7fcda0c00640 1 --2- 172.21.15.27:0/2764402251 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcd9c0b35a0 0x7fcd9c0b5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.424+0000 7fcda2000640 1 --2- 172.21.15.27:0/2764402251 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd9c15e7d0 0x7fcd9c159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.424+0000 7fcda1600640 1 -- 172.21.15.27:0/2764402251 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd9c15e7d0 msgr2=0x7fcd9c159fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.424+0000 7fcda1600640 1 --2- 172.21.15.27:0/2764402251 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd9c15e7d0 0x7fcd9c159fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.424+0000 7fcda1600640 1 -- 172.21.15.27:0/2764402251 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcd9c0b35a0 msgr2=0x7fcd9c0b5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.424+0000 7fcda1600640 1 --2- 172.21.15.27:0/2764402251 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcd9c0b35a0 0x7fcd9c0b5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.424+0000 7fcda1600640 1 -- 172.21.15.27:0/2764402251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd84009420 con 0x7fcd9c09a120 2024-01-15T18:11:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.424+0000 7fcda1600640 1 --2- 172.21.15.27:0/2764402251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd9c09a120 0x7fcd9c0ca8f0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c00c430 tx=0x7fcd8c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.424+0000 7fcd99600640 1 -- 172.21.15.27:0/2764402251 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c0052d0 con 0x7fcd9c09a120 2024-01-15T18:11:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.424+0000 7fcd99600640 1 -- 172.21.15.27:0/2764402251 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd8c005450 con 0x7fcd9c09a120 2024-01-15T18:11:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.424+0000 7fcd99600640 1 -- 172.21.15.27:0/2764402251 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c003840 con 0x7fcd9c09a120 2024-01-15T18:11:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.424+0000 7fcda4000640 1 -- 172.21.15.27:0/2764402251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd9c0b4080 con 0x7fcd9c09a120 2024-01-15T18:11:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.424+0000 7fcda4000640 1 -- 172.21.15.27:0/2764402251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd9c06ca20 con 0x7fcd9c09a120 2024-01-15T18:11:55.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.428+0000 7fcda4000640 1 -- 172.21.15.27:0/2764402251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd9c04e850 con 0x7fcd9c09a120 2024-01-15T18:11:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.428+0000 7fcd99600640 1 -- 172.21.15.27:0/2764402251 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7fcd8c0039a0 con 0x7fcd9c09a120 2024-01-15T18:11:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.428+0000 7fcd99600640 1 --2- 172.21.15.27:0/2764402251 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcd74067100 0x7fcd740695c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.428+0000 7fcd99600640 1 -- 172.21.15.27:0/2764402251 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4999+0+0 (secure 0 0 0) 0x7fcd8c082080 con 0x7fcd9c09a120 2024-01-15T18:11:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.428+0000 7fcda0c00640 1 --2- 172.21.15.27:0/2764402251 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcd74067100 0x7fcd740695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.428+0000 7fcda0c00640 1 --2- 172.21.15.27:0/2764402251 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcd74067100 0x7fcd740695c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fcd840ae000 tx=0x7fcd840a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:55.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.432+0000 7fcd99600640 1 -- 172.21.15.27:0/2764402251 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fcd8c050f70 con 0x7fcd9c09a120 2024-01-15T18:11:55.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:55.600+0000 7fcda4000640 1 -- 172.21.15.27:0/2764402251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fcd9c0b6ed0 con 0x7fcd9c09a120 2024-01-15T18:11:56.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.212+0000 7fcd99600640 1 -- 172.21.15.27:0/2764402251 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v64) v1 ==== 97+0+0 (secure 0 0 0) 0x7fcd8c017080 con 0x7fcd9c09a120 2024-01-15T18:11:56.216 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-01-15T18:11:56.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.216+0000 7fcda4000640 1 -- 172.21.15.27:0/2764402251 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcd74067100 msgr2=0x7fcd740695c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:56.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.216+0000 7fcda4000640 1 --2- 172.21.15.27:0/2764402251 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcd74067100 0x7fcd740695c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fcd840ae000 tx=0x7fcd840a7000 comp rx=0 tx=0).stop 2024-01-15T18:11:56.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.216+0000 7fcda4000640 1 -- 172.21.15.27:0/2764402251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd9c09a120 msgr2=0x7fcd9c0ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:56.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.216+0000 7fcda4000640 1 --2- 172.21.15.27:0/2764402251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd9c09a120 0x7fcd9c0ca8f0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c00c430 tx=0x7fcd8c00c8f0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.216+0000 7fcda4000640 1 -- 172.21.15.27:0/2764402251 shutdown_connections 2024-01-15T18:11:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.216+0000 7fcda4000640 1 --2- 172.21.15.27:0/2764402251 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcd74067100 0x7fcd740695c0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.216+0000 7fcda4000640 1 --2- 172.21.15.27:0/2764402251 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcd9c15e7d0 0x7fcd9c159fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.216+0000 7fcda4000640 1 --2- 172.21.15.27:0/2764402251 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcd9c0b35a0 0x7fcd9c0b5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.216+0000 7fcda4000640 1 --2- 172.21.15.27:0/2764402251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcd9c09a120 0x7fcd9c0ca8f0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.216+0000 7fcda4000640 1 -- 172.21.15.27:0/2764402251 >> 172.21.15.27:0/2764402251 conn(0x7fcd9c0b2a80 msgr2=0x7fcd9c0aed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.216+0000 7fcda4000640 1 -- 172.21.15.27:0/2764402251 shutdown_connections 2024-01-15T18:11:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.220+0000 7fcda4000640 1 -- 172.21.15.27:0/2764402251 wait complete. 2024-01-15T18:11:56.245 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-15T18:11:56.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.388+0000 7fce57200640 1 Processor -- start 2024-01-15T18:11:56.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.388+0000 7fce57200640 1 -- start start 2024-01-15T18:11:56.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.388+0000 7fce57200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fce500c3e90 0x7fce500b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:56.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.388+0000 7fce57200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce500b23e0 0x7fce5006b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:56.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.388+0000 7fce57200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce50161d10 con 0x7fce500c4270 2024-01-15T18:11:56.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.388+0000 7fce57200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce50165de0 con 0x7fce500c3e90 2024-01-15T18:11:56.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.388+0000 7fce57200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce500cf260 con 0x7fce500b23e0 2024-01-15T18:11:56.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.388+0000 7fce55200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fce500c4270 0x7fce500cda80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48550/0 (socket says 172.21.15.27:48550) 2024-01-15T18:11:56.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.388+0000 7fce55200640 1 -- 172.21.15.27:0/3096776776 learned_addr learned my addr 172.21.15.27:0/3096776776 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:56.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.388+0000 7fce4f400640 1 --2- 172.21.15.27:0/3096776776 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce500b23e0 0x7fce5006b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:56.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.388+0000 7fce4fe00640 1 --2- 172.21.15.27:0/3096776776 >> v2:172.21.15.93:3300/0 conn(0x7fce500c3e90 0x7fce500b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:56.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.392+0000 7fce4ea00640 1 -- 172.21.15.27:0/3096776776 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832780215 0 0) 0x7fce480014a0 con 0x7fce500c4270 2024-01-15T18:11:56.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.392+0000 7fce4ea00640 1 -- 172.21.15.27:0/3096776776 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce30003530 con 0x7fce500c4270 2024-01-15T18:11:56.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.392+0000 7fce4f400640 1 -- 172.21.15.27:0/3096776776 >> v2:172.21.15.93:3300/0 conn(0x7fce500c3e90 msgr2=0x7fce500b1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:56.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.392+0000 7fce4f400640 1 --2- 172.21.15.27:0/3096776776 >> v2:172.21.15.93:3300/0 conn(0x7fce500c3e90 0x7fce500b1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.392+0000 7fce4f400640 1 -- 172.21.15.27:0/3096776776 >> v1:172.21.15.27:6789/0 conn(0x7fce500c4270 legacy=0x7fce500cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:56.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.392+0000 7fce4f400640 1 -- 172.21.15.27:0/3096776776 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce50166310 con 0x7fce500b23e0 2024-01-15T18:11:56.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.392+0000 7fce4f400640 1 --2- 172.21.15.27:0/3096776776 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce500b23e0 0x7fce5006b600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fce380097c0 tx=0x7fce3801c390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93ed971d5703a71 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:56.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.392+0000 7fce4ea00640 1 -- 172.21.15.27:0/3096776776 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce380ab600 con 0x7fce500b23e0 2024-01-15T18:11:56.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.392+0000 7fce4ea00640 1 -- 172.21.15.27:0/3096776776 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce380a7050 con 0x7fce500b23e0 2024-01-15T18:11:56.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.392+0000 7fce4ea00640 1 -- 172.21.15.27:0/3096776776 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce380ab980 con 0x7fce500b23e0 2024-01-15T18:11:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.396+0000 7fce57200640 1 -- 172.21.15.27:0/3096776776 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce500b23e0 msgr2=0x7fce5006b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.396+0000 7fce57200640 1 --2- 172.21.15.27:0/3096776776 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce500b23e0 0x7fce5006b600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fce380097c0 tx=0x7fce3801c390 comp rx=0 tx=0).stop 2024-01-15T18:11:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.396+0000 7fce57200640 1 -- 172.21.15.27:0/3096776776 shutdown_connections 2024-01-15T18:11:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.396+0000 7fce57200640 1 --2- 172.21.15.27:0/3096776776 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce500b23e0 0x7fce5006b600 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.396+0000 7fce57200640 1 --2- 172.21.15.27:0/3096776776 >> v2:172.21.15.93:3300/0 conn(0x7fce500c3e90 0x7fce500b1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.396+0000 7fce57200640 1 -- 172.21.15.27:0/3096776776 >> 172.21.15.27:0/3096776776 conn(0x7fce500b2a10 msgr2=0x7fce500b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:56.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.396+0000 7fce57200640 1 -- 172.21.15.27:0/3096776776 shutdown_connections 2024-01-15T18:11:56.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.396+0000 7fce57200640 1 -- 172.21.15.27:0/3096776776 wait complete. 2024-01-15T18:11:56.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.396+0000 7fce57200640 1 Processor -- start 2024-01-15T18:11:56.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.396+0000 7fce57200640 1 -- start start 2024-01-15T18:11:56.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.396+0000 7fce57200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce500b23e0 0x7fce500b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:56.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.396+0000 7fce57200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce500c3e90 0x7fce50160470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:56.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.396+0000 7fce57200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce500c4270 0x7fce5014bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:56.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.396+0000 7fce57200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce500b54a0 con 0x7fce500c4270 2024-01-15T18:11:56.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.396+0000 7fce57200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce500b5780 con 0x7fce500b23e0 2024-01-15T18:11:56.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.396+0000 7fce57200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce500b5a60 con 0x7fce500c3e90 2024-01-15T18:11:56.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.396+0000 7fce55200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce500c4270 0x7fce5014bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:56.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.396+0000 7fce55200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce500c4270 0x7fce5014bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58482/0 (socket says 172.21.15.27:58482) 2024-01-15T18:11:56.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.396+0000 7fce55200640 1 -- 172.21.15.27:0/657857971 learned_addr learned my addr 172.21.15.27:0/657857971 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:56.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.400+0000 7fce4f400640 1 --2- 172.21.15.27:0/657857971 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce500c3e90 0x7fce50160470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:56.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.400+0000 7fce4fe00640 1 --2- 172.21.15.27:0/657857971 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce500b23e0 0x7fce500b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:56.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.400+0000 7fce55200640 1 -- 172.21.15.27:0/657857971 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce500c3e90 msgr2=0x7fce50160470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:56.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.400+0000 7fce55200640 1 --2- 172.21.15.27:0/657857971 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce500c3e90 0x7fce50160470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.400+0000 7fce55200640 1 -- 172.21.15.27:0/657857971 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce500b23e0 msgr2=0x7fce500b4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:56.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.400+0000 7fce55200640 1 --2- 172.21.15.27:0/657857971 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce500b23e0 0x7fce500b4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.400+0000 7fce55200640 1 -- 172.21.15.27:0/657857971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce38009420 con 0x7fce500c4270 2024-01-15T18:11:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.400+0000 7fce4fe00640 1 --2- 172.21.15.27:0/657857971 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce500b23e0 0x7fce500b4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:11:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.400+0000 7fce55200640 1 --2- 172.21.15.27:0/657857971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce500c4270 0x7fce5014bee0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fce4800cae0 tx=0x7fce4800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.400+0000 7fce47e00640 1 -- 172.21.15.27:0/657857971 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4800ac40 con 0x7fce500c4270 2024-01-15T18:11:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.400+0000 7fce47e00640 1 -- 172.21.15.27:0/657857971 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce4800ada0 con 0x7fce500c4270 2024-01-15T18:11:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.400+0000 7fce57200640 1 -- 172.21.15.27:0/657857971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce500b5d40 con 0x7fce500c4270 2024-01-15T18:11:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.400+0000 7fce47e00640 1 -- 172.21.15.27:0/657857971 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce48012620 con 0x7fce500c4270 2024-01-15T18:11:56.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.400+0000 7fce57200640 1 -- 172.21.15.27:0/657857971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce500b5190 con 0x7fce500c4270 2024-01-15T18:11:56.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.400+0000 7fce57200640 1 -- 172.21.15.27:0/657857971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce500b6260 con 0x7fce500c4270 2024-01-15T18:11:56.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.400+0000 7fce47e00640 1 -- 172.21.15.27:0/657857971 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7fce48007070 con 0x7fce500c4270 2024-01-15T18:11:56.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.400+0000 7fce47e00640 1 --2- 172.21.15.27:0/657857971 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce280670d0 0x7fce28069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:56.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.404+0000 7fce47e00640 1 -- 172.21.15.27:0/657857971 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4913+0+0 (secure 0 0 0) 0x7fce48082830 con 0x7fce500c4270 2024-01-15T18:11:56.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.404+0000 7fce4fe00640 1 --2- 172.21.15.27:0/657857971 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce280670d0 0x7fce28069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:56.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.404+0000 7fce4fe00640 1 --2- 172.21.15.27:0/657857971 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce280670d0 0x7fce28069590 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fce5016c5b0 tx=0x7fce40008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:56.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.404+0000 7fce47e00640 1 -- 172.21.15.27:0/657857971 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fce500b6260 con 0x7fce500c4270 2024-01-15T18:11:56.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.624+0000 7fce57200640 1 -- 172.21.15.27:0/657857971 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fce500b6260 con 0x7fce500c4270 2024-01-15T18:11:56.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.624+0000 7fce47e00640 1 -- 172.21.15.27:0/657857971 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fce500b6260 con 0x7fce500c4270 2024-01-15T18:11:56.625 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:11:56.625 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:11:56.625 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-01-15T18:11:56.625 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:56.625 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-01-15T18:11:56.625 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlbMZIGBAAhSU4y2DGJd3MH0v6v3LsDg==", 2024-01-15T18:11:56.625 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:11:56.625 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:11:56.625 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:11:56.625 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:11:56.626 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:11:56.626 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:56.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:11:56.626 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:56.626 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-01-15T18:11:56.626 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlC7B9JBAAsXmBjV4KwPZejOmowXO2Zg==", 2024-01-15T18:11:56.626 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:11:56.626 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:11:56.626 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:11:56.626 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:11:56.626 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:11:56.626 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:56.626 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:11:56.626 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:56.627 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-01-15T18:11:56.627 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6Vla6a3HxAA/AHc6FSTccj0JGwiR4Vl7g==", 2024-01-15T18:11:56.627 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:11:56.627 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:11:56.627 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:11:56.627 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:11:56.627 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:11:56.627 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:56.627 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:11:56.627 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:56.627 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-01-15T18:11:56.627 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6Vl9nqzKxAAx2aDA8UwidAUrU9DBxGldA==", 2024-01-15T18:11:56.627 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-01-15T18:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCSc6Vly4TACRAAYBmMIcsZsRp4CZ5a3xCWgA==", 2024-01-15T18:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:56.628 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:11:56.629 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:56.629 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-01-15T18:11:56.629 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCVc6VlW/HEERAAOBgPz+eGpHwNXDe8fkYo7w==", 2024-01-15T18:11:56.629 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:11:56.629 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:11:56.629 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:11:56.629 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:11:56.629 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:56.629 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:11:56.629 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:56.629 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-01-15T18:11:56.629 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCYc6Vl+Ec4GBAAWRe/6u7u4RXApd57H8ihng==", 2024-01-15T18:11:56.629 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:11:56.629 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:11:56.629 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:11:56.630 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:11:56.630 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:56.630 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:11:56.630 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:56.630 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-01-15T18:11:56.630 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCbc6VlNdMPHxAAHzJvzLltsTl1wrEnI/iyrQ==", 2024-01-15T18:11:56.630 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:11:56.630 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:11:56.630 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:11:56.630 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:11:56.630 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:56.630 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:11:56.630 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:56.630 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-01-15T18:11:56.630 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQChc6Vlg8eVKBAAwZWAncCE2Lra+CxmSIxZkA==", 2024-01-15T18:11:56.631 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:11:56.631 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:11:56.631 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:11:56.631 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:11:56.631 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:56.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:11:56.631 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:56.631 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-01-15T18:11:56.631 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCkc6VlIXDOLxAAFISn6ILqq/aNFK07RxNnaw==", 2024-01-15T18:11:56.631 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:11:56.631 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:11:56.631 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:11:56.631 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:11:56.631 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:56.631 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:11:56.631 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:56.632 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-01-15T18:11:56.632 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCnc6Vl9Ru2NRAAl996VBDudw23SGiFPusIcw==", 2024-01-15T18:11:56.632 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:11:56.632 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:11:56.632 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:11:56.632 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:11:56.632 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:56.632 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:11:56.632 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:56.632 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-01-15T18:11:56.632 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCrc6VlHoAsARAAIoPMridLsIcpU3I9Hr5XLQ==", 2024-01-15T18:11:56.632 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:11:56.632 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:11:56.632 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:11:56.633 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:11:56.633 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:56.633 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:11:56.633 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:56.633 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-01-15T18:11:56.633 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlH+DCLxAA1sKVut+3WjA/B+aYJ6/Jeg==", 2024-01-15T18:11:56.633 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:11:56.633 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:11:56.633 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-01-15T18:11:56.633 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-15T18:11:56.633 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:56.633 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:11:56.633 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:56.633 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-01-15T18:11:56.633 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlaRBaORAAijn3UJLHbWm8ol2NiKNSaw==", 2024-01-15T18:11:56.634 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:11:56.634 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:11:56.634 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-15T18:11:56.634 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-15T18:11:56.634 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-15T18:11:56.634 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:56.634 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:11:56.634 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:56.634 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-01-15T18:11:56.634 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlDbLQNRAA82Lwz+pKY+3nHPAr1DichA==", 2024-01-15T18:11:56.634 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:11:56.634 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:11:56.634 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-15T18:11:56.634 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-15T18:11:56.634 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-15T18:11:56.635 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:56.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:11:56.635 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:56.635 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-01-15T18:11:56.635 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCPc6VlOyDaAxAADea0K1OEDTfrzsM7VEkixg==", 2024-01-15T18:11:56.635 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:11:56.635 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:11:56.635 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-15T18:11:56.635 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-15T18:11:56.635 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-15T18:11:56.635 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:56.635 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:11:56.635 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:56.635 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-01-15T18:11:56.635 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNc6VlK70BNxAAgYXkW/pMlzy8GZoZhX9NMA==", 2024-01-15T18:11:56.635 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:11:56.636 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-15T18:11:56.636 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:11:56.636 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:11:56.636 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:11:56.636 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:56.636 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:11:56.636 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:56.636 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-01-15T18:11:56.636 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlQbinExAAWtUdgkUNR3cCKV+ytcbUuQ==", 2024-01-15T18:11:56.636 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:11:56.636 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-15T18:11:56.636 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-15T18:11:56.636 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:11:56.636 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:56.636 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:11:56.637 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:56.637 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-01-15T18:11:56.637 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlzafrDxAAVVEBuX8BDT3cB9NfD2xTdQ==", 2024-01-15T18:11:56.637 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:11:56.637 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-15T18:11:56.637 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-15T18:11:56.637 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:11:56.637 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:56.637 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:56.637 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-01-15T18:11:56.637 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:11:56.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.624+0000 7fce57200640 1 -- 172.21.15.27:0/657857971 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce280670d0 msgr2=0x7fce28069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:56.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.624+0000 7fce57200640 1 --2- 172.21.15.27:0/657857971 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce280670d0 0x7fce28069590 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fce5016c5b0 tx=0x7fce40008040 comp rx=0 tx=0).stop 2024-01-15T18:11:56.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.624+0000 7fce57200640 1 -- 172.21.15.27:0/657857971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce500c4270 msgr2=0x7fce5014bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:56.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.624+0000 7fce57200640 1 --2- 172.21.15.27:0/657857971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce500c4270 0x7fce5014bee0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fce4800cae0 tx=0x7fce4800a060 comp rx=0 tx=0).stop 2024-01-15T18:11:56.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.624+0000 7fce57200640 1 -- 172.21.15.27:0/657857971 shutdown_connections 2024-01-15T18:11:56.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.624+0000 7fce57200640 1 --2- 172.21.15.27:0/657857971 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce280670d0 0x7fce28069590 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.624+0000 7fce57200640 1 --2- 172.21.15.27:0/657857971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce500c4270 0x7fce5014bee0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.624+0000 7fce57200640 1 --2- 172.21.15.27:0/657857971 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce500c3e90 0x7fce50160470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.624+0000 7fce57200640 1 --2- 172.21.15.27:0/657857971 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce500b23e0 0x7fce500b4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.624+0000 7fce57200640 1 -- 172.21.15.27:0/657857971 >> 172.21.15.27:0/657857971 conn(0x7fce500b2a10 msgr2=0x7fce500d0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:56.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.628+0000 7fce57200640 1 -- 172.21.15.27:0/657857971 shutdown_connections 2024-01-15T18:11:56.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.628+0000 7fce57200640 1 -- 172.21.15.27:0/657857971 wait complete. 2024-01-15T18:11:56.647 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:11:56.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.788+0000 7f9234800640 1 Processor -- start 2024-01-15T18:11:56.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.788+0000 7f9234800640 1 -- start start 2024-01-15T18:11:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.788+0000 7f9234800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f922c160b10 0x7f922c0afe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.788+0000 7f9234800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f922c0ce330 0x7f922c0cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.788+0000 7f9234800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f922c0d05b0 con 0x7f922c0cd7a0 2024-01-15T18:11:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.788+0000 7f9234800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f922c0ce7d0 con 0x7f922c160b10 2024-01-15T18:11:56.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.788+0000 7f9234800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f922c0cfc80 con 0x7f922c0ce330 2024-01-15T18:11:56.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.788+0000 7f9231e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f922c160b10 0x7f922c0afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:56.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.788+0000 7f9232800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f922c0ce330 0x7f922c0cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:56.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.788+0000 7f9232800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f922c0ce330 0x7f922c0cb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:32854/0 (socket says 172.21.15.27:32854) 2024-01-15T18:11:56.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.788+0000 7f9232800640 1 -- 172.21.15.27:0/1420633709 learned_addr learned my addr 172.21.15.27:0/1420633709 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:56.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.788+0000 7f9231e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f922c160b10 0x7f922c0afe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:35848/0 (socket says 172.21.15.27:35848) 2024-01-15T18:11:56.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.788+0000 7f9231e00640 1 -- 172.21.15.27:0/1420633709 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f922c0ce330 msgr2=0x7f922c0cb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:56.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.788+0000 7f9231e00640 1 --2- 172.21.15.27:0/1420633709 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f922c0ce330 0x7f922c0cb7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.788+0000 7f9231e00640 1 -- 172.21.15.27:0/1420633709 >> v1:172.21.15.93:6789/0 conn(0x7f922c0cd7a0 legacy=0x7f922c0cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:56.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.788+0000 7f9231e00640 1 -- 172.21.15.27:0/1420633709 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f922c0cff60 con 0x7f922c160b10 2024-01-15T18:11:56.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.788+0000 7f9232800640 1 --2- 172.21.15.27:0/1420633709 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f922c0ce330 0x7f922c0cb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:11:56.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.792+0000 7f9231e00640 1 --2- 172.21.15.27:0/1420633709 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f922c160b10 0x7f922c0afe70 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f921c0097c0 tx=0x7f921c01c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d239ce485959d46 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:56.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.792+0000 7f9230a00640 1 -- 172.21.15.27:0/1420633709 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c0ac640 con 0x7f922c160b10 2024-01-15T18:11:56.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.792+0000 7f9230a00640 1 -- 172.21.15.27:0/1420633709 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f921c0ac7a0 con 0x7f922c160b10 2024-01-15T18:11:56.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.792+0000 7f9230a00640 1 -- 172.21.15.27:0/1420633709 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c0acb30 con 0x7f922c160b10 2024-01-15T18:11:56.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.792+0000 7f9234800640 1 -- 172.21.15.27:0/1420633709 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f922c160b10 msgr2=0x7f922c0afe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:56.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.792+0000 7f9234800640 1 --2- 172.21.15.27:0/1420633709 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f922c160b10 0x7f922c0afe70 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f921c0097c0 tx=0x7f921c01c3c0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.792+0000 7f9234800640 1 -- 172.21.15.27:0/1420633709 shutdown_connections 2024-01-15T18:11:56.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.792+0000 7f9234800640 1 --2- 172.21.15.27:0/1420633709 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f922c0ce330 0x7f922c0cb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.792+0000 7f9234800640 1 --2- 172.21.15.27:0/1420633709 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f922c160b10 0x7f922c0afe70 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.792+0000 7f9234800640 1 -- 172.21.15.27:0/1420633709 >> 172.21.15.27:0/1420633709 conn(0x7f922c0d0f20 msgr2=0x7f922c0d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:56.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.792+0000 7f9234800640 1 -- 172.21.15.27:0/1420633709 shutdown_connections 2024-01-15T18:11:56.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.792+0000 7f9234800640 1 -- 172.21.15.27:0/1420633709 wait complete. 2024-01-15T18:11:56.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.796+0000 7f9234800640 1 Processor -- start 2024-01-15T18:11:56.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.796+0000 7f9234800640 1 -- start start 2024-01-15T18:11:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.796+0000 7f9234800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f922c0cd7a0 0x7f922c0c9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.796+0000 7f9234800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f922c0ce330 0x7f922c0d15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.796+0000 7f9234800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f922c160b10 0x7f922c1562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.796+0000 7f9234800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922c0c5150 con 0x7f922c0cd7a0 2024-01-15T18:11:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.796+0000 7f9234800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922c0c9940 con 0x7f922c160b10 2024-01-15T18:11:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.796+0000 7f9234800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f922c0c9c20 con 0x7f922c0ce330 2024-01-15T18:11:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.796+0000 7f9231e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f922c0cd7a0 0x7f922c0c9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:56.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.796+0000 7f9231e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f922c0cd7a0 0x7f922c0c9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58498/0 (socket says 172.21.15.27:58498) 2024-01-15T18:11:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.796+0000 7f9231e00640 1 -- 172.21.15.27:0/3981470879 learned_addr learned my addr 172.21.15.27:0/3981470879 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.796+0000 7f9232800640 1 --2- 172.21.15.27:0/3981470879 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f922c160b10 0x7f922c1562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.796+0000 7f9231400640 1 --2- 172.21.15.27:0/3981470879 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f922c0ce330 0x7f922c0d15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.796+0000 7f9231e00640 1 -- 172.21.15.27:0/3981470879 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f922c0ce330 msgr2=0x7f922c0d15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.796+0000 7f9231e00640 1 --2- 172.21.15.27:0/3981470879 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f922c0ce330 0x7f922c0d15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.796+0000 7f9231e00640 1 -- 172.21.15.27:0/3981470879 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f922c160b10 msgr2=0x7f922c1562c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.796+0000 7f9231e00640 1 --2- 172.21.15.27:0/3981470879 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f922c160b10 0x7f922c1562c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.796+0000 7f9231e00640 1 -- 172.21.15.27:0/3981470879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9224009b30 con 0x7f922c0cd7a0 2024-01-15T18:11:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.796+0000 7f9231e00640 1 --2- 172.21.15.27:0/3981470879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f922c0cd7a0 0x7f922c0c9040 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f921c019ff0 tx=0x7f921c019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.796+0000 7f922a000640 1 -- 172.21.15.27:0/3981470879 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c0b5610 con 0x7f922c0cd7a0 2024-01-15T18:11:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.796+0000 7f922a000640 1 -- 172.21.15.27:0/3981470879 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f921c0b5770 con 0x7f922c0cd7a0 2024-01-15T18:11:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.796+0000 7f922a000640 1 -- 172.21.15.27:0/3981470879 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c0b5b20 con 0x7f922c0cd7a0 2024-01-15T18:11:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.796+0000 7f9234800640 1 -- 172.21.15.27:0/3981470879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f921c009420 con 0x7f922c0cd7a0 2024-01-15T18:11:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.796+0000 7f9234800640 1 -- 172.21.15.27:0/3981470879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f922c0ca1e0 con 0x7f922c0cd7a0 2024-01-15T18:11:56.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.800+0000 7f9234800640 1 -- 172.21.15.27:0/3981470879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f922c0c52c0 con 0x7f922c0cd7a0 2024-01-15T18:11:56.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.800+0000 7f922a000640 1 -- 172.21.15.27:0/3981470879 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f921c0b3040 con 0x7f922c0cd7a0 2024-01-15T18:11:56.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.800+0000 7f922a000640 1 --2- 172.21.15.27:0/3981470879 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9204067100 0x7f92040695c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:56.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.800+0000 7f922a000640 1 -- 172.21.15.27:0/3981470879 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4913+0+0 (secure 0 0 0) 0x7f921c11cd90 con 0x7f922c0cd7a0 2024-01-15T18:11:56.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.800+0000 7f9231400640 1 --2- 172.21.15.27:0/3981470879 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9204067100 0x7f92040695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.800+0000 7f9231400640 1 --2- 172.21.15.27:0/3981470879 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9204067100 0x7f92040695c0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f922c0c7ef0 tx=0x7f9214009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:56.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.804+0000 7f922a000640 1 -- 172.21.15.27:0/3981470879 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f922c0c52c0 con 0x7f922c0cd7a0 2024-01-15T18:11:56.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.960+0000 7f9234800640 1 -- 172.21.15.27:0/3981470879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f922c0c5290 con 0x7f922c0cd7a0 2024-01-15T18:11:56.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.964+0000 7f922a000640 1 -- 172.21.15.27:0/3981470879 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+12848 (secure 0 0 0) 0x7f921c0b1030 con 0x7f922c0cd7a0 2024-01-15T18:11:56.964 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:11:56.964 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":66,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:11:56.227530+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:11:56.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.964+0000 7f9234800640 1 -- 172.21.15.27:0/3981470879 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9204067100 msgr2=0x7f92040695c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:56.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.964+0000 7f9234800640 1 --2- 172.21.15.27:0/3981470879 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9204067100 0x7f92040695c0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f922c0c7ef0 tx=0x7f9214009040 comp rx=0 tx=0).stop 2024-01-15T18:11:56.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.964+0000 7f9234800640 1 -- 172.21.15.27:0/3981470879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f922c0cd7a0 msgr2=0x7f922c0c9040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:56.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.964+0000 7f9234800640 1 --2- 172.21.15.27:0/3981470879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f922c0cd7a0 0x7f922c0c9040 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f921c019ff0 tx=0x7f921c019160 comp rx=0 tx=0).stop 2024-01-15T18:11:56.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.964+0000 7f9234800640 1 -- 172.21.15.27:0/3981470879 shutdown_connections 2024-01-15T18:11:56.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.964+0000 7f9234800640 1 --2- 172.21.15.27:0/3981470879 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9204067100 0x7f92040695c0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.964+0000 7f9234800640 1 --2- 172.21.15.27:0/3981470879 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f922c160b10 0x7f922c1562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.964+0000 7f9234800640 1 --2- 172.21.15.27:0/3981470879 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f922c0ce330 0x7f922c0d15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.964+0000 7f9234800640 1 --2- 172.21.15.27:0/3981470879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f922c0cd7a0 0x7f922c0c9040 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.964+0000 7f9234800640 1 -- 172.21.15.27:0/3981470879 >> 172.21.15.27:0/3981470879 conn(0x7f922c0d0f20 msgr2=0x7f922c156f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:56.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.964+0000 7f9234800640 1 -- 172.21.15.27:0/3981470879 shutdown_connections 2024-01-15T18:11:56.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:56.964+0000 7f9234800640 1 -- 172.21.15.27:0/3981470879 wait complete. 2024-01-15T18:11:56.986 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:11:56.987 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:11:57.038 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:11:57.042 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:11:57.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.128+0000 7f4162c00640 1 Processor -- start 2024-01-15T18:11:57.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.132+0000 7f4162c00640 1 -- start start 2024-01-15T18:11:57.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.132+0000 7f4162c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f415c0b1db0 0x7f415c0b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:57.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.132+0000 7f4162c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f415c0ce050 0x7f415c0ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:57.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.132+0000 7f4162c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c0712b0 con 0x7f415c06c430 2024-01-15T18:11:57.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.132+0000 7f4162c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f415c0cea20 con 0x7f415c0ce050 2024-01-15T18:11:57.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.132+0000 7f4162c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f415c0ccfb0 con 0x7f415c0b1db0 2024-01-15T18:11:57.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.132+0000 7f4160c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f415c0ce050 0x7f415c0ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:57.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.132+0000 7f415be00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f415c06c430 0x7f415c070ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48558/0 (socket says 172.21.15.27:48558) 2024-01-15T18:11:57.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.132+0000 7f415be00640 1 -- 172.21.15.27:0/2322297984 learned_addr learned my addr 172.21.15.27:0/2322297984 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:57.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.132+0000 7f415b400640 1 --2- 172.21.15.27:0/2322297984 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f415c0b1db0 0x7f415c0b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:57.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.132+0000 7f4160c00640 1 -- 172.21.15.27:0/2322297984 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f415c0b1db0 msgr2=0x7f415c0b2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:57.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.132+0000 7f4160c00640 1 --2- 172.21.15.27:0/2322297984 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f415c0b1db0 0x7f415c0b2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:57.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.132+0000 7f4160c00640 1 -- 172.21.15.27:0/2322297984 >> v1:172.21.15.27:6789/0 conn(0x7f415c06c430 legacy=0x7f415c070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:57.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.132+0000 7f4160c00640 1 -- 172.21.15.27:0/2322297984 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415c0cd290 con 0x7f415c0ce050 2024-01-15T18:11:57.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.136+0000 7f4160c00640 1 --2- 172.21.15.27:0/2322297984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f415c0ce050 0x7f415c0ce4e0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f415400aea0 tx=0x7f415401d9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cad77526e350a99 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:57.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.136+0000 7f415aa00640 1 -- 172.21.15.27:0/2322297984 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415400a2d0 con 0x7f415c0ce050 2024-01-15T18:11:57.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.136+0000 7f415aa00640 1 -- 172.21.15.27:0/2322297984 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41540a66f0 con 0x7f415c0ce050 2024-01-15T18:11:57.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.136+0000 7f415aa00640 1 -- 172.21.15.27:0/2322297984 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41540a6a80 con 0x7f415c0ce050 2024-01-15T18:11:57.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.136+0000 7f4162c00640 1 -- 172.21.15.27:0/2322297984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f415c0ce050 msgr2=0x7f415c0ce4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:57.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.136+0000 7f4162c00640 1 --2- 172.21.15.27:0/2322297984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f415c0ce050 0x7f415c0ce4e0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f415400aea0 tx=0x7f415401d9c0 comp rx=0 tx=0).stop 2024-01-15T18:11:57.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.136+0000 7f4162c00640 1 -- 172.21.15.27:0/2322297984 shutdown_connections 2024-01-15T18:11:57.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.136+0000 7f4162c00640 1 --2- 172.21.15.27:0/2322297984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f415c0ce050 0x7f415c0ce4e0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:57.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.136+0000 7f4162c00640 1 --2- 172.21.15.27:0/2322297984 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f415c0b1db0 0x7f415c0b2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:57.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.136+0000 7f4162c00640 1 -- 172.21.15.27:0/2322297984 >> 172.21.15.27:0/2322297984 conn(0x7f415c0cb700 msgr2=0x7f415c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:57.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.136+0000 7f4162c00640 1 -- 172.21.15.27:0/2322297984 shutdown_connections 2024-01-15T18:11:57.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.136+0000 7f4162c00640 1 -- 172.21.15.27:0/2322297984 wait complete. 2024-01-15T18:11:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.136+0000 7f4162c00640 1 Processor -- start 2024-01-15T18:11:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.140+0000 7f4162c00640 1 -- start start 2024-01-15T18:11:57.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.140+0000 7f4162c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f415c06c430 0x7f415c14a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:57.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.140+0000 7f4162c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f415c0b1db0 0x7f415c14f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:57.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.140+0000 7f4162c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f415c0ce050 0x7f415c14fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:57.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.140+0000 7f4162c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415c15be10 con 0x7f415c0ce050 2024-01-15T18:11:57.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.140+0000 7f4162c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415c154e90 con 0x7f415c06c430 2024-01-15T18:11:57.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.140+0000 7f4162c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f415c14ac50 con 0x7f415c0b1db0 2024-01-15T18:11:57.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.140+0000 7f4160c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f415c0ce050 0x7f415c14fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:57.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.140+0000 7f4160c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f415c0ce050 0x7f415c14fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58516/0 (socket says 172.21.15.27:58516) 2024-01-15T18:11:57.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.140+0000 7f4160c00640 1 -- 172.21.15.27:0/2456632966 learned_addr learned my addr 172.21.15.27:0/2456632966 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:57.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.140+0000 7f415b400640 1 --2- 172.21.15.27:0/2456632966 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f415c0b1db0 0x7f415c14f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:57.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.140+0000 7f415be00640 1 --2- 172.21.15.27:0/2456632966 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f415c06c430 0x7f415c14a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:57.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.140+0000 7f4160c00640 1 -- 172.21.15.27:0/2456632966 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f415c0b1db0 msgr2=0x7f415c14f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:57.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.140+0000 7f4160c00640 1 --2- 172.21.15.27:0/2456632966 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f415c0b1db0 0x7f415c14f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:57.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.140+0000 7f4160c00640 1 -- 172.21.15.27:0/2456632966 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f415c06c430 msgr2=0x7f415c14a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:57.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.140+0000 7f4160c00640 1 --2- 172.21.15.27:0/2456632966 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f415c06c430 0x7f415c14a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:57.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.140+0000 7f4160c00640 1 -- 172.21.15.27:0/2456632966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415400ab80 con 0x7f415c0ce050 2024-01-15T18:11:57.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.140+0000 7f4160c00640 1 --2- 172.21.15.27:0/2456632966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f415c0ce050 0x7f415c14fd80 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f41540b0040 tx=0x7f415401a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:57.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.140+0000 7f4153e00640 1 -- 172.21.15.27:0/2456632966 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41540bc070 con 0x7f415c0ce050 2024-01-15T18:11:57.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.140+0000 7f4153e00640 1 -- 172.21.15.27:0/2456632966 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f415400a2e0 con 0x7f415c0ce050 2024-01-15T18:11:57.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.140+0000 7f4162c00640 1 -- 172.21.15.27:0/2456632966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415c1f5a00 con 0x7f415c0ce050 2024-01-15T18:11:57.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.140+0000 7f4153e00640 1 -- 172.21.15.27:0/2456632966 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41540b76a0 con 0x7f415c0ce050 2024-01-15T18:11:57.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.140+0000 7f4162c00640 1 -- 172.21.15.27:0/2456632966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f415c1f5fa0 con 0x7f415c0ce050 2024-01-15T18:11:57.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.140+0000 7f4162c00640 1 -- 172.21.15.27:0/2456632966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f415c04e850 con 0x7f415c0ce050 2024-01-15T18:11:57.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.144+0000 7f4153e00640 1 -- 172.21.15.27:0/2456632966 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f41540b7800 con 0x7f415c0ce050 2024-01-15T18:11:57.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.144+0000 7f4153e00640 1 --2- 172.21.15.27:0/2456632966 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4134067100 0x7f41340695c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:57.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.144+0000 7f4153e00640 1 -- 172.21.15.27:0/2456632966 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4913+0+0 (secure 0 0 0) 0x7f415411d060 con 0x7f415c0ce050 2024-01-15T18:11:57.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.144+0000 7f415be00640 1 --2- 172.21.15.27:0/2456632966 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4134067100 0x7f41340695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:57.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.144+0000 7f415be00640 1 --2- 172.21.15.27:0/2456632966 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4134067100 0x7f41340695c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f414c005890 tx=0x7f414c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:57.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.148+0000 7f4153e00640 1 -- 172.21.15.27:0/2456632966 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f41540ebf50 con 0x7f415c0ce050 2024-01-15T18:11:57.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.308+0000 7f4162c00640 1 -- 172.21.15.27:0/2456632966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f415c15ced0 con 0x7f415c0ce050 2024-01-15T18:11:57.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.312+0000 7f4153e00640 1 -- 172.21.15.27:0/2456632966 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v66) v1 ==== 93+0+10 (secure 0 0 0) 0x7f41540ef570 con 0x7f415c0ce050 2024-01-15T18:11:57.312 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:11:57.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.312+0000 7f4162c00640 1 -- 172.21.15.27:0/2456632966 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4134067100 msgr2=0x7f41340695c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:57.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.312+0000 7f4162c00640 1 --2- 172.21.15.27:0/2456632966 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4134067100 0x7f41340695c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f414c005890 tx=0x7f414c009040 comp rx=0 tx=0).stop 2024-01-15T18:11:57.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.312+0000 7f4162c00640 1 -- 172.21.15.27:0/2456632966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f415c0ce050 msgr2=0x7f415c14fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:57.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.312+0000 7f4162c00640 1 --2- 172.21.15.27:0/2456632966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f415c0ce050 0x7f415c14fd80 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f41540b0040 tx=0x7f415401a760 comp rx=0 tx=0).stop 2024-01-15T18:11:57.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.312+0000 7f4162c00640 1 -- 172.21.15.27:0/2456632966 shutdown_connections 2024-01-15T18:11:57.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.312+0000 7f4162c00640 1 --2- 172.21.15.27:0/2456632966 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4134067100 0x7f41340695c0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:57.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.312+0000 7f4162c00640 1 --2- 172.21.15.27:0/2456632966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f415c0ce050 0x7f415c14fd80 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:57.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.312+0000 7f4162c00640 1 --2- 172.21.15.27:0/2456632966 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f415c0b1db0 0x7f415c14f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:57.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.312+0000 7f4162c00640 1 --2- 172.21.15.27:0/2456632966 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f415c06c430 0x7f415c14a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:57.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.312+0000 7f4162c00640 1 -- 172.21.15.27:0/2456632966 >> 172.21.15.27:0/2456632966 conn(0x7f415c0cb700 msgr2=0x7f415c0ae920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:57.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.312+0000 7f4162c00640 1 -- 172.21.15.27:0/2456632966 shutdown_connections 2024-01-15T18:11:57.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.312+0000 7f4162c00640 1 -- 172.21.15.27:0/2456632966 wait complete. 2024-01-15T18:11:57.334 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-15T18:11:57.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.472+0000 7f0cdd000640 1 Processor -- start 2024-01-15T18:11:57.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.472+0000 7f0cdd000640 1 -- start start 2024-01-15T18:11:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.476+0000 7f0cdd000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0cd80af600 0x7f0cd815fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.476+0000 7f0cdd000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd80d1ec0 con 0x7f0cd80b0130 2024-01-15T18:11:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.476+0000 7f0cdd000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd80ae160 con 0x7f0cd80cf080 2024-01-15T18:11:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.476+0000 7f0cdd000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd80cfb70 con 0x7f0cd80af600 2024-01-15T18:11:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.476+0000 7f0cd5400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0cd80b0130 0x7f0cd815f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48568/0 (socket says 172.21.15.27:48568) 2024-01-15T18:11:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.476+0000 7f0cd5400640 1 -- 172.21.15.27:0/3369448574 learned_addr learned my addr 172.21.15.27:0/3369448574 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:57.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.476+0000 7f0cd6800640 1 --2- 172.21.15.27:0/3369448574 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0cd80af600 0x7f0cd815fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.476+0000 7f0cd4a00640 1 -- 172.21.15.27:0/3369448574 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183366069 0 0) 0x7f0cd80d1ec0 con 0x7f0cd80b0130 2024-01-15T18:11:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.476+0000 7f0cd4a00640 1 -- 172.21.15.27:0/3369448574 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cb8003530 con 0x7f0cd80b0130 2024-01-15T18:11:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.476+0000 7f0cd4a00640 1 -- 172.21.15.27:0/3369448574 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922453771 0 0) 0x7f0cd80ae160 con 0x7f0cd80cf080 2024-01-15T18:11:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.476+0000 7f0cd4a00640 1 -- 172.21.15.27:0/3369448574 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cd80d1ec0 con 0x7f0cd80cf080 2024-01-15T18:11:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.476+0000 7f0cd6800640 1 -- 172.21.15.27:0/3369448574 >> v1:172.21.15.93:6789/0 conn(0x7f0cd80cf080 legacy=0x7f0cd80cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.476+0000 7f0cd6800640 1 -- 172.21.15.27:0/3369448574 >> v1:172.21.15.27:6789/0 conn(0x7f0cd80b0130 legacy=0x7f0cd815f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:57.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.476+0000 7f0cd6800640 1 -- 172.21.15.27:0/3369448574 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd80d1be0 con 0x7f0cd80af600 2024-01-15T18:11:57.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.480+0000 7f0cd6800640 1 --2- 172.21.15.27:0/3369448574 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0cd80af600 0x7f0cd815fa10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0009dc0 tx=0x7f0cd001c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19fdff587d167565 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:57.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.480+0000 7f0cd4a00640 1 -- 172.21.15.27:0/3369448574 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd00a5a20 con 0x7f0cd80af600 2024-01-15T18:11:57.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.480+0000 7f0cd4a00640 1 -- 172.21.15.27:0/3369448574 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd00a5b80 con 0x7f0cd80af600 2024-01-15T18:11:57.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.480+0000 7f0cd4a00640 1 -- 172.21.15.27:0/3369448574 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd00ad440 con 0x7f0cd80af600 2024-01-15T18:11:57.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.480+0000 7f0cdd000640 1 -- 172.21.15.27:0/3369448574 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0cd80af600 msgr2=0x7f0cd815fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:57.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.480+0000 7f0cdd000640 1 --2- 172.21.15.27:0/3369448574 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0cd80af600 0x7f0cd815fa10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0009dc0 tx=0x7f0cd001c940 comp rx=0 tx=0).stop 2024-01-15T18:11:57.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.480+0000 7f0cdd000640 1 -- 172.21.15.27:0/3369448574 shutdown_connections 2024-01-15T18:11:57.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.480+0000 7f0cdd000640 1 --2- 172.21.15.27:0/3369448574 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0cd80af600 0x7f0cd815fa10 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:57.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.480+0000 7f0cdd000640 1 -- 172.21.15.27:0/3369448574 >> 172.21.15.27:0/3369448574 conn(0x7f0cd80d0f20 msgr2=0x7f0cd80d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:57.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.480+0000 7f0cdd000640 1 -- 172.21.15.27:0/3369448574 shutdown_connections 2024-01-15T18:11:57.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.480+0000 7f0cdd000640 1 -- 172.21.15.27:0/3369448574 wait complete. 2024-01-15T18:11:57.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cdd000640 1 Processor -- start 2024-01-15T18:11:57.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cdd000640 1 -- start start 2024-01-15T18:11:57.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cdd000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0cd80af600 0x7f0cd815f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:57.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cdd000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0cd80b0130 0x7f0cd814efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:57.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cdd000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd80cf080 0x7f0cd80c9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cdd000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd80c85b0 con 0x7f0cd80cf080 2024-01-15T18:11:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cdd000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd80cdd40 con 0x7f0cd80af600 2024-01-15T18:11:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cdd000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd80cbdf0 con 0x7f0cd80b0130 2024-01-15T18:11:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cd6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd80cf080 0x7f0cd80c9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cd6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd80cf080 0x7f0cd80c9980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58520/0 (socket says 172.21.15.27:58520) 2024-01-15T18:11:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cd6800640 1 -- 172.21.15.27:0/91881009 learned_addr learned my addr 172.21.15.27:0/91881009 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cd5400640 1 --2- 172.21.15.27:0/91881009 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0cd80b0130 0x7f0cd814efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cd5e00640 1 --2- 172.21.15.27:0/91881009 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0cd80af600 0x7f0cd815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cd6800640 1 -- 172.21.15.27:0/91881009 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0cd80b0130 msgr2=0x7f0cd814efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cd6800640 1 --2- 172.21.15.27:0/91881009 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0cd80b0130 0x7f0cd814efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cd6800640 1 -- 172.21.15.27:0/91881009 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0cd80af600 msgr2=0x7f0cd815f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:57.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cd6800640 1 --2- 172.21.15.27:0/91881009 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0cd80af600 0x7f0cd815f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cd6800640 1 -- 172.21.15.27:0/91881009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd0009a70 con 0x7f0cd80cf080 2024-01-15T18:11:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cd5e00640 1 --2- 172.21.15.27:0/91881009 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0cd80af600 0x7f0cd815f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:11:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cd6800640 1 --2- 172.21.15.27:0/91881009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd80cf080 0x7f0cd80c9980 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f0cd001a570 tx=0x7f0cd00196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cce000640 1 -- 172.21.15.27:0/91881009 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd00adca0 con 0x7f0cd80cf080 2024-01-15T18:11:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cce000640 1 -- 172.21.15.27:0/91881009 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd00ade00 con 0x7f0cd80cf080 2024-01-15T18:11:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cdd000640 1 -- 172.21.15.27:0/91881009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd8150010 con 0x7f0cd80cf080 2024-01-15T18:11:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cce000640 1 -- 172.21.15.27:0/91881009 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd00b9690 con 0x7f0cd80cf080 2024-01-15T18:11:57.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cdd000640 1 -- 172.21.15.27:0/91881009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd80ce230 con 0x7f0cd80cf080 2024-01-15T18:11:57.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.484+0000 7f0cdd000640 1 -- 172.21.15.27:0/91881009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd804e850 con 0x7f0cd80cf080 2024-01-15T18:11:57.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.488+0000 7f0cce000640 1 -- 172.21.15.27:0/91881009 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f0cd00b97f0 con 0x7f0cd80cf080 2024-01-15T18:11:57.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.488+0000 7f0cce000640 1 --2- 172.21.15.27:0/91881009 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0cb00670d0 0x7f0cb0069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:57.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.488+0000 7f0cce000640 1 -- 172.21.15.27:0/91881009 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4913+0+0 (secure 0 0 0) 0x7f0cd011e110 con 0x7f0cd80cf080 2024-01-15T18:11:57.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.488+0000 7f0cd5e00640 1 --2- 172.21.15.27:0/91881009 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0cb00670d0 0x7f0cb0069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:57.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.488+0000 7f0cd5e00640 1 --2- 172.21.15.27:0/91881009 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0cb00670d0 0x7f0cb0069590 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80011f0 tx=0x7f0cc8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:57.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.492+0000 7f0cce000640 1 -- 172.21.15.27:0/91881009 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0cd00ed010 con 0x7f0cd80cf080 2024-01-15T18:11:57.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.652+0000 7f0cdd000640 1 -- 172.21.15.27:0/91881009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0cd80cbf30 con 0x7f0cd80cf080 2024-01-15T18:11:57.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.652+0000 7f0cce000640 1 -- 172.21.15.27:0/91881009 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v66) v1 ==== 84+0+65 (secure 0 0 0) 0x7f0cd0009450 con 0x7f0cd80cf080 2024-01-15T18:11:57.653 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:11:57.653 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:11:57.653 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:11:57.653 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-01-15T18:11:57.653 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-01-15T18:11:57.654 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:11:57.654 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:11:57.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.652+0000 7f0cdd000640 1 -- 172.21.15.27:0/91881009 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0cb00670d0 msgr2=0x7f0cb0069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:57.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.652+0000 7f0cdd000640 1 --2- 172.21.15.27:0/91881009 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0cb00670d0 0x7f0cb0069590 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80011f0 tx=0x7f0cc8009040 comp rx=0 tx=0).stop 2024-01-15T18:11:57.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.652+0000 7f0cdd000640 1 -- 172.21.15.27:0/91881009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd80cf080 msgr2=0x7f0cd80c9980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:57.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.652+0000 7f0cdd000640 1 --2- 172.21.15.27:0/91881009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd80cf080 0x7f0cd80c9980 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f0cd001a570 tx=0x7f0cd00196e0 comp rx=0 tx=0).stop 2024-01-15T18:11:57.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.652+0000 7f0cdd000640 1 -- 172.21.15.27:0/91881009 shutdown_connections 2024-01-15T18:11:57.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.652+0000 7f0cdd000640 1 --2- 172.21.15.27:0/91881009 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0cb00670d0 0x7f0cb0069590 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:57.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.652+0000 7f0cdd000640 1 --2- 172.21.15.27:0/91881009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0cd80cf080 0x7f0cd80c9980 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:57.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.652+0000 7f0cdd000640 1 --2- 172.21.15.27:0/91881009 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0cd80b0130 0x7f0cd814efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:57.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.652+0000 7f0cdd000640 1 --2- 172.21.15.27:0/91881009 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0cd80af600 0x7f0cd815f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:57.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.652+0000 7f0cdd000640 1 -- 172.21.15.27:0/91881009 >> 172.21.15.27:0/91881009 conn(0x7f0cd80d0f20 msgr2=0x7f0cd815cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:57.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.652+0000 7f0cdd000640 1 -- 172.21.15.27:0/91881009 shutdown_connections 2024-01-15T18:11:57.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.652+0000 7f0cdd000640 1 -- 172.21.15.27:0/91881009 wait complete. 2024-01-15T18:11:57.676 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-15T18:11:57.676 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-15T18:11:57.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.820+0000 7fe956200640 1 Processor -- start 2024-01-15T18:11:57.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.820+0000 7fe956200640 1 -- start start 2024-01-15T18:11:57.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.820+0000 7fe956200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe95015b4b0 0x7fe9500b1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:57.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.820+0000 7fe956200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9500cec60 0x7fe9500b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:57.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.820+0000 7fe956200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9500d19a0 con 0x7fe9500cf460 2024-01-15T18:11:57.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.820+0000 7fe956200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9500d1c80 con 0x7fe9500cec60 2024-01-15T18:11:57.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.820+0000 7fe956200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9500b07c0 con 0x7fe95015b4b0 2024-01-15T18:11:57.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.824+0000 7fe94fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9500cec60 0x7fe9500b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:57.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.824+0000 7fe94f400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe9500cf460 0x7fe9500cf840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48570/0 (socket says 172.21.15.27:48570) 2024-01-15T18:11:57.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.824+0000 7fe94f400640 1 -- 172.21.15.27:0/2025949748 learned_addr learned my addr 172.21.15.27:0/2025949748 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:57.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.824+0000 7fe94ea00640 1 --2- 172.21.15.27:0/2025949748 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe95015b4b0 0x7fe9500b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:57.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.824+0000 7fe94fe00640 1 -- 172.21.15.27:0/2025949748 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe95015b4b0 msgr2=0x7fe9500b1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:57.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.824+0000 7fe94fe00640 1 --2- 172.21.15.27:0/2025949748 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe95015b4b0 0x7fe9500b1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:57.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.824+0000 7fe94fe00640 1 -- 172.21.15.27:0/2025949748 >> v1:172.21.15.27:6789/0 conn(0x7fe9500cf460 legacy=0x7fe9500cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:57.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.824+0000 7fe94fe00640 1 -- 172.21.15.27:0/2025949748 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9500b0aa0 con 0x7fe9500cec60 2024-01-15T18:11:57.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.824+0000 7fe94fe00640 1 --2- 172.21.15.27:0/2025949748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9500cec60 0x7fe9500b1ac0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fe948009d90 tx=0x7fe94801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66ca33372ecdce server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:57.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.824+0000 7fe94e000640 1 -- 172.21.15.27:0/2025949748 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9500b2810 con 0x7fe9500cec60 2024-01-15T18:11:57.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.824+0000 7fe94e000640 1 -- 172.21.15.27:0/2025949748 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9480a5a10 con 0x7fe9500cec60 2024-01-15T18:11:57.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.824+0000 7fe94e000640 1 -- 172.21.15.27:0/2025949748 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9480a5da0 con 0x7fe9500cec60 2024-01-15T18:11:57.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.828+0000 7fe956200640 1 -- 172.21.15.27:0/2025949748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9500cec60 msgr2=0x7fe9500b1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:57.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.828+0000 7fe956200640 1 --2- 172.21.15.27:0/2025949748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9500cec60 0x7fe9500b1ac0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fe948009d90 tx=0x7fe94801c950 comp rx=0 tx=0).stop 2024-01-15T18:11:57.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.828+0000 7fe956200640 1 -- 172.21.15.27:0/2025949748 shutdown_connections 2024-01-15T18:11:57.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.828+0000 7fe956200640 1 --2- 172.21.15.27:0/2025949748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9500cec60 0x7fe9500b1ac0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:57.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.828+0000 7fe956200640 1 --2- 172.21.15.27:0/2025949748 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe95015b4b0 0x7fe9500b1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:57.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.828+0000 7fe956200640 1 -- 172.21.15.27:0/2025949748 >> 172.21.15.27:0/2025949748 conn(0x7fe9500d0e80 msgr2=0x7fe9500d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:57.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.828+0000 7fe956200640 1 -- 172.21.15.27:0/2025949748 shutdown_connections 2024-01-15T18:11:57.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.828+0000 7fe956200640 1 -- 172.21.15.27:0/2025949748 wait complete. 2024-01-15T18:11:57.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.828+0000 7fe956200640 1 Processor -- start 2024-01-15T18:11:57.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.828+0000 7fe956200640 1 -- start start 2024-01-15T18:11:57.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.828+0000 7fe956200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9500cec60 0x7fe9501f5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:57.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.828+0000 7fe956200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9500cf460 0x7fe9501f5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:57.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.828+0000 7fe956200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe95015b4b0 0x7fe9501f97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:57.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.828+0000 7fe956200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9501f5ed0 con 0x7fe9500cec60 2024-01-15T18:11:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.828+0000 7fe956200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9500aeeb0 con 0x7fe95015b4b0 2024-01-15T18:11:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.828+0000 7fe956200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9500af190 con 0x7fe9500cf460 2024-01-15T18:11:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.828+0000 7fe94f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9500cec60 0x7fe9501f5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.828+0000 7fe94f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9500cec60 0x7fe9501f5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58544/0 (socket says 172.21.15.27:58544) 2024-01-15T18:11:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.828+0000 7fe94f400640 1 -- 172.21.15.27:0/3290875139 learned_addr learned my addr 172.21.15.27:0/3290875139 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.832+0000 7fe94ea00640 1 --2- 172.21.15.27:0/3290875139 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9500cf460 0x7fe9501f5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.832+0000 7fe94fe00640 1 --2- 172.21.15.27:0/3290875139 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe95015b4b0 0x7fe9501f97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.832+0000 7fe94f400640 1 -- 172.21.15.27:0/3290875139 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9500cf460 msgr2=0x7fe9501f5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.832+0000 7fe94f400640 1 --2- 172.21.15.27:0/3290875139 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9500cf460 0x7fe9501f5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.832+0000 7fe94f400640 1 -- 172.21.15.27:0/3290875139 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe95015b4b0 msgr2=0x7fe9501f97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.832+0000 7fe94f400640 1 --2- 172.21.15.27:0/3290875139 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe95015b4b0 0x7fe9501f97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.832+0000 7fe94f400640 1 -- 172.21.15.27:0/3290875139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe948009a70 con 0x7fe9500cec60 2024-01-15T18:11:57.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.832+0000 7fe94f400640 1 --2- 172.21.15.27:0/3290875139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9500cec60 0x7fe9501f5450 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fe94000c500 tx=0x7fe94000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.832+0000 7fe947400640 1 -- 172.21.15.27:0/3290875139 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe940005cf0 con 0x7fe9500cec60 2024-01-15T18:11:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.832+0000 7fe947400640 1 -- 172.21.15.27:0/3290875139 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe940005e50 con 0x7fe9500cec60 2024-01-15T18:11:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.832+0000 7fe947400640 1 -- 172.21.15.27:0/3290875139 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9400072f0 con 0x7fe9500cec60 2024-01-15T18:11:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.832+0000 7fe956200640 1 -- 172.21.15.27:0/3290875139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9500af470 con 0x7fe9500cec60 2024-01-15T18:11:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.832+0000 7fe956200640 1 -- 172.21.15.27:0/3290875139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9500c8250 con 0x7fe9500cec60 2024-01-15T18:11:57.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.832+0000 7fe956200640 1 -- 172.21.15.27:0/3290875139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe95004e850 con 0x7fe9500cec60 2024-01-15T18:11:57.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.832+0000 7fe947400640 1 -- 172.21.15.27:0/3290875139 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7fe940007450 con 0x7fe9500cec60 2024-01-15T18:11:57.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.832+0000 7fe947400640 1 --2- 172.21.15.27:0/3290875139 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe9280670d0 0x7fe928069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:57.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.836+0000 7fe947400640 1 -- 172.21.15.27:0/3290875139 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4913+0+0 (secure 0 0 0) 0x7fe940081f90 con 0x7fe9500cec60 2024-01-15T18:11:57.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.836+0000 7fe94ea00640 1 --2- 172.21.15.27:0/3290875139 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe9280670d0 0x7fe928069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:57.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.836+0000 7fe94ea00640 1 --2- 172.21.15.27:0/3290875139 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe9280670d0 0x7fe928069590 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe9500c77b0 tx=0x7fe938008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:57.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.840+0000 7fe947400640 1 -- 172.21.15.27:0/3290875139 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe940050e80 con 0x7fe9500cec60 2024-01-15T18:11:57.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:57.996+0000 7fe956200640 1 -- 172.21.15.27:0/3290875139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fe9500af920 con 0x7fe9500cec60 2024-01-15T18:11:58.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:58.984+0000 7fe947400640 1 -- 172.21.15.27:0/3290875139 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v67) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe940017020 con 0x7fe9500cec60 2024-01-15T18:11:58.985 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-01-15T18:11:58.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:58.984+0000 7fe956200640 1 -- 172.21.15.27:0/3290875139 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe9280670d0 msgr2=0x7fe928069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:58.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:58.984+0000 7fe956200640 1 --2- 172.21.15.27:0/3290875139 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe9280670d0 0x7fe928069590 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe9500c77b0 tx=0x7fe938008040 comp rx=0 tx=0).stop 2024-01-15T18:11:58.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:58.984+0000 7fe956200640 1 -- 172.21.15.27:0/3290875139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9500cec60 msgr2=0x7fe9501f5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:58.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:58.984+0000 7fe956200640 1 --2- 172.21.15.27:0/3290875139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9500cec60 0x7fe9501f5450 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fe94000c500 tx=0x7fe94000c9c0 comp rx=0 tx=0).stop 2024-01-15T18:11:58.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:58.984+0000 7fe956200640 1 -- 172.21.15.27:0/3290875139 shutdown_connections 2024-01-15T18:11:58.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:58.984+0000 7fe956200640 1 --2- 172.21.15.27:0/3290875139 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe9280670d0 0x7fe928069590 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:58.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:58.984+0000 7fe956200640 1 --2- 172.21.15.27:0/3290875139 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe95015b4b0 0x7fe9501f97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:58.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:58.984+0000 7fe956200640 1 --2- 172.21.15.27:0/3290875139 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9500cf460 0x7fe9501f5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:58.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:58.984+0000 7fe956200640 1 --2- 172.21.15.27:0/3290875139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9500cec60 0x7fe9501f5450 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:58.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:58.984+0000 7fe956200640 1 -- 172.21.15.27:0/3290875139 >> 172.21.15.27:0/3290875139 conn(0x7fe9500d0e80 msgr2=0x7fe9500cd1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:58.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:58.988+0000 7fe956200640 1 -- 172.21.15.27:0/3290875139 shutdown_connections 2024-01-15T18:11:58.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:58.988+0000 7fe956200640 1 -- 172.21.15.27:0/3290875139 wait complete. 2024-01-15T18:11:59.017 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-15T18:11:59.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.168+0000 7f3e49400640 1 Processor -- start 2024-01-15T18:11:59.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.168+0000 7f3e49400640 1 -- start start 2024-01-15T18:11:59.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.172+0000 7f3e49400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e440cde10 0x7f3e440cbf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:59.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.172+0000 7f3e49400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f3e440cfc30 0x7f3e440cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:59.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.172+0000 7f3e49400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e440cb4d0 con 0x7f3e4415d680 2024-01-15T18:11:59.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.172+0000 7f3e49400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e440d0120 con 0x7f3e440cfc30 2024-01-15T18:11:59.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.172+0000 7f3e49400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e440d0400 con 0x7f3e440cde10 2024-01-15T18:11:59.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.172+0000 7f3e42800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3e4415d680 0x7f3e440cd700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48576/0 (socket says 172.21.15.27:48576) 2024-01-15T18:11:59.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.172+0000 7f3e42800640 1 -- 172.21.15.27:0/3454413050 learned_addr learned my addr 172.21.15.27:0/3454413050 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:11:59.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.172+0000 7f3e41e00640 1 --2- 172.21.15.27:0/3454413050 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e440cde10 0x7f3e440cbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:59.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.172+0000 7f3e43200640 1 --2- 172.21.15.27:0/3454413050 >> v2:172.21.15.93:3300/0 conn(0x7f3e440cfc30 0x7f3e440cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:59.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.172+0000 7f3e41400640 1 -- 172.21.15.27:0/3454413050 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3866065225 0 0) 0x7f3e440cb4d0 con 0x7f3e4415d680 2024-01-15T18:11:59.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.172+0000 7f3e41400640 1 -- 172.21.15.27:0/3454413050 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e24003530 con 0x7f3e4415d680 2024-01-15T18:11:59.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.172+0000 7f3e41e00640 1 -- 172.21.15.27:0/3454413050 >> v2:172.21.15.93:3300/0 conn(0x7f3e440cfc30 msgr2=0x7f3e440cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:59.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.172+0000 7f3e41400640 1 -- 172.21.15.27:0/3454413050 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1627421280 0 0) 0x7f3e24003530 con 0x7f3e4415d680 2024-01-15T18:11:59.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.172+0000 7f3e41e00640 1 --2- 172.21.15.27:0/3454413050 >> v2:172.21.15.93:3300/0 conn(0x7f3e440cfc30 0x7f3e440cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:59.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.172+0000 7f3e41e00640 1 -- 172.21.15.27:0/3454413050 >> v1:172.21.15.27:6789/0 conn(0x7f3e4415d680 legacy=0x7f3e440cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:59.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.172+0000 7f3e41e00640 1 -- 172.21.15.27:0/3454413050 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e440cb770 con 0x7f3e440cde10 2024-01-15T18:11:59.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.172+0000 7f3e43200640 1 --2- 172.21.15.27:0/3454413050 >> v2:172.21.15.93:3300/0 conn(0x7f3e440cfc30 0x7f3e440cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:11:59.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.176+0000 7f3e41e00640 1 --2- 172.21.15.27:0/3454413050 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e440cde10 0x7f3e440cbf70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c0097c0 tx=0x7f3e2c01c370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd3c0f5a0f3ae106 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:59.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.176+0000 7f3e41400640 1 -- 172.21.15.27:0/3454413050 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e440ce850 con 0x7f3e440cde10 2024-01-15T18:11:59.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.176+0000 7f3e41400640 1 -- 172.21.15.27:0/3454413050 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e2c0a7050 con 0x7f3e440cde10 2024-01-15T18:11:59.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.176+0000 7f3e41400640 1 -- 172.21.15.27:0/3454413050 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e34001680 con 0x7f3e440cde10 2024-01-15T18:11:59.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.176+0000 7f3e49400640 1 -- 172.21.15.27:0/3454413050 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e440cde10 msgr2=0x7f3e440cbf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:59.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.176+0000 7f3e49400640 1 --2- 172.21.15.27:0/3454413050 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e440cde10 0x7f3e440cbf70 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c0097c0 tx=0x7f3e2c01c370 comp rx=0 tx=0).stop 2024-01-15T18:11:59.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.176+0000 7f3e49400640 1 -- 172.21.15.27:0/3454413050 shutdown_connections 2024-01-15T18:11:59.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.176+0000 7f3e49400640 1 --2- 172.21.15.27:0/3454413050 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e440cde10 0x7f3e440cbf70 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:59.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.176+0000 7f3e49400640 1 --2- 172.21.15.27:0/3454413050 >> v2:172.21.15.93:3300/0 conn(0x7f3e440cfc30 0x7f3e440cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:59.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.176+0000 7f3e49400640 1 -- 172.21.15.27:0/3454413050 >> 172.21.15.27:0/3454413050 conn(0x7f3e440d0e80 msgr2=0x7f3e440d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:59.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.176+0000 7f3e49400640 1 -- 172.21.15.27:0/3454413050 shutdown_connections 2024-01-15T18:11:59.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.176+0000 7f3e49400640 1 -- 172.21.15.27:0/3454413050 wait complete. 2024-01-15T18:11:59.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.176+0000 7f3e49400640 1 Processor -- start 2024-01-15T18:11:59.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.176+0000 7f3e49400640 1 -- start start 2024-01-15T18:11:59.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.180+0000 7f3e49400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e440cde10 0x7f3e44152850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:59.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.180+0000 7f3e49400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e440cfc30 0x7f3e44155420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:59.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.180+0000 7f3e49400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3e4415d680 0x7f3e4415f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:59.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.180+0000 7f3e49400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e440c9c50 con 0x7f3e440cde10 2024-01-15T18:11:59.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.180+0000 7f3e49400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e44152d90 con 0x7f3e4415d680 2024-01-15T18:11:59.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.180+0000 7f3e49400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e440c9040 con 0x7f3e440cfc30 2024-01-15T18:11:59.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.180+0000 7f3e42800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e440cde10 0x7f3e44152850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:59.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.180+0000 7f3e41e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e440cfc30 0x7f3e44155420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:59.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.180+0000 7f3e42800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e440cde10 0x7f3e44152850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58550/0 (socket says 172.21.15.27:58550) 2024-01-15T18:11:59.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.180+0000 7f3e42800640 1 -- 172.21.15.27:0/2804393731 learned_addr learned my addr 172.21.15.27:0/2804393731 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:11:59.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.180+0000 7f3e43200640 1 --2- 172.21.15.27:0/2804393731 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3e4415d680 0x7f3e4415f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:59.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.180+0000 7f3e41e00640 1 -- 172.21.15.27:0/2804393731 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3e4415d680 msgr2=0x7f3e4415f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:59.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.180+0000 7f3e41e00640 1 --2- 172.21.15.27:0/2804393731 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3e4415d680 0x7f3e4415f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:59.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.180+0000 7f3e41e00640 1 -- 172.21.15.27:0/2804393731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e440cde10 msgr2=0x7f3e44152850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:59.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.180+0000 7f3e41e00640 1 --2- 172.21.15.27:0/2804393731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e440cde10 0x7f3e44152850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:59.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.180+0000 7f3e41e00640 1 -- 172.21.15.27:0/2804393731 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3c009a70 con 0x7f3e440cfc30 2024-01-15T18:11:59.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.180+0000 7f3e41e00640 1 --2- 172.21.15.27:0/2804393731 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e440cfc30 0x7f3e44155420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c0ab000 tx=0x7f3e2c019110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:59.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.180+0000 7f3e3aa00640 1 -- 172.21.15.27:0/2804393731 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c0bb070 con 0x7f3e440cfc30 2024-01-15T18:11:59.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.180+0000 7f3e3aa00640 1 -- 172.21.15.27:0/2804393731 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e2c003830 con 0x7f3e440cfc30 2024-01-15T18:11:59.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.180+0000 7f3e49400640 1 -- 172.21.15.27:0/2804393731 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e2c009420 con 0x7f3e440cfc30 2024-01-15T18:11:59.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.180+0000 7f3e3aa00640 1 -- 172.21.15.27:0/2804393731 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c003ba0 con 0x7f3e440cfc30 2024-01-15T18:11:59.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.180+0000 7f3e49400640 1 -- 172.21.15.27:0/2804393731 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e440c9600 con 0x7f3e440cfc30 2024-01-15T18:11:59.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.180+0000 7f3e49400640 1 -- 172.21.15.27:0/2804393731 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e440c9dc0 con 0x7f3e440cfc30 2024-01-15T18:11:59.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.180+0000 7f3e3aa00640 1 -- 172.21.15.27:0/2804393731 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f3e2c0b2050 con 0x7f3e440cfc30 2024-01-15T18:11:59.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.184+0000 7f3e3aa00640 1 --2- 172.21.15.27:0/2804393731 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e1c067100 0x7f3e1c0695c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:59.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.184+0000 7f3e3aa00640 1 -- 172.21.15.27:0/2804393731 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5283+0+0 (secure 0 0 0) 0x7f3e2c11c480 con 0x7f3e440cfc30 2024-01-15T18:11:59.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.184+0000 7f3e42800640 1 --2- 172.21.15.27:0/2804393731 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e1c067100 0x7f3e1c0695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:59.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.184+0000 7f3e42800640 1 --2- 172.21.15.27:0/2804393731 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e1c067100 0x7f3e1c0695c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f3e34001ec0 tx=0x7f3e34001740 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:59.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.188+0000 7f3e3aa00640 1 -- 172.21.15.27:0/2804393731 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3e2c0eb410 con 0x7f3e440cfc30 2024-01-15T18:11:59.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:11:59.344+0000 7f3e49400640 1 -- 172.21.15.27:0/2804393731 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f3e4404e850 con 0x7f3e440cfc30 2024-01-15T18:12:00.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.004+0000 7f3e3aa00640 1 -- 172.21.15.27:0/2804393731 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v68) v1 ==== 170+0+0 (secure 0 0 0) 0x7f3e2c0eea30 con 0x7f3e440cfc30 2024-01-15T18:12:00.010 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-01-15T18:12:00.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.016+0000 7f3e49400640 1 -- 172.21.15.27:0/2804393731 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e1c067100 msgr2=0x7f3e1c0695c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:00.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.016+0000 7f3e49400640 1 --2- 172.21.15.27:0/2804393731 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e1c067100 0x7f3e1c0695c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f3e34001ec0 tx=0x7f3e34001740 comp rx=0 tx=0).stop 2024-01-15T18:12:00.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.016+0000 7f3e49400640 1 -- 172.21.15.27:0/2804393731 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e440cfc30 msgr2=0x7f3e44155420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:00.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.016+0000 7f3e49400640 1 --2- 172.21.15.27:0/2804393731 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e440cfc30 0x7f3e44155420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c0ab000 tx=0x7f3e2c019110 comp rx=0 tx=0).stop 2024-01-15T18:12:00.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.016+0000 7f3e49400640 1 -- 172.21.15.27:0/2804393731 shutdown_connections 2024-01-15T18:12:00.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.016+0000 7f3e49400640 1 --2- 172.21.15.27:0/2804393731 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e1c067100 0x7f3e1c0695c0 secure :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f3e34001ec0 tx=0x7f3e34001740 comp rx=0 tx=0).stop 2024-01-15T18:12:00.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.016+0000 7f3e49400640 1 --2- 172.21.15.27:0/2804393731 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3e4415d680 0x7f3e4415f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:00.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.016+0000 7f3e49400640 1 --2- 172.21.15.27:0/2804393731 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e440cfc30 0x7f3e44155420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:00.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.016+0000 7f3e49400640 1 --2- 172.21.15.27:0/2804393731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e440cde10 0x7f3e44152850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:00.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.016+0000 7f3e49400640 1 -- 172.21.15.27:0/2804393731 >> 172.21.15.27:0/2804393731 conn(0x7f3e440d0e80 msgr2=0x7f3e440d1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:00.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.020+0000 7f3e49400640 1 -- 172.21.15.27:0/2804393731 shutdown_connections 2024-01-15T18:12:00.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.020+0000 7f3e49400640 1 -- 172.21.15.27:0/2804393731 wait complete. 2024-01-15T18:12:00.046 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-15T18:12:00.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.188+0000 7fa426400640 1 Processor -- start 2024-01-15T18:12:00.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.188+0000 7fa426400640 1 -- start start 2024-01-15T18:12:00.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.192+0000 7fa426400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa4200cf460 0x7fa4200cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:00.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.192+0000 7fa426400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa4200b2590 0x7fa4200cc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:00.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.192+0000 7fa426400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4200cfd80 con 0x7fa42015b4b0 2024-01-15T18:12:00.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.192+0000 7fa426400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4200d19a0 con 0x7fa4200cf460 2024-01-15T18:12:00.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.192+0000 7fa426400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4200d1c80 con 0x7fa4200b2590 2024-01-15T18:12:00.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.192+0000 7fa41f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa4200cf460 0x7fa4200cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:00.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.192+0000 7fa41ea00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa42015b4b0 0x7fa4200cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48580/0 (socket says 172.21.15.27:48580) 2024-01-15T18:12:00.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.192+0000 7fa41ea00640 1 -- 172.21.15.27:0/3498020104 learned_addr learned my addr 172.21.15.27:0/3498020104 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:12:00.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.192+0000 7fa41f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa4200cf460 0x7fa4200cf840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58562/0 (socket says 172.21.15.27:58562) 2024-01-15T18:12:00.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.192+0000 7fa41fe00640 1 --2- 172.21.15.27:0/3498020104 >> v2:172.21.15.93:3300/0 conn(0x7fa4200b2590 0x7fa4200cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:00.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.192+0000 7fa41f400640 1 -- 172.21.15.27:0/3498020104 >> v2:172.21.15.93:3300/0 conn(0x7fa4200b2590 msgr2=0x7fa4200cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:00.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.192+0000 7fa41f400640 1 --2- 172.21.15.27:0/3498020104 >> v2:172.21.15.93:3300/0 conn(0x7fa4200b2590 0x7fa4200cc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:00.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.192+0000 7fa41f400640 1 -- 172.21.15.27:0/3498020104 >> v1:172.21.15.27:6789/0 conn(0x7fa42015b4b0 legacy=0x7fa4200cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:00.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.192+0000 7fa41f400640 1 -- 172.21.15.27:0/3498020104 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4200b07c0 con 0x7fa4200cf460 2024-01-15T18:12:00.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.196+0000 7fa41f400640 1 --2- 172.21.15.27:0/3498020104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4200cf460 0x7fa4200cf840 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fa4200d08b0 tx=0x7fa41001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a6b90bcae29f0fb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:00.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.196+0000 7fa41e000640 1 -- 172.21.15.27:0/3498020104 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4100ab660 con 0x7fa4200cf460 2024-01-15T18:12:00.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.196+0000 7fa41e000640 1 -- 172.21.15.27:0/3498020104 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4100ab7c0 con 0x7fa4200cf460 2024-01-15T18:12:00.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.196+0000 7fa41e000640 1 -- 172.21.15.27:0/3498020104 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4100abb50 con 0x7fa4200cf460 2024-01-15T18:12:00.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.196+0000 7fa426400640 1 -- 172.21.15.27:0/3498020104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4200cf460 msgr2=0x7fa4200cf840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:00.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.196+0000 7fa426400640 1 --2- 172.21.15.27:0/3498020104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4200cf460 0x7fa4200cf840 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fa4200d08b0 tx=0x7fa41001c460 comp rx=0 tx=0).stop 2024-01-15T18:12:00.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.196+0000 7fa426400640 1 -- 172.21.15.27:0/3498020104 shutdown_connections 2024-01-15T18:12:00.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.196+0000 7fa426400640 1 --2- 172.21.15.27:0/3498020104 >> v2:172.21.15.93:3300/0 conn(0x7fa4200b2590 0x7fa4200cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:00.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.196+0000 7fa426400640 1 --2- 172.21.15.27:0/3498020104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4200cf460 0x7fa4200cf840 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:00.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.196+0000 7fa426400640 1 -- 172.21.15.27:0/3498020104 >> 172.21.15.27:0/3498020104 conn(0x7fa4200d0e80 msgr2=0x7fa4200d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:00.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.196+0000 7fa426400640 1 -- 172.21.15.27:0/3498020104 shutdown_connections 2024-01-15T18:12:00.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.196+0000 7fa426400640 1 -- 172.21.15.27:0/3498020104 wait complete. 2024-01-15T18:12:00.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.196+0000 7fa426400640 1 Processor -- start 2024-01-15T18:12:00.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.196+0000 7fa426400640 1 -- start start 2024-01-15T18:12:00.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.200+0000 7fa426400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4200b2590 0x7fa420148c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:00.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.200+0000 7fa426400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa4200cf460 0x7fa4200c9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:00.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.200+0000 7fa426400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa42015b4b0 0x7fa420152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:00.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.200+0000 7fa426400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42015eb80 con 0x7fa42015b4b0 2024-01-15T18:12:00.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.200+0000 7fa426400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4200ca250 con 0x7fa4200cf460 2024-01-15T18:12:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.200+0000 7fa426400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa420158f80 con 0x7fa4200b2590 2024-01-15T18:12:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.200+0000 7fa41fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa42015b4b0 0x7fa420152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.200+0000 7fa41fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa42015b4b0 0x7fa420152a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58570/0 (socket says 172.21.15.27:58570) 2024-01-15T18:12:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.200+0000 7fa41fe00640 1 -- 172.21.15.27:0/3923671711 learned_addr learned my addr 172.21.15.27:0/3923671711 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.200+0000 7fa41f400640 1 --2- 172.21.15.27:0/3923671711 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4200b2590 0x7fa420148c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:00.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.200+0000 7fa41ea00640 1 --2- 172.21.15.27:0/3923671711 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa4200cf460 0x7fa4200c9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.200+0000 7fa41fe00640 1 -- 172.21.15.27:0/3923671711 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4200b2590 msgr2=0x7fa420148c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:00.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.200+0000 7fa41fe00640 1 --2- 172.21.15.27:0/3923671711 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4200b2590 0x7fa420148c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:00.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.200+0000 7fa41fe00640 1 -- 172.21.15.27:0/3923671711 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa4200cf460 msgr2=0x7fa4200c9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:00.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.200+0000 7fa41fe00640 1 --2- 172.21.15.27:0/3923671711 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa4200cf460 0x7fa4200c9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:00.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.200+0000 7fa41fe00640 1 -- 172.21.15.27:0/3923671711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa410009420 con 0x7fa42015b4b0 2024-01-15T18:12:00.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.200+0000 7fa41fe00640 1 --2- 172.21.15.27:0/3923671711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa42015b4b0 0x7fa420152a70 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fa418007e60 tx=0x7fa41800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:00.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.200+0000 7fa417400640 1 -- 172.21.15.27:0/3923671711 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa418014620 con 0x7fa42015b4b0 2024-01-15T18:12:00.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.200+0000 7fa417400640 1 -- 172.21.15.27:0/3923671711 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa418014780 con 0x7fa42015b4b0 2024-01-15T18:12:00.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.200+0000 7fa417400640 1 -- 172.21.15.27:0/3923671711 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa418014b30 con 0x7fa42015b4b0 2024-01-15T18:12:00.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.200+0000 7fa426400640 1 -- 172.21.15.27:0/3923671711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa420149150 con 0x7fa42015b4b0 2024-01-15T18:12:00.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.200+0000 7fa426400640 1 -- 172.21.15.27:0/3923671711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa420158930 con 0x7fa42015b4b0 2024-01-15T18:12:00.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.200+0000 7fa426400640 1 -- 172.21.15.27:0/3923671711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa420159110 con 0x7fa42015b4b0 2024-01-15T18:12:00.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.204+0000 7fa417400640 1 -- 172.21.15.27:0/3923671711 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7fa418014c90 con 0x7fa42015b4b0 2024-01-15T18:12:00.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.204+0000 7fa417400640 1 --2- 172.21.15.27:0/3923671711 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa3f8067100 0x7fa3f80695c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:00.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.204+0000 7fa417400640 1 -- 172.21.15.27:0/3923671711 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5665+0+0 (secure 0 0 0) 0x7fa418083e40 con 0x7fa42015b4b0 2024-01-15T18:12:00.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.204+0000 7fa41f400640 1 --2- 172.21.15.27:0/3923671711 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa3f8067100 0x7fa3f80695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:00.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.204+0000 7fa41f400640 1 --2- 172.21.15.27:0/3923671711 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa3f8067100 0x7fa3f80695c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fa4100af000 tx=0x7fa4100a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:00.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.208+0000 7fa417400640 1 -- 172.21.15.27:0/3923671711 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa418052d30 con 0x7fa42015b4b0 2024-01-15T18:12:00.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:00.400+0000 7fa426400640 1 -- 172.21.15.27:0/3923671711 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa42004e850 con 0x7fa42015b4b0 2024-01-15T18:12:01.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.016+0000 7fa417400640 1 -- 172.21.15.27:0/3923671711 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v43) v1 ==== 174+0+0 (secure 0 0 0) 0x7fa418056350 con 0x7fa42015b4b0 2024-01-15T18:12:01.022 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 10 and data pool 11 2024-01-15T18:12:01.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.024+0000 7fa426400640 1 -- 172.21.15.27:0/3923671711 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa3f8067100 msgr2=0x7fa3f80695c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:01.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.024+0000 7fa426400640 1 --2- 172.21.15.27:0/3923671711 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa3f8067100 0x7fa3f80695c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fa4100af000 tx=0x7fa4100a7000 comp rx=0 tx=0).stop 2024-01-15T18:12:01.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.024+0000 7fa426400640 1 -- 172.21.15.27:0/3923671711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa42015b4b0 msgr2=0x7fa420152a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:01.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.024+0000 7fa426400640 1 --2- 172.21.15.27:0/3923671711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa42015b4b0 0x7fa420152a70 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fa418007e60 tx=0x7fa41800e4b0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.028+0000 7fa426400640 1 -- 172.21.15.27:0/3923671711 shutdown_connections 2024-01-15T18:12:01.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.028+0000 7fa426400640 1 --2- 172.21.15.27:0/3923671711 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa3f8067100 0x7fa3f80695c0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.028+0000 7fa426400640 1 --2- 172.21.15.27:0/3923671711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa42015b4b0 0x7fa420152a70 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.028+0000 7fa426400640 1 --2- 172.21.15.27:0/3923671711 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa4200cf460 0x7fa4200c9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.028+0000 7fa426400640 1 --2- 172.21.15.27:0/3923671711 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4200b2590 0x7fa420148c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.028+0000 7fa426400640 1 -- 172.21.15.27:0/3923671711 >> 172.21.15.27:0/3923671711 conn(0x7fa4200d0e80 msgr2=0x7fa4200ada70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:01.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.032+0000 7fa426400640 1 -- 172.21.15.27:0/3923671711 shutdown_connections 2024-01-15T18:12:01.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.036+0000 7fa426400640 1 -- 172.21.15.27:0/3923671711 wait complete. 2024-01-15T18:12:01.065 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:12:01.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.208+0000 7ff250800640 1 Processor -- start 2024-01-15T18:12:01.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.208+0000 7ff250800640 1 -- start start 2024-01-15T18:12:01.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.208+0000 7ff250800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff24815e7d0 0x7ff2480cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.208+0000 7ff250800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2480cedc0 0x7ff2480cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.208+0000 7ff250800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff2480ce2e0 0x7ff2480cba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.208+0000 7ff250800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2480b38b0 con 0x7ff24815e7d0 2024-01-15T18:12:01.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.208+0000 7ff250800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2480cd550 con 0x7ff2480ce2e0 2024-01-15T18:12:01.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.208+0000 7ff250800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2480cd270 con 0x7ff2480cedc0 2024-01-15T18:12:01.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.208+0000 7ff24de00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff24815e7d0 0x7ff2480cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.208+0000 7ff24de00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff24815e7d0 0x7ff2480cf2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58580/0 (socket says 172.21.15.27:58580) 2024-01-15T18:12:01.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.208+0000 7ff24de00640 1 -- 172.21.15.27:0/2992459519 learned_addr learned my addr 172.21.15.27:0/2992459519 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:01.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.208+0000 7ff24e800640 1 --2- 172.21.15.27:0/2992459519 >> v2:172.21.15.93:3300/0 conn(0x7ff2480ce2e0 0x7ff2480cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.208+0000 7ff24d400640 1 --2- 172.21.15.27:0/2992459519 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2480cedc0 0x7ff2480cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.208+0000 7ff24de00640 1 -- 172.21.15.27:0/2992459519 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2480cedc0 msgr2=0x7ff2480cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:01.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.208+0000 7ff24de00640 1 --2- 172.21.15.27:0/2992459519 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2480cedc0 0x7ff2480cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.208+0000 7ff24de00640 1 -- 172.21.15.27:0/2992459519 >> v2:172.21.15.93:3300/0 conn(0x7ff2480ce2e0 msgr2=0x7ff2480cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:01.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.208+0000 7ff24de00640 1 --2- 172.21.15.27:0/2992459519 >> v2:172.21.15.93:3300/0 conn(0x7ff2480ce2e0 0x7ff2480cba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.208+0000 7ff24de00640 1 -- 172.21.15.27:0/2992459519 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2480b35d0 con 0x7ff24815e7d0 2024-01-15T18:12:01.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.212+0000 7ff24de00640 1 --2- 172.21.15.27:0/2992459519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff24815e7d0 0x7ff2480cf2a0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7ff248071b60 tx=0x7ff23801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84dd73723a1908c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:01.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.212+0000 7ff24ca00640 1 -- 172.21.15.27:0/2992459519 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2380ac410 con 0x7ff24815e7d0 2024-01-15T18:12:01.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.212+0000 7ff24ca00640 1 -- 172.21.15.27:0/2992459519 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2380ac570 con 0x7ff24815e7d0 2024-01-15T18:12:01.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.212+0000 7ff24ca00640 1 -- 172.21.15.27:0/2992459519 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2380ac900 con 0x7ff24815e7d0 2024-01-15T18:12:01.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.212+0000 7ff250800640 1 -- 172.21.15.27:0/2992459519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff24815e7d0 msgr2=0x7ff2480cf2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:01.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.212+0000 7ff250800640 1 --2- 172.21.15.27:0/2992459519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff24815e7d0 0x7ff2480cf2a0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7ff248071b60 tx=0x7ff23801c500 comp rx=0 tx=0).stop 2024-01-15T18:12:01.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.212+0000 7ff250800640 1 -- 172.21.15.27:0/2992459519 shutdown_connections 2024-01-15T18:12:01.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.212+0000 7ff250800640 1 --2- 172.21.15.27:0/2992459519 >> v2:172.21.15.93:3300/0 conn(0x7ff2480ce2e0 0x7ff2480cba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.212+0000 7ff250800640 1 --2- 172.21.15.27:0/2992459519 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2480cedc0 0x7ff2480cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.212+0000 7ff250800640 1 --2- 172.21.15.27:0/2992459519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff24815e7d0 0x7ff2480cf2a0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.212+0000 7ff250800640 1 -- 172.21.15.27:0/2992459519 >> 172.21.15.27:0/2992459519 conn(0x7ff2480b2a10 msgr2=0x7ff2480b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:01.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.212+0000 7ff250800640 1 -- 172.21.15.27:0/2992459519 shutdown_connections 2024-01-15T18:12:01.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.212+0000 7ff250800640 1 -- 172.21.15.27:0/2992459519 wait complete. 2024-01-15T18:12:01.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff250800640 1 Processor -- start 2024-01-15T18:12:01.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff250800640 1 -- start start 2024-01-15T18:12:01.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff250800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2480ce2e0 0x7ff24814b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff250800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2480cedc0 0x7ff24806b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff250800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff24815e7d0 0x7ff2480af840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff250800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff248159cc0 con 0x7ff24815e7d0 2024-01-15T18:12:01.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff250800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24815a560 con 0x7ff2480cedc0 2024-01-15T18:12:01.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff250800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24814bb50 con 0x7ff2480ce2e0 2024-01-15T18:12:01.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff24e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff24815e7d0 0x7ff2480af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff24e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff24815e7d0 0x7ff2480af840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58592/0 (socket says 172.21.15.27:58592) 2024-01-15T18:12:01.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff24e800640 1 -- 172.21.15.27:0/1450645232 learned_addr learned my addr 172.21.15.27:0/1450645232 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:01.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff24de00640 1 --2- 172.21.15.27:0/1450645232 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2480ce2e0 0x7ff24814b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff24d400640 1 --2- 172.21.15.27:0/1450645232 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2480cedc0 0x7ff24806b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff24e800640 1 -- 172.21.15.27:0/1450645232 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2480ce2e0 msgr2=0x7ff24814b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:01.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff24e800640 1 --2- 172.21.15.27:0/1450645232 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2480ce2e0 0x7ff24814b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff24e800640 1 -- 172.21.15.27:0/1450645232 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2480cedc0 msgr2=0x7ff24806b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:01.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff24e800640 1 --2- 172.21.15.27:0/1450645232 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2480cedc0 0x7ff24806b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff24e800640 1 -- 172.21.15.27:0/1450645232 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2380094d0 con 0x7ff24815e7d0 2024-01-15T18:12:01.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff24de00640 1 --2- 172.21.15.27:0/1450645232 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2480ce2e0 0x7ff24814b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:12:01.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff24e800640 1 --2- 172.21.15.27:0/1450645232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff24815e7d0 0x7ff2480af840 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7ff240007ec0 tx=0x7ff24000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:01.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff246000640 1 -- 172.21.15.27:0/1450645232 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff240014420 con 0x7ff24815e7d0 2024-01-15T18:12:01.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff246000640 1 -- 172.21.15.27:0/1450645232 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2400145a0 con 0x7ff24815e7d0 2024-01-15T18:12:01.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff250800640 1 -- 172.21.15.27:0/1450645232 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2480b02c0 con 0x7ff24815e7d0 2024-01-15T18:12:01.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff246000640 1 -- 172.21.15.27:0/1450645232 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff240014950 con 0x7ff24815e7d0 2024-01-15T18:12:01.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff250800640 1 -- 172.21.15.27:0/1450645232 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff24815a1f0 con 0x7ff24815e7d0 2024-01-15T18:12:01.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.216+0000 7ff250800640 1 -- 172.21.15.27:0/1450645232 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff24804e850 con 0x7ff24815e7d0 2024-01-15T18:12:01.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.220+0000 7ff246000640 1 -- 172.21.15.27:0/1450645232 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7ff240014ab0 con 0x7ff24815e7d0 2024-01-15T18:12:01.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.220+0000 7ff246000640 1 --2- 172.21.15.27:0/1450645232 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff2200670d0 0x7ff220069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.220+0000 7ff246000640 1 -- 172.21.15.27:0/1450645232 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5765+0+0 (secure 0 0 0) 0x7ff2400832c0 con 0x7ff24815e7d0 2024-01-15T18:12:01.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.220+0000 7ff24de00640 1 --2- 172.21.15.27:0/1450645232 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff2200670d0 0x7ff220069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.220+0000 7ff24de00640 1 --2- 172.21.15.27:0/1450645232 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff2200670d0 0x7ff220069590 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff2380ae000 tx=0x7ff238009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:01.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.224+0000 7ff246000640 1 -- 172.21.15.27:0/1450645232 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff2400521b0 con 0x7ff24815e7d0 2024-01-15T18:12:01.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.380+0000 7ff250800640 1 -- 172.21.15.27:0/1450645232 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff24815a6a0 con 0x7ff24815e7d0 2024-01-15T18:12:01.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.384+0000 7ff246000640 1 -- 172.21.15.27:0/1450645232 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16572 (secure 0 0 0) 0x7ff2400557d0 con 0x7ff24815e7d0 2024-01-15T18:12:01.384 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:12:01.384 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":69,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:12:00.405082+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:11:58.044871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-15T18:11:59.394705+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:12:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.384+0000 7ff250800640 1 -- 172.21.15.27:0/1450645232 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff2200670d0 msgr2=0x7ff220069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.384+0000 7ff250800640 1 --2- 172.21.15.27:0/1450645232 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff2200670d0 0x7ff220069590 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff2380ae000 tx=0x7ff238009040 comp rx=0 tx=0).stop 2024-01-15T18:12:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.384+0000 7ff250800640 1 -- 172.21.15.27:0/1450645232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff24815e7d0 msgr2=0x7ff2480af840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:01.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.384+0000 7ff250800640 1 --2- 172.21.15.27:0/1450645232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff24815e7d0 0x7ff2480af840 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7ff240007ec0 tx=0x7ff24000e4b0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.384+0000 7ff250800640 1 -- 172.21.15.27:0/1450645232 shutdown_connections 2024-01-15T18:12:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.384+0000 7ff250800640 1 --2- 172.21.15.27:0/1450645232 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff2200670d0 0x7ff220069590 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.384+0000 7ff250800640 1 --2- 172.21.15.27:0/1450645232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff24815e7d0 0x7ff2480af840 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.384+0000 7ff250800640 1 --2- 172.21.15.27:0/1450645232 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2480cedc0 0x7ff24806b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.384+0000 7ff250800640 1 --2- 172.21.15.27:0/1450645232 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2480ce2e0 0x7ff24814b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.384+0000 7ff250800640 1 -- 172.21.15.27:0/1450645232 >> 172.21.15.27:0/1450645232 conn(0x7ff2480b2a10 msgr2=0x7ff24815c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.384+0000 7ff250800640 1 -- 172.21.15.27:0/1450645232 shutdown_connections 2024-01-15T18:12:01.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.384+0000 7ff250800640 1 -- 172.21.15.27:0/1450645232 wait complete. 2024-01-15T18:12:01.407 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:12:01.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.548+0000 7f7055a00640 1 Processor -- start 2024-01-15T18:12:01.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.548+0000 7f7055a00640 1 -- start start 2024-01-15T18:12:01.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.548+0000 7f7055a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7050159bf0 0x7f7050159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.548+0000 7f7055a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f705006d430 0x7f70500cd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.548+0000 7f7055a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7050159a50 con 0x7f7050071e40 2024-01-15T18:12:01.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.548+0000 7f7055a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70500cce00 con 0x7f705006d430 2024-01-15T18:12:01.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.548+0000 7f7055a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f705015a510 con 0x7f7050159bf0 2024-01-15T18:12:01.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.548+0000 7f704f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f705006d430 0x7f70500cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.548+0000 7f704f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f705006d430 0x7f70500cd250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58596/0 (socket says 172.21.15.27:58596) 2024-01-15T18:12:01.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.548+0000 7f704f400640 1 -- 172.21.15.27:0/2366548044 learned_addr learned my addr 172.21.15.27:0/2366548044 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:01.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.548+0000 7f704ea00640 1 --2- 172.21.15.27:0/2366548044 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7050159bf0 0x7f7050159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.548+0000 7f704f400640 1 -- 172.21.15.27:0/2366548044 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7050159bf0 msgr2=0x7f7050159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:01.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.548+0000 7f704f400640 1 --2- 172.21.15.27:0/2366548044 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7050159bf0 0x7f7050159fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.548+0000 7f704f400640 1 -- 172.21.15.27:0/2366548044 >> v1:172.21.15.93:6789/0 conn(0x7f7050071e40 legacy=0x7f70500c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:01.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.548+0000 7f704f400640 1 -- 172.21.15.27:0/2366548044 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f705006d980 con 0x7f705006d430 2024-01-15T18:12:01.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.552+0000 7f704f400640 1 --2- 172.21.15.27:0/2366548044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705006d430 0x7f70500cd250 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f7048009e50 tx=0x7f704801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a65a1012ac44d4ac server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:01.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.552+0000 7f704d600640 1 -- 172.21.15.27:0/2366548044 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70480af400 con 0x7f705006d430 2024-01-15T18:12:01.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.552+0000 7f704d600640 1 -- 172.21.15.27:0/2366548044 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70480ab050 con 0x7f705006d430 2024-01-15T18:12:01.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.552+0000 7f704d600640 1 -- 172.21.15.27:0/2366548044 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70480af780 con 0x7f705006d430 2024-01-15T18:12:01.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.552+0000 7f7055a00640 1 -- 172.21.15.27:0/2366548044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705006d430 msgr2=0x7f70500cd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:01.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.552+0000 7f7055a00640 1 --2- 172.21.15.27:0/2366548044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705006d430 0x7f70500cd250 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f7048009e50 tx=0x7f704801f1f0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.552+0000 7f7055a00640 1 -- 172.21.15.27:0/2366548044 shutdown_connections 2024-01-15T18:12:01.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.552+0000 7f7055a00640 1 --2- 172.21.15.27:0/2366548044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705006d430 0x7f70500cd250 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.552+0000 7f7055a00640 1 --2- 172.21.15.27:0/2366548044 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7050159bf0 0x7f7050159fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.552+0000 7f7055a00640 1 -- 172.21.15.27:0/2366548044 >> 172.21.15.27:0/2366548044 conn(0x7f70500cec60 msgr2=0x7f70500cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:01.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.552+0000 7f7055a00640 1 -- 172.21.15.27:0/2366548044 shutdown_connections 2024-01-15T18:12:01.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.552+0000 7f7055a00640 1 -- 172.21.15.27:0/2366548044 wait complete. 2024-01-15T18:12:01.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.552+0000 7f7055a00640 1 Processor -- start 2024-01-15T18:12:01.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f7055a00640 1 -- start start 2024-01-15T18:12:01.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f7055a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705006d430 0x7f70501510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f7055a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7050159bf0 0x7f70500b3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f7055a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f70500b4290 0x7f70500b2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f7055a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705015a780 con 0x7f705006d430 2024-01-15T18:12:01.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f7055a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7050151f60 con 0x7f7050159bf0 2024-01-15T18:12:01.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f7055a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70501516c0 con 0x7f70500b4290 2024-01-15T18:12:01.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f704ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705006d430 0x7f70501510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f704ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705006d430 0x7f70501510e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58610/0 (socket says 172.21.15.27:58610) 2024-01-15T18:12:01.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f704ea00640 1 -- 172.21.15.27:0/2461684930 learned_addr learned my addr 172.21.15.27:0/2461684930 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:01.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f704f400640 1 --2- 172.21.15.27:0/2461684930 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f70500b4290 0x7f70500b2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f704e000640 1 --2- 172.21.15.27:0/2461684930 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7050159bf0 0x7f70500b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f704ea00640 1 -- 172.21.15.27:0/2461684930 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f70500b4290 msgr2=0x7f70500b2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:01.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f704ea00640 1 --2- 172.21.15.27:0/2461684930 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f70500b4290 0x7f70500b2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f704ea00640 1 -- 172.21.15.27:0/2461684930 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7050159bf0 msgr2=0x7f70500b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:01.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f704ea00640 1 --2- 172.21.15.27:0/2461684930 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7050159bf0 0x7f70500b3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f704ea00640 1 -- 172.21.15.27:0/2461684930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7048009b30 con 0x7f705006d430 2024-01-15T18:12:01.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f704f400640 1 --2- 172.21.15.27:0/2461684930 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f70500b4290 0x7f70500b2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:01.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f704e000640 1 --2- 172.21.15.27:0/2461684930 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7050159bf0 0x7f70500b3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:01.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f704ea00640 1 --2- 172.21.15.27:0/2461684930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705006d430 0x7f70501510e0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f704000e870 tx=0x7f704000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:01.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f7046a00640 1 -- 172.21.15.27:0/2461684930 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7040009510 con 0x7f705006d430 2024-01-15T18:12:01.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f7046a00640 1 -- 172.21.15.27:0/2461684930 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704000eec0 con 0x7f705006d430 2024-01-15T18:12:01.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f7055a00640 1 -- 172.21.15.27:0/2461684930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70501519a0 con 0x7f705006d430 2024-01-15T18:12:01.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f7046a00640 1 -- 172.21.15.27:0/2461684930 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704000cb40 con 0x7f705006d430 2024-01-15T18:12:01.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f7055a00640 1 -- 172.21.15.27:0/2461684930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70500c4b90 con 0x7f705006d430 2024-01-15T18:12:01.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.556+0000 7f7055a00640 1 -- 172.21.15.27:0/2461684930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7050073c00 con 0x7f705006d430 2024-01-15T18:12:01.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.560+0000 7f7046a00640 1 -- 172.21.15.27:0/2461684930 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f704000cca0 con 0x7f705006d430 2024-01-15T18:12:01.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.560+0000 7f7046a00640 1 --2- 172.21.15.27:0/2461684930 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f70280670d0 0x7f7028069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.560+0000 7f7046a00640 1 -- 172.21.15.27:0/2461684930 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f7040082cb0 con 0x7f705006d430 2024-01-15T18:12:01.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.560+0000 7f704e000640 1 --2- 172.21.15.27:0/2461684930 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f70280670d0 0x7f7028069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.560+0000 7f704e000640 1 --2- 172.21.15.27:0/2461684930 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f70280670d0 0x7f7028069590 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f705015ba40 tx=0x7f7038009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:01.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.564+0000 7f7046a00640 1 -- 172.21.15.27:0/2461684930 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7050073c00 con 0x7f705006d430 2024-01-15T18:12:01.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.720+0000 7f7055a00640 1 -- 172.21.15.27:0/2461684930 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7050073c00 con 0x7f705006d430 2024-01-15T18:12:01.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.724+0000 7f7046a00640 1 -- 172.21.15.27:0/2461684930 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16572 (secure 0 0 0) 0x7f7050073c00 con 0x7f705006d430 2024-01-15T18:12:01.724 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:12:01.724 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":69,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:12:00.405082+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:11:58.044871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-15T18:11:59.394705+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:12:01.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.724+0000 7f7055a00640 1 -- 172.21.15.27:0/2461684930 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f70280670d0 msgr2=0x7f7028069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:01.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.724+0000 7f7055a00640 1 --2- 172.21.15.27:0/2461684930 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f70280670d0 0x7f7028069590 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f705015ba40 tx=0x7f7038009040 comp rx=0 tx=0).stop 2024-01-15T18:12:01.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.724+0000 7f7055a00640 1 -- 172.21.15.27:0/2461684930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705006d430 msgr2=0x7f70501510e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:01.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.724+0000 7f7055a00640 1 --2- 172.21.15.27:0/2461684930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705006d430 0x7f70501510e0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f704000e870 tx=0x7f704000ec70 comp rx=0 tx=0).stop 2024-01-15T18:12:01.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.724+0000 7f7055a00640 1 -- 172.21.15.27:0/2461684930 shutdown_connections 2024-01-15T18:12:01.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.724+0000 7f7055a00640 1 --2- 172.21.15.27:0/2461684930 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f70280670d0 0x7f7028069590 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.724+0000 7f7055a00640 1 --2- 172.21.15.27:0/2461684930 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f70500b4290 0x7f70500b2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.724+0000 7f7055a00640 1 --2- 172.21.15.27:0/2461684930 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7050159bf0 0x7f70500b3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.724+0000 7f7055a00640 1 --2- 172.21.15.27:0/2461684930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f705006d430 0x7f70501510e0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.724+0000 7f7055a00640 1 -- 172.21.15.27:0/2461684930 >> 172.21.15.27:0/2461684930 conn(0x7f70500cec60 msgr2=0x7f705006c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:01.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.724+0000 7f7055a00640 1 -- 172.21.15.27:0/2461684930 shutdown_connections 2024-01-15T18:12:01.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.724+0000 7f7055a00640 1 -- 172.21.15.27:0/2461684930 wait complete. 2024-01-15T18:12:01.746 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-15T18:12:01.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.888+0000 7fc2d4a00640 1 Processor -- start 2024-01-15T18:12:01.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.888+0000 7fc2d4a00640 1 -- start start 2024-01-15T18:12:01.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d4a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2cc0d1a40 0x7fc2cc0cba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d4a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2cc0cbfa0 con 0x7fc2cc0cf450 2024-01-15T18:12:01.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d4a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2cc0cefa0 con 0x7fc2cc0b0650 2024-01-15T18:12:01.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d4a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc0ce240 con 0x7fc2cc0d1a40 2024-01-15T18:12:01.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d2000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc2cc0cf450 0x7fc2cc0cf830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48588/0 (socket says 172.21.15.27:48588) 2024-01-15T18:12:01.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d2000640 1 -- 172.21.15.27:0/2478470499 learned_addr learned my addr 172.21.15.27:0/2478470499 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:12:01.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d1600640 1 --2- 172.21.15.27:0/2478470499 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2cc0d1a40 0x7fc2cc0cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d0c00640 1 -- 172.21.15.27:0/2478470499 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4208195631 0 0) 0x7fc2cc0cbfa0 con 0x7fc2cc0cf450 2024-01-15T18:12:01.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d0c00640 1 -- 172.21.15.27:0/2478470499 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2ac003530 con 0x7fc2cc0cf450 2024-01-15T18:12:01.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d0c00640 1 -- 172.21.15.27:0/2478470499 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3933480667 0 0) 0x7fc2ac003530 con 0x7fc2cc0cf450 2024-01-15T18:12:01.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d0c00640 1 -- 172.21.15.27:0/2478470499 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2cc0d1a40 msgr2=0x7fc2cc0cba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:01.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d0c00640 1 --2- 172.21.15.27:0/2478470499 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2cc0d1a40 0x7fc2cc0cba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d0c00640 1 -- 172.21.15.27:0/2478470499 >> v1:172.21.15.93:6789/0 conn(0x7fc2cc0b0650 legacy=0x7fc2cc0b0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:01.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d0c00640 1 -- 172.21.15.27:0/2478470499 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2cc0ccd70 con 0x7fc2cc0cf450 2024-01-15T18:12:01.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d0c00640 1 -- 172.21.15.27:0/2478470499 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fc2bc001680 con 0x7fc2cc0cf450 2024-01-15T18:12:01.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d0c00640 1 -- 172.21.15.27:0/2478470499 >> v1:172.21.15.27:6789/0 conn(0x7fc2cc0cf450 legacy=0x7fc2cc0cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:01.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d0c00640 1 --2- 172.21.15.27:0/2478470499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2ac003a20 0x7fc2ac003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d0c00640 1 --2- 172.21.15.27:0/2478470499 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2ac0047d0 0x7fc2ac004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d0c00640 1 --2- 172.21.15.27:0/2478470499 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2cc0d1a40 0x7fc2ac005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d0c00640 1 -- 172.21.15.27:0/2478470499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc0ce240 con 0x7fc2ac003a20 2024-01-15T18:12:01.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d0c00640 1 -- 172.21.15.27:0/2478470499 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ac00bac0 con 0x7fc2ac0047d0 2024-01-15T18:12:01.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d0c00640 1 -- 172.21.15.27:0/2478470499 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ac00bc80 con 0x7fc2cc0d1a40 2024-01-15T18:12:01.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d2000640 1 --2- 172.21.15.27:0/2478470499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2ac003a20 0x7fc2ac003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d1600640 1 --2- 172.21.15.27:0/2478470499 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2ac0047d0 0x7fc2ac004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d2a00640 1 --2- 172.21.15.27:0/2478470499 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2cc0d1a40 0x7fc2ac005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d2000640 1 -- 172.21.15.27:0/2478470499 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2cc0d1a40 msgr2=0x7fc2ac005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:01.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d2000640 1 --2- 172.21.15.27:0/2478470499 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2cc0d1a40 0x7fc2ac005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d2000640 1 -- 172.21.15.27:0/2478470499 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2ac0047d0 msgr2=0x7fc2ac004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:01.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d2000640 1 --2- 172.21.15.27:0/2478470499 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2ac0047d0 0x7fc2ac004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.892+0000 7fc2d2000640 1 -- 172.21.15.27:0/2478470499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ac003530 con 0x7fc2ac003a20 2024-01-15T18:12:01.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.896+0000 7fc2d2000640 1 --2- 172.21.15.27:0/2478470499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2ac003a20 0x7fc2ac003e40 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc001060 tx=0x7fc2bc01cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e8e9fcc9100dbba server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:01.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.896+0000 7fc2d0c00640 1 -- 172.21.15.27:0/2478470499 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc0058e0 con 0x7fc2ac003a20 2024-01-15T18:12:01.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.896+0000 7fc2d0c00640 1 -- 172.21.15.27:0/2478470499 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2bc005a40 con 0x7fc2ac003a20 2024-01-15T18:12:01.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.896+0000 7fc2d0c00640 1 -- 172.21.15.27:0/2478470499 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc005df0 con 0x7fc2ac003a20 2024-01-15T18:12:01.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.896+0000 7fc2d4a00640 1 -- 172.21.15.27:0/2478470499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2ac003a20 msgr2=0x7fc2ac003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:01.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.896+0000 7fc2d4a00640 1 --2- 172.21.15.27:0/2478470499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2ac003a20 0x7fc2ac003e40 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc001060 tx=0x7fc2bc01cc00 comp rx=0 tx=0).stop 2024-01-15T18:12:01.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.896+0000 7fc2d2a00640 1 -- 172.21.15.27:0/2478470499 reap_dead start 2024-01-15T18:12:01.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.896+0000 7fc2d4a00640 1 -- 172.21.15.27:0/2478470499 shutdown_connections 2024-01-15T18:12:01.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.896+0000 7fc2d4a00640 1 -- 172.21.15.27:0/2478470499 >> 172.21.15.27:0/2478470499 conn(0x7fc2cc0d0e70 msgr2=0x7fc2cc0d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:01.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.896+0000 7fc2d4a00640 1 -- 172.21.15.27:0/2478470499 shutdown_connections 2024-01-15T18:12:01.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.896+0000 7fc2d4a00640 1 -- 172.21.15.27:0/2478470499 wait complete. 2024-01-15T18:12:01.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.900+0000 7fc2d4a00640 1 Processor -- start 2024-01-15T18:12:01.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.900+0000 7fc2d4a00640 1 -- start start 2024-01-15T18:12:01.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.900+0000 7fc2d4a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2ac003a20 0x7fc2cc0b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.900+0000 7fc2d4a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc158270 0x7fc2cc157a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.900+0000 7fc2d4a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2cc166420 0x7fc2cc1668a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.900+0000 7fc2d4a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc166de0 con 0x7fc2cc158270 2024-01-15T18:12:01.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.900+0000 7fc2d4a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc0c55f0 con 0x7fc2ac003a20 2024-01-15T18:12:01.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.900+0000 7fc2d4a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2cc0c52f0 con 0x7fc2cc166420 2024-01-15T18:12:01.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.900+0000 7fc2d1600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc158270 0x7fc2cc157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.900+0000 7fc2d1600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc158270 0x7fc2cc157a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58616/0 (socket says 172.21.15.27:58616) 2024-01-15T18:12:01.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.900+0000 7fc2d1600640 1 -- 172.21.15.27:0/2940560680 learned_addr learned my addr 172.21.15.27:0/2940560680 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:01.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.900+0000 7fc2d2000640 1 --2- 172.21.15.27:0/2940560680 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2ac003a20 0x7fc2cc0b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.900+0000 7fc2d2a00640 1 --2- 172.21.15.27:0/2940560680 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2cc166420 0x7fc2cc1668a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.900+0000 7fc2d1600640 1 -- 172.21.15.27:0/2940560680 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2cc166420 msgr2=0x7fc2cc1668a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:01.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.900+0000 7fc2d1600640 1 --2- 172.21.15.27:0/2940560680 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2cc166420 0x7fc2cc1668a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.900+0000 7fc2d1600640 1 -- 172.21.15.27:0/2940560680 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2ac003a20 msgr2=0x7fc2cc0b3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:01.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.900+0000 7fc2d1600640 1 --2- 172.21.15.27:0/2940560680 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2ac003a20 0x7fc2cc0b3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.900+0000 7fc2d1600640 1 -- 172.21.15.27:0/2940560680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2cc0b4350 con 0x7fc2cc158270 2024-01-15T18:12:01.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.900+0000 7fc2d1600640 1 --2- 172.21.15.27:0/2940560680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc158270 0x7fc2cc157a90 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4011640 tx=0x7fc2b4011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:01.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.900+0000 7fc2ca000640 1 -- 172.21.15.27:0/2940560680 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b400f8c0 con 0x7fc2cc158270 2024-01-15T18:12:01.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.900+0000 7fc2ca000640 1 -- 172.21.15.27:0/2940560680 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b400fa20 con 0x7fc2cc158270 2024-01-15T18:12:01.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.900+0000 7fc2ca000640 1 -- 172.21.15.27:0/2940560680 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b400fd90 con 0x7fc2cc158270 2024-01-15T18:12:01.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.900+0000 7fc2d4a00640 1 -- 172.21.15.27:0/2940560680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2cc0b4630 con 0x7fc2cc158270 2024-01-15T18:12:01.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.900+0000 7fc2d4a00640 1 -- 172.21.15.27:0/2940560680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2cc157fd0 con 0x7fc2cc158270 2024-01-15T18:12:01.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.904+0000 7fc2d4a00640 1 -- 172.21.15.27:0/2940560680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2cc04e850 con 0x7fc2cc158270 2024-01-15T18:12:01.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.904+0000 7fc2ca000640 1 -- 172.21.15.27:0/2940560680 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7fc2b4015460 con 0x7fc2cc158270 2024-01-15T18:12:01.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.904+0000 7fc2ca000640 1 --2- 172.21.15.27:0/2940560680 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc2a4067100 0x7fc2a40695c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.904+0000 7fc2ca000640 1 -- 172.21.15.27:0/2940560680 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fc2b4082d70 con 0x7fc2cc158270 2024-01-15T18:12:01.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.904+0000 7fc2d2000640 1 --2- 172.21.15.27:0/2940560680 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc2a4067100 0x7fc2a40695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.904+0000 7fc2d2000640 1 --2- 172.21.15.27:0/2940560680 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc2a4067100 0x7fc2a40695c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc0aa000 tx=0x7fc2bc001a10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:01.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:01.908+0000 7fc2ca000640 1 -- 172.21.15.27:0/2940560680 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc2b4051c60 con 0x7fc2cc158270 2024-01-15T18:12:02.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:02.099+0000 7fc2d4a00640 1 -- 172.21.15.27:0/2940560680 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fc2cc0c5430 con 0x7fc2cc158270 2024-01-15T18:12:02.221 INFO:tasks.ceph.mds.c.smithi027.stderr:2024-01-15T18:12:02.219+0000 7f575a600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-15T18:12:03.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.027+0000 7fc2ca000640 1 -- 172.21.15.27:0/2940560680 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v46) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc2cc0c5430 con 0x7fc2cc158270 2024-01-15T18:12:03.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.031+0000 7fc2d4a00640 1 -- 172.21.15.27:0/2940560680 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc2a4067100 msgr2=0x7fc2a40695c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:03.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.031+0000 7fc2d4a00640 1 --2- 172.21.15.27:0/2940560680 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc2a4067100 0x7fc2a40695c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc0aa000 tx=0x7fc2bc001a10 comp rx=0 tx=0).stop 2024-01-15T18:12:03.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.031+0000 7fc2d4a00640 1 -- 172.21.15.27:0/2940560680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc158270 msgr2=0x7fc2cc157a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:03.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.031+0000 7fc2d4a00640 1 --2- 172.21.15.27:0/2940560680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc158270 0x7fc2cc157a90 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4011640 tx=0x7fc2b4011b00 comp rx=0 tx=0).stop 2024-01-15T18:12:03.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.031+0000 7fc2d4a00640 1 -- 172.21.15.27:0/2940560680 shutdown_connections 2024-01-15T18:12:03.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.031+0000 7fc2d4a00640 1 --2- 172.21.15.27:0/2940560680 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc2a4067100 0x7fc2a40695c0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:03.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.031+0000 7fc2d4a00640 1 --2- 172.21.15.27:0/2940560680 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2cc166420 0x7fc2cc1668a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:03.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.031+0000 7fc2d4a00640 1 --2- 172.21.15.27:0/2940560680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2cc158270 0x7fc2cc157a90 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:03.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.031+0000 7fc2d4a00640 1 --2- 172.21.15.27:0/2940560680 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2ac003a20 0x7fc2cc0b3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:03.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.031+0000 7fc2d4a00640 1 -- 172.21.15.27:0/2940560680 >> 172.21.15.27:0/2940560680 conn(0x7fc2cc0d0e70 msgr2=0x7fc2cc0ae4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:03.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.031+0000 7fc2d4a00640 1 -- 172.21.15.27:0/2940560680 shutdown_connections 2024-01-15T18:12:03.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.031+0000 7fc2d4a00640 1 -- 172.21.15.27:0/2940560680 wait complete. 2024-01-15T18:12:03.057 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-15T18:12:03.057 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-15T18:12:03.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.207+0000 7f55ef000640 1 Processor -- start 2024-01-15T18:12:03.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.207+0000 7f55ef000640 1 -- start start 2024-01-15T18:12:03.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.207+0000 7f55ef000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f55e8158900 0x7f55e815f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:03.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.207+0000 7f55ef000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f55e815bf00 0x7f55e80cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:03.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.207+0000 7f55ef000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f55e80cf110 0x7f55e80701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:03.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.207+0000 7f55ef000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55e80b1cb0 con 0x7f55e80cf110 2024-01-15T18:12:03.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.207+0000 7f55ef000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55e80cf510 con 0x7f55e815bf00 2024-01-15T18:12:03.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.207+0000 7f55ef000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55e80b1f00 con 0x7f55e8158900 2024-01-15T18:12:03.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.207+0000 7f55ed000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f55e80cf110 0x7f55e80701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:03.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.207+0000 7f55ed000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f55e80cf110 0x7f55e80701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58624/0 (socket says 172.21.15.27:58624) 2024-01-15T18:12:03.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.207+0000 7f55ed000640 1 -- 172.21.15.27:0/1774971962 learned_addr learned my addr 172.21.15.27:0/1774971962 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:03.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.207+0000 7f55e7e00640 1 --2- 172.21.15.27:0/1774971962 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f55e8158900 0x7f55e815f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:03.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.207+0000 7f55e7400640 1 --2- 172.21.15.27:0/1774971962 >> v2:172.21.15.93:3300/0 conn(0x7f55e815bf00 0x7f55e80cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:03.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.207+0000 7f55ed000640 1 -- 172.21.15.27:0/1774971962 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f55e8158900 msgr2=0x7f55e815f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:03.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.207+0000 7f55ed000640 1 --2- 172.21.15.27:0/1774971962 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f55e8158900 0x7f55e815f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:03.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.207+0000 7f55ed000640 1 -- 172.21.15.27:0/1774971962 >> v2:172.21.15.93:3300/0 conn(0x7f55e815bf00 msgr2=0x7f55e80cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:03.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.207+0000 7f55ed000640 1 --2- 172.21.15.27:0/1774971962 >> v2:172.21.15.93:3300/0 conn(0x7f55e815bf00 0x7f55e80cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:03.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.207+0000 7f55ed000640 1 -- 172.21.15.27:0/1774971962 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e80b21e0 con 0x7f55e80cf110 2024-01-15T18:12:03.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.211+0000 7f55ed000640 1 --2- 172.21.15.27:0/1774971962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55e80cf110 0x7f55e80701e0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f55e80cead0 tx=0x7f55e001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b3ea4e54e7f3391 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:03.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.211+0000 7f55e6a00640 1 -- 172.21.15.27:0/1774971962 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e8070f30 con 0x7f55e80cf110 2024-01-15T18:12:03.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.211+0000 7f55e6a00640 1 -- 172.21.15.27:0/1774971962 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e00a5740 con 0x7f55e80cf110 2024-01-15T18:12:03.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.211+0000 7f55e6a00640 1 -- 172.21.15.27:0/1774971962 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e00a5ad0 con 0x7f55e80cf110 2024-01-15T18:12:03.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.211+0000 7f55ef000640 1 -- 172.21.15.27:0/1774971962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55e80cf110 msgr2=0x7f55e80701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:03.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.211+0000 7f55ef000640 1 --2- 172.21.15.27:0/1774971962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55e80cf110 0x7f55e80701e0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f55e80cead0 tx=0x7f55e001c9d0 comp rx=0 tx=0).stop 2024-01-15T18:12:03.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.211+0000 7f55ef000640 1 -- 172.21.15.27:0/1774971962 shutdown_connections 2024-01-15T18:12:03.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.211+0000 7f55ef000640 1 --2- 172.21.15.27:0/1774971962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55e80cf110 0x7f55e80701e0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:03.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.211+0000 7f55ef000640 1 --2- 172.21.15.27:0/1774971962 >> v2:172.21.15.93:3300/0 conn(0x7f55e815bf00 0x7f55e80cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:03.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.211+0000 7f55ef000640 1 --2- 172.21.15.27:0/1774971962 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f55e8158900 0x7f55e815f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:03.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.211+0000 7f55ef000640 1 -- 172.21.15.27:0/1774971962 >> 172.21.15.27:0/1774971962 conn(0x7f55e80cb700 msgr2=0x7f55e80cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:03.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.211+0000 7f55ef000640 1 -- 172.21.15.27:0/1774971962 shutdown_connections 2024-01-15T18:12:03.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.211+0000 7f55ef000640 1 -- 172.21.15.27:0/1774971962 wait complete. 2024-01-15T18:12:03.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.215+0000 7f55ef000640 1 Processor -- start 2024-01-15T18:12:03.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.215+0000 7f55ef000640 1 -- start start 2024-01-15T18:12:03.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.215+0000 7f55ef000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f55e80cf110 0x7f55e814d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:03.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.215+0000 7f55ef000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55e8158900 0x7f55e8149220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:03.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.215+0000 7f55ef000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55e815bf00 0x7f55e80af6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:03.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.215+0000 7f55ef000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e8149da0 con 0x7f55e815bf00 2024-01-15T18:12:03.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.215+0000 7f55ef000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e80cdb50 con 0x7f55e8158900 2024-01-15T18:12:03.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.215+0000 7f55ef000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55e80cde10 con 0x7f55e80cf110 2024-01-15T18:12:03.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.215+0000 7f55ed000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55e815bf00 0x7f55e80af6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:03.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.215+0000 7f55ed000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55e815bf00 0x7f55e80af6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58636/0 (socket says 172.21.15.27:58636) 2024-01-15T18:12:03.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.215+0000 7f55ed000640 1 -- 172.21.15.27:0/2320347695 learned_addr learned my addr 172.21.15.27:0/2320347695 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:03.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.215+0000 7f55e7400640 1 --2- 172.21.15.27:0/2320347695 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55e8158900 0x7f55e8149220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:03.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.215+0000 7f55e7e00640 1 --2- 172.21.15.27:0/2320347695 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f55e80cf110 0x7f55e814d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:03.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.215+0000 7f55ed000640 1 -- 172.21.15.27:0/2320347695 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f55e80cf110 msgr2=0x7f55e814d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:03.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.215+0000 7f55ed000640 1 --2- 172.21.15.27:0/2320347695 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f55e80cf110 0x7f55e814d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:03.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.215+0000 7f55ed000640 1 -- 172.21.15.27:0/2320347695 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55e8158900 msgr2=0x7f55e8149220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:03.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.215+0000 7f55ed000640 1 --2- 172.21.15.27:0/2320347695 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55e8158900 0x7f55e8149220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:03.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.215+0000 7f55ed000640 1 -- 172.21.15.27:0/2320347695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e0009b30 con 0x7f55e815bf00 2024-01-15T18:12:03.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.215+0000 7f55ed000640 1 --2- 172.21.15.27:0/2320347695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55e815bf00 0x7f55e80af6f0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f55e00af040 tx=0x7f55e0019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:03.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.215+0000 7f55dfe00640 1 -- 172.21.15.27:0/2320347695 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e00b8670 con 0x7f55e815bf00 2024-01-15T18:12:03.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.215+0000 7f55dfe00640 1 -- 172.21.15.27:0/2320347695 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e00b87f0 con 0x7f55e815bf00 2024-01-15T18:12:03.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.215+0000 7f55ef000640 1 -- 172.21.15.27:0/2320347695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55e80ce0f0 con 0x7f55e815bf00 2024-01-15T18:12:03.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.215+0000 7f55dfe00640 1 -- 172.21.15.27:0/2320347695 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e00b8ba0 con 0x7f55e815bf00 2024-01-15T18:12:03.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.215+0000 7f55ef000640 1 -- 172.21.15.27:0/2320347695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55e80ce6b0 con 0x7f55e815bf00 2024-01-15T18:12:03.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.219+0000 7f55ef000640 1 -- 172.21.15.27:0/2320347695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55e80cdf50 con 0x7f55e815bf00 2024-01-15T18:12:03.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.219+0000 7f55dfe00640 1 -- 172.21.15.27:0/2320347695 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f55e00b8d00 con 0x7f55e815bf00 2024-01-15T18:12:03.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.219+0000 7f55dfe00640 1 --2- 172.21.15.27:0/2320347695 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f55c00670d0 0x7f55c0069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:03.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.219+0000 7f55dfe00640 1 -- 172.21.15.27:0/2320347695 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f55e011ddf0 con 0x7f55e815bf00 2024-01-15T18:12:03.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.219+0000 7f55e7e00640 1 --2- 172.21.15.27:0/2320347695 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f55c00670d0 0x7f55c0069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:03.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.219+0000 7f55e7e00640 1 --2- 172.21.15.27:0/2320347695 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f55c00670d0 0x7f55c0069590 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f55d8009b60 tx=0x7f55d8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:03.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.223+0000 7f55dfe00640 1 -- 172.21.15.27:0/2320347695 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f55e80cdf50 con 0x7f55e815bf00 2024-01-15T18:12:03.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:03.415+0000 7f55ef000640 1 -- 172.21.15.27:0/2320347695 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f55e80cdf50 con 0x7f55e815bf00 2024-01-15T18:12:04.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.031+0000 7f55dfe00640 1 -- 172.21.15.27:0/2320347695 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v47) v1 ==== 122+0+0 (secure 0 0 0) 0x7f55e80cdf50 con 0x7f55e815bf00 2024-01-15T18:12:04.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.035+0000 7f55ef000640 1 -- 172.21.15.27:0/2320347695 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f55c00670d0 msgr2=0x7f55c0069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:04.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.035+0000 7f55ef000640 1 --2- 172.21.15.27:0/2320347695 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f55c00670d0 0x7f55c0069590 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f55d8009b60 tx=0x7f55d8009190 comp rx=0 tx=0).stop 2024-01-15T18:12:04.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.035+0000 7f55ef000640 1 -- 172.21.15.27:0/2320347695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55e815bf00 msgr2=0x7f55e80af6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:04.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.035+0000 7f55ef000640 1 --2- 172.21.15.27:0/2320347695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55e815bf00 0x7f55e80af6f0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f55e00af040 tx=0x7f55e0019770 comp rx=0 tx=0).stop 2024-01-15T18:12:04.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.035+0000 7f55ef000640 1 -- 172.21.15.27:0/2320347695 shutdown_connections 2024-01-15T18:12:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.035+0000 7f55ef000640 1 --2- 172.21.15.27:0/2320347695 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f55c00670d0 0x7f55c0069590 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.035+0000 7f55ef000640 1 --2- 172.21.15.27:0/2320347695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f55e815bf00 0x7f55e80af6f0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.035+0000 7f55ef000640 1 --2- 172.21.15.27:0/2320347695 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f55e8158900 0x7f55e8149220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.035+0000 7f55ef000640 1 --2- 172.21.15.27:0/2320347695 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f55e80cf110 0x7f55e814d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.035+0000 7f55ef000640 1 -- 172.21.15.27:0/2320347695 >> 172.21.15.27:0/2320347695 conn(0x7f55e80cb700 msgr2=0x7f55e806c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.035+0000 7f55ef000640 1 -- 172.21.15.27:0/2320347695 shutdown_connections 2024-01-15T18:12:04.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.035+0000 7f55ef000640 1 -- 172.21.15.27:0/2320347695 wait complete. 2024-01-15T18:12:04.061 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-15T18:12:04.061 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:12:04.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.207+0000 7f7e5e600640 1 Processor -- start 2024-01-15T18:12:04.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.207+0000 7f7e5e600640 1 -- start start 2024-01-15T18:12:04.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.207+0000 7f7e5e600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f7e58071e40 0x7f7e580c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:04.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.207+0000 7f7e5e600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e58159a50 con 0x7f7e5806d430 2024-01-15T18:12:04.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.207+0000 7f7e5e600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e5806d980 con 0x7f7e58159bf0 2024-01-15T18:12:04.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.207+0000 7f7e5e600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e580cce00 con 0x7f7e58071e40 2024-01-15T18:12:04.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.207+0000 7f7e57e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7e5806d430 0x7f7e580cd250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59390/0 (socket says 172.21.15.27:59390) 2024-01-15T18:12:04.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.207+0000 7f7e57e00640 1 -- 172.21.15.27:0/1214457853 learned_addr learned my addr 172.21.15.27:0/1214457853 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:12:04.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.207+0000 7f7e56a00640 1 --2- 172.21.15.27:0/1214457853 >> v2:172.21.15.93:3300/0 conn(0x7f7e58071e40 0x7f7e580c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:04.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.207+0000 7f7e56000640 1 -- 172.21.15.27:0/1214457853 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046335793 0 0) 0x7f7e500014a0 con 0x7f7e5806d430 2024-01-15T18:12:04.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.207+0000 7f7e56000640 1 -- 172.21.15.27:0/1214457853 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e38003530 con 0x7f7e5806d430 2024-01-15T18:12:04.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.207+0000 7f7e56000640 1 -- 172.21.15.27:0/1214457853 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979613066 0 0) 0x7f7e5806d980 con 0x7f7e58159bf0 2024-01-15T18:12:04.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.207+0000 7f7e56000640 1 -- 172.21.15.27:0/1214457853 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e500014a0 con 0x7f7e58159bf0 2024-01-15T18:12:04.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.207+0000 7f7e56a00640 1 -- 172.21.15.27:0/1214457853 >> v1:172.21.15.93:6789/0 conn(0x7f7e58159bf0 legacy=0x7f7e58159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:04.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.207+0000 7f7e56a00640 1 -- 172.21.15.27:0/1214457853 >> v1:172.21.15.27:6789/0 conn(0x7f7e5806d430 legacy=0x7f7e580cd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:04.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.207+0000 7f7e56a00640 1 -- 172.21.15.27:0/1214457853 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e5806dc60 con 0x7f7e58071e40 2024-01-15T18:12:04.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.211+0000 7f7e56a00640 1 --2- 172.21.15.27:0/1214457853 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e58071e40 0x7f7e580c9be0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7e4000a960 tx=0x7f7e40020040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25752997c8273603 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:04.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.211+0000 7f7e56000640 1 -- 172.21.15.27:0/1214457853 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e400a9e50 con 0x7f7e58071e40 2024-01-15T18:12:04.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.211+0000 7f7e56000640 1 -- 172.21.15.27:0/1214457853 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e400b1440 con 0x7f7e58071e40 2024-01-15T18:12:04.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.211+0000 7f7e56000640 1 -- 172.21.15.27:0/1214457853 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e400b17b0 con 0x7f7e58071e40 2024-01-15T18:12:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.211+0000 7f7e5e600640 1 -- 172.21.15.27:0/1214457853 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e58071e40 msgr2=0x7f7e580c9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.211+0000 7f7e5e600640 1 --2- 172.21.15.27:0/1214457853 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e58071e40 0x7f7e580c9be0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7e4000a960 tx=0x7f7e40020040 comp rx=0 tx=0).stop 2024-01-15T18:12:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.211+0000 7f7e5e600640 1 -- 172.21.15.27:0/1214457853 shutdown_connections 2024-01-15T18:12:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.211+0000 7f7e5e600640 1 --2- 172.21.15.27:0/1214457853 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e58071e40 0x7f7e580c9be0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.211+0000 7f7e5e600640 1 -- 172.21.15.27:0/1214457853 >> 172.21.15.27:0/1214457853 conn(0x7f7e580cec60 msgr2=0x7f7e580cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:04.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.211+0000 7f7e5e600640 1 -- 172.21.15.27:0/1214457853 shutdown_connections 2024-01-15T18:12:04.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.211+0000 7f7e5e600640 1 -- 172.21.15.27:0/1214457853 wait complete. 2024-01-15T18:12:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.215+0000 7f7e5e600640 1 Processor -- start 2024-01-15T18:12:04.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.215+0000 7f7e5e600640 1 -- start start 2024-01-15T18:12:04.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.215+0000 7f7e5e600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e5806d430 0x7f7e581512a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:04.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.215+0000 7f7e5e600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7e58159bf0 0x7f7e58153920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:04.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.215+0000 7f7e5e600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e581f0f10 0x7f7e58157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:04.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.215+0000 7f7e5e600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e58153490 con 0x7f7e5806d430 2024-01-15T18:12:04.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.215+0000 7f7e5e600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e581517e0 con 0x7f7e581f0f10 2024-01-15T18:12:04.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.215+0000 7f7e5e600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e58153140 con 0x7f7e58159bf0 2024-01-15T18:12:04.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.215+0000 7f7e57400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e5806d430 0x7f7e581512a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:04.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.215+0000 7f7e57400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e5806d430 0x7f7e581512a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58236/0 (socket says 172.21.15.27:58236) 2024-01-15T18:12:04.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.215+0000 7f7e57400640 1 -- 172.21.15.27:0/1187017247 learned_addr learned my addr 172.21.15.27:0/1187017247 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:04.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.215+0000 7f7e57e00640 1 --2- 172.21.15.27:0/1187017247 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e581f0f10 0x7f7e58157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:04.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.215+0000 7f7e56a00640 1 --2- 172.21.15.27:0/1187017247 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7e58159bf0 0x7f7e58153920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:04.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.215+0000 7f7e57400640 1 -- 172.21.15.27:0/1187017247 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7e58159bf0 msgr2=0x7f7e58153920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:04.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.215+0000 7f7e57400640 1 --2- 172.21.15.27:0/1187017247 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7e58159bf0 0x7f7e58153920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.215+0000 7f7e57400640 1 -- 172.21.15.27:0/1187017247 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e581f0f10 msgr2=0x7f7e58157580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:04.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.215+0000 7f7e57400640 1 --2- 172.21.15.27:0/1187017247 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e581f0f10 0x7f7e58157580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.215+0000 7f7e57400640 1 -- 172.21.15.27:0/1187017247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e4000a5c0 con 0x7f7e5806d430 2024-01-15T18:12:04.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.215+0000 7f7e57400640 1 --2- 172.21.15.27:0/1187017247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e5806d430 0x7f7e581512a0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f7e4800c430 tx=0x7f7e4800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:04.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.215+0000 7f7e4f400640 1 -- 172.21.15.27:0/1187017247 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e480052d0 con 0x7f7e5806d430 2024-01-15T18:12:04.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.215+0000 7f7e4f400640 1 -- 172.21.15.27:0/1187017247 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e48005450 con 0x7f7e5806d430 2024-01-15T18:12:04.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.215+0000 7f7e4f400640 1 -- 172.21.15.27:0/1187017247 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e48003840 con 0x7f7e5806d430 2024-01-15T18:12:04.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.215+0000 7f7e5e600640 1 -- 172.21.15.27:0/1187017247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e58157f60 con 0x7f7e5806d430 2024-01-15T18:12:04.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.215+0000 7f7e5e600640 1 -- 172.21.15.27:0/1187017247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e58153e60 con 0x7f7e5806d430 2024-01-15T18:12:04.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.219+0000 7f7e5e600640 1 -- 172.21.15.27:0/1187017247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e58154050 con 0x7f7e5806d430 2024-01-15T18:12:04.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.219+0000 7f7e4f400640 1 -- 172.21.15.27:0/1187017247 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f7e480039a0 con 0x7f7e5806d430 2024-01-15T18:12:04.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.219+0000 7f7e4f400640 1 --2- 172.21.15.27:0/1187017247 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7e300670d0 0x7f7e30069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:04.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.219+0000 7f7e4f400640 1 -- 172.21.15.27:0/1187017247 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f7e48082040 con 0x7f7e5806d430 2024-01-15T18:12:04.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.219+0000 7f7e56a00640 1 --2- 172.21.15.27:0/1187017247 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7e300670d0 0x7f7e30069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:04.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.219+0000 7f7e56a00640 1 --2- 172.21.15.27:0/1187017247 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7e300670d0 0x7f7e30069590 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f7e400b3000 tx=0x7f7e400ab070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:04.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.223+0000 7f7e4f400640 1 -- 172.21.15.27:0/1187017247 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7e58154050 con 0x7f7e5806d430 2024-01-15T18:12:04.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.415+0000 7f7e5e600640 1 -- 172.21.15.27:0/1187017247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e58154050 con 0x7f7e5806d430 2024-01-15T18:12:04.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.415+0000 7f7e4f400640 1 -- 172.21.15.27:0/1187017247 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+4909 (secure 0 0 0) 0x7f7e58154050 con 0x7f7e5806d430 2024-01-15T18:12:04.417 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:12:04.417 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":47,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6889,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6835/914017554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":914017554},{"type":"v1","addr":"172.21.15.93:6835","nonce":914017554}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":6892,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/4099412077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6836","nonce":4099412077},{"type":"v1","addr":"172.21.15.93:6837","nonce":4099412077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":7215,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/902229728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":902229728},{"type":"v1","addr":"172.21.15.27:6833","nonce":902229728}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:12:00.405416+0000","modified":"2024-01-15T18:12:03.417514+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7227},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7227":{"gid":7227,"name":"c","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/2027824196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2027824196},{"type":"v1","addr":"172.21.15.27:6835","nonce":2027824196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-01-15T18:12:04.418 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 47 2024-01-15T18:12:04.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.415+0000 7f7e5e600640 1 -- 172.21.15.27:0/1187017247 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7e300670d0 msgr2=0x7f7e30069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:04.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.415+0000 7f7e5e600640 1 --2- 172.21.15.27:0/1187017247 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7e300670d0 0x7f7e30069590 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f7e400b3000 tx=0x7f7e400ab070 comp rx=0 tx=0).stop 2024-01-15T18:12:04.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.415+0000 7f7e5e600640 1 -- 172.21.15.27:0/1187017247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e5806d430 msgr2=0x7f7e581512a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:04.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.415+0000 7f7e5e600640 1 --2- 172.21.15.27:0/1187017247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e5806d430 0x7f7e581512a0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f7e4800c430 tx=0x7f7e4800c8f0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.419+0000 7f7e5e600640 1 -- 172.21.15.27:0/1187017247 shutdown_connections 2024-01-15T18:12:04.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.419+0000 7f7e5e600640 1 --2- 172.21.15.27:0/1187017247 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7e300670d0 0x7f7e30069590 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.419+0000 7f7e5e600640 1 --2- 172.21.15.27:0/1187017247 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e581f0f10 0x7f7e58157580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.419+0000 7f7e5e600640 1 --2- 172.21.15.27:0/1187017247 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7e58159bf0 0x7f7e58153920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.419+0000 7f7e5e600640 1 --2- 172.21.15.27:0/1187017247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e5806d430 0x7f7e581512a0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.419+0000 7f7e5e600640 1 -- 172.21.15.27:0/1187017247 >> 172.21.15.27:0/1187017247 conn(0x7f7e580cec60 msgr2=0x7f7e58159100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:04.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.419+0000 7f7e5e600640 1 -- 172.21.15.27:0/1187017247 shutdown_connections 2024-01-15T18:12:04.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.419+0000 7f7e5e600640 1 -- 172.21.15.27:0/1187017247 wait complete. 2024-01-15T18:12:04.439 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:12:04.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.579+0000 7f4647800640 1 Processor -- start 2024-01-15T18:12:04.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.579+0000 7f4647800640 1 -- start start 2024-01-15T18:12:04.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.579+0000 7f4647800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f464015c130 0x7f464006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:04.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.579+0000 7f4647800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f464006e240 0x7f464015d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:04.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.579+0000 7f4647800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f464006c4a0 con 0x7f46400701a0 2024-01-15T18:12:04.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.579+0000 7f4647800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f464015c660 con 0x7f464015c130 2024-01-15T18:12:04.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.579+0000 7f4647800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464015c940 con 0x7f464006e240 2024-01-15T18:12:04.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.579+0000 7f463fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f464015c130 0x7f464006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:04.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.579+0000 7f463fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f464015c130 0x7f464006dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58252/0 (socket says 172.21.15.27:58252) 2024-01-15T18:12:04.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.579+0000 7f463fe00640 1 -- 172.21.15.27:0/2869282044 learned_addr learned my addr 172.21.15.27:0/2869282044 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:04.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.579+0000 7f4644e00640 1 --1- 172.21.15.27:0/2869282044 >> v1:172.21.15.27:6789/0 conn(0x7f46400701a0 0x7f4640070580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59404/0 (socket says 172.21.15.27:59404) 2024-01-15T18:12:04.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.579+0000 7f4645800640 1 --2- 172.21.15.27:0/2869282044 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f464006e240 0x7f464015d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:04.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.579+0000 7f463fe00640 1 -- 172.21.15.27:0/2869282044 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f464006e240 msgr2=0x7f464015d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:04.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.579+0000 7f463fe00640 1 --2- 172.21.15.27:0/2869282044 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f464006e240 0x7f464015d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.579+0000 7f463fe00640 1 -- 172.21.15.27:0/2869282044 >> v1:172.21.15.27:6789/0 conn(0x7f46400701a0 legacy=0x7f4640070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:04.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.579+0000 7f463fe00640 1 -- 172.21.15.27:0/2869282044 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464015cc20 con 0x7f464015c130 2024-01-15T18:12:04.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.579+0000 7f463f400640 1 -- 172.21.15.27:0/2869282044 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673857113 0 0) 0x7f464006c4a0 con 0x7f46400701a0 2024-01-15T18:12:04.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.583+0000 7f463fe00640 1 --2- 172.21.15.27:0/2869282044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f464015c130 0x7f464006dd00 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f4628009870 tx=0x7f462801c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b15ffb1e6341f573 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:04.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.583+0000 7f463f400640 1 -- 172.21.15.27:0/2869282044 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46280aa670 con 0x7f464015c130 2024-01-15T18:12:04.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.583+0000 7f463f400640 1 -- 172.21.15.27:0/2869282044 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46280aa7d0 con 0x7f464015c130 2024-01-15T18:12:04.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.583+0000 7f463f400640 1 -- 172.21.15.27:0/2869282044 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46280aab60 con 0x7f464015c130 2024-01-15T18:12:04.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.583+0000 7f4647800640 1 -- 172.21.15.27:0/2869282044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f464015c130 msgr2=0x7f464006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:04.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.583+0000 7f4647800640 1 --2- 172.21.15.27:0/2869282044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f464015c130 0x7f464006dd00 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f4628009870 tx=0x7f462801c510 comp rx=0 tx=0).stop 2024-01-15T18:12:04.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.583+0000 7f4647800640 1 -- 172.21.15.27:0/2869282044 shutdown_connections 2024-01-15T18:12:04.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.583+0000 7f4647800640 1 --2- 172.21.15.27:0/2869282044 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f464006e240 0x7f464015d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.583+0000 7f4647800640 1 --2- 172.21.15.27:0/2869282044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f464015c130 0x7f464006dd00 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.583+0000 7f4647800640 1 -- 172.21.15.27:0/2869282044 >> 172.21.15.27:0/2869282044 conn(0x7f46400cec60 msgr2=0x7f46400cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:04.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.583+0000 7f4647800640 1 -- 172.21.15.27:0/2869282044 shutdown_connections 2024-01-15T18:12:04.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.583+0000 7f4647800640 1 -- 172.21.15.27:0/2869282044 wait complete. 2024-01-15T18:12:04.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.587+0000 7f4647800640 1 Processor -- start 2024-01-15T18:12:04.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.587+0000 7f4647800640 1 -- start start 2024-01-15T18:12:04.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.587+0000 7f4647800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f464006e240 0x7f4640159750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:04.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.587+0000 7f4647800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f46400701a0 0x7f464013ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:04.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.587+0000 7f4647800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f464015c130 0x7f46401491d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:04.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.587+0000 7f4647800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4640159330 con 0x7f464006e240 2024-01-15T18:12:04.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.587+0000 7f4647800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464014aaf0 con 0x7f46400701a0 2024-01-15T18:12:04.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.587+0000 7f4647800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46401586b0 con 0x7f464015c130 2024-01-15T18:12:04.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.587+0000 7f4644e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f464006e240 0x7f4640159750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:04.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.587+0000 7f4644e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f464006e240 0x7f4640159750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58258/0 (socket says 172.21.15.27:58258) 2024-01-15T18:12:04.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.587+0000 7f4644e00640 1 -- 172.21.15.27:0/2797359914 learned_addr learned my addr 172.21.15.27:0/2797359914 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:04.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.587+0000 7f4645800640 1 --2- 172.21.15.27:0/2797359914 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f464015c130 0x7f46401491d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:04.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.587+0000 7f463fe00640 1 --2- 172.21.15.27:0/2797359914 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f46400701a0 0x7f464013ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:04.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.587+0000 7f4644e00640 1 -- 172.21.15.27:0/2797359914 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f464015c130 msgr2=0x7f46401491d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:04.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.587+0000 7f4644e00640 1 --2- 172.21.15.27:0/2797359914 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f464015c130 0x7f46401491d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.587+0000 7f4644e00640 1 -- 172.21.15.27:0/2797359914 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f46400701a0 msgr2=0x7f464013ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:04.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.587+0000 7f4644e00640 1 --2- 172.21.15.27:0/2797359914 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f46400701a0 0x7f464013ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.587+0000 7f4644e00640 1 -- 172.21.15.27:0/2797359914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46280094d0 con 0x7f464006e240 2024-01-15T18:12:04.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.587+0000 7f4644e00640 1 --2- 172.21.15.27:0/2797359914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f464006e240 0x7f4640159750 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f46300011f0 tx=0x7f463000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:04.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.587+0000 7f463cc00640 1 -- 172.21.15.27:0/2797359914 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46300074e0 con 0x7f464006e240 2024-01-15T18:12:04.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.587+0000 7f463cc00640 1 -- 172.21.15.27:0/2797359914 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4630007640 con 0x7f464006e240 2024-01-15T18:12:04.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.587+0000 7f463cc00640 1 -- 172.21.15.27:0/2797359914 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4630004a10 con 0x7f464006e240 2024-01-15T18:12:04.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.587+0000 7f4647800640 1 -- 172.21.15.27:0/2797359914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4640158990 con 0x7f464006e240 2024-01-15T18:12:04.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.587+0000 7f4647800640 1 -- 172.21.15.27:0/2797359914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4640158f50 con 0x7f464006e240 2024-01-15T18:12:04.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.591+0000 7f4647800640 1 -- 172.21.15.27:0/2797359914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4640159470 con 0x7f464006e240 2024-01-15T18:12:04.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.591+0000 7f463cc00640 1 -- 172.21.15.27:0/2797359914 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f4630004b70 con 0x7f464006e240 2024-01-15T18:12:04.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.591+0000 7f463cc00640 1 --2- 172.21.15.27:0/2797359914 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f46180670d0 0x7f4618069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:04.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.591+0000 7f463cc00640 1 -- 172.21.15.27:0/2797359914 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f4630081d50 con 0x7f464006e240 2024-01-15T18:12:04.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.591+0000 7f463fe00640 1 --2- 172.21.15.27:0/2797359914 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f46180670d0 0x7f4618069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:04.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.595+0000 7f463fe00640 1 --2- 172.21.15.27:0/2797359914 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f46180670d0 0x7f4618069590 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f46280ae000 tx=0x7f46280a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:04.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.595+0000 7f463cc00640 1 -- 172.21.15.27:0/2797359914 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4640159470 con 0x7f464006e240 2024-01-15T18:12:04.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.755+0000 7f4647800640 1 -- 172.21.15.27:0/2797359914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4640159470 con 0x7f464006e240 2024-01-15T18:12:04.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.755+0000 7f463cc00640 1 -- 172.21.15.27:0/2797359914 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16559 (secure 0 0 0) 0x7f4640159470 con 0x7f464006e240 2024-01-15T18:12:04.757 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:12:04.757 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":70,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:12:02.000420+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:11:58.044871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-15T18:11:59.394705+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:12:04.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.755+0000 7f4647800640 1 -- 172.21.15.27:0/2797359914 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f46180670d0 msgr2=0x7f4618069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:04.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.755+0000 7f4647800640 1 --2- 172.21.15.27:0/2797359914 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f46180670d0 0x7f4618069590 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f46280ae000 tx=0x7f46280a7000 comp rx=0 tx=0).stop 2024-01-15T18:12:04.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.755+0000 7f4647800640 1 -- 172.21.15.27:0/2797359914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f464006e240 msgr2=0x7f4640159750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:04.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.755+0000 7f4647800640 1 --2- 172.21.15.27:0/2797359914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f464006e240 0x7f4640159750 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f46300011f0 tx=0x7f463000c9b0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.755+0000 7f4647800640 1 -- 172.21.15.27:0/2797359914 shutdown_connections 2024-01-15T18:12:04.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.755+0000 7f4647800640 1 --2- 172.21.15.27:0/2797359914 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f46180670d0 0x7f4618069590 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.755+0000 7f4647800640 1 --2- 172.21.15.27:0/2797359914 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f464015c130 0x7f46401491d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.755+0000 7f4647800640 1 --2- 172.21.15.27:0/2797359914 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f46400701a0 0x7f464013ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.755+0000 7f4647800640 1 --2- 172.21.15.27:0/2797359914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f464006e240 0x7f4640159750 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.755+0000 7f4647800640 1 -- 172.21.15.27:0/2797359914 >> 172.21.15.27:0/2797359914 conn(0x7f46400cec60 msgr2=0x7f464015b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:04.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.755+0000 7f4647800640 1 -- 172.21.15.27:0/2797359914 shutdown_connections 2024-01-15T18:12:04.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.755+0000 7f4647800640 1 -- 172.21.15.27:0/2797359914 wait complete. 2024-01-15T18:12:04.780 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-15T18:12:04.780 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:12:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.919+0000 7f239e600640 1 Processor -- start 2024-01-15T18:12:04.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.919+0000 7f239e600640 1 -- start start 2024-01-15T18:12:04.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.919+0000 7f239e600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f23980d1ab0 0x7f23980cf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:04.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.919+0000 7f239e600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f23980c5030 0x7f23980c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:04.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.919+0000 7f239e600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23980c4ba0 con 0x7f239815e830 2024-01-15T18:12:04.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.919+0000 7f239e600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23980cc5a0 con 0x7f23980d1ab0 2024-01-15T18:12:04.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.919+0000 7f239e600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23980cc840 con 0x7f23980c5030 2024-01-15T18:12:04.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.919+0000 7f2396a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f23980d1ab0 0x7f23980cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:04.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.919+0000 7f2397400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f239815e830 0x7f23980ceaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59418/0 (socket says 172.21.15.27:59418) 2024-01-15T18:12:04.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.919+0000 7f2397400640 1 -- 172.21.15.27:0/2617711811 learned_addr learned my addr 172.21.15.27:0/2617711811 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:12:04.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.919+0000 7f2397e00640 1 --2- 172.21.15.27:0/2617711811 >> v2:172.21.15.93:3300/0 conn(0x7f23980c5030 0x7f23980c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:04.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.923+0000 7f2396a00640 1 -- 172.21.15.27:0/2617711811 >> v2:172.21.15.93:3300/0 conn(0x7f23980c5030 msgr2=0x7f23980c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:04.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.923+0000 7f2396a00640 1 --2- 172.21.15.27:0/2617711811 >> v2:172.21.15.93:3300/0 conn(0x7f23980c5030 0x7f23980c54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.923+0000 7f2396a00640 1 -- 172.21.15.27:0/2617711811 >> v1:172.21.15.27:6789/0 conn(0x7f239815e830 legacy=0x7f23980ceaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:04.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.923+0000 7f2396000640 1 -- 172.21.15.27:0/2617711811 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 164907329 0 0) 0x7f23980c4ba0 con 0x7f239815e830 2024-01-15T18:12:04.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.923+0000 7f2396a00640 1 -- 172.21.15.27:0/2617711811 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23980ccb20 con 0x7f23980d1ab0 2024-01-15T18:12:04.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.923+0000 7f2396a00640 1 --2- 172.21.15.27:0/2617711811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23980d1ab0 0x7f23980cf200 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f23800097c0 tx=0x7f238001c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=214aeac175c9443c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:04.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.923+0000 7f2396000640 1 -- 172.21.15.27:0/2617711811 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23800ab600 con 0x7f23980d1ab0 2024-01-15T18:12:04.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.923+0000 7f2396000640 1 -- 172.21.15.27:0/2617711811 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23800a7050 con 0x7f23980d1ab0 2024-01-15T18:12:04.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.923+0000 7f2396000640 1 -- 172.21.15.27:0/2617711811 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23800ab980 con 0x7f23980d1ab0 2024-01-15T18:12:04.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.923+0000 7f239e600640 1 -- 172.21.15.27:0/2617711811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23980d1ab0 msgr2=0x7f23980cf200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:04.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.923+0000 7f239e600640 1 --2- 172.21.15.27:0/2617711811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23980d1ab0 0x7f23980cf200 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f23800097c0 tx=0x7f238001c3b0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.927+0000 7f239e600640 1 -- 172.21.15.27:0/2617711811 shutdown_connections 2024-01-15T18:12:04.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.927+0000 7f239e600640 1 --2- 172.21.15.27:0/2617711811 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23980d1ab0 0x7f23980cf200 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.927+0000 7f239e600640 1 --2- 172.21.15.27:0/2617711811 >> v2:172.21.15.93:3300/0 conn(0x7f23980c5030 0x7f23980c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.927+0000 7f239e600640 1 -- 172.21.15.27:0/2617711811 >> 172.21.15.27:0/2617711811 conn(0x7f23980d0f20 msgr2=0x7f23980d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:04.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.927+0000 7f239e600640 1 -- 172.21.15.27:0/2617711811 shutdown_connections 2024-01-15T18:12:04.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.927+0000 7f239e600640 1 -- 172.21.15.27:0/2617711811 wait complete. 2024-01-15T18:12:04.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.927+0000 7f239e600640 1 Processor -- start 2024-01-15T18:12:04.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.927+0000 7f239e600640 1 -- start start 2024-01-15T18:12:04.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.927+0000 7f239e600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23980c5030 0x7f23981f9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:04.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.927+0000 7f239e600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f23980d1ab0 0x7f23980ca150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:04.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.927+0000 7f239e600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f239815e830 0x7f239815c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:04.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.927+0000 7f239e600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23980ca690 con 0x7f23980c5030 2024-01-15T18:12:04.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.927+0000 7f239e600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2398146df0 con 0x7f239815e830 2024-01-15T18:12:04.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.927+0000 7f239e600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2398157bd0 con 0x7f23980d1ab0 2024-01-15T18:12:04.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.927+0000 7f2397400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23980c5030 0x7f23981f9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:04.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.927+0000 7f2397400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23980c5030 0x7f23981f9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58278/0 (socket says 172.21.15.27:58278) 2024-01-15T18:12:04.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.927+0000 7f2397400640 1 -- 172.21.15.27:0/734391389 learned_addr learned my addr 172.21.15.27:0/734391389 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:04.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.927+0000 7f2397e00640 1 --2- 172.21.15.27:0/734391389 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f239815e830 0x7f239815c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:04.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.927+0000 7f2396a00640 1 --2- 172.21.15.27:0/734391389 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f23980d1ab0 0x7f23980ca150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:04.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.927+0000 7f2397400640 1 -- 172.21.15.27:0/734391389 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f23980d1ab0 msgr2=0x7f23980ca150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:04.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.927+0000 7f2397400640 1 --2- 172.21.15.27:0/734391389 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f23980d1ab0 0x7f23980ca150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.927+0000 7f2397400640 1 -- 172.21.15.27:0/734391389 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f239815e830 msgr2=0x7f239815c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:04.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.927+0000 7f2397400640 1 --2- 172.21.15.27:0/734391389 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f239815e830 0x7f239815c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.927+0000 7f2397400640 1 -- 172.21.15.27:0/734391389 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2380009420 con 0x7f23980c5030 2024-01-15T18:12:04.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.931+0000 7f2397400640 1 --2- 172.21.15.27:0/734391389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23980c5030 0x7f23981f9a60 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f238800c4b0 tx=0x7f238800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:04.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.931+0000 7f238f400640 1 -- 172.21.15.27:0/734391389 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23880074e0 con 0x7f23980c5030 2024-01-15T18:12:04.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.931+0000 7f238f400640 1 -- 172.21.15.27:0/734391389 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2388007660 con 0x7f23980c5030 2024-01-15T18:12:04.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.931+0000 7f239e600640 1 -- 172.21.15.27:0/734391389 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23980ca890 con 0x7f23980c5030 2024-01-15T18:12:04.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.931+0000 7f238f400640 1 -- 172.21.15.27:0/734391389 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2388004a30 con 0x7f23980c5030 2024-01-15T18:12:04.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.931+0000 7f239e600640 1 -- 172.21.15.27:0/734391389 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2398157e20 con 0x7f23980c5030 2024-01-15T18:12:04.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.931+0000 7f239e600640 1 -- 172.21.15.27:0/734391389 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f239804e850 con 0x7f23980c5030 2024-01-15T18:12:04.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.931+0000 7f238f400640 1 -- 172.21.15.27:0/734391389 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f2388004b90 con 0x7f23980c5030 2024-01-15T18:12:04.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.931+0000 7f238f400640 1 --2- 172.21.15.27:0/734391389 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f23700670d0 0x7f2370069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:04.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.931+0000 7f238f400640 1 -- 172.21.15.27:0/734391389 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f2388081c10 con 0x7f23980c5030 2024-01-15T18:12:04.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.935+0000 7f2396a00640 1 --2- 172.21.15.27:0/734391389 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f23700670d0 0x7f2370069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:04.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.935+0000 7f2396a00640 1 --2- 172.21.15.27:0/734391389 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f23700670d0 0x7f2370069590 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f23800ad000 tx=0x7f23800a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:04.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:04.935+0000 7f238f400640 1 -- 172.21.15.27:0/734391389 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f23880544e0 con 0x7f23980c5030 2024-01-15T18:12:05.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.095+0000 7f239e600640 1 -- 172.21.15.27:0/734391389 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f23980cf950 con 0x7f23700670d0 2024-01-15T18:12:05.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.103+0000 7f238f400640 1 -- 172.21.15.27:0/734391389 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+283262 (secure 0 0 0) 0x7f23800bc090 con 0x7f23700670d0 2024-01-15T18:12:05.109 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:12:05.110 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:12:05.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.107+0000 7f239e600640 1 -- 172.21.15.27:0/734391389 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f23700670d0 msgr2=0x7f2370069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:05.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.107+0000 7f239e600640 1 --2- 172.21.15.27:0/734391389 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f23700670d0 0x7f2370069590 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f23800ad000 tx=0x7f23800a6000 comp rx=0 tx=0).stop 2024-01-15T18:12:05.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.107+0000 7f239e600640 1 -- 172.21.15.27:0/734391389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23980c5030 msgr2=0x7f23981f9a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:05.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.107+0000 7f239e600640 1 --2- 172.21.15.27:0/734391389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23980c5030 0x7f23981f9a60 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f238800c4b0 tx=0x7f238800c970 comp rx=0 tx=0).stop 2024-01-15T18:12:05.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.107+0000 7f239e600640 1 -- 172.21.15.27:0/734391389 shutdown_connections 2024-01-15T18:12:05.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.107+0000 7f239e600640 1 --2- 172.21.15.27:0/734391389 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f23700670d0 0x7f2370069590 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:05.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.107+0000 7f239e600640 1 --2- 172.21.15.27:0/734391389 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f239815e830 0x7f239815c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:05.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.107+0000 7f239e600640 1 --2- 172.21.15.27:0/734391389 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f23980d1ab0 0x7f23980ca150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:05.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.107+0000 7f239e600640 1 --2- 172.21.15.27:0/734391389 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23980c5030 0x7f23981f9a60 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:05.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.107+0000 7f239e600640 1 -- 172.21.15.27:0/734391389 >> 172.21.15.27:0/734391389 conn(0x7f23980d0f20 msgr2=0x7f23980c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:05.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.107+0000 7f239e600640 1 -- 172.21.15.27:0/734391389 shutdown_connections 2024-01-15T18:12:05.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.107+0000 7f239e600640 1 -- 172.21.15.27:0/734391389 wait complete. 2024-01-15T18:12:05.129 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2024-01-15T18:12:04.259217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":182,"acting":182,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":397208,"kb_used_data":7528,"kb_used_omap":12,"kb_used_meta":389555,"kb_avail":754577512,"statfs":{"total":773094113280,"available":772687372288,"internally_reserved":0,"allocated":7708672,"data_stored":3920296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":398904880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.270525"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.018152+0000","last_change":"2024-01-15T18:12:00.018152+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.018152+0000","last_undegraded":"2024-01-15T18:12:00.018152+0000","last_fullsized":"2024-01-15T18:12:00.018152+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.010620+0000","last_change":"2024-01-15T18:12:00.010620+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.010620+0000","last_undegraded":"2024-01-15T18:12:00.010620+0000","last_fullsized":"2024-01-15T18:12:00.010620+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.008063+0000","last_change":"2024-01-15T18:12:00.008063+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.008063+0000","last_undegraded":"2024-01-15T18:12:00.008063+0000","last_fullsized":"2024-01-15T18:12:00.008063+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.006826+0000","last_change":"2024-01-15T18:12:00.006826+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.006826+0000","last_undegraded":"2024-01-15T18:12:00.006826+0000","last_fullsized":"2024-01-15T18:12:00.006826+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.003755+0000","last_change":"2024-01-15T18:12:00.003755+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.003755+0000","last_undegraded":"2024-01-15T18:12:00.003755+0000","last_fullsized":"2024-01-15T18:12:00.003755+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.008474+0000","last_change":"2024-01-15T18:12:00.008474+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.008474+0000","last_undegraded":"2024-01-15T18:12:00.008474+0000","last_fullsized":"2024-01-15T18:12:00.008474+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005570+0000","last_change":"2024-01-15T18:12:01.023552+0000","last_active":"2024-01-15T18:12:02.005570+0000","last_peered":"2024-01-15T18:12:02.005570+0000","last_clean":"2024-01-15T18:12:02.005570+0000","last_became_active":"2024-01-15T18:12:01.023332+0000","last_became_peered":"2024-01-15T18:12:01.023332+0000","last_unstale":"2024-01-15T18:12:02.005570+0000","last_undegraded":"2024-01-15T18:12:02.005570+0000","last_fullsized":"2024-01-15T18:12:02.005570+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:13:50.232455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.009793+0000","last_change":"2024-01-15T18:12:00.009793+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.009793+0000","last_undegraded":"2024-01-15T18:12:00.009793+0000","last_fullsized":"2024-01-15T18:12:00.009793+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.005086+0000","last_change":"2024-01-15T18:12:00.005086+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.005086+0000","last_undegraded":"2024-01-15T18:12:00.005086+0000","last_fullsized":"2024-01-15T18:12:00.005086+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.006522+0000","last_change":"2024-01-15T18:12:00.006522+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.006522+0000","last_undegraded":"2024-01-15T18:12:00.006522+0000","last_fullsized":"2024-01-15T18:12:00.006522+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.017853+0000","last_change":"2024-01-15T18:12:00.017853+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.017853+0000","last_undegraded":"2024-01-15T18:12:00.017853+0000","last_fullsized":"2024-01-15T18:12:00.017853+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.006367+0000","last_change":"2024-01-15T18:12:00.006367+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.006367+0000","last_undegraded":"2024-01-15T18:12:00.006367+0000","last_fullsized":"2024-01-15T18:12:00.006367+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:11:59.986005+0000","last_change":"2024-01-15T18:11:59.986005+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:59.986005+0000","last_undegraded":"2024-01-15T18:11:59.986005+0000","last_fullsized":"2024-01-15T18:11:59.986005+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004318+0000","last_change":"2024-01-15T18:12:00.004318+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004318+0000","last_undegraded":"2024-01-15T18:12:00.004318+0000","last_fullsized":"2024-01-15T18:12:00.004318+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:11:59.986005+0000","last_change":"2024-01-15T18:11:59.986005+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:59.986005+0000","last_undegraded":"2024-01-15T18:11:59.986005+0000","last_fullsized":"2024-01-15T18:11:59.986005+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004080+0000","last_change":"2024-01-15T18:12:00.004080+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004080+0000","last_undegraded":"2024-01-15T18:12:00.004080+0000","last_fullsized":"2024-01-15T18:12:00.004080+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004665+0000","last_change":"2024-01-15T18:12:00.004665+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004665+0000","last_undegraded":"2024-01-15T18:12:00.004665+0000","last_fullsized":"2024-01-15T18:12:00.004665+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004833+0000","last_change":"2024-01-15T18:12:00.004833+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004833+0000","last_undegraded":"2024-01-15T18:12:00.004833+0000","last_fullsized":"2024-01-15T18:12:00.004833+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.010019+0000","last_change":"2024-01-15T18:12:00.010019+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.010019+0000","last_undegraded":"2024-01-15T18:12:00.010019+0000","last_fullsized":"2024-01-15T18:12:00.010019+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:11:59.986005+0000","last_change":"2024-01-15T18:11:59.986005+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:59.986005+0000","last_undegraded":"2024-01-15T18:11:59.986005+0000","last_fullsized":"2024-01-15T18:11:59.986005+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004967+0000","last_change":"2024-01-15T18:12:00.004967+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004967+0000","last_undegraded":"2024-01-15T18:12:00.004967+0000","last_fullsized":"2024-01-15T18:12:00.004967+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.005039+0000","last_change":"2024-01-15T18:12:00.005039+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.005039+0000","last_undegraded":"2024-01-15T18:12:00.005039+0000","last_fullsized":"2024-01-15T18:12:00.005039+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.014261+0000","last_change":"2024-01-15T18:12:00.014261+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.014261+0000","last_undegraded":"2024-01-15T18:12:00.014261+0000","last_fullsized":"2024-01-15T18:12:00.014261+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.014771+0000","last_change":"2024-01-15T18:12:00.014771+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.014771+0000","last_undegraded":"2024-01-15T18:12:00.014771+0000","last_fullsized":"2024-01-15T18:12:00.014771+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.005063+0000","last_change":"2024-01-15T18:12:00.005063+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.005063+0000","last_undegraded":"2024-01-15T18:12:00.005063+0000","last_fullsized":"2024-01-15T18:12:00.005063+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005958+0000","last_change":"2024-01-15T18:12:01.026872+0000","last_active":"2024-01-15T18:12:02.005958+0000","last_peered":"2024-01-15T18:12:02.005958+0000","last_clean":"2024-01-15T18:12:02.005958+0000","last_became_active":"2024-01-15T18:12:01.026554+0000","last_became_peered":"2024-01-15T18:12:01.026554+0000","last_unstale":"2024-01-15T18:12:02.005958+0000","last_undegraded":"2024-01-15T18:12:02.005958+0000","last_fullsized":"2024-01-15T18:12:02.005958+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:05:44.296057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.007628+0000","last_change":"2024-01-15T18:12:00.007628+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.007628+0000","last_undegraded":"2024-01-15T18:12:00.007628+0000","last_fullsized":"2024-01-15T18:12:00.007628+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.003323+0000","last_change":"2024-01-15T18:12:00.003323+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.003323+0000","last_undegraded":"2024-01-15T18:12:00.003323+0000","last_fullsized":"2024-01-15T18:12:00.003323+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.006443+0000","last_change":"2024-01-15T18:12:00.006443+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.006443+0000","last_undegraded":"2024-01-15T18:12:00.006443+0000","last_fullsized":"2024-01-15T18:12:00.006443+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005852+0000","last_change":"2024-01-15T18:12:01.025557+0000","last_active":"2024-01-15T18:12:02.005852+0000","last_peered":"2024-01-15T18:12:02.005852+0000","last_clean":"2024-01-15T18:12:02.005852+0000","last_became_active":"2024-01-15T18:12:01.024972+0000","last_became_peered":"2024-01-15T18:12:01.024972+0000","last_unstale":"2024-01-15T18:12:02.005852+0000","last_undegraded":"2024-01-15T18:12:02.005852+0000","last_fullsized":"2024-01-15T18:12:02.005852+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:46:42.441463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005732+0000","last_change":"2024-01-15T18:12:01.023561+0000","last_active":"2024-01-15T18:12:02.005732+0000","last_peered":"2024-01-15T18:12:02.005732+0000","last_clean":"2024-01-15T18:12:02.005732+0000","last_became_active":"2024-01-15T18:12:01.022979+0000","last_became_peered":"2024-01-15T18:12:01.022979+0000","last_unstale":"2024-01-15T18:12:02.005732+0000","last_undegraded":"2024-01-15T18:12:02.005732+0000","last_fullsized":"2024-01-15T18:12:02.005732+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:58:49.958276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.009341+0000","last_change":"2024-01-15T18:12:00.009341+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.009341+0000","last_undegraded":"2024-01-15T18:12:00.009341+0000","last_fullsized":"2024-01-15T18:12:00.009341+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.014983+0000","last_change":"2024-01-15T18:12:00.014983+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.014983+0000","last_undegraded":"2024-01-15T18:12:00.014983+0000","last_fullsized":"2024-01-15T18:12:00.014983+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.028642+0000","last_change":"2024-01-15T18:12:00.026981+0000","last_active":"2024-01-15T18:12:00.028642+0000","last_peered":"2024-01-15T18:12:00.028642+0000","last_clean":"2024-01-15T18:12:00.028642+0000","last_became_active":"2024-01-15T18:12:00.026446+0000","last_became_peered":"2024-01-15T18:12:00.026446+0000","last_unstale":"2024-01-15T18:12:00.028642+0000","last_undegraded":"2024-01-15T18:12:00.028642+0000","last_fullsized":"2024-01-15T18:12:00.028642+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:10:55.247395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004241+0000","last_change":"2024-01-15T18:12:00.004241+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004241+0000","last_undegraded":"2024-01-15T18:12:00.004241+0000","last_fullsized":"2024-01-15T18:12:00.004241+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-15T18:11:58.996438+0000","last_change":"2024-01-15T18:11:58.996438+0000","last_active":"2024-01-15T18:11:58.982718+0000","last_peered":"2024-01-15T18:11:58.982718+0000","last_clean":"2024-01-15T18:11:58.982718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:58.996438+0000","last_undegraded":"2024-01-15T18:11:58.996438+0000","last_fullsized":"2024-01-15T18:11:58.996438+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.008802+0000","last_change":"2024-01-15T18:12:00.008802+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.008802+0000","last_undegraded":"2024-01-15T18:12:00.008802+0000","last_fullsized":"2024-01-15T18:12:00.008802+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.022940+0000","last_change":"2024-01-15T18:12:00.022131+0000","last_active":"2024-01-15T18:12:00.022940+0000","last_peered":"2024-01-15T18:12:00.022940+0000","last_clean":"2024-01-15T18:12:00.022940+0000","last_became_active":"2024-01-15T18:12:00.021836+0000","last_became_peered":"2024-01-15T18:12:00.021836+0000","last_unstale":"2024-01-15T18:12:00.022940+0000","last_undegraded":"2024-01-15T18:12:00.022940+0000","last_fullsized":"2024-01-15T18:12:00.022940+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:25.596374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.014971+0000","last_change":"2024-01-15T18:12:00.014971+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.014971+0000","last_undegraded":"2024-01-15T18:12:00.014971+0000","last_fullsized":"2024-01-15T18:12:00.014971+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.022099+0000","last_change":"2024-01-15T18:12:00.019217+0000","last_active":"2024-01-15T18:12:00.022099+0000","last_peered":"2024-01-15T18:12:00.022099+0000","last_clean":"2024-01-15T18:12:00.022099+0000","last_became_active":"2024-01-15T18:12:00.018924+0000","last_became_peered":"2024-01-15T18:12:00.018924+0000","last_unstale":"2024-01-15T18:12:00.022099+0000","last_undegraded":"2024-01-15T18:12:00.022099+0000","last_fullsized":"2024-01-15T18:12:00.022099+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:49:48.318924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.006011+0000","last_change":"2024-01-15T18:12:01.028720+0000","last_active":"2024-01-15T18:12:02.006011+0000","last_peered":"2024-01-15T18:12:02.006011+0000","last_clean":"2024-01-15T18:12:02.006011+0000","last_became_active":"2024-01-15T18:12:01.028453+0000","last_became_peered":"2024-01-15T18:12:01.028453+0000","last_unstale":"2024-01-15T18:12:02.006011+0000","last_undegraded":"2024-01-15T18:12:02.006011+0000","last_fullsized":"2024-01-15T18:12:02.006011+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:20.609600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.020405+0000","last_change":"2024-01-15T18:12:00.017547+0000","last_active":"2024-01-15T18:12:00.020405+0000","last_peered":"2024-01-15T18:12:00.020405+0000","last_clean":"2024-01-15T18:12:00.020405+0000","last_became_active":"2024-01-15T18:12:00.017260+0000","last_became_peered":"2024-01-15T18:12:00.017260+0000","last_unstale":"2024-01-15T18:12:00.020405+0000","last_undegraded":"2024-01-15T18:12:00.020405+0000","last_fullsized":"2024-01-15T18:12:00.020405+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:14:22.833631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004400+0000","last_change":"2024-01-15T18:12:00.004400+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004400+0000","last_undegraded":"2024-01-15T18:12:00.004400+0000","last_fullsized":"2024-01-15T18:12:00.004400+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.028035+0000","last_change":"2024-01-15T18:12:00.025697+0000","last_active":"2024-01-15T18:12:00.028035+0000","last_peered":"2024-01-15T18:12:00.028035+0000","last_clean":"2024-01-15T18:12:00.028035+0000","last_became_active":"2024-01-15T18:12:00.025383+0000","last_became_peered":"2024-01-15T18:12:00.025383+0000","last_unstale":"2024-01-15T18:12:00.028035+0000","last_undegraded":"2024-01-15T18:12:00.028035+0000","last_fullsized":"2024-01-15T18:12:00.028035+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:53:35.224736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004715+0000","last_change":"2024-01-15T18:12:00.004715+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004715+0000","last_undegraded":"2024-01-15T18:12:00.004715+0000","last_fullsized":"2024-01-15T18:12:00.004715+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.020327+0000","last_change":"2024-01-15T18:12:00.019136+0000","last_active":"2024-01-15T18:12:00.020327+0000","last_peered":"2024-01-15T18:12:00.020327+0000","last_clean":"2024-01-15T18:12:00.020327+0000","last_became_active":"2024-01-15T18:12:00.018512+0000","last_became_peered":"2024-01-15T18:12:00.018512+0000","last_unstale":"2024-01-15T18:12:00.020327+0000","last_undegraded":"2024-01-15T18:12:00.020327+0000","last_fullsized":"2024-01-15T18:12:00.020327+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:09:59.518166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:11:59.986005+0000","last_change":"2024-01-15T18:11:59.986005+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:59.986005+0000","last_undegraded":"2024-01-15T18:11:59.986005+0000","last_fullsized":"2024-01-15T18:11:59.986005+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.024655+0000","last_change":"2024-01-15T18:12:00.021284+0000","last_active":"2024-01-15T18:12:00.024655+0000","last_peered":"2024-01-15T18:12:00.024655+0000","last_clean":"2024-01-15T18:12:00.024655+0000","last_became_active":"2024-01-15T18:12:00.021017+0000","last_became_peered":"2024-01-15T18:12:00.021017+0000","last_unstale":"2024-01-15T18:12:00.024655+0000","last_undegraded":"2024-01-15T18:12:00.024655+0000","last_fullsized":"2024-01-15T18:12:00.024655+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:39:44.736330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005945+0000","last_change":"2024-01-15T18:12:01.042737+0000","last_active":"2024-01-15T18:12:02.005945+0000","last_peered":"2024-01-15T18:12:02.005945+0000","last_clean":"2024-01-15T18:12:02.005945+0000","last_became_active":"2024-01-15T18:12:01.042482+0000","last_became_peered":"2024-01-15T18:12:01.042482+0000","last_unstale":"2024-01-15T18:12:02.005945+0000","last_undegraded":"2024-01-15T18:12:02.005945+0000","last_fullsized":"2024-01-15T18:12:02.005945+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:04:19.070396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.019580+0000","last_change":"2024-01-15T18:12:00.017218+0000","last_active":"2024-01-15T18:12:00.019580+0000","last_peered":"2024-01-15T18:12:00.019580+0000","last_clean":"2024-01-15T18:12:00.019580+0000","last_became_active":"2024-01-15T18:12:00.016921+0000","last_became_peered":"2024-01-15T18:12:00.016921+0000","last_unstale":"2024-01-15T18:12:00.019580+0000","last_undegraded":"2024-01-15T18:12:00.019580+0000","last_fullsized":"2024-01-15T18:12:00.019580+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:12:19.017434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.017663+0000","last_change":"2024-01-15T18:12:00.017663+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.017663+0000","last_undegraded":"2024-01-15T18:12:00.017663+0000","last_fullsized":"2024-01-15T18:12:00.017663+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.021218+0000","last_change":"2024-01-15T18:12:00.018102+0000","last_active":"2024-01-15T18:12:00.021218+0000","last_peered":"2024-01-15T18:12:00.021218+0000","last_clean":"2024-01-15T18:12:00.021218+0000","last_became_active":"2024-01-15T18:12:00.017781+0000","last_became_peered":"2024-01-15T18:12:00.017781+0000","last_unstale":"2024-01-15T18:12:00.021218+0000","last_undegraded":"2024-01-15T18:12:00.021218+0000","last_fullsized":"2024-01-15T18:12:00.021218+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:28:35.159833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.008315+0000","last_change":"2024-01-15T18:12:00.008315+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.008315+0000","last_undegraded":"2024-01-15T18:12:00.008315+0000","last_fullsized":"2024-01-15T18:12:00.008315+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.022940+0000","last_change":"2024-01-15T18:12:00.019006+0000","last_active":"2024-01-15T18:12:00.022940+0000","last_peered":"2024-01-15T18:12:00.022940+0000","last_clean":"2024-01-15T18:12:00.022940+0000","last_became_active":"2024-01-15T18:12:00.017853+0000","last_became_peered":"2024-01-15T18:12:00.017853+0000","last_unstale":"2024-01-15T18:12:00.022940+0000","last_undegraded":"2024-01-15T18:12:00.022940+0000","last_fullsized":"2024-01-15T18:12:00.022940+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:56:18.635395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004379+0000","last_change":"2024-01-15T18:12:00.004379+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004379+0000","last_undegraded":"2024-01-15T18:12:00.004379+0000","last_fullsized":"2024-01-15T18:12:00.004379+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.013357+0000","last_change":"2024-01-15T18:12:00.011850+0000","last_active":"2024-01-15T18:12:00.013357+0000","last_peered":"2024-01-15T18:12:00.013357+0000","last_clean":"2024-01-15T18:12:00.013357+0000","last_became_active":"2024-01-15T18:12:00.011519+0000","last_became_peered":"2024-01-15T18:12:00.011519+0000","last_unstale":"2024-01-15T18:12:00.013357+0000","last_undegraded":"2024-01-15T18:12:00.013357+0000","last_fullsized":"2024-01-15T18:12:00.013357+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:33:21.015220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.013658+0000","last_change":"2024-01-15T18:12:00.013658+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.013658+0000","last_undegraded":"2024-01-15T18:12:00.013658+0000","last_fullsized":"2024-01-15T18:12:00.013658+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.028897+0000","last_change":"2024-01-15T18:12:00.028242+0000","last_active":"2024-01-15T18:12:00.028897+0000","last_peered":"2024-01-15T18:12:00.028897+0000","last_clean":"2024-01-15T18:12:00.028897+0000","last_became_active":"2024-01-15T18:12:00.027987+0000","last_became_peered":"2024-01-15T18:12:00.027987+0000","last_unstale":"2024-01-15T18:12:00.028897+0000","last_undegraded":"2024-01-15T18:12:00.028897+0000","last_fullsized":"2024-01-15T18:12:00.028897+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:27:28.185512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005827+0000","last_change":"2024-01-15T18:12:01.024849+0000","last_active":"2024-01-15T18:12:02.005827+0000","last_peered":"2024-01-15T18:12:02.005827+0000","last_clean":"2024-01-15T18:12:02.005827+0000","last_became_active":"2024-01-15T18:12:01.024579+0000","last_became_peered":"2024-01-15T18:12:01.024579+0000","last_unstale":"2024-01-15T18:12:02.005827+0000","last_undegraded":"2024-01-15T18:12:02.005827+0000","last_fullsized":"2024-01-15T18:12:02.005827+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:43:37.142127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.020356+0000","last_change":"2024-01-15T18:12:00.014156+0000","last_active":"2024-01-15T18:12:00.020356+0000","last_peered":"2024-01-15T18:12:00.020356+0000","last_clean":"2024-01-15T18:12:00.020356+0000","last_became_active":"2024-01-15T18:12:00.013881+0000","last_became_peered":"2024-01-15T18:12:00.013881+0000","last_unstale":"2024-01-15T18:12:00.020356+0000","last_undegraded":"2024-01-15T18:12:00.020356+0000","last_fullsized":"2024-01-15T18:12:00.020356+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:55:26.157604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:11:59.986005+0000","last_change":"2024-01-15T18:11:59.986005+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:59.986005+0000","last_undegraded":"2024-01-15T18:11:59.986005+0000","last_fullsized":"2024-01-15T18:11:59.986005+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.014661+0000","last_change":"2024-01-15T18:12:00.025503+0000","last_active":"2024-01-15T18:12:01.014661+0000","last_peered":"2024-01-15T18:12:01.014661+0000","last_clean":"2024-01-15T18:12:01.014661+0000","last_became_active":"2024-01-15T18:12:00.025198+0000","last_became_peered":"2024-01-15T18:12:00.025198+0000","last_unstale":"2024-01-15T18:12:01.014661+0000","last_undegraded":"2024-01-15T18:12:01.014661+0000","last_fullsized":"2024-01-15T18:12:01.014661+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:06:28.115069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.003520+0000","last_change":"2024-01-15T18:12:00.003520+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.003520+0000","last_undegraded":"2024-01-15T18:12:00.003520+0000","last_fullsized":"2024-01-15T18:12:00.003520+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.027906+0000","last_change":"2024-01-15T18:12:00.025058+0000","last_active":"2024-01-15T18:12:00.027906+0000","last_peered":"2024-01-15T18:12:00.027906+0000","last_clean":"2024-01-15T18:12:00.027906+0000","last_became_active":"2024-01-15T18:12:00.024655+0000","last_became_peered":"2024-01-15T18:12:00.024655+0000","last_unstale":"2024-01-15T18:12:00.027906+0000","last_undegraded":"2024-01-15T18:12:00.027906+0000","last_fullsized":"2024-01-15T18:12:00.027906+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:06:24.967935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.005892+0000","last_change":"2024-01-15T18:12:00.005892+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.005892+0000","last_undegraded":"2024-01-15T18:12:00.005892+0000","last_fullsized":"2024-01-15T18:12:00.005892+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.022025+0000","last_change":"2024-01-15T18:12:00.019039+0000","last_active":"2024-01-15T18:12:00.022025+0000","last_peered":"2024-01-15T18:12:00.022025+0000","last_clean":"2024-01-15T18:12:00.022025+0000","last_became_active":"2024-01-15T18:12:00.018741+0000","last_became_peered":"2024-01-15T18:12:00.018741+0000","last_unstale":"2024-01-15T18:12:00.022025+0000","last_undegraded":"2024-01-15T18:12:00.022025+0000","last_fullsized":"2024-01-15T18:12:00.022025+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:05:09.561750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.016990+0000","last_change":"2024-01-15T18:12:00.016990+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.016990+0000","last_undegraded":"2024-01-15T18:12:00.016990+0000","last_fullsized":"2024-01-15T18:12:00.016990+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.023601+0000","last_change":"2024-01-15T18:12:00.022509+0000","last_active":"2024-01-15T18:12:00.023601+0000","last_peered":"2024-01-15T18:12:00.023601+0000","last_clean":"2024-01-15T18:12:00.023601+0000","last_became_active":"2024-01-15T18:12:00.021725+0000","last_became_peered":"2024-01-15T18:12:00.021725+0000","last_unstale":"2024-01-15T18:12:00.023601+0000","last_undegraded":"2024-01-15T18:12:00.023601+0000","last_fullsized":"2024-01-15T18:12:00.023601+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:25.993795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.003474+0000","last_change":"2024-01-15T18:12:00.003474+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.003474+0000","last_undegraded":"2024-01-15T18:12:00.003474+0000","last_fullsized":"2024-01-15T18:12:00.003474+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.026284+0000","last_change":"2024-01-15T18:12:00.022642+0000","last_active":"2024-01-15T18:12:00.026284+0000","last_peered":"2024-01-15T18:12:00.026284+0000","last_clean":"2024-01-15T18:12:00.026284+0000","last_became_active":"2024-01-15T18:12:00.021842+0000","last_became_peered":"2024-01-15T18:12:00.021842+0000","last_unstale":"2024-01-15T18:12:00.026284+0000","last_undegraded":"2024-01-15T18:12:00.026284+0000","last_fullsized":"2024-01-15T18:12:00.026284+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:54:49.641387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.003892+0000","last_change":"2024-01-15T18:12:00.003892+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.003892+0000","last_undegraded":"2024-01-15T18:12:00.003892+0000","last_fullsized":"2024-01-15T18:12:00.003892+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.028661+0000","last_change":"2024-01-15T18:12:00.027374+0000","last_active":"2024-01-15T18:12:00.028661+0000","last_peered":"2024-01-15T18:12:00.028661+0000","last_clean":"2024-01-15T18:12:00.028661+0000","last_became_active":"2024-01-15T18:12:00.027064+0000","last_became_peered":"2024-01-15T18:12:00.027064+0000","last_unstale":"2024-01-15T18:12:00.028661+0000","last_undegraded":"2024-01-15T18:12:00.028661+0000","last_fullsized":"2024-01-15T18:12:00.028661+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:54:28.422641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.006003+0000","last_change":"2024-01-15T18:12:00.006003+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.006003+0000","last_undegraded":"2024-01-15T18:12:00.006003+0000","last_fullsized":"2024-01-15T18:12:00.006003+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.028017+0000","last_change":"2024-01-15T18:12:00.023536+0000","last_active":"2024-01-15T18:12:00.028017+0000","last_peered":"2024-01-15T18:12:00.028017+0000","last_clean":"2024-01-15T18:12:00.028017+0000","last_became_active":"2024-01-15T18:12:00.023249+0000","last_became_peered":"2024-01-15T18:12:00.023249+0000","last_unstale":"2024-01-15T18:12:00.028017+0000","last_undegraded":"2024-01-15T18:12:00.028017+0000","last_fullsized":"2024-01-15T18:12:00.028017+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:48:59.611157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.005765+0000","last_change":"2024-01-15T18:12:00.005765+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.005765+0000","last_undegraded":"2024-01-15T18:12:00.005765+0000","last_fullsized":"2024-01-15T18:12:00.005765+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.017179+0000","last_change":"2024-01-15T18:12:00.020946+0000","last_active":"2024-01-15T18:12:01.017179+0000","last_peered":"2024-01-15T18:12:01.017179+0000","last_clean":"2024-01-15T18:12:01.017179+0000","last_became_active":"2024-01-15T18:12:00.020641+0000","last_became_peered":"2024-01-15T18:12:00.020641+0000","last_unstale":"2024-01-15T18:12:01.017179+0000","last_undegraded":"2024-01-15T18:12:01.017179+0000","last_fullsized":"2024-01-15T18:12:01.017179+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:14:51.219226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.005856+0000","last_change":"2024-01-15T18:12:00.005856+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.005856+0000","last_undegraded":"2024-01-15T18:12:00.005856+0000","last_fullsized":"2024-01-15T18:12:00.005856+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-15T18:11:58.996468+0000","last_change":"2024-01-15T18:11:58.996468+0000","last_active":"2024-01-15T18:11:58.982718+0000","last_peered":"2024-01-15T18:11:58.982718+0000","last_clean":"2024-01-15T18:11:58.982718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:58.996468+0000","last_undegraded":"2024-01-15T18:11:58.996468+0000","last_fullsized":"2024-01-15T18:11:58.996468+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:11:59.986005+0000","last_change":"2024-01-15T18:11:59.986005+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:59.986005+0000","last_undegraded":"2024-01-15T18:11:59.986005+0000","last_fullsized":"2024-01-15T18:11:59.986005+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.021188+0000","last_change":"2024-01-15T18:12:00.016456+0000","last_active":"2024-01-15T18:12:00.021188+0000","last_peered":"2024-01-15T18:12:00.021188+0000","last_clean":"2024-01-15T18:12:00.021188+0000","last_became_active":"2024-01-15T18:12:00.016145+0000","last_became_peered":"2024-01-15T18:12:00.016145+0000","last_unstale":"2024-01-15T18:12:00.021188+0000","last_undegraded":"2024-01-15T18:12:00.021188+0000","last_fullsized":"2024-01-15T18:12:00.021188+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:02:23.099071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.006010+0000","last_change":"2024-01-15T18:12:01.024249+0000","last_active":"2024-01-15T18:12:02.006010+0000","last_peered":"2024-01-15T18:12:02.006010+0000","last_clean":"2024-01-15T18:12:02.006010+0000","last_became_active":"2024-01-15T18:12:01.023862+0000","last_became_peered":"2024-01-15T18:12:01.023862+0000","last_unstale":"2024-01-15T18:12:02.006010+0000","last_undegraded":"2024-01-15T18:12:02.006010+0000","last_fullsized":"2024-01-15T18:12:02.006010+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:18:47.967166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.016071+0000","last_change":"2024-01-15T18:12:00.023511+0000","last_active":"2024-01-15T18:12:01.016071+0000","last_peered":"2024-01-15T18:12:01.016071+0000","last_clean":"2024-01-15T18:12:01.016071+0000","last_became_active":"2024-01-15T18:12:00.023145+0000","last_became_peered":"2024-01-15T18:12:00.023145+0000","last_unstale":"2024-01-15T18:12:01.016071+0000","last_undegraded":"2024-01-15T18:12:01.016071+0000","last_fullsized":"2024-01-15T18:12:01.016071+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:27:30.560351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.016823+0000","last_change":"2024-01-15T18:12:00.016823+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.016823+0000","last_undegraded":"2024-01-15T18:12:00.016823+0000","last_fullsized":"2024-01-15T18:12:00.016823+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.025706+0000","last_change":"2024-01-15T18:12:00.023014+0000","last_active":"2024-01-15T18:12:00.025706+0000","last_peered":"2024-01-15T18:12:00.025706+0000","last_clean":"2024-01-15T18:12:00.025706+0000","last_became_active":"2024-01-15T18:12:00.022748+0000","last_became_peered":"2024-01-15T18:12:00.022748+0000","last_unstale":"2024-01-15T18:12:00.025706+0000","last_undegraded":"2024-01-15T18:12:00.025706+0000","last_fullsized":"2024-01-15T18:12:00.025706+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:13:33.148973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.014451+0000","last_change":"2024-01-15T18:12:00.014451+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.014451+0000","last_undegraded":"2024-01-15T18:12:00.014451+0000","last_fullsized":"2024-01-15T18:12:00.014451+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-15T18:11:58.997807+0000","last_change":"2024-01-15T18:11:58.997807+0000","last_active":"2024-01-15T18:11:58.982718+0000","last_peered":"2024-01-15T18:11:58.982718+0000","last_clean":"2024-01-15T18:11:58.982718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:58.997807+0000","last_undegraded":"2024-01-15T18:11:58.997807+0000","last_fullsized":"2024-01-15T18:11:58.997807+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.003849+0000","last_change":"2024-01-15T18:12:00.003849+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.003849+0000","last_undegraded":"2024-01-15T18:12:00.003849+0000","last_fullsized":"2024-01-15T18:12:00.003849+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.021303+0000","last_change":"2024-01-15T18:12:00.019406+0000","last_active":"2024-01-15T18:12:00.021303+0000","last_peered":"2024-01-15T18:12:00.021303+0000","last_clean":"2024-01-15T18:12:00.021303+0000","last_became_active":"2024-01-15T18:12:00.019122+0000","last_became_peered":"2024-01-15T18:12:00.019122+0000","last_unstale":"2024-01-15T18:12:00.021303+0000","last_undegraded":"2024-01-15T18:12:00.021303+0000","last_fullsized":"2024-01-15T18:12:00.021303+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:51:33.138526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.007471+0000","last_change":"2024-01-15T18:12:00.007471+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.007471+0000","last_undegraded":"2024-01-15T18:12:00.007471+0000","last_fullsized":"2024-01-15T18:12:00.007471+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.018689+0000","last_change":"2024-01-15T18:12:00.015364+0000","last_active":"2024-01-15T18:12:00.018689+0000","last_peered":"2024-01-15T18:12:00.018689+0000","last_clean":"2024-01-15T18:12:00.018689+0000","last_became_active":"2024-01-15T18:12:00.014208+0000","last_became_peered":"2024-01-15T18:12:00.014208+0000","last_unstale":"2024-01-15T18:12:00.018689+0000","last_undegraded":"2024-01-15T18:12:00.018689+0000","last_fullsized":"2024-01-15T18:12:00.018689+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:59:42.431920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004040+0000","last_change":"2024-01-15T18:12:00.004040+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004040+0000","last_undegraded":"2024-01-15T18:12:00.004040+0000","last_fullsized":"2024-01-15T18:12:00.004040+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.027927+0000","last_change":"2024-01-15T18:12:00.025623+0000","last_active":"2024-01-15T18:12:00.027927+0000","last_peered":"2024-01-15T18:12:00.027927+0000","last_clean":"2024-01-15T18:12:00.027927+0000","last_became_active":"2024-01-15T18:12:00.025294+0000","last_became_peered":"2024-01-15T18:12:00.025294+0000","last_unstale":"2024-01-15T18:12:00.027927+0000","last_undegraded":"2024-01-15T18:12:00.027927+0000","last_fullsized":"2024-01-15T18:12:00.027927+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:02:43.133788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.007044+0000","last_change":"2024-01-15T18:12:00.007044+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.007044+0000","last_undegraded":"2024-01-15T18:12:00.007044+0000","last_fullsized":"2024-01-15T18:12:00.007044+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.023758+0000","last_change":"2024-01-15T18:12:00.019753+0000","last_active":"2024-01-15T18:12:00.023758+0000","last_peered":"2024-01-15T18:12:00.023758+0000","last_clean":"2024-01-15T18:12:00.023758+0000","last_became_active":"2024-01-15T18:12:00.019482+0000","last_became_peered":"2024-01-15T18:12:00.019482+0000","last_unstale":"2024-01-15T18:12:00.023758+0000","last_undegraded":"2024-01-15T18:12:00.023758+0000","last_fullsized":"2024-01-15T18:12:00.023758+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:53:26.480802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.009189+0000","last_change":"2024-01-15T18:12:00.009189+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.009189+0000","last_undegraded":"2024-01-15T18:12:00.009189+0000","last_fullsized":"2024-01-15T18:12:00.009189+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.022076+0000","last_change":"2024-01-15T18:12:00.019108+0000","last_active":"2024-01-15T18:12:00.022076+0000","last_peered":"2024-01-15T18:12:00.022076+0000","last_clean":"2024-01-15T18:12:00.022076+0000","last_became_active":"2024-01-15T18:12:00.018827+0000","last_became_peered":"2024-01-15T18:12:00.018827+0000","last_unstale":"2024-01-15T18:12:00.022076+0000","last_undegraded":"2024-01-15T18:12:00.022076+0000","last_fullsized":"2024-01-15T18:12:00.022076+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:48:57.061505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":203,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-01-15T18:11:58.994013+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:11:58.994013+0000","last_peered":"2024-01-15T18:11:58.994013+0000","last_clean":"2024-01-15T18:11:58.994013+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:11:58.994013+0000","last_undegraded":"2024-01-15T18:11:58.994013+0000","last_fullsized":"2024-01-15T18:11:58.994013+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":116,"acting":116,"num_store_stats":7},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509530,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50596,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321244,"statfs":{"total":96636764160,"available":96584953856,"internally_reserved":0,"allocated":1400832,"data_stored":928722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.1200000000000001,"15min":0.78900000000000003},"min":{"1min":0.66000000000000003,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":9.6989999999999998,"5min":10.284000000000001,"15min":10.284000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":2.4159999999999999,"5min":1.2290000000000001,"15min":0.86299999999999999},"min":{"1min":0.39100000000000001,"5min":0.378,"15min":0.378},"max":{"1min":16.968,"5min":16.968,"15min":16.968},"last":0.56599999999999995}]},{"osd":1,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.7749999999999999,"5min":1.1830000000000001,"15min":0.83899999999999997},"min":{"1min":0.64800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":15.791,"5min":15.791,"15min":15.791},"last":0.54300000000000004},{"interface":"front","average":{"1min":2.1339999999999999,"5min":1.1779999999999999,"15min":0.85899999999999999},"min":{"1min":0.44600000000000001,"5min":0.378,"15min":0.378},"max":{"1min":10.842000000000001,"5min":10.842000000000001,"15min":10.842000000000001},"last":0.64800000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1,"15min":0.76400000000000001},"min":{"1min":0.42299999999999999,"5min":0.378,"15min":0.378},"max":{"1min":4.9029999999999996,"5min":10.119999999999999,"15min":10.119999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":2.0579999999999998,"5min":1.081,"15min":0.83499999999999996},"min":{"1min":0.66800000000000004,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":9.7539999999999996,"5min":9.7539999999999996,"15min":9.7539999999999996},"last":0.75900000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":1.802,"5min":1.143,"15min":0.82299999999999995},"min":{"1min":0.47099999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":11.481,"5min":11.481,"15min":11.481},"last":0.69299999999999995},{"interface":"front","average":{"1min":2.3220000000000001,"5min":1.2310000000000001,"15min":0.91900000000000004},"min":{"1min":0.70199999999999996,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":9.7959999999999994,"5min":10.34,"15min":10.34},"last":0.67200000000000004}]},{"osd":4,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.5630000000000002,"5min":1.3360000000000001,"15min":0.85799999999999998},"min":{"1min":0.41099999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":20.387,"5min":20.387,"15min":20.387},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.649,"5min":1.131,"15min":0.81999999999999995},"min":{"1min":0.46200000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":11.305,"5min":14.871,"15min":14.871},"last":0.623}]},{"osd":5,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.2719999999999998,"5min":1.2070000000000001,"15min":0.86899999999999999},"min":{"1min":0.47799999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":9.8450000000000006,"5min":19.959,"15min":19.959},"last":0.59099999999999997},{"interface":"front","average":{"1min":2.6240000000000001,"5min":1.3009999999999999,"15min":0.92000000000000004},"min":{"1min":0.48299999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":19.905000000000001,"5min":19.905000000000001,"15min":19.905000000000001},"last":0.83199999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":3.157,"5min":1.667,"15min":1.093},"min":{"1min":0.63800000000000001,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":20.492000000000001,"5min":22.478999999999999,"15min":22.478999999999999},"last":0.877},{"interface":"front","average":{"1min":1.397,"5min":1.3049999999999999,"15min":0.92500000000000004},"min":{"1min":0.45100000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.2549999999999999,"5min":21.509,"15min":21.509},"last":0.73899999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509530,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45856,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325984,"statfs":{"total":96636764160,"available":96589807616,"internally_reserved":0,"allocated":806912,"data_stored":338354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":1.157,"15min":0.872},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.91800000000000004,"5min":12.41,"15min":12.41},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.69199999999999995,"5min":1.0009999999999999,"15min":0.79000000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.93500000000000005,"5min":6.7759999999999998,"15min":6.7759999999999998},"last":0.57399999999999995}]},{"osd":1,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":1.454,"15min":0.96399999999999997},"min":{"1min":0.46999999999999997,"5min":0.41899999999999998,"15min":0.41299999999999998},"max":{"1min":0.83199999999999996,"5min":53.451999999999998,"15min":53.451999999999998},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.71299999999999997,"5min":1.008,"15min":0.83699999999999997},"min":{"1min":0.56100000000000005,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.92900000000000005,"5min":7.7329999999999997,"15min":7.7329999999999997},"last":0.56200000000000006}]},{"osd":2,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":1.004,"15min":0.89400000000000002},"min":{"1min":0.46700000000000003,"5min":0.45900000000000002,"15min":0.38900000000000001},"max":{"1min":1.1060000000000001,"5min":7.9409999999999998,"15min":7.9409999999999998},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.93700000000000006,"15min":0.82999999999999996},"min":{"1min":0.57099999999999995,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.90800000000000003,"5min":5.1760000000000002,"15min":5.1760000000000002},"last":0.75800000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":1.038,"15min":0.876},"min":{"1min":0.46000000000000002,"5min":0.42099999999999999,"15min":0.41399999999999998},"max":{"1min":0.94899999999999995,"5min":6.758,"15min":6.758},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.70499999999999996,"5min":1.5600000000000001,"15min":1.0660000000000001},"min":{"1min":0.49199999999999999,"5min":0.45200000000000001,"15min":0.41599999999999998},"max":{"1min":0.877,"5min":53.398000000000003,"15min":53.398000000000003},"last":0.80600000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":1.0029999999999999,"15min":0.79500000000000004},"min":{"1min":0.38500000000000001,"5min":0.378,"15min":0.34999999999999998},"max":{"1min":0.88,"5min":8.6259999999999994,"15min":8.6259999999999994},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.94899999999999995,"15min":0.79100000000000004},"min":{"1min":0.42599999999999999,"5min":0.38200000000000001,"15min":0.34499999999999997},"max":{"1min":0.85099999999999998,"5min":6.9589999999999996,"15min":6.9589999999999996},"last":0.63}]},{"osd":5,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":1.641,"15min":0.98299999999999998},"min":{"1min":0.41299999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.96399999999999997,"5min":53.229999999999997,"15min":53.229999999999997},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.69299999999999995,"5min":1.228,"15min":0.92900000000000005},"min":{"1min":0.45500000000000002,"5min":0.435,"15min":0.435},"max":{"1min":1.149,"5min":10.029999999999999,"15min":10.029999999999999},"last":0.745}]},{"osd":7,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":1.5760000000000001,"15min":1.071},"min":{"1min":0.437,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.90400000000000003,"5min":45.070999999999998,"15min":45.070999999999998},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.70099999999999996,"5min":1.6910000000000001,"15min":1.0940000000000001},"min":{"1min":0.52800000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.92200000000000004,"5min":53.338000000000001,"15min":53.338000000000001},"last":0.77300000000000002}]}]},{"osd":4,"up_from":15,"seq":64424509530,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50028,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321812,"statfs":{"total":96636764160,"available":96585535488,"internally_reserved":0,"allocated":819200,"data_stored":344685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":1.2,"15min":0.81899999999999995},"min":{"1min":0.502,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.2509999999999999,"5min":23.318000000000001,"15min":23.318000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.70699999999999996,"5min":1.046,"15min":0.77500000000000002},"min":{"1min":0.442,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.97899999999999998,"5min":14.723000000000001,"15min":14.723000000000001},"last":0.442}]},{"osd":1,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.89500000000000002,"15min":0.72999999999999998},"min":{"1min":0.442,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.876,"5min":7.8200000000000003,"15min":7.8200000000000003},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.73899999999999999,"5min":1.1879999999999999,"15min":0.82299999999999995},"min":{"1min":0.47299999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.613,"5min":22.408999999999999,"15min":22.408999999999999},"last":0.58999999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.91800000000000004,"15min":0.72399999999999998},"min":{"1min":0.48699999999999999,"5min":0.40600000000000003,"15min":0.34399999999999997},"max":{"1min":1.0009999999999999,"5min":8.8529999999999998,"15min":8.8529999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.77900000000000003,"5min":1.288,"15min":0.89200000000000002},"min":{"1min":0.55500000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.2969999999999999,"5min":22.338999999999999,"15min":22.338999999999999},"last":0.65800000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":1.149,"15min":0.84299999999999997},"min":{"1min":0.52500000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.3939999999999999,"5min":14.763999999999999,"15min":14.763999999999999},"last":0.622},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0509999999999999,"15min":0.83399999999999996},"min":{"1min":0.59899999999999998,"5min":0.504,"15min":0.45500000000000002},"max":{"1min":1.7669999999999999,"5min":8.9459999999999997,"15min":8.9459999999999997},"last":0.67300000000000004}]},{"osd":5,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":1.2609999999999999,"15min":0.86399999999999999},"min":{"1min":0.39800000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.4670000000000001,"5min":31.617999999999999,"15min":31.617999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":1.1910000000000001,"15min":0.83099999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.377},"max":{"1min":1.022,"5min":17.145,"15min":17.145},"last":0.55100000000000005}]},{"osd":6,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.95299999999999996,"15min":0.77100000000000002},"min":{"1min":0.46500000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.89300000000000002,"5min":11.124000000000001,"15min":11.124000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.749,"5min":1.7629999999999999,"15min":1.0780000000000001},"min":{"1min":0.45600000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.339,"5min":28.872,"15min":28.872},"last":0.76700000000000002}]},{"osd":7,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":1.5409999999999999,"15min":0.99299999999999999},"min":{"1min":0.45800000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.3169999999999999,"5min":39.256,"15min":39.256},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":1.1160000000000001,"15min":0.89200000000000002},"min":{"1min":0.51900000000000002,"5min":0.46500000000000002,"15min":0.42199999999999999},"max":{"1min":1.0329999999999999,"5min":26.573,"15min":26.573},"last":0.72899999999999998}]}]},{"osd":3,"up_from":15,"seq":64424509529,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50016,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321824,"statfs":{"total":96636764160,"available":96585547776,"internally_reserved":0,"allocated":806912,"data_stored":338354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":0.58899999999999997,"15min":0.47699999999999998},"min":{"1min":0.29799999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":5.8849999999999998,"5min":5.8849999999999998,"15min":5.8849999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.1299999999999999,"5min":0.81999999999999995,"15min":0.58699999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":4.7990000000000004,"5min":17.756,"15min":17.756},"last":0.44400000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":0.755,"15min":0.59099999999999997},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":12.307,"5min":12.307,"15min":12.307},"last":0.56999999999999995},{"interface":"front","average":{"1min":1.1080000000000001,"5min":0.66300000000000003,"15min":0.57799999999999996},"min":{"1min":0.34899999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":6.0640000000000001,"5min":6.0640000000000001,"15min":6.0640000000000001},"last":0.51400000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":0.84999999999999998,"15min":0.65800000000000003},"min":{"1min":0.36099999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":12.468999999999999,"5min":12.468999999999999,"15min":12.468999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.2549999999999999,"5min":0.77400000000000002,"15min":0.63400000000000001},"min":{"1min":0.41599999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":6.0330000000000004,"5min":6.0330000000000004,"15min":6.0330000000000004},"last":0.46899999999999997}]},{"osd":4,"last update":"Mon Jan 15 18:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.141,"15min":0.91900000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":6.3970000000000002,"5min":17.853999999999999,"15min":17.853999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":2.1240000000000001,"5min":1.115,"15min":0.89300000000000002},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":13.31,"5min":13.31,"15min":13.31},"last":0.67000000000000004}]},{"osd":5,"last update":"Mon Jan 15 18:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":1.0109999999999999,"15min":0.83099999999999996},"min":{"1min":0.52100000000000002,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":6.1139999999999999,"5min":6.1139999999999999,"15min":6.1139999999999999},"last":0.81899999999999995},{"interface":"front","average":{"1min":2.1720000000000002,"5min":1.284,"15min":0.90800000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":20.515999999999998,"5min":20.515999999999998,"15min":20.515999999999998},"last":0.83199999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:10:58 2024","interfaces":[{"interface":"back","average":{"1min":2.2879999999999998,"5min":1.333,"15min":0.96899999999999997},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":13.250999999999999,"5min":23.100999999999999,"15min":23.100999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":2.246,"5min":1.1339999999999999,"15min":0.88600000000000001},"min":{"1min":0.5,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":12.76,"5min":12.76,"15min":12.76},"last":0.86199999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:10:58 2024","interfaces":[{"interface":"back","average":{"1min":2.9430000000000001,"5min":1.3819999999999999,"15min":0.97899999999999998},"min":{"1min":0.48599999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":21.545000000000002,"5min":21.545000000000002,"15min":21.545000000000002},"last":0.70199999999999996},{"interface":"front","average":{"1min":2.7650000000000001,"5min":1.3200000000000001,"15min":0.98899999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":13.385,"5min":13.385,"15min":13.385},"last":0.80500000000000005}]}]},{"osd":2,"up_from":15,"seq":64424509529,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45872,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325968,"statfs":{"total":96636764160,"available":96589791232,"internally_reserved":0,"allocated":819200,"data_stored":344685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.61699999999999999,"15min":0.51200000000000001},"min":{"1min":0.29399999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.8480000000000001,"5min":5.1849999999999996,"15min":5.1849999999999996},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.64200000000000002,"15min":0.52300000000000002},"min":{"1min":0.32600000000000001,"5min":0.29399999999999998,"15min":0.28299999999999997},"max":{"1min":2.3759999999999999,"5min":7.516,"15min":7.516},"last":0.50700000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.748,"15min":0.61199999999999999},"min":{"1min":0.34100000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":2.2850000000000001,"5min":10.907,"15min":10.907},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.81599999999999995,"15min":0.57399999999999995},"min":{"1min":0.40500000000000003,"5min":0.311,"15min":0.27000000000000002},"max":{"1min":1.7150000000000001,"5min":8.9190000000000005,"15min":8.9190000000000005},"last":0.68999999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.73599999999999999,"15min":0.628},"min":{"1min":0.36599999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.86399999999999999,"5min":12.185,"15min":12.185},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.83499999999999996,"15min":0.64700000000000002},"min":{"1min":0.443,"5min":0.39100000000000001,"15min":0.34000000000000002},"max":{"1min":2.3330000000000002,"5min":7.6760000000000002,"15min":7.6760000000000002},"last":0.63500000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.99299999999999999,"15min":0.82699999999999996},"min":{"1min":0.58699999999999997,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":2.21,"5min":8.6609999999999996,"15min":8.6609999999999996},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.76000000000000001,"5min":1.1899999999999999,"15min":0.90600000000000003},"min":{"1min":0.54000000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.6339999999999999,"5min":19.146999999999998,"15min":19.146999999999998},"last":0.77500000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":1.1319999999999999,"15min":0.90800000000000003},"min":{"1min":0.442,"5min":0.44,"15min":0.44},"max":{"1min":0.90800000000000003,"5min":31.710000000000001,"15min":31.710000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.875,"5min":1.373,"15min":0.98299999999999998},"min":{"1min":0.59399999999999997,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.1669999999999998,"5min":22.539000000000001,"15min":22.539000000000001},"last":0.871}]},{"osd":6,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":1.107,"15min":0.88600000000000001},"min":{"1min":0.61699999999999999,"5min":0.50800000000000001,"15min":0.48299999999999998},"max":{"1min":1.675,"5min":9.4540000000000006,"15min":9.4540000000000006},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.1399999999999999,"15min":0.90900000000000003},"min":{"1min":0.53100000000000003,"5min":0.503,"15min":0.503},"max":{"1min":1,"5min":18.614999999999998,"15min":18.614999999999998},"last":0.74199999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":1.113,"15min":0.90400000000000003},"min":{"1min":0.58199999999999996,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.752,"5min":11.432,"15min":11.432},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.264,"15min":0.97099999999999997},"min":{"1min":0.54500000000000004,"5min":0.502,"15min":0.502},"max":{"1min":2.2480000000000002,"5min":17.013000000000002,"15min":17.013000000000002},"last":0.90600000000000003}]}]},{"osd":1,"up_from":15,"seq":64424509531,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50032,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321808,"statfs":{"total":96636764160,"available":96585531392,"internally_reserved":0,"allocated":819200,"data_stored":344685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.96399999999999997,"15min":0.65500000000000003},"min":{"1min":0.309,"5min":0.26500000000000001,"15min":0.254},"max":{"1min":0.72899999999999998,"5min":19.577000000000002,"15min":19.577000000000002},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":1.1779999999999999,"15min":0.71499999999999997},"min":{"1min":0.35399999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.80000000000000004,"5min":48.277999999999999,"15min":48.277999999999999},"last":0.42899999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":1.036,"15min":0.70599999999999996},"min":{"1min":0.35599999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.65000000000000002,"5min":40.698,"15min":40.698},"last":0.379},{"interface":"front","average":{"1min":0.53200000000000003,"5min":1.337,"15min":0.92900000000000005},"min":{"1min":0.36899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93300000000000005,"5min":48.316000000000003,"15min":48.316000000000003},"last":0.35399999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.2370000000000001,"15min":0.83999999999999997},"min":{"1min":0.46600000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.88200000000000001,"5min":48.234999999999999,"15min":48.234999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.75800000000000001,"15min":0.64500000000000002},"min":{"1min":0.36099999999999999,"5min":0.34699999999999998,"15min":0.311},"max":{"1min":0.74399999999999999,"5min":6.468,"15min":6.468},"last":0.69299999999999995}]},{"osd":4,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":1.0589999999999999,"15min":0.81799999999999995},"min":{"1min":0.48999999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.0720000000000001,"5min":8.3409999999999993,"15min":8.3409999999999993},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.80200000000000005,"5min":1.165,"15min":0.90600000000000003},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.42499999999999999},"max":{"1min":1.278,"5min":19.829000000000001,"15min":19.829000000000001},"last":0.52200000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":1.4399999999999999,"15min":0.99399999999999999},"min":{"1min":0.59499999999999997,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.306,"5min":22.594999999999999,"15min":22.594999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.79400000000000004,"5min":1.8939999999999999,"15min":1.145},"min":{"1min":0.54500000000000004,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1180000000000001,"5min":48.198,"15min":48.198},"last":0.57699999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":1.4590000000000001,"15min":0.98899999999999999},"min":{"1min":0.59899999999999998,"5min":0.53000000000000003,"15min":0.47499999999999998},"max":{"1min":1.1499999999999999,"5min":35.112000000000002,"15min":35.112000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.81200000000000006,"5min":1.1779999999999999,"15min":0.93999999999999995},"min":{"1min":0.53100000000000003,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.175,"5min":19.780999999999999,"15min":19.780999999999999},"last":0.59999999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":1.8069999999999999,"15min":1.1259999999999999},"min":{"1min":0.628,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2450000000000001,"5min":48.148000000000003,"15min":48.148000000000003},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.80300000000000005,"5min":1.194,"15min":0.93300000000000005},"min":{"1min":0.64000000000000001,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.214,"5min":9.3840000000000003,"15min":9.3840000000000003},"last":0.84199999999999997}]}]},{"osd":5,"up_from":14,"seq":60129542235,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50064,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321776,"statfs":{"total":96636764160,"available":96585498624,"internally_reserved":0,"allocated":847872,"data_stored":357962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.872,"15min":0.77600000000000002},"min":{"1min":0.45900000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.3019999999999996,"5min":21.385000000000002,"15min":21.385000000000002},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.732,"5min":1.071,"15min":0.88800000000000001},"min":{"1min":0.46999999999999997,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":9.359,"5min":9.359,"15min":9.359},"last":0.78200000000000003}]},{"osd":1,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":3.0960000000000001,"5min":1.1839999999999999,"15min":0.84599999999999997},"min":{"1min":0.48799999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":21.120000000000001,"5min":21.120000000000001,"15min":21.120000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0329999999999999,"15min":0.80100000000000005},"min":{"1min":0.52400000000000002,"5min":0.315,"15min":0.315},"max":{"1min":4.3730000000000002,"5min":21.5,"15min":21.5},"last":0.753}]},{"osd":2,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.1419999999999999,"15min":0.85399999999999998},"min":{"1min":0.56399999999999995,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":6.2750000000000004,"5min":8.7750000000000004,"15min":8.7750000000000004},"last":0.73799999999999999},{"interface":"front","average":{"1min":2.9780000000000002,"5min":1.2869999999999999,"15min":0.94999999999999996},"min":{"1min":0.58099999999999996,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":21.201000000000001,"5min":21.201000000000001,"15min":21.201000000000001},"last":0.76600000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.0069999999999999,"15min":0.875},"min":{"1min":0.57399999999999995,"5min":0.44,"15min":0.44},"max":{"1min":4.4100000000000001,"5min":21.457000000000001,"15min":21.457000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.583,"5min":1.196,"15min":0.96799999999999997},"min":{"1min":0.54200000000000004,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":6.8650000000000002,"5min":8.7300000000000004,"15min":8.7300000000000004},"last":0.64900000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.012,"15min":0.82799999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":5.8929999999999998,"5min":9.6679999999999993,"15min":9.6679999999999993},"last":0.55700000000000005},{"interface":"front","average":{"1min":3.8380000000000001,"5min":1.5329999999999999,"15min":1.008},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":21.724,"5min":26.146999999999998,"15min":26.146999999999998},"last":0.70099999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.91900000000000004,"15min":0.79800000000000004},"min":{"1min":0.36099999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.835,"5min":21.535,"15min":21.535},"last":0.67600000000000005},{"interface":"front","average":{"1min":1.6160000000000001,"5min":1.3839999999999999,"15min":0.999},"min":{"1min":0.40100000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":5.7539999999999996,"5min":30.488,"15min":30.488},"last":0.82099999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":4.7309999999999999,"5min":1.6259999999999999,"15min":1.0760000000000001},"min":{"1min":0.42599999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":30.483000000000001,"5min":30.483000000000001,"15min":30.483000000000001},"last":0.629},{"interface":"front","average":{"1min":3.508,"5min":1.4650000000000001,"15min":1.0429999999999999},"min":{"1min":0.42899999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":25.437999999999999,"5min":25.437999999999999,"15min":25.437999999999999},"last":0.72199999999999998}]}]},{"osd":0,"up_from":14,"seq":60129542234,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54744,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317096,"statfs":{"total":96636764160,"available":96580706304,"internally_reserved":0,"allocated":1388544,"data_stored":922849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.89300000000000002,"15min":0.60699999999999998},"min":{"1min":0.26000000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9179999999999999,"5min":19.843,"15min":19.843},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.88700000000000001,"15min":0.66400000000000003},"min":{"1min":0.29199999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73299999999999998,"5min":10.154,"15min":10.154},"last":0.42999999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":1.115,"15min":0.753},"min":{"1min":0.38800000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.754,"5min":25.905000000000001,"15min":25.905000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.5,"5min":0.77900000000000003,"15min":0.58199999999999996},"min":{"1min":0.29699999999999999,"5min":0.28599999999999998,"15min":0.26700000000000002},"max":{"1min":1.2549999999999999,"5min":9.3829999999999991,"15min":9.3829999999999991},"last":0.38800000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.95699999999999996,"15min":0.70299999999999996},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.308},"max":{"1min":1.952,"5min":9.1780000000000008,"15min":9.1780000000000008},"last":0.496},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.97399999999999998,"15min":0.73099999999999998},"min":{"1min":0.42899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":2.9809999999999999,"5min":10.210000000000001,"15min":10.210000000000001},"last":0.51200000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":1.343,"15min":0.96399999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.38},"max":{"1min":1.306,"5min":22.814,"15min":22.814},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.446,"15min":1.03},"min":{"1min":0.49099999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":2.169,"5min":23.341999999999999,"15min":23.341999999999999},"last":0.73299999999999998}]},{"osd":5,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":1.262,"15min":0.94899999999999995},"min":{"1min":0.46899999999999997,"5min":0.44600000000000001,"15min":0.441},"max":{"1min":1.365,"5min":12.428000000000001,"15min":12.428000000000001},"last":0.874},{"interface":"front","average":{"1min":0.82499999999999996,"5min":1.6930000000000001,"15min":1.1259999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.46300000000000002},"max":{"1min":2.0880000000000001,"5min":19.774000000000001,"15min":19.774000000000001},"last":0.78600000000000003}]},{"osd":6,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":1.9219999999999999,"15min":1.1719999999999999},"min":{"1min":0.46700000000000003,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.0449999999999999,"5min":31.059000000000001,"15min":31.059000000000001},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.83299999999999996,"5min":1.488,"15min":1.0580000000000001},"min":{"1min":0.57999999999999996,"5min":0.51500000000000001,"15min":0.49099999999999999},"max":{"1min":1.3899999999999999,"5min":40.731000000000002,"15min":40.731000000000002},"last":0.80800000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":1.673,"15min":1.1100000000000001},"min":{"1min":0.61599999999999999,"5min":0.47199999999999998,"15min":0.42499999999999999},"max":{"1min":1.4239999999999999,"5min":24.492999999999999,"15min":24.492999999999999},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.875,"5min":1.5780000000000001,"15min":1.097},"min":{"1min":0.61199999999999999,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":2.2130000000000001,"5min":23.524000000000001,"15min":23.524000000000001},"last":0.82599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:12:05.135 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:12:05.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.311+0000 7fce1a200640 1 Processor -- start 2024-01-15T18:12:05.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.311+0000 7fce1a200640 1 -- start start 2024-01-15T18:12:05.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.311+0000 7fce1a200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce14071e40 0x7fce140c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:05.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.311+0000 7fce1a200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fce1406d430 0x7fce140cd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:05.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.311+0000 7fce1a200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce14159a50 con 0x7fce14159bf0 2024-01-15T18:12:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.311+0000 7fce1a200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce140cce00 con 0x7fce1406d430 2024-01-15T18:12:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.311+0000 7fce1a200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce1406d980 con 0x7fce14071e40 2024-01-15T18:12:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.311+0000 7fce13400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fce14159bf0 0x7fce14159fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59430/0 (socket says 172.21.15.27:59430) 2024-01-15T18:12:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.311+0000 7fce13400640 1 -- 172.21.15.27:0/351446001 learned_addr learned my addr 172.21.15.27:0/351446001 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:12:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.311+0000 7fce13e00640 1 --2- 172.21.15.27:0/351446001 >> v2:172.21.15.27:3300/0 conn(0x7fce1406d430 0x7fce140cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.311+0000 7fce12a00640 1 --2- 172.21.15.27:0/351446001 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce14071e40 0x7fce140c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.311+0000 7fce12000640 1 -- 172.21.15.27:0/351446001 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151547868 0 0) 0x7fce04001120 con 0x7fce14159bf0 2024-01-15T18:12:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.311+0000 7fce12000640 1 -- 172.21.15.27:0/351446001 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdf4003530 con 0x7fce14159bf0 2024-01-15T18:12:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.311+0000 7fce13e00640 1 -- 172.21.15.27:0/351446001 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce14071e40 msgr2=0x7fce140c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.311+0000 7fce13e00640 1 --2- 172.21.15.27:0/351446001 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce14071e40 0x7fce140c9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.311+0000 7fce13e00640 1 -- 172.21.15.27:0/351446001 >> v1:172.21.15.27:6789/0 conn(0x7fce14159bf0 legacy=0x7fce14159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:05.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.311+0000 7fce13e00640 1 -- 172.21.15.27:0/351446001 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce1406dc60 con 0x7fce1406d430 2024-01-15T18:12:05.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.315+0000 7fce13e00640 1 --2- 172.21.15.27:0/351446001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce1406d430 0x7fce140cd250 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fce0c00aed0 tx=0x7fce0c020240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a7537be2169fbf4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:05.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.315+0000 7fce12000640 1 -- 172.21.15.27:0/351446001 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0c00a2d0 con 0x7fce1406d430 2024-01-15T18:12:05.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.315+0000 7fce12000640 1 -- 172.21.15.27:0/351446001 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce0c0ac050 con 0x7fce1406d430 2024-01-15T18:12:05.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.315+0000 7fce12000640 1 -- 172.21.15.27:0/351446001 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0c008640 con 0x7fce1406d430 2024-01-15T18:12:05.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.315+0000 7fce1a200640 1 -- 172.21.15.27:0/351446001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce1406d430 msgr2=0x7fce140cd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:05.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.315+0000 7fce1a200640 1 --2- 172.21.15.27:0/351446001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce1406d430 0x7fce140cd250 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fce0c00aed0 tx=0x7fce0c020240 comp rx=0 tx=0).stop 2024-01-15T18:12:05.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.315+0000 7fce1a200640 1 -- 172.21.15.27:0/351446001 shutdown_connections 2024-01-15T18:12:05.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.315+0000 7fce1a200640 1 --2- 172.21.15.27:0/351446001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce1406d430 0x7fce140cd250 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:05.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.315+0000 7fce1a200640 1 --2- 172.21.15.27:0/351446001 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce14071e40 0x7fce140c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:05.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.315+0000 7fce1a200640 1 -- 172.21.15.27:0/351446001 >> 172.21.15.27:0/351446001 conn(0x7fce140cec60 msgr2=0x7fce140cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:05.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.315+0000 7fce1a200640 1 -- 172.21.15.27:0/351446001 shutdown_connections 2024-01-15T18:12:05.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.315+0000 7fce1a200640 1 -- 172.21.15.27:0/351446001 wait complete. 2024-01-15T18:12:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce1a200640 1 Processor -- start 2024-01-15T18:12:05.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce1a200640 1 -- start start 2024-01-15T18:12:05.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce1a200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce1406d430 0x7fce141e4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:05.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce1a200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce14159bf0 0x7fce141e6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:05.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce1a200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce141e67a0 0x7fce141e8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:05.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce1a200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce141e48e0 con 0x7fce1406d430 2024-01-15T18:12:05.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce1a200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce141e3980 con 0x7fce141e67a0 2024-01-15T18:12:05.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce1a200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce141e3c60 con 0x7fce14159bf0 2024-01-15T18:12:05.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce13400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce1406d430 0x7fce141e4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:05.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce13400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce1406d430 0x7fce141e4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58296/0 (socket says 172.21.15.27:58296) 2024-01-15T18:12:05.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce13400640 1 -- 172.21.15.27:0/89623855 learned_addr learned my addr 172.21.15.27:0/89623855 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:05.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce12a00640 1 --2- 172.21.15.27:0/89623855 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce14159bf0 0x7fce141e6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:05.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce13e00640 1 --2- 172.21.15.27:0/89623855 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce141e67a0 0x7fce141e8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:05.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce13400640 1 -- 172.21.15.27:0/89623855 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce14159bf0 msgr2=0x7fce141e6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:05.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce13400640 1 --2- 172.21.15.27:0/89623855 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce14159bf0 0x7fce141e6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:05.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce13400640 1 -- 172.21.15.27:0/89623855 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce141e67a0 msgr2=0x7fce141e8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:05.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce13400640 1 --2- 172.21.15.27:0/89623855 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce141e67a0 0x7fce141e8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:05.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce13400640 1 -- 172.21.15.27:0/89623855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0c00ab80 con 0x7fce1406d430 2024-01-15T18:12:05.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce12a00640 1 --2- 172.21.15.27:0/89623855 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce14159bf0 0x7fce141e6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:12:05.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce13400640 1 --2- 172.21.15.27:0/89623855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce1406d430 0x7fce141e4d00 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fce0400c400 tx=0x7fce0400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:05.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce0b400640 1 -- 172.21.15.27:0/89623855 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce04005cf0 con 0x7fce1406d430 2024-01-15T18:12:05.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce0b400640 1 -- 172.21.15.27:0/89623855 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce04005e70 con 0x7fce1406d430 2024-01-15T18:12:05.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce1a200640 1 -- 172.21.15.27:0/89623855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce141e3f40 con 0x7fce1406d430 2024-01-15T18:12:05.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce0b400640 1 -- 172.21.15.27:0/89623855 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce040072f0 con 0x7fce1406d430 2024-01-15T18:12:05.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce1a200640 1 -- 172.21.15.27:0/89623855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce141e4500 con 0x7fce1406d430 2024-01-15T18:12:05.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.319+0000 7fce1a200640 1 -- 172.21.15.27:0/89623855 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce1404e850 con 0x7fce1406d430 2024-01-15T18:12:05.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.323+0000 7fce0b400640 1 -- 172.21.15.27:0/89623855 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7fce04007450 con 0x7fce1406d430 2024-01-15T18:12:05.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.323+0000 7fce0b400640 1 --2- 172.21.15.27:0/89623855 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcdec0670d0 0x7fcdec069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:05.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.323+0000 7fce0b400640 1 -- 172.21.15.27:0/89623855 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fce04081f50 con 0x7fce1406d430 2024-01-15T18:12:05.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.323+0000 7fce12a00640 1 --2- 172.21.15.27:0/89623855 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcdec0670d0 0x7fcdec069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:05.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.323+0000 7fce12a00640 1 --2- 172.21.15.27:0/89623855 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcdec0670d0 0x7fcdec069590 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fce140c54f0 tx=0x7fcdfc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:05.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.327+0000 7fce0b400640 1 -- 172.21.15.27:0/89623855 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fce04054820 con 0x7fce1406d430 2024-01-15T18:12:05.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.483+0000 7fce1a200640 1 -- 172.21.15.27:0/89623855 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fce140ca120 con 0x7fcdec0670d0 2024-01-15T18:12:05.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.491+0000 7fce0b400640 1 -- 172.21.15.27:0/89623855 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+283262 (secure 0 0 0) 0x7fcdfc011a40 con 0x7fcdec0670d0 2024-01-15T18:12:05.495 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:12:05.497 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:12:05.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.495+0000 7fce1a200640 1 -- 172.21.15.27:0/89623855 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcdec0670d0 msgr2=0x7fcdec069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:05.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.495+0000 7fce1a200640 1 --2- 172.21.15.27:0/89623855 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcdec0670d0 0x7fcdec069590 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fce140c54f0 tx=0x7fcdfc008040 comp rx=0 tx=0).stop 2024-01-15T18:12:05.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.495+0000 7fce1a200640 1 -- 172.21.15.27:0/89623855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce1406d430 msgr2=0x7fce141e4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:05.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.495+0000 7fce1a200640 1 --2- 172.21.15.27:0/89623855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce1406d430 0x7fce141e4d00 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fce0400c400 tx=0x7fce0400c8c0 comp rx=0 tx=0).stop 2024-01-15T18:12:05.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.495+0000 7fce1a200640 1 -- 172.21.15.27:0/89623855 shutdown_connections 2024-01-15T18:12:05.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.495+0000 7fce1a200640 1 --2- 172.21.15.27:0/89623855 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcdec0670d0 0x7fcdec069590 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:05.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.495+0000 7fce1a200640 1 --2- 172.21.15.27:0/89623855 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce141e67a0 0x7fce141e8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:05.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.495+0000 7fce1a200640 1 --2- 172.21.15.27:0/89623855 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce14159bf0 0x7fce141e6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:05.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.495+0000 7fce1a200640 1 --2- 172.21.15.27:0/89623855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce1406d430 0x7fce141e4d00 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:05.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.495+0000 7fce1a200640 1 -- 172.21.15.27:0/89623855 >> 172.21.15.27:0/89623855 conn(0x7fce140cec60 msgr2=0x7fce140afc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:05.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.495+0000 7fce1a200640 1 -- 172.21.15.27:0/89623855 shutdown_connections 2024-01-15T18:12:05.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.495+0000 7fce1a200640 1 -- 172.21.15.27:0/89623855 wait complete. 2024-01-15T18:12:05.517 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2024-01-15T18:12:04.259217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":182,"acting":182,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":397208,"kb_used_data":7528,"kb_used_omap":12,"kb_used_meta":389555,"kb_avail":754577512,"statfs":{"total":773094113280,"available":772687372288,"internally_reserved":0,"allocated":7708672,"data_stored":3920296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":398904880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.270525"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.018152+0000","last_change":"2024-01-15T18:12:00.018152+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.018152+0000","last_undegraded":"2024-01-15T18:12:00.018152+0000","last_fullsized":"2024-01-15T18:12:00.018152+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.010620+0000","last_change":"2024-01-15T18:12:00.010620+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.010620+0000","last_undegraded":"2024-01-15T18:12:00.010620+0000","last_fullsized":"2024-01-15T18:12:00.010620+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.008063+0000","last_change":"2024-01-15T18:12:00.008063+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.008063+0000","last_undegraded":"2024-01-15T18:12:00.008063+0000","last_fullsized":"2024-01-15T18:12:00.008063+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.006826+0000","last_change":"2024-01-15T18:12:00.006826+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.006826+0000","last_undegraded":"2024-01-15T18:12:00.006826+0000","last_fullsized":"2024-01-15T18:12:00.006826+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.003755+0000","last_change":"2024-01-15T18:12:00.003755+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.003755+0000","last_undegraded":"2024-01-15T18:12:00.003755+0000","last_fullsized":"2024-01-15T18:12:00.003755+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.008474+0000","last_change":"2024-01-15T18:12:00.008474+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.008474+0000","last_undegraded":"2024-01-15T18:12:00.008474+0000","last_fullsized":"2024-01-15T18:12:00.008474+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005570+0000","last_change":"2024-01-15T18:12:01.023552+0000","last_active":"2024-01-15T18:12:02.005570+0000","last_peered":"2024-01-15T18:12:02.005570+0000","last_clean":"2024-01-15T18:12:02.005570+0000","last_became_active":"2024-01-15T18:12:01.023332+0000","last_became_peered":"2024-01-15T18:12:01.023332+0000","last_unstale":"2024-01-15T18:12:02.005570+0000","last_undegraded":"2024-01-15T18:12:02.005570+0000","last_fullsized":"2024-01-15T18:12:02.005570+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:13:50.232455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.009793+0000","last_change":"2024-01-15T18:12:00.009793+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.009793+0000","last_undegraded":"2024-01-15T18:12:00.009793+0000","last_fullsized":"2024-01-15T18:12:00.009793+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.005086+0000","last_change":"2024-01-15T18:12:00.005086+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.005086+0000","last_undegraded":"2024-01-15T18:12:00.005086+0000","last_fullsized":"2024-01-15T18:12:00.005086+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.006522+0000","last_change":"2024-01-15T18:12:00.006522+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.006522+0000","last_undegraded":"2024-01-15T18:12:00.006522+0000","last_fullsized":"2024-01-15T18:12:00.006522+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.017853+0000","last_change":"2024-01-15T18:12:00.017853+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.017853+0000","last_undegraded":"2024-01-15T18:12:00.017853+0000","last_fullsized":"2024-01-15T18:12:00.017853+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.006367+0000","last_change":"2024-01-15T18:12:00.006367+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.006367+0000","last_undegraded":"2024-01-15T18:12:00.006367+0000","last_fullsized":"2024-01-15T18:12:00.006367+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:11:59.986005+0000","last_change":"2024-01-15T18:11:59.986005+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:59.986005+0000","last_undegraded":"2024-01-15T18:11:59.986005+0000","last_fullsized":"2024-01-15T18:11:59.986005+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004318+0000","last_change":"2024-01-15T18:12:00.004318+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004318+0000","last_undegraded":"2024-01-15T18:12:00.004318+0000","last_fullsized":"2024-01-15T18:12:00.004318+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:11:59.986005+0000","last_change":"2024-01-15T18:11:59.986005+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:59.986005+0000","last_undegraded":"2024-01-15T18:11:59.986005+0000","last_fullsized":"2024-01-15T18:11:59.986005+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004080+0000","last_change":"2024-01-15T18:12:00.004080+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004080+0000","last_undegraded":"2024-01-15T18:12:00.004080+0000","last_fullsized":"2024-01-15T18:12:00.004080+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004665+0000","last_change":"2024-01-15T18:12:00.004665+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004665+0000","last_undegraded":"2024-01-15T18:12:00.004665+0000","last_fullsized":"2024-01-15T18:12:00.004665+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004833+0000","last_change":"2024-01-15T18:12:00.004833+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004833+0000","last_undegraded":"2024-01-15T18:12:00.004833+0000","last_fullsized":"2024-01-15T18:12:00.004833+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.010019+0000","last_change":"2024-01-15T18:12:00.010019+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.010019+0000","last_undegraded":"2024-01-15T18:12:00.010019+0000","last_fullsized":"2024-01-15T18:12:00.010019+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:11:59.986005+0000","last_change":"2024-01-15T18:11:59.986005+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:59.986005+0000","last_undegraded":"2024-01-15T18:11:59.986005+0000","last_fullsized":"2024-01-15T18:11:59.986005+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004967+0000","last_change":"2024-01-15T18:12:00.004967+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004967+0000","last_undegraded":"2024-01-15T18:12:00.004967+0000","last_fullsized":"2024-01-15T18:12:00.004967+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.005039+0000","last_change":"2024-01-15T18:12:00.005039+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.005039+0000","last_undegraded":"2024-01-15T18:12:00.005039+0000","last_fullsized":"2024-01-15T18:12:00.005039+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.014261+0000","last_change":"2024-01-15T18:12:00.014261+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.014261+0000","last_undegraded":"2024-01-15T18:12:00.014261+0000","last_fullsized":"2024-01-15T18:12:00.014261+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.014771+0000","last_change":"2024-01-15T18:12:00.014771+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.014771+0000","last_undegraded":"2024-01-15T18:12:00.014771+0000","last_fullsized":"2024-01-15T18:12:00.014771+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.005063+0000","last_change":"2024-01-15T18:12:00.005063+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.005063+0000","last_undegraded":"2024-01-15T18:12:00.005063+0000","last_fullsized":"2024-01-15T18:12:00.005063+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005958+0000","last_change":"2024-01-15T18:12:01.026872+0000","last_active":"2024-01-15T18:12:02.005958+0000","last_peered":"2024-01-15T18:12:02.005958+0000","last_clean":"2024-01-15T18:12:02.005958+0000","last_became_active":"2024-01-15T18:12:01.026554+0000","last_became_peered":"2024-01-15T18:12:01.026554+0000","last_unstale":"2024-01-15T18:12:02.005958+0000","last_undegraded":"2024-01-15T18:12:02.005958+0000","last_fullsized":"2024-01-15T18:12:02.005958+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:05:44.296057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.007628+0000","last_change":"2024-01-15T18:12:00.007628+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.007628+0000","last_undegraded":"2024-01-15T18:12:00.007628+0000","last_fullsized":"2024-01-15T18:12:00.007628+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.003323+0000","last_change":"2024-01-15T18:12:00.003323+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.003323+0000","last_undegraded":"2024-01-15T18:12:00.003323+0000","last_fullsized":"2024-01-15T18:12:00.003323+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.006443+0000","last_change":"2024-01-15T18:12:00.006443+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.006443+0000","last_undegraded":"2024-01-15T18:12:00.006443+0000","last_fullsized":"2024-01-15T18:12:00.006443+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005852+0000","last_change":"2024-01-15T18:12:01.025557+0000","last_active":"2024-01-15T18:12:02.005852+0000","last_peered":"2024-01-15T18:12:02.005852+0000","last_clean":"2024-01-15T18:12:02.005852+0000","last_became_active":"2024-01-15T18:12:01.024972+0000","last_became_peered":"2024-01-15T18:12:01.024972+0000","last_unstale":"2024-01-15T18:12:02.005852+0000","last_undegraded":"2024-01-15T18:12:02.005852+0000","last_fullsized":"2024-01-15T18:12:02.005852+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:46:42.441463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005732+0000","last_change":"2024-01-15T18:12:01.023561+0000","last_active":"2024-01-15T18:12:02.005732+0000","last_peered":"2024-01-15T18:12:02.005732+0000","last_clean":"2024-01-15T18:12:02.005732+0000","last_became_active":"2024-01-15T18:12:01.022979+0000","last_became_peered":"2024-01-15T18:12:01.022979+0000","last_unstale":"2024-01-15T18:12:02.005732+0000","last_undegraded":"2024-01-15T18:12:02.005732+0000","last_fullsized":"2024-01-15T18:12:02.005732+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:58:49.958276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.009341+0000","last_change":"2024-01-15T18:12:00.009341+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.009341+0000","last_undegraded":"2024-01-15T18:12:00.009341+0000","last_fullsized":"2024-01-15T18:12:00.009341+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.014983+0000","last_change":"2024-01-15T18:12:00.014983+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.014983+0000","last_undegraded":"2024-01-15T18:12:00.014983+0000","last_fullsized":"2024-01-15T18:12:00.014983+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.028642+0000","last_change":"2024-01-15T18:12:00.026981+0000","last_active":"2024-01-15T18:12:00.028642+0000","last_peered":"2024-01-15T18:12:00.028642+0000","last_clean":"2024-01-15T18:12:00.028642+0000","last_became_active":"2024-01-15T18:12:00.026446+0000","last_became_peered":"2024-01-15T18:12:00.026446+0000","last_unstale":"2024-01-15T18:12:00.028642+0000","last_undegraded":"2024-01-15T18:12:00.028642+0000","last_fullsized":"2024-01-15T18:12:00.028642+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:10:55.247395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004241+0000","last_change":"2024-01-15T18:12:00.004241+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004241+0000","last_undegraded":"2024-01-15T18:12:00.004241+0000","last_fullsized":"2024-01-15T18:12:00.004241+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-15T18:11:58.996438+0000","last_change":"2024-01-15T18:11:58.996438+0000","last_active":"2024-01-15T18:11:58.982718+0000","last_peered":"2024-01-15T18:11:58.982718+0000","last_clean":"2024-01-15T18:11:58.982718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:58.996438+0000","last_undegraded":"2024-01-15T18:11:58.996438+0000","last_fullsized":"2024-01-15T18:11:58.996438+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.008802+0000","last_change":"2024-01-15T18:12:00.008802+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.008802+0000","last_undegraded":"2024-01-15T18:12:00.008802+0000","last_fullsized":"2024-01-15T18:12:00.008802+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.022940+0000","last_change":"2024-01-15T18:12:00.022131+0000","last_active":"2024-01-15T18:12:00.022940+0000","last_peered":"2024-01-15T18:12:00.022940+0000","last_clean":"2024-01-15T18:12:00.022940+0000","last_became_active":"2024-01-15T18:12:00.021836+0000","last_became_peered":"2024-01-15T18:12:00.021836+0000","last_unstale":"2024-01-15T18:12:00.022940+0000","last_undegraded":"2024-01-15T18:12:00.022940+0000","last_fullsized":"2024-01-15T18:12:00.022940+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:25.596374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.014971+0000","last_change":"2024-01-15T18:12:00.014971+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.014971+0000","last_undegraded":"2024-01-15T18:12:00.014971+0000","last_fullsized":"2024-01-15T18:12:00.014971+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.022099+0000","last_change":"2024-01-15T18:12:00.019217+0000","last_active":"2024-01-15T18:12:00.022099+0000","last_peered":"2024-01-15T18:12:00.022099+0000","last_clean":"2024-01-15T18:12:00.022099+0000","last_became_active":"2024-01-15T18:12:00.018924+0000","last_became_peered":"2024-01-15T18:12:00.018924+0000","last_unstale":"2024-01-15T18:12:00.022099+0000","last_undegraded":"2024-01-15T18:12:00.022099+0000","last_fullsized":"2024-01-15T18:12:00.022099+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:49:48.318924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.006011+0000","last_change":"2024-01-15T18:12:01.028720+0000","last_active":"2024-01-15T18:12:02.006011+0000","last_peered":"2024-01-15T18:12:02.006011+0000","last_clean":"2024-01-15T18:12:02.006011+0000","last_became_active":"2024-01-15T18:12:01.028453+0000","last_became_peered":"2024-01-15T18:12:01.028453+0000","last_unstale":"2024-01-15T18:12:02.006011+0000","last_undegraded":"2024-01-15T18:12:02.006011+0000","last_fullsized":"2024-01-15T18:12:02.006011+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:20.609600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.020405+0000","last_change":"2024-01-15T18:12:00.017547+0000","last_active":"2024-01-15T18:12:00.020405+0000","last_peered":"2024-01-15T18:12:00.020405+0000","last_clean":"2024-01-15T18:12:00.020405+0000","last_became_active":"2024-01-15T18:12:00.017260+0000","last_became_peered":"2024-01-15T18:12:00.017260+0000","last_unstale":"2024-01-15T18:12:00.020405+0000","last_undegraded":"2024-01-15T18:12:00.020405+0000","last_fullsized":"2024-01-15T18:12:00.020405+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:14:22.833631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004400+0000","last_change":"2024-01-15T18:12:00.004400+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004400+0000","last_undegraded":"2024-01-15T18:12:00.004400+0000","last_fullsized":"2024-01-15T18:12:00.004400+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.028035+0000","last_change":"2024-01-15T18:12:00.025697+0000","last_active":"2024-01-15T18:12:00.028035+0000","last_peered":"2024-01-15T18:12:00.028035+0000","last_clean":"2024-01-15T18:12:00.028035+0000","last_became_active":"2024-01-15T18:12:00.025383+0000","last_became_peered":"2024-01-15T18:12:00.025383+0000","last_unstale":"2024-01-15T18:12:00.028035+0000","last_undegraded":"2024-01-15T18:12:00.028035+0000","last_fullsized":"2024-01-15T18:12:00.028035+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:53:35.224736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004715+0000","last_change":"2024-01-15T18:12:00.004715+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004715+0000","last_undegraded":"2024-01-15T18:12:00.004715+0000","last_fullsized":"2024-01-15T18:12:00.004715+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.020327+0000","last_change":"2024-01-15T18:12:00.019136+0000","last_active":"2024-01-15T18:12:00.020327+0000","last_peered":"2024-01-15T18:12:00.020327+0000","last_clean":"2024-01-15T18:12:00.020327+0000","last_became_active":"2024-01-15T18:12:00.018512+0000","last_became_peered":"2024-01-15T18:12:00.018512+0000","last_unstale":"2024-01-15T18:12:00.020327+0000","last_undegraded":"2024-01-15T18:12:00.020327+0000","last_fullsized":"2024-01-15T18:12:00.020327+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:09:59.518166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:11:59.986005+0000","last_change":"2024-01-15T18:11:59.986005+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:59.986005+0000","last_undegraded":"2024-01-15T18:11:59.986005+0000","last_fullsized":"2024-01-15T18:11:59.986005+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.024655+0000","last_change":"2024-01-15T18:12:00.021284+0000","last_active":"2024-01-15T18:12:00.024655+0000","last_peered":"2024-01-15T18:12:00.024655+0000","last_clean":"2024-01-15T18:12:00.024655+0000","last_became_active":"2024-01-15T18:12:00.021017+0000","last_became_peered":"2024-01-15T18:12:00.021017+0000","last_unstale":"2024-01-15T18:12:00.024655+0000","last_undegraded":"2024-01-15T18:12:00.024655+0000","last_fullsized":"2024-01-15T18:12:00.024655+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:39:44.736330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005945+0000","last_change":"2024-01-15T18:12:01.042737+0000","last_active":"2024-01-15T18:12:02.005945+0000","last_peered":"2024-01-15T18:12:02.005945+0000","last_clean":"2024-01-15T18:12:02.005945+0000","last_became_active":"2024-01-15T18:12:01.042482+0000","last_became_peered":"2024-01-15T18:12:01.042482+0000","last_unstale":"2024-01-15T18:12:02.005945+0000","last_undegraded":"2024-01-15T18:12:02.005945+0000","last_fullsized":"2024-01-15T18:12:02.005945+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:04:19.070396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.019580+0000","last_change":"2024-01-15T18:12:00.017218+0000","last_active":"2024-01-15T18:12:00.019580+0000","last_peered":"2024-01-15T18:12:00.019580+0000","last_clean":"2024-01-15T18:12:00.019580+0000","last_became_active":"2024-01-15T18:12:00.016921+0000","last_became_peered":"2024-01-15T18:12:00.016921+0000","last_unstale":"2024-01-15T18:12:00.019580+0000","last_undegraded":"2024-01-15T18:12:00.019580+0000","last_fullsized":"2024-01-15T18:12:00.019580+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:12:19.017434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.017663+0000","last_change":"2024-01-15T18:12:00.017663+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.017663+0000","last_undegraded":"2024-01-15T18:12:00.017663+0000","last_fullsized":"2024-01-15T18:12:00.017663+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.021218+0000","last_change":"2024-01-15T18:12:00.018102+0000","last_active":"2024-01-15T18:12:00.021218+0000","last_peered":"2024-01-15T18:12:00.021218+0000","last_clean":"2024-01-15T18:12:00.021218+0000","last_became_active":"2024-01-15T18:12:00.017781+0000","last_became_peered":"2024-01-15T18:12:00.017781+0000","last_unstale":"2024-01-15T18:12:00.021218+0000","last_undegraded":"2024-01-15T18:12:00.021218+0000","last_fullsized":"2024-01-15T18:12:00.021218+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:28:35.159833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.008315+0000","last_change":"2024-01-15T18:12:00.008315+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.008315+0000","last_undegraded":"2024-01-15T18:12:00.008315+0000","last_fullsized":"2024-01-15T18:12:00.008315+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.022940+0000","last_change":"2024-01-15T18:12:00.019006+0000","last_active":"2024-01-15T18:12:00.022940+0000","last_peered":"2024-01-15T18:12:00.022940+0000","last_clean":"2024-01-15T18:12:00.022940+0000","last_became_active":"2024-01-15T18:12:00.017853+0000","last_became_peered":"2024-01-15T18:12:00.017853+0000","last_unstale":"2024-01-15T18:12:00.022940+0000","last_undegraded":"2024-01-15T18:12:00.022940+0000","last_fullsized":"2024-01-15T18:12:00.022940+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:56:18.635395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004379+0000","last_change":"2024-01-15T18:12:00.004379+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004379+0000","last_undegraded":"2024-01-15T18:12:00.004379+0000","last_fullsized":"2024-01-15T18:12:00.004379+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.013357+0000","last_change":"2024-01-15T18:12:00.011850+0000","last_active":"2024-01-15T18:12:00.013357+0000","last_peered":"2024-01-15T18:12:00.013357+0000","last_clean":"2024-01-15T18:12:00.013357+0000","last_became_active":"2024-01-15T18:12:00.011519+0000","last_became_peered":"2024-01-15T18:12:00.011519+0000","last_unstale":"2024-01-15T18:12:00.013357+0000","last_undegraded":"2024-01-15T18:12:00.013357+0000","last_fullsized":"2024-01-15T18:12:00.013357+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:33:21.015220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.013658+0000","last_change":"2024-01-15T18:12:00.013658+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.013658+0000","last_undegraded":"2024-01-15T18:12:00.013658+0000","last_fullsized":"2024-01-15T18:12:00.013658+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.028897+0000","last_change":"2024-01-15T18:12:00.028242+0000","last_active":"2024-01-15T18:12:00.028897+0000","last_peered":"2024-01-15T18:12:00.028897+0000","last_clean":"2024-01-15T18:12:00.028897+0000","last_became_active":"2024-01-15T18:12:00.027987+0000","last_became_peered":"2024-01-15T18:12:00.027987+0000","last_unstale":"2024-01-15T18:12:00.028897+0000","last_undegraded":"2024-01-15T18:12:00.028897+0000","last_fullsized":"2024-01-15T18:12:00.028897+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:27:28.185512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005827+0000","last_change":"2024-01-15T18:12:01.024849+0000","last_active":"2024-01-15T18:12:02.005827+0000","last_peered":"2024-01-15T18:12:02.005827+0000","last_clean":"2024-01-15T18:12:02.005827+0000","last_became_active":"2024-01-15T18:12:01.024579+0000","last_became_peered":"2024-01-15T18:12:01.024579+0000","last_unstale":"2024-01-15T18:12:02.005827+0000","last_undegraded":"2024-01-15T18:12:02.005827+0000","last_fullsized":"2024-01-15T18:12:02.005827+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:43:37.142127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.020356+0000","last_change":"2024-01-15T18:12:00.014156+0000","last_active":"2024-01-15T18:12:00.020356+0000","last_peered":"2024-01-15T18:12:00.020356+0000","last_clean":"2024-01-15T18:12:00.020356+0000","last_became_active":"2024-01-15T18:12:00.013881+0000","last_became_peered":"2024-01-15T18:12:00.013881+0000","last_unstale":"2024-01-15T18:12:00.020356+0000","last_undegraded":"2024-01-15T18:12:00.020356+0000","last_fullsized":"2024-01-15T18:12:00.020356+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:55:26.157604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:11:59.986005+0000","last_change":"2024-01-15T18:11:59.986005+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:59.986005+0000","last_undegraded":"2024-01-15T18:11:59.986005+0000","last_fullsized":"2024-01-15T18:11:59.986005+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.014661+0000","last_change":"2024-01-15T18:12:00.025503+0000","last_active":"2024-01-15T18:12:01.014661+0000","last_peered":"2024-01-15T18:12:01.014661+0000","last_clean":"2024-01-15T18:12:01.014661+0000","last_became_active":"2024-01-15T18:12:00.025198+0000","last_became_peered":"2024-01-15T18:12:00.025198+0000","last_unstale":"2024-01-15T18:12:01.014661+0000","last_undegraded":"2024-01-15T18:12:01.014661+0000","last_fullsized":"2024-01-15T18:12:01.014661+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:06:28.115069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.003520+0000","last_change":"2024-01-15T18:12:00.003520+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.003520+0000","last_undegraded":"2024-01-15T18:12:00.003520+0000","last_fullsized":"2024-01-15T18:12:00.003520+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.027906+0000","last_change":"2024-01-15T18:12:00.025058+0000","last_active":"2024-01-15T18:12:00.027906+0000","last_peered":"2024-01-15T18:12:00.027906+0000","last_clean":"2024-01-15T18:12:00.027906+0000","last_became_active":"2024-01-15T18:12:00.024655+0000","last_became_peered":"2024-01-15T18:12:00.024655+0000","last_unstale":"2024-01-15T18:12:00.027906+0000","last_undegraded":"2024-01-15T18:12:00.027906+0000","last_fullsized":"2024-01-15T18:12:00.027906+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:06:24.967935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.005892+0000","last_change":"2024-01-15T18:12:00.005892+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.005892+0000","last_undegraded":"2024-01-15T18:12:00.005892+0000","last_fullsized":"2024-01-15T18:12:00.005892+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.022025+0000","last_change":"2024-01-15T18:12:00.019039+0000","last_active":"2024-01-15T18:12:00.022025+0000","last_peered":"2024-01-15T18:12:00.022025+0000","last_clean":"2024-01-15T18:12:00.022025+0000","last_became_active":"2024-01-15T18:12:00.018741+0000","last_became_peered":"2024-01-15T18:12:00.018741+0000","last_unstale":"2024-01-15T18:12:00.022025+0000","last_undegraded":"2024-01-15T18:12:00.022025+0000","last_fullsized":"2024-01-15T18:12:00.022025+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:05:09.561750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.016990+0000","last_change":"2024-01-15T18:12:00.016990+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.016990+0000","last_undegraded":"2024-01-15T18:12:00.016990+0000","last_fullsized":"2024-01-15T18:12:00.016990+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.023601+0000","last_change":"2024-01-15T18:12:00.022509+0000","last_active":"2024-01-15T18:12:00.023601+0000","last_peered":"2024-01-15T18:12:00.023601+0000","last_clean":"2024-01-15T18:12:00.023601+0000","last_became_active":"2024-01-15T18:12:00.021725+0000","last_became_peered":"2024-01-15T18:12:00.021725+0000","last_unstale":"2024-01-15T18:12:00.023601+0000","last_undegraded":"2024-01-15T18:12:00.023601+0000","last_fullsized":"2024-01-15T18:12:00.023601+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:25.993795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.003474+0000","last_change":"2024-01-15T18:12:00.003474+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.003474+0000","last_undegraded":"2024-01-15T18:12:00.003474+0000","last_fullsized":"2024-01-15T18:12:00.003474+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.026284+0000","last_change":"2024-01-15T18:12:00.022642+0000","last_active":"2024-01-15T18:12:00.026284+0000","last_peered":"2024-01-15T18:12:00.026284+0000","last_clean":"2024-01-15T18:12:00.026284+0000","last_became_active":"2024-01-15T18:12:00.021842+0000","last_became_peered":"2024-01-15T18:12:00.021842+0000","last_unstale":"2024-01-15T18:12:00.026284+0000","last_undegraded":"2024-01-15T18:12:00.026284+0000","last_fullsized":"2024-01-15T18:12:00.026284+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:54:49.641387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.003892+0000","last_change":"2024-01-15T18:12:00.003892+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.003892+0000","last_undegraded":"2024-01-15T18:12:00.003892+0000","last_fullsized":"2024-01-15T18:12:00.003892+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.028661+0000","last_change":"2024-01-15T18:12:00.027374+0000","last_active":"2024-01-15T18:12:00.028661+0000","last_peered":"2024-01-15T18:12:00.028661+0000","last_clean":"2024-01-15T18:12:00.028661+0000","last_became_active":"2024-01-15T18:12:00.027064+0000","last_became_peered":"2024-01-15T18:12:00.027064+0000","last_unstale":"2024-01-15T18:12:00.028661+0000","last_undegraded":"2024-01-15T18:12:00.028661+0000","last_fullsized":"2024-01-15T18:12:00.028661+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:54:28.422641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.006003+0000","last_change":"2024-01-15T18:12:00.006003+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.006003+0000","last_undegraded":"2024-01-15T18:12:00.006003+0000","last_fullsized":"2024-01-15T18:12:00.006003+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.028017+0000","last_change":"2024-01-15T18:12:00.023536+0000","last_active":"2024-01-15T18:12:00.028017+0000","last_peered":"2024-01-15T18:12:00.028017+0000","last_clean":"2024-01-15T18:12:00.028017+0000","last_became_active":"2024-01-15T18:12:00.023249+0000","last_became_peered":"2024-01-15T18:12:00.023249+0000","last_unstale":"2024-01-15T18:12:00.028017+0000","last_undegraded":"2024-01-15T18:12:00.028017+0000","last_fullsized":"2024-01-15T18:12:00.028017+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:48:59.611157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.005765+0000","last_change":"2024-01-15T18:12:00.005765+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.005765+0000","last_undegraded":"2024-01-15T18:12:00.005765+0000","last_fullsized":"2024-01-15T18:12:00.005765+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.017179+0000","last_change":"2024-01-15T18:12:00.020946+0000","last_active":"2024-01-15T18:12:01.017179+0000","last_peered":"2024-01-15T18:12:01.017179+0000","last_clean":"2024-01-15T18:12:01.017179+0000","last_became_active":"2024-01-15T18:12:00.020641+0000","last_became_peered":"2024-01-15T18:12:00.020641+0000","last_unstale":"2024-01-15T18:12:01.017179+0000","last_undegraded":"2024-01-15T18:12:01.017179+0000","last_fullsized":"2024-01-15T18:12:01.017179+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:14:51.219226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.005856+0000","last_change":"2024-01-15T18:12:00.005856+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.005856+0000","last_undegraded":"2024-01-15T18:12:00.005856+0000","last_fullsized":"2024-01-15T18:12:00.005856+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-15T18:11:58.996468+0000","last_change":"2024-01-15T18:11:58.996468+0000","last_active":"2024-01-15T18:11:58.982718+0000","last_peered":"2024-01-15T18:11:58.982718+0000","last_clean":"2024-01-15T18:11:58.982718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:58.996468+0000","last_undegraded":"2024-01-15T18:11:58.996468+0000","last_fullsized":"2024-01-15T18:11:58.996468+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:11:59.986005+0000","last_change":"2024-01-15T18:11:59.986005+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:59.986005+0000","last_undegraded":"2024-01-15T18:11:59.986005+0000","last_fullsized":"2024-01-15T18:11:59.986005+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.021188+0000","last_change":"2024-01-15T18:12:00.016456+0000","last_active":"2024-01-15T18:12:00.021188+0000","last_peered":"2024-01-15T18:12:00.021188+0000","last_clean":"2024-01-15T18:12:00.021188+0000","last_became_active":"2024-01-15T18:12:00.016145+0000","last_became_peered":"2024-01-15T18:12:00.016145+0000","last_unstale":"2024-01-15T18:12:00.021188+0000","last_undegraded":"2024-01-15T18:12:00.021188+0000","last_fullsized":"2024-01-15T18:12:00.021188+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:02:23.099071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.006010+0000","last_change":"2024-01-15T18:12:01.024249+0000","last_active":"2024-01-15T18:12:02.006010+0000","last_peered":"2024-01-15T18:12:02.006010+0000","last_clean":"2024-01-15T18:12:02.006010+0000","last_became_active":"2024-01-15T18:12:01.023862+0000","last_became_peered":"2024-01-15T18:12:01.023862+0000","last_unstale":"2024-01-15T18:12:02.006010+0000","last_undegraded":"2024-01-15T18:12:02.006010+0000","last_fullsized":"2024-01-15T18:12:02.006010+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:18:47.967166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.016071+0000","last_change":"2024-01-15T18:12:00.023511+0000","last_active":"2024-01-15T18:12:01.016071+0000","last_peered":"2024-01-15T18:12:01.016071+0000","last_clean":"2024-01-15T18:12:01.016071+0000","last_became_active":"2024-01-15T18:12:00.023145+0000","last_became_peered":"2024-01-15T18:12:00.023145+0000","last_unstale":"2024-01-15T18:12:01.016071+0000","last_undegraded":"2024-01-15T18:12:01.016071+0000","last_fullsized":"2024-01-15T18:12:01.016071+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:27:30.560351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.016823+0000","last_change":"2024-01-15T18:12:00.016823+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.016823+0000","last_undegraded":"2024-01-15T18:12:00.016823+0000","last_fullsized":"2024-01-15T18:12:00.016823+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.025706+0000","last_change":"2024-01-15T18:12:00.023014+0000","last_active":"2024-01-15T18:12:00.025706+0000","last_peered":"2024-01-15T18:12:00.025706+0000","last_clean":"2024-01-15T18:12:00.025706+0000","last_became_active":"2024-01-15T18:12:00.022748+0000","last_became_peered":"2024-01-15T18:12:00.022748+0000","last_unstale":"2024-01-15T18:12:00.025706+0000","last_undegraded":"2024-01-15T18:12:00.025706+0000","last_fullsized":"2024-01-15T18:12:00.025706+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:13:33.148973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.014451+0000","last_change":"2024-01-15T18:12:00.014451+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.014451+0000","last_undegraded":"2024-01-15T18:12:00.014451+0000","last_fullsized":"2024-01-15T18:12:00.014451+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-15T18:11:58.997807+0000","last_change":"2024-01-15T18:11:58.997807+0000","last_active":"2024-01-15T18:11:58.982718+0000","last_peered":"2024-01-15T18:11:58.982718+0000","last_clean":"2024-01-15T18:11:58.982718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:58.997807+0000","last_undegraded":"2024-01-15T18:11:58.997807+0000","last_fullsized":"2024-01-15T18:11:58.997807+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.003849+0000","last_change":"2024-01-15T18:12:00.003849+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.003849+0000","last_undegraded":"2024-01-15T18:12:00.003849+0000","last_fullsized":"2024-01-15T18:12:00.003849+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.021303+0000","last_change":"2024-01-15T18:12:00.019406+0000","last_active":"2024-01-15T18:12:00.021303+0000","last_peered":"2024-01-15T18:12:00.021303+0000","last_clean":"2024-01-15T18:12:00.021303+0000","last_became_active":"2024-01-15T18:12:00.019122+0000","last_became_peered":"2024-01-15T18:12:00.019122+0000","last_unstale":"2024-01-15T18:12:00.021303+0000","last_undegraded":"2024-01-15T18:12:00.021303+0000","last_fullsized":"2024-01-15T18:12:00.021303+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:51:33.138526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.007471+0000","last_change":"2024-01-15T18:12:00.007471+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.007471+0000","last_undegraded":"2024-01-15T18:12:00.007471+0000","last_fullsized":"2024-01-15T18:12:00.007471+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.018689+0000","last_change":"2024-01-15T18:12:00.015364+0000","last_active":"2024-01-15T18:12:00.018689+0000","last_peered":"2024-01-15T18:12:00.018689+0000","last_clean":"2024-01-15T18:12:00.018689+0000","last_became_active":"2024-01-15T18:12:00.014208+0000","last_became_peered":"2024-01-15T18:12:00.014208+0000","last_unstale":"2024-01-15T18:12:00.018689+0000","last_undegraded":"2024-01-15T18:12:00.018689+0000","last_fullsized":"2024-01-15T18:12:00.018689+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:59:42.431920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004040+0000","last_change":"2024-01-15T18:12:00.004040+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004040+0000","last_undegraded":"2024-01-15T18:12:00.004040+0000","last_fullsized":"2024-01-15T18:12:00.004040+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.027927+0000","last_change":"2024-01-15T18:12:00.025623+0000","last_active":"2024-01-15T18:12:00.027927+0000","last_peered":"2024-01-15T18:12:00.027927+0000","last_clean":"2024-01-15T18:12:00.027927+0000","last_became_active":"2024-01-15T18:12:00.025294+0000","last_became_peered":"2024-01-15T18:12:00.025294+0000","last_unstale":"2024-01-15T18:12:00.027927+0000","last_undegraded":"2024-01-15T18:12:00.027927+0000","last_fullsized":"2024-01-15T18:12:00.027927+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:02:43.133788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.007044+0000","last_change":"2024-01-15T18:12:00.007044+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.007044+0000","last_undegraded":"2024-01-15T18:12:00.007044+0000","last_fullsized":"2024-01-15T18:12:00.007044+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.023758+0000","last_change":"2024-01-15T18:12:00.019753+0000","last_active":"2024-01-15T18:12:00.023758+0000","last_peered":"2024-01-15T18:12:00.023758+0000","last_clean":"2024-01-15T18:12:00.023758+0000","last_became_active":"2024-01-15T18:12:00.019482+0000","last_became_peered":"2024-01-15T18:12:00.019482+0000","last_unstale":"2024-01-15T18:12:00.023758+0000","last_undegraded":"2024-01-15T18:12:00.023758+0000","last_fullsized":"2024-01-15T18:12:00.023758+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:53:26.480802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.009189+0000","last_change":"2024-01-15T18:12:00.009189+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.009189+0000","last_undegraded":"2024-01-15T18:12:00.009189+0000","last_fullsized":"2024-01-15T18:12:00.009189+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.022076+0000","last_change":"2024-01-15T18:12:00.019108+0000","last_active":"2024-01-15T18:12:00.022076+0000","last_peered":"2024-01-15T18:12:00.022076+0000","last_clean":"2024-01-15T18:12:00.022076+0000","last_became_active":"2024-01-15T18:12:00.018827+0000","last_became_peered":"2024-01-15T18:12:00.018827+0000","last_unstale":"2024-01-15T18:12:00.022076+0000","last_undegraded":"2024-01-15T18:12:00.022076+0000","last_fullsized":"2024-01-15T18:12:00.022076+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:48:57.061505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":203,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-01-15T18:11:58.994013+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:11:58.994013+0000","last_peered":"2024-01-15T18:11:58.994013+0000","last_clean":"2024-01-15T18:11:58.994013+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:11:58.994013+0000","last_undegraded":"2024-01-15T18:11:58.994013+0000","last_fullsized":"2024-01-15T18:11:58.994013+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":116,"acting":116,"num_store_stats":7},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509530,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50596,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321244,"statfs":{"total":96636764160,"available":96584953856,"internally_reserved":0,"allocated":1400832,"data_stored":928722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.1200000000000001,"15min":0.78900000000000003},"min":{"1min":0.66000000000000003,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":9.6989999999999998,"5min":10.284000000000001,"15min":10.284000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":2.4159999999999999,"5min":1.2290000000000001,"15min":0.86299999999999999},"min":{"1min":0.39100000000000001,"5min":0.378,"15min":0.378},"max":{"1min":16.968,"5min":16.968,"15min":16.968},"last":0.56599999999999995}]},{"osd":1,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.7749999999999999,"5min":1.1830000000000001,"15min":0.83899999999999997},"min":{"1min":0.64800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":15.791,"5min":15.791,"15min":15.791},"last":0.54300000000000004},{"interface":"front","average":{"1min":2.1339999999999999,"5min":1.1779999999999999,"15min":0.85899999999999999},"min":{"1min":0.44600000000000001,"5min":0.378,"15min":0.378},"max":{"1min":10.842000000000001,"5min":10.842000000000001,"15min":10.842000000000001},"last":0.64800000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1,"15min":0.76400000000000001},"min":{"1min":0.42299999999999999,"5min":0.378,"15min":0.378},"max":{"1min":4.9029999999999996,"5min":10.119999999999999,"15min":10.119999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":2.0579999999999998,"5min":1.081,"15min":0.83499999999999996},"min":{"1min":0.66800000000000004,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":9.7539999999999996,"5min":9.7539999999999996,"15min":9.7539999999999996},"last":0.75900000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":1.802,"5min":1.143,"15min":0.82299999999999995},"min":{"1min":0.47099999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":11.481,"5min":11.481,"15min":11.481},"last":0.69299999999999995},{"interface":"front","average":{"1min":2.3220000000000001,"5min":1.2310000000000001,"15min":0.91900000000000004},"min":{"1min":0.70199999999999996,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":9.7959999999999994,"5min":10.34,"15min":10.34},"last":0.67200000000000004}]},{"osd":4,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.5630000000000002,"5min":1.3360000000000001,"15min":0.85799999999999998},"min":{"1min":0.41099999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":20.387,"5min":20.387,"15min":20.387},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.649,"5min":1.131,"15min":0.81999999999999995},"min":{"1min":0.46200000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":11.305,"5min":14.871,"15min":14.871},"last":0.623}]},{"osd":5,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.2719999999999998,"5min":1.2070000000000001,"15min":0.86899999999999999},"min":{"1min":0.47799999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":9.8450000000000006,"5min":19.959,"15min":19.959},"last":0.59099999999999997},{"interface":"front","average":{"1min":2.6240000000000001,"5min":1.3009999999999999,"15min":0.92000000000000004},"min":{"1min":0.48299999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":19.905000000000001,"5min":19.905000000000001,"15min":19.905000000000001},"last":0.83199999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":3.157,"5min":1.667,"15min":1.093},"min":{"1min":0.63800000000000001,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":20.492000000000001,"5min":22.478999999999999,"15min":22.478999999999999},"last":0.877},{"interface":"front","average":{"1min":1.397,"5min":1.3049999999999999,"15min":0.92500000000000004},"min":{"1min":0.45100000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.2549999999999999,"5min":21.509,"15min":21.509},"last":0.73899999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509530,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45856,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325984,"statfs":{"total":96636764160,"available":96589807616,"internally_reserved":0,"allocated":806912,"data_stored":338354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":1.157,"15min":0.872},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.91800000000000004,"5min":12.41,"15min":12.41},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.69199999999999995,"5min":1.0009999999999999,"15min":0.79000000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.93500000000000005,"5min":6.7759999999999998,"15min":6.7759999999999998},"last":0.57399999999999995}]},{"osd":1,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":1.454,"15min":0.96399999999999997},"min":{"1min":0.46999999999999997,"5min":0.41899999999999998,"15min":0.41299999999999998},"max":{"1min":0.83199999999999996,"5min":53.451999999999998,"15min":53.451999999999998},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.71299999999999997,"5min":1.008,"15min":0.83699999999999997},"min":{"1min":0.56100000000000005,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.92900000000000005,"5min":7.7329999999999997,"15min":7.7329999999999997},"last":0.56200000000000006}]},{"osd":2,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":1.004,"15min":0.89400000000000002},"min":{"1min":0.46700000000000003,"5min":0.45900000000000002,"15min":0.38900000000000001},"max":{"1min":1.1060000000000001,"5min":7.9409999999999998,"15min":7.9409999999999998},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.93700000000000006,"15min":0.82999999999999996},"min":{"1min":0.57099999999999995,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.90800000000000003,"5min":5.1760000000000002,"15min":5.1760000000000002},"last":0.75800000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":1.038,"15min":0.876},"min":{"1min":0.46000000000000002,"5min":0.42099999999999999,"15min":0.41399999999999998},"max":{"1min":0.94899999999999995,"5min":6.758,"15min":6.758},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.70499999999999996,"5min":1.5600000000000001,"15min":1.0660000000000001},"min":{"1min":0.49199999999999999,"5min":0.45200000000000001,"15min":0.41599999999999998},"max":{"1min":0.877,"5min":53.398000000000003,"15min":53.398000000000003},"last":0.80600000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":1.0029999999999999,"15min":0.79500000000000004},"min":{"1min":0.38500000000000001,"5min":0.378,"15min":0.34999999999999998},"max":{"1min":0.88,"5min":8.6259999999999994,"15min":8.6259999999999994},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.94899999999999995,"15min":0.79100000000000004},"min":{"1min":0.42599999999999999,"5min":0.38200000000000001,"15min":0.34499999999999997},"max":{"1min":0.85099999999999998,"5min":6.9589999999999996,"15min":6.9589999999999996},"last":0.63}]},{"osd":5,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":1.641,"15min":0.98299999999999998},"min":{"1min":0.41299999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.96399999999999997,"5min":53.229999999999997,"15min":53.229999999999997},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.69299999999999995,"5min":1.228,"15min":0.92900000000000005},"min":{"1min":0.45500000000000002,"5min":0.435,"15min":0.435},"max":{"1min":1.149,"5min":10.029999999999999,"15min":10.029999999999999},"last":0.745}]},{"osd":7,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":1.5760000000000001,"15min":1.071},"min":{"1min":0.437,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.90400000000000003,"5min":45.070999999999998,"15min":45.070999999999998},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.70099999999999996,"5min":1.6910000000000001,"15min":1.0940000000000001},"min":{"1min":0.52800000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.92200000000000004,"5min":53.338000000000001,"15min":53.338000000000001},"last":0.77300000000000002}]}]},{"osd":4,"up_from":15,"seq":64424509530,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50028,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321812,"statfs":{"total":96636764160,"available":96585535488,"internally_reserved":0,"allocated":819200,"data_stored":344685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":1.2,"15min":0.81899999999999995},"min":{"1min":0.502,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.2509999999999999,"5min":23.318000000000001,"15min":23.318000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.70699999999999996,"5min":1.046,"15min":0.77500000000000002},"min":{"1min":0.442,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.97899999999999998,"5min":14.723000000000001,"15min":14.723000000000001},"last":0.442}]},{"osd":1,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.89500000000000002,"15min":0.72999999999999998},"min":{"1min":0.442,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.876,"5min":7.8200000000000003,"15min":7.8200000000000003},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.73899999999999999,"5min":1.1879999999999999,"15min":0.82299999999999995},"min":{"1min":0.47299999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.613,"5min":22.408999999999999,"15min":22.408999999999999},"last":0.58999999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.91800000000000004,"15min":0.72399999999999998},"min":{"1min":0.48699999999999999,"5min":0.40600000000000003,"15min":0.34399999999999997},"max":{"1min":1.0009999999999999,"5min":8.8529999999999998,"15min":8.8529999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.77900000000000003,"5min":1.288,"15min":0.89200000000000002},"min":{"1min":0.55500000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.2969999999999999,"5min":22.338999999999999,"15min":22.338999999999999},"last":0.65800000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":1.149,"15min":0.84299999999999997},"min":{"1min":0.52500000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.3939999999999999,"5min":14.763999999999999,"15min":14.763999999999999},"last":0.622},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0509999999999999,"15min":0.83399999999999996},"min":{"1min":0.59899999999999998,"5min":0.504,"15min":0.45500000000000002},"max":{"1min":1.7669999999999999,"5min":8.9459999999999997,"15min":8.9459999999999997},"last":0.67300000000000004}]},{"osd":5,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":1.2609999999999999,"15min":0.86399999999999999},"min":{"1min":0.39800000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.4670000000000001,"5min":31.617999999999999,"15min":31.617999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":1.1910000000000001,"15min":0.83099999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.377},"max":{"1min":1.022,"5min":17.145,"15min":17.145},"last":0.55100000000000005}]},{"osd":6,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.95299999999999996,"15min":0.77100000000000002},"min":{"1min":0.46500000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.89300000000000002,"5min":11.124000000000001,"15min":11.124000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.749,"5min":1.7629999999999999,"15min":1.0780000000000001},"min":{"1min":0.45600000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.339,"5min":28.872,"15min":28.872},"last":0.76700000000000002}]},{"osd":7,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":1.5409999999999999,"15min":0.99299999999999999},"min":{"1min":0.45800000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.3169999999999999,"5min":39.256,"15min":39.256},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":1.1160000000000001,"15min":0.89200000000000002},"min":{"1min":0.51900000000000002,"5min":0.46500000000000002,"15min":0.42199999999999999},"max":{"1min":1.0329999999999999,"5min":26.573,"15min":26.573},"last":0.72899999999999998}]}]},{"osd":3,"up_from":15,"seq":64424509529,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50016,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321824,"statfs":{"total":96636764160,"available":96585547776,"internally_reserved":0,"allocated":806912,"data_stored":338354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":0.58899999999999997,"15min":0.47699999999999998},"min":{"1min":0.29799999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":5.8849999999999998,"5min":5.8849999999999998,"15min":5.8849999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.1299999999999999,"5min":0.81999999999999995,"15min":0.58699999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":4.7990000000000004,"5min":17.756,"15min":17.756},"last":0.44400000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":0.755,"15min":0.59099999999999997},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":12.307,"5min":12.307,"15min":12.307},"last":0.56999999999999995},{"interface":"front","average":{"1min":1.1080000000000001,"5min":0.66300000000000003,"15min":0.57799999999999996},"min":{"1min":0.34899999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":6.0640000000000001,"5min":6.0640000000000001,"15min":6.0640000000000001},"last":0.51400000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":0.84999999999999998,"15min":0.65800000000000003},"min":{"1min":0.36099999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":12.468999999999999,"5min":12.468999999999999,"15min":12.468999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.2549999999999999,"5min":0.77400000000000002,"15min":0.63400000000000001},"min":{"1min":0.41599999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":6.0330000000000004,"5min":6.0330000000000004,"15min":6.0330000000000004},"last":0.46899999999999997}]},{"osd":4,"last update":"Mon Jan 15 18:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.141,"15min":0.91900000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":6.3970000000000002,"5min":17.853999999999999,"15min":17.853999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":2.1240000000000001,"5min":1.115,"15min":0.89300000000000002},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":13.31,"5min":13.31,"15min":13.31},"last":0.67000000000000004}]},{"osd":5,"last update":"Mon Jan 15 18:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":1.0109999999999999,"15min":0.83099999999999996},"min":{"1min":0.52100000000000002,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":6.1139999999999999,"5min":6.1139999999999999,"15min":6.1139999999999999},"last":0.81899999999999995},{"interface":"front","average":{"1min":2.1720000000000002,"5min":1.284,"15min":0.90800000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":20.515999999999998,"5min":20.515999999999998,"15min":20.515999999999998},"last":0.83199999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:10:58 2024","interfaces":[{"interface":"back","average":{"1min":2.2879999999999998,"5min":1.333,"15min":0.96899999999999997},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":13.250999999999999,"5min":23.100999999999999,"15min":23.100999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":2.246,"5min":1.1339999999999999,"15min":0.88600000000000001},"min":{"1min":0.5,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":12.76,"5min":12.76,"15min":12.76},"last":0.86199999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:10:58 2024","interfaces":[{"interface":"back","average":{"1min":2.9430000000000001,"5min":1.3819999999999999,"15min":0.97899999999999998},"min":{"1min":0.48599999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":21.545000000000002,"5min":21.545000000000002,"15min":21.545000000000002},"last":0.70199999999999996},{"interface":"front","average":{"1min":2.7650000000000001,"5min":1.3200000000000001,"15min":0.98899999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":13.385,"5min":13.385,"15min":13.385},"last":0.80500000000000005}]}]},{"osd":2,"up_from":15,"seq":64424509529,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45872,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325968,"statfs":{"total":96636764160,"available":96589791232,"internally_reserved":0,"allocated":819200,"data_stored":344685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.61699999999999999,"15min":0.51200000000000001},"min":{"1min":0.29399999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.8480000000000001,"5min":5.1849999999999996,"15min":5.1849999999999996},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.64200000000000002,"15min":0.52300000000000002},"min":{"1min":0.32600000000000001,"5min":0.29399999999999998,"15min":0.28299999999999997},"max":{"1min":2.3759999999999999,"5min":7.516,"15min":7.516},"last":0.50700000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.748,"15min":0.61199999999999999},"min":{"1min":0.34100000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":2.2850000000000001,"5min":10.907,"15min":10.907},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.81599999999999995,"15min":0.57399999999999995},"min":{"1min":0.40500000000000003,"5min":0.311,"15min":0.27000000000000002},"max":{"1min":1.7150000000000001,"5min":8.9190000000000005,"15min":8.9190000000000005},"last":0.68999999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.73599999999999999,"15min":0.628},"min":{"1min":0.36599999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.86399999999999999,"5min":12.185,"15min":12.185},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.83499999999999996,"15min":0.64700000000000002},"min":{"1min":0.443,"5min":0.39100000000000001,"15min":0.34000000000000002},"max":{"1min":2.3330000000000002,"5min":7.6760000000000002,"15min":7.6760000000000002},"last":0.63500000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.99299999999999999,"15min":0.82699999999999996},"min":{"1min":0.58699999999999997,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":2.21,"5min":8.6609999999999996,"15min":8.6609999999999996},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.76000000000000001,"5min":1.1899999999999999,"15min":0.90600000000000003},"min":{"1min":0.54000000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.6339999999999999,"5min":19.146999999999998,"15min":19.146999999999998},"last":0.77500000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":1.1319999999999999,"15min":0.90800000000000003},"min":{"1min":0.442,"5min":0.44,"15min":0.44},"max":{"1min":0.90800000000000003,"5min":31.710000000000001,"15min":31.710000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.875,"5min":1.373,"15min":0.98299999999999998},"min":{"1min":0.59399999999999997,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.1669999999999998,"5min":22.539000000000001,"15min":22.539000000000001},"last":0.871}]},{"osd":6,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":1.107,"15min":0.88600000000000001},"min":{"1min":0.61699999999999999,"5min":0.50800000000000001,"15min":0.48299999999999998},"max":{"1min":1.675,"5min":9.4540000000000006,"15min":9.4540000000000006},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.1399999999999999,"15min":0.90900000000000003},"min":{"1min":0.53100000000000003,"5min":0.503,"15min":0.503},"max":{"1min":1,"5min":18.614999999999998,"15min":18.614999999999998},"last":0.74199999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":1.113,"15min":0.90400000000000003},"min":{"1min":0.58199999999999996,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.752,"5min":11.432,"15min":11.432},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.264,"15min":0.97099999999999997},"min":{"1min":0.54500000000000004,"5min":0.502,"15min":0.502},"max":{"1min":2.2480000000000002,"5min":17.013000000000002,"15min":17.013000000000002},"last":0.90600000000000003}]}]},{"osd":1,"up_from":15,"seq":64424509531,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50032,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321808,"statfs":{"total":96636764160,"available":96585531392,"internally_reserved":0,"allocated":819200,"data_stored":344685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.96399999999999997,"15min":0.65500000000000003},"min":{"1min":0.309,"5min":0.26500000000000001,"15min":0.254},"max":{"1min":0.72899999999999998,"5min":19.577000000000002,"15min":19.577000000000002},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":1.1779999999999999,"15min":0.71499999999999997},"min":{"1min":0.35399999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.80000000000000004,"5min":48.277999999999999,"15min":48.277999999999999},"last":0.42899999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":1.036,"15min":0.70599999999999996},"min":{"1min":0.35599999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.65000000000000002,"5min":40.698,"15min":40.698},"last":0.379},{"interface":"front","average":{"1min":0.53200000000000003,"5min":1.337,"15min":0.92900000000000005},"min":{"1min":0.36899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93300000000000005,"5min":48.316000000000003,"15min":48.316000000000003},"last":0.35399999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.2370000000000001,"15min":0.83999999999999997},"min":{"1min":0.46600000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.88200000000000001,"5min":48.234999999999999,"15min":48.234999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.75800000000000001,"15min":0.64500000000000002},"min":{"1min":0.36099999999999999,"5min":0.34699999999999998,"15min":0.311},"max":{"1min":0.74399999999999999,"5min":6.468,"15min":6.468},"last":0.69299999999999995}]},{"osd":4,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":1.0589999999999999,"15min":0.81799999999999995},"min":{"1min":0.48999999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.0720000000000001,"5min":8.3409999999999993,"15min":8.3409999999999993},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.80200000000000005,"5min":1.165,"15min":0.90600000000000003},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.42499999999999999},"max":{"1min":1.278,"5min":19.829000000000001,"15min":19.829000000000001},"last":0.52200000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":1.4399999999999999,"15min":0.99399999999999999},"min":{"1min":0.59499999999999997,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.306,"5min":22.594999999999999,"15min":22.594999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.79400000000000004,"5min":1.8939999999999999,"15min":1.145},"min":{"1min":0.54500000000000004,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1180000000000001,"5min":48.198,"15min":48.198},"last":0.57699999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":1.4590000000000001,"15min":0.98899999999999999},"min":{"1min":0.59899999999999998,"5min":0.53000000000000003,"15min":0.47499999999999998},"max":{"1min":1.1499999999999999,"5min":35.112000000000002,"15min":35.112000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.81200000000000006,"5min":1.1779999999999999,"15min":0.93999999999999995},"min":{"1min":0.53100000000000003,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.175,"5min":19.780999999999999,"15min":19.780999999999999},"last":0.59999999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":1.8069999999999999,"15min":1.1259999999999999},"min":{"1min":0.628,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2450000000000001,"5min":48.148000000000003,"15min":48.148000000000003},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.80300000000000005,"5min":1.194,"15min":0.93300000000000005},"min":{"1min":0.64000000000000001,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.214,"5min":9.3840000000000003,"15min":9.3840000000000003},"last":0.84199999999999997}]}]},{"osd":5,"up_from":14,"seq":60129542235,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50064,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321776,"statfs":{"total":96636764160,"available":96585498624,"internally_reserved":0,"allocated":847872,"data_stored":357962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.872,"15min":0.77600000000000002},"min":{"1min":0.45900000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.3019999999999996,"5min":21.385000000000002,"15min":21.385000000000002},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.732,"5min":1.071,"15min":0.88800000000000001},"min":{"1min":0.46999999999999997,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":9.359,"5min":9.359,"15min":9.359},"last":0.78200000000000003}]},{"osd":1,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":3.0960000000000001,"5min":1.1839999999999999,"15min":0.84599999999999997},"min":{"1min":0.48799999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":21.120000000000001,"5min":21.120000000000001,"15min":21.120000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0329999999999999,"15min":0.80100000000000005},"min":{"1min":0.52400000000000002,"5min":0.315,"15min":0.315},"max":{"1min":4.3730000000000002,"5min":21.5,"15min":21.5},"last":0.753}]},{"osd":2,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.1419999999999999,"15min":0.85399999999999998},"min":{"1min":0.56399999999999995,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":6.2750000000000004,"5min":8.7750000000000004,"15min":8.7750000000000004},"last":0.73799999999999999},{"interface":"front","average":{"1min":2.9780000000000002,"5min":1.2869999999999999,"15min":0.94999999999999996},"min":{"1min":0.58099999999999996,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":21.201000000000001,"5min":21.201000000000001,"15min":21.201000000000001},"last":0.76600000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.0069999999999999,"15min":0.875},"min":{"1min":0.57399999999999995,"5min":0.44,"15min":0.44},"max":{"1min":4.4100000000000001,"5min":21.457000000000001,"15min":21.457000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.583,"5min":1.196,"15min":0.96799999999999997},"min":{"1min":0.54200000000000004,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":6.8650000000000002,"5min":8.7300000000000004,"15min":8.7300000000000004},"last":0.64900000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.012,"15min":0.82799999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":5.8929999999999998,"5min":9.6679999999999993,"15min":9.6679999999999993},"last":0.55700000000000005},{"interface":"front","average":{"1min":3.8380000000000001,"5min":1.5329999999999999,"15min":1.008},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":21.724,"5min":26.146999999999998,"15min":26.146999999999998},"last":0.70099999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.91900000000000004,"15min":0.79800000000000004},"min":{"1min":0.36099999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.835,"5min":21.535,"15min":21.535},"last":0.67600000000000005},{"interface":"front","average":{"1min":1.6160000000000001,"5min":1.3839999999999999,"15min":0.999},"min":{"1min":0.40100000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":5.7539999999999996,"5min":30.488,"15min":30.488},"last":0.82099999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":4.7309999999999999,"5min":1.6259999999999999,"15min":1.0760000000000001},"min":{"1min":0.42599999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":30.483000000000001,"5min":30.483000000000001,"15min":30.483000000000001},"last":0.629},{"interface":"front","average":{"1min":3.508,"5min":1.4650000000000001,"15min":1.0429999999999999},"min":{"1min":0.42899999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":25.437999999999999,"5min":25.437999999999999,"15min":25.437999999999999},"last":0.72199999999999998}]}]},{"osd":0,"up_from":14,"seq":60129542234,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54744,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317096,"statfs":{"total":96636764160,"available":96580706304,"internally_reserved":0,"allocated":1388544,"data_stored":922849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.89300000000000002,"15min":0.60699999999999998},"min":{"1min":0.26000000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9179999999999999,"5min":19.843,"15min":19.843},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.88700000000000001,"15min":0.66400000000000003},"min":{"1min":0.29199999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73299999999999998,"5min":10.154,"15min":10.154},"last":0.42999999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":1.115,"15min":0.753},"min":{"1min":0.38800000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.754,"5min":25.905000000000001,"15min":25.905000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.5,"5min":0.77900000000000003,"15min":0.58199999999999996},"min":{"1min":0.29699999999999999,"5min":0.28599999999999998,"15min":0.26700000000000002},"max":{"1min":1.2549999999999999,"5min":9.3829999999999991,"15min":9.3829999999999991},"last":0.38800000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.95699999999999996,"15min":0.70299999999999996},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.308},"max":{"1min":1.952,"5min":9.1780000000000008,"15min":9.1780000000000008},"last":0.496},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.97399999999999998,"15min":0.73099999999999998},"min":{"1min":0.42899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":2.9809999999999999,"5min":10.210000000000001,"15min":10.210000000000001},"last":0.51200000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":1.343,"15min":0.96399999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.38},"max":{"1min":1.306,"5min":22.814,"15min":22.814},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.446,"15min":1.03},"min":{"1min":0.49099999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":2.169,"5min":23.341999999999999,"15min":23.341999999999999},"last":0.73299999999999998}]},{"osd":5,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":1.262,"15min":0.94899999999999995},"min":{"1min":0.46899999999999997,"5min":0.44600000000000001,"15min":0.441},"max":{"1min":1.365,"5min":12.428000000000001,"15min":12.428000000000001},"last":0.874},{"interface":"front","average":{"1min":0.82499999999999996,"5min":1.6930000000000001,"15min":1.1259999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.46300000000000002},"max":{"1min":2.0880000000000001,"5min":19.774000000000001,"15min":19.774000000000001},"last":0.78600000000000003}]},{"osd":6,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":1.9219999999999999,"15min":1.1719999999999999},"min":{"1min":0.46700000000000003,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.0449999999999999,"5min":31.059000000000001,"15min":31.059000000000001},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.83299999999999996,"5min":1.488,"15min":1.0580000000000001},"min":{"1min":0.57999999999999996,"5min":0.51500000000000001,"15min":0.49099999999999999},"max":{"1min":1.3899999999999999,"5min":40.731000000000002,"15min":40.731000000000002},"last":0.80800000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":1.673,"15min":1.1100000000000001},"min":{"1min":0.61599999999999999,"5min":0.47199999999999998,"15min":0.42499999999999999},"max":{"1min":1.4239999999999999,"5min":24.492999999999999,"15min":24.492999999999999},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.875,"5min":1.5780000000000001,"15min":1.097},"min":{"1min":0.61199999999999999,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":2.2130000000000001,"5min":23.524000000000001,"15min":23.524000000000001},"last":0.82599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:12:05.555 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:12:05.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.699+0000 7fd010200640 1 Processor -- start 2024-01-15T18:12:05.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.699+0000 7fd010200640 1 -- start start 2024-01-15T18:12:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.699+0000 7fd010200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fd0080701a0 0x7fd008070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.699+0000 7fd010200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd00815c130 0x7fd00815d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.699+0000 7fd010200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd00806c4a0 con 0x7fd0080c92b0 2024-01-15T18:12:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.699+0000 7fd010200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd008070ac0 con 0x7fd00815c130 2024-01-15T18:12:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.699+0000 7fd010200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd00815c660 con 0x7fd0080701a0 2024-01-15T18:12:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.699+0000 7fd00e200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd00815c130 0x7fd00815d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.699+0000 7fd00e200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd00815c130 0x7fd00815d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58302/0 (socket says 172.21.15.27:58302) 2024-01-15T18:12:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.699+0000 7fd00ce00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd0080c92b0 0x7fd00806dd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59432/0 (socket says 172.21.15.27:59432) 2024-01-15T18:12:05.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.699+0000 7fd00e200640 1 -- 172.21.15.27:0/192645383 learned_addr learned my addr 172.21.15.27:0/192645383 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.699+0000 7fd00d800640 1 --2- 172.21.15.27:0/192645383 >> v2:172.21.15.93:3300/0 conn(0x7fd0080701a0 0x7fd008070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.699+0000 7fd00e200640 1 -- 172.21.15.27:0/192645383 >> v2:172.21.15.93:3300/0 conn(0x7fd0080701a0 msgr2=0x7fd008070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.699+0000 7fd00e200640 1 --2- 172.21.15.27:0/192645383 >> v2:172.21.15.93:3300/0 conn(0x7fd0080701a0 0x7fd008070580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:05.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.699+0000 7fd00e200640 1 -- 172.21.15.27:0/192645383 >> v1:172.21.15.27:6789/0 conn(0x7fd0080c92b0 legacy=0x7fd00806dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:05.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.699+0000 7fd00e200640 1 -- 172.21.15.27:0/192645383 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd00815c940 con 0x7fd00815c130 2024-01-15T18:12:05.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.699+0000 7fd007e00640 1 -- 172.21.15.27:0/192645383 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 693706350 0 0) 0x7fd00806c4a0 con 0x7fd0080c92b0 2024-01-15T18:12:05.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.703+0000 7fd00e200640 1 --2- 172.21.15.27:0/192645383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00815c130 0x7fd00815d9e0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fd0080681f0 tx=0x7fd00001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65ab5f1c9536c445 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:05.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.703+0000 7fd007e00640 1 -- 172.21.15.27:0/192645383 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0000a5860 con 0x7fd00815c130 2024-01-15T18:12:05.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.703+0000 7fd007e00640 1 -- 172.21.15.27:0/192645383 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0000a59c0 con 0x7fd00815c130 2024-01-15T18:12:05.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.703+0000 7fd007e00640 1 -- 172.21.15.27:0/192645383 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0000a5d50 con 0x7fd00815c130 2024-01-15T18:12:05.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.703+0000 7fd010200640 1 -- 172.21.15.27:0/192645383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00815c130 msgr2=0x7fd00815d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:05.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.703+0000 7fd010200640 1 --2- 172.21.15.27:0/192645383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00815c130 0x7fd00815d9e0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fd0080681f0 tx=0x7fd00001c9a0 comp rx=0 tx=0).stop 2024-01-15T18:12:05.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.703+0000 7fd010200640 1 -- 172.21.15.27:0/192645383 shutdown_connections 2024-01-15T18:12:05.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.703+0000 7fd010200640 1 --2- 172.21.15.27:0/192645383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00815c130 0x7fd00815d9e0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:05.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.703+0000 7fd010200640 1 --2- 172.21.15.27:0/192645383 >> v2:172.21.15.93:3300/0 conn(0x7fd0080701a0 0x7fd008070580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:05.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.703+0000 7fd010200640 1 -- 172.21.15.27:0/192645383 >> 172.21.15.27:0/192645383 conn(0x7fd0080cec60 msgr2=0x7fd0080cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:05.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.703+0000 7fd010200640 1 -- 172.21.15.27:0/192645383 shutdown_connections 2024-01-15T18:12:05.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.703+0000 7fd010200640 1 -- 172.21.15.27:0/192645383 wait complete. 2024-01-15T18:12:05.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd010200640 1 Processor -- start 2024-01-15T18:12:05.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd010200640 1 -- start start 2024-01-15T18:12:05.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd010200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0080701a0 0x7fd008156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:05.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd010200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd0080c92b0 0x7fd0080c5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:05.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd010200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd00815c130 0x7fd0080c5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:05.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd010200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0080c8700 con 0x7fd0080701a0 2024-01-15T18:12:05.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd010200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd008157310 con 0x7fd0080c92b0 2024-01-15T18:12:05.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd010200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0080c80f0 con 0x7fd00815c130 2024-01-15T18:12:05.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd00d800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0080701a0 0x7fd008156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:05.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd00d800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0080701a0 0x7fd008156a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58316/0 (socket says 172.21.15.27:58316) 2024-01-15T18:12:05.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd00d800640 1 -- 172.21.15.27:0/4060145714 learned_addr learned my addr 172.21.15.27:0/4060145714 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:05.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd00ce00640 1 --2- 172.21.15.27:0/4060145714 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd0080c92b0 0x7fd0080c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:05.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd00e200640 1 --2- 172.21.15.27:0/4060145714 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd00815c130 0x7fd0080c5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:05.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd00d800640 1 -- 172.21.15.27:0/4060145714 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd00815c130 msgr2=0x7fd0080c5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:05.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd00d800640 1 --2- 172.21.15.27:0/4060145714 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd00815c130 0x7fd0080c5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:05.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd00d800640 1 -- 172.21.15.27:0/4060145714 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd0080c92b0 msgr2=0x7fd0080c5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:05.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd00d800640 1 --2- 172.21.15.27:0/4060145714 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd0080c92b0 0x7fd0080c5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:05.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd00d800640 1 -- 172.21.15.27:0/4060145714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd000009b30 con 0x7fd0080701a0 2024-01-15T18:12:05.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd00ce00640 1 --2- 172.21.15.27:0/4060145714 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd0080c92b0 0x7fd0080c5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:05.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd00e200640 1 --2- 172.21.15.27:0/4060145714 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd00815c130 0x7fd0080c5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:05.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd00d800640 1 --2- 172.21.15.27:0/4060145714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0080701a0 0x7fd008156a90 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fcff800d800 tx=0x7fcff800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:05.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd005600640 1 -- 172.21.15.27:0/4060145714 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff800b6c0 con 0x7fd0080701a0 2024-01-15T18:12:05.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd005600640 1 -- 172.21.15.27:0/4060145714 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff800b820 con 0x7fd0080701a0 2024-01-15T18:12:05.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd010200640 1 -- 172.21.15.27:0/4060145714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0080c8b90 con 0x7fd0080701a0 2024-01-15T18:12:05.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd005600640 1 -- 172.21.15.27:0/4060145714 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff800bb90 con 0x7fd0080701a0 2024-01-15T18:12:05.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd010200640 1 -- 172.21.15.27:0/4060145714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0080c8390 con 0x7fd0080701a0 2024-01-15T18:12:05.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.707+0000 7fd010200640 1 -- 172.21.15.27:0/4060145714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0080c7f50 con 0x7fd0080701a0 2024-01-15T18:12:05.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.711+0000 7fd005600640 1 -- 172.21.15.27:0/4060145714 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7fcff800bcf0 con 0x7fd0080701a0 2024-01-15T18:12:05.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.711+0000 7fd005600640 1 --2- 172.21.15.27:0/4060145714 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcfe00670d0 0x7fcfe0069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:05.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.711+0000 7fd005600640 1 -- 172.21.15.27:0/4060145714 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fcff8083080 con 0x7fd0080701a0 2024-01-15T18:12:05.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.711+0000 7fd00ce00640 1 --2- 172.21.15.27:0/4060145714 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcfe00670d0 0x7fcfe0069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:05.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.711+0000 7fd00ce00640 1 --2- 172.21.15.27:0/4060145714 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcfe00670d0 0x7fcfe0069590 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fd0080c6b40 tx=0x7fcff0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:05.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.715+0000 7fd005600640 1 -- 172.21.15.27:0/4060145714 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd0080c7f50 con 0x7fd0080701a0 2024-01-15T18:12:05.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.867+0000 7fd010200640 1 -- 172.21.15.27:0/4060145714 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd00806e410 con 0x7fcfe00670d0 2024-01-15T18:12:05.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.875+0000 7fd005600640 1 -- 172.21.15.27:0/4060145714 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+283262 (secure 0 0 0) 0x7fcff00013e0 con 0x7fcfe00670d0 2024-01-15T18:12:05.881 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:12:05.882 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:12:05.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.879+0000 7fd010200640 1 -- 172.21.15.27:0/4060145714 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcfe00670d0 msgr2=0x7fcfe0069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:05.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.879+0000 7fd010200640 1 --2- 172.21.15.27:0/4060145714 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcfe00670d0 0x7fcfe0069590 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fd0080c6b40 tx=0x7fcff0009040 comp rx=0 tx=0).stop 2024-01-15T18:12:05.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.879+0000 7fd010200640 1 -- 172.21.15.27:0/4060145714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0080701a0 msgr2=0x7fd008156a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:05.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.879+0000 7fd010200640 1 --2- 172.21.15.27:0/4060145714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0080701a0 0x7fd008156a90 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fcff800d800 tx=0x7fcff800dcc0 comp rx=0 tx=0).stop 2024-01-15T18:12:05.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.879+0000 7fd010200640 1 -- 172.21.15.27:0/4060145714 shutdown_connections 2024-01-15T18:12:05.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.879+0000 7fd010200640 1 --2- 172.21.15.27:0/4060145714 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcfe00670d0 0x7fcfe0069590 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:05.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.879+0000 7fd010200640 1 --2- 172.21.15.27:0/4060145714 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd00815c130 0x7fd0080c5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:05.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.879+0000 7fd010200640 1 --2- 172.21.15.27:0/4060145714 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd0080c92b0 0x7fd0080c5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:05.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.879+0000 7fd010200640 1 --2- 172.21.15.27:0/4060145714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0080701a0 0x7fd008156a90 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:05.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.879+0000 7fd010200640 1 -- 172.21.15.27:0/4060145714 >> 172.21.15.27:0/4060145714 conn(0x7fd0080cec60 msgr2=0x7fd008071390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:05.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.879+0000 7fd010200640 1 -- 172.21.15.27:0/4060145714 shutdown_connections 2024-01-15T18:12:05.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:05.879+0000 7fd010200640 1 -- 172.21.15.27:0/4060145714 wait complete. 2024-01-15T18:12:05.903 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2024-01-15T18:12:04.259217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":182,"acting":182,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":397208,"kb_used_data":7528,"kb_used_omap":12,"kb_used_meta":389555,"kb_avail":754577512,"statfs":{"total":773094113280,"available":772687372288,"internally_reserved":0,"allocated":7708672,"data_stored":3920296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":398904880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.270525"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.018152+0000","last_change":"2024-01-15T18:12:00.018152+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.018152+0000","last_undegraded":"2024-01-15T18:12:00.018152+0000","last_fullsized":"2024-01-15T18:12:00.018152+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.010620+0000","last_change":"2024-01-15T18:12:00.010620+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.010620+0000","last_undegraded":"2024-01-15T18:12:00.010620+0000","last_fullsized":"2024-01-15T18:12:00.010620+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.008063+0000","last_change":"2024-01-15T18:12:00.008063+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.008063+0000","last_undegraded":"2024-01-15T18:12:00.008063+0000","last_fullsized":"2024-01-15T18:12:00.008063+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.006826+0000","last_change":"2024-01-15T18:12:00.006826+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.006826+0000","last_undegraded":"2024-01-15T18:12:00.006826+0000","last_fullsized":"2024-01-15T18:12:00.006826+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.003755+0000","last_change":"2024-01-15T18:12:00.003755+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.003755+0000","last_undegraded":"2024-01-15T18:12:00.003755+0000","last_fullsized":"2024-01-15T18:12:00.003755+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.008474+0000","last_change":"2024-01-15T18:12:00.008474+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.008474+0000","last_undegraded":"2024-01-15T18:12:00.008474+0000","last_fullsized":"2024-01-15T18:12:00.008474+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005570+0000","last_change":"2024-01-15T18:12:01.023552+0000","last_active":"2024-01-15T18:12:02.005570+0000","last_peered":"2024-01-15T18:12:02.005570+0000","last_clean":"2024-01-15T18:12:02.005570+0000","last_became_active":"2024-01-15T18:12:01.023332+0000","last_became_peered":"2024-01-15T18:12:01.023332+0000","last_unstale":"2024-01-15T18:12:02.005570+0000","last_undegraded":"2024-01-15T18:12:02.005570+0000","last_fullsized":"2024-01-15T18:12:02.005570+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:13:50.232455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.009793+0000","last_change":"2024-01-15T18:12:00.009793+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.009793+0000","last_undegraded":"2024-01-15T18:12:00.009793+0000","last_fullsized":"2024-01-15T18:12:00.009793+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.005086+0000","last_change":"2024-01-15T18:12:00.005086+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.005086+0000","last_undegraded":"2024-01-15T18:12:00.005086+0000","last_fullsized":"2024-01-15T18:12:00.005086+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.006522+0000","last_change":"2024-01-15T18:12:00.006522+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.006522+0000","last_undegraded":"2024-01-15T18:12:00.006522+0000","last_fullsized":"2024-01-15T18:12:00.006522+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.017853+0000","last_change":"2024-01-15T18:12:00.017853+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.017853+0000","last_undegraded":"2024-01-15T18:12:00.017853+0000","last_fullsized":"2024-01-15T18:12:00.017853+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.006367+0000","last_change":"2024-01-15T18:12:00.006367+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.006367+0000","last_undegraded":"2024-01-15T18:12:00.006367+0000","last_fullsized":"2024-01-15T18:12:00.006367+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:11:59.986005+0000","last_change":"2024-01-15T18:11:59.986005+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:59.986005+0000","last_undegraded":"2024-01-15T18:11:59.986005+0000","last_fullsized":"2024-01-15T18:11:59.986005+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004318+0000","last_change":"2024-01-15T18:12:00.004318+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004318+0000","last_undegraded":"2024-01-15T18:12:00.004318+0000","last_fullsized":"2024-01-15T18:12:00.004318+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:11:59.986005+0000","last_change":"2024-01-15T18:11:59.986005+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:59.986005+0000","last_undegraded":"2024-01-15T18:11:59.986005+0000","last_fullsized":"2024-01-15T18:11:59.986005+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004080+0000","last_change":"2024-01-15T18:12:00.004080+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004080+0000","last_undegraded":"2024-01-15T18:12:00.004080+0000","last_fullsized":"2024-01-15T18:12:00.004080+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004665+0000","last_change":"2024-01-15T18:12:00.004665+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004665+0000","last_undegraded":"2024-01-15T18:12:00.004665+0000","last_fullsized":"2024-01-15T18:12:00.004665+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004833+0000","last_change":"2024-01-15T18:12:00.004833+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004833+0000","last_undegraded":"2024-01-15T18:12:00.004833+0000","last_fullsized":"2024-01-15T18:12:00.004833+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.010019+0000","last_change":"2024-01-15T18:12:00.010019+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.010019+0000","last_undegraded":"2024-01-15T18:12:00.010019+0000","last_fullsized":"2024-01-15T18:12:00.010019+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:11:59.986005+0000","last_change":"2024-01-15T18:11:59.986005+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:59.986005+0000","last_undegraded":"2024-01-15T18:11:59.986005+0000","last_fullsized":"2024-01-15T18:11:59.986005+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004967+0000","last_change":"2024-01-15T18:12:00.004967+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004967+0000","last_undegraded":"2024-01-15T18:12:00.004967+0000","last_fullsized":"2024-01-15T18:12:00.004967+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.005039+0000","last_change":"2024-01-15T18:12:00.005039+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.005039+0000","last_undegraded":"2024-01-15T18:12:00.005039+0000","last_fullsized":"2024-01-15T18:12:00.005039+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.014261+0000","last_change":"2024-01-15T18:12:00.014261+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.014261+0000","last_undegraded":"2024-01-15T18:12:00.014261+0000","last_fullsized":"2024-01-15T18:12:00.014261+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.014771+0000","last_change":"2024-01-15T18:12:00.014771+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.014771+0000","last_undegraded":"2024-01-15T18:12:00.014771+0000","last_fullsized":"2024-01-15T18:12:00.014771+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.005063+0000","last_change":"2024-01-15T18:12:00.005063+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.005063+0000","last_undegraded":"2024-01-15T18:12:00.005063+0000","last_fullsized":"2024-01-15T18:12:00.005063+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005958+0000","last_change":"2024-01-15T18:12:01.026872+0000","last_active":"2024-01-15T18:12:02.005958+0000","last_peered":"2024-01-15T18:12:02.005958+0000","last_clean":"2024-01-15T18:12:02.005958+0000","last_became_active":"2024-01-15T18:12:01.026554+0000","last_became_peered":"2024-01-15T18:12:01.026554+0000","last_unstale":"2024-01-15T18:12:02.005958+0000","last_undegraded":"2024-01-15T18:12:02.005958+0000","last_fullsized":"2024-01-15T18:12:02.005958+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:05:44.296057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.007628+0000","last_change":"2024-01-15T18:12:00.007628+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.007628+0000","last_undegraded":"2024-01-15T18:12:00.007628+0000","last_fullsized":"2024-01-15T18:12:00.007628+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.003323+0000","last_change":"2024-01-15T18:12:00.003323+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.003323+0000","last_undegraded":"2024-01-15T18:12:00.003323+0000","last_fullsized":"2024-01-15T18:12:00.003323+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.006443+0000","last_change":"2024-01-15T18:12:00.006443+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.006443+0000","last_undegraded":"2024-01-15T18:12:00.006443+0000","last_fullsized":"2024-01-15T18:12:00.006443+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005852+0000","last_change":"2024-01-15T18:12:01.025557+0000","last_active":"2024-01-15T18:12:02.005852+0000","last_peered":"2024-01-15T18:12:02.005852+0000","last_clean":"2024-01-15T18:12:02.005852+0000","last_became_active":"2024-01-15T18:12:01.024972+0000","last_became_peered":"2024-01-15T18:12:01.024972+0000","last_unstale":"2024-01-15T18:12:02.005852+0000","last_undegraded":"2024-01-15T18:12:02.005852+0000","last_fullsized":"2024-01-15T18:12:02.005852+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:46:42.441463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005732+0000","last_change":"2024-01-15T18:12:01.023561+0000","last_active":"2024-01-15T18:12:02.005732+0000","last_peered":"2024-01-15T18:12:02.005732+0000","last_clean":"2024-01-15T18:12:02.005732+0000","last_became_active":"2024-01-15T18:12:01.022979+0000","last_became_peered":"2024-01-15T18:12:01.022979+0000","last_unstale":"2024-01-15T18:12:02.005732+0000","last_undegraded":"2024-01-15T18:12:02.005732+0000","last_fullsized":"2024-01-15T18:12:02.005732+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:58:49.958276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.009341+0000","last_change":"2024-01-15T18:12:00.009341+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.009341+0000","last_undegraded":"2024-01-15T18:12:00.009341+0000","last_fullsized":"2024-01-15T18:12:00.009341+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.014983+0000","last_change":"2024-01-15T18:12:00.014983+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.014983+0000","last_undegraded":"2024-01-15T18:12:00.014983+0000","last_fullsized":"2024-01-15T18:12:00.014983+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.028642+0000","last_change":"2024-01-15T18:12:00.026981+0000","last_active":"2024-01-15T18:12:00.028642+0000","last_peered":"2024-01-15T18:12:00.028642+0000","last_clean":"2024-01-15T18:12:00.028642+0000","last_became_active":"2024-01-15T18:12:00.026446+0000","last_became_peered":"2024-01-15T18:12:00.026446+0000","last_unstale":"2024-01-15T18:12:00.028642+0000","last_undegraded":"2024-01-15T18:12:00.028642+0000","last_fullsized":"2024-01-15T18:12:00.028642+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:10:55.247395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004241+0000","last_change":"2024-01-15T18:12:00.004241+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004241+0000","last_undegraded":"2024-01-15T18:12:00.004241+0000","last_fullsized":"2024-01-15T18:12:00.004241+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-15T18:11:58.996438+0000","last_change":"2024-01-15T18:11:58.996438+0000","last_active":"2024-01-15T18:11:58.982718+0000","last_peered":"2024-01-15T18:11:58.982718+0000","last_clean":"2024-01-15T18:11:58.982718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:58.996438+0000","last_undegraded":"2024-01-15T18:11:58.996438+0000","last_fullsized":"2024-01-15T18:11:58.996438+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.008802+0000","last_change":"2024-01-15T18:12:00.008802+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.008802+0000","last_undegraded":"2024-01-15T18:12:00.008802+0000","last_fullsized":"2024-01-15T18:12:00.008802+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.022940+0000","last_change":"2024-01-15T18:12:00.022131+0000","last_active":"2024-01-15T18:12:00.022940+0000","last_peered":"2024-01-15T18:12:00.022940+0000","last_clean":"2024-01-15T18:12:00.022940+0000","last_became_active":"2024-01-15T18:12:00.021836+0000","last_became_peered":"2024-01-15T18:12:00.021836+0000","last_unstale":"2024-01-15T18:12:00.022940+0000","last_undegraded":"2024-01-15T18:12:00.022940+0000","last_fullsized":"2024-01-15T18:12:00.022940+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:25.596374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.014971+0000","last_change":"2024-01-15T18:12:00.014971+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.014971+0000","last_undegraded":"2024-01-15T18:12:00.014971+0000","last_fullsized":"2024-01-15T18:12:00.014971+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.022099+0000","last_change":"2024-01-15T18:12:00.019217+0000","last_active":"2024-01-15T18:12:00.022099+0000","last_peered":"2024-01-15T18:12:00.022099+0000","last_clean":"2024-01-15T18:12:00.022099+0000","last_became_active":"2024-01-15T18:12:00.018924+0000","last_became_peered":"2024-01-15T18:12:00.018924+0000","last_unstale":"2024-01-15T18:12:00.022099+0000","last_undegraded":"2024-01-15T18:12:00.022099+0000","last_fullsized":"2024-01-15T18:12:00.022099+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:49:48.318924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.006011+0000","last_change":"2024-01-15T18:12:01.028720+0000","last_active":"2024-01-15T18:12:02.006011+0000","last_peered":"2024-01-15T18:12:02.006011+0000","last_clean":"2024-01-15T18:12:02.006011+0000","last_became_active":"2024-01-15T18:12:01.028453+0000","last_became_peered":"2024-01-15T18:12:01.028453+0000","last_unstale":"2024-01-15T18:12:02.006011+0000","last_undegraded":"2024-01-15T18:12:02.006011+0000","last_fullsized":"2024-01-15T18:12:02.006011+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:20.609600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.020405+0000","last_change":"2024-01-15T18:12:00.017547+0000","last_active":"2024-01-15T18:12:00.020405+0000","last_peered":"2024-01-15T18:12:00.020405+0000","last_clean":"2024-01-15T18:12:00.020405+0000","last_became_active":"2024-01-15T18:12:00.017260+0000","last_became_peered":"2024-01-15T18:12:00.017260+0000","last_unstale":"2024-01-15T18:12:00.020405+0000","last_undegraded":"2024-01-15T18:12:00.020405+0000","last_fullsized":"2024-01-15T18:12:00.020405+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:14:22.833631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004400+0000","last_change":"2024-01-15T18:12:00.004400+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004400+0000","last_undegraded":"2024-01-15T18:12:00.004400+0000","last_fullsized":"2024-01-15T18:12:00.004400+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.028035+0000","last_change":"2024-01-15T18:12:00.025697+0000","last_active":"2024-01-15T18:12:00.028035+0000","last_peered":"2024-01-15T18:12:00.028035+0000","last_clean":"2024-01-15T18:12:00.028035+0000","last_became_active":"2024-01-15T18:12:00.025383+0000","last_became_peered":"2024-01-15T18:12:00.025383+0000","last_unstale":"2024-01-15T18:12:00.028035+0000","last_undegraded":"2024-01-15T18:12:00.028035+0000","last_fullsized":"2024-01-15T18:12:00.028035+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:53:35.224736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004715+0000","last_change":"2024-01-15T18:12:00.004715+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004715+0000","last_undegraded":"2024-01-15T18:12:00.004715+0000","last_fullsized":"2024-01-15T18:12:00.004715+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.020327+0000","last_change":"2024-01-15T18:12:00.019136+0000","last_active":"2024-01-15T18:12:00.020327+0000","last_peered":"2024-01-15T18:12:00.020327+0000","last_clean":"2024-01-15T18:12:00.020327+0000","last_became_active":"2024-01-15T18:12:00.018512+0000","last_became_peered":"2024-01-15T18:12:00.018512+0000","last_unstale":"2024-01-15T18:12:00.020327+0000","last_undegraded":"2024-01-15T18:12:00.020327+0000","last_fullsized":"2024-01-15T18:12:00.020327+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:09:59.518166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:11:59.986005+0000","last_change":"2024-01-15T18:11:59.986005+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:59.986005+0000","last_undegraded":"2024-01-15T18:11:59.986005+0000","last_fullsized":"2024-01-15T18:11:59.986005+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.024655+0000","last_change":"2024-01-15T18:12:00.021284+0000","last_active":"2024-01-15T18:12:00.024655+0000","last_peered":"2024-01-15T18:12:00.024655+0000","last_clean":"2024-01-15T18:12:00.024655+0000","last_became_active":"2024-01-15T18:12:00.021017+0000","last_became_peered":"2024-01-15T18:12:00.021017+0000","last_unstale":"2024-01-15T18:12:00.024655+0000","last_undegraded":"2024-01-15T18:12:00.024655+0000","last_fullsized":"2024-01-15T18:12:00.024655+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:39:44.736330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005945+0000","last_change":"2024-01-15T18:12:01.042737+0000","last_active":"2024-01-15T18:12:02.005945+0000","last_peered":"2024-01-15T18:12:02.005945+0000","last_clean":"2024-01-15T18:12:02.005945+0000","last_became_active":"2024-01-15T18:12:01.042482+0000","last_became_peered":"2024-01-15T18:12:01.042482+0000","last_unstale":"2024-01-15T18:12:02.005945+0000","last_undegraded":"2024-01-15T18:12:02.005945+0000","last_fullsized":"2024-01-15T18:12:02.005945+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:04:19.070396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.019580+0000","last_change":"2024-01-15T18:12:00.017218+0000","last_active":"2024-01-15T18:12:00.019580+0000","last_peered":"2024-01-15T18:12:00.019580+0000","last_clean":"2024-01-15T18:12:00.019580+0000","last_became_active":"2024-01-15T18:12:00.016921+0000","last_became_peered":"2024-01-15T18:12:00.016921+0000","last_unstale":"2024-01-15T18:12:00.019580+0000","last_undegraded":"2024-01-15T18:12:00.019580+0000","last_fullsized":"2024-01-15T18:12:00.019580+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:12:19.017434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.017663+0000","last_change":"2024-01-15T18:12:00.017663+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.017663+0000","last_undegraded":"2024-01-15T18:12:00.017663+0000","last_fullsized":"2024-01-15T18:12:00.017663+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.021218+0000","last_change":"2024-01-15T18:12:00.018102+0000","last_active":"2024-01-15T18:12:00.021218+0000","last_peered":"2024-01-15T18:12:00.021218+0000","last_clean":"2024-01-15T18:12:00.021218+0000","last_became_active":"2024-01-15T18:12:00.017781+0000","last_became_peered":"2024-01-15T18:12:00.017781+0000","last_unstale":"2024-01-15T18:12:00.021218+0000","last_undegraded":"2024-01-15T18:12:00.021218+0000","last_fullsized":"2024-01-15T18:12:00.021218+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:28:35.159833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.008315+0000","last_change":"2024-01-15T18:12:00.008315+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.008315+0000","last_undegraded":"2024-01-15T18:12:00.008315+0000","last_fullsized":"2024-01-15T18:12:00.008315+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.022940+0000","last_change":"2024-01-15T18:12:00.019006+0000","last_active":"2024-01-15T18:12:00.022940+0000","last_peered":"2024-01-15T18:12:00.022940+0000","last_clean":"2024-01-15T18:12:00.022940+0000","last_became_active":"2024-01-15T18:12:00.017853+0000","last_became_peered":"2024-01-15T18:12:00.017853+0000","last_unstale":"2024-01-15T18:12:00.022940+0000","last_undegraded":"2024-01-15T18:12:00.022940+0000","last_fullsized":"2024-01-15T18:12:00.022940+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:56:18.635395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004379+0000","last_change":"2024-01-15T18:12:00.004379+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004379+0000","last_undegraded":"2024-01-15T18:12:00.004379+0000","last_fullsized":"2024-01-15T18:12:00.004379+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.013357+0000","last_change":"2024-01-15T18:12:00.011850+0000","last_active":"2024-01-15T18:12:00.013357+0000","last_peered":"2024-01-15T18:12:00.013357+0000","last_clean":"2024-01-15T18:12:00.013357+0000","last_became_active":"2024-01-15T18:12:00.011519+0000","last_became_peered":"2024-01-15T18:12:00.011519+0000","last_unstale":"2024-01-15T18:12:00.013357+0000","last_undegraded":"2024-01-15T18:12:00.013357+0000","last_fullsized":"2024-01-15T18:12:00.013357+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:33:21.015220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.013658+0000","last_change":"2024-01-15T18:12:00.013658+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.013658+0000","last_undegraded":"2024-01-15T18:12:00.013658+0000","last_fullsized":"2024-01-15T18:12:00.013658+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.028897+0000","last_change":"2024-01-15T18:12:00.028242+0000","last_active":"2024-01-15T18:12:00.028897+0000","last_peered":"2024-01-15T18:12:00.028897+0000","last_clean":"2024-01-15T18:12:00.028897+0000","last_became_active":"2024-01-15T18:12:00.027987+0000","last_became_peered":"2024-01-15T18:12:00.027987+0000","last_unstale":"2024-01-15T18:12:00.028897+0000","last_undegraded":"2024-01-15T18:12:00.028897+0000","last_fullsized":"2024-01-15T18:12:00.028897+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:27:28.185512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005827+0000","last_change":"2024-01-15T18:12:01.024849+0000","last_active":"2024-01-15T18:12:02.005827+0000","last_peered":"2024-01-15T18:12:02.005827+0000","last_clean":"2024-01-15T18:12:02.005827+0000","last_became_active":"2024-01-15T18:12:01.024579+0000","last_became_peered":"2024-01-15T18:12:01.024579+0000","last_unstale":"2024-01-15T18:12:02.005827+0000","last_undegraded":"2024-01-15T18:12:02.005827+0000","last_fullsized":"2024-01-15T18:12:02.005827+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:43:37.142127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.020356+0000","last_change":"2024-01-15T18:12:00.014156+0000","last_active":"2024-01-15T18:12:00.020356+0000","last_peered":"2024-01-15T18:12:00.020356+0000","last_clean":"2024-01-15T18:12:00.020356+0000","last_became_active":"2024-01-15T18:12:00.013881+0000","last_became_peered":"2024-01-15T18:12:00.013881+0000","last_unstale":"2024-01-15T18:12:00.020356+0000","last_undegraded":"2024-01-15T18:12:00.020356+0000","last_fullsized":"2024-01-15T18:12:00.020356+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:55:26.157604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:11:59.986005+0000","last_change":"2024-01-15T18:11:59.986005+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:59.986005+0000","last_undegraded":"2024-01-15T18:11:59.986005+0000","last_fullsized":"2024-01-15T18:11:59.986005+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.014661+0000","last_change":"2024-01-15T18:12:00.025503+0000","last_active":"2024-01-15T18:12:01.014661+0000","last_peered":"2024-01-15T18:12:01.014661+0000","last_clean":"2024-01-15T18:12:01.014661+0000","last_became_active":"2024-01-15T18:12:00.025198+0000","last_became_peered":"2024-01-15T18:12:00.025198+0000","last_unstale":"2024-01-15T18:12:01.014661+0000","last_undegraded":"2024-01-15T18:12:01.014661+0000","last_fullsized":"2024-01-15T18:12:01.014661+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:06:28.115069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.003520+0000","last_change":"2024-01-15T18:12:00.003520+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.003520+0000","last_undegraded":"2024-01-15T18:12:00.003520+0000","last_fullsized":"2024-01-15T18:12:00.003520+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.027906+0000","last_change":"2024-01-15T18:12:00.025058+0000","last_active":"2024-01-15T18:12:00.027906+0000","last_peered":"2024-01-15T18:12:00.027906+0000","last_clean":"2024-01-15T18:12:00.027906+0000","last_became_active":"2024-01-15T18:12:00.024655+0000","last_became_peered":"2024-01-15T18:12:00.024655+0000","last_unstale":"2024-01-15T18:12:00.027906+0000","last_undegraded":"2024-01-15T18:12:00.027906+0000","last_fullsized":"2024-01-15T18:12:00.027906+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:06:24.967935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.005892+0000","last_change":"2024-01-15T18:12:00.005892+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.005892+0000","last_undegraded":"2024-01-15T18:12:00.005892+0000","last_fullsized":"2024-01-15T18:12:00.005892+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.022025+0000","last_change":"2024-01-15T18:12:00.019039+0000","last_active":"2024-01-15T18:12:00.022025+0000","last_peered":"2024-01-15T18:12:00.022025+0000","last_clean":"2024-01-15T18:12:00.022025+0000","last_became_active":"2024-01-15T18:12:00.018741+0000","last_became_peered":"2024-01-15T18:12:00.018741+0000","last_unstale":"2024-01-15T18:12:00.022025+0000","last_undegraded":"2024-01-15T18:12:00.022025+0000","last_fullsized":"2024-01-15T18:12:00.022025+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:05:09.561750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.016990+0000","last_change":"2024-01-15T18:12:00.016990+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.016990+0000","last_undegraded":"2024-01-15T18:12:00.016990+0000","last_fullsized":"2024-01-15T18:12:00.016990+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.023601+0000","last_change":"2024-01-15T18:12:00.022509+0000","last_active":"2024-01-15T18:12:00.023601+0000","last_peered":"2024-01-15T18:12:00.023601+0000","last_clean":"2024-01-15T18:12:00.023601+0000","last_became_active":"2024-01-15T18:12:00.021725+0000","last_became_peered":"2024-01-15T18:12:00.021725+0000","last_unstale":"2024-01-15T18:12:00.023601+0000","last_undegraded":"2024-01-15T18:12:00.023601+0000","last_fullsized":"2024-01-15T18:12:00.023601+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:25.993795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.003474+0000","last_change":"2024-01-15T18:12:00.003474+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.003474+0000","last_undegraded":"2024-01-15T18:12:00.003474+0000","last_fullsized":"2024-01-15T18:12:00.003474+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.026284+0000","last_change":"2024-01-15T18:12:00.022642+0000","last_active":"2024-01-15T18:12:00.026284+0000","last_peered":"2024-01-15T18:12:00.026284+0000","last_clean":"2024-01-15T18:12:00.026284+0000","last_became_active":"2024-01-15T18:12:00.021842+0000","last_became_peered":"2024-01-15T18:12:00.021842+0000","last_unstale":"2024-01-15T18:12:00.026284+0000","last_undegraded":"2024-01-15T18:12:00.026284+0000","last_fullsized":"2024-01-15T18:12:00.026284+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:54:49.641387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.003892+0000","last_change":"2024-01-15T18:12:00.003892+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.003892+0000","last_undegraded":"2024-01-15T18:12:00.003892+0000","last_fullsized":"2024-01-15T18:12:00.003892+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.028661+0000","last_change":"2024-01-15T18:12:00.027374+0000","last_active":"2024-01-15T18:12:00.028661+0000","last_peered":"2024-01-15T18:12:00.028661+0000","last_clean":"2024-01-15T18:12:00.028661+0000","last_became_active":"2024-01-15T18:12:00.027064+0000","last_became_peered":"2024-01-15T18:12:00.027064+0000","last_unstale":"2024-01-15T18:12:00.028661+0000","last_undegraded":"2024-01-15T18:12:00.028661+0000","last_fullsized":"2024-01-15T18:12:00.028661+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:54:28.422641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.006003+0000","last_change":"2024-01-15T18:12:00.006003+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.006003+0000","last_undegraded":"2024-01-15T18:12:00.006003+0000","last_fullsized":"2024-01-15T18:12:00.006003+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.028017+0000","last_change":"2024-01-15T18:12:00.023536+0000","last_active":"2024-01-15T18:12:00.028017+0000","last_peered":"2024-01-15T18:12:00.028017+0000","last_clean":"2024-01-15T18:12:00.028017+0000","last_became_active":"2024-01-15T18:12:00.023249+0000","last_became_peered":"2024-01-15T18:12:00.023249+0000","last_unstale":"2024-01-15T18:12:00.028017+0000","last_undegraded":"2024-01-15T18:12:00.028017+0000","last_fullsized":"2024-01-15T18:12:00.028017+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:48:59.611157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.005765+0000","last_change":"2024-01-15T18:12:00.005765+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.005765+0000","last_undegraded":"2024-01-15T18:12:00.005765+0000","last_fullsized":"2024-01-15T18:12:00.005765+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.017179+0000","last_change":"2024-01-15T18:12:00.020946+0000","last_active":"2024-01-15T18:12:01.017179+0000","last_peered":"2024-01-15T18:12:01.017179+0000","last_clean":"2024-01-15T18:12:01.017179+0000","last_became_active":"2024-01-15T18:12:00.020641+0000","last_became_peered":"2024-01-15T18:12:00.020641+0000","last_unstale":"2024-01-15T18:12:01.017179+0000","last_undegraded":"2024-01-15T18:12:01.017179+0000","last_fullsized":"2024-01-15T18:12:01.017179+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:14:51.219226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.005856+0000","last_change":"2024-01-15T18:12:00.005856+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.005856+0000","last_undegraded":"2024-01-15T18:12:00.005856+0000","last_fullsized":"2024-01-15T18:12:00.005856+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-15T18:11:58.996468+0000","last_change":"2024-01-15T18:11:58.996468+0000","last_active":"2024-01-15T18:11:58.982718+0000","last_peered":"2024-01-15T18:11:58.982718+0000","last_clean":"2024-01-15T18:11:58.982718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:58.996468+0000","last_undegraded":"2024-01-15T18:11:58.996468+0000","last_fullsized":"2024-01-15T18:11:58.996468+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:11:59.986005+0000","last_change":"2024-01-15T18:11:59.986005+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:59.986005+0000","last_undegraded":"2024-01-15T18:11:59.986005+0000","last_fullsized":"2024-01-15T18:11:59.986005+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.021188+0000","last_change":"2024-01-15T18:12:00.016456+0000","last_active":"2024-01-15T18:12:00.021188+0000","last_peered":"2024-01-15T18:12:00.021188+0000","last_clean":"2024-01-15T18:12:00.021188+0000","last_became_active":"2024-01-15T18:12:00.016145+0000","last_became_peered":"2024-01-15T18:12:00.016145+0000","last_unstale":"2024-01-15T18:12:00.021188+0000","last_undegraded":"2024-01-15T18:12:00.021188+0000","last_fullsized":"2024-01-15T18:12:00.021188+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:02:23.099071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.006010+0000","last_change":"2024-01-15T18:12:01.024249+0000","last_active":"2024-01-15T18:12:02.006010+0000","last_peered":"2024-01-15T18:12:02.006010+0000","last_clean":"2024-01-15T18:12:02.006010+0000","last_became_active":"2024-01-15T18:12:01.023862+0000","last_became_peered":"2024-01-15T18:12:01.023862+0000","last_unstale":"2024-01-15T18:12:02.006010+0000","last_undegraded":"2024-01-15T18:12:02.006010+0000","last_fullsized":"2024-01-15T18:12:02.006010+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:18:47.967166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.016071+0000","last_change":"2024-01-15T18:12:00.023511+0000","last_active":"2024-01-15T18:12:01.016071+0000","last_peered":"2024-01-15T18:12:01.016071+0000","last_clean":"2024-01-15T18:12:01.016071+0000","last_became_active":"2024-01-15T18:12:00.023145+0000","last_became_peered":"2024-01-15T18:12:00.023145+0000","last_unstale":"2024-01-15T18:12:01.016071+0000","last_undegraded":"2024-01-15T18:12:01.016071+0000","last_fullsized":"2024-01-15T18:12:01.016071+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:27:30.560351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.016823+0000","last_change":"2024-01-15T18:12:00.016823+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.016823+0000","last_undegraded":"2024-01-15T18:12:00.016823+0000","last_fullsized":"2024-01-15T18:12:00.016823+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.025706+0000","last_change":"2024-01-15T18:12:00.023014+0000","last_active":"2024-01-15T18:12:00.025706+0000","last_peered":"2024-01-15T18:12:00.025706+0000","last_clean":"2024-01-15T18:12:00.025706+0000","last_became_active":"2024-01-15T18:12:00.022748+0000","last_became_peered":"2024-01-15T18:12:00.022748+0000","last_unstale":"2024-01-15T18:12:00.025706+0000","last_undegraded":"2024-01-15T18:12:00.025706+0000","last_fullsized":"2024-01-15T18:12:00.025706+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:13:33.148973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.014451+0000","last_change":"2024-01-15T18:12:00.014451+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.014451+0000","last_undegraded":"2024-01-15T18:12:00.014451+0000","last_fullsized":"2024-01-15T18:12:00.014451+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":1,"reported_epoch":67,"state":"creating+peering","last_fresh":"2024-01-15T18:11:58.997807+0000","last_change":"2024-01-15T18:11:58.997807+0000","last_active":"2024-01-15T18:11:58.982718+0000","last_peered":"2024-01-15T18:11:58.982718+0000","last_clean":"2024-01-15T18:11:58.982718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:11:58.997807+0000","last_undegraded":"2024-01-15T18:11:58.997807+0000","last_fullsized":"2024-01-15T18:11:58.997807+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.003849+0000","last_change":"2024-01-15T18:12:00.003849+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.003849+0000","last_undegraded":"2024-01-15T18:12:00.003849+0000","last_fullsized":"2024-01-15T18:12:00.003849+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.021303+0000","last_change":"2024-01-15T18:12:00.019406+0000","last_active":"2024-01-15T18:12:00.021303+0000","last_peered":"2024-01-15T18:12:00.021303+0000","last_clean":"2024-01-15T18:12:00.021303+0000","last_became_active":"2024-01-15T18:12:00.019122+0000","last_became_peered":"2024-01-15T18:12:00.019122+0000","last_unstale":"2024-01-15T18:12:00.021303+0000","last_undegraded":"2024-01-15T18:12:00.021303+0000","last_fullsized":"2024-01-15T18:12:00.021303+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:51:33.138526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.007471+0000","last_change":"2024-01-15T18:12:00.007471+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.007471+0000","last_undegraded":"2024-01-15T18:12:00.007471+0000","last_fullsized":"2024-01-15T18:12:00.007471+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.018689+0000","last_change":"2024-01-15T18:12:00.015364+0000","last_active":"2024-01-15T18:12:00.018689+0000","last_peered":"2024-01-15T18:12:00.018689+0000","last_clean":"2024-01-15T18:12:00.018689+0000","last_became_active":"2024-01-15T18:12:00.014208+0000","last_became_peered":"2024-01-15T18:12:00.014208+0000","last_unstale":"2024-01-15T18:12:00.018689+0000","last_undegraded":"2024-01-15T18:12:00.018689+0000","last_fullsized":"2024-01-15T18:12:00.018689+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:59:42.431920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.004040+0000","last_change":"2024-01-15T18:12:00.004040+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.004040+0000","last_undegraded":"2024-01-15T18:12:00.004040+0000","last_fullsized":"2024-01-15T18:12:00.004040+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.027927+0000","last_change":"2024-01-15T18:12:00.025623+0000","last_active":"2024-01-15T18:12:00.027927+0000","last_peered":"2024-01-15T18:12:00.027927+0000","last_clean":"2024-01-15T18:12:00.027927+0000","last_became_active":"2024-01-15T18:12:00.025294+0000","last_became_peered":"2024-01-15T18:12:00.025294+0000","last_unstale":"2024-01-15T18:12:00.027927+0000","last_undegraded":"2024-01-15T18:12:00.027927+0000","last_fullsized":"2024-01-15T18:12:00.027927+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:02:43.133788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.007044+0000","last_change":"2024-01-15T18:12:00.007044+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.007044+0000","last_undegraded":"2024-01-15T18:12:00.007044+0000","last_fullsized":"2024-01-15T18:12:00.007044+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.023758+0000","last_change":"2024-01-15T18:12:00.019753+0000","last_active":"2024-01-15T18:12:00.023758+0000","last_peered":"2024-01-15T18:12:00.023758+0000","last_clean":"2024-01-15T18:12:00.023758+0000","last_became_active":"2024-01-15T18:12:00.019482+0000","last_became_peered":"2024-01-15T18:12:00.019482+0000","last_unstale":"2024-01-15T18:12:00.023758+0000","last_undegraded":"2024-01-15T18:12:00.023758+0000","last_fullsized":"2024-01-15T18:12:00.023758+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:53:26.480802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":68,"state":"creating+peering","last_fresh":"2024-01-15T18:12:00.009189+0000","last_change":"2024-01-15T18:12:00.009189+0000","last_active":"2024-01-15T18:11:59.986005+0000","last_peered":"2024-01-15T18:11:59.986005+0000","last_clean":"2024-01-15T18:11:59.986005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:12:00.009189+0000","last_undegraded":"2024-01-15T18:12:00.009189+0000","last_fullsized":"2024-01-15T18:12:00.009189+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-01-15T18:12:00.022076+0000","last_change":"2024-01-15T18:12:00.019108+0000","last_active":"2024-01-15T18:12:00.022076+0000","last_peered":"2024-01-15T18:12:00.022076+0000","last_clean":"2024-01-15T18:12:00.022076+0000","last_became_active":"2024-01-15T18:12:00.018827+0000","last_became_peered":"2024-01-15T18:12:00.018827+0000","last_unstale":"2024-01-15T18:12:00.022076+0000","last_undegraded":"2024-01-15T18:12:00.022076+0000","last_fullsized":"2024-01-15T18:12:00.022076+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:48:57.061505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":203,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-01-15T18:11:58.994013+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:11:58.994013+0000","last_peered":"2024-01-15T18:11:58.994013+0000","last_clean":"2024-01-15T18:11:58.994013+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:11:58.994013+0000","last_undegraded":"2024-01-15T18:11:58.994013+0000","last_fullsized":"2024-01-15T18:11:58.994013+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":116,"acting":116,"num_store_stats":7},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509530,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50596,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321244,"statfs":{"total":96636764160,"available":96584953856,"internally_reserved":0,"allocated":1400832,"data_stored":928722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.1200000000000001,"15min":0.78900000000000003},"min":{"1min":0.66000000000000003,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":9.6989999999999998,"5min":10.284000000000001,"15min":10.284000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":2.4159999999999999,"5min":1.2290000000000001,"15min":0.86299999999999999},"min":{"1min":0.39100000000000001,"5min":0.378,"15min":0.378},"max":{"1min":16.968,"5min":16.968,"15min":16.968},"last":0.56599999999999995}]},{"osd":1,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.7749999999999999,"5min":1.1830000000000001,"15min":0.83899999999999997},"min":{"1min":0.64800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":15.791,"5min":15.791,"15min":15.791},"last":0.54300000000000004},{"interface":"front","average":{"1min":2.1339999999999999,"5min":1.1779999999999999,"15min":0.85899999999999999},"min":{"1min":0.44600000000000001,"5min":0.378,"15min":0.378},"max":{"1min":10.842000000000001,"5min":10.842000000000001,"15min":10.842000000000001},"last":0.64800000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1,"15min":0.76400000000000001},"min":{"1min":0.42299999999999999,"5min":0.378,"15min":0.378},"max":{"1min":4.9029999999999996,"5min":10.119999999999999,"15min":10.119999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":2.0579999999999998,"5min":1.081,"15min":0.83499999999999996},"min":{"1min":0.66800000000000004,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":9.7539999999999996,"5min":9.7539999999999996,"15min":9.7539999999999996},"last":0.75900000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":1.802,"5min":1.143,"15min":0.82299999999999995},"min":{"1min":0.47099999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":11.481,"5min":11.481,"15min":11.481},"last":0.69299999999999995},{"interface":"front","average":{"1min":2.3220000000000001,"5min":1.2310000000000001,"15min":0.91900000000000004},"min":{"1min":0.70199999999999996,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":9.7959999999999994,"5min":10.34,"15min":10.34},"last":0.67200000000000004}]},{"osd":4,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.5630000000000002,"5min":1.3360000000000001,"15min":0.85799999999999998},"min":{"1min":0.41099999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":20.387,"5min":20.387,"15min":20.387},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.649,"5min":1.131,"15min":0.81999999999999995},"min":{"1min":0.46200000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":11.305,"5min":14.871,"15min":14.871},"last":0.623}]},{"osd":5,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.2719999999999998,"5min":1.2070000000000001,"15min":0.86899999999999999},"min":{"1min":0.47799999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":9.8450000000000006,"5min":19.959,"15min":19.959},"last":0.59099999999999997},{"interface":"front","average":{"1min":2.6240000000000001,"5min":1.3009999999999999,"15min":0.92000000000000004},"min":{"1min":0.48299999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":19.905000000000001,"5min":19.905000000000001,"15min":19.905000000000001},"last":0.83199999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":3.157,"5min":1.667,"15min":1.093},"min":{"1min":0.63800000000000001,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":20.492000000000001,"5min":22.478999999999999,"15min":22.478999999999999},"last":0.877},{"interface":"front","average":{"1min":1.397,"5min":1.3049999999999999,"15min":0.92500000000000004},"min":{"1min":0.45100000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.2549999999999999,"5min":21.509,"15min":21.509},"last":0.73899999999999999}]}]},{"osd":6,"up_from":15,"seq":64424509530,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45856,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325984,"statfs":{"total":96636764160,"available":96589807616,"internally_reserved":0,"allocated":806912,"data_stored":338354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":1.157,"15min":0.872},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.91800000000000004,"5min":12.41,"15min":12.41},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.69199999999999995,"5min":1.0009999999999999,"15min":0.79000000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.93500000000000005,"5min":6.7759999999999998,"15min":6.7759999999999998},"last":0.57399999999999995}]},{"osd":1,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":1.454,"15min":0.96399999999999997},"min":{"1min":0.46999999999999997,"5min":0.41899999999999998,"15min":0.41299999999999998},"max":{"1min":0.83199999999999996,"5min":53.451999999999998,"15min":53.451999999999998},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.71299999999999997,"5min":1.008,"15min":0.83699999999999997},"min":{"1min":0.56100000000000005,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.92900000000000005,"5min":7.7329999999999997,"15min":7.7329999999999997},"last":0.56200000000000006}]},{"osd":2,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":1.004,"15min":0.89400000000000002},"min":{"1min":0.46700000000000003,"5min":0.45900000000000002,"15min":0.38900000000000001},"max":{"1min":1.1060000000000001,"5min":7.9409999999999998,"15min":7.9409999999999998},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.93700000000000006,"15min":0.82999999999999996},"min":{"1min":0.57099999999999995,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.90800000000000003,"5min":5.1760000000000002,"15min":5.1760000000000002},"last":0.75800000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":1.038,"15min":0.876},"min":{"1min":0.46000000000000002,"5min":0.42099999999999999,"15min":0.41399999999999998},"max":{"1min":0.94899999999999995,"5min":6.758,"15min":6.758},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.70499999999999996,"5min":1.5600000000000001,"15min":1.0660000000000001},"min":{"1min":0.49199999999999999,"5min":0.45200000000000001,"15min":0.41599999999999998},"max":{"1min":0.877,"5min":53.398000000000003,"15min":53.398000000000003},"last":0.80600000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":1.0029999999999999,"15min":0.79500000000000004},"min":{"1min":0.38500000000000001,"5min":0.378,"15min":0.34999999999999998},"max":{"1min":0.88,"5min":8.6259999999999994,"15min":8.6259999999999994},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.94899999999999995,"15min":0.79100000000000004},"min":{"1min":0.42599999999999999,"5min":0.38200000000000001,"15min":0.34499999999999997},"max":{"1min":0.85099999999999998,"5min":6.9589999999999996,"15min":6.9589999999999996},"last":0.63}]},{"osd":5,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":1.641,"15min":0.98299999999999998},"min":{"1min":0.41299999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.96399999999999997,"5min":53.229999999999997,"15min":53.229999999999997},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.69299999999999995,"5min":1.228,"15min":0.92900000000000005},"min":{"1min":0.45500000000000002,"5min":0.435,"15min":0.435},"max":{"1min":1.149,"5min":10.029999999999999,"15min":10.029999999999999},"last":0.745}]},{"osd":7,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":1.5760000000000001,"15min":1.071},"min":{"1min":0.437,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.90400000000000003,"5min":45.070999999999998,"15min":45.070999999999998},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.70099999999999996,"5min":1.6910000000000001,"15min":1.0940000000000001},"min":{"1min":0.52800000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.92200000000000004,"5min":53.338000000000001,"15min":53.338000000000001},"last":0.77300000000000002}]}]},{"osd":4,"up_from":15,"seq":64424509530,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50028,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321812,"statfs":{"total":96636764160,"available":96585535488,"internally_reserved":0,"allocated":819200,"data_stored":344685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":1.2,"15min":0.81899999999999995},"min":{"1min":0.502,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.2509999999999999,"5min":23.318000000000001,"15min":23.318000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.70699999999999996,"5min":1.046,"15min":0.77500000000000002},"min":{"1min":0.442,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.97899999999999998,"5min":14.723000000000001,"15min":14.723000000000001},"last":0.442}]},{"osd":1,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.89500000000000002,"15min":0.72999999999999998},"min":{"1min":0.442,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.876,"5min":7.8200000000000003,"15min":7.8200000000000003},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.73899999999999999,"5min":1.1879999999999999,"15min":0.82299999999999995},"min":{"1min":0.47299999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.613,"5min":22.408999999999999,"15min":22.408999999999999},"last":0.58999999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.91800000000000004,"15min":0.72399999999999998},"min":{"1min":0.48699999999999999,"5min":0.40600000000000003,"15min":0.34399999999999997},"max":{"1min":1.0009999999999999,"5min":8.8529999999999998,"15min":8.8529999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.77900000000000003,"5min":1.288,"15min":0.89200000000000002},"min":{"1min":0.55500000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.2969999999999999,"5min":22.338999999999999,"15min":22.338999999999999},"last":0.65800000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":1.149,"15min":0.84299999999999997},"min":{"1min":0.52500000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.3939999999999999,"5min":14.763999999999999,"15min":14.763999999999999},"last":0.622},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0509999999999999,"15min":0.83399999999999996},"min":{"1min":0.59899999999999998,"5min":0.504,"15min":0.45500000000000002},"max":{"1min":1.7669999999999999,"5min":8.9459999999999997,"15min":8.9459999999999997},"last":0.67300000000000004}]},{"osd":5,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":1.2609999999999999,"15min":0.86399999999999999},"min":{"1min":0.39800000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.4670000000000001,"5min":31.617999999999999,"15min":31.617999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":1.1910000000000001,"15min":0.83099999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.377},"max":{"1min":1.022,"5min":17.145,"15min":17.145},"last":0.55100000000000005}]},{"osd":6,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.95299999999999996,"15min":0.77100000000000002},"min":{"1min":0.46500000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.89300000000000002,"5min":11.124000000000001,"15min":11.124000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.749,"5min":1.7629999999999999,"15min":1.0780000000000001},"min":{"1min":0.45600000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.339,"5min":28.872,"15min":28.872},"last":0.76700000000000002}]},{"osd":7,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":1.5409999999999999,"15min":0.99299999999999999},"min":{"1min":0.45800000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.3169999999999999,"5min":39.256,"15min":39.256},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":1.1160000000000001,"15min":0.89200000000000002},"min":{"1min":0.51900000000000002,"5min":0.46500000000000002,"15min":0.42199999999999999},"max":{"1min":1.0329999999999999,"5min":26.573,"15min":26.573},"last":0.72899999999999998}]}]},{"osd":3,"up_from":15,"seq":64424509529,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50016,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321824,"statfs":{"total":96636764160,"available":96585547776,"internally_reserved":0,"allocated":806912,"data_stored":338354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":0.58899999999999997,"15min":0.47699999999999998},"min":{"1min":0.29799999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":5.8849999999999998,"5min":5.8849999999999998,"15min":5.8849999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.1299999999999999,"5min":0.81999999999999995,"15min":0.58699999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":4.7990000000000004,"5min":17.756,"15min":17.756},"last":0.44400000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":0.755,"15min":0.59099999999999997},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":12.307,"5min":12.307,"15min":12.307},"last":0.56999999999999995},{"interface":"front","average":{"1min":1.1080000000000001,"5min":0.66300000000000003,"15min":0.57799999999999996},"min":{"1min":0.34899999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":6.0640000000000001,"5min":6.0640000000000001,"15min":6.0640000000000001},"last":0.51400000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":0.84999999999999998,"15min":0.65800000000000003},"min":{"1min":0.36099999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":12.468999999999999,"5min":12.468999999999999,"15min":12.468999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.2549999999999999,"5min":0.77400000000000002,"15min":0.63400000000000001},"min":{"1min":0.41599999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":6.0330000000000004,"5min":6.0330000000000004,"15min":6.0330000000000004},"last":0.46899999999999997}]},{"osd":4,"last update":"Mon Jan 15 18:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.141,"15min":0.91900000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":6.3970000000000002,"5min":17.853999999999999,"15min":17.853999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":2.1240000000000001,"5min":1.115,"15min":0.89300000000000002},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":13.31,"5min":13.31,"15min":13.31},"last":0.67000000000000004}]},{"osd":5,"last update":"Mon Jan 15 18:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":1.0109999999999999,"15min":0.83099999999999996},"min":{"1min":0.52100000000000002,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":6.1139999999999999,"5min":6.1139999999999999,"15min":6.1139999999999999},"last":0.81899999999999995},{"interface":"front","average":{"1min":2.1720000000000002,"5min":1.284,"15min":0.90800000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":20.515999999999998,"5min":20.515999999999998,"15min":20.515999999999998},"last":0.83199999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:10:58 2024","interfaces":[{"interface":"back","average":{"1min":2.2879999999999998,"5min":1.333,"15min":0.96899999999999997},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":13.250999999999999,"5min":23.100999999999999,"15min":23.100999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":2.246,"5min":1.1339999999999999,"15min":0.88600000000000001},"min":{"1min":0.5,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":12.76,"5min":12.76,"15min":12.76},"last":0.86199999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:10:58 2024","interfaces":[{"interface":"back","average":{"1min":2.9430000000000001,"5min":1.3819999999999999,"15min":0.97899999999999998},"min":{"1min":0.48599999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":21.545000000000002,"5min":21.545000000000002,"15min":21.545000000000002},"last":0.70199999999999996},{"interface":"front","average":{"1min":2.7650000000000001,"5min":1.3200000000000001,"15min":0.98899999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":13.385,"5min":13.385,"15min":13.385},"last":0.80500000000000005}]}]},{"osd":2,"up_from":15,"seq":64424509529,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45872,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325968,"statfs":{"total":96636764160,"available":96589791232,"internally_reserved":0,"allocated":819200,"data_stored":344685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.61699999999999999,"15min":0.51200000000000001},"min":{"1min":0.29399999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.8480000000000001,"5min":5.1849999999999996,"15min":5.1849999999999996},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.64200000000000002,"15min":0.52300000000000002},"min":{"1min":0.32600000000000001,"5min":0.29399999999999998,"15min":0.28299999999999997},"max":{"1min":2.3759999999999999,"5min":7.516,"15min":7.516},"last":0.50700000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.748,"15min":0.61199999999999999},"min":{"1min":0.34100000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":2.2850000000000001,"5min":10.907,"15min":10.907},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.81599999999999995,"15min":0.57399999999999995},"min":{"1min":0.40500000000000003,"5min":0.311,"15min":0.27000000000000002},"max":{"1min":1.7150000000000001,"5min":8.9190000000000005,"15min":8.9190000000000005},"last":0.68999999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.73599999999999999,"15min":0.628},"min":{"1min":0.36599999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.86399999999999999,"5min":12.185,"15min":12.185},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.83499999999999996,"15min":0.64700000000000002},"min":{"1min":0.443,"5min":0.39100000000000001,"15min":0.34000000000000002},"max":{"1min":2.3330000000000002,"5min":7.6760000000000002,"15min":7.6760000000000002},"last":0.63500000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.99299999999999999,"15min":0.82699999999999996},"min":{"1min":0.58699999999999997,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":2.21,"5min":8.6609999999999996,"15min":8.6609999999999996},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.76000000000000001,"5min":1.1899999999999999,"15min":0.90600000000000003},"min":{"1min":0.54000000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.6339999999999999,"5min":19.146999999999998,"15min":19.146999999999998},"last":0.77500000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":1.1319999999999999,"15min":0.90800000000000003},"min":{"1min":0.442,"5min":0.44,"15min":0.44},"max":{"1min":0.90800000000000003,"5min":31.710000000000001,"15min":31.710000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.875,"5min":1.373,"15min":0.98299999999999998},"min":{"1min":0.59399999999999997,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.1669999999999998,"5min":22.539000000000001,"15min":22.539000000000001},"last":0.871}]},{"osd":6,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":1.107,"15min":0.88600000000000001},"min":{"1min":0.61699999999999999,"5min":0.50800000000000001,"15min":0.48299999999999998},"max":{"1min":1.675,"5min":9.4540000000000006,"15min":9.4540000000000006},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.1399999999999999,"15min":0.90900000000000003},"min":{"1min":0.53100000000000003,"5min":0.503,"15min":0.503},"max":{"1min":1,"5min":18.614999999999998,"15min":18.614999999999998},"last":0.74199999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":1.113,"15min":0.90400000000000003},"min":{"1min":0.58199999999999996,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.752,"5min":11.432,"15min":11.432},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.264,"15min":0.97099999999999997},"min":{"1min":0.54500000000000004,"5min":0.502,"15min":0.502},"max":{"1min":2.2480000000000002,"5min":17.013000000000002,"15min":17.013000000000002},"last":0.90600000000000003}]}]},{"osd":1,"up_from":15,"seq":64424509531,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50032,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321808,"statfs":{"total":96636764160,"available":96585531392,"internally_reserved":0,"allocated":819200,"data_stored":344685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.96399999999999997,"15min":0.65500000000000003},"min":{"1min":0.309,"5min":0.26500000000000001,"15min":0.254},"max":{"1min":0.72899999999999998,"5min":19.577000000000002,"15min":19.577000000000002},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":1.1779999999999999,"15min":0.71499999999999997},"min":{"1min":0.35399999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.80000000000000004,"5min":48.277999999999999,"15min":48.277999999999999},"last":0.42899999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":1.036,"15min":0.70599999999999996},"min":{"1min":0.35599999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.65000000000000002,"5min":40.698,"15min":40.698},"last":0.379},{"interface":"front","average":{"1min":0.53200000000000003,"5min":1.337,"15min":0.92900000000000005},"min":{"1min":0.36899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93300000000000005,"5min":48.316000000000003,"15min":48.316000000000003},"last":0.35399999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.2370000000000001,"15min":0.83999999999999997},"min":{"1min":0.46600000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.88200000000000001,"5min":48.234999999999999,"15min":48.234999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.75800000000000001,"15min":0.64500000000000002},"min":{"1min":0.36099999999999999,"5min":0.34699999999999998,"15min":0.311},"max":{"1min":0.74399999999999999,"5min":6.468,"15min":6.468},"last":0.69299999999999995}]},{"osd":4,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":1.0589999999999999,"15min":0.81799999999999995},"min":{"1min":0.48999999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.0720000000000001,"5min":8.3409999999999993,"15min":8.3409999999999993},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.80200000000000005,"5min":1.165,"15min":0.90600000000000003},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.42499999999999999},"max":{"1min":1.278,"5min":19.829000000000001,"15min":19.829000000000001},"last":0.52200000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":1.4399999999999999,"15min":0.99399999999999999},"min":{"1min":0.59499999999999997,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.306,"5min":22.594999999999999,"15min":22.594999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.79400000000000004,"5min":1.8939999999999999,"15min":1.145},"min":{"1min":0.54500000000000004,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1180000000000001,"5min":48.198,"15min":48.198},"last":0.57699999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":1.4590000000000001,"15min":0.98899999999999999},"min":{"1min":0.59899999999999998,"5min":0.53000000000000003,"15min":0.47499999999999998},"max":{"1min":1.1499999999999999,"5min":35.112000000000002,"15min":35.112000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.81200000000000006,"5min":1.1779999999999999,"15min":0.93999999999999995},"min":{"1min":0.53100000000000003,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.175,"5min":19.780999999999999,"15min":19.780999999999999},"last":0.59999999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":1.8069999999999999,"15min":1.1259999999999999},"min":{"1min":0.628,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2450000000000001,"5min":48.148000000000003,"15min":48.148000000000003},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.80300000000000005,"5min":1.194,"15min":0.93300000000000005},"min":{"1min":0.64000000000000001,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.214,"5min":9.3840000000000003,"15min":9.3840000000000003},"last":0.84199999999999997}]}]},{"osd":5,"up_from":14,"seq":60129542235,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50064,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321776,"statfs":{"total":96636764160,"available":96585498624,"internally_reserved":0,"allocated":847872,"data_stored":357962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.872,"15min":0.77600000000000002},"min":{"1min":0.45900000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.3019999999999996,"5min":21.385000000000002,"15min":21.385000000000002},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.732,"5min":1.071,"15min":0.88800000000000001},"min":{"1min":0.46999999999999997,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":9.359,"5min":9.359,"15min":9.359},"last":0.78200000000000003}]},{"osd":1,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":3.0960000000000001,"5min":1.1839999999999999,"15min":0.84599999999999997},"min":{"1min":0.48799999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":21.120000000000001,"5min":21.120000000000001,"15min":21.120000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0329999999999999,"15min":0.80100000000000005},"min":{"1min":0.52400000000000002,"5min":0.315,"15min":0.315},"max":{"1min":4.3730000000000002,"5min":21.5,"15min":21.5},"last":0.753}]},{"osd":2,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.1419999999999999,"15min":0.85399999999999998},"min":{"1min":0.56399999999999995,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":6.2750000000000004,"5min":8.7750000000000004,"15min":8.7750000000000004},"last":0.73799999999999999},{"interface":"front","average":{"1min":2.9780000000000002,"5min":1.2869999999999999,"15min":0.94999999999999996},"min":{"1min":0.58099999999999996,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":21.201000000000001,"5min":21.201000000000001,"15min":21.201000000000001},"last":0.76600000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.0069999999999999,"15min":0.875},"min":{"1min":0.57399999999999995,"5min":0.44,"15min":0.44},"max":{"1min":4.4100000000000001,"5min":21.457000000000001,"15min":21.457000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.583,"5min":1.196,"15min":0.96799999999999997},"min":{"1min":0.54200000000000004,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":6.8650000000000002,"5min":8.7300000000000004,"15min":8.7300000000000004},"last":0.64900000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.012,"15min":0.82799999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":5.8929999999999998,"5min":9.6679999999999993,"15min":9.6679999999999993},"last":0.55700000000000005},{"interface":"front","average":{"1min":3.8380000000000001,"5min":1.5329999999999999,"15min":1.008},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":21.724,"5min":26.146999999999998,"15min":26.146999999999998},"last":0.70099999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.91900000000000004,"15min":0.79800000000000004},"min":{"1min":0.36099999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.835,"5min":21.535,"15min":21.535},"last":0.67600000000000005},{"interface":"front","average":{"1min":1.6160000000000001,"5min":1.3839999999999999,"15min":0.999},"min":{"1min":0.40100000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":5.7539999999999996,"5min":30.488,"15min":30.488},"last":0.82099999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":4.7309999999999999,"5min":1.6259999999999999,"15min":1.0760000000000001},"min":{"1min":0.42599999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":30.483000000000001,"5min":30.483000000000001,"15min":30.483000000000001},"last":0.629},{"interface":"front","average":{"1min":3.508,"5min":1.4650000000000001,"15min":1.0429999999999999},"min":{"1min":0.42899999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":25.437999999999999,"5min":25.437999999999999,"15min":25.437999999999999},"last":0.72199999999999998}]}]},{"osd":0,"up_from":14,"seq":60129542234,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54744,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317096,"statfs":{"total":96636764160,"available":96580706304,"internally_reserved":0,"allocated":1388544,"data_stored":922849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.89300000000000002,"15min":0.60699999999999998},"min":{"1min":0.26000000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9179999999999999,"5min":19.843,"15min":19.843},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.88700000000000001,"15min":0.66400000000000003},"min":{"1min":0.29199999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73299999999999998,"5min":10.154,"15min":10.154},"last":0.42999999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":1.115,"15min":0.753},"min":{"1min":0.38800000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.754,"5min":25.905000000000001,"15min":25.905000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.5,"5min":0.77900000000000003,"15min":0.58199999999999996},"min":{"1min":0.29699999999999999,"5min":0.28599999999999998,"15min":0.26700000000000002},"max":{"1min":1.2549999999999999,"5min":9.3829999999999991,"15min":9.3829999999999991},"last":0.38800000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.95699999999999996,"15min":0.70299999999999996},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.308},"max":{"1min":1.952,"5min":9.1780000000000008,"15min":9.1780000000000008},"last":0.496},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.97399999999999998,"15min":0.73099999999999998},"min":{"1min":0.42899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":2.9809999999999999,"5min":10.210000000000001,"15min":10.210000000000001},"last":0.51200000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":1.343,"15min":0.96399999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.38},"max":{"1min":1.306,"5min":22.814,"15min":22.814},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.446,"15min":1.03},"min":{"1min":0.49099999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":2.169,"5min":23.341999999999999,"15min":23.341999999999999},"last":0.73299999999999998}]},{"osd":5,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":1.262,"15min":0.94899999999999995},"min":{"1min":0.46899999999999997,"5min":0.44600000000000001,"15min":0.441},"max":{"1min":1.365,"5min":12.428000000000001,"15min":12.428000000000001},"last":0.874},{"interface":"front","average":{"1min":0.82499999999999996,"5min":1.6930000000000001,"15min":1.1259999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.46300000000000002},"max":{"1min":2.0880000000000001,"5min":19.774000000000001,"15min":19.774000000000001},"last":0.78600000000000003}]},{"osd":6,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":1.9219999999999999,"15min":1.1719999999999999},"min":{"1min":0.46700000000000003,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.0449999999999999,"5min":31.059000000000001,"15min":31.059000000000001},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.83299999999999996,"5min":1.488,"15min":1.0580000000000001},"min":{"1min":0.57999999999999996,"5min":0.51500000000000001,"15min":0.49099999999999999},"max":{"1min":1.3899999999999999,"5min":40.731000000000002,"15min":40.731000000000002},"last":0.80800000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":1.673,"15min":1.1100000000000001},"min":{"1min":0.61599999999999999,"5min":0.47199999999999998,"15min":0.42499999999999999},"max":{"1min":1.4239999999999999,"5min":24.492999999999999,"15min":24.492999999999999},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.875,"5min":1.5780000000000001,"15min":1.097},"min":{"1min":0.61199999999999999,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":2.2130000000000001,"5min":23.524000000000001,"15min":23.524000000000001},"last":0.82599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:12:05.908 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3f is not active+clean 2024-01-15T18:12:05.908 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.018152+0000', 'last_change': '2024-01-15T18:12:00.018152+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.018152+0000', 'last_undegraded': '2024-01-15T18:12:00.018152+0000', 'last_fullsized': '2024-01-15T18:12:00.018152+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-15T18:12:05.908 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3e is not active+clean 2024-01-15T18:12:05.908 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.010620+0000', 'last_change': '2024-01-15T18:12:00.010620+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.010620+0000', 'last_undegraded': '2024-01-15T18:12:00.010620+0000', 'last_fullsized': '2024-01-15T18:12:00.010620+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:12:05.908 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3d is not active+clean 2024-01-15T18:12:05.908 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.008063+0000', 'last_change': '2024-01-15T18:12:00.008063+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.008063+0000', 'last_undegraded': '2024-01-15T18:12:00.008063+0000', 'last_fullsized': '2024-01-15T18:12:00.008063+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:12:05.909 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3c is not active+clean 2024-01-15T18:12:05.909 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.006826+0000', 'last_change': '2024-01-15T18:12:00.006826+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.006826+0000', 'last_undegraded': '2024-01-15T18:12:00.006826+0000', 'last_fullsized': '2024-01-15T18:12:00.006826+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:12:05.909 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3b is not active+clean 2024-01-15T18:12:05.909 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.003755+0000', 'last_change': '2024-01-15T18:12:00.003755+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.003755+0000', 'last_undegraded': '2024-01-15T18:12:00.003755+0000', 'last_fullsized': '2024-01-15T18:12:00.003755+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:12:05.909 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3a is not active+clean 2024-01-15T18:12:05.909 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.008474+0000', 'last_change': '2024-01-15T18:12:00.008474+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.008474+0000', 'last_undegraded': '2024-01-15T18:12:00.008474+0000', 'last_fullsized': '2024-01-15T18:12:00.008474+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:12:05.909 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.38 is not active+clean 2024-01-15T18:12:05.909 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.009793+0000', 'last_change': '2024-01-15T18:12:00.009793+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.009793+0000', 'last_undegraded': '2024-01-15T18:12:00.009793+0000', 'last_fullsized': '2024-01-15T18:12:00.009793+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:12:05.909 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.37 is not active+clean 2024-01-15T18:12:05.909 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.005086+0000', 'last_change': '2024-01-15T18:12:00.005086+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.005086+0000', 'last_undegraded': '2024-01-15T18:12:00.005086+0000', 'last_fullsized': '2024-01-15T18:12:00.005086+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:12:05.909 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.36 is not active+clean 2024-01-15T18:12:05.909 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.006522+0000', 'last_change': '2024-01-15T18:12:00.006522+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.006522+0000', 'last_undegraded': '2024-01-15T18:12:00.006522+0000', 'last_fullsized': '2024-01-15T18:12:00.006522+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:12:05.910 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.35 is not active+clean 2024-01-15T18:12:05.910 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.017853+0000', 'last_change': '2024-01-15T18:12:00.017853+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.017853+0000', 'last_undegraded': '2024-01-15T18:12:00.017853+0000', 'last_fullsized': '2024-01-15T18:12:00.017853+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-15T18:12:05.910 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.34 is not active+clean 2024-01-15T18:12:05.910 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.006367+0000', 'last_change': '2024-01-15T18:12:00.006367+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.006367+0000', 'last_undegraded': '2024-01-15T18:12:00.006367+0000', 'last_fullsized': '2024-01-15T18:12:00.006367+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:12:05.910 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.33 is not active+clean 2024-01-15T18:12:05.910 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:11:59.986005+0000', 'last_change': '2024-01-15T18:11:59.986005+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:11:59.986005+0000', 'last_undegraded': '2024-01-15T18:11:59.986005+0000', 'last_fullsized': '2024-01-15T18:11:59.986005+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:12:05.910 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.32 is not active+clean 2024-01-15T18:12:05.910 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.004318+0000', 'last_change': '2024-01-15T18:12:00.004318+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.004318+0000', 'last_undegraded': '2024-01-15T18:12:00.004318+0000', 'last_fullsized': '2024-01-15T18:12:00.004318+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:12:05.910 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.31 is not active+clean 2024-01-15T18:12:05.910 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:11:59.986005+0000', 'last_change': '2024-01-15T18:11:59.986005+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:11:59.986005+0000', 'last_undegraded': '2024-01-15T18:11:59.986005+0000', 'last_fullsized': '2024-01-15T18:11:59.986005+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:12:05.910 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.30 is not active+clean 2024-01-15T18:12:05.911 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.004080+0000', 'last_change': '2024-01-15T18:12:00.004080+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.004080+0000', 'last_undegraded': '2024-01-15T18:12:00.004080+0000', 'last_fullsized': '2024-01-15T18:12:00.004080+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:12:05.911 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2f is not active+clean 2024-01-15T18:12:05.911 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.004665+0000', 'last_change': '2024-01-15T18:12:00.004665+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.004665+0000', 'last_undegraded': '2024-01-15T18:12:00.004665+0000', 'last_fullsized': '2024-01-15T18:12:00.004665+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:12:05.911 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2e is not active+clean 2024-01-15T18:12:05.911 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.004833+0000', 'last_change': '2024-01-15T18:12:00.004833+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.004833+0000', 'last_undegraded': '2024-01-15T18:12:00.004833+0000', 'last_fullsized': '2024-01-15T18:12:00.004833+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:12:05.911 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2d is not active+clean 2024-01-15T18:12:05.911 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.010019+0000', 'last_change': '2024-01-15T18:12:00.010019+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.010019+0000', 'last_undegraded': '2024-01-15T18:12:00.010019+0000', 'last_fullsized': '2024-01-15T18:12:00.010019+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:12:05.911 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2c is not active+clean 2024-01-15T18:12:05.911 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:11:59.986005+0000', 'last_change': '2024-01-15T18:11:59.986005+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:11:59.986005+0000', 'last_undegraded': '2024-01-15T18:11:59.986005+0000', 'last_fullsized': '2024-01-15T18:11:59.986005+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:12:05.911 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2b is not active+clean 2024-01-15T18:12:05.911 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.004967+0000', 'last_change': '2024-01-15T18:12:00.004967+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.004967+0000', 'last_undegraded': '2024-01-15T18:12:00.004967+0000', 'last_fullsized': '2024-01-15T18:12:00.004967+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:12:05.912 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2a is not active+clean 2024-01-15T18:12:05.912 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.005039+0000', 'last_change': '2024-01-15T18:12:00.005039+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.005039+0000', 'last_undegraded': '2024-01-15T18:12:00.005039+0000', 'last_fullsized': '2024-01-15T18:12:00.005039+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:12:05.912 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.29 is not active+clean 2024-01-15T18:12:05.912 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.014261+0000', 'last_change': '2024-01-15T18:12:00.014261+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.014261+0000', 'last_undegraded': '2024-01-15T18:12:00.014261+0000', 'last_fullsized': '2024-01-15T18:12:00.014261+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-15T18:12:05.912 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.28 is not active+clean 2024-01-15T18:12:05.912 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.014771+0000', 'last_change': '2024-01-15T18:12:00.014771+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.014771+0000', 'last_undegraded': '2024-01-15T18:12:00.014771+0000', 'last_fullsized': '2024-01-15T18:12:00.014771+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-15T18:12:05.912 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.27 is not active+clean 2024-01-15T18:12:05.912 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.005063+0000', 'last_change': '2024-01-15T18:12:00.005063+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.005063+0000', 'last_undegraded': '2024-01-15T18:12:00.005063+0000', 'last_fullsized': '2024-01-15T18:12:00.005063+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:12:05.912 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.25 is not active+clean 2024-01-15T18:12:05.912 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.007628+0000', 'last_change': '2024-01-15T18:12:00.007628+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.007628+0000', 'last_undegraded': '2024-01-15T18:12:00.007628+0000', 'last_fullsized': '2024-01-15T18:12:00.007628+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:12:05.913 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.24 is not active+clean 2024-01-15T18:12:05.913 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.003323+0000', 'last_change': '2024-01-15T18:12:00.003323+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.003323+0000', 'last_undegraded': '2024-01-15T18:12:00.003323+0000', 'last_fullsized': '2024-01-15T18:12:00.003323+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:12:05.913 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.23 is not active+clean 2024-01-15T18:12:05.913 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.006443+0000', 'last_change': '2024-01-15T18:12:00.006443+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.006443+0000', 'last_undegraded': '2024-01-15T18:12:00.006443+0000', 'last_fullsized': '2024-01-15T18:12:00.006443+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:12:05.913 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.20 is not active+clean 2024-01-15T18:12:05.913 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.009341+0000', 'last_change': '2024-01-15T18:12:00.009341+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.009341+0000', 'last_undegraded': '2024-01-15T18:12:00.009341+0000', 'last_fullsized': '2024-01-15T18:12:00.009341+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:12:05.913 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1e is not active+clean 2024-01-15T18:12:05.913 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.014983+0000', 'last_change': '2024-01-15T18:12:00.014983+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.014983+0000', 'last_undegraded': '2024-01-15T18:12:00.014983+0000', 'last_fullsized': '2024-01-15T18:12:00.014983+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-15T18:12:05.913 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1f is not active+clean 2024-01-15T18:12:05.913 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.004241+0000', 'last_change': '2024-01-15T18:12:00.004241+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.004241+0000', 'last_undegraded': '2024-01-15T18:12:00.004241+0000', 'last_fullsized': '2024-01-15T18:12:00.004241+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:12:05.913 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1e is not active+clean 2024-01-15T18:12:05.913 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:11:58.996438+0000', 'last_change': '2024-01-15T18:11:58.996438+0000', 'last_active': '2024-01-15T18:11:58.982718+0000', 'last_peered': '2024-01-15T18:11:58.982718+0000', 'last_clean': '2024-01-15T18:11:58.982718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:11:58.996438+0000', 'last_undegraded': '2024-01-15T18:11:58.996438+0000', 'last_fullsized': '2024-01-15T18:11:58.996438+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:58.982718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:58.982718+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:58.982718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:12:05.914 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1c is not active+clean 2024-01-15T18:12:05.914 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.008802+0000', 'last_change': '2024-01-15T18:12:00.008802+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.008802+0000', 'last_undegraded': '2024-01-15T18:12:00.008802+0000', 'last_fullsized': '2024-01-15T18:12:00.008802+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:12:05.914 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1d is not active+clean 2024-01-15T18:12:05.914 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.014971+0000', 'last_change': '2024-01-15T18:12:00.014971+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.014971+0000', 'last_undegraded': '2024-01-15T18:12:00.014971+0000', 'last_fullsized': '2024-01-15T18:12:00.014971+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-15T18:12:05.914 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1b is not active+clean 2024-01-15T18:12:05.914 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.004400+0000', 'last_change': '2024-01-15T18:12:00.004400+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.004400+0000', 'last_undegraded': '2024-01-15T18:12:00.004400+0000', 'last_fullsized': '2024-01-15T18:12:00.004400+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:12:05.914 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.18 is not active+clean 2024-01-15T18:12:05.914 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.004715+0000', 'last_change': '2024-01-15T18:12:00.004715+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.004715+0000', 'last_undegraded': '2024-01-15T18:12:00.004715+0000', 'last_fullsized': '2024-01-15T18:12:00.004715+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:12:05.914 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.19 is not active+clean 2024-01-15T18:12:05.914 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:11:59.986005+0000', 'last_change': '2024-01-15T18:11:59.986005+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:11:59.986005+0000', 'last_undegraded': '2024-01-15T18:11:59.986005+0000', 'last_fullsized': '2024-01-15T18:11:59.986005+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:12:05.914 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.17 is not active+clean 2024-01-15T18:12:05.915 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.017663+0000', 'last_change': '2024-01-15T18:12:00.017663+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.017663+0000', 'last_undegraded': '2024-01-15T18:12:00.017663+0000', 'last_fullsized': '2024-01-15T18:12:00.017663+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-15T18:12:05.915 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.14 is not active+clean 2024-01-15T18:12:05.915 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.008315+0000', 'last_change': '2024-01-15T18:12:00.008315+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.008315+0000', 'last_undegraded': '2024-01-15T18:12:00.008315+0000', 'last_fullsized': '2024-01-15T18:12:00.008315+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:12:05.915 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.15 is not active+clean 2024-01-15T18:12:05.915 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.004379+0000', 'last_change': '2024-01-15T18:12:00.004379+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.004379+0000', 'last_undegraded': '2024-01-15T18:12:00.004379+0000', 'last_fullsized': '2024-01-15T18:12:00.004379+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:12:05.915 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.12 is not active+clean 2024-01-15T18:12:05.915 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.013658+0000', 'last_change': '2024-01-15T18:12:00.013658+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.013658+0000', 'last_undegraded': '2024-01-15T18:12:00.013658+0000', 'last_fullsized': '2024-01-15T18:12:00.013658+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-15T18:12:05.915 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.10 is not active+clean 2024-01-15T18:12:05.915 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:11:59.986005+0000', 'last_change': '2024-01-15T18:11:59.986005+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:11:59.986005+0000', 'last_undegraded': '2024-01-15T18:11:59.986005+0000', 'last_fullsized': '2024-01-15T18:11:59.986005+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:12:05.915 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.11 is not active+clean 2024-01-15T18:12:05.915 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.003520+0000', 'last_change': '2024-01-15T18:12:00.003520+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.003520+0000', 'last_undegraded': '2024-01-15T18:12:00.003520+0000', 'last_fullsized': '2024-01-15T18:12:00.003520+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:12:05.916 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.e is not active+clean 2024-01-15T18:12:05.916 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.005892+0000', 'last_change': '2024-01-15T18:12:00.005892+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.005892+0000', 'last_undegraded': '2024-01-15T18:12:00.005892+0000', 'last_fullsized': '2024-01-15T18:12:00.005892+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:12:05.916 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.f is not active+clean 2024-01-15T18:12:05.916 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.016990+0000', 'last_change': '2024-01-15T18:12:00.016990+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.016990+0000', 'last_undegraded': '2024-01-15T18:12:00.016990+0000', 'last_fullsized': '2024-01-15T18:12:00.016990+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-15T18:12:05.916 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.c is not active+clean 2024-01-15T18:12:05.916 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.003474+0000', 'last_change': '2024-01-15T18:12:00.003474+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.003474+0000', 'last_undegraded': '2024-01-15T18:12:00.003474+0000', 'last_fullsized': '2024-01-15T18:12:00.003474+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:12:05.916 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.d is not active+clean 2024-01-15T18:12:05.916 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.003892+0000', 'last_change': '2024-01-15T18:12:00.003892+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.003892+0000', 'last_undegraded': '2024-01-15T18:12:00.003892+0000', 'last_fullsized': '2024-01-15T18:12:00.003892+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:12:05.916 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.b is not active+clean 2024-01-15T18:12:05.916 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.006003+0000', 'last_change': '2024-01-15T18:12:00.006003+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.006003+0000', 'last_undegraded': '2024-01-15T18:12:00.006003+0000', 'last_fullsized': '2024-01-15T18:12:00.006003+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:12:05.916 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.8 is not active+clean 2024-01-15T18:12:05.916 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.005765+0000', 'last_change': '2024-01-15T18:12:00.005765+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.005765+0000', 'last_undegraded': '2024-01-15T18:12:00.005765+0000', 'last_fullsized': '2024-01-15T18:12:00.005765+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:12:05.917 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.9 is not active+clean 2024-01-15T18:12:05.917 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.005856+0000', 'last_change': '2024-01-15T18:12:00.005856+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.005856+0000', 'last_undegraded': '2024-01-15T18:12:00.005856+0000', 'last_fullsized': '2024-01-15T18:12:00.005856+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:12:05.917 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.8 is not active+clean 2024-01-15T18:12:05.917 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:11:58.996468+0000', 'last_change': '2024-01-15T18:11:58.996468+0000', 'last_active': '2024-01-15T18:11:58.982718+0000', 'last_peered': '2024-01-15T18:11:58.982718+0000', 'last_clean': '2024-01-15T18:11:58.982718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:11:58.996468+0000', 'last_undegraded': '2024-01-15T18:11:58.996468+0000', 'last_fullsized': '2024-01-15T18:11:58.996468+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:58.982718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:58.982718+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:58.982718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:12:05.917 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.6 is not active+clean 2024-01-15T18:12:05.917 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:11:59.986005+0000', 'last_change': '2024-01-15T18:11:59.986005+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:11:59.986005+0000', 'last_undegraded': '2024-01-15T18:11:59.986005+0000', 'last_fullsized': '2024-01-15T18:11:59.986005+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:12:05.917 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.4 is not active+clean 2024-01-15T18:12:05.917 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.016823+0000', 'last_change': '2024-01-15T18:12:00.016823+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.016823+0000', 'last_undegraded': '2024-01-15T18:12:00.016823+0000', 'last_fullsized': '2024-01-15T18:12:00.016823+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-15T18:12:05.917 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.5 is not active+clean 2024-01-15T18:12:05.917 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.014451+0000', 'last_change': '2024-01-15T18:12:00.014451+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.014451+0000', 'last_undegraded': '2024-01-15T18:12:00.014451+0000', 'last_fullsized': '2024-01-15T18:12:00.014451+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-15T18:12:05.917 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.4 is not active+clean 2024-01-15T18:12:05.918 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 67, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:11:58.997807+0000', 'last_change': '2024-01-15T18:11:58.997807+0000', 'last_active': '2024-01-15T18:11:58.982718+0000', 'last_peered': '2024-01-15T18:11:58.982718+0000', 'last_clean': '2024-01-15T18:11:58.982718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:11:58.997807+0000', 'last_undegraded': '2024-01-15T18:11:58.997807+0000', 'last_fullsized': '2024-01-15T18:11:58.997807+0000', 'mapping_epoch': 67, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 67, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:58.982718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:58.982718+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:58.982718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:12:05.918 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2 is not active+clean 2024-01-15T18:12:05.918 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.003849+0000', 'last_change': '2024-01-15T18:12:00.003849+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.003849+0000', 'last_undegraded': '2024-01-15T18:12:00.003849+0000', 'last_fullsized': '2024-01-15T18:12:00.003849+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:12:05.918 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3 is not active+clean 2024-01-15T18:12:05.918 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.007471+0000', 'last_change': '2024-01-15T18:12:00.007471+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.007471+0000', 'last_undegraded': '2024-01-15T18:12:00.007471+0000', 'last_fullsized': '2024-01-15T18:12:00.007471+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:12:05.918 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.0 is not active+clean 2024-01-15T18:12:05.918 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.004040+0000', 'last_change': '2024-01-15T18:12:00.004040+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.004040+0000', 'last_undegraded': '2024-01-15T18:12:00.004040+0000', 'last_fullsized': '2024-01-15T18:12:00.004040+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:12:05.918 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1 is not active+clean 2024-01-15T18:12:05.918 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.007044+0000', 'last_change': '2024-01-15T18:12:00.007044+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.007044+0000', 'last_undegraded': '2024-01-15T18:12:00.007044+0000', 'last_fullsized': '2024-01-15T18:12:00.007044+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:12:05.918 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.a is not active+clean 2024-01-15T18:12:05.918 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 68, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:12:00.009189+0000', 'last_change': '2024-01-15T18:12:00.009189+0000', 'last_active': '2024-01-15T18:11:59.986005+0000', 'last_peered': '2024-01-15T18:11:59.986005+0000', 'last_clean': '2024-01-15T18:11:59.986005+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:12:00.009189+0000', 'last_undegraded': '2024-01-15T18:12:00.009189+0000', 'last_fullsized': '2024-01-15T18:12:00.009189+0000', 'mapping_epoch': 68, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 68, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'last_clean_scrub_stamp': '2024-01-15T18:11:59.986005+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:12:05.919 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-15T18:12:06.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.083+0000 7f0b64c00640 1 Processor -- start 2024-01-15T18:12:06.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.083+0000 7f0b64c00640 1 -- start start 2024-01-15T18:12:06.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.087+0000 7f0b64c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0b6015e7d0 0x7f0b600cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:06.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.087+0000 7f0b64c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0b600cedc0 0x7f0b600cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:06.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.087+0000 7f0b64c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b600b38b0 con 0x7f0b600ce2e0 2024-01-15T18:12:06.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.087+0000 7f0b64c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b600cd550 con 0x7f0b600cedc0 2024-01-15T18:12:06.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.087+0000 7f0b64c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b600cd270 con 0x7f0b6015e7d0 2024-01-15T18:12:06.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.087+0000 7f0b5d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0b600cedc0 0x7f0b600cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:06.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.087+0000 7f0b5d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0b600cedc0 0x7f0b600cf7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58330/0 (socket says 172.21.15.27:58330) 2024-01-15T18:12:06.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.087+0000 7f0b5d400640 1 -- 172.21.15.27:0/628150752 learned_addr learned my addr 172.21.15.27:0/628150752 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:06.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.087+0000 7f0b5de00640 1 --2- 172.21.15.27:0/628150752 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0b6015e7d0 0x7f0b600cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:06.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.087+0000 7f0b5d400640 1 -- 172.21.15.27:0/628150752 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0b6015e7d0 msgr2=0x7f0b600cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:06.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.087+0000 7f0b5d400640 1 --2- 172.21.15.27:0/628150752 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0b6015e7d0 0x7f0b600cf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.087+0000 7f0b5d400640 1 -- 172.21.15.27:0/628150752 >> v1:172.21.15.93:6789/0 conn(0x7f0b600ce2e0 legacy=0x7f0b600cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:06.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.087+0000 7f0b5d400640 1 -- 172.21.15.27:0/628150752 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b600b35d0 con 0x7f0b600cedc0 2024-01-15T18:12:06.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.091+0000 7f0b5d400640 1 --2- 172.21.15.27:0/628150752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b600cedc0 0x7f0b600cf7e0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f0b48009870 tx=0x7f0b4801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78931f7aa3a4ce7b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:06.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.091+0000 7f0b5ca00640 1 -- 172.21.15.27:0/628150752 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b480a7070 con 0x7f0b600cedc0 2024-01-15T18:12:06.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.091+0000 7f0b5ca00640 1 -- 172.21.15.27:0/628150752 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b480ab430 con 0x7f0b600cedc0 2024-01-15T18:12:06.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.091+0000 7f0b5ca00640 1 -- 172.21.15.27:0/628150752 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b480ab7c0 con 0x7f0b600cedc0 2024-01-15T18:12:06.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.091+0000 7f0b64c00640 1 -- 172.21.15.27:0/628150752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b600cedc0 msgr2=0x7f0b600cf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:06.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.091+0000 7f0b64c00640 1 --2- 172.21.15.27:0/628150752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b600cedc0 0x7f0b600cf7e0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f0b48009870 tx=0x7f0b4801c540 comp rx=0 tx=0).stop 2024-01-15T18:12:06.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.091+0000 7f0b64c00640 1 -- 172.21.15.27:0/628150752 shutdown_connections 2024-01-15T18:12:06.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.091+0000 7f0b64c00640 1 --2- 172.21.15.27:0/628150752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b600cedc0 0x7f0b600cf7e0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.091+0000 7f0b64c00640 1 --2- 172.21.15.27:0/628150752 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0b6015e7d0 0x7f0b600cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.091+0000 7f0b64c00640 1 -- 172.21.15.27:0/628150752 >> 172.21.15.27:0/628150752 conn(0x7f0b600b2a10 msgr2=0x7f0b600b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:06.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.091+0000 7f0b64c00640 1 -- 172.21.15.27:0/628150752 shutdown_connections 2024-01-15T18:12:06.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.091+0000 7f0b64c00640 1 -- 172.21.15.27:0/628150752 wait complete. 2024-01-15T18:12:06.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b64c00640 1 Processor -- start 2024-01-15T18:12:06.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b64c00640 1 -- start start 2024-01-15T18:12:06.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b64c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0b600ce2e0 0x7f0b60157660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:06.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b64c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0b600cedc0 0x7f0b600c6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:06.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b64c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b6015e7d0 0x7f0b6006f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:06.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b64c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b600c61e0 con 0x7f0b6015e7d0 2024-01-15T18:12:06.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b64c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b600c58f0 con 0x7f0b600ce2e0 2024-01-15T18:12:06.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b64c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b600c5b90 con 0x7f0b600cedc0 2024-01-15T18:12:06.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b5e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b6015e7d0 0x7f0b6006f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:06.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b5d400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0b600cedc0 0x7f0b600c6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:06.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b5e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b6015e7d0 0x7f0b6006f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58336/0 (socket says 172.21.15.27:58336) 2024-01-15T18:12:06.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b5e800640 1 -- 172.21.15.27:0/1280774480 learned_addr learned my addr 172.21.15.27:0/1280774480 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:06.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b5de00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0b600ce2e0 0x7f0b60157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:06.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b5d400640 1 -- 172.21.15.27:0/1280774480 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0b600ce2e0 msgr2=0x7f0b60157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:06.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b5d400640 1 --2- 172.21.15.27:0/1280774480 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0b600ce2e0 0x7f0b60157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b5d400640 1 -- 172.21.15.27:0/1280774480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b6015e7d0 msgr2=0x7f0b6006f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:06.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b5d400640 1 --2- 172.21.15.27:0/1280774480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b6015e7d0 0x7f0b6006f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b5d400640 1 -- 172.21.15.27:0/1280774480 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b480094d0 con 0x7f0b600cedc0 2024-01-15T18:12:06.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b5e800640 1 --2- 172.21.15.27:0/1280774480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b6015e7d0 0x7f0b6006f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:12:06.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b5d400640 1 --2- 172.21.15.27:0/1280774480 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0b600cedc0 0x7f0b600c6670 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0b480ac000 tx=0x7f0b480192e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:06.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b56000640 1 -- 172.21.15.27:0/1280774480 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b480b4620 con 0x7f0b600cedc0 2024-01-15T18:12:06.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b56000640 1 -- 172.21.15.27:0/1280774480 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b480b47a0 con 0x7f0b600cedc0 2024-01-15T18:12:06.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b56000640 1 -- 172.21.15.27:0/1280774480 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b480b4b50 con 0x7f0b600cedc0 2024-01-15T18:12:06.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b64c00640 1 -- 172.21.15.27:0/1280774480 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b600c5e70 con 0x7f0b600cedc0 2024-01-15T18:12:06.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b64c00640 1 -- 172.21.15.27:0/1280774480 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b6006f910 con 0x7f0b600cedc0 2024-01-15T18:12:06.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.095+0000 7f0b64c00640 1 -- 172.21.15.27:0/1280774480 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b600c6350 con 0x7f0b600cedc0 2024-01-15T18:12:06.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.099+0000 7f0b56000640 1 -- 172.21.15.27:0/1280774480 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f0b480bf3a0 con 0x7f0b600cedc0 2024-01-15T18:12:06.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.099+0000 7f0b56000640 1 --2- 172.21.15.27:0/1280774480 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0b38067100 0x7f0b380695c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:06.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.099+0000 7f0b56000640 1 -- 172.21.15.27:0/1280774480 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f0b4811c360 con 0x7f0b600cedc0 2024-01-15T18:12:06.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.099+0000 7f0b5de00640 1 --2- 172.21.15.27:0/1280774480 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0b38067100 0x7f0b380695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:06.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.099+0000 7f0b5de00640 1 --2- 172.21.15.27:0/1280774480 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0b38067100 0x7f0b380695c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0b50009ac0 tx=0x7f0b50009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:06.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.103+0000 7f0b56000640 1 -- 172.21.15.27:0/1280774480 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0b480eb250 con 0x7f0b600cedc0 2024-01-15T18:12:06.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.315+0000 7f0b64c00640 1 -- 172.21.15.27:0/1280774480 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f0b6004e850 con 0x7f0b600cedc0 2024-01-15T18:12:06.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.319+0000 7f0b56000640 1 -- 172.21.15.27:0/1280774480 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1110 (secure 0 0 0) 0x7f0b480ee870 con 0x7f0b600cedc0 2024-01-15T18:12:06.319 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:12:06.320 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":453,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":70,"num_osds":8,"num_up_osds":8,"osd_up_since":1705341884,"num_in_osds":8,"osd_in_since":1705341875,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":53},{"state_name":"active+clean","count":38},{"state_name":"unknown","count":6}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":406740992,"bytes_avail":772687372288,"bytes_total":773094113280,"unknown_pgs_ratio":0.061855670064687729,"inactive_pgs_ratio":0.54639172554016113},"fsmap":{"epoch":47,"id":5,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":5,"rank":0,"name":"c","status":"up:active","gid":7227}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-01-15T18:10:16.226140+0000","services":{}},"progress_events":{}} 2024-01-15T18:12:06.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.319+0000 7f0b64c00640 1 -- 172.21.15.27:0/1280774480 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0b38067100 msgr2=0x7f0b380695c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:06.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.319+0000 7f0b64c00640 1 --2- 172.21.15.27:0/1280774480 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0b38067100 0x7f0b380695c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0b50009ac0 tx=0x7f0b50009190 comp rx=0 tx=0).stop 2024-01-15T18:12:06.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.319+0000 7f0b64c00640 1 -- 172.21.15.27:0/1280774480 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0b600cedc0 msgr2=0x7f0b600c6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:06.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.319+0000 7f0b64c00640 1 --2- 172.21.15.27:0/1280774480 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0b600cedc0 0x7f0b600c6670 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0b480ac000 tx=0x7f0b480192e0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.319+0000 7f0b64c00640 1 -- 172.21.15.27:0/1280774480 shutdown_connections 2024-01-15T18:12:06.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.319+0000 7f0b64c00640 1 --2- 172.21.15.27:0/1280774480 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0b38067100 0x7f0b380695c0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.319+0000 7f0b64c00640 1 --2- 172.21.15.27:0/1280774480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b6015e7d0 0x7f0b6006f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.319+0000 7f0b64c00640 1 --2- 172.21.15.27:0/1280774480 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0b600cedc0 0x7f0b600c6670 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.319+0000 7f0b64c00640 1 --2- 172.21.15.27:0/1280774480 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0b600ce2e0 0x7f0b60157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.319+0000 7f0b64c00640 1 -- 172.21.15.27:0/1280774480 >> 172.21.15.27:0/1280774480 conn(0x7f0b600b2a10 msgr2=0x7f0b6015d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:06.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.319+0000 7f0b64c00640 1 -- 172.21.15.27:0/1280774480 shutdown_connections 2024-01-15T18:12:06.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.319+0000 7f0b64c00640 1 -- 172.21.15.27:0/1280774480 wait complete. 2024-01-15T18:12:06.342 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-01-15T18:12:06.342 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:12:06.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.479+0000 7fa0d0800640 1 Processor -- start 2024-01-15T18:12:06.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0d0800640 1 -- start start 2024-01-15T18:12:06.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0d0800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa0c80afe70 0x7fa0c80cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:06.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0d0800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c80d05b0 con 0x7fa0c80cb7b0 2024-01-15T18:12:06.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0d0800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c80ce7d0 con 0x7fa0c8160b10 2024-01-15T18:12:06.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0d0800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0c80cfc80 con 0x7fa0c80afe70 2024-01-15T18:12:06.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0ce800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa0c80cb7b0 0x7fa0c80cbc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59438/0 (socket says 172.21.15.27:59438) 2024-01-15T18:12:06.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0ce800640 1 -- 172.21.15.27:0/633001 learned_addr learned my addr 172.21.15.27:0/633001 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:12:06.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0cd400640 1 --2- 172.21.15.27:0/633001 >> v2:172.21.15.93:3300/0 conn(0x7fa0c80afe70 0x7fa0c80cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:06.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0cca00640 1 -- 172.21.15.27:0/633001 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416289658 0 0) 0x7fa0c80d05b0 con 0x7fa0c80cb7b0 2024-01-15T18:12:06.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0cca00640 1 -- 172.21.15.27:0/633001 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0a8003530 con 0x7fa0c80cb7b0 2024-01-15T18:12:06.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0cca00640 1 -- 172.21.15.27:0/633001 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1582303684 0 0) 0x7fa0a8003530 con 0x7fa0c80cb7b0 2024-01-15T18:12:06.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0cca00640 1 -- 172.21.15.27:0/633001 >> v2:172.21.15.93:3300/0 conn(0x7fa0c80afe70 msgr2=0x7fa0c80cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:06.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0cca00640 1 --2- 172.21.15.27:0/633001 >> v2:172.21.15.93:3300/0 conn(0x7fa0c80afe70 0x7fa0c80cdeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0cca00640 1 -- 172.21.15.27:0/633001 >> v1:172.21.15.93:6789/0 conn(0x7fa0c8160b10 legacy=0x7fa0c80cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:06.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0cca00640 1 -- 172.21.15.27:0/633001 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c80cff60 con 0x7fa0c80cb7b0 2024-01-15T18:12:06.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0cca00640 1 -- 172.21.15.27:0/633001 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fa0c0001a00 con 0x7fa0c80cb7b0 2024-01-15T18:12:06.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0cca00640 1 -- 172.21.15.27:0/633001 >> v1:172.21.15.27:6789/0 conn(0x7fa0c80cb7b0 legacy=0x7fa0c80cbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:06.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0cca00640 1 --2- 172.21.15.27:0/633001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0a80039f0 0x7fa0a8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:06.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0cca00640 1 --2- 172.21.15.27:0/633001 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa0a80047d0 0x7fa0a8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:06.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0cca00640 1 --2- 172.21.15.27:0/633001 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa0a8005200 0x7fa0a800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:06.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0cca00640 1 -- 172.21.15.27:0/633001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c80cfc80 con 0x7fa0a80039f0 2024-01-15T18:12:06.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0cca00640 1 -- 172.21.15.27:0/633001 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a800c160 con 0x7fa0a80047d0 2024-01-15T18:12:06.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0cca00640 1 -- 172.21.15.27:0/633001 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a800c320 con 0x7fa0a8005200 2024-01-15T18:12:06.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0cde00640 1 --2- 172.21.15.27:0/633001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0a80039f0 0x7fa0a8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:06.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0cd400640 1 --2- 172.21.15.27:0/633001 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa0a80047d0 0x7fa0a8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:06.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0ce800640 1 --2- 172.21.15.27:0/633001 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa0a8005200 0x7fa0a800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:06.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0cde00640 1 -- 172.21.15.27:0/633001 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa0a8005200 msgr2=0x7fa0a800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:06.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0cde00640 1 --2- 172.21.15.27:0/633001 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa0a8005200 0x7fa0a800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0cde00640 1 -- 172.21.15.27:0/633001 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa0a80047d0 msgr2=0x7fa0a8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:06.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0cde00640 1 --2- 172.21.15.27:0/633001 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa0a80047d0 0x7fa0a8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0cde00640 1 -- 172.21.15.27:0/633001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a8003530 con 0x7fa0a80039f0 2024-01-15T18:12:06.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0ce800640 1 --2- 172.21.15.27:0/633001 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa0a8005200 0x7fa0a800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:12:06.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.483+0000 7fa0ce800640 1 -- 172.21.15.27:0/633001 reap_dead start 2024-01-15T18:12:06.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.487+0000 7fa0cde00640 1 --2- 172.21.15.27:0/633001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0a80039f0 0x7fa0a8003e10 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8001780 tx=0x7fa0b801c2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c658d37e49e9f85d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.487+0000 7fa0cca00640 1 -- 172.21.15.27:0/633001 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b80052d0 con 0x7fa0a80039f0 2024-01-15T18:12:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.487+0000 7fa0cca00640 1 -- 172.21.15.27:0/633001 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b80a7050 con 0x7fa0a80039f0 2024-01-15T18:12:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.487+0000 7fa0cca00640 1 -- 172.21.15.27:0/633001 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8003600 con 0x7fa0a80039f0 2024-01-15T18:12:06.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.487+0000 7fa0d0800640 1 -- 172.21.15.27:0/633001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0a80039f0 msgr2=0x7fa0a8003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:06.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.487+0000 7fa0d0800640 1 --2- 172.21.15.27:0/633001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0a80039f0 0x7fa0a8003e10 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8001780 tx=0x7fa0b801c2f0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.487+0000 7fa0d0800640 1 -- 172.21.15.27:0/633001 shutdown_connections 2024-01-15T18:12:06.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.487+0000 7fa0d0800640 1 --2- 172.21.15.27:0/633001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0a80039f0 0x7fa0a8003e10 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.487+0000 7fa0d0800640 1 -- 172.21.15.27:0/633001 >> 172.21.15.27:0/633001 conn(0x7fa0c80d0f20 msgr2=0x7fa0c80d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:06.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.487+0000 7fa0d0800640 1 -- 172.21.15.27:0/633001 shutdown_connections 2024-01-15T18:12:06.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.487+0000 7fa0d0800640 1 -- 172.21.15.27:0/633001 wait complete. 2024-01-15T18:12:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.491+0000 7fa0d0800640 1 Processor -- start 2024-01-15T18:12:06.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.491+0000 7fa0d0800640 1 -- start start 2024-01-15T18:12:06.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.491+0000 7fa0d0800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0a80039f0 0x7fa0c81ea8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:06.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.491+0000 7fa0d0800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa0c8152f80 0x7fa0c815ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:06.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.491+0000 7fa0d0800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa0c815f340 0x7fa0c8157710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:06.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.491+0000 7fa0d0800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c81ea420 con 0x7fa0a80039f0 2024-01-15T18:12:06.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.491+0000 7fa0d0800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c81e9810 con 0x7fa0c8152f80 2024-01-15T18:12:06.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.491+0000 7fa0d0800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c81e9af0 con 0x7fa0c815f340 2024-01-15T18:12:06.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.491+0000 7fa0cde00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0a80039f0 0x7fa0c81ea8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:06.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.491+0000 7fa0cde00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0a80039f0 0x7fa0c81ea8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58350/0 (socket says 172.21.15.27:58350) 2024-01-15T18:12:06.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.491+0000 7fa0cde00640 1 -- 172.21.15.27:0/743201730 learned_addr learned my addr 172.21.15.27:0/743201730 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:06.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.491+0000 7fa0cd400640 1 --2- 172.21.15.27:0/743201730 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa0c8152f80 0x7fa0c815ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:06.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.491+0000 7fa0ce800640 1 --2- 172.21.15.27:0/743201730 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa0c815f340 0x7fa0c8157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:06.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.491+0000 7fa0cde00640 1 -- 172.21.15.27:0/743201730 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa0c815f340 msgr2=0x7fa0c8157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.491+0000 7fa0cde00640 1 --2- 172.21.15.27:0/743201730 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa0c815f340 0x7fa0c8157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.491+0000 7fa0cde00640 1 -- 172.21.15.27:0/743201730 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa0c8152f80 msgr2=0x7fa0c815ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.491+0000 7fa0cde00640 1 --2- 172.21.15.27:0/743201730 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa0c8152f80 0x7fa0c815ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.491+0000 7fa0cde00640 1 -- 172.21.15.27:0/743201730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c81e9dd0 con 0x7fa0a80039f0 2024-01-15T18:12:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.491+0000 7fa0cde00640 1 --2- 172.21.15.27:0/743201730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0a80039f0 0x7fa0c81ea8b0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fa0b801c120 tx=0x7fa0a8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.491+0000 7fa0c6000640 1 -- 172.21.15.27:0/743201730 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8007230 con 0x7fa0a80039f0 2024-01-15T18:12:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.491+0000 7fa0c6000640 1 -- 172.21.15.27:0/743201730 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b8007390 con 0x7fa0a80039f0 2024-01-15T18:12:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.491+0000 7fa0c6000640 1 -- 172.21.15.27:0/743201730 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b80067e0 con 0x7fa0a80039f0 2024-01-15T18:12:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.491+0000 7fa0d0800640 1 -- 172.21.15.27:0/743201730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c81ea0b0 con 0x7fa0a80039f0 2024-01-15T18:12:06.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.491+0000 7fa0d0800640 1 -- 172.21.15.27:0/743201730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c81603b0 con 0x7fa0a80039f0 2024-01-15T18:12:06.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.495+0000 7fa0d0800640 1 -- 172.21.15.27:0/743201730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c81ea240 con 0x7fa0a80039f0 2024-01-15T18:12:06.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.495+0000 7fa0c6000640 1 -- 172.21.15.27:0/743201730 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7fa0b8006940 con 0x7fa0a80039f0 2024-01-15T18:12:06.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.495+0000 7fa0c6000640 1 --2- 172.21.15.27:0/743201730 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0a00670d0 0x7fa0a0069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:06.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.495+0000 7fa0c6000640 1 -- 172.21.15.27:0/743201730 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fa0b811caa0 con 0x7fa0a80039f0 2024-01-15T18:12:06.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.495+0000 7fa0cd400640 1 --2- 172.21.15.27:0/743201730 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0a00670d0 0x7fa0a0069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:06.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.495+0000 7fa0cd400640 1 --2- 172.21.15.27:0/743201730 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0a00670d0 0x7fa0a0069590 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa0c815f9c0 tx=0x7fa0b0009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:06.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.499+0000 7fa0c6000640 1 -- 172.21.15.27:0/743201730 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa0b80eb8e0 con 0x7fa0a80039f0 2024-01-15T18:12:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.655+0000 7fa0d0800640 1 -- 172.21.15.27:0/743201730 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa0c80d0cc0 con 0x7fa0a00670d0 2024-01-15T18:12:06.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.663+0000 7fa0c6000640 1 -- 172.21.15.27:0/743201730 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288658 (secure 0 0 0) 0x7fa0b0019070 con 0x7fa0a00670d0 2024-01-15T18:12:06.668 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:12:06.670 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:12:06.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.667+0000 7fa0d0800640 1 -- 172.21.15.27:0/743201730 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0a00670d0 msgr2=0x7fa0a0069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:06.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.667+0000 7fa0d0800640 1 --2- 172.21.15.27:0/743201730 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0a00670d0 0x7fa0a0069590 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa0c815f9c0 tx=0x7fa0b0009190 comp rx=0 tx=0).stop 2024-01-15T18:12:06.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.667+0000 7fa0d0800640 1 -- 172.21.15.27:0/743201730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0a80039f0 msgr2=0x7fa0c81ea8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:06.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.667+0000 7fa0d0800640 1 --2- 172.21.15.27:0/743201730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0a80039f0 0x7fa0c81ea8b0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fa0b801c120 tx=0x7fa0a8005800 comp rx=0 tx=0).stop 2024-01-15T18:12:06.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.667+0000 7fa0d0800640 1 -- 172.21.15.27:0/743201730 shutdown_connections 2024-01-15T18:12:06.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.667+0000 7fa0d0800640 1 --2- 172.21.15.27:0/743201730 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0a00670d0 0x7fa0a0069590 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.667+0000 7fa0d0800640 1 --2- 172.21.15.27:0/743201730 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa0c815f340 0x7fa0c8157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.667+0000 7fa0d0800640 1 --2- 172.21.15.27:0/743201730 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa0c8152f80 0x7fa0c815ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.667+0000 7fa0d0800640 1 --2- 172.21.15.27:0/743201730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0a80039f0 0x7fa0c81ea8b0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.667+0000 7fa0d0800640 1 -- 172.21.15.27:0/743201730 >> 172.21.15.27:0/743201730 conn(0x7fa0c80d0f20 msgr2=0x7fa0c806fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:06.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.667+0000 7fa0d0800640 1 -- 172.21.15.27:0/743201730 shutdown_connections 2024-01-15T18:12:06.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:06.667+0000 7fa0d0800640 1 -- 172.21.15.27:0/743201730 wait complete. 2024-01-15T18:12:06.688 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":284,"stamp":"2024-01-15T18:12:06.259617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":397500,"kb_used_data":7816,"kb_used_omap":12,"kb_used_meta":389555,"kb_avail":754577220,"statfs":{"total":773094113280,"available":772687073280,"internally_reserved":0,"allocated":8003584,"data_stored":4048532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":398904880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.259024"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756789+0000","last_change":"2024-01-15T18:12:01.024863+0000","last_active":"2024-01-15T18:12:02.756789+0000","last_peered":"2024-01-15T18:12:02.756789+0000","last_clean":"2024-01-15T18:12:02.756789+0000","last_became_active":"2024-01-15T18:12:01.024550+0000","last_became_peered":"2024-01-15T18:12:01.024550+0000","last_unstale":"2024-01-15T18:12:02.756789+0000","last_undegraded":"2024-01-15T18:12:02.756789+0000","last_fullsized":"2024-01-15T18:12:02.756789+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:25:17.902934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.757395+0000","last_change":"2024-01-15T18:12:01.011995+0000","last_active":"2024-01-15T18:12:02.757395+0000","last_peered":"2024-01-15T18:12:02.757395+0000","last_clean":"2024-01-15T18:12:02.757395+0000","last_became_active":"2024-01-15T18:12:01.011593+0000","last_became_peered":"2024-01-15T18:12:01.011593+0000","last_unstale":"2024-01-15T18:12:02.757395+0000","last_undegraded":"2024-01-15T18:12:02.757395+0000","last_fullsized":"2024-01-15T18:12:02.757395+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:18:04.676758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756930+0000","last_change":"2024-01-15T18:12:01.027262+0000","last_active":"2024-01-15T18:12:02.756930+0000","last_peered":"2024-01-15T18:12:02.756930+0000","last_clean":"2024-01-15T18:12:02.756930+0000","last_became_active":"2024-01-15T18:12:01.026976+0000","last_became_peered":"2024-01-15T18:12:01.026976+0000","last_unstale":"2024-01-15T18:12:02.756930+0000","last_undegraded":"2024-01-15T18:12:02.756930+0000","last_fullsized":"2024-01-15T18:12:02.756930+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:02:46.221698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756314+0000","last_change":"2024-01-15T18:12:01.023153+0000","last_active":"2024-01-15T18:12:02.756314+0000","last_peered":"2024-01-15T18:12:02.756314+0000","last_clean":"2024-01-15T18:12:02.756314+0000","last_became_active":"2024-01-15T18:12:01.022804+0000","last_became_peered":"2024-01-15T18:12:01.022804+0000","last_unstale":"2024-01-15T18:12:02.756314+0000","last_undegraded":"2024-01-15T18:12:02.756314+0000","last_fullsized":"2024-01-15T18:12:02.756314+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:26.599661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.007112+0000","last_change":"2024-01-15T18:12:01.030691+0000","last_active":"2024-01-15T18:12:02.007112+0000","last_peered":"2024-01-15T18:12:02.007112+0000","last_clean":"2024-01-15T18:12:02.007112+0000","last_became_active":"2024-01-15T18:12:01.030418+0000","last_became_peered":"2024-01-15T18:12:01.030418+0000","last_unstale":"2024-01-15T18:12:02.007112+0000","last_undegraded":"2024-01-15T18:12:02.007112+0000","last_fullsized":"2024-01-15T18:12:02.007112+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:54:50.644674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.755789+0000","last_change":"2024-01-15T18:12:01.011618+0000","last_active":"2024-01-15T18:12:02.755789+0000","last_peered":"2024-01-15T18:12:02.755789+0000","last_clean":"2024-01-15T18:12:02.755789+0000","last_became_active":"2024-01-15T18:12:01.011331+0000","last_became_peered":"2024-01-15T18:12:01.011331+0000","last_unstale":"2024-01-15T18:12:02.755789+0000","last_undegraded":"2024-01-15T18:12:02.755789+0000","last_fullsized":"2024-01-15T18:12:02.755789+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:12:20.020722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005570+0000","last_change":"2024-01-15T18:12:01.023552+0000","last_active":"2024-01-15T18:12:02.005570+0000","last_peered":"2024-01-15T18:12:02.005570+0000","last_clean":"2024-01-15T18:12:02.005570+0000","last_became_active":"2024-01-15T18:12:01.023332+0000","last_became_peered":"2024-01-15T18:12:01.023332+0000","last_unstale":"2024-01-15T18:12:02.005570+0000","last_undegraded":"2024-01-15T18:12:02.005570+0000","last_fullsized":"2024-01-15T18:12:02.005570+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:13:50.232455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.755868+0000","last_change":"2024-01-15T18:12:01.026981+0000","last_active":"2024-01-15T18:12:02.755868+0000","last_peered":"2024-01-15T18:12:02.755868+0000","last_clean":"2024-01-15T18:12:02.755868+0000","last_became_active":"2024-01-15T18:12:01.026616+0000","last_became_peered":"2024-01-15T18:12:01.026616+0000","last_unstale":"2024-01-15T18:12:02.755868+0000","last_undegraded":"2024-01-15T18:12:02.755868+0000","last_fullsized":"2024-01-15T18:12:02.755868+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:26.599661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.007768+0000","last_change":"2024-01-15T18:12:01.030330+0000","last_active":"2024-01-15T18:12:02.007768+0000","last_peered":"2024-01-15T18:12:02.007768+0000","last_clean":"2024-01-15T18:12:02.007768+0000","last_became_active":"2024-01-15T18:12:01.030085+0000","last_became_peered":"2024-01-15T18:12:01.030085+0000","last_unstale":"2024-01-15T18:12:02.007768+0000","last_undegraded":"2024-01-15T18:12:02.007768+0000","last_fullsized":"2024-01-15T18:12:02.007768+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:08:20.018505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756486+0000","last_change":"2024-01-15T18:12:01.014677+0000","last_active":"2024-01-15T18:12:02.756486+0000","last_peered":"2024-01-15T18:12:02.756486+0000","last_clean":"2024-01-15T18:12:02.756486+0000","last_became_active":"2024-01-15T18:12:01.014379+0000","last_became_peered":"2024-01-15T18:12:01.014379+0000","last_unstale":"2024-01-15T18:12:02.756486+0000","last_undegraded":"2024-01-15T18:12:02.756486+0000","last_fullsized":"2024-01-15T18:12:02.756486+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:03:56.065254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756472+0000","last_change":"2024-01-15T18:12:01.027756+0000","last_active":"2024-01-15T18:12:02.756472+0000","last_peered":"2024-01-15T18:12:02.756472+0000","last_clean":"2024-01-15T18:12:02.756472+0000","last_became_active":"2024-01-15T18:12:01.027426+0000","last_became_peered":"2024-01-15T18:12:01.027426+0000","last_unstale":"2024-01-15T18:12:02.756472+0000","last_undegraded":"2024-01-15T18:12:02.756472+0000","last_fullsized":"2024-01-15T18:12:02.756472+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:05.621191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756455+0000","last_change":"2024-01-15T18:12:01.023192+0000","last_active":"2024-01-15T18:12:02.756455+0000","last_peered":"2024-01-15T18:12:02.756455+0000","last_clean":"2024-01-15T18:12:02.756455+0000","last_became_active":"2024-01-15T18:12:01.022417+0000","last_became_peered":"2024-01-15T18:12:01.022417+0000","last_unstale":"2024-01-15T18:12:02.756455+0000","last_undegraded":"2024-01-15T18:12:02.756455+0000","last_fullsized":"2024-01-15T18:12:02.756455+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:02:44.137075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.006064+0000","last_change":"2024-01-15T18:12:01.030704+0000","last_active":"2024-01-15T18:12:02.006064+0000","last_peered":"2024-01-15T18:12:02.006064+0000","last_clean":"2024-01-15T18:12:02.006064+0000","last_became_active":"2024-01-15T18:12:01.030404+0000","last_became_peered":"2024-01-15T18:12:01.030404+0000","last_unstale":"2024-01-15T18:12:02.006064+0000","last_undegraded":"2024-01-15T18:12:02.006064+0000","last_fullsized":"2024-01-15T18:12:02.006064+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:05:44.296057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.007386+0000","last_change":"2024-01-15T18:12:01.026060+0000","last_active":"2024-01-15T18:12:02.007386+0000","last_peered":"2024-01-15T18:12:02.007386+0000","last_clean":"2024-01-15T18:12:02.007386+0000","last_became_active":"2024-01-15T18:12:01.025781+0000","last_became_peered":"2024-01-15T18:12:01.025781+0000","last_unstale":"2024-01-15T18:12:02.007386+0000","last_undegraded":"2024-01-15T18:12:02.007386+0000","last_fullsized":"2024-01-15T18:12:02.007386+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:00:42.420373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.006053+0000","last_change":"2024-01-15T18:12:01.026597+0000","last_active":"2024-01-15T18:12:02.006053+0000","last_peered":"2024-01-15T18:12:02.006053+0000","last_clean":"2024-01-15T18:12:02.006053+0000","last_became_active":"2024-01-15T18:12:01.026178+0000","last_became_peered":"2024-01-15T18:12:01.026178+0000","last_unstale":"2024-01-15T18:12:02.006053+0000","last_undegraded":"2024-01-15T18:12:02.006053+0000","last_fullsized":"2024-01-15T18:12:02.006053+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:04:19.070396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.757153+0000","last_change":"2024-01-15T18:12:01.011394+0000","last_active":"2024-01-15T18:12:02.757153+0000","last_peered":"2024-01-15T18:12:02.757153+0000","last_clean":"2024-01-15T18:12:02.757153+0000","last_became_active":"2024-01-15T18:12:01.011072+0000","last_became_peered":"2024-01-15T18:12:01.011072+0000","last_unstale":"2024-01-15T18:12:02.757153+0000","last_undegraded":"2024-01-15T18:12:02.757153+0000","last_fullsized":"2024-01-15T18:12:02.757153+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:41:30.756491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.007777+0000","last_change":"2024-01-15T18:12:01.029694+0000","last_active":"2024-01-15T18:12:02.007777+0000","last_peered":"2024-01-15T18:12:02.007777+0000","last_clean":"2024-01-15T18:12:02.007777+0000","last_became_active":"2024-01-15T18:12:01.029127+0000","last_became_peered":"2024-01-15T18:12:01.029127+0000","last_unstale":"2024-01-15T18:12:02.007777+0000","last_undegraded":"2024-01-15T18:12:02.007777+0000","last_fullsized":"2024-01-15T18:12:02.007777+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:50:00.994032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756791+0000","last_change":"2024-01-15T18:12:01.029765+0000","last_active":"2024-01-15T18:12:02.756791+0000","last_peered":"2024-01-15T18:12:02.756791+0000","last_clean":"2024-01-15T18:12:02.756791+0000","last_became_active":"2024-01-15T18:12:01.029504+0000","last_became_peered":"2024-01-15T18:12:01.029504+0000","last_unstale":"2024-01-15T18:12:02.756791+0000","last_undegraded":"2024-01-15T18:12:02.756791+0000","last_fullsized":"2024-01-15T18:12:02.756791+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:36:37.178674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756561+0000","last_change":"2024-01-15T18:12:01.026480+0000","last_active":"2024-01-15T18:12:02.756561+0000","last_peered":"2024-01-15T18:12:02.756561+0000","last_clean":"2024-01-15T18:12:02.756561+0000","last_became_active":"2024-01-15T18:12:01.026104+0000","last_became_peered":"2024-01-15T18:12:01.026104+0000","last_unstale":"2024-01-15T18:12:02.756561+0000","last_undegraded":"2024-01-15T18:12:02.756561+0000","last_fullsized":"2024-01-15T18:12:02.756561+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:37:54.171968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.006074+0000","last_change":"2024-01-15T18:12:01.030452+0000","last_active":"2024-01-15T18:12:02.006074+0000","last_peered":"2024-01-15T18:12:02.006074+0000","last_clean":"2024-01-15T18:12:02.006074+0000","last_became_active":"2024-01-15T18:12:01.027541+0000","last_became_peered":"2024-01-15T18:12:01.027541+0000","last_unstale":"2024-01-15T18:12:02.006074+0000","last_undegraded":"2024-01-15T18:12:02.006074+0000","last_fullsized":"2024-01-15T18:12:02.006074+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:18:47.967166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756659+0000","last_change":"2024-01-15T18:12:01.031213+0000","last_active":"2024-01-15T18:12:02.756659+0000","last_peered":"2024-01-15T18:12:02.756659+0000","last_clean":"2024-01-15T18:12:02.756659+0000","last_became_active":"2024-01-15T18:12:01.027530+0000","last_became_peered":"2024-01-15T18:12:01.027530+0000","last_unstale":"2024-01-15T18:12:02.756659+0000","last_undegraded":"2024-01-15T18:12:02.756659+0000","last_fullsized":"2024-01-15T18:12:02.756659+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:10:56.250682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756491+0000","last_change":"2024-01-15T18:12:01.019809+0000","last_active":"2024-01-15T18:12:02.756491+0000","last_peered":"2024-01-15T18:12:02.756491+0000","last_clean":"2024-01-15T18:12:02.756491+0000","last_became_active":"2024-01-15T18:12:01.019490+0000","last_became_peered":"2024-01-15T18:12:01.019490+0000","last_unstale":"2024-01-15T18:12:02.756491+0000","last_undegraded":"2024-01-15T18:12:02.756491+0000","last_fullsized":"2024-01-15T18:12:02.756491+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:54:29.425928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.755552+0000","last_change":"2024-01-15T18:12:01.037632+0000","last_active":"2024-01-15T18:12:02.755552+0000","last_peered":"2024-01-15T18:12:02.755552+0000","last_clean":"2024-01-15T18:12:02.755552+0000","last_became_active":"2024-01-15T18:12:01.037364+0000","last_became_peered":"2024-01-15T18:12:01.037364+0000","last_unstale":"2024-01-15T18:12:02.755552+0000","last_undegraded":"2024-01-15T18:12:02.755552+0000","last_fullsized":"2024-01-15T18:12:02.755552+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:29:44.252608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.755556+0000","last_change":"2024-01-15T18:12:01.033589+0000","last_active":"2024-01-15T18:12:02.755556+0000","last_peered":"2024-01-15T18:12:02.755556+0000","last_clean":"2024-01-15T18:12:02.755556+0000","last_became_active":"2024-01-15T18:12:01.033232+0000","last_became_peered":"2024-01-15T18:12:01.033232+0000","last_unstale":"2024-01-15T18:12:02.755556+0000","last_undegraded":"2024-01-15T18:12:02.755556+0000","last_fullsized":"2024-01-15T18:12:02.755556+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:33:41.475245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.757431+0000","last_change":"2024-01-15T18:12:01.027376+0000","last_active":"2024-01-15T18:12:02.757431+0000","last_peered":"2024-01-15T18:12:02.757431+0000","last_clean":"2024-01-15T18:12:02.757431+0000","last_became_active":"2024-01-15T18:12:01.026894+0000","last_became_peered":"2024-01-15T18:12:01.026894+0000","last_unstale":"2024-01-15T18:12:02.757431+0000","last_undegraded":"2024-01-15T18:12:02.757431+0000","last_fullsized":"2024-01-15T18:12:02.757431+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:11.559886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005958+0000","last_change":"2024-01-15T18:12:01.026872+0000","last_active":"2024-01-15T18:12:02.005958+0000","last_peered":"2024-01-15T18:12:02.005958+0000","last_clean":"2024-01-15T18:12:02.005958+0000","last_became_active":"2024-01-15T18:12:01.026554+0000","last_became_peered":"2024-01-15T18:12:01.026554+0000","last_unstale":"2024-01-15T18:12:02.005958+0000","last_undegraded":"2024-01-15T18:12:02.005958+0000","last_fullsized":"2024-01-15T18:12:02.005958+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:05:44.296057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.757074+0000","last_change":"2024-01-15T18:12:01.029998+0000","last_active":"2024-01-15T18:12:02.757074+0000","last_peered":"2024-01-15T18:12:02.757074+0000","last_clean":"2024-01-15T18:12:02.757074+0000","last_became_active":"2024-01-15T18:12:01.029687+0000","last_became_peered":"2024-01-15T18:12:01.029687+0000","last_unstale":"2024-01-15T18:12:02.757074+0000","last_undegraded":"2024-01-15T18:12:02.757074+0000","last_fullsized":"2024-01-15T18:12:02.757074+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:04:28.577498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756412+0000","last_change":"2024-01-15T18:12:01.014692+0000","last_active":"2024-01-15T18:12:02.756412+0000","last_peered":"2024-01-15T18:12:02.756412+0000","last_clean":"2024-01-15T18:12:02.756412+0000","last_became_active":"2024-01-15T18:12:01.014384+0000","last_became_peered":"2024-01-15T18:12:01.014384+0000","last_unstale":"2024-01-15T18:12:02.756412+0000","last_undegraded":"2024-01-15T18:12:02.756412+0000","last_fullsized":"2024-01-15T18:12:02.756412+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:55:27.047888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756790+0000","last_change":"2024-01-15T18:12:01.027026+0000","last_active":"2024-01-15T18:12:02.756790+0000","last_peered":"2024-01-15T18:12:02.756790+0000","last_clean":"2024-01-15T18:12:02.756790+0000","last_became_active":"2024-01-15T18:12:01.026727+0000","last_became_peered":"2024-01-15T18:12:01.026727+0000","last_unstale":"2024-01-15T18:12:02.756790+0000","last_undegraded":"2024-01-15T18:12:02.756790+0000","last_fullsized":"2024-01-15T18:12:02.756790+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:21:01.515615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005852+0000","last_change":"2024-01-15T18:12:01.025557+0000","last_active":"2024-01-15T18:12:02.005852+0000","last_peered":"2024-01-15T18:12:02.005852+0000","last_clean":"2024-01-15T18:12:02.005852+0000","last_became_active":"2024-01-15T18:12:01.024972+0000","last_became_peered":"2024-01-15T18:12:01.024972+0000","last_unstale":"2024-01-15T18:12:02.005852+0000","last_undegraded":"2024-01-15T18:12:02.005852+0000","last_fullsized":"2024-01-15T18:12:02.005852+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:46:42.441463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005732+0000","last_change":"2024-01-15T18:12:01.023561+0000","last_active":"2024-01-15T18:12:02.005732+0000","last_peered":"2024-01-15T18:12:02.005732+0000","last_clean":"2024-01-15T18:12:02.005732+0000","last_became_active":"2024-01-15T18:12:01.022979+0000","last_became_peered":"2024-01-15T18:12:01.022979+0000","last_unstale":"2024-01-15T18:12:02.005732+0000","last_undegraded":"2024-01-15T18:12:02.005732+0000","last_fullsized":"2024-01-15T18:12:02.005732+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:58:49.958276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.755914+0000","last_change":"2024-01-15T18:12:01.027732+0000","last_active":"2024-01-15T18:12:02.755914+0000","last_peered":"2024-01-15T18:12:02.755914+0000","last_clean":"2024-01-15T18:12:02.755914+0000","last_became_active":"2024-01-15T18:12:01.027455+0000","last_became_peered":"2024-01-15T18:12:01.027455+0000","last_unstale":"2024-01-15T18:12:02.755914+0000","last_undegraded":"2024-01-15T18:12:02.755914+0000","last_fullsized":"2024-01-15T18:12:02.755914+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:02:44.137075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756530+0000","last_change":"2024-01-15T18:12:01.034752+0000","last_active":"2024-01-15T18:12:02.756530+0000","last_peered":"2024-01-15T18:12:02.756530+0000","last_clean":"2024-01-15T18:12:02.756530+0000","last_became_active":"2024-01-15T18:12:01.033222+0000","last_became_peered":"2024-01-15T18:12:01.033222+0000","last_unstale":"2024-01-15T18:12:02.756530+0000","last_undegraded":"2024-01-15T18:12:02.756530+0000","last_fullsized":"2024-01-15T18:12:02.756530+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:53:50.857705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"69'4","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.045456+0000","last_change":"2024-01-15T18:12:00.026981+0000","last_active":"2024-01-15T18:12:01.045456+0000","last_peered":"2024-01-15T18:12:01.045456+0000","last_clean":"2024-01-15T18:12:01.045456+0000","last_became_active":"2024-01-15T18:12:00.026446+0000","last_became_peered":"2024-01-15T18:12:00.026446+0000","last_unstale":"2024-01-15T18:12:01.045456+0000","last_undegraded":"2024-01-15T18:12:01.045456+0000","last_fullsized":"2024-01-15T18:12:01.045456+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:56:18.635395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.007841+0000","last_change":"2024-01-15T18:12:01.028777+0000","last_active":"2024-01-15T18:12:02.007841+0000","last_peered":"2024-01-15T18:12:02.007841+0000","last_clean":"2024-01-15T18:12:02.007841+0000","last_became_active":"2024-01-15T18:12:01.028450+0000","last_became_peered":"2024-01-15T18:12:01.028450+0000","last_unstale":"2024-01-15T18:12:02.007841+0000","last_undegraded":"2024-01-15T18:12:02.007841+0000","last_fullsized":"2024-01-15T18:12:02.007841+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:36:16.553457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"69'1","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.042664+0000","last_change":"2024-01-15T18:12:00.018709+0000","last_active":"2024-01-15T18:12:01.042664+0000","last_peered":"2024-01-15T18:12:01.042664+0000","last_clean":"2024-01-15T18:12:01.042664+0000","last_became_active":"2024-01-15T18:12:00.018436+0000","last_became_peered":"2024-01-15T18:12:00.018436+0000","last_unstale":"2024-01-15T18:12:01.042664+0000","last_undegraded":"2024-01-15T18:12:01.042664+0000","last_fullsized":"2024-01-15T18:12:01.042664+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:13:49.229168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.757001+0000","last_change":"2024-01-15T18:12:01.026312+0000","last_active":"2024-01-15T18:12:02.757001+0000","last_peered":"2024-01-15T18:12:02.757001+0000","last_clean":"2024-01-15T18:12:02.757001+0000","last_became_active":"2024-01-15T18:12:01.025949+0000","last_became_peered":"2024-01-15T18:12:01.025949+0000","last_unstale":"2024-01-15T18:12:02.757001+0000","last_undegraded":"2024-01-15T18:12:02.757001+0000","last_fullsized":"2024-01-15T18:12:02.757001+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:40:33.473282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.044385+0000","last_change":"2024-01-15T18:12:00.022131+0000","last_active":"2024-01-15T18:12:01.044385+0000","last_peered":"2024-01-15T18:12:01.044385+0000","last_clean":"2024-01-15T18:12:01.044385+0000","last_became_active":"2024-01-15T18:12:00.021836+0000","last_became_peered":"2024-01-15T18:12:00.021836+0000","last_unstale":"2024-01-15T18:12:01.044385+0000","last_undegraded":"2024-01-15T18:12:01.044385+0000","last_fullsized":"2024-01-15T18:12:01.044385+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:11:11.204604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756616+0000","last_change":"2024-01-15T18:12:01.034660+0000","last_active":"2024-01-15T18:12:02.756616+0000","last_peered":"2024-01-15T18:12:02.756616+0000","last_clean":"2024-01-15T18:12:02.756616+0000","last_became_active":"2024-01-15T18:12:01.034177+0000","last_became_peered":"2024-01-15T18:12:01.034177+0000","last_unstale":"2024-01-15T18:12:02.756616+0000","last_undegraded":"2024-01-15T18:12:02.756616+0000","last_fullsized":"2024-01-15T18:12:02.756616+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:37:09.206892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.043171+0000","last_change":"2024-01-15T18:12:00.019217+0000","last_active":"2024-01-15T18:12:01.043171+0000","last_peered":"2024-01-15T18:12:01.043171+0000","last_clean":"2024-01-15T18:12:01.043171+0000","last_became_active":"2024-01-15T18:12:00.018924+0000","last_became_peered":"2024-01-15T18:12:00.018924+0000","last_unstale":"2024-01-15T18:12:01.043171+0000","last_undegraded":"2024-01-15T18:12:01.043171+0000","last_fullsized":"2024-01-15T18:12:01.043171+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:06:24.967935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.006011+0000","last_change":"2024-01-15T18:12:01.028720+0000","last_active":"2024-01-15T18:12:02.006011+0000","last_peered":"2024-01-15T18:12:02.006011+0000","last_clean":"2024-01-15T18:12:02.006011+0000","last_became_active":"2024-01-15T18:12:01.028453+0000","last_became_peered":"2024-01-15T18:12:01.028453+0000","last_unstale":"2024-01-15T18:12:02.006011+0000","last_undegraded":"2024-01-15T18:12:02.006011+0000","last_fullsized":"2024-01-15T18:12:02.006011+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:20.609600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"69'1","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.039628+0000","last_change":"2024-01-15T18:12:00.017547+0000","last_active":"2024-01-15T18:12:01.039628+0000","last_peered":"2024-01-15T18:12:01.039628+0000","last_clean":"2024-01-15T18:12:01.039628+0000","last_became_active":"2024-01-15T18:12:00.017260+0000","last_became_peered":"2024-01-15T18:12:00.017260+0000","last_unstale":"2024-01-15T18:12:01.039628+0000","last_undegraded":"2024-01-15T18:12:01.039628+0000","last_fullsized":"2024-01-15T18:12:01.039628+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:37:08.203604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756307+0000","last_change":"2024-01-15T18:12:01.027542+0000","last_active":"2024-01-15T18:12:02.756307+0000","last_peered":"2024-01-15T18:12:02.756307+0000","last_clean":"2024-01-15T18:12:02.756307+0000","last_became_active":"2024-01-15T18:12:01.027247+0000","last_became_peered":"2024-01-15T18:12:01.027247+0000","last_unstale":"2024-01-15T18:12:02.756307+0000","last_undegraded":"2024-01-15T18:12:02.756307+0000","last_fullsized":"2024-01-15T18:12:02.756307+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:07.393446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"69'1","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.041945+0000","last_change":"2024-01-15T18:12:00.025697+0000","last_active":"2024-01-15T18:12:01.041945+0000","last_peered":"2024-01-15T18:12:01.041945+0000","last_clean":"2024-01-15T18:12:01.041945+0000","last_became_active":"2024-01-15T18:12:00.025383+0000","last_became_peered":"2024-01-15T18:12:00.025383+0000","last_unstale":"2024-01-15T18:12:01.041945+0000","last_undegraded":"2024-01-15T18:12:01.041945+0000","last_fullsized":"2024-01-15T18:12:01.041945+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:27:28.185512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.006976+0000","last_change":"2024-01-15T18:12:01.014933+0000","last_active":"2024-01-15T18:12:02.006976+0000","last_peered":"2024-01-15T18:12:02.006976+0000","last_clean":"2024-01-15T18:12:02.006976+0000","last_became_active":"2024-01-15T18:12:01.014638+0000","last_became_peered":"2024-01-15T18:12:01.014638+0000","last_unstale":"2024-01-15T18:12:02.006976+0000","last_undegraded":"2024-01-15T18:12:02.006976+0000","last_fullsized":"2024-01-15T18:12:02.006976+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:53:36.228024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.013121+0000","last_change":"2024-01-15T18:12:00.019136+0000","last_active":"2024-01-15T18:12:01.013121+0000","last_peered":"2024-01-15T18:12:01.013121+0000","last_clean":"2024-01-15T18:12:01.013121+0000","last_became_active":"2024-01-15T18:12:00.018512+0000","last_became_peered":"2024-01-15T18:12:00.018512+0000","last_unstale":"2024-01-15T18:12:01.013121+0000","last_undegraded":"2024-01-15T18:12:01.013121+0000","last_fullsized":"2024-01-15T18:12:01.013121+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:13:56.868830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005875+0000","last_change":"2024-01-15T18:12:01.042936+0000","last_active":"2024-01-15T18:12:02.005875+0000","last_peered":"2024-01-15T18:12:02.005875+0000","last_clean":"2024-01-15T18:12:02.005875+0000","last_became_active":"2024-01-15T18:12:01.042276+0000","last_became_peered":"2024-01-15T18:12:01.042276+0000","last_unstale":"2024-01-15T18:12:02.005875+0000","last_undegraded":"2024-01-15T18:12:02.005875+0000","last_fullsized":"2024-01-15T18:12:02.005875+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:46:42.441463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.005620+0000","last_change":"2024-01-15T18:12:00.021284+0000","last_active":"2024-01-15T18:12:01.005620+0000","last_peered":"2024-01-15T18:12:01.005620+0000","last_clean":"2024-01-15T18:12:01.005620+0000","last_became_active":"2024-01-15T18:12:00.021017+0000","last_became_peered":"2024-01-15T18:12:00.021017+0000","last_unstale":"2024-01-15T18:12:01.005620+0000","last_undegraded":"2024-01-15T18:12:01.005620+0000","last_fullsized":"2024-01-15T18:12:01.005620+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:45.089127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005945+0000","last_change":"2024-01-15T18:12:01.042737+0000","last_active":"2024-01-15T18:12:02.005945+0000","last_peered":"2024-01-15T18:12:02.005945+0000","last_clean":"2024-01-15T18:12:02.005945+0000","last_became_active":"2024-01-15T18:12:01.042482+0000","last_became_peered":"2024-01-15T18:12:01.042482+0000","last_unstale":"2024-01-15T18:12:02.005945+0000","last_undegraded":"2024-01-15T18:12:02.005945+0000","last_fullsized":"2024-01-15T18:12:02.005945+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:04:19.070396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.008493+0000","last_change":"2024-01-15T18:12:00.017218+0000","last_active":"2024-01-15T18:12:01.008493+0000","last_peered":"2024-01-15T18:12:01.008493+0000","last_clean":"2024-01-15T18:12:01.008493+0000","last_became_active":"2024-01-15T18:12:00.016921+0000","last_became_peered":"2024-01-15T18:12:00.016921+0000","last_unstale":"2024-01-15T18:12:01.008493+0000","last_undegraded":"2024-01-15T18:12:01.008493+0000","last_fullsized":"2024-01-15T18:12:01.008493+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:08:25.595491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756886+0000","last_change":"2024-01-15T18:12:01.034009+0000","last_active":"2024-01-15T18:12:02.756886+0000","last_peered":"2024-01-15T18:12:02.756886+0000","last_clean":"2024-01-15T18:12:02.756886+0000","last_became_active":"2024-01-15T18:12:01.033712+0000","last_became_peered":"2024-01-15T18:12:01.033712+0000","last_unstale":"2024-01-15T18:12:02.756886+0000","last_undegraded":"2024-01-15T18:12:02.756886+0000","last_fullsized":"2024-01-15T18:12:02.756886+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:36:21.428758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.009024+0000","last_change":"2024-01-15T18:12:00.018102+0000","last_active":"2024-01-15T18:12:01.009024+0000","last_peered":"2024-01-15T18:12:01.009024+0000","last_clean":"2024-01-15T18:12:01.009024+0000","last_became_active":"2024-01-15T18:12:00.017781+0000","last_became_peered":"2024-01-15T18:12:00.017781+0000","last_unstale":"2024-01-15T18:12:01.009024+0000","last_undegraded":"2024-01-15T18:12:01.009024+0000","last_fullsized":"2024-01-15T18:12:01.009024+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:46:51.225073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.757140+0000","last_change":"2024-01-15T18:12:01.012486+0000","last_active":"2024-01-15T18:12:02.757140+0000","last_peered":"2024-01-15T18:12:02.757140+0000","last_clean":"2024-01-15T18:12:02.757140+0000","last_became_active":"2024-01-15T18:12:01.010951+0000","last_became_peered":"2024-01-15T18:12:01.010951+0000","last_unstale":"2024-01-15T18:12:02.757140+0000","last_undegraded":"2024-01-15T18:12:02.757140+0000","last_fullsized":"2024-01-15T18:12:02.757140+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:19:30.413967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"69'3","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.041771+0000","last_change":"2024-01-15T18:12:00.019006+0000","last_active":"2024-01-15T18:12:01.041771+0000","last_peered":"2024-01-15T18:12:01.041771+0000","last_clean":"2024-01-15T18:12:01.041771+0000","last_became_active":"2024-01-15T18:12:00.017853+0000","last_became_peered":"2024-01-15T18:12:00.017853+0000","last_unstale":"2024-01-15T18:12:01.041771+0000","last_undegraded":"2024-01-15T18:12:01.041771+0000","last_fullsized":"2024-01-15T18:12:01.041771+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:29:43.249320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.007141+0000","last_change":"2024-01-15T18:12:01.025987+0000","last_active":"2024-01-15T18:12:02.007141+0000","last_peered":"2024-01-15T18:12:02.007141+0000","last_clean":"2024-01-15T18:12:02.007141+0000","last_became_active":"2024-01-15T18:12:01.025672+0000","last_became_peered":"2024-01-15T18:12:01.025672+0000","last_unstale":"2024-01-15T18:12:02.007141+0000","last_undegraded":"2024-01-15T18:12:02.007141+0000","last_fullsized":"2024-01-15T18:12:02.007141+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:28:36.163121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"69'6","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.046630+0000","last_change":"2024-01-15T18:12:00.011850+0000","last_active":"2024-01-15T18:12:01.046630+0000","last_peered":"2024-01-15T18:12:01.046630+0000","last_clean":"2024-01-15T18:12:01.046630+0000","last_became_active":"2024-01-15T18:12:00.011519+0000","last_became_peered":"2024-01-15T18:12:00.011519+0000","last_unstale":"2024-01-15T18:12:01.046630+0000","last_undegraded":"2024-01-15T18:12:01.046630+0000","last_fullsized":"2024-01-15T18:12:01.046630+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:17:27.915008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.755541+0000","last_change":"2024-01-15T18:12:01.033995+0000","last_active":"2024-01-15T18:12:02.755541+0000","last_peered":"2024-01-15T18:12:02.755541+0000","last_clean":"2024-01-15T18:12:02.755541+0000","last_became_active":"2024-01-15T18:12:01.033680+0000","last_became_peered":"2024-01-15T18:12:01.033680+0000","last_unstale":"2024-01-15T18:12:02.755541+0000","last_undegraded":"2024-01-15T18:12:02.755541+0000","last_fullsized":"2024-01-15T18:12:02.755541+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:09:53.302299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.044808+0000","last_change":"2024-01-15T18:12:00.028242+0000","last_active":"2024-01-15T18:12:01.044808+0000","last_peered":"2024-01-15T18:12:01.044808+0000","last_clean":"2024-01-15T18:12:01.044808+0000","last_became_active":"2024-01-15T18:12:00.027987+0000","last_became_peered":"2024-01-15T18:12:00.027987+0000","last_unstale":"2024-01-15T18:12:01.044808+0000","last_undegraded":"2024-01-15T18:12:01.044808+0000","last_fullsized":"2024-01-15T18:12:01.044808+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:06:24.967935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005827+0000","last_change":"2024-01-15T18:12:01.024849+0000","last_active":"2024-01-15T18:12:02.005827+0000","last_peered":"2024-01-15T18:12:02.005827+0000","last_clean":"2024-01-15T18:12:02.005827+0000","last_became_active":"2024-01-15T18:12:01.024579+0000","last_became_peered":"2024-01-15T18:12:01.024579+0000","last_unstale":"2024-01-15T18:12:02.005827+0000","last_undegraded":"2024-01-15T18:12:02.005827+0000","last_fullsized":"2024-01-15T18:12:02.005827+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:43:37.142127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.007485+0000","last_change":"2024-01-15T18:12:00.014156+0000","last_active":"2024-01-15T18:12:01.007485+0000","last_peered":"2024-01-15T18:12:01.007485+0000","last_clean":"2024-01-15T18:12:01.007485+0000","last_became_active":"2024-01-15T18:12:00.013881+0000","last_became_peered":"2024-01-15T18:12:00.013881+0000","last_unstale":"2024-01-15T18:12:01.007485+0000","last_undegraded":"2024-01-15T18:12:01.007485+0000","last_fullsized":"2024-01-15T18:12:01.007485+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:49:33.673954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005777+0000","last_change":"2024-01-15T18:12:01.032149+0000","last_active":"2024-01-15T18:12:02.005777+0000","last_peered":"2024-01-15T18:12:02.005777+0000","last_clean":"2024-01-15T18:12:02.005777+0000","last_became_active":"2024-01-15T18:12:01.030842+0000","last_became_peered":"2024-01-15T18:12:01.030842+0000","last_unstale":"2024-01-15T18:12:02.005777+0000","last_undegraded":"2024-01-15T18:12:02.005777+0000","last_fullsized":"2024-01-15T18:12:02.005777+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:43:37.142127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.014661+0000","last_change":"2024-01-15T18:12:00.025503+0000","last_active":"2024-01-15T18:12:01.014661+0000","last_peered":"2024-01-15T18:12:01.014661+0000","last_clean":"2024-01-15T18:12:01.014661+0000","last_became_active":"2024-01-15T18:12:00.025198+0000","last_became_peered":"2024-01-15T18:12:00.025198+0000","last_unstale":"2024-01-15T18:12:01.014661+0000","last_undegraded":"2024-01-15T18:12:01.014661+0000","last_fullsized":"2024-01-15T18:12:01.014661+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:06:28.115069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.007094+0000","last_change":"2024-01-15T18:12:01.026199+0000","last_active":"2024-01-15T18:12:02.007094+0000","last_peered":"2024-01-15T18:12:02.007094+0000","last_clean":"2024-01-15T18:12:02.007094+0000","last_became_active":"2024-01-15T18:12:01.025920+0000","last_became_peered":"2024-01-15T18:12:01.025920+0000","last_unstale":"2024-01-15T18:12:02.007094+0000","last_undegraded":"2024-01-15T18:12:02.007094+0000","last_fullsized":"2024-01-15T18:12:02.007094+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:53:27.484089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"69'8","reported_seq":20,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.045203+0000","last_change":"2024-01-15T18:12:00.025058+0000","last_active":"2024-01-15T18:12:01.045203+0000","last_peered":"2024-01-15T18:12:01.045203+0000","last_clean":"2024-01-15T18:12:01.045203+0000","last_became_active":"2024-01-15T18:12:00.024655+0000","last_became_peered":"2024-01-15T18:12:00.024655+0000","last_unstale":"2024-01-15T18:12:01.045203+0000","last_undegraded":"2024-01-15T18:12:01.045203+0000","last_fullsized":"2024-01-15T18:12:01.045203+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:33:10.196557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.757369+0000","last_change":"2024-01-15T18:12:01.023212+0000","last_active":"2024-01-15T18:12:02.757369+0000","last_peered":"2024-01-15T18:12:02.757369+0000","last_clean":"2024-01-15T18:12:02.757369+0000","last_became_active":"2024-01-15T18:12:01.022574+0000","last_became_peered":"2024-01-15T18:12:01.022574+0000","last_unstale":"2024-01-15T18:12:02.757369+0000","last_undegraded":"2024-01-15T18:12:02.757369+0000","last_fullsized":"2024-01-15T18:12:02.757369+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:17:25.624685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"69'1","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.043094+0000","last_change":"2024-01-15T18:12:00.019039+0000","last_active":"2024-01-15T18:12:01.043094+0000","last_peered":"2024-01-15T18:12:01.043094+0000","last_clean":"2024-01-15T18:12:01.043094+0000","last_became_active":"2024-01-15T18:12:00.018741+0000","last_became_peered":"2024-01-15T18:12:00.018741+0000","last_unstale":"2024-01-15T18:12:01.043094+0000","last_undegraded":"2024-01-15T18:12:01.043094+0000","last_fullsized":"2024-01-15T18:12:01.043094+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:40:14.291921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756941+0000","last_change":"2024-01-15T18:12:01.034613+0000","last_active":"2024-01-15T18:12:02.756941+0000","last_peered":"2024-01-15T18:12:02.756941+0000","last_clean":"2024-01-15T18:12:02.756941+0000","last_became_active":"2024-01-15T18:12:01.034323+0000","last_became_peered":"2024-01-15T18:12:01.034323+0000","last_unstale":"2024-01-15T18:12:02.756941+0000","last_undegraded":"2024-01-15T18:12:02.756941+0000","last_fullsized":"2024-01-15T18:12:02.756941+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:41:38.814257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.042539+0000","last_change":"2024-01-15T18:12:00.022509+0000","last_active":"2024-01-15T18:12:01.042539+0000","last_peered":"2024-01-15T18:12:01.042539+0000","last_clean":"2024-01-15T18:12:01.042539+0000","last_became_active":"2024-01-15T18:12:00.021725+0000","last_became_peered":"2024-01-15T18:12:00.021725+0000","last_unstale":"2024-01-15T18:12:01.042539+0000","last_undegraded":"2024-01-15T18:12:01.042539+0000","last_fullsized":"2024-01-15T18:12:01.042539+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:07:42.595399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756509+0000","last_change":"2024-01-15T18:12:01.017630+0000","last_active":"2024-01-15T18:12:02.756509+0000","last_peered":"2024-01-15T18:12:02.756509+0000","last_clean":"2024-01-15T18:12:02.756509+0000","last_became_active":"2024-01-15T18:12:01.016364+0000","last_became_peered":"2024-01-15T18:12:01.016364+0000","last_unstale":"2024-01-15T18:12:02.756509+0000","last_undegraded":"2024-01-15T18:12:02.756509+0000","last_fullsized":"2024-01-15T18:12:02.756509+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:37:54.171968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"69'1","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.043471+0000","last_change":"2024-01-15T18:12:00.022642+0000","last_active":"2024-01-15T18:12:01.043471+0000","last_peered":"2024-01-15T18:12:01.043471+0000","last_clean":"2024-01-15T18:12:01.043471+0000","last_became_active":"2024-01-15T18:12:00.021842+0000","last_became_peered":"2024-01-15T18:12:00.021842+0000","last_unstale":"2024-01-15T18:12:01.043471+0000","last_undegraded":"2024-01-15T18:12:01.043471+0000","last_fullsized":"2024-01-15T18:12:01.043471+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:38:51.921542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.007215+0000","last_change":"2024-01-15T18:12:01.013299+0000","last_active":"2024-01-15T18:12:02.007215+0000","last_peered":"2024-01-15T18:12:02.007215+0000","last_clean":"2024-01-15T18:12:02.007215+0000","last_became_active":"2024-01-15T18:12:01.012799+0000","last_became_peered":"2024-01-15T18:12:01.012799+0000","last_unstale":"2024-01-15T18:12:02.007215+0000","last_undegraded":"2024-01-15T18:12:02.007215+0000","last_fullsized":"2024-01-15T18:12:02.007215+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:27:06.509091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.014213+0000","last_change":"2024-01-15T18:12:00.027374+0000","last_active":"2024-01-15T18:12:01.014213+0000","last_peered":"2024-01-15T18:12:01.014213+0000","last_clean":"2024-01-15T18:12:01.014213+0000","last_became_active":"2024-01-15T18:12:00.027064+0000","last_became_peered":"2024-01-15T18:12:00.027064+0000","last_unstale":"2024-01-15T18:12:01.014213+0000","last_undegraded":"2024-01-15T18:12:01.014213+0000","last_fullsized":"2024-01-15T18:12:01.014213+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:17:24.621398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756841+0000","last_change":"2024-01-15T18:12:01.025265+0000","last_active":"2024-01-15T18:12:02.756841+0000","last_peered":"2024-01-15T18:12:02.756841+0000","last_clean":"2024-01-15T18:12:02.756841+0000","last_became_active":"2024-01-15T18:12:01.024338+0000","last_became_peered":"2024-01-15T18:12:01.024338+0000","last_unstale":"2024-01-15T18:12:02.756841+0000","last_undegraded":"2024-01-15T18:12:02.756841+0000","last_fullsized":"2024-01-15T18:12:02.756841+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:30:16.972579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.007589+0000","last_change":"2024-01-15T18:12:00.023536+0000","last_active":"2024-01-15T18:12:01.007589+0000","last_peered":"2024-01-15T18:12:01.007589+0000","last_clean":"2024-01-15T18:12:01.007589+0000","last_became_active":"2024-01-15T18:12:00.023249+0000","last_became_peered":"2024-01-15T18:12:00.023249+0000","last_unstale":"2024-01-15T18:12:01.007589+0000","last_undegraded":"2024-01-15T18:12:01.007589+0000","last_fullsized":"2024-01-15T18:12:01.007589+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:55.181455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756009+0000","last_change":"2024-01-15T18:12:01.025190+0000","last_active":"2024-01-15T18:12:02.756009+0000","last_peered":"2024-01-15T18:12:02.756009+0000","last_clean":"2024-01-15T18:12:02.756009+0000","last_became_active":"2024-01-15T18:12:01.024888+0000","last_became_peered":"2024-01-15T18:12:01.024888+0000","last_unstale":"2024-01-15T18:12:02.756009+0000","last_undegraded":"2024-01-15T18:12:02.756009+0000","last_fullsized":"2024-01-15T18:12:02.756009+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:25:17.902934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.017179+0000","last_change":"2024-01-15T18:12:00.020946+0000","last_active":"2024-01-15T18:12:01.017179+0000","last_peered":"2024-01-15T18:12:01.017179+0000","last_clean":"2024-01-15T18:12:01.017179+0000","last_became_active":"2024-01-15T18:12:00.020641+0000","last_became_peered":"2024-01-15T18:12:00.020641+0000","last_unstale":"2024-01-15T18:12:01.017179+0000","last_undegraded":"2024-01-15T18:12:01.017179+0000","last_fullsized":"2024-01-15T18:12:01.017179+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:14:51.219226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756518+0000","last_change":"2024-01-15T18:12:01.027693+0000","last_active":"2024-01-15T18:12:02.756518+0000","last_peered":"2024-01-15T18:12:02.756518+0000","last_clean":"2024-01-15T18:12:02.756518+0000","last_became_active":"2024-01-15T18:12:01.025511+0000","last_became_peered":"2024-01-15T18:12:01.025511+0000","last_unstale":"2024-01-15T18:12:02.756518+0000","last_undegraded":"2024-01-15T18:12:02.756518+0000","last_fullsized":"2024-01-15T18:12:02.756518+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:36:21.428758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.012850+0000","last_change":"2024-01-15T18:12:00.018802+0000","last_active":"2024-01-15T18:12:01.012850+0000","last_peered":"2024-01-15T18:12:01.012850+0000","last_clean":"2024-01-15T18:12:01.012850+0000","last_became_active":"2024-01-15T18:12:00.017356+0000","last_became_peered":"2024-01-15T18:12:00.017356+0000","last_unstale":"2024-01-15T18:12:01.012850+0000","last_undegraded":"2024-01-15T18:12:01.012850+0000","last_fullsized":"2024-01-15T18:12:01.012850+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:00:57.752979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.006344+0000","last_change":"2024-01-15T18:12:01.028988+0000","last_active":"2024-01-15T18:12:02.006344+0000","last_peered":"2024-01-15T18:12:02.006344+0000","last_clean":"2024-01-15T18:12:02.006344+0000","last_became_active":"2024-01-15T18:12:01.028024+0000","last_became_peered":"2024-01-15T18:12:01.028024+0000","last_unstale":"2024-01-15T18:12:02.006344+0000","last_undegraded":"2024-01-15T18:12:02.006344+0000","last_fullsized":"2024-01-15T18:12:02.006344+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:20.609600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.042991+0000","last_change":"2024-01-15T18:12:00.016456+0000","last_active":"2024-01-15T18:12:01.042991+0000","last_peered":"2024-01-15T18:12:01.042991+0000","last_clean":"2024-01-15T18:12:01.042991+0000","last_became_active":"2024-01-15T18:12:00.016145+0000","last_became_peered":"2024-01-15T18:12:00.016145+0000","last_unstale":"2024-01-15T18:12:01.042991+0000","last_undegraded":"2024-01-15T18:12:01.042991+0000","last_fullsized":"2024-01-15T18:12:01.042991+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:13:56.868830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.006010+0000","last_change":"2024-01-15T18:12:01.024249+0000","last_active":"2024-01-15T18:12:02.006010+0000","last_peered":"2024-01-15T18:12:02.006010+0000","last_clean":"2024-01-15T18:12:02.006010+0000","last_became_active":"2024-01-15T18:12:01.023862+0000","last_became_peered":"2024-01-15T18:12:01.023862+0000","last_unstale":"2024-01-15T18:12:02.006010+0000","last_undegraded":"2024-01-15T18:12:02.006010+0000","last_fullsized":"2024-01-15T18:12:02.006010+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:18:47.967166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.016071+0000","last_change":"2024-01-15T18:12:00.023511+0000","last_active":"2024-01-15T18:12:01.016071+0000","last_peered":"2024-01-15T18:12:01.016071+0000","last_clean":"2024-01-15T18:12:01.016071+0000","last_became_active":"2024-01-15T18:12:00.023145+0000","last_became_peered":"2024-01-15T18:12:00.023145+0000","last_unstale":"2024-01-15T18:12:01.016071+0000","last_undegraded":"2024-01-15T18:12:01.016071+0000","last_fullsized":"2024-01-15T18:12:01.016071+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:27:30.560351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756286+0000","last_change":"2024-01-15T18:12:01.026130+0000","last_active":"2024-01-15T18:12:02.756286+0000","last_peered":"2024-01-15T18:12:02.756286+0000","last_clean":"2024-01-15T18:12:02.756286+0000","last_became_active":"2024-01-15T18:12:01.025861+0000","last_became_peered":"2024-01-15T18:12:01.025861+0000","last_unstale":"2024-01-15T18:12:02.756286+0000","last_undegraded":"2024-01-15T18:12:02.756286+0000","last_fullsized":"2024-01-15T18:12:02.756286+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:57.993053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.006728+0000","last_change":"2024-01-15T18:12:00.023014+0000","last_active":"2024-01-15T18:12:01.006728+0000","last_peered":"2024-01-15T18:12:01.006728+0000","last_clean":"2024-01-15T18:12:01.006728+0000","last_became_active":"2024-01-15T18:12:00.022748+0000","last_became_peered":"2024-01-15T18:12:00.022748+0000","last_unstale":"2024-01-15T18:12:01.006728+0000","last_undegraded":"2024-01-15T18:12:01.006728+0000","last_fullsized":"2024-01-15T18:12:01.006728+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:05:09.561750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756754+0000","last_change":"2024-01-15T18:12:01.034310+0000","last_active":"2024-01-15T18:12:02.756754+0000","last_peered":"2024-01-15T18:12:02.756754+0000","last_clean":"2024-01-15T18:12:02.756754+0000","last_became_active":"2024-01-15T18:12:01.034041+0000","last_became_peered":"2024-01-15T18:12:01.034041+0000","last_unstale":"2024-01-15T18:12:02.756754+0000","last_undegraded":"2024-01-15T18:12:02.756754+0000","last_fullsized":"2024-01-15T18:12:02.756754+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:45:12.201484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.012619+0000","last_change":"2024-01-15T18:12:00.024610+0000","last_active":"2024-01-15T18:12:01.012619+0000","last_peered":"2024-01-15T18:12:01.012619+0000","last_clean":"2024-01-15T18:12:01.012619+0000","last_became_active":"2024-01-15T18:12:00.024272+0000","last_became_peered":"2024-01-15T18:12:00.024272+0000","last_unstale":"2024-01-15T18:12:01.012619+0000","last_undegraded":"2024-01-15T18:12:01.012619+0000","last_fullsized":"2024-01-15T18:12:01.012619+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:20:15.640627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.007502+0000","last_change":"2024-01-15T18:12:01.028311+0000","last_active":"2024-01-15T18:12:02.007502+0000","last_peered":"2024-01-15T18:12:02.007502+0000","last_clean":"2024-01-15T18:12:02.007502+0000","last_became_active":"2024-01-15T18:12:01.027889+0000","last_became_peered":"2024-01-15T18:12:01.027889+0000","last_unstale":"2024-01-15T18:12:02.007502+0000","last_undegraded":"2024-01-15T18:12:02.007502+0000","last_fullsized":"2024-01-15T18:12:02.007502+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:19:20.426156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.005931+0000","last_change":"2024-01-15T18:12:00.019406+0000","last_active":"2024-01-15T18:12:01.005931+0000","last_peered":"2024-01-15T18:12:01.005931+0000","last_clean":"2024-01-15T18:12:01.005931+0000","last_became_active":"2024-01-15T18:12:00.019122+0000","last_became_peered":"2024-01-15T18:12:00.019122+0000","last_unstale":"2024-01-15T18:12:01.005931+0000","last_undegraded":"2024-01-15T18:12:01.005931+0000","last_fullsized":"2024-01-15T18:12:01.005931+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:50:34.976203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756988+0000","last_change":"2024-01-15T18:12:01.026385+0000","last_active":"2024-01-15T18:12:02.756988+0000","last_peered":"2024-01-15T18:12:02.756988+0000","last_clean":"2024-01-15T18:12:02.756988+0000","last_became_active":"2024-01-15T18:12:01.026085+0000","last_became_peered":"2024-01-15T18:12:01.026085+0000","last_unstale":"2024-01-15T18:12:02.756988+0000","last_undegraded":"2024-01-15T18:12:02.756988+0000","last_fullsized":"2024-01-15T18:12:02.756988+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:36:37.178674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.013295+0000","last_change":"2024-01-15T18:12:00.015364+0000","last_active":"2024-01-15T18:12:01.013295+0000","last_peered":"2024-01-15T18:12:01.013295+0000","last_clean":"2024-01-15T18:12:01.013295+0000","last_became_active":"2024-01-15T18:12:00.014208+0000","last_became_peered":"2024-01-15T18:12:00.014208+0000","last_unstale":"2024-01-15T18:12:01.013295+0000","last_undegraded":"2024-01-15T18:12:01.013295+0000","last_fullsized":"2024-01-15T18:12:01.013295+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:46:51.225073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756081+0000","last_change":"2024-01-15T18:12:01.032004+0000","last_active":"2024-01-15T18:12:02.756081+0000","last_peered":"2024-01-15T18:12:02.756081+0000","last_clean":"2024-01-15T18:12:02.756081+0000","last_became_active":"2024-01-15T18:12:01.031719+0000","last_became_peered":"2024-01-15T18:12:01.031719+0000","last_unstale":"2024-01-15T18:12:02.756081+0000","last_undegraded":"2024-01-15T18:12:02.756081+0000","last_fullsized":"2024-01-15T18:12:02.756081+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:26.997082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.006423+0000","last_change":"2024-01-15T18:12:00.025623+0000","last_active":"2024-01-15T18:12:01.006423+0000","last_peered":"2024-01-15T18:12:01.006423+0000","last_clean":"2024-01-15T18:12:01.006423+0000","last_became_active":"2024-01-15T18:12:00.025294+0000","last_became_peered":"2024-01-15T18:12:00.025294+0000","last_unstale":"2024-01-15T18:12:01.006423+0000","last_undegraded":"2024-01-15T18:12:01.006423+0000","last_fullsized":"2024-01-15T18:12:01.006423+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:05.587795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.755841+0000","last_change":"2024-01-15T18:12:01.030920+0000","last_active":"2024-01-15T18:12:02.755841+0000","last_peered":"2024-01-15T18:12:02.755841+0000","last_clean":"2024-01-15T18:12:02.755841+0000","last_became_active":"2024-01-15T18:12:01.030565+0000","last_became_peered":"2024-01-15T18:12:01.030565+0000","last_unstale":"2024-01-15T18:12:02.755841+0000","last_undegraded":"2024-01-15T18:12:02.755841+0000","last_fullsized":"2024-01-15T18:12:02.755841+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:26.997082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.007385+0000","last_change":"2024-01-15T18:12:00.019753+0000","last_active":"2024-01-15T18:12:01.007385+0000","last_peered":"2024-01-15T18:12:01.007385+0000","last_clean":"2024-01-15T18:12:01.007385+0000","last_became_active":"2024-01-15T18:12:00.019482+0000","last_became_peered":"2024-01-15T18:12:00.019482+0000","last_unstale":"2024-01-15T18:12:01.007385+0000","last_undegraded":"2024-01-15T18:12:01.007385+0000","last_fullsized":"2024-01-15T18:12:01.007385+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:09:59.518166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756765+0000","last_change":"2024-01-15T18:12:01.026201+0000","last_active":"2024-01-15T18:12:02.756765+0000","last_peered":"2024-01-15T18:12:02.756765+0000","last_clean":"2024-01-15T18:12:02.756765+0000","last_became_active":"2024-01-15T18:12:01.025920+0000","last_became_peered":"2024-01-15T18:12:01.025920+0000","last_unstale":"2024-01-15T18:12:02.756765+0000","last_undegraded":"2024-01-15T18:12:02.756765+0000","last_fullsized":"2024-01-15T18:12:02.756765+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:41:38.814257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.043831+0000","last_change":"2024-01-15T18:12:00.019108+0000","last_active":"2024-01-15T18:12:01.043831+0000","last_peered":"2024-01-15T18:12:01.043831+0000","last_clean":"2024-01-15T18:12:01.043831+0000","last_became_active":"2024-01-15T18:12:00.018827+0000","last_became_peered":"2024-01-15T18:12:00.018827+0000","last_unstale":"2024-01-15T18:12:01.043831+0000","last_undegraded":"2024-01-15T18:12:01.043831+0000","last_fullsized":"2024-01-15T18:12:01.043831+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:17:27.915008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":205,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.019972+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:12:01.019972+0000","last_peered":"2024-01-15T18:12:01.019972+0000","last_clean":"2024-01-15T18:12:01.019972+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:12:01.019972+0000","last_undegraded":"2024-01-15T18:12:01.019972+0000","last_fullsized":"2024-01-15T18:12:01.019972+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509531,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50640,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321200,"statfs":{"total":96636764160,"available":96584908800,"internally_reserved":0,"allocated":1445888,"data_stored":948856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.1200000000000001,"15min":0.78900000000000003},"min":{"1min":0.66000000000000003,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":9.6989999999999998,"5min":10.284000000000001,"15min":10.284000000000001},"last":0.82599999999999996},{"interface":"front","average":{"1min":2.4159999999999999,"5min":1.2290000000000001,"15min":0.86299999999999999},"min":{"1min":0.39100000000000001,"5min":0.378,"15min":0.378},"max":{"1min":16.968,"5min":16.968,"15min":16.968},"last":0.76400000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.7749999999999999,"5min":1.1830000000000001,"15min":0.83899999999999997},"min":{"1min":0.64800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":15.791,"5min":15.791,"15min":15.791},"last":0.86699999999999999},{"interface":"front","average":{"1min":2.1339999999999999,"5min":1.1779999999999999,"15min":0.85899999999999999},"min":{"1min":0.44600000000000001,"5min":0.378,"15min":0.378},"max":{"1min":10.842000000000001,"5min":10.842000000000001,"15min":10.842000000000001},"last":0.84499999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1,"15min":0.76400000000000001},"min":{"1min":0.42299999999999999,"5min":0.378,"15min":0.378},"max":{"1min":4.9029999999999996,"5min":10.119999999999999,"15min":10.119999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":2.0579999999999998,"5min":1.081,"15min":0.83499999999999996},"min":{"1min":0.66800000000000004,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":9.7539999999999996,"5min":9.7539999999999996,"15min":9.7539999999999996},"last":0.78300000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":1.802,"5min":1.143,"15min":0.82299999999999995},"min":{"1min":0.47099999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":11.481,"5min":11.481,"15min":11.481},"last":0.80400000000000005},{"interface":"front","average":{"1min":2.3220000000000001,"5min":1.2310000000000001,"15min":0.91900000000000004},"min":{"1min":0.70199999999999996,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":9.7959999999999994,"5min":10.34,"15min":10.34},"last":0.625}]},{"osd":4,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.5630000000000002,"5min":1.3360000000000001,"15min":0.85799999999999998},"min":{"1min":0.41099999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":20.387,"5min":20.387,"15min":20.387},"last":0.56999999999999995},{"interface":"front","average":{"1min":1.649,"5min":1.131,"15min":0.81999999999999995},"min":{"1min":0.46200000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":11.305,"5min":14.871,"15min":14.871},"last":0.59899999999999998}]},{"osd":5,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.2719999999999998,"5min":1.2070000000000001,"15min":0.86899999999999999},"min":{"1min":0.47799999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":9.8450000000000006,"5min":19.959,"15min":19.959},"last":0.54300000000000004},{"interface":"front","average":{"1min":2.6240000000000001,"5min":1.3009999999999999,"15min":0.92000000000000004},"min":{"1min":0.48299999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":19.905000000000001,"5min":19.905000000000001,"15min":19.905000000000001},"last":0.69399999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":3.157,"5min":1.667,"15min":1.093},"min":{"1min":0.63800000000000001,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":20.492000000000001,"5min":22.478999999999999,"15min":22.478999999999999},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.397,"5min":1.3049999999999999,"15min":0.92500000000000004},"min":{"1min":0.45100000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.2549999999999999,"5min":21.509,"15min":21.509},"last":0.65500000000000003}]}]},{"osd":6,"up_from":15,"seq":64424509531,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45904,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325936,"statfs":{"total":96636764160,"available":96589758464,"internally_reserved":0,"allocated":851968,"data_stored":357964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":1.157,"15min":0.872},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.91800000000000004,"5min":12.41,"15min":12.41},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":1.0009999999999999,"15min":0.79000000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.93500000000000005,"5min":6.7759999999999998,"15min":6.7759999999999998},"last":0.747}]},{"osd":1,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":1.454,"15min":0.96399999999999997},"min":{"1min":0.46999999999999997,"5min":0.41899999999999998,"15min":0.41299999999999998},"max":{"1min":0.83199999999999996,"5min":53.451999999999998,"15min":53.451999999999998},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.71299999999999997,"5min":1.008,"15min":0.83699999999999997},"min":{"1min":0.56100000000000005,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.92900000000000005,"5min":7.7329999999999997,"15min":7.7329999999999997},"last":0.72399999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":1.004,"15min":0.89400000000000002},"min":{"1min":0.46700000000000003,"5min":0.45900000000000002,"15min":0.38900000000000001},"max":{"1min":1.1060000000000001,"5min":7.9409999999999998,"15min":7.9409999999999998},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.93700000000000006,"15min":0.82999999999999996},"min":{"1min":0.57099999999999995,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.90800000000000003,"5min":5.1760000000000002,"15min":5.1760000000000002},"last":0.60599999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":1.038,"15min":0.876},"min":{"1min":0.46000000000000002,"5min":0.42099999999999999,"15min":0.41399999999999998},"max":{"1min":0.94899999999999995,"5min":6.758,"15min":6.758},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.70499999999999996,"5min":1.5600000000000001,"15min":1.0660000000000001},"min":{"1min":0.49199999999999999,"5min":0.45200000000000001,"15min":0.41599999999999998},"max":{"1min":0.877,"5min":53.398000000000003,"15min":53.398000000000003},"last":0.63200000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":1.0029999999999999,"15min":0.79500000000000004},"min":{"1min":0.38500000000000001,"5min":0.378,"15min":0.34999999999999998},"max":{"1min":0.88,"5min":8.6259999999999994,"15min":8.6259999999999994},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.94899999999999995,"15min":0.79100000000000004},"min":{"1min":0.42599999999999999,"5min":0.38200000000000001,"15min":0.34499999999999997},"max":{"1min":0.85099999999999998,"5min":6.9589999999999996,"15min":6.9589999999999996},"last":0.65900000000000003}]},{"osd":5,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":1.641,"15min":0.98299999999999998},"min":{"1min":0.41299999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.96399999999999997,"5min":53.229999999999997,"15min":53.229999999999997},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":1.228,"15min":0.92900000000000005},"min":{"1min":0.45500000000000002,"5min":0.435,"15min":0.435},"max":{"1min":1.149,"5min":10.029999999999999,"15min":10.029999999999999},"last":0.80800000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":1.5760000000000001,"15min":1.071},"min":{"1min":0.437,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.90400000000000003,"5min":45.070999999999998,"15min":45.070999999999998},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.70099999999999996,"5min":1.6910000000000001,"15min":1.0940000000000001},"min":{"1min":0.52800000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.92200000000000004,"5min":53.338000000000001,"15min":53.338000000000001},"last":0.70699999999999996}]}]},{"osd":4,"up_from":15,"seq":64424509531,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50056,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321784,"statfs":{"total":96636764160,"available":96585506816,"internally_reserved":0,"allocated":847872,"data_stored":357942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":1.2,"15min":0.81899999999999995},"min":{"1min":0.502,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.2509999999999999,"5min":23.318000000000001,"15min":23.318000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":1.046,"15min":0.77500000000000002},"min":{"1min":0.442,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.97899999999999998,"5min":14.723000000000001,"15min":14.723000000000001},"last":0.84599999999999997}]},{"osd":1,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.89500000000000002,"15min":0.72999999999999998},"min":{"1min":0.442,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.876,"5min":7.8200000000000003,"15min":7.8200000000000003},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.73899999999999999,"5min":1.1879999999999999,"15min":0.82299999999999995},"min":{"1min":0.47299999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.613,"5min":22.408999999999999,"15min":22.408999999999999},"last":0.751}]},{"osd":2,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.91800000000000004,"15min":0.72399999999999998},"min":{"1min":0.48699999999999999,"5min":0.40600000000000003,"15min":0.34399999999999997},"max":{"1min":1.0009999999999999,"5min":8.8529999999999998,"15min":8.8529999999999998},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.77900000000000003,"5min":1.288,"15min":0.89200000000000002},"min":{"1min":0.55500000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.2969999999999999,"5min":22.338999999999999,"15min":22.338999999999999},"last":0.67100000000000004}]},{"osd":3,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":1.149,"15min":0.84299999999999997},"min":{"1min":0.52500000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.3939999999999999,"5min":14.763999999999999,"15min":14.763999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0509999999999999,"15min":0.83399999999999996},"min":{"1min":0.59899999999999998,"5min":0.504,"15min":0.45500000000000002},"max":{"1min":1.7669999999999999,"5min":8.9459999999999997,"15min":8.9459999999999997},"last":0.86799999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":1.2609999999999999,"15min":0.86399999999999999},"min":{"1min":0.39800000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.4670000000000001,"5min":31.617999999999999,"15min":31.617999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":1.1910000000000001,"15min":0.83099999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.377},"max":{"1min":1.022,"5min":17.145,"15min":17.145},"last":0.79900000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.95299999999999996,"15min":0.77100000000000002},"min":{"1min":0.46500000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.89300000000000002,"5min":11.124000000000001,"15min":11.124000000000001},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.749,"5min":1.7629999999999999,"15min":1.0780000000000001},"min":{"1min":0.45600000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.339,"5min":28.872,"15min":28.872},"last":0.82299999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":1.5409999999999999,"15min":0.99299999999999999},"min":{"1min":0.45800000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.3169999999999999,"5min":39.256,"15min":39.256},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.70799999999999996,"5min":1.1160000000000001,"15min":0.89200000000000002},"min":{"1min":0.51900000000000002,"5min":0.46500000000000002,"15min":0.42199999999999999},"max":{"1min":1.0329999999999999,"5min":26.573,"15min":26.573},"last":0.94399999999999995}]}]},{"osd":3,"up_from":15,"seq":64424509530,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50060,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321780,"statfs":{"total":96636764160,"available":96585502720,"internally_reserved":0,"allocated":851968,"data_stored":357984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58899999999999997,"15min":0.47999999999999998},"min":{"1min":0.26400000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67700000000000005,"5min":5.8849999999999998,"15min":5.8849999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.80900000000000005,"15min":0.59299999999999997},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.96999999999999997,"5min":17.756,"15min":17.756},"last":0.39600000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.75,"15min":0.58999999999999997},"min":{"1min":0.29499999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.68999999999999995,"5min":12.307,"15min":12.307},"last":0.44},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.66400000000000003,"15min":0.57899999999999996},"min":{"1min":0.35199999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.72099999999999997,"5min":6.0640000000000001,"15min":6.0640000000000001},"last":0.56299999999999994}]},{"osd":2,"last update":"Mon Jan 15 18:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.85799999999999998,"15min":0.65900000000000003},"min":{"1min":0.44700000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.89600000000000002,"5min":12.468999999999999,"15min":12.468999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.76200000000000001,"15min":0.63700000000000001},"min":{"1min":0.442,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79100000000000004,"5min":6.0330000000000004,"15min":6.0330000000000004},"last":0.53200000000000003}]},{"osd":4,"last update":"Mon Jan 15 18:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.1859999999999999,"15min":0.92900000000000005},"min":{"1min":0.53300000000000003,"5min":0.375,"15min":0.375},"max":{"1min":4.1239999999999997,"5min":17.853999999999999,"15min":17.853999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.1639999999999999,"15min":0.90600000000000003},"min":{"1min":0.61799999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":4.0960000000000001,"5min":13.31,"15min":13.31},"last":0.63600000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":1.0289999999999999,"15min":0.83699999999999997},"min":{"1min":0.58199999999999996,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.6419999999999999,"5min":6.1139999999999999,"15min":6.1139999999999999},"last":0.753},{"interface":"front","average":{"1min":0.82499999999999996,"5min":1.302,"15min":0.91500000000000004},"min":{"1min":0.59899999999999998,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.458,"5min":20.515999999999998,"15min":20.515999999999998},"last":0.83699999999999997}]},{"osd":6,"last update":"Mon Jan 15 18:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":1.345,"15min":0.97099999999999997},"min":{"1min":0.61499999999999999,"5min":0.432,"15min":0.432},"max":{"1min":1.845,"5min":23.100999999999999,"15min":23.100999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.82699999999999996,"5min":1.133,"15min":0.89000000000000001},"min":{"1min":0.624,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.133,"5min":12.76,"15min":12.76},"last":0.59999999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":1.379,"15min":0.98099999999999998},"min":{"1min":0.53800000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":2.016,"5min":21.545000000000002,"15min":21.545000000000002},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.94599999999999995,"5min":1.3560000000000001,"15min":0.997},"min":{"1min":0.63800000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":3.2229999999999999,"5min":13.385,"15min":13.385},"last":0.82199999999999995}]}]},{"osd":2,"up_from":15,"seq":64424509530,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45904,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325936,"statfs":{"total":96636764160,"available":96589758464,"internally_reserved":0,"allocated":856064,"data_stored":359484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.61699999999999999,"15min":0.51200000000000001},"min":{"1min":0.29399999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.8480000000000001,"5min":5.1849999999999996,"15min":5.1849999999999996},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.64200000000000002,"15min":0.52300000000000002},"min":{"1min":0.32600000000000001,"5min":0.29399999999999998,"15min":0.28299999999999997},"max":{"1min":2.3759999999999999,"5min":7.516,"15min":7.516},"last":0.40200000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.748,"15min":0.61199999999999999},"min":{"1min":0.34100000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":2.2850000000000001,"5min":10.907,"15min":10.907},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.81599999999999995,"15min":0.57399999999999995},"min":{"1min":0.40500000000000003,"5min":0.311,"15min":0.27000000000000002},"max":{"1min":1.7150000000000001,"5min":8.9190000000000005,"15min":8.9190000000000005},"last":0.54600000000000004}]},{"osd":3,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.73599999999999999,"15min":0.628},"min":{"1min":0.36599999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.86399999999999999,"5min":12.185,"15min":12.185},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.83499999999999996,"15min":0.64700000000000002},"min":{"1min":0.443,"5min":0.39100000000000001,"15min":0.34000000000000002},"max":{"1min":2.3330000000000002,"5min":7.6760000000000002,"15min":7.6760000000000002},"last":0.622}]},{"osd":4,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.99299999999999999,"15min":0.82699999999999996},"min":{"1min":0.58699999999999997,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":2.21,"5min":8.6609999999999996,"15min":8.6609999999999996},"last":1.018},{"interface":"front","average":{"1min":0.76000000000000001,"5min":1.1899999999999999,"15min":0.90600000000000003},"min":{"1min":0.54000000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.6339999999999999,"5min":19.146999999999998,"15min":19.146999999999998},"last":1.0649999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":1.1319999999999999,"15min":0.90800000000000003},"min":{"1min":0.442,"5min":0.44,"15min":0.44},"max":{"1min":0.90800000000000003,"5min":31.710000000000001,"15min":31.710000000000001},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.875,"5min":1.373,"15min":0.98299999999999998},"min":{"1min":0.59399999999999997,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.1669999999999998,"5min":22.539000000000001,"15min":22.539000000000001},"last":1.0940000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":1.107,"15min":0.88600000000000001},"min":{"1min":0.61699999999999999,"5min":0.50800000000000001,"15min":0.48299999999999998},"max":{"1min":1.675,"5min":9.4540000000000006,"15min":9.4540000000000006},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.1399999999999999,"15min":0.90900000000000003},"min":{"1min":0.53100000000000003,"5min":0.503,"15min":0.503},"max":{"1min":1,"5min":18.614999999999998,"15min":18.614999999999998},"last":0.97399999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":1.113,"15min":0.90400000000000003},"min":{"1min":0.58199999999999996,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.752,"5min":11.432,"15min":11.432},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.264,"15min":0.97099999999999997},"min":{"1min":0.54500000000000004,"5min":0.502,"15min":0.502},"max":{"1min":2.2480000000000002,"5min":17.013000000000002,"15min":17.013000000000002},"last":0.94499999999999995}]}]},{"osd":1,"up_from":15,"seq":64424509532,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50072,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321768,"statfs":{"total":96636764160,"available":96585490432,"internally_reserved":0,"allocated":864256,"data_stored":360100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.96399999999999997,"15min":0.65500000000000003},"min":{"1min":0.309,"5min":0.26500000000000001,"15min":0.254},"max":{"1min":0.72899999999999998,"5min":19.577000000000002,"15min":19.577000000000002},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":1.1779999999999999,"15min":0.71499999999999997},"min":{"1min":0.35399999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.80000000000000004,"5min":48.277999999999999,"15min":48.277999999999999},"last":0.56999999999999995}]},{"osd":2,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":1.036,"15min":0.70599999999999996},"min":{"1min":0.35599999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.65000000000000002,"5min":40.698,"15min":40.698},"last":0.504},{"interface":"front","average":{"1min":0.53200000000000003,"5min":1.337,"15min":0.92900000000000005},"min":{"1min":0.36899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93300000000000005,"5min":48.316000000000003,"15min":48.316000000000003},"last":0.48399999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.2370000000000001,"15min":0.83999999999999997},"min":{"1min":0.46600000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.88200000000000001,"5min":48.234999999999999,"15min":48.234999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.75800000000000001,"15min":0.64500000000000002},"min":{"1min":0.36099999999999999,"5min":0.34699999999999998,"15min":0.311},"max":{"1min":0.74399999999999999,"5min":6.468,"15min":6.468},"last":0.54700000000000004}]},{"osd":4,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":1.0589999999999999,"15min":0.81799999999999995},"min":{"1min":0.48999999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.0720000000000001,"5min":8.3409999999999993,"15min":8.3409999999999993},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.80200000000000005,"5min":1.165,"15min":0.90600000000000003},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.42499999999999999},"max":{"1min":1.278,"5min":19.829000000000001,"15min":19.829000000000001},"last":0.74199999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":1.4399999999999999,"15min":0.99399999999999999},"min":{"1min":0.59499999999999997,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.306,"5min":22.594999999999999,"15min":22.594999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.79400000000000004,"5min":1.8939999999999999,"15min":1.145},"min":{"1min":0.54500000000000004,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1180000000000001,"5min":48.198,"15min":48.198},"last":0.69499999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":1.4590000000000001,"15min":0.98899999999999999},"min":{"1min":0.59899999999999998,"5min":0.53000000000000003,"15min":0.47499999999999998},"max":{"1min":1.1499999999999999,"5min":35.112000000000002,"15min":35.112000000000002},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.81200000000000006,"5min":1.1779999999999999,"15min":0.93999999999999995},"min":{"1min":0.53100000000000003,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.175,"5min":19.780999999999999,"15min":19.780999999999999},"last":0.86099999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":1.8069999999999999,"15min":1.1259999999999999},"min":{"1min":0.628,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2450000000000001,"5min":48.148000000000003,"15min":48.148000000000003},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.80300000000000005,"5min":1.194,"15min":0.93300000000000005},"min":{"1min":0.64000000000000001,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.214,"5min":9.3840000000000003,"15min":9.3840000000000003},"last":0.76000000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542235,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50064,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321776,"statfs":{"total":96636764160,"available":96585498624,"internally_reserved":0,"allocated":847872,"data_stored":357962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.872,"15min":0.77600000000000002},"min":{"1min":0.45900000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.3019999999999996,"5min":21.385000000000002,"15min":21.385000000000002},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.732,"5min":1.071,"15min":0.88800000000000001},"min":{"1min":0.46999999999999997,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":9.359,"5min":9.359,"15min":9.359},"last":0.78200000000000003}]},{"osd":1,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":3.0960000000000001,"5min":1.1839999999999999,"15min":0.84599999999999997},"min":{"1min":0.48799999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":21.120000000000001,"5min":21.120000000000001,"15min":21.120000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0329999999999999,"15min":0.80100000000000005},"min":{"1min":0.52400000000000002,"5min":0.315,"15min":0.315},"max":{"1min":4.3730000000000002,"5min":21.5,"15min":21.5},"last":0.753}]},{"osd":2,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.1419999999999999,"15min":0.85399999999999998},"min":{"1min":0.56399999999999995,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":6.2750000000000004,"5min":8.7750000000000004,"15min":8.7750000000000004},"last":0.73799999999999999},{"interface":"front","average":{"1min":2.9780000000000002,"5min":1.2869999999999999,"15min":0.94999999999999996},"min":{"1min":0.58099999999999996,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":21.201000000000001,"5min":21.201000000000001,"15min":21.201000000000001},"last":0.76600000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.0069999999999999,"15min":0.875},"min":{"1min":0.57399999999999995,"5min":0.44,"15min":0.44},"max":{"1min":4.4100000000000001,"5min":21.457000000000001,"15min":21.457000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.583,"5min":1.196,"15min":0.96799999999999997},"min":{"1min":0.54200000000000004,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":6.8650000000000002,"5min":8.7300000000000004,"15min":8.7300000000000004},"last":0.64900000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.012,"15min":0.82799999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":5.8929999999999998,"5min":9.6679999999999993,"15min":9.6679999999999993},"last":0.55700000000000005},{"interface":"front","average":{"1min":3.8380000000000001,"5min":1.5329999999999999,"15min":1.008},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":21.724,"5min":26.146999999999998,"15min":26.146999999999998},"last":0.70099999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.91900000000000004,"15min":0.79800000000000004},"min":{"1min":0.36099999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.835,"5min":21.535,"15min":21.535},"last":0.67600000000000005},{"interface":"front","average":{"1min":1.6160000000000001,"5min":1.3839999999999999,"15min":0.999},"min":{"1min":0.40100000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":5.7539999999999996,"5min":30.488,"15min":30.488},"last":0.82099999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":4.7309999999999999,"5min":1.6259999999999999,"15min":1.0760000000000001},"min":{"1min":0.42599999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":30.483000000000001,"5min":30.483000000000001,"15min":30.483000000000001},"last":0.629},{"interface":"front","average":{"1min":3.508,"5min":1.4650000000000001,"15min":1.0429999999999999},"min":{"1min":0.42899999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":25.437999999999999,"5min":25.437999999999999,"15min":25.437999999999999},"last":0.72199999999999998}]}]},{"osd":0,"up_from":14,"seq":60129542235,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54800,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317040,"statfs":{"total":96636764160,"available":96580648960,"internally_reserved":0,"allocated":1437696,"data_stored":948240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.89300000000000002,"15min":0.60699999999999998},"min":{"1min":0.26000000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9179999999999999,"5min":19.843,"15min":19.843},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.88700000000000001,"15min":0.66400000000000003},"min":{"1min":0.29199999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73299999999999998,"5min":10.154,"15min":10.154},"last":0.51600000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":1.115,"15min":0.753},"min":{"1min":0.38800000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.754,"5min":25.905000000000001,"15min":25.905000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.77900000000000003,"15min":0.58199999999999996},"min":{"1min":0.29699999999999999,"5min":0.28599999999999998,"15min":0.26700000000000002},"max":{"1min":1.2549999999999999,"5min":9.3829999999999991,"15min":9.3829999999999991},"last":0.47199999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.95699999999999996,"15min":0.70299999999999996},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.308},"max":{"1min":1.952,"5min":9.1780000000000008,"15min":9.1780000000000008},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.97399999999999998,"15min":0.73099999999999998},"min":{"1min":0.42899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":2.9809999999999999,"5min":10.210000000000001,"15min":10.210000000000001},"last":0.48899999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":1.343,"15min":0.96399999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.38},"max":{"1min":1.306,"5min":22.814,"15min":22.814},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.446,"15min":1.03},"min":{"1min":0.49099999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":2.169,"5min":23.341999999999999,"15min":23.341999999999999},"last":0.877}]},{"osd":5,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":1.262,"15min":0.94899999999999995},"min":{"1min":0.46899999999999997,"5min":0.44600000000000001,"15min":0.441},"max":{"1min":1.365,"5min":12.428000000000001,"15min":12.428000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.82499999999999996,"5min":1.6930000000000001,"15min":1.1259999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.46300000000000002},"max":{"1min":2.0880000000000001,"5min":19.774000000000001,"15min":19.774000000000001},"last":0.81399999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":1.9219999999999999,"15min":1.1719999999999999},"min":{"1min":0.46700000000000003,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.0449999999999999,"5min":31.059000000000001,"15min":31.059000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.83299999999999996,"5min":1.488,"15min":1.0580000000000001},"min":{"1min":0.57999999999999996,"5min":0.51500000000000001,"15min":0.49099999999999999},"max":{"1min":1.3899999999999999,"5min":40.731000000000002,"15min":40.731000000000002},"last":0.90600000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":1.673,"15min":1.1100000000000001},"min":{"1min":0.61599999999999999,"5min":0.47199999999999998,"15min":0.42499999999999999},"max":{"1min":1.4239999999999999,"5min":24.492999999999999,"15min":24.492999999999999},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.875,"5min":1.5780000000000001,"15min":1.097},"min":{"1min":0.61199999999999999,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":2.2130000000000001,"5min":23.524000000000001,"15min":23.524000000000001},"last":0.77800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:12:09.694 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:12:09.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e7c600640 1 Processor -- start 2024-01-15T18:12:09.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e7c600640 1 -- start start 2024-01-15T18:12:09.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e7c600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9e7415e7d0 0x7f9e740cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:09.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e7c600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e740b38b0 con 0x7f9e740cedc0 2024-01-15T18:12:09.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e7c600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e740cd550 con 0x7f9e740b20d0 2024-01-15T18:12:09.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e7c600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e740cd270 con 0x7f9e7415e7d0 2024-01-15T18:12:09.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e79200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9e740cedc0 0x7f9e740cba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59440/0 (socket says 172.21.15.27:59440) 2024-01-15T18:12:09.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e79200640 1 -- 172.21.15.27:0/915934226 learned_addr learned my addr 172.21.15.27:0/915934226 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:12:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e79c00640 1 --2- 172.21.15.27:0/915934226 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9e7415e7d0 0x7f9e740cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e73e00640 1 -- 172.21.15.27:0/915934226 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285839801 0 0) 0x7f9e740b38b0 con 0x7f9e740cedc0 2024-01-15T18:12:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e73e00640 1 -- 172.21.15.27:0/915934226 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e54003530 con 0x7f9e740cedc0 2024-01-15T18:12:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e73e00640 1 -- 172.21.15.27:0/915934226 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010016256 0 0) 0x7f9e740cd550 con 0x7f9e740b20d0 2024-01-15T18:12:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e73e00640 1 -- 172.21.15.27:0/915934226 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e740b38b0 con 0x7f9e740b20d0 2024-01-15T18:12:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e73e00640 1 -- 172.21.15.27:0/915934226 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1732981275 0 0) 0x7f9e54003530 con 0x7f9e740cedc0 2024-01-15T18:12:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e73e00640 1 -- 172.21.15.27:0/915934226 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9e7415e7d0 msgr2=0x7f9e740cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e73e00640 1 --2- 172.21.15.27:0/915934226 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9e7415e7d0 0x7f9e740cf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e73e00640 1 -- 172.21.15.27:0/915934226 >> v1:172.21.15.93:6789/0 conn(0x7f9e740b20d0 legacy=0x7f9e740cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e73e00640 1 -- 172.21.15.27:0/915934226 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e740b35d0 con 0x7f9e740cedc0 2024-01-15T18:12:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e73e00640 1 -- 172.21.15.27:0/915934226 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f9e5c001680 con 0x7f9e740cedc0 2024-01-15T18:12:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e73e00640 1 -- 172.21.15.27:0/915934226 >> v1:172.21.15.27:6789/0 conn(0x7f9e740cedc0 legacy=0x7f9e740cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e73e00640 1 --2- 172.21.15.27:0/915934226 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9e7415e7d0 0x7f9e54003bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:09.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e73e00640 1 --2- 172.21.15.27:0/915934226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e54004160 0x7f9e540047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e73e00640 1 --2- 172.21.15.27:0/915934226 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e54004d10 0x7f9e54005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e73e00640 1 -- 172.21.15.27:0/915934226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e740cd270 con 0x7f9e54004160 2024-01-15T18:12:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e73e00640 1 -- 172.21.15.27:0/915934226 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5400bb20 con 0x7f9e54004d10 2024-01-15T18:12:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e73e00640 1 -- 172.21.15.27:0/915934226 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5400bce0 con 0x7f9e7415e7d0 2024-01-15T18:12:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.847+0000 7f9e79200640 1 --2- 172.21.15.27:0/915934226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e54004160 0x7f9e540047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.851+0000 7f9e79c00640 1 --2- 172.21.15.27:0/915934226 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9e7415e7d0 0x7f9e54003bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.851+0000 7f9e7a600640 1 --2- 172.21.15.27:0/915934226 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e54004d10 0x7f9e54005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.851+0000 7f9e79200640 1 -- 172.21.15.27:0/915934226 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9e7415e7d0 msgr2=0x7f9e54003bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.851+0000 7f9e79200640 1 --2- 172.21.15.27:0/915934226 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9e7415e7d0 0x7f9e54003bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.851+0000 7f9e79200640 1 -- 172.21.15.27:0/915934226 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e54004d10 msgr2=0x7f9e54005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.851+0000 7f9e79200640 1 --2- 172.21.15.27:0/915934226 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e54004d10 0x7f9e54005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.851+0000 7f9e79200640 1 -- 172.21.15.27:0/915934226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e54003530 con 0x7f9e54004160 2024-01-15T18:12:09.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.851+0000 7f9e79200640 1 --2- 172.21.15.27:0/915934226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e54004160 0x7f9e540047d0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c007f80 tx=0x7f9e5c01cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=242e2bd215c5c91d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:09.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.851+0000 7f9e73e00640 1 -- 172.21.15.27:0/915934226 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c005ae0 con 0x7f9e54004160 2024-01-15T18:12:09.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.851+0000 7f9e73e00640 1 -- 172.21.15.27:0/915934226 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5c005c40 con 0x7f9e54004160 2024-01-15T18:12:09.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.851+0000 7f9e73e00640 1 -- 172.21.15.27:0/915934226 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c0a57e0 con 0x7f9e54004160 2024-01-15T18:12:09.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.855+0000 7f9e7c600640 1 -- 172.21.15.27:0/915934226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e54004160 msgr2=0x7f9e540047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:09.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.855+0000 7f9e7c600640 1 --2- 172.21.15.27:0/915934226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e54004160 0x7f9e540047d0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c007f80 tx=0x7f9e5c01cae0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.855+0000 7f9e7a600640 1 -- 172.21.15.27:0/915934226 reap_dead start 2024-01-15T18:12:09.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.855+0000 7f9e7c600640 1 -- 172.21.15.27:0/915934226 shutdown_connections 2024-01-15T18:12:09.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.855+0000 7f9e7c600640 1 -- 172.21.15.27:0/915934226 >> 172.21.15.27:0/915934226 conn(0x7f9e740b2a10 msgr2=0x7f9e740b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:09.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.855+0000 7f9e7c600640 1 -- 172.21.15.27:0/915934226 shutdown_connections 2024-01-15T18:12:09.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.855+0000 7f9e7c600640 1 -- 172.21.15.27:0/915934226 wait complete. 2024-01-15T18:12:09.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.855+0000 7f9e7c600640 1 Processor -- start 2024-01-15T18:12:09.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.855+0000 7f9e7c600640 1 -- start start 2024-01-15T18:12:09.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.855+0000 7f9e7c600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9e54004160 0x7f9e740b5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:09.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.855+0000 7f9e7c600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e74153220 0x7f9e7415cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:09.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.855+0000 7f9e7c600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e74153680 0x7f9e7415d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:09.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.855+0000 7f9e7c600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e740b4c00 con 0x7f9e74153680 2024-01-15T18:12:09.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.855+0000 7f9e7c600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e740b6150 con 0x7f9e74153220 2024-01-15T18:12:09.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.855+0000 7f9e7c600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e740b4e50 con 0x7f9e54004160 2024-01-15T18:12:09.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.855+0000 7f9e7a600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e74153680 0x7f9e7415d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:09.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.855+0000 7f9e7a600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e74153680 0x7f9e7415d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58376/0 (socket says 172.21.15.27:58376) 2024-01-15T18:12:09.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.855+0000 7f9e7a600640 1 -- 172.21.15.27:0/312975304 learned_addr learned my addr 172.21.15.27:0/312975304 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:09.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.855+0000 7f9e79200640 1 --2- 172.21.15.27:0/312975304 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e74153220 0x7f9e7415cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:09.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.855+0000 7f9e79c00640 1 --2- 172.21.15.27:0/312975304 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9e54004160 0x7f9e740b5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:09.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.859+0000 7f9e7a600640 1 -- 172.21.15.27:0/312975304 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9e54004160 msgr2=0x7f9e740b5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:09.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.859+0000 7f9e7a600640 1 --2- 172.21.15.27:0/312975304 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9e54004160 0x7f9e740b5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.859+0000 7f9e7a600640 1 -- 172.21.15.27:0/312975304 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e74153220 msgr2=0x7f9e7415cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:09.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.859+0000 7f9e7a600640 1 --2- 172.21.15.27:0/312975304 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e74153220 0x7f9e7415cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.859+0000 7f9e7a600640 1 -- 172.21.15.27:0/312975304 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e740b5130 con 0x7f9e74153680 2024-01-15T18:12:09.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.859+0000 7f9e7a600640 1 --2- 172.21.15.27:0/312975304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e74153680 0x7f9e7415d140 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c0059b0 tx=0x7f9e6c012c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:09.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.859+0000 7f9e71600640 1 -- 172.21.15.27:0/312975304 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c010a20 con 0x7f9e74153680 2024-01-15T18:12:09.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.859+0000 7f9e71600640 1 -- 172.21.15.27:0/312975304 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6c010b80 con 0x7f9e74153680 2024-01-15T18:12:09.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.859+0000 7f9e7c600640 1 -- 172.21.15.27:0/312975304 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e740b5410 con 0x7f9e74153680 2024-01-15T18:12:09.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.859+0000 7f9e71600640 1 -- 172.21.15.27:0/312975304 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c012e70 con 0x7f9e74153680 2024-01-15T18:12:09.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.859+0000 7f9e7c600640 1 -- 172.21.15.27:0/312975304 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e740c8950 con 0x7f9e74153680 2024-01-15T18:12:09.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.859+0000 7f9e7c600640 1 -- 172.21.15.27:0/312975304 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e7404e850 con 0x7f9e74153680 2024-01-15T18:12:09.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.859+0000 7f9e71600640 1 -- 172.21.15.27:0/312975304 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f9e6c016480 con 0x7f9e74153680 2024-01-15T18:12:09.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.859+0000 7f9e71600640 1 --2- 172.21.15.27:0/312975304 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9e4c067100 0x7f9e4c0695c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:09.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.859+0000 7f9e71600640 1 -- 172.21.15.27:0/312975304 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f9e6c085f60 con 0x7f9e74153680 2024-01-15T18:12:09.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.863+0000 7f9e79c00640 1 --2- 172.21.15.27:0/312975304 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9e4c067100 0x7f9e4c0695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:09.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.863+0000 7f9e79c00640 1 --2- 172.21.15.27:0/312975304 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9e4c067100 0x7f9e4c0695c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9e64009b00 tx=0x7f9e64009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:09.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:09.863+0000 7f9e71600640 1 -- 172.21.15.27:0/312975304 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9e6c054da0 con 0x7f9e74153680 2024-01-15T18:12:10.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.019+0000 7f9e7c600640 1 -- 172.21.15.27:0/312975304 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9e7415ebb0 con 0x7f9e4c067100 2024-01-15T18:12:10.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.031+0000 7f9e71600640 1 -- 172.21.15.27:0/312975304 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288658 (secure 0 0 0) 0x7f9e64019070 con 0x7f9e4c067100 2024-01-15T18:12:10.035 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:12:10.036 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:12:10.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.035+0000 7f9e7c600640 1 -- 172.21.15.27:0/312975304 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9e4c067100 msgr2=0x7f9e4c0695c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.035+0000 7f9e7c600640 1 --2- 172.21.15.27:0/312975304 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9e4c067100 0x7f9e4c0695c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9e64009b00 tx=0x7f9e64009190 comp rx=0 tx=0).stop 2024-01-15T18:12:10.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.035+0000 7f9e7c600640 1 -- 172.21.15.27:0/312975304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e74153680 msgr2=0x7f9e7415d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.035+0000 7f9e7c600640 1 --2- 172.21.15.27:0/312975304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e74153680 0x7f9e7415d140 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c0059b0 tx=0x7f9e6c012c20 comp rx=0 tx=0).stop 2024-01-15T18:12:10.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.035+0000 7f9e7c600640 1 -- 172.21.15.27:0/312975304 shutdown_connections 2024-01-15T18:12:10.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.035+0000 7f9e7c600640 1 --2- 172.21.15.27:0/312975304 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9e4c067100 0x7f9e4c0695c0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.035+0000 7f9e7c600640 1 --2- 172.21.15.27:0/312975304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e74153680 0x7f9e7415d140 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.035+0000 7f9e7c600640 1 --2- 172.21.15.27:0/312975304 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9e74153220 0x7f9e7415cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.035+0000 7f9e7c600640 1 --2- 172.21.15.27:0/312975304 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9e54004160 0x7f9e740b5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.035+0000 7f9e7c600640 1 -- 172.21.15.27:0/312975304 >> 172.21.15.27:0/312975304 conn(0x7f9e740b2a10 msgr2=0x7f9e740ce160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:10.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.035+0000 7f9e7c600640 1 -- 172.21.15.27:0/312975304 shutdown_connections 2024-01-15T18:12:10.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.035+0000 7f9e7c600640 1 -- 172.21.15.27:0/312975304 wait complete. 2024-01-15T18:12:10.056 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-01-15T18:12:08.260270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":397500,"kb_used_data":7816,"kb_used_omap":12,"kb_used_meta":389555,"kb_avail":754577220,"statfs":{"total":773094113280,"available":772687073280,"internally_reserved":0,"allocated":8003584,"data_stored":4048532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":398904880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002004"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756789+0000","last_change":"2024-01-15T18:12:01.024863+0000","last_active":"2024-01-15T18:12:02.756789+0000","last_peered":"2024-01-15T18:12:02.756789+0000","last_clean":"2024-01-15T18:12:02.756789+0000","last_became_active":"2024-01-15T18:12:01.024550+0000","last_became_peered":"2024-01-15T18:12:01.024550+0000","last_unstale":"2024-01-15T18:12:02.756789+0000","last_undegraded":"2024-01-15T18:12:02.756789+0000","last_fullsized":"2024-01-15T18:12:02.756789+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:25:17.902934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.757395+0000","last_change":"2024-01-15T18:12:01.011995+0000","last_active":"2024-01-15T18:12:02.757395+0000","last_peered":"2024-01-15T18:12:02.757395+0000","last_clean":"2024-01-15T18:12:02.757395+0000","last_became_active":"2024-01-15T18:12:01.011593+0000","last_became_peered":"2024-01-15T18:12:01.011593+0000","last_unstale":"2024-01-15T18:12:02.757395+0000","last_undegraded":"2024-01-15T18:12:02.757395+0000","last_fullsized":"2024-01-15T18:12:02.757395+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:18:04.676758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756930+0000","last_change":"2024-01-15T18:12:01.027262+0000","last_active":"2024-01-15T18:12:02.756930+0000","last_peered":"2024-01-15T18:12:02.756930+0000","last_clean":"2024-01-15T18:12:02.756930+0000","last_became_active":"2024-01-15T18:12:01.026976+0000","last_became_peered":"2024-01-15T18:12:01.026976+0000","last_unstale":"2024-01-15T18:12:02.756930+0000","last_undegraded":"2024-01-15T18:12:02.756930+0000","last_fullsized":"2024-01-15T18:12:02.756930+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:02:46.221698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756314+0000","last_change":"2024-01-15T18:12:01.023153+0000","last_active":"2024-01-15T18:12:02.756314+0000","last_peered":"2024-01-15T18:12:02.756314+0000","last_clean":"2024-01-15T18:12:02.756314+0000","last_became_active":"2024-01-15T18:12:01.022804+0000","last_became_peered":"2024-01-15T18:12:01.022804+0000","last_unstale":"2024-01-15T18:12:02.756314+0000","last_undegraded":"2024-01-15T18:12:02.756314+0000","last_fullsized":"2024-01-15T18:12:02.756314+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:26.599661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.007112+0000","last_change":"2024-01-15T18:12:01.030691+0000","last_active":"2024-01-15T18:12:02.007112+0000","last_peered":"2024-01-15T18:12:02.007112+0000","last_clean":"2024-01-15T18:12:02.007112+0000","last_became_active":"2024-01-15T18:12:01.030418+0000","last_became_peered":"2024-01-15T18:12:01.030418+0000","last_unstale":"2024-01-15T18:12:02.007112+0000","last_undegraded":"2024-01-15T18:12:02.007112+0000","last_fullsized":"2024-01-15T18:12:02.007112+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:54:50.644674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.755789+0000","last_change":"2024-01-15T18:12:01.011618+0000","last_active":"2024-01-15T18:12:02.755789+0000","last_peered":"2024-01-15T18:12:02.755789+0000","last_clean":"2024-01-15T18:12:02.755789+0000","last_became_active":"2024-01-15T18:12:01.011331+0000","last_became_peered":"2024-01-15T18:12:01.011331+0000","last_unstale":"2024-01-15T18:12:02.755789+0000","last_undegraded":"2024-01-15T18:12:02.755789+0000","last_fullsized":"2024-01-15T18:12:02.755789+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:12:20.020722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005570+0000","last_change":"2024-01-15T18:12:01.023552+0000","last_active":"2024-01-15T18:12:02.005570+0000","last_peered":"2024-01-15T18:12:02.005570+0000","last_clean":"2024-01-15T18:12:02.005570+0000","last_became_active":"2024-01-15T18:12:01.023332+0000","last_became_peered":"2024-01-15T18:12:01.023332+0000","last_unstale":"2024-01-15T18:12:02.005570+0000","last_undegraded":"2024-01-15T18:12:02.005570+0000","last_fullsized":"2024-01-15T18:12:02.005570+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:13:50.232455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.755868+0000","last_change":"2024-01-15T18:12:01.026981+0000","last_active":"2024-01-15T18:12:02.755868+0000","last_peered":"2024-01-15T18:12:02.755868+0000","last_clean":"2024-01-15T18:12:02.755868+0000","last_became_active":"2024-01-15T18:12:01.026616+0000","last_became_peered":"2024-01-15T18:12:01.026616+0000","last_unstale":"2024-01-15T18:12:02.755868+0000","last_undegraded":"2024-01-15T18:12:02.755868+0000","last_fullsized":"2024-01-15T18:12:02.755868+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:26.599661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.007768+0000","last_change":"2024-01-15T18:12:01.030330+0000","last_active":"2024-01-15T18:12:02.007768+0000","last_peered":"2024-01-15T18:12:02.007768+0000","last_clean":"2024-01-15T18:12:02.007768+0000","last_became_active":"2024-01-15T18:12:01.030085+0000","last_became_peered":"2024-01-15T18:12:01.030085+0000","last_unstale":"2024-01-15T18:12:02.007768+0000","last_undegraded":"2024-01-15T18:12:02.007768+0000","last_fullsized":"2024-01-15T18:12:02.007768+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:08:20.018505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756486+0000","last_change":"2024-01-15T18:12:01.014677+0000","last_active":"2024-01-15T18:12:02.756486+0000","last_peered":"2024-01-15T18:12:02.756486+0000","last_clean":"2024-01-15T18:12:02.756486+0000","last_became_active":"2024-01-15T18:12:01.014379+0000","last_became_peered":"2024-01-15T18:12:01.014379+0000","last_unstale":"2024-01-15T18:12:02.756486+0000","last_undegraded":"2024-01-15T18:12:02.756486+0000","last_fullsized":"2024-01-15T18:12:02.756486+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:03:56.065254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756472+0000","last_change":"2024-01-15T18:12:01.027756+0000","last_active":"2024-01-15T18:12:02.756472+0000","last_peered":"2024-01-15T18:12:02.756472+0000","last_clean":"2024-01-15T18:12:02.756472+0000","last_became_active":"2024-01-15T18:12:01.027426+0000","last_became_peered":"2024-01-15T18:12:01.027426+0000","last_unstale":"2024-01-15T18:12:02.756472+0000","last_undegraded":"2024-01-15T18:12:02.756472+0000","last_fullsized":"2024-01-15T18:12:02.756472+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:05.621191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756455+0000","last_change":"2024-01-15T18:12:01.023192+0000","last_active":"2024-01-15T18:12:02.756455+0000","last_peered":"2024-01-15T18:12:02.756455+0000","last_clean":"2024-01-15T18:12:02.756455+0000","last_became_active":"2024-01-15T18:12:01.022417+0000","last_became_peered":"2024-01-15T18:12:01.022417+0000","last_unstale":"2024-01-15T18:12:02.756455+0000","last_undegraded":"2024-01-15T18:12:02.756455+0000","last_fullsized":"2024-01-15T18:12:02.756455+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:02:44.137075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.006064+0000","last_change":"2024-01-15T18:12:01.030704+0000","last_active":"2024-01-15T18:12:02.006064+0000","last_peered":"2024-01-15T18:12:02.006064+0000","last_clean":"2024-01-15T18:12:02.006064+0000","last_became_active":"2024-01-15T18:12:01.030404+0000","last_became_peered":"2024-01-15T18:12:01.030404+0000","last_unstale":"2024-01-15T18:12:02.006064+0000","last_undegraded":"2024-01-15T18:12:02.006064+0000","last_fullsized":"2024-01-15T18:12:02.006064+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:05:44.296057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.007386+0000","last_change":"2024-01-15T18:12:01.026060+0000","last_active":"2024-01-15T18:12:02.007386+0000","last_peered":"2024-01-15T18:12:02.007386+0000","last_clean":"2024-01-15T18:12:02.007386+0000","last_became_active":"2024-01-15T18:12:01.025781+0000","last_became_peered":"2024-01-15T18:12:01.025781+0000","last_unstale":"2024-01-15T18:12:02.007386+0000","last_undegraded":"2024-01-15T18:12:02.007386+0000","last_fullsized":"2024-01-15T18:12:02.007386+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:00:42.420373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.006053+0000","last_change":"2024-01-15T18:12:01.026597+0000","last_active":"2024-01-15T18:12:02.006053+0000","last_peered":"2024-01-15T18:12:02.006053+0000","last_clean":"2024-01-15T18:12:02.006053+0000","last_became_active":"2024-01-15T18:12:01.026178+0000","last_became_peered":"2024-01-15T18:12:01.026178+0000","last_unstale":"2024-01-15T18:12:02.006053+0000","last_undegraded":"2024-01-15T18:12:02.006053+0000","last_fullsized":"2024-01-15T18:12:02.006053+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:04:19.070396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.757153+0000","last_change":"2024-01-15T18:12:01.011394+0000","last_active":"2024-01-15T18:12:02.757153+0000","last_peered":"2024-01-15T18:12:02.757153+0000","last_clean":"2024-01-15T18:12:02.757153+0000","last_became_active":"2024-01-15T18:12:01.011072+0000","last_became_peered":"2024-01-15T18:12:01.011072+0000","last_unstale":"2024-01-15T18:12:02.757153+0000","last_undegraded":"2024-01-15T18:12:02.757153+0000","last_fullsized":"2024-01-15T18:12:02.757153+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:41:30.756491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.007777+0000","last_change":"2024-01-15T18:12:01.029694+0000","last_active":"2024-01-15T18:12:02.007777+0000","last_peered":"2024-01-15T18:12:02.007777+0000","last_clean":"2024-01-15T18:12:02.007777+0000","last_became_active":"2024-01-15T18:12:01.029127+0000","last_became_peered":"2024-01-15T18:12:01.029127+0000","last_unstale":"2024-01-15T18:12:02.007777+0000","last_undegraded":"2024-01-15T18:12:02.007777+0000","last_fullsized":"2024-01-15T18:12:02.007777+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:50:00.994032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756791+0000","last_change":"2024-01-15T18:12:01.029765+0000","last_active":"2024-01-15T18:12:02.756791+0000","last_peered":"2024-01-15T18:12:02.756791+0000","last_clean":"2024-01-15T18:12:02.756791+0000","last_became_active":"2024-01-15T18:12:01.029504+0000","last_became_peered":"2024-01-15T18:12:01.029504+0000","last_unstale":"2024-01-15T18:12:02.756791+0000","last_undegraded":"2024-01-15T18:12:02.756791+0000","last_fullsized":"2024-01-15T18:12:02.756791+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:36:37.178674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756561+0000","last_change":"2024-01-15T18:12:01.026480+0000","last_active":"2024-01-15T18:12:02.756561+0000","last_peered":"2024-01-15T18:12:02.756561+0000","last_clean":"2024-01-15T18:12:02.756561+0000","last_became_active":"2024-01-15T18:12:01.026104+0000","last_became_peered":"2024-01-15T18:12:01.026104+0000","last_unstale":"2024-01-15T18:12:02.756561+0000","last_undegraded":"2024-01-15T18:12:02.756561+0000","last_fullsized":"2024-01-15T18:12:02.756561+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:37:54.171968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.006074+0000","last_change":"2024-01-15T18:12:01.030452+0000","last_active":"2024-01-15T18:12:02.006074+0000","last_peered":"2024-01-15T18:12:02.006074+0000","last_clean":"2024-01-15T18:12:02.006074+0000","last_became_active":"2024-01-15T18:12:01.027541+0000","last_became_peered":"2024-01-15T18:12:01.027541+0000","last_unstale":"2024-01-15T18:12:02.006074+0000","last_undegraded":"2024-01-15T18:12:02.006074+0000","last_fullsized":"2024-01-15T18:12:02.006074+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:18:47.967166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756659+0000","last_change":"2024-01-15T18:12:01.031213+0000","last_active":"2024-01-15T18:12:02.756659+0000","last_peered":"2024-01-15T18:12:02.756659+0000","last_clean":"2024-01-15T18:12:02.756659+0000","last_became_active":"2024-01-15T18:12:01.027530+0000","last_became_peered":"2024-01-15T18:12:01.027530+0000","last_unstale":"2024-01-15T18:12:02.756659+0000","last_undegraded":"2024-01-15T18:12:02.756659+0000","last_fullsized":"2024-01-15T18:12:02.756659+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:10:56.250682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756491+0000","last_change":"2024-01-15T18:12:01.019809+0000","last_active":"2024-01-15T18:12:02.756491+0000","last_peered":"2024-01-15T18:12:02.756491+0000","last_clean":"2024-01-15T18:12:02.756491+0000","last_became_active":"2024-01-15T18:12:01.019490+0000","last_became_peered":"2024-01-15T18:12:01.019490+0000","last_unstale":"2024-01-15T18:12:02.756491+0000","last_undegraded":"2024-01-15T18:12:02.756491+0000","last_fullsized":"2024-01-15T18:12:02.756491+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:54:29.425928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.755552+0000","last_change":"2024-01-15T18:12:01.037632+0000","last_active":"2024-01-15T18:12:02.755552+0000","last_peered":"2024-01-15T18:12:02.755552+0000","last_clean":"2024-01-15T18:12:02.755552+0000","last_became_active":"2024-01-15T18:12:01.037364+0000","last_became_peered":"2024-01-15T18:12:01.037364+0000","last_unstale":"2024-01-15T18:12:02.755552+0000","last_undegraded":"2024-01-15T18:12:02.755552+0000","last_fullsized":"2024-01-15T18:12:02.755552+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:29:44.252608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.755556+0000","last_change":"2024-01-15T18:12:01.033589+0000","last_active":"2024-01-15T18:12:02.755556+0000","last_peered":"2024-01-15T18:12:02.755556+0000","last_clean":"2024-01-15T18:12:02.755556+0000","last_became_active":"2024-01-15T18:12:01.033232+0000","last_became_peered":"2024-01-15T18:12:01.033232+0000","last_unstale":"2024-01-15T18:12:02.755556+0000","last_undegraded":"2024-01-15T18:12:02.755556+0000","last_fullsized":"2024-01-15T18:12:02.755556+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:33:41.475245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.757431+0000","last_change":"2024-01-15T18:12:01.027376+0000","last_active":"2024-01-15T18:12:02.757431+0000","last_peered":"2024-01-15T18:12:02.757431+0000","last_clean":"2024-01-15T18:12:02.757431+0000","last_became_active":"2024-01-15T18:12:01.026894+0000","last_became_peered":"2024-01-15T18:12:01.026894+0000","last_unstale":"2024-01-15T18:12:02.757431+0000","last_undegraded":"2024-01-15T18:12:02.757431+0000","last_fullsized":"2024-01-15T18:12:02.757431+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:11.559886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005958+0000","last_change":"2024-01-15T18:12:01.026872+0000","last_active":"2024-01-15T18:12:02.005958+0000","last_peered":"2024-01-15T18:12:02.005958+0000","last_clean":"2024-01-15T18:12:02.005958+0000","last_became_active":"2024-01-15T18:12:01.026554+0000","last_became_peered":"2024-01-15T18:12:01.026554+0000","last_unstale":"2024-01-15T18:12:02.005958+0000","last_undegraded":"2024-01-15T18:12:02.005958+0000","last_fullsized":"2024-01-15T18:12:02.005958+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:05:44.296057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.757074+0000","last_change":"2024-01-15T18:12:01.029998+0000","last_active":"2024-01-15T18:12:02.757074+0000","last_peered":"2024-01-15T18:12:02.757074+0000","last_clean":"2024-01-15T18:12:02.757074+0000","last_became_active":"2024-01-15T18:12:01.029687+0000","last_became_peered":"2024-01-15T18:12:01.029687+0000","last_unstale":"2024-01-15T18:12:02.757074+0000","last_undegraded":"2024-01-15T18:12:02.757074+0000","last_fullsized":"2024-01-15T18:12:02.757074+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:04:28.577498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756412+0000","last_change":"2024-01-15T18:12:01.014692+0000","last_active":"2024-01-15T18:12:02.756412+0000","last_peered":"2024-01-15T18:12:02.756412+0000","last_clean":"2024-01-15T18:12:02.756412+0000","last_became_active":"2024-01-15T18:12:01.014384+0000","last_became_peered":"2024-01-15T18:12:01.014384+0000","last_unstale":"2024-01-15T18:12:02.756412+0000","last_undegraded":"2024-01-15T18:12:02.756412+0000","last_fullsized":"2024-01-15T18:12:02.756412+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:55:27.047888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756790+0000","last_change":"2024-01-15T18:12:01.027026+0000","last_active":"2024-01-15T18:12:02.756790+0000","last_peered":"2024-01-15T18:12:02.756790+0000","last_clean":"2024-01-15T18:12:02.756790+0000","last_became_active":"2024-01-15T18:12:01.026727+0000","last_became_peered":"2024-01-15T18:12:01.026727+0000","last_unstale":"2024-01-15T18:12:02.756790+0000","last_undegraded":"2024-01-15T18:12:02.756790+0000","last_fullsized":"2024-01-15T18:12:02.756790+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:21:01.515615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005852+0000","last_change":"2024-01-15T18:12:01.025557+0000","last_active":"2024-01-15T18:12:02.005852+0000","last_peered":"2024-01-15T18:12:02.005852+0000","last_clean":"2024-01-15T18:12:02.005852+0000","last_became_active":"2024-01-15T18:12:01.024972+0000","last_became_peered":"2024-01-15T18:12:01.024972+0000","last_unstale":"2024-01-15T18:12:02.005852+0000","last_undegraded":"2024-01-15T18:12:02.005852+0000","last_fullsized":"2024-01-15T18:12:02.005852+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:46:42.441463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005732+0000","last_change":"2024-01-15T18:12:01.023561+0000","last_active":"2024-01-15T18:12:02.005732+0000","last_peered":"2024-01-15T18:12:02.005732+0000","last_clean":"2024-01-15T18:12:02.005732+0000","last_became_active":"2024-01-15T18:12:01.022979+0000","last_became_peered":"2024-01-15T18:12:01.022979+0000","last_unstale":"2024-01-15T18:12:02.005732+0000","last_undegraded":"2024-01-15T18:12:02.005732+0000","last_fullsized":"2024-01-15T18:12:02.005732+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:58:49.958276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.755914+0000","last_change":"2024-01-15T18:12:01.027732+0000","last_active":"2024-01-15T18:12:02.755914+0000","last_peered":"2024-01-15T18:12:02.755914+0000","last_clean":"2024-01-15T18:12:02.755914+0000","last_became_active":"2024-01-15T18:12:01.027455+0000","last_became_peered":"2024-01-15T18:12:01.027455+0000","last_unstale":"2024-01-15T18:12:02.755914+0000","last_undegraded":"2024-01-15T18:12:02.755914+0000","last_fullsized":"2024-01-15T18:12:02.755914+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:02:44.137075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756530+0000","last_change":"2024-01-15T18:12:01.034752+0000","last_active":"2024-01-15T18:12:02.756530+0000","last_peered":"2024-01-15T18:12:02.756530+0000","last_clean":"2024-01-15T18:12:02.756530+0000","last_became_active":"2024-01-15T18:12:01.033222+0000","last_became_peered":"2024-01-15T18:12:01.033222+0000","last_unstale":"2024-01-15T18:12:02.756530+0000","last_undegraded":"2024-01-15T18:12:02.756530+0000","last_fullsized":"2024-01-15T18:12:02.756530+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:53:50.857705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"69'4","reported_seq":16,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.045456+0000","last_change":"2024-01-15T18:12:00.026981+0000","last_active":"2024-01-15T18:12:01.045456+0000","last_peered":"2024-01-15T18:12:01.045456+0000","last_clean":"2024-01-15T18:12:01.045456+0000","last_became_active":"2024-01-15T18:12:00.026446+0000","last_became_peered":"2024-01-15T18:12:00.026446+0000","last_unstale":"2024-01-15T18:12:01.045456+0000","last_undegraded":"2024-01-15T18:12:01.045456+0000","last_fullsized":"2024-01-15T18:12:01.045456+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:56:18.635395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.007841+0000","last_change":"2024-01-15T18:12:01.028777+0000","last_active":"2024-01-15T18:12:02.007841+0000","last_peered":"2024-01-15T18:12:02.007841+0000","last_clean":"2024-01-15T18:12:02.007841+0000","last_became_active":"2024-01-15T18:12:01.028450+0000","last_became_peered":"2024-01-15T18:12:01.028450+0000","last_unstale":"2024-01-15T18:12:02.007841+0000","last_undegraded":"2024-01-15T18:12:02.007841+0000","last_fullsized":"2024-01-15T18:12:02.007841+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:36:16.553457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"69'1","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.042664+0000","last_change":"2024-01-15T18:12:00.018709+0000","last_active":"2024-01-15T18:12:01.042664+0000","last_peered":"2024-01-15T18:12:01.042664+0000","last_clean":"2024-01-15T18:12:01.042664+0000","last_became_active":"2024-01-15T18:12:00.018436+0000","last_became_peered":"2024-01-15T18:12:00.018436+0000","last_unstale":"2024-01-15T18:12:01.042664+0000","last_undegraded":"2024-01-15T18:12:01.042664+0000","last_fullsized":"2024-01-15T18:12:01.042664+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:13:49.229168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.757001+0000","last_change":"2024-01-15T18:12:01.026312+0000","last_active":"2024-01-15T18:12:02.757001+0000","last_peered":"2024-01-15T18:12:02.757001+0000","last_clean":"2024-01-15T18:12:02.757001+0000","last_became_active":"2024-01-15T18:12:01.025949+0000","last_became_peered":"2024-01-15T18:12:01.025949+0000","last_unstale":"2024-01-15T18:12:02.757001+0000","last_undegraded":"2024-01-15T18:12:02.757001+0000","last_fullsized":"2024-01-15T18:12:02.757001+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:40:33.473282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.044385+0000","last_change":"2024-01-15T18:12:00.022131+0000","last_active":"2024-01-15T18:12:01.044385+0000","last_peered":"2024-01-15T18:12:01.044385+0000","last_clean":"2024-01-15T18:12:01.044385+0000","last_became_active":"2024-01-15T18:12:00.021836+0000","last_became_peered":"2024-01-15T18:12:00.021836+0000","last_unstale":"2024-01-15T18:12:01.044385+0000","last_undegraded":"2024-01-15T18:12:01.044385+0000","last_fullsized":"2024-01-15T18:12:01.044385+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:11:11.204604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756616+0000","last_change":"2024-01-15T18:12:01.034660+0000","last_active":"2024-01-15T18:12:02.756616+0000","last_peered":"2024-01-15T18:12:02.756616+0000","last_clean":"2024-01-15T18:12:02.756616+0000","last_became_active":"2024-01-15T18:12:01.034177+0000","last_became_peered":"2024-01-15T18:12:01.034177+0000","last_unstale":"2024-01-15T18:12:02.756616+0000","last_undegraded":"2024-01-15T18:12:02.756616+0000","last_fullsized":"2024-01-15T18:12:02.756616+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:37:09.206892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.043171+0000","last_change":"2024-01-15T18:12:00.019217+0000","last_active":"2024-01-15T18:12:01.043171+0000","last_peered":"2024-01-15T18:12:01.043171+0000","last_clean":"2024-01-15T18:12:01.043171+0000","last_became_active":"2024-01-15T18:12:00.018924+0000","last_became_peered":"2024-01-15T18:12:00.018924+0000","last_unstale":"2024-01-15T18:12:01.043171+0000","last_undegraded":"2024-01-15T18:12:01.043171+0000","last_fullsized":"2024-01-15T18:12:01.043171+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:06:24.967935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.006011+0000","last_change":"2024-01-15T18:12:01.028720+0000","last_active":"2024-01-15T18:12:02.006011+0000","last_peered":"2024-01-15T18:12:02.006011+0000","last_clean":"2024-01-15T18:12:02.006011+0000","last_became_active":"2024-01-15T18:12:01.028453+0000","last_became_peered":"2024-01-15T18:12:01.028453+0000","last_unstale":"2024-01-15T18:12:02.006011+0000","last_undegraded":"2024-01-15T18:12:02.006011+0000","last_fullsized":"2024-01-15T18:12:02.006011+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:20.609600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"69'1","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.039628+0000","last_change":"2024-01-15T18:12:00.017547+0000","last_active":"2024-01-15T18:12:01.039628+0000","last_peered":"2024-01-15T18:12:01.039628+0000","last_clean":"2024-01-15T18:12:01.039628+0000","last_became_active":"2024-01-15T18:12:00.017260+0000","last_became_peered":"2024-01-15T18:12:00.017260+0000","last_unstale":"2024-01-15T18:12:01.039628+0000","last_undegraded":"2024-01-15T18:12:01.039628+0000","last_fullsized":"2024-01-15T18:12:01.039628+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:37:08.203604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756307+0000","last_change":"2024-01-15T18:12:01.027542+0000","last_active":"2024-01-15T18:12:02.756307+0000","last_peered":"2024-01-15T18:12:02.756307+0000","last_clean":"2024-01-15T18:12:02.756307+0000","last_became_active":"2024-01-15T18:12:01.027247+0000","last_became_peered":"2024-01-15T18:12:01.027247+0000","last_unstale":"2024-01-15T18:12:02.756307+0000","last_undegraded":"2024-01-15T18:12:02.756307+0000","last_fullsized":"2024-01-15T18:12:02.756307+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:07.393446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"69'1","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.041945+0000","last_change":"2024-01-15T18:12:00.025697+0000","last_active":"2024-01-15T18:12:01.041945+0000","last_peered":"2024-01-15T18:12:01.041945+0000","last_clean":"2024-01-15T18:12:01.041945+0000","last_became_active":"2024-01-15T18:12:00.025383+0000","last_became_peered":"2024-01-15T18:12:00.025383+0000","last_unstale":"2024-01-15T18:12:01.041945+0000","last_undegraded":"2024-01-15T18:12:01.041945+0000","last_fullsized":"2024-01-15T18:12:01.041945+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:27:28.185512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.006976+0000","last_change":"2024-01-15T18:12:01.014933+0000","last_active":"2024-01-15T18:12:02.006976+0000","last_peered":"2024-01-15T18:12:02.006976+0000","last_clean":"2024-01-15T18:12:02.006976+0000","last_became_active":"2024-01-15T18:12:01.014638+0000","last_became_peered":"2024-01-15T18:12:01.014638+0000","last_unstale":"2024-01-15T18:12:02.006976+0000","last_undegraded":"2024-01-15T18:12:02.006976+0000","last_fullsized":"2024-01-15T18:12:02.006976+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:53:36.228024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.013121+0000","last_change":"2024-01-15T18:12:00.019136+0000","last_active":"2024-01-15T18:12:01.013121+0000","last_peered":"2024-01-15T18:12:01.013121+0000","last_clean":"2024-01-15T18:12:01.013121+0000","last_became_active":"2024-01-15T18:12:00.018512+0000","last_became_peered":"2024-01-15T18:12:00.018512+0000","last_unstale":"2024-01-15T18:12:01.013121+0000","last_undegraded":"2024-01-15T18:12:01.013121+0000","last_fullsized":"2024-01-15T18:12:01.013121+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:13:56.868830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005875+0000","last_change":"2024-01-15T18:12:01.042936+0000","last_active":"2024-01-15T18:12:02.005875+0000","last_peered":"2024-01-15T18:12:02.005875+0000","last_clean":"2024-01-15T18:12:02.005875+0000","last_became_active":"2024-01-15T18:12:01.042276+0000","last_became_peered":"2024-01-15T18:12:01.042276+0000","last_unstale":"2024-01-15T18:12:02.005875+0000","last_undegraded":"2024-01-15T18:12:02.005875+0000","last_fullsized":"2024-01-15T18:12:02.005875+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:46:42.441463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.005620+0000","last_change":"2024-01-15T18:12:00.021284+0000","last_active":"2024-01-15T18:12:01.005620+0000","last_peered":"2024-01-15T18:12:01.005620+0000","last_clean":"2024-01-15T18:12:01.005620+0000","last_became_active":"2024-01-15T18:12:00.021017+0000","last_became_peered":"2024-01-15T18:12:00.021017+0000","last_unstale":"2024-01-15T18:12:01.005620+0000","last_undegraded":"2024-01-15T18:12:01.005620+0000","last_fullsized":"2024-01-15T18:12:01.005620+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:45.089127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005945+0000","last_change":"2024-01-15T18:12:01.042737+0000","last_active":"2024-01-15T18:12:02.005945+0000","last_peered":"2024-01-15T18:12:02.005945+0000","last_clean":"2024-01-15T18:12:02.005945+0000","last_became_active":"2024-01-15T18:12:01.042482+0000","last_became_peered":"2024-01-15T18:12:01.042482+0000","last_unstale":"2024-01-15T18:12:02.005945+0000","last_undegraded":"2024-01-15T18:12:02.005945+0000","last_fullsized":"2024-01-15T18:12:02.005945+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:04:19.070396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.008493+0000","last_change":"2024-01-15T18:12:00.017218+0000","last_active":"2024-01-15T18:12:01.008493+0000","last_peered":"2024-01-15T18:12:01.008493+0000","last_clean":"2024-01-15T18:12:01.008493+0000","last_became_active":"2024-01-15T18:12:00.016921+0000","last_became_peered":"2024-01-15T18:12:00.016921+0000","last_unstale":"2024-01-15T18:12:01.008493+0000","last_undegraded":"2024-01-15T18:12:01.008493+0000","last_fullsized":"2024-01-15T18:12:01.008493+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:08:25.595491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756886+0000","last_change":"2024-01-15T18:12:01.034009+0000","last_active":"2024-01-15T18:12:02.756886+0000","last_peered":"2024-01-15T18:12:02.756886+0000","last_clean":"2024-01-15T18:12:02.756886+0000","last_became_active":"2024-01-15T18:12:01.033712+0000","last_became_peered":"2024-01-15T18:12:01.033712+0000","last_unstale":"2024-01-15T18:12:02.756886+0000","last_undegraded":"2024-01-15T18:12:02.756886+0000","last_fullsized":"2024-01-15T18:12:02.756886+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:36:21.428758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.009024+0000","last_change":"2024-01-15T18:12:00.018102+0000","last_active":"2024-01-15T18:12:01.009024+0000","last_peered":"2024-01-15T18:12:01.009024+0000","last_clean":"2024-01-15T18:12:01.009024+0000","last_became_active":"2024-01-15T18:12:00.017781+0000","last_became_peered":"2024-01-15T18:12:00.017781+0000","last_unstale":"2024-01-15T18:12:01.009024+0000","last_undegraded":"2024-01-15T18:12:01.009024+0000","last_fullsized":"2024-01-15T18:12:01.009024+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:46:51.225073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.757140+0000","last_change":"2024-01-15T18:12:01.012486+0000","last_active":"2024-01-15T18:12:02.757140+0000","last_peered":"2024-01-15T18:12:02.757140+0000","last_clean":"2024-01-15T18:12:02.757140+0000","last_became_active":"2024-01-15T18:12:01.010951+0000","last_became_peered":"2024-01-15T18:12:01.010951+0000","last_unstale":"2024-01-15T18:12:02.757140+0000","last_undegraded":"2024-01-15T18:12:02.757140+0000","last_fullsized":"2024-01-15T18:12:02.757140+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:19:30.413967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"69'3","reported_seq":15,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.041771+0000","last_change":"2024-01-15T18:12:00.019006+0000","last_active":"2024-01-15T18:12:01.041771+0000","last_peered":"2024-01-15T18:12:01.041771+0000","last_clean":"2024-01-15T18:12:01.041771+0000","last_became_active":"2024-01-15T18:12:00.017853+0000","last_became_peered":"2024-01-15T18:12:00.017853+0000","last_unstale":"2024-01-15T18:12:01.041771+0000","last_undegraded":"2024-01-15T18:12:01.041771+0000","last_fullsized":"2024-01-15T18:12:01.041771+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:29:43.249320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.007141+0000","last_change":"2024-01-15T18:12:01.025987+0000","last_active":"2024-01-15T18:12:02.007141+0000","last_peered":"2024-01-15T18:12:02.007141+0000","last_clean":"2024-01-15T18:12:02.007141+0000","last_became_active":"2024-01-15T18:12:01.025672+0000","last_became_peered":"2024-01-15T18:12:01.025672+0000","last_unstale":"2024-01-15T18:12:02.007141+0000","last_undegraded":"2024-01-15T18:12:02.007141+0000","last_fullsized":"2024-01-15T18:12:02.007141+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:28:36.163121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"69'6","reported_seq":18,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.046630+0000","last_change":"2024-01-15T18:12:00.011850+0000","last_active":"2024-01-15T18:12:01.046630+0000","last_peered":"2024-01-15T18:12:01.046630+0000","last_clean":"2024-01-15T18:12:01.046630+0000","last_became_active":"2024-01-15T18:12:00.011519+0000","last_became_peered":"2024-01-15T18:12:00.011519+0000","last_unstale":"2024-01-15T18:12:01.046630+0000","last_undegraded":"2024-01-15T18:12:01.046630+0000","last_fullsized":"2024-01-15T18:12:01.046630+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:17:27.915008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.755541+0000","last_change":"2024-01-15T18:12:01.033995+0000","last_active":"2024-01-15T18:12:02.755541+0000","last_peered":"2024-01-15T18:12:02.755541+0000","last_clean":"2024-01-15T18:12:02.755541+0000","last_became_active":"2024-01-15T18:12:01.033680+0000","last_became_peered":"2024-01-15T18:12:01.033680+0000","last_unstale":"2024-01-15T18:12:02.755541+0000","last_undegraded":"2024-01-15T18:12:02.755541+0000","last_fullsized":"2024-01-15T18:12:02.755541+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:09:53.302299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.044808+0000","last_change":"2024-01-15T18:12:00.028242+0000","last_active":"2024-01-15T18:12:01.044808+0000","last_peered":"2024-01-15T18:12:01.044808+0000","last_clean":"2024-01-15T18:12:01.044808+0000","last_became_active":"2024-01-15T18:12:00.027987+0000","last_became_peered":"2024-01-15T18:12:00.027987+0000","last_unstale":"2024-01-15T18:12:01.044808+0000","last_undegraded":"2024-01-15T18:12:01.044808+0000","last_fullsized":"2024-01-15T18:12:01.044808+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:06:24.967935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005827+0000","last_change":"2024-01-15T18:12:01.024849+0000","last_active":"2024-01-15T18:12:02.005827+0000","last_peered":"2024-01-15T18:12:02.005827+0000","last_clean":"2024-01-15T18:12:02.005827+0000","last_became_active":"2024-01-15T18:12:01.024579+0000","last_became_peered":"2024-01-15T18:12:01.024579+0000","last_unstale":"2024-01-15T18:12:02.005827+0000","last_undegraded":"2024-01-15T18:12:02.005827+0000","last_fullsized":"2024-01-15T18:12:02.005827+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:43:37.142127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.007485+0000","last_change":"2024-01-15T18:12:00.014156+0000","last_active":"2024-01-15T18:12:01.007485+0000","last_peered":"2024-01-15T18:12:01.007485+0000","last_clean":"2024-01-15T18:12:01.007485+0000","last_became_active":"2024-01-15T18:12:00.013881+0000","last_became_peered":"2024-01-15T18:12:00.013881+0000","last_unstale":"2024-01-15T18:12:01.007485+0000","last_undegraded":"2024-01-15T18:12:01.007485+0000","last_fullsized":"2024-01-15T18:12:01.007485+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:49:33.673954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.005777+0000","last_change":"2024-01-15T18:12:01.032149+0000","last_active":"2024-01-15T18:12:02.005777+0000","last_peered":"2024-01-15T18:12:02.005777+0000","last_clean":"2024-01-15T18:12:02.005777+0000","last_became_active":"2024-01-15T18:12:01.030842+0000","last_became_peered":"2024-01-15T18:12:01.030842+0000","last_unstale":"2024-01-15T18:12:02.005777+0000","last_undegraded":"2024-01-15T18:12:02.005777+0000","last_fullsized":"2024-01-15T18:12:02.005777+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:43:37.142127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.014661+0000","last_change":"2024-01-15T18:12:00.025503+0000","last_active":"2024-01-15T18:12:01.014661+0000","last_peered":"2024-01-15T18:12:01.014661+0000","last_clean":"2024-01-15T18:12:01.014661+0000","last_became_active":"2024-01-15T18:12:00.025198+0000","last_became_peered":"2024-01-15T18:12:00.025198+0000","last_unstale":"2024-01-15T18:12:01.014661+0000","last_undegraded":"2024-01-15T18:12:01.014661+0000","last_fullsized":"2024-01-15T18:12:01.014661+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:06:28.115069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.007094+0000","last_change":"2024-01-15T18:12:01.026199+0000","last_active":"2024-01-15T18:12:02.007094+0000","last_peered":"2024-01-15T18:12:02.007094+0000","last_clean":"2024-01-15T18:12:02.007094+0000","last_became_active":"2024-01-15T18:12:01.025920+0000","last_became_peered":"2024-01-15T18:12:01.025920+0000","last_unstale":"2024-01-15T18:12:02.007094+0000","last_undegraded":"2024-01-15T18:12:02.007094+0000","last_fullsized":"2024-01-15T18:12:02.007094+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:53:27.484089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"69'8","reported_seq":20,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.045203+0000","last_change":"2024-01-15T18:12:00.025058+0000","last_active":"2024-01-15T18:12:01.045203+0000","last_peered":"2024-01-15T18:12:01.045203+0000","last_clean":"2024-01-15T18:12:01.045203+0000","last_became_active":"2024-01-15T18:12:00.024655+0000","last_became_peered":"2024-01-15T18:12:00.024655+0000","last_unstale":"2024-01-15T18:12:01.045203+0000","last_undegraded":"2024-01-15T18:12:01.045203+0000","last_fullsized":"2024-01-15T18:12:01.045203+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:33:10.196557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.757369+0000","last_change":"2024-01-15T18:12:01.023212+0000","last_active":"2024-01-15T18:12:02.757369+0000","last_peered":"2024-01-15T18:12:02.757369+0000","last_clean":"2024-01-15T18:12:02.757369+0000","last_became_active":"2024-01-15T18:12:01.022574+0000","last_became_peered":"2024-01-15T18:12:01.022574+0000","last_unstale":"2024-01-15T18:12:02.757369+0000","last_undegraded":"2024-01-15T18:12:02.757369+0000","last_fullsized":"2024-01-15T18:12:02.757369+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:17:25.624685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"69'1","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.043094+0000","last_change":"2024-01-15T18:12:00.019039+0000","last_active":"2024-01-15T18:12:01.043094+0000","last_peered":"2024-01-15T18:12:01.043094+0000","last_clean":"2024-01-15T18:12:01.043094+0000","last_became_active":"2024-01-15T18:12:00.018741+0000","last_became_peered":"2024-01-15T18:12:00.018741+0000","last_unstale":"2024-01-15T18:12:01.043094+0000","last_undegraded":"2024-01-15T18:12:01.043094+0000","last_fullsized":"2024-01-15T18:12:01.043094+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:40:14.291921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756941+0000","last_change":"2024-01-15T18:12:01.034613+0000","last_active":"2024-01-15T18:12:02.756941+0000","last_peered":"2024-01-15T18:12:02.756941+0000","last_clean":"2024-01-15T18:12:02.756941+0000","last_became_active":"2024-01-15T18:12:01.034323+0000","last_became_peered":"2024-01-15T18:12:01.034323+0000","last_unstale":"2024-01-15T18:12:02.756941+0000","last_undegraded":"2024-01-15T18:12:02.756941+0000","last_fullsized":"2024-01-15T18:12:02.756941+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:41:38.814257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.042539+0000","last_change":"2024-01-15T18:12:00.022509+0000","last_active":"2024-01-15T18:12:01.042539+0000","last_peered":"2024-01-15T18:12:01.042539+0000","last_clean":"2024-01-15T18:12:01.042539+0000","last_became_active":"2024-01-15T18:12:00.021725+0000","last_became_peered":"2024-01-15T18:12:00.021725+0000","last_unstale":"2024-01-15T18:12:01.042539+0000","last_undegraded":"2024-01-15T18:12:01.042539+0000","last_fullsized":"2024-01-15T18:12:01.042539+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:07:42.595399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756509+0000","last_change":"2024-01-15T18:12:01.017630+0000","last_active":"2024-01-15T18:12:02.756509+0000","last_peered":"2024-01-15T18:12:02.756509+0000","last_clean":"2024-01-15T18:12:02.756509+0000","last_became_active":"2024-01-15T18:12:01.016364+0000","last_became_peered":"2024-01-15T18:12:01.016364+0000","last_unstale":"2024-01-15T18:12:02.756509+0000","last_undegraded":"2024-01-15T18:12:02.756509+0000","last_fullsized":"2024-01-15T18:12:02.756509+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:37:54.171968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"69'1","reported_seq":13,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.043471+0000","last_change":"2024-01-15T18:12:00.022642+0000","last_active":"2024-01-15T18:12:01.043471+0000","last_peered":"2024-01-15T18:12:01.043471+0000","last_clean":"2024-01-15T18:12:01.043471+0000","last_became_active":"2024-01-15T18:12:00.021842+0000","last_became_peered":"2024-01-15T18:12:00.021842+0000","last_unstale":"2024-01-15T18:12:01.043471+0000","last_undegraded":"2024-01-15T18:12:01.043471+0000","last_fullsized":"2024-01-15T18:12:01.043471+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:38:51.921542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.007215+0000","last_change":"2024-01-15T18:12:01.013299+0000","last_active":"2024-01-15T18:12:02.007215+0000","last_peered":"2024-01-15T18:12:02.007215+0000","last_clean":"2024-01-15T18:12:02.007215+0000","last_became_active":"2024-01-15T18:12:01.012799+0000","last_became_peered":"2024-01-15T18:12:01.012799+0000","last_unstale":"2024-01-15T18:12:02.007215+0000","last_undegraded":"2024-01-15T18:12:02.007215+0000","last_fullsized":"2024-01-15T18:12:02.007215+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:27:06.509091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.014213+0000","last_change":"2024-01-15T18:12:00.027374+0000","last_active":"2024-01-15T18:12:01.014213+0000","last_peered":"2024-01-15T18:12:01.014213+0000","last_clean":"2024-01-15T18:12:01.014213+0000","last_became_active":"2024-01-15T18:12:00.027064+0000","last_became_peered":"2024-01-15T18:12:00.027064+0000","last_unstale":"2024-01-15T18:12:01.014213+0000","last_undegraded":"2024-01-15T18:12:01.014213+0000","last_fullsized":"2024-01-15T18:12:01.014213+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:17:24.621398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756841+0000","last_change":"2024-01-15T18:12:01.025265+0000","last_active":"2024-01-15T18:12:02.756841+0000","last_peered":"2024-01-15T18:12:02.756841+0000","last_clean":"2024-01-15T18:12:02.756841+0000","last_became_active":"2024-01-15T18:12:01.024338+0000","last_became_peered":"2024-01-15T18:12:01.024338+0000","last_unstale":"2024-01-15T18:12:02.756841+0000","last_undegraded":"2024-01-15T18:12:02.756841+0000","last_fullsized":"2024-01-15T18:12:02.756841+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:30:16.972579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.007589+0000","last_change":"2024-01-15T18:12:00.023536+0000","last_active":"2024-01-15T18:12:01.007589+0000","last_peered":"2024-01-15T18:12:01.007589+0000","last_clean":"2024-01-15T18:12:01.007589+0000","last_became_active":"2024-01-15T18:12:00.023249+0000","last_became_peered":"2024-01-15T18:12:00.023249+0000","last_unstale":"2024-01-15T18:12:01.007589+0000","last_undegraded":"2024-01-15T18:12:01.007589+0000","last_fullsized":"2024-01-15T18:12:01.007589+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:55.181455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756009+0000","last_change":"2024-01-15T18:12:01.025190+0000","last_active":"2024-01-15T18:12:02.756009+0000","last_peered":"2024-01-15T18:12:02.756009+0000","last_clean":"2024-01-15T18:12:02.756009+0000","last_became_active":"2024-01-15T18:12:01.024888+0000","last_became_peered":"2024-01-15T18:12:01.024888+0000","last_unstale":"2024-01-15T18:12:02.756009+0000","last_undegraded":"2024-01-15T18:12:02.756009+0000","last_fullsized":"2024-01-15T18:12:02.756009+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:25:17.902934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.017179+0000","last_change":"2024-01-15T18:12:00.020946+0000","last_active":"2024-01-15T18:12:01.017179+0000","last_peered":"2024-01-15T18:12:01.017179+0000","last_clean":"2024-01-15T18:12:01.017179+0000","last_became_active":"2024-01-15T18:12:00.020641+0000","last_became_peered":"2024-01-15T18:12:00.020641+0000","last_unstale":"2024-01-15T18:12:01.017179+0000","last_undegraded":"2024-01-15T18:12:01.017179+0000","last_fullsized":"2024-01-15T18:12:01.017179+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:14:51.219226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756518+0000","last_change":"2024-01-15T18:12:01.027693+0000","last_active":"2024-01-15T18:12:02.756518+0000","last_peered":"2024-01-15T18:12:02.756518+0000","last_clean":"2024-01-15T18:12:02.756518+0000","last_became_active":"2024-01-15T18:12:01.025511+0000","last_became_peered":"2024-01-15T18:12:01.025511+0000","last_unstale":"2024-01-15T18:12:02.756518+0000","last_undegraded":"2024-01-15T18:12:02.756518+0000","last_fullsized":"2024-01-15T18:12:02.756518+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:36:21.428758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.012850+0000","last_change":"2024-01-15T18:12:00.018802+0000","last_active":"2024-01-15T18:12:01.012850+0000","last_peered":"2024-01-15T18:12:01.012850+0000","last_clean":"2024-01-15T18:12:01.012850+0000","last_became_active":"2024-01-15T18:12:00.017356+0000","last_became_peered":"2024-01-15T18:12:00.017356+0000","last_unstale":"2024-01-15T18:12:01.012850+0000","last_undegraded":"2024-01-15T18:12:01.012850+0000","last_fullsized":"2024-01-15T18:12:01.012850+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:00:57.752979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.006344+0000","last_change":"2024-01-15T18:12:01.028988+0000","last_active":"2024-01-15T18:12:02.006344+0000","last_peered":"2024-01-15T18:12:02.006344+0000","last_clean":"2024-01-15T18:12:02.006344+0000","last_became_active":"2024-01-15T18:12:01.028024+0000","last_became_peered":"2024-01-15T18:12:01.028024+0000","last_unstale":"2024-01-15T18:12:02.006344+0000","last_undegraded":"2024-01-15T18:12:02.006344+0000","last_fullsized":"2024-01-15T18:12:02.006344+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:32:20.609600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.042991+0000","last_change":"2024-01-15T18:12:00.016456+0000","last_active":"2024-01-15T18:12:01.042991+0000","last_peered":"2024-01-15T18:12:01.042991+0000","last_clean":"2024-01-15T18:12:01.042991+0000","last_became_active":"2024-01-15T18:12:00.016145+0000","last_became_peered":"2024-01-15T18:12:00.016145+0000","last_unstale":"2024-01-15T18:12:01.042991+0000","last_undegraded":"2024-01-15T18:12:01.042991+0000","last_fullsized":"2024-01-15T18:12:01.042991+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:13:56.868830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.006010+0000","last_change":"2024-01-15T18:12:01.024249+0000","last_active":"2024-01-15T18:12:02.006010+0000","last_peered":"2024-01-15T18:12:02.006010+0000","last_clean":"2024-01-15T18:12:02.006010+0000","last_became_active":"2024-01-15T18:12:01.023862+0000","last_became_peered":"2024-01-15T18:12:01.023862+0000","last_unstale":"2024-01-15T18:12:02.006010+0000","last_undegraded":"2024-01-15T18:12:02.006010+0000","last_fullsized":"2024-01-15T18:12:02.006010+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:18:47.967166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.016071+0000","last_change":"2024-01-15T18:12:00.023511+0000","last_active":"2024-01-15T18:12:01.016071+0000","last_peered":"2024-01-15T18:12:01.016071+0000","last_clean":"2024-01-15T18:12:01.016071+0000","last_became_active":"2024-01-15T18:12:00.023145+0000","last_became_peered":"2024-01-15T18:12:00.023145+0000","last_unstale":"2024-01-15T18:12:01.016071+0000","last_undegraded":"2024-01-15T18:12:01.016071+0000","last_fullsized":"2024-01-15T18:12:01.016071+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:27:30.560351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756286+0000","last_change":"2024-01-15T18:12:01.026130+0000","last_active":"2024-01-15T18:12:02.756286+0000","last_peered":"2024-01-15T18:12:02.756286+0000","last_clean":"2024-01-15T18:12:02.756286+0000","last_became_active":"2024-01-15T18:12:01.025861+0000","last_became_peered":"2024-01-15T18:12:01.025861+0000","last_unstale":"2024-01-15T18:12:02.756286+0000","last_undegraded":"2024-01-15T18:12:02.756286+0000","last_fullsized":"2024-01-15T18:12:02.756286+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:57.993053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.006728+0000","last_change":"2024-01-15T18:12:00.023014+0000","last_active":"2024-01-15T18:12:01.006728+0000","last_peered":"2024-01-15T18:12:01.006728+0000","last_clean":"2024-01-15T18:12:01.006728+0000","last_became_active":"2024-01-15T18:12:00.022748+0000","last_became_peered":"2024-01-15T18:12:00.022748+0000","last_unstale":"2024-01-15T18:12:01.006728+0000","last_undegraded":"2024-01-15T18:12:01.006728+0000","last_fullsized":"2024-01-15T18:12:01.006728+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:05:09.561750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756754+0000","last_change":"2024-01-15T18:12:01.034310+0000","last_active":"2024-01-15T18:12:02.756754+0000","last_peered":"2024-01-15T18:12:02.756754+0000","last_clean":"2024-01-15T18:12:02.756754+0000","last_became_active":"2024-01-15T18:12:01.034041+0000","last_became_peered":"2024-01-15T18:12:01.034041+0000","last_unstale":"2024-01-15T18:12:02.756754+0000","last_undegraded":"2024-01-15T18:12:02.756754+0000","last_fullsized":"2024-01-15T18:12:02.756754+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:45:12.201484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.012619+0000","last_change":"2024-01-15T18:12:00.024610+0000","last_active":"2024-01-15T18:12:01.012619+0000","last_peered":"2024-01-15T18:12:01.012619+0000","last_clean":"2024-01-15T18:12:01.012619+0000","last_became_active":"2024-01-15T18:12:00.024272+0000","last_became_peered":"2024-01-15T18:12:00.024272+0000","last_unstale":"2024-01-15T18:12:01.012619+0000","last_undegraded":"2024-01-15T18:12:01.012619+0000","last_fullsized":"2024-01-15T18:12:01.012619+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:20:15.640627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.007502+0000","last_change":"2024-01-15T18:12:01.028311+0000","last_active":"2024-01-15T18:12:02.007502+0000","last_peered":"2024-01-15T18:12:02.007502+0000","last_clean":"2024-01-15T18:12:02.007502+0000","last_became_active":"2024-01-15T18:12:01.027889+0000","last_became_peered":"2024-01-15T18:12:01.027889+0000","last_unstale":"2024-01-15T18:12:02.007502+0000","last_undegraded":"2024-01-15T18:12:02.007502+0000","last_fullsized":"2024-01-15T18:12:02.007502+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:19:20.426156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.005931+0000","last_change":"2024-01-15T18:12:00.019406+0000","last_active":"2024-01-15T18:12:01.005931+0000","last_peered":"2024-01-15T18:12:01.005931+0000","last_clean":"2024-01-15T18:12:01.005931+0000","last_became_active":"2024-01-15T18:12:00.019122+0000","last_became_peered":"2024-01-15T18:12:00.019122+0000","last_unstale":"2024-01-15T18:12:01.005931+0000","last_undegraded":"2024-01-15T18:12:01.005931+0000","last_fullsized":"2024-01-15T18:12:01.005931+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:50:34.976203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756988+0000","last_change":"2024-01-15T18:12:01.026385+0000","last_active":"2024-01-15T18:12:02.756988+0000","last_peered":"2024-01-15T18:12:02.756988+0000","last_clean":"2024-01-15T18:12:02.756988+0000","last_became_active":"2024-01-15T18:12:01.026085+0000","last_became_peered":"2024-01-15T18:12:01.026085+0000","last_unstale":"2024-01-15T18:12:02.756988+0000","last_undegraded":"2024-01-15T18:12:02.756988+0000","last_fullsized":"2024-01-15T18:12:02.756988+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:36:37.178674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.013295+0000","last_change":"2024-01-15T18:12:00.015364+0000","last_active":"2024-01-15T18:12:01.013295+0000","last_peered":"2024-01-15T18:12:01.013295+0000","last_clean":"2024-01-15T18:12:01.013295+0000","last_became_active":"2024-01-15T18:12:00.014208+0000","last_became_peered":"2024-01-15T18:12:00.014208+0000","last_unstale":"2024-01-15T18:12:01.013295+0000","last_undegraded":"2024-01-15T18:12:01.013295+0000","last_fullsized":"2024-01-15T18:12:01.013295+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:46:51.225073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756081+0000","last_change":"2024-01-15T18:12:01.032004+0000","last_active":"2024-01-15T18:12:02.756081+0000","last_peered":"2024-01-15T18:12:02.756081+0000","last_clean":"2024-01-15T18:12:02.756081+0000","last_became_active":"2024-01-15T18:12:01.031719+0000","last_became_peered":"2024-01-15T18:12:01.031719+0000","last_unstale":"2024-01-15T18:12:02.756081+0000","last_undegraded":"2024-01-15T18:12:02.756081+0000","last_fullsized":"2024-01-15T18:12:02.756081+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:26.997082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.006423+0000","last_change":"2024-01-15T18:12:00.025623+0000","last_active":"2024-01-15T18:12:01.006423+0000","last_peered":"2024-01-15T18:12:01.006423+0000","last_clean":"2024-01-15T18:12:01.006423+0000","last_became_active":"2024-01-15T18:12:00.025294+0000","last_became_peered":"2024-01-15T18:12:00.025294+0000","last_unstale":"2024-01-15T18:12:01.006423+0000","last_undegraded":"2024-01-15T18:12:01.006423+0000","last_fullsized":"2024-01-15T18:12:01.006423+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:05.587795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.755841+0000","last_change":"2024-01-15T18:12:01.030920+0000","last_active":"2024-01-15T18:12:02.755841+0000","last_peered":"2024-01-15T18:12:02.755841+0000","last_clean":"2024-01-15T18:12:02.755841+0000","last_became_active":"2024-01-15T18:12:01.030565+0000","last_became_peered":"2024-01-15T18:12:01.030565+0000","last_unstale":"2024-01-15T18:12:02.755841+0000","last_undegraded":"2024-01-15T18:12:02.755841+0000","last_fullsized":"2024-01-15T18:12:02.755841+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:26.997082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.007385+0000","last_change":"2024-01-15T18:12:00.019753+0000","last_active":"2024-01-15T18:12:01.007385+0000","last_peered":"2024-01-15T18:12:01.007385+0000","last_clean":"2024-01-15T18:12:01.007385+0000","last_became_active":"2024-01-15T18:12:00.019482+0000","last_became_peered":"2024-01-15T18:12:00.019482+0000","last_unstale":"2024-01-15T18:12:01.007385+0000","last_undegraded":"2024-01-15T18:12:01.007385+0000","last_fullsized":"2024-01-15T18:12:01.007385+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:09:59.518166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":11,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-15T18:12:02.756765+0000","last_change":"2024-01-15T18:12:01.026201+0000","last_active":"2024-01-15T18:12:02.756765+0000","last_peered":"2024-01-15T18:12:02.756765+0000","last_clean":"2024-01-15T18:12:02.756765+0000","last_became_active":"2024-01-15T18:12:01.025920+0000","last_became_peered":"2024-01-15T18:12:01.025920+0000","last_unstale":"2024-01-15T18:12:02.756765+0000","last_undegraded":"2024-01-15T18:12:02.756765+0000","last_fullsized":"2024-01-15T18:12:02.756765+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:59.986005+0000","last_clean_scrub_stamp":"2024-01-15T18:11:59.986005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:41:38.814257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"69'2","reported_seq":14,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.043831+0000","last_change":"2024-01-15T18:12:00.019108+0000","last_active":"2024-01-15T18:12:01.043831+0000","last_peered":"2024-01-15T18:12:01.043831+0000","last_clean":"2024-01-15T18:12:01.043831+0000","last_became_active":"2024-01-15T18:12:00.018827+0000","last_became_peered":"2024-01-15T18:12:00.018827+0000","last_unstale":"2024-01-15T18:12:01.043831+0000","last_undegraded":"2024-01-15T18:12:01.043831+0000","last_fullsized":"2024-01-15T18:12:01.043831+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:11:58.982718+0000","last_clean_scrub_stamp":"2024-01-15T18:11:58.982718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:17:27.915008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":205,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-01-15T18:12:01.019972+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:12:01.019972+0000","last_peered":"2024-01-15T18:12:01.019972+0000","last_clean":"2024-01-15T18:12:01.019972+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:12:01.019972+0000","last_undegraded":"2024-01-15T18:12:01.019972+0000","last_fullsized":"2024-01-15T18:12:01.019972+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509531,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50640,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321200,"statfs":{"total":96636764160,"available":96584908800,"internally_reserved":0,"allocated":1445888,"data_stored":948856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.1200000000000001,"15min":0.78900000000000003},"min":{"1min":0.66000000000000003,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":9.6989999999999998,"5min":10.284000000000001,"15min":10.284000000000001},"last":0.82599999999999996},{"interface":"front","average":{"1min":2.4159999999999999,"5min":1.2290000000000001,"15min":0.86299999999999999},"min":{"1min":0.39100000000000001,"5min":0.378,"15min":0.378},"max":{"1min":16.968,"5min":16.968,"15min":16.968},"last":0.76400000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.7749999999999999,"5min":1.1830000000000001,"15min":0.83899999999999997},"min":{"1min":0.64800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":15.791,"5min":15.791,"15min":15.791},"last":0.86699999999999999},{"interface":"front","average":{"1min":2.1339999999999999,"5min":1.1779999999999999,"15min":0.85899999999999999},"min":{"1min":0.44600000000000001,"5min":0.378,"15min":0.378},"max":{"1min":10.842000000000001,"5min":10.842000000000001,"15min":10.842000000000001},"last":0.84499999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1,"15min":0.76400000000000001},"min":{"1min":0.42299999999999999,"5min":0.378,"15min":0.378},"max":{"1min":4.9029999999999996,"5min":10.119999999999999,"15min":10.119999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":2.0579999999999998,"5min":1.081,"15min":0.83499999999999996},"min":{"1min":0.66800000000000004,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":9.7539999999999996,"5min":9.7539999999999996,"15min":9.7539999999999996},"last":0.78300000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":1.802,"5min":1.143,"15min":0.82299999999999995},"min":{"1min":0.47099999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":11.481,"5min":11.481,"15min":11.481},"last":0.80400000000000005},{"interface":"front","average":{"1min":2.3220000000000001,"5min":1.2310000000000001,"15min":0.91900000000000004},"min":{"1min":0.70199999999999996,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":9.7959999999999994,"5min":10.34,"15min":10.34},"last":0.625}]},{"osd":4,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.5630000000000002,"5min":1.3360000000000001,"15min":0.85799999999999998},"min":{"1min":0.41099999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":20.387,"5min":20.387,"15min":20.387},"last":0.56999999999999995},{"interface":"front","average":{"1min":1.649,"5min":1.131,"15min":0.81999999999999995},"min":{"1min":0.46200000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":11.305,"5min":14.871,"15min":14.871},"last":0.59899999999999998}]},{"osd":5,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":2.2719999999999998,"5min":1.2070000000000001,"15min":0.86899999999999999},"min":{"1min":0.47799999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":9.8450000000000006,"5min":19.959,"15min":19.959},"last":0.54300000000000004},{"interface":"front","average":{"1min":2.6240000000000001,"5min":1.3009999999999999,"15min":0.92000000000000004},"min":{"1min":0.48299999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":19.905000000000001,"5min":19.905000000000001,"15min":19.905000000000001},"last":0.69399999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:11:06 2024","interfaces":[{"interface":"back","average":{"1min":3.157,"5min":1.667,"15min":1.093},"min":{"1min":0.63800000000000001,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":20.492000000000001,"5min":22.478999999999999,"15min":22.478999999999999},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.397,"5min":1.3049999999999999,"15min":0.92500000000000004},"min":{"1min":0.45100000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.2549999999999999,"5min":21.509,"15min":21.509},"last":0.65500000000000003}]}]},{"osd":6,"up_from":15,"seq":64424509531,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45904,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325936,"statfs":{"total":96636764160,"available":96589758464,"internally_reserved":0,"allocated":851968,"data_stored":357964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":1.157,"15min":0.872},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.91800000000000004,"5min":12.41,"15min":12.41},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":1.0009999999999999,"15min":0.79000000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.93500000000000005,"5min":6.7759999999999998,"15min":6.7759999999999998},"last":0.747}]},{"osd":1,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":1.454,"15min":0.96399999999999997},"min":{"1min":0.46999999999999997,"5min":0.41899999999999998,"15min":0.41299999999999998},"max":{"1min":0.83199999999999996,"5min":53.451999999999998,"15min":53.451999999999998},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.71299999999999997,"5min":1.008,"15min":0.83699999999999997},"min":{"1min":0.56100000000000005,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.92900000000000005,"5min":7.7329999999999997,"15min":7.7329999999999997},"last":0.72399999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":1.004,"15min":0.89400000000000002},"min":{"1min":0.46700000000000003,"5min":0.45900000000000002,"15min":0.38900000000000001},"max":{"1min":1.1060000000000001,"5min":7.9409999999999998,"15min":7.9409999999999998},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.93700000000000006,"15min":0.82999999999999996},"min":{"1min":0.57099999999999995,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.90800000000000003,"5min":5.1760000000000002,"15min":5.1760000000000002},"last":0.60599999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":1.038,"15min":0.876},"min":{"1min":0.46000000000000002,"5min":0.42099999999999999,"15min":0.41399999999999998},"max":{"1min":0.94899999999999995,"5min":6.758,"15min":6.758},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.70499999999999996,"5min":1.5600000000000001,"15min":1.0660000000000001},"min":{"1min":0.49199999999999999,"5min":0.45200000000000001,"15min":0.41599999999999998},"max":{"1min":0.877,"5min":53.398000000000003,"15min":53.398000000000003},"last":0.63200000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":1.0029999999999999,"15min":0.79500000000000004},"min":{"1min":0.38500000000000001,"5min":0.378,"15min":0.34999999999999998},"max":{"1min":0.88,"5min":8.6259999999999994,"15min":8.6259999999999994},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.94899999999999995,"15min":0.79100000000000004},"min":{"1min":0.42599999999999999,"5min":0.38200000000000001,"15min":0.34499999999999997},"max":{"1min":0.85099999999999998,"5min":6.9589999999999996,"15min":6.9589999999999996},"last":0.65900000000000003}]},{"osd":5,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":1.641,"15min":0.98299999999999998},"min":{"1min":0.41299999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.96399999999999997,"5min":53.229999999999997,"15min":53.229999999999997},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":1.228,"15min":0.92900000000000005},"min":{"1min":0.45500000000000002,"5min":0.435,"15min":0.435},"max":{"1min":1.149,"5min":10.029999999999999,"15min":10.029999999999999},"last":0.80800000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":1.5760000000000001,"15min":1.071},"min":{"1min":0.437,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.90400000000000003,"5min":45.070999999999998,"15min":45.070999999999998},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.70099999999999996,"5min":1.6910000000000001,"15min":1.0940000000000001},"min":{"1min":0.52800000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.92200000000000004,"5min":53.338000000000001,"15min":53.338000000000001},"last":0.70699999999999996}]}]},{"osd":4,"up_from":15,"seq":64424509531,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50056,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321784,"statfs":{"total":96636764160,"available":96585506816,"internally_reserved":0,"allocated":847872,"data_stored":357942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":1.2,"15min":0.81899999999999995},"min":{"1min":0.502,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.2509999999999999,"5min":23.318000000000001,"15min":23.318000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":1.046,"15min":0.77500000000000002},"min":{"1min":0.442,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.97899999999999998,"5min":14.723000000000001,"15min":14.723000000000001},"last":0.84599999999999997}]},{"osd":1,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.89500000000000002,"15min":0.72999999999999998},"min":{"1min":0.442,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.876,"5min":7.8200000000000003,"15min":7.8200000000000003},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.73899999999999999,"5min":1.1879999999999999,"15min":0.82299999999999995},"min":{"1min":0.47299999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.613,"5min":22.408999999999999,"15min":22.408999999999999},"last":0.751}]},{"osd":2,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.91800000000000004,"15min":0.72399999999999998},"min":{"1min":0.48699999999999999,"5min":0.40600000000000003,"15min":0.34399999999999997},"max":{"1min":1.0009999999999999,"5min":8.8529999999999998,"15min":8.8529999999999998},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.77900000000000003,"5min":1.288,"15min":0.89200000000000002},"min":{"1min":0.55500000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.2969999999999999,"5min":22.338999999999999,"15min":22.338999999999999},"last":0.67100000000000004}]},{"osd":3,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":1.149,"15min":0.84299999999999997},"min":{"1min":0.52500000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.3939999999999999,"5min":14.763999999999999,"15min":14.763999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.0509999999999999,"15min":0.83399999999999996},"min":{"1min":0.59899999999999998,"5min":0.504,"15min":0.45500000000000002},"max":{"1min":1.7669999999999999,"5min":8.9459999999999997,"15min":8.9459999999999997},"last":0.86799999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":1.2609999999999999,"15min":0.86399999999999999},"min":{"1min":0.39800000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.4670000000000001,"5min":31.617999999999999,"15min":31.617999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":1.1910000000000001,"15min":0.83099999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.377},"max":{"1min":1.022,"5min":17.145,"15min":17.145},"last":0.79900000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.95299999999999996,"15min":0.77100000000000002},"min":{"1min":0.46500000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.89300000000000002,"5min":11.124000000000001,"15min":11.124000000000001},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.749,"5min":1.7629999999999999,"15min":1.0780000000000001},"min":{"1min":0.45600000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.339,"5min":28.872,"15min":28.872},"last":0.82299999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":1.5409999999999999,"15min":0.99299999999999999},"min":{"1min":0.45800000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.3169999999999999,"5min":39.256,"15min":39.256},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.70799999999999996,"5min":1.1160000000000001,"15min":0.89200000000000002},"min":{"1min":0.51900000000000002,"5min":0.46500000000000002,"15min":0.42199999999999999},"max":{"1min":1.0329999999999999,"5min":26.573,"15min":26.573},"last":0.94399999999999995}]}]},{"osd":3,"up_from":15,"seq":64424509530,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50060,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321780,"statfs":{"total":96636764160,"available":96585502720,"internally_reserved":0,"allocated":851968,"data_stored":357984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.58899999999999997,"15min":0.47999999999999998},"min":{"1min":0.26400000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67700000000000005,"5min":5.8849999999999998,"15min":5.8849999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.80900000000000005,"15min":0.59299999999999997},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.96999999999999997,"5min":17.756,"15min":17.756},"last":0.39600000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.75,"15min":0.58999999999999997},"min":{"1min":0.29499999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.68999999999999995,"5min":12.307,"15min":12.307},"last":0.44},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.66400000000000003,"15min":0.57899999999999996},"min":{"1min":0.35199999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.72099999999999997,"5min":6.0640000000000001,"15min":6.0640000000000001},"last":0.56299999999999994}]},{"osd":2,"last update":"Mon Jan 15 18:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.85799999999999998,"15min":0.65900000000000003},"min":{"1min":0.44700000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.89600000000000002,"5min":12.468999999999999,"15min":12.468999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.76200000000000001,"15min":0.63700000000000001},"min":{"1min":0.442,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79100000000000004,"5min":6.0330000000000004,"15min":6.0330000000000004},"last":0.53200000000000003}]},{"osd":4,"last update":"Mon Jan 15 18:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.1859999999999999,"15min":0.92900000000000005},"min":{"1min":0.53300000000000003,"5min":0.375,"15min":0.375},"max":{"1min":4.1239999999999997,"5min":17.853999999999999,"15min":17.853999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.1639999999999999,"15min":0.90600000000000003},"min":{"1min":0.61799999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":4.0960000000000001,"5min":13.31,"15min":13.31},"last":0.63600000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":1.0289999999999999,"15min":0.83699999999999997},"min":{"1min":0.58199999999999996,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.6419999999999999,"5min":6.1139999999999999,"15min":6.1139999999999999},"last":0.753},{"interface":"front","average":{"1min":0.82499999999999996,"5min":1.302,"15min":0.91500000000000004},"min":{"1min":0.59899999999999998,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.458,"5min":20.515999999999998,"15min":20.515999999999998},"last":0.83699999999999997}]},{"osd":6,"last update":"Mon Jan 15 18:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":1.345,"15min":0.97099999999999997},"min":{"1min":0.61499999999999999,"5min":0.432,"15min":0.432},"max":{"1min":1.845,"5min":23.100999999999999,"15min":23.100999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.82699999999999996,"5min":1.133,"15min":0.89000000000000001},"min":{"1min":0.624,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.133,"5min":12.76,"15min":12.76},"last":0.59999999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":1.379,"15min":0.98099999999999998},"min":{"1min":0.53800000000000003,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":2.016,"5min":21.545000000000002,"15min":21.545000000000002},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.94599999999999995,"5min":1.3560000000000001,"15min":0.997},"min":{"1min":0.63800000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":3.2229999999999999,"5min":13.385,"15min":13.385},"last":0.82199999999999995}]}]},{"osd":2,"up_from":15,"seq":64424509530,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45904,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325936,"statfs":{"total":96636764160,"available":96589758464,"internally_reserved":0,"allocated":856064,"data_stored":359484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.61699999999999999,"15min":0.51200000000000001},"min":{"1min":0.29399999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.8480000000000001,"5min":5.1849999999999996,"15min":5.1849999999999996},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.64200000000000002,"15min":0.52300000000000002},"min":{"1min":0.32600000000000001,"5min":0.29399999999999998,"15min":0.28299999999999997},"max":{"1min":2.3759999999999999,"5min":7.516,"15min":7.516},"last":0.40200000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.748,"15min":0.61199999999999999},"min":{"1min":0.34100000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":2.2850000000000001,"5min":10.907,"15min":10.907},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.81599999999999995,"15min":0.57399999999999995},"min":{"1min":0.40500000000000003,"5min":0.311,"15min":0.27000000000000002},"max":{"1min":1.7150000000000001,"5min":8.9190000000000005,"15min":8.9190000000000005},"last":0.54600000000000004}]},{"osd":3,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.73599999999999999,"15min":0.628},"min":{"1min":0.36599999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.86399999999999999,"5min":12.185,"15min":12.185},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.83499999999999996,"15min":0.64700000000000002},"min":{"1min":0.443,"5min":0.39100000000000001,"15min":0.34000000000000002},"max":{"1min":2.3330000000000002,"5min":7.6760000000000002,"15min":7.6760000000000002},"last":0.622}]},{"osd":4,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.99299999999999999,"15min":0.82699999999999996},"min":{"1min":0.58699999999999997,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":2.21,"5min":8.6609999999999996,"15min":8.6609999999999996},"last":1.018},{"interface":"front","average":{"1min":0.76000000000000001,"5min":1.1899999999999999,"15min":0.90600000000000003},"min":{"1min":0.54000000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.6339999999999999,"5min":19.146999999999998,"15min":19.146999999999998},"last":1.0649999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":1.1319999999999999,"15min":0.90800000000000003},"min":{"1min":0.442,"5min":0.44,"15min":0.44},"max":{"1min":0.90800000000000003,"5min":31.710000000000001,"15min":31.710000000000001},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.875,"5min":1.373,"15min":0.98299999999999998},"min":{"1min":0.59399999999999997,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":2.1669999999999998,"5min":22.539000000000001,"15min":22.539000000000001},"last":1.0940000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":1.107,"15min":0.88600000000000001},"min":{"1min":0.61699999999999999,"5min":0.50800000000000001,"15min":0.48299999999999998},"max":{"1min":1.675,"5min":9.4540000000000006,"15min":9.4540000000000006},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.1399999999999999,"15min":0.90900000000000003},"min":{"1min":0.53100000000000003,"5min":0.503,"15min":0.503},"max":{"1min":1,"5min":18.614999999999998,"15min":18.614999999999998},"last":0.97399999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:11:53 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":1.113,"15min":0.90400000000000003},"min":{"1min":0.58199999999999996,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.752,"5min":11.432,"15min":11.432},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.264,"15min":0.97099999999999997},"min":{"1min":0.54500000000000004,"5min":0.502,"15min":0.502},"max":{"1min":2.2480000000000002,"5min":17.013000000000002,"15min":17.013000000000002},"last":0.94499999999999995}]}]},{"osd":1,"up_from":15,"seq":64424509532,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50072,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321768,"statfs":{"total":96636764160,"available":96585490432,"internally_reserved":0,"allocated":864256,"data_stored":360100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.96399999999999997,"15min":0.65500000000000003},"min":{"1min":0.309,"5min":0.26500000000000001,"15min":0.254},"max":{"1min":0.72899999999999998,"5min":19.577000000000002,"15min":19.577000000000002},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":1.1779999999999999,"15min":0.71499999999999997},"min":{"1min":0.35399999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.80000000000000004,"5min":48.277999999999999,"15min":48.277999999999999},"last":0.56999999999999995}]},{"osd":2,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":1.036,"15min":0.70599999999999996},"min":{"1min":0.35599999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.65000000000000002,"5min":40.698,"15min":40.698},"last":0.504},{"interface":"front","average":{"1min":0.53200000000000003,"5min":1.337,"15min":0.92900000000000005},"min":{"1min":0.36899999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93300000000000005,"5min":48.316000000000003,"15min":48.316000000000003},"last":0.48399999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.2370000000000001,"15min":0.83999999999999997},"min":{"1min":0.46600000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.88200000000000001,"5min":48.234999999999999,"15min":48.234999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.75800000000000001,"15min":0.64500000000000002},"min":{"1min":0.36099999999999999,"5min":0.34699999999999998,"15min":0.311},"max":{"1min":0.74399999999999999,"5min":6.468,"15min":6.468},"last":0.54700000000000004}]},{"osd":4,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":1.0589999999999999,"15min":0.81799999999999995},"min":{"1min":0.48999999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.0720000000000001,"5min":8.3409999999999993,"15min":8.3409999999999993},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.80200000000000005,"5min":1.165,"15min":0.90600000000000003},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.42499999999999999},"max":{"1min":1.278,"5min":19.829000000000001,"15min":19.829000000000001},"last":0.74199999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":1.4399999999999999,"15min":0.99399999999999999},"min":{"1min":0.59499999999999997,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.306,"5min":22.594999999999999,"15min":22.594999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.79400000000000004,"5min":1.8939999999999999,"15min":1.145},"min":{"1min":0.54500000000000004,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1180000000000001,"5min":48.198,"15min":48.198},"last":0.69499999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":1.4590000000000001,"15min":0.98899999999999999},"min":{"1min":0.59899999999999998,"5min":0.53000000000000003,"15min":0.47499999999999998},"max":{"1min":1.1499999999999999,"5min":35.112000000000002,"15min":35.112000000000002},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.81200000000000006,"5min":1.1779999999999999,"15min":0.93999999999999995},"min":{"1min":0.53100000000000003,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.175,"5min":19.780999999999999,"15min":19.780999999999999},"last":0.86099999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":1.8069999999999999,"15min":1.1259999999999999},"min":{"1min":0.628,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.2450000000000001,"5min":48.148000000000003,"15min":48.148000000000003},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.80300000000000005,"5min":1.194,"15min":0.93300000000000005},"min":{"1min":0.64000000000000001,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.214,"5min":9.3840000000000003,"15min":9.3840000000000003},"last":0.76000000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542235,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50064,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321776,"statfs":{"total":96636764160,"available":96585498624,"internally_reserved":0,"allocated":847872,"data_stored":357962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.872,"15min":0.77600000000000002},"min":{"1min":0.45900000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.3019999999999996,"5min":21.385000000000002,"15min":21.385000000000002},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.732,"5min":1.071,"15min":0.88800000000000001},"min":{"1min":0.46999999999999997,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":9.359,"5min":9.359,"15min":9.359},"last":0.78200000000000003}]},{"osd":1,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":3.0960000000000001,"5min":1.1839999999999999,"15min":0.84599999999999997},"min":{"1min":0.48799999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":21.120000000000001,"5min":21.120000000000001,"15min":21.120000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0329999999999999,"15min":0.80100000000000005},"min":{"1min":0.52400000000000002,"5min":0.315,"15min":0.315},"max":{"1min":4.3730000000000002,"5min":21.5,"15min":21.5},"last":0.753}]},{"osd":2,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.1419999999999999,"15min":0.85399999999999998},"min":{"1min":0.56399999999999995,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":6.2750000000000004,"5min":8.7750000000000004,"15min":8.7750000000000004},"last":0.73799999999999999},{"interface":"front","average":{"1min":2.9780000000000002,"5min":1.2869999999999999,"15min":0.94999999999999996},"min":{"1min":0.58099999999999996,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":21.201000000000001,"5min":21.201000000000001,"15min":21.201000000000001},"last":0.76600000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.0069999999999999,"15min":0.875},"min":{"1min":0.57399999999999995,"5min":0.44,"15min":0.44},"max":{"1min":4.4100000000000001,"5min":21.457000000000001,"15min":21.457000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.583,"5min":1.196,"15min":0.96799999999999997},"min":{"1min":0.54200000000000004,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":6.8650000000000002,"5min":8.7300000000000004,"15min":8.7300000000000004},"last":0.64900000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.012,"15min":0.82799999999999996},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":5.8929999999999998,"5min":9.6679999999999993,"15min":9.6679999999999993},"last":0.55700000000000005},{"interface":"front","average":{"1min":3.8380000000000001,"5min":1.5329999999999999,"15min":1.008},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":21.724,"5min":26.146999999999998,"15min":26.146999999999998},"last":0.70099999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.91900000000000004,"15min":0.79800000000000004},"min":{"1min":0.36099999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.835,"5min":21.535,"15min":21.535},"last":0.67600000000000005},{"interface":"front","average":{"1min":1.6160000000000001,"5min":1.3839999999999999,"15min":0.999},"min":{"1min":0.40100000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":5.7539999999999996,"5min":30.488,"15min":30.488},"last":0.82099999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:11:05 2024","interfaces":[{"interface":"back","average":{"1min":4.7309999999999999,"5min":1.6259999999999999,"15min":1.0760000000000001},"min":{"1min":0.42599999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":30.483000000000001,"5min":30.483000000000001,"15min":30.483000000000001},"last":0.629},{"interface":"front","average":{"1min":3.508,"5min":1.4650000000000001,"15min":1.0429999999999999},"min":{"1min":0.42899999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":25.437999999999999,"5min":25.437999999999999,"15min":25.437999999999999},"last":0.72199999999999998}]}]},{"osd":0,"up_from":14,"seq":60129542235,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54800,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317040,"statfs":{"total":96636764160,"available":96580648960,"internally_reserved":0,"allocated":1437696,"data_stored":948240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.89300000000000002,"15min":0.60699999999999998},"min":{"1min":0.26000000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9179999999999999,"5min":19.843,"15min":19.843},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.88700000000000001,"15min":0.66400000000000003},"min":{"1min":0.29199999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73299999999999998,"5min":10.154,"15min":10.154},"last":0.51600000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":1.115,"15min":0.753},"min":{"1min":0.38800000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.754,"5min":25.905000000000001,"15min":25.905000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.77900000000000003,"15min":0.58199999999999996},"min":{"1min":0.29699999999999999,"5min":0.28599999999999998,"15min":0.26700000000000002},"max":{"1min":1.2549999999999999,"5min":9.3829999999999991,"15min":9.3829999999999991},"last":0.47199999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.95699999999999996,"15min":0.70299999999999996},"min":{"1min":0.36599999999999999,"5min":0.308,"15min":0.308},"max":{"1min":1.952,"5min":9.1780000000000008,"15min":9.1780000000000008},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.97399999999999998,"15min":0.73099999999999998},"min":{"1min":0.42899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":2.9809999999999999,"5min":10.210000000000001,"15min":10.210000000000001},"last":0.48899999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":1.343,"15min":0.96399999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.38},"max":{"1min":1.306,"5min":22.814,"15min":22.814},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.446,"15min":1.03},"min":{"1min":0.49099999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":2.169,"5min":23.341999999999999,"15min":23.341999999999999},"last":0.877}]},{"osd":5,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":1.262,"15min":0.94899999999999995},"min":{"1min":0.46899999999999997,"5min":0.44600000000000001,"15min":0.441},"max":{"1min":1.365,"5min":12.428000000000001,"15min":12.428000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.82499999999999996,"5min":1.6930000000000001,"15min":1.1259999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.46300000000000002},"max":{"1min":2.0880000000000001,"5min":19.774000000000001,"15min":19.774000000000001},"last":0.81399999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":1.9219999999999999,"15min":1.1719999999999999},"min":{"1min":0.46700000000000003,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.0449999999999999,"5min":31.059000000000001,"15min":31.059000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.83299999999999996,"5min":1.488,"15min":1.0580000000000001},"min":{"1min":0.57999999999999996,"5min":0.51500000000000001,"15min":0.49099999999999999},"max":{"1min":1.3899999999999999,"5min":40.731000000000002,"15min":40.731000000000002},"last":0.90600000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":1.673,"15min":1.1100000000000001},"min":{"1min":0.61599999999999999,"5min":0.47199999999999998,"15min":0.42499999999999999},"max":{"1min":1.4239999999999999,"5min":24.492999999999999,"15min":24.492999999999999},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.875,"5min":1.5780000000000001,"15min":1.097},"min":{"1min":0.61199999999999999,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":2.2130000000000001,"5min":23.524000000000001,"15min":23.524000000000001},"last":0.77800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:12:10.061 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-15T18:12:10.062 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-15T18:12:10.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.243+0000 7f95b5000640 1 Processor -- start 2024-01-15T18:12:10.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.243+0000 7f95b5000640 1 -- start start 2024-01-15T18:12:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.247+0000 7f95b5000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f95b015f550 0x7f95b00cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.247+0000 7f95b5000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f95b00cf110 0x7f95b00701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.247+0000 7f95b5000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95b00b1cb0 con 0x7f95b0158900 2024-01-15T18:12:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.247+0000 7f95b5000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95b00cf580 con 0x7f95b015f550 2024-01-15T18:12:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.247+0000 7f95b5000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95b015fa00 con 0x7f95b00cf110 2024-01-15T18:12:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.247+0000 7f95ade00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f95b0158900 0x7f95b015bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59448/0 (socket says 172.21.15.27:59448) 2024-01-15T18:12:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.247+0000 7f95ade00640 1 -- 172.21.15.27:0/3318343822 learned_addr learned my addr 172.21.15.27:0/3318343822 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:12:10.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.247+0000 7f95ad400640 1 --2- 172.21.15.27:0/3318343822 >> v2:172.21.15.93:3300/0 conn(0x7f95b015f550 0x7f95b00cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.247+0000 7f95ae800640 1 --2- 172.21.15.27:0/3318343822 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f95b00cf110 0x7f95b00701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.247+0000 7f95aca00640 1 -- 172.21.15.27:0/3318343822 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424665151 0 0) 0x7f95b00b1cb0 con 0x7f95b0158900 2024-01-15T18:12:10.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.247+0000 7f95aca00640 1 -- 172.21.15.27:0/3318343822 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9590003530 con 0x7f95b0158900 2024-01-15T18:12:10.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.247+0000 7f95ae800640 1 -- 172.21.15.27:0/3318343822 >> v2:172.21.15.93:3300/0 conn(0x7f95b015f550 msgr2=0x7f95b00cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:10.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.247+0000 7f95ae800640 1 --2- 172.21.15.27:0/3318343822 >> v2:172.21.15.93:3300/0 conn(0x7f95b015f550 0x7f95b00cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.247+0000 7f95ae800640 1 -- 172.21.15.27:0/3318343822 >> v1:172.21.15.27:6789/0 conn(0x7f95b0158900 legacy=0x7f95b015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.247+0000 7f95ae800640 1 -- 172.21.15.27:0/3318343822 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b00b1f00 con 0x7f95b00cf110 2024-01-15T18:12:10.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.247+0000 7f95ad400640 1 --2- 172.21.15.27:0/3318343822 >> v2:172.21.15.93:3300/0 conn(0x7f95b015f550 0x7f95b00cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:12:10.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.251+0000 7f95ae800640 1 --2- 172.21.15.27:0/3318343822 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f95b00cf110 0x7f95b00701e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f95a8009d90 tx=0x7f95a801c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=132c16431a230daf server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:10.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.251+0000 7f95aca00640 1 -- 172.21.15.27:0/3318343822 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b0070f30 con 0x7f95b00cf110 2024-01-15T18:12:10.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.251+0000 7f95aca00640 1 -- 172.21.15.27:0/3318343822 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95a80a59e0 con 0x7f95b00cf110 2024-01-15T18:12:10.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.251+0000 7f95aca00640 1 -- 172.21.15.27:0/3318343822 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a80a5d70 con 0x7f95b00cf110 2024-01-15T18:12:10.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.251+0000 7f95b5000640 1 -- 172.21.15.27:0/3318343822 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f95b00cf110 msgr2=0x7f95b00701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.251+0000 7f95b5000640 1 --2- 172.21.15.27:0/3318343822 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f95b00cf110 0x7f95b00701e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f95a8009d90 tx=0x7f95a801c920 comp rx=0 tx=0).stop 2024-01-15T18:12:10.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.251+0000 7f95b5000640 1 -- 172.21.15.27:0/3318343822 shutdown_connections 2024-01-15T18:12:10.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.251+0000 7f95b5000640 1 --2- 172.21.15.27:0/3318343822 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f95b00cf110 0x7f95b00701e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.251+0000 7f95b5000640 1 --2- 172.21.15.27:0/3318343822 >> v2:172.21.15.93:3300/0 conn(0x7f95b015f550 0x7f95b00cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.251+0000 7f95b5000640 1 -- 172.21.15.27:0/3318343822 >> 172.21.15.27:0/3318343822 conn(0x7f95b00cb700 msgr2=0x7f95b00cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:10.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.251+0000 7f95b5000640 1 -- 172.21.15.27:0/3318343822 shutdown_connections 2024-01-15T18:12:10.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.251+0000 7f95b5000640 1 -- 172.21.15.27:0/3318343822 wait complete. 2024-01-15T18:12:10.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.251+0000 7f95b5000640 1 Processor -- start 2024-01-15T18:12:10.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.251+0000 7f95b5000640 1 -- start start 2024-01-15T18:12:10.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.255+0000 7f95b5000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f95b00cf110 0x7f95b015a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.255+0000 7f95b5000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95b0158900 0x7f95b0142cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.255+0000 7f95b5000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95b015f550 0x7f95b013f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.255+0000 7f95b5000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b0159f70 con 0x7f95b0158900 2024-01-15T18:12:10.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.255+0000 7f95b5000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b01590a0 con 0x7f95b015f550 2024-01-15T18:12:10.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.255+0000 7f95b5000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95b0143210 con 0x7f95b00cf110 2024-01-15T18:12:10.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.255+0000 7f95ad400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95b0158900 0x7f95b0142cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.255+0000 7f95ad400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95b0158900 0x7f95b0142cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58378/0 (socket says 172.21.15.27:58378) 2024-01-15T18:12:10.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.255+0000 7f95ad400640 1 -- 172.21.15.27:0/3096946637 learned_addr learned my addr 172.21.15.27:0/3096946637 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:10.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.255+0000 7f95ae800640 1 --2- 172.21.15.27:0/3096946637 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95b015f550 0x7f95b013f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.255+0000 7f95ade00640 1 --2- 172.21.15.27:0/3096946637 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f95b00cf110 0x7f95b015a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.255+0000 7f95ad400640 1 -- 172.21.15.27:0/3096946637 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f95b00cf110 msgr2=0x7f95b015a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.255+0000 7f95ad400640 1 --2- 172.21.15.27:0/3096946637 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f95b00cf110 0x7f95b015a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.255+0000 7f95ad400640 1 -- 172.21.15.27:0/3096946637 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95b015f550 msgr2=0x7f95b013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.255+0000 7f95ad400640 1 --2- 172.21.15.27:0/3096946637 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95b015f550 0x7f95b013f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.255+0000 7f95ad400640 1 -- 172.21.15.27:0/3096946637 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9598009420 con 0x7f95b0158900 2024-01-15T18:12:10.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.255+0000 7f95ad400640 1 --2- 172.21.15.27:0/3096946637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95b0158900 0x7f95b0142cd0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f959800e980 tx=0x7f959800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:10.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.255+0000 7f95a6000640 1 -- 172.21.15.27:0/3096946637 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959800ca00 con 0x7f95b0158900 2024-01-15T18:12:10.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.255+0000 7f95a6000640 1 -- 172.21.15.27:0/3096946637 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f959800cb60 con 0x7f95b0158900 2024-01-15T18:12:10.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.255+0000 7f95a6000640 1 -- 172.21.15.27:0/3096946637 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9598013470 con 0x7f95b0158900 2024-01-15T18:12:10.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.255+0000 7f95b5000640 1 -- 172.21.15.27:0/3096946637 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a8009a70 con 0x7f95b0158900 2024-01-15T18:12:10.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.255+0000 7f95b5000640 1 -- 172.21.15.27:0/3096946637 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95b01595d0 con 0x7f95b0158900 2024-01-15T18:12:10.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.255+0000 7f95b5000640 1 -- 172.21.15.27:0/3096946637 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95b015a1e0 con 0x7f95b0158900 2024-01-15T18:12:10.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.259+0000 7f95a6000640 1 -- 172.21.15.27:0/3096946637 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f95980135d0 con 0x7f95b0158900 2024-01-15T18:12:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.259+0000 7f95a6000640 1 --2- 172.21.15.27:0/3096946637 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f95880670d0 0x7f9588069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.259+0000 7f95a6000640 1 -- 172.21.15.27:0/3096946637 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f9598082c50 con 0x7f95b0158900 2024-01-15T18:12:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.259+0000 7f95ade00640 1 --2- 172.21.15.27:0/3096946637 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f95880670d0 0x7f9588069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.259+0000 7f95ade00640 1 --2- 172.21.15.27:0/3096946637 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f95880670d0 0x7f9588069590 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f95a00011f0 tx=0x7f95a0007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:10.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.263+0000 7f95a6000640 1 -- 172.21.15.27:0/3096946637 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9598051b40 con 0x7f95b0158900 2024-01-15T18:12:10.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.475+0000 7f95b5000640 1 -- 172.21.15.27:0/3096946637 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f95b004e850 con 0x7f95b0158900 2024-01-15T18:12:10.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.479+0000 7f95a6000640 1 -- 172.21.15.27:0/3096946637 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+147 (secure 0 0 0) 0x7f95980210b0 con 0x7f95b0158900 2024-01-15T18:12:10.480 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-01-15T18:12:10.480 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOc6VlH+DCLxAA1sKVut+3WjA/B+aYJ6/Jeg== 2024-01-15T18:12:10.480 INFO:teuthology.orchestra.run.smithi027.stdout: caps mds = "allow" 2024-01-15T18:12:10.481 INFO:teuthology.orchestra.run.smithi027.stdout: caps mon = "allow r" 2024-01-15T18:12:10.481 INFO:teuthology.orchestra.run.smithi027.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-15T18:12:10.481 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-01-15T18:12:10.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.483+0000 7f95b5000640 1 -- 172.21.15.27:0/3096946637 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f95880670d0 msgr2=0x7f9588069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.483+0000 7f95b5000640 1 --2- 172.21.15.27:0/3096946637 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f95880670d0 0x7f9588069590 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f95a00011f0 tx=0x7f95a0007d40 comp rx=0 tx=0).stop 2024-01-15T18:12:10.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.483+0000 7f95b5000640 1 -- 172.21.15.27:0/3096946637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95b0158900 msgr2=0x7f95b0142cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.483+0000 7f95b5000640 1 --2- 172.21.15.27:0/3096946637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95b0158900 0x7f95b0142cd0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f959800e980 tx=0x7f959800ee40 comp rx=0 tx=0).stop 2024-01-15T18:12:10.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.483+0000 7f95b5000640 1 -- 172.21.15.27:0/3096946637 shutdown_connections 2024-01-15T18:12:10.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.483+0000 7f95b5000640 1 --2- 172.21.15.27:0/3096946637 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f95880670d0 0x7f9588069590 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.483+0000 7f95b5000640 1 --2- 172.21.15.27:0/3096946637 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f95b015f550 0x7f95b013f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.483+0000 7f95b5000640 1 --2- 172.21.15.27:0/3096946637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95b0158900 0x7f95b0142cd0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.483+0000 7f95b5000640 1 --2- 172.21.15.27:0/3096946637 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f95b00cf110 0x7f95b015a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.483+0000 7f95b5000640 1 -- 172.21.15.27:0/3096946637 >> 172.21.15.27:0/3096946637 conn(0x7f95b00cb700 msgr2=0x7f95b00afc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:10.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.483+0000 7f95b5000640 1 -- 172.21.15.27:0/3096946637 shutdown_connections 2024-01-15T18:12:10.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.483+0000 7f95b5000640 1 -- 172.21.15.27:0/3096946637 wait complete. 2024-01-15T18:12:10.504 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:12:10.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.643+0000 7f4e3b800640 1 Processor -- start 2024-01-15T18:12:10.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.643+0000 7f4e3b800640 1 -- start start 2024-01-15T18:12:10.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.643+0000 7f4e3b800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f4e340c92b0 0x7f4e3415d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.643+0000 7f4e3b800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e3406c4a0 con 0x7f4e340701a0 2024-01-15T18:12:10.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.643+0000 7f4e3b800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e3415c660 con 0x7f4e3415c130 2024-01-15T18:12:10.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.643+0000 7f4e3b800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e3415c940 con 0x7f4e340c92b0 2024-01-15T18:12:10.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.643+0000 7f4e38e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4e340701a0 0x7f4e34070580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59458/0 (socket says 172.21.15.27:59458) 2024-01-15T18:12:10.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.643+0000 7f4e38e00640 1 -- 172.21.15.27:0/640575792 learned_addr learned my addr 172.21.15.27:0/640575792 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:12:10.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.643+0000 7f4e39800640 1 --2- 172.21.15.27:0/640575792 >> v2:172.21.15.93:3300/0 conn(0x7f4e340c92b0 0x7f4e3415d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.643+0000 7f4e33400640 1 -- 172.21.15.27:0/640575792 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30881096 0 0) 0x7f4e3406c4a0 con 0x7f4e340701a0 2024-01-15T18:12:10.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.643+0000 7f4e33400640 1 -- 172.21.15.27:0/640575792 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e14003530 con 0x7f4e340701a0 2024-01-15T18:12:10.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.643+0000 7f4e33400640 1 -- 172.21.15.27:0/640575792 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766243700 0 0) 0x7f4e3415c660 con 0x7f4e3415c130 2024-01-15T18:12:10.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.643+0000 7f4e33400640 1 -- 172.21.15.27:0/640575792 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e3406c4a0 con 0x7f4e3415c130 2024-01-15T18:12:10.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.643+0000 7f4e39800640 1 -- 172.21.15.27:0/640575792 >> v1:172.21.15.93:6789/0 conn(0x7f4e3415c130 legacy=0x7f4e3406dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.643+0000 7f4e39800640 1 -- 172.21.15.27:0/640575792 >> v1:172.21.15.27:6789/0 conn(0x7f4e340701a0 legacy=0x7f4e34070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.643+0000 7f4e39800640 1 -- 172.21.15.27:0/640575792 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3415cc20 con 0x7f4e340c92b0 2024-01-15T18:12:10.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.647+0000 7f4e39800640 1 --2- 172.21.15.27:0/640575792 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e340c92b0 0x7f4e3415d9e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c009d90 tx=0x7f4e2c01c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44d72a6780e685a1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:10.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.647+0000 7f4e33400640 1 -- 172.21.15.27:0/640575792 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c0a5950 con 0x7f4e340c92b0 2024-01-15T18:12:10.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.647+0000 7f4e33400640 1 -- 172.21.15.27:0/640575792 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2c0a5ab0 con 0x7f4e340c92b0 2024-01-15T18:12:10.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.647+0000 7f4e33400640 1 -- 172.21.15.27:0/640575792 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c0a5e40 con 0x7f4e340c92b0 2024-01-15T18:12:10.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.647+0000 7f4e3b800640 1 -- 172.21.15.27:0/640575792 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e340c92b0 msgr2=0x7f4e3415d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.647+0000 7f4e3b800640 1 --2- 172.21.15.27:0/640575792 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e340c92b0 0x7f4e3415d9e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c009d90 tx=0x7f4e2c01c890 comp rx=0 tx=0).stop 2024-01-15T18:12:10.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.651+0000 7f4e3b800640 1 -- 172.21.15.27:0/640575792 shutdown_connections 2024-01-15T18:12:10.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.651+0000 7f4e3b800640 1 --2- 172.21.15.27:0/640575792 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e340c92b0 0x7f4e3415d9e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.651+0000 7f4e3b800640 1 -- 172.21.15.27:0/640575792 >> 172.21.15.27:0/640575792 conn(0x7f4e340cec60 msgr2=0x7f4e340cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:10.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.651+0000 7f4e3b800640 1 -- 172.21.15.27:0/640575792 shutdown_connections 2024-01-15T18:12:10.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.651+0000 7f4e3b800640 1 -- 172.21.15.27:0/640575792 wait complete. 2024-01-15T18:12:10.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.651+0000 7f4e3b800640 1 Processor -- start 2024-01-15T18:12:10.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.651+0000 7f4e3b800640 1 -- start start 2024-01-15T18:12:10.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.651+0000 7f4e3b800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e340701a0 0x7f4e340b4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.651+0000 7f4e3b800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e340c92b0 0x7f4e340c8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.651+0000 7f4e3b800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e3415c130 0x7f4e3414b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.651+0000 7f4e3b800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e340b3dd0 con 0x7f4e340701a0 2024-01-15T18:12:10.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.651+0000 7f4e3b800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e340b3200 con 0x7f4e3415c130 2024-01-15T18:12:10.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.651+0000 7f4e3b800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e340b34a0 con 0x7f4e340c92b0 2024-01-15T18:12:10.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.651+0000 7f4e38e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e340701a0 0x7f4e340b4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.651+0000 7f4e38e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e340701a0 0x7f4e340b4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58386/0 (socket says 172.21.15.27:58386) 2024-01-15T18:12:10.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.651+0000 7f4e38e00640 1 -- 172.21.15.27:0/2562566359 learned_addr learned my addr 172.21.15.27:0/2562566359 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:10.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.651+0000 7f4e39800640 1 --2- 172.21.15.27:0/2562566359 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e3415c130 0x7f4e3414b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.651+0000 7f4e33e00640 1 --2- 172.21.15.27:0/2562566359 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e340c92b0 0x7f4e340c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.651+0000 7f4e38e00640 1 -- 172.21.15.27:0/2562566359 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e340c92b0 msgr2=0x7f4e340c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.651+0000 7f4e38e00640 1 --2- 172.21.15.27:0/2562566359 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e340c92b0 0x7f4e340c8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.651+0000 7f4e38e00640 1 -- 172.21.15.27:0/2562566359 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e3415c130 msgr2=0x7f4e3414b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.651+0000 7f4e38e00640 1 --2- 172.21.15.27:0/2562566359 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e3415c130 0x7f4e3414b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.651+0000 7f4e38e00640 1 -- 172.21.15.27:0/2562566359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2c009a70 con 0x7f4e340701a0 2024-01-15T18:12:10.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.655+0000 7f4e38e00640 1 --2- 172.21.15.27:0/2562566359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e340701a0 0x7f4e340b4260 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f4e2400c4d0 tx=0x7f4e2400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:10.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.655+0000 7f4e30c00640 1 -- 172.21.15.27:0/2562566359 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e24005cf0 con 0x7f4e340701a0 2024-01-15T18:12:10.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.655+0000 7f4e30c00640 1 -- 172.21.15.27:0/2562566359 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e24005e50 con 0x7f4e340701a0 2024-01-15T18:12:10.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.655+0000 7f4e3b800640 1 -- 172.21.15.27:0/2562566359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e340b3780 con 0x7f4e340701a0 2024-01-15T18:12:10.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.655+0000 7f4e30c00640 1 -- 172.21.15.27:0/2562566359 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e240072b0 con 0x7f4e340701a0 2024-01-15T18:12:10.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.655+0000 7f4e3b800640 1 -- 172.21.15.27:0/2562566359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e3414b5f0 con 0x7f4e340701a0 2024-01-15T18:12:10.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.655+0000 7f4e3b800640 1 -- 172.21.15.27:0/2562566359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e3404e850 con 0x7f4e340701a0 2024-01-15T18:12:10.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.655+0000 7f4e30c00640 1 -- 172.21.15.27:0/2562566359 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f4e24007410 con 0x7f4e340701a0 2024-01-15T18:12:10.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.655+0000 7f4e30c00640 1 --2- 172.21.15.27:0/2562566359 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4e0c0670d0 0x7f4e0c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.655+0000 7f4e30c00640 1 -- 172.21.15.27:0/2562566359 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f4e24082140 con 0x7f4e340701a0 2024-01-15T18:12:10.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.655+0000 7f4e33e00640 1 --2- 172.21.15.27:0/2562566359 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4e0c0670d0 0x7f4e0c069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.659+0000 7f4e33e00640 1 --2- 172.21.15.27:0/2562566359 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4e0c0670d0 0x7f4e0c069590 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f4e34152620 tx=0x7f4e1c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:10.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.659+0000 7f4e30c00640 1 -- 172.21.15.27:0/2562566359 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4e24050f20 con 0x7f4e340701a0 2024-01-15T18:12:10.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.851+0000 7f4e3b800640 1 -- 172.21.15.27:0/2562566359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e3406dcc0 con 0x7f4e340701a0 2024-01-15T18:12:10.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.851+0000 7f4e30c00640 1 -- 172.21.15.27:0/2562566359 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+4909 (secure 0 0 0) 0x7f4e24017070 con 0x7f4e340701a0 2024-01-15T18:12:10.853 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:12:10.853 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":47,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6889,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6835/914017554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":914017554},{"type":"v1","addr":"172.21.15.93:6835","nonce":914017554}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":6892,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/4099412077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6836","nonce":4099412077},{"type":"v1","addr":"172.21.15.93:6837","nonce":4099412077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":7215,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/902229728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":902229728},{"type":"v1","addr":"172.21.15.27:6833","nonce":902229728}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:12:00.405416+0000","modified":"2024-01-15T18:12:03.417514+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7227},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7227":{"gid":7227,"name":"c","rank":0,"incarnation":43,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/2027824196","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2027824196},{"type":"v1","addr":"172.21.15.27:6835","nonce":2027824196}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-01-15T18:12:10.853 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 47 2024-01-15T18:12:10.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.851+0000 7f4e3b800640 1 -- 172.21.15.27:0/2562566359 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4e0c0670d0 msgr2=0x7f4e0c069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.851+0000 7f4e3b800640 1 --2- 172.21.15.27:0/2562566359 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4e0c0670d0 0x7f4e0c069590 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f4e34152620 tx=0x7f4e1c009040 comp rx=0 tx=0).stop 2024-01-15T18:12:10.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.851+0000 7f4e3b800640 1 -- 172.21.15.27:0/2562566359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e340701a0 msgr2=0x7f4e340b4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.851+0000 7f4e3b800640 1 --2- 172.21.15.27:0/2562566359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e340701a0 0x7f4e340b4260 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f4e2400c4d0 tx=0x7f4e2400c990 comp rx=0 tx=0).stop 2024-01-15T18:12:10.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.851+0000 7f4e3b800640 1 -- 172.21.15.27:0/2562566359 shutdown_connections 2024-01-15T18:12:10.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.851+0000 7f4e3b800640 1 --2- 172.21.15.27:0/2562566359 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4e0c0670d0 0x7f4e0c069590 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.851+0000 7f4e3b800640 1 --2- 172.21.15.27:0/2562566359 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e3415c130 0x7f4e3414b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.851+0000 7f4e3b800640 1 --2- 172.21.15.27:0/2562566359 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e340c92b0 0x7f4e340c8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.851+0000 7f4e3b800640 1 --2- 172.21.15.27:0/2562566359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e340701a0 0x7f4e340b4260 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.851+0000 7f4e3b800640 1 -- 172.21.15.27:0/2562566359 >> 172.21.15.27:0/2562566359 conn(0x7f4e340cec60 msgr2=0x7f4e34070a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:10.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.851+0000 7f4e3b800640 1 -- 172.21.15.27:0/2562566359 shutdown_connections 2024-01-15T18:12:10.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:10.851+0000 7f4e3b800640 1 -- 172.21.15.27:0/2562566359 wait complete. 2024-01-15T18:12:11.277 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 47, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-15T18:12:00.405416+0000', 'modified': '2024-01-15T18:12:03.417514+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7227}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7227': {'gid': 7227, 'name': 'c', 'rank': 0, 'incarnation': 43, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6835/2027824196', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6834', 'nonce': 2027824196}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 2027824196}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-15T18:12:11.277 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-15T18:12:11.277 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-15T18:12:11.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.419+0000 7fe9c3c00640 1 Processor -- start 2024-01-15T18:12:11.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.419+0000 7fe9c3c00640 1 -- start start 2024-01-15T18:12:11.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.419+0000 7fe9c3c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9bc06ca90 0x7fe9bc06cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.419+0000 7fe9c3c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9bc159c10 0x7fe9bc15a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.419+0000 7fe9c3c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9bc070b90 con 0x7fe9bc153c80 2024-01-15T18:12:11.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.419+0000 7fe9c3c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9bc06d460 con 0x7fe9bc06ca90 2024-01-15T18:12:11.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.419+0000 7fe9c3c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9bc15a980 con 0x7fe9bc159c10 2024-01-15T18:12:11.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.423+0000 7fe9bbe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9bc06ca90 0x7fe9bc06cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.423+0000 7fe9bbe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9bc06ca90 0x7fe9bc06cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58400/0 (socket says 172.21.15.27:58400) 2024-01-15T18:12:11.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.423+0000 7fe9bbe00640 1 -- 172.21.15.27:0/3204571967 learned_addr learned my addr 172.21.15.27:0/3204571967 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:11.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.423+0000 7fe9c1c00640 1 --2- 172.21.15.27:0/3204571967 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9bc159c10 0x7fe9bc15a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.423+0000 7fe9bbe00640 1 -- 172.21.15.27:0/3204571967 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9bc159c10 msgr2=0x7fe9bc15a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:11.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.423+0000 7fe9bbe00640 1 --2- 172.21.15.27:0/3204571967 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9bc159c10 0x7fe9bc15a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.423+0000 7fe9bbe00640 1 -- 172.21.15.27:0/3204571967 >> v1:172.21.15.93:6789/0 conn(0x7fe9bc153c80 legacy=0x7fe9bc071020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.423+0000 7fe9bbe00640 1 -- 172.21.15.27:0/3204571967 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9bc070260 con 0x7fe9bc06ca90 2024-01-15T18:12:11.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.423+0000 7fe9bbe00640 1 --2- 172.21.15.27:0/3204571967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9bc06ca90 0x7fe9bc06cf20 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4009870 tx=0x7fe9a401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d0f31494c9e93dc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:11.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.423+0000 7fe9bb400640 1 -- 172.21.15.27:0/3204571967 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a40a4ea0 con 0x7fe9bc06ca90 2024-01-15T18:12:11.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.423+0000 7fe9bb400640 1 -- 172.21.15.27:0/3204571967 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a40a7050 con 0x7fe9bc06ca90 2024-01-15T18:12:11.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.423+0000 7fe9bb400640 1 -- 172.21.15.27:0/3204571967 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a40ab820 con 0x7fe9bc06ca90 2024-01-15T18:12:11.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.427+0000 7fe9c3c00640 1 -- 172.21.15.27:0/3204571967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9bc06ca90 msgr2=0x7fe9bc06cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.427+0000 7fe9c3c00640 1 --2- 172.21.15.27:0/3204571967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9bc06ca90 0x7fe9bc06cf20 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4009870 tx=0x7fe9a401c4b0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.427+0000 7fe9c3c00640 1 -- 172.21.15.27:0/3204571967 shutdown_connections 2024-01-15T18:12:11.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.427+0000 7fe9c3c00640 1 --2- 172.21.15.27:0/3204571967 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9bc159c10 0x7fe9bc15a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.427+0000 7fe9c3c00640 1 --2- 172.21.15.27:0/3204571967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9bc06ca90 0x7fe9bc06cf20 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.427+0000 7fe9c3c00640 1 -- 172.21.15.27:0/3204571967 >> 172.21.15.27:0/3204571967 conn(0x7fe9bc0b0850 msgr2=0x7fe9bc0b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:11.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.427+0000 7fe9c3c00640 1 -- 172.21.15.27:0/3204571967 shutdown_connections 2024-01-15T18:12:11.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.427+0000 7fe9c3c00640 1 -- 172.21.15.27:0/3204571967 wait complete. 2024-01-15T18:12:11.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.427+0000 7fe9c3c00640 1 Processor -- start 2024-01-15T18:12:11.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.427+0000 7fe9c3c00640 1 -- start start 2024-01-15T18:12:11.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.427+0000 7fe9c3c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe9bc06ca90 0x7fe9bc13c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.427+0000 7fe9c3c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9bc153c80 0x7fe9bc1665d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.427+0000 7fe9c3c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9bc159c10 0x7fe9bc0aeb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.427+0000 7fe9c3c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9bc154060 con 0x7fe9bc159c10 2024-01-15T18:12:11.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.427+0000 7fe9c3c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9bc13c9f0 con 0x7fe9bc06ca90 2024-01-15T18:12:11.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.427+0000 7fe9c3c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9bc166ed0 con 0x7fe9bc153c80 2024-01-15T18:12:11.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.427+0000 7fe9c1c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9bc159c10 0x7fe9bc0aeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.427+0000 7fe9c1c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9bc159c10 0x7fe9bc0aeb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58410/0 (socket says 172.21.15.27:58410) 2024-01-15T18:12:11.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.427+0000 7fe9c1c00640 1 -- 172.21.15.27:0/2910678777 learned_addr learned my addr 172.21.15.27:0/2910678777 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:11.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.427+0000 7fe9bbe00640 1 --2- 172.21.15.27:0/2910678777 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9bc153c80 0x7fe9bc1665d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.427+0000 7fe9c1200640 1 --2- 172.21.15.27:0/2910678777 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe9bc06ca90 0x7fe9bc13c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.431+0000 7fe9c1c00640 1 -- 172.21.15.27:0/2910678777 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9bc153c80 msgr2=0x7fe9bc1665d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.431+0000 7fe9c1c00640 1 --2- 172.21.15.27:0/2910678777 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9bc153c80 0x7fe9bc1665d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.431+0000 7fe9c1c00640 1 -- 172.21.15.27:0/2910678777 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe9bc06ca90 msgr2=0x7fe9bc13c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.431+0000 7fe9c1c00640 1 --2- 172.21.15.27:0/2910678777 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe9bc06ca90 0x7fe9bc13c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.431+0000 7fe9c1c00640 1 -- 172.21.15.27:0/2910678777 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a40094d0 con 0x7fe9bc159c10 2024-01-15T18:12:11.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.431+0000 7fe9bbe00640 1 --2- 172.21.15.27:0/2910678777 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9bc153c80 0x7fe9bc1665d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:12:11.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.431+0000 7fe9c1c00640 1 --2- 172.21.15.27:0/2910678777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9bc159c10 0x7fe9bc0aeb70 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4009d30 tx=0x7fe9b400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:11.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.431+0000 7fe9b8c00640 1 -- 172.21.15.27:0/2910678777 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4009ba0 con 0x7fe9bc159c10 2024-01-15T18:12:11.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.431+0000 7fe9b8c00640 1 -- 172.21.15.27:0/2910678777 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b400f040 con 0x7fe9bc159c10 2024-01-15T18:12:11.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.431+0000 7fe9b8c00640 1 -- 172.21.15.27:0/2910678777 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b40146c0 con 0x7fe9bc159c10 2024-01-15T18:12:11.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.431+0000 7fe9c3c00640 1 -- 172.21.15.27:0/2910678777 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9bc1671b0 con 0x7fe9bc159c10 2024-01-15T18:12:11.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.431+0000 7fe9c3c00640 1 -- 172.21.15.27:0/2910678777 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9bc167770 con 0x7fe9bc159c10 2024-01-15T18:12:11.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.431+0000 7fe9b8c00640 1 -- 172.21.15.27:0/2910678777 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7fe9b4014820 con 0x7fe9bc159c10 2024-01-15T18:12:11.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.431+0000 7fe9b8c00640 1 --2- 172.21.15.27:0/2910678777 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe9940670d0 0x7fe994069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9b8c00640 1 -- 172.21.15.27:0/2910678777 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fe9b4083500 con 0x7fe9bc159c10 2024-01-15T18:12:11.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c7118000 1 Processor -- start 2024-01-15T18:12:11.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c1200640 1 --2- 172.21.15.27:0/2910678777 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe9940670d0 0x7fe994069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c7118000 1 -- start start 2024-01-15T18:12:11.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c1200640 1 --2- 172.21.15.27:0/2910678777 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe9940670d0 0x7fe994069590 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac007fd0 tx=0x7fe9ac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:11.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c7118000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 0x55db5411e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c7118000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55db5411e5b0 0x55db540d3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c7118000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55db540d4320 0x55db540d4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c7118000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db53f3e130 con 0x55db5411dc90 2024-01-15T18:12:11.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c7118000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db53f2b9a0 con 0x55db540d4320 2024-01-15T18:12:11.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c7118000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x55db53f28ed0 con 0x55db5411e5b0 2024-01-15T18:12:11.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c1200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 0x55db5411e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c1200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 0x55db5411e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58418/0 (socket says 172.21.15.27:58418) 2024-01-15T18:12:11.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c1200640 1 -- 172.21.15.27:0/3271603089 learned_addr learned my addr 172.21.15.27:0/3271603089 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:11.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9bbe00640 1 --2- 172.21.15.27:0/3271603089 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55db5411e5b0 0x55db540d3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c1c00640 1 --2- 172.21.15.27:0/3271603089 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55db540d4320 0x55db540d4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c1200640 1 -- 172.21.15.27:0/3271603089 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55db5411e5b0 msgr2=0x55db540d3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:11.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c1200640 1 --2- 172.21.15.27:0/3271603089 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55db5411e5b0 0x55db540d3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c1200640 1 -- 172.21.15.27:0/3271603089 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55db540d4320 msgr2=0x55db540d4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:11.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c1200640 1 --2- 172.21.15.27:0/3271603089 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55db540d4320 0x55db540d4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c1200640 1 -- 172.21.15.27:0/3271603089 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db53c1c890 con 0x55db5411dc90 2024-01-15T18:12:11.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c1200640 1 --2- 172.21.15.27:0/3271603089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 0x55db5411e070 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac00f950 tx=0x7fe9ac012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe691c84d3fe248b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:11.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9b1600640 1 -- 172.21.15.27:0/3271603089 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac00fed0 con 0x55db5411dc90 2024-01-15T18:12:11.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9b1600640 1 -- 172.21.15.27:0/3271603089 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9ac014be0 con 0x55db5411dc90 2024-01-15T18:12:11.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9b1600640 1 -- 172.21.15.27:0/3271603089 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac020410 con 0x55db5411dc90 2024-01-15T18:12:11.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c7118000 1 -- 172.21.15.27:0/3271603089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 msgr2=0x55db5411e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c7118000 1 --2- 172.21.15.27:0/3271603089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 0x55db5411e070 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac00f950 tx=0x7fe9ac012000 comp rx=0 tx=0).stop 2024-01-15T18:12:11.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c7118000 1 -- 172.21.15.27:0/3271603089 shutdown_connections 2024-01-15T18:12:11.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c7118000 1 --2- 172.21.15.27:0/3271603089 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55db540d4320 0x55db540d4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c7118000 1 --2- 172.21.15.27:0/3271603089 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55db5411e5b0 0x55db540d3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c7118000 1 --2- 172.21.15.27:0/3271603089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 0x55db5411e070 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c7118000 1 -- 172.21.15.27:0/3271603089 >> 172.21.15.27:0/3271603089 conn(0x55db540d02d0 msgr2=0x55db540d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:11.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c7118000 1 -- 172.21.15.27:0/3271603089 shutdown_connections 2024-01-15T18:12:11.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.435+0000 7fe9c7118000 1 -- 172.21.15.27:0/3271603089 wait complete. 2024-01-15T18:12:11.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9c7118000 1 -- start start 2024-01-15T18:12:11.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9c7118000 1 Processor -- start 2024-01-15T18:12:11.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9c7118000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55db540d4320 0x55db540d8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9c7118000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 0x55db540d9280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9c7118000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55db5411e5b0 0x55db540ec910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9c7118000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db53f247c0 con 0x55db5411dc90 2024-01-15T18:12:11.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9c7118000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db53edcbf0 con 0x55db5411e5b0 2024-01-15T18:12:11.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9c7118000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x55db53ee2a90 con 0x55db540d4320 2024-01-15T18:12:11.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9c7118000 10 client.0 _ll_register_callbacks cb 0x55db540bf3c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:12:11.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9bbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 0x55db540d9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9bbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 0x55db540d9280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58428/0 (socket says 172.21.15.27:58428) 2024-01-15T18:12:11.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9bbe00640 1 -- 172.21.15.27:0/1865892097 learned_addr learned my addr 172.21.15.27:0/1865892097 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:11.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9c1200640 1 --2- 172.21.15.27:0/1865892097 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55db540d4320 0x55db540d8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9c1c00640 1 --2- 172.21.15.27:0/1865892097 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55db5411e5b0 0x55db540ec910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9bbe00640 1 -- 172.21.15.27:0/1865892097 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55db540d4320 msgr2=0x55db540d8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:11.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9bbe00640 1 --2- 172.21.15.27:0/1865892097 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55db540d4320 0x55db540d8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9bbe00640 1 -- 172.21.15.27:0/1865892097 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55db5411e5b0 msgr2=0x55db540ec910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:11.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9bbe00640 1 --2- 172.21.15.27:0/1865892097 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55db5411e5b0 0x55db540ec910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9bbe00640 1 -- 172.21.15.27:0/1865892097 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9bc070b90 con 0x55db5411dc90 2024-01-15T18:12:11.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9bbe00640 1 --2- 172.21.15.27:0/1865892097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 0x55db540d9280 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fe9a401c4f0 tx=0x7fe9a401acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c66ac2c5fc8afb40 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:11.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9aa000640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-15T18:12:11.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9aa000640 1 -- 172.21.15.27:0/1865892097 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a40c0610 con 0x55db5411dc90 2024-01-15T18:12:11.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9c7118000 1 -- 172.21.15.27:0/1865892097 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55db53ee6470 con 0x55db5411dc90 2024-01-15T18:12:11.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9aa000640 1 -- 172.21.15.27:0/1865892097 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(70..70 src has 1..70) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fe9a40abe30 con 0x55db5411dc90 2024-01-15T18:12:11.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9aa000640 1 -- 172.21.15.27:0/1865892097 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe9a40c0770 con 0x55db5411dc90 2024-01-15T18:12:11.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9c7118000 10 client.7395 fetch_fsmap learned FSMap version 47 2024-01-15T18:12:11.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9c7118000 1 -- 172.21.15.27:0/1865892097 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=2+}) v3 -- 0x55db53ee5190 con 0x55db5411dc90 2024-01-15T18:12:11.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9aa000640 1 -- 172.21.15.27:0/1865892097 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55db53ee6470 con 0x55db5411dc90 2024-01-15T18:12:11.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9aa000640 1 -- 172.21.15.27:0/1865892097 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 47) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe9a40b63f0 con 0x55db5411dc90 2024-01-15T18:12:11.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9c7118000 10 client.7395 fetch_fsmap finished waiting for FSMap version 47 2024-01-15T18:12:11.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9c7118000 10 client.7395 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-15T18:12:11.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9c7118000 20 client.7395 populate_metadata read hostname 'smithi027' 2024-01-15T18:12:11.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9c7118000 1 --2- 172.21.15.27:0/1865892097 >> [v2:172.21.15.27:6834/2027824196,v1:172.21.15.27:6835/2027824196] conn(0x55db540ed320 0x55db5412ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9c7118000 4 client.7395 mds_command: new command op to 7227 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-15T18:12:11.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.439+0000 7fe9c7118000 1 -- 172.21.15.27:0/1865892097 --> [v2:172.21.15.27:6834/2027824196,v1:172.21.15.27:6835/2027824196] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55db53ee9b50 con 0x55db540ed320 2024-01-15T18:12:11.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.443+0000 7fe9c1200640 1 --2- 172.21.15.27:0/1865892097 >> [v2:172.21.15.27:6834/2027824196,v1:172.21.15.27:6835/2027824196] conn(0x55db540ed320 0x55db5412ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.443+0000 7fe9c1200640 1 --2- 172.21.15.27:0/1865892097 >> [v2:172.21.15.27:6834/2027824196,v1:172.21.15.27:6835/2027824196] conn(0x55db540ed320 0x55db5412ff50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59e58afff27c90cf server_cookie=1e695aa333d63e9c in_seq=0 out_seq=0 2024-01-15T18:12:11.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.443+0000 7fe9aa000640 10 client.7395 ms_handle_connect on v2:172.21.15.27:6834/2027824196 2024-01-15T18:12:11.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.443+0000 7fe9aa000640 1 -- 172.21.15.27:0/1865892097 <== mds.0 v2:172.21.15.27:6834/2027824196 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55db53ee9b50 con 0x55db540ed320 2024-01-15T18:12:11.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.443+0000 7fe9aa000640 10 client.7395 handle_command_reply: tid=0 2024-01-15T18:12:11.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.443+0000 7fe9c7118000 1 client.7395 shutdown 2024-01-15T18:12:11.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.443+0000 7fe9c7118000 1 -- 172.21.15.27:0/1865892097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 msgr2=0x55db540d9280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.443+0000 7fe9c7118000 1 --2- 172.21.15.27:0/1865892097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 0x55db540d9280 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fe9a401c4f0 tx=0x7fe9a401acf0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.443+0000 7fe9c7118000 1 -- 172.21.15.27:0/1865892097 shutdown_connections 2024-01-15T18:12:11.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.443+0000 7fe9c7118000 1 --2- 172.21.15.27:0/1865892097 >> [v2:172.21.15.27:6834/2027824196,v1:172.21.15.27:6835/2027824196] conn(0x55db540ed320 0x55db5412ff50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.443+0000 7fe9c7118000 1 --2- 172.21.15.27:0/1865892097 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55db5411e5b0 0x55db540ec910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.443+0000 7fe9c7118000 1 --2- 172.21.15.27:0/1865892097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 0x55db540d9280 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.443+0000 7fe9c7118000 1 --2- 172.21.15.27:0/1865892097 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55db540d4320 0x55db540d8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.443+0000 7fe9aa000640 0 client.7395 ms_handle_reset on v2:172.21.15.27:6834/2027824196 2024-01-15T18:12:11.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.443+0000 7fe9c7118000 1 -- 172.21.15.27:0/1865892097 >> 172.21.15.27:0/1865892097 conn(0x55db540d02d0 msgr2=0x55db540d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:11.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.443+0000 7fe9c7118000 1 -- 172.21.15.27:0/1865892097 shutdown_connections 2024-01-15T18:12:11.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.443+0000 7fe9c7118000 1 -- 172.21.15.27:0/1865892097 wait complete. 2024-01-15T18:12:11.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.443+0000 7fe9c7118000 20 client.7395 trim_cache size 0 max 16384 2024-01-15T18:12:11.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.459+0000 7fe9c7118000 1 Processor -- start 2024-01-15T18:12:11.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.459+0000 7fe9c7118000 1 -- start start 2024-01-15T18:12:11.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.459+0000 7fe9c7118000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55db540d4320 0x55db5411cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.459+0000 7fe9c7118000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 0x55db5411bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.459+0000 7fe9c7118000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55db5411e5b0 0x55db5411c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.459+0000 7fe9c7118000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db53bdd940 con 0x55db5411dc90 2024-01-15T18:12:11.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.459+0000 7fe9c7118000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db53f21bc0 con 0x55db5411e5b0 2024-01-15T18:12:11.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.459+0000 7fe9c7118000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x55db53ed9760 con 0x55db540d4320 2024-01-15T18:12:11.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.459+0000 7fe9bbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 0x55db5411bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.459+0000 7fe9bbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 0x55db5411bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58446/0 (socket says 172.21.15.27:58446) 2024-01-15T18:12:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.459+0000 7fe9bbe00640 1 -- 172.21.15.27:0/3915276407 learned_addr learned my addr 172.21.15.27:0/3915276407 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.459+0000 7fe9c1c00640 1 --2- 172.21.15.27:0/3915276407 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55db5411e5b0 0x55db5411c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.459+0000 7fe9c1200640 1 --2- 172.21.15.27:0/3915276407 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55db540d4320 0x55db5411cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.459+0000 7fe9bbe00640 1 -- 172.21.15.27:0/3915276407 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55db540d4320 msgr2=0x55db5411cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.459+0000 7fe9bbe00640 1 --2- 172.21.15.27:0/3915276407 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55db540d4320 0x55db5411cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9bbe00640 1 -- 172.21.15.27:0/3915276407 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55db5411e5b0 msgr2=0x55db5411c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9bbe00640 1 --2- 172.21.15.27:0/3915276407 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55db5411e5b0 0x55db5411c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9bbe00640 1 -- 172.21.15.27:0/3915276407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db53ecb490 con 0x55db5411dc90 2024-01-15T18:12:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9c1200640 1 --2- 172.21.15.27:0/3915276407 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55db540d4320 0x55db5411cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:12:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9c1c00640 1 --2- 172.21.15.27:0/3915276407 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55db5411e5b0 0x55db5411c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:12:11.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9bbe00640 1 --2- 172.21.15.27:0/3915276407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 0x55db5411bb30 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4008010 tx=0x7fe9a40b7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd7d075300e70aac server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:11.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9b2000640 1 -- 172.21.15.27:0/3915276407 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a40b6b90 con 0x55db5411dc90 2024-01-15T18:12:11.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9b2000640 1 -- 172.21.15.27:0/3915276407 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a40b6cf0 con 0x55db5411dc90 2024-01-15T18:12:11.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9b2000640 1 -- 172.21.15.27:0/3915276407 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a40abd50 con 0x55db5411dc90 2024-01-15T18:12:11.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9c7118000 1 -- 172.21.15.27:0/3915276407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 msgr2=0x55db5411bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9c7118000 1 --2- 172.21.15.27:0/3915276407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 0x55db5411bb30 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4008010 tx=0x7fe9a40b7000 comp rx=0 tx=0).stop 2024-01-15T18:12:11.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9c7118000 1 -- 172.21.15.27:0/3915276407 shutdown_connections 2024-01-15T18:12:11.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9c7118000 1 --2- 172.21.15.27:0/3915276407 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55db5411e5b0 0x55db5411c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9c7118000 1 --2- 172.21.15.27:0/3915276407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 0x55db5411bb30 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9c7118000 1 --2- 172.21.15.27:0/3915276407 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55db540d4320 0x55db5411cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9c7118000 1 -- 172.21.15.27:0/3915276407 >> 172.21.15.27:0/3915276407 conn(0x55db540d02d0 msgr2=0x55db5412b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:11.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9c7118000 1 -- 172.21.15.27:0/3915276407 shutdown_connections 2024-01-15T18:12:11.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9c7118000 1 -- 172.21.15.27:0/3915276407 wait complete. 2024-01-15T18:12:11.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9c7118000 1 -- start start 2024-01-15T18:12:11.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9c7118000 1 Processor -- start 2024-01-15T18:12:11.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9c7118000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55db540d4320 0x55db54100ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9c7118000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 0x55db540d56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9c7118000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55db5411e5b0 0x55db540d5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9c7118000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db53fabfa0 con 0x55db5411dc90 2024-01-15T18:12:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9c7118000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db53fdc0a0 con 0x55db5411e5b0 2024-01-15T18:12:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9c7118000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x55db53f977a0 con 0x55db540d4320 2024-01-15T18:12:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9c7118000 10 client.0 _ll_register_callbacks cb 0x55db53f72c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:12:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9bbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 0x55db540d56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9bbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 0x55db540d56a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58448/0 (socket says 172.21.15.27:58448) 2024-01-15T18:12:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9bbe00640 1 -- 172.21.15.27:0/808338158 learned_addr learned my addr 172.21.15.27:0/808338158 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9c1200640 1 --2- 172.21.15.27:0/808338158 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55db540d4320 0x55db54100ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.463+0000 7fe9c1c00640 1 --2- 172.21.15.27:0/808338158 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55db5411e5b0 0x55db540d5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9bbe00640 1 -- 172.21.15.27:0/808338158 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55db540d4320 msgr2=0x55db54100ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9bbe00640 1 --2- 172.21.15.27:0/808338158 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55db540d4320 0x55db54100ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9bbe00640 1 -- 172.21.15.27:0/808338158 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55db5411e5b0 msgr2=0x55db540d5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9bbe00640 1 --2- 172.21.15.27:0/808338158 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55db5411e5b0 0x55db540d5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9bbe00640 1 -- 172.21.15.27:0/808338158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55db53ee5190 con 0x55db5411dc90 2024-01-15T18:12:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9bbe00640 1 --2- 172.21.15.27:0/808338158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 0x55db540d56a0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4009870 tx=0x7fe9a40a8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e7de2c77b716e29 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9aa000640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-15T18:12:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9aa000640 1 -- 172.21.15.27:0/808338158 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a4004240 con 0x55db5411dc90 2024-01-15T18:12:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9aa000640 1 -- 172.21.15.27:0/808338158 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(70..70 src has 1..70) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fe9a40043a0 con 0x55db5411dc90 2024-01-15T18:12:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9c7118000 1 -- 172.21.15.27:0/808338158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55db53f23110 con 0x55db5411dc90 2024-01-15T18:12:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9aa000640 1 -- 172.21.15.27:0/808338158 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe9a4003620 con 0x55db5411dc90 2024-01-15T18:12:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9c7118000 10 client.7401 fetch_fsmap learned FSMap version 47 2024-01-15T18:12:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9c7118000 1 -- 172.21.15.27:0/808338158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=2+}) v3 -- 0x55db53c1ccc0 con 0x55db5411dc90 2024-01-15T18:12:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9aa000640 1 -- 172.21.15.27:0/808338158 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55db53f23110 con 0x55db5411dc90 2024-01-15T18:12:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9aa000640 1 -- 172.21.15.27:0/808338158 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 47) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe9a4003780 con 0x55db5411dc90 2024-01-15T18:12:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9c7118000 10 client.7401 fetch_fsmap finished waiting for FSMap version 47 2024-01-15T18:12:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9c7118000 10 client.7401 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-15T18:12:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9c7118000 20 client.7401 populate_metadata read hostname 'smithi027' 2024-01-15T18:12:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9c7118000 1 --2- 172.21.15.27:0/808338158 >> [v2:172.21.15.27:6834/2027824196,v1:172.21.15.27:6835/2027824196] conn(0x55db54130340 0x55db54130720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9c7118000 4 client.7401 mds_command: new command op to 7227 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-15T18:12:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9c7118000 1 -- 172.21.15.27:0/808338158 --> [v2:172.21.15.27:6834/2027824196,v1:172.21.15.27:6835/2027824196] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55db53f529a0 con 0x55db54130340 2024-01-15T18:12:11.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9c1200640 1 --2- 172.21.15.27:0/808338158 >> [v2:172.21.15.27:6834/2027824196,v1:172.21.15.27:6835/2027824196] conn(0x55db54130340 0x55db54130720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9c1200640 1 --2- 172.21.15.27:0/808338158 >> [v2:172.21.15.27:6834/2027824196,v1:172.21.15.27:6835/2027824196] conn(0x55db54130340 0x55db54130720 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=416f5460625ccd77 server_cookie=326aacaea933ecf4 in_seq=0 out_seq=0 2024-01-15T18:12:11.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9aa000640 10 client.7401 ms_handle_connect on v2:172.21.15.27:6834/2027824196 2024-01-15T18:12:11.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9aa000640 1 -- 172.21.15.27:0/808338158 <== mds.0 v2:172.21.15.27:6834/2027824196 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+319 (crc 0 0 0) 0x55db53f529a0 con 0x55db54130340 2024-01-15T18:12:11.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9aa000640 10 client.7401 handle_command_reply: tid=0 2024-01-15T18:12:11.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9c7118000 1 client.7401 shutdown 2024-01-15T18:12:11.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9c7118000 1 -- 172.21.15.27:0/808338158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 msgr2=0x55db540d56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9c7118000 1 --2- 172.21.15.27:0/808338158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 0x55db540d56a0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4009870 tx=0x7fe9a40a8000 comp rx=0 tx=0).stop 2024-01-15T18:12:11.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9c7118000 1 -- 172.21.15.27:0/808338158 shutdown_connections 2024-01-15T18:12:11.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9c7118000 1 --2- 172.21.15.27:0/808338158 >> [v2:172.21.15.27:6834/2027824196,v1:172.21.15.27:6835/2027824196] conn(0x55db54130340 0x55db54130720 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9c7118000 1 --2- 172.21.15.27:0/808338158 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55db5411e5b0 0x55db540d5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9c7118000 1 --2- 172.21.15.27:0/808338158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55db5411dc90 0x55db540d56a0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9c7118000 1 --2- 172.21.15.27:0/808338158 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55db540d4320 0x55db54100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9aa000640 0 client.7401 ms_handle_reset on v2:172.21.15.27:6834/2027824196 2024-01-15T18:12:11.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9c7118000 1 -- 172.21.15.27:0/808338158 >> 172.21.15.27:0/808338158 conn(0x55db540d02d0 msgr2=0x55db540d26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:11.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9c7118000 1 -- 172.21.15.27:0/808338158 shutdown_connections 2024-01-15T18:12:11.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9c7118000 1 -- 172.21.15.27:0/808338158 wait complete. 2024-01-15T18:12:11.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.467+0000 7fe9c7118000 20 client.7401 trim_cache size 0 max 16384 2024-01-15T18:12:11.474 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:12:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:12:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-01-15T18:12:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 7227, 2024-01-15T18:12:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-01-15T18:12:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-01-15T18:12:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-01-15T18:12:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 10.453654837, 2024-01-15T18:12:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 47, 2024-01-15T18:12:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 70, 2024-01-15T18:12:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 70, 2024-01-15T18:12:11.475 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 16.114105624 2024-01-15T18:12:11.475 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:12:11.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.471+0000 7fe9c3c00640 1 -- 172.21.15.27:0/2910678777 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe9940670d0 msgr2=0x7fe994069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.471+0000 7fe9c3c00640 1 --2- 172.21.15.27:0/2910678777 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe9940670d0 0x7fe994069590 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac007fd0 tx=0x7fe9ac009040 comp rx=0 tx=0).stop 2024-01-15T18:12:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.471+0000 7fe9c3c00640 1 -- 172.21.15.27:0/2910678777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9bc159c10 msgr2=0x7fe9bc0aeb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.471+0000 7fe9c3c00640 1 --2- 172.21.15.27:0/2910678777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9bc159c10 0x7fe9bc0aeb70 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4009d30 tx=0x7fe9b400c490 comp rx=0 tx=0).stop 2024-01-15T18:12:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.471+0000 7fe9c3c00640 1 -- 172.21.15.27:0/2910678777 shutdown_connections 2024-01-15T18:12:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.471+0000 7fe9c3c00640 1 --2- 172.21.15.27:0/2910678777 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe9940670d0 0x7fe994069590 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.471+0000 7fe9c3c00640 1 --2- 172.21.15.27:0/2910678777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9bc159c10 0x7fe9bc0aeb70 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.471+0000 7fe9c3c00640 1 --2- 172.21.15.27:0/2910678777 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9bc153c80 0x7fe9bc1665d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.471+0000 7fe9c3c00640 1 --2- 172.21.15.27:0/2910678777 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe9bc06ca90 0x7fe9bc13c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.471+0000 7fe9c3c00640 1 -- 172.21.15.27:0/2910678777 >> 172.21.15.27:0/2910678777 conn(0x7fe9bc0b0850 msgr2=0x7fe9bc0cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.471+0000 7fe9c3c00640 1 -- 172.21.15.27:0/2910678777 shutdown_connections 2024-01-15T18:12:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.471+0000 7fe9c3c00640 1 -- 172.21.15.27:0/2910678777 wait complete. 2024-01-15T18:12:11.492 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:12:11.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.631+0000 7f187a600640 1 Processor -- start 2024-01-15T18:12:11.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.631+0000 7f187a600640 1 -- start start 2024-01-15T18:12:11.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.635+0000 7f187a600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f18740701a0 0x7f1874070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.635+0000 7f187a600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f18740c92b0 0x7f187406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.635+0000 7f187a600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f187406c4a0 con 0x7f187406e240 2024-01-15T18:12:11.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.635+0000 7f187a600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1874070ac0 con 0x7f18740c92b0 2024-01-15T18:12:11.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.635+0000 7f187a600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f187415c660 con 0x7f18740701a0 2024-01-15T18:12:11.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.635+0000 7f1872a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f18740c92b0 0x7f187406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.635+0000 7f1872a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f18740c92b0 0x7f187406dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58460/0 (socket says 172.21.15.27:58460) 2024-01-15T18:12:11.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.635+0000 7f1872a00640 1 -- 172.21.15.27:0/4090901139 learned_addr learned my addr 172.21.15.27:0/4090901139 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:11.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.635+0000 7f1873400640 1 --2- 172.21.15.27:0/4090901139 >> v2:172.21.15.93:3300/0 conn(0x7f18740701a0 0x7f1874070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.635+0000 7f1872a00640 1 -- 172.21.15.27:0/4090901139 >> v2:172.21.15.93:3300/0 conn(0x7f18740701a0 msgr2=0x7f1874070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:11.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.635+0000 7f1872a00640 1 --2- 172.21.15.27:0/4090901139 >> v2:172.21.15.93:3300/0 conn(0x7f18740701a0 0x7f1874070580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.635+0000 7f1872a00640 1 -- 172.21.15.27:0/4090901139 >> v1:172.21.15.93:6789/0 conn(0x7f187406e240 legacy=0x7f187415d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:11.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.635+0000 7f1872a00640 1 -- 172.21.15.27:0/4090901139 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f187415c940 con 0x7f18740c92b0 2024-01-15T18:12:11.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.635+0000 7f1873400640 1 --2- 172.21.15.27:0/4090901139 >> v2:172.21.15.93:3300/0 conn(0x7f18740701a0 0x7f1874070580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:11.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.635+0000 7f1872a00640 1 --2- 172.21.15.27:0/4090901139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18740c92b0 0x7f187406dd00 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f18740681f0 tx=0x7f185c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d55bc85ba689c5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:11.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.635+0000 7f1872000640 1 -- 172.21.15.27:0/4090901139 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c0aa5f0 con 0x7f18740c92b0 2024-01-15T18:12:11.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.635+0000 7f1872000640 1 -- 172.21.15.27:0/4090901139 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185c0aa750 con 0x7f18740c92b0 2024-01-15T18:12:11.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.635+0000 7f1872000640 1 -- 172.21.15.27:0/4090901139 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c0aaae0 con 0x7f18740c92b0 2024-01-15T18:12:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.639+0000 7f187a600640 1 -- 172.21.15.27:0/4090901139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18740c92b0 msgr2=0x7f187406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.639+0000 7f187a600640 1 --2- 172.21.15.27:0/4090901139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18740c92b0 0x7f187406dd00 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f18740681f0 tx=0x7f185c01c490 comp rx=0 tx=0).stop 2024-01-15T18:12:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.639+0000 7f187a600640 1 -- 172.21.15.27:0/4090901139 shutdown_connections 2024-01-15T18:12:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.639+0000 7f187a600640 1 --2- 172.21.15.27:0/4090901139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18740c92b0 0x7f187406dd00 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.639+0000 7f187a600640 1 --2- 172.21.15.27:0/4090901139 >> v2:172.21.15.93:3300/0 conn(0x7f18740701a0 0x7f1874070580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.639+0000 7f187a600640 1 -- 172.21.15.27:0/4090901139 >> 172.21.15.27:0/4090901139 conn(0x7f18740cec60 msgr2=0x7f18740cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:11.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.639+0000 7f187a600640 1 -- 172.21.15.27:0/4090901139 shutdown_connections 2024-01-15T18:12:11.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.639+0000 7f187a600640 1 -- 172.21.15.27:0/4090901139 wait complete. 2024-01-15T18:12:11.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.639+0000 7f187a600640 1 Processor -- start 2024-01-15T18:12:11.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.639+0000 7f187a600640 1 -- start start 2024-01-15T18:12:11.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.639+0000 7f187a600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f187406e240 0x7f18741524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.639+0000 7f187a600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18740701a0 0x7f18740cc830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.639+0000 7f187a600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f18740c92b0 0x7f18740ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.639+0000 7f187a600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18740c6200 con 0x7f18740701a0 2024-01-15T18:12:11.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.639+0000 7f187a600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18740cd6e0 con 0x7f18740c92b0 2024-01-15T18:12:11.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.639+0000 7f187a600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18740c5e60 con 0x7f187406e240 2024-01-15T18:12:11.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.639+0000 7f1872a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18740701a0 0x7f18740cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.639+0000 7f1872a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18740701a0 0x7f18740cc830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58474/0 (socket says 172.21.15.27:58474) 2024-01-15T18:12:11.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.639+0000 7f1872a00640 1 -- 172.21.15.27:0/241459321 learned_addr learned my addr 172.21.15.27:0/241459321 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:11.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.643+0000 7f1873400640 1 --2- 172.21.15.27:0/241459321 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f187406e240 0x7f18741524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.639+0000 7f1873e00640 1 --2- 172.21.15.27:0/241459321 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f18740c92b0 0x7f18740ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.643+0000 7f1872a00640 1 -- 172.21.15.27:0/241459321 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f187406e240 msgr2=0x7f18741524b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.643+0000 7f1872a00640 1 --2- 172.21.15.27:0/241459321 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f187406e240 0x7f18741524b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.643+0000 7f1872a00640 1 -- 172.21.15.27:0/241459321 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f18740c92b0 msgr2=0x7f18740ccd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.643+0000 7f1872a00640 1 --2- 172.21.15.27:0/241459321 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f18740c92b0 0x7f18740ccd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.643+0000 7f1872a00640 1 -- 172.21.15.27:0/241459321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185c0094d0 con 0x7f18740701a0 2024-01-15T18:12:11.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.643+0000 7f1872a00640 1 --2- 172.21.15.27:0/241459321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18740701a0 0x7f18740cc830 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f185c01a0c0 tx=0x7f185c01acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:11.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.643+0000 7f186b400640 1 -- 172.21.15.27:0/241459321 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c0b3800 con 0x7f18740701a0 2024-01-15T18:12:11.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.643+0000 7f186b400640 1 -- 172.21.15.27:0/241459321 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185c0b3980 con 0x7f18740701a0 2024-01-15T18:12:11.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.643+0000 7f187a600640 1 -- 172.21.15.27:0/241459321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18740c8b80 con 0x7f18740701a0 2024-01-15T18:12:11.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.643+0000 7f186b400640 1 -- 172.21.15.27:0/241459321 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c0b3d30 con 0x7f18740701a0 2024-01-15T18:12:11.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.643+0000 7f187a600640 1 -- 172.21.15.27:0/241459321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1874152ae0 con 0x7f18740701a0 2024-01-15T18:12:11.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.643+0000 7f187a600640 1 -- 172.21.15.27:0/241459321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1874152dc0 con 0x7f18740701a0 2024-01-15T18:12:11.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.643+0000 7f186b400640 1 -- 172.21.15.27:0/241459321 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f185c0c6440 con 0x7f18740701a0 2024-01-15T18:12:11.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.647+0000 7f186b400640 1 --2- 172.21.15.27:0/241459321 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f184c0670d0 0x7f184c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.647+0000 7f186b400640 1 -- 172.21.15.27:0/241459321 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f185c0a7070 con 0x7f18740701a0 2024-01-15T18:12:11.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.647+0000 7f1873400640 1 --2- 172.21.15.27:0/241459321 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f184c0670d0 0x7f184c069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.647+0000 7f1873400640 1 --2- 172.21.15.27:0/241459321 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f184c0670d0 0x7f184c069590 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f186400ab60 tx=0x7f1864008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:11.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.651+0000 7f186b400640 1 -- 172.21.15.27:0/241459321 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f185c0ec460 con 0x7f18740701a0 2024-01-15T18:12:11.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.839+0000 7f187a600640 1 -- 172.21.15.27:0/241459321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f187404e850 con 0x7f18740701a0 2024-01-15T18:12:11.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.839+0000 7f186b400640 1 -- 172.21.15.27:0/241459321 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v47) v1 ==== 53+0+73 (secure 0 0 0) 0x7f185c0efa80 con 0x7f18740701a0 2024-01-15T18:12:11.842 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:12:11.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.843+0000 7f187a600640 1 -- 172.21.15.27:0/241459321 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f184c0670d0 msgr2=0x7f184c069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.843+0000 7f187a600640 1 --2- 172.21.15.27:0/241459321 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f184c0670d0 0x7f184c069590 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f186400ab60 tx=0x7f1864008040 comp rx=0 tx=0).stop 2024-01-15T18:12:11.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.843+0000 7f187a600640 1 -- 172.21.15.27:0/241459321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18740701a0 msgr2=0x7f18740cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.843+0000 7f187a600640 1 --2- 172.21.15.27:0/241459321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18740701a0 0x7f18740cc830 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f185c01a0c0 tx=0x7f185c01acd0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.843+0000 7f187a600640 1 -- 172.21.15.27:0/241459321 shutdown_connections 2024-01-15T18:12:11.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.843+0000 7f187a600640 1 --2- 172.21.15.27:0/241459321 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f184c0670d0 0x7f184c069590 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.843+0000 7f187a600640 1 --2- 172.21.15.27:0/241459321 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f18740c92b0 0x7f18740ccd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.843+0000 7f187a600640 1 --2- 172.21.15.27:0/241459321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18740701a0 0x7f18740cc830 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.843+0000 7f187a600640 1 --2- 172.21.15.27:0/241459321 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f187406e240 0x7f18741524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.843+0000 7f187a600640 1 -- 172.21.15.27:0/241459321 >> 172.21.15.27:0/241459321 conn(0x7f18740cec60 msgr2=0x7f1874071470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:11.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.843+0000 7f187a600640 1 -- 172.21.15.27:0/241459321 shutdown_connections 2024-01-15T18:12:11.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:11.843+0000 7f187a600640 1 -- 172.21.15.27:0/241459321 wait complete. 2024-01-15T18:12:11.864 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:12:11.864 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:12:11.864 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-01-15T18:12:11.865 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-15T18:12:11.865 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-15T18:12:11.865 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:12:11.865 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:12:11.865 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:12:11.865 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:12:11.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:11.865 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-01-15T18:12:11.874 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:12:11.875 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:12:11.875 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:12:11.875 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:11.875 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-15T18:12:11.875 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:11.875 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:12:11.875 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:12:11.875 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-01-15T18:12:11.875 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:12:11.875 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:12:11.875 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-01-15T18:12:11.875 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:12:11.875 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-01-15T18:12:11.875 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-01-15T18:12:11.876 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-01-15T18:12:11.876 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42027sec preferred_lft 42027sec 2024-01-15T18:12:11.876 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-01-15T18:12:11.876 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:11.876 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:12:11.876 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-01-15T18:12:11.876 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-01-15T18:12:11.876 INFO:teuthology.orchestra.run.smithi027.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-01-15T18:12:11.876 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 02:42:65:20:14:72 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:12:11.876 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-01-15T18:12:11.876 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:11.876 INFO:teuthology.orchestra.run.smithi027.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:12:11.876 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-01-15T18:12:11.877 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:12:11.877 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:11.877 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-01-15T18:12:11.877 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:11.877 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:12:11.877 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:12:11.877 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-01-15T18:12:11.877 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:11.877 INFO:teuthology.orchestra.run.smithi027.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:12:11.877 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:12:11.877 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-01-15T18:12:11.877 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:11.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:11.878 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:12:11.937 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-01-15T18:12:11.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:11.939 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-01-15T18:12:11.992 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-01-15T18:12:11.992 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:12:11.992 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-01-15T18:12:11.993 INFO:teuthology.orchestra.run.smithi027.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-01-15T18:12:11.993 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-01-15T18:12:11.993 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:12:11.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:11.993 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-15T18:12:11.993 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-15T18:12:11.993 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-15T18:12:11.993 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-15T18:12:11.993 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-15T18:12:11.993 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-15T18:12:12.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:12.089 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-15T18:12:12.098 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-15T18:12:12.098 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-15T18:12:12.099 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:12:12.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:12.099 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-01-15T18:12:12.167 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-15T18:12:12.167 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:12:12.167 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-15T18:12:12.219 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:12:12.219 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:12:12.219 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:12:12.270 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-01-15T18:12:12.331 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:12:12.388 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-01-15T18:12:12.388 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-01-15T18:12:12.388 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-01-15T18:12:12.388 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-01-15T18:12:12.388 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-01-15T18:12:12.389 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-01-15T18:12:12.389 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-01-15T18:12:12.389 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-01-15T18:12:12.389 INFO:teuthology.orchestra.run.smithi027.stdout:/run/lock 2024-01-15T18:12:12.389 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-01-15T18:12:12.389 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-01-15T18:12:12.389 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-01-15T18:12:12.389 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:12:12.389 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-01-15T18:12:12.389 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-01-15T18:12:12.389 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-01-15T18:12:12.389 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-01-15T18:12:12.389 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/fs/nfsd 2024-01-15T18:12:12.389 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-01-15T18:12:12.389 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-01-15T18:12:12.390 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:12:12.390 INFO:teuthology.orchestra.run.smithi027.stdout:/run/rpc_pipefs 2024-01-15T18:12:12.390 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-01-15T18:12:12.390 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-01-15T18:12:12.390 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:12:12.390 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:12:12.390 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:12:12.390 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:12:12.390 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-01-15T18:12:12.390 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:12:12.390 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:12:12.390 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:12:12.390 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:12:12.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:12.391 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-01-15T18:12:12.447 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:12:12.447 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-15T18:12:12.494 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-01-15T18:12:12.516 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:12:12.572 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-01-15T18:12:12.572 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-01-15T18:12:12.572 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-01-15T18:12:12.572 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-01-15T18:12:12.573 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-01-15T18:12:12.573 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-01-15T18:12:12.573 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-01-15T18:12:12.573 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-01-15T18:12:12.573 INFO:teuthology.orchestra.run.smithi027.stdout:/run/lock 2024-01-15T18:12:12.573 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-01-15T18:12:12.573 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-01-15T18:12:12.573 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-01-15T18:12:12.573 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:12:12.573 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-01-15T18:12:12.573 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-01-15T18:12:12.573 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-01-15T18:12:12.573 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-01-15T18:12:12.573 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/fs/nfsd 2024-01-15T18:12:12.573 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-01-15T18:12:12.574 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-01-15T18:12:12.574 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:12:12.574 INFO:teuthology.orchestra.run.smithi027.stdout:/run/rpc_pipefs 2024-01-15T18:12:12.574 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-01-15T18:12:12.574 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-01-15T18:12:12.574 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:12:12.574 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:12:12.574 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:12:12.574 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:12:12.574 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-01-15T18:12:12.574 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:12:12.574 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:12:12.574 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:12:12.574 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:12:12.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:12.575 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-01-15T18:12:12.591 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:2024-01-15T18:12:12.587+0000 7f02c3294580 -1 init, newargv = 0x558d5e685ff0 newargc=15 2024-01-15T18:12:12.591 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[44283]: starting ceph client 2024-01-15T18:12:12.597 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[44283]: starting fuse 2024-01-15T18:12:12.599 INFO:teuthology.orchestra.run.smithi027.stdout:43 2024-01-15T18:12:12.599 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-01-15T18:12:12.600 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:12:12.600 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-01-15T18:12:12.600 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-01-15T18:12:12.600 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-01-15T18:12:12.600 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-01-15T18:12:12.600 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:12:12.600 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-01-15T18:12:12.600 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-15T18:12:12.600 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-01-15T18:12:12.600 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-15T18:12:12.600 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:12:12.600 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-01-15T18:12:12.600 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-01-15T18:12:12.600 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-01-15T18:12:12.600 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-01-15T18:12:12.600 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:12:12.601 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-01-15T18:12:12.601 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:12:12.601 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:12:12.601 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:12:12.601 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-01-15T18:12:12.601 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-01-15T18:12:12.601 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-01-15T18:12:12.601 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:12:12.601 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:12:12.601 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.0")) 2024-01-15T18:12:12.601 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-01-15T18:12:12.729 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.0.44283.asok 2024-01-15T18:12:12.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:12:12.741 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.44283.asok 2024-01-15T18:12:12.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:12.741 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.44283.asok status 2024-01-15T18:12:12.913 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:12:12.913 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-01-15T18:12:12.913 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:12:12.913 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:12:12.913 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "0", 2024-01-15T18:12:12.913 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-01-15T18:12:12.914 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-15T18:12:12.914 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "44283", 2024-01-15T18:12:12.914 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-01-15T18:12:12.914 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:12:12.914 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-01-15T18:12:12.914 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-01-15T18:12:12.914 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 7413, 2024-01-15T18:12:12.914 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-01-15T18:12:12.914 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-01-15T18:12:12.914 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-01-15T18:12:12.914 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 7413 2024-01-15T18:12:12.914 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:12:12.914 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-01-15T18:12:12.914 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-01-15T18:12:12.914 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:12:12.915 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3555611547 2024-01-15T18:12:12.915 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:12:12.915 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:12:12.915 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-01-15T18:12:12.915 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-01-15T18:12:12.915 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:12:12.915 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 3555611547 2024-01-15T18:12:12.915 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:12:12.915 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.7413 192.168.0.1:0/3555611547", 2024-01-15T18:12:12.915 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.1:0/3555611547", 2024-01-15T18:12:12.915 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-01-15T18:12:12.915 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 47, 2024-01-15T18:12:12.915 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 70, 2024-01-15T18:12:12.915 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:12:12.915 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-01-15T18:12:12.915 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-01-15T18:12:12.916 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:12:12.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:12.928 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-01-15T18:12:12.928 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-15T18:12:12.938 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-01-15T18:12:12.938 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-15T18:12:12.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:12.938 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:12:12.998 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:12:13.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.179+0000 7f0f99000640 1 Processor -- start 2024-01-15T18:12:13.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.179+0000 7f0f99000640 1 -- start start 2024-01-15T18:12:13.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.179+0000 7f0f99000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0f940cfaa0 0x7f0f9406bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:13.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.179+0000 7f0f99000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f940ae280 con 0x7f0f940ae710 2024-01-15T18:12:13.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.179+0000 7f0f99000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f940adc50 con 0x7f0f94160af0 2024-01-15T18:12:13.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.179+0000 7f0f99000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f940cff80 con 0x7f0f940cfaa0 2024-01-15T18:12:13.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.183+0000 7f0f92800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0f940ae710 0x7f0f940aeba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59460/0 (socket says 172.21.15.27:59460) 2024-01-15T18:12:13.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.183+0000 7f0f92800640 1 -- 172.21.15.27:0/1971702237 learned_addr learned my addr 172.21.15.27:0/1971702237 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:12:13.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.183+0000 7f0f91400640 1 --2- 172.21.15.27:0/1971702237 >> v2:172.21.15.93:3300/0 conn(0x7f0f940cfaa0 0x7f0f9406bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:13.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.183+0000 7f0f90a00640 1 -- 172.21.15.27:0/1971702237 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4179595428 0 0) 0x7f0f940ae280 con 0x7f0f940ae710 2024-01-15T18:12:13.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.183+0000 7f0f90a00640 1 -- 172.21.15.27:0/1971702237 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f74003530 con 0x7f0f940ae710 2024-01-15T18:12:13.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.183+0000 7f0f91400640 1 -- 172.21.15.27:0/1971702237 >> v1:172.21.15.93:6789/0 conn(0x7f0f94160af0 legacy=0x7f0f9406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:13.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.183+0000 7f0f91400640 1 -- 172.21.15.27:0/1971702237 >> v1:172.21.15.27:6789/0 conn(0x7f0f940ae710 legacy=0x7f0f940aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:13.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.183+0000 7f0f90a00640 1 -- 172.21.15.27:0/1971702237 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346739481 0 0) 0x7f0f940adc50 con 0x7f0f94160af0 2024-01-15T18:12:13.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.183+0000 7f0f91400640 1 -- 172.21.15.27:0/1971702237 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f940d0260 con 0x7f0f940cfaa0 2024-01-15T18:12:13.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.187+0000 7f0f91400640 1 --2- 172.21.15.27:0/1971702237 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f940cfaa0 0x7f0f9406bd10 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c0097c0 tx=0x7f0f7c01c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c46aca5eec7034c7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:13.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.187+0000 7f0f90a00640 1 -- 172.21.15.27:0/1971702237 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c0ab400 con 0x7f0f940cfaa0 2024-01-15T18:12:13.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.187+0000 7f0f90a00640 1 -- 172.21.15.27:0/1971702237 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7c0a7050 con 0x7f0f940cfaa0 2024-01-15T18:12:13.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.187+0000 7f0f90a00640 1 -- 172.21.15.27:0/1971702237 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c0ab780 con 0x7f0f940cfaa0 2024-01-15T18:12:13.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.187+0000 7f0f99000640 1 -- 172.21.15.27:0/1971702237 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f940cfaa0 msgr2=0x7f0f9406bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:13.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.187+0000 7f0f99000640 1 --2- 172.21.15.27:0/1971702237 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f940cfaa0 0x7f0f9406bd10 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c0097c0 tx=0x7f0f7c01c3a0 comp rx=0 tx=0).stop 2024-01-15T18:12:13.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.187+0000 7f0f99000640 1 -- 172.21.15.27:0/1971702237 shutdown_connections 2024-01-15T18:12:13.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.187+0000 7f0f99000640 1 --2- 172.21.15.27:0/1971702237 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f940cfaa0 0x7f0f9406bd10 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:13.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.187+0000 7f0f99000640 1 -- 172.21.15.27:0/1971702237 >> 172.21.15.27:0/1971702237 conn(0x7f0f940d0f20 msgr2=0x7f0f940d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:13.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.187+0000 7f0f99000640 1 -- 172.21.15.27:0/1971702237 shutdown_connections 2024-01-15T18:12:13.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.187+0000 7f0f99000640 1 -- 172.21.15.27:0/1971702237 wait complete. 2024-01-15T18:12:13.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.187+0000 7f0f99000640 1 Processor -- start 2024-01-15T18:12:13.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.187+0000 7f0f99000640 1 -- start start 2024-01-15T18:12:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.191+0000 7f0f99000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f940ae710 0x7f0f9413f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.191+0000 7f0f99000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0f940cfaa0 0x7f0f94154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.191+0000 7f0f99000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f94160af0 0x7f0f94154bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.191+0000 7f0f99000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f940cc1d0 con 0x7f0f940ae710 2024-01-15T18:12:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.191+0000 7f0f99000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f940cc4b0 con 0x7f0f94160af0 2024-01-15T18:12:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.191+0000 7f0f99000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f940cc790 con 0x7f0f940cfaa0 2024-01-15T18:12:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.191+0000 7f0f91e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f940ae710 0x7f0f9413f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.191+0000 7f0f91e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f940ae710 0x7f0f9413f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58484/0 (socket says 172.21.15.27:58484) 2024-01-15T18:12:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.191+0000 7f0f91e00640 1 -- 172.21.15.27:0/1705610549 learned_addr learned my addr 172.21.15.27:0/1705610549 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.191+0000 7f0f91400640 1 --2- 172.21.15.27:0/1705610549 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0f940cfaa0 0x7f0f94154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.191+0000 7f0f92800640 1 --2- 172.21.15.27:0/1705610549 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f94160af0 0x7f0f94154bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.191+0000 7f0f91e00640 1 -- 172.21.15.27:0/1705610549 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0f940cfaa0 msgr2=0x7f0f94154680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.191+0000 7f0f91e00640 1 --2- 172.21.15.27:0/1705610549 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0f940cfaa0 0x7f0f94154680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.191+0000 7f0f91e00640 1 -- 172.21.15.27:0/1705610549 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f94160af0 msgr2=0x7f0f94154bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.191+0000 7f0f91e00640 1 --2- 172.21.15.27:0/1705610549 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f94160af0 0x7f0f94154bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.191+0000 7f0f91e00640 1 -- 172.21.15.27:0/1705610549 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f7c009420 con 0x7f0f940ae710 2024-01-15T18:12:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.191+0000 7f0f92800640 1 --2- 172.21.15.27:0/1705610549 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f94160af0 0x7f0f94154bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.191+0000 7f0f91e00640 1 --2- 172.21.15.27:0/1705610549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f940ae710 0x7f0f9413f0c0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f0f8400c4d0 tx=0x7f0f8400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.191+0000 7f0f8a000640 1 -- 172.21.15.27:0/1705610549 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f84005480 con 0x7f0f940ae710 2024-01-15T18:12:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.191+0000 7f0f8a000640 1 -- 172.21.15.27:0/1705610549 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f84003600 con 0x7f0f940ae710 2024-01-15T18:12:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.191+0000 7f0f99000640 1 -- 172.21.15.27:0/1705610549 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f940cca70 con 0x7f0f940ae710 2024-01-15T18:12:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.191+0000 7f0f8a000640 1 -- 172.21.15.27:0/1705610549 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f840039b0 con 0x7f0f940ae710 2024-01-15T18:12:13.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.191+0000 7f0f99000640 1 -- 172.21.15.27:0/1705610549 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9413f600 con 0x7f0f940ae710 2024-01-15T18:12:13.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.191+0000 7f0f99000640 1 -- 172.21.15.27:0/1705610549 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9404e850 con 0x7f0f940ae710 2024-01-15T18:12:13.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.191+0000 7f0f8a000640 1 -- 172.21.15.27:0/1705610549 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f0f84003b10 con 0x7f0f940ae710 2024-01-15T18:12:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.195+0000 7f0f8a000640 1 --2- 172.21.15.27:0/1705610549 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0f6c0670d0 0x7f0f6c069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.195+0000 7f0f8a000640 1 -- 172.21.15.27:0/1705610549 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f0f840820e0 con 0x7f0f940ae710 2024-01-15T18:12:13.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.195+0000 7f0f91400640 1 --2- 172.21.15.27:0/1705610549 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0f6c0670d0 0x7f0f6c069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.195+0000 7f0f91400640 1 --2- 172.21.15.27:0/1705610549 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0f6c0670d0 0x7f0f6c069590 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c0ad000 tx=0x7f0f7c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.199+0000 7f0f8a000640 1 -- 172.21.15.27:0/1705610549 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0f84050fe0 con 0x7f0f940ae710 2024-01-15T18:12:13.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.391+0000 7f0f99000640 1 -- 172.21.15.27:0/1705610549 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0f940ccf20 con 0x7f0f940ae710 2024-01-15T18:12:13.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.391+0000 7f0f8a000640 1 -- 172.21.15.27:0/1705610549 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v47) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0f84017070 con 0x7f0f940ae710 2024-01-15T18:12:13.394 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:12:13.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.395+0000 7f0f99000640 1 -- 172.21.15.27:0/1705610549 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0f6c0670d0 msgr2=0x7f0f6c069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:13.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.395+0000 7f0f99000640 1 --2- 172.21.15.27:0/1705610549 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0f6c0670d0 0x7f0f6c069590 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c0ad000 tx=0x7f0f7c008040 comp rx=0 tx=0).stop 2024-01-15T18:12:13.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.395+0000 7f0f99000640 1 -- 172.21.15.27:0/1705610549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f940ae710 msgr2=0x7f0f9413f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:13.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.395+0000 7f0f99000640 1 --2- 172.21.15.27:0/1705610549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f940ae710 0x7f0f9413f0c0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f0f8400c4d0 tx=0x7f0f8400c990 comp rx=0 tx=0).stop 2024-01-15T18:12:13.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.395+0000 7f0f99000640 1 -- 172.21.15.27:0/1705610549 shutdown_connections 2024-01-15T18:12:13.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.395+0000 7f0f99000640 1 --2- 172.21.15.27:0/1705610549 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0f6c0670d0 0x7f0f6c069590 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:13.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.395+0000 7f0f99000640 1 --2- 172.21.15.27:0/1705610549 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f94160af0 0x7f0f94154bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:13.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.395+0000 7f0f99000640 1 --2- 172.21.15.27:0/1705610549 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0f940cfaa0 0x7f0f94154680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:13.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.395+0000 7f0f99000640 1 --2- 172.21.15.27:0/1705610549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f940ae710 0x7f0f9413f0c0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:13.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.395+0000 7f0f99000640 1 -- 172.21.15.27:0/1705610549 >> 172.21.15.27:0/1705610549 conn(0x7f0f940d0f20 msgr2=0x7f0f940aeb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:13.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.395+0000 7f0f99000640 1 -- 172.21.15.27:0/1705610549 shutdown_connections 2024-01-15T18:12:13.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:13.395+0000 7f0f99000640 1 -- 172.21.15.27:0/1705610549 wait complete. 2024-01-15T18:12:13.416 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:12:13.416 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:12:13.416 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi093.front.sepia.ceph.com 2024-01-15T18:12:13.416 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-15T18:12:13.416 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-15T18:12:13.416 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:12:13.416 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:12:13.417 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:12:13.417 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:12:13.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:13.417 DEBUG:teuthology.orchestra.run.smithi093:> ip addr 2024-01-15T18:12:13.426 INFO:teuthology.orchestra.run.smithi093.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:12:13.426 INFO:teuthology.orchestra.run.smithi093.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:12:13.427 INFO:teuthology.orchestra.run.smithi093.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:12:13.427 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:13.427 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 ::1/128 scope host 2024-01-15T18:12:13.427 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:13.427 INFO:teuthology.orchestra.run.smithi093.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:12:13.427 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:d8:da:1c brd ff:ff:ff:ff:ff:ff 2024-01-15T18:12:13.427 INFO:teuthology.orchestra.run.smithi093.stdout: altname enp7s0f0 2024-01-15T18:12:13.427 INFO:teuthology.orchestra.run.smithi093.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:12:13.427 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:d8:da:1d brd ff:ff:ff:ff:ff:ff 2024-01-15T18:12:13.427 INFO:teuthology.orchestra.run.smithi093.stdout: altname enp7s0f1 2024-01-15T18:12:13.427 INFO:teuthology.orchestra.run.smithi093.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:12:13.427 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:8f:d6:34 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:12:13.427 INFO:teuthology.orchestra.run.smithi093.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:12:13.427 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:8f:d6:35 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:12:13.427 INFO:teuthology.orchestra.run.smithi093.stdout: inet 172.21.15.93/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-01-15T18:12:13.428 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft 42005sec preferred_lft 42005sec 2024-01-15T18:12:13.428 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::ec4:7aff:fe8f:d635/64 scope link 2024-01-15T18:12:13.428 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:13.428 INFO:teuthology.orchestra.run.smithi093.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-01-15T18:12:13.428 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 02:42:2a:61:fc:7d brd ff:ff:ff:ff:ff:ff 2024-01-15T18:12:13.428 INFO:teuthology.orchestra.run.smithi093.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-01-15T18:12:13.428 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:13.428 INFO:teuthology.orchestra.run.smithi093.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:12:13.428 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-01-15T18:12:13.428 INFO:teuthology.orchestra.run.smithi093.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:12:13.428 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:13.428 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-01-15T18:12:13.428 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:13.428 INFO:teuthology.orchestra.run.smithi093.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:12:13.429 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:12:13.429 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-01-15T18:12:13.429 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:13.429 INFO:teuthology.orchestra.run.smithi093.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:12:13.429 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:12:13.429 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-01-15T18:12:13.429 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:13.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:13.430 DEBUG:teuthology.orchestra.run.smithi093:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:12:13.491 INFO:teuthology.orchestra.run.smithi093.stdout:1 2024-01-15T18:12:13.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:13.493 DEBUG:teuthology.orchestra.run.smithi093:> route 2024-01-15T18:12:13.546 INFO:teuthology.orchestra.run.smithi093.stdout:Kernel IP routing table 2024-01-15T18:12:13.546 INFO:teuthology.orchestra.run.smithi093.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:12:13.546 INFO:teuthology.orchestra.run.smithi093.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-01-15T18:12:13.546 INFO:teuthology.orchestra.run.smithi093.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T18:12:13.546 INFO:teuthology.orchestra.run.smithi093.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-01-15T18:12:13.546 INFO:teuthology.orchestra.run.smithi093.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T18:12:13.546 INFO:teuthology.orchestra.run.smithi093.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:12:13.546 INFO:teuthology.orchestra.run.smithi093.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:12:13.546 INFO:teuthology.orchestra.run.smithi093.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:12:13.546 INFO:teuthology.orchestra.run.smithi093.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:12:13.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:13.547 DEBUG:teuthology.orchestra.run.smithi093:> (cd / && exec bash -c ' 2024-01-15T18:12:13.547 DEBUG:teuthology.orchestra.run.smithi093:> set -e 2024-01-15T18:12:13.547 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T18:12:13.547 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T18:12:13.547 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T18:12:13.547 DEBUG:teuthology.orchestra.run.smithi093:> ') 2024-01-15T18:12:13.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:13.638 DEBUG:teuthology.orchestra.run.smithi093:> ip netns list 2024-01-15T18:12:13.647 INFO:teuthology.orchestra.run.smithi093.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-15T18:12:13.647 INFO:teuthology.orchestra.run.smithi093.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-15T18:12:13.647 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:12:13.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:13.648 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip link set brx.0 up 2024-01-15T18:12:13.714 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-15T18:12:13.715 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:12:13.715 DEBUG:teuthology.orchestra.run.smithi093:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-15T18:12:13.767 INFO:teuthology.orchestra.run.smithi093.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:12:13.767 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:12:13.768 DEBUG:teuthology.orchestra.run.smithi093:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:12:13.818 DEBUG:teuthology.orchestra.run.smithi093:> sudo modprobe fuse 2024-01-15T18:12:13.879 DEBUG:teuthology.orchestra.run.smithi093:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:12:13.932 INFO:teuthology.orchestra.run.smithi093.stdout:/sys 2024-01-15T18:12:13.933 INFO:teuthology.orchestra.run.smithi093.stdout:/proc 2024-01-15T18:12:13.933 INFO:teuthology.orchestra.run.smithi093.stdout:/dev 2024-01-15T18:12:13.933 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/pts 2024-01-15T18:12:13.933 INFO:teuthology.orchestra.run.smithi093.stdout:/run 2024-01-15T18:12:13.933 INFO:teuthology.orchestra.run.smithi093.stdout:/ 2024-01-15T18:12:13.933 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/security 2024-01-15T18:12:13.933 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/shm 2024-01-15T18:12:13.933 INFO:teuthology.orchestra.run.smithi093.stdout:/run/lock 2024-01-15T18:12:13.933 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/cgroup 2024-01-15T18:12:13.933 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/pstore 2024-01-15T18:12:13.933 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/bpf 2024-01-15T18:12:13.933 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:12:13.933 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/hugepages 2024-01-15T18:12:13.933 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/mqueue 2024-01-15T18:12:13.934 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/debug 2024-01-15T18:12:13.934 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/tracing 2024-01-15T18:12:13.934 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/fs/nfsd 2024-01-15T18:12:13.934 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/fuse/connections 2024-01-15T18:12:13.934 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/config 2024-01-15T18:12:13.934 INFO:teuthology.orchestra.run.smithi093.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:12:13.934 INFO:teuthology.orchestra.run.smithi093.stdout:/run/rpc_pipefs 2024-01-15T18:12:13.934 INFO:teuthology.orchestra.run.smithi093.stdout:/run/user/1000 2024-01-15T18:12:13.934 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph 2024-01-15T18:12:13.934 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:12:13.934 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:12:13.934 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:12:13.934 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:12:13.934 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns 2024-01-15T18:12:13.934 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:12:13.935 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:12:13.935 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:12:13.935 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:12:13.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:13.936 DEBUG:teuthology.orchestra.run.smithi093:> ls /sys/fs/fuse/connections 2024-01-15T18:12:13.991 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:12:13.991 DEBUG:teuthology.orchestra.run.smithi093:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-15T18:12:14.034 DEBUG:teuthology.orchestra.run.smithi093:> sudo modprobe fuse 2024-01-15T18:12:14.054 DEBUG:teuthology.orchestra.run.smithi093:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:12:14.109 INFO:teuthology.orchestra.run.smithi093.stdout:/sys 2024-01-15T18:12:14.109 INFO:teuthology.orchestra.run.smithi093.stdout:/proc 2024-01-15T18:12:14.109 INFO:teuthology.orchestra.run.smithi093.stdout:/dev 2024-01-15T18:12:14.109 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/pts 2024-01-15T18:12:14.109 INFO:teuthology.orchestra.run.smithi093.stdout:/run 2024-01-15T18:12:14.109 INFO:teuthology.orchestra.run.smithi093.stdout:/ 2024-01-15T18:12:14.110 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/security 2024-01-15T18:12:14.110 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/shm 2024-01-15T18:12:14.110 INFO:teuthology.orchestra.run.smithi093.stdout:/run/lock 2024-01-15T18:12:14.110 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/cgroup 2024-01-15T18:12:14.110 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/pstore 2024-01-15T18:12:14.110 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/bpf 2024-01-15T18:12:14.110 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:12:14.110 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/hugepages 2024-01-15T18:12:14.110 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/mqueue 2024-01-15T18:12:14.110 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/debug 2024-01-15T18:12:14.110 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/tracing 2024-01-15T18:12:14.110 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/fs/nfsd 2024-01-15T18:12:14.110 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/fuse/connections 2024-01-15T18:12:14.110 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/config 2024-01-15T18:12:14.110 INFO:teuthology.orchestra.run.smithi093.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:12:14.110 INFO:teuthology.orchestra.run.smithi093.stdout:/run/rpc_pipefs 2024-01-15T18:12:14.111 INFO:teuthology.orchestra.run.smithi093.stdout:/run/user/1000 2024-01-15T18:12:14.111 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph 2024-01-15T18:12:14.111 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:12:14.111 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:12:14.111 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:12:14.111 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:12:14.111 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns 2024-01-15T18:12:14.111 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:12:14.111 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:12:14.111 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:12:14.111 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:12:14.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:14.112 DEBUG:teuthology.orchestra.run.smithi093:> ls /sys/fs/fuse/connections 2024-01-15T18:12:14.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:2024-01-15T18:12:14.131+0000 7ff5db646580 -1 init, newargv = 0x55db4a8e73c0 newargc=15 2024-01-15T18:12:14.132 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:ceph-fuse[34052]: starting ceph client 2024-01-15T18:12:14.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:ceph-fuse[34052]: starting fuse 2024-01-15T18:12:15.241 DEBUG:teuthology.orchestra.run.smithi093:> sudo modprobe fuse 2024-01-15T18:12:15.264 DEBUG:teuthology.orchestra.run.smithi093:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:12:15.321 INFO:teuthology.orchestra.run.smithi093.stdout:/sys 2024-01-15T18:12:15.321 INFO:teuthology.orchestra.run.smithi093.stdout:/proc 2024-01-15T18:12:15.321 INFO:teuthology.orchestra.run.smithi093.stdout:/dev 2024-01-15T18:12:15.321 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/pts 2024-01-15T18:12:15.321 INFO:teuthology.orchestra.run.smithi093.stdout:/run 2024-01-15T18:12:15.321 INFO:teuthology.orchestra.run.smithi093.stdout:/ 2024-01-15T18:12:15.321 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/security 2024-01-15T18:12:15.322 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/shm 2024-01-15T18:12:15.322 INFO:teuthology.orchestra.run.smithi093.stdout:/run/lock 2024-01-15T18:12:15.322 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/cgroup 2024-01-15T18:12:15.322 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/pstore 2024-01-15T18:12:15.322 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/bpf 2024-01-15T18:12:15.322 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:12:15.322 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/hugepages 2024-01-15T18:12:15.322 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/mqueue 2024-01-15T18:12:15.322 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/debug 2024-01-15T18:12:15.322 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/tracing 2024-01-15T18:12:15.322 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/fs/nfsd 2024-01-15T18:12:15.322 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/fuse/connections 2024-01-15T18:12:15.322 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/config 2024-01-15T18:12:15.322 INFO:teuthology.orchestra.run.smithi093.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:12:15.322 INFO:teuthology.orchestra.run.smithi093.stdout:/run/rpc_pipefs 2024-01-15T18:12:15.323 INFO:teuthology.orchestra.run.smithi093.stdout:/run/user/1000 2024-01-15T18:12:15.323 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph 2024-01-15T18:12:15.323 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:12:15.323 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:12:15.323 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:12:15.323 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:12:15.323 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns 2024-01-15T18:12:15.323 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:12:15.323 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:12:15.323 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:12:15.323 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:12:15.323 INFO:teuthology.orchestra.run.smithi093.stdout:/home/ubuntu/cephtest/mnt.1 2024-01-15T18:12:15.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:15.324 DEBUG:teuthology.orchestra.run.smithi093:> ls /sys/fs/fuse/connections 2024-01-15T18:12:15.375 INFO:teuthology.orchestra.run.smithi093.stdout:43 2024-01-15T18:12:15.375 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-01-15T18:12:15.375 DEBUG:teuthology.orchestra.run.smithi093:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:12:15.375 DEBUG:teuthology.orchestra.run.smithi093:> import glob 2024-01-15T18:12:15.376 DEBUG:teuthology.orchestra.run.smithi093:> import re 2024-01-15T18:12:15.376 DEBUG:teuthology.orchestra.run.smithi093:> import os 2024-01-15T18:12:15.376 DEBUG:teuthology.orchestra.run.smithi093:> import subprocess 2024-01-15T18:12:15.376 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:12:15.376 DEBUG:teuthology.orchestra.run.smithi093:> def _find_admin_socket(client_name): 2024-01-15T18:12:15.376 DEBUG:teuthology.orchestra.run.smithi093:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-15T18:12:15.376 DEBUG:teuthology.orchestra.run.smithi093:> files = glob.glob(asok_path) 2024-01-15T18:12:15.376 DEBUG:teuthology.orchestra.run.smithi093:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-15T18:12:15.376 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:12:15.376 DEBUG:teuthology.orchestra.run.smithi093:> # Given a non-glob path, it better be there 2024-01-15T18:12:15.376 DEBUG:teuthology.orchestra.run.smithi093:> if "*" not in asok_path: 2024-01-15T18:12:15.376 DEBUG:teuthology.orchestra.run.smithi093:> assert(len(files) == 1) 2024-01-15T18:12:15.376 DEBUG:teuthology.orchestra.run.smithi093:> return files[0] 2024-01-15T18:12:15.376 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:12:15.376 DEBUG:teuthology.orchestra.run.smithi093:> for f in files: 2024-01-15T18:12:15.377 DEBUG:teuthology.orchestra.run.smithi093:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:12:15.377 DEBUG:teuthology.orchestra.run.smithi093:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:12:15.377 DEBUG:teuthology.orchestra.run.smithi093:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:12:15.377 DEBUG:teuthology.orchestra.run.smithi093:> contents = proc_f.read() 2024-01-15T18:12:15.377 DEBUG:teuthology.orchestra.run.smithi093:> if mountpoint in contents: 2024-01-15T18:12:15.377 DEBUG:teuthology.orchestra.run.smithi093:> return f 2024-01-15T18:12:15.377 DEBUG:teuthology.orchestra.run.smithi093:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:12:15.377 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:12:15.377 DEBUG:teuthology.orchestra.run.smithi093:> print(_find_admin_socket("client.1")) 2024-01-15T18:12:15.377 DEBUG:teuthology.orchestra.run.smithi093:> ' 2024-01-15T18:12:15.503 INFO:teuthology.orchestra.run.smithi093.stdout:/var/run/ceph/ceph-client.1.34052.asok 2024-01-15T18:12:15.507 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:12:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:12:15.515 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.34052.asok 2024-01-15T18:12:15.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:15.515 DEBUG:teuthology.orchestra.run.smithi093:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.34052.asok status 2024-01-15T18:12:15.685 INFO:teuthology.orchestra.run.smithi093.stdout:{ 2024-01-15T18:12:15.685 INFO:teuthology.orchestra.run.smithi093.stdout: "metadata": { 2024-01-15T18:12:15.685 INFO:teuthology.orchestra.run.smithi093.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:12:15.685 INFO:teuthology.orchestra.run.smithi093.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:12:15.685 INFO:teuthology.orchestra.run.smithi093.stdout: "entity_id": "1", 2024-01-15T18:12:15.685 INFO:teuthology.orchestra.run.smithi093.stdout: "hostname": "smithi093", 2024-01-15T18:12:15.685 INFO:teuthology.orchestra.run.smithi093.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-15T18:12:15.685 INFO:teuthology.orchestra.run.smithi093.stdout: "pid": "34052", 2024-01-15T18:12:15.685 INFO:teuthology.orchestra.run.smithi093.stdout: "root": "/" 2024-01-15T18:12:15.685 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:12:15.685 INFO:teuthology.orchestra.run.smithi093.stdout: "dentry_count": 0, 2024-01-15T18:12:15.685 INFO:teuthology.orchestra.run.smithi093.stdout: "dentry_pinned_count": 0, 2024-01-15T18:12:15.685 INFO:teuthology.orchestra.run.smithi093.stdout: "id": 7054, 2024-01-15T18:12:15.685 INFO:teuthology.orchestra.run.smithi093.stdout: "inst": { 2024-01-15T18:12:15.686 INFO:teuthology.orchestra.run.smithi093.stdout: "name": { 2024-01-15T18:12:15.686 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "client", 2024-01-15T18:12:15.686 INFO:teuthology.orchestra.run.smithi093.stdout: "num": 7054 2024-01-15T18:12:15.686 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:12:15.686 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": { 2024-01-15T18:12:15.686 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "v1", 2024-01-15T18:12:15.686 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:12:15.686 INFO:teuthology.orchestra.run.smithi093.stdout: "nonce": 2563923422 2024-01-15T18:12:15.686 INFO:teuthology.orchestra.run.smithi093.stdout: } 2024-01-15T18:12:15.686 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:12:15.686 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": { 2024-01-15T18:12:15.686 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "v1", 2024-01-15T18:12:15.686 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:12:15.686 INFO:teuthology.orchestra.run.smithi093.stdout: "nonce": 2563923422 2024-01-15T18:12:15.686 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:12:15.686 INFO:teuthology.orchestra.run.smithi093.stdout: "inst_str": "client.7054 192.168.0.1:0/2563923422", 2024-01-15T18:12:15.687 INFO:teuthology.orchestra.run.smithi093.stdout: "addr_str": "192.168.0.1:0/2563923422", 2024-01-15T18:12:15.687 INFO:teuthology.orchestra.run.smithi093.stdout: "inode_count": 1, 2024-01-15T18:12:15.687 INFO:teuthology.orchestra.run.smithi093.stdout: "mds_epoch": 47, 2024-01-15T18:12:15.687 INFO:teuthology.orchestra.run.smithi093.stdout: "osd_epoch": 70, 2024-01-15T18:12:15.687 INFO:teuthology.orchestra.run.smithi093.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:12:15.687 INFO:teuthology.orchestra.run.smithi093.stdout: "blocklisted": false, 2024-01-15T18:12:15.687 INFO:teuthology.orchestra.run.smithi093.stdout: "fs_name": "cephfs" 2024-01-15T18:12:15.687 INFO:teuthology.orchestra.run.smithi093.stdout:} 2024-01-15T18:12:15.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:15.699 DEBUG:teuthology.orchestra.run.smithi093:> stat --file-system '--printf=%T 2024-01-15T18:12:15.699 DEBUG:teuthology.orchestra.run.smithi093:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-15T18:12:15.708 INFO:teuthology.orchestra.run.smithi093.stdout:fuseblk 2024-01-15T18:12:15.708 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-15T18:12:15.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:15.709 DEBUG:teuthology.orchestra.run.smithi093:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:12:15.770 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-15T18:12:15.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.923+0000 7fe27e600640 1 Processor -- start 2024-01-15T18:12:15.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.927+0000 7fe27e600640 1 -- start start 2024-01-15T18:12:15.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.927+0000 7fe27e600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fe27815f550 0x7fe2780cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:15.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.927+0000 7fe27e600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe2780cf110 0x7fe2780701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:15.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.927+0000 7fe27e600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2780b1cb0 con 0x7fe278158900 2024-01-15T18:12:15.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.927+0000 7fe27e600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2780cf580 con 0x7fe2780cf110 2024-01-15T18:12:15.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.927+0000 7fe27e600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe27815fa00 con 0x7fe27815f550 2024-01-15T18:12:15.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.927+0000 7fe277400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe278158900 0x7fe27815bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38306/0 (socket says 172.21.15.27:38306) 2024-01-15T18:12:15.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.927+0000 7fe277400640 1 -- 172.21.15.27:0/1312829652 learned_addr learned my addr 172.21.15.27:0/1312829652 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:12:15.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.927+0000 7fe277e00640 1 --2- 172.21.15.27:0/1312829652 >> v2:172.21.15.27:3300/0 conn(0x7fe2780cf110 0x7fe2780701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:15.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.927+0000 7fe276a00640 1 --2- 172.21.15.27:0/1312829652 >> v2:172.21.15.93:3300/0 conn(0x7fe27815f550 0x7fe2780cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:15.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.927+0000 7fe276000640 1 -- 172.21.15.27:0/1312829652 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968479512 0 0) 0x7fe2780b1cb0 con 0x7fe278158900 2024-01-15T18:12:15.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.927+0000 7fe276000640 1 -- 172.21.15.27:0/1312829652 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe258003530 con 0x7fe278158900 2024-01-15T18:12:15.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.927+0000 7fe277e00640 1 -- 172.21.15.27:0/1312829652 >> v2:172.21.15.93:3300/0 conn(0x7fe27815f550 msgr2=0x7fe2780cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:15.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.927+0000 7fe277e00640 1 --2- 172.21.15.27:0/1312829652 >> v2:172.21.15.93:3300/0 conn(0x7fe27815f550 0x7fe2780cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:15.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.927+0000 7fe277e00640 1 -- 172.21.15.27:0/1312829652 >> v1:172.21.15.27:6789/0 conn(0x7fe278158900 legacy=0x7fe27815bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:15.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.927+0000 7fe277e00640 1 -- 172.21.15.27:0/1312829652 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2780b1f00 con 0x7fe2780cf110 2024-01-15T18:12:15.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.931+0000 7fe277e00640 1 --2- 172.21.15.27:0/1312829652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2780cf110 0x7fe2780701e0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fe270009d90 tx=0x7fe27001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83d9bf4e89d8cafb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:15.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.931+0000 7fe276000640 1 -- 172.21.15.27:0/1312829652 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe278070f30 con 0x7fe2780cf110 2024-01-15T18:12:15.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.931+0000 7fe276000640 1 -- 172.21.15.27:0/1312829652 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2700a59a0 con 0x7fe2780cf110 2024-01-15T18:12:15.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.931+0000 7fe276000640 1 -- 172.21.15.27:0/1312829652 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2700a5d10 con 0x7fe2780cf110 2024-01-15T18:12:15.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.931+0000 7fe27e600640 1 -- 172.21.15.27:0/1312829652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2780cf110 msgr2=0x7fe2780701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:15.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.931+0000 7fe27e600640 1 --2- 172.21.15.27:0/1312829652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2780cf110 0x7fe2780701e0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fe270009d90 tx=0x7fe27001c8e0 comp rx=0 tx=0).stop 2024-01-15T18:12:15.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.931+0000 7fe27e600640 1 -- 172.21.15.27:0/1312829652 shutdown_connections 2024-01-15T18:12:15.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.931+0000 7fe27e600640 1 --2- 172.21.15.27:0/1312829652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2780cf110 0x7fe2780701e0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:15.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.931+0000 7fe27e600640 1 --2- 172.21.15.27:0/1312829652 >> v2:172.21.15.93:3300/0 conn(0x7fe27815f550 0x7fe2780cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:15.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.931+0000 7fe27e600640 1 -- 172.21.15.27:0/1312829652 >> 172.21.15.27:0/1312829652 conn(0x7fe2780cb700 msgr2=0x7fe2780cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:15.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.931+0000 7fe27e600640 1 -- 172.21.15.27:0/1312829652 shutdown_connections 2024-01-15T18:12:15.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.931+0000 7fe27e600640 1 -- 172.21.15.27:0/1312829652 wait complete. 2024-01-15T18:12:15.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.935+0000 7fe27e600640 1 Processor -- start 2024-01-15T18:12:15.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.935+0000 7fe27e600640 1 -- start start 2024-01-15T18:12:15.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.935+0000 7fe27e600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2780cf110 0x7fe278150200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:15.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.935+0000 7fe27e600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe278158900 0x7fe2780aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:15.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.935+0000 7fe27e600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe27815f550 0x7fe2780af2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:15.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.935+0000 7fe27e600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27815c7c0 con 0x7fe2780cf110 2024-01-15T18:12:15.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.935+0000 7fe27e600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe278157f40 con 0x7fe278158900 2024-01-15T18:12:15.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.935+0000 7fe27e600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe278150c80 con 0x7fe27815f550 2024-01-15T18:12:15.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.935+0000 7fe277400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2780cf110 0x7fe278150200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:15.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.935+0000 7fe277400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2780cf110 0x7fe278150200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52840/0 (socket says 172.21.15.27:52840) 2024-01-15T18:12:15.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.935+0000 7fe277400640 1 -- 172.21.15.27:0/1450502175 learned_addr learned my addr 172.21.15.27:0/1450502175 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.935+0000 7fe277e00640 1 --2- 172.21.15.27:0/1450502175 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe27815f550 0x7fe2780af2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.935+0000 7fe276a00640 1 --2- 172.21.15.27:0/1450502175 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe278158900 0x7fe2780aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.935+0000 7fe277400640 1 -- 172.21.15.27:0/1450502175 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe27815f550 msgr2=0x7fe2780af2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.935+0000 7fe277400640 1 --2- 172.21.15.27:0/1450502175 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe27815f550 0x7fe2780af2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.935+0000 7fe277400640 1 -- 172.21.15.27:0/1450502175 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe278158900 msgr2=0x7fe2780aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.935+0000 7fe277400640 1 --2- 172.21.15.27:0/1450502175 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe278158900 0x7fe2780aeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.935+0000 7fe277400640 1 -- 172.21.15.27:0/1450502175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe270009a70 con 0x7fe2780cf110 2024-01-15T18:12:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.935+0000 7fe277400640 1 --2- 172.21.15.27:0/1450502175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2780cf110 0x7fe278150200 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fe26800c480 tx=0x7fe26800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.935+0000 7fe26f400640 1 -- 172.21.15.27:0/1450502175 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe268005cf0 con 0x7fe2780cf110 2024-01-15T18:12:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.935+0000 7fe26f400640 1 -- 172.21.15.27:0/1450502175 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe268005e50 con 0x7fe2780cf110 2024-01-15T18:12:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.935+0000 7fe26f400640 1 -- 172.21.15.27:0/1450502175 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe268007290 con 0x7fe2780cf110 2024-01-15T18:12:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.935+0000 7fe27e600640 1 -- 172.21.15.27:0/1450502175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe27815be20 con 0x7fe2780cf110 2024-01-15T18:12:15.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.935+0000 7fe27e600640 1 -- 172.21.15.27:0/1450502175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe27815c3e0 con 0x7fe2780cf110 2024-01-15T18:12:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.939+0000 7fe27e600640 1 -- 172.21.15.27:0/1450502175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe27815c900 con 0x7fe2780cf110 2024-01-15T18:12:15.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.939+0000 7fe26f400640 1 -- 172.21.15.27:0/1450502175 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7fe2680073f0 con 0x7fe2780cf110 2024-01-15T18:12:15.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.939+0000 7fe26f400640 1 --2- 172.21.15.27:0/1450502175 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe2500670d0 0x7fe250069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:15.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.939+0000 7fe26f400640 1 -- 172.21.15.27:0/1450502175 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fe2680820d0 con 0x7fe2780cf110 2024-01-15T18:12:15.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.939+0000 7fe276a00640 1 --2- 172.21.15.27:0/1450502175 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe2500670d0 0x7fe250069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:15.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.939+0000 7fe276a00640 1 --2- 172.21.15.27:0/1450502175 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe2500670d0 0x7fe250069590 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe2600079c0 tx=0x7fe260008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:15.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:15.943+0000 7fe26f400640 1 -- 172.21.15.27:0/1450502175 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe268050fc0 con 0x7fe2780cf110 2024-01-15T18:12:16.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:16.139+0000 7fe27e600640 1 -- 172.21.15.27:0/1450502175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe278073c00 con 0x7fe2780cf110 2024-01-15T18:12:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.083+0000 7fe26f400640 1 -- 172.21.15.27:0/1450502175 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v48) v1 ==== 136+0+0 (secure 0 0 0) 0x7fe278073c00 con 0x7fe2780cf110 2024-01-15T18:12:17.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.087+0000 7fe27e600640 1 -- 172.21.15.27:0/1450502175 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe2500670d0 msgr2=0x7fe250069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.087+0000 7fe27e600640 1 --2- 172.21.15.27:0/1450502175 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe2500670d0 0x7fe250069590 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe2600079c0 tx=0x7fe260008040 comp rx=0 tx=0).stop 2024-01-15T18:12:17.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.087+0000 7fe27e600640 1 -- 172.21.15.27:0/1450502175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2780cf110 msgr2=0x7fe278150200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.087+0000 7fe27e600640 1 --2- 172.21.15.27:0/1450502175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2780cf110 0x7fe278150200 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fe26800c480 tx=0x7fe26800c940 comp rx=0 tx=0).stop 2024-01-15T18:12:17.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.087+0000 7fe27e600640 1 -- 172.21.15.27:0/1450502175 shutdown_connections 2024-01-15T18:12:17.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.087+0000 7fe27e600640 1 --2- 172.21.15.27:0/1450502175 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe2500670d0 0x7fe250069590 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.087+0000 7fe27e600640 1 --2- 172.21.15.27:0/1450502175 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe27815f550 0x7fe2780af2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.087+0000 7fe27e600640 1 --2- 172.21.15.27:0/1450502175 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe278158900 0x7fe2780aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.087+0000 7fe27e600640 1 --2- 172.21.15.27:0/1450502175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2780cf110 0x7fe278150200 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.087+0000 7fe27e600640 1 -- 172.21.15.27:0/1450502175 >> 172.21.15.27:0/1450502175 conn(0x7fe2780cb700 msgr2=0x7fe2780b0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:17.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.087+0000 7fe27e600640 1 -- 172.21.15.27:0/1450502175 shutdown_connections 2024-01-15T18:12:17.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.087+0000 7fe27e600640 1 -- 172.21.15.27:0/1450502175 wait complete. 2024-01-15T18:12:17.113 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:12:17.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.259+0000 7f4ea0800640 1 Processor -- start 2024-01-15T18:12:17.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.259+0000 7f4ea0800640 1 -- start start 2024-01-15T18:12:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.259+0000 7f4ea0800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f4e9809a120 0x7f4e980cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.259+0000 7f4ea0800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e9809a700 con 0x7f4e980b35a0 2024-01-15T18:12:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.259+0000 7f4ea0800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e980cd590 con 0x7f4e9815e7d0 2024-01-15T18:12:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.259+0000 7f4ea0800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e980cd870 con 0x7f4e9809a120 2024-01-15T18:12:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.259+0000 7f4e9e800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4e980b35a0 0x7f4e980cc530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38314/0 (socket says 172.21.15.27:38314) 2024-01-15T18:12:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.259+0000 7f4e9e800640 1 -- 172.21.15.27:0/3920572932 learned_addr learned my addr 172.21.15.27:0/3920572932 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:12:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.259+0000 7f4e9d400640 1 --2- 172.21.15.27:0/3920572932 >> v2:172.21.15.93:3300/0 conn(0x7f4e9809a120 0x7f4e980cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.259+0000 7f4e9ca00640 1 -- 172.21.15.27:0/3920572932 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3964109397 0 0) 0x7f4e9809a700 con 0x7f4e980b35a0 2024-01-15T18:12:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.259+0000 7f4e9ca00640 1 -- 172.21.15.27:0/3920572932 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e78003530 con 0x7f4e980b35a0 2024-01-15T18:12:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.259+0000 7f4e9ca00640 1 -- 172.21.15.27:0/3920572932 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3088011369 0 0) 0x7f4e980cd590 con 0x7f4e9815e7d0 2024-01-15T18:12:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.259+0000 7f4e9ca00640 1 -- 172.21.15.27:0/3920572932 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e9809a700 con 0x7f4e9815e7d0 2024-01-15T18:12:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.259+0000 7f4e9ca00640 1 -- 172.21.15.27:0/3920572932 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2834866207 0 0) 0x7f4e78003530 con 0x7f4e980b35a0 2024-01-15T18:12:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.259+0000 7f4e9ca00640 1 -- 172.21.15.27:0/3920572932 >> v2:172.21.15.93:3300/0 conn(0x7f4e9809a120 msgr2=0x7f4e980cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:17.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.259+0000 7f4e9ca00640 1 --2- 172.21.15.27:0/3920572932 >> v2:172.21.15.93:3300/0 conn(0x7f4e9809a120 0x7f4e980cbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.259+0000 7f4e9ca00640 1 -- 172.21.15.27:0/3920572932 >> v1:172.21.15.93:6789/0 conn(0x7f4e9815e7d0 legacy=0x7f4e98099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.259+0000 7f4e9ca00640 1 -- 172.21.15.27:0/3920572932 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e980cfa00 con 0x7f4e980b35a0 2024-01-15T18:12:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.259+0000 7f4e9ca00640 1 -- 172.21.15.27:0/3920572932 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f4e90001a00 con 0x7f4e980b35a0 2024-01-15T18:12:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.259+0000 7f4e9d400640 1 --2- 172.21.15.27:0/3920572932 >> v2:172.21.15.93:3300/0 conn(0x7f4e9809a120 0x7f4e980cbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:12:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.263+0000 7f4e9ca00640 1 -- 172.21.15.27:0/3920572932 >> v1:172.21.15.27:6789/0 conn(0x7f4e980b35a0 legacy=0x7f4e980cc530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.263+0000 7f4e9ca00640 1 --2- 172.21.15.27:0/3920572932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e78003b80 0x7f4e78003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.263+0000 7f4e9ca00640 1 --2- 172.21.15.27:0/3920572932 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e780047d0 0x7f4e78004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.263+0000 7f4e9ca00640 1 --2- 172.21.15.27:0/3920572932 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e78005190 0x7f4e7800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.263+0000 7f4e9ca00640 1 -- 172.21.15.27:0/3920572932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e980cd870 con 0x7f4e78003b80 2024-01-15T18:12:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.263+0000 7f4e9ca00640 1 -- 172.21.15.27:0/3920572932 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7800c060 con 0x7f4e780047d0 2024-01-15T18:12:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.263+0000 7f4e9ca00640 1 -- 172.21.15.27:0/3920572932 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7800c220 con 0x7f4e78005190 2024-01-15T18:12:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.263+0000 7f4e9de00640 1 --2- 172.21.15.27:0/3920572932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e78003b80 0x7f4e78003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.263+0000 7f4e9e800640 1 --2- 172.21.15.27:0/3920572932 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e78005190 0x7f4e7800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.263+0000 7f4e9d400640 1 --2- 172.21.15.27:0/3920572932 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e780047d0 0x7f4e78004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.263+0000 7f4e9de00640 1 -- 172.21.15.27:0/3920572932 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e78005190 msgr2=0x7f4e7800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:17.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.263+0000 7f4e9de00640 1 --2- 172.21.15.27:0/3920572932 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e78005190 0x7f4e7800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.263+0000 7f4e9de00640 1 -- 172.21.15.27:0/3920572932 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e780047d0 msgr2=0x7f4e78004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:17.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.263+0000 7f4e9de00640 1 --2- 172.21.15.27:0/3920572932 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e780047d0 0x7f4e78004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.263+0000 7f4e9de00640 1 -- 172.21.15.27:0/3920572932 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e78003530 con 0x7f4e78003b80 2024-01-15T18:12:17.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.263+0000 7f4e9e800640 1 -- 172.21.15.27:0/3920572932 reap_dead start 2024-01-15T18:12:17.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.263+0000 7f4e9de00640 1 --2- 172.21.15.27:0/3920572932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e78003b80 0x7f4e78003f60 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f4e880018d0 tx=0x7f4e8801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c59cb60c50781ea8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:17.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.263+0000 7f4e9ca00640 1 -- 172.21.15.27:0/3920572932 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e880052d0 con 0x7f4e78003b80 2024-01-15T18:12:17.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.263+0000 7f4e9ca00640 1 -- 172.21.15.27:0/3920572932 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e880a7050 con 0x7f4e78003b80 2024-01-15T18:12:17.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.263+0000 7f4e9ca00640 1 -- 172.21.15.27:0/3920572932 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e88003600 con 0x7f4e78003b80 2024-01-15T18:12:17.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.267+0000 7f4ea0800640 1 -- 172.21.15.27:0/3920572932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e78003b80 msgr2=0x7f4e78003f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.267+0000 7f4ea0800640 1 --2- 172.21.15.27:0/3920572932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e78003b80 0x7f4e78003f60 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f4e880018d0 tx=0x7f4e8801c4f0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.267+0000 7f4ea0800640 1 -- 172.21.15.27:0/3920572932 shutdown_connections 2024-01-15T18:12:17.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.267+0000 7f4ea0800640 1 --2- 172.21.15.27:0/3920572932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e78003b80 0x7f4e78003f60 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.267+0000 7f4ea0800640 1 -- 172.21.15.27:0/3920572932 >> 172.21.15.27:0/3920572932 conn(0x7f4e980b2a80 msgr2=0x7f4e980b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:17.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.267+0000 7f4ea0800640 1 -- 172.21.15.27:0/3920572932 shutdown_connections 2024-01-15T18:12:17.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.267+0000 7f4ea0800640 1 -- 172.21.15.27:0/3920572932 wait complete. 2024-01-15T18:12:17.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.267+0000 7f4ea0800640 1 Processor -- start 2024-01-15T18:12:17.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.267+0000 7f4ea0800640 1 -- start start 2024-01-15T18:12:17.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.267+0000 7f4ea0800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e78003b80 0x7f4e9814a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.267+0000 7f4ea0800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e980b3da0 0x7f4e980c8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.267+0000 7f4ea0800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e980c8560 0x7f4e980b4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.267+0000 7f4ea0800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9815c980 con 0x7f4e980b3da0 2024-01-15T18:12:17.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.267+0000 7f4ea0800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9814a840 con 0x7f4e78003b80 2024-01-15T18:12:17.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.267+0000 7f4ea0800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e98158d10 con 0x7f4e980c8560 2024-01-15T18:12:17.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.271+0000 7f4e9d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e980b3da0 0x7f4e980c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.271+0000 7f4e9d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e980b3da0 0x7f4e980c8020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52866/0 (socket says 172.21.15.27:52866) 2024-01-15T18:12:17.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.271+0000 7f4e9d400640 1 -- 172.21.15.27:0/2445782578 learned_addr learned my addr 172.21.15.27:0/2445782578 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:17.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.271+0000 7f4e9e800640 1 --2- 172.21.15.27:0/2445782578 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e980c8560 0x7f4e980b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.271+0000 7f4e9de00640 1 --2- 172.21.15.27:0/2445782578 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e78003b80 0x7f4e9814a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.271+0000 7f4e9d400640 1 -- 172.21.15.27:0/2445782578 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e980c8560 msgr2=0x7f4e980b4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.271+0000 7f4e9d400640 1 --2- 172.21.15.27:0/2445782578 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e980c8560 0x7f4e980b4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.271+0000 7f4e9d400640 1 -- 172.21.15.27:0/2445782578 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e78003b80 msgr2=0x7f4e9814a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.271+0000 7f4e9d400640 1 --2- 172.21.15.27:0/2445782578 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e78003b80 0x7f4e9814a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.271+0000 7f4e9d400640 1 -- 172.21.15.27:0/2445782578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e981577d0 con 0x7f4e980b3da0 2024-01-15T18:12:17.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.271+0000 7f4e9d400640 1 --2- 172.21.15.27:0/2445782578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e980b3da0 0x7f4e980c8020 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f4e80011cd0 tx=0x7f4e8000f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:17.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.271+0000 7f4e96000640 1 -- 172.21.15.27:0/2445782578 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8000fd20 con 0x7f4e980b3da0 2024-01-15T18:12:17.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.271+0000 7f4e96000640 1 -- 172.21.15.27:0/2445782578 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e8000fe80 con 0x7f4e980b3da0 2024-01-15T18:12:17.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.271+0000 7f4e96000640 1 -- 172.21.15.27:0/2445782578 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e80004270 con 0x7f4e980b3da0 2024-01-15T18:12:17.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.271+0000 7f4ea0800640 1 -- 172.21.15.27:0/2445782578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e981574f0 con 0x7f4e980b3da0 2024-01-15T18:12:17.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.271+0000 7f4ea0800640 1 -- 172.21.15.27:0/2445782578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e98158a30 con 0x7f4e980b3da0 2024-01-15T18:12:17.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.271+0000 7f4ea0800640 1 -- 172.21.15.27:0/2445782578 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e9804e850 con 0x7f4e980b3da0 2024-01-15T18:12:17.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.271+0000 7f4e96000640 1 -- 172.21.15.27:0/2445782578 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f4e800043d0 con 0x7f4e980b3da0 2024-01-15T18:12:17.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.271+0000 7f4e96000640 1 --2- 172.21.15.27:0/2445782578 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4e700670d0 0x7f4e70069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.271+0000 7f4e96000640 1 -- 172.21.15.27:0/2445782578 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f4e80083300 con 0x7f4e980b3da0 2024-01-15T18:12:17.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.275+0000 7f4e9de00640 1 --2- 172.21.15.27:0/2445782578 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4e700670d0 0x7f4e70069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.275+0000 7f4e9de00640 1 --2- 172.21.15.27:0/2445782578 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4e700670d0 0x7f4e70069590 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f4e880ac000 tx=0x7f4e880a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:17.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.275+0000 7f4e96000640 1 -- 172.21.15.27:0/2445782578 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4e80052160 con 0x7f4e980b3da0 2024-01-15T18:12:17.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.451+0000 7f4ea0800640 1 -- 172.21.15.27:0/2445782578 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e9815ebb0 con 0x7f4e700670d0 2024-01-15T18:12:17.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.455+0000 7f4e96000640 1 -- 172.21.15.27:0/2445782578 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f4e88005330 con 0x7f4e700670d0 2024-01-15T18:12:17.457 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:12:17.457 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:12:17.457 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-15T18:12:17.457 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:12:17.457 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:12:17.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.455+0000 7f4ea0800640 1 -- 172.21.15.27:0/2445782578 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4e700670d0 msgr2=0x7f4e70069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.455+0000 7f4ea0800640 1 --2- 172.21.15.27:0/2445782578 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4e700670d0 0x7f4e70069590 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f4e880ac000 tx=0x7f4e880a6000 comp rx=0 tx=0).stop 2024-01-15T18:12:17.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.455+0000 7f4ea0800640 1 -- 172.21.15.27:0/2445782578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e980b3da0 msgr2=0x7f4e980c8020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.455+0000 7f4ea0800640 1 --2- 172.21.15.27:0/2445782578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e980b3da0 0x7f4e980c8020 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f4e80011cd0 tx=0x7f4e8000f610 comp rx=0 tx=0).stop 2024-01-15T18:12:17.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.455+0000 7f4ea0800640 1 -- 172.21.15.27:0/2445782578 shutdown_connections 2024-01-15T18:12:17.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.455+0000 7f4ea0800640 1 --2- 172.21.15.27:0/2445782578 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4e700670d0 0x7f4e70069590 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.455+0000 7f4ea0800640 1 --2- 172.21.15.27:0/2445782578 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e980c8560 0x7f4e980b4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.455+0000 7f4ea0800640 1 --2- 172.21.15.27:0/2445782578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e980b3da0 0x7f4e980c8020 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.455+0000 7f4ea0800640 1 --2- 172.21.15.27:0/2445782578 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e78003b80 0x7f4e9814a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.455+0000 7f4ea0800640 1 -- 172.21.15.27:0/2445782578 >> 172.21.15.27:0/2445782578 conn(0x7f4e980b2a80 msgr2=0x7f4e980adff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:17.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.455+0000 7f4ea0800640 1 -- 172.21.15.27:0/2445782578 shutdown_connections 2024-01-15T18:12:17.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.455+0000 7f4ea0800640 1 -- 172.21.15.27:0/2445782578 wait complete. 2024-01-15T18:12:17.479 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:12:17.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.623+0000 7fd2ed200640 1 Processor -- start 2024-01-15T18:12:17.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.623+0000 7fd2ed200640 1 -- start start 2024-01-15T18:12:17.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.623+0000 7fd2ed200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd2e815d3b0 0x7fd2e80cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.623+0000 7fd2ed200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd2e806b600 0x7fd2e806bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.623+0000 7fd2ed200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fd2e806c220 0x7fd2e8160f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.623+0000 7fd2ed200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2e806c6b0 con 0x7fd2e806b600 2024-01-15T18:12:17.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.623+0000 7fd2ed200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2e80b04b0 con 0x7fd2e806c220 2024-01-15T18:12:17.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.623+0000 7fd2ed200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e80d0160 con 0x7fd2e815d3b0 2024-01-15T18:12:17.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.623+0000 7fd2e5400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd2e806b600 0x7fd2e806bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.623+0000 7fd2e5400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd2e806b600 0x7fd2e806bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52870/0 (socket says 172.21.15.27:52870) 2024-01-15T18:12:17.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.623+0000 7fd2e5400640 1 -- 172.21.15.27:0/324829583 learned_addr learned my addr 172.21.15.27:0/324829583 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:17.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.623+0000 7fd2e5e00640 1 --2- 172.21.15.27:0/324829583 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd2e815d3b0 0x7fd2e80cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.623+0000 7fd2e6800640 1 --2- 172.21.15.27:0/324829583 >> v2:172.21.15.93:3300/0 conn(0x7fd2e806c220 0x7fd2e8160f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.623+0000 7fd2e5400640 1 -- 172.21.15.27:0/324829583 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd2e815d3b0 msgr2=0x7fd2e80cfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:17.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.623+0000 7fd2e5400640 1 --2- 172.21.15.27:0/324829583 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd2e815d3b0 0x7fd2e80cfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.623+0000 7fd2e5400640 1 -- 172.21.15.27:0/324829583 >> v2:172.21.15.93:3300/0 conn(0x7fd2e806c220 msgr2=0x7fd2e8160f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:17.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.623+0000 7fd2e5400640 1 --2- 172.21.15.27:0/324829583 >> v2:172.21.15.93:3300/0 conn(0x7fd2e806c220 0x7fd2e8160f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.623+0000 7fd2e5400640 1 -- 172.21.15.27:0/324829583 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e80b0720 con 0x7fd2e806b600 2024-01-15T18:12:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.627+0000 7fd2e5400640 1 --2- 172.21.15.27:0/324829583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2e806b600 0x7fd2e806bce0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0009870 tx=0x7fd2d001c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75bfec0a3f455b90 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.627+0000 7fd2e4a00640 1 -- 172.21.15.27:0/324829583 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d00ab600 con 0x7fd2e806b600 2024-01-15T18:12:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.627+0000 7fd2e4a00640 1 -- 172.21.15.27:0/324829583 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d00a7050 con 0x7fd2e806b600 2024-01-15T18:12:17.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.627+0000 7fd2e4a00640 1 -- 172.21.15.27:0/324829583 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d00ab980 con 0x7fd2e806b600 2024-01-15T18:12:17.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.627+0000 7fd2ed200640 1 -- 172.21.15.27:0/324829583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2e806b600 msgr2=0x7fd2e806bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.627+0000 7fd2ed200640 1 --2- 172.21.15.27:0/324829583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2e806b600 0x7fd2e806bce0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0009870 tx=0x7fd2d001c210 comp rx=0 tx=0).stop 2024-01-15T18:12:17.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.627+0000 7fd2ed200640 1 -- 172.21.15.27:0/324829583 shutdown_connections 2024-01-15T18:12:17.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.627+0000 7fd2ed200640 1 --2- 172.21.15.27:0/324829583 >> v2:172.21.15.93:3300/0 conn(0x7fd2e806c220 0x7fd2e8160f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.627+0000 7fd2ed200640 1 --2- 172.21.15.27:0/324829583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2e806b600 0x7fd2e806bce0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.627+0000 7fd2ed200640 1 --2- 172.21.15.27:0/324829583 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd2e815d3b0 0x7fd2e80cfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.627+0000 7fd2ed200640 1 -- 172.21.15.27:0/324829583 >> 172.21.15.27:0/324829583 conn(0x7fd2e80d0ed0 msgr2=0x7fd2e80d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.627+0000 7fd2ed200640 1 -- 172.21.15.27:0/324829583 shutdown_connections 2024-01-15T18:12:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.627+0000 7fd2ed200640 1 -- 172.21.15.27:0/324829583 wait complete. 2024-01-15T18:12:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.631+0000 7fd2ed200640 1 Processor -- start 2024-01-15T18:12:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.631+0000 7fd2ed200640 1 -- start start 2024-01-15T18:12:17.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.631+0000 7fd2ed200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd2e806b600 0x7fd2e815f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.631+0000 7fd2ed200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd2e806c220 0x7fd2e81608a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.631+0000 7fd2ed200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2e815d3b0 0x7fd2e815e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.631+0000 7fd2ed200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e815f800 con 0x7fd2e815d3b0 2024-01-15T18:12:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.631+0000 7fd2ed200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e815fae0 con 0x7fd2e806b600 2024-01-15T18:12:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.631+0000 7fd2ed200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e815fdc0 con 0x7fd2e806c220 2024-01-15T18:12:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.631+0000 7fd2e6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2e815d3b0 0x7fd2e815e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.631+0000 7fd2e6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2e815d3b0 0x7fd2e815e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52878/0 (socket says 172.21.15.27:52878) 2024-01-15T18:12:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.631+0000 7fd2e6800640 1 -- 172.21.15.27:0/2598417946 learned_addr learned my addr 172.21.15.27:0/2598417946 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.631+0000 7fd2e5400640 1 --2- 172.21.15.27:0/2598417946 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd2e806c220 0x7fd2e81608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.631+0000 7fd2e5e00640 1 --2- 172.21.15.27:0/2598417946 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd2e806b600 0x7fd2e815f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.631+0000 7fd2e6800640 1 -- 172.21.15.27:0/2598417946 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd2e806c220 msgr2=0x7fd2e81608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.631+0000 7fd2e6800640 1 --2- 172.21.15.27:0/2598417946 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd2e806c220 0x7fd2e81608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.631+0000 7fd2e6800640 1 -- 172.21.15.27:0/2598417946 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd2e806b600 msgr2=0x7fd2e815f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.631+0000 7fd2e6800640 1 --2- 172.21.15.27:0/2598417946 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd2e806b600 0x7fd2e815f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.631+0000 7fd2e6800640 1 -- 172.21.15.27:0/2598417946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d00094d0 con 0x7fd2e815d3b0 2024-01-15T18:12:17.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.631+0000 7fd2e6800640 1 --2- 172.21.15.27:0/2598417946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2e815d3b0 0x7fd2e815e3d0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0007e60 tx=0x7fd2e000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:17.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.631+0000 7fd2de000640 1 -- 172.21.15.27:0/2598417946 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0014620 con 0x7fd2e815d3b0 2024-01-15T18:12:17.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.631+0000 7fd2de000640 1 -- 172.21.15.27:0/2598417946 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e000ee90 con 0x7fd2e815d3b0 2024-01-15T18:12:17.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.631+0000 7fd2de000640 1 -- 172.21.15.27:0/2598417946 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e00149e0 con 0x7fd2e815d3b0 2024-01-15T18:12:17.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.631+0000 7fd2ed200640 1 -- 172.21.15.27:0/2598417946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e81600a0 con 0x7fd2e815d3b0 2024-01-15T18:12:17.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.631+0000 7fd2ed200640 1 -- 172.21.15.27:0/2598417946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e815e910 con 0x7fd2e815d3b0 2024-01-15T18:12:17.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.635+0000 7fd2ed200640 1 -- 172.21.15.27:0/2598417946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e806d550 con 0x7fd2e815d3b0 2024-01-15T18:12:17.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.635+0000 7fd2de000640 1 -- 172.21.15.27:0/2598417946 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7fd2e0014b40 con 0x7fd2e815d3b0 2024-01-15T18:12:17.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.635+0000 7fd2de000640 1 --2- 172.21.15.27:0/2598417946 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd2c00670d0 0x7fd2c0069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.635+0000 7fd2de000640 1 -- 172.21.15.27:0/2598417946 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fd2e00832f0 con 0x7fd2e815d3b0 2024-01-15T18:12:17.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.635+0000 7fd2e5e00640 1 --2- 172.21.15.27:0/2598417946 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd2c00670d0 0x7fd2c0069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.635+0000 7fd2e5e00640 1 --2- 172.21.15.27:0/2598417946 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd2c00670d0 0x7fd2c0069590 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80079c0 tx=0x7fd2d8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:17.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.639+0000 7fd2de000640 1 -- 172.21.15.27:0/2598417946 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd2e00521e0 con 0x7fd2e815d3b0 2024-01-15T18:12:17.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.795+0000 7fd2ed200640 1 -- 172.21.15.27:0/2598417946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd2e804e850 con 0x7fd2e815d3b0 2024-01-15T18:12:17.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.795+0000 7fd2de000640 1 -- 172.21.15.27:0/2598417946 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd2e0055800 con 0x7fd2e815d3b0 2024-01-15T18:12:17.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.799+0000 7fd2ed200640 1 -- 172.21.15.27:0/2598417946 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd2c00670d0 msgr2=0x7fd2c0069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.799+0000 7fd2ed200640 1 --2- 172.21.15.27:0/2598417946 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd2c00670d0 0x7fd2c0069590 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80079c0 tx=0x7fd2d8008040 comp rx=0 tx=0).stop 2024-01-15T18:12:17.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.799+0000 7fd2ed200640 1 -- 172.21.15.27:0/2598417946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2e815d3b0 msgr2=0x7fd2e815e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.799+0000 7fd2ed200640 1 --2- 172.21.15.27:0/2598417946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2e815d3b0 0x7fd2e815e3d0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0007e60 tx=0x7fd2e000e4b0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.799+0000 7fd2ed200640 1 -- 172.21.15.27:0/2598417946 shutdown_connections 2024-01-15T18:12:17.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.799+0000 7fd2ed200640 1 --2- 172.21.15.27:0/2598417946 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd2c00670d0 0x7fd2c0069590 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.799+0000 7fd2ed200640 1 --2- 172.21.15.27:0/2598417946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2e815d3b0 0x7fd2e815e3d0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.799+0000 7fd2ed200640 1 --2- 172.21.15.27:0/2598417946 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd2e806c220 0x7fd2e81608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.799+0000 7fd2ed200640 1 --2- 172.21.15.27:0/2598417946 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd2e806b600 0x7fd2e815f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.799+0000 7fd2ed200640 1 -- 172.21.15.27:0/2598417946 >> 172.21.15.27:0/2598417946 conn(0x7fd2e80d0ed0 msgr2=0x7fd2e80aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:17.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.799+0000 7fd2ed200640 1 -- 172.21.15.27:0/2598417946 shutdown_connections 2024-01-15T18:12:17.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.799+0000 7fd2ed200640 1 -- 172.21.15.27:0/2598417946 wait complete. 2024-01-15T18:12:17.821 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-15T18:12:17.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.963+0000 7f0401c00640 1 Processor -- start 2024-01-15T18:12:17.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.963+0000 7f0401c00640 1 -- start start 2024-01-15T18:12:17.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f0401c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f03fc0cf2e0 0x7f03fc0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f0401c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03fc0b1cb0 con 0x7f03fc158900 2024-01-15T18:12:17.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f0401c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03fc15fa00 con 0x7f03fc15f550 2024-01-15T18:12:17.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f0401c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03fc0b1f00 con 0x7f03fc0cf2e0 2024-01-15T18:12:17.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03faa00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f03fc158900 0x7f03fc15bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38324/0 (socket says 172.21.15.27:38324) 2024-01-15T18:12:17.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03faa00640 1 -- 172.21.15.27:0/3201380655 learned_addr learned my addr 172.21.15.27:0/3201380655 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:12:17.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03fb400640 1 --2- 172.21.15.27:0/3201380655 >> v2:172.21.15.93:3300/0 conn(0x7f03fc0cf2e0 0x7f03fc0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03f9600640 1 -- 172.21.15.27:0/3201380655 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378459270 0 0) 0x7f03fc0b1cb0 con 0x7f03fc158900 2024-01-15T18:12:17.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03f9600640 1 -- 172.21.15.27:0/3201380655 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03dc003530 con 0x7f03fc158900 2024-01-15T18:12:17.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03f9600640 1 -- 172.21.15.27:0/3201380655 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 272311572 0 0) 0x7f03dc003530 con 0x7f03fc158900 2024-01-15T18:12:17.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03f9600640 1 -- 172.21.15.27:0/3201380655 >> v2:172.21.15.93:3300/0 conn(0x7f03fc0cf2e0 msgr2=0x7f03fc0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:17.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03f9600640 1 --2- 172.21.15.27:0/3201380655 >> v2:172.21.15.93:3300/0 conn(0x7f03fc0cf2e0 0x7f03fc0701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03f9600640 1 -- 172.21.15.27:0/3201380655 >> v1:172.21.15.93:6789/0 conn(0x7f03fc15f550 legacy=0x7f03fc0cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03f9600640 1 -- 172.21.15.27:0/3201380655 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03fc0b21e0 con 0x7f03fc158900 2024-01-15T18:12:17.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03f9600640 1 -- 172.21.15.27:0/3201380655 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f03ec001680 con 0x7f03fc158900 2024-01-15T18:12:17.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03f9600640 1 -- 172.21.15.27:0/3201380655 >> v1:172.21.15.27:6789/0 conn(0x7f03fc158900 legacy=0x7f03fc15bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03f9600640 1 --2- 172.21.15.27:0/3201380655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03dc0039f0 0x7f03dc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03f9600640 1 --2- 172.21.15.27:0/3201380655 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03dc0047d0 0x7f03dc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03f9600640 1 --2- 172.21.15.27:0/3201380655 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03dc005190 0x7f03dc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03f9600640 1 -- 172.21.15.27:0/3201380655 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03fc0b1f00 con 0x7f03dc0039f0 2024-01-15T18:12:17.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03f9600640 1 -- 172.21.15.27:0/3201380655 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc00c060 con 0x7f03dc005190 2024-01-15T18:12:17.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03f9600640 1 -- 172.21.15.27:0/3201380655 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03dc00c220 con 0x7f03dc0047d0 2024-01-15T18:12:17.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03faa00640 1 --2- 172.21.15.27:0/3201380655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03dc0039f0 0x7f03dc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03fa000640 1 --2- 172.21.15.27:0/3201380655 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03dc0047d0 0x7f03dc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03fb400640 1 --2- 172.21.15.27:0/3201380655 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03dc005190 0x7f03dc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03faa00640 1 -- 172.21.15.27:0/3201380655 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03dc0047d0 msgr2=0x7f03dc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:17.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03faa00640 1 --2- 172.21.15.27:0/3201380655 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03dc0047d0 0x7f03dc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03faa00640 1 -- 172.21.15.27:0/3201380655 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03dc005190 msgr2=0x7f03dc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:17.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03faa00640 1 --2- 172.21.15.27:0/3201380655 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03dc005190 0x7f03dc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03faa00640 1 -- 172.21.15.27:0/3201380655 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03dc003530 con 0x7f03dc0039f0 2024-01-15T18:12:17.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.967+0000 7f03fb400640 1 -- 172.21.15.27:0/3201380655 reap_dead start 2024-01-15T18:12:17.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.971+0000 7f03faa00640 1 --2- 172.21.15.27:0/3201380655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03dc0039f0 0x7f03dc003dd0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f03ec00c190 tx=0x7f03ec01cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=779854848549feb8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:17.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.971+0000 7f03f9600640 1 -- 172.21.15.27:0/3201380655 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec005ae0 con 0x7f03dc0039f0 2024-01-15T18:12:17.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.971+0000 7f03f9600640 1 -- 172.21.15.27:0/3201380655 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03ec005c40 con 0x7f03dc0039f0 2024-01-15T18:12:17.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.971+0000 7f03f9600640 1 -- 172.21.15.27:0/3201380655 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec007070 con 0x7f03dc0039f0 2024-01-15T18:12:17.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.971+0000 7f0401c00640 1 -- 172.21.15.27:0/3201380655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03dc0039f0 msgr2=0x7f03dc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.971+0000 7f0401c00640 1 --2- 172.21.15.27:0/3201380655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03dc0039f0 0x7f03dc003dd0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f03ec00c190 tx=0x7f03ec01cb70 comp rx=0 tx=0).stop 2024-01-15T18:12:17.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.971+0000 7f0401c00640 1 -- 172.21.15.27:0/3201380655 shutdown_connections 2024-01-15T18:12:17.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.971+0000 7f0401c00640 1 --2- 172.21.15.27:0/3201380655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03dc0039f0 0x7f03dc003dd0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.971+0000 7f0401c00640 1 -- 172.21.15.27:0/3201380655 >> 172.21.15.27:0/3201380655 conn(0x7f03fc0cb700 msgr2=0x7f03fc0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:17.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.971+0000 7f0401c00640 1 -- 172.21.15.27:0/3201380655 shutdown_connections 2024-01-15T18:12:17.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.971+0000 7f0401c00640 1 -- 172.21.15.27:0/3201380655 wait complete. 2024-01-15T18:12:17.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.971+0000 7f0401c00640 1 Processor -- start 2024-01-15T18:12:17.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.975+0000 7f0401c00640 1 -- start start 2024-01-15T18:12:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.975+0000 7f0401c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03dc0039f0 0x7f03fc1f56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.975+0000 7f0401c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03fc1f5bf0 0x7f03fc1f70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.975+0000 7f0401c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03fc1f7610 0x7f03fc1f7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.975+0000 7f0401c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03fc1f6070 con 0x7f03dc0039f0 2024-01-15T18:12:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.975+0000 7f0401c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03fc1f6ea0 con 0x7f03fc1f5bf0 2024-01-15T18:12:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.975+0000 7f0401c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03fc1f49e0 con 0x7f03fc1f7610 2024-01-15T18:12:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.975+0000 7f03faa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03dc0039f0 0x7f03fc1f56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.975+0000 7f03faa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03dc0039f0 0x7f03fc1f56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52906/0 (socket says 172.21.15.27:52906) 2024-01-15T18:12:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.975+0000 7f03faa00640 1 -- 172.21.15.27:0/430648545 learned_addr learned my addr 172.21.15.27:0/430648545 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.975+0000 7f03fa000640 1 --2- 172.21.15.27:0/430648545 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03fc1f5bf0 0x7f03fc1f70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.975+0000 7f03fb400640 1 --2- 172.21.15.27:0/430648545 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03fc1f7610 0x7f03fc1f7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.975+0000 7f03faa00640 1 -- 172.21.15.27:0/430648545 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03fc1f7610 msgr2=0x7f03fc1f7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.975+0000 7f03faa00640 1 --2- 172.21.15.27:0/430648545 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03fc1f7610 0x7f03fc1f7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.975+0000 7f03faa00640 1 -- 172.21.15.27:0/430648545 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03fc1f5bf0 msgr2=0x7f03fc1f70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.975+0000 7f03faa00640 1 --2- 172.21.15.27:0/430648545 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03fc1f5bf0 0x7f03fc1f70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.975+0000 7f03faa00640 1 -- 172.21.15.27:0/430648545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03fc1f6290 con 0x7f03dc0039f0 2024-01-15T18:12:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.975+0000 7f03faa00640 1 --2- 172.21.15.27:0/430648545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03dc0039f0 0x7f03fc1f56b0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f03ec01c5e0 tx=0x7f03dc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:17.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.975+0000 7f03f2a00640 1 -- 172.21.15.27:0/430648545 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec0b4070 con 0x7f03dc0039f0 2024-01-15T18:12:17.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.975+0000 7f03f2a00640 1 -- 172.21.15.27:0/430648545 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03ec0015c0 con 0x7f03dc0039f0 2024-01-15T18:12:17.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.975+0000 7f0401c00640 1 -- 172.21.15.27:0/430648545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03fc1f6570 con 0x7f03dc0039f0 2024-01-15T18:12:17.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.975+0000 7f03f2a00640 1 -- 172.21.15.27:0/430648545 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec007290 con 0x7f03dc0039f0 2024-01-15T18:12:17.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.975+0000 7f0401c00640 1 -- 172.21.15.27:0/430648545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03fc1f6b30 con 0x7f03dc0039f0 2024-01-15T18:12:17.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.975+0000 7f0401c00640 1 -- 172.21.15.27:0/430648545 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03fc073bd0 con 0x7f03dc0039f0 2024-01-15T18:12:17.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.979+0000 7f03f2a00640 1 -- 172.21.15.27:0/430648545 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f03ec0073f0 con 0x7f03dc0039f0 2024-01-15T18:12:17.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.979+0000 7f03f2a00640 1 --2- 172.21.15.27:0/430648545 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03d4067100 0x7f03d40695c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.979+0000 7f03f2a00640 1 -- 172.21.15.27:0/430648545 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f03ec11c420 con 0x7f03dc0039f0 2024-01-15T18:12:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.979+0000 7f03fa000640 1 --2- 172.21.15.27:0/430648545 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03d4067100 0x7f03d40695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.979+0000 7f03fa000640 1 --2- 172.21.15.27:0/430648545 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03d4067100 0x7f03d40695c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f03fc15d710 tx=0x7f03e4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:17.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:17.983+0000 7f03f2a00640 1 -- 172.21.15.27:0/430648545 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f03ec0eb310 con 0x7f03dc0039f0 2024-01-15T18:12:18.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.163+0000 7f0401c00640 1 -- 172.21.15.27:0/430648545 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f03fc15c310 con 0x7f03d4067100 2024-01-15T18:12:18.172 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:12:18.171+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:18.172 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:12:18.171+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:18.172 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:12:18.171+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:18.172 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:12:18.171+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:18.172 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:12:18.171+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:18.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.207+0000 7f03f2a00640 1 -- 172.21.15.27:0/430648545 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f03e4005020 con 0x7f03d4067100 2024-01-15T18:12:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.207+0000 7f0401c00640 1 -- 172.21.15.27:0/430648545 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03d4067100 msgr2=0x7f03d40695c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.207+0000 7f0401c00640 1 --2- 172.21.15.27:0/430648545 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03d4067100 0x7f03d40695c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f03fc15d710 tx=0x7f03e4009040 comp rx=0 tx=0).stop 2024-01-15T18:12:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.207+0000 7f0401c00640 1 -- 172.21.15.27:0/430648545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03dc0039f0 msgr2=0x7f03fc1f56b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.207+0000 7f0401c00640 1 --2- 172.21.15.27:0/430648545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03dc0039f0 0x7f03fc1f56b0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f03ec01c5e0 tx=0x7f03dc005800 comp rx=0 tx=0).stop 2024-01-15T18:12:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.207+0000 7f0401c00640 1 -- 172.21.15.27:0/430648545 shutdown_connections 2024-01-15T18:12:18.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.207+0000 7f0401c00640 1 --2- 172.21.15.27:0/430648545 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03d4067100 0x7f03d40695c0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.207+0000 7f0401c00640 1 --2- 172.21.15.27:0/430648545 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03fc1f7610 0x7f03fc1f7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.207+0000 7f0401c00640 1 --2- 172.21.15.27:0/430648545 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03fc1f5bf0 0x7f03fc1f70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.207+0000 7f0401c00640 1 --2- 172.21.15.27:0/430648545 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03dc0039f0 0x7f03fc1f56b0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.207+0000 7f0401c00640 1 -- 172.21.15.27:0/430648545 >> 172.21.15.27:0/430648545 conn(0x7f03fc0cb700 msgr2=0x7f03fc15dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:18.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.211+0000 7f0401c00640 1 -- 172.21.15.27:0/430648545 shutdown_connections 2024-01-15T18:12:18.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.211+0000 7f0401c00640 1 -- 172.21.15.27:0/430648545 wait complete. 2024-01-15T18:12:18.231 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:12:18.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.375+0000 7f17d1600640 1 Processor -- start 2024-01-15T18:12:18.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.375+0000 7f17d1600640 1 -- start start 2024-01-15T18:12:18.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.375+0000 7f17d1600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f17cc15d0d0 0x7f17cc0b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:18.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.375+0000 7f17d1600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f17cc0cff40 0x7f17cc161410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:18.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.375+0000 7f17d1600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17cc0d0400 con 0x7f17cc0cfb20 2024-01-15T18:12:18.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.375+0000 7f17d1600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17cc0721c0 con 0x7f17cc0cff40 2024-01-15T18:12:18.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.375+0000 7f17d1600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17cc0b4f90 con 0x7f17cc15d0d0 2024-01-15T18:12:18.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.375+0000 7f17ca000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f17cc0cfb20 0x7f17cc0b4880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38338/0 (socket says 172.21.15.27:38338) 2024-01-15T18:12:18.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.375+0000 7f17ca000640 1 -- 172.21.15.27:0/3490165376 learned_addr learned my addr 172.21.15.27:0/3490165376 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:12:18.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.375+0000 7f17cb400640 1 --2- 172.21.15.27:0/3490165376 >> v2:172.21.15.27:3300/0 conn(0x7f17cc0cff40 0x7f17cc161410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:18.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.375+0000 7f17caa00640 1 --2- 172.21.15.27:0/3490165376 >> v2:172.21.15.93:3300/0 conn(0x7f17cc15d0d0 0x7f17cc0b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:18.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.375+0000 7f17cb400640 1 -- 172.21.15.27:0/3490165376 >> v2:172.21.15.93:3300/0 conn(0x7f17cc15d0d0 msgr2=0x7f17cc0b2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:18.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.375+0000 7f17cb400640 1 --2- 172.21.15.27:0/3490165376 >> v2:172.21.15.93:3300/0 conn(0x7f17cc15d0d0 0x7f17cc0b2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.375+0000 7f17cb400640 1 -- 172.21.15.27:0/3490165376 >> v1:172.21.15.27:6789/0 conn(0x7f17cc0cfb20 legacy=0x7f17cc0b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:18.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.375+0000 7f17c9600640 1 -- 172.21.15.27:0/3490165376 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 815616530 0 0) 0x7f17cc0d0400 con 0x7f17cc0cfb20 2024-01-15T18:12:18.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.375+0000 7f17cb400640 1 -- 172.21.15.27:0/3490165376 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17cc071940 con 0x7f17cc0cff40 2024-01-15T18:12:18.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.379+0000 7f17cb400640 1 --2- 172.21.15.27:0/3490165376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17cc0cff40 0x7f17cc161410 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f17cc071b30 tx=0x7f17c401c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f971ee67e7f1f411 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:18.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.379+0000 7f17c9600640 1 -- 172.21.15.27:0/3490165376 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c40a5910 con 0x7f17cc0cff40 2024-01-15T18:12:18.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.379+0000 7f17c9600640 1 -- 172.21.15.27:0/3490165376 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c40a5a70 con 0x7f17cc0cff40 2024-01-15T18:12:18.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.379+0000 7f17c9600640 1 -- 172.21.15.27:0/3490165376 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c40a5e00 con 0x7f17cc0cff40 2024-01-15T18:12:18.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.379+0000 7f17d1600640 1 -- 172.21.15.27:0/3490165376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17cc0cff40 msgr2=0x7f17cc161410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:18.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.379+0000 7f17d1600640 1 --2- 172.21.15.27:0/3490165376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17cc0cff40 0x7f17cc161410 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f17cc071b30 tx=0x7f17c401c850 comp rx=0 tx=0).stop 2024-01-15T18:12:18.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.379+0000 7f17d1600640 1 -- 172.21.15.27:0/3490165376 shutdown_connections 2024-01-15T18:12:18.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.379+0000 7f17d1600640 1 --2- 172.21.15.27:0/3490165376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17cc0cff40 0x7f17cc161410 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.379+0000 7f17d1600640 1 --2- 172.21.15.27:0/3490165376 >> v2:172.21.15.93:3300/0 conn(0x7f17cc15d0d0 0x7f17cc0b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.379+0000 7f17d1600640 1 -- 172.21.15.27:0/3490165376 >> 172.21.15.27:0/3490165376 conn(0x7f17cc06b600 msgr2=0x7f17cc06ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:18.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.379+0000 7f17d1600640 1 -- 172.21.15.27:0/3490165376 shutdown_connections 2024-01-15T18:12:18.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.379+0000 7f17d1600640 1 -- 172.21.15.27:0/3490165376 wait complete. 2024-01-15T18:12:18.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.383+0000 7f17d1600640 1 Processor -- start 2024-01-15T18:12:18.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.383+0000 7f17d1600640 1 -- start start 2024-01-15T18:12:18.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.383+0000 7f17d1600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17cc0cfb20 0x7f17cc1438a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:18.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.383+0000 7f17d1600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17cc0cff40 0x7f17cc1555b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:18.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.383+0000 7f17d1600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17cc15d0d0 0x7f17cc156300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:18.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.383+0000 7f17d1600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc155fe0 con 0x7f17cc0cfb20 2024-01-15T18:12:18.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.383+0000 7f17d1600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc143de0 con 0x7f17cc0cff40 2024-01-15T18:12:18.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.383+0000 7f17d1600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc15b610 con 0x7f17cc15d0d0 2024-01-15T18:12:18.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.383+0000 7f17caa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17cc0cfb20 0x7f17cc1438a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:18.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.383+0000 7f17caa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17cc0cfb20 0x7f17cc1438a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52914/0 (socket says 172.21.15.27:52914) 2024-01-15T18:12:18.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.383+0000 7f17caa00640 1 -- 172.21.15.27:0/2059112682 learned_addr learned my addr 172.21.15.27:0/2059112682 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:18.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.383+0000 7f17ca000640 1 --2- 172.21.15.27:0/2059112682 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17cc0cff40 0x7f17cc1555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:18.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.383+0000 7f17cb400640 1 --2- 172.21.15.27:0/2059112682 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17cc15d0d0 0x7f17cc156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:18.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.383+0000 7f17caa00640 1 -- 172.21.15.27:0/2059112682 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17cc15d0d0 msgr2=0x7f17cc156300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:18.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.383+0000 7f17caa00640 1 --2- 172.21.15.27:0/2059112682 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17cc15d0d0 0x7f17cc156300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.383+0000 7f17caa00640 1 -- 172.21.15.27:0/2059112682 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17cc0cff40 msgr2=0x7f17cc1555b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:18.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.383+0000 7f17caa00640 1 --2- 172.21.15.27:0/2059112682 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17cc0cff40 0x7f17cc1555b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.383+0000 7f17caa00640 1 -- 172.21.15.27:0/2059112682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c4009a70 con 0x7f17cc0cfb20 2024-01-15T18:12:18.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.383+0000 7f17caa00640 1 --2- 172.21.15.27:0/2059112682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17cc0cfb20 0x7f17cc1438a0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f17bc00d830 tx=0x7f17bc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:18.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.383+0000 7f17c2a00640 1 -- 172.21.15.27:0/2059112682 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc00b6c0 con 0x7f17cc0cfb20 2024-01-15T18:12:18.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.383+0000 7f17c2a00640 1 -- 172.21.15.27:0/2059112682 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17bc00b820 con 0x7f17cc0cfb20 2024-01-15T18:12:18.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.383+0000 7f17c2a00640 1 -- 172.21.15.27:0/2059112682 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc00bb90 con 0x7f17cc0cfb20 2024-01-15T18:12:18.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.383+0000 7f17d1600640 1 -- 172.21.15.27:0/2059112682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17cc15ace0 con 0x7f17cc0cfb20 2024-01-15T18:12:18.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.387+0000 7f17d1600640 1 -- 172.21.15.27:0/2059112682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17cc15b2a0 con 0x7f17cc0cfb20 2024-01-15T18:12:18.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.387+0000 7f17d1600640 1 -- 172.21.15.27:0/2059112682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17cc04e850 con 0x7f17cc0cfb20 2024-01-15T18:12:18.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.387+0000 7f17c2a00640 1 -- 172.21.15.27:0/2059112682 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 21) v1 ==== 86874+0+0 (secure 0 0 0) 0x7f17bc01e460 con 0x7f17cc0cfb20 2024-01-15T18:12:18.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.387+0000 7f17c2a00640 1 --2- 172.21.15.27:0/2059112682 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f17a40670d0 0x7f17a4069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:18.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.387+0000 7f17c2a00640 1 -- 172.21.15.27:0/2059112682 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f17bc083380 con 0x7f17cc0cfb20 2024-01-15T18:12:18.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.387+0000 7f17ca000640 1 --2- 172.21.15.27:0/2059112682 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f17a40670d0 0x7f17a4069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:18.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.387+0000 7f17ca000640 1 --2- 172.21.15.27:0/2059112682 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f17a40670d0 0x7f17a4069590 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f17cc1624f0 tx=0x7f17b4007c10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:18.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.391+0000 7f17c2a00640 1 -- 172.21.15.27:0/2059112682 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f17bc052140 con 0x7f17cc0cfb20 2024-01-15T18:12:18.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.575+0000 7f17d1600640 1 -- 172.21.15.27:0/2059112682 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f17cc161950 con 0x7f17a40670d0 2024-01-15T18:12:18.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.579+0000 7f17c2a00640 1 -- 172.21.15.27:0/2059112682 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f17b4007da0 con 0x7f17a40670d0 2024-01-15T18:12:18.583 INFO:teuthology.orchestra.run.smithi027.stdout:/volumes/_nogroup/sv1/10f4c67f-ff21-41d3-a8bc-abca26a3de8e 2024-01-15T18:12:18.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.583+0000 7f17d1600640 1 -- 172.21.15.27:0/2059112682 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f17a40670d0 msgr2=0x7f17a4069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:18.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.583+0000 7f17d1600640 1 --2- 172.21.15.27:0/2059112682 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f17a40670d0 0x7f17a4069590 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f17cc1624f0 tx=0x7f17b4007c10 comp rx=0 tx=0).stop 2024-01-15T18:12:18.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.583+0000 7f17d1600640 1 -- 172.21.15.27:0/2059112682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17cc0cfb20 msgr2=0x7f17cc1438a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:18.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.583+0000 7f17d1600640 1 --2- 172.21.15.27:0/2059112682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17cc0cfb20 0x7f17cc1438a0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f17bc00d830 tx=0x7f17bc00dcf0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.583+0000 7f17d1600640 1 -- 172.21.15.27:0/2059112682 shutdown_connections 2024-01-15T18:12:18.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.583+0000 7f17d1600640 1 --2- 172.21.15.27:0/2059112682 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f17a40670d0 0x7f17a4069590 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.583+0000 7f17d1600640 1 --2- 172.21.15.27:0/2059112682 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17cc15d0d0 0x7f17cc156300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.583+0000 7f17d1600640 1 --2- 172.21.15.27:0/2059112682 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17cc0cff40 0x7f17cc1555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.583+0000 7f17d1600640 1 --2- 172.21.15.27:0/2059112682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17cc0cfb20 0x7f17cc1438a0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.583+0000 7f17d1600640 1 -- 172.21.15.27:0/2059112682 >> 172.21.15.27:0/2059112682 conn(0x7f17cc06b600 msgr2=0x7f17cc1596f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:18.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.583+0000 7f17d1600640 1 -- 172.21.15.27:0/2059112682 shutdown_connections 2024-01-15T18:12:18.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:18.583+0000 7f17d1600640 1 -- 172.21.15.27:0/2059112682 wait complete. 2024-01-15T18:12:18.605 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/10f4c67f-ff21-41d3-a8bc-abca26a3de8e 2024-01-15T18:12:18.605 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:12:18.606 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:12:18.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:18.614 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/10f4c67f-ff21-41d3-a8bc-abca26a3de8e/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:12:25.814 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records in 2024-01-15T18:12:25.814 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records out 2024-01-15T18:12:25.814 INFO:teuthology.orchestra.run.smithi027.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.13633 s, 150 MB/s 2024-01-15T18:12:25.815 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:12:25.815 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:12:25.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:25.824 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/10f4c67f-ff21-41d3-a8bc-abca26a3de8e/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:12:27.095 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:12:27.098 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:12:32.736 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records in 2024-01-15T18:12:32.736 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records out 2024-01-15T18:12:32.736 INFO:teuthology.orchestra.run.smithi027.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.85159 s, 157 MB/s 2024-01-15T18:12:32.736 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:12:32.736 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:12:32.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:32.746 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/10f4c67f-ff21-41d3-a8bc-abca26a3de8e/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:12:39.606 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records in 2024-01-15T18:12:39.606 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records out 2024-01-15T18:12:39.607 INFO:teuthology.orchestra.run.smithi027.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.8074 s, 158 MB/s 2024-01-15T18:12:39.608 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 10 2024-01-15T18:12:39.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.751+0000 7f01c7200640 1 Processor -- start 2024-01-15T18:12:39.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.751+0000 7f01c7200640 1 -- start start 2024-01-15T18:12:39.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.751+0000 7f01c7200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f01c00cf460 0x7f01c00cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.751+0000 7f01c7200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f01c015b4b0 0x7f01c00cba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.751+0000 7f01c7200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01c00cfd80 con 0x7f01c00cec60 2024-01-15T18:12:39.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.751+0000 7f01c7200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01c00d19a0 con 0x7f01c015b4b0 2024-01-15T18:12:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.751+0000 7f01c7200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01c00d1c80 con 0x7f01c00cf460 2024-01-15T18:12:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.751+0000 7f01c5200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f01c00cec60 0x7f01c00cbf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45480/0 (socket says 172.21.15.27:45480) 2024-01-15T18:12:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.751+0000 7f01c5200640 1 -- 172.21.15.27:0/328303639 learned_addr learned my addr 172.21.15.27:0/328303639 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:12:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.751+0000 7f01bfe00640 1 --2- 172.21.15.27:0/328303639 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f01c00cf460 0x7f01c00cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.751+0000 7f01bf400640 1 --2- 172.21.15.27:0/328303639 >> v2:172.21.15.93:3300/0 conn(0x7f01c015b4b0 0x7f01c00cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.751+0000 7f01bea00640 1 -- 172.21.15.27:0/328303639 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1319696807 0 0) 0x7f01c00cfd80 con 0x7f01c00cec60 2024-01-15T18:12:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.751+0000 7f01bea00640 1 -- 172.21.15.27:0/328303639 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01a0003530 con 0x7f01c00cec60 2024-01-15T18:12:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.755+0000 7f01bf400640 1 -- 172.21.15.27:0/328303639 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f01c00cf460 msgr2=0x7f01c00cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:39.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.755+0000 7f01bea00640 1 -- 172.21.15.27:0/328303639 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4147523030 0 0) 0x7f01a0003530 con 0x7f01c00cec60 2024-01-15T18:12:39.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.755+0000 7f01bf400640 1 --2- 172.21.15.27:0/328303639 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f01c00cf460 0x7f01c00cf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.755+0000 7f01bf400640 1 -- 172.21.15.27:0/328303639 >> v1:172.21.15.27:6789/0 conn(0x7f01c00cec60 legacy=0x7f01c00cbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:39.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.755+0000 7f01bf400640 1 -- 172.21.15.27:0/328303639 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c00b07c0 con 0x7f01c015b4b0 2024-01-15T18:12:39.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.755+0000 7f01bf400640 1 --2- 172.21.15.27:0/328303639 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f01c015b4b0 0x7f01c00cba20 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f01a80097c0 tx=0x7f01a801c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14ca54c68bde6bdb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:39.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.755+0000 7f01bea00640 1 -- 172.21.15.27:0/328303639 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b8001a00 con 0x7f01c015b4b0 2024-01-15T18:12:39.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.755+0000 7f01bea00640 1 -- 172.21.15.27:0/328303639 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a80a7050 con 0x7f01c015b4b0 2024-01-15T18:12:39.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.755+0000 7f01bea00640 1 -- 172.21.15.27:0/328303639 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a80ab620 con 0x7f01c015b4b0 2024-01-15T18:12:39.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.755+0000 7f01c7200640 1 -- 172.21.15.27:0/328303639 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f01c015b4b0 msgr2=0x7f01c00cba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:39.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.755+0000 7f01c7200640 1 --2- 172.21.15.27:0/328303639 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f01c015b4b0 0x7f01c00cba20 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f01a80097c0 tx=0x7f01a801c3a0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.759+0000 7f01c7200640 1 -- 172.21.15.27:0/328303639 shutdown_connections 2024-01-15T18:12:39.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.759+0000 7f01c7200640 1 --2- 172.21.15.27:0/328303639 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f01c015b4b0 0x7f01c00cba20 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.759+0000 7f01c7200640 1 --2- 172.21.15.27:0/328303639 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f01c00cf460 0x7f01c00cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.759+0000 7f01c7200640 1 -- 172.21.15.27:0/328303639 >> 172.21.15.27:0/328303639 conn(0x7f01c00d0e80 msgr2=0x7f01c00d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:39.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.759+0000 7f01c7200640 1 -- 172.21.15.27:0/328303639 shutdown_connections 2024-01-15T18:12:39.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.759+0000 7f01c7200640 1 -- 172.21.15.27:0/328303639 wait complete. 2024-01-15T18:12:39.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.759+0000 7f01c7200640 1 Processor -- start 2024-01-15T18:12:39.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.759+0000 7f01c7200640 1 -- start start 2024-01-15T18:12:39.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.759+0000 7f01c7200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f01c00cec60 0x7f01c0149e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.759+0000 7f01c7200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f01c00cf460 0x7f01c00c8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.759+0000 7f01c7200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01c015b4b0 0x7f01c00c8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.759+0000 7f01c7200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c0153070 con 0x7f01c015b4b0 2024-01-15T18:12:39.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.759+0000 7f01c7200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c00c9620 con 0x7f01c00cec60 2024-01-15T18:12:39.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.759+0000 7f01c7200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01c01529b0 con 0x7f01c00cf460 2024-01-15T18:12:39.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.759+0000 7f01c5200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01c015b4b0 0x7f01c00c8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.759+0000 7f01c5200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01c015b4b0 0x7f01c00c8cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36588/0 (socket says 172.21.15.27:36588) 2024-01-15T18:12:39.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.759+0000 7f01c5200640 1 -- 172.21.15.27:0/2413662050 learned_addr learned my addr 172.21.15.27:0/2413662050 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:39.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.759+0000 7f01bfe00640 1 --2- 172.21.15.27:0/2413662050 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f01c00cec60 0x7f01c0149e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.759+0000 7f01bf400640 1 --2- 172.21.15.27:0/2413662050 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f01c00cf460 0x7f01c00c8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.763+0000 7f01c5200640 1 -- 172.21.15.27:0/2413662050 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f01c00cf460 msgr2=0x7f01c00c8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:39.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.763+0000 7f01c5200640 1 --2- 172.21.15.27:0/2413662050 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f01c00cf460 0x7f01c00c8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.763+0000 7f01c5200640 1 -- 172.21.15.27:0/2413662050 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f01c00cec60 msgr2=0x7f01c0149e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:39.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.763+0000 7f01c5200640 1 --2- 172.21.15.27:0/2413662050 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f01c00cec60 0x7f01c0149e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.763+0000 7f01c5200640 1 -- 172.21.15.27:0/2413662050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a8009420 con 0x7f01c015b4b0 2024-01-15T18:12:39.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.763+0000 7f01c5200640 1 --2- 172.21.15.27:0/2413662050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01c015b4b0 0x7f01c00c8cb0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f01b800cc70 tx=0x7f01b800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:39.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.763+0000 7f01b7e00640 1 -- 172.21.15.27:0/2413662050 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b800ce80 con 0x7f01c015b4b0 2024-01-15T18:12:39.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.763+0000 7f01b7e00640 1 -- 172.21.15.27:0/2413662050 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01b8012400 con 0x7f01c015b4b0 2024-01-15T18:12:39.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.763+0000 7f01b7e00640 1 -- 172.21.15.27:0/2413662050 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b80127d0 con 0x7f01c015b4b0 2024-01-15T18:12:39.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.763+0000 7f01c7200640 1 -- 172.21.15.27:0/2413662050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01c0152c90 con 0x7f01c015b4b0 2024-01-15T18:12:39.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.763+0000 7f01c7200640 1 -- 172.21.15.27:0/2413662050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c0149bc0 con 0x7f01c015b4b0 2024-01-15T18:12:39.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.763+0000 7f01c7200640 1 -- 172.21.15.27:0/2413662050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01c01531b0 con 0x7f01c015b4b0 2024-01-15T18:12:39.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.763+0000 7f01b7e00640 1 -- 172.21.15.27:0/2413662050 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 22) v1 ==== 86925+0+0 (secure 0 0 0) 0x7f01b8007070 con 0x7f01c015b4b0 2024-01-15T18:12:39.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.763+0000 7f01b7e00640 1 --2- 172.21.15.27:0/2413662050 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0198067160 0x7f0198069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.763+0000 7f01b7e00640 1 -- 172.21.15.27:0/2413662050 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f01b8082f80 con 0x7f01c015b4b0 2024-01-15T18:12:39.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.767+0000 7f01bfe00640 1 --2- 172.21.15.27:0/2413662050 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0198067160 0x7f0198069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.767+0000 7f01bfe00640 1 --2- 172.21.15.27:0/2413662050 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0198067160 0x7f0198069620 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f01c00b0570 tx=0x7f01b0009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:39.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.771+0000 7f01b7e00640 1 -- 172.21.15.27:0/2413662050 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f01c01531b0 con 0x7f01c015b4b0 2024-01-15T18:12:39.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.927+0000 7f01c7200640 1 -- 172.21.15.27:0/2413662050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f01c01531b0 con 0x7f01c015b4b0 2024-01-15T18:12:39.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.931+0000 7f01b7e00640 1 -- 172.21.15.27:0/2413662050 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v27)=0 v27) v1 ==== 136+0+0 (secure 0 0 0) 0x7f01c01531b0 con 0x7f01c015b4b0 2024-01-15T18:12:39.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.935+0000 7f01c7200640 1 -- 172.21.15.27:0/2413662050 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0198067160 msgr2=0x7f0198069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:39.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.935+0000 7f01c7200640 1 --2- 172.21.15.27:0/2413662050 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0198067160 0x7f0198069620 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f01c00b0570 tx=0x7f01b0009190 comp rx=0 tx=0).stop 2024-01-15T18:12:39.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.935+0000 7f01c7200640 1 -- 172.21.15.27:0/2413662050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01c015b4b0 msgr2=0x7f01c00c8cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:39.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.935+0000 7f01c7200640 1 --2- 172.21.15.27:0/2413662050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01c015b4b0 0x7f01c00c8cb0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f01b800cc70 tx=0x7f01b800a430 comp rx=0 tx=0).stop 2024-01-15T18:12:39.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.935+0000 7f01c7200640 1 -- 172.21.15.27:0/2413662050 shutdown_connections 2024-01-15T18:12:39.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.935+0000 7f01c7200640 1 --2- 172.21.15.27:0/2413662050 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0198067160 0x7f0198069620 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.935+0000 7f01c7200640 1 --2- 172.21.15.27:0/2413662050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01c015b4b0 0x7f01c00c8cb0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.935+0000 7f01c7200640 1 --2- 172.21.15.27:0/2413662050 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f01c00cf460 0x7f01c00c8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.935+0000 7f01c7200640 1 --2- 172.21.15.27:0/2413662050 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f01c00cec60 0x7f01c0149e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.935+0000 7f01c7200640 1 -- 172.21.15.27:0/2413662050 >> 172.21.15.27:0/2413662050 conn(0x7f01c00d0e80 msgr2=0x7f01c00c7450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:39.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.935+0000 7f01c7200640 1 -- 172.21.15.27:0/2413662050 shutdown_connections 2024-01-15T18:12:39.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:39.935+0000 7f01c7200640 1 -- 172.21.15.27:0/2413662050 wait complete. 2024-01-15T18:12:39.954 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-15T18:12:40.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.099+0000 7f91ee800640 1 Processor -- start 2024-01-15T18:12:40.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.099+0000 7f91ee800640 1 -- start start 2024-01-15T18:12:40.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.103+0000 7f91ee800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f91e8158900 0x7f91e815f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:40.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.103+0000 7f91ee800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f91e815bf00 0x7f91e80cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.103+0000 7f91ee800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e80b1cb0 con 0x7f91e80cf110 2024-01-15T18:12:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.103+0000 7f91ee800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91e80cf580 con 0x7f91e815bf00 2024-01-15T18:12:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.103+0000 7f91ee800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91e80b1f00 con 0x7f91e8158900 2024-01-15T18:12:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.103+0000 7f91e6a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f91e815bf00 0x7f91e80cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.103+0000 7f91e7e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f91e80cf110 0x7f91e80701e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45488/0 (socket says 172.21.15.27:45488) 2024-01-15T18:12:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.103+0000 7f91e7e00640 1 -- 172.21.15.27:0/679587288 learned_addr learned my addr 172.21.15.27:0/679587288 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:12:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.103+0000 7f91e7400640 1 --2- 172.21.15.27:0/679587288 >> v2:172.21.15.93:3300/0 conn(0x7f91e8158900 0x7f91e815f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.103+0000 7f91e6000640 1 -- 172.21.15.27:0/679587288 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2143388048 0 0) 0x7f91e80b1cb0 con 0x7f91e80cf110 2024-01-15T18:12:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.103+0000 7f91e6000640 1 -- 172.21.15.27:0/679587288 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c8003530 con 0x7f91e80cf110 2024-01-15T18:12:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.103+0000 7f91e6a00640 1 -- 172.21.15.27:0/679587288 >> v2:172.21.15.93:3300/0 conn(0x7f91e8158900 msgr2=0x7f91e815f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.103+0000 7f91e6a00640 1 --2- 172.21.15.27:0/679587288 >> v2:172.21.15.93:3300/0 conn(0x7f91e8158900 0x7f91e815f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.103+0000 7f91e6a00640 1 -- 172.21.15.27:0/679587288 >> v1:172.21.15.27:6789/0 conn(0x7f91e80cf110 legacy=0x7f91e80701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:40.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.103+0000 7f91e6a00640 1 -- 172.21.15.27:0/679587288 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e80b21e0 con 0x7f91e815bf00 2024-01-15T18:12:40.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.107+0000 7f91e6a00640 1 --2- 172.21.15.27:0/679587288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91e815bf00 0x7f91e80cebd0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f91d00097c0 tx=0x7f91d001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9425901d0b9ce2e3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:40.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.107+0000 7f91e6000640 1 -- 172.21.15.27:0/679587288 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d00ab600 con 0x7f91e815bf00 2024-01-15T18:12:40.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.107+0000 7f91e6000640 1 -- 172.21.15.27:0/679587288 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d00a7050 con 0x7f91e815bf00 2024-01-15T18:12:40.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.107+0000 7f91e6000640 1 -- 172.21.15.27:0/679587288 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d00ab980 con 0x7f91e815bf00 2024-01-15T18:12:40.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.107+0000 7f91ee800640 1 -- 172.21.15.27:0/679587288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91e815bf00 msgr2=0x7f91e80cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:40.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.107+0000 7f91ee800640 1 --2- 172.21.15.27:0/679587288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91e815bf00 0x7f91e80cebd0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f91d00097c0 tx=0x7f91d001c440 comp rx=0 tx=0).stop 2024-01-15T18:12:40.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.107+0000 7f91ee800640 1 -- 172.21.15.27:0/679587288 shutdown_connections 2024-01-15T18:12:40.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.107+0000 7f91ee800640 1 --2- 172.21.15.27:0/679587288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91e815bf00 0x7f91e80cebd0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.107+0000 7f91ee800640 1 --2- 172.21.15.27:0/679587288 >> v2:172.21.15.93:3300/0 conn(0x7f91e8158900 0x7f91e815f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.107+0000 7f91ee800640 1 -- 172.21.15.27:0/679587288 >> 172.21.15.27:0/679587288 conn(0x7f91e80cb700 msgr2=0x7f91e80cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:40.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.107+0000 7f91ee800640 1 -- 172.21.15.27:0/679587288 shutdown_connections 2024-01-15T18:12:40.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.107+0000 7f91ee800640 1 -- 172.21.15.27:0/679587288 wait complete. 2024-01-15T18:12:40.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.107+0000 7f91ee800640 1 Processor -- start 2024-01-15T18:12:40.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.107+0000 7f91ee800640 1 -- start start 2024-01-15T18:12:40.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.111+0000 7f91ee800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f91e80cf110 0x7f91e81478d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:40.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.111+0000 7f91ee800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91e8158900 0x7f91e8145da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:40.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.111+0000 7f91ee800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f91e815bf00 0x7f91e8156950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:40.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.111+0000 7f91ee800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e8156170 con 0x7f91e8158900 2024-01-15T18:12:40.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.111+0000 7f91ee800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e80af290 con 0x7f91e815bf00 2024-01-15T18:12:40.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.111+0000 7f91ee800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e80af570 con 0x7f91e80cf110 2024-01-15T18:12:40.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.111+0000 7f91e6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91e8158900 0x7f91e8145da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:40.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.111+0000 7f91e6a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91e8158900 0x7f91e8145da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36604/0 (socket says 172.21.15.27:36604) 2024-01-15T18:12:40.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.111+0000 7f91e7400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f91e80cf110 0x7f91e81478d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:40.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.111+0000 7f91e6a00640 1 -- 172.21.15.27:0/4050196370 learned_addr learned my addr 172.21.15.27:0/4050196370 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:40.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.111+0000 7f91e7e00640 1 --2- 172.21.15.27:0/4050196370 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f91e815bf00 0x7f91e8156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:40.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.111+0000 7f91e6a00640 1 -- 172.21.15.27:0/4050196370 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f91e80cf110 msgr2=0x7f91e81478d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:40.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.111+0000 7f91e6a00640 1 --2- 172.21.15.27:0/4050196370 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f91e80cf110 0x7f91e81478d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.111+0000 7f91e6a00640 1 -- 172.21.15.27:0/4050196370 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f91e815bf00 msgr2=0x7f91e8156950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:40.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.111+0000 7f91e6a00640 1 --2- 172.21.15.27:0/4050196370 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f91e815bf00 0x7f91e8156950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.111+0000 7f91e6a00640 1 -- 172.21.15.27:0/4050196370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d0009420 con 0x7f91e8158900 2024-01-15T18:12:40.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.111+0000 7f91e6a00640 1 --2- 172.21.15.27:0/4050196370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91e8158900 0x7f91e8145da0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f91d00ae040 tx=0x7f91d00191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:40.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.111+0000 7f91df400640 1 -- 172.21.15.27:0/4050196370 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d00b4410 con 0x7f91e8158900 2024-01-15T18:12:40.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.111+0000 7f91df400640 1 -- 172.21.15.27:0/4050196370 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d00b4590 con 0x7f91e8158900 2024-01-15T18:12:40.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.111+0000 7f91df400640 1 -- 172.21.15.27:0/4050196370 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d00b4940 con 0x7f91e8158900 2024-01-15T18:12:40.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.111+0000 7f91ee800640 1 -- 172.21.15.27:0/4050196370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91e8156430 con 0x7f91e8158900 2024-01-15T18:12:40.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.111+0000 7f91ee800640 1 -- 172.21.15.27:0/4050196370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e8156e90 con 0x7f91e8158900 2024-01-15T18:12:40.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.111+0000 7f91ee800640 1 -- 172.21.15.27:0/4050196370 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91e8073bd0 con 0x7f91e8158900 2024-01-15T18:12:40.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.111+0000 7f91df400640 1 -- 172.21.15.27:0/4050196370 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 22) v1 ==== 86925+0+0 (secure 0 0 0) 0x7f91d00b2040 con 0x7f91e8158900 2024-01-15T18:12:40.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.115+0000 7f91df400640 1 --2- 172.21.15.27:0/4050196370 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f91c0067160 0x7f91c0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:40.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.115+0000 7f91df400640 1 -- 172.21.15.27:0/4050196370 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f91d011cd10 con 0x7f91e8158900 2024-01-15T18:12:40.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.115+0000 7f91e7400640 1 --2- 172.21.15.27:0/4050196370 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f91c0067160 0x7f91c0069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:40.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.115+0000 7f91e7400640 1 --2- 172.21.15.27:0/4050196370 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f91c0067160 0x7f91c0069620 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f91e80ae9f0 tx=0x7f91d8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:40.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.119+0000 7f91df400640 1 -- 172.21.15.27:0/4050196370 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f91d00eb9a0 con 0x7f91e8158900 2024-01-15T18:12:40.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.303+0000 7f91ee800640 1 -- 172.21.15.27:0/4050196370 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f91e8157080 con 0x7f91c0067160 2024-01-15T18:12:40.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.319+0000 7f91df400640 1 -- 172.21.15.27:0/4050196370 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f91d8011b50 con 0x7f91c0067160 2024-01-15T18:12:40.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.319+0000 7f91ee800640 1 -- 172.21.15.27:0/4050196370 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f91c0067160 msgr2=0x7f91c0069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:40.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.319+0000 7f91ee800640 1 --2- 172.21.15.27:0/4050196370 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f91c0067160 0x7f91c0069620 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f91e80ae9f0 tx=0x7f91d8009190 comp rx=0 tx=0).stop 2024-01-15T18:12:40.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.319+0000 7f91ee800640 1 -- 172.21.15.27:0/4050196370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91e8158900 msgr2=0x7f91e8145da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:40.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.319+0000 7f91ee800640 1 --2- 172.21.15.27:0/4050196370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91e8158900 0x7f91e8145da0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f91d00ae040 tx=0x7f91d00191e0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.319+0000 7f91ee800640 1 -- 172.21.15.27:0/4050196370 shutdown_connections 2024-01-15T18:12:40.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.319+0000 7f91ee800640 1 --2- 172.21.15.27:0/4050196370 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f91c0067160 0x7f91c0069620 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.319+0000 7f91ee800640 1 --2- 172.21.15.27:0/4050196370 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f91e815bf00 0x7f91e8156950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.319+0000 7f91ee800640 1 --2- 172.21.15.27:0/4050196370 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91e8158900 0x7f91e8145da0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.319+0000 7f91ee800640 1 --2- 172.21.15.27:0/4050196370 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f91e80cf110 0x7f91e81478d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.319+0000 7f91ee800640 1 -- 172.21.15.27:0/4050196370 >> 172.21.15.27:0/4050196370 conn(0x7f91e80cb700 msgr2=0x7f91e80720b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:40.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.319+0000 7f91ee800640 1 -- 172.21.15.27:0/4050196370 shutdown_connections 2024-01-15T18:12:40.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.319+0000 7f91ee800640 1 -- 172.21.15.27:0/4050196370 wait complete. 2024-01-15T18:12:40.341 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:12:40.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.483+0000 7f7aabc00640 1 Processor -- start 2024-01-15T18:12:40.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.483+0000 7f7aabc00640 1 -- start start 2024-01-15T18:12:40.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.487+0000 7f7aabc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7aa40cf450 0x7f7aa40d1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:40.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.487+0000 7f7aabc00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa40cefa0 con 0x7f7aa40b0650 2024-01-15T18:12:40.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.487+0000 7f7aabc00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa40ce240 con 0x7f7aa40cba60 2024-01-15T18:12:40.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.487+0000 7f7aabc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa40ccd70 con 0x7f7aa40cf450 2024-01-15T18:12:40.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.487+0000 7f7aa9c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7aa40b0650 0x7f7aa40b0ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45496/0 (socket says 172.21.15.27:45496) 2024-01-15T18:12:40.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.487+0000 7f7aa9c00640 1 -- 172.21.15.27:0/2603385459 learned_addr learned my addr 172.21.15.27:0/2603385459 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:12:40.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.487+0000 7f7aa9200640 1 --2- 172.21.15.27:0/2603385459 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7aa40cf450 0x7f7aa40d1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:40.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.487+0000 7f7aa3400640 1 -- 172.21.15.27:0/2603385459 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764017216 0 0) 0x7f7aa40cefa0 con 0x7f7aa40b0650 2024-01-15T18:12:40.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.487+0000 7f7aa3400640 1 -- 172.21.15.27:0/2603385459 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a84003530 con 0x7f7aa40b0650 2024-01-15T18:12:40.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.487+0000 7f7aa3400640 1 -- 172.21.15.27:0/2603385459 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756485982 0 0) 0x7f7aa40ce240 con 0x7f7aa40cba60 2024-01-15T18:12:40.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.487+0000 7f7aa9200640 1 -- 172.21.15.27:0/2603385459 >> v1:172.21.15.93:6789/0 conn(0x7f7aa40cba60 legacy=0x7f7aa40cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:40.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.487+0000 7f7aa9200640 1 -- 172.21.15.27:0/2603385459 >> v1:172.21.15.27:6789/0 conn(0x7f7aa40b0650 legacy=0x7f7aa40b0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:40.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.487+0000 7f7aa9200640 1 -- 172.21.15.27:0/2603385459 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa40cc1f0 con 0x7f7aa40cf450 2024-01-15T18:12:40.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.491+0000 7f7aa9200640 1 --2- 172.21.15.27:0/2603385459 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7aa40cf450 0x7f7aa40d1a40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7a940097f0 tx=0x7f7a9401c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57c82ae3cdd0728c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:40.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.491+0000 7f7aa3400640 1 -- 172.21.15.27:0/2603385459 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a940ac410 con 0x7f7aa40cf450 2024-01-15T18:12:40.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.491+0000 7f7aa3400640 1 -- 172.21.15.27:0/2603385459 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a940ac570 con 0x7f7aa40cf450 2024-01-15T18:12:40.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.491+0000 7f7aa3400640 1 -- 172.21.15.27:0/2603385459 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a940ac900 con 0x7f7aa40cf450 2024-01-15T18:12:40.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.491+0000 7f7aabc00640 1 -- 172.21.15.27:0/2603385459 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7aa40cf450 msgr2=0x7f7aa40d1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:40.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.491+0000 7f7aabc00640 1 --2- 172.21.15.27:0/2603385459 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7aa40cf450 0x7f7aa40d1a40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7a940097f0 tx=0x7f7a9401c4c0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.491+0000 7f7aabc00640 1 -- 172.21.15.27:0/2603385459 shutdown_connections 2024-01-15T18:12:40.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.491+0000 7f7aabc00640 1 --2- 172.21.15.27:0/2603385459 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7aa40cf450 0x7f7aa40d1a40 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.491+0000 7f7aabc00640 1 -- 172.21.15.27:0/2603385459 >> 172.21.15.27:0/2603385459 conn(0x7f7aa40d0e70 msgr2=0x7f7aa40d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:40.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.491+0000 7f7aabc00640 1 -- 172.21.15.27:0/2603385459 shutdown_connections 2024-01-15T18:12:40.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.491+0000 7f7aabc00640 1 -- 172.21.15.27:0/2603385459 wait complete. 2024-01-15T18:12:40.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.495+0000 7f7aabc00640 1 Processor -- start 2024-01-15T18:12:40.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.495+0000 7f7aabc00640 1 -- start start 2024-01-15T18:12:40.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.495+0000 7f7aabc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7aa40b0650 0x7f7aa40ca960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:40.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.495+0000 7f7aabc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7aa40cba60 0x7f7aa415a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:40.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.495+0000 7f7aabc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7aa40cf450 0x7f7aa415b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:40.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.495+0000 7f7aabc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa40caea0 con 0x7f7aa40cba60 2024-01-15T18:12:40.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.495+0000 7f7aabc00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa40c98c0 con 0x7f7aa40cf450 2024-01-15T18:12:40.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.495+0000 7f7aabc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa40c9ba0 con 0x7f7aa40b0650 2024-01-15T18:12:40.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.495+0000 7f7aa3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7aa40cba60 0x7f7aa415a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:40.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.495+0000 7f7aa3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7aa40cba60 0x7f7aa415a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36614/0 (socket says 172.21.15.27:36614) 2024-01-15T18:12:40.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.495+0000 7f7aa3e00640 1 -- 172.21.15.27:0/670316475 learned_addr learned my addr 172.21.15.27:0/670316475 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:40.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.495+0000 7f7aa9c00640 1 --2- 172.21.15.27:0/670316475 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7aa40cf450 0x7f7aa415b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:40.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.495+0000 7f7aa9200640 1 --2- 172.21.15.27:0/670316475 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7aa40b0650 0x7f7aa40ca960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:40.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.495+0000 7f7aa3e00640 1 -- 172.21.15.27:0/670316475 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7aa40b0650 msgr2=0x7f7aa40ca960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:40.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.495+0000 7f7aa3e00640 1 --2- 172.21.15.27:0/670316475 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7aa40b0650 0x7f7aa40ca960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.495+0000 7f7aa3e00640 1 -- 172.21.15.27:0/670316475 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7aa40cf450 msgr2=0x7f7aa415b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:40.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.495+0000 7f7aa3e00640 1 --2- 172.21.15.27:0/670316475 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7aa40cf450 0x7f7aa415b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.495+0000 7f7aa3e00640 1 -- 172.21.15.27:0/670316475 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a94009420 con 0x7f7aa40cba60 2024-01-15T18:12:40.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.495+0000 7f7aa3e00640 1 --2- 172.21.15.27:0/670316475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7aa40cba60 0x7f7aa415a0d0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c001ec0 tx=0x7f7a8c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:40.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.495+0000 7f7aa0c00640 1 -- 172.21.15.27:0/670316475 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c0054e0 con 0x7f7aa40cba60 2024-01-15T18:12:40.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.495+0000 7f7aa0c00640 1 -- 172.21.15.27:0/670316475 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a8c003600 con 0x7f7aa40cba60 2024-01-15T18:12:40.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.495+0000 7f7aabc00640 1 -- 172.21.15.27:0/670316475 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa40c9e80 con 0x7f7aa40cba60 2024-01-15T18:12:40.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.495+0000 7f7aa0c00640 1 -- 172.21.15.27:0/670316475 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c0039b0 con 0x7f7aa40cba60 2024-01-15T18:12:40.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.495+0000 7f7aabc00640 1 -- 172.21.15.27:0/670316475 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa40ca440 con 0x7f7aa40cba60 2024-01-15T18:12:40.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.495+0000 7f7aabc00640 1 -- 172.21.15.27:0/670316475 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa404e850 con 0x7f7aa40cba60 2024-01-15T18:12:40.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.499+0000 7f7aa0c00640 1 -- 172.21.15.27:0/670316475 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 22) v1 ==== 86925+0+0 (secure 0 0 0) 0x7f7a8c003b10 con 0x7f7aa40cba60 2024-01-15T18:12:40.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.499+0000 7f7aa0c00640 1 --2- 172.21.15.27:0/670316475 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7a7c067190 0x7f7a7c069650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:40.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.499+0000 7f7aa0c00640 1 -- 172.21.15.27:0/670316475 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f7a8c0823e0 con 0x7f7aa40cba60 2024-01-15T18:12:40.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.499+0000 7f7aa9200640 1 --2- 172.21.15.27:0/670316475 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7a7c067190 0x7f7a7c069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:40.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.499+0000 7f7aa9200640 1 --2- 172.21.15.27:0/670316475 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7a7c067190 0x7f7a7c069650 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f7a940ae000 tx=0x7f7a940a6040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:40.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.503+0000 7f7aa0c00640 1 -- 172.21.15.27:0/670316475 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7a8c050ed0 con 0x7f7aa40cba60 2024-01-15T18:12:40.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.687+0000 7f7aabc00640 1 -- 172.21.15.27:0/670316475 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7aa406f7f0 con 0x7f7a7c067190 2024-01-15T18:12:40.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.691+0000 7f7aa0c00640 1 -- 172.21.15.27:0/670316475 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f7aa406f7f0 con 0x7f7a7c067190 2024-01-15T18:12:40.693 INFO:teuthology.orchestra.run.smithi027.stdout:/volumes/_nogroup/sv1/10f4c67f-ff21-41d3-a8bc-abca26a3de8e 2024-01-15T18:12:40.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.695+0000 7f7aabc00640 1 -- 172.21.15.27:0/670316475 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7a7c067190 msgr2=0x7f7a7c069650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:40.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.695+0000 7f7aabc00640 1 --2- 172.21.15.27:0/670316475 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7a7c067190 0x7f7a7c069650 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f7a940ae000 tx=0x7f7a940a6040 comp rx=0 tx=0).stop 2024-01-15T18:12:40.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.695+0000 7f7aabc00640 1 -- 172.21.15.27:0/670316475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7aa40cba60 msgr2=0x7f7aa415a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:40.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.695+0000 7f7aabc00640 1 --2- 172.21.15.27:0/670316475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7aa40cba60 0x7f7aa415a0d0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c001ec0 tx=0x7f7a8c00c8e0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.695+0000 7f7aabc00640 1 -- 172.21.15.27:0/670316475 shutdown_connections 2024-01-15T18:12:40.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.695+0000 7f7aabc00640 1 --2- 172.21.15.27:0/670316475 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7a7c067190 0x7f7a7c069650 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.695+0000 7f7aabc00640 1 --2- 172.21.15.27:0/670316475 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7aa40cf450 0x7f7aa415b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.695+0000 7f7aabc00640 1 --2- 172.21.15.27:0/670316475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7aa40cba60 0x7f7aa415a0d0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.695+0000 7f7aabc00640 1 --2- 172.21.15.27:0/670316475 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7aa40b0650 0x7f7aa40ca960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.695+0000 7f7aabc00640 1 -- 172.21.15.27:0/670316475 >> 172.21.15.27:0/670316475 conn(0x7f7aa40d0e70 msgr2=0x7f7aa406f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:40.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.695+0000 7f7aabc00640 1 -- 172.21.15.27:0/670316475 shutdown_connections 2024-01-15T18:12:40.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.695+0000 7f7aabc00640 1 -- 172.21.15.27:0/670316475 wait complete. 2024-01-15T18:12:40.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:40.715 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/10f4c67f-ff21-41d3-a8bc-abca26a3de8e) 2024-01-15T18:12:40.728 INFO:teuthology.orchestra.run.smithi027.stdout:# file: volumes/_nogroup/sv1/10f4c67f-ff21-41d3-a8bc-abca26a3de8e 2024-01-15T18:12:40.728 INFO:teuthology.orchestra.run.smithi027.stdout:ceph.dir.rbytes="3221225472" 2024-01-15T18:12:40.728 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:12:40.729 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-15T18:12:40.729 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone51931 2024-01-15T18:12:40.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.915+0000 7fb19aa00640 1 Processor -- start 2024-01-15T18:12:40.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.915+0000 7fb19aa00640 1 -- start start 2024-01-15T18:12:40.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.915+0000 7fb19aa00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb19415d570 0x7fb19406be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:40.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.915+0000 7fb19aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb19406b810 0x7fb19406c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:40.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.915+0000 7fb19aa00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1940b0d70 con 0x7fb19415d190 2024-01-15T18:12:40.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.915+0000 7fb19aa00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1940d0540 con 0x7fb19406b810 2024-01-15T18:12:40.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.915+0000 7fb19aa00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1940b0440 con 0x7fb19415d570 2024-01-15T18:12:40.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.915+0000 7fb193e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb19415d190 0x7fb1940cfcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45510/0 (socket says 172.21.15.27:45510) 2024-01-15T18:12:40.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.915+0000 7fb193e00640 1 -- 172.21.15.27:0/4288153262 learned_addr learned my addr 172.21.15.27:0/4288153262 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:12:40.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.915+0000 7fb198a00640 1 --2- 172.21.15.27:0/4288153262 >> v2:172.21.15.27:3300/0 conn(0x7fb19406b810 0x7fb19406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:40.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.915+0000 7fb193400640 1 --2- 172.21.15.27:0/4288153262 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb19415d570 0x7fb19406be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:40.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.915+0000 7fb192a00640 1 -- 172.21.15.27:0/4288153262 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632064011 0 0) 0x7fb1940b0d70 con 0x7fb19415d190 2024-01-15T18:12:40.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.915+0000 7fb192a00640 1 -- 172.21.15.27:0/4288153262 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb174003530 con 0x7fb19415d190 2024-01-15T18:12:40.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.915+0000 7fb198a00640 1 -- 172.21.15.27:0/4288153262 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb19415d570 msgr2=0x7fb19406be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:40.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.915+0000 7fb198a00640 1 --2- 172.21.15.27:0/4288153262 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb19415d570 0x7fb19406be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.915+0000 7fb198a00640 1 -- 172.21.15.27:0/4288153262 >> v1:172.21.15.27:6789/0 conn(0x7fb19415d190 legacy=0x7fb1940cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:40.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.915+0000 7fb198a00640 1 -- 172.21.15.27:0/4288153262 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1940b0720 con 0x7fb19406b810 2024-01-15T18:12:40.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.919+0000 7fb198a00640 1 --2- 172.21.15.27:0/4288153262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb19406b810 0x7fb19406c350 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fb18c009d90 tx=0x7fb18c01c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64953aecf04bfeed server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:40.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.919+0000 7fb192a00640 1 -- 172.21.15.27:0/4288153262 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18c0a5a00 con 0x7fb19406b810 2024-01-15T18:12:40.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.919+0000 7fb192a00640 1 -- 172.21.15.27:0/4288153262 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb18c0a5b60 con 0x7fb19406b810 2024-01-15T18:12:40.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.919+0000 7fb192a00640 1 -- 172.21.15.27:0/4288153262 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18c0ad440 con 0x7fb19406b810 2024-01-15T18:12:40.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.919+0000 7fb19aa00640 1 -- 172.21.15.27:0/4288153262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb19406b810 msgr2=0x7fb19406c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:40.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.919+0000 7fb19aa00640 1 --2- 172.21.15.27:0/4288153262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb19406b810 0x7fb19406c350 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fb18c009d90 tx=0x7fb18c01c920 comp rx=0 tx=0).stop 2024-01-15T18:12:40.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.919+0000 7fb19aa00640 1 -- 172.21.15.27:0/4288153262 shutdown_connections 2024-01-15T18:12:40.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.919+0000 7fb19aa00640 1 --2- 172.21.15.27:0/4288153262 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb19406b810 0x7fb19406c350 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.919+0000 7fb19aa00640 1 --2- 172.21.15.27:0/4288153262 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb19415d570 0x7fb19406be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.919+0000 7fb19aa00640 1 -- 172.21.15.27:0/4288153262 >> 172.21.15.27:0/4288153262 conn(0x7fb1940d0e70 msgr2=0x7fb1940d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:40.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.919+0000 7fb19aa00640 1 -- 172.21.15.27:0/4288153262 shutdown_connections 2024-01-15T18:12:40.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.919+0000 7fb19aa00640 1 -- 172.21.15.27:0/4288153262 wait complete. 2024-01-15T18:12:40.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.923+0000 7fb19aa00640 1 Processor -- start 2024-01-15T18:12:40.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.923+0000 7fb19aa00640 1 -- start start 2024-01-15T18:12:40.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.923+0000 7fb19aa00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb19406b810 0x7fb1940b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.923+0000 7fb19aa00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb19415d190 0x7fb1940b5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.923+0000 7fb19aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb19415d570 0x7fb1940adae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.923+0000 7fb19aa00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19415a0f0 con 0x7fb19415d570 2024-01-15T18:12:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.923+0000 7fb19aa00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1940b64d0 con 0x7fb19415d190 2024-01-15T18:12:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.923+0000 7fb19aa00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1940b67b0 con 0x7fb19406b810 2024-01-15T18:12:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.923+0000 7fb198a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb19415d570 0x7fb1940adae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.923+0000 7fb198a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb19415d570 0x7fb1940adae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36644/0 (socket says 172.21.15.27:36644) 2024-01-15T18:12:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.923+0000 7fb198a00640 1 -- 172.21.15.27:0/2336309453 learned_addr learned my addr 172.21.15.27:0/2336309453 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.923+0000 7fb193e00640 1 --2- 172.21.15.27:0/2336309453 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb19406b810 0x7fb1940b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.923+0000 7fb193400640 1 --2- 172.21.15.27:0/2336309453 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb19415d190 0x7fb1940b5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.923+0000 7fb198a00640 1 -- 172.21.15.27:0/2336309453 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb19406b810 msgr2=0x7fb1940b3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.923+0000 7fb198a00640 1 --2- 172.21.15.27:0/2336309453 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb19406b810 0x7fb1940b3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.923+0000 7fb198a00640 1 -- 172.21.15.27:0/2336309453 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb19415d190 msgr2=0x7fb1940b5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:40.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.923+0000 7fb198a00640 1 --2- 172.21.15.27:0/2336309453 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb19415d190 0x7fb1940b5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.923+0000 7fb198a00640 1 -- 172.21.15.27:0/2336309453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb18c009a70 con 0x7fb19415d570 2024-01-15T18:12:40.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.923+0000 7fb198a00640 1 --2- 172.21.15.27:0/2336309453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb19415d570 0x7fb1940adae0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fb18c0b0040 tx=0x7fb18c0196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:40.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.923+0000 7fb18be00640 1 -- 172.21.15.27:0/2336309453 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18c0a5a00 con 0x7fb19415d570 2024-01-15T18:12:40.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.923+0000 7fb18be00640 1 -- 172.21.15.27:0/2336309453 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb18c009450 con 0x7fb19415d570 2024-01-15T18:12:40.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.923+0000 7fb19aa00640 1 -- 172.21.15.27:0/2336309453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1940b42a0 con 0x7fb19415d570 2024-01-15T18:12:40.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.923+0000 7fb18be00640 1 -- 172.21.15.27:0/2336309453 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18c0b9690 con 0x7fb19415d570 2024-01-15T18:12:40.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.927+0000 7fb19aa00640 1 -- 172.21.15.27:0/2336309453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1940b0140 con 0x7fb19415d570 2024-01-15T18:12:40.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.927+0000 7fb19aa00640 1 -- 172.21.15.27:0/2336309453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb19404e850 con 0x7fb19415d570 2024-01-15T18:12:40.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.927+0000 7fb18be00640 1 -- 172.21.15.27:0/2336309453 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 22) v1 ==== 86925+0+0 (secure 0 0 0) 0x7fb18c0039a0 con 0x7fb19415d570 2024-01-15T18:12:40.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.927+0000 7fb18be00640 1 --2- 172.21.15.27:0/2336309453 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb16c067190 0x7fb16c069650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:40.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.927+0000 7fb18be00640 1 -- 172.21.15.27:0/2336309453 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fb18c120700 con 0x7fb19415d570 2024-01-15T18:12:40.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.927+0000 7fb193e00640 1 --2- 172.21.15.27:0/2336309453 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb16c067190 0x7fb16c069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:40.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.931+0000 7fb193e00640 1 --2- 172.21.15.27:0/2336309453 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb16c067190 0x7fb16c069650 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fb1840011f0 tx=0x7fb184007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:40.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:40.931+0000 7fb18be00640 1 -- 172.21.15.27:0/2336309453 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb18c0ecf10 con 0x7fb19415d570 2024-01-15T18:12:41.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.123+0000 7fb19aa00640 1 -- 172.21.15.27:0/2336309453 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone51931", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1940b6640 con 0x7fb16c067190 2024-01-15T18:12:41.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.195+0000 7fb18be00640 1 -- 172.21.15.27:0/2336309453 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb1940b6640 con 0x7fb16c067190 2024-01-15T18:12:41.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.199+0000 7fb19aa00640 1 -- 172.21.15.27:0/2336309453 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb16c067190 msgr2=0x7fb16c069650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:41.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.199+0000 7fb19aa00640 1 --2- 172.21.15.27:0/2336309453 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb16c067190 0x7fb16c069650 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fb1840011f0 tx=0x7fb184007d40 comp rx=0 tx=0).stop 2024-01-15T18:12:41.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.199+0000 7fb19aa00640 1 -- 172.21.15.27:0/2336309453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb19415d570 msgr2=0x7fb1940adae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:41.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.199+0000 7fb19aa00640 1 --2- 172.21.15.27:0/2336309453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb19415d570 0x7fb1940adae0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fb18c0b0040 tx=0x7fb18c0196c0 comp rx=0 tx=0).stop 2024-01-15T18:12:41.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.199+0000 7fb19aa00640 1 -- 172.21.15.27:0/2336309453 shutdown_connections 2024-01-15T18:12:41.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.199+0000 7fb19aa00640 1 --2- 172.21.15.27:0/2336309453 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb16c067190 0x7fb16c069650 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:41.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.199+0000 7fb19aa00640 1 --2- 172.21.15.27:0/2336309453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb19415d570 0x7fb1940adae0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:41.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.199+0000 7fb19aa00640 1 --2- 172.21.15.27:0/2336309453 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb19415d190 0x7fb1940b5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:41.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.199+0000 7fb19aa00640 1 --2- 172.21.15.27:0/2336309453 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb19406b810 0x7fb1940b3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:41.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.199+0000 7fb19aa00640 1 -- 172.21.15.27:0/2336309453 >> 172.21.15.27:0/2336309453 conn(0x7fb1940d0e70 msgr2=0x7fb19406ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:41.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.199+0000 7fb19aa00640 1 -- 172.21.15.27:0/2336309453 shutdown_connections 2024-01-15T18:12:41.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.199+0000 7fb19aa00640 1 -- 172.21.15.27:0/2336309453 wait complete. 2024-01-15T18:12:41.204 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:12:41.203+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:41.204 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:12:41.203+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:41.204 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:12:41.203+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:41.204 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:12:41.203+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:41.204 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:12:41.203+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:41.223 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:12:41.219+0000 7f59f8800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:41.223 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:12:41.219+0000 7f59f8800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:41.223 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:12:41.219+0000 7f59f8800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:41.223 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:12:41.219+0000 7f59f8800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:41.223 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:12:41.219+0000 7f59f8800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:41.230 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone51932 2024-01-15T18:12:41.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.399+0000 7f9b2bc00640 1 Processor -- start 2024-01-15T18:12:41.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.403+0000 7f9b2bc00640 1 -- start start 2024-01-15T18:12:41.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.403+0000 7f9b2bc00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f9b2415d5d0 0x7f9b240d3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:41.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.403+0000 7f9b2bc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b240cf890 0x7f9b240cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:41.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.403+0000 7f9b2bc00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b240cf2d0 con 0x7f9b240cde00 2024-01-15T18:12:41.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.403+0000 7f9b2bc00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b240cf5b0 con 0x7f9b2415d5d0 2024-01-15T18:12:41.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.403+0000 7f9b2bc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b240d0240 con 0x7f9b240cf890 2024-01-15T18:12:41.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.403+0000 7f9b29c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9b240cde00 0x7f9b240ce230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45522/0 (socket says 172.21.15.27:45522) 2024-01-15T18:12:41.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.403+0000 7f9b29c00640 1 -- 172.21.15.27:0/369932096 learned_addr learned my addr 172.21.15.27:0/369932096 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:12:41.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.403+0000 7f9b23e00640 1 --2- 172.21.15.27:0/369932096 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b240cf890 0x7f9b240cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:41.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.403+0000 7f9b29200640 1 --2- 172.21.15.27:0/369932096 >> v2:172.21.15.93:3300/0 conn(0x7f9b2415d5d0 0x7f9b240d3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:41.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.403+0000 7f9b23400640 1 -- 172.21.15.27:0/369932096 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754511002 0 0) 0x7f9b240cf2d0 con 0x7f9b240cde00 2024-01-15T18:12:41.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.403+0000 7f9b23400640 1 -- 172.21.15.27:0/369932096 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b04003530 con 0x7f9b240cde00 2024-01-15T18:12:41.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.403+0000 7f9b23e00640 1 -- 172.21.15.27:0/369932096 >> v2:172.21.15.93:3300/0 conn(0x7f9b2415d5d0 msgr2=0x7f9b240d3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:41.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.403+0000 7f9b23e00640 1 --2- 172.21.15.27:0/369932096 >> v2:172.21.15.93:3300/0 conn(0x7f9b2415d5d0 0x7f9b240d3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:41.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.403+0000 7f9b23e00640 1 -- 172.21.15.27:0/369932096 >> v1:172.21.15.27:6789/0 conn(0x7f9b240cde00 legacy=0x7f9b240ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:41.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.403+0000 7f9b23e00640 1 -- 172.21.15.27:0/369932096 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b240b2610 con 0x7f9b240cf890 2024-01-15T18:12:41.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.403+0000 7f9b29200640 1 --2- 172.21.15.27:0/369932096 >> v2:172.21.15.93:3300/0 conn(0x7f9b2415d5d0 0x7f9b240d3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:12:41.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.407+0000 7f9b23e00640 1 --2- 172.21.15.27:0/369932096 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b240cf890 0x7f9b240cfd00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c0097c0 tx=0x7f9b0c01c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26b55dadff5dc94d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:41.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.407+0000 7f9b23400640 1 -- 172.21.15.27:0/369932096 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c0ab680 con 0x7f9b240cf890 2024-01-15T18:12:41.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.407+0000 7f9b23400640 1 -- 172.21.15.27:0/369932096 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b0c0ab7e0 con 0x7f9b240cf890 2024-01-15T18:12:41.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.407+0000 7f9b23400640 1 -- 172.21.15.27:0/369932096 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c0abb70 con 0x7f9b240cf890 2024-01-15T18:12:41.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.407+0000 7f9b2bc00640 1 -- 172.21.15.27:0/369932096 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b240cf890 msgr2=0x7f9b240cfd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:41.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.407+0000 7f9b2bc00640 1 --2- 172.21.15.27:0/369932096 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b240cf890 0x7f9b240cfd00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c0097c0 tx=0x7f9b0c01c480 comp rx=0 tx=0).stop 2024-01-15T18:12:41.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.407+0000 7f9b2bc00640 1 -- 172.21.15.27:0/369932096 shutdown_connections 2024-01-15T18:12:41.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.407+0000 7f9b2bc00640 1 --2- 172.21.15.27:0/369932096 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b240cf890 0x7f9b240cfd00 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:41.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.407+0000 7f9b2bc00640 1 --2- 172.21.15.27:0/369932096 >> v2:172.21.15.93:3300/0 conn(0x7f9b2415d5d0 0x7f9b240d3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:41.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.407+0000 7f9b2bc00640 1 -- 172.21.15.27:0/369932096 >> 172.21.15.27:0/369932096 conn(0x7f9b240d3010 msgr2=0x7f9b240d3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:41.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.407+0000 7f9b2bc00640 1 -- 172.21.15.27:0/369932096 shutdown_connections 2024-01-15T18:12:41.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.407+0000 7f9b2bc00640 1 -- 172.21.15.27:0/369932096 wait complete. 2024-01-15T18:12:41.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.411+0000 7f9b2bc00640 1 Processor -- start 2024-01-15T18:12:41.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.411+0000 7f9b2bc00640 1 -- start start 2024-01-15T18:12:41.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.411+0000 7f9b2bc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9b240cde00 0x7f9b240b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:41.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.411+0000 7f9b2bc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b2415d5d0 0x7f9b24153410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:41.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.411+0000 7f9b2bc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b2406cb50 0x7f9b2415c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:41.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.411+0000 7f9b2bc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b240c9b40 con 0x7f9b2415d5d0 2024-01-15T18:12:41.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.411+0000 7f9b2bc00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b240c9de0 con 0x7f9b240cde00 2024-01-15T18:12:41.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.411+0000 7f9b2bc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b24153d70 con 0x7f9b2406cb50 2024-01-15T18:12:41.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.411+0000 7f9b23e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b2415d5d0 0x7f9b24153410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:41.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.411+0000 7f9b23e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b2415d5d0 0x7f9b24153410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36650/0 (socket says 172.21.15.27:36650) 2024-01-15T18:12:41.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.411+0000 7f9b23e00640 1 -- 172.21.15.27:0/642369718 learned_addr learned my addr 172.21.15.27:0/642369718 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:41.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.415+0000 7f9b29200640 1 --2- 172.21.15.27:0/642369718 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9b240cde00 0x7f9b240b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:41.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.415+0000 7f9b29c00640 1 --2- 172.21.15.27:0/642369718 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b2406cb50 0x7f9b2415c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:41.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.415+0000 7f9b23e00640 1 -- 172.21.15.27:0/642369718 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b2406cb50 msgr2=0x7f9b2415c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:41.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.415+0000 7f9b23e00640 1 --2- 172.21.15.27:0/642369718 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b2406cb50 0x7f9b2415c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:41.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.415+0000 7f9b23e00640 1 -- 172.21.15.27:0/642369718 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9b240cde00 msgr2=0x7f9b240b5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:41.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.415+0000 7f9b23e00640 1 --2- 172.21.15.27:0/642369718 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9b240cde00 0x7f9b240b5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:41.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.415+0000 7f9b23e00640 1 -- 172.21.15.27:0/642369718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b14009420 con 0x7f9b2415d5d0 2024-01-15T18:12:41.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.415+0000 7f9b29200640 1 --2- 172.21.15.27:0/642369718 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9b240cde00 0x7f9b240b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:12:41.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.415+0000 7f9b29c00640 1 --2- 172.21.15.27:0/642369718 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b2406cb50 0x7f9b2415c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:41.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.415+0000 7f9b23e00640 1 --2- 172.21.15.27:0/642369718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b2415d5d0 0x7f9b24153410 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c0ad040 tx=0x7f9b0c01acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:41.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.415+0000 7f9b20c00640 1 -- 172.21.15.27:0/642369718 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c004220 con 0x7f9b2415d5d0 2024-01-15T18:12:41.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.415+0000 7f9b20c00640 1 -- 172.21.15.27:0/642369718 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b0c004380 con 0x7f9b2415d5d0 2024-01-15T18:12:41.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.415+0000 7f9b2bc00640 1 -- 172.21.15.27:0/642369718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b0c009420 con 0x7f9b2415d5d0 2024-01-15T18:12:41.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.415+0000 7f9b20c00640 1 -- 172.21.15.27:0/642369718 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c003820 con 0x7f9b2415d5d0 2024-01-15T18:12:41.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.415+0000 7f9b2bc00640 1 -- 172.21.15.27:0/642369718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b240caed0 con 0x7f9b2415d5d0 2024-01-15T18:12:41.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.415+0000 7f9b2bc00640 1 -- 172.21.15.27:0/642369718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b2404e850 con 0x7f9b2415d5d0 2024-01-15T18:12:41.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.415+0000 7f9b20c00640 1 -- 172.21.15.27:0/642369718 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 22) v1 ==== 86925+0+0 (secure 0 0 0) 0x7f9b0c003980 con 0x7f9b2415d5d0 2024-01-15T18:12:41.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.419+0000 7f9b20c00640 1 --2- 172.21.15.27:0/642369718 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9afc067160 0x7f9afc069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:41.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.419+0000 7f9b20c00640 1 -- 172.21.15.27:0/642369718 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f9b0c11d9a0 con 0x7f9b2415d5d0 2024-01-15T18:12:41.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.419+0000 7f9b29200640 1 --2- 172.21.15.27:0/642369718 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9afc067160 0x7f9afc069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:41.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.419+0000 7f9b29200640 1 --2- 172.21.15.27:0/642369718 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9afc067160 0x7f9afc069620 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f9b240afe40 tx=0x7f9b14008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:41.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.423+0000 7f9b20c00640 1 -- 172.21.15.27:0/642369718 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9b0c0ec490 con 0x7f9b2415d5d0 2024-01-15T18:12:41.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.623+0000 7f9b2bc00640 1 -- 172.21.15.27:0/642369718 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone51932", "target": ["mon-mgr", ""]}) v1 -- 0x7f9b24154220 con 0x7f9afc067160 2024-01-15T18:12:41.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.679+0000 7f9b20c00640 1 -- 172.21.15.27:0/642369718 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9b24154220 con 0x7f9afc067160 2024-01-15T18:12:41.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.683+0000 7f9b2bc00640 1 -- 172.21.15.27:0/642369718 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9afc067160 msgr2=0x7f9afc069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:41.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.683+0000 7f9b2bc00640 1 --2- 172.21.15.27:0/642369718 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9afc067160 0x7f9afc069620 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f9b240afe40 tx=0x7f9b14008040 comp rx=0 tx=0).stop 2024-01-15T18:12:41.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.683+0000 7f9b2bc00640 1 -- 172.21.15.27:0/642369718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b2415d5d0 msgr2=0x7f9b24153410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:41.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.683+0000 7f9b2bc00640 1 --2- 172.21.15.27:0/642369718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b2415d5d0 0x7f9b24153410 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c0ad040 tx=0x7f9b0c01acc0 comp rx=0 tx=0).stop 2024-01-15T18:12:41.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.683+0000 7f9b2bc00640 1 -- 172.21.15.27:0/642369718 shutdown_connections 2024-01-15T18:12:41.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.683+0000 7f9b2bc00640 1 --2- 172.21.15.27:0/642369718 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9afc067160 0x7f9afc069620 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:41.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.683+0000 7f9b2bc00640 1 --2- 172.21.15.27:0/642369718 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b2406cb50 0x7f9b2415c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:41.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.683+0000 7f9b2bc00640 1 --2- 172.21.15.27:0/642369718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b2415d5d0 0x7f9b24153410 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:41.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.683+0000 7f9b2bc00640 1 --2- 172.21.15.27:0/642369718 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9b240cde00 0x7f9b240b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:41.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.683+0000 7f9b2bc00640 1 -- 172.21.15.27:0/642369718 >> 172.21.15.27:0/642369718 conn(0x7f9b240d3010 msgr2=0x7f9b240b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:41.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.683+0000 7f9b2bc00640 1 -- 172.21.15.27:0/642369718 shutdown_connections 2024-01-15T18:12:41.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.683+0000 7f9b2bc00640 1 -- 172.21.15.27:0/642369718 wait complete. 2024-01-15T18:12:41.706 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone51933 2024-01-15T18:12:41.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.851+0000 7f4e51a00640 1 Processor -- start 2024-01-15T18:12:41.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.851+0000 7f4e51a00640 1 -- start start 2024-01-15T18:12:41.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.851+0000 7f4e51a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e4c15b350 0x7f4e4c0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:41.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.851+0000 7f4e51a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4e4c099a10 0x7f4e4c099e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:41.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.851+0000 7f4e51a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e4c0cfec0 con 0x7f4e4c09a5b0 2024-01-15T18:12:41.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.851+0000 7f4e51a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e4c09a3c0 con 0x7f4e4c099a10 2024-01-15T18:12:41.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.851+0000 7f4e51a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c0ccd50 con 0x7f4e4c15b350 2024-01-15T18:12:41.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.851+0000 7f4e4a000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4e4c09a5b0 0x7f4e4c0cc500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45534/0 (socket says 172.21.15.27:45534) 2024-01-15T18:12:41.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.851+0000 7f4e4a000640 1 -- 172.21.15.27:0/3876086965 learned_addr learned my addr 172.21.15.27:0/3876086965 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:12:41.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.851+0000 7f4e4b400640 1 --2- 172.21.15.27:0/3876086965 >> v2:172.21.15.27:3300/0 conn(0x7f4e4c099a10 0x7f4e4c099e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:41.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.855+0000 7f4e4aa00640 1 --2- 172.21.15.27:0/3876086965 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e4c15b350 0x7f4e4c0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:41.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.855+0000 7f4e4b400640 1 -- 172.21.15.27:0/3876086965 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e4c15b350 msgr2=0x7f4e4c0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:41.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.855+0000 7f4e4b400640 1 --2- 172.21.15.27:0/3876086965 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e4c15b350 0x7f4e4c0cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:41.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.855+0000 7f4e4b400640 1 -- 172.21.15.27:0/3876086965 >> v1:172.21.15.27:6789/0 conn(0x7f4e4c09a5b0 legacy=0x7f4e4c0cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:41.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.855+0000 7f4e4b400640 1 -- 172.21.15.27:0/3876086965 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e4c0cebc0 con 0x7f4e4c099a10 2024-01-15T18:12:41.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.855+0000 7f4e49600640 1 -- 172.21.15.27:0/3876086965 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3475908042 0 0) 0x7f4e4c0cfec0 con 0x7f4e4c09a5b0 2024-01-15T18:12:41.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.855+0000 7f4e4b400640 1 --2- 172.21.15.27:0/3876086965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e4c099a10 0x7f4e4c099e80 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f4e44009d90 tx=0x7f4e4401c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35079cf0b9836a5c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:41.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.855+0000 7f4e49600640 1 -- 172.21.15.27:0/3876086965 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e440a5950 con 0x7f4e4c099a10 2024-01-15T18:12:41.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.855+0000 7f4e49600640 1 -- 172.21.15.27:0/3876086965 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e440a5ab0 con 0x7f4e4c099a10 2024-01-15T18:12:41.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.855+0000 7f4e49600640 1 -- 172.21.15.27:0/3876086965 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e440a5e40 con 0x7f4e4c099a10 2024-01-15T18:12:41.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.859+0000 7f4e51a00640 1 -- 172.21.15.27:0/3876086965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e4c099a10 msgr2=0x7f4e4c099e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:41.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.859+0000 7f4e51a00640 1 --2- 172.21.15.27:0/3876086965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e4c099a10 0x7f4e4c099e80 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f4e44009d90 tx=0x7f4e4401c890 comp rx=0 tx=0).stop 2024-01-15T18:12:41.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.859+0000 7f4e51a00640 1 -- 172.21.15.27:0/3876086965 shutdown_connections 2024-01-15T18:12:41.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.859+0000 7f4e51a00640 1 --2- 172.21.15.27:0/3876086965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e4c099a10 0x7f4e4c099e80 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:41.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.859+0000 7f4e51a00640 1 --2- 172.21.15.27:0/3876086965 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e4c15b350 0x7f4e4c0cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:41.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.859+0000 7f4e51a00640 1 -- 172.21.15.27:0/3876086965 >> 172.21.15.27:0/3876086965 conn(0x7f4e4c0b2a80 msgr2=0x7f4e4c0b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:41.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.859+0000 7f4e51a00640 1 -- 172.21.15.27:0/3876086965 shutdown_connections 2024-01-15T18:12:41.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.859+0000 7f4e51a00640 1 -- 172.21.15.27:0/3876086965 wait complete. 2024-01-15T18:12:41.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.859+0000 7f4e51a00640 1 Processor -- start 2024-01-15T18:12:41.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.859+0000 7f4e51a00640 1 -- start start 2024-01-15T18:12:41.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.859+0000 7f4e51a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e4c099a10 0x7f4e4c155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:41.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.859+0000 7f4e51a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e4c09a5b0 0x7f4e4c0b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:41.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.859+0000 7f4e51a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e4c15b350 0x7f4e4c0b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:41.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.859+0000 7f4e51a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c0c8530 con 0x7f4e4c099a10 2024-01-15T18:12:41.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.859+0000 7f4e51a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c155fc0 con 0x7f4e4c09a5b0 2024-01-15T18:12:41.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.859+0000 7f4e51a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c1562a0 con 0x7f4e4c15b350 2024-01-15T18:12:41.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.859+0000 7f4e4aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e4c099a10 0x7f4e4c155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:41.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.859+0000 7f4e4aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e4c099a10 0x7f4e4c155670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36672/0 (socket says 172.21.15.27:36672) 2024-01-15T18:12:41.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.859+0000 7f4e4aa00640 1 -- 172.21.15.27:0/4121370306 learned_addr learned my addr 172.21.15.27:0/4121370306 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:41.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.859+0000 7f4e4a000640 1 --2- 172.21.15.27:0/4121370306 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e4c09a5b0 0x7f4e4c0b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:41.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.859+0000 7f4e4b400640 1 --2- 172.21.15.27:0/4121370306 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e4c15b350 0x7f4e4c0b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:41.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.859+0000 7f4e4aa00640 1 -- 172.21.15.27:0/4121370306 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e4c15b350 msgr2=0x7f4e4c0b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:41.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.863+0000 7f4e4aa00640 1 --2- 172.21.15.27:0/4121370306 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e4c15b350 0x7f4e4c0b33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:41.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.863+0000 7f4e4aa00640 1 -- 172.21.15.27:0/4121370306 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e4c09a5b0 msgr2=0x7f4e4c0b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:41.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.863+0000 7f4e4aa00640 1 --2- 172.21.15.27:0/4121370306 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e4c09a5b0 0x7f4e4c0b2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:41.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.863+0000 7f4e4aa00640 1 -- 172.21.15.27:0/4121370306 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e44009a70 con 0x7f4e4c099a10 2024-01-15T18:12:41.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.863+0000 7f4e4aa00640 1 --2- 172.21.15.27:0/4121370306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e4c099a10 0x7f4e4c155670 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c00e870 tx=0x7f4e3c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:41.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.863+0000 7f4e42a00640 1 -- 172.21.15.27:0/4121370306 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c009510 con 0x7f4e4c099a10 2024-01-15T18:12:41.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.863+0000 7f4e42a00640 1 -- 172.21.15.27:0/4121370306 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e3c00eec0 con 0x7f4e4c099a10 2024-01-15T18:12:41.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.863+0000 7f4e51a00640 1 -- 172.21.15.27:0/4121370306 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e4c156580 con 0x7f4e4c099a10 2024-01-15T18:12:41.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.863+0000 7f4e42a00640 1 -- 172.21.15.27:0/4121370306 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c00cb40 con 0x7f4e4c099a10 2024-01-15T18:12:41.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.863+0000 7f4e51a00640 1 -- 172.21.15.27:0/4121370306 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e4c155bb0 con 0x7f4e4c099a10 2024-01-15T18:12:41.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.863+0000 7f4e51a00640 1 -- 172.21.15.27:0/4121370306 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e4c0d0720 con 0x7f4e4c099a10 2024-01-15T18:12:41.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.863+0000 7f4e42a00640 1 -- 172.21.15.27:0/4121370306 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 22) v1 ==== 86925+0+0 (secure 0 0 0) 0x7f4e3c00cca0 con 0x7f4e4c099a10 2024-01-15T18:12:41.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.867+0000 7f4e42a00640 1 --2- 172.21.15.27:0/4121370306 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4e24067190 0x7f4e24069650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:41.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.867+0000 7f4e42a00640 1 -- 172.21.15.27:0/4121370306 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f4e3c082e80 con 0x7f4e4c099a10 2024-01-15T18:12:41.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.867+0000 7f4e4a000640 1 --2- 172.21.15.27:0/4121370306 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4e24067190 0x7f4e24069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:41.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.867+0000 7f4e4a000640 1 --2- 172.21.15.27:0/4121370306 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4e24067190 0x7f4e24069650 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c15a250 tx=0x7f4e34007d60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:41.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:41.867+0000 7f4e42a00640 1 -- 172.21.15.27:0/4121370306 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4e3c0519f0 con 0x7f4e4c099a10 2024-01-15T18:12:42.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:42.055+0000 7f4e51a00640 1 -- 172.21.15.27:0/4121370306 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone51933", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e4c06f930 con 0x7f4e24067190 2024-01-15T18:12:43.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:43.187+0000 7f4e42a00640 1 -- 172.21.15.27:0/4121370306 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 23) v1 ==== 87027+0+0 (secure 0 0 0) 0x7f4e3c051dc0 con 0x7f4e4c099a10 2024-01-15T18:12:45.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.435+0000 7f4e42a00640 1 -- 172.21.15.27:0/4121370306 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4e4c06f930 con 0x7f4e24067190 2024-01-15T18:12:45.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.435+0000 7f4e51a00640 1 -- 172.21.15.27:0/4121370306 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4e24067190 msgr2=0x7f4e24069650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:45.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.435+0000 7f4e51a00640 1 --2- 172.21.15.27:0/4121370306 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4e24067190 0x7f4e24069650 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c15a250 tx=0x7f4e34007d60 comp rx=0 tx=0).stop 2024-01-15T18:12:45.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.435+0000 7f4e51a00640 1 -- 172.21.15.27:0/4121370306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e4c099a10 msgr2=0x7f4e4c155670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:45.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.435+0000 7f4e51a00640 1 --2- 172.21.15.27:0/4121370306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e4c099a10 0x7f4e4c155670 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c00e870 tx=0x7f4e3c00ec70 comp rx=0 tx=0).stop 2024-01-15T18:12:45.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.435+0000 7f4e51a00640 1 -- 172.21.15.27:0/4121370306 shutdown_connections 2024-01-15T18:12:45.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.435+0000 7f4e51a00640 1 --2- 172.21.15.27:0/4121370306 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4e24067190 0x7f4e24069650 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:45.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.435+0000 7f4e51a00640 1 --2- 172.21.15.27:0/4121370306 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e4c15b350 0x7f4e4c0b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:45.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.435+0000 7f4e51a00640 1 --2- 172.21.15.27:0/4121370306 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e4c09a5b0 0x7f4e4c0b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:45.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.435+0000 7f4e51a00640 1 --2- 172.21.15.27:0/4121370306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e4c099a10 0x7f4e4c155670 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:45.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.435+0000 7f4e51a00640 1 -- 172.21.15.27:0/4121370306 >> 172.21.15.27:0/4121370306 conn(0x7f4e4c0b2a80 msgr2=0x7f4e4c0cc500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:45.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.435+0000 7f4e51a00640 1 -- 172.21.15.27:0/4121370306 shutdown_connections 2024-01-15T18:12:45.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.435+0000 7f4e51a00640 1 -- 172.21.15.27:0/4121370306 wait complete. 2024-01-15T18:12:45.463 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone51934 2024-01-15T18:12:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.611+0000 7f4dd8800640 1 Processor -- start 2024-01-15T18:12:45.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.611+0000 7f4dd8800640 1 -- start start 2024-01-15T18:12:45.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.615+0000 7f4dd8800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4dd00ccee0 0x7f4dd00cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:45.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.615+0000 7f4dd8800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4dd006eac0 0x7f4dd00cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:45.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.615+0000 7f4dd8800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd006a700 con 0x7f4dd00cb750 2024-01-15T18:12:45.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.615+0000 7f4dd8800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd006b2a0 con 0x7f4dd00ccee0 2024-01-15T18:12:45.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.615+0000 7f4dd8800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd006c100 con 0x7f4dd006eac0 2024-01-15T18:12:45.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.615+0000 7f4dd5e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4dd00ccee0 0x7f4dd00cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:45.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.615+0000 7f4dd5400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4dd00cb750 0x7f4dd0071c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58820/0 (socket says 172.21.15.27:58820) 2024-01-15T18:12:45.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.615+0000 7f4dd5400640 1 -- 172.21.15.27:0/3560984971 learned_addr learned my addr 172.21.15.27:0/3560984971 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:12:45.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.615+0000 7f4dd7200640 1 --2- 172.21.15.27:0/3560984971 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4dd006eac0 0x7f4dd00cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:45.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.615+0000 7f4dd5e00640 1 -- 172.21.15.27:0/3560984971 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4dd006eac0 msgr2=0x7f4dd00cbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:45.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.615+0000 7f4dd5e00640 1 --2- 172.21.15.27:0/3560984971 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4dd006eac0 0x7f4dd00cbbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:45.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.615+0000 7f4dd5e00640 1 -- 172.21.15.27:0/3560984971 >> v1:172.21.15.27:6789/0 conn(0x7f4dd00cb750 legacy=0x7f4dd0071c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:45.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.615+0000 7f4dd4a00640 1 -- 172.21.15.27:0/3560984971 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3696204183 0 0) 0x7f4dd006a700 con 0x7f4dd00cb750 2024-01-15T18:12:45.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.615+0000 7f4dd5e00640 1 -- 172.21.15.27:0/3560984971 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd006b4f0 con 0x7f4dd00ccee0 2024-01-15T18:12:45.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.619+0000 7f4dd5e00640 1 --2- 172.21.15.27:0/3560984971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd00ccee0 0x7f4dd00cd2c0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f4dc00097c0 tx=0x7f4dc001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7be728e19abf3eab server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:45.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.619+0000 7f4dd4a00640 1 -- 172.21.15.27:0/3560984971 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc00a8970 con 0x7f4dd00ccee0 2024-01-15T18:12:45.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.619+0000 7f4dd4a00640 1 -- 172.21.15.27:0/3560984971 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc00a8ad0 con 0x7f4dd00ccee0 2024-01-15T18:12:45.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.619+0000 7f4dd4a00640 1 -- 172.21.15.27:0/3560984971 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc00a8e60 con 0x7f4dd00ccee0 2024-01-15T18:12:45.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.619+0000 7f4dd8800640 1 -- 172.21.15.27:0/3560984971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd00ccee0 msgr2=0x7f4dd00cd2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:45.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.619+0000 7f4dd8800640 1 --2- 172.21.15.27:0/3560984971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd00ccee0 0x7f4dd00cd2c0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f4dc00097c0 tx=0x7f4dc001ec70 comp rx=0 tx=0).stop 2024-01-15T18:12:45.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.619+0000 7f4dd8800640 1 -- 172.21.15.27:0/3560984971 shutdown_connections 2024-01-15T18:12:45.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.619+0000 7f4dd8800640 1 --2- 172.21.15.27:0/3560984971 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4dd006eac0 0x7f4dd00cbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:45.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.619+0000 7f4dd8800640 1 --2- 172.21.15.27:0/3560984971 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd00ccee0 0x7f4dd00cd2c0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:45.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.619+0000 7f4dd8800640 1 -- 172.21.15.27:0/3560984971 >> 172.21.15.27:0/3560984971 conn(0x7f4dd0099a10 msgr2=0x7f4dd00b2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:45.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.619+0000 7f4dd8800640 1 -- 172.21.15.27:0/3560984971 shutdown_connections 2024-01-15T18:12:45.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.619+0000 7f4dd8800640 1 -- 172.21.15.27:0/3560984971 wait complete. 2024-01-15T18:12:45.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.619+0000 7f4dd8800640 1 Processor -- start 2024-01-15T18:12:45.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.619+0000 7f4dd8800640 1 -- start start 2024-01-15T18:12:45.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.623+0000 7f4dd8800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd006eac0 0x7f4dd00c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:45.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.623+0000 7f4dd8800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4dd00cb750 0x7f4dd00b4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:45.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.623+0000 7f4dd8800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4dd00ccee0 0x7f4dd01f1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:45.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.623+0000 7f4dd8800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd00c6320 con 0x7f4dd006eac0 2024-01-15T18:12:45.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.623+0000 7f4dd8800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd015e550 con 0x7f4dd00ccee0 2024-01-15T18:12:45.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.623+0000 7f4dd8800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd00c6550 con 0x7f4dd00cb750 2024-01-15T18:12:45.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.623+0000 7f4dd5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd006eac0 0x7f4dd00c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:45.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.623+0000 7f4dd5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd006eac0 0x7f4dd00c5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52154/0 (socket says 172.21.15.27:52154) 2024-01-15T18:12:45.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.623+0000 7f4dd5e00640 1 -- 172.21.15.27:0/166022152 learned_addr learned my addr 172.21.15.27:0/166022152 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:45.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.623+0000 7f4dd5400640 1 --2- 172.21.15.27:0/166022152 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4dd00cb750 0x7f4dd00b4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:45.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.623+0000 7f4dd7200640 1 --2- 172.21.15.27:0/166022152 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4dd00ccee0 0x7f4dd01f1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:45.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.623+0000 7f4dd5e00640 1 -- 172.21.15.27:0/166022152 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4dd00cb750 msgr2=0x7f4dd00b4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:45.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.623+0000 7f4dd5e00640 1 --2- 172.21.15.27:0/166022152 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4dd00cb750 0x7f4dd00b4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:45.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.623+0000 7f4dd5e00640 1 -- 172.21.15.27:0/166022152 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4dd00ccee0 msgr2=0x7f4dd01f1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:45.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.623+0000 7f4dd5e00640 1 --2- 172.21.15.27:0/166022152 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4dd00ccee0 0x7f4dd01f1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:45.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.623+0000 7f4dd5e00640 1 -- 172.21.15.27:0/166022152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc0009420 con 0x7f4dd006eac0 2024-01-15T18:12:45.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.623+0000 7f4dd5400640 1 --2- 172.21.15.27:0/166022152 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4dd00cb750 0x7f4dd00b4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:12:45.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.623+0000 7f4dd5e00640 1 --2- 172.21.15.27:0/166022152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd006eac0 0x7f4dd00c5de0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f4dc001c8a0 tx=0x7f4dc001d4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:45.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.623+0000 7f4dce000640 1 -- 172.21.15.27:0/166022152 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc00b1e00 con 0x7f4dd006eac0 2024-01-15T18:12:45.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.623+0000 7f4dce000640 1 -- 172.21.15.27:0/166022152 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc0006e20 con 0x7f4dd006eac0 2024-01-15T18:12:45.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.623+0000 7f4dd8800640 1 -- 172.21.15.27:0/166022152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd00c6830 con 0x7f4dd006eac0 2024-01-15T18:12:45.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.623+0000 7f4dce000640 1 -- 172.21.15.27:0/166022152 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0004230 con 0x7f4dd006eac0 2024-01-15T18:12:45.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.623+0000 7f4dd8800640 1 -- 172.21.15.27:0/166022152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd00c6df0 con 0x7f4dd006eac0 2024-01-15T18:12:45.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.623+0000 7f4dd8800640 1 -- 172.21.15.27:0/166022152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd004e8c0 con 0x7f4dd006eac0 2024-01-15T18:12:45.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.627+0000 7f4dce000640 1 -- 172.21.15.27:0/166022152 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) v1 ==== 87027+0+0 (secure 0 0 0) 0x7f4dc0004390 con 0x7f4dd006eac0 2024-01-15T18:12:45.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.627+0000 7f4dce000640 1 --2- 172.21.15.27:0/166022152 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4da8067280 0x7f4da8069740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:45.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.627+0000 7f4dce000640 1 -- 172.21.15.27:0/166022152 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f4dc01205e0 con 0x7f4dd006eac0 2024-01-15T18:12:45.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.627+0000 7f4dd5400640 1 --2- 172.21.15.27:0/166022152 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4da8067280 0x7f4da8069740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:45.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.627+0000 7f4dd5400640 1 --2- 172.21.15.27:0/166022152 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4da8067280 0x7f4da8069740 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f4db8007fd0 tx=0x7f4db8007d60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:45.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.631+0000 7f4dce000640 1 -- 172.21.15.27:0/166022152 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4dc00ef180 con 0x7f4dd006eac0 2024-01-15T18:12:45.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:45.819+0000 7f4dd8800640 1 -- 172.21.15.27:0/166022152 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone51934", "target": ["mon-mgr", ""]}) v1 -- 0x7f4dd006bd90 con 0x7f4da8067280 2024-01-15T18:12:50.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.427+0000 7f4dce000640 1 -- 172.21.15.27:0/166022152 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4db8007320 con 0x7f4da8067280 2024-01-15T18:12:50.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.431+0000 7f4dd8800640 1 -- 172.21.15.27:0/166022152 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4da8067280 msgr2=0x7f4da8069740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:50.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.431+0000 7f4dd8800640 1 --2- 172.21.15.27:0/166022152 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4da8067280 0x7f4da8069740 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f4db8007fd0 tx=0x7f4db8007d60 comp rx=0 tx=0).stop 2024-01-15T18:12:50.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.431+0000 7f4dd8800640 1 -- 172.21.15.27:0/166022152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd006eac0 msgr2=0x7f4dd00c5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:50.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.431+0000 7f4dd8800640 1 --2- 172.21.15.27:0/166022152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd006eac0 0x7f4dd00c5de0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f4dc001c8a0 tx=0x7f4dc001d4b0 comp rx=0 tx=0).stop 2024-01-15T18:12:50.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.431+0000 7f4dd8800640 1 -- 172.21.15.27:0/166022152 shutdown_connections 2024-01-15T18:12:50.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.431+0000 7f4dd8800640 1 --2- 172.21.15.27:0/166022152 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4da8067280 0x7f4da8069740 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:50.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.431+0000 7f4dd8800640 1 --2- 172.21.15.27:0/166022152 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4dd00ccee0 0x7f4dd01f1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:50.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.431+0000 7f4dd8800640 1 --2- 172.21.15.27:0/166022152 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4dd00cb750 0x7f4dd00b4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:50.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.431+0000 7f4dd8800640 1 --2- 172.21.15.27:0/166022152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd006eac0 0x7f4dd00c5de0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:50.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.431+0000 7f4dd8800640 1 -- 172.21.15.27:0/166022152 >> 172.21.15.27:0/166022152 conn(0x7f4dd0099a10 msgr2=0x7f4dd00b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:50.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.431+0000 7f4dd8800640 1 -- 172.21.15.27:0/166022152 shutdown_connections 2024-01-15T18:12:50.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.431+0000 7f4dd8800640 1 -- 172.21.15.27:0/166022152 wait complete. 2024-01-15T18:12:50.455 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone51935 2024-01-15T18:12:50.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.603+0000 7fdb68000640 1 Processor -- start 2024-01-15T18:12:50.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.603+0000 7fdb68000640 1 -- start start 2024-01-15T18:12:50.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.607+0000 7fdb68000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb600b4300 0x7fdb600cf940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:50.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.607+0000 7fdb68000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fdb60071c50 0x7fdb60072070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:50.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.607+0000 7fdb68000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdb6015b730 0x7fdb6015bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:50.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.607+0000 7fdb68000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb6006a4f0 con 0x7fdb6015b730 2024-01-15T18:12:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.607+0000 7fdb68000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb600cebf0 con 0x7fdb60071c50 2024-01-15T18:12:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.607+0000 7fdb68000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb600725b0 con 0x7fdb600b4300 2024-01-15T18:12:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.607+0000 7fdb66a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdb6015b730 0x7fdb6015bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.607+0000 7fdb66a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdb6015b730 0x7fdb6015bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52164/0 (socket says 172.21.15.27:52164) 2024-01-15T18:12:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.607+0000 7fdb66a00640 1 -- 172.21.15.27:0/296032444 learned_addr learned my addr 172.21.15.27:0/296032444 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.607+0000 7fdb64c00640 1 --2- 172.21.15.27:0/296032444 >> v2:172.21.15.93:3300/0 conn(0x7fdb60071c50 0x7fdb60072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.607+0000 7fdb65600640 1 --2- 172.21.15.27:0/296032444 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb600b4300 0x7fdb600cf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.607+0000 7fdb66a00640 1 -- 172.21.15.27:0/296032444 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb600b4300 msgr2=0x7fdb600cf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.607+0000 7fdb66a00640 1 --2- 172.21.15.27:0/296032444 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb600b4300 0x7fdb600cf940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.607+0000 7fdb66a00640 1 -- 172.21.15.27:0/296032444 >> v2:172.21.15.93:3300/0 conn(0x7fdb60071c50 msgr2=0x7fdb60072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.607+0000 7fdb66a00640 1 --2- 172.21.15.27:0/296032444 >> v2:172.21.15.93:3300/0 conn(0x7fdb60071c50 0x7fdb60072070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.607+0000 7fdb66a00640 1 -- 172.21.15.27:0/296032444 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb600b46e0 con 0x7fdb6015b730 2024-01-15T18:12:50.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.611+0000 7fdb66a00640 1 --2- 172.21.15.27:0/296032444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb6015b730 0x7fdb6015bb90 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fdb6006fce0 tx=0x7fdb5801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55fe5c5a04cc2c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:50.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.611+0000 7fdb5fe00640 1 -- 172.21.15.27:0/296032444 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb580af400 con 0x7fdb6015b730 2024-01-15T18:12:50.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.611+0000 7fdb5fe00640 1 -- 172.21.15.27:0/296032444 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb580ab050 con 0x7fdb6015b730 2024-01-15T18:12:50.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.611+0000 7fdb5fe00640 1 -- 172.21.15.27:0/296032444 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb580af780 con 0x7fdb6015b730 2024-01-15T18:12:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.611+0000 7fdb68000640 1 -- 172.21.15.27:0/296032444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb6015b730 msgr2=0x7fdb6015bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.611+0000 7fdb68000640 1 --2- 172.21.15.27:0/296032444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb6015b730 0x7fdb6015bb90 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fdb6006fce0 tx=0x7fdb5801f1c0 comp rx=0 tx=0).stop 2024-01-15T18:12:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.611+0000 7fdb68000640 1 -- 172.21.15.27:0/296032444 shutdown_connections 2024-01-15T18:12:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.611+0000 7fdb68000640 1 --2- 172.21.15.27:0/296032444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb6015b730 0x7fdb6015bb90 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.611+0000 7fdb68000640 1 --2- 172.21.15.27:0/296032444 >> v2:172.21.15.93:3300/0 conn(0x7fdb60071c50 0x7fdb60072070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.611+0000 7fdb68000640 1 --2- 172.21.15.27:0/296032444 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb600b4300 0x7fdb600cf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.611+0000 7fdb68000640 1 -- 172.21.15.27:0/296032444 >> 172.21.15.27:0/296032444 conn(0x7fdb60099e10 msgr2=0x7fdb600b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.611+0000 7fdb68000640 1 -- 172.21.15.27:0/296032444 shutdown_connections 2024-01-15T18:12:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.611+0000 7fdb68000640 1 -- 172.21.15.27:0/296032444 wait complete. 2024-01-15T18:12:50.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.611+0000 7fdb68000640 1 Processor -- start 2024-01-15T18:12:50.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.611+0000 7fdb68000640 1 -- start start 2024-01-15T18:12:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.615+0000 7fdb68000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb60071c50 0x7fdb6014f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.615+0000 7fdb68000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdb600b4300 0x7fdb601e53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.615+0000 7fdb68000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb6015b730 0x7fdb60154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.615+0000 7fdb68000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb601e5e80 con 0x7fdb60071c50 2024-01-15T18:12:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.615+0000 7fdb68000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb601e6160 con 0x7fdb600b4300 2024-01-15T18:12:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.615+0000 7fdb68000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb601e6440 con 0x7fdb6015b730 2024-01-15T18:12:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.615+0000 7fdb65600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb60071c50 0x7fdb6014f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:50.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.615+0000 7fdb64c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdb600b4300 0x7fdb601e53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:50.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.615+0000 7fdb65600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb60071c50 0x7fdb6014f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52170/0 (socket says 172.21.15.27:52170) 2024-01-15T18:12:50.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.615+0000 7fdb65600640 1 -- 172.21.15.27:0/3163668205 learned_addr learned my addr 172.21.15.27:0/3163668205 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:50.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.615+0000 7fdb66a00640 1 --2- 172.21.15.27:0/3163668205 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb6015b730 0x7fdb60154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:50.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.615+0000 7fdb65600640 1 -- 172.21.15.27:0/3163668205 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb6015b730 msgr2=0x7fdb60154290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:50.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.615+0000 7fdb65600640 1 --2- 172.21.15.27:0/3163668205 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb6015b730 0x7fdb60154290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:50.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.615+0000 7fdb65600640 1 -- 172.21.15.27:0/3163668205 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdb600b4300 msgr2=0x7fdb601e53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:50.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.615+0000 7fdb65600640 1 --2- 172.21.15.27:0/3163668205 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdb600b4300 0x7fdb601e53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:50.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.615+0000 7fdb65600640 1 -- 172.21.15.27:0/3163668205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb58009b30 con 0x7fdb60071c50 2024-01-15T18:12:50.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.615+0000 7fdb65600640 1 --2- 172.21.15.27:0/3163668205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb60071c50 0x7fdb6014f750 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fdb5000e930 tx=0x7fdb5000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:50.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.615+0000 7fdb5d600640 1 -- 172.21.15.27:0/3163668205 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb50009690 con 0x7fdb60071c50 2024-01-15T18:12:50.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.615+0000 7fdb5d600640 1 -- 172.21.15.27:0/3163668205 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb5000ca00 con 0x7fdb60071c50 2024-01-15T18:12:50.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.615+0000 7fdb5d600640 1 -- 172.21.15.27:0/3163668205 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5000cd70 con 0x7fdb60071c50 2024-01-15T18:12:50.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.615+0000 7fdb68000640 1 -- 172.21.15.27:0/3163668205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb601e6720 con 0x7fdb60071c50 2024-01-15T18:12:50.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.615+0000 7fdb68000640 1 -- 172.21.15.27:0/3163668205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb60157b50 con 0x7fdb60071c50 2024-01-15T18:12:50.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.615+0000 7fdb68000640 1 -- 172.21.15.27:0/3163668205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb60150170 con 0x7fdb60071c50 2024-01-15T18:12:50.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.615+0000 7fdb5d600640 1 -- 172.21.15.27:0/3163668205 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) v1 ==== 87027+0+0 (secure 0 0 0) 0x7fdb50013470 con 0x7fdb60071c50 2024-01-15T18:12:50.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.619+0000 7fdb5d600640 1 --2- 172.21.15.27:0/3163668205 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdb38067280 0x7fdb38069740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:50.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.619+0000 7fdb5d600640 1 -- 172.21.15.27:0/3163668205 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fdb500830d0 con 0x7fdb60071c50 2024-01-15T18:12:50.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.619+0000 7fdb64c00640 1 --2- 172.21.15.27:0/3163668205 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdb38067280 0x7fdb38069740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:50.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.619+0000 7fdb64c00640 1 --2- 172.21.15.27:0/3163668205 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdb38067280 0x7fdb38069740 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fdb600b1e20 tx=0x7fdb48009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:50.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.623+0000 7fdb5d600640 1 -- 172.21.15.27:0/3163668205 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fdb50051cd0 con 0x7fdb60071c50 2024-01-15T18:12:50.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:50.807+0000 7fdb68000640 1 -- 172.21.15.27:0/3163668205 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone51935", "target": ["mon-mgr", ""]}) v1 -- 0x7fdb600cf420 con 0x7fdb38067280 2024-01-15T18:12:55.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.439+0000 7fdb5d600640 1 -- 172.21.15.27:0/3163668205 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdb48009810 con 0x7fdb38067280 2024-01-15T18:12:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.439+0000 7fdb68000640 1 -- 172.21.15.27:0/3163668205 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdb38067280 msgr2=0x7fdb38069740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.439+0000 7fdb68000640 1 --2- 172.21.15.27:0/3163668205 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdb38067280 0x7fdb38069740 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fdb600b1e20 tx=0x7fdb48009190 comp rx=0 tx=0).stop 2024-01-15T18:12:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.439+0000 7fdb68000640 1 -- 172.21.15.27:0/3163668205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb60071c50 msgr2=0x7fdb6014f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:55.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.439+0000 7fdb68000640 1 --2- 172.21.15.27:0/3163668205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb60071c50 0x7fdb6014f750 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fdb5000e930 tx=0x7fdb5000edf0 comp rx=0 tx=0).stop 2024-01-15T18:12:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.439+0000 7fdb68000640 1 -- 172.21.15.27:0/3163668205 shutdown_connections 2024-01-15T18:12:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.439+0000 7fdb68000640 1 --2- 172.21.15.27:0/3163668205 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdb38067280 0x7fdb38069740 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.439+0000 7fdb68000640 1 --2- 172.21.15.27:0/3163668205 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb6015b730 0x7fdb60154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.439+0000 7fdb68000640 1 --2- 172.21.15.27:0/3163668205 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdb600b4300 0x7fdb601e53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.439+0000 7fdb68000640 1 --2- 172.21.15.27:0/3163668205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb60071c50 0x7fdb6014f750 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.439+0000 7fdb68000640 1 -- 172.21.15.27:0/3163668205 >> 172.21.15.27:0/3163668205 conn(0x7fdb60099e10 msgr2=0x7fdb600cd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.443+0000 7fdb68000640 1 -- 172.21.15.27:0/3163668205 shutdown_connections 2024-01-15T18:12:55.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.443+0000 7fdb68000640 1 -- 172.21.15.27:0/3163668205 wait complete. 2024-01-15T18:12:55.468 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone51936 2024-01-15T18:12:55.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.623+0000 7fa0eaa00640 1 Processor -- start 2024-01-15T18:12:55.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.623+0000 7fa0eaa00640 1 -- start start 2024-01-15T18:12:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.623+0000 7fa0eaa00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa0e415ad20 0x7fa0e415b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.623+0000 7fa0eaa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa0e40715f0 0x7fa0e415dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.623+0000 7fa0eaa00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e415a610 con 0x7fa0e415d900 2024-01-15T18:12:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.623+0000 7fa0eaa00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0e415e3c0 con 0x7fa0e40715f0 2024-01-15T18:12:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.623+0000 7fa0eaa00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e415e6a0 con 0x7fa0e415ad20 2024-01-15T18:12:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.623+0000 7fa0e8a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa0e40715f0 0x7fa0e415dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:55.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.623+0000 7fa0e8a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa0e40715f0 0x7fa0e415dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46528/0 (socket says 172.21.15.27:46528) 2024-01-15T18:12:55.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.623+0000 7fa0e8a00640 1 -- 172.21.15.27:0/673973997 learned_addr learned my addr 172.21.15.27:0/673973997 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:55.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.623+0000 7fa0e3e00640 1 --2- 172.21.15.27:0/673973997 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa0e415ad20 0x7fa0e415b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:55.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.623+0000 7fa0e8a00640 1 -- 172.21.15.27:0/673973997 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa0e415ad20 msgr2=0x7fa0e415b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:55.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.623+0000 7fa0e8a00640 1 --2- 172.21.15.27:0/673973997 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa0e415ad20 0x7fa0e415b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:55.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.623+0000 7fa0e8a00640 1 -- 172.21.15.27:0/673973997 >> v1:172.21.15.93:6789/0 conn(0x7fa0e415d900 legacy=0x7fa0e4070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:55.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.623+0000 7fa0e8a00640 1 -- 172.21.15.27:0/673973997 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e415b6e0 con 0x7fa0e40715f0 2024-01-15T18:12:55.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.623+0000 7fa0e3e00640 1 --2- 172.21.15.27:0/673973997 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa0e415ad20 0x7fa0e415b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:55.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.627+0000 7fa0e8a00640 1 --2- 172.21.15.27:0/673973997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0e40715f0 0x7fa0e415dd80 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc009e50 tx=0x7fa0dc01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e36ddcabf5b18924 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:55.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.627+0000 7fa0e2a00640 1 -- 172.21.15.27:0/673973997 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc0a5840 con 0x7fa0e40715f0 2024-01-15T18:12:55.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.627+0000 7fa0e2a00640 1 -- 172.21.15.27:0/673973997 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0dc0a59a0 con 0x7fa0e40715f0 2024-01-15T18:12:55.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.627+0000 7fa0e2a00640 1 -- 172.21.15.27:0/673973997 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc0a5d30 con 0x7fa0e40715f0 2024-01-15T18:12:55.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.627+0000 7fa0eaa00640 1 -- 172.21.15.27:0/673973997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0e40715f0 msgr2=0x7fa0e415dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:55.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.627+0000 7fa0eaa00640 1 --2- 172.21.15.27:0/673973997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0e40715f0 0x7fa0e415dd80 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc009e50 tx=0x7fa0dc01c9a0 comp rx=0 tx=0).stop 2024-01-15T18:12:55.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.627+0000 7fa0eaa00640 1 -- 172.21.15.27:0/673973997 shutdown_connections 2024-01-15T18:12:55.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.627+0000 7fa0eaa00640 1 --2- 172.21.15.27:0/673973997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0e40715f0 0x7fa0e415dd80 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:55.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.627+0000 7fa0eaa00640 1 --2- 172.21.15.27:0/673973997 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa0e415ad20 0x7fa0e415b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:55.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.627+0000 7fa0eaa00640 1 -- 172.21.15.27:0/673973997 >> 172.21.15.27:0/673973997 conn(0x7fa0e40cb700 msgr2=0x7fa0e40cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:55.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.627+0000 7fa0eaa00640 1 -- 172.21.15.27:0/673973997 shutdown_connections 2024-01-15T18:12:55.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.627+0000 7fa0eaa00640 1 -- 172.21.15.27:0/673973997 wait complete. 2024-01-15T18:12:55.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.631+0000 7fa0eaa00640 1 Processor -- start 2024-01-15T18:12:55.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.631+0000 7fa0eaa00640 1 -- start start 2024-01-15T18:12:55.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.631+0000 7fa0eaa00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa0e40715f0 0x7fa0e4156670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:55.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.631+0000 7fa0eaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0e415ad20 0x7fa0e40ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:55.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.631+0000 7fa0eaa00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa0e415d900 0x7fa0e41529b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.631+0000 7fa0eaa00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e4156250 con 0x7fa0e415ad20 2024-01-15T18:12:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.631+0000 7fa0eaa00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e40cdc90 con 0x7fa0e40715f0 2024-01-15T18:12:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.631+0000 7fa0eaa00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e40cd6d0 con 0x7fa0e415d900 2024-01-15T18:12:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.631+0000 7fa0e3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0e415ad20 0x7fa0e40ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.631+0000 7fa0e3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0e415ad20 0x7fa0e40ccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46536/0 (socket says 172.21.15.27:46536) 2024-01-15T18:12:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.631+0000 7fa0e3400640 1 -- 172.21.15.27:0/711654970 learned_addr learned my addr 172.21.15.27:0/711654970 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.631+0000 7fa0e8a00640 1 --2- 172.21.15.27:0/711654970 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa0e415d900 0x7fa0e41529b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.631+0000 7fa0e3e00640 1 --2- 172.21.15.27:0/711654970 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa0e40715f0 0x7fa0e4156670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.631+0000 7fa0e3400640 1 -- 172.21.15.27:0/711654970 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa0e415d900 msgr2=0x7fa0e41529b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.631+0000 7fa0e3400640 1 --2- 172.21.15.27:0/711654970 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa0e415d900 0x7fa0e41529b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.631+0000 7fa0e3400640 1 -- 172.21.15.27:0/711654970 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa0e40715f0 msgr2=0x7fa0e4156670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.631+0000 7fa0e3400640 1 --2- 172.21.15.27:0/711654970 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa0e40715f0 0x7fa0e4156670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.631+0000 7fa0e3400640 1 -- 172.21.15.27:0/711654970 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0dc009b30 con 0x7fa0e415ad20 2024-01-15T18:12:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.631+0000 7fa0e3400640 1 --2- 172.21.15.27:0/711654970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0e415ad20 0x7fa0e40ccd60 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc00b410 tx=0x7fa0cc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.631+0000 7fa0dbe00640 1 -- 172.21.15.27:0/711654970 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc004020 con 0x7fa0e415ad20 2024-01-15T18:12:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.631+0000 7fa0dbe00640 1 -- 172.21.15.27:0/711654970 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0cc004180 con 0x7fa0e415ad20 2024-01-15T18:12:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.631+0000 7fa0eaa00640 1 -- 172.21.15.27:0/711654970 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e40cd9b0 con 0x7fa0e415ad20 2024-01-15T18:12:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.631+0000 7fa0dbe00640 1 -- 172.21.15.27:0/711654970 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc011630 con 0x7fa0e415ad20 2024-01-15T18:12:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.631+0000 7fa0eaa00640 1 -- 172.21.15.27:0/711654970 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e4156bb0 con 0x7fa0e415ad20 2024-01-15T18:12:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.631+0000 7fa0eaa00640 1 -- 172.21.15.27:0/711654970 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e4156390 con 0x7fa0e415ad20 2024-01-15T18:12:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.635+0000 7fa0dbe00640 1 -- 172.21.15.27:0/711654970 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) v1 ==== 87027+0+0 (secure 0 0 0) 0x7fa0cc011790 con 0x7fa0e415ad20 2024-01-15T18:12:55.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.635+0000 7fa0dbe00640 1 --2- 172.21.15.27:0/711654970 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0bc067280 0x7fa0bc069740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:55.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.635+0000 7fa0dbe00640 1 -- 172.21.15.27:0/711654970 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fa0cc082bb0 con 0x7fa0e415ad20 2024-01-15T18:12:55.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.635+0000 7fa0e3e00640 1 --2- 172.21.15.27:0/711654970 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0bc067280 0x7fa0bc069740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.635+0000 7fa0e3e00640 1 --2- 172.21.15.27:0/711654970 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0bc067280 0x7fa0bc069740 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fa0e40cc180 tx=0x7fa0d4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:55.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.639+0000 7fa0dbe00640 1 -- 172.21.15.27:0/711654970 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa0e4156390 con 0x7fa0e415ad20 2024-01-15T18:12:55.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.827+0000 7fa0eaa00640 1 -- 172.21.15.27:0/711654970 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone51936", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0e4156390 con 0x7fa0bc067280 2024-01-15T18:12:55.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.895+0000 7fa0dbe00640 1 -- 172.21.15.27:0/711654970 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa0e4156390 con 0x7fa0bc067280 2024-01-15T18:12:55.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.899+0000 7fa0eaa00640 1 -- 172.21.15.27:0/711654970 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0bc067280 msgr2=0x7fa0bc069740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:55.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.899+0000 7fa0eaa00640 1 --2- 172.21.15.27:0/711654970 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0bc067280 0x7fa0bc069740 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fa0e40cc180 tx=0x7fa0d4008040 comp rx=0 tx=0).stop 2024-01-15T18:12:55.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.899+0000 7fa0eaa00640 1 -- 172.21.15.27:0/711654970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0e415ad20 msgr2=0x7fa0e40ccd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:55.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.899+0000 7fa0eaa00640 1 --2- 172.21.15.27:0/711654970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0e415ad20 0x7fa0e40ccd60 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc00b410 tx=0x7fa0cc00b8d0 comp rx=0 tx=0).stop 2024-01-15T18:12:55.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.899+0000 7fa0eaa00640 1 -- 172.21.15.27:0/711654970 shutdown_connections 2024-01-15T18:12:55.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.899+0000 7fa0eaa00640 1 --2- 172.21.15.27:0/711654970 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0bc067280 0x7fa0bc069740 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:55.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.899+0000 7fa0eaa00640 1 --2- 172.21.15.27:0/711654970 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa0e415d900 0x7fa0e41529b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:55.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.899+0000 7fa0eaa00640 1 --2- 172.21.15.27:0/711654970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa0e415ad20 0x7fa0e40ccd60 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:55.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.899+0000 7fa0eaa00640 1 --2- 172.21.15.27:0/711654970 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa0e40715f0 0x7fa0e4156670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:55.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.899+0000 7fa0eaa00640 1 -- 172.21.15.27:0/711654970 >> 172.21.15.27:0/711654970 conn(0x7fa0e40cb700 msgr2=0x7fa0e4070ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:55.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.899+0000 7fa0eaa00640 1 -- 172.21.15.27:0/711654970 shutdown_connections 2024-01-15T18:12:55.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:55.899+0000 7fa0eaa00640 1 -- 172.21.15.27:0/711654970 wait complete. 2024-01-15T18:12:55.919 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone51937 2024-01-15T18:12:56.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.067+0000 7f9aa7400640 1 Processor -- start 2024-01-15T18:12:56.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.067+0000 7f9aa7400640 1 -- start start 2024-01-15T18:12:56.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.067+0000 7f9aa7400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9aa00d1a40 0x7f9aa00cba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:56.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.067+0000 7f9aa7400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f9aa00b0650 0x7f9aa00b0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:56.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.067+0000 7f9aa7400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa00cbfa0 con 0x7f9aa00cf450 2024-01-15T18:12:56.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.067+0000 7f9aa7400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aa00cefa0 con 0x7f9aa00b0650 2024-01-15T18:12:56.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.067+0000 7f9aa7400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa00ce240 con 0x7f9aa00d1a40 2024-01-15T18:12:56.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.067+0000 7f9aa4a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9aa00cf450 0x7f9aa00cf830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42514/0 (socket says 172.21.15.27:42514) 2024-01-15T18:12:56.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.067+0000 7f9aa4a00640 1 -- 172.21.15.27:0/3240626756 learned_addr learned my addr 172.21.15.27:0/3240626756 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:12:56.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.067+0000 7f9aa5400640 1 --2- 172.21.15.27:0/3240626756 >> v2:172.21.15.93:3300/0 conn(0x7f9aa00b0650 0x7f9aa00b0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:56.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.067+0000 7f9a9fe00640 1 --2- 172.21.15.27:0/3240626756 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9aa00d1a40 0x7f9aa00cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:56.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.067+0000 7f9a9f400640 1 -- 172.21.15.27:0/3240626756 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3158995713 0 0) 0x7f9aa00cbfa0 con 0x7f9aa00cf450 2024-01-15T18:12:56.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.067+0000 7f9a9f400640 1 -- 172.21.15.27:0/3240626756 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a80003530 con 0x7f9aa00cf450 2024-01-15T18:12:56.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.067+0000 7f9a9fe00640 1 -- 172.21.15.27:0/3240626756 >> v2:172.21.15.93:3300/0 conn(0x7f9aa00b0650 msgr2=0x7f9aa00b0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:56.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.067+0000 7f9a9fe00640 1 --2- 172.21.15.27:0/3240626756 >> v2:172.21.15.93:3300/0 conn(0x7f9aa00b0650 0x7f9aa00b0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:56.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.067+0000 7f9a9fe00640 1 -- 172.21.15.27:0/3240626756 >> v1:172.21.15.27:6789/0 conn(0x7f9aa00cf450 legacy=0x7f9aa00cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:56.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.067+0000 7f9a9fe00640 1 -- 172.21.15.27:0/3240626756 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa00ccd70 con 0x7f9aa00d1a40 2024-01-15T18:12:56.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.067+0000 7f9aa5400640 1 --2- 172.21.15.27:0/3240626756 >> v2:172.21.15.93:3300/0 conn(0x7f9aa00b0650 0x7f9aa00b0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:12:56.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.071+0000 7f9a9fe00640 1 --2- 172.21.15.27:0/3240626756 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9aa00d1a40 0x7f9aa00cba60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9a880097c0 tx=0x7f9a8801c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=573d32528872236e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:56.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.071+0000 7f9a9f400640 1 -- 172.21.15.27:0/3240626756 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a880ab600 con 0x7f9aa00d1a40 2024-01-15T18:12:56.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.071+0000 7f9a9f400640 1 -- 172.21.15.27:0/3240626756 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a880a7050 con 0x7f9aa00d1a40 2024-01-15T18:12:56.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.071+0000 7f9a9f400640 1 -- 172.21.15.27:0/3240626756 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a880ab980 con 0x7f9aa00d1a40 2024-01-15T18:12:56.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.071+0000 7f9aa7400640 1 -- 172.21.15.27:0/3240626756 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9aa00d1a40 msgr2=0x7f9aa00cba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:56.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.071+0000 7f9aa7400640 1 --2- 172.21.15.27:0/3240626756 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9aa00d1a40 0x7f9aa00cba60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9a880097c0 tx=0x7f9a8801c480 comp rx=0 tx=0).stop 2024-01-15T18:12:56.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.071+0000 7f9aa7400640 1 -- 172.21.15.27:0/3240626756 shutdown_connections 2024-01-15T18:12:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.071+0000 7f9aa7400640 1 --2- 172.21.15.27:0/3240626756 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9aa00d1a40 0x7f9aa00cba60 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.071+0000 7f9aa7400640 1 --2- 172.21.15.27:0/3240626756 >> v2:172.21.15.93:3300/0 conn(0x7f9aa00b0650 0x7f9aa00b0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.071+0000 7f9aa7400640 1 -- 172.21.15.27:0/3240626756 >> 172.21.15.27:0/3240626756 conn(0x7f9aa00d0e70 msgr2=0x7f9aa00d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9aa7400640 1 -- 172.21.15.27:0/3240626756 shutdown_connections 2024-01-15T18:12:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9aa7400640 1 -- 172.21.15.27:0/3240626756 wait complete. 2024-01-15T18:12:56.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9aa7400640 1 Processor -- start 2024-01-15T18:12:56.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9aa7400640 1 -- start start 2024-01-15T18:12:56.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9aa7400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9aa00b0650 0x7f9aa014e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:56.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9aa7400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa00cf450 0x7f9aa006c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:56.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9aa7400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9aa00d1a40 0x7f9aa00ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:56.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9aa7400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa00b6560 con 0x7f9aa00cf450 2024-01-15T18:12:56.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9aa7400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa00b58e0 con 0x7f9aa00b0650 2024-01-15T18:12:56.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9aa7400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa00b5bc0 con 0x7f9aa00d1a40 2024-01-15T18:12:56.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9a9fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa00cf450 0x7f9aa006c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:56.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9a9fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa00cf450 0x7f9aa006c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46538/0 (socket says 172.21.15.27:46538) 2024-01-15T18:12:56.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9a9fe00640 1 -- 172.21.15.27:0/1652632090 learned_addr learned my addr 172.21.15.27:0/1652632090 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:56.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9aa5400640 1 --2- 172.21.15.27:0/1652632090 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9aa00d1a40 0x7f9aa00ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:56.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9aa4a00640 1 --2- 172.21.15.27:0/1652632090 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9aa00b0650 0x7f9aa014e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:56.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9a9fe00640 1 -- 172.21.15.27:0/1652632090 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9aa00d1a40 msgr2=0x7f9aa00ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:56.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9a9fe00640 1 --2- 172.21.15.27:0/1652632090 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9aa00d1a40 0x7f9aa00ca110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:56.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9a9fe00640 1 -- 172.21.15.27:0/1652632090 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9aa00b0650 msgr2=0x7f9aa014e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:56.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9a9fe00640 1 --2- 172.21.15.27:0/1652632090 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9aa00b0650 0x7f9aa014e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:56.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9a9fe00640 1 -- 172.21.15.27:0/1652632090 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a98009a70 con 0x7f9aa00cf450 2024-01-15T18:12:56.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9aa4a00640 1 --2- 172.21.15.27:0/1652632090 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9aa00b0650 0x7f9aa014e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:12:56.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9a9fe00640 1 --2- 172.21.15.27:0/1652632090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa00cf450 0x7f9aa006c010 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f9a8801ab30 tx=0x7f9a88019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:56.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9a9cc00640 1 -- 172.21.15.27:0/1652632090 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a88006e20 con 0x7f9aa00cf450 2024-01-15T18:12:56.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9a9cc00640 1 -- 172.21.15.27:0/1652632090 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a880b5470 con 0x7f9aa00cf450 2024-01-15T18:12:56.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9a9cc00640 1 -- 172.21.15.27:0/1652632090 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a880b5840 con 0x7f9aa00cf450 2024-01-15T18:12:56.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.075+0000 7f9aa7400640 1 -- 172.21.15.27:0/1652632090 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a88009420 con 0x7f9aa00cf450 2024-01-15T18:12:56.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.079+0000 7f9aa7400640 1 -- 172.21.15.27:0/1652632090 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa00b6180 con 0x7f9aa00cf450 2024-01-15T18:12:56.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.079+0000 7f9aa7400640 1 -- 172.21.15.27:0/1652632090 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa004e850 con 0x7f9aa00cf450 2024-01-15T18:12:56.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.079+0000 7f9a9cc00640 1 -- 172.21.15.27:0/1652632090 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) v1 ==== 87027+0+0 (secure 0 0 0) 0x7f9a880b2040 con 0x7f9aa00cf450 2024-01-15T18:12:56.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.079+0000 7f9a9cc00640 1 --2- 172.21.15.27:0/1652632090 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9a78067280 0x7f9a78069740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:56.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.079+0000 7f9a9cc00640 1 -- 172.21.15.27:0/1652632090 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f9a8811d250 con 0x7f9aa00cf450 2024-01-15T18:12:56.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.079+0000 7f9aa4a00640 1 --2- 172.21.15.27:0/1652632090 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9a78067280 0x7f9a78069740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:56.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.079+0000 7f9aa4a00640 1 --2- 172.21.15.27:0/1652632090 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9a78067280 0x7f9a78069740 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f9a900011f0 tx=0x7f9a90009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:56.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.083+0000 7f9a9cc00640 1 -- 172.21.15.27:0/1652632090 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9a880ebce0 con 0x7f9aa00cf450 2024-01-15T18:12:56.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.271+0000 7f9aa7400640 1 -- 172.21.15.27:0/1652632090 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone51937", "target": ["mon-mgr", ""]}) v1 -- 0x7f9aa00cca90 con 0x7f9a78067280 2024-01-15T18:12:56.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.331+0000 7f9a9cc00640 1 -- 172.21.15.27:0/1652632090 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9a90001d10 con 0x7f9a78067280 2024-01-15T18:12:56.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.335+0000 7f9aa7400640 1 -- 172.21.15.27:0/1652632090 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9a78067280 msgr2=0x7f9a78069740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:56.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.335+0000 7f9aa7400640 1 --2- 172.21.15.27:0/1652632090 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9a78067280 0x7f9a78069740 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f9a900011f0 tx=0x7f9a90009040 comp rx=0 tx=0).stop 2024-01-15T18:12:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.335+0000 7f9aa7400640 1 -- 172.21.15.27:0/1652632090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa00cf450 msgr2=0x7f9aa006c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.335+0000 7f9aa7400640 1 --2- 172.21.15.27:0/1652632090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa00cf450 0x7f9aa006c010 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f9a8801ab30 tx=0x7f9a88019220 comp rx=0 tx=0).stop 2024-01-15T18:12:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.335+0000 7f9aa7400640 1 -- 172.21.15.27:0/1652632090 shutdown_connections 2024-01-15T18:12:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.335+0000 7f9aa7400640 1 --2- 172.21.15.27:0/1652632090 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9a78067280 0x7f9a78069740 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.335+0000 7f9aa7400640 1 --2- 172.21.15.27:0/1652632090 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9aa00d1a40 0x7f9aa00ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.335+0000 7f9aa7400640 1 --2- 172.21.15.27:0/1652632090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa00cf450 0x7f9aa006c010 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.335+0000 7f9aa7400640 1 --2- 172.21.15.27:0/1652632090 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9aa00b0650 0x7f9aa014e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.335+0000 7f9aa7400640 1 -- 172.21.15.27:0/1652632090 >> 172.21.15.27:0/1652632090 conn(0x7f9aa00d0e70 msgr2=0x7f9aa015e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.335+0000 7f9aa7400640 1 -- 172.21.15.27:0/1652632090 shutdown_connections 2024-01-15T18:12:56.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.335+0000 7f9aa7400640 1 -- 172.21.15.27:0/1652632090 wait complete. 2024-01-15T18:12:56.355 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-15T18:12:56.355 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone51934 2024-01-15T18:12:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.499+0000 7f8e60000640 1 Processor -- start 2024-01-15T18:12:56.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.499+0000 7f8e60000640 1 -- start start 2024-01-15T18:12:56.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.499+0000 7f8e60000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f8e580cde10 0x7f8e580cbf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:56.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.499+0000 7f8e60000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e580cb4d0 con 0x7f8e580cfc30 2024-01-15T18:12:56.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.499+0000 7f8e60000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e580d0120 con 0x7f8e5815d680 2024-01-15T18:12:56.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.499+0000 7f8e60000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e580d0400 con 0x7f8e580cde10 2024-01-15T18:12:56.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.499+0000 7f8e5e000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8e580cfc30 0x7f8e580cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42524/0 (socket says 172.21.15.27:42524) 2024-01-15T18:12:56.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.499+0000 7f8e5e000640 1 -- 172.21.15.27:0/1599482034 learned_addr learned my addr 172.21.15.27:0/1599482034 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:12:56.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.499+0000 7f8e5cc00640 1 --2- 172.21.15.27:0/1599482034 >> v2:172.21.15.93:3300/0 conn(0x7f8e580cde10 0x7f8e580cbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:56.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.499+0000 7f8e57e00640 1 -- 172.21.15.27:0/1599482034 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1931754542 0 0) 0x7f8e580cb4d0 con 0x7f8e580cfc30 2024-01-15T18:12:56.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.499+0000 7f8e57e00640 1 -- 172.21.15.27:0/1599482034 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e38003530 con 0x7f8e580cfc30 2024-01-15T18:12:56.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.499+0000 7f8e57e00640 1 -- 172.21.15.27:0/1599482034 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1463425645 0 0) 0x7f8e38003530 con 0x7f8e580cfc30 2024-01-15T18:12:56.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.499+0000 7f8e57e00640 1 -- 172.21.15.27:0/1599482034 >> v2:172.21.15.93:3300/0 conn(0x7f8e580cde10 msgr2=0x7f8e580cbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:56.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.499+0000 7f8e57e00640 1 --2- 172.21.15.27:0/1599482034 >> v2:172.21.15.93:3300/0 conn(0x7f8e580cde10 0x7f8e580cbf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:56.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.499+0000 7f8e57e00640 1 -- 172.21.15.27:0/1599482034 >> v1:172.21.15.93:6789/0 conn(0x7f8e5815d680 legacy=0x7f8e580cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:56.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.499+0000 7f8e57e00640 1 -- 172.21.15.27:0/1599482034 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e580cb770 con 0x7f8e580cfc30 2024-01-15T18:12:56.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.499+0000 7f8e57e00640 1 -- 172.21.15.27:0/1599482034 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f8e50001a20 con 0x7f8e580cfc30 2024-01-15T18:12:56.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.499+0000 7f8e57e00640 1 -- 172.21.15.27:0/1599482034 >> v1:172.21.15.27:6789/0 conn(0x7f8e580cfc30 legacy=0x7f8e580cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:56.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.499+0000 7f8e57e00640 1 --2- 172.21.15.27:0/1599482034 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8e380039f0 0x7f8e38003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:56.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.499+0000 7f8e57e00640 1 --2- 172.21.15.27:0/1599482034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e380047d0 0x7f8e38004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:56.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.503+0000 7f8e57e00640 1 --2- 172.21.15.27:0/1599482034 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8e38005190 0x7f8e3800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:56.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.503+0000 7f8e57e00640 1 -- 172.21.15.27:0/1599482034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e580d0400 con 0x7f8e380047d0 2024-01-15T18:12:56.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.503+0000 7f8e57e00640 1 -- 172.21.15.27:0/1599482034 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3800c0c0 con 0x7f8e380039f0 2024-01-15T18:12:56.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.503+0000 7f8e57e00640 1 -- 172.21.15.27:0/1599482034 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3800c280 con 0x7f8e38005190 2024-01-15T18:12:56.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.503+0000 7f8e5cc00640 1 --2- 172.21.15.27:0/1599482034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e380047d0 0x7f8e38004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:56.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.503+0000 7f8e5d600640 1 --2- 172.21.15.27:0/1599482034 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8e380039f0 0x7f8e38003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:56.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.503+0000 7f8e5e000640 1 --2- 172.21.15.27:0/1599482034 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8e38005190 0x7f8e3800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:56.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.503+0000 7f8e5cc00640 1 -- 172.21.15.27:0/1599482034 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8e38005190 msgr2=0x7f8e3800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:56.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.503+0000 7f8e5cc00640 1 --2- 172.21.15.27:0/1599482034 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8e38005190 0x7f8e3800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:56.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.503+0000 7f8e5cc00640 1 -- 172.21.15.27:0/1599482034 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8e380039f0 msgr2=0x7f8e38003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:56.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.503+0000 7f8e5cc00640 1 --2- 172.21.15.27:0/1599482034 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8e380039f0 0x7f8e38003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:56.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.503+0000 7f8e5cc00640 1 -- 172.21.15.27:0/1599482034 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e38003530 con 0x7f8e380047d0 2024-01-15T18:12:56.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.503+0000 7f8e5e000640 1 --2- 172.21.15.27:0/1599482034 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8e38005190 0x7f8e3800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:12:56.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.503+0000 7f8e5e000640 1 -- 172.21.15.27:0/1599482034 reap_dead start 2024-01-15T18:12:56.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.503+0000 7f8e5cc00640 1 --2- 172.21.15.27:0/1599482034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e380047d0 0x7f8e38004c50 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f8e40009ae0 tx=0x7f8e4001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9151a154c729822d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:56.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.503+0000 7f8e57e00640 1 -- 172.21.15.27:0/1599482034 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e400a5640 con 0x7f8e380047d0 2024-01-15T18:12:56.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.507+0000 7f8e57e00640 1 -- 172.21.15.27:0/1599482034 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e400a57a0 con 0x7f8e380047d0 2024-01-15T18:12:56.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.507+0000 7f8e57e00640 1 -- 172.21.15.27:0/1599482034 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e400a5b50 con 0x7f8e380047d0 2024-01-15T18:12:56.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.507+0000 7f8e60000640 1 -- 172.21.15.27:0/1599482034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e380047d0 msgr2=0x7f8e38004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:56.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.507+0000 7f8e60000640 1 --2- 172.21.15.27:0/1599482034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e380047d0 0x7f8e38004c50 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f8e40009ae0 tx=0x7f8e4001ca50 comp rx=0 tx=0).stop 2024-01-15T18:12:56.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.507+0000 7f8e60000640 1 -- 172.21.15.27:0/1599482034 shutdown_connections 2024-01-15T18:12:56.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.507+0000 7f8e60000640 1 --2- 172.21.15.27:0/1599482034 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e380047d0 0x7f8e38004c50 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:56.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.507+0000 7f8e60000640 1 -- 172.21.15.27:0/1599482034 >> 172.21.15.27:0/1599482034 conn(0x7f8e580d0e80 msgr2=0x7f8e580d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:56.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.507+0000 7f8e60000640 1 -- 172.21.15.27:0/1599482034 shutdown_connections 2024-01-15T18:12:56.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.507+0000 7f8e60000640 1 -- 172.21.15.27:0/1599482034 wait complete. 2024-01-15T18:12:56.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.507+0000 7f8e60000640 1 Processor -- start 2024-01-15T18:12:56.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.507+0000 7f8e60000640 1 -- start start 2024-01-15T18:12:56.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.507+0000 7f8e60000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8e380047d0 0x7f8e5815e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:56.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.507+0000 7f8e60000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e5815f310 0x7f8e58156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:56.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.507+0000 7f8e60000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8e5815b6d0 0x7f8e5815bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:56.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.507+0000 7f8e60000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e581f6df0 con 0x7f8e5815f310 2024-01-15T18:12:56.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.507+0000 7f8e60000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5815c090 con 0x7f8e380047d0 2024-01-15T18:12:56.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.507+0000 7f8e60000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e580c9d30 con 0x7f8e5815b6d0 2024-01-15T18:12:56.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.511+0000 7f8e5cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e5815f310 0x7f8e58156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:56.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.511+0000 7f8e5cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e5815f310 0x7f8e58156cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46554/0 (socket says 172.21.15.27:46554) 2024-01-15T18:12:56.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.511+0000 7f8e5cc00640 1 -- 172.21.15.27:0/240744002 learned_addr learned my addr 172.21.15.27:0/240744002 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:56.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.511+0000 7f8e5d600640 1 --2- 172.21.15.27:0/240744002 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8e380047d0 0x7f8e5815e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:56.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.511+0000 7f8e5e000640 1 --2- 172.21.15.27:0/240744002 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8e5815b6d0 0x7f8e5815bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:56.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.511+0000 7f8e5cc00640 1 -- 172.21.15.27:0/240744002 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8e5815b6d0 msgr2=0x7f8e5815bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:56.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.511+0000 7f8e5cc00640 1 --2- 172.21.15.27:0/240744002 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8e5815b6d0 0x7f8e5815bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:56.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.511+0000 7f8e5cc00640 1 -- 172.21.15.27:0/240744002 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8e380047d0 msgr2=0x7f8e5815e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:56.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.511+0000 7f8e5cc00640 1 --2- 172.21.15.27:0/240744002 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8e380047d0 0x7f8e5815e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:56.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.511+0000 7f8e5cc00640 1 -- 172.21.15.27:0/240744002 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5815faf0 con 0x7f8e5815f310 2024-01-15T18:12:56.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.511+0000 7f8e5cc00640 1 --2- 172.21.15.27:0/240744002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e5815f310 0x7f8e58156cf0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f8e4001c880 tx=0x7f8e400197f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:56.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.511+0000 7f8e55600640 1 -- 172.21.15.27:0/240744002 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e40004220 con 0x7f8e5815f310 2024-01-15T18:12:56.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.511+0000 7f8e55600640 1 -- 172.21.15.27:0/240744002 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e400043c0 con 0x7f8e5815f310 2024-01-15T18:12:56.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.511+0000 7f8e55600640 1 -- 172.21.15.27:0/240744002 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e400b68b0 con 0x7f8e5815f310 2024-01-15T18:12:56.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.511+0000 7f8e60000640 1 -- 172.21.15.27:0/240744002 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e581f6490 con 0x7f8e5815f310 2024-01-15T18:12:56.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.511+0000 7f8e60000640 1 -- 172.21.15.27:0/240744002 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e581f6730 con 0x7f8e5815f310 2024-01-15T18:12:56.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.511+0000 7f8e60000640 1 -- 172.21.15.27:0/240744002 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e5804e850 con 0x7f8e5815f310 2024-01-15T18:12:56.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.511+0000 7f8e55600640 1 -- 172.21.15.27:0/240744002 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) v1 ==== 87027+0+0 (secure 0 0 0) 0x7f8e400b4070 con 0x7f8e5815f310 2024-01-15T18:12:56.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.515+0000 7f8e55600640 1 --2- 172.21.15.27:0/240744002 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8e300672b0 0x7f8e30069770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:56.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.515+0000 7f8e55600640 1 -- 172.21.15.27:0/240744002 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f8e4011f700 con 0x7f8e5815f310 2024-01-15T18:12:56.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.515+0000 7f8e5d600640 1 --2- 172.21.15.27:0/240744002 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8e300672b0 0x7f8e30069770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:56.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.515+0000 7f8e5d600640 1 --2- 172.21.15.27:0/240744002 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8e300672b0 0x7f8e30069770 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8e48007fd0 tx=0x7f8e48009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:56.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.519+0000 7f8e55600640 1 -- 172.21.15.27:0/240744002 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8e400ebc10 con 0x7f8e5815f310 2024-01-15T18:12:56.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.707+0000 7f8e60000640 1 -- 172.21.15.27:0/240744002 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone51934", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e580d19a0 con 0x7f8e300672b0 2024-01-15T18:12:56.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.715+0000 7f8e55600640 1 -- 172.21.15.27:0/240744002 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f8e480052d0 con 0x7f8e300672b0 2024-01-15T18:12:56.715 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:12:56.715 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:12:56.715 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "pending", 2024-01-15T18:12:56.716 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:12:56.716 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:12:56.716 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:12:56.716 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:12:56.716 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:12:56.716 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:12:56.716 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:12:56.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.715+0000 7f8e60000640 1 -- 172.21.15.27:0/240744002 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8e300672b0 msgr2=0x7f8e30069770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:56.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.715+0000 7f8e60000640 1 --2- 172.21.15.27:0/240744002 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8e300672b0 0x7f8e30069770 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8e48007fd0 tx=0x7f8e48009040 comp rx=0 tx=0).stop 2024-01-15T18:12:56.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.715+0000 7f8e60000640 1 -- 172.21.15.27:0/240744002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e5815f310 msgr2=0x7f8e58156cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:56.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.715+0000 7f8e60000640 1 --2- 172.21.15.27:0/240744002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e5815f310 0x7f8e58156cf0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f8e4001c880 tx=0x7f8e400197f0 comp rx=0 tx=0).stop 2024-01-15T18:12:56.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.715+0000 7f8e60000640 1 -- 172.21.15.27:0/240744002 shutdown_connections 2024-01-15T18:12:56.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.715+0000 7f8e60000640 1 --2- 172.21.15.27:0/240744002 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8e300672b0 0x7f8e30069770 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:56.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.715+0000 7f8e60000640 1 --2- 172.21.15.27:0/240744002 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8e5815b6d0 0x7f8e5815bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:56.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.715+0000 7f8e60000640 1 --2- 172.21.15.27:0/240744002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e5815f310 0x7f8e58156cf0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:56.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.715+0000 7f8e60000640 1 --2- 172.21.15.27:0/240744002 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8e380047d0 0x7f8e5815e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:56.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.715+0000 7f8e60000640 1 -- 172.21.15.27:0/240744002 >> 172.21.15.27:0/240744002 conn(0x7f8e580d0e80 msgr2=0x7f8e580cf150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:56.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.715+0000 7f8e60000640 1 -- 172.21.15.27:0/240744002 shutdown_connections 2024-01-15T18:12:56.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:56.715+0000 7f8e60000640 1 -- 172.21.15.27:0/240744002 wait complete. 2024-01-15T18:12:56.737 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-15T18:12:57.157 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:12:57.160 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:12:57.738 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone51934 2024-01-15T18:12:57.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.891+0000 7f0380a00640 1 Processor -- start 2024-01-15T18:12:57.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.891+0000 7f0380a00640 1 -- start start 2024-01-15T18:12:57.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.895+0000 7f0380a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f037815ad20 0x7f037815b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:57.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.895+0000 7f0380a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03780715f0 0x7f037815ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:57.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.895+0000 7f0380a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f037815a610 con 0x7f037815d900 2024-01-15T18:12:57.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.895+0000 7f0380a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f037815e3c0 con 0x7f037815ad20 2024-01-15T18:12:57.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.895+0000 7f0380a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f037815e6a0 con 0x7f03780715f0 2024-01-15T18:12:57.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.895+0000 7f037e000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f037815ad20 0x7f037815b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:57.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.895+0000 7f037d600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f037815d900 0x7f0378070ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42532/0 (socket says 172.21.15.27:42532) 2024-01-15T18:12:57.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.895+0000 7f037d600640 1 -- 172.21.15.27:0/1949767280 learned_addr learned my addr 172.21.15.27:0/1949767280 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:12:57.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.895+0000 7f037ea00640 1 --2- 172.21.15.27:0/1949767280 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03780715f0 0x7f037815ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:57.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.895+0000 7f037e000640 1 -- 172.21.15.27:0/1949767280 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03780715f0 msgr2=0x7f037815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:57.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.895+0000 7f037e000640 1 --2- 172.21.15.27:0/1949767280 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03780715f0 0x7f037815ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:57.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.895+0000 7f037e000640 1 -- 172.21.15.27:0/1949767280 >> v1:172.21.15.27:6789/0 conn(0x7f037815d900 legacy=0x7f0378070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:57.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.895+0000 7f037cc00640 1 -- 172.21.15.27:0/1949767280 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1230079820 0 0) 0x7f037815a610 con 0x7f037815d900 2024-01-15T18:12:57.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.895+0000 7f037e000640 1 -- 172.21.15.27:0/1949767280 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f037815b6e0 con 0x7f037815ad20 2024-01-15T18:12:57.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.899+0000 7f037e000640 1 --2- 172.21.15.27:0/1949767280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f037815ad20 0x7f037815b100 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f03680097f0 tx=0x7f036801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea0a098f49ceb289 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:57.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.899+0000 7f037cc00640 1 -- 172.21.15.27:0/1949767280 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03680ab670 con 0x7f037815ad20 2024-01-15T18:12:57.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.899+0000 7f037cc00640 1 -- 172.21.15.27:0/1949767280 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03680ab7d0 con 0x7f037815ad20 2024-01-15T18:12:57.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.899+0000 7f037cc00640 1 -- 172.21.15.27:0/1949767280 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03680abb60 con 0x7f037815ad20 2024-01-15T18:12:57.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.899+0000 7f0380a00640 1 -- 172.21.15.27:0/1949767280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f037815ad20 msgr2=0x7f037815b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:57.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.899+0000 7f0380a00640 1 --2- 172.21.15.27:0/1949767280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f037815ad20 0x7f037815b100 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f03680097f0 tx=0x7f036801c450 comp rx=0 tx=0).stop 2024-01-15T18:12:57.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.899+0000 7f0380a00640 1 -- 172.21.15.27:0/1949767280 shutdown_connections 2024-01-15T18:12:57.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.899+0000 7f0380a00640 1 --2- 172.21.15.27:0/1949767280 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03780715f0 0x7f037815ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:57.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.899+0000 7f0380a00640 1 --2- 172.21.15.27:0/1949767280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f037815ad20 0x7f037815b100 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:57.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.899+0000 7f0380a00640 1 -- 172.21.15.27:0/1949767280 >> 172.21.15.27:0/1949767280 conn(0x7f03780cb700 msgr2=0x7f03780cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:57.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.899+0000 7f0380a00640 1 -- 172.21.15.27:0/1949767280 shutdown_connections 2024-01-15T18:12:57.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.899+0000 7f0380a00640 1 -- 172.21.15.27:0/1949767280 wait complete. 2024-01-15T18:12:57.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.899+0000 7f0380a00640 1 Processor -- start 2024-01-15T18:12:57.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.899+0000 7f0380a00640 1 -- start start 2024-01-15T18:12:57.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.903+0000 7f0380a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03780715f0 0x7f037814fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:57.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.903+0000 7f0380a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f037815ad20 0x7f0378157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:57.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.903+0000 7f0380a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f037815d900 0x7f0378162e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:57.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.903+0000 7f0380a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0378155d80 con 0x7f037815ad20 2024-01-15T18:12:57.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.903+0000 7f0380a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03781f6700 con 0x7f03780715f0 2024-01-15T18:12:57.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.903+0000 7f0380a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03781f5b80 con 0x7f037815d900 2024-01-15T18:12:57.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.903+0000 7f037d600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f037815ad20 0x7f0378157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:57.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.903+0000 7f037d600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f037815ad20 0x7f0378157160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46570/0 (socket says 172.21.15.27:46570) 2024-01-15T18:12:57.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.903+0000 7f037d600640 1 -- 172.21.15.27:0/496625047 learned_addr learned my addr 172.21.15.27:0/496625047 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:57.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.903+0000 7f037ea00640 1 --2- 172.21.15.27:0/496625047 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f037815d900 0x7f0378162e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:57.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.903+0000 7f037e000640 1 --2- 172.21.15.27:0/496625047 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03780715f0 0x7f037814fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:57.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.903+0000 7f037d600640 1 -- 172.21.15.27:0/496625047 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f037815d900 msgr2=0x7f0378162e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:57.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.903+0000 7f037d600640 1 --2- 172.21.15.27:0/496625047 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f037815d900 0x7f0378162e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:57.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.903+0000 7f037d600640 1 -- 172.21.15.27:0/496625047 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03780715f0 msgr2=0x7f037814fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:57.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.903+0000 7f037d600640 1 --2- 172.21.15.27:0/496625047 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03780715f0 0x7f037814fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:57.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.903+0000 7f037d600640 1 -- 172.21.15.27:0/496625047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0368009420 con 0x7f037815ad20 2024-01-15T18:12:57.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.903+0000 7f037d600640 1 --2- 172.21.15.27:0/496625047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f037815ad20 0x7f0378157160 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f036000c480 tx=0x7f036000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:57.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.903+0000 7f0376000640 1 -- 172.21.15.27:0/496625047 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03600074e0 con 0x7f037815ad20 2024-01-15T18:12:57.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.903+0000 7f0376000640 1 -- 172.21.15.27:0/496625047 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0360007660 con 0x7f037815ad20 2024-01-15T18:12:57.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.903+0000 7f0376000640 1 -- 172.21.15.27:0/496625047 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0360004a30 con 0x7f037815ad20 2024-01-15T18:12:57.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.903+0000 7f0380a00640 1 -- 172.21.15.27:0/496625047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03781f5e60 con 0x7f037815ad20 2024-01-15T18:12:57.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.903+0000 7f0380a00640 1 -- 172.21.15.27:0/496625047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03781f6420 con 0x7f037815ad20 2024-01-15T18:12:57.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.903+0000 7f0380a00640 1 -- 172.21.15.27:0/496625047 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f037804e850 con 0x7f037815ad20 2024-01-15T18:12:57.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.907+0000 7f0376000640 1 -- 172.21.15.27:0/496625047 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) v1 ==== 87027+0+0 (secure 0 0 0) 0x7f0360004b90 con 0x7f037815ad20 2024-01-15T18:12:57.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.907+0000 7f0376000640 1 --2- 172.21.15.27:0/496625047 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0350067280 0x7f0350069740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:57.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.907+0000 7f0376000640 1 -- 172.21.15.27:0/496625047 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f0360082050 con 0x7f037815ad20 2024-01-15T18:12:57.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.907+0000 7f037e000640 1 --2- 172.21.15.27:0/496625047 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0350067280 0x7f0350069740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:57.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.907+0000 7f037e000640 1 --2- 172.21.15.27:0/496625047 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0350067280 0x7f0350069740 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f03680af000 tx=0x7f03680a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:57.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:57.911+0000 7f0376000640 1 -- 172.21.15.27:0/496625047 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0360050b90 con 0x7f037815ad20 2024-01-15T18:12:58.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:58.103+0000 7f0380a00640 1 -- 172.21.15.27:0/496625047 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone51934", "target": ["mon-mgr", ""]}) v1 -- 0x7f037806d060 con 0x7f0350067280 2024-01-15T18:12:58.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:58.111+0000 7f0376000640 1 -- 172.21.15.27:0/496625047 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f037806d060 con 0x7f0350067280 2024-01-15T18:12:58.111 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:12:58.111 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:12:58.111 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "pending", 2024-01-15T18:12:58.111 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:12:58.111 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:12:58.111 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:12:58.111 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:12:58.112 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:12:58.112 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:12:58.112 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:12:58.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:58.111+0000 7f0380a00640 1 -- 172.21.15.27:0/496625047 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0350067280 msgr2=0x7f0350069740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:58.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:58.111+0000 7f0380a00640 1 --2- 172.21.15.27:0/496625047 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0350067280 0x7f0350069740 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f03680af000 tx=0x7f03680a7000 comp rx=0 tx=0).stop 2024-01-15T18:12:58.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:58.111+0000 7f0380a00640 1 -- 172.21.15.27:0/496625047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f037815ad20 msgr2=0x7f0378157160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:58.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:58.111+0000 7f0380a00640 1 --2- 172.21.15.27:0/496625047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f037815ad20 0x7f0378157160 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f036000c480 tx=0x7f036000c940 comp rx=0 tx=0).stop 2024-01-15T18:12:58.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:58.111+0000 7f0380a00640 1 -- 172.21.15.27:0/496625047 shutdown_connections 2024-01-15T18:12:58.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:58.111+0000 7f0380a00640 1 --2- 172.21.15.27:0/496625047 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0350067280 0x7f0350069740 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:58.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:58.111+0000 7f0380a00640 1 --2- 172.21.15.27:0/496625047 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f037815d900 0x7f0378162e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:58.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:58.111+0000 7f0380a00640 1 --2- 172.21.15.27:0/496625047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f037815ad20 0x7f0378157160 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:58.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:58.111+0000 7f0380a00640 1 --2- 172.21.15.27:0/496625047 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03780715f0 0x7f037814fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:58.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:58.111+0000 7f0380a00640 1 -- 172.21.15.27:0/496625047 >> 172.21.15.27:0/496625047 conn(0x7f03780cb700 msgr2=0x7f0378153860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:58.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:58.111+0000 7f0380a00640 1 -- 172.21.15.27:0/496625047 shutdown_connections 2024-01-15T18:12:58.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:58.111+0000 7f0380a00640 1 -- 172.21.15.27:0/496625047 wait complete. 2024-01-15T18:12:58.133 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-15T18:12:59.134 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone51934 2024-01-15T18:12:59.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.287+0000 7f9792800640 1 Processor -- start 2024-01-15T18:12:59.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.287+0000 7f9792800640 1 -- start start 2024-01-15T18:12:59.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.287+0000 7f9792800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f978c15d570 0x7f978c06be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:59.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.291+0000 7f9792800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f978c06b810 0x7f978c06c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:59.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.291+0000 7f9792800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f978c0b0d70 con 0x7f978c15d190 2024-01-15T18:12:59.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.291+0000 7f9792800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f978c0d0540 con 0x7f978c06b810 2024-01-15T18:12:59.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.291+0000 7f9792800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c0b0440 con 0x7f978c15d570 2024-01-15T18:12:59.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.291+0000 7f978b400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f978c15d190 0x7f978c0cfcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42548/0 (socket says 172.21.15.27:42548) 2024-01-15T18:12:59.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.291+0000 7f978b400640 1 -- 172.21.15.27:0/2848331849 learned_addr learned my addr 172.21.15.27:0/2848331849 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:12:59.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.291+0000 7f978be00640 1 --2- 172.21.15.27:0/2848331849 >> v2:172.21.15.27:3300/0 conn(0x7f978c06b810 0x7f978c06c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:59.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.291+0000 7f978aa00640 1 --2- 172.21.15.27:0/2848331849 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f978c15d570 0x7f978c06be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:59.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.291+0000 7f978a000640 1 -- 172.21.15.27:0/2848331849 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2821031101 0 0) 0x7f978c0b0d70 con 0x7f978c15d190 2024-01-15T18:12:59.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.291+0000 7f978a000640 1 -- 172.21.15.27:0/2848331849 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f976c003530 con 0x7f978c15d190 2024-01-15T18:12:59.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.291+0000 7f978be00640 1 -- 172.21.15.27:0/2848331849 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f978c15d570 msgr2=0x7f978c06be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:59.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.291+0000 7f978be00640 1 --2- 172.21.15.27:0/2848331849 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f978c15d570 0x7f978c06be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:59.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.291+0000 7f978be00640 1 -- 172.21.15.27:0/2848331849 >> v1:172.21.15.27:6789/0 conn(0x7f978c15d190 legacy=0x7f978c0cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:59.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.291+0000 7f978be00640 1 -- 172.21.15.27:0/2848331849 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c0b0720 con 0x7f978c06b810 2024-01-15T18:12:59.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.291+0000 7f978be00640 1 --2- 172.21.15.27:0/2848331849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f978c06b810 0x7f978c06c350 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f9784009d90 tx=0x7f978401c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=463cb2555cce5a36 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:59.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.295+0000 7f978a000640 1 -- 172.21.15.27:0/2848331849 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97840a5a00 con 0x7f978c06b810 2024-01-15T18:12:59.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.295+0000 7f978a000640 1 -- 172.21.15.27:0/2848331849 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97840a5b60 con 0x7f978c06b810 2024-01-15T18:12:59.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.295+0000 7f978a000640 1 -- 172.21.15.27:0/2848331849 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97840ad440 con 0x7f978c06b810 2024-01-15T18:12:59.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.295+0000 7f9792800640 1 -- 172.21.15.27:0/2848331849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f978c06b810 msgr2=0x7f978c06c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:59.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.295+0000 7f9792800640 1 --2- 172.21.15.27:0/2848331849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f978c06b810 0x7f978c06c350 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f9784009d90 tx=0x7f978401c920 comp rx=0 tx=0).stop 2024-01-15T18:12:59.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.295+0000 7f9792800640 1 -- 172.21.15.27:0/2848331849 shutdown_connections 2024-01-15T18:12:59.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.295+0000 7f9792800640 1 --2- 172.21.15.27:0/2848331849 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f978c06b810 0x7f978c06c350 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:59.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.295+0000 7f9792800640 1 --2- 172.21.15.27:0/2848331849 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f978c15d570 0x7f978c06be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:59.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.295+0000 7f9792800640 1 -- 172.21.15.27:0/2848331849 >> 172.21.15.27:0/2848331849 conn(0x7f978c0d0e70 msgr2=0x7f978c0d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:59.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.295+0000 7f9792800640 1 -- 172.21.15.27:0/2848331849 shutdown_connections 2024-01-15T18:12:59.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.295+0000 7f9792800640 1 -- 172.21.15.27:0/2848331849 wait complete. 2024-01-15T18:12:59.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.295+0000 7f9792800640 1 Processor -- start 2024-01-15T18:12:59.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.295+0000 7f9792800640 1 -- start start 2024-01-15T18:12:59.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.295+0000 7f9792800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f978c06b810 0x7f978c0b6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:59.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.295+0000 7f9792800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f978c15d190 0x7f978c1528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:59.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.295+0000 7f9792800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f978c15d570 0x7f978c0c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:59.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.295+0000 7f9792800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c0c5a20 con 0x7f978c15d570 2024-01-15T18:12:59.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.295+0000 7f9792800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c0b5ff0 con 0x7f978c15d190 2024-01-15T18:12:59.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.295+0000 7f9792800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c0c5410 con 0x7f978c06b810 2024-01-15T18:12:59.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.299+0000 7f978be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f978c15d570 0x7f978c0c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:59.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.299+0000 7f978be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f978c15d570 0x7f978c0c5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46598/0 (socket says 172.21.15.27:46598) 2024-01-15T18:12:59.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.299+0000 7f978be00640 1 -- 172.21.15.27:0/2990015747 learned_addr learned my addr 172.21.15.27:0/2990015747 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:12:59.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.299+0000 7f978b400640 1 --2- 172.21.15.27:0/2990015747 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f978c06b810 0x7f978c0b6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:59.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.299+0000 7f978be00640 1 -- 172.21.15.27:0/2990015747 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f978c06b810 msgr2=0x7f978c0b6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:59.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.299+0000 7f978be00640 1 --2- 172.21.15.27:0/2990015747 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f978c06b810 0x7f978c0b6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:59.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.299+0000 7f978be00640 1 -- 172.21.15.27:0/2990015747 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f978c15d190 msgr2=0x7f978c1528f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:59.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.299+0000 7f978aa00640 1 --2- 172.21.15.27:0/2990015747 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f978c15d190 0x7f978c1528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:59.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.299+0000 7f978be00640 1 --2- 172.21.15.27:0/2990015747 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f978c15d190 0x7f978c1528f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:59.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.299+0000 7f978be00640 1 -- 172.21.15.27:0/2990015747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9784009a70 con 0x7f978c15d570 2024-01-15T18:12:59.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.299+0000 7f978b400640 1 --2- 172.21.15.27:0/2990015747 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f978c06b810 0x7f978c0b6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:59.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.299+0000 7f978aa00640 1 --2- 172.21.15.27:0/2990015747 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f978c15d190 0x7f978c1528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:59.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.299+0000 7f978be00640 1 --2- 172.21.15.27:0/2990015747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f978c15d570 0x7f978c0c5eb0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f97840b0040 tx=0x7f97840196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:59.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.299+0000 7f9783400640 1 -- 172.21.15.27:0/2990015747 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97840a5a00 con 0x7f978c15d570 2024-01-15T18:12:59.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.299+0000 7f9783400640 1 -- 172.21.15.27:0/2990015747 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9784009450 con 0x7f978c15d570 2024-01-15T18:12:59.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.299+0000 7f9783400640 1 -- 172.21.15.27:0/2990015747 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97840b9690 con 0x7f978c15d570 2024-01-15T18:12:59.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.299+0000 7f9792800640 1 -- 172.21.15.27:0/2990015747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978c0b5d10 con 0x7f978c15d570 2024-01-15T18:12:59.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.299+0000 7f9792800640 1 -- 172.21.15.27:0/2990015747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978c152e30 con 0x7f978c15d570 2024-01-15T18:12:59.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.299+0000 7f9792800640 1 -- 172.21.15.27:0/2990015747 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f978c0b11c0 con 0x7f978c15d570 2024-01-15T18:12:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.299+0000 7f9783400640 1 -- 172.21.15.27:0/2990015747 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) v1 ==== 87027+0+0 (secure 0 0 0) 0x7f97840b97f0 con 0x7f978c15d570 2024-01-15T18:12:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.303+0000 7f9783400640 1 --2- 172.21.15.27:0/2990015747 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9764067280 0x7f9764069740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.303+0000 7f9783400640 1 -- 172.21.15.27:0/2990015747 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f978411e8a0 con 0x7f978c15d570 2024-01-15T18:12:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.303+0000 7f978b400640 1 --2- 172.21.15.27:0/2990015747 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9764067280 0x7f9764069740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:59.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.303+0000 7f978b400640 1 --2- 172.21.15.27:0/2990015747 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9764067280 0x7f9764069740 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f977c0011f0 tx=0x7f977c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:59.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.307+0000 7f9783400640 1 -- 172.21.15.27:0/2990015747 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f978c0b11c0 con 0x7f978c15d570 2024-01-15T18:12:59.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.499+0000 7f9792800640 1 -- 172.21.15.27:0/2990015747 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone51934", "target": ["mon-mgr", ""]}) v1 -- 0x7f978c0c5880 con 0x7f9764067280 2024-01-15T18:12:59.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.503+0000 7f9783400640 1 -- 172.21.15.27:0/2990015747 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f977c001d30 con 0x7f9764067280 2024-01-15T18:12:59.506 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:12:59.506 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:12:59.506 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "pending", 2024-01-15T18:12:59.506 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:12:59.506 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:12:59.507 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:12:59.507 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:12:59.507 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:12:59.507 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:12:59.507 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:12:59.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.507+0000 7f9792800640 1 -- 172.21.15.27:0/2990015747 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9764067280 msgr2=0x7f9764069740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:59.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.507+0000 7f9792800640 1 --2- 172.21.15.27:0/2990015747 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9764067280 0x7f9764069740 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f977c0011f0 tx=0x7f977c009040 comp rx=0 tx=0).stop 2024-01-15T18:12:59.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.507+0000 7f9792800640 1 -- 172.21.15.27:0/2990015747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f978c15d570 msgr2=0x7f978c0c5eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:59.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.507+0000 7f9792800640 1 --2- 172.21.15.27:0/2990015747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f978c15d570 0x7f978c0c5eb0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f97840b0040 tx=0x7f97840196c0 comp rx=0 tx=0).stop 2024-01-15T18:12:59.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.507+0000 7f9792800640 1 -- 172.21.15.27:0/2990015747 shutdown_connections 2024-01-15T18:12:59.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.507+0000 7f9792800640 1 --2- 172.21.15.27:0/2990015747 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9764067280 0x7f9764069740 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:59.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.507+0000 7f9792800640 1 --2- 172.21.15.27:0/2990015747 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f978c15d570 0x7f978c0c5eb0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:59.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.507+0000 7f9792800640 1 --2- 172.21.15.27:0/2990015747 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f978c15d190 0x7f978c1528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:59.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.507+0000 7f9792800640 1 --2- 172.21.15.27:0/2990015747 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f978c06b810 0x7f978c0b6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:59.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.507+0000 7f9792800640 1 -- 172.21.15.27:0/2990015747 >> 172.21.15.27:0/2990015747 conn(0x7f978c0d0e70 msgr2=0x7f978c0afaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:59.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.507+0000 7f9792800640 1 -- 172.21.15.27:0/2990015747 shutdown_connections 2024-01-15T18:12:59.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:12:59.507+0000 7f9792800640 1 -- 172.21.15.27:0/2990015747 wait complete. 2024-01-15T18:12:59.528 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-15T18:13:00.385 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:13:00.383+0000 7f59f9c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:00.385 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:13:00.383+0000 7f59f9c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:00.385 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:13:00.383+0000 7f59f9c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:00.385 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:13:00.383+0000 7f59f9c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:00.385 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:13:00.383+0000 7f59f9c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:00.529 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone51934 2024-01-15T18:13:00.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.671+0000 7f663aa00640 1 Processor -- start 2024-01-15T18:13:00.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.671+0000 7f663aa00640 1 -- start start 2024-01-15T18:13:00.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f663aa00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f663415f550 0x7f66340cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:00.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f663aa00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66340b1cb0 con 0x7f66340cf110 2024-01-15T18:13:00.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f663aa00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663415fa00 con 0x7f6634158900 2024-01-15T18:13:00.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f663aa00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66340cf580 con 0x7f663415f550 2024-01-15T18:13:00.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6638a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f66340cf110 0x7f66340701e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42556/0 (socket says 172.21.15.27:42556) 2024-01-15T18:13:00.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6638a00640 1 -- 172.21.15.27:0/2163136955 learned_addr learned my addr 172.21.15.27:0/2163136955 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:13:00.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6633400640 1 --2- 172.21.15.27:0/2163136955 >> v2:172.21.15.93:3300/0 conn(0x7f663415f550 0x7f66340cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:00.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6632a00640 1 -- 172.21.15.27:0/2163136955 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3433891239 0 0) 0x7f66340b1cb0 con 0x7f66340cf110 2024-01-15T18:13:00.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6632a00640 1 -- 172.21.15.27:0/2163136955 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6614003530 con 0x7f66340cf110 2024-01-15T18:13:00.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6632a00640 1 -- 172.21.15.27:0/2163136955 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4100376837 0 0) 0x7f6614003530 con 0x7f66340cf110 2024-01-15T18:13:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6632a00640 1 -- 172.21.15.27:0/2163136955 >> v2:172.21.15.93:3300/0 conn(0x7f663415f550 msgr2=0x7f66340cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6632a00640 1 --2- 172.21.15.27:0/2163136955 >> v2:172.21.15.93:3300/0 conn(0x7f663415f550 0x7f66340cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6632a00640 1 -- 172.21.15.27:0/2163136955 >> v1:172.21.15.93:6789/0 conn(0x7f6634158900 legacy=0x7f663415bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6632a00640 1 -- 172.21.15.27:0/2163136955 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66340b1f00 con 0x7f66340cf110 2024-01-15T18:13:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6632a00640 1 -- 172.21.15.27:0/2163136955 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f662c001a00 con 0x7f66340cf110 2024-01-15T18:13:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6632a00640 1 -- 172.21.15.27:0/2163136955 >> v1:172.21.15.27:6789/0 conn(0x7f66340cf110 legacy=0x7f66340701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6632a00640 1 --2- 172.21.15.27:0/2163136955 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66140039f0 0x7f6614003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6632a00640 1 --2- 172.21.15.27:0/2163136955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66140047d0 0x7f6614004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6632a00640 1 --2- 172.21.15.27:0/2163136955 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6614005190 0x7f661400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:00.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6632a00640 1 -- 172.21.15.27:0/2163136955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66340cf580 con 0x7f66140047d0 2024-01-15T18:13:00.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6632a00640 1 -- 172.21.15.27:0/2163136955 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661400c0c0 con 0x7f6614005190 2024-01-15T18:13:00.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6632a00640 1 -- 172.21.15.27:0/2163136955 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f661400c280 con 0x7f66140039f0 2024-01-15T18:13:00.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6633400640 1 --2- 172.21.15.27:0/2163136955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66140047d0 0x7f6614004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:00.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6638a00640 1 --2- 172.21.15.27:0/2163136955 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6614005190 0x7f661400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:00.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6633e00640 1 --2- 172.21.15.27:0/2163136955 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66140039f0 0x7f6614003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:00.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6633400640 1 -- 172.21.15.27:0/2163136955 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66140039f0 msgr2=0x7f6614003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:00.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6633400640 1 --2- 172.21.15.27:0/2163136955 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66140039f0 0x7f6614003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:00.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6633400640 1 -- 172.21.15.27:0/2163136955 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6614005190 msgr2=0x7f661400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:00.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6633400640 1 --2- 172.21.15.27:0/2163136955 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6614005190 0x7f661400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:00.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6633400640 1 -- 172.21.15.27:0/2163136955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6614003530 con 0x7f66140047d0 2024-01-15T18:13:00.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6638a00640 1 --2- 172.21.15.27:0/2163136955 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6614005190 0x7f661400b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:13:00.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.675+0000 7f6638a00640 1 -- 172.21.15.27:0/2163136955 reap_dead start 2024-01-15T18:13:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.679+0000 7f6633400640 1 --2- 172.21.15.27:0/2163136955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66140047d0 0x7f6614004c50 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f661c009ae0 tx=0x7f661c01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b902b0c9db424a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.679+0000 7f6632a00640 1 -- 172.21.15.27:0/2163136955 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661c0a5640 con 0x7f66140047d0 2024-01-15T18:13:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.679+0000 7f6632a00640 1 -- 172.21.15.27:0/2163136955 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f661c0a57a0 con 0x7f66140047d0 2024-01-15T18:13:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.679+0000 7f6632a00640 1 -- 172.21.15.27:0/2163136955 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661c0a5b50 con 0x7f66140047d0 2024-01-15T18:13:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.679+0000 7f663aa00640 1 -- 172.21.15.27:0/2163136955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66140047d0 msgr2=0x7f6614004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.679+0000 7f663aa00640 1 --2- 172.21.15.27:0/2163136955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66140047d0 0x7f6614004c50 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f661c009ae0 tx=0x7f661c01ca50 comp rx=0 tx=0).stop 2024-01-15T18:13:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.679+0000 7f663aa00640 1 -- 172.21.15.27:0/2163136955 shutdown_connections 2024-01-15T18:13:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.679+0000 7f663aa00640 1 --2- 172.21.15.27:0/2163136955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66140047d0 0x7f6614004c50 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.679+0000 7f663aa00640 1 -- 172.21.15.27:0/2163136955 >> 172.21.15.27:0/2163136955 conn(0x7f66340cb700 msgr2=0x7f66340cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.679+0000 7f663aa00640 1 -- 172.21.15.27:0/2163136955 shutdown_connections 2024-01-15T18:13:00.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.679+0000 7f663aa00640 1 -- 172.21.15.27:0/2163136955 wait complete. 2024-01-15T18:13:00.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.683+0000 7f663aa00640 1 Processor -- start 2024-01-15T18:13:00.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.683+0000 7f663aa00640 1 -- start start 2024-01-15T18:13:00.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.683+0000 7f663aa00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66140047d0 0x7f66340ad260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:00.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.683+0000 7f663aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f663413efe0 0x7f663415ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:00.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.683+0000 7f663aa00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f663415ce70 0x7f663415d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:00.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.683+0000 7f663aa00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663415d810 con 0x7f663413efe0 2024-01-15T18:13:00.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.683+0000 7f663aa00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663413f720 con 0x7f663415ce70 2024-01-15T18:13:00.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.683+0000 7f663aa00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f663413f460 con 0x7f66140047d0 2024-01-15T18:13:00.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.683+0000 7f6633400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f663413efe0 0x7f663415ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:00.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.683+0000 7f6633400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f663413efe0 0x7f663415ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46614/0 (socket says 172.21.15.27:46614) 2024-01-15T18:13:00.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.683+0000 7f6633400640 1 -- 172.21.15.27:0/3582031174 learned_addr learned my addr 172.21.15.27:0/3582031174 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:00.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.683+0000 7f6638a00640 1 --2- 172.21.15.27:0/3582031174 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f663415ce70 0x7f663415d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:00.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.683+0000 7f6633e00640 1 --2- 172.21.15.27:0/3582031174 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66140047d0 0x7f66340ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:00.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.683+0000 7f6633400640 1 -- 172.21.15.27:0/3582031174 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66140047d0 msgr2=0x7f66340ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:00.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.683+0000 7f6633400640 1 --2- 172.21.15.27:0/3582031174 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66140047d0 0x7f66340ad260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:00.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.683+0000 7f6633400640 1 -- 172.21.15.27:0/3582031174 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f663415ce70 msgr2=0x7f663415d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:00.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.683+0000 7f6633400640 1 --2- 172.21.15.27:0/3582031174 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f663415ce70 0x7f663415d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:00.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.683+0000 7f6633400640 1 -- 172.21.15.27:0/3582031174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663415b220 con 0x7f663413efe0 2024-01-15T18:13:00.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.683+0000 7f6633400640 1 --2- 172.21.15.27:0/3582031174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f663413efe0 0x7f663415ace0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f661c01c880 tx=0x7f661c0197f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:00.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.683+0000 7f662be00640 1 -- 172.21.15.27:0/3582031174 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661c004020 con 0x7f663413efe0 2024-01-15T18:13:00.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.683+0000 7f662be00640 1 -- 172.21.15.27:0/3582031174 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f661c004180 con 0x7f663413efe0 2024-01-15T18:13:00.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.683+0000 7f663aa00640 1 -- 172.21.15.27:0/3582031174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66340ad920 con 0x7f663413efe0 2024-01-15T18:13:00.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.683+0000 7f662be00640 1 -- 172.21.15.27:0/3582031174 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661c0b8660 con 0x7f663413efe0 2024-01-15T18:13:00.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.683+0000 7f663aa00640 1 -- 172.21.15.27:0/3582031174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66340adee0 con 0x7f663413efe0 2024-01-15T18:13:00.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.687+0000 7f663aa00640 1 -- 172.21.15.27:0/3582031174 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f663404e850 con 0x7f663413efe0 2024-01-15T18:13:00.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.687+0000 7f662be00640 1 -- 172.21.15.27:0/3582031174 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 23) v1 ==== 87027+0+0 (secure 0 0 0) 0x7f661c0b4070 con 0x7f663413efe0 2024-01-15T18:13:00.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.687+0000 7f662be00640 1 --2- 172.21.15.27:0/3582031174 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f660c0672b0 0x7f660c069770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:00.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.687+0000 7f662be00640 1 -- 172.21.15.27:0/3582031174 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f661c11d330 con 0x7f663413efe0 2024-01-15T18:13:00.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.687+0000 7f6633e00640 1 --2- 172.21.15.27:0/3582031174 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f660c0672b0 0x7f660c069770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:00.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.687+0000 7f6633e00640 1 --2- 172.21.15.27:0/3582031174 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f660c0672b0 0x7f660c069770 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6624007fd0 tx=0x7f6624009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:00.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.691+0000 7f662be00640 1 -- 172.21.15.27:0/3582031174 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f661c0ebe70 con 0x7f663413efe0 2024-01-15T18:13:00.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:00.879+0000 7f663aa00640 1 -- 172.21.15.27:0/3582031174 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone51934", "target": ["mon-mgr", ""]}) v1 -- 0x7f66340cf990 con 0x7f660c0672b0 2024-01-15T18:13:03.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:03.067+0000 7f662be00640 1 -- 172.21.15.27:0/3582031174 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 24) v1 ==== 87078+0+0 (secure 0 0 0) 0x7f661c0ec240 con 0x7f663413efe0 2024-01-15T18:13:05.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:05.371+0000 7f662be00640 1 -- 172.21.15.27:0/3582031174 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f66340cf990 con 0x7f660c0672b0 2024-01-15T18:13:05.373 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:13:05.373 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:13:05.373 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "pending", 2024-01-15T18:13:05.373 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:13:05.373 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:13:05.373 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:13:05.373 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:13:05.373 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:05.373 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:05.374 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:13:05.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:05.375+0000 7f663aa00640 1 -- 172.21.15.27:0/3582031174 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f660c0672b0 msgr2=0x7f660c069770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:05.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:05.375+0000 7f663aa00640 1 --2- 172.21.15.27:0/3582031174 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f660c0672b0 0x7f660c069770 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6624007fd0 tx=0x7f6624009040 comp rx=0 tx=0).stop 2024-01-15T18:13:05.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:05.375+0000 7f663aa00640 1 -- 172.21.15.27:0/3582031174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f663413efe0 msgr2=0x7f663415ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:05.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:05.375+0000 7f663aa00640 1 --2- 172.21.15.27:0/3582031174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f663413efe0 0x7f663415ace0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f661c01c880 tx=0x7f661c0197f0 comp rx=0 tx=0).stop 2024-01-15T18:13:05.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:05.375+0000 7f663aa00640 1 -- 172.21.15.27:0/3582031174 shutdown_connections 2024-01-15T18:13:05.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:05.375+0000 7f663aa00640 1 --2- 172.21.15.27:0/3582031174 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f660c0672b0 0x7f660c069770 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:05.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:05.375+0000 7f663aa00640 1 --2- 172.21.15.27:0/3582031174 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f663415ce70 0x7f663415d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:05.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:05.375+0000 7f663aa00640 1 --2- 172.21.15.27:0/3582031174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f663413efe0 0x7f663415ace0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:05.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:05.375+0000 7f663aa00640 1 --2- 172.21.15.27:0/3582031174 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f66140047d0 0x7f66340ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:05.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:05.375+0000 7f663aa00640 1 -- 172.21.15.27:0/3582031174 >> 172.21.15.27:0/3582031174 conn(0x7f66340cb700 msgr2=0x7f66340700e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:05.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:05.375+0000 7f663aa00640 1 -- 172.21.15.27:0/3582031174 shutdown_connections 2024-01-15T18:13:05.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:05.375+0000 7f663aa00640 1 -- 172.21.15.27:0/3582031174 wait complete. 2024-01-15T18:13:05.385 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:13:05.382+0000 7f59f8800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:05.385 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:13:05.382+0000 7f59f8800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:05.385 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:13:05.382+0000 7f59f8800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:05.385 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:13:05.382+0000 7f59f8800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:05.385 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:13:05.382+0000 7f59f8800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:05.402 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-15T18:13:06.403 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone51934 2024-01-15T18:13:06.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.631+0000 7f6960e00640 1 Processor -- start 2024-01-15T18:13:06.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.635+0000 7f6960e00640 1 -- start start 2024-01-15T18:13:06.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.635+0000 7f6960e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f695c1597c0 0x7f695c0aec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:06.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.635+0000 7f6960e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f695c0ae780 0x7f695c06de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:06.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.635+0000 7f6960e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f695c06d100 con 0x7f695c070d50 2024-01-15T18:13:06.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.635+0000 7f6960e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f695c06b2e0 con 0x7f695c0ae780 2024-01-15T18:13:06.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.635+0000 7f6960e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f695c06d350 con 0x7f695c1597c0 2024-01-15T18:13:06.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.635+0000 7f695a800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f695c0ae780 0x7f695c06de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:06.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.635+0000 7f6959e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f695c070d50 0x7f695c071130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60006/0 (socket says 172.21.15.27:60006) 2024-01-15T18:13:06.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.635+0000 7f6959e00640 1 -- 172.21.15.27:0/2279234408 learned_addr learned my addr 172.21.15.27:0/2279234408 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:13:06.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.635+0000 7f6959400640 1 --2- 172.21.15.27:0/2279234408 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f695c1597c0 0x7f695c0aec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:06.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.635+0000 7f695a800640 1 -- 172.21.15.27:0/2279234408 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f695c1597c0 msgr2=0x7f695c0aec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:06.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.635+0000 7f695a800640 1 --2- 172.21.15.27:0/2279234408 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f695c1597c0 0x7f695c0aec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:06.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.635+0000 7f695a800640 1 -- 172.21.15.27:0/2279234408 >> v1:172.21.15.27:6789/0 conn(0x7f695c070d50 legacy=0x7f695c071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:06.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.635+0000 7f695a800640 1 -- 172.21.15.27:0/2279234408 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695c06d630 con 0x7f695c0ae780 2024-01-15T18:13:06.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.635+0000 7f6958a00640 1 -- 172.21.15.27:0/2279234408 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693365786 0 0) 0x7f695c06d100 con 0x7f695c070d50 2024-01-15T18:13:06.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.639+0000 7f695a800640 1 --2- 172.21.15.27:0/2279234408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f695c0ae780 0x7f695c06de30 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f6954009d90 tx=0x7f695401c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3018b81d0d5cc404 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:06.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.639+0000 7f6958a00640 1 -- 172.21.15.27:0/2279234408 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69540a5860 con 0x7f695c0ae780 2024-01-15T18:13:06.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.639+0000 7f6958a00640 1 -- 172.21.15.27:0/2279234408 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69540a59c0 con 0x7f695c0ae780 2024-01-15T18:13:06.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.639+0000 7f6958a00640 1 -- 172.21.15.27:0/2279234408 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69540a5d50 con 0x7f695c0ae780 2024-01-15T18:13:06.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.639+0000 7f6960e00640 1 -- 172.21.15.27:0/2279234408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f695c0ae780 msgr2=0x7f695c06de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:06.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.639+0000 7f6960e00640 1 --2- 172.21.15.27:0/2279234408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f695c0ae780 0x7f695c06de30 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f6954009d90 tx=0x7f695401c800 comp rx=0 tx=0).stop 2024-01-15T18:13:06.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.639+0000 7f6960e00640 1 -- 172.21.15.27:0/2279234408 shutdown_connections 2024-01-15T18:13:06.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.639+0000 7f6960e00640 1 --2- 172.21.15.27:0/2279234408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f695c0ae780 0x7f695c06de30 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:06.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.639+0000 7f6960e00640 1 --2- 172.21.15.27:0/2279234408 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f695c1597c0 0x7f695c0aec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:06.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.639+0000 7f6960e00640 1 -- 172.21.15.27:0/2279234408 >> 172.21.15.27:0/2279234408 conn(0x7f695c0b1620 msgr2=0x7f695c0b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:06.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.639+0000 7f6960e00640 1 -- 172.21.15.27:0/2279234408 shutdown_connections 2024-01-15T18:13:06.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.639+0000 7f6960e00640 1 -- 172.21.15.27:0/2279234408 wait complete. 2024-01-15T18:13:06.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.643+0000 7f6960e00640 1 Processor -- start 2024-01-15T18:13:06.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.643+0000 7f6960e00640 1 -- start start 2024-01-15T18:13:06.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.647+0000 7f6960e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f695c070d50 0x7f695c15c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:06.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.647+0000 7f6960e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f695c0ae780 0x7f695c15a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:06.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.647+0000 7f6960e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f695c1597c0 0x7f695c15ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:06.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.647+0000 7f6960e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c15b490 con 0x7f695c0ae780 2024-01-15T18:13:06.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.647+0000 7f6960e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c0af3a0 con 0x7f695c070d50 2024-01-15T18:13:06.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.647+0000 7f6960e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f695c1512f0 con 0x7f695c1597c0 2024-01-15T18:13:06.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.647+0000 7f6959400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f695c0ae780 0x7f695c15a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:06.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.647+0000 7f6959400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f695c0ae780 0x7f695c15a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38004/0 (socket says 172.21.15.27:38004) 2024-01-15T18:13:06.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.647+0000 7f6959400640 1 -- 172.21.15.27:0/1987729263 learned_addr learned my addr 172.21.15.27:0/1987729263 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:06.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.647+0000 7f6959400640 1 -- 172.21.15.27:0/1987729263 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f695c1597c0 msgr2=0x7f695c15ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:06.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.647+0000 7f6959400640 1 --2- 172.21.15.27:0/1987729263 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f695c1597c0 0x7f695c15ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:06.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.647+0000 7f6959400640 1 -- 172.21.15.27:0/1987729263 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f695c070d50 msgr2=0x7f695c15c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:06.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.647+0000 7f6959400640 1 --2- 172.21.15.27:0/1987729263 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f695c070d50 0x7f695c15c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:06.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.647+0000 7f6959400640 1 -- 172.21.15.27:0/1987729263 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6954009a70 con 0x7f695c0ae780 2024-01-15T18:13:06.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.647+0000 7f6959400640 1 --2- 172.21.15.27:0/1987729263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f695c0ae780 0x7f695c15a5d0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f694400d620 tx=0x7f694400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.647+0000 7f6952000640 1 -- 172.21.15.27:0/1987729263 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694400b620 con 0x7f695c0ae780 2024-01-15T18:13:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.647+0000 7f6952000640 1 -- 172.21.15.27:0/1987729263 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f694400b780 con 0x7f695c0ae780 2024-01-15T18:13:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.647+0000 7f6952000640 1 -- 172.21.15.27:0/1987729263 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694400bb10 con 0x7f695c0ae780 2024-01-15T18:13:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.647+0000 7f6960e00640 1 -- 172.21.15.27:0/1987729263 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f695c15b1b0 con 0x7f695c0ae780 2024-01-15T18:13:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.647+0000 7f6960e00640 1 -- 172.21.15.27:0/1987729263 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f695c0af030 con 0x7f695c0ae780 2024-01-15T18:13:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.651+0000 7f6960e00640 1 -- 172.21.15.27:0/1987729263 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f695c04e850 con 0x7f695c0ae780 2024-01-15T18:13:06.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.651+0000 7f6952000640 1 -- 172.21.15.27:0/1987729263 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) v1 ==== 87078+0+0 (secure 0 0 0) 0x7f694400bc70 con 0x7f695c0ae780 2024-01-15T18:13:06.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.651+0000 7f6952000640 1 --2- 172.21.15.27:0/1987729263 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6934067310 0x7f69340697d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:06.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.651+0000 7f6952000640 1 -- 172.21.15.27:0/1987729263 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f6944083400 con 0x7f695c0ae780 2024-01-15T18:13:06.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.663+0000 7f6952000640 1 -- 172.21.15.27:0/1987729263 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6944051ed0 con 0x7f695c0ae780 2024-01-15T18:13:06.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.667+0000 7f6959e00640 1 --2- 172.21.15.27:0/1987729263 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6934067310 0x7f69340697d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:06.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.671+0000 7f6959e00640 1 --2- 172.21.15.27:0/1987729263 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6934067310 0x7f69340697d0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f694c0011f0 tx=0x7f694c007930 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:06.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:06.955+0000 7f6960e00640 1 -- 172.21.15.27:0/1987729263 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone51934", "target": ["mon-mgr", ""]}) v1 -- 0x7f695c0cf270 con 0x7f6934067310 2024-01-15T18:13:07.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.003+0000 7f6952000640 1 -- 172.21.15.27:0/1987729263 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f695c0cf270 con 0x7f6934067310 2024-01-15T18:13:07.006 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:13:07.006 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:13:07.006 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:13:07.006 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:13:07.006 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:13:07.006 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:13:07.007 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:13:07.007 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:07.007 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:13:07.007 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 0.0, 2024-01-15T18:13:07.007 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "0 B/3 GB", 2024-01-15T18:13:07.007 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "0/3", 2024-01-15T18:13:07.007 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:13:07.007 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:13:07.007 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:07.007 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:07.008 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:13:07.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.007+0000 7f6960e00640 1 -- 172.21.15.27:0/1987729263 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6934067310 msgr2=0x7f69340697d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:07.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.007+0000 7f6960e00640 1 --2- 172.21.15.27:0/1987729263 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6934067310 0x7f69340697d0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f694c0011f0 tx=0x7f694c007930 comp rx=0 tx=0).stop 2024-01-15T18:13:07.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.007+0000 7f6960e00640 1 -- 172.21.15.27:0/1987729263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f695c0ae780 msgr2=0x7f695c15a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:07.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.007+0000 7f6960e00640 1 --2- 172.21.15.27:0/1987729263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f695c0ae780 0x7f695c15a5d0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f694400d620 tx=0x7f694400dae0 comp rx=0 tx=0).stop 2024-01-15T18:13:07.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.007+0000 7f6960e00640 1 -- 172.21.15.27:0/1987729263 shutdown_connections 2024-01-15T18:13:07.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.007+0000 7f6960e00640 1 --2- 172.21.15.27:0/1987729263 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6934067310 0x7f69340697d0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:07.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.007+0000 7f6960e00640 1 --2- 172.21.15.27:0/1987729263 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f695c1597c0 0x7f695c15ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:07.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.007+0000 7f6960e00640 1 --2- 172.21.15.27:0/1987729263 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f695c0ae780 0x7f695c15a5d0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:07.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.007+0000 7f6960e00640 1 --2- 172.21.15.27:0/1987729263 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f695c070d50 0x7f695c15c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:07.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.007+0000 7f6960e00640 1 -- 172.21.15.27:0/1987729263 >> 172.21.15.27:0/1987729263 conn(0x7f695c0b1620 msgr2=0x7f695c0b1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:07.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.007+0000 7f6960e00640 1 -- 172.21.15.27:0/1987729263 shutdown_connections 2024-01-15T18:13:07.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.007+0000 7f6960e00640 1 -- 172.21.15.27:0/1987729263 wait complete. 2024-01-15T18:13:07.046 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-15T18:13:07.046 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone51937 2024-01-15T18:13:07.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.291+0000 7f88a8200640 1 Processor -- start 2024-01-15T18:13:07.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.291+0000 7f88a8200640 1 -- start start 2024-01-15T18:13:07.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.295+0000 7f88a8200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88a00cf460 0x7f88a00cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:07.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.295+0000 7f88a8200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f88a015b4b0 0x7f88a00cba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:07.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.295+0000 7f88a8200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f88a00cec60 0x7f88a00cbf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:07.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.295+0000 7f88a8200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88a00cfd80 con 0x7f88a00cec60 2024-01-15T18:13:07.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.295+0000 7f88a8200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88a00d19a0 con 0x7f88a015b4b0 2024-01-15T18:13:07.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.295+0000 7f88a8200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a00d1c80 con 0x7f88a00cf460 2024-01-15T18:13:07.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.295+0000 7f88a6200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f88a00cec60 0x7f88a00cbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:07.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.295+0000 7f88a6200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f88a00cec60 0x7f88a00cbf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38016/0 (socket says 172.21.15.27:38016) 2024-01-15T18:13:07.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.295+0000 7f88a6200640 1 -- 172.21.15.27:0/849213673 learned_addr learned my addr 172.21.15.27:0/849213673 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:07.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.295+0000 7f88a6200640 1 -- 172.21.15.27:0/849213673 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88a00cf460 msgr2=0x7f88a00cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:07.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.295+0000 7f88a6200640 1 --2- 172.21.15.27:0/849213673 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88a00cf460 0x7f88a00cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:07.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.295+0000 7f88a6200640 1 -- 172.21.15.27:0/849213673 >> v2:172.21.15.93:3300/0 conn(0x7f88a015b4b0 msgr2=0x7f88a00cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:07.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.295+0000 7f88a6200640 1 --2- 172.21.15.27:0/849213673 >> v2:172.21.15.93:3300/0 conn(0x7f88a015b4b0 0x7f88a00cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:07.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.295+0000 7f88a6200640 1 -- 172.21.15.27:0/849213673 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a00b07c0 con 0x7f88a00cec60 2024-01-15T18:13:07.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.299+0000 7f88a6200640 1 --2- 172.21.15.27:0/849213673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88a00cec60 0x7f88a00cbf60 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f88a0067280 tx=0x7f889001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88bec7e4a7720d5c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:07.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.299+0000 7f889fe00640 1 -- 172.21.15.27:0/849213673 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88900a5850 con 0x7f88a00cec60 2024-01-15T18:13:07.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.299+0000 7f889fe00640 1 -- 172.21.15.27:0/849213673 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88900a59b0 con 0x7f88a00cec60 2024-01-15T18:13:07.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.299+0000 7f889fe00640 1 -- 172.21.15.27:0/849213673 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88900a5d40 con 0x7f88a00cec60 2024-01-15T18:13:07.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.299+0000 7f88a8200640 1 -- 172.21.15.27:0/849213673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88a00cec60 msgr2=0x7f88a00cbf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:07.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.299+0000 7f88a8200640 1 --2- 172.21.15.27:0/849213673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88a00cec60 0x7f88a00cbf60 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f88a0067280 tx=0x7f889001c9d0 comp rx=0 tx=0).stop 2024-01-15T18:13:07.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.303+0000 7f88a8200640 1 -- 172.21.15.27:0/849213673 shutdown_connections 2024-01-15T18:13:07.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.303+0000 7f88a8200640 1 --2- 172.21.15.27:0/849213673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88a00cec60 0x7f88a00cbf60 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:07.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.303+0000 7f88a8200640 1 --2- 172.21.15.27:0/849213673 >> v2:172.21.15.93:3300/0 conn(0x7f88a015b4b0 0x7f88a00cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:07.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.303+0000 7f88a8200640 1 --2- 172.21.15.27:0/849213673 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88a00cf460 0x7f88a00cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:07.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.303+0000 7f88a8200640 1 -- 172.21.15.27:0/849213673 >> 172.21.15.27:0/849213673 conn(0x7f88a00d0e80 msgr2=0x7f88a00d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:07.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.303+0000 7f88a8200640 1 -- 172.21.15.27:0/849213673 shutdown_connections 2024-01-15T18:13:07.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.303+0000 7f88a8200640 1 -- 172.21.15.27:0/849213673 wait complete. 2024-01-15T18:13:07.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.307+0000 7f88a8200640 1 Processor -- start 2024-01-15T18:13:07.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.307+0000 7f88a8200640 1 -- start start 2024-01-15T18:13:07.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.307+0000 7f88a8200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88a00cec60 0x7f88a01444b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:07.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.307+0000 7f88a8200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88a00cf460 0x7f88a01579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:07.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.307+0000 7f88a8200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88a015b4b0 0x7f88a0157f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:07.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.307+0000 7f88a8200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a0158e00 con 0x7f88a015b4b0 2024-01-15T18:13:07.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.307+0000 7f88a8200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a00c7340 con 0x7f88a00cf460 2024-01-15T18:13:07.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.307+0000 7f88a8200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a0158560 con 0x7f88a00cec60 2024-01-15T18:13:07.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.307+0000 7f88a6200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88a015b4b0 0x7f88a0157f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:07.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.307+0000 7f88a6200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88a015b4b0 0x7f88a0157f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38020/0 (socket says 172.21.15.27:38020) 2024-01-15T18:13:07.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.307+0000 7f88a6200640 1 -- 172.21.15.27:0/2623292968 learned_addr learned my addr 172.21.15.27:0/2623292968 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:07.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.307+0000 7f88a6200640 1 -- 172.21.15.27:0/2623292968 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88a00cec60 msgr2=0x7f88a01444b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:07.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.307+0000 7f88a6200640 1 --2- 172.21.15.27:0/2623292968 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88a00cec60 0x7f88a01444b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:07.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.307+0000 7f88a6200640 1 -- 172.21.15.27:0/2623292968 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88a00cf460 msgr2=0x7f88a01579f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:07.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.307+0000 7f88a6200640 1 --2- 172.21.15.27:0/2623292968 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88a00cf460 0x7f88a01579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:07.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.307+0000 7f88a6200640 1 -- 172.21.15.27:0/2623292968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8890009b30 con 0x7f88a015b4b0 2024-01-15T18:13:07.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.307+0000 7f88a6200640 1 --2- 172.21.15.27:0/2623292968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88a015b4b0 0x7f88a0157f30 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f889001b080 tx=0x7f8890019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.307+0000 7f889d600640 1 -- 172.21.15.27:0/2623292968 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88900b8880 con 0x7f88a015b4b0 2024-01-15T18:13:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.307+0000 7f889d600640 1 -- 172.21.15.27:0/2623292968 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88900b8a00 con 0x7f88a015b4b0 2024-01-15T18:13:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.307+0000 7f889d600640 1 -- 172.21.15.27:0/2623292968 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88900b8db0 con 0x7f88a015b4b0 2024-01-15T18:13:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.307+0000 7f88a8200640 1 -- 172.21.15.27:0/2623292968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a0158840 con 0x7f88a015b4b0 2024-01-15T18:13:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.307+0000 7f88a8200640 1 -- 172.21.15.27:0/2623292968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a013ebb0 con 0x7f88a015b4b0 2024-01-15T18:13:07.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.311+0000 7f88a8200640 1 -- 172.21.15.27:0/2623292968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a013ea10 con 0x7f88a015b4b0 2024-01-15T18:13:07.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.311+0000 7f889d600640 1 -- 172.21.15.27:0/2623292968 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f88900c3460 con 0x7f88a015b4b0 2024-01-15T18:13:07.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.311+0000 7f889d600640 1 --2- 172.21.15.27:0/2623292968 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88780673a0 0x7f8878069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:07.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.311+0000 7f889d600640 1 -- 172.21.15.27:0/2623292968 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f889011e420 con 0x7f88a015b4b0 2024-01-15T18:13:07.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.323+0000 7f889d600640 1 -- 172.21.15.27:0/2623292968 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f88a013ea10 con 0x7f88a015b4b0 2024-01-15T18:13:07.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.331+0000 7f88a5800640 1 --2- 172.21.15.27:0/2623292968 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88780673a0 0x7f8878069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:07.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.335+0000 7f88a5800640 1 --2- 172.21.15.27:0/2623292968 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88780673a0 0x7f8878069860 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8898009730 tx=0x7f8898006ca0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:07.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.655+0000 7f88a8200640 1 -- 172.21.15.27:0/2623292968 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone51937", "target": ["mon-mgr", ""]}) v1 -- 0x7f88a00cc8b0 con 0x7f88780673a0 2024-01-15T18:13:07.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.707+0000 7f889d600640 1 -- 172.21.15.27:0/2623292968 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f8898009340 con 0x7f88780673a0 2024-01-15T18:13:07.708 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:13:07.708 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:13:07.708 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "pending", 2024-01-15T18:13:07.708 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:13:07.708 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:13:07.709 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:13:07.709 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:13:07.709 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:07.709 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:07.709 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:13:07.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.711+0000 7f88a8200640 1 -- 172.21.15.27:0/2623292968 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88780673a0 msgr2=0x7f8878069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:07.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.711+0000 7f88a8200640 1 --2- 172.21.15.27:0/2623292968 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88780673a0 0x7f8878069860 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f8898009730 tx=0x7f8898006ca0 comp rx=0 tx=0).stop 2024-01-15T18:13:07.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.711+0000 7f88a8200640 1 -- 172.21.15.27:0/2623292968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88a015b4b0 msgr2=0x7f88a0157f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:07.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.711+0000 7f88a8200640 1 --2- 172.21.15.27:0/2623292968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88a015b4b0 0x7f88a0157f30 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f889001b080 tx=0x7f8890019770 comp rx=0 tx=0).stop 2024-01-15T18:13:07.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.711+0000 7f88a8200640 1 -- 172.21.15.27:0/2623292968 shutdown_connections 2024-01-15T18:13:07.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.711+0000 7f88a8200640 1 --2- 172.21.15.27:0/2623292968 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88780673a0 0x7f8878069860 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:07.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.711+0000 7f88a8200640 1 --2- 172.21.15.27:0/2623292968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88a015b4b0 0x7f88a0157f30 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:07.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.711+0000 7f88a8200640 1 --2- 172.21.15.27:0/2623292968 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88a00cf460 0x7f88a01579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:07.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.711+0000 7f88a8200640 1 --2- 172.21.15.27:0/2623292968 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88a00cec60 0x7f88a01444b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:07.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.711+0000 7f88a8200640 1 -- 172.21.15.27:0/2623292968 >> 172.21.15.27:0/2623292968 conn(0x7f88a00d0e80 msgr2=0x7f88a006c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:07.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.711+0000 7f88a8200640 1 -- 172.21.15.27:0/2623292968 shutdown_connections 2024-01-15T18:13:07.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.711+0000 7f88a8200640 1 -- 172.21.15.27:0/2623292968 wait complete. 2024-01-15T18:13:07.755 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:13:07.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.971+0000 7fbfdfe00640 1 Processor -- start 2024-01-15T18:13:07.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.975+0000 7fbfdfe00640 1 -- start start 2024-01-15T18:13:07.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.975+0000 7fbfdfe00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fbfe00cb4a0 0x7fbfe00c8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:07.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.975+0000 7fbfdfe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbfe006bc10 0x7fbfe015dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:07.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.975+0000 7fbfdfe00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe015d800 con 0x7fbfe00aed30 2024-01-15T18:13:07.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.975+0000 7fbfdfe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe01612d0 con 0x7fbfe006bc10 2024-01-15T18:13:07.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.975+0000 7fbfdfe00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe01615b0 con 0x7fbfe00cb4a0 2024-01-15T18:13:07.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.979+0000 7fbfdf400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbfe006bc10 0x7fbfe015dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:07.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.979+0000 7fbfdf400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbfe006bc10 0x7fbfe015dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38022/0 (socket says 172.21.15.27:38022) 2024-01-15T18:13:07.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.979+0000 7fbfdf400640 1 -- 172.21.15.27:0/3466683188 learned_addr learned my addr 172.21.15.27:0/3466683188 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:07.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.979+0000 7fbfdea00640 1 --2- 172.21.15.27:0/3466683188 >> v2:172.21.15.93:3300/0 conn(0x7fbfe00cb4a0 0x7fbfe00c8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:07.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.979+0000 7fbfdf400640 1 -- 172.21.15.27:0/3466683188 >> v2:172.21.15.93:3300/0 conn(0x7fbfe00cb4a0 msgr2=0x7fbfe00c8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:07.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.979+0000 7fbfdf400640 1 --2- 172.21.15.27:0/3466683188 >> v2:172.21.15.93:3300/0 conn(0x7fbfe00cb4a0 0x7fbfe00c8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:07.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.979+0000 7fbfdf400640 1 -- 172.21.15.27:0/3466683188 >> v1:172.21.15.93:6789/0 conn(0x7fbfe00aed30 legacy=0x7fbfe006b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:07.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.979+0000 7fbfdf400640 1 -- 172.21.15.27:0/3466683188 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe0161890 con 0x7fbfe006bc10 2024-01-15T18:13:07.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.983+0000 7fbfdf400640 1 --2- 172.21.15.27:0/3466683188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfe006bc10 0x7fbfe015dcc0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fbfe004eea0 tx=0x7fbfd801c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27d9e1dd83904eb4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.983+0000 7fbfdd600640 1 -- 172.21.15.27:0/3466683188 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd80a57b0 con 0x7fbfe006bc10 2024-01-15T18:13:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.983+0000 7fbfdd600640 1 -- 172.21.15.27:0/3466683188 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd80a5910 con 0x7fbfe006bc10 2024-01-15T18:13:07.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.983+0000 7fbfdd600640 1 -- 172.21.15.27:0/3466683188 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd80a5ca0 con 0x7fbfe006bc10 2024-01-15T18:13:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.987+0000 7fbfdfe00640 1 -- 172.21.15.27:0/3466683188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfe006bc10 msgr2=0x7fbfe015dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:07.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.987+0000 7fbfdfe00640 1 --2- 172.21.15.27:0/3466683188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfe006bc10 0x7fbfe015dcc0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fbfe004eea0 tx=0x7fbfd801c8d0 comp rx=0 tx=0).stop 2024-01-15T18:13:07.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.987+0000 7fbfdfe00640 1 -- 172.21.15.27:0/3466683188 shutdown_connections 2024-01-15T18:13:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.987+0000 7fbfdfe00640 1 --2- 172.21.15.27:0/3466683188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfe006bc10 0x7fbfe015dcc0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.987+0000 7fbfdfe00640 1 --2- 172.21.15.27:0/3466683188 >> v2:172.21.15.93:3300/0 conn(0x7fbfe00cb4a0 0x7fbfe00c8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.987+0000 7fbfdfe00640 1 -- 172.21.15.27:0/3466683188 >> 172.21.15.27:0/3466683188 conn(0x7fbfe0070af0 msgr2=0x7fbfe0070f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.987+0000 7fbfdfe00640 1 -- 172.21.15.27:0/3466683188 shutdown_connections 2024-01-15T18:13:07.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.987+0000 7fbfdfe00640 1 -- 172.21.15.27:0/3466683188 wait complete. 2024-01-15T18:13:07.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.991+0000 7fbfdfe00640 1 Processor -- start 2024-01-15T18:13:07.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.991+0000 7fbfdfe00640 1 -- start start 2024-01-15T18:13:07.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.991+0000 7fbfdfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfe006bc10 0x7fbfe0153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:07.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.991+0000 7fbfdfe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbfe00cb4a0 0x7fbfe006dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:07.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.991+0000 7fbfdfe00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfe006e0e0 0x7fbfe01fddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:07.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.991+0000 7fbfdfe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe00b22a0 con 0x7fbfe006bc10 2024-01-15T18:13:07.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.991+0000 7fbfdfe00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe00b2de0 con 0x7fbfe006e0e0 2024-01-15T18:13:07.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.991+0000 7fbfdfe00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe006e570 con 0x7fbfe00cb4a0 2024-01-15T18:13:07.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.991+0000 7fbfdea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfe006bc10 0x7fbfe0153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:07.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.991+0000 7fbfdea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfe006bc10 0x7fbfe0153ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38036/0 (socket says 172.21.15.27:38036) 2024-01-15T18:13:07.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.991+0000 7fbfdea00640 1 -- 172.21.15.27:0/781592560 learned_addr learned my addr 172.21.15.27:0/781592560 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:07.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.995+0000 7fbfde000640 1 --2- 172.21.15.27:0/781592560 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbfe00cb4a0 0x7fbfe006dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:07.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.995+0000 7fbfdea00640 1 -- 172.21.15.27:0/781592560 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbfe00cb4a0 msgr2=0x7fbfe006dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:07.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.995+0000 7fbfdea00640 1 --2- 172.21.15.27:0/781592560 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbfe00cb4a0 0x7fbfe006dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:07.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.995+0000 7fbfdea00640 1 -- 172.21.15.27:0/781592560 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfe006e0e0 msgr2=0x7fbfe01fddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:07.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.995+0000 7fbfdea00640 1 --2- 172.21.15.27:0/781592560 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfe006e0e0 0x7fbfe01fddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:07.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.995+0000 7fbfdea00640 1 -- 172.21.15.27:0/781592560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd8009b30 con 0x7fbfe006bc10 2024-01-15T18:13:07.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.995+0000 7fbfdea00640 1 --2- 172.21.15.27:0/781592560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfe006bc10 0x7fbfe0153ea0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fbfd000d5f0 tx=0x7fbfd000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:07.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.995+0000 7fbfd6a00640 1 -- 172.21.15.27:0/781592560 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd00125f0 con 0x7fbfe006bc10 2024-01-15T18:13:07.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.995+0000 7fbfd6a00640 1 -- 172.21.15.27:0/781592560 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd0012750 con 0x7fbfe006bc10 2024-01-15T18:13:07.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.995+0000 7fbfdfe00640 1 -- 172.21.15.27:0/781592560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe00b24b0 con 0x7fbfe006bc10 2024-01-15T18:13:07.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.995+0000 7fbfd6a00640 1 -- 172.21.15.27:0/781592560 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd0012ae0 con 0x7fbfe006bc10 2024-01-15T18:13:07.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.995+0000 7fbfdfe00640 1 -- 172.21.15.27:0/781592560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe00b2a70 con 0x7fbfe006bc10 2024-01-15T18:13:07.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.995+0000 7fbfdfe00640 1 -- 172.21.15.27:0/781592560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe004e850 con 0x7fbfe006bc10 2024-01-15T18:13:08.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.999+0000 7fbfd6a00640 1 -- 172.21.15.27:0/781592560 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7fbfd0012c40 con 0x7fbfe006bc10 2024-01-15T18:13:08.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.999+0000 7fbfd6a00640 1 --2- 172.21.15.27:0/781592560 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbfb80673d0 0x7fbfb8069890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:08.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.999+0000 7fbfd6a00640 1 -- 172.21.15.27:0/781592560 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fbfd0083200 con 0x7fbfe006bc10 2024-01-15T18:13:08.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:07.999+0000 7fbfde000640 1 --2- 172.21.15.27:0/781592560 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbfb80673d0 0x7fbfb8069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:08.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:08.003+0000 7fbfde000640 1 --2- 172.21.15.27:0/781592560 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbfb80673d0 0x7fbfb8069890 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fbfe00b53a0 tx=0x7fbfc8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:08.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:08.027+0000 7fbfd6a00640 1 -- 172.21.15.27:0/781592560 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbfd0051ef0 con 0x7fbfe006bc10 2024-01-15T18:13:08.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:08.379+0000 7fbfdfe00640 1 -- 172.21.15.27:0/781592560 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbfe0067d00 con 0x7fbfe006bc10 2024-01-15T18:13:08.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:08.379+0000 7fbfd6a00640 1 -- 172.21.15.27:0/781592560 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2791 (secure 0 0 0) 0x7fbfd0055510 con 0x7fbfe006bc10 2024-01-15T18:13:08.383 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:13:08.383 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:13:08.383 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:13:08.383 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:13:08.383 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:13:08.384 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:13:08.384 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:13:08.384 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:08.384 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:13:08.384 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:13:08.384 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:13:08.385 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:13:08.385 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:13:08.385 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:08.385 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:13:08.385 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:13:08.385 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:13:08.385 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:13:08.386 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:08.386 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 515, 2024-01-15T18:13:08.386 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:13:08.386 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:13:08.386 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:13:08.386 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:13:08.386 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:08.386 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:13:08.386 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 72, 2024-01-15T18:13:08.387 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:13:08.387 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:13:08.387 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:13:08.387 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:13:08.387 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:13:08.387 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:13:08.387 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:08.387 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:13:08.388 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:13:08.388 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:08.388 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:13:08.388 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:13:08.388 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:08.388 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:08.388 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:13:08.389 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:13:08.389 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 803, 2024-01-15T18:13:08.389 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 3222393243, 2024-01-15T18:13:08.389 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 6867857408, 2024-01-15T18:13:08.389 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 766226255872, 2024-01-15T18:13:08.389 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:13:08.389 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 1108, 2024-01-15T18:13:08.389 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 24898, 2024-01-15T18:13:08.390 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 1, 2024-01-15T18:13:08.390 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 7 2024-01-15T18:13:08.390 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:08.390 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:13:08.390 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 48, 2024-01-15T18:13:08.390 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 5, 2024-01-15T18:13:08.390 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:13:08.390 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:13:08.390 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:13:08.391 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:13:08.391 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:08.391 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 5, 2024-01-15T18:13:08.391 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:13:08.391 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:13:08.391 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:13:08.391 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 7227 2024-01-15T18:13:08.391 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:08.391 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:08.392 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:13:08.392 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:08.392 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:13:08.392 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:13:08.392 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:13:08.392 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:13:08.392 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:13:08.392 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:13:08.393 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:13:08.393 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:08.393 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:13:08.393 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:08.393 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:13:08.393 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 15, 2024-01-15T18:13:08.393 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:12:58.272961+0000", 2024-01-15T18:13:08.393 INFO:teuthology.orchestra.run.smithi027.stdout: "services": { 2024-01-15T18:13:08.393 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-01-15T18:13:08.394 INFO:teuthology.orchestra.run.smithi027.stdout: "daemons": { 2024-01-15T18:13:08.394 INFO:teuthology.orchestra.run.smithi027.stdout: "summary": "", 2024-01-15T18:13:08.394 INFO:teuthology.orchestra.run.smithi027.stdout: "c": { 2024-01-15T18:13:08.394 INFO:teuthology.orchestra.run.smithi027.stdout: "start_epoch": 0, 2024-01-15T18:13:08.394 INFO:teuthology.orchestra.run.smithi027.stdout: "start_stamp": "0.000000", 2024-01-15T18:13:08.394 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 0, 2024-01-15T18:13:08.394 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "(unrecognized address family 0)/0", 2024-01-15T18:13:08.394 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": {}, 2024-01-15T18:13:08.394 INFO:teuthology.orchestra.run.smithi027.stdout: "task_status": {} 2024-01-15T18:13:08.395 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:08.395 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:08.395 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:08.395 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:08.395 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:08.395 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:13:08.395 INFO:teuthology.orchestra.run.smithi027.stdout: "6eefee50-4f20-4629-a81e-c700e83869a8": { 2024-01-15T18:13:08.395 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "4 ongoing cloning jobs; Avg progress = 0.0% (9s)\n [............................] ", 2024-01-15T18:13:08.395 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0, 2024-01-15T18:13:08.396 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:08.396 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:08.396 INFO:teuthology.orchestra.run.smithi027.stdout: "8fb3fd3d-b405-4726-bf6b-75de006e8a09": { 2024-01-15T18:13:08.396 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 0.0% (9s)\n [............................] ", 2024-01-15T18:13:08.396 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0, 2024-01-15T18:13:08.396 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:08.396 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:08.396 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:08.396 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:13:08.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:08.383+0000 7fbfdfe00640 1 -- 172.21.15.27:0/781592560 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbfb80673d0 msgr2=0x7fbfb8069890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:08.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:08.383+0000 7fbfdfe00640 1 --2- 172.21.15.27:0/781592560 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbfb80673d0 0x7fbfb8069890 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fbfe00b53a0 tx=0x7fbfc8009040 comp rx=0 tx=0).stop 2024-01-15T18:13:08.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:08.383+0000 7fbfdfe00640 1 -- 172.21.15.27:0/781592560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfe006bc10 msgr2=0x7fbfe0153ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:08.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:08.383+0000 7fbfdfe00640 1 --2- 172.21.15.27:0/781592560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfe006bc10 0x7fbfe0153ea0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fbfd000d5f0 tx=0x7fbfd000dab0 comp rx=0 tx=0).stop 2024-01-15T18:13:08.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:08.383+0000 7fbfdfe00640 1 -- 172.21.15.27:0/781592560 shutdown_connections 2024-01-15T18:13:08.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:08.383+0000 7fbfdfe00640 1 --2- 172.21.15.27:0/781592560 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbfb80673d0 0x7fbfb8069890 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:08.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:08.383+0000 7fbfdfe00640 1 --2- 172.21.15.27:0/781592560 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfe006e0e0 0x7fbfe01fddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:08.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:08.383+0000 7fbfdfe00640 1 --2- 172.21.15.27:0/781592560 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbfe00cb4a0 0x7fbfe006dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:08.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:08.383+0000 7fbfdfe00640 1 --2- 172.21.15.27:0/781592560 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfe006bc10 0x7fbfe0153ea0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:08.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:08.383+0000 7fbfdfe00640 1 -- 172.21.15.27:0/781592560 >> 172.21.15.27:0/781592560 conn(0x7fbfe0070af0 msgr2=0x7fbfe00b0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:08.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:08.383+0000 7fbfdfe00640 1 -- 172.21.15.27:0/781592560 shutdown_connections 2024-01-15T18:13:08.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:08.383+0000 7fbfdfe00640 1 -- 172.21.15.27:0/781592560 wait complete. 2024-01-15T18:13:09.406 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-15T18:13:09.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.555+0000 7f93a6e00640 1 Processor -- start 2024-01-15T18:13:09.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.555+0000 7f93a6e00640 1 -- start start 2024-01-15T18:13:09.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.559+0000 7f93a6e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f93a00cf9b0 0x7f93a00b10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:09.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.559+0000 7f93a6e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93a00b38b0 con 0x7f93a00cedc0 2024-01-15T18:13:09.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.559+0000 7f93a6e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93a00cd550 con 0x7f93a015e7d0 2024-01-15T18:13:09.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.559+0000 7f93a6e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a00cd270 con 0x7f93a00cf9b0 2024-01-15T18:13:09.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.559+0000 7f93a4e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f93a00cedc0 0x7f93a00b1600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60010/0 (socket says 172.21.15.27:60010) 2024-01-15T18:13:09.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.559+0000 7f93a4e00640 1 -- 172.21.15.27:0/3723761842 learned_addr learned my addr 172.21.15.27:0/3723761842 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:13:09.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.559+0000 7f939f400640 1 --2- 172.21.15.27:0/3723761842 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f93a00cf9b0 0x7f93a00b10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:09.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.559+0000 7f939ea00640 1 -- 172.21.15.27:0/3723761842 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 38182882 0 0) 0x7f93a00b38b0 con 0x7f93a00cedc0 2024-01-15T18:13:09.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.559+0000 7f939ea00640 1 -- 172.21.15.27:0/3723761842 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9380003530 con 0x7f93a00cedc0 2024-01-15T18:13:09.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.559+0000 7f939ea00640 1 -- 172.21.15.27:0/3723761842 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920977568 0 0) 0x7f93a00cd550 con 0x7f93a015e7d0 2024-01-15T18:13:09.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.559+0000 7f939ea00640 1 -- 172.21.15.27:0/3723761842 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93a00b38b0 con 0x7f93a015e7d0 2024-01-15T18:13:09.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.559+0000 7f939f400640 1 -- 172.21.15.27:0/3723761842 >> v1:172.21.15.93:6789/0 conn(0x7f93a015e7d0 legacy=0x7f93a00cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:09.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.559+0000 7f939f400640 1 -- 172.21.15.27:0/3723761842 >> v1:172.21.15.27:6789/0 conn(0x7f93a00cedc0 legacy=0x7f93a00b1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:09.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.559+0000 7f939f400640 1 -- 172.21.15.27:0/3723761842 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a00b35d0 con 0x7f93a00cf9b0 2024-01-15T18:13:09.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.563+0000 7f939f400640 1 --2- 172.21.15.27:0/3723761842 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f93a00cf9b0 0x7f93a00b10c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f93880097f0 tx=0x7f93a00b2520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56940f41ca9c8e7c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:09.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.563+0000 7f939ea00640 1 -- 172.21.15.27:0/3723761842 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93880ab600 con 0x7f93a00cf9b0 2024-01-15T18:13:09.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.563+0000 7f939ea00640 1 -- 172.21.15.27:0/3723761842 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93880a7050 con 0x7f93a00cf9b0 2024-01-15T18:13:09.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.563+0000 7f939ea00640 1 -- 172.21.15.27:0/3723761842 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93880ab960 con 0x7f93a00cf9b0 2024-01-15T18:13:09.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.563+0000 7f93a6e00640 1 -- 172.21.15.27:0/3723761842 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f93a00cf9b0 msgr2=0x7f93a00b10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:09.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.563+0000 7f93a6e00640 1 --2- 172.21.15.27:0/3723761842 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f93a00cf9b0 0x7f93a00b10c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f93880097f0 tx=0x7f93a00b2520 comp rx=0 tx=0).stop 2024-01-15T18:13:09.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.563+0000 7f93a6e00640 1 -- 172.21.15.27:0/3723761842 shutdown_connections 2024-01-15T18:13:09.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.563+0000 7f93a6e00640 1 --2- 172.21.15.27:0/3723761842 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f93a00cf9b0 0x7f93a00b10c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:09.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.563+0000 7f93a6e00640 1 -- 172.21.15.27:0/3723761842 >> 172.21.15.27:0/3723761842 conn(0x7f93a00b2a10 msgr2=0x7f93a00b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:09.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.563+0000 7f93a6e00640 1 -- 172.21.15.27:0/3723761842 shutdown_connections 2024-01-15T18:13:09.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.563+0000 7f93a6e00640 1 -- 172.21.15.27:0/3723761842 wait complete. 2024-01-15T18:13:09.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.567+0000 7f93a6e00640 1 Processor -- start 2024-01-15T18:13:09.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.567+0000 7f93a6e00640 1 -- start start 2024-01-15T18:13:09.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.567+0000 7f93a6e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93a00cedc0 0x7f93a015d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:09.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.567+0000 7f93a6e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f93a00cf9b0 0x7f93a00af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:09.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.567+0000 7f93a6e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f93a015e7d0 0x7f93a00c5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:09.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.567+0000 7f93a6e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a00af440 con 0x7f93a00cedc0 2024-01-15T18:13:09.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.567+0000 7f93a6e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a00ae830 con 0x7f93a015e7d0 2024-01-15T18:13:09.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.567+0000 7f93a6e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a00aeb10 con 0x7f93a00cf9b0 2024-01-15T18:13:09.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.567+0000 7f939fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93a00cedc0 0x7f93a015d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:09.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.567+0000 7f939fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93a00cedc0 0x7f93a015d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38048/0 (socket says 172.21.15.27:38048) 2024-01-15T18:13:09.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.567+0000 7f939fe00640 1 -- 172.21.15.27:0/2389597743 learned_addr learned my addr 172.21.15.27:0/2389597743 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:09.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.567+0000 7f939f400640 1 --2- 172.21.15.27:0/2389597743 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f93a00cf9b0 0x7f93a00af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:09.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.567+0000 7f93a4e00640 1 --2- 172.21.15.27:0/2389597743 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f93a015e7d0 0x7f93a00c5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:09.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.567+0000 7f939fe00640 1 -- 172.21.15.27:0/2389597743 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f93a00cf9b0 msgr2=0x7f93a00af8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:09.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.567+0000 7f939fe00640 1 --2- 172.21.15.27:0/2389597743 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f93a00cf9b0 0x7f93a00af8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:09.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.567+0000 7f939fe00640 1 -- 172.21.15.27:0/2389597743 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f93a015e7d0 msgr2=0x7f93a00c5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:09.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.567+0000 7f939fe00640 1 --2- 172.21.15.27:0/2389597743 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f93a015e7d0 0x7f93a00c5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:09.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.567+0000 7f939fe00640 1 -- 172.21.15.27:0/2389597743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9388009420 con 0x7f93a00cedc0 2024-01-15T18:13:09.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.567+0000 7f939fe00640 1 --2- 172.21.15.27:0/2389597743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93a00cedc0 0x7f93a015d8d0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f939000c550 tx=0x7f939000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:09.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.567+0000 7f9397e00640 1 -- 172.21.15.27:0/2389597743 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9390005480 con 0x7f93a00cedc0 2024-01-15T18:13:09.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.567+0000 7f9397e00640 1 -- 172.21.15.27:0/2389597743 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9390003600 con 0x7f93a00cedc0 2024-01-15T18:13:09.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.567+0000 7f93a6e00640 1 -- 172.21.15.27:0/2389597743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a00aedf0 con 0x7f93a00cedc0 2024-01-15T18:13:09.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.567+0000 7f9397e00640 1 -- 172.21.15.27:0/2389597743 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93900039f0 con 0x7f93a00cedc0 2024-01-15T18:13:09.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.567+0000 7f93a6e00640 1 -- 172.21.15.27:0/2389597743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a00c6900 con 0x7f93a00cedc0 2024-01-15T18:13:09.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.571+0000 7f93a6e00640 1 -- 172.21.15.27:0/2389597743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a004e850 con 0x7f93a00cedc0 2024-01-15T18:13:09.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.571+0000 7f9397e00640 1 -- 172.21.15.27:0/2389597743 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f9390003b50 con 0x7f93a00cedc0 2024-01-15T18:13:09.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.571+0000 7f9397e00640 1 --2- 172.21.15.27:0/2389597743 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f93780673a0 0x7f9378069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:09.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.571+0000 7f9397e00640 1 -- 172.21.15.27:0/2389597743 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f9390082a20 con 0x7f93a00cedc0 2024-01-15T18:13:09.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.571+0000 7f939f400640 1 --2- 172.21.15.27:0/2389597743 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f93780673a0 0x7f9378069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:09.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.571+0000 7f939f400640 1 --2- 172.21.15.27:0/2389597743 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f93780673a0 0x7f9378069860 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f93880ad000 tx=0x7f9388004740 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:09.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.575+0000 7f9397e00640 1 -- 172.21.15.27:0/2389597743 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9390055140 con 0x7f93a00cedc0 2024-01-15T18:13:09.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.787+0000 7f93a6e00640 1 -- 172.21.15.27:0/2389597743 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f93a00ce0a0 con 0x7f93a00cedc0 2024-01-15T18:13:09.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.787+0000 7f9397e00640 1 -- 172.21.15.27:0/2389597743 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2337 (secure 0 0 0) 0x7f9390054cd0 con 0x7f93a00cedc0 2024-01-15T18:13:09.789 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:13:09.790 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:13:09.790 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:13:09.790 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:13:09.790 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:13:09.790 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:13:09.790 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:13:09.790 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:09.790 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:13:09.790 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:13:09.791 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:13:09.791 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:13:09.791 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:13:09.791 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:09.791 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:13:09.791 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:13:09.791 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:13:09.791 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:13:09.791 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:09.791 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 516, 2024-01-15T18:13:09.791 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:13:09.792 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:13:09.792 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:13:09.792 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:13:09.792 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:09.792 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:13:09.792 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 72, 2024-01-15T18:13:09.792 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:13:09.792 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:13:09.792 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:13:09.793 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:13:09.793 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:13:09.793 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:13:09.793 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:09.793 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:13:09.793 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:13:09.793 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:09.793 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:13:09.793 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:13:09.793 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:09.793 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:09.794 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:13:09.794 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:13:09.794 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 803, 2024-01-15T18:13:09.794 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 3222393243, 2024-01-15T18:13:09.794 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 6867857408, 2024-01-15T18:13:09.794 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 766226255872, 2024-01-15T18:13:09.794 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:13:09.794 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 852, 2024-01-15T18:13:09.794 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 19611, 2024-01-15T18:13:09.794 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 0, 2024-01-15T18:13:09.794 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 5 2024-01-15T18:13:09.795 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:09.795 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:13:09.795 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 48, 2024-01-15T18:13:09.795 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 5, 2024-01-15T18:13:09.795 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:13:09.795 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:13:09.795 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:13:09.795 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:13:09.795 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:09.795 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 5, 2024-01-15T18:13:09.796 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:13:09.796 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:13:09.796 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:13:09.796 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 7227 2024-01-15T18:13:09.796 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:09.796 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:09.796 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:13:09.796 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:09.796 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:13:09.796 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:13:09.797 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:13:09.797 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:13:09.797 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:13:09.797 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:13:09.797 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:13:09.797 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:09.797 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:13:09.797 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:09.797 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:13:09.797 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 16, 2024-01-15T18:13:09.797 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:13:08.281503+0000", 2024-01-15T18:13:09.797 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:13:09.798 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:09.798 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:13:09.798 INFO:teuthology.orchestra.run.smithi027.stdout: "6eefee50-4f20-4629-a81e-c700e83869a8": { 2024-01-15T18:13:09.798 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "4 ongoing cloning jobs; Avg progress = 0.0% (9s)\n [............................] ", 2024-01-15T18:13:09.798 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0, 2024-01-15T18:13:09.798 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:09.798 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:09.798 INFO:teuthology.orchestra.run.smithi027.stdout: "8fb3fd3d-b405-4726-bf6b-75de006e8a09": { 2024-01-15T18:13:09.798 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 0.0% (9s)\n [............................] ", 2024-01-15T18:13:09.798 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0, 2024-01-15T18:13:09.799 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:09.799 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:09.799 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:09.799 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:13:09.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.791+0000 7f93a6e00640 1 -- 172.21.15.27:0/2389597743 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f93780673a0 msgr2=0x7f9378069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:09.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.791+0000 7f93a6e00640 1 --2- 172.21.15.27:0/2389597743 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f93780673a0 0x7f9378069860 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f93880ad000 tx=0x7f9388004740 comp rx=0 tx=0).stop 2024-01-15T18:13:09.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.791+0000 7f93a6e00640 1 -- 172.21.15.27:0/2389597743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93a00cedc0 msgr2=0x7f93a015d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:09.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.791+0000 7f93a6e00640 1 --2- 172.21.15.27:0/2389597743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93a00cedc0 0x7f93a015d8d0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f939000c550 tx=0x7f939000ca10 comp rx=0 tx=0).stop 2024-01-15T18:13:09.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.791+0000 7f93a6e00640 1 -- 172.21.15.27:0/2389597743 shutdown_connections 2024-01-15T18:13:09.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.791+0000 7f93a6e00640 1 --2- 172.21.15.27:0/2389597743 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f93780673a0 0x7f9378069860 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:09.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.791+0000 7f93a6e00640 1 --2- 172.21.15.27:0/2389597743 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f93a015e7d0 0x7f93a00c5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:09.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.791+0000 7f93a6e00640 1 --2- 172.21.15.27:0/2389597743 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f93a00cf9b0 0x7f93a00af8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:09.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.791+0000 7f93a6e00640 1 --2- 172.21.15.27:0/2389597743 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93a00cedc0 0x7f93a015d8d0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:09.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.791+0000 7f93a6e00640 1 -- 172.21.15.27:0/2389597743 >> 172.21.15.27:0/2389597743 conn(0x7f93a00b2a10 msgr2=0x7f93a00b10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:09.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.791+0000 7f93a6e00640 1 -- 172.21.15.27:0/2389597743 shutdown_connections 2024-01-15T18:13:09.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:09.791+0000 7f93a6e00640 1 -- 172.21.15.27:0/2389597743 wait complete. 2024-01-15T18:13:10.812 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-15T18:13:10.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.963+0000 7f2ec2200640 1 Processor -- start 2024-01-15T18:13:10.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.963+0000 7f2ec2200640 1 -- start start 2024-01-15T18:13:10.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.963+0000 7f2ec2200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f2ebc0cedc0 0x7f2ebc0b17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:10.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.963+0000 7f2ec2200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ebc0b38b0 con 0x7f2ebc0cf9b0 2024-01-15T18:13:10.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.963+0000 7f2ec2200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ebc0cd550 con 0x7f2ebc15e7d0 2024-01-15T18:13:10.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.963+0000 7f2ec2200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ebc0cd270 con 0x7f2ebc0cedc0 2024-01-15T18:13:10.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.963+0000 7f2ebaa00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2ebc0cf9b0 0x7f2ebc0b10c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60012/0 (socket says 172.21.15.27:60012) 2024-01-15T18:13:10.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.963+0000 7f2ebaa00640 1 -- 172.21.15.27:0/1101333261 learned_addr learned my addr 172.21.15.27:0/1101333261 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:13:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.963+0000 7f2ebbe00640 1 --2- 172.21.15.27:0/1101333261 >> v2:172.21.15.93:3300/0 conn(0x7f2ebc0cedc0 0x7f2ebc0b17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.963+0000 7f2eba000640 1 -- 172.21.15.27:0/1101333261 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3134666878 0 0) 0x7f2ebc0b38b0 con 0x7f2ebc0cf9b0 2024-01-15T18:13:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.963+0000 7f2eba000640 1 -- 172.21.15.27:0/1101333261 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e9c003530 con 0x7f2ebc0cf9b0 2024-01-15T18:13:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.963+0000 7f2eba000640 1 -- 172.21.15.27:0/1101333261 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3003915428 0 0) 0x7f2e9c003530 con 0x7f2ebc0cf9b0 2024-01-15T18:13:10.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.963+0000 7f2eba000640 1 -- 172.21.15.27:0/1101333261 >> v2:172.21.15.93:3300/0 conn(0x7f2ebc0cedc0 msgr2=0x7f2ebc0b17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.963+0000 7f2eba000640 1 --2- 172.21.15.27:0/1101333261 >> v2:172.21.15.93:3300/0 conn(0x7f2ebc0cedc0 0x7f2ebc0b17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.963+0000 7f2eba000640 1 -- 172.21.15.27:0/1101333261 >> v1:172.21.15.93:6789/0 conn(0x7f2ebc15e7d0 legacy=0x7f2ebc0cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.963+0000 7f2eba000640 1 -- 172.21.15.27:0/1101333261 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ebc0b35d0 con 0x7f2ebc0cf9b0 2024-01-15T18:13:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.963+0000 7f2eba000640 1 -- 172.21.15.27:0/1101333261 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f2ea4001680 con 0x7f2ebc0cf9b0 2024-01-15T18:13:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.963+0000 7f2eba000640 1 -- 172.21.15.27:0/1101333261 >> v1:172.21.15.27:6789/0 conn(0x7f2ebc0cf9b0 legacy=0x7f2ebc0b10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.963+0000 7f2eba000640 1 --2- 172.21.15.27:0/1101333261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e9c003a40 0x7f2e9c003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.963+0000 7f2eba000640 1 --2- 172.21.15.27:0/1101333261 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2e9c0047d0 0x7f2e9c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.963+0000 7f2eba000640 1 --2- 172.21.15.27:0/1101333261 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2e9c0052b0 0x7f2e9c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.963+0000 7f2eba000640 1 -- 172.21.15.27:0/1101333261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc0cd270 con 0x7f2e9c003a40 2024-01-15T18:13:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.963+0000 7f2eba000640 1 -- 172.21.15.27:0/1101333261 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9c00c180 con 0x7f2e9c0047d0 2024-01-15T18:13:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.963+0000 7f2eba000640 1 -- 172.21.15.27:0/1101333261 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9c00c340 con 0x7f2e9c0052b0 2024-01-15T18:13:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.967+0000 7f2ebb400640 1 --2- 172.21.15.27:0/1101333261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e9c003a40 0x7f2e9c003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:10.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.967+0000 7f2ebaa00640 1 --2- 172.21.15.27:0/1101333261 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2e9c0047d0 0x7f2e9c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:10.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.967+0000 7f2ebbe00640 1 --2- 172.21.15.27:0/1101333261 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2e9c0052b0 0x7f2e9c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:10.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.967+0000 7f2ebb400640 1 -- 172.21.15.27:0/1101333261 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2e9c0052b0 msgr2=0x7f2e9c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:10.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.967+0000 7f2ebb400640 1 --2- 172.21.15.27:0/1101333261 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2e9c0052b0 0x7f2e9c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:10.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.967+0000 7f2ebb400640 1 -- 172.21.15.27:0/1101333261 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2e9c0047d0 msgr2=0x7f2e9c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:10.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.967+0000 7f2ebb400640 1 --2- 172.21.15.27:0/1101333261 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2e9c0047d0 0x7f2e9c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:10.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.967+0000 7f2ebb400640 1 -- 172.21.15.27:0/1101333261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e9c003530 con 0x7f2e9c003a40 2024-01-15T18:13:10.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.967+0000 7f2ebbe00640 1 -- 172.21.15.27:0/1101333261 reap_dead start 2024-01-15T18:13:10.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.967+0000 7f2ebb400640 1 --2- 172.21.15.27:0/1101333261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e9c003a40 0x7f2e9c003e60 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f2eac001780 tx=0x7f2eac01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e31defb17852d8e9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:10.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.967+0000 7f2eba000640 1 -- 172.21.15.27:0/1101333261 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eac0052d0 con 0x7f2e9c003a40 2024-01-15T18:13:10.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.967+0000 7f2eba000640 1 -- 172.21.15.27:0/1101333261 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eac0a7050 con 0x7f2e9c003a40 2024-01-15T18:13:10.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.967+0000 7f2eba000640 1 -- 172.21.15.27:0/1101333261 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eac003600 con 0x7f2e9c003a40 2024-01-15T18:13:10.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.971+0000 7f2ec2200640 1 -- 172.21.15.27:0/1101333261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e9c003a40 msgr2=0x7f2e9c003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:10.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.971+0000 7f2ec2200640 1 --2- 172.21.15.27:0/1101333261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e9c003a40 0x7f2e9c003e60 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f2eac001780 tx=0x7f2eac01c400 comp rx=0 tx=0).stop 2024-01-15T18:13:10.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.971+0000 7f2ec2200640 1 -- 172.21.15.27:0/1101333261 shutdown_connections 2024-01-15T18:13:10.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.971+0000 7f2ec2200640 1 --2- 172.21.15.27:0/1101333261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2e9c003a40 0x7f2e9c003e60 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:10.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.971+0000 7f2ec2200640 1 -- 172.21.15.27:0/1101333261 >> 172.21.15.27:0/1101333261 conn(0x7f2ebc0b2a10 msgr2=0x7f2ebc0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:10.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.971+0000 7f2ec2200640 1 -- 172.21.15.27:0/1101333261 shutdown_connections 2024-01-15T18:13:10.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.971+0000 7f2ec2200640 1 -- 172.21.15.27:0/1101333261 wait complete. 2024-01-15T18:13:10.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.971+0000 7f2ec2200640 1 Processor -- start 2024-01-15T18:13:10.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.971+0000 7f2ec2200640 1 -- start start 2024-01-15T18:13:10.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.971+0000 7f2ec2200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2e9c003a40 0x7f2ebc0b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:10.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.971+0000 7f2ec2200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ebc0b6570 0x7f2ebc06c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:10.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.971+0000 7f2ec2200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ebc06cb90 0x7f2ebc0c5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:10.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.971+0000 7f2ec2200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc06d010 con 0x7f2ebc0b6570 2024-01-15T18:13:10.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.971+0000 7f2ec2200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc0b4a20 con 0x7f2ebc06cb90 2024-01-15T18:13:10.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.971+0000 7f2ec2200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc0b5060 con 0x7f2e9c003a40 2024-01-15T18:13:10.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.971+0000 7f2ebaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ebc0b6570 0x7f2ebc06c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:10.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.971+0000 7f2ebaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ebc0b6570 0x7f2ebc06c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38054/0 (socket says 172.21.15.27:38054) 2024-01-15T18:13:10.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.971+0000 7f2ebaa00640 1 -- 172.21.15.27:0/2074379879 learned_addr learned my addr 172.21.15.27:0/2074379879 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:10.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.975+0000 7f2ebbe00640 1 --2- 172.21.15.27:0/2074379879 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ebc06cb90 0x7f2ebc0c5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:10.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.975+0000 7f2ebb400640 1 --2- 172.21.15.27:0/2074379879 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2e9c003a40 0x7f2ebc0b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:10.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.975+0000 7f2ebaa00640 1 -- 172.21.15.27:0/2074379879 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2e9c003a40 msgr2=0x7f2ebc0b44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:10.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.975+0000 7f2ebaa00640 1 --2- 172.21.15.27:0/2074379879 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2e9c003a40 0x7f2ebc0b44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:10.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.975+0000 7f2ebaa00640 1 -- 172.21.15.27:0/2074379879 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ebc06cb90 msgr2=0x7f2ebc0c5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:10.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.975+0000 7f2ebaa00640 1 --2- 172.21.15.27:0/2074379879 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ebc06cb90 0x7f2ebc0c5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:10.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.975+0000 7f2ebaa00640 1 -- 172.21.15.27:0/2074379879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ebc0b5340 con 0x7f2ebc0b6570 2024-01-15T18:13:10.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.975+0000 7f2ebaa00640 1 --2- 172.21.15.27:0/2074379879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ebc0b6570 0x7f2ebc06c650 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f2ea40017b0 tx=0x7f2ea4007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:10.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.975+0000 7f2eb3400640 1 -- 172.21.15.27:0/2074379879 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea4007770 con 0x7f2ebc0b6570 2024-01-15T18:13:10.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.975+0000 7f2eb3400640 1 -- 172.21.15.27:0/2074379879 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea400f750 con 0x7f2ebc0b6570 2024-01-15T18:13:10.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.975+0000 7f2eb3400640 1 -- 172.21.15.27:0/2074379879 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea400fb00 con 0x7f2ebc0b6570 2024-01-15T18:13:10.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.975+0000 7f2ec2200640 1 -- 172.21.15.27:0/2074379879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ebc0b5a90 con 0x7f2ebc0b6570 2024-01-15T18:13:10.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.975+0000 7f2ec2200640 1 -- 172.21.15.27:0/2074379879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ebc0b6050 con 0x7f2ebc0b6570 2024-01-15T18:13:10.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.975+0000 7f2ec2200640 1 -- 172.21.15.27:0/2074379879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ebc04e850 con 0x7f2ebc0b6570 2024-01-15T18:13:10.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.975+0000 7f2eb3400640 1 -- 172.21.15.27:0/2074379879 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f2ea400fc60 con 0x7f2ebc0b6570 2024-01-15T18:13:10.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.975+0000 7f2eb3400640 1 --2- 172.21.15.27:0/2074379879 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2e940673d0 0x7f2e94069890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:10.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.975+0000 7f2eb3400640 1 -- 172.21.15.27:0/2074379879 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f2ea4082be0 con 0x7f2ebc0b6570 2024-01-15T18:13:10.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.979+0000 7f2ebb400640 1 --2- 172.21.15.27:0/2074379879 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2e940673d0 0x7f2e94069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:10.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.979+0000 7f2ebb400640 1 --2- 172.21.15.27:0/2074379879 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2e940673d0 0x7f2e94069890 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2eac001750 tx=0x7f2eac0011a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:10.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:10.979+0000 7f2eb3400640 1 -- 172.21.15.27:0/2074379879 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2ea4051920 con 0x7f2ebc0b6570 2024-01-15T18:13:11.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:11.195+0000 7f2ec2200640 1 -- 172.21.15.27:0/2074379879 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2ebc0b2f90 con 0x7f2ebc0b6570 2024-01-15T18:13:11.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:11.195+0000 7f2eb3400640 1 -- 172.21.15.27:0/2074379879 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2337 (secure 0 0 0) 0x7f2ea4054f40 con 0x7f2ebc0b6570 2024-01-15T18:13:11.196 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:13:11.196 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:13:11.196 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:13:11.196 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:13:11.196 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:13:11.196 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:13:11.196 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:13:11.196 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:11.196 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:13:11.197 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:13:11.197 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:13:11.197 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:13:11.197 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:13:11.197 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:11.197 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:13:11.197 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:13:11.197 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:13:11.197 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:13:11.197 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:11.197 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 518, 2024-01-15T18:13:11.197 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:13:11.197 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:13:11.197 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:13:11.197 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:13:11.197 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:11.198 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:13:11.198 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 72, 2024-01-15T18:13:11.198 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:13:11.198 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:13:11.198 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:13:11.198 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:13:11.198 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:13:11.198 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:13:11.198 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:11.198 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:13:11.198 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:13:11.198 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:11.198 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:13:11.198 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:13:11.199 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:11.199 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:11.199 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:13:11.199 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:13:11.199 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 803, 2024-01-15T18:13:11.199 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 3222393243, 2024-01-15T18:13:11.199 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 6867857408, 2024-01-15T18:13:11.199 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 766226255872, 2024-01-15T18:13:11.199 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:13:11.199 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 852, 2024-01-15T18:13:11.199 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 19611, 2024-01-15T18:13:11.199 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 0, 2024-01-15T18:13:11.199 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 5 2024-01-15T18:13:11.199 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:11.200 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:13:11.200 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 48, 2024-01-15T18:13:11.200 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 5, 2024-01-15T18:13:11.200 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:13:11.200 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:13:11.200 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:13:11.200 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:13:11.200 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:11.200 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 5, 2024-01-15T18:13:11.200 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:13:11.200 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:13:11.200 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:13:11.200 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 7227 2024-01-15T18:13:11.200 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:11.200 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:11.201 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:13:11.201 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:11.201 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:13:11.201 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:13:11.201 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:13:11.201 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:13:11.201 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:13:11.201 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:13:11.201 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:13:11.201 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:11.201 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:13:11.201 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:11.201 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:13:11.201 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 16, 2024-01-15T18:13:11.201 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:13:08.281503+0000", 2024-01-15T18:13:11.201 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:13:11.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:11.202 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:13:11.202 INFO:teuthology.orchestra.run.smithi027.stdout: "6eefee50-4f20-4629-a81e-c700e83869a8": { 2024-01-15T18:13:11.202 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "4 ongoing cloning jobs; Avg progress = 0.0% (9s)\n [............................] ", 2024-01-15T18:13:11.202 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0, 2024-01-15T18:13:11.202 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:11.202 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:11.202 INFO:teuthology.orchestra.run.smithi027.stdout: "8fb3fd3d-b405-4726-bf6b-75de006e8a09": { 2024-01-15T18:13:11.202 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 0.0% (9s)\n [............................] ", 2024-01-15T18:13:11.202 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0, 2024-01-15T18:13:11.202 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:11.202 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:11.202 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:11.202 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:13:11.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:11.195+0000 7f2ec2200640 1 -- 172.21.15.27:0/2074379879 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2e940673d0 msgr2=0x7f2e94069890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:11.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:11.195+0000 7f2ec2200640 1 --2- 172.21.15.27:0/2074379879 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2e940673d0 0x7f2e94069890 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2eac001750 tx=0x7f2eac0011a0 comp rx=0 tx=0).stop 2024-01-15T18:13:11.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:11.195+0000 7f2ec2200640 1 -- 172.21.15.27:0/2074379879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ebc0b6570 msgr2=0x7f2ebc06c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:11.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:11.195+0000 7f2ec2200640 1 --2- 172.21.15.27:0/2074379879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ebc0b6570 0x7f2ebc06c650 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f2ea40017b0 tx=0x7f2ea4007430 comp rx=0 tx=0).stop 2024-01-15T18:13:11.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:11.195+0000 7f2ec2200640 1 -- 172.21.15.27:0/2074379879 shutdown_connections 2024-01-15T18:13:11.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:11.195+0000 7f2ec2200640 1 --2- 172.21.15.27:0/2074379879 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2e940673d0 0x7f2e94069890 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:11.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:11.195+0000 7f2ec2200640 1 --2- 172.21.15.27:0/2074379879 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ebc06cb90 0x7f2ebc0c5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:11.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:11.195+0000 7f2ec2200640 1 --2- 172.21.15.27:0/2074379879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ebc0b6570 0x7f2ebc06c650 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:11.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:11.195+0000 7f2ec2200640 1 --2- 172.21.15.27:0/2074379879 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2e9c003a40 0x7f2ebc0b44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:11.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:11.195+0000 7f2ec2200640 1 -- 172.21.15.27:0/2074379879 >> 172.21.15.27:0/2074379879 conn(0x7f2ebc0b2a10 msgr2=0x7f2ebc0b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:11.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:11.199+0000 7f2ec2200640 1 -- 172.21.15.27:0/2074379879 shutdown_connections 2024-01-15T18:13:11.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:11.199+0000 7f2ec2200640 1 -- 172.21.15.27:0/2074379879 wait complete. 2024-01-15T18:13:12.220 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-15T18:13:12.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.371+0000 7f2c01c00640 1 Processor -- start 2024-01-15T18:13:12.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.371+0000 7f2c01c00640 1 -- start start 2024-01-15T18:13:12.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.375+0000 7f2c01c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f2bfc15e7d0 0x7f2bfc0cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:12.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.375+0000 7f2c01c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2bfc0cedc0 0x7f2bfc0cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:12.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.375+0000 7f2c01c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bfc0b38b0 con 0x7f2bfc0ce2e0 2024-01-15T18:13:12.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.375+0000 7f2c01c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bfc0cd550 con 0x7f2bfc0cedc0 2024-01-15T18:13:12.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.375+0000 7f2c01c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bfc0cd270 con 0x7f2bfc15e7d0 2024-01-15T18:13:12.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.375+0000 7f2bfa000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2bfc0cedc0 0x7f2bfc0cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:12.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.375+0000 7f2bfb400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2bfc0ce2e0 0x7f2bfc0cba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60018/0 (socket says 172.21.15.27:60018) 2024-01-15T18:13:12.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.375+0000 7f2bfb400640 1 -- 172.21.15.27:0/879805547 learned_addr learned my addr 172.21.15.27:0/879805547 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:13:12.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.375+0000 7f2bfaa00640 1 --2- 172.21.15.27:0/879805547 >> v2:172.21.15.93:3300/0 conn(0x7f2bfc15e7d0 0x7f2bfc0cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:12.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.375+0000 7f2bfa000640 1 -- 172.21.15.27:0/879805547 >> v2:172.21.15.93:3300/0 conn(0x7f2bfc15e7d0 msgr2=0x7f2bfc0cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:12.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.375+0000 7f2bfa000640 1 --2- 172.21.15.27:0/879805547 >> v2:172.21.15.93:3300/0 conn(0x7f2bfc15e7d0 0x7f2bfc0cf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:12.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.375+0000 7f2bfa000640 1 -- 172.21.15.27:0/879805547 >> v1:172.21.15.27:6789/0 conn(0x7f2bfc0ce2e0 legacy=0x7f2bfc0cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:12.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.375+0000 7f2bfa000640 1 -- 172.21.15.27:0/879805547 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bfc0b35d0 con 0x7f2bfc0cedc0 2024-01-15T18:13:12.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.375+0000 7f2bfaa00640 1 --2- 172.21.15.27:0/879805547 >> v2:172.21.15.93:3300/0 conn(0x7f2bfc15e7d0 0x7f2bfc0cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:13:12.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.375+0000 7f2bf9600640 1 -- 172.21.15.27:0/879805547 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259735398 0 0) 0x7f2bfc0b38b0 con 0x7f2bfc0ce2e0 2024-01-15T18:13:12.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.379+0000 7f2bfa000640 1 --2- 172.21.15.27:0/879805547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bfc0cedc0 0x7f2bfc0cf7e0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc071b00 tx=0x7f2be401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6c86ccba59f43c6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:12.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.379+0000 7f2bf9600640 1 -- 172.21.15.27:0/879805547 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be40ab600 con 0x7f2bfc0cedc0 2024-01-15T18:13:12.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.379+0000 7f2bf9600640 1 -- 172.21.15.27:0/879805547 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be40a7050 con 0x7f2bfc0cedc0 2024-01-15T18:13:12.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.379+0000 7f2bf9600640 1 -- 172.21.15.27:0/879805547 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be40ab980 con 0x7f2bfc0cedc0 2024-01-15T18:13:12.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.379+0000 7f2c01c00640 1 -- 172.21.15.27:0/879805547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bfc0cedc0 msgr2=0x7f2bfc0cf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:12.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.379+0000 7f2c01c00640 1 --2- 172.21.15.27:0/879805547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bfc0cedc0 0x7f2bfc0cf7e0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc071b00 tx=0x7f2be401c3d0 comp rx=0 tx=0).stop 2024-01-15T18:13:12.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.379+0000 7f2c01c00640 1 -- 172.21.15.27:0/879805547 shutdown_connections 2024-01-15T18:13:12.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.379+0000 7f2c01c00640 1 --2- 172.21.15.27:0/879805547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bfc0cedc0 0x7f2bfc0cf7e0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:12.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.379+0000 7f2c01c00640 1 --2- 172.21.15.27:0/879805547 >> v2:172.21.15.93:3300/0 conn(0x7f2bfc15e7d0 0x7f2bfc0cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:12.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.379+0000 7f2c01c00640 1 -- 172.21.15.27:0/879805547 >> 172.21.15.27:0/879805547 conn(0x7f2bfc0b2a10 msgr2=0x7f2bfc0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:12.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.379+0000 7f2c01c00640 1 -- 172.21.15.27:0/879805547 shutdown_connections 2024-01-15T18:13:12.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.379+0000 7f2c01c00640 1 -- 172.21.15.27:0/879805547 wait complete. 2024-01-15T18:13:12.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.379+0000 7f2c01c00640 1 Processor -- start 2024-01-15T18:13:12.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.383+0000 7f2c01c00640 1 -- start start 2024-01-15T18:13:12.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.383+0000 7f2c01c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bfc0ce2e0 0x7f2bfc14e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:12.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.383+0000 7f2c01c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2bfc0cedc0 0x7f2bfc0c6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:12.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.383+0000 7f2c01c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2bfc15e7d0 0x7f2bfc0c4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:12.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.383+0000 7f2c01c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc0c62e0 con 0x7f2bfc0ce2e0 2024-01-15T18:13:12.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.383+0000 7f2c01c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc0c56d0 con 0x7f2bfc0cedc0 2024-01-15T18:13:12.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.383+0000 7f2c01c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc0c59b0 con 0x7f2bfc15e7d0 2024-01-15T18:13:12.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.383+0000 7f2bfaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bfc0ce2e0 0x7f2bfc14e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:12.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.383+0000 7f2bfaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bfc0ce2e0 0x7f2bfc14e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38074/0 (socket says 172.21.15.27:38074) 2024-01-15T18:13:12.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.383+0000 7f2bfaa00640 1 -- 172.21.15.27:0/3265625115 learned_addr learned my addr 172.21.15.27:0/3265625115 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:12.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.383+0000 7f2bfb400640 1 --2- 172.21.15.27:0/3265625115 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2bfc15e7d0 0x7f2bfc0c4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:12.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.383+0000 7f2bfa000640 1 --2- 172.21.15.27:0/3265625115 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2bfc0cedc0 0x7f2bfc0c6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:12.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.383+0000 7f2bfaa00640 1 -- 172.21.15.27:0/3265625115 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2bfc15e7d0 msgr2=0x7f2bfc0c4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:12.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.383+0000 7f2bfaa00640 1 --2- 172.21.15.27:0/3265625115 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2bfc15e7d0 0x7f2bfc0c4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:12.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.383+0000 7f2bfaa00640 1 -- 172.21.15.27:0/3265625115 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2bfc0cedc0 msgr2=0x7f2bfc0c6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:12.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.383+0000 7f2bfaa00640 1 --2- 172.21.15.27:0/3265625115 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2bfc0cedc0 0x7f2bfc0c6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:12.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.383+0000 7f2bfaa00640 1 -- 172.21.15.27:0/3265625115 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be4009420 con 0x7f2bfc0ce2e0 2024-01-15T18:13:12.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.383+0000 7f2bfaa00640 1 --2- 172.21.15.27:0/3265625115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bfc0ce2e0 0x7f2bfc14e640 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f2bec00e810 tx=0x7f2bec00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:12.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.383+0000 7f2bf2a00640 1 -- 172.21.15.27:0/3265625115 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bec00c8d0 con 0x7f2bfc0ce2e0 2024-01-15T18:13:12.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.383+0000 7f2c01c00640 1 -- 172.21.15.27:0/3265625115 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bfc0c5c90 con 0x7f2bfc0ce2e0 2024-01-15T18:13:12.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.383+0000 7f2bf2a00640 1 -- 172.21.15.27:0/3265625115 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bec00ca30 con 0x7f2bfc0ce2e0 2024-01-15T18:13:12.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.383+0000 7f2bf2a00640 1 -- 172.21.15.27:0/3265625115 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bec00cdc0 con 0x7f2bfc0ce2e0 2024-01-15T18:13:12.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.383+0000 7f2c01c00640 1 -- 172.21.15.27:0/3265625115 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bfc0b4ce0 con 0x7f2bfc0ce2e0 2024-01-15T18:13:12.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.383+0000 7f2c01c00640 1 -- 172.21.15.27:0/3265625115 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bfc04e850 con 0x7f2bfc0ce2e0 2024-01-15T18:13:12.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.387+0000 7f2bf2a00640 1 -- 172.21.15.27:0/3265625115 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f2bec013410 con 0x7f2bfc0ce2e0 2024-01-15T18:13:12.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.387+0000 7f2bf2a00640 1 --2- 172.21.15.27:0/3265625115 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2bd40673a0 0x7f2bd4069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:12.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.387+0000 7f2bf2a00640 1 -- 172.21.15.27:0/3265625115 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f2bec082ba0 con 0x7f2bfc0ce2e0 2024-01-15T18:13:12.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.387+0000 7f2bfa000640 1 --2- 172.21.15.27:0/3265625115 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2bd40673a0 0x7f2bd4069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:12.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.387+0000 7f2bfa000640 1 --2- 172.21.15.27:0/3265625115 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2bd40673a0 0x7f2bd4069860 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f2be40ad000 tx=0x7f2be4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:12.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.391+0000 7f2bf2a00640 1 -- 172.21.15.27:0/3265625115 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2bec051810 con 0x7f2bfc0ce2e0 2024-01-15T18:13:12.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.603+0000 7f2c01c00640 1 -- 172.21.15.27:0/3265625115 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2bfc0d02e0 con 0x7f2bfc0ce2e0 2024-01-15T18:13:12.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.603+0000 7f2bf2a00640 1 -- 172.21.15.27:0/3265625115 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f2bec021990 con 0x7f2bfc0ce2e0 2024-01-15T18:13:12.605 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:13:12.605 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:13:12.605 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:13:12.606 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:13:12.606 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:13:12.606 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:13:12.606 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:13:12.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:12.606 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:13:12.606 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:13:12.606 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:13:12.606 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:13:12.606 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:13:12.606 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:12.606 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:13:12.606 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:13:12.606 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:13:12.606 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:13:12.607 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:12.607 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 519, 2024-01-15T18:13:12.607 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:13:12.607 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:13:12.607 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:13:12.607 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:13:12.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:12.607 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:13:12.607 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 72, 2024-01-15T18:13:12.607 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:13:12.607 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:13:12.607 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:13:12.607 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:13:12.607 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:13:12.607 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:13:12.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:12.608 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:13:12.608 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:13:12.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:12.608 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:13:12.608 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:13:12.608 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:12.608 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:12.608 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:13:12.608 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:13:12.608 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 1315, 2024-01-15T18:13:12.608 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 5369958725, 2024-01-15T18:13:12.608 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 11167285248, 2024-01-15T18:13:12.608 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 761926828032, 2024-01-15T18:13:12.608 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:13:12.609 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 178816785, 2024-01-15T18:13:12.609 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 178842535, 2024-01-15T18:13:12.609 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 43, 2024-01-15T18:13:12.609 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 50 2024-01-15T18:13:12.609 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:12.609 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:13:12.609 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 48, 2024-01-15T18:13:12.609 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 5, 2024-01-15T18:13:12.609 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:13:12.609 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:13:12.609 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:13:12.609 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:13:12.609 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:12.609 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 5, 2024-01-15T18:13:12.609 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:13:12.609 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:13:12.610 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:13:12.610 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 7227 2024-01-15T18:13:12.610 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:12.610 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:12.610 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:13:12.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:12.610 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:13:12.610 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:13:12.610 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:13:12.610 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:13:12.610 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:13:12.610 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:13:12.610 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:13:12.610 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:12.610 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:13:12.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:12.611 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:13:12.611 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 16, 2024-01-15T18:13:12.611 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:13:08.281503+0000", 2024-01-15T18:13:12.611 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:13:12.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:12.611 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:13:12.611 INFO:teuthology.orchestra.run.smithi027.stdout: "6eefee50-4f20-4629-a81e-c700e83869a8": { 2024-01-15T18:13:12.611 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (12s)\n [====........................] (remaining: 64s)", 2024-01-15T18:13:12.611 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.16664999723434448, 2024-01-15T18:13:12.611 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:12.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:12.611 INFO:teuthology.orchestra.run.smithi027.stdout: "8fb3fd3d-b405-4726-bf6b-75de006e8a09": { 2024-01-15T18:13:12.611 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (12s)\n [==..........................] (remaining: 2m)", 2024-01-15T18:13:12.611 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.095228567719459534, 2024-01-15T18:13:12.611 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:12.611 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:12.612 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:12.612 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:13:12.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.607+0000 7f2c01c00640 1 -- 172.21.15.27:0/3265625115 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2bd40673a0 msgr2=0x7f2bd4069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:12.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.607+0000 7f2c01c00640 1 --2- 172.21.15.27:0/3265625115 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2bd40673a0 0x7f2bd4069860 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f2be40ad000 tx=0x7f2be4008040 comp rx=0 tx=0).stop 2024-01-15T18:13:12.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.607+0000 7f2c01c00640 1 -- 172.21.15.27:0/3265625115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bfc0ce2e0 msgr2=0x7f2bfc14e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:12.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.607+0000 7f2c01c00640 1 --2- 172.21.15.27:0/3265625115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bfc0ce2e0 0x7f2bfc14e640 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f2bec00e810 tx=0x7f2bec00ecd0 comp rx=0 tx=0).stop 2024-01-15T18:13:12.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.607+0000 7f2c01c00640 1 -- 172.21.15.27:0/3265625115 shutdown_connections 2024-01-15T18:13:12.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.607+0000 7f2c01c00640 1 --2- 172.21.15.27:0/3265625115 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2bd40673a0 0x7f2bd4069860 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:12.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.607+0000 7f2c01c00640 1 --2- 172.21.15.27:0/3265625115 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2bfc15e7d0 0x7f2bfc0c4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:12.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.607+0000 7f2c01c00640 1 --2- 172.21.15.27:0/3265625115 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2bfc0cedc0 0x7f2bfc0c6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:12.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.607+0000 7f2c01c00640 1 --2- 172.21.15.27:0/3265625115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bfc0ce2e0 0x7f2bfc14e640 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:12.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.607+0000 7f2c01c00640 1 -- 172.21.15.27:0/3265625115 >> 172.21.15.27:0/3265625115 conn(0x7f2bfc0b2a10 msgr2=0x7f2bfc15d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:12.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.607+0000 7f2c01c00640 1 -- 172.21.15.27:0/3265625115 shutdown_connections 2024-01-15T18:13:12.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:12.607+0000 7f2c01c00640 1 -- 172.21.15.27:0/3265625115 wait complete. 2024-01-15T18:13:13.628 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-15T18:13:13.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.775+0000 7f5f73a00640 1 Processor -- start 2024-01-15T18:13:13.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.775+0000 7f5f73a00640 1 -- start start 2024-01-15T18:13:13.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.775+0000 7f5f73a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5f6c0b2390 0x7f5f6c0cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:13.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.775+0000 7f5f73a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f5f6c06bb80 0x7f5f6c0cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:13.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.775+0000 7f5f73a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5f6c0cfcc0 0x7f5f6c0cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:13.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.775+0000 7f5f73a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f6c0b2770 con 0x7f5f6c0b2390 2024-01-15T18:13:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.775+0000 7f5f73a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f6c06c060 con 0x7f5f6c06bb80 2024-01-15T18:13:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.775+0000 7f5f73a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c06c340 con 0x7f5f6c0cfcc0 2024-01-15T18:13:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.775+0000 7f5f71000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5f6c0b2390 0x7f5f6c0cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.775+0000 7f5f71000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5f6c0b2390 0x7f5f6c0cf780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56784/0 (socket says 172.21.15.27:56784) 2024-01-15T18:13:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.775+0000 7f5f71000640 1 -- 172.21.15.27:0/2815231767 learned_addr learned my addr 172.21.15.27:0/2815231767 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.775+0000 7f5f71a00640 1 --2- 172.21.15.27:0/2815231767 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5f6c0cfcc0 0x7f5f6c0cddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.775+0000 7f5f6be00640 1 --2- 172.21.15.27:0/2815231767 >> v2:172.21.15.93:3300/0 conn(0x7f5f6c06bb80 0x7f5f6c0cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.775+0000 7f5f71000640 1 -- 172.21.15.27:0/2815231767 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5f6c0cfcc0 msgr2=0x7f5f6c0cddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.775+0000 7f5f71000640 1 --2- 172.21.15.27:0/2815231767 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5f6c0cfcc0 0x7f5f6c0cddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.775+0000 7f5f71000640 1 -- 172.21.15.27:0/2815231767 >> v2:172.21.15.93:3300/0 conn(0x7f5f6c06bb80 msgr2=0x7f5f6c0cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.775+0000 7f5f71000640 1 --2- 172.21.15.27:0/2815231767 >> v2:172.21.15.93:3300/0 conn(0x7f5f6c06bb80 0x7f5f6c0cd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.775+0000 7f5f71000640 1 -- 172.21.15.27:0/2815231767 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f6c06c620 con 0x7f5f6c0b2390 2024-01-15T18:13:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.775+0000 7f5f71a00640 1 --2- 172.21.15.27:0/2815231767 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5f6c0cfcc0 0x7f5f6c0cddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:13:13.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.775+0000 7f5f6be00640 1 --2- 172.21.15.27:0/2815231767 >> v2:172.21.15.93:3300/0 conn(0x7f5f6c06bb80 0x7f5f6c0cd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:13:13.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.779+0000 7f5f71000640 1 --2- 172.21.15.27:0/2815231767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f6c0b2390 0x7f5f6c0cf780 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c070ea0 tx=0x7f5f5c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2e4b6621f74994 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:13.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.779+0000 7f5f6b400640 1 -- 172.21.15.27:0/2815231767 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c0aa5f0 con 0x7f5f6c0b2390 2024-01-15T18:13:13.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.779+0000 7f5f6b400640 1 -- 172.21.15.27:0/2815231767 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f5c0aa750 con 0x7f5f6c0b2390 2024-01-15T18:13:13.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.779+0000 7f5f6b400640 1 -- 172.21.15.27:0/2815231767 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c0aaae0 con 0x7f5f6c0b2390 2024-01-15T18:13:13.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.779+0000 7f5f73a00640 1 -- 172.21.15.27:0/2815231767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f6c0b2390 msgr2=0x7f5f6c0cf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:13.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.779+0000 7f5f73a00640 1 --2- 172.21.15.27:0/2815231767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f6c0b2390 0x7f5f6c0cf780 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c070ea0 tx=0x7f5f5c01c4d0 comp rx=0 tx=0).stop 2024-01-15T18:13:13.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.779+0000 7f5f73a00640 1 -- 172.21.15.27:0/2815231767 shutdown_connections 2024-01-15T18:13:13.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.779+0000 7f5f73a00640 1 --2- 172.21.15.27:0/2815231767 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5f6c0cfcc0 0x7f5f6c0cddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:13.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.779+0000 7f5f73a00640 1 --2- 172.21.15.27:0/2815231767 >> v2:172.21.15.93:3300/0 conn(0x7f5f6c06bb80 0x7f5f6c0cd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:13.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.779+0000 7f5f73a00640 1 --2- 172.21.15.27:0/2815231767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f6c0b2390 0x7f5f6c0cf780 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:13.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.779+0000 7f5f73a00640 1 -- 172.21.15.27:0/2815231767 >> 172.21.15.27:0/2815231767 conn(0x7f5f6c0b2a10 msgr2=0x7f5f6c0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:13.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.779+0000 7f5f73a00640 1 -- 172.21.15.27:0/2815231767 shutdown_connections 2024-01-15T18:13:13.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.779+0000 7f5f73a00640 1 -- 172.21.15.27:0/2815231767 wait complete. 2024-01-15T18:13:13.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f73a00640 1 Processor -- start 2024-01-15T18:13:13.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f73a00640 1 -- start start 2024-01-15T18:13:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f73a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5f6c06bb80 0x7f5f6c0af410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f73a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f6c0b2390 0x7f5f6c0af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f73a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5f6c0cfcc0 0x7f5f6c0b5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f73a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c15eb60 con 0x7f5f6c0b2390 2024-01-15T18:13:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f73a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c15e250 con 0x7f5f6c06bb80 2024-01-15T18:13:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f73a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c0b66c0 con 0x7f5f6c0cfcc0 2024-01-15T18:13:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f6be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f6c0b2390 0x7f5f6c0af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f6be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f6c0b2390 0x7f5f6c0af950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56798/0 (socket says 172.21.15.27:56798) 2024-01-15T18:13:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f6be00640 1 -- 172.21.15.27:0/98932787 learned_addr learned my addr 172.21.15.27:0/98932787 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f71a00640 1 --2- 172.21.15.27:0/98932787 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5f6c0cfcc0 0x7f5f6c0b5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f71000640 1 --2- 172.21.15.27:0/98932787 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5f6c06bb80 0x7f5f6c0af410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f6be00640 1 -- 172.21.15.27:0/98932787 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5f6c0cfcc0 msgr2=0x7f5f6c0b5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f6be00640 1 --2- 172.21.15.27:0/98932787 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5f6c0cfcc0 0x7f5f6c0b5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:13.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f6be00640 1 -- 172.21.15.27:0/98932787 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5f6c06bb80 msgr2=0x7f5f6c0af410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:13.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f6be00640 1 --2- 172.21.15.27:0/98932787 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5f6c06bb80 0x7f5f6c0af410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:13.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f6be00640 1 -- 172.21.15.27:0/98932787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5c0094d0 con 0x7f5f6c0b2390 2024-01-15T18:13:13.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f71000640 1 --2- 172.21.15.27:0/98932787 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5f6c06bb80 0x7f5f6c0af410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:13:13.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f71a00640 1 --2- 172.21.15.27:0/98932787 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5f6c0cfcc0 0x7f5f6c0b5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:13:13.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f6be00640 1 --2- 172.21.15.27:0/98932787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f6c0b2390 0x7f5f6c0af950 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f5f5400d830 tx=0x7f5f5400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:13.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f68c00640 1 -- 172.21.15.27:0/98932787 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5400be10 con 0x7f5f6c0b2390 2024-01-15T18:13:13.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f68c00640 1 -- 172.21.15.27:0/98932787 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f54004030 con 0x7f5f6c0b2390 2024-01-15T18:13:13.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f68c00640 1 -- 172.21.15.27:0/98932787 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f540043c0 con 0x7f5f6c0b2390 2024-01-15T18:13:13.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f73a00640 1 -- 172.21.15.27:0/98932787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f6c15e4a0 con 0x7f5f6c0b2390 2024-01-15T18:13:13.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.783+0000 7f5f73a00640 1 -- 172.21.15.27:0/98932787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f6c15a750 con 0x7f5f6c0b2390 2024-01-15T18:13:13.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.787+0000 7f5f73a00640 1 -- 172.21.15.27:0/98932787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f6c0d07c0 con 0x7f5f6c0b2390 2024-01-15T18:13:13.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.787+0000 7f5f68c00640 1 -- 172.21.15.27:0/98932787 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f5f540098b0 con 0x7f5f6c0b2390 2024-01-15T18:13:13.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.787+0000 7f5f68c00640 1 --2- 172.21.15.27:0/98932787 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5f440673a0 0x7f5f44069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:13.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.787+0000 7f5f68c00640 1 -- 172.21.15.27:0/98932787 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f5f54082f00 con 0x7f5f6c0b2390 2024-01-15T18:13:13.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.787+0000 7f5f71000640 1 --2- 172.21.15.27:0/98932787 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5f440673a0 0x7f5f44069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:13.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.787+0000 7f5f71000640 1 --2- 172.21.15.27:0/98932787 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5f440673a0 0x7f5f44069860 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c0ae000 tx=0x7f5f5c0a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:13.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:13.791+0000 7f5f68c00640 1 -- 172.21.15.27:0/98932787 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5f54051bf0 con 0x7f5f6c0b2390 2024-01-15T18:13:14.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:14.003+0000 7f5f73a00640 1 -- 172.21.15.27:0/98932787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5f6c15eca0 con 0x7f5f6c0b2390 2024-01-15T18:13:14.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:14.003+0000 7f5f68c00640 1 -- 172.21.15.27:0/98932787 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f5f6c15eca0 con 0x7f5f6c0b2390 2024-01-15T18:13:14.005 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:13:14.005 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:13:14.005 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:13:14.005 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:13:14.005 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:13:14.005 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:13:14.005 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:13:14.005 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:14.005 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:13:14.005 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:13:14.005 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:13:14.005 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:13:14.005 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:13:14.005 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:14.005 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:13:14.006 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:13:14.006 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:13:14.006 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:13:14.006 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:14.006 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 521, 2024-01-15T18:13:14.006 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:13:14.006 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:13:14.006 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:13:14.006 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:13:14.006 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:14.006 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:13:14.006 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 72, 2024-01-15T18:13:14.006 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:13:14.006 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:13:14.006 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:13:14.007 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:13:14.007 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:13:14.007 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:13:14.007 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:14.007 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:13:14.007 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:13:14.007 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:14.007 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:13:14.007 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:13:14.007 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:14.007 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:14.007 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:13:14.007 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:13:14.007 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 1315, 2024-01-15T18:13:14.007 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 5369958725, 2024-01-15T18:13:14.007 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 11167285248, 2024-01-15T18:13:14.008 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 761926828032, 2024-01-15T18:13:14.008 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:13:14.008 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 178816785, 2024-01-15T18:13:14.008 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 178842535, 2024-01-15T18:13:14.008 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 43, 2024-01-15T18:13:14.008 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 50 2024-01-15T18:13:14.008 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:14.008 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:13:14.008 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 48, 2024-01-15T18:13:14.008 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 5, 2024-01-15T18:13:14.008 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:13:14.008 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:13:14.008 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:13:14.008 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:13:14.008 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:14.008 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 5, 2024-01-15T18:13:14.009 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:13:14.009 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:13:14.009 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:13:14.009 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 7227 2024-01-15T18:13:14.009 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:14.009 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:14.009 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:13:14.009 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:14.009 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:13:14.009 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:13:14.009 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:13:14.009 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:13:14.009 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:13:14.009 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:13:14.009 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:13:14.010 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:14.010 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:13:14.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:14.010 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:13:14.010 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 16, 2024-01-15T18:13:14.010 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:13:08.281503+0000", 2024-01-15T18:13:14.010 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:13:14.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:14.010 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:13:14.010 INFO:teuthology.orchestra.run.smithi027.stdout: "6eefee50-4f20-4629-a81e-c700e83869a8": { 2024-01-15T18:13:14.010 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (12s)\n [====........................] (remaining: 64s)", 2024-01-15T18:13:14.010 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.16664999723434448, 2024-01-15T18:13:14.010 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:14.010 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:14.010 INFO:teuthology.orchestra.run.smithi027.stdout: "8fb3fd3d-b405-4726-bf6b-75de006e8a09": { 2024-01-15T18:13:14.011 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (12s)\n [==..........................] (remaining: 2m)", 2024-01-15T18:13:14.011 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.095228567719459534, 2024-01-15T18:13:14.011 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:14.011 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:14.011 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:14.011 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:13:14.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:14.007+0000 7f5f73a00640 1 -- 172.21.15.27:0/98932787 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5f440673a0 msgr2=0x7f5f44069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:14.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:14.007+0000 7f5f73a00640 1 --2- 172.21.15.27:0/98932787 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5f440673a0 0x7f5f44069860 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c0ae000 tx=0x7f5f5c0a7000 comp rx=0 tx=0).stop 2024-01-15T18:13:14.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:14.007+0000 7f5f73a00640 1 -- 172.21.15.27:0/98932787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f6c0b2390 msgr2=0x7f5f6c0af950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:14.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:14.007+0000 7f5f73a00640 1 --2- 172.21.15.27:0/98932787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f6c0b2390 0x7f5f6c0af950 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f5f5400d830 tx=0x7f5f5400dcf0 comp rx=0 tx=0).stop 2024-01-15T18:13:14.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:14.007+0000 7f5f73a00640 1 -- 172.21.15.27:0/98932787 shutdown_connections 2024-01-15T18:13:14.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:14.007+0000 7f5f73a00640 1 --2- 172.21.15.27:0/98932787 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5f440673a0 0x7f5f44069860 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:14.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:14.007+0000 7f5f73a00640 1 --2- 172.21.15.27:0/98932787 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5f6c0cfcc0 0x7f5f6c0b5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:14.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:14.007+0000 7f5f73a00640 1 --2- 172.21.15.27:0/98932787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f6c0b2390 0x7f5f6c0af950 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:14.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:14.007+0000 7f5f73a00640 1 --2- 172.21.15.27:0/98932787 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5f6c06bb80 0x7f5f6c0af410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:14.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:14.007+0000 7f5f73a00640 1 -- 172.21.15.27:0/98932787 >> 172.21.15.27:0/98932787 conn(0x7f5f6c0b2a10 msgr2=0x7f5f6c0b2df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:14.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:14.007+0000 7f5f73a00640 1 -- 172.21.15.27:0/98932787 shutdown_connections 2024-01-15T18:13:14.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:14.007+0000 7f5f73a00640 1 -- 172.21.15.27:0/98932787 wait complete. 2024-01-15T18:13:15.028 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-15T18:13:15.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.179+0000 7fefe6000640 1 Processor -- start 2024-01-15T18:13:15.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.179+0000 7fefe6000640 1 -- start start 2024-01-15T18:13:15.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.179+0000 7fefe6000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fefe00cf080 0x7fefe00cf460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:15.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.179+0000 7fefe6000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fefe00b0130 0x7fefe00b0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:15.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.179+0000 7fefe6000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fefe00b24a0 0x7fefe00b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:15.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.179+0000 7fefe6000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefe00d1ec0 con 0x7fefe00cf080 2024-01-15T18:13:15.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.179+0000 7fefe6000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefe00ae160 con 0x7fefe00b24a0 2024-01-15T18:13:15.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.179+0000 7fefe6000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe00d1be0 con 0x7fefe00b0130 2024-01-15T18:13:15.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.179+0000 7fefdf400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fefe00cf080 0x7fefe00cf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:15.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.179+0000 7fefdf400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fefe00cf080 0x7fefe00cf460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56810/0 (socket says 172.21.15.27:56810) 2024-01-15T18:13:15.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.179+0000 7fefdf400640 1 -- 172.21.15.27:0/1093685351 learned_addr learned my addr 172.21.15.27:0/1093685351 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:15.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.179+0000 7fefdfe00640 1 --2- 172.21.15.27:0/1093685351 >> v2:172.21.15.93:3300/0 conn(0x7fefe00b24a0 0x7fefe00b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:15.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.179+0000 7fefdf400640 1 -- 172.21.15.27:0/1093685351 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fefe00b0130 msgr2=0x7fefe00b0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:15.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.179+0000 7fefdea00640 1 --2- 172.21.15.27:0/1093685351 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fefe00b0130 0x7fefe00b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:15.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.179+0000 7fefdf400640 1 --2- 172.21.15.27:0/1093685351 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fefe00b0130 0x7fefe00b0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:15.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.179+0000 7fefdf400640 1 -- 172.21.15.27:0/1093685351 >> v2:172.21.15.93:3300/0 conn(0x7fefe00b24a0 msgr2=0x7fefe00b0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:15.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.179+0000 7fefdf400640 1 --2- 172.21.15.27:0/1093685351 >> v2:172.21.15.93:3300/0 conn(0x7fefe00b24a0 0x7fefe00b0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:15.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.179+0000 7fefdf400640 1 -- 172.21.15.27:0/1093685351 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe00addf0 con 0x7fefe00cf080 2024-01-15T18:13:15.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.179+0000 7fefdea00640 1 --2- 172.21.15.27:0/1093685351 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fefe00b0130 0x7fefe00b0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:13:15.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.183+0000 7fefdf400640 1 --2- 172.21.15.27:0/1093685351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe00cf080 0x7fefe00cf460 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fefe00b07c0 tx=0x7fefd001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30c2c05d40bc28d8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:15.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.183+0000 7fefde000640 1 -- 172.21.15.27:0/1093685351 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd00ac630 con 0x7fefe00cf080 2024-01-15T18:13:15.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.183+0000 7fefde000640 1 -- 172.21.15.27:0/1093685351 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd00ac790 con 0x7fefe00cf080 2024-01-15T18:13:15.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.183+0000 7fefde000640 1 -- 172.21.15.27:0/1093685351 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd00acb20 con 0x7fefe00cf080 2024-01-15T18:13:15.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.183+0000 7fefe6000640 1 -- 172.21.15.27:0/1093685351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe00cf080 msgr2=0x7fefe00cf460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:15.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.183+0000 7fefe6000640 1 --2- 172.21.15.27:0/1093685351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe00cf080 0x7fefe00cf460 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fefe00b07c0 tx=0x7fefd001c500 comp rx=0 tx=0).stop 2024-01-15T18:13:15.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.183+0000 7fefe6000640 1 -- 172.21.15.27:0/1093685351 shutdown_connections 2024-01-15T18:13:15.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.183+0000 7fefe6000640 1 --2- 172.21.15.27:0/1093685351 >> v2:172.21.15.93:3300/0 conn(0x7fefe00b24a0 0x7fefe00b0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:15.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.183+0000 7fefe6000640 1 --2- 172.21.15.27:0/1093685351 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fefe00b0130 0x7fefe00b0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:15.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.183+0000 7fefe6000640 1 --2- 172.21.15.27:0/1093685351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe00cf080 0x7fefe00cf460 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:15.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.183+0000 7fefe6000640 1 -- 172.21.15.27:0/1093685351 >> 172.21.15.27:0/1093685351 conn(0x7fefe00d0f20 msgr2=0x7fefe00d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:15.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.183+0000 7fefe6000640 1 -- 172.21.15.27:0/1093685351 shutdown_connections 2024-01-15T18:13:15.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.183+0000 7fefe6000640 1 -- 172.21.15.27:0/1093685351 wait complete. 2024-01-15T18:13:15.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.187+0000 7fefe6000640 1 Processor -- start 2024-01-15T18:13:15.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.187+0000 7fefe6000640 1 -- start start 2024-01-15T18:13:15.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.187+0000 7fefe6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe00b0130 0x7fefe0158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:15.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.187+0000 7fefe6000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fefe00b24a0 0x7fefe0145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:15.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.187+0000 7fefe6000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fefe00cf080 0x7fefe0145640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:15.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.187+0000 7fefe6000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe01f5f30 con 0x7fefe00b0130 2024-01-15T18:13:15.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.187+0000 7fefe6000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe015dde0 con 0x7fefe00b24a0 2024-01-15T18:13:15.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.187+0000 7fefe6000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe015e0c0 con 0x7fefe00cf080 2024-01-15T18:13:15.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.187+0000 7fefdf400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe00b0130 0x7fefe0158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:15.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.187+0000 7fefdf400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe00b0130 0x7fefe0158320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56812/0 (socket says 172.21.15.27:56812) 2024-01-15T18:13:15.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.187+0000 7fefdf400640 1 -- 172.21.15.27:0/2672149112 learned_addr learned my addr 172.21.15.27:0/2672149112 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:15.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.187+0000 7fefdfe00640 1 --2- 172.21.15.27:0/2672149112 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fefe00cf080 0x7fefe0145640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:15.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.187+0000 7fefdea00640 1 --2- 172.21.15.27:0/2672149112 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fefe00b24a0 0x7fefe0145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:15.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.187+0000 7fefdf400640 1 -- 172.21.15.27:0/2672149112 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fefe00cf080 msgr2=0x7fefe0145640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:15.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.187+0000 7fefdf400640 1 --2- 172.21.15.27:0/2672149112 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fefe00cf080 0x7fefe0145640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:15.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.187+0000 7fefdf400640 1 -- 172.21.15.27:0/2672149112 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fefe00b24a0 msgr2=0x7fefe0145100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:15.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.187+0000 7fefdf400640 1 --2- 172.21.15.27:0/2672149112 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fefe00b24a0 0x7fefe0145100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:15.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.187+0000 7fefdf400640 1 -- 172.21.15.27:0/2672149112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd00094d0 con 0x7fefe00b0130 2024-01-15T18:13:15.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.187+0000 7fefdf400640 1 --2- 172.21.15.27:0/2672149112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe00b0130 0x7fefe0158320 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fefd00af040 tx=0x7fefd00192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:15.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.191+0000 7fefd7400640 1 -- 172.21.15.27:0/2672149112 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd00b5830 con 0x7fefe00b0130 2024-01-15T18:13:15.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.191+0000 7fefd7400640 1 -- 172.21.15.27:0/2672149112 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd00b59b0 con 0x7fefe00b0130 2024-01-15T18:13:15.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.191+0000 7fefe6000640 1 -- 172.21.15.27:0/2672149112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe0157840 con 0x7fefe00b0130 2024-01-15T18:13:15.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.191+0000 7fefd7400640 1 -- 172.21.15.27:0/2672149112 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd00b5d60 con 0x7fefe00b0130 2024-01-15T18:13:15.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.191+0000 7fefe6000640 1 -- 172.21.15.27:0/2672149112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe0157e00 con 0x7fefe00b0130 2024-01-15T18:13:15.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.191+0000 7fefe6000640 1 -- 172.21.15.27:0/2672149112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefe00c48c0 con 0x7fefe00b0130 2024-01-15T18:13:15.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.191+0000 7fefd7400640 1 -- 172.21.15.27:0/2672149112 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7fefd00b3070 con 0x7fefe00b0130 2024-01-15T18:13:15.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.191+0000 7fefd7400640 1 --2- 172.21.15.27:0/2672149112 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fefb80673a0 0x7fefb8069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:15.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.191+0000 7fefd7400640 1 -- 172.21.15.27:0/2672149112 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fefd011d660 con 0x7fefe00b0130 2024-01-15T18:13:15.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.191+0000 7fefdea00640 1 --2- 172.21.15.27:0/2672149112 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fefb80673a0 0x7fefb8069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:15.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.195+0000 7fefdea00640 1 --2- 172.21.15.27:0/2672149112 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fefb80673a0 0x7fefb8069860 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fefe0147c60 tx=0x7fefc8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:15.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.195+0000 7fefd7400640 1 -- 172.21.15.27:0/2672149112 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fefd0124070 con 0x7fefe00b0130 2024-01-15T18:13:15.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.411+0000 7fefe6000640 1 -- 172.21.15.27:0/2672149112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fefe015e200 con 0x7fefe00b0130 2024-01-15T18:13:15.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.411+0000 7fefd7400640 1 -- 172.21.15.27:0/2672149112 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7fefd00a8990 con 0x7fefe00b0130 2024-01-15T18:13:15.413 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:13:15.413 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:13:15.413 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:13:15.413 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:13:15.413 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:13:15.413 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:13:15.413 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:13:15.413 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:15.413 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:13:15.414 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:13:15.414 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:13:15.414 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:13:15.414 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:13:15.414 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:15.414 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:13:15.414 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:13:15.414 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:13:15.414 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:13:15.414 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:15.414 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 522, 2024-01-15T18:13:15.414 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:13:15.415 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:13:15.415 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:13:15.415 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:13:15.415 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:15.415 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:13:15.415 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 72, 2024-01-15T18:13:15.415 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:13:15.415 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:13:15.415 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:13:15.415 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:13:15.415 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:13:15.415 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:13:15.415 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:15.415 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:13:15.416 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:13:15.416 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:15.416 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:13:15.416 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:13:15.416 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:15.416 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:15.416 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:13:15.416 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:13:15.416 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 1315, 2024-01-15T18:13:15.416 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 5369958725, 2024-01-15T18:13:15.416 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 11167285248, 2024-01-15T18:13:15.416 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 761926828032, 2024-01-15T18:13:15.416 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:13:15.417 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 178812190, 2024-01-15T18:13:15.417 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 178823872, 2024-01-15T18:13:15.417 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 43, 2024-01-15T18:13:15.417 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 47 2024-01-15T18:13:15.417 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:15.417 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:13:15.417 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 48, 2024-01-15T18:13:15.417 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 5, 2024-01-15T18:13:15.417 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:13:15.417 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:13:15.417 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:13:15.417 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:13:15.417 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:15.417 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 5, 2024-01-15T18:13:15.418 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:13:15.418 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:13:15.418 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:13:15.418 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 7227 2024-01-15T18:13:15.418 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:15.418 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:15.418 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:13:15.418 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:15.418 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:13:15.418 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:13:15.418 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:13:15.418 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:13:15.418 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:13:15.419 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:13:15.419 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:13:15.419 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:15.419 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:13:15.419 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:15.419 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:13:15.419 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 16, 2024-01-15T18:13:15.419 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:13:08.281503+0000", 2024-01-15T18:13:15.419 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:13:15.419 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:15.419 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:13:15.419 INFO:teuthology.orchestra.run.smithi027.stdout: "6eefee50-4f20-4629-a81e-c700e83869a8": { 2024-01-15T18:13:15.419 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (12s)\n [====........................] (remaining: 64s)", 2024-01-15T18:13:15.419 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.16664999723434448, 2024-01-15T18:13:15.419 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:15.420 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:15.420 INFO:teuthology.orchestra.run.smithi027.stdout: "8fb3fd3d-b405-4726-bf6b-75de006e8a09": { 2024-01-15T18:13:15.420 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (12s)\n [==..........................] (remaining: 2m)", 2024-01-15T18:13:15.420 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.095228567719459534, 2024-01-15T18:13:15.420 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:15.420 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:15.420 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:15.420 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:13:15.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.415+0000 7fefe6000640 1 -- 172.21.15.27:0/2672149112 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fefb80673a0 msgr2=0x7fefb8069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:15.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.415+0000 7fefe6000640 1 --2- 172.21.15.27:0/2672149112 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fefb80673a0 0x7fefb8069860 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fefe0147c60 tx=0x7fefc8008040 comp rx=0 tx=0).stop 2024-01-15T18:13:15.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.415+0000 7fefe6000640 1 -- 172.21.15.27:0/2672149112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe00b0130 msgr2=0x7fefe0158320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:15.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.415+0000 7fefe6000640 1 --2- 172.21.15.27:0/2672149112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe00b0130 0x7fefe0158320 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fefd00af040 tx=0x7fefd00192a0 comp rx=0 tx=0).stop 2024-01-15T18:13:15.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.415+0000 7fefe6000640 1 -- 172.21.15.27:0/2672149112 shutdown_connections 2024-01-15T18:13:15.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.415+0000 7fefe6000640 1 --2- 172.21.15.27:0/2672149112 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fefb80673a0 0x7fefb8069860 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:15.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.415+0000 7fefe6000640 1 --2- 172.21.15.27:0/2672149112 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fefe00cf080 0x7fefe0145640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:15.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.415+0000 7fefe6000640 1 --2- 172.21.15.27:0/2672149112 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fefe00b24a0 0x7fefe0145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:15.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.415+0000 7fefe6000640 1 --2- 172.21.15.27:0/2672149112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe00b0130 0x7fefe0158320 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:15.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.415+0000 7fefe6000640 1 -- 172.21.15.27:0/2672149112 >> 172.21.15.27:0/2672149112 conn(0x7fefe00d0f20 msgr2=0x7fefe006b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:15.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.415+0000 7fefe6000640 1 -- 172.21.15.27:0/2672149112 shutdown_connections 2024-01-15T18:13:15.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:15.415+0000 7fefe6000640 1 -- 172.21.15.27:0/2672149112 wait complete. 2024-01-15T18:13:16.440 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-15T18:13:16.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.683+0000 7f1ffa600640 1 Processor -- start 2024-01-15T18:13:16.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.683+0000 7f1ffa600640 1 -- start start 2024-01-15T18:13:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.687+0000 7f1ffa600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f1ff4160b20 0x7f1ff40cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.687+0000 7f1ffa600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1ff40b1e90 0x7f1ff40cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.687+0000 7f1ffa600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ff40b2ad0 0x7f1ff40b2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.687+0000 7f1ffa600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ff4160f00 con 0x7f1ff40b1e90 2024-01-15T18:13:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.687+0000 7f1ffa600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ff40cf330 con 0x7f1ff4160b20 2024-01-15T18:13:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.687+0000 7f1ffa600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff40d15b0 con 0x7f1ff40b2ad0 2024-01-15T18:13:16.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.687+0000 7f1ff2a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1ff40b1e90 0x7f1ff40cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:16.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.687+0000 7f1ff2a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1ff40b1e90 0x7f1ff40cffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56828/0 (socket says 172.21.15.27:56828) 2024-01-15T18:13:16.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.687+0000 7f1ff2a00640 1 -- 172.21.15.27:0/4190759094 learned_addr learned my addr 172.21.15.27:0/4190759094 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:16.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.687+0000 7f1ff2a00640 1 -- 172.21.15.27:0/4190759094 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ff40b2ad0 msgr2=0x7f1ff40b2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:16.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.687+0000 7f1ff2a00640 1 --2- 172.21.15.27:0/4190759094 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ff40b2ad0 0x7f1ff40b2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:16.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.687+0000 7f1ff2a00640 1 -- 172.21.15.27:0/4190759094 >> v2:172.21.15.93:3300/0 conn(0x7f1ff4160b20 msgr2=0x7f1ff40cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:16.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.687+0000 7f1ff2a00640 1 --2- 172.21.15.27:0/4190759094 >> v2:172.21.15.93:3300/0 conn(0x7f1ff4160b20 0x7f1ff40cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:16.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.687+0000 7f1ff2a00640 1 -- 172.21.15.27:0/4190759094 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff40d1890 con 0x7f1ff40b1e90 2024-01-15T18:13:16.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.691+0000 7f1ff2a00640 1 --2- 172.21.15.27:0/4190759094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff40b1e90 0x7f1ff40cffd0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f1ff408d420 tx=0x7f1fdc01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=232aabe00049f61f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:16.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.691+0000 7f1ff2000640 1 -- 172.21.15.27:0/4190759094 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc0ac410 con 0x7f1ff40b1e90 2024-01-15T18:13:16.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.691+0000 7f1ff2000640 1 -- 172.21.15.27:0/4190759094 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fdc0ac570 con 0x7f1ff40b1e90 2024-01-15T18:13:16.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.691+0000 7f1ff2000640 1 -- 172.21.15.27:0/4190759094 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc0ac900 con 0x7f1ff40b1e90 2024-01-15T18:13:16.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.691+0000 7f1ffa600640 1 -- 172.21.15.27:0/4190759094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff40b1e90 msgr2=0x7f1ff40cffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:16.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.691+0000 7f1ffa600640 1 --2- 172.21.15.27:0/4190759094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff40b1e90 0x7f1ff40cffd0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f1ff408d420 tx=0x7f1fdc01c500 comp rx=0 tx=0).stop 2024-01-15T18:13:16.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.695+0000 7f1ffa600640 1 -- 172.21.15.27:0/4190759094 shutdown_connections 2024-01-15T18:13:16.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.695+0000 7f1ffa600640 1 --2- 172.21.15.27:0/4190759094 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ff40b2ad0 0x7f1ff40b2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:16.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.695+0000 7f1ffa600640 1 --2- 172.21.15.27:0/4190759094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff40b1e90 0x7f1ff40cffd0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:16.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.695+0000 7f1ffa600640 1 --2- 172.21.15.27:0/4190759094 >> v2:172.21.15.93:3300/0 conn(0x7f1ff4160b20 0x7f1ff40cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:16.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.695+0000 7f1ffa600640 1 -- 172.21.15.27:0/4190759094 >> 172.21.15.27:0/4190759094 conn(0x7f1ff409c410 msgr2=0x7f1ff40d30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:16.695 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.695+0000 7f1ffa600640 1 -- 172.21.15.27:0/4190759094 shutdown_connections 2024-01-15T18:13:16.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.695+0000 7f1ffa600640 1 -- 172.21.15.27:0/4190759094 wait complete. 2024-01-15T18:13:16.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.695+0000 7f1ffa600640 1 Processor -- start 2024-01-15T18:13:16.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.695+0000 7f1ffa600640 1 -- start start 2024-01-15T18:13:16.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.699+0000 7f1ffa600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff40b1e90 0x7f1ff40c6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:16.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.699+0000 7f1ffa600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ff40b2ad0 0x7f1ff40b8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:16.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.699+0000 7f1ffa600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ff4160b20 0x7f1ff415d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:16.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.699+0000 7f1ffa600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff415a740 con 0x7f1ff40b1e90 2024-01-15T18:13:16.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.699+0000 7f1ffa600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff4153680 con 0x7f1ff40b2ad0 2024-01-15T18:13:16.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.699+0000 7f1ffa600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff4157790 con 0x7f1ff4160b20 2024-01-15T18:13:16.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.699+0000 7f1ff3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff40b1e90 0x7f1ff40c6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:16.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.699+0000 7f1ff3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff40b1e90 0x7f1ff40c6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56832/0 (socket says 172.21.15.27:56832) 2024-01-15T18:13:16.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.699+0000 7f1ff3400640 1 -- 172.21.15.27:0/851201686 learned_addr learned my addr 172.21.15.27:0/851201686 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:16.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.699+0000 7f1ff3e00640 1 --2- 172.21.15.27:0/851201686 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ff4160b20 0x7f1ff415d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:16.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.699+0000 7f1ff2a00640 1 --2- 172.21.15.27:0/851201686 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ff40b2ad0 0x7f1ff40b8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:16.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.699+0000 7f1ff3400640 1 -- 172.21.15.27:0/851201686 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ff4160b20 msgr2=0x7f1ff415d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:16.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.699+0000 7f1ff3400640 1 --2- 172.21.15.27:0/851201686 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ff4160b20 0x7f1ff415d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:16.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.699+0000 7f1ff3400640 1 -- 172.21.15.27:0/851201686 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ff40b2ad0 msgr2=0x7f1ff40b8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:16.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.699+0000 7f1ff3400640 1 --2- 172.21.15.27:0/851201686 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ff40b2ad0 0x7f1ff40b8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:16.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.699+0000 7f1ff3400640 1 -- 172.21.15.27:0/851201686 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fdc0094d0 con 0x7f1ff40b1e90 2024-01-15T18:13:16.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.699+0000 7f1ff3400640 1 --2- 172.21.15.27:0/851201686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff40b1e90 0x7f1ff40c6c10 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00ba50 tx=0x7f1fec00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:16.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.699+0000 7f1feb400640 1 -- 172.21.15.27:0/851201686 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec00cbe0 con 0x7f1ff40b1e90 2024-01-15T18:13:16.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.699+0000 7f1feb400640 1 -- 172.21.15.27:0/851201686 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fec00cd60 con 0x7f1ff40b1e90 2024-01-15T18:13:16.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.699+0000 7f1ffa600640 1 -- 172.21.15.27:0/851201686 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff4157ad0 con 0x7f1ff40b1e90 2024-01-15T18:13:16.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.699+0000 7f1feb400640 1 -- 172.21.15.27:0/851201686 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec014680 con 0x7f1ff40b1e90 2024-01-15T18:13:16.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.699+0000 7f1ffa600640 1 -- 172.21.15.27:0/851201686 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff40c7150 con 0x7f1ff40b1e90 2024-01-15T18:13:16.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.703+0000 7f1ffa600640 1 -- 172.21.15.27:0/851201686 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff404e850 con 0x7f1ff40b1e90 2024-01-15T18:13:16.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.703+0000 7f1feb400640 1 -- 172.21.15.27:0/851201686 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f1fec01d3b0 con 0x7f1ff40b1e90 2024-01-15T18:13:16.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.703+0000 7f1feb400640 1 --2- 172.21.15.27:0/851201686 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1fcc0673a0 0x7f1fcc069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:16.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.703+0000 7f1feb400640 1 -- 172.21.15.27:0/851201686 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f1fec082c50 con 0x7f1ff40b1e90 2024-01-15T18:13:16.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.711+0000 7f1feb400640 1 -- 172.21.15.27:0/851201686 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1fec0519f0 con 0x7f1ff40b1e90 2024-01-15T18:13:16.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.711+0000 7f1ff2a00640 1 --2- 172.21.15.27:0/851201686 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1fcc0673a0 0x7f1fcc069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:16.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:16.715+0000 7f1ff2a00640 1 --2- 172.21.15.27:0/851201686 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1fcc0673a0 0x7f1fcc069860 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0ae000 tx=0x7f1fdc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:17.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:17.087+0000 7f1ffa600640 1 -- 172.21.15.27:0/851201686 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1ff40c6810 con 0x7f1ff40b1e90 2024-01-15T18:13:17.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:17.091+0000 7f1feb400640 1 -- 172.21.15.27:0/851201686 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f1fec019090 con 0x7f1ff40b1e90 2024-01-15T18:13:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:13:17.093 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:13:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:13:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:13:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:13:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:13:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:13:17.093 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:13:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:13:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:13:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:13:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:13:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:13:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:13:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:13:17.094 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:13:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 524, 2024-01-15T18:13:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:13:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:13:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:13:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:13:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:13:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 72, 2024-01-15T18:13:17.095 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:13:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:13:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:13:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:13:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:13:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:13:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:13:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:13:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:17.096 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:13:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:13:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:13:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:13:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 1315, 2024-01-15T18:13:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 5369983076, 2024-01-15T18:13:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 11167285248, 2024-01-15T18:13:17.097 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 761926828032, 2024-01-15T18:13:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:13:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 179166403, 2024-01-15T18:13:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 178831050, 2024-01-15T18:13:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 43, 2024-01-15T18:13:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 47 2024-01-15T18:13:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:13:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 48, 2024-01-15T18:13:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 5, 2024-01-15T18:13:17.098 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:13:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:13:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:13:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:13:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 5, 2024-01-15T18:13:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:13:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:13:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:13:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 7227 2024-01-15T18:13:17.099 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:13:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:13:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:13:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:13:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:13:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:13:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:13:17.100 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:13:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:13:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:13:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 16, 2024-01-15T18:13:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:13:08.281503+0000", 2024-01-15T18:13:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:13:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:13:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: "6eefee50-4f20-4629-a81e-c700e83869a8": { 2024-01-15T18:13:17.101 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (12s)\n [====........................] (remaining: 64s)", 2024-01-15T18:13:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.16664999723434448, 2024-01-15T18:13:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: "8fb3fd3d-b405-4726-bf6b-75de006e8a09": { 2024-01-15T18:13:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (12s)\n [==..........................] (remaining: 2m)", 2024-01-15T18:13:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.095228567719459534, 2024-01-15T18:13:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:17.102 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:17.102 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:13:17.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:17.091+0000 7f1ffa600640 1 -- 172.21.15.27:0/851201686 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1fcc0673a0 msgr2=0x7f1fcc069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:17.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:17.091+0000 7f1ffa600640 1 --2- 172.21.15.27:0/851201686 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1fcc0673a0 0x7f1fcc069860 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0ae000 tx=0x7f1fdc008040 comp rx=0 tx=0).stop 2024-01-15T18:13:17.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:17.095+0000 7f1ffa600640 1 -- 172.21.15.27:0/851201686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff40b1e90 msgr2=0x7f1ff40c6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:17.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:17.095+0000 7f1ffa600640 1 --2- 172.21.15.27:0/851201686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff40b1e90 0x7f1ff40c6c10 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00ba50 tx=0x7f1fec00bf10 comp rx=0 tx=0).stop 2024-01-15T18:13:17.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:17.095+0000 7f1ffa600640 1 -- 172.21.15.27:0/851201686 shutdown_connections 2024-01-15T18:13:17.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:17.095+0000 7f1ffa600640 1 --2- 172.21.15.27:0/851201686 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1fcc0673a0 0x7f1fcc069860 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:17.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:17.095+0000 7f1ffa600640 1 --2- 172.21.15.27:0/851201686 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ff4160b20 0x7f1ff415d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:17.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:17.095+0000 7f1ffa600640 1 --2- 172.21.15.27:0/851201686 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ff40b2ad0 0x7f1ff40b8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:17.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:17.095+0000 7f1ffa600640 1 --2- 172.21.15.27:0/851201686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff40b1e90 0x7f1ff40c6c10 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:17.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:17.095+0000 7f1ffa600640 1 -- 172.21.15.27:0/851201686 >> 172.21.15.27:0/851201686 conn(0x7f1ff409c410 msgr2=0x7f1ff406b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:17.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:17.095+0000 7f1ffa600640 1 -- 172.21.15.27:0/851201686 shutdown_connections 2024-01-15T18:13:17.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:17.095+0000 7f1ffa600640 1 -- 172.21.15.27:0/851201686 wait complete. 2024-01-15T18:13:18.138 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-15T18:13:18.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.279+0000 7fbf44400640 1 Processor -- start 2024-01-15T18:13:18.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.279+0000 7fbf44400640 1 -- start start 2024-01-15T18:13:18.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.279+0000 7fbf44400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbf3c160af0 0x7fbf3c06b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:18.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.279+0000 7fbf44400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf3c0cfaa0 0x7fbf3c06bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:18.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.279+0000 7fbf44400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf3c0ae280 con 0x7fbf3c06c080 2024-01-15T18:13:18.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.279+0000 7fbf44400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf3c0adc50 con 0x7fbf3c160af0 2024-01-15T18:13:18.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.279+0000 7fbf44400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c0cff80 con 0x7fbf3c0cfaa0 2024-01-15T18:13:18.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.283+0000 7fbf41a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbf3c160af0 0x7fbf3c06b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:18.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.283+0000 7fbf41a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbf3c160af0 0x7fbf3c06b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56844/0 (socket says 172.21.15.27:56844) 2024-01-15T18:13:18.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.283+0000 7fbf41a00640 1 -- 172.21.15.27:0/2303661937 learned_addr learned my addr 172.21.15.27:0/2303661937 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:18.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.283+0000 7fbf41000640 1 --2- 172.21.15.27:0/2303661937 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf3c0cfaa0 0x7fbf3c06bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:18.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.283+0000 7fbf41a00640 1 -- 172.21.15.27:0/2303661937 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf3c0cfaa0 msgr2=0x7fbf3c06bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:18.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.283+0000 7fbf41a00640 1 --2- 172.21.15.27:0/2303661937 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf3c0cfaa0 0x7fbf3c06bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:18.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.283+0000 7fbf41a00640 1 -- 172.21.15.27:0/2303661937 >> v1:172.21.15.93:6789/0 conn(0x7fbf3c06c080 legacy=0x7fbf3c161210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:18.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.283+0000 7fbf41a00640 1 -- 172.21.15.27:0/2303661937 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3c0d0260 con 0x7fbf3c160af0 2024-01-15T18:13:18.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.283+0000 7fbf41000640 1 --2- 172.21.15.27:0/2303661937 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf3c0cfaa0 0x7fbf3c06bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:13:18.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.283+0000 7fbf41a00640 1 --2- 172.21.15.27:0/2303661937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf3c160af0 0x7fbf3c06b600 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c009870 tx=0x7fbf2c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43583d8b9f017880 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:18.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.287+0000 7fbf3be00640 1 -- 172.21.15.27:0/2303661937 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c0ac640 con 0x7fbf3c160af0 2024-01-15T18:13:18.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.287+0000 7fbf3be00640 1 -- 172.21.15.27:0/2303661937 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2c0ac7a0 con 0x7fbf3c160af0 2024-01-15T18:13:18.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.287+0000 7fbf3be00640 1 -- 172.21.15.27:0/2303661937 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c0acb30 con 0x7fbf3c160af0 2024-01-15T18:13:18.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.287+0000 7fbf44400640 1 -- 172.21.15.27:0/2303661937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf3c160af0 msgr2=0x7fbf3c06b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:18.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.287+0000 7fbf44400640 1 --2- 172.21.15.27:0/2303661937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf3c160af0 0x7fbf3c06b600 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c009870 tx=0x7fbf2c01c500 comp rx=0 tx=0).stop 2024-01-15T18:13:18.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.287+0000 7fbf44400640 1 -- 172.21.15.27:0/2303661937 shutdown_connections 2024-01-15T18:13:18.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.287+0000 7fbf44400640 1 --2- 172.21.15.27:0/2303661937 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf3c0cfaa0 0x7fbf3c06bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:18.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.287+0000 7fbf44400640 1 --2- 172.21.15.27:0/2303661937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf3c160af0 0x7fbf3c06b600 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:18.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.287+0000 7fbf44400640 1 -- 172.21.15.27:0/2303661937 >> 172.21.15.27:0/2303661937 conn(0x7fbf3c0d0f20 msgr2=0x7fbf3c0d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:18.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.287+0000 7fbf44400640 1 -- 172.21.15.27:0/2303661937 shutdown_connections 2024-01-15T18:13:18.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.287+0000 7fbf44400640 1 -- 172.21.15.27:0/2303661937 wait complete. 2024-01-15T18:13:18.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.287+0000 7fbf44400640 1 Processor -- start 2024-01-15T18:13:18.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.287+0000 7fbf44400640 1 -- start start 2024-01-15T18:13:18.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.287+0000 7fbf44400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf3c06c080 0x7fbf3c0c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:18.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.287+0000 7fbf44400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf3c0cfaa0 0x7fbf3c06eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:18.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.287+0000 7fbf44400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf3c160af0 0x7fbf3c14d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:18.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.287+0000 7fbf44400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c14ef20 con 0x7fbf3c0cfaa0 2024-01-15T18:13:18.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.287+0000 7fbf44400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c0c7800 con 0x7fbf3c160af0 2024-01-15T18:13:18.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.287+0000 7fbf44400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3c14e310 con 0x7fbf3c06c080 2024-01-15T18:13:18.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.291+0000 7fbf41000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf3c0cfaa0 0x7fbf3c06eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:18.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.291+0000 7fbf41000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf3c0cfaa0 0x7fbf3c06eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56860/0 (socket says 172.21.15.27:56860) 2024-01-15T18:13:18.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.291+0000 7fbf41000640 1 -- 172.21.15.27:0/1720648732 learned_addr learned my addr 172.21.15.27:0/1720648732 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:18.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.291+0000 7fbf41a00640 1 --2- 172.21.15.27:0/1720648732 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf3c06c080 0x7fbf3c0c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:18.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.291+0000 7fbf42400640 1 --2- 172.21.15.27:0/1720648732 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf3c160af0 0x7fbf3c14d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:18.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.291+0000 7fbf42400640 1 -- 172.21.15.27:0/1720648732 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf3c06c080 msgr2=0x7fbf3c0c72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:18.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.291+0000 7fbf42400640 1 --2- 172.21.15.27:0/1720648732 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf3c06c080 0x7fbf3c0c72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:18.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.291+0000 7fbf42400640 1 -- 172.21.15.27:0/1720648732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf3c0cfaa0 msgr2=0x7fbf3c06eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:18.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.291+0000 7fbf42400640 1 --2- 172.21.15.27:0/1720648732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf3c0cfaa0 0x7fbf3c06eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:18.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.291+0000 7fbf42400640 1 -- 172.21.15.27:0/1720648732 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf2c0094d0 con 0x7fbf3c160af0 2024-01-15T18:13:18.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.291+0000 7fbf41000640 1 --2- 172.21.15.27:0/1720648732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf3c0cfaa0 0x7fbf3c06eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:13:18.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.291+0000 7fbf41a00640 1 --2- 172.21.15.27:0/1720648732 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf3c06c080 0x7fbf3c0c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:13:18.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.291+0000 7fbf42400640 1 --2- 172.21.15.27:0/1720648732 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf3c160af0 0x7fbf3c14d2a0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fbf3400be00 tx=0x7fbf3400bef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:18.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.291+0000 7fbf39600640 1 -- 172.21.15.27:0/1720648732 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3400ce70 con 0x7fbf3c160af0 2024-01-15T18:13:18.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.291+0000 7fbf39600640 1 -- 172.21.15.27:0/1720648732 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf34003980 con 0x7fbf3c160af0 2024-01-15T18:13:18.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.291+0000 7fbf39600640 1 -- 172.21.15.27:0/1720648732 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf34003d70 con 0x7fbf3c160af0 2024-01-15T18:13:18.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.291+0000 7fbf44400640 1 -- 172.21.15.27:0/1720648732 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf3c14e5f0 con 0x7fbf3c160af0 2024-01-15T18:13:18.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.291+0000 7fbf44400640 1 -- 172.21.15.27:0/1720648732 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf3c14ebb0 con 0x7fbf3c160af0 2024-01-15T18:13:18.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.291+0000 7fbf44400640 1 -- 172.21.15.27:0/1720648732 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf3c14f0d0 con 0x7fbf3c160af0 2024-01-15T18:13:18.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.291+0000 7fbf39600640 1 -- 172.21.15.27:0/1720648732 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7fbf34003ed0 con 0x7fbf3c160af0 2024-01-15T18:13:18.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.295+0000 7fbf39600640 1 --2- 172.21.15.27:0/1720648732 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbf140673d0 0x7fbf14069890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:18.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.295+0000 7fbf39600640 1 -- 172.21.15.27:0/1720648732 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fbf34082d30 con 0x7fbf3c160af0 2024-01-15T18:13:18.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.295+0000 7fbf41a00640 1 --2- 172.21.15.27:0/1720648732 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbf140673d0 0x7fbf14069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:18.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.295+0000 7fbf41a00640 1 --2- 172.21.15.27:0/1720648732 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbf140673d0 0x7fbf14069890 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0ae000 tx=0x7fbf2c0a6070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:18.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.299+0000 7fbf39600640 1 -- 172.21.15.27:0/1720648732 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbf3c14f0d0 con 0x7fbf3c160af0 2024-01-15T18:13:18.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.563+0000 7fbf44400640 1 -- 172.21.15.27:0/1720648732 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbf3c14f090 con 0x7fbf3c160af0 2024-01-15T18:13:18.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.567+0000 7fbf39600640 1 -- 172.21.15.27:0/1720648732 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7fbf340519e0 con 0x7fbf3c160af0 2024-01-15T18:13:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:13:18.568 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:13:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:13:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:13:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:13:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:13:18.568 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:13:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:13:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:13:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:13:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:13:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:13:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:13:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:13:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:13:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:13:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 525, 2024-01-15T18:13:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:13:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:13:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:13:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:13:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:13:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 72, 2024-01-15T18:13:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:13:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:13:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:13:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:13:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:13:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:13:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:13:18.570 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:13:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:13:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:13:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:13:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:13:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 1315, 2024-01-15T18:13:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 5369983076, 2024-01-15T18:13:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 11167285248, 2024-01-15T18:13:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 761926828032, 2024-01-15T18:13:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:13:18.571 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 179166403, 2024-01-15T18:13:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 178831050, 2024-01-15T18:13:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 43, 2024-01-15T18:13:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 47 2024-01-15T18:13:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:13:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 48, 2024-01-15T18:13:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 5, 2024-01-15T18:13:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:13:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:13:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:13:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:13:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 5, 2024-01-15T18:13:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:13:18.572 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:13:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:13:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 7227 2024-01-15T18:13:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:13:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:13:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:13:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:13:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:13:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:13:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:13:18.573 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:13:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:13:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:13:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 16, 2024-01-15T18:13:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:13:08.281503+0000", 2024-01-15T18:13:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:13:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:13:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "6eefee50-4f20-4629-a81e-c700e83869a8": { 2024-01-15T18:13:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (12s)\n [====........................] (remaining: 64s)", 2024-01-15T18:13:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.16664999723434448, 2024-01-15T18:13:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:18.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "8fb3fd3d-b405-4726-bf6b-75de006e8a09": { 2024-01-15T18:13:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (12s)\n [==..........................] (remaining: 2m)", 2024-01-15T18:13:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.095228567719459534, 2024-01-15T18:13:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:18.575 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:18.575 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:13:18.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.571+0000 7fbf44400640 1 -- 172.21.15.27:0/1720648732 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbf140673d0 msgr2=0x7fbf14069890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:18.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.571+0000 7fbf44400640 1 --2- 172.21.15.27:0/1720648732 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbf140673d0 0x7fbf14069890 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0ae000 tx=0x7fbf2c0a6070 comp rx=0 tx=0).stop 2024-01-15T18:13:18.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.571+0000 7fbf44400640 1 -- 172.21.15.27:0/1720648732 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf3c160af0 msgr2=0x7fbf3c14d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:18.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.571+0000 7fbf44400640 1 --2- 172.21.15.27:0/1720648732 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf3c160af0 0x7fbf3c14d2a0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fbf3400be00 tx=0x7fbf3400bef0 comp rx=0 tx=0).stop 2024-01-15T18:13:18.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.571+0000 7fbf44400640 1 -- 172.21.15.27:0/1720648732 shutdown_connections 2024-01-15T18:13:18.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.571+0000 7fbf44400640 1 --2- 172.21.15.27:0/1720648732 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbf140673d0 0x7fbf14069890 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:18.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.571+0000 7fbf44400640 1 --2- 172.21.15.27:0/1720648732 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf3c160af0 0x7fbf3c14d2a0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:18.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.571+0000 7fbf44400640 1 --2- 172.21.15.27:0/1720648732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf3c0cfaa0 0x7fbf3c06eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:18.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.571+0000 7fbf44400640 1 --2- 172.21.15.27:0/1720648732 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf3c06c080 0x7fbf3c0c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:18.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.571+0000 7fbf44400640 1 -- 172.21.15.27:0/1720648732 >> 172.21.15.27:0/1720648732 conn(0x7fbf3c0d0f20 msgr2=0x7fbf3c0cef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:18.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.571+0000 7fbf44400640 1 -- 172.21.15.27:0/1720648732 shutdown_connections 2024-01-15T18:13:18.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:18.571+0000 7fbf44400640 1 -- 172.21.15.27:0/1720648732 wait complete. 2024-01-15T18:13:19.614 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-15T18:13:19.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.855+0000 7fd1e0000640 1 Processor -- start 2024-01-15T18:13:19.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.855+0000 7fd1e0000640 1 -- start start 2024-01-15T18:13:19.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.855+0000 7fd1e0000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd1d81614d0 0x7fd1d80b4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:19.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.855+0000 7fd1e0000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd1d81618b0 0x7fd1d8071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:19.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.859+0000 7fd1e0000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d806a620 con 0x7fd1d806b6c0 2024-01-15T18:13:19.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.859+0000 7fd1e0000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d806a900 con 0x7fd1d81618b0 2024-01-15T18:13:19.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.859+0000 7fd1e0000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d806abe0 con 0x7fd1d81614d0 2024-01-15T18:13:19.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.859+0000 7fd1dea00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd1d806b6c0 0x7fd1d81651e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56282/0 (socket says 172.21.15.27:56282) 2024-01-15T18:13:19.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.859+0000 7fd1dea00640 1 -- 172.21.15.27:0/3809717261 learned_addr learned my addr 172.21.15.27:0/3809717261 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:13:19.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.859+0000 7fd1dcc00640 1 --2- 172.21.15.27:0/3809717261 >> v2:172.21.15.27:3300/0 conn(0x7fd1d81618b0 0x7fd1d8071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:19.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.859+0000 7fd1dcc00640 1 -- 172.21.15.27:0/3809717261 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd1d81614d0 msgr2=0x7fd1d80b4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:19.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.859+0000 7fd1dcc00640 1 --2- 172.21.15.27:0/3809717261 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd1d81614d0 0x7fd1d80b4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:19.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.859+0000 7fd1dcc00640 1 -- 172.21.15.27:0/3809717261 >> v1:172.21.15.27:6789/0 conn(0x7fd1d806b6c0 legacy=0x7fd1d81651e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:19.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.859+0000 7fd1dcc00640 1 -- 172.21.15.27:0/3809717261 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d806aec0 con 0x7fd1d81618b0 2024-01-15T18:13:19.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.863+0000 7fd1dcc00640 1 --2- 172.21.15.27:0/3809717261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d81618b0 0x7fd1d8071c70 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fd1c00097c0 tx=0x7fd1c001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94e8b0711d8f37f3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:19.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.863+0000 7fd1d7e00640 1 -- 172.21.15.27:0/3809717261 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c00ac640 con 0x7fd1d81618b0 2024-01-15T18:13:19.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.863+0000 7fd1d7e00640 1 -- 172.21.15.27:0/3809717261 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c00ac7a0 con 0x7fd1d81618b0 2024-01-15T18:13:19.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.863+0000 7fd1d7e00640 1 -- 172.21.15.27:0/3809717261 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c00acb30 con 0x7fd1d81618b0 2024-01-15T18:13:19.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.867+0000 7fd1e0000640 1 -- 172.21.15.27:0/3809717261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d81618b0 msgr2=0x7fd1d8071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:19.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.867+0000 7fd1e0000640 1 --2- 172.21.15.27:0/3809717261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d81618b0 0x7fd1d8071c70 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fd1c00097c0 tx=0x7fd1c001c3c0 comp rx=0 tx=0).stop 2024-01-15T18:13:19.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.867+0000 7fd1e0000640 1 -- 172.21.15.27:0/3809717261 shutdown_connections 2024-01-15T18:13:19.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.867+0000 7fd1e0000640 1 --2- 172.21.15.27:0/3809717261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d81618b0 0x7fd1d8071c70 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:19.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.867+0000 7fd1e0000640 1 --2- 172.21.15.27:0/3809717261 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd1d81614d0 0x7fd1d80b4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:19.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.867+0000 7fd1e0000640 1 -- 172.21.15.27:0/3809717261 >> 172.21.15.27:0/3809717261 conn(0x7fd1d80b2360 msgr2=0x7fd1d80b2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:19.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.867+0000 7fd1e0000640 1 -- 172.21.15.27:0/3809717261 shutdown_connections 2024-01-15T18:13:19.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.867+0000 7fd1e0000640 1 -- 172.21.15.27:0/3809717261 wait complete. 2024-01-15T18:13:19.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.867+0000 7fd1e0000640 1 Processor -- start 2024-01-15T18:13:19.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.867+0000 7fd1e0000640 1 -- start start 2024-01-15T18:13:19.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.871+0000 7fd1e0000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d806b6c0 0x7fd1d8154530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:19.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.871+0000 7fd1e0000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1d81614d0 0x7fd1d8162650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:19.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.871+0000 7fd1e0000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd1d81618b0 0x7fd1d81527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:19.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.871+0000 7fd1e0000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d80cdb20 con 0x7fd1d806b6c0 2024-01-15T18:13:19.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.871+0000 7fd1e0000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d80cde00 con 0x7fd1d81614d0 2024-01-15T18:13:19.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.871+0000 7fd1e0000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d80ce0e0 con 0x7fd1d81618b0 2024-01-15T18:13:19.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.871+0000 7fd1dd600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d806b6c0 0x7fd1d8154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:19.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.871+0000 7fd1dd600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d806b6c0 0x7fd1d8154530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56872/0 (socket says 172.21.15.27:56872) 2024-01-15T18:13:19.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.871+0000 7fd1dd600640 1 -- 172.21.15.27:0/960919918 learned_addr learned my addr 172.21.15.27:0/960919918 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:19.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.871+0000 7fd1dd600640 1 -- 172.21.15.27:0/960919918 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd1d81618b0 msgr2=0x7fd1d81527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:19.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.871+0000 7fd1dd600640 1 --2- 172.21.15.27:0/960919918 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd1d81618b0 0x7fd1d81527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:19.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.871+0000 7fd1dd600640 1 -- 172.21.15.27:0/960919918 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1d81614d0 msgr2=0x7fd1d8162650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:19.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.871+0000 7fd1dd600640 1 --2- 172.21.15.27:0/960919918 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1d81614d0 0x7fd1d8162650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:19.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.871+0000 7fd1dd600640 1 -- 172.21.15.27:0/960919918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c0009420 con 0x7fd1d806b6c0 2024-01-15T18:13:19.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.871+0000 7fd1dd600640 1 --2- 172.21.15.27:0/960919918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d806b6c0 0x7fd1d8154530 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fd1c800b550 tx=0x7fd1c800ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:19.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.871+0000 7fd1d5600640 1 -- 172.21.15.27:0/960919918 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8010870 con 0x7fd1d806b6c0 2024-01-15T18:13:19.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.871+0000 7fd1d5600640 1 -- 172.21.15.27:0/960919918 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c80109f0 con 0x7fd1d806b6c0 2024-01-15T18:13:19.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.871+0000 7fd1d5600640 1 -- 172.21.15.27:0/960919918 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8010da0 con 0x7fd1d806b6c0 2024-01-15T18:13:19.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.871+0000 7fd1e0000640 1 -- 172.21.15.27:0/960919918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d80ce3c0 con 0x7fd1d806b6c0 2024-01-15T18:13:19.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.871+0000 7fd1e0000640 1 -- 172.21.15.27:0/960919918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d8162b90 con 0x7fd1d806b6c0 2024-01-15T18:13:19.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.875+0000 7fd1e0000640 1 -- 172.21.15.27:0/960919918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1d8067a10 con 0x7fd1d806b6c0 2024-01-15T18:13:19.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.875+0000 7fd1d5600640 1 -- 172.21.15.27:0/960919918 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7fd1c800f3c0 con 0x7fd1d806b6c0 2024-01-15T18:13:19.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.875+0000 7fd1d5600640 1 --2- 172.21.15.27:0/960919918 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd1b00673a0 0x7fd1b0069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:19.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.875+0000 7fd1d5600640 1 -- 172.21.15.27:0/960919918 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fd1c8082df0 con 0x7fd1d806b6c0 2024-01-15T18:13:19.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.883+0000 7fd1d5600640 1 -- 172.21.15.27:0/960919918 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd1c8051a80 con 0x7fd1d806b6c0 2024-01-15T18:13:19.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.903+0000 7fd1dcc00640 1 --2- 172.21.15.27:0/960919918 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd1b00673a0 0x7fd1b0069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:19.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:19.907+0000 7fd1dcc00640 1 --2- 172.21.15.27:0/960919918 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd1b00673a0 0x7fd1b0069860 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fd1c00ae000 tx=0x7fd1c0004740 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:20.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:20.163+0000 7fd1e0000640 1 -- 172.21.15.27:0/960919918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd1d8074820 con 0x7fd1d806b6c0 2024-01-15T18:13:20.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:20.163+0000 7fd1d5600640 1 -- 172.21.15.27:0/960919918 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7fd1c8014990 con 0x7fd1d806b6c0 2024-01-15T18:13:20.165 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:13:20.165 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:13:20.165 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:13:20.165 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:13:20.165 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:13:20.165 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:13:20.165 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:13:20.165 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:20.165 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:13:20.165 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:13:20.165 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:13:20.165 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:13:20.166 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:13:20.166 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:20.166 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:13:20.166 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:13:20.166 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:13:20.166 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:13:20.166 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:20.166 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 527, 2024-01-15T18:13:20.166 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:13:20.166 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:13:20.166 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:13:20.166 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:13:20.166 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:20.166 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:13:20.166 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 72, 2024-01-15T18:13:20.166 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:13:20.167 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:13:20.167 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:13:20.167 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:13:20.167 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:13:20.167 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:13:20.167 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:20.167 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:13:20.167 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:13:20.167 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:20.167 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:13:20.167 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:13:20.167 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:20.167 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:20.167 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:13:20.168 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:13:20.168 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 1315, 2024-01-15T18:13:20.168 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 5369983076, 2024-01-15T18:13:20.168 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 11167285248, 2024-01-15T18:13:20.168 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 761926828032, 2024-01-15T18:13:20.168 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:13:20.168 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 179238622, 2024-01-15T18:13:20.168 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 178898783, 2024-01-15T18:13:20.168 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 43, 2024-01-15T18:13:20.168 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 46 2024-01-15T18:13:20.168 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:20.168 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:13:20.168 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 48, 2024-01-15T18:13:20.168 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 5, 2024-01-15T18:13:20.169 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:13:20.169 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:13:20.169 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:13:20.169 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:13:20.169 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:20.169 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 5, 2024-01-15T18:13:20.169 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:13:20.169 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:13:20.169 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:13:20.169 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 7227 2024-01-15T18:13:20.169 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:20.169 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:20.169 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:13:20.169 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:20.169 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:13:20.170 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:13:20.170 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:13:20.170 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:13:20.170 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:13:20.170 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:13:20.170 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:13:20.170 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:20.170 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:13:20.170 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:20.170 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:13:20.170 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 16, 2024-01-15T18:13:20.171 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:13:08.281503+0000", 2024-01-15T18:13:20.171 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:13:20.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:20.171 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:13:20.171 INFO:teuthology.orchestra.run.smithi027.stdout: "6eefee50-4f20-4629-a81e-c700e83869a8": { 2024-01-15T18:13:20.171 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "4 ongoing cloning jobs; Avg progress = 33.33% (22s)\n [=========...................] (remaining: 45s)", 2024-01-15T18:13:20.171 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.33329999446868896, 2024-01-15T18:13:20.171 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:20.171 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:20.171 INFO:teuthology.orchestra.run.smithi027.stdout: "8fb3fd3d-b405-4726-bf6b-75de006e8a09": { 2024-01-15T18:13:20.171 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 19.046% (22s)\n [=====.......................] (remaining: 96s)", 2024-01-15T18:13:20.171 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.19045713543891907, 2024-01-15T18:13:20.171 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:20.171 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:20.171 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:20.172 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:13:20.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:20.167+0000 7fd1e0000640 1 -- 172.21.15.27:0/960919918 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd1b00673a0 msgr2=0x7fd1b0069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:20.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:20.167+0000 7fd1e0000640 1 --2- 172.21.15.27:0/960919918 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd1b00673a0 0x7fd1b0069860 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fd1c00ae000 tx=0x7fd1c0004740 comp rx=0 tx=0).stop 2024-01-15T18:13:20.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:20.167+0000 7fd1e0000640 1 -- 172.21.15.27:0/960919918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d806b6c0 msgr2=0x7fd1d8154530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:20.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:20.167+0000 7fd1e0000640 1 --2- 172.21.15.27:0/960919918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d806b6c0 0x7fd1d8154530 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fd1c800b550 tx=0x7fd1c800ba10 comp rx=0 tx=0).stop 2024-01-15T18:13:20.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:20.167+0000 7fd1e0000640 1 -- 172.21.15.27:0/960919918 shutdown_connections 2024-01-15T18:13:20.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:20.167+0000 7fd1e0000640 1 --2- 172.21.15.27:0/960919918 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd1b00673a0 0x7fd1b0069860 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:20.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:20.167+0000 7fd1e0000640 1 --2- 172.21.15.27:0/960919918 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd1d81618b0 0x7fd1d81527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:20.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:20.167+0000 7fd1e0000640 1 --2- 172.21.15.27:0/960919918 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1d81614d0 0x7fd1d8162650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:20.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:20.167+0000 7fd1e0000640 1 --2- 172.21.15.27:0/960919918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1d806b6c0 0x7fd1d8154530 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:20.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:20.167+0000 7fd1e0000640 1 -- 172.21.15.27:0/960919918 >> 172.21.15.27:0/960919918 conn(0x7fd1d80b2360 msgr2=0x7fd1d8158360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:20.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:20.167+0000 7fd1e0000640 1 -- 172.21.15.27:0/960919918 shutdown_connections 2024-01-15T18:13:20.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:20.167+0000 7fd1e0000640 1 -- 172.21.15.27:0/960919918 wait complete. 2024-01-15T18:13:21.206 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-15T18:13:21.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.347+0000 7fc544a00640 1 Processor -- start 2024-01-15T18:13:21.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.347+0000 7fc544a00640 1 -- start start 2024-01-15T18:13:21.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.351+0000 7fc544a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc53c15e7d0 0x7fc53c0cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:21.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.351+0000 7fc544a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc53c0b38b0 con 0x7fc53c0cedc0 2024-01-15T18:13:21.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.351+0000 7fc544a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc53c0cd550 con 0x7fc53c0b20d0 2024-01-15T18:13:21.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.351+0000 7fc544a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc53c0cd270 con 0x7fc53c15e7d0 2024-01-15T18:13:21.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.351+0000 7fc542000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc53c15e7d0 0x7fc53c0cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:21.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.351+0000 7fc541600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc53c0cedc0 0x7fc53c0cba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56294/0 (socket says 172.21.15.27:56294) 2024-01-15T18:13:21.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.351+0000 7fc541600640 1 -- 172.21.15.27:0/1636749619 learned_addr learned my addr 172.21.15.27:0/1636749619 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:13:21.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.351+0000 7fc540c00640 1 -- 172.21.15.27:0/1636749619 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258693712 0 0) 0x7fc53c0cd550 con 0x7fc53c0b20d0 2024-01-15T18:13:21.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.351+0000 7fc542000640 1 -- 172.21.15.27:0/1636749619 >> v1:172.21.15.93:6789/0 conn(0x7fc53c0b20d0 legacy=0x7fc53c0cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:21.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.351+0000 7fc542000640 1 -- 172.21.15.27:0/1636749619 >> v1:172.21.15.27:6789/0 conn(0x7fc53c0cedc0 legacy=0x7fc53c0cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:21.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.351+0000 7fc542000640 1 -- 172.21.15.27:0/1636749619 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc53c0b35d0 con 0x7fc53c15e7d0 2024-01-15T18:13:21.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.355+0000 7fc542000640 1 --2- 172.21.15.27:0/1636749619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc53c15e7d0 0x7fc53c0cf2a0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fc52c0097c0 tx=0x7fc52c01c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82a24ee2d14b0683 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:21.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.355+0000 7fc540c00640 1 -- 172.21.15.27:0/1636749619 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c0ab600 con 0x7fc53c15e7d0 2024-01-15T18:13:21.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.355+0000 7fc540c00640 1 -- 172.21.15.27:0/1636749619 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc52c0a7050 con 0x7fc53c15e7d0 2024-01-15T18:13:21.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.355+0000 7fc540c00640 1 -- 172.21.15.27:0/1636749619 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c0ab980 con 0x7fc53c15e7d0 2024-01-15T18:13:21.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.355+0000 7fc544a00640 1 -- 172.21.15.27:0/1636749619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc53c15e7d0 msgr2=0x7fc53c0cf2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:21.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.355+0000 7fc544a00640 1 --2- 172.21.15.27:0/1636749619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc53c15e7d0 0x7fc53c0cf2a0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fc52c0097c0 tx=0x7fc52c01c310 comp rx=0 tx=0).stop 2024-01-15T18:13:21.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.355+0000 7fc544a00640 1 -- 172.21.15.27:0/1636749619 shutdown_connections 2024-01-15T18:13:21.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.355+0000 7fc544a00640 1 --2- 172.21.15.27:0/1636749619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc53c15e7d0 0x7fc53c0cf2a0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:21.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.355+0000 7fc544a00640 1 -- 172.21.15.27:0/1636749619 >> 172.21.15.27:0/1636749619 conn(0x7fc53c0b2a10 msgr2=0x7fc53c0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:21.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.355+0000 7fc544a00640 1 -- 172.21.15.27:0/1636749619 shutdown_connections 2024-01-15T18:13:21.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.355+0000 7fc544a00640 1 -- 172.21.15.27:0/1636749619 wait complete. 2024-01-15T18:13:21.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.355+0000 7fc544a00640 1 Processor -- start 2024-01-15T18:13:21.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.355+0000 7fc544a00640 1 -- start start 2024-01-15T18:13:21.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc544a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc53c0b20d0 0x7fc53c15b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:21.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc544a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc53c0cedc0 0x7fc53c0aed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:21.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc544a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc53c15e7d0 0x7fc53c0af280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:21.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc544a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c15b9a0 con 0x7fc53c15e7d0 2024-01-15T18:13:21.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc544a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c0c9240 con 0x7fc53c0b20d0 2024-01-15T18:13:21.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc544a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c1f3a80 con 0x7fc53c0cedc0 2024-01-15T18:13:21.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc542a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc53c15e7d0 0x7fc53c0af280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:21.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc542000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc53c0b20d0 0x7fc53c15b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:21.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc542000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc53c0b20d0 0x7fc53c15b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:59774/0 (socket says 172.21.15.27:59774) 2024-01-15T18:13:21.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc542000640 1 -- 172.21.15.27:0/1515870557 learned_addr learned my addr 172.21.15.27:0/1515870557 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:21.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc542a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc53c15e7d0 0x7fc53c0af280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56888/0 (socket says 172.21.15.27:56888) 2024-01-15T18:13:21.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc541600640 1 --2- 172.21.15.27:0/1515870557 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc53c0cedc0 0x7fc53c0aed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:21.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc542a00640 1 -- 172.21.15.27:0/1515870557 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc53c0cedc0 msgr2=0x7fc53c0aed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:21.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc542a00640 1 --2- 172.21.15.27:0/1515870557 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc53c0cedc0 0x7fc53c0aed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:21.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc542a00640 1 -- 172.21.15.27:0/1515870557 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc53c0b20d0 msgr2=0x7fc53c15b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:21.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc542a00640 1 --2- 172.21.15.27:0/1515870557 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc53c0b20d0 0x7fc53c15b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:21.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc542a00640 1 -- 172.21.15.27:0/1515870557 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52c009420 con 0x7fc53c15e7d0 2024-01-15T18:13:21.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc542000640 1 --2- 172.21.15.27:0/1515870557 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc53c0b20d0 0x7fc53c15b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:13:21.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc542a00640 1 --2- 172.21.15.27:0/1515870557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc53c15e7d0 0x7fc53c0af280 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fc53400c780 tx=0x7fc53400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:21.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc53a000640 1 -- 172.21.15.27:0/1515870557 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc534003b80 con 0x7fc53c15e7d0 2024-01-15T18:13:21.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc53a000640 1 -- 172.21.15.27:0/1515870557 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc534003ce0 con 0x7fc53c15e7d0 2024-01-15T18:13:21.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc544a00640 1 -- 172.21.15.27:0/1515870557 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc53c0c9500 con 0x7fc53c15e7d0 2024-01-15T18:13:21.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc53a000640 1 -- 172.21.15.27:0/1515870557 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5340052b0 con 0x7fc53c15e7d0 2024-01-15T18:13:21.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc544a00640 1 -- 172.21.15.27:0/1515870557 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc53c1f3ea0 con 0x7fc53c15e7d0 2024-01-15T18:13:21.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc544a00640 1 -- 172.21.15.27:0/1515870557 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc53c04e850 con 0x7fc53c15e7d0 2024-01-15T18:13:21.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.359+0000 7fc53a000640 1 -- 172.21.15.27:0/1515870557 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7fc534005410 con 0x7fc53c15e7d0 2024-01-15T18:13:21.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.363+0000 7fc53a000640 1 --2- 172.21.15.27:0/1515870557 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc5140673a0 0x7fc514069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:21.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.363+0000 7fc53a000640 1 -- 172.21.15.27:0/1515870557 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fc534082670 con 0x7fc53c15e7d0 2024-01-15T18:13:21.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.363+0000 7fc542000640 1 --2- 172.21.15.27:0/1515870557 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc5140673a0 0x7fc514069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:21.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.363+0000 7fc542000640 1 --2- 172.21.15.27:0/1515870557 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc5140673a0 0x7fc514069860 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fc52c019f40 tx=0x7fc52c0a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:21.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.367+0000 7fc53a000640 1 -- 172.21.15.27:0/1515870557 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc534051180 con 0x7fc53c15e7d0 2024-01-15T18:13:21.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.579+0000 7fc544a00640 1 -- 172.21.15.27:0/1515870557 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc53c0c99b0 con 0x7fc53c15e7d0 2024-01-15T18:13:21.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.579+0000 7fc53a000640 1 -- 172.21.15.27:0/1515870557 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7fc534017990 con 0x7fc53c15e7d0 2024-01-15T18:13:21.579 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:13:21.579 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:13:21.579 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:13:21.579 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:13:21.579 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:13:21.579 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:13:21.579 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:13:21.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:21.579 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:13:21.580 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:13:21.580 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:13:21.580 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:13:21.580 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:13:21.580 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:21.580 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:13:21.580 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:13:21.580 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:13:21.580 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:13:21.580 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:21.580 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 528, 2024-01-15T18:13:21.580 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:13:21.580 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:13:21.580 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:13:21.581 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:13:21.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:21.581 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:13:21.581 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 72, 2024-01-15T18:13:21.581 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:13:21.581 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:13:21.581 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:13:21.581 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:13:21.581 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:13:21.581 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:13:21.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:21.581 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:13:21.581 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:13:21.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:21.581 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:13:21.582 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:13:21.582 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:21.582 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:21.582 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:13:21.582 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:13:21.582 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 1490, 2024-01-15T18:13:21.582 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 6103986276, 2024-01-15T18:13:21.582 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 12228444160, 2024-01-15T18:13:21.582 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 760865669120, 2024-01-15T18:13:21.582 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:13:21.582 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 236514899, 2024-01-15T18:13:21.582 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 240018020, 2024-01-15T18:13:21.582 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 56, 2024-01-15T18:13:21.582 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 60 2024-01-15T18:13:21.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:21.582 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:13:21.583 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 48, 2024-01-15T18:13:21.583 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 5, 2024-01-15T18:13:21.583 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:13:21.583 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:13:21.583 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:13:21.583 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:13:21.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:21.583 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 5, 2024-01-15T18:13:21.583 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:13:21.583 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:13:21.583 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:13:21.583 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 7227 2024-01-15T18:13:21.584 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:21.584 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:21.584 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:13:21.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:21.584 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:13:21.584 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:13:21.584 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:13:21.584 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:13:21.584 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:13:21.584 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:13:21.584 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:13:21.584 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:21.584 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:13:21.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:21.584 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:13:21.584 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 16, 2024-01-15T18:13:21.585 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:13:08.281503+0000", 2024-01-15T18:13:21.585 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:13:21.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:21.585 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:13:21.585 INFO:teuthology.orchestra.run.smithi027.stdout: "6eefee50-4f20-4629-a81e-c700e83869a8": { 2024-01-15T18:13:21.585 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "4 ongoing cloning jobs; Avg progress = 33.33% (22s)\n [=========...................] (remaining: 45s)", 2024-01-15T18:13:21.585 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.33329999446868896, 2024-01-15T18:13:21.585 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:21.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:21.585 INFO:teuthology.orchestra.run.smithi027.stdout: "8fb3fd3d-b405-4726-bf6b-75de006e8a09": { 2024-01-15T18:13:21.585 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 19.046% (22s)\n [=====.......................] (remaining: 96s)", 2024-01-15T18:13:21.585 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.19045713543891907, 2024-01-15T18:13:21.585 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:21.585 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:21.586 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:21.586 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:13:21.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.579+0000 7fc544a00640 1 -- 172.21.15.27:0/1515870557 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc5140673a0 msgr2=0x7fc514069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:21.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.579+0000 7fc544a00640 1 --2- 172.21.15.27:0/1515870557 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc5140673a0 0x7fc514069860 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fc52c019f40 tx=0x7fc52c0a6000 comp rx=0 tx=0).stop 2024-01-15T18:13:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.579+0000 7fc544a00640 1 -- 172.21.15.27:0/1515870557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc53c15e7d0 msgr2=0x7fc53c0af280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.579+0000 7fc544a00640 1 --2- 172.21.15.27:0/1515870557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc53c15e7d0 0x7fc53c0af280 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fc53400c780 tx=0x7fc53400cc40 comp rx=0 tx=0).stop 2024-01-15T18:13:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.579+0000 7fc544a00640 1 -- 172.21.15.27:0/1515870557 shutdown_connections 2024-01-15T18:13:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.579+0000 7fc544a00640 1 --2- 172.21.15.27:0/1515870557 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc5140673a0 0x7fc514069860 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.579+0000 7fc544a00640 1 --2- 172.21.15.27:0/1515870557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc53c15e7d0 0x7fc53c0af280 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.579+0000 7fc544a00640 1 --2- 172.21.15.27:0/1515870557 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc53c0cedc0 0x7fc53c0aed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.579+0000 7fc544a00640 1 --2- 172.21.15.27:0/1515870557 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc53c0b20d0 0x7fc53c15b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.579+0000 7fc544a00640 1 -- 172.21.15.27:0/1515870557 >> 172.21.15.27:0/1515870557 conn(0x7fc53c0b2a10 msgr2=0x7fc53c0cf1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.579+0000 7fc544a00640 1 -- 172.21.15.27:0/1515870557 shutdown_connections 2024-01-15T18:13:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:21.579+0000 7fc544a00640 1 -- 172.21.15.27:0/1515870557 wait complete. 2024-01-15T18:13:22.601 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-15T18:13:22.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.751+0000 7f4dd9400640 1 Processor -- start 2024-01-15T18:13:22.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.751+0000 7f4dd9400640 1 -- start start 2024-01-15T18:13:22.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.755+0000 7f4dd9400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4dd406c010 0x7f4dd406b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:22.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.755+0000 7f4dd9400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4dd415caf0 0x7f4dd415cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:22.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.755+0000 7f4dd9400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd40cc310 con 0x7f4dd40b3cd0 2024-01-15T18:13:22.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.755+0000 7f4dd9400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd40cc5d0 con 0x7f4dd415caf0 2024-01-15T18:13:22.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.755+0000 7f4dd9400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd40cc8b0 con 0x7f4dd406c010 2024-01-15T18:13:22.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.755+0000 7f4dd3200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4dd415caf0 0x7f4dd415cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:22.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.755+0000 7f4dd3200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4dd415caf0 0x7f4dd415cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56898/0 (socket says 172.21.15.27:56898) 2024-01-15T18:13:22.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.755+0000 7f4dd3200640 1 -- 172.21.15.27:0/4078809176 learned_addr learned my addr 172.21.15.27:0/4078809176 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:22.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.755+0000 7f4dd2800640 1 --2- 172.21.15.27:0/4078809176 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4dd406c010 0x7f4dd406b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:22.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.755+0000 7f4dd3200640 1 -- 172.21.15.27:0/4078809176 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4dd406c010 msgr2=0x7f4dd406b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:22.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.755+0000 7f4dd3200640 1 --2- 172.21.15.27:0/4078809176 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4dd406c010 0x7f4dd406b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:22.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.755+0000 7f4dd3200640 1 -- 172.21.15.27:0/4078809176 >> v1:172.21.15.93:6789/0 conn(0x7f4dd40b3cd0 legacy=0x7f4dd406c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:22.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.755+0000 7f4dd3200640 1 -- 172.21.15.27:0/4078809176 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd40ccb90 con 0x7f4dd415caf0 2024-01-15T18:13:22.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.759+0000 7f4dd3200640 1 --2- 172.21.15.27:0/4078809176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd415caf0 0x7f4dd415cf80 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc009e50 tx=0x7f4dcc01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6de0832fa64058 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:22.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.759+0000 7f4dd1400640 1 -- 172.21.15.27:0/4078809176 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc0a55a0 con 0x7f4dd415caf0 2024-01-15T18:13:22.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.759+0000 7f4dd1400640 1 -- 172.21.15.27:0/4078809176 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dcc0a5700 con 0x7f4dd415caf0 2024-01-15T18:13:22.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.759+0000 7f4dd1400640 1 -- 172.21.15.27:0/4078809176 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc0a5a90 con 0x7f4dd415caf0 2024-01-15T18:13:22.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.759+0000 7f4dd9400640 1 -- 172.21.15.27:0/4078809176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd415caf0 msgr2=0x7f4dd415cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:22.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.759+0000 7f4dd9400640 1 --2- 172.21.15.27:0/4078809176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd415caf0 0x7f4dd415cf80 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc009e50 tx=0x7f4dcc01c910 comp rx=0 tx=0).stop 2024-01-15T18:13:22.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.759+0000 7f4dd9400640 1 -- 172.21.15.27:0/4078809176 shutdown_connections 2024-01-15T18:13:22.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.759+0000 7f4dd9400640 1 --2- 172.21.15.27:0/4078809176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd415caf0 0x7f4dd415cf80 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:22.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.759+0000 7f4dd9400640 1 --2- 172.21.15.27:0/4078809176 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4dd406c010 0x7f4dd406b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:22.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.759+0000 7f4dd9400640 1 -- 172.21.15.27:0/4078809176 >> 172.21.15.27:0/4078809176 conn(0x7f4dd40d0e80 msgr2=0x7f4dd40d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:22.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.759+0000 7f4dd9400640 1 -- 172.21.15.27:0/4078809176 shutdown_connections 2024-01-15T18:13:22.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.759+0000 7f4dd9400640 1 -- 172.21.15.27:0/4078809176 wait complete. 2024-01-15T18:13:22.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.759+0000 7f4dd9400640 1 Processor -- start 2024-01-15T18:13:22.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.759+0000 7f4dd9400640 1 -- start start 2024-01-15T18:13:22.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.763+0000 7f4dd9400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4dd406c010 0x7f4dd41539f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:22.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.763+0000 7f4dd9400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4dd40b3cd0 0x7f4dd4153f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:22.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.763+0000 7f4dd9400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd415caf0 0x7f4dd41524a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:22.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.763+0000 7f4dd9400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd40c8b90 con 0x7f4dd415caf0 2024-01-15T18:13:22.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.763+0000 7f4dd9400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd40cafc0 con 0x7f4dd40b3cd0 2024-01-15T18:13:22.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.763+0000 7f4dd9400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd40ca3b0 con 0x7f4dd406c010 2024-01-15T18:13:22.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.763+0000 7f4dd3200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd415caf0 0x7f4dd41524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:22.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.763+0000 7f4dd3200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd415caf0 0x7f4dd41524a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56910/0 (socket says 172.21.15.27:56910) 2024-01-15T18:13:22.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.763+0000 7f4dd3200640 1 -- 172.21.15.27:0/578656992 learned_addr learned my addr 172.21.15.27:0/578656992 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:22.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.763+0000 7f4dd2800640 1 --2- 172.21.15.27:0/578656992 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4dd406c010 0x7f4dd41539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:22.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.763+0000 7f4dd1e00640 1 --2- 172.21.15.27:0/578656992 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4dd40b3cd0 0x7f4dd4153f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:22.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.763+0000 7f4dd3200640 1 -- 172.21.15.27:0/578656992 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4dd406c010 msgr2=0x7f4dd41539f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:22.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.763+0000 7f4dd3200640 1 --2- 172.21.15.27:0/578656992 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4dd406c010 0x7f4dd41539f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:22.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.763+0000 7f4dd3200640 1 -- 172.21.15.27:0/578656992 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4dd40b3cd0 msgr2=0x7f4dd4153f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:22.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.763+0000 7f4dd3200640 1 --2- 172.21.15.27:0/578656992 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4dd40b3cd0 0x7f4dd4153f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:22.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.763+0000 7f4dd3200640 1 -- 172.21.15.27:0/578656992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dcc009b30 con 0x7f4dd415caf0 2024-01-15T18:13:22.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.763+0000 7f4dd3200640 1 --2- 172.21.15.27:0/578656992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd415caf0 0x7f4dd41524a0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc0af040 tx=0x7f4dcc0196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:22.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.763+0000 7f4dcaa00640 1 -- 172.21.15.27:0/578656992 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc0a55a0 con 0x7f4dd415caf0 2024-01-15T18:13:22.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.763+0000 7f4dcaa00640 1 -- 172.21.15.27:0/578656992 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dcc0b8670 con 0x7f4dd415caf0 2024-01-15T18:13:22.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.763+0000 7f4dcaa00640 1 -- 172.21.15.27:0/578656992 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc0b8a20 con 0x7f4dd415caf0 2024-01-15T18:13:22.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.763+0000 7f4dd9400640 1 -- 172.21.15.27:0/578656992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd40ca690 con 0x7f4dd415caf0 2024-01-15T18:13:22.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.763+0000 7f4dd9400640 1 -- 172.21.15.27:0/578656992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd40cac50 con 0x7f4dd415caf0 2024-01-15T18:13:22.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.763+0000 7f4dd9400640 1 -- 172.21.15.27:0/578656992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd40cb130 con 0x7f4dd415caf0 2024-01-15T18:13:22.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.767+0000 7f4dcaa00640 1 -- 172.21.15.27:0/578656992 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f4dcc0b8b80 con 0x7f4dd415caf0 2024-01-15T18:13:22.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.767+0000 7f4dcaa00640 1 --2- 172.21.15.27:0/578656992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4dac0673a0 0x7f4dac069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:22.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.767+0000 7f4dcaa00640 1 -- 172.21.15.27:0/578656992 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f4dcc11e060 con 0x7f4dd415caf0 2024-01-15T18:13:22.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.767+0000 7f4dd2800640 1 --2- 172.21.15.27:0/578656992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4dac0673a0 0x7f4dac069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:22.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.767+0000 7f4dd2800640 1 --2- 172.21.15.27:0/578656992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4dac0673a0 0x7f4dac069860 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4154760 tx=0x7f4dc4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:22.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.771+0000 7f4dcaa00640 1 -- 172.21.15.27:0/578656992 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4dcc0ecda0 con 0x7f4dd415caf0 2024-01-15T18:13:22.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.987+0000 7f4dd9400640 1 -- 172.21.15.27:0/578656992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4dd404e850 con 0x7f4dd415caf0 2024-01-15T18:13:22.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.987+0000 7f4dcaa00640 1 -- 172.21.15.27:0/578656992 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f4dcc0bc990 con 0x7f4dd415caf0 2024-01-15T18:13:22.989 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:13:22.989 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:13:22.989 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:13:22.989 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:13:22.989 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:13:22.989 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:13:22.989 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:13:22.989 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:22.989 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:13:22.989 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:13:22.989 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:13:22.990 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:13:22.990 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:13:22.990 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:22.990 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:13:22.990 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:13:22.990 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:13:22.990 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:13:22.990 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:22.990 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 529, 2024-01-15T18:13:22.990 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:13:22.990 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:13:22.990 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:13:22.990 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:13:22.990 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:22.991 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:13:22.991 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 72, 2024-01-15T18:13:22.991 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:13:22.991 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:13:22.991 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:13:22.991 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:13:22.991 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:13:22.991 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:13:22.991 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:22.991 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:13:22.991 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:13:22.991 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:22.991 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:13:22.991 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:13:22.991 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:22.992 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:22.992 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:13:22.992 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:13:22.992 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 1490, 2024-01-15T18:13:22.992 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 6103986276, 2024-01-15T18:13:22.992 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 12228444160, 2024-01-15T18:13:22.992 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 760865669120, 2024-01-15T18:13:22.992 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:13:22.992 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 236514899, 2024-01-15T18:13:22.992 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 240018020, 2024-01-15T18:13:22.992 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 56, 2024-01-15T18:13:22.992 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 60 2024-01-15T18:13:22.992 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:22.992 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:13:22.993 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 48, 2024-01-15T18:13:22.993 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 5, 2024-01-15T18:13:22.993 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:13:22.993 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:13:22.993 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:13:22.993 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:13:22.993 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:22.993 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 5, 2024-01-15T18:13:22.993 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:13:22.993 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:13:22.993 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:13:22.993 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 7227 2024-01-15T18:13:22.993 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:22.993 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:22.993 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:13:22.993 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:22.994 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:13:22.994 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:13:22.994 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:13:22.994 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:13:22.994 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:13:22.994 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:13:22.994 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:13:22.994 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:22.994 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:13:22.994 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:22.994 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:13:22.994 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 16, 2024-01-15T18:13:22.994 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:13:08.281503+0000", 2024-01-15T18:13:22.994 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:13:22.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:22.995 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:13:22.995 INFO:teuthology.orchestra.run.smithi027.stdout: "6eefee50-4f20-4629-a81e-c700e83869a8": { 2024-01-15T18:13:22.995 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "4 ongoing cloning jobs; Avg progress = 33.33% (22s)\n [=========...................] (remaining: 45s)", 2024-01-15T18:13:22.995 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.33329999446868896, 2024-01-15T18:13:22.995 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:22.995 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:22.995 INFO:teuthology.orchestra.run.smithi027.stdout: "8fb3fd3d-b405-4726-bf6b-75de006e8a09": { 2024-01-15T18:13:22.995 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 19.046% (22s)\n [=====.......................] (remaining: 96s)", 2024-01-15T18:13:22.995 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.19045713543891907, 2024-01-15T18:13:22.995 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:22.995 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:22.995 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:22.995 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:13:22.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.991+0000 7f4dd9400640 1 -- 172.21.15.27:0/578656992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4dac0673a0 msgr2=0x7f4dac069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:22.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.991+0000 7f4dd9400640 1 --2- 172.21.15.27:0/578656992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4dac0673a0 0x7f4dac069860 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4154760 tx=0x7f4dc4008040 comp rx=0 tx=0).stop 2024-01-15T18:13:22.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.991+0000 7f4dd9400640 1 -- 172.21.15.27:0/578656992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd415caf0 msgr2=0x7f4dd41524a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:22.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.991+0000 7f4dd9400640 1 --2- 172.21.15.27:0/578656992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd415caf0 0x7f4dd41524a0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc0af040 tx=0x7f4dcc0196b0 comp rx=0 tx=0).stop 2024-01-15T18:13:22.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.991+0000 7f4dd9400640 1 -- 172.21.15.27:0/578656992 shutdown_connections 2024-01-15T18:13:22.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.991+0000 7f4dd9400640 1 --2- 172.21.15.27:0/578656992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4dac0673a0 0x7f4dac069860 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:22.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.991+0000 7f4dd9400640 1 --2- 172.21.15.27:0/578656992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4dd415caf0 0x7f4dd41524a0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:22.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.991+0000 7f4dd9400640 1 --2- 172.21.15.27:0/578656992 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4dd40b3cd0 0x7f4dd4153f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:22.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.991+0000 7f4dd9400640 1 --2- 172.21.15.27:0/578656992 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4dd406c010 0x7f4dd41539f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:22.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.991+0000 7f4dd9400640 1 -- 172.21.15.27:0/578656992 >> 172.21.15.27:0/578656992 conn(0x7f4dd40d0e80 msgr2=0x7f4dd406b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:22.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.991+0000 7f4dd9400640 1 -- 172.21.15.27:0/578656992 shutdown_connections 2024-01-15T18:13:22.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:22.991+0000 7f4dd9400640 1 -- 172.21.15.27:0/578656992 wait complete. 2024-01-15T18:13:24.011 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-15T18:13:24.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.163+0000 7f4affe00640 1 Processor -- start 2024-01-15T18:13:24.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.163+0000 7f4affe00640 1 -- start start 2024-01-15T18:13:24.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.167+0000 7f4affe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4af80b2390 0x7f4af80cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:24.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.167+0000 7f4affe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4af806bb80 0x7f4af80cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:24.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.167+0000 7f4affe00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af80b2770 con 0x7f4af80cfcc0 2024-01-15T18:13:24.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.167+0000 7f4affe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4af806c060 con 0x7f4af806bb80 2024-01-15T18:13:24.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.167+0000 7f4affe00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4af806c340 con 0x7f4af80b2390 2024-01-15T18:13:24.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.167+0000 7f4afca00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4af806bb80 0x7f4af80cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:24.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.167+0000 7f4afde00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4af80cfcc0 0x7f4af8161120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53874/0 (socket says 172.21.15.27:53874) 2024-01-15T18:13:24.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.167+0000 7f4afde00640 1 -- 172.21.15.27:0/2527163641 learned_addr learned my addr 172.21.15.27:0/2527163641 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:13:24.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.167+0000 7f4afd400640 1 --2- 172.21.15.27:0/2527163641 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4af80b2390 0x7f4af80cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:24.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.167+0000 7f4afca00640 1 -- 172.21.15.27:0/2527163641 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4af80b2390 msgr2=0x7f4af80cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:24.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.167+0000 7f4afca00640 1 --2- 172.21.15.27:0/2527163641 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4af80b2390 0x7f4af80cf780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:24.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.167+0000 7f4afca00640 1 -- 172.21.15.27:0/2527163641 >> v1:172.21.15.27:6789/0 conn(0x7f4af80cfcc0 legacy=0x7f4af8161120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:24.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.167+0000 7f4afca00640 1 -- 172.21.15.27:0/2527163641 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af806c620 con 0x7f4af806bb80 2024-01-15T18:13:24.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.167+0000 7f4af7e00640 1 -- 172.21.15.27:0/2527163641 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1003114411 0 0) 0x7f4af80b2770 con 0x7f4af80cfcc0 2024-01-15T18:13:24.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.167+0000 7f4afca00640 1 --2- 172.21.15.27:0/2527163641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4af806bb80 0x7f4af80cd880 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f4ae00097c0 tx=0x7f4ae001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16c7444c8d8d3ca1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:24.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.171+0000 7f4af7e00640 1 -- 172.21.15.27:0/2527163641 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae00ab650 con 0x7f4af806bb80 2024-01-15T18:13:24.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.171+0000 7f4af7e00640 1 -- 172.21.15.27:0/2527163641 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ae00ab7b0 con 0x7f4af806bb80 2024-01-15T18:13:24.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.171+0000 7f4af7e00640 1 -- 172.21.15.27:0/2527163641 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae00abb40 con 0x7f4af806bb80 2024-01-15T18:13:24.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.171+0000 7f4affe00640 1 -- 172.21.15.27:0/2527163641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4af806bb80 msgr2=0x7f4af80cd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:24.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.171+0000 7f4affe00640 1 --2- 172.21.15.27:0/2527163641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4af806bb80 0x7f4af80cd880 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f4ae00097c0 tx=0x7f4ae001c450 comp rx=0 tx=0).stop 2024-01-15T18:13:24.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.171+0000 7f4affe00640 1 -- 172.21.15.27:0/2527163641 shutdown_connections 2024-01-15T18:13:24.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.171+0000 7f4affe00640 1 --2- 172.21.15.27:0/2527163641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4af806bb80 0x7f4af80cd880 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:24.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.171+0000 7f4affe00640 1 --2- 172.21.15.27:0/2527163641 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4af80b2390 0x7f4af80cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:24.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.171+0000 7f4affe00640 1 -- 172.21.15.27:0/2527163641 >> 172.21.15.27:0/2527163641 conn(0x7f4af80b2a10 msgr2=0x7f4af80b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:24.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.171+0000 7f4affe00640 1 -- 172.21.15.27:0/2527163641 shutdown_connections 2024-01-15T18:13:24.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.171+0000 7f4affe00640 1 -- 172.21.15.27:0/2527163641 wait complete. 2024-01-15T18:13:24.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.171+0000 7f4affe00640 1 Processor -- start 2024-01-15T18:13:24.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.171+0000 7f4affe00640 1 -- start start 2024-01-15T18:13:24.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.175+0000 7f4affe00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4af806bb80 0x7f4af80c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:24.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.175+0000 7f4affe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4af80b2390 0x7f4af815d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:24.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.175+0000 7f4affe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4af80cfcc0 0x7f4af815d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:24.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.175+0000 7f4affe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af815bfd0 con 0x7f4af80b2390 2024-01-15T18:13:24.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.175+0000 7f4affe00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af815c270 con 0x7f4af806bb80 2024-01-15T18:13:24.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.175+0000 7f4affe00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4af815c550 con 0x7f4af80cfcc0 2024-01-15T18:13:24.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.175+0000 7f4afca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4af80b2390 0x7f4af815d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:24.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.175+0000 7f4afca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4af80b2390 0x7f4af815d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40678/0 (socket says 172.21.15.27:40678) 2024-01-15T18:13:24.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.175+0000 7f4afca00640 1 -- 172.21.15.27:0/1663609033 learned_addr learned my addr 172.21.15.27:0/1663609033 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:24.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.175+0000 7f4afde00640 1 --2- 172.21.15.27:0/1663609033 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4af80cfcc0 0x7f4af815d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:24.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.175+0000 7f4afd400640 1 --2- 172.21.15.27:0/1663609033 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4af806bb80 0x7f4af80c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:24.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.175+0000 7f4afca00640 1 -- 172.21.15.27:0/1663609033 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4af80cfcc0 msgr2=0x7f4af815d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:24.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.175+0000 7f4afca00640 1 --2- 172.21.15.27:0/1663609033 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4af80cfcc0 0x7f4af815d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:24.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.175+0000 7f4afca00640 1 -- 172.21.15.27:0/1663609033 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4af806bb80 msgr2=0x7f4af80c7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:24.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.175+0000 7f4afca00640 1 --2- 172.21.15.27:0/1663609033 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4af806bb80 0x7f4af80c7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:24.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.175+0000 7f4afca00640 1 -- 172.21.15.27:0/1663609033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae0009420 con 0x7f4af80b2390 2024-01-15T18:13:24.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.175+0000 7f4afca00640 1 --2- 172.21.15.27:0/1663609033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4af80b2390 0x7f4af815d030 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f4ae00ad040 tx=0x7f4ae001ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:24.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.175+0000 7f4af5600640 1 -- 172.21.15.27:0/1663609033 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae00bf610 con 0x7f4af80b2390 2024-01-15T18:13:24.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.175+0000 7f4af5600640 1 -- 172.21.15.27:0/1663609033 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ae00bf790 con 0x7f4af80b2390 2024-01-15T18:13:24.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.175+0000 7f4affe00640 1 -- 172.21.15.27:0/1663609033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af815c830 con 0x7f4af80b2390 2024-01-15T18:13:24.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.175+0000 7f4af5600640 1 -- 172.21.15.27:0/1663609033 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae00bfb40 con 0x7f4af80b2390 2024-01-15T18:13:24.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.175+0000 7f4affe00640 1 -- 172.21.15.27:0/1663609033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af81561a0 con 0x7f4af80b2390 2024-01-15T18:13:24.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.175+0000 7f4affe00640 1 -- 172.21.15.27:0/1663609033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af80d07c0 con 0x7f4af80b2390 2024-01-15T18:13:24.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.175+0000 7f4af5600640 1 -- 172.21.15.27:0/1663609033 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f4ae00bfca0 con 0x7f4af80b2390 2024-01-15T18:13:24.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.179+0000 7f4af5600640 1 --2- 172.21.15.27:0/1663609033 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4ad00673a0 0x7f4ad0069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:24.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.179+0000 7f4af5600640 1 -- 172.21.15.27:0/1663609033 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f4ae011ddb0 con 0x7f4af80b2390 2024-01-15T18:13:24.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.179+0000 7f4afd400640 1 --2- 172.21.15.27:0/1663609033 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4ad00673a0 0x7f4ad0069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:24.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.179+0000 7f4afd400640 1 --2- 172.21.15.27:0/1663609033 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4ad00673a0 0x7f4ad0069860 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8009ac0 tx=0x7f4ae8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:24.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.183+0000 7f4af5600640 1 -- 172.21.15.27:0/1663609033 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4ae00ec9e0 con 0x7f4af80b2390 2024-01-15T18:13:24.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.395+0000 7f4affe00640 1 -- 172.21.15.27:0/1663609033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4af804e850 con 0x7f4af80b2390 2024-01-15T18:13:24.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.395+0000 7f4af5600640 1 -- 172.21.15.27:0/1663609033 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2411 (secure 0 0 0) 0x7f4ae00f0000 con 0x7f4af80b2390 2024-01-15T18:13:24.398 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:13:24.399 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:13:24.399 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:13:24.399 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:13:24.399 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:13:24.399 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:13:24.399 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:13:24.399 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:24.399 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:13:24.399 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:13:24.400 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:13:24.400 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:13:24.400 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:13:24.400 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:24.400 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:13:24.400 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:13:24.400 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:13:24.400 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:13:24.400 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:24.401 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 531, 2024-01-15T18:13:24.401 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:13:24.401 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:13:24.401 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:13:24.401 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:13:24.401 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:24.401 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:13:24.401 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 72, 2024-01-15T18:13:24.401 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:13:24.402 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:13:24.402 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:13:24.402 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:13:24.402 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:13:24.402 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:13:24.402 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:24.402 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:13:24.402 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:13:24.402 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:24.402 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:13:24.403 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:13:24.403 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:24.403 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:24.403 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:13:24.403 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:13:24.403 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 2125, 2024-01-15T18:13:24.403 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 8767488719, 2024-01-15T18:13:24.403 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 17689714688, 2024-01-15T18:13:24.403 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 755404398592, 2024-01-15T18:13:24.403 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:13:24.404 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 436006034, 2024-01-15T18:13:24.404 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 423448820, 2024-01-15T18:13:24.404 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 104, 2024-01-15T18:13:24.404 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 107 2024-01-15T18:13:24.404 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:24.404 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:13:24.404 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 48, 2024-01-15T18:13:24.404 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 5, 2024-01-15T18:13:24.404 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:13:24.404 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:13:24.405 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:13:24.405 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:13:24.405 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:24.405 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 5, 2024-01-15T18:13:24.405 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:13:24.405 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:13:24.405 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:13:24.405 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 7227 2024-01-15T18:13:24.405 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:24.405 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:24.405 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:13:24.406 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:24.406 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:13:24.406 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:13:24.406 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:13:24.406 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:13:24.406 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:13:24.406 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:13:24.406 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:13:24.406 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:24.406 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:13:24.407 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:24.407 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:13:24.407 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 16, 2024-01-15T18:13:24.407 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:13:08.281503+0000", 2024-01-15T18:13:24.407 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:13:24.407 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:24.407 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:13:24.407 INFO:teuthology.orchestra.run.smithi027.stdout: "6eefee50-4f20-4629-a81e-c700e83869a8": { 2024-01-15T18:13:24.407 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "4 ongoing cloning jobs; Avg progress = 50.0% (25s)\n [==============..............] (remaining: 25s)", 2024-01-15T18:13:24.407 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.5, 2024-01-15T18:13:24.407 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:24.408 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:24.408 INFO:teuthology.orchestra.run.smithi027.stdout: "8fb3fd3d-b405-4726-bf6b-75de006e8a09": { 2024-01-15T18:13:24.408 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 28.571% (25s)\n [========....................] (remaining: 64s)", 2024-01-15T18:13:24.408 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.28571429848670959, 2024-01-15T18:13:24.408 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:24.408 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:24.408 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:24.408 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:13:24.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.399+0000 7f4affe00640 1 -- 172.21.15.27:0/1663609033 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4ad00673a0 msgr2=0x7f4ad0069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:24.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.399+0000 7f4affe00640 1 --2- 172.21.15.27:0/1663609033 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4ad00673a0 0x7f4ad0069860 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8009ac0 tx=0x7f4ae8009190 comp rx=0 tx=0).stop 2024-01-15T18:13:24.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.399+0000 7f4affe00640 1 -- 172.21.15.27:0/1663609033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4af80b2390 msgr2=0x7f4af815d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:24.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.399+0000 7f4affe00640 1 --2- 172.21.15.27:0/1663609033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4af80b2390 0x7f4af815d030 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f4ae00ad040 tx=0x7f4ae001ac90 comp rx=0 tx=0).stop 2024-01-15T18:13:24.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.399+0000 7f4affe00640 1 -- 172.21.15.27:0/1663609033 shutdown_connections 2024-01-15T18:13:24.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.399+0000 7f4affe00640 1 --2- 172.21.15.27:0/1663609033 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4ad00673a0 0x7f4ad0069860 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:24.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.399+0000 7f4affe00640 1 --2- 172.21.15.27:0/1663609033 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4af80cfcc0 0x7f4af815d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:24.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.399+0000 7f4affe00640 1 --2- 172.21.15.27:0/1663609033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4af80b2390 0x7f4af815d030 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:24.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.399+0000 7f4affe00640 1 --2- 172.21.15.27:0/1663609033 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4af806bb80 0x7f4af80c7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:24.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.399+0000 7f4affe00640 1 -- 172.21.15.27:0/1663609033 >> 172.21.15.27:0/1663609033 conn(0x7f4af80b2a10 msgr2=0x7f4af815a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:24.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.399+0000 7f4affe00640 1 -- 172.21.15.27:0/1663609033 shutdown_connections 2024-01-15T18:13:24.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:24.399+0000 7f4affe00640 1 -- 172.21.15.27:0/1663609033 wait complete. 2024-01-15T18:13:25.421 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone51938 2024-01-15T18:13:25.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.578+0000 7f5a71000640 1 Processor -- start 2024-01-15T18:13:25.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.578+0000 7f5a71000640 1 -- start start 2024-01-15T18:13:25.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.578+0000 7f5a71000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f5a6c15d5c0 0x7f5a6c071c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:25.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.578+0000 7f5a71000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5a6c0cf570 0x7f5a6c072190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:25.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.578+0000 7f5a71000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a6c0b5980 con 0x7f5a6c0b4b10 2024-01-15T18:13:25.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.578+0000 7f5a71000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a6c0ce7b0 con 0x7f5a6c0cf570 2024-01-15T18:13:25.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.578+0000 7f5a71000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a6c0cea90 con 0x7f5a6c15d5c0 2024-01-15T18:13:25.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.578+0000 7f5a69400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5a6c0cf570 0x7f5a6c072190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:25.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.578+0000 7f5a69400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5a6c0cf570 0x7f5a6c072190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40680/0 (socket says 172.21.15.27:40680) 2024-01-15T18:13:25.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.578+0000 7f5a69400640 1 -- 172.21.15.27:0/843424196 learned_addr learned my addr 172.21.15.27:0/843424196 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:25.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.578+0000 7f5a69400640 1 -- 172.21.15.27:0/843424196 >> v2:172.21.15.93:3300/0 conn(0x7f5a6c15d5c0 msgr2=0x7f5a6c071c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:25.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.578+0000 7f5a69e00640 1 --2- 172.21.15.27:0/843424196 >> v2:172.21.15.93:3300/0 conn(0x7f5a6c15d5c0 0x7f5a6c071c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:25.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.578+0000 7f5a69400640 1 --2- 172.21.15.27:0/843424196 >> v2:172.21.15.93:3300/0 conn(0x7f5a6c15d5c0 0x7f5a6c071c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:25.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.578+0000 7f5a69400640 1 -- 172.21.15.27:0/843424196 >> v1:172.21.15.93:6789/0 conn(0x7f5a6c0b4b10 legacy=0x7f5a6c0b4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:25.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.578+0000 7f5a69400640 1 -- 172.21.15.27:0/843424196 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a6c0ced70 con 0x7f5a6c0cf570 2024-01-15T18:13:25.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.578+0000 7f5a69e00640 1 --2- 172.21.15.27:0/843424196 >> v2:172.21.15.93:3300/0 conn(0x7f5a6c15d5c0 0x7f5a6c071c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:13:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.582+0000 7f5a69400640 1 --2- 172.21.15.27:0/843424196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a6c0cf570 0x7f5a6c072190 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c0ce380 tx=0x7f5a5401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bce5765405ea26d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:25.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.582+0000 7f5a68a00640 1 -- 172.21.15.27:0/843424196 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a540ab600 con 0x7f5a6c0cf570 2024-01-15T18:13:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.582+0000 7f5a68a00640 1 -- 172.21.15.27:0/843424196 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a540a7050 con 0x7f5a6c0cf570 2024-01-15T18:13:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.582+0000 7f5a68a00640 1 -- 172.21.15.27:0/843424196 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a540ab980 con 0x7f5a6c0cf570 2024-01-15T18:13:25.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.582+0000 7f5a71000640 1 -- 172.21.15.27:0/843424196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a6c0cf570 msgr2=0x7f5a6c072190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.582+0000 7f5a71000640 1 --2- 172.21.15.27:0/843424196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a6c0cf570 0x7f5a6c072190 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c0ce380 tx=0x7f5a5401c430 comp rx=0 tx=0).stop 2024-01-15T18:13:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.582+0000 7f5a71000640 1 -- 172.21.15.27:0/843424196 shutdown_connections 2024-01-15T18:13:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.582+0000 7f5a71000640 1 --2- 172.21.15.27:0/843424196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a6c0cf570 0x7f5a6c072190 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.582+0000 7f5a71000640 1 --2- 172.21.15.27:0/843424196 >> v2:172.21.15.93:3300/0 conn(0x7f5a6c15d5c0 0x7f5a6c071c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.582+0000 7f5a71000640 1 -- 172.21.15.27:0/843424196 >> 172.21.15.27:0/843424196 conn(0x7f5a6c099e10 msgr2=0x7f5a6c0b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.582+0000 7f5a71000640 1 -- 172.21.15.27:0/843424196 shutdown_connections 2024-01-15T18:13:25.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.582+0000 7f5a71000640 1 -- 172.21.15.27:0/843424196 wait complete. 2024-01-15T18:13:25.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.586+0000 7f5a71000640 1 Processor -- start 2024-01-15T18:13:25.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.586+0000 7f5a71000640 1 -- start start 2024-01-15T18:13:25.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.586+0000 7f5a71000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a6c0b4b10 0x7f5a6c158050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:25.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.586+0000 7f5a71000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a6c0cf570 0x7f5a6c1441c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:25.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.586+0000 7f5a71000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a6c15d5c0 0x7f5a6c06bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:25.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.586+0000 7f5a71000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c158710 con 0x7f5a6c0cf570 2024-01-15T18:13:25.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.586+0000 7f5a71000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c1589f0 con 0x7f5a6c15d5c0 2024-01-15T18:13:25.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.586+0000 7f5a71000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c158cd0 con 0x7f5a6c0b4b10 2024-01-15T18:13:25.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.586+0000 7f5a69400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a6c0cf570 0x7f5a6c1441c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:25.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.586+0000 7f5a69400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a6c0cf570 0x7f5a6c1441c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40694/0 (socket says 172.21.15.27:40694) 2024-01-15T18:13:25.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.586+0000 7f5a69e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a6c0b4b10 0x7f5a6c158050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:25.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.586+0000 7f5a69400640 1 -- 172.21.15.27:0/212280679 learned_addr learned my addr 172.21.15.27:0/212280679 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:25.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.586+0000 7f5a6b200640 1 --2- 172.21.15.27:0/212280679 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a6c15d5c0 0x7f5a6c06bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:25.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.586+0000 7f5a69400640 1 -- 172.21.15.27:0/212280679 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a6c0b4b10 msgr2=0x7f5a6c158050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:25.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.586+0000 7f5a69400640 1 --2- 172.21.15.27:0/212280679 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a6c0b4b10 0x7f5a6c158050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:25.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.586+0000 7f5a69400640 1 -- 172.21.15.27:0/212280679 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a6c15d5c0 msgr2=0x7f5a6c06bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:25.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.586+0000 7f5a69400640 1 --2- 172.21.15.27:0/212280679 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a6c15d5c0 0x7f5a6c06bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:25.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.586+0000 7f5a69400640 1 -- 172.21.15.27:0/212280679 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a540094d0 con 0x7f5a6c0cf570 2024-01-15T18:13:25.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.586+0000 7f5a69400640 1 --2- 172.21.15.27:0/212280679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a6c0cf570 0x7f5a6c1441c0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f5a5401a060 tx=0x7f5a540191d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:25.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.586+0000 7f5a62000640 1 -- 172.21.15.27:0/212280679 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a540b37d0 con 0x7f5a6c0cf570 2024-01-15T18:13:25.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.586+0000 7f5a62000640 1 -- 172.21.15.27:0/212280679 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a540b3930 con 0x7f5a6c0cf570 2024-01-15T18:13:25.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.586+0000 7f5a71000640 1 -- 172.21.15.27:0/212280679 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a6c158fb0 con 0x7f5a6c0cf570 2024-01-15T18:13:25.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.586+0000 7f5a62000640 1 -- 172.21.15.27:0/212280679 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a540b3ce0 con 0x7f5a6c0cf570 2024-01-15T18:13:25.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.586+0000 7f5a71000640 1 -- 172.21.15.27:0/212280679 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a6c1597b0 con 0x7f5a6c0cf570 2024-01-15T18:13:25.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.586+0000 7f5a71000640 1 -- 172.21.15.27:0/212280679 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a6c09e950 con 0x7f5a6c0cf570 2024-01-15T18:13:25.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.590+0000 7f5a62000640 1 -- 172.21.15.27:0/212280679 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f5a540b2070 con 0x7f5a6c0cf570 2024-01-15T18:13:25.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.590+0000 7f5a62000640 1 --2- 172.21.15.27:0/212280679 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5a440673a0 0x7f5a44069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:25.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.590+0000 7f5a62000640 1 -- 172.21.15.27:0/212280679 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f5a5411cf00 con 0x7f5a6c0cf570 2024-01-15T18:13:25.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.590+0000 7f5a69e00640 1 --2- 172.21.15.27:0/212280679 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5a440673a0 0x7f5a44069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:25.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.590+0000 7f5a69e00640 1 --2- 172.21.15.27:0/212280679 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5a440673a0 0x7f5a44069860 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c006fd0 tx=0x7f5a5c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:25.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.594+0000 7f5a62000640 1 -- 172.21.15.27:0/212280679 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5a540ebc10 con 0x7f5a6c0cf570 2024-01-15T18:13:25.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.782+0000 7f5a71000640 1 -- 172.21.15.27:0/212280679 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone51938", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a6c0ae800 con 0x7f5a440673a0 2024-01-15T18:13:25.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.854+0000 7f5a62000640 1 -- 172.21.15.27:0/212280679 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5a6c0ae800 con 0x7f5a440673a0 2024-01-15T18:13:25.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.854+0000 7f5a71000640 1 -- 172.21.15.27:0/212280679 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5a440673a0 msgr2=0x7f5a44069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:25.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.854+0000 7f5a71000640 1 --2- 172.21.15.27:0/212280679 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5a440673a0 0x7f5a44069860 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c006fd0 tx=0x7f5a5c008040 comp rx=0 tx=0).stop 2024-01-15T18:13:25.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.854+0000 7f5a71000640 1 -- 172.21.15.27:0/212280679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a6c0cf570 msgr2=0x7f5a6c1441c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:25.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.854+0000 7f5a71000640 1 --2- 172.21.15.27:0/212280679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a6c0cf570 0x7f5a6c1441c0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f5a5401a060 tx=0x7f5a540191d0 comp rx=0 tx=0).stop 2024-01-15T18:13:25.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.858+0000 7f5a71000640 1 -- 172.21.15.27:0/212280679 shutdown_connections 2024-01-15T18:13:25.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.858+0000 7f5a71000640 1 --2- 172.21.15.27:0/212280679 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5a440673a0 0x7f5a44069860 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:25.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.858+0000 7f5a71000640 1 --2- 172.21.15.27:0/212280679 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a6c15d5c0 0x7f5a6c06bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:25.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.858+0000 7f5a71000640 1 --2- 172.21.15.27:0/212280679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a6c0cf570 0x7f5a6c1441c0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:25.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.858+0000 7f5a71000640 1 --2- 172.21.15.27:0/212280679 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a6c0b4b10 0x7f5a6c158050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:25.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.858+0000 7f5a71000640 1 -- 172.21.15.27:0/212280679 >> 172.21.15.27:0/212280679 conn(0x7f5a6c099e10 msgr2=0x7f5a6c0b22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:25.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.858+0000 7f5a71000640 1 -- 172.21.15.27:0/212280679 shutdown_connections 2024-01-15T18:13:25.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:25.858+0000 7f5a71000640 1 -- 172.21.15.27:0/212280679 wait complete. 2024-01-15T18:13:25.878 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone51939 2024-01-15T18:13:26.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.022+0000 7f5221200640 1 Processor -- start 2024-01-15T18:13:26.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.022+0000 7f5221200640 1 -- start start 2024-01-15T18:13:26.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.022+0000 7f5221200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f521c0d1a40 0x7f521c0cba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:26.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.022+0000 7f5221200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521c0cbfa0 con 0x7f521c0b0650 2024-01-15T18:13:26.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.022+0000 7f5221200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521c0cefa0 con 0x7f521c0cf450 2024-01-15T18:13:26.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.022+0000 7f5221200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f521c0ce240 con 0x7f521c0d1a40 2024-01-15T18:13:26.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.022+0000 7f521a800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f521c0b0650 0x7f521c0b0ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53878/0 (socket says 172.21.15.27:53878) 2024-01-15T18:13:26.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.022+0000 7f521a800640 1 -- 172.21.15.27:0/2924194863 learned_addr learned my addr 172.21.15.27:0/2924194863 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:13:26.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.022+0000 7f5219400640 1 --2- 172.21.15.27:0/2924194863 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f521c0d1a40 0x7f521c0cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:26.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.026+0000 7f5218a00640 1 -- 172.21.15.27:0/2924194863 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638332840 0 0) 0x7f521c0cbfa0 con 0x7f521c0b0650 2024-01-15T18:13:26.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.026+0000 7f5218a00640 1 -- 172.21.15.27:0/2924194863 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51fc003530 con 0x7f521c0b0650 2024-01-15T18:13:26.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.026+0000 7f5218a00640 1 -- 172.21.15.27:0/2924194863 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 502177583 0 0) 0x7f51fc003530 con 0x7f521c0b0650 2024-01-15T18:13:26.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.026+0000 7f5218a00640 1 -- 172.21.15.27:0/2924194863 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f521c0d1a40 msgr2=0x7f521c0cba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:26.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.026+0000 7f5218a00640 1 --2- 172.21.15.27:0/2924194863 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f521c0d1a40 0x7f521c0cba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:26.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.026+0000 7f5218a00640 1 -- 172.21.15.27:0/2924194863 >> v1:172.21.15.93:6789/0 conn(0x7f521c0cf450 legacy=0x7f521c0cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:26.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.026+0000 7f5218a00640 1 -- 172.21.15.27:0/2924194863 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521c0ccd70 con 0x7f521c0b0650 2024-01-15T18:13:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.026+0000 7f5218a00640 1 -- 172.21.15.27:0/2924194863 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f5214001a00 con 0x7f521c0b0650 2024-01-15T18:13:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.026+0000 7f5218a00640 1 -- 172.21.15.27:0/2924194863 >> v1:172.21.15.27:6789/0 conn(0x7f521c0b0650 legacy=0x7f521c0b0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.026+0000 7f5218a00640 1 --2- 172.21.15.27:0/2924194863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51fc003a20 0x7f51fc003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.026+0000 7f5218a00640 1 --2- 172.21.15.27:0/2924194863 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f521c0d1a40 0x7f51fc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.026+0000 7f5218a00640 1 --2- 172.21.15.27:0/2924194863 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f51fc004d10 0x7f51fc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.026+0000 7f5218a00640 1 -- 172.21.15.27:0/2924194863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521c0ce240 con 0x7f51fc003a20 2024-01-15T18:13:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.026+0000 7f5218a00640 1 -- 172.21.15.27:0/2924194863 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51fc00bad0 con 0x7f51fc004d10 2024-01-15T18:13:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.026+0000 7f5218a00640 1 -- 172.21.15.27:0/2924194863 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51fc00bc90 con 0x7f521c0d1a40 2024-01-15T18:13:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.026+0000 7f5219e00640 1 --2- 172.21.15.27:0/2924194863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51fc003a20 0x7f51fc003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.026+0000 7f5219400640 1 --2- 172.21.15.27:0/2924194863 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f521c0d1a40 0x7f51fc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.026+0000 7f521a800640 1 --2- 172.21.15.27:0/2924194863 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f51fc004d10 0x7f51fc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.026+0000 7f5219e00640 1 -- 172.21.15.27:0/2924194863 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f521c0d1a40 msgr2=0x7f51fc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.026+0000 7f5219e00640 1 --2- 172.21.15.27:0/2924194863 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f521c0d1a40 0x7f51fc0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:26.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.026+0000 7f5219e00640 1 -- 172.21.15.27:0/2924194863 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f51fc004d10 msgr2=0x7f51fc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:26.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.026+0000 7f5219e00640 1 --2- 172.21.15.27:0/2924194863 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f51fc004d10 0x7f51fc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:26.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.026+0000 7f5219e00640 1 -- 172.21.15.27:0/2924194863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51fc003530 con 0x7f51fc003a20 2024-01-15T18:13:26.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.026+0000 7f5219e00640 1 --2- 172.21.15.27:0/2924194863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51fc003a20 0x7f51fc003e40 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f520c001750 tx=0x7f520c01c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bddb771c2f71da9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:26.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.030+0000 7f5218a00640 1 -- 172.21.15.27:0/2924194863 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c0052d0 con 0x7f51fc003a20 2024-01-15T18:13:26.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.030+0000 7f5218a00640 1 -- 172.21.15.27:0/2924194863 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520c0a7050 con 0x7f51fc003a20 2024-01-15T18:13:26.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.030+0000 7f5218a00640 1 -- 172.21.15.27:0/2924194863 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c003600 con 0x7f51fc003a20 2024-01-15T18:13:26.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.030+0000 7f5221200640 1 -- 172.21.15.27:0/2924194863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51fc003a20 msgr2=0x7f51fc003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:26.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.030+0000 7f5221200640 1 --2- 172.21.15.27:0/2924194863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51fc003a20 0x7f51fc003e40 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f520c001750 tx=0x7f520c01c300 comp rx=0 tx=0).stop 2024-01-15T18:13:26.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.030+0000 7f521a800640 1 -- 172.21.15.27:0/2924194863 reap_dead start 2024-01-15T18:13:26.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.030+0000 7f5221200640 1 -- 172.21.15.27:0/2924194863 shutdown_connections 2024-01-15T18:13:26.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.030+0000 7f5221200640 1 -- 172.21.15.27:0/2924194863 >> 172.21.15.27:0/2924194863 conn(0x7f521c0d0e70 msgr2=0x7f521c0d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:26.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.030+0000 7f5221200640 1 -- 172.21.15.27:0/2924194863 shutdown_connections 2024-01-15T18:13:26.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.030+0000 7f5221200640 1 -- 172.21.15.27:0/2924194863 wait complete. 2024-01-15T18:13:26.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.030+0000 7f5221200640 1 Processor -- start 2024-01-15T18:13:26.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.030+0000 7f5221200640 1 -- start start 2024-01-15T18:13:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.034+0000 7f5221200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51fc003a20 0x7f521c0ca840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.034+0000 7f5221200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f521c0aefd0 0x7f521c0af450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.034+0000 7f5221200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f521c0b5df0 0x7f521c0b6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.034+0000 7f5221200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521c0c7f40 con 0x7f51fc003a20 2024-01-15T18:13:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.034+0000 7f5221200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521c0af990 con 0x7f521c0aefd0 2024-01-15T18:13:26.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.034+0000 7f5221200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f521c0c8a50 con 0x7f521c0b5df0 2024-01-15T18:13:26.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.034+0000 7f5219e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51fc003a20 0x7f521c0ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:26.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.034+0000 7f5219e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51fc003a20 0x7f521c0ca840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40706/0 (socket says 172.21.15.27:40706) 2024-01-15T18:13:26.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.034+0000 7f5219e00640 1 -- 172.21.15.27:0/2853874333 learned_addr learned my addr 172.21.15.27:0/2853874333 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:26.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.034+0000 7f5219400640 1 --2- 172.21.15.27:0/2853874333 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f521c0aefd0 0x7f521c0af450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:26.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.034+0000 7f521a800640 1 --2- 172.21.15.27:0/2853874333 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f521c0b5df0 0x7f521c0b6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:26.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.034+0000 7f5219e00640 1 -- 172.21.15.27:0/2853874333 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f521c0b5df0 msgr2=0x7f521c0b6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:26.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.034+0000 7f5219e00640 1 --2- 172.21.15.27:0/2853874333 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f521c0b5df0 0x7f521c0b6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:26.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.034+0000 7f5219e00640 1 -- 172.21.15.27:0/2853874333 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f521c0aefd0 msgr2=0x7f521c0af450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:26.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.034+0000 7f5219e00640 1 --2- 172.21.15.27:0/2853874333 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f521c0aefd0 0x7f521c0af450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:26.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.034+0000 7f5219e00640 1 -- 172.21.15.27:0/2853874333 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521c0c8120 con 0x7f51fc003a20 2024-01-15T18:13:26.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.034+0000 7f521a800640 1 --2- 172.21.15.27:0/2853874333 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f521c0b5df0 0x7f521c0b6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:13:26.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.034+0000 7f5219e00640 1 --2- 172.21.15.27:0/2853874333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51fc003a20 0x7f521c0ca840 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f520c005040 tx=0x7f51fc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:26.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.034+0000 7f5212000640 1 -- 172.21.15.27:0/2853874333 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c0070c0 con 0x7f51fc003a20 2024-01-15T18:13:26.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.034+0000 7f5212000640 1 -- 172.21.15.27:0/2853874333 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520c007220 con 0x7f51fc003a20 2024-01-15T18:13:26.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.034+0000 7f5212000640 1 -- 172.21.15.27:0/2853874333 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c0075d0 con 0x7f51fc003a20 2024-01-15T18:13:26.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.034+0000 7f5221200640 1 -- 172.21.15.27:0/2853874333 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f521c0c8400 con 0x7f51fc003a20 2024-01-15T18:13:26.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.034+0000 7f5221200640 1 -- 172.21.15.27:0/2853874333 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f521c0cad80 con 0x7f51fc003a20 2024-01-15T18:13:26.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.034+0000 7f5221200640 1 -- 172.21.15.27:0/2853874333 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f521c04e850 con 0x7f51fc003a20 2024-01-15T18:13:26.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.034+0000 7f5212000640 1 -- 172.21.15.27:0/2853874333 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f520c0067e0 con 0x7f51fc003a20 2024-01-15T18:13:26.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.038+0000 7f5212000640 1 --2- 172.21.15.27:0/2853874333 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f51f40673a0 0x7f51f4069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:26.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.038+0000 7f5212000640 1 -- 172.21.15.27:0/2853874333 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f520c11cf70 con 0x7f51fc003a20 2024-01-15T18:13:26.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.038+0000 7f5219400640 1 --2- 172.21.15.27:0/2853874333 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f51f40673a0 0x7f51f4069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:26.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.038+0000 7f5219400640 1 --2- 172.21.15.27:0/2853874333 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f51f40673a0 0x7f51f4069860 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f521c15e080 tx=0x7f5204009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:26.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.042+0000 7f5212000640 1 -- 172.21.15.27:0/2853874333 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f520c0ebc60 con 0x7f51fc003a20 2024-01-15T18:13:26.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.226+0000 7f5221200640 1 -- 172.21.15.27:0/2853874333 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone51939", "target": ["mon-mgr", ""]}) v1 -- 0x7f521c0cca90 con 0x7f51f40673a0 2024-01-15T18:13:26.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.290+0000 7f5212000640 1 -- 172.21.15.27:0/2853874333 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5204019070 con 0x7f51f40673a0 2024-01-15T18:13:26.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.290+0000 7f5221200640 1 -- 172.21.15.27:0/2853874333 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f51f40673a0 msgr2=0x7f51f4069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:26.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.290+0000 7f5221200640 1 --2- 172.21.15.27:0/2853874333 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f51f40673a0 0x7f51f4069860 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f521c15e080 tx=0x7f5204009190 comp rx=0 tx=0).stop 2024-01-15T18:13:26.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.290+0000 7f5221200640 1 -- 172.21.15.27:0/2853874333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51fc003a20 msgr2=0x7f521c0ca840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:26.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.290+0000 7f5221200640 1 --2- 172.21.15.27:0/2853874333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51fc003a20 0x7f521c0ca840 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f520c005040 tx=0x7f51fc005800 comp rx=0 tx=0).stop 2024-01-15T18:13:26.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.290+0000 7f5221200640 1 -- 172.21.15.27:0/2853874333 shutdown_connections 2024-01-15T18:13:26.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.290+0000 7f5221200640 1 --2- 172.21.15.27:0/2853874333 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f51f40673a0 0x7f51f4069860 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:26.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.290+0000 7f5221200640 1 --2- 172.21.15.27:0/2853874333 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f521c0b5df0 0x7f521c0b6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:26.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.290+0000 7f5221200640 1 --2- 172.21.15.27:0/2853874333 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f521c0aefd0 0x7f521c0af450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:26.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.290+0000 7f5221200640 1 --2- 172.21.15.27:0/2853874333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51fc003a20 0x7f521c0ca840 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:26.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.290+0000 7f5221200640 1 -- 172.21.15.27:0/2853874333 >> 172.21.15.27:0/2853874333 conn(0x7f521c0d0e70 msgr2=0x7f521c0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:26.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.290+0000 7f5221200640 1 -- 172.21.15.27:0/2853874333 shutdown_connections 2024-01-15T18:13:26.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.290+0000 7f5221200640 1 -- 172.21.15.27:0/2853874333 wait complete. 2024-01-15T18:13:26.313 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone519310 2024-01-15T18:13:26.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.458+0000 7f1f17800640 1 Processor -- start 2024-01-15T18:13:26.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.458+0000 7f1f17800640 1 -- start start 2024-01-15T18:13:26.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f17800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f1f100cba60 0x7f1f100b0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:26.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f17800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f100cefa0 con 0x7f1f100cf450 2024-01-15T18:13:26.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f17800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f100ce240 con 0x7f1f100d1a40 2024-01-15T18:13:26.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f17800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f100ccd70 con 0x7f1f100cba60 2024-01-15T18:13:26.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f14e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1f100cf450 0x7f1f100cf830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53888/0 (socket says 172.21.15.27:53888) 2024-01-15T18:13:26.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f14e00640 1 -- 172.21.15.27:0/1748596212 learned_addr learned my addr 172.21.15.27:0/1748596212 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:13:26.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f15800640 1 --2- 172.21.15.27:0/1748596212 >> v2:172.21.15.93:3300/0 conn(0x7f1f100cba60 0x7f1f100b0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:26.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f0f400640 1 -- 172.21.15.27:0/1748596212 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2884783750 0 0) 0x7f1f100cefa0 con 0x7f1f100cf450 2024-01-15T18:13:26.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f0f400640 1 -- 172.21.15.27:0/1748596212 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ef0003530 con 0x7f1f100cf450 2024-01-15T18:13:26.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f0f400640 1 -- 172.21.15.27:0/1748596212 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1503299285 0 0) 0x7f1ef0003530 con 0x7f1f100cf450 2024-01-15T18:13:26.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f0f400640 1 -- 172.21.15.27:0/1748596212 >> v2:172.21.15.93:3300/0 conn(0x7f1f100cba60 msgr2=0x7f1f100b0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:26.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f0f400640 1 --2- 172.21.15.27:0/1748596212 >> v2:172.21.15.93:3300/0 conn(0x7f1f100cba60 0x7f1f100b0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:26.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f0f400640 1 -- 172.21.15.27:0/1748596212 >> v1:172.21.15.93:6789/0 conn(0x7f1f100d1a40 legacy=0x7f1f100b0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:26.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f0f400640 1 -- 172.21.15.27:0/1748596212 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f100cc1f0 con 0x7f1f100cf450 2024-01-15T18:13:26.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f0f400640 1 -- 172.21.15.27:0/1748596212 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f1f00001680 con 0x7f1f100cf450 2024-01-15T18:13:26.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f0f400640 1 -- 172.21.15.27:0/1748596212 >> v1:172.21.15.27:6789/0 conn(0x7f1f100cf450 legacy=0x7f1f100cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:26.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f0f400640 1 --2- 172.21.15.27:0/1748596212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ef0003a40 0x7f1ef0003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:26.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f0f400640 1 --2- 172.21.15.27:0/1748596212 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ef00047d0 0x7f1ef0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:26.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f0f400640 1 --2- 172.21.15.27:0/1748596212 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ef00051e0 0x7f1ef000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:26.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f0f400640 1 -- 172.21.15.27:0/1748596212 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f100ccd70 con 0x7f1ef0003a40 2024-01-15T18:13:26.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f0f400640 1 -- 172.21.15.27:0/1748596212 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef000c110 con 0x7f1ef00051e0 2024-01-15T18:13:26.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f0f400640 1 -- 172.21.15.27:0/1748596212 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef000c2d0 con 0x7f1ef00047d0 2024-01-15T18:13:26.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f14e00640 1 --2- 172.21.15.27:0/1748596212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ef0003a40 0x7f1ef0003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:26.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f0fe00640 1 --2- 172.21.15.27:0/1748596212 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ef00047d0 0x7f1ef0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:26.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f15800640 1 --2- 172.21.15.27:0/1748596212 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ef00051e0 0x7f1ef000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:26.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f14e00640 1 -- 172.21.15.27:0/1748596212 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ef00047d0 msgr2=0x7f1ef0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:26.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f14e00640 1 --2- 172.21.15.27:0/1748596212 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ef00047d0 0x7f1ef0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:26.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f14e00640 1 -- 172.21.15.27:0/1748596212 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ef00051e0 msgr2=0x7f1ef000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:26.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f14e00640 1 --2- 172.21.15.27:0/1748596212 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ef00051e0 0x7f1ef000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:26.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f14e00640 1 -- 172.21.15.27:0/1748596212 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef0003530 con 0x7f1ef0003a40 2024-01-15T18:13:26.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.462+0000 7f1f15800640 1 -- 172.21.15.27:0/1748596212 reap_dead start 2024-01-15T18:13:26.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.466+0000 7f1f14e00640 1 --2- 172.21.15.27:0/1748596212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ef0003a40 0x7f1ef0003e60 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f1f0000c0e0 tx=0x7f1f0001cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc3f633c5a1d7346 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:26.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.466+0000 7f1f0f400640 1 -- 172.21.15.27:0/1748596212 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f00005ae0 con 0x7f1ef0003a40 2024-01-15T18:13:26.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.466+0000 7f1f0f400640 1 -- 172.21.15.27:0/1748596212 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f00005c40 con 0x7f1ef0003a40 2024-01-15T18:13:26.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.466+0000 7f1f0f400640 1 -- 172.21.15.27:0/1748596212 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f000072b0 con 0x7f1ef0003a40 2024-01-15T18:13:26.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.466+0000 7f1f17800640 1 -- 172.21.15.27:0/1748596212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ef0003a40 msgr2=0x7f1ef0003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:26.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.466+0000 7f1f17800640 1 --2- 172.21.15.27:0/1748596212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ef0003a40 0x7f1ef0003e60 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f1f0000c0e0 tx=0x7f1f0001cae0 comp rx=0 tx=0).stop 2024-01-15T18:13:26.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.466+0000 7f1f17800640 1 -- 172.21.15.27:0/1748596212 shutdown_connections 2024-01-15T18:13:26.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.466+0000 7f1f17800640 1 --2- 172.21.15.27:0/1748596212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ef0003a40 0x7f1ef0003e60 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:26.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.466+0000 7f1f17800640 1 -- 172.21.15.27:0/1748596212 >> 172.21.15.27:0/1748596212 conn(0x7f1f100d0e70 msgr2=0x7f1f100d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:26.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.466+0000 7f1f17800640 1 -- 172.21.15.27:0/1748596212 shutdown_connections 2024-01-15T18:13:26.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.466+0000 7f1f17800640 1 -- 172.21.15.27:0/1748596212 wait complete. 2024-01-15T18:13:26.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.470+0000 7f1f17800640 1 Processor -- start 2024-01-15T18:13:26.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.470+0000 7f1f17800640 1 -- start start 2024-01-15T18:13:26.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.470+0000 7f1f17800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ef0003a40 0x7f1f100ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:26.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.470+0000 7f1f17800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f100c7550 0x7f1f100c4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:26.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.470+0000 7f1f17800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1f100c79b0 0x7f1f100b4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:26.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.470+0000 7f1f17800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f100c9400 con 0x7f1f100c7550 2024-01-15T18:13:26.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.470+0000 7f1f17800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f100c96e0 con 0x7f1ef0003a40 2024-01-15T18:13:26.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.470+0000 7f1f17800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f100c99c0 con 0x7f1f100c79b0 2024-01-15T18:13:26.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.470+0000 7f1f0fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f100c7550 0x7f1f100c4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:26.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.470+0000 7f1f0fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f100c7550 0x7f1f100c4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40734/0 (socket says 172.21.15.27:40734) 2024-01-15T18:13:26.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.470+0000 7f1f0fe00640 1 -- 172.21.15.27:0/1912245425 learned_addr learned my addr 172.21.15.27:0/1912245425 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:26.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.470+0000 7f1f14e00640 1 --2- 172.21.15.27:0/1912245425 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ef0003a40 0x7f1f100ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:26.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.470+0000 7f1f15800640 1 --2- 172.21.15.27:0/1912245425 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1f100c79b0 0x7f1f100b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:26.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.470+0000 7f1f0fe00640 1 -- 172.21.15.27:0/1912245425 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1f100c79b0 msgr2=0x7f1f100b4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:26.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.470+0000 7f1f0fe00640 1 --2- 172.21.15.27:0/1912245425 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1f100c79b0 0x7f1f100b4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:26.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.470+0000 7f1f0fe00640 1 -- 172.21.15.27:0/1912245425 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ef0003a40 msgr2=0x7f1f100ca4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:26.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.470+0000 7f1f0fe00640 1 --2- 172.21.15.27:0/1912245425 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ef0003a40 0x7f1f100ca4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:26.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.470+0000 7f1f0fe00640 1 -- 172.21.15.27:0/1912245425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f100cbfc0 con 0x7f1f100c7550 2024-01-15T18:13:26.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.470+0000 7f1f15800640 1 --2- 172.21.15.27:0/1912245425 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1f100c79b0 0x7f1f100b4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:13:26.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.470+0000 7f1f0fe00640 1 --2- 172.21.15.27:0/1912245425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f100c7550 0x7f1f100c4c60 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f1ef800f770 tx=0x7f1ef800fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:26.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.470+0000 7f1f0cc00640 1 -- 172.21.15.27:0/1912245425 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8001350 con 0x7f1f100c7550 2024-01-15T18:13:26.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.470+0000 7f1f0cc00640 1 -- 172.21.15.27:0/1912245425 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef80050d0 con 0x7f1f100c7550 2024-01-15T18:13:26.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.470+0000 7f1f17800640 1 -- 172.21.15.27:0/1912245425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f100c9ca0 con 0x7f1f100c7550 2024-01-15T18:13:26.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.470+0000 7f1f0cc00640 1 -- 172.21.15.27:0/1912245425 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8005480 con 0x7f1f100c7550 2024-01-15T18:13:26.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.470+0000 7f1f17800640 1 -- 172.21.15.27:0/1912245425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f100c51a0 con 0x7f1f100c7550 2024-01-15T18:13:26.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.474+0000 7f1f17800640 1 -- 172.21.15.27:0/1912245425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f1004e850 con 0x7f1f100c7550 2024-01-15T18:13:26.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.474+0000 7f1f0cc00640 1 -- 172.21.15.27:0/1912245425 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f1ef800d730 con 0x7f1f100c7550 2024-01-15T18:13:26.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.474+0000 7f1f0cc00640 1 --2- 172.21.15.27:0/1912245425 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1ee80673a0 0x7f1ee8069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:26.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.474+0000 7f1f0cc00640 1 -- 172.21.15.27:0/1912245425 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f1ef8082c70 con 0x7f1f100c7550 2024-01-15T18:13:26.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.474+0000 7f1f14e00640 1 --2- 172.21.15.27:0/1912245425 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1ee80673a0 0x7f1ee8069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:26.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.474+0000 7f1f14e00640 1 --2- 172.21.15.27:0/1912245425 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1ee80673a0 0x7f1ee8069860 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f1f000ab000 tx=0x7f1f000b1040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:26.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.478+0000 7f1f0cc00640 1 -- 172.21.15.27:0/1912245425 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1ef8051870 con 0x7f1f100c7550 2024-01-15T18:13:26.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.666+0000 7f1f17800640 1 -- 172.21.15.27:0/1912245425 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone519310", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f1006aa50 con 0x7f1ee80673a0 2024-01-15T18:13:26.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.730+0000 7f1f0cc00640 1 -- 172.21.15.27:0/1912245425 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1f1006aa50 con 0x7f1ee80673a0 2024-01-15T18:13:26.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.730+0000 7f1f17800640 1 -- 172.21.15.27:0/1912245425 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1ee80673a0 msgr2=0x7f1ee8069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:26.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.730+0000 7f1f17800640 1 --2- 172.21.15.27:0/1912245425 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1ee80673a0 0x7f1ee8069860 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f1f000ab000 tx=0x7f1f000b1040 comp rx=0 tx=0).stop 2024-01-15T18:13:26.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.730+0000 7f1f17800640 1 -- 172.21.15.27:0/1912245425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f100c7550 msgr2=0x7f1f100c4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:26.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.730+0000 7f1f17800640 1 --2- 172.21.15.27:0/1912245425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f100c7550 0x7f1f100c4c60 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f1ef800f770 tx=0x7f1ef800fc30 comp rx=0 tx=0).stop 2024-01-15T18:13:26.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.730+0000 7f1f17800640 1 -- 172.21.15.27:0/1912245425 shutdown_connections 2024-01-15T18:13:26.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.730+0000 7f1f17800640 1 --2- 172.21.15.27:0/1912245425 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1ee80673a0 0x7f1ee8069860 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:26.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.730+0000 7f1f17800640 1 --2- 172.21.15.27:0/1912245425 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1f100c79b0 0x7f1f100b4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:26.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.730+0000 7f1f17800640 1 --2- 172.21.15.27:0/1912245425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f100c7550 0x7f1f100c4c60 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:26.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.730+0000 7f1f17800640 1 --2- 172.21.15.27:0/1912245425 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ef0003a40 0x7f1f100ca4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:26.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.730+0000 7f1f17800640 1 -- 172.21.15.27:0/1912245425 >> 172.21.15.27:0/1912245425 conn(0x7f1f100d0e70 msgr2=0x7f1f100afaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:26.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.730+0000 7f1f17800640 1 -- 172.21.15.27:0/1912245425 shutdown_connections 2024-01-15T18:13:26.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.730+0000 7f1f17800640 1 -- 172.21.15.27:0/1912245425 wait complete. 2024-01-15T18:13:26.754 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone519311 2024-01-15T18:13:26.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.898+0000 7f8830000640 1 Processor -- start 2024-01-15T18:13:26.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.898+0000 7f8830000640 1 -- start start 2024-01-15T18:13:26.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.902+0000 7f8830000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f882815f550 0x7f88280cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:26.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.902+0000 7f8830000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88280b1cb0 con 0x7f8828158900 2024-01-15T18:13:26.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.902+0000 7f8830000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f882815fa00 con 0x7f88280cf110 2024-01-15T18:13:26.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.902+0000 7f8830000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88280cf580 con 0x7f882815f550 2024-01-15T18:13:26.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.902+0000 7f882d600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8828158900 0x7f882815bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53904/0 (socket says 172.21.15.27:53904) 2024-01-15T18:13:26.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.902+0000 7f882d600640 1 -- 172.21.15.27:0/4149424071 learned_addr learned my addr 172.21.15.27:0/4149424071 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:13:26.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.902+0000 7f882cc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f882815f550 0x7f88280cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:26.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.902+0000 7f882cc00640 1 -- 172.21.15.27:0/4149424071 >> v1:172.21.15.93:6789/0 conn(0x7f88280cf110 legacy=0x7f88280701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:26.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.902+0000 7f882cc00640 1 -- 172.21.15.27:0/4149424071 >> v1:172.21.15.27:6789/0 conn(0x7f8828158900 legacy=0x7f882815bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:26.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.902+0000 7f882cc00640 1 -- 172.21.15.27:0/4149424071 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88280b1f00 con 0x7f882815f550 2024-01-15T18:13:26.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.902+0000 7f8827e00640 1 -- 172.21.15.27:0/4149424071 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1219263615 0 0) 0x7f88280b1cb0 con 0x7f8828158900 2024-01-15T18:13:26.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.906+0000 7f882cc00640 1 --2- 172.21.15.27:0/4149424071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f882815f550 0x7f88280cebd0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f88100097f0 tx=0x7f881001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e5d2d66152b0457 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:26.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.906+0000 7f8827e00640 1 -- 172.21.15.27:0/4149424071 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88100ab610 con 0x7f882815f550 2024-01-15T18:13:26.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.906+0000 7f8827e00640 1 -- 172.21.15.27:0/4149424071 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88100ab770 con 0x7f882815f550 2024-01-15T18:13:26.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.906+0000 7f8827e00640 1 -- 172.21.15.27:0/4149424071 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88100abb00 con 0x7f882815f550 2024-01-15T18:13:26.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.906+0000 7f8830000640 1 -- 172.21.15.27:0/4149424071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f882815f550 msgr2=0x7f88280cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:26.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.906+0000 7f8830000640 1 --2- 172.21.15.27:0/4149424071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f882815f550 0x7f88280cebd0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f88100097f0 tx=0x7f881001c410 comp rx=0 tx=0).stop 2024-01-15T18:13:26.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.906+0000 7f8830000640 1 -- 172.21.15.27:0/4149424071 shutdown_connections 2024-01-15T18:13:26.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.906+0000 7f8830000640 1 --2- 172.21.15.27:0/4149424071 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f882815f550 0x7f88280cebd0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:26.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.906+0000 7f8830000640 1 -- 172.21.15.27:0/4149424071 >> 172.21.15.27:0/4149424071 conn(0x7f88280cb700 msgr2=0x7f88280cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:26.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.906+0000 7f8830000640 1 -- 172.21.15.27:0/4149424071 shutdown_connections 2024-01-15T18:13:26.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.906+0000 7f8830000640 1 -- 172.21.15.27:0/4149424071 wait complete. 2024-01-15T18:13:26.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.906+0000 7f8830000640 1 Processor -- start 2024-01-15T18:13:26.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.906+0000 7f8830000640 1 -- start start 2024-01-15T18:13:26.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.910+0000 7f8830000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88280cf110 0x7f8828154fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:26.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.910+0000 7f8830000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8828158900 0x7f882815ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:26.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.910+0000 7f8830000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f882815f550 0x7f8828151070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:26.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.910+0000 7f8830000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882815a700 con 0x7f882815f550 2024-01-15T18:13:26.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.910+0000 7f8830000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8828154500 con 0x7f88280cf110 2024-01-15T18:13:26.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.910+0000 7f8830000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88281547e0 con 0x7f8828158900 2024-01-15T18:13:26.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.910+0000 7f882e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f882815f550 0x7f8828151070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:26.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.910+0000 7f882e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f882815f550 0x7f8828151070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40752/0 (socket says 172.21.15.27:40752) 2024-01-15T18:13:26.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.910+0000 7f882e000640 1 -- 172.21.15.27:0/1675301153 learned_addr learned my addr 172.21.15.27:0/1675301153 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:26.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.910+0000 7f882d600640 1 --2- 172.21.15.27:0/1675301153 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88280cf110 0x7f8828154fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:26.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.910+0000 7f882cc00640 1 --2- 172.21.15.27:0/1675301153 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8828158900 0x7f882815ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:26.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.910+0000 7f882e000640 1 -- 172.21.15.27:0/1675301153 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8828158900 msgr2=0x7f882815ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:26.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.910+0000 7f882e000640 1 --2- 172.21.15.27:0/1675301153 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8828158900 0x7f882815ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:26.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.910+0000 7f882e000640 1 -- 172.21.15.27:0/1675301153 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88280cf110 msgr2=0x7f8828154fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:26.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.910+0000 7f882e000640 1 --2- 172.21.15.27:0/1675301153 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88280cf110 0x7f8828154fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:26.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.910+0000 7f882e000640 1 -- 172.21.15.27:0/1675301153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8810009420 con 0x7f882815f550 2024-01-15T18:13:26.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.910+0000 7f882e000640 1 --2- 172.21.15.27:0/1675301153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f882815f550 0x7f8828151070 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f882000c780 tx=0x7f882000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:26.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.910+0000 7f8825600640 1 -- 172.21.15.27:0/1675301153 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8820004b70 con 0x7f882815f550 2024-01-15T18:13:26.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.910+0000 7f8825600640 1 -- 172.21.15.27:0/1675301153 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f882000ce90 con 0x7f882815f550 2024-01-15T18:13:26.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.910+0000 7f8830000640 1 -- 172.21.15.27:0/1675301153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8828154ac0 con 0x7f882815f550 2024-01-15T18:13:26.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.910+0000 7f8825600640 1 -- 172.21.15.27:0/1675301153 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882000a850 con 0x7f882815f550 2024-01-15T18:13:26.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.910+0000 7f8830000640 1 -- 172.21.15.27:0/1675301153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f882815b0d0 con 0x7f882815f550 2024-01-15T18:13:26.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.910+0000 7f8830000640 1 -- 172.21.15.27:0/1675301153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8828154cb0 con 0x7f882815f550 2024-01-15T18:13:26.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.910+0000 7f8825600640 1 -- 172.21.15.27:0/1675301153 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f882000a9b0 con 0x7f882815f550 2024-01-15T18:13:26.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.914+0000 7f8825600640 1 --2- 172.21.15.27:0/1675301153 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88000673a0 0x7f8800069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:26.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.914+0000 7f8825600640 1 -- 172.21.15.27:0/1675301153 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f8820083150 con 0x7f882815f550 2024-01-15T18:13:26.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.914+0000 7f882d600640 1 --2- 172.21.15.27:0/1675301153 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88000673a0 0x7f8800069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:26.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.914+0000 7f882d600640 1 --2- 172.21.15.27:0/1675301153 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88000673a0 0x7f8800069860 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8818000f70 tx=0x7f8818007b80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:26.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:26.918+0000 7f8825600640 1 -- 172.21.15.27:0/1675301153 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8820051bb0 con 0x7f882815f550 2024-01-15T18:13:27.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.106+0000 7f8830000640 1 -- 172.21.15.27:0/1675301153 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone519311", "target": ["mon-mgr", ""]}) v1 -- 0x7f88280cf990 con 0x7f88000673a0 2024-01-15T18:13:27.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.170+0000 7f8825600640 1 -- 172.21.15.27:0/1675301153 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f88280cf990 con 0x7f88000673a0 2024-01-15T18:13:27.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.174+0000 7f8830000640 1 -- 172.21.15.27:0/1675301153 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88000673a0 msgr2=0x7f8800069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:27.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.174+0000 7f8830000640 1 --2- 172.21.15.27:0/1675301153 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88000673a0 0x7f8800069860 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8818000f70 tx=0x7f8818007b80 comp rx=0 tx=0).stop 2024-01-15T18:13:27.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.174+0000 7f8830000640 1 -- 172.21.15.27:0/1675301153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f882815f550 msgr2=0x7f8828151070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:27.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.174+0000 7f8830000640 1 --2- 172.21.15.27:0/1675301153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f882815f550 0x7f8828151070 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f882000c780 tx=0x7f882000cc40 comp rx=0 tx=0).stop 2024-01-15T18:13:27.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.174+0000 7f8830000640 1 -- 172.21.15.27:0/1675301153 shutdown_connections 2024-01-15T18:13:27.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.174+0000 7f8830000640 1 --2- 172.21.15.27:0/1675301153 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88000673a0 0x7f8800069860 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:27.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.174+0000 7f8830000640 1 --2- 172.21.15.27:0/1675301153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f882815f550 0x7f8828151070 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:27.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.174+0000 7f8830000640 1 --2- 172.21.15.27:0/1675301153 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8828158900 0x7f882815ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:27.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.174+0000 7f8830000640 1 --2- 172.21.15.27:0/1675301153 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88280cf110 0x7f8828154fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:27.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.174+0000 7f8830000640 1 -- 172.21.15.27:0/1675301153 >> 172.21.15.27:0/1675301153 conn(0x7f88280cb700 msgr2=0x7f8828071060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:27.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.174+0000 7f8830000640 1 -- 172.21.15.27:0/1675301153 shutdown_connections 2024-01-15T18:13:27.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.174+0000 7f8830000640 1 -- 172.21.15.27:0/1675301153 wait complete. 2024-01-15T18:13:27.196 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone519312 2024-01-15T18:13:27.206 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:13:27.244 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:13:27.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.342+0000 7f8eb4000640 1 Processor -- start 2024-01-15T18:13:27.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.342+0000 7f8eb4000640 1 -- start start 2024-01-15T18:13:27.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.342+0000 7f8eb4000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8eac1557e0 0x7f8eac0acd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:27.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.342+0000 7f8eb4000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8eac06ac00 0x7f8eac15c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:27.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.342+0000 7f8eb4000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eac0ad410 con 0x7f8eac0b3d10 2024-01-15T18:13:27.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.342+0000 7f8eb4000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eac0ae360 con 0x7f8eac06ac00 2024-01-15T18:13:27.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.342+0000 7f8eb4000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac15c570 con 0x7f8eac1557e0 2024-01-15T18:13:27.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.342+0000 7f8eb2000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8eac06ac00 0x7f8eac15c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:27.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.342+0000 7f8eb0c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8eac0b3d10 0x7f8eac06a4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53920/0 (socket says 172.21.15.27:53920) 2024-01-15T18:13:27.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.342+0000 7f8eb0c00640 1 -- 172.21.15.27:0/2306204423 learned_addr learned my addr 172.21.15.27:0/2306204423 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:13:27.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.342+0000 7f8eb1600640 1 --2- 172.21.15.27:0/2306204423 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8eac1557e0 0x7f8eac0acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:27.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.342+0000 7f8eb2000640 1 -- 172.21.15.27:0/2306204423 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8eac1557e0 msgr2=0x7f8eac0acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:27.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.342+0000 7f8eb2000640 1 --2- 172.21.15.27:0/2306204423 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8eac1557e0 0x7f8eac0acd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:27.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.342+0000 7f8eb2000640 1 -- 172.21.15.27:0/2306204423 >> v1:172.21.15.27:6789/0 conn(0x7f8eac0b3d10 legacy=0x7f8eac06a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:27.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.342+0000 7f8eb2000640 1 -- 172.21.15.27:0/2306204423 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eac0b2d00 con 0x7f8eac06ac00 2024-01-15T18:13:27.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.342+0000 7f8eabe00640 1 -- 172.21.15.27:0/2306204423 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3622390086 0 0) 0x7f8eac0ad410 con 0x7f8eac0b3d10 2024-01-15T18:13:27.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.342+0000 7f8eb1600640 1 --2- 172.21.15.27:0/2306204423 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8eac1557e0 0x7f8eac0acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:13:27.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.346+0000 7f8eb2000640 1 --2- 172.21.15.27:0/2306204423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8eac06ac00 0x7f8eac15c030 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4009d90 tx=0x7f8ea401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec670b8e90b70751 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:27.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.346+0000 7f8eabe00640 1 -- 172.21.15.27:0/2306204423 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea40a5800 con 0x7f8eac06ac00 2024-01-15T18:13:27.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.346+0000 7f8eabe00640 1 -- 172.21.15.27:0/2306204423 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea40a5960 con 0x7f8eac06ac00 2024-01-15T18:13:27.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.346+0000 7f8eabe00640 1 -- 172.21.15.27:0/2306204423 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea40a5cf0 con 0x7f8eac06ac00 2024-01-15T18:13:27.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.346+0000 7f8eb4000640 1 -- 172.21.15.27:0/2306204423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8eac06ac00 msgr2=0x7f8eac15c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:27.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.346+0000 7f8eb4000640 1 --2- 172.21.15.27:0/2306204423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8eac06ac00 0x7f8eac15c030 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4009d90 tx=0x7f8ea401c950 comp rx=0 tx=0).stop 2024-01-15T18:13:27.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.346+0000 7f8eb4000640 1 -- 172.21.15.27:0/2306204423 shutdown_connections 2024-01-15T18:13:27.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.346+0000 7f8eb4000640 1 --2- 172.21.15.27:0/2306204423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8eac06ac00 0x7f8eac15c030 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:27.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.346+0000 7f8eb4000640 1 --2- 172.21.15.27:0/2306204423 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8eac1557e0 0x7f8eac0acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:27.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.346+0000 7f8eb4000640 1 -- 172.21.15.27:0/2306204423 >> 172.21.15.27:0/2306204423 conn(0x7f8eac06b600 msgr2=0x7f8eac06ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:27.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.346+0000 7f8eb4000640 1 -- 172.21.15.27:0/2306204423 shutdown_connections 2024-01-15T18:13:27.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.346+0000 7f8eb4000640 1 -- 172.21.15.27:0/2306204423 wait complete. 2024-01-15T18:13:27.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.350+0000 7f8eb4000640 1 Processor -- start 2024-01-15T18:13:27.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.350+0000 7f8eb4000640 1 -- start start 2024-01-15T18:13:27.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.350+0000 7f8eb4000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8eac06ac00 0x7f8eac142390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:27.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.350+0000 7f8eb4000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8eac0b3d10 0x7f8eac1525a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:27.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.350+0000 7f8eb4000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8eac1557e0 0x7f8eac14a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:27.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.350+0000 7f8eb4000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac159eb0 con 0x7f8eac06ac00 2024-01-15T18:13:27.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.350+0000 7f8eb4000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac1521a0 con 0x7f8eac0b3d10 2024-01-15T18:13:27.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.350+0000 7f8eb4000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eac159230 con 0x7f8eac1557e0 2024-01-15T18:13:27.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.350+0000 7f8eb1600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8eac06ac00 0x7f8eac142390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:27.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.350+0000 7f8eb1600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8eac06ac00 0x7f8eac142390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40780/0 (socket says 172.21.15.27:40780) 2024-01-15T18:13:27.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.350+0000 7f8eb1600640 1 -- 172.21.15.27:0/3387258298 learned_addr learned my addr 172.21.15.27:0/3387258298 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:27.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.350+0000 7f8eb1600640 1 -- 172.21.15.27:0/3387258298 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8eac1557e0 msgr2=0x7f8eac14a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:27.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.350+0000 7f8eb1600640 1 --2- 172.21.15.27:0/3387258298 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8eac1557e0 0x7f8eac14a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:27.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.350+0000 7f8eb1600640 1 -- 172.21.15.27:0/3387258298 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8eac0b3d10 msgr2=0x7f8eac1525a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:27.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.350+0000 7f8eb1600640 1 --2- 172.21.15.27:0/3387258298 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8eac0b3d10 0x7f8eac1525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:27.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.350+0000 7f8eb1600640 1 -- 172.21.15.27:0/3387258298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea4009a70 con 0x7f8eac06ac00 2024-01-15T18:13:27.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.350+0000 7f8eb1600640 1 --2- 172.21.15.27:0/3387258298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8eac06ac00 0x7f8eac142390 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c00d800 tx=0x7f8e9c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:27.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.350+0000 7f8ea9600640 1 -- 172.21.15.27:0/3387258298 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c0096c0 con 0x7f8eac06ac00 2024-01-15T18:13:27.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.350+0000 7f8ea9600640 1 -- 172.21.15.27:0/3387258298 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9c009820 con 0x7f8eac06ac00 2024-01-15T18:13:27.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.350+0000 7f8ea9600640 1 -- 172.21.15.27:0/3387258298 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c009bd0 con 0x7f8eac06ac00 2024-01-15T18:13:27.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.350+0000 7f8eb4000640 1 -- 172.21.15.27:0/3387258298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eac159510 con 0x7f8eac06ac00 2024-01-15T18:13:27.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.354+0000 7f8eb4000640 1 -- 172.21.15.27:0/3387258298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eac159ad0 con 0x7f8eac06ac00 2024-01-15T18:13:27.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.354+0000 7f8eb4000640 1 -- 172.21.15.27:0/3387258298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eac04e850 con 0x7f8eac06ac00 2024-01-15T18:13:27.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.354+0000 7f8ea9600640 1 -- 172.21.15.27:0/3387258298 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f8e9c009d30 con 0x7f8eac06ac00 2024-01-15T18:13:27.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.354+0000 7f8ea9600640 1 --2- 172.21.15.27:0/3387258298 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8e840673a0 0x7f8e84069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:27.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.354+0000 7f8ea9600640 1 -- 172.21.15.27:0/3387258298 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f8e9c083450 con 0x7f8eac06ac00 2024-01-15T18:13:27.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.354+0000 7f8eb0c00640 1 --2- 172.21.15.27:0/3387258298 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8e840673a0 0x7f8e84069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:27.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.358+0000 7f8eb0c00640 1 --2- 172.21.15.27:0/3387258298 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8e840673a0 0x7f8e84069860 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8e940011f0 tx=0x7f8e94007930 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:27.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.358+0000 7f8ea9600640 1 -- 172.21.15.27:0/3387258298 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8e9c052140 con 0x7f8eac06ac00 2024-01-15T18:13:27.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.550+0000 7f8eb4000640 1 -- 172.21.15.27:0/3387258298 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone519312", "target": ["mon-mgr", ""]}) v1 -- 0x7f8eac06b2a0 con 0x7f8e840673a0 2024-01-15T18:13:27.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.614+0000 7f8ea9600640 1 -- 172.21.15.27:0/3387258298 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8eac06b2a0 con 0x7f8e840673a0 2024-01-15T18:13:27.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.614+0000 7f8eb4000640 1 -- 172.21.15.27:0/3387258298 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8e840673a0 msgr2=0x7f8e84069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:27.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.614+0000 7f8eb4000640 1 --2- 172.21.15.27:0/3387258298 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8e840673a0 0x7f8e84069860 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8e940011f0 tx=0x7f8e94007930 comp rx=0 tx=0).stop 2024-01-15T18:13:27.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.614+0000 7f8eb4000640 1 -- 172.21.15.27:0/3387258298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8eac06ac00 msgr2=0x7f8eac142390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:27.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.614+0000 7f8eb4000640 1 --2- 172.21.15.27:0/3387258298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8eac06ac00 0x7f8eac142390 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c00d800 tx=0x7f8e9c00dcc0 comp rx=0 tx=0).stop 2024-01-15T18:13:27.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.614+0000 7f8eb4000640 1 -- 172.21.15.27:0/3387258298 shutdown_connections 2024-01-15T18:13:27.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.614+0000 7f8eb4000640 1 --2- 172.21.15.27:0/3387258298 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8e840673a0 0x7f8e84069860 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:27.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.614+0000 7f8eb4000640 1 --2- 172.21.15.27:0/3387258298 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8eac1557e0 0x7f8eac14a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:27.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.614+0000 7f8eb4000640 1 --2- 172.21.15.27:0/3387258298 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8eac0b3d10 0x7f8eac1525a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:27.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.614+0000 7f8eb4000640 1 --2- 172.21.15.27:0/3387258298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8eac06ac00 0x7f8eac142390 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:27.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.614+0000 7f8eb4000640 1 -- 172.21.15.27:0/3387258298 >> 172.21.15.27:0/3387258298 conn(0x7f8eac06b600 msgr2=0x7f8eac152ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:27.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.614+0000 7f8eb4000640 1 -- 172.21.15.27:0/3387258298 shutdown_connections 2024-01-15T18:13:27.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.614+0000 7f8eb4000640 1 -- 172.21.15.27:0/3387258298 wait complete. 2024-01-15T18:13:27.638 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone519313 2024-01-15T18:13:27.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.782+0000 7fec6ba00640 1 Processor -- start 2024-01-15T18:13:27.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.782+0000 7fec6ba00640 1 -- start start 2024-01-15T18:13:27.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.782+0000 7fec6ba00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fec6415d570 0x7fec6406be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:27.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.782+0000 7fec6ba00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fec6406b810 0x7fec6406c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:27.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.782+0000 7fec6ba00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec640b0d70 con 0x7fec6415d190 2024-01-15T18:13:27.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.782+0000 7fec6ba00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec640d0540 con 0x7fec6406b810 2024-01-15T18:13:27.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.782+0000 7fec6ba00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec640b0440 con 0x7fec6415d570 2024-01-15T18:13:27.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.782+0000 7fec69a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fec6406b810 0x7fec6406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:27.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.782+0000 7fec69a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fec6406b810 0x7fec6406c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40796/0 (socket says 172.21.15.27:40796) 2024-01-15T18:13:27.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.782+0000 7fec69a00640 1 -- 172.21.15.27:0/116082046 learned_addr learned my addr 172.21.15.27:0/116082046 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:27.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.782+0000 7fec63e00640 1 --2- 172.21.15.27:0/116082046 >> v2:172.21.15.93:3300/0 conn(0x7fec6415d570 0x7fec6406be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:27.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.782+0000 7fec69a00640 1 -- 172.21.15.27:0/116082046 >> v2:172.21.15.93:3300/0 conn(0x7fec6415d570 msgr2=0x7fec6406be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:27.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.782+0000 7fec69a00640 1 --2- 172.21.15.27:0/116082046 >> v2:172.21.15.93:3300/0 conn(0x7fec6415d570 0x7fec6406be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:27.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.782+0000 7fec69a00640 1 -- 172.21.15.27:0/116082046 >> v1:172.21.15.93:6789/0 conn(0x7fec6415d190 legacy=0x7fec640cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:27.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.786+0000 7fec69a00640 1 -- 172.21.15.27:0/116082046 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec640b0720 con 0x7fec6406b810 2024-01-15T18:13:27.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.786+0000 7fec69a00640 1 --2- 172.21.15.27:0/116082046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec6406b810 0x7fec6406c350 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fec640713a0 tx=0x7fec5c01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cce98498ba80e22 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:27.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.786+0000 7fec63400640 1 -- 172.21.15.27:0/116082046 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5c0a5840 con 0x7fec6406b810 2024-01-15T18:13:27.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.786+0000 7fec63400640 1 -- 172.21.15.27:0/116082046 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec5c0a59a0 con 0x7fec6406b810 2024-01-15T18:13:27.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.786+0000 7fec63400640 1 -- 172.21.15.27:0/116082046 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5c0a5d30 con 0x7fec6406b810 2024-01-15T18:13:27.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.786+0000 7fec6ba00640 1 -- 172.21.15.27:0/116082046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec6406b810 msgr2=0x7fec6406c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:27.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.786+0000 7fec6ba00640 1 --2- 172.21.15.27:0/116082046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec6406b810 0x7fec6406c350 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fec640713a0 tx=0x7fec5c01c960 comp rx=0 tx=0).stop 2024-01-15T18:13:27.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec6ba00640 1 -- 172.21.15.27:0/116082046 shutdown_connections 2024-01-15T18:13:27.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec6ba00640 1 --2- 172.21.15.27:0/116082046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec6406b810 0x7fec6406c350 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:27.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec6ba00640 1 --2- 172.21.15.27:0/116082046 >> v2:172.21.15.93:3300/0 conn(0x7fec6415d570 0x7fec6406be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:27.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec6ba00640 1 -- 172.21.15.27:0/116082046 >> 172.21.15.27:0/116082046 conn(0x7fec640d0e70 msgr2=0x7fec640d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:27.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec6ba00640 1 -- 172.21.15.27:0/116082046 shutdown_connections 2024-01-15T18:13:27.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec6ba00640 1 -- 172.21.15.27:0/116082046 wait complete. 2024-01-15T18:13:27.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec6ba00640 1 Processor -- start 2024-01-15T18:13:27.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec6ba00640 1 -- start start 2024-01-15T18:13:27.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec6ba00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fec6406b810 0x7fec641581d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:27.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec6ba00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec6415d190 0x7fec6409bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:27.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec6ba00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fec6415d570 0x7fec640c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:27.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec6ba00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec640c5cc0 con 0x7fec6415d190 2024-01-15T18:13:27.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec6ba00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6409c510 con 0x7fec6415d570 2024-01-15T18:13:27.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec6ba00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec640c4e10 con 0x7fec6406b810 2024-01-15T18:13:27.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec63e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec6415d190 0x7fec6409bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:27.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec63e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec6415d190 0x7fec6409bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40808/0 (socket says 172.21.15.27:40808) 2024-01-15T18:13:27.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec63e00640 1 -- 172.21.15.27:0/2556394939 learned_addr learned my addr 172.21.15.27:0/2556394939 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:27.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec69a00640 1 --2- 172.21.15.27:0/2556394939 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fec6415d570 0x7fec640c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:27.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec69000640 1 --2- 172.21.15.27:0/2556394939 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fec6406b810 0x7fec641581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:27.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec63e00640 1 -- 172.21.15.27:0/2556394939 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fec6406b810 msgr2=0x7fec641581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:27.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec63e00640 1 --2- 172.21.15.27:0/2556394939 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fec6406b810 0x7fec641581d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:27.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec63e00640 1 -- 172.21.15.27:0/2556394939 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fec6415d570 msgr2=0x7fec640c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:27.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec63e00640 1 --2- 172.21.15.27:0/2556394939 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fec6415d570 0x7fec640c48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:27.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec63e00640 1 -- 172.21.15.27:0/2556394939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5c009b30 con 0x7fec6415d190 2024-01-15T18:13:27.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec69000640 1 --2- 172.21.15.27:0/2556394939 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fec6406b810 0x7fec641581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:13:27.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.790+0000 7fec63e00640 1 --2- 172.21.15.27:0/2556394939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec6415d190 0x7fec6409bfd0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fec4c00d5f0 tx=0x7fec4c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:27.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.794+0000 7fec60c00640 1 -- 172.21.15.27:0/2556394939 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4c0125f0 con 0x7fec6415d190 2024-01-15T18:13:27.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.794+0000 7fec60c00640 1 -- 172.21.15.27:0/2556394939 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec4c012750 con 0x7fec6415d190 2024-01-15T18:13:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.794+0000 7fec6ba00640 1 -- 172.21.15.27:0/2556394939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec640c50b0 con 0x7fec6415d190 2024-01-15T18:13:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.794+0000 7fec60c00640 1 -- 172.21.15.27:0/2556394939 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4c012ae0 con 0x7fec6415d190 2024-01-15T18:13:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.794+0000 7fec6ba00640 1 -- 172.21.15.27:0/2556394939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec640c5670 con 0x7fec6415d190 2024-01-15T18:13:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.794+0000 7fec6ba00640 1 -- 172.21.15.27:0/2556394939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec640b11e0 con 0x7fec6415d190 2024-01-15T18:13:27.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.794+0000 7fec60c00640 1 -- 172.21.15.27:0/2556394939 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7fec4c012c40 con 0x7fec6415d190 2024-01-15T18:13:27.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.794+0000 7fec60c00640 1 --2- 172.21.15.27:0/2556394939 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fec3c0673a0 0x7fec3c069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:27.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.794+0000 7fec60c00640 1 -- 172.21.15.27:0/2556394939 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fec4c083220 con 0x7fec6415d190 2024-01-15T18:13:27.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.794+0000 7fec69000640 1 --2- 172.21.15.27:0/2556394939 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fec3c0673a0 0x7fec3c069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:27.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.798+0000 7fec69000640 1 --2- 172.21.15.27:0/2556394939 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fec3c0673a0 0x7fec3c069860 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fec54003ec0 tx=0x7fec54009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:27.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.798+0000 7fec60c00640 1 -- 172.21.15.27:0/2556394939 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fec640b11e0 con 0x7fec6415d190 2024-01-15T18:13:27.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:27.986+0000 7fec6ba00640 1 -- 172.21.15.27:0/2556394939 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone519313", "target": ["mon-mgr", ""]}) v1 -- 0x7fec640b11e0 con 0x7fec3c0673a0 2024-01-15T18:13:28.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.050+0000 7fec60c00640 1 -- 172.21.15.27:0/2556394939 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fec640b11e0 con 0x7fec3c0673a0 2024-01-15T18:13:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.054+0000 7fec6ba00640 1 -- 172.21.15.27:0/2556394939 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fec3c0673a0 msgr2=0x7fec3c069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.054+0000 7fec6ba00640 1 --2- 172.21.15.27:0/2556394939 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fec3c0673a0 0x7fec3c069860 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fec54003ec0 tx=0x7fec54009040 comp rx=0 tx=0).stop 2024-01-15T18:13:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.054+0000 7fec6ba00640 1 -- 172.21.15.27:0/2556394939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec6415d190 msgr2=0x7fec6409bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.054+0000 7fec6ba00640 1 --2- 172.21.15.27:0/2556394939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec6415d190 0x7fec6409bfd0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fec4c00d5f0 tx=0x7fec4c00dab0 comp rx=0 tx=0).stop 2024-01-15T18:13:28.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.054+0000 7fec6ba00640 1 -- 172.21.15.27:0/2556394939 shutdown_connections 2024-01-15T18:13:28.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.054+0000 7fec6ba00640 1 --2- 172.21.15.27:0/2556394939 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fec3c0673a0 0x7fec3c069860 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:28.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.054+0000 7fec6ba00640 1 --2- 172.21.15.27:0/2556394939 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fec6415d570 0x7fec640c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:28.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.054+0000 7fec6ba00640 1 --2- 172.21.15.27:0/2556394939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec6415d190 0x7fec6409bfd0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:28.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.054+0000 7fec6ba00640 1 --2- 172.21.15.27:0/2556394939 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fec6406b810 0x7fec641581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:28.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.054+0000 7fec6ba00640 1 -- 172.21.15.27:0/2556394939 >> 172.21.15.27:0/2556394939 conn(0x7fec640d0e70 msgr2=0x7fec640ae660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:28.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.054+0000 7fec6ba00640 1 -- 172.21.15.27:0/2556394939 shutdown_connections 2024-01-15T18:13:28.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.054+0000 7fec6ba00640 1 -- 172.21.15.27:0/2556394939 wait complete. 2024-01-15T18:13:28.075 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone519314 2024-01-15T18:13:28.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.230+0000 7fe0f8600640 1 Processor -- start 2024-01-15T18:13:28.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.230+0000 7fe0f8600640 1 -- start start 2024-01-15T18:13:28.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.230+0000 7fe0f8600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fe0f006ac00 0x7fe0f015ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:28.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.230+0000 7fe0f8600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe0f015efb0 0x7fe0f00b4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:28.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.230+0000 7fe0f8600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f013e230 con 0x7fe0f00cc100 2024-01-15T18:13:28.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.230+0000 7fe0f8600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0f0140560 con 0x7fe0f015efb0 2024-01-15T18:13:28.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.230+0000 7fe0f8600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0f015f760 con 0x7fe0f006ac00 2024-01-15T18:13:28.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.230+0000 7fe0f6600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe0f015efb0 0x7fe0f00b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:28.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.230+0000 7fe0f5c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe0f00cc100 0x7fe0f006a4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53936/0 (socket says 172.21.15.27:53936) 2024-01-15T18:13:28.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.230+0000 7fe0f5c00640 1 -- 172.21.15.27:0/414275830 learned_addr learned my addr 172.21.15.27:0/414275830 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:13:28.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.230+0000 7fe0f6600640 1 -- 172.21.15.27:0/414275830 >> v2:172.21.15.93:3300/0 conn(0x7fe0f006ac00 msgr2=0x7fe0f015ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:28.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.230+0000 7fe0f6600640 1 --2- 172.21.15.27:0/414275830 >> v2:172.21.15.93:3300/0 conn(0x7fe0f006ac00 0x7fe0f015ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:28.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.230+0000 7fe0f6600640 1 -- 172.21.15.27:0/414275830 >> v1:172.21.15.27:6789/0 conn(0x7fe0f00cc100 legacy=0x7fe0f006a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:28.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.234+0000 7fe0f6600640 1 -- 172.21.15.27:0/414275830 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f01407b0 con 0x7fe0f015efb0 2024-01-15T18:13:28.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.234+0000 7fe0efe00640 1 -- 172.21.15.27:0/414275830 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383516030 0 0) 0x7fe0e0001120 con 0x7fe0f00cc100 2024-01-15T18:13:28.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.234+0000 7fe0f6600640 1 --2- 172.21.15.27:0/414275830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0f015efb0 0x7fe0f00b4870 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fe0e8009d90 tx=0x7fe0e801c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0fe69d779b6f399 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:28.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.234+0000 7fe0efe00640 1 -- 172.21.15.27:0/414275830 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e80a5910 con 0x7fe0f015efb0 2024-01-15T18:13:28.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.234+0000 7fe0efe00640 1 -- 172.21.15.27:0/414275830 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e80a5a70 con 0x7fe0f015efb0 2024-01-15T18:13:28.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.234+0000 7fe0efe00640 1 -- 172.21.15.27:0/414275830 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e80a5e00 con 0x7fe0f015efb0 2024-01-15T18:13:28.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.238+0000 7fe0f8600640 1 -- 172.21.15.27:0/414275830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0f015efb0 msgr2=0x7fe0f00b4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:28.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.238+0000 7fe0f8600640 1 --2- 172.21.15.27:0/414275830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0f015efb0 0x7fe0f00b4870 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fe0e8009d90 tx=0x7fe0e801c850 comp rx=0 tx=0).stop 2024-01-15T18:13:28.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.238+0000 7fe0f8600640 1 -- 172.21.15.27:0/414275830 shutdown_connections 2024-01-15T18:13:28.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.238+0000 7fe0f8600640 1 --2- 172.21.15.27:0/414275830 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0f015efb0 0x7fe0f00b4870 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:28.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.238+0000 7fe0f8600640 1 --2- 172.21.15.27:0/414275830 >> v2:172.21.15.93:3300/0 conn(0x7fe0f006ac00 0x7fe0f015ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:28.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.238+0000 7fe0f8600640 1 -- 172.21.15.27:0/414275830 >> 172.21.15.27:0/414275830 conn(0x7fe0f00cce10 msgr2=0x7fe0f00b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:28.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.238+0000 7fe0f8600640 1 -- 172.21.15.27:0/414275830 shutdown_connections 2024-01-15T18:13:28.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.238+0000 7fe0f8600640 1 -- 172.21.15.27:0/414275830 wait complete. 2024-01-15T18:13:28.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.238+0000 7fe0f8600640 1 Processor -- start 2024-01-15T18:13:28.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.238+0000 7fe0f8600640 1 -- start start 2024-01-15T18:13:28.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.238+0000 7fe0f8600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe0f006ac00 0x7fe0f0159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:28.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.238+0000 7fe0f8600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0f00cc100 0x7fe0f01354a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:28.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.238+0000 7fe0f8600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0f015efb0 0x7fe0f015a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:28.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.238+0000 7fe0f8600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f015a290 con 0x7fe0f00cc100 2024-01-15T18:13:28.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.238+0000 7fe0f8600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f0159680 con 0x7fe0f015efb0 2024-01-15T18:13:28.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.238+0000 7fe0f8600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f0159960 con 0x7fe0f006ac00 2024-01-15T18:13:28.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.238+0000 7fe0f5200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0f00cc100 0x7fe0f01354a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:28.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.242+0000 7fe0f5200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0f00cc100 0x7fe0f01354a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40812/0 (socket says 172.21.15.27:40812) 2024-01-15T18:13:28.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.242+0000 7fe0f5200640 1 -- 172.21.15.27:0/963600803 learned_addr learned my addr 172.21.15.27:0/963600803 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:28.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.242+0000 7fe0f6600640 1 --2- 172.21.15.27:0/963600803 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0f015efb0 0x7fe0f015a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:28.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.242+0000 7fe0f5200640 1 -- 172.21.15.27:0/963600803 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe0f006ac00 msgr2=0x7fe0f0159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:28.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.242+0000 7fe0f5200640 1 --2- 172.21.15.27:0/963600803 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe0f006ac00 0x7fe0f0159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:28.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.242+0000 7fe0f5200640 1 -- 172.21.15.27:0/963600803 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0f015efb0 msgr2=0x7fe0f015a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:28.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.242+0000 7fe0f5200640 1 --2- 172.21.15.27:0/963600803 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0f015efb0 0x7fe0f015a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:28.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.242+0000 7fe0f5200640 1 -- 172.21.15.27:0/963600803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e8009a70 con 0x7fe0f00cc100 2024-01-15T18:13:28.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.242+0000 7fe0f5200640 1 --2- 172.21.15.27:0/963600803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0f00cc100 0x7fe0f01354a0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fe0d800b550 tx=0x7fe0d800ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:28.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.242+0000 7fe0ed600640 1 -- 172.21.15.27:0/963600803 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d8010870 con 0x7fe0f00cc100 2024-01-15T18:13:28.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.242+0000 7fe0ed600640 1 -- 172.21.15.27:0/963600803 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d80109f0 con 0x7fe0f00cc100 2024-01-15T18:13:28.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.242+0000 7fe0ed600640 1 -- 172.21.15.27:0/963600803 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d8010da0 con 0x7fe0f00cc100 2024-01-15T18:13:28.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.242+0000 7fe0f8600640 1 -- 172.21.15.27:0/963600803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0f0159c40 con 0x7fe0f00cc100 2024-01-15T18:13:28.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.242+0000 7fe0f8600640 1 -- 172.21.15.27:0/963600803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0f013aab0 con 0x7fe0f00cc100 2024-01-15T18:13:28.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.242+0000 7fe0f8600640 1 -- 172.21.15.27:0/963600803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0f004e850 con 0x7fe0f00cc100 2024-01-15T18:13:28.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.242+0000 7fe0ed600640 1 -- 172.21.15.27:0/963600803 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7fe0d800f3c0 con 0x7fe0f00cc100 2024-01-15T18:13:28.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.246+0000 7fe0ed600640 1 --2- 172.21.15.27:0/963600803 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe0c80673a0 0x7fe0c8069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:28.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.246+0000 7fe0ed600640 1 -- 172.21.15.27:0/963600803 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fe0d8082e40 con 0x7fe0f00cc100 2024-01-15T18:13:28.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.246+0000 7fe0f5c00640 1 --2- 172.21.15.27:0/963600803 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe0c80673a0 0x7fe0c8069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:28.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.250+0000 7fe0ed600640 1 -- 172.21.15.27:0/963600803 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe0d8051ad0 con 0x7fe0f00cc100 2024-01-15T18:13:28.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.258+0000 7fe0f5c00640 1 --2- 172.21.15.27:0/963600803 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe0c80673a0 0x7fe0c8069860 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0001640 tx=0x7fe0e0001320 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:28.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:28.578+0000 7fe0f8600640 1 -- 172.21.15.27:0/963600803 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone519314", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0f0072190 con 0x7fe0c80673a0 2024-01-15T18:13:29.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.050+0000 7fe0ed600640 1 -- 172.21.15.27:0/963600803 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe0f0072190 con 0x7fe0c80673a0 2024-01-15T18:13:29.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.058+0000 7fe0f8600640 1 -- 172.21.15.27:0/963600803 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe0c80673a0 msgr2=0x7fe0c8069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:29.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.058+0000 7fe0f8600640 1 --2- 172.21.15.27:0/963600803 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe0c80673a0 0x7fe0c8069860 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0001640 tx=0x7fe0e0001320 comp rx=0 tx=0).stop 2024-01-15T18:13:29.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.058+0000 7fe0f8600640 1 -- 172.21.15.27:0/963600803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0f00cc100 msgr2=0x7fe0f01354a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:29.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.058+0000 7fe0f8600640 1 --2- 172.21.15.27:0/963600803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0f00cc100 0x7fe0f01354a0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fe0d800b550 tx=0x7fe0d800ba10 comp rx=0 tx=0).stop 2024-01-15T18:13:29.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.062+0000 7fe0f8600640 1 -- 172.21.15.27:0/963600803 shutdown_connections 2024-01-15T18:13:29.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.062+0000 7fe0f8600640 1 --2- 172.21.15.27:0/963600803 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe0c80673a0 0x7fe0c8069860 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:29.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.062+0000 7fe0f8600640 1 --2- 172.21.15.27:0/963600803 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0f015efb0 0x7fe0f015a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:29.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.062+0000 7fe0f8600640 1 --2- 172.21.15.27:0/963600803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0f00cc100 0x7fe0f01354a0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:29.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.062+0000 7fe0f8600640 1 --2- 172.21.15.27:0/963600803 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe0f006ac00 0x7fe0f0159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:29.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.062+0000 7fe0f8600640 1 -- 172.21.15.27:0/963600803 >> 172.21.15.27:0/963600803 conn(0x7fe0f00cce10 msgr2=0x7fe0f0071a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:29.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.062+0000 7fe0f8600640 1 -- 172.21.15.27:0/963600803 shutdown_connections 2024-01-15T18:13:29.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.062+0000 7fe0f8600640 1 -- 172.21.15.27:0/963600803 wait complete. 2024-01-15T18:13:29.112 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone519315 2024-01-15T18:13:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.338+0000 7f530f800640 1 Processor -- start 2024-01-15T18:13:29.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.342+0000 7f530f800640 1 -- start start 2024-01-15T18:13:29.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.342+0000 7f530f800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5308161a90 0x7f5308161ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:29.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.342+0000 7f530f800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f53080ce810 0x7f53080ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:29.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.342+0000 7f530f800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530806c5d0 con 0x7f53080cdb20 2024-01-15T18:13:29.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.342+0000 7f530f800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53080cf1e0 con 0x7f53080ce810 2024-01-15T18:13:29.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.342+0000 7f530f800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5308162b00 con 0x7f5308161a90 2024-01-15T18:13:29.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.342+0000 7f5307e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5308161a90 0x7f5308161ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:29.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.342+0000 7f5307e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5308161a90 0x7f5308161ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:37364/0 (socket says 172.21.15.27:37364) 2024-01-15T18:13:29.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.342+0000 7f5307e00640 1 -- 172.21.15.27:0/629890857 learned_addr learned my addr 172.21.15.27:0/629890857 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:29.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.342+0000 7f5307e00640 1 -- 172.21.15.27:0/629890857 >> v2:172.21.15.93:3300/0 conn(0x7f53080ce810 msgr2=0x7f53080ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:29.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.342+0000 7f5307e00640 1 --2- 172.21.15.27:0/629890857 >> v2:172.21.15.93:3300/0 conn(0x7f53080ce810 0x7f53080ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:29.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.342+0000 7f5307e00640 1 -- 172.21.15.27:0/629890857 >> v1:172.21.15.93:6789/0 conn(0x7f53080cdb20 legacy=0x7f530806cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:29.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.342+0000 7f5307e00640 1 -- 172.21.15.27:0/629890857 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530806d4a0 con 0x7f5308161a90 2024-01-15T18:13:29.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.346+0000 7f5307e00640 1 --2- 172.21.15.27:0/629890857 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5308161a90 0x7f5308161ef0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f52f0009870 tx=0x7f52f001c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a87d9a9efa70c8fb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:29.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.350+0000 7f5307400640 1 -- 172.21.15.27:0/629890857 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f00aa670 con 0x7f5308161a90 2024-01-15T18:13:29.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.350+0000 7f5307400640 1 -- 172.21.15.27:0/629890857 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52f00aa7d0 con 0x7f5308161a90 2024-01-15T18:13:29.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.350+0000 7f5307400640 1 -- 172.21.15.27:0/629890857 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52f00aab60 con 0x7f5308161a90 2024-01-15T18:13:29.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.350+0000 7f530f800640 1 -- 172.21.15.27:0/629890857 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5308161a90 msgr2=0x7f5308161ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:29.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.350+0000 7f530f800640 1 --2- 172.21.15.27:0/629890857 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5308161a90 0x7f5308161ef0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f52f0009870 tx=0x7f52f001c510 comp rx=0 tx=0).stop 2024-01-15T18:13:29.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.350+0000 7f530f800640 1 -- 172.21.15.27:0/629890857 shutdown_connections 2024-01-15T18:13:29.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.350+0000 7f530f800640 1 --2- 172.21.15.27:0/629890857 >> v2:172.21.15.93:3300/0 conn(0x7f53080ce810 0x7f53080ceca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:29.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.350+0000 7f530f800640 1 --2- 172.21.15.27:0/629890857 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5308161a90 0x7f5308161ef0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:29.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.350+0000 7f530f800640 1 -- 172.21.15.27:0/629890857 >> 172.21.15.27:0/629890857 conn(0x7f53080cb700 msgr2=0x7f53080cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:29.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.350+0000 7f530f800640 1 -- 172.21.15.27:0/629890857 shutdown_connections 2024-01-15T18:13:29.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.350+0000 7f530f800640 1 -- 172.21.15.27:0/629890857 wait complete. 2024-01-15T18:13:29.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.354+0000 7f530f800640 1 Processor -- start 2024-01-15T18:13:29.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.354+0000 7f530f800640 1 -- start start 2024-01-15T18:13:29.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.354+0000 7f530f800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f53080cdb20 0x7f5308137570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:29.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.354+0000 7f530f800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f53080ce810 0x7f530814a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:29.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.358+0000 7f530f800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5308161a90 0x7f5308155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:29.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.358+0000 7f530f800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5308156950 con 0x7f5308161a90 2024-01-15T18:13:29.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.358+0000 7f530f800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530814ade0 con 0x7f53080ce810 2024-01-15T18:13:29.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.358+0000 7f530f800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5308155a90 con 0x7f53080cdb20 2024-01-15T18:13:29.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.358+0000 7f530d800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5308161a90 0x7f5308155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:29.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.358+0000 7f530d800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5308161a90 0x7f5308155550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40822/0 (socket says 172.21.15.27:40822) 2024-01-15T18:13:29.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.358+0000 7f530d800640 1 -- 172.21.15.27:0/3044184072 learned_addr learned my addr 172.21.15.27:0/3044184072 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:29.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.358+0000 7f530d800640 1 -- 172.21.15.27:0/3044184072 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f53080cdb20 msgr2=0x7f5308137570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:29.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.358+0000 7f530d800640 1 --2- 172.21.15.27:0/3044184072 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f53080cdb20 0x7f5308137570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:29.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.358+0000 7f530d800640 1 -- 172.21.15.27:0/3044184072 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f53080ce810 msgr2=0x7f530814a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:29.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.358+0000 7f530d800640 1 --2- 172.21.15.27:0/3044184072 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f53080ce810 0x7f530814a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:29.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.358+0000 7f530d800640 1 -- 172.21.15.27:0/3044184072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f00094d0 con 0x7f5308161a90 2024-01-15T18:13:29.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.358+0000 7f530d800640 1 --2- 172.21.15.27:0/3044184072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5308161a90 0x7f5308155550 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f530000b970 tx=0x7f530000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:29.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.358+0000 7f5304c00640 1 -- 172.21.15.27:0/3044184072 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530000cb40 con 0x7f5308161a90 2024-01-15T18:13:29.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.358+0000 7f530f800640 1 -- 172.21.15.27:0/3044184072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5308155d40 con 0x7f5308161a90 2024-01-15T18:13:29.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.358+0000 7f5304c00640 1 -- 172.21.15.27:0/3044184072 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f530000ccc0 con 0x7f5308161a90 2024-01-15T18:13:29.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.358+0000 7f5304c00640 1 -- 172.21.15.27:0/3044184072 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5300014680 con 0x7f5308161a90 2024-01-15T18:13:29.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.358+0000 7f530f800640 1 -- 172.21.15.27:0/3044184072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5308156300 con 0x7f5308161a90 2024-01-15T18:13:29.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.358+0000 7f530f800640 1 -- 172.21.15.27:0/3044184072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52d8005260 con 0x7f5308161a90 2024-01-15T18:13:29.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.362+0000 7f5304c00640 1 -- 172.21.15.27:0/3044184072 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f530001d3b0 con 0x7f5308161a90 2024-01-15T18:13:29.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.362+0000 7f5304c00640 1 --2- 172.21.15.27:0/3044184072 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f52e00673a0 0x7f52e0069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:29.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.362+0000 7f5304c00640 1 -- 172.21.15.27:0/3044184072 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f5300082d10 con 0x7f5308161a90 2024-01-15T18:13:29.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.370+0000 7f5304c00640 1 -- 172.21.15.27:0/3044184072 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5300051980 con 0x7f5308161a90 2024-01-15T18:13:29.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.374+0000 7f530ce00640 1 --2- 172.21.15.27:0/3044184072 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f52e00673a0 0x7f52e0069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:29.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.386+0000 7f530ce00640 1 --2- 172.21.15.27:0/3044184072 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f52e00673a0 0x7f52e0069860 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f52f8000f70 tx=0x7f52f80079c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:29.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:29.726+0000 7f530f800640 1 -- 172.21.15.27:0/3044184072 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone519315", "target": ["mon-mgr", ""]}) v1 -- 0x7f52d8000c90 con 0x7f52e00673a0 2024-01-15T18:13:30.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.302+0000 7f5304c00640 1 -- 172.21.15.27:0/3044184072 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f52d8000c90 con 0x7f52e00673a0 2024-01-15T18:13:30.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.310+0000 7f530f800640 1 -- 172.21.15.27:0/3044184072 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f52e00673a0 msgr2=0x7f52e0069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:30.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.310+0000 7f530f800640 1 --2- 172.21.15.27:0/3044184072 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f52e00673a0 0x7f52e0069860 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f52f8000f70 tx=0x7f52f80079c0 comp rx=0 tx=0).stop 2024-01-15T18:13:30.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.310+0000 7f530f800640 1 -- 172.21.15.27:0/3044184072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5308161a90 msgr2=0x7f5308155550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:30.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.310+0000 7f530f800640 1 --2- 172.21.15.27:0/3044184072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5308161a90 0x7f5308155550 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f530000b970 tx=0x7f530000be30 comp rx=0 tx=0).stop 2024-01-15T18:13:30.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.310+0000 7f530f800640 1 -- 172.21.15.27:0/3044184072 shutdown_connections 2024-01-15T18:13:30.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.310+0000 7f530f800640 1 --2- 172.21.15.27:0/3044184072 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f52e00673a0 0x7f52e0069860 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:30.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.310+0000 7f530f800640 1 --2- 172.21.15.27:0/3044184072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5308161a90 0x7f5308155550 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:30.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.310+0000 7f530f800640 1 --2- 172.21.15.27:0/3044184072 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f53080ce810 0x7f530814a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:30.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.310+0000 7f530f800640 1 --2- 172.21.15.27:0/3044184072 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f53080cdb20 0x7f5308137570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:30.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.310+0000 7f530f800640 1 -- 172.21.15.27:0/3044184072 >> 172.21.15.27:0/3044184072 conn(0x7f53080cb700 msgr2=0x7f5308158330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:30.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.310+0000 7f530f800640 1 -- 172.21.15.27:0/3044184072 shutdown_connections 2024-01-15T18:13:30.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.310+0000 7f530f800640 1 -- 172.21.15.27:0/3044184072 wait complete. 2024-01-15T18:13:30.343 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone519316 2024-01-15T18:13:30.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.590+0000 7f09a5200640 1 Processor -- start 2024-01-15T18:13:30.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.590+0000 7f09a5200640 1 -- start start 2024-01-15T18:13:30.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f09a5200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f09a0159f70 0x7f09a006daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:30.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f09a5200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a006e520 con 0x7f09a006e030 2024-01-15T18:13:30.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f09a5200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a006e800 con 0x7f09a00b2ff0 2024-01-15T18:13:30.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f09a5200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a006eae0 con 0x7f09a0159f70 2024-01-15T18:13:30.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f099ea00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f09a006e030 0x7f09a015a350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53952/0 (socket says 172.21.15.27:53952) 2024-01-15T18:13:30.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f099ea00640 1 -- 172.21.15.27:0/3858063615 learned_addr learned my addr 172.21.15.27:0/3858063615 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:13:30.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f099e000640 1 -- 172.21.15.27:0/3858063615 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3411410803 0 0) 0x7f09a006e520 con 0x7f09a006e030 2024-01-15T18:13:30.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f099e000640 1 -- 172.21.15.27:0/3858063615 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0980003530 con 0x7f09a006e030 2024-01-15T18:13:30.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f099e000640 1 -- 172.21.15.27:0/3858063615 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3836159235 0 0) 0x7f0980003530 con 0x7f09a006e030 2024-01-15T18:13:30.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f099e000640 1 -- 172.21.15.27:0/3858063615 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f09a0159f70 msgr2=0x7f09a006daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:30.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f099e000640 1 --2- 172.21.15.27:0/3858063615 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f09a0159f70 0x7f09a006daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:30.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f099e000640 1 -- 172.21.15.27:0/3858063615 >> v1:172.21.15.93:6789/0 conn(0x7f09a00b2ff0 legacy=0x7f09a00cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:30.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f099e000640 1 -- 172.21.15.27:0/3858063615 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a006edc0 con 0x7f09a006e030 2024-01-15T18:13:30.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f099e000640 1 -- 172.21.15.27:0/3858063615 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f0988001680 con 0x7f09a006e030 2024-01-15T18:13:30.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f099e000640 1 -- 172.21.15.27:0/3858063615 >> v1:172.21.15.27:6789/0 conn(0x7f09a006e030 legacy=0x7f09a015a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:30.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f099e000640 1 --2- 172.21.15.27:0/3858063615 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f09a0159f70 0x7f0980003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:30.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f099e000640 1 --2- 172.21.15.27:0/3858063615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0980004160 0x7f09800047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:30.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f099e000640 1 --2- 172.21.15.27:0/3858063615 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0980004e30 0x7f098000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:30.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f099e000640 1 -- 172.21.15.27:0/3858063615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a006eae0 con 0x7f0980004160 2024-01-15T18:13:30.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f099e000640 1 -- 172.21.15.27:0/3858063615 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09800055b0 con 0x7f0980004e30 2024-01-15T18:13:30.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f099e000640 1 -- 172.21.15.27:0/3858063615 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f098000bed0 con 0x7f09a0159f70 2024-01-15T18:13:30.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f099ea00640 1 --2- 172.21.15.27:0/3858063615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0980004160 0x7f09800047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:30.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f099ea00640 1 -- 172.21.15.27:0/3858063615 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f09a0159f70 msgr2=0x7f0980003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:30.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f099ea00640 1 --2- 172.21.15.27:0/3858063615 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f09a0159f70 0x7f0980003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:30.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f099ea00640 1 -- 172.21.15.27:0/3858063615 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0980004e30 msgr2=0x7f098000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:30.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f099ea00640 1 --2- 172.21.15.27:0/3858063615 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0980004e30 0x7f098000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:30.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.594+0000 7f099ea00640 1 -- 172.21.15.27:0/3858063615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0980003530 con 0x7f0980004160 2024-01-15T18:13:30.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.598+0000 7f099ea00640 1 --2- 172.21.15.27:0/3858063615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0980004160 0x7f09800047d0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f0988007c10 tx=0x7f0988023740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=558760517f6cea63 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:30.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.602+0000 7f099e000640 1 -- 172.21.15.27:0/3858063615 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0988005ae0 con 0x7f0980004160 2024-01-15T18:13:30.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.602+0000 7f099e000640 1 -- 172.21.15.27:0/3858063615 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09880af070 con 0x7f0980004160 2024-01-15T18:13:30.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.602+0000 7f099e000640 1 -- 172.21.15.27:0/3858063615 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0988005e80 con 0x7f0980004160 2024-01-15T18:13:30.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.602+0000 7f09a5200640 1 -- 172.21.15.27:0/3858063615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0980004160 msgr2=0x7f09800047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:30.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.602+0000 7f09a5200640 1 --2- 172.21.15.27:0/3858063615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0980004160 0x7f09800047d0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f0988007c10 tx=0x7f0988023740 comp rx=0 tx=0).stop 2024-01-15T18:13:30.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.602+0000 7f099fe00640 1 -- 172.21.15.27:0/3858063615 reap_dead start 2024-01-15T18:13:30.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.602+0000 7f09a5200640 1 -- 172.21.15.27:0/3858063615 shutdown_connections 2024-01-15T18:13:30.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.602+0000 7f09a5200640 1 -- 172.21.15.27:0/3858063615 >> 172.21.15.27:0/3858063615 conn(0x7f09a006bc30 msgr2=0x7f09a00b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:30.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.602+0000 7f09a5200640 1 -- 172.21.15.27:0/3858063615 shutdown_connections 2024-01-15T18:13:30.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.602+0000 7f09a5200640 1 -- 172.21.15.27:0/3858063615 wait complete. 2024-01-15T18:13:30.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.606+0000 7f09a5200640 1 Processor -- start 2024-01-15T18:13:30.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.606+0000 7f09a5200640 1 -- start start 2024-01-15T18:13:30.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.606+0000 7f09a5200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0980004160 0x7f09a0157b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:30.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.606+0000 7f09a5200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f09a014ebd0 0x7f09a01562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:30.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.606+0000 7f09a5200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f09a0151ce0 0x7f09a00b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:30.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.606+0000 7f09a5200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a0156800 con 0x7f0980004160 2024-01-15T18:13:30.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.606+0000 7f09a5200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a0156ad0 con 0x7f09a0151ce0 2024-01-15T18:13:30.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.606+0000 7f09a5200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a0156db0 con 0x7f09a014ebd0 2024-01-15T18:13:30.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.610+0000 7f099f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0980004160 0x7f09a0157b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:30.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.610+0000 7f099f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0980004160 0x7f09a0157b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40842/0 (socket says 172.21.15.27:40842) 2024-01-15T18:13:30.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.610+0000 7f099f400640 1 -- 172.21.15.27:0/3698806579 learned_addr learned my addr 172.21.15.27:0/3698806579 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:30.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.614+0000 7f099f400640 1 -- 172.21.15.27:0/3698806579 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f09a014ebd0 msgr2=0x7f09a01562c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:30.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.614+0000 7f099f400640 1 --2- 172.21.15.27:0/3698806579 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f09a014ebd0 0x7f09a01562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:30.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.614+0000 7f099f400640 1 -- 172.21.15.27:0/3698806579 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f09a0151ce0 msgr2=0x7f09a00b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:30.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.614+0000 7f099f400640 1 --2- 172.21.15.27:0/3698806579 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f09a0151ce0 0x7f09a00b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:30.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.614+0000 7f099f400640 1 -- 172.21.15.27:0/3698806579 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a0157090 con 0x7f0980004160 2024-01-15T18:13:30.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.614+0000 7f099f400640 1 --2- 172.21.15.27:0/3698806579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0980004160 0x7f09a0157b70 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f099000b4d0 tx=0x7f099000b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:30.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.614+0000 7f0997400640 1 -- 172.21.15.27:0/3698806579 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0990010870 con 0x7f0980004160 2024-01-15T18:13:30.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.614+0000 7f0997400640 1 -- 172.21.15.27:0/3698806579 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09900109d0 con 0x7f0980004160 2024-01-15T18:13:30.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.614+0000 7f0997400640 1 -- 172.21.15.27:0/3698806579 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0990010d60 con 0x7f0980004160 2024-01-15T18:13:30.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.614+0000 7f09a5200640 1 -- 172.21.15.27:0/3698806579 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a0157370 con 0x7f0980004160 2024-01-15T18:13:30.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.614+0000 7f09a5200640 1 -- 172.21.15.27:0/3698806579 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a0152190 con 0x7f0980004160 2024-01-15T18:13:30.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.618+0000 7f09a5200640 1 -- 172.21.15.27:0/3698806579 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0970005260 con 0x7f0980004160 2024-01-15T18:13:30.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.618+0000 7f0997400640 1 -- 172.21.15.27:0/3698806579 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f099000f3c0 con 0x7f0980004160 2024-01-15T18:13:30.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.626+0000 7f0997400640 1 --2- 172.21.15.27:0/3698806579 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f09780673f0 0x7f09780698b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:30.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.626+0000 7f0997400640 1 -- 172.21.15.27:0/3698806579 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f0990082d50 con 0x7f0980004160 2024-01-15T18:13:30.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.626+0000 7f099ea00640 1 --2- 172.21.15.27:0/3698806579 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f09780673f0 0x7f09780698b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:30.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.630+0000 7f099ea00640 1 --2- 172.21.15.27:0/3698806579 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f09780673f0 0x7f09780698b0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f09880b3000 tx=0x7f0988001940 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:30.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.630+0000 7f0997400640 1 -- 172.21.15.27:0/3698806579 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0990051a20 con 0x7f0980004160 2024-01-15T18:13:30.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:30.962+0000 7f09a5200640 1 -- 172.21.15.27:0/3698806579 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone519316", "target": ["mon-mgr", ""]}) v1 -- 0x7f0970000c90 con 0x7f09780673f0 2024-01-15T18:13:31.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.014+0000 7f0997400640 1 -- 172.21.15.27:0/3698806579 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0970000c90 con 0x7f09780673f0 2024-01-15T18:13:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.014+0000 7f09a5200640 1 -- 172.21.15.27:0/3698806579 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f09780673f0 msgr2=0x7f09780698b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.014+0000 7f09a5200640 1 --2- 172.21.15.27:0/3698806579 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f09780673f0 0x7f09780698b0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f09880b3000 tx=0x7f0988001940 comp rx=0 tx=0).stop 2024-01-15T18:13:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.014+0000 7f09a5200640 1 -- 172.21.15.27:0/3698806579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0980004160 msgr2=0x7f09a0157b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.014+0000 7f09a5200640 1 --2- 172.21.15.27:0/3698806579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0980004160 0x7f09a0157b70 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f099000b4d0 tx=0x7f099000b990 comp rx=0 tx=0).stop 2024-01-15T18:13:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.014+0000 7f09a5200640 1 -- 172.21.15.27:0/3698806579 shutdown_connections 2024-01-15T18:13:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.014+0000 7f09a5200640 1 --2- 172.21.15.27:0/3698806579 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f09780673f0 0x7f09780698b0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.014+0000 7f09a5200640 1 --2- 172.21.15.27:0/3698806579 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f09a0151ce0 0x7f09a00b3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.014+0000 7f09a5200640 1 --2- 172.21.15.27:0/3698806579 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f09a014ebd0 0x7f09a01562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.014+0000 7f09a5200640 1 --2- 172.21.15.27:0/3698806579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0980004160 0x7f09a0157b70 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:31.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.014+0000 7f09a5200640 1 -- 172.21.15.27:0/3698806579 >> 172.21.15.27:0/3698806579 conn(0x7f09a006bc30 msgr2=0x7f09a00cf670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:31.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.014+0000 7f09a5200640 1 -- 172.21.15.27:0/3698806579 shutdown_connections 2024-01-15T18:13:31.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.014+0000 7f09a5200640 1 -- 172.21.15.27:0/3698806579 wait complete. 2024-01-15T18:13:31.038 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone519317 2024-01-15T18:13:31.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.182+0000 7ff9a6e00640 1 Processor -- start 2024-01-15T18:13:31.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.182+0000 7ff9a6e00640 1 -- start start 2024-01-15T18:13:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.186+0000 7ff9a6e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff9a00cf450 0x7ff9a00d1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.186+0000 7ff9a6e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a00cefa0 con 0x7ff9a00b0650 2024-01-15T18:13:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.186+0000 7ff9a6e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a00ce240 con 0x7ff9a00cba60 2024-01-15T18:13:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.186+0000 7ff9a6e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9a00ccd70 con 0x7ff9a00cf450 2024-01-15T18:13:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.186+0000 7ff99fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff9a00cf450 0x7ff9a00d1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.186+0000 7ff9a4e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff9a00b0650 0x7ff9a00b0a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53964/0 (socket says 172.21.15.27:53964) 2024-01-15T18:13:31.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.186+0000 7ff9a4e00640 1 -- 172.21.15.27:0/3873476740 learned_addr learned my addr 172.21.15.27:0/3873476740 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:13:31.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.186+0000 7ff99fe00640 1 -- 172.21.15.27:0/3873476740 >> v1:172.21.15.93:6789/0 conn(0x7ff9a00cba60 legacy=0x7ff9a00cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:31.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.186+0000 7ff99fe00640 1 -- 172.21.15.27:0/3873476740 >> v1:172.21.15.27:6789/0 conn(0x7ff9a00b0650 legacy=0x7ff9a00b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:31.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.186+0000 7ff99ea00640 1 -- 172.21.15.27:0/3873476740 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469087228 0 0) 0x7ff9a00ce240 con 0x7ff9a00cba60 2024-01-15T18:13:31.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.186+0000 7ff99fe00640 1 -- 172.21.15.27:0/3873476740 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a00cc1f0 con 0x7ff9a00cf450 2024-01-15T18:13:31.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.190+0000 7ff99fe00640 1 --2- 172.21.15.27:0/3873476740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00cf450 0x7ff9a00d1a40 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7ff9900097c0 tx=0x7ff99001c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdb8da1c6f0f7821 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:31.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.190+0000 7ff99ea00640 1 -- 172.21.15.27:0/3873476740 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9900ab600 con 0x7ff9a00cf450 2024-01-15T18:13:31.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.190+0000 7ff99ea00640 1 -- 172.21.15.27:0/3873476740 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9900a7050 con 0x7ff9a00cf450 2024-01-15T18:13:31.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.190+0000 7ff99ea00640 1 -- 172.21.15.27:0/3873476740 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9900ab980 con 0x7ff9a00cf450 2024-01-15T18:13:31.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.190+0000 7ff9a6e00640 1 -- 172.21.15.27:0/3873476740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00cf450 msgr2=0x7ff9a00d1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:31.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.190+0000 7ff9a6e00640 1 --2- 172.21.15.27:0/3873476740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00cf450 0x7ff9a00d1a40 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7ff9900097c0 tx=0x7ff99001c310 comp rx=0 tx=0).stop 2024-01-15T18:13:31.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.190+0000 7ff9a6e00640 1 -- 172.21.15.27:0/3873476740 shutdown_connections 2024-01-15T18:13:31.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.190+0000 7ff9a6e00640 1 --2- 172.21.15.27:0/3873476740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00cf450 0x7ff9a00d1a40 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:31.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.190+0000 7ff9a6e00640 1 -- 172.21.15.27:0/3873476740 >> 172.21.15.27:0/3873476740 conn(0x7ff9a00d0e70 msgr2=0x7ff9a00d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:31.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.190+0000 7ff9a6e00640 1 -- 172.21.15.27:0/3873476740 shutdown_connections 2024-01-15T18:13:31.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.190+0000 7ff9a6e00640 1 -- 172.21.15.27:0/3873476740 wait complete. 2024-01-15T18:13:31.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.190+0000 7ff9a6e00640 1 Processor -- start 2024-01-15T18:13:31.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.190+0000 7ff9a6e00640 1 -- start start 2024-01-15T18:13:31.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.194+0000 7ff9a6e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00b0650 0x7ff9a01556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:31.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.194+0000 7ff9a6e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9a00cba60 0x7ff9a00c8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:31.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.194+0000 7ff9a6e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff9a00cf450 0x7ff9a0159d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:31.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.194+0000 7ff9a6e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a00c93a0 con 0x7ff9a00b0650 2024-01-15T18:13:31.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.194+0000 7ff9a6e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a00c9680 con 0x7ff9a00cf450 2024-01-15T18:13:31.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.194+0000 7ff9a6e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a00c9960 con 0x7ff9a00cba60 2024-01-15T18:13:31.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.194+0000 7ff99fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00b0650 0x7ff9a01556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:31.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.194+0000 7ff99fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00b0650 0x7ff9a01556f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40872/0 (socket says 172.21.15.27:40872) 2024-01-15T18:13:31.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.194+0000 7ff99fe00640 1 -- 172.21.15.27:0/2309410544 learned_addr learned my addr 172.21.15.27:0/2309410544 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:31.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.194+0000 7ff9a4e00640 1 --2- 172.21.15.27:0/2309410544 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff9a00cf450 0x7ff9a0159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:31.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.194+0000 7ff99f400640 1 --2- 172.21.15.27:0/2309410544 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9a00cba60 0x7ff9a00c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:31.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.194+0000 7ff99fe00640 1 -- 172.21.15.27:0/2309410544 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9a00cba60 msgr2=0x7ff9a00c8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:31.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.194+0000 7ff99fe00640 1 --2- 172.21.15.27:0/2309410544 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9a00cba60 0x7ff9a00c8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:31.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.194+0000 7ff99fe00640 1 -- 172.21.15.27:0/2309410544 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff9a00cf450 msgr2=0x7ff9a0159d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:31.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.194+0000 7ff99fe00640 1 --2- 172.21.15.27:0/2309410544 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff9a00cf450 0x7ff9a0159d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:31.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.194+0000 7ff99fe00640 1 -- 172.21.15.27:0/2309410544 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff990009420 con 0x7ff9a00b0650 2024-01-15T18:13:31.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.194+0000 7ff99f400640 1 --2- 172.21.15.27:0/2309410544 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9a00cba60 0x7ff9a00c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:13:31.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.194+0000 7ff99fe00640 1 --2- 172.21.15.27:0/2309410544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00b0650 0x7ff9a01556f0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7ff990018f20 tx=0x7ff99001ab50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:31.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.194+0000 7ff997e00640 1 -- 172.21.15.27:0/2309410544 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9900aa8b0 con 0x7ff9a00b0650 2024-01-15T18:13:31.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.194+0000 7ff997e00640 1 -- 172.21.15.27:0/2309410544 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9900aaa30 con 0x7ff9a00b0650 2024-01-15T18:13:31.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.194+0000 7ff9a6e00640 1 -- 172.21.15.27:0/2309410544 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a00c9c40 con 0x7ff9a00b0650 2024-01-15T18:13:31.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.194+0000 7ff997e00640 1 -- 172.21.15.27:0/2309410544 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9900aade0 con 0x7ff9a00b0650 2024-01-15T18:13:31.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.194+0000 7ff9a6e00640 1 -- 172.21.15.27:0/2309410544 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a00b6310 con 0x7ff9a00b0650 2024-01-15T18:13:31.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.194+0000 7ff9a6e00640 1 -- 172.21.15.27:0/2309410544 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9a00b64a0 con 0x7ff9a00b0650 2024-01-15T18:13:31.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.198+0000 7ff997e00640 1 -- 172.21.15.27:0/2309410544 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7ff9900c2460 con 0x7ff9a00b0650 2024-01-15T18:13:31.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.198+0000 7ff997e00640 1 --2- 172.21.15.27:0/2309410544 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff9780673a0 0x7ff978069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:31.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.198+0000 7ff997e00640 1 -- 172.21.15.27:0/2309410544 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7ff99011da90 con 0x7ff9a00b0650 2024-01-15T18:13:31.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.198+0000 7ff99f400640 1 --2- 172.21.15.27:0/2309410544 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff9780673a0 0x7ff978069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:31.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.198+0000 7ff99f400640 1 --2- 172.21.15.27:0/2309410544 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff9780673a0 0x7ff978069860 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ff9a01f69e0 tx=0x7ff988009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:31.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.202+0000 7ff997e00640 1 -- 172.21.15.27:0/2309410544 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff9a00b64a0 con 0x7ff9a00b0650 2024-01-15T18:13:31.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.478+0000 7ff9a6e00640 1 -- 172.21.15.27:0/2309410544 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone519317", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9a00cca90 con 0x7ff9780673a0 2024-01-15T18:13:31.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.922+0000 7ff997e00640 1 -- 172.21.15.27:0/2309410544 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff988001330 con 0x7ff9780673a0 2024-01-15T18:13:31.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.926+0000 7ff9a6e00640 1 -- 172.21.15.27:0/2309410544 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff9780673a0 msgr2=0x7ff978069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:31.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.926+0000 7ff9a6e00640 1 --2- 172.21.15.27:0/2309410544 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff9780673a0 0x7ff978069860 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ff9a01f69e0 tx=0x7ff988009040 comp rx=0 tx=0).stop 2024-01-15T18:13:31.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.926+0000 7ff9a6e00640 1 -- 172.21.15.27:0/2309410544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00b0650 msgr2=0x7ff9a01556f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:31.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.926+0000 7ff9a6e00640 1 --2- 172.21.15.27:0/2309410544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00b0650 0x7ff9a01556f0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7ff990018f20 tx=0x7ff99001ab50 comp rx=0 tx=0).stop 2024-01-15T18:13:31.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.926+0000 7ff9a6e00640 1 -- 172.21.15.27:0/2309410544 shutdown_connections 2024-01-15T18:13:31.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.926+0000 7ff9a6e00640 1 --2- 172.21.15.27:0/2309410544 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff9780673a0 0x7ff978069860 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:31.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.926+0000 7ff9a6e00640 1 --2- 172.21.15.27:0/2309410544 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff9a00cf450 0x7ff9a0159d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:31.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.926+0000 7ff9a6e00640 1 --2- 172.21.15.27:0/2309410544 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9a00cba60 0x7ff9a00c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:31.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.926+0000 7ff9a6e00640 1 --2- 172.21.15.27:0/2309410544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9a00b0650 0x7ff9a01556f0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:31.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.926+0000 7ff9a6e00640 1 -- 172.21.15.27:0/2309410544 >> 172.21.15.27:0/2309410544 conn(0x7ff9a00d0e70 msgr2=0x7ff9a00ccf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:31.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.926+0000 7ff9a6e00640 1 -- 172.21.15.27:0/2309410544 shutdown_connections 2024-01-15T18:13:31.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:31.926+0000 7ff9a6e00640 1 -- 172.21.15.27:0/2309410544 wait complete. 2024-01-15T18:13:31.968 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone519318 2024-01-15T18:13:32.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.214+0000 7f5b0d800640 1 Processor -- start 2024-01-15T18:13:32.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.214+0000 7f5b0d800640 1 -- start start 2024-01-15T18:13:32.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.214+0000 7f5b0d800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b080b5300 0x7f5b080721e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:32.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.214+0000 7f5b0d800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5b08071c90 0x7f5b080b56e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:32.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.214+0000 7f5b0d800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b080cdee0 con 0x7f5b080b1c80 2024-01-15T18:13:32.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.214+0000 7f5b0d800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b080ce1c0 con 0x7f5b08071c90 2024-01-15T18:13:32.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.214+0000 7f5b0d800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b080ce4a0 con 0x7f5b080b5300 2024-01-15T18:13:32.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.214+0000 7f5b06000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5b08071c90 0x7f5b080b56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:32.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.214+0000 7f5b06000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5b08071c90 0x7f5b080b56e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40882/0 (socket says 172.21.15.27:40882) 2024-01-15T18:13:32.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.214+0000 7f5b06000640 1 -- 172.21.15.27:0/1577688168 learned_addr learned my addr 172.21.15.27:0/1577688168 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:32.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.214+0000 7f5b06000640 1 -- 172.21.15.27:0/1577688168 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b080b5300 msgr2=0x7f5b080721e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:32.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.214+0000 7f5b06000640 1 --2- 172.21.15.27:0/1577688168 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b080b5300 0x7f5b080721e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:32.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.214+0000 7f5b06000640 1 -- 172.21.15.27:0/1577688168 >> v1:172.21.15.27:6789/0 conn(0x7f5b080b1c80 legacy=0x7f5b080b20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:32.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.214+0000 7f5b06000640 1 -- 172.21.15.27:0/1577688168 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b0815c130 con 0x7f5b08071c90 2024-01-15T18:13:32.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.214+0000 7f5b05600640 1 -- 172.21.15.27:0/1577688168 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2687630838 0 0) 0x7f5b080cdee0 con 0x7f5b080b1c80 2024-01-15T18:13:32.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.218+0000 7f5b06000640 1 --2- 172.21.15.27:0/1577688168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b08071c90 0x7f5b080b56e0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f5af0009870 tx=0x7f5af001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8548c73944fe15bc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:32.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.218+0000 7f5b05600640 1 -- 172.21.15.27:0/1577688168 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af00a95f0 con 0x7f5b08071c90 2024-01-15T18:13:32.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.218+0000 7f5b05600640 1 -- 172.21.15.27:0/1577688168 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af00a9750 con 0x7f5b08071c90 2024-01-15T18:13:32.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.218+0000 7f5b05600640 1 -- 172.21.15.27:0/1577688168 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af00a9ae0 con 0x7f5b08071c90 2024-01-15T18:13:32.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.222+0000 7f5b0d800640 1 -- 172.21.15.27:0/1577688168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b08071c90 msgr2=0x7f5b080b56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:32.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.222+0000 7f5b0d800640 1 --2- 172.21.15.27:0/1577688168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b08071c90 0x7f5b080b56e0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f5af0009870 tx=0x7f5af001c510 comp rx=0 tx=0).stop 2024-01-15T18:13:32.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.222+0000 7f5b0d800640 1 -- 172.21.15.27:0/1577688168 shutdown_connections 2024-01-15T18:13:32.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.222+0000 7f5b0d800640 1 --2- 172.21.15.27:0/1577688168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b08071c90 0x7f5b080b56e0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:32.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.222+0000 7f5b0d800640 1 --2- 172.21.15.27:0/1577688168 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b080b5300 0x7f5b080721e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:32.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.222+0000 7f5b0d800640 1 -- 172.21.15.27:0/1577688168 >> 172.21.15.27:0/1577688168 conn(0x7f5b0806b600 msgr2=0x7f5b0806ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:32.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.222+0000 7f5b0d800640 1 -- 172.21.15.27:0/1577688168 shutdown_connections 2024-01-15T18:13:32.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.222+0000 7f5b0d800640 1 -- 172.21.15.27:0/1577688168 wait complete. 2024-01-15T18:13:32.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.226+0000 7f5b0d800640 1 Processor -- start 2024-01-15T18:13:32.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.226+0000 7f5b0d800640 1 -- start start 2024-01-15T18:13:32.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.230+0000 7f5b0d800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b08071c90 0x7f5b0814db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:32.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.230+0000 7f5b0d800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b080b1c80 0x7f5b080b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:32.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.230+0000 7f5b0d800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b080b5300 0x7f5b08150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:32.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.230+0000 7f5b0d800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0814eb00 con 0x7f5b080b5300 2024-01-15T18:13:32.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.230+0000 7f5b0d800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0814e0a0 con 0x7f5b08071c90 2024-01-15T18:13:32.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.230+0000 7f5b0d800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b080b3200 con 0x7f5b080b1c80 2024-01-15T18:13:32.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.230+0000 7f5b07400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b080b5300 0x7f5b08150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:32.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.230+0000 7f5b06a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b08071c90 0x7f5b0814db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:32.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.230+0000 7f5b06000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b080b1c80 0x7f5b080b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:32.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.230+0000 7f5b06a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b08071c90 0x7f5b0814db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:50892/0 (socket says 172.21.15.27:50892) 2024-01-15T18:13:32.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.230+0000 7f5b06a00640 1 -- 172.21.15.27:0/3364297052 learned_addr learned my addr 172.21.15.27:0/3364297052 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:32.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.230+0000 7f5b07400640 1 -- 172.21.15.27:0/3364297052 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b080b1c80 msgr2=0x7f5b080b42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:32.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.230+0000 7f5b07400640 1 --2- 172.21.15.27:0/3364297052 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b080b1c80 0x7f5b080b42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:32.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.230+0000 7f5b07400640 1 -- 172.21.15.27:0/3364297052 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b08071c90 msgr2=0x7f5b0814db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:32.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.230+0000 7f5b07400640 1 --2- 172.21.15.27:0/3364297052 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b08071c90 0x7f5b0814db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:32.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.230+0000 7f5b07400640 1 -- 172.21.15.27:0/3364297052 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af00094d0 con 0x7f5b080b5300 2024-01-15T18:13:32.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.230+0000 7f5b07400640 1 --2- 172.21.15.27:0/3364297052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b080b5300 0x7f5b08150950 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f5b0000c700 tx=0x7f5b0000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:32.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.230+0000 7f5afea00640 1 -- 172.21.15.27:0/3364297052 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b00003db0 con 0x7f5b080b5300 2024-01-15T18:13:32.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.230+0000 7f5afea00640 1 -- 172.21.15.27:0/3364297052 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b0000ce10 con 0x7f5b080b5300 2024-01-15T18:13:32.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.230+0000 7f5b0d800640 1 -- 172.21.15.27:0/3364297052 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b080b34e0 con 0x7f5b080b5300 2024-01-15T18:13:32.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.230+0000 7f5afea00640 1 -- 172.21.15.27:0/3364297052 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b000052b0 con 0x7f5b080b5300 2024-01-15T18:13:32.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.230+0000 7f5b0d800640 1 -- 172.21.15.27:0/3364297052 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b080b3aa0 con 0x7f5b080b5300 2024-01-15T18:13:32.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.234+0000 7f5b0d800640 1 -- 172.21.15.27:0/3364297052 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b08073bd0 con 0x7f5b080b5300 2024-01-15T18:13:32.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.234+0000 7f5afea00640 1 -- 172.21.15.27:0/3364297052 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f5b00005410 con 0x7f5b080b5300 2024-01-15T18:13:32.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.238+0000 7f5afea00640 1 --2- 172.21.15.27:0/3364297052 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5ae00673d0 0x7f5ae0069890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:32.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.238+0000 7f5afea00640 1 -- 172.21.15.27:0/3364297052 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f5b000827e0 con 0x7f5b080b5300 2024-01-15T18:13:32.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.250+0000 7f5b06a00640 1 --2- 172.21.15.27:0/3364297052 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5ae00673d0 0x7f5ae0069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:32.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.250+0000 7f5afea00640 1 -- 172.21.15.27:0/3364297052 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5b00051240 con 0x7f5b080b5300 2024-01-15T18:13:32.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.254+0000 7f5b06a00640 1 --2- 172.21.15.27:0/3364297052 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5ae00673d0 0x7f5ae0069890 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5af8006fd0 tx=0x7f5af8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:32.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:32.530+0000 7f5b0d800640 1 -- 172.21.15.27:0/3364297052 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone519318", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b0804e850 con 0x7f5ae00673d0 2024-01-15T18:13:33.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.114+0000 7f5afea00640 1 -- 172.21.15.27:0/3364297052 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5b0804e850 con 0x7f5ae00673d0 2024-01-15T18:13:33.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.118+0000 7f5b0d800640 1 -- 172.21.15.27:0/3364297052 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5ae00673d0 msgr2=0x7f5ae0069890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:33.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.118+0000 7f5b0d800640 1 --2- 172.21.15.27:0/3364297052 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5ae00673d0 0x7f5ae0069890 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5af8006fd0 tx=0x7f5af8008040 comp rx=0 tx=0).stop 2024-01-15T18:13:33.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.118+0000 7f5b0d800640 1 -- 172.21.15.27:0/3364297052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b080b5300 msgr2=0x7f5b08150950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:33.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.118+0000 7f5b0d800640 1 --2- 172.21.15.27:0/3364297052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b080b5300 0x7f5b08150950 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f5b0000c700 tx=0x7f5b0000cbc0 comp rx=0 tx=0).stop 2024-01-15T18:13:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.118+0000 7f5b0d800640 1 -- 172.21.15.27:0/3364297052 shutdown_connections 2024-01-15T18:13:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.118+0000 7f5b0d800640 1 --2- 172.21.15.27:0/3364297052 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5ae00673d0 0x7f5ae0069890 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.118+0000 7f5b0d800640 1 --2- 172.21.15.27:0/3364297052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b080b5300 0x7f5b08150950 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.118+0000 7f5b0d800640 1 --2- 172.21.15.27:0/3364297052 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b080b1c80 0x7f5b080b42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.118+0000 7f5b0d800640 1 --2- 172.21.15.27:0/3364297052 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b08071c90 0x7f5b0814db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.118+0000 7f5b0d800640 1 -- 172.21.15.27:0/3364297052 >> 172.21.15.27:0/3364297052 conn(0x7f5b0806b600 msgr2=0x7f5b0815b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.118+0000 7f5b0d800640 1 -- 172.21.15.27:0/3364297052 shutdown_connections 2024-01-15T18:13:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.118+0000 7f5b0d800640 1 -- 172.21.15.27:0/3364297052 wait complete. 2024-01-15T18:13:33.160 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone519319 2024-01-15T18:13:33.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.398+0000 7f0077800640 1 Processor -- start 2024-01-15T18:13:33.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.398+0000 7f0077800640 1 -- start start 2024-01-15T18:13:33.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.402+0000 7f0077800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f00700b04b0 0x7f00700d19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:33.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.402+0000 7f0077800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f007009a120 0x7f00700b0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:33.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.402+0000 7f0077800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00700cc520 con 0x7f007015b370 2024-01-15T18:13:33.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.402+0000 7f0077800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f007015b750 con 0x7f007009a120 2024-01-15T18:13:33.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.402+0000 7f0077800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00700cc790 con 0x7f00700b04b0 2024-01-15T18:13:33.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.402+0000 7f0075800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f007009a120 0x7f00700b0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:33.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.402+0000 7f0074e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f007015b370 0x7f0070099a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53986/0 (socket says 172.21.15.27:53986) 2024-01-15T18:13:33.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.402+0000 7f0074e00640 1 -- 172.21.15.27:0/580339364 learned_addr learned my addr 172.21.15.27:0/580339364 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:13:33.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.402+0000 7f006fe00640 1 --2- 172.21.15.27:0/580339364 >> v2:172.21.15.93:3300/0 conn(0x7f00700b04b0 0x7f00700d19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:33.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.402+0000 7f0075800640 1 -- 172.21.15.27:0/580339364 >> v2:172.21.15.93:3300/0 conn(0x7f00700b04b0 msgr2=0x7f00700d19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:33.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.402+0000 7f0075800640 1 --2- 172.21.15.27:0/580339364 >> v2:172.21.15.93:3300/0 conn(0x7f00700b04b0 0x7f00700d19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:33.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.402+0000 7f0075800640 1 -- 172.21.15.27:0/580339364 >> v1:172.21.15.27:6789/0 conn(0x7f007015b370 legacy=0x7f0070099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:33.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.402+0000 7f0075800640 1 -- 172.21.15.27:0/580339364 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00700cca70 con 0x7f007009a120 2024-01-15T18:13:33.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.406+0000 7f0075800640 1 --2- 172.21.15.27:0/580339364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f007009a120 0x7f00700b0b50 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f0070071640 tx=0x7f006001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99bdfad5a8486d1b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:33.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.406+0000 7f006f400640 1 -- 172.21.15.27:0/580339364 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00600ab680 con 0x7f007009a120 2024-01-15T18:13:33.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.406+0000 7f006f400640 1 -- 172.21.15.27:0/580339364 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00600ab7e0 con 0x7f007009a120 2024-01-15T18:13:33.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.406+0000 7f006f400640 1 -- 172.21.15.27:0/580339364 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00600abb70 con 0x7f007009a120 2024-01-15T18:13:33.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.410+0000 7f0077800640 1 -- 172.21.15.27:0/580339364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f007009a120 msgr2=0x7f00700b0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:33.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.410+0000 7f0077800640 1 --2- 172.21.15.27:0/580339364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f007009a120 0x7f00700b0b50 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f0070071640 tx=0x7f006001c460 comp rx=0 tx=0).stop 2024-01-15T18:13:33.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.410+0000 7f0077800640 1 -- 172.21.15.27:0/580339364 shutdown_connections 2024-01-15T18:13:33.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.410+0000 7f0077800640 1 --2- 172.21.15.27:0/580339364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f007009a120 0x7f00700b0b50 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:33.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.410+0000 7f0077800640 1 --2- 172.21.15.27:0/580339364 >> v2:172.21.15.93:3300/0 conn(0x7f00700b04b0 0x7f00700d19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:33.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.410+0000 7f0077800640 1 -- 172.21.15.27:0/580339364 >> 172.21.15.27:0/580339364 conn(0x7f00700d0ed0 msgr2=0x7f00700d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:33.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.410+0000 7f0077800640 1 -- 172.21.15.27:0/580339364 shutdown_connections 2024-01-15T18:13:33.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.410+0000 7f0077800640 1 -- 172.21.15.27:0/580339364 wait complete. 2024-01-15T18:13:33.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.414+0000 7f0077800640 1 Processor -- start 2024-01-15T18:13:33.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.414+0000 7f0077800640 1 -- start start 2024-01-15T18:13:33.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.414+0000 7f0077800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f007009a120 0x7f00700b6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:33.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.414+0000 7f0077800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00700b04b0 0x7f0070156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:33.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.414+0000 7f0077800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f007015b370 0x7f0070159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:33.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.414+0000 7f0077800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0070158f40 con 0x7f00700b04b0 2024-01-15T18:13:33.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.414+0000 7f0077800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00700b69b0 con 0x7f007009a120 2024-01-15T18:13:33.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.414+0000 7f0077800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0070158300 con 0x7f007015b370 2024-01-15T18:13:33.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.414+0000 7f006fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00700b04b0 0x7f0070156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:33.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.414+0000 7f006fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00700b04b0 0x7f0070156460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40916/0 (socket says 172.21.15.27:40916) 2024-01-15T18:13:33.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.414+0000 7f006fe00640 1 -- 172.21.15.27:0/681252189 learned_addr learned my addr 172.21.15.27:0/681252189 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:33.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.418+0000 7f0075800640 1 --2- 172.21.15.27:0/681252189 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f007015b370 0x7f0070159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:33.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.418+0000 7f006fe00640 1 -- 172.21.15.27:0/681252189 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f007015b370 msgr2=0x7f0070159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:33.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.418+0000 7f006fe00640 1 --2- 172.21.15.27:0/681252189 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f007015b370 0x7f0070159360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:33.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.418+0000 7f006fe00640 1 -- 172.21.15.27:0/681252189 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f007009a120 msgr2=0x7f00700b6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:33.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.418+0000 7f006fe00640 1 --2- 172.21.15.27:0/681252189 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f007009a120 0x7f00700b6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:33.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.418+0000 7f006fe00640 1 -- 172.21.15.27:0/681252189 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0060009420 con 0x7f00700b04b0 2024-01-15T18:13:33.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.418+0000 7f006fe00640 1 --2- 172.21.15.27:0/681252189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00700b04b0 0x7f0070156460 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f005800d7d0 tx=0x7f005800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:33.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.418+0000 7f006cc00640 1 -- 172.21.15.27:0/681252189 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0058018070 con 0x7f00700b04b0 2024-01-15T18:13:33.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.418+0000 7f006cc00640 1 -- 172.21.15.27:0/681252189 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0058004020 con 0x7f00700b04b0 2024-01-15T18:13:33.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.418+0000 7f006cc00640 1 -- 172.21.15.27:0/681252189 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00580041d0 con 0x7f00700b04b0 2024-01-15T18:13:33.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.418+0000 7f0077800640 1 -- 172.21.15.27:0/681252189 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00701585a0 con 0x7f00700b04b0 2024-01-15T18:13:33.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.418+0000 7f0077800640 1 -- 172.21.15.27:0/681252189 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0070158b60 con 0x7f00700b04b0 2024-01-15T18:13:33.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.418+0000 7f0077800640 1 -- 172.21.15.27:0/681252189 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00701591b0 con 0x7f00700b04b0 2024-01-15T18:13:33.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.422+0000 7f006cc00640 1 -- 172.21.15.27:0/681252189 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f0058004330 con 0x7f00700b04b0 2024-01-15T18:13:33.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.422+0000 7f006cc00640 1 --2- 172.21.15.27:0/681252189 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f00480673a0 0x7f0048069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:33.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.422+0000 7f006cc00640 1 -- 172.21.15.27:0/681252189 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f0058083740 con 0x7f00700b04b0 2024-01-15T18:13:33.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.426+0000 7f0074e00640 1 --2- 172.21.15.27:0/681252189 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f00480673a0 0x7f0048069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:33.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.430+0000 7f006cc00640 1 -- 172.21.15.27:0/681252189 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0058052410 con 0x7f00700b04b0 2024-01-15T18:13:33.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.454+0000 7f0074e00640 1 --2- 172.21.15.27:0/681252189 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f00480673a0 0x7f0048069860 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0068001fe0 tx=0x7f00680015d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:33.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:33.770+0000 7f0077800640 1 -- 172.21.15.27:0/681252189 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone519319", "target": ["mon-mgr", ""]}) v1 -- 0x7f0070067e70 con 0x7f00480673a0 2024-01-15T18:13:34.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.050+0000 7f006cc00640 1 -- 172.21.15.27:0/681252189 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0070067e70 con 0x7f00480673a0 2024-01-15T18:13:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.054+0000 7f0077800640 1 -- 172.21.15.27:0/681252189 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f00480673a0 msgr2=0x7f0048069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.054+0000 7f0077800640 1 --2- 172.21.15.27:0/681252189 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f00480673a0 0x7f0048069860 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0068001fe0 tx=0x7f00680015d0 comp rx=0 tx=0).stop 2024-01-15T18:13:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.054+0000 7f0077800640 1 -- 172.21.15.27:0/681252189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00700b04b0 msgr2=0x7f0070156460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.054+0000 7f0077800640 1 --2- 172.21.15.27:0/681252189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00700b04b0 0x7f0070156460 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f005800d7d0 tx=0x7f005800dc90 comp rx=0 tx=0).stop 2024-01-15T18:13:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.054+0000 7f0077800640 1 -- 172.21.15.27:0/681252189 shutdown_connections 2024-01-15T18:13:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.054+0000 7f0077800640 1 --2- 172.21.15.27:0/681252189 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f00480673a0 0x7f0048069860 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:34.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.054+0000 7f0077800640 1 --2- 172.21.15.27:0/681252189 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f007015b370 0x7f0070159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:34.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.054+0000 7f0077800640 1 --2- 172.21.15.27:0/681252189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00700b04b0 0x7f0070156460 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:34.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.054+0000 7f0077800640 1 --2- 172.21.15.27:0/681252189 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f007009a120 0x7f00700b6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:34.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.054+0000 7f0077800640 1 -- 172.21.15.27:0/681252189 >> 172.21.15.27:0/681252189 conn(0x7f00700d0ed0 msgr2=0x7f00700ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:34.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.054+0000 7f0077800640 1 -- 172.21.15.27:0/681252189 shutdown_connections 2024-01-15T18:13:34.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.054+0000 7f0077800640 1 -- 172.21.15.27:0/681252189 wait complete. 2024-01-15T18:13:34.078 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone519320 2024-01-15T18:13:34.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.226+0000 7f52c5000640 1 Processor -- start 2024-01-15T18:13:34.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.226+0000 7f52c5000640 1 -- start start 2024-01-15T18:13:34.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.226+0000 7f52c5000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f52c015d3b0 0x7f52c00cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:34.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.226+0000 7f52c5000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f52c006b600 0x7f52c006bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:34.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.226+0000 7f52c5000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f52c006c220 0x7f52c0160f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:34.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.226+0000 7f52c5000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c006c6b0 con 0x7f52c015d3b0 2024-01-15T18:13:34.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.226+0000 7f52c5000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52c00b04b0 con 0x7f52c006b600 2024-01-15T18:13:34.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.226+0000 7f52c5000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c00d0160 con 0x7f52c006c220 2024-01-15T18:13:34.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.226+0000 7f52bde00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f52c015d3b0 0x7f52c00cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:34.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.226+0000 7f52bde00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f52c015d3b0 0x7f52c00cfc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40630/0 (socket says 172.21.15.27:40630) 2024-01-15T18:13:34.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.226+0000 7f52bde00640 1 -- 172.21.15.27:0/3932337130 learned_addr learned my addr 172.21.15.27:0/3932337130 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:34.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.226+0000 7f52bd400640 1 --2- 172.21.15.27:0/3932337130 >> v2:172.21.15.93:3300/0 conn(0x7f52c006b600 0x7f52c006bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:34.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.226+0000 7f52be800640 1 --2- 172.21.15.27:0/3932337130 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f52c006c220 0x7f52c0160f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:34.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.226+0000 7f52bde00640 1 -- 172.21.15.27:0/3932337130 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f52c006c220 msgr2=0x7f52c0160f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:34.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.226+0000 7f52bde00640 1 --2- 172.21.15.27:0/3932337130 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f52c006c220 0x7f52c0160f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:34.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.226+0000 7f52bde00640 1 -- 172.21.15.27:0/3932337130 >> v2:172.21.15.93:3300/0 conn(0x7f52c006b600 msgr2=0x7f52c006bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:34.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.226+0000 7f52bde00640 1 --2- 172.21.15.27:0/3932337130 >> v2:172.21.15.93:3300/0 conn(0x7f52c006b600 0x7f52c006bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:34.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.226+0000 7f52bde00640 1 -- 172.21.15.27:0/3932337130 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c00b0720 con 0x7f52c015d3b0 2024-01-15T18:13:34.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.230+0000 7f52bde00640 1 --2- 172.21.15.27:0/3932337130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c015d3b0 0x7f52c00cfc20 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f52c0070e80 tx=0x7f52b001c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1edeffbc895404f4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:34.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.230+0000 7f52bca00640 1 -- 172.21.15.27:0/3932337130 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b00ab600 con 0x7f52c015d3b0 2024-01-15T18:13:34.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.230+0000 7f52bca00640 1 -- 172.21.15.27:0/3932337130 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b00a7050 con 0x7f52c015d3b0 2024-01-15T18:13:34.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.230+0000 7f52bca00640 1 -- 172.21.15.27:0/3932337130 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b00ab980 con 0x7f52c015d3b0 2024-01-15T18:13:34.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.230+0000 7f52c5000640 1 -- 172.21.15.27:0/3932337130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c015d3b0 msgr2=0x7f52c00cfc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:34.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.230+0000 7f52c5000640 1 --2- 172.21.15.27:0/3932337130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c015d3b0 0x7f52c00cfc20 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f52c0070e80 tx=0x7f52b001c210 comp rx=0 tx=0).stop 2024-01-15T18:13:34.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.230+0000 7f52c5000640 1 -- 172.21.15.27:0/3932337130 shutdown_connections 2024-01-15T18:13:34.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.230+0000 7f52c5000640 1 --2- 172.21.15.27:0/3932337130 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f52c006c220 0x7f52c0160f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:34.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.230+0000 7f52c5000640 1 --2- 172.21.15.27:0/3932337130 >> v2:172.21.15.93:3300/0 conn(0x7f52c006b600 0x7f52c006bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:34.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.230+0000 7f52c5000640 1 --2- 172.21.15.27:0/3932337130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c015d3b0 0x7f52c00cfc20 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:34.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.230+0000 7f52c5000640 1 -- 172.21.15.27:0/3932337130 >> 172.21.15.27:0/3932337130 conn(0x7f52c00d0ed0 msgr2=0x7f52c00d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:34.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.230+0000 7f52c5000640 1 -- 172.21.15.27:0/3932337130 shutdown_connections 2024-01-15T18:13:34.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.234+0000 7f52c5000640 1 -- 172.21.15.27:0/3932337130 wait complete. 2024-01-15T18:13:34.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.234+0000 7f52c5000640 1 Processor -- start 2024-01-15T18:13:34.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.234+0000 7f52c5000640 1 -- start start 2024-01-15T18:13:34.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.234+0000 7f52c5000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f52c006b600 0x7f52c00c7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:34.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.234+0000 7f52c5000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52c006c220 0x7f52c0154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:34.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.234+0000 7f52c5000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c015d3b0 0x7f52c01546c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:34.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.234+0000 7f52c5000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c00c5220 con 0x7f52c015d3b0 2024-01-15T18:13:34.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.234+0000 7f52c5000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c00c4930 con 0x7f52c006c220 2024-01-15T18:13:34.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.234+0000 7f52c5000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52c015f390 con 0x7f52c006b600 2024-01-15T18:13:34.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.234+0000 7f52be800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c015d3b0 0x7f52c01546c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:34.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.234+0000 7f52be800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c015d3b0 0x7f52c01546c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40636/0 (socket says 172.21.15.27:40636) 2024-01-15T18:13:34.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.234+0000 7f52bd400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52c006c220 0x7f52c0154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:34.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.234+0000 7f52be800640 1 -- 172.21.15.27:0/2299355781 learned_addr learned my addr 172.21.15.27:0/2299355781 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:34.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.234+0000 7f52bde00640 1 --2- 172.21.15.27:0/2299355781 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f52c006b600 0x7f52c00c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:34.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.234+0000 7f52be800640 1 -- 172.21.15.27:0/2299355781 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f52c006b600 msgr2=0x7f52c00c7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:34.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.234+0000 7f52be800640 1 --2- 172.21.15.27:0/2299355781 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f52c006b600 0x7f52c00c7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:34.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.234+0000 7f52be800640 1 -- 172.21.15.27:0/2299355781 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52c006c220 msgr2=0x7f52c0154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:34.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.234+0000 7f52be800640 1 --2- 172.21.15.27:0/2299355781 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52c006c220 0x7f52c0154180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:34.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.234+0000 7f52be800640 1 -- 172.21.15.27:0/2299355781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b00094d0 con 0x7f52c015d3b0 2024-01-15T18:13:34.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.234+0000 7f52bd400640 1 --2- 172.21.15.27:0/2299355781 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52c006c220 0x7f52c0154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:13:34.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.234+0000 7f52bde00640 1 --2- 172.21.15.27:0/2299355781 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f52c006b600 0x7f52c00c7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:13:34.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.234+0000 7f52be800640 1 --2- 172.21.15.27:0/2299355781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c015d3b0 0x7f52c01546c0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f52b8007e40 tx=0x7f52b800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:34.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.238+0000 7f52b6000640 1 -- 172.21.15.27:0/2299355781 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b8014620 con 0x7f52c015d3b0 2024-01-15T18:13:34.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.238+0000 7f52b6000640 1 -- 172.21.15.27:0/2299355781 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b800ee90 con 0x7f52c015d3b0 2024-01-15T18:13:34.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.238+0000 7f52b6000640 1 -- 172.21.15.27:0/2299355781 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b80149e0 con 0x7f52c015d3b0 2024-01-15T18:13:34.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.238+0000 7f52c5000640 1 -- 172.21.15.27:0/2299355781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c00c4bd0 con 0x7f52c015d3b0 2024-01-15T18:13:34.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.238+0000 7f52c5000640 1 -- 172.21.15.27:0/2299355781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c00c7940 con 0x7f52c015d3b0 2024-01-15T18:13:34.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.238+0000 7f52c5000640 1 -- 172.21.15.27:0/2299355781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c006d550 con 0x7f52c015d3b0 2024-01-15T18:13:34.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.238+0000 7f52b6000640 1 -- 172.21.15.27:0/2299355781 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f52b8014b40 con 0x7f52c015d3b0 2024-01-15T18:13:34.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.238+0000 7f52b6000640 1 --2- 172.21.15.27:0/2299355781 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f52980673d0 0x7f5298069890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:34.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.238+0000 7f52b6000640 1 -- 172.21.15.27:0/2299355781 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f52b8083590 con 0x7f52c015d3b0 2024-01-15T18:13:34.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.238+0000 7f52bde00640 1 --2- 172.21.15.27:0/2299355781 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f52980673d0 0x7f5298069890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:34.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.242+0000 7f52bde00640 1 --2- 172.21.15.27:0/2299355781 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f52980673d0 0x7f5298069890 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f52b00ad000 tx=0x7f52b0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:34.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.242+0000 7f52b6000640 1 -- 172.21.15.27:0/2299355781 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f52b80b7280 con 0x7f52c015d3b0 2024-01-15T18:13:34.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:34.430+0000 7f52c5000640 1 -- 172.21.15.27:0/2299355781 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone519320", "target": ["mon-mgr", ""]}) v1 -- 0x7f52c00d0430 con 0x7f52980673d0 2024-01-15T18:13:36.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.478+0000 7f52b6000640 1 -- 172.21.15.27:0/2299355781 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f52b00ab8d0 con 0x7f52980673d0 2024-01-15T18:13:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.478+0000 7f52c5000640 1 -- 172.21.15.27:0/2299355781 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f52980673d0 msgr2=0x7f5298069890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.478+0000 7f52c5000640 1 --2- 172.21.15.27:0/2299355781 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f52980673d0 0x7f5298069890 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f52b00ad000 tx=0x7f52b0008040 comp rx=0 tx=0).stop 2024-01-15T18:13:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.478+0000 7f52c5000640 1 -- 172.21.15.27:0/2299355781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c015d3b0 msgr2=0x7f52c01546c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.478+0000 7f52c5000640 1 --2- 172.21.15.27:0/2299355781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c015d3b0 0x7f52c01546c0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f52b8007e40 tx=0x7f52b800e4b0 comp rx=0 tx=0).stop 2024-01-15T18:13:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.478+0000 7f52c5000640 1 -- 172.21.15.27:0/2299355781 shutdown_connections 2024-01-15T18:13:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.478+0000 7f52c5000640 1 --2- 172.21.15.27:0/2299355781 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f52980673d0 0x7f5298069890 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.478+0000 7f52c5000640 1 --2- 172.21.15.27:0/2299355781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52c015d3b0 0x7f52c01546c0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.478+0000 7f52c5000640 1 --2- 172.21.15.27:0/2299355781 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f52c006c220 0x7f52c0154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.478+0000 7f52c5000640 1 --2- 172.21.15.27:0/2299355781 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f52c006b600 0x7f52c00c7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:36.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.478+0000 7f52c5000640 1 -- 172.21.15.27:0/2299355781 >> 172.21.15.27:0/2299355781 conn(0x7f52c00d0ed0 msgr2=0x7f52c00d12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:36.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.482+0000 7f52c5000640 1 -- 172.21.15.27:0/2299355781 shutdown_connections 2024-01-15T18:13:36.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.482+0000 7f52c5000640 1 -- 172.21.15.27:0/2299355781 wait complete. 2024-01-15T18:13:36.506 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-15T18:13:36.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.646+0000 7ff09a600640 1 Processor -- start 2024-01-15T18:13:36.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.646+0000 7ff09a600640 1 -- start start 2024-01-15T18:13:36.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.650+0000 7ff09a600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff09406bb80 0x7ff0940cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:36.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.650+0000 7ff09a600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff0940cddc0 0x7ff0940b0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:36.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.650+0000 7ff09a600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0940b2770 con 0x7ff0940b2390 2024-01-15T18:13:36.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.650+0000 7ff09a600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0940ce270 con 0x7ff0940cddc0 2024-01-15T18:13:36.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.650+0000 7ff09a600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff09406c060 con 0x7ff09406bb80 2024-01-15T18:13:36.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.650+0000 7ff092a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff09406bb80 0x7ff0940cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:36.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.650+0000 7ff093e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff0940cddc0 0x7ff0940b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:36.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.650+0000 7ff093400640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7ff0940b2390 0x7ff0940cf780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:47156/0 (socket says 172.21.15.27:47156) 2024-01-15T18:13:36.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.650+0000 7ff092a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff09406bb80 0x7ff0940cd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:53246/0 (socket says 172.21.15.27:53246) 2024-01-15T18:13:36.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.650+0000 7ff093400640 1 -- 172.21.15.27:0/526407943 learned_addr learned my addr 172.21.15.27:0/526407943 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:13:36.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.650+0000 7ff093e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff0940cddc0 0x7ff0940b0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:38442/0 (socket says 172.21.15.27:38442) 2024-01-15T18:13:36.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.650+0000 7ff092a00640 1 -- 172.21.15.27:0/526407943 >> v2:172.21.15.93:3300/0 conn(0x7ff0940cddc0 msgr2=0x7ff0940b0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:36.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.650+0000 7ff092a00640 1 --2- 172.21.15.27:0/526407943 >> v2:172.21.15.93:3300/0 conn(0x7ff0940cddc0 0x7ff0940b0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:36.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.650+0000 7ff092a00640 1 -- 172.21.15.27:0/526407943 >> v1:172.21.15.93:6789/0 conn(0x7ff0940b2390 legacy=0x7ff0940cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:36.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.650+0000 7ff092a00640 1 -- 172.21.15.27:0/526407943 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09406c340 con 0x7ff09406bb80 2024-01-15T18:13:36.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.650+0000 7ff093e00640 1 --2- 172.21.15.27:0/526407943 >> v2:172.21.15.93:3300/0 conn(0x7ff0940cddc0 0x7ff0940b0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:13:36.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.654+0000 7ff092a00640 1 --2- 172.21.15.27:0/526407943 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff09406bb80 0x7ff0940cd880 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff07c0097c0 tx=0x7ff07c01c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9eeb7cbe71150e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:36.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.654+0000 7ff092000640 1 -- 172.21.15.27:0/526407943 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c0ab650 con 0x7ff09406bb80 2024-01-15T18:13:36.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.654+0000 7ff092000640 1 -- 172.21.15.27:0/526407943 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07c0ab7b0 con 0x7ff09406bb80 2024-01-15T18:13:36.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.654+0000 7ff092000640 1 -- 172.21.15.27:0/526407943 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c0abb40 con 0x7ff09406bb80 2024-01-15T18:13:36.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.654+0000 7ff09a600640 1 -- 172.21.15.27:0/526407943 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff09406bb80 msgr2=0x7ff0940cd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:36.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.654+0000 7ff09a600640 1 --2- 172.21.15.27:0/526407943 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff09406bb80 0x7ff0940cd880 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff07c0097c0 tx=0x7ff07c01c450 comp rx=0 tx=0).stop 2024-01-15T18:13:36.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.654+0000 7ff09a600640 1 -- 172.21.15.27:0/526407943 shutdown_connections 2024-01-15T18:13:36.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.654+0000 7ff09a600640 1 --2- 172.21.15.27:0/526407943 >> v2:172.21.15.93:3300/0 conn(0x7ff0940cddc0 0x7ff0940b0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:36.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.654+0000 7ff09a600640 1 --2- 172.21.15.27:0/526407943 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff09406bb80 0x7ff0940cd880 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:36.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.654+0000 7ff09a600640 1 -- 172.21.15.27:0/526407943 >> 172.21.15.27:0/526407943 conn(0x7ff0940b2a10 msgr2=0x7ff0940b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:36.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.654+0000 7ff09a600640 1 -- 172.21.15.27:0/526407943 shutdown_connections 2024-01-15T18:13:36.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.654+0000 7ff09a600640 1 -- 172.21.15.27:0/526407943 wait complete. 2024-01-15T18:13:36.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.658+0000 7ff09a600640 1 Processor -- start 2024-01-15T18:13:36.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.658+0000 7ff09a600640 1 -- start start 2024-01-15T18:13:36.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.658+0000 7ff09a600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff09406bb80 0x7ff09415d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:36.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.658+0000 7ff09a600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0940b2390 0x7ff09406e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:36.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.658+0000 7ff09a600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0940cddc0 0x7ff09415bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:36.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.658+0000 7ff09a600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0940c9bc0 con 0x7ff09406bb80 2024-01-15T18:13:36.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.658+0000 7ff09a600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0940c9ea0 con 0x7ff0940b2390 2024-01-15T18:13:36.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.658+0000 7ff09a600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0940ca180 con 0x7ff0940cddc0 2024-01-15T18:13:36.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.658+0000 7ff093400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff09406bb80 0x7ff09415d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:36.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.658+0000 7ff093400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff09406bb80 0x7ff09415d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40646/0 (socket says 172.21.15.27:40646) 2024-01-15T18:13:36.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.658+0000 7ff093400640 1 -- 172.21.15.27:0/3861315575 learned_addr learned my addr 172.21.15.27:0/3861315575 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:36.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.658+0000 7ff092a00640 1 --2- 172.21.15.27:0/3861315575 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0940b2390 0x7ff09406e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:36.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.658+0000 7ff093e00640 1 --2- 172.21.15.27:0/3861315575 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0940cddc0 0x7ff09415bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:36.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.658+0000 7ff093400640 1 -- 172.21.15.27:0/3861315575 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0940cddc0 msgr2=0x7ff09415bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:36.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.658+0000 7ff093400640 1 --2- 172.21.15.27:0/3861315575 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0940cddc0 0x7ff09415bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:36.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.658+0000 7ff093400640 1 -- 172.21.15.27:0/3861315575 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0940b2390 msgr2=0x7ff09406e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:36.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.658+0000 7ff093400640 1 --2- 172.21.15.27:0/3861315575 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0940b2390 0x7ff09406e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:36.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.658+0000 7ff093400640 1 -- 172.21.15.27:0/3861315575 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08c009a70 con 0x7ff09406bb80 2024-01-15T18:13:36.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.658+0000 7ff093400640 1 --2- 172.21.15.27:0/3861315575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff09406bb80 0x7ff09415d3e0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7ff08400c500 tx=0x7ff08400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:36.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.658+0000 7ff08b400640 1 -- 172.21.15.27:0/3861315575 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff084005cf0 con 0x7ff09406bb80 2024-01-15T18:13:36.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.658+0000 7ff08b400640 1 -- 172.21.15.27:0/3861315575 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff084005e50 con 0x7ff09406bb80 2024-01-15T18:13:36.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.658+0000 7ff09a600640 1 -- 172.21.15.27:0/3861315575 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff07c009420 con 0x7ff09406bb80 2024-01-15T18:13:36.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.658+0000 7ff08b400640 1 -- 172.21.15.27:0/3861315575 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0840072d0 con 0x7ff09406bb80 2024-01-15T18:13:36.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.658+0000 7ff09a600640 1 -- 172.21.15.27:0/3861315575 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0940ca740 con 0x7ff09406bb80 2024-01-15T18:13:36.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.658+0000 7ff09a600640 1 -- 172.21.15.27:0/3861315575 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0940d07c0 con 0x7ff09406bb80 2024-01-15T18:13:36.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.662+0000 7ff08b400640 1 -- 172.21.15.27:0/3861315575 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7ff084007430 con 0x7ff09406bb80 2024-01-15T18:13:36.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.662+0000 7ff08b400640 1 --2- 172.21.15.27:0/3861315575 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff06c0673a0 0x7ff06c069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:36.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.662+0000 7ff08b400640 1 -- 172.21.15.27:0/3861315575 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7ff084082510 con 0x7ff09406bb80 2024-01-15T18:13:36.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.662+0000 7ff092a00640 1 --2- 172.21.15.27:0/3861315575 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff06c0673a0 0x7ff06c069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:36.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.662+0000 7ff092a00640 1 --2- 172.21.15.27:0/3861315575 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff06c0673a0 0x7ff06c069860 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ff07c0af000 tx=0x7ff07c0a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:36.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.666+0000 7ff08b400640 1 -- 172.21.15.27:0/3861315575 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff084050fa0 con 0x7ff09406bb80 2024-01-15T18:13:36.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.882+0000 7ff09a600640 1 -- 172.21.15.27:0/3861315575 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff09404e850 con 0x7ff09406bb80 2024-01-15T18:13:36.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.882+0000 7ff08b400640 1 -- 172.21.15.27:0/3861315575 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2425 (secure 0 0 0) 0x7ff0840545c0 con 0x7ff09406bb80 2024-01-15T18:13:36.884 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:13:36.885 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:13:36.885 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:13:36.885 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:13:36.885 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:13:36.885 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:13:36.885 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:13:36.885 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:36.885 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:13:36.885 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:13:36.885 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:13:36.885 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:13:36.885 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:13:36.885 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:36.885 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:13:36.885 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:13:36.885 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:13:36.886 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:13:36.886 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:36.886 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 543, 2024-01-15T18:13:36.886 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:13:36.886 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:13:36.886 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:13:36.886 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:13:36.886 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:36.886 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:13:36.886 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 72, 2024-01-15T18:13:36.886 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:13:36.886 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:13:36.886 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:13:36.886 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:13:36.886 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:13:36.887 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:13:36.887 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:36.887 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:13:36.887 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:13:36.887 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:36.887 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:13:36.887 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:13:36.887 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:36.887 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:36.887 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:13:36.887 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:13:36.887 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 2809, 2024-01-15T18:13:36.887 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 11590646627, 2024-01-15T18:13:36.887 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 22686003200, 2024-01-15T18:13:36.887 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 750408110080, 2024-01-15T18:13:36.887 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:13:36.888 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 228104630, 2024-01-15T18:13:36.888 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 235128674, 2024-01-15T18:13:36.888 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 54, 2024-01-15T18:13:36.888 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 66 2024-01-15T18:13:36.888 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:36.888 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:13:36.888 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 48, 2024-01-15T18:13:36.888 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 5, 2024-01-15T18:13:36.888 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:13:36.888 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:13:36.888 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:13:36.888 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:13:36.888 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:13:36.888 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 5, 2024-01-15T18:13:36.889 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:13:36.889 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "c", 2024-01-15T18:13:36.889 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:13:36.889 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 7227 2024-01-15T18:13:36.889 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:36.889 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:36.889 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:13:36.889 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:36.889 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:13:36.889 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:13:36.889 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:13:36.889 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:13:36.889 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:13:36.889 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:13:36.889 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:13:36.889 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:13:36.890 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:13:36.890 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:36.890 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:13:36.890 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 16, 2024-01-15T18:13:36.890 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:13:08.281503+0000", 2024-01-15T18:13:36.890 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:13:36.890 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:36.890 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": { 2024-01-15T18:13:36.890 INFO:teuthology.orchestra.run.smithi027.stdout: "6eefee50-4f20-4629-a81e-c700e83869a8": { 2024-01-15T18:13:36.890 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "4 ongoing cloning jobs; Avg progress = 83.335% (38s)\n [=======================.....] (remaining: 7s)", 2024-01-15T18:13:36.890 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.83335000276565552, 2024-01-15T18:13:36.890 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:36.890 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:13:36.890 INFO:teuthology.orchestra.run.smithi027.stdout: "8fb3fd3d-b405-4726-bf6b-75de006e8a09": { 2024-01-15T18:13:36.891 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "18 ongoing+pending cloning jobs; Avg progress = 18.519% (38s)\n [=====.......................] (remaining: 2m)", 2024-01-15T18:13:36.891 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": 0.185188889503479, 2024-01-15T18:13:36.891 INFO:teuthology.orchestra.run.smithi027.stdout: "add_to_ceph_s": true 2024-01-15T18:13:36.891 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:36.891 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:13:36.891 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:13:36.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.886+0000 7ff09a600640 1 -- 172.21.15.27:0/3861315575 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff06c0673a0 msgr2=0x7ff06c069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:36.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.886+0000 7ff09a600640 1 --2- 172.21.15.27:0/3861315575 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff06c0673a0 0x7ff06c069860 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ff07c0af000 tx=0x7ff07c0a7000 comp rx=0 tx=0).stop 2024-01-15T18:13:36.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.886+0000 7ff09a600640 1 -- 172.21.15.27:0/3861315575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff09406bb80 msgr2=0x7ff09415d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:36.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.886+0000 7ff09a600640 1 --2- 172.21.15.27:0/3861315575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff09406bb80 0x7ff09415d3e0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7ff08400c500 tx=0x7ff08400c900 comp rx=0 tx=0).stop 2024-01-15T18:13:36.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.886+0000 7ff09a600640 1 -- 172.21.15.27:0/3861315575 shutdown_connections 2024-01-15T18:13:36.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.886+0000 7ff09a600640 1 --2- 172.21.15.27:0/3861315575 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff06c0673a0 0x7ff06c069860 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:36.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.886+0000 7ff09a600640 1 --2- 172.21.15.27:0/3861315575 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff0940cddc0 0x7ff09415bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:36.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.886+0000 7ff09a600640 1 --2- 172.21.15.27:0/3861315575 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff0940b2390 0x7ff09406e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:36.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.886+0000 7ff09a600640 1 --2- 172.21.15.27:0/3861315575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff09406bb80 0x7ff09415d3e0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:36.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.886+0000 7ff09a600640 1 -- 172.21.15.27:0/3861315575 >> 172.21.15.27:0/3861315575 conn(0x7ff0940b2a10 msgr2=0x7ff0940af800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:36.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.886+0000 7ff09a600640 1 -- 172.21.15.27:0/3861315575 shutdown_connections 2024-01-15T18:13:36.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:36.886+0000 7ff09a600640 1 -- 172.21.15.27:0/3861315575 wait complete. 2024-01-15T18:13:36.906 INFO:tasks.cephfs.test_volumes:new_onpen_p is less than onpen_p. 2024-01-15T18:13:36.906 INFO:tasks.cephfs.test_volumes:new_onpen_p = 0.185188889503479; onpen_p = 0.2857142984867096 2024-01-15T18:13:36.906 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone51931 2024-01-15T18:13:37.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.050+0000 7f37b0e00640 1 Processor -- start 2024-01-15T18:13:37.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.050+0000 7f37b0e00640 1 -- start start 2024-01-15T18:13:37.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.054+0000 7f37b0e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f37ac071550 0x7f37ac0cec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:37.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.054+0000 7f37b0e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f37ac06b630 0x7f37ac0cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:37.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.054+0000 7f37b0e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ac06b1a0 con 0x7f37ac15af80 2024-01-15T18:13:37.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.054+0000 7f37b0e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37ac15c370 con 0x7f37ac071550 2024-01-15T18:13:37.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.054+0000 7f37b0e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37ac06a590 con 0x7f37ac06b630 2024-01-15T18:13:37.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.054+0000 7f37a9e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f37ac15af80 0x7f37ac070e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37034/0 (socket says 172.21.15.27:37034) 2024-01-15T18:13:37.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.054+0000 7f37a9e00640 1 -- 172.21.15.27:0/2163468319 learned_addr learned my addr 172.21.15.27:0/2163468319 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:13:37.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.054+0000 7f37a9400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f37ac071550 0x7f37ac0cec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:37.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.054+0000 7f37aa800640 1 --2- 172.21.15.27:0/2163468319 >> v2:172.21.15.93:3300/0 conn(0x7f37ac06b630 0x7f37ac0cdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:37.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.054+0000 7f37a9400640 1 -- 172.21.15.27:0/2163468319 >> v2:172.21.15.93:3300/0 conn(0x7f37ac06b630 msgr2=0x7f37ac0cdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:37.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.054+0000 7f37a9400640 1 --2- 172.21.15.27:0/2163468319 >> v2:172.21.15.93:3300/0 conn(0x7f37ac06b630 0x7f37ac0cdc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:37.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.054+0000 7f37a9400640 1 -- 172.21.15.27:0/2163468319 >> v1:172.21.15.27:6789/0 conn(0x7f37ac15af80 legacy=0x7f37ac070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:37.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.054+0000 7f37a9400640 1 -- 172.21.15.27:0/2163468319 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ac06a870 con 0x7f37ac071550 2024-01-15T18:13:37.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.054+0000 7f37aa800640 1 --2- 172.21.15.27:0/2163468319 >> v2:172.21.15.93:3300/0 conn(0x7f37ac06b630 0x7f37ac0cdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:13:37.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.054+0000 7f37a9400640 1 --2- 172.21.15.27:0/2163468319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37ac071550 0x7f37ac0cec70 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f37ac0cc960 tx=0x7f379401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4074524427e6fe8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:37.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.058+0000 7f37a8a00640 1 -- 172.21.15.27:0/2163468319 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37940a7070 con 0x7f37ac071550 2024-01-15T18:13:37.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.058+0000 7f37a8a00640 1 -- 172.21.15.27:0/2163468319 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37940ab430 con 0x7f37ac071550 2024-01-15T18:13:37.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.058+0000 7f37a8a00640 1 -- 172.21.15.27:0/2163468319 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37940ab7c0 con 0x7f37ac071550 2024-01-15T18:13:37.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.058+0000 7f37b0e00640 1 -- 172.21.15.27:0/2163468319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37ac071550 msgr2=0x7f37ac0cec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:37.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.058+0000 7f37b0e00640 1 --2- 172.21.15.27:0/2163468319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37ac071550 0x7f37ac0cec70 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f37ac0cc960 tx=0x7f379401c430 comp rx=0 tx=0).stop 2024-01-15T18:13:37.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.058+0000 7f37b0e00640 1 -- 172.21.15.27:0/2163468319 shutdown_connections 2024-01-15T18:13:37.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.058+0000 7f37b0e00640 1 --2- 172.21.15.27:0/2163468319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37ac071550 0x7f37ac0cec70 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:37.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.058+0000 7f37b0e00640 1 --2- 172.21.15.27:0/2163468319 >> v2:172.21.15.93:3300/0 conn(0x7f37ac06b630 0x7f37ac0cdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:37.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.058+0000 7f37b0e00640 1 -- 172.21.15.27:0/2163468319 >> 172.21.15.27:0/2163468319 conn(0x7f37ac0b15b0 msgr2=0x7f37ac0b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:37.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.058+0000 7f37b0e00640 1 -- 172.21.15.27:0/2163468319 shutdown_connections 2024-01-15T18:13:37.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.058+0000 7f37b0e00640 1 -- 172.21.15.27:0/2163468319 wait complete. 2024-01-15T18:13:37.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.058+0000 7f37b0e00640 1 Processor -- start 2024-01-15T18:13:37.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.058+0000 7f37b0e00640 1 -- start start 2024-01-15T18:13:37.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.058+0000 7f37b0e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37ac06b630 0x7f37ac144f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:37.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.058+0000 7f37b0e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f37ac071550 0x7f37ac0b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:37.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.058+0000 7f37b0e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f37ac15af80 0x7f37ac0cbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:37.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.058+0000 7f37b0e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac153da0 con 0x7f37ac06b630 2024-01-15T18:13:37.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.058+0000 7f37b0e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac0b3800 con 0x7f37ac15af80 2024-01-15T18:13:37.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.058+0000 7f37b0e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac0b3ae0 con 0x7f37ac071550 2024-01-15T18:13:37.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.062+0000 7f37a9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37ac06b630 0x7f37ac144f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:37.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.062+0000 7f37a9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37ac06b630 0x7f37ac144f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40660/0 (socket says 172.21.15.27:40660) 2024-01-15T18:13:37.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.062+0000 7f37a9e00640 1 -- 172.21.15.27:0/2082900148 learned_addr learned my addr 172.21.15.27:0/2082900148 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:37.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.062+0000 7f37aa800640 1 --2- 172.21.15.27:0/2082900148 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f37ac15af80 0x7f37ac0cbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:37.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.062+0000 7f37a9400640 1 --2- 172.21.15.27:0/2082900148 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f37ac071550 0x7f37ac0b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:37.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.062+0000 7f37a9e00640 1 -- 172.21.15.27:0/2082900148 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f37ac071550 msgr2=0x7f37ac0b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:37.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.062+0000 7f37a9e00640 1 --2- 172.21.15.27:0/2082900148 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f37ac071550 0x7f37ac0b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:37.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.062+0000 7f37a9e00640 1 -- 172.21.15.27:0/2082900148 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f37ac15af80 msgr2=0x7f37ac0cbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:37.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.062+0000 7f37a9e00640 1 --2- 172.21.15.27:0/2082900148 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f37ac15af80 0x7f37ac0cbdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:37.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.062+0000 7f37a9e00640 1 -- 172.21.15.27:0/2082900148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3794009420 con 0x7f37ac06b630 2024-01-15T18:13:37.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.062+0000 7f37a9e00640 1 --2- 172.21.15.27:0/2082900148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37ac06b630 0x7f37ac144f70 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f379c00c370 tx=0x7f379c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:37.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.062+0000 7f37a2000640 1 -- 172.21.15.27:0/2082900148 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379c005cf0 con 0x7f37ac06b630 2024-01-15T18:13:37.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.062+0000 7f37a2000640 1 -- 172.21.15.27:0/2082900148 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f379c005e50 con 0x7f37ac06b630 2024-01-15T18:13:37.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.062+0000 7f37a2000640 1 -- 172.21.15.27:0/2082900148 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379c0072f0 con 0x7f37ac06b630 2024-01-15T18:13:37.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.062+0000 7f37b0e00640 1 -- 172.21.15.27:0/2082900148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37ac0b3dc0 con 0x7f37ac06b630 2024-01-15T18:13:37.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.062+0000 7f37b0e00640 1 -- 172.21.15.27:0/2082900148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37ac0b34f0 con 0x7f37ac06b630 2024-01-15T18:13:37.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.062+0000 7f37b0e00640 1 -- 172.21.15.27:0/2082900148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37ac0b4000 con 0x7f37ac06b630 2024-01-15T18:13:37.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.062+0000 7f37a2000640 1 -- 172.21.15.27:0/2082900148 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f379c007450 con 0x7f37ac06b630 2024-01-15T18:13:37.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.066+0000 7f37a2000640 1 --2- 172.21.15.27:0/2082900148 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f37840673a0 0x7f3784069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:37.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.066+0000 7f37a2000640 1 -- 172.21.15.27:0/2082900148 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f379c0824b0 con 0x7f37ac06b630 2024-01-15T18:13:37.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.066+0000 7f37a9400640 1 --2- 172.21.15.27:0/2082900148 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f37840673a0 0x7f3784069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:37.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.066+0000 7f37a9400640 1 --2- 172.21.15.27:0/2082900148 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f37840673a0 0x7f3784069860 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f37940af000 tx=0x7f37940a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:37.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.066+0000 7f37a2000640 1 -- 172.21.15.27:0/2082900148 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f379c050fc0 con 0x7f37ac06b630 2024-01-15T18:13:37.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.258+0000 7f37b0e00640 1 -- 172.21.15.27:0/2082900148 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone51931", "target": ["mon-mgr", ""]}) v1 -- 0x7f37ac04e850 con 0x7f37840673a0 2024-01-15T18:13:37.270 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:13:37.270+0000 7f59f2400640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-01-15T18:13:37.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.270+0000 7f37a2000640 1 -- 172.21.15.27:0/2082900148 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) v1 ==== 60+0+0 (secure 0 0 0) 0x7f37ac04e850 con 0x7f37840673a0 2024-01-15T18:13:37.271 INFO:teuthology.orchestra.run.smithi027.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-01-15T18:13:37.280 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-15T18:13:37.281 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone51932 2024-01-15T18:13:37.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.422+0000 7fb1e4800640 1 Processor -- start 2024-01-15T18:13:37.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.422+0000 7fb1e4800640 1 -- start start 2024-01-15T18:13:37.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.422+0000 7fb1e4800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb1dc0b1db0 0x7fb1dc0b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:37.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.422+0000 7fb1e4800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1dc0712b0 con 0x7fb1dc0ce050 2024-01-15T18:13:37.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.422+0000 7fb1e4800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1dc0ccfb0 con 0x7fb1dc06c430 2024-01-15T18:13:37.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.422+0000 7fb1e4800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1dc0cd290 con 0x7fb1dc0b1db0 2024-01-15T18:13:37.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.426+0000 7fb1e1400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb1dc0b1db0 0x7fb1dc0b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:37.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.426+0000 7fb1e2800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb1dc0ce050 0x7fb1dc06c810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37048/0 (socket says 172.21.15.27:37048) 2024-01-15T18:13:37.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.426+0000 7fb1e2800640 1 -- 172.21.15.27:0/312932442 learned_addr learned my addr 172.21.15.27:0/312932442 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:13:37.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.426+0000 7fb1e1400640 1 -- 172.21.15.27:0/312932442 >> v1:172.21.15.93:6789/0 conn(0x7fb1dc06c430 legacy=0x7fb1dc070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:37.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.426+0000 7fb1e1400640 1 -- 172.21.15.27:0/312932442 >> v1:172.21.15.27:6789/0 conn(0x7fb1dc0ce050 legacy=0x7fb1dc06c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:37.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.426+0000 7fb1e1400640 1 -- 172.21.15.27:0/312932442 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1dc0cd570 con 0x7fb1dc0b1db0 2024-01-15T18:13:37.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.426+0000 7fb1e1400640 1 --2- 172.21.15.27:0/312932442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1dc0b1db0 0x7fb1dc0b2240 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fb1c40097f0 tx=0x7fb1c401d5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7341a511e8ade1e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:37.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.426+0000 7fb1e0a00640 1 -- 172.21.15.27:0/312932442 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c40a9640 con 0x7fb1dc0b1db0 2024-01-15T18:13:37.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.426+0000 7fb1e0a00640 1 -- 172.21.15.27:0/312932442 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1c40a97a0 con 0x7fb1dc0b1db0 2024-01-15T18:13:37.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.426+0000 7fb1e0a00640 1 -- 172.21.15.27:0/312932442 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c40a9b30 con 0x7fb1dc0b1db0 2024-01-15T18:13:37.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.430+0000 7fb1e4800640 1 -- 172.21.15.27:0/312932442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1dc0b1db0 msgr2=0x7fb1dc0b2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:37.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.430+0000 7fb1e4800640 1 --2- 172.21.15.27:0/312932442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1dc0b1db0 0x7fb1dc0b2240 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fb1c40097f0 tx=0x7fb1c401d5f0 comp rx=0 tx=0).stop 2024-01-15T18:13:37.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.430+0000 7fb1e4800640 1 -- 172.21.15.27:0/312932442 shutdown_connections 2024-01-15T18:13:37.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.430+0000 7fb1e4800640 1 --2- 172.21.15.27:0/312932442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1dc0b1db0 0x7fb1dc0b2240 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:37.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.430+0000 7fb1e4800640 1 -- 172.21.15.27:0/312932442 >> 172.21.15.27:0/312932442 conn(0x7fb1dc0cb700 msgr2=0x7fb1dc0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:37.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.430+0000 7fb1e4800640 1 -- 172.21.15.27:0/312932442 shutdown_connections 2024-01-15T18:13:37.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.430+0000 7fb1e4800640 1 -- 172.21.15.27:0/312932442 wait complete. 2024-01-15T18:13:37.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.430+0000 7fb1e4800640 1 Processor -- start 2024-01-15T18:13:37.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.430+0000 7fb1e4800640 1 -- start start 2024-01-15T18:13:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.430+0000 7fb1e4800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb1dc06c430 0x7fb1dc1454e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.430+0000 7fb1e4800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1dc0b1db0 0x7fb1dc147f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.430+0000 7fb1e4800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1dc0ce050 0x7fb1dc148eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.430+0000 7fb1e4800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc145a20 con 0x7fb1dc0ce050 2024-01-15T18:13:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.430+0000 7fb1e4800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc147210 con 0x7fb1dc0b1db0 2024-01-15T18:13:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.430+0000 7fb1e4800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc156cd0 con 0x7fb1dc06c430 2024-01-15T18:13:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.430+0000 7fb1e2800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1dc0ce050 0x7fb1dc148eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.430+0000 7fb1e2800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1dc0ce050 0x7fb1dc148eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40682/0 (socket says 172.21.15.27:40682) 2024-01-15T18:13:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.430+0000 7fb1e2800640 1 -- 172.21.15.27:0/1899797049 learned_addr learned my addr 172.21.15.27:0/1899797049 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.434+0000 7fb1e1400640 1 --2- 172.21.15.27:0/1899797049 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1dc0b1db0 0x7fb1dc147f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:37.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.434+0000 7fb1e1e00640 1 --2- 172.21.15.27:0/1899797049 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb1dc06c430 0x7fb1dc1454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.434+0000 7fb1e2800640 1 -- 172.21.15.27:0/1899797049 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb1dc06c430 msgr2=0x7fb1dc1454e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.434+0000 7fb1e2800640 1 --2- 172.21.15.27:0/1899797049 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb1dc06c430 0x7fb1dc1454e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.434+0000 7fb1e2800640 1 -- 172.21.15.27:0/1899797049 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1dc0b1db0 msgr2=0x7fb1dc147f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.434+0000 7fb1e2800640 1 --2- 172.21.15.27:0/1899797049 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1dc0b1db0 0x7fb1dc147f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.434+0000 7fb1e2800640 1 -- 172.21.15.27:0/1899797049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c4009420 con 0x7fb1dc0ce050 2024-01-15T18:13:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.434+0000 7fb1e1400640 1 --2- 172.21.15.27:0/1899797049 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1dc0b1db0 0x7fb1dc147f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:13:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.434+0000 7fb1e1e00640 1 --2- 172.21.15.27:0/1899797049 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb1dc06c430 0x7fb1dc1454e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:13:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.434+0000 7fb1e2800640 1 --2- 172.21.15.27:0/1899797049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1dc0ce050 0x7fb1dc148eb0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fb1d400ca80 tx=0x7fb1d400cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.434+0000 7fb1da000640 1 -- 172.21.15.27:0/1899797049 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d400ac30 con 0x7fb1dc0ce050 2024-01-15T18:13:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.434+0000 7fb1da000640 1 -- 172.21.15.27:0/1899797049 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d400ad90 con 0x7fb1dc0ce050 2024-01-15T18:13:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.434+0000 7fb1e4800640 1 -- 172.21.15.27:0/1899797049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1dc147490 con 0x7fb1dc0ce050 2024-01-15T18:13:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.434+0000 7fb1da000640 1 -- 172.21.15.27:0/1899797049 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4012600 con 0x7fb1dc0ce050 2024-01-15T18:13:37.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.434+0000 7fb1e4800640 1 -- 172.21.15.27:0/1899797049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1dc147a50 con 0x7fb1dc0ce050 2024-01-15T18:13:37.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.434+0000 7fb1e4800640 1 -- 172.21.15.27:0/1899797049 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1dc147c40 con 0x7fb1dc0ce050 2024-01-15T18:13:37.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.434+0000 7fb1da000640 1 -- 172.21.15.27:0/1899797049 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7fb1d4007070 con 0x7fb1dc0ce050 2024-01-15T18:13:37.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.434+0000 7fb1da000640 1 --2- 172.21.15.27:0/1899797049 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb1b40673a0 0x7fb1b4069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:37.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.434+0000 7fb1da000640 1 -- 172.21.15.27:0/1899797049 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fb1d4082df0 con 0x7fb1dc0ce050 2024-01-15T18:13:37.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.438+0000 7fb1e1e00640 1 --2- 172.21.15.27:0/1899797049 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb1b40673a0 0x7fb1b4069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:37.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.438+0000 7fb1e1e00640 1 --2- 172.21.15.27:0/1899797049 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb1b40673a0 0x7fb1b4069860 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc000f70 tx=0x7fb1cc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:37.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.438+0000 7fb1da000640 1 -- 172.21.15.27:0/1899797049 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb1d4051900 con 0x7fb1dc0ce050 2024-01-15T18:13:37.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.634+0000 7fb1e4800640 1 -- 172.21.15.27:0/1899797049 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone51932", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1dc0af3d0 con 0x7fb1b40673a0 2024-01-15T18:13:37.643 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:13:37.642+0000 7f59f2400640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-01-15T18:13:37.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.642+0000 7fb1da000640 1 -- 172.21.15.27:0/1899797049 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) v1 ==== 60+0+0 (secure 0 0 0) 0x7fb1dc0af3d0 con 0x7fb1b40673a0 2024-01-15T18:13:37.644 INFO:teuthology.orchestra.run.smithi027.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-01-15T18:13:37.653 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-15T18:13:37.653 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone51933 2024-01-15T18:13:37.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.794+0000 7f28ae600640 1 Processor -- start 2024-01-15T18:13:37.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.794+0000 7f28ae600640 1 -- start start 2024-01-15T18:13:37.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.798+0000 7f28ae600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f28a815ad20 0x7f28a815b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:37.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.798+0000 7f28ae600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28a80715f0 0x7f28a815ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:37.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.798+0000 7f28ae600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a815a610 con 0x7f28a815d900 2024-01-15T18:13:37.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.798+0000 7f28ae600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28a815e3c0 con 0x7f28a815ad20 2024-01-15T18:13:37.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.798+0000 7f28ae600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28a815e6a0 con 0x7f28a80715f0 2024-01-15T18:13:37.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.798+0000 7f28a6a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f28a815d900 0x7f28a8070ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37050/0 (socket says 172.21.15.27:37050) 2024-01-15T18:13:37.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.798+0000 7f28a6a00640 1 -- 172.21.15.27:0/2865241601 learned_addr learned my addr 172.21.15.27:0/2865241601 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:13:37.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.798+0000 7f28a7e00640 1 --2- 172.21.15.27:0/2865241601 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28a80715f0 0x7f28a815ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:37.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.798+0000 7f28a7400640 1 --2- 172.21.15.27:0/2865241601 >> v2:172.21.15.93:3300/0 conn(0x7f28a815ad20 0x7f28a815b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:37.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.798+0000 7f28a6000640 1 -- 172.21.15.27:0/2865241601 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2459186335 0 0) 0x7f28a815a610 con 0x7f28a815d900 2024-01-15T18:13:37.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.798+0000 7f28a6000640 1 -- 172.21.15.27:0/2865241601 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2888003530 con 0x7f28a815d900 2024-01-15T18:13:37.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.798+0000 7f28a7400640 1 -- 172.21.15.27:0/2865241601 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28a80715f0 msgr2=0x7f28a815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:37.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.798+0000 7f28a7400640 1 --2- 172.21.15.27:0/2865241601 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28a80715f0 0x7f28a815ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:37.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.798+0000 7f28a7400640 1 -- 172.21.15.27:0/2865241601 >> v1:172.21.15.27:6789/0 conn(0x7f28a815d900 legacy=0x7f28a8070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:37.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.798+0000 7f28a7400640 1 -- 172.21.15.27:0/2865241601 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a815b6e0 con 0x7f28a815ad20 2024-01-15T18:13:37.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.798+0000 7f28a7e00640 1 --2- 172.21.15.27:0/2865241601 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28a80715f0 0x7f28a815ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:13:37.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.802+0000 7f28a7400640 1 --2- 172.21.15.27:0/2865241601 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28a815ad20 0x7f28a815b100 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f28980097f0 tx=0x7f289801c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1c698554362031 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:37.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.802+0000 7f28a6000640 1 -- 172.21.15.27:0/2865241601 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28980ab670 con 0x7f28a815ad20 2024-01-15T18:13:37.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.802+0000 7f28a6000640 1 -- 172.21.15.27:0/2865241601 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28980ab7d0 con 0x7f28a815ad20 2024-01-15T18:13:37.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.802+0000 7f28a6000640 1 -- 172.21.15.27:0/2865241601 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28980abb60 con 0x7f28a815ad20 2024-01-15T18:13:37.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.802+0000 7f28ae600640 1 -- 172.21.15.27:0/2865241601 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28a815ad20 msgr2=0x7f28a815b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:37.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.802+0000 7f28ae600640 1 --2- 172.21.15.27:0/2865241601 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28a815ad20 0x7f28a815b100 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f28980097f0 tx=0x7f289801c450 comp rx=0 tx=0).stop 2024-01-15T18:13:37.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.802+0000 7f28ae600640 1 -- 172.21.15.27:0/2865241601 shutdown_connections 2024-01-15T18:13:37.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.802+0000 7f28ae600640 1 --2- 172.21.15.27:0/2865241601 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28a80715f0 0x7f28a815ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:37.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.802+0000 7f28ae600640 1 --2- 172.21.15.27:0/2865241601 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28a815ad20 0x7f28a815b100 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:37.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.802+0000 7f28ae600640 1 -- 172.21.15.27:0/2865241601 >> 172.21.15.27:0/2865241601 conn(0x7f28a80cb700 msgr2=0x7f28a80cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:37.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.802+0000 7f28ae600640 1 -- 172.21.15.27:0/2865241601 shutdown_connections 2024-01-15T18:13:37.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.802+0000 7f28ae600640 1 -- 172.21.15.27:0/2865241601 wait complete. 2024-01-15T18:13:37.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.806+0000 7f28ae600640 1 Processor -- start 2024-01-15T18:13:37.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.806+0000 7f28ae600640 1 -- start start 2024-01-15T18:13:37.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.806+0000 7f28ae600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28a80715f0 0x7f28a8151fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:37.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.806+0000 7f28ae600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28a815ad20 0x7f28a806ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:37.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.806+0000 7f28ae600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28a815d900 0x7f28a8149d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:37.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.806+0000 7f28ae600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a814cb70 con 0x7f28a80715f0 2024-01-15T18:13:37.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.806+0000 7f28ae600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a8153c60 con 0x7f28a815d900 2024-01-15T18:13:37.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.806+0000 7f28ae600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28a8153f40 con 0x7f28a815ad20 2024-01-15T18:13:37.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.806+0000 7f28a7400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28a80715f0 0x7f28a8151fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:37.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.806+0000 7f28a7400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28a80715f0 0x7f28a8151fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40688/0 (socket says 172.21.15.27:40688) 2024-01-15T18:13:37.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.806+0000 7f28a7400640 1 -- 172.21.15.27:0/1938513595 learned_addr learned my addr 172.21.15.27:0/1938513595 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:37.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.806+0000 7f28a6a00640 1 --2- 172.21.15.27:0/1938513595 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28a815ad20 0x7f28a806ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:37.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.806+0000 7f28a7e00640 1 --2- 172.21.15.27:0/1938513595 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28a815d900 0x7f28a8149d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:37.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.806+0000 7f28a7400640 1 -- 172.21.15.27:0/1938513595 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28a815ad20 msgr2=0x7f28a806ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:37.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.806+0000 7f28a7400640 1 --2- 172.21.15.27:0/1938513595 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28a815ad20 0x7f28a806ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:37.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.806+0000 7f28a7400640 1 -- 172.21.15.27:0/1938513595 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28a815d900 msgr2=0x7f28a8149d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:37.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.806+0000 7f28a7400640 1 --2- 172.21.15.27:0/1938513595 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28a815d900 0x7f28a8149d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:37.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.806+0000 7f28a7400640 1 -- 172.21.15.27:0/1938513595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a0009a70 con 0x7f28a80715f0 2024-01-15T18:13:37.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.806+0000 7f28a7400640 1 --2- 172.21.15.27:0/1938513595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28a80715f0 0x7f28a8151fc0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f289801a080 tx=0x7f289801ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:37.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.806+0000 7f289f400640 1 -- 172.21.15.27:0/1938513595 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28980b5810 con 0x7f28a80715f0 2024-01-15T18:13:37.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.806+0000 7f289f400640 1 -- 172.21.15.27:0/1938513595 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28980b5970 con 0x7f28a80715f0 2024-01-15T18:13:37.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.806+0000 7f28ae600640 1 -- 172.21.15.27:0/1938513595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2898009420 con 0x7f28a80715f0 2024-01-15T18:13:37.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.806+0000 7f289f400640 1 -- 172.21.15.27:0/1938513595 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28980b5d20 con 0x7f28a80715f0 2024-01-15T18:13:37.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.806+0000 7f28ae600640 1 -- 172.21.15.27:0/1938513595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a806d210 con 0x7f28a80715f0 2024-01-15T18:13:37.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.806+0000 7f28ae600640 1 -- 172.21.15.27:0/1938513595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a8070900 con 0x7f28a80715f0 2024-01-15T18:13:37.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.810+0000 7f289f400640 1 -- 172.21.15.27:0/1938513595 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f28980bf410 con 0x7f28a80715f0 2024-01-15T18:13:37.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.810+0000 7f289f400640 1 --2- 172.21.15.27:0/1938513595 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f28800673a0 0x7f2880069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:37.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.810+0000 7f289f400640 1 -- 172.21.15.27:0/1938513595 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f289811dec0 con 0x7f28a80715f0 2024-01-15T18:13:37.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.810+0000 7f28a6a00640 1 --2- 172.21.15.27:0/1938513595 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f28800673a0 0x7f2880069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.810+0000 7f28a6a00640 1 --2- 172.21.15.27:0/1938513595 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f28800673a0 0x7f2880069860 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2890007fd0 tx=0x7f2890009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:37.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:37.814+0000 7f289f400640 1 -- 172.21.15.27:0/1938513595 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f28980ec990 con 0x7f28a80715f0 2024-01-15T18:13:38.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:38.006+0000 7f28ae600640 1 -- 172.21.15.27:0/1938513595 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone51933", "target": ["mon-mgr", ""]}) v1 -- 0x7f28a8153dd0 con 0x7f28800673a0 2024-01-15T18:13:41.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.190+0000 7f289f400640 1 -- 172.21.15.27:0/1938513595 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f28a8153dd0 con 0x7f28800673a0 2024-01-15T18:13:41.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.194+0000 7f28ae600640 1 -- 172.21.15.27:0/1938513595 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f28800673a0 msgr2=0x7f2880069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:41.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.194+0000 7f28ae600640 1 --2- 172.21.15.27:0/1938513595 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f28800673a0 0x7f2880069860 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2890007fd0 tx=0x7f2890009040 comp rx=0 tx=0).stop 2024-01-15T18:13:41.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.194+0000 7f28ae600640 1 -- 172.21.15.27:0/1938513595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28a80715f0 msgr2=0x7f28a8151fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:41.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.194+0000 7f28ae600640 1 --2- 172.21.15.27:0/1938513595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28a80715f0 0x7f28a8151fc0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f289801a080 tx=0x7f289801ac90 comp rx=0 tx=0).stop 2024-01-15T18:13:41.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.194+0000 7f28ae600640 1 -- 172.21.15.27:0/1938513595 shutdown_connections 2024-01-15T18:13:41.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.194+0000 7f28ae600640 1 --2- 172.21.15.27:0/1938513595 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f28800673a0 0x7f2880069860 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:41.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.194+0000 7f28ae600640 1 --2- 172.21.15.27:0/1938513595 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28a815d900 0x7f28a8149d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:41.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.194+0000 7f28ae600640 1 --2- 172.21.15.27:0/1938513595 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28a815ad20 0x7f28a806ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:41.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.194+0000 7f28ae600640 1 --2- 172.21.15.27:0/1938513595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28a80715f0 0x7f28a8151fc0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:41.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.194+0000 7f28ae600640 1 -- 172.21.15.27:0/1938513595 >> 172.21.15.27:0/1938513595 conn(0x7f28a80cb700 msgr2=0x7f28a8070ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:41.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.194+0000 7f28ae600640 1 -- 172.21.15.27:0/1938513595 shutdown_connections 2024-01-15T18:13:41.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.194+0000 7f28ae600640 1 -- 172.21.15.27:0/1938513595 wait complete. 2024-01-15T18:13:41.230 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone51934 2024-01-15T18:13:41.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.454+0000 7fba49a00640 1 Processor -- start 2024-01-15T18:13:41.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.454+0000 7fba49a00640 1 -- start start 2024-01-15T18:13:41.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.454+0000 7fba49a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fba440cfc30 0x7fba440cc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:41.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.454+0000 7fba49a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba440cb4d0 con 0x7fba440cde10 2024-01-15T18:13:41.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.454+0000 7fba49a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba440d0120 con 0x7fba4415d680 2024-01-15T18:13:41.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.454+0000 7fba49a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba440d0400 con 0x7fba440cfc30 2024-01-15T18:13:41.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.454+0000 7fba42000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fba440cde10 0x7fba440cbf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37058/0 (socket says 172.21.15.27:37058) 2024-01-15T18:13:41.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.454+0000 7fba42000640 1 -- 172.21.15.27:0/894813772 learned_addr learned my addr 172.21.15.27:0/894813772 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:13:41.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.454+0000 7fba43400640 1 --2- 172.21.15.27:0/894813772 >> v2:172.21.15.93:3300/0 conn(0x7fba440cfc30 0x7fba440cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:41.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.458+0000 7fba41600640 1 -- 172.21.15.27:0/894813772 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 268896669 0 0) 0x7fba440cb4d0 con 0x7fba440cde10 2024-01-15T18:13:41.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.458+0000 7fba41600640 1 -- 172.21.15.27:0/894813772 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba24003530 con 0x7fba440cde10 2024-01-15T18:13:41.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.458+0000 7fba41600640 1 -- 172.21.15.27:0/894813772 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3532583397 0 0) 0x7fba24003530 con 0x7fba440cde10 2024-01-15T18:13:41.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.458+0000 7fba41600640 1 -- 172.21.15.27:0/894813772 >> v2:172.21.15.93:3300/0 conn(0x7fba440cfc30 msgr2=0x7fba440cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:41.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.458+0000 7fba41600640 1 --2- 172.21.15.27:0/894813772 >> v2:172.21.15.93:3300/0 conn(0x7fba440cfc30 0x7fba440cc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:41.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.458+0000 7fba41600640 1 -- 172.21.15.27:0/894813772 >> v1:172.21.15.93:6789/0 conn(0x7fba4415d680 legacy=0x7fba440cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:41.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.458+0000 7fba41600640 1 -- 172.21.15.27:0/894813772 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba440cb770 con 0x7fba440cde10 2024-01-15T18:13:41.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.458+0000 7fba41600640 1 -- 172.21.15.27:0/894813772 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fba2c0016a0 con 0x7fba440cde10 2024-01-15T18:13:41.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.458+0000 7fba41600640 1 -- 172.21.15.27:0/894813772 >> v1:172.21.15.27:6789/0 conn(0x7fba440cde10 legacy=0x7fba440cbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:41.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.458+0000 7fba41600640 1 --2- 172.21.15.27:0/894813772 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fba24003a40 0x7fba24003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:41.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.458+0000 7fba41600640 1 --2- 172.21.15.27:0/894813772 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fba240047d0 0x7fba24004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:41.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.458+0000 7fba41600640 1 --2- 172.21.15.27:0/894813772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba240052b0 0x7fba2400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:41.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.458+0000 7fba41600640 1 -- 172.21.15.27:0/894813772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba440d0400 con 0x7fba240052b0 2024-01-15T18:13:41.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.458+0000 7fba41600640 1 -- 172.21.15.27:0/894813772 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba2400c1b0 con 0x7fba240047d0 2024-01-15T18:13:41.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.458+0000 7fba41600640 1 -- 172.21.15.27:0/894813772 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba2400c370 con 0x7fba24003a40 2024-01-15T18:13:41.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.458+0000 7fba43400640 1 --2- 172.21.15.27:0/894813772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba240052b0 0x7fba2400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:41.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.458+0000 7fba43400640 1 -- 172.21.15.27:0/894813772 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fba24003a40 msgr2=0x7fba24003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:41.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.458+0000 7fba43400640 1 --2- 172.21.15.27:0/894813772 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fba24003a40 0x7fba24003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:41.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.458+0000 7fba42000640 1 --2- 172.21.15.27:0/894813772 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fba240047d0 0x7fba24004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:41.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.458+0000 7fba43400640 1 -- 172.21.15.27:0/894813772 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fba240047d0 msgr2=0x7fba24004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:41.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.458+0000 7fba43400640 1 --2- 172.21.15.27:0/894813772 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fba240047d0 0x7fba24004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:41.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.458+0000 7fba43400640 1 -- 172.21.15.27:0/894813772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba24003530 con 0x7fba240052b0 2024-01-15T18:13:41.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.458+0000 7fba42000640 1 --2- 172.21.15.27:0/894813772 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fba240047d0 0x7fba24004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:13:41.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.462+0000 7fba43400640 1 --2- 172.21.15.27:0/894813772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba240052b0 0x7fba2400b990 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fba3c007330 tx=0x7fba3c01ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5661a9f6f26bbfd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:41.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.462+0000 7fba43400640 1 -- 172.21.15.27:0/894813772 reap_dead start 2024-01-15T18:13:41.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.462+0000 7fba41600640 1 -- 172.21.15.27:0/894813772 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3c0a5d40 con 0x7fba240052b0 2024-01-15T18:13:41.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.462+0000 7fba41600640 1 -- 172.21.15.27:0/894813772 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3c0ad440 con 0x7fba240052b0 2024-01-15T18:13:41.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.462+0000 7fba41600640 1 -- 172.21.15.27:0/894813772 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3c0ad7f0 con 0x7fba240052b0 2024-01-15T18:13:41.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.462+0000 7fba49a00640 1 -- 172.21.15.27:0/894813772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba240052b0 msgr2=0x7fba2400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:41.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.462+0000 7fba49a00640 1 --2- 172.21.15.27:0/894813772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba240052b0 0x7fba2400b990 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fba3c007330 tx=0x7fba3c01ce40 comp rx=0 tx=0).stop 2024-01-15T18:13:41.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.462+0000 7fba49a00640 1 -- 172.21.15.27:0/894813772 shutdown_connections 2024-01-15T18:13:41.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.462+0000 7fba49a00640 1 --2- 172.21.15.27:0/894813772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba240052b0 0x7fba2400b990 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:41.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.462+0000 7fba49a00640 1 -- 172.21.15.27:0/894813772 >> 172.21.15.27:0/894813772 conn(0x7fba440d0e80 msgr2=0x7fba440d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:41.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.462+0000 7fba49a00640 1 -- 172.21.15.27:0/894813772 shutdown_connections 2024-01-15T18:13:41.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.462+0000 7fba49a00640 1 -- 172.21.15.27:0/894813772 wait complete. 2024-01-15T18:13:41.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.466+0000 7fba49a00640 1 Processor -- start 2024-01-15T18:13:41.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.466+0000 7fba49a00640 1 -- start start 2024-01-15T18:13:41.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.470+0000 7fba49a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fba240052b0 0x7fba44148c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:41.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.470+0000 7fba49a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fba4414bdb0 0x7fba44160630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:41.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.470+0000 7fba49a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba4406f600 0x7fba4406fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:41.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.470+0000 7fba49a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba441f2a90 con 0x7fba4406f600 2024-01-15T18:13:41.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.470+0000 7fba49a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4406ffc0 con 0x7fba240052b0 2024-01-15T18:13:41.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.470+0000 7fba49a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba4415f590 con 0x7fba4414bdb0 2024-01-15T18:13:41.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.470+0000 7fba43400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba4406f600 0x7fba4406fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:41.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.470+0000 7fba43400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba4406f600 0x7fba4406fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40702/0 (socket says 172.21.15.27:40702) 2024-01-15T18:13:41.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.470+0000 7fba43400640 1 -- 172.21.15.27:0/445336026 learned_addr learned my addr 172.21.15.27:0/445336026 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:41.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.470+0000 7fba43400640 1 -- 172.21.15.27:0/445336026 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fba4414bdb0 msgr2=0x7fba44160630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:41.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.470+0000 7fba43400640 1 --2- 172.21.15.27:0/445336026 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fba4414bdb0 0x7fba44160630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:41.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.470+0000 7fba43400640 1 -- 172.21.15.27:0/445336026 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fba240052b0 msgr2=0x7fba44148c10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-15T18:13:41.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.470+0000 7fba43400640 1 --2- 172.21.15.27:0/445336026 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fba240052b0 0x7fba44148c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:41.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.470+0000 7fba43400640 1 -- 172.21.15.27:0/445336026 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba4415f870 con 0x7fba4406f600 2024-01-15T18:13:41.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.470+0000 7fba43400640 1 --2- 172.21.15.27:0/445336026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba4406f600 0x7fba4406fa80 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fba3c01ce80 tx=0x7fba3c019be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:41.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.470+0000 7fba3aa00640 1 -- 172.21.15.27:0/445336026 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3c0ad9c0 con 0x7fba4406f600 2024-01-15T18:13:41.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.470+0000 7fba3aa00640 1 -- 172.21.15.27:0/445336026 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3c0adb40 con 0x7fba4406f600 2024-01-15T18:13:41.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.470+0000 7fba3aa00640 1 -- 172.21.15.27:0/445336026 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3c0b6410 con 0x7fba4406f600 2024-01-15T18:13:41.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.470+0000 7fba49a00640 1 -- 172.21.15.27:0/445336026 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba4415fb50 con 0x7fba4406f600 2024-01-15T18:13:41.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.470+0000 7fba49a00640 1 -- 172.21.15.27:0/445336026 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba44160110 con 0x7fba4406f600 2024-01-15T18:13:41.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.470+0000 7fba49a00640 1 -- 172.21.15.27:0/445336026 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba44160330 con 0x7fba4406f600 2024-01-15T18:13:41.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.474+0000 7fba3aa00640 1 -- 172.21.15.27:0/445336026 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7fba3c0b6570 con 0x7fba4406f600 2024-01-15T18:13:41.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.474+0000 7fba3aa00640 1 --2- 172.21.15.27:0/445336026 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fba1c0673f0 0x7fba1c0698b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:41.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.474+0000 7fba3aa00640 1 -- 172.21.15.27:0/445336026 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fba3c11d8e0 con 0x7fba4406f600 2024-01-15T18:13:41.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.478+0000 7fba3aa00640 1 -- 172.21.15.27:0/445336026 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fba3c0ec5b0 con 0x7fba4406f600 2024-01-15T18:13:41.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.482+0000 7fba42a00640 1 --2- 172.21.15.27:0/445336026 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fba1c0673f0 0x7fba1c0698b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:41.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.486+0000 7fba42a00640 1 --2- 172.21.15.27:0/445336026 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fba1c0673f0 0x7fba1c0698b0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fba34007fd0 tx=0x7fba340079a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:41.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.750+0000 7fba49a00640 1 -- 172.21.15.27:0/445336026 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone51934", "target": ["mon-mgr", ""]}) v1 -- 0x7fba440d19a0 con 0x7fba1c0673f0 2024-01-15T18:13:41.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.918+0000 7fba3aa00640 1 -- 172.21.15.27:0/445336026 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fba34001230 con 0x7fba1c0673f0 2024-01-15T18:13:41.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.922+0000 7fba49a00640 1 -- 172.21.15.27:0/445336026 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fba1c0673f0 msgr2=0x7fba1c0698b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:41.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.922+0000 7fba49a00640 1 --2- 172.21.15.27:0/445336026 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fba1c0673f0 0x7fba1c0698b0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fba34007fd0 tx=0x7fba340079a0 comp rx=0 tx=0).stop 2024-01-15T18:13:41.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.922+0000 7fba49a00640 1 -- 172.21.15.27:0/445336026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba4406f600 msgr2=0x7fba4406fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:41.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.922+0000 7fba49a00640 1 --2- 172.21.15.27:0/445336026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba4406f600 0x7fba4406fa80 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fba3c01ce80 tx=0x7fba3c019be0 comp rx=0 tx=0).stop 2024-01-15T18:13:41.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.922+0000 7fba49a00640 1 -- 172.21.15.27:0/445336026 shutdown_connections 2024-01-15T18:13:41.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.922+0000 7fba49a00640 1 --2- 172.21.15.27:0/445336026 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fba1c0673f0 0x7fba1c0698b0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:41.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.922+0000 7fba49a00640 1 --2- 172.21.15.27:0/445336026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba4406f600 0x7fba4406fa80 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:41.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.922+0000 7fba49a00640 1 --2- 172.21.15.27:0/445336026 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fba4414bdb0 0x7fba44160630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:41.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.922+0000 7fba49a00640 1 --2- 172.21.15.27:0/445336026 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fba240052b0 0x7fba44148c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:41.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.922+0000 7fba49a00640 1 -- 172.21.15.27:0/445336026 >> 172.21.15.27:0/445336026 conn(0x7fba440d0e80 msgr2=0x7fba440d1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:41.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.922+0000 7fba49a00640 1 -- 172.21.15.27:0/445336026 shutdown_connections 2024-01-15T18:13:41.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:41.922+0000 7fba49a00640 1 -- 172.21.15.27:0/445336026 wait complete. 2024-01-15T18:13:41.943 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone51935 2024-01-15T18:13:42.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.086+0000 7efc0a000640 1 Processor -- start 2024-01-15T18:13:42.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.086+0000 7efc0a000640 1 -- start start 2024-01-15T18:13:42.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.086+0000 7efc0a000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efc0415d680 0x7efc040cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:42.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.086+0000 7efc0a000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc0406c750 con 0x7efc040ccf80 2024-01-15T18:13:42.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.086+0000 7efc0a000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc040cb4d0 con 0x7efc040cfc30 2024-01-15T18:13:42.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.086+0000 7efc0a000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc040d0120 con 0x7efc0415d680 2024-01-15T18:13:42.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.086+0000 7efc03400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efc0415d680 0x7efc040cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:42.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.086+0000 7efc03e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7efc040ccf80 0x7efc0406c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37066/0 (socket says 172.21.15.27:37066) 2024-01-15T18:13:42.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.086+0000 7efc03e00640 1 -- 172.21.15.27:0/4050096958 learned_addr learned my addr 172.21.15.27:0/4050096958 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:13:42.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.086+0000 7efc03400640 1 -- 172.21.15.27:0/4050096958 >> v1:172.21.15.93:6789/0 conn(0x7efc040cfc30 legacy=0x7efc0406b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:42.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.086+0000 7efc03400640 1 -- 172.21.15.27:0/4050096958 >> v1:172.21.15.27:6789/0 conn(0x7efc040ccf80 legacy=0x7efc0406c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:42.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.086+0000 7efc03400640 1 -- 172.21.15.27:0/4050096958 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc040d0400 con 0x7efc0415d680 2024-01-15T18:13:42.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.086+0000 7efc02000640 1 -- 172.21.15.27:0/4050096958 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991547357 0 0) 0x7efc0406c750 con 0x7efc040ccf80 2024-01-15T18:13:42.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.090+0000 7efc03400640 1 --2- 172.21.15.27:0/4050096958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc0415d680 0x7efc040cd700 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7efbf40097c0 tx=0x7efbf401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60564f0c0367a888 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:42.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.090+0000 7efc02000640 1 -- 172.21.15.27:0/4050096958 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf40ab650 con 0x7efc0415d680 2024-01-15T18:13:42.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.090+0000 7efc02000640 1 -- 172.21.15.27:0/4050096958 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf40ab7b0 con 0x7efc0415d680 2024-01-15T18:13:42.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.090+0000 7efc02000640 1 -- 172.21.15.27:0/4050096958 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf40abb40 con 0x7efc0415d680 2024-01-15T18:13:42.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.090+0000 7efc0a000640 1 -- 172.21.15.27:0/4050096958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc0415d680 msgr2=0x7efc040cd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:42.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.090+0000 7efc0a000640 1 --2- 172.21.15.27:0/4050096958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc0415d680 0x7efc040cd700 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7efbf40097c0 tx=0x7efbf401c450 comp rx=0 tx=0).stop 2024-01-15T18:13:42.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.090+0000 7efc0a000640 1 -- 172.21.15.27:0/4050096958 shutdown_connections 2024-01-15T18:13:42.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.090+0000 7efc0a000640 1 --2- 172.21.15.27:0/4050096958 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc0415d680 0x7efc040cd700 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:42.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.090+0000 7efc0a000640 1 -- 172.21.15.27:0/4050096958 >> 172.21.15.27:0/4050096958 conn(0x7efc040d0e80 msgr2=0x7efc040d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:42.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.090+0000 7efc0a000640 1 -- 172.21.15.27:0/4050096958 shutdown_connections 2024-01-15T18:13:42.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.090+0000 7efc0a000640 1 -- 172.21.15.27:0/4050096958 wait complete. 2024-01-15T18:13:42.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.094+0000 7efc0a000640 1 Processor -- start 2024-01-15T18:13:42.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.094+0000 7efc0a000640 1 -- start start 2024-01-15T18:13:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.094+0000 7efc0a000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efc040ccf80 0x7efc04156630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.094+0000 7efc0a000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efc040cfc30 0x7efc0414c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.094+0000 7efc0a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc0415d680 0x7efc0415fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.094+0000 7efc0a000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc041554d0 con 0x7efc0415d680 2024-01-15T18:13:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.094+0000 7efc0a000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc041557b0 con 0x7efc040cfc30 2024-01-15T18:13:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.094+0000 7efc0a000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc04155a90 con 0x7efc040ccf80 2024-01-15T18:13:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.094+0000 7efc03e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc0415d680 0x7efc0415fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.094+0000 7efc03e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc0415d680 0x7efc0415fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40724/0 (socket says 172.21.15.27:40724) 2024-01-15T18:13:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.094+0000 7efc03e00640 1 -- 172.21.15.27:0/2919511136 learned_addr learned my addr 172.21.15.27:0/2919511136 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:42.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.094+0000 7efc02a00640 1 --2- 172.21.15.27:0/2919511136 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efc040cfc30 0x7efc0414c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:42.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.094+0000 7efc03400640 1 --2- 172.21.15.27:0/2919511136 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efc040ccf80 0x7efc04156630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:42.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.094+0000 7efc03e00640 1 -- 172.21.15.27:0/2919511136 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efc040ccf80 msgr2=0x7efc04156630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:42.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.094+0000 7efc03e00640 1 --2- 172.21.15.27:0/2919511136 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efc040ccf80 0x7efc04156630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:42.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.094+0000 7efc03e00640 1 -- 172.21.15.27:0/2919511136 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efc040cfc30 msgr2=0x7efc0414c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:42.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.094+0000 7efc03e00640 1 --2- 172.21.15.27:0/2919511136 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efc040cfc30 0x7efc0414c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:42.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.094+0000 7efc03e00640 1 -- 172.21.15.27:0/2919511136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf4009420 con 0x7efc0415d680 2024-01-15T18:13:42.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.094+0000 7efc03e00640 1 --2- 172.21.15.27:0/2919511136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc0415d680 0x7efc0415fe40 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7efbfc00cb30 tx=0x7efbfc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:42.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.094+0000 7efbfb400640 1 -- 172.21.15.27:0/2919511136 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc00aca0 con 0x7efc0415d680 2024-01-15T18:13:42.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.098+0000 7efbfb400640 1 -- 172.21.15.27:0/2919511136 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbfc00ae20 con 0x7efc0415d680 2024-01-15T18:13:42.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.098+0000 7efc0a000640 1 -- 172.21.15.27:0/2919511136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc04155d70 con 0x7efc0415d680 2024-01-15T18:13:42.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.098+0000 7efbfb400640 1 -- 172.21.15.27:0/2919511136 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc012640 con 0x7efc0415d680 2024-01-15T18:13:42.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.098+0000 7efc0a000640 1 -- 172.21.15.27:0/2919511136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc041551c0 con 0x7efc0415d680 2024-01-15T18:13:42.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.098+0000 7efc0a000640 1 -- 172.21.15.27:0/2919511136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc04155f00 con 0x7efc0415d680 2024-01-15T18:13:42.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.098+0000 7efbfb400640 1 -- 172.21.15.27:0/2919511136 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7efbfc007070 con 0x7efc0415d680 2024-01-15T18:13:42.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.098+0000 7efbfb400640 1 --2- 172.21.15.27:0/2919511136 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efbdc0673a0 0x7efbdc069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:42.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.098+0000 7efbfb400640 1 -- 172.21.15.27:0/2919511136 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7efbfc082ef0 con 0x7efc0415d680 2024-01-15T18:13:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.098+0000 7efc03400640 1 --2- 172.21.15.27:0/2919511136 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efbdc0673a0 0x7efbdc069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:42.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.102+0000 7efc03400640 1 --2- 172.21.15.27:0/2919511136 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efbdc0673a0 0x7efbdc069860 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7efbf40af000 tx=0x7efbf40a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:42.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.102+0000 7efbfb400640 1 -- 172.21.15.27:0/2919511136 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7efbfc051a00 con 0x7efc0415d680 2024-01-15T18:13:42.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:42.298+0000 7efc0a000640 1 -- 172.21.15.27:0/2919511136 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone51935", "target": ["mon-mgr", ""]}) v1 -- 0x7efc04155920 con 0x7efbdc0673a0 2024-01-15T18:13:50.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.426+0000 7efbfb400640 1 -- 172.21.15.27:0/2919511136 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7efc04155920 con 0x7efbdc0673a0 2024-01-15T18:13:50.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.430+0000 7efc0a000640 1 -- 172.21.15.27:0/2919511136 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efbdc0673a0 msgr2=0x7efbdc069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:50.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.430+0000 7efc0a000640 1 --2- 172.21.15.27:0/2919511136 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efbdc0673a0 0x7efbdc069860 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7efbf40af000 tx=0x7efbf40a7000 comp rx=0 tx=0).stop 2024-01-15T18:13:50.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.430+0000 7efc0a000640 1 -- 172.21.15.27:0/2919511136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc0415d680 msgr2=0x7efc0415fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:50.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.430+0000 7efc0a000640 1 --2- 172.21.15.27:0/2919511136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc0415d680 0x7efc0415fe40 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7efbfc00cb30 tx=0x7efbfc00a060 comp rx=0 tx=0).stop 2024-01-15T18:13:50.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.430+0000 7efc0a000640 1 -- 172.21.15.27:0/2919511136 shutdown_connections 2024-01-15T18:13:50.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.430+0000 7efc0a000640 1 --2- 172.21.15.27:0/2919511136 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efbdc0673a0 0x7efbdc069860 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:50.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.430+0000 7efc0a000640 1 --2- 172.21.15.27:0/2919511136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc0415d680 0x7efc0415fe40 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:50.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.430+0000 7efc0a000640 1 --2- 172.21.15.27:0/2919511136 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efc040cfc30 0x7efc0414c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:50.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.430+0000 7efc0a000640 1 --2- 172.21.15.27:0/2919511136 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efc040ccf80 0x7efc04156630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:50.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.430+0000 7efc0a000640 1 -- 172.21.15.27:0/2919511136 >> 172.21.15.27:0/2919511136 conn(0x7efc040d0e80 msgr2=0x7efc040ae840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:50.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.430+0000 7efc0a000640 1 -- 172.21.15.27:0/2919511136 shutdown_connections 2024-01-15T18:13:50.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.430+0000 7efc0a000640 1 -- 172.21.15.27:0/2919511136 wait complete. 2024-01-15T18:13:50.454 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone51936 2024-01-15T18:13:50.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.598+0000 7f3c06600640 1 Processor -- start 2024-01-15T18:13:50.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.598+0000 7f3c06600640 1 -- start start 2024-01-15T18:13:50.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.602+0000 7f3c06600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f3c0015d190 0x7f3c000cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:50.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.602+0000 7f3c06600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3c0006c520 0x7f3c000cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:50.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.602+0000 7f3c06600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c000b0d70 con 0x7f3c0015d570 2024-01-15T18:13:50.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.602+0000 7f3c06600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c000d0540 con 0x7f3c0015d190 2024-01-15T18:13:50.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.602+0000 7f3c06600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c000b0440 con 0x7f3c0006c520 2024-01-15T18:13:50.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.602+0000 7f3bffe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3c0006c520 0x7f3c000cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:50.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.602+0000 7f3bff400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f3c0015d190 0x7f3c000cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:50.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.602+0000 7f3bffe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3c0006c520 0x7f3c000cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:59014/0 (socket says 172.21.15.27:59014) 2024-01-15T18:13:50.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.602+0000 7f3bfea00640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7f3c0015d570 0x7f3c0006be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:60656/0 (socket says 172.21.15.27:60656) 2024-01-15T18:13:50.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.602+0000 7f3bffe00640 1 -- 172.21.15.27:0/1882396333 learned_addr learned my addr 172.21.15.27:0/1882396333 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:50.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.602+0000 7f3bffe00640 1 -- 172.21.15.27:0/1882396333 >> v2:172.21.15.93:3300/0 conn(0x7f3c0015d190 msgr2=0x7f3c000cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:50.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.602+0000 7f3bffe00640 1 --2- 172.21.15.27:0/1882396333 >> v2:172.21.15.93:3300/0 conn(0x7f3c0015d190 0x7f3c000cfcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:50.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.602+0000 7f3bffe00640 1 -- 172.21.15.27:0/1882396333 >> v1:172.21.15.93:6789/0 conn(0x7f3c0015d570 legacy=0x7f3c0006be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:50.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.602+0000 7f3bffe00640 1 -- 172.21.15.27:0/1882396333 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c000b0720 con 0x7f3c0006c520 2024-01-15T18:13:50.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.606+0000 7f3bffe00640 1 --2- 172.21.15.27:0/1882396333 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3c0006c520 0x7f3c000cd5e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8009e50 tx=0x7f3bf801c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b759bc7beb6ce0b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:50.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.606+0000 7f3bfe000640 1 -- 172.21.15.27:0/1882396333 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf80a5870 con 0x7f3c0006c520 2024-01-15T18:13:50.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.606+0000 7f3bfe000640 1 -- 172.21.15.27:0/1882396333 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf80a59d0 con 0x7f3c0006c520 2024-01-15T18:13:50.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.606+0000 7f3bfe000640 1 -- 172.21.15.27:0/1882396333 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf80a5d60 con 0x7f3c0006c520 2024-01-15T18:13:50.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.606+0000 7f3c06600640 1 -- 172.21.15.27:0/1882396333 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3c0006c520 msgr2=0x7f3c000cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:50.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.606+0000 7f3c06600640 1 --2- 172.21.15.27:0/1882396333 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3c0006c520 0x7f3c000cd5e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8009e50 tx=0x7f3bf801c9e0 comp rx=0 tx=0).stop 2024-01-15T18:13:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.606+0000 7f3c06600640 1 -- 172.21.15.27:0/1882396333 shutdown_connections 2024-01-15T18:13:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.606+0000 7f3c06600640 1 --2- 172.21.15.27:0/1882396333 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3c0006c520 0x7f3c000cd5e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.606+0000 7f3c06600640 1 --2- 172.21.15.27:0/1882396333 >> v2:172.21.15.93:3300/0 conn(0x7f3c0015d190 0x7f3c000cfcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.606+0000 7f3c06600640 1 -- 172.21.15.27:0/1882396333 >> 172.21.15.27:0/1882396333 conn(0x7f3c000d0e70 msgr2=0x7f3c000d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.606+0000 7f3c06600640 1 -- 172.21.15.27:0/1882396333 shutdown_connections 2024-01-15T18:13:50.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.606+0000 7f3c06600640 1 -- 172.21.15.27:0/1882396333 wait complete. 2024-01-15T18:13:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.606+0000 7f3c06600640 1 Processor -- start 2024-01-15T18:13:50.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.606+0000 7f3c06600640 1 -- start start 2024-01-15T18:13:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.610+0000 7f3c06600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3c0006c520 0x7f3c000b3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.610+0000 7f3c06600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3c0015d190 0x7f3c000b6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.610+0000 7f3c06600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c0015d570 0x7f3c0006f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.610+0000 7f3c06600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c00159f60 con 0x7f3c0015d570 2024-01-15T18:13:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.610+0000 7f3c06600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c000c7230 con 0x7f3c0015d190 2024-01-15T18:13:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.610+0000 7f3c06600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0015a0f0 con 0x7f3c0006c520 2024-01-15T18:13:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.610+0000 7f3bffe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c0015d570 0x7f3c0006f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.610+0000 7f3bffe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c0015d570 0x7f3c0006f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54884/0 (socket says 172.21.15.27:54884) 2024-01-15T18:13:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.610+0000 7f3bffe00640 1 -- 172.21.15.27:0/4020829992 learned_addr learned my addr 172.21.15.27:0/4020829992 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:13:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.610+0000 7f3bfea00640 1 --2- 172.21.15.27:0/4020829992 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3c0015d190 0x7f3c000b6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.610+0000 7f3bff400640 1 --2- 172.21.15.27:0/4020829992 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3c0006c520 0x7f3c000b3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.610+0000 7f3bffe00640 1 -- 172.21.15.27:0/4020829992 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3c0006c520 msgr2=0x7f3c000b3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:50.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.610+0000 7f3bffe00640 1 --2- 172.21.15.27:0/4020829992 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3c0006c520 0x7f3c000b3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:50.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.610+0000 7f3bffe00640 1 -- 172.21.15.27:0/4020829992 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3c0015d190 msgr2=0x7f3c000b6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:50.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.610+0000 7f3bffe00640 1 --2- 172.21.15.27:0/4020829992 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3c0015d190 0x7f3c000b6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:50.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.610+0000 7f3bffe00640 1 -- 172.21.15.27:0/4020829992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf8009b30 con 0x7f3c0015d570 2024-01-15T18:13:50.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.610+0000 7f3bff400640 1 --2- 172.21.15.27:0/4020829992 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3c0006c520 0x7f3c000b3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:13:50.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.610+0000 7f3bfea00640 1 --2- 172.21.15.27:0/4020829992 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3c0015d190 0x7f3c000b6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:13:50.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.610+0000 7f3bffe00640 1 --2- 172.21.15.27:0/4020829992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c0015d570 0x7f3c0006f860 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80af040 tx=0x7f3bf8019780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:50.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.610+0000 7f3bf7400640 1 -- 172.21.15.27:0/4020829992 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf80b6680 con 0x7f3c0015d570 2024-01-15T18:13:50.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.610+0000 7f3bf7400640 1 -- 172.21.15.27:0/4020829992 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf80b67e0 con 0x7f3c0015d570 2024-01-15T18:13:50.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.610+0000 7f3c06600640 1 -- 172.21.15.27:0/4020829992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c000b4380 con 0x7f3c0015d570 2024-01-15T18:13:50.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.610+0000 7f3bf7400640 1 -- 172.21.15.27:0/4020829992 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf80b6b90 con 0x7f3c0015d570 2024-01-15T18:13:50.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.610+0000 7f3c06600640 1 -- 172.21.15.27:0/4020829992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c000b4940 con 0x7f3c0015d570 2024-01-15T18:13:50.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.610+0000 7f3c06600640 1 -- 172.21.15.27:0/4020829992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c0015a280 con 0x7f3c0015d570 2024-01-15T18:13:50.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.614+0000 7f3bf7400640 1 -- 172.21.15.27:0/4020829992 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f3bf80b6cf0 con 0x7f3c0015d570 2024-01-15T18:13:50.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.614+0000 7f3bf7400640 1 --2- 172.21.15.27:0/4020829992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3bd80673a0 0x7f3bd8069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:50.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.614+0000 7f3bf7400640 1 -- 172.21.15.27:0/4020829992 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f3bf811dfa0 con 0x7f3c0015d570 2024-01-15T18:13:50.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.614+0000 7f3bff400640 1 --2- 172.21.15.27:0/4020829992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3bd80673a0 0x7f3bd8069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.614+0000 7f3bff400640 1 --2- 172.21.15.27:0/4020829992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3bd80673a0 0x7f3bd8069860 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0009ae0 tx=0x7f3bf0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:50.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.618+0000 7f3bf7400640 1 -- 172.21.15.27:0/4020829992 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3bf80eccb0 con 0x7f3c0015d570 2024-01-15T18:13:50.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:13:50.806+0000 7f3c06600640 1 -- 172.21.15.27:0/4020829992 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone51936", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c0004e850 con 0x7f3bd80673a0 2024-01-15T18:13:57.312 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:13:57.316 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:14:00.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.482+0000 7f3bf7400640 1 -- 172.21.15.27:0/4020829992 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3c0004e850 con 0x7f3bd80673a0 2024-01-15T18:14:00.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.486+0000 7f3c06600640 1 -- 172.21.15.27:0/4020829992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3bd80673a0 msgr2=0x7f3bd8069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:00.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.486+0000 7f3c06600640 1 --2- 172.21.15.27:0/4020829992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3bd80673a0 0x7f3bd8069860 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0009ae0 tx=0x7f3bf0008040 comp rx=0 tx=0).stop 2024-01-15T18:14:00.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.486+0000 7f3c06600640 1 -- 172.21.15.27:0/4020829992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c0015d570 msgr2=0x7f3c0006f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:00.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.486+0000 7f3c06600640 1 --2- 172.21.15.27:0/4020829992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c0015d570 0x7f3c0006f860 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80af040 tx=0x7f3bf8019780 comp rx=0 tx=0).stop 2024-01-15T18:14:00.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.486+0000 7f3c06600640 1 -- 172.21.15.27:0/4020829992 shutdown_connections 2024-01-15T18:14:00.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.486+0000 7f3c06600640 1 --2- 172.21.15.27:0/4020829992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3bd80673a0 0x7f3bd8069860 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:00.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.486+0000 7f3c06600640 1 --2- 172.21.15.27:0/4020829992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c0015d570 0x7f3c0006f860 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:00.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.486+0000 7f3c06600640 1 --2- 172.21.15.27:0/4020829992 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3c0015d190 0x7f3c000b6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:00.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.486+0000 7f3c06600640 1 --2- 172.21.15.27:0/4020829992 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3c0006c520 0x7f3c000b3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:00.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.486+0000 7f3c06600640 1 -- 172.21.15.27:0/4020829992 >> 172.21.15.27:0/4020829992 conn(0x7f3c000d0e70 msgr2=0x7f3c000b1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:00.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.486+0000 7f3c06600640 1 -- 172.21.15.27:0/4020829992 shutdown_connections 2024-01-15T18:14:00.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.486+0000 7f3c06600640 1 -- 172.21.15.27:0/4020829992 wait complete. 2024-01-15T18:14:00.506 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone51937 2024-01-15T18:14:00.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b55600640 1 Processor -- start 2024-01-15T18:14:00.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b55600640 1 -- start start 2024-01-15T18:14:00.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b55600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f1b500cca40 0x7f1b500d19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:00.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b55600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b5015b6b0 con 0x7f1b500cef20 2024-01-15T18:14:00.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b55600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b500cfdb0 con 0x7f1b500ce2c0 2024-01-15T18:14:00.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b55600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b500cb9e0 con 0x7f1b500cca40 2024-01-15T18:14:00.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4ea00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1b500cef20 0x7f1b500cf300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56506/0 (socket says 172.21.15.27:56506) 2024-01-15T18:14:00.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4ea00640 1 -- 172.21.15.27:0/212368288 learned_addr learned my addr 172.21.15.27:0/212368288 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:14:00.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4f400640 1 --2- 172.21.15.27:0/212368288 >> v2:172.21.15.93:3300/0 conn(0x7f1b500cca40 0x7f1b500d19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:00.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4d600640 1 -- 172.21.15.27:0/212368288 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908263388 0 0) 0x7f1b5015b6b0 con 0x7f1b500cef20 2024-01-15T18:14:00.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4d600640 1 -- 172.21.15.27:0/212368288 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b30003530 con 0x7f1b500cef20 2024-01-15T18:14:00.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4d600640 1 -- 172.21.15.27:0/212368288 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 386471912 0 0) 0x7f1b30003530 con 0x7f1b500cef20 2024-01-15T18:14:00.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4d600640 1 -- 172.21.15.27:0/212368288 >> v2:172.21.15.93:3300/0 conn(0x7f1b500cca40 msgr2=0x7f1b500d19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:00.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4d600640 1 --2- 172.21.15.27:0/212368288 >> v2:172.21.15.93:3300/0 conn(0x7f1b500cca40 0x7f1b500d19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:00.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4d600640 1 -- 172.21.15.27:0/212368288 >> v1:172.21.15.93:6789/0 conn(0x7f1b500ce2c0 legacy=0x7f1b500b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:00.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4d600640 1 -- 172.21.15.27:0/212368288 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b500cbc80 con 0x7f1b500cef20 2024-01-15T18:14:00.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4d600640 1 -- 172.21.15.27:0/212368288 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f1b40001680 con 0x7f1b500cef20 2024-01-15T18:14:00.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4d600640 1 -- 172.21.15.27:0/212368288 >> v1:172.21.15.27:6789/0 conn(0x7f1b500cef20 legacy=0x7f1b500cf300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:00.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4d600640 1 --2- 172.21.15.27:0/212368288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b30003a40 0x7f1b30003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:00.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4d600640 1 --2- 172.21.15.27:0/212368288 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1b300047d0 0x7f1b30004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:00.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4d600640 1 --2- 172.21.15.27:0/212368288 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1b300051e0 0x7f1b3000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:00.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4d600640 1 -- 172.21.15.27:0/212368288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b500cb9e0 con 0x7f1b30003a40 2024-01-15T18:14:00.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4d600640 1 -- 172.21.15.27:0/212368288 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3000c110 con 0x7f1b300051e0 2024-01-15T18:14:00.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4d600640 1 -- 172.21.15.27:0/212368288 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3000c2d0 con 0x7f1b300047d0 2024-01-15T18:14:00.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4ea00640 1 --2- 172.21.15.27:0/212368288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b30003a40 0x7f1b30003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:00.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4f400640 1 --2- 172.21.15.27:0/212368288 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1b300051e0 0x7f1b3000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:00.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4e000640 1 --2- 172.21.15.27:0/212368288 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1b300047d0 0x7f1b30004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:00.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4ea00640 1 -- 172.21.15.27:0/212368288 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1b300047d0 msgr2=0x7f1b30004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:00.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4ea00640 1 --2- 172.21.15.27:0/212368288 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1b300047d0 0x7f1b30004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:00.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4ea00640 1 -- 172.21.15.27:0/212368288 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1b300051e0 msgr2=0x7f1b3000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:00.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4ea00640 1 --2- 172.21.15.27:0/212368288 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1b300051e0 0x7f1b3000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:00.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4ea00640 1 -- 172.21.15.27:0/212368288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b30003530 con 0x7f1b30003a40 2024-01-15T18:14:00.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4f400640 1 --2- 172.21.15.27:0/212368288 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1b300051e0 0x7f1b3000b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:14:00.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.650+0000 7f1b4f400640 1 -- 172.21.15.27:0/212368288 reap_dead start 2024-01-15T18:14:00.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.654+0000 7f1b4ea00640 1 --2- 172.21.15.27:0/212368288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b30003a40 0x7f1b30003e60 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f1b4000c0e0 tx=0x7f1b4001cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ea640428ef25477 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:00.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.654+0000 7f1b4d600640 1 -- 172.21.15.27:0/212368288 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b40005ae0 con 0x7f1b30003a40 2024-01-15T18:14:00.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.654+0000 7f1b4d600640 1 -- 172.21.15.27:0/212368288 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b40005c60 con 0x7f1b30003a40 2024-01-15T18:14:00.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.654+0000 7f1b4d600640 1 -- 172.21.15.27:0/212368288 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b40007070 con 0x7f1b30003a40 2024-01-15T18:14:00.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.654+0000 7f1b55600640 1 -- 172.21.15.27:0/212368288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b30003a40 msgr2=0x7f1b30003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:00.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.654+0000 7f1b55600640 1 --2- 172.21.15.27:0/212368288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b30003a40 0x7f1b30003e60 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f1b4000c0e0 tx=0x7f1b4001cb20 comp rx=0 tx=0).stop 2024-01-15T18:14:00.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.654+0000 7f1b55600640 1 -- 172.21.15.27:0/212368288 shutdown_connections 2024-01-15T18:14:00.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.654+0000 7f1b55600640 1 --2- 172.21.15.27:0/212368288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b30003a40 0x7f1b30003e60 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:00.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.654+0000 7f1b55600640 1 -- 172.21.15.27:0/212368288 >> 172.21.15.27:0/212368288 conn(0x7f1b500d0ed0 msgr2=0x7f1b500d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:00.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.654+0000 7f1b55600640 1 -- 172.21.15.27:0/212368288 shutdown_connections 2024-01-15T18:14:00.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.654+0000 7f1b55600640 1 -- 172.21.15.27:0/212368288 wait complete. 2024-01-15T18:14:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.658+0000 7f1b55600640 1 Processor -- start 2024-01-15T18:14:00.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.658+0000 7f1b55600640 1 -- start start 2024-01-15T18:14:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.658+0000 7f1b55600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b30003a40 0x7f1b50146ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.658+0000 7f1b55600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1b5015c700 0x7f1b500ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.658+0000 7f1b55600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1b500cabc0 0x7f1b50146260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.658+0000 7f1b55600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5015db90 con 0x7f1b30003a40 2024-01-15T18:14:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.658+0000 7f1b55600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5015dfb0 con 0x7f1b5015c700 2024-01-15T18:14:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.658+0000 7f1b55600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5015cf10 con 0x7f1b500cabc0 2024-01-15T18:14:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.658+0000 7f1b4ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b30003a40 0x7f1b50146ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.658+0000 7f1b4ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b30003a40 0x7f1b50146ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44796/0 (socket says 172.21.15.27:44796) 2024-01-15T18:14:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.658+0000 7f1b4ea00640 1 -- 172.21.15.27:0/2458043447 learned_addr learned my addr 172.21.15.27:0/2458043447 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:14:00.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.658+0000 7f1b4f400640 1 --2- 172.21.15.27:0/2458043447 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1b500cabc0 0x7f1b50146260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:00.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.658+0000 7f1b4e000640 1 --2- 172.21.15.27:0/2458043447 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1b5015c700 0x7f1b500ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:00.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.658+0000 7f1b4ea00640 1 -- 172.21.15.27:0/2458043447 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1b500cabc0 msgr2=0x7f1b50146260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:00.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.658+0000 7f1b4ea00640 1 --2- 172.21.15.27:0/2458043447 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1b500cabc0 0x7f1b50146260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:00.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.658+0000 7f1b4ea00640 1 -- 172.21.15.27:0/2458043447 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1b5015c700 msgr2=0x7f1b500ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:00.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.658+0000 7f1b4ea00640 1 --2- 172.21.15.27:0/2458043447 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1b5015c700 0x7f1b500ca680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:00.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.658+0000 7f1b4ea00640 1 -- 172.21.15.27:0/2458043447 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5015d1f0 con 0x7f1b30003a40 2024-01-15T18:14:00.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.658+0000 7f1b4ea00640 1 --2- 172.21.15.27:0/2458043447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b30003a40 0x7f1b50146ea0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f1b40000c10 tx=0x7f1b30005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:00.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.658+0000 7f1b46a00640 1 -- 172.21.15.27:0/2458043447 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b400b4070 con 0x7f1b30003a40 2024-01-15T18:14:00.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.658+0000 7f1b46a00640 1 -- 172.21.15.27:0/2458043447 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b40001540 con 0x7f1b30003a40 2024-01-15T18:14:00.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.658+0000 7f1b55600640 1 -- 172.21.15.27:0/2458043447 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b5015d4d0 con 0x7f1b30003a40 2024-01-15T18:14:00.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.658+0000 7f1b46a00640 1 -- 172.21.15.27:0/2458043447 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b400072b0 con 0x7f1b30003a40 2024-01-15T18:14:00.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.658+0000 7f1b55600640 1 -- 172.21.15.27:0/2458043447 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b5015cb80 con 0x7f1b30003a40 2024-01-15T18:14:00.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.662+0000 7f1b55600640 1 -- 172.21.15.27:0/2458043447 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b500d09f0 con 0x7f1b30003a40 2024-01-15T18:14:00.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.662+0000 7f1b46a00640 1 -- 172.21.15.27:0/2458043447 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f1b40007410 con 0x7f1b30003a40 2024-01-15T18:14:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.662+0000 7f1b46a00640 1 --2- 172.21.15.27:0/2458043447 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1b280673a0 0x7f1b28069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.662+0000 7f1b46a00640 1 -- 172.21.15.27:0/2458043447 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f1b4011c5a0 con 0x7f1b30003a40 2024-01-15T18:14:00.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.662+0000 7f1b4e000640 1 --2- 172.21.15.27:0/2458043447 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1b280673a0 0x7f1b28069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:00.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.662+0000 7f1b4e000640 1 --2- 172.21.15.27:0/2458043447 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1b280673a0 0x7f1b28069860 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f1b5015e6c0 tx=0x7f1b38009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:00.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.666+0000 7f1b46a00640 1 -- 172.21.15.27:0/2458043447 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1b400eb2e0 con 0x7f1b30003a40 2024-01-15T18:14:00.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:00.854+0000 7f1b55600640 1 -- 172.21.15.27:0/2458043447 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone51937", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b5006d610 con 0x7f1b280673a0 2024-01-15T18:14:01.491 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:14:01.490+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:14:01.491 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:14:01.490+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:14:01.491 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:14:01.490+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:14:01.491 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:14:01.490+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:14:01.491 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:14:01.490+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:14:02.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.366+0000 7f1b46a00640 1 -- 172.21.15.27:0/2458043447 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1b38001c70 con 0x7f1b280673a0 2024-01-15T18:14:02.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.370+0000 7f1b55600640 1 -- 172.21.15.27:0/2458043447 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1b280673a0 msgr2=0x7f1b28069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:02.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.370+0000 7f1b55600640 1 --2- 172.21.15.27:0/2458043447 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1b280673a0 0x7f1b28069860 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f1b5015e6c0 tx=0x7f1b38009040 comp rx=0 tx=0).stop 2024-01-15T18:14:02.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.370+0000 7f1b55600640 1 -- 172.21.15.27:0/2458043447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b30003a40 msgr2=0x7f1b50146ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.370+0000 7f1b55600640 1 --2- 172.21.15.27:0/2458043447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b30003a40 0x7f1b50146ea0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f1b40000c10 tx=0x7f1b30005800 comp rx=0 tx=0).stop 2024-01-15T18:14:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.370+0000 7f1b55600640 1 -- 172.21.15.27:0/2458043447 shutdown_connections 2024-01-15T18:14:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.370+0000 7f1b55600640 1 --2- 172.21.15.27:0/2458043447 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1b280673a0 0x7f1b28069860 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.370+0000 7f1b55600640 1 --2- 172.21.15.27:0/2458043447 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1b500cabc0 0x7f1b50146260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.370+0000 7f1b55600640 1 --2- 172.21.15.27:0/2458043447 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1b5015c700 0x7f1b500ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.370+0000 7f1b55600640 1 --2- 172.21.15.27:0/2458043447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1b30003a40 0x7f1b50146ea0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.370+0000 7f1b55600640 1 -- 172.21.15.27:0/2458043447 >> 172.21.15.27:0/2458043447 conn(0x7f1b500d0ed0 msgr2=0x7f1b500ccf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.370+0000 7f1b55600640 1 -- 172.21.15.27:0/2458043447 shutdown_connections 2024-01-15T18:14:02.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.370+0000 7f1b55600640 1 -- 172.21.15.27:0/2458043447 wait complete. 2024-01-15T18:14:02.392 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone51938 2024-01-15T18:14:02.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f76400640 1 Processor -- start 2024-01-15T18:14:02.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f76400640 1 -- start start 2024-01-15T18:14:02.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f76400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3f7015b4b0 0x7f3f700b1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:02.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f76400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f700d19a0 con 0x7f3f700cf460 2024-01-15T18:14:02.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f76400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f700d1c80 con 0x7f3f700cec60 2024-01-15T18:14:02.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f76400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f700b07c0 con 0x7f3f7015b4b0 2024-01-15T18:14:02.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6f400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3f700cf460 0x7f3f700cf840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56514/0 (socket says 172.21.15.27:56514) 2024-01-15T18:14:02.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6f400640 1 -- 172.21.15.27:0/875288905 learned_addr learned my addr 172.21.15.27:0/875288905 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:14:02.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6ea00640 1 --2- 172.21.15.27:0/875288905 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3f7015b4b0 0x7f3f700b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:02.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6e000640 1 -- 172.21.15.27:0/875288905 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230475688 0 0) 0x7f3f700d19a0 con 0x7f3f700cf460 2024-01-15T18:14:02.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6e000640 1 -- 172.21.15.27:0/875288905 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f50003530 con 0x7f3f700cf460 2024-01-15T18:14:02.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6e000640 1 -- 172.21.15.27:0/875288905 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1199570755 0 0) 0x7f3f50003530 con 0x7f3f700cf460 2024-01-15T18:14:02.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6e000640 1 -- 172.21.15.27:0/875288905 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3f7015b4b0 msgr2=0x7f3f700b1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:02.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6e000640 1 --2- 172.21.15.27:0/875288905 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3f7015b4b0 0x7f3f700b1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:02.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6e000640 1 -- 172.21.15.27:0/875288905 >> v1:172.21.15.93:6789/0 conn(0x7f3f700cec60 legacy=0x7f3f700b1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:02.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6e000640 1 -- 172.21.15.27:0/875288905 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f700b0aa0 con 0x7f3f700cf460 2024-01-15T18:14:02.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6e000640 1 -- 172.21.15.27:0/875288905 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f3f60001680 con 0x7f3f700cf460 2024-01-15T18:14:02.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6e000640 1 -- 172.21.15.27:0/875288905 >> v1:172.21.15.27:6789/0 conn(0x7f3f700cf460 legacy=0x7f3f700cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:02.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6e000640 1 --2- 172.21.15.27:0/875288905 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3f7015b4b0 0x7f3f50003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:02.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6e000640 1 --2- 172.21.15.27:0/875288905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f50003f60 0x7f3f500047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:02.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6e000640 1 --2- 172.21.15.27:0/875288905 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3f50004d10 0x7f3f50005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:02.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6e000640 1 -- 172.21.15.27:0/875288905 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f700b07c0 con 0x7f3f50003f60 2024-01-15T18:14:02.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6e000640 1 -- 172.21.15.27:0/875288905 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5000ba70 con 0x7f3f50004d10 2024-01-15T18:14:02.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6e000640 1 -- 172.21.15.27:0/875288905 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5000bc30 con 0x7f3f7015b4b0 2024-01-15T18:14:02.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6ea00640 1 --2- 172.21.15.27:0/875288905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f50003f60 0x7f3f500047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:02.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6f400640 1 --2- 172.21.15.27:0/875288905 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3f7015b4b0 0x7f3f50003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:02.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6fe00640 1 --2- 172.21.15.27:0/875288905 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3f50004d10 0x7f3f50005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:02.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6ea00640 1 -- 172.21.15.27:0/875288905 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3f7015b4b0 msgr2=0x7f3f50003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:02.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6ea00640 1 --2- 172.21.15.27:0/875288905 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3f7015b4b0 0x7f3f50003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:02.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6ea00640 1 -- 172.21.15.27:0/875288905 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3f50004d10 msgr2=0x7f3f50005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:02.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6ea00640 1 --2- 172.21.15.27:0/875288905 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3f50004d10 0x7f3f50005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:02.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.538+0000 7f3f6ea00640 1 -- 172.21.15.27:0/875288905 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f50003530 con 0x7f3f50003f60 2024-01-15T18:14:02.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.542+0000 7f3f6ea00640 1 --2- 172.21.15.27:0/875288905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f50003f60 0x7f3f500047d0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f3f58009ae0 tx=0x7f3f5801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5856dfc17d9e13a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:02.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.542+0000 7f3f6e000640 1 -- 172.21.15.27:0/875288905 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f580a5640 con 0x7f3f50003f60 2024-01-15T18:14:02.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.542+0000 7f3f6e000640 1 -- 172.21.15.27:0/875288905 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f580a57a0 con 0x7f3f50003f60 2024-01-15T18:14:02.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.542+0000 7f3f6e000640 1 -- 172.21.15.27:0/875288905 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f580a5b50 con 0x7f3f50003f60 2024-01-15T18:14:02.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.542+0000 7f3f76400640 1 -- 172.21.15.27:0/875288905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f50003f60 msgr2=0x7f3f500047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:02.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.542+0000 7f3f76400640 1 --2- 172.21.15.27:0/875288905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f50003f60 0x7f3f500047d0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f3f58009ae0 tx=0x7f3f5801ca50 comp rx=0 tx=0).stop 2024-01-15T18:14:02.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.542+0000 7f3f6fe00640 1 -- 172.21.15.27:0/875288905 reap_dead start 2024-01-15T18:14:02.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.542+0000 7f3f76400640 1 -- 172.21.15.27:0/875288905 shutdown_connections 2024-01-15T18:14:02.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.542+0000 7f3f76400640 1 -- 172.21.15.27:0/875288905 >> 172.21.15.27:0/875288905 conn(0x7f3f700d0e80 msgr2=0x7f3f700d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:02.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.542+0000 7f3f76400640 1 -- 172.21.15.27:0/875288905 shutdown_connections 2024-01-15T18:14:02.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.542+0000 7f3f76400640 1 -- 172.21.15.27:0/875288905 wait complete. 2024-01-15T18:14:02.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.546+0000 7f3f76400640 1 Processor -- start 2024-01-15T18:14:02.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.546+0000 7f3f76400640 1 -- start start 2024-01-15T18:14:02.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.546+0000 7f3f76400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f50003f60 0x7f3f70159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:02.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.546+0000 7f3f76400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3f7009d0f0 0x7f3f700b5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:02.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.546+0000 7f3f76400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3f700b5e40 0x7f3f701f6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:02.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.546+0000 7f3f76400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f700b62c0 con 0x7f3f50003f60 2024-01-15T18:14:02.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.546+0000 7f3f76400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7009e9e0 con 0x7f3f700b5e40 2024-01-15T18:14:02.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.546+0000 7f3f76400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f70159960 con 0x7f3f7009d0f0 2024-01-15T18:14:02.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.546+0000 7f3f6f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f50003f60 0x7f3f70159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:02.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.546+0000 7f3f6f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f50003f60 0x7f3f70159420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44808/0 (socket says 172.21.15.27:44808) 2024-01-15T18:14:02.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.546+0000 7f3f6f400640 1 -- 172.21.15.27:0/1117141641 learned_addr learned my addr 172.21.15.27:0/1117141641 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:14:02.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.546+0000 7f3f6ea00640 1 --2- 172.21.15.27:0/1117141641 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3f7009d0f0 0x7f3f700b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:02.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.546+0000 7f3f6fe00640 1 --2- 172.21.15.27:0/1117141641 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3f700b5e40 0x7f3f701f6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:02.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.546+0000 7f3f6f400640 1 -- 172.21.15.27:0/1117141641 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3f7009d0f0 msgr2=0x7f3f700b5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:02.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.546+0000 7f3f6f400640 1 --2- 172.21.15.27:0/1117141641 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3f7009d0f0 0x7f3f700b5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:02.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.546+0000 7f3f6f400640 1 -- 172.21.15.27:0/1117141641 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3f700b5e40 msgr2=0x7f3f701f6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:02.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.546+0000 7f3f6f400640 1 --2- 172.21.15.27:0/1117141641 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3f700b5e40 0x7f3f701f6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:02.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.546+0000 7f3f6f400640 1 -- 172.21.15.27:0/1117141641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7009d940 con 0x7f3f50003f60 2024-01-15T18:14:02.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.546+0000 7f3f6fe00640 1 --2- 172.21.15.27:0/1117141641 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3f700b5e40 0x7f3f701f6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:14:02.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.546+0000 7f3f6f400640 1 --2- 172.21.15.27:0/1117141641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f50003f60 0x7f3f70159420 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f3f60007290 tx=0x7f3f60007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:02.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.546+0000 7f3f67400640 1 -- 172.21.15.27:0/1117141641 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6000f940 con 0x7f3f50003f60 2024-01-15T18:14:02.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.546+0000 7f3f67400640 1 -- 172.21.15.27:0/1117141641 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f6000faa0 con 0x7f3f50003f60 2024-01-15T18:14:02.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.546+0000 7f3f67400640 1 -- 172.21.15.27:0/1117141641 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6000fe10 con 0x7f3f50003f60 2024-01-15T18:14:02.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.546+0000 7f3f76400640 1 -- 172.21.15.27:0/1117141641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f7009dc20 con 0x7f3f50003f60 2024-01-15T18:14:02.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.550+0000 7f3f76400640 1 -- 172.21.15.27:0/1117141641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f7009e1e0 con 0x7f3f50003f60 2024-01-15T18:14:02.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.550+0000 7f3f76400640 1 -- 172.21.15.27:0/1117141641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f7004e850 con 0x7f3f50003f60 2024-01-15T18:14:02.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.550+0000 7f3f67400640 1 -- 172.21.15.27:0/1117141641 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f3f6001d460 con 0x7f3f50003f60 2024-01-15T18:14:02.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.550+0000 7f3f67400640 1 --2- 172.21.15.27:0/1117141641 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3f480673a0 0x7f3f48069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:02.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.550+0000 7f3f67400640 1 -- 172.21.15.27:0/1117141641 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f3f60082e50 con 0x7f3f50003f60 2024-01-15T18:14:02.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.550+0000 7f3f6ea00640 1 --2- 172.21.15.27:0/1117141641 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3f480673a0 0x7f3f48069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:02.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.550+0000 7f3f6ea00640 1 --2- 172.21.15.27:0/1117141641 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3f480673a0 0x7f3f48069860 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f3f580af000 tx=0x7f3f580046d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:02.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.554+0000 7f3f67400640 1 -- 172.21.15.27:0/1117141641 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3f60051ae0 con 0x7f3f50003f60 2024-01-15T18:14:02.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:02.742+0000 7f3f76400640 1 -- 172.21.15.27:0/1117141641 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone51938", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f7015a3f0 con 0x7f3f480673a0 2024-01-15T18:14:03.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.090+0000 7f3f67400640 1 -- 172.21.15.27:0/1117141641 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3f7015a3f0 con 0x7f3f480673a0 2024-01-15T18:14:03.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.094+0000 7f3f76400640 1 -- 172.21.15.27:0/1117141641 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3f480673a0 msgr2=0x7f3f48069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:03.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.094+0000 7f3f76400640 1 --2- 172.21.15.27:0/1117141641 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3f480673a0 0x7f3f48069860 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f3f580af000 tx=0x7f3f580046d0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.094+0000 7f3f76400640 1 -- 172.21.15.27:0/1117141641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f50003f60 msgr2=0x7f3f70159420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:03.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.094+0000 7f3f76400640 1 --2- 172.21.15.27:0/1117141641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f50003f60 0x7f3f70159420 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f3f60007290 tx=0x7f3f60007750 comp rx=0 tx=0).stop 2024-01-15T18:14:03.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.094+0000 7f3f76400640 1 -- 172.21.15.27:0/1117141641 shutdown_connections 2024-01-15T18:14:03.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.094+0000 7f3f76400640 1 --2- 172.21.15.27:0/1117141641 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3f480673a0 0x7f3f48069860 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.094+0000 7f3f76400640 1 --2- 172.21.15.27:0/1117141641 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3f700b5e40 0x7f3f701f6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.094+0000 7f3f76400640 1 --2- 172.21.15.27:0/1117141641 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3f7009d0f0 0x7f3f700b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.094+0000 7f3f76400640 1 --2- 172.21.15.27:0/1117141641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f50003f60 0x7f3f70159420 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.094+0000 7f3f76400640 1 -- 172.21.15.27:0/1117141641 >> 172.21.15.27:0/1117141641 conn(0x7f3f700d0e80 msgr2=0x7f3f7006b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:03.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.094+0000 7f3f76400640 1 -- 172.21.15.27:0/1117141641 shutdown_connections 2024-01-15T18:14:03.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.094+0000 7f3f76400640 1 -- 172.21.15.27:0/1117141641 wait complete. 2024-01-15T18:14:03.116 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone51939 2024-01-15T18:14:03.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.262+0000 7fdc8f000640 1 Processor -- start 2024-01-15T18:14:03.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.262+0000 7fdc8f000640 1 -- start start 2024-01-15T18:14:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.262+0000 7fdc8f000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fdc8815f550 0x7fdc880cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.262+0000 7fdc8f000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdc880cf110 0x7fdc880701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.262+0000 7fdc8f000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc880b1cb0 con 0x7fdc88158900 2024-01-15T18:14:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.262+0000 7fdc8f000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc880cf580 con 0x7fdc8815f550 2024-01-15T18:14:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.262+0000 7fdc8f000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8815fa00 con 0x7fdc880cf110 2024-01-15T18:14:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.262+0000 7fdc87e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdc88158900 0x7fdc8815bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:56520/0 (socket says 172.21.15.27:56520) 2024-01-15T18:14:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.262+0000 7fdc87e00640 1 -- 172.21.15.27:0/3460916155 learned_addr learned my addr 172.21.15.27:0/3460916155 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:14:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.262+0000 7fdc8d000640 1 --2- 172.21.15.27:0/3460916155 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdc880cf110 0x7fdc880701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:03.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.262+0000 7fdc87400640 1 --2- 172.21.15.27:0/3460916155 >> v2:172.21.15.93:3300/0 conn(0x7fdc8815f550 0x7fdc880cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:03.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.262+0000 7fdc86a00640 1 -- 172.21.15.27:0/3460916155 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719642257 0 0) 0x7fdc880b1cb0 con 0x7fdc88158900 2024-01-15T18:14:03.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.262+0000 7fdc86a00640 1 -- 172.21.15.27:0/3460916155 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc68003530 con 0x7fdc88158900 2024-01-15T18:14:03.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.266+0000 7fdc8d000640 1 -- 172.21.15.27:0/3460916155 >> v2:172.21.15.93:3300/0 conn(0x7fdc8815f550 msgr2=0x7fdc880cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:03.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.266+0000 7fdc8d000640 1 --2- 172.21.15.27:0/3460916155 >> v2:172.21.15.93:3300/0 conn(0x7fdc8815f550 0x7fdc880cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.266+0000 7fdc8d000640 1 -- 172.21.15.27:0/3460916155 >> v1:172.21.15.27:6789/0 conn(0x7fdc88158900 legacy=0x7fdc8815bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:03.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.266+0000 7fdc86a00640 1 -- 172.21.15.27:0/3460916155 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1902310195 0 0) 0x7fdc68003530 con 0x7fdc88158900 2024-01-15T18:14:03.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.266+0000 7fdc8d000640 1 -- 172.21.15.27:0/3460916155 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc880b1f00 con 0x7fdc880cf110 2024-01-15T18:14:03.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.266+0000 7fdc87400640 1 --2- 172.21.15.27:0/3460916155 >> v2:172.21.15.93:3300/0 conn(0x7fdc8815f550 0x7fdc880cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:14:03.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.266+0000 7fdc8d000640 1 --2- 172.21.15.27:0/3460916155 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdc880cf110 0x7fdc880701e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdc80009d90 tx=0x7fdc8001c840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5a755492306d037 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:03.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.266+0000 7fdc86a00640 1 -- 172.21.15.27:0/3460916155 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc88070f30 con 0x7fdc880cf110 2024-01-15T18:14:03.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.266+0000 7fdc86a00640 1 -- 172.21.15.27:0/3460916155 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc800a5640 con 0x7fdc880cf110 2024-01-15T18:14:03.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.266+0000 7fdc86a00640 1 -- 172.21.15.27:0/3460916155 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc78001680 con 0x7fdc880cf110 2024-01-15T18:14:03.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.270+0000 7fdc8f000640 1 -- 172.21.15.27:0/3460916155 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdc880cf110 msgr2=0x7fdc880701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:03.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.270+0000 7fdc8f000640 1 --2- 172.21.15.27:0/3460916155 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdc880cf110 0x7fdc880701e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdc80009d90 tx=0x7fdc8001c840 comp rx=0 tx=0).stop 2024-01-15T18:14:03.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.270+0000 7fdc8f000640 1 -- 172.21.15.27:0/3460916155 shutdown_connections 2024-01-15T18:14:03.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.270+0000 7fdc8f000640 1 --2- 172.21.15.27:0/3460916155 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdc880cf110 0x7fdc880701e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.270+0000 7fdc8f000640 1 --2- 172.21.15.27:0/3460916155 >> v2:172.21.15.93:3300/0 conn(0x7fdc8815f550 0x7fdc880cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.270+0000 7fdc8f000640 1 -- 172.21.15.27:0/3460916155 >> 172.21.15.27:0/3460916155 conn(0x7fdc880cb700 msgr2=0x7fdc880cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:03.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.270+0000 7fdc8f000640 1 -- 172.21.15.27:0/3460916155 shutdown_connections 2024-01-15T18:14:03.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.270+0000 7fdc8f000640 1 -- 172.21.15.27:0/3460916155 wait complete. 2024-01-15T18:14:03.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.270+0000 7fdc8f000640 1 Processor -- start 2024-01-15T18:14:03.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.270+0000 7fdc8f000640 1 -- start start 2024-01-15T18:14:03.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.270+0000 7fdc8f000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdc880cf110 0x7fdc8813f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:03.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.270+0000 7fdc8f000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc88158900 0x7fdc88150270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:03.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.270+0000 7fdc8f000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdc8815f550 0x7fdc88155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:03.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.270+0000 7fdc8f000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc881507b0 con 0x7fdc88158900 2024-01-15T18:14:03.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.270+0000 7fdc8f000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc88140290 con 0x7fdc8815f550 2024-01-15T18:14:03.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.270+0000 7fdc8f000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8813d490 con 0x7fdc880cf110 2024-01-15T18:14:03.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.270+0000 7fdc87400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc88158900 0x7fdc88150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:03.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.270+0000 7fdc87400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc88158900 0x7fdc88150270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44818/0 (socket says 172.21.15.27:44818) 2024-01-15T18:14:03.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.270+0000 7fdc87400640 1 -- 172.21.15.27:0/165596791 learned_addr learned my addr 172.21.15.27:0/165596791 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:14:03.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.270+0000 7fdc8d000640 1 --2- 172.21.15.27:0/165596791 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdc8815f550 0x7fdc88155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:03.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.270+0000 7fdc87e00640 1 --2- 172.21.15.27:0/165596791 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdc880cf110 0x7fdc8813f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:03.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.274+0000 7fdc87400640 1 -- 172.21.15.27:0/165596791 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdc880cf110 msgr2=0x7fdc8813f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:03.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.274+0000 7fdc87400640 1 --2- 172.21.15.27:0/165596791 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdc880cf110 0x7fdc8813f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.274+0000 7fdc87400640 1 -- 172.21.15.27:0/165596791 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdc8815f550 msgr2=0x7fdc88155aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:03.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.274+0000 7fdc87400640 1 --2- 172.21.15.27:0/165596791 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdc8815f550 0x7fdc88155aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.274+0000 7fdc87400640 1 -- 172.21.15.27:0/165596791 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc70009420 con 0x7fdc88158900 2024-01-15T18:14:03.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.274+0000 7fdc8d000640 1 --2- 172.21.15.27:0/165596791 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdc8815f550 0x7fdc88155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:14:03.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.274+0000 7fdc87400640 1 --2- 172.21.15.27:0/165596791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc88158900 0x7fdc88150270 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fdc7000e980 tx=0x7fdc7000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:03.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.274+0000 7fdc7fe00640 1 -- 172.21.15.27:0/165596791 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7000ca00 con 0x7fdc88158900 2024-01-15T18:14:03.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.274+0000 7fdc7fe00640 1 -- 172.21.15.27:0/165596791 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc7000cb60 con 0x7fdc88158900 2024-01-15T18:14:03.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.274+0000 7fdc7fe00640 1 -- 172.21.15.27:0/165596791 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc70013470 con 0x7fdc88158900 2024-01-15T18:14:03.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.274+0000 7fdc8f000640 1 -- 172.21.15.27:0/165596791 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc80009a70 con 0x7fdc88158900 2024-01-15T18:14:03.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.274+0000 7fdc8f000640 1 -- 172.21.15.27:0/165596791 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc8813fcd0 con 0x7fdc88158900 2024-01-15T18:14:03.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.274+0000 7fdc8f000640 1 -- 172.21.15.27:0/165596791 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc8804e850 con 0x7fdc88158900 2024-01-15T18:14:03.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.274+0000 7fdc7fe00640 1 -- 172.21.15.27:0/165596791 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 87180+0+0 (secure 0 0 0) 0x7fdc700135d0 con 0x7fdc88158900 2024-01-15T18:14:03.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.274+0000 7fdc7fe00640 1 --2- 172.21.15.27:0/165596791 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdc60067430 0x7fdc600698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:03.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.274+0000 7fdc7fe00640 1 -- 172.21.15.27:0/165596791 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fdc70083270 con 0x7fdc88158900 2024-01-15T18:14:03.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.278+0000 7fdc87e00640 1 --2- 172.21.15.27:0/165596791 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdc60067430 0x7fdc600698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:03.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.278+0000 7fdc87e00640 1 --2- 172.21.15.27:0/165596791 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdc60067430 0x7fdc600698f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fdc78001190 tx=0x7fdc78001120 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:03.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.282+0000 7fdc7fe00640 1 -- 172.21.15.27:0/165596791 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fdc70051ce0 con 0x7fdc88158900 2024-01-15T18:14:03.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:03.474+0000 7fdc8f000640 1 -- 172.21.15.27:0/165596791 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone51939", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc8815abf0 con 0x7fdc60067430 2024-01-15T18:14:15.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.490+0000 7fdc7fe00640 1 -- 172.21.15.27:0/165596791 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdc8815abf0 con 0x7fdc60067430 2024-01-15T18:14:15.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.490+0000 7fdc8f000640 1 -- 172.21.15.27:0/165596791 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdc60067430 msgr2=0x7fdc600698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:15.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.490+0000 7fdc8f000640 1 --2- 172.21.15.27:0/165596791 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdc60067430 0x7fdc600698f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fdc78001190 tx=0x7fdc78001120 comp rx=0 tx=0).stop 2024-01-15T18:14:15.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.490+0000 7fdc8f000640 1 -- 172.21.15.27:0/165596791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc88158900 msgr2=0x7fdc88150270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:15.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.490+0000 7fdc8f000640 1 --2- 172.21.15.27:0/165596791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc88158900 0x7fdc88150270 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fdc7000e980 tx=0x7fdc7000ee40 comp rx=0 tx=0).stop 2024-01-15T18:14:15.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.490+0000 7fdc8f000640 1 -- 172.21.15.27:0/165596791 shutdown_connections 2024-01-15T18:14:15.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.490+0000 7fdc8f000640 1 --2- 172.21.15.27:0/165596791 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdc60067430 0x7fdc600698f0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:15.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.490+0000 7fdc8f000640 1 --2- 172.21.15.27:0/165596791 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdc8815f550 0x7fdc88155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:15.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.490+0000 7fdc8f000640 1 --2- 172.21.15.27:0/165596791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc88158900 0x7fdc88150270 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:15.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.490+0000 7fdc8f000640 1 --2- 172.21.15.27:0/165596791 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdc880cf110 0x7fdc8813f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:15.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.490+0000 7fdc8f000640 1 -- 172.21.15.27:0/165596791 >> 172.21.15.27:0/165596791 conn(0x7fdc880cb700 msgr2=0x7fdc880aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:15.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.490+0000 7fdc8f000640 1 -- 172.21.15.27:0/165596791 shutdown_connections 2024-01-15T18:14:15.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.490+0000 7fdc8f000640 1 -- 172.21.15.27:0/165596791 wait complete. 2024-01-15T18:14:15.513 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone519310 2024-01-15T18:14:15.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.654+0000 7f8e70200640 1 Processor -- start 2024-01-15T18:14:15.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.654+0000 7f8e70200640 1 -- start start 2024-01-15T18:14:15.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.658+0000 7f8e70200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8e68144880 0x7f8e680cd490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:15.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.658+0000 7f8e70200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8e6815bcb0 0x7f8e6815c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:15.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.658+0000 7f8e70200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e680ccd20 con 0x7f8e6806eaa0 2024-01-15T18:14:15.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.658+0000 7f8e70200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e6815c680 con 0x7f8e6815bcb0 2024-01-15T18:14:15.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.658+0000 7f8e70200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6806ef20 con 0x7f8e68144880 2024-01-15T18:14:15.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.658+0000 7f8e6e200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8e6815bcb0 0x7f8e6815c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:15.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.658+0000 7f8e6e200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8e6815bcb0 0x7f8e6815c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56342/0 (socket says 172.21.15.27:56342) 2024-01-15T18:14:15.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.658+0000 7f8e6e200640 1 -- 172.21.15.27:0/2773501895 learned_addr learned my addr 172.21.15.27:0/2773501895 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:14:15.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.658+0000 7f8e6d800640 1 --2- 172.21.15.27:0/2773501895 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8e68144880 0x7f8e680cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:15.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.658+0000 7f8e6e200640 1 -- 172.21.15.27:0/2773501895 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8e68144880 msgr2=0x7f8e680cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:15.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.658+0000 7f8e6e200640 1 --2- 172.21.15.27:0/2773501895 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8e68144880 0x7f8e680cd490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:15.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.658+0000 7f8e6e200640 1 -- 172.21.15.27:0/2773501895 >> v1:172.21.15.93:6789/0 conn(0x7f8e6806eaa0 legacy=0x7f8e6815ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:15.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.658+0000 7f8e6e200640 1 -- 172.21.15.27:0/2773501895 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6815b7d0 con 0x7f8e6815bcb0 2024-01-15T18:14:15.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.662+0000 7f8e6e200640 1 --2- 172.21.15.27:0/2773501895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e6815bcb0 0x7f8e6815c140 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009e50 tx=0x7f8e6001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4621e1b79c4d7414 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:15.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.662+0000 7f8e67e00640 1 -- 172.21.15.27:0/2773501895 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e600a5820 con 0x7f8e6815bcb0 2024-01-15T18:14:15.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.662+0000 7f8e67e00640 1 -- 172.21.15.27:0/2773501895 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e600a5980 con 0x7f8e6815bcb0 2024-01-15T18:14:15.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.662+0000 7f8e67e00640 1 -- 172.21.15.27:0/2773501895 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e600a5d10 con 0x7f8e6815bcb0 2024-01-15T18:14:15.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.662+0000 7f8e70200640 1 -- 172.21.15.27:0/2773501895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e6815bcb0 msgr2=0x7f8e6815c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:15.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.662+0000 7f8e70200640 1 --2- 172.21.15.27:0/2773501895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e6815bcb0 0x7f8e6815c140 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009e50 tx=0x7f8e6001c9a0 comp rx=0 tx=0).stop 2024-01-15T18:14:15.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.662+0000 7f8e70200640 1 -- 172.21.15.27:0/2773501895 shutdown_connections 2024-01-15T18:14:15.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.662+0000 7f8e70200640 1 --2- 172.21.15.27:0/2773501895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e6815bcb0 0x7f8e6815c140 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:15.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.662+0000 7f8e70200640 1 --2- 172.21.15.27:0/2773501895 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8e68144880 0x7f8e680cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:15.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.662+0000 7f8e70200640 1 -- 172.21.15.27:0/2773501895 >> 172.21.15.27:0/2773501895 conn(0x7f8e6806b600 msgr2=0x7f8e6806ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:15.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.662+0000 7f8e70200640 1 -- 172.21.15.27:0/2773501895 shutdown_connections 2024-01-15T18:14:15.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.662+0000 7f8e70200640 1 -- 172.21.15.27:0/2773501895 wait complete. 2024-01-15T18:14:15.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.662+0000 7f8e70200640 1 Processor -- start 2024-01-15T18:14:15.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.662+0000 7f8e70200640 1 -- start start 2024-01-15T18:14:15.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.666+0000 7f8e70200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8e6806eaa0 0x7f8e681587f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:15.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.666+0000 7f8e70200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e68144880 0x7f8e6813d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:15.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.666+0000 7f8e70200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8e6815bcb0 0x7f8e6813e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:15.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.666+0000 7f8e70200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e68164e70 con 0x7f8e68144880 2024-01-15T18:14:15.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.666+0000 7f8e70200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e68165150 con 0x7f8e6815bcb0 2024-01-15T18:14:15.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.666+0000 7f8e70200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e68165430 con 0x7f8e6806eaa0 2024-01-15T18:14:15.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.666+0000 7f8e6ce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e68144880 0x7f8e6813d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:15.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.666+0000 7f8e6ce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e68144880 0x7f8e6813d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56358/0 (socket says 172.21.15.27:56358) 2024-01-15T18:14:15.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.666+0000 7f8e6ce00640 1 -- 172.21.15.27:0/1483423641 learned_addr learned my addr 172.21.15.27:0/1483423641 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:14:15.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.666+0000 7f8e6d800640 1 --2- 172.21.15.27:0/1483423641 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8e6806eaa0 0x7f8e681587f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:15.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.666+0000 7f8e6e200640 1 --2- 172.21.15.27:0/1483423641 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8e6815bcb0 0x7f8e6813e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:15.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.666+0000 7f8e6ce00640 1 -- 172.21.15.27:0/1483423641 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8e6806eaa0 msgr2=0x7f8e681587f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:15.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.666+0000 7f8e6ce00640 1 --2- 172.21.15.27:0/1483423641 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8e6806eaa0 0x7f8e681587f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:15.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.666+0000 7f8e6ce00640 1 -- 172.21.15.27:0/1483423641 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8e6815bcb0 msgr2=0x7f8e6813e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:15.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.666+0000 7f8e6ce00640 1 --2- 172.21.15.27:0/1483423641 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8e6815bcb0 0x7f8e6813e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:15.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.666+0000 7f8e6ce00640 1 -- 172.21.15.27:0/1483423641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e60009b30 con 0x7f8e68144880 2024-01-15T18:14:15.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.666+0000 7f8e6ce00640 1 --2- 172.21.15.27:0/1483423641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e68144880 0x7f8e6813d720 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f8e5000c450 tx=0x7f8e5000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:15.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.666+0000 7f8e65600640 1 -- 172.21.15.27:0/1483423641 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e50012830 con 0x7f8e68144880 2024-01-15T18:14:15.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.666+0000 7f8e65600640 1 -- 172.21.15.27:0/1483423641 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e500129b0 con 0x7f8e68144880 2024-01-15T18:14:15.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.666+0000 7f8e70200640 1 -- 172.21.15.27:0/1483423641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e68165710 con 0x7f8e68144880 2024-01-15T18:14:15.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.666+0000 7f8e65600640 1 -- 172.21.15.27:0/1483423641 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e50012d60 con 0x7f8e68144880 2024-01-15T18:14:15.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.666+0000 7f8e70200640 1 -- 172.21.15.27:0/1483423641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e68164b60 con 0x7f8e68144880 2024-01-15T18:14:15.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.666+0000 7f8e70200640 1 -- 172.21.15.27:0/1483423641 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e680cc7d0 con 0x7f8e68144880 2024-01-15T18:14:15.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.666+0000 7f8e65600640 1 -- 172.21.15.27:0/1483423641 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 87180+0+0 (secure 0 0 0) 0x7f8e5001c460 con 0x7f8e68144880 2024-01-15T18:14:15.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.670+0000 7f8e65600640 1 --2- 172.21.15.27:0/1483423641 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8e40067430 0x7f8e400698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:15.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.670+0000 7f8e65600640 1 -- 172.21.15.27:0/1483423641 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f8e50082af0 con 0x7f8e68144880 2024-01-15T18:14:15.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.670+0000 7f8e6d800640 1 --2- 172.21.15.27:0/1483423641 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8e40067430 0x7f8e400698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:15.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.670+0000 7f8e6d800640 1 --2- 172.21.15.27:0/1483423641 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8e40067430 0x7f8e400698f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8e58007a60 tx=0x7f8e58008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:15.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.674+0000 7f8e65600640 1 -- 172.21.15.27:0/1483423641 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8e500517b0 con 0x7f8e68144880 2024-01-15T18:14:15.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:15.862+0000 7f8e70200640 1 -- 172.21.15.27:0/1483423641 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone519310", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e68144c60 con 0x7f8e40067430 2024-01-15T18:14:25.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.618+0000 7f8e65600640 1 -- 172.21.15.27:0/1483423641 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8e58010990 con 0x7f8e40067430 2024-01-15T18:14:25.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.622+0000 7f8e70200640 1 -- 172.21.15.27:0/1483423641 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8e40067430 msgr2=0x7f8e400698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:25.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.622+0000 7f8e70200640 1 --2- 172.21.15.27:0/1483423641 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8e40067430 0x7f8e400698f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f8e58007a60 tx=0x7f8e58008040 comp rx=0 tx=0).stop 2024-01-15T18:14:25.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.622+0000 7f8e70200640 1 -- 172.21.15.27:0/1483423641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e68144880 msgr2=0x7f8e6813d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:25.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.622+0000 7f8e70200640 1 --2- 172.21.15.27:0/1483423641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e68144880 0x7f8e6813d720 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f8e5000c450 tx=0x7f8e5000c910 comp rx=0 tx=0).stop 2024-01-15T18:14:25.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.622+0000 7f8e70200640 1 -- 172.21.15.27:0/1483423641 shutdown_connections 2024-01-15T18:14:25.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.622+0000 7f8e70200640 1 --2- 172.21.15.27:0/1483423641 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8e40067430 0x7f8e400698f0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:25.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.622+0000 7f8e70200640 1 --2- 172.21.15.27:0/1483423641 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8e6815bcb0 0x7f8e6813e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:25.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.622+0000 7f8e70200640 1 --2- 172.21.15.27:0/1483423641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e68144880 0x7f8e6813d720 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:25.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.622+0000 7f8e70200640 1 --2- 172.21.15.27:0/1483423641 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8e6806eaa0 0x7f8e681587f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:25.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.622+0000 7f8e70200640 1 -- 172.21.15.27:0/1483423641 >> 172.21.15.27:0/1483423641 conn(0x7f8e6806b600 msgr2=0x7f8e68158f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:25.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.622+0000 7f8e70200640 1 -- 172.21.15.27:0/1483423641 shutdown_connections 2024-01-15T18:14:25.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.622+0000 7f8e70200640 1 -- 172.21.15.27:0/1483423641 wait complete. 2024-01-15T18:14:25.659 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone519311 2024-01-15T18:14:25.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.930+0000 7ff9f2000640 1 Processor -- start 2024-01-15T18:14:25.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.930+0000 7ff9f2000640 1 -- start start 2024-01-15T18:14:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.934+0000 7ff9f2000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9ec143de0 0x7ff9ec0ce620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.934+0000 7ff9f2000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff9ec1448a0 0x7ff9ec06c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:25.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.934+0000 7ff9f2000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9ec0b41d0 con 0x7ff9ec06ce20 2024-01-15T18:14:25.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.934+0000 7ff9f2000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9ec1441c0 con 0x7ff9ec1448a0 2024-01-15T18:14:25.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.934+0000 7ff9f2000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ec06d230 con 0x7ff9ec143de0 2024-01-15T18:14:25.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.934+0000 7ff9eb400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9ec143de0 0x7ff9ec0ce620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:25.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.934+0000 7ff9eb400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9ec143de0 0x7ff9ec0ce620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:35126/0 (socket says 172.21.15.27:35126) 2024-01-15T18:14:25.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.934+0000 7ff9eb400640 1 -- 172.21.15.27:0/1344632650 learned_addr learned my addr 172.21.15.27:0/1344632650 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:14:25.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.934+0000 7ff9ebe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff9ec06ce20 0x7ff9ec15ec80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55374/0 (socket says 172.21.15.27:55374) 2024-01-15T18:14:25.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.934+0000 7ff9eaa00640 1 --2- 172.21.15.27:0/1344632650 >> v2:172.21.15.27:3300/0 conn(0x7ff9ec1448a0 0x7ff9ec06c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:25.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.934+0000 7ff9eaa00640 1 -- 172.21.15.27:0/1344632650 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9ec143de0 msgr2=0x7ff9ec0ce620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:25.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.934+0000 7ff9eaa00640 1 --2- 172.21.15.27:0/1344632650 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9ec143de0 0x7ff9ec0ce620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:25.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.934+0000 7ff9eaa00640 1 -- 172.21.15.27:0/1344632650 >> v1:172.21.15.27:6789/0 conn(0x7ff9ec06ce20 legacy=0x7ff9ec15ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:25.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.934+0000 7ff9eaa00640 1 -- 172.21.15.27:0/1344632650 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ec0b2120 con 0x7ff9ec1448a0 2024-01-15T18:14:25.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.938+0000 7ff9eaa00640 1 --2- 172.21.15.27:0/1344632650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9ec1448a0 0x7ff9ec06c8e0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40097c0 tx=0x7ff9d401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83dabfb80d58e5fb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:25.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.938+0000 7ff9ea000640 1 -- 172.21.15.27:0/1344632650 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d40ab400 con 0x7ff9ec1448a0 2024-01-15T18:14:25.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.938+0000 7ff9ea000640 1 -- 172.21.15.27:0/1344632650 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d40a7050 con 0x7ff9ec1448a0 2024-01-15T18:14:25.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.938+0000 7ff9ea000640 1 -- 172.21.15.27:0/1344632650 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d40ab780 con 0x7ff9ec1448a0 2024-01-15T18:14:25.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.942+0000 7ff9f2000640 1 -- 172.21.15.27:0/1344632650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9ec1448a0 msgr2=0x7ff9ec06c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:25.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.942+0000 7ff9f2000640 1 --2- 172.21.15.27:0/1344632650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9ec1448a0 0x7ff9ec06c8e0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40097c0 tx=0x7ff9d401c450 comp rx=0 tx=0).stop 2024-01-15T18:14:25.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.942+0000 7ff9f2000640 1 -- 172.21.15.27:0/1344632650 shutdown_connections 2024-01-15T18:14:25.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.942+0000 7ff9f2000640 1 --2- 172.21.15.27:0/1344632650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9ec1448a0 0x7ff9ec06c8e0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:25.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.942+0000 7ff9f2000640 1 --2- 172.21.15.27:0/1344632650 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9ec143de0 0x7ff9ec0ce620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:25.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.942+0000 7ff9f2000640 1 -- 172.21.15.27:0/1344632650 >> 172.21.15.27:0/1344632650 conn(0x7ff9ec06a8f0 msgr2=0x7ff9ec06ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:25.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.942+0000 7ff9f2000640 1 -- 172.21.15.27:0/1344632650 shutdown_connections 2024-01-15T18:14:25.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.942+0000 7ff9f2000640 1 -- 172.21.15.27:0/1344632650 wait complete. 2024-01-15T18:14:25.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.946+0000 7ff9f2000640 1 Processor -- start 2024-01-15T18:14:25.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.946+0000 7ff9f2000640 1 -- start start 2024-01-15T18:14:25.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.950+0000 7ff9f2000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9ec06ce20 0x7ff9ec158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:25.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.950+0000 7ff9f2000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9ec143de0 0x7ff9ec1382e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:25.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.950+0000 7ff9f2000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff9ec1448a0 0x7ff9ec1f9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:25.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.950+0000 7ff9f2000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ec137890 con 0x7ff9ec143de0 2024-01-15T18:14:25.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.950+0000 7ff9f2000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ec1faa10 con 0x7ff9ec1448a0 2024-01-15T18:14:25.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.950+0000 7ff9f2000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ec1facf0 con 0x7ff9ec06ce20 2024-01-15T18:14:25.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.950+0000 7ff9eaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9ec143de0 0x7ff9ec1382e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:25.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.950+0000 7ff9eaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9ec143de0 0x7ff9ec1382e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36864/0 (socket says 172.21.15.27:36864) 2024-01-15T18:14:25.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.950+0000 7ff9eaa00640 1 -- 172.21.15.27:0/3013201983 learned_addr learned my addr 172.21.15.27:0/3013201983 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:14:25.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.950+0000 7ff9ebe00640 1 --2- 172.21.15.27:0/3013201983 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff9ec1448a0 0x7ff9ec1f9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:25.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.950+0000 7ff9eaa00640 1 -- 172.21.15.27:0/3013201983 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9ec06ce20 msgr2=0x7ff9ec158440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:25.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.950+0000 7ff9eaa00640 1 --2- 172.21.15.27:0/3013201983 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9ec06ce20 0x7ff9ec158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:25.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.950+0000 7ff9eaa00640 1 -- 172.21.15.27:0/3013201983 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff9ec1448a0 msgr2=0x7ff9ec1f9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:25.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.950+0000 7ff9eaa00640 1 --2- 172.21.15.27:0/3013201983 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff9ec1448a0 0x7ff9ec1f9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:25.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.950+0000 7ff9eaa00640 1 -- 172.21.15.27:0/3013201983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d4009420 con 0x7ff9ec143de0 2024-01-15T18:14:25.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.950+0000 7ff9eaa00640 1 --2- 172.21.15.27:0/3013201983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9ec143de0 0x7ff9ec1382e0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40ae040 tx=0x7ff9d40191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:25.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.950+0000 7ff9e3400640 1 -- 172.21.15.27:0/3013201983 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d40b4840 con 0x7ff9ec143de0 2024-01-15T18:14:25.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.950+0000 7ff9e3400640 1 -- 172.21.15.27:0/3013201983 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d40b49c0 con 0x7ff9ec143de0 2024-01-15T18:14:25.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.950+0000 7ff9e3400640 1 -- 172.21.15.27:0/3013201983 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d40b4d70 con 0x7ff9ec143de0 2024-01-15T18:14:25.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.950+0000 7ff9f2000640 1 -- 172.21.15.27:0/3013201983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9ec137ae0 con 0x7ff9ec143de0 2024-01-15T18:14:25.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.950+0000 7ff9f2000640 1 -- 172.21.15.27:0/3013201983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9ec13a0b0 con 0x7ff9ec143de0 2024-01-15T18:14:25.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.950+0000 7ff9f2000640 1 -- 172.21.15.27:0/3013201983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9ec1fae80 con 0x7ff9ec143de0 2024-01-15T18:14:25.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.954+0000 7ff9e3400640 1 -- 172.21.15.27:0/3013201983 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 87180+0+0 (secure 0 0 0) 0x7ff9d40b2040 con 0x7ff9ec143de0 2024-01-15T18:14:25.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.954+0000 7ff9e3400640 1 --2- 172.21.15.27:0/3013201983 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff9c4067460 0x7ff9c4069920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:25.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.954+0000 7ff9e3400640 1 -- 172.21.15.27:0/3013201983 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7ff9d411d190 con 0x7ff9ec143de0 2024-01-15T18:14:25.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.962+0000 7ff9e3400640 1 -- 172.21.15.27:0/3013201983 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff9d40ebed0 con 0x7ff9ec143de0 2024-01-15T18:14:25.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.962+0000 7ff9eb400640 1 --2- 172.21.15.27:0/3013201983 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff9c4067460 0x7ff9c4069920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:25.970+0000 7ff9eb400640 1 --2- 172.21.15.27:0/3013201983 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff9c4067460 0x7ff9c4069920 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec13e890 tx=0x7ff9dc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:26.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.250+0000 7ff9f2000640 1 -- 172.21.15.27:0/3013201983 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone519311", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9ec0b1db0 con 0x7ff9c4067460 2024-01-15T18:14:26.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.634+0000 7ff9e3400640 1 -- 172.21.15.27:0/3013201983 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff9dc009400 con 0x7ff9c4067460 2024-01-15T18:14:26.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.638+0000 7ff9f2000640 1 -- 172.21.15.27:0/3013201983 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff9c4067460 msgr2=0x7ff9c4069920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:26.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.638+0000 7ff9f2000640 1 --2- 172.21.15.27:0/3013201983 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff9c4067460 0x7ff9c4069920 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec13e890 tx=0x7ff9dc008040 comp rx=0 tx=0).stop 2024-01-15T18:14:26.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.638+0000 7ff9f2000640 1 -- 172.21.15.27:0/3013201983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9ec143de0 msgr2=0x7ff9ec1382e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:26.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.638+0000 7ff9f2000640 1 --2- 172.21.15.27:0/3013201983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9ec143de0 0x7ff9ec1382e0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40ae040 tx=0x7ff9d40191f0 comp rx=0 tx=0).stop 2024-01-15T18:14:26.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.638+0000 7ff9f2000640 1 -- 172.21.15.27:0/3013201983 shutdown_connections 2024-01-15T18:14:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.638+0000 7ff9f2000640 1 --2- 172.21.15.27:0/3013201983 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff9c4067460 0x7ff9c4069920 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.638+0000 7ff9f2000640 1 --2- 172.21.15.27:0/3013201983 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff9ec1448a0 0x7ff9ec1f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.638+0000 7ff9f2000640 1 --2- 172.21.15.27:0/3013201983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9ec143de0 0x7ff9ec1382e0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.638+0000 7ff9f2000640 1 --2- 172.21.15.27:0/3013201983 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9ec06ce20 0x7ff9ec158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.638+0000 7ff9f2000640 1 -- 172.21.15.27:0/3013201983 >> 172.21.15.27:0/3013201983 conn(0x7ff9ec06a8f0 msgr2=0x7ff9ec143120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.638+0000 7ff9f2000640 1 -- 172.21.15.27:0/3013201983 shutdown_connections 2024-01-15T18:14:26.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.638+0000 7ff9f2000640 1 -- 172.21.15.27:0/3013201983 wait complete. 2024-01-15T18:14:26.674 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone519312 2024-01-15T18:14:26.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.894+0000 7f9ec2200640 1 Processor -- start 2024-01-15T18:14:26.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.894+0000 7f9ec2200640 1 -- start start 2024-01-15T18:14:26.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.898+0000 7f9ec2200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9ebc0b2fc0 0x7f9ebc06b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:26.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.898+0000 7f9ec2200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f9ebc06bc30 0x7f9ebc06a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:26.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.898+0000 7f9ec2200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ebc161cc0 con 0x7f9ebc070cd0 2024-01-15T18:14:26.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.898+0000 7f9ec2200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ebc1628d0 con 0x7f9ebc06bc30 2024-01-15T18:14:26.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.898+0000 7f9ec2200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ebc161f30 con 0x7f9ebc0b2fc0 2024-01-15T18:14:26.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.902+0000 7f9ebaa00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9ebc0b2fc0 0x7f9ebc06b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:26.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.902+0000 7f9ebaa00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9ebc0b2fc0 0x7f9ebc06b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:35138/0 (socket says 172.21.15.27:35138) 2024-01-15T18:14:26.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.902+0000 7f9ebaa00640 1 -- 172.21.15.27:0/2001095419 learned_addr learned my addr 172.21.15.27:0/2001095419 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:14:26.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.902+0000 7f9ebbe00640 1 --2- 172.21.15.27:0/2001095419 >> v2:172.21.15.93:3300/0 conn(0x7f9ebc06bc30 0x7f9ebc06a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:26.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.902+0000 7f9ebaa00640 1 -- 172.21.15.27:0/2001095419 >> v2:172.21.15.93:3300/0 conn(0x7f9ebc06bc30 msgr2=0x7f9ebc06a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:26.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.902+0000 7f9ebaa00640 1 --2- 172.21.15.27:0/2001095419 >> v2:172.21.15.93:3300/0 conn(0x7f9ebc06bc30 0x7f9ebc06a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:26.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.902+0000 7f9ebaa00640 1 -- 172.21.15.27:0/2001095419 >> v1:172.21.15.93:6789/0 conn(0x7f9ebc070cd0 legacy=0x7f9ebc0710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:26.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.902+0000 7f9ebaa00640 1 -- 172.21.15.27:0/2001095419 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ebc162210 con 0x7f9ebc0b2fc0 2024-01-15T18:14:26.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.906+0000 7f9ebaa00640 1 --2- 172.21.15.27:0/2001095419 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9ebc0b2fc0 0x7f9ebc06b6f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4009870 tx=0x7f9ea401c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24cc971f54793501 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:26.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.906+0000 7f9eba000640 1 -- 172.21.15.27:0/2001095419 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea40ac410 con 0x7f9ebc0b2fc0 2024-01-15T18:14:26.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.906+0000 7f9eba000640 1 -- 172.21.15.27:0/2001095419 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ea40ac570 con 0x7f9ebc0b2fc0 2024-01-15T18:14:26.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.906+0000 7f9eba000640 1 -- 172.21.15.27:0/2001095419 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea40ac900 con 0x7f9ebc0b2fc0 2024-01-15T18:14:26.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.906+0000 7f9ec2200640 1 -- 172.21.15.27:0/2001095419 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9ebc0b2fc0 msgr2=0x7f9ebc06b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:26.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.906+0000 7f9ec2200640 1 --2- 172.21.15.27:0/2001095419 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9ebc0b2fc0 0x7f9ebc06b6f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4009870 tx=0x7f9ea401c480 comp rx=0 tx=0).stop 2024-01-15T18:14:26.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.906+0000 7f9ec2200640 1 -- 172.21.15.27:0/2001095419 shutdown_connections 2024-01-15T18:14:26.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.906+0000 7f9ec2200640 1 --2- 172.21.15.27:0/2001095419 >> v2:172.21.15.93:3300/0 conn(0x7f9ebc06bc30 0x7f9ebc06a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:26.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.906+0000 7f9ec2200640 1 --2- 172.21.15.27:0/2001095419 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9ebc0b2fc0 0x7f9ebc06b6f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:26.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.906+0000 7f9ec2200640 1 -- 172.21.15.27:0/2001095419 >> 172.21.15.27:0/2001095419 conn(0x7f9ebc0b1620 msgr2=0x7f9ebc0b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:26.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.906+0000 7f9ec2200640 1 -- 172.21.15.27:0/2001095419 shutdown_connections 2024-01-15T18:14:26.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.906+0000 7f9ec2200640 1 -- 172.21.15.27:0/2001095419 wait complete. 2024-01-15T18:14:26.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.910+0000 7f9ec2200640 1 Processor -- start 2024-01-15T18:14:26.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.910+0000 7f9ec2200640 1 -- start start 2024-01-15T18:14:26.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.910+0000 7f9ec2200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9ebc06bc30 0x7f9ebc15cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:26.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.910+0000 7f9ec2200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9ebc070cd0 0x7f9ebc153370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:26.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.910+0000 7f9ec2200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ebc0b2fc0 0x7f9ebc14d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:26.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.910+0000 7f9ec2200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ebc15e1e0 con 0x7f9ebc0b2fc0 2024-01-15T18:14:26.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.910+0000 7f9ec2200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ebc1538b0 con 0x7f9ebc070cd0 2024-01-15T18:14:26.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.910+0000 7f9ec2200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ebc152410 con 0x7f9ebc06bc30 2024-01-15T18:14:26.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.910+0000 7f9ebbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ebc0b2fc0 0x7f9ebc14d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:26.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.910+0000 7f9ebbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ebc0b2fc0 0x7f9ebc14d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36878/0 (socket says 172.21.15.27:36878) 2024-01-15T18:14:26.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.910+0000 7f9ebbe00640 1 -- 172.21.15.27:0/1035989463 learned_addr learned my addr 172.21.15.27:0/1035989463 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:14:26.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.910+0000 7f9ebaa00640 1 --2- 172.21.15.27:0/1035989463 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9ebc070cd0 0x7f9ebc153370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:26.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.910+0000 7f9ebbe00640 1 -- 172.21.15.27:0/1035989463 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9ebc06bc30 msgr2=0x7f9ebc15cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:26.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.910+0000 7f9ebbe00640 1 --2- 172.21.15.27:0/1035989463 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9ebc06bc30 0x7f9ebc15cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:26.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.910+0000 7f9ebbe00640 1 -- 172.21.15.27:0/1035989463 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9ebc070cd0 msgr2=0x7f9ebc153370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:26.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.910+0000 7f9ebbe00640 1 --2- 172.21.15.27:0/1035989463 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9ebc070cd0 0x7f9ebc153370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:26.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.910+0000 7f9ebbe00640 1 -- 172.21.15.27:0/1035989463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea40094d0 con 0x7f9ebc0b2fc0 2024-01-15T18:14:26.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.914+0000 7f9ebbe00640 1 --2- 172.21.15.27:0/1035989463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ebc0b2fc0 0x7f9ebc14d7b0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f9eb400cd70 tx=0x7f9eb400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:26.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.914+0000 7f9eb3400640 1 -- 172.21.15.27:0/1035989463 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb4003ba0 con 0x7f9ebc0b2fc0 2024-01-15T18:14:26.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.914+0000 7f9eb3400640 1 -- 172.21.15.27:0/1035989463 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb4003d00 con 0x7f9ebc0b2fc0 2024-01-15T18:14:26.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.914+0000 7f9ec2200640 1 -- 172.21.15.27:0/1035989463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ebc06d1f0 con 0x7f9ebc0b2fc0 2024-01-15T18:14:26.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.914+0000 7f9eb3400640 1 -- 172.21.15.27:0/1035989463 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb4004240 con 0x7f9ebc0b2fc0 2024-01-15T18:14:26.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.914+0000 7f9ec2200640 1 -- 172.21.15.27:0/1035989463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ebc15d8b0 con 0x7f9ebc0b2fc0 2024-01-15T18:14:26.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.914+0000 7f9ec2200640 1 -- 172.21.15.27:0/1035989463 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ebc15e390 con 0x7f9ebc0b2fc0 2024-01-15T18:14:26.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.918+0000 7f9eb3400640 1 -- 172.21.15.27:0/1035989463 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 87180+0+0 (secure 0 0 0) 0x7f9eb40043a0 con 0x7f9ebc0b2fc0 2024-01-15T18:14:26.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.918+0000 7f9eb3400640 1 --2- 172.21.15.27:0/1035989463 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9e94067460 0x7f9e94069920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:26.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.918+0000 7f9eb3400640 1 -- 172.21.15.27:0/1035989463 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f9eb4012070 con 0x7f9ebc0b2fc0 2024-01-15T18:14:26.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.918+0000 7f9ebb400640 1 --2- 172.21.15.27:0/1035989463 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9e94067460 0x7f9e94069920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.926+0000 7f9ebb400640 1 --2- 172.21.15.27:0/1035989463 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9e94067460 0x7f9e94069920 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f9eac000f70 tx=0x7f9eac0079c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:26.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:26.926+0000 7f9eb3400640 1 -- 172.21.15.27:0/1035989463 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9ebc15e390 con 0x7f9ebc0b2fc0 2024-01-15T18:14:27.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.198+0000 7f9ec2200640 1 -- 172.21.15.27:0/1035989463 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone519312", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ebc16e310 con 0x7f9e94067460 2024-01-15T18:14:27.371 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:14:27.374 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:14:27.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.422+0000 7f9eb3400640 1 -- 172.21.15.27:0/1035989463 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9eac0012e0 con 0x7f9e94067460 2024-01-15T18:14:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.426+0000 7f9ec2200640 1 -- 172.21.15.27:0/1035989463 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9e94067460 msgr2=0x7f9e94069920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.426+0000 7f9ec2200640 1 --2- 172.21.15.27:0/1035989463 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9e94067460 0x7f9e94069920 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f9eac000f70 tx=0x7f9eac0079c0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.426+0000 7f9ec2200640 1 -- 172.21.15.27:0/1035989463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ebc0b2fc0 msgr2=0x7f9ebc14d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.426+0000 7f9ec2200640 1 --2- 172.21.15.27:0/1035989463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ebc0b2fc0 0x7f9ebc14d7b0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f9eb400cd70 tx=0x7f9eb400a660 comp rx=0 tx=0).stop 2024-01-15T18:14:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.426+0000 7f9ec2200640 1 -- 172.21.15.27:0/1035989463 shutdown_connections 2024-01-15T18:14:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.426+0000 7f9ec2200640 1 --2- 172.21.15.27:0/1035989463 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9e94067460 0x7f9e94069920 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.426+0000 7f9ec2200640 1 --2- 172.21.15.27:0/1035989463 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ebc0b2fc0 0x7f9ebc14d7b0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.426+0000 7f9ec2200640 1 --2- 172.21.15.27:0/1035989463 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9ebc070cd0 0x7f9ebc153370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.426+0000 7f9ec2200640 1 --2- 172.21.15.27:0/1035989463 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9ebc06bc30 0x7f9ebc15cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.426+0000 7f9ec2200640 1 -- 172.21.15.27:0/1035989463 >> 172.21.15.27:0/1035989463 conn(0x7f9ebc0b1620 msgr2=0x7f9ebc0710b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:27.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.426+0000 7f9ec2200640 1 -- 172.21.15.27:0/1035989463 shutdown_connections 2024-01-15T18:14:27.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.426+0000 7f9ec2200640 1 -- 172.21.15.27:0/1035989463 wait complete. 2024-01-15T18:14:27.446 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone519313 2024-01-15T18:14:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.634+0000 7f3408800640 1 Processor -- start 2024-01-15T18:14:27.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.634+0000 7f3408800640 1 -- start start 2024-01-15T18:14:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.638+0000 7f3408800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f340006c430 0x7f3400070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.638+0000 7f3408800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34000ccfb0 con 0x7f34000ce760 2024-01-15T18:14:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.638+0000 7f3408800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34000cd290 con 0x7f34000b1db0 2024-01-15T18:14:27.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.638+0000 7f3408800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34000cd570 con 0x7f340006c430 2024-01-15T18:14:27.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.638+0000 7f3406800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f34000ce760 0x7f340006c810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55388/0 (socket says 172.21.15.27:55388) 2024-01-15T18:14:27.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.638+0000 7f3406800640 1 -- 172.21.15.27:0/1814656745 learned_addr learned my addr 172.21.15.27:0/1814656745 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:14:27.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.638+0000 7f3405e00640 1 --2- 172.21.15.27:0/1814656745 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f340006c430 0x7f3400070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:27.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.638+0000 7f3405e00640 1 -- 172.21.15.27:0/1814656745 >> v1:172.21.15.93:6789/0 conn(0x7f34000b1db0 legacy=0x7f34000ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:27.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.638+0000 7f3405e00640 1 -- 172.21.15.27:0/1814656745 >> v1:172.21.15.27:6789/0 conn(0x7f34000ce760 legacy=0x7f340006c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:27.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.638+0000 7f3404a00640 1 -- 172.21.15.27:0/1814656745 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684721681 0 0) 0x7f34000ccfb0 con 0x7f34000ce760 2024-01-15T18:14:27.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.638+0000 7f3405e00640 1 -- 172.21.15.27:0/1814656745 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34000cd850 con 0x7f340006c430 2024-01-15T18:14:27.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.642+0000 7f3405e00640 1 --2- 172.21.15.27:0/1814656745 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f340006c430 0x7f3400070ba0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f33f00097f0 tx=0x7f33f001e620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c983cb02b1248b1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:27.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.642+0000 7f3404a00640 1 -- 172.21.15.27:0/1814656745 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f00ab400 con 0x7f340006c430 2024-01-15T18:14:27.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.642+0000 7f3404a00640 1 -- 172.21.15.27:0/1814656745 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f00ab560 con 0x7f340006c430 2024-01-15T18:14:27.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.642+0000 7f3404a00640 1 -- 172.21.15.27:0/1814656745 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f00ab8f0 con 0x7f340006c430 2024-01-15T18:14:27.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.642+0000 7f3408800640 1 -- 172.21.15.27:0/1814656745 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f340006c430 msgr2=0x7f3400070ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:27.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.642+0000 7f3408800640 1 --2- 172.21.15.27:0/1814656745 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f340006c430 0x7f3400070ba0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f33f00097f0 tx=0x7f33f001e620 comp rx=0 tx=0).stop 2024-01-15T18:14:27.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.642+0000 7f3408800640 1 -- 172.21.15.27:0/1814656745 shutdown_connections 2024-01-15T18:14:27.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.642+0000 7f3408800640 1 --2- 172.21.15.27:0/1814656745 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f340006c430 0x7f3400070ba0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.642+0000 7f3408800640 1 -- 172.21.15.27:0/1814656745 >> 172.21.15.27:0/1814656745 conn(0x7f34000cb700 msgr2=0x7f34000cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:27.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.642+0000 7f3408800640 1 -- 172.21.15.27:0/1814656745 shutdown_connections 2024-01-15T18:14:27.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.642+0000 7f3408800640 1 -- 172.21.15.27:0/1814656745 wait complete. 2024-01-15T18:14:27.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.642+0000 7f3408800640 1 Processor -- start 2024-01-15T18:14:27.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.642+0000 7f3408800640 1 -- start start 2024-01-15T18:14:27.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.646+0000 7f3408800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f340006c430 0x7f34001421e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:27.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.646+0000 7f3408800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34000b1db0 0x7f340013f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:27.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.646+0000 7f3408800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f34000ce760 0x7f3400140e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:27.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.646+0000 7f3408800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340013faf0 con 0x7f34000b1db0 2024-01-15T18:14:27.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.646+0000 7f3408800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340013fd90 con 0x7f34000ce760 2024-01-15T18:14:27.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.646+0000 7f3408800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3400140070 con 0x7f340006c430 2024-01-15T18:14:27.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.646+0000 7f3405e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f340006c430 0x7f34001421e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:27.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.646+0000 7f3405400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34000b1db0 0x7f340013f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:27.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.646+0000 7f3405e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f340006c430 0x7f34001421e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:35162/0 (socket says 172.21.15.27:35162) 2024-01-15T18:14:27.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.646+0000 7f3405e00640 1 -- 172.21.15.27:0/2458944850 learned_addr learned my addr 172.21.15.27:0/2458944850 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:14:27.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.646+0000 7f3406800640 1 --2- 172.21.15.27:0/2458944850 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f34000ce760 0x7f3400140e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:27.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.646+0000 7f3405e00640 1 -- 172.21.15.27:0/2458944850 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f34000ce760 msgr2=0x7f3400140e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:27.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.646+0000 7f3405e00640 1 --2- 172.21.15.27:0/2458944850 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f34000ce760 0x7f3400140e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.646+0000 7f3405e00640 1 -- 172.21.15.27:0/2458944850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34000b1db0 msgr2=0x7f340013f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:27.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.646+0000 7f3405e00640 1 --2- 172.21.15.27:0/2458944850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34000b1db0 0x7f340013f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.646+0000 7f3405e00640 1 -- 172.21.15.27:0/2458944850 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f0009420 con 0x7f340006c430 2024-01-15T18:14:27.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.646+0000 7f3405e00640 1 --2- 172.21.15.27:0/2458944850 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f340006c430 0x7f34001421e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f33f001b230 tx=0x7f33f001ce60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:27.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.646+0000 7f33fe000640 1 -- 172.21.15.27:0/2458944850 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f00b3800 con 0x7f340006c430 2024-01-15T18:14:27.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.646+0000 7f33fe000640 1 -- 172.21.15.27:0/2458944850 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f00b3980 con 0x7f340006c430 2024-01-15T18:14:27.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.646+0000 7f3408800640 1 -- 172.21.15.27:0/2458944850 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3400140350 con 0x7f340006c430 2024-01-15T18:14:27.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.646+0000 7f33fe000640 1 -- 172.21.15.27:0/2458944850 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f00b3d30 con 0x7f340006c430 2024-01-15T18:14:27.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.646+0000 7f3408800640 1 -- 172.21.15.27:0/2458944850 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3400140910 con 0x7f340006c430 2024-01-15T18:14:27.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.646+0000 7f3408800640 1 -- 172.21.15.27:0/2458944850 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f340004e850 con 0x7f340006c430 2024-01-15T18:14:27.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.650+0000 7f33fe000640 1 -- 172.21.15.27:0/2458944850 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 26) v1 ==== 87180+0+0 (secure 0 0 0) 0x7f33f00c6390 con 0x7f340006c430 2024-01-15T18:14:27.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.650+0000 7f33fe000640 1 --2- 172.21.15.27:0/2458944850 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f33d8067430 0x7f33d80698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:27.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.650+0000 7f33fe000640 1 -- 172.21.15.27:0/2458944850 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f33f011daf0 con 0x7f340006c430 2024-01-15T18:14:27.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.650+0000 7f3405400640 1 --2- 172.21.15.27:0/2458944850 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f33d8067430 0x7f33d80698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:27.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.650+0000 7f3405400640 1 --2- 172.21.15.27:0/2458944850 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f33d8067430 0x7f33d80698f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f340015c990 tx=0x7f33e8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:27.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.654+0000 7f33fe000640 1 -- 172.21.15.27:0/2458944850 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f33f00ec5e0 con 0x7f340006c430 2024-01-15T18:14:27.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:27.842+0000 7f3408800640 1 -- 172.21.15.27:0/2458944850 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone519313", "target": ["mon-mgr", ""]}) v1 -- 0x7f34001671c0 con 0x7f33d8067430 2024-01-15T18:14:37.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.458+0000 7f33fe000640 1 -- 172.21.15.27:0/2458944850 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f33e80014f0 con 0x7f33d8067430 2024-01-15T18:14:37.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.462+0000 7f3408800640 1 -- 172.21.15.27:0/2458944850 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f33d8067430 msgr2=0x7f33d80698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:37.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.462+0000 7f3408800640 1 --2- 172.21.15.27:0/2458944850 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f33d8067430 0x7f33d80698f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f340015c990 tx=0x7f33e8009040 comp rx=0 tx=0).stop 2024-01-15T18:14:37.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.462+0000 7f3408800640 1 -- 172.21.15.27:0/2458944850 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f340006c430 msgr2=0x7f34001421e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:37.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.462+0000 7f3408800640 1 --2- 172.21.15.27:0/2458944850 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f340006c430 0x7f34001421e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f33f001b230 tx=0x7f33f001ce60 comp rx=0 tx=0).stop 2024-01-15T18:14:37.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.462+0000 7f3408800640 1 -- 172.21.15.27:0/2458944850 shutdown_connections 2024-01-15T18:14:37.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.462+0000 7f3408800640 1 --2- 172.21.15.27:0/2458944850 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f33d8067430 0x7f33d80698f0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:37.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.462+0000 7f3408800640 1 --2- 172.21.15.27:0/2458944850 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f34000ce760 0x7f3400140e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:37.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.462+0000 7f3408800640 1 --2- 172.21.15.27:0/2458944850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34000b1db0 0x7f340013f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:37.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.462+0000 7f3408800640 1 --2- 172.21.15.27:0/2458944850 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f340006c430 0x7f34001421e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:37.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.462+0000 7f3408800640 1 -- 172.21.15.27:0/2458944850 >> 172.21.15.27:0/2458944850 conn(0x7f34000cb700 msgr2=0x7f34000b2840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:37.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.462+0000 7f3408800640 1 -- 172.21.15.27:0/2458944850 shutdown_connections 2024-01-15T18:14:37.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.462+0000 7f3408800640 1 -- 172.21.15.27:0/2458944850 wait complete. 2024-01-15T18:14:37.485 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone519314 2024-01-15T18:14:37.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.630+0000 7f31a1c00640 1 Processor -- start 2024-01-15T18:14:37.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.630+0000 7f31a1c00640 1 -- start start 2024-01-15T18:14:37.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.634+0000 7f31a1c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f319c0ce760 0x7f319c0b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:37.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.634+0000 7f31a1c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f319c0712b0 con 0x7f319c0b1db0 2024-01-15T18:14:37.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.634+0000 7f31a1c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f319c0ccfb0 con 0x7f319c06c430 2024-01-15T18:14:37.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.634+0000 7f31a1c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f319c0cd290 con 0x7f319c0ce760 2024-01-15T18:14:37.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.634+0000 7f319b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f319c0ce760 0x7f319c0b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:37.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.634+0000 7f319a000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f319c0b1db0 0x7f319c0ce050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47684/0 (socket says 172.21.15.27:47684) 2024-01-15T18:14:37.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.634+0000 7f319a000640 1 -- 172.21.15.27:0/2775929035 learned_addr learned my addr 172.21.15.27:0/2775929035 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:14:37.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.634+0000 7f319b400640 1 -- 172.21.15.27:0/2775929035 >> v1:172.21.15.93:6789/0 conn(0x7f319c06c430 legacy=0x7f319c070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:37.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.634+0000 7f3199600640 1 -- 172.21.15.27:0/2775929035 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564623577 0 0) 0x7f319c0712b0 con 0x7f319c0b1db0 2024-01-15T18:14:37.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.634+0000 7f319b400640 1 -- 172.21.15.27:0/2775929035 >> v1:172.21.15.27:6789/0 conn(0x7f319c0b1db0 legacy=0x7f319c0ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:37.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.634+0000 7f319b400640 1 -- 172.21.15.27:0/2775929035 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f319c0cd570 con 0x7f319c0ce760 2024-01-15T18:14:37.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.634+0000 7f319b400640 1 --2- 172.21.15.27:0/2775929035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f319c0ce760 0x7f319c0b2240 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f3194009dc0 tx=0x7f319401c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=972401c79d5400be server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:37.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.638+0000 7f3199600640 1 -- 172.21.15.27:0/2775929035 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31940a58f0 con 0x7f319c0ce760 2024-01-15T18:14:37.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.638+0000 7f3199600640 1 -- 172.21.15.27:0/2775929035 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31940a5a50 con 0x7f319c0ce760 2024-01-15T18:14:37.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.638+0000 7f3199600640 1 -- 172.21.15.27:0/2775929035 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31940a5de0 con 0x7f319c0ce760 2024-01-15T18:14:37.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.638+0000 7f31a1c00640 1 -- 172.21.15.27:0/2775929035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f319c0ce760 msgr2=0x7f319c0b2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:37.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.638+0000 7f31a1c00640 1 --2- 172.21.15.27:0/2775929035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f319c0ce760 0x7f319c0b2240 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f3194009dc0 tx=0x7f319401c830 comp rx=0 tx=0).stop 2024-01-15T18:14:37.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.638+0000 7f31a1c00640 1 -- 172.21.15.27:0/2775929035 shutdown_connections 2024-01-15T18:14:37.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.638+0000 7f31a1c00640 1 --2- 172.21.15.27:0/2775929035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f319c0ce760 0x7f319c0b2240 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:37.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.638+0000 7f31a1c00640 1 -- 172.21.15.27:0/2775929035 >> 172.21.15.27:0/2775929035 conn(0x7f319c0cb700 msgr2=0x7f319c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:37.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.638+0000 7f31a1c00640 1 -- 172.21.15.27:0/2775929035 shutdown_connections 2024-01-15T18:14:37.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.638+0000 7f31a1c00640 1 -- 172.21.15.27:0/2775929035 wait complete. 2024-01-15T18:14:37.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.638+0000 7f31a1c00640 1 Processor -- start 2024-01-15T18:14:37.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.638+0000 7f31a1c00640 1 -- start start 2024-01-15T18:14:37.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.638+0000 7f31a1c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f319c06c430 0x7f319c0adf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:37.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.642+0000 7f31a1c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f319c0b1db0 0x7f319c151ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:37.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.642+0000 7f31a1c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f319c0ce760 0x7f319c1451a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:37.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.642+0000 7f31a1c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319c153d90 con 0x7f319c0ce760 2024-01-15T18:14:37.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.642+0000 7f31a1c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319c154630 con 0x7f319c0b1db0 2024-01-15T18:14:37.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.642+0000 7f31a1c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f319c158fc0 con 0x7f319c06c430 2024-01-15T18:14:37.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.642+0000 7f319b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f319c0ce760 0x7f319c1451a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:37.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.642+0000 7f319b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f319c0ce760 0x7f319c1451a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35380/0 (socket says 172.21.15.27:35380) 2024-01-15T18:14:37.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.642+0000 7f319b400640 1 -- 172.21.15.27:0/2973308328 learned_addr learned my addr 172.21.15.27:0/2973308328 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:14:37.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.642+0000 7f319a000640 1 --2- 172.21.15.27:0/2973308328 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f319c0b1db0 0x7f319c151ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:37.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.642+0000 7f319aa00640 1 --2- 172.21.15.27:0/2973308328 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f319c06c430 0x7f319c0adf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:37.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.642+0000 7f319b400640 1 -- 172.21.15.27:0/2973308328 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f319c06c430 msgr2=0x7f319c0adf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:37.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.642+0000 7f319b400640 1 --2- 172.21.15.27:0/2973308328 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f319c06c430 0x7f319c0adf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:37.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.642+0000 7f319b400640 1 -- 172.21.15.27:0/2973308328 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f319c0b1db0 msgr2=0x7f319c151ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:37.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.642+0000 7f319b400640 1 --2- 172.21.15.27:0/2973308328 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f319c0b1db0 0x7f319c151ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:37.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.642+0000 7f319b400640 1 -- 172.21.15.27:0/2973308328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3194009a70 con 0x7f319c0ce760 2024-01-15T18:14:37.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.642+0000 7f319aa00640 1 --2- 172.21.15.27:0/2973308328 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f319c06c430 0x7f319c0adf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:14:37.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.642+0000 7f319b400640 1 --2- 172.21.15.27:0/2973308328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f319c0ce760 0x7f319c1451a0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f31940b0040 tx=0x7f31940195d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:37.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.642+0000 7f3192a00640 1 -- 172.21.15.27:0/2973308328 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31940ad850 con 0x7f319c0ce760 2024-01-15T18:14:37.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.642+0000 7f3192a00640 1 -- 172.21.15.27:0/2973308328 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31940ad9b0 con 0x7f319c0ce760 2024-01-15T18:14:37.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.642+0000 7f31a1c00640 1 -- 172.21.15.27:0/2973308328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f319c153fe0 con 0x7f319c0ce760 2024-01-15T18:14:37.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.642+0000 7f3192a00640 1 -- 172.21.15.27:0/2973308328 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31940add80 con 0x7f319c0ce760 2024-01-15T18:14:37.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.642+0000 7f31a1c00640 1 -- 172.21.15.27:0/2973308328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f319c154ac0 con 0x7f319c0ce760 2024-01-15T18:14:37.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.642+0000 7f31a1c00640 1 -- 172.21.15.27:0/2973308328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f319c0cb500 con 0x7f319c0ce760 2024-01-15T18:14:37.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.642+0000 7f3192a00640 1 -- 172.21.15.27:0/2973308328 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 87180+0+0 (secure 0 0 0) 0x7f31940b4050 con 0x7f319c0ce760 2024-01-15T18:14:37.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.646+0000 7f3192a00640 1 --2- 172.21.15.27:0/2973308328 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3174067430 0x7f31740698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:37.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.646+0000 7f3192a00640 1 -- 172.21.15.27:0/2973308328 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f319411dd50 con 0x7f319c0ce760 2024-01-15T18:14:37.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.646+0000 7f319aa00640 1 --2- 172.21.15.27:0/2973308328 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3174067430 0x7f31740698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:37.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.646+0000 7f319aa00640 1 --2- 172.21.15.27:0/2973308328 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3174067430 0x7f31740698f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f319c152dc0 tx=0x7f318c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:37.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.650+0000 7f3192a00640 1 -- 172.21.15.27:0/2973308328 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f31940ecbb0 con 0x7f319c0ce760 2024-01-15T18:14:37.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:37.854+0000 7f31a1c00640 1 -- 172.21.15.27:0/2973308328 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone519314", "target": ["mon-mgr", ""]}) v1 -- 0x7f319c0cbd10 con 0x7f3174067430 2024-01-15T18:14:39.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:39.178+0000 7f3192a00640 1 -- 172.21.15.27:0/2973308328 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 27) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f31940ecf80 con 0x7f319c0ce760 2024-01-15T18:14:46.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:46.954+0000 7f3192a00640 1 -- 172.21.15.27:0/2973308328 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f319c0cbd10 con 0x7f3174067430 2024-01-15T18:14:46.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:46.958+0000 7f31a1c00640 1 -- 172.21.15.27:0/2973308328 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3174067430 msgr2=0x7f31740698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:46.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:46.958+0000 7f31a1c00640 1 --2- 172.21.15.27:0/2973308328 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3174067430 0x7f31740698f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f319c152dc0 tx=0x7f318c009040 comp rx=0 tx=0).stop 2024-01-15T18:14:46.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:46.958+0000 7f31a1c00640 1 -- 172.21.15.27:0/2973308328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f319c0ce760 msgr2=0x7f319c1451a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:46.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:46.958+0000 7f31a1c00640 1 --2- 172.21.15.27:0/2973308328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f319c0ce760 0x7f319c1451a0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f31940b0040 tx=0x7f31940195d0 comp rx=0 tx=0).stop 2024-01-15T18:14:46.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:46.958+0000 7f31a1c00640 1 -- 172.21.15.27:0/2973308328 shutdown_connections 2024-01-15T18:14:46.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:46.958+0000 7f31a1c00640 1 --2- 172.21.15.27:0/2973308328 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3174067430 0x7f31740698f0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:46.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:46.958+0000 7f31a1c00640 1 --2- 172.21.15.27:0/2973308328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f319c0ce760 0x7f319c1451a0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:46.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:46.958+0000 7f31a1c00640 1 --2- 172.21.15.27:0/2973308328 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f319c0b1db0 0x7f319c151ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:46.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:46.958+0000 7f31a1c00640 1 --2- 172.21.15.27:0/2973308328 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f319c06c430 0x7f319c0adf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:46.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:46.958+0000 7f31a1c00640 1 -- 172.21.15.27:0/2973308328 >> 172.21.15.27:0/2973308328 conn(0x7f319c0cb700 msgr2=0x7f319c0b31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:46.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:46.958+0000 7f31a1c00640 1 -- 172.21.15.27:0/2973308328 shutdown_connections 2024-01-15T18:14:46.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:46.958+0000 7f31a1c00640 1 -- 172.21.15.27:0/2973308328 wait complete. 2024-01-15T18:14:46.982 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone519315 2024-01-15T18:14:47.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.130+0000 7f92f9a00640 1 Processor -- start 2024-01-15T18:14:47.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.130+0000 7f92f9a00640 1 -- start start 2024-01-15T18:14:47.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.130+0000 7f92f9a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f92f4146a60 0x7f92f40ac880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:47.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.130+0000 7f92f9a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f92f4160cb0 0x7f92f41610f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:47.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.130+0000 7f92f9a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f92f406a4f0 0x7f92f406a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.130+0000 7f92f9a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92f4161630 con 0x7f92f406a4f0 2024-01-15T18:14:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.130+0000 7f92f9a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92f406aec0 con 0x7f92f4160cb0 2024-01-15T18:14:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.130+0000 7f92f9a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f40cde30 con 0x7f92f4146a60 2024-01-15T18:14:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.130+0000 7f92f3400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f92f406a4f0 0x7f92f406a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.130+0000 7f92f3400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f92f406a4f0 0x7f92f406a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55316/0 (socket says 172.21.15.27:55316) 2024-01-15T18:14:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.130+0000 7f92f3400640 1 -- 172.21.15.27:0/4294509927 learned_addr learned my addr 172.21.15.27:0/4294509927 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:14:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.130+0000 7f92f2000640 1 --2- 172.21.15.27:0/4294509927 >> v2:172.21.15.93:3300/0 conn(0x7f92f4160cb0 0x7f92f41610f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.130+0000 7f92f2a00640 1 --2- 172.21.15.27:0/4294509927 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f92f4146a60 0x7f92f40ac880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.130+0000 7f92f3400640 1 -- 172.21.15.27:0/4294509927 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f92f4146a60 msgr2=0x7f92f40ac880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.130+0000 7f92f3400640 1 --2- 172.21.15.27:0/4294509927 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f92f4146a60 0x7f92f40ac880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.130+0000 7f92f3400640 1 -- 172.21.15.27:0/4294509927 >> v2:172.21.15.93:3300/0 conn(0x7f92f4160cb0 msgr2=0x7f92f41610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:47.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.130+0000 7f92f3400640 1 --2- 172.21.15.27:0/4294509927 >> v2:172.21.15.93:3300/0 conn(0x7f92f4160cb0 0x7f92f41610f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:47.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.130+0000 7f92f3400640 1 -- 172.21.15.27:0/4294509927 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f406fa50 con 0x7f92f406a4f0 2024-01-15T18:14:47.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.134+0000 7f92f3400640 1 --2- 172.21.15.27:0/4294509927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92f406a4f0 0x7f92f406a980 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f92f40cd820 tx=0x7f92ec01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d96dc4e2057e823 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:47.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.134+0000 7f92f1600640 1 -- 172.21.15.27:0/4294509927 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec0a5820 con 0x7f92f406a4f0 2024-01-15T18:14:47.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.134+0000 7f92f1600640 1 -- 172.21.15.27:0/4294509927 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ec0a5980 con 0x7f92f406a4f0 2024-01-15T18:14:47.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.134+0000 7f92f1600640 1 -- 172.21.15.27:0/4294509927 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec0a5d10 con 0x7f92f406a4f0 2024-01-15T18:14:47.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.134+0000 7f92f9a00640 1 -- 172.21.15.27:0/4294509927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92f406a4f0 msgr2=0x7f92f406a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:47.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.134+0000 7f92f9a00640 1 --2- 172.21.15.27:0/4294509927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92f406a4f0 0x7f92f406a980 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f92f40cd820 tx=0x7f92ec01c9a0 comp rx=0 tx=0).stop 2024-01-15T18:14:47.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.134+0000 7f92f9a00640 1 -- 172.21.15.27:0/4294509927 shutdown_connections 2024-01-15T18:14:47.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.134+0000 7f92f9a00640 1 --2- 172.21.15.27:0/4294509927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92f406a4f0 0x7f92f406a980 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:47.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.134+0000 7f92f9a00640 1 --2- 172.21.15.27:0/4294509927 >> v2:172.21.15.93:3300/0 conn(0x7f92f4160cb0 0x7f92f41610f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:47.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.134+0000 7f92f9a00640 1 --2- 172.21.15.27:0/4294509927 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f92f4146a60 0x7f92f40ac880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:47.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.134+0000 7f92f9a00640 1 -- 172.21.15.27:0/4294509927 >> 172.21.15.27:0/4294509927 conn(0x7f92f40cce10 msgr2=0x7f92f40b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:47.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.134+0000 7f92f9a00640 1 -- 172.21.15.27:0/4294509927 shutdown_connections 2024-01-15T18:14:47.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.134+0000 7f92f9a00640 1 -- 172.21.15.27:0/4294509927 wait complete. 2024-01-15T18:14:47.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.138+0000 7f92f9a00640 1 Processor -- start 2024-01-15T18:14:47.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.138+0000 7f92f9a00640 1 -- start start 2024-01-15T18:14:47.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.138+0000 7f92f9a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92f406a4f0 0x7f92f4138610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:47.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.138+0000 7f92f9a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f92f4146a60 0x7f92f4138b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:47.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.138+0000 7f92f9a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f92f4160cb0 0x7f92f40b3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:47.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.138+0000 7f92f9a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f4139570 con 0x7f92f406a4f0 2024-01-15T18:14:47.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.138+0000 7f92f9a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f4136c10 con 0x7f92f4146a60 2024-01-15T18:14:47.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.138+0000 7f92f9a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f41377d0 con 0x7f92f4160cb0 2024-01-15T18:14:47.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.138+0000 7f92f2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92f406a4f0 0x7f92f4138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:47.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.138+0000 7f92f2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92f406a4f0 0x7f92f4138610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55322/0 (socket says 172.21.15.27:55322) 2024-01-15T18:14:47.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.138+0000 7f92f2a00640 1 -- 172.21.15.27:0/3707681611 learned_addr learned my addr 172.21.15.27:0/3707681611 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:14:47.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.138+0000 7f92f2000640 1 --2- 172.21.15.27:0/3707681611 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f92f4146a60 0x7f92f4138b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:47.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.138+0000 7f92f3400640 1 --2- 172.21.15.27:0/3707681611 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f92f4160cb0 0x7f92f40b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:47.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.138+0000 7f92f2a00640 1 -- 172.21.15.27:0/3707681611 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f92f4160cb0 msgr2=0x7f92f40b3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:47.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.138+0000 7f92f2a00640 1 --2- 172.21.15.27:0/3707681611 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f92f4160cb0 0x7f92f40b3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:47.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.138+0000 7f92f2a00640 1 -- 172.21.15.27:0/3707681611 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f92f4146a60 msgr2=0x7f92f4138b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:47.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.138+0000 7f92f2a00640 1 --2- 172.21.15.27:0/3707681611 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f92f4146a60 0x7f92f4138b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:47.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.138+0000 7f92f2a00640 1 -- 172.21.15.27:0/3707681611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ec009b30 con 0x7f92f406a4f0 2024-01-15T18:14:47.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.138+0000 7f92f2a00640 1 --2- 172.21.15.27:0/3707681611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92f406a4f0 0x7f92f4138610 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f92e400d800 tx=0x7f92e400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:47.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.138+0000 7f92eaa00640 1 -- 172.21.15.27:0/3707681611 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e40096a0 con 0x7f92f406a4f0 2024-01-15T18:14:47.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.138+0000 7f92eaa00640 1 -- 172.21.15.27:0/3707681611 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e4009800 con 0x7f92f406a4f0 2024-01-15T18:14:47.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.138+0000 7f92eaa00640 1 -- 172.21.15.27:0/3707681611 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e4009b70 con 0x7f92f406a4f0 2024-01-15T18:14:47.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.138+0000 7f92f9a00640 1 -- 172.21.15.27:0/3707681611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f415e5a0 con 0x7f92f406a4f0 2024-01-15T18:14:47.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.138+0000 7f92f9a00640 1 -- 172.21.15.27:0/3707681611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f4137180 con 0x7f92f406a4f0 2024-01-15T18:14:47.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.142+0000 7f92f9a00640 1 -- 172.21.15.27:0/3707681611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92f404e850 con 0x7f92f406a4f0 2024-01-15T18:14:47.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.142+0000 7f92eaa00640 1 -- 172.21.15.27:0/3707681611 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 27) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f92e4009cd0 con 0x7f92f406a4f0 2024-01-15T18:14:47.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.142+0000 7f92eaa00640 1 --2- 172.21.15.27:0/3707681611 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f92cc0673a0 0x7f92cc069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:47.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.142+0000 7f92eaa00640 1 -- 172.21.15.27:0/3707681611 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f92e4083370 con 0x7f92f406a4f0 2024-01-15T18:14:47.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.142+0000 7f92f2000640 1 --2- 172.21.15.27:0/3707681611 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f92cc0673a0 0x7f92cc069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:47.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.142+0000 7f92f2000640 1 --2- 172.21.15.27:0/3707681611 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f92cc0673a0 0x7f92cc069860 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f92f406ee20 tx=0x7f92dc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:47.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.146+0000 7f92eaa00640 1 -- 172.21.15.27:0/3707681611 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f92e40520b0 con 0x7f92f406a4f0 2024-01-15T18:14:47.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.338+0000 7f92f9a00640 1 -- 172.21.15.27:0/3707681611 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone519315", "target": ["mon-mgr", ""]}) v1 -- 0x7f92f413cd40 con 0x7f92cc0673a0 2024-01-15T18:14:47.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.802+0000 7f92eaa00640 1 -- 172.21.15.27:0/3707681611 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f92f413cd40 con 0x7f92cc0673a0 2024-01-15T18:14:47.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.806+0000 7f92f9a00640 1 -- 172.21.15.27:0/3707681611 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f92cc0673a0 msgr2=0x7f92cc069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:47.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.806+0000 7f92f9a00640 1 --2- 172.21.15.27:0/3707681611 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f92cc0673a0 0x7f92cc069860 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f92f406ee20 tx=0x7f92dc008040 comp rx=0 tx=0).stop 2024-01-15T18:14:47.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.806+0000 7f92f9a00640 1 -- 172.21.15.27:0/3707681611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92f406a4f0 msgr2=0x7f92f4138610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:47.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.806+0000 7f92f9a00640 1 --2- 172.21.15.27:0/3707681611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92f406a4f0 0x7f92f4138610 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f92e400d800 tx=0x7f92e400dcc0 comp rx=0 tx=0).stop 2024-01-15T18:14:47.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.806+0000 7f92f9a00640 1 -- 172.21.15.27:0/3707681611 shutdown_connections 2024-01-15T18:14:47.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.806+0000 7f92f9a00640 1 --2- 172.21.15.27:0/3707681611 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f92cc0673a0 0x7f92cc069860 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:47.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.806+0000 7f92f9a00640 1 --2- 172.21.15.27:0/3707681611 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f92f4160cb0 0x7f92f40b3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:47.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.806+0000 7f92f9a00640 1 --2- 172.21.15.27:0/3707681611 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f92f4146a60 0x7f92f4138b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:47.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.806+0000 7f92f9a00640 1 --2- 172.21.15.27:0/3707681611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92f406a4f0 0x7f92f4138610 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:47.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.806+0000 7f92f9a00640 1 -- 172.21.15.27:0/3707681611 >> 172.21.15.27:0/3707681611 conn(0x7f92f40cce10 msgr2=0x7f92f406dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:47.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.806+0000 7f92f9a00640 1 -- 172.21.15.27:0/3707681611 shutdown_connections 2024-01-15T18:14:47.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.806+0000 7f92f9a00640 1 -- 172.21.15.27:0/3707681611 wait complete. 2024-01-15T18:14:47.827 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone519316 2024-01-15T18:14:47.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.970+0000 7f4e24800640 1 Processor -- start 2024-01-15T18:14:47.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.970+0000 7f4e24800640 1 -- start start 2024-01-15T18:14:47.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.974+0000 7f4e24800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e1c06ac00 0x7f4e1c161120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:47.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.974+0000 7f4e24800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e1c0cde30 con 0x7f4e1c160cb0 2024-01-15T18:14:47.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.974+0000 7f4e24800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e1c06fa50 con 0x7f4e1c146a60 2024-01-15T18:14:47.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.974+0000 7f4e24800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1c161820 con 0x7f4e1c06ac00 2024-01-15T18:14:47.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.974+0000 7f4e21400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4e1c160cb0 0x7f4e1c06a4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47182/0 (socket says 172.21.15.27:47182) 2024-01-15T18:14:47.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.974+0000 7f4e21400640 1 -- 172.21.15.27:0/3996624065 learned_addr learned my addr 172.21.15.27:0/3996624065 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:14:47.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.974+0000 7f4e22800640 1 --2- 172.21.15.27:0/3996624065 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e1c06ac00 0x7f4e1c161120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:47.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.974+0000 7f4e20a00640 1 -- 172.21.15.27:0/3996624065 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2976482070 0 0) 0x7f4e1c0cde30 con 0x7f4e1c160cb0 2024-01-15T18:14:47.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.974+0000 7f4e20a00640 1 -- 172.21.15.27:0/3996624065 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dfc003530 con 0x7f4e1c160cb0 2024-01-15T18:14:47.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.974+0000 7f4e22800640 1 -- 172.21.15.27:0/3996624065 >> v1:172.21.15.93:6789/0 conn(0x7f4e1c146a60 legacy=0x7f4e1c0ac880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:47.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.974+0000 7f4e22800640 1 -- 172.21.15.27:0/3996624065 >> v1:172.21.15.27:6789/0 conn(0x7f4e1c160cb0 legacy=0x7f4e1c06a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:47.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.974+0000 7f4e22800640 1 -- 172.21.15.27:0/3996624065 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e1c161b00 con 0x7f4e1c06ac00 2024-01-15T18:14:47.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.978+0000 7f4e22800640 1 --2- 172.21.15.27:0/3996624065 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e1c06ac00 0x7f4e1c161120 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4e14009dc0 tx=0x7f4e1401c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87e392df3bf2e864 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:47.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.978+0000 7f4e20a00640 1 -- 172.21.15.27:0/3996624065 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e140a5a30 con 0x7f4e1c06ac00 2024-01-15T18:14:47.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.978+0000 7f4e20a00640 1 -- 172.21.15.27:0/3996624065 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e140a5b90 con 0x7f4e1c06ac00 2024-01-15T18:14:47.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.978+0000 7f4e20a00640 1 -- 172.21.15.27:0/3996624065 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e140ad440 con 0x7f4e1c06ac00 2024-01-15T18:14:47.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.978+0000 7f4e24800640 1 -- 172.21.15.27:0/3996624065 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e1c06ac00 msgr2=0x7f4e1c161120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:47.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.978+0000 7f4e24800640 1 --2- 172.21.15.27:0/3996624065 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e1c06ac00 0x7f4e1c161120 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4e14009dc0 tx=0x7f4e1401c970 comp rx=0 tx=0).stop 2024-01-15T18:14:47.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.978+0000 7f4e24800640 1 -- 172.21.15.27:0/3996624065 shutdown_connections 2024-01-15T18:14:47.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.978+0000 7f4e24800640 1 --2- 172.21.15.27:0/3996624065 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e1c06ac00 0x7f4e1c161120 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:47.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.978+0000 7f4e24800640 1 -- 172.21.15.27:0/3996624065 >> 172.21.15.27:0/3996624065 conn(0x7f4e1c0cce10 msgr2=0x7f4e1c0b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:47.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.978+0000 7f4e24800640 1 -- 172.21.15.27:0/3996624065 shutdown_connections 2024-01-15T18:14:47.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.978+0000 7f4e24800640 1 -- 172.21.15.27:0/3996624065 wait complete. 2024-01-15T18:14:47.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.978+0000 7f4e24800640 1 Processor -- start 2024-01-15T18:14:47.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.978+0000 7f4e24800640 1 -- start start 2024-01-15T18:14:47.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.982+0000 7f4e24800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e1c06ac00 0x7f4e1c15d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:47.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.982+0000 7f4e24800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e1c146a60 0x7f4e1c156b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:47.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.982+0000 7f4e24800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e1c160cb0 0x7f4e1c157090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:47.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.982+0000 7f4e24800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1c15dcb0 con 0x7f4e1c146a60 2024-01-15T18:14:47.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.982+0000 7f4e24800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1c15dec0 con 0x7f4e1c160cb0 2024-01-15T18:14:47.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.982+0000 7f4e24800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1c1f5990 con 0x7f4e1c06ac00 2024-01-15T18:14:47.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.982+0000 7f4e22800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e1c160cb0 0x7f4e1c157090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:47.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.982+0000 7f4e21400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e1c146a60 0x7f4e1c156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:47.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.982+0000 7f4e22800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e1c160cb0 0x7f4e1c157090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:49216/0 (socket says 172.21.15.27:49216) 2024-01-15T18:14:47.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.982+0000 7f4e22800640 1 -- 172.21.15.27:0/605388198 learned_addr learned my addr 172.21.15.27:0/605388198 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:14:47.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.982+0000 7f4e21e00640 1 --2- 172.21.15.27:0/605388198 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e1c06ac00 0x7f4e1c15d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:47.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.982+0000 7f4e22800640 1 -- 172.21.15.27:0/605388198 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e1c06ac00 msgr2=0x7f4e1c15d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:47.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.982+0000 7f4e22800640 1 --2- 172.21.15.27:0/605388198 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e1c06ac00 0x7f4e1c15d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:47.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.982+0000 7f4e22800640 1 -- 172.21.15.27:0/605388198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e1c146a60 msgr2=0x7f4e1c156b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:47.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.982+0000 7f4e22800640 1 --2- 172.21.15.27:0/605388198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e1c146a60 0x7f4e1c156b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:47.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.982+0000 7f4e22800640 1 -- 172.21.15.27:0/605388198 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e14009a70 con 0x7f4e1c160cb0 2024-01-15T18:14:47.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.982+0000 7f4e22800640 1 --2- 172.21.15.27:0/605388198 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e1c160cb0 0x7f4e1c157090 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4e1401a5a0 tx=0x7f4e14019710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:47.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.982+0000 7f4e1a000640 1 -- 172.21.15.27:0/605388198 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e140add50 con 0x7f4e1c160cb0 2024-01-15T18:14:47.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.982+0000 7f4e1a000640 1 -- 172.21.15.27:0/605388198 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e14009450 con 0x7f4e1c160cb0 2024-01-15T18:14:47.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.982+0000 7f4e1a000640 1 -- 172.21.15.27:0/605388198 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e140b9670 con 0x7f4e1c160cb0 2024-01-15T18:14:47.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.982+0000 7f4e24800640 1 -- 172.21.15.27:0/605388198 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e1c1f5c70 con 0x7f4e1c160cb0 2024-01-15T18:14:47.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.982+0000 7f4e24800640 1 -- 172.21.15.27:0/605388198 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e1c1f5660 con 0x7f4e1c160cb0 2024-01-15T18:14:47.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.982+0000 7f4e24800640 1 -- 172.21.15.27:0/605388198 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e1c04e850 con 0x7f4e1c160cb0 2024-01-15T18:14:47.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.982+0000 7f4e1a000640 1 -- 172.21.15.27:0/605388198 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 27) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f4e140b97d0 con 0x7f4e1c160cb0 2024-01-15T18:14:47.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.986+0000 7f4e1a000640 1 --2- 172.21.15.27:0/605388198 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4df40673a0 0x7f4df4069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:47.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.986+0000 7f4e1a000640 1 -- 172.21.15.27:0/605388198 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f4e1411e230 con 0x7f4e1c160cb0 2024-01-15T18:14:47.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.986+0000 7f4e21e00640 1 --2- 172.21.15.27:0/605388198 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4df40673a0 0x7f4df4069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:47.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.986+0000 7f4e21e00640 1 --2- 172.21.15.27:0/605388198 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4df40673a0 0x7f4df4069860 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c0b2590 tx=0x7f4e0c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:47.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:47.990+0000 7f4e1a000640 1 -- 172.21.15.27:0/605388198 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4e140ecfd0 con 0x7f4e1c160cb0 2024-01-15T18:14:48.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.182+0000 7f4e24800640 1 -- 172.21.15.27:0/605388198 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone519316", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e1c0cf3d0 con 0x7f4df40673a0 2024-01-15T18:14:48.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.578+0000 7f4e1a000640 1 -- 172.21.15.27:0/605388198 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4e0c001510 con 0x7f4df40673a0 2024-01-15T18:14:48.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.582+0000 7f4e24800640 1 -- 172.21.15.27:0/605388198 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4df40673a0 msgr2=0x7f4df4069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:48.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.582+0000 7f4e24800640 1 --2- 172.21.15.27:0/605388198 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4df40673a0 0x7f4df4069860 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c0b2590 tx=0x7f4e0c009040 comp rx=0 tx=0).stop 2024-01-15T18:14:48.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.582+0000 7f4e24800640 1 -- 172.21.15.27:0/605388198 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e1c160cb0 msgr2=0x7f4e1c157090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:48.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.582+0000 7f4e24800640 1 --2- 172.21.15.27:0/605388198 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e1c160cb0 0x7f4e1c157090 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4e1401a5a0 tx=0x7f4e14019710 comp rx=0 tx=0).stop 2024-01-15T18:14:48.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.582+0000 7f4e24800640 1 -- 172.21.15.27:0/605388198 shutdown_connections 2024-01-15T18:14:48.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.582+0000 7f4e24800640 1 --2- 172.21.15.27:0/605388198 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4df40673a0 0x7f4df4069860 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:48.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.582+0000 7f4e24800640 1 --2- 172.21.15.27:0/605388198 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4e1c160cb0 0x7f4e1c157090 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:48.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.582+0000 7f4e24800640 1 --2- 172.21.15.27:0/605388198 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e1c146a60 0x7f4e1c156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:48.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.582+0000 7f4e24800640 1 --2- 172.21.15.27:0/605388198 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4e1c06ac00 0x7f4e1c15d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:48.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.582+0000 7f4e24800640 1 -- 172.21.15.27:0/605388198 >> 172.21.15.27:0/605388198 conn(0x7f4e1c0cce10 msgr2=0x7f4e1c0711f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:48.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.582+0000 7f4e24800640 1 -- 172.21.15.27:0/605388198 shutdown_connections 2024-01-15T18:14:48.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.582+0000 7f4e24800640 1 -- 172.21.15.27:0/605388198 wait complete. 2024-01-15T18:14:48.603 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone519317 2024-01-15T18:14:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.746+0000 7f4adbc00640 1 Processor -- start 2024-01-15T18:14:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.746+0000 7f4adbc00640 1 -- start start 2024-01-15T18:14:48.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.746+0000 7f4adbc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4ad4158900 0x7f4ad415f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:48.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.746+0000 7f4adbc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4ad415bf00 0x7f4ad40cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:48.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.746+0000 7f4adbc00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad40b1cb0 con 0x7f4ad40cf110 2024-01-15T18:14:48.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.746+0000 7f4adbc00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ad40cf510 con 0x7f4ad415bf00 2024-01-15T18:14:48.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.746+0000 7f4adbc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad40b1f00 con 0x7f4ad4158900 2024-01-15T18:14:48.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.746+0000 7f4ad3e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4ad415bf00 0x7f4ad40cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:48.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.746+0000 7f4ad9c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4ad40cf110 0x7f4ad40701e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47194/0 (socket says 172.21.15.27:47194) 2024-01-15T18:14:48.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.746+0000 7f4ad9c00640 1 -- 172.21.15.27:0/3662308789 learned_addr learned my addr 172.21.15.27:0/3662308789 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:14:48.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.750+0000 7f4ad9200640 1 --2- 172.21.15.27:0/3662308789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4ad4158900 0x7f4ad415f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.750+0000 7f4ad3e00640 1 -- 172.21.15.27:0/3662308789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4ad4158900 msgr2=0x7f4ad415f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.750+0000 7f4ad3e00640 1 --2- 172.21.15.27:0/3662308789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4ad4158900 0x7f4ad415f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.750+0000 7f4ad3e00640 1 -- 172.21.15.27:0/3662308789 >> v1:172.21.15.27:6789/0 conn(0x7f4ad40cf110 legacy=0x7f4ad40701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.750+0000 7f4ad3400640 1 -- 172.21.15.27:0/3662308789 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497123593 0 0) 0x7f4ad40b1cb0 con 0x7f4ad40cf110 2024-01-15T18:14:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.750+0000 7f4ad3e00640 1 -- 172.21.15.27:0/3662308789 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad40b21e0 con 0x7f4ad415bf00 2024-01-15T18:14:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.750+0000 7f4ad9200640 1 --2- 172.21.15.27:0/3662308789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4ad4158900 0x7f4ad415f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:14:48.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.750+0000 7f4ad3e00640 1 --2- 172.21.15.27:0/3662308789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad415bf00 0x7f4ad40cebd0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f4abc0097c0 tx=0x7f4abc01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2b35a30cf03633 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:48.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.750+0000 7f4ad3400640 1 -- 172.21.15.27:0/3662308789 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4abc0ab600 con 0x7f4ad415bf00 2024-01-15T18:14:48.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.750+0000 7f4ad3400640 1 -- 172.21.15.27:0/3662308789 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4abc0a7050 con 0x7f4ad415bf00 2024-01-15T18:14:48.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.750+0000 7f4ad3400640 1 -- 172.21.15.27:0/3662308789 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4abc0ab980 con 0x7f4ad415bf00 2024-01-15T18:14:48.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.754+0000 7f4adbc00640 1 -- 172.21.15.27:0/3662308789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad415bf00 msgr2=0x7f4ad40cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:48.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.754+0000 7f4adbc00640 1 --2- 172.21.15.27:0/3662308789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad415bf00 0x7f4ad40cebd0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f4abc0097c0 tx=0x7f4abc01c480 comp rx=0 tx=0).stop 2024-01-15T18:14:48.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.754+0000 7f4adbc00640 1 -- 172.21.15.27:0/3662308789 shutdown_connections 2024-01-15T18:14:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.754+0000 7f4adbc00640 1 --2- 172.21.15.27:0/3662308789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad415bf00 0x7f4ad40cebd0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.754+0000 7f4adbc00640 1 --2- 172.21.15.27:0/3662308789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4ad4158900 0x7f4ad415f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.754+0000 7f4adbc00640 1 -- 172.21.15.27:0/3662308789 >> 172.21.15.27:0/3662308789 conn(0x7f4ad40cb700 msgr2=0x7f4ad40cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.754+0000 7f4adbc00640 1 -- 172.21.15.27:0/3662308789 shutdown_connections 2024-01-15T18:14:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.754+0000 7f4adbc00640 1 -- 172.21.15.27:0/3662308789 wait complete. 2024-01-15T18:14:48.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.754+0000 7f4adbc00640 1 Processor -- start 2024-01-15T18:14:48.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.754+0000 7f4adbc00640 1 -- start start 2024-01-15T18:14:48.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.754+0000 7f4adbc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ad40cf110 0x7f4ad41520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:48.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.754+0000 7f4adbc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad4158900 0x7f4ad415a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:48.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.754+0000 7f4adbc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4ad415bf00 0x7f4ad40b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:48.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.754+0000 7f4adbc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad415aa20 con 0x7f4ad4158900 2024-01-15T18:14:48.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.754+0000 7f4adbc00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad415ad00 con 0x7f4ad40cf110 2024-01-15T18:14:48.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.754+0000 7f4adbc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad415afe0 con 0x7f4ad415bf00 2024-01-15T18:14:48.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.758+0000 7f4ad3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad4158900 0x7f4ad415a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:48.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.758+0000 7f4ad3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad4158900 0x7f4ad415a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55346/0 (socket says 172.21.15.27:55346) 2024-01-15T18:14:48.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.758+0000 7f4ad3e00640 1 -- 172.21.15.27:0/722555858 learned_addr learned my addr 172.21.15.27:0/722555858 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:14:48.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.758+0000 7f4ad9200640 1 --2- 172.21.15.27:0/722555858 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ad40cf110 0x7f4ad41520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:48.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.758+0000 7f4ad9c00640 1 --2- 172.21.15.27:0/722555858 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4ad415bf00 0x7f4ad40b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:48.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.758+0000 7f4ad3e00640 1 -- 172.21.15.27:0/722555858 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4ad415bf00 msgr2=0x7f4ad40b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:48.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.758+0000 7f4ad3e00640 1 --2- 172.21.15.27:0/722555858 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4ad415bf00 0x7f4ad40b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:48.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.758+0000 7f4ad3e00640 1 -- 172.21.15.27:0/722555858 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ad40cf110 msgr2=0x7f4ad41520f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:48.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.758+0000 7f4ad3e00640 1 --2- 172.21.15.27:0/722555858 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ad40cf110 0x7f4ad41520f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:48.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.758+0000 7f4ad3e00640 1 -- 172.21.15.27:0/722555858 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4abc009420 con 0x7f4ad4158900 2024-01-15T18:14:48.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.758+0000 7f4ad9200640 1 --2- 172.21.15.27:0/722555858 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ad40cf110 0x7f4ad41520f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:14:48.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.758+0000 7f4ad3e00640 1 --2- 172.21.15.27:0/722555858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad4158900 0x7f4ad415a1d0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f4abc01ab30 tx=0x7f4abc019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:48.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.758+0000 7f4ad0c00640 1 -- 172.21.15.27:0/722555858 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4abc004220 con 0x7f4ad4158900 2024-01-15T18:14:48.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.758+0000 7f4ad0c00640 1 -- 172.21.15.27:0/722555858 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4abc004380 con 0x7f4ad4158900 2024-01-15T18:14:48.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.758+0000 7f4ad0c00640 1 -- 172.21.15.27:0/722555858 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4abc003820 con 0x7f4ad4158900 2024-01-15T18:14:48.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.758+0000 7f4adbc00640 1 -- 172.21.15.27:0/722555858 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad40ccf30 con 0x7f4ad4158900 2024-01-15T18:14:48.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.758+0000 7f4adbc00640 1 -- 172.21.15.27:0/722555858 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad415d0d0 con 0x7f4ad4158900 2024-01-15T18:14:48.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.758+0000 7f4adbc00640 1 -- 172.21.15.27:0/722555858 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ad40cd120 con 0x7f4ad4158900 2024-01-15T18:14:48.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.758+0000 7f4ad0c00640 1 -- 172.21.15.27:0/722555858 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 27) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f4abc0b2050 con 0x7f4ad4158900 2024-01-15T18:14:48.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.762+0000 7f4ad0c00640 1 --2- 172.21.15.27:0/722555858 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4aac0673a0 0x7f4aac069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:48.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.762+0000 7f4ad0c00640 1 -- 172.21.15.27:0/722555858 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f4abc11d260 con 0x7f4ad4158900 2024-01-15T18:14:48.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.762+0000 7f4ad9200640 1 --2- 172.21.15.27:0/722555858 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4aac0673a0 0x7f4aac069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:48.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.762+0000 7f4ad9200640 1 --2- 172.21.15.27:0/722555858 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4aac0673a0 0x7f4aac069860 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f4ac4009970 tx=0x7f4ac4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:48.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.766+0000 7f4ad0c00640 1 -- 172.21.15.27:0/722555858 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4abc0ebcc0 con 0x7f4ad4158900 2024-01-15T18:14:48.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:48.962+0000 7f4adbc00640 1 -- 172.21.15.27:0/722555858 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone519317", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ad415c380 con 0x7f4aac0673a0 2024-01-15T18:14:57.423 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:14:57.427 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:14:58.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.645+0000 7f4ad0c00640 1 -- 172.21.15.27:0/722555858 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4ac4009600 con 0x7f4aac0673a0 2024-01-15T18:14:58.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.649+0000 7f4adbc00640 1 -- 172.21.15.27:0/722555858 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4aac0673a0 msgr2=0x7f4aac069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:58.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.649+0000 7f4adbc00640 1 --2- 172.21.15.27:0/722555858 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4aac0673a0 0x7f4aac069860 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f4ac4009970 tx=0x7f4ac4008040 comp rx=0 tx=0).stop 2024-01-15T18:14:58.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.649+0000 7f4adbc00640 1 -- 172.21.15.27:0/722555858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad4158900 msgr2=0x7f4ad415a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:58.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.649+0000 7f4adbc00640 1 --2- 172.21.15.27:0/722555858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad4158900 0x7f4ad415a1d0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f4abc01ab30 tx=0x7f4abc019220 comp rx=0 tx=0).stop 2024-01-15T18:14:58.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.649+0000 7f4adbc00640 1 -- 172.21.15.27:0/722555858 shutdown_connections 2024-01-15T18:14:58.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.649+0000 7f4adbc00640 1 --2- 172.21.15.27:0/722555858 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4aac0673a0 0x7f4aac069860 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:58.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.649+0000 7f4adbc00640 1 --2- 172.21.15.27:0/722555858 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4ad415bf00 0x7f4ad40b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:58.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.649+0000 7f4adbc00640 1 --2- 172.21.15.27:0/722555858 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ad4158900 0x7f4ad415a1d0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:58.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.649+0000 7f4adbc00640 1 --2- 172.21.15.27:0/722555858 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ad40cf110 0x7f4ad41520f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:58.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.649+0000 7f4adbc00640 1 -- 172.21.15.27:0/722555858 >> 172.21.15.27:0/722555858 conn(0x7f4ad40cb700 msgr2=0x7f4ad41526d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:58.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.649+0000 7f4adbc00640 1 -- 172.21.15.27:0/722555858 shutdown_connections 2024-01-15T18:14:58.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.649+0000 7f4adbc00640 1 -- 172.21.15.27:0/722555858 wait complete. 2024-01-15T18:14:58.671 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone519318 2024-01-15T18:14:58.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.817+0000 7fade3200640 1 Processor -- start 2024-01-15T18:14:58.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.817+0000 7fade3200640 1 -- start start 2024-01-15T18:14:58.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.817+0000 7fade3200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7faddc146a60 0x7faddc0ac880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:58.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.817+0000 7fade3200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faddc06ac00 0x7faddc161120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:58.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.817+0000 7fade3200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faddc0cde30 con 0x7faddc160cb0 2024-01-15T18:14:58.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.817+0000 7fade3200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faddc06fa50 con 0x7faddc146a60 2024-01-15T18:14:58.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.817+0000 7fade3200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faddc161820 con 0x7faddc06ac00 2024-01-15T18:14:58.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.817+0000 7fade1200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faddc06ac00 0x7faddc161120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:58.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.817+0000 7faddb400640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7faddc160cb0 0x7faddc06a4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:37926/0 (socket says 172.21.15.27:37926) 2024-01-15T18:14:58.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.817+0000 7faddbe00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7faddc146a60 0x7faddc0ac880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:58.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.817+0000 7fade1200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faddc06ac00 0x7faddc161120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:34602/0 (socket says 172.21.15.27:34602) 2024-01-15T18:14:58.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.817+0000 7faddb400640 1 -- 172.21.15.27:0/3907103811 learned_addr learned my addr 172.21.15.27:0/3907103811 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:14:58.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.817+0000 7faddaa00640 1 -- 172.21.15.27:0/3907103811 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860808740 0 0) 0x7faddc0cde30 con 0x7faddc160cb0 2024-01-15T18:14:58.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.817+0000 7faddaa00640 1 -- 172.21.15.27:0/3907103811 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadbc003530 con 0x7faddc160cb0 2024-01-15T18:14:58.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.817+0000 7faddbe00640 1 -- 172.21.15.27:0/3907103811 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faddc06ac00 msgr2=0x7faddc161120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:58.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.817+0000 7faddbe00640 1 --2- 172.21.15.27:0/3907103811 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faddc06ac00 0x7faddc161120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:58.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.817+0000 7faddbe00640 1 -- 172.21.15.27:0/3907103811 >> v1:172.21.15.93:6789/0 conn(0x7faddc160cb0 legacy=0x7faddc06a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:58.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.817+0000 7faddbe00640 1 -- 172.21.15.27:0/3907103811 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faddc161b00 con 0x7faddc146a60 2024-01-15T18:14:58.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.821+0000 7faddbe00640 1 --2- 172.21.15.27:0/3907103811 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faddc146a60 0x7faddc0ac880 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fadcc0097c0 tx=0x7fadcc01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea9382ab4a85092e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:58.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.821+0000 7faddaa00640 1 -- 172.21.15.27:0/3907103811 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc0ab680 con 0x7faddc146a60 2024-01-15T18:14:58.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.821+0000 7faddaa00640 1 -- 172.21.15.27:0/3907103811 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadcc0ab7e0 con 0x7faddc146a60 2024-01-15T18:14:58.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.821+0000 7faddaa00640 1 -- 172.21.15.27:0/3907103811 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc0abb70 con 0x7faddc146a60 2024-01-15T18:14:58.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.821+0000 7fade3200640 1 -- 172.21.15.27:0/3907103811 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faddc146a60 msgr2=0x7faddc0ac880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:58.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.821+0000 7fade3200640 1 --2- 172.21.15.27:0/3907103811 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faddc146a60 0x7faddc0ac880 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fadcc0097c0 tx=0x7fadcc01c480 comp rx=0 tx=0).stop 2024-01-15T18:14:58.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.821+0000 7fade3200640 1 -- 172.21.15.27:0/3907103811 shutdown_connections 2024-01-15T18:14:58.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.821+0000 7fade3200640 1 --2- 172.21.15.27:0/3907103811 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faddc06ac00 0x7faddc161120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:58.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.821+0000 7fade3200640 1 --2- 172.21.15.27:0/3907103811 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faddc146a60 0x7faddc0ac880 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:58.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.821+0000 7fade3200640 1 -- 172.21.15.27:0/3907103811 >> 172.21.15.27:0/3907103811 conn(0x7faddc0cce10 msgr2=0x7faddc0b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:58.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.825+0000 7fade3200640 1 -- 172.21.15.27:0/3907103811 shutdown_connections 2024-01-15T18:14:58.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.825+0000 7fade3200640 1 -- 172.21.15.27:0/3907103811 wait complete. 2024-01-15T18:14:58.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.825+0000 7fade3200640 1 Processor -- start 2024-01-15T18:14:58.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.825+0000 7fade3200640 1 -- start start 2024-01-15T18:14:58.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.825+0000 7fade3200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faddc06ac00 0x7faddc15a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:58.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.825+0000 7fade3200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faddc146a60 0x7faddc0af3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:58.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.825+0000 7fade3200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faddc160cb0 0x7faddc157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:58.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.825+0000 7fade3200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faddc0ae5e0 con 0x7faddc146a60 2024-01-15T18:14:58.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.825+0000 7fade3200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faddc13ff40 con 0x7faddc06ac00 2024-01-15T18:14:58.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.825+0000 7fade3200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faddc0adb40 con 0x7faddc160cb0 2024-01-15T18:14:58.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.825+0000 7faddb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faddc146a60 0x7faddc0af3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:58.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.825+0000 7faddbe00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faddc06ac00 0x7faddc15a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:58.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.825+0000 7faddb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faddc146a60 0x7faddc0af3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38082/0 (socket says 172.21.15.27:38082) 2024-01-15T18:14:58.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.825+0000 7faddb400640 1 -- 172.21.15.27:0/4016679306 learned_addr learned my addr 172.21.15.27:0/4016679306 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:14:58.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.825+0000 7fade1200640 1 --2- 172.21.15.27:0/4016679306 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faddc160cb0 0x7faddc157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:58.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.825+0000 7faddbe00640 1 -- 172.21.15.27:0/4016679306 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faddc160cb0 msgr2=0x7faddc157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:58.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.825+0000 7faddbe00640 1 --2- 172.21.15.27:0/4016679306 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faddc160cb0 0x7faddc157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:58.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.825+0000 7faddbe00640 1 -- 172.21.15.27:0/4016679306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faddc146a60 msgr2=0x7faddc0af3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:58.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.825+0000 7faddbe00640 1 --2- 172.21.15.27:0/4016679306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faddc146a60 0x7faddc0af3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:58.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.825+0000 7faddbe00640 1 -- 172.21.15.27:0/4016679306 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadcc009420 con 0x7faddc06ac00 2024-01-15T18:14:58.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.825+0000 7faddb400640 1 --2- 172.21.15.27:0/4016679306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faddc146a60 0x7faddc0af3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:14:58.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.829+0000 7faddbe00640 1 --2- 172.21.15.27:0/4016679306 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faddc06ac00 0x7faddc15a470 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fadcc0ac000 tx=0x7fadcc01acc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:58.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.829+0000 7fadd3e00640 1 -- 172.21.15.27:0/4016679306 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc0bf610 con 0x7faddc06ac00 2024-01-15T18:14:58.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.829+0000 7fadd3e00640 1 -- 172.21.15.27:0/4016679306 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadcc0bf770 con 0x7faddc06ac00 2024-01-15T18:14:58.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.829+0000 7fade3200640 1 -- 172.21.15.27:0/4016679306 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faddc13fc60 con 0x7faddc06ac00 2024-01-15T18:14:58.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.829+0000 7fadd3e00640 1 -- 172.21.15.27:0/4016679306 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc0bfb20 con 0x7faddc06ac00 2024-01-15T18:14:58.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.829+0000 7fade3200640 1 -- 172.21.15.27:0/4016679306 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faddc0ae0c0 con 0x7faddc06ac00 2024-01-15T18:14:58.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.829+0000 7fade3200640 1 -- 172.21.15.27:0/4016679306 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faddc0ae2b0 con 0x7faddc06ac00 2024-01-15T18:14:58.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.829+0000 7fadd3e00640 1 -- 172.21.15.27:0/4016679306 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 27) v1 ==== 87129+0+0 (secure 0 0 0) 0x7fadcc0bfc80 con 0x7faddc06ac00 2024-01-15T18:14:58.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.829+0000 7fadd3e00640 1 --2- 172.21.15.27:0/4016679306 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fadb40673a0 0x7fadb4069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:58.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.829+0000 7fadd3e00640 1 -- 172.21.15.27:0/4016679306 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fadcc11dd60 con 0x7faddc06ac00 2024-01-15T18:14:58.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.833+0000 7faddb400640 1 --2- 172.21.15.27:0/4016679306 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fadb40673a0 0x7fadb4069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:58.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.833+0000 7faddb400640 1 --2- 172.21.15.27:0/4016679306 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fadb40673a0 0x7fadb4069860 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7faddc0ae970 tx=0x7fadc4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:58.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:58.837+0000 7fadd3e00640 1 -- 172.21.15.27:0/4016679306 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fadcc0ec8d0 con 0x7faddc06ac00 2024-01-15T18:14:59.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:14:59.029+0000 7fade3200640 1 -- 172.21.15.27:0/4016679306 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone519318", "target": ["mon-mgr", ""]}) v1 -- 0x7faddc04e850 con 0x7fadb40673a0 2024-01-15T18:15:08.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.145+0000 7fadd3e00640 1 -- 172.21.15.27:0/4016679306 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7faddc04e850 con 0x7fadb40673a0 2024-01-15T18:15:08.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.149+0000 7fade3200640 1 -- 172.21.15.27:0/4016679306 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fadb40673a0 msgr2=0x7fadb4069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:08.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.149+0000 7fade3200640 1 --2- 172.21.15.27:0/4016679306 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fadb40673a0 0x7fadb4069860 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7faddc0ae970 tx=0x7fadc4009040 comp rx=0 tx=0).stop 2024-01-15T18:15:08.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.149+0000 7fade3200640 1 -- 172.21.15.27:0/4016679306 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faddc06ac00 msgr2=0x7faddc15a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:08.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.149+0000 7fade3200640 1 --2- 172.21.15.27:0/4016679306 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faddc06ac00 0x7faddc15a470 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fadcc0ac000 tx=0x7fadcc01acc0 comp rx=0 tx=0).stop 2024-01-15T18:15:08.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.149+0000 7fade3200640 1 -- 172.21.15.27:0/4016679306 shutdown_connections 2024-01-15T18:15:08.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.149+0000 7fade3200640 1 --2- 172.21.15.27:0/4016679306 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fadb40673a0 0x7fadb4069860 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:08.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.149+0000 7fade3200640 1 --2- 172.21.15.27:0/4016679306 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faddc160cb0 0x7faddc157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:08.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.149+0000 7fade3200640 1 --2- 172.21.15.27:0/4016679306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faddc146a60 0x7faddc0af3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:08.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.149+0000 7fade3200640 1 --2- 172.21.15.27:0/4016679306 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faddc06ac00 0x7faddc15a470 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:08.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.149+0000 7fade3200640 1 -- 172.21.15.27:0/4016679306 >> 172.21.15.27:0/4016679306 conn(0x7faddc0cce10 msgr2=0x7faddc0b49a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:08.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.149+0000 7fade3200640 1 -- 172.21.15.27:0/4016679306 shutdown_connections 2024-01-15T18:15:08.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.149+0000 7fade3200640 1 -- 172.21.15.27:0/4016679306 wait complete. 2024-01-15T18:15:08.171 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone519319 2024-01-15T18:15:08.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.313+0000 7f5940000640 1 Processor -- start 2024-01-15T18:15:08.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.317+0000 7f5940000640 1 -- start start 2024-01-15T18:15:08.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.317+0000 7f5940000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f593815b100 0x7f5938070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:08.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.317+0000 7f5940000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5938071420 0x7f593815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:08.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.317+0000 7f5940000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5938071860 con 0x7f593815ad20 2024-01-15T18:15:08.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.317+0000 7f5940000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f593815a610 con 0x7f5938071420 2024-01-15T18:15:08.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.317+0000 7f5940000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f593815e3c0 con 0x7f593815b100 2024-01-15T18:15:08.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.317+0000 7f593d600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f593815ad20 0x7f593815d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54604/0 (socket says 172.21.15.27:54604) 2024-01-15T18:15:08.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.317+0000 7f593d600640 1 -- 172.21.15.27:0/209322186 learned_addr learned my addr 172.21.15.27:0/209322186 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:08.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.317+0000 7f593e000640 1 --2- 172.21.15.27:0/209322186 >> v2:172.21.15.27:3300/0 conn(0x7f5938071420 0x7f593815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:08.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.317+0000 7f5937e00640 1 -- 172.21.15.27:0/209322186 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121803180 0 0) 0x7f5938071860 con 0x7f593815ad20 2024-01-15T18:15:08.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.317+0000 7f593cc00640 1 --2- 172.21.15.27:0/209322186 >> v2:172.21.15.93:3300/0 conn(0x7f593815b100 0x7f5938070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:08.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.317+0000 7f5937e00640 1 -- 172.21.15.27:0/209322186 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5918003530 con 0x7f593815ad20 2024-01-15T18:15:08.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.317+0000 7f593e000640 1 -- 172.21.15.27:0/209322186 >> v2:172.21.15.93:3300/0 conn(0x7f593815b100 msgr2=0x7f5938070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:08.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.317+0000 7f593e000640 1 --2- 172.21.15.27:0/209322186 >> v2:172.21.15.93:3300/0 conn(0x7f593815b100 0x7f5938070ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:08.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.317+0000 7f593e000640 1 -- 172.21.15.27:0/209322186 >> v1:172.21.15.27:6789/0 conn(0x7f593815ad20 legacy=0x7f593815d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:08.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.317+0000 7f593e000640 1 -- 172.21.15.27:0/209322186 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593815e6a0 con 0x7f5938071420 2024-01-15T18:15:08.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.317+0000 7f593cc00640 1 --2- 172.21.15.27:0/209322186 >> v2:172.21.15.93:3300/0 conn(0x7f593815b100 0x7f5938070ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:08.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.321+0000 7f593e000640 1 --2- 172.21.15.27:0/209322186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5938071420 0x7f593815bee0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f5930009d90 tx=0x7f593001c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97d2b2be34618dd8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:08.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.321+0000 7f5937e00640 1 -- 172.21.15.27:0/209322186 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59300a5910 con 0x7f5938071420 2024-01-15T18:15:08.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.321+0000 7f5937e00640 1 -- 172.21.15.27:0/209322186 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59300a5a70 con 0x7f5938071420 2024-01-15T18:15:08.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.321+0000 7f5937e00640 1 -- 172.21.15.27:0/209322186 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59300a5de0 con 0x7f5938071420 2024-01-15T18:15:08.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.321+0000 7f5940000640 1 -- 172.21.15.27:0/209322186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5938071420 msgr2=0x7f593815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:08.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.321+0000 7f5940000640 1 --2- 172.21.15.27:0/209322186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5938071420 0x7f593815bee0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f5930009d90 tx=0x7f593001c850 comp rx=0 tx=0).stop 2024-01-15T18:15:08.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.321+0000 7f5940000640 1 -- 172.21.15.27:0/209322186 shutdown_connections 2024-01-15T18:15:08.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.321+0000 7f5940000640 1 --2- 172.21.15.27:0/209322186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5938071420 0x7f593815bee0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:08.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.321+0000 7f5940000640 1 --2- 172.21.15.27:0/209322186 >> v2:172.21.15.93:3300/0 conn(0x7f593815b100 0x7f5938070ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:08.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.321+0000 7f5940000640 1 -- 172.21.15.27:0/209322186 >> 172.21.15.27:0/209322186 conn(0x7f59380cb700 msgr2=0x7f59380cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:08.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.321+0000 7f5940000640 1 -- 172.21.15.27:0/209322186 shutdown_connections 2024-01-15T18:15:08.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.321+0000 7f5940000640 1 -- 172.21.15.27:0/209322186 wait complete. 2024-01-15T18:15:08.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f5940000640 1 Processor -- start 2024-01-15T18:15:08.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f5940000640 1 -- start start 2024-01-15T18:15:08.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f5940000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5938071420 0x7f5938157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:08.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f5940000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f593815ad20 0x7f59380c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:08.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f5940000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f593815b100 0x7f593806cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:08.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f5940000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59380cab50 con 0x7f593815ad20 2024-01-15T18:15:08.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f5940000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5938156f20 con 0x7f593815b100 2024-01-15T18:15:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f5940000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59381565c0 con 0x7f5938071420 2024-01-15T18:15:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f593cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f593815ad20 0x7f59380c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f593cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f593815ad20 0x7f59380c9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34154/0 (socket says 172.21.15.27:34154) 2024-01-15T18:15:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f593cc00640 1 -- 172.21.15.27:0/2135996278 learned_addr learned my addr 172.21.15.27:0/2135996278 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f593d600640 1 --2- 172.21.15.27:0/2135996278 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5938071420 0x7f5938157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f593e000640 1 --2- 172.21.15.27:0/2135996278 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f593815b100 0x7f593806cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f593cc00640 1 -- 172.21.15.27:0/2135996278 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5938071420 msgr2=0x7f5938157320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f593cc00640 1 --2- 172.21.15.27:0/2135996278 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5938071420 0x7f5938157320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f593cc00640 1 -- 172.21.15.27:0/2135996278 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f593815b100 msgr2=0x7f593806cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f593cc00640 1 --2- 172.21.15.27:0/2135996278 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f593815b100 0x7f593806cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f593cc00640 1 -- 172.21.15.27:0/2135996278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5930009a70 con 0x7f593815ad20 2024-01-15T18:15:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f593e000640 1 --2- 172.21.15.27:0/2135996278 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f593815b100 0x7f593806cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f593cc00640 1 --2- 172.21.15.27:0/2135996278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f593815ad20 0x7f59380c9fb0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f592000ca60 tx=0x7f592000cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f5935600640 1 -- 172.21.15.27:0/2135996278 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592000ad70 con 0x7f593815ad20 2024-01-15T18:15:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f5935600640 1 -- 172.21.15.27:0/2135996278 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f592000f400 con 0x7f593815ad20 2024-01-15T18:15:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f5940000640 1 -- 172.21.15.27:0/2135996278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59380ca8d0 con 0x7f593815ad20 2024-01-15T18:15:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f5935600640 1 -- 172.21.15.27:0/2135996278 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592000f7b0 con 0x7f593815ad20 2024-01-15T18:15:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f5940000640 1 -- 172.21.15.27:0/2135996278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59380ca4f0 con 0x7f593815ad20 2024-01-15T18:15:08.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.325+0000 7f5940000640 1 -- 172.21.15.27:0/2135996278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f593804e850 con 0x7f593815ad20 2024-01-15T18:15:08.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.329+0000 7f5935600640 1 -- 172.21.15.27:0/2135996278 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 27) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f592000d040 con 0x7f593815ad20 2024-01-15T18:15:08.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.329+0000 7f5935600640 1 --2- 172.21.15.27:0/2135996278 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f59100673a0 0x7f5910069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:08.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.329+0000 7f5935600640 1 -- 172.21.15.27:0/2135996278 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f5920082d20 con 0x7f593815ad20 2024-01-15T18:15:08.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.329+0000 7f593d600640 1 --2- 172.21.15.27:0/2135996278 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f59100673a0 0x7f5910069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:08.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.329+0000 7f593d600640 1 --2- 172.21.15.27:0/2135996278 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f59100673a0 0x7f5910069860 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f5928005890 tx=0x7f5928009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:08.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.333+0000 7f5935600640 1 -- 172.21.15.27:0/2135996278 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f59200519b0 con 0x7f593815ad20 2024-01-15T18:15:08.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.525+0000 7f5940000640 1 -- 172.21.15.27:0/2135996278 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone519319", "target": ["mon-mgr", ""]}) v1 -- 0x7f5938072380 con 0x7f59100673a0 2024-01-15T18:15:08.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.953+0000 7f5935600640 1 -- 172.21.15.27:0/2135996278 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5938072380 con 0x7f59100673a0 2024-01-15T18:15:08.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.957+0000 7f5940000640 1 -- 172.21.15.27:0/2135996278 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f59100673a0 msgr2=0x7f5910069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:08.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.957+0000 7f5940000640 1 --2- 172.21.15.27:0/2135996278 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f59100673a0 0x7f5910069860 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f5928005890 tx=0x7f5928009040 comp rx=0 tx=0).stop 2024-01-15T18:15:08.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.957+0000 7f5940000640 1 -- 172.21.15.27:0/2135996278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f593815ad20 msgr2=0x7f59380c9fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:08.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.957+0000 7f5940000640 1 --2- 172.21.15.27:0/2135996278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f593815ad20 0x7f59380c9fb0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f592000ca60 tx=0x7f592000cf20 comp rx=0 tx=0).stop 2024-01-15T18:15:08.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.957+0000 7f5940000640 1 -- 172.21.15.27:0/2135996278 shutdown_connections 2024-01-15T18:15:08.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.957+0000 7f5940000640 1 --2- 172.21.15.27:0/2135996278 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f59100673a0 0x7f5910069860 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:08.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.957+0000 7f5940000640 1 --2- 172.21.15.27:0/2135996278 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f593815b100 0x7f593806cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:08.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.957+0000 7f5940000640 1 --2- 172.21.15.27:0/2135996278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f593815ad20 0x7f59380c9fb0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:08.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.957+0000 7f5940000640 1 --2- 172.21.15.27:0/2135996278 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5938071420 0x7f5938157320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:08.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.957+0000 7f5940000640 1 -- 172.21.15.27:0/2135996278 >> 172.21.15.27:0/2135996278 conn(0x7f59380cb700 msgr2=0x7f59380cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:08.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.957+0000 7f5940000640 1 -- 172.21.15.27:0/2135996278 shutdown_connections 2024-01-15T18:15:08.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:08.957+0000 7f5940000640 1 -- 172.21.15.27:0/2135996278 wait complete. 2024-01-15T18:15:08.977 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone519320 2024-01-15T18:15:09.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.121+0000 7fb2bbe00640 1 Processor -- start 2024-01-15T18:15:09.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.121+0000 7fb2bbe00640 1 -- start start 2024-01-15T18:15:09.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.121+0000 7fb2bbe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb2b41468d0 0x7fb2b40ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:09.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.121+0000 7fb2bbe00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2b40ad920 con 0x7fb2b40cef30 2024-01-15T18:15:09.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.121+0000 7fb2bbe00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2b40ac880 con 0x7fb2b40cd070 2024-01-15T18:15:09.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.121+0000 7fb2bbe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2b40cd4c0 con 0x7fb2b41468d0 2024-01-15T18:15:09.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.121+0000 7fb2b9400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb2b41468d0 0x7fb2b40ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:09.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.121+0000 7fb2b8a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb2b40cef30 0x7fb2b40cc960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54618/0 (socket says 172.21.15.27:54618) 2024-01-15T18:15:09.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.121+0000 7fb2b8a00640 1 -- 172.21.15.27:0/334060810 learned_addr learned my addr 172.21.15.27:0/334060810 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:09.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.121+0000 7fb2b9400640 1 -- 172.21.15.27:0/334060810 >> v1:172.21.15.93:6789/0 conn(0x7fb2b40cd070 legacy=0x7fb2b406c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:09.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.121+0000 7fb2b9400640 1 -- 172.21.15.27:0/334060810 >> v1:172.21.15.27:6789/0 conn(0x7fb2b40cef30 legacy=0x7fb2b40cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:09.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.121+0000 7fb2b9400640 1 -- 172.21.15.27:0/334060810 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b40acb60 con 0x7fb2b41468d0 2024-01-15T18:15:09.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.125+0000 7fb2b9400640 1 --2- 172.21.15.27:0/334060810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2b41468d0 0x7fb2b40ce9f0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fb2a40097c0 tx=0x7fb2a401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56e2be57a4f36d29 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:09.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.125+0000 7fb2b3e00640 1 -- 172.21.15.27:0/334060810 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a40ab650 con 0x7fb2b41468d0 2024-01-15T18:15:09.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.125+0000 7fb2b3e00640 1 -- 172.21.15.27:0/334060810 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2a40ab7b0 con 0x7fb2b41468d0 2024-01-15T18:15:09.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.125+0000 7fb2b3e00640 1 -- 172.21.15.27:0/334060810 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a40abb40 con 0x7fb2b41468d0 2024-01-15T18:15:09.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.125+0000 7fb2bbe00640 1 -- 172.21.15.27:0/334060810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2b41468d0 msgr2=0x7fb2b40ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:09.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.125+0000 7fb2bbe00640 1 --2- 172.21.15.27:0/334060810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2b41468d0 0x7fb2b40ce9f0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fb2a40097c0 tx=0x7fb2a401c450 comp rx=0 tx=0).stop 2024-01-15T18:15:09.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.125+0000 7fb2bbe00640 1 -- 172.21.15.27:0/334060810 shutdown_connections 2024-01-15T18:15:09.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.125+0000 7fb2bbe00640 1 --2- 172.21.15.27:0/334060810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2b41468d0 0x7fb2b40ce9f0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:09.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.125+0000 7fb2bbe00640 1 -- 172.21.15.27:0/334060810 >> 172.21.15.27:0/334060810 conn(0x7fb2b40cb700 msgr2=0x7fb2b40cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:09.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.125+0000 7fb2bbe00640 1 -- 172.21.15.27:0/334060810 shutdown_connections 2024-01-15T18:15:09.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.125+0000 7fb2bbe00640 1 -- 172.21.15.27:0/334060810 wait complete. 2024-01-15T18:15:09.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.129+0000 7fb2bbe00640 1 Processor -- start 2024-01-15T18:15:09.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.129+0000 7fb2bbe00640 1 -- start start 2024-01-15T18:15:09.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.129+0000 7fb2bbe00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb2b40cd070 0x7fb2b41f70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:09.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.129+0000 7fb2bbe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb2b40cef30 0x7fb2b415ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:09.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.129+0000 7fb2bbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2b41468d0 0x7fb2b40b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:09.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.129+0000 7fb2bbe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b40b3c60 con 0x7fb2b41468d0 2024-01-15T18:15:09.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.129+0000 7fb2bbe00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b41f65f0 con 0x7fb2b40cd070 2024-01-15T18:15:09.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.129+0000 7fb2bbe00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b41f68d0 con 0x7fb2b40cef30 2024-01-15T18:15:09.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.129+0000 7fb2b9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2b41468d0 0x7fb2b40b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:09.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.129+0000 7fb2b9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2b41468d0 0x7fb2b40b31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34164/0 (socket says 172.21.15.27:34164) 2024-01-15T18:15:09.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.129+0000 7fb2b9e00640 1 -- 172.21.15.27:0/445678815 learned_addr learned my addr 172.21.15.27:0/445678815 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:09.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.129+0000 7fb2b9400640 1 --2- 172.21.15.27:0/445678815 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb2b40cd070 0x7fb2b41f70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:09.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.129+0000 7fb2b8a00640 1 --2- 172.21.15.27:0/445678815 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb2b40cef30 0x7fb2b415ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:09.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.129+0000 7fb2b9e00640 1 -- 172.21.15.27:0/445678815 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb2b40cef30 msgr2=0x7fb2b415ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:09.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.129+0000 7fb2b9e00640 1 --2- 172.21.15.27:0/445678815 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb2b40cef30 0x7fb2b415ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:09.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.129+0000 7fb2b9e00640 1 -- 172.21.15.27:0/445678815 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb2b40cd070 msgr2=0x7fb2b41f70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:09.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.129+0000 7fb2b9e00640 1 --2- 172.21.15.27:0/445678815 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb2b40cd070 0x7fb2b41f70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:09.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.129+0000 7fb2b9e00640 1 -- 172.21.15.27:0/445678815 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a4009420 con 0x7fb2b41468d0 2024-01-15T18:15:09.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.129+0000 7fb2b9e00640 1 --2- 172.21.15.27:0/445678815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2b41468d0 0x7fb2b40b31a0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac00c770 tx=0x7fb2ac00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:09.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.129+0000 7fb2b1600640 1 -- 172.21.15.27:0/445678815 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac003b80 con 0x7fb2b41468d0 2024-01-15T18:15:09.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.129+0000 7fb2b1600640 1 -- 172.21.15.27:0/445678815 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2ac003ce0 con 0x7fb2b41468d0 2024-01-15T18:15:09.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.129+0000 7fb2b1600640 1 -- 172.21.15.27:0/445678815 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac00aa90 con 0x7fb2b41468d0 2024-01-15T18:15:09.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.129+0000 7fb2bbe00640 1 -- 172.21.15.27:0/445678815 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2b40cf490 con 0x7fb2b41468d0 2024-01-15T18:15:09.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.129+0000 7fb2bbe00640 1 -- 172.21.15.27:0/445678815 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2b41f62d0 con 0x7fb2b41468d0 2024-01-15T18:15:09.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.129+0000 7fb2bbe00640 1 -- 172.21.15.27:0/445678815 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2b415b0a0 con 0x7fb2b41468d0 2024-01-15T18:15:09.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.133+0000 7fb2b1600640 1 -- 172.21.15.27:0/445678815 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 27) v1 ==== 87129+0+0 (secure 0 0 0) 0x7fb2ac00abf0 con 0x7fb2b41468d0 2024-01-15T18:15:09.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.133+0000 7fb2b1600640 1 --2- 172.21.15.27:0/445678815 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb28c0673a0 0x7fb28c069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:09.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.133+0000 7fb2b1600640 1 -- 172.21.15.27:0/445678815 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fb2ac082aa0 con 0x7fb2b41468d0 2024-01-15T18:15:09.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.133+0000 7fb2b9400640 1 --2- 172.21.15.27:0/445678815 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb28c0673a0 0x7fb28c069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:09.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.133+0000 7fb2b9400640 1 --2- 172.21.15.27:0/445678815 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb28c0673a0 0x7fb28c069860 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fb2a40af000 tx=0x7fb2a40a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:09.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.137+0000 7fb2b1600640 1 -- 172.21.15.27:0/445678815 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb2ac0517e0 con 0x7fb2b41468d0 2024-01-15T18:15:09.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.329+0000 7fb2bbe00640 1 -- 172.21.15.27:0/445678815 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone519320", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2b40cf310 con 0x7fb28c0673a0 2024-01-15T18:15:09.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.761+0000 7fb2b1600640 1 -- 172.21.15.27:0/445678815 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb2a40ab9a0 con 0x7fb28c0673a0 2024-01-15T18:15:09.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.761+0000 7fb2bbe00640 1 -- 172.21.15.27:0/445678815 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb28c0673a0 msgr2=0x7fb28c069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:09.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.761+0000 7fb2bbe00640 1 --2- 172.21.15.27:0/445678815 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb28c0673a0 0x7fb28c069860 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fb2a40af000 tx=0x7fb2a40a7000 comp rx=0 tx=0).stop 2024-01-15T18:15:09.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.761+0000 7fb2bbe00640 1 -- 172.21.15.27:0/445678815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2b41468d0 msgr2=0x7fb2b40b31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:09.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.761+0000 7fb2bbe00640 1 --2- 172.21.15.27:0/445678815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2b41468d0 0x7fb2b40b31a0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac00c770 tx=0x7fb2ac00cc30 comp rx=0 tx=0).stop 2024-01-15T18:15:09.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.761+0000 7fb2bbe00640 1 -- 172.21.15.27:0/445678815 shutdown_connections 2024-01-15T18:15:09.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.761+0000 7fb2bbe00640 1 --2- 172.21.15.27:0/445678815 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb28c0673a0 0x7fb28c069860 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:09.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.761+0000 7fb2bbe00640 1 --2- 172.21.15.27:0/445678815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2b41468d0 0x7fb2b40b31a0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:09.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.761+0000 7fb2bbe00640 1 --2- 172.21.15.27:0/445678815 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb2b40cef30 0x7fb2b415ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:09.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.761+0000 7fb2bbe00640 1 --2- 172.21.15.27:0/445678815 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb2b40cd070 0x7fb2b41f70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:09.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.761+0000 7fb2bbe00640 1 -- 172.21.15.27:0/445678815 >> 172.21.15.27:0/445678815 conn(0x7fb2b40cb700 msgr2=0x7fb2b415bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:09.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.761+0000 7fb2bbe00640 1 -- 172.21.15.27:0/445678815 shutdown_connections 2024-01-15T18:15:09.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.761+0000 7fb2bbe00640 1 -- 172.21.15.27:0/445678815 wait complete. 2024-01-15T18:15:09.783 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:15:09.784 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone51931 2024-01-15T18:15:09.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.925+0000 7f4fa6e00640 1 Processor -- start 2024-01-15T18:15:09.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.925+0000 7f4fa6e00640 1 -- start start 2024-01-15T18:15:09.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.929+0000 7f4fa6e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4fa015f550 0x7f4fa00cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:09.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.929+0000 7f4fa6e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4fa00cf110 0x7f4fa00701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:09.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.929+0000 7f4fa6e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fa00b1cb0 con 0x7f4fa0158900 2024-01-15T18:15:09.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.929+0000 7f4fa6e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fa015fa00 con 0x7f4fa00cf110 2024-01-15T18:15:09.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.929+0000 7f4fa6e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa00cf540 con 0x7f4fa015f550 2024-01-15T18:15:09.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.929+0000 7f4fa4e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4fa00cf110 0x7f4fa00701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:09.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.929+0000 7f4fa4e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4fa00cf110 0x7f4fa00701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34166/0 (socket says 172.21.15.27:34166) 2024-01-15T18:15:09.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.929+0000 7f4fa4e00640 1 -- 172.21.15.27:0/2107762327 learned_addr learned my addr 172.21.15.27:0/2107762327 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:09.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.929+0000 7f4f9f400640 1 --2- 172.21.15.27:0/2107762327 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4fa015f550 0x7f4fa00cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:09.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.929+0000 7f4fa4e00640 1 -- 172.21.15.27:0/2107762327 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4fa015f550 msgr2=0x7f4fa00cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:09.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.929+0000 7f4fa4e00640 1 --2- 172.21.15.27:0/2107762327 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4fa015f550 0x7f4fa00cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:09.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.929+0000 7f4fa4e00640 1 -- 172.21.15.27:0/2107762327 >> v1:172.21.15.93:6789/0 conn(0x7f4fa0158900 legacy=0x7f4fa015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:09.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.929+0000 7f4fa4e00640 1 -- 172.21.15.27:0/2107762327 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa00b1f00 con 0x7f4fa00cf110 2024-01-15T18:15:09.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.933+0000 7f4fa4e00640 1 --2- 172.21.15.27:0/2107762327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fa00cf110 0x7f4fa00701e0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f4f98009e50 tx=0x7f4f9801c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e3af633e90827b1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:09.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.933+0000 7f4f9ea00640 1 -- 172.21.15.27:0/2107762327 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa0070f30 con 0x7f4fa00cf110 2024-01-15T18:15:09.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.933+0000 7f4f9ea00640 1 -- 172.21.15.27:0/2107762327 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f980a58a0 con 0x7f4fa00cf110 2024-01-15T18:15:09.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.933+0000 7f4f9ea00640 1 -- 172.21.15.27:0/2107762327 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f980a5c30 con 0x7f4fa00cf110 2024-01-15T18:15:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.933+0000 7f4fa6e00640 1 -- 172.21.15.27:0/2107762327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fa00cf110 msgr2=0x7f4fa00701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.933+0000 7f4fa6e00640 1 --2- 172.21.15.27:0/2107762327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fa00cf110 0x7f4fa00701e0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f4f98009e50 tx=0x7f4f9801c9e0 comp rx=0 tx=0).stop 2024-01-15T18:15:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.933+0000 7f4fa6e00640 1 -- 172.21.15.27:0/2107762327 shutdown_connections 2024-01-15T18:15:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.933+0000 7f4fa6e00640 1 --2- 172.21.15.27:0/2107762327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fa00cf110 0x7f4fa00701e0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.933+0000 7f4fa6e00640 1 --2- 172.21.15.27:0/2107762327 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4fa015f550 0x7f4fa00cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:09.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.933+0000 7f4fa6e00640 1 -- 172.21.15.27:0/2107762327 >> 172.21.15.27:0/2107762327 conn(0x7f4fa00cb700 msgr2=0x7f4fa00cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:09.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.933+0000 7f4fa6e00640 1 -- 172.21.15.27:0/2107762327 shutdown_connections 2024-01-15T18:15:09.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.933+0000 7f4fa6e00640 1 -- 172.21.15.27:0/2107762327 wait complete. 2024-01-15T18:15:09.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.933+0000 7f4fa6e00640 1 Processor -- start 2024-01-15T18:15:09.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.933+0000 7f4fa6e00640 1 -- start start 2024-01-15T18:15:09.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.937+0000 7f4fa6e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fa00cf110 0x7f4fa0159be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:09.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.937+0000 7f4fa6e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4fa0158900 0x7f4fa0140540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:09.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.937+0000 7f4fa6e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4fa015f550 0x7f4fa00b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:09.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.937+0000 7f4fa6e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa015b140 con 0x7f4fa00cf110 2024-01-15T18:15:09.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.937+0000 7f4fa6e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa015a4c0 con 0x7f4fa0158900 2024-01-15T18:15:09.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.937+0000 7f4fa6e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa015a7a0 con 0x7f4fa015f550 2024-01-15T18:15:09.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.937+0000 7f4f9fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fa00cf110 0x7f4fa0159be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:09.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.937+0000 7f4fa4e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4fa015f550 0x7f4fa00b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:09.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.937+0000 7f4f9fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fa00cf110 0x7f4fa0159be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34178/0 (socket says 172.21.15.27:34178) 2024-01-15T18:15:09.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.937+0000 7f4f9fe00640 1 -- 172.21.15.27:0/4158756744 learned_addr learned my addr 172.21.15.27:0/4158756744 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:09.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.937+0000 7f4f9f400640 1 --2- 172.21.15.27:0/4158756744 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4fa0158900 0x7f4fa0140540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:09.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.937+0000 7f4f9fe00640 1 -- 172.21.15.27:0/4158756744 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4fa015f550 msgr2=0x7f4fa00b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:09.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.937+0000 7f4f9fe00640 1 --2- 172.21.15.27:0/4158756744 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4fa015f550 0x7f4fa00b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:09.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.937+0000 7f4f9fe00640 1 -- 172.21.15.27:0/4158756744 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4fa0158900 msgr2=0x7f4fa0140540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:09.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.937+0000 7f4f9fe00640 1 --2- 172.21.15.27:0/4158756744 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4fa0158900 0x7f4fa0140540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:09.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.937+0000 7f4f9fe00640 1 -- 172.21.15.27:0/4158756744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f98009b30 con 0x7f4fa00cf110 2024-01-15T18:15:09.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.937+0000 7f4f9fe00640 1 --2- 172.21.15.27:0/4158756744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fa00cf110 0x7f4fa0159be0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f4f9000c500 tx=0x7f4f9000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.937+0000 7f4f97e00640 1 -- 172.21.15.27:0/4158756744 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f90005220 con 0x7f4fa00cf110 2024-01-15T18:15:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.937+0000 7f4f97e00640 1 -- 172.21.15.27:0/4158756744 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f90005380 con 0x7f4fa00cf110 2024-01-15T18:15:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.937+0000 7f4fa6e00640 1 -- 172.21.15.27:0/4158756744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fa015aa80 con 0x7f4fa00cf110 2024-01-15T18:15:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.937+0000 7f4f97e00640 1 -- 172.21.15.27:0/4158756744 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f90007270 con 0x7f4fa00cf110 2024-01-15T18:15:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.937+0000 7f4fa6e00640 1 -- 172.21.15.27:0/4158756744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fa015a120 con 0x7f4fa00cf110 2024-01-15T18:15:09.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.937+0000 7f4fa6e00640 1 -- 172.21.15.27:0/4158756744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fa015b280 con 0x7f4fa00cf110 2024-01-15T18:15:09.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.941+0000 7f4f97e00640 1 -- 172.21.15.27:0/4158756744 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 27) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f4f900073d0 con 0x7f4fa00cf110 2024-01-15T18:15:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.941+0000 7f4f97e00640 1 --2- 172.21.15.27:0/4158756744 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4f780673a0 0x7f4f78069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.941+0000 7f4f97e00640 1 -- 172.21.15.27:0/4158756744 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f4f90082630 con 0x7f4fa00cf110 2024-01-15T18:15:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.941+0000 7f4f9f400640 1 --2- 172.21.15.27:0/4158756744 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4f780673a0 0x7f4f78069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:09.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.941+0000 7f4f9f400640 1 --2- 172.21.15.27:0/4158756744 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4f780673a0 0x7f4f78069860 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f4fa01512a0 tx=0x7f4f88009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:09.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:09.945+0000 7f4f97e00640 1 -- 172.21.15.27:0/4158756744 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4fa015b280 con 0x7f4fa00cf110 2024-01-15T18:15:10.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.137+0000 7f4fa6e00640 1 -- 172.21.15.27:0/4158756744 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone51931", "target": ["mon-mgr", ""]}) v1 -- 0x7f4fa006ced0 con 0x7f4f780673a0 2024-01-15T18:15:10.144 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:15:10.141+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:10.144 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:15:10.141+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:10.144 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:15:10.141+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:10.144 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:15:10.141+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:10.144 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:15:10.141+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:10.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.161+0000 7f4f97e00640 1 -- 172.21.15.27:0/4158756744 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f4fa006ced0 con 0x7f4f780673a0 2024-01-15T18:15:10.161 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:15:10.161 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:15:10.161 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "complete" 2024-01-15T18:15:10.161 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:10.161 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:15:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.161+0000 7f4fa6e00640 1 -- 172.21.15.27:0/4158756744 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4f780673a0 msgr2=0x7f4f78069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.161+0000 7f4fa6e00640 1 --2- 172.21.15.27:0/4158756744 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4f780673a0 0x7f4f78069860 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f4fa01512a0 tx=0x7f4f88009190 comp rx=0 tx=0).stop 2024-01-15T18:15:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.161+0000 7f4fa6e00640 1 -- 172.21.15.27:0/4158756744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fa00cf110 msgr2=0x7f4fa0159be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.161+0000 7f4fa6e00640 1 --2- 172.21.15.27:0/4158756744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fa00cf110 0x7f4fa0159be0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f4f9000c500 tx=0x7f4f9000c9c0 comp rx=0 tx=0).stop 2024-01-15T18:15:10.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.161+0000 7f4fa6e00640 1 -- 172.21.15.27:0/4158756744 shutdown_connections 2024-01-15T18:15:10.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.161+0000 7f4fa6e00640 1 --2- 172.21.15.27:0/4158756744 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4f780673a0 0x7f4f78069860 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:10.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.161+0000 7f4fa6e00640 1 --2- 172.21.15.27:0/4158756744 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4fa015f550 0x7f4fa00b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:10.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.161+0000 7f4fa6e00640 1 --2- 172.21.15.27:0/4158756744 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4fa0158900 0x7f4fa0140540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:10.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.161+0000 7f4fa6e00640 1 --2- 172.21.15.27:0/4158756744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fa00cf110 0x7f4fa0159be0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:10.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.161+0000 7f4fa6e00640 1 -- 172.21.15.27:0/4158756744 >> 172.21.15.27:0/4158756744 conn(0x7f4fa00cb700 msgr2=0x7f4fa00cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:10.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.161+0000 7f4fa6e00640 1 -- 172.21.15.27:0/4158756744 shutdown_connections 2024-01-15T18:15:10.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.161+0000 7f4fa6e00640 1 -- 172.21.15.27:0/4158756744 wait complete. 2024-01-15T18:15:10.183 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:15:10.183 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone51932 2024-01-15T18:15:10.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.329+0000 7f193a200640 1 Processor -- start 2024-01-15T18:15:10.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.329+0000 7f193a200640 1 -- start start 2024-01-15T18:15:10.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.329+0000 7f193a200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f193415d190 0x7f19340cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:10.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.329+0000 7f193a200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f193415d570 0x7f193406be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:10.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.329+0000 7f193a200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19340b0d70 con 0x7f193406b810 2024-01-15T18:15:10.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.329+0000 7f193a200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19340d0540 con 0x7f193415d190 2024-01-15T18:15:10.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.329+0000 7f193a200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19340b0440 con 0x7f193415d570 2024-01-15T18:15:10.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.329+0000 7f1933400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f193415d190 0x7f19340cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:10.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.329+0000 7f1933400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f193415d190 0x7f19340cfcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34192/0 (socket says 172.21.15.27:34192) 2024-01-15T18:15:10.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.329+0000 7f1933400640 1 -- 172.21.15.27:0/273362641 learned_addr learned my addr 172.21.15.27:0/273362641 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:10.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.329+0000 7f1933400640 1 -- 172.21.15.27:0/273362641 >> v2:172.21.15.93:3300/0 conn(0x7f193415d570 msgr2=0x7f193406be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:10.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.329+0000 7f1932a00640 1 --2- 172.21.15.27:0/273362641 >> v2:172.21.15.93:3300/0 conn(0x7f193415d570 0x7f193406be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:10.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.329+0000 7f1933400640 1 --2- 172.21.15.27:0/273362641 >> v2:172.21.15.93:3300/0 conn(0x7f193415d570 0x7f193406be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:10.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.329+0000 7f1933400640 1 -- 172.21.15.27:0/273362641 >> v1:172.21.15.93:6789/0 conn(0x7f193406b810 legacy=0x7f19340cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:10.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.329+0000 7f1933400640 1 -- 172.21.15.27:0/273362641 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19340b0720 con 0x7f193415d190 2024-01-15T18:15:10.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.329+0000 7f1932a00640 1 --2- 172.21.15.27:0/273362641 >> v2:172.21.15.93:3300/0 conn(0x7f193415d570 0x7f193406be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:10.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.333+0000 7f1933400640 1 --2- 172.21.15.27:0/273362641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f193415d190 0x7f19340cfcf0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f19340713a0 tx=0x7f192401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37da783870ac666c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:10.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.333+0000 7f1932000640 1 -- 172.21.15.27:0/273362641 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19240ac600 con 0x7f193415d190 2024-01-15T18:15:10.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.333+0000 7f1932000640 1 -- 172.21.15.27:0/273362641 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19240a8050 con 0x7f193415d190 2024-01-15T18:15:10.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.333+0000 7f1932000640 1 -- 172.21.15.27:0/273362641 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19240ac980 con 0x7f193415d190 2024-01-15T18:15:10.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.333+0000 7f193a200640 1 -- 172.21.15.27:0/273362641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f193415d190 msgr2=0x7f19340cfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:10.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.333+0000 7f193a200640 1 --2- 172.21.15.27:0/273362641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f193415d190 0x7f19340cfcf0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f19340713a0 tx=0x7f192401c490 comp rx=0 tx=0).stop 2024-01-15T18:15:10.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.333+0000 7f193a200640 1 -- 172.21.15.27:0/273362641 shutdown_connections 2024-01-15T18:15:10.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.333+0000 7f193a200640 1 --2- 172.21.15.27:0/273362641 >> v2:172.21.15.93:3300/0 conn(0x7f193415d570 0x7f193406be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:10.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.333+0000 7f193a200640 1 --2- 172.21.15.27:0/273362641 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f193415d190 0x7f19340cfcf0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:10.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.333+0000 7f193a200640 1 -- 172.21.15.27:0/273362641 >> 172.21.15.27:0/273362641 conn(0x7f19340d0e70 msgr2=0x7f19340d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:10.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.333+0000 7f193a200640 1 -- 172.21.15.27:0/273362641 shutdown_connections 2024-01-15T18:15:10.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.333+0000 7f193a200640 1 -- 172.21.15.27:0/273362641 wait complete. 2024-01-15T18:15:10.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.337+0000 7f193a200640 1 Processor -- start 2024-01-15T18:15:10.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.337+0000 7f193a200640 1 -- start start 2024-01-15T18:15:10.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.337+0000 7f193a200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f193406b810 0x7f19341536b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:10.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.337+0000 7f193a200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f193415d190 0x7f19340c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:10.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.337+0000 7f193a200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f193415d570 0x7f1934153d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:10.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.337+0000 7f193a200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1934153bf0 con 0x7f193415d190 2024-01-15T18:15:10.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.337+0000 7f193a200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19340c5800 con 0x7f193415d570 2024-01-15T18:15:10.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.337+0000 7f193a200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19340c5240 con 0x7f193406b810 2024-01-15T18:15:10.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.337+0000 7f1932a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f193415d190 0x7f19340c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:10.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.337+0000 7f1932a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f193415d190 0x7f19340c48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34194/0 (socket says 172.21.15.27:34194) 2024-01-15T18:15:10.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.337+0000 7f1932a00640 1 -- 172.21.15.27:0/3545202568 learned_addr learned my addr 172.21.15.27:0/3545202568 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:10.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.337+0000 7f1933400640 1 --2- 172.21.15.27:0/3545202568 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f193406b810 0x7f19341536b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:10.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.337+0000 7f1933e00640 1 --2- 172.21.15.27:0/3545202568 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f193415d570 0x7f1934153d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.337+0000 7f1932a00640 1 -- 172.21.15.27:0/3545202568 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f193406b810 msgr2=0x7f19341536b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.337+0000 7f1932a00640 1 --2- 172.21.15.27:0/3545202568 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f193406b810 0x7f19341536b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.337+0000 7f1932a00640 1 -- 172.21.15.27:0/3545202568 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f193415d570 msgr2=0x7f1934153d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.337+0000 7f1932a00640 1 --2- 172.21.15.27:0/3545202568 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f193415d570 0x7f1934153d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.337+0000 7f1932a00640 1 -- 172.21.15.27:0/3545202568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19240094d0 con 0x7f193415d190 2024-01-15T18:15:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.337+0000 7f1932a00640 1 --2- 172.21.15.27:0/3545202568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f193415d190 0x7f19340c48d0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f191c00c7f0 tx=0x7f191c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:10.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.337+0000 7f192b400640 1 -- 172.21.15.27:0/3545202568 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f191c007820 con 0x7f193415d190 2024-01-15T18:15:10.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.337+0000 7f192b400640 1 -- 172.21.15.27:0/3545202568 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f191c007980 con 0x7f193415d190 2024-01-15T18:15:10.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.337+0000 7f192b400640 1 -- 172.21.15.27:0/3545202568 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f191c007d30 con 0x7f193415d190 2024-01-15T18:15:10.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.337+0000 7f193a200640 1 -- 172.21.15.27:0/3545202568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19340c5520 con 0x7f193415d190 2024-01-15T18:15:10.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.337+0000 7f193a200640 1 -- 172.21.15.27:0/3545202568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19340c66e0 con 0x7f193415d190 2024-01-15T18:15:10.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.341+0000 7f193a200640 1 -- 172.21.15.27:0/3545202568 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f193404e850 con 0x7f193415d190 2024-01-15T18:15:10.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.341+0000 7f192b400640 1 -- 172.21.15.27:0/3545202568 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 27) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f191c0113f0 con 0x7f193415d190 2024-01-15T18:15:10.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.341+0000 7f192b400640 1 --2- 172.21.15.27:0/3545202568 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f190c0673a0 0x7f190c069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:10.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.341+0000 7f192b400640 1 -- 172.21.15.27:0/3545202568 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f191c083060 con 0x7f193415d190 2024-01-15T18:15:10.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.341+0000 7f1933400640 1 --2- 172.21.15.27:0/3545202568 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f190c0673a0 0x7f190c069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:10.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.341+0000 7f1933400640 1 --2- 172.21.15.27:0/3545202568 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f190c0673a0 0x7f190c069860 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f19240ae000 tx=0x7f19240a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:10.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.345+0000 7f192b400640 1 -- 172.21.15.27:0/3545202568 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f191c051da0 con 0x7f193415d190 2024-01-15T18:15:10.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.537+0000 7f193a200640 1 -- 172.21.15.27:0/3545202568 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone51932", "target": ["mon-mgr", ""]}) v1 -- 0x7f19340ce420 con 0x7f190c0673a0 2024-01-15T18:15:10.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.549+0000 7f192b400640 1 -- 172.21.15.27:0/3545202568 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f19340ce420 con 0x7f190c0673a0 2024-01-15T18:15:10.552 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:15:10.552 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:15:10.552 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "complete" 2024-01-15T18:15:10.553 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:10.553 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:15:10.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.553+0000 7f193a200640 1 -- 172.21.15.27:0/3545202568 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f190c0673a0 msgr2=0x7f190c069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:10.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.553+0000 7f193a200640 1 --2- 172.21.15.27:0/3545202568 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f190c0673a0 0x7f190c069860 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f19240ae000 tx=0x7f19240a7000 comp rx=0 tx=0).stop 2024-01-15T18:15:10.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.553+0000 7f193a200640 1 -- 172.21.15.27:0/3545202568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f193415d190 msgr2=0x7f19340c48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:10.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.553+0000 7f193a200640 1 --2- 172.21.15.27:0/3545202568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f193415d190 0x7f19340c48d0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f191c00c7f0 tx=0x7f191c00ccb0 comp rx=0 tx=0).stop 2024-01-15T18:15:10.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.553+0000 7f193a200640 1 -- 172.21.15.27:0/3545202568 shutdown_connections 2024-01-15T18:15:10.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.553+0000 7f193a200640 1 --2- 172.21.15.27:0/3545202568 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f190c0673a0 0x7f190c069860 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:10.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.553+0000 7f193a200640 1 --2- 172.21.15.27:0/3545202568 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f193415d570 0x7f1934153d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:10.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.553+0000 7f193a200640 1 --2- 172.21.15.27:0/3545202568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f193415d190 0x7f19340c48d0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:10.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.553+0000 7f193a200640 1 --2- 172.21.15.27:0/3545202568 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f193406b810 0x7f19341536b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:10.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.553+0000 7f193a200640 1 -- 172.21.15.27:0/3545202568 >> 172.21.15.27:0/3545202568 conn(0x7f19340d0e70 msgr2=0x7f19340aec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:10.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.553+0000 7f193a200640 1 -- 172.21.15.27:0/3545202568 shutdown_connections 2024-01-15T18:15:10.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.553+0000 7f193a200640 1 -- 172.21.15.27:0/3545202568 wait complete. 2024-01-15T18:15:10.574 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:15:10.574 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone51933 2024-01-15T18:15:10.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.717+0000 7fe267400640 1 Processor -- start 2024-01-15T18:15:10.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.717+0000 7fe267400640 1 -- start start 2024-01-15T18:15:10.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.717+0000 7fe267400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe260158900 0x7fe26015f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:10.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.717+0000 7fe267400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe2600cf2e0 0x7fe2600701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:10.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.717+0000 7fe267400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2600b1cb0 con 0x7fe26015bf00 2024-01-15T18:15:10.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.717+0000 7fe267400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2600b1f00 con 0x7fe2600cf2e0 2024-01-15T18:15:10.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.717+0000 7fe267400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2600b21e0 con 0x7fe260158900 2024-01-15T18:15:10.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.717+0000 7fe265400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe2600cf2e0 0x7fe2600701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:10.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.717+0000 7fe265400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe2600cf2e0 0x7fe2600701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34200/0 (socket says 172.21.15.27:34200) 2024-01-15T18:15:10.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.717+0000 7fe265400640 1 -- 172.21.15.27:0/3906585011 learned_addr learned my addr 172.21.15.27:0/3906585011 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:10.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.717+0000 7fe264a00640 1 --2- 172.21.15.27:0/3906585011 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe260158900 0x7fe26015f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:10.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.721+0000 7fe265400640 1 -- 172.21.15.27:0/3906585011 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe260158900 msgr2=0x7fe26015f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:10.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.721+0000 7fe265400640 1 --2- 172.21.15.27:0/3906585011 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe260158900 0x7fe26015f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:10.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.721+0000 7fe265400640 1 -- 172.21.15.27:0/3906585011 >> v1:172.21.15.93:6789/0 conn(0x7fe26015bf00 legacy=0x7fe2600cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:10.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.721+0000 7fe265400640 1 -- 172.21.15.27:0/3906585011 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe26015ea70 con 0x7fe2600cf2e0 2024-01-15T18:15:10.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.721+0000 7fe264a00640 1 --2- 172.21.15.27:0/3906585011 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe260158900 0x7fe26015f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:10.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.721+0000 7fe265400640 1 --2- 172.21.15.27:0/3906585011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2600cf2e0 0x7fe2600701e0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fe258009e50 tx=0x7fe25801ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f7b6ab1a14fbc0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:10.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.721+0000 7fe25f400640 1 -- 172.21.15.27:0/3906585011 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe260070f30 con 0x7fe2600cf2e0 2024-01-15T18:15:10.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.721+0000 7fe25f400640 1 -- 172.21.15.27:0/3906585011 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2580a57d0 con 0x7fe2600cf2e0 2024-01-15T18:15:10.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.721+0000 7fe25f400640 1 -- 172.21.15.27:0/3906585011 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2580a5b60 con 0x7fe2600cf2e0 2024-01-15T18:15:10.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.725+0000 7fe267400640 1 -- 172.21.15.27:0/3906585011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2600cf2e0 msgr2=0x7fe2600701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:10.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.725+0000 7fe267400640 1 --2- 172.21.15.27:0/3906585011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2600cf2e0 0x7fe2600701e0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fe258009e50 tx=0x7fe25801ca10 comp rx=0 tx=0).stop 2024-01-15T18:15:10.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.725+0000 7fe267400640 1 -- 172.21.15.27:0/3906585011 shutdown_connections 2024-01-15T18:15:10.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.725+0000 7fe267400640 1 --2- 172.21.15.27:0/3906585011 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2600cf2e0 0x7fe2600701e0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:10.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.725+0000 7fe267400640 1 --2- 172.21.15.27:0/3906585011 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe260158900 0x7fe26015f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:10.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.725+0000 7fe267400640 1 -- 172.21.15.27:0/3906585011 >> 172.21.15.27:0/3906585011 conn(0x7fe2600cb700 msgr2=0x7fe2600cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:10.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.725+0000 7fe267400640 1 -- 172.21.15.27:0/3906585011 shutdown_connections 2024-01-15T18:15:10.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.725+0000 7fe267400640 1 -- 172.21.15.27:0/3906585011 wait complete. 2024-01-15T18:15:10.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.725+0000 7fe267400640 1 Processor -- start 2024-01-15T18:15:10.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.725+0000 7fe267400640 1 -- start start 2024-01-15T18:15:10.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.725+0000 7fe267400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe2600cf2e0 0x7fe260150b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:10.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.725+0000 7fe267400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe260158900 0x7fe2601f8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:10.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.725+0000 7fe267400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe26015bf00 0x7fe2601f9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:10.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.725+0000 7fe267400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2601f9d80 con 0x7fe260158900 2024-01-15T18:15:10.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.725+0000 7fe267400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2600cde00 con 0x7fe26015bf00 2024-01-15T18:15:10.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.725+0000 7fe267400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2600ce0e0 con 0x7fe2600cf2e0 2024-01-15T18:15:10.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.725+0000 7fe25fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe260158900 0x7fe2601f8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:10.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.725+0000 7fe264a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe2600cf2e0 0x7fe260150b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:10.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.725+0000 7fe265400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe26015bf00 0x7fe2601f9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:10.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.725+0000 7fe25fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe260158900 0x7fe2601f8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34214/0 (socket says 172.21.15.27:34214) 2024-01-15T18:15:10.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.725+0000 7fe25fe00640 1 -- 172.21.15.27:0/1640936948 learned_addr learned my addr 172.21.15.27:0/1640936948 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:10.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.729+0000 7fe25fe00640 1 -- 172.21.15.27:0/1640936948 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe2600cf2e0 msgr2=0x7fe260150b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:10.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.729+0000 7fe25fe00640 1 --2- 172.21.15.27:0/1640936948 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe2600cf2e0 0x7fe260150b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:10.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.729+0000 7fe25fe00640 1 -- 172.21.15.27:0/1640936948 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe26015bf00 msgr2=0x7fe2601f9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:10.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.729+0000 7fe25fe00640 1 --2- 172.21.15.27:0/1640936948 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe26015bf00 0x7fe2601f9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:10.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.729+0000 7fe25fe00640 1 -- 172.21.15.27:0/1640936948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe258009b30 con 0x7fe260158900 2024-01-15T18:15:10.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.729+0000 7fe265400640 1 --2- 172.21.15.27:0/1640936948 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe26015bf00 0x7fe2601f9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:15:10.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.729+0000 7fe25fe00640 1 --2- 172.21.15.27:0/1640936948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe260158900 0x7fe2601f8f10 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fe24800c500 tx=0x7fe24800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:10.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.729+0000 7fe25cc00640 1 -- 172.21.15.27:0/1640936948 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe248005220 con 0x7fe260158900 2024-01-15T18:15:10.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.729+0000 7fe25cc00640 1 -- 172.21.15.27:0/1640936948 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe248005380 con 0x7fe260158900 2024-01-15T18:15:10.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.729+0000 7fe267400640 1 -- 172.21.15.27:0/1640936948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2601591c0 con 0x7fe260158900 2024-01-15T18:15:10.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.729+0000 7fe25cc00640 1 -- 172.21.15.27:0/1640936948 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe248007270 con 0x7fe260158900 2024-01-15T18:15:10.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.729+0000 7fe267400640 1 -- 172.21.15.27:0/1640936948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe260159780 con 0x7fe260158900 2024-01-15T18:15:10.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.729+0000 7fe267400640 1 -- 172.21.15.27:0/1640936948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe26004e850 con 0x7fe260158900 2024-01-15T18:15:10.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.729+0000 7fe25cc00640 1 -- 172.21.15.27:0/1640936948 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 27) v1 ==== 87129+0+0 (secure 0 0 0) 0x7fe2480073d0 con 0x7fe260158900 2024-01-15T18:15:10.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.729+0000 7fe25cc00640 1 --2- 172.21.15.27:0/1640936948 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe2380673a0 0x7fe238069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:10.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.729+0000 7fe25cc00640 1 -- 172.21.15.27:0/1640936948 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fe248082640 con 0x7fe260158900 2024-01-15T18:15:10.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.733+0000 7fe264a00640 1 --2- 172.21.15.27:0/1640936948 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe2380673a0 0x7fe238069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:10.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.733+0000 7fe264a00640 1 --2- 172.21.15.27:0/1640936948 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe2380673a0 0x7fe238069860 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe250006e90 tx=0x7fe250008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:10.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.733+0000 7fe25cc00640 1 -- 172.21.15.27:0/1640936948 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe248051300 con 0x7fe260158900 2024-01-15T18:15:10.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.921+0000 7fe267400640 1 -- 172.21.15.27:0/1640936948 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone51933", "target": ["mon-mgr", ""]}) v1 -- 0x7fe26006e170 con 0x7fe2380673a0 2024-01-15T18:15:10.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.933+0000 7fe25cc00640 1 -- 172.21.15.27:0/1640936948 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fe2500118e0 con 0x7fe2380673a0 2024-01-15T18:15:10.936 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:15:10.936 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:15:10.936 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "canceled", 2024-01-15T18:15:10.936 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:15:10.936 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:15:10.936 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:15:10.936 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:15:10.936 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:10.936 INFO:teuthology.orchestra.run.smithi027.stdout: "failure": { 2024-01-15T18:15:10.937 INFO:teuthology.orchestra.run.smithi027.stdout: "errno": "4", 2024-01-15T18:15:10.937 INFO:teuthology.orchestra.run.smithi027.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:15:10.937 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:10.937 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:10.937 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:15:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.937+0000 7fe267400640 1 -- 172.21.15.27:0/1640936948 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe2380673a0 msgr2=0x7fe238069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.937+0000 7fe267400640 1 --2- 172.21.15.27:0/1640936948 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe2380673a0 0x7fe238069860 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe250006e90 tx=0x7fe250008040 comp rx=0 tx=0).stop 2024-01-15T18:15:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.937+0000 7fe267400640 1 -- 172.21.15.27:0/1640936948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe260158900 msgr2=0x7fe2601f8f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.937+0000 7fe267400640 1 --2- 172.21.15.27:0/1640936948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe260158900 0x7fe2601f8f10 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fe24800c500 tx=0x7fe24800c9c0 comp rx=0 tx=0).stop 2024-01-15T18:15:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.937+0000 7fe267400640 1 -- 172.21.15.27:0/1640936948 shutdown_connections 2024-01-15T18:15:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.937+0000 7fe267400640 1 --2- 172.21.15.27:0/1640936948 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe2380673a0 0x7fe238069860 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:10.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.937+0000 7fe267400640 1 --2- 172.21.15.27:0/1640936948 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe26015bf00 0x7fe2601f9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:10.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.937+0000 7fe267400640 1 --2- 172.21.15.27:0/1640936948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe260158900 0x7fe2601f8f10 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:10.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.937+0000 7fe267400640 1 --2- 172.21.15.27:0/1640936948 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe2600cf2e0 0x7fe260150b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:10.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.937+0000 7fe267400640 1 -- 172.21.15.27:0/1640936948 >> 172.21.15.27:0/1640936948 conn(0x7fe2600cb700 msgr2=0x7fe2600cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:10.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.937+0000 7fe267400640 1 -- 172.21.15.27:0/1640936948 shutdown_connections 2024-01-15T18:15:10.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:10.937+0000 7fe267400640 1 -- 172.21.15.27:0/1640936948 wait complete. 2024-01-15T18:15:10.958 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:15:10.959 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone51934 2024-01-15T18:15:11.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.101+0000 7f4622a00640 1 Processor -- start 2024-01-15T18:15:11.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.101+0000 7f4622a00640 1 -- start start 2024-01-15T18:15:11.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.101+0000 7f4622a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f461c158900 0x7f461c15f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:11.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.101+0000 7f4622a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f461c15bf00 0x7f461c0cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:11.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.101+0000 7f4622a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f461c0cf110 0x7f461c0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:11.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.101+0000 7f4622a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f461c0b1cb0 con 0x7f461c158900 2024-01-15T18:15:11.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.101+0000 7f4622a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f461c0cf540 con 0x7f461c0cf110 2024-01-15T18:15:11.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.101+0000 7f4622a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f461c0b1f00 con 0x7f461c15bf00 2024-01-15T18:15:11.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.101+0000 7f461be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f461c158900 0x7f461c15f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:11.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.101+0000 7f461be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f461c158900 0x7f461c15f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34222/0 (socket says 172.21.15.27:34222) 2024-01-15T18:15:11.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.101+0000 7f461be00640 1 -- 172.21.15.27:0/501314256 learned_addr learned my addr 172.21.15.27:0/501314256 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:11.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.101+0000 7f4620a00640 1 --2- 172.21.15.27:0/501314256 >> v2:172.21.15.93:3300/0 conn(0x7f461c0cf110 0x7f461c0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:11.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.101+0000 7f461b400640 1 --2- 172.21.15.27:0/501314256 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f461c15bf00 0x7f461c0cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:11.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.101+0000 7f461be00640 1 -- 172.21.15.27:0/501314256 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f461c15bf00 msgr2=0x7f461c0cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:11.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.101+0000 7f461be00640 1 --2- 172.21.15.27:0/501314256 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f461c15bf00 0x7f461c0cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.101+0000 7f461be00640 1 -- 172.21.15.27:0/501314256 >> v2:172.21.15.93:3300/0 conn(0x7f461c0cf110 msgr2=0x7f461c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:11.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.101+0000 7f461be00640 1 --2- 172.21.15.27:0/501314256 >> v2:172.21.15.93:3300/0 conn(0x7f461c0cf110 0x7f461c0701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.101+0000 7f461be00640 1 -- 172.21.15.27:0/501314256 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f461c0b21e0 con 0x7f461c158900 2024-01-15T18:15:11.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.101+0000 7f4620a00640 1 --2- 172.21.15.27:0/501314256 >> v2:172.21.15.93:3300/0 conn(0x7f461c0cf110 0x7f461c0701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:11.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.105+0000 7f461be00640 1 --2- 172.21.15.27:0/501314256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f461c158900 0x7f461c15f550 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f461c0cfc20 tx=0x7f460c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3d0ebd86316bbb8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:11.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.105+0000 7f461aa00640 1 -- 172.21.15.27:0/501314256 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c070f30 con 0x7f461c158900 2024-01-15T18:15:11.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.105+0000 7f461aa00640 1 -- 172.21.15.27:0/501314256 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f460c0ac410 con 0x7f461c158900 2024-01-15T18:15:11.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.105+0000 7f461aa00640 1 -- 172.21.15.27:0/501314256 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c0ac7a0 con 0x7f461c158900 2024-01-15T18:15:11.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.105+0000 7f4622a00640 1 -- 172.21.15.27:0/501314256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f461c158900 msgr2=0x7f461c15f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:11.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.105+0000 7f4622a00640 1 --2- 172.21.15.27:0/501314256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f461c158900 0x7f461c15f550 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f461c0cfc20 tx=0x7f460c01c500 comp rx=0 tx=0).stop 2024-01-15T18:15:11.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.105+0000 7f4622a00640 1 -- 172.21.15.27:0/501314256 shutdown_connections 2024-01-15T18:15:11.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.105+0000 7f4622a00640 1 --2- 172.21.15.27:0/501314256 >> v2:172.21.15.93:3300/0 conn(0x7f461c0cf110 0x7f461c0701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.105+0000 7f4622a00640 1 --2- 172.21.15.27:0/501314256 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f461c15bf00 0x7f461c0cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.105+0000 7f4622a00640 1 --2- 172.21.15.27:0/501314256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f461c158900 0x7f461c15f550 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.105+0000 7f4622a00640 1 -- 172.21.15.27:0/501314256 >> 172.21.15.27:0/501314256 conn(0x7f461c0cb700 msgr2=0x7f461c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:11.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.105+0000 7f4622a00640 1 -- 172.21.15.27:0/501314256 shutdown_connections 2024-01-15T18:15:11.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.105+0000 7f4622a00640 1 -- 172.21.15.27:0/501314256 wait complete. 2024-01-15T18:15:11.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.109+0000 7f4622a00640 1 Processor -- start 2024-01-15T18:15:11.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.109+0000 7f4622a00640 1 -- start start 2024-01-15T18:15:11.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.109+0000 7f4622a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f461c0cf110 0x7f461c138cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:11.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.109+0000 7f4622a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f461c158900 0x7f461c153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:11.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.109+0000 7f4622a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f461c15bf00 0x7f461c0aebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:11.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.109+0000 7f4622a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461c153610 con 0x7f461c158900 2024-01-15T18:15:11.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.109+0000 7f4622a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461c152720 con 0x7f461c15bf00 2024-01-15T18:15:11.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.109+0000 7f4622a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f461c152a00 con 0x7f461c0cf110 2024-01-15T18:15:11.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.109+0000 7f461b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f461c158900 0x7f461c153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:11.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.109+0000 7f461b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f461c158900 0x7f461c153ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34228/0 (socket says 172.21.15.27:34228) 2024-01-15T18:15:11.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.109+0000 7f461b400640 1 -- 172.21.15.27:0/890417002 learned_addr learned my addr 172.21.15.27:0/890417002 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:11.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.109+0000 7f461be00640 1 --2- 172.21.15.27:0/890417002 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f461c0cf110 0x7f461c138cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:11.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.109+0000 7f4620a00640 1 --2- 172.21.15.27:0/890417002 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f461c15bf00 0x7f461c0aebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:11.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.109+0000 7f461b400640 1 -- 172.21.15.27:0/890417002 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f461c0cf110 msgr2=0x7f461c138cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:11.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.109+0000 7f461b400640 1 --2- 172.21.15.27:0/890417002 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f461c0cf110 0x7f461c138cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.109+0000 7f461b400640 1 -- 172.21.15.27:0/890417002 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f461c15bf00 msgr2=0x7f461c0aebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:11.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.109+0000 7f461b400640 1 --2- 172.21.15.27:0/890417002 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f461c15bf00 0x7f461c0aebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.109+0000 7f461b400640 1 -- 172.21.15.27:0/890417002 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460c0094d0 con 0x7f461c158900 2024-01-15T18:15:11.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.109+0000 7f461b400640 1 --2- 172.21.15.27:0/890417002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f461c158900 0x7f461c153ee0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f460400d5f0 tx=0x7f460400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:11.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.113+0000 7f4613e00640 1 -- 172.21.15.27:0/890417002 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46040125f0 con 0x7f461c158900 2024-01-15T18:15:11.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.113+0000 7f4613e00640 1 -- 172.21.15.27:0/890417002 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4604012750 con 0x7f461c158900 2024-01-15T18:15:11.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.113+0000 7f4622a00640 1 -- 172.21.15.27:0/890417002 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f461c15c490 con 0x7f461c158900 2024-01-15T18:15:11.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.113+0000 7f4613e00640 1 -- 172.21.15.27:0/890417002 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4604012ae0 con 0x7f461c158900 2024-01-15T18:15:11.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.113+0000 7f4622a00640 1 -- 172.21.15.27:0/890417002 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f461c152fc0 con 0x7f461c158900 2024-01-15T18:15:11.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.113+0000 7f4622a00640 1 -- 172.21.15.27:0/890417002 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f461c1537c0 con 0x7f461c158900 2024-01-15T18:15:11.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.117+0000 7f4613e00640 1 -- 172.21.15.27:0/890417002 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 27) v1 ==== 87129+0+0 (secure 0 0 0) 0x7f4604012c40 con 0x7f461c158900 2024-01-15T18:15:11.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.117+0000 7f4613e00640 1 --2- 172.21.15.27:0/890417002 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f45f40673f0 0x7f45f40698b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:11.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.117+0000 7f4613e00640 1 -- 172.21.15.27:0/890417002 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f4604083090 con 0x7f461c158900 2024-01-15T18:15:11.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.117+0000 7f461be00640 1 --2- 172.21.15.27:0/890417002 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f45f40673f0 0x7f45f40698b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:11.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.117+0000 7f461be00640 1 --2- 172.21.15.27:0/890417002 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f45f40673f0 0x7f45f40698b0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f460c0ae000 tx=0x7f460c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:11.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.121+0000 7f4613e00640 1 -- 172.21.15.27:0/890417002 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f461c1537c0 con 0x7f461c158900 2024-01-15T18:15:11.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.129+0000 7f4613e00640 1 -- 172.21.15.27:0/890417002 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7f46040521a0 con 0x7f461c158900 2024-01-15T18:15:11.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.313+0000 7f4622a00640 1 -- 172.21.15.27:0/890417002 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone51934", "target": ["mon-mgr", ""]}) v1 -- 0x7f461c165c90 con 0x7f45f40673f0 2024-01-15T18:15:11.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.325+0000 7f4613e00640 1 -- 172.21.15.27:0/890417002 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f461c165c90 con 0x7f45f40673f0 2024-01-15T18:15:11.326 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:15:11.326 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:15:11.327 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "canceled", 2024-01-15T18:15:11.327 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:15:11.327 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:15:11.327 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:15:11.327 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:15:11.327 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:11.327 INFO:teuthology.orchestra.run.smithi027.stdout: "failure": { 2024-01-15T18:15:11.327 INFO:teuthology.orchestra.run.smithi027.stdout: "errno": "4", 2024-01-15T18:15:11.327 INFO:teuthology.orchestra.run.smithi027.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:15:11.327 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:11.327 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:11.327 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:15:11.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.325+0000 7f4622a00640 1 -- 172.21.15.27:0/890417002 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f45f40673f0 msgr2=0x7f45f40698b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:11.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.325+0000 7f4622a00640 1 --2- 172.21.15.27:0/890417002 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f45f40673f0 0x7f45f40698b0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f460c0ae000 tx=0x7f460c009040 comp rx=0 tx=0).stop 2024-01-15T18:15:11.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.325+0000 7f4622a00640 1 -- 172.21.15.27:0/890417002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f461c158900 msgr2=0x7f461c153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:11.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.325+0000 7f4622a00640 1 --2- 172.21.15.27:0/890417002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f461c158900 0x7f461c153ee0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f460400d5f0 tx=0x7f460400dab0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.325+0000 7f4622a00640 1 -- 172.21.15.27:0/890417002 shutdown_connections 2024-01-15T18:15:11.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.325+0000 7f4622a00640 1 --2- 172.21.15.27:0/890417002 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f45f40673f0 0x7f45f40698b0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.325+0000 7f4622a00640 1 --2- 172.21.15.27:0/890417002 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f461c15bf00 0x7f461c0aebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.325+0000 7f4622a00640 1 --2- 172.21.15.27:0/890417002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f461c158900 0x7f461c153ee0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.325+0000 7f4622a00640 1 --2- 172.21.15.27:0/890417002 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f461c0cf110 0x7f461c138cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.325+0000 7f4622a00640 1 -- 172.21.15.27:0/890417002 >> 172.21.15.27:0/890417002 conn(0x7f461c0cb700 msgr2=0x7f461c15b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:11.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.325+0000 7f4622a00640 1 -- 172.21.15.27:0/890417002 shutdown_connections 2024-01-15T18:15:11.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.329+0000 7f4622a00640 1 -- 172.21.15.27:0/890417002 wait complete. 2024-01-15T18:15:11.348 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:15:11.348 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone51935 2024-01-15T18:15:11.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d68000640 1 Processor -- start 2024-01-15T18:15:11.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d68000640 1 -- start start 2024-01-15T18:15:11.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d68000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d600cf460 0x7f0d600cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:11.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d68000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d600cfd80 con 0x7f0d600b2590 2024-01-15T18:15:11.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d68000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d600d19a0 con 0x7f0d6015b4b0 2024-01-15T18:15:11.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d68000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d600d1c80 con 0x7f0d600cf460 2024-01-15T18:15:11.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d66000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0d600b2590 0x7f0d600cc130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54624/0 (socket says 172.21.15.27:54624) 2024-01-15T18:15:11.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d66000640 1 -- 172.21.15.27:0/1338651734 learned_addr learned my addr 172.21.15.27:0/1338651734 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:11.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d65600640 1 --2- 172.21.15.27:0/1338651734 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d600cf460 0x7f0d600cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:11.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d5fe00640 1 -- 172.21.15.27:0/1338651734 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3084422865 0 0) 0x7f0d600cfd80 con 0x7f0d600b2590 2024-01-15T18:15:11.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d5fe00640 1 -- 172.21.15.27:0/1338651734 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d40003530 con 0x7f0d600b2590 2024-01-15T18:15:11.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d5fe00640 1 -- 172.21.15.27:0/1338651734 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 137132883 0 0) 0x7f0d40003530 con 0x7f0d600b2590 2024-01-15T18:15:11.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d5fe00640 1 -- 172.21.15.27:0/1338651734 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d600cf460 msgr2=0x7f0d600cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:11.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d5fe00640 1 --2- 172.21.15.27:0/1338651734 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d600cf460 0x7f0d600cf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d5fe00640 1 -- 172.21.15.27:0/1338651734 >> v1:172.21.15.93:6789/0 conn(0x7f0d6015b4b0 legacy=0x7f0d600cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:11.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d5fe00640 1 -- 172.21.15.27:0/1338651734 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d600b07c0 con 0x7f0d600b2590 2024-01-15T18:15:11.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d5fe00640 1 -- 172.21.15.27:0/1338651734 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f0d58001a00 con 0x7f0d600b2590 2024-01-15T18:15:11.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d5fe00640 1 -- 172.21.15.27:0/1338651734 >> v1:172.21.15.27:6789/0 conn(0x7f0d600b2590 legacy=0x7f0d600cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:11.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d5fe00640 1 --2- 172.21.15.27:0/1338651734 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d40003a20 0x7f0d40003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:11.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d5fe00640 1 --2- 172.21.15.27:0/1338651734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d400047d0 0x7f0d40004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:11.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d5fe00640 1 --2- 172.21.15.27:0/1338651734 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d600cf460 0x7f0d40005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:11.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d5fe00640 1 -- 172.21.15.27:0/1338651734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d600d1c80 con 0x7f0d400047d0 2024-01-15T18:15:11.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d5fe00640 1 -- 172.21.15.27:0/1338651734 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4000ba40 con 0x7f0d40003a20 2024-01-15T18:15:11.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d5fe00640 1 -- 172.21.15.27:0/1338651734 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4000bbd0 con 0x7f0d600cf460 2024-01-15T18:15:11.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d64c00640 1 --2- 172.21.15.27:0/1338651734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d400047d0 0x7f0d40004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:11.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d65600640 1 --2- 172.21.15.27:0/1338651734 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d40003a20 0x7f0d40003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:11.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d66000640 1 --2- 172.21.15.27:0/1338651734 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d600cf460 0x7f0d40005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:11.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d64c00640 1 -- 172.21.15.27:0/1338651734 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d600cf460 msgr2=0x7f0d40005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:11.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d64c00640 1 --2- 172.21.15.27:0/1338651734 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d600cf460 0x7f0d40005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d64c00640 1 -- 172.21.15.27:0/1338651734 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d40003a20 msgr2=0x7f0d40003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:11.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d64c00640 1 --2- 172.21.15.27:0/1338651734 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d40003a20 0x7f0d40003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.493+0000 7f0d64c00640 1 -- 172.21.15.27:0/1338651734 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d40003530 con 0x7f0d400047d0 2024-01-15T18:15:11.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.497+0000 7f0d64c00640 1 --2- 172.21.15.27:0/1338651734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d400047d0 0x7f0d40004c50 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f0d48001860 tx=0x7f0d4801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfe6cfe1551bf1d7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:11.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.497+0000 7f0d5fe00640 1 -- 172.21.15.27:0/1338651734 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d480ac600 con 0x7f0d400047d0 2024-01-15T18:15:11.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.497+0000 7f0d5fe00640 1 -- 172.21.15.27:0/1338651734 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d480a8050 con 0x7f0d400047d0 2024-01-15T18:15:11.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.497+0000 7f0d5fe00640 1 -- 172.21.15.27:0/1338651734 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d480ac9a0 con 0x7f0d400047d0 2024-01-15T18:15:11.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.497+0000 7f0d68000640 1 -- 172.21.15.27:0/1338651734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d400047d0 msgr2=0x7f0d40004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:11.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.497+0000 7f0d68000640 1 --2- 172.21.15.27:0/1338651734 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d400047d0 0x7f0d40004c50 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f0d48001860 tx=0x7f0d4801c480 comp rx=0 tx=0).stop 2024-01-15T18:15:11.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.497+0000 7f0d66000640 1 -- 172.21.15.27:0/1338651734 reap_dead start 2024-01-15T18:15:11.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.497+0000 7f0d68000640 1 -- 172.21.15.27:0/1338651734 shutdown_connections 2024-01-15T18:15:11.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.497+0000 7f0d68000640 1 -- 172.21.15.27:0/1338651734 >> 172.21.15.27:0/1338651734 conn(0x7f0d600d0e80 msgr2=0x7f0d600d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:11.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.501+0000 7f0d68000640 1 -- 172.21.15.27:0/1338651734 shutdown_connections 2024-01-15T18:15:11.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.501+0000 7f0d68000640 1 -- 172.21.15.27:0/1338651734 wait complete. 2024-01-15T18:15:11.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.501+0000 7f0d68000640 1 Processor -- start 2024-01-15T18:15:11.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.501+0000 7f0d68000640 1 -- start start 2024-01-15T18:15:11.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.501+0000 7f0d68000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d400047d0 0x7f0d601577b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:11.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.501+0000 7f0d68000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d601fa120 0x7f0d601544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:11.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.501+0000 7f0d68000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d600c96c0 0x7f0d600c9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:11.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.501+0000 7f0d68000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d600ca0b0 con 0x7f0d601fa120 2024-01-15T18:15:11.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.501+0000 7f0d68000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d601f61a0 con 0x7f0d600c96c0 2024-01-15T18:15:11.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.501+0000 7f0d68000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d601fa5e0 con 0x7f0d400047d0 2024-01-15T18:15:11.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.501+0000 7f0d64c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d601fa120 0x7f0d601544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:11.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.501+0000 7f0d64c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d601fa120 0x7f0d601544d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34248/0 (socket says 172.21.15.27:34248) 2024-01-15T18:15:11.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.501+0000 7f0d64c00640 1 -- 172.21.15.27:0/1045336330 learned_addr learned my addr 172.21.15.27:0/1045336330 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:11.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.501+0000 7f0d65600640 1 --2- 172.21.15.27:0/1045336330 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d400047d0 0x7f0d601577b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:11.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.501+0000 7f0d66000640 1 --2- 172.21.15.27:0/1045336330 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d600c96c0 0x7f0d600c9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:11.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.501+0000 7f0d64c00640 1 -- 172.21.15.27:0/1045336330 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d400047d0 msgr2=0x7f0d601577b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:11.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.501+0000 7f0d64c00640 1 --2- 172.21.15.27:0/1045336330 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d400047d0 0x7f0d601577b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.501+0000 7f0d64c00640 1 -- 172.21.15.27:0/1045336330 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d600c96c0 msgr2=0x7f0d600c9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:11.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.501+0000 7f0d64c00640 1 --2- 172.21.15.27:0/1045336330 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d600c96c0 0x7f0d600c9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.501+0000 7f0d64c00640 1 -- 172.21.15.27:0/1045336330 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6006b6a0 con 0x7f0d601fa120 2024-01-15T18:15:11.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.501+0000 7f0d65600640 1 --2- 172.21.15.27:0/1045336330 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d400047d0 0x7f0d601577b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:11.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.501+0000 7f0d64c00640 1 --2- 172.21.15.27:0/1045336330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d601fa120 0x7f0d601544d0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f0d4801c2b0 tx=0x7f0d40005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:11.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.501+0000 7f0d5d600640 1 -- 172.21.15.27:0/1045336330 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d480acdc0 con 0x7f0d601fa120 2024-01-15T18:15:11.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.501+0000 7f0d5d600640 1 -- 172.21.15.27:0/1045336330 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d48005110 con 0x7f0d601fa120 2024-01-15T18:15:11.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.501+0000 7f0d68000640 1 -- 172.21.15.27:0/1045336330 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d6006b980 con 0x7f0d601fa120 2024-01-15T18:15:11.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.505+0000 7f0d5d600640 1 -- 172.21.15.27:0/1045336330 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d48005480 con 0x7f0d601fa120 2024-01-15T18:15:11.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.505+0000 7f0d68000640 1 -- 172.21.15.27:0/1045336330 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d6006bf40 con 0x7f0d601fa120 2024-01-15T18:15:11.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.505+0000 7f0d68000640 1 -- 172.21.15.27:0/1045336330 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d6004e850 con 0x7f0d601fa120 2024-01-15T18:15:11.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.505+0000 7f0d5d600640 1 -- 172.21.15.27:0/1045336330 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7f0d480b3040 con 0x7f0d601fa120 2024-01-15T18:15:11.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.505+0000 7f0d5d600640 1 --2- 172.21.15.27:0/1045336330 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0d38067460 0x7f0d38069920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:11.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.505+0000 7f0d5d600640 1 -- 172.21.15.27:0/1045336330 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f0d4811c230 con 0x7f0d601fa120 2024-01-15T18:15:11.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.505+0000 7f0d65600640 1 --2- 172.21.15.27:0/1045336330 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0d38067460 0x7f0d38069920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:11.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.509+0000 7f0d65600640 1 --2- 172.21.15.27:0/1045336330 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0d38067460 0x7f0d38069920 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0d6015af80 tx=0x7f0d50008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:11.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.509+0000 7f0d5d600640 1 -- 172.21.15.27:0/1045336330 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0d480ead20 con 0x7f0d601fa120 2024-01-15T18:15:11.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.701+0000 7f0d68000640 1 -- 172.21.15.27:0/1045336330 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone51935", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d600cc840 con 0x7f0d38067460 2024-01-15T18:15:11.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.709+0000 7f0d5d600640 1 -- 172.21.15.27:0/1045336330 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f0d50009850 con 0x7f0d38067460 2024-01-15T18:15:11.712 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:15:11.713 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:15:11.713 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "canceled", 2024-01-15T18:15:11.713 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:15:11.713 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:15:11.713 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:15:11.713 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:15:11.713 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:11.713 INFO:teuthology.orchestra.run.smithi027.stdout: "failure": { 2024-01-15T18:15:11.713 INFO:teuthology.orchestra.run.smithi027.stdout: "errno": "4", 2024-01-15T18:15:11.713 INFO:teuthology.orchestra.run.smithi027.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:15:11.713 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:11.713 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:11.713 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:15:11.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.713+0000 7f0d68000640 1 -- 172.21.15.27:0/1045336330 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0d38067460 msgr2=0x7f0d38069920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:11.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.713+0000 7f0d68000640 1 --2- 172.21.15.27:0/1045336330 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0d38067460 0x7f0d38069920 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0d6015af80 tx=0x7f0d50008040 comp rx=0 tx=0).stop 2024-01-15T18:15:11.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.713+0000 7f0d68000640 1 -- 172.21.15.27:0/1045336330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d601fa120 msgr2=0x7f0d601544d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:11.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.713+0000 7f0d68000640 1 --2- 172.21.15.27:0/1045336330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d601fa120 0x7f0d601544d0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f0d4801c2b0 tx=0x7f0d40005800 comp rx=0 tx=0).stop 2024-01-15T18:15:11.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.713+0000 7f0d68000640 1 -- 172.21.15.27:0/1045336330 shutdown_connections 2024-01-15T18:15:11.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.713+0000 7f0d68000640 1 --2- 172.21.15.27:0/1045336330 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0d38067460 0x7f0d38069920 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.713+0000 7f0d68000640 1 --2- 172.21.15.27:0/1045336330 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d600c96c0 0x7f0d600c9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.713+0000 7f0d68000640 1 --2- 172.21.15.27:0/1045336330 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d601fa120 0x7f0d601544d0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.713+0000 7f0d68000640 1 --2- 172.21.15.27:0/1045336330 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d400047d0 0x7f0d601577b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.713+0000 7f0d68000640 1 -- 172.21.15.27:0/1045336330 >> 172.21.15.27:0/1045336330 conn(0x7f0d600d0e80 msgr2=0x7f0d60154be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:11.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.713+0000 7f0d68000640 1 -- 172.21.15.27:0/1045336330 shutdown_connections 2024-01-15T18:15:11.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.713+0000 7f0d68000640 1 -- 172.21.15.27:0/1045336330 wait complete. 2024-01-15T18:15:11.733 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:15:11.734 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone51936 2024-01-15T18:15:11.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.877+0000 7f3fc1400640 1 Processor -- start 2024-01-15T18:15:11.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.877+0000 7f3fc1400640 1 -- start start 2024-01-15T18:15:11.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.877+0000 7f3fc1400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3fbc0cf450 0x7f3fbc0d1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:11.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.877+0000 7f3fc1400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3fbc0cba60 0x7f3fbc0cf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:11.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.877+0000 7f3fc1400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fbc0cefa0 con 0x7f3fbc15ee30 2024-01-15T18:15:11.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.877+0000 7f3fc1400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fbc0ce240 con 0x7f3fbc0cf450 2024-01-15T18:15:11.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.877+0000 7f3fc1400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0cfd70 con 0x7f3fbc0cba60 2024-01-15T18:15:11.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.877+0000 7f3fbb200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3fbc15ee30 0x7f3fbc15f2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54638/0 (socket says 172.21.15.27:54638) 2024-01-15T18:15:11.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.877+0000 7f3fbb200640 1 -- 172.21.15.27:0/528988208 learned_addr learned my addr 172.21.15.27:0/528988208 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:11.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.877+0000 7f3fba800640 1 --2- 172.21.15.27:0/528988208 >> v2:172.21.15.27:3300/0 conn(0x7f3fbc0cf450 0x7f3fbc0d1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:11.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.877+0000 7f3fb9e00640 1 --2- 172.21.15.27:0/528988208 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3fbc0cba60 0x7f3fbc0cf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:11.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.877+0000 7f3fba800640 1 -- 172.21.15.27:0/528988208 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3fbc0cba60 msgr2=0x7f3fbc0cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:11.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.877+0000 7f3fba800640 1 --2- 172.21.15.27:0/528988208 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3fbc0cba60 0x7f3fbc0cf830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.877+0000 7f3fba800640 1 -- 172.21.15.27:0/528988208 >> v1:172.21.15.27:6789/0 conn(0x7f3fbc15ee30 legacy=0x7f3fbc15f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:11.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.877+0000 7f3fba800640 1 -- 172.21.15.27:0/528988208 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc0ccd70 con 0x7f3fbc0cf450 2024-01-15T18:15:11.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.881+0000 7f3fba800640 1 --2- 172.21.15.27:0/528988208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0cf450 0x7f3fbc0d1a40 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f3fac0097c0 tx=0x7f3fac01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27099165ae2ba32e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:11.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.881+0000 7f3fb9400640 1 -- 172.21.15.27:0/528988208 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac0ab600 con 0x7f3fbc0cf450 2024-01-15T18:15:11.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.881+0000 7f3fb9400640 1 -- 172.21.15.27:0/528988208 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fac0a7050 con 0x7f3fbc0cf450 2024-01-15T18:15:11.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.881+0000 7f3fb9400640 1 -- 172.21.15.27:0/528988208 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac0ab980 con 0x7f3fbc0cf450 2024-01-15T18:15:11.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.881+0000 7f3fc1400640 1 -- 172.21.15.27:0/528988208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0cf450 msgr2=0x7f3fbc0d1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:11.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.881+0000 7f3fc1400640 1 --2- 172.21.15.27:0/528988208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0cf450 0x7f3fbc0d1a40 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f3fac0097c0 tx=0x7f3fac01c480 comp rx=0 tx=0).stop 2024-01-15T18:15:11.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.881+0000 7f3fc1400640 1 -- 172.21.15.27:0/528988208 shutdown_connections 2024-01-15T18:15:11.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.881+0000 7f3fc1400640 1 --2- 172.21.15.27:0/528988208 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3fbc0cba60 0x7f3fbc0cf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.881+0000 7f3fc1400640 1 --2- 172.21.15.27:0/528988208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0cf450 0x7f3fbc0d1a40 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.881+0000 7f3fc1400640 1 -- 172.21.15.27:0/528988208 >> 172.21.15.27:0/528988208 conn(0x7f3fbc0d0e70 msgr2=0x7f3fbc0d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:11.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.881+0000 7f3fc1400640 1 -- 172.21.15.27:0/528988208 shutdown_connections 2024-01-15T18:15:11.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.881+0000 7f3fc1400640 1 -- 172.21.15.27:0/528988208 wait complete. 2024-01-15T18:15:11.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.885+0000 7f3fc1400640 1 Processor -- start 2024-01-15T18:15:11.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.885+0000 7f3fc1400640 1 -- start start 2024-01-15T18:15:11.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.885+0000 7f3fc1400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3fbc0cba60 0x7f3fbc156910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:11.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.885+0000 7f3fc1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0cf450 0x7f3fbc165820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:11.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.885+0000 7f3fc1400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3fbc15ee30 0x7f3fbc157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:11.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.885+0000 7f3fc1400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc165400 con 0x7f3fbc0cf450 2024-01-15T18:15:11.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.885+0000 7f3fc1400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc164810 con 0x7f3fbc15ee30 2024-01-15T18:15:11.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.885+0000 7f3fc1400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc156e50 con 0x7f3fbc0cba60 2024-01-15T18:15:11.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.885+0000 7f3fb9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0cf450 0x7f3fbc165820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:11.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.885+0000 7f3fb9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0cf450 0x7f3fbc165820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34266/0 (socket says 172.21.15.27:34266) 2024-01-15T18:15:11.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.885+0000 7f3fb9e00640 1 -- 172.21.15.27:0/3789747594 learned_addr learned my addr 172.21.15.27:0/3789747594 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:11.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.885+0000 7f3fba800640 1 --2- 172.21.15.27:0/3789747594 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3fbc0cba60 0x7f3fbc156910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:11.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.885+0000 7f3fbb200640 1 --2- 172.21.15.27:0/3789747594 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3fbc15ee30 0x7f3fbc157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:11.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.885+0000 7f3fb9e00640 1 -- 172.21.15.27:0/3789747594 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3fbc0cba60 msgr2=0x7f3fbc156910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:11.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.885+0000 7f3fb9e00640 1 --2- 172.21.15.27:0/3789747594 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3fbc0cba60 0x7f3fbc156910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.885+0000 7f3fb9e00640 1 -- 172.21.15.27:0/3789747594 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3fbc15ee30 msgr2=0x7f3fbc157d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:11.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.885+0000 7f3fb9e00640 1 --2- 172.21.15.27:0/3789747594 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3fbc15ee30 0x7f3fbc157d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:11.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.885+0000 7f3fb9e00640 1 -- 172.21.15.27:0/3789747594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fac009420 con 0x7f3fbc0cf450 2024-01-15T18:15:11.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.885+0000 7f3fba800640 1 --2- 172.21.15.27:0/3789747594 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3fbc0cba60 0x7f3fbc156910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:11.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.885+0000 7f3fb9e00640 1 --2- 172.21.15.27:0/3789747594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0cf450 0x7f3fbc165820 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f3fa400d5f0 tx=0x7f3fa400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:11.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.885+0000 7f3fb2a00640 1 -- 172.21.15.27:0/3789747594 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa40125f0 con 0x7f3fbc0cf450 2024-01-15T18:15:11.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.885+0000 7f3fb2a00640 1 -- 172.21.15.27:0/3789747594 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa4012750 con 0x7f3fbc0cf450 2024-01-15T18:15:11.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.885+0000 7f3fb2a00640 1 -- 172.21.15.27:0/3789747594 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa4012ae0 con 0x7f3fbc0cf450 2024-01-15T18:15:11.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.885+0000 7f3fc1400640 1 -- 172.21.15.27:0/3789747594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fbc0cbfc0 con 0x7f3fbc0cf450 2024-01-15T18:15:11.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.885+0000 7f3fc1400640 1 -- 172.21.15.27:0/3789747594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fbc164d40 con 0x7f3fbc0cf450 2024-01-15T18:15:11.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.889+0000 7f3fc1400640 1 -- 172.21.15.27:0/3789747594 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fbc165670 con 0x7f3fbc0cf450 2024-01-15T18:15:11.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.889+0000 7f3fb2a00640 1 -- 172.21.15.27:0/3789747594 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7f3fa4012c40 con 0x7f3fbc0cf450 2024-01-15T18:15:11.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.889+0000 7f3fb2a00640 1 --2- 172.21.15.27:0/3789747594 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3f94067430 0x7f3f940698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:11.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.889+0000 7f3fb2a00640 1 -- 172.21.15.27:0/3789747594 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f3fa40832f0 con 0x7f3fbc0cf450 2024-01-15T18:15:11.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.889+0000 7f3fba800640 1 --2- 172.21.15.27:0/3789747594 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3f94067430 0x7f3f940698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:11.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.889+0000 7f3fba800640 1 --2- 172.21.15.27:0/3789747594 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3f94067430 0x7f3f940698f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f3fac0ad000 tx=0x7f3fac008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:11.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:11.893+0000 7f3fb2a00640 1 -- 172.21.15.27:0/3789747594 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3fa4051f60 con 0x7f3fbc0cf450 2024-01-15T18:15:12.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.085+0000 7f3fc1400640 1 -- 172.21.15.27:0/3789747594 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone51936", "target": ["mon-mgr", ""]}) v1 -- 0x7f3fbc0cbe40 con 0x7f3f94067430 2024-01-15T18:15:12.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.097+0000 7f3fb2a00640 1 -- 172.21.15.27:0/3789747594 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f3fac0ab780 con 0x7f3f94067430 2024-01-15T18:15:12.099 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:15:12.099 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:15:12.099 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "canceled", 2024-01-15T18:15:12.099 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:15:12.099 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:15:12.100 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:15:12.100 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:15:12.100 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:12.100 INFO:teuthology.orchestra.run.smithi027.stdout: "failure": { 2024-01-15T18:15:12.100 INFO:teuthology.orchestra.run.smithi027.stdout: "errno": "4", 2024-01-15T18:15:12.100 INFO:teuthology.orchestra.run.smithi027.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:15:12.100 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:12.100 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:12.100 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:15:12.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.101+0000 7f3fc1400640 1 -- 172.21.15.27:0/3789747594 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3f94067430 msgr2=0x7f3f940698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:12.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.101+0000 7f3fc1400640 1 --2- 172.21.15.27:0/3789747594 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3f94067430 0x7f3f940698f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f3fac0ad000 tx=0x7f3fac008040 comp rx=0 tx=0).stop 2024-01-15T18:15:12.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.101+0000 7f3fc1400640 1 -- 172.21.15.27:0/3789747594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0cf450 msgr2=0x7f3fbc165820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:12.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.101+0000 7f3fc1400640 1 --2- 172.21.15.27:0/3789747594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0cf450 0x7f3fbc165820 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f3fa400d5f0 tx=0x7f3fa400dab0 comp rx=0 tx=0).stop 2024-01-15T18:15:12.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.101+0000 7f3fc1400640 1 -- 172.21.15.27:0/3789747594 shutdown_connections 2024-01-15T18:15:12.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.101+0000 7f3fc1400640 1 --2- 172.21.15.27:0/3789747594 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3f94067430 0x7f3f940698f0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:12.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.101+0000 7f3fc1400640 1 --2- 172.21.15.27:0/3789747594 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3fbc15ee30 0x7f3fbc157d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:12.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.101+0000 7f3fc1400640 1 --2- 172.21.15.27:0/3789747594 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc0cf450 0x7f3fbc165820 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:12.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.101+0000 7f3fc1400640 1 --2- 172.21.15.27:0/3789747594 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3fbc0cba60 0x7f3fbc156910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:12.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.101+0000 7f3fc1400640 1 -- 172.21.15.27:0/3789747594 >> 172.21.15.27:0/3789747594 conn(0x7f3fbc0d0e70 msgr2=0x7f3fbc06fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:12.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.101+0000 7f3fc1400640 1 -- 172.21.15.27:0/3789747594 shutdown_connections 2024-01-15T18:15:12.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.101+0000 7f3fc1400640 1 -- 172.21.15.27:0/3789747594 wait complete. 2024-01-15T18:15:12.121 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:15:12.121 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone51937 2024-01-15T18:15:12.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.265+0000 7fb323600640 1 Processor -- start 2024-01-15T18:15:12.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.265+0000 7fb323600640 1 -- start start 2024-01-15T18:15:12.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.265+0000 7fb323600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fb31c15d680 0x7fb31c0cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.265+0000 7fb323600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb31c0cfc30 0x7fb31c0cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.265+0000 7fb323600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31c0cb4d0 con 0x7fb31c0d19a0 2024-01-15T18:15:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.265+0000 7fb323600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb31c0d0120 con 0x7fb31c0cfc30 2024-01-15T18:15:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.265+0000 7fb323600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb31c0d0400 con 0x7fb31c15d680 2024-01-15T18:15:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.265+0000 7fb31be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb31c0cfc30 0x7fb31c0cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.265+0000 7fb31be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb31c0cfc30 0x7fb31c0cdc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34278/0 (socket says 172.21.15.27:34278) 2024-01-15T18:15:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.265+0000 7fb31be00640 1 -- 172.21.15.27:0/1809473381 learned_addr learned my addr 172.21.15.27:0/1809473381 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.265+0000 7fb320c00640 1 --2- 172.21.15.27:0/1809473381 >> v2:172.21.15.93:3300/0 conn(0x7fb31c15d680 0x7fb31c0cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.269+0000 7fb31be00640 1 -- 172.21.15.27:0/1809473381 >> v2:172.21.15.93:3300/0 conn(0x7fb31c15d680 msgr2=0x7fb31c0cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.269+0000 7fb31be00640 1 --2- 172.21.15.27:0/1809473381 >> v2:172.21.15.93:3300/0 conn(0x7fb31c15d680 0x7fb31c0cd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.269+0000 7fb31be00640 1 -- 172.21.15.27:0/1809473381 >> v1:172.21.15.93:6789/0 conn(0x7fb31c0d19a0 legacy=0x7fb31c06b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:12.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.269+0000 7fb31be00640 1 -- 172.21.15.27:0/1809473381 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31c0cb770 con 0x7fb31c0cfc30 2024-01-15T18:15:12.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.269+0000 7fb31be00640 1 --2- 172.21.15.27:0/1809473381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31c0cfc30 0x7fb31c0cdc40 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fb31c0d0e50 tx=0x7fb30401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f9bf433110c6a78 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:12.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.269+0000 7fb31b400640 1 -- 172.21.15.27:0/1809473381 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c0ce850 con 0x7fb31c0cfc30 2024-01-15T18:15:12.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.269+0000 7fb31b400640 1 -- 172.21.15.27:0/1809473381 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3040aa650 con 0x7fb31c0cfc30 2024-01-15T18:15:12.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.269+0000 7fb31b400640 1 -- 172.21.15.27:0/1809473381 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3040aa9e0 con 0x7fb31c0cfc30 2024-01-15T18:15:12.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.273+0000 7fb323600640 1 -- 172.21.15.27:0/1809473381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31c0cfc30 msgr2=0x7fb31c0cdc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:12.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.273+0000 7fb323600640 1 --2- 172.21.15.27:0/1809473381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31c0cfc30 0x7fb31c0cdc40 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fb31c0d0e50 tx=0x7fb30401c4d0 comp rx=0 tx=0).stop 2024-01-15T18:15:12.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.273+0000 7fb323600640 1 -- 172.21.15.27:0/1809473381 shutdown_connections 2024-01-15T18:15:12.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.273+0000 7fb323600640 1 --2- 172.21.15.27:0/1809473381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31c0cfc30 0x7fb31c0cdc40 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:12.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.273+0000 7fb323600640 1 --2- 172.21.15.27:0/1809473381 >> v2:172.21.15.93:3300/0 conn(0x7fb31c15d680 0x7fb31c0cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:12.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.273+0000 7fb323600640 1 -- 172.21.15.27:0/1809473381 >> 172.21.15.27:0/1809473381 conn(0x7fb31c0d0e80 msgr2=0x7fb31c0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:12.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.273+0000 7fb323600640 1 -- 172.21.15.27:0/1809473381 shutdown_connections 2024-01-15T18:15:12.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.273+0000 7fb323600640 1 -- 172.21.15.27:0/1809473381 wait complete. 2024-01-15T18:15:12.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.273+0000 7fb323600640 1 Processor -- start 2024-01-15T18:15:12.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.273+0000 7fb323600640 1 -- start start 2024-01-15T18:15:12.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.273+0000 7fb323600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb31c0cfc30 0x7fb31c148a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:12.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.273+0000 7fb323600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31c0d19a0 0x7fb31c0c4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:12.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.273+0000 7fb323600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb31c15d680 0x7fb31c0c7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:12.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.273+0000 7fb323600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c150c80 con 0x7fb31c0d19a0 2024-01-15T18:15:12.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.273+0000 7fb323600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c149220 con 0x7fb31c15d680 2024-01-15T18:15:12.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.273+0000 7fb323600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c1503e0 con 0x7fb31c0cfc30 2024-01-15T18:15:12.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.273+0000 7fb31be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31c0d19a0 0x7fb31c0c4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:12.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.273+0000 7fb31be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31c0d19a0 0x7fb31c0c4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34288/0 (socket says 172.21.15.27:34288) 2024-01-15T18:15:12.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.273+0000 7fb31be00640 1 -- 172.21.15.27:0/2793217078 learned_addr learned my addr 172.21.15.27:0/2793217078 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:12.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.273+0000 7fb31be00640 1 -- 172.21.15.27:0/2793217078 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb31c0cfc30 msgr2=0x7fb31c148a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:12.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.273+0000 7fb31be00640 1 --2- 172.21.15.27:0/2793217078 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb31c0cfc30 0x7fb31c148a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:12.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.273+0000 7fb31be00640 1 -- 172.21.15.27:0/2793217078 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb31c15d680 msgr2=0x7fb31c0c7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:12.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.273+0000 7fb31be00640 1 --2- 172.21.15.27:0/2793217078 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb31c15d680 0x7fb31c0c7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:12.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.273+0000 7fb31be00640 1 -- 172.21.15.27:0/2793217078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3040094d0 con 0x7fb31c0d19a0 2024-01-15T18:15:12.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.277+0000 7fb31be00640 1 --2- 172.21.15.27:0/2793217078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31c0d19a0 0x7fb31c0c4990 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fb3040190e0 tx=0x7fb30401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.277+0000 7fb318c00640 1 -- 172.21.15.27:0/2793217078 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3040b3800 con 0x7fb31c0d19a0 2024-01-15T18:15:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.277+0000 7fb318c00640 1 -- 172.21.15.27:0/2793217078 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3040b3960 con 0x7fb31c0d19a0 2024-01-15T18:15:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.277+0000 7fb318c00640 1 -- 172.21.15.27:0/2793217078 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3040b3d10 con 0x7fb31c0d19a0 2024-01-15T18:15:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.277+0000 7fb323600640 1 -- 172.21.15.27:0/2793217078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb31c1506c0 con 0x7fb31c0d19a0 2024-01-15T18:15:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.277+0000 7fb323600640 1 -- 172.21.15.27:0/2793217078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb31c14a770 con 0x7fb31c0d19a0 2024-01-15T18:15:12.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.277+0000 7fb323600640 1 -- 172.21.15.27:0/2793217078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb31c149080 con 0x7fb31c0d19a0 2024-01-15T18:15:12.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.277+0000 7fb318c00640 1 -- 172.21.15.27:0/2793217078 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7fb3040b2380 con 0x7fb31c0d19a0 2024-01-15T18:15:12.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.277+0000 7fb318c00640 1 --2- 172.21.15.27:0/2793217078 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb2f4067430 0x7fb2f40698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:12.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.277+0000 7fb318c00640 1 -- 172.21.15.27:0/2793217078 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fb30411d5d0 con 0x7fb31c0d19a0 2024-01-15T18:15:12.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.281+0000 7fb320c00640 1 --2- 172.21.15.27:0/2793217078 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb2f4067430 0x7fb2f40698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:12.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.281+0000 7fb320c00640 1 --2- 172.21.15.27:0/2793217078 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb2f4067430 0x7fb2f40698f0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00b850 tx=0x7fb30c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:12.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.281+0000 7fb318c00640 1 -- 172.21.15.27:0/2793217078 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb31c149080 con 0x7fb31c0d19a0 2024-01-15T18:15:12.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.469+0000 7fb323600640 1 -- 172.21.15.27:0/2793217078 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone51937", "target": ["mon-mgr", ""]}) v1 -- 0x7fb31c149080 con 0x7fb2f4067430 2024-01-15T18:15:12.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.481+0000 7fb318c00640 1 -- 172.21.15.27:0/2793217078 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fb31c149080 con 0x7fb2f4067430 2024-01-15T18:15:12.482 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:15:12.482 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:15:12.483 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "canceled", 2024-01-15T18:15:12.483 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:15:12.483 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:15:12.483 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:15:12.483 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:15:12.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:12.483 INFO:teuthology.orchestra.run.smithi027.stdout: "failure": { 2024-01-15T18:15:12.483 INFO:teuthology.orchestra.run.smithi027.stdout: "errno": "4", 2024-01-15T18:15:12.483 INFO:teuthology.orchestra.run.smithi027.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:15:12.483 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:12.483 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:12.483 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:15:12.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.481+0000 7fb323600640 1 -- 172.21.15.27:0/2793217078 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb2f4067430 msgr2=0x7fb2f40698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:12.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.481+0000 7fb323600640 1 --2- 172.21.15.27:0/2793217078 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb2f4067430 0x7fb2f40698f0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00b850 tx=0x7fb30c008040 comp rx=0 tx=0).stop 2024-01-15T18:15:12.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.481+0000 7fb323600640 1 -- 172.21.15.27:0/2793217078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31c0d19a0 msgr2=0x7fb31c0c4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:12.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.481+0000 7fb323600640 1 --2- 172.21.15.27:0/2793217078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31c0d19a0 0x7fb31c0c4990 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fb3040190e0 tx=0x7fb30401ad10 comp rx=0 tx=0).stop 2024-01-15T18:15:12.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.481+0000 7fb323600640 1 -- 172.21.15.27:0/2793217078 shutdown_connections 2024-01-15T18:15:12.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.481+0000 7fb323600640 1 --2- 172.21.15.27:0/2793217078 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb2f4067430 0x7fb2f40698f0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:12.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.481+0000 7fb323600640 1 --2- 172.21.15.27:0/2793217078 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb31c15d680 0x7fb31c0c7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:12.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.481+0000 7fb323600640 1 --2- 172.21.15.27:0/2793217078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31c0d19a0 0x7fb31c0c4990 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:12.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.481+0000 7fb323600640 1 --2- 172.21.15.27:0/2793217078 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb31c0cfc30 0x7fb31c148a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:12.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.481+0000 7fb323600640 1 -- 172.21.15.27:0/2793217078 >> 172.21.15.27:0/2793217078 conn(0x7fb31c0d0e80 msgr2=0x7fb31c06f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:12.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.485+0000 7fb323600640 1 -- 172.21.15.27:0/2793217078 shutdown_connections 2024-01-15T18:15:12.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.485+0000 7fb323600640 1 -- 172.21.15.27:0/2793217078 wait complete. 2024-01-15T18:15:12.505 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:15:12.505 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone51938 2024-01-15T18:15:12.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.649+0000 7f214ea00640 1 Processor -- start 2024-01-15T18:15:12.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.649+0000 7f214ea00640 1 -- start start 2024-01-15T18:15:12.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.649+0000 7f214ea00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f214815b4b0 0x7f21480b1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:12.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.649+0000 7f214ea00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21480d19a0 con 0x7f21480cf460 2024-01-15T18:15:12.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.649+0000 7f214ea00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21480d1c80 con 0x7f21480cec60 2024-01-15T18:15:12.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.649+0000 7f214ea00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21480b07c0 con 0x7f214815b4b0 2024-01-15T18:15:12.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.649+0000 7f2147e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f21480cf460 0x7f21480cf840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54648/0 (socket says 172.21.15.27:54648) 2024-01-15T18:15:12.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.649+0000 7f2147e00640 1 -- 172.21.15.27:0/1210962110 learned_addr learned my addr 172.21.15.27:0/1210962110 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:12.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.649+0000 7f2147400640 1 --2- 172.21.15.27:0/1210962110 >> v2:172.21.15.93:3300/0 conn(0x7f214815b4b0 0x7f21480b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:12.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.649+0000 7f2146a00640 1 -- 172.21.15.27:0/1210962110 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135843076 0 0) 0x7f21480d19a0 con 0x7f21480cf460 2024-01-15T18:15:12.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.649+0000 7f2146a00640 1 -- 172.21.15.27:0/1210962110 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2128003530 con 0x7f21480cf460 2024-01-15T18:15:12.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.649+0000 7f2147400640 1 -- 172.21.15.27:0/1210962110 >> v1:172.21.15.93:6789/0 conn(0x7f21480cec60 legacy=0x7f21480b1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:12.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.649+0000 7f2147400640 1 -- 172.21.15.27:0/1210962110 >> v1:172.21.15.27:6789/0 conn(0x7f21480cf460 legacy=0x7f21480cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:12.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.649+0000 7f2147400640 1 -- 172.21.15.27:0/1210962110 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21480b0aa0 con 0x7f214815b4b0 2024-01-15T18:15:12.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.653+0000 7f2147400640 1 --2- 172.21.15.27:0/1210962110 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f214815b4b0 0x7f21480b1580 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f21300097c0 tx=0x7f213001c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f948507e901bd1e7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:12.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.653+0000 7f2146a00640 1 -- 172.21.15.27:0/1210962110 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21300ab680 con 0x7f214815b4b0 2024-01-15T18:15:12.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.653+0000 7f2146a00640 1 -- 172.21.15.27:0/1210962110 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21300ab7e0 con 0x7f214815b4b0 2024-01-15T18:15:12.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.653+0000 7f2146a00640 1 -- 172.21.15.27:0/1210962110 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21300abb50 con 0x7f214815b4b0 2024-01-15T18:15:12.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.653+0000 7f214ea00640 1 -- 172.21.15.27:0/1210962110 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f214815b4b0 msgr2=0x7f21480b1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:12.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.653+0000 7f214ea00640 1 --2- 172.21.15.27:0/1210962110 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f214815b4b0 0x7f21480b1580 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f21300097c0 tx=0x7f213001c480 comp rx=0 tx=0).stop 2024-01-15T18:15:12.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.653+0000 7f214ea00640 1 -- 172.21.15.27:0/1210962110 shutdown_connections 2024-01-15T18:15:12.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.653+0000 7f214ea00640 1 --2- 172.21.15.27:0/1210962110 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f214815b4b0 0x7f21480b1580 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:12.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.653+0000 7f214ea00640 1 -- 172.21.15.27:0/1210962110 >> 172.21.15.27:0/1210962110 conn(0x7f21480d0e80 msgr2=0x7f21480d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:12.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.653+0000 7f214ea00640 1 -- 172.21.15.27:0/1210962110 shutdown_connections 2024-01-15T18:15:12.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.653+0000 7f214ea00640 1 -- 172.21.15.27:0/1210962110 wait complete. 2024-01-15T18:15:12.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.657+0000 7f214ea00640 1 Processor -- start 2024-01-15T18:15:12.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.657+0000 7f214ea00640 1 -- start start 2024-01-15T18:15:12.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.657+0000 7f214ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21480cec60 0x7f2148156300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:12.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.657+0000 7f214ea00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f21480cf460 0x7f214815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:12.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.657+0000 7f214ea00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f214815b4b0 0x7f214815c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:12.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.657+0000 7f214ea00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214814f1c0 con 0x7f21480cec60 2024-01-15T18:15:12.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.657+0000 7f214ea00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214814f4a0 con 0x7f214815b4b0 2024-01-15T18:15:12.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.657+0000 7f214ea00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f214814f780 con 0x7f21480cf460 2024-01-15T18:15:12.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.657+0000 7f2147e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21480cec60 0x7f2148156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:12.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.657+0000 7f2147e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21480cec60 0x7f2148156300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34290/0 (socket says 172.21.15.27:34290) 2024-01-15T18:15:12.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.657+0000 7f2147e00640 1 -- 172.21.15.27:0/907836145 learned_addr learned my addr 172.21.15.27:0/907836145 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:12.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.657+0000 7f214ca00640 1 --2- 172.21.15.27:0/907836145 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f214815b4b0 0x7f214815c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:12.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.657+0000 7f2147400640 1 --2- 172.21.15.27:0/907836145 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f21480cf460 0x7f214815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:12.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.657+0000 7f2147e00640 1 -- 172.21.15.27:0/907836145 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f21480cf460 msgr2=0x7f214815bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:12.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.657+0000 7f2147e00640 1 --2- 172.21.15.27:0/907836145 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f21480cf460 0x7f214815bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:12.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.657+0000 7f2147e00640 1 -- 172.21.15.27:0/907836145 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f214815b4b0 msgr2=0x7f214815c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:12.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.657+0000 7f2147e00640 1 --2- 172.21.15.27:0/907836145 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f214815b4b0 0x7f214815c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:12.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.657+0000 7f2147e00640 1 -- 172.21.15.27:0/907836145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2130009420 con 0x7f21480cec60 2024-01-15T18:15:12.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.657+0000 7f2147e00640 1 --2- 172.21.15.27:0/907836145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21480cec60 0x7f2148156300 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f213800c4d0 tx=0x7f213800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:12.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.657+0000 7f213fe00640 1 -- 172.21.15.27:0/907836145 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2138005cf0 con 0x7f21480cec60 2024-01-15T18:15:12.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.657+0000 7f213fe00640 1 -- 172.21.15.27:0/907836145 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2138005e50 con 0x7f21480cec60 2024-01-15T18:15:12.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.657+0000 7f214ea00640 1 -- 172.21.15.27:0/907836145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f214814fa60 con 0x7f21480cec60 2024-01-15T18:15:12.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.657+0000 7f213fe00640 1 -- 172.21.15.27:0/907836145 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2138007270 con 0x7f21480cec60 2024-01-15T18:15:12.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.661+0000 7f214ea00640 1 -- 172.21.15.27:0/907836145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2148151e40 con 0x7f21480cec60 2024-01-15T18:15:12.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.661+0000 7f214ea00640 1 -- 172.21.15.27:0/907836145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21480d0440 con 0x7f21480cec60 2024-01-15T18:15:12.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.661+0000 7f213fe00640 1 -- 172.21.15.27:0/907836145 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7f21380073d0 con 0x7f21480cec60 2024-01-15T18:15:12.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.661+0000 7f213fe00640 1 --2- 172.21.15.27:0/907836145 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2120067430 0x7f21200698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:12.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.661+0000 7f213fe00640 1 -- 172.21.15.27:0/907836145 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f2138082520 con 0x7f21480cec60 2024-01-15T18:15:12.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.661+0000 7f2147400640 1 --2- 172.21.15.27:0/907836145 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2120067430 0x7f21200698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:12.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.665+0000 7f2147400640 1 --2- 172.21.15.27:0/907836145 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2120067430 0x7f21200698f0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f21300af000 tx=0x7f2130008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:12.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.665+0000 7f213fe00640 1 -- 172.21.15.27:0/907836145 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2138050ff0 con 0x7f21480cec60 2024-01-15T18:15:12.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.857+0000 7f214ea00640 1 -- 172.21.15.27:0/907836145 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone51938", "target": ["mon-mgr", ""]}) v1 -- 0x7f214804e850 con 0x7f2120067430 2024-01-15T18:15:12.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.869+0000 7f213fe00640 1 -- 172.21.15.27:0/907836145 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f214804e850 con 0x7f2120067430 2024-01-15T18:15:12.869 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:15:12.869 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:15:12.869 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "canceled", 2024-01-15T18:15:12.870 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:15:12.870 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:15:12.870 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:15:12.870 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:15:12.870 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:12.870 INFO:teuthology.orchestra.run.smithi027.stdout: "failure": { 2024-01-15T18:15:12.870 INFO:teuthology.orchestra.run.smithi027.stdout: "errno": "4", 2024-01-15T18:15:12.870 INFO:teuthology.orchestra.run.smithi027.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:15:12.870 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:12.870 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:12.870 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:15:12.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.869+0000 7f214ea00640 1 -- 172.21.15.27:0/907836145 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2120067430 msgr2=0x7f21200698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:12.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.869+0000 7f214ea00640 1 --2- 172.21.15.27:0/907836145 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2120067430 0x7f21200698f0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f21300af000 tx=0x7f2130008040 comp rx=0 tx=0).stop 2024-01-15T18:15:12.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.869+0000 7f214ea00640 1 -- 172.21.15.27:0/907836145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21480cec60 msgr2=0x7f2148156300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:12.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.869+0000 7f214ea00640 1 --2- 172.21.15.27:0/907836145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21480cec60 0x7f2148156300 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f213800c4d0 tx=0x7f213800c990 comp rx=0 tx=0).stop 2024-01-15T18:15:12.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.869+0000 7f214ea00640 1 -- 172.21.15.27:0/907836145 shutdown_connections 2024-01-15T18:15:12.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.869+0000 7f214ea00640 1 --2- 172.21.15.27:0/907836145 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2120067430 0x7f21200698f0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:12.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.869+0000 7f214ea00640 1 --2- 172.21.15.27:0/907836145 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f214815b4b0 0x7f214815c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:12.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.869+0000 7f214ea00640 1 --2- 172.21.15.27:0/907836145 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f21480cf460 0x7f214815bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:12.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.869+0000 7f214ea00640 1 --2- 172.21.15.27:0/907836145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21480cec60 0x7f2148156300 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:12.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.869+0000 7f214ea00640 1 -- 172.21.15.27:0/907836145 >> 172.21.15.27:0/907836145 conn(0x7f21480d0e80 msgr2=0x7f21480b13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:12.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.869+0000 7f214ea00640 1 -- 172.21.15.27:0/907836145 shutdown_connections 2024-01-15T18:15:12.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:12.869+0000 7f214ea00640 1 -- 172.21.15.27:0/907836145 wait complete. 2024-01-15T18:15:12.893 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:15:12.893 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone51939 2024-01-15T18:15:13.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.037+0000 7fefbfc00640 1 Processor -- start 2024-01-15T18:15:13.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.037+0000 7fefbfc00640 1 -- start start 2024-01-15T18:15:13.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.037+0000 7fefbfc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fefb80b4300 0x7fefb80cf940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:13.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.037+0000 7fefbfc00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fefb815be40 0x7fefb80720a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:13.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.037+0000 7fefbfc00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb80725e0 con 0x7fefb8071c50 2024-01-15T18:15:13.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.037+0000 7fefbfc00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefb806a4f0 con 0x7fefb80b4300 2024-01-15T18:15:13.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.037+0000 7fefbfc00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefb80cebf0 con 0x7fefb815be40 2024-01-15T18:15:13.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.041+0000 7fefbd200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fefb80b4300 0x7fefb80cf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:13.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.041+0000 7fefbd200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fefb80b4300 0x7fefb80cf940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34296/0 (socket says 172.21.15.27:34296) 2024-01-15T18:15:13.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.041+0000 7fefbd200640 1 -- 172.21.15.27:0/2570030321 learned_addr learned my addr 172.21.15.27:0/2570030321 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:13.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.041+0000 7fefbe600640 1 --2- 172.21.15.27:0/2570030321 >> v2:172.21.15.93:3300/0 conn(0x7fefb815be40 0x7fefb80720a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:13.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.041+0000 7fefbd200640 1 -- 172.21.15.27:0/2570030321 >> v2:172.21.15.93:3300/0 conn(0x7fefb815be40 msgr2=0x7fefb80720a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:13.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.041+0000 7fefbd200640 1 --2- 172.21.15.27:0/2570030321 >> v2:172.21.15.93:3300/0 conn(0x7fefb815be40 0x7fefb80720a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:13.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.041+0000 7fefbd200640 1 -- 172.21.15.27:0/2570030321 >> v1:172.21.15.93:6789/0 conn(0x7fefb8071c50 legacy=0x7fefb815b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:13.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.041+0000 7fefbd200640 1 -- 172.21.15.27:0/2570030321 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb80b46e0 con 0x7fefb80b4300 2024-01-15T18:15:13.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.041+0000 7fefbe600640 1 --2- 172.21.15.27:0/2570030321 >> v2:172.21.15.93:3300/0 conn(0x7fefb815be40 0x7fefb80720a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:13.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.041+0000 7fefbd200640 1 --2- 172.21.15.27:0/2570030321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefb80b4300 0x7fefb80cf940 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fefb80b5150 tx=0x7fefa801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b40bb1b00775199 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:13.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.045+0000 7fefb7400640 1 -- 172.21.15.27:0/2570030321 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa80a89f0 con 0x7fefb80b4300 2024-01-15T18:15:13.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.045+0000 7fefb7400640 1 -- 172.21.15.27:0/2570030321 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefa80a8b50 con 0x7fefb80b4300 2024-01-15T18:15:13.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.045+0000 7fefb7400640 1 -- 172.21.15.27:0/2570030321 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa80b0440 con 0x7fefb80b4300 2024-01-15T18:15:13.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.045+0000 7fefbfc00640 1 -- 172.21.15.27:0/2570030321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefb80b4300 msgr2=0x7fefb80cf940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:13.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.045+0000 7fefbfc00640 1 --2- 172.21.15.27:0/2570030321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefb80b4300 0x7fefb80cf940 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fefb80b5150 tx=0x7fefa801ecf0 comp rx=0 tx=0).stop 2024-01-15T18:15:13.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.045+0000 7fefbfc00640 1 -- 172.21.15.27:0/2570030321 shutdown_connections 2024-01-15T18:15:13.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.045+0000 7fefbfc00640 1 --2- 172.21.15.27:0/2570030321 >> v2:172.21.15.93:3300/0 conn(0x7fefb815be40 0x7fefb80720a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:13.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.045+0000 7fefbfc00640 1 --2- 172.21.15.27:0/2570030321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefb80b4300 0x7fefb80cf940 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:13.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.045+0000 7fefbfc00640 1 -- 172.21.15.27:0/2570030321 >> 172.21.15.27:0/2570030321 conn(0x7fefb8099e10 msgr2=0x7fefb80b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:13.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.045+0000 7fefbfc00640 1 -- 172.21.15.27:0/2570030321 shutdown_connections 2024-01-15T18:15:13.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.045+0000 7fefbfc00640 1 -- 172.21.15.27:0/2570030321 wait complete. 2024-01-15T18:15:13.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.045+0000 7fefbfc00640 1 Processor -- start 2024-01-15T18:15:13.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.045+0000 7fefbfc00640 1 -- start start 2024-01-15T18:15:13.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.045+0000 7fefbfc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefb8071c50 0x7fefb80cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:13.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.045+0000 7fefbfc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fefb80b4300 0x7fefb80c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:13.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.045+0000 7fefbfc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fefb815be40 0x7fefb8145520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:13.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.045+0000 7fefbfc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb81550d0 con 0x7fefb8071c50 2024-01-15T18:15:13.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.045+0000 7fefbfc00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb8146700 con 0x7fefb80b4300 2024-01-15T18:15:13.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.045+0000 7fefbfc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb8145dd0 con 0x7fefb815be40 2024-01-15T18:15:13.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.049+0000 7fefbd200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefb8071c50 0x7fefb80cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:13.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.049+0000 7fefbd200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefb8071c50 0x7fefb80cc1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34302/0 (socket says 172.21.15.27:34302) 2024-01-15T18:15:13.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.049+0000 7fefbd200640 1 -- 172.21.15.27:0/3089265028 learned_addr learned my addr 172.21.15.27:0/3089265028 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:13.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.049+0000 7fefb7e00640 1 --2- 172.21.15.27:0/3089265028 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fefb80b4300 0x7fefb80c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:13.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.049+0000 7fefbe600640 1 --2- 172.21.15.27:0/3089265028 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fefb815be40 0x7fefb8145520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:13.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.049+0000 7fefbd200640 1 -- 172.21.15.27:0/3089265028 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fefb815be40 msgr2=0x7fefb8145520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:13.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.049+0000 7fefbd200640 1 --2- 172.21.15.27:0/3089265028 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fefb815be40 0x7fefb8145520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:13.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.049+0000 7fefbd200640 1 -- 172.21.15.27:0/3089265028 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fefb80b4300 msgr2=0x7fefb80c99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:13.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.049+0000 7fefbd200640 1 --2- 172.21.15.27:0/3089265028 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fefb80b4300 0x7fefb80c99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:13.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.049+0000 7fefbd200640 1 -- 172.21.15.27:0/3089265028 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa80094d0 con 0x7fefb8071c50 2024-01-15T18:15:13.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.049+0000 7fefbd200640 1 --2- 172.21.15.27:0/3089265028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefb8071c50 0x7fefb80cc1a0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fefa80b3040 tx=0x7fefa801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:13.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.049+0000 7fefb4c00640 1 -- 172.21.15.27:0/3089265028 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa80ba600 con 0x7fefb8071c50 2024-01-15T18:15:13.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.049+0000 7fefb4c00640 1 -- 172.21.15.27:0/3089265028 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefa80ba760 con 0x7fefb8071c50 2024-01-15T18:15:13.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.049+0000 7fefbfc00640 1 -- 172.21.15.27:0/3089265028 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb81460b0 con 0x7fefb8071c50 2024-01-15T18:15:13.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.049+0000 7fefb4c00640 1 -- 172.21.15.27:0/3089265028 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa80bab10 con 0x7fefb8071c50 2024-01-15T18:15:13.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.049+0000 7fefbfc00640 1 -- 172.21.15.27:0/3089265028 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb8146b90 con 0x7fefb8071c50 2024-01-15T18:15:13.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.049+0000 7fefbfc00640 1 -- 172.21.15.27:0/3089265028 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefb809e950 con 0x7fefb8071c50 2024-01-15T18:15:13.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.049+0000 7fefb4c00640 1 -- 172.21.15.27:0/3089265028 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7fefa80bac70 con 0x7fefb8071c50 2024-01-15T18:15:13.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.053+0000 7fefb4c00640 1 --2- 172.21.15.27:0/3089265028 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fef90067460 0x7fef90069920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:13.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.053+0000 7fefb4c00640 1 -- 172.21.15.27:0/3089265028 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fefa81212d0 con 0x7fefb8071c50 2024-01-15T18:15:13.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.053+0000 7fefb7e00640 1 --2- 172.21.15.27:0/3089265028 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fef90067460 0x7fef90069920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:13.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.053+0000 7fefb7e00640 1 --2- 172.21.15.27:0/3089265028 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fef90067460 0x7fef90069920 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fefb809e8f0 tx=0x7fefa0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:13.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.057+0000 7fefb4c00640 1 -- 172.21.15.27:0/3089265028 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fefa80eff80 con 0x7fefb8071c50 2024-01-15T18:15:13.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.249+0000 7fefbfc00640 1 -- 172.21.15.27:0/3089265028 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone51939", "target": ["mon-mgr", ""]}) v1 -- 0x7fefb80cf420 con 0x7fef90067460 2024-01-15T18:15:13.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.261+0000 7fefb4c00640 1 -- 172.21.15.27:0/3089265028 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fefa00014f0 con 0x7fef90067460 2024-01-15T18:15:13.261 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:15:13.261 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:15:13.262 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "canceled", 2024-01-15T18:15:13.262 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:15:13.262 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:15:13.262 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:15:13.262 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:15:13.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:13.262 INFO:teuthology.orchestra.run.smithi027.stdout: "failure": { 2024-01-15T18:15:13.262 INFO:teuthology.orchestra.run.smithi027.stdout: "errno": "4", 2024-01-15T18:15:13.262 INFO:teuthology.orchestra.run.smithi027.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:15:13.262 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:13.262 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:13.262 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:15:13.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.261+0000 7fefbfc00640 1 -- 172.21.15.27:0/3089265028 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fef90067460 msgr2=0x7fef90069920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:13.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.261+0000 7fefbfc00640 1 --2- 172.21.15.27:0/3089265028 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fef90067460 0x7fef90069920 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fefb809e8f0 tx=0x7fefa0009040 comp rx=0 tx=0).stop 2024-01-15T18:15:13.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.261+0000 7fefbfc00640 1 -- 172.21.15.27:0/3089265028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefb8071c50 msgr2=0x7fefb80cc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:13.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.261+0000 7fefbfc00640 1 --2- 172.21.15.27:0/3089265028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefb8071c50 0x7fefb80cc1a0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fefa80b3040 tx=0x7fefa801ba90 comp rx=0 tx=0).stop 2024-01-15T18:15:13.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.261+0000 7fefbfc00640 1 -- 172.21.15.27:0/3089265028 shutdown_connections 2024-01-15T18:15:13.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.261+0000 7fefbfc00640 1 --2- 172.21.15.27:0/3089265028 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fef90067460 0x7fef90069920 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:13.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.261+0000 7fefbfc00640 1 --2- 172.21.15.27:0/3089265028 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fefb815be40 0x7fefb8145520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:13.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.261+0000 7fefbfc00640 1 --2- 172.21.15.27:0/3089265028 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fefb80b4300 0x7fefb80c99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:13.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.261+0000 7fefbfc00640 1 --2- 172.21.15.27:0/3089265028 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefb8071c50 0x7fefb80cc1a0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:13.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.261+0000 7fefbfc00640 1 -- 172.21.15.27:0/3089265028 >> 172.21.15.27:0/3089265028 conn(0x7fefb8099e10 msgr2=0x7fefb815d0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:13.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.261+0000 7fefbfc00640 1 -- 172.21.15.27:0/3089265028 shutdown_connections 2024-01-15T18:15:13.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.261+0000 7fefbfc00640 1 -- 172.21.15.27:0/3089265028 wait complete. 2024-01-15T18:15:13.284 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:15:13.284 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone519310 2024-01-15T18:15:13.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.425+0000 7fb7e2600640 1 Processor -- start 2024-01-15T18:15:13.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.425+0000 7fb7e2600640 1 -- start start 2024-01-15T18:15:13.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.429+0000 7fb7e2600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb7dc06c430 0x7fb7dc070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:13.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.429+0000 7fb7e2600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb7dc0ce760 0x7fb7dc0b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:13.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.429+0000 7fb7e2600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7dc0ccfb0 con 0x7fb7dc0b1db0 2024-01-15T18:15:13.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.429+0000 7fb7e2600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7dc0cd290 con 0x7fb7dc0ce760 2024-01-15T18:15:13.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.429+0000 7fb7e2600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc0cd570 con 0x7fb7dc06c430 2024-01-15T18:15:13.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.429+0000 7fb7daa00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb7dc0b1db0 0x7fb7dc0ce050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54652/0 (socket says 172.21.15.27:54652) 2024-01-15T18:15:13.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.429+0000 7fb7dbe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb7dc0ce760 0x7fb7dc0b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:13.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.429+0000 7fb7daa00640 1 -- 172.21.15.27:0/1947890928 learned_addr learned my addr 172.21.15.27:0/1947890928 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:13.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.429+0000 7fb7db400640 1 --2- 172.21.15.27:0/1947890928 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb7dc06c430 0x7fb7dc070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:13.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.429+0000 7fb7da000640 1 -- 172.21.15.27:0/1947890928 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702536487 0 0) 0x7fb7dc0ccfb0 con 0x7fb7dc0b1db0 2024-01-15T18:15:13.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.429+0000 7fb7da000640 1 -- 172.21.15.27:0/1947890928 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7bc003530 con 0x7fb7dc0b1db0 2024-01-15T18:15:13.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.429+0000 7fb7dbe00640 1 -- 172.21.15.27:0/1947890928 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb7dc06c430 msgr2=0x7fb7dc070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:13.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.429+0000 7fb7dbe00640 1 --2- 172.21.15.27:0/1947890928 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb7dc06c430 0x7fb7dc070ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:13.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.429+0000 7fb7dbe00640 1 -- 172.21.15.27:0/1947890928 >> v1:172.21.15.27:6789/0 conn(0x7fb7dc0b1db0 legacy=0x7fb7dc0ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:13.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.429+0000 7fb7dbe00640 1 -- 172.21.15.27:0/1947890928 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7dc0cd850 con 0x7fb7dc0ce760 2024-01-15T18:15:13.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.433+0000 7fb7dbe00640 1 --2- 172.21.15.27:0/1947890928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7dc0ce760 0x7fb7dc0b2210 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4009d90 tx=0x7fb7d401c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7f6c6ab95028dad server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:13.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.433+0000 7fb7da000640 1 -- 172.21.15.27:0/1947890928 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d40a6710 con 0x7fb7dc0ce760 2024-01-15T18:15:13.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.433+0000 7fb7da000640 1 -- 172.21.15.27:0/1947890928 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d40a6870 con 0x7fb7dc0ce760 2024-01-15T18:15:13.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.433+0000 7fb7da000640 1 -- 172.21.15.27:0/1947890928 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d40a6c00 con 0x7fb7dc0ce760 2024-01-15T18:15:13.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.433+0000 7fb7e2600640 1 -- 172.21.15.27:0/1947890928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7dc0ce760 msgr2=0x7fb7dc0b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:13.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.433+0000 7fb7e2600640 1 --2- 172.21.15.27:0/1947890928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7dc0ce760 0x7fb7dc0b2210 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4009d90 tx=0x7fb7d401c8c0 comp rx=0 tx=0).stop 2024-01-15T18:15:13.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.433+0000 7fb7e2600640 1 -- 172.21.15.27:0/1947890928 shutdown_connections 2024-01-15T18:15:13.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.433+0000 7fb7e2600640 1 --2- 172.21.15.27:0/1947890928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7dc0ce760 0x7fb7dc0b2210 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:13.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.433+0000 7fb7e2600640 1 --2- 172.21.15.27:0/1947890928 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb7dc06c430 0x7fb7dc070ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:13.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.433+0000 7fb7e2600640 1 -- 172.21.15.27:0/1947890928 >> 172.21.15.27:0/1947890928 conn(0x7fb7dc0cb700 msgr2=0x7fb7dc0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:13.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.433+0000 7fb7e2600640 1 -- 172.21.15.27:0/1947890928 shutdown_connections 2024-01-15T18:15:13.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.433+0000 7fb7e2600640 1 -- 172.21.15.27:0/1947890928 wait complete. 2024-01-15T18:15:13.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.433+0000 7fb7e2600640 1 Processor -- start 2024-01-15T18:15:13.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.433+0000 7fb7e2600640 1 -- start start 2024-01-15T18:15:13.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.437+0000 7fb7e2600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7dc06c430 0x7fb7dc14d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:13.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.437+0000 7fb7e2600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb7dc0b1db0 0x7fb7dc14ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:13.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.437+0000 7fb7e2600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb7dc0ce760 0x7fb7dc0ae920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:13.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.437+0000 7fb7e2600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc14f7b0 con 0x7fb7dc06c430 2024-01-15T18:15:13.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.437+0000 7fb7e2600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc14fa90 con 0x7fb7dc0ce760 2024-01-15T18:15:13.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.437+0000 7fb7e2600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc14fd70 con 0x7fb7dc0b1db0 2024-01-15T18:15:13.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.437+0000 7fb7db400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7dc06c430 0x7fb7dc14d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:13.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.437+0000 7fb7db400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7dc06c430 0x7fb7dc14d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34316/0 (socket says 172.21.15.27:34316) 2024-01-15T18:15:13.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.437+0000 7fb7db400640 1 -- 172.21.15.27:0/1072191472 learned_addr learned my addr 172.21.15.27:0/1072191472 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:13.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.437+0000 7fb7dbe00640 1 --2- 172.21.15.27:0/1072191472 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb7dc0ce760 0x7fb7dc0ae920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:13.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.437+0000 7fb7daa00640 1 --2- 172.21.15.27:0/1072191472 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb7dc0b1db0 0x7fb7dc14ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:13.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.437+0000 7fb7db400640 1 -- 172.21.15.27:0/1072191472 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb7dc0b1db0 msgr2=0x7fb7dc14ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:13.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.437+0000 7fb7db400640 1 --2- 172.21.15.27:0/1072191472 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb7dc0b1db0 0x7fb7dc14ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:13.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.437+0000 7fb7db400640 1 -- 172.21.15.27:0/1072191472 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb7dc0ce760 msgr2=0x7fb7dc0ae920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:13.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.437+0000 7fb7db400640 1 --2- 172.21.15.27:0/1072191472 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb7dc0ce760 0x7fb7dc0ae920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:13.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.437+0000 7fb7db400640 1 -- 172.21.15.27:0/1072191472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d4009a70 con 0x7fb7dc06c430 2024-01-15T18:15:13.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.437+0000 7fb7db400640 1 --2- 172.21.15.27:0/1072191472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7dc06c430 0x7fb7dc14d6b0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00e930 tx=0x7fb7cc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.437+0000 7fb7d3400640 1 -- 172.21.15.27:0/1072191472 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc009690 con 0x7fb7dc06c430 2024-01-15T18:15:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.437+0000 7fb7d3400640 1 -- 172.21.15.27:0/1072191472 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7cc00ca00 con 0x7fb7dc06c430 2024-01-15T18:15:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.437+0000 7fb7d3400640 1 -- 172.21.15.27:0/1072191472 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc00cd70 con 0x7fb7dc06c430 2024-01-15T18:15:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.437+0000 7fb7e2600640 1 -- 172.21.15.27:0/1072191472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7dc0712f0 con 0x7fb7dc06c430 2024-01-15T18:15:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.437+0000 7fb7e2600640 1 -- 172.21.15.27:0/1072191472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7dc1571c0 con 0x7fb7dc06c430 2024-01-15T18:15:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.437+0000 7fb7e2600640 1 -- 172.21.15.27:0/1072191472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7dc04e850 con 0x7fb7dc06c430 2024-01-15T18:15:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.437+0000 7fb7d3400640 1 -- 172.21.15.27:0/1072191472 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7fb7cc013470 con 0x7fb7dc06c430 2024-01-15T18:15:13.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.441+0000 7fb7d3400640 1 --2- 172.21.15.27:0/1072191472 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb7b4067430 0x7fb7b40698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:13.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.441+0000 7fb7d3400640 1 -- 172.21.15.27:0/1072191472 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fb7cc0831d0 con 0x7fb7dc06c430 2024-01-15T18:15:13.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.441+0000 7fb7daa00640 1 --2- 172.21.15.27:0/1072191472 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb7b4067430 0x7fb7b40698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:13.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.441+0000 7fb7daa00640 1 --2- 172.21.15.27:0/1072191472 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb7b4067430 0x7fb7b40698f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc0b2c90 tx=0x7fb7c4007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:13.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.445+0000 7fb7d3400640 1 -- 172.21.15.27:0/1072191472 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb7cc051d80 con 0x7fb7dc06c430 2024-01-15T18:15:13.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.633+0000 7fb7e2600640 1 -- 172.21.15.27:0/1072191472 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone519310", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7dc15ced0 con 0x7fb7b4067430 2024-01-15T18:15:13.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.653+0000 7fb7d3400640 1 -- 172.21.15.27:0/1072191472 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fb7dc15ced0 con 0x7fb7b4067430 2024-01-15T18:15:13.655 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:15:13.655 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:15:13.656 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "canceled", 2024-01-15T18:15:13.656 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:15:13.656 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:15:13.656 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:15:13.656 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:15:13.656 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:13.656 INFO:teuthology.orchestra.run.smithi027.stdout: "failure": { 2024-01-15T18:15:13.656 INFO:teuthology.orchestra.run.smithi027.stdout: "errno": "4", 2024-01-15T18:15:13.656 INFO:teuthology.orchestra.run.smithi027.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:15:13.656 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:13.656 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:13.656 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:15:13.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.657+0000 7fb7e2600640 1 -- 172.21.15.27:0/1072191472 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb7b4067430 msgr2=0x7fb7b40698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:13.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.657+0000 7fb7e2600640 1 --2- 172.21.15.27:0/1072191472 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb7b4067430 0x7fb7b40698f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc0b2c90 tx=0x7fb7c4007d40 comp rx=0 tx=0).stop 2024-01-15T18:15:13.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.657+0000 7fb7e2600640 1 -- 172.21.15.27:0/1072191472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7dc06c430 msgr2=0x7fb7dc14d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:13.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.657+0000 7fb7e2600640 1 --2- 172.21.15.27:0/1072191472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7dc06c430 0x7fb7dc14d6b0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00e930 tx=0x7fb7cc00edf0 comp rx=0 tx=0).stop 2024-01-15T18:15:13.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.657+0000 7fb7e2600640 1 -- 172.21.15.27:0/1072191472 shutdown_connections 2024-01-15T18:15:13.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.657+0000 7fb7e2600640 1 --2- 172.21.15.27:0/1072191472 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb7b4067430 0x7fb7b40698f0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:13.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.657+0000 7fb7e2600640 1 --2- 172.21.15.27:0/1072191472 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb7dc0ce760 0x7fb7dc0ae920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:13.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.657+0000 7fb7e2600640 1 --2- 172.21.15.27:0/1072191472 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb7dc0b1db0 0x7fb7dc14ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:13.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.657+0000 7fb7e2600640 1 --2- 172.21.15.27:0/1072191472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7dc06c430 0x7fb7dc14d6b0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:13.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.657+0000 7fb7e2600640 1 -- 172.21.15.27:0/1072191472 >> 172.21.15.27:0/1072191472 conn(0x7fb7dc0cb700 msgr2=0x7fb7dc0b4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:13.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.657+0000 7fb7e2600640 1 -- 172.21.15.27:0/1072191472 shutdown_connections 2024-01-15T18:15:13.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.657+0000 7fb7e2600640 1 -- 172.21.15.27:0/1072191472 wait complete. 2024-01-15T18:15:13.677 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:15:13.678 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone519311 2024-01-15T18:15:13.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.821+0000 7f3221600640 1 Processor -- start 2024-01-15T18:15:13.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.821+0000 7f3221600640 1 -- start start 2024-01-15T18:15:13.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.821+0000 7f3221600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f321c0cf460 0x7f321c0cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:13.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.821+0000 7f3221600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f321c0cfd80 con 0x7f321c15b4b0 2024-01-15T18:15:13.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.821+0000 7f3221600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f321c0d19a0 con 0x7f321c0b2590 2024-01-15T18:15:13.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.821+0000 7f3221600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f321c0d1c80 con 0x7f321c0cf460 2024-01-15T18:15:13.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.821+0000 7f321aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f321c0cf460 0x7f321c0cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:13.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.821+0000 7f321a000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f321c15b4b0 0x7f321c0cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60748/0 (socket says 172.21.15.27:60748) 2024-01-15T18:15:13.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.821+0000 7f321a000640 1 -- 172.21.15.27:0/3493726363 learned_addr learned my addr 172.21.15.27:0/3493726363 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:13.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.821+0000 7f321aa00640 1 -- 172.21.15.27:0/3493726363 >> v1:172.21.15.93:6789/0 conn(0x7f321c0b2590 legacy=0x7f321c0cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:13.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.821+0000 7f321aa00640 1 -- 172.21.15.27:0/3493726363 >> v1:172.21.15.27:6789/0 conn(0x7f321c15b4b0 legacy=0x7f321c0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:13.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.821+0000 7f321aa00640 1 -- 172.21.15.27:0/3493726363 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f321c0b07c0 con 0x7f321c0cf460 2024-01-15T18:15:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.825+0000 7f321aa00640 1 --2- 172.21.15.27:0/3493726363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f321c0cf460 0x7f321c0cf840 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f320c0097c0 tx=0x7f320c01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a04b2e0cce4e66b9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.825+0000 7f3219600640 1 -- 172.21.15.27:0/3493726363 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f320c0ab640 con 0x7f321c0cf460 2024-01-15T18:15:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.825+0000 7f3219600640 1 -- 172.21.15.27:0/3493726363 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f320c0a7070 con 0x7f321c0cf460 2024-01-15T18:15:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.825+0000 7f3219600640 1 -- 172.21.15.27:0/3493726363 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f320c0ab9c0 con 0x7f321c0cf460 2024-01-15T18:15:13.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.825+0000 7f3221600640 1 -- 172.21.15.27:0/3493726363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f321c0cf460 msgr2=0x7f321c0cf840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:13.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.825+0000 7f3221600640 1 --2- 172.21.15.27:0/3493726363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f321c0cf460 0x7f321c0cf840 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f320c0097c0 tx=0x7f320c01c440 comp rx=0 tx=0).stop 2024-01-15T18:15:13.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.825+0000 7f3221600640 1 -- 172.21.15.27:0/3493726363 shutdown_connections 2024-01-15T18:15:13.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.825+0000 7f3221600640 1 --2- 172.21.15.27:0/3493726363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f321c0cf460 0x7f321c0cf840 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:13.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.825+0000 7f3221600640 1 -- 172.21.15.27:0/3493726363 >> 172.21.15.27:0/3493726363 conn(0x7f321c0d0e80 msgr2=0x7f321c0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:13.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.825+0000 7f3221600640 1 -- 172.21.15.27:0/3493726363 shutdown_connections 2024-01-15T18:15:13.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.825+0000 7f3221600640 1 -- 172.21.15.27:0/3493726363 wait complete. 2024-01-15T18:15:13.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.829+0000 7f3221600640 1 Processor -- start 2024-01-15T18:15:13.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.829+0000 7f3221600640 1 -- start start 2024-01-15T18:15:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.829+0000 7f3221600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f321c0b2590 0x7f321c0af0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.829+0000 7f3221600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f321c0cf460 0x7f321c15c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.829+0000 7f3221600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f321c15b4b0 0x7f321c15cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.829+0000 7f3221600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321c15d420 con 0x7f321c0cf460 2024-01-15T18:15:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.829+0000 7f3221600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321c0ae030 con 0x7f321c0b2590 2024-01-15T18:15:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.829+0000 7f3221600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f321c0ae310 con 0x7f321c15b4b0 2024-01-15T18:15:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.829+0000 7f321a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f321c0cf460 0x7f321c15c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.829+0000 7f321a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f321c0cf460 0x7f321c15c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51222/0 (socket says 172.21.15.27:51222) 2024-01-15T18:15:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.829+0000 7f321a000640 1 -- 172.21.15.27:0/917915277 learned_addr learned my addr 172.21.15.27:0/917915277 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.829+0000 7f321aa00640 1 --2- 172.21.15.27:0/917915277 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f321c0b2590 0x7f321c0af0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.829+0000 7f321a000640 1 -- 172.21.15.27:0/917915277 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f321c15b4b0 msgr2=0x7f321c15cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.829+0000 7f321b400640 1 --2- 172.21.15.27:0/917915277 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f321c15b4b0 0x7f321c15cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.829+0000 7f321a000640 1 --2- 172.21.15.27:0/917915277 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f321c15b4b0 0x7f321c15cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.829+0000 7f321a000640 1 -- 172.21.15.27:0/917915277 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f321c0b2590 msgr2=0x7f321c0af0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.829+0000 7f321a000640 1 --2- 172.21.15.27:0/917915277 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f321c0b2590 0x7f321c0af0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.829+0000 7f321a000640 1 -- 172.21.15.27:0/917915277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f320c009420 con 0x7f321c0cf460 2024-01-15T18:15:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.829+0000 7f321aa00640 1 --2- 172.21.15.27:0/917915277 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f321c0b2590 0x7f321c0af0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.829+0000 7f321b400640 1 --2- 172.21.15.27:0/917915277 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f321c15b4b0 0x7f321c15cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.829+0000 7f321a000640 1 --2- 172.21.15.27:0/917915277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f321c0cf460 0x7f321c15c9a0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f320400c400 tx=0x7f320400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.829+0000 7f3212a00640 1 -- 172.21.15.27:0/917915277 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3204005cf0 con 0x7f321c0cf460 2024-01-15T18:15:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.829+0000 7f3212a00640 1 -- 172.21.15.27:0/917915277 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3204005e50 con 0x7f321c0cf460 2024-01-15T18:15:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.829+0000 7f3212a00640 1 -- 172.21.15.27:0/917915277 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32040072d0 con 0x7f321c0cf460 2024-01-15T18:15:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.829+0000 7f3221600640 1 -- 172.21.15.27:0/917915277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f321c0ae5f0 con 0x7f321c0cf460 2024-01-15T18:15:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.833+0000 7f3221600640 1 -- 172.21.15.27:0/917915277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f321c0aebb0 con 0x7f321c0cf460 2024-01-15T18:15:13.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.833+0000 7f3221600640 1 -- 172.21.15.27:0/917915277 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f321c0aedf0 con 0x7f321c0cf460 2024-01-15T18:15:13.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.833+0000 7f3212a00640 1 -- 172.21.15.27:0/917915277 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7f3204007430 con 0x7f321c0cf460 2024-01-15T18:15:13.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.833+0000 7f3212a00640 1 --2- 172.21.15.27:0/917915277 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f31f4067430 0x7f31f40698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:13.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.833+0000 7f3212a00640 1 -- 172.21.15.27:0/917915277 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f3204082490 con 0x7f321c0cf460 2024-01-15T18:15:13.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.833+0000 7f321aa00640 1 --2- 172.21.15.27:0/917915277 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f31f4067430 0x7f31f40698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:13.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.833+0000 7f321aa00640 1 --2- 172.21.15.27:0/917915277 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f31f4067430 0x7f31f40698f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f320c0af000 tx=0x7f320c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:13.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:13.837+0000 7f3212a00640 1 -- 172.21.15.27:0/917915277 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f321c0aedf0 con 0x7f321c0cf460 2024-01-15T18:15:14.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.029+0000 7f3221600640 1 -- 172.21.15.27:0/917915277 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone519311", "target": ["mon-mgr", ""]}) v1 -- 0x7f321c0cc840 con 0x7f31f4067430 2024-01-15T18:15:14.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.041+0000 7f3212a00640 1 -- 172.21.15.27:0/917915277 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f320c0bc0c0 con 0x7f31f4067430 2024-01-15T18:15:14.043 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:15:14.043 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:15:14.043 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "canceled", 2024-01-15T18:15:14.043 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:15:14.043 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:15:14.043 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:15:14.043 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:15:14.044 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:14.044 INFO:teuthology.orchestra.run.smithi027.stdout: "failure": { 2024-01-15T18:15:14.044 INFO:teuthology.orchestra.run.smithi027.stdout: "errno": "4", 2024-01-15T18:15:14.044 INFO:teuthology.orchestra.run.smithi027.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:15:14.044 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:14.044 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:14.044 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:15:14.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.045+0000 7f3221600640 1 -- 172.21.15.27:0/917915277 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f31f4067430 msgr2=0x7f31f40698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:14.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.045+0000 7f3221600640 1 --2- 172.21.15.27:0/917915277 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f31f4067430 0x7f31f40698f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f320c0af000 tx=0x7f320c008040 comp rx=0 tx=0).stop 2024-01-15T18:15:14.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.045+0000 7f3221600640 1 -- 172.21.15.27:0/917915277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f321c0cf460 msgr2=0x7f321c15c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:14.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.045+0000 7f3221600640 1 --2- 172.21.15.27:0/917915277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f321c0cf460 0x7f321c15c9a0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f320400c400 tx=0x7f320400c8c0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.045+0000 7f3221600640 1 -- 172.21.15.27:0/917915277 shutdown_connections 2024-01-15T18:15:14.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.045+0000 7f3221600640 1 --2- 172.21.15.27:0/917915277 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f31f4067430 0x7f31f40698f0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.045+0000 7f3221600640 1 --2- 172.21.15.27:0/917915277 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f321c15b4b0 0x7f321c15cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.045+0000 7f3221600640 1 --2- 172.21.15.27:0/917915277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f321c0cf460 0x7f321c15c9a0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.045+0000 7f3221600640 1 --2- 172.21.15.27:0/917915277 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f321c0b2590 0x7f321c0af0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.045+0000 7f3221600640 1 -- 172.21.15.27:0/917915277 >> 172.21.15.27:0/917915277 conn(0x7f321c0d0e80 msgr2=0x7f321c06bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:14.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.045+0000 7f3221600640 1 -- 172.21.15.27:0/917915277 shutdown_connections 2024-01-15T18:15:14.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.045+0000 7f3221600640 1 -- 172.21.15.27:0/917915277 wait complete. 2024-01-15T18:15:14.065 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:15:14.066 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone519312 2024-01-15T18:15:14.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.209+0000 7f078fa00640 1 Processor -- start 2024-01-15T18:15:14.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.209+0000 7f078fa00640 1 -- start start 2024-01-15T18:15:14.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.213+0000 7f078fa00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f078815ad20 0x7f078815b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:14.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.213+0000 7f078fa00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f07880715f0 0x7f078815dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:14.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.213+0000 7f078fa00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f078815a610 con 0x7f078815d900 2024-01-15T18:15:14.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.213+0000 7f078fa00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f078815e3c0 con 0x7f07880715f0 2024-01-15T18:15:14.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.213+0000 7f078fa00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f078815e6a0 con 0x7f078815ad20 2024-01-15T18:15:14.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.213+0000 7f078da00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f07880715f0 0x7f078815dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:14.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.213+0000 7f078d000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f078815ad20 0x7f078815b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:14.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.213+0000 7f0787e00640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7f078815d900 0x7f0788070ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:48566/0 (socket says 172.21.15.27:48566) 2024-01-15T18:15:14.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.213+0000 7f0787e00640 1 -- 172.21.15.27:0/3089829082 learned_addr learned my addr 172.21.15.27:0/3089829082 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:14.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.213+0000 7f0787400640 1 -- 172.21.15.27:0/3089829082 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002080614 0 0) 0x7f078815a610 con 0x7f078815d900 2024-01-15T18:15:14.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.213+0000 7f0787400640 1 -- 172.21.15.27:0/3089829082 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0768003530 con 0x7f078815d900 2024-01-15T18:15:14.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.213+0000 7f078d000640 1 -- 172.21.15.27:0/3089829082 >> v2:172.21.15.93:3300/0 conn(0x7f07880715f0 msgr2=0x7f078815dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:14.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.213+0000 7f078d000640 1 --2- 172.21.15.27:0/3089829082 >> v2:172.21.15.93:3300/0 conn(0x7f07880715f0 0x7f078815dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.213+0000 7f078d000640 1 -- 172.21.15.27:0/3089829082 >> v1:172.21.15.93:6789/0 conn(0x7f078815d900 legacy=0x7f0788070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:14.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.213+0000 7f078d000640 1 -- 172.21.15.27:0/3089829082 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f078815b6e0 con 0x7f078815ad20 2024-01-15T18:15:14.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.213+0000 7f078da00640 1 --2- 172.21.15.27:0/3089829082 >> v2:172.21.15.93:3300/0 conn(0x7f07880715f0 0x7f078815dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:15:14.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.217+0000 7f078d000640 1 --2- 172.21.15.27:0/3089829082 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f078815ad20 0x7f078815b100 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f07780097c0 tx=0x7f077801c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc65721a1eb9fdb9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:14.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.217+0000 7f0787400640 1 -- 172.21.15.27:0/3089829082 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07780ab670 con 0x7f078815ad20 2024-01-15T18:15:14.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.217+0000 7f0787400640 1 -- 172.21.15.27:0/3089829082 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07780ab7d0 con 0x7f078815ad20 2024-01-15T18:15:14.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.217+0000 7f0787400640 1 -- 172.21.15.27:0/3089829082 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07780abb60 con 0x7f078815ad20 2024-01-15T18:15:14.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.217+0000 7f078fa00640 1 -- 172.21.15.27:0/3089829082 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f078815ad20 msgr2=0x7f078815b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:14.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.217+0000 7f078fa00640 1 --2- 172.21.15.27:0/3089829082 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f078815ad20 0x7f078815b100 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f07780097c0 tx=0x7f077801c450 comp rx=0 tx=0).stop 2024-01-15T18:15:14.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.217+0000 7f078fa00640 1 -- 172.21.15.27:0/3089829082 shutdown_connections 2024-01-15T18:15:14.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.217+0000 7f078fa00640 1 --2- 172.21.15.27:0/3089829082 >> v2:172.21.15.93:3300/0 conn(0x7f07880715f0 0x7f078815dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.217+0000 7f078fa00640 1 --2- 172.21.15.27:0/3089829082 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f078815ad20 0x7f078815b100 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.217+0000 7f078fa00640 1 -- 172.21.15.27:0/3089829082 >> 172.21.15.27:0/3089829082 conn(0x7f07880cb700 msgr2=0x7f07880cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:14.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.217+0000 7f078fa00640 1 -- 172.21.15.27:0/3089829082 shutdown_connections 2024-01-15T18:15:14.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.217+0000 7f078fa00640 1 -- 172.21.15.27:0/3089829082 wait complete. 2024-01-15T18:15:14.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.217+0000 7f078fa00640 1 Processor -- start 2024-01-15T18:15:14.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.217+0000 7f078fa00640 1 -- start start 2024-01-15T18:15:14.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.221+0000 7f078fa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07880715f0 0x7f078814f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:14.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.221+0000 7f078fa00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f078815ad20 0x7f078814c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:14.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.221+0000 7f078fa00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f078815d900 0x7f07880ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:14.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.221+0000 7f078fa00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0788155640 con 0x7f07880715f0 2024-01-15T18:15:14.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.221+0000 7f078fa00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07880cad10 con 0x7f078815d900 2024-01-15T18:15:14.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.221+0000 7f078fa00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f078814ff20 con 0x7f078815ad20 2024-01-15T18:15:14.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.221+0000 7f078d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07880715f0 0x7f078814f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:14.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.221+0000 7f078d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07880715f0 0x7f078814f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51234/0 (socket says 172.21.15.27:51234) 2024-01-15T18:15:14.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.221+0000 7f078d000640 1 -- 172.21.15.27:0/3218136988 learned_addr learned my addr 172.21.15.27:0/3218136988 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:14.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.221+0000 7f078da00640 1 --2- 172.21.15.27:0/3218136988 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f078815d900 0x7f07880ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:14.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.221+0000 7f0787e00640 1 --2- 172.21.15.27:0/3218136988 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f078815ad20 0x7f078814c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:14.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.221+0000 7f078d000640 1 -- 172.21.15.27:0/3218136988 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f078815ad20 msgr2=0x7f078814c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:14.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.221+0000 7f078d000640 1 --2- 172.21.15.27:0/3218136988 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f078815ad20 0x7f078814c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.221+0000 7f078d000640 1 -- 172.21.15.27:0/3218136988 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f078815d900 msgr2=0x7f07880ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:14.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.221+0000 7f078d000640 1 --2- 172.21.15.27:0/3218136988 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f078815d900 0x7f07880ca7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.221+0000 7f078d000640 1 -- 172.21.15.27:0/3218136988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0780009a70 con 0x7f07880715f0 2024-01-15T18:15:14.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.221+0000 7f078d000640 1 --2- 172.21.15.27:0/3218136988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07880715f0 0x7f078814f8b0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f07780ad040 tx=0x7f077801ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:14.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.221+0000 7f0784c00640 1 -- 172.21.15.27:0/3218136988 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07780b5810 con 0x7f07880715f0 2024-01-15T18:15:14.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.221+0000 7f0784c00640 1 -- 172.21.15.27:0/3218136988 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07780b59b0 con 0x7f07880715f0 2024-01-15T18:15:14.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.221+0000 7f0784c00640 1 -- 172.21.15.27:0/3218136988 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07780b5d60 con 0x7f07880715f0 2024-01-15T18:15:14.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.221+0000 7f078fa00640 1 -- 172.21.15.27:0/3218136988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0778009420 con 0x7f07880715f0 2024-01-15T18:15:14.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.221+0000 7f078fa00640 1 -- 172.21.15.27:0/3218136988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0788155b70 con 0x7f07880715f0 2024-01-15T18:15:14.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.221+0000 7f078fa00640 1 -- 172.21.15.27:0/3218136988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f078804e850 con 0x7f07880715f0 2024-01-15T18:15:14.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.221+0000 7f0784c00640 1 -- 172.21.15.27:0/3218136988 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7f07780c93f0 con 0x7f07880715f0 2024-01-15T18:15:14.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.225+0000 7f0784c00640 1 --2- 172.21.15.27:0/3218136988 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0760067430 0x7f07600698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:14.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.225+0000 7f0784c00640 1 -- 172.21.15.27:0/3218136988 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f077811e050 con 0x7f07880715f0 2024-01-15T18:15:14.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.225+0000 7f0787e00640 1 --2- 172.21.15.27:0/3218136988 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0760067430 0x7f07600698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:14.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.225+0000 7f0787e00640 1 --2- 172.21.15.27:0/3218136988 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0760067430 0x7f07600698f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f07880707e0 tx=0x7f0770007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:14.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.225+0000 7f0784c00640 1 -- 172.21.15.27:0/3218136988 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f07780ecdd0 con 0x7f07880715f0 2024-01-15T18:15:14.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.417+0000 7f078fa00640 1 -- 172.21.15.27:0/3218136988 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone519312", "target": ["mon-mgr", ""]}) v1 -- 0x7f078815df20 con 0x7f0760067430 2024-01-15T18:15:14.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.429+0000 7f0784c00640 1 -- 172.21.15.27:0/3218136988 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f078815df20 con 0x7f0760067430 2024-01-15T18:15:14.430 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:15:14.430 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:15:14.430 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "canceled", 2024-01-15T18:15:14.430 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:15:14.430 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:15:14.430 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:15:14.430 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:15:14.431 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:14.431 INFO:teuthology.orchestra.run.smithi027.stdout: "failure": { 2024-01-15T18:15:14.431 INFO:teuthology.orchestra.run.smithi027.stdout: "errno": "4", 2024-01-15T18:15:14.431 INFO:teuthology.orchestra.run.smithi027.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:15:14.431 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:14.431 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:14.431 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:15:14.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.429+0000 7f078fa00640 1 -- 172.21.15.27:0/3218136988 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0760067430 msgr2=0x7f07600698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:14.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.429+0000 7f078fa00640 1 --2- 172.21.15.27:0/3218136988 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0760067430 0x7f07600698f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f07880707e0 tx=0x7f0770007d40 comp rx=0 tx=0).stop 2024-01-15T18:15:14.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.429+0000 7f078fa00640 1 -- 172.21.15.27:0/3218136988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07880715f0 msgr2=0x7f078814f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:14.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.429+0000 7f078fa00640 1 --2- 172.21.15.27:0/3218136988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07880715f0 0x7f078814f8b0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f07780ad040 tx=0x7f077801ac90 comp rx=0 tx=0).stop 2024-01-15T18:15:14.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.429+0000 7f078fa00640 1 -- 172.21.15.27:0/3218136988 shutdown_connections 2024-01-15T18:15:14.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.429+0000 7f078fa00640 1 --2- 172.21.15.27:0/3218136988 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0760067430 0x7f07600698f0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.429+0000 7f078fa00640 1 --2- 172.21.15.27:0/3218136988 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f078815d900 0x7f07880ca7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.429+0000 7f078fa00640 1 --2- 172.21.15.27:0/3218136988 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f078815ad20 0x7f078814c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.429+0000 7f078fa00640 1 --2- 172.21.15.27:0/3218136988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07880715f0 0x7f078814f8b0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.429+0000 7f078fa00640 1 -- 172.21.15.27:0/3218136988 >> 172.21.15.27:0/3218136988 conn(0x7f07880cb700 msgr2=0x7f0788070ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:14.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.429+0000 7f078fa00640 1 -- 172.21.15.27:0/3218136988 shutdown_connections 2024-01-15T18:15:14.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.429+0000 7f078fa00640 1 -- 172.21.15.27:0/3218136988 wait complete. 2024-01-15T18:15:14.451 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:15:14.452 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone519313 2024-01-15T18:15:14.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.593+0000 7fd1faa00640 1 Processor -- start 2024-01-15T18:15:14.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.593+0000 7fd1faa00640 1 -- start start 2024-01-15T18:15:14.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.597+0000 7fd1faa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd1f406c430 0x7fd1f4070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:14.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.597+0000 7fd1faa00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd1f40b1db0 0x7fd1f40b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:14.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.597+0000 7fd1faa00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1f40ccfb0 con 0x7fd1f40ce050 2024-01-15T18:15:14.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.597+0000 7fd1faa00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1f40cd290 con 0x7fd1f406c430 2024-01-15T18:15:14.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.597+0000 7fd1faa00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f40cd570 con 0x7fd1f40b1db0 2024-01-15T18:15:14.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.597+0000 7fd1f3e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd1f406c430 0x7fd1f4070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:14.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.597+0000 7fd1f8a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd1f40ce050 0x7fd1f406c810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60756/0 (socket says 172.21.15.27:60756) 2024-01-15T18:15:14.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.597+0000 7fd1f8a00640 1 -- 172.21.15.27:0/3438593542 learned_addr learned my addr 172.21.15.27:0/3438593542 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:14.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.597+0000 7fd1f3400640 1 --2- 172.21.15.27:0/3438593542 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd1f40b1db0 0x7fd1f40b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:14.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.597+0000 7fd1f3e00640 1 -- 172.21.15.27:0/3438593542 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd1f40b1db0 msgr2=0x7fd1f40b2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:14.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.597+0000 7fd1f3e00640 1 --2- 172.21.15.27:0/3438593542 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd1f40b1db0 0x7fd1f40b2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.597+0000 7fd1f3e00640 1 -- 172.21.15.27:0/3438593542 >> v1:172.21.15.27:6789/0 conn(0x7fd1f40ce050 legacy=0x7fd1f406c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:14.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.597+0000 7fd1f3e00640 1 -- 172.21.15.27:0/3438593542 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f40cd850 con 0x7fd1f406c430 2024-01-15T18:15:14.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.597+0000 7fd1f2a00640 1 -- 172.21.15.27:0/3438593542 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3162029159 0 0) 0x7fd1f40ccfb0 con 0x7fd1f40ce050 2024-01-15T18:15:14.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.597+0000 7fd1f3e00640 1 --2- 172.21.15.27:0/3438593542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1f406c430 0x7fd1f4070ba0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fd1e40097c0 tx=0x7fd1e401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=769eae477d0eb916 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:14.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.601+0000 7fd1f2a00640 1 -- 172.21.15.27:0/3438593542 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e40ab600 con 0x7fd1f406c430 2024-01-15T18:15:14.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.601+0000 7fd1f2a00640 1 -- 172.21.15.27:0/3438593542 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1e40a7050 con 0x7fd1f406c430 2024-01-15T18:15:14.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.601+0000 7fd1f2a00640 1 -- 172.21.15.27:0/3438593542 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e40ab980 con 0x7fd1f406c430 2024-01-15T18:15:14.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.601+0000 7fd1faa00640 1 -- 172.21.15.27:0/3438593542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1f406c430 msgr2=0x7fd1f4070ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:14.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.601+0000 7fd1faa00640 1 --2- 172.21.15.27:0/3438593542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1f406c430 0x7fd1f4070ba0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fd1e40097c0 tx=0x7fd1e401c3f0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.601+0000 7fd1faa00640 1 -- 172.21.15.27:0/3438593542 shutdown_connections 2024-01-15T18:15:14.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.601+0000 7fd1faa00640 1 --2- 172.21.15.27:0/3438593542 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd1f40b1db0 0x7fd1f40b2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.601+0000 7fd1faa00640 1 --2- 172.21.15.27:0/3438593542 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1f406c430 0x7fd1f4070ba0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.601+0000 7fd1faa00640 1 -- 172.21.15.27:0/3438593542 >> 172.21.15.27:0/3438593542 conn(0x7fd1f40cb700 msgr2=0x7fd1f40cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:14.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.601+0000 7fd1faa00640 1 -- 172.21.15.27:0/3438593542 shutdown_connections 2024-01-15T18:15:14.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.601+0000 7fd1faa00640 1 -- 172.21.15.27:0/3438593542 wait complete. 2024-01-15T18:15:14.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.601+0000 7fd1faa00640 1 Processor -- start 2024-01-15T18:15:14.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.601+0000 7fd1faa00640 1 -- start start 2024-01-15T18:15:14.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.601+0000 7fd1faa00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd1f406c430 0x7fd1f4153cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:14.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.601+0000 7fd1faa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1f40b1db0 0x7fd1f4152370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:14.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.601+0000 7fd1faa00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1f40ce050 0x7fd1f413f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:14.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.601+0000 7fd1faa00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f4152c30 con 0x7fd1f40b1db0 2024-01-15T18:15:14.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.601+0000 7fd1faa00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f4152f10 con 0x7fd1f40ce050 2024-01-15T18:15:14.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.601+0000 7fd1faa00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f41531f0 con 0x7fd1f406c430 2024-01-15T18:15:14.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.605+0000 7fd1f3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1f40b1db0 0x7fd1f4152370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:14.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.605+0000 7fd1f3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1f40b1db0 0x7fd1f4152370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51246/0 (socket says 172.21.15.27:51246) 2024-01-15T18:15:14.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.605+0000 7fd1f3400640 1 -- 172.21.15.27:0/1957288044 learned_addr learned my addr 172.21.15.27:0/1957288044 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:14.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.605+0000 7fd1f8a00640 1 --2- 172.21.15.27:0/1957288044 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1f40ce050 0x7fd1f413f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:14.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.605+0000 7fd1f3e00640 1 --2- 172.21.15.27:0/1957288044 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd1f406c430 0x7fd1f4153cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:14.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.605+0000 7fd1f3400640 1 -- 172.21.15.27:0/1957288044 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd1f406c430 msgr2=0x7fd1f4153cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:14.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.605+0000 7fd1f3400640 1 --2- 172.21.15.27:0/1957288044 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd1f406c430 0x7fd1f4153cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.605+0000 7fd1f3400640 1 -- 172.21.15.27:0/1957288044 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1f40ce050 msgr2=0x7fd1f413f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:14.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.605+0000 7fd1f3400640 1 --2- 172.21.15.27:0/1957288044 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1f40ce050 0x7fd1f413f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.605+0000 7fd1f3400640 1 -- 172.21.15.27:0/1957288044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e4009420 con 0x7fd1f40b1db0 2024-01-15T18:15:14.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.605+0000 7fd1f3400640 1 --2- 172.21.15.27:0/1957288044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1f40b1db0 0x7fd1f4152370 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc00d5f0 tx=0x7fd1dc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:14.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.605+0000 7fd1ebe00640 1 -- 172.21.15.27:0/1957288044 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc0125f0 con 0x7fd1f40b1db0 2024-01-15T18:15:14.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.605+0000 7fd1ebe00640 1 -- 172.21.15.27:0/1957288044 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1dc012750 con 0x7fd1f40b1db0 2024-01-15T18:15:14.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.605+0000 7fd1ebe00640 1 -- 172.21.15.27:0/1957288044 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc012ae0 con 0x7fd1f40b1db0 2024-01-15T18:15:14.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.605+0000 7fd1faa00640 1 -- 172.21.15.27:0/1957288044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1f41534d0 con 0x7fd1f40b1db0 2024-01-15T18:15:14.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.605+0000 7fd1faa00640 1 -- 172.21.15.27:0/1957288044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1f41528b0 con 0x7fd1f40b1db0 2024-01-15T18:15:14.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.605+0000 7fd1faa00640 1 -- 172.21.15.27:0/1957288044 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1f40b2210 con 0x7fd1f40b1db0 2024-01-15T18:15:14.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.605+0000 7fd1ebe00640 1 -- 172.21.15.27:0/1957288044 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7fd1dc012c40 con 0x7fd1f40b1db0 2024-01-15T18:15:14.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.609+0000 7fd1ebe00640 1 --2- 172.21.15.27:0/1957288044 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd1cc067430 0x7fd1cc0698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:14.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.609+0000 7fd1ebe00640 1 -- 172.21.15.27:0/1957288044 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fd1dc0832e0 con 0x7fd1f40b1db0 2024-01-15T18:15:14.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.609+0000 7fd1f3e00640 1 --2- 172.21.15.27:0/1957288044 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd1cc067430 0x7fd1cc0698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:14.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.609+0000 7fd1f3e00640 1 --2- 172.21.15.27:0/1957288044 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd1cc067430 0x7fd1cc0698f0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fd1e40ad000 tx=0x7fd1e40a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:14.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.613+0000 7fd1ebe00640 1 -- 172.21.15.27:0/1957288044 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd1dc051f70 con 0x7fd1f40b1db0 2024-01-15T18:15:14.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.801+0000 7fd1faa00640 1 -- 172.21.15.27:0/1957288044 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone519313", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1f4153080 con 0x7fd1cc067430 2024-01-15T18:15:14.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.813+0000 7fd1ebe00640 1 -- 172.21.15.27:0/1957288044 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fd1f4153080 con 0x7fd1cc067430 2024-01-15T18:15:14.813 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:15:14.813 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:15:14.814 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "canceled", 2024-01-15T18:15:14.814 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:15:14.814 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:15:14.814 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:15:14.814 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:15:14.814 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:14.814 INFO:teuthology.orchestra.run.smithi027.stdout: "failure": { 2024-01-15T18:15:14.814 INFO:teuthology.orchestra.run.smithi027.stdout: "errno": "4", 2024-01-15T18:15:14.814 INFO:teuthology.orchestra.run.smithi027.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:15:14.814 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:14.814 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:14.814 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:15:14.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.813+0000 7fd1faa00640 1 -- 172.21.15.27:0/1957288044 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd1cc067430 msgr2=0x7fd1cc0698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:14.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.813+0000 7fd1faa00640 1 --2- 172.21.15.27:0/1957288044 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd1cc067430 0x7fd1cc0698f0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fd1e40ad000 tx=0x7fd1e40a6000 comp rx=0 tx=0).stop 2024-01-15T18:15:14.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.813+0000 7fd1faa00640 1 -- 172.21.15.27:0/1957288044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1f40b1db0 msgr2=0x7fd1f4152370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:14.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.813+0000 7fd1faa00640 1 --2- 172.21.15.27:0/1957288044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1f40b1db0 0x7fd1f4152370 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc00d5f0 tx=0x7fd1dc00dab0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.813+0000 7fd1faa00640 1 -- 172.21.15.27:0/1957288044 shutdown_connections 2024-01-15T18:15:14.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.813+0000 7fd1faa00640 1 --2- 172.21.15.27:0/1957288044 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd1cc067430 0x7fd1cc0698f0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.813+0000 7fd1faa00640 1 --2- 172.21.15.27:0/1957288044 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1f40ce050 0x7fd1f413f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.813+0000 7fd1faa00640 1 --2- 172.21.15.27:0/1957288044 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1f40b1db0 0x7fd1f4152370 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.813+0000 7fd1faa00640 1 --2- 172.21.15.27:0/1957288044 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd1f406c430 0x7fd1f4153cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.813+0000 7fd1faa00640 1 -- 172.21.15.27:0/1957288044 >> 172.21.15.27:0/1957288044 conn(0x7fd1f40cb700 msgr2=0x7fd1f40cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:14.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.813+0000 7fd1faa00640 1 -- 172.21.15.27:0/1957288044 shutdown_connections 2024-01-15T18:15:14.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.813+0000 7fd1faa00640 1 -- 172.21.15.27:0/1957288044 wait complete. 2024-01-15T18:15:14.836 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:15:14.836 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone519314 2024-01-15T18:15:14.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.977+0000 7f7ad3200640 1 Processor -- start 2024-01-15T18:15:14.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.977+0000 7f7ad3200640 1 -- start start 2024-01-15T18:15:14.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.981+0000 7f7ad3200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7acc158900 0x7f7acc15f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:14.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.981+0000 7f7ad3200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f7acc15bf00 0x7f7acc0cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.981+0000 7f7ad3200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7acc0cf110 0x7f7acc0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.981+0000 7f7ad3200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7acc0cf580 con 0x7f7acc158900 2024-01-15T18:15:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.981+0000 7f7ad3200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7acc0b1cb0 con 0x7f7acc15bf00 2024-01-15T18:15:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.981+0000 7f7ad3200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc0b1f00 con 0x7f7acc0cf110 2024-01-15T18:15:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.981+0000 7f7acbe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7acc158900 0x7f7acc15f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.981+0000 7f7acbe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7acc158900 0x7f7acc15f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51254/0 (socket says 172.21.15.27:51254) 2024-01-15T18:15:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.981+0000 7f7acbe00640 1 -- 172.21.15.27:0/2763042464 learned_addr learned my addr 172.21.15.27:0/2763042464 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.981+0000 7f7acb400640 1 --2- 172.21.15.27:0/2763042464 >> v2:172.21.15.93:3300/0 conn(0x7f7acc15bf00 0x7f7acc0cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.981+0000 7f7ad1200640 1 --2- 172.21.15.27:0/2763042464 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7acc0cf110 0x7f7acc0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.981+0000 7f7acbe00640 1 -- 172.21.15.27:0/2763042464 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7acc0cf110 msgr2=0x7f7acc0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.981+0000 7f7acbe00640 1 --2- 172.21.15.27:0/2763042464 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7acc0cf110 0x7f7acc0701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.981+0000 7f7acbe00640 1 -- 172.21.15.27:0/2763042464 >> v2:172.21.15.93:3300/0 conn(0x7f7acc15bf00 msgr2=0x7f7acc0cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.981+0000 7f7acbe00640 1 --2- 172.21.15.27:0/2763042464 >> v2:172.21.15.93:3300/0 conn(0x7f7acc15bf00 0x7f7acc0cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.981+0000 7f7acbe00640 1 -- 172.21.15.27:0/2763042464 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7acc0b21e0 con 0x7f7acc158900 2024-01-15T18:15:14.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.981+0000 7f7ad1200640 1 --2- 172.21.15.27:0/2763042464 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7acc0cf110 0x7f7acc0701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:14.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.985+0000 7f7acbe00640 1 --2- 172.21.15.27:0/2763042464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7acc158900 0x7f7acc15f550 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f7acc072050 tx=0x7f7abc01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed73105f84a75fce server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:14.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.985+0000 7f7acaa00640 1 -- 172.21.15.27:0/2763042464 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7acc070f30 con 0x7f7acc158900 2024-01-15T18:15:14.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.985+0000 7f7acaa00640 1 -- 172.21.15.27:0/2763042464 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7abc0ac410 con 0x7f7acc158900 2024-01-15T18:15:14.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.985+0000 7f7acaa00640 1 -- 172.21.15.27:0/2763042464 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc0ac7a0 con 0x7f7acc158900 2024-01-15T18:15:14.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.985+0000 7f7ad3200640 1 -- 172.21.15.27:0/2763042464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7acc158900 msgr2=0x7f7acc15f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:14.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.985+0000 7f7ad3200640 1 --2- 172.21.15.27:0/2763042464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7acc158900 0x7f7acc15f550 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f7acc072050 tx=0x7f7abc01c500 comp rx=0 tx=0).stop 2024-01-15T18:15:14.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.985+0000 7f7ad3200640 1 -- 172.21.15.27:0/2763042464 shutdown_connections 2024-01-15T18:15:14.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.985+0000 7f7ad3200640 1 --2- 172.21.15.27:0/2763042464 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7acc0cf110 0x7f7acc0701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.985+0000 7f7ad3200640 1 --2- 172.21.15.27:0/2763042464 >> v2:172.21.15.93:3300/0 conn(0x7f7acc15bf00 0x7f7acc0cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.985+0000 7f7ad3200640 1 --2- 172.21.15.27:0/2763042464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7acc158900 0x7f7acc15f550 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.985+0000 7f7ad3200640 1 -- 172.21.15.27:0/2763042464 >> 172.21.15.27:0/2763042464 conn(0x7f7acc0cb700 msgr2=0x7f7acc0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:14.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.985+0000 7f7ad3200640 1 -- 172.21.15.27:0/2763042464 shutdown_connections 2024-01-15T18:15:14.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.985+0000 7f7ad3200640 1 -- 172.21.15.27:0/2763042464 wait complete. 2024-01-15T18:15:14.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.985+0000 7f7ad3200640 1 Processor -- start 2024-01-15T18:15:14.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.985+0000 7f7ad3200640 1 -- start start 2024-01-15T18:15:14.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.989+0000 7f7ad3200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7acc0cf110 0x7f7acc0afec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:14.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.989+0000 7f7ad3200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7acc158900 0x7f7acc149670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:14.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.989+0000 7f7ad3200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7acc15bf00 0x7f7acc151ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:14.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.989+0000 7f7ad3200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc149bb0 con 0x7f7acc0cf110 2024-01-15T18:15:14.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.989+0000 7f7ad3200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc151a40 con 0x7f7acc158900 2024-01-15T18:15:14.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.989+0000 7f7ad3200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc0b05a0 con 0x7f7acc15bf00 2024-01-15T18:15:14.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.989+0000 7f7acbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7acc0cf110 0x7f7acc0afec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:14.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.989+0000 7f7acbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7acc0cf110 0x7f7acc0afec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51260/0 (socket says 172.21.15.27:51260) 2024-01-15T18:15:14.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.989+0000 7f7acbe00640 1 -- 172.21.15.27:0/4209156524 learned_addr learned my addr 172.21.15.27:0/4209156524 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:14.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.989+0000 7f7acb400640 1 --2- 172.21.15.27:0/4209156524 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7acc158900 0x7f7acc149670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:14.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.989+0000 7f7ad1200640 1 --2- 172.21.15.27:0/4209156524 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7acc15bf00 0x7f7acc151ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:14.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.989+0000 7f7acbe00640 1 -- 172.21.15.27:0/4209156524 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7acc15bf00 msgr2=0x7f7acc151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:14.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.989+0000 7f7acbe00640 1 --2- 172.21.15.27:0/4209156524 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7acc15bf00 0x7f7acc151ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.989+0000 7f7acbe00640 1 -- 172.21.15.27:0/4209156524 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7acc158900 msgr2=0x7f7acc149670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:14.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.989+0000 7f7acbe00640 1 --2- 172.21.15.27:0/4209156524 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7acc158900 0x7f7acc149670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:14.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.989+0000 7f7acbe00640 1 -- 172.21.15.27:0/4209156524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7abc0094d0 con 0x7f7acc0cf110 2024-01-15T18:15:14.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.989+0000 7f7ad1200640 1 --2- 172.21.15.27:0/4209156524 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7acc15bf00 0x7f7acc151ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:14.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.989+0000 7f7acb400640 1 --2- 172.21.15.27:0/4209156524 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7acc158900 0x7f7acc149670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:14.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.989+0000 7f7acbe00640 1 --2- 172.21.15.27:0/4209156524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7acc0cf110 0x7f7acc0afec0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f7abc0af040 tx=0x7f7abc0192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:14.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.989+0000 7f7ac3e00640 1 -- 172.21.15.27:0/4209156524 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc0b7470 con 0x7f7acc0cf110 2024-01-15T18:15:14.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.989+0000 7f7ac3e00640 1 -- 172.21.15.27:0/4209156524 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7abc0b75f0 con 0x7f7acc0cf110 2024-01-15T18:15:14.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.989+0000 7f7ac3e00640 1 -- 172.21.15.27:0/4209156524 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc0b79a0 con 0x7f7acc0cf110 2024-01-15T18:15:14.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.989+0000 7f7ad3200640 1 -- 172.21.15.27:0/4209156524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7acc15c490 con 0x7f7acc0cf110 2024-01-15T18:15:14.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.989+0000 7f7ad3200640 1 -- 172.21.15.27:0/4209156524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7acc1513f0 con 0x7f7acc0cf110 2024-01-15T18:15:14.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.989+0000 7f7ad3200640 1 -- 172.21.15.27:0/4209156524 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7acc04e850 con 0x7f7acc0cf110 2024-01-15T18:15:14.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.993+0000 7f7ac3e00640 1 -- 172.21.15.27:0/4209156524 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7f7abc0b3070 con 0x7f7acc0cf110 2024-01-15T18:15:14.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.993+0000 7f7ac3e00640 1 --2- 172.21.15.27:0/4209156524 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7aa4067430 0x7f7aa40698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:14.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.993+0000 7f7ac3e00640 1 -- 172.21.15.27:0/4209156524 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f7abc11d4c0 con 0x7f7acc0cf110 2024-01-15T18:15:14.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.993+0000 7f7acb400640 1 --2- 172.21.15.27:0/4209156524 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7aa4067430 0x7f7aa40698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:14.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.993+0000 7f7acb400640 1 --2- 172.21.15.27:0/4209156524 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7aa4067430 0x7f7aa40698f0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f7acc0b2e20 tx=0x7f7ab4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:14.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:14.997+0000 7f7ac3e00640 1 -- 172.21.15.27:0/4209156524 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7abc0ec150 con 0x7f7acc0cf110 2024-01-15T18:15:15.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.189+0000 7f7ad3200640 1 -- 172.21.15.27:0/4209156524 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone519314", "target": ["mon-mgr", ""]}) v1 -- 0x7f7acc15c310 con 0x7f7aa4067430 2024-01-15T18:15:15.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.201+0000 7f7ac3e00640 1 -- 172.21.15.27:0/4209156524 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f7ab4007200 con 0x7f7aa4067430 2024-01-15T18:15:15.202 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:15:15.202 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:15:15.202 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "canceled", 2024-01-15T18:15:15.202 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:15:15.202 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:15:15.202 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:15:15.202 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:15:15.203 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:15.203 INFO:teuthology.orchestra.run.smithi027.stdout: "failure": { 2024-01-15T18:15:15.203 INFO:teuthology.orchestra.run.smithi027.stdout: "errno": "4", 2024-01-15T18:15:15.203 INFO:teuthology.orchestra.run.smithi027.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:15:15.203 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:15.203 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:15.203 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:15:15.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.201+0000 7f7ad3200640 1 -- 172.21.15.27:0/4209156524 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7aa4067430 msgr2=0x7f7aa40698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:15.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.201+0000 7f7ad3200640 1 --2- 172.21.15.27:0/4209156524 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7aa4067430 0x7f7aa40698f0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f7acc0b2e20 tx=0x7f7ab4008040 comp rx=0 tx=0).stop 2024-01-15T18:15:15.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.201+0000 7f7ad3200640 1 -- 172.21.15.27:0/4209156524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7acc0cf110 msgr2=0x7f7acc0afec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:15.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.201+0000 7f7ad3200640 1 --2- 172.21.15.27:0/4209156524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7acc0cf110 0x7f7acc0afec0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f7abc0af040 tx=0x7f7abc0192a0 comp rx=0 tx=0).stop 2024-01-15T18:15:15.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.201+0000 7f7ad3200640 1 -- 172.21.15.27:0/4209156524 shutdown_connections 2024-01-15T18:15:15.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.201+0000 7f7ad3200640 1 --2- 172.21.15.27:0/4209156524 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7aa4067430 0x7f7aa40698f0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:15.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.201+0000 7f7ad3200640 1 --2- 172.21.15.27:0/4209156524 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7acc15bf00 0x7f7acc151ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:15.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.201+0000 7f7ad3200640 1 --2- 172.21.15.27:0/4209156524 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7acc158900 0x7f7acc149670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:15.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.201+0000 7f7ad3200640 1 --2- 172.21.15.27:0/4209156524 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7acc0cf110 0x7f7acc0afec0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:15.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.201+0000 7f7ad3200640 1 -- 172.21.15.27:0/4209156524 >> 172.21.15.27:0/4209156524 conn(0x7f7acc0cb700 msgr2=0x7f7acc0acda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:15.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.205+0000 7f7ad3200640 1 -- 172.21.15.27:0/4209156524 shutdown_connections 2024-01-15T18:15:15.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.205+0000 7f7ad3200640 1 -- 172.21.15.27:0/4209156524 wait complete. 2024-01-15T18:15:15.225 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:15:15.225 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone519315 2024-01-15T18:15:15.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.373+0000 7fb698000640 1 Processor -- start 2024-01-15T18:15:15.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.373+0000 7fb698000640 1 -- start start 2024-01-15T18:15:15.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.373+0000 7fb698000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fb69015ad20 0x7fb69015b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:15.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.373+0000 7fb698000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb6900715f0 0x7fb69015ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:15.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.373+0000 7fb698000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb69015a610 con 0x7fb69015d900 2024-01-15T18:15:15.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.373+0000 7fb698000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb69015e3c0 con 0x7fb6900715f0 2024-01-15T18:15:15.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.373+0000 7fb698000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb69015e6a0 con 0x7fb69015ad20 2024-01-15T18:15:15.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.373+0000 7fb696000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb6900715f0 0x7fb69015ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:15.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.377+0000 7fb696000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb6900715f0 0x7fb69015ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51266/0 (socket says 172.21.15.27:51266) 2024-01-15T18:15:15.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.377+0000 7fb696000640 1 -- 172.21.15.27:0/4127451267 learned_addr learned my addr 172.21.15.27:0/4127451267 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:15.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.377+0000 7fb695600640 1 --2- 172.21.15.27:0/4127451267 >> v2:172.21.15.93:3300/0 conn(0x7fb69015ad20 0x7fb69015b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:15.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.377+0000 7fb696000640 1 -- 172.21.15.27:0/4127451267 >> v2:172.21.15.93:3300/0 conn(0x7fb69015ad20 msgr2=0x7fb69015b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:15.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.377+0000 7fb696000640 1 --2- 172.21.15.27:0/4127451267 >> v2:172.21.15.93:3300/0 conn(0x7fb69015ad20 0x7fb69015b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:15.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.377+0000 7fb696000640 1 -- 172.21.15.27:0/4127451267 >> v1:172.21.15.93:6789/0 conn(0x7fb69015d900 legacy=0x7fb690070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:15.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.377+0000 7fb696000640 1 -- 172.21.15.27:0/4127451267 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69015b6e0 con 0x7fb6900715f0 2024-01-15T18:15:15.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.377+0000 7fb696000640 1 --2- 172.21.15.27:0/4127451267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6900715f0 0x7fb69015ddb0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fb688009e50 tx=0x7fb68801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f46d9192b94eabc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:15.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.377+0000 7fb68fe00640 1 -- 172.21.15.27:0/4127451267 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6880a5880 con 0x7fb6900715f0 2024-01-15T18:15:15.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.377+0000 7fb68fe00640 1 -- 172.21.15.27:0/4127451267 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6880a59e0 con 0x7fb6900715f0 2024-01-15T18:15:15.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.377+0000 7fb68fe00640 1 -- 172.21.15.27:0/4127451267 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6880a5d70 con 0x7fb6900715f0 2024-01-15T18:15:15.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.381+0000 7fb698000640 1 -- 172.21.15.27:0/4127451267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6900715f0 msgr2=0x7fb69015ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:15.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.381+0000 7fb698000640 1 --2- 172.21.15.27:0/4127451267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6900715f0 0x7fb69015ddb0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fb688009e50 tx=0x7fb68801c9a0 comp rx=0 tx=0).stop 2024-01-15T18:15:15.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.381+0000 7fb698000640 1 -- 172.21.15.27:0/4127451267 shutdown_connections 2024-01-15T18:15:15.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.381+0000 7fb698000640 1 --2- 172.21.15.27:0/4127451267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6900715f0 0x7fb69015ddb0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:15.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.381+0000 7fb698000640 1 --2- 172.21.15.27:0/4127451267 >> v2:172.21.15.93:3300/0 conn(0x7fb69015ad20 0x7fb69015b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:15.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.381+0000 7fb698000640 1 -- 172.21.15.27:0/4127451267 >> 172.21.15.27:0/4127451267 conn(0x7fb6900cb700 msgr2=0x7fb6900cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:15.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.381+0000 7fb698000640 1 -- 172.21.15.27:0/4127451267 shutdown_connections 2024-01-15T18:15:15.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.381+0000 7fb698000640 1 -- 172.21.15.27:0/4127451267 wait complete. 2024-01-15T18:15:15.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.381+0000 7fb698000640 1 Processor -- start 2024-01-15T18:15:15.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.381+0000 7fb698000640 1 -- start start 2024-01-15T18:15:15.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.381+0000 7fb698000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb6900715f0 0x7fb69013e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:15.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.381+0000 7fb698000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb69015ad20 0x7fb690153730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:15.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.381+0000 7fb698000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb69015d900 0x7fb690153c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:15.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.381+0000 7fb698000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69014b220 con 0x7fb69015d900 2024-01-15T18:15:15.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.381+0000 7fb698000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69014b4c0 con 0x7fb6900715f0 2024-01-15T18:15:15.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.381+0000 7fb698000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6900ce810 con 0x7fb69015ad20 2024-01-15T18:15:15.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.381+0000 7fb696000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb69015d900 0x7fb690153c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:15.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.381+0000 7fb696000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb69015d900 0x7fb690153c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51268/0 (socket says 172.21.15.27:51268) 2024-01-15T18:15:15.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.381+0000 7fb696000640 1 -- 172.21.15.27:0/2912305576 learned_addr learned my addr 172.21.15.27:0/2912305576 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:15.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.381+0000 7fb695600640 1 --2- 172.21.15.27:0/2912305576 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb6900715f0 0x7fb69013e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:15.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.381+0000 7fb694c00640 1 --2- 172.21.15.27:0/2912305576 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb69015ad20 0x7fb690153730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:15.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.385+0000 7fb696000640 1 -- 172.21.15.27:0/2912305576 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb69015ad20 msgr2=0x7fb690153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:15.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.385+0000 7fb696000640 1 --2- 172.21.15.27:0/2912305576 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb69015ad20 0x7fb690153730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:15.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.385+0000 7fb696000640 1 -- 172.21.15.27:0/2912305576 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb6900715f0 msgr2=0x7fb69013e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:15.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.385+0000 7fb696000640 1 --2- 172.21.15.27:0/2912305576 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb6900715f0 0x7fb69013e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:15.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.385+0000 7fb696000640 1 -- 172.21.15.27:0/2912305576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb688009b30 con 0x7fb69015d900 2024-01-15T18:15:15.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.385+0000 7fb694c00640 1 --2- 172.21.15.27:0/2912305576 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb69015ad20 0x7fb690153730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:15.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.385+0000 7fb696000640 1 --2- 172.21.15.27:0/2912305576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb69015d900 0x7fb690153c70 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fb6880195b0 tx=0x7fb688019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:15.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.385+0000 7fb68d600640 1 -- 172.21.15.27:0/2912305576 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6880b8880 con 0x7fb69015d900 2024-01-15T18:15:15.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.385+0000 7fb68d600640 1 -- 172.21.15.27:0/2912305576 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6880b8a00 con 0x7fb69015d900 2024-01-15T18:15:15.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.385+0000 7fb698000640 1 -- 172.21.15.27:0/2912305576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6900ceaf0 con 0x7fb69015d900 2024-01-15T18:15:15.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.385+0000 7fb68d600640 1 -- 172.21.15.27:0/2912305576 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6880b8db0 con 0x7fb69015d900 2024-01-15T18:15:15.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.385+0000 7fb698000640 1 -- 172.21.15.27:0/2912305576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6900cda00 con 0x7fb69015d900 2024-01-15T18:15:15.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.385+0000 7fb698000640 1 -- 172.21.15.27:0/2912305576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb69004e850 con 0x7fb69015d900 2024-01-15T18:15:15.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.385+0000 7fb68d600640 1 -- 172.21.15.27:0/2912305576 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7fb6880c3460 con 0x7fb69015d900 2024-01-15T18:15:15.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.385+0000 7fb68d600640 1 --2- 172.21.15.27:0/2912305576 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb668067430 0x7fb6680698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:15.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.385+0000 7fb68d600640 1 -- 172.21.15.27:0/2912305576 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fb68811e620 con 0x7fb69015d900 2024-01-15T18:15:15.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.389+0000 7fb695600640 1 --2- 172.21.15.27:0/2912305576 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb668067430 0x7fb6680698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:15.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.389+0000 7fb695600640 1 --2- 172.21.15.27:0/2912305576 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb668067430 0x7fb6680698f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fb680007a60 tx=0x7fb680008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:15.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.393+0000 7fb68d600640 1 -- 172.21.15.27:0/2912305576 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb6880ed410 con 0x7fb69015d900 2024-01-15T18:15:15.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.581+0000 7fb698000640 1 -- 172.21.15.27:0/2912305576 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone519315", "target": ["mon-mgr", ""]}) v1 -- 0x7fb69014b600 con 0x7fb668067430 2024-01-15T18:15:15.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.589+0000 7fb68d600640 1 -- 172.21.15.27:0/2912305576 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fb69014b600 con 0x7fb668067430 2024-01-15T18:15:15.592 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:15:15.592 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:15:15.592 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "canceled", 2024-01-15T18:15:15.592 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:15:15.592 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:15:15.593 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:15:15.593 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:15:15.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:15.593 INFO:teuthology.orchestra.run.smithi027.stdout: "failure": { 2024-01-15T18:15:15.593 INFO:teuthology.orchestra.run.smithi027.stdout: "errno": "4", 2024-01-15T18:15:15.593 INFO:teuthology.orchestra.run.smithi027.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:15:15.593 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:15.593 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:15.593 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:15:15.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.593+0000 7fb698000640 1 -- 172.21.15.27:0/2912305576 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb668067430 msgr2=0x7fb6680698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:15.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.593+0000 7fb698000640 1 --2- 172.21.15.27:0/2912305576 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb668067430 0x7fb6680698f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fb680007a60 tx=0x7fb680008040 comp rx=0 tx=0).stop 2024-01-15T18:15:15.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.593+0000 7fb698000640 1 -- 172.21.15.27:0/2912305576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb69015d900 msgr2=0x7fb690153c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:15.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.593+0000 7fb698000640 1 --2- 172.21.15.27:0/2912305576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb69015d900 0x7fb690153c70 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fb6880195b0 tx=0x7fb688019740 comp rx=0 tx=0).stop 2024-01-15T18:15:15.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.593+0000 7fb698000640 1 -- 172.21.15.27:0/2912305576 shutdown_connections 2024-01-15T18:15:15.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.593+0000 7fb698000640 1 --2- 172.21.15.27:0/2912305576 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb668067430 0x7fb6680698f0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:15.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.593+0000 7fb698000640 1 --2- 172.21.15.27:0/2912305576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb69015d900 0x7fb690153c70 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:15.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.593+0000 7fb698000640 1 --2- 172.21.15.27:0/2912305576 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb69015ad20 0x7fb690153730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:15.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.593+0000 7fb698000640 1 --2- 172.21.15.27:0/2912305576 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb6900715f0 0x7fb69013e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:15.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.593+0000 7fb698000640 1 -- 172.21.15.27:0/2912305576 >> 172.21.15.27:0/2912305576 conn(0x7fb6900cb700 msgr2=0x7fb69006c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:15.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.593+0000 7fb698000640 1 -- 172.21.15.27:0/2912305576 shutdown_connections 2024-01-15T18:15:15.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.593+0000 7fb698000640 1 -- 172.21.15.27:0/2912305576 wait complete. 2024-01-15T18:15:15.613 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:15:15.614 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone519316 2024-01-15T18:15:15.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.757+0000 7f0f5e800640 1 Processor -- start 2024-01-15T18:15:15.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.757+0000 7f0f5e800640 1 -- start start 2024-01-15T18:15:15.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.757+0000 7f0f5e800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0f580cf2e0 0x7f0f580701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:15.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.757+0000 7f0f5e800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f580b1cb0 con 0x7f0f58158900 2024-01-15T18:15:15.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.757+0000 7f0f5e800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f5815fa00 con 0x7f0f5815f550 2024-01-15T18:15:15.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.757+0000 7f0f5e800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f580b1f00 con 0x7f0f580cf2e0 2024-01-15T18:15:15.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.757+0000 7f0f57400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0f58158900 0x7f0f5815bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60772/0 (socket says 172.21.15.27:60772) 2024-01-15T18:15:15.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.757+0000 7f0f57400640 1 -- 172.21.15.27:0/3493299990 learned_addr learned my addr 172.21.15.27:0/3493299990 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:15.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.757+0000 7f0f57e00640 1 --2- 172.21.15.27:0/3493299990 >> v2:172.21.15.93:3300/0 conn(0x7f0f580cf2e0 0x7f0f580701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:15.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.757+0000 7f0f56000640 1 -- 172.21.15.27:0/3493299990 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384491451 0 0) 0x7f0f580b1cb0 con 0x7f0f58158900 2024-01-15T18:15:15.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.757+0000 7f0f56000640 1 -- 172.21.15.27:0/3493299990 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f38003530 con 0x7f0f58158900 2024-01-15T18:15:15.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.757+0000 7f0f57e00640 1 -- 172.21.15.27:0/3493299990 >> v1:172.21.15.93:6789/0 conn(0x7f0f5815f550 legacy=0x7f0f580cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:15.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.757+0000 7f0f57e00640 1 -- 172.21.15.27:0/3493299990 >> v1:172.21.15.27:6789/0 conn(0x7f0f58158900 legacy=0x7f0f5815bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:15.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.757+0000 7f0f57e00640 1 -- 172.21.15.27:0/3493299990 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f580b21e0 con 0x7f0f580cf2e0 2024-01-15T18:15:15.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.761+0000 7f0f57e00640 1 --2- 172.21.15.27:0/3493299990 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f580cf2e0 0x7f0f580701e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f0f50009d90 tx=0x7f0f5001c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a9330fea0a01aa1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:15.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.761+0000 7f0f56000640 1 -- 172.21.15.27:0/3493299990 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f58070f30 con 0x7f0f580cf2e0 2024-01-15T18:15:15.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.761+0000 7f0f56000640 1 -- 172.21.15.27:0/3493299990 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f500a59e0 con 0x7f0f580cf2e0 2024-01-15T18:15:15.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.761+0000 7f0f56000640 1 -- 172.21.15.27:0/3493299990 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f500a5d70 con 0x7f0f580cf2e0 2024-01-15T18:15:15.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.761+0000 7f0f5e800640 1 -- 172.21.15.27:0/3493299990 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f580cf2e0 msgr2=0x7f0f580701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:15.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.761+0000 7f0f5e800640 1 --2- 172.21.15.27:0/3493299990 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f580cf2e0 0x7f0f580701e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f0f50009d90 tx=0x7f0f5001c920 comp rx=0 tx=0).stop 2024-01-15T18:15:15.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.761+0000 7f0f5e800640 1 -- 172.21.15.27:0/3493299990 shutdown_connections 2024-01-15T18:15:15.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.761+0000 7f0f5e800640 1 --2- 172.21.15.27:0/3493299990 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f580cf2e0 0x7f0f580701e0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:15.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.761+0000 7f0f5e800640 1 -- 172.21.15.27:0/3493299990 >> 172.21.15.27:0/3493299990 conn(0x7f0f580cb700 msgr2=0x7f0f580cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:15.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.761+0000 7f0f5e800640 1 -- 172.21.15.27:0/3493299990 shutdown_connections 2024-01-15T18:15:15.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.761+0000 7f0f5e800640 1 -- 172.21.15.27:0/3493299990 wait complete. 2024-01-15T18:15:15.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.765+0000 7f0f5e800640 1 Processor -- start 2024-01-15T18:15:15.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.765+0000 7f0f5e800640 1 -- start start 2024-01-15T18:15:15.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.765+0000 7f0f5e800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f580cf2e0 0x7f0f58150390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:15.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.765+0000 7f0f5e800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0f58158900 0x7f0f5813f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:15.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.765+0000 7f0f5e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f5815f550 0x7f0f5813f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:15.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.765+0000 7f0f5e800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f580a8bf0 con 0x7f0f5815f550 2024-01-15T18:15:15.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.765+0000 7f0f5e800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5813fc30 con 0x7f0f580cf2e0 2024-01-15T18:15:15.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.765+0000 7f0f5e800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f58156090 con 0x7f0f58158900 2024-01-15T18:15:15.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.765+0000 7f0f57e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f5815f550 0x7f0f5813f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:15.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.765+0000 7f0f57e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f5815f550 0x7f0f5813f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51282/0 (socket says 172.21.15.27:51282) 2024-01-15T18:15:15.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.765+0000 7f0f57e00640 1 -- 172.21.15.27:0/4241841339 learned_addr learned my addr 172.21.15.27:0/4241841339 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:15.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.765+0000 7f0f57400640 1 --2- 172.21.15.27:0/4241841339 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f580cf2e0 0x7f0f58150390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:15.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.765+0000 7f0f56a00640 1 --2- 172.21.15.27:0/4241841339 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0f58158900 0x7f0f5813f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:15.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.765+0000 7f0f57e00640 1 -- 172.21.15.27:0/4241841339 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0f58158900 msgr2=0x7f0f5813f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:15.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.765+0000 7f0f57e00640 1 --2- 172.21.15.27:0/4241841339 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0f58158900 0x7f0f5813f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:15.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.765+0000 7f0f57e00640 1 -- 172.21.15.27:0/4241841339 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f580cf2e0 msgr2=0x7f0f58150390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:15.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.765+0000 7f0f57e00640 1 --2- 172.21.15.27:0/4241841339 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f580cf2e0 0x7f0f58150390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:15.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.765+0000 7f0f57e00640 1 -- 172.21.15.27:0/4241841339 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f50009a70 con 0x7f0f5815f550 2024-01-15T18:15:15.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.765+0000 7f0f57e00640 1 --2- 172.21.15.27:0/4241841339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f5815f550 0x7f0f5813f6f0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f0f5001a550 tx=0x7f0f500196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:15.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.765+0000 7f0f4f400640 1 -- 172.21.15.27:0/4241841339 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f500b9860 con 0x7f0f5815f550 2024-01-15T18:15:15.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.765+0000 7f0f4f400640 1 -- 172.21.15.27:0/4241841339 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f500b99c0 con 0x7f0f5815f550 2024-01-15T18:15:15.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.765+0000 7f0f4f400640 1 -- 172.21.15.27:0/4241841339 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f500b9d30 con 0x7f0f5815f550 2024-01-15T18:15:15.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.765+0000 7f0f5e800640 1 -- 172.21.15.27:0/4241841339 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f58156370 con 0x7f0f5815f550 2024-01-15T18:15:15.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.765+0000 7f0f5e800640 1 -- 172.21.15.27:0/4241841339 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f58156930 con 0x7f0f5815f550 2024-01-15T18:15:15.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.769+0000 7f0f5e800640 1 -- 172.21.15.27:0/4241841339 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f580a8d30 con 0x7f0f5815f550 2024-01-15T18:15:15.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.769+0000 7f0f4f400640 1 -- 172.21.15.27:0/4241841339 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7f0f500ad440 con 0x7f0f5815f550 2024-01-15T18:15:15.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.769+0000 7f0f4f400640 1 --2- 172.21.15.27:0/4241841339 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0f30067430 0x7f0f300698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:15.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.769+0000 7f0f4f400640 1 -- 172.21.15.27:0/4241841339 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f0f5011e380 con 0x7f0f5815f550 2024-01-15T18:15:15.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.769+0000 7f0f57400640 1 --2- 172.21.15.27:0/4241841339 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0f30067430 0x7f0f300698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:15.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.769+0000 7f0f57400640 1 --2- 172.21.15.27:0/4241841339 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0f30067430 0x7f0f300698f0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f0f480011f0 tx=0x7f0f48009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:15.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.773+0000 7f0f4f400640 1 -- 172.21.15.27:0/4241841339 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0f580a8d30 con 0x7f0f5815f550 2024-01-15T18:15:15.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.961+0000 7f0f5e800640 1 -- 172.21.15.27:0/4241841339 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone519316", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f5806e170 con 0x7f0f30067430 2024-01-15T18:15:15.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.973+0000 7f0f4f400640 1 -- 172.21.15.27:0/4241841339 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f0f48001400 con 0x7f0f30067430 2024-01-15T18:15:15.975 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:15:15.975 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:15:15.975 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "canceled", 2024-01-15T18:15:15.975 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:15:15.976 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:15:15.976 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:15:15.976 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:15:15.976 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:15.976 INFO:teuthology.orchestra.run.smithi027.stdout: "failure": { 2024-01-15T18:15:15.976 INFO:teuthology.orchestra.run.smithi027.stdout: "errno": "4", 2024-01-15T18:15:15.976 INFO:teuthology.orchestra.run.smithi027.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:15:15.976 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:15.976 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:15.976 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:15:15.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.977+0000 7f0f5e800640 1 -- 172.21.15.27:0/4241841339 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0f30067430 msgr2=0x7f0f300698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:15.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.977+0000 7f0f5e800640 1 --2- 172.21.15.27:0/4241841339 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0f30067430 0x7f0f300698f0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f0f480011f0 tx=0x7f0f48009040 comp rx=0 tx=0).stop 2024-01-15T18:15:15.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.977+0000 7f0f5e800640 1 -- 172.21.15.27:0/4241841339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f5815f550 msgr2=0x7f0f5813f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:15.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.977+0000 7f0f5e800640 1 --2- 172.21.15.27:0/4241841339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f5815f550 0x7f0f5813f6f0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f0f5001a550 tx=0x7f0f500196c0 comp rx=0 tx=0).stop 2024-01-15T18:15:15.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.977+0000 7f0f5e800640 1 -- 172.21.15.27:0/4241841339 shutdown_connections 2024-01-15T18:15:15.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.977+0000 7f0f5e800640 1 --2- 172.21.15.27:0/4241841339 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0f30067430 0x7f0f300698f0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:15.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.977+0000 7f0f5e800640 1 --2- 172.21.15.27:0/4241841339 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0f5815f550 0x7f0f5813f6f0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:15.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.977+0000 7f0f5e800640 1 --2- 172.21.15.27:0/4241841339 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0f58158900 0x7f0f5813f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:15.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.977+0000 7f0f5e800640 1 --2- 172.21.15.27:0/4241841339 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0f580cf2e0 0x7f0f58150390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:15.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.977+0000 7f0f5e800640 1 -- 172.21.15.27:0/4241841339 >> 172.21.15.27:0/4241841339 conn(0x7f0f580cb700 msgr2=0x7f0f580733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:15.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.977+0000 7f0f5e800640 1 -- 172.21.15.27:0/4241841339 shutdown_connections 2024-01-15T18:15:15.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:15.977+0000 7f0f5e800640 1 -- 172.21.15.27:0/4241841339 wait complete. 2024-01-15T18:15:15.997 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:15:15.997 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone519317 2024-01-15T18:15:16.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.141+0000 7fe09a400640 1 Processor -- start 2024-01-15T18:15:16.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.141+0000 7fe09a400640 1 -- start start 2024-01-15T18:15:16.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.141+0000 7fe09a400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fe09415ad20 0x7fe09415b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:16.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.141+0000 7fe09a400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe0940715f0 0x7fe09415ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:16.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.141+0000 7fe09a400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe09415a610 con 0x7fe09415d900 2024-01-15T18:15:16.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.141+0000 7fe09a400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe09415e3c0 con 0x7fe0940715f0 2024-01-15T18:15:16.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.141+0000 7fe09a400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe09415e6a0 con 0x7fe09415ad20 2024-01-15T18:15:16.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.141+0000 7fe093e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe0940715f0 0x7fe09415ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:16.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.141+0000 7fe092a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe09415d900 0x7fe094070ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60780/0 (socket says 172.21.15.27:60780) 2024-01-15T18:15:16.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.141+0000 7fe092a00640 1 -- 172.21.15.27:0/2801868871 learned_addr learned my addr 172.21.15.27:0/2801868871 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:16.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.141+0000 7fe093400640 1 --2- 172.21.15.27:0/2801868871 >> v2:172.21.15.93:3300/0 conn(0x7fe09415ad20 0x7fe09415b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:16.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.141+0000 7fe093e00640 1 -- 172.21.15.27:0/2801868871 >> v2:172.21.15.93:3300/0 conn(0x7fe09415ad20 msgr2=0x7fe09415b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:16.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.141+0000 7fe093e00640 1 --2- 172.21.15.27:0/2801868871 >> v2:172.21.15.93:3300/0 conn(0x7fe09415ad20 0x7fe09415b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:16.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.141+0000 7fe093e00640 1 -- 172.21.15.27:0/2801868871 >> v1:172.21.15.27:6789/0 conn(0x7fe09415d900 legacy=0x7fe094070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:16.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.141+0000 7fe093e00640 1 -- 172.21.15.27:0/2801868871 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe09415b6e0 con 0x7fe0940715f0 2024-01-15T18:15:16.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.145+0000 7fe093e00640 1 --2- 172.21.15.27:0/2801868871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0940715f0 0x7fe09415ddb0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fe08c009d90 tx=0x7fe08c01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=344c73f7bd7226a2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:16.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.145+0000 7fe092000640 1 -- 172.21.15.27:0/2801868871 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c0a57b0 con 0x7fe0940715f0 2024-01-15T18:15:16.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.145+0000 7fe092000640 1 -- 172.21.15.27:0/2801868871 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08c0a5910 con 0x7fe0940715f0 2024-01-15T18:15:16.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.145+0000 7fe092000640 1 -- 172.21.15.27:0/2801868871 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c0a5ca0 con 0x7fe0940715f0 2024-01-15T18:15:16.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.149+0000 7fe09a400640 1 -- 172.21.15.27:0/2801868871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0940715f0 msgr2=0x7fe09415ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:16.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.149+0000 7fe09a400640 1 --2- 172.21.15.27:0/2801868871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0940715f0 0x7fe09415ddb0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fe08c009d90 tx=0x7fe08c01c8e0 comp rx=0 tx=0).stop 2024-01-15T18:15:16.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.149+0000 7fe09a400640 1 -- 172.21.15.27:0/2801868871 shutdown_connections 2024-01-15T18:15:16.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.149+0000 7fe09a400640 1 --2- 172.21.15.27:0/2801868871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0940715f0 0x7fe09415ddb0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:16.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.149+0000 7fe09a400640 1 --2- 172.21.15.27:0/2801868871 >> v2:172.21.15.93:3300/0 conn(0x7fe09415ad20 0x7fe09415b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:16.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.149+0000 7fe09a400640 1 -- 172.21.15.27:0/2801868871 >> 172.21.15.27:0/2801868871 conn(0x7fe0940cb700 msgr2=0x7fe0940cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:16.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.149+0000 7fe09a400640 1 -- 172.21.15.27:0/2801868871 shutdown_connections 2024-01-15T18:15:16.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.149+0000 7fe09a400640 1 -- 172.21.15.27:0/2801868871 wait complete. 2024-01-15T18:15:16.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.149+0000 7fe09a400640 1 Processor -- start 2024-01-15T18:15:16.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.149+0000 7fe09a400640 1 -- start start 2024-01-15T18:15:16.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.149+0000 7fe09a400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe0940715f0 0x7fe094157680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:16.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.149+0000 7fe09a400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe09415ad20 0x7fe0941535c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:16.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.149+0000 7fe09a400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe09415d900 0x7fe0940cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:16.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.149+0000 7fe09a400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0940cd6a0 con 0x7fe09415d900 2024-01-15T18:15:16.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.149+0000 7fe09a400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0940ccaf0 con 0x7fe09415ad20 2024-01-15T18:15:16.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.149+0000 7fe09a400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe094151d80 con 0x7fe0940715f0 2024-01-15T18:15:16.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.153+0000 7fe093e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe09415d900 0x7fe0940cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:16.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.153+0000 7fe092a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe09415ad20 0x7fe0941535c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:16.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.153+0000 7fe093400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe0940715f0 0x7fe094157680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:16.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.153+0000 7fe092a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe09415ad20 0x7fe0941535c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:58864/0 (socket says 172.21.15.27:58864) 2024-01-15T18:15:16.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.153+0000 7fe093400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe0940715f0 0x7fe094157680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:33776/0 (socket says 172.21.15.27:33776) 2024-01-15T18:15:16.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.153+0000 7fe093400640 1 -- 172.21.15.27:0/3669041204 learned_addr learned my addr 172.21.15.27:0/3669041204 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:16.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.153+0000 7fe093e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe09415d900 0x7fe0940cdac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51302/0 (socket says 172.21.15.27:51302) 2024-01-15T18:15:16.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.153+0000 7fe093e00640 1 -- 172.21.15.27:0/3669041204 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe0940715f0 msgr2=0x7fe094157680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:16.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.153+0000 7fe093e00640 1 --2- 172.21.15.27:0/3669041204 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe0940715f0 0x7fe094157680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:16.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.153+0000 7fe093e00640 1 -- 172.21.15.27:0/3669041204 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe09415ad20 msgr2=0x7fe0941535c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:16.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.153+0000 7fe093e00640 1 --2- 172.21.15.27:0/3669041204 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe09415ad20 0x7fe0941535c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:16.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.153+0000 7fe093e00640 1 -- 172.21.15.27:0/3669041204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe08c009a70 con 0x7fe09415d900 2024-01-15T18:15:16.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.153+0000 7fe093e00640 1 --2- 172.21.15.27:0/3669041204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe09415d900 0x7fe0940cdac0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fe08c0ac000 tx=0x7fe08c019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:16.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.153+0000 7fe08b400640 1 -- 172.21.15.27:0/3669041204 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c0b9860 con 0x7fe09415d900 2024-01-15T18:15:16.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.153+0000 7fe08b400640 1 -- 172.21.15.27:0/3669041204 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08c0b99c0 con 0x7fe09415d900 2024-01-15T18:15:16.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.153+0000 7fe08b400640 1 -- 172.21.15.27:0/3669041204 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c0b9d50 con 0x7fe09415d900 2024-01-15T18:15:16.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.153+0000 7fe09a400640 1 -- 172.21.15.27:0/3669041204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0940ccd00 con 0x7fe09415d900 2024-01-15T18:15:16.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.153+0000 7fe09a400640 1 -- 172.21.15.27:0/3669041204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0940cd2c0 con 0x7fe09415d900 2024-01-15T18:15:16.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.153+0000 7fe09a400640 1 -- 172.21.15.27:0/3669041204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0940cd7e0 con 0x7fe09415d900 2024-01-15T18:15:16.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.153+0000 7fe08b400640 1 -- 172.21.15.27:0/3669041204 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7fe08c0ad440 con 0x7fe09415d900 2024-01-15T18:15:16.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.157+0000 7fe08b400640 1 --2- 172.21.15.27:0/3669041204 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe06c067430 0x7fe06c0698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:16.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.157+0000 7fe08b400640 1 -- 172.21.15.27:0/3669041204 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fe08c11d6e0 con 0x7fe09415d900 2024-01-15T18:15:16.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.157+0000 7fe093400640 1 --2- 172.21.15.27:0/3669041204 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe06c067430 0x7fe06c0698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:16.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.157+0000 7fe093400640 1 --2- 172.21.15.27:0/3669041204 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe06c067430 0x7fe06c0698f0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe084007b10 tx=0x7fe084008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:16.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.161+0000 7fe08b400640 1 -- 172.21.15.27:0/3669041204 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe0940cd7e0 con 0x7fe09415d900 2024-01-15T18:15:16.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.349+0000 7fe09a400640 1 -- 172.21.15.27:0/3669041204 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone519317", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0940cd7e0 con 0x7fe06c067430 2024-01-15T18:15:16.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.361+0000 7fe08b400640 1 -- 172.21.15.27:0/3669041204 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fe0940cd7e0 con 0x7fe06c067430 2024-01-15T18:15:16.362 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:15:16.362 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:15:16.362 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "canceled", 2024-01-15T18:15:16.362 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:15:16.362 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:15:16.362 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:15:16.362 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:15:16.362 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:16.362 INFO:teuthology.orchestra.run.smithi027.stdout: "failure": { 2024-01-15T18:15:16.363 INFO:teuthology.orchestra.run.smithi027.stdout: "errno": "4", 2024-01-15T18:15:16.363 INFO:teuthology.orchestra.run.smithi027.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:15:16.363 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:16.363 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:16.363 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:15:16.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.361+0000 7fe09a400640 1 -- 172.21.15.27:0/3669041204 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe06c067430 msgr2=0x7fe06c0698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:16.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.361+0000 7fe09a400640 1 --2- 172.21.15.27:0/3669041204 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe06c067430 0x7fe06c0698f0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe084007b10 tx=0x7fe084008040 comp rx=0 tx=0).stop 2024-01-15T18:15:16.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.361+0000 7fe09a400640 1 -- 172.21.15.27:0/3669041204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe09415d900 msgr2=0x7fe0940cdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:16.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.361+0000 7fe09a400640 1 --2- 172.21.15.27:0/3669041204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe09415d900 0x7fe0940cdac0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fe08c0ac000 tx=0x7fe08c019680 comp rx=0 tx=0).stop 2024-01-15T18:15:16.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.361+0000 7fe09a400640 1 -- 172.21.15.27:0/3669041204 shutdown_connections 2024-01-15T18:15:16.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.361+0000 7fe09a400640 1 --2- 172.21.15.27:0/3669041204 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe06c067430 0x7fe06c0698f0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:16.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.361+0000 7fe09a400640 1 --2- 172.21.15.27:0/3669041204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe09415d900 0x7fe0940cdac0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:16.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.361+0000 7fe09a400640 1 --2- 172.21.15.27:0/3669041204 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe09415ad20 0x7fe0941535c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:16.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.361+0000 7fe09a400640 1 --2- 172.21.15.27:0/3669041204 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe0940715f0 0x7fe094157680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:16.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.361+0000 7fe09a400640 1 -- 172.21.15.27:0/3669041204 >> 172.21.15.27:0/3669041204 conn(0x7fe0940cb700 msgr2=0x7fe0940cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:16.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.361+0000 7fe09a400640 1 -- 172.21.15.27:0/3669041204 shutdown_connections 2024-01-15T18:15:16.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.361+0000 7fe09a400640 1 -- 172.21.15.27:0/3669041204 wait complete. 2024-01-15T18:15:16.383 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:15:16.384 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone519318 2024-01-15T18:15:16.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.525+0000 7fb714a00640 1 Processor -- start 2024-01-15T18:15:16.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.525+0000 7fb714a00640 1 -- start start 2024-01-15T18:15:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.529+0000 7fb714a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fb70c158900 0x7fb70c15f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.529+0000 7fb714a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb70c15bf00 0x7fb70c0cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.529+0000 7fb714a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb70c0cf110 0x7fb70c0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.529+0000 7fb714a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb70c0cf580 con 0x7fb70c15bf00 2024-01-15T18:15:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.529+0000 7fb714a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb70c0b1cb0 con 0x7fb70c158900 2024-01-15T18:15:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.529+0000 7fb714a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb70c0b1f00 con 0x7fb70c0cf110 2024-01-15T18:15:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.529+0000 7fb711600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb70c15bf00 0x7fb70c0cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.529+0000 7fb711600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb70c15bf00 0x7fb70c0cebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51304/0 (socket says 172.21.15.27:51304) 2024-01-15T18:15:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.529+0000 7fb711600640 1 -- 172.21.15.27:0/3653591225 learned_addr learned my addr 172.21.15.27:0/3653591225 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.529+0000 7fb712a00640 1 --2- 172.21.15.27:0/3653591225 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb70c0cf110 0x7fb70c0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:16.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.529+0000 7fb712000640 1 --2- 172.21.15.27:0/3653591225 >> v2:172.21.15.93:3300/0 conn(0x7fb70c158900 0x7fb70c15f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:16.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.529+0000 7fb711600640 1 -- 172.21.15.27:0/3653591225 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb70c0cf110 msgr2=0x7fb70c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:16.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.529+0000 7fb711600640 1 --2- 172.21.15.27:0/3653591225 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb70c0cf110 0x7fb70c0701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:16.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.529+0000 7fb711600640 1 -- 172.21.15.27:0/3653591225 >> v2:172.21.15.93:3300/0 conn(0x7fb70c158900 msgr2=0x7fb70c15f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:16.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.529+0000 7fb711600640 1 --2- 172.21.15.27:0/3653591225 >> v2:172.21.15.93:3300/0 conn(0x7fb70c158900 0x7fb70c15f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:16.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.529+0000 7fb711600640 1 -- 172.21.15.27:0/3653591225 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70c0b21e0 con 0x7fb70c15bf00 2024-01-15T18:15:16.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.529+0000 7fb712a00640 1 --2- 172.21.15.27:0/3653591225 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb70c0cf110 0x7fb70c0701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:16.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.529+0000 7fb712000640 1 --2- 172.21.15.27:0/3653591225 >> v2:172.21.15.93:3300/0 conn(0x7fb70c158900 0x7fb70c15f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:16.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.533+0000 7fb711600640 1 --2- 172.21.15.27:0/3653591225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb70c15bf00 0x7fb70c0cebd0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fb70c0ceaa0 tx=0x7fb6f401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d42f0d7169d551 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:16.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.533+0000 7fb710c00640 1 -- 172.21.15.27:0/3653591225 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70c070f30 con 0x7fb70c15bf00 2024-01-15T18:15:16.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.533+0000 7fb710c00640 1 -- 172.21.15.27:0/3653591225 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f40ac410 con 0x7fb70c15bf00 2024-01-15T18:15:16.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.533+0000 7fb710c00640 1 -- 172.21.15.27:0/3653591225 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f40ac7a0 con 0x7fb70c15bf00 2024-01-15T18:15:16.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.533+0000 7fb714a00640 1 -- 172.21.15.27:0/3653591225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb70c15bf00 msgr2=0x7fb70c0cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:16.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.533+0000 7fb714a00640 1 --2- 172.21.15.27:0/3653591225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb70c15bf00 0x7fb70c0cebd0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fb70c0ceaa0 tx=0x7fb6f401c500 comp rx=0 tx=0).stop 2024-01-15T18:15:16.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.533+0000 7fb714a00640 1 -- 172.21.15.27:0/3653591225 shutdown_connections 2024-01-15T18:15:16.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.533+0000 7fb714a00640 1 --2- 172.21.15.27:0/3653591225 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb70c0cf110 0x7fb70c0701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:16.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.533+0000 7fb714a00640 1 --2- 172.21.15.27:0/3653591225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb70c15bf00 0x7fb70c0cebd0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:16.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.533+0000 7fb714a00640 1 --2- 172.21.15.27:0/3653591225 >> v2:172.21.15.93:3300/0 conn(0x7fb70c158900 0x7fb70c15f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:16.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.533+0000 7fb714a00640 1 -- 172.21.15.27:0/3653591225 >> 172.21.15.27:0/3653591225 conn(0x7fb70c0cb700 msgr2=0x7fb70c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:16.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.533+0000 7fb714a00640 1 -- 172.21.15.27:0/3653591225 shutdown_connections 2024-01-15T18:15:16.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.533+0000 7fb714a00640 1 -- 172.21.15.27:0/3653591225 wait complete. 2024-01-15T18:15:16.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.533+0000 7fb714a00640 1 Processor -- start 2024-01-15T18:15:16.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.533+0000 7fb714a00640 1 -- start start 2024-01-15T18:15:16.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb714a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb70c0cf110 0x7fb70c15ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:16.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb714a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb70c158900 0x7fb70c0cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:16.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb714a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb70c15bf00 0x7fb70c153c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:16.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb714a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70c153820 con 0x7fb70c158900 2024-01-15T18:15:16.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb714a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70c1595c0 con 0x7fb70c0cf110 2024-01-15T18:15:16.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb714a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb70c1598a0 con 0x7fb70c15bf00 2024-01-15T18:15:16.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb711600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb70c158900 0x7fb70c0cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:16.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb711600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb70c158900 0x7fb70c0cdac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51310/0 (socket says 172.21.15.27:51310) 2024-01-15T18:15:16.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb711600640 1 -- 172.21.15.27:0/1434105200 learned_addr learned my addr 172.21.15.27:0/1434105200 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:16.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb712000640 1 --2- 172.21.15.27:0/1434105200 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb70c0cf110 0x7fb70c15ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:16.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb712a00640 1 --2- 172.21.15.27:0/1434105200 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb70c15bf00 0x7fb70c153c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:16.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb711600640 1 -- 172.21.15.27:0/1434105200 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb70c15bf00 msgr2=0x7fb70c153c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:16.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb711600640 1 --2- 172.21.15.27:0/1434105200 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb70c15bf00 0x7fb70c153c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:16.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb711600640 1 -- 172.21.15.27:0/1434105200 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb70c0cf110 msgr2=0x7fb70c15ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:16.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb711600640 1 --2- 172.21.15.27:0/1434105200 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb70c0cf110 0x7fb70c15ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:16.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb711600640 1 -- 172.21.15.27:0/1434105200 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f40094d0 con 0x7fb70c158900 2024-01-15T18:15:16.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb712a00640 1 --2- 172.21.15.27:0/1434105200 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb70c15bf00 0x7fb70c153c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:16.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb712000640 1 --2- 172.21.15.27:0/1434105200 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb70c0cf110 0x7fb70c15ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:16.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb711600640 1 --2- 172.21.15.27:0/1434105200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb70c158900 0x7fb70c0cdac0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fb6f401abb0 tx=0x7fb6f40192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:16.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb70a000640 1 -- 172.21.15.27:0/1434105200 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f40ace20 con 0x7fb70c158900 2024-01-15T18:15:16.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb70a000640 1 -- 172.21.15.27:0/1434105200 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f40b65f0 con 0x7fb70c158900 2024-01-15T18:15:16.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb714a00640 1 -- 172.21.15.27:0/1434105200 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb70c159b80 con 0x7fb70c158900 2024-01-15T18:15:16.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb70a000640 1 -- 172.21.15.27:0/1434105200 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f40b69a0 con 0x7fb70c158900 2024-01-15T18:15:16.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb714a00640 1 -- 172.21.15.27:0/1434105200 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb70c15b180 con 0x7fb70c158900 2024-01-15T18:15:16.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb714a00640 1 -- 172.21.15.27:0/1434105200 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb70c04e850 con 0x7fb70c158900 2024-01-15T18:15:16.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.537+0000 7fb70a000640 1 -- 172.21.15.27:0/1434105200 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7fb6f40b6b00 con 0x7fb70c158900 2024-01-15T18:15:16.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.541+0000 7fb70a000640 1 --2- 172.21.15.27:0/1434105200 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb6e4067460 0x7fb6e4069920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:16.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.541+0000 7fb70a000640 1 -- 172.21.15.27:0/1434105200 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fb6f411d7e0 con 0x7fb70c158900 2024-01-15T18:15:16.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.541+0000 7fb712000640 1 --2- 172.21.15.27:0/1434105200 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb6e4067460 0x7fb6e4069920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:16.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.541+0000 7fb712000640 1 --2- 172.21.15.27:0/1434105200 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb6e4067460 0x7fb6e4069920 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc006fd0 tx=0x7fb6fc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:16.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.545+0000 7fb70a000640 1 -- 172.21.15.27:0/1434105200 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb6f40ec440 con 0x7fb70c158900 2024-01-15T18:15:16.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.733+0000 7fb714a00640 1 -- 172.21.15.27:0/1434105200 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone519318", "target": ["mon-mgr", ""]}) v1 -- 0x7fb70c159a10 con 0x7fb6e4067460 2024-01-15T18:15:16.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.741+0000 7fb70a000640 1 -- 172.21.15.27:0/1434105200 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fb70c159a10 con 0x7fb6e4067460 2024-01-15T18:15:16.743 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:15:16.743 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:15:16.743 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "canceled", 2024-01-15T18:15:16.743 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:15:16.744 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:15:16.744 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:15:16.744 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:15:16.744 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:16.744 INFO:teuthology.orchestra.run.smithi027.stdout: "failure": { 2024-01-15T18:15:16.744 INFO:teuthology.orchestra.run.smithi027.stdout: "errno": "4", 2024-01-15T18:15:16.744 INFO:teuthology.orchestra.run.smithi027.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:15:16.744 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:16.744 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:16.744 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:15:16.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.745+0000 7fb714a00640 1 -- 172.21.15.27:0/1434105200 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb6e4067460 msgr2=0x7fb6e4069920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:16.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.745+0000 7fb714a00640 1 --2- 172.21.15.27:0/1434105200 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb6e4067460 0x7fb6e4069920 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc006fd0 tx=0x7fb6fc008040 comp rx=0 tx=0).stop 2024-01-15T18:15:16.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.745+0000 7fb714a00640 1 -- 172.21.15.27:0/1434105200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb70c158900 msgr2=0x7fb70c0cdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:16.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.745+0000 7fb714a00640 1 --2- 172.21.15.27:0/1434105200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb70c158900 0x7fb70c0cdac0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fb6f401abb0 tx=0x7fb6f40192a0 comp rx=0 tx=0).stop 2024-01-15T18:15:16.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.745+0000 7fb714a00640 1 -- 172.21.15.27:0/1434105200 shutdown_connections 2024-01-15T18:15:16.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.745+0000 7fb714a00640 1 --2- 172.21.15.27:0/1434105200 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb6e4067460 0x7fb6e4069920 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:16.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.745+0000 7fb714a00640 1 --2- 172.21.15.27:0/1434105200 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb70c15bf00 0x7fb70c153c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:16.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.745+0000 7fb714a00640 1 --2- 172.21.15.27:0/1434105200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb70c158900 0x7fb70c0cdac0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:16.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.745+0000 7fb714a00640 1 --2- 172.21.15.27:0/1434105200 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb70c0cf110 0x7fb70c15ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:16.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.745+0000 7fb714a00640 1 -- 172.21.15.27:0/1434105200 >> 172.21.15.27:0/1434105200 conn(0x7fb70c0cb700 msgr2=0x7fb70c073450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:16.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.745+0000 7fb714a00640 1 -- 172.21.15.27:0/1434105200 shutdown_connections 2024-01-15T18:15:16.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.745+0000 7fb714a00640 1 -- 172.21.15.27:0/1434105200 wait complete. 2024-01-15T18:15:16.765 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:15:16.766 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone519319 2024-01-15T18:15:16.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.909+0000 7f040ac00640 1 Processor -- start 2024-01-15T18:15:16.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.909+0000 7f040ac00640 1 -- start start 2024-01-15T18:15:16.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.909+0000 7f040ac00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f04040cf2e0 0x7f04040701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:16.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.909+0000 7f040ac00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04040b1cb0 con 0x7f040415f550 2024-01-15T18:15:16.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.909+0000 7f040ac00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f040415fa00 con 0x7f0404158900 2024-01-15T18:15:16.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.909+0000 7f040ac00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04040b1f00 con 0x7f04040cf2e0 2024-01-15T18:15:16.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.909+0000 7f0403400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f040415f550 0x7f04040cebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60784/0 (socket says 172.21.15.27:60784) 2024-01-15T18:15:16.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.909+0000 7f0403400640 1 -- 172.21.15.27:0/1403202867 learned_addr learned my addr 172.21.15.27:0/1403202867 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:16.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.909+0000 7f0408c00640 1 --2- 172.21.15.27:0/1403202867 >> v2:172.21.15.93:3300/0 conn(0x7f04040cf2e0 0x7f04040701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:16.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.909+0000 7f0402a00640 1 -- 172.21.15.27:0/1403202867 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790446908 0 0) 0x7f04040b1cb0 con 0x7f040415f550 2024-01-15T18:15:16.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.909+0000 7f0402a00640 1 -- 172.21.15.27:0/1403202867 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03e4003530 con 0x7f040415f550 2024-01-15T18:15:16.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.909+0000 7f0408c00640 1 -- 172.21.15.27:0/1403202867 >> v1:172.21.15.93:6789/0 conn(0x7f0404158900 legacy=0x7f040415bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:16.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.909+0000 7f0402a00640 1 -- 172.21.15.27:0/1403202867 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 563930843 0 0) 0x7f040415fa00 con 0x7f0404158900 2024-01-15T18:15:16.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.909+0000 7f0408c00640 1 -- 172.21.15.27:0/1403202867 >> v1:172.21.15.27:6789/0 conn(0x7f040415f550 legacy=0x7f04040cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:16.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.909+0000 7f0408c00640 1 -- 172.21.15.27:0/1403202867 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04040b21e0 con 0x7f04040cf2e0 2024-01-15T18:15:16.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.913+0000 7f0408c00640 1 --2- 172.21.15.27:0/1403202867 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f04040cf2e0 0x7f04040701e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f03fc009d90 tx=0x7f03fc01c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4de25e0d59916ee4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:16.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.913+0000 7f0402a00640 1 -- 172.21.15.27:0/1403202867 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0404070f30 con 0x7f04040cf2e0 2024-01-15T18:15:16.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.913+0000 7f0402a00640 1 -- 172.21.15.27:0/1403202867 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03fc0a59e0 con 0x7f04040cf2e0 2024-01-15T18:15:16.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.913+0000 7f0402a00640 1 -- 172.21.15.27:0/1403202867 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03fc0a5d70 con 0x7f04040cf2e0 2024-01-15T18:15:16.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.913+0000 7f040ac00640 1 -- 172.21.15.27:0/1403202867 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f04040cf2e0 msgr2=0x7f04040701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:16.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.913+0000 7f040ac00640 1 --2- 172.21.15.27:0/1403202867 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f04040cf2e0 0x7f04040701e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f03fc009d90 tx=0x7f03fc01c920 comp rx=0 tx=0).stop 2024-01-15T18:15:16.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.913+0000 7f040ac00640 1 -- 172.21.15.27:0/1403202867 shutdown_connections 2024-01-15T18:15:16.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.913+0000 7f040ac00640 1 --2- 172.21.15.27:0/1403202867 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f04040cf2e0 0x7f04040701e0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:16.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.913+0000 7f040ac00640 1 -- 172.21.15.27:0/1403202867 >> 172.21.15.27:0/1403202867 conn(0x7f04040cb700 msgr2=0x7f04040cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:16.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.913+0000 7f040ac00640 1 -- 172.21.15.27:0/1403202867 shutdown_connections 2024-01-15T18:15:16.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.913+0000 7f040ac00640 1 -- 172.21.15.27:0/1403202867 wait complete. 2024-01-15T18:15:16.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.917+0000 7f040ac00640 1 Processor -- start 2024-01-15T18:15:16.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.917+0000 7f040ac00640 1 -- start start 2024-01-15T18:15:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.917+0000 7f040ac00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f04040cf2e0 0x7f04040ad3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.917+0000 7f040ac00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0404158900 0x7f040413f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.917+0000 7f040ac00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f040415f550 0x7f04040cc810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.917+0000 7f040ac00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0404151680 con 0x7f0404158900 2024-01-15T18:15:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.917+0000 7f040ac00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04040ad8f0 con 0x7f040415f550 2024-01-15T18:15:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.917+0000 7f040ac00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0404153410 con 0x7f04040cf2e0 2024-01-15T18:15:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.917+0000 7f0403400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0404158900 0x7f040413f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.917+0000 7f0403400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0404158900 0x7f040413f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51316/0 (socket says 172.21.15.27:51316) 2024-01-15T18:15:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.917+0000 7f0403400640 1 -- 172.21.15.27:0/2486170582 learned_addr learned my addr 172.21.15.27:0/2486170582 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.917+0000 7f0403e00640 1 --2- 172.21.15.27:0/2486170582 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f04040cf2e0 0x7f04040ad3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.917+0000 7f0408c00640 1 --2- 172.21.15.27:0/2486170582 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f040415f550 0x7f04040cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.917+0000 7f0403400640 1 -- 172.21.15.27:0/2486170582 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f04040cf2e0 msgr2=0x7f04040ad3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.917+0000 7f0403400640 1 --2- 172.21.15.27:0/2486170582 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f04040cf2e0 0x7f04040ad3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:16.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.917+0000 7f0403400640 1 -- 172.21.15.27:0/2486170582 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f040415f550 msgr2=0x7f04040cc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:16.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.917+0000 7f0403400640 1 --2- 172.21.15.27:0/2486170582 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f040415f550 0x7f04040cc810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:16.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.917+0000 7f0403400640 1 -- 172.21.15.27:0/2486170582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03fc009a70 con 0x7f0404158900 2024-01-15T18:15:16.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.917+0000 7f0403400640 1 --2- 172.21.15.27:0/2486170582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0404158900 0x7f040413f5f0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f03ec00c3c0 tx=0x7f03ec00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:16.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.917+0000 7f03fbe00640 1 -- 172.21.15.27:0/2486170582 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec005cf0 con 0x7f0404158900 2024-01-15T18:15:16.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.917+0000 7f03fbe00640 1 -- 172.21.15.27:0/2486170582 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03ec005e50 con 0x7f0404158900 2024-01-15T18:15:16.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.917+0000 7f040ac00640 1 -- 172.21.15.27:0/2486170582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04041536f0 con 0x7f0404158900 2024-01-15T18:15:16.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.917+0000 7f03fbe00640 1 -- 172.21.15.27:0/2486170582 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec0072f0 con 0x7f0404158900 2024-01-15T18:15:16.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.921+0000 7f040ac00640 1 -- 172.21.15.27:0/2486170582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0404153cb0 con 0x7f0404158900 2024-01-15T18:15:16.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.921+0000 7f040ac00640 1 -- 172.21.15.27:0/2486170582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0404073bd0 con 0x7f0404158900 2024-01-15T18:15:16.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.921+0000 7f03fbe00640 1 -- 172.21.15.27:0/2486170582 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7f03ec007450 con 0x7f0404158900 2024-01-15T18:15:16.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.921+0000 7f03fbe00640 1 --2- 172.21.15.27:0/2486170582 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03dc067430 0x7f03dc0698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:16.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.921+0000 7f03fbe00640 1 -- 172.21.15.27:0/2486170582 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f03ec0825b0 con 0x7f0404158900 2024-01-15T18:15:16.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.921+0000 7f0403e00640 1 --2- 172.21.15.27:0/2486170582 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03dc067430 0x7f03dc0698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:16.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.925+0000 7f0403e00640 1 --2- 172.21.15.27:0/2486170582 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03dc067430 0x7f03dc0698f0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f03f4000f70 tx=0x7f03f4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:16.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:16.925+0000 7f03fbe00640 1 -- 172.21.15.27:0/2486170582 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f03ec051050 con 0x7f0404158900 2024-01-15T18:15:17.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.113+0000 7f040ac00640 1 -- 172.21.15.27:0/2486170582 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone519319", "target": ["mon-mgr", ""]}) v1 -- 0x7f040406e170 con 0x7f03dc067430 2024-01-15T18:15:17.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.125+0000 7f03fbe00640 1 -- 172.21.15.27:0/2486170582 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f03f40013e0 con 0x7f03dc067430 2024-01-15T18:15:17.127 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:15:17.127 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:15:17.127 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "canceled", 2024-01-15T18:15:17.127 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:15:17.127 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:15:17.127 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:15:17.127 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:15:17.127 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:17.127 INFO:teuthology.orchestra.run.smithi027.stdout: "failure": { 2024-01-15T18:15:17.127 INFO:teuthology.orchestra.run.smithi027.stdout: "errno": "4", 2024-01-15T18:15:17.127 INFO:teuthology.orchestra.run.smithi027.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:15:17.127 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:17.127 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:17.127 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:15:17.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.129+0000 7f040ac00640 1 -- 172.21.15.27:0/2486170582 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03dc067430 msgr2=0x7f03dc0698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:17.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.129+0000 7f040ac00640 1 --2- 172.21.15.27:0/2486170582 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03dc067430 0x7f03dc0698f0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f03f4000f70 tx=0x7f03f4009040 comp rx=0 tx=0).stop 2024-01-15T18:15:17.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.129+0000 7f040ac00640 1 -- 172.21.15.27:0/2486170582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0404158900 msgr2=0x7f040413f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:17.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.129+0000 7f040ac00640 1 --2- 172.21.15.27:0/2486170582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0404158900 0x7f040413f5f0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f03ec00c3c0 tx=0x7f03ec00c880 comp rx=0 tx=0).stop 2024-01-15T18:15:17.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.129+0000 7f040ac00640 1 -- 172.21.15.27:0/2486170582 shutdown_connections 2024-01-15T18:15:17.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.129+0000 7f040ac00640 1 --2- 172.21.15.27:0/2486170582 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03dc067430 0x7f03dc0698f0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:17.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.129+0000 7f040ac00640 1 --2- 172.21.15.27:0/2486170582 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f040415f550 0x7f04040cc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:17.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.129+0000 7f040ac00640 1 --2- 172.21.15.27:0/2486170582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0404158900 0x7f040413f5f0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:17.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.129+0000 7f040ac00640 1 --2- 172.21.15.27:0/2486170582 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f04040cf2e0 0x7f04040ad3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:17.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.129+0000 7f040ac00640 1 -- 172.21.15.27:0/2486170582 >> 172.21.15.27:0/2486170582 conn(0x7f04040cb700 msgr2=0x7f0404157cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:17.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.129+0000 7f040ac00640 1 -- 172.21.15.27:0/2486170582 shutdown_connections 2024-01-15T18:15:17.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.129+0000 7f040ac00640 1 -- 172.21.15.27:0/2486170582 wait complete. 2024-01-15T18:15:17.149 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:15:17.149 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone519320 2024-01-15T18:15:17.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.293+0000 7f32b0c00640 1 Processor -- start 2024-01-15T18:15:17.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.293+0000 7f32b0c00640 1 -- start start 2024-01-15T18:15:17.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.293+0000 7f32b0c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32ac15d680 0x7f32ac0cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:17.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.293+0000 7f32b0c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f32ac0ccf80 0x7f32ac0cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:17.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.293+0000 7f32b0c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32ac0cb4d0 con 0x7f32ac0cfc30 2024-01-15T18:15:17.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.293+0000 7f32b0c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32ac0d0120 con 0x7f32ac0ccf80 2024-01-15T18:15:17.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.293+0000 7f32b0c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32ac0d0400 con 0x7f32ac15d680 2024-01-15T18:15:17.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.293+0000 7f32aa800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f32ac0ccf80 0x7f32ac0cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:17.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.293+0000 7f32a9400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f32ac0cfc30 0x7f32ac06b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60788/0 (socket says 172.21.15.27:60788) 2024-01-15T18:15:17.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.293+0000 7f32a9400640 1 -- 172.21.15.27:0/85783712 learned_addr learned my addr 172.21.15.27:0/85783712 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:17.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.293+0000 7f32a9e00640 1 --2- 172.21.15.27:0/85783712 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32ac15d680 0x7f32ac0cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:17.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.293+0000 7f32aa800640 1 -- 172.21.15.27:0/85783712 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32ac15d680 msgr2=0x7f32ac0cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:17.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.293+0000 7f32aa800640 1 --2- 172.21.15.27:0/85783712 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32ac15d680 0x7f32ac0cd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:17.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.293+0000 7f32aa800640 1 -- 172.21.15.27:0/85783712 >> v1:172.21.15.27:6789/0 conn(0x7f32ac0cfc30 legacy=0x7f32ac06b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:17.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.293+0000 7f32aa800640 1 -- 172.21.15.27:0/85783712 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32ac0cb770 con 0x7f32ac0ccf80 2024-01-15T18:15:17.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.293+0000 7f32a8a00640 1 -- 172.21.15.27:0/85783712 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673468365 0 0) 0x7f32ac0cb4d0 con 0x7f32ac0cfc30 2024-01-15T18:15:17.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.297+0000 7f32aa800640 1 --2- 172.21.15.27:0/85783712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32ac0ccf80 0x7f32ac0cdc40 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f32a4009d90 tx=0x7f32a401c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d2004d22dbb3701 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:17.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.297+0000 7f32a8a00640 1 -- 172.21.15.27:0/85783712 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ac0ce850 con 0x7f32ac0ccf80 2024-01-15T18:15:17.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.297+0000 7f32a8a00640 1 -- 172.21.15.27:0/85783712 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a40a5a00 con 0x7f32ac0ccf80 2024-01-15T18:15:17.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.297+0000 7f32a8a00640 1 -- 172.21.15.27:0/85783712 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a40a5d90 con 0x7f32ac0ccf80 2024-01-15T18:15:17.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.297+0000 7f32b0c00640 1 -- 172.21.15.27:0/85783712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32ac0ccf80 msgr2=0x7f32ac0cdc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:17.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.297+0000 7f32b0c00640 1 --2- 172.21.15.27:0/85783712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32ac0ccf80 0x7f32ac0cdc40 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f32a4009d90 tx=0x7f32a401c920 comp rx=0 tx=0).stop 2024-01-15T18:15:17.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.297+0000 7f32b0c00640 1 -- 172.21.15.27:0/85783712 shutdown_connections 2024-01-15T18:15:17.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.297+0000 7f32b0c00640 1 --2- 172.21.15.27:0/85783712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32ac0ccf80 0x7f32ac0cdc40 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:17.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.297+0000 7f32b0c00640 1 --2- 172.21.15.27:0/85783712 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32ac15d680 0x7f32ac0cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:17.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.297+0000 7f32b0c00640 1 -- 172.21.15.27:0/85783712 >> 172.21.15.27:0/85783712 conn(0x7f32ac0d0e80 msgr2=0x7f32ac0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:17.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.301+0000 7f32b0c00640 1 -- 172.21.15.27:0/85783712 shutdown_connections 2024-01-15T18:15:17.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.301+0000 7f32b0c00640 1 -- 172.21.15.27:0/85783712 wait complete. 2024-01-15T18:15:17.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.301+0000 7f32b0c00640 1 Processor -- start 2024-01-15T18:15:17.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.301+0000 7f32b0c00640 1 -- start start 2024-01-15T18:15:17.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.301+0000 7f32b0c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32ac0ccf80 0x7f32ac1605a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:17.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.301+0000 7f32b0c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32ac0cfc30 0x7f32ac0c59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:17.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.301+0000 7f32b0c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32ac15d680 0x7f32ac15f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:17.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.301+0000 7f32b0c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ac0c5f20 con 0x7f32ac0cfc30 2024-01-15T18:15:17.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.301+0000 7f32b0c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ac0c69f0 con 0x7f32ac0ccf80 2024-01-15T18:15:17.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.301+0000 7f32b0c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32ac15f950 con 0x7f32ac15d680 2024-01-15T18:15:17.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.301+0000 7f32a9400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32ac0cfc30 0x7f32ac0c59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.301+0000 7f32a9400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32ac0cfc30 0x7f32ac0c59e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51338/0 (socket says 172.21.15.27:51338) 2024-01-15T18:15:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.301+0000 7f32a9400640 1 -- 172.21.15.27:0/1169085898 learned_addr learned my addr 172.21.15.27:0/1169085898 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.301+0000 7f32aa800640 1 --2- 172.21.15.27:0/1169085898 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32ac15d680 0x7f32ac15f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.301+0000 7f32a9e00640 1 --2- 172.21.15.27:0/1169085898 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32ac0ccf80 0x7f32ac1605a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.301+0000 7f32a9400640 1 -- 172.21.15.27:0/1169085898 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32ac15d680 msgr2=0x7f32ac15f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.301+0000 7f32a9400640 1 --2- 172.21.15.27:0/1169085898 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32ac15d680 0x7f32ac15f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.301+0000 7f32a9400640 1 -- 172.21.15.27:0/1169085898 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32ac0ccf80 msgr2=0x7f32ac1605a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.301+0000 7f32a9400640 1 --2- 172.21.15.27:0/1169085898 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32ac0ccf80 0x7f32ac1605a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.301+0000 7f32a9400640 1 -- 172.21.15.27:0/1169085898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a4009a70 con 0x7f32ac0cfc30 2024-01-15T18:15:17.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.305+0000 7f32a9400640 1 --2- 172.21.15.27:0/1169085898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32ac0cfc30 0x7f32ac0c59e0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f329400c4b0 tx=0x7f329400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:17.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.305+0000 7f32a2000640 1 -- 172.21.15.27:0/1169085898 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32940074e0 con 0x7f32ac0cfc30 2024-01-15T18:15:17.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.305+0000 7f32a2000640 1 -- 172.21.15.27:0/1169085898 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3294007660 con 0x7f32ac0cfc30 2024-01-15T18:15:17.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.305+0000 7f32a2000640 1 -- 172.21.15.27:0/1169085898 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3294004a30 con 0x7f32ac0cfc30 2024-01-15T18:15:17.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.305+0000 7f32b0c00640 1 -- 172.21.15.27:0/1169085898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32ac0c6150 con 0x7f32ac0cfc30 2024-01-15T18:15:17.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.305+0000 7f32b0c00640 1 -- 172.21.15.27:0/1169085898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32ac0c6710 con 0x7f32ac0cfc30 2024-01-15T18:15:17.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.305+0000 7f32b0c00640 1 -- 172.21.15.27:0/1169085898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32ac04e850 con 0x7f32ac0cfc30 2024-01-15T18:15:17.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.305+0000 7f32a2000640 1 -- 172.21.15.27:0/1169085898 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7f3294004b90 con 0x7f32ac0cfc30 2024-01-15T18:15:17.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.305+0000 7f32a2000640 1 --2- 172.21.15.27:0/1169085898 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3284067430 0x7f32840698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:17.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.305+0000 7f32a2000640 1 -- 172.21.15.27:0/1169085898 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f3294082210 con 0x7f32ac0cfc30 2024-01-15T18:15:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.305+0000 7f32a9e00640 1 --2- 172.21.15.27:0/1169085898 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3284067430 0x7f32840698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.309+0000 7f32a9e00640 1 --2- 172.21.15.27:0/1169085898 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3284067430 0x7f32840698f0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f32ac0c74c0 tx=0x7f329c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.309+0000 7f32a2000640 1 -- 172.21.15.27:0/1169085898 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3294050dc0 con 0x7f32ac0cfc30 2024-01-15T18:15:17.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.501+0000 7f32b0c00640 1 -- 172.21.15.27:0/1169085898 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone519320", "target": ["mon-mgr", ""]}) v1 -- 0x7f32ac067060 con 0x7f3284067430 2024-01-15T18:15:17.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.513+0000 7f32a2000640 1 -- 172.21.15.27:0/1169085898 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f32ac067060 con 0x7f3284067430 2024-01-15T18:15:17.515 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:15:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:15:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "canceled", 2024-01-15T18:15:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:15:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:15:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:15:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:15:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "failure": { 2024-01-15T18:15:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "errno": "4", 2024-01-15T18:15:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:15:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:17.516 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:15:17.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.517+0000 7f32b0c00640 1 -- 172.21.15.27:0/1169085898 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3284067430 msgr2=0x7f32840698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:17.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.517+0000 7f32b0c00640 1 --2- 172.21.15.27:0/1169085898 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3284067430 0x7f32840698f0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f32ac0c74c0 tx=0x7f329c008040 comp rx=0 tx=0).stop 2024-01-15T18:15:17.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.517+0000 7f32b0c00640 1 -- 172.21.15.27:0/1169085898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32ac0cfc30 msgr2=0x7f32ac0c59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:17.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.517+0000 7f32b0c00640 1 --2- 172.21.15.27:0/1169085898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32ac0cfc30 0x7f32ac0c59e0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f329400c4b0 tx=0x7f329400c970 comp rx=0 tx=0).stop 2024-01-15T18:15:17.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.517+0000 7f32b0c00640 1 -- 172.21.15.27:0/1169085898 shutdown_connections 2024-01-15T18:15:17.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.517+0000 7f32b0c00640 1 --2- 172.21.15.27:0/1169085898 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3284067430 0x7f32840698f0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:17.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.517+0000 7f32b0c00640 1 --2- 172.21.15.27:0/1169085898 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32ac15d680 0x7f32ac15f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:17.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.517+0000 7f32b0c00640 1 --2- 172.21.15.27:0/1169085898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32ac0cfc30 0x7f32ac0c59e0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:17.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.517+0000 7f32b0c00640 1 --2- 172.21.15.27:0/1169085898 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32ac0ccf80 0x7f32ac1605a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:17.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.517+0000 7f32b0c00640 1 -- 172.21.15.27:0/1169085898 >> 172.21.15.27:0/1169085898 conn(0x7f32ac0d0e80 msgr2=0x7f32ac06b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:17.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.517+0000 7f32b0c00640 1 -- 172.21.15.27:0/1169085898 shutdown_connections 2024-01-15T18:15:17.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.517+0000 7f32b0c00640 1 -- 172.21.15.27:0/1169085898 wait complete. 2024-01-15T18:15:17.538 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:15:17.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.677+0000 7f6c5aa00640 1 Processor -- start 2024-01-15T18:15:17.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.677+0000 7f6c5aa00640 1 -- start start 2024-01-15T18:15:17.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.681+0000 7f6c5aa00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6c5415b180 0x7f6c540cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:17.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.681+0000 7f6c5aa00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c540ce7d0 con 0x7f6c540cc9a0 2024-01-15T18:15:17.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.681+0000 7f6c5aa00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c540d1b40 con 0x7f6c540cd7e0 2024-01-15T18:15:17.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.681+0000 7f6c5aa00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c540d1e20 con 0x7f6c5415b180 2024-01-15T18:15:17.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.681+0000 7f6c58a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6c540cc9a0 0x7f6c540b10e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60794/0 (socket says 172.21.15.27:60794) 2024-01-15T18:15:17.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.681+0000 7f6c58a00640 1 -- 172.21.15.27:0/1627347070 learned_addr learned my addr 172.21.15.27:0/1627347070 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:17.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.681+0000 7f6c53e00640 1 --2- 172.21.15.27:0/1627347070 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6c5415b180 0x7f6c540cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:17.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.681+0000 7f6c52a00640 1 -- 172.21.15.27:0/1627347070 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292543420 0 0) 0x7f6c540ce7d0 con 0x7f6c540cc9a0 2024-01-15T18:15:17.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.681+0000 7f6c52a00640 1 -- 172.21.15.27:0/1627347070 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c34003530 con 0x7f6c540cc9a0 2024-01-15T18:15:17.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.681+0000 7f6c52a00640 1 -- 172.21.15.27:0/1627347070 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318056502 0 0) 0x7f6c540d1b40 con 0x7f6c540cd7e0 2024-01-15T18:15:17.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.681+0000 7f6c52a00640 1 -- 172.21.15.27:0/1627347070 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c540ce7d0 con 0x7f6c540cd7e0 2024-01-15T18:15:17.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.681+0000 7f6c52a00640 1 -- 172.21.15.27:0/1627347070 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1290446524 0 0) 0x7f6c34003530 con 0x7f6c540cc9a0 2024-01-15T18:15:17.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.681+0000 7f6c53e00640 1 -- 172.21.15.27:0/1627347070 >> v1:172.21.15.93:6789/0 conn(0x7f6c540cd7e0 legacy=0x7f6c540b09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:17.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.681+0000 7f6c53e00640 1 -- 172.21.15.27:0/1627347070 >> v1:172.21.15.27:6789/0 conn(0x7f6c540cc9a0 legacy=0x7f6c540b10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:17.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.681+0000 7f6c53e00640 1 -- 172.21.15.27:0/1627347070 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c540cebf0 con 0x7f6c5415b180 2024-01-15T18:15:17.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.685+0000 7f6c53e00640 1 --2- 172.21.15.27:0/1627347070 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6c5415b180 0x7f6c540cfa90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6c440097f0 tx=0x7f6c4401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51eff04cf75437e7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:17.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.685+0000 7f6c52a00640 1 -- 172.21.15.27:0/1627347070 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c001a00 con 0x7f6c5415b180 2024-01-15T18:15:17.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.685+0000 7f6c52a00640 1 -- 172.21.15.27:0/1627347070 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c440a7050 con 0x7f6c5415b180 2024-01-15T18:15:17.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.685+0000 7f6c52a00640 1 -- 172.21.15.27:0/1627347070 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c440ab600 con 0x7f6c5415b180 2024-01-15T18:15:17.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.685+0000 7f6c5aa00640 1 -- 172.21.15.27:0/1627347070 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6c5415b180 msgr2=0x7f6c540cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:17.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.685+0000 7f6c5aa00640 1 --2- 172.21.15.27:0/1627347070 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6c5415b180 0x7f6c540cfa90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6c440097f0 tx=0x7f6c4401c3e0 comp rx=0 tx=0).stop 2024-01-15T18:15:17.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.685+0000 7f6c5aa00640 1 -- 172.21.15.27:0/1627347070 shutdown_connections 2024-01-15T18:15:17.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.685+0000 7f6c5aa00640 1 --2- 172.21.15.27:0/1627347070 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6c5415b180 0x7f6c540cfa90 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:17.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.685+0000 7f6c5aa00640 1 -- 172.21.15.27:0/1627347070 >> 172.21.15.27:0/1627347070 conn(0x7f6c540d0e80 msgr2=0x7f6c540d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:17.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.685+0000 7f6c5aa00640 1 -- 172.21.15.27:0/1627347070 shutdown_connections 2024-01-15T18:15:17.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.685+0000 7f6c5aa00640 1 -- 172.21.15.27:0/1627347070 wait complete. 2024-01-15T18:15:17.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.685+0000 7f6c5aa00640 1 Processor -- start 2024-01-15T18:15:17.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.685+0000 7f6c5aa00640 1 -- start start 2024-01-15T18:15:17.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.689+0000 7f6c5aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c540cc9a0 0x7f6c5414a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:17.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.689+0000 7f6c5aa00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6c540cd7e0 0x7f6c54147b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:17.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.689+0000 7f6c5aa00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6c5415b180 0x7f6c54154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:17.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.689+0000 7f6c5aa00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c540ceed0 con 0x7f6c540cc9a0 2024-01-15T18:15:17.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.689+0000 7f6c5aa00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c54150b10 con 0x7f6c540cd7e0 2024-01-15T18:15:17.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.689+0000 7f6c5aa00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c54150df0 con 0x7f6c5415b180 2024-01-15T18:15:17.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.689+0000 7f6c53e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c540cc9a0 0x7f6c5414a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:17.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.689+0000 7f6c53e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c540cc9a0 0x7f6c5414a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51348/0 (socket says 172.21.15.27:51348) 2024-01-15T18:15:17.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.689+0000 7f6c53e00640 1 -- 172.21.15.27:0/3424033988 learned_addr learned my addr 172.21.15.27:0/3424033988 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:17.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.689+0000 7f6c53400640 1 --2- 172.21.15.27:0/3424033988 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6c540cd7e0 0x7f6c54147b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:17.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.689+0000 7f6c58a00640 1 --2- 172.21.15.27:0/3424033988 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6c5415b180 0x7f6c54154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:17.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.689+0000 7f6c53e00640 1 -- 172.21.15.27:0/3424033988 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6c5415b180 msgr2=0x7f6c54154d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:17.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.689+0000 7f6c53e00640 1 --2- 172.21.15.27:0/3424033988 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6c5415b180 0x7f6c54154d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:17.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.689+0000 7f6c53e00640 1 -- 172.21.15.27:0/3424033988 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6c540cd7e0 msgr2=0x7f6c54147b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:17.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.689+0000 7f6c53e00640 1 --2- 172.21.15.27:0/3424033988 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6c540cd7e0 0x7f6c54147b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:17.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.689+0000 7f6c53e00640 1 -- 172.21.15.27:0/3424033988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c44009420 con 0x7f6c540cc9a0 2024-01-15T18:15:17.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.689+0000 7f6c53e00640 1 --2- 172.21.15.27:0/3424033988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c540cc9a0 0x7f6c5414a050 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f6c44000c10 tx=0x7f6c44019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:17.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.689+0000 7f6c4be00640 1 -- 172.21.15.27:0/3424033988 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c440b4400 con 0x7f6c540cc9a0 2024-01-15T18:15:17.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.689+0000 7f6c4be00640 1 -- 172.21.15.27:0/3424033988 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c440b4560 con 0x7f6c540cc9a0 2024-01-15T18:15:17.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.689+0000 7f6c4be00640 1 -- 172.21.15.27:0/3424033988 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c440b48d0 con 0x7f6c540cc9a0 2024-01-15T18:15:17.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.689+0000 7f6c5aa00640 1 -- 172.21.15.27:0/3424033988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c541510d0 con 0x7f6c540cc9a0 2024-01-15T18:15:17.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.689+0000 7f6c5aa00640 1 -- 172.21.15.27:0/3424033988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c54151690 con 0x7f6c540cc9a0 2024-01-15T18:15:17.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.689+0000 7f6c5aa00640 1 -- 172.21.15.27:0/3424033988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c5404e850 con 0x7f6c540cc9a0 2024-01-15T18:15:17.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.693+0000 7f6c4be00640 1 -- 172.21.15.27:0/3424033988 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7f6c440b2040 con 0x7f6c540cc9a0 2024-01-15T18:15:17.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.693+0000 7f6c4be00640 1 --2- 172.21.15.27:0/3424033988 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6c2c067430 0x7f6c2c0698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:17.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.693+0000 7f6c4be00640 1 -- 172.21.15.27:0/3424033988 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f6c4411cae0 con 0x7f6c540cc9a0 2024-01-15T18:15:17.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.693+0000 7f6c53400640 1 --2- 172.21.15.27:0/3424033988 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6c2c067430 0x7f6c2c0698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:17.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.693+0000 7f6c53400640 1 --2- 172.21.15.27:0/3424033988 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6c2c067430 0x7f6c2c0698f0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f6c540cb420 tx=0x7f6c3c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:17.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.697+0000 7f6c4be00640 1 -- 172.21.15.27:0/3424033988 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6c440eb820 con 0x7f6c540cc9a0 2024-01-15T18:15:17.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.873+0000 7f6c5aa00640 1 -- 172.21.15.27:0/3424033988 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c54169640 con 0x7f6c2c067430 2024-01-15T18:15:17.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.877+0000 7f6c4be00640 1 -- 172.21.15.27:0/3424033988 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f6c3c001440 con 0x7f6c2c067430 2024-01-15T18:15:17.878 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:15:17.878 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:17.878 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-15T18:15:17.878 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:17.878 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:15:17.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.877+0000 7f6c5aa00640 1 -- 172.21.15.27:0/3424033988 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6c2c067430 msgr2=0x7f6c2c0698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:17.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.877+0000 7f6c5aa00640 1 --2- 172.21.15.27:0/3424033988 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6c2c067430 0x7f6c2c0698f0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f6c540cb420 tx=0x7f6c3c009040 comp rx=0 tx=0).stop 2024-01-15T18:15:17.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.877+0000 7f6c5aa00640 1 -- 172.21.15.27:0/3424033988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c540cc9a0 msgr2=0x7f6c5414a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:17.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.877+0000 7f6c5aa00640 1 --2- 172.21.15.27:0/3424033988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c540cc9a0 0x7f6c5414a050 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f6c44000c10 tx=0x7f6c44019180 comp rx=0 tx=0).stop 2024-01-15T18:15:17.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.877+0000 7f6c5aa00640 1 -- 172.21.15.27:0/3424033988 shutdown_connections 2024-01-15T18:15:17.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.877+0000 7f6c5aa00640 1 --2- 172.21.15.27:0/3424033988 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6c2c067430 0x7f6c2c0698f0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:17.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.877+0000 7f6c5aa00640 1 --2- 172.21.15.27:0/3424033988 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6c5415b180 0x7f6c54154d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:17.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.877+0000 7f6c5aa00640 1 --2- 172.21.15.27:0/3424033988 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6c540cd7e0 0x7f6c54147b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:17.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.881+0000 7f6c5aa00640 1 --2- 172.21.15.27:0/3424033988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c540cc9a0 0x7f6c5414a050 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:17.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.881+0000 7f6c5aa00640 1 -- 172.21.15.27:0/3424033988 >> 172.21.15.27:0/3424033988 conn(0x7f6c540d0e80 msgr2=0x7f6c540ae9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:17.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.881+0000 7f6c5aa00640 1 -- 172.21.15.27:0/3424033988 shutdown_connections 2024-01-15T18:15:17.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:17.881+0000 7f6c5aa00640 1 -- 172.21.15.27:0/3424033988 wait complete. 2024-01-15T18:15:17.901 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-15T18:15:18.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.041+0000 7fd3c4200640 1 Processor -- start 2024-01-15T18:15:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.045+0000 7fd3c4200640 1 -- start start 2024-01-15T18:15:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.045+0000 7fd3c4200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fd3bc158900 0x7fd3bc15f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.045+0000 7fd3c4200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3bc0b1cb0 con 0x7fd3bc0cf2e0 2024-01-15T18:15:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.045+0000 7fd3c4200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3bc0b1f00 con 0x7fd3bc15bf00 2024-01-15T18:15:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.045+0000 7fd3c4200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3bc0b21e0 con 0x7fd3bc158900 2024-01-15T18:15:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.045+0000 7fd3c2200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd3bc0cf2e0 0x7fd3bc0701e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60798/0 (socket says 172.21.15.27:60798) 2024-01-15T18:15:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.045+0000 7fd3c2200640 1 -- 172.21.15.27:0/57880596 learned_addr learned my addr 172.21.15.27:0/57880596 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.045+0000 7fd3c1800640 1 --2- 172.21.15.27:0/57880596 >> v2:172.21.15.93:3300/0 conn(0x7fd3bc158900 0x7fd3bc15f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.045+0000 7fd3bbe00640 1 -- 172.21.15.27:0/57880596 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087776668 0 0) 0x7fd3bc0b1cb0 con 0x7fd3bc0cf2e0 2024-01-15T18:15:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.045+0000 7fd3bbe00640 1 -- 172.21.15.27:0/57880596 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd39c003530 con 0x7fd3bc0cf2e0 2024-01-15T18:15:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.045+0000 7fd3bbe00640 1 -- 172.21.15.27:0/57880596 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159071149 0 0) 0x7fd3bc0b1f00 con 0x7fd3bc15bf00 2024-01-15T18:15:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.045+0000 7fd3bbe00640 1 -- 172.21.15.27:0/57880596 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3bc0b1cb0 con 0x7fd3bc15bf00 2024-01-15T18:15:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.045+0000 7fd3c1800640 1 -- 172.21.15.27:0/57880596 >> v1:172.21.15.93:6789/0 conn(0x7fd3bc15bf00 legacy=0x7fd3bc0cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.045+0000 7fd3c1800640 1 -- 172.21.15.27:0/57880596 >> v1:172.21.15.27:6789/0 conn(0x7fd3bc0cf2e0 legacy=0x7fd3bc0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.045+0000 7fd3c1800640 1 -- 172.21.15.27:0/57880596 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3bc15ea70 con 0x7fd3bc158900 2024-01-15T18:15:18.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.049+0000 7fd3c1800640 1 --2- 172.21.15.27:0/57880596 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd3bc158900 0x7fd3bc15f550 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac0097c0 tx=0x7fd3ac01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65e32b3832c9d6f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:18.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.049+0000 7fd3bbe00640 1 -- 172.21.15.27:0/57880596 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac0ab600 con 0x7fd3bc158900 2024-01-15T18:15:18.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.049+0000 7fd3bbe00640 1 -- 172.21.15.27:0/57880596 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3ac0a7050 con 0x7fd3bc158900 2024-01-15T18:15:18.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.049+0000 7fd3bbe00640 1 -- 172.21.15.27:0/57880596 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac0ab980 con 0x7fd3bc158900 2024-01-15T18:15:18.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.049+0000 7fd3c4200640 1 -- 172.21.15.27:0/57880596 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd3bc158900 msgr2=0x7fd3bc15f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:18.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.049+0000 7fd3c4200640 1 --2- 172.21.15.27:0/57880596 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd3bc158900 0x7fd3bc15f550 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac0097c0 tx=0x7fd3ac01c480 comp rx=0 tx=0).stop 2024-01-15T18:15:18.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.049+0000 7fd3c4200640 1 -- 172.21.15.27:0/57880596 shutdown_connections 2024-01-15T18:15:18.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.049+0000 7fd3c4200640 1 --2- 172.21.15.27:0/57880596 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd3bc158900 0x7fd3bc15f550 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:18.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.049+0000 7fd3c4200640 1 -- 172.21.15.27:0/57880596 >> 172.21.15.27:0/57880596 conn(0x7fd3bc0cb700 msgr2=0x7fd3bc0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:18.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.049+0000 7fd3c4200640 1 -- 172.21.15.27:0/57880596 shutdown_connections 2024-01-15T18:15:18.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.049+0000 7fd3c4200640 1 -- 172.21.15.27:0/57880596 wait complete. 2024-01-15T18:15:18.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3c4200640 1 Processor -- start 2024-01-15T18:15:18.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3c4200640 1 -- start start 2024-01-15T18:15:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3c4200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3bc0cf2e0 0x7fd3bc06c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3c4200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd3bc158900 0x7fd3bc0ae9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3c4200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd3bc15bf00 0x7fd3bc0cdda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3c4200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc071620 con 0x7fd3bc0cf2e0 2024-01-15T18:15:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3c4200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc15a4c0 con 0x7fd3bc15bf00 2024-01-15T18:15:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3c4200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3bc14f2e0 con 0x7fd3bc158900 2024-01-15T18:15:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3c1800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3bc0cf2e0 0x7fd3bc06c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3c1800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3bc0cf2e0 0x7fd3bc06c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51354/0 (socket says 172.21.15.27:51354) 2024-01-15T18:15:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3c1800640 1 -- 172.21.15.27:0/3364143441 learned_addr learned my addr 172.21.15.27:0/3364143441 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3c2200640 1 --2- 172.21.15.27:0/3364143441 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd3bc15bf00 0x7fd3bc0cdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3c0e00640 1 --2- 172.21.15.27:0/3364143441 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd3bc158900 0x7fd3bc0ae9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3c1800640 1 -- 172.21.15.27:0/3364143441 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd3bc158900 msgr2=0x7fd3bc0ae9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3c1800640 1 --2- 172.21.15.27:0/3364143441 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd3bc158900 0x7fd3bc0ae9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3c1800640 1 -- 172.21.15.27:0/3364143441 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd3bc15bf00 msgr2=0x7fd3bc0cdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3c1800640 1 --2- 172.21.15.27:0/3364143441 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd3bc15bf00 0x7fd3bc0cdda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3c1800640 1 -- 172.21.15.27:0/3364143441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ac009420 con 0x7fd3bc0cf2e0 2024-01-15T18:15:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3c0e00640 1 --2- 172.21.15.27:0/3364143441 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd3bc158900 0x7fd3bc0ae9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3c1800640 1 --2- 172.21.15.27:0/3364143441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3bc0cf2e0 0x7fd3bc06c430 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac019090 tx=0x7fd3ac01acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3b9600640 1 -- 172.21.15.27:0/3364143441 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac0b7450 con 0x7fd3bc0cf2e0 2024-01-15T18:15:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3b9600640 1 -- 172.21.15.27:0/3364143441 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3ac0b75d0 con 0x7fd3bc0cf2e0 2024-01-15T18:15:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3b9600640 1 -- 172.21.15.27:0/3364143441 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac0b7980 con 0x7fd3bc0cf2e0 2024-01-15T18:15:18.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3c4200640 1 -- 172.21.15.27:0/3364143441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3bc15ad00 con 0x7fd3bc0cf2e0 2024-01-15T18:15:18.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3c4200640 1 -- 172.21.15.27:0/3364143441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3bc15a190 con 0x7fd3bc0cf2e0 2024-01-15T18:15:18.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.053+0000 7fd3c4200640 1 -- 172.21.15.27:0/3364143441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3bc15ab40 con 0x7fd3bc0cf2e0 2024-01-15T18:15:18.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.057+0000 7fd3b9600640 1 -- 172.21.15.27:0/3364143441 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7fd3ac0b7ae0 con 0x7fd3bc0cf2e0 2024-01-15T18:15:18.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.057+0000 7fd3b9600640 1 --2- 172.21.15.27:0/3364143441 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd394067430 0x7fd3940698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:18.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.057+0000 7fd3b9600640 1 -- 172.21.15.27:0/3364143441 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fd3ac11d460 con 0x7fd3bc0cf2e0 2024-01-15T18:15:18.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.057+0000 7fd3c0e00640 1 --2- 172.21.15.27:0/3364143441 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd394067430 0x7fd3940698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:18.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.057+0000 7fd3c0e00640 1 --2- 172.21.15.27:0/3364143441 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd394067430 0x7fd3940698f0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc067560 tx=0x7fd3a4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:18.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.061+0000 7fd3b9600640 1 -- 172.21.15.27:0/3364143441 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd3ac0ec0f0 con 0x7fd3bc0cf2e0 2024-01-15T18:15:18.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.241+0000 7fd3c4200640 1 -- 172.21.15.27:0/3364143441 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd3bc0cdbc0 con 0x7fd394067430 2024-01-15T18:15:18.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.257+0000 7fd3b9600640 1 -- 172.21.15.27:0/3364143441 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1029 (secure 0 0 0) 0x7fd3bc0cdbc0 con 0x7fd394067430 2024-01-15T18:15:18.259 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:15:18.259 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:15:18.259 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:18.260 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone51937" 2024-01-15T18:15:18.260 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:18.260 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:18.260 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone51931" 2024-01-15T18:15:18.260 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:18.260 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:18.260 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone519315" 2024-01-15T18:15:18.260 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:18.260 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:18.260 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone519316" 2024-01-15T18:15:18.260 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:18.260 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:18.260 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone519319" 2024-01-15T18:15:18.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:18.261 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:18.261 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone51932" 2024-01-15T18:15:18.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:18.261 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:18.261 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone519310" 2024-01-15T18:15:18.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:18.261 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:18.261 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sv1" 2024-01-15T18:15:18.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:18.261 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:18.261 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone51935" 2024-01-15T18:15:18.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:18.261 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:18.261 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone519311" 2024-01-15T18:15:18.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:18.262 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:18.262 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone519312" 2024-01-15T18:15:18.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:18.262 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:18.262 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone519313" 2024-01-15T18:15:18.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:18.262 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:18.262 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone51938" 2024-01-15T18:15:18.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:18.262 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:18.262 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone519320" 2024-01-15T18:15:18.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:18.262 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:18.262 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone51933" 2024-01-15T18:15:18.263 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:18.263 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:18.263 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone51934" 2024-01-15T18:15:18.263 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:18.263 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:18.263 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone519318" 2024-01-15T18:15:18.263 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:18.263 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:18.263 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone51939" 2024-01-15T18:15:18.263 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:18.263 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:18.263 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone519317" 2024-01-15T18:15:18.263 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:18.263 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:18.264 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone51936" 2024-01-15T18:15:18.264 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:18.264 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:18.264 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone519314" 2024-01-15T18:15:18.264 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:18.264 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:15:18.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.261+0000 7fd3c4200640 1 -- 172.21.15.27:0/3364143441 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd394067430 msgr2=0x7fd3940698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:18.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.261+0000 7fd3c4200640 1 --2- 172.21.15.27:0/3364143441 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd394067430 0x7fd3940698f0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc067560 tx=0x7fd3a4009040 comp rx=0 tx=0).stop 2024-01-15T18:15:18.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.261+0000 7fd3c4200640 1 -- 172.21.15.27:0/3364143441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3bc0cf2e0 msgr2=0x7fd3bc06c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:18.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.261+0000 7fd3c4200640 1 --2- 172.21.15.27:0/3364143441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3bc0cf2e0 0x7fd3bc06c430 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac019090 tx=0x7fd3ac01acc0 comp rx=0 tx=0).stop 2024-01-15T18:15:18.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.261+0000 7fd3c4200640 1 -- 172.21.15.27:0/3364143441 shutdown_connections 2024-01-15T18:15:18.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.261+0000 7fd3c4200640 1 --2- 172.21.15.27:0/3364143441 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd394067430 0x7fd3940698f0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:18.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.261+0000 7fd3c4200640 1 --2- 172.21.15.27:0/3364143441 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd3bc15bf00 0x7fd3bc0cdda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:18.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.261+0000 7fd3c4200640 1 --2- 172.21.15.27:0/3364143441 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd3bc158900 0x7fd3bc0ae9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:18.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.261+0000 7fd3c4200640 1 --2- 172.21.15.27:0/3364143441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3bc0cf2e0 0x7fd3bc06c430 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:18.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.261+0000 7fd3c4200640 1 -- 172.21.15.27:0/3364143441 >> 172.21.15.27:0/3364143441 conn(0x7fd3bc0cb700 msgr2=0x7fd3bc0cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:18.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.261+0000 7fd3c4200640 1 -- 172.21.15.27:0/3364143441 shutdown_connections 2024-01-15T18:15:18.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.261+0000 7fd3c4200640 1 -- 172.21.15.27:0/3364143441 wait complete. 2024-01-15T18:15:18.282 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone51937 2024-01-15T18:15:18.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.425+0000 7fe399c00640 1 Processor -- start 2024-01-15T18:15:18.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.425+0000 7fe399c00640 1 -- start start 2024-01-15T18:15:18.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe399c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe39415be40 0x7fe3940cf940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:18.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe399c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3940cebf0 con 0x7fe394072360 2024-01-15T18:15:18.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe399c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3940b46e0 con 0x7fe3940b4300 2024-01-15T18:15:18.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe399c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe39406a4f0 con 0x7fe39415be40 2024-01-15T18:15:18.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe392000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe394072360 0x7fe39415b730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60808/0 (socket says 172.21.15.27:60808) 2024-01-15T18:15:18.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe392000640 1 -- 172.21.15.27:0/1063380607 learned_addr learned my addr 172.21.15.27:0/1063380607 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:18.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe393e00640 1 --2- 172.21.15.27:0/1063380607 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe39415be40 0x7fe3940cf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:18.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe391600640 1 -- 172.21.15.27:0/1063380607 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212669313 0 0) 0x7fe3940cebf0 con 0x7fe394072360 2024-01-15T18:15:18.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe391600640 1 -- 172.21.15.27:0/1063380607 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe374003530 con 0x7fe394072360 2024-01-15T18:15:18.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe391600640 1 -- 172.21.15.27:0/1063380607 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2848988358 0 0) 0x7fe3940b46e0 con 0x7fe3940b4300 2024-01-15T18:15:18.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe391600640 1 -- 172.21.15.27:0/1063380607 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3940cebf0 con 0x7fe3940b4300 2024-01-15T18:15:18.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe391600640 1 -- 172.21.15.27:0/1063380607 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2738786990 0 0) 0x7fe374003530 con 0x7fe394072360 2024-01-15T18:15:18.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe391600640 1 -- 172.21.15.27:0/1063380607 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe39415be40 msgr2=0x7fe3940cf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:18.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe391600640 1 --2- 172.21.15.27:0/1063380607 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe39415be40 0x7fe3940cf940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:18.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe391600640 1 -- 172.21.15.27:0/1063380607 >> v1:172.21.15.93:6789/0 conn(0x7fe3940b4300 legacy=0x7fe394071c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:18.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe391600640 1 -- 172.21.15.27:0/1063380607 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3940cee60 con 0x7fe394072360 2024-01-15T18:15:18.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe391600640 1 -- 172.21.15.27:0/1063380607 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fe37c001680 con 0x7fe394072360 2024-01-15T18:15:18.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe391600640 1 -- 172.21.15.27:0/1063380607 >> v1:172.21.15.27:6789/0 conn(0x7fe394072360 legacy=0x7fe39415b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:18.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe391600640 1 --2- 172.21.15.27:0/1063380607 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe39415be40 0x7fe374003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:18.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe391600640 1 --2- 172.21.15.27:0/1063380607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3740040a0 0x7fe3740047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:18.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe391600640 1 --2- 172.21.15.27:0/1063380607 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe374004d10 0x7fe374005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:18.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe391600640 1 -- 172.21.15.27:0/1063380607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39406a4f0 con 0x7fe3740040a0 2024-01-15T18:15:18.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe391600640 1 -- 172.21.15.27:0/1063380607 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe37400ba70 con 0x7fe374004d10 2024-01-15T18:15:18.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe391600640 1 -- 172.21.15.27:0/1063380607 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe37400bc30 con 0x7fe39415be40 2024-01-15T18:15:18.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe392000640 1 --2- 172.21.15.27:0/1063380607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3740040a0 0x7fe3740047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:18.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe392a00640 1 --2- 172.21.15.27:0/1063380607 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe39415be40 0x7fe374003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:18.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe393e00640 1 --2- 172.21.15.27:0/1063380607 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe374004d10 0x7fe374005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:18.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe392000640 1 -- 172.21.15.27:0/1063380607 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe39415be40 msgr2=0x7fe374003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:18.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe392000640 1 --2- 172.21.15.27:0/1063380607 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe39415be40 0x7fe374003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:18.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe392000640 1 -- 172.21.15.27:0/1063380607 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe374004d10 msgr2=0x7fe374005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:18.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe392000640 1 --2- 172.21.15.27:0/1063380607 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe374004d10 0x7fe374005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:18.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.429+0000 7fe392000640 1 -- 172.21.15.27:0/1063380607 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe374003530 con 0x7fe3740040a0 2024-01-15T18:15:18.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.433+0000 7fe392000640 1 --2- 172.21.15.27:0/1063380607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3740040a0 0x7fe3740047d0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fe37c00c160 tx=0x7fe37c01f310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c316ad6e7f51060b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:18.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.433+0000 7fe391600640 1 -- 172.21.15.27:0/1063380607 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37c005ae0 con 0x7fe3740040a0 2024-01-15T18:15:18.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.433+0000 7fe391600640 1 -- 172.21.15.27:0/1063380607 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe37c0ab070 con 0x7fe3740040a0 2024-01-15T18:15:18.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.433+0000 7fe391600640 1 -- 172.21.15.27:0/1063380607 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37c007070 con 0x7fe3740040a0 2024-01-15T18:15:18.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.433+0000 7fe399c00640 1 -- 172.21.15.27:0/1063380607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3740040a0 msgr2=0x7fe3740047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:18.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.433+0000 7fe399c00640 1 --2- 172.21.15.27:0/1063380607 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3740040a0 0x7fe3740047d0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fe37c00c160 tx=0x7fe37c01f310 comp rx=0 tx=0).stop 2024-01-15T18:15:18.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.433+0000 7fe393e00640 1 -- 172.21.15.27:0/1063380607 reap_dead start 2024-01-15T18:15:18.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.433+0000 7fe399c00640 1 -- 172.21.15.27:0/1063380607 shutdown_connections 2024-01-15T18:15:18.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.433+0000 7fe399c00640 1 -- 172.21.15.27:0/1063380607 >> 172.21.15.27:0/1063380607 conn(0x7fe394099e10 msgr2=0x7fe3940b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:18.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.433+0000 7fe399c00640 1 -- 172.21.15.27:0/1063380607 shutdown_connections 2024-01-15T18:15:18.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.433+0000 7fe399c00640 1 -- 172.21.15.27:0/1063380607 wait complete. 2024-01-15T18:15:18.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.437+0000 7fe399c00640 1 Processor -- start 2024-01-15T18:15:18.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.437+0000 7fe399c00640 1 -- start start 2024-01-15T18:15:18.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.437+0000 7fe399c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe3740040a0 0x7fe394153410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:18.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.437+0000 7fe399c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe39414ee70 0x7fe394154e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:18.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.437+0000 7fe399c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3941565e0 0x7fe394156a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:18.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.437+0000 7fe399c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe394156fb0 con 0x7fe3941565e0 2024-01-15T18:15:18.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.437+0000 7fe399c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39414f2f0 con 0x7fe39414ee70 2024-01-15T18:15:18.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.437+0000 7fe399c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3940b0c30 con 0x7fe3740040a0 2024-01-15T18:15:18.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.437+0000 7fe393e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3941565e0 0x7fe394156a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:18.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.437+0000 7fe392a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe3740040a0 0x7fe394153410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:18.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.437+0000 7fe392000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe39414ee70 0x7fe394154e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:18.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.437+0000 7fe392a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe3740040a0 0x7fe394153410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:33910/0 (socket says 172.21.15.27:33910) 2024-01-15T18:15:18.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.437+0000 7fe392a00640 1 -- 172.21.15.27:0/1526353873 learned_addr learned my addr 172.21.15.27:0/1526353873 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:18.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.437+0000 7fe392a00640 1 -- 172.21.15.27:0/1526353873 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe39414ee70 msgr2=0x7fe394154e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:18.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.437+0000 7fe392a00640 1 --2- 172.21.15.27:0/1526353873 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe39414ee70 0x7fe394154e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:18.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.437+0000 7fe392a00640 1 -- 172.21.15.27:0/1526353873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3941565e0 msgr2=0x7fe394156a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:18.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.437+0000 7fe392a00640 1 --2- 172.21.15.27:0/1526353873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3941565e0 0x7fe394156a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:18.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.437+0000 7fe392a00640 1 -- 172.21.15.27:0/1526353873 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3940b0ec0 con 0x7fe3740040a0 2024-01-15T18:15:18.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.437+0000 7fe392a00640 1 --2- 172.21.15.27:0/1526353873 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe3740040a0 0x7fe394153410 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe38400e7b0 tx=0x7fe38400ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:18.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.437+0000 7fe38aa00640 1 -- 172.21.15.27:0/1526353873 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38400ee60 con 0x7fe3740040a0 2024-01-15T18:15:18.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.437+0000 7fe38aa00640 1 -- 172.21.15.27:0/1526353873 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3840052e0 con 0x7fe3740040a0 2024-01-15T18:15:18.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.437+0000 7fe399c00640 1 -- 172.21.15.27:0/1526353873 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3940b11a0 con 0x7fe3740040a0 2024-01-15T18:15:18.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.437+0000 7fe38aa00640 1 -- 172.21.15.27:0/1526353873 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38400c900 con 0x7fe3740040a0 2024-01-15T18:15:18.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.437+0000 7fe399c00640 1 -- 172.21.15.27:0/1526353873 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3940b1760 con 0x7fe3740040a0 2024-01-15T18:15:18.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.441+0000 7fe399c00640 1 -- 172.21.15.27:0/1526353873 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe39409e950 con 0x7fe3740040a0 2024-01-15T18:15:18.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.441+0000 7fe38aa00640 1 -- 172.21.15.27:0/1526353873 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7fe38400ca60 con 0x7fe3740040a0 2024-01-15T18:15:18.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.441+0000 7fe38aa00640 1 --2- 172.21.15.27:0/1526353873 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe36c067430 0x7fe36c0698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:18.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.441+0000 7fe38aa00640 1 -- 172.21.15.27:0/1526353873 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fe384082c00 con 0x7fe3740040a0 2024-01-15T18:15:18.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.441+0000 7fe392000640 1 --2- 172.21.15.27:0/1526353873 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe36c067430 0x7fe36c0698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:18.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.441+0000 7fe392000640 1 --2- 172.21.15.27:0/1526353873 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe36c067430 0x7fe36c0698f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe37c0ae000 tx=0x7fe37c0b4040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:18.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.445+0000 7fe38aa00640 1 -- 172.21.15.27:0/1526353873 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe384051940 con 0x7fe3740040a0 2024-01-15T18:15:18.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.629+0000 7fe399c00640 1 -- 172.21.15.27:0/1526353873 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone51937", "target": ["mon-mgr", ""]}) v1 -- 0x7fe39415c490 con 0x7fe36c067430 2024-01-15T18:15:18.676 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:15:18.653+0000 7f59f5600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:18.676 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:15:18.653+0000 7f59f5600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:18.676 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:15:18.653+0000 7f59f5600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:18.676 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:15:18.653+0000 7f59f5600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:18.676 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:15:18.653+0000 7f59f5600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:18.676 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:15:18.673+0000 7f59f6000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:18.677 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:15:18.673+0000 7f59f6000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:18.677 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:15:18.673+0000 7f59f6000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:18.677 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:15:18.673+0000 7f59f6000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:18.677 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:15:18.673+0000 7f59f6000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:18.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.649+0000 7fe38aa00640 1 -- 172.21.15.27:0/1526353873 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe37c0b8040 con 0x7fe36c067430 2024-01-15T18:15:18.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.649+0000 7fe399c00640 1 -- 172.21.15.27:0/1526353873 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe36c067430 msgr2=0x7fe36c0698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:18.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.649+0000 7fe399c00640 1 --2- 172.21.15.27:0/1526353873 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe36c067430 0x7fe36c0698f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe37c0ae000 tx=0x7fe37c0b4040 comp rx=0 tx=0).stop 2024-01-15T18:15:18.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.649+0000 7fe399c00640 1 -- 172.21.15.27:0/1526353873 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe3740040a0 msgr2=0x7fe394153410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:18.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.649+0000 7fe399c00640 1 --2- 172.21.15.27:0/1526353873 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe3740040a0 0x7fe394153410 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe38400e7b0 tx=0x7fe38400ec70 comp rx=0 tx=0).stop 2024-01-15T18:15:18.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.653+0000 7fe399c00640 1 -- 172.21.15.27:0/1526353873 shutdown_connections 2024-01-15T18:15:18.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.653+0000 7fe399c00640 1 --2- 172.21.15.27:0/1526353873 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe36c067430 0x7fe36c0698f0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:18.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.653+0000 7fe399c00640 1 --2- 172.21.15.27:0/1526353873 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3941565e0 0x7fe394156a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:18.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.653+0000 7fe399c00640 1 --2- 172.21.15.27:0/1526353873 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe39414ee70 0x7fe394154e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:18.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.653+0000 7fe399c00640 1 --2- 172.21.15.27:0/1526353873 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe3740040a0 0x7fe394153410 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:18.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.653+0000 7fe399c00640 1 -- 172.21.15.27:0/1526353873 >> 172.21.15.27:0/1526353873 conn(0x7fe394099e10 msgr2=0x7fe39415e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:18.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.653+0000 7fe399c00640 1 -- 172.21.15.27:0/1526353873 shutdown_connections 2024-01-15T18:15:18.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.653+0000 7fe399c00640 1 -- 172.21.15.27:0/1526353873 wait complete. 2024-01-15T18:15:18.679 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone51931 2024-01-15T18:15:18.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.825+0000 7f03eb000640 1 Processor -- start 2024-01-15T18:15:18.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.825+0000 7f03eb000640 1 -- start start 2024-01-15T18:15:18.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.825+0000 7f03eb000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f03e40b4300 0x7f03e40cf940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:18.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.825+0000 7f03eb000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03e4071c50 0x7f03e4072070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:18.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.825+0000 7f03eb000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f03e415b730 0x7f03e415bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:18.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.825+0000 7f03eb000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03e406a4f0 con 0x7f03e415b730 2024-01-15T18:15:18.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.825+0000 7f03eb000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03e40cebf0 con 0x7f03e40b4300 2024-01-15T18:15:18.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.825+0000 7f03eb000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e40725b0 con 0x7f03e4071c50 2024-01-15T18:15:18.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.825+0000 7f03e9a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f03e415b730 0x7f03e415bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:18.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.825+0000 7f03e9a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f03e415b730 0x7f03e415bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51386/0 (socket says 172.21.15.27:51386) 2024-01-15T18:15:18.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.825+0000 7f03e9a00640 1 -- 172.21.15.27:0/2810574655 learned_addr learned my addr 172.21.15.27:0/2810574655 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:18.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.825+0000 7f03e3400640 1 --2- 172.21.15.27:0/2810574655 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03e4071c50 0x7f03e4072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:18.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.825+0000 7f03e3e00640 1 --2- 172.21.15.27:0/2810574655 >> v2:172.21.15.93:3300/0 conn(0x7f03e40b4300 0x7f03e40cf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:18.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.825+0000 7f03e9a00640 1 -- 172.21.15.27:0/2810574655 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03e4071c50 msgr2=0x7f03e4072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:18.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.829+0000 7f03e9a00640 1 --2- 172.21.15.27:0/2810574655 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03e4071c50 0x7f03e4072070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:18.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.829+0000 7f03e9a00640 1 -- 172.21.15.27:0/2810574655 >> v2:172.21.15.93:3300/0 conn(0x7f03e40b4300 msgr2=0x7f03e40cf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:18.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.829+0000 7f03e9a00640 1 --2- 172.21.15.27:0/2810574655 >> v2:172.21.15.93:3300/0 conn(0x7f03e40b4300 0x7f03e40cf940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:18.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.829+0000 7f03e9a00640 1 -- 172.21.15.27:0/2810574655 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e40b46e0 con 0x7f03e415b730 2024-01-15T18:15:18.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.829+0000 7f03e9a00640 1 --2- 172.21.15.27:0/2810574655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e415b730 0x7f03e415bb90 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f03e40ce370 tx=0x7f03dc01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d454d6104a13bc8f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:18.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.829+0000 7f03e2a00640 1 -- 172.21.15.27:0/2810574655 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03dc0af400 con 0x7f03e415b730 2024-01-15T18:15:18.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.829+0000 7f03e2a00640 1 -- 172.21.15.27:0/2810574655 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03dc0ab050 con 0x7f03e415b730 2024-01-15T18:15:18.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.829+0000 7f03e2a00640 1 -- 172.21.15.27:0/2810574655 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03dc0af780 con 0x7f03e415b730 2024-01-15T18:15:18.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.829+0000 7f03eb000640 1 -- 172.21.15.27:0/2810574655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e415b730 msgr2=0x7f03e415bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:18.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.829+0000 7f03eb000640 1 --2- 172.21.15.27:0/2810574655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e415b730 0x7f03e415bb90 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f03e40ce370 tx=0x7f03dc01f1c0 comp rx=0 tx=0).stop 2024-01-15T18:15:18.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03eb000640 1 -- 172.21.15.27:0/2810574655 shutdown_connections 2024-01-15T18:15:18.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03eb000640 1 --2- 172.21.15.27:0/2810574655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e415b730 0x7f03e415bb90 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:18.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03eb000640 1 --2- 172.21.15.27:0/2810574655 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03e4071c50 0x7f03e4072070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:18.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03eb000640 1 --2- 172.21.15.27:0/2810574655 >> v2:172.21.15.93:3300/0 conn(0x7f03e40b4300 0x7f03e40cf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:18.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03eb000640 1 -- 172.21.15.27:0/2810574655 >> 172.21.15.27:0/2810574655 conn(0x7f03e4099e10 msgr2=0x7f03e40b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:18.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03eb000640 1 -- 172.21.15.27:0/2810574655 shutdown_connections 2024-01-15T18:15:18.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03eb000640 1 -- 172.21.15.27:0/2810574655 wait complete. 2024-01-15T18:15:18.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03eb000640 1 Processor -- start 2024-01-15T18:15:18.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03eb000640 1 -- start start 2024-01-15T18:15:18.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03eb000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03e4071c50 0x7f03e4154a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:18.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03eb000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e40b4300 0x7f03e40ae730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:18.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03eb000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03e415b730 0x7f03e40aec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:18.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03eb000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e40cba60 con 0x7f03e40b4300 2024-01-15T18:15:18.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03eb000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e40cb4a0 con 0x7f03e415b730 2024-01-15T18:15:18.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03eb000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e40cb780 con 0x7f03e4071c50 2024-01-15T18:15:18.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03e3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e40b4300 0x7f03e40ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:18.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03e3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e40b4300 0x7f03e40ae730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51390/0 (socket says 172.21.15.27:51390) 2024-01-15T18:15:18.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03e3400640 1 -- 172.21.15.27:0/500919276 learned_addr learned my addr 172.21.15.27:0/500919276 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:18.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03e9a00640 1 --2- 172.21.15.27:0/500919276 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03e415b730 0x7f03e40aec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:18.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03e3e00640 1 --2- 172.21.15.27:0/500919276 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03e4071c50 0x7f03e4154a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:18.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03e3400640 1 -- 172.21.15.27:0/500919276 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03e4071c50 msgr2=0x7f03e4154a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:18.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03e3400640 1 --2- 172.21.15.27:0/500919276 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03e4071c50 0x7f03e4154a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:18.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03e3400640 1 -- 172.21.15.27:0/500919276 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03e415b730 msgr2=0x7f03e40aec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:18.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03e3400640 1 --2- 172.21.15.27:0/500919276 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03e415b730 0x7f03e40aec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:18.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03e3400640 1 -- 172.21.15.27:0/500919276 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03dc009b30 con 0x7f03e40b4300 2024-01-15T18:15:18.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.833+0000 7f03e3e00640 1 --2- 172.21.15.27:0/500919276 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03e4071c50 0x7f03e4154a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:18.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.837+0000 7f03e3400640 1 --2- 172.21.15.27:0/500919276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e40b4300 0x7f03e40ae730 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f03cc00e8b0 tx=0x7f03cc00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:18.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.837+0000 7f03dbe00640 1 -- 172.21.15.27:0/500919276 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc00c940 con 0x7f03e40b4300 2024-01-15T18:15:18.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.837+0000 7f03dbe00640 1 -- 172.21.15.27:0/500919276 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03cc00caa0 con 0x7f03e40b4300 2024-01-15T18:15:18.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.837+0000 7f03dbe00640 1 -- 172.21.15.27:0/500919276 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc00ce30 con 0x7f03e40b4300 2024-01-15T18:15:18.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.837+0000 7f03eb000640 1 -- 172.21.15.27:0/500919276 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03e40b0030 con 0x7f03e40b4300 2024-01-15T18:15:18.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.837+0000 7f03eb000640 1 -- 172.21.15.27:0/500919276 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03e4155400 con 0x7f03e40b4300 2024-01-15T18:15:18.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.837+0000 7f03eb000640 1 -- 172.21.15.27:0/500919276 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03e40afe90 con 0x7f03e40b4300 2024-01-15T18:15:18.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.837+0000 7f03dbe00640 1 -- 172.21.15.27:0/500919276 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7f03cc013470 con 0x7f03e40b4300 2024-01-15T18:15:18.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.837+0000 7f03dbe00640 1 --2- 172.21.15.27:0/500919276 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03bc067430 0x7f03bc0698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:18.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.837+0000 7f03dbe00640 1 -- 172.21.15.27:0/500919276 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f03cc0831d0 con 0x7f03e40b4300 2024-01-15T18:15:18.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.837+0000 7f03e3e00640 1 --2- 172.21.15.27:0/500919276 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03bc067430 0x7f03bc0698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:18.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.841+0000 7f03e3e00640 1 --2- 172.21.15.27:0/500919276 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03bc067430 0x7f03bc0698f0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f03d4009840 tx=0x7f03d4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:18.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:18.841+0000 7f03dbe00640 1 -- 172.21.15.27:0/500919276 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f03e40afe90 con 0x7f03e40b4300 2024-01-15T18:15:19.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.021+0000 7f03eb000640 1 -- 172.21.15.27:0/500919276 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone51931", "target": ["mon-mgr", ""]}) v1 -- 0x7f03e40cf420 con 0x7f03bc067430 2024-01-15T18:15:19.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.049+0000 7f03dbe00640 1 -- 172.21.15.27:0/500919276 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f03d4009e20 con 0x7f03bc067430 2024-01-15T18:15:19.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.049+0000 7f03eb000640 1 -- 172.21.15.27:0/500919276 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03bc067430 msgr2=0x7f03bc0698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:19.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.049+0000 7f03eb000640 1 --2- 172.21.15.27:0/500919276 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03bc067430 0x7f03bc0698f0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f03d4009840 tx=0x7f03d4008040 comp rx=0 tx=0).stop 2024-01-15T18:15:19.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.049+0000 7f03eb000640 1 -- 172.21.15.27:0/500919276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e40b4300 msgr2=0x7f03e40ae730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:19.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.049+0000 7f03eb000640 1 --2- 172.21.15.27:0/500919276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e40b4300 0x7f03e40ae730 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f03cc00e8b0 tx=0x7f03cc00ed70 comp rx=0 tx=0).stop 2024-01-15T18:15:19.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.053+0000 7f03eb000640 1 -- 172.21.15.27:0/500919276 shutdown_connections 2024-01-15T18:15:19.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.053+0000 7f03eb000640 1 --2- 172.21.15.27:0/500919276 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03bc067430 0x7f03bc0698f0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:19.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.053+0000 7f03eb000640 1 --2- 172.21.15.27:0/500919276 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03e415b730 0x7f03e40aec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:19.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.053+0000 7f03eb000640 1 --2- 172.21.15.27:0/500919276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e40b4300 0x7f03e40ae730 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:19.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.053+0000 7f03eb000640 1 --2- 172.21.15.27:0/500919276 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03e4071c50 0x7f03e4154a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:19.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.053+0000 7f03eb000640 1 -- 172.21.15.27:0/500919276 >> 172.21.15.27:0/500919276 conn(0x7f03e4099e10 msgr2=0x7f03e415c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:19.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.053+0000 7f03eb000640 1 -- 172.21.15.27:0/500919276 shutdown_connections 2024-01-15T18:15:19.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.053+0000 7f03eb000640 1 -- 172.21.15.27:0/500919276 wait complete. 2024-01-15T18:15:19.075 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone519315 2024-01-15T18:15:19.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.221+0000 7f8fad400640 1 Processor -- start 2024-01-15T18:15:19.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.221+0000 7f8fad400640 1 -- start start 2024-01-15T18:15:19.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.221+0000 7f8fad400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8fa815d480 0x7f8fa80cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:19.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.221+0000 7f8fad400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f8fa809a120 0x7f8fa80d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:19.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.221+0000 7f8fad400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa815d860 con 0x7f8fa806c740 2024-01-15T18:15:19.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.221+0000 7f8fad400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fa806b6a0 con 0x7f8fa815d480 2024-01-15T18:15:19.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.221+0000 7f8fad400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fa806b980 con 0x7f8fa809a120 2024-01-15T18:15:19.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.221+0000 7f8fa6800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8fa815d480 0x7f8fa80cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:19.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.221+0000 7f8fa5e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8fa806c740 0x7f8fa8099a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60816/0 (socket says 172.21.15.27:60816) 2024-01-15T18:15:19.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.221+0000 7f8fa5e00640 1 -- 172.21.15.27:0/1510986547 learned_addr learned my addr 172.21.15.27:0/1510986547 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:19.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.225+0000 7f8fa7200640 1 --2- 172.21.15.27:0/1510986547 >> v2:172.21.15.93:3300/0 conn(0x7f8fa809a120 0x7f8fa80d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:19.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.225+0000 7f8fa6800640 1 -- 172.21.15.27:0/1510986547 >> v2:172.21.15.93:3300/0 conn(0x7f8fa809a120 msgr2=0x7f8fa80d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:19.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.225+0000 7f8fa6800640 1 --2- 172.21.15.27:0/1510986547 >> v2:172.21.15.93:3300/0 conn(0x7f8fa809a120 0x7f8fa80d0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:19.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.225+0000 7f8fa6800640 1 -- 172.21.15.27:0/1510986547 >> v1:172.21.15.27:6789/0 conn(0x7f8fa806c740 legacy=0x7f8fa8099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:19.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.225+0000 7f8fa5400640 1 -- 172.21.15.27:0/1510986547 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669725682 0 0) 0x7f8fa815d860 con 0x7f8fa806c740 2024-01-15T18:15:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.225+0000 7f8fa6800640 1 -- 172.21.15.27:0/1510986547 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa806bc60 con 0x7f8fa815d480 2024-01-15T18:15:19.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.225+0000 7f8fa6800640 1 --2- 172.21.15.27:0/1510986547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fa815d480 0x7f8fa80cfbf0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f8f980097c0 tx=0x7f8f9801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b12821a4bef95c5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:19.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.225+0000 7f8fa5400640 1 -- 172.21.15.27:0/1510986547 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f980ab600 con 0x7f8fa815d480 2024-01-15T18:15:19.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.225+0000 7f8fa5400640 1 -- 172.21.15.27:0/1510986547 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f980a7050 con 0x7f8fa815d480 2024-01-15T18:15:19.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.225+0000 7f8fa5400640 1 -- 172.21.15.27:0/1510986547 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f980ab980 con 0x7f8fa815d480 2024-01-15T18:15:19.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fad400640 1 -- 172.21.15.27:0/1510986547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fa815d480 msgr2=0x7f8fa80cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:19.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fad400640 1 --2- 172.21.15.27:0/1510986547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fa815d480 0x7f8fa80cfbf0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f8f980097c0 tx=0x7f8f9801c410 comp rx=0 tx=0).stop 2024-01-15T18:15:19.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fad400640 1 -- 172.21.15.27:0/1510986547 shutdown_connections 2024-01-15T18:15:19.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fad400640 1 --2- 172.21.15.27:0/1510986547 >> v2:172.21.15.93:3300/0 conn(0x7f8fa809a120 0x7f8fa80d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:19.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fad400640 1 --2- 172.21.15.27:0/1510986547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fa815d480 0x7f8fa80cfbf0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:19.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fad400640 1 -- 172.21.15.27:0/1510986547 >> 172.21.15.27:0/1510986547 conn(0x7f8fa80b2a80 msgr2=0x7f8fa80b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:19.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fad400640 1 -- 172.21.15.27:0/1510986547 shutdown_connections 2024-01-15T18:15:19.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fad400640 1 -- 172.21.15.27:0/1510986547 wait complete. 2024-01-15T18:15:19.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fad400640 1 Processor -- start 2024-01-15T18:15:19.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fad400640 1 -- start start 2024-01-15T18:15:19.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fad400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fa806c740 0x7f8fa80c49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:19.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fad400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8fa809a120 0x7f8fa8152120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:19.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fad400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8fa815d480 0x7f8fa81546f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:19.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fad400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa815cd60 con 0x7f8fa806c740 2024-01-15T18:15:19.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fad400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa80c8af0 con 0x7f8fa815d480 2024-01-15T18:15:19.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fad400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa8152d30 con 0x7f8fa809a120 2024-01-15T18:15:19.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fa6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fa806c740 0x7f8fa80c49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:19.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fa6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fa806c740 0x7f8fa80c49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51394/0 (socket says 172.21.15.27:51394) 2024-01-15T18:15:19.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fa6800640 1 -- 172.21.15.27:0/2506296121 learned_addr learned my addr 172.21.15.27:0/2506296121 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:19.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fa5e00640 1 --2- 172.21.15.27:0/2506296121 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8fa809a120 0x7f8fa8152120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:19.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fa7200640 1 --2- 172.21.15.27:0/2506296121 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8fa815d480 0x7f8fa81546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:19.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fa6800640 1 -- 172.21.15.27:0/2506296121 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8fa809a120 msgr2=0x7f8fa8152120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:19.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fa6800640 1 --2- 172.21.15.27:0/2506296121 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8fa809a120 0x7f8fa8152120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:19.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fa6800640 1 -- 172.21.15.27:0/2506296121 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8fa815d480 msgr2=0x7f8fa81546f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:19.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fa6800640 1 --2- 172.21.15.27:0/2506296121 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8fa815d480 0x7f8fa81546f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:19.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.229+0000 7f8fa6800640 1 -- 172.21.15.27:0/2506296121 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f98009420 con 0x7f8fa806c740 2024-01-15T18:15:19.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.233+0000 7f8fa6800640 1 --2- 172.21.15.27:0/2506296121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fa806c740 0x7f8fa80c49b0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f8f980097c0 tx=0x7f8f980191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:19.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.233+0000 7f8f9ea00640 1 -- 172.21.15.27:0/2506296121 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f98006e20 con 0x7f8fa806c740 2024-01-15T18:15:19.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.233+0000 7f8f9ea00640 1 -- 172.21.15.27:0/2506296121 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f98004030 con 0x7f8fa806c740 2024-01-15T18:15:19.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.233+0000 7f8fad400640 1 -- 172.21.15.27:0/2506296121 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa80c81c0 con 0x7f8fa806c740 2024-01-15T18:15:19.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.233+0000 7f8f9ea00640 1 -- 172.21.15.27:0/2506296121 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f980043a0 con 0x7f8fa806c740 2024-01-15T18:15:19.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.233+0000 7f8fad400640 1 -- 172.21.15.27:0/2506296121 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa80c8780 con 0x7f8fa806c740 2024-01-15T18:15:19.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.233+0000 7f8fad400640 1 -- 172.21.15.27:0/2506296121 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa804e850 con 0x7f8fa806c740 2024-01-15T18:15:19.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.233+0000 7f8f9ea00640 1 -- 172.21.15.27:0/2506296121 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7f8f980b2040 con 0x7f8fa806c740 2024-01-15T18:15:19.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.233+0000 7f8f9ea00640 1 --2- 172.21.15.27:0/2506296121 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8f80067430 0x7f8f800698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:19.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.233+0000 7f8f9ea00640 1 -- 172.21.15.27:0/2506296121 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f8f9811c070 con 0x7f8fa806c740 2024-01-15T18:15:19.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.233+0000 7f8fa5e00640 1 --2- 172.21.15.27:0/2506296121 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8f80067430 0x7f8f800698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:19.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.237+0000 7f8fa5e00640 1 --2- 172.21.15.27:0/2506296121 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8f80067430 0x7f8f800698f0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8fa815a340 tx=0x7f8f90007b60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:19.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.237+0000 7f8f9ea00640 1 -- 172.21.15.27:0/2506296121 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8f980eab40 con 0x7f8fa806c740 2024-01-15T18:15:19.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.417+0000 7f8fad400640 1 -- 172.21.15.27:0/2506296121 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone519315", "target": ["mon-mgr", ""]}) v1 -- 0x7f8fa806cbb0 con 0x7f8f80067430 2024-01-15T18:15:19.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.441+0000 7f8f9ea00640 1 -- 172.21.15.27:0/2506296121 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8fa806cbb0 con 0x7f8f80067430 2024-01-15T18:15:19.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.441+0000 7f8fad400640 1 -- 172.21.15.27:0/2506296121 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8f80067430 msgr2=0x7f8f800698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:19.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.441+0000 7f8fad400640 1 --2- 172.21.15.27:0/2506296121 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8f80067430 0x7f8f800698f0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8fa815a340 tx=0x7f8f90007b60 comp rx=0 tx=0).stop 2024-01-15T18:15:19.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.441+0000 7f8fad400640 1 -- 172.21.15.27:0/2506296121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fa806c740 msgr2=0x7f8fa80c49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:19.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.441+0000 7f8fad400640 1 --2- 172.21.15.27:0/2506296121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fa806c740 0x7f8fa80c49b0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f8f980097c0 tx=0x7f8f980191b0 comp rx=0 tx=0).stop 2024-01-15T18:15:19.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.445+0000 7f8fad400640 1 -- 172.21.15.27:0/2506296121 shutdown_connections 2024-01-15T18:15:19.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.445+0000 7f8fad400640 1 --2- 172.21.15.27:0/2506296121 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8f80067430 0x7f8f800698f0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:19.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.445+0000 7f8fad400640 1 --2- 172.21.15.27:0/2506296121 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8fa815d480 0x7f8fa81546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:19.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.445+0000 7f8fad400640 1 --2- 172.21.15.27:0/2506296121 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8fa809a120 0x7f8fa8152120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:19.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.445+0000 7f8fad400640 1 --2- 172.21.15.27:0/2506296121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fa806c740 0x7f8fa80c49b0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:19.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.445+0000 7f8fad400640 1 -- 172.21.15.27:0/2506296121 >> 172.21.15.27:0/2506296121 conn(0x7f8fa80b2a80 msgr2=0x7f8fa806faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:19.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.445+0000 7f8fad400640 1 -- 172.21.15.27:0/2506296121 shutdown_connections 2024-01-15T18:15:19.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.445+0000 7f8fad400640 1 -- 172.21.15.27:0/2506296121 wait complete. 2024-01-15T18:15:19.471 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone519316 2024-01-15T18:15:19.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.613+0000 7f223b600640 1 Processor -- start 2024-01-15T18:15:19.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.613+0000 7f223b600640 1 -- start start 2024-01-15T18:15:19.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.617+0000 7f223b600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f223415d190 0x7f22340cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:19.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.617+0000 7f223b600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f223406c520 0x7f22340cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:19.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.617+0000 7f223b600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22340b0d70 con 0x7f223415d570 2024-01-15T18:15:19.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.617+0000 7f223b600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22340d0540 con 0x7f223415d190 2024-01-15T18:15:19.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.617+0000 7f223b600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22340b0440 con 0x7f223406c520 2024-01-15T18:15:19.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.617+0000 7f2238c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f223415d190 0x7f22340cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:19.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.617+0000 7f2233e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f223415d570 0x7f223406be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60832/0 (socket says 172.21.15.27:60832) 2024-01-15T18:15:19.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.617+0000 7f2233e00640 1 -- 172.21.15.27:0/1667696529 learned_addr learned my addr 172.21.15.27:0/1667696529 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:19.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.617+0000 7f2239600640 1 --2- 172.21.15.27:0/1667696529 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f223406c520 0x7f22340cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:19.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.617+0000 7f2238c00640 1 -- 172.21.15.27:0/1667696529 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f223406c520 msgr2=0x7f22340cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:19.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.617+0000 7f2238c00640 1 --2- 172.21.15.27:0/1667696529 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f223406c520 0x7f22340cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:19.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.617+0000 7f2238c00640 1 -- 172.21.15.27:0/1667696529 >> v1:172.21.15.27:6789/0 conn(0x7f223415d570 legacy=0x7f223406be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:19.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.617+0000 7f2238c00640 1 -- 172.21.15.27:0/1667696529 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22340b0720 con 0x7f223415d190 2024-01-15T18:15:19.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.621+0000 7f2238c00640 1 --2- 172.21.15.27:0/1667696529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f223415d190 0x7f22340cfcf0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f22240097c0 tx=0x7f222401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a08a8ce14e268de7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:19.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.621+0000 7f2233400640 1 -- 172.21.15.27:0/1667696529 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22240ab610 con 0x7f223415d190 2024-01-15T18:15:19.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.621+0000 7f2233400640 1 -- 172.21.15.27:0/1667696529 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22240ab770 con 0x7f223415d190 2024-01-15T18:15:19.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.621+0000 7f2233400640 1 -- 172.21.15.27:0/1667696529 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22240abb00 con 0x7f223415d190 2024-01-15T18:15:19.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.621+0000 7f223b600640 1 -- 172.21.15.27:0/1667696529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f223415d190 msgr2=0x7f22340cfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:19.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.621+0000 7f223b600640 1 --2- 172.21.15.27:0/1667696529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f223415d190 0x7f22340cfcf0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f22240097c0 tx=0x7f222401c450 comp rx=0 tx=0).stop 2024-01-15T18:15:19.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.621+0000 7f223b600640 1 -- 172.21.15.27:0/1667696529 shutdown_connections 2024-01-15T18:15:19.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.621+0000 7f223b600640 1 --2- 172.21.15.27:0/1667696529 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f223406c520 0x7f22340cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:19.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.621+0000 7f223b600640 1 --2- 172.21.15.27:0/1667696529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f223415d190 0x7f22340cfcf0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:19.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.621+0000 7f223b600640 1 -- 172.21.15.27:0/1667696529 >> 172.21.15.27:0/1667696529 conn(0x7f22340d0e70 msgr2=0x7f22340d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:19.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.621+0000 7f223b600640 1 -- 172.21.15.27:0/1667696529 shutdown_connections 2024-01-15T18:15:19.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.621+0000 7f223b600640 1 -- 172.21.15.27:0/1667696529 wait complete. 2024-01-15T18:15:19.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.621+0000 7f223b600640 1 Processor -- start 2024-01-15T18:15:19.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.621+0000 7f223b600640 1 -- start start 2024-01-15T18:15:19.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.625+0000 7f223b600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f223406c520 0x7f22340b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:19.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.625+0000 7f223b600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f223415d190 0x7f22341493b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:19.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.625+0000 7f223b600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f223415d570 0x7f2234152eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:19.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.625+0000 7f223b600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22341481f0 con 0x7f223415d190 2024-01-15T18:15:19.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.625+0000 7f223b600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22340b5ee0 con 0x7f223406c520 2024-01-15T18:15:19.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.625+0000 7f223b600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22340c7a50 con 0x7f223415d570 2024-01-15T18:15:19.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.625+0000 7f2233e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f223415d190 0x7f22341493b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:19.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.625+0000 7f2233e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f223415d190 0x7f22341493b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51410/0 (socket says 172.21.15.27:51410) 2024-01-15T18:15:19.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.625+0000 7f2233e00640 1 -- 172.21.15.27:0/1141729175 learned_addr learned my addr 172.21.15.27:0/1141729175 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:19.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.625+0000 7f2238c00640 1 --2- 172.21.15.27:0/1141729175 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f223406c520 0x7f22340b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:19.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.625+0000 7f2239600640 1 --2- 172.21.15.27:0/1141729175 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f223415d570 0x7f2234152eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:19.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.625+0000 7f2233e00640 1 -- 172.21.15.27:0/1141729175 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f223415d570 msgr2=0x7f2234152eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:19.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.625+0000 7f2233e00640 1 --2- 172.21.15.27:0/1141729175 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f223415d570 0x7f2234152eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:19.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.625+0000 7f2233e00640 1 -- 172.21.15.27:0/1141729175 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f223406c520 msgr2=0x7f22340b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:19.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.625+0000 7f2233e00640 1 --2- 172.21.15.27:0/1141729175 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f223406c520 0x7f22340b46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:19.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.625+0000 7f2233e00640 1 -- 172.21.15.27:0/1141729175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2224009420 con 0x7f223415d190 2024-01-15T18:15:19.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.625+0000 7f2233e00640 1 --2- 172.21.15.27:0/1141729175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f223415d190 0x7f22341493b0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f221c00c430 tx=0x7f221c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:19.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.625+0000 7f2230c00640 1 -- 172.21.15.27:0/1141729175 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f221c005cf0 con 0x7f223415d190 2024-01-15T18:15:19.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.625+0000 7f2230c00640 1 -- 172.21.15.27:0/1141729175 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f221c005e50 con 0x7f223415d190 2024-01-15T18:15:19.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.625+0000 7f2230c00640 1 -- 172.21.15.27:0/1141729175 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f221c0072d0 con 0x7f223415d190 2024-01-15T18:15:19.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.625+0000 7f223b600640 1 -- 172.21.15.27:0/1141729175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2234147f10 con 0x7f223415d190 2024-01-15T18:15:19.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.625+0000 7f223b600640 1 -- 172.21.15.27:0/1141729175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22340c7770 con 0x7f223415d190 2024-01-15T18:15:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.625+0000 7f223b600640 1 -- 172.21.15.27:0/1141729175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22340c72f0 con 0x7f223415d190 2024-01-15T18:15:19.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.625+0000 7f2230c00640 1 -- 172.21.15.27:0/1141729175 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7f221c007430 con 0x7f223415d190 2024-01-15T18:15:19.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.629+0000 7f2230c00640 1 --2- 172.21.15.27:0/1141729175 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f220c067460 0x7f220c069920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:19.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.629+0000 7f2230c00640 1 -- 172.21.15.27:0/1141729175 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f221c0824b0 con 0x7f223415d190 2024-01-15T18:15:19.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.629+0000 7f2238c00640 1 --2- 172.21.15.27:0/1141729175 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f220c067460 0x7f220c069920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:19.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.629+0000 7f2238c00640 1 --2- 172.21.15.27:0/1141729175 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f220c067460 0x7f220c069920 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f22240af000 tx=0x7f22240a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:19.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.633+0000 7f2230c00640 1 -- 172.21.15.27:0/1141729175 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f22340c72f0 con 0x7f223415d190 2024-01-15T18:15:19.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.817+0000 7f223b600640 1 -- 172.21.15.27:0/1141729175 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone519316", "target": ["mon-mgr", ""]}) v1 -- 0x7f22340c72f0 con 0x7f220c067460 2024-01-15T18:15:19.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.837+0000 7f2230c00640 1 -- 172.21.15.27:0/1141729175 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f22340c72f0 con 0x7f220c067460 2024-01-15T18:15:19.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.841+0000 7f223b600640 1 -- 172.21.15.27:0/1141729175 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f220c067460 msgr2=0x7f220c069920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:19.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.841+0000 7f223b600640 1 --2- 172.21.15.27:0/1141729175 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f220c067460 0x7f220c069920 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f22240af000 tx=0x7f22240a7000 comp rx=0 tx=0).stop 2024-01-15T18:15:19.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.841+0000 7f223b600640 1 -- 172.21.15.27:0/1141729175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f223415d190 msgr2=0x7f22341493b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:19.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.841+0000 7f223b600640 1 --2- 172.21.15.27:0/1141729175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f223415d190 0x7f22341493b0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f221c00c430 tx=0x7f221c00c8f0 comp rx=0 tx=0).stop 2024-01-15T18:15:19.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.841+0000 7f223b600640 1 -- 172.21.15.27:0/1141729175 shutdown_connections 2024-01-15T18:15:19.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.841+0000 7f223b600640 1 --2- 172.21.15.27:0/1141729175 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f220c067460 0x7f220c069920 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:19.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.841+0000 7f223b600640 1 --2- 172.21.15.27:0/1141729175 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f223415d570 0x7f2234152eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:19.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.841+0000 7f223b600640 1 --2- 172.21.15.27:0/1141729175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f223415d190 0x7f22341493b0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:19.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.841+0000 7f223b600640 1 --2- 172.21.15.27:0/1141729175 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f223406c520 0x7f22340b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:19.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.841+0000 7f223b600640 1 -- 172.21.15.27:0/1141729175 >> 172.21.15.27:0/1141729175 conn(0x7f22340d0e70 msgr2=0x7f22340ae0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:19.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.841+0000 7f223b600640 1 -- 172.21.15.27:0/1141729175 shutdown_connections 2024-01-15T18:15:19.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:19.841+0000 7f223b600640 1 -- 172.21.15.27:0/1141729175 wait complete. 2024-01-15T18:15:19.863 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone519319 2024-01-15T18:15:20.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.009+0000 7fd487400640 1 Processor -- start 2024-01-15T18:15:20.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.009+0000 7fd487400640 1 -- start start 2024-01-15T18:15:20.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.009+0000 7fd487400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fd4800cf2e0 0x7fd4800701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:20.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.009+0000 7fd487400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4800b1cb0 con 0x7fd480158900 2024-01-15T18:15:20.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.009+0000 7fd487400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48015fa00 con 0x7fd48015f550 2024-01-15T18:15:20.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.009+0000 7fd487400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4800b1f00 con 0x7fd4800cf2e0 2024-01-15T18:15:20.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.009+0000 7fd484a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd480158900 0x7fd48015bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60844/0 (socket says 172.21.15.27:60844) 2024-01-15T18:15:20.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.009+0000 7fd484a00640 1 -- 172.21.15.27:0/2094018116 learned_addr learned my addr 172.21.15.27:0/2094018116 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:20.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.009+0000 7fd485400640 1 --2- 172.21.15.27:0/2094018116 >> v2:172.21.15.93:3300/0 conn(0x7fd4800cf2e0 0x7fd4800701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:20.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.009+0000 7fd47f400640 1 -- 172.21.15.27:0/2094018116 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2496648461 0 0) 0x7fd4800b1cb0 con 0x7fd480158900 2024-01-15T18:15:20.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.009+0000 7fd47f400640 1 -- 172.21.15.27:0/2094018116 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd460003530 con 0x7fd480158900 2024-01-15T18:15:20.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.009+0000 7fd485400640 1 -- 172.21.15.27:0/2094018116 >> v1:172.21.15.93:6789/0 conn(0x7fd48015f550 legacy=0x7fd4800cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:20.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.009+0000 7fd485400640 1 -- 172.21.15.27:0/2094018116 >> v1:172.21.15.27:6789/0 conn(0x7fd480158900 legacy=0x7fd48015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:20.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.009+0000 7fd485400640 1 -- 172.21.15.27:0/2094018116 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4800b21e0 con 0x7fd4800cf2e0 2024-01-15T18:15:20.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.013+0000 7fd485400640 1 --2- 172.21.15.27:0/2094018116 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4800cf2e0 0x7fd4800701e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fd478009d90 tx=0x7fd47801c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7656bd7898092ff6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:20.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.013+0000 7fd47f400640 1 -- 172.21.15.27:0/2094018116 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd480070f30 con 0x7fd4800cf2e0 2024-01-15T18:15:20.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.013+0000 7fd47f400640 1 -- 172.21.15.27:0/2094018116 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4780a59e0 con 0x7fd4800cf2e0 2024-01-15T18:15:20.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.013+0000 7fd47f400640 1 -- 172.21.15.27:0/2094018116 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4780a5d70 con 0x7fd4800cf2e0 2024-01-15T18:15:20.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.013+0000 7fd487400640 1 -- 172.21.15.27:0/2094018116 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4800cf2e0 msgr2=0x7fd4800701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:20.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.013+0000 7fd487400640 1 --2- 172.21.15.27:0/2094018116 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4800cf2e0 0x7fd4800701e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fd478009d90 tx=0x7fd47801c920 comp rx=0 tx=0).stop 2024-01-15T18:15:20.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.013+0000 7fd487400640 1 -- 172.21.15.27:0/2094018116 shutdown_connections 2024-01-15T18:15:20.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.013+0000 7fd487400640 1 --2- 172.21.15.27:0/2094018116 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4800cf2e0 0x7fd4800701e0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:20.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.013+0000 7fd487400640 1 -- 172.21.15.27:0/2094018116 >> 172.21.15.27:0/2094018116 conn(0x7fd4800cb700 msgr2=0x7fd4800cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:20.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.013+0000 7fd487400640 1 -- 172.21.15.27:0/2094018116 shutdown_connections 2024-01-15T18:15:20.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.013+0000 7fd487400640 1 -- 172.21.15.27:0/2094018116 wait complete. 2024-01-15T18:15:20.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.017+0000 7fd487400640 1 Processor -- start 2024-01-15T18:15:20.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.017+0000 7fd487400640 1 -- start start 2024-01-15T18:15:20.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.017+0000 7fd487400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4800cf2e0 0x7fd4801f5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:20.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.017+0000 7fd487400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd480158900 0x7fd48015dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:20.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.017+0000 7fd487400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd48015f550 0x7fd480150490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:20.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.017+0000 7fd487400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4800cc9b0 con 0x7fd480158900 2024-01-15T18:15:20.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.017+0000 7fd487400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48015c7a0 con 0x7fd48015f550 2024-01-15T18:15:20.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.017+0000 7fd487400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd48015abb0 con 0x7fd4800cf2e0 2024-01-15T18:15:20.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.017+0000 7fd47fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd480158900 0x7fd48015dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:20.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.017+0000 7fd484a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4800cf2e0 0x7fd4801f5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:20.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.017+0000 7fd47fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd480158900 0x7fd48015dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51420/0 (socket says 172.21.15.27:51420) 2024-01-15T18:15:20.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.017+0000 7fd47fe00640 1 -- 172.21.15.27:0/1955837748 learned_addr learned my addr 172.21.15.27:0/1955837748 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:20.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.017+0000 7fd485400640 1 --2- 172.21.15.27:0/1955837748 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd48015f550 0x7fd480150490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:20.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.017+0000 7fd47fe00640 1 -- 172.21.15.27:0/1955837748 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4800cf2e0 msgr2=0x7fd4801f5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:20.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.017+0000 7fd47fe00640 1 --2- 172.21.15.27:0/1955837748 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4800cf2e0 0x7fd4801f5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:20.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.017+0000 7fd47fe00640 1 -- 172.21.15.27:0/1955837748 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd48015f550 msgr2=0x7fd480150490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:20.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.017+0000 7fd47fe00640 1 --2- 172.21.15.27:0/1955837748 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd48015f550 0x7fd480150490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:20.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.017+0000 7fd47fe00640 1 -- 172.21.15.27:0/1955837748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd478009a70 con 0x7fd480158900 2024-01-15T18:15:20.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.017+0000 7fd47fe00640 1 --2- 172.21.15.27:0/1955837748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd480158900 0x7fd48015dff0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fd46800c3c0 tx=0x7fd46800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:20.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.017+0000 7fd47cc00640 1 -- 172.21.15.27:0/1955837748 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4680052d0 con 0x7fd480158900 2024-01-15T18:15:20.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.017+0000 7fd47cc00640 1 -- 172.21.15.27:0/1955837748 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd468005430 con 0x7fd480158900 2024-01-15T18:15:20.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.017+0000 7fd47cc00640 1 -- 172.21.15.27:0/1955837748 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd468003840 con 0x7fd480158900 2024-01-15T18:15:20.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.017+0000 7fd487400640 1 -- 172.21.15.27:0/1955837748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd48015a8d0 con 0x7fd480158900 2024-01-15T18:15:20.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.017+0000 7fd487400640 1 -- 172.21.15.27:0/1955837748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd48015ca40 con 0x7fd480158900 2024-01-15T18:15:20.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.021+0000 7fd487400640 1 -- 172.21.15.27:0/1955837748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4800ccaf0 con 0x7fd480158900 2024-01-15T18:15:20.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.021+0000 7fd47cc00640 1 -- 172.21.15.27:0/1955837748 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7fd4680039a0 con 0x7fd480158900 2024-01-15T18:15:20.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.021+0000 7fd47cc00640 1 --2- 172.21.15.27:0/1955837748 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd458067430 0x7fd4580698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:20.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.021+0000 7fd47cc00640 1 -- 172.21.15.27:0/1955837748 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fd4680824d0 con 0x7fd480158900 2024-01-15T18:15:20.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.021+0000 7fd484a00640 1 --2- 172.21.15.27:0/1955837748 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd458067430 0x7fd4580698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:20.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.025+0000 7fd484a00640 1 --2- 172.21.15.27:0/1955837748 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd458067430 0x7fd4580698f0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fd4700011f0 tx=0x7fd470007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:20.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.025+0000 7fd47cc00640 1 -- 172.21.15.27:0/1955837748 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd4800ccaf0 con 0x7fd480158900 2024-01-15T18:15:20.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.209+0000 7fd487400640 1 -- 172.21.15.27:0/1955837748 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone519319", "target": ["mon-mgr", ""]}) v1 -- 0x7fd48006e170 con 0x7fd458067430 2024-01-15T18:15:20.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.225+0000 7fd47cc00640 1 -- 172.21.15.27:0/1955837748 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd470005860 con 0x7fd458067430 2024-01-15T18:15:20.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.225+0000 7fd487400640 1 -- 172.21.15.27:0/1955837748 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd458067430 msgr2=0x7fd4580698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:20.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.225+0000 7fd487400640 1 --2- 172.21.15.27:0/1955837748 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd458067430 0x7fd4580698f0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fd4700011f0 tx=0x7fd470007d40 comp rx=0 tx=0).stop 2024-01-15T18:15:20.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.225+0000 7fd487400640 1 -- 172.21.15.27:0/1955837748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd480158900 msgr2=0x7fd48015dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:20.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.225+0000 7fd487400640 1 --2- 172.21.15.27:0/1955837748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd480158900 0x7fd48015dff0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fd46800c3c0 tx=0x7fd46800c880 comp rx=0 tx=0).stop 2024-01-15T18:15:20.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.225+0000 7fd487400640 1 -- 172.21.15.27:0/1955837748 shutdown_connections 2024-01-15T18:15:20.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.225+0000 7fd487400640 1 --2- 172.21.15.27:0/1955837748 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd458067430 0x7fd4580698f0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:20.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.225+0000 7fd487400640 1 --2- 172.21.15.27:0/1955837748 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd48015f550 0x7fd480150490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:20.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.225+0000 7fd487400640 1 --2- 172.21.15.27:0/1955837748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd480158900 0x7fd48015dff0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:20.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.225+0000 7fd487400640 1 --2- 172.21.15.27:0/1955837748 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4800cf2e0 0x7fd4801f5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:20.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.225+0000 7fd487400640 1 -- 172.21.15.27:0/1955837748 >> 172.21.15.27:0/1955837748 conn(0x7fd4800cb700 msgr2=0x7fd48006c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:20.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.229+0000 7fd487400640 1 -- 172.21.15.27:0/1955837748 shutdown_connections 2024-01-15T18:15:20.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.229+0000 7fd487400640 1 -- 172.21.15.27:0/1955837748 wait complete. 2024-01-15T18:15:20.249 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone51932 2024-01-15T18:15:20.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.397+0000 7fbadc800640 1 Processor -- start 2024-01-15T18:15:20.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.397+0000 7fbadc800640 1 -- start start 2024-01-15T18:15:20.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.397+0000 7fbadc800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fbad40cfc30 0x7fbad40cc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:20.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.397+0000 7fbadc800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad40cb4d0 con 0x7fbad40cde10 2024-01-15T18:15:20.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.397+0000 7fbadc800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad40d0120 con 0x7fbad415d680 2024-01-15T18:15:20.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.397+0000 7fbadc800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbad40d0400 con 0x7fbad40cfc30 2024-01-15T18:15:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.397+0000 7fbad9400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbad40cde10 0x7fbad40cbf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60860/0 (socket says 172.21.15.27:60860) 2024-01-15T18:15:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.397+0000 7fbad9400640 1 -- 172.21.15.27:0/2585433908 learned_addr learned my addr 172.21.15.27:0/2585433908 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.397+0000 7fbada800640 1 --2- 172.21.15.27:0/2585433908 >> v2:172.21.15.93:3300/0 conn(0x7fbad40cfc30 0x7fbad40cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.397+0000 7fbad8a00640 1 -- 172.21.15.27:0/2585433908 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3965235752 0 0) 0x7fbad40cb4d0 con 0x7fbad40cde10 2024-01-15T18:15:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.397+0000 7fbad8a00640 1 -- 172.21.15.27:0/2585433908 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbab4003530 con 0x7fbad40cde10 2024-01-15T18:15:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.397+0000 7fbad8a00640 1 -- 172.21.15.27:0/2585433908 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630054015 0 0) 0x7fbad40d0120 con 0x7fbad415d680 2024-01-15T18:15:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.397+0000 7fbad8a00640 1 -- 172.21.15.27:0/2585433908 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbad40cb4d0 con 0x7fbad415d680 2024-01-15T18:15:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.397+0000 7fbada800640 1 -- 172.21.15.27:0/2585433908 >> v1:172.21.15.93:6789/0 conn(0x7fbad415d680 legacy=0x7fbad40cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.397+0000 7fbada800640 1 -- 172.21.15.27:0/2585433908 >> v1:172.21.15.27:6789/0 conn(0x7fbad40cde10 legacy=0x7fbad40cbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.397+0000 7fbad8a00640 1 -- 172.21.15.27:0/2585433908 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3433070575 0 0) 0x7fbab4003530 con 0x7fbad40cde10 2024-01-15T18:15:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.397+0000 7fbada800640 1 -- 172.21.15.27:0/2585433908 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad40cb770 con 0x7fbad40cfc30 2024-01-15T18:15:20.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.401+0000 7fbada800640 1 --2- 172.21.15.27:0/2585433908 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbad40cfc30 0x7fbad40cc680 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fbacc009d90 tx=0x7fbad40cd3d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=deea15ebb8e58f5d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:20.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.401+0000 7fbad8a00640 1 -- 172.21.15.27:0/2585433908 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad40ce850 con 0x7fbad40cfc30 2024-01-15T18:15:20.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.401+0000 7fbad8a00640 1 -- 172.21.15.27:0/2585433908 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbacc0a56a0 con 0x7fbad40cfc30 2024-01-15T18:15:20.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.401+0000 7fbad8a00640 1 -- 172.21.15.27:0/2585433908 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc0016a0 con 0x7fbad40cfc30 2024-01-15T18:15:20.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.401+0000 7fbadc800640 1 -- 172.21.15.27:0/2585433908 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbad40cfc30 msgr2=0x7fbad40cc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:20.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.401+0000 7fbadc800640 1 --2- 172.21.15.27:0/2585433908 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbad40cfc30 0x7fbad40cc680 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fbacc009d90 tx=0x7fbad40cd3d0 comp rx=0 tx=0).stop 2024-01-15T18:15:20.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.401+0000 7fbadc800640 1 -- 172.21.15.27:0/2585433908 shutdown_connections 2024-01-15T18:15:20.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.401+0000 7fbadc800640 1 --2- 172.21.15.27:0/2585433908 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbad40cfc30 0x7fbad40cc680 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:20.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.401+0000 7fbadc800640 1 -- 172.21.15.27:0/2585433908 >> 172.21.15.27:0/2585433908 conn(0x7fbad40d0e80 msgr2=0x7fbad40d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:20.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.401+0000 7fbadc800640 1 -- 172.21.15.27:0/2585433908 shutdown_connections 2024-01-15T18:15:20.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.401+0000 7fbadc800640 1 -- 172.21.15.27:0/2585433908 wait complete. 2024-01-15T18:15:20.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.405+0000 7fbadc800640 1 Processor -- start 2024-01-15T18:15:20.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.405+0000 7fbadc800640 1 -- start start 2024-01-15T18:15:20.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.405+0000 7fbadc800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbad40cde10 0x7fbad415dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:20.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.405+0000 7fbadc800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbad40cfc30 0x7fbad4153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:20.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.405+0000 7fbadc800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad415d680 0x7fbad415fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:20.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.405+0000 7fbadc800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad41591f0 con 0x7fbad415d680 2024-01-15T18:15:20.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.405+0000 7fbadc800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad415e160 con 0x7fbad40cfc30 2024-01-15T18:15:20.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.405+0000 7fbadc800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad415e440 con 0x7fbad40cde10 2024-01-15T18:15:20.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.405+0000 7fbada800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad415d680 0x7fbad415fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.405+0000 7fbada800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad415d680 0x7fbad415fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51426/0 (socket says 172.21.15.27:51426) 2024-01-15T18:15:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.405+0000 7fbada800640 1 -- 172.21.15.27:0/3295682948 learned_addr learned my addr 172.21.15.27:0/3295682948 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.405+0000 7fbad9400640 1 --2- 172.21.15.27:0/3295682948 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbad40cfc30 0x7fbad4153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.405+0000 7fbad9e00640 1 --2- 172.21.15.27:0/3295682948 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbad40cde10 0x7fbad415dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.405+0000 7fbada800640 1 -- 172.21.15.27:0/3295682948 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbad40cde10 msgr2=0x7fbad415dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.405+0000 7fbada800640 1 --2- 172.21.15.27:0/3295682948 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbad40cde10 0x7fbad415dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.405+0000 7fbada800640 1 -- 172.21.15.27:0/3295682948 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbad40cfc30 msgr2=0x7fbad4153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.405+0000 7fbada800640 1 --2- 172.21.15.27:0/3295682948 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbad40cfc30 0x7fbad4153ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.405+0000 7fbada800640 1 -- 172.21.15.27:0/3295682948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbacc009a70 con 0x7fbad415d680 2024-01-15T18:15:20.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.405+0000 7fbada800640 1 --2- 172.21.15.27:0/3295682948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad415d680 0x7fbad415fb30 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fbacc01a470 tx=0x7fbacc0195e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:20.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.405+0000 7fbad2000640 1 -- 172.21.15.27:0/3295682948 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbacc0b9860 con 0x7fbad415d680 2024-01-15T18:15:20.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.405+0000 7fbad2000640 1 -- 172.21.15.27:0/3295682948 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbacc0b99c0 con 0x7fbad415d680 2024-01-15T18:15:20.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.405+0000 7fbad2000640 1 -- 172.21.15.27:0/3295682948 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbacc0b9d70 con 0x7fbad415d680 2024-01-15T18:15:20.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.405+0000 7fbadc800640 1 -- 172.21.15.27:0/3295682948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad415e720 con 0x7fbad415d680 2024-01-15T18:15:20.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.405+0000 7fbadc800640 1 -- 172.21.15.27:0/3295682948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad415ece0 con 0x7fbad415d680 2024-01-15T18:15:20.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.409+0000 7fbadc800640 1 -- 172.21.15.27:0/3295682948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbad40afe50 con 0x7fbad415d680 2024-01-15T18:15:20.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.409+0000 7fbad2000640 1 -- 172.21.15.27:0/3295682948 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7fbacc0c4460 con 0x7fbad415d680 2024-01-15T18:15:20.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.409+0000 7fbad2000640 1 --2- 172.21.15.27:0/3295682948 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbaac067430 0x7fbaac0698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:20.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.409+0000 7fbad2000640 1 -- 172.21.15.27:0/3295682948 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fbacc11e070 con 0x7fbad415d680 2024-01-15T18:15:20.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.409+0000 7fbad9e00640 1 --2- 172.21.15.27:0/3295682948 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbaac067430 0x7fbaac0698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:20.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.409+0000 7fbad9e00640 1 --2- 172.21.15.27:0/3295682948 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbaac067430 0x7fbaac0698f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fbac40011f0 tx=0x7fbac4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:20.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.413+0000 7fbad2000640 1 -- 172.21.15.27:0/3295682948 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbacc0ece20 con 0x7fbad415d680 2024-01-15T18:15:20.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.597+0000 7fbadc800640 1 -- 172.21.15.27:0/3295682948 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone51932", "target": ["mon-mgr", ""]}) v1 -- 0x7fbad40ceae0 con 0x7fbaac067430 2024-01-15T18:15:20.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.613+0000 7fbad2000640 1 -- 172.21.15.27:0/3295682948 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbad40ceae0 con 0x7fbaac067430 2024-01-15T18:15:20.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.613+0000 7fbadc800640 1 -- 172.21.15.27:0/3295682948 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbaac067430 msgr2=0x7fbaac0698f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:20.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.613+0000 7fbadc800640 1 --2- 172.21.15.27:0/3295682948 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbaac067430 0x7fbaac0698f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fbac40011f0 tx=0x7fbac4009040 comp rx=0 tx=0).stop 2024-01-15T18:15:20.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.613+0000 7fbadc800640 1 -- 172.21.15.27:0/3295682948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad415d680 msgr2=0x7fbad415fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:20.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.613+0000 7fbadc800640 1 --2- 172.21.15.27:0/3295682948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad415d680 0x7fbad415fb30 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fbacc01a470 tx=0x7fbacc0195e0 comp rx=0 tx=0).stop 2024-01-15T18:15:20.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.613+0000 7fbadc800640 1 -- 172.21.15.27:0/3295682948 shutdown_connections 2024-01-15T18:15:20.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.613+0000 7fbadc800640 1 --2- 172.21.15.27:0/3295682948 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbaac067430 0x7fbaac0698f0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:20.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.613+0000 7fbadc800640 1 --2- 172.21.15.27:0/3295682948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad415d680 0x7fbad415fb30 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:20.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.613+0000 7fbadc800640 1 --2- 172.21.15.27:0/3295682948 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbad40cfc30 0x7fbad4153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:20.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.613+0000 7fbadc800640 1 --2- 172.21.15.27:0/3295682948 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbad40cde10 0x7fbad415dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:20.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.613+0000 7fbadc800640 1 -- 172.21.15.27:0/3295682948 >> 172.21.15.27:0/3295682948 conn(0x7fbad40d0e80 msgr2=0x7fbad40d1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:20.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.613+0000 7fbadc800640 1 -- 172.21.15.27:0/3295682948 shutdown_connections 2024-01-15T18:15:20.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.613+0000 7fbadc800640 1 -- 172.21.15.27:0/3295682948 wait complete. 2024-01-15T18:15:20.637 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone519310 2024-01-15T18:15:20.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.781+0000 7f8683400640 1 Processor -- start 2024-01-15T18:15:20.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.781+0000 7f8683400640 1 -- start start 2024-01-15T18:15:20.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f8683400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f867c15f550 0x7f867c0cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f8683400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f867c0cf110 0x7f867c0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f8683400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f867c0b1cb0 con 0x7f867c158900 2024-01-15T18:15:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f8683400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f867c15fa00 con 0x7f867c0cf110 2024-01-15T18:15:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f8683400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f867c0cf540 con 0x7f867c15f550 2024-01-15T18:15:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f8680a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f867c158900 0x7f867c15bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60866/0 (socket says 172.21.15.27:60866) 2024-01-15T18:15:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f8680a00640 1 -- 172.21.15.27:0/3346202679 learned_addr learned my addr 172.21.15.27:0/3346202679 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f867be00640 1 --2- 172.21.15.27:0/3346202679 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f867c15f550 0x7f867c0cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f8681400640 1 --2- 172.21.15.27:0/3346202679 >> v2:172.21.15.93:3300/0 conn(0x7f867c0cf110 0x7f867c0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f867b400640 1 -- 172.21.15.27:0/3346202679 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2150457587 0 0) 0x7f867c0b1cb0 con 0x7f867c158900 2024-01-15T18:15:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f867b400640 1 -- 172.21.15.27:0/3346202679 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f865c003530 con 0x7f867c158900 2024-01-15T18:15:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f867b400640 1 -- 172.21.15.27:0/3346202679 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1353278121 0 0) 0x7f865c003530 con 0x7f867c158900 2024-01-15T18:15:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f867b400640 1 -- 172.21.15.27:0/3346202679 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f867c15f550 msgr2=0x7f867c0cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:20.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f867b400640 1 --2- 172.21.15.27:0/3346202679 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f867c15f550 0x7f867c0cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f867b400640 1 -- 172.21.15.27:0/3346202679 >> v2:172.21.15.93:3300/0 conn(0x7f867c0cf110 msgr2=0x7f867c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f867b400640 1 --2- 172.21.15.27:0/3346202679 >> v2:172.21.15.93:3300/0 conn(0x7f867c0cf110 0x7f867c0701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f867b400640 1 -- 172.21.15.27:0/3346202679 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867c0b1f00 con 0x7f867c158900 2024-01-15T18:15:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f867b400640 1 -- 172.21.15.27:0/3346202679 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f866c001680 con 0x7f867c158900 2024-01-15T18:15:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f867b400640 1 -- 172.21.15.27:0/3346202679 >> v1:172.21.15.27:6789/0 conn(0x7f867c158900 legacy=0x7f867c15bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f867b400640 1 --2- 172.21.15.27:0/3346202679 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f865c0039f0 0x7f865c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f867b400640 1 --2- 172.21.15.27:0/3346202679 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f867c15f550 0x7f865c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f867b400640 1 --2- 172.21.15.27:0/3346202679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f865c004d10 0x7f865c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f867b400640 1 -- 172.21.15.27:0/3346202679 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c15fa00 con 0x7f865c004d10 2024-01-15T18:15:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f867b400640 1 -- 172.21.15.27:0/3346202679 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c0cf540 con 0x7f865c0039f0 2024-01-15T18:15:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f867b400640 1 -- 172.21.15.27:0/3346202679 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f865c00b990 con 0x7f867c15f550 2024-01-15T18:15:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f8681400640 1 --2- 172.21.15.27:0/3346202679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f865c004d10 0x7f865c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f8680a00640 1 --2- 172.21.15.27:0/3346202679 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f865c0039f0 0x7f865c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:20.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f867be00640 1 --2- 172.21.15.27:0/3346202679 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f867c15f550 0x7f865c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f8681400640 1 -- 172.21.15.27:0/3346202679 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f867c15f550 msgr2=0x7f865c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f8681400640 1 --2- 172.21.15.27:0/3346202679 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f867c15f550 0x7f865c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f8681400640 1 -- 172.21.15.27:0/3346202679 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f865c0039f0 msgr2=0x7f865c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f8681400640 1 --2- 172.21.15.27:0/3346202679 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f865c0039f0 0x7f865c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:20.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.785+0000 7f8681400640 1 -- 172.21.15.27:0/3346202679 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865c003530 con 0x7f865c004d10 2024-01-15T18:15:20.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.789+0000 7f8681400640 1 --2- 172.21.15.27:0/3346202679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f865c004d10 0x7f865c005190 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f8674009fc0 tx=0x7f867401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd60e0f0511ee677 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:20.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.789+0000 7f867b400640 1 -- 172.21.15.27:0/3346202679 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86740a5cb0 con 0x7f865c004d10 2024-01-15T18:15:20.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.789+0000 7f867b400640 1 -- 172.21.15.27:0/3346202679 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867400f040 con 0x7f865c004d10 2024-01-15T18:15:20.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.789+0000 7f867b400640 1 -- 172.21.15.27:0/3346202679 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86740ad680 con 0x7f865c004d10 2024-01-15T18:15:20.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.789+0000 7f8683400640 1 -- 172.21.15.27:0/3346202679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f865c004d10 msgr2=0x7f865c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:20.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.789+0000 7f8683400640 1 --2- 172.21.15.27:0/3346202679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f865c004d10 0x7f865c005190 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f8674009fc0 tx=0x7f867401cca0 comp rx=0 tx=0).stop 2024-01-15T18:15:20.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.789+0000 7f8681400640 1 -- 172.21.15.27:0/3346202679 reap_dead start 2024-01-15T18:15:20.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.789+0000 7f8683400640 1 -- 172.21.15.27:0/3346202679 shutdown_connections 2024-01-15T18:15:20.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.789+0000 7f8683400640 1 -- 172.21.15.27:0/3346202679 >> 172.21.15.27:0/3346202679 conn(0x7f867c0cb700 msgr2=0x7f867c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:20.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.789+0000 7f8683400640 1 -- 172.21.15.27:0/3346202679 shutdown_connections 2024-01-15T18:15:20.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.789+0000 7f8683400640 1 -- 172.21.15.27:0/3346202679 wait complete. 2024-01-15T18:15:20.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.789+0000 7f8683400640 1 Processor -- start 2024-01-15T18:15:20.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.793+0000 7f8683400640 1 -- start start 2024-01-15T18:15:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.793+0000 7f8683400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f865c004d10 0x7f867c1f2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.793+0000 7f8683400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f867c0af9b0 0x7f867c0aca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.793+0000 7f8683400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f867c1f65e0 0x7f867c1f6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.793+0000 7f8683400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c0b0750 con 0x7f865c004d10 2024-01-15T18:15:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.793+0000 7f8683400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c0afeb0 con 0x7f867c1f65e0 2024-01-15T18:15:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.793+0000 7f8683400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f867c0b0190 con 0x7f867c0af9b0 2024-01-15T18:15:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.793+0000 7f8680a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f865c004d10 0x7f867c1f2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.793+0000 7f8680a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f865c004d10 0x7f867c1f2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51442/0 (socket says 172.21.15.27:51442) 2024-01-15T18:15:20.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.793+0000 7f8680a00640 1 -- 172.21.15.27:0/180431163 learned_addr learned my addr 172.21.15.27:0/180431163 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:20.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.793+0000 7f8681400640 1 --2- 172.21.15.27:0/180431163 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f867c1f65e0 0x7f867c1f6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:20.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.793+0000 7f867be00640 1 --2- 172.21.15.27:0/180431163 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f867c0af9b0 0x7f867c0aca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:20.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.793+0000 7f8680a00640 1 -- 172.21.15.27:0/180431163 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f867c0af9b0 msgr2=0x7f867c0aca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:20.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.793+0000 7f8680a00640 1 --2- 172.21.15.27:0/180431163 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f867c0af9b0 0x7f867c0aca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:20.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.793+0000 7f8680a00640 1 -- 172.21.15.27:0/180431163 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f867c1f65e0 msgr2=0x7f867c1f6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:20.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.793+0000 7f8680a00640 1 --2- 172.21.15.27:0/180431163 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f867c1f65e0 0x7f867c1f6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:20.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.793+0000 7f8680a00640 1 -- 172.21.15.27:0/180431163 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867c0b0470 con 0x7f865c004d10 2024-01-15T18:15:20.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.793+0000 7f867be00640 1 --2- 172.21.15.27:0/180431163 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f867c0af9b0 0x7f867c0aca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:20.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.793+0000 7f8680a00640 1 --2- 172.21.15.27:0/180431163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f865c004d10 0x7f867c1f2260 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f866c007340 tx=0x7f866c007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:20.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.793+0000 7f8678c00640 1 -- 172.21.15.27:0/180431163 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866c00f9c0 con 0x7f865c004d10 2024-01-15T18:15:20.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.793+0000 7f8678c00640 1 -- 172.21.15.27:0/180431163 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f866c00fb20 con 0x7f865c004d10 2024-01-15T18:15:20.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.793+0000 7f8683400640 1 -- 172.21.15.27:0/180431163 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f867c1f27a0 con 0x7f865c004d10 2024-01-15T18:15:20.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.793+0000 7f8678c00640 1 -- 172.21.15.27:0/180431163 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866c00fed0 con 0x7f865c004d10 2024-01-15T18:15:20.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.793+0000 7f8683400640 1 -- 172.21.15.27:0/180431163 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f867c0ad400 con 0x7f865c004d10 2024-01-15T18:15:20.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.793+0000 7f8683400640 1 -- 172.21.15.27:0/180431163 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f867c15b290 con 0x7f865c004d10 2024-01-15T18:15:20.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.797+0000 7f8678c00640 1 -- 172.21.15.27:0/180431163 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 28) v1 ==== 87180+0+0 (secure 0 0 0) 0x7f866c01d460 con 0x7f865c004d10 2024-01-15T18:15:20.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.797+0000 7f8678c00640 1 --2- 172.21.15.27:0/180431163 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8654067460 0x7f8654069920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:20.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.797+0000 7f8678c00640 1 -- 172.21.15.27:0/180431163 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f866c082c90 con 0x7f865c004d10 2024-01-15T18:15:20.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.797+0000 7f867be00640 1 --2- 172.21.15.27:0/180431163 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8654067460 0x7f8654069920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:20.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.797+0000 7f867be00640 1 --2- 172.21.15.27:0/180431163 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8654067460 0x7f8654069920 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f867c0ce9f0 tx=0x7f8664008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:20.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.801+0000 7f8678c00640 1 -- 172.21.15.27:0/180431163 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f866c051980 con 0x7f865c004d10 2024-01-15T18:15:20.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:20.989+0000 7f8683400640 1 -- 172.21.15.27:0/180431163 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone519310", "target": ["mon-mgr", ""]}) v1 -- 0x7f867c0cf990 con 0x7f8654067460 2024-01-15T18:15:21.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.005+0000 7f8678c00640 1 -- 172.21.15.27:0/180431163 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f867c0cf990 con 0x7f8654067460 2024-01-15T18:15:21.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.009+0000 7f8683400640 1 -- 172.21.15.27:0/180431163 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8654067460 msgr2=0x7f8654069920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.009+0000 7f8683400640 1 --2- 172.21.15.27:0/180431163 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8654067460 0x7f8654069920 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f867c0ce9f0 tx=0x7f8664008040 comp rx=0 tx=0).stop 2024-01-15T18:15:21.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.009+0000 7f8683400640 1 -- 172.21.15.27:0/180431163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f865c004d10 msgr2=0x7f867c1f2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.009+0000 7f8683400640 1 --2- 172.21.15.27:0/180431163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f865c004d10 0x7f867c1f2260 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f866c007340 tx=0x7f866c007800 comp rx=0 tx=0).stop 2024-01-15T18:15:21.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.009+0000 7f8683400640 1 -- 172.21.15.27:0/180431163 shutdown_connections 2024-01-15T18:15:21.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.009+0000 7f8683400640 1 --2- 172.21.15.27:0/180431163 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8654067460 0x7f8654069920 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.009+0000 7f8683400640 1 --2- 172.21.15.27:0/180431163 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f867c1f65e0 0x7f867c1f6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.009+0000 7f8683400640 1 --2- 172.21.15.27:0/180431163 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f867c0af9b0 0x7f867c0aca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.009+0000 7f8683400640 1 --2- 172.21.15.27:0/180431163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f865c004d10 0x7f867c1f2260 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.009+0000 7f8683400640 1 -- 172.21.15.27:0/180431163 >> 172.21.15.27:0/180431163 conn(0x7f867c0cb700 msgr2=0x7f867c15df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:21.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.009+0000 7f8683400640 1 -- 172.21.15.27:0/180431163 shutdown_connections 2024-01-15T18:15:21.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.009+0000 7f8683400640 1 -- 172.21.15.27:0/180431163 wait complete. 2024-01-15T18:15:21.034 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-15T18:15:21.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.181+0000 7fd339600640 1 Processor -- start 2024-01-15T18:15:21.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.181+0000 7fd339600640 1 -- start start 2024-01-15T18:15:21.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.185+0000 7fd339600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd3340cf460 0x7fd3340cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:21.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.185+0000 7fd339600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3340cfd80 con 0x7fd33415b4b0 2024-01-15T18:15:21.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.185+0000 7fd339600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3340d19a0 con 0x7fd3340b2590 2024-01-15T18:15:21.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.185+0000 7fd339600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3340d1c80 con 0x7fd3340cf460 2024-01-15T18:15:21.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.185+0000 7fd332a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd3340cf460 0x7fd3340cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:21.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.185+0000 7fd332000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd33415b4b0 0x7fd3340cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60872/0 (socket says 172.21.15.27:60872) 2024-01-15T18:15:21.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.185+0000 7fd332000640 1 -- 172.21.15.27:0/3014690484 learned_addr learned my addr 172.21.15.27:0/3014690484 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:21.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.185+0000 7fd332a00640 1 -- 172.21.15.27:0/3014690484 >> v1:172.21.15.93:6789/0 conn(0x7fd3340b2590 legacy=0x7fd3340cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.185+0000 7fd332a00640 1 -- 172.21.15.27:0/3014690484 >> v1:172.21.15.27:6789/0 conn(0x7fd33415b4b0 legacy=0x7fd3340cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.185+0000 7fd331600640 1 -- 172.21.15.27:0/3014690484 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634244676 0 0) 0x7fd3340cfd80 con 0x7fd33415b4b0 2024-01-15T18:15:21.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.185+0000 7fd332a00640 1 -- 172.21.15.27:0/3014690484 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3340b07c0 con 0x7fd3340cf460 2024-01-15T18:15:21.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.189+0000 7fd332a00640 1 --2- 172.21.15.27:0/3014690484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3340cf460 0x7fd3340cf840 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fd3240097c0 tx=0x7fd32401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=455f44764d9379dd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:21.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.189+0000 7fd331600640 1 -- 172.21.15.27:0/3014690484 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3240ab680 con 0x7fd3340cf460 2024-01-15T18:15:21.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.189+0000 7fd331600640 1 -- 172.21.15.27:0/3014690484 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3240ab7e0 con 0x7fd3340cf460 2024-01-15T18:15:21.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.189+0000 7fd331600640 1 -- 172.21.15.27:0/3014690484 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3240abb70 con 0x7fd3340cf460 2024-01-15T18:15:21.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.189+0000 7fd339600640 1 -- 172.21.15.27:0/3014690484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3340cf460 msgr2=0x7fd3340cf840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.189+0000 7fd339600640 1 --2- 172.21.15.27:0/3014690484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3340cf460 0x7fd3340cf840 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fd3240097c0 tx=0x7fd32401c480 comp rx=0 tx=0).stop 2024-01-15T18:15:21.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.189+0000 7fd339600640 1 -- 172.21.15.27:0/3014690484 shutdown_connections 2024-01-15T18:15:21.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.189+0000 7fd339600640 1 --2- 172.21.15.27:0/3014690484 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd3340cf460 0x7fd3340cf840 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.189+0000 7fd339600640 1 -- 172.21.15.27:0/3014690484 >> 172.21.15.27:0/3014690484 conn(0x7fd3340d0e80 msgr2=0x7fd3340d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:21.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.189+0000 7fd339600640 1 -- 172.21.15.27:0/3014690484 shutdown_connections 2024-01-15T18:15:21.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.189+0000 7fd339600640 1 -- 172.21.15.27:0/3014690484 wait complete. 2024-01-15T18:15:21.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.189+0000 7fd339600640 1 Processor -- start 2024-01-15T18:15:21.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.189+0000 7fd339600640 1 -- start start 2024-01-15T18:15:21.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.193+0000 7fd339600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd3340b2590 0x7fd3341492a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:21.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.193+0000 7fd339600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd3340cf460 0x7fd3341485a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:21.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.193+0000 7fd339600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd33415b4b0 0x7fd3341570c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:21.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.193+0000 7fd339600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33406bbf0 con 0x7fd33415b4b0 2024-01-15T18:15:21.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.193+0000 7fd339600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33406bed0 con 0x7fd3340b2590 2024-01-15T18:15:21.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.193+0000 7fd339600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33406c1b0 con 0x7fd3340cf460 2024-01-15T18:15:21.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.193+0000 7fd333400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd33415b4b0 0x7fd3341570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:21.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.193+0000 7fd333400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd33415b4b0 0x7fd3341570c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51456/0 (socket says 172.21.15.27:51456) 2024-01-15T18:15:21.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.193+0000 7fd333400640 1 -- 172.21.15.27:0/2607699487 learned_addr learned my addr 172.21.15.27:0/2607699487 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:21.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.193+0000 7fd332a00640 1 --2- 172.21.15.27:0/2607699487 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd3340b2590 0x7fd3341492a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:21.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.193+0000 7fd332000640 1 --2- 172.21.15.27:0/2607699487 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd3340cf460 0x7fd3341485a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:21.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.193+0000 7fd333400640 1 -- 172.21.15.27:0/2607699487 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd3340cf460 msgr2=0x7fd3341485a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.193+0000 7fd333400640 1 --2- 172.21.15.27:0/2607699487 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd3340cf460 0x7fd3341485a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.193+0000 7fd333400640 1 -- 172.21.15.27:0/2607699487 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd3340b2590 msgr2=0x7fd3341492a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.193+0000 7fd333400640 1 --2- 172.21.15.27:0/2607699487 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd3340b2590 0x7fd3341492a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.193+0000 7fd333400640 1 -- 172.21.15.27:0/2607699487 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd324009420 con 0x7fd33415b4b0 2024-01-15T18:15:21.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.193+0000 7fd332a00640 1 --2- 172.21.15.27:0/2607699487 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd3340b2590 0x7fd3341492a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:21.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.193+0000 7fd333400640 1 --2- 172.21.15.27:0/2607699487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd33415b4b0 0x7fd3341570c0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fd32c00c7a0 tx=0x7fd32c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:21.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.193+0000 7fd32aa00640 1 -- 172.21.15.27:0/2607699487 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c003b80 con 0x7fd33415b4b0 2024-01-15T18:15:21.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.193+0000 7fd32aa00640 1 -- 172.21.15.27:0/2607699487 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32c003ce0 con 0x7fd33415b4b0 2024-01-15T18:15:21.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.193+0000 7fd339600640 1 -- 172.21.15.27:0/2607699487 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd33406c490 con 0x7fd33415b4b0 2024-01-15T18:15:21.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.193+0000 7fd32aa00640 1 -- 172.21.15.27:0/2607699487 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32c00aa30 con 0x7fd33415b4b0 2024-01-15T18:15:21.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.193+0000 7fd339600640 1 -- 172.21.15.27:0/2607699487 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd334148ae0 con 0x7fd33415b4b0 2024-01-15T18:15:21.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.193+0000 7fd339600640 1 -- 172.21.15.27:0/2607699487 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd33404e850 con 0x7fd33415b4b0 2024-01-15T18:15:21.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.193+0000 7fd32aa00640 1 -- 172.21.15.27:0/2607699487 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 29) v1 ==== 87282+0+0 (secure 0 0 0) 0x7fd32c00ab90 con 0x7fd33415b4b0 2024-01-15T18:15:21.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.197+0000 7fd32aa00640 1 --2- 172.21.15.27:0/2607699487 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd30c067550 0x7fd30c069a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:21.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.197+0000 7fd32aa00640 1 -- 172.21.15.27:0/2607699487 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fd32c082c60 con 0x7fd33415b4b0 2024-01-15T18:15:21.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.197+0000 7fd332a00640 1 --2- 172.21.15.27:0/2607699487 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd30c067550 0x7fd30c069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:21.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.197+0000 7fd332a00640 1 --2- 172.21.15.27:0/2607699487 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd30c067550 0x7fd30c069a10 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd3240af000 tx=0x7fd3240a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:21.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.201+0000 7fd32aa00640 1 -- 172.21.15.27:0/2607699487 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd32c051950 con 0x7fd33415b4b0 2024-01-15T18:15:21.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.553+0000 7fd339600640 1 -- 172.21.15.27:0/2607699487 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd3340cc840 con 0x7fd30c067550 2024-01-15T18:15:21.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.581+0000 7fd32aa00640 1 -- 172.21.15.27:0/2607699487 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fd3240ab950 con 0x7fd30c067550 2024-01-15T18:15:21.583 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:15:21.583 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:15:21.583 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:21.583 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ss1" 2024-01-15T18:15:21.583 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:21.583 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:15:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.585+0000 7fd339600640 1 -- 172.21.15.27:0/2607699487 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd30c067550 msgr2=0x7fd30c069a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.585+0000 7fd339600640 1 --2- 172.21.15.27:0/2607699487 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd30c067550 0x7fd30c069a10 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd3240af000 tx=0x7fd3240a7000 comp rx=0 tx=0).stop 2024-01-15T18:15:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.585+0000 7fd339600640 1 -- 172.21.15.27:0/2607699487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd33415b4b0 msgr2=0x7fd3341570c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.585+0000 7fd339600640 1 --2- 172.21.15.27:0/2607699487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd33415b4b0 0x7fd3341570c0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fd32c00c7a0 tx=0x7fd32c00cc60 comp rx=0 tx=0).stop 2024-01-15T18:15:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.585+0000 7fd339600640 1 -- 172.21.15.27:0/2607699487 shutdown_connections 2024-01-15T18:15:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.585+0000 7fd339600640 1 --2- 172.21.15.27:0/2607699487 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd30c067550 0x7fd30c069a10 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.585+0000 7fd339600640 1 --2- 172.21.15.27:0/2607699487 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd33415b4b0 0x7fd3341570c0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.585+0000 7fd339600640 1 --2- 172.21.15.27:0/2607699487 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd3340cf460 0x7fd3341485a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.585+0000 7fd339600640 1 --2- 172.21.15.27:0/2607699487 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd3340b2590 0x7fd3341492a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.585+0000 7fd339600640 1 -- 172.21.15.27:0/2607699487 >> 172.21.15.27:0/2607699487 conn(0x7fd3340d0e80 msgr2=0x7fd33415a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:21.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.585+0000 7fd339600640 1 -- 172.21.15.27:0/2607699487 shutdown_connections 2024-01-15T18:15:21.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.585+0000 7fd339600640 1 -- 172.21.15.27:0/2607699487 wait complete. 2024-01-15T18:15:21.622 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:15:21.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.781+0000 7fa5cc600640 1 Processor -- start 2024-01-15T18:15:21.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.781+0000 7fa5cc600640 1 -- start start 2024-01-15T18:15:21.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.781+0000 7fa5cc600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5c406ac00 0x7fa5c4161120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:21.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.781+0000 7fa5cc600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c40cde30 con 0x7fa5c4146a60 2024-01-15T18:15:21.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.781+0000 7fa5cc600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c406fa50 con 0x7fa5c4160cb0 2024-01-15T18:15:21.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.781+0000 7fa5cc600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5c4161820 con 0x7fa5c406ac00 2024-01-15T18:15:21.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.781+0000 7fa5c9c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa5c4146a60 0x7fa5c40ac880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60884/0 (socket says 172.21.15.27:60884) 2024-01-15T18:15:21.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.781+0000 7fa5c9c00640 1 -- 172.21.15.27:0/4075140273 learned_addr learned my addr 172.21.15.27:0/4075140273 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:21.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.781+0000 7fa5ca600640 1 --2- 172.21.15.27:0/4075140273 >> v2:172.21.15.27:3300/0 conn(0x7fa5c406ac00 0x7fa5c4161120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:21.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.781+0000 7fa5c3e00640 1 -- 172.21.15.27:0/4075140273 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170126247 0 0) 0x7fa5c40cde30 con 0x7fa5c4146a60 2024-01-15T18:15:21.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.781+0000 7fa5c3e00640 1 -- 172.21.15.27:0/4075140273 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5a4003530 con 0x7fa5c4146a60 2024-01-15T18:15:21.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.781+0000 7fa5ca600640 1 -- 172.21.15.27:0/4075140273 >> v1:172.21.15.93:6789/0 conn(0x7fa5c4160cb0 legacy=0x7fa5c406a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.781+0000 7fa5ca600640 1 -- 172.21.15.27:0/4075140273 >> v1:172.21.15.27:6789/0 conn(0x7fa5c4146a60 legacy=0x7fa5c40ac880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.781+0000 7fa5ca600640 1 -- 172.21.15.27:0/4075140273 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c4161b00 con 0x7fa5c406ac00 2024-01-15T18:15:21.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.785+0000 7fa5ca600640 1 --2- 172.21.15.27:0/4075140273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5c406ac00 0x7fa5c4161120 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc009d90 tx=0x7fa5bc01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4f1b861df6ead8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:21.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.785+0000 7fa5c3e00640 1 -- 172.21.15.27:0/4075140273 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc0a5a10 con 0x7fa5c406ac00 2024-01-15T18:15:21.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.785+0000 7fa5c3e00640 1 -- 172.21.15.27:0/4075140273 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5bc0a5b70 con 0x7fa5c406ac00 2024-01-15T18:15:21.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.785+0000 7fa5c3e00640 1 -- 172.21.15.27:0/4075140273 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc0ad440 con 0x7fa5c406ac00 2024-01-15T18:15:21.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.785+0000 7fa5cc600640 1 -- 172.21.15.27:0/4075140273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5c406ac00 msgr2=0x7fa5c4161120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.785+0000 7fa5cc600640 1 --2- 172.21.15.27:0/4075140273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5c406ac00 0x7fa5c4161120 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc009d90 tx=0x7fa5bc01c950 comp rx=0 tx=0).stop 2024-01-15T18:15:21.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.785+0000 7fa5cc600640 1 -- 172.21.15.27:0/4075140273 shutdown_connections 2024-01-15T18:15:21.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.785+0000 7fa5cc600640 1 --2- 172.21.15.27:0/4075140273 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5c406ac00 0x7fa5c4161120 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.785+0000 7fa5cc600640 1 -- 172.21.15.27:0/4075140273 >> 172.21.15.27:0/4075140273 conn(0x7fa5c40cce10 msgr2=0x7fa5c40b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:21.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.785+0000 7fa5cc600640 1 -- 172.21.15.27:0/4075140273 shutdown_connections 2024-01-15T18:15:21.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.785+0000 7fa5cc600640 1 -- 172.21.15.27:0/4075140273 wait complete. 2024-01-15T18:15:21.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.789+0000 7fa5cc600640 1 Processor -- start 2024-01-15T18:15:21.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.789+0000 7fa5cc600640 1 -- start start 2024-01-15T18:15:21.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.789+0000 7fa5cc600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa5c406ac00 0x7fa5c415b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:21.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.789+0000 7fa5cc600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5c4146a60 0x7fa5c4145e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:21.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.789+0000 7fa5cc600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5c4160cb0 0x7fa5c415cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:21.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.789+0000 7fa5cc600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c415d320 con 0x7fa5c4146a60 2024-01-15T18:15:21.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.789+0000 7fa5cc600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c415d110 con 0x7fa5c4160cb0 2024-01-15T18:15:21.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.789+0000 7fa5cc600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c415b560 con 0x7fa5c406ac00 2024-01-15T18:15:21.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.789+0000 7fa5c9200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5c4146a60 0x7fa5c4145e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:21.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.789+0000 7fa5c9200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5c4146a60 0x7fa5c4145e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51486/0 (socket says 172.21.15.27:51486) 2024-01-15T18:15:21.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.789+0000 7fa5c9200640 1 -- 172.21.15.27:0/4134503220 learned_addr learned my addr 172.21.15.27:0/4134503220 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:21.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.789+0000 7fa5c9c00640 1 --2- 172.21.15.27:0/4134503220 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa5c406ac00 0x7fa5c415b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:21.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.789+0000 7fa5ca600640 1 --2- 172.21.15.27:0/4134503220 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5c4160cb0 0x7fa5c415cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:21.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.789+0000 7fa5c9200640 1 -- 172.21.15.27:0/4134503220 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa5c406ac00 msgr2=0x7fa5c415b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.789+0000 7fa5c9200640 1 --2- 172.21.15.27:0/4134503220 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa5c406ac00 0x7fa5c415b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.789+0000 7fa5c9200640 1 -- 172.21.15.27:0/4134503220 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5c4160cb0 msgr2=0x7fa5c415cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.789+0000 7fa5c9200640 1 --2- 172.21.15.27:0/4134503220 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5c4160cb0 0x7fa5c415cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.789+0000 7fa5c9200640 1 -- 172.21.15.27:0/4134503220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5bc009a70 con 0x7fa5c4146a60 2024-01-15T18:15:21.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.789+0000 7fa5c9200640 1 --2- 172.21.15.27:0/4134503220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5c4146a60 0x7fa5c4145e90 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac00c3c0 tx=0x7fa5ac00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:21.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.789+0000 7fa5c1600640 1 -- 172.21.15.27:0/4134503220 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac0052d0 con 0x7fa5c4146a60 2024-01-15T18:15:21.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.789+0000 7fa5c1600640 1 -- 172.21.15.27:0/4134503220 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ac005430 con 0x7fa5c4146a60 2024-01-15T18:15:21.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.789+0000 7fa5cc600640 1 -- 172.21.15.27:0/4134503220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c415ac30 con 0x7fa5c4146a60 2024-01-15T18:15:21.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.789+0000 7fa5c1600640 1 -- 172.21.15.27:0/4134503220 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac003840 con 0x7fa5c4146a60 2024-01-15T18:15:21.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.789+0000 7fa5cc600640 1 -- 172.21.15.27:0/4134503220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c415b1f0 con 0x7fa5c4146a60 2024-01-15T18:15:21.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.789+0000 7fa5cc600640 1 -- 172.21.15.27:0/4134503220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c415b6f0 con 0x7fa5c4146a60 2024-01-15T18:15:21.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.793+0000 7fa5c1600640 1 -- 172.21.15.27:0/4134503220 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 29) v1 ==== 87282+0+0 (secure 0 0 0) 0x7fa5ac0039a0 con 0x7fa5c4146a60 2024-01-15T18:15:21.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.793+0000 7fa5c1600640 1 --2- 172.21.15.27:0/4134503220 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa59c067550 0x7fa59c069a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:21.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.793+0000 7fa5c1600640 1 -- 172.21.15.27:0/4134503220 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fa5ac082560 con 0x7fa5c4146a60 2024-01-15T18:15:21.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.793+0000 7fa5c9c00640 1 --2- 172.21.15.27:0/4134503220 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa59c067550 0x7fa59c069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:21.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.793+0000 7fa5c9c00640 1 --2- 172.21.15.27:0/4134503220 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa59c067550 0x7fa59c069a10 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fa5b40011f0 tx=0x7fa5b4007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:21.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.797+0000 7fa5c1600640 1 -- 172.21.15.27:0/4134503220 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa5ac051000 con 0x7fa5c4146a60 2024-01-15T18:15:21.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:21.985+0000 7fa5cc600640 1 -- 172.21.15.27:0/4134503220 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa5c416b060 con 0x7fa59c067550 2024-01-15T18:15:22.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.009+0000 7fa5c1600640 1 -- 172.21.15.27:0/4134503220 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa5b4006b50 con 0x7fa59c067550 2024-01-15T18:15:22.010 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:15:22.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.013+0000 7fa5cc600640 1 -- 172.21.15.27:0/4134503220 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa59c067550 msgr2=0x7fa59c069a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.013+0000 7fa5cc600640 1 --2- 172.21.15.27:0/4134503220 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa59c067550 0x7fa59c069a10 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fa5b40011f0 tx=0x7fa5b4007d40 comp rx=0 tx=0).stop 2024-01-15T18:15:22.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.013+0000 7fa5cc600640 1 -- 172.21.15.27:0/4134503220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5c4146a60 msgr2=0x7fa5c4145e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.013+0000 7fa5cc600640 1 --2- 172.21.15.27:0/4134503220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5c4146a60 0x7fa5c4145e90 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac00c3c0 tx=0x7fa5ac00c880 comp rx=0 tx=0).stop 2024-01-15T18:15:22.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.013+0000 7fa5cc600640 1 -- 172.21.15.27:0/4134503220 shutdown_connections 2024-01-15T18:15:22.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.013+0000 7fa5cc600640 1 --2- 172.21.15.27:0/4134503220 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa59c067550 0x7fa59c069a10 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.013+0000 7fa5cc600640 1 --2- 172.21.15.27:0/4134503220 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5c4160cb0 0x7fa5c415cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.013+0000 7fa5cc600640 1 --2- 172.21.15.27:0/4134503220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5c4146a60 0x7fa5c4145e90 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.013+0000 7fa5cc600640 1 --2- 172.21.15.27:0/4134503220 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa5c406ac00 0x7fa5c415b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.013+0000 7fa5cc600640 1 -- 172.21.15.27:0/4134503220 >> 172.21.15.27:0/4134503220 conn(0x7fa5c40cce10 msgr2=0x7fa5c4071a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:22.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.013+0000 7fa5cc600640 1 -- 172.21.15.27:0/4134503220 shutdown_connections 2024-01-15T18:15:22.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.013+0000 7fa5cc600640 1 -- 172.21.15.27:0/4134503220 wait complete. 2024-01-15T18:15:22.033 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-15T18:15:22.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.177+0000 7fbbc6800640 1 Processor -- start 2024-01-15T18:15:22.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.177+0000 7fbbc6800640 1 -- start start 2024-01-15T18:15:22.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.177+0000 7fbbc6800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbbc015b4b0 0x7fbbc00b1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.177+0000 7fbbc6800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fbbc00cec60 0x7fbbc00b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.177+0000 7fbbc6800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc00d19a0 con 0x7fbbc00cf460 2024-01-15T18:15:22.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.177+0000 7fbbc6800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbc00d1c80 con 0x7fbbc00cec60 2024-01-15T18:15:22.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.177+0000 7fbbc6800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc00b07c0 con 0x7fbbc015b4b0 2024-01-15T18:15:22.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.177+0000 7fbbbf400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbbc00cf460 0x7fbbc00cf840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60898/0 (socket says 172.21.15.27:60898) 2024-01-15T18:15:22.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.177+0000 7fbbbf400640 1 -- 172.21.15.27:0/2760367757 learned_addr learned my addr 172.21.15.27:0/2760367757 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:22.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.177+0000 7fbbbea00640 1 --2- 172.21.15.27:0/2760367757 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbbc015b4b0 0x7fbbc00b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.177+0000 7fbbbfe00640 1 --2- 172.21.15.27:0/2760367757 >> v2:172.21.15.93:3300/0 conn(0x7fbbc00cec60 0x7fbbc00b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.177+0000 7fbbbe000640 1 -- 172.21.15.27:0/2760367757 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902167487 0 0) 0x7fbbc00d19a0 con 0x7fbbc00cf460 2024-01-15T18:15:22.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.177+0000 7fbbbe000640 1 -- 172.21.15.27:0/2760367757 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbba0003530 con 0x7fbbc00cf460 2024-01-15T18:15:22.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.177+0000 7fbbbea00640 1 -- 172.21.15.27:0/2760367757 >> v2:172.21.15.93:3300/0 conn(0x7fbbc00cec60 msgr2=0x7fbbc00b1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:22.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.177+0000 7fbbbea00640 1 --2- 172.21.15.27:0/2760367757 >> v2:172.21.15.93:3300/0 conn(0x7fbbc00cec60 0x7fbbc00b1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.177+0000 7fbbbea00640 1 -- 172.21.15.27:0/2760367757 >> v1:172.21.15.27:6789/0 conn(0x7fbbc00cf460 legacy=0x7fbbc00cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.177+0000 7fbbbea00640 1 -- 172.21.15.27:0/2760367757 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc00b0aa0 con 0x7fbbc015b4b0 2024-01-15T18:15:22.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.177+0000 7fbbbfe00640 1 --2- 172.21.15.27:0/2760367757 >> v2:172.21.15.93:3300/0 conn(0x7fbbc00cec60 0x7fbbc00b1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:15:22.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.181+0000 7fbbbea00640 1 --2- 172.21.15.27:0/2760367757 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbbc015b4b0 0x7fbbc00b1580 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbba80097c0 tx=0x7fbba801c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=604e144bcaf7320 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:22.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.181+0000 7fbbbe000640 1 -- 172.21.15.27:0/2760367757 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc00b2810 con 0x7fbbc015b4b0 2024-01-15T18:15:22.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.181+0000 7fbbbe000640 1 -- 172.21.15.27:0/2760367757 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba80ab680 con 0x7fbbc015b4b0 2024-01-15T18:15:22.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.181+0000 7fbbbe000640 1 -- 172.21.15.27:0/2760367757 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba80aba10 con 0x7fbbc015b4b0 2024-01-15T18:15:22.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.181+0000 7fbbc6800640 1 -- 172.21.15.27:0/2760367757 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbbc015b4b0 msgr2=0x7fbbc00b1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.181+0000 7fbbc6800640 1 --2- 172.21.15.27:0/2760367757 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbbc015b4b0 0x7fbbc00b1580 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbba80097c0 tx=0x7fbba801c480 comp rx=0 tx=0).stop 2024-01-15T18:15:22.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.181+0000 7fbbc6800640 1 -- 172.21.15.27:0/2760367757 shutdown_connections 2024-01-15T18:15:22.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.181+0000 7fbbc6800640 1 --2- 172.21.15.27:0/2760367757 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbbc015b4b0 0x7fbbc00b1580 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.181+0000 7fbbc6800640 1 --2- 172.21.15.27:0/2760367757 >> v2:172.21.15.93:3300/0 conn(0x7fbbc00cec60 0x7fbbc00b1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.181+0000 7fbbc6800640 1 -- 172.21.15.27:0/2760367757 >> 172.21.15.27:0/2760367757 conn(0x7fbbc00d0e80 msgr2=0x7fbbc00d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:22.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.181+0000 7fbbc6800640 1 -- 172.21.15.27:0/2760367757 shutdown_connections 2024-01-15T18:15:22.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.181+0000 7fbbc6800640 1 -- 172.21.15.27:0/2760367757 wait complete. 2024-01-15T18:15:22.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.185+0000 7fbbc6800640 1 Processor -- start 2024-01-15T18:15:22.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.185+0000 7fbbc6800640 1 -- start start 2024-01-15T18:15:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.185+0000 7fbbc6800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbbc00cec60 0x7fbbc0150830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.185+0000 7fbbc6800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbbc00cf460 0x7fbbc00aeac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.185+0000 7fbbc6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbc015b4b0 0x7fbbc00af000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.185+0000 7fbbc6800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc0150f60 con 0x7fbbc015b4b0 2024-01-15T18:15:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.185+0000 7fbbc6800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc0151240 con 0x7fbbc00cf460 2024-01-15T18:15:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.185+0000 7fbbc6800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc014fd50 con 0x7fbbc00cec60 2024-01-15T18:15:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.185+0000 7fbbbfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbc015b4b0 0x7fbbc00af000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.185+0000 7fbbbea00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbbc00cf460 0x7fbbc00aeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.185+0000 7fbbbea00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbbc00cf460 0x7fbbc00aeac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:59096/0 (socket says 172.21.15.27:59096) 2024-01-15T18:15:22.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.185+0000 7fbbbfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbc015b4b0 0x7fbbc00af000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51488/0 (socket says 172.21.15.27:51488) 2024-01-15T18:15:22.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.185+0000 7fbbbea00640 1 -- 172.21.15.27:0/15992502 learned_addr learned my addr 172.21.15.27:0/15992502 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:22.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.185+0000 7fbbbf400640 1 --2- 172.21.15.27:0/15992502 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbbc00cec60 0x7fbbc0150830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.185+0000 7fbbbfe00640 1 -- 172.21.15.27:0/15992502 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbbc00cec60 msgr2=0x7fbbc0150830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.185+0000 7fbbbfe00640 1 --2- 172.21.15.27:0/15992502 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbbc00cec60 0x7fbbc0150830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.185+0000 7fbbbfe00640 1 -- 172.21.15.27:0/15992502 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbbc00cf460 msgr2=0x7fbbc00aeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.185+0000 7fbbbfe00640 1 --2- 172.21.15.27:0/15992502 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbbc00cf460 0x7fbbc00aeac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.185+0000 7fbbbfe00640 1 -- 172.21.15.27:0/15992502 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb8009a70 con 0x7fbbc015b4b0 2024-01-15T18:15:22.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.185+0000 7fbbbfe00640 1 --2- 172.21.15.27:0/15992502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbc015b4b0 0x7fbbc00af000 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fbbb800efc0 tx=0x7fbbb800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:22.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.185+0000 7fbbb7400640 1 -- 172.21.15.27:0/15992502 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb800cd30 con 0x7fbbc015b4b0 2024-01-15T18:15:22.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.185+0000 7fbbb7400640 1 -- 172.21.15.27:0/15992502 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb800f040 con 0x7fbbc015b4b0 2024-01-15T18:15:22.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.185+0000 7fbbc6800640 1 -- 172.21.15.27:0/15992502 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbba8009420 con 0x7fbbc015b4b0 2024-01-15T18:15:22.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.185+0000 7fbbb7400640 1 -- 172.21.15.27:0/15992502 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb8003be0 con 0x7fbbc015b4b0 2024-01-15T18:15:22.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.185+0000 7fbbc6800640 1 -- 172.21.15.27:0/15992502 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc0150310 con 0x7fbbc015b4b0 2024-01-15T18:15:22.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.189+0000 7fbbc6800640 1 -- 172.21.15.27:0/15992502 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc0150500 con 0x7fbbc015b4b0 2024-01-15T18:15:22.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.189+0000 7fbbb7400640 1 -- 172.21.15.27:0/15992502 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 29) v1 ==== 87282+0+0 (secure 0 0 0) 0x7fbbb8014480 con 0x7fbbc015b4b0 2024-01-15T18:15:22.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.189+0000 7fbbb7400640 1 --2- 172.21.15.27:0/15992502 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbb98067550 0x7fbb98069a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.189+0000 7fbbb7400640 1 -- 172.21.15.27:0/15992502 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fbbb80838e0 con 0x7fbbc015b4b0 2024-01-15T18:15:22.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.189+0000 7fbbbf400640 1 --2- 172.21.15.27:0/15992502 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbb98067550 0x7fbb98069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.193+0000 7fbbbf400640 1 --2- 172.21.15.27:0/15992502 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbb98067550 0x7fbb98069a10 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fbbb00011f0 tx=0x7fbbb0007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:22.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.193+0000 7fbbb7400640 1 -- 172.21.15.27:0/15992502 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbbb80525a0 con 0x7fbbc015b4b0 2024-01-15T18:15:22.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.381+0000 7fbbc6800640 1 -- 172.21.15.27:0/15992502 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbc00ce590 con 0x7fbb98067550 2024-01-15T18:15:22.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.397+0000 7fbbb7400640 1 -- 172.21.15.27:0/15992502 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbbc00ce590 con 0x7fbb98067550 2024-01-15T18:15:22.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.397+0000 7fbbc6800640 1 -- 172.21.15.27:0/15992502 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbb98067550 msgr2=0x7fbb98069a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.397+0000 7fbbc6800640 1 --2- 172.21.15.27:0/15992502 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbb98067550 0x7fbb98069a10 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fbbb00011f0 tx=0x7fbbb0007d40 comp rx=0 tx=0).stop 2024-01-15T18:15:22.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.397+0000 7fbbc6800640 1 -- 172.21.15.27:0/15992502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbc015b4b0 msgr2=0x7fbbc00af000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.397+0000 7fbbc6800640 1 --2- 172.21.15.27:0/15992502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbc015b4b0 0x7fbbc00af000 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fbbb800efc0 tx=0x7fbbb800c490 comp rx=0 tx=0).stop 2024-01-15T18:15:22.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.401+0000 7fbbc6800640 1 -- 172.21.15.27:0/15992502 shutdown_connections 2024-01-15T18:15:22.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.401+0000 7fbbc6800640 1 --2- 172.21.15.27:0/15992502 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbb98067550 0x7fbb98069a10 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.401+0000 7fbbc6800640 1 --2- 172.21.15.27:0/15992502 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbbc015b4b0 0x7fbbc00af000 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.401+0000 7fbbc6800640 1 --2- 172.21.15.27:0/15992502 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbbc00cf460 0x7fbbc00aeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.401+0000 7fbbc6800640 1 --2- 172.21.15.27:0/15992502 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbbc00cec60 0x7fbbc0150830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.401+0000 7fbbc6800640 1 -- 172.21.15.27:0/15992502 >> 172.21.15.27:0/15992502 conn(0x7fbbc00d0e80 msgr2=0x7fbbc00cd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:22.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.401+0000 7fbbc6800640 1 -- 172.21.15.27:0/15992502 shutdown_connections 2024-01-15T18:15:22.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.401+0000 7fbbc6800640 1 -- 172.21.15.27:0/15992502 wait complete. 2024-01-15T18:15:22.423 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone51935 2024-01-15T18:15:22.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.569+0000 7fa87f400640 1 Processor -- start 2024-01-15T18:15:22.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.569+0000 7fa87f400640 1 -- start start 2024-01-15T18:15:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.573+0000 7fa87f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa8780cde10 0x7fa8780cbf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.573+0000 7fa87f400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8780cfc30 0x7fa8780cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.573+0000 7fa87f400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8780cb4d0 con 0x7fa87815d680 2024-01-15T18:15:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.573+0000 7fa87f400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8780d0120 con 0x7fa8780cde10 2024-01-15T18:15:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.573+0000 7fa87f400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8780d0400 con 0x7fa8780cfc30 2024-01-15T18:15:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.573+0000 7fa877e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa8780cde10 0x7fa8780cbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.573+0000 7fa877e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa8780cde10 0x7fa8780cbf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51496/0 (socket says 172.21.15.27:51496) 2024-01-15T18:15:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.573+0000 7fa877e00640 1 -- 172.21.15.27:0/4135295103 learned_addr learned my addr 172.21.15.27:0/4135295103 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.573+0000 7fa87d400640 1 --2- 172.21.15.27:0/4135295103 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8780cfc30 0x7fa8780cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.573+0000 7fa877e00640 1 -- 172.21.15.27:0/4135295103 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8780cfc30 msgr2=0x7fa8780cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.573+0000 7fa877e00640 1 --2- 172.21.15.27:0/4135295103 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8780cfc30 0x7fa8780cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.573+0000 7fa877e00640 1 -- 172.21.15.27:0/4135295103 >> v1:172.21.15.93:6789/0 conn(0x7fa87815d680 legacy=0x7fa8780cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.573+0000 7fa877e00640 1 -- 172.21.15.27:0/4135295103 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8780cb770 con 0x7fa8780cde10 2024-01-15T18:15:22.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.577+0000 7fa877e00640 1 --2- 172.21.15.27:0/4135295103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8780cde10 0x7fa8780cbf70 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fa860009870 tx=0x7fa86001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4cf0de537fa4217 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:22.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.577+0000 7fa877400640 1 -- 172.21.15.27:0/4135295103 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8780ce850 con 0x7fa8780cde10 2024-01-15T18:15:22.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.577+0000 7fa877400640 1 -- 172.21.15.27:0/4135295103 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8600a95f0 con 0x7fa8780cde10 2024-01-15T18:15:22.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.577+0000 7fa877400640 1 -- 172.21.15.27:0/4135295103 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8600a9980 con 0x7fa8780cde10 2024-01-15T18:15:22.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.577+0000 7fa87f400640 1 -- 172.21.15.27:0/4135295103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8780cde10 msgr2=0x7fa8780cbf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.577+0000 7fa87f400640 1 --2- 172.21.15.27:0/4135295103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8780cde10 0x7fa8780cbf70 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fa860009870 tx=0x7fa86001c510 comp rx=0 tx=0).stop 2024-01-15T18:15:22.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.577+0000 7fa87f400640 1 -- 172.21.15.27:0/4135295103 shutdown_connections 2024-01-15T18:15:22.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.577+0000 7fa87f400640 1 --2- 172.21.15.27:0/4135295103 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa8780cfc30 0x7fa8780cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.577+0000 7fa87f400640 1 --2- 172.21.15.27:0/4135295103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8780cde10 0x7fa8780cbf70 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.577+0000 7fa87f400640 1 -- 172.21.15.27:0/4135295103 >> 172.21.15.27:0/4135295103 conn(0x7fa8780d0e80 msgr2=0x7fa8780d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:22.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.577+0000 7fa87f400640 1 -- 172.21.15.27:0/4135295103 shutdown_connections 2024-01-15T18:15:22.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.577+0000 7fa87f400640 1 -- 172.21.15.27:0/4135295103 wait complete. 2024-01-15T18:15:22.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.577+0000 7fa87f400640 1 Processor -- start 2024-01-15T18:15:22.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.577+0000 7fa87f400640 1 -- start start 2024-01-15T18:15:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.581+0000 7fa87f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8780cde10 0x7fa87806b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.581+0000 7fa87f400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8780cfc30 0x7fa878152f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.581+0000 7fa87f400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa87815d680 0x7fa8781534a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.581+0000 7fa87f400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8781539e0 con 0x7fa8780cde10 2024-01-15T18:15:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.581+0000 7fa87f400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8780c62a0 con 0x7fa8780cfc30 2024-01-15T18:15:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.581+0000 7fa87f400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8780c6580 con 0x7fa87815d680 2024-01-15T18:15:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.581+0000 7fa87ca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8780cde10 0x7fa87806b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.581+0000 7fa87ca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8780cde10 0x7fa87806b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51512/0 (socket says 172.21.15.27:51512) 2024-01-15T18:15:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.581+0000 7fa87ca00640 1 -- 172.21.15.27:0/1398949935 learned_addr learned my addr 172.21.15.27:0/1398949935 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.581+0000 7fa877e00640 1 --2- 172.21.15.27:0/1398949935 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8780cfc30 0x7fa878152f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.581+0000 7fa87d400640 1 --2- 172.21.15.27:0/1398949935 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa87815d680 0x7fa8781534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.581+0000 7fa87ca00640 1 -- 172.21.15.27:0/1398949935 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa87815d680 msgr2=0x7fa8781534a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.581+0000 7fa87ca00640 1 --2- 172.21.15.27:0/1398949935 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa87815d680 0x7fa8781534a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.581+0000 7fa87ca00640 1 -- 172.21.15.27:0/1398949935 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8780cfc30 msgr2=0x7fa878152f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.581+0000 7fa87ca00640 1 --2- 172.21.15.27:0/1398949935 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8780cfc30 0x7fa878152f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.581+0000 7fa87ca00640 1 -- 172.21.15.27:0/1398949935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8600094d0 con 0x7fa8780cde10 2024-01-15T18:15:22.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.581+0000 7fa877e00640 1 --2- 172.21.15.27:0/1398949935 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8780cfc30 0x7fa878152f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:15:22.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.581+0000 7fa87ca00640 1 --2- 172.21.15.27:0/1398949935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8780cde10 0x7fa87806b910 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fa86800c3c0 tx=0x7fa86800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:22.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.581+0000 7fa874c00640 1 -- 172.21.15.27:0/1398949935 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8680052d0 con 0x7fa8780cde10 2024-01-15T18:15:22.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.581+0000 7fa874c00640 1 -- 172.21.15.27:0/1398949935 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa868005430 con 0x7fa8780cde10 2024-01-15T18:15:22.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.581+0000 7fa874c00640 1 -- 172.21.15.27:0/1398949935 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa868003840 con 0x7fa8780cde10 2024-01-15T18:15:22.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.581+0000 7fa87f400640 1 -- 172.21.15.27:0/1398949935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8780c6860 con 0x7fa8780cde10 2024-01-15T18:15:22.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.581+0000 7fa87f400640 1 -- 172.21.15.27:0/1398949935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8780c6e20 con 0x7fa8780cde10 2024-01-15T18:15:22.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.581+0000 7fa87f400640 1 -- 172.21.15.27:0/1398949935 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8780c7010 con 0x7fa8780cde10 2024-01-15T18:15:22.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.585+0000 7fa874c00640 1 -- 172.21.15.27:0/1398949935 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 29) v1 ==== 87282+0+0 (secure 0 0 0) 0x7fa8680039a0 con 0x7fa8780cde10 2024-01-15T18:15:22.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.585+0000 7fa874c00640 1 --2- 172.21.15.27:0/1398949935 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa850067550 0x7fa850069a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.585+0000 7fa874c00640 1 -- 172.21.15.27:0/1398949935 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fa868082570 con 0x7fa8780cde10 2024-01-15T18:15:22.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.585+0000 7fa877e00640 1 --2- 172.21.15.27:0/1398949935 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa850067550 0x7fa850069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.585+0000 7fa877e00640 1 --2- 172.21.15.27:0/1398949935 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa850067550 0x7fa850069a10 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa8600ae000 tx=0x7fa8600b3040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:22.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.589+0000 7fa874c00640 1 -- 172.21.15.27:0/1398949935 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa868051030 con 0x7fa8780cde10 2024-01-15T18:15:22.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.769+0000 7fa87f400640 1 -- 172.21.15.27:0/1398949935 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone51935", "target": ["mon-mgr", ""]}) v1 -- 0x7fa87815a9d0 con 0x7fa850067550 2024-01-15T18:15:22.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.789+0000 7fa874c00640 1 -- 172.21.15.27:0/1398949935 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa87815a9d0 con 0x7fa850067550 2024-01-15T18:15:22.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.789+0000 7fa87f400640 1 -- 172.21.15.27:0/1398949935 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa850067550 msgr2=0x7fa850069a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.789+0000 7fa87f400640 1 --2- 172.21.15.27:0/1398949935 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa850067550 0x7fa850069a10 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa8600ae000 tx=0x7fa8600b3040 comp rx=0 tx=0).stop 2024-01-15T18:15:22.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.789+0000 7fa87f400640 1 -- 172.21.15.27:0/1398949935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8780cde10 msgr2=0x7fa87806b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.789+0000 7fa87f400640 1 --2- 172.21.15.27:0/1398949935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8780cde10 0x7fa87806b910 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fa86800c3c0 tx=0x7fa86800c880 comp rx=0 tx=0).stop 2024-01-15T18:15:22.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.789+0000 7fa87f400640 1 -- 172.21.15.27:0/1398949935 shutdown_connections 2024-01-15T18:15:22.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.789+0000 7fa87f400640 1 --2- 172.21.15.27:0/1398949935 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa850067550 0x7fa850069a10 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.789+0000 7fa87f400640 1 --2- 172.21.15.27:0/1398949935 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa87815d680 0x7fa8781534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.789+0000 7fa87f400640 1 --2- 172.21.15.27:0/1398949935 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa8780cfc30 0x7fa878152f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.789+0000 7fa87f400640 1 --2- 172.21.15.27:0/1398949935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa8780cde10 0x7fa87806b910 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.789+0000 7fa87f400640 1 -- 172.21.15.27:0/1398949935 >> 172.21.15.27:0/1398949935 conn(0x7fa8780d0e80 msgr2=0x7fa8780d1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:22.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.789+0000 7fa87f400640 1 -- 172.21.15.27:0/1398949935 shutdown_connections 2024-01-15T18:15:22.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.789+0000 7fa87f400640 1 -- 172.21.15.27:0/1398949935 wait complete. 2024-01-15T18:15:22.815 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone519311 2024-01-15T18:15:22.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.961+0000 7f6eb4400640 1 Processor -- start 2024-01-15T18:15:22.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.961+0000 7f6eb4400640 1 -- start start 2024-01-15T18:15:22.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.961+0000 7f6eb4400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f6eac15b4b0 0x7f6eac0b1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.961+0000 7f6eb4400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6eac0cec60 0x7f6eac0b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.961+0000 7f6eb4400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eac0d19a0 con 0x7f6eac0cf460 2024-01-15T18:15:22.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.961+0000 7f6eb4400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eac0d1c80 con 0x7f6eac0cec60 2024-01-15T18:15:22.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.961+0000 7f6eb4400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eac0b07c0 con 0x7f6eac15b4b0 2024-01-15T18:15:22.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.961+0000 7f6eb2400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6eac0cec60 0x7f6eac0b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.961+0000 7f6eb1a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6eac0cf460 0x7f6eac0cf840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:60900/0 (socket says 172.21.15.27:60900) 2024-01-15T18:15:22.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.961+0000 7f6eb1a00640 1 -- 172.21.15.27:0/1022176645 learned_addr learned my addr 172.21.15.27:0/1022176645 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:22.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.961+0000 7f6eb1000640 1 --2- 172.21.15.27:0/1022176645 >> v2:172.21.15.93:3300/0 conn(0x7f6eac15b4b0 0x7f6eac0b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.961+0000 7f6eb2400640 1 -- 172.21.15.27:0/1022176645 >> v2:172.21.15.93:3300/0 conn(0x7f6eac15b4b0 msgr2=0x7f6eac0b1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:22.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.961+0000 7f6eb2400640 1 --2- 172.21.15.27:0/1022176645 >> v2:172.21.15.93:3300/0 conn(0x7f6eac15b4b0 0x7f6eac0b1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.961+0000 7f6eb2400640 1 -- 172.21.15.27:0/1022176645 >> v1:172.21.15.27:6789/0 conn(0x7f6eac0cf460 legacy=0x7f6eac0cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.961+0000 7f6eb2400640 1 -- 172.21.15.27:0/1022176645 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eac0b0aa0 con 0x7f6eac0cec60 2024-01-15T18:15:22.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.961+0000 7f6eabe00640 1 -- 172.21.15.27:0/1022176645 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156591109 0 0) 0x7f6eac0d19a0 con 0x7f6eac0cf460 2024-01-15T18:15:22.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.965+0000 7f6eb2400640 1 --2- 172.21.15.27:0/1022176645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eac0cec60 0x7f6eac0b1ac0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f6eac0717b0 tx=0x7f6ea401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f675df648f66cf7e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:22.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.965+0000 7f6eabe00640 1 -- 172.21.15.27:0/1022176645 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eac0b2810 con 0x7f6eac0cec60 2024-01-15T18:15:22.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.965+0000 7f6eabe00640 1 -- 172.21.15.27:0/1022176645 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ea40a59d0 con 0x7f6eac0cec60 2024-01-15T18:15:22.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.965+0000 7f6eabe00640 1 -- 172.21.15.27:0/1022176645 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea40a5d60 con 0x7f6eac0cec60 2024-01-15T18:15:22.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.965+0000 7f6eb4400640 1 -- 172.21.15.27:0/1022176645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eac0cec60 msgr2=0x7f6eac0b1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.965+0000 7f6eb4400640 1 --2- 172.21.15.27:0/1022176645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eac0cec60 0x7f6eac0b1ac0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f6eac0717b0 tx=0x7f6ea401c910 comp rx=0 tx=0).stop 2024-01-15T18:15:22.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.965+0000 7f6eb4400640 1 -- 172.21.15.27:0/1022176645 shutdown_connections 2024-01-15T18:15:22.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.965+0000 7f6eb4400640 1 --2- 172.21.15.27:0/1022176645 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eac0cec60 0x7f6eac0b1ac0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.965+0000 7f6eb4400640 1 --2- 172.21.15.27:0/1022176645 >> v2:172.21.15.93:3300/0 conn(0x7f6eac15b4b0 0x7f6eac0b1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.965+0000 7f6eb4400640 1 -- 172.21.15.27:0/1022176645 >> 172.21.15.27:0/1022176645 conn(0x7f6eac0d0e80 msgr2=0x7f6eac0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:22.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.965+0000 7f6eb4400640 1 -- 172.21.15.27:0/1022176645 shutdown_connections 2024-01-15T18:15:22.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.965+0000 7f6eb4400640 1 -- 172.21.15.27:0/1022176645 wait complete. 2024-01-15T18:15:22.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.969+0000 7f6eb4400640 1 Processor -- start 2024-01-15T18:15:22.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.969+0000 7f6eb4400640 1 -- start start 2024-01-15T18:15:22.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.969+0000 7f6eb4400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eac0cec60 0x7f6eac149bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.969+0000 7f6eb4400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6eac0cf460 0x7f6eac1591c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.969+0000 7f6eb4400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6eac15b4b0 0x7f6eac159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.969+0000 7f6eb4400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eac15a5b0 con 0x7f6eac0cec60 2024-01-15T18:15:22.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.969+0000 7f6eb4400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eac159d10 con 0x7f6eac0cf460 2024-01-15T18:15:22.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.969+0000 7f6eb4400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eac159ff0 con 0x7f6eac15b4b0 2024-01-15T18:15:22.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.969+0000 7f6eb1a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eac0cec60 0x7f6eac149bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.969+0000 7f6eb2400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6eac15b4b0 0x7f6eac159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.969+0000 7f6eb2400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6eac15b4b0 0x7f6eac159700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:34054/0 (socket says 172.21.15.27:34054) 2024-01-15T18:15:22.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.969+0000 7f6eb2400640 1 -- 172.21.15.27:0/3450981727 learned_addr learned my addr 172.21.15.27:0/3450981727 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:22.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.969+0000 7f6eb1000640 1 --2- 172.21.15.27:0/3450981727 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6eac0cf460 0x7f6eac1591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.969+0000 7f6eb1a00640 1 -- 172.21.15.27:0/3450981727 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6eac15b4b0 msgr2=0x7f6eac159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.969+0000 7f6eb1a00640 1 --2- 172.21.15.27:0/3450981727 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6eac15b4b0 0x7f6eac159700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.969+0000 7f6eb1a00640 1 -- 172.21.15.27:0/3450981727 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6eac0cf460 msgr2=0x7f6eac1591c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.969+0000 7f6eb1a00640 1 --2- 172.21.15.27:0/3450981727 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6eac0cf460 0x7f6eac1591c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.969+0000 7f6eb1a00640 1 -- 172.21.15.27:0/3450981727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea4009a70 con 0x7f6eac0cec60 2024-01-15T18:15:22.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.969+0000 7f6eb1a00640 1 --2- 172.21.15.27:0/3450981727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eac0cec60 0x7f6eac149bf0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c00c430 tx=0x7f6e9c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:22.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.969+0000 7f6ea9600640 1 -- 172.21.15.27:0/3450981727 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0074e0 con 0x7f6eac0cec60 2024-01-15T18:15:22.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.969+0000 7f6ea9600640 1 -- 172.21.15.27:0/3450981727 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9c007660 con 0x7f6eac0cec60 2024-01-15T18:15:22.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.969+0000 7f6ea9600640 1 -- 172.21.15.27:0/3450981727 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c004a30 con 0x7f6eac0cec60 2024-01-15T18:15:22.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.969+0000 7f6eb4400640 1 -- 172.21.15.27:0/3450981727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eac15a2d0 con 0x7f6eac0cec60 2024-01-15T18:15:22.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.969+0000 7f6eb4400640 1 -- 172.21.15.27:0/3450981727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eac14a130 con 0x7f6eac0cec60 2024-01-15T18:15:22.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.973+0000 7f6eb4400640 1 -- 172.21.15.27:0/3450981727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eac1634c0 con 0x7f6eac0cec60 2024-01-15T18:15:22.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.973+0000 7f6ea9600640 1 -- 172.21.15.27:0/3450981727 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 29) v1 ==== 87282+0+0 (secure 0 0 0) 0x7f6e9c004b90 con 0x7f6eac0cec60 2024-01-15T18:15:22.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.973+0000 7f6ea9600640 1 --2- 172.21.15.27:0/3450981727 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6e84067550 0x7f6e84069a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.973+0000 7f6ea9600640 1 -- 172.21.15.27:0/3450981727 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f6e9c082da0 con 0x7f6eac0cec60 2024-01-15T18:15:22.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.973+0000 7f6eb1000640 1 --2- 172.21.15.27:0/3450981727 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6e84067550 0x7f6e84069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.973+0000 7f6eb1000640 1 --2- 172.21.15.27:0/3450981727 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6e84067550 0x7f6e84069a10 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f6eac0ade30 tx=0x7f6e94008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:22.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:22.977+0000 7f6ea9600640 1 -- 172.21.15.27:0/3450981727 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6e9c051980 con 0x7f6eac0cec60 2024-01-15T18:15:23.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.157+0000 7f6eb4400640 1 -- 172.21.15.27:0/3450981727 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone519311", "target": ["mon-mgr", ""]}) v1 -- 0x7f6eac15a160 con 0x7f6e84067550 2024-01-15T18:15:23.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.177+0000 7f6ea9600640 1 -- 172.21.15.27:0/3450981727 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6eac15a160 con 0x7f6e84067550 2024-01-15T18:15:23.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.181+0000 7f6eb4400640 1 -- 172.21.15.27:0/3450981727 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6e84067550 msgr2=0x7f6e84069a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:23.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.181+0000 7f6eb4400640 1 --2- 172.21.15.27:0/3450981727 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6e84067550 0x7f6e84069a10 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f6eac0ade30 tx=0x7f6e94008040 comp rx=0 tx=0).stop 2024-01-15T18:15:23.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.181+0000 7f6eb4400640 1 -- 172.21.15.27:0/3450981727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eac0cec60 msgr2=0x7f6eac149bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:23.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.181+0000 7f6eb4400640 1 --2- 172.21.15.27:0/3450981727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eac0cec60 0x7f6eac149bf0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c00c430 tx=0x7f6e9c00c8f0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.181+0000 7f6eb4400640 1 -- 172.21.15.27:0/3450981727 shutdown_connections 2024-01-15T18:15:23.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.181+0000 7f6eb4400640 1 --2- 172.21.15.27:0/3450981727 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6e84067550 0x7f6e84069a10 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.181+0000 7f6eb4400640 1 --2- 172.21.15.27:0/3450981727 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6eac15b4b0 0x7f6eac159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.181+0000 7f6eb4400640 1 --2- 172.21.15.27:0/3450981727 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6eac0cf460 0x7f6eac1591c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.181+0000 7f6eb4400640 1 --2- 172.21.15.27:0/3450981727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6eac0cec60 0x7f6eac149bf0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.181+0000 7f6eb4400640 1 -- 172.21.15.27:0/3450981727 >> 172.21.15.27:0/3450981727 conn(0x7f6eac0d0e80 msgr2=0x7f6eac0ae380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:23.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.181+0000 7f6eb4400640 1 -- 172.21.15.27:0/3450981727 shutdown_connections 2024-01-15T18:15:23.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.181+0000 7f6eb4400640 1 -- 172.21.15.27:0/3450981727 wait complete. 2024-01-15T18:15:23.204 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone519312 2024-01-15T18:15:23.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.461+0000 7f5264a00640 1 Processor -- start 2024-01-15T18:15:23.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.461+0000 7f5264a00640 1 -- start start 2024-01-15T18:15:23.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.461+0000 7f5264a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f525c0ac170 0x7f525c0b1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:23.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.461+0000 7f5264a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f525c0ac590 0x7f525c06ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:23.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.461+0000 7f5264a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f525c070730 con 0x7f525c159a30 2024-01-15T18:15:23.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.461+0000 7f5264a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f525c070540 con 0x7f525c0ac590 2024-01-15T18:15:23.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.461+0000 7f5264a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f525c0709f0 con 0x7f525c0ac170 2024-01-15T18:15:23.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.461+0000 7f5262a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f525c0ac590 0x7f525c06ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:23.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.461+0000 7f5262a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f525c0ac590 0x7f525c06ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:59140/0 (socket says 172.21.15.27:59140) 2024-01-15T18:15:23.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.465+0000 7f5262a00640 1 -- 172.21.15.27:0/893245340 learned_addr learned my addr 172.21.15.27:0/893245340 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:23.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.465+0000 7f5261600640 1 --2- 172.21.15.27:0/893245340 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f525c0ac170 0x7f525c0b1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:23.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.465+0000 7f5262a00640 1 -- 172.21.15.27:0/893245340 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f525c0ac170 msgr2=0x7f525c0b1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:23.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.465+0000 7f5262a00640 1 --2- 172.21.15.27:0/893245340 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f525c0ac170 0x7f525c0b1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.465+0000 7f5262a00640 1 -- 172.21.15.27:0/893245340 >> v1:172.21.15.27:6789/0 conn(0x7f525c159a30 legacy=0x7f525c0b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:23.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.465+0000 7f5262a00640 1 -- 172.21.15.27:0/893245340 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f525c070cd0 con 0x7f525c0ac590 2024-01-15T18:15:23.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.469+0000 7f5262a00640 1 --2- 172.21.15.27:0/893245340 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f525c0ac590 0x7f525c06ce30 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f5254013150 tx=0x7f5254024be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12d1a418b7928f2f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:23.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.469+0000 7f5260c00640 1 -- 172.21.15.27:0/893245340 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52540adb80 con 0x7f525c0ac590 2024-01-15T18:15:23.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.469+0000 7f5260c00640 1 -- 172.21.15.27:0/893245340 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52540adce0 con 0x7f525c0ac590 2024-01-15T18:15:23.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.469+0000 7f5260c00640 1 -- 172.21.15.27:0/893245340 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52540112f0 con 0x7f525c0ac590 2024-01-15T18:15:23.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.469+0000 7f5264a00640 1 -- 172.21.15.27:0/893245340 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f525c0ac590 msgr2=0x7f525c06ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:23.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.473+0000 7f5264a00640 1 --2- 172.21.15.27:0/893245340 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f525c0ac590 0x7f525c06ce30 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f5254013150 tx=0x7f5254024be0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.473+0000 7f5264a00640 1 -- 172.21.15.27:0/893245340 shutdown_connections 2024-01-15T18:15:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.473+0000 7f5264a00640 1 --2- 172.21.15.27:0/893245340 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f525c0ac170 0x7f525c0b1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.473+0000 7f5264a00640 1 --2- 172.21.15.27:0/893245340 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f525c0ac590 0x7f525c06ce30 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.473+0000 7f5264a00640 1 -- 172.21.15.27:0/893245340 >> 172.21.15.27:0/893245340 conn(0x7f525c0b0bb0 msgr2=0x7f525c0b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:23.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.473+0000 7f5264a00640 1 -- 172.21.15.27:0/893245340 shutdown_connections 2024-01-15T18:15:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.473+0000 7f5264a00640 1 -- 172.21.15.27:0/893245340 wait complete. 2024-01-15T18:15:23.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.477+0000 7f5264a00640 1 Processor -- start 2024-01-15T18:15:23.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.477+0000 7f5264a00640 1 -- start start 2024-01-15T18:15:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.481+0000 7f5264a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f525c0ac170 0x7f525c09c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.481+0000 7f5264a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f525c0ac590 0x7f525c202f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.481+0000 7f5264a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f525c159a30 0x7f525c154970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.481+0000 7f5264a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525c203490 con 0x7f525c159a30 2024-01-15T18:15:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.481+0000 7f5264a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f525c203730 con 0x7f525c0ac590 2024-01-15T18:15:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.481+0000 7f5264a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f525c203a10 con 0x7f525c0ac170 2024-01-15T18:15:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.481+0000 7f5262000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f525c0ac170 0x7f525c09c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.481+0000 7f5262000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f525c0ac170 0x7f525c09c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:34070/0 (socket says 172.21.15.27:34070) 2024-01-15T18:15:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.481+0000 7f5262000640 1 -- 172.21.15.27:0/340662970 learned_addr learned my addr 172.21.15.27:0/340662970 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.481+0000 7f5262a00640 1 --2- 172.21.15.27:0/340662970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f525c159a30 0x7f525c154970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.481+0000 7f5261600640 1 --2- 172.21.15.27:0/340662970 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f525c0ac590 0x7f525c202f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.481+0000 7f5262000640 1 -- 172.21.15.27:0/340662970 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f525c0ac590 msgr2=0x7f525c202f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:23.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.481+0000 7f5262000640 1 --2- 172.21.15.27:0/340662970 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f525c0ac590 0x7f525c202f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.481+0000 7f5262000640 1 -- 172.21.15.27:0/340662970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f525c159a30 msgr2=0x7f525c154970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:23.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.481+0000 7f5262000640 1 --2- 172.21.15.27:0/340662970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f525c159a30 0x7f525c154970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.481+0000 7f5262000640 1 -- 172.21.15.27:0/340662970 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5254011d70 con 0x7f525c0ac170 2024-01-15T18:15:23.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.481+0000 7f5262000640 1 --2- 172.21.15.27:0/340662970 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f525c0ac170 0x7f525c09c120 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f524c00c7c0 tx=0x7f524c00cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:23.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.481+0000 7f525a000640 1 -- 172.21.15.27:0/340662970 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f524c00aa50 con 0x7f525c0ac170 2024-01-15T18:15:23.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.481+0000 7f525a000640 1 -- 172.21.15.27:0/340662970 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f524c00abb0 con 0x7f525c0ac170 2024-01-15T18:15:23.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.481+0000 7f525a000640 1 -- 172.21.15.27:0/340662970 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f524c012420 con 0x7f525c0ac170 2024-01-15T18:15:23.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.485+0000 7f5264a00640 1 -- 172.21.15.27:0/340662970 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f525c203cf0 con 0x7f525c0ac170 2024-01-15T18:15:23.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.485+0000 7f5264a00640 1 -- 172.21.15.27:0/340662970 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f525c154eb0 con 0x7f525c0ac170 2024-01-15T18:15:23.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.485+0000 7f5264a00640 1 -- 172.21.15.27:0/340662970 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f525c04e930 con 0x7f525c0ac170 2024-01-15T18:15:23.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.489+0000 7f525a000640 1 -- 172.21.15.27:0/340662970 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 29) v1 ==== 87282+0+0 (secure 0 0 0) 0x7f524c012580 con 0x7f525c0ac170 2024-01-15T18:15:23.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.489+0000 7f525a000640 1 --2- 172.21.15.27:0/340662970 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5234067620 0x7f5234069ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:23.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.489+0000 7f525a000640 1 -- 172.21.15.27:0/340662970 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f524c0834c0 con 0x7f525c0ac170 2024-01-15T18:15:23.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.489+0000 7f5261600640 1 --2- 172.21.15.27:0/340662970 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5234067620 0x7f5234069ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:23.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.493+0000 7f5261600640 1 --2- 172.21.15.27:0/340662970 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5234067620 0x7f5234069ae0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f525c0b0ad0 tx=0x7f5244009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:23.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.497+0000 7f525a000640 1 -- 172.21.15.27:0/340662970 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f524c051f80 con 0x7f525c0ac170 2024-01-15T18:15:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.749+0000 7f5264a00640 1 -- 172.21.15.27:0/340662970 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone519312", "target": ["mon-mgr", ""]}) v1 -- 0x7f525c0aebb0 con 0x7f5234067620 2024-01-15T18:15:23.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.769+0000 7f525a000640 1 -- 172.21.15.27:0/340662970 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f525c0aebb0 con 0x7f5234067620 2024-01-15T18:15:23.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.769+0000 7f5264a00640 1 -- 172.21.15.27:0/340662970 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5234067620 msgr2=0x7f5234069ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:23.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.769+0000 7f5264a00640 1 --2- 172.21.15.27:0/340662970 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5234067620 0x7f5234069ae0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f525c0b0ad0 tx=0x7f5244009190 comp rx=0 tx=0).stop 2024-01-15T18:15:23.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.769+0000 7f5264a00640 1 -- 172.21.15.27:0/340662970 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f525c0ac170 msgr2=0x7f525c09c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:23.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.769+0000 7f5264a00640 1 --2- 172.21.15.27:0/340662970 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f525c0ac170 0x7f525c09c120 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f524c00c7c0 tx=0x7f524c00cc80 comp rx=0 tx=0).stop 2024-01-15T18:15:23.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.769+0000 7f5264a00640 1 -- 172.21.15.27:0/340662970 shutdown_connections 2024-01-15T18:15:23.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.769+0000 7f5264a00640 1 --2- 172.21.15.27:0/340662970 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5234067620 0x7f5234069ae0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.769+0000 7f5264a00640 1 --2- 172.21.15.27:0/340662970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f525c159a30 0x7f525c154970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.769+0000 7f5264a00640 1 --2- 172.21.15.27:0/340662970 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f525c0ac590 0x7f525c202f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.769+0000 7f5264a00640 1 --2- 172.21.15.27:0/340662970 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f525c0ac170 0x7f525c09c120 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.769+0000 7f5264a00640 1 -- 172.21.15.27:0/340662970 >> 172.21.15.27:0/340662970 conn(0x7f525c0b0bb0 msgr2=0x7f525c1573f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:23.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.769+0000 7f5264a00640 1 -- 172.21.15.27:0/340662970 shutdown_connections 2024-01-15T18:15:23.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.769+0000 7f5264a00640 1 -- 172.21.15.27:0/340662970 wait complete. 2024-01-15T18:15:23.794 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone519313 2024-01-15T18:15:23.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.941+0000 7fe599000640 1 Processor -- start 2024-01-15T18:15:23.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.941+0000 7fe599000640 1 -- start start 2024-01-15T18:15:23.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.941+0000 7fe599000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fe59415ad20 0x7fe59415b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:23.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.941+0000 7fe599000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe5940715f0 0x7fe59415ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:23.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.941+0000 7fe599000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe59415a610 con 0x7fe59415d900 2024-01-15T18:15:23.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.941+0000 7fe599000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe59415e3c0 con 0x7fe59415ad20 2024-01-15T18:15:23.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.941+0000 7fe599000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe59415e6a0 con 0x7fe5940715f0 2024-01-15T18:15:23.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.941+0000 7fe592800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe5940715f0 0x7fe59415ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:23.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.941+0000 7fe591e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fe59415ad20 0x7fe59415b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:23.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.941+0000 7fe591400640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7fe59415d900 0x7fe594070ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:36648/0 (socket says 172.21.15.27:36648) 2024-01-15T18:15:23.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.941+0000 7fe591400640 1 -- 172.21.15.27:0/3389412889 learned_addr learned my addr 172.21.15.27:0/3389412889 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:23.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.941+0000 7fe591e00640 1 -- 172.21.15.27:0/3389412889 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe5940715f0 msgr2=0x7fe59415ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:23.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.941+0000 7fe591e00640 1 --2- 172.21.15.27:0/3389412889 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe5940715f0 0x7fe59415ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.941+0000 7fe591e00640 1 -- 172.21.15.27:0/3389412889 >> v1:172.21.15.93:6789/0 conn(0x7fe59415d900 legacy=0x7fe594070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:23.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.941+0000 7fe591e00640 1 -- 172.21.15.27:0/3389412889 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe59415b6e0 con 0x7fe59415ad20 2024-01-15T18:15:23.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.941+0000 7fe590a00640 1 -- 172.21.15.27:0/3389412889 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2108420575 0 0) 0x7fe59415a610 con 0x7fe59415d900 2024-01-15T18:15:23.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.945+0000 7fe591e00640 1 --2- 172.21.15.27:0/3389412889 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe59415ad20 0x7fe59415b100 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe5840097f0 tx=0x7fe58401c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1f6b7d8b168b869 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:23.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.945+0000 7fe590a00640 1 -- 172.21.15.27:0/3389412889 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5840ab670 con 0x7fe59415ad20 2024-01-15T18:15:23.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.945+0000 7fe590a00640 1 -- 172.21.15.27:0/3389412889 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5840ab7d0 con 0x7fe59415ad20 2024-01-15T18:15:23.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.945+0000 7fe590a00640 1 -- 172.21.15.27:0/3389412889 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5840abb60 con 0x7fe59415ad20 2024-01-15T18:15:23.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.945+0000 7fe599000640 1 -- 172.21.15.27:0/3389412889 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe59415ad20 msgr2=0x7fe59415b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:23.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.945+0000 7fe599000640 1 --2- 172.21.15.27:0/3389412889 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe59415ad20 0x7fe59415b100 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe5840097f0 tx=0x7fe58401c450 comp rx=0 tx=0).stop 2024-01-15T18:15:23.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.945+0000 7fe599000640 1 -- 172.21.15.27:0/3389412889 shutdown_connections 2024-01-15T18:15:23.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.945+0000 7fe599000640 1 --2- 172.21.15.27:0/3389412889 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe5940715f0 0x7fe59415ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.945+0000 7fe599000640 1 --2- 172.21.15.27:0/3389412889 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe59415ad20 0x7fe59415b100 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.945+0000 7fe599000640 1 -- 172.21.15.27:0/3389412889 >> 172.21.15.27:0/3389412889 conn(0x7fe5940cb700 msgr2=0x7fe5940cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:23.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.945+0000 7fe599000640 1 -- 172.21.15.27:0/3389412889 shutdown_connections 2024-01-15T18:15:23.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.945+0000 7fe599000640 1 -- 172.21.15.27:0/3389412889 wait complete. 2024-01-15T18:15:23.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.949+0000 7fe599000640 1 Processor -- start 2024-01-15T18:15:23.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.949+0000 7fe599000640 1 -- start start 2024-01-15T18:15:23.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.949+0000 7fe599000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5940715f0 0x7fe594155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:23.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.949+0000 7fe599000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe59415ad20 0x7fe59406c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:23.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.949+0000 7fe599000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe59415d900 0x7fe594152a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:23.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.949+0000 7fe599000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5941f53c0 con 0x7fe5940715f0 2024-01-15T18:15:23.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.949+0000 7fe599000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe594156540 con 0x7fe59415d900 2024-01-15T18:15:23.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.949+0000 7fe599000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe594156820 con 0x7fe59415ad20 2024-01-15T18:15:23.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.949+0000 7fe591e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5940715f0 0x7fe594155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:23.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.949+0000 7fe591e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5940715f0 0x7fe594155cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52332/0 (socket says 172.21.15.27:52332) 2024-01-15T18:15:23.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.949+0000 7fe591e00640 1 -- 172.21.15.27:0/1526353135 learned_addr learned my addr 172.21.15.27:0/1526353135 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:23.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.949+0000 7fe591400640 1 --2- 172.21.15.27:0/1526353135 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe59415ad20 0x7fe59406c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:23.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.949+0000 7fe592800640 1 --2- 172.21.15.27:0/1526353135 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe59415d900 0x7fe594152a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:23.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.949+0000 7fe591e00640 1 -- 172.21.15.27:0/1526353135 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe59415ad20 msgr2=0x7fe59406c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:23.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.949+0000 7fe591e00640 1 --2- 172.21.15.27:0/1526353135 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe59415ad20 0x7fe59406c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.949+0000 7fe591e00640 1 -- 172.21.15.27:0/1526353135 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe59415d900 msgr2=0x7fe594152a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:23.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.949+0000 7fe591e00640 1 --2- 172.21.15.27:0/1526353135 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe59415d900 0x7fe594152a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.949+0000 7fe591e00640 1 -- 172.21.15.27:0/1526353135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe584009420 con 0x7fe5940715f0 2024-01-15T18:15:23.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.949+0000 7fe591e00640 1 --2- 172.21.15.27:0/1526353135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5940715f0 0x7fe594155cc0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fe5840ad040 tx=0x7fe58401ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:23.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.949+0000 7fe58a000640 1 -- 172.21.15.27:0/1526353135 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5840b5810 con 0x7fe5940715f0 2024-01-15T18:15:23.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.949+0000 7fe58a000640 1 -- 172.21.15.27:0/1526353135 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5840b5970 con 0x7fe5940715f0 2024-01-15T18:15:23.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.949+0000 7fe58a000640 1 -- 172.21.15.27:0/1526353135 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5840b5d20 con 0x7fe5940715f0 2024-01-15T18:15:23.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.949+0000 7fe599000640 1 -- 172.21.15.27:0/1526353135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe594156b00 con 0x7fe5940715f0 2024-01-15T18:15:23.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.949+0000 7fe599000640 1 -- 172.21.15.27:0/1526353135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe594156200 con 0x7fe5940715f0 2024-01-15T18:15:23.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.953+0000 7fe599000640 1 -- 172.21.15.27:0/1526353135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe59404e850 con 0x7fe5940715f0 2024-01-15T18:15:23.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.953+0000 7fe58a000640 1 -- 172.21.15.27:0/1526353135 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 29) v1 ==== 87282+0+0 (secure 0 0 0) 0x7fe5840bf410 con 0x7fe5940715f0 2024-01-15T18:15:23.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.953+0000 7fe58a000640 1 --2- 172.21.15.27:0/1526353135 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe56c067550 0x7fe56c069a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:23.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.953+0000 7fe58a000640 1 -- 172.21.15.27:0/1526353135 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fe58411dcf0 con 0x7fe5940715f0 2024-01-15T18:15:23.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.953+0000 7fe591400640 1 --2- 172.21.15.27:0/1526353135 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe56c067550 0x7fe56c069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:23.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.953+0000 7fe591400640 1 --2- 172.21.15.27:0/1526353135 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe56c067550 0x7fe56c069a10 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe59406d080 tx=0x7fe57c007df0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:23.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:23.957+0000 7fe58a000640 1 -- 172.21.15.27:0/1526353135 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe5840ec780 con 0x7fe5940715f0 2024-01-15T18:15:24.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.137+0000 7fe599000640 1 -- 172.21.15.27:0/1526353135 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone519313", "target": ["mon-mgr", ""]}) v1 -- 0x7fe594156960 con 0x7fe56c067550 2024-01-15T18:15:24.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.157+0000 7fe58a000640 1 -- 172.21.15.27:0/1526353135 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe594156960 con 0x7fe56c067550 2024-01-15T18:15:24.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.161+0000 7fe599000640 1 -- 172.21.15.27:0/1526353135 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe56c067550 msgr2=0x7fe56c069a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:24.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.161+0000 7fe599000640 1 --2- 172.21.15.27:0/1526353135 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe56c067550 0x7fe56c069a10 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe59406d080 tx=0x7fe57c007df0 comp rx=0 tx=0).stop 2024-01-15T18:15:24.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.161+0000 7fe599000640 1 -- 172.21.15.27:0/1526353135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5940715f0 msgr2=0x7fe594155cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:24.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.161+0000 7fe599000640 1 --2- 172.21.15.27:0/1526353135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5940715f0 0x7fe594155cc0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fe5840ad040 tx=0x7fe58401ac90 comp rx=0 tx=0).stop 2024-01-15T18:15:24.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.161+0000 7fe599000640 1 -- 172.21.15.27:0/1526353135 shutdown_connections 2024-01-15T18:15:24.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.161+0000 7fe599000640 1 --2- 172.21.15.27:0/1526353135 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe56c067550 0x7fe56c069a10 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:24.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.161+0000 7fe599000640 1 --2- 172.21.15.27:0/1526353135 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe59415d900 0x7fe594152a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:24.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.161+0000 7fe599000640 1 --2- 172.21.15.27:0/1526353135 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe59415ad20 0x7fe59406c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:24.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.161+0000 7fe599000640 1 --2- 172.21.15.27:0/1526353135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5940715f0 0x7fe594155cc0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:24.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.161+0000 7fe599000640 1 -- 172.21.15.27:0/1526353135 >> 172.21.15.27:0/1526353135 conn(0x7fe5940cb700 msgr2=0x7fe5940cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:24.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.161+0000 7fe599000640 1 -- 172.21.15.27:0/1526353135 shutdown_connections 2024-01-15T18:15:24.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.161+0000 7fe599000640 1 -- 172.21.15.27:0/1526353135 wait complete. 2024-01-15T18:15:24.185 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone51938 2024-01-15T18:15:24.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.389+0000 7ff720a00640 1 Processor -- start 2024-01-15T18:15:24.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.389+0000 7ff720a00640 1 -- start start 2024-01-15T18:15:24.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.393+0000 7ff720a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff71815ad20 0x7ff71815b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:24.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.393+0000 7ff720a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff71815d900 0x7ff71815dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:24.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.393+0000 7ff720a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff71815a610 con 0x7ff718070ee0 2024-01-15T18:15:24.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.393+0000 7ff720a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff71815e3c0 con 0x7ff71815d900 2024-01-15T18:15:24.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.393+0000 7ff720a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff71815e6a0 con 0x7ff71815ad20 2024-01-15T18:15:24.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.393+0000 7ff71d600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff71815d900 0x7ff71815dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:24.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.393+0000 7ff71e000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff71815ad20 0x7ff71815b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:24.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.393+0000 7ff71ea00640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7ff718070ee0 0x7ff71806d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:36654/0 (socket says 172.21.15.27:36654) 2024-01-15T18:15:24.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.393+0000 7ff71ea00640 1 -- 172.21.15.27:0/2600545904 learned_addr learned my addr 172.21.15.27:0/2600545904 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:24.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.393+0000 7ff71cc00640 1 -- 172.21.15.27:0/2600545904 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716337858 0 0) 0x7ff71815a610 con 0x7ff718070ee0 2024-01-15T18:15:24.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.393+0000 7ff71cc00640 1 -- 172.21.15.27:0/2600545904 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f8003530 con 0x7ff718070ee0 2024-01-15T18:15:24.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.393+0000 7ff71d600640 1 -- 172.21.15.27:0/2600545904 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff71815ad20 msgr2=0x7ff71815b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:24.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.393+0000 7ff71d600640 1 --2- 172.21.15.27:0/2600545904 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff71815ad20 0x7ff71815b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:24.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.393+0000 7ff71d600640 1 -- 172.21.15.27:0/2600545904 >> v1:172.21.15.93:6789/0 conn(0x7ff718070ee0 legacy=0x7ff71806d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:24.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.393+0000 7ff71d600640 1 -- 172.21.15.27:0/2600545904 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff71815b6e0 con 0x7ff71815d900 2024-01-15T18:15:24.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.393+0000 7ff71e000640 1 --2- 172.21.15.27:0/2600545904 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff71815ad20 0x7ff71815b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:15:24.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.397+0000 7ff71d600640 1 --2- 172.21.15.27:0/2600545904 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff71815d900 0x7ff71815dd80 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ff7000097c0 tx=0x7ff70001c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60bebde69251839c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:24.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.397+0000 7ff71cc00640 1 -- 172.21.15.27:0/2600545904 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7000ab670 con 0x7ff71815d900 2024-01-15T18:15:24.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.397+0000 7ff71cc00640 1 -- 172.21.15.27:0/2600545904 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7000ab7d0 con 0x7ff71815d900 2024-01-15T18:15:24.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.397+0000 7ff71cc00640 1 -- 172.21.15.27:0/2600545904 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7000abb60 con 0x7ff71815d900 2024-01-15T18:15:24.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.397+0000 7ff720a00640 1 -- 172.21.15.27:0/2600545904 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff71815d900 msgr2=0x7ff71815dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:24.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.397+0000 7ff720a00640 1 --2- 172.21.15.27:0/2600545904 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff71815d900 0x7ff71815dd80 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ff7000097c0 tx=0x7ff70001c450 comp rx=0 tx=0).stop 2024-01-15T18:15:24.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.397+0000 7ff720a00640 1 -- 172.21.15.27:0/2600545904 shutdown_connections 2024-01-15T18:15:24.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.397+0000 7ff720a00640 1 --2- 172.21.15.27:0/2600545904 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff71815d900 0x7ff71815dd80 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:24.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.397+0000 7ff720a00640 1 --2- 172.21.15.27:0/2600545904 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff71815ad20 0x7ff71815b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:24.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.397+0000 7ff720a00640 1 -- 172.21.15.27:0/2600545904 >> 172.21.15.27:0/2600545904 conn(0x7ff7180cb700 msgr2=0x7ff7180cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:24.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.397+0000 7ff720a00640 1 -- 172.21.15.27:0/2600545904 shutdown_connections 2024-01-15T18:15:24.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.397+0000 7ff720a00640 1 -- 172.21.15.27:0/2600545904 wait complete. 2024-01-15T18:15:24.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.397+0000 7ff720a00640 1 Processor -- start 2024-01-15T18:15:24.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.397+0000 7ff720a00640 1 -- start start 2024-01-15T18:15:24.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.401+0000 7ff720a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff718070ee0 0x7ff7181465d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:24.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.401+0000 7ff720a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff71815ad20 0x7ff718156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:24.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.401+0000 7ff720a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71815d900 0x7ff718153280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:24.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.401+0000 7ff720a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff718157570 con 0x7ff71815d900 2024-01-15T18:15:24.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.401+0000 7ff720a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7181f96f0 con 0x7ff71815ad20 2024-01-15T18:15:24.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.401+0000 7ff720a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7181f99d0 con 0x7ff718070ee0 2024-01-15T18:15:24.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.401+0000 7ff71ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71815d900 0x7ff718153280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:24.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.401+0000 7ff71ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71815d900 0x7ff718153280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52344/0 (socket says 172.21.15.27:52344) 2024-01-15T18:15:24.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.401+0000 7ff71ea00640 1 -- 172.21.15.27:0/876340793 learned_addr learned my addr 172.21.15.27:0/876340793 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:24.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.401+0000 7ff71e000640 1 --2- 172.21.15.27:0/876340793 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff718070ee0 0x7ff7181465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:24.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.401+0000 7ff71d600640 1 --2- 172.21.15.27:0/876340793 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff71815ad20 0x7ff718156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:24.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.401+0000 7ff71ea00640 1 -- 172.21.15.27:0/876340793 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff718070ee0 msgr2=0x7ff7181465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:24.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.401+0000 7ff71ea00640 1 --2- 172.21.15.27:0/876340793 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff718070ee0 0x7ff7181465d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:24.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.401+0000 7ff71ea00640 1 -- 172.21.15.27:0/876340793 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff71815ad20 msgr2=0x7ff718156ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:24.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.401+0000 7ff71ea00640 1 --2- 172.21.15.27:0/876340793 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff71815ad20 0x7ff718156ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:24.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.401+0000 7ff71ea00640 1 -- 172.21.15.27:0/876340793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff708009420 con 0x7ff71815d900 2024-01-15T18:15:24.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.401+0000 7ff71ea00640 1 --2- 172.21.15.27:0/876340793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71815d900 0x7ff718153280 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7ff71000cb40 tx=0x7ff71000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:24.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.401+0000 7ff716000640 1 -- 172.21.15.27:0/876340793 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff71000aca0 con 0x7ff71815d900 2024-01-15T18:15:24.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.401+0000 7ff716000640 1 -- 172.21.15.27:0/876340793 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff71000ae00 con 0x7ff71815d900 2024-01-15T18:15:24.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.401+0000 7ff716000640 1 -- 172.21.15.27:0/876340793 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff710012620 con 0x7ff71815d900 2024-01-15T18:15:24.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.401+0000 7ff720a00640 1 -- 172.21.15.27:0/876340793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff700009420 con 0x7ff71815d900 2024-01-15T18:15:24.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.401+0000 7ff720a00640 1 -- 172.21.15.27:0/876340793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff718146b10 con 0x7ff71815d900 2024-01-15T18:15:24.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.401+0000 7ff720a00640 1 -- 172.21.15.27:0/876340793 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff71804e850 con 0x7ff71815d900 2024-01-15T18:15:24.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.405+0000 7ff716000640 1 -- 172.21.15.27:0/876340793 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 29) v1 ==== 87282+0+0 (secure 0 0 0) 0x7ff710007070 con 0x7ff71815d900 2024-01-15T18:15:24.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.405+0000 7ff716000640 1 --2- 172.21.15.27:0/876340793 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff6f0067550 0x7ff6f0069a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:24.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.405+0000 7ff716000640 1 -- 172.21.15.27:0/876340793 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7ff7100830a0 con 0x7ff71815d900 2024-01-15T18:15:24.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.405+0000 7ff71e000640 1 --2- 172.21.15.27:0/876340793 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff6f0067550 0x7ff6f0069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:24.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.405+0000 7ff71e000640 1 --2- 172.21.15.27:0/876340793 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff6f0067550 0x7ff6f0069a10 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ff708009cf0 tx=0x7ff708009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:24.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.409+0000 7ff716000640 1 -- 172.21.15.27:0/876340793 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff710051b60 con 0x7ff71815d900 2024-01-15T18:15:24.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.585+0000 7ff720a00640 1 -- 172.21.15.27:0/876340793 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone51938", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7181f9860 con 0x7ff6f0067550 2024-01-15T18:15:24.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.605+0000 7ff716000640 1 -- 172.21.15.27:0/876340793 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff7181f9860 con 0x7ff6f0067550 2024-01-15T18:15:24.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.609+0000 7ff720a00640 1 -- 172.21.15.27:0/876340793 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff6f0067550 msgr2=0x7ff6f0069a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:24.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.609+0000 7ff720a00640 1 --2- 172.21.15.27:0/876340793 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff6f0067550 0x7ff6f0069a10 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7ff708009cf0 tx=0x7ff708009190 comp rx=0 tx=0).stop 2024-01-15T18:15:24.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.609+0000 7ff720a00640 1 -- 172.21.15.27:0/876340793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71815d900 msgr2=0x7ff718153280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:24.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.609+0000 7ff720a00640 1 --2- 172.21.15.27:0/876340793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71815d900 0x7ff718153280 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7ff71000cb40 tx=0x7ff71000a060 comp rx=0 tx=0).stop 2024-01-15T18:15:24.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.609+0000 7ff720a00640 1 -- 172.21.15.27:0/876340793 shutdown_connections 2024-01-15T18:15:24.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.609+0000 7ff720a00640 1 --2- 172.21.15.27:0/876340793 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff6f0067550 0x7ff6f0069a10 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:24.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.609+0000 7ff720a00640 1 --2- 172.21.15.27:0/876340793 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff71815d900 0x7ff718153280 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:24.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.609+0000 7ff720a00640 1 --2- 172.21.15.27:0/876340793 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff71815ad20 0x7ff718156ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:24.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.609+0000 7ff720a00640 1 --2- 172.21.15.27:0/876340793 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff718070ee0 0x7ff7181465d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:24.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.609+0000 7ff720a00640 1 -- 172.21.15.27:0/876340793 >> 172.21.15.27:0/876340793 conn(0x7ff7180cb700 msgr2=0x7ff71815a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:24.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.609+0000 7ff720a00640 1 -- 172.21.15.27:0/876340793 shutdown_connections 2024-01-15T18:15:24.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.609+0000 7ff720a00640 1 -- 172.21.15.27:0/876340793 wait complete. 2024-01-15T18:15:24.632 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone519320 2024-01-15T18:15:24.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.777+0000 7fbe3fa00640 1 Processor -- start 2024-01-15T18:15:24.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.777+0000 7fbe3fa00640 1 -- start start 2024-01-15T18:15:24.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.777+0000 7fbe3fa00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fbe38158900 0x7fbe3815f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:24.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.777+0000 7fbe3fa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbe380cf2e0 0x7fbe380701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:24.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.777+0000 7fbe3fa00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe380b1cb0 con 0x7fbe3815bf00 2024-01-15T18:15:24.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.777+0000 7fbe3fa00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe380b1f00 con 0x7fbe380cf2e0 2024-01-15T18:15:24.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.777+0000 7fbe3fa00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe380b21e0 con 0x7fbe38158900 2024-01-15T18:15:24.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.777+0000 7fbe3da00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbe380cf2e0 0x7fbe380701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:24.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.777+0000 7fbe3da00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbe380cf2e0 0x7fbe380701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52346/0 (socket says 172.21.15.27:52346) 2024-01-15T18:15:24.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.777+0000 7fbe3da00640 1 -- 172.21.15.27:0/3409313377 learned_addr learned my addr 172.21.15.27:0/3409313377 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:24.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.777+0000 7fbe3d000640 1 --2- 172.21.15.27:0/3409313377 >> v2:172.21.15.93:3300/0 conn(0x7fbe38158900 0x7fbe3815f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:24.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.777+0000 7fbe3da00640 1 -- 172.21.15.27:0/3409313377 >> v2:172.21.15.93:3300/0 conn(0x7fbe38158900 msgr2=0x7fbe3815f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:24.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.777+0000 7fbe3da00640 1 --2- 172.21.15.27:0/3409313377 >> v2:172.21.15.93:3300/0 conn(0x7fbe38158900 0x7fbe3815f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.777+0000 7fbe3da00640 1 -- 172.21.15.27:0/3409313377 >> v1:172.21.15.93:6789/0 conn(0x7fbe3815bf00 legacy=0x7fbe380cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.777+0000 7fbe3da00640 1 -- 172.21.15.27:0/3409313377 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe3815ea70 con 0x7fbe380cf2e0 2024-01-15T18:15:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.777+0000 7fbe3d000640 1 --2- 172.21.15.27:0/3409313377 >> v2:172.21.15.93:3300/0 conn(0x7fbe38158900 0x7fbe3815f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:24.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.781+0000 7fbe3da00640 1 --2- 172.21.15.27:0/3409313377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe380cf2e0 0x7fbe380701e0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fbe380ceaa0 tx=0x7fbe3001c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20853d7214f5bd9c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:24.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.781+0000 7fbe37400640 1 -- 172.21.15.27:0/3409313377 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe38070f30 con 0x7fbe380cf2e0 2024-01-15T18:15:24.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.781+0000 7fbe37400640 1 -- 172.21.15.27:0/3409313377 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe300a5740 con 0x7fbe380cf2e0 2024-01-15T18:15:24.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.781+0000 7fbe37400640 1 -- 172.21.15.27:0/3409313377 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe300a5ad0 con 0x7fbe380cf2e0 2024-01-15T18:15:24.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.781+0000 7fbe3fa00640 1 -- 172.21.15.27:0/3409313377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe380cf2e0 msgr2=0x7fbe380701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:24.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.781+0000 7fbe3fa00640 1 --2- 172.21.15.27:0/3409313377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe380cf2e0 0x7fbe380701e0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fbe380ceaa0 tx=0x7fbe3001c990 comp rx=0 tx=0).stop 2024-01-15T18:15:24.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.781+0000 7fbe3fa00640 1 -- 172.21.15.27:0/3409313377 shutdown_connections 2024-01-15T18:15:24.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.781+0000 7fbe3fa00640 1 --2- 172.21.15.27:0/3409313377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe380cf2e0 0x7fbe380701e0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:24.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.781+0000 7fbe3fa00640 1 --2- 172.21.15.27:0/3409313377 >> v2:172.21.15.93:3300/0 conn(0x7fbe38158900 0x7fbe3815f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:24.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.781+0000 7fbe3fa00640 1 -- 172.21.15.27:0/3409313377 >> 172.21.15.27:0/3409313377 conn(0x7fbe380cb700 msgr2=0x7fbe380cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:24.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.781+0000 7fbe3fa00640 1 -- 172.21.15.27:0/3409313377 shutdown_connections 2024-01-15T18:15:24.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.781+0000 7fbe3fa00640 1 -- 172.21.15.27:0/3409313377 wait complete. 2024-01-15T18:15:24.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.785+0000 7fbe3fa00640 1 Processor -- start 2024-01-15T18:15:24.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.785+0000 7fbe3fa00640 1 -- start start 2024-01-15T18:15:24.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.785+0000 7fbe3fa00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbe380cf2e0 0x7fbe380b0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:24.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.785+0000 7fbe3fa00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbe38158900 0x7fbe3814c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:24.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.785+0000 7fbe3fa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe3815bf00 0x7fbe380cd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:24.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.785+0000 7fbe3fa00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe380b0650 con 0x7fbe3815bf00 2024-01-15T18:15:24.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.785+0000 7fbe3fa00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe381593f0 con 0x7fbe380cf2e0 2024-01-15T18:15:24.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.785+0000 7fbe3fa00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3814cf10 con 0x7fbe38158900 2024-01-15T18:15:24.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.785+0000 7fbe3da00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe3815bf00 0x7fbe380cd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:24.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.785+0000 7fbe3da00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe3815bf00 0x7fbe380cd9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52358/0 (socket says 172.21.15.27:52358) 2024-01-15T18:15:24.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.785+0000 7fbe3da00640 1 -- 172.21.15.27:0/3060391454 learned_addr learned my addr 172.21.15.27:0/3060391454 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:24.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.785+0000 7fbe3d000640 1 --2- 172.21.15.27:0/3060391454 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbe380cf2e0 0x7fbe380b0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:24.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.785+0000 7fbe37e00640 1 --2- 172.21.15.27:0/3060391454 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbe38158900 0x7fbe3814c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:24.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.785+0000 7fbe3da00640 1 -- 172.21.15.27:0/3060391454 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbe38158900 msgr2=0x7fbe3814c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:24.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.785+0000 7fbe3da00640 1 --2- 172.21.15.27:0/3060391454 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbe38158900 0x7fbe3814c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:24.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.785+0000 7fbe3da00640 1 -- 172.21.15.27:0/3060391454 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbe380cf2e0 msgr2=0x7fbe380b0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:24.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.785+0000 7fbe3da00640 1 --2- 172.21.15.27:0/3060391454 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbe380cf2e0 0x7fbe380b0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:24.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.785+0000 7fbe3da00640 1 -- 172.21.15.27:0/3060391454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe30009b30 con 0x7fbe3815bf00 2024-01-15T18:15:24.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.785+0000 7fbe3da00640 1 --2- 172.21.15.27:0/3060391454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe3815bf00 0x7fbe380cd9f0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fbe3001b040 tx=0x7fbe30019730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.785+0000 7fbe34c00640 1 -- 172.21.15.27:0/3060391454 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe300b8670 con 0x7fbe3815bf00 2024-01-15T18:15:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.785+0000 7fbe34c00640 1 -- 172.21.15.27:0/3060391454 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe300b87d0 con 0x7fbe3815bf00 2024-01-15T18:15:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.785+0000 7fbe3fa00640 1 -- 172.21.15.27:0/3060391454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe380cdf30 con 0x7fbe3815bf00 2024-01-15T18:15:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.785+0000 7fbe34c00640 1 -- 172.21.15.27:0/3060391454 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe300b8b80 con 0x7fbe3815bf00 2024-01-15T18:15:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.785+0000 7fbe3fa00640 1 -- 172.21.15.27:0/3060391454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe3814d440 con 0x7fbe3815bf00 2024-01-15T18:15:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.785+0000 7fbe3fa00640 1 -- 172.21.15.27:0/3060391454 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe381595a0 con 0x7fbe3815bf00 2024-01-15T18:15:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.789+0000 7fbe34c00640 1 -- 172.21.15.27:0/3060391454 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 29) v1 ==== 87282+0+0 (secure 0 0 0) 0x7fbe300b8ce0 con 0x7fbe3815bf00 2024-01-15T18:15:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.789+0000 7fbe34c00640 1 --2- 172.21.15.27:0/3060391454 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbe10067550 0x7fbe10069a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.789+0000 7fbe34c00640 1 -- 172.21.15.27:0/3060391454 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fbe3011e320 con 0x7fbe3815bf00 2024-01-15T18:15:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.789+0000 7fbe3d000640 1 --2- 172.21.15.27:0/3060391454 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbe10067550 0x7fbe10069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:24.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.789+0000 7fbe3d000640 1 --2- 172.21.15.27:0/3060391454 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbe10067550 0x7fbe10069a10 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fbe2800ab60 tx=0x7fbe28008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:24.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.793+0000 7fbe34c00640 1 -- 172.21.15.27:0/3060391454 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbe381595a0 con 0x7fbe3815bf00 2024-01-15T18:15:24.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.977+0000 7fbe3fa00640 1 -- 172.21.15.27:0/3060391454 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone519320", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe380735d0 con 0x7fbe10067550 2024-01-15T18:15:24.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.993+0000 7fbe34c00640 1 -- 172.21.15.27:0/3060391454 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbe380735d0 con 0x7fbe10067550 2024-01-15T18:15:24.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.997+0000 7fbe3fa00640 1 -- 172.21.15.27:0/3060391454 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbe10067550 msgr2=0x7fbe10069a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.997+0000 7fbe3fa00640 1 --2- 172.21.15.27:0/3060391454 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbe10067550 0x7fbe10069a10 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fbe2800ab60 tx=0x7fbe28008040 comp rx=0 tx=0).stop 2024-01-15T18:15:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.997+0000 7fbe3fa00640 1 -- 172.21.15.27:0/3060391454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe3815bf00 msgr2=0x7fbe380cd9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.997+0000 7fbe3fa00640 1 --2- 172.21.15.27:0/3060391454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe3815bf00 0x7fbe380cd9f0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fbe3001b040 tx=0x7fbe30019730 comp rx=0 tx=0).stop 2024-01-15T18:15:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.997+0000 7fbe3fa00640 1 -- 172.21.15.27:0/3060391454 shutdown_connections 2024-01-15T18:15:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.997+0000 7fbe3fa00640 1 --2- 172.21.15.27:0/3060391454 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbe10067550 0x7fbe10069a10 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.997+0000 7fbe3fa00640 1 --2- 172.21.15.27:0/3060391454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe3815bf00 0x7fbe380cd9f0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.997+0000 7fbe3fa00640 1 --2- 172.21.15.27:0/3060391454 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbe38158900 0x7fbe3814c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.997+0000 7fbe3fa00640 1 --2- 172.21.15.27:0/3060391454 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbe380cf2e0 0x7fbe380b0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.997+0000 7fbe3fa00640 1 -- 172.21.15.27:0/3060391454 >> 172.21.15.27:0/3060391454 conn(0x7fbe380cb700 msgr2=0x7fbe380cebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.997+0000 7fbe3fa00640 1 -- 172.21.15.27:0/3060391454 shutdown_connections 2024-01-15T18:15:24.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:24.997+0000 7fbe3fa00640 1 -- 172.21.15.27:0/3060391454 wait complete. 2024-01-15T18:15:25.021 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone51933 2024-01-15T18:15:25.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.165+0000 7f94e1200640 1 Processor -- start 2024-01-15T18:15:25.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.165+0000 7f94e1200640 1 -- start start 2024-01-15T18:15:25.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.165+0000 7f94e1200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f94dc0cf2e0 0x7f94dc0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:25.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.165+0000 7f94e1200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94dc0b1cb0 con 0x7f94dc15f550 2024-01-15T18:15:25.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.165+0000 7f94e1200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94dc15fa00 con 0x7f94dc158900 2024-01-15T18:15:25.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.165+0000 7f94e1200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94dc0b1f00 con 0x7f94dc0cf2e0 2024-01-15T18:15:25.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.165+0000 7f94d9400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f94dc15f550 0x7f94dc0cebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49152/0 (socket says 172.21.15.27:49152) 2024-01-15T18:15:25.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.165+0000 7f94d9400640 1 -- 172.21.15.27:0/644491602 learned_addr learned my addr 172.21.15.27:0/644491602 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:25.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.165+0000 7f94da800640 1 --2- 172.21.15.27:0/644491602 >> v2:172.21.15.27:3300/0 conn(0x7f94dc0cf2e0 0x7f94dc0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:25.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.165+0000 7f94d8a00640 1 -- 172.21.15.27:0/644491602 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 547487690 0 0) 0x7f94dc0b1cb0 con 0x7f94dc15f550 2024-01-15T18:15:25.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.165+0000 7f94d8a00640 1 -- 172.21.15.27:0/644491602 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94bc003530 con 0x7f94dc15f550 2024-01-15T18:15:25.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.165+0000 7f94da800640 1 -- 172.21.15.27:0/644491602 >> v1:172.21.15.93:6789/0 conn(0x7f94dc158900 legacy=0x7f94dc15bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:25.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.165+0000 7f94d8a00640 1 -- 172.21.15.27:0/644491602 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 103569122 0 0) 0x7f94dc15fa00 con 0x7f94dc158900 2024-01-15T18:15:25.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.165+0000 7f94da800640 1 -- 172.21.15.27:0/644491602 >> v1:172.21.15.27:6789/0 conn(0x7f94dc15f550 legacy=0x7f94dc0cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:25.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.165+0000 7f94da800640 1 -- 172.21.15.27:0/644491602 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94dc0b21e0 con 0x7f94dc0cf2e0 2024-01-15T18:15:25.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.169+0000 7f94da800640 1 --2- 172.21.15.27:0/644491602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94dc0cf2e0 0x7f94dc0701e0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f94d4009d90 tx=0x7f94d401c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d635572b371a093 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:25.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.169+0000 7f94d8a00640 1 -- 172.21.15.27:0/644491602 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc070f30 con 0x7f94dc0cf2e0 2024-01-15T18:15:25.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.169+0000 7f94d8a00640 1 -- 172.21.15.27:0/644491602 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94d40a59e0 con 0x7f94dc0cf2e0 2024-01-15T18:15:25.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.169+0000 7f94d8a00640 1 -- 172.21.15.27:0/644491602 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d40a5d70 con 0x7f94dc0cf2e0 2024-01-15T18:15:25.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.169+0000 7f94e1200640 1 -- 172.21.15.27:0/644491602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94dc0cf2e0 msgr2=0x7f94dc0701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:25.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.169+0000 7f94e1200640 1 --2- 172.21.15.27:0/644491602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94dc0cf2e0 0x7f94dc0701e0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f94d4009d90 tx=0x7f94d401c920 comp rx=0 tx=0).stop 2024-01-15T18:15:25.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.169+0000 7f94e1200640 1 -- 172.21.15.27:0/644491602 shutdown_connections 2024-01-15T18:15:25.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.169+0000 7f94e1200640 1 --2- 172.21.15.27:0/644491602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94dc0cf2e0 0x7f94dc0701e0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:25.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.169+0000 7f94e1200640 1 -- 172.21.15.27:0/644491602 >> 172.21.15.27:0/644491602 conn(0x7f94dc0cb700 msgr2=0x7f94dc0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:25.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.169+0000 7f94e1200640 1 -- 172.21.15.27:0/644491602 shutdown_connections 2024-01-15T18:15:25.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.169+0000 7f94e1200640 1 -- 172.21.15.27:0/644491602 wait complete. 2024-01-15T18:15:25.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94e1200640 1 Processor -- start 2024-01-15T18:15:25.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94e1200640 1 -- start start 2024-01-15T18:15:25.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94e1200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94dc0cf2e0 0x7f94dc0ce1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:25.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94e1200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f94dc158900 0x7f94dc0b3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:25.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94e1200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f94dc15f550 0x7f94dc14ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:25.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94e1200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94dc14fa90 con 0x7f94dc0cf2e0 2024-01-15T18:15:25.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94e1200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94dc15c0b0 con 0x7f94dc158900 2024-01-15T18:15:25.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94e1200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94dc15c390 con 0x7f94dc15f550 2024-01-15T18:15:25.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94d9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94dc0cf2e0 0x7f94dc0ce1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:25.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94d9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94dc0cf2e0 0x7f94dc0ce1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52380/0 (socket says 172.21.15.27:52380) 2024-01-15T18:15:25.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94d9e00640 1 -- 172.21.15.27:0/2153343299 learned_addr learned my addr 172.21.15.27:0/2153343299 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:25.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94d9400640 1 --2- 172.21.15.27:0/2153343299 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f94dc158900 0x7f94dc0b3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:25.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94da800640 1 --2- 172.21.15.27:0/2153343299 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f94dc15f550 0x7f94dc14ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:25.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94d9e00640 1 -- 172.21.15.27:0/2153343299 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f94dc15f550 msgr2=0x7f94dc14ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:25.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94d9e00640 1 --2- 172.21.15.27:0/2153343299 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f94dc15f550 0x7f94dc14ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:25.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94d9e00640 1 -- 172.21.15.27:0/2153343299 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f94dc158900 msgr2=0x7f94dc0b3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:25.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94d9e00640 1 --2- 172.21.15.27:0/2153343299 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f94dc158900 0x7f94dc0b3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:25.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94d9e00640 1 -- 172.21.15.27:0/2153343299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d4009a70 con 0x7f94dc0cf2e0 2024-01-15T18:15:25.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94d9400640 1 --2- 172.21.15.27:0/2153343299 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f94dc158900 0x7f94dc0b3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:25.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94da800640 1 --2- 172.21.15.27:0/2153343299 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f94dc15f550 0x7f94dc14ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:25.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94d9e00640 1 --2- 172.21.15.27:0/2153343299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94dc0cf2e0 0x7f94dc0ce1f0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f94cc00c4d0 tx=0x7f94cc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:25.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94d2000640 1 -- 172.21.15.27:0/2153343299 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94cc005480 con 0x7f94dc0cf2e0 2024-01-15T18:15:25.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94d2000640 1 -- 172.21.15.27:0/2153343299 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94cc003600 con 0x7f94dc0cf2e0 2024-01-15T18:15:25.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94e1200640 1 -- 172.21.15.27:0/2153343299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94dc0aca20 con 0x7f94dc0cf2e0 2024-01-15T18:15:25.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94d2000640 1 -- 172.21.15.27:0/2153343299 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94cc0039b0 con 0x7f94dc0cf2e0 2024-01-15T18:15:25.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94e1200640 1 -- 172.21.15.27:0/2153343299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94dc0acfe0 con 0x7f94dc0cf2e0 2024-01-15T18:15:25.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.173+0000 7f94e1200640 1 -- 172.21.15.27:0/2153343299 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94dc0ac880 con 0x7f94dc0cf2e0 2024-01-15T18:15:25.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.177+0000 7f94d2000640 1 -- 172.21.15.27:0/2153343299 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 29) v1 ==== 87282+0+0 (secure 0 0 0) 0x7f94cc003b10 con 0x7f94dc0cf2e0 2024-01-15T18:15:25.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.177+0000 7f94d2000640 1 --2- 172.21.15.27:0/2153343299 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f94b4067580 0x7f94b4069a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:25.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.177+0000 7f94d2000640 1 -- 172.21.15.27:0/2153343299 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f94cc082650 con 0x7f94dc0cf2e0 2024-01-15T18:15:25.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.177+0000 7f94d9400640 1 --2- 172.21.15.27:0/2153343299 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f94b4067580 0x7f94b4069a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:25.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.177+0000 7f94d9400640 1 --2- 172.21.15.27:0/2153343299 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f94b4067580 0x7f94b4069a40 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f94dc0b4430 tx=0x7f94c4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:25.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.181+0000 7f94d2000640 1 -- 172.21.15.27:0/2153343299 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f94dc0ac880 con 0x7f94dc0cf2e0 2024-01-15T18:15:25.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.521+0000 7f94e1200640 1 -- 172.21.15.27:0/2153343299 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone51933", "target": ["mon-mgr", ""]}) v1 -- 0x7f94dc0cf990 con 0x7f94b4067580 2024-01-15T18:15:25.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.553+0000 7f94d2000640 1 -- 172.21.15.27:0/2153343299 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f94dc0cf990 con 0x7f94b4067580 2024-01-15T18:15:25.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.557+0000 7f94e1200640 1 -- 172.21.15.27:0/2153343299 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f94b4067580 msgr2=0x7f94b4069a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:25.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.557+0000 7f94e1200640 1 --2- 172.21.15.27:0/2153343299 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f94b4067580 0x7f94b4069a40 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f94dc0b4430 tx=0x7f94c4009040 comp rx=0 tx=0).stop 2024-01-15T18:15:25.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.557+0000 7f94e1200640 1 -- 172.21.15.27:0/2153343299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94dc0cf2e0 msgr2=0x7f94dc0ce1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:25.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.557+0000 7f94e1200640 1 --2- 172.21.15.27:0/2153343299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94dc0cf2e0 0x7f94dc0ce1f0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f94cc00c4d0 tx=0x7f94cc00c990 comp rx=0 tx=0).stop 2024-01-15T18:15:25.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.557+0000 7f94e1200640 1 -- 172.21.15.27:0/2153343299 shutdown_connections 2024-01-15T18:15:25.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.557+0000 7f94e1200640 1 --2- 172.21.15.27:0/2153343299 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f94b4067580 0x7f94b4069a40 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:25.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.557+0000 7f94e1200640 1 --2- 172.21.15.27:0/2153343299 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f94dc15f550 0x7f94dc14ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:25.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.557+0000 7f94e1200640 1 --2- 172.21.15.27:0/2153343299 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f94dc158900 0x7f94dc0b3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:25.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.557+0000 7f94e1200640 1 --2- 172.21.15.27:0/2153343299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94dc0cf2e0 0x7f94dc0ce1f0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:25.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.557+0000 7f94e1200640 1 -- 172.21.15.27:0/2153343299 >> 172.21.15.27:0/2153343299 conn(0x7f94dc0cb700 msgr2=0x7f94dc157cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:25.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.557+0000 7f94e1200640 1 -- 172.21.15.27:0/2153343299 shutdown_connections 2024-01-15T18:15:25.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.557+0000 7f94e1200640 1 -- 172.21.15.27:0/2153343299 wait complete. 2024-01-15T18:15:25.581 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone51934 2024-01-15T18:15:25.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.725+0000 7fafef200640 1 Processor -- start 2024-01-15T18:15:25.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.725+0000 7fafef200640 1 -- start start 2024-01-15T18:15:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.725+0000 7fafef200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fafe815f550 0x7fafe80cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:25.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.725+0000 7fafef200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fafe80cf110 0x7fafe80701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:25.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.725+0000 7fafef200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe80b1cb0 con 0x7fafe8158900 2024-01-15T18:15:25.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.725+0000 7fafef200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafe80cf580 con 0x7fafe815f550 2024-01-15T18:15:25.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.725+0000 7fafef200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe815fa00 con 0x7fafe80cf110 2024-01-15T18:15:25.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.725+0000 7fafe7400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fafe815f550 0x7fafe80cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:25.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.725+0000 7fafe7e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fafe8158900 0x7fafe815bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49158/0 (socket says 172.21.15.27:49158) 2024-01-15T18:15:25.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.725+0000 7fafe7e00640 1 -- 172.21.15.27:0/2641012987 learned_addr learned my addr 172.21.15.27:0/2641012987 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:25.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.725+0000 7fafed200640 1 --2- 172.21.15.27:0/2641012987 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fafe80cf110 0x7fafe80701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:25.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.729+0000 7fafe7400640 1 -- 172.21.15.27:0/2641012987 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fafe80cf110 msgr2=0x7fafe80701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:25.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.729+0000 7fafe7400640 1 --2- 172.21.15.27:0/2641012987 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fafe80cf110 0x7fafe80701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:25.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.729+0000 7fafe7400640 1 -- 172.21.15.27:0/2641012987 >> v1:172.21.15.27:6789/0 conn(0x7fafe8158900 legacy=0x7fafe815bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:25.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.729+0000 7fafe7400640 1 -- 172.21.15.27:0/2641012987 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe80b1f00 con 0x7fafe815f550 2024-01-15T18:15:25.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.729+0000 7fafe7400640 1 --2- 172.21.15.27:0/2641012987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafe815f550 0x7fafe80cebd0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fafd00097c0 tx=0x7fafd001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2131cee415832e3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:25.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.729+0000 7fafe6a00640 1 -- 172.21.15.27:0/2641012987 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe8070f30 con 0x7fafe815f550 2024-01-15T18:15:25.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.729+0000 7fafe6a00640 1 -- 172.21.15.27:0/2641012987 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd00ab650 con 0x7fafe815f550 2024-01-15T18:15:25.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.729+0000 7fafe6a00640 1 -- 172.21.15.27:0/2641012987 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd00ab9e0 con 0x7fafe815f550 2024-01-15T18:15:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.733+0000 7fafef200640 1 -- 172.21.15.27:0/2641012987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafe815f550 msgr2=0x7fafe80cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.733+0000 7fafef200640 1 --2- 172.21.15.27:0/2641012987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafe815f550 0x7fafe80cebd0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fafd00097c0 tx=0x7fafd001c450 comp rx=0 tx=0).stop 2024-01-15T18:15:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.733+0000 7fafef200640 1 -- 172.21.15.27:0/2641012987 shutdown_connections 2024-01-15T18:15:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.733+0000 7fafef200640 1 --2- 172.21.15.27:0/2641012987 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fafe80cf110 0x7fafe80701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.733+0000 7fafef200640 1 --2- 172.21.15.27:0/2641012987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafe815f550 0x7fafe80cebd0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.733+0000 7fafef200640 1 -- 172.21.15.27:0/2641012987 >> 172.21.15.27:0/2641012987 conn(0x7fafe80cb700 msgr2=0x7fafe80cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.733+0000 7fafef200640 1 -- 172.21.15.27:0/2641012987 shutdown_connections 2024-01-15T18:15:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.733+0000 7fafef200640 1 -- 172.21.15.27:0/2641012987 wait complete. 2024-01-15T18:15:25.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.733+0000 7fafef200640 1 Processor -- start 2024-01-15T18:15:25.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.733+0000 7fafef200640 1 -- start start 2024-01-15T18:15:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.733+0000 7fafef200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fafe80cf110 0x7fafe8152620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.733+0000 7fafef200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fafe8158900 0x7fafe8151a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.733+0000 7fafef200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafe815f550 0x7fafe8151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.733+0000 7fafef200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe80ccc50 con 0x7fafe815f550 2024-01-15T18:15:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.733+0000 7fafef200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe80ccf30 con 0x7fafe8158900 2024-01-15T18:15:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.733+0000 7fafef200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe80cd210 con 0x7fafe80cf110 2024-01-15T18:15:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.733+0000 7fafed200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafe815f550 0x7fafe8151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.733+0000 7fafed200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafe815f550 0x7fafe8151f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52406/0 (socket says 172.21.15.27:52406) 2024-01-15T18:15:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.733+0000 7fafed200640 1 -- 172.21.15.27:0/3425042846 learned_addr learned my addr 172.21.15.27:0/3425042846 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.733+0000 7fafe7e00640 1 --2- 172.21.15.27:0/3425042846 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fafe80cf110 0x7fafe8152620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.733+0000 7fafe7400640 1 --2- 172.21.15.27:0/3425042846 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fafe8158900 0x7fafe8151a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.737+0000 7fafed200640 1 -- 172.21.15.27:0/3425042846 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fafe80cf110 msgr2=0x7fafe8152620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.737+0000 7fafed200640 1 --2- 172.21.15.27:0/3425042846 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fafe80cf110 0x7fafe8152620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.737+0000 7fafed200640 1 -- 172.21.15.27:0/3425042846 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fafe8158900 msgr2=0x7fafe8151a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.737+0000 7fafed200640 1 --2- 172.21.15.27:0/3425042846 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fafe8158900 0x7fafe8151a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.737+0000 7fafed200640 1 -- 172.21.15.27:0/3425042846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd0009420 con 0x7fafe815f550 2024-01-15T18:15:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.737+0000 7fafed200640 1 --2- 172.21.15.27:0/3425042846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafe815f550 0x7fafe8151f40 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fafe0007e40 tx=0x7fafe000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.737+0000 7fafdfe00640 1 -- 172.21.15.27:0/3425042846 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe0014620 con 0x7fafe815f550 2024-01-15T18:15:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.737+0000 7fafdfe00640 1 -- 172.21.15.27:0/3425042846 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe000ee90 con 0x7fafe815f550 2024-01-15T18:15:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.737+0000 7fafef200640 1 -- 172.21.15.27:0/3425042846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe80cd4f0 con 0x7fafe815f550 2024-01-15T18:15:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.737+0000 7fafdfe00640 1 -- 172.21.15.27:0/3425042846 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe00149e0 con 0x7fafe815f550 2024-01-15T18:15:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.737+0000 7fafef200640 1 -- 172.21.15.27:0/3425042846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe8152b60 con 0x7fafe815f550 2024-01-15T18:15:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.737+0000 7fafef200640 1 -- 172.21.15.27:0/3425042846 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafe8070570 con 0x7fafe815f550 2024-01-15T18:15:25.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.737+0000 7fafdfe00640 1 -- 172.21.15.27:0/3425042846 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 29) v1 ==== 87282+0+0 (secure 0 0 0) 0x7fafe0014b40 con 0x7fafe815f550 2024-01-15T18:15:25.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.741+0000 7fafdfe00640 1 --2- 172.21.15.27:0/3425042846 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fafc0067550 0x7fafc0069a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:25.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.741+0000 7fafdfe00640 1 -- 172.21.15.27:0/3425042846 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7fafe00836c0 con 0x7fafe815f550 2024-01-15T18:15:25.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.741+0000 7fafe7e00640 1 --2- 172.21.15.27:0/3425042846 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fafc0067550 0x7fafc0069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:25.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.741+0000 7fafe7e00640 1 --2- 172.21.15.27:0/3425042846 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fafc0067550 0x7fafc0069a10 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fafd8005890 tx=0x7fafd8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:25.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.741+0000 7fafdfe00640 1 -- 172.21.15.27:0/3425042846 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fafe0052400 con 0x7fafe815f550 2024-01-15T18:15:25.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.925+0000 7fafef200640 1 -- 172.21.15.27:0/3425042846 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone51934", "target": ["mon-mgr", ""]}) v1 -- 0x7fafe80cf990 con 0x7fafc0067550 2024-01-15T18:15:25.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.941+0000 7fafdfe00640 1 -- 172.21.15.27:0/3425042846 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fafe80cf990 con 0x7fafc0067550 2024-01-15T18:15:25.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.945+0000 7fafef200640 1 -- 172.21.15.27:0/3425042846 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fafc0067550 msgr2=0x7fafc0069a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:25.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.945+0000 7fafef200640 1 --2- 172.21.15.27:0/3425042846 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fafc0067550 0x7fafc0069a10 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fafd8005890 tx=0x7fafd8009040 comp rx=0 tx=0).stop 2024-01-15T18:15:25.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.945+0000 7fafef200640 1 -- 172.21.15.27:0/3425042846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafe815f550 msgr2=0x7fafe8151f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:25.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.945+0000 7fafef200640 1 --2- 172.21.15.27:0/3425042846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafe815f550 0x7fafe8151f40 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fafe0007e40 tx=0x7fafe000e4b0 comp rx=0 tx=0).stop 2024-01-15T18:15:25.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.945+0000 7fafef200640 1 -- 172.21.15.27:0/3425042846 shutdown_connections 2024-01-15T18:15:25.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.945+0000 7fafef200640 1 --2- 172.21.15.27:0/3425042846 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fafc0067550 0x7fafc0069a10 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:25.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.945+0000 7fafef200640 1 --2- 172.21.15.27:0/3425042846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafe815f550 0x7fafe8151f40 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:25.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.945+0000 7fafef200640 1 --2- 172.21.15.27:0/3425042846 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fafe8158900 0x7fafe8151a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:25.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.945+0000 7fafef200640 1 --2- 172.21.15.27:0/3425042846 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fafe80cf110 0x7fafe8152620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:25.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.945+0000 7fafef200640 1 -- 172.21.15.27:0/3425042846 >> 172.21.15.27:0/3425042846 conn(0x7fafe80cb700 msgr2=0x7fafe8155030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:25.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.945+0000 7fafef200640 1 -- 172.21.15.27:0/3425042846 shutdown_connections 2024-01-15T18:15:25.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:25.945+0000 7fafef200640 1 -- 172.21.15.27:0/3425042846 wait complete. 2024-01-15T18:15:25.967 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone519318 2024-01-15T18:15:26.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.117+0000 7f2725e00640 1 Processor -- start 2024-01-15T18:15:26.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.117+0000 7f2725e00640 1 -- start start 2024-01-15T18:15:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.117+0000 7f2725e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f27201468d0 0x7f27200ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.117+0000 7f2725e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27200cd070 0x7f272006c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.117+0000 7f2725e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27200ad920 con 0x7f27200cef30 2024-01-15T18:15:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.117+0000 7f2725e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27200cd4e0 con 0x7f27201468d0 2024-01-15T18:15:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.117+0000 7f2725e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27200ac880 con 0x7f27200cd070 2024-01-15T18:15:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.117+0000 7f271ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f27201468d0 0x7f27200ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.117+0000 7f271ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f27201468d0 0x7f27200ce9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52410/0 (socket says 172.21.15.27:52410) 2024-01-15T18:15:26.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.117+0000 7f271ea00640 1 -- 172.21.15.27:0/1673067480 learned_addr learned my addr 172.21.15.27:0/1673067480 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:26.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.117+0000 7f271f400640 1 --2- 172.21.15.27:0/1673067480 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27200cd070 0x7f272006c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:26.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.117+0000 7f271ea00640 1 -- 172.21.15.27:0/1673067480 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27200cd070 msgr2=0x7f272006c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:26.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.117+0000 7f271ea00640 1 --2- 172.21.15.27:0/1673067480 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27200cd070 0x7f272006c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.117+0000 7f271ea00640 1 -- 172.21.15.27:0/1673067480 >> v1:172.21.15.93:6789/0 conn(0x7f27200cef30 legacy=0x7f27200cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:26.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.117+0000 7f271ea00640 1 -- 172.21.15.27:0/1673067480 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27200acb60 con 0x7f27201468d0 2024-01-15T18:15:26.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.121+0000 7f271ea00640 1 --2- 172.21.15.27:0/1673067480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27201468d0 0x7f27200ce9f0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f2710009870 tx=0x7f271001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f56f87ee04cfe2b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:26.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.121+0000 7f271d600640 1 -- 172.21.15.27:0/1673067480 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27100aa670 con 0x7f27201468d0 2024-01-15T18:15:26.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.121+0000 7f271d600640 1 -- 172.21.15.27:0/1673067480 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27100aa7d0 con 0x7f27201468d0 2024-01-15T18:15:26.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.121+0000 7f271d600640 1 -- 172.21.15.27:0/1673067480 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27100aab60 con 0x7f27201468d0 2024-01-15T18:15:26.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.121+0000 7f2725e00640 1 -- 172.21.15.27:0/1673067480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27201468d0 msgr2=0x7f27200ce9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:26.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.121+0000 7f2725e00640 1 --2- 172.21.15.27:0/1673067480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27201468d0 0x7f27200ce9f0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f2710009870 tx=0x7f271001c510 comp rx=0 tx=0).stop 2024-01-15T18:15:26.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.121+0000 7f2725e00640 1 -- 172.21.15.27:0/1673067480 shutdown_connections 2024-01-15T18:15:26.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.121+0000 7f2725e00640 1 --2- 172.21.15.27:0/1673067480 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27200cd070 0x7f272006c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.121+0000 7f2725e00640 1 --2- 172.21.15.27:0/1673067480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27201468d0 0x7f27200ce9f0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.121+0000 7f2725e00640 1 -- 172.21.15.27:0/1673067480 >> 172.21.15.27:0/1673067480 conn(0x7f27200cb700 msgr2=0x7f27200cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:26.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.121+0000 7f2725e00640 1 -- 172.21.15.27:0/1673067480 shutdown_connections 2024-01-15T18:15:26.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.121+0000 7f2725e00640 1 -- 172.21.15.27:0/1673067480 wait complete. 2024-01-15T18:15:26.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.125+0000 7f2725e00640 1 Processor -- start 2024-01-15T18:15:26.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.125+0000 7f2725e00640 1 -- start start 2024-01-15T18:15:26.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.125+0000 7f2725e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27200cd070 0x7f2720160340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:26.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.125+0000 7f2725e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27200cef30 0x7f272015d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:26.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.125+0000 7f2725e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27201468d0 0x7f272015aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:26.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.125+0000 7f2725e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272015eb40 con 0x7f27201468d0 2024-01-15T18:15:26.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.125+0000 7f2725e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272015dc90 con 0x7f27200cd070 2024-01-15T18:15:26.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.125+0000 7f2725e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f272015af80 con 0x7f27200cef30 2024-01-15T18:15:26.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.125+0000 7f271f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27201468d0 0x7f272015aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:26.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.125+0000 7f271ea00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27200cd070 0x7f2720160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:26.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.125+0000 7f271f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27201468d0 0x7f272015aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52426/0 (socket says 172.21.15.27:52426) 2024-01-15T18:15:26.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.125+0000 7f271f400640 1 -- 172.21.15.27:0/4225761989 learned_addr learned my addr 172.21.15.27:0/4225761989 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:26.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.125+0000 7f271e000640 1 --2- 172.21.15.27:0/4225761989 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27200cef30 0x7f272015d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:26.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.125+0000 7f271ea00640 1 -- 172.21.15.27:0/4225761989 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27200cef30 msgr2=0x7f272015d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:26.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.125+0000 7f271ea00640 1 --2- 172.21.15.27:0/4225761989 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27200cef30 0x7f272015d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.125+0000 7f271ea00640 1 -- 172.21.15.27:0/4225761989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27201468d0 msgr2=0x7f272015aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:26.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.125+0000 7f271ea00640 1 --2- 172.21.15.27:0/4225761989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27201468d0 0x7f272015aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.125+0000 7f271ea00640 1 -- 172.21.15.27:0/4225761989 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27100094d0 con 0x7f27200cd070 2024-01-15T18:15:26.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.125+0000 7f271f400640 1 --2- 172.21.15.27:0/4225761989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27201468d0 0x7f272015aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:15:26.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.125+0000 7f271ea00640 1 --2- 172.21.15.27:0/4225761989 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27200cd070 0x7f2720160340 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f27100ab000 tx=0x7f271001ad50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:26.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.125+0000 7f2717400640 1 -- 172.21.15.27:0/4225761989 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27100b3800 con 0x7f27200cd070 2024-01-15T18:15:26.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.125+0000 7f2717400640 1 -- 172.21.15.27:0/4225761989 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27100b3960 con 0x7f27200cd070 2024-01-15T18:15:26.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.125+0000 7f2725e00640 1 -- 172.21.15.27:0/4225761989 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27200cf490 con 0x7f27200cd070 2024-01-15T18:15:26.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.125+0000 7f2717400640 1 -- 172.21.15.27:0/4225761989 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27100b3d10 con 0x7f27200cd070 2024-01-15T18:15:26.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.125+0000 7f2725e00640 1 -- 172.21.15.27:0/4225761989 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f272015e210 con 0x7f27200cd070 2024-01-15T18:15:26.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.129+0000 7f2725e00640 1 -- 172.21.15.27:0/4225761989 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f272006e520 con 0x7f27200cd070 2024-01-15T18:15:26.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.129+0000 7f2717400640 1 -- 172.21.15.27:0/4225761989 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 29) v1 ==== 87282+0+0 (secure 0 0 0) 0x7f27100c6440 con 0x7f27200cd070 2024-01-15T18:15:26.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.129+0000 7f2717400640 1 --2- 172.21.15.27:0/4225761989 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f26f8067550 0x7f26f8069a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:26.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.129+0000 7f2717400640 1 -- 172.21.15.27:0/4225761989 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5823+0+0 (secure 0 0 0) 0x7f271011de00 con 0x7f27200cd070 2024-01-15T18:15:26.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.129+0000 7f271e000640 1 --2- 172.21.15.27:0/4225761989 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f26f8067550 0x7f26f8069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:26.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.129+0000 7f271e000640 1 --2- 172.21.15.27:0/4225761989 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f26f8067550 0x7f26f8069a10 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f27201f9900 tx=0x7f2708009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:26.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.133+0000 7f2717400640 1 -- 172.21.15.27:0/4225761989 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f27100ec780 con 0x7f27200cd070 2024-01-15T18:15:26.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.393+0000 7f2725e00640 1 -- 172.21.15.27:0/4225761989 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone519318", "target": ["mon-mgr", ""]}) v1 -- 0x7f272015c0e0 con 0x7f26f8067550 2024-01-15T18:15:26.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.409+0000 7f2717400640 1 -- 172.21.15.27:0/4225761989 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f272015c0e0 con 0x7f26f8067550 2024-01-15T18:15:26.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.409+0000 7f2725e00640 1 -- 172.21.15.27:0/4225761989 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f26f8067550 msgr2=0x7f26f8069a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:26.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.409+0000 7f2725e00640 1 --2- 172.21.15.27:0/4225761989 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f26f8067550 0x7f26f8069a10 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f27201f9900 tx=0x7f2708009040 comp rx=0 tx=0).stop 2024-01-15T18:15:26.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.409+0000 7f2725e00640 1 -- 172.21.15.27:0/4225761989 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27200cd070 msgr2=0x7f2720160340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:26.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.409+0000 7f2725e00640 1 --2- 172.21.15.27:0/4225761989 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27200cd070 0x7f2720160340 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f27100ab000 tx=0x7f271001ad50 comp rx=0 tx=0).stop 2024-01-15T18:15:26.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.409+0000 7f2725e00640 1 -- 172.21.15.27:0/4225761989 shutdown_connections 2024-01-15T18:15:26.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.409+0000 7f2725e00640 1 --2- 172.21.15.27:0/4225761989 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f26f8067550 0x7f26f8069a10 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.409+0000 7f2725e00640 1 --2- 172.21.15.27:0/4225761989 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f27201468d0 0x7f272015aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.409+0000 7f2725e00640 1 --2- 172.21.15.27:0/4225761989 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f27200cef30 0x7f272015d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.409+0000 7f2725e00640 1 --2- 172.21.15.27:0/4225761989 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f27200cd070 0x7f2720160340 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.409+0000 7f2725e00640 1 -- 172.21.15.27:0/4225761989 >> 172.21.15.27:0/4225761989 conn(0x7f27200cb700 msgr2=0x7f2720160bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:26.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.409+0000 7f2725e00640 1 -- 172.21.15.27:0/4225761989 shutdown_connections 2024-01-15T18:15:26.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.409+0000 7f2725e00640 1 -- 172.21.15.27:0/4225761989 wait complete. 2024-01-15T18:15:26.434 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone51939 2024-01-15T18:15:26.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.577+0000 7f499e000640 1 Processor -- start 2024-01-15T18:15:26.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.577+0000 7f499e000640 1 -- start start 2024-01-15T18:15:26.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.577+0000 7f499e000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f499815d570 0x7f499806be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:26.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.577+0000 7f499e000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f499806b810 0x7f499806c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:26.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.577+0000 7f499e000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49980b0d70 con 0x7f499815d190 2024-01-15T18:15:26.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.577+0000 7f499e000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49980d0540 con 0x7f499806b810 2024-01-15T18:15:26.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.577+0000 7f499e000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49980b0440 con 0x7f499815d570 2024-01-15T18:15:26.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.577+0000 7f4997400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f499815d190 0x7f49980cfcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49170/0 (socket says 172.21.15.27:49170) 2024-01-15T18:15:26.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.577+0000 7f4997400640 1 -- 172.21.15.27:0/359576094 learned_addr learned my addr 172.21.15.27:0/359576094 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:26.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.577+0000 7f4997e00640 1 --2- 172.21.15.27:0/359576094 >> v2:172.21.15.27:3300/0 conn(0x7f499806b810 0x7f499806c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:26.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.581+0000 7f4996a00640 1 --2- 172.21.15.27:0/359576094 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f499815d570 0x7f499806be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.581+0000 7f4996000640 1 -- 172.21.15.27:0/359576094 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3558461615 0 0) 0x7f49980b0d70 con 0x7f499815d190 2024-01-15T18:15:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.581+0000 7f4996000640 1 -- 172.21.15.27:0/359576094 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4978003530 con 0x7f499815d190 2024-01-15T18:15:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.581+0000 7f4997e00640 1 -- 172.21.15.27:0/359576094 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f499815d570 msgr2=0x7f499806be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.581+0000 7f4997e00640 1 --2- 172.21.15.27:0/359576094 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f499815d570 0x7f499806be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.581+0000 7f4997e00640 1 -- 172.21.15.27:0/359576094 >> v1:172.21.15.27:6789/0 conn(0x7f499815d190 legacy=0x7f49980cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:26.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.581+0000 7f4997e00640 1 -- 172.21.15.27:0/359576094 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49980b0720 con 0x7f499806b810 2024-01-15T18:15:26.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.581+0000 7f4997e00640 1 --2- 172.21.15.27:0/359576094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f499806b810 0x7f499806c350 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f4990009d90 tx=0x7f499001c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f93e6ade7cbe919d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:26.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.581+0000 7f4996000640 1 -- 172.21.15.27:0/359576094 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49900a5a00 con 0x7f499806b810 2024-01-15T18:15:26.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.581+0000 7f4996000640 1 -- 172.21.15.27:0/359576094 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49900a5b60 con 0x7f499806b810 2024-01-15T18:15:26.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.581+0000 7f4996000640 1 -- 172.21.15.27:0/359576094 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49900ad440 con 0x7f499806b810 2024-01-15T18:15:26.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.585+0000 7f499e000640 1 -- 172.21.15.27:0/359576094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f499806b810 msgr2=0x7f499806c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:26.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.585+0000 7f499e000640 1 --2- 172.21.15.27:0/359576094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f499806b810 0x7f499806c350 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f4990009d90 tx=0x7f499001c920 comp rx=0 tx=0).stop 2024-01-15T18:15:26.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.585+0000 7f499e000640 1 -- 172.21.15.27:0/359576094 shutdown_connections 2024-01-15T18:15:26.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.585+0000 7f499e000640 1 --2- 172.21.15.27:0/359576094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f499806b810 0x7f499806c350 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.585+0000 7f499e000640 1 --2- 172.21.15.27:0/359576094 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f499815d570 0x7f499806be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.585+0000 7f499e000640 1 -- 172.21.15.27:0/359576094 >> 172.21.15.27:0/359576094 conn(0x7f49980d0e70 msgr2=0x7f49980d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:26.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.585+0000 7f499e000640 1 -- 172.21.15.27:0/359576094 shutdown_connections 2024-01-15T18:15:26.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.585+0000 7f499e000640 1 -- 172.21.15.27:0/359576094 wait complete. 2024-01-15T18:15:26.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.585+0000 7f499e000640 1 Processor -- start 2024-01-15T18:15:26.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.585+0000 7f499e000640 1 -- start start 2024-01-15T18:15:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.585+0000 7f499e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f499806b810 0x7f49980b6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.585+0000 7f499e000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f499815d190 0x7f49981528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.585+0000 7f499e000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f499815d570 0x7f49980c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.585+0000 7f499e000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49980c5a20 con 0x7f499806b810 2024-01-15T18:15:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.585+0000 7f499e000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49980b5ff0 con 0x7f499815d570 2024-01-15T18:15:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.585+0000 7f499e000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49980c5410 con 0x7f499815d190 2024-01-15T18:15:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.585+0000 7f4997400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f499806b810 0x7f49980b6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:26.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.585+0000 7f4997400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f499806b810 0x7f49980b6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52454/0 (socket says 172.21.15.27:52454) 2024-01-15T18:15:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.585+0000 7f4997400640 1 -- 172.21.15.27:0/3446488423 learned_addr learned my addr 172.21.15.27:0/3446488423 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.585+0000 7f4997e00640 1 --2- 172.21.15.27:0/3446488423 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f499815d570 0x7f49980c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:26.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.585+0000 7f4996a00640 1 --2- 172.21.15.27:0/3446488423 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f499815d190 0x7f49981528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:26.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.589+0000 7f4997400640 1 -- 172.21.15.27:0/3446488423 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f499815d190 msgr2=0x7f49981528f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:26.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.589+0000 7f4997400640 1 --2- 172.21.15.27:0/3446488423 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f499815d190 0x7f49981528f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.589+0000 7f4997400640 1 -- 172.21.15.27:0/3446488423 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f499815d570 msgr2=0x7f49980c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:26.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.589+0000 7f4997400640 1 --2- 172.21.15.27:0/3446488423 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f499815d570 0x7f49980c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.589+0000 7f4997400640 1 -- 172.21.15.27:0/3446488423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4990009a70 con 0x7f499806b810 2024-01-15T18:15:26.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.589+0000 7f4997400640 1 --2- 172.21.15.27:0/3446488423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f499806b810 0x7f49980b6940 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f498800c4d0 tx=0x7f498800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:26.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.589+0000 7f498f400640 1 -- 172.21.15.27:0/3446488423 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4988005cf0 con 0x7f499806b810 2024-01-15T18:15:26.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.589+0000 7f498f400640 1 -- 172.21.15.27:0/3446488423 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4988005e50 con 0x7f499806b810 2024-01-15T18:15:26.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.589+0000 7f498f400640 1 -- 172.21.15.27:0/3446488423 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4988007270 con 0x7f499806b810 2024-01-15T18:15:26.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.589+0000 7f499e000640 1 -- 172.21.15.27:0/3446488423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49980b5d10 con 0x7f499806b810 2024-01-15T18:15:26.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.589+0000 7f499e000640 1 -- 172.21.15.27:0/3446488423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4998152e30 con 0x7f499806b810 2024-01-15T18:15:26.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.589+0000 7f499e000640 1 -- 172.21.15.27:0/3446488423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f499804e850 con 0x7f499806b810 2024-01-15T18:15:26.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.589+0000 7f498f400640 1 -- 172.21.15.27:0/3446488423 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 29) v1 ==== 87282+0+0 (secure 0 0 0) 0x7f49880073d0 con 0x7f499806b810 2024-01-15T18:15:26.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.593+0000 7f498f400640 1 --2- 172.21.15.27:0/3446488423 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4970067550 0x7f4970069a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:26.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.593+0000 7f498f400640 1 -- 172.21.15.27:0/3446488423 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f4988082620 con 0x7f499806b810 2024-01-15T18:15:26.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.593+0000 7f4996a00640 1 --2- 172.21.15.27:0/3446488423 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4970067550 0x7f4970069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:26.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.593+0000 7f4996a00640 1 --2- 172.21.15.27:0/3446488423 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4970067550 0x7f4970069a10 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f49980cd570 tx=0x7f4980009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:26.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.597+0000 7f498f400640 1 -- 172.21.15.27:0/3446488423 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f49880510e0 con 0x7f499806b810 2024-01-15T18:15:26.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.773+0000 7f499e000640 1 -- 172.21.15.27:0/3446488423 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone51939", "target": ["mon-mgr", ""]}) v1 -- 0x7f49980b11e0 con 0x7f4970067550 2024-01-15T18:15:26.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.789+0000 7f498f400640 1 -- 172.21.15.27:0/3446488423 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4980004020 con 0x7f4970067550 2024-01-15T18:15:26.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.793+0000 7f499e000640 1 -- 172.21.15.27:0/3446488423 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4970067550 msgr2=0x7f4970069a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:26.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.793+0000 7f499e000640 1 --2- 172.21.15.27:0/3446488423 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4970067550 0x7f4970069a10 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f49980cd570 tx=0x7f4980009190 comp rx=0 tx=0).stop 2024-01-15T18:15:26.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.793+0000 7f499e000640 1 -- 172.21.15.27:0/3446488423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f499806b810 msgr2=0x7f49980b6940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:26.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.793+0000 7f499e000640 1 --2- 172.21.15.27:0/3446488423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f499806b810 0x7f49980b6940 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f498800c4d0 tx=0x7f498800c990 comp rx=0 tx=0).stop 2024-01-15T18:15:26.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.793+0000 7f499e000640 1 -- 172.21.15.27:0/3446488423 shutdown_connections 2024-01-15T18:15:26.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.793+0000 7f499e000640 1 --2- 172.21.15.27:0/3446488423 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4970067550 0x7f4970069a10 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.793+0000 7f499e000640 1 --2- 172.21.15.27:0/3446488423 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f499815d570 0x7f49980c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.793+0000 7f499e000640 1 --2- 172.21.15.27:0/3446488423 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f499815d190 0x7f49981528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.793+0000 7f499e000640 1 --2- 172.21.15.27:0/3446488423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f499806b810 0x7f49980b6940 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.793+0000 7f499e000640 1 -- 172.21.15.27:0/3446488423 >> 172.21.15.27:0/3446488423 conn(0x7f49980d0e70 msgr2=0x7f49980afaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:26.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.793+0000 7f499e000640 1 -- 172.21.15.27:0/3446488423 shutdown_connections 2024-01-15T18:15:26.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.793+0000 7f499e000640 1 -- 172.21.15.27:0/3446488423 wait complete. 2024-01-15T18:15:26.816 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone519317 2024-01-15T18:15:26.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.957+0000 7f1092e00640 1 Processor -- start 2024-01-15T18:15:26.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.957+0000 7f1092e00640 1 -- start start 2024-01-15T18:15:26.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.961+0000 7f1092e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f108c0cde10 0x7f108c0cbf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:26.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.961+0000 7f1092e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f108c0cfc30 0x7f108c0cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:26.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.961+0000 7f1092e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f108c0cb4d0 con 0x7f108c15d680 2024-01-15T18:15:26.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.961+0000 7f1092e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f108c0d0120 con 0x7f108c0cfc30 2024-01-15T18:15:26.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.961+0000 7f1092e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f108c0d0400 con 0x7f108c0cde10 2024-01-15T18:15:26.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.961+0000 7f108be00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f108c15d680 0x7f108c0cd700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49184/0 (socket says 172.21.15.27:49184) 2024-01-15T18:15:26.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.961+0000 7f108be00640 1 -- 172.21.15.27:0/4272958821 learned_addr learned my addr 172.21.15.27:0/4272958821 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:26.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.961+0000 7f1090e00640 1 --2- 172.21.15.27:0/4272958821 >> v2:172.21.15.27:3300/0 conn(0x7f108c0cfc30 0x7f108c0cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:26.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.961+0000 7f108b400640 1 --2- 172.21.15.27:0/4272958821 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f108c0cde10 0x7f108c0cbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:26.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.961+0000 7f108aa00640 1 -- 172.21.15.27:0/4272958821 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691130271 0 0) 0x7f108c0cb4d0 con 0x7f108c15d680 2024-01-15T18:15:26.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.961+0000 7f108aa00640 1 -- 172.21.15.27:0/4272958821 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f106c003530 con 0x7f108c15d680 2024-01-15T18:15:26.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.961+0000 7f108b400640 1 -- 172.21.15.27:0/4272958821 >> v2:172.21.15.27:3300/0 conn(0x7f108c0cfc30 msgr2=0x7f108c0cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:26.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.961+0000 7f108b400640 1 --2- 172.21.15.27:0/4272958821 >> v2:172.21.15.27:3300/0 conn(0x7f108c0cfc30 0x7f108c0cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.961+0000 7f108b400640 1 -- 172.21.15.27:0/4272958821 >> v1:172.21.15.27:6789/0 conn(0x7f108c15d680 legacy=0x7f108c0cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:26.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.961+0000 7f108b400640 1 -- 172.21.15.27:0/4272958821 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f108c0cb770 con 0x7f108c0cde10 2024-01-15T18:15:26.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.961+0000 7f1090e00640 1 --2- 172.21.15.27:0/4272958821 >> v2:172.21.15.27:3300/0 conn(0x7f108c0cfc30 0x7f108c0cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:15:26.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.965+0000 7f108b400640 1 --2- 172.21.15.27:0/4272958821 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f108c0cde10 0x7f108c0cbf70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f10740097c0 tx=0x7f107401c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af83d97b346a132 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:26.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.965+0000 7f108aa00640 1 -- 172.21.15.27:0/4272958821 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f108c0ce850 con 0x7f108c0cde10 2024-01-15T18:15:26.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.965+0000 7f108aa00640 1 -- 172.21.15.27:0/4272958821 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10740ab670 con 0x7f108c0cde10 2024-01-15T18:15:26.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.965+0000 7f108aa00640 1 -- 172.21.15.27:0/4272958821 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10740aba00 con 0x7f108c0cde10 2024-01-15T18:15:26.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.965+0000 7f1092e00640 1 -- 172.21.15.27:0/4272958821 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f108c0cde10 msgr2=0x7f108c0cbf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:26.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.965+0000 7f1092e00640 1 --2- 172.21.15.27:0/4272958821 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f108c0cde10 0x7f108c0cbf70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f10740097c0 tx=0x7f107401c450 comp rx=0 tx=0).stop 2024-01-15T18:15:26.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.965+0000 7f1092e00640 1 -- 172.21.15.27:0/4272958821 shutdown_connections 2024-01-15T18:15:26.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.965+0000 7f1092e00640 1 --2- 172.21.15.27:0/4272958821 >> v2:172.21.15.27:3300/0 conn(0x7f108c0cfc30 0x7f108c0cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.965+0000 7f1092e00640 1 --2- 172.21.15.27:0/4272958821 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f108c0cde10 0x7f108c0cbf70 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.965+0000 7f1092e00640 1 -- 172.21.15.27:0/4272958821 >> 172.21.15.27:0/4272958821 conn(0x7f108c0d0e80 msgr2=0x7f108c0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:26.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.965+0000 7f1092e00640 1 -- 172.21.15.27:0/4272958821 shutdown_connections 2024-01-15T18:15:26.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.965+0000 7f1092e00640 1 -- 172.21.15.27:0/4272958821 wait complete. 2024-01-15T18:15:26.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.965+0000 7f1092e00640 1 Processor -- start 2024-01-15T18:15:26.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.969+0000 7f1092e00640 1 -- start start 2024-01-15T18:15:26.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.969+0000 7f1092e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f108c0cde10 0x7f108c1604e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:26.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.969+0000 7f1092e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f108c0cfc30 0x7f108c06bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:26.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.969+0000 7f1092e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f108c15d680 0x7f108c152830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:26.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.969+0000 7f1092e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108c0ca7b0 con 0x7f108c0cfc30 2024-01-15T18:15:26.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.969+0000 7f1092e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108c0c4ea0 con 0x7f108c15d680 2024-01-15T18:15:26.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.969+0000 7f1092e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f108c0ca1f0 con 0x7f108c0cde10 2024-01-15T18:15:26.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.969+0000 7f108b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f108c0cfc30 0x7f108c06bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:26.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.969+0000 7f108be00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f108c0cde10 0x7f108c1604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:26.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.969+0000 7f108b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f108c0cfc30 0x7f108c06bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52468/0 (socket says 172.21.15.27:52468) 2024-01-15T18:15:26.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.969+0000 7f108b400640 1 -- 172.21.15.27:0/4103323765 learned_addr learned my addr 172.21.15.27:0/4103323765 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:26.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.969+0000 7f1090e00640 1 --2- 172.21.15.27:0/4103323765 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f108c15d680 0x7f108c152830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:26.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.969+0000 7f108b400640 1 -- 172.21.15.27:0/4103323765 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f108c0cde10 msgr2=0x7f108c1604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:26.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.969+0000 7f108b400640 1 --2- 172.21.15.27:0/4103323765 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f108c0cde10 0x7f108c1604e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.969+0000 7f108b400640 1 -- 172.21.15.27:0/4103323765 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f108c15d680 msgr2=0x7f108c152830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:26.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.969+0000 7f108b400640 1 --2- 172.21.15.27:0/4103323765 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f108c15d680 0x7f108c152830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.969+0000 7f108b400640 1 -- 172.21.15.27:0/4103323765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1084009a70 con 0x7f108c0cfc30 2024-01-15T18:15:26.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.969+0000 7f108b400640 1 --2- 172.21.15.27:0/4103323765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f108c0cfc30 0x7f108c06bd60 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f10740ad040 tx=0x7f107401ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:26.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.969+0000 7f1083e00640 1 -- 172.21.15.27:0/4103323765 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10740b5610 con 0x7f108c0cfc30 2024-01-15T18:15:26.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.969+0000 7f1083e00640 1 -- 172.21.15.27:0/4103323765 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10740b57b0 con 0x7f108c0cfc30 2024-01-15T18:15:26.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.969+0000 7f1092e00640 1 -- 172.21.15.27:0/4103323765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1074009420 con 0x7f108c0cfc30 2024-01-15T18:15:26.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.969+0000 7f1083e00640 1 -- 172.21.15.27:0/4103323765 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10740b5b60 con 0x7f108c0cfc30 2024-01-15T18:15:26.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.969+0000 7f1092e00640 1 -- 172.21.15.27:0/4103323765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f108c0c4b30 con 0x7f108c0cfc30 2024-01-15T18:15:26.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.969+0000 7f1092e00640 1 -- 172.21.15.27:0/4103323765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f108c0c4970 con 0x7f108c0cfc30 2024-01-15T18:15:26.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.973+0000 7f1083e00640 1 -- 172.21.15.27:0/4103323765 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 29) v1 ==== 87282+0+0 (secure 0 0 0) 0x7f10740b5cc0 con 0x7f108c0cfc30 2024-01-15T18:15:26.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.973+0000 7f1083e00640 1 --2- 172.21.15.27:0/4103323765 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1064067550 0x7f1064069a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:26.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.973+0000 7f1083e00640 1 -- 172.21.15.27:0/4103323765 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f107411dcf0 con 0x7f108c0cfc30 2024-01-15T18:15:26.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.973+0000 7f108be00640 1 --2- 172.21.15.27:0/4103323765 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1064067550 0x7f1064069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:26.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.973+0000 7f108be00640 1 --2- 172.21.15.27:0/4103323765 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1064067550 0x7f1064069a10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f107c0011f0 tx=0x7f107c007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:26.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:26.977+0000 7f1083e00640 1 -- 172.21.15.27:0/4103323765 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f10740ec720 con 0x7f108c0cfc30 2024-01-15T18:15:27.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.161+0000 7f1092e00640 1 -- 172.21.15.27:0/4103323765 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone519317", "target": ["mon-mgr", ""]}) v1 -- 0x7f108c0c5010 con 0x7f1064067550 2024-01-15T18:15:27.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.177+0000 7f1083e00640 1 -- 172.21.15.27:0/4103323765 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f108c0c5010 con 0x7f1064067550 2024-01-15T18:15:27.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.177+0000 7f1092e00640 1 -- 172.21.15.27:0/4103323765 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1064067550 msgr2=0x7f1064069a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:27.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.177+0000 7f1092e00640 1 --2- 172.21.15.27:0/4103323765 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1064067550 0x7f1064069a10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f107c0011f0 tx=0x7f107c007d40 comp rx=0 tx=0).stop 2024-01-15T18:15:27.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.177+0000 7f1092e00640 1 -- 172.21.15.27:0/4103323765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f108c0cfc30 msgr2=0x7f108c06bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:27.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.177+0000 7f1092e00640 1 --2- 172.21.15.27:0/4103323765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f108c0cfc30 0x7f108c06bd60 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f10740ad040 tx=0x7f107401ac90 comp rx=0 tx=0).stop 2024-01-15T18:15:27.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.181+0000 7f1092e00640 1 -- 172.21.15.27:0/4103323765 shutdown_connections 2024-01-15T18:15:27.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.181+0000 7f1092e00640 1 --2- 172.21.15.27:0/4103323765 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1064067550 0x7f1064069a10 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:27.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.181+0000 7f1092e00640 1 --2- 172.21.15.27:0/4103323765 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f108c15d680 0x7f108c152830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:27.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.181+0000 7f1092e00640 1 --2- 172.21.15.27:0/4103323765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f108c0cfc30 0x7f108c06bd60 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:27.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.181+0000 7f1092e00640 1 --2- 172.21.15.27:0/4103323765 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f108c0cde10 0x7f108c1604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:27.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.181+0000 7f1092e00640 1 -- 172.21.15.27:0/4103323765 >> 172.21.15.27:0/4103323765 conn(0x7f108c0d0e80 msgr2=0x7f108c0cbe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:27.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.181+0000 7f1092e00640 1 -- 172.21.15.27:0/4103323765 shutdown_connections 2024-01-15T18:15:27.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.181+0000 7f1092e00640 1 -- 172.21.15.27:0/4103323765 wait complete. 2024-01-15T18:15:27.205 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone51936 2024-01-15T18:15:27.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.349+0000 7f3ac3c00640 1 Processor -- start 2024-01-15T18:15:27.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.353+0000 7f3ac3c00640 1 -- start start 2024-01-15T18:15:27.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.353+0000 7f3ac3c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3abc15d480 0x7f3abc0cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:27.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.353+0000 7f3ac3c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f3abc09a120 0x7f3abc0d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:27.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.353+0000 7f3ac3c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3abc15d860 con 0x7f3abc06c740 2024-01-15T18:15:27.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.353+0000 7f3ac3c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3abc06b6a0 con 0x7f3abc15d480 2024-01-15T18:15:27.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.353+0000 7f3ac3c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3abc06b980 con 0x7f3abc09a120 2024-01-15T18:15:27.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.353+0000 7f3ac1200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3abc15d480 0x7f3abc0cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:27.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.353+0000 7f3ac1200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3abc15d480 0x7f3abc0cfbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52480/0 (socket says 172.21.15.27:52480) 2024-01-15T18:15:27.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.353+0000 7f3ac1200640 1 -- 172.21.15.27:0/2190863018 learned_addr learned my addr 172.21.15.27:0/2190863018 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:27.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.353+0000 7f3ac1c00640 1 --2- 172.21.15.27:0/2190863018 >> v2:172.21.15.93:3300/0 conn(0x7f3abc09a120 0x7f3abc0d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:27.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.353+0000 7f3ac1200640 1 -- 172.21.15.27:0/2190863018 >> v2:172.21.15.93:3300/0 conn(0x7f3abc09a120 msgr2=0x7f3abc0d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:27.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.353+0000 7f3ac1200640 1 --2- 172.21.15.27:0/2190863018 >> v2:172.21.15.93:3300/0 conn(0x7f3abc09a120 0x7f3abc0d0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:27.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.353+0000 7f3ac1200640 1 -- 172.21.15.27:0/2190863018 >> v1:172.21.15.93:6789/0 conn(0x7f3abc06c740 legacy=0x7f3abc099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:27.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.353+0000 7f3ac1200640 1 -- 172.21.15.27:0/2190863018 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3abc06bc60 con 0x7f3abc15d480 2024-01-15T18:15:27.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.357+0000 7f3ac1200640 1 --2- 172.21.15.27:0/2190863018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3abc15d480 0x7f3abc0cfbf0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f3aac009870 tx=0x7f3aac01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fff369e58bb54fa server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:27.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.357+0000 7f3abb400640 1 -- 172.21.15.27:0/2190863018 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aac0ad680 con 0x7f3abc15d480 2024-01-15T18:15:27.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.357+0000 7f3abb400640 1 -- 172.21.15.27:0/2190863018 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aac0ad7e0 con 0x7f3abc15d480 2024-01-15T18:15:27.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.357+0000 7f3abb400640 1 -- 172.21.15.27:0/2190863018 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aac0adb70 con 0x7f3abc15d480 2024-01-15T18:15:27.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.357+0000 7f3ac3c00640 1 -- 172.21.15.27:0/2190863018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3abc15d480 msgr2=0x7f3abc0cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:27.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.357+0000 7f3ac3c00640 1 --2- 172.21.15.27:0/2190863018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3abc15d480 0x7f3abc0cfbf0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f3aac009870 tx=0x7f3aac01c490 comp rx=0 tx=0).stop 2024-01-15T18:15:27.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.357+0000 7f3ac3c00640 1 -- 172.21.15.27:0/2190863018 shutdown_connections 2024-01-15T18:15:27.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.357+0000 7f3ac3c00640 1 --2- 172.21.15.27:0/2190863018 >> v2:172.21.15.93:3300/0 conn(0x7f3abc09a120 0x7f3abc0d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:27.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.357+0000 7f3ac3c00640 1 --2- 172.21.15.27:0/2190863018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3abc15d480 0x7f3abc0cfbf0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:27.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.357+0000 7f3ac3c00640 1 -- 172.21.15.27:0/2190863018 >> 172.21.15.27:0/2190863018 conn(0x7f3abc0b2a80 msgr2=0x7f3abc0b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:27.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.357+0000 7f3ac3c00640 1 -- 172.21.15.27:0/2190863018 shutdown_connections 2024-01-15T18:15:27.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.357+0000 7f3ac3c00640 1 -- 172.21.15.27:0/2190863018 wait complete. 2024-01-15T18:15:27.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.357+0000 7f3ac3c00640 1 Processor -- start 2024-01-15T18:15:27.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.357+0000 7f3ac3c00640 1 -- start start 2024-01-15T18:15:27.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.361+0000 7f3ac3c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3abc06c740 0x7f3abc15fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.361+0000 7f3ac3c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3abc09a120 0x7f3abc160750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.361+0000 7f3ac3c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3abc15d480 0x7f3abc14c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.361+0000 7f3ac3c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3abc15a7d0 con 0x7f3abc15d480 2024-01-15T18:15:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.361+0000 7f3ac3c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3abc1603e0 con 0x7f3abc09a120 2024-01-15T18:15:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.361+0000 7f3ac3c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3abc153120 con 0x7f3abc06c740 2024-01-15T18:15:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.361+0000 7f3ac1c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3abc15d480 0x7f3abc14c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.361+0000 7f3ac1c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3abc15d480 0x7f3abc14c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52492/0 (socket says 172.21.15.27:52492) 2024-01-15T18:15:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.361+0000 7f3ac1c00640 1 -- 172.21.15.27:0/669661916 learned_addr learned my addr 172.21.15.27:0/669661916 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.361+0000 7f3abbe00640 1 --2- 172.21.15.27:0/669661916 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3abc09a120 0x7f3abc160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.361+0000 7f3ac1200640 1 --2- 172.21.15.27:0/669661916 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3abc06c740 0x7f3abc15fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.361+0000 7f3ac1c00640 1 -- 172.21.15.27:0/669661916 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3abc06c740 msgr2=0x7f3abc15fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.361+0000 7f3ac1c00640 1 --2- 172.21.15.27:0/669661916 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3abc06c740 0x7f3abc15fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.361+0000 7f3ac1c00640 1 -- 172.21.15.27:0/669661916 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3abc09a120 msgr2=0x7f3abc160750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.361+0000 7f3ac1c00640 1 --2- 172.21.15.27:0/669661916 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3abc09a120 0x7f3abc160750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.361+0000 7f3ac1c00640 1 -- 172.21.15.27:0/669661916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aac0094d0 con 0x7f3abc15d480 2024-01-15T18:15:27.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.361+0000 7f3abbe00640 1 --2- 172.21.15.27:0/669661916 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3abc09a120 0x7f3abc160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:27.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.361+0000 7f3ac1c00640 1 --2- 172.21.15.27:0/669661916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3abc15d480 0x7f3abc14c0f0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4007de0 tx=0x7f3ab400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:27.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.361+0000 7f3ab8c00640 1 -- 172.21.15.27:0/669661916 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab4014620 con 0x7f3abc15d480 2024-01-15T18:15:27.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.361+0000 7f3ab8c00640 1 -- 172.21.15.27:0/669661916 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ab400ee90 con 0x7f3abc15d480 2024-01-15T18:15:27.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.361+0000 7f3ab8c00640 1 -- 172.21.15.27:0/669661916 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab40149e0 con 0x7f3abc15d480 2024-01-15T18:15:27.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.361+0000 7f3ac3c00640 1 -- 172.21.15.27:0/669661916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3abc153400 con 0x7f3abc15d480 2024-01-15T18:15:27.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.361+0000 7f3ac3c00640 1 -- 172.21.15.27:0/669661916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3abc1515a0 con 0x7f3abc15d480 2024-01-15T18:15:27.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.361+0000 7f3ac3c00640 1 -- 172.21.15.27:0/669661916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3abc04e850 con 0x7f3abc15d480 2024-01-15T18:15:27.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.365+0000 7f3ab8c00640 1 -- 172.21.15.27:0/669661916 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 29) v1 ==== 87282+0+0 (secure 0 0 0) 0x7f3ab4014b40 con 0x7f3abc15d480 2024-01-15T18:15:27.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.365+0000 7f3ab8c00640 1 --2- 172.21.15.27:0/669661916 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a94067550 0x7f3a94069a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:27.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.365+0000 7f3ab8c00640 1 -- 172.21.15.27:0/669661916 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f3ab40837f0 con 0x7f3abc15d480 2024-01-15T18:15:27.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.365+0000 7f3ac1200640 1 --2- 172.21.15.27:0/669661916 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a94067550 0x7f3a94069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:27.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.365+0000 7f3ac1200640 1 --2- 172.21.15.27:0/669661916 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a94067550 0x7f3a94069a10 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f3aac0af000 tx=0x7f3aac0a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:27.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.369+0000 7f3ab8c00640 1 -- 172.21.15.27:0/669661916 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3ab4052480 con 0x7f3abc15d480 2024-01-15T18:15:27.497 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:15:27.500 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:15:27.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.553+0000 7f3ac3c00640 1 -- 172.21.15.27:0/669661916 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone51936", "target": ["mon-mgr", ""]}) v1 -- 0x7f3abc067110 con 0x7f3a94067550 2024-01-15T18:15:27.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.569+0000 7f3ab8c00640 1 -- 172.21.15.27:0/669661916 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3abc067110 con 0x7f3a94067550 2024-01-15T18:15:27.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.569+0000 7f3ac3c00640 1 -- 172.21.15.27:0/669661916 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a94067550 msgr2=0x7f3a94069a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:27.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.569+0000 7f3ac3c00640 1 --2- 172.21.15.27:0/669661916 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a94067550 0x7f3a94069a10 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f3aac0af000 tx=0x7f3aac0a7000 comp rx=0 tx=0).stop 2024-01-15T18:15:27.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.569+0000 7f3ac3c00640 1 -- 172.21.15.27:0/669661916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3abc15d480 msgr2=0x7f3abc14c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:27.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.569+0000 7f3ac3c00640 1 --2- 172.21.15.27:0/669661916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3abc15d480 0x7f3abc14c0f0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4007de0 tx=0x7f3ab400e4b0 comp rx=0 tx=0).stop 2024-01-15T18:15:27.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.569+0000 7f3ac3c00640 1 -- 172.21.15.27:0/669661916 shutdown_connections 2024-01-15T18:15:27.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.569+0000 7f3ac3c00640 1 --2- 172.21.15.27:0/669661916 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a94067550 0x7f3a94069a10 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:27.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.569+0000 7f3ac3c00640 1 --2- 172.21.15.27:0/669661916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3abc15d480 0x7f3abc14c0f0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:27.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.569+0000 7f3ac3c00640 1 --2- 172.21.15.27:0/669661916 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3abc09a120 0x7f3abc160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:27.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.569+0000 7f3ac3c00640 1 --2- 172.21.15.27:0/669661916 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3abc06c740 0x7f3abc15fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:27.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.569+0000 7f3ac3c00640 1 -- 172.21.15.27:0/669661916 >> 172.21.15.27:0/669661916 conn(0x7f3abc0b2a80 msgr2=0x7f3abc0ce310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:27.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.573+0000 7f3ac3c00640 1 -- 172.21.15.27:0/669661916 shutdown_connections 2024-01-15T18:15:27.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.573+0000 7f3ac3c00640 1 -- 172.21.15.27:0/669661916 wait complete. 2024-01-15T18:15:27.594 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone519314 2024-01-15T18:15:27.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.781+0000 7fd163400640 1 Processor -- start 2024-01-15T18:15:27.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.781+0000 7fd163400640 1 -- start start 2024-01-15T18:15:27.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.781+0000 7fd163400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fd15c0d1a40 0x7fd15c0cba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:27.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.781+0000 7fd163400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd15c0b0650 0x7fd15c0b0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:27.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.781+0000 7fd163400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd15c0cbfa0 con 0x7fd15c0cf450 2024-01-15T18:15:27.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.781+0000 7fd163400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd15c0cefa0 con 0x7fd15c0d1a40 2024-01-15T18:15:27.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.781+0000 7fd163400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c0ce240 con 0x7fd15c0b0650 2024-01-15T18:15:27.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.781+0000 7fd160a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd15c0cf450 0x7fd15c0cf830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49196/0 (socket says 172.21.15.27:49196) 2024-01-15T18:15:27.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.781+0000 7fd160a00640 1 -- 172.21.15.27:0/2590947047 learned_addr learned my addr 172.21.15.27:0/2590947047 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:27.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.781+0000 7fd15be00640 1 --2- 172.21.15.27:0/2590947047 >> v2:172.21.15.93:3300/0 conn(0x7fd15c0d1a40 0x7fd15c0cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:27.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.781+0000 7fd161400640 1 --2- 172.21.15.27:0/2590947047 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd15c0b0650 0x7fd15c0b0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:27.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.781+0000 7fd15b400640 1 -- 172.21.15.27:0/2590947047 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776963037 0 0) 0x7fd15c0cbfa0 con 0x7fd15c0cf450 2024-01-15T18:15:27.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.781+0000 7fd15b400640 1 -- 172.21.15.27:0/2590947047 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd13c003530 con 0x7fd15c0cf450 2024-01-15T18:15:27.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.781+0000 7fd161400640 1 -- 172.21.15.27:0/2590947047 >> v2:172.21.15.93:3300/0 conn(0x7fd15c0d1a40 msgr2=0x7fd15c0cba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:27.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.781+0000 7fd161400640 1 --2- 172.21.15.27:0/2590947047 >> v2:172.21.15.93:3300/0 conn(0x7fd15c0d1a40 0x7fd15c0cba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:27.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.781+0000 7fd161400640 1 -- 172.21.15.27:0/2590947047 >> v1:172.21.15.27:6789/0 conn(0x7fd15c0cf450 legacy=0x7fd15c0cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:27.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.781+0000 7fd161400640 1 -- 172.21.15.27:0/2590947047 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c0ccd70 con 0x7fd15c0b0650 2024-01-15T18:15:27.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.785+0000 7fd161400640 1 --2- 172.21.15.27:0/2590947047 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd15c0b0650 0x7fd15c0b0ac0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd154009d90 tx=0x7fd15401c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5435dfebe96245e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:27.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.785+0000 7fd15b400640 1 -- 172.21.15.27:0/2590947047 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1540a5810 con 0x7fd15c0b0650 2024-01-15T18:15:27.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.785+0000 7fd15b400640 1 -- 172.21.15.27:0/2590947047 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1540a5970 con 0x7fd15c0b0650 2024-01-15T18:15:27.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.785+0000 7fd15b400640 1 -- 172.21.15.27:0/2590947047 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1540a5d00 con 0x7fd15c0b0650 2024-01-15T18:15:27.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.785+0000 7fd163400640 1 -- 172.21.15.27:0/2590947047 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd15c0b0650 msgr2=0x7fd15c0b0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:27.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.785+0000 7fd163400640 1 --2- 172.21.15.27:0/2590947047 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd15c0b0650 0x7fd15c0b0ac0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd154009d90 tx=0x7fd15401c950 comp rx=0 tx=0).stop 2024-01-15T18:15:27.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.785+0000 7fd163400640 1 -- 172.21.15.27:0/2590947047 shutdown_connections 2024-01-15T18:15:27.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.785+0000 7fd163400640 1 --2- 172.21.15.27:0/2590947047 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd15c0b0650 0x7fd15c0b0ac0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:27.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.785+0000 7fd163400640 1 --2- 172.21.15.27:0/2590947047 >> v2:172.21.15.93:3300/0 conn(0x7fd15c0d1a40 0x7fd15c0cba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:27.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.785+0000 7fd163400640 1 -- 172.21.15.27:0/2590947047 >> 172.21.15.27:0/2590947047 conn(0x7fd15c0d0e70 msgr2=0x7fd15c0d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:27.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.785+0000 7fd163400640 1 -- 172.21.15.27:0/2590947047 shutdown_connections 2024-01-15T18:15:27.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.785+0000 7fd163400640 1 -- 172.21.15.27:0/2590947047 wait complete. 2024-01-15T18:15:27.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.789+0000 7fd163400640 1 Processor -- start 2024-01-15T18:15:27.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.789+0000 7fd163400640 1 -- start start 2024-01-15T18:15:27.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.789+0000 7fd163400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd15c0b0650 0x7fd15c159260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:27.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.789+0000 7fd163400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd15c0cf450 0x7fd15c1584e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:27.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.789+0000 7fd163400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd15c0d1a40 0x7fd15c0ca240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:27.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.789+0000 7fd163400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c0adb90 con 0x7fd15c0b0650 2024-01-15T18:15:27.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.789+0000 7fd163400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c15b030 con 0x7fd15c0cf450 2024-01-15T18:15:27.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.789+0000 7fd163400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c15ace0 con 0x7fd15c0d1a40 2024-01-15T18:15:27.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.789+0000 7fd160a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd15c0b0650 0x7fd15c159260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:27.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.789+0000 7fd160a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd15c0b0650 0x7fd15c159260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52494/0 (socket says 172.21.15.27:52494) 2024-01-15T18:15:27.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.789+0000 7fd160a00640 1 -- 172.21.15.27:0/1942667540 learned_addr learned my addr 172.21.15.27:0/1942667540 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:27.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.789+0000 7fd161400640 1 --2- 172.21.15.27:0/1942667540 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd15c0d1a40 0x7fd15c0ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:27.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.789+0000 7fd15be00640 1 --2- 172.21.15.27:0/1942667540 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd15c0cf450 0x7fd15c1584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:27.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.789+0000 7fd160a00640 1 -- 172.21.15.27:0/1942667540 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd15c0d1a40 msgr2=0x7fd15c0ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:27.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.789+0000 7fd160a00640 1 --2- 172.21.15.27:0/1942667540 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd15c0d1a40 0x7fd15c0ca240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:27.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.789+0000 7fd160a00640 1 -- 172.21.15.27:0/1942667540 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd15c0cf450 msgr2=0x7fd15c1584e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:27.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.789+0000 7fd160a00640 1 --2- 172.21.15.27:0/1942667540 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd15c0cf450 0x7fd15c1584e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:27.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.789+0000 7fd160a00640 1 -- 172.21.15.27:0/1942667540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd154009a70 con 0x7fd15c0b0650 2024-01-15T18:15:27.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.789+0000 7fd15be00640 1 --2- 172.21.15.27:0/1942667540 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd15c0cf450 0x7fd15c1584e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:27.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.789+0000 7fd160a00640 1 --2- 172.21.15.27:0/1942667540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd15c0b0650 0x7fd15c159260 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fd14c00c420 tx=0x7fd14c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:27.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.789+0000 7fd158c00640 1 -- 172.21.15.27:0/1942667540 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c005cf0 con 0x7fd15c0b0650 2024-01-15T18:15:27.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.789+0000 7fd158c00640 1 -- 172.21.15.27:0/1942667540 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14c005e50 con 0x7fd15c0b0650 2024-01-15T18:15:27.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.789+0000 7fd158c00640 1 -- 172.21.15.27:0/1942667540 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c0072d0 con 0x7fd15c0b0650 2024-01-15T18:15:27.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.789+0000 7fd163400640 1 -- 172.21.15.27:0/1942667540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd15c0add70 con 0x7fd15c0b0650 2024-01-15T18:15:27.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.789+0000 7fd163400640 1 -- 172.21.15.27:0/1942667540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd15c158a20 con 0x7fd15c0b0650 2024-01-15T18:15:27.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.793+0000 7fd163400640 1 -- 172.21.15.27:0/1942667540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd15c06d610 con 0x7fd15c0b0650 2024-01-15T18:15:27.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.793+0000 7fd158c00640 1 -- 172.21.15.27:0/1942667540 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 29) v1 ==== 87282+0+0 (secure 0 0 0) 0x7fd14c007430 con 0x7fd15c0b0650 2024-01-15T18:15:27.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.793+0000 7fd158c00640 1 --2- 172.21.15.27:0/1942667540 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd1340675a0 0x7fd134069a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:27.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.793+0000 7fd158c00640 1 -- 172.21.15.27:0/1942667540 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fd14c0825e0 con 0x7fd15c0b0650 2024-01-15T18:15:27.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.793+0000 7fd15be00640 1 --2- 172.21.15.27:0/1942667540 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd1340675a0 0x7fd134069a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:27.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.793+0000 7fd15be00640 1 --2- 172.21.15.27:0/1942667540 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd1340675a0 0x7fd134069a60 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fd15c15dff0 tx=0x7fd144008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.797+0000 7fd158c00640 1 -- 172.21.15.27:0/1942667540 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd14c0510e0 con 0x7fd15c0b0650 2024-01-15T18:15:27.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:27.981+0000 7fd163400640 1 -- 172.21.15.27:0/1942667540 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone519314", "target": ["mon-mgr", ""]}) v1 -- 0x7fd15c0cca90 con 0x7fd1340675a0 2024-01-15T18:15:28.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.001+0000 7fd158c00640 1 -- 172.21.15.27:0/1942667540 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd144011a10 con 0x7fd1340675a0 2024-01-15T18:15:28.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.001+0000 7fd163400640 1 -- 172.21.15.27:0/1942667540 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd1340675a0 msgr2=0x7fd134069a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:28.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.001+0000 7fd163400640 1 --2- 172.21.15.27:0/1942667540 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd1340675a0 0x7fd134069a60 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fd15c15dff0 tx=0x7fd144008040 comp rx=0 tx=0).stop 2024-01-15T18:15:28.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.001+0000 7fd163400640 1 -- 172.21.15.27:0/1942667540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd15c0b0650 msgr2=0x7fd15c159260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:28.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.001+0000 7fd163400640 1 --2- 172.21.15.27:0/1942667540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd15c0b0650 0x7fd15c159260 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fd14c00c420 tx=0x7fd14c00c8e0 comp rx=0 tx=0).stop 2024-01-15T18:15:28.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.001+0000 7fd163400640 1 -- 172.21.15.27:0/1942667540 shutdown_connections 2024-01-15T18:15:28.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.001+0000 7fd163400640 1 --2- 172.21.15.27:0/1942667540 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd1340675a0 0x7fd134069a60 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:28.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.001+0000 7fd163400640 1 --2- 172.21.15.27:0/1942667540 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd15c0d1a40 0x7fd15c0ca240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:28.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.001+0000 7fd163400640 1 --2- 172.21.15.27:0/1942667540 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd15c0cf450 0x7fd15c1584e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:28.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.001+0000 7fd163400640 1 --2- 172.21.15.27:0/1942667540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd15c0b0650 0x7fd15c159260 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:28.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.001+0000 7fd163400640 1 -- 172.21.15.27:0/1942667540 >> 172.21.15.27:0/1942667540 conn(0x7fd15c0d0e70 msgr2=0x7fd15c06f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:28.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.001+0000 7fd163400640 1 -- 172.21.15.27:0/1942667540 shutdown_connections 2024-01-15T18:15:28.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.001+0000 7fd163400640 1 -- 172.21.15.27:0/1942667540 wait complete. 2024-01-15T18:15:28.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:28.027 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-15T18:15:28.043 INFO:teuthology.orchestra.run.smithi027.stdout:2 2024-01-15T18:15:28.044 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-15T18:15:28.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.253+0000 7fa6f9600640 1 Processor -- start 2024-01-15T18:15:28.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.253+0000 7fa6f9600640 1 -- start start 2024-01-15T18:15:28.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.253+0000 7fa6f9600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa6f41619b0 0x7fa6f4161d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:28.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.253+0000 7fa6f9600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f40d3ff0 con 0x7fa6f4072500 2024-01-15T18:15:28.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.253+0000 7fa6f9600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f406c940 con 0x7fa6f4070540 2024-01-15T18:15:28.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.253+0000 7fa6f9600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f406cc20 con 0x7fa6f41619b0 2024-01-15T18:15:28.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.257+0000 7fa6f2000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa6f4072500 0x7fa6f406dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49206/0 (socket says 172.21.15.27:49206) 2024-01-15T18:15:28.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.257+0000 7fa6f2000640 1 -- 172.21.15.27:0/473077322 learned_addr learned my addr 172.21.15.27:0/473077322 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:28.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.257+0000 7fa6f2a00640 1 --2- 172.21.15.27:0/473077322 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa6f41619b0 0x7fa6f4161d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:28.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.257+0000 7fa6f2a00640 1 -- 172.21.15.27:0/473077322 >> v1:172.21.15.93:6789/0 conn(0x7fa6f4070540 legacy=0x7fa6f40ba6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:28.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.257+0000 7fa6f2a00640 1 -- 172.21.15.27:0/473077322 >> v1:172.21.15.27:6789/0 conn(0x7fa6f4072500 legacy=0x7fa6f406dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:28.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.257+0000 7fa6f1600640 1 -- 172.21.15.27:0/473077322 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455420318 0 0) 0x7fa6f40d3ff0 con 0x7fa6f4072500 2024-01-15T18:15:28.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.257+0000 7fa6f2a00640 1 -- 172.21.15.27:0/473077322 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f406cf00 con 0x7fa6f41619b0 2024-01-15T18:15:28.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.261+0000 7fa6f2a00640 1 --2- 172.21.15.27:0/473077322 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa6f41619b0 0x7fa6f4161d90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa6e40097f0 tx=0x7fa6e401c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a7b08f51d279ae0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:28.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.261+0000 7fa6f1600640 1 -- 172.21.15.27:0/473077322 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e40aa5f0 con 0x7fa6f41619b0 2024-01-15T18:15:28.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.261+0000 7fa6f1600640 1 -- 172.21.15.27:0/473077322 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e40aa750 con 0x7fa6f41619b0 2024-01-15T18:15:28.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.261+0000 7fa6f1600640 1 -- 172.21.15.27:0/473077322 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e40aaae0 con 0x7fa6f41619b0 2024-01-15T18:15:28.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.261+0000 7fa6f9600640 1 -- 172.21.15.27:0/473077322 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa6f41619b0 msgr2=0x7fa6f4161d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:28.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.261+0000 7fa6f9600640 1 --2- 172.21.15.27:0/473077322 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa6f41619b0 0x7fa6f4161d90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa6e40097f0 tx=0x7fa6e401c490 comp rx=0 tx=0).stop 2024-01-15T18:15:28.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.265+0000 7fa6f9600640 1 -- 172.21.15.27:0/473077322 shutdown_connections 2024-01-15T18:15:28.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.265+0000 7fa6f9600640 1 --2- 172.21.15.27:0/473077322 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa6f41619b0 0x7fa6f4161d90 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:28.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.265+0000 7fa6f9600640 1 -- 172.21.15.27:0/473077322 >> 172.21.15.27:0/473077322 conn(0x7fa6f40b93c0 msgr2=0x7fa6f40b97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:28.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.265+0000 7fa6f9600640 1 -- 172.21.15.27:0/473077322 shutdown_connections 2024-01-15T18:15:28.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.265+0000 7fa6f9600640 1 -- 172.21.15.27:0/473077322 wait complete. 2024-01-15T18:15:28.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.265+0000 7fa6f9600640 1 Processor -- start 2024-01-15T18:15:28.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.265+0000 7fa6f9600640 1 -- start start 2024-01-15T18:15:28.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.269+0000 7fa6f9600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa6f4070540 0x7fa6f40d6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:28.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.269+0000 7fa6f9600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa6f4072500 0x7fa6f4150dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:28.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.269+0000 7fa6f9600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6f41619b0 0x7fa6f4151310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:28.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.269+0000 7fa6f9600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f40d66d0 con 0x7fa6f41619b0 2024-01-15T18:15:28.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.269+0000 7fa6f9600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f415fc00 con 0x7fa6f4072500 2024-01-15T18:15:28.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.269+0000 7fa6f9600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f415fe10 con 0x7fa6f4070540 2024-01-15T18:15:28.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.269+0000 7fa6f3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6f41619b0 0x7fa6f4151310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:28.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.269+0000 7fa6f3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6f41619b0 0x7fa6f4151310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52510/0 (socket says 172.21.15.27:52510) 2024-01-15T18:15:28.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.269+0000 7fa6f3400640 1 -- 172.21.15.27:0/3621852265 learned_addr learned my addr 172.21.15.27:0/3621852265 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:28.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.269+0000 7fa6f2a00640 1 --2- 172.21.15.27:0/3621852265 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa6f4070540 0x7fa6f40d6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:28.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.269+0000 7fa6f3400640 1 -- 172.21.15.27:0/3621852265 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa6f4070540 msgr2=0x7fa6f40d6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:28.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.269+0000 7fa6f3400640 1 --2- 172.21.15.27:0/3621852265 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa6f4070540 0x7fa6f40d6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:28.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.269+0000 7fa6f3400640 1 -- 172.21.15.27:0/3621852265 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa6f4072500 msgr2=0x7fa6f4150dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-15T18:15:28.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.269+0000 7fa6f3400640 1 --2- 172.21.15.27:0/3621852265 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa6f4072500 0x7fa6f4150dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:28.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.269+0000 7fa6f3400640 1 -- 172.21.15.27:0/3621852265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e4009420 con 0x7fa6f41619b0 2024-01-15T18:15:28.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.273+0000 7fa6f3400640 1 --2- 172.21.15.27:0/3621852265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6f41619b0 0x7fa6f4151310 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec00c730 tx=0x7fa6ec00cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:28.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.273+0000 7fa6eaa00640 1 -- 172.21.15.27:0/3621852265 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec003b80 con 0x7fa6f41619b0 2024-01-15T18:15:28.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.273+0000 7fa6f9600640 1 -- 172.21.15.27:0/3621852265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f4070ee0 con 0x7fa6f41619b0 2024-01-15T18:15:28.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.273+0000 7fa6eaa00640 1 -- 172.21.15.27:0/3621852265 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ec003d00 con 0x7fa6f41619b0 2024-01-15T18:15:28.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.273+0000 7fa6f9600640 1 -- 172.21.15.27:0/3621852265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6f41603d0 con 0x7fa6f41619b0 2024-01-15T18:15:28.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.273+0000 7fa6eaa00640 1 -- 172.21.15.27:0/3621852265 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec005310 con 0x7fa6f41619b0 2024-01-15T18:15:28.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.273+0000 7fa6f9600640 1 -- 172.21.15.27:0/3621852265 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6f40a5a20 con 0x7fa6f41619b0 2024-01-15T18:15:28.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.277+0000 7fa6eaa00640 1 -- 172.21.15.27:0/3621852265 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 29) v1 ==== 87282+0+0 (secure 0 0 0) 0x7fa6ec005470 con 0x7fa6f41619b0 2024-01-15T18:15:28.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.281+0000 7fa6eaa00640 1 --2- 172.21.15.27:0/3621852265 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa6cc0675b0 0x7fa6cc069a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.281+0000 7fa6eaa00640 1 -- 172.21.15.27:0/3621852265 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fa6ec083650 con 0x7fa6f41619b0 2024-01-15T18:15:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.281+0000 7fa6f2a00640 1 --2- 172.21.15.27:0/3621852265 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa6cc0675b0 0x7fa6cc069a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:28.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.281+0000 7fa6f2a00640 1 --2- 172.21.15.27:0/3621852265 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa6cc0675b0 0x7fa6cc069a70 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa6e40ae000 tx=0x7fa6e40a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:28.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.289+0000 7fa6eaa00640 1 -- 172.21.15.27:0/3621852265 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa6ec052340 con 0x7fa6f41619b0 2024-01-15T18:15:28.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:28.649+0000 7fa6f9600640 1 -- 172.21.15.27:0/3621852265 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6f40a5740 con 0x7fa6cc0675b0 2024-01-15T18:15:28.723 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-01-15T18:15:28.721+0000 7f053dc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-15T18:15:28.808 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-15T18:15:31.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:31.229+0000 7fa6eaa00640 1 -- 172.21.15.27:0/3621852265 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7fa6ec005720 con 0x7fa6f41619b0 2024-01-15T18:15:31.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:31.773+0000 7fa6eaa00640 1 -- 172.21.15.27:0/3621852265 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fa6f40a5740 con 0x7fa6cc0675b0 2024-01-15T18:15:31.775 INFO:teuthology.orchestra.run.smithi027.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-15T18:15:31.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:31.777+0000 7fa6f9600640 1 -- 172.21.15.27:0/3621852265 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa6cc0675b0 msgr2=0x7fa6cc069a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:31.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:31.777+0000 7fa6f9600640 1 --2- 172.21.15.27:0/3621852265 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa6cc0675b0 0x7fa6cc069a70 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa6e40ae000 tx=0x7fa6e40a7000 comp rx=0 tx=0).stop 2024-01-15T18:15:31.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:31.777+0000 7fa6f9600640 1 -- 172.21.15.27:0/3621852265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6f41619b0 msgr2=0x7fa6f4151310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:31.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:31.777+0000 7fa6f9600640 1 --2- 172.21.15.27:0/3621852265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6f41619b0 0x7fa6f4151310 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec00c730 tx=0x7fa6ec00cb30 comp rx=0 tx=0).stop 2024-01-15T18:15:31.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:31.777+0000 7fa6f9600640 1 -- 172.21.15.27:0/3621852265 shutdown_connections 2024-01-15T18:15:31.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:31.777+0000 7fa6f9600640 1 --2- 172.21.15.27:0/3621852265 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa6cc0675b0 0x7fa6cc069a70 secure :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa6e40ae000 tx=0x7fa6e40a7000 comp rx=0 tx=0).stop 2024-01-15T18:15:31.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:31.777+0000 7fa6f9600640 1 --2- 172.21.15.27:0/3621852265 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6f41619b0 0x7fa6f4151310 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:31.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:31.777+0000 7fa6f9600640 1 --2- 172.21.15.27:0/3621852265 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa6f4072500 0x7fa6f4150dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:31.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:31.777+0000 7fa6f9600640 1 --2- 172.21.15.27:0/3621852265 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa6f4070540 0x7fa6f40d6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:31.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:31.777+0000 7fa6f9600640 1 -- 172.21.15.27:0/3621852265 >> 172.21.15.27:0/3621852265 conn(0x7fa6f40b93c0 msgr2=0x7fa6f40b8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:31.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:31.781+0000 7fa6f9600640 1 -- 172.21.15.27:0/3621852265 shutdown_connections 2024-01-15T18:15:31.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:31.781+0000 7fa6f9600640 1 -- 172.21.15.27:0/3621852265 wait complete. 2024-01-15T18:15:31.813 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T18:15:31.813 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:15:31.875 DEBUG:teuthology.orchestra.run.smithi093:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:15:31.960 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T18:15:31.960 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:15:31.960 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:15:31.969 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:15:32.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.149+0000 7fd8e6400640 1 Processor -- start 2024-01-15T18:15:32.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.149+0000 7fd8e6400640 1 -- start start 2024-01-15T18:15:32.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.153+0000 7fd8e6400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8e00701a0 0x7fd8e0070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:32.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.153+0000 7fd8e6400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e006c4a0 con 0x7fd8e00c92b0 2024-01-15T18:15:32.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.153+0000 7fd8e6400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e0070ac0 con 0x7fd8e015c130 2024-01-15T18:15:32.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.153+0000 7fd8e6400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e015c660 con 0x7fd8e00701a0 2024-01-15T18:15:32.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.153+0000 7fd8dea00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd8e00c92b0 0x7fd8e006dd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49216/0 (socket says 172.21.15.27:49216) 2024-01-15T18:15:32.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.153+0000 7fd8dea00640 1 -- 172.21.15.27:0/4007049351 learned_addr learned my addr 172.21.15.27:0/4007049351 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:32.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.153+0000 7fd8df400640 1 --2- 172.21.15.27:0/4007049351 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8e00701a0 0x7fd8e0070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:32.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.153+0000 7fd8de000640 1 -- 172.21.15.27:0/4007049351 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105573447 0 0) 0x7fd8e006c4a0 con 0x7fd8e00c92b0 2024-01-15T18:15:32.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.153+0000 7fd8de000640 1 -- 172.21.15.27:0/4007049351 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8c0003530 con 0x7fd8e00c92b0 2024-01-15T18:15:32.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.153+0000 7fd8de000640 1 -- 172.21.15.27:0/4007049351 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3165873838 0 0) 0x7fd8e0070ac0 con 0x7fd8e015c130 2024-01-15T18:15:32.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.153+0000 7fd8de000640 1 -- 172.21.15.27:0/4007049351 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8e006c4a0 con 0x7fd8e015c130 2024-01-15T18:15:32.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.153+0000 7fd8df400640 1 -- 172.21.15.27:0/4007049351 >> v1:172.21.15.93:6789/0 conn(0x7fd8e015c130 legacy=0x7fd8e015d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:32.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.153+0000 7fd8df400640 1 -- 172.21.15.27:0/4007049351 >> v1:172.21.15.27:6789/0 conn(0x7fd8e00c92b0 legacy=0x7fd8e006dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:32.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.153+0000 7fd8de000640 1 -- 172.21.15.27:0/4007049351 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2534898023 0 0) 0x7fd8c0003530 con 0x7fd8e00c92b0 2024-01-15T18:15:32.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.153+0000 7fd8df400640 1 -- 172.21.15.27:0/4007049351 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e015c940 con 0x7fd8e00701a0 2024-01-15T18:15:32.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.157+0000 7fd8df400640 1 --2- 172.21.15.27:0/4007049351 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8e00701a0 0x7fd8e0070580 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd8d00097f0 tx=0x7fd8d001c3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=baae430822af86a3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:32.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.157+0000 7fd8de000640 1 -- 172.21.15.27:0/4007049351 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c8001680 con 0x7fd8e00701a0 2024-01-15T18:15:32.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.157+0000 7fd8de000640 1 -- 172.21.15.27:0/4007049351 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d00a7050 con 0x7fd8e00701a0 2024-01-15T18:15:32.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.157+0000 7fd8de000640 1 -- 172.21.15.27:0/4007049351 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d00ab620 con 0x7fd8e00701a0 2024-01-15T18:15:32.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.157+0000 7fd8e6400640 1 -- 172.21.15.27:0/4007049351 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8e00701a0 msgr2=0x7fd8e0070580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:32.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.157+0000 7fd8e6400640 1 --2- 172.21.15.27:0/4007049351 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8e00701a0 0x7fd8e0070580 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd8d00097f0 tx=0x7fd8d001c3b0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.157+0000 7fd8e6400640 1 -- 172.21.15.27:0/4007049351 shutdown_connections 2024-01-15T18:15:32.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.157+0000 7fd8e6400640 1 --2- 172.21.15.27:0/4007049351 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8e00701a0 0x7fd8e0070580 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.157+0000 7fd8e6400640 1 -- 172.21.15.27:0/4007049351 >> 172.21.15.27:0/4007049351 conn(0x7fd8e00cec60 msgr2=0x7fd8e00cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:32.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.157+0000 7fd8e6400640 1 -- 172.21.15.27:0/4007049351 shutdown_connections 2024-01-15T18:15:32.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.157+0000 7fd8e6400640 1 -- 172.21.15.27:0/4007049351 wait complete. 2024-01-15T18:15:32.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.157+0000 7fd8e6400640 1 Processor -- start 2024-01-15T18:15:32.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.157+0000 7fd8e6400640 1 -- start start 2024-01-15T18:15:32.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.161+0000 7fd8e6400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd8e00701a0 0x7fd8e00c5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:32.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.161+0000 7fd8e6400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8e00c92b0 0x7fd8e00c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:32.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.161+0000 7fd8e6400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8e015c130 0x7fd8e0140e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:32.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.161+0000 7fd8e6400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e00c4bc0 con 0x7fd8e015c130 2024-01-15T18:15:32.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.161+0000 7fd8e6400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e0153750 con 0x7fd8e00701a0 2024-01-15T18:15:32.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.161+0000 7fd8e6400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e00c5590 con 0x7fd8e00c92b0 2024-01-15T18:15:32.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.161+0000 7fd8dfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8e015c130 0x7fd8e0140e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:32.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.161+0000 7fd8df400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd8e00701a0 0x7fd8e00c5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:32.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.161+0000 7fd8dfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8e015c130 0x7fd8e0140e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52522/0 (socket says 172.21.15.27:52522) 2024-01-15T18:15:32.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.161+0000 7fd8dfe00640 1 -- 172.21.15.27:0/3440473341 learned_addr learned my addr 172.21.15.27:0/3440473341 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:32.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.161+0000 7fd8dea00640 1 --2- 172.21.15.27:0/3440473341 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8e00c92b0 0x7fd8e00c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:32.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.161+0000 7fd8df400640 1 -- 172.21.15.27:0/3440473341 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8e00c92b0 msgr2=0x7fd8e00c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:32.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.161+0000 7fd8df400640 1 --2- 172.21.15.27:0/3440473341 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8e00c92b0 0x7fd8e00c7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.161+0000 7fd8df400640 1 -- 172.21.15.27:0/3440473341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8e015c130 msgr2=0x7fd8e0140e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:32.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.161+0000 7fd8df400640 1 --2- 172.21.15.27:0/3440473341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8e015c130 0x7fd8e0140e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.161+0000 7fd8df400640 1 -- 172.21.15.27:0/3440473341 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d0009420 con 0x7fd8e00701a0 2024-01-15T18:15:32.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.161+0000 7fd8df400640 1 --2- 172.21.15.27:0/3440473341 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd8e00701a0 0x7fd8e00c5050 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fd8d00ab000 tx=0x7fd8d0019150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:32.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.161+0000 7fd8d7400640 1 -- 172.21.15.27:0/3440473341 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d00b4410 con 0x7fd8e00701a0 2024-01-15T18:15:32.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.161+0000 7fd8d7400640 1 -- 172.21.15.27:0/3440473341 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d00b4570 con 0x7fd8e00701a0 2024-01-15T18:15:32.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.161+0000 7fd8e6400640 1 -- 172.21.15.27:0/3440473341 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e01419f0 con 0x7fd8e00701a0 2024-01-15T18:15:32.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.161+0000 7fd8d7400640 1 -- 172.21.15.27:0/3440473341 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d00b48e0 con 0x7fd8e00701a0 2024-01-15T18:15:32.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.161+0000 7fd8e6400640 1 -- 172.21.15.27:0/3440473341 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e00c4850 con 0x7fd8e00701a0 2024-01-15T18:15:32.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.161+0000 7fd8e6400640 1 -- 172.21.15.27:0/3440473341 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e004e850 con 0x7fd8e00701a0 2024-01-15T18:15:32.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.161+0000 7fd8d7400640 1 -- 172.21.15.27:0/3440473341 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7fd8d00b2040 con 0x7fd8e00701a0 2024-01-15T18:15:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.165+0000 7fd8d7400640 1 --2- 172.21.15.27:0/3440473341 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd8b80674c0 0x7fd8b8069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.165+0000 7fd8d7400640 1 -- 172.21.15.27:0/3440473341 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5027+0+0 (secure 0 0 0) 0x7fd8d011cfb0 con 0x7fd8e00701a0 2024-01-15T18:15:32.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.165+0000 7fd8dea00640 1 --2- 172.21.15.27:0/3440473341 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd8b80674c0 0x7fd8b8069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:32.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.165+0000 7fd8dea00640 1 --2- 172.21.15.27:0/3440473341 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd8b80674c0 0x7fd8b8069980 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00ca860 tx=0x7fd8c8001a60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:32.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.169+0000 7fd8d7400640 1 -- 172.21.15.27:0/3440473341 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd8d00eba90 con 0x7fd8e00701a0 2024-01-15T18:15:32.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.325+0000 7fd8e6400640 1 -- 172.21.15.27:0/3440473341 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8e00c1d40 con 0x7fd8e00701a0 2024-01-15T18:15:32.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.329+0000 7fd8d7400640 1 -- 172.21.15.27:0/3440473341 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+13023 (secure 0 0 0) 0x7fd8d00ef0b0 con 0x7fd8e00701a0 2024-01-15T18:15:32.329 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:15:32.329 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":76,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:15:31.734298+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6835/2027824196":"2024-01-16T18:15:28.727053+0000","172.21.15.27:6834/2027824196":"2024-01-16T18:15:28.727053+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:15:32.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.329+0000 7fd8e6400640 1 -- 172.21.15.27:0/3440473341 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd8b80674c0 msgr2=0x7fd8b8069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:32.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.329+0000 7fd8e6400640 1 --2- 172.21.15.27:0/3440473341 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd8b80674c0 0x7fd8b8069980 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00ca860 tx=0x7fd8c8001a60 comp rx=0 tx=0).stop 2024-01-15T18:15:32.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.329+0000 7fd8e6400640 1 -- 172.21.15.27:0/3440473341 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd8e00701a0 msgr2=0x7fd8e00c5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:32.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.329+0000 7fd8e6400640 1 --2- 172.21.15.27:0/3440473341 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd8e00701a0 0x7fd8e00c5050 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fd8d00ab000 tx=0x7fd8d0019150 comp rx=0 tx=0).stop 2024-01-15T18:15:32.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.329+0000 7fd8e6400640 1 -- 172.21.15.27:0/3440473341 shutdown_connections 2024-01-15T18:15:32.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.329+0000 7fd8e6400640 1 --2- 172.21.15.27:0/3440473341 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd8b80674c0 0x7fd8b8069980 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.329+0000 7fd8e6400640 1 --2- 172.21.15.27:0/3440473341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8e015c130 0x7fd8e0140e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.329+0000 7fd8e6400640 1 --2- 172.21.15.27:0/3440473341 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd8e00c92b0 0x7fd8e00c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.329+0000 7fd8e6400640 1 --2- 172.21.15.27:0/3440473341 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd8e00701a0 0x7fd8e00c5050 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.329+0000 7fd8e6400640 1 -- 172.21.15.27:0/3440473341 >> 172.21.15.27:0/3440473341 conn(0x7fd8e00cec60 msgr2=0x7fd8e015b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:32.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.329+0000 7fd8e6400640 1 -- 172.21.15.27:0/3440473341 shutdown_connections 2024-01-15T18:15:32.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.329+0000 7fd8e6400640 1 -- 172.21.15.27:0/3440473341 wait complete. 2024-01-15T18:15:32.351 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:15:32.352 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:15:32.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.509+0000 7f7794000640 1 Processor -- start 2024-01-15T18:15:32.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.513+0000 7f7794000640 1 -- start start 2024-01-15T18:15:32.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.513+0000 7f7794000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f778c070c50 0x7f778c06dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:32.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.513+0000 7f7794000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f778c0abe90 0x7f778c06d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:32.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.513+0000 7f7794000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f778c0b1eb0 con 0x7f778c15d5b0 2024-01-15T18:15:32.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.513+0000 7f7794000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f778c0b27c0 con 0x7f778c0abe90 2024-01-15T18:15:32.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.513+0000 7f7794000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f778c0728c0 con 0x7f778c070c50 2024-01-15T18:15:32.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.513+0000 7f7791600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f778c15d5b0 0x7f778c070540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49232/0 (socket says 172.21.15.27:49232) 2024-01-15T18:15:32.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.513+0000 7f7791600640 1 -- 172.21.15.27:0/1762069565 learned_addr learned my addr 172.21.15.27:0/1762069565 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:32.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.513+0000 7f7792000640 1 --2- 172.21.15.27:0/1762069565 >> v2:172.21.15.93:3300/0 conn(0x7f778c0abe90 0x7f778c06d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:32.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.513+0000 7f7790c00640 1 --2- 172.21.15.27:0/1762069565 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f778c070c50 0x7f778c06dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:32.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.517+0000 7f7790c00640 1 -- 172.21.15.27:0/1762069565 >> v2:172.21.15.93:3300/0 conn(0x7f778c0abe90 msgr2=0x7f778c06d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:32.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.517+0000 7f7790c00640 1 --2- 172.21.15.27:0/1762069565 >> v2:172.21.15.93:3300/0 conn(0x7f778c0abe90 0x7f778c06d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.517+0000 7f7790c00640 1 -- 172.21.15.27:0/1762069565 >> v1:172.21.15.27:6789/0 conn(0x7f778c15d5b0 legacy=0x7f778c070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:32.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.517+0000 7f7790c00640 1 -- 172.21.15.27:0/1762069565 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778c0ae0e0 con 0x7f778c070c50 2024-01-15T18:15:32.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.521+0000 7f7790c00640 1 --2- 172.21.15.27:0/1762069565 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f778c070c50 0x7f778c06dfe0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f77740097c0 tx=0x7f777401c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba140a5e0b1be53b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:32.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.521+0000 7f778be00640 1 -- 172.21.15.27:0/1762069565 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77740ab600 con 0x7f778c070c50 2024-01-15T18:15:32.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.521+0000 7f778be00640 1 -- 172.21.15.27:0/1762069565 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77740a7050 con 0x7f778c070c50 2024-01-15T18:15:32.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.521+0000 7f7794000640 1 -- 172.21.15.27:0/1762069565 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f778c070c50 msgr2=0x7f778c06dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:32.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.521+0000 7f7794000640 1 --2- 172.21.15.27:0/1762069565 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f778c070c50 0x7f778c06dfe0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f77740097c0 tx=0x7f777401c3f0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.521+0000 7f7794000640 1 -- 172.21.15.27:0/1762069565 shutdown_connections 2024-01-15T18:15:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.521+0000 7f7794000640 1 --2- 172.21.15.27:0/1762069565 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f778c070c50 0x7f778c06dfe0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.521+0000 7f7794000640 1 --2- 172.21.15.27:0/1762069565 >> v2:172.21.15.93:3300/0 conn(0x7f778c0abe90 0x7f778c06d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.521+0000 7f7794000640 1 -- 172.21.15.27:0/1762069565 >> 172.21.15.27:0/1762069565 conn(0x7f778c0b0bb0 msgr2=0x7f778c0b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.525+0000 7f7794000640 1 -- 172.21.15.27:0/1762069565 shutdown_connections 2024-01-15T18:15:32.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.525+0000 7f7794000640 1 -- 172.21.15.27:0/1762069565 wait complete. 2024-01-15T18:15:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.525+0000 7f7794000640 1 Processor -- start 2024-01-15T18:15:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.529+0000 7f7794000640 1 -- start start 2024-01-15T18:15:32.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.529+0000 7f7794000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f778c070c50 0x7f778c1544b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.529+0000 7f7794000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f778c0abe90 0x7f778c15fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.529+0000 7f7794000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f778c15d5b0 0x7f778c160120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.529+0000 7f7794000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778c14d660 con 0x7f778c15d5b0 2024-01-15T18:15:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.529+0000 7f7794000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778c14c700 con 0x7f778c0abe90 2024-01-15T18:15:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.529+0000 7f7794000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f778c14c9e0 con 0x7f778c070c50 2024-01-15T18:15:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.529+0000 7f7792000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f778c15d5b0 0x7f778c160120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:32.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.529+0000 7f7792000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f778c15d5b0 0x7f778c160120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52538/0 (socket says 172.21.15.27:52538) 2024-01-15T18:15:32.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.529+0000 7f7792000640 1 -- 172.21.15.27:0/3596629313 learned_addr learned my addr 172.21.15.27:0/3596629313 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:32.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.529+0000 7f7790c00640 1 --2- 172.21.15.27:0/3596629313 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f778c0abe90 0x7f778c15fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:32.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.529+0000 7f7791600640 1 --2- 172.21.15.27:0/3596629313 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f778c070c50 0x7f778c1544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.529+0000 7f7792000640 1 -- 172.21.15.27:0/3596629313 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f778c070c50 msgr2=0x7f778c1544b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.529+0000 7f7792000640 1 --2- 172.21.15.27:0/3596629313 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f778c070c50 0x7f778c1544b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.529+0000 7f7792000640 1 -- 172.21.15.27:0/3596629313 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f778c0abe90 msgr2=0x7f778c15fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.529+0000 7f7792000640 1 --2- 172.21.15.27:0/3596629313 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f778c0abe90 0x7f778c15fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.529+0000 7f7792000640 1 -- 172.21.15.27:0/3596629313 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7774009420 con 0x7f778c15d5b0 2024-01-15T18:15:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.529+0000 7f7790c00640 1 --2- 172.21.15.27:0/3596629313 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f778c0abe90 0x7f778c15fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:15:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.529+0000 7f7792000640 1 --2- 172.21.15.27:0/3596629313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f778c15d5b0 0x7f778c160120 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f777c00e7c0 tx=0x7f777c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.529+0000 7f7789600640 1 -- 172.21.15.27:0/3596629313 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c00eed0 con 0x7f778c15d5b0 2024-01-15T18:15:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.529+0000 7f7789600640 1 -- 172.21.15.27:0/3596629313 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f777c00c940 con 0x7f778c15d5b0 2024-01-15T18:15:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.529+0000 7f7789600640 1 -- 172.21.15.27:0/3596629313 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c00ccd0 con 0x7f778c15d5b0 2024-01-15T18:15:32.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.529+0000 7f7794000640 1 -- 172.21.15.27:0/3596629313 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f778c14ccc0 con 0x7f778c15d5b0 2024-01-15T18:15:32.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.529+0000 7f7794000640 1 -- 172.21.15.27:0/3596629313 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f778c14d280 con 0x7f778c15d5b0 2024-01-15T18:15:32.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.533+0000 7f7794000640 1 -- 172.21.15.27:0/3596629313 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f775c005260 con 0x7f778c15d5b0 2024-01-15T18:15:32.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.533+0000 7f7789600640 1 -- 172.21.15.27:0/3596629313 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7f777c013470 con 0x7f778c15d5b0 2024-01-15T18:15:32.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.533+0000 7f7789600640 1 --2- 172.21.15.27:0/3596629313 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7764067510 0x7f77640699d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:32.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.533+0000 7f7789600640 1 -- 172.21.15.27:0/3596629313 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f777c0830f0 con 0x7f778c15d5b0 2024-01-15T18:15:32.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.533+0000 7f7791600640 1 --2- 172.21.15.27:0/3596629313 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7764067510 0x7f77640699d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:32.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.541+0000 7f7791600640 1 --2- 172.21.15.27:0/3596629313 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7764067510 0x7f77640699d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f7784005c10 tx=0x7f7784009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:32.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.545+0000 7f7789600640 1 -- 172.21.15.27:0/3596629313 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f777c051bd0 con 0x7f778c15d5b0 2024-01-15T18:15:32.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.725+0000 7f7794000640 1 -- 172.21.15.27:0/3596629313 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f775c005a90 con 0x7f778c15d5b0 2024-01-15T18:15:32.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.725+0000 7f7789600640 1 -- 172.21.15.27:0/3596629313 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7f777c0210a0 con 0x7f778c15d5b0 2024-01-15T18:15:32.725 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:15:32.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.725+0000 7f7794000640 1 -- 172.21.15.27:0/3596629313 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7764067510 msgr2=0x7f77640699d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:32.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.725+0000 7f7794000640 1 --2- 172.21.15.27:0/3596629313 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7764067510 0x7f77640699d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f7784005c10 tx=0x7f7784009040 comp rx=0 tx=0).stop 2024-01-15T18:15:32.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.725+0000 7f7794000640 1 -- 172.21.15.27:0/3596629313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f778c15d5b0 msgr2=0x7f778c160120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:32.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.725+0000 7f7794000640 1 --2- 172.21.15.27:0/3596629313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f778c15d5b0 0x7f778c160120 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f777c00e7c0 tx=0x7f777c00ec80 comp rx=0 tx=0).stop 2024-01-15T18:15:32.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.725+0000 7f7794000640 1 -- 172.21.15.27:0/3596629313 shutdown_connections 2024-01-15T18:15:32.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.725+0000 7f7794000640 1 --2- 172.21.15.27:0/3596629313 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7764067510 0x7f77640699d0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.725+0000 7f7794000640 1 --2- 172.21.15.27:0/3596629313 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f778c15d5b0 0x7f778c160120 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.725+0000 7f7794000640 1 --2- 172.21.15.27:0/3596629313 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f778c0abe90 0x7f778c15fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.725+0000 7f7794000640 1 --2- 172.21.15.27:0/3596629313 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f778c070c50 0x7f778c1544b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.725+0000 7f7794000640 1 -- 172.21.15.27:0/3596629313 >> 172.21.15.27:0/3596629313 conn(0x7f778c0b0bb0 msgr2=0x7f778c0b0010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:32.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.725+0000 7f7794000640 1 -- 172.21.15.27:0/3596629313 shutdown_connections 2024-01-15T18:15:32.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.725+0000 7f7794000640 1 -- 172.21.15.27:0/3596629313 wait complete. 2024-01-15T18:15:32.747 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:15:32.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.933+0000 7fc242c00640 1 Processor -- start 2024-01-15T18:15:32.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.933+0000 7fc242c00640 1 -- start start 2024-01-15T18:15:32.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.933+0000 7fc242c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fc23c15e7d0 0x7fc23c099a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:32.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.933+0000 7fc242c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc23c0b35a0 0x7fc23c099f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:32.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.933+0000 7fc242c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc23c0afe10 0x7fc23c15efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:32.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.933+0000 7fc242c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc23c0cd870 con 0x7fc23c0afe10 2024-01-15T18:15:32.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.933+0000 7fc242c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc23c09a700 con 0x7fc23c15e7d0 2024-01-15T18:15:32.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.933+0000 7fc242c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c0cd590 con 0x7fc23c0b35a0 2024-01-15T18:15:32.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.933+0000 7fc240c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc23c0afe10 0x7fc23c15efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:32.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.933+0000 7fc240c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc23c0afe10 0x7fc23c15efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52550/0 (socket says 172.21.15.27:52550) 2024-01-15T18:15:32.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.933+0000 7fc240c00640 1 -- 172.21.15.27:0/3069344844 learned_addr learned my addr 172.21.15.27:0/3069344844 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:32.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.933+0000 7fc23b400640 1 --2- 172.21.15.27:0/3069344844 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc23c0b35a0 0x7fc23c099f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:32.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.933+0000 7fc23be00640 1 --2- 172.21.15.27:0/3069344844 >> v2:172.21.15.93:3300/0 conn(0x7fc23c15e7d0 0x7fc23c099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:32.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.933+0000 7fc240c00640 1 -- 172.21.15.27:0/3069344844 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc23c0b35a0 msgr2=0x7fc23c099f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:32.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.933+0000 7fc240c00640 1 --2- 172.21.15.27:0/3069344844 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc23c0b35a0 0x7fc23c099f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.933+0000 7fc240c00640 1 -- 172.21.15.27:0/3069344844 >> v2:172.21.15.93:3300/0 conn(0x7fc23c15e7d0 msgr2=0x7fc23c099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:32.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.933+0000 7fc240c00640 1 --2- 172.21.15.27:0/3069344844 >> v2:172.21.15.93:3300/0 conn(0x7fc23c15e7d0 0x7fc23c099a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.933+0000 7fc240c00640 1 -- 172.21.15.27:0/3069344844 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc23c0cfa00 con 0x7fc23c0afe10 2024-01-15T18:15:32.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.937+0000 7fc240c00640 1 --2- 172.21.15.27:0/3069344844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc23c0afe10 0x7fc23c15efd0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fc23c071eb0 tx=0x7fc23401c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb493047b6dfec2d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:32.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.937+0000 7fc23aa00640 1 -- 172.21.15.27:0/3069344844 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2340a5580 con 0x7fc23c0afe10 2024-01-15T18:15:32.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.937+0000 7fc23aa00640 1 -- 172.21.15.27:0/3069344844 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2340a56e0 con 0x7fc23c0afe10 2024-01-15T18:15:32.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.937+0000 7fc23aa00640 1 -- 172.21.15.27:0/3069344844 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2340a5a70 con 0x7fc23c0afe10 2024-01-15T18:15:32.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.937+0000 7fc242c00640 1 -- 172.21.15.27:0/3069344844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc23c0afe10 msgr2=0x7fc23c15efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:32.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.937+0000 7fc242c00640 1 --2- 172.21.15.27:0/3069344844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc23c0afe10 0x7fc23c15efd0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fc23c071eb0 tx=0x7fc23401c6e0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.937+0000 7fc242c00640 1 -- 172.21.15.27:0/3069344844 shutdown_connections 2024-01-15T18:15:32.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.937+0000 7fc242c00640 1 --2- 172.21.15.27:0/3069344844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc23c0afe10 0x7fc23c15efd0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.937+0000 7fc242c00640 1 --2- 172.21.15.27:0/3069344844 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc23c0b35a0 0x7fc23c099f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.937+0000 7fc242c00640 1 --2- 172.21.15.27:0/3069344844 >> v2:172.21.15.93:3300/0 conn(0x7fc23c15e7d0 0x7fc23c099a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.937+0000 7fc242c00640 1 -- 172.21.15.27:0/3069344844 >> 172.21.15.27:0/3069344844 conn(0x7fc23c0b2a80 msgr2=0x7fc23c0b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:32.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.937+0000 7fc242c00640 1 -- 172.21.15.27:0/3069344844 shutdown_connections 2024-01-15T18:15:32.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.937+0000 7fc242c00640 1 -- 172.21.15.27:0/3069344844 wait complete. 2024-01-15T18:15:32.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.941+0000 7fc242c00640 1 Processor -- start 2024-01-15T18:15:32.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.941+0000 7fc242c00640 1 -- start start 2024-01-15T18:15:32.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.941+0000 7fc242c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc23c0afe10 0x7fc23c1f5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:32.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.941+0000 7fc242c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc23c0b35a0 0x7fc23c156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:32.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.941+0000 7fc242c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc23c15e7d0 0x7fc23c0af080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:32.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.941+0000 7fc242c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c06c4b0 con 0x7fc23c0b35a0 2024-01-15T18:15:32.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.941+0000 7fc242c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c06c940 con 0x7fc23c0afe10 2024-01-15T18:15:32.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.941+0000 7fc242c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c1f5c90 con 0x7fc23c15e7d0 2024-01-15T18:15:32.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.941+0000 7fc23b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc23c0b35a0 0x7fc23c156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:32.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.941+0000 7fc23b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc23c0b35a0 0x7fc23c156320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52562/0 (socket says 172.21.15.27:52562) 2024-01-15T18:15:32.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.941+0000 7fc23b400640 1 -- 172.21.15.27:0/1050219896 learned_addr learned my addr 172.21.15.27:0/1050219896 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:32.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.941+0000 7fc23be00640 1 --2- 172.21.15.27:0/1050219896 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc23c0afe10 0x7fc23c1f5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:32.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.941+0000 7fc240c00640 1 --2- 172.21.15.27:0/1050219896 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc23c15e7d0 0x7fc23c0af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:32.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.941+0000 7fc23b400640 1 -- 172.21.15.27:0/1050219896 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc23c15e7d0 msgr2=0x7fc23c0af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:32.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.941+0000 7fc23b400640 1 --2- 172.21.15.27:0/1050219896 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc23c15e7d0 0x7fc23c0af080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.941+0000 7fc23b400640 1 -- 172.21.15.27:0/1050219896 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc23c0afe10 msgr2=0x7fc23c1f5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:32.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.941+0000 7fc23b400640 1 --2- 172.21.15.27:0/1050219896 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc23c0afe10 0x7fc23c1f5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:32.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.941+0000 7fc23b400640 1 -- 172.21.15.27:0/1050219896 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc234009b30 con 0x7fc23c0b35a0 2024-01-15T18:15:32.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.941+0000 7fc23b400640 1 --2- 172.21.15.27:0/1050219896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc23c0b35a0 0x7fc23c156320 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fc22400e8b0 tx=0x7fc22400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:32.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.941+0000 7fc233e00640 1 -- 172.21.15.27:0/1050219896 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22400c940 con 0x7fc23c0b35a0 2024-01-15T18:15:32.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.941+0000 7fc233e00640 1 -- 172.21.15.27:0/1050219896 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22400caa0 con 0x7fc23c0b35a0 2024-01-15T18:15:32.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.941+0000 7fc242c00640 1 -- 172.21.15.27:0/1050219896 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc23c06b8a0 con 0x7fc23c0b35a0 2024-01-15T18:15:32.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.941+0000 7fc233e00640 1 -- 172.21.15.27:0/1050219896 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22400ce50 con 0x7fc23c0b35a0 2024-01-15T18:15:32.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.941+0000 7fc242c00640 1 -- 172.21.15.27:0/1050219896 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc23c06be60 con 0x7fc23c0b35a0 2024-01-15T18:15:32.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.945+0000 7fc242c00640 1 -- 172.21.15.27:0/1050219896 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc23c04e850 con 0x7fc23c0b35a0 2024-01-15T18:15:32.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.945+0000 7fc233e00640 1 -- 172.21.15.27:0/1050219896 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7fc224013470 con 0x7fc23c0b35a0 2024-01-15T18:15:32.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.945+0000 7fc233e00640 1 --2- 172.21.15.27:0/1050219896 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc2140674f0 0x7fc2140699b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:32.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.945+0000 7fc233e00640 1 -- 172.21.15.27:0/1050219896 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5027+0+0 (secure 0 0 0) 0x7fc2240830e0 con 0x7fc23c0b35a0 2024-01-15T18:15:32.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.945+0000 7fc23be00640 1 --2- 172.21.15.27:0/1050219896 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc2140674f0 0x7fc2140699b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:32.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.945+0000 7fc23be00640 1 --2- 172.21.15.27:0/1050219896 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc2140674f0 0x7fc2140699b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fc22c009ac0 tx=0x7fc22c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:32.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:32.949+0000 7fc233e00640 1 -- 172.21.15.27:0/1050219896 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc224051bc0 con 0x7fc23c0b35a0 2024-01-15T18:15:33.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.121+0000 7fc242c00640 1 -- 172.21.15.27:0/1050219896 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc23c0b6ed0 con 0x7fc23c0b35a0 2024-01-15T18:15:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.121+0000 7fc233e00640 1 -- 172.21.15.27:0/1050219896 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v76) v1 ==== 80+0+122 (secure 0 0 0) 0x7fc224021080 con 0x7fc23c0b35a0 2024-01-15T18:15:33.123 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6835/2027824196 2024-01-16T18:15:28.727053+0000 2024-01-15T18:15:33.123 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6834/2027824196 2024-01-16T18:15:28.727053+0000 2024-01-15T18:15:33.123 INFO:teuthology.orchestra.run.smithi027.stderr:listed 2 entries 2024-01-15T18:15:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.125+0000 7fc242c00640 1 -- 172.21.15.27:0/1050219896 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc2140674f0 msgr2=0x7fc2140699b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.125+0000 7fc242c00640 1 --2- 172.21.15.27:0/1050219896 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc2140674f0 0x7fc2140699b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fc22c009ac0 tx=0x7fc22c009190 comp rx=0 tx=0).stop 2024-01-15T18:15:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.125+0000 7fc242c00640 1 -- 172.21.15.27:0/1050219896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc23c0b35a0 msgr2=0x7fc23c156320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.125+0000 7fc242c00640 1 --2- 172.21.15.27:0/1050219896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc23c0b35a0 0x7fc23c156320 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fc22400e8b0 tx=0x7fc22400ed70 comp rx=0 tx=0).stop 2024-01-15T18:15:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.125+0000 7fc242c00640 1 -- 172.21.15.27:0/1050219896 shutdown_connections 2024-01-15T18:15:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.125+0000 7fc242c00640 1 --2- 172.21.15.27:0/1050219896 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc2140674f0 0x7fc2140699b0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.125+0000 7fc242c00640 1 --2- 172.21.15.27:0/1050219896 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc23c15e7d0 0x7fc23c0af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.125+0000 7fc242c00640 1 --2- 172.21.15.27:0/1050219896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc23c0b35a0 0x7fc23c156320 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:33.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.125+0000 7fc242c00640 1 --2- 172.21.15.27:0/1050219896 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc23c0afe10 0x7fc23c1f5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:33.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.125+0000 7fc242c00640 1 -- 172.21.15.27:0/1050219896 >> 172.21.15.27:0/1050219896 conn(0x7fc23c0b2a80 msgr2=0x7fc23c0b0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:33.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.125+0000 7fc242c00640 1 -- 172.21.15.27:0/1050219896 shutdown_connections 2024-01-15T18:15:33.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.125+0000 7fc242c00640 1 -- 172.21.15.27:0/1050219896 wait complete. 2024-01-15T18:15:33.145 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-01-15T18:15:33.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:33.145 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-15T18:15:33.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[44283]: fuse finished with error 0 and tester_r 0 2024-01-15T18:15:33.166 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T18:15:33.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:33.167 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-15T18:15:33.364 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:15:33.364 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:15:33.373 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:15:33.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.513+0000 7fd728600640 1 Processor -- start 2024-01-15T18:15:33.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.513+0000 7fd728600640 1 -- start start 2024-01-15T18:15:33.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.513+0000 7fd728600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd72015c130 0x7fd72006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:33.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.513+0000 7fd728600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd72006e240 0x7fd72015d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:33.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.513+0000 7fd728600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd72006c4a0 con 0x7fd7200701a0 2024-01-15T18:15:33.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.513+0000 7fd728600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd72015c660 con 0x7fd72015c130 2024-01-15T18:15:33.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.513+0000 7fd728600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd72015c940 con 0x7fd72006e240 2024-01-15T18:15:33.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.513+0000 7fd725200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd72015c130 0x7fd72006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:33.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.513+0000 7fd725c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd7200701a0 0x7fd720070580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:49238/0 (socket says 172.21.15.27:49238) 2024-01-15T18:15:33.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.513+0000 7fd725c00640 1 -- 172.21.15.27:0/3735435364 learned_addr learned my addr 172.21.15.27:0/3735435364 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:33.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.513+0000 7fd726600640 1 --2- 172.21.15.27:0/3735435364 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd72006e240 0x7fd72015d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:33.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.517+0000 7fd725200640 1 -- 172.21.15.27:0/3735435364 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd72006e240 msgr2=0x7fd72015d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:33.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.517+0000 7fd725200640 1 --2- 172.21.15.27:0/3735435364 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd72006e240 0x7fd72015d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:33.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.517+0000 7fd725200640 1 -- 172.21.15.27:0/3735435364 >> v1:172.21.15.27:6789/0 conn(0x7fd7200701a0 legacy=0x7fd720070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:33.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.517+0000 7fd725200640 1 -- 172.21.15.27:0/3735435364 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd72015cc20 con 0x7fd72015c130 2024-01-15T18:15:33.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.517+0000 7fd71fe00640 1 -- 172.21.15.27:0/3735435364 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135352425 0 0) 0x7fd72006c4a0 con 0x7fd7200701a0 2024-01-15T18:15:33.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.517+0000 7fd725200640 1 --2- 172.21.15.27:0/3735435364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd72015c130 0x7fd72006dd00 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fd7080097c0 tx=0x7fd70801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b8b2471a9e14a0c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:33.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.517+0000 7fd71fe00640 1 -- 172.21.15.27:0/3735435364 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7080ab650 con 0x7fd72015c130 2024-01-15T18:15:33.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.517+0000 7fd71fe00640 1 -- 172.21.15.27:0/3735435364 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7080ab7b0 con 0x7fd72015c130 2024-01-15T18:15:33.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.517+0000 7fd71fe00640 1 -- 172.21.15.27:0/3735435364 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7080abb40 con 0x7fd72015c130 2024-01-15T18:15:33.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.517+0000 7fd728600640 1 -- 172.21.15.27:0/3735435364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd72015c130 msgr2=0x7fd72006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:33.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.517+0000 7fd728600640 1 --2- 172.21.15.27:0/3735435364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd72015c130 0x7fd72006dd00 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7fd7080097c0 tx=0x7fd70801c450 comp rx=0 tx=0).stop 2024-01-15T18:15:33.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.521+0000 7fd728600640 1 -- 172.21.15.27:0/3735435364 shutdown_connections 2024-01-15T18:15:33.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.521+0000 7fd728600640 1 --2- 172.21.15.27:0/3735435364 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd72006e240 0x7fd72015d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:33.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.521+0000 7fd728600640 1 --2- 172.21.15.27:0/3735435364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd72015c130 0x7fd72006dd00 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:33.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.521+0000 7fd728600640 1 -- 172.21.15.27:0/3735435364 >> 172.21.15.27:0/3735435364 conn(0x7fd7200cec60 msgr2=0x7fd7200cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:33.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.521+0000 7fd728600640 1 -- 172.21.15.27:0/3735435364 shutdown_connections 2024-01-15T18:15:33.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.521+0000 7fd728600640 1 -- 172.21.15.27:0/3735435364 wait complete. 2024-01-15T18:15:33.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.521+0000 7fd728600640 1 Processor -- start 2024-01-15T18:15:33.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.521+0000 7fd728600640 1 -- start start 2024-01-15T18:15:33.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.521+0000 7fd728600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd72006e240 0x7fd7200b2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:33.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.521+0000 7fd728600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7200701a0 0x7fd7200c84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:33.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.521+0000 7fd728600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd72015c130 0x7fd720141a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:33.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.521+0000 7fd728600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7200b2eb0 con 0x7fd7200701a0 2024-01-15T18:15:33.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.521+0000 7fd728600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd720144a20 con 0x7fd72006e240 2024-01-15T18:15:33.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.521+0000 7fd728600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7200b30a0 con 0x7fd72015c130 2024-01-15T18:15:33.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.521+0000 7fd725200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7200701a0 0x7fd7200c84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:33.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.521+0000 7fd725200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7200701a0 0x7fd7200c84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52576/0 (socket says 172.21.15.27:52576) 2024-01-15T18:15:33.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.521+0000 7fd725200640 1 -- 172.21.15.27:0/3145427819 learned_addr learned my addr 172.21.15.27:0/3145427819 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:33.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.521+0000 7fd725c00640 1 --2- 172.21.15.27:0/3145427819 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd72006e240 0x7fd7200b2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:33.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.521+0000 7fd726600640 1 --2- 172.21.15.27:0/3145427819 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd72015c130 0x7fd720141a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:33.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.525+0000 7fd725c00640 1 -- 172.21.15.27:0/3145427819 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd72015c130 msgr2=0x7fd720141a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:33.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.525+0000 7fd725c00640 1 --2- 172.21.15.27:0/3145427819 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd72015c130 0x7fd720141a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:33.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.525+0000 7fd725c00640 1 -- 172.21.15.27:0/3145427819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7200701a0 msgr2=0x7fd7200c84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:33.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.525+0000 7fd725c00640 1 --2- 172.21.15.27:0/3145427819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7200701a0 0x7fd7200c84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:33.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.525+0000 7fd725c00640 1 -- 172.21.15.27:0/3145427819 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd708009420 con 0x7fd72006e240 2024-01-15T18:15:33.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.525+0000 7fd725c00640 1 --2- 172.21.15.27:0/3145427819 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd72006e240 0x7fd7200b2970 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fd71000c460 tx=0x7fd71000c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:33.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.525+0000 7fd71d600640 1 -- 172.21.15.27:0/3145427819 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7100074e0 con 0x7fd72006e240 2024-01-15T18:15:33.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.525+0000 7fd71d600640 1 -- 172.21.15.27:0/3145427819 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd710007660 con 0x7fd72006e240 2024-01-15T18:15:33.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.525+0000 7fd71d600640 1 -- 172.21.15.27:0/3145427819 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd710004a30 con 0x7fd72006e240 2024-01-15T18:15:33.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.525+0000 7fd728600640 1 -- 172.21.15.27:0/3145427819 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7200b3380 con 0x7fd72006e240 2024-01-15T18:15:33.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.525+0000 7fd728600640 1 -- 172.21.15.27:0/3145427819 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7200b3940 con 0x7fd72006e240 2024-01-15T18:15:33.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.525+0000 7fd728600640 1 -- 172.21.15.27:0/3145427819 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd72004e850 con 0x7fd72006e240 2024-01-15T18:15:33.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.525+0000 7fd71d600640 1 -- 172.21.15.27:0/3145427819 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7fd710004b90 con 0x7fd72006e240 2024-01-15T18:15:33.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.525+0000 7fd71d600640 1 --2- 172.21.15.27:0/3145427819 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd6f80674c0 0x7fd6f8069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:33.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.525+0000 7fd71d600640 1 -- 172.21.15.27:0/3145427819 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5027+0+0 (secure 0 0 0) 0x7fd7100821a0 con 0x7fd72006e240 2024-01-15T18:15:33.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.525+0000 7fd725200640 1 --2- 172.21.15.27:0/3145427819 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd6f80674c0 0x7fd6f8069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:33.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.529+0000 7fd725200640 1 --2- 172.21.15.27:0/3145427819 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd6f80674c0 0x7fd6f8069980 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fd70801c490 tx=0x7fd7080b6040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:33.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.533+0000 7fd71d600640 1 -- 172.21.15.27:0/3145427819 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd710050da0 con 0x7fd72006e240 2024-01-15T18:15:33.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.689+0000 7fd728600640 1 -- 172.21.15.27:0/3145427819 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7200accd0 con 0x7fd72006e240 2024-01-15T18:15:33.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.689+0000 7fd71d600640 1 -- 172.21.15.27:0/3145427819 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+13023 (secure 0 0 0) 0x7fd710018070 con 0x7fd72006e240 2024-01-15T18:15:33.690 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:15:33.690 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":76,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:15:31.734298+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6835/2027824196":"2024-01-16T18:15:28.727053+0000","172.21.15.27:6834/2027824196":"2024-01-16T18:15:28.727053+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:15:33.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.689+0000 7fd728600640 1 -- 172.21.15.27:0/3145427819 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd6f80674c0 msgr2=0x7fd6f8069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:33.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.689+0000 7fd728600640 1 --2- 172.21.15.27:0/3145427819 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd6f80674c0 0x7fd6f8069980 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fd70801c490 tx=0x7fd7080b6040 comp rx=0 tx=0).stop 2024-01-15T18:15:33.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.689+0000 7fd728600640 1 -- 172.21.15.27:0/3145427819 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd72006e240 msgr2=0x7fd7200b2970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:33.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.689+0000 7fd728600640 1 --2- 172.21.15.27:0/3145427819 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd72006e240 0x7fd7200b2970 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fd71000c460 tx=0x7fd71000c920 comp rx=0 tx=0).stop 2024-01-15T18:15:33.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.689+0000 7fd728600640 1 -- 172.21.15.27:0/3145427819 shutdown_connections 2024-01-15T18:15:33.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.689+0000 7fd728600640 1 --2- 172.21.15.27:0/3145427819 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd6f80674c0 0x7fd6f8069980 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:33.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.689+0000 7fd728600640 1 --2- 172.21.15.27:0/3145427819 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd72015c130 0x7fd720141a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:33.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.689+0000 7fd728600640 1 --2- 172.21.15.27:0/3145427819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7200701a0 0x7fd7200c84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:33.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.689+0000 7fd728600640 1 --2- 172.21.15.27:0/3145427819 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd72006e240 0x7fd7200b2970 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:33.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.689+0000 7fd728600640 1 -- 172.21.15.27:0/3145427819 >> 172.21.15.27:0/3145427819 conn(0x7fd7200cec60 msgr2=0x7fd720070890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:33.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.689+0000 7fd728600640 1 -- 172.21.15.27:0/3145427819 shutdown_connections 2024-01-15T18:15:33.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.689+0000 7fd728600640 1 -- 172.21.15.27:0/3145427819 wait complete. 2024-01-15T18:15:33.713 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:15:33.713 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:15:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.853+0000 7fca82e00640 1 Processor -- start 2024-01-15T18:15:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.857+0000 7fca82e00640 1 -- start start 2024-01-15T18:15:33.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.857+0000 7fca82e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fca7c158900 0x7fca7c15f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:33.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.857+0000 7fca82e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fca7c0cf2e0 0x7fca7c0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:33.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.857+0000 7fca82e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7c0b1cb0 con 0x7fca7c15bf00 2024-01-15T18:15:33.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.857+0000 7fca82e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca7c0b1f00 con 0x7fca7c158900 2024-01-15T18:15:33.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.857+0000 7fca82e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c0b21e0 con 0x7fca7c0cf2e0 2024-01-15T18:15:33.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.857+0000 7fca7be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fca7c158900 0x7fca7c15f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:33.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.857+0000 7fca7be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fca7c158900 0x7fca7c15f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44920/0 (socket says 172.21.15.27:44920) 2024-01-15T18:15:33.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.857+0000 7fca7b400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fca7c15bf00 0x7fca7c0cebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55616/0 (socket says 172.21.15.27:55616) 2024-01-15T18:15:33.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.857+0000 7fca7be00640 1 -- 172.21.15.27:0/2306097269 learned_addr learned my addr 172.21.15.27:0/2306097269 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:33.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.857+0000 7fca80e00640 1 --2- 172.21.15.27:0/2306097269 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fca7c0cf2e0 0x7fca7c0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:33.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.857+0000 7fca7be00640 1 -- 172.21.15.27:0/2306097269 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fca7c0cf2e0 msgr2=0x7fca7c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:33.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.857+0000 7fca7be00640 1 --2- 172.21.15.27:0/2306097269 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fca7c0cf2e0 0x7fca7c0701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:33.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.857+0000 7fca7be00640 1 -- 172.21.15.27:0/2306097269 >> v1:172.21.15.27:6789/0 conn(0x7fca7c15bf00 legacy=0x7fca7c0cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:33.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.857+0000 7fca7aa00640 1 -- 172.21.15.27:0/2306097269 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724001107 0 0) 0x7fca7c0b1cb0 con 0x7fca7c15bf00 2024-01-15T18:15:33.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.857+0000 7fca7be00640 1 -- 172.21.15.27:0/2306097269 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7c15ea70 con 0x7fca7c158900 2024-01-15T18:15:33.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.861+0000 7fca7be00640 1 --2- 172.21.15.27:0/2306097269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca7c158900 0x7fca7c15f550 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fca6c009870 tx=0x7fca6c01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d06922060ae96f9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:33.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.861+0000 7fca7aa00640 1 -- 172.21.15.27:0/2306097269 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c0a7070 con 0x7fca7c158900 2024-01-15T18:15:33.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.861+0000 7fca7aa00640 1 -- 172.21.15.27:0/2306097269 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca6c0ab430 con 0x7fca7c158900 2024-01-15T18:15:33.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.861+0000 7fca7aa00640 1 -- 172.21.15.27:0/2306097269 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c070f30 con 0x7fca7c158900 2024-01-15T18:15:33.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.861+0000 7fca82e00640 1 -- 172.21.15.27:0/2306097269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca7c158900 msgr2=0x7fca7c15f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:33.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.861+0000 7fca82e00640 1 --2- 172.21.15.27:0/2306097269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca7c158900 0x7fca7c15f550 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fca6c009870 tx=0x7fca6c01c540 comp rx=0 tx=0).stop 2024-01-15T18:15:33.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.861+0000 7fca82e00640 1 -- 172.21.15.27:0/2306097269 shutdown_connections 2024-01-15T18:15:33.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.861+0000 7fca82e00640 1 --2- 172.21.15.27:0/2306097269 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fca7c0cf2e0 0x7fca7c0701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:33.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.861+0000 7fca82e00640 1 --2- 172.21.15.27:0/2306097269 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca7c158900 0x7fca7c15f550 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:33.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.861+0000 7fca82e00640 1 -- 172.21.15.27:0/2306097269 >> 172.21.15.27:0/2306097269 conn(0x7fca7c0cb700 msgr2=0x7fca7c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:33.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.861+0000 7fca82e00640 1 -- 172.21.15.27:0/2306097269 shutdown_connections 2024-01-15T18:15:33.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.861+0000 7fca82e00640 1 -- 172.21.15.27:0/2306097269 wait complete. 2024-01-15T18:15:33.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.865+0000 7fca82e00640 1 Processor -- start 2024-01-15T18:15:33.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.865+0000 7fca82e00640 1 -- start start 2024-01-15T18:15:33.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.865+0000 7fca82e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca7c0cf2e0 0x7fca7c15a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:33.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.865+0000 7fca82e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fca7c158900 0x7fca7c153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:33.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.865+0000 7fca82e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fca7c15bf00 0x7fca7c143570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:33.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.865+0000 7fca82e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c15a8a0 con 0x7fca7c0cf2e0 2024-01-15T18:15:33.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.865+0000 7fca82e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c159070 con 0x7fca7c15bf00 2024-01-15T18:15:33.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.865+0000 7fca82e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c1592c0 con 0x7fca7c158900 2024-01-15T18:15:33.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.865+0000 7fca7be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca7c0cf2e0 0x7fca7c15a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:33.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.865+0000 7fca7be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca7c0cf2e0 0x7fca7c15a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44924/0 (socket says 172.21.15.27:44924) 2024-01-15T18:15:33.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.865+0000 7fca7be00640 1 -- 172.21.15.27:0/858516706 learned_addr learned my addr 172.21.15.27:0/858516706 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:33.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.865+0000 7fca7b400640 1 --2- 172.21.15.27:0/858516706 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fca7c158900 0x7fca7c153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:33.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.865+0000 7fca80e00640 1 --2- 172.21.15.27:0/858516706 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fca7c15bf00 0x7fca7c143570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:33.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.865+0000 7fca7be00640 1 -- 172.21.15.27:0/858516706 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fca7c158900 msgr2=0x7fca7c153dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:33.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.865+0000 7fca7be00640 1 --2- 172.21.15.27:0/858516706 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fca7c158900 0x7fca7c153dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:33.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.865+0000 7fca7be00640 1 -- 172.21.15.27:0/858516706 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fca7c15bf00 msgr2=0x7fca7c143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:33.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.865+0000 7fca7be00640 1 --2- 172.21.15.27:0/858516706 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fca7c15bf00 0x7fca7c143570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:33.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.865+0000 7fca7be00640 1 -- 172.21.15.27:0/858516706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca6c0094d0 con 0x7fca7c0cf2e0 2024-01-15T18:15:33.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.865+0000 7fca7be00640 1 --2- 172.21.15.27:0/858516706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca7c0cf2e0 0x7fca7c15a360 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0ad040 tx=0x7fca6c0192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:33.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.865+0000 7fca73e00640 1 -- 172.21.15.27:0/858516706 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c0b4620 con 0x7fca7c0cf2e0 2024-01-15T18:15:33.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.865+0000 7fca73e00640 1 -- 172.21.15.27:0/858516706 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca6c0b47a0 con 0x7fca7c0cf2e0 2024-01-15T18:15:33.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.865+0000 7fca82e00640 1 -- 172.21.15.27:0/858516706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca7c1595a0 con 0x7fca7c0cf2e0 2024-01-15T18:15:33.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.865+0000 7fca73e00640 1 -- 172.21.15.27:0/858516706 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c0b4b50 con 0x7fca7c0cf2e0 2024-01-15T18:15:33.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.865+0000 7fca82e00640 1 -- 172.21.15.27:0/858516706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca7c159b60 con 0x7fca7c0cf2e0 2024-01-15T18:15:33.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.865+0000 7fca82e00640 1 -- 172.21.15.27:0/858516706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca7c04e850 con 0x7fca7c0cf2e0 2024-01-15T18:15:33.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.869+0000 7fca73e00640 1 -- 172.21.15.27:0/858516706 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7fca6c0b4cb0 con 0x7fca7c0cf2e0 2024-01-15T18:15:33.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.869+0000 7fca73e00640 1 --2- 172.21.15.27:0/858516706 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fca540674c0 0x7fca54069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:33.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.869+0000 7fca73e00640 1 -- 172.21.15.27:0/858516706 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5027+0+0 (secure 0 0 0) 0x7fca6c11d420 con 0x7fca7c0cf2e0 2024-01-15T18:15:33.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.869+0000 7fca7b400640 1 --2- 172.21.15.27:0/858516706 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fca540674c0 0x7fca54069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:33.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.869+0000 7fca7b400640 1 --2- 172.21.15.27:0/858516706 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fca540674c0 0x7fca54069980 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fca7c1f9400 tx=0x7fca64007b60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:33.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:33.873+0000 7fca73e00640 1 -- 172.21.15.27:0/858516706 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fca6c0ebdd0 con 0x7fca7c0cf2e0 2024-01-15T18:15:34.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.029+0000 7fca82e00640 1 -- 172.21.15.27:0/858516706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fca7c06d540 con 0x7fca7c0cf2e0 2024-01-15T18:15:34.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.029+0000 7fca73e00640 1 -- 172.21.15.27:0/858516706 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7fca6c0ef3f0 con 0x7fca7c0cf2e0 2024-01-15T18:15:34.031 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:15:34.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.033+0000 7fca82e00640 1 -- 172.21.15.27:0/858516706 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fca540674c0 msgr2=0x7fca54069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:34.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.033+0000 7fca82e00640 1 --2- 172.21.15.27:0/858516706 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fca540674c0 0x7fca54069980 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fca7c1f9400 tx=0x7fca64007b60 comp rx=0 tx=0).stop 2024-01-15T18:15:34.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.033+0000 7fca82e00640 1 -- 172.21.15.27:0/858516706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca7c0cf2e0 msgr2=0x7fca7c15a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:34.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.033+0000 7fca82e00640 1 --2- 172.21.15.27:0/858516706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca7c0cf2e0 0x7fca7c15a360 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0ad040 tx=0x7fca6c0192e0 comp rx=0 tx=0).stop 2024-01-15T18:15:34.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.033+0000 7fca82e00640 1 -- 172.21.15.27:0/858516706 shutdown_connections 2024-01-15T18:15:34.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.033+0000 7fca82e00640 1 --2- 172.21.15.27:0/858516706 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fca540674c0 0x7fca54069980 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:34.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.033+0000 7fca82e00640 1 --2- 172.21.15.27:0/858516706 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fca7c15bf00 0x7fca7c143570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:34.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.033+0000 7fca82e00640 1 --2- 172.21.15.27:0/858516706 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fca7c158900 0x7fca7c153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:34.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.033+0000 7fca82e00640 1 --2- 172.21.15.27:0/858516706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca7c0cf2e0 0x7fca7c15a360 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:34.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.033+0000 7fca82e00640 1 -- 172.21.15.27:0/858516706 >> 172.21.15.27:0/858516706 conn(0x7fca7c0cb700 msgr2=0x7fca7c0cebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:34.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.033+0000 7fca82e00640 1 -- 172.21.15.27:0/858516706 shutdown_connections 2024-01-15T18:15:34.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.033+0000 7fca82e00640 1 -- 172.21.15.27:0/858516706 wait complete. 2024-01-15T18:15:34.054 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:15:34.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.193+0000 7fa041400640 1 Processor -- start 2024-01-15T18:15:34.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.193+0000 7fa041400640 1 -- start start 2024-01-15T18:15:34.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.197+0000 7fa041400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa03c15d9e0 0x7fa03c15a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:34.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.197+0000 7fa041400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa03c15dfa0 con 0x7fa03c0719f0 2024-01-15T18:15:34.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.197+0000 7fa041400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa03c15e280 con 0x7fa03c15b060 2024-01-15T18:15:34.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.197+0000 7fa041400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa03c15b440 con 0x7fa03c15d9e0 2024-01-15T18:15:34.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.197+0000 7fa03b200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa03c15d9e0 0x7fa03c15a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:34.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.197+0000 7fa039e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa03c0719f0 0x7fa03c15bad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55630/0 (socket says 172.21.15.27:55630) 2024-01-15T18:15:34.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.197+0000 7fa039e00640 1 -- 172.21.15.27:0/65494214 learned_addr learned my addr 172.21.15.27:0/65494214 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:34.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.197+0000 7fa03b200640 1 -- 172.21.15.27:0/65494214 >> v1:172.21.15.93:6789/0 conn(0x7fa03c15b060 legacy=0x7fa03c0712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:34.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.197+0000 7fa03b200640 1 -- 172.21.15.27:0/65494214 >> v1:172.21.15.27:6789/0 conn(0x7fa03c0719f0 legacy=0x7fa03c15bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:34.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.197+0000 7fa03b200640 1 -- 172.21.15.27:0/65494214 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa03c15e560 con 0x7fa03c15d9e0 2024-01-15T18:15:34.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.201+0000 7fa03b200640 1 --2- 172.21.15.27:0/65494214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa03c15d9e0 0x7fa03c15a3f0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fa034009d90 tx=0x7fa03401c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=156415aae6051c06 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:34.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.201+0000 7fa039400640 1 -- 172.21.15.27:0/65494214 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0340a5950 con 0x7fa03c15d9e0 2024-01-15T18:15:34.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.201+0000 7fa039400640 1 -- 172.21.15.27:0/65494214 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0340a5ab0 con 0x7fa03c15d9e0 2024-01-15T18:15:34.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.201+0000 7fa039400640 1 -- 172.21.15.27:0/65494214 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0340a5e40 con 0x7fa03c15d9e0 2024-01-15T18:15:34.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.201+0000 7fa041400640 1 -- 172.21.15.27:0/65494214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa03c15d9e0 msgr2=0x7fa03c15a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:34.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.201+0000 7fa041400640 1 --2- 172.21.15.27:0/65494214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa03c15d9e0 0x7fa03c15a3f0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fa034009d90 tx=0x7fa03401c890 comp rx=0 tx=0).stop 2024-01-15T18:15:34.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.201+0000 7fa041400640 1 -- 172.21.15.27:0/65494214 shutdown_connections 2024-01-15T18:15:34.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.201+0000 7fa041400640 1 --2- 172.21.15.27:0/65494214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa03c15d9e0 0x7fa03c15a3f0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:34.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.201+0000 7fa041400640 1 -- 172.21.15.27:0/65494214 >> 172.21.15.27:0/65494214 conn(0x7fa03c0b0850 msgr2=0x7fa03c0b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:34.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.201+0000 7fa041400640 1 -- 172.21.15.27:0/65494214 shutdown_connections 2024-01-15T18:15:34.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.201+0000 7fa041400640 1 -- 172.21.15.27:0/65494214 wait complete. 2024-01-15T18:15:34.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.201+0000 7fa041400640 1 Processor -- start 2024-01-15T18:15:34.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.201+0000 7fa041400640 1 -- start start 2024-01-15T18:15:34.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.205+0000 7fa041400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa03c0719f0 0x7fa03c0b3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:34.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.205+0000 7fa041400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa03c15b060 0x7fa03c148230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:34.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.205+0000 7fa041400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa03c15d9e0 0x7fa03c153880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:34.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.205+0000 7fa041400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03c156040 con 0x7fa03c15d9e0 2024-01-15T18:15:34.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.205+0000 7fa041400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03c156320 con 0x7fa03c15b060 2024-01-15T18:15:34.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.205+0000 7fa041400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa03c156600 con 0x7fa03c0719f0 2024-01-15T18:15:34.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.205+0000 7fa03b200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa03c15d9e0 0x7fa03c153880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:34.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.205+0000 7fa03b200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa03c15d9e0 0x7fa03c153880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44950/0 (socket says 172.21.15.27:44950) 2024-01-15T18:15:34.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.205+0000 7fa03b200640 1 -- 172.21.15.27:0/282585325 learned_addr learned my addr 172.21.15.27:0/282585325 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:34.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.205+0000 7fa03a800640 1 --2- 172.21.15.27:0/282585325 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa03c0719f0 0x7fa03c0b3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:34.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.205+0000 7fa039e00640 1 --2- 172.21.15.27:0/282585325 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa03c15b060 0x7fa03c148230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.205+0000 7fa03b200640 1 -- 172.21.15.27:0/282585325 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa03c0719f0 msgr2=0x7fa03c0b3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.205+0000 7fa03b200640 1 --2- 172.21.15.27:0/282585325 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa03c0719f0 0x7fa03c0b3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.205+0000 7fa03b200640 1 -- 172.21.15.27:0/282585325 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa03c15b060 msgr2=0x7fa03c148230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.205+0000 7fa03b200640 1 --2- 172.21.15.27:0/282585325 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa03c15b060 0x7fa03c148230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.205+0000 7fa03b200640 1 -- 172.21.15.27:0/282585325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa034009a70 con 0x7fa03c15d9e0 2024-01-15T18:15:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.205+0000 7fa03b200640 1 --2- 172.21.15.27:0/282585325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa03c15d9e0 0x7fa03c153880 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fa03401a4c0 tx=0x7fa034019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.205+0000 7fa032a00640 1 -- 172.21.15.27:0/282585325 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0340addc0 con 0x7fa03c15d9e0 2024-01-15T18:15:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.205+0000 7fa032a00640 1 -- 172.21.15.27:0/282585325 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0340b9450 con 0x7fa03c15d9e0 2024-01-15T18:15:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.205+0000 7fa041400640 1 -- 172.21.15.27:0/282585325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa03c1568e0 con 0x7fa03c15d9e0 2024-01-15T18:15:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.205+0000 7fa032a00640 1 -- 172.21.15.27:0/282585325 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0340b97e0 con 0x7fa03c15d9e0 2024-01-15T18:15:34.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.205+0000 7fa041400640 1 -- 172.21.15.27:0/282585325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa03c155cf0 con 0x7fa03c15d9e0 2024-01-15T18:15:34.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.205+0000 7fa041400640 1 -- 172.21.15.27:0/282585325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa03c156b20 con 0x7fa03c15d9e0 2024-01-15T18:15:34.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.205+0000 7fa032a00640 1 -- 172.21.15.27:0/282585325 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7fa0340b9940 con 0x7fa03c15d9e0 2024-01-15T18:15:34.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.209+0000 7fa032a00640 1 --2- 172.21.15.27:0/282585325 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0140674c0 0x7fa014069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:34.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.209+0000 7fa032a00640 1 -- 172.21.15.27:0/282585325 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5027+0+0 (secure 0 0 0) 0x7fa03411e7d0 con 0x7fa03c15d9e0 2024-01-15T18:15:34.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.209+0000 7fa03a800640 1 --2- 172.21.15.27:0/282585325 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0140674c0 0x7fa014069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.209+0000 7fa03a800640 1 --2- 172.21.15.27:0/282585325 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0140674c0 0x7fa014069980 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fa02c007fd0 tx=0x7fa02c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:34.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.213+0000 7fa032a00640 1 -- 172.21.15.27:0/282585325 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa03c156b20 con 0x7fa03c15d9e0 2024-01-15T18:15:34.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.381+0000 7fa041400640 1 -- 172.21.15.27:0/282585325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa03c156b20 con 0x7fa03c15d9e0 2024-01-15T18:15:34.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.381+0000 7fa032a00640 1 -- 172.21.15.27:0/282585325 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v76) v1 ==== 80+0+122 (secure 0 0 0) 0x7fa0340bd090 con 0x7fa03c15d9e0 2024-01-15T18:15:34.382 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6835/2027824196 2024-01-16T18:15:28.727053+0000 2024-01-15T18:15:34.382 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6834/2027824196 2024-01-16T18:15:28.727053+0000 2024-01-15T18:15:34.382 INFO:teuthology.orchestra.run.smithi027.stderr:listed 2 entries 2024-01-15T18:15:34.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.381+0000 7fa041400640 1 -- 172.21.15.27:0/282585325 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0140674c0 msgr2=0x7fa014069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:34.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.381+0000 7fa041400640 1 --2- 172.21.15.27:0/282585325 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0140674c0 0x7fa014069980 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fa02c007fd0 tx=0x7fa02c009040 comp rx=0 tx=0).stop 2024-01-15T18:15:34.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.381+0000 7fa041400640 1 -- 172.21.15.27:0/282585325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa03c15d9e0 msgr2=0x7fa03c153880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:34.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.381+0000 7fa041400640 1 --2- 172.21.15.27:0/282585325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa03c15d9e0 0x7fa03c153880 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7fa03401a4c0 tx=0x7fa034019630 comp rx=0 tx=0).stop 2024-01-15T18:15:34.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.381+0000 7fa041400640 1 -- 172.21.15.27:0/282585325 shutdown_connections 2024-01-15T18:15:34.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.381+0000 7fa041400640 1 --2- 172.21.15.27:0/282585325 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa0140674c0 0x7fa014069980 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:34.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.381+0000 7fa041400640 1 --2- 172.21.15.27:0/282585325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa03c15d9e0 0x7fa03c153880 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:34.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.381+0000 7fa041400640 1 --2- 172.21.15.27:0/282585325 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa03c15b060 0x7fa03c148230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:34.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.381+0000 7fa041400640 1 --2- 172.21.15.27:0/282585325 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa03c0719f0 0x7fa03c0b3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:34.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.381+0000 7fa041400640 1 -- 172.21.15.27:0/282585325 >> 172.21.15.27:0/282585325 conn(0x7fa03c0b0850 msgr2=0x7fa03c0733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:34.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.385+0000 7fa041400640 1 -- 172.21.15.27:0/282585325 shutdown_connections 2024-01-15T18:15:34.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.385+0000 7fa041400640 1 -- 172.21.15.27:0/282585325 wait complete. 2024-01-15T18:15:34.404 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi093.front.sepia.ceph.com... 2024-01-15T18:15:34.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:34.404 DEBUG:teuthology.orchestra.run.smithi093:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-15T18:15:34.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:ceph-fuse[34052]: fuse finished with error 0 and tester_r 0 2024-01-15T18:15:34.426 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T18:15:34.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:34.426 DEBUG:teuthology.orchestra.run.smithi093:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-15T18:15:34.600 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:15:34.600 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:15:34.609 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354580k,nr_inodes=4088645,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9249 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:15:34.610 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T18:15:34.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:34.610 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-15T18:15:34.659 INFO:teuthology.orchestra.run.smithi027.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-15T18:15:34.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:15:34.661 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:15:34.661 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:15:34.670 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354484k,nr_inodes=4088621,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8403 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:15:34.670 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T18:15:34.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:34.670 DEBUG:teuthology.orchestra.run.smithi093:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-15T18:15:34.720 INFO:teuthology.orchestra.run.smithi093.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-15T18:15:34.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:15:34.722 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:15:34.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.861+0000 7f448ae00640 1 Processor -- start 2024-01-15T18:15:34.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.861+0000 7f448ae00640 1 -- start start 2024-01-15T18:15:34.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.865+0000 7f448ae00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f448415e810 0x7f44840afc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:34.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.865+0000 7f448ae00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44840b2070 0x7f44840cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:34.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.865+0000 7f448ae00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f44840cf4f0 0x7f448415ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:34.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.865+0000 7f448ae00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44840cf9a0 con 0x7f44840cf4f0 2024-01-15T18:15:34.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.865+0000 7f448ae00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44840b18a0 con 0x7f448415e810 2024-01-15T18:15:34.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.865+0000 7f448ae00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44840b27f0 con 0x7f44840b2070 2024-01-15T18:15:34.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.865+0000 7f4488e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f44840cf4f0 0x7f448415ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:34.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.865+0000 7f4488e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f44840cf4f0 0x7f448415ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44952/0 (socket says 172.21.15.27:44952) 2024-01-15T18:15:34.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.865+0000 7f4488e00640 1 -- 172.21.15.27:0/15895566 learned_addr learned my addr 172.21.15.27:0/15895566 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:34.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.865+0000 7f4483400640 1 --2- 172.21.15.27:0/15895566 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44840b2070 0x7f44840cefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:34.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.865+0000 7f4483e00640 1 --2- 172.21.15.27:0/15895566 >> v2:172.21.15.93:3300/0 conn(0x7f448415e810 0x7f44840afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:34.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.865+0000 7f4488e00640 1 -- 172.21.15.27:0/15895566 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44840b2070 msgr2=0x7f44840cefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:34.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.865+0000 7f4488e00640 1 --2- 172.21.15.27:0/15895566 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44840b2070 0x7f44840cefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:34.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.865+0000 7f4488e00640 1 -- 172.21.15.27:0/15895566 >> v2:172.21.15.93:3300/0 conn(0x7f448415e810 msgr2=0x7f44840afc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:34.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.865+0000 7f4488e00640 1 --2- 172.21.15.27:0/15895566 >> v2:172.21.15.93:3300/0 conn(0x7f448415e810 0x7f44840afc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:34.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.865+0000 7f4488e00640 1 -- 172.21.15.27:0/15895566 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44840b2510 con 0x7f44840cf4f0 2024-01-15T18:15:34.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.865+0000 7f4483e00640 1 --2- 172.21.15.27:0/15895566 >> v2:172.21.15.93:3300/0 conn(0x7f448415e810 0x7f44840afc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:34.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.865+0000 7f4488e00640 1 --2- 172.21.15.27:0/15895566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44840cf4f0 0x7f448415ef30 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f44840d0e80 tx=0x7f447c01c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df9cc87c5ded1f59 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:34.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.865+0000 7f4482a00640 1 -- 172.21.15.27:0/15895566 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447c0a5560 con 0x7f44840cf4f0 2024-01-15T18:15:34.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.869+0000 7f4482a00640 1 -- 172.21.15.27:0/15895566 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f447c0a56c0 con 0x7f44840cf4f0 2024-01-15T18:15:34.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.869+0000 7f4482a00640 1 -- 172.21.15.27:0/15895566 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447c0a5a50 con 0x7f44840cf4f0 2024-01-15T18:15:34.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.869+0000 7f448ae00640 1 -- 172.21.15.27:0/15895566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44840cf4f0 msgr2=0x7f448415ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:34.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.869+0000 7f448ae00640 1 --2- 172.21.15.27:0/15895566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44840cf4f0 0x7f448415ef30 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f44840d0e80 tx=0x7f447c01c6e0 comp rx=0 tx=0).stop 2024-01-15T18:15:34.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.869+0000 7f448ae00640 1 -- 172.21.15.27:0/15895566 shutdown_connections 2024-01-15T18:15:34.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.869+0000 7f448ae00640 1 --2- 172.21.15.27:0/15895566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44840cf4f0 0x7f448415ef30 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:34.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.869+0000 7f448ae00640 1 --2- 172.21.15.27:0/15895566 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44840b2070 0x7f44840cefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:34.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.869+0000 7f448ae00640 1 --2- 172.21.15.27:0/15895566 >> v2:172.21.15.93:3300/0 conn(0x7f448415e810 0x7f44840afc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:34.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.869+0000 7f448ae00640 1 -- 172.21.15.27:0/15895566 >> 172.21.15.27:0/15895566 conn(0x7f44840d0f20 msgr2=0x7f44840d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:34.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.869+0000 7f448ae00640 1 -- 172.21.15.27:0/15895566 shutdown_connections 2024-01-15T18:15:34.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.869+0000 7f448ae00640 1 -- 172.21.15.27:0/15895566 wait complete. 2024-01-15T18:15:34.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.869+0000 7f448ae00640 1 Processor -- start 2024-01-15T18:15:34.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.869+0000 7f448ae00640 1 -- start start 2024-01-15T18:15:34.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.869+0000 7f448ae00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44840b2070 0x7f448414d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.869+0000 7f448ae00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44840cf4f0 0x7f44841574a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.869+0000 7f448ae00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f448415e810 0x7f44840c91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.869+0000 7f448ae00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f448414de90 con 0x7f44840cf4f0 2024-01-15T18:15:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.869+0000 7f448ae00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4484157010 con 0x7f44840b2070 2024-01-15T18:15:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.869+0000 7f448ae00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4484156400 con 0x7f448415e810 2024-01-15T18:15:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.873+0000 7f4483400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44840cf4f0 0x7f44841574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.873+0000 7f4483400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44840cf4f0 0x7f44841574a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44956/0 (socket says 172.21.15.27:44956) 2024-01-15T18:15:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.873+0000 7f4483400640 1 -- 172.21.15.27:0/1530831988 learned_addr learned my addr 172.21.15.27:0/1530831988 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.873+0000 7f4483e00640 1 --2- 172.21.15.27:0/1530831988 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44840b2070 0x7f448414d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.873+0000 7f4488e00640 1 --2- 172.21.15.27:0/1530831988 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f448415e810 0x7f44840c91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.873+0000 7f4483400640 1 -- 172.21.15.27:0/1530831988 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f448415e810 msgr2=0x7f44840c91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.873+0000 7f4483400640 1 --2- 172.21.15.27:0/1530831988 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f448415e810 0x7f44840c91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:34.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.873+0000 7f4483400640 1 -- 172.21.15.27:0/1530831988 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44840b2070 msgr2=0x7f448414d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:34.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.873+0000 7f4483400640 1 --2- 172.21.15.27:0/1530831988 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44840b2070 0x7f448414d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:34.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.873+0000 7f4483400640 1 -- 172.21.15.27:0/1530831988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f447c009b30 con 0x7f44840cf4f0 2024-01-15T18:15:34.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.873+0000 7f4483400640 1 --2- 172.21.15.27:0/1530831988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44840cf4f0 0x7f44841574a0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f446c00d800 tx=0x7f446c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:34.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.873+0000 7f447be00640 1 -- 172.21.15.27:0/1530831988 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446c0096a0 con 0x7f44840cf4f0 2024-01-15T18:15:34.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.873+0000 7f447be00640 1 -- 172.21.15.27:0/1530831988 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f446c009800 con 0x7f44840cf4f0 2024-01-15T18:15:34.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.873+0000 7f447be00640 1 -- 172.21.15.27:0/1530831988 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446c009990 con 0x7f44840cf4f0 2024-01-15T18:15:34.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.873+0000 7f448ae00640 1 -- 172.21.15.27:0/1530831988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44841566e0 con 0x7f44840cf4f0 2024-01-15T18:15:34.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.873+0000 7f448ae00640 1 -- 172.21.15.27:0/1530831988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4484156ca0 con 0x7f44840cf4f0 2024-01-15T18:15:34.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.873+0000 7f448ae00640 1 -- 172.21.15.27:0/1530831988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44840d0870 con 0x7f44840cf4f0 2024-01-15T18:15:34.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.873+0000 7f447be00640 1 -- 172.21.15.27:0/1530831988 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7f446c009af0 con 0x7f44840cf4f0 2024-01-15T18:15:34.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.877+0000 7f447be00640 1 --2- 172.21.15.27:0/1530831988 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f445c0674f0 0x7f445c0699b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:34.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.877+0000 7f447be00640 1 -- 172.21.15.27:0/1530831988 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f446c083320 con 0x7f44840cf4f0 2024-01-15T18:15:34.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.877+0000 7f4483e00640 1 --2- 172.21.15.27:0/1530831988 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f445c0674f0 0x7f445c0699b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:34.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.877+0000 7f4483e00640 1 --2- 172.21.15.27:0/1530831988 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f445c0674f0 0x7f445c0699b0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f448415f050 tx=0x7f4474008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:34.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:34.881+0000 7f447be00640 1 -- 172.21.15.27:0/1530831988 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f446c087e60 con 0x7f44840cf4f0 2024-01-15T18:15:35.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.069+0000 7f448ae00640 1 -- 172.21.15.27:0/1530831988 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4484157180 con 0x7f44840cf4f0 2024-01-15T18:15:35.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.069+0000 7f447be00640 1 -- 172.21.15.27:0/1530831988 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+3693 (secure 0 0 0) 0x7f446c018090 con 0x7f44840cf4f0 2024-01-15T18:15:35.071 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:15:35.071 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":50,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6889,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6835/914017554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":914017554},{"type":"v1","addr":"172.21.15.93:6835","nonce":914017554}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":6892,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/4099412077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6836","nonce":4099412077},{"type":"v1","addr":"172.21.15.93:6837","nonce":4099412077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":7215,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/902229728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":902229728},{"type":"v1","addr":"172.21.15.27:6833","nonce":902229728}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":8262,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3388033973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3388033973},{"type":"v1","addr":"172.21.15.27:6835","nonce":3388033973}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[]} 2024-01-15T18:15:35.072 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 50 2024-01-15T18:15:35.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.073+0000 7f448ae00640 1 -- 172.21.15.27:0/1530831988 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f445c0674f0 msgr2=0x7f445c0699b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:35.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.073+0000 7f448ae00640 1 --2- 172.21.15.27:0/1530831988 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f445c0674f0 0x7f445c0699b0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f448415f050 tx=0x7f4474008040 comp rx=0 tx=0).stop 2024-01-15T18:15:35.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.073+0000 7f448ae00640 1 -- 172.21.15.27:0/1530831988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44840cf4f0 msgr2=0x7f44841574a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:35.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.073+0000 7f448ae00640 1 --2- 172.21.15.27:0/1530831988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44840cf4f0 0x7f44841574a0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f446c00d800 tx=0x7f446c00dcc0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.073+0000 7f448ae00640 1 -- 172.21.15.27:0/1530831988 shutdown_connections 2024-01-15T18:15:35.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.073+0000 7f448ae00640 1 --2- 172.21.15.27:0/1530831988 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f445c0674f0 0x7f445c0699b0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.073+0000 7f448ae00640 1 --2- 172.21.15.27:0/1530831988 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f448415e810 0x7f44840c91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.073+0000 7f448ae00640 1 --2- 172.21.15.27:0/1530831988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44840cf4f0 0x7f44841574a0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.073+0000 7f448ae00640 1 --2- 172.21.15.27:0/1530831988 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44840b2070 0x7f448414d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.073+0000 7f448ae00640 1 -- 172.21.15.27:0/1530831988 >> 172.21.15.27:0/1530831988 conn(0x7f44840d0f20 msgr2=0x7f44840cc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:35.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.073+0000 7f448ae00640 1 -- 172.21.15.27:0/1530831988 shutdown_connections 2024-01-15T18:15:35.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.073+0000 7f448ae00640 1 -- 172.21.15.27:0/1530831988 wait complete. 2024-01-15T18:15:35.094 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-15T18:15:35.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.237+0000 7f3845200640 1 Processor -- start 2024-01-15T18:15:35.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.237+0000 7f3845200640 1 -- start start 2024-01-15T18:15:35.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.237+0000 7f3845200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38400cf100 0x7f38400cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:35.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.237+0000 7f3845200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f38400ccea0 0x7f384006c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:35.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.237+0000 7f3845200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38400ad920 con 0x7f38401468d0 2024-01-15T18:15:35.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.237+0000 7f3845200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38400ac880 con 0x7f38400ccea0 2024-01-15T18:15:35.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.237+0000 7f3845200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38400acb60 con 0x7f38400cf100 2024-01-15T18:15:35.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.237+0000 7f383e800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f38400ccea0 0x7f384006c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:35.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.237+0000 7f383de00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f38401468d0 0x7f38400ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55640/0 (socket says 172.21.15.27:55640) 2024-01-15T18:15:35.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.237+0000 7f383de00640 1 -- 172.21.15.27:0/2415865893 learned_addr learned my addr 172.21.15.27:0/2415865893 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:35.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.237+0000 7f383d400640 1 --2- 172.21.15.27:0/2415865893 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38400cf100 0x7f38400cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:35.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.237+0000 7f383e800640 1 -- 172.21.15.27:0/2415865893 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38400cf100 msgr2=0x7f38400cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:35.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.237+0000 7f383e800640 1 --2- 172.21.15.27:0/2415865893 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38400cf100 0x7f38400cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.237+0000 7f383e800640 1 -- 172.21.15.27:0/2415865893 >> v1:172.21.15.27:6789/0 conn(0x7f38401468d0 legacy=0x7f38400ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:35.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.237+0000 7f383ca00640 1 -- 172.21.15.27:0/2415865893 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285256732 0 0) 0x7f3830001120 con 0x7f38401468d0 2024-01-15T18:15:35.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.237+0000 7f383e800640 1 -- 172.21.15.27:0/2415865893 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38400ace40 con 0x7f38400ccea0 2024-01-15T18:15:35.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.241+0000 7f383e800640 1 --2- 172.21.15.27:0/2415865893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38400ccea0 0x7f384006c940 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f3838009dc0 tx=0x7f383801c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89bc5171ae1ef822 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:35.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.241+0000 7f383ca00640 1 -- 172.21.15.27:0/2415865893 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38380a58d0 con 0x7f38400ccea0 2024-01-15T18:15:35.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.241+0000 7f383ca00640 1 -- 172.21.15.27:0/2415865893 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38380a5a30 con 0x7f38400ccea0 2024-01-15T18:15:35.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.241+0000 7f383ca00640 1 -- 172.21.15.27:0/2415865893 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38380a5dc0 con 0x7f38400ccea0 2024-01-15T18:15:35.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.241+0000 7f3845200640 1 -- 172.21.15.27:0/2415865893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38400ccea0 msgr2=0x7f384006c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:35.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.241+0000 7f3845200640 1 --2- 172.21.15.27:0/2415865893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38400ccea0 0x7f384006c940 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f3838009dc0 tx=0x7f383801c900 comp rx=0 tx=0).stop 2024-01-15T18:15:35.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.241+0000 7f3845200640 1 -- 172.21.15.27:0/2415865893 shutdown_connections 2024-01-15T18:15:35.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.241+0000 7f3845200640 1 --2- 172.21.15.27:0/2415865893 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38400ccea0 0x7f384006c940 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.241+0000 7f3845200640 1 --2- 172.21.15.27:0/2415865893 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38400cf100 0x7f38400cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.241+0000 7f3845200640 1 -- 172.21.15.27:0/2415865893 >> 172.21.15.27:0/2415865893 conn(0x7f38400cb700 msgr2=0x7f38400cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:35.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.241+0000 7f3845200640 1 -- 172.21.15.27:0/2415865893 shutdown_connections 2024-01-15T18:15:35.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.241+0000 7f3845200640 1 -- 172.21.15.27:0/2415865893 wait complete. 2024-01-15T18:15:35.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.245+0000 7f3845200640 1 Processor -- start 2024-01-15T18:15:35.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.245+0000 7f3845200640 1 -- start start 2024-01-15T18:15:35.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.245+0000 7f3845200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f38400ccea0 0x7f384013e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:35.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.245+0000 7f3845200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38400cf100 0x7f384013a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:35.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.245+0000 7f3845200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38401468d0 0x7f38400707e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:35.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.245+0000 7f3845200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38400af7c0 con 0x7f38400cf100 2024-01-15T18:15:35.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.245+0000 7f3845200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3840158f50 con 0x7f38400ccea0 2024-01-15T18:15:35.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.245+0000 7f3845200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f384013e910 con 0x7f38401468d0 2024-01-15T18:15:35.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.245+0000 7f383d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38400cf100 0x7f384013a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:35.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.245+0000 7f383d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38400cf100 0x7f384013a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44982/0 (socket says 172.21.15.27:44982) 2024-01-15T18:15:35.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.245+0000 7f383d400640 1 -- 172.21.15.27:0/672488546 learned_addr learned my addr 172.21.15.27:0/672488546 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:35.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.245+0000 7f383de00640 1 --2- 172.21.15.27:0/672488546 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f38400ccea0 0x7f384013e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:35.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.245+0000 7f383e800640 1 --2- 172.21.15.27:0/672488546 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38401468d0 0x7f38400707e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:35.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.245+0000 7f383d400640 1 -- 172.21.15.27:0/672488546 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38401468d0 msgr2=0x7f38400707e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:35.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.245+0000 7f383d400640 1 --2- 172.21.15.27:0/672488546 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38401468d0 0x7f38400707e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.245+0000 7f383d400640 1 -- 172.21.15.27:0/672488546 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f38400ccea0 msgr2=0x7f384013e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:35.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.245+0000 7f383d400640 1 --2- 172.21.15.27:0/672488546 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f38400ccea0 0x7f384013e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.245+0000 7f383d400640 1 -- 172.21.15.27:0/672488546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3838009a70 con 0x7f38400cf100 2024-01-15T18:15:35.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.245+0000 7f383d400640 1 --2- 172.21.15.27:0/672488546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38400cf100 0x7f384013a4c0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f382800d800 tx=0x7f382800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:35.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.245+0000 7f3836000640 1 -- 172.21.15.27:0/672488546 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382800b6c0 con 0x7f38400cf100 2024-01-15T18:15:35.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.245+0000 7f3836000640 1 -- 172.21.15.27:0/672488546 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382800b820 con 0x7f38400cf100 2024-01-15T18:15:35.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.245+0000 7f3836000640 1 -- 172.21.15.27:0/672488546 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382800bb90 con 0x7f38400cf100 2024-01-15T18:15:35.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.245+0000 7f3845200640 1 -- 172.21.15.27:0/672488546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38401591a0 con 0x7f38400cf100 2024-01-15T18:15:35.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.245+0000 7f3845200640 1 -- 172.21.15.27:0/672488546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3840159760 con 0x7f38400cf100 2024-01-15T18:15:35.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.249+0000 7f3845200640 1 -- 172.21.15.27:0/672488546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3840159920 con 0x7f38400cf100 2024-01-15T18:15:35.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.249+0000 7f3836000640 1 -- 172.21.15.27:0/672488546 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7f382800bcf0 con 0x7f38400cf100 2024-01-15T18:15:35.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.249+0000 7f3836000640 1 --2- 172.21.15.27:0/672488546 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f38180674c0 0x7f3818069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:35.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.249+0000 7f3836000640 1 -- 172.21.15.27:0/672488546 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f3828083440 con 0x7f38400cf100 2024-01-15T18:15:35.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.249+0000 7f383de00640 1 --2- 172.21.15.27:0/672488546 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f38180674c0 0x7f3818069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:35.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.253+0000 7f383de00640 1 --2- 172.21.15.27:0/672488546 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f38180674c0 0x7f3818069980 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f3830000f70 tx=0x7f3830007df0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:35.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.253+0000 7f3836000640 1 -- 172.21.15.27:0/672488546 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3828051f20 con 0x7f38400cf100 2024-01-15T18:15:35.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.417+0000 7f3845200640 1 -- 172.21.15.27:0/672488546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7f384004e850 con 0x7f38400cf100 2024-01-15T18:15:35.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.421+0000 7f3836000640 1 -- 172.21.15.27:0/672488546 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v28) v1 ==== 120+0+0 (secure 0 0 0) 0x7f3828055540 con 0x7f38400cf100 2024-01-15T18:15:35.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.421+0000 7f3845200640 1 -- 172.21.15.27:0/672488546 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f38180674c0 msgr2=0x7f3818069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:35.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.421+0000 7f3845200640 1 --2- 172.21.15.27:0/672488546 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f38180674c0 0x7f3818069980 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f3830000f70 tx=0x7f3830007df0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.421+0000 7f3845200640 1 -- 172.21.15.27:0/672488546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38400cf100 msgr2=0x7f384013a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:35.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.421+0000 7f3845200640 1 --2- 172.21.15.27:0/672488546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38400cf100 0x7f384013a4c0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f382800d800 tx=0x7f382800dcc0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.421+0000 7f3845200640 1 -- 172.21.15.27:0/672488546 shutdown_connections 2024-01-15T18:15:35.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.421+0000 7f3845200640 1 --2- 172.21.15.27:0/672488546 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f38180674c0 0x7f3818069980 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.421+0000 7f3845200640 1 --2- 172.21.15.27:0/672488546 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38401468d0 0x7f38400707e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.421+0000 7f3845200640 1 --2- 172.21.15.27:0/672488546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38400cf100 0x7f384013a4c0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.421+0000 7f3845200640 1 --2- 172.21.15.27:0/672488546 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f38400ccea0 0x7f384013e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.421+0000 7f3845200640 1 -- 172.21.15.27:0/672488546 >> 172.21.15.27:0/672488546 conn(0x7f38400cb700 msgr2=0x7f38400ce9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:35.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.425+0000 7f3845200640 1 -- 172.21.15.27:0/672488546 shutdown_connections 2024-01-15T18:15:35.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.425+0000 7f3845200640 1 -- 172.21.15.27:0/672488546 wait complete. 2024-01-15T18:15:35.445 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-15T18:15:35.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.589+0000 7f8b3ee00640 1 Processor -- start 2024-01-15T18:15:35.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.589+0000 7f8b3ee00640 1 -- start start 2024-01-15T18:15:35.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.589+0000 7f8b3ee00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8b3806c430 0x7f8b38070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:35.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.589+0000 7f8b3ee00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f8b380ce760 0x7f8b380b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:35.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.589+0000 7f8b3ee00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b380ccfb0 con 0x7f8b380b1db0 2024-01-15T18:15:35.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.589+0000 7f8b3ee00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b380cd290 con 0x7f8b3806c430 2024-01-15T18:15:35.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.589+0000 7f8b3ee00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b380cd570 con 0x7f8b380ce760 2024-01-15T18:15:35.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.589+0000 7f8b37e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8b3806c430 0x7f8b38070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:35.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.589+0000 7f8b37e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8b3806c430 0x7f8b38070ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44998/0 (socket says 172.21.15.27:44998) 2024-01-15T18:15:35.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.589+0000 7f8b37e00640 1 -- 172.21.15.27:0/1617770049 learned_addr learned my addr 172.21.15.27:0/1617770049 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:35.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.589+0000 7f8b3ce00640 1 --2- 172.21.15.27:0/1617770049 >> v2:172.21.15.93:3300/0 conn(0x7f8b380ce760 0x7f8b380b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:35.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.589+0000 7f8b37e00640 1 -- 172.21.15.27:0/1617770049 >> v2:172.21.15.93:3300/0 conn(0x7f8b380ce760 msgr2=0x7f8b380b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:35.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.589+0000 7f8b37e00640 1 --2- 172.21.15.27:0/1617770049 >> v2:172.21.15.93:3300/0 conn(0x7f8b380ce760 0x7f8b380b2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.589+0000 7f8b37e00640 1 -- 172.21.15.27:0/1617770049 >> v1:172.21.15.93:6789/0 conn(0x7f8b380b1db0 legacy=0x7f8b380ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:35.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.589+0000 7f8b37e00640 1 -- 172.21.15.27:0/1617770049 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b380cd850 con 0x7f8b3806c430 2024-01-15T18:15:35.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.593+0000 7f8b37e00640 1 --2- 172.21.15.27:0/1617770049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b3806c430 0x7f8b38070ba0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f8b28009870 tx=0x7f8b2801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb22de386fcef992 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:35.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.593+0000 7f8b36a00640 1 -- 172.21.15.27:0/1617770049 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b280ac600 con 0x7f8b3806c430 2024-01-15T18:15:35.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.593+0000 7f8b36a00640 1 -- 172.21.15.27:0/1617770049 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b280a8050 con 0x7f8b3806c430 2024-01-15T18:15:35.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.593+0000 7f8b36a00640 1 -- 172.21.15.27:0/1617770049 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b280ac980 con 0x7f8b3806c430 2024-01-15T18:15:35.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.593+0000 7f8b3ee00640 1 -- 172.21.15.27:0/1617770049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b3806c430 msgr2=0x7f8b38070ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:35.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.593+0000 7f8b3ee00640 1 --2- 172.21.15.27:0/1617770049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b3806c430 0x7f8b38070ba0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f8b28009870 tx=0x7f8b2801c400 comp rx=0 tx=0).stop 2024-01-15T18:15:35.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.593+0000 7f8b3ee00640 1 -- 172.21.15.27:0/1617770049 shutdown_connections 2024-01-15T18:15:35.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.593+0000 7f8b3ee00640 1 --2- 172.21.15.27:0/1617770049 >> v2:172.21.15.93:3300/0 conn(0x7f8b380ce760 0x7f8b380b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.593+0000 7f8b3ee00640 1 --2- 172.21.15.27:0/1617770049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b3806c430 0x7f8b38070ba0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.593+0000 7f8b3ee00640 1 -- 172.21.15.27:0/1617770049 >> 172.21.15.27:0/1617770049 conn(0x7f8b380cb700 msgr2=0x7f8b380cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:35.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.593+0000 7f8b3ee00640 1 -- 172.21.15.27:0/1617770049 shutdown_connections 2024-01-15T18:15:35.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.593+0000 7f8b3ee00640 1 -- 172.21.15.27:0/1617770049 wait complete. 2024-01-15T18:15:35.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.597+0000 7f8b3ee00640 1 Processor -- start 2024-01-15T18:15:35.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.597+0000 7f8b3ee00640 1 -- start start 2024-01-15T18:15:35.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.597+0000 7f8b3ee00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8b3806c430 0x7f8b38159c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:35.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.597+0000 7f8b3ee00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8b380b1db0 0x7f8b3814f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:35.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.597+0000 7f8b3ee00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b380ce760 0x7f8b3814fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:35.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.597+0000 7f8b3ee00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b38151ee0 con 0x7f8b380ce760 2024-01-15T18:15:35.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.597+0000 7f8b3ee00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3815a730 con 0x7f8b3806c430 2024-01-15T18:15:35.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.597+0000 7f8b3ee00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3815aa10 con 0x7f8b380b1db0 2024-01-15T18:15:35.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.597+0000 7f8b3ce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b380ce760 0x7f8b3814fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:35.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.597+0000 7f8b3ce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b380ce760 0x7f8b3814fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45010/0 (socket says 172.21.15.27:45010) 2024-01-15T18:15:35.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.597+0000 7f8b3ce00640 1 -- 172.21.15.27:0/3866460882 learned_addr learned my addr 172.21.15.27:0/3866460882 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:35.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.597+0000 7f8b37e00640 1 --2- 172.21.15.27:0/3866460882 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8b3806c430 0x7f8b38159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:35.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.597+0000 7f8b37400640 1 --2- 172.21.15.27:0/3866460882 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8b380b1db0 0x7f8b3814f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:35.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.597+0000 7f8b3ce00640 1 -- 172.21.15.27:0/3866460882 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8b380b1db0 msgr2=0x7f8b3814f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:35.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.597+0000 7f8b3ce00640 1 --2- 172.21.15.27:0/3866460882 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8b380b1db0 0x7f8b3814f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.597+0000 7f8b3ce00640 1 -- 172.21.15.27:0/3866460882 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8b3806c430 msgr2=0x7f8b38159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:35.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.597+0000 7f8b3ce00640 1 --2- 172.21.15.27:0/3866460882 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8b3806c430 0x7f8b38159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.597+0000 7f8b3ce00640 1 -- 172.21.15.27:0/3866460882 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b280094d0 con 0x7f8b380ce760 2024-01-15T18:15:35.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.597+0000 7f8b3ce00640 1 --2- 172.21.15.27:0/3866460882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b380ce760 0x7f8b3814fb80 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f8b30009fd0 tx=0x7f8b3000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:35.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.597+0000 7f8b2fe00640 1 -- 172.21.15.27:0/3866460882 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b30014420 con 0x7f8b380ce760 2024-01-15T18:15:35.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.597+0000 7f8b2fe00640 1 -- 172.21.15.27:0/3866460882 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b30014580 con 0x7f8b380ce760 2024-01-15T18:15:35.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.597+0000 7f8b2fe00640 1 -- 172.21.15.27:0/3866460882 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b30014930 con 0x7f8b380ce760 2024-01-15T18:15:35.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.597+0000 7f8b3ee00640 1 -- 172.21.15.27:0/3866460882 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b380712f0 con 0x7f8b380ce760 2024-01-15T18:15:35.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.597+0000 7f8b3ee00640 1 -- 172.21.15.27:0/3866460882 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b3815afd0 con 0x7f8b380ce760 2024-01-15T18:15:35.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.597+0000 7f8b3ee00640 1 -- 172.21.15.27:0/3866460882 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b381520b0 con 0x7f8b380ce760 2024-01-15T18:15:35.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.601+0000 7f8b2fe00640 1 -- 172.21.15.27:0/3866460882 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7f8b30014a90 con 0x7f8b380ce760 2024-01-15T18:15:35.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.601+0000 7f8b2fe00640 1 --2- 172.21.15.27:0/3866460882 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8b100674f0 0x7f8b100699b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:35.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.601+0000 7f8b2fe00640 1 -- 172.21.15.27:0/3866460882 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f8b300837b0 con 0x7f8b380ce760 2024-01-15T18:15:35.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.601+0000 7f8b37e00640 1 --2- 172.21.15.27:0/3866460882 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8b100674f0 0x7f8b100699b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:35.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.601+0000 7f8b37e00640 1 --2- 172.21.15.27:0/3866460882 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8b100674f0 0x7f8b100699b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f8b280ae000 tx=0x7f8b280a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:35.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.605+0000 7f8b2fe00640 1 -- 172.21.15.27:0/3866460882 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8b381520b0 con 0x7f8b380ce760 2024-01-15T18:15:35.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.761+0000 7f8b3ee00640 1 -- 172.21.15.27:0/3866460882 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f8b381520b0 con 0x7f8b380ce760 2024-01-15T18:15:35.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.769+0000 7f8b2fe00640 1 -- 172.21.15.27:0/3866460882 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8b381520b0 con 0x7f8b380ce760 2024-01-15T18:15:35.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.769+0000 7f8b3ee00640 1 -- 172.21.15.27:0/3866460882 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8b100674f0 msgr2=0x7f8b100699b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:35.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.769+0000 7f8b3ee00640 1 --2- 172.21.15.27:0/3866460882 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8b100674f0 0x7f8b100699b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f8b280ae000 tx=0x7f8b280a7000 comp rx=0 tx=0).stop 2024-01-15T18:15:35.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.769+0000 7f8b3ee00640 1 -- 172.21.15.27:0/3866460882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b380ce760 msgr2=0x7f8b3814fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:35.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.769+0000 7f8b3ee00640 1 --2- 172.21.15.27:0/3866460882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b380ce760 0x7f8b3814fb80 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f8b30009fd0 tx=0x7f8b3000e4b0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.769+0000 7f8b3ee00640 1 -- 172.21.15.27:0/3866460882 shutdown_connections 2024-01-15T18:15:35.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.769+0000 7f8b3ee00640 1 --2- 172.21.15.27:0/3866460882 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8b100674f0 0x7f8b100699b0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.769+0000 7f8b3ee00640 1 --2- 172.21.15.27:0/3866460882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b380ce760 0x7f8b3814fb80 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.769+0000 7f8b3ee00640 1 --2- 172.21.15.27:0/3866460882 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8b380b1db0 0x7f8b3814f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.769+0000 7f8b3ee00640 1 --2- 172.21.15.27:0/3866460882 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8b3806c430 0x7f8b38159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.769+0000 7f8b3ee00640 1 -- 172.21.15.27:0/3866460882 >> 172.21.15.27:0/3866460882 conn(0x7f8b380cb700 msgr2=0x7f8b38154a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:35.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.769+0000 7f8b3ee00640 1 -- 172.21.15.27:0/3866460882 shutdown_connections 2024-01-15T18:15:35.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.769+0000 7f8b3ee00640 1 -- 172.21.15.27:0/3866460882 wait complete. 2024-01-15T18:15:35.791 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added' 2024-01-15T18:15:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.929+0000 7fb303e00640 1 Processor -- start 2024-01-15T18:15:35.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.929+0000 7fb303e00640 1 -- start start 2024-01-15T18:15:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb303e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb2fc15e730 0x7fb2fc0cd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb303e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc0ce920 con 0x7fb2fc0cdd10 2024-01-15T18:15:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb303e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc0ce640 con 0x7fb2fc06db80 2024-01-15T18:15:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb303e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc0cc940 con 0x7fb2fc15e730 2024-01-15T18:15:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb301e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb2fc0cdd10 0x7fb2fc070c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55652/0 (socket says 172.21.15.27:55652) 2024-01-15T18:15:35.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb301e00640 1 -- 172.21.15.27:0/1046457019 learned_addr learned my addr 172.21.15.27:0/1046457019 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:35.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb300a00640 1 --2- 172.21.15.27:0/1046457019 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb2fc15e730 0x7fb2fc0cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:35.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb2fbe00640 1 -- 172.21.15.27:0/1046457019 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783837896 0 0) 0x7fb2fc0ce920 con 0x7fb2fc0cdd10 2024-01-15T18:15:35.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb2fbe00640 1 -- 172.21.15.27:0/1046457019 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2dc003530 con 0x7fb2fc0cdd10 2024-01-15T18:15:35.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb2fbe00640 1 -- 172.21.15.27:0/1046457019 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908802912 0 0) 0x7fb2fc0ce640 con 0x7fb2fc06db80 2024-01-15T18:15:35.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb2fbe00640 1 -- 172.21.15.27:0/1046457019 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2fc0ce920 con 0x7fb2fc06db80 2024-01-15T18:15:35.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb2fbe00640 1 -- 172.21.15.27:0/1046457019 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3667474673 0 0) 0x7fb2dc003530 con 0x7fb2fc0cdd10 2024-01-15T18:15:35.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb2fbe00640 1 -- 172.21.15.27:0/1046457019 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb2fc15e730 msgr2=0x7fb2fc0cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:35.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb2fbe00640 1 --2- 172.21.15.27:0/1046457019 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb2fc15e730 0x7fb2fc0cd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb2fbe00640 1 -- 172.21.15.27:0/1046457019 >> v1:172.21.15.93:6789/0 conn(0x7fb2fc06db80 legacy=0x7fb2fc06df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:35.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb2fbe00640 1 -- 172.21.15.27:0/1046457019 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc0ccc20 con 0x7fb2fc0cdd10 2024-01-15T18:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb2fbe00640 1 -- 172.21.15.27:0/1046457019 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fb2f4001a00 con 0x7fb2fc0cdd10 2024-01-15T18:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb2fbe00640 1 -- 172.21.15.27:0/1046457019 >> v1:172.21.15.27:6789/0 conn(0x7fb2fc0cdd10 legacy=0x7fb2fc070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb2fbe00640 1 --2- 172.21.15.27:0/1046457019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc003bb0 0x7fb2dc003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb2fbe00640 1 --2- 172.21.15.27:0/1046457019 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb2dc0047d0 0x7fb2dc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb2fbe00640 1 --2- 172.21.15.27:0/1046457019 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb2fc15e730 0x7fb2dc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb2fbe00640 1 -- 172.21.15.27:0/1046457019 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc0cc940 con 0x7fb2dc003bb0 2024-01-15T18:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb2fbe00640 1 -- 172.21.15.27:0/1046457019 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc00baa0 con 0x7fb2dc0047d0 2024-01-15T18:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb2fbe00640 1 -- 172.21.15.27:0/1046457019 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc00bc60 con 0x7fb2fc15e730 2024-01-15T18:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb301400640 1 --2- 172.21.15.27:0/1046457019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc003bb0 0x7fb2dc003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb300a00640 1 --2- 172.21.15.27:0/1046457019 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb2dc0047d0 0x7fb2dc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb301e00640 1 --2- 172.21.15.27:0/1046457019 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb2fc15e730 0x7fb2dc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb301400640 1 -- 172.21.15.27:0/1046457019 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb2fc15e730 msgr2=0x7fb2dc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb301400640 1 --2- 172.21.15.27:0/1046457019 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb2fc15e730 0x7fb2dc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb301400640 1 -- 172.21.15.27:0/1046457019 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb2dc0047d0 msgr2=0x7fb2dc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:35.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb301400640 1 --2- 172.21.15.27:0/1046457019 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb2dc0047d0 0x7fb2dc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.933+0000 7fb301400640 1 -- 172.21.15.27:0/1046457019 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2dc003530 con 0x7fb2dc003bb0 2024-01-15T18:15:35.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.937+0000 7fb301400640 1 --2- 172.21.15.27:0/1046457019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc003bb0 0x7fb2dc003fb0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec001950 tx=0x7fb2ec01ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fd6b540d2d2732a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:35.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.937+0000 7fb2fbe00640 1 -- 172.21.15.27:0/1046457019 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec0052d0 con 0x7fb2dc003bb0 2024-01-15T18:15:35.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.937+0000 7fb2fbe00640 1 -- 172.21.15.27:0/1046457019 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2ec005450 con 0x7fb2dc003bb0 2024-01-15T18:15:35.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.937+0000 7fb2fbe00640 1 -- 172.21.15.27:0/1046457019 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec0a7c10 con 0x7fb2dc003bb0 2024-01-15T18:15:35.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.937+0000 7fb303e00640 1 -- 172.21.15.27:0/1046457019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc003bb0 msgr2=0x7fb2dc003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:35.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.937+0000 7fb303e00640 1 --2- 172.21.15.27:0/1046457019 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc003bb0 0x7fb2dc003fb0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec001950 tx=0x7fb2ec01ebc0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.937+0000 7fb301e00640 1 -- 172.21.15.27:0/1046457019 reap_dead start 2024-01-15T18:15:35.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.937+0000 7fb303e00640 1 -- 172.21.15.27:0/1046457019 shutdown_connections 2024-01-15T18:15:35.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.937+0000 7fb303e00640 1 -- 172.21.15.27:0/1046457019 >> 172.21.15.27:0/1046457019 conn(0x7fb2fc0cec60 msgr2=0x7fb2fc0cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:35.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.937+0000 7fb303e00640 1 -- 172.21.15.27:0/1046457019 shutdown_connections 2024-01-15T18:15:35.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.937+0000 7fb303e00640 1 -- 172.21.15.27:0/1046457019 wait complete. 2024-01-15T18:15:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.941+0000 7fb303e00640 1 Processor -- start 2024-01-15T18:15:35.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.941+0000 7fb303e00640 1 -- start start 2024-01-15T18:15:35.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.941+0000 7fb303e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc003bb0 0x7fb2fc0c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:35.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.941+0000 7fb303e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb2fc0c36d0 0x7fb2fc0cb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:35.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.941+0000 7fb303e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb2fc1f67c0 0x7fb2fc0cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:35.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.941+0000 7fb303e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc0af470 con 0x7fb2dc003bb0 2024-01-15T18:15:35.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.941+0000 7fb303e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc1f6d00 con 0x7fb2fc0c36d0 2024-01-15T18:15:35.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.941+0000 7fb303e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc1f6fe0 con 0x7fb2fc1f67c0 2024-01-15T18:15:35.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.941+0000 7fb301400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc003bb0 0x7fb2fc0c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:35.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.941+0000 7fb301400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc003bb0 0x7fb2fc0c7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45026/0 (socket says 172.21.15.27:45026) 2024-01-15T18:15:35.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.941+0000 7fb301400640 1 -- 172.21.15.27:0/2634366296 learned_addr learned my addr 172.21.15.27:0/2634366296 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:35.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.941+0000 7fb300a00640 1 --2- 172.21.15.27:0/2634366296 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb2fc0c36d0 0x7fb2fc0cb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:35.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.941+0000 7fb301e00640 1 --2- 172.21.15.27:0/2634366296 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb2fc1f67c0 0x7fb2fc0cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:35.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.941+0000 7fb301400640 1 -- 172.21.15.27:0/2634366296 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb2fc1f67c0 msgr2=0x7fb2fc0cbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:35.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.941+0000 7fb301400640 1 --2- 172.21.15.27:0/2634366296 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb2fc1f67c0 0x7fb2fc0cbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.941+0000 7fb301400640 1 -- 172.21.15.27:0/2634366296 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb2fc0c36d0 msgr2=0x7fb2fc0cb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:35.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.941+0000 7fb301400640 1 --2- 172.21.15.27:0/2634366296 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb2fc0c36d0 0x7fb2fc0cb6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:35.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.941+0000 7fb301400640 1 -- 172.21.15.27:0/2634366296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc1f72c0 con 0x7fb2dc003bb0 2024-01-15T18:15:35.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.941+0000 7fb301400640 1 --2- 172.21.15.27:0/2634366296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc003bb0 0x7fb2fc0c7a00 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec005040 tx=0x7fb2dc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:35.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.941+0000 7fb2f9600640 1 -- 172.21.15.27:0/2634366296 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec0050d0 con 0x7fb2dc003bb0 2024-01-15T18:15:35.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.941+0000 7fb2f9600640 1 -- 172.21.15.27:0/2634366296 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2ec005230 con 0x7fb2dc003bb0 2024-01-15T18:15:35.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.941+0000 7fb303e00640 1 -- 172.21.15.27:0/2634366296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2fc1f75a0 con 0x7fb2dc003bb0 2024-01-15T18:15:35.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.941+0000 7fb2f9600640 1 -- 172.21.15.27:0/2634366296 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec006a20 con 0x7fb2dc003bb0 2024-01-15T18:15:35.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.941+0000 7fb303e00640 1 -- 172.21.15.27:0/2634366296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2fc158310 con 0x7fb2dc003bb0 2024-01-15T18:15:35.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.941+0000 7fb303e00640 1 -- 172.21.15.27:0/2634366296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2fc1f77e0 con 0x7fb2dc003bb0 2024-01-15T18:15:35.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.945+0000 7fb2f9600640 1 -- 172.21.15.27:0/2634366296 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7fb2ec006b80 con 0x7fb2dc003bb0 2024-01-15T18:15:35.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.945+0000 7fb2f9600640 1 --2- 172.21.15.27:0/2634366296 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb2d40674f0 0x7fb2d40699b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:35.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.945+0000 7fb2f9600640 1 -- 172.21.15.27:0/2634366296 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5027+0+0 (secure 0 0 0) 0x7fb2fc15b3c0 con 0x7fb2dc003bb0 2024-01-15T18:15:35.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.945+0000 7fb300a00640 1 --2- 172.21.15.27:0/2634366296 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb2d40674f0 0x7fb2d40699b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:35.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.945+0000 7fb300a00640 1 --2- 172.21.15.27:0/2634366296 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb2d40674f0 0x7fb2d40699b0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc0c54b0 tx=0x7fb2e4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:35.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:35.949+0000 7fb2f9600640 1 -- 172.21.15.27:0/2634366296 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb2ec0eb4f0 con 0x7fb2dc003bb0 2024-01-15T18:15:36.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:36.161+0000 7fb303e00640 1 -- 172.21.15.27:0/2634366296 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added"]} v 0) v1 -- 0x7fb2fc1f7120 con 0x7fb2dc003bb0 2024-01-15T18:15:37.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.105+0000 7fb2f9600640 1 -- 172.21.15.27:0/2634366296 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added"]}]=0 v499) v1 ==== 166+0+0 (secure 0 0 0) 0x7fb2fc1f7120 con 0x7fb2dc003bb0 2024-01-15T18:15:37.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.105+0000 7fb303e00640 1 -- 172.21.15.27:0/2634366296 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb2d40674f0 msgr2=0x7fb2d40699b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:37.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.105+0000 7fb303e00640 1 --2- 172.21.15.27:0/2634366296 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb2d40674f0 0x7fb2d40699b0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc0c54b0 tx=0x7fb2e4008040 comp rx=0 tx=0).stop 2024-01-15T18:15:37.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.105+0000 7fb303e00640 1 -- 172.21.15.27:0/2634366296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc003bb0 msgr2=0x7fb2fc0c7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:37.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.105+0000 7fb303e00640 1 --2- 172.21.15.27:0/2634366296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc003bb0 0x7fb2fc0c7a00 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec005040 tx=0x7fb2dc005800 comp rx=0 tx=0).stop 2024-01-15T18:15:37.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.105+0000 7fb303e00640 1 -- 172.21.15.27:0/2634366296 shutdown_connections 2024-01-15T18:15:37.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.105+0000 7fb303e00640 1 --2- 172.21.15.27:0/2634366296 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb2d40674f0 0x7fb2d40699b0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.105+0000 7fb303e00640 1 --2- 172.21.15.27:0/2634366296 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb2fc1f67c0 0x7fb2fc0cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.105+0000 7fb303e00640 1 --2- 172.21.15.27:0/2634366296 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb2fc0c36d0 0x7fb2fc0cb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.105+0000 7fb303e00640 1 --2- 172.21.15.27:0/2634366296 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2dc003bb0 0x7fb2fc0c7a00 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.105+0000 7fb303e00640 1 -- 172.21.15.27:0/2634366296 >> 172.21.15.27:0/2634366296 conn(0x7fb2fc0cec60 msgr2=0x7fb2fc06df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:37.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.105+0000 7fb303e00640 1 -- 172.21.15.27:0/2634366296 shutdown_connections 2024-01-15T18:15:37.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.105+0000 7fb303e00640 1 -- 172.21.15.27:0/2634366296 wait complete. 2024-01-15T18:15:37.132 INFO:tasks.cephfs_test_runner:X = number of clones issued. ... ok 2024-01-15T18:15:37.133 INFO:tasks.cephfs_test_runner:Starting test: test_stats_are_printed (tasks.cephfs.test_volumes.TestCloneProgress) Test that the command "ceph fs clone status" prints progress stats 2024-01-15T18:15:37.133 INFO:tasks.cephfs_test_runner:test_stats_are_printed (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T18:15:37.133 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:15:37.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.277+0000 7f5a87a00640 1 Processor -- start 2024-01-15T18:15:37.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.277+0000 7f5a87a00640 1 -- start start 2024-01-15T18:15:37.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.277+0000 7f5a87a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a8015c130 0x7f5a8006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:37.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.277+0000 7f5a87a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f5a8006e240 0x7f5a8015d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:37.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.277+0000 7f5a87a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a8006c4a0 con 0x7f5a800701a0 2024-01-15T18:15:37.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a87a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a8015c660 con 0x7f5a8006e240 2024-01-15T18:15:37.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a87a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8015c940 con 0x7f5a8015c130 2024-01-15T18:15:37.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a85000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5a800701a0 0x7f5a80070580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55668/0 (socket says 172.21.15.27:55668) 2024-01-15T18:15:37.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a85000640 1 -- 172.21.15.27:0/940812211 learned_addr learned my addr 172.21.15.27:0/940812211 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:37.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a7fe00640 1 --2- 172.21.15.27:0/940812211 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a8015c130 0x7f5a8006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:37.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a85a00640 1 --2- 172.21.15.27:0/940812211 >> v2:172.21.15.93:3300/0 conn(0x7f5a8006e240 0x7f5a8015d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:37.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a7f400640 1 -- 172.21.15.27:0/940812211 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1209206410 0 0) 0x7f5a8006c4a0 con 0x7f5a800701a0 2024-01-15T18:15:37.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a7f400640 1 -- 172.21.15.27:0/940812211 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a60003530 con 0x7f5a800701a0 2024-01-15T18:15:37.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a7f400640 1 -- 172.21.15.27:0/940812211 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3130812005 0 0) 0x7f5a60003530 con 0x7f5a800701a0 2024-01-15T18:15:37.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a7f400640 1 -- 172.21.15.27:0/940812211 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a8015c130 msgr2=0x7f5a8006dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:37.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a7f400640 1 --2- 172.21.15.27:0/940812211 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a8015c130 0x7f5a8006dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a7f400640 1 -- 172.21.15.27:0/940812211 >> v2:172.21.15.93:3300/0 conn(0x7f5a8006e240 msgr2=0x7f5a8015d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:37.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a7f400640 1 --2- 172.21.15.27:0/940812211 >> v2:172.21.15.93:3300/0 conn(0x7f5a8006e240 0x7f5a8015d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a7f400640 1 -- 172.21.15.27:0/940812211 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8015cc20 con 0x7f5a800701a0 2024-01-15T18:15:37.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a7f400640 1 -- 172.21.15.27:0/940812211 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f5a70001680 con 0x7f5a800701a0 2024-01-15T18:15:37.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a7f400640 1 -- 172.21.15.27:0/940812211 >> v1:172.21.15.27:6789/0 conn(0x7f5a800701a0 legacy=0x7f5a80070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:37.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a7f400640 1 --2- 172.21.15.27:0/940812211 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a8015c130 0x7f5a600039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:37.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a7f400640 1 --2- 172.21.15.27:0/940812211 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a60003fc0 0x7f5a600047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:37.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a7f400640 1 --2- 172.21.15.27:0/940812211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a60004d10 0x7f5a60005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:37.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a7f400640 1 -- 172.21.15.27:0/940812211 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8015c660 con 0x7f5a60004d10 2024-01-15T18:15:37.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a7f400640 1 -- 172.21.15.27:0/940812211 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8015c940 con 0x7f5a60003fc0 2024-01-15T18:15:37.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a7f400640 1 -- 172.21.15.27:0/940812211 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6000ba10 con 0x7f5a8015c130 2024-01-15T18:15:37.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a85a00640 1 --2- 172.21.15.27:0/940812211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a60004d10 0x7f5a60005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:37.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a85000640 1 --2- 172.21.15.27:0/940812211 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a8015c130 0x7f5a600039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:37.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a7fe00640 1 --2- 172.21.15.27:0/940812211 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a60003fc0 0x7f5a600047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:37.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a85a00640 1 -- 172.21.15.27:0/940812211 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a8015c130 msgr2=0x7f5a600039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:37.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a85a00640 1 --2- 172.21.15.27:0/940812211 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a8015c130 0x7f5a600039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a85a00640 1 -- 172.21.15.27:0/940812211 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a60003fc0 msgr2=0x7f5a600047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:37.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a85a00640 1 --2- 172.21.15.27:0/940812211 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a60003fc0 0x7f5a600047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a85a00640 1 -- 172.21.15.27:0/940812211 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a60003530 con 0x7f5a60004d10 2024-01-15T18:15:37.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.281+0000 7f5a7fe00640 1 --2- 172.21.15.27:0/940812211 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a60003fc0 0x7f5a600047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:37.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.285+0000 7f5a85a00640 1 --2- 172.21.15.27:0/940812211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a60004d10 0x7f5a60005190 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f5a800cfb30 tx=0x7f5a7801cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52f5a75fafa99bcc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:37.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.285+0000 7f5a7f400640 1 -- 172.21.15.27:0/940812211 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a780a5c50 con 0x7f5a60004d10 2024-01-15T18:15:37.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.285+0000 7f5a7f400640 1 -- 172.21.15.27:0/940812211 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7800f040 con 0x7f5a60004d10 2024-01-15T18:15:37.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.285+0000 7f5a7f400640 1 -- 172.21.15.27:0/940812211 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a780ad440 con 0x7f5a60004d10 2024-01-15T18:15:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.285+0000 7f5a87a00640 1 -- 172.21.15.27:0/940812211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a60004d10 msgr2=0x7f5a60005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.285+0000 7f5a87a00640 1 --2- 172.21.15.27:0/940812211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a60004d10 0x7f5a60005190 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f5a800cfb30 tx=0x7f5a7801cc40 comp rx=0 tx=0).stop 2024-01-15T18:15:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.285+0000 7f5a85a00640 1 -- 172.21.15.27:0/940812211 reap_dead start 2024-01-15T18:15:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.285+0000 7f5a87a00640 1 -- 172.21.15.27:0/940812211 shutdown_connections 2024-01-15T18:15:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.285+0000 7f5a87a00640 1 -- 172.21.15.27:0/940812211 >> 172.21.15.27:0/940812211 conn(0x7f5a800cec60 msgr2=0x7f5a800cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.285+0000 7f5a87a00640 1 -- 172.21.15.27:0/940812211 shutdown_connections 2024-01-15T18:15:37.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.285+0000 7f5a87a00640 1 -- 172.21.15.27:0/940812211 wait complete. 2024-01-15T18:15:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.285+0000 7f5a87a00640 1 Processor -- start 2024-01-15T18:15:37.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.289+0000 7f5a87a00640 1 -- start start 2024-01-15T18:15:37.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.289+0000 7f5a87a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a60004d10 0x7f5a80159af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:37.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.289+0000 7f5a87a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a801e65b0 0x7f5a801e5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:37.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.289+0000 7f5a87a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a800aee00 0x7f5a800af280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:37.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.289+0000 7f5a87a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a800af7c0 con 0x7f5a60004d10 2024-01-15T18:15:37.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.289+0000 7f5a87a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a801e7760 con 0x7f5a800aee00 2024-01-15T18:15:37.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.289+0000 7f5a87a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a801e6b50 con 0x7f5a801e65b0 2024-01-15T18:15:37.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.289+0000 7f5a85000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a60004d10 0x7f5a80159af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:37.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.289+0000 7f5a85000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a60004d10 0x7f5a80159af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45042/0 (socket says 172.21.15.27:45042) 2024-01-15T18:15:37.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.289+0000 7f5a85000640 1 -- 172.21.15.27:0/2709315552 learned_addr learned my addr 172.21.15.27:0/2709315552 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:37.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.289+0000 7f5a85a00640 1 --2- 172.21.15.27:0/2709315552 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a800aee00 0x7f5a800af280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:37.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.289+0000 7f5a7fe00640 1 --2- 172.21.15.27:0/2709315552 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a801e65b0 0x7f5a801e5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:37.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.289+0000 7f5a85000640 1 -- 172.21.15.27:0/2709315552 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a801e65b0 msgr2=0x7f5a801e5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:37.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.289+0000 7f5a85000640 1 --2- 172.21.15.27:0/2709315552 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a801e65b0 0x7f5a801e5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.289+0000 7f5a85000640 1 -- 172.21.15.27:0/2709315552 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a800aee00 msgr2=0x7f5a800af280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:37.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.289+0000 7f5a85000640 1 --2- 172.21.15.27:0/2709315552 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a800aee00 0x7f5a800af280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.289+0000 7f5a85000640 1 -- 172.21.15.27:0/2709315552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a801e6e30 con 0x7f5a60004d10 2024-01-15T18:15:37.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.289+0000 7f5a7fe00640 1 --2- 172.21.15.27:0/2709315552 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a801e65b0 0x7f5a801e5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:37.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.289+0000 7f5a85000640 1 --2- 172.21.15.27:0/2709315552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a60004d10 0x7f5a80159af0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f5a70007310 tx=0x7f5a700077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:37.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.289+0000 7f5a7cc00640 1 -- 172.21.15.27:0/2709315552 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7000f9c0 con 0x7f5a60004d10 2024-01-15T18:15:37.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.289+0000 7f5a7cc00640 1 -- 172.21.15.27:0/2709315552 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7000fb40 con 0x7f5a60004d10 2024-01-15T18:15:37.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.289+0000 7f5a7cc00640 1 -- 172.21.15.27:0/2709315552 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a700133f0 con 0x7f5a60004d10 2024-01-15T18:15:37.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.289+0000 7f5a87a00640 1 -- 172.21.15.27:0/2709315552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a801e7110 con 0x7f5a60004d10 2024-01-15T18:15:37.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.289+0000 7f5a87a00640 1 -- 172.21.15.27:0/2709315552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a801e6240 con 0x7f5a60004d10 2024-01-15T18:15:37.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.289+0000 7f5a87a00640 1 -- 172.21.15.27:0/2709315552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a8004e850 con 0x7f5a60004d10 2024-01-15T18:15:37.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.293+0000 7f5a7cc00640 1 -- 172.21.15.27:0/2709315552 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7f5a70013550 con 0x7f5a60004d10 2024-01-15T18:15:37.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.293+0000 7f5a7cc00640 1 --2- 172.21.15.27:0/2709315552 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5a580674f0 0x7f5a580699b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:37.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.293+0000 7f5a7cc00640 1 -- 172.21.15.27:0/2709315552 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f5a700831d0 con 0x7f5a60004d10 2024-01-15T18:15:37.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.293+0000 7f5a7fe00640 1 --2- 172.21.15.27:0/2709315552 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5a580674f0 0x7f5a580699b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:37.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.293+0000 7f5a7fe00640 1 --2- 172.21.15.27:0/2709315552 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5a580674f0 0x7f5a580699b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f5a8006e110 tx=0x7f5a68008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:37.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.297+0000 7f5a7cc00640 1 -- 172.21.15.27:0/2709315552 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5a70051c00 con 0x7f5a60004d10 2024-01-15T18:15:37.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.453+0000 7f5a87a00640 1 -- 172.21.15.27:0/2709315552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a800c9280 con 0x7f5a60004d10 2024-01-15T18:15:37.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.453+0000 7f5a7cc00640 1 -- 172.21.15.27:0/2709315552 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+13023 (secure 0 0 0) 0x7f5a70055220 con 0x7f5a60004d10 2024-01-15T18:15:37.456 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:15:37.456 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":76,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:15:31.734298+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6835/2027824196":"2024-01-16T18:15:28.727053+0000","172.21.15.27:6834/2027824196":"2024-01-16T18:15:28.727053+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:15:37.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.457+0000 7f5a87a00640 1 -- 172.21.15.27:0/2709315552 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5a580674f0 msgr2=0x7f5a580699b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:37.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.457+0000 7f5a87a00640 1 --2- 172.21.15.27:0/2709315552 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5a580674f0 0x7f5a580699b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f5a8006e110 tx=0x7f5a68008040 comp rx=0 tx=0).stop 2024-01-15T18:15:37.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.457+0000 7f5a87a00640 1 -- 172.21.15.27:0/2709315552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a60004d10 msgr2=0x7f5a80159af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:37.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.457+0000 7f5a87a00640 1 --2- 172.21.15.27:0/2709315552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a60004d10 0x7f5a80159af0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f5a70007310 tx=0x7f5a700077d0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.457+0000 7f5a87a00640 1 -- 172.21.15.27:0/2709315552 shutdown_connections 2024-01-15T18:15:37.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.457+0000 7f5a87a00640 1 --2- 172.21.15.27:0/2709315552 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5a580674f0 0x7f5a580699b0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.457+0000 7f5a87a00640 1 --2- 172.21.15.27:0/2709315552 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a800aee00 0x7f5a800af280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.457+0000 7f5a87a00640 1 --2- 172.21.15.27:0/2709315552 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a801e65b0 0x7f5a801e5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.457+0000 7f5a87a00640 1 --2- 172.21.15.27:0/2709315552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a60004d10 0x7f5a80159af0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.457+0000 7f5a87a00640 1 -- 172.21.15.27:0/2709315552 >> 172.21.15.27:0/2709315552 conn(0x7f5a800cec60 msgr2=0x7f5a800ac990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:37.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.457+0000 7f5a87a00640 1 -- 172.21.15.27:0/2709315552 shutdown_connections 2024-01-15T18:15:37.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.457+0000 7f5a87a00640 1 -- 172.21.15.27:0/2709315552 wait complete. 2024-01-15T18:15:37.479 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:15:37.479 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:15:37.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.621+0000 7f0ea8e00640 1 Processor -- start 2024-01-15T18:15:37.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.621+0000 7f0ea8e00640 1 -- start start 2024-01-15T18:15:37.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.621+0000 7f0ea8e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0ea4158900 0x7f0ea415f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:37.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.621+0000 7f0ea8e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ea415bf00 0x7f0ea40cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:37.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.621+0000 7f0ea8e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea40b1cb0 con 0x7f0ea40cf110 2024-01-15T18:15:37.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.621+0000 7f0ea8e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ea40cf540 con 0x7f0ea4158900 2024-01-15T18:15:37.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.621+0000 7f0ea8e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea40b1f00 con 0x7f0ea415bf00 2024-01-15T18:15:37.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.621+0000 7f0ea2800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0ea40cf110 0x7f0ea40701e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55678/0 (socket says 172.21.15.27:55678) 2024-01-15T18:15:37.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.621+0000 7f0ea2800640 1 -- 172.21.15.27:0/2778893916 learned_addr learned my addr 172.21.15.27:0/2778893916 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:37.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.625+0000 7f0ea1400640 1 --2- 172.21.15.27:0/2778893916 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ea415bf00 0x7f0ea40cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:37.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.625+0000 7f0ea1e00640 1 --2- 172.21.15.27:0/2778893916 >> v2:172.21.15.93:3300/0 conn(0x7f0ea4158900 0x7f0ea415f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:37.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.625+0000 7f0ea0a00640 1 -- 172.21.15.27:0/2778893916 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41993625 0 0) 0x7f0ea40b1cb0 con 0x7f0ea40cf110 2024-01-15T18:15:37.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.625+0000 7f0ea0a00640 1 -- 172.21.15.27:0/2778893916 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e84003530 con 0x7f0ea40cf110 2024-01-15T18:15:37.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.625+0000 7f0ea1400640 1 -- 172.21.15.27:0/2778893916 >> v2:172.21.15.93:3300/0 conn(0x7f0ea4158900 msgr2=0x7f0ea415f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:37.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.625+0000 7f0ea1400640 1 --2- 172.21.15.27:0/2778893916 >> v2:172.21.15.93:3300/0 conn(0x7f0ea4158900 0x7f0ea415f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.625+0000 7f0ea1400640 1 -- 172.21.15.27:0/2778893916 >> v1:172.21.15.27:6789/0 conn(0x7f0ea40cf110 legacy=0x7f0ea40701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:37.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.625+0000 7f0ea1400640 1 -- 172.21.15.27:0/2778893916 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea40b21e0 con 0x7f0ea415bf00 2024-01-15T18:15:37.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.625+0000 7f0ea1400640 1 --2- 172.21.15.27:0/2778893916 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ea415bf00 0x7f0ea40cebd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c0097c0 tx=0x7f0e8c01c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=206df237a2bb0f1b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:37.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.625+0000 7f0ea0a00640 1 -- 172.21.15.27:0/2778893916 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c0ab600 con 0x7f0ea415bf00 2024-01-15T18:15:37.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.625+0000 7f0ea0a00640 1 -- 172.21.15.27:0/2778893916 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e8c0a7050 con 0x7f0ea415bf00 2024-01-15T18:15:37.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.625+0000 7f0ea0a00640 1 -- 172.21.15.27:0/2778893916 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c0ab960 con 0x7f0ea415bf00 2024-01-15T18:15:37.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.629+0000 7f0ea8e00640 1 -- 172.21.15.27:0/2778893916 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ea415bf00 msgr2=0x7f0ea40cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:37.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.629+0000 7f0ea8e00640 1 --2- 172.21.15.27:0/2778893916 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ea415bf00 0x7f0ea40cebd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c0097c0 tx=0x7f0e8c01c480 comp rx=0 tx=0).stop 2024-01-15T18:15:37.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.629+0000 7f0ea8e00640 1 -- 172.21.15.27:0/2778893916 shutdown_connections 2024-01-15T18:15:37.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.629+0000 7f0ea8e00640 1 --2- 172.21.15.27:0/2778893916 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ea415bf00 0x7f0ea40cebd0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.629+0000 7f0ea8e00640 1 --2- 172.21.15.27:0/2778893916 >> v2:172.21.15.93:3300/0 conn(0x7f0ea4158900 0x7f0ea415f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.629+0000 7f0ea8e00640 1 -- 172.21.15.27:0/2778893916 >> 172.21.15.27:0/2778893916 conn(0x7f0ea40cb700 msgr2=0x7f0ea40cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:37.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.629+0000 7f0ea8e00640 1 -- 172.21.15.27:0/2778893916 shutdown_connections 2024-01-15T18:15:37.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.629+0000 7f0ea8e00640 1 -- 172.21.15.27:0/2778893916 wait complete. 2024-01-15T18:15:37.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.629+0000 7f0ea8e00640 1 Processor -- start 2024-01-15T18:15:37.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.629+0000 7f0ea8e00640 1 -- start start 2024-01-15T18:15:37.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.629+0000 7f0ea8e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea40cf110 0x7f0ea41f5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:37.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.629+0000 7f0ea8e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ea4158900 0x7f0ea4138d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:37.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.629+0000 7f0ea8e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ea415bf00 0x7f0ea414f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:37.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.629+0000 7f0ea8e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea41f5fa0 con 0x7f0ea40cf110 2024-01-15T18:15:37.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.629+0000 7f0ea8e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea40cc810 con 0x7f0ea415bf00 2024-01-15T18:15:37.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.629+0000 7f0ea8e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea40ccdd0 con 0x7f0ea4158900 2024-01-15T18:15:37.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.629+0000 7f0ea1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea40cf110 0x7f0ea41f5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:37.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.633+0000 7f0ea1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea40cf110 0x7f0ea41f5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45052/0 (socket says 172.21.15.27:45052) 2024-01-15T18:15:37.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.633+0000 7f0ea1e00640 1 -- 172.21.15.27:0/2649537319 learned_addr learned my addr 172.21.15.27:0/2649537319 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:37.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.633+0000 7f0ea2800640 1 --2- 172.21.15.27:0/2649537319 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ea415bf00 0x7f0ea414f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:37.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.633+0000 7f0ea1400640 1 --2- 172.21.15.27:0/2649537319 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ea4158900 0x7f0ea4138d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:37.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.633+0000 7f0ea1e00640 1 -- 172.21.15.27:0/2649537319 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ea4158900 msgr2=0x7f0ea4138d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:37.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.633+0000 7f0ea1e00640 1 --2- 172.21.15.27:0/2649537319 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ea4158900 0x7f0ea4138d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.633+0000 7f0ea1e00640 1 -- 172.21.15.27:0/2649537319 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ea415bf00 msgr2=0x7f0ea414f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:37.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.633+0000 7f0ea1e00640 1 --2- 172.21.15.27:0/2649537319 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ea415bf00 0x7f0ea414f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.633+0000 7f0ea1e00640 1 -- 172.21.15.27:0/2649537319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8c009420 con 0x7f0ea40cf110 2024-01-15T18:15:37.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.633+0000 7f0ea1e00640 1 --2- 172.21.15.27:0/2649537319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea40cf110 0x7f0ea41f5a60 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400e960 tx=0x7f0e9400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:37.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.633+0000 7f0e9a000640 1 -- 172.21.15.27:0/2649537319 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e94009690 con 0x7f0ea40cf110 2024-01-15T18:15:37.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.633+0000 7f0e9a000640 1 -- 172.21.15.27:0/2649537319 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9400c9a0 con 0x7f0ea40cf110 2024-01-15T18:15:37.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.633+0000 7f0e9a000640 1 -- 172.21.15.27:0/2649537319 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9400cd10 con 0x7f0ea40cf110 2024-01-15T18:15:37.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.633+0000 7f0ea8e00640 1 -- 172.21.15.27:0/2649537319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ea4139810 con 0x7f0ea40cf110 2024-01-15T18:15:37.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.633+0000 7f0ea8e00640 1 -- 172.21.15.27:0/2649537319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ea40cca60 con 0x7f0ea40cf110 2024-01-15T18:15:37.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.633+0000 7f0ea8e00640 1 -- 172.21.15.27:0/2649537319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ea4139d30 con 0x7f0ea40cf110 2024-01-15T18:15:37.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.633+0000 7f0e9a000640 1 -- 172.21.15.27:0/2649537319 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7f0e94013470 con 0x7f0ea40cf110 2024-01-15T18:15:37.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.637+0000 7f0e9a000640 1 --2- 172.21.15.27:0/2649537319 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0e7c0674c0 0x7f0e7c069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:37.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.637+0000 7f0e9a000640 1 -- 172.21.15.27:0/2649537319 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f0e94083580 con 0x7f0ea40cf110 2024-01-15T18:15:37.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.637+0000 7f0ea1400640 1 --2- 172.21.15.27:0/2649537319 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0e7c0674c0 0x7f0e7c069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:37.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.637+0000 7f0ea1400640 1 --2- 172.21.15.27:0/2649537319 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0e7c0674c0 0x7f0e7c069980 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c0ad000 tx=0x7f0e8c0a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:37.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.637+0000 7f0e9a000640 1 -- 172.21.15.27:0/2649537319 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0e94052060 con 0x7f0ea40cf110 2024-01-15T18:15:37.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.797+0000 7f0ea8e00640 1 -- 172.21.15.27:0/2649537319 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0ea404e850 con 0x7f0ea40cf110 2024-01-15T18:15:37.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.797+0000 7f0e9a000640 1 -- 172.21.15.27:0/2649537319 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0e940210a0 con 0x7f0ea40cf110 2024-01-15T18:15:37.799 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:15:37.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.797+0000 7f0ea8e00640 1 -- 172.21.15.27:0/2649537319 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0e7c0674c0 msgr2=0x7f0e7c069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:37.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.797+0000 7f0ea8e00640 1 --2- 172.21.15.27:0/2649537319 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0e7c0674c0 0x7f0e7c069980 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c0ad000 tx=0x7f0e8c0a6000 comp rx=0 tx=0).stop 2024-01-15T18:15:37.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.797+0000 7f0ea8e00640 1 -- 172.21.15.27:0/2649537319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea40cf110 msgr2=0x7f0ea41f5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:37.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.797+0000 7f0ea8e00640 1 --2- 172.21.15.27:0/2649537319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea40cf110 0x7f0ea41f5a60 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400e960 tx=0x7f0e9400ee20 comp rx=0 tx=0).stop 2024-01-15T18:15:37.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.797+0000 7f0ea8e00640 1 -- 172.21.15.27:0/2649537319 shutdown_connections 2024-01-15T18:15:37.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.797+0000 7f0ea8e00640 1 --2- 172.21.15.27:0/2649537319 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0e7c0674c0 0x7f0e7c069980 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.797+0000 7f0ea8e00640 1 --2- 172.21.15.27:0/2649537319 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0ea415bf00 0x7f0ea414f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.797+0000 7f0ea8e00640 1 --2- 172.21.15.27:0/2649537319 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0ea4158900 0x7f0ea4138d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.797+0000 7f0ea8e00640 1 --2- 172.21.15.27:0/2649537319 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ea40cf110 0x7f0ea41f5a60 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.797+0000 7f0ea8e00640 1 -- 172.21.15.27:0/2649537319 >> 172.21.15.27:0/2649537319 conn(0x7f0ea40cb700 msgr2=0x7f0ea415c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:37.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.801+0000 7f0ea8e00640 1 -- 172.21.15.27:0/2649537319 shutdown_connections 2024-01-15T18:15:37.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.801+0000 7f0ea8e00640 1 -- 172.21.15.27:0/2649537319 wait complete. 2024-01-15T18:15:37.820 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_stats_are_printed' 2024-01-15T18:15:37.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.961+0000 7fc59c200640 1 Processor -- start 2024-01-15T18:15:37.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.961+0000 7fc59c200640 1 -- start start 2024-01-15T18:15:37.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.961+0000 7fc59c200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc59415d0a0 0x7fc59406c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:37.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.961+0000 7fc59c200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5940ce710 0x7fc5940cd240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:37.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.961+0000 7fc59c200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5940ad180 con 0x7fc5940c94c0 2024-01-15T18:15:37.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.961+0000 7fc59c200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5940ac890 con 0x7fc59415d0a0 2024-01-15T18:15:37.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.961+0000 7fc59c200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5940cda90 con 0x7fc5940ce710 2024-01-15T18:15:37.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.961+0000 7fc599800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc59415d0a0 0x7fc59406c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:37.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.961+0000 7fc599800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc59415d0a0 0x7fc59406c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45068/0 (socket says 172.21.15.27:45068) 2024-01-15T18:15:37.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.961+0000 7fc599800640 1 -- 172.21.15.27:0/1053774049 learned_addr learned my addr 172.21.15.27:0/1053774049 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:37.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.961+0000 7fc598e00640 1 --2- 172.21.15.27:0/1053774049 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5940ce710 0x7fc5940cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:37.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.964+0000 7fc599800640 1 -- 172.21.15.27:0/1053774049 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5940ce710 msgr2=0x7fc5940cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:37.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.964+0000 7fc599800640 1 --2- 172.21.15.27:0/1053774049 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5940ce710 0x7fc5940cd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.964+0000 7fc599800640 1 -- 172.21.15.27:0/1053774049 >> v1:172.21.15.93:6789/0 conn(0x7fc5940c94c0 legacy=0x7fc5940ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:37.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.964+0000 7fc599800640 1 -- 172.21.15.27:0/1053774049 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5940acb30 con 0x7fc59415d0a0 2024-01-15T18:15:37.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.964+0000 7fc598e00640 1 --2- 172.21.15.27:0/1053774049 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5940ce710 0x7fc5940cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:37.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.964+0000 7fc599800640 1 --2- 172.21.15.27:0/1053774049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc59415d0a0 0x7fc59406c1e0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fc584009870 tx=0x7fc58401c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ae34ddc968dfcb6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:37.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.964+0000 7fc593e00640 1 -- 172.21.15.27:0/1053774049 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5840aa430 con 0x7fc59415d0a0 2024-01-15T18:15:37.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.964+0000 7fc593e00640 1 -- 172.21.15.27:0/1053774049 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5840aa590 con 0x7fc59415d0a0 2024-01-15T18:15:37.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.964+0000 7fc593e00640 1 -- 172.21.15.27:0/1053774049 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5840aa920 con 0x7fc59415d0a0 2024-01-15T18:15:37.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc59c200640 1 -- 172.21.15.27:0/1053774049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc59415d0a0 msgr2=0x7fc59406c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:37.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc59c200640 1 --2- 172.21.15.27:0/1053774049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc59415d0a0 0x7fc59406c1e0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fc584009870 tx=0x7fc58401c510 comp rx=0 tx=0).stop 2024-01-15T18:15:37.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc59c200640 1 -- 172.21.15.27:0/1053774049 shutdown_connections 2024-01-15T18:15:37.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc59c200640 1 --2- 172.21.15.27:0/1053774049 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5940ce710 0x7fc5940cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc59c200640 1 --2- 172.21.15.27:0/1053774049 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc59415d0a0 0x7fc59406c1e0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc59c200640 1 -- 172.21.15.27:0/1053774049 >> 172.21.15.27:0/1053774049 conn(0x7fc5940cec60 msgr2=0x7fc5940cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:37.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc59c200640 1 -- 172.21.15.27:0/1053774049 shutdown_connections 2024-01-15T18:15:37.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc59c200640 1 -- 172.21.15.27:0/1053774049 wait complete. 2024-01-15T18:15:37.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc59c200640 1 Processor -- start 2024-01-15T18:15:37.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc59c200640 1 -- start start 2024-01-15T18:15:37.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc59c200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5940c94c0 0x7fc59415b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:37.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc59c200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5940ce710 0x7fc59406c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:37.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc59c200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc59415d0a0 0x7fc59414ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:37.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc59c200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59414eb80 con 0x7fc5940c94c0 2024-01-15T18:15:37.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc59c200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59414df70 con 0x7fc59415d0a0 2024-01-15T18:15:37.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc59c200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc59414e250 con 0x7fc5940ce710 2024-01-15T18:15:37.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc599800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5940c94c0 0x7fc59415b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:37.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc599800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5940c94c0 0x7fc59415b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45082/0 (socket says 172.21.15.27:45082) 2024-01-15T18:15:37.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc599800640 1 -- 172.21.15.27:0/508814517 learned_addr learned my addr 172.21.15.27:0/508814517 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:37.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc598e00640 1 --2- 172.21.15.27:0/508814517 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5940ce710 0x7fc59406c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:37.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc59a200640 1 --2- 172.21.15.27:0/508814517 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc59415d0a0 0x7fc59414ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:37.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc599800640 1 -- 172.21.15.27:0/508814517 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5940ce710 msgr2=0x7fc59406c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:37.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc599800640 1 --2- 172.21.15.27:0/508814517 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5940ce710 0x7fc59406c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc599800640 1 -- 172.21.15.27:0/508814517 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc59415d0a0 msgr2=0x7fc59414ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:37.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc599800640 1 --2- 172.21.15.27:0/508814517 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc59415d0a0 0x7fc59414ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:37.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.968+0000 7fc599800640 1 -- 172.21.15.27:0/508814517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5840094d0 con 0x7fc5940c94c0 2024-01-15T18:15:37.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.972+0000 7fc599800640 1 --2- 172.21.15.27:0/508814517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5940c94c0 0x7fc59415b0c0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fc5840ac040 tx=0x7fc58401ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:37.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.972+0000 7fc591600640 1 -- 172.21.15.27:0/508814517 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5840b3800 con 0x7fc5940c94c0 2024-01-15T18:15:37.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.972+0000 7fc591600640 1 -- 172.21.15.27:0/508814517 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5840b3960 con 0x7fc5940c94c0 2024-01-15T18:15:37.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.972+0000 7fc591600640 1 -- 172.21.15.27:0/508814517 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5840b3d10 con 0x7fc5940c94c0 2024-01-15T18:15:37.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.972+0000 7fc59c200640 1 -- 172.21.15.27:0/508814517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5940ae2b0 con 0x7fc5940c94c0 2024-01-15T18:15:37.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.972+0000 7fc59c200640 1 -- 172.21.15.27:0/508814517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc59414e810 con 0x7fc5940c94c0 2024-01-15T18:15:37.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.972+0000 7fc59c200640 1 -- 172.21.15.27:0/508814517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc59404e850 con 0x7fc5940c94c0 2024-01-15T18:15:37.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.972+0000 7fc591600640 1 -- 172.21.15.27:0/508814517 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7fc5840b2380 con 0x7fc5940c94c0 2024-01-15T18:15:37.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.972+0000 7fc591600640 1 --2- 172.21.15.27:0/508814517 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc56c0674c0 0x7fc56c069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.972+0000 7fc591600640 1 -- 172.21.15.27:0/508814517 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5027+0+0 (secure 0 0 0) 0x7fc58411d600 con 0x7fc5940c94c0 2024-01-15T18:15:37.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.972+0000 7fc598e00640 1 --2- 172.21.15.27:0/508814517 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc56c0674c0 0x7fc56c069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:37.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.976+0000 7fc598e00640 1 --2- 172.21.15.27:0/508814517 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc56c0674c0 0x7fc56c069980 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc5940ae770 tx=0x7fc57c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:37.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:37.976+0000 7fc591600640 1 -- 172.21.15.27:0/508814517 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc5840ec0e0 con 0x7fc5940c94c0 2024-01-15T18:15:38.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:38.188+0000 7fc59c200640 1 -- 172.21.15.27:0/508814517 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_stats_are_printed"]} v 0) v1 -- 0x7fc594072300 con 0x7fc5940c94c0 2024-01-15T18:15:39.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.112+0000 7fc591600640 1 -- 172.21.15.27:0/508814517 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_stats_are_printed"]}]=0 v501) v1 ==== 148+0+0 (secure 0 0 0) 0x7fc5840ef700 con 0x7fc5940c94c0 2024-01-15T18:15:39.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.112+0000 7fc59c200640 1 -- 172.21.15.27:0/508814517 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc56c0674c0 msgr2=0x7fc56c069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:39.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.112+0000 7fc59c200640 1 --2- 172.21.15.27:0/508814517 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc56c0674c0 0x7fc56c069980 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc5940ae770 tx=0x7fc57c008040 comp rx=0 tx=0).stop 2024-01-15T18:15:39.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.112+0000 7fc59c200640 1 -- 172.21.15.27:0/508814517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5940c94c0 msgr2=0x7fc59415b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:39.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.112+0000 7fc59c200640 1 --2- 172.21.15.27:0/508814517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5940c94c0 0x7fc59415b0c0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fc5840ac040 tx=0x7fc58401ad50 comp rx=0 tx=0).stop 2024-01-15T18:15:39.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.112+0000 7fc59c200640 1 -- 172.21.15.27:0/508814517 shutdown_connections 2024-01-15T18:15:39.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.112+0000 7fc59c200640 1 --2- 172.21.15.27:0/508814517 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc56c0674c0 0x7fc56c069980 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:39.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.112+0000 7fc59c200640 1 --2- 172.21.15.27:0/508814517 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc59415d0a0 0x7fc59414ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:39.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.112+0000 7fc59c200640 1 --2- 172.21.15.27:0/508814517 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5940ce710 0x7fc59406c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:39.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.112+0000 7fc59c200640 1 --2- 172.21.15.27:0/508814517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5940c94c0 0x7fc59415b0c0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:39.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.112+0000 7fc59c200640 1 -- 172.21.15.27:0/508814517 >> 172.21.15.27:0/508814517 conn(0x7fc5940cec60 msgr2=0x7fc5940703d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:39.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.116+0000 7fc59c200640 1 -- 172.21.15.27:0/508814517 shutdown_connections 2024-01-15T18:15:39.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.116+0000 7fc59c200640 1 -- 172.21.15.27:0/508814517 wait complete. 2024-01-15T18:15:39.142 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:15:39.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.288+0000 7ff886800640 1 Processor -- start 2024-01-15T18:15:39.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.288+0000 7ff886800640 1 -- start start 2024-01-15T18:15:39.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.288+0000 7ff886800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff88015d480 0x7ff8800cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:39.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.288+0000 7ff886800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff88006c740 0x7ff8800d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:39.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.288+0000 7ff886800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff88015d860 con 0x7ff880099a10 2024-01-15T18:15:39.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.288+0000 7ff886800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff88006b6a0 con 0x7ff88006c740 2024-01-15T18:15:39.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.288+0000 7ff886800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff88006b980 con 0x7ff88015d480 2024-01-15T18:15:39.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.292+0000 7ff87ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff88006c740 0x7ff8800d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:39.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.292+0000 7ff87ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff88006c740 0x7ff8800d0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45084/0 (socket says 172.21.15.27:45084) 2024-01-15T18:15:39.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.292+0000 7ff87ea00640 1 -- 172.21.15.27:0/1057560473 learned_addr learned my addr 172.21.15.27:0/1057560473 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:39.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.292+0000 7ff87f400640 1 --2- 172.21.15.27:0/1057560473 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff88015d480 0x7ff8800cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:39.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.292+0000 7ff87ea00640 1 -- 172.21.15.27:0/1057560473 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff88015d480 msgr2=0x7ff8800cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:39.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.292+0000 7ff87ea00640 1 --2- 172.21.15.27:0/1057560473 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff88015d480 0x7ff8800cfbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:39.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.292+0000 7ff87ea00640 1 -- 172.21.15.27:0/1057560473 >> v1:172.21.15.93:6789/0 conn(0x7ff880099a10 legacy=0x7ff880160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:39.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.292+0000 7ff87ea00640 1 -- 172.21.15.27:0/1057560473 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88006bc60 con 0x7ff88006c740 2024-01-15T18:15:39.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.292+0000 7ff87ea00640 1 --2- 172.21.15.27:0/1057560473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff88006c740 0x7ff8800d0130 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7ff868009870 tx=0x7ff86801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24eeb1abd1eac741 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:39.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.292+0000 7ff87e000640 1 -- 172.21.15.27:0/1057560473 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8680a7070 con 0x7ff88006c740 2024-01-15T18:15:39.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.292+0000 7ff87e000640 1 -- 172.21.15.27:0/1057560473 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8680ab430 con 0x7ff88006c740 2024-01-15T18:15:39.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.292+0000 7ff87e000640 1 -- 172.21.15.27:0/1057560473 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8680ab7c0 con 0x7ff88006c740 2024-01-15T18:15:39.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.296+0000 7ff886800640 1 -- 172.21.15.27:0/1057560473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff88006c740 msgr2=0x7ff8800d0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:39.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.296+0000 7ff886800640 1 --2- 172.21.15.27:0/1057560473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff88006c740 0x7ff8800d0130 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7ff868009870 tx=0x7ff86801c540 comp rx=0 tx=0).stop 2024-01-15T18:15:39.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.296+0000 7ff886800640 1 -- 172.21.15.27:0/1057560473 shutdown_connections 2024-01-15T18:15:39.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.296+0000 7ff886800640 1 --2- 172.21.15.27:0/1057560473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff88006c740 0x7ff8800d0130 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:39.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.296+0000 7ff886800640 1 --2- 172.21.15.27:0/1057560473 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff88015d480 0x7ff8800cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:39.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.296+0000 7ff886800640 1 -- 172.21.15.27:0/1057560473 >> 172.21.15.27:0/1057560473 conn(0x7ff8800b2a80 msgr2=0x7ff8800b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:39.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.296+0000 7ff886800640 1 -- 172.21.15.27:0/1057560473 shutdown_connections 2024-01-15T18:15:39.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.296+0000 7ff886800640 1 -- 172.21.15.27:0/1057560473 wait complete. 2024-01-15T18:15:39.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.296+0000 7ff886800640 1 Processor -- start 2024-01-15T18:15:39.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.296+0000 7ff886800640 1 -- start start 2024-01-15T18:15:39.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.296+0000 7ff886800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff88006c740 0x7ff8800b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:39.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.296+0000 7ff886800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff880099a10 0x7ff8800b5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:39.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.296+0000 7ff886800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff88015d480 0x7ff8800b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:39.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.296+0000 7ff886800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88015cce0 con 0x7ff88015d480 2024-01-15T18:15:39.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.296+0000 7ff886800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88015bec0 con 0x7ff880099a10 2024-01-15T18:15:39.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.296+0000 7ff886800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff88015c160 con 0x7ff88006c740 2024-01-15T18:15:39.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.296+0000 7ff87fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff88015d480 0x7ff8800b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:39.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.300+0000 7ff87fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff88015d480 0x7ff8800b5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45094/0 (socket says 172.21.15.27:45094) 2024-01-15T18:15:39.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.300+0000 7ff87fe00640 1 -- 172.21.15.27:0/828592642 learned_addr learned my addr 172.21.15.27:0/828592642 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:39.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.300+0000 7ff87ea00640 1 --2- 172.21.15.27:0/828592642 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff880099a10 0x7ff8800b5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:39.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.300+0000 7ff87f400640 1 --2- 172.21.15.27:0/828592642 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff88006c740 0x7ff8800b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:39.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.300+0000 7ff87fe00640 1 -- 172.21.15.27:0/828592642 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff88006c740 msgr2=0x7ff8800b5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:39.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.300+0000 7ff87fe00640 1 --2- 172.21.15.27:0/828592642 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff88006c740 0x7ff8800b5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:39.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.300+0000 7ff87fe00640 1 -- 172.21.15.27:0/828592642 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff880099a10 msgr2=0x7ff8800b5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:39.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.300+0000 7ff87fe00640 1 --2- 172.21.15.27:0/828592642 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff880099a10 0x7ff8800b5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:39.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.300+0000 7ff87fe00640 1 -- 172.21.15.27:0/828592642 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8680094d0 con 0x7ff88015d480 2024-01-15T18:15:39.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.300+0000 7ff87fe00640 1 --2- 172.21.15.27:0/828592642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff88015d480 0x7ff8800b5ab0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7ff87800c770 tx=0x7ff87800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:39.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.300+0000 7ff877400640 1 -- 172.21.15.27:0/828592642 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff878003b80 con 0x7ff88015d480 2024-01-15T18:15:39.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.300+0000 7ff877400640 1 -- 172.21.15.27:0/828592642 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff878003ce0 con 0x7ff88015d480 2024-01-15T18:15:39.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.300+0000 7ff877400640 1 -- 172.21.15.27:0/828592642 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87800aa90 con 0x7ff88015d480 2024-01-15T18:15:39.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.300+0000 7ff886800640 1 -- 172.21.15.27:0/828592642 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff88015c440 con 0x7ff88015d480 2024-01-15T18:15:39.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.300+0000 7ff886800640 1 -- 172.21.15.27:0/828592642 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff88015ca00 con 0x7ff88015d480 2024-01-15T18:15:39.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.300+0000 7ff886800640 1 -- 172.21.15.27:0/828592642 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff88004e850 con 0x7ff88015d480 2024-01-15T18:15:39.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.300+0000 7ff877400640 1 -- 172.21.15.27:0/828592642 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7ff87800abf0 con 0x7ff88015d480 2024-01-15T18:15:39.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.300+0000 7ff877400640 1 --2- 172.21.15.27:0/828592642 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff8580674c0 0x7ff858069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:39.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.304+0000 7ff877400640 1 -- 172.21.15.27:0/828592642 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5027+0+0 (secure 0 0 0) 0x7ff878082d60 con 0x7ff88015d480 2024-01-15T18:15:39.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.304+0000 7ff87f400640 1 --2- 172.21.15.27:0/828592642 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff8580674c0 0x7ff858069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:39.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.304+0000 7ff87f400640 1 --2- 172.21.15.27:0/828592642 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff8580674c0 0x7ff858069980 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7ff8700079c0 tx=0x7ff870008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:39.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.308+0000 7ff877400640 1 -- 172.21.15.27:0/828592642 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff878051840 con 0x7ff88015d480 2024-01-15T18:15:39.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.464+0000 7ff886800640 1 -- 172.21.15.27:0/828592642 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff880154140 con 0x7ff88015d480 2024-01-15T18:15:39.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.468+0000 7ff877400640 1 -- 172.21.15.27:0/828592642 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff878054e60 con 0x7ff88015d480 2024-01-15T18:15:39.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.468+0000 7ff886800640 1 -- 172.21.15.27:0/828592642 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff8580674c0 msgr2=0x7ff858069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:39.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.468+0000 7ff886800640 1 --2- 172.21.15.27:0/828592642 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff8580674c0 0x7ff858069980 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7ff8700079c0 tx=0x7ff870008040 comp rx=0 tx=0).stop 2024-01-15T18:15:39.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.468+0000 7ff886800640 1 -- 172.21.15.27:0/828592642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff88015d480 msgr2=0x7ff8800b5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:39.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.468+0000 7ff886800640 1 --2- 172.21.15.27:0/828592642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff88015d480 0x7ff8800b5ab0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7ff87800c770 tx=0x7ff87800cc30 comp rx=0 tx=0).stop 2024-01-15T18:15:39.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.472+0000 7ff886800640 1 -- 172.21.15.27:0/828592642 shutdown_connections 2024-01-15T18:15:39.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.472+0000 7ff886800640 1 --2- 172.21.15.27:0/828592642 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff8580674c0 0x7ff858069980 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:39.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.472+0000 7ff886800640 1 --2- 172.21.15.27:0/828592642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff88015d480 0x7ff8800b5ab0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:39.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.472+0000 7ff886800640 1 --2- 172.21.15.27:0/828592642 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff880099a10 0x7ff8800b5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:39.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.472+0000 7ff886800640 1 --2- 172.21.15.27:0/828592642 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff88006c740 0x7ff8800b5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:39.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.472+0000 7ff886800640 1 -- 172.21.15.27:0/828592642 >> 172.21.15.27:0/828592642 conn(0x7ff8800b2a80 msgr2=0x7ff880159400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:39.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.472+0000 7ff886800640 1 -- 172.21.15.27:0/828592642 shutdown_connections 2024-01-15T18:15:39.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.472+0000 7ff886800640 1 -- 172.21.15.27:0/828592642 wait complete. 2024-01-15T18:15:39.493 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T18:15:39.494 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:15:39.563 DEBUG:teuthology.orchestra.run.smithi093:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:15:39.639 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T18:15:39.640 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:15:39.640 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:15:39.649 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354580k,nr_inodes=4088645,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9249 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:15:39.649 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:15:39.649 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:15:39.658 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354484k,nr_inodes=4088621,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8403 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:15:39.658 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:15:39.658 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:15:39.700 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354580k,nr_inodes=4088645,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9249 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:15:39.700 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:15:39.700 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:15:39.709 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354484k,nr_inodes=4088621,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8403 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:15:39.709 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-15T18:15:39.709 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:15:39.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.880+0000 7fbadb600640 1 Processor -- start 2024-01-15T18:15:39.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.880+0000 7fbadb600640 1 -- start start 2024-01-15T18:15:39.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.884+0000 7fbadb600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbad4159bf0 0x7fbad4159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:39.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.884+0000 7fbadb600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbad4071e40 0x7fbad40c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:39.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.884+0000 7fbadb600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fbad406d430 0x7fbad415d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:39.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.884+0000 7fbadb600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbad40cce00 con 0x7fbad4159bf0 2024-01-15T18:15:39.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.884+0000 7fbadb600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbad4159a50 con 0x7fbad406d430 2024-01-15T18:15:39.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.884+0000 7fbadb600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad415a510 con 0x7fbad4071e40 2024-01-15T18:15:39.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.884+0000 7fbad8c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbad4159bf0 0x7fbad4159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:39.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.884+0000 7fbad8c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbad4159bf0 0x7fbad4159fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45110/0 (socket says 172.21.15.27:45110) 2024-01-15T18:15:39.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.884+0000 7fbad8c00640 1 -- 172.21.15.27:0/2104440674 learned_addr learned my addr 172.21.15.27:0/2104440674 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:39.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.884+0000 7fbad3e00640 1 --2- 172.21.15.27:0/2104440674 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbad4071e40 0x7fbad40c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:39.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.884+0000 7fbad9600640 1 --2- 172.21.15.27:0/2104440674 >> v2:172.21.15.93:3300/0 conn(0x7fbad406d430 0x7fbad415d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:39.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.884+0000 7fbad8c00640 1 -- 172.21.15.27:0/2104440674 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbad4071e40 msgr2=0x7fbad40c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:39.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.884+0000 7fbad8c00640 1 --2- 172.21.15.27:0/2104440674 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbad4071e40 0x7fbad40c9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:39.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.884+0000 7fbad8c00640 1 -- 172.21.15.27:0/2104440674 >> v2:172.21.15.93:3300/0 conn(0x7fbad406d430 msgr2=0x7fbad415d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:39.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.884+0000 7fbad8c00640 1 --2- 172.21.15.27:0/2104440674 >> v2:172.21.15.93:3300/0 conn(0x7fbad406d430 0x7fbad415d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:39.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.884+0000 7fbad8c00640 1 -- 172.21.15.27:0/2104440674 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad406d980 con 0x7fbad4159bf0 2024-01-15T18:15:39.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.888+0000 7fbad8c00640 1 --2- 172.21.15.27:0/2104440674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad4159bf0 0x7fbad4159fd0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fbad40cfa80 tx=0x7fbac40200c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2423be6cbcef7c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:39.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.888+0000 7fbad3400640 1 -- 172.21.15.27:0/2104440674 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac40a9e90 con 0x7fbad4159bf0 2024-01-15T18:15:39.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.888+0000 7fbad3400640 1 -- 172.21.15.27:0/2104440674 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac40b1440 con 0x7fbad4159bf0 2024-01-15T18:15:39.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.888+0000 7fbad3400640 1 -- 172.21.15.27:0/2104440674 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac40b17d0 con 0x7fbad4159bf0 2024-01-15T18:15:39.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.888+0000 7fbadb600640 1 -- 172.21.15.27:0/2104440674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad4159bf0 msgr2=0x7fbad4159fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:39.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.888+0000 7fbadb600640 1 --2- 172.21.15.27:0/2104440674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad4159bf0 0x7fbad4159fd0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fbad40cfa80 tx=0x7fbac40200c0 comp rx=0 tx=0).stop 2024-01-15T18:15:39.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.888+0000 7fbadb600640 1 -- 172.21.15.27:0/2104440674 shutdown_connections 2024-01-15T18:15:39.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.888+0000 7fbadb600640 1 --2- 172.21.15.27:0/2104440674 >> v2:172.21.15.93:3300/0 conn(0x7fbad406d430 0x7fbad415d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:39.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.888+0000 7fbadb600640 1 --2- 172.21.15.27:0/2104440674 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbad4071e40 0x7fbad40c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:39.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.888+0000 7fbadb600640 1 --2- 172.21.15.27:0/2104440674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad4159bf0 0x7fbad4159fd0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:39.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.888+0000 7fbadb600640 1 -- 172.21.15.27:0/2104440674 >> 172.21.15.27:0/2104440674 conn(0x7fbad40cec60 msgr2=0x7fbad40cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:39.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.888+0000 7fbadb600640 1 -- 172.21.15.27:0/2104440674 shutdown_connections 2024-01-15T18:15:39.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.888+0000 7fbadb600640 1 -- 172.21.15.27:0/2104440674 wait complete. 2024-01-15T18:15:39.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.888+0000 7fbadb600640 1 Processor -- start 2024-01-15T18:15:39.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.888+0000 7fbadb600640 1 -- start start 2024-01-15T18:15:39.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.892+0000 7fbadb600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbad406d430 0x7fbad40c48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:39.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.892+0000 7fbadb600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbad4159bf0 0x7fbad4151eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:39.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.892+0000 7fbadb600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad41523f0 0x7fbad4152870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:39.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.892+0000 7fbadb600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad40c4e20 con 0x7fbad41523f0 2024-01-15T18:15:39.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.892+0000 7fbadb600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad40c6a70 con 0x7fbad4159bf0 2024-01-15T18:15:39.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.892+0000 7fbadb600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad41f30a0 con 0x7fbad406d430 2024-01-15T18:15:39.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.892+0000 7fbad9600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad41523f0 0x7fbad4152870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:39.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.892+0000 7fbad9600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad41523f0 0x7fbad4152870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45126/0 (socket says 172.21.15.27:45126) 2024-01-15T18:15:39.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.892+0000 7fbad9600640 1 -- 172.21.15.27:0/2590819498 learned_addr learned my addr 172.21.15.27:0/2590819498 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:39.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.892+0000 7fbad3e00640 1 --2- 172.21.15.27:0/2590819498 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbad4159bf0 0x7fbad4151eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:39.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.892+0000 7fbad8c00640 1 --2- 172.21.15.27:0/2590819498 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbad406d430 0x7fbad40c48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:39.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.892+0000 7fbad9600640 1 -- 172.21.15.27:0/2590819498 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbad406d430 msgr2=0x7fbad40c48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:39.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.892+0000 7fbad9600640 1 --2- 172.21.15.27:0/2590819498 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbad406d430 0x7fbad40c48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:39.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.892+0000 7fbad9600640 1 -- 172.21.15.27:0/2590819498 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbad4159bf0 msgr2=0x7fbad4151eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:39.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.892+0000 7fbad9600640 1 --2- 172.21.15.27:0/2590819498 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbad4159bf0 0x7fbad4151eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:39.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.892+0000 7fbad9600640 1 -- 172.21.15.27:0/2590819498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac400a670 con 0x7fbad41523f0 2024-01-15T18:15:39.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.892+0000 7fbad9600640 1 --2- 172.21.15.27:0/2590819498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad41523f0 0x7fbad4152870 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fbacc007e40 tx=0x7fbacc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:39.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.892+0000 7fbad0c00640 1 -- 172.21.15.27:0/2590819498 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbacc014620 con 0x7fbad41523f0 2024-01-15T18:15:39.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.892+0000 7fbad0c00640 1 -- 172.21.15.27:0/2590819498 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbacc00ee90 con 0x7fbad41523f0 2024-01-15T18:15:39.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.892+0000 7fbadb600640 1 -- 172.21.15.27:0/2590819498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad40c5e60 con 0x7fbad41523f0 2024-01-15T18:15:39.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.892+0000 7fbad0c00640 1 -- 172.21.15.27:0/2590819498 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbacc0149e0 con 0x7fbad41523f0 2024-01-15T18:15:39.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.892+0000 7fbadb600640 1 -- 172.21.15.27:0/2590819498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad40c6420 con 0x7fbad41523f0 2024-01-15T18:15:39.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.892+0000 7fbadb600640 1 -- 172.21.15.27:0/2590819498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbad4073bd0 con 0x7fbad41523f0 2024-01-15T18:15:39.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.892+0000 7fbad0c00640 1 -- 172.21.15.27:0/2590819498 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7fbacc014b40 con 0x7fbad41523f0 2024-01-15T18:15:39.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.896+0000 7fbad0c00640 1 --2- 172.21.15.27:0/2590819498 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbaac0674c0 0x7fbaac069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:39.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.896+0000 7fbad0c00640 1 -- 172.21.15.27:0/2590819498 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5027+0+0 (secure 0 0 0) 0x7fbacc0838b0 con 0x7fbad41523f0 2024-01-15T18:15:39.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.896+0000 7fbad8c00640 1 --2- 172.21.15.27:0/2590819498 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbaac0674c0 0x7fbaac069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:39.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.896+0000 7fbad8c00640 1 --2- 172.21.15.27:0/2590819498 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbaac0674c0 0x7fbaac069980 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fbac40b3000 tx=0x7fbac4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:39.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:39.900+0000 7fbad0c00640 1 -- 172.21.15.27:0/2590819498 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbacc052390 con 0x7fbad41523f0 2024-01-15T18:15:40.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.088+0000 7fbadb600640 1 -- 172.21.15.27:0/2590819498 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbad40ca0f0 con 0x7fbad41523f0 2024-01-15T18:15:40.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.088+0000 7fbad0c00640 1 -- 172.21.15.27:0/2590819498 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+3693 (secure 0 0 0) 0x7fbacc0559b0 con 0x7fbad41523f0 2024-01-15T18:15:40.091 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:15:40.091 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":50,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6889,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6835/914017554","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":914017554},{"type":"v1","addr":"172.21.15.93:6835","nonce":914017554}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":6892,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/4099412077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6836","nonce":4099412077},{"type":"v1","addr":"172.21.15.93:6837","nonce":4099412077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42},{"gid":7215,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/902229728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":902229728},{"type":"v1","addr":"172.21.15.27:6833","nonce":902229728}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":8262,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3388033973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":3388033973},{"type":"v1","addr":"172.21.15.27:6835","nonce":3388033973}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[]} 2024-01-15T18:15:40.091 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 50 2024-01-15T18:15:40.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.092+0000 7fbadb600640 1 -- 172.21.15.27:0/2590819498 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbaac0674c0 msgr2=0x7fbaac069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:40.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.092+0000 7fbadb600640 1 --2- 172.21.15.27:0/2590819498 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbaac0674c0 0x7fbaac069980 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fbac40b3000 tx=0x7fbac4008040 comp rx=0 tx=0).stop 2024-01-15T18:15:40.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.092+0000 7fbadb600640 1 -- 172.21.15.27:0/2590819498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad41523f0 msgr2=0x7fbad4152870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:40.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.092+0000 7fbadb600640 1 --2- 172.21.15.27:0/2590819498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad41523f0 0x7fbad4152870 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fbacc007e40 tx=0x7fbacc00e4b0 comp rx=0 tx=0).stop 2024-01-15T18:15:40.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.092+0000 7fbadb600640 1 -- 172.21.15.27:0/2590819498 shutdown_connections 2024-01-15T18:15:40.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.092+0000 7fbadb600640 1 --2- 172.21.15.27:0/2590819498 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbaac0674c0 0x7fbaac069980 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:40.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.092+0000 7fbadb600640 1 --2- 172.21.15.27:0/2590819498 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad41523f0 0x7fbad4152870 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:40.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.092+0000 7fbadb600640 1 --2- 172.21.15.27:0/2590819498 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbad4159bf0 0x7fbad4151eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:40.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.092+0000 7fbadb600640 1 --2- 172.21.15.27:0/2590819498 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbad406d430 0x7fbad40c48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:40.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.092+0000 7fbadb600640 1 -- 172.21.15.27:0/2590819498 >> 172.21.15.27:0/2590819498 conn(0x7fbad40cec60 msgr2=0x7fbad4158f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:40.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.092+0000 7fbadb600640 1 -- 172.21.15.27:0/2590819498 shutdown_connections 2024-01-15T18:15:40.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.092+0000 7fbadb600640 1 -- 172.21.15.27:0/2590819498 wait complete. 2024-01-15T18:15:40.113 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-15T18:15:40.113 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-15T18:15:40.113 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-15T18:15:40.114 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:15:40.114 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-15T18:15:40.114 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-15T18:15:40.114 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-15T18:15:40.114 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:15:40.114 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-15T18:15:40.114 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-15T18:15:40.115 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-15T18:15:40.115 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:15:40.115 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-15T18:15:40.115 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-15T18:15:40.115 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-15T18:15:40.115 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:15:40.126 INFO:tasks.ceph.mds.a:Stopped 2024-01-15T18:15:40.127 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-15T18:15:40.128 INFO:tasks.ceph.mds.c:Stopped 2024-01-15T18:15:40.128 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-15T18:15:40.130 INFO:tasks.ceph.mds.b:Stopped 2024-01-15T18:15:40.130 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-15T18:15:40.131 INFO:tasks.ceph.mds.d:Stopped 2024-01-15T18:15:40.131 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-15T18:15:40.170 INFO:tasks.ceph.mds.a:Started 2024-01-15T18:15:40.170 DEBUG:teuthology.parallel:result is None 2024-01-15T18:15:40.172 INFO:tasks.ceph.mds.c:Started 2024-01-15T18:15:40.173 DEBUG:teuthology.parallel:result is None 2024-01-15T18:15:40.180 INFO:tasks.ceph.mds.b:Started 2024-01-15T18:15:40.180 DEBUG:teuthology.parallel:result is None 2024-01-15T18:15:40.181 INFO:tasks.ceph.mds.d:Started 2024-01-15T18:15:40.182 DEBUG:teuthology.parallel:result is None 2024-01-15T18:15:40.182 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-15T18:15:40.257 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-15T18:15:40.260 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-15T18:15:40.270 INFO:tasks.ceph.mds.d.smithi093.stdout:starting mds.d at 2024-01-15T18:15:40.270 INFO:tasks.ceph.mds.b.smithi093.stdout:starting mds.b at 2024-01-15T18:15:40.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.332+0000 7f888e200640 1 Processor -- start 2024-01-15T18:15:40.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.332+0000 7f888e200640 1 -- start start 2024-01-15T18:15:40.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.332+0000 7f888e200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f88880b35a0 0x7f88880cc700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:40.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.332+0000 7f888e200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888809a700 con 0x7f888809a120 2024-01-15T18:15:40.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.332+0000 7f888e200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88880cd590 con 0x7f888815e7d0 2024-01-15T18:15:40.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.332+0000 7f888e200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88880cd870 con 0x7f88880b35a0 2024-01-15T18:15:40.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.332+0000 7f8886a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f888809a120 0x7f88880cbff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55706/0 (socket says 172.21.15.27:55706) 2024-01-15T18:15:40.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.332+0000 7f8886a00640 1 -- 172.21.15.27:0/2563760186 learned_addr learned my addr 172.21.15.27:0/2563760186 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:40.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.332+0000 7f8887e00640 1 --2- 172.21.15.27:0/2563760186 >> v2:172.21.15.93:3300/0 conn(0x7f88880b35a0 0x7f88880cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:40.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.332+0000 7f8887e00640 1 -- 172.21.15.27:0/2563760186 >> v1:172.21.15.93:6789/0 conn(0x7f888815e7d0 legacy=0x7f8888099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:40.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.332+0000 7f8887e00640 1 -- 172.21.15.27:0/2563760186 >> v1:172.21.15.27:6789/0 conn(0x7f888809a120 legacy=0x7f88880cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:40.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.332+0000 7f8886000640 1 -- 172.21.15.27:0/2563760186 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366646962 0 0) 0x7f888809a700 con 0x7f888809a120 2024-01-15T18:15:40.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.332+0000 7f8887e00640 1 -- 172.21.15.27:0/2563760186 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88880cfa00 con 0x7f88880b35a0 2024-01-15T18:15:40.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.336+0000 7f8887e00640 1 --2- 172.21.15.27:0/2563760186 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88880b35a0 0x7f88880cc700 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f8880009d90 tx=0x7f888001c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac0012c441a27646 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:40.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.336+0000 7f8886000640 1 -- 172.21.15.27:0/2563760186 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88800a5a00 con 0x7f88880b35a0 2024-01-15T18:15:40.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.336+0000 7f8886000640 1 -- 172.21.15.27:0/2563760186 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88800a5b60 con 0x7f88880b35a0 2024-01-15T18:15:40.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.336+0000 7f8886000640 1 -- 172.21.15.27:0/2563760186 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88800ad440 con 0x7f88880b35a0 2024-01-15T18:15:40.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.336+0000 7f888e200640 1 -- 172.21.15.27:0/2563760186 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88880b35a0 msgr2=0x7f88880cc700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:40.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.336+0000 7f888e200640 1 --2- 172.21.15.27:0/2563760186 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88880b35a0 0x7f88880cc700 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f8880009d90 tx=0x7f888001c920 comp rx=0 tx=0).stop 2024-01-15T18:15:40.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.336+0000 7f888e200640 1 -- 172.21.15.27:0/2563760186 shutdown_connections 2024-01-15T18:15:40.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.336+0000 7f888e200640 1 --2- 172.21.15.27:0/2563760186 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88880b35a0 0x7f88880cc700 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:40.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.336+0000 7f888e200640 1 -- 172.21.15.27:0/2563760186 >> 172.21.15.27:0/2563760186 conn(0x7f88880b2a80 msgr2=0x7f88880b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:40.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.336+0000 7f888e200640 1 -- 172.21.15.27:0/2563760186 shutdown_connections 2024-01-15T18:15:40.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.336+0000 7f888e200640 1 -- 172.21.15.27:0/2563760186 wait complete. 2024-01-15T18:15:40.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f888e200640 1 Processor -- start 2024-01-15T18:15:40.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f888e200640 1 -- start start 2024-01-15T18:15:40.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f888e200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f888809a120 0x7f88881fa560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:40.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f888e200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88880b35a0 0x7f8888151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:40.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f888e200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f888815e7d0 0x7f88881f6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:40.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f888e200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88881f63d0 con 0x7f888815e7d0 2024-01-15T18:15:40.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f888e200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8888165c20 con 0x7f88880b35a0 2024-01-15T18:15:40.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f888e200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8888165f00 con 0x7f888809a120 2024-01-15T18:15:40.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f8887e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f888815e7d0 0x7f88881f6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f8887e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f888815e7d0 0x7f88881f6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45168/0 (socket says 172.21.15.27:45168) 2024-01-15T18:15:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f8887e00640 1 -- 172.21.15.27:0/2756602696 learned_addr learned my addr 172.21.15.27:0/2756602696 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f8887400640 1 --2- 172.21.15.27:0/2756602696 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f888809a120 0x7f88881fa560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f8886a00640 1 --2- 172.21.15.27:0/2756602696 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88880b35a0 0x7f8888151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f8887e00640 1 -- 172.21.15.27:0/2756602696 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f888809a120 msgr2=0x7f88881fa560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f8887e00640 1 --2- 172.21.15.27:0/2756602696 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f888809a120 0x7f88881fa560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f8887e00640 1 -- 172.21.15.27:0/2756602696 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88880b35a0 msgr2=0x7f8888151fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f8887e00640 1 --2- 172.21.15.27:0/2756602696 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88880b35a0 0x7f8888151fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f8887e00640 1 -- 172.21.15.27:0/2756602696 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8880009a70 con 0x7f888815e7d0 2024-01-15T18:15:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f8887400640 1 --2- 172.21.15.27:0/2756602696 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f888809a120 0x7f88881fa560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f8886a00640 1 --2- 172.21.15.27:0/2756602696 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88880b35a0 0x7f8888151fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:40.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f8887e00640 1 --2- 172.21.15.27:0/2756602696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f888815e7d0 0x7f88881f6bb0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f88800b0040 tx=0x7f88800196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:40.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f887f400640 1 -- 172.21.15.27:0/2756602696 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88800ade40 con 0x7f888815e7d0 2024-01-15T18:15:40.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f887f400640 1 -- 172.21.15.27:0/2756602696 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88800b9450 con 0x7f888815e7d0 2024-01-15T18:15:40.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f888e200640 1 -- 172.21.15.27:0/2756602696 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88881661e0 con 0x7f888815e7d0 2024-01-15T18:15:40.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f887f400640 1 -- 172.21.15.27:0/2756602696 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88800b9800 con 0x7f888815e7d0 2024-01-15T18:15:40.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.340+0000 7f888e200640 1 -- 172.21.15.27:0/2756602696 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88881524e0 con 0x7f888815e7d0 2024-01-15T18:15:40.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.344+0000 7f888e200640 1 -- 172.21.15.27:0/2756602696 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f888804e850 con 0x7f888815e7d0 2024-01-15T18:15:40.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.344+0000 7f887f400640 1 -- 172.21.15.27:0/2756602696 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7f88800b9960 con 0x7f888815e7d0 2024-01-15T18:15:40.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.344+0000 7f887f400640 1 --2- 172.21.15.27:0/2756602696 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88600674f0 0x7f88600699b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:40.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.344+0000 7f887f400640 1 -- 172.21.15.27:0/2756602696 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f888011e7c0 con 0x7f888815e7d0 2024-01-15T18:15:40.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.344+0000 7f8887400640 1 --2- 172.21.15.27:0/2756602696 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88600674f0 0x7f88600699b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:40.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.344+0000 7f8887400640 1 --2- 172.21.15.27:0/2756602696 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88600674f0 0x7f88600699b0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f8888153c00 tx=0x7f8878009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:40.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.348+0000 7f887f400640 1 -- 172.21.15.27:0/2756602696 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f88800ed310 con 0x7f888815e7d0 2024-01-15T18:15:40.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:40.516+0000 7f888e200640 1 -- 172.21.15.27:0/2756602696 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f88881f6860 con 0x7f888815e7d0 2024-01-15T18:15:41.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.128+0000 7f887f400640 1 -- 172.21.15.27:0/2756602696 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v78) v1 ==== 97+0+0 (secure 0 0 0) 0x7f88800f5e60 con 0x7f888815e7d0 2024-01-15T18:15:41.130 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-01-15T18:15:41.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.132+0000 7f888e200640 1 -- 172.21.15.27:0/2756602696 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88600674f0 msgr2=0x7f88600699b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:41.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.132+0000 7f888e200640 1 --2- 172.21.15.27:0/2756602696 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88600674f0 0x7f88600699b0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f8888153c00 tx=0x7f8878009040 comp rx=0 tx=0).stop 2024-01-15T18:15:41.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.132+0000 7f888e200640 1 -- 172.21.15.27:0/2756602696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f888815e7d0 msgr2=0x7f88881f6bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:41.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.132+0000 7f888e200640 1 --2- 172.21.15.27:0/2756602696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f888815e7d0 0x7f88881f6bb0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f88800b0040 tx=0x7f88800196c0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.132+0000 7f888e200640 1 -- 172.21.15.27:0/2756602696 shutdown_connections 2024-01-15T18:15:41.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.132+0000 7f888e200640 1 --2- 172.21.15.27:0/2756602696 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88600674f0 0x7f88600699b0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.132+0000 7f888e200640 1 --2- 172.21.15.27:0/2756602696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f888815e7d0 0x7f88881f6bb0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.132+0000 7f888e200640 1 --2- 172.21.15.27:0/2756602696 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88880b35a0 0x7f8888151fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.132+0000 7f888e200640 1 --2- 172.21.15.27:0/2756602696 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f888809a120 0x7f88881fa560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.132+0000 7f888e200640 1 -- 172.21.15.27:0/2756602696 >> 172.21.15.27:0/2756602696 conn(0x7f88880b2a80 msgr2=0x7f88880b2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:41.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.132+0000 7f888e200640 1 -- 172.21.15.27:0/2756602696 shutdown_connections 2024-01-15T18:15:41.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.132+0000 7f888e200640 1 -- 172.21.15.27:0/2756602696 wait complete. 2024-01-15T18:15:41.159 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-15T18:15:41.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.304+0000 7fbf71a00640 1 Processor -- start 2024-01-15T18:15:41.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.304+0000 7fbf71a00640 1 -- start start 2024-01-15T18:15:41.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.304+0000 7fbf71a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fbf6c0701a0 0x7fbf6c070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:41.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.304+0000 7fbf71a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbf6c0c92b0 0x7fbf6c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:41.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.304+0000 7fbf71a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf6c06e240 0x7fbf6c15d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:41.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.304+0000 7fbf71a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf6c06c4a0 con 0x7fbf6c0c92b0 2024-01-15T18:15:41.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.304+0000 7fbf71a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf6c070ac0 con 0x7fbf6c0701a0 2024-01-15T18:15:41.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.304+0000 7fbf71a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c15c660 con 0x7fbf6c06e240 2024-01-15T18:15:41.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.304+0000 7fbf6a000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbf6c0c92b0 0x7fbf6c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:41.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.304+0000 7fbf6a000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbf6c0c92b0 0x7fbf6c06dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45172/0 (socket says 172.21.15.27:45172) 2024-01-15T18:15:41.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.304+0000 7fbf6a000640 1 -- 172.21.15.27:0/1327182538 learned_addr learned my addr 172.21.15.27:0/1327182538 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:41.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.304+0000 7fbf6b400640 1 --2- 172.21.15.27:0/1327182538 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf6c06e240 0x7fbf6c15d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:41.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.304+0000 7fbf6a000640 1 -- 172.21.15.27:0/1327182538 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf6c06e240 msgr2=0x7fbf6c15d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:41.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.304+0000 7fbf6aa00640 1 --2- 172.21.15.27:0/1327182538 >> v2:172.21.15.93:3300/0 conn(0x7fbf6c0701a0 0x7fbf6c070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:41.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.304+0000 7fbf6a000640 1 --2- 172.21.15.27:0/1327182538 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf6c06e240 0x7fbf6c15d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.304+0000 7fbf6a000640 1 -- 172.21.15.27:0/1327182538 >> v2:172.21.15.93:3300/0 conn(0x7fbf6c0701a0 msgr2=0x7fbf6c070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:41.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.304+0000 7fbf6a000640 1 --2- 172.21.15.27:0/1327182538 >> v2:172.21.15.93:3300/0 conn(0x7fbf6c0701a0 0x7fbf6c070580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.304+0000 7fbf6a000640 1 -- 172.21.15.27:0/1327182538 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf6c15c940 con 0x7fbf6c0c92b0 2024-01-15T18:15:41.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.304+0000 7fbf6b400640 1 --2- 172.21.15.27:0/1327182538 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf6c06e240 0x7fbf6c15d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:41.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.304+0000 7fbf6aa00640 1 --2- 172.21.15.27:0/1327182538 >> v2:172.21.15.93:3300/0 conn(0x7fbf6c0701a0 0x7fbf6c070580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:41.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.308+0000 7fbf6a000640 1 --2- 172.21.15.27:0/1327182538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6c0c92b0 0x7fbf6c06dd00 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c0681f0 tx=0x7fbf5401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a17ace3685e78e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:41.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.308+0000 7fbf69600640 1 -- 172.21.15.27:0/1327182538 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf540aa5f0 con 0x7fbf6c0c92b0 2024-01-15T18:15:41.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.308+0000 7fbf69600640 1 -- 172.21.15.27:0/1327182538 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf540aa750 con 0x7fbf6c0c92b0 2024-01-15T18:15:41.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.308+0000 7fbf69600640 1 -- 172.21.15.27:0/1327182538 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf540aaae0 con 0x7fbf6c0c92b0 2024-01-15T18:15:41.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.308+0000 7fbf71a00640 1 -- 172.21.15.27:0/1327182538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6c0c92b0 msgr2=0x7fbf6c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:41.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.308+0000 7fbf71a00640 1 --2- 172.21.15.27:0/1327182538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6c0c92b0 0x7fbf6c06dd00 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c0681f0 tx=0x7fbf5401c4d0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.308+0000 7fbf71a00640 1 -- 172.21.15.27:0/1327182538 shutdown_connections 2024-01-15T18:15:41.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.308+0000 7fbf71a00640 1 --2- 172.21.15.27:0/1327182538 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf6c06e240 0x7fbf6c15d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.308+0000 7fbf71a00640 1 --2- 172.21.15.27:0/1327182538 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6c0c92b0 0x7fbf6c06dd00 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.308+0000 7fbf71a00640 1 --2- 172.21.15.27:0/1327182538 >> v2:172.21.15.93:3300/0 conn(0x7fbf6c0701a0 0x7fbf6c070580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.308+0000 7fbf71a00640 1 -- 172.21.15.27:0/1327182538 >> 172.21.15.27:0/1327182538 conn(0x7fbf6c0cec60 msgr2=0x7fbf6c0cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:41.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.308+0000 7fbf71a00640 1 -- 172.21.15.27:0/1327182538 shutdown_connections 2024-01-15T18:15:41.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.308+0000 7fbf71a00640 1 -- 172.21.15.27:0/1327182538 wait complete. 2024-01-15T18:15:41.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf71a00640 1 Processor -- start 2024-01-15T18:15:41.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf71a00640 1 -- start start 2024-01-15T18:15:41.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf71a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf6c06e240 0x7fbf6c14a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:41.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf71a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6c0701a0 0x7fbf6c0ae470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:41.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf71a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf6c0c92b0 0x7fbf6c0cce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:41.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf71a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c14acc0 con 0x7fbf6c0701a0 2024-01-15T18:15:41.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf71a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c0ad6b0 con 0x7fbf6c0c92b0 2024-01-15T18:15:41.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf71a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c0ad990 con 0x7fbf6c06e240 2024-01-15T18:15:41.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf6a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6c0701a0 0x7fbf6c0ae470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:41.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf6a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6c0701a0 0x7fbf6c0ae470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45182/0 (socket says 172.21.15.27:45182) 2024-01-15T18:15:41.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf6a000640 1 -- 172.21.15.27:0/2035545368 learned_addr learned my addr 172.21.15.27:0/2035545368 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:41.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf6b400640 1 --2- 172.21.15.27:0/2035545368 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf6c0c92b0 0x7fbf6c0cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:41.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf6aa00640 1 --2- 172.21.15.27:0/2035545368 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf6c06e240 0x7fbf6c14a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:41.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf6a000640 1 -- 172.21.15.27:0/2035545368 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf6c06e240 msgr2=0x7fbf6c14a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:41.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf6a000640 1 --2- 172.21.15.27:0/2035545368 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf6c06e240 0x7fbf6c14a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf6a000640 1 -- 172.21.15.27:0/2035545368 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf6c0c92b0 msgr2=0x7fbf6c0cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:41.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf6a000640 1 --2- 172.21.15.27:0/2035545368 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf6c0c92b0 0x7fbf6c0cce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf6a000640 1 -- 172.21.15.27:0/2035545368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf540094d0 con 0x7fbf6c0701a0 2024-01-15T18:15:41.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf6aa00640 1 --2- 172.21.15.27:0/2035545368 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf6c06e240 0x7fbf6c14a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:41.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf6a000640 1 --2- 172.21.15.27:0/2035545368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6c0701a0 0x7fbf6c0ae470 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fbf5401ab80 tx=0x7fbf5401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:41.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf62a00640 1 -- 172.21.15.27:0/2035545368 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf540b3800 con 0x7fbf6c0701a0 2024-01-15T18:15:41.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf62a00640 1 -- 172.21.15.27:0/2035545368 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf540b3980 con 0x7fbf6c0701a0 2024-01-15T18:15:41.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf71a00640 1 -- 172.21.15.27:0/2035545368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf6c0adc70 con 0x7fbf6c0701a0 2024-01-15T18:15:41.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf62a00640 1 -- 172.21.15.27:0/2035545368 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf540b3d30 con 0x7fbf6c0701a0 2024-01-15T18:15:41.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf71a00640 1 -- 172.21.15.27:0/2035545368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf6c0ad310 con 0x7fbf6c0701a0 2024-01-15T18:15:41.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.312+0000 7fbf71a00640 1 -- 172.21.15.27:0/2035545368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf6c04e850 con 0x7fbf6c0701a0 2024-01-15T18:15:41.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.316+0000 7fbf62a00640 1 -- 172.21.15.27:0/2035545368 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7fbf540b2380 con 0x7fbf6c0701a0 2024-01-15T18:15:41.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.316+0000 7fbf62a00640 1 --2- 172.21.15.27:0/2035545368 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbf440674f0 0x7fbf440699b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:41.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.316+0000 7fbf62a00640 1 -- 172.21.15.27:0/2035545368 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4941+0+0 (secure 0 0 0) 0x7fbf540a7070 con 0x7fbf6c0701a0 2024-01-15T18:15:41.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.316+0000 7fbf6aa00640 1 --2- 172.21.15.27:0/2035545368 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbf440674f0 0x7fbf440699b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:41.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.316+0000 7fbf6aa00640 1 --2- 172.21.15.27:0/2035545368 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbf440674f0 0x7fbf440699b0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c006fd0 tx=0x7fbf5c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:41.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.320+0000 7fbf62a00640 1 -- 172.21.15.27:0/2035545368 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbf540ebd10 con 0x7fbf6c0701a0 2024-01-15T18:15:41.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.532+0000 7fbf71a00640 1 -- 172.21.15.27:0/2035545368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fbf6c0ae120 con 0x7fbf6c0701a0 2024-01-15T18:15:41.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.532+0000 7fbf62a00640 1 -- 172.21.15.27:0/2035545368 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fbf540c3020 con 0x7fbf6c0701a0 2024-01-15T18:15:41.534 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:15:41.534 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:15:41.534 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-01-15T18:15:41.534 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:41.534 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-01-15T18:15:41.534 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlbMZIGBAAhSU4y2DGJd3MH0v6v3LsDg==", 2024-01-15T18:15:41.534 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:15:41.535 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:15:41.535 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:15:41.535 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:15:41.535 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:15:41.535 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:41.535 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:41.535 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:41.535 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-01-15T18:15:41.535 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlC7B9JBAAsXmBjV4KwPZejOmowXO2Zg==", 2024-01-15T18:15:41.535 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:15:41.535 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:15:41.535 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:15:41.535 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:15:41.535 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:15:41.536 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:41.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:41.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:41.536 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-01-15T18:15:41.536 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6Vla6a3HxAA/AHc6FSTccj0JGwiR4Vl7g==", 2024-01-15T18:15:41.536 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:15:41.536 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:15:41.536 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:15:41.536 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:15:41.536 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:15:41.536 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:41.536 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:41.536 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:41.537 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-01-15T18:15:41.537 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6Vl9nqzKxAAx2aDA8UwidAUrU9DBxGldA==", 2024-01-15T18:15:41.537 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:15:41.537 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:15:41.537 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:15:41.537 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:15:41.537 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:15:41.537 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:41.537 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:41.537 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:41.537 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-01-15T18:15:41.537 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCSc6Vly4TACRAAYBmMIcsZsRp4CZ5a3xCWgA==", 2024-01-15T18:15:41.537 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:15:41.537 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:15:41.538 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:15:41.538 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:15:41.538 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:41.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:41.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:41.538 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-01-15T18:15:41.538 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCVc6VlW/HEERAAOBgPz+eGpHwNXDe8fkYo7w==", 2024-01-15T18:15:41.538 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:15:41.538 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:15:41.538 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:15:41.538 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:15:41.538 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:41.538 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:41.538 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:41.539 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-01-15T18:15:41.539 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCYc6Vl+Ec4GBAAWRe/6u7u4RXApd57H8ihng==", 2024-01-15T18:15:41.539 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:15:41.539 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:15:41.539 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:15:41.539 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:15:41.539 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:41.539 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:41.539 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:41.539 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-01-15T18:15:41.539 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCbc6VlNdMPHxAAHzJvzLltsTl1wrEnI/iyrQ==", 2024-01-15T18:15:41.539 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:15:41.539 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:15:41.539 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:15:41.539 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:15:41.540 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:41.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:41.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:41.540 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-01-15T18:15:41.540 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQChc6Vlg8eVKBAAwZWAncCE2Lra+CxmSIxZkA==", 2024-01-15T18:15:41.540 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:15:41.540 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:15:41.540 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:15:41.540 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:15:41.540 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:41.540 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:41.540 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:41.540 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-01-15T18:15:41.540 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCkc6VlIXDOLxAAFISn6ILqq/aNFK07RxNnaw==", 2024-01-15T18:15:41.541 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:15:41.541 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:15:41.541 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:15:41.541 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:15:41.541 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:41.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:41.541 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:41.541 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-01-15T18:15:41.541 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCnc6Vl9Ru2NRAAl996VBDudw23SGiFPusIcw==", 2024-01-15T18:15:41.541 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:15:41.541 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:15:41.541 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:15:41.541 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:15:41.541 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:41.541 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:41.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:41.542 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-01-15T18:15:41.542 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCrc6VlHoAsARAAIoPMridLsIcpU3I9Hr5XLQ==", 2024-01-15T18:15:41.542 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:15:41.542 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:15:41.542 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:15:41.542 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:15:41.542 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:41.542 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:41.542 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:41.542 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-01-15T18:15:41.542 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlH+DCLxAA1sKVut+3WjA/B+aYJ6/Jeg==", 2024-01-15T18:15:41.542 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:15:41.542 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:15:41.542 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-01-15T18:15:41.543 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-15T18:15:41.543 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:41.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:41.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:41.543 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-01-15T18:15:41.543 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlaRBaORAAijn3UJLHbWm8ol2NiKNSaw==", 2024-01-15T18:15:41.543 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:15:41.543 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:15:41.543 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-15T18:15:41.543 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-15T18:15:41.543 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-15T18:15:41.543 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:41.543 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:41.543 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:41.543 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-01-15T18:15:41.544 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlDbLQNRAA82Lwz+pKY+3nHPAr1DichA==", 2024-01-15T18:15:41.544 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:15:41.544 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:15:41.544 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-15T18:15:41.544 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-15T18:15:41.544 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-15T18:15:41.544 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:41.544 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:41.544 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:41.544 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-01-15T18:15:41.544 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCPc6VlOyDaAxAADea0K1OEDTfrzsM7VEkixg==", 2024-01-15T18:15:41.544 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:15:41.544 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:15:41.544 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-15T18:15:41.544 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-15T18:15:41.545 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-15T18:15:41.545 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:41.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:41.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:41.545 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-01-15T18:15:41.545 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNc6VlK70BNxAAgYXkW/pMlzy8GZoZhX9NMA==", 2024-01-15T18:15:41.545 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:15:41.545 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-15T18:15:41.545 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:15:41.545 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:15:41.545 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:15:41.545 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:41.545 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:41.545 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:41.545 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-01-15T18:15:41.545 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlQbinExAAWtUdgkUNR3cCKV+ytcbUuQ==", 2024-01-15T18:15:41.546 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:15:41.546 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-15T18:15:41.546 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-15T18:15:41.546 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:15:41.546 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:41.546 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:41.546 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:41.546 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-01-15T18:15:41.546 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlzafrDxAAVVEBuX8BDT3cB9NfD2xTdQ==", 2024-01-15T18:15:41.546 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:15:41.546 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-15T18:15:41.546 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-15T18:15:41.546 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:15:41.546 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:41.546 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:41.546 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-01-15T18:15:41.547 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:15:41.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.532+0000 7fbf71a00640 1 -- 172.21.15.27:0/2035545368 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbf440674f0 msgr2=0x7fbf440699b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:41.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.532+0000 7fbf71a00640 1 --2- 172.21.15.27:0/2035545368 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbf440674f0 0x7fbf440699b0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c006fd0 tx=0x7fbf5c008040 comp rx=0 tx=0).stop 2024-01-15T18:15:41.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.532+0000 7fbf71a00640 1 -- 172.21.15.27:0/2035545368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6c0701a0 msgr2=0x7fbf6c0ae470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:41.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.532+0000 7fbf71a00640 1 --2- 172.21.15.27:0/2035545368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6c0701a0 0x7fbf6c0ae470 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fbf5401ab80 tx=0x7fbf5401ad10 comp rx=0 tx=0).stop 2024-01-15T18:15:41.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.532+0000 7fbf71a00640 1 -- 172.21.15.27:0/2035545368 shutdown_connections 2024-01-15T18:15:41.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.532+0000 7fbf71a00640 1 --2- 172.21.15.27:0/2035545368 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbf440674f0 0x7fbf440699b0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.532+0000 7fbf71a00640 1 --2- 172.21.15.27:0/2035545368 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf6c0c92b0 0x7fbf6c0cce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.532+0000 7fbf71a00640 1 --2- 172.21.15.27:0/2035545368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6c0701a0 0x7fbf6c0ae470 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.532+0000 7fbf71a00640 1 --2- 172.21.15.27:0/2035545368 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf6c06e240 0x7fbf6c14a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.532+0000 7fbf71a00640 1 -- 172.21.15.27:0/2035545368 >> 172.21.15.27:0/2035545368 conn(0x7fbf6c0cec60 msgr2=0x7fbf6c0aef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:41.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.532+0000 7fbf71a00640 1 -- 172.21.15.27:0/2035545368 shutdown_connections 2024-01-15T18:15:41.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.532+0000 7fbf71a00640 1 -- 172.21.15.27:0/2035545368 wait complete. 2024-01-15T18:15:41.556 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:15:41.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.696+0000 7fb4d8400640 1 Processor -- start 2024-01-15T18:15:41.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.696+0000 7fb4d8400640 1 -- start start 2024-01-15T18:15:41.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.696+0000 7fb4d8400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb4d015d1d0 0x7fb4d00d1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:41.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.696+0000 7fb4d8400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb4d00ce790 0x7fb4d00cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:41.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.696+0000 7fb4d8400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4d00d3e00 con 0x7fb4d00cfb00 2024-01-15T18:15:41.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.696+0000 7fb4d8400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4d00cf110 con 0x7fb4d015d1d0 2024-01-15T18:15:41.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.696+0000 7fb4d8400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d00d0d60 con 0x7fb4d00ce790 2024-01-15T18:15:41.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.696+0000 7fb4d5a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb4d015d1d0 0x7fb4d00d1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:41.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.696+0000 7fb4d5a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb4d015d1d0 0x7fb4d00d1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45184/0 (socket says 172.21.15.27:45184) 2024-01-15T18:15:41.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.696+0000 7fb4d6400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb4d00cfb00 0x7fb4d00b5f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55712/0 (socket says 172.21.15.27:55712) 2024-01-15T18:15:41.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.696+0000 7fb4d5a00640 1 -- 172.21.15.27:0/3050438555 learned_addr learned my addr 172.21.15.27:0/3050438555 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:41.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.696+0000 7fb4d5000640 1 --2- 172.21.15.27:0/3050438555 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb4d00ce790 0x7fb4d00cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:41.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.696+0000 7fb4d5a00640 1 -- 172.21.15.27:0/3050438555 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb4d00ce790 msgr2=0x7fb4d00cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:41.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.696+0000 7fb4d5a00640 1 --2- 172.21.15.27:0/3050438555 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb4d00ce790 0x7fb4d00cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.696+0000 7fb4d5a00640 1 -- 172.21.15.27:0/3050438555 >> v1:172.21.15.27:6789/0 conn(0x7fb4d00cfb00 legacy=0x7fb4d00b5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:41.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.696+0000 7fb4d5a00640 1 -- 172.21.15.27:0/3050438555 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d00d1040 con 0x7fb4d015d1d0 2024-01-15T18:15:41.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.696+0000 7fb4d5000640 1 --2- 172.21.15.27:0/3050438555 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb4d00ce790 0x7fb4d00cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:41.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.700+0000 7fb4d5a00640 1 --2- 172.21.15.27:0/3050438555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4d015d1d0 0x7fb4d00d1e00 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0009870 tx=0x7fb4d00b6e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16b20e882ce865e4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:41.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.700+0000 7fb4cfe00640 1 -- 172.21.15.27:0/3050438555 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c00aa670 con 0x7fb4d015d1d0 2024-01-15T18:15:41.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.700+0000 7fb4cfe00640 1 -- 172.21.15.27:0/3050438555 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c00aa7d0 con 0x7fb4d015d1d0 2024-01-15T18:15:41.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.700+0000 7fb4cfe00640 1 -- 172.21.15.27:0/3050438555 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c00aab60 con 0x7fb4d015d1d0 2024-01-15T18:15:41.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.700+0000 7fb4d8400640 1 -- 172.21.15.27:0/3050438555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4d015d1d0 msgr2=0x7fb4d00d1e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:41.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.700+0000 7fb4d8400640 1 --2- 172.21.15.27:0/3050438555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4d015d1d0 0x7fb4d00d1e00 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0009870 tx=0x7fb4d00b6e60 comp rx=0 tx=0).stop 2024-01-15T18:15:41.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.700+0000 7fb4d8400640 1 -- 172.21.15.27:0/3050438555 shutdown_connections 2024-01-15T18:15:41.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.700+0000 7fb4d8400640 1 --2- 172.21.15.27:0/3050438555 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb4d00ce790 0x7fb4d00cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.700+0000 7fb4d8400640 1 --2- 172.21.15.27:0/3050438555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4d015d1d0 0x7fb4d00d1e00 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.700+0000 7fb4d8400640 1 -- 172.21.15.27:0/3050438555 >> 172.21.15.27:0/3050438555 conn(0x7fb4d009c440 msgr2=0x7fb4d00d30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:41.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.700+0000 7fb4d8400640 1 -- 172.21.15.27:0/3050438555 shutdown_connections 2024-01-15T18:15:41.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.704+0000 7fb4d8400640 1 -- 172.21.15.27:0/3050438555 wait complete. 2024-01-15T18:15:41.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.704+0000 7fb4d8400640 1 Processor -- start 2024-01-15T18:15:41.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.704+0000 7fb4d8400640 1 -- start start 2024-01-15T18:15:41.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.704+0000 7fb4d8400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb4d00ce790 0x7fb4d0159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:41.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.704+0000 7fb4d8400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4d00cfb00 0x7fb4d00cb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:41.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.704+0000 7fb4d8400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb4d015d1d0 0x7fb4d0152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:41.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.704+0000 7fb4d8400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d0153fd0 con 0x7fb4d00cfb00 2024-01-15T18:15:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.704+0000 7fb4d8400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d015aad0 con 0x7fb4d015d1d0 2024-01-15T18:15:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.704+0000 7fb4d8400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d00cbed0 con 0x7fb4d00ce790 2024-01-15T18:15:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.704+0000 7fb4d5000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4d00cfb00 0x7fb4d00cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.704+0000 7fb4d5000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4d00cfb00 0x7fb4d00cb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45188/0 (socket says 172.21.15.27:45188) 2024-01-15T18:15:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.704+0000 7fb4d5000640 1 -- 172.21.15.27:0/2356308039 learned_addr learned my addr 172.21.15.27:0/2356308039 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.704+0000 7fb4d5a00640 1 --2- 172.21.15.27:0/2356308039 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb4d00ce790 0x7fb4d0159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.704+0000 7fb4d6400640 1 --2- 172.21.15.27:0/2356308039 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb4d015d1d0 0x7fb4d0152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.704+0000 7fb4d5000640 1 -- 172.21.15.27:0/2356308039 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb4d00ce790 msgr2=0x7fb4d0159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.704+0000 7fb4d5000640 1 --2- 172.21.15.27:0/2356308039 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb4d00ce790 0x7fb4d0159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.704+0000 7fb4d5000640 1 -- 172.21.15.27:0/2356308039 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb4d015d1d0 msgr2=0x7fb4d0152600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.704+0000 7fb4d5000640 1 --2- 172.21.15.27:0/2356308039 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb4d015d1d0 0x7fb4d0152600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.704+0000 7fb4d5000640 1 -- 172.21.15.27:0/2356308039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c00094d0 con 0x7fb4d00cfb00 2024-01-15T18:15:41.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.708+0000 7fb4d5000640 1 --2- 172.21.15.27:0/2356308039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4d00cfb00 0x7fb4d00cb990 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fb4b800e810 tx=0x7fb4b800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:41.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.708+0000 7fb4cd600640 1 -- 172.21.15.27:0/2356308039 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b800c8d0 con 0x7fb4d00cfb00 2024-01-15T18:15:41.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.708+0000 7fb4cd600640 1 -- 172.21.15.27:0/2356308039 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b800ca30 con 0x7fb4d00cfb00 2024-01-15T18:15:41.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.708+0000 7fb4d8400640 1 -- 172.21.15.27:0/2356308039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4d00d04a0 con 0x7fb4d00cfb00 2024-01-15T18:15:41.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.708+0000 7fb4cd600640 1 -- 172.21.15.27:0/2356308039 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b800cdc0 con 0x7fb4d00cfb00 2024-01-15T18:15:41.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.708+0000 7fb4d8400640 1 -- 172.21.15.27:0/2356308039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4d00cc480 con 0x7fb4d00cfb00 2024-01-15T18:15:41.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.708+0000 7fb4d8400640 1 -- 172.21.15.27:0/2356308039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4d0067a10 con 0x7fb4d00cfb00 2024-01-15T18:15:41.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.708+0000 7fb4cd600640 1 -- 172.21.15.27:0/2356308039 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7fb4b8013410 con 0x7fb4d00cfb00 2024-01-15T18:15:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.708+0000 7fb4cd600640 1 --2- 172.21.15.27:0/2356308039 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb4a80674c0 0x7fb4a8069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.708+0000 7fb4cd600640 1 -- 172.21.15.27:0/2356308039 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4941+0+0 (secure 0 0 0) 0x7fb4b8082d40 con 0x7fb4d00cfb00 2024-01-15T18:15:41.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.712+0000 7fb4d5a00640 1 --2- 172.21.15.27:0/2356308039 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb4a80674c0 0x7fb4a8069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:41.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.712+0000 7fb4d5a00640 1 --2- 172.21.15.27:0/2356308039 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb4a80674c0 0x7fb4a8069980 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb4c00ae000 tx=0x7fb4c00b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:41.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.712+0000 7fb4cd600640 1 -- 172.21.15.27:0/2356308039 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb4b8051880 con 0x7fb4d00cfb00 2024-01-15T18:15:41.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.872+0000 7fb4d8400640 1 -- 172.21.15.27:0/2356308039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4d004e850 con 0x7fb4d00cfb00 2024-01-15T18:15:41.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.876+0000 7fb4cd600640 1 -- 172.21.15.27:0/2356308039 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+12894 (secure 0 0 0) 0x7fb4b8054ea0 con 0x7fb4d00cfb00 2024-01-15T18:15:41.877 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:15:41.877 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":80,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:15:41.147736+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:15:41.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.876+0000 7fb4d8400640 1 -- 172.21.15.27:0/2356308039 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb4a80674c0 msgr2=0x7fb4a8069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:41.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.876+0000 7fb4d8400640 1 --2- 172.21.15.27:0/2356308039 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb4a80674c0 0x7fb4a8069980 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb4c00ae000 tx=0x7fb4c00b5040 comp rx=0 tx=0).stop 2024-01-15T18:15:41.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.876+0000 7fb4d8400640 1 -- 172.21.15.27:0/2356308039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4d00cfb00 msgr2=0x7fb4d00cb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:41.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.876+0000 7fb4d8400640 1 --2- 172.21.15.27:0/2356308039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4d00cfb00 0x7fb4d00cb990 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fb4b800e810 tx=0x7fb4b800ecd0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.876+0000 7fb4d8400640 1 -- 172.21.15.27:0/2356308039 shutdown_connections 2024-01-15T18:15:41.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.876+0000 7fb4d8400640 1 --2- 172.21.15.27:0/2356308039 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb4a80674c0 0x7fb4a8069980 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.876+0000 7fb4d8400640 1 --2- 172.21.15.27:0/2356308039 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb4d015d1d0 0x7fb4d0152600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.876+0000 7fb4d8400640 1 --2- 172.21.15.27:0/2356308039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4d00cfb00 0x7fb4d00cb990 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.876+0000 7fb4d8400640 1 --2- 172.21.15.27:0/2356308039 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb4d00ce790 0x7fb4d0159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:41.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.876+0000 7fb4d8400640 1 -- 172.21.15.27:0/2356308039 >> 172.21.15.27:0/2356308039 conn(0x7fb4d009c440 msgr2=0x7fb4d0156c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:41.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.876+0000 7fb4d8400640 1 -- 172.21.15.27:0/2356308039 shutdown_connections 2024-01-15T18:15:41.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:41.876+0000 7fb4d8400640 1 -- 172.21.15.27:0/2356308039 wait complete. 2024-01-15T18:15:41.899 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:15:41.900 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:15:42.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.040+0000 7fdd52400640 1 Processor -- start 2024-01-15T18:15:42.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.040+0000 7fdd52400640 1 -- start start 2024-01-15T18:15:42.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.044+0000 7fdd52400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdd4c158900 0x7fdd4c15f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:42.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.044+0000 7fdd52400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fdd4c15bf00 0x7fdd4c0cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:42.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.044+0000 7fdd52400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd4c0b1cb0 con 0x7fdd4c0cf110 2024-01-15T18:15:42.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.044+0000 7fdd52400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd4c0cf580 con 0x7fdd4c158900 2024-01-15T18:15:42.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.044+0000 7fdd52400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd4c0b1f00 con 0x7fdd4c15bf00 2024-01-15T18:15:42.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.044+0000 7fdd4b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdd4c158900 0x7fdd4c15f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:42.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.044+0000 7fdd4b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdd4c158900 0x7fdd4c15f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45196/0 (socket says 172.21.15.27:45196) 2024-01-15T18:15:42.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.044+0000 7fdd4b400640 1 -- 172.21.15.27:0/684587465 learned_addr learned my addr 172.21.15.27:0/684587465 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:42.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.044+0000 7fdd4aa00640 1 --2- 172.21.15.27:0/684587465 >> v2:172.21.15.93:3300/0 conn(0x7fdd4c15bf00 0x7fdd4c0cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:42.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.044+0000 7fdd4b400640 1 -- 172.21.15.27:0/684587465 >> v2:172.21.15.93:3300/0 conn(0x7fdd4c15bf00 msgr2=0x7fdd4c0cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:42.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.044+0000 7fdd4b400640 1 --2- 172.21.15.27:0/684587465 >> v2:172.21.15.93:3300/0 conn(0x7fdd4c15bf00 0x7fdd4c0cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:42.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.044+0000 7fdd4b400640 1 -- 172.21.15.27:0/684587465 >> v1:172.21.15.93:6789/0 conn(0x7fdd4c0cf110 legacy=0x7fdd4c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:42.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.044+0000 7fdd4b400640 1 -- 172.21.15.27:0/684587465 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd4c0b21e0 con 0x7fdd4c158900 2024-01-15T18:15:42.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.044+0000 7fdd4b400640 1 --2- 172.21.15.27:0/684587465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd4c158900 0x7fdd4c15f550 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c009870 tx=0x7fdd3c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0fa32939036cb73 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:42.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.048+0000 7fdd4a000640 1 -- 172.21.15.27:0/684587465 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c0ab5f0 con 0x7fdd4c158900 2024-01-15T18:15:42.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.048+0000 7fdd4a000640 1 -- 172.21.15.27:0/684587465 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd3c0ab750 con 0x7fdd4c158900 2024-01-15T18:15:42.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.048+0000 7fdd4a000640 1 -- 172.21.15.27:0/684587465 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c0abae0 con 0x7fdd4c158900 2024-01-15T18:15:42.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.048+0000 7fdd52400640 1 -- 172.21.15.27:0/684587465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd4c158900 msgr2=0x7fdd4c15f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:42.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.048+0000 7fdd52400640 1 --2- 172.21.15.27:0/684587465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd4c158900 0x7fdd4c15f550 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c009870 tx=0x7fdd3c01c500 comp rx=0 tx=0).stop 2024-01-15T18:15:42.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.048+0000 7fdd52400640 1 -- 172.21.15.27:0/684587465 shutdown_connections 2024-01-15T18:15:42.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.048+0000 7fdd52400640 1 --2- 172.21.15.27:0/684587465 >> v2:172.21.15.93:3300/0 conn(0x7fdd4c15bf00 0x7fdd4c0cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:42.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.048+0000 7fdd52400640 1 --2- 172.21.15.27:0/684587465 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd4c158900 0x7fdd4c15f550 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:42.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.048+0000 7fdd52400640 1 -- 172.21.15.27:0/684587465 >> 172.21.15.27:0/684587465 conn(0x7fdd4c0cb700 msgr2=0x7fdd4c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:42.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.048+0000 7fdd52400640 1 -- 172.21.15.27:0/684587465 shutdown_connections 2024-01-15T18:15:42.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.048+0000 7fdd52400640 1 -- 172.21.15.27:0/684587465 wait complete. 2024-01-15T18:15:42.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.048+0000 7fdd52400640 1 Processor -- start 2024-01-15T18:15:42.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.048+0000 7fdd52400640 1 -- start start 2024-01-15T18:15:42.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.048+0000 7fdd52400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd4c0cf110 0x7fdd4c14f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:42.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.048+0000 7fdd52400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd4c158900 0x7fdd4c0cde60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:42.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.048+0000 7fdd52400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd4c15bf00 0x7fdd4c151440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:42.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.048+0000 7fdd52400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c150980 con 0x7fdd4c158900 2024-01-15T18:15:42.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.048+0000 7fdd52400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c14fec0 con 0x7fdd4c15bf00 2024-01-15T18:15:42.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.048+0000 7fdd52400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c0ccdc0 con 0x7fdd4c0cf110 2024-01-15T18:15:42.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.052+0000 7fdd4aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd4c158900 0x7fdd4c0cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:42.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.052+0000 7fdd4aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd4c158900 0x7fdd4c0cde60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45210/0 (socket says 172.21.15.27:45210) 2024-01-15T18:15:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.052+0000 7fdd4aa00640 1 -- 172.21.15.27:0/4213970337 learned_addr learned my addr 172.21.15.27:0/4213970337 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.052+0000 7fdd4b400640 1 --2- 172.21.15.27:0/4213970337 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd4c0cf110 0x7fdd4c14f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.052+0000 7fdd4be00640 1 --2- 172.21.15.27:0/4213970337 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd4c15bf00 0x7fdd4c151440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.052+0000 7fdd4aa00640 1 -- 172.21.15.27:0/4213970337 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd4c0cf110 msgr2=0x7fdd4c14f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.052+0000 7fdd4aa00640 1 --2- 172.21.15.27:0/4213970337 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd4c0cf110 0x7fdd4c14f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.052+0000 7fdd4aa00640 1 -- 172.21.15.27:0/4213970337 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd4c15bf00 msgr2=0x7fdd4c151440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.052+0000 7fdd4aa00640 1 --2- 172.21.15.27:0/4213970337 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd4c15bf00 0x7fdd4c151440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.052+0000 7fdd4aa00640 1 -- 172.21.15.27:0/4213970337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd3c0094d0 con 0x7fdd4c158900 2024-01-15T18:15:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.052+0000 7fdd4b400640 1 --2- 172.21.15.27:0/4213970337 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd4c0cf110 0x7fdd4c14f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:42.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.052+0000 7fdd4aa00640 1 --2- 172.21.15.27:0/4213970337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd4c158900 0x7fdd4c0cde60 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fdd3400e870 tx=0x7fdd3400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:42.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.052+0000 7fdd43400640 1 -- 172.21.15.27:0/4213970337 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd34009510 con 0x7fdd4c158900 2024-01-15T18:15:42.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.052+0000 7fdd43400640 1 -- 172.21.15.27:0/4213970337 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd3400eec0 con 0x7fdd4c158900 2024-01-15T18:15:42.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.052+0000 7fdd52400640 1 -- 172.21.15.27:0/4213970337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd4c15c490 con 0x7fdd4c158900 2024-01-15T18:15:42.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.052+0000 7fdd43400640 1 -- 172.21.15.27:0/4213970337 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3400cb40 con 0x7fdd4c158900 2024-01-15T18:15:42.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.052+0000 7fdd52400640 1 -- 172.21.15.27:0/4213970337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd4c0cd380 con 0x7fdd4c158900 2024-01-15T18:15:42.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.052+0000 7fdd52400640 1 -- 172.21.15.27:0/4213970337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd4c04e850 con 0x7fdd4c158900 2024-01-15T18:15:42.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.052+0000 7fdd43400640 1 -- 172.21.15.27:0/4213970337 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7fdd3400cca0 con 0x7fdd4c158900 2024-01-15T18:15:42.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.056+0000 7fdd43400640 1 --2- 172.21.15.27:0/4213970337 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd240674c0 0x7fdd24069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.056+0000 7fdd43400640 1 -- 172.21.15.27:0/4213970337 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4941+0+0 (secure 0 0 0) 0x7fdd34083550 con 0x7fdd4c158900 2024-01-15T18:15:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.056+0000 7fdd4b400640 1 --2- 172.21.15.27:0/4213970337 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd240674c0 0x7fdd24069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:42.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.056+0000 7fdd4b400640 1 --2- 172.21.15.27:0/4213970337 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd240674c0 0x7fdd24069980 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c0af000 tx=0x7fdd3c0a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:42.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.060+0000 7fdd43400640 1 -- 172.21.15.27:0/4213970337 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fdd34051f80 con 0x7fdd4c158900 2024-01-15T18:15:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.216+0000 7fdd52400640 1 -- 172.21.15.27:0/4213970337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdd4c06d540 con 0x7fdd4c158900 2024-01-15T18:15:42.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.220+0000 7fdd43400640 1 -- 172.21.15.27:0/4213970337 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdd340210a0 con 0x7fdd4c158900 2024-01-15T18:15:42.221 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:15:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.220+0000 7fdd52400640 1 -- 172.21.15.27:0/4213970337 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd240674c0 msgr2=0x7fdd24069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.220+0000 7fdd52400640 1 --2- 172.21.15.27:0/4213970337 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd240674c0 0x7fdd24069980 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c0af000 tx=0x7fdd3c0a7000 comp rx=0 tx=0).stop 2024-01-15T18:15:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.220+0000 7fdd52400640 1 -- 172.21.15.27:0/4213970337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd4c158900 msgr2=0x7fdd4c0cde60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.220+0000 7fdd52400640 1 --2- 172.21.15.27:0/4213970337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd4c158900 0x7fdd4c0cde60 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fdd3400e870 tx=0x7fdd3400ec70 comp rx=0 tx=0).stop 2024-01-15T18:15:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.220+0000 7fdd52400640 1 -- 172.21.15.27:0/4213970337 shutdown_connections 2024-01-15T18:15:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.220+0000 7fdd52400640 1 --2- 172.21.15.27:0/4213970337 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd240674c0 0x7fdd24069980 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.220+0000 7fdd52400640 1 --2- 172.21.15.27:0/4213970337 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd4c15bf00 0x7fdd4c151440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.220+0000 7fdd52400640 1 --2- 172.21.15.27:0/4213970337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd4c158900 0x7fdd4c0cde60 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.220+0000 7fdd52400640 1 --2- 172.21.15.27:0/4213970337 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd4c0cf110 0x7fdd4c14f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.220+0000 7fdd52400640 1 -- 172.21.15.27:0/4213970337 >> 172.21.15.27:0/4213970337 conn(0x7fdd4c0cb700 msgr2=0x7fdd4c0700e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.220+0000 7fdd52400640 1 -- 172.21.15.27:0/4213970337 shutdown_connections 2024-01-15T18:15:42.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.220+0000 7fdd52400640 1 -- 172.21.15.27:0/4213970337 wait complete. 2024-01-15T18:15:42.242 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-15T18:15:42.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.380+0000 7f70e3a00640 1 Processor -- start 2024-01-15T18:15:42.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.380+0000 7f70e3a00640 1 -- start start 2024-01-15T18:15:42.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.384+0000 7f70e3a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f70dc159bf0 0x7f70dc159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:42.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.384+0000 7f70e3a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70dc159a50 con 0x7f70dc071e40 2024-01-15T18:15:42.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.384+0000 7f70e3a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70dc15a510 con 0x7f70dc06d430 2024-01-15T18:15:42.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.384+0000 7f70e3a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70dc0cce00 con 0x7f70dc159bf0 2024-01-15T18:15:42.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.384+0000 7f70e1000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f70dc159bf0 0x7f70dc159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:42.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.384+0000 7f70dbe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f70dc071e40 0x7f70dc0c9be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55724/0 (socket says 172.21.15.27:55724) 2024-01-15T18:15:42.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.384+0000 7f70dbe00640 1 -- 172.21.15.27:0/3043916684 learned_addr learned my addr 172.21.15.27:0/3043916684 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:42.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.384+0000 7f70e1000640 1 -- 172.21.15.27:0/3043916684 >> v1:172.21.15.93:6789/0 conn(0x7f70dc06d430 legacy=0x7f70dc0cd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:42.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.384+0000 7f70e1000640 1 -- 172.21.15.27:0/3043916684 >> v1:172.21.15.27:6789/0 conn(0x7f70dc071e40 legacy=0x7f70dc0c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:42.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.384+0000 7f70e1000640 1 -- 172.21.15.27:0/3043916684 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70dc06d980 con 0x7f70dc159bf0 2024-01-15T18:15:42.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.384+0000 7f70db400640 1 -- 172.21.15.27:0/3043916684 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363315233 0 0) 0x7f70c4001120 con 0x7f70dc071e40 2024-01-15T18:15:42.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.388+0000 7f70e1000640 1 --2- 172.21.15.27:0/3043916684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70dc159bf0 0x7f70dc159fd0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f70cc0097c0 tx=0x7f70cc01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4d036e13048bce1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:42.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.388+0000 7f70db400640 1 -- 172.21.15.27:0/3043916684 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70cc0a8970 con 0x7f70dc159bf0 2024-01-15T18:15:42.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.388+0000 7f70db400640 1 -- 172.21.15.27:0/3043916684 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70cc0a8ad0 con 0x7f70dc159bf0 2024-01-15T18:15:42.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.388+0000 7f70db400640 1 -- 172.21.15.27:0/3043916684 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70cc0a8e60 con 0x7f70dc159bf0 2024-01-15T18:15:42.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.388+0000 7f70e3a00640 1 -- 172.21.15.27:0/3043916684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70dc159bf0 msgr2=0x7f70dc159fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:42.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.388+0000 7f70e3a00640 1 --2- 172.21.15.27:0/3043916684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70dc159bf0 0x7f70dc159fd0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f70cc0097c0 tx=0x7f70cc01ec70 comp rx=0 tx=0).stop 2024-01-15T18:15:42.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.388+0000 7f70e3a00640 1 -- 172.21.15.27:0/3043916684 shutdown_connections 2024-01-15T18:15:42.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.388+0000 7f70e3a00640 1 --2- 172.21.15.27:0/3043916684 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70dc159bf0 0x7f70dc159fd0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:42.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.388+0000 7f70e3a00640 1 -- 172.21.15.27:0/3043916684 >> 172.21.15.27:0/3043916684 conn(0x7f70dc0cec60 msgr2=0x7f70dc0cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:42.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.388+0000 7f70e3a00640 1 -- 172.21.15.27:0/3043916684 shutdown_connections 2024-01-15T18:15:42.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.388+0000 7f70e3a00640 1 -- 172.21.15.27:0/3043916684 wait complete. 2024-01-15T18:15:42.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.388+0000 7f70e3a00640 1 Processor -- start 2024-01-15T18:15:42.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.388+0000 7f70e3a00640 1 -- start start 2024-01-15T18:15:42.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.392+0000 7f70e3a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70dc06d430 0x7f70dc144c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:42.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.392+0000 7f70e3a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f70dc159bf0 0x7f70dc15c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:42.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.392+0000 7f70e3a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f70dc0c53c0 0x7f70dc15d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:42.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.392+0000 7f70e3a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc15c6a0 con 0x7f70dc06d430 2024-01-15T18:15:42.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.392+0000 7f70e3a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc152220 con 0x7f70dc159bf0 2024-01-15T18:15:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.392+0000 7f70e3a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc145310 con 0x7f70dc0c53c0 2024-01-15T18:15:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.392+0000 7f70e1000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70dc06d430 0x7f70dc144c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.392+0000 7f70e1000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70dc06d430 0x7f70dc144c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45232/0 (socket says 172.21.15.27:45232) 2024-01-15T18:15:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.392+0000 7f70e1000640 1 -- 172.21.15.27:0/1034746050 learned_addr learned my addr 172.21.15.27:0/1034746050 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.392+0000 7f70dbe00640 1 --2- 172.21.15.27:0/1034746050 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f70dc159bf0 0x7f70dc15c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.392+0000 7f70e1a00640 1 --2- 172.21.15.27:0/1034746050 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f70dc0c53c0 0x7f70dc15d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.392+0000 7f70e1000640 1 -- 172.21.15.27:0/1034746050 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f70dc0c53c0 msgr2=0x7f70dc15d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.392+0000 7f70e1000640 1 --2- 172.21.15.27:0/1034746050 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f70dc0c53c0 0x7f70dc15d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.392+0000 7f70e1000640 1 -- 172.21.15.27:0/1034746050 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f70dc159bf0 msgr2=0x7f70dc15c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.392+0000 7f70e1000640 1 --2- 172.21.15.27:0/1034746050 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f70dc159bf0 0x7f70dc15c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.392+0000 7f70e1000640 1 -- 172.21.15.27:0/1034746050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70cc009420 con 0x7f70dc06d430 2024-01-15T18:15:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.392+0000 7f70e1a00640 1 --2- 172.21.15.27:0/1034746050 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f70dc0c53c0 0x7f70dc15d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.392+0000 7f70e1000640 1 --2- 172.21.15.27:0/1034746050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70dc06d430 0x7f70dc144c70 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f70cc0b3040 tx=0x7f70cc01ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.392+0000 7f70d8c00640 1 -- 172.21.15.27:0/1034746050 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70cc0b9630 con 0x7f70dc06d430 2024-01-15T18:15:42.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.392+0000 7f70d8c00640 1 -- 172.21.15.27:0/1034746050 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70cc0b97b0 con 0x7f70dc06d430 2024-01-15T18:15:42.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.392+0000 7f70d8c00640 1 -- 172.21.15.27:0/1034746050 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70cc0b9b60 con 0x7f70dc06d430 2024-01-15T18:15:42.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.392+0000 7f70e3a00640 1 -- 172.21.15.27:0/1034746050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70dc1455f0 con 0x7f70dc06d430 2024-01-15T18:15:42.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.392+0000 7f70e3a00640 1 -- 172.21.15.27:0/1034746050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70dc145bb0 con 0x7f70dc06d430 2024-01-15T18:15:42.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.392+0000 7f70e3a00640 1 -- 172.21.15.27:0/1034746050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70dc152440 con 0x7f70dc06d430 2024-01-15T18:15:42.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.392+0000 7f70d8c00640 1 -- 172.21.15.27:0/1034746050 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7f70cc0b7040 con 0x7f70dc06d430 2024-01-15T18:15:42.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.396+0000 7f70d8c00640 1 --2- 172.21.15.27:0/1034746050 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f70b40674c0 0x7f70b4069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:42.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.396+0000 7f70d8c00640 1 -- 172.21.15.27:0/1034746050 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4941+0+0 (secure 0 0 0) 0x7f70cc120f70 con 0x7f70dc06d430 2024-01-15T18:15:42.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.396+0000 7f70dbe00640 1 --2- 172.21.15.27:0/1034746050 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f70b40674c0 0x7f70b4069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:42.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.396+0000 7f70dbe00640 1 --2- 172.21.15.27:0/1034746050 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f70b40674c0 0x7f70b4069980 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f70c4007e00 tx=0x7f70c4007d90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:42.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.400+0000 7f70d8c00640 1 -- 172.21.15.27:0/1034746050 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f70dc152440 con 0x7f70dc06d430 2024-01-15T18:15:42.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.556+0000 7f70e3a00640 1 -- 172.21.15.27:0/1034746050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f70dc152440 con 0x7f70dc06d430 2024-01-15T18:15:42.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.556+0000 7f70d8c00640 1 -- 172.21.15.27:0/1034746050 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v80) v1 ==== 84+0+65 (secure 0 0 0) 0x7f70dc152440 con 0x7f70dc06d430 2024-01-15T18:15:42.560 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:15:42.560 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:15:42.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:42.560 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-01-15T18:15:42.560 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-01-15T18:15:42.560 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:42.561 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:15:42.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.560+0000 7f70e3a00640 1 -- 172.21.15.27:0/1034746050 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f70b40674c0 msgr2=0x7f70b4069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:42.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.560+0000 7f70e3a00640 1 --2- 172.21.15.27:0/1034746050 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f70b40674c0 0x7f70b4069980 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f70c4007e00 tx=0x7f70c4007d90 comp rx=0 tx=0).stop 2024-01-15T18:15:42.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.560+0000 7f70e3a00640 1 -- 172.21.15.27:0/1034746050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70dc06d430 msgr2=0x7f70dc144c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:42.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.560+0000 7f70e3a00640 1 --2- 172.21.15.27:0/1034746050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70dc06d430 0x7f70dc144c70 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f70cc0b3040 tx=0x7f70cc01ba10 comp rx=0 tx=0).stop 2024-01-15T18:15:42.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.560+0000 7f70e3a00640 1 -- 172.21.15.27:0/1034746050 shutdown_connections 2024-01-15T18:15:42.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.560+0000 7f70e3a00640 1 --2- 172.21.15.27:0/1034746050 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f70b40674c0 0x7f70b4069980 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:42.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.560+0000 7f70e3a00640 1 --2- 172.21.15.27:0/1034746050 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f70dc0c53c0 0x7f70dc15d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:42.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.560+0000 7f70e3a00640 1 --2- 172.21.15.27:0/1034746050 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f70dc159bf0 0x7f70dc15c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:42.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.560+0000 7f70e3a00640 1 --2- 172.21.15.27:0/1034746050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70dc06d430 0x7f70dc144c70 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:42.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.560+0000 7f70e3a00640 1 -- 172.21.15.27:0/1034746050 >> 172.21.15.27:0/1034746050 conn(0x7f70dc0cec60 msgr2=0x7f70dc0c9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:42.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.560+0000 7f70e3a00640 1 -- 172.21.15.27:0/1034746050 shutdown_connections 2024-01-15T18:15:42.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.560+0000 7f70e3a00640 1 -- 172.21.15.27:0/1034746050 wait complete. 2024-01-15T18:15:42.582 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-15T18:15:42.583 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-15T18:15:42.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.724+0000 7f8b8c200640 1 Processor -- start 2024-01-15T18:15:42.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.724+0000 7f8b8c200640 1 -- start start 2024-01-15T18:15:42.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.728+0000 7f8b8c200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8b8415d190 0x7f8b840cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:42.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.728+0000 7f8b8c200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8b8406c520 0x7f8b840cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:42.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.728+0000 7f8b8c200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b840b0d70 con 0x7f8b8415d570 2024-01-15T18:15:42.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.728+0000 7f8b8c200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b840d0540 con 0x7f8b8415d190 2024-01-15T18:15:42.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.728+0000 7f8b8c200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b840b0440 con 0x7f8b8406c520 2024-01-15T18:15:42.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.728+0000 7f8b89800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8b8415d190 0x7f8b840cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:42.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.728+0000 7f8b89800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8b8415d190 0x7f8b840cfcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45246/0 (socket says 172.21.15.27:45246) 2024-01-15T18:15:42.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.728+0000 7f8b89800640 1 -- 172.21.15.27:0/3667353053 learned_addr learned my addr 172.21.15.27:0/3667353053 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:42.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.728+0000 7f8b8a200640 1 --2- 172.21.15.27:0/3667353053 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8b8406c520 0x7f8b840cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:42.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.728+0000 7f8b89800640 1 -- 172.21.15.27:0/3667353053 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8b8406c520 msgr2=0x7f8b840cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:42.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.728+0000 7f8b89800640 1 --2- 172.21.15.27:0/3667353053 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8b8406c520 0x7f8b840cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:42.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.728+0000 7f8b89800640 1 -- 172.21.15.27:0/3667353053 >> v1:172.21.15.93:6789/0 conn(0x7f8b8415d570 legacy=0x7f8b8406be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:42.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.728+0000 7f8b89800640 1 -- 172.21.15.27:0/3667353053 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b840b0720 con 0x7f8b8415d190 2024-01-15T18:15:42.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.732+0000 7f8b89800640 1 --2- 172.21.15.27:0/3667353053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b8415d190 0x7f8b840cfcf0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f8b74009870 tx=0x7f8b7401c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b49a22a718029d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:42.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.732+0000 7f8b83e00640 1 -- 172.21.15.27:0/3667353053 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b740aa430 con 0x7f8b8415d190 2024-01-15T18:15:42.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.732+0000 7f8b83e00640 1 -- 172.21.15.27:0/3667353053 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b740aa590 con 0x7f8b8415d190 2024-01-15T18:15:42.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.732+0000 7f8b83e00640 1 -- 172.21.15.27:0/3667353053 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b740aa920 con 0x7f8b8415d190 2024-01-15T18:15:42.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.732+0000 7f8b8c200640 1 -- 172.21.15.27:0/3667353053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b8415d190 msgr2=0x7f8b840cfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:42.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.732+0000 7f8b8c200640 1 --2- 172.21.15.27:0/3667353053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b8415d190 0x7f8b840cfcf0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f8b74009870 tx=0x7f8b7401c510 comp rx=0 tx=0).stop 2024-01-15T18:15:42.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.732+0000 7f8b8c200640 1 -- 172.21.15.27:0/3667353053 shutdown_connections 2024-01-15T18:15:42.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.732+0000 7f8b8c200640 1 --2- 172.21.15.27:0/3667353053 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8b8406c520 0x7f8b840cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:42.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.732+0000 7f8b8c200640 1 --2- 172.21.15.27:0/3667353053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b8415d190 0x7f8b840cfcf0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:42.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.732+0000 7f8b8c200640 1 -- 172.21.15.27:0/3667353053 >> 172.21.15.27:0/3667353053 conn(0x7f8b840d0e70 msgr2=0x7f8b840d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:42.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.732+0000 7f8b8c200640 1 -- 172.21.15.27:0/3667353053 shutdown_connections 2024-01-15T18:15:42.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.732+0000 7f8b8c200640 1 -- 172.21.15.27:0/3667353053 wait complete. 2024-01-15T18:15:42.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.732+0000 7f8b8c200640 1 Processor -- start 2024-01-15T18:15:42.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.732+0000 7f8b8c200640 1 -- start start 2024-01-15T18:15:42.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.736+0000 7f8b8c200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8b8406c520 0x7f8b84145e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:42.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.736+0000 7f8b8c200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8b8415d190 0x7f8b84156820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:42.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.736+0000 7f8b8c200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b8415d570 0x7f8b84156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:42.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.736+0000 7f8b8c200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b84154700 con 0x7f8b8415d570 2024-01-15T18:15:42.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.736+0000 7f8b8c200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b840d1530 con 0x7f8b8415d190 2024-01-15T18:15:42.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.736+0000 7f8b8c200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b84154140 con 0x7f8b8406c520 2024-01-15T18:15:42.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.736+0000 7f8b8a200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b8415d570 0x7f8b84156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:42.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.736+0000 7f8b8a200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b8415d570 0x7f8b84156d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45252/0 (socket says 172.21.15.27:45252) 2024-01-15T18:15:42.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.736+0000 7f8b8a200640 1 -- 172.21.15.27:0/2416603323 learned_addr learned my addr 172.21.15.27:0/2416603323 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:42.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.736+0000 7f8b89800640 1 --2- 172.21.15.27:0/2416603323 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8b8406c520 0x7f8b84145e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:42.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.736+0000 7f8b88e00640 1 --2- 172.21.15.27:0/2416603323 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8b8415d190 0x7f8b84156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:42.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.736+0000 7f8b8a200640 1 -- 172.21.15.27:0/2416603323 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8b8406c520 msgr2=0x7f8b84145e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:42.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.736+0000 7f8b8a200640 1 --2- 172.21.15.27:0/2416603323 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8b8406c520 0x7f8b84145e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:42.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.736+0000 7f8b8a200640 1 -- 172.21.15.27:0/2416603323 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8b8415d190 msgr2=0x7f8b84156820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:42.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.736+0000 7f8b8a200640 1 --2- 172.21.15.27:0/2416603323 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8b8415d190 0x7f8b84156820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:42.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.736+0000 7f8b8a200640 1 -- 172.21.15.27:0/2416603323 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b740094d0 con 0x7f8b8415d570 2024-01-15T18:15:42.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.736+0000 7f8b8a200640 1 --2- 172.21.15.27:0/2416603323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b8415d570 0x7f8b84156d60 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c007e10 tx=0x7f8b7c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:42.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.736+0000 7f8b81600640 1 -- 172.21.15.27:0/2416603323 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c014620 con 0x7f8b8415d570 2024-01-15T18:15:42.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.736+0000 7f8b81600640 1 -- 172.21.15.27:0/2416603323 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b7c0147a0 con 0x7f8b8415d570 2024-01-15T18:15:42.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.736+0000 7f8b8c200640 1 -- 172.21.15.27:0/2416603323 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b84154420 con 0x7f8b8415d570 2024-01-15T18:15:42.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.736+0000 7f8b81600640 1 -- 172.21.15.27:0/2416603323 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c014b50 con 0x7f8b8415d570 2024-01-15T18:15:42.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.736+0000 7f8b8c200640 1 -- 172.21.15.27:0/2416603323 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b84146340 con 0x7f8b8415d570 2024-01-15T18:15:42.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.736+0000 7f8b8c200640 1 -- 172.21.15.27:0/2416603323 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b840c4c40 con 0x7f8b8415d570 2024-01-15T18:15:42.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.736+0000 7f8b81600640 1 -- 172.21.15.27:0/2416603323 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7f8b7c014cb0 con 0x7f8b8415d570 2024-01-15T18:15:42.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.740+0000 7f8b81600640 1 --2- 172.21.15.27:0/2416603323 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8b5c0674c0 0x7f8b5c069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:42.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.740+0000 7f8b81600640 1 -- 172.21.15.27:0/2416603323 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 4941+0+0 (secure 0 0 0) 0x7f8b7c0837b0 con 0x7f8b8415d570 2024-01-15T18:15:42.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.740+0000 7f8b89800640 1 --2- 172.21.15.27:0/2416603323 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8b5c0674c0 0x7f8b5c069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:42.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.740+0000 7f8b89800640 1 --2- 172.21.15.27:0/2416603323 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8b5c0674c0 0x7f8b5c069980 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8b740ae000 tx=0x7f8b740b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:42.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.744+0000 7f8b81600640 1 -- 172.21.15.27:0/2416603323 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8b840c4c40 con 0x7f8b8415d570 2024-01-15T18:15:42.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:42.900+0000 7f8b8c200640 1 -- 172.21.15.27:0/2416603323 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8b840c4c40 con 0x7f8b8415d570 2024-01-15T18:15:43.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.132+0000 7f8b81600640 1 -- 172.21.15.27:0/2416603323 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v81) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8b840c4c40 con 0x7f8b8415d570 2024-01-15T18:15:43.133 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-01-15T18:15:43.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.132+0000 7f8b8c200640 1 -- 172.21.15.27:0/2416603323 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8b5c0674c0 msgr2=0x7f8b5c069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:43.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.132+0000 7f8b8c200640 1 --2- 172.21.15.27:0/2416603323 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8b5c0674c0 0x7f8b5c069980 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8b740ae000 tx=0x7f8b740b5040 comp rx=0 tx=0).stop 2024-01-15T18:15:43.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.132+0000 7f8b8c200640 1 -- 172.21.15.27:0/2416603323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b8415d570 msgr2=0x7f8b84156d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:43.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.132+0000 7f8b8c200640 1 --2- 172.21.15.27:0/2416603323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b8415d570 0x7f8b84156d60 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c007e10 tx=0x7f8b7c00e4b0 comp rx=0 tx=0).stop 2024-01-15T18:15:43.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.132+0000 7f8b8c200640 1 -- 172.21.15.27:0/2416603323 shutdown_connections 2024-01-15T18:15:43.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.132+0000 7f8b8c200640 1 --2- 172.21.15.27:0/2416603323 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8b5c0674c0 0x7f8b5c069980 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:43.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.132+0000 7f8b8c200640 1 --2- 172.21.15.27:0/2416603323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b8415d570 0x7f8b84156d60 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:43.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.132+0000 7f8b8c200640 1 --2- 172.21.15.27:0/2416603323 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8b8415d190 0x7f8b84156820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:43.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.132+0000 7f8b8c200640 1 --2- 172.21.15.27:0/2416603323 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8b8406c520 0x7f8b84145e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:43.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.132+0000 7f8b8c200640 1 -- 172.21.15.27:0/2416603323 >> 172.21.15.27:0/2416603323 conn(0x7f8b840d0e70 msgr2=0x7f8b840cfa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:43.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.132+0000 7f8b8c200640 1 -- 172.21.15.27:0/2416603323 shutdown_connections 2024-01-15T18:15:43.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.132+0000 7f8b8c200640 1 -- 172.21.15.27:0/2416603323 wait complete. 2024-01-15T18:15:43.161 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-15T18:15:43.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.308+0000 7fb6d1e00640 1 Processor -- start 2024-01-15T18:15:43.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.308+0000 7fb6d1e00640 1 -- start start 2024-01-15T18:15:43.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.308+0000 7fb6d1e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6cc0cde10 0x7fb6cc0cbf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:43.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.308+0000 7fb6d1e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6cc0cb4d0 con 0x7fb6cc0cfc30 2024-01-15T18:15:43.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.308+0000 7fb6d1e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6cc0d0120 con 0x7fb6cc15d680 2024-01-15T18:15:43.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.308+0000 7fb6d1e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6cc0d0400 con 0x7fb6cc0cde10 2024-01-15T18:15:43.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.308+0000 7fb6cb400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb6cc0cfc30 0x7fb6cc0cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55728/0 (socket says 172.21.15.27:55728) 2024-01-15T18:15:43.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.308+0000 7fb6cb400640 1 -- 172.21.15.27:0/2865934814 learned_addr learned my addr 172.21.15.27:0/2865934814 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:43.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.308+0000 7fb6ca000640 1 --2- 172.21.15.27:0/2865934814 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6cc0cde10 0x7fb6cc0cbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:43.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.308+0000 7fb6c9600640 1 -- 172.21.15.27:0/2865934814 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3240781571 0 0) 0x7fb6cc0cb4d0 con 0x7fb6cc0cfc30 2024-01-15T18:15:43.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.308+0000 7fb6c9600640 1 -- 172.21.15.27:0/2865934814 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6ac003530 con 0x7fb6cc0cfc30 2024-01-15T18:15:43.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.308+0000 7fb6c9600640 1 -- 172.21.15.27:0/2865934814 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063828348 0 0) 0x7fb6cc0d0120 con 0x7fb6cc15d680 2024-01-15T18:15:43.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.308+0000 7fb6c9600640 1 -- 172.21.15.27:0/2865934814 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6cc0cb4d0 con 0x7fb6cc15d680 2024-01-15T18:15:43.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.308+0000 7fb6ca000640 1 -- 172.21.15.27:0/2865934814 >> v1:172.21.15.93:6789/0 conn(0x7fb6cc15d680 legacy=0x7fb6cc0cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:43.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.308+0000 7fb6ca000640 1 -- 172.21.15.27:0/2865934814 >> v1:172.21.15.27:6789/0 conn(0x7fb6cc0cfc30 legacy=0x7fb6cc0cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:43.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.308+0000 7fb6ca000640 1 -- 172.21.15.27:0/2865934814 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6cc0cb770 con 0x7fb6cc0cde10 2024-01-15T18:15:43.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.312+0000 7fb6ca000640 1 --2- 172.21.15.27:0/2865934814 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6cc0cde10 0x7fb6cc0cbf70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40097f0 tx=0x7fb6cc0cd3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a0ad90ef437c47a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:43.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.312+0000 7fb6c9600640 1 -- 172.21.15.27:0/2865934814 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc0ce850 con 0x7fb6cc0cde10 2024-01-15T18:15:43.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.312+0000 7fb6c9600640 1 -- 172.21.15.27:0/2865934814 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b40a6040 con 0x7fb6cc0cde10 2024-01-15T18:15:43.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.312+0000 7fb6c9600640 1 -- 172.21.15.27:0/2865934814 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b40aa830 con 0x7fb6cc0cde10 2024-01-15T18:15:43.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.312+0000 7fb6d1e00640 1 -- 172.21.15.27:0/2865934814 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6cc0cde10 msgr2=0x7fb6cc0cbf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:43.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.312+0000 7fb6d1e00640 1 --2- 172.21.15.27:0/2865934814 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6cc0cde10 0x7fb6cc0cbf70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40097f0 tx=0x7fb6cc0cd3d0 comp rx=0 tx=0).stop 2024-01-15T18:15:43.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.312+0000 7fb6d1e00640 1 -- 172.21.15.27:0/2865934814 shutdown_connections 2024-01-15T18:15:43.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.312+0000 7fb6d1e00640 1 --2- 172.21.15.27:0/2865934814 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6cc0cde10 0x7fb6cc0cbf70 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:43.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.312+0000 7fb6d1e00640 1 -- 172.21.15.27:0/2865934814 >> 172.21.15.27:0/2865934814 conn(0x7fb6cc0d0e80 msgr2=0x7fb6cc0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:43.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.312+0000 7fb6d1e00640 1 -- 172.21.15.27:0/2865934814 shutdown_connections 2024-01-15T18:15:43.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.312+0000 7fb6d1e00640 1 -- 172.21.15.27:0/2865934814 wait complete. 2024-01-15T18:15:43.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.316+0000 7fb6d1e00640 1 Processor -- start 2024-01-15T18:15:43.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.316+0000 7fb6d1e00640 1 -- start start 2024-01-15T18:15:43.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.316+0000 7fb6d1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6cc0cde10 0x7fb6cc0c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:43.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.316+0000 7fb6d1e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6cc0cfc30 0x7fb6cc15fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:43.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.316+0000 7fb6d1e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb6cc15d680 0x7fb6cc155ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:43.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.316+0000 7fb6d1e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6cc0c8650 con 0x7fb6cc0cde10 2024-01-15T18:15:43.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.316+0000 7fb6d1e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6cc0c8930 con 0x7fb6cc15d680 2024-01-15T18:15:43.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.316+0000 7fb6d1e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6cc0c8c10 con 0x7fb6cc0cfc30 2024-01-15T18:15:43.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.316+0000 7fb6caa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6cc0cde10 0x7fb6cc0c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:43.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.316+0000 7fb6caa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6cc0cde10 0x7fb6cc0c7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45258/0 (socket says 172.21.15.27:45258) 2024-01-15T18:15:43.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.316+0000 7fb6caa00640 1 -- 172.21.15.27:0/2123433323 learned_addr learned my addr 172.21.15.27:0/2123433323 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:43.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.316+0000 7fb6ca000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6cc0cfc30 0x7fb6cc15fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:43.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.316+0000 7fb6cb400640 1 --2- 172.21.15.27:0/2123433323 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb6cc15d680 0x7fb6cc155ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:43.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.316+0000 7fb6ca000640 1 -- 172.21.15.27:0/2123433323 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb6cc15d680 msgr2=0x7fb6cc155ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:43.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.316+0000 7fb6ca000640 1 --2- 172.21.15.27:0/2123433323 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb6cc15d680 0x7fb6cc155ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:43.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.316+0000 7fb6ca000640 1 -- 172.21.15.27:0/2123433323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6cc0cde10 msgr2=0x7fb6cc0c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:43.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.316+0000 7fb6ca000640 1 --2- 172.21.15.27:0/2123433323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6cc0cde10 0x7fb6cc0c7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:43.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.316+0000 7fb6ca000640 1 -- 172.21.15.27:0/2123433323 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b4009420 con 0x7fb6cc0cfc30 2024-01-15T18:15:43.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.316+0000 7fb6caa00640 1 --2- 172.21.15.27:0/2123433323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6cc0cde10 0x7fb6cc0c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:15:43.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.316+0000 7fb6ca000640 1 --2- 172.21.15.27:0/2123433323 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6cc0cfc30 0x7fb6cc15fc60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40ab000 tx=0x7fb6b401acd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:43.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.320+0000 7fb6c3400640 1 -- 172.21.15.27:0/2123433323 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b40be610 con 0x7fb6cc0cfc30 2024-01-15T18:15:43.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.320+0000 7fb6c3400640 1 -- 172.21.15.27:0/2123433323 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b40be790 con 0x7fb6cc0cfc30 2024-01-15T18:15:43.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.320+0000 7fb6c3400640 1 -- 172.21.15.27:0/2123433323 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b40beb40 con 0x7fb6cc0cfc30 2024-01-15T18:15:43.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.320+0000 7fb6d1e00640 1 -- 172.21.15.27:0/2123433323 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6cc0c8ef0 con 0x7fb6cc0cfc30 2024-01-15T18:15:43.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.320+0000 7fb6d1e00640 1 -- 172.21.15.27:0/2123433323 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6cc156000 con 0x7fb6cc0cfc30 2024-01-15T18:15:43.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.320+0000 7fb6d1e00640 1 -- 172.21.15.27:0/2123433323 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6cc0c8a70 con 0x7fb6cc0cfc30 2024-01-15T18:15:43.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.320+0000 7fb6c3400640 1 -- 172.21.15.27:0/2123433323 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7fb6b40beca0 con 0x7fb6cc0cfc30 2024-01-15T18:15:43.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.320+0000 7fb6c3400640 1 --2- 172.21.15.27:0/2123433323 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb6a40674c0 0x7fb6a4069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:43.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.320+0000 7fb6c3400640 1 -- 172.21.15.27:0/2123433323 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5311+0+0 (secure 0 0 0) 0x7fb6b411d6c0 con 0x7fb6cc0cfc30 2024-01-15T18:15:43.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.320+0000 7fb6caa00640 1 --2- 172.21.15.27:0/2123433323 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb6a40674c0 0x7fb6a4069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:43.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.324+0000 7fb6caa00640 1 --2- 172.21.15.27:0/2123433323 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb6a40674c0 0x7fb6a4069980 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc0011f0 tx=0x7fb6bc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:43.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.324+0000 7fb6c3400640 1 -- 172.21.15.27:0/2123433323 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb6b40ec1e0 con 0x7fb6cc0cfc30 2024-01-15T18:15:43.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:43.484+0000 7fb6d1e00640 1 -- 172.21.15.27:0/2123433323 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb6cc0c8a70 con 0x7fb6cc0cfc30 2024-01-15T18:15:44.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.140+0000 7fb6c3400640 1 -- 172.21.15.27:0/2123433323 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v82) v1 ==== 170+0+0 (secure 0 0 0) 0x7fb6cc0c8a70 con 0x7fb6cc0cfc30 2024-01-15T18:15:44.142 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-01-15T18:15:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.144+0000 7fb6d1e00640 1 -- 172.21.15.27:0/2123433323 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb6a40674c0 msgr2=0x7fb6a4069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.144+0000 7fb6d1e00640 1 --2- 172.21.15.27:0/2123433323 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb6a40674c0 0x7fb6a4069980 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc0011f0 tx=0x7fb6bc009040 comp rx=0 tx=0).stop 2024-01-15T18:15:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.144+0000 7fb6d1e00640 1 -- 172.21.15.27:0/2123433323 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6cc0cfc30 msgr2=0x7fb6cc15fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.144+0000 7fb6d1e00640 1 --2- 172.21.15.27:0/2123433323 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6cc0cfc30 0x7fb6cc15fc60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40ab000 tx=0x7fb6b401acd0 comp rx=0 tx=0).stop 2024-01-15T18:15:44.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.148+0000 7fb6d1e00640 1 -- 172.21.15.27:0/2123433323 shutdown_connections 2024-01-15T18:15:44.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.148+0000 7fb6d1e00640 1 --2- 172.21.15.27:0/2123433323 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb6a40674c0 0x7fb6a4069980 secure :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc0011f0 tx=0x7fb6bc009040 comp rx=0 tx=0).stop 2024-01-15T18:15:44.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.148+0000 7fb6d1e00640 1 --2- 172.21.15.27:0/2123433323 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb6cc15d680 0x7fb6cc155ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:44.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.148+0000 7fb6d1e00640 1 --2- 172.21.15.27:0/2123433323 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6cc0cfc30 0x7fb6cc15fc60 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:44.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.148+0000 7fb6d1e00640 1 --2- 172.21.15.27:0/2123433323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6cc0cde10 0x7fb6cc0c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:44.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.148+0000 7fb6d1e00640 1 -- 172.21.15.27:0/2123433323 >> 172.21.15.27:0/2123433323 conn(0x7fb6cc0d0e80 msgr2=0x7fb6cc15c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:44.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.148+0000 7fb6d1e00640 1 -- 172.21.15.27:0/2123433323 shutdown_connections 2024-01-15T18:15:44.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.148+0000 7fb6d1e00640 1 -- 172.21.15.27:0/2123433323 wait complete. 2024-01-15T18:15:44.184 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-15T18:15:44.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.328+0000 7f3eb4e00640 1 Processor -- start 2024-01-15T18:15:44.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.328+0000 7f3eb4e00640 1 -- start start 2024-01-15T18:15:44.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.332+0000 7f3eb4e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3eb0158900 0x7f3eb015f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:44.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.332+0000 7f3eb4e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f3eb00cf2e0 0x7f3eb00701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:44.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.332+0000 7f3eb4e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb00b1cb0 con 0x7f3eb015bf00 2024-01-15T18:15:44.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.332+0000 7f3eb4e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb00b1f00 con 0x7f3eb00cf2e0 2024-01-15T18:15:44.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.332+0000 7f3eb4e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb00b21e0 con 0x7f3eb0158900 2024-01-15T18:15:44.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.332+0000 7f3ead400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3eb015bf00 0x7f3eb00cebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46088/0 (socket says 172.21.15.27:46088) 2024-01-15T18:15:44.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.332+0000 7f3ead400640 1 -- 172.21.15.27:0/1836116310 learned_addr learned my addr 172.21.15.27:0/1836116310 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:44.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.332+0000 7f3eade00640 1 --2- 172.21.15.27:0/1836116310 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3eb0158900 0x7f3eb015f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:44.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.332+0000 7f3eae800640 1 --2- 172.21.15.27:0/1836116310 >> v2:172.21.15.93:3300/0 conn(0x7f3eb00cf2e0 0x7f3eb00701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:44.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.332+0000 7f3eaca00640 1 -- 172.21.15.27:0/1836116310 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801248651 0 0) 0x7f3eb00b1cb0 con 0x7f3eb015bf00 2024-01-15T18:15:44.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.332+0000 7f3eaca00640 1 -- 172.21.15.27:0/1836116310 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e90003530 con 0x7f3eb015bf00 2024-01-15T18:15:44.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.332+0000 7f3eade00640 1 -- 172.21.15.27:0/1836116310 >> v2:172.21.15.93:3300/0 conn(0x7f3eb00cf2e0 msgr2=0x7f3eb00701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:44.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.332+0000 7f3eade00640 1 --2- 172.21.15.27:0/1836116310 >> v2:172.21.15.93:3300/0 conn(0x7f3eb00cf2e0 0x7f3eb00701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:44.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.332+0000 7f3eade00640 1 -- 172.21.15.27:0/1836116310 >> v1:172.21.15.27:6789/0 conn(0x7f3eb015bf00 legacy=0x7f3eb00cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:44.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.332+0000 7f3eade00640 1 -- 172.21.15.27:0/1836116310 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb015ea70 con 0x7f3eb0158900 2024-01-15T18:15:44.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.332+0000 7f3eae800640 1 --2- 172.21.15.27:0/1836116310 >> v2:172.21.15.93:3300/0 conn(0x7f3eb00cf2e0 0x7f3eb00701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:15:44.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.336+0000 7f3eade00640 1 --2- 172.21.15.27:0/1836116310 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3eb0158900 0x7f3eb015f550 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3ea00097c0 tx=0x7f3ea001c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc5f2348a2915c84 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:44.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.336+0000 7f3eaca00640 1 -- 172.21.15.27:0/1836116310 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb0070f30 con 0x7f3eb0158900 2024-01-15T18:15:44.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.336+0000 7f3eaca00640 1 -- 172.21.15.27:0/1836116310 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea00a7050 con 0x7f3eb0158900 2024-01-15T18:15:44.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.336+0000 7f3eaca00640 1 -- 172.21.15.27:0/1836116310 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea00ab820 con 0x7f3eb0158900 2024-01-15T18:15:44.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.336+0000 7f3eb4e00640 1 -- 172.21.15.27:0/1836116310 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3eb0158900 msgr2=0x7f3eb015f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:44.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.336+0000 7f3eb4e00640 1 --2- 172.21.15.27:0/1836116310 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3eb0158900 0x7f3eb015f550 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3ea00097c0 tx=0x7f3ea001c480 comp rx=0 tx=0).stop 2024-01-15T18:15:44.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.336+0000 7f3eb4e00640 1 -- 172.21.15.27:0/1836116310 shutdown_connections 2024-01-15T18:15:44.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.336+0000 7f3eb4e00640 1 --2- 172.21.15.27:0/1836116310 >> v2:172.21.15.93:3300/0 conn(0x7f3eb00cf2e0 0x7f3eb00701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:44.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.336+0000 7f3eb4e00640 1 --2- 172.21.15.27:0/1836116310 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3eb0158900 0x7f3eb015f550 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:44.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.336+0000 7f3eb4e00640 1 -- 172.21.15.27:0/1836116310 >> 172.21.15.27:0/1836116310 conn(0x7f3eb00cb700 msgr2=0x7f3eb00cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:44.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.336+0000 7f3eb4e00640 1 -- 172.21.15.27:0/1836116310 shutdown_connections 2024-01-15T18:15:44.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.336+0000 7f3eb4e00640 1 -- 172.21.15.27:0/1836116310 wait complete. 2024-01-15T18:15:44.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.336+0000 7f3eb4e00640 1 Processor -- start 2024-01-15T18:15:44.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.340+0000 7f3eb4e00640 1 -- start start 2024-01-15T18:15:44.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.340+0000 7f3eb4e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3eb00cf2e0 0x7f3eb014a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:44.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.340+0000 7f3eb4e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eb0158900 0x7f3eb00cc810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:44.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.340+0000 7f3eb4e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3eb015bf00 0x7f3eb00adc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:44.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.340+0000 7f3eb4e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb014fb40 con 0x7f3eb0158900 2024-01-15T18:15:44.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.340+0000 7f3eb4e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb014ef30 con 0x7f3eb00cf2e0 2024-01-15T18:15:44.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.340+0000 7f3eb4e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb014f210 con 0x7f3eb015bf00 2024-01-15T18:15:44.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.340+0000 7f3ead400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eb0158900 0x7f3eb00cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:44.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.340+0000 7f3ead400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eb0158900 0x7f3eb00cc810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56522/0 (socket says 172.21.15.27:56522) 2024-01-15T18:15:44.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.340+0000 7f3ead400640 1 -- 172.21.15.27:0/3208114805 learned_addr learned my addr 172.21.15.27:0/3208114805 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:44.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.340+0000 7f3eade00640 1 --2- 172.21.15.27:0/3208114805 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3eb00cf2e0 0x7f3eb014a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:44.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.340+0000 7f3eae800640 1 --2- 172.21.15.27:0/3208114805 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3eb015bf00 0x7f3eb00adc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:44.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.340+0000 7f3ead400640 1 -- 172.21.15.27:0/3208114805 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3eb015bf00 msgr2=0x7f3eb00adc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:44.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.340+0000 7f3ead400640 1 --2- 172.21.15.27:0/3208114805 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3eb015bf00 0x7f3eb00adc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:44.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.340+0000 7f3ead400640 1 -- 172.21.15.27:0/3208114805 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3eb00cf2e0 msgr2=0x7f3eb014a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:44.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.340+0000 7f3ead400640 1 --2- 172.21.15.27:0/3208114805 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3eb00cf2e0 0x7f3eb014a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:44.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.340+0000 7f3ead400640 1 -- 172.21.15.27:0/3208114805 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea8009a70 con 0x7f3eb0158900 2024-01-15T18:15:44.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.340+0000 7f3eade00640 1 --2- 172.21.15.27:0/3208114805 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3eb00cf2e0 0x7f3eb014a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:15:44.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.340+0000 7f3ead400640 1 --2- 172.21.15.27:0/3208114805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eb0158900 0x7f3eb00cc810 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f3e9800c420 tx=0x7f3e9800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:44.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.340+0000 7f3ea6000640 1 -- 172.21.15.27:0/3208114805 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e98005cf0 con 0x7f3eb0158900 2024-01-15T18:15:44.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.340+0000 7f3eb4e00640 1 -- 172.21.15.27:0/3208114805 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea0009420 con 0x7f3eb0158900 2024-01-15T18:15:44.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.340+0000 7f3ea6000640 1 -- 172.21.15.27:0/3208114805 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e98005e50 con 0x7f3eb0158900 2024-01-15T18:15:44.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.340+0000 7f3ea6000640 1 -- 172.21.15.27:0/3208114805 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e980072d0 con 0x7f3eb0158900 2024-01-15T18:15:44.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.340+0000 7f3eb4e00640 1 -- 172.21.15.27:0/3208114805 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb014f7d0 con 0x7f3eb0158900 2024-01-15T18:15:44.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.340+0000 7f3eb4e00640 1 -- 172.21.15.27:0/3208114805 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb014f960 con 0x7f3eb0158900 2024-01-15T18:15:44.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.344+0000 7f3ea6000640 1 -- 172.21.15.27:0/3208114805 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7f3e98007430 con 0x7f3eb0158900 2024-01-15T18:15:44.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.344+0000 7f3ea6000640 1 --2- 172.21.15.27:0/3208114805 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e880674c0 0x7f3e88069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:44.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.344+0000 7f3ea6000640 1 -- 172.21.15.27:0/3208114805 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5693+0+0 (secure 0 0 0) 0x7f3e98082570 con 0x7f3eb0158900 2024-01-15T18:15:44.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.344+0000 7f3eade00640 1 --2- 172.21.15.27:0/3208114805 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e880674c0 0x7f3e88069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:44.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.344+0000 7f3eade00640 1 --2- 172.21.15.27:0/3208114805 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e880674c0 0x7f3e88069980 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f3ea00ad000 tx=0x7f3ea0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:44.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.348+0000 7f3ea6000640 1 -- 172.21.15.27:0/3208114805 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3e98050fd0 con 0x7f3eb0158900 2024-01-15T18:15:44.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:44.536+0000 7f3eb4e00640 1 -- 172.21.15.27:0/3208114805 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f3eb004e850 con 0x7f3eb0158900 2024-01-15T18:15:45.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.156+0000 7f3ea6000640 1 -- 172.21.15.27:0/3208114805 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v53) v1 ==== 174+0+0 (secure 0 0 0) 0x7f3e980170c0 con 0x7f3eb0158900 2024-01-15T18:15:45.153 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 12 and data pool 13 2024-01-15T18:15:45.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.156+0000 7f3eb4e00640 1 -- 172.21.15.27:0/3208114805 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e880674c0 msgr2=0x7f3e88069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.156+0000 7f3eb4e00640 1 --2- 172.21.15.27:0/3208114805 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e880674c0 0x7f3e88069980 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f3ea00ad000 tx=0x7f3ea0008040 comp rx=0 tx=0).stop 2024-01-15T18:15:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.156+0000 7f3eb4e00640 1 -- 172.21.15.27:0/3208114805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eb0158900 msgr2=0x7f3eb00cc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.156+0000 7f3eb4e00640 1 --2- 172.21.15.27:0/3208114805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eb0158900 0x7f3eb00cc810 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f3e9800c420 tx=0x7f3e9800c820 comp rx=0 tx=0).stop 2024-01-15T18:15:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.156+0000 7f3eb4e00640 1 -- 172.21.15.27:0/3208114805 shutdown_connections 2024-01-15T18:15:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.156+0000 7f3eb4e00640 1 --2- 172.21.15.27:0/3208114805 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e880674c0 0x7f3e88069980 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.156+0000 7f3eb4e00640 1 --2- 172.21.15.27:0/3208114805 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3eb015bf00 0x7f3eb00adc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.156+0000 7f3eb4e00640 1 --2- 172.21.15.27:0/3208114805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3eb0158900 0x7f3eb00cc810 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.156+0000 7f3eb4e00640 1 --2- 172.21.15.27:0/3208114805 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3eb00cf2e0 0x7f3eb014a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.156+0000 7f3eb4e00640 1 -- 172.21.15.27:0/3208114805 >> 172.21.15.27:0/3208114805 conn(0x7f3eb00cb700 msgr2=0x7f3eb015cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.156+0000 7f3eb4e00640 1 -- 172.21.15.27:0/3208114805 shutdown_connections 2024-01-15T18:15:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.156+0000 7f3eb4e00640 1 -- 172.21.15.27:0/3208114805 wait complete. 2024-01-15T18:15:45.202 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:15:45.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.344+0000 7f1070000640 1 Processor -- start 2024-01-15T18:15:45.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.344+0000 7f1070000640 1 -- start start 2024-01-15T18:15:45.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.348+0000 7f1070000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f106806d430 0x7f10681644e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:45.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.348+0000 7f1070000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1068159a50 con 0x7f1068159bf0 2024-01-15T18:15:45.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.348+0000 7f1070000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106806d980 con 0x7f1068071e40 2024-01-15T18:15:45.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.348+0000 7f1070000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10680cce00 con 0x7f106806d430 2024-01-15T18:15:45.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.348+0000 7f106d600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1068159bf0 0x7f1068159fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46098/0 (socket says 172.21.15.27:46098) 2024-01-15T18:15:45.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.348+0000 7f106d600640 1 -- 172.21.15.27:0/325802577 learned_addr learned my addr 172.21.15.27:0/325802577 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:45.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.348+0000 7f106e000640 1 --2- 172.21.15.27:0/325802577 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f106806d430 0x7f10681644e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:45.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.348+0000 7f1067e00640 1 -- 172.21.15.27:0/325802577 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2671206478 0 0) 0x7f1058001120 con 0x7f1068159bf0 2024-01-15T18:15:45.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.348+0000 7f1067e00640 1 -- 172.21.15.27:0/325802577 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1048003530 con 0x7f1068159bf0 2024-01-15T18:15:45.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.348+0000 7f1067e00640 1 -- 172.21.15.27:0/325802577 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249774458 0 0) 0x7f106806d980 con 0x7f1068071e40 2024-01-15T18:15:45.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.348+0000 7f1067e00640 1 -- 172.21.15.27:0/325802577 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1058001120 con 0x7f1068071e40 2024-01-15T18:15:45.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.348+0000 7f106e000640 1 -- 172.21.15.27:0/325802577 >> v1:172.21.15.93:6789/0 conn(0x7f1068071e40 legacy=0x7f10680c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:45.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.348+0000 7f106e000640 1 -- 172.21.15.27:0/325802577 >> v1:172.21.15.27:6789/0 conn(0x7f1068159bf0 legacy=0x7f1068159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:45.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.348+0000 7f106e000640 1 -- 172.21.15.27:0/325802577 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106806dc60 con 0x7f106806d430 2024-01-15T18:15:45.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.352+0000 7f106e000640 1 --2- 172.21.15.27:0/325802577 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f106806d430 0x7f10681644e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1060009dc0 tx=0x7f106001f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52a99379ab7d2902 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:45.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.352+0000 7f1067e00640 1 -- 172.21.15.27:0/325802577 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10600af600 con 0x7f106806d430 2024-01-15T18:15:45.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.352+0000 7f1067e00640 1 -- 172.21.15.27:0/325802577 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10600ab050 con 0x7f106806d430 2024-01-15T18:15:45.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.352+0000 7f1067e00640 1 -- 172.21.15.27:0/325802577 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10600af980 con 0x7f106806d430 2024-01-15T18:15:45.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.352+0000 7f1070000640 1 -- 172.21.15.27:0/325802577 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f106806d430 msgr2=0x7f10681644e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:45.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.352+0000 7f1070000640 1 --2- 172.21.15.27:0/325802577 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f106806d430 0x7f10681644e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1060009dc0 tx=0x7f106001f150 comp rx=0 tx=0).stop 2024-01-15T18:15:45.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.352+0000 7f1070000640 1 -- 172.21.15.27:0/325802577 shutdown_connections 2024-01-15T18:15:45.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.352+0000 7f1070000640 1 --2- 172.21.15.27:0/325802577 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f106806d430 0x7f10681644e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:45.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.352+0000 7f1070000640 1 -- 172.21.15.27:0/325802577 >> 172.21.15.27:0/325802577 conn(0x7f10680cec60 msgr2=0x7f10680cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:45.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.352+0000 7f1070000640 1 -- 172.21.15.27:0/325802577 shutdown_connections 2024-01-15T18:15:45.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.352+0000 7f1070000640 1 -- 172.21.15.27:0/325802577 wait complete. 2024-01-15T18:15:45.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.352+0000 7f1070000640 1 Processor -- start 2024-01-15T18:15:45.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.352+0000 7f1070000640 1 -- start start 2024-01-15T18:15:45.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.356+0000 7f1070000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f106806d430 0x7f106814f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:45.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.356+0000 7f1070000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1068159bf0 0x7f10680c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:45.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.356+0000 7f1070000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f10680c6bf0 0x7f1068156420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:45.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.356+0000 7f1070000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10680c70a0 con 0x7f106806d430 2024-01-15T18:15:45.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.356+0000 7f1070000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1068157d40 con 0x7f1068159bf0 2024-01-15T18:15:45.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.356+0000 7f1070000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1068157130 con 0x7f10680c6bf0 2024-01-15T18:15:45.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.356+0000 7f106d600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f106806d430 0x7f106814f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:45.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.356+0000 7f106d600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f106806d430 0x7f106814f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56530/0 (socket says 172.21.15.27:56530) 2024-01-15T18:15:45.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.356+0000 7f106d600640 1 -- 172.21.15.27:0/736990490 learned_addr learned my addr 172.21.15.27:0/736990490 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:45.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.356+0000 7f106e000640 1 --2- 172.21.15.27:0/736990490 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f10680c6bf0 0x7f1068156420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:45.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.356+0000 7f106cc00640 1 --2- 172.21.15.27:0/736990490 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1068159bf0 0x7f10680c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:45.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.356+0000 7f106d600640 1 -- 172.21.15.27:0/736990490 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f10680c6bf0 msgr2=0x7f1068156420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:45.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.356+0000 7f106d600640 1 --2- 172.21.15.27:0/736990490 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f10680c6bf0 0x7f1068156420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:45.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.356+0000 7f106d600640 1 -- 172.21.15.27:0/736990490 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1068159bf0 msgr2=0x7f10680c66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:45.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.356+0000 7f106d600640 1 --2- 172.21.15.27:0/736990490 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1068159bf0 0x7f10680c66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:45.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.356+0000 7f106d600640 1 -- 172.21.15.27:0/736990490 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1060009a70 con 0x7f106806d430 2024-01-15T18:15:45.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.356+0000 7f106cc00640 1 --2- 172.21.15.27:0/736990490 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1068159bf0 0x7f10680c66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:45.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.356+0000 7f106d600640 1 --2- 172.21.15.27:0/736990490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f106806d430 0x7f106814f060 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f105800c700 tx=0x7f105800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:45.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.356+0000 7f1065600640 1 -- 172.21.15.27:0/736990490 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1058005ac0 con 0x7f106806d430 2024-01-15T18:15:45.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.356+0000 7f1065600640 1 -- 172.21.15.27:0/736990490 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1058005c20 con 0x7f106806d430 2024-01-15T18:15:45.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.356+0000 7f1070000640 1 -- 172.21.15.27:0/736990490 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1068157410 con 0x7f106806d430 2024-01-15T18:15:45.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.356+0000 7f1065600640 1 -- 172.21.15.27:0/736990490 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1058007070 con 0x7f106806d430 2024-01-15T18:15:45.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.356+0000 7f1070000640 1 -- 172.21.15.27:0/736990490 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10681579d0 con 0x7f106806d430 2024-01-15T18:15:45.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.356+0000 7f1070000640 1 -- 172.21.15.27:0/736990490 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f106804e850 con 0x7f106806d430 2024-01-15T18:15:45.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.360+0000 7f1065600640 1 -- 172.21.15.27:0/736990490 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7f10580071d0 con 0x7f106806d430 2024-01-15T18:15:45.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.360+0000 7f1065600640 1 --2- 172.21.15.27:0/736990490 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f10400674c0 0x7f1040069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:45.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.360+0000 7f1065600640 1 -- 172.21.15.27:0/736990490 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f1058082670 con 0x7f106806d430 2024-01-15T18:15:45.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.360+0000 7f106cc00640 1 --2- 172.21.15.27:0/736990490 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f10400674c0 0x7f1040069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:45.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.360+0000 7f106cc00640 1 --2- 172.21.15.27:0/736990490 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f10400674c0 0x7f1040069980 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f10681e37f0 tx=0x7f1050009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:45.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.364+0000 7f1065600640 1 -- 172.21.15.27:0/736990490 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f10580510a0 con 0x7f106806d430 2024-01-15T18:15:45.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.520+0000 7f1070000640 1 -- 172.21.15.27:0/736990490 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1068159a50 con 0x7f106806d430 2024-01-15T18:15:45.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.524+0000 7f1065600640 1 -- 172.21.15.27:0/736990490 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f1068159a50 con 0x7f106806d430 2024-01-15T18:15:45.521 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:15:45.521 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:15:44.541180+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:15:42.953021+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-15T18:15:43.535741+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:15:45.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.524+0000 7f1070000640 1 -- 172.21.15.27:0/736990490 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f10400674c0 msgr2=0x7f1040069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:45.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.524+0000 7f1070000640 1 --2- 172.21.15.27:0/736990490 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f10400674c0 0x7f1040069980 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f10681e37f0 tx=0x7f1050009040 comp rx=0 tx=0).stop 2024-01-15T18:15:45.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.524+0000 7f1070000640 1 -- 172.21.15.27:0/736990490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f106806d430 msgr2=0x7f106814f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:45.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.524+0000 7f1070000640 1 --2- 172.21.15.27:0/736990490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f106806d430 0x7f106814f060 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f105800c700 tx=0x7f105800cbc0 comp rx=0 tx=0).stop 2024-01-15T18:15:45.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.524+0000 7f1070000640 1 -- 172.21.15.27:0/736990490 shutdown_connections 2024-01-15T18:15:45.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.524+0000 7f1070000640 1 --2- 172.21.15.27:0/736990490 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f10400674c0 0x7f1040069980 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:45.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.524+0000 7f1070000640 1 --2- 172.21.15.27:0/736990490 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f10680c6bf0 0x7f1068156420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:45.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.524+0000 7f1070000640 1 --2- 172.21.15.27:0/736990490 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1068159bf0 0x7f10680c66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:45.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.524+0000 7f1070000640 1 --2- 172.21.15.27:0/736990490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f106806d430 0x7f106814f060 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:45.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.524+0000 7f1070000640 1 -- 172.21.15.27:0/736990490 >> 172.21.15.27:0/736990490 conn(0x7f10680cec60 msgr2=0x7f1068070180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:45.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.524+0000 7f1070000640 1 -- 172.21.15.27:0/736990490 shutdown_connections 2024-01-15T18:15:45.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.524+0000 7f1070000640 1 -- 172.21.15.27:0/736990490 wait complete. 2024-01-15T18:15:45.545 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:15:45.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.688+0000 7f570d200640 1 Processor -- start 2024-01-15T18:15:45.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.688+0000 7f570d200640 1 -- start start 2024-01-15T18:15:45.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.688+0000 7f570d200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f570815e7d0 0x7f57080cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:45.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.688+0000 7f570d200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f57080b20d0 0x7f57080cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:45.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.688+0000 7f570d200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57080b38b0 con 0x7f57080cedc0 2024-01-15T18:15:45.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.688+0000 7f570d200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57080cd550 con 0x7f57080b20d0 2024-01-15T18:15:45.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.688+0000 7f570d200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57080cd270 con 0x7f570815e7d0 2024-01-15T18:15:45.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.688+0000 7f5706800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f57080b20d0 0x7f57080cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:45.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.688+0000 7f5705400640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7f57080cedc0 0x7f57080cba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:49472/0 (socket says 172.21.15.27:49472) 2024-01-15T18:15:45.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.688+0000 7f5705e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f570815e7d0 0x7f57080cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:45.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.688+0000 7f5705400640 1 -- 172.21.15.27:0/7988770 learned_addr learned my addr 172.21.15.27:0/7988770 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:45.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.692+0000 7f5706800640 1 -- 172.21.15.27:0/7988770 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f570815e7d0 msgr2=0x7f57080cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:45.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.692+0000 7f5706800640 1 --2- 172.21.15.27:0/7988770 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f570815e7d0 0x7f57080cf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:45.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.692+0000 7f5706800640 1 -- 172.21.15.27:0/7988770 >> v1:172.21.15.93:6789/0 conn(0x7f57080cedc0 legacy=0x7f57080cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:45.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.692+0000 7f5706800640 1 -- 172.21.15.27:0/7988770 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57080b35d0 con 0x7f57080b20d0 2024-01-15T18:15:45.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.692+0000 7f5704a00640 1 -- 172.21.15.27:0/7988770 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174437935 0 0) 0x7f57080b38b0 con 0x7f57080cedc0 2024-01-15T18:15:45.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.692+0000 7f5705e00640 1 --2- 172.21.15.27:0/7988770 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f570815e7d0 0x7f57080cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:15:45.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.692+0000 7f5706800640 1 --2- 172.21.15.27:0/7988770 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f57080b20d0 0x7f57080cf7e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5700009d90 tx=0x7f570001c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6eb119009f27c08 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:45.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.692+0000 7f5704a00640 1 -- 172.21.15.27:0/7988770 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57000a5900 con 0x7f57080b20d0 2024-01-15T18:15:45.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.692+0000 7f5704a00640 1 -- 172.21.15.27:0/7988770 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57000a5a60 con 0x7f57080b20d0 2024-01-15T18:15:45.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.692+0000 7f5704a00640 1 -- 172.21.15.27:0/7988770 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57000a5df0 con 0x7f57080b20d0 2024-01-15T18:15:45.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.696+0000 7f570d200640 1 -- 172.21.15.27:0/7988770 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f57080b20d0 msgr2=0x7f57080cf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:45.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.696+0000 7f570d200640 1 --2- 172.21.15.27:0/7988770 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f57080b20d0 0x7f57080cf7e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5700009d90 tx=0x7f570001c950 comp rx=0 tx=0).stop 2024-01-15T18:15:45.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.696+0000 7f570d200640 1 -- 172.21.15.27:0/7988770 shutdown_connections 2024-01-15T18:15:45.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.696+0000 7f570d200640 1 --2- 172.21.15.27:0/7988770 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f57080b20d0 0x7f57080cf7e0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:45.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.696+0000 7f570d200640 1 --2- 172.21.15.27:0/7988770 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f570815e7d0 0x7f57080cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:45.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.696+0000 7f570d200640 1 -- 172.21.15.27:0/7988770 >> 172.21.15.27:0/7988770 conn(0x7f57080b2a10 msgr2=0x7f57080b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:45.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.696+0000 7f570d200640 1 -- 172.21.15.27:0/7988770 shutdown_connections 2024-01-15T18:15:45.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.696+0000 7f570d200640 1 -- 172.21.15.27:0/7988770 wait complete. 2024-01-15T18:15:45.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.696+0000 7f570d200640 1 Processor -- start 2024-01-15T18:15:45.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.696+0000 7f570d200640 1 -- start start 2024-01-15T18:15:45.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.696+0000 7f570d200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57080b20d0 0x7f570814ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:45.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.696+0000 7f570d200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f57080cedc0 0x7f570806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:45.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.696+0000 7f570d200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f570815e7d0 0x7f57080c6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:45.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.696+0000 7f570d200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570814ec00 con 0x7f57080b20d0 2024-01-15T18:15:45.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.696+0000 7f570d200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570814f630 con 0x7f570815e7d0 2024-01-15T18:15:45.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.696+0000 7f570d200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f570806c010 con 0x7f57080cedc0 2024-01-15T18:15:45.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.696+0000 7f5705e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57080b20d0 0x7f570814ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:45.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.696+0000 7f5705e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57080b20d0 0x7f570814ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56536/0 (socket says 172.21.15.27:56536) 2024-01-15T18:15:45.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.696+0000 7f5705e00640 1 -- 172.21.15.27:0/1183945336 learned_addr learned my addr 172.21.15.27:0/1183945336 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:45.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.700+0000 7f5706800640 1 --2- 172.21.15.27:0/1183945336 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f570815e7d0 0x7f57080c6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:45.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.700+0000 7f5705400640 1 --2- 172.21.15.27:0/1183945336 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f57080cedc0 0x7f570806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:45.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.700+0000 7f5705e00640 1 -- 172.21.15.27:0/1183945336 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f57080cedc0 msgr2=0x7f570806d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:45.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.700+0000 7f5705e00640 1 --2- 172.21.15.27:0/1183945336 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f57080cedc0 0x7f570806d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:45.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.700+0000 7f5705e00640 1 -- 172.21.15.27:0/1183945336 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f570815e7d0 msgr2=0x7f57080c6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:45.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.700+0000 7f5705e00640 1 --2- 172.21.15.27:0/1183945336 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f570815e7d0 0x7f57080c6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:45.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.700+0000 7f5705e00640 1 -- 172.21.15.27:0/1183945336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f8009420 con 0x7f57080b20d0 2024-01-15T18:15:45.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.700+0000 7f5706800640 1 --2- 172.21.15.27:0/1183945336 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f570815e7d0 0x7f57080c6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:45.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.700+0000 7f5705400640 1 --2- 172.21.15.27:0/1183945336 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f57080cedc0 0x7f570806d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:45.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.700+0000 7f5705e00640 1 --2- 172.21.15.27:0/1183945336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57080b20d0 0x7f570814ed70 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f56f800ec10 tx=0x7f56f800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:45.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.700+0000 7f56fe000640 1 -- 172.21.15.27:0/1183945336 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f800cbd0 con 0x7f57080b20d0 2024-01-15T18:15:45.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.700+0000 7f56fe000640 1 -- 172.21.15.27:0/1183945336 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f800cd30 con 0x7f57080b20d0 2024-01-15T18:15:45.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.700+0000 7f570d200640 1 -- 172.21.15.27:0/1183945336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5700009a70 con 0x7f57080b20d0 2024-01-15T18:15:45.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.700+0000 7f56fe000640 1 -- 172.21.15.27:0/1183945336 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f8013690 con 0x7f57080b20d0 2024-01-15T18:15:45.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.700+0000 7f570d200640 1 -- 172.21.15.27:0/1183945336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57081636f0 con 0x7f57080b20d0 2024-01-15T18:15:45.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.700+0000 7f570d200640 1 -- 172.21.15.27:0/1183945336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5708163910 con 0x7f57080b20d0 2024-01-15T18:15:45.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.700+0000 7f56fe000640 1 -- 172.21.15.27:0/1183945336 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7f56f80137f0 con 0x7f57080b20d0 2024-01-15T18:15:45.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.700+0000 7f56fe000640 1 --2- 172.21.15.27:0/1183945336 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f56e00674c0 0x7f56e0069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:45.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.700+0000 7f56fe000640 1 -- 172.21.15.27:0/1183945336 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f56f8083380 con 0x7f57080b20d0 2024-01-15T18:15:45.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.704+0000 7f5705400640 1 --2- 172.21.15.27:0/1183945336 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f56e00674c0 0x7f56e0069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:45.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.704+0000 7f5705400640 1 --2- 172.21.15.27:0/1183945336 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f56e00674c0 0x7f56e0069980 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f56f0007fd0 tx=0x7f56f0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:45.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.704+0000 7f56fe000640 1 -- 172.21.15.27:0/1183945336 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f56f8051db0 con 0x7f57080b20d0 2024-01-15T18:15:45.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.864+0000 7f570d200640 1 -- 172.21.15.27:0/1183945336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f570804e850 con 0x7f57080b20d0 2024-01-15T18:15:45.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.864+0000 7f56fe000640 1 -- 172.21.15.27:0/1183945336 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f56f8021090 con 0x7f57080b20d0 2024-01-15T18:15:45.863 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:15:45.863 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":83,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:15:44.541180+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:15:42.953021+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-15T18:15:43.535741+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:15:45.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.868+0000 7f570d200640 1 -- 172.21.15.27:0/1183945336 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f56e00674c0 msgr2=0x7f56e0069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:45.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.868+0000 7f570d200640 1 --2- 172.21.15.27:0/1183945336 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f56e00674c0 0x7f56e0069980 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f56f0007fd0 tx=0x7f56f0009040 comp rx=0 tx=0).stop 2024-01-15T18:15:45.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.868+0000 7f570d200640 1 -- 172.21.15.27:0/1183945336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57080b20d0 msgr2=0x7f570814ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:45.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.868+0000 7f570d200640 1 --2- 172.21.15.27:0/1183945336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57080b20d0 0x7f570814ed70 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f56f800ec10 tx=0x7f56f800c520 comp rx=0 tx=0).stop 2024-01-15T18:15:45.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.868+0000 7f570d200640 1 -- 172.21.15.27:0/1183945336 shutdown_connections 2024-01-15T18:15:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.868+0000 7f570d200640 1 --2- 172.21.15.27:0/1183945336 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f56e00674c0 0x7f56e0069980 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.868+0000 7f570d200640 1 --2- 172.21.15.27:0/1183945336 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f570815e7d0 0x7f57080c6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.868+0000 7f570d200640 1 --2- 172.21.15.27:0/1183945336 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f57080cedc0 0x7f570806d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.868+0000 7f570d200640 1 --2- 172.21.15.27:0/1183945336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57080b20d0 0x7f570814ed70 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.868+0000 7f570d200640 1 -- 172.21.15.27:0/1183945336 >> 172.21.15.27:0/1183945336 conn(0x7f57080b2a10 msgr2=0x7f57080cf470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.868+0000 7f570d200640 1 -- 172.21.15.27:0/1183945336 shutdown_connections 2024-01-15T18:15:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:45.868+0000 7f570d200640 1 -- 172.21.15.27:0/1183945336 wait complete. 2024-01-15T18:15:45.885 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-15T18:15:46.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.032+0000 7f9dcc400640 1 Processor -- start 2024-01-15T18:15:46.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.032+0000 7f9dcc400640 1 -- start start 2024-01-15T18:15:46.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.032+0000 7f9dcc400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f9dc40b1db0 0x7f9dc40b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:46.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.032+0000 7f9dcc400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9dc40ce050 0x7f9dc40ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:46.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.032+0000 7f9dcc400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dc40712b0 con 0x7f9dc406c430 2024-01-15T18:15:46.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.032+0000 7f9dcc400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dc40cea20 con 0x7f9dc40b1db0 2024-01-15T18:15:46.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.032+0000 7f9dcc400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc40ccfb0 con 0x7f9dc40ce050 2024-01-15T18:15:46.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.032+0000 7f9dca400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9dc40ce050 0x7f9dc40ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:46.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.032+0000 7f9dc9000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f9dc40b1db0 0x7f9dc40b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:46.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.032+0000 7f9dca400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9dc40ce050 0x7f9dc40ce4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:43008/0 (socket says 172.21.15.27:43008) 2024-01-15T18:15:46.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.032+0000 7f9dca400640 1 -- 172.21.15.27:0/2445162775 learned_addr learned my addr 172.21.15.27:0/2445162775 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:46.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.032+0000 7f9dca400640 1 -- 172.21.15.27:0/2445162775 >> v2:172.21.15.93:3300/0 conn(0x7f9dc40b1db0 msgr2=0x7f9dc40b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:46.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.032+0000 7f9dca400640 1 --2- 172.21.15.27:0/2445162775 >> v2:172.21.15.93:3300/0 conn(0x7f9dc40b1db0 0x7f9dc40b2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:46.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.032+0000 7f9dca400640 1 -- 172.21.15.27:0/2445162775 >> v1:172.21.15.93:6789/0 conn(0x7f9dc406c430 legacy=0x7f9dc4070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:46.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.032+0000 7f9dca400640 1 -- 172.21.15.27:0/2445162775 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc40cd290 con 0x7f9dc40ce050 2024-01-15T18:15:46.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.036+0000 7f9dca400640 1 --2- 172.21.15.27:0/2445162775 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9dc40ce050 0x7f9dc40ce4e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc009e80 tx=0x7f9dbc01c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa1131429531175d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:46.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.036+0000 7f9dc3e00640 1 -- 172.21.15.27:0/2445162775 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc0a5660 con 0x7f9dc40ce050 2024-01-15T18:15:46.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.036+0000 7f9dc3e00640 1 -- 172.21.15.27:0/2445162775 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dbc0a57c0 con 0x7f9dc40ce050 2024-01-15T18:15:46.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.036+0000 7f9dc3e00640 1 -- 172.21.15.27:0/2445162775 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc0a5b50 con 0x7f9dc40ce050 2024-01-15T18:15:46.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.036+0000 7f9dcc400640 1 -- 172.21.15.27:0/2445162775 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9dc40ce050 msgr2=0x7f9dc40ce4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:46.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.036+0000 7f9dcc400640 1 --2- 172.21.15.27:0/2445162775 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9dc40ce050 0x7f9dc40ce4e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc009e80 tx=0x7f9dbc01c930 comp rx=0 tx=0).stop 2024-01-15T18:15:46.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.036+0000 7f9dcc400640 1 -- 172.21.15.27:0/2445162775 shutdown_connections 2024-01-15T18:15:46.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.036+0000 7f9dcc400640 1 --2- 172.21.15.27:0/2445162775 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9dc40ce050 0x7f9dc40ce4e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:46.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.036+0000 7f9dcc400640 1 --2- 172.21.15.27:0/2445162775 >> v2:172.21.15.93:3300/0 conn(0x7f9dc40b1db0 0x7f9dc40b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:46.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.036+0000 7f9dcc400640 1 -- 172.21.15.27:0/2445162775 >> 172.21.15.27:0/2445162775 conn(0x7f9dc40cb700 msgr2=0x7f9dc40cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:46.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.036+0000 7f9dcc400640 1 -- 172.21.15.27:0/2445162775 shutdown_connections 2024-01-15T18:15:46.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.036+0000 7f9dcc400640 1 -- 172.21.15.27:0/2445162775 wait complete. 2024-01-15T18:15:46.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dcc400640 1 Processor -- start 2024-01-15T18:15:46.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dcc400640 1 -- start start 2024-01-15T18:15:46.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dcc400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9dc406c430 0x7f9dc41662c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:46.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dcc400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9dc40b1db0 0x7f9dc4141780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:46.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dcc400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dc40ce050 0x7f9dc415c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:46.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dcc400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc415cab0 con 0x7f9dc40ce050 2024-01-15T18:15:46.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dcc400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc4166910 con 0x7f9dc40b1db0 2024-01-15T18:15:46.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dcc400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc4166bf0 con 0x7f9dc406c430 2024-01-15T18:15:46.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dca400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dc40ce050 0x7f9dc415c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:46.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dca400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dc40ce050 0x7f9dc415c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56546/0 (socket says 172.21.15.27:56546) 2024-01-15T18:15:46.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dca400640 1 -- 172.21.15.27:0/3961467450 learned_addr learned my addr 172.21.15.27:0/3961467450 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:46.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dca400640 1 -- 172.21.15.27:0/3961467450 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9dc406c430 msgr2=0x7f9dc41662c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:46.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dc9a00640 1 --2- 172.21.15.27:0/3961467450 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9dc406c430 0x7f9dc41662c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:46.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dc9000640 1 --2- 172.21.15.27:0/3961467450 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9dc40b1db0 0x7f9dc4141780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:46.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dca400640 1 --2- 172.21.15.27:0/3961467450 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9dc406c430 0x7f9dc41662c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:46.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dca400640 1 -- 172.21.15.27:0/3961467450 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9dc40b1db0 msgr2=0x7f9dc4141780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:46.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dca400640 1 --2- 172.21.15.27:0/3961467450 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9dc40b1db0 0x7f9dc4141780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:46.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dca400640 1 -- 172.21.15.27:0/3961467450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dbc009b30 con 0x7f9dc40ce050 2024-01-15T18:15:46.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dc9000640 1 --2- 172.21.15.27:0/3961467450 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9dc40b1db0 0x7f9dc4141780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:46.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dc9a00640 1 --2- 172.21.15.27:0/3961467450 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9dc406c430 0x7f9dc41662c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:46.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dca400640 1 --2- 172.21.15.27:0/3961467450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dc40ce050 0x7f9dc415c570 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc0af040 tx=0x7f9dbc0196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:46.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dc1600640 1 -- 172.21.15.27:0/3961467450 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc0b8670 con 0x7f9dc40ce050 2024-01-15T18:15:46.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dc1600640 1 -- 172.21.15.27:0/3961467450 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dbc0b8810 con 0x7f9dc40ce050 2024-01-15T18:15:46.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dcc400640 1 -- 172.21.15.27:0/3961467450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dc4166ed0 con 0x7f9dc40ce050 2024-01-15T18:15:46.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dc1600640 1 -- 172.21.15.27:0/3961467450 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc0b8bc0 con 0x7f9dc40ce050 2024-01-15T18:15:46.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.040+0000 7f9dcc400640 1 -- 172.21.15.27:0/3961467450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dc4167490 con 0x7f9dc40ce050 2024-01-15T18:15:46.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.044+0000 7f9dcc400640 1 -- 172.21.15.27:0/3961467450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dc4167680 con 0x7f9dc40ce050 2024-01-15T18:15:46.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.044+0000 7f9dc1600640 1 -- 172.21.15.27:0/3961467450 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7f9dbc0b8d20 con 0x7f9dc40ce050 2024-01-15T18:15:46.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.044+0000 7f9dc1600640 1 --2- 172.21.15.27:0/3961467450 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9d9c0674c0 0x7f9d9c069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:46.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.044+0000 7f9dc1600640 1 -- 172.21.15.27:0/3961467450 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f9dbc11e0f0 con 0x7f9dc40ce050 2024-01-15T18:15:46.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.044+0000 7f9dc9a00640 1 --2- 172.21.15.27:0/3961467450 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9d9c0674c0 0x7f9d9c069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:46.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.044+0000 7f9dc9a00640 1 --2- 172.21.15.27:0/3961467450 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9d9c0674c0 0x7f9d9c069980 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f9db4000f70 tx=0x7f9db4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:46.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.048+0000 7f9dc1600640 1 -- 172.21.15.27:0/3961467450 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9dbc0ece30 con 0x7f9dc40ce050 2024-01-15T18:15:46.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:46.244+0000 7f9dcc400640 1 -- 172.21.15.27:0/3961467450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f9dc404e850 con 0x7f9dc40ce050 2024-01-15T18:15:47.135 INFO:tasks.ceph.mds.b.smithi093.stderr:2024-01-15T18:15:47.133+0000 7f9211c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-15T18:15:47.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.192+0000 7f9dc1600640 1 -- 172.21.15.27:0/3961467450 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v56) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9dbc009510 con 0x7f9dc40ce050 2024-01-15T18:15:47.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.196+0000 7f9dcc400640 1 -- 172.21.15.27:0/3961467450 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9d9c0674c0 msgr2=0x7f9d9c069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:47.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.196+0000 7f9dcc400640 1 --2- 172.21.15.27:0/3961467450 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9d9c0674c0 0x7f9d9c069980 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f9db4000f70 tx=0x7f9db4009040 comp rx=0 tx=0).stop 2024-01-15T18:15:47.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.196+0000 7f9dcc400640 1 -- 172.21.15.27:0/3961467450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dc40ce050 msgr2=0x7f9dc415c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:47.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.196+0000 7f9dcc400640 1 --2- 172.21.15.27:0/3961467450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dc40ce050 0x7f9dc415c570 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc0af040 tx=0x7f9dbc0196d0 comp rx=0 tx=0).stop 2024-01-15T18:15:47.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.196+0000 7f9dcc400640 1 -- 172.21.15.27:0/3961467450 shutdown_connections 2024-01-15T18:15:47.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.196+0000 7f9dcc400640 1 --2- 172.21.15.27:0/3961467450 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9d9c0674c0 0x7f9d9c069980 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:47.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.196+0000 7f9dcc400640 1 --2- 172.21.15.27:0/3961467450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9dc40ce050 0x7f9dc415c570 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:47.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.196+0000 7f9dcc400640 1 --2- 172.21.15.27:0/3961467450 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9dc40b1db0 0x7f9dc4141780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:47.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.196+0000 7f9dcc400640 1 --2- 172.21.15.27:0/3961467450 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9dc406c430 0x7f9dc41662c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:47.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.196+0000 7f9dcc400640 1 -- 172.21.15.27:0/3961467450 >> 172.21.15.27:0/3961467450 conn(0x7f9dc40cb700 msgr2=0x7f9dc40b3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:47.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.196+0000 7f9dcc400640 1 -- 172.21.15.27:0/3961467450 shutdown_connections 2024-01-15T18:15:47.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.196+0000 7f9dcc400640 1 -- 172.21.15.27:0/3961467450 wait complete. 2024-01-15T18:15:47.218 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-15T18:15:47.219 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-15T18:15:47.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.368+0000 7fbba7600640 1 Processor -- start 2024-01-15T18:15:47.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.368+0000 7fbba7600640 1 -- start start 2024-01-15T18:15:47.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.372+0000 7fbba7600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbba015b350 0x7fbba00cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:47.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.372+0000 7fbba7600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fbba0099a10 0x7fbba0099e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:47.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.372+0000 7fbba7600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba00cfec0 con 0x7fbba009a5b0 2024-01-15T18:15:47.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.372+0000 7fbba7600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbba009a3c0 con 0x7fbba0099a10 2024-01-15T18:15:47.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.372+0000 7fbba7600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba00ccd50 con 0x7fbba015b350 2024-01-15T18:15:47.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.372+0000 7fbb9fe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbba009a5b0 0x7fbba00cc500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46112/0 (socket says 172.21.15.27:46112) 2024-01-15T18:15:47.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.372+0000 7fbb9fe00640 1 -- 172.21.15.27:0/118013573 learned_addr learned my addr 172.21.15.27:0/118013573 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:47.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.372+0000 7fbba4c00640 1 --2- 172.21.15.27:0/118013573 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbba015b350 0x7fbba00cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:47.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.372+0000 7fbba5600640 1 --2- 172.21.15.27:0/118013573 >> v2:172.21.15.93:3300/0 conn(0x7fbba0099a10 0x7fbba0099e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:47.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.372+0000 7fbb9f400640 1 -- 172.21.15.27:0/118013573 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537522442 0 0) 0x7fbba00cfec0 con 0x7fbba009a5b0 2024-01-15T18:15:47.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.372+0000 7fbb9f400640 1 -- 172.21.15.27:0/118013573 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb80003530 con 0x7fbba009a5b0 2024-01-15T18:15:47.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.372+0000 7fbba4c00640 1 -- 172.21.15.27:0/118013573 >> v2:172.21.15.93:3300/0 conn(0x7fbba0099a10 msgr2=0x7fbba0099e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:47.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.372+0000 7fbba4c00640 1 --2- 172.21.15.27:0/118013573 >> v2:172.21.15.93:3300/0 conn(0x7fbba0099a10 0x7fbba0099e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:47.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.372+0000 7fbba4c00640 1 -- 172.21.15.27:0/118013573 >> v1:172.21.15.27:6789/0 conn(0x7fbba009a5b0 legacy=0x7fbba00cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:47.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.372+0000 7fbba4c00640 1 -- 172.21.15.27:0/118013573 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba00cebc0 con 0x7fbba015b350 2024-01-15T18:15:47.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.372+0000 7fbba5600640 1 --2- 172.21.15.27:0/118013573 >> v2:172.21.15.93:3300/0 conn(0x7fbba0099a10 0x7fbba0099e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:15:47.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.376+0000 7fbba4c00640 1 --2- 172.21.15.27:0/118013573 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbba015b350 0x7fbba00cf980 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbb900097c0 tx=0x7fbb9001c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bbe7a0d1d85a585 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:47.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.376+0000 7fbb9f400640 1 -- 172.21.15.27:0/118013573 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb900ab600 con 0x7fbba015b350 2024-01-15T18:15:47.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.376+0000 7fbb9f400640 1 -- 172.21.15.27:0/118013573 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb900a7050 con 0x7fbba015b350 2024-01-15T18:15:47.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.376+0000 7fbb9f400640 1 -- 172.21.15.27:0/118013573 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb900ab980 con 0x7fbba015b350 2024-01-15T18:15:47.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.376+0000 7fbba7600640 1 -- 172.21.15.27:0/118013573 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbba015b350 msgr2=0x7fbba00cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:47.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.376+0000 7fbba7600640 1 --2- 172.21.15.27:0/118013573 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbba015b350 0x7fbba00cf980 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbb900097c0 tx=0x7fbb9001c3c0 comp rx=0 tx=0).stop 2024-01-15T18:15:47.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.376+0000 7fbba7600640 1 -- 172.21.15.27:0/118013573 shutdown_connections 2024-01-15T18:15:47.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.376+0000 7fbba7600640 1 --2- 172.21.15.27:0/118013573 >> v2:172.21.15.93:3300/0 conn(0x7fbba0099a10 0x7fbba0099e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:47.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.376+0000 7fbba7600640 1 --2- 172.21.15.27:0/118013573 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbba015b350 0x7fbba00cf980 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:47.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.376+0000 7fbba7600640 1 -- 172.21.15.27:0/118013573 >> 172.21.15.27:0/118013573 conn(0x7fbba00b2a80 msgr2=0x7fbba00b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:47.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.376+0000 7fbba7600640 1 -- 172.21.15.27:0/118013573 shutdown_connections 2024-01-15T18:15:47.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.376+0000 7fbba7600640 1 -- 172.21.15.27:0/118013573 wait complete. 2024-01-15T18:15:47.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.376+0000 7fbba7600640 1 Processor -- start 2024-01-15T18:15:47.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.376+0000 7fbba7600640 1 -- start start 2024-01-15T18:15:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbba7600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbba0099a10 0x7fbba00c8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbba7600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbba009a5b0 0x7fbba0151a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:47.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbba7600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbba015b350 0x7fbba00c9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:47.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbba7600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba00c8670 con 0x7fbba0099a10 2024-01-15T18:15:47.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbba7600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba00c9040 con 0x7fbba009a5b0 2024-01-15T18:15:47.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbba7600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba00ca6a0 con 0x7fbba015b350 2024-01-15T18:15:47.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbba4c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbba0099a10 0x7fbba00c8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:47.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbba4c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbba0099a10 0x7fbba00c8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56554/0 (socket says 172.21.15.27:56554) 2024-01-15T18:15:47.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbba4c00640 1 -- 172.21.15.27:0/2875140086 learned_addr learned my addr 172.21.15.27:0/2875140086 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:47.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbba5600640 1 --2- 172.21.15.27:0/2875140086 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbba015b350 0x7fbba00c9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:47.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbb9fe00640 1 --2- 172.21.15.27:0/2875140086 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbba009a5b0 0x7fbba0151a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:47.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbba4c00640 1 -- 172.21.15.27:0/2875140086 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbba015b350 msgr2=0x7fbba00c9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:47.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbba4c00640 1 --2- 172.21.15.27:0/2875140086 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbba015b350 0x7fbba00c9b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:47.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbba4c00640 1 -- 172.21.15.27:0/2875140086 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbba009a5b0 msgr2=0x7fbba0151a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:47.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbba4c00640 1 --2- 172.21.15.27:0/2875140086 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbba009a5b0 0x7fbba0151a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:47.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbba4c00640 1 -- 172.21.15.27:0/2875140086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb98009a70 con 0x7fbba0099a10 2024-01-15T18:15:47.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbba5600640 1 --2- 172.21.15.27:0/2875140086 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbba015b350 0x7fbba00c9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:47.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbb9fe00640 1 --2- 172.21.15.27:0/2875140086 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbba009a5b0 0x7fbba0151a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:47.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbba4c00640 1 --2- 172.21.15.27:0/2875140086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbba0099a10 0x7fbba00c8b00 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fbb90019ff0 tx=0x7fbb90019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:47.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbb9cc00640 1 -- 172.21.15.27:0/2875140086 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb900b5470 con 0x7fbba0099a10 2024-01-15T18:15:47.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbb9cc00640 1 -- 172.21.15.27:0/2875140086 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb900b55f0 con 0x7fbba0099a10 2024-01-15T18:15:47.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbba7600640 1 -- 172.21.15.27:0/2875140086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb90009420 con 0x7fbba0099a10 2024-01-15T18:15:47.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbb9cc00640 1 -- 172.21.15.27:0/2875140086 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb900b59a0 con 0x7fbba0099a10 2024-01-15T18:15:47.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbba7600640 1 -- 172.21.15.27:0/2875140086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbba00c8020 con 0x7fbba0099a10 2024-01-15T18:15:47.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbba7600640 1 -- 172.21.15.27:0/2875140086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbba00d0720 con 0x7fbba0099a10 2024-01-15T18:15:47.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.380+0000 7fbb9cc00640 1 -- 172.21.15.27:0/2875140086 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7fbb900b2040 con 0x7fbba0099a10 2024-01-15T18:15:47.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.384+0000 7fbb9cc00640 1 --2- 172.21.15.27:0/2875140086 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbb780674c0 0x7fbb78069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:47.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.384+0000 7fbb9cc00640 1 -- 172.21.15.27:0/2875140086 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fbb9011d490 con 0x7fbba0099a10 2024-01-15T18:15:47.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.384+0000 7fbb9fe00640 1 --2- 172.21.15.27:0/2875140086 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbb780674c0 0x7fbb78069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:47.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.384+0000 7fbb9fe00640 1 --2- 172.21.15.27:0/2875140086 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbb780674c0 0x7fbb78069980 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fbba006f530 tx=0x7fbb88009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:47.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.388+0000 7fbb9cc00640 1 -- 172.21.15.27:0/2875140086 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbb900ebfe0 con 0x7fbba0099a10 2024-01-15T18:15:47.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:47.580+0000 7fbba7600640 1 -- 172.21.15.27:0/2875140086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fbba004e850 con 0x7fbba0099a10 2024-01-15T18:15:48.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.196+0000 7fbb9cc00640 1 -- 172.21.15.27:0/2875140086 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v57) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbb900b0020 con 0x7fbba0099a10 2024-01-15T18:15:48.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.196+0000 7fbba7600640 1 -- 172.21.15.27:0/2875140086 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbb780674c0 msgr2=0x7fbb78069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:48.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.196+0000 7fbba7600640 1 --2- 172.21.15.27:0/2875140086 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbb780674c0 0x7fbb78069980 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fbba006f530 tx=0x7fbb88009040 comp rx=0 tx=0).stop 2024-01-15T18:15:48.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.196+0000 7fbba7600640 1 -- 172.21.15.27:0/2875140086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbba0099a10 msgr2=0x7fbba00c8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:48.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.196+0000 7fbba7600640 1 --2- 172.21.15.27:0/2875140086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbba0099a10 0x7fbba00c8b00 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fbb90019ff0 tx=0x7fbb90019160 comp rx=0 tx=0).stop 2024-01-15T18:15:48.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.196+0000 7fbba7600640 1 -- 172.21.15.27:0/2875140086 shutdown_connections 2024-01-15T18:15:48.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.196+0000 7fbba7600640 1 --2- 172.21.15.27:0/2875140086 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbb780674c0 0x7fbb78069980 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:48.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.196+0000 7fbba7600640 1 --2- 172.21.15.27:0/2875140086 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbba015b350 0x7fbba00c9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:48.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.196+0000 7fbba7600640 1 --2- 172.21.15.27:0/2875140086 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbba009a5b0 0x7fbba0151a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:48.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.196+0000 7fbba7600640 1 --2- 172.21.15.27:0/2875140086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbba0099a10 0x7fbba00c8b00 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:48.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.196+0000 7fbba7600640 1 -- 172.21.15.27:0/2875140086 >> 172.21.15.27:0/2875140086 conn(0x7fbba00b2a80 msgr2=0x7fbba0159ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:48.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.196+0000 7fbba7600640 1 -- 172.21.15.27:0/2875140086 shutdown_connections 2024-01-15T18:15:48.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.196+0000 7fbba7600640 1 -- 172.21.15.27:0/2875140086 wait complete. 2024-01-15T18:15:48.217 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-15T18:15:48.217 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:15:48.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.360+0000 7fa567800640 1 Processor -- start 2024-01-15T18:15:48.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.360+0000 7fa567800640 1 -- start start 2024-01-15T18:15:48.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.364+0000 7fa567800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa5600701a0 0x7fa560070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:48.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.364+0000 7fa567800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa5600c92b0 0x7fa56006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:48.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.364+0000 7fa567800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56006c4a0 con 0x7fa56006e240 2024-01-15T18:15:48.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.364+0000 7fa567800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa560070ac0 con 0x7fa5600701a0 2024-01-15T18:15:48.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.364+0000 7fa567800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56015c660 con 0x7fa5600c92b0 2024-01-15T18:15:48.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.364+0000 7fa564e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa5600701a0 0x7fa560070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:48.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.364+0000 7fa55fe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa5600c92b0 0x7fa56006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:48.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.364+0000 7fa55fe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa5600c92b0 0x7fa56006dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:43040/0 (socket says 172.21.15.27:43040) 2024-01-15T18:15:48.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.364+0000 7fa55fe00640 1 -- 172.21.15.27:0/2403346900 learned_addr learned my addr 172.21.15.27:0/2403346900 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:48.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.364+0000 7fa55fe00640 1 -- 172.21.15.27:0/2403346900 >> v2:172.21.15.93:3300/0 conn(0x7fa5600701a0 msgr2=0x7fa560070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:48.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.364+0000 7fa55fe00640 1 --2- 172.21.15.27:0/2403346900 >> v2:172.21.15.93:3300/0 conn(0x7fa5600701a0 0x7fa560070580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:48.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.364+0000 7fa55fe00640 1 -- 172.21.15.27:0/2403346900 >> v1:172.21.15.93:6789/0 conn(0x7fa56006e240 legacy=0x7fa56015d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:48.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.364+0000 7fa55fe00640 1 -- 172.21.15.27:0/2403346900 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56015c940 con 0x7fa5600c92b0 2024-01-15T18:15:48.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.368+0000 7fa55fe00640 1 --2- 172.21.15.27:0/2403346900 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa5600c92b0 0x7fa56006dd00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa548009870 tx=0x7fa54801c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6a70939860483ca server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:48.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.368+0000 7fa55f400640 1 -- 172.21.15.27:0/2403346900 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5480aa670 con 0x7fa5600c92b0 2024-01-15T18:15:48.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.368+0000 7fa55f400640 1 -- 172.21.15.27:0/2403346900 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5480aa7d0 con 0x7fa5600c92b0 2024-01-15T18:15:48.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.368+0000 7fa55f400640 1 -- 172.21.15.27:0/2403346900 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5480aab60 con 0x7fa5600c92b0 2024-01-15T18:15:48.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.368+0000 7fa567800640 1 -- 172.21.15.27:0/2403346900 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa5600c92b0 msgr2=0x7fa56006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:48.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.368+0000 7fa567800640 1 --2- 172.21.15.27:0/2403346900 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa5600c92b0 0x7fa56006dd00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa548009870 tx=0x7fa54801c510 comp rx=0 tx=0).stop 2024-01-15T18:15:48.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.368+0000 7fa567800640 1 -- 172.21.15.27:0/2403346900 shutdown_connections 2024-01-15T18:15:48.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.368+0000 7fa567800640 1 --2- 172.21.15.27:0/2403346900 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa5600c92b0 0x7fa56006dd00 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:48.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.368+0000 7fa567800640 1 --2- 172.21.15.27:0/2403346900 >> v2:172.21.15.93:3300/0 conn(0x7fa5600701a0 0x7fa560070580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:48.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.368+0000 7fa567800640 1 -- 172.21.15.27:0/2403346900 >> 172.21.15.27:0/2403346900 conn(0x7fa5600cec60 msgr2=0x7fa5600cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:48.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.368+0000 7fa567800640 1 -- 172.21.15.27:0/2403346900 shutdown_connections 2024-01-15T18:15:48.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.368+0000 7fa567800640 1 -- 172.21.15.27:0/2403346900 wait complete. 2024-01-15T18:15:48.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.372+0000 7fa567800640 1 Processor -- start 2024-01-15T18:15:48.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.372+0000 7fa567800640 1 -- start start 2024-01-15T18:15:48.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.372+0000 7fa567800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa56006e240 0x7fa5600ae550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:48.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.372+0000 7fa567800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5600701a0 0x7fa560150a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:48.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.372+0000 7fa567800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5600c92b0 0x7fa5601f9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:48.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.372+0000 7fa567800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa560158c70 con 0x7fa5600c92b0 2024-01-15T18:15:48.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.372+0000 7fa567800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5600cc830 con 0x7fa5600701a0 2024-01-15T18:15:48.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.372+0000 7fa567800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5601f6620 con 0x7fa56006e240 2024-01-15T18:15:48.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.372+0000 7fa565800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5600c92b0 0x7fa5601f9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:48.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.372+0000 7fa565800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5600c92b0 0x7fa5601f9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56562/0 (socket says 172.21.15.27:56562) 2024-01-15T18:15:48.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.372+0000 7fa565800640 1 -- 172.21.15.27:0/450378136 learned_addr learned my addr 172.21.15.27:0/450378136 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:48.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.372+0000 7fa55fe00640 1 --2- 172.21.15.27:0/450378136 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5600701a0 0x7fa560150a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:48.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.372+0000 7fa564e00640 1 --2- 172.21.15.27:0/450378136 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa56006e240 0x7fa5600ae550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:48.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.372+0000 7fa565800640 1 -- 172.21.15.27:0/450378136 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa56006e240 msgr2=0x7fa5600ae550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:48.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.372+0000 7fa565800640 1 --2- 172.21.15.27:0/450378136 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa56006e240 0x7fa5600ae550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:48.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.372+0000 7fa565800640 1 -- 172.21.15.27:0/450378136 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5600701a0 msgr2=0x7fa560150a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:48.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.372+0000 7fa565800640 1 --2- 172.21.15.27:0/450378136 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5600701a0 0x7fa560150a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:48.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.372+0000 7fa565800640 1 -- 172.21.15.27:0/450378136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5480094d0 con 0x7fa5600c92b0 2024-01-15T18:15:48.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.372+0000 7fa565800640 1 --2- 172.21.15.27:0/450378136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5600c92b0 0x7fa5601f9d10 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fa55800c7a0 tx=0x7fa55800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:48.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.372+0000 7fa55cc00640 1 -- 172.21.15.27:0/450378136 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa558003b80 con 0x7fa5600c92b0 2024-01-15T18:15:48.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.372+0000 7fa55cc00640 1 -- 172.21.15.27:0/450378136 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa558003ce0 con 0x7fa5600c92b0 2024-01-15T18:15:48.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.372+0000 7fa55cc00640 1 -- 172.21.15.27:0/450378136 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55800aa30 con 0x7fa5600c92b0 2024-01-15T18:15:48.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.372+0000 7fa567800640 1 -- 172.21.15.27:0/450378136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5600aea90 con 0x7fa5600c92b0 2024-01-15T18:15:48.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.372+0000 7fa567800640 1 -- 172.21.15.27:0/450378136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa560158990 con 0x7fa5600c92b0 2024-01-15T18:15:48.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.372+0000 7fa567800640 1 -- 172.21.15.27:0/450378136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa56004e850 con 0x7fa5600c92b0 2024-01-15T18:15:48.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.376+0000 7fa55cc00640 1 -- 172.21.15.27:0/450378136 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7fa55800ab90 con 0x7fa5600c92b0 2024-01-15T18:15:48.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.376+0000 7fa55cc00640 1 --2- 172.21.15.27:0/450378136 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa5380674c0 0x7fa538069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:48.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.376+0000 7fa55cc00640 1 -- 172.21.15.27:0/450378136 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fa558082d60 con 0x7fa5600c92b0 2024-01-15T18:15:48.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.376+0000 7fa564e00640 1 --2- 172.21.15.27:0/450378136 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa5380674c0 0x7fa538069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:48.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.376+0000 7fa564e00640 1 --2- 172.21.15.27:0/450378136 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa5380674c0 0x7fa538069980 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fa550009b60 tx=0x7fa550009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:48.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.380+0000 7fa55cc00640 1 -- 172.21.15.27:0/450378136 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa558051840 con 0x7fa5600c92b0 2024-01-15T18:15:48.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.572+0000 7fa567800640 1 -- 172.21.15.27:0/450378136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5600af800 con 0x7fa5600c92b0 2024-01-15T18:15:48.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.572+0000 7fa55cc00640 1 -- 172.21.15.27:0/450378136 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4912 (secure 0 0 0) 0x7fa558054e60 con 0x7fa5600c92b0 2024-01-15T18:15:48.571 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:15:48.571 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":57,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8376,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/2584136338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2584136338},{"type":"v1","addr":"172.21.15.27:6833","nonce":2584136338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":8385,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2635542478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2635542478},{"type":"v1","addr":"172.21.15.27:6835","nonce":2635542478}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52},{"gid":8388,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/2822170069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":2822170069},{"type":"v1","addr":"172.21.15.93:6836","nonce":2822170069}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:15:44.541523+0000","modified":"2024-01-15T18:15:47.581737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8391},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8391":{"gid":8391,"name":"b","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.93:6837/980414657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":980414657},{"type":"v1","addr":"172.21.15.93:6837","nonce":980414657}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-01-15T18:15:48.571 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 57 2024-01-15T18:15:48.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.576+0000 7fa567800640 1 -- 172.21.15.27:0/450378136 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa5380674c0 msgr2=0x7fa538069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:48.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.576+0000 7fa567800640 1 --2- 172.21.15.27:0/450378136 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa5380674c0 0x7fa538069980 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fa550009b60 tx=0x7fa550009190 comp rx=0 tx=0).stop 2024-01-15T18:15:48.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.576+0000 7fa567800640 1 -- 172.21.15.27:0/450378136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5600c92b0 msgr2=0x7fa5601f9d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:48.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.576+0000 7fa567800640 1 --2- 172.21.15.27:0/450378136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5600c92b0 0x7fa5601f9d10 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fa55800c7a0 tx=0x7fa55800cc60 comp rx=0 tx=0).stop 2024-01-15T18:15:48.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.576+0000 7fa567800640 1 -- 172.21.15.27:0/450378136 shutdown_connections 2024-01-15T18:15:48.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.576+0000 7fa567800640 1 --2- 172.21.15.27:0/450378136 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa5380674c0 0x7fa538069980 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:48.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.576+0000 7fa567800640 1 --2- 172.21.15.27:0/450378136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5600c92b0 0x7fa5601f9d10 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:48.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.576+0000 7fa567800640 1 --2- 172.21.15.27:0/450378136 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa5600701a0 0x7fa560150a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:48.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.576+0000 7fa567800640 1 --2- 172.21.15.27:0/450378136 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa56006e240 0x7fa5600ae550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:48.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.576+0000 7fa567800640 1 -- 172.21.15.27:0/450378136 >> 172.21.15.27:0/450378136 conn(0x7fa5600cec60 msgr2=0x7fa560153c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:48.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.576+0000 7fa567800640 1 -- 172.21.15.27:0/450378136 shutdown_connections 2024-01-15T18:15:48.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.576+0000 7fa567800640 1 -- 172.21.15.27:0/450378136 wait complete. 2024-01-15T18:15:48.593 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:15:48.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.736+0000 7ff2fa400640 1 Processor -- start 2024-01-15T18:15:48.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.736+0000 7ff2fa400640 1 -- start start 2024-01-15T18:15:48.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.736+0000 7ff2fa400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2f415d240 0x7ff2f406b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:48.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.736+0000 7ff2fa400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff2f406c680 0x7ff2f4166dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:48.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.736+0000 7ff2fa400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f40cfe30 con 0x7ff2f40ce570 2024-01-15T18:15:48.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.736+0000 7ff2fa400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2f415e630 con 0x7ff2f406c680 2024-01-15T18:15:48.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.736+0000 7ff2fa400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f40cf500 con 0x7ff2f415d240 2024-01-15T18:15:48.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.736+0000 7ff2f2a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff2f40ce570 0x7ff2f406bf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46126/0 (socket says 172.21.15.27:46126) 2024-01-15T18:15:48.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.736+0000 7ff2f2a00640 1 -- 172.21.15.27:0/786979590 learned_addr learned my addr 172.21.15.27:0/786979590 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:48.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f3e00640 1 --2- 172.21.15.27:0/786979590 >> v2:172.21.15.93:3300/0 conn(0x7ff2f406c680 0x7ff2f4166dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:48.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f3400640 1 --2- 172.21.15.27:0/786979590 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2f415d240 0x7ff2f406b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:48.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f2000640 1 -- 172.21.15.27:0/786979590 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4241238647 0 0) 0x7ff2f40cfe30 con 0x7ff2f40ce570 2024-01-15T18:15:48.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f2000640 1 -- 172.21.15.27:0/786979590 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2d4003530 con 0x7ff2f40ce570 2024-01-15T18:15:48.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f2000640 1 -- 172.21.15.27:0/786979590 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3776480279 0 0) 0x7ff2d4003530 con 0x7ff2f40ce570 2024-01-15T18:15:48.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f2000640 1 -- 172.21.15.27:0/786979590 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2f415d240 msgr2=0x7ff2f406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:48.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f2000640 1 --2- 172.21.15.27:0/786979590 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2f415d240 0x7ff2f406b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:48.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f2000640 1 -- 172.21.15.27:0/786979590 >> v2:172.21.15.93:3300/0 conn(0x7ff2f406c680 msgr2=0x7ff2f4166dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:48.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f2000640 1 --2- 172.21.15.27:0/786979590 >> v2:172.21.15.93:3300/0 conn(0x7ff2f406c680 0x7ff2f4166dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:48.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f2000640 1 -- 172.21.15.27:0/786979590 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f40cf7e0 con 0x7ff2f40ce570 2024-01-15T18:15:48.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f2000640 1 -- 172.21.15.27:0/786979590 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7ff2dc001680 con 0x7ff2f40ce570 2024-01-15T18:15:48.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f2000640 1 -- 172.21.15.27:0/786979590 >> v1:172.21.15.27:6789/0 conn(0x7ff2f40ce570 legacy=0x7ff2f406bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:48.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f2000640 1 --2- 172.21.15.27:0/786979590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2d40039f0 0x7ff2d4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:48.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f2000640 1 --2- 172.21.15.27:0/786979590 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2f415d240 0x7ff2d40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:48.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f2000640 1 --2- 172.21.15.27:0/786979590 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2d4004d10 0x7ff2d4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f2000640 1 -- 172.21.15.27:0/786979590 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f415e630 con 0x7ff2d40039f0 2024-01-15T18:15:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f2000640 1 -- 172.21.15.27:0/786979590 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f40cf500 con 0x7ff2d4004d10 2024-01-15T18:15:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f2000640 1 -- 172.21.15.27:0/786979590 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d400b990 con 0x7ff2f415d240 2024-01-15T18:15:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f3400640 1 --2- 172.21.15.27:0/786979590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2d40039f0 0x7ff2d4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f2a00640 1 --2- 172.21.15.27:0/786979590 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2f415d240 0x7ff2d40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f3e00640 1 --2- 172.21.15.27:0/786979590 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2d4004d10 0x7ff2d4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f3400640 1 -- 172.21.15.27:0/786979590 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2f415d240 msgr2=0x7ff2d40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f3400640 1 --2- 172.21.15.27:0/786979590 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2f415d240 0x7ff2d40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f3400640 1 -- 172.21.15.27:0/786979590 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2d4004d10 msgr2=0x7ff2d4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f3400640 1 --2- 172.21.15.27:0/786979590 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2d4004d10 0x7ff2d4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f3400640 1 -- 172.21.15.27:0/786979590 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d4003530 con 0x7ff2d40039f0 2024-01-15T18:15:48.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.740+0000 7ff2f3e00640 1 --2- 172.21.15.27:0/786979590 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2d4004d10 0x7ff2d4005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:48.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.744+0000 7ff2f3400640 1 --2- 172.21.15.27:0/786979590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2d40039f0 0x7ff2d4003e10 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4009ae0 tx=0x7ff2e401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e554910a90c9bbf server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:48.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.744+0000 7ff2f2000640 1 -- 172.21.15.27:0/786979590 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e40a5550 con 0x7ff2d40039f0 2024-01-15T18:15:48.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.744+0000 7ff2f2000640 1 -- 172.21.15.27:0/786979590 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e40a56b0 con 0x7ff2d40039f0 2024-01-15T18:15:48.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.744+0000 7ff2f2000640 1 -- 172.21.15.27:0/786979590 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e40a5a60 con 0x7ff2d40039f0 2024-01-15T18:15:48.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.744+0000 7ff2fa400640 1 -- 172.21.15.27:0/786979590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2d40039f0 msgr2=0x7ff2d4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:48.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.744+0000 7ff2fa400640 1 --2- 172.21.15.27:0/786979590 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2d40039f0 0x7ff2d4003e10 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4009ae0 tx=0x7ff2e401c960 comp rx=0 tx=0).stop 2024-01-15T18:15:48.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.744+0000 7ff2f3e00640 1 -- 172.21.15.27:0/786979590 reap_dead start 2024-01-15T18:15:48.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.744+0000 7ff2fa400640 1 -- 172.21.15.27:0/786979590 shutdown_connections 2024-01-15T18:15:48.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.744+0000 7ff2fa400640 1 -- 172.21.15.27:0/786979590 >> 172.21.15.27:0/786979590 conn(0x7ff2f40d0f20 msgr2=0x7ff2f40d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:48.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.744+0000 7ff2fa400640 1 -- 172.21.15.27:0/786979590 shutdown_connections 2024-01-15T18:15:48.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.744+0000 7ff2fa400640 1 -- 172.21.15.27:0/786979590 wait complete. 2024-01-15T18:15:48.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.744+0000 7ff2fa400640 1 Processor -- start 2024-01-15T18:15:48.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.744+0000 7ff2fa400640 1 -- start start 2024-01-15T18:15:48.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.748+0000 7ff2fa400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2d40039f0 0x7ff2f4143ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:48.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.748+0000 7ff2fa400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2f409e5a0 0x7ff2f40c8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:48.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.748+0000 7ff2fa400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2f40c90b0 0x7ff2f40c9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:48.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.748+0000 7ff2fa400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f41444a0 con 0x7ff2f409e5a0 2024-01-15T18:15:48.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.748+0000 7ff2fa400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f4144780 con 0x7ff2f40c90b0 2024-01-15T18:15:48.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.748+0000 7ff2fa400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f4144a60 con 0x7ff2d40039f0 2024-01-15T18:15:48.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.748+0000 7ff2f2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2f409e5a0 0x7ff2f40c8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:48.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.748+0000 7ff2f2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2f409e5a0 0x7ff2f40c8b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56588/0 (socket says 172.21.15.27:56588) 2024-01-15T18:15:48.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.748+0000 7ff2f2a00640 1 -- 172.21.15.27:0/1140945945 learned_addr learned my addr 172.21.15.27:0/1140945945 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:48.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.748+0000 7ff2f3400640 1 --2- 172.21.15.27:0/1140945945 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2d40039f0 0x7ff2f4143ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:48.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.748+0000 7ff2f3e00640 1 --2- 172.21.15.27:0/1140945945 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2f40c90b0 0x7ff2f40c9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.748+0000 7ff2f2a00640 1 -- 172.21.15.27:0/1140945945 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2d40039f0 msgr2=0x7ff2f4143ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.748+0000 7ff2f2a00640 1 --2- 172.21.15.27:0/1140945945 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2d40039f0 0x7ff2f4143ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.748+0000 7ff2f2a00640 1 -- 172.21.15.27:0/1140945945 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2f40c90b0 msgr2=0x7ff2f40c9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.748+0000 7ff2f2a00640 1 --2- 172.21.15.27:0/1140945945 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2f40c90b0 0x7ff2f40c9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.748+0000 7ff2f2a00640 1 -- 172.21.15.27:0/1140945945 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f4144d40 con 0x7ff2f409e5a0 2024-01-15T18:15:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.748+0000 7ff2f2a00640 1 --2- 172.21.15.27:0/1140945945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2f409e5a0 0x7ff2f40c8b70 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc00e7b0 tx=0x7ff2dc00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.748+0000 7ff2eb400640 1 -- 172.21.15.27:0/1140945945 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc00c870 con 0x7ff2f409e5a0 2024-01-15T18:15:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.748+0000 7ff2eb400640 1 -- 172.21.15.27:0/1140945945 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2dc00c9d0 con 0x7ff2f409e5a0 2024-01-15T18:15:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.748+0000 7ff2eb400640 1 -- 172.21.15.27:0/1140945945 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc00cd80 con 0x7ff2f409e5a0 2024-01-15T18:15:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.748+0000 7ff2fa400640 1 -- 172.21.15.27:0/1140945945 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f4145020 con 0x7ff2f409e5a0 2024-01-15T18:15:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.748+0000 7ff2fa400640 1 -- 172.21.15.27:0/1140945945 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f409ea20 con 0x7ff2f409e5a0 2024-01-15T18:15:48.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.748+0000 7ff2fa400640 1 -- 172.21.15.27:0/1140945945 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f4145210 con 0x7ff2f409e5a0 2024-01-15T18:15:48.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.748+0000 7ff2eb400640 1 -- 172.21.15.27:0/1140945945 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7ff2dc01d460 con 0x7ff2f409e5a0 2024-01-15T18:15:48.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.752+0000 7ff2eb400640 1 --2- 172.21.15.27:0/1140945945 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff2cc0674c0 0x7ff2cc069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:48.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.752+0000 7ff2eb400640 1 -- 172.21.15.27:0/1140945945 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7ff2dc082ef0 con 0x7ff2f409e5a0 2024-01-15T18:15:48.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.752+0000 7ff2f3400640 1 --2- 172.21.15.27:0/1140945945 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff2cc0674c0 0x7ff2cc069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.752+0000 7ff2f3400640 1 --2- 172.21.15.27:0/1140945945 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff2cc0674c0 0x7ff2cc069980 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff2e40af000 tx=0x7ff2e40046d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:48.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.756+0000 7ff2eb400640 1 -- 172.21.15.27:0/1140945945 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff2dc0519d0 con 0x7ff2f409e5a0 2024-01-15T18:15:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.916+0000 7ff2fa400640 1 -- 172.21.15.27:0/1140945945 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2f404e850 con 0x7ff2f409e5a0 2024-01-15T18:15:48.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.916+0000 7ff2eb400640 1 -- 172.21.15.27:0/1140945945 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+16596 (secure 0 0 0) 0x7ff2dc054ff0 con 0x7ff2f409e5a0 2024-01-15T18:15:48.916 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:15:48.916 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":84,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:15:46.170619+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:15:42.953021+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-15T18:15:43.535741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:15:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.920+0000 7ff2fa400640 1 -- 172.21.15.27:0/1140945945 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff2cc0674c0 msgr2=0x7ff2cc069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.920+0000 7ff2fa400640 1 --2- 172.21.15.27:0/1140945945 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff2cc0674c0 0x7ff2cc069980 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff2e40af000 tx=0x7ff2e40046d0 comp rx=0 tx=0).stop 2024-01-15T18:15:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.920+0000 7ff2fa400640 1 -- 172.21.15.27:0/1140945945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2f409e5a0 msgr2=0x7ff2f40c8b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.920+0000 7ff2fa400640 1 --2- 172.21.15.27:0/1140945945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2f409e5a0 0x7ff2f40c8b70 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc00e7b0 tx=0x7ff2dc00ec70 comp rx=0 tx=0).stop 2024-01-15T18:15:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.920+0000 7ff2fa400640 1 -- 172.21.15.27:0/1140945945 shutdown_connections 2024-01-15T18:15:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.920+0000 7ff2fa400640 1 --2- 172.21.15.27:0/1140945945 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff2cc0674c0 0x7ff2cc069980 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.920+0000 7ff2fa400640 1 --2- 172.21.15.27:0/1140945945 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2f40c90b0 0x7ff2f40c9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.920+0000 7ff2fa400640 1 --2- 172.21.15.27:0/1140945945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2f409e5a0 0x7ff2f40c8b70 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.920+0000 7ff2fa400640 1 --2- 172.21.15.27:0/1140945945 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2d40039f0 0x7ff2f4143ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.920+0000 7ff2fa400640 1 -- 172.21.15.27:0/1140945945 >> 172.21.15.27:0/1140945945 conn(0x7ff2f40d0f20 msgr2=0x7ff2f40af310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.920+0000 7ff2fa400640 1 -- 172.21.15.27:0/1140945945 shutdown_connections 2024-01-15T18:15:48.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:48.920+0000 7ff2fa400640 1 -- 172.21.15.27:0/1140945945 wait complete. 2024-01-15T18:15:48.938 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-15T18:15:48.938 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:15:49.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.080+0000 7f141e600640 1 Processor -- start 2024-01-15T18:15:49.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.080+0000 7f141e600640 1 -- start start 2024-01-15T18:15:49.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.080+0000 7f141e600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f141815d0a0 0x7f141806c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:49.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.080+0000 7f141e600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f14180c94c0 0x7f14180c9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:49.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.080+0000 7f141e600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14180ad180 con 0x7f14180ce710 2024-01-15T18:15:49.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.080+0000 7f141e600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14180c9e90 con 0x7f14180c94c0 2024-01-15T18:15:49.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.080+0000 7f141e600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14180ac890 con 0x7f141815d0a0 2024-01-15T18:15:49.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.080+0000 7f1417e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f14180c94c0 0x7f14180c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:49.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.080+0000 7f1417e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f14180c94c0 0x7f14180c9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56600/0 (socket says 172.21.15.27:56600) 2024-01-15T18:15:49.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.080+0000 7f1417e00640 1 -- 172.21.15.27:0/580481996 learned_addr learned my addr 172.21.15.27:0/580481996 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:49.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.080+0000 7f1416a00640 1 --1- 172.21.15.27:0/580481996 >> v1:172.21.15.27:6789/0 conn(0x7f14180ce710 0x7f14180cd240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46142/0 (socket says 172.21.15.27:46142) 2024-01-15T18:15:49.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.080+0000 7f1417400640 1 --2- 172.21.15.27:0/580481996 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f141815d0a0 0x7f141806c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:49.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.084+0000 7f1417e00640 1 -- 172.21.15.27:0/580481996 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f141815d0a0 msgr2=0x7f141806c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:49.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.084+0000 7f1417e00640 1 --2- 172.21.15.27:0/580481996 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f141815d0a0 0x7f141806c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:49.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.084+0000 7f1417e00640 1 -- 172.21.15.27:0/580481996 >> v1:172.21.15.27:6789/0 conn(0x7f14180ce710 legacy=0x7f14180cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:49.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.084+0000 7f1417e00640 1 -- 172.21.15.27:0/580481996 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14180cda90 con 0x7f14180c94c0 2024-01-15T18:15:49.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.084+0000 7f1416000640 1 -- 172.21.15.27:0/580481996 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30143801 0 0) 0x7f14180ad180 con 0x7f14180ce710 2024-01-15T18:15:49.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.084+0000 7f1417e00640 1 --2- 172.21.15.27:0/580481996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14180c94c0 0x7f14180c9950 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f1410009e50 tx=0x7f141001c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30ac3492b0aedf12 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:49.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.084+0000 7f1416000640 1 -- 172.21.15.27:0/580481996 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14100a58a0 con 0x7f14180c94c0 2024-01-15T18:15:49.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.084+0000 7f1416000640 1 -- 172.21.15.27:0/580481996 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14100a5a00 con 0x7f14180c94c0 2024-01-15T18:15:49.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.084+0000 7f1416000640 1 -- 172.21.15.27:0/580481996 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14100a5d90 con 0x7f14180c94c0 2024-01-15T18:15:49.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f141e600640 1 -- 172.21.15.27:0/580481996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14180c94c0 msgr2=0x7f14180c9950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:49.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f141e600640 1 --2- 172.21.15.27:0/580481996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14180c94c0 0x7f14180c9950 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f1410009e50 tx=0x7f141001c9e0 comp rx=0 tx=0).stop 2024-01-15T18:15:49.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f141e600640 1 -- 172.21.15.27:0/580481996 shutdown_connections 2024-01-15T18:15:49.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f141e600640 1 --2- 172.21.15.27:0/580481996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14180c94c0 0x7f14180c9950 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:49.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f141e600640 1 --2- 172.21.15.27:0/580481996 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f141815d0a0 0x7f141806c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:49.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f141e600640 1 -- 172.21.15.27:0/580481996 >> 172.21.15.27:0/580481996 conn(0x7f14180cec60 msgr2=0x7f14180cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:49.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f141e600640 1 -- 172.21.15.27:0/580481996 shutdown_connections 2024-01-15T18:15:49.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f141e600640 1 -- 172.21.15.27:0/580481996 wait complete. 2024-01-15T18:15:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f141e600640 1 Processor -- start 2024-01-15T18:15:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f141e600640 1 -- start start 2024-01-15T18:15:49.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f141e600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f14180c94c0 0x7f1418159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:49.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f141e600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14180ce710 0x7f141815e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:49.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f141e600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f141815d0a0 0x7f141815bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:49.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f141e600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1418161f20 con 0x7f14180ce710 2024-01-15T18:15:49.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f141e600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f141815f3d0 con 0x7f141815d0a0 2024-01-15T18:15:49.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f141e600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f141815f6b0 con 0x7f14180c94c0 2024-01-15T18:15:49.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f1416a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14180ce710 0x7f141815e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:49.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f1416a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14180ce710 0x7f141815e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56606/0 (socket says 172.21.15.27:56606) 2024-01-15T18:15:49.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f1416a00640 1 -- 172.21.15.27:0/667625585 learned_addr learned my addr 172.21.15.27:0/667625585 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:49.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f1417400640 1 --2- 172.21.15.27:0/667625585 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f14180c94c0 0x7f1418159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:49.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f1417e00640 1 --2- 172.21.15.27:0/667625585 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f141815d0a0 0x7f141815bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:49.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f1416a00640 1 -- 172.21.15.27:0/667625585 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f14180c94c0 msgr2=0x7f1418159490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:49.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f1416a00640 1 --2- 172.21.15.27:0/667625585 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f14180c94c0 0x7f1418159490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:49.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f1416a00640 1 -- 172.21.15.27:0/667625585 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f141815d0a0 msgr2=0x7f141815bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:49.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f1416a00640 1 --2- 172.21.15.27:0/667625585 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f141815d0a0 0x7f141815bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:49.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.088+0000 7f1416a00640 1 -- 172.21.15.27:0/667625585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1410009b30 con 0x7f14180ce710 2024-01-15T18:15:49.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.092+0000 7f1416a00640 1 --2- 172.21.15.27:0/667625585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14180ce710 0x7f141815e910 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f140000c480 tx=0x7f140000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:49.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.092+0000 7f140f400640 1 -- 172.21.15.27:0/667625585 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14000074e0 con 0x7f14180ce710 2024-01-15T18:15:49.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.092+0000 7f140f400640 1 -- 172.21.15.27:0/667625585 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1400007660 con 0x7f14180ce710 2024-01-15T18:15:49.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.092+0000 7f141e600640 1 -- 172.21.15.27:0/667625585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f141815d690 con 0x7f14180ce710 2024-01-15T18:15:49.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.092+0000 7f140f400640 1 -- 172.21.15.27:0/667625585 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1400004a30 con 0x7f14180ce710 2024-01-15T18:15:49.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.092+0000 7f141e600640 1 -- 172.21.15.27:0/667625585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f141815fc70 con 0x7f14180ce710 2024-01-15T18:15:49.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.092+0000 7f141e600640 1 -- 172.21.15.27:0/667625585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f141815fe90 con 0x7f14180ce710 2024-01-15T18:15:49.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.092+0000 7f140f400640 1 -- 172.21.15.27:0/667625585 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7f1400004b90 con 0x7f14180ce710 2024-01-15T18:15:49.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.092+0000 7f140f400640 1 --2- 172.21.15.27:0/667625585 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f13f00674c0 0x7f13f0069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:49.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.092+0000 7f140f400640 1 -- 172.21.15.27:0/667625585 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f1400082490 con 0x7f14180ce710 2024-01-15T18:15:49.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.092+0000 7f1417400640 1 --2- 172.21.15.27:0/667625585 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f13f00674c0 0x7f13f0069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:49.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.096+0000 7f1417400640 1 --2- 172.21.15.27:0/667625585 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f13f00674c0 0x7f13f0069980 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f14181676a0 tx=0x7f1408009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:49.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.096+0000 7f140f400640 1 -- 172.21.15.27:0/667625585 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1400050fd0 con 0x7f14180ce710 2024-01-15T18:15:49.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.252+0000 7f141e600640 1 -- 172.21.15.27:0/667625585 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f141804e850 con 0x7f13f00674c0 2024-01-15T18:15:49.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.264+0000 7f140f400640 1 -- 172.21.15.27:0/667625585 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287360 (secure 0 0 0) 0x7f141804e850 con 0x7f13f00674c0 2024-01-15T18:15:49.263 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:15:49.265 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:15:49.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.264+0000 7f141e600640 1 -- 172.21.15.27:0/667625585 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f13f00674c0 msgr2=0x7f13f0069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:49.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.264+0000 7f141e600640 1 --2- 172.21.15.27:0/667625585 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f13f00674c0 0x7f13f0069980 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f14181676a0 tx=0x7f1408009190 comp rx=0 tx=0).stop 2024-01-15T18:15:49.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.264+0000 7f141e600640 1 -- 172.21.15.27:0/667625585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14180ce710 msgr2=0x7f141815e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:49.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.264+0000 7f141e600640 1 --2- 172.21.15.27:0/667625585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14180ce710 0x7f141815e910 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f140000c480 tx=0x7f140000c940 comp rx=0 tx=0).stop 2024-01-15T18:15:49.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.268+0000 7f141e600640 1 -- 172.21.15.27:0/667625585 shutdown_connections 2024-01-15T18:15:49.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.268+0000 7f141e600640 1 --2- 172.21.15.27:0/667625585 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f13f00674c0 0x7f13f0069980 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:49.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.268+0000 7f141e600640 1 --2- 172.21.15.27:0/667625585 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f141815d0a0 0x7f141815bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:49.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.268+0000 7f141e600640 1 --2- 172.21.15.27:0/667625585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14180ce710 0x7f141815e910 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:49.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.268+0000 7f141e600640 1 --2- 172.21.15.27:0/667625585 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f14180c94c0 0x7f1418159490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:49.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.268+0000 7f141e600640 1 -- 172.21.15.27:0/667625585 >> 172.21.15.27:0/667625585 conn(0x7f14180cec60 msgr2=0x7f141806dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:49.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.268+0000 7f141e600640 1 -- 172.21.15.27:0/667625585 shutdown_connections 2024-01-15T18:15:49.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.268+0000 7f141e600640 1 -- 172.21.15.27:0/667625585 wait complete. 2024-01-15T18:15:49.285 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-01-15T18:15:48.331070+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591166,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":176,"num_write_kb":2427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":79,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":563496,"kb_used_data":8996,"kb_used_omap":12,"kb_used_meta":554355,"kb_avail":754411224,"statfs":{"total":773094113280,"available":772517093376,"internally_reserved":0,"allocated":9211904,"data_stored":4609039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":567660080},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":798,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.192938"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.197673+0000","last_change":"2024-01-15T18:15:45.194410+0000","last_active":"2024-01-15T18:15:45.197673+0000","last_peered":"2024-01-15T18:15:45.197673+0000","last_clean":"2024-01-15T18:15:45.197673+0000","last_became_active":"2024-01-15T18:15:45.193977+0000","last_became_peered":"2024-01-15T18:15:45.193977+0000","last_unstale":"2024-01-15T18:15:45.197673+0000","last_undegraded":"2024-01-15T18:15:45.197673+0000","last_fullsized":"2024-01-15T18:15:45.197673+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:59:33.238710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.193135+0000","last_change":"2024-01-15T18:15:45.185773+0000","last_active":"2024-01-15T18:15:45.193135+0000","last_peered":"2024-01-15T18:15:45.193135+0000","last_clean":"2024-01-15T18:15:45.193135+0000","last_became_active":"2024-01-15T18:15:45.185459+0000","last_became_peered":"2024-01-15T18:15:45.185459+0000","last_unstale":"2024-01-15T18:15:45.193135+0000","last_undegraded":"2024-01-15T18:15:45.193135+0000","last_fullsized":"2024-01-15T18:15:45.193135+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:33:22.524087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196740+0000","last_change":"2024-01-15T18:15:45.190593+0000","last_active":"2024-01-15T18:15:45.196740+0000","last_peered":"2024-01-15T18:15:45.196740+0000","last_clean":"2024-01-15T18:15:45.196740+0000","last_became_active":"2024-01-15T18:15:45.190292+0000","last_became_peered":"2024-01-15T18:15:45.190292+0000","last_unstale":"2024-01-15T18:15:45.196740+0000","last_undegraded":"2024-01-15T18:15:45.196740+0000","last_fullsized":"2024-01-15T18:15:45.196740+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:41:44.153310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.197017+0000","last_change":"2024-01-15T18:15:45.192728+0000","last_active":"2024-01-15T18:15:45.197017+0000","last_peered":"2024-01-15T18:15:45.197017+0000","last_clean":"2024-01-15T18:15:45.197017+0000","last_became_active":"2024-01-15T18:15:45.192270+0000","last_became_peered":"2024-01-15T18:15:45.192270+0000","last_unstale":"2024-01-15T18:15:45.197017+0000","last_undegraded":"2024-01-15T18:15:45.197017+0000","last_fullsized":"2024-01-15T18:15:45.197017+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:10:12.400165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191097+0000","last_change":"2024-01-15T18:15:45.181959+0000","last_active":"2024-01-15T18:15:45.191097+0000","last_peered":"2024-01-15T18:15:45.191097+0000","last_clean":"2024-01-15T18:15:45.191097+0000","last_became_active":"2024-01-15T18:15:45.181526+0000","last_became_peered":"2024-01-15T18:15:45.181526+0000","last_unstale":"2024-01-15T18:15:45.191097+0000","last_undegraded":"2024-01-15T18:15:45.191097+0000","last_fullsized":"2024-01-15T18:15:45.191097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:10:10.202003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.145723+0000","last_change":"2024-01-15T18:15:44.145723+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.145723+0000","last_undegraded":"2024-01-15T18:15:44.145723+0000","last_fullsized":"2024-01-15T18:15:44.145723+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.188332+0000","last_change":"2024-01-15T18:15:45.177382+0000","last_active":"2024-01-15T18:15:45.188332+0000","last_peered":"2024-01-15T18:15:45.188332+0000","last_clean":"2024-01-15T18:15:45.188332+0000","last_became_active":"2024-01-15T18:15:45.173825+0000","last_became_peered":"2024-01-15T18:15:45.173825+0000","last_unstale":"2024-01-15T18:15:45.188332+0000","last_undegraded":"2024-01-15T18:15:45.188332+0000","last_fullsized":"2024-01-15T18:15:45.188332+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:03:55.153676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.144794+0000","last_change":"2024-01-15T18:15:44.144794+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.144794+0000","last_undegraded":"2024-01-15T18:15:44.144794+0000","last_fullsized":"2024-01-15T18:15:44.144794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.145532+0000","last_change":"2024-01-15T18:15:44.145532+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.145532+0000","last_undegraded":"2024-01-15T18:15:44.145532+0000","last_fullsized":"2024-01-15T18:15:44.145532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.195302+0000","last_change":"2024-01-15T18:15:45.190970+0000","last_active":"2024-01-15T18:15:45.195302+0000","last_peered":"2024-01-15T18:15:45.195302+0000","last_clean":"2024-01-15T18:15:45.195302+0000","last_became_active":"2024-01-15T18:15:45.187235+0000","last_became_peered":"2024-01-15T18:15:45.187235+0000","last_unstale":"2024-01-15T18:15:45.195302+0000","last_undegraded":"2024-01-15T18:15:45.195302+0000","last_fullsized":"2024-01-15T18:15:45.195302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:35:49.681365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.186724+0000","last_change":"2024-01-15T18:15:45.179561+0000","last_active":"2024-01-15T18:15:45.186724+0000","last_peered":"2024-01-15T18:15:45.186724+0000","last_clean":"2024-01-15T18:15:45.186724+0000","last_became_active":"2024-01-15T18:15:45.179305+0000","last_became_peered":"2024-01-15T18:15:45.179305+0000","last_unstale":"2024-01-15T18:15:45.186724+0000","last_undegraded":"2024-01-15T18:15:45.186724+0000","last_fullsized":"2024-01-15T18:15:45.186724+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:56:26.621247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191872+0000","last_change":"2024-01-15T18:15:45.189460+0000","last_active":"2024-01-15T18:15:45.191872+0000","last_peered":"2024-01-15T18:15:45.191872+0000","last_clean":"2024-01-15T18:15:45.191872+0000","last_became_active":"2024-01-15T18:15:45.188105+0000","last_became_peered":"2024-01-15T18:15:45.188105+0000","last_unstale":"2024-01-15T18:15:45.191872+0000","last_undegraded":"2024-01-15T18:15:45.191872+0000","last_fullsized":"2024-01-15T18:15:45.191872+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:45:42.971410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196682+0000","last_change":"2024-01-15T18:15:45.190326+0000","last_active":"2024-01-15T18:15:45.196682+0000","last_peered":"2024-01-15T18:15:45.196682+0000","last_clean":"2024-01-15T18:15:45.196682+0000","last_became_active":"2024-01-15T18:15:45.188312+0000","last_became_peered":"2024-01-15T18:15:45.188312+0000","last_unstale":"2024-01-15T18:15:45.196682+0000","last_undegraded":"2024-01-15T18:15:45.196682+0000","last_fullsized":"2024-01-15T18:15:45.196682+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:28:05.260374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191135+0000","last_change":"2024-01-15T18:15:45.180022+0000","last_active":"2024-01-15T18:15:45.191135+0000","last_peered":"2024-01-15T18:15:45.191135+0000","last_clean":"2024-01-15T18:15:45.191135+0000","last_became_active":"2024-01-15T18:15:45.179537+0000","last_became_peered":"2024-01-15T18:15:45.179537+0000","last_unstale":"2024-01-15T18:15:45.191135+0000","last_undegraded":"2024-01-15T18:15:45.191135+0000","last_fullsized":"2024-01-15T18:15:45.191135+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:55.575874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196300+0000","last_change":"2024-01-15T18:15:45.185924+0000","last_active":"2024-01-15T18:15:45.196300+0000","last_peered":"2024-01-15T18:15:45.196300+0000","last_clean":"2024-01-15T18:15:45.196300+0000","last_became_active":"2024-01-15T18:15:45.185669+0000","last_became_peered":"2024-01-15T18:15:45.185669+0000","last_unstale":"2024-01-15T18:15:45.196300+0000","last_undegraded":"2024-01-15T18:15:45.196300+0000","last_fullsized":"2024-01-15T18:15:45.196300+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:08:31.073685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.184890+0000","last_change":"2024-01-15T18:15:45.176262+0000","last_active":"2024-01-15T18:15:45.184890+0000","last_peered":"2024-01-15T18:15:45.184890+0000","last_clean":"2024-01-15T18:15:45.184890+0000","last_became_active":"2024-01-15T18:15:45.175976+0000","last_became_peered":"2024-01-15T18:15:45.175976+0000","last_unstale":"2024-01-15T18:15:45.184890+0000","last_undegraded":"2024-01-15T18:15:45.184890+0000","last_fullsized":"2024-01-15T18:15:45.184890+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:16.351733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191295+0000","last_change":"2024-01-15T18:15:45.184436+0000","last_active":"2024-01-15T18:15:45.191295+0000","last_peered":"2024-01-15T18:15:45.191295+0000","last_clean":"2024-01-15T18:15:45.191295+0000","last_became_active":"2024-01-15T18:15:45.181072+0000","last_became_peered":"2024-01-15T18:15:45.181072+0000","last_unstale":"2024-01-15T18:15:45.191295+0000","last_undegraded":"2024-01-15T18:15:45.191295+0000","last_fullsized":"2024-01-15T18:15:45.191295+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:04:09.741683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191667+0000","last_change":"2024-01-15T18:15:45.184794+0000","last_active":"2024-01-15T18:15:45.191667+0000","last_peered":"2024-01-15T18:15:45.191667+0000","last_clean":"2024-01-15T18:15:45.191667+0000","last_became_active":"2024-01-15T18:15:45.184528+0000","last_became_peered":"2024-01-15T18:15:45.184528+0000","last_unstale":"2024-01-15T18:15:45.191667+0000","last_undegraded":"2024-01-15T18:15:45.191667+0000","last_fullsized":"2024-01-15T18:15:45.191667+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:33.583003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.158290+0000","last_change":"2024-01-15T18:15:44.158290+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.158290+0000","last_undegraded":"2024-01-15T18:15:44.158290+0000","last_fullsized":"2024-01-15T18:15:44.158290+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.203247+0000","last_change":"2024-01-15T18:15:45.202436+0000","last_active":"2024-01-15T18:15:45.203247+0000","last_peered":"2024-01-15T18:15:45.203247+0000","last_clean":"2024-01-15T18:15:45.203247+0000","last_became_active":"2024-01-15T18:15:45.202075+0000","last_became_peered":"2024-01-15T18:15:45.202075+0000","last_unstale":"2024-01-15T18:15:45.203247+0000","last_undegraded":"2024-01-15T18:15:45.203247+0000","last_fullsized":"2024-01-15T18:15:45.203247+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:29:34.404126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198796+0000","last_change":"2024-01-15T18:15:45.197310+0000","last_active":"2024-01-15T18:15:45.198796+0000","last_peered":"2024-01-15T18:15:45.198796+0000","last_clean":"2024-01-15T18:15:45.198796+0000","last_became_active":"2024-01-15T18:15:45.196364+0000","last_became_peered":"2024-01-15T18:15:45.196364+0000","last_unstale":"2024-01-15T18:15:45.198796+0000","last_undegraded":"2024-01-15T18:15:45.198796+0000","last_fullsized":"2024-01-15T18:15:45.198796+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:28:16.596449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191310+0000","last_change":"2024-01-15T18:15:45.185683+0000","last_active":"2024-01-15T18:15:45.191310+0000","last_peered":"2024-01-15T18:15:45.191310+0000","last_clean":"2024-01-15T18:15:45.191310+0000","last_became_active":"2024-01-15T18:15:45.185303+0000","last_became_peered":"2024-01-15T18:15:45.185303+0000","last_unstale":"2024-01-15T18:15:45.191310+0000","last_undegraded":"2024-01-15T18:15:45.191310+0000","last_fullsized":"2024-01-15T18:15:45.191310+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:12:38.030672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198110+0000","last_change":"2024-01-15T18:15:45.195966+0000","last_active":"2024-01-15T18:15:45.198110+0000","last_peered":"2024-01-15T18:15:45.198110+0000","last_clean":"2024-01-15T18:15:45.198110+0000","last_became_active":"2024-01-15T18:15:45.194717+0000","last_became_peered":"2024-01-15T18:15:45.194717+0000","last_unstale":"2024-01-15T18:15:45.198110+0000","last_undegraded":"2024-01-15T18:15:45.198110+0000","last_fullsized":"2024-01-15T18:15:45.198110+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:50:32.308793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.190685+0000","last_change":"2024-01-15T18:15:45.179745+0000","last_active":"2024-01-15T18:15:45.190685+0000","last_peered":"2024-01-15T18:15:45.190685+0000","last_clean":"2024-01-15T18:15:45.190685+0000","last_became_active":"2024-01-15T18:15:45.179329+0000","last_became_peered":"2024-01-15T18:15:45.179329+0000","last_unstale":"2024-01-15T18:15:45.190685+0000","last_undegraded":"2024-01-15T18:15:45.190685+0000","last_fullsized":"2024-01-15T18:15:45.190685+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:01:10.870535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198301+0000","last_change":"2024-01-15T18:15:45.192970+0000","last_active":"2024-01-15T18:15:45.198301+0000","last_peered":"2024-01-15T18:15:45.198301+0000","last_clean":"2024-01-15T18:15:45.198301+0000","last_became_active":"2024-01-15T18:15:45.192474+0000","last_became_peered":"2024-01-15T18:15:45.192474+0000","last_unstale":"2024-01-15T18:15:45.198301+0000","last_undegraded":"2024-01-15T18:15:45.198301+0000","last_fullsized":"2024-01-15T18:15:45.198301+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:38.217871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.183678+0000","last_change":"2024-01-15T18:15:45.180211+0000","last_active":"2024-01-15T18:15:45.183678+0000","last_peered":"2024-01-15T18:15:45.183678+0000","last_clean":"2024-01-15T18:15:45.183678+0000","last_became_active":"2024-01-15T18:15:45.179566+0000","last_became_peered":"2024-01-15T18:15:45.179566+0000","last_unstale":"2024-01-15T18:15:45.183678+0000","last_undegraded":"2024-01-15T18:15:45.183678+0000","last_fullsized":"2024-01-15T18:15:45.183678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:40:13.829197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.192821+0000","last_change":"2024-01-15T18:15:45.189295+0000","last_active":"2024-01-15T18:15:45.192821+0000","last_peered":"2024-01-15T18:15:45.192821+0000","last_clean":"2024-01-15T18:15:45.192821+0000","last_became_active":"2024-01-15T18:15:45.187101+0000","last_became_peered":"2024-01-15T18:15:45.187101+0000","last_unstale":"2024-01-15T18:15:45.192821+0000","last_undegraded":"2024-01-15T18:15:45.192821+0000","last_fullsized":"2024-01-15T18:15:45.192821+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:01:13.745010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.195405+0000","last_change":"2024-01-15T18:15:45.192353+0000","last_active":"2024-01-15T18:15:45.195405+0000","last_peered":"2024-01-15T18:15:45.195405+0000","last_clean":"2024-01-15T18:15:45.195405+0000","last_became_active":"2024-01-15T18:15:45.191563+0000","last_became_peered":"2024-01-15T18:15:45.191563+0000","last_unstale":"2024-01-15T18:15:45.195405+0000","last_undegraded":"2024-01-15T18:15:45.195405+0000","last_fullsized":"2024-01-15T18:15:45.195405+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:57:43.892130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.197122+0000","last_change":"2024-01-15T18:15:45.190607+0000","last_active":"2024-01-15T18:15:45.197122+0000","last_peered":"2024-01-15T18:15:45.197122+0000","last_clean":"2024-01-15T18:15:45.197122+0000","last_became_active":"2024-01-15T18:15:45.189633+0000","last_became_peered":"2024-01-15T18:15:45.189633+0000","last_unstale":"2024-01-15T18:15:45.197122+0000","last_undegraded":"2024-01-15T18:15:45.197122+0000","last_fullsized":"2024-01-15T18:15:45.197122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:35:49.681365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.187329+0000","last_change":"2024-01-15T18:15:45.180268+0000","last_active":"2024-01-15T18:15:45.187329+0000","last_peered":"2024-01-15T18:15:45.187329+0000","last_clean":"2024-01-15T18:15:45.187329+0000","last_became_active":"2024-01-15T18:15:45.179617+0000","last_became_peered":"2024-01-15T18:15:45.179617+0000","last_unstale":"2024-01-15T18:15:45.187329+0000","last_undegraded":"2024-01-15T18:15:45.187329+0000","last_fullsized":"2024-01-15T18:15:45.187329+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:18:22.468719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.186449+0000","last_change":"2024-01-15T18:15:45.177309+0000","last_active":"2024-01-15T18:15:45.186449+0000","last_peered":"2024-01-15T18:15:45.186449+0000","last_clean":"2024-01-15T18:15:45.186449+0000","last_became_active":"2024-01-15T18:15:45.177022+0000","last_became_peered":"2024-01-15T18:15:45.177022+0000","last_unstale":"2024-01-15T18:15:45.186449+0000","last_undegraded":"2024-01-15T18:15:45.186449+0000","last_fullsized":"2024-01-15T18:15:45.186449+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:53.495194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.187438+0000","last_change":"2024-01-15T18:15:45.182964+0000","last_active":"2024-01-15T18:15:45.187438+0000","last_peered":"2024-01-15T18:15:45.187438+0000","last_clean":"2024-01-15T18:15:45.187438+0000","last_became_active":"2024-01-15T18:15:45.182690+0000","last_became_peered":"2024-01-15T18:15:45.182690+0000","last_unstale":"2024-01-15T18:15:45.187438+0000","last_undegraded":"2024-01-15T18:15:45.187438+0000","last_fullsized":"2024-01-15T18:15:45.187438+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:31.079315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.192979+0000","last_change":"2024-01-15T18:15:45.189818+0000","last_active":"2024-01-15T18:15:45.192979+0000","last_peered":"2024-01-15T18:15:45.192979+0000","last_clean":"2024-01-15T18:15:45.192979+0000","last_became_active":"2024-01-15T18:15:45.189534+0000","last_became_peered":"2024-01-15T18:15:45.189534+0000","last_unstale":"2024-01-15T18:15:45.192979+0000","last_undegraded":"2024-01-15T18:15:45.192979+0000","last_fullsized":"2024-01-15T18:15:45.192979+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:13:12.060989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"83'4","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.197594+0000","last_change":"2024-01-15T18:15:44.163810+0000","last_active":"2024-01-15T18:15:45.197594+0000","last_peered":"2024-01-15T18:15:45.197594+0000","last_clean":"2024-01-15T18:15:45.197594+0000","last_became_active":"2024-01-15T18:15:44.163128+0000","last_became_peered":"2024-01-15T18:15:44.163128+0000","last_unstale":"2024-01-15T18:15:45.197594+0000","last_undegraded":"2024-01-15T18:15:45.197594+0000","last_fullsized":"2024-01-15T18:15:45.197594+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:22:07.304644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.202605+0000","last_change":"2024-01-15T18:15:45.201966+0000","last_active":"2024-01-15T18:15:45.202605+0000","last_peered":"2024-01-15T18:15:45.202605+0000","last_clean":"2024-01-15T18:15:45.202605+0000","last_became_active":"2024-01-15T18:15:45.201568+0000","last_became_peered":"2024-01-15T18:15:45.201568+0000","last_unstale":"2024-01-15T18:15:45.202605+0000","last_undegraded":"2024-01-15T18:15:45.202605+0000","last_fullsized":"2024-01-15T18:15:45.202605+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:16.351733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.187621+0000","last_change":"2024-01-15T18:15:44.156714+0000","last_active":"2024-01-15T18:15:45.187621+0000","last_peered":"2024-01-15T18:15:45.187621+0000","last_clean":"2024-01-15T18:15:45.187621+0000","last_became_active":"2024-01-15T18:15:44.156419+0000","last_became_peered":"2024-01-15T18:15:44.156419+0000","last_unstale":"2024-01-15T18:15:45.187621+0000","last_undegraded":"2024-01-15T18:15:45.187621+0000","last_fullsized":"2024-01-15T18:15:45.187621+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:14:41.627743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.197147+0000","last_change":"2024-01-15T18:15:45.192955+0000","last_active":"2024-01-15T18:15:45.197147+0000","last_peered":"2024-01-15T18:15:45.197147+0000","last_clean":"2024-01-15T18:15:45.197147+0000","last_became_active":"2024-01-15T18:15:45.192565+0000","last_became_peered":"2024-01-15T18:15:45.192565+0000","last_unstale":"2024-01-15T18:15:45.197147+0000","last_undegraded":"2024-01-15T18:15:45.197147+0000","last_fullsized":"2024-01-15T18:15:45.197147+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:07:58.498146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.171318+0000","last_change":"2024-01-15T18:15:44.167059+0000","last_active":"2024-01-15T18:15:44.171318+0000","last_peered":"2024-01-15T18:15:44.171318+0000","last_clean":"2024-01-15T18:15:44.171318+0000","last_became_active":"2024-01-15T18:15:44.166829+0000","last_became_peered":"2024-01-15T18:15:44.166829+0000","last_unstale":"2024-01-15T18:15:44.171318+0000","last_undegraded":"2024-01-15T18:15:44.171318+0000","last_fullsized":"2024-01-15T18:15:44.171318+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:39:01.209507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.194382+0000","last_change":"2024-01-15T18:15:45.186505+0000","last_active":"2024-01-15T18:15:45.194382+0000","last_peered":"2024-01-15T18:15:45.194382+0000","last_clean":"2024-01-15T18:15:45.194382+0000","last_became_active":"2024-01-15T18:15:45.186181+0000","last_became_peered":"2024-01-15T18:15:45.186181+0000","last_unstale":"2024-01-15T18:15:45.194382+0000","last_undegraded":"2024-01-15T18:15:45.194382+0000","last_fullsized":"2024-01-15T18:15:45.194382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:03:29.579452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.170558+0000","last_change":"2024-01-15T18:15:44.166697+0000","last_active":"2024-01-15T18:15:44.170558+0000","last_peered":"2024-01-15T18:15:44.170558+0000","last_clean":"2024-01-15T18:15:44.170558+0000","last_became_active":"2024-01-15T18:15:44.166389+0000","last_became_peered":"2024-01-15T18:15:44.166389+0000","last_unstale":"2024-01-15T18:15:44.170558+0000","last_undegraded":"2024-01-15T18:15:44.170558+0000","last_fullsized":"2024-01-15T18:15:44.170558+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:19:36.045488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.187351+0000","last_change":"2024-01-15T18:15:45.180445+0000","last_active":"2024-01-15T18:15:45.187351+0000","last_peered":"2024-01-15T18:15:45.187351+0000","last_clean":"2024-01-15T18:15:45.187351+0000","last_became_active":"2024-01-15T18:15:45.179946+0000","last_became_peered":"2024-01-15T18:15:45.179946+0000","last_unstale":"2024-01-15T18:15:45.187351+0000","last_undegraded":"2024-01-15T18:15:45.187351+0000","last_fullsized":"2024-01-15T18:15:45.187351+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:50:32.308793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.181787+0000","last_change":"2024-01-15T18:15:44.157992+0000","last_active":"2024-01-15T18:15:45.181787+0000","last_peered":"2024-01-15T18:15:45.181787+0000","last_clean":"2024-01-15T18:15:45.181787+0000","last_became_active":"2024-01-15T18:15:44.157721+0000","last_became_peered":"2024-01-15T18:15:44.157721+0000","last_unstale":"2024-01-15T18:15:45.181787+0000","last_undegraded":"2024-01-15T18:15:45.181787+0000","last_fullsized":"2024-01-15T18:15:45.181787+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:27:47.584018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.185831+0000","last_change":"2024-01-15T18:15:45.178738+0000","last_active":"2024-01-15T18:15:45.185831+0000","last_peered":"2024-01-15T18:15:45.185831+0000","last_clean":"2024-01-15T18:15:45.185831+0000","last_became_active":"2024-01-15T18:15:45.178446+0000","last_became_peered":"2024-01-15T18:15:45.178446+0000","last_unstale":"2024-01-15T18:15:45.185831+0000","last_undegraded":"2024-01-15T18:15:45.185831+0000","last_fullsized":"2024-01-15T18:15:45.185831+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:59:33.238710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.187378+0000","last_change":"2024-01-15T18:15:44.154715+0000","last_active":"2024-01-15T18:15:45.187378+0000","last_peered":"2024-01-15T18:15:45.187378+0000","last_clean":"2024-01-15T18:15:45.187378+0000","last_became_active":"2024-01-15T18:15:44.154433+0000","last_became_peered":"2024-01-15T18:15:44.154433+0000","last_unstale":"2024-01-15T18:15:45.187378+0000","last_undegraded":"2024-01-15T18:15:45.187378+0000","last_fullsized":"2024-01-15T18:15:45.187378+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:42:00.915026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.202740+0000","last_change":"2024-01-15T18:15:45.202049+0000","last_active":"2024-01-15T18:15:45.202740+0000","last_peered":"2024-01-15T18:15:45.202740+0000","last_clean":"2024-01-15T18:15:45.202740+0000","last_became_active":"2024-01-15T18:15:45.201742+0000","last_became_peered":"2024-01-15T18:15:45.201742+0000","last_unstale":"2024-01-15T18:15:45.202740+0000","last_undegraded":"2024-01-15T18:15:45.202740+0000","last_fullsized":"2024-01-15T18:15:45.202740+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:56.553890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.172316+0000","last_change":"2024-01-15T18:15:44.167859+0000","last_active":"2024-01-15T18:15:44.172316+0000","last_peered":"2024-01-15T18:15:44.172316+0000","last_clean":"2024-01-15T18:15:44.172316+0000","last_became_active":"2024-01-15T18:15:44.167561+0000","last_became_peered":"2024-01-15T18:15:44.167561+0000","last_unstale":"2024-01-15T18:15:44.172316+0000","last_undegraded":"2024-01-15T18:15:44.172316+0000","last_fullsized":"2024-01-15T18:15:44.172316+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:17.332973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196319+0000","last_change":"2024-01-15T18:15:45.187853+0000","last_active":"2024-01-15T18:15:45.196319+0000","last_peered":"2024-01-15T18:15:45.196319+0000","last_clean":"2024-01-15T18:15:45.196319+0000","last_became_active":"2024-01-15T18:15:45.187552+0000","last_became_peered":"2024-01-15T18:15:45.187552+0000","last_unstale":"2024-01-15T18:15:45.196319+0000","last_undegraded":"2024-01-15T18:15:45.196319+0000","last_fullsized":"2024-01-15T18:15:45.196319+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:56.553890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.162106+0000","last_change":"2024-01-15T18:15:44.155651+0000","last_active":"2024-01-15T18:15:45.162106+0000","last_peered":"2024-01-15T18:15:45.162106+0000","last_clean":"2024-01-15T18:15:45.162106+0000","last_became_active":"2024-01-15T18:15:44.155306+0000","last_became_peered":"2024-01-15T18:15:44.155306+0000","last_unstale":"2024-01-15T18:15:45.162106+0000","last_undegraded":"2024-01-15T18:15:45.162106+0000","last_fullsized":"2024-01-15T18:15:45.162106+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:06:15.812490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.189876+0000","last_change":"2024-01-15T18:15:45.184002+0000","last_active":"2024-01-15T18:15:45.189876+0000","last_peered":"2024-01-15T18:15:45.189876+0000","last_clean":"2024-01-15T18:15:45.189876+0000","last_became_active":"2024-01-15T18:15:45.183624+0000","last_became_peered":"2024-01-15T18:15:45.183624+0000","last_unstale":"2024-01-15T18:15:45.189876+0000","last_undegraded":"2024-01-15T18:15:45.189876+0000","last_fullsized":"2024-01-15T18:15:45.189876+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:03:29.579452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.186739+0000","last_change":"2024-01-15T18:15:44.160123+0000","last_active":"2024-01-15T18:15:45.186739+0000","last_peered":"2024-01-15T18:15:45.186739+0000","last_clean":"2024-01-15T18:15:45.186739+0000","last_became_active":"2024-01-15T18:15:44.159761+0000","last_became_peered":"2024-01-15T18:15:44.159761+0000","last_unstale":"2024-01-15T18:15:45.186739+0000","last_undegraded":"2024-01-15T18:15:45.186739+0000","last_fullsized":"2024-01-15T18:15:45.186739+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:42:00.915026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.144921+0000","last_change":"2024-01-15T18:15:44.144921+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.144921+0000","last_undegraded":"2024-01-15T18:15:44.144921+0000","last_fullsized":"2024-01-15T18:15:44.144921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.187555+0000","last_change":"2024-01-15T18:15:44.166336+0000","last_active":"2024-01-15T18:15:45.187555+0000","last_peered":"2024-01-15T18:15:45.187555+0000","last_clean":"2024-01-15T18:15:45.187555+0000","last_became_active":"2024-01-15T18:15:44.165403+0000","last_became_peered":"2024-01-15T18:15:44.165403+0000","last_unstale":"2024-01-15T18:15:45.187555+0000","last_undegraded":"2024-01-15T18:15:45.187555+0000","last_fullsized":"2024-01-15T18:15:45.187555+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:42:53.652872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.155839+0000","last_change":"2024-01-15T18:15:44.155839+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.155839+0000","last_undegraded":"2024-01-15T18:15:44.155839+0000","last_fullsized":"2024-01-15T18:15:44.155839+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"83'3","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.197477+0000","last_change":"2024-01-15T18:15:44.160184+0000","last_active":"2024-01-15T18:15:45.197477+0000","last_peered":"2024-01-15T18:15:45.197477+0000","last_clean":"2024-01-15T18:15:45.197477+0000","last_became_active":"2024-01-15T18:15:44.159855+0000","last_became_peered":"2024-01-15T18:15:44.159855+0000","last_unstale":"2024-01-15T18:15:45.197477+0000","last_undegraded":"2024-01-15T18:15:45.197477+0000","last_fullsized":"2024-01-15T18:15:45.197477+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:04:08.736438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.192668+0000","last_change":"2024-01-15T18:15:45.187059+0000","last_active":"2024-01-15T18:15:45.192668+0000","last_peered":"2024-01-15T18:15:45.192668+0000","last_clean":"2024-01-15T18:15:45.192668+0000","last_became_active":"2024-01-15T18:15:45.184830+0000","last_became_peered":"2024-01-15T18:15:45.184830+0000","last_unstale":"2024-01-15T18:15:45.192668+0000","last_undegraded":"2024-01-15T18:15:45.192668+0000","last_fullsized":"2024-01-15T18:15:45.192668+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:58:35.413117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.163096+0000","last_change":"2024-01-15T18:15:44.158973+0000","last_active":"2024-01-15T18:15:44.163096+0000","last_peered":"2024-01-15T18:15:44.163096+0000","last_clean":"2024-01-15T18:15:44.163096+0000","last_became_active":"2024-01-15T18:15:44.158684+0000","last_became_peered":"2024-01-15T18:15:44.158684+0000","last_unstale":"2024-01-15T18:15:44.163096+0000","last_undegraded":"2024-01-15T18:15:44.163096+0000","last_fullsized":"2024-01-15T18:15:44.163096+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:22:27.571591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.153863+0000","last_change":"2024-01-15T18:15:44.153863+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.153863+0000","last_undegraded":"2024-01-15T18:15:44.153863+0000","last_fullsized":"2024-01-15T18:15:44.153863+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.201845+0000","last_change":"2024-01-15T18:15:44.166340+0000","last_active":"2024-01-15T18:15:45.201845+0000","last_peered":"2024-01-15T18:15:45.201845+0000","last_clean":"2024-01-15T18:15:45.201845+0000","last_became_active":"2024-01-15T18:15:44.165947+0000","last_became_peered":"2024-01-15T18:15:44.165947+0000","last_unstale":"2024-01-15T18:15:45.201845+0000","last_undegraded":"2024-01-15T18:15:45.201845+0000","last_fullsized":"2024-01-15T18:15:45.201845+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:12:37.715165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.199984+0000","last_change":"2024-01-15T18:15:45.199248+0000","last_active":"2024-01-15T18:15:45.199984+0000","last_peered":"2024-01-15T18:15:45.199984+0000","last_clean":"2024-01-15T18:15:45.199984+0000","last_became_active":"2024-01-15T18:15:45.198929+0000","last_became_peered":"2024-01-15T18:15:45.198929+0000","last_unstale":"2024-01-15T18:15:45.199984+0000","last_undegraded":"2024-01-15T18:15:45.199984+0000","last_fullsized":"2024-01-15T18:15:45.199984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:26:01.224733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.182090+0000","last_change":"2024-01-15T18:15:44.161901+0000","last_active":"2024-01-15T18:15:45.182090+0000","last_peered":"2024-01-15T18:15:45.182090+0000","last_clean":"2024-01-15T18:15:45.182090+0000","last_became_active":"2024-01-15T18:15:44.159756+0000","last_became_peered":"2024-01-15T18:15:44.159756+0000","last_unstale":"2024-01-15T18:15:45.182090+0000","last_undegraded":"2024-01-15T18:15:45.182090+0000","last_fullsized":"2024-01-15T18:15:45.182090+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:41:43.148064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.180743+0000","last_change":"2024-01-15T18:15:45.174549+0000","last_active":"2024-01-15T18:15:45.180743+0000","last_peered":"2024-01-15T18:15:45.180743+0000","last_clean":"2024-01-15T18:15:45.180743+0000","last_became_active":"2024-01-15T18:15:45.174046+0000","last_became_peered":"2024-01-15T18:15:45.174046+0000","last_unstale":"2024-01-15T18:15:45.180743+0000","last_undegraded":"2024-01-15T18:15:45.180743+0000","last_fullsized":"2024-01-15T18:15:45.180743+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:12:42.277859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.159621+0000","last_change":"2024-01-15T18:15:44.154875+0000","last_active":"2024-01-15T18:15:44.159621+0000","last_peered":"2024-01-15T18:15:44.159621+0000","last_clean":"2024-01-15T18:15:44.159621+0000","last_became_active":"2024-01-15T18:15:44.154585+0000","last_became_peered":"2024-01-15T18:15:44.154585+0000","last_unstale":"2024-01-15T18:15:44.159621+0000","last_undegraded":"2024-01-15T18:15:44.159621+0000","last_fullsized":"2024-01-15T18:15:44.159621+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:39:01.209507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198412+0000","last_change":"2024-01-15T18:15:45.193330+0000","last_active":"2024-01-15T18:15:45.198412+0000","last_peered":"2024-01-15T18:15:45.198412+0000","last_clean":"2024-01-15T18:15:45.198412+0000","last_became_active":"2024-01-15T18:15:45.192354+0000","last_became_peered":"2024-01-15T18:15:45.192354+0000","last_unstale":"2024-01-15T18:15:45.198412+0000","last_undegraded":"2024-01-15T18:15:45.198412+0000","last_fullsized":"2024-01-15T18:15:45.198412+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:12:58.057727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.172772+0000","last_change":"2024-01-15T18:15:44.168821+0000","last_active":"2024-01-15T18:15:44.172772+0000","last_peered":"2024-01-15T18:15:44.172772+0000","last_clean":"2024-01-15T18:15:44.172772+0000","last_became_active":"2024-01-15T18:15:44.168551+0000","last_became_peered":"2024-01-15T18:15:44.168551+0000","last_unstale":"2024-01-15T18:15:44.172772+0000","last_undegraded":"2024-01-15T18:15:44.172772+0000","last_fullsized":"2024-01-15T18:15:44.172772+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:14:11.832883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191678+0000","last_change":"2024-01-15T18:15:45.180405+0000","last_active":"2024-01-15T18:15:45.191678+0000","last_peered":"2024-01-15T18:15:45.191678+0000","last_clean":"2024-01-15T18:15:45.191678+0000","last_became_active":"2024-01-15T18:15:45.179918+0000","last_became_peered":"2024-01-15T18:15:45.179918+0000","last_unstale":"2024-01-15T18:15:45.191678+0000","last_undegraded":"2024-01-15T18:15:45.191678+0000","last_fullsized":"2024-01-15T18:15:45.191678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:01:49.697849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.192261+0000","last_change":"2024-01-15T18:15:44.159443+0000","last_active":"2024-01-15T18:15:45.192261+0000","last_peered":"2024-01-15T18:15:45.192261+0000","last_clean":"2024-01-15T18:15:45.192261+0000","last_became_active":"2024-01-15T18:15:44.159118+0000","last_became_peered":"2024-01-15T18:15:44.159118+0000","last_unstale":"2024-01-15T18:15:45.192261+0000","last_undegraded":"2024-01-15T18:15:45.192261+0000","last_fullsized":"2024-01-15T18:15:45.192261+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:13:11.055744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198173+0000","last_change":"2024-01-15T18:15:45.195167+0000","last_active":"2024-01-15T18:15:45.198173+0000","last_peered":"2024-01-15T18:15:45.198173+0000","last_clean":"2024-01-15T18:15:45.198173+0000","last_became_active":"2024-01-15T18:15:45.194897+0000","last_became_peered":"2024-01-15T18:15:45.194897+0000","last_unstale":"2024-01-15T18:15:45.198173+0000","last_undegraded":"2024-01-15T18:15:45.198173+0000","last_fullsized":"2024-01-15T18:15:45.198173+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:53:25.231875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.172853+0000","last_change":"2024-01-15T18:15:44.169724+0000","last_active":"2024-01-15T18:15:44.172853+0000","last_peered":"2024-01-15T18:15:44.172853+0000","last_clean":"2024-01-15T18:15:44.172853+0000","last_became_active":"2024-01-15T18:15:44.169420+0000","last_became_peered":"2024-01-15T18:15:44.169420+0000","last_unstale":"2024-01-15T18:15:44.172853+0000","last_undegraded":"2024-01-15T18:15:44.172853+0000","last_fullsized":"2024-01-15T18:15:44.172853+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:50:20.346298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.185177+0000","last_change":"2024-01-15T18:15:45.175568+0000","last_active":"2024-01-15T18:15:45.185177+0000","last_peered":"2024-01-15T18:15:45.185177+0000","last_clean":"2024-01-15T18:15:45.185177+0000","last_became_active":"2024-01-15T18:15:45.173399+0000","last_became_peered":"2024-01-15T18:15:45.173399+0000","last_unstale":"2024-01-15T18:15:45.185177+0000","last_undegraded":"2024-01-15T18:15:45.185177+0000","last_fullsized":"2024-01-15T18:15:45.185177+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:42:54.658117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.172758+0000","last_change":"2024-01-15T18:15:44.168665+0000","last_active":"2024-01-15T18:15:44.172758+0000","last_peered":"2024-01-15T18:15:44.172758+0000","last_clean":"2024-01-15T18:15:44.172758+0000","last_became_active":"2024-01-15T18:15:44.168375+0000","last_became_peered":"2024-01-15T18:15:44.168375+0000","last_unstale":"2024-01-15T18:15:44.172758+0000","last_undegraded":"2024-01-15T18:15:44.172758+0000","last_fullsized":"2024-01-15T18:15:44.172758+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:09:06.608452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191116+0000","last_change":"2024-01-15T18:15:45.179812+0000","last_active":"2024-01-15T18:15:45.191116+0000","last_peered":"2024-01-15T18:15:45.191116+0000","last_clean":"2024-01-15T18:15:45.191116+0000","last_became_active":"2024-01-15T18:15:45.179148+0000","last_became_peered":"2024-01-15T18:15:45.179148+0000","last_unstale":"2024-01-15T18:15:45.191116+0000","last_undegraded":"2024-01-15T18:15:45.191116+0000","last_fullsized":"2024-01-15T18:15:45.191116+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:47:52.953626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196980+0000","last_change":"2024-01-15T18:15:44.163128+0000","last_active":"2024-01-15T18:15:45.196980+0000","last_peered":"2024-01-15T18:15:45.196980+0000","last_clean":"2024-01-15T18:15:45.196980+0000","last_became_active":"2024-01-15T18:15:44.162839+0000","last_became_peered":"2024-01-15T18:15:44.162839+0000","last_unstale":"2024-01-15T18:15:45.196980+0000","last_undegraded":"2024-01-15T18:15:45.196980+0000","last_fullsized":"2024-01-15T18:15:45.196980+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:34:15.516676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.146439+0000","last_change":"2024-01-15T18:15:44.146439+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.146439+0000","last_undegraded":"2024-01-15T18:15:44.146439+0000","last_fullsized":"2024-01-15T18:15:44.146439+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.174831+0000","last_change":"2024-01-15T18:15:44.167859+0000","last_active":"2024-01-15T18:15:45.174831+0000","last_peered":"2024-01-15T18:15:45.174831+0000","last_clean":"2024-01-15T18:15:45.174831+0000","last_became_active":"2024-01-15T18:15:44.167592+0000","last_became_peered":"2024-01-15T18:15:44.167592+0000","last_unstale":"2024-01-15T18:15:45.174831+0000","last_undegraded":"2024-01-15T18:15:45.174831+0000","last_fullsized":"2024-01-15T18:15:45.174831+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:02:39.986413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.190664+0000","last_change":"2024-01-15T18:15:45.179572+0000","last_active":"2024-01-15T18:15:45.190664+0000","last_peered":"2024-01-15T18:15:45.190664+0000","last_clean":"2024-01-15T18:15:45.190664+0000","last_became_active":"2024-01-15T18:15:45.178802+0000","last_became_peered":"2024-01-15T18:15:45.178802+0000","last_unstale":"2024-01-15T18:15:45.190664+0000","last_undegraded":"2024-01-15T18:15:45.190664+0000","last_fullsized":"2024-01-15T18:15:45.190664+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:26:43.360415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.173615+0000","last_change":"2024-01-15T18:15:44.168997+0000","last_active":"2024-01-15T18:15:45.173615+0000","last_peered":"2024-01-15T18:15:45.173615+0000","last_clean":"2024-01-15T18:15:45.173615+0000","last_became_active":"2024-01-15T18:15:44.168678+0000","last_became_peered":"2024-01-15T18:15:44.168678+0000","last_unstale":"2024-01-15T18:15:45.173615+0000","last_undegraded":"2024-01-15T18:15:45.173615+0000","last_fullsized":"2024-01-15T18:15:45.173615+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:20:19.634573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.145752+0000","last_change":"2024-01-15T18:15:44.145752+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.145752+0000","last_undegraded":"2024-01-15T18:15:44.145752+0000","last_fullsized":"2024-01-15T18:15:44.145752+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.178682+0000","last_change":"2024-01-15T18:15:44.162077+0000","last_active":"2024-01-15T18:15:45.178682+0000","last_peered":"2024-01-15T18:15:45.178682+0000","last_clean":"2024-01-15T18:15:45.178682+0000","last_became_active":"2024-01-15T18:15:44.161794+0000","last_became_peered":"2024-01-15T18:15:44.161794+0000","last_unstale":"2024-01-15T18:15:45.178682+0000","last_undegraded":"2024-01-15T18:15:45.178682+0000","last_fullsized":"2024-01-15T18:15:45.178682+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:05:24.652003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.185502+0000","last_change":"2024-01-15T18:15:45.175862+0000","last_active":"2024-01-15T18:15:45.185502+0000","last_peered":"2024-01-15T18:15:45.185502+0000","last_clean":"2024-01-15T18:15:45.185502+0000","last_became_active":"2024-01-15T18:15:45.175569+0000","last_became_peered":"2024-01-15T18:15:45.175569+0000","last_unstale":"2024-01-15T18:15:45.185502+0000","last_undegraded":"2024-01-15T18:15:45.185502+0000","last_fullsized":"2024-01-15T18:15:45.185502+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:22:08.309890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.193674+0000","last_change":"2024-01-15T18:15:44.168092+0000","last_active":"2024-01-15T18:15:45.193674+0000","last_peered":"2024-01-15T18:15:45.193674+0000","last_clean":"2024-01-15T18:15:45.193674+0000","last_became_active":"2024-01-15T18:15:44.167828+0000","last_became_peered":"2024-01-15T18:15:44.167828+0000","last_unstale":"2024-01-15T18:15:45.193674+0000","last_undegraded":"2024-01-15T18:15:45.193674+0000","last_fullsized":"2024-01-15T18:15:45.193674+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:27:26.285133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198128+0000","last_change":"2024-01-15T18:15:45.194246+0000","last_active":"2024-01-15T18:15:45.198128+0000","last_peered":"2024-01-15T18:15:45.198128+0000","last_clean":"2024-01-15T18:15:45.198128+0000","last_became_active":"2024-01-15T18:15:45.193864+0000","last_became_peered":"2024-01-15T18:15:45.193864+0000","last_unstale":"2024-01-15T18:15:45.198128+0000","last_undegraded":"2024-01-15T18:15:45.198128+0000","last_fullsized":"2024-01-15T18:15:45.198128+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:05:25.657249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.162597+0000","last_change":"2024-01-15T18:15:44.166561+0000","last_active":"2024-01-15T18:15:45.162597+0000","last_peered":"2024-01-15T18:15:45.162597+0000","last_clean":"2024-01-15T18:15:45.162597+0000","last_became_active":"2024-01-15T18:15:44.166280+0000","last_became_peered":"2024-01-15T18:15:44.166280+0000","last_unstale":"2024-01-15T18:15:45.162597+0000","last_undegraded":"2024-01-15T18:15:45.162597+0000","last_fullsized":"2024-01-15T18:15:45.162597+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:30.074069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198142+0000","last_change":"2024-01-15T18:15:45.196412+0000","last_active":"2024-01-15T18:15:45.198142+0000","last_peered":"2024-01-15T18:15:45.198142+0000","last_clean":"2024-01-15T18:15:45.198142+0000","last_became_active":"2024-01-15T18:15:45.196022+0000","last_became_peered":"2024-01-15T18:15:45.196022+0000","last_unstale":"2024-01-15T18:15:45.198142+0000","last_undegraded":"2024-01-15T18:15:45.198142+0000","last_fullsized":"2024-01-15T18:15:45.198142+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:09:32.875279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.184853+0000","last_change":"2024-01-15T18:15:44.163120+0000","last_active":"2024-01-15T18:15:45.184853+0000","last_peered":"2024-01-15T18:15:45.184853+0000","last_clean":"2024-01-15T18:15:45.184853+0000","last_became_active":"2024-01-15T18:15:44.162805+0000","last_became_peered":"2024-01-15T18:15:44.162805+0000","last_unstale":"2024-01-15T18:15:45.184853+0000","last_undegraded":"2024-01-15T18:15:45.184853+0000","last_fullsized":"2024-01-15T18:15:45.184853+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:49.344761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191100+0000","last_change":"2024-01-15T18:15:45.185283+0000","last_active":"2024-01-15T18:15:45.191100+0000","last_peered":"2024-01-15T18:15:45.191100+0000","last_clean":"2024-01-15T18:15:45.191100+0000","last_became_active":"2024-01-15T18:15:45.185001+0000","last_became_peered":"2024-01-15T18:15:45.185001+0000","last_unstale":"2024-01-15T18:15:45.191100+0000","last_undegraded":"2024-01-15T18:15:45.191100+0000","last_fullsized":"2024-01-15T18:15:45.191100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:41:05.797910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.183836+0000","last_change":"2024-01-15T18:15:44.163840+0000","last_active":"2024-01-15T18:15:45.183836+0000","last_peered":"2024-01-15T18:15:45.183836+0000","last_clean":"2024-01-15T18:15:45.183836+0000","last_became_active":"2024-01-15T18:15:44.163441+0000","last_became_peered":"2024-01-15T18:15:44.163441+0000","last_unstale":"2024-01-15T18:15:45.183836+0000","last_undegraded":"2024-01-15T18:15:45.183836+0000","last_fullsized":"2024-01-15T18:15:45.183836+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:41:04.792664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.145185+0000","last_change":"2024-01-15T18:15:44.145185+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.145185+0000","last_undegraded":"2024-01-15T18:15:44.145185+0000","last_fullsized":"2024-01-15T18:15:44.145185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.172834+0000","last_change":"2024-01-15T18:15:44.169437+0000","last_active":"2024-01-15T18:15:44.172834+0000","last_peered":"2024-01-15T18:15:44.172834+0000","last_clean":"2024-01-15T18:15:44.172834+0000","last_became_active":"2024-01-15T18:15:44.169170+0000","last_became_peered":"2024-01-15T18:15:44.169170+0000","last_unstale":"2024-01-15T18:15:44.172834+0000","last_undegraded":"2024-01-15T18:15:44.172834+0000","last_fullsized":"2024-01-15T18:15:44.172834+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:50:41.854134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.156826+0000","last_change":"2024-01-15T18:15:44.156826+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.156826+0000","last_undegraded":"2024-01-15T18:15:44.156826+0000","last_fullsized":"2024-01-15T18:15:44.156826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.164910+0000","last_change":"2024-01-15T18:15:44.154568+0000","last_active":"2024-01-15T18:15:45.164910+0000","last_peered":"2024-01-15T18:15:45.164910+0000","last_clean":"2024-01-15T18:15:45.164910+0000","last_became_active":"2024-01-15T18:15:44.154277+0000","last_became_peered":"2024-01-15T18:15:44.154277+0000","last_unstale":"2024-01-15T18:15:45.164910+0000","last_undegraded":"2024-01-15T18:15:45.164910+0000","last_fullsized":"2024-01-15T18:15:45.164910+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:39:47.546718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191500+0000","last_change":"2024-01-15T18:15:45.182784+0000","last_active":"2024-01-15T18:15:45.191500+0000","last_peered":"2024-01-15T18:15:45.191500+0000","last_clean":"2024-01-15T18:15:45.191500+0000","last_became_active":"2024-01-15T18:15:45.179168+0000","last_became_peered":"2024-01-15T18:15:45.179168+0000","last_unstale":"2024-01-15T18:15:45.191500+0000","last_undegraded":"2024-01-15T18:15:45.191500+0000","last_fullsized":"2024-01-15T18:15:45.191500+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:34:16.521921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.183666+0000","last_change":"2024-01-15T18:15:44.160891+0000","last_active":"2024-01-15T18:15:45.183666+0000","last_peered":"2024-01-15T18:15:45.183666+0000","last_clean":"2024-01-15T18:15:45.183666+0000","last_became_active":"2024-01-15T18:15:44.160625+0000","last_became_peered":"2024-01-15T18:15:44.160625+0000","last_unstale":"2024-01-15T18:15:45.183666+0000","last_undegraded":"2024-01-15T18:15:45.183666+0000","last_fullsized":"2024-01-15T18:15:45.183666+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:52.489949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196072+0000","last_change":"2024-01-15T18:15:45.192344+0000","last_active":"2024-01-15T18:15:45.196072+0000","last_peered":"2024-01-15T18:15:45.196072+0000","last_clean":"2024-01-15T18:15:45.196072+0000","last_became_active":"2024-01-15T18:15:45.191845+0000","last_became_peered":"2024-01-15T18:15:45.191845+0000","last_unstale":"2024-01-15T18:15:45.196072+0000","last_undegraded":"2024-01-15T18:15:45.196072+0000","last_fullsized":"2024-01-15T18:15:45.196072+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:01:31.532362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.163378+0000","last_change":"2024-01-15T18:15:44.156428+0000","last_active":"2024-01-15T18:15:45.163378+0000","last_peered":"2024-01-15T18:15:45.163378+0000","last_clean":"2024-01-15T18:15:45.163378+0000","last_became_active":"2024-01-15T18:15:44.156132+0000","last_became_peered":"2024-01-15T18:15:44.156132+0000","last_unstale":"2024-01-15T18:15:45.163378+0000","last_undegraded":"2024-01-15T18:15:45.163378+0000","last_fullsized":"2024-01-15T18:15:45.163378+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:49.344761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.186454+0000","last_change":"2024-01-15T18:15:45.178846+0000","last_active":"2024-01-15T18:15:45.186454+0000","last_peered":"2024-01-15T18:15:45.186454+0000","last_clean":"2024-01-15T18:15:45.186454+0000","last_became_active":"2024-01-15T18:15:45.178552+0000","last_became_peered":"2024-01-15T18:15:45.178552+0000","last_unstale":"2024-01-15T18:15:45.186454+0000","last_undegraded":"2024-01-15T18:15:45.186454+0000","last_fullsized":"2024-01-15T18:15:45.186454+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:45:31.736245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.183987+0000","last_change":"2024-01-15T18:15:44.170831+0000","last_active":"2024-01-15T18:15:45.183987+0000","last_peered":"2024-01-15T18:15:45.183987+0000","last_clean":"2024-01-15T18:15:45.183987+0000","last_became_active":"2024-01-15T18:15:44.170489+0000","last_became_peered":"2024-01-15T18:15:44.170489+0000","last_unstale":"2024-01-15T18:15:45.183987+0000","last_undegraded":"2024-01-15T18:15:45.183987+0000","last_fullsized":"2024-01-15T18:15:45.183987+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:54:03.053923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":218,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.162031+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:15:44.162031+0000","last_peered":"2024-01-15T18:15:44.162031+0000","last_clean":"2024-01-15T18:15:44.162031+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:15:44.162031+0000","last_undegraded":"2024-01-15T18:15:44.162031+0000","last_fullsized":"2024-01-15T18:15:44.162031+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":798,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509575,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77092,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294748,"statfs":{"total":96636764160,"available":96557821952,"internally_reserved":0,"allocated":1593344,"data_stored":1017889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77330890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":1.171,"15min":0.874},"min":{"1min":0.52400000000000002,"5min":0.38100000000000001,"15min":0.36199999999999999},"max":{"1min":1.0089999999999999,"5min":9.6989999999999998,"15min":10.284000000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.68799999999999994,"5min":1.1399999999999999,"15min":0.90000000000000002},"min":{"1min":0.48599999999999999,"5min":0.39100000000000001,"15min":0.378},"max":{"1min":0.83099999999999996,"5min":16.968,"15min":16.968},"last":6.6189999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":1.3129999999999999,"15min":0.91400000000000003},"min":{"1min":0.52600000000000002,"5min":0.36099999999999999,"15min":0.33800000000000002},"max":{"1min":0.90100000000000002,"5min":15.791,"15min":15.791},"last":4.944},{"interface":"front","average":{"1min":0.71099999999999997,"5min":1.109,"15min":0.90000000000000002},"min":{"1min":0.51500000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.85799999999999998,"5min":10.842000000000001,"15min":10.842000000000001},"last":4.6699999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":1.2609999999999999,"15min":0.92500000000000004},"min":{"1min":0.502,"5min":0.40400000000000003,"15min":0.378},"max":{"1min":0.98299999999999998,"5min":22.318000000000001,"15min":22.318000000000001},"last":4.4669999999999996},{"interface":"front","average":{"1min":0.76200000000000001,"5min":1.159,"15min":0.89400000000000002},"min":{"1min":0.58399999999999996,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.93700000000000006,"5min":9.7539999999999996,"15min":9.7539999999999996},"last":4.7030000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":1.387,"15min":0.98899999999999999},"min":{"1min":0.59799999999999998,"5min":0.47099999999999997,"15min":0.38100000000000001},"max":{"1min":0.93300000000000005,"5min":18.152999999999999,"15min":18.152999999999999},"last":1.456},{"interface":"front","average":{"1min":0.81599999999999995,"5min":1.274,"15min":0.98599999999999999},"min":{"1min":0.622,"5min":0.433,"15min":0.43099999999999999},"max":{"1min":0.96599999999999997,"5min":9.7959999999999994,"15min":10.34},"last":2.6909999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":1.2150000000000001,"15min":0.92700000000000005},"min":{"1min":0.44,"5min":0.314,"15min":0.314},"max":{"1min":0.85199999999999998,"5min":20.387,"15min":20.387},"last":4.4480000000000004},{"interface":"front","average":{"1min":0.623,"5min":1.353,"15min":0.98399999999999999},"min":{"1min":0.47099999999999997,"5min":0.38800000000000001,"15min":0.36099999999999999},"max":{"1min":1.032,"5min":17.215,"15min":17.215},"last":3.6309999999999998}]},{"osd":5,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":1.2410000000000001,"15min":0.94499999999999995},"min":{"1min":0.46400000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.95299999999999996,"5min":9.8450000000000006,"15min":19.959},"last":4.8639999999999999},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1.278,"15min":0.97699999999999998},"min":{"1min":0.439,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83999999999999997,"5min":19.905000000000001,"15min":19.905000000000001},"last":4.3959999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":1.4470000000000001,"15min":1.1499999999999999},"min":{"1min":0.59499999999999997,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97099999999999997,"5min":20.492000000000001,"15min":22.478999999999999},"last":4.3330000000000002},{"interface":"front","average":{"1min":0.749,"5min":1.272,"15min":1.0600000000000001},"min":{"1min":0.52200000000000002,"5min":0.44,"15min":0.38300000000000001},"max":{"1min":0.998,"5min":17.132000000000001,"15min":21.509},"last":3.573}]}]},{"osd":6,"up_from":15,"seq":64424509575,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58524,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313316,"statfs":{"total":96636764160,"available":96576835584,"internally_reserved":0,"allocated":999424,"data_stored":427521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":1.3009999999999999,"15min":0.95699999999999996},"min":{"1min":0.55700000000000005,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.96099999999999997,"5min":32.587000000000003,"15min":32.587000000000003},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.75,"5min":1.0660000000000001,"15min":0.86199999999999999},"min":{"1min":0.49199999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.97799999999999998,"5min":16.193000000000001,"15min":16.193000000000001},"last":0.69399999999999995}]},{"osd":1,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.92400000000000004,"15min":0.98699999999999999},"min":{"1min":0.439,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.90700000000000003,"5min":7.7649999999999997,"15min":53.451999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.72499999999999998,"5min":1.2090000000000001,"15min":0.92600000000000005},"min":{"1min":0.53400000000000003,"5min":0.437,"15min":0.42899999999999999},"max":{"1min":0.86299999999999999,"5min":32.512,"15min":32.512},"last":0.51700000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":1.24,"15min":0.96799999999999997},"min":{"1min":0.58399999999999996,"5min":0.41699999999999998,"15min":0.38900000000000001},"max":{"1min":0.92200000000000004,"5min":32.472999999999999,"15min":32.472999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.754,"5min":1.0649999999999999,"15min":0.88500000000000001},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.38500000000000001},"max":{"1min":1.02,"5min":16.602,"15min":16.602},"last":0.65300000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":1.1699999999999999,"15min":0.93600000000000005},"min":{"1min":0.56399999999999995,"5min":0.46000000000000002,"15min":0.41399999999999998},"max":{"1min":0.92100000000000004,"5min":16.388000000000002,"15min":16.388000000000002},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.0680000000000001,"15min":1.075},"min":{"1min":0.54400000000000004,"5min":0.432,"15min":0.41599999999999998},"max":{"1min":0.95999999999999996,"5min":9.3719999999999999,"15min":53.398000000000003},"last":0.67200000000000004}]},{"osd":4,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":1.149,"15min":0.89500000000000002},"min":{"1min":0.435,"5min":0.38100000000000001,"15min":0.34999999999999998},"max":{"1min":0.79700000000000004,"5min":32.549999999999997,"15min":32.549999999999997},"last":0.63},{"interface":"front","average":{"1min":0.59499999999999997,"5min":1.0409999999999999,"15min":0.84899999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.34499999999999997},"max":{"1min":0.96499999999999997,"5min":11.852,"15min":11.852},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":1.0720000000000001,"15min":1.034},"min":{"1min":0.42699999999999999,"5min":0.35799999999999998,"15min":0.30399999999999999},"max":{"1min":0.77300000000000002,"5min":13.087999999999999,"15min":53.229999999999997},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.68700000000000006,"5min":1.375,"15min":1.0349999999999999},"min":{"1min":0.42599999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.0009999999999999,"5min":32.404000000000003,"15min":32.404000000000003},"last":0.71899999999999997}]},{"osd":7,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":1.294,"15min":1.1200000000000001},"min":{"1min":0.52900000000000003,"5min":0.436,"15min":0.40400000000000003},"max":{"1min":0.94099999999999995,"5min":23.991,"15min":45.070999999999998},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":1.3260000000000001,"15min":1.2170000000000001},"min":{"1min":0.51200000000000001,"5min":0.40999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88200000000000001,"5min":40.972999999999999,"15min":53.338000000000001},"last":0.753}]}]},{"osd":4,"up_from":15,"seq":64424509575,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58528,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313312,"statfs":{"total":96636764160,"available":96576831488,"internally_reserved":0,"allocated":999424,"data_stored":427521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":1.111,"15min":0.88500000000000001},"min":{"1min":0.46000000000000002,"5min":0.437,"15min":0.32900000000000001},"max":{"1min":0.96199999999999997,"5min":13.492000000000001,"15min":23.318000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.92100000000000004,"15min":0.80200000000000005},"min":{"1min":0.41999999999999998,"5min":0.378,"15min":0.32600000000000001},"max":{"1min":0.875,"5min":14.723000000000001,"15min":14.723000000000001},"last":0.71099999999999997}]},{"osd":1,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":1.0049999999999999,"15min":0.78800000000000003},"min":{"1min":0.50800000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.84299999999999997,"5min":9.5700000000000003,"15min":9.5700000000000003},"last":0.63},{"interface":"front","average":{"1min":0.71999999999999997,"5min":1.155,"15min":0.91400000000000003},"min":{"1min":0.47399999999999998,"5min":0.40200000000000002,"15min":0.36899999999999999},"max":{"1min":0.84999999999999998,"5min":13.57,"15min":22.408999999999999},"last":0.751}]},{"osd":2,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":1.038,"15min":0.79400000000000004},"min":{"1min":0.53000000000000003,"5min":0.41599999999999998,"15min":0.34399999999999997},"max":{"1min":0.84199999999999997,"5min":9.7089999999999996,"15min":9.7089999999999996},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.73499999999999999,"5min":1.2949999999999999,"15min":1.014},"min":{"1min":0.439,"5min":0.439,"15min":0.39100000000000001},"max":{"1min":0.92600000000000005,"5min":14.118,"15min":22.338999999999999},"last":0.69999999999999996}]},{"osd":3,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":1.016,"15min":0.873},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.41199999999999998},"max":{"1min":0.83199999999999996,"5min":14.763999999999999,"15min":14.763999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.77300000000000002,"5min":1.1479999999999999,"15min":0.90500000000000003},"min":{"1min":0.622,"5min":0.48099999999999998,"15min":0.45500000000000002},"max":{"1min":0.89400000000000002,"5min":9.6289999999999996,"15min":9.6289999999999996},"last":0.72999999999999998}]},{"osd":5,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":1.1299999999999999,"15min":0.96299999999999997},"min":{"1min":0.39000000000000001,"5min":0.34799999999999998,"15min":0.33200000000000002},"max":{"1min":0.875,"5min":15.787000000000001,"15min":31.617999999999999},"last":0.501},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.94699999999999995,"15min":0.86699999999999999},"min":{"1min":0.44,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.92300000000000004,"5min":16.388000000000002,"15min":17.145},"last":0.59899999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.98399999999999999,"15min":0.83099999999999996},"min":{"1min":0.42799999999999999,"5min":0.37,"15min":0.35499999999999998},"max":{"1min":0.94599999999999995,"5min":11.124000000000001,"15min":11.124000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.71699999999999997,"5min":1.3580000000000001,"15min":1.226},"min":{"1min":0.53700000000000003,"5min":0.434,"15min":0.40200000000000002},"max":{"1min":1.018,"5min":32.308,"15min":32.308},"last":0.55500000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":1.002,"15min":1.0109999999999999},"min":{"1min":0.49399999999999999,"5min":0.41099999999999998,"15min":0.34799999999999998},"max":{"1min":0.89900000000000002,"5min":14.653,"15min":39.256},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.746,"5min":1.2290000000000001,"15min":1.0129999999999999},"min":{"1min":0.57799999999999996,"5min":0.46500000000000002,"15min":0.42199999999999999},"max":{"1min":0.98599999999999999,"5min":16.277000000000001,"15min":26.573},"last":0.57399999999999995}]}]},{"osd":3,"up_from":15,"seq":64424509574,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80676,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94291164,"statfs":{"total":96636764160,"available":96554151936,"internally_reserved":0,"allocated":999424,"data_stored":427521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":81590727},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.59199999999999997,"15min":0.49099999999999999},"min":{"1min":0.29299999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.751,"5min":5.8849999999999998,"15min":5.8849999999999998},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.68999999999999995,"15min":0.61599999999999999},"min":{"1min":0.38600000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.77500000000000002,"5min":4.7990000000000004,"15min":17.756},"last":0.29799999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.73099999999999998,"15min":0.59299999999999997},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.621,"5min":12.307,"15min":12.307},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.69399999999999995,"15min":0.59299999999999997},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.32200000000000001},"max":{"1min":0.83799999999999997,"5min":6.0640000000000001,"15min":6.0640000000000001},"last":0.38600000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.83099999999999996,"15min":0.69599999999999995},"min":{"1min":0.46800000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":3.633,"5min":12.468999999999999,"15min":12.468999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.85399999999999998,"15min":0.68500000000000005},"min":{"1min":0.48599999999999999,"5min":0.41599999999999998,"15min":0.34599999999999997},"max":{"1min":1.0129999999999999,"5min":10.837999999999999,"15min":10.837999999999999},"last":0.47199999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":1.292,"15min":1.028},"min":{"1min":0.59499999999999997,"5min":0.375,"15min":0.375},"max":{"1min":1.014,"5min":13.863,"15min":17.853999999999999},"last":0.874},{"interface":"front","average":{"1min":0.82899999999999996,"5min":1.2809999999999999,"15min":0.96899999999999997},"min":{"1min":0.63,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.0469999999999999,"5min":13.31,"15min":13.31},"last":0.72099999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":1.369,"15min":0.97599999999999998},"min":{"1min":0.66500000000000004,"5min":0.52100000000000002,"15min":0.51000000000000001},"max":{"1min":0.94399999999999995,"5min":17.687999999999999,"15min":17.687999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.206,"15min":0.97399999999999998},"min":{"1min":0.51700000000000002,"5min":0.438,"15min":0.438},"max":{"1min":3.2160000000000002,"5min":20.515999999999998,"15min":20.515999999999998},"last":0.82099999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":1.1499999999999999,"15min":0.98999999999999999},"min":{"1min":0.55600000000000005,"5min":0.432,"15min":0.432},"max":{"1min":1.1060000000000001,"5min":13.250999999999999,"15min":23.100999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.82099999999999995,"5min":1.2150000000000001,"15min":0.93799999999999994},"min":{"1min":0.60799999999999998,"5min":0.5,"15min":0.49399999999999999},"max":{"1min":1.0169999999999999,"5min":12.76,"15min":12.76},"last":0.93400000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.7889999999999999,"15min":1.1719999999999999},"min":{"1min":0.625,"5min":0.48599999999999999,"15min":0.48199999999999998},"max":{"1min":0.99099999999999999,"5min":24.321999999999999,"15min":24.321999999999999},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.81999999999999995,"5min":1.329,"15min":1.028},"min":{"1min":0.58399999999999996,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0569999999999999,"5min":13.385,"15min":13.385},"last":0.78100000000000003}]}]},{"osd":2,"up_from":15,"seq":64424509574,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76532,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295308,"statfs":{"total":96636764160,"available":96558395392,"internally_reserved":0,"allocated":1015808,"data_stored":434962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.51900000000000002,"15min":0.51200000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26500000000000001},"max":{"1min":0.66500000000000004,"5min":1.857,"15min":5.1849999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54200000000000004,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.252,"15min":0.252},"max":{"1min":0.89300000000000002,"5min":2.3759999999999999,"15min":7.516},"last":0.55900000000000005}]},{"osd":1,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.70599999999999996,"15min":0.61599999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.875,"5min":10.907,"15min":10.907},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.68500000000000005,"15min":0.58799999999999997},"min":{"1min":0.38100000000000001,"5min":0.28100000000000003,"15min":0.27000000000000002},"max":{"1min":0.73799999999999999,"5min":7.2119999999999997,"15min":8.9190000000000005},"last":0.53500000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.60599999999999998,"15min":0.626},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.72999999999999998,"5min":2.7290000000000001,"15min":12.185},"last":0.621},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.747,"15min":0.66100000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.34000000000000002},"max":{"1min":0.96699999999999997,"5min":7.6760000000000002,"15min":7.6760000000000002},"last":0.67900000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.89500000000000002,"15min":0.85799999999999998},"min":{"1min":0.55700000000000005,"5min":0.47499999999999998,"15min":0.45800000000000002},"max":{"1min":1.1870000000000001,"5min":4.3150000000000004,"15min":8.6609999999999996},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.755,"5min":0.878,"15min":0.90900000000000003},"min":{"1min":0.5,"5min":0.45800000000000002,"15min":0.42499999999999999},"max":{"1min":1.075,"5min":8.3190000000000008,"15min":19.146999999999998},"last":0.84199999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.80400000000000005,"15min":0.90200000000000002},"min":{"1min":0.52400000000000002,"5min":0.442,"15min":0.44},"max":{"1min":1.1599999999999999,"5min":2.7789999999999999,"15min":31.710000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.996,"15min":1},"min":{"1min":0.54400000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.155,"5min":10.460000000000001,"15min":22.539000000000001},"last":0.77300000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.99399999999999999,"15min":0.89700000000000002},"min":{"1min":0.501,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.1220000000000001,"5min":8.3580000000000005,"15min":9.4540000000000006},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.83799999999999997,"15min":0.90500000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0489999999999999,"5min":2.669,"15min":18.614999999999998},"last":0.88}]},{"osd":7,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.125,"15min":0.95399999999999996},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.47999999999999998},"max":{"1min":1.157,"5min":11.606,"15min":11.606},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.92200000000000004,"15min":0.98699999999999999},"min":{"1min":0.60399999999999998,"5min":0.51700000000000002,"15min":0.502},"max":{"1min":1.1180000000000001,"5min":3.6850000000000001,"15min":17.013000000000002},"last":0.86199999999999999}]}]},{"osd":1,"up_from":15,"seq":64424509576,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80700,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94291140,"statfs":{"total":96636764160,"available":96554127360,"internally_reserved":0,"allocated":1024000,"data_stored":434574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81590725},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.70699999999999996,"15min":0.66300000000000003},"min":{"1min":0.26300000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.745,"5min":17.396999999999998,"15min":19.577000000000002},"last":0.623},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.74099999999999999,"15min":0.76000000000000001},"min":{"1min":0.33900000000000002,"5min":0.29199999999999998,"15min":0.251},"max":{"1min":0.83999999999999997,"5min":12.378,"15min":48.277999999999999},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.61399999999999999,"15min":0.71399999999999997},"min":{"1min":0.33600000000000002,"5min":0.309,"15min":0.30299999999999999},"max":{"1min":0.82799999999999996,"5min":2.8610000000000002,"15min":40.698},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.65900000000000003,"15min":0.90100000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.317},"max":{"1min":0.89500000000000002,"5min":3.0449999999999999,"15min":48.316000000000003},"last":0.441}]},{"osd":3,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.71799999999999997,"15min":0.84499999999999997},"min":{"1min":0.434,"5min":0.39800000000000002,"15min":0.33700000000000002},"max":{"1min":1.2509999999999999,"5min":3.0899999999999999,"15min":48.234999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.83799999999999997,"15min":0.68999999999999995},"min":{"1min":0.315,"5min":0.315,"15min":0.311},"max":{"1min":4.0780000000000003,"5min":9.9719999999999995,"15min":9.9719999999999995},"last":0.46000000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":1.0980000000000001,"15min":0.88700000000000001},"min":{"1min":0.49299999999999999,"5min":0.48999999999999999,"15min":0.44500000000000001},"max":{"1min":1.446,"5min":9.9329999999999998,"15min":9.9329999999999998},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.80800000000000005,"5min":1.026,"15min":0.93600000000000005},"min":{"1min":0.47499999999999998,"5min":0.45800000000000002,"15min":0.42499999999999999},"max":{"1min":1.163,"5min":9.8610000000000007,"15min":19.829000000000001},"last":0.88300000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.95399999999999996,"15min":1.0169999999999999},"min":{"1min":0.502,"5min":0.48599999999999999,"15min":0.46100000000000002},"max":{"1min":1.425,"5min":8.5839999999999996,"15min":22.594999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.82099999999999995,"5min":1.2629999999999999,"15min":1.163},"min":{"1min":0.58499999999999996,"5min":0.443,"15min":0.443},"max":{"1min":1.109,"5min":26.748999999999999,"15min":48.198},"last":0.76900000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":1.125,"15min":1.046},"min":{"1min":0.52200000000000002,"5min":0.50600000000000001,"15min":0.47499999999999998},"max":{"1min":1.879,"5min":9.6349999999999998,"15min":35.112000000000002},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.999,"15min":0.97599999999999998},"min":{"1min":0.58599999999999997,"5min":0.53100000000000003,"15min":0.50800000000000001},"max":{"1min":1.77,"5min":5.8849999999999998,"15min":19.780999999999999},"last":0.95199999999999996}]},{"osd":7,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3340000000000001,"15min":1.169},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.97499999999999998,"5min":20.872,"15min":48.148000000000003},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.85899999999999999,"5min":1.246,"15min":0.996},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.52100000000000002},"max":{"1min":1.52,"5min":9.8640000000000008,"15min":9.8640000000000008},"last":0.873}]}]},{"osd":5,"up_from":14,"seq":60129542279,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54352,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317488,"statfs":{"total":96636764160,"available":96581107712,"internally_reserved":0,"allocated":987136,"data_stored":421162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.86699999999999999,"15min":0.80900000000000005},"min":{"1min":0.51800000000000002,"5min":0.29899999999999999,"15min":0.28999999999999998},"max":{"1min":3.8410000000000002,"5min":5.1639999999999997,"15min":21.385000000000002},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.0069999999999999,"15min":0.89600000000000002},"min":{"1min":0.57099999999999995,"5min":0.41599999999999998,"15min":0.34899999999999998},"max":{"1min":3.1080000000000001,"5min":9.359,"15min":9.359},"last":0.71099999999999997}]},{"osd":1,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":1.3120000000000001,"15min":0.89600000000000002},"min":{"1min":0.55400000000000005,"5min":0.435,"15min":0.34899999999999998},"max":{"1min":1.0469999999999999,"5min":21.120000000000001,"15min":21.120000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.2529999999999999,"15min":0.96499999999999997},"min":{"1min":0.628,"5min":0.38600000000000001,"15min":0.315},"max":{"1min":6.1310000000000002,"5min":15.843,"15min":21.5},"last":0.52900000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.98699999999999999,"15min":0.89800000000000002},"min":{"1min":0.51400000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":3.9220000000000002,"5min":6.2750000000000004,"15min":8.7750000000000004},"last":0.54500000000000004},{"interface":"front","average":{"1min":1.135,"5min":1.3640000000000001,"15min":0.998},"min":{"1min":0.50800000000000001,"5min":0.42499999999999999,"15min":0.40400000000000003},"max":{"1min":8.3529999999999998,"5min":21.201000000000001,"15min":21.201000000000001},"last":0.69499999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":0.98599999999999999,"15min":0.92200000000000004},"min":{"1min":0.60499999999999998,"5min":0.37,"15min":0.37},"max":{"1min":7.7199999999999998,"5min":7.7199999999999998,"15min":21.457000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.90300000000000002,"5min":1.034,"15min":0.97899999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.45400000000000001},"max":{"1min":3.8239999999999998,"5min":6.8650000000000002,"15min":8.7300000000000004},"last":0.47699999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":1.0740000000000001,"15min":0.88800000000000001},"min":{"1min":0.51800000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.99199999999999999,"5min":10.962,"15min":10.962},"last":0.626},{"interface":"front","average":{"1min":0.69299999999999995,"5min":1.4410000000000001,"15min":1.034},"min":{"1min":0.48399999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0189999999999999,"5min":21.724,"15min":26.146999999999998},"last":0.51300000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.83899999999999997},"min":{"1min":0.44500000000000001,"5min":0.36099999999999999,"15min":0.32000000000000001},"max":{"1min":0.95599999999999996,"5min":5.2160000000000002,"15min":21.535},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.79000000000000004,"5min":1.012,"15min":1.0129999999999999},"min":{"1min":0.52400000000000002,"5min":0.379,"15min":0.36299999999999999},"max":{"1min":1.03,"5min":5.7539999999999996,"15min":30.488},"last":0.67800000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":2.0070000000000001,"15min":1.216},"min":{"1min":0.47699999999999998,"5min":0.42599999999999999,"15min":0.39700000000000002},"max":{"1min":1.008,"5min":33.499000000000002,"15min":33.499000000000002},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.81000000000000005,"5min":1.502,"15min":1.0880000000000001},"min":{"1min":0.54300000000000004,"5min":0.42899999999999999,"15min":0.36499999999999999},"max":{"1min":1.0609999999999999,"5min":25.437999999999999,"15min":25.437999999999999},"last":0.64300000000000002}]}]},{"osd":0,"up_from":14,"seq":60129542279,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77092,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294748,"statfs":{"total":96636764160,"available":96557821952,"internally_reserved":0,"allocated":1593344,"data_stored":1017889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.90400000000000003,"15min":0.63},"min":{"1min":0.245,"5min":0.245,"15min":0.23699999999999999},"max":{"1min":0.78600000000000003,"5min":19.843,"15min":19.843},"last":0.37},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.94699999999999995,"15min":0.68000000000000005},"min":{"1min":0.29999999999999999,"5min":0.29199999999999998,"15min":0.27300000000000002},"max":{"1min":0.746,"5min":10.154,"15min":10.154},"last":0.39200000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":1.2729999999999999,"15min":0.80400000000000005},"min":{"1min":0.39500000000000002,"5min":0.34899999999999998,"15min":0.30399999999999999},"max":{"1min":0.84099999999999997,"5min":25.905000000000001,"15min":25.905000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.82699999999999996,"15min":0.61199999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.26700000000000002},"max":{"1min":0.80000000000000004,"5min":9.3829999999999991,"15min":9.3829999999999991},"last":0.51900000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":1.026,"15min":0.74199999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.80400000000000005,"5min":9.1780000000000008,"15min":9.1780000000000008},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":1.016,"15min":0.76700000000000002},"min":{"1min":0.41699999999999998,"5min":0.40000000000000002,"15min":0.36899999999999999},"max":{"1min":0.89900000000000002,"5min":10.210000000000001,"15min":10.210000000000001},"last":0.44500000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":1.1319999999999999,"15min":0.98799999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.38},"max":{"1min":1.1459999999999999,"5min":10.691000000000001,"15min":22.814},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.80400000000000005,"5min":1.395,"15min":1.042},"min":{"1min":0.50600000000000001,"5min":0.49099999999999999,"15min":0.41199999999999998},"max":{"1min":1.046,"5min":23.341999999999999,"15min":23.341999999999999},"last":0.66000000000000003}]},{"osd":5,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":1.141,"15min":0.96899999999999997},"min":{"1min":0.55700000000000005,"5min":0.46899999999999997,"15min":0.441},"max":{"1min":1.046,"5min":10.737,"15min":12.428000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.83799999999999997,"5min":1.962,"15min":1.27},"min":{"1min":0.56999999999999995,"5min":0.498,"15min":0.46300000000000002},"max":{"1min":1.3049999999999999,"5min":23.021000000000001,"15min":23.021000000000001},"last":0.68999999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":1.696,"15min":1.218},"min":{"1min":0.52700000000000002,"5min":0.46700000000000003,"15min":0.46200000000000002},"max":{"1min":1.0229999999999999,"5min":26.036000000000001,"15min":31.059000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.88800000000000001,"5min":1.083,"15min":1.0920000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.49099999999999999},"max":{"1min":1.931,"5min":6.9690000000000003,"15min":40.731000000000002},"last":0.70299999999999996}]},{"osd":7,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.8400000000000001,"15min":1.208},"min":{"1min":0.58999999999999997,"5min":0.54900000000000004,"15min":0.42499999999999999},"max":{"1min":4.1609999999999996,"5min":24.492999999999999,"15min":24.492999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.83499999999999996,"5min":1.242,"15min":1.1140000000000001},"min":{"1min":0.64500000000000002,"5min":0.58399999999999996,"15min":0.46600000000000003},"max":{"1min":1.008,"5min":9.2789999999999999,"15min":23.524000000000001},"last":0.72899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:15:49.291 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:15:49.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.472+0000 7f3118600640 1 Processor -- start 2024-01-15T18:15:49.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.472+0000 7f3118600640 1 -- start start 2024-01-15T18:15:49.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.472+0000 7f3118600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f31100cf9b0 0x7f31100b10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:49.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.472+0000 7f3118600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31100b38b0 con 0x7f311015e7d0 2024-01-15T18:15:49.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.472+0000 7f3118600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31100cd550 con 0x7f31100cedc0 2024-01-15T18:15:49.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.472+0000 7f3118600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31100cd270 con 0x7f31100cf9b0 2024-01-15T18:15:49.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.472+0000 7f3115200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f31100cf9b0 0x7f31100b10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:49.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.472+0000 7f3115c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f311015e7d0 0x7f31100cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46150/0 (socket says 172.21.15.27:46150) 2024-01-15T18:15:49.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.472+0000 7f3115c00640 1 -- 172.21.15.27:0/1877620896 learned_addr learned my addr 172.21.15.27:0/1877620896 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:49.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.472+0000 7f3115200640 1 -- 172.21.15.27:0/1877620896 >> v1:172.21.15.93:6789/0 conn(0x7f31100cedc0 legacy=0x7f31100b1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:49.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.472+0000 7f3115200640 1 -- 172.21.15.27:0/1877620896 >> v1:172.21.15.27:6789/0 conn(0x7f311015e7d0 legacy=0x7f31100cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:49.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.472+0000 7f310fe00640 1 -- 172.21.15.27:0/1877620896 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1536318873 0 0) 0x7f31100b38b0 con 0x7f311015e7d0 2024-01-15T18:15:49.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.472+0000 7f3115200640 1 -- 172.21.15.27:0/1877620896 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31100b35d0 con 0x7f31100cf9b0 2024-01-15T18:15:49.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.476+0000 7f3115200640 1 --2- 172.21.15.27:0/1877620896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31100cf9b0 0x7f31100b10c0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f30f80097c0 tx=0x7f31100b2520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acf5f669dc34373f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:49.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.476+0000 7f310fe00640 1 -- 172.21.15.27:0/1877620896 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f80ab600 con 0x7f31100cf9b0 2024-01-15T18:15:49.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.476+0000 7f310fe00640 1 -- 172.21.15.27:0/1877620896 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f80a7050 con 0x7f31100cf9b0 2024-01-15T18:15:49.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.476+0000 7f310fe00640 1 -- 172.21.15.27:0/1877620896 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f80ab980 con 0x7f31100cf9b0 2024-01-15T18:15:49.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.476+0000 7f3118600640 1 -- 172.21.15.27:0/1877620896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31100cf9b0 msgr2=0x7f31100b10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:49.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.476+0000 7f3118600640 1 --2- 172.21.15.27:0/1877620896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31100cf9b0 0x7f31100b10c0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f30f80097c0 tx=0x7f31100b2520 comp rx=0 tx=0).stop 2024-01-15T18:15:49.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.476+0000 7f3118600640 1 -- 172.21.15.27:0/1877620896 shutdown_connections 2024-01-15T18:15:49.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.476+0000 7f3118600640 1 --2- 172.21.15.27:0/1877620896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31100cf9b0 0x7f31100b10c0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:49.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.476+0000 7f3118600640 1 -- 172.21.15.27:0/1877620896 >> 172.21.15.27:0/1877620896 conn(0x7f31100b2a10 msgr2=0x7f31100b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:49.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.476+0000 7f3118600640 1 -- 172.21.15.27:0/1877620896 shutdown_connections 2024-01-15T18:15:49.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.476+0000 7f3118600640 1 -- 172.21.15.27:0/1877620896 wait complete. 2024-01-15T18:15:49.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.480+0000 7f3118600640 1 Processor -- start 2024-01-15T18:15:49.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.480+0000 7f3118600640 1 -- start start 2024-01-15T18:15:49.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.480+0000 7f3118600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f31100cedc0 0x7f31100af870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:49.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.480+0000 7f3118600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31100cf9b0 0x7f3110154510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:49.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.480+0000 7f3118600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f311015e7d0 0x7f31101f74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:49.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.480+0000 7f3118600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311015bb10 con 0x7f31100cf9b0 2024-01-15T18:15:49.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.480+0000 7f3118600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3110157b40 con 0x7f311015e7d0 2024-01-15T18:15:49.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.480+0000 7f3118600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3110157810 con 0x7f31100cedc0 2024-01-15T18:15:49.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.480+0000 7f3115200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31100cf9b0 0x7f3110154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:49.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.480+0000 7f3115200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31100cf9b0 0x7f3110154510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56612/0 (socket says 172.21.15.27:56612) 2024-01-15T18:15:49.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.480+0000 7f3115200640 1 -- 172.21.15.27:0/3229240888 learned_addr learned my addr 172.21.15.27:0/3229240888 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:49.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.480+0000 7f3116600640 1 --2- 172.21.15.27:0/3229240888 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f311015e7d0 0x7f31101f74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:49.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.480+0000 7f3115c00640 1 --2- 172.21.15.27:0/3229240888 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f31100cedc0 0x7f31100af870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:49.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.480+0000 7f3115200640 1 -- 172.21.15.27:0/3229240888 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f31100cedc0 msgr2=0x7f31100af870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:49.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.480+0000 7f3115200640 1 --2- 172.21.15.27:0/3229240888 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f31100cedc0 0x7f31100af870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:49.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.480+0000 7f3115200640 1 -- 172.21.15.27:0/3229240888 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f311015e7d0 msgr2=0x7f31101f74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:49.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.480+0000 7f3115200640 1 --2- 172.21.15.27:0/3229240888 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f311015e7d0 0x7f31101f74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:49.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.480+0000 7f3115200640 1 -- 172.21.15.27:0/3229240888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f8009420 con 0x7f31100cf9b0 2024-01-15T18:15:49.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.480+0000 7f3115200640 1 --2- 172.21.15.27:0/3229240888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31100cf9b0 0x7f3110154510 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f30f80097c0 tx=0x7f30f8019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:49.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.480+0000 7f310d600640 1 -- 172.21.15.27:0/3229240888 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f8004030 con 0x7f31100cf9b0 2024-01-15T18:15:49.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.480+0000 7f310d600640 1 -- 172.21.15.27:0/3229240888 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f8004190 con 0x7f31100cf9b0 2024-01-15T18:15:49.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.480+0000 7f310d600640 1 -- 172.21.15.27:0/3229240888 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f8004500 con 0x7f31100cf9b0 2024-01-15T18:15:49.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.480+0000 7f3118600640 1 -- 172.21.15.27:0/3229240888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f311015b550 con 0x7f31100cf9b0 2024-01-15T18:15:49.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.484+0000 7f3118600640 1 -- 172.21.15.27:0/3229240888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31101552e0 con 0x7f31100cf9b0 2024-01-15T18:15:49.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.484+0000 7f3118600640 1 -- 172.21.15.27:0/3229240888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3110157670 con 0x7f31100cf9b0 2024-01-15T18:15:49.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.484+0000 7f310d600640 1 -- 172.21.15.27:0/3229240888 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7f30f80b2040 con 0x7f31100cf9b0 2024-01-15T18:15:49.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.484+0000 7f310d600640 1 --2- 172.21.15.27:0/3229240888 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f30e80674c0 0x7f30e8069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:49.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.484+0000 7f310d600640 1 -- 172.21.15.27:0/3229240888 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f30f811d070 con 0x7f31100cf9b0 2024-01-15T18:15:49.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.484+0000 7f3115c00640 1 --2- 172.21.15.27:0/3229240888 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f30e80674c0 0x7f30e8069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:49.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.488+0000 7f3115c00640 1 --2- 172.21.15.27:0/3229240888 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f30e80674c0 0x7f30e8069980 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f3100000f70 tx=0x7f3100007240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:49.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.488+0000 7f310d600640 1 -- 172.21.15.27:0/3229240888 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3110157670 con 0x7f31100cf9b0 2024-01-15T18:15:49.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.648+0000 7f3118600640 1 -- 172.21.15.27:0/3229240888 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3110157670 con 0x7f30e80674c0 2024-01-15T18:15:49.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.660+0000 7f310d600640 1 -- 172.21.15.27:0/3229240888 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287360 (secure 0 0 0) 0x7f3110157670 con 0x7f30e80674c0 2024-01-15T18:15:49.660 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:15:49.662 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:15:49.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.664+0000 7f3118600640 1 -- 172.21.15.27:0/3229240888 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f30e80674c0 msgr2=0x7f30e8069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:49.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.664+0000 7f3118600640 1 --2- 172.21.15.27:0/3229240888 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f30e80674c0 0x7f30e8069980 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f3100000f70 tx=0x7f3100007240 comp rx=0 tx=0).stop 2024-01-15T18:15:49.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.664+0000 7f3118600640 1 -- 172.21.15.27:0/3229240888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31100cf9b0 msgr2=0x7f3110154510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:49.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.664+0000 7f3118600640 1 --2- 172.21.15.27:0/3229240888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31100cf9b0 0x7f3110154510 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f30f80097c0 tx=0x7f30f8019190 comp rx=0 tx=0).stop 2024-01-15T18:15:49.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.664+0000 7f3118600640 1 -- 172.21.15.27:0/3229240888 shutdown_connections 2024-01-15T18:15:49.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.664+0000 7f3118600640 1 --2- 172.21.15.27:0/3229240888 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f30e80674c0 0x7f30e8069980 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:49.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.664+0000 7f3118600640 1 --2- 172.21.15.27:0/3229240888 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f311015e7d0 0x7f31101f74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:49.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.664+0000 7f3118600640 1 --2- 172.21.15.27:0/3229240888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31100cf9b0 0x7f3110154510 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:49.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.664+0000 7f3118600640 1 --2- 172.21.15.27:0/3229240888 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f31100cedc0 0x7f31100af870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:49.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.664+0000 7f3118600640 1 -- 172.21.15.27:0/3229240888 >> 172.21.15.27:0/3229240888 conn(0x7f31100b2a10 msgr2=0x7f31100cc660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:49.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.664+0000 7f3118600640 1 -- 172.21.15.27:0/3229240888 shutdown_connections 2024-01-15T18:15:49.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.664+0000 7f3118600640 1 -- 172.21.15.27:0/3229240888 wait complete. 2024-01-15T18:15:49.682 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-01-15T18:15:48.331070+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591166,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":176,"num_write_kb":2427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":79,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":563496,"kb_used_data":8996,"kb_used_omap":12,"kb_used_meta":554355,"kb_avail":754411224,"statfs":{"total":773094113280,"available":772517093376,"internally_reserved":0,"allocated":9211904,"data_stored":4609039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":567660080},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":798,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.192938"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.197673+0000","last_change":"2024-01-15T18:15:45.194410+0000","last_active":"2024-01-15T18:15:45.197673+0000","last_peered":"2024-01-15T18:15:45.197673+0000","last_clean":"2024-01-15T18:15:45.197673+0000","last_became_active":"2024-01-15T18:15:45.193977+0000","last_became_peered":"2024-01-15T18:15:45.193977+0000","last_unstale":"2024-01-15T18:15:45.197673+0000","last_undegraded":"2024-01-15T18:15:45.197673+0000","last_fullsized":"2024-01-15T18:15:45.197673+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:59:33.238710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.193135+0000","last_change":"2024-01-15T18:15:45.185773+0000","last_active":"2024-01-15T18:15:45.193135+0000","last_peered":"2024-01-15T18:15:45.193135+0000","last_clean":"2024-01-15T18:15:45.193135+0000","last_became_active":"2024-01-15T18:15:45.185459+0000","last_became_peered":"2024-01-15T18:15:45.185459+0000","last_unstale":"2024-01-15T18:15:45.193135+0000","last_undegraded":"2024-01-15T18:15:45.193135+0000","last_fullsized":"2024-01-15T18:15:45.193135+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:33:22.524087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196740+0000","last_change":"2024-01-15T18:15:45.190593+0000","last_active":"2024-01-15T18:15:45.196740+0000","last_peered":"2024-01-15T18:15:45.196740+0000","last_clean":"2024-01-15T18:15:45.196740+0000","last_became_active":"2024-01-15T18:15:45.190292+0000","last_became_peered":"2024-01-15T18:15:45.190292+0000","last_unstale":"2024-01-15T18:15:45.196740+0000","last_undegraded":"2024-01-15T18:15:45.196740+0000","last_fullsized":"2024-01-15T18:15:45.196740+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:41:44.153310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.197017+0000","last_change":"2024-01-15T18:15:45.192728+0000","last_active":"2024-01-15T18:15:45.197017+0000","last_peered":"2024-01-15T18:15:45.197017+0000","last_clean":"2024-01-15T18:15:45.197017+0000","last_became_active":"2024-01-15T18:15:45.192270+0000","last_became_peered":"2024-01-15T18:15:45.192270+0000","last_unstale":"2024-01-15T18:15:45.197017+0000","last_undegraded":"2024-01-15T18:15:45.197017+0000","last_fullsized":"2024-01-15T18:15:45.197017+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:10:12.400165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191097+0000","last_change":"2024-01-15T18:15:45.181959+0000","last_active":"2024-01-15T18:15:45.191097+0000","last_peered":"2024-01-15T18:15:45.191097+0000","last_clean":"2024-01-15T18:15:45.191097+0000","last_became_active":"2024-01-15T18:15:45.181526+0000","last_became_peered":"2024-01-15T18:15:45.181526+0000","last_unstale":"2024-01-15T18:15:45.191097+0000","last_undegraded":"2024-01-15T18:15:45.191097+0000","last_fullsized":"2024-01-15T18:15:45.191097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:10:10.202003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.145723+0000","last_change":"2024-01-15T18:15:44.145723+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.145723+0000","last_undegraded":"2024-01-15T18:15:44.145723+0000","last_fullsized":"2024-01-15T18:15:44.145723+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.188332+0000","last_change":"2024-01-15T18:15:45.177382+0000","last_active":"2024-01-15T18:15:45.188332+0000","last_peered":"2024-01-15T18:15:45.188332+0000","last_clean":"2024-01-15T18:15:45.188332+0000","last_became_active":"2024-01-15T18:15:45.173825+0000","last_became_peered":"2024-01-15T18:15:45.173825+0000","last_unstale":"2024-01-15T18:15:45.188332+0000","last_undegraded":"2024-01-15T18:15:45.188332+0000","last_fullsized":"2024-01-15T18:15:45.188332+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:03:55.153676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.144794+0000","last_change":"2024-01-15T18:15:44.144794+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.144794+0000","last_undegraded":"2024-01-15T18:15:44.144794+0000","last_fullsized":"2024-01-15T18:15:44.144794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.145532+0000","last_change":"2024-01-15T18:15:44.145532+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.145532+0000","last_undegraded":"2024-01-15T18:15:44.145532+0000","last_fullsized":"2024-01-15T18:15:44.145532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.195302+0000","last_change":"2024-01-15T18:15:45.190970+0000","last_active":"2024-01-15T18:15:45.195302+0000","last_peered":"2024-01-15T18:15:45.195302+0000","last_clean":"2024-01-15T18:15:45.195302+0000","last_became_active":"2024-01-15T18:15:45.187235+0000","last_became_peered":"2024-01-15T18:15:45.187235+0000","last_unstale":"2024-01-15T18:15:45.195302+0000","last_undegraded":"2024-01-15T18:15:45.195302+0000","last_fullsized":"2024-01-15T18:15:45.195302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:35:49.681365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.186724+0000","last_change":"2024-01-15T18:15:45.179561+0000","last_active":"2024-01-15T18:15:45.186724+0000","last_peered":"2024-01-15T18:15:45.186724+0000","last_clean":"2024-01-15T18:15:45.186724+0000","last_became_active":"2024-01-15T18:15:45.179305+0000","last_became_peered":"2024-01-15T18:15:45.179305+0000","last_unstale":"2024-01-15T18:15:45.186724+0000","last_undegraded":"2024-01-15T18:15:45.186724+0000","last_fullsized":"2024-01-15T18:15:45.186724+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:56:26.621247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191872+0000","last_change":"2024-01-15T18:15:45.189460+0000","last_active":"2024-01-15T18:15:45.191872+0000","last_peered":"2024-01-15T18:15:45.191872+0000","last_clean":"2024-01-15T18:15:45.191872+0000","last_became_active":"2024-01-15T18:15:45.188105+0000","last_became_peered":"2024-01-15T18:15:45.188105+0000","last_unstale":"2024-01-15T18:15:45.191872+0000","last_undegraded":"2024-01-15T18:15:45.191872+0000","last_fullsized":"2024-01-15T18:15:45.191872+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:45:42.971410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196682+0000","last_change":"2024-01-15T18:15:45.190326+0000","last_active":"2024-01-15T18:15:45.196682+0000","last_peered":"2024-01-15T18:15:45.196682+0000","last_clean":"2024-01-15T18:15:45.196682+0000","last_became_active":"2024-01-15T18:15:45.188312+0000","last_became_peered":"2024-01-15T18:15:45.188312+0000","last_unstale":"2024-01-15T18:15:45.196682+0000","last_undegraded":"2024-01-15T18:15:45.196682+0000","last_fullsized":"2024-01-15T18:15:45.196682+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:28:05.260374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191135+0000","last_change":"2024-01-15T18:15:45.180022+0000","last_active":"2024-01-15T18:15:45.191135+0000","last_peered":"2024-01-15T18:15:45.191135+0000","last_clean":"2024-01-15T18:15:45.191135+0000","last_became_active":"2024-01-15T18:15:45.179537+0000","last_became_peered":"2024-01-15T18:15:45.179537+0000","last_unstale":"2024-01-15T18:15:45.191135+0000","last_undegraded":"2024-01-15T18:15:45.191135+0000","last_fullsized":"2024-01-15T18:15:45.191135+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:55.575874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196300+0000","last_change":"2024-01-15T18:15:45.185924+0000","last_active":"2024-01-15T18:15:45.196300+0000","last_peered":"2024-01-15T18:15:45.196300+0000","last_clean":"2024-01-15T18:15:45.196300+0000","last_became_active":"2024-01-15T18:15:45.185669+0000","last_became_peered":"2024-01-15T18:15:45.185669+0000","last_unstale":"2024-01-15T18:15:45.196300+0000","last_undegraded":"2024-01-15T18:15:45.196300+0000","last_fullsized":"2024-01-15T18:15:45.196300+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:08:31.073685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.184890+0000","last_change":"2024-01-15T18:15:45.176262+0000","last_active":"2024-01-15T18:15:45.184890+0000","last_peered":"2024-01-15T18:15:45.184890+0000","last_clean":"2024-01-15T18:15:45.184890+0000","last_became_active":"2024-01-15T18:15:45.175976+0000","last_became_peered":"2024-01-15T18:15:45.175976+0000","last_unstale":"2024-01-15T18:15:45.184890+0000","last_undegraded":"2024-01-15T18:15:45.184890+0000","last_fullsized":"2024-01-15T18:15:45.184890+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:16.351733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191295+0000","last_change":"2024-01-15T18:15:45.184436+0000","last_active":"2024-01-15T18:15:45.191295+0000","last_peered":"2024-01-15T18:15:45.191295+0000","last_clean":"2024-01-15T18:15:45.191295+0000","last_became_active":"2024-01-15T18:15:45.181072+0000","last_became_peered":"2024-01-15T18:15:45.181072+0000","last_unstale":"2024-01-15T18:15:45.191295+0000","last_undegraded":"2024-01-15T18:15:45.191295+0000","last_fullsized":"2024-01-15T18:15:45.191295+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:04:09.741683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191667+0000","last_change":"2024-01-15T18:15:45.184794+0000","last_active":"2024-01-15T18:15:45.191667+0000","last_peered":"2024-01-15T18:15:45.191667+0000","last_clean":"2024-01-15T18:15:45.191667+0000","last_became_active":"2024-01-15T18:15:45.184528+0000","last_became_peered":"2024-01-15T18:15:45.184528+0000","last_unstale":"2024-01-15T18:15:45.191667+0000","last_undegraded":"2024-01-15T18:15:45.191667+0000","last_fullsized":"2024-01-15T18:15:45.191667+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:33.583003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.158290+0000","last_change":"2024-01-15T18:15:44.158290+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.158290+0000","last_undegraded":"2024-01-15T18:15:44.158290+0000","last_fullsized":"2024-01-15T18:15:44.158290+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.203247+0000","last_change":"2024-01-15T18:15:45.202436+0000","last_active":"2024-01-15T18:15:45.203247+0000","last_peered":"2024-01-15T18:15:45.203247+0000","last_clean":"2024-01-15T18:15:45.203247+0000","last_became_active":"2024-01-15T18:15:45.202075+0000","last_became_peered":"2024-01-15T18:15:45.202075+0000","last_unstale":"2024-01-15T18:15:45.203247+0000","last_undegraded":"2024-01-15T18:15:45.203247+0000","last_fullsized":"2024-01-15T18:15:45.203247+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:29:34.404126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198796+0000","last_change":"2024-01-15T18:15:45.197310+0000","last_active":"2024-01-15T18:15:45.198796+0000","last_peered":"2024-01-15T18:15:45.198796+0000","last_clean":"2024-01-15T18:15:45.198796+0000","last_became_active":"2024-01-15T18:15:45.196364+0000","last_became_peered":"2024-01-15T18:15:45.196364+0000","last_unstale":"2024-01-15T18:15:45.198796+0000","last_undegraded":"2024-01-15T18:15:45.198796+0000","last_fullsized":"2024-01-15T18:15:45.198796+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:28:16.596449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191310+0000","last_change":"2024-01-15T18:15:45.185683+0000","last_active":"2024-01-15T18:15:45.191310+0000","last_peered":"2024-01-15T18:15:45.191310+0000","last_clean":"2024-01-15T18:15:45.191310+0000","last_became_active":"2024-01-15T18:15:45.185303+0000","last_became_peered":"2024-01-15T18:15:45.185303+0000","last_unstale":"2024-01-15T18:15:45.191310+0000","last_undegraded":"2024-01-15T18:15:45.191310+0000","last_fullsized":"2024-01-15T18:15:45.191310+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:12:38.030672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198110+0000","last_change":"2024-01-15T18:15:45.195966+0000","last_active":"2024-01-15T18:15:45.198110+0000","last_peered":"2024-01-15T18:15:45.198110+0000","last_clean":"2024-01-15T18:15:45.198110+0000","last_became_active":"2024-01-15T18:15:45.194717+0000","last_became_peered":"2024-01-15T18:15:45.194717+0000","last_unstale":"2024-01-15T18:15:45.198110+0000","last_undegraded":"2024-01-15T18:15:45.198110+0000","last_fullsized":"2024-01-15T18:15:45.198110+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:50:32.308793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.190685+0000","last_change":"2024-01-15T18:15:45.179745+0000","last_active":"2024-01-15T18:15:45.190685+0000","last_peered":"2024-01-15T18:15:45.190685+0000","last_clean":"2024-01-15T18:15:45.190685+0000","last_became_active":"2024-01-15T18:15:45.179329+0000","last_became_peered":"2024-01-15T18:15:45.179329+0000","last_unstale":"2024-01-15T18:15:45.190685+0000","last_undegraded":"2024-01-15T18:15:45.190685+0000","last_fullsized":"2024-01-15T18:15:45.190685+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:01:10.870535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198301+0000","last_change":"2024-01-15T18:15:45.192970+0000","last_active":"2024-01-15T18:15:45.198301+0000","last_peered":"2024-01-15T18:15:45.198301+0000","last_clean":"2024-01-15T18:15:45.198301+0000","last_became_active":"2024-01-15T18:15:45.192474+0000","last_became_peered":"2024-01-15T18:15:45.192474+0000","last_unstale":"2024-01-15T18:15:45.198301+0000","last_undegraded":"2024-01-15T18:15:45.198301+0000","last_fullsized":"2024-01-15T18:15:45.198301+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:38.217871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.183678+0000","last_change":"2024-01-15T18:15:45.180211+0000","last_active":"2024-01-15T18:15:45.183678+0000","last_peered":"2024-01-15T18:15:45.183678+0000","last_clean":"2024-01-15T18:15:45.183678+0000","last_became_active":"2024-01-15T18:15:45.179566+0000","last_became_peered":"2024-01-15T18:15:45.179566+0000","last_unstale":"2024-01-15T18:15:45.183678+0000","last_undegraded":"2024-01-15T18:15:45.183678+0000","last_fullsized":"2024-01-15T18:15:45.183678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:40:13.829197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.192821+0000","last_change":"2024-01-15T18:15:45.189295+0000","last_active":"2024-01-15T18:15:45.192821+0000","last_peered":"2024-01-15T18:15:45.192821+0000","last_clean":"2024-01-15T18:15:45.192821+0000","last_became_active":"2024-01-15T18:15:45.187101+0000","last_became_peered":"2024-01-15T18:15:45.187101+0000","last_unstale":"2024-01-15T18:15:45.192821+0000","last_undegraded":"2024-01-15T18:15:45.192821+0000","last_fullsized":"2024-01-15T18:15:45.192821+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:01:13.745010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.195405+0000","last_change":"2024-01-15T18:15:45.192353+0000","last_active":"2024-01-15T18:15:45.195405+0000","last_peered":"2024-01-15T18:15:45.195405+0000","last_clean":"2024-01-15T18:15:45.195405+0000","last_became_active":"2024-01-15T18:15:45.191563+0000","last_became_peered":"2024-01-15T18:15:45.191563+0000","last_unstale":"2024-01-15T18:15:45.195405+0000","last_undegraded":"2024-01-15T18:15:45.195405+0000","last_fullsized":"2024-01-15T18:15:45.195405+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:57:43.892130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.197122+0000","last_change":"2024-01-15T18:15:45.190607+0000","last_active":"2024-01-15T18:15:45.197122+0000","last_peered":"2024-01-15T18:15:45.197122+0000","last_clean":"2024-01-15T18:15:45.197122+0000","last_became_active":"2024-01-15T18:15:45.189633+0000","last_became_peered":"2024-01-15T18:15:45.189633+0000","last_unstale":"2024-01-15T18:15:45.197122+0000","last_undegraded":"2024-01-15T18:15:45.197122+0000","last_fullsized":"2024-01-15T18:15:45.197122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:35:49.681365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.187329+0000","last_change":"2024-01-15T18:15:45.180268+0000","last_active":"2024-01-15T18:15:45.187329+0000","last_peered":"2024-01-15T18:15:45.187329+0000","last_clean":"2024-01-15T18:15:45.187329+0000","last_became_active":"2024-01-15T18:15:45.179617+0000","last_became_peered":"2024-01-15T18:15:45.179617+0000","last_unstale":"2024-01-15T18:15:45.187329+0000","last_undegraded":"2024-01-15T18:15:45.187329+0000","last_fullsized":"2024-01-15T18:15:45.187329+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:18:22.468719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.186449+0000","last_change":"2024-01-15T18:15:45.177309+0000","last_active":"2024-01-15T18:15:45.186449+0000","last_peered":"2024-01-15T18:15:45.186449+0000","last_clean":"2024-01-15T18:15:45.186449+0000","last_became_active":"2024-01-15T18:15:45.177022+0000","last_became_peered":"2024-01-15T18:15:45.177022+0000","last_unstale":"2024-01-15T18:15:45.186449+0000","last_undegraded":"2024-01-15T18:15:45.186449+0000","last_fullsized":"2024-01-15T18:15:45.186449+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:53.495194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.187438+0000","last_change":"2024-01-15T18:15:45.182964+0000","last_active":"2024-01-15T18:15:45.187438+0000","last_peered":"2024-01-15T18:15:45.187438+0000","last_clean":"2024-01-15T18:15:45.187438+0000","last_became_active":"2024-01-15T18:15:45.182690+0000","last_became_peered":"2024-01-15T18:15:45.182690+0000","last_unstale":"2024-01-15T18:15:45.187438+0000","last_undegraded":"2024-01-15T18:15:45.187438+0000","last_fullsized":"2024-01-15T18:15:45.187438+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:31.079315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.192979+0000","last_change":"2024-01-15T18:15:45.189818+0000","last_active":"2024-01-15T18:15:45.192979+0000","last_peered":"2024-01-15T18:15:45.192979+0000","last_clean":"2024-01-15T18:15:45.192979+0000","last_became_active":"2024-01-15T18:15:45.189534+0000","last_became_peered":"2024-01-15T18:15:45.189534+0000","last_unstale":"2024-01-15T18:15:45.192979+0000","last_undegraded":"2024-01-15T18:15:45.192979+0000","last_fullsized":"2024-01-15T18:15:45.192979+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:13:12.060989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"83'4","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.197594+0000","last_change":"2024-01-15T18:15:44.163810+0000","last_active":"2024-01-15T18:15:45.197594+0000","last_peered":"2024-01-15T18:15:45.197594+0000","last_clean":"2024-01-15T18:15:45.197594+0000","last_became_active":"2024-01-15T18:15:44.163128+0000","last_became_peered":"2024-01-15T18:15:44.163128+0000","last_unstale":"2024-01-15T18:15:45.197594+0000","last_undegraded":"2024-01-15T18:15:45.197594+0000","last_fullsized":"2024-01-15T18:15:45.197594+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:22:07.304644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.202605+0000","last_change":"2024-01-15T18:15:45.201966+0000","last_active":"2024-01-15T18:15:45.202605+0000","last_peered":"2024-01-15T18:15:45.202605+0000","last_clean":"2024-01-15T18:15:45.202605+0000","last_became_active":"2024-01-15T18:15:45.201568+0000","last_became_peered":"2024-01-15T18:15:45.201568+0000","last_unstale":"2024-01-15T18:15:45.202605+0000","last_undegraded":"2024-01-15T18:15:45.202605+0000","last_fullsized":"2024-01-15T18:15:45.202605+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:16.351733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.187621+0000","last_change":"2024-01-15T18:15:44.156714+0000","last_active":"2024-01-15T18:15:45.187621+0000","last_peered":"2024-01-15T18:15:45.187621+0000","last_clean":"2024-01-15T18:15:45.187621+0000","last_became_active":"2024-01-15T18:15:44.156419+0000","last_became_peered":"2024-01-15T18:15:44.156419+0000","last_unstale":"2024-01-15T18:15:45.187621+0000","last_undegraded":"2024-01-15T18:15:45.187621+0000","last_fullsized":"2024-01-15T18:15:45.187621+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:14:41.627743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.197147+0000","last_change":"2024-01-15T18:15:45.192955+0000","last_active":"2024-01-15T18:15:45.197147+0000","last_peered":"2024-01-15T18:15:45.197147+0000","last_clean":"2024-01-15T18:15:45.197147+0000","last_became_active":"2024-01-15T18:15:45.192565+0000","last_became_peered":"2024-01-15T18:15:45.192565+0000","last_unstale":"2024-01-15T18:15:45.197147+0000","last_undegraded":"2024-01-15T18:15:45.197147+0000","last_fullsized":"2024-01-15T18:15:45.197147+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:07:58.498146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.171318+0000","last_change":"2024-01-15T18:15:44.167059+0000","last_active":"2024-01-15T18:15:44.171318+0000","last_peered":"2024-01-15T18:15:44.171318+0000","last_clean":"2024-01-15T18:15:44.171318+0000","last_became_active":"2024-01-15T18:15:44.166829+0000","last_became_peered":"2024-01-15T18:15:44.166829+0000","last_unstale":"2024-01-15T18:15:44.171318+0000","last_undegraded":"2024-01-15T18:15:44.171318+0000","last_fullsized":"2024-01-15T18:15:44.171318+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:39:01.209507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.194382+0000","last_change":"2024-01-15T18:15:45.186505+0000","last_active":"2024-01-15T18:15:45.194382+0000","last_peered":"2024-01-15T18:15:45.194382+0000","last_clean":"2024-01-15T18:15:45.194382+0000","last_became_active":"2024-01-15T18:15:45.186181+0000","last_became_peered":"2024-01-15T18:15:45.186181+0000","last_unstale":"2024-01-15T18:15:45.194382+0000","last_undegraded":"2024-01-15T18:15:45.194382+0000","last_fullsized":"2024-01-15T18:15:45.194382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:03:29.579452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.170558+0000","last_change":"2024-01-15T18:15:44.166697+0000","last_active":"2024-01-15T18:15:44.170558+0000","last_peered":"2024-01-15T18:15:44.170558+0000","last_clean":"2024-01-15T18:15:44.170558+0000","last_became_active":"2024-01-15T18:15:44.166389+0000","last_became_peered":"2024-01-15T18:15:44.166389+0000","last_unstale":"2024-01-15T18:15:44.170558+0000","last_undegraded":"2024-01-15T18:15:44.170558+0000","last_fullsized":"2024-01-15T18:15:44.170558+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:19:36.045488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.187351+0000","last_change":"2024-01-15T18:15:45.180445+0000","last_active":"2024-01-15T18:15:45.187351+0000","last_peered":"2024-01-15T18:15:45.187351+0000","last_clean":"2024-01-15T18:15:45.187351+0000","last_became_active":"2024-01-15T18:15:45.179946+0000","last_became_peered":"2024-01-15T18:15:45.179946+0000","last_unstale":"2024-01-15T18:15:45.187351+0000","last_undegraded":"2024-01-15T18:15:45.187351+0000","last_fullsized":"2024-01-15T18:15:45.187351+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:50:32.308793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.181787+0000","last_change":"2024-01-15T18:15:44.157992+0000","last_active":"2024-01-15T18:15:45.181787+0000","last_peered":"2024-01-15T18:15:45.181787+0000","last_clean":"2024-01-15T18:15:45.181787+0000","last_became_active":"2024-01-15T18:15:44.157721+0000","last_became_peered":"2024-01-15T18:15:44.157721+0000","last_unstale":"2024-01-15T18:15:45.181787+0000","last_undegraded":"2024-01-15T18:15:45.181787+0000","last_fullsized":"2024-01-15T18:15:45.181787+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:27:47.584018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.185831+0000","last_change":"2024-01-15T18:15:45.178738+0000","last_active":"2024-01-15T18:15:45.185831+0000","last_peered":"2024-01-15T18:15:45.185831+0000","last_clean":"2024-01-15T18:15:45.185831+0000","last_became_active":"2024-01-15T18:15:45.178446+0000","last_became_peered":"2024-01-15T18:15:45.178446+0000","last_unstale":"2024-01-15T18:15:45.185831+0000","last_undegraded":"2024-01-15T18:15:45.185831+0000","last_fullsized":"2024-01-15T18:15:45.185831+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:59:33.238710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.187378+0000","last_change":"2024-01-15T18:15:44.154715+0000","last_active":"2024-01-15T18:15:45.187378+0000","last_peered":"2024-01-15T18:15:45.187378+0000","last_clean":"2024-01-15T18:15:45.187378+0000","last_became_active":"2024-01-15T18:15:44.154433+0000","last_became_peered":"2024-01-15T18:15:44.154433+0000","last_unstale":"2024-01-15T18:15:45.187378+0000","last_undegraded":"2024-01-15T18:15:45.187378+0000","last_fullsized":"2024-01-15T18:15:45.187378+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:42:00.915026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.202740+0000","last_change":"2024-01-15T18:15:45.202049+0000","last_active":"2024-01-15T18:15:45.202740+0000","last_peered":"2024-01-15T18:15:45.202740+0000","last_clean":"2024-01-15T18:15:45.202740+0000","last_became_active":"2024-01-15T18:15:45.201742+0000","last_became_peered":"2024-01-15T18:15:45.201742+0000","last_unstale":"2024-01-15T18:15:45.202740+0000","last_undegraded":"2024-01-15T18:15:45.202740+0000","last_fullsized":"2024-01-15T18:15:45.202740+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:56.553890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.172316+0000","last_change":"2024-01-15T18:15:44.167859+0000","last_active":"2024-01-15T18:15:44.172316+0000","last_peered":"2024-01-15T18:15:44.172316+0000","last_clean":"2024-01-15T18:15:44.172316+0000","last_became_active":"2024-01-15T18:15:44.167561+0000","last_became_peered":"2024-01-15T18:15:44.167561+0000","last_unstale":"2024-01-15T18:15:44.172316+0000","last_undegraded":"2024-01-15T18:15:44.172316+0000","last_fullsized":"2024-01-15T18:15:44.172316+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:17.332973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196319+0000","last_change":"2024-01-15T18:15:45.187853+0000","last_active":"2024-01-15T18:15:45.196319+0000","last_peered":"2024-01-15T18:15:45.196319+0000","last_clean":"2024-01-15T18:15:45.196319+0000","last_became_active":"2024-01-15T18:15:45.187552+0000","last_became_peered":"2024-01-15T18:15:45.187552+0000","last_unstale":"2024-01-15T18:15:45.196319+0000","last_undegraded":"2024-01-15T18:15:45.196319+0000","last_fullsized":"2024-01-15T18:15:45.196319+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:56.553890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.162106+0000","last_change":"2024-01-15T18:15:44.155651+0000","last_active":"2024-01-15T18:15:45.162106+0000","last_peered":"2024-01-15T18:15:45.162106+0000","last_clean":"2024-01-15T18:15:45.162106+0000","last_became_active":"2024-01-15T18:15:44.155306+0000","last_became_peered":"2024-01-15T18:15:44.155306+0000","last_unstale":"2024-01-15T18:15:45.162106+0000","last_undegraded":"2024-01-15T18:15:45.162106+0000","last_fullsized":"2024-01-15T18:15:45.162106+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:06:15.812490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.189876+0000","last_change":"2024-01-15T18:15:45.184002+0000","last_active":"2024-01-15T18:15:45.189876+0000","last_peered":"2024-01-15T18:15:45.189876+0000","last_clean":"2024-01-15T18:15:45.189876+0000","last_became_active":"2024-01-15T18:15:45.183624+0000","last_became_peered":"2024-01-15T18:15:45.183624+0000","last_unstale":"2024-01-15T18:15:45.189876+0000","last_undegraded":"2024-01-15T18:15:45.189876+0000","last_fullsized":"2024-01-15T18:15:45.189876+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:03:29.579452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.186739+0000","last_change":"2024-01-15T18:15:44.160123+0000","last_active":"2024-01-15T18:15:45.186739+0000","last_peered":"2024-01-15T18:15:45.186739+0000","last_clean":"2024-01-15T18:15:45.186739+0000","last_became_active":"2024-01-15T18:15:44.159761+0000","last_became_peered":"2024-01-15T18:15:44.159761+0000","last_unstale":"2024-01-15T18:15:45.186739+0000","last_undegraded":"2024-01-15T18:15:45.186739+0000","last_fullsized":"2024-01-15T18:15:45.186739+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:42:00.915026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.144921+0000","last_change":"2024-01-15T18:15:44.144921+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.144921+0000","last_undegraded":"2024-01-15T18:15:44.144921+0000","last_fullsized":"2024-01-15T18:15:44.144921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.187555+0000","last_change":"2024-01-15T18:15:44.166336+0000","last_active":"2024-01-15T18:15:45.187555+0000","last_peered":"2024-01-15T18:15:45.187555+0000","last_clean":"2024-01-15T18:15:45.187555+0000","last_became_active":"2024-01-15T18:15:44.165403+0000","last_became_peered":"2024-01-15T18:15:44.165403+0000","last_unstale":"2024-01-15T18:15:45.187555+0000","last_undegraded":"2024-01-15T18:15:45.187555+0000","last_fullsized":"2024-01-15T18:15:45.187555+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:42:53.652872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.155839+0000","last_change":"2024-01-15T18:15:44.155839+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.155839+0000","last_undegraded":"2024-01-15T18:15:44.155839+0000","last_fullsized":"2024-01-15T18:15:44.155839+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"83'3","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.197477+0000","last_change":"2024-01-15T18:15:44.160184+0000","last_active":"2024-01-15T18:15:45.197477+0000","last_peered":"2024-01-15T18:15:45.197477+0000","last_clean":"2024-01-15T18:15:45.197477+0000","last_became_active":"2024-01-15T18:15:44.159855+0000","last_became_peered":"2024-01-15T18:15:44.159855+0000","last_unstale":"2024-01-15T18:15:45.197477+0000","last_undegraded":"2024-01-15T18:15:45.197477+0000","last_fullsized":"2024-01-15T18:15:45.197477+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:04:08.736438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.192668+0000","last_change":"2024-01-15T18:15:45.187059+0000","last_active":"2024-01-15T18:15:45.192668+0000","last_peered":"2024-01-15T18:15:45.192668+0000","last_clean":"2024-01-15T18:15:45.192668+0000","last_became_active":"2024-01-15T18:15:45.184830+0000","last_became_peered":"2024-01-15T18:15:45.184830+0000","last_unstale":"2024-01-15T18:15:45.192668+0000","last_undegraded":"2024-01-15T18:15:45.192668+0000","last_fullsized":"2024-01-15T18:15:45.192668+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:58:35.413117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.163096+0000","last_change":"2024-01-15T18:15:44.158973+0000","last_active":"2024-01-15T18:15:44.163096+0000","last_peered":"2024-01-15T18:15:44.163096+0000","last_clean":"2024-01-15T18:15:44.163096+0000","last_became_active":"2024-01-15T18:15:44.158684+0000","last_became_peered":"2024-01-15T18:15:44.158684+0000","last_unstale":"2024-01-15T18:15:44.163096+0000","last_undegraded":"2024-01-15T18:15:44.163096+0000","last_fullsized":"2024-01-15T18:15:44.163096+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:22:27.571591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.153863+0000","last_change":"2024-01-15T18:15:44.153863+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.153863+0000","last_undegraded":"2024-01-15T18:15:44.153863+0000","last_fullsized":"2024-01-15T18:15:44.153863+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.201845+0000","last_change":"2024-01-15T18:15:44.166340+0000","last_active":"2024-01-15T18:15:45.201845+0000","last_peered":"2024-01-15T18:15:45.201845+0000","last_clean":"2024-01-15T18:15:45.201845+0000","last_became_active":"2024-01-15T18:15:44.165947+0000","last_became_peered":"2024-01-15T18:15:44.165947+0000","last_unstale":"2024-01-15T18:15:45.201845+0000","last_undegraded":"2024-01-15T18:15:45.201845+0000","last_fullsized":"2024-01-15T18:15:45.201845+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:12:37.715165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.199984+0000","last_change":"2024-01-15T18:15:45.199248+0000","last_active":"2024-01-15T18:15:45.199984+0000","last_peered":"2024-01-15T18:15:45.199984+0000","last_clean":"2024-01-15T18:15:45.199984+0000","last_became_active":"2024-01-15T18:15:45.198929+0000","last_became_peered":"2024-01-15T18:15:45.198929+0000","last_unstale":"2024-01-15T18:15:45.199984+0000","last_undegraded":"2024-01-15T18:15:45.199984+0000","last_fullsized":"2024-01-15T18:15:45.199984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:26:01.224733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.182090+0000","last_change":"2024-01-15T18:15:44.161901+0000","last_active":"2024-01-15T18:15:45.182090+0000","last_peered":"2024-01-15T18:15:45.182090+0000","last_clean":"2024-01-15T18:15:45.182090+0000","last_became_active":"2024-01-15T18:15:44.159756+0000","last_became_peered":"2024-01-15T18:15:44.159756+0000","last_unstale":"2024-01-15T18:15:45.182090+0000","last_undegraded":"2024-01-15T18:15:45.182090+0000","last_fullsized":"2024-01-15T18:15:45.182090+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:41:43.148064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.180743+0000","last_change":"2024-01-15T18:15:45.174549+0000","last_active":"2024-01-15T18:15:45.180743+0000","last_peered":"2024-01-15T18:15:45.180743+0000","last_clean":"2024-01-15T18:15:45.180743+0000","last_became_active":"2024-01-15T18:15:45.174046+0000","last_became_peered":"2024-01-15T18:15:45.174046+0000","last_unstale":"2024-01-15T18:15:45.180743+0000","last_undegraded":"2024-01-15T18:15:45.180743+0000","last_fullsized":"2024-01-15T18:15:45.180743+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:12:42.277859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.159621+0000","last_change":"2024-01-15T18:15:44.154875+0000","last_active":"2024-01-15T18:15:44.159621+0000","last_peered":"2024-01-15T18:15:44.159621+0000","last_clean":"2024-01-15T18:15:44.159621+0000","last_became_active":"2024-01-15T18:15:44.154585+0000","last_became_peered":"2024-01-15T18:15:44.154585+0000","last_unstale":"2024-01-15T18:15:44.159621+0000","last_undegraded":"2024-01-15T18:15:44.159621+0000","last_fullsized":"2024-01-15T18:15:44.159621+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:39:01.209507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198412+0000","last_change":"2024-01-15T18:15:45.193330+0000","last_active":"2024-01-15T18:15:45.198412+0000","last_peered":"2024-01-15T18:15:45.198412+0000","last_clean":"2024-01-15T18:15:45.198412+0000","last_became_active":"2024-01-15T18:15:45.192354+0000","last_became_peered":"2024-01-15T18:15:45.192354+0000","last_unstale":"2024-01-15T18:15:45.198412+0000","last_undegraded":"2024-01-15T18:15:45.198412+0000","last_fullsized":"2024-01-15T18:15:45.198412+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:12:58.057727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.172772+0000","last_change":"2024-01-15T18:15:44.168821+0000","last_active":"2024-01-15T18:15:44.172772+0000","last_peered":"2024-01-15T18:15:44.172772+0000","last_clean":"2024-01-15T18:15:44.172772+0000","last_became_active":"2024-01-15T18:15:44.168551+0000","last_became_peered":"2024-01-15T18:15:44.168551+0000","last_unstale":"2024-01-15T18:15:44.172772+0000","last_undegraded":"2024-01-15T18:15:44.172772+0000","last_fullsized":"2024-01-15T18:15:44.172772+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:14:11.832883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191678+0000","last_change":"2024-01-15T18:15:45.180405+0000","last_active":"2024-01-15T18:15:45.191678+0000","last_peered":"2024-01-15T18:15:45.191678+0000","last_clean":"2024-01-15T18:15:45.191678+0000","last_became_active":"2024-01-15T18:15:45.179918+0000","last_became_peered":"2024-01-15T18:15:45.179918+0000","last_unstale":"2024-01-15T18:15:45.191678+0000","last_undegraded":"2024-01-15T18:15:45.191678+0000","last_fullsized":"2024-01-15T18:15:45.191678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:01:49.697849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.192261+0000","last_change":"2024-01-15T18:15:44.159443+0000","last_active":"2024-01-15T18:15:45.192261+0000","last_peered":"2024-01-15T18:15:45.192261+0000","last_clean":"2024-01-15T18:15:45.192261+0000","last_became_active":"2024-01-15T18:15:44.159118+0000","last_became_peered":"2024-01-15T18:15:44.159118+0000","last_unstale":"2024-01-15T18:15:45.192261+0000","last_undegraded":"2024-01-15T18:15:45.192261+0000","last_fullsized":"2024-01-15T18:15:45.192261+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:13:11.055744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198173+0000","last_change":"2024-01-15T18:15:45.195167+0000","last_active":"2024-01-15T18:15:45.198173+0000","last_peered":"2024-01-15T18:15:45.198173+0000","last_clean":"2024-01-15T18:15:45.198173+0000","last_became_active":"2024-01-15T18:15:45.194897+0000","last_became_peered":"2024-01-15T18:15:45.194897+0000","last_unstale":"2024-01-15T18:15:45.198173+0000","last_undegraded":"2024-01-15T18:15:45.198173+0000","last_fullsized":"2024-01-15T18:15:45.198173+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:53:25.231875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.172853+0000","last_change":"2024-01-15T18:15:44.169724+0000","last_active":"2024-01-15T18:15:44.172853+0000","last_peered":"2024-01-15T18:15:44.172853+0000","last_clean":"2024-01-15T18:15:44.172853+0000","last_became_active":"2024-01-15T18:15:44.169420+0000","last_became_peered":"2024-01-15T18:15:44.169420+0000","last_unstale":"2024-01-15T18:15:44.172853+0000","last_undegraded":"2024-01-15T18:15:44.172853+0000","last_fullsized":"2024-01-15T18:15:44.172853+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:50:20.346298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.185177+0000","last_change":"2024-01-15T18:15:45.175568+0000","last_active":"2024-01-15T18:15:45.185177+0000","last_peered":"2024-01-15T18:15:45.185177+0000","last_clean":"2024-01-15T18:15:45.185177+0000","last_became_active":"2024-01-15T18:15:45.173399+0000","last_became_peered":"2024-01-15T18:15:45.173399+0000","last_unstale":"2024-01-15T18:15:45.185177+0000","last_undegraded":"2024-01-15T18:15:45.185177+0000","last_fullsized":"2024-01-15T18:15:45.185177+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:42:54.658117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.172758+0000","last_change":"2024-01-15T18:15:44.168665+0000","last_active":"2024-01-15T18:15:44.172758+0000","last_peered":"2024-01-15T18:15:44.172758+0000","last_clean":"2024-01-15T18:15:44.172758+0000","last_became_active":"2024-01-15T18:15:44.168375+0000","last_became_peered":"2024-01-15T18:15:44.168375+0000","last_unstale":"2024-01-15T18:15:44.172758+0000","last_undegraded":"2024-01-15T18:15:44.172758+0000","last_fullsized":"2024-01-15T18:15:44.172758+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:09:06.608452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191116+0000","last_change":"2024-01-15T18:15:45.179812+0000","last_active":"2024-01-15T18:15:45.191116+0000","last_peered":"2024-01-15T18:15:45.191116+0000","last_clean":"2024-01-15T18:15:45.191116+0000","last_became_active":"2024-01-15T18:15:45.179148+0000","last_became_peered":"2024-01-15T18:15:45.179148+0000","last_unstale":"2024-01-15T18:15:45.191116+0000","last_undegraded":"2024-01-15T18:15:45.191116+0000","last_fullsized":"2024-01-15T18:15:45.191116+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:47:52.953626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196980+0000","last_change":"2024-01-15T18:15:44.163128+0000","last_active":"2024-01-15T18:15:45.196980+0000","last_peered":"2024-01-15T18:15:45.196980+0000","last_clean":"2024-01-15T18:15:45.196980+0000","last_became_active":"2024-01-15T18:15:44.162839+0000","last_became_peered":"2024-01-15T18:15:44.162839+0000","last_unstale":"2024-01-15T18:15:45.196980+0000","last_undegraded":"2024-01-15T18:15:45.196980+0000","last_fullsized":"2024-01-15T18:15:45.196980+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:34:15.516676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.146439+0000","last_change":"2024-01-15T18:15:44.146439+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.146439+0000","last_undegraded":"2024-01-15T18:15:44.146439+0000","last_fullsized":"2024-01-15T18:15:44.146439+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.174831+0000","last_change":"2024-01-15T18:15:44.167859+0000","last_active":"2024-01-15T18:15:45.174831+0000","last_peered":"2024-01-15T18:15:45.174831+0000","last_clean":"2024-01-15T18:15:45.174831+0000","last_became_active":"2024-01-15T18:15:44.167592+0000","last_became_peered":"2024-01-15T18:15:44.167592+0000","last_unstale":"2024-01-15T18:15:45.174831+0000","last_undegraded":"2024-01-15T18:15:45.174831+0000","last_fullsized":"2024-01-15T18:15:45.174831+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:02:39.986413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.190664+0000","last_change":"2024-01-15T18:15:45.179572+0000","last_active":"2024-01-15T18:15:45.190664+0000","last_peered":"2024-01-15T18:15:45.190664+0000","last_clean":"2024-01-15T18:15:45.190664+0000","last_became_active":"2024-01-15T18:15:45.178802+0000","last_became_peered":"2024-01-15T18:15:45.178802+0000","last_unstale":"2024-01-15T18:15:45.190664+0000","last_undegraded":"2024-01-15T18:15:45.190664+0000","last_fullsized":"2024-01-15T18:15:45.190664+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:26:43.360415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.173615+0000","last_change":"2024-01-15T18:15:44.168997+0000","last_active":"2024-01-15T18:15:45.173615+0000","last_peered":"2024-01-15T18:15:45.173615+0000","last_clean":"2024-01-15T18:15:45.173615+0000","last_became_active":"2024-01-15T18:15:44.168678+0000","last_became_peered":"2024-01-15T18:15:44.168678+0000","last_unstale":"2024-01-15T18:15:45.173615+0000","last_undegraded":"2024-01-15T18:15:45.173615+0000","last_fullsized":"2024-01-15T18:15:45.173615+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:20:19.634573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.145752+0000","last_change":"2024-01-15T18:15:44.145752+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.145752+0000","last_undegraded":"2024-01-15T18:15:44.145752+0000","last_fullsized":"2024-01-15T18:15:44.145752+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.178682+0000","last_change":"2024-01-15T18:15:44.162077+0000","last_active":"2024-01-15T18:15:45.178682+0000","last_peered":"2024-01-15T18:15:45.178682+0000","last_clean":"2024-01-15T18:15:45.178682+0000","last_became_active":"2024-01-15T18:15:44.161794+0000","last_became_peered":"2024-01-15T18:15:44.161794+0000","last_unstale":"2024-01-15T18:15:45.178682+0000","last_undegraded":"2024-01-15T18:15:45.178682+0000","last_fullsized":"2024-01-15T18:15:45.178682+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:05:24.652003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.185502+0000","last_change":"2024-01-15T18:15:45.175862+0000","last_active":"2024-01-15T18:15:45.185502+0000","last_peered":"2024-01-15T18:15:45.185502+0000","last_clean":"2024-01-15T18:15:45.185502+0000","last_became_active":"2024-01-15T18:15:45.175569+0000","last_became_peered":"2024-01-15T18:15:45.175569+0000","last_unstale":"2024-01-15T18:15:45.185502+0000","last_undegraded":"2024-01-15T18:15:45.185502+0000","last_fullsized":"2024-01-15T18:15:45.185502+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:22:08.309890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.193674+0000","last_change":"2024-01-15T18:15:44.168092+0000","last_active":"2024-01-15T18:15:45.193674+0000","last_peered":"2024-01-15T18:15:45.193674+0000","last_clean":"2024-01-15T18:15:45.193674+0000","last_became_active":"2024-01-15T18:15:44.167828+0000","last_became_peered":"2024-01-15T18:15:44.167828+0000","last_unstale":"2024-01-15T18:15:45.193674+0000","last_undegraded":"2024-01-15T18:15:45.193674+0000","last_fullsized":"2024-01-15T18:15:45.193674+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:27:26.285133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198128+0000","last_change":"2024-01-15T18:15:45.194246+0000","last_active":"2024-01-15T18:15:45.198128+0000","last_peered":"2024-01-15T18:15:45.198128+0000","last_clean":"2024-01-15T18:15:45.198128+0000","last_became_active":"2024-01-15T18:15:45.193864+0000","last_became_peered":"2024-01-15T18:15:45.193864+0000","last_unstale":"2024-01-15T18:15:45.198128+0000","last_undegraded":"2024-01-15T18:15:45.198128+0000","last_fullsized":"2024-01-15T18:15:45.198128+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:05:25.657249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.162597+0000","last_change":"2024-01-15T18:15:44.166561+0000","last_active":"2024-01-15T18:15:45.162597+0000","last_peered":"2024-01-15T18:15:45.162597+0000","last_clean":"2024-01-15T18:15:45.162597+0000","last_became_active":"2024-01-15T18:15:44.166280+0000","last_became_peered":"2024-01-15T18:15:44.166280+0000","last_unstale":"2024-01-15T18:15:45.162597+0000","last_undegraded":"2024-01-15T18:15:45.162597+0000","last_fullsized":"2024-01-15T18:15:45.162597+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:30.074069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198142+0000","last_change":"2024-01-15T18:15:45.196412+0000","last_active":"2024-01-15T18:15:45.198142+0000","last_peered":"2024-01-15T18:15:45.198142+0000","last_clean":"2024-01-15T18:15:45.198142+0000","last_became_active":"2024-01-15T18:15:45.196022+0000","last_became_peered":"2024-01-15T18:15:45.196022+0000","last_unstale":"2024-01-15T18:15:45.198142+0000","last_undegraded":"2024-01-15T18:15:45.198142+0000","last_fullsized":"2024-01-15T18:15:45.198142+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:09:32.875279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.184853+0000","last_change":"2024-01-15T18:15:44.163120+0000","last_active":"2024-01-15T18:15:45.184853+0000","last_peered":"2024-01-15T18:15:45.184853+0000","last_clean":"2024-01-15T18:15:45.184853+0000","last_became_active":"2024-01-15T18:15:44.162805+0000","last_became_peered":"2024-01-15T18:15:44.162805+0000","last_unstale":"2024-01-15T18:15:45.184853+0000","last_undegraded":"2024-01-15T18:15:45.184853+0000","last_fullsized":"2024-01-15T18:15:45.184853+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:49.344761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191100+0000","last_change":"2024-01-15T18:15:45.185283+0000","last_active":"2024-01-15T18:15:45.191100+0000","last_peered":"2024-01-15T18:15:45.191100+0000","last_clean":"2024-01-15T18:15:45.191100+0000","last_became_active":"2024-01-15T18:15:45.185001+0000","last_became_peered":"2024-01-15T18:15:45.185001+0000","last_unstale":"2024-01-15T18:15:45.191100+0000","last_undegraded":"2024-01-15T18:15:45.191100+0000","last_fullsized":"2024-01-15T18:15:45.191100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:41:05.797910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.183836+0000","last_change":"2024-01-15T18:15:44.163840+0000","last_active":"2024-01-15T18:15:45.183836+0000","last_peered":"2024-01-15T18:15:45.183836+0000","last_clean":"2024-01-15T18:15:45.183836+0000","last_became_active":"2024-01-15T18:15:44.163441+0000","last_became_peered":"2024-01-15T18:15:44.163441+0000","last_unstale":"2024-01-15T18:15:45.183836+0000","last_undegraded":"2024-01-15T18:15:45.183836+0000","last_fullsized":"2024-01-15T18:15:45.183836+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:41:04.792664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.145185+0000","last_change":"2024-01-15T18:15:44.145185+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.145185+0000","last_undegraded":"2024-01-15T18:15:44.145185+0000","last_fullsized":"2024-01-15T18:15:44.145185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.172834+0000","last_change":"2024-01-15T18:15:44.169437+0000","last_active":"2024-01-15T18:15:44.172834+0000","last_peered":"2024-01-15T18:15:44.172834+0000","last_clean":"2024-01-15T18:15:44.172834+0000","last_became_active":"2024-01-15T18:15:44.169170+0000","last_became_peered":"2024-01-15T18:15:44.169170+0000","last_unstale":"2024-01-15T18:15:44.172834+0000","last_undegraded":"2024-01-15T18:15:44.172834+0000","last_fullsized":"2024-01-15T18:15:44.172834+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:50:41.854134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.156826+0000","last_change":"2024-01-15T18:15:44.156826+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.156826+0000","last_undegraded":"2024-01-15T18:15:44.156826+0000","last_fullsized":"2024-01-15T18:15:44.156826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.164910+0000","last_change":"2024-01-15T18:15:44.154568+0000","last_active":"2024-01-15T18:15:45.164910+0000","last_peered":"2024-01-15T18:15:45.164910+0000","last_clean":"2024-01-15T18:15:45.164910+0000","last_became_active":"2024-01-15T18:15:44.154277+0000","last_became_peered":"2024-01-15T18:15:44.154277+0000","last_unstale":"2024-01-15T18:15:45.164910+0000","last_undegraded":"2024-01-15T18:15:45.164910+0000","last_fullsized":"2024-01-15T18:15:45.164910+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:39:47.546718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191500+0000","last_change":"2024-01-15T18:15:45.182784+0000","last_active":"2024-01-15T18:15:45.191500+0000","last_peered":"2024-01-15T18:15:45.191500+0000","last_clean":"2024-01-15T18:15:45.191500+0000","last_became_active":"2024-01-15T18:15:45.179168+0000","last_became_peered":"2024-01-15T18:15:45.179168+0000","last_unstale":"2024-01-15T18:15:45.191500+0000","last_undegraded":"2024-01-15T18:15:45.191500+0000","last_fullsized":"2024-01-15T18:15:45.191500+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:34:16.521921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.183666+0000","last_change":"2024-01-15T18:15:44.160891+0000","last_active":"2024-01-15T18:15:45.183666+0000","last_peered":"2024-01-15T18:15:45.183666+0000","last_clean":"2024-01-15T18:15:45.183666+0000","last_became_active":"2024-01-15T18:15:44.160625+0000","last_became_peered":"2024-01-15T18:15:44.160625+0000","last_unstale":"2024-01-15T18:15:45.183666+0000","last_undegraded":"2024-01-15T18:15:45.183666+0000","last_fullsized":"2024-01-15T18:15:45.183666+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:52.489949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196072+0000","last_change":"2024-01-15T18:15:45.192344+0000","last_active":"2024-01-15T18:15:45.196072+0000","last_peered":"2024-01-15T18:15:45.196072+0000","last_clean":"2024-01-15T18:15:45.196072+0000","last_became_active":"2024-01-15T18:15:45.191845+0000","last_became_peered":"2024-01-15T18:15:45.191845+0000","last_unstale":"2024-01-15T18:15:45.196072+0000","last_undegraded":"2024-01-15T18:15:45.196072+0000","last_fullsized":"2024-01-15T18:15:45.196072+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:01:31.532362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.163378+0000","last_change":"2024-01-15T18:15:44.156428+0000","last_active":"2024-01-15T18:15:45.163378+0000","last_peered":"2024-01-15T18:15:45.163378+0000","last_clean":"2024-01-15T18:15:45.163378+0000","last_became_active":"2024-01-15T18:15:44.156132+0000","last_became_peered":"2024-01-15T18:15:44.156132+0000","last_unstale":"2024-01-15T18:15:45.163378+0000","last_undegraded":"2024-01-15T18:15:45.163378+0000","last_fullsized":"2024-01-15T18:15:45.163378+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:49.344761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.186454+0000","last_change":"2024-01-15T18:15:45.178846+0000","last_active":"2024-01-15T18:15:45.186454+0000","last_peered":"2024-01-15T18:15:45.186454+0000","last_clean":"2024-01-15T18:15:45.186454+0000","last_became_active":"2024-01-15T18:15:45.178552+0000","last_became_peered":"2024-01-15T18:15:45.178552+0000","last_unstale":"2024-01-15T18:15:45.186454+0000","last_undegraded":"2024-01-15T18:15:45.186454+0000","last_fullsized":"2024-01-15T18:15:45.186454+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:45:31.736245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.183987+0000","last_change":"2024-01-15T18:15:44.170831+0000","last_active":"2024-01-15T18:15:45.183987+0000","last_peered":"2024-01-15T18:15:45.183987+0000","last_clean":"2024-01-15T18:15:45.183987+0000","last_became_active":"2024-01-15T18:15:44.170489+0000","last_became_peered":"2024-01-15T18:15:44.170489+0000","last_unstale":"2024-01-15T18:15:45.183987+0000","last_undegraded":"2024-01-15T18:15:45.183987+0000","last_fullsized":"2024-01-15T18:15:45.183987+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:54:03.053923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":218,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.162031+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:15:44.162031+0000","last_peered":"2024-01-15T18:15:44.162031+0000","last_clean":"2024-01-15T18:15:44.162031+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:15:44.162031+0000","last_undegraded":"2024-01-15T18:15:44.162031+0000","last_fullsized":"2024-01-15T18:15:44.162031+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":798,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509575,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77092,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294748,"statfs":{"total":96636764160,"available":96557821952,"internally_reserved":0,"allocated":1593344,"data_stored":1017889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77330890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":1.171,"15min":0.874},"min":{"1min":0.52400000000000002,"5min":0.38100000000000001,"15min":0.36199999999999999},"max":{"1min":1.0089999999999999,"5min":9.6989999999999998,"15min":10.284000000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.68799999999999994,"5min":1.1399999999999999,"15min":0.90000000000000002},"min":{"1min":0.48599999999999999,"5min":0.39100000000000001,"15min":0.378},"max":{"1min":0.83099999999999996,"5min":16.968,"15min":16.968},"last":6.6189999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":1.3129999999999999,"15min":0.91400000000000003},"min":{"1min":0.52600000000000002,"5min":0.36099999999999999,"15min":0.33800000000000002},"max":{"1min":0.90100000000000002,"5min":15.791,"15min":15.791},"last":4.944},{"interface":"front","average":{"1min":0.71099999999999997,"5min":1.109,"15min":0.90000000000000002},"min":{"1min":0.51500000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.85799999999999998,"5min":10.842000000000001,"15min":10.842000000000001},"last":4.6699999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":1.2609999999999999,"15min":0.92500000000000004},"min":{"1min":0.502,"5min":0.40400000000000003,"15min":0.378},"max":{"1min":0.98299999999999998,"5min":22.318000000000001,"15min":22.318000000000001},"last":4.4669999999999996},{"interface":"front","average":{"1min":0.76200000000000001,"5min":1.159,"15min":0.89400000000000002},"min":{"1min":0.58399999999999996,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.93700000000000006,"5min":9.7539999999999996,"15min":9.7539999999999996},"last":4.7030000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":1.387,"15min":0.98899999999999999},"min":{"1min":0.59799999999999998,"5min":0.47099999999999997,"15min":0.38100000000000001},"max":{"1min":0.93300000000000005,"5min":18.152999999999999,"15min":18.152999999999999},"last":1.456},{"interface":"front","average":{"1min":0.81599999999999995,"5min":1.274,"15min":0.98599999999999999},"min":{"1min":0.622,"5min":0.433,"15min":0.43099999999999999},"max":{"1min":0.96599999999999997,"5min":9.7959999999999994,"15min":10.34},"last":2.6909999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":1.2150000000000001,"15min":0.92700000000000005},"min":{"1min":0.44,"5min":0.314,"15min":0.314},"max":{"1min":0.85199999999999998,"5min":20.387,"15min":20.387},"last":4.4480000000000004},{"interface":"front","average":{"1min":0.623,"5min":1.353,"15min":0.98399999999999999},"min":{"1min":0.47099999999999997,"5min":0.38800000000000001,"15min":0.36099999999999999},"max":{"1min":1.032,"5min":17.215,"15min":17.215},"last":3.6309999999999998}]},{"osd":5,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":1.2410000000000001,"15min":0.94499999999999995},"min":{"1min":0.46400000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.95299999999999996,"5min":9.8450000000000006,"15min":19.959},"last":4.8639999999999999},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1.278,"15min":0.97699999999999998},"min":{"1min":0.439,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83999999999999997,"5min":19.905000000000001,"15min":19.905000000000001},"last":4.3959999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":1.4470000000000001,"15min":1.1499999999999999},"min":{"1min":0.59499999999999997,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97099999999999997,"5min":20.492000000000001,"15min":22.478999999999999},"last":4.3330000000000002},{"interface":"front","average":{"1min":0.749,"5min":1.272,"15min":1.0600000000000001},"min":{"1min":0.52200000000000002,"5min":0.44,"15min":0.38300000000000001},"max":{"1min":0.998,"5min":17.132000000000001,"15min":21.509},"last":3.573}]}]},{"osd":6,"up_from":15,"seq":64424509575,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58524,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313316,"statfs":{"total":96636764160,"available":96576835584,"internally_reserved":0,"allocated":999424,"data_stored":427521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":1.3009999999999999,"15min":0.95699999999999996},"min":{"1min":0.55700000000000005,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.96099999999999997,"5min":32.587000000000003,"15min":32.587000000000003},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.75,"5min":1.0660000000000001,"15min":0.86199999999999999},"min":{"1min":0.49199999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.97799999999999998,"5min":16.193000000000001,"15min":16.193000000000001},"last":0.69399999999999995}]},{"osd":1,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.92400000000000004,"15min":0.98699999999999999},"min":{"1min":0.439,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.90700000000000003,"5min":7.7649999999999997,"15min":53.451999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.72499999999999998,"5min":1.2090000000000001,"15min":0.92600000000000005},"min":{"1min":0.53400000000000003,"5min":0.437,"15min":0.42899999999999999},"max":{"1min":0.86299999999999999,"5min":32.512,"15min":32.512},"last":0.51700000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":1.24,"15min":0.96799999999999997},"min":{"1min":0.58399999999999996,"5min":0.41699999999999998,"15min":0.38900000000000001},"max":{"1min":0.92200000000000004,"5min":32.472999999999999,"15min":32.472999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.754,"5min":1.0649999999999999,"15min":0.88500000000000001},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.38500000000000001},"max":{"1min":1.02,"5min":16.602,"15min":16.602},"last":0.65300000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":1.1699999999999999,"15min":0.93600000000000005},"min":{"1min":0.56399999999999995,"5min":0.46000000000000002,"15min":0.41399999999999998},"max":{"1min":0.92100000000000004,"5min":16.388000000000002,"15min":16.388000000000002},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.0680000000000001,"15min":1.075},"min":{"1min":0.54400000000000004,"5min":0.432,"15min":0.41599999999999998},"max":{"1min":0.95999999999999996,"5min":9.3719999999999999,"15min":53.398000000000003},"last":0.67200000000000004}]},{"osd":4,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":1.149,"15min":0.89500000000000002},"min":{"1min":0.435,"5min":0.38100000000000001,"15min":0.34999999999999998},"max":{"1min":0.79700000000000004,"5min":32.549999999999997,"15min":32.549999999999997},"last":0.63},{"interface":"front","average":{"1min":0.59499999999999997,"5min":1.0409999999999999,"15min":0.84899999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.34499999999999997},"max":{"1min":0.96499999999999997,"5min":11.852,"15min":11.852},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":1.0720000000000001,"15min":1.034},"min":{"1min":0.42699999999999999,"5min":0.35799999999999998,"15min":0.30399999999999999},"max":{"1min":0.77300000000000002,"5min":13.087999999999999,"15min":53.229999999999997},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.68700000000000006,"5min":1.375,"15min":1.0349999999999999},"min":{"1min":0.42599999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.0009999999999999,"5min":32.404000000000003,"15min":32.404000000000003},"last":0.71899999999999997}]},{"osd":7,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":1.294,"15min":1.1200000000000001},"min":{"1min":0.52900000000000003,"5min":0.436,"15min":0.40400000000000003},"max":{"1min":0.94099999999999995,"5min":23.991,"15min":45.070999999999998},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":1.3260000000000001,"15min":1.2170000000000001},"min":{"1min":0.51200000000000001,"5min":0.40999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88200000000000001,"5min":40.972999999999999,"15min":53.338000000000001},"last":0.753}]}]},{"osd":4,"up_from":15,"seq":64424509575,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58528,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313312,"statfs":{"total":96636764160,"available":96576831488,"internally_reserved":0,"allocated":999424,"data_stored":427521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":1.111,"15min":0.88500000000000001},"min":{"1min":0.46000000000000002,"5min":0.437,"15min":0.32900000000000001},"max":{"1min":0.96199999999999997,"5min":13.492000000000001,"15min":23.318000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.92100000000000004,"15min":0.80200000000000005},"min":{"1min":0.41999999999999998,"5min":0.378,"15min":0.32600000000000001},"max":{"1min":0.875,"5min":14.723000000000001,"15min":14.723000000000001},"last":0.71099999999999997}]},{"osd":1,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":1.0049999999999999,"15min":0.78800000000000003},"min":{"1min":0.50800000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.84299999999999997,"5min":9.5700000000000003,"15min":9.5700000000000003},"last":0.63},{"interface":"front","average":{"1min":0.71999999999999997,"5min":1.155,"15min":0.91400000000000003},"min":{"1min":0.47399999999999998,"5min":0.40200000000000002,"15min":0.36899999999999999},"max":{"1min":0.84999999999999998,"5min":13.57,"15min":22.408999999999999},"last":0.751}]},{"osd":2,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":1.038,"15min":0.79400000000000004},"min":{"1min":0.53000000000000003,"5min":0.41599999999999998,"15min":0.34399999999999997},"max":{"1min":0.84199999999999997,"5min":9.7089999999999996,"15min":9.7089999999999996},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.73499999999999999,"5min":1.2949999999999999,"15min":1.014},"min":{"1min":0.439,"5min":0.439,"15min":0.39100000000000001},"max":{"1min":0.92600000000000005,"5min":14.118,"15min":22.338999999999999},"last":0.69999999999999996}]},{"osd":3,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":1.016,"15min":0.873},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.41199999999999998},"max":{"1min":0.83199999999999996,"5min":14.763999999999999,"15min":14.763999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.77300000000000002,"5min":1.1479999999999999,"15min":0.90500000000000003},"min":{"1min":0.622,"5min":0.48099999999999998,"15min":0.45500000000000002},"max":{"1min":0.89400000000000002,"5min":9.6289999999999996,"15min":9.6289999999999996},"last":0.72999999999999998}]},{"osd":5,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":1.1299999999999999,"15min":0.96299999999999997},"min":{"1min":0.39000000000000001,"5min":0.34799999999999998,"15min":0.33200000000000002},"max":{"1min":0.875,"5min":15.787000000000001,"15min":31.617999999999999},"last":0.501},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.94699999999999995,"15min":0.86699999999999999},"min":{"1min":0.44,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.92300000000000004,"5min":16.388000000000002,"15min":17.145},"last":0.59899999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.98399999999999999,"15min":0.83099999999999996},"min":{"1min":0.42799999999999999,"5min":0.37,"15min":0.35499999999999998},"max":{"1min":0.94599999999999995,"5min":11.124000000000001,"15min":11.124000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.71699999999999997,"5min":1.3580000000000001,"15min":1.226},"min":{"1min":0.53700000000000003,"5min":0.434,"15min":0.40200000000000002},"max":{"1min":1.018,"5min":32.308,"15min":32.308},"last":0.55500000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":1.002,"15min":1.0109999999999999},"min":{"1min":0.49399999999999999,"5min":0.41099999999999998,"15min":0.34799999999999998},"max":{"1min":0.89900000000000002,"5min":14.653,"15min":39.256},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.746,"5min":1.2290000000000001,"15min":1.0129999999999999},"min":{"1min":0.57799999999999996,"5min":0.46500000000000002,"15min":0.42199999999999999},"max":{"1min":0.98599999999999999,"5min":16.277000000000001,"15min":26.573},"last":0.57399999999999995}]}]},{"osd":3,"up_from":15,"seq":64424509574,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80676,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94291164,"statfs":{"total":96636764160,"available":96554151936,"internally_reserved":0,"allocated":999424,"data_stored":427521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":81590727},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.59199999999999997,"15min":0.49099999999999999},"min":{"1min":0.29299999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.751,"5min":5.8849999999999998,"15min":5.8849999999999998},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.68999999999999995,"15min":0.61599999999999999},"min":{"1min":0.38600000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.77500000000000002,"5min":4.7990000000000004,"15min":17.756},"last":0.29799999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.73099999999999998,"15min":0.59299999999999997},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.621,"5min":12.307,"15min":12.307},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.69399999999999995,"15min":0.59299999999999997},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.32200000000000001},"max":{"1min":0.83799999999999997,"5min":6.0640000000000001,"15min":6.0640000000000001},"last":0.38600000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.83099999999999996,"15min":0.69599999999999995},"min":{"1min":0.46800000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":3.633,"5min":12.468999999999999,"15min":12.468999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.85399999999999998,"15min":0.68500000000000005},"min":{"1min":0.48599999999999999,"5min":0.41599999999999998,"15min":0.34599999999999997},"max":{"1min":1.0129999999999999,"5min":10.837999999999999,"15min":10.837999999999999},"last":0.47199999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":1.292,"15min":1.028},"min":{"1min":0.59499999999999997,"5min":0.375,"15min":0.375},"max":{"1min":1.014,"5min":13.863,"15min":17.853999999999999},"last":0.874},{"interface":"front","average":{"1min":0.82899999999999996,"5min":1.2809999999999999,"15min":0.96899999999999997},"min":{"1min":0.63,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.0469999999999999,"5min":13.31,"15min":13.31},"last":0.72099999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":1.369,"15min":0.97599999999999998},"min":{"1min":0.66500000000000004,"5min":0.52100000000000002,"15min":0.51000000000000001},"max":{"1min":0.94399999999999995,"5min":17.687999999999999,"15min":17.687999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.206,"15min":0.97399999999999998},"min":{"1min":0.51700000000000002,"5min":0.438,"15min":0.438},"max":{"1min":3.2160000000000002,"5min":20.515999999999998,"15min":20.515999999999998},"last":0.82099999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":1.1499999999999999,"15min":0.98999999999999999},"min":{"1min":0.55600000000000005,"5min":0.432,"15min":0.432},"max":{"1min":1.1060000000000001,"5min":13.250999999999999,"15min":23.100999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.82099999999999995,"5min":1.2150000000000001,"15min":0.93799999999999994},"min":{"1min":0.60799999999999998,"5min":0.5,"15min":0.49399999999999999},"max":{"1min":1.0169999999999999,"5min":12.76,"15min":12.76},"last":0.93400000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.7889999999999999,"15min":1.1719999999999999},"min":{"1min":0.625,"5min":0.48599999999999999,"15min":0.48199999999999998},"max":{"1min":0.99099999999999999,"5min":24.321999999999999,"15min":24.321999999999999},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.81999999999999995,"5min":1.329,"15min":1.028},"min":{"1min":0.58399999999999996,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0569999999999999,"5min":13.385,"15min":13.385},"last":0.78100000000000003}]}]},{"osd":2,"up_from":15,"seq":64424509574,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76532,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295308,"statfs":{"total":96636764160,"available":96558395392,"internally_reserved":0,"allocated":1015808,"data_stored":434962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.51900000000000002,"15min":0.51200000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26500000000000001},"max":{"1min":0.66500000000000004,"5min":1.857,"15min":5.1849999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54200000000000004,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.252,"15min":0.252},"max":{"1min":0.89300000000000002,"5min":2.3759999999999999,"15min":7.516},"last":0.55900000000000005}]},{"osd":1,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.70599999999999996,"15min":0.61599999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.875,"5min":10.907,"15min":10.907},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.68500000000000005,"15min":0.58799999999999997},"min":{"1min":0.38100000000000001,"5min":0.28100000000000003,"15min":0.27000000000000002},"max":{"1min":0.73799999999999999,"5min":7.2119999999999997,"15min":8.9190000000000005},"last":0.53500000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.60599999999999998,"15min":0.626},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.72999999999999998,"5min":2.7290000000000001,"15min":12.185},"last":0.621},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.747,"15min":0.66100000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.34000000000000002},"max":{"1min":0.96699999999999997,"5min":7.6760000000000002,"15min":7.6760000000000002},"last":0.67900000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.89500000000000002,"15min":0.85799999999999998},"min":{"1min":0.55700000000000005,"5min":0.47499999999999998,"15min":0.45800000000000002},"max":{"1min":1.1870000000000001,"5min":4.3150000000000004,"15min":8.6609999999999996},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.755,"5min":0.878,"15min":0.90900000000000003},"min":{"1min":0.5,"5min":0.45800000000000002,"15min":0.42499999999999999},"max":{"1min":1.075,"5min":8.3190000000000008,"15min":19.146999999999998},"last":0.84199999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.80400000000000005,"15min":0.90200000000000002},"min":{"1min":0.52400000000000002,"5min":0.442,"15min":0.44},"max":{"1min":1.1599999999999999,"5min":2.7789999999999999,"15min":31.710000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.996,"15min":1},"min":{"1min":0.54400000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.155,"5min":10.460000000000001,"15min":22.539000000000001},"last":0.77300000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.99399999999999999,"15min":0.89700000000000002},"min":{"1min":0.501,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.1220000000000001,"5min":8.3580000000000005,"15min":9.4540000000000006},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.83799999999999997,"15min":0.90500000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0489999999999999,"5min":2.669,"15min":18.614999999999998},"last":0.88}]},{"osd":7,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.125,"15min":0.95399999999999996},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.47999999999999998},"max":{"1min":1.157,"5min":11.606,"15min":11.606},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.92200000000000004,"15min":0.98699999999999999},"min":{"1min":0.60399999999999998,"5min":0.51700000000000002,"15min":0.502},"max":{"1min":1.1180000000000001,"5min":3.6850000000000001,"15min":17.013000000000002},"last":0.86199999999999999}]}]},{"osd":1,"up_from":15,"seq":64424509576,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80700,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94291140,"statfs":{"total":96636764160,"available":96554127360,"internally_reserved":0,"allocated":1024000,"data_stored":434574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81590725},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.70699999999999996,"15min":0.66300000000000003},"min":{"1min":0.26300000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.745,"5min":17.396999999999998,"15min":19.577000000000002},"last":0.623},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.74099999999999999,"15min":0.76000000000000001},"min":{"1min":0.33900000000000002,"5min":0.29199999999999998,"15min":0.251},"max":{"1min":0.83999999999999997,"5min":12.378,"15min":48.277999999999999},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.61399999999999999,"15min":0.71399999999999997},"min":{"1min":0.33600000000000002,"5min":0.309,"15min":0.30299999999999999},"max":{"1min":0.82799999999999996,"5min":2.8610000000000002,"15min":40.698},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.65900000000000003,"15min":0.90100000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.317},"max":{"1min":0.89500000000000002,"5min":3.0449999999999999,"15min":48.316000000000003},"last":0.441}]},{"osd":3,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.71799999999999997,"15min":0.84499999999999997},"min":{"1min":0.434,"5min":0.39800000000000002,"15min":0.33700000000000002},"max":{"1min":1.2509999999999999,"5min":3.0899999999999999,"15min":48.234999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.83799999999999997,"15min":0.68999999999999995},"min":{"1min":0.315,"5min":0.315,"15min":0.311},"max":{"1min":4.0780000000000003,"5min":9.9719999999999995,"15min":9.9719999999999995},"last":0.46000000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":1.0980000000000001,"15min":0.88700000000000001},"min":{"1min":0.49299999999999999,"5min":0.48999999999999999,"15min":0.44500000000000001},"max":{"1min":1.446,"5min":9.9329999999999998,"15min":9.9329999999999998},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.80800000000000005,"5min":1.026,"15min":0.93600000000000005},"min":{"1min":0.47499999999999998,"5min":0.45800000000000002,"15min":0.42499999999999999},"max":{"1min":1.163,"5min":9.8610000000000007,"15min":19.829000000000001},"last":0.88300000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.95399999999999996,"15min":1.0169999999999999},"min":{"1min":0.502,"5min":0.48599999999999999,"15min":0.46100000000000002},"max":{"1min":1.425,"5min":8.5839999999999996,"15min":22.594999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.82099999999999995,"5min":1.2629999999999999,"15min":1.163},"min":{"1min":0.58499999999999996,"5min":0.443,"15min":0.443},"max":{"1min":1.109,"5min":26.748999999999999,"15min":48.198},"last":0.76900000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":1.125,"15min":1.046},"min":{"1min":0.52200000000000002,"5min":0.50600000000000001,"15min":0.47499999999999998},"max":{"1min":1.879,"5min":9.6349999999999998,"15min":35.112000000000002},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.999,"15min":0.97599999999999998},"min":{"1min":0.58599999999999997,"5min":0.53100000000000003,"15min":0.50800000000000001},"max":{"1min":1.77,"5min":5.8849999999999998,"15min":19.780999999999999},"last":0.95199999999999996}]},{"osd":7,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3340000000000001,"15min":1.169},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.97499999999999998,"5min":20.872,"15min":48.148000000000003},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.85899999999999999,"5min":1.246,"15min":0.996},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.52100000000000002},"max":{"1min":1.52,"5min":9.8640000000000008,"15min":9.8640000000000008},"last":0.873}]}]},{"osd":5,"up_from":14,"seq":60129542279,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54352,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317488,"statfs":{"total":96636764160,"available":96581107712,"internally_reserved":0,"allocated":987136,"data_stored":421162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.86699999999999999,"15min":0.80900000000000005},"min":{"1min":0.51800000000000002,"5min":0.29899999999999999,"15min":0.28999999999999998},"max":{"1min":3.8410000000000002,"5min":5.1639999999999997,"15min":21.385000000000002},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.0069999999999999,"15min":0.89600000000000002},"min":{"1min":0.57099999999999995,"5min":0.41599999999999998,"15min":0.34899999999999998},"max":{"1min":3.1080000000000001,"5min":9.359,"15min":9.359},"last":0.71099999999999997}]},{"osd":1,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":1.3120000000000001,"15min":0.89600000000000002},"min":{"1min":0.55400000000000005,"5min":0.435,"15min":0.34899999999999998},"max":{"1min":1.0469999999999999,"5min":21.120000000000001,"15min":21.120000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.2529999999999999,"15min":0.96499999999999997},"min":{"1min":0.628,"5min":0.38600000000000001,"15min":0.315},"max":{"1min":6.1310000000000002,"5min":15.843,"15min":21.5},"last":0.52900000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.98699999999999999,"15min":0.89800000000000002},"min":{"1min":0.51400000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":3.9220000000000002,"5min":6.2750000000000004,"15min":8.7750000000000004},"last":0.54500000000000004},{"interface":"front","average":{"1min":1.135,"5min":1.3640000000000001,"15min":0.998},"min":{"1min":0.50800000000000001,"5min":0.42499999999999999,"15min":0.40400000000000003},"max":{"1min":8.3529999999999998,"5min":21.201000000000001,"15min":21.201000000000001},"last":0.69499999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":0.98599999999999999,"15min":0.92200000000000004},"min":{"1min":0.60499999999999998,"5min":0.37,"15min":0.37},"max":{"1min":7.7199999999999998,"5min":7.7199999999999998,"15min":21.457000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.90300000000000002,"5min":1.034,"15min":0.97899999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.45400000000000001},"max":{"1min":3.8239999999999998,"5min":6.8650000000000002,"15min":8.7300000000000004},"last":0.47699999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":1.0740000000000001,"15min":0.88800000000000001},"min":{"1min":0.51800000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.99199999999999999,"5min":10.962,"15min":10.962},"last":0.626},{"interface":"front","average":{"1min":0.69299999999999995,"5min":1.4410000000000001,"15min":1.034},"min":{"1min":0.48399999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0189999999999999,"5min":21.724,"15min":26.146999999999998},"last":0.51300000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.83899999999999997},"min":{"1min":0.44500000000000001,"5min":0.36099999999999999,"15min":0.32000000000000001},"max":{"1min":0.95599999999999996,"5min":5.2160000000000002,"15min":21.535},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.79000000000000004,"5min":1.012,"15min":1.0129999999999999},"min":{"1min":0.52400000000000002,"5min":0.379,"15min":0.36299999999999999},"max":{"1min":1.03,"5min":5.7539999999999996,"15min":30.488},"last":0.67800000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":2.0070000000000001,"15min":1.216},"min":{"1min":0.47699999999999998,"5min":0.42599999999999999,"15min":0.39700000000000002},"max":{"1min":1.008,"5min":33.499000000000002,"15min":33.499000000000002},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.81000000000000005,"5min":1.502,"15min":1.0880000000000001},"min":{"1min":0.54300000000000004,"5min":0.42899999999999999,"15min":0.36499999999999999},"max":{"1min":1.0609999999999999,"5min":25.437999999999999,"15min":25.437999999999999},"last":0.64300000000000002}]}]},{"osd":0,"up_from":14,"seq":60129542279,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77092,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294748,"statfs":{"total":96636764160,"available":96557821952,"internally_reserved":0,"allocated":1593344,"data_stored":1017889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.90400000000000003,"15min":0.63},"min":{"1min":0.245,"5min":0.245,"15min":0.23699999999999999},"max":{"1min":0.78600000000000003,"5min":19.843,"15min":19.843},"last":0.37},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.94699999999999995,"15min":0.68000000000000005},"min":{"1min":0.29999999999999999,"5min":0.29199999999999998,"15min":0.27300000000000002},"max":{"1min":0.746,"5min":10.154,"15min":10.154},"last":0.39200000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":1.2729999999999999,"15min":0.80400000000000005},"min":{"1min":0.39500000000000002,"5min":0.34899999999999998,"15min":0.30399999999999999},"max":{"1min":0.84099999999999997,"5min":25.905000000000001,"15min":25.905000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.82699999999999996,"15min":0.61199999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.26700000000000002},"max":{"1min":0.80000000000000004,"5min":9.3829999999999991,"15min":9.3829999999999991},"last":0.51900000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":1.026,"15min":0.74199999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.80400000000000005,"5min":9.1780000000000008,"15min":9.1780000000000008},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":1.016,"15min":0.76700000000000002},"min":{"1min":0.41699999999999998,"5min":0.40000000000000002,"15min":0.36899999999999999},"max":{"1min":0.89900000000000002,"5min":10.210000000000001,"15min":10.210000000000001},"last":0.44500000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":1.1319999999999999,"15min":0.98799999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.38},"max":{"1min":1.1459999999999999,"5min":10.691000000000001,"15min":22.814},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.80400000000000005,"5min":1.395,"15min":1.042},"min":{"1min":0.50600000000000001,"5min":0.49099999999999999,"15min":0.41199999999999998},"max":{"1min":1.046,"5min":23.341999999999999,"15min":23.341999999999999},"last":0.66000000000000003}]},{"osd":5,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":1.141,"15min":0.96899999999999997},"min":{"1min":0.55700000000000005,"5min":0.46899999999999997,"15min":0.441},"max":{"1min":1.046,"5min":10.737,"15min":12.428000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.83799999999999997,"5min":1.962,"15min":1.27},"min":{"1min":0.56999999999999995,"5min":0.498,"15min":0.46300000000000002},"max":{"1min":1.3049999999999999,"5min":23.021000000000001,"15min":23.021000000000001},"last":0.68999999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":1.696,"15min":1.218},"min":{"1min":0.52700000000000002,"5min":0.46700000000000003,"15min":0.46200000000000002},"max":{"1min":1.0229999999999999,"5min":26.036000000000001,"15min":31.059000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.88800000000000001,"5min":1.083,"15min":1.0920000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.49099999999999999},"max":{"1min":1.931,"5min":6.9690000000000003,"15min":40.731000000000002},"last":0.70299999999999996}]},{"osd":7,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.8400000000000001,"15min":1.208},"min":{"1min":0.58999999999999997,"5min":0.54900000000000004,"15min":0.42499999999999999},"max":{"1min":4.1609999999999996,"5min":24.492999999999999,"15min":24.492999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.83499999999999996,"5min":1.242,"15min":1.1140000000000001},"min":{"1min":0.64500000000000002,"5min":0.58399999999999996,"15min":0.46600000000000003},"max":{"1min":1.008,"5min":9.2789999999999999,"15min":23.524000000000001},"last":0.72899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:15:49.686 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:15:49.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.872+0000 7f590b800640 1 Processor -- start 2024-01-15T18:15:49.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.872+0000 7f590b800640 1 -- start start 2024-01-15T18:15:49.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.872+0000 7f590b800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f590415e7d0 0x7f59040cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:49.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.872+0000 7f590b800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f59040cedc0 0x7f59040cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:49.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.872+0000 7f590b800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59040b38b0 con 0x7f59040ce2e0 2024-01-15T18:15:49.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.872+0000 7f590b800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59040cd550 con 0x7f590415e7d0 2024-01-15T18:15:49.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.872+0000 7f590b800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59040cd270 con 0x7f59040cedc0 2024-01-15T18:15:49.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.872+0000 7f5908e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f590415e7d0 0x7f59040cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:49.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.872+0000 7f5908e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f590415e7d0 0x7f59040cf2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56618/0 (socket says 172.21.15.27:56618) 2024-01-15T18:15:49.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.872+0000 7f5908e00640 1 -- 172.21.15.27:0/2814849271 learned_addr learned my addr 172.21.15.27:0/2814849271 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:49.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.872+0000 7f5903e00640 1 --2- 172.21.15.27:0/2814849271 >> v2:172.21.15.93:3300/0 conn(0x7f59040cedc0 0x7f59040cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:49.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.872+0000 7f5908e00640 1 -- 172.21.15.27:0/2814849271 >> v2:172.21.15.93:3300/0 conn(0x7f59040cedc0 msgr2=0x7f59040cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:49.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.872+0000 7f5908e00640 1 --2- 172.21.15.27:0/2814849271 >> v2:172.21.15.93:3300/0 conn(0x7f59040cedc0 0x7f59040cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:49.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.872+0000 7f5908e00640 1 -- 172.21.15.27:0/2814849271 >> v1:172.21.15.93:6789/0 conn(0x7f59040ce2e0 legacy=0x7f59040cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:49.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.872+0000 7f5908e00640 1 -- 172.21.15.27:0/2814849271 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59040b35d0 con 0x7f590415e7d0 2024-01-15T18:15:49.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.876+0000 7f5908e00640 1 --2- 172.21.15.27:0/2814849271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f590415e7d0 0x7f59040cf2a0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f59040718a0 tx=0x7f58f401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a43925a8b6ef707 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:49.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.876+0000 7f5903400640 1 -- 172.21.15.27:0/2814849271 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f40ad660 con 0x7f590415e7d0 2024-01-15T18:15:49.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.876+0000 7f5903400640 1 -- 172.21.15.27:0/2814849271 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f40ad7c0 con 0x7f590415e7d0 2024-01-15T18:15:49.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.876+0000 7f5903400640 1 -- 172.21.15.27:0/2814849271 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f40adb50 con 0x7f590415e7d0 2024-01-15T18:15:49.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.876+0000 7f590b800640 1 -- 172.21.15.27:0/2814849271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f590415e7d0 msgr2=0x7f59040cf2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:49.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.876+0000 7f590b800640 1 --2- 172.21.15.27:0/2814849271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f590415e7d0 0x7f59040cf2a0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f59040718a0 tx=0x7f58f401c470 comp rx=0 tx=0).stop 2024-01-15T18:15:49.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.876+0000 7f590b800640 1 -- 172.21.15.27:0/2814849271 shutdown_connections 2024-01-15T18:15:49.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.876+0000 7f590b800640 1 --2- 172.21.15.27:0/2814849271 >> v2:172.21.15.93:3300/0 conn(0x7f59040cedc0 0x7f59040cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:49.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.876+0000 7f590b800640 1 --2- 172.21.15.27:0/2814849271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f590415e7d0 0x7f59040cf2a0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:49.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.876+0000 7f590b800640 1 -- 172.21.15.27:0/2814849271 >> 172.21.15.27:0/2814849271 conn(0x7f59040b2a10 msgr2=0x7f59040b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:49.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.876+0000 7f590b800640 1 -- 172.21.15.27:0/2814849271 shutdown_connections 2024-01-15T18:15:49.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.876+0000 7f590b800640 1 -- 172.21.15.27:0/2814849271 wait complete. 2024-01-15T18:15:49.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.880+0000 7f590b800640 1 Processor -- start 2024-01-15T18:15:49.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.880+0000 7f590b800640 1 -- start start 2024-01-15T18:15:49.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.880+0000 7f590b800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59040ce2e0 0x7f5904153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:49.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.880+0000 7f590b800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59040cedc0 0x7f590406c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:49.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.880+0000 7f590b800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f590415e7d0 0x7f59040aed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:49.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.880+0000 7f590b800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590406d4a0 con 0x7f59040cedc0 2024-01-15T18:15:49.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.880+0000 7f590b800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59040c8c50 con 0x7f590415e7d0 2024-01-15T18:15:49.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.880+0000 7f590b800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f590406b9e0 con 0x7f59040ce2e0 2024-01-15T18:15:49.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.880+0000 7f5903e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59040cedc0 0x7f590406c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:49.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.880+0000 7f5903e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59040cedc0 0x7f590406c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56628/0 (socket says 172.21.15.27:56628) 2024-01-15T18:15:49.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.880+0000 7f5903e00640 1 -- 172.21.15.27:0/2838947852 learned_addr learned my addr 172.21.15.27:0/2838947852 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:49.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.880+0000 7f5908e00640 1 --2- 172.21.15.27:0/2838947852 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59040ce2e0 0x7f5904153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:49.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.880+0000 7f5909800640 1 --2- 172.21.15.27:0/2838947852 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f590415e7d0 0x7f59040aed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:49.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.880+0000 7f5903e00640 1 -- 172.21.15.27:0/2838947852 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59040ce2e0 msgr2=0x7f5904153fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:49.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.880+0000 7f5903e00640 1 --2- 172.21.15.27:0/2838947852 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59040ce2e0 0x7f5904153fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:49.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.880+0000 7f5903e00640 1 -- 172.21.15.27:0/2838947852 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f590415e7d0 msgr2=0x7f59040aed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:49.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.880+0000 7f5903e00640 1 --2- 172.21.15.27:0/2838947852 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f590415e7d0 0x7f59040aed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:49.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.880+0000 7f5903e00640 1 -- 172.21.15.27:0/2838947852 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f40094d0 con 0x7f59040cedc0 2024-01-15T18:15:49.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.880+0000 7f5903e00640 1 --2- 172.21.15.27:0/2838947852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59040cedc0 0x7f590406c560 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f58ec00d5f0 tx=0x7f58ec00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:49.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.880+0000 7f5900c00640 1 -- 172.21.15.27:0/2838947852 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ec0125f0 con 0x7f59040cedc0 2024-01-15T18:15:49.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.880+0000 7f5900c00640 1 -- 172.21.15.27:0/2838947852 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58ec012750 con 0x7f59040cedc0 2024-01-15T18:15:49.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.880+0000 7f5900c00640 1 -- 172.21.15.27:0/2838947852 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ec012ae0 con 0x7f59040cedc0 2024-01-15T18:15:49.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.880+0000 7f590b800640 1 -- 172.21.15.27:0/2838947852 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f590406bcc0 con 0x7f59040cedc0 2024-01-15T18:15:49.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.880+0000 7f590b800640 1 -- 172.21.15.27:0/2838947852 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f590406c280 con 0x7f59040cedc0 2024-01-15T18:15:49.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.884+0000 7f590b800640 1 -- 172.21.15.27:0/2838947852 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f590404e850 con 0x7f59040cedc0 2024-01-15T18:15:49.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.884+0000 7f5900c00640 1 -- 172.21.15.27:0/2838947852 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7f58ec012c40 con 0x7f59040cedc0 2024-01-15T18:15:49.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.884+0000 7f5900c00640 1 --2- 172.21.15.27:0/2838947852 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f58dc0674c0 0x7f58dc069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:49.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.884+0000 7f5900c00640 1 -- 172.21.15.27:0/2838947852 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f58ec083400 con 0x7f59040cedc0 2024-01-15T18:15:49.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.884+0000 7f5908e00640 1 --2- 172.21.15.27:0/2838947852 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f58dc0674c0 0x7f58dc069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:49.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.884+0000 7f5908e00640 1 --2- 172.21.15.27:0/2838947852 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f58dc0674c0 0x7f58dc069980 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f58f40af000 tx=0x7f58f4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:49.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:49.888+0000 7f5900c00640 1 -- 172.21.15.27:0/2838947852 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f58ec051ee0 con 0x7f59040cedc0 2024-01-15T18:15:50.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.044+0000 7f590b800640 1 -- 172.21.15.27:0/2838947852 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5904067070 con 0x7f58dc0674c0 2024-01-15T18:15:50.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.052+0000 7f5900c00640 1 -- 172.21.15.27:0/2838947852 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287360 (secure 0 0 0) 0x7f5904067070 con 0x7f58dc0674c0 2024-01-15T18:15:50.052 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:15:50.053 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:15:50.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.056+0000 7f590b800640 1 -- 172.21.15.27:0/2838947852 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f58dc0674c0 msgr2=0x7f58dc069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:50.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.056+0000 7f590b800640 1 --2- 172.21.15.27:0/2838947852 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f58dc0674c0 0x7f58dc069980 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f58f40af000 tx=0x7f58f4008040 comp rx=0 tx=0).stop 2024-01-15T18:15:50.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.056+0000 7f590b800640 1 -- 172.21.15.27:0/2838947852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59040cedc0 msgr2=0x7f590406c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:50.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.056+0000 7f590b800640 1 --2- 172.21.15.27:0/2838947852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59040cedc0 0x7f590406c560 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f58ec00d5f0 tx=0x7f58ec00dab0 comp rx=0 tx=0).stop 2024-01-15T18:15:50.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.056+0000 7f590b800640 1 -- 172.21.15.27:0/2838947852 shutdown_connections 2024-01-15T18:15:50.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.056+0000 7f590b800640 1 --2- 172.21.15.27:0/2838947852 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f58dc0674c0 0x7f58dc069980 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:50.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.056+0000 7f590b800640 1 --2- 172.21.15.27:0/2838947852 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f590415e7d0 0x7f59040aed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:50.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.056+0000 7f590b800640 1 --2- 172.21.15.27:0/2838947852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59040cedc0 0x7f590406c560 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:50.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.056+0000 7f590b800640 1 --2- 172.21.15.27:0/2838947852 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59040ce2e0 0x7f5904153fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:50.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.056+0000 7f590b800640 1 -- 172.21.15.27:0/2838947852 >> 172.21.15.27:0/2838947852 conn(0x7f59040b2a10 msgr2=0x7f590415b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:50.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.056+0000 7f590b800640 1 -- 172.21.15.27:0/2838947852 shutdown_connections 2024-01-15T18:15:50.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.056+0000 7f590b800640 1 -- 172.21.15.27:0/2838947852 wait complete. 2024-01-15T18:15:50.073 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-01-15T18:15:48.331070+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591166,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":176,"num_write_kb":2427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":79,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":563496,"kb_used_data":8996,"kb_used_omap":12,"kb_used_meta":554355,"kb_avail":754411224,"statfs":{"total":773094113280,"available":772517093376,"internally_reserved":0,"allocated":9211904,"data_stored":4609039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":567660080},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":798,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.192938"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.197673+0000","last_change":"2024-01-15T18:15:45.194410+0000","last_active":"2024-01-15T18:15:45.197673+0000","last_peered":"2024-01-15T18:15:45.197673+0000","last_clean":"2024-01-15T18:15:45.197673+0000","last_became_active":"2024-01-15T18:15:45.193977+0000","last_became_peered":"2024-01-15T18:15:45.193977+0000","last_unstale":"2024-01-15T18:15:45.197673+0000","last_undegraded":"2024-01-15T18:15:45.197673+0000","last_fullsized":"2024-01-15T18:15:45.197673+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:59:33.238710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.193135+0000","last_change":"2024-01-15T18:15:45.185773+0000","last_active":"2024-01-15T18:15:45.193135+0000","last_peered":"2024-01-15T18:15:45.193135+0000","last_clean":"2024-01-15T18:15:45.193135+0000","last_became_active":"2024-01-15T18:15:45.185459+0000","last_became_peered":"2024-01-15T18:15:45.185459+0000","last_unstale":"2024-01-15T18:15:45.193135+0000","last_undegraded":"2024-01-15T18:15:45.193135+0000","last_fullsized":"2024-01-15T18:15:45.193135+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:33:22.524087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196740+0000","last_change":"2024-01-15T18:15:45.190593+0000","last_active":"2024-01-15T18:15:45.196740+0000","last_peered":"2024-01-15T18:15:45.196740+0000","last_clean":"2024-01-15T18:15:45.196740+0000","last_became_active":"2024-01-15T18:15:45.190292+0000","last_became_peered":"2024-01-15T18:15:45.190292+0000","last_unstale":"2024-01-15T18:15:45.196740+0000","last_undegraded":"2024-01-15T18:15:45.196740+0000","last_fullsized":"2024-01-15T18:15:45.196740+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:41:44.153310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.197017+0000","last_change":"2024-01-15T18:15:45.192728+0000","last_active":"2024-01-15T18:15:45.197017+0000","last_peered":"2024-01-15T18:15:45.197017+0000","last_clean":"2024-01-15T18:15:45.197017+0000","last_became_active":"2024-01-15T18:15:45.192270+0000","last_became_peered":"2024-01-15T18:15:45.192270+0000","last_unstale":"2024-01-15T18:15:45.197017+0000","last_undegraded":"2024-01-15T18:15:45.197017+0000","last_fullsized":"2024-01-15T18:15:45.197017+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:10:12.400165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191097+0000","last_change":"2024-01-15T18:15:45.181959+0000","last_active":"2024-01-15T18:15:45.191097+0000","last_peered":"2024-01-15T18:15:45.191097+0000","last_clean":"2024-01-15T18:15:45.191097+0000","last_became_active":"2024-01-15T18:15:45.181526+0000","last_became_peered":"2024-01-15T18:15:45.181526+0000","last_unstale":"2024-01-15T18:15:45.191097+0000","last_undegraded":"2024-01-15T18:15:45.191097+0000","last_fullsized":"2024-01-15T18:15:45.191097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:10:10.202003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.145723+0000","last_change":"2024-01-15T18:15:44.145723+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.145723+0000","last_undegraded":"2024-01-15T18:15:44.145723+0000","last_fullsized":"2024-01-15T18:15:44.145723+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.188332+0000","last_change":"2024-01-15T18:15:45.177382+0000","last_active":"2024-01-15T18:15:45.188332+0000","last_peered":"2024-01-15T18:15:45.188332+0000","last_clean":"2024-01-15T18:15:45.188332+0000","last_became_active":"2024-01-15T18:15:45.173825+0000","last_became_peered":"2024-01-15T18:15:45.173825+0000","last_unstale":"2024-01-15T18:15:45.188332+0000","last_undegraded":"2024-01-15T18:15:45.188332+0000","last_fullsized":"2024-01-15T18:15:45.188332+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:03:55.153676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.144794+0000","last_change":"2024-01-15T18:15:44.144794+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.144794+0000","last_undegraded":"2024-01-15T18:15:44.144794+0000","last_fullsized":"2024-01-15T18:15:44.144794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.145532+0000","last_change":"2024-01-15T18:15:44.145532+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.145532+0000","last_undegraded":"2024-01-15T18:15:44.145532+0000","last_fullsized":"2024-01-15T18:15:44.145532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.195302+0000","last_change":"2024-01-15T18:15:45.190970+0000","last_active":"2024-01-15T18:15:45.195302+0000","last_peered":"2024-01-15T18:15:45.195302+0000","last_clean":"2024-01-15T18:15:45.195302+0000","last_became_active":"2024-01-15T18:15:45.187235+0000","last_became_peered":"2024-01-15T18:15:45.187235+0000","last_unstale":"2024-01-15T18:15:45.195302+0000","last_undegraded":"2024-01-15T18:15:45.195302+0000","last_fullsized":"2024-01-15T18:15:45.195302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:35:49.681365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.186724+0000","last_change":"2024-01-15T18:15:45.179561+0000","last_active":"2024-01-15T18:15:45.186724+0000","last_peered":"2024-01-15T18:15:45.186724+0000","last_clean":"2024-01-15T18:15:45.186724+0000","last_became_active":"2024-01-15T18:15:45.179305+0000","last_became_peered":"2024-01-15T18:15:45.179305+0000","last_unstale":"2024-01-15T18:15:45.186724+0000","last_undegraded":"2024-01-15T18:15:45.186724+0000","last_fullsized":"2024-01-15T18:15:45.186724+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:56:26.621247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191872+0000","last_change":"2024-01-15T18:15:45.189460+0000","last_active":"2024-01-15T18:15:45.191872+0000","last_peered":"2024-01-15T18:15:45.191872+0000","last_clean":"2024-01-15T18:15:45.191872+0000","last_became_active":"2024-01-15T18:15:45.188105+0000","last_became_peered":"2024-01-15T18:15:45.188105+0000","last_unstale":"2024-01-15T18:15:45.191872+0000","last_undegraded":"2024-01-15T18:15:45.191872+0000","last_fullsized":"2024-01-15T18:15:45.191872+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:45:42.971410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196682+0000","last_change":"2024-01-15T18:15:45.190326+0000","last_active":"2024-01-15T18:15:45.196682+0000","last_peered":"2024-01-15T18:15:45.196682+0000","last_clean":"2024-01-15T18:15:45.196682+0000","last_became_active":"2024-01-15T18:15:45.188312+0000","last_became_peered":"2024-01-15T18:15:45.188312+0000","last_unstale":"2024-01-15T18:15:45.196682+0000","last_undegraded":"2024-01-15T18:15:45.196682+0000","last_fullsized":"2024-01-15T18:15:45.196682+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:28:05.260374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191135+0000","last_change":"2024-01-15T18:15:45.180022+0000","last_active":"2024-01-15T18:15:45.191135+0000","last_peered":"2024-01-15T18:15:45.191135+0000","last_clean":"2024-01-15T18:15:45.191135+0000","last_became_active":"2024-01-15T18:15:45.179537+0000","last_became_peered":"2024-01-15T18:15:45.179537+0000","last_unstale":"2024-01-15T18:15:45.191135+0000","last_undegraded":"2024-01-15T18:15:45.191135+0000","last_fullsized":"2024-01-15T18:15:45.191135+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:55.575874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196300+0000","last_change":"2024-01-15T18:15:45.185924+0000","last_active":"2024-01-15T18:15:45.196300+0000","last_peered":"2024-01-15T18:15:45.196300+0000","last_clean":"2024-01-15T18:15:45.196300+0000","last_became_active":"2024-01-15T18:15:45.185669+0000","last_became_peered":"2024-01-15T18:15:45.185669+0000","last_unstale":"2024-01-15T18:15:45.196300+0000","last_undegraded":"2024-01-15T18:15:45.196300+0000","last_fullsized":"2024-01-15T18:15:45.196300+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:08:31.073685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.184890+0000","last_change":"2024-01-15T18:15:45.176262+0000","last_active":"2024-01-15T18:15:45.184890+0000","last_peered":"2024-01-15T18:15:45.184890+0000","last_clean":"2024-01-15T18:15:45.184890+0000","last_became_active":"2024-01-15T18:15:45.175976+0000","last_became_peered":"2024-01-15T18:15:45.175976+0000","last_unstale":"2024-01-15T18:15:45.184890+0000","last_undegraded":"2024-01-15T18:15:45.184890+0000","last_fullsized":"2024-01-15T18:15:45.184890+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:16.351733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191295+0000","last_change":"2024-01-15T18:15:45.184436+0000","last_active":"2024-01-15T18:15:45.191295+0000","last_peered":"2024-01-15T18:15:45.191295+0000","last_clean":"2024-01-15T18:15:45.191295+0000","last_became_active":"2024-01-15T18:15:45.181072+0000","last_became_peered":"2024-01-15T18:15:45.181072+0000","last_unstale":"2024-01-15T18:15:45.191295+0000","last_undegraded":"2024-01-15T18:15:45.191295+0000","last_fullsized":"2024-01-15T18:15:45.191295+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:04:09.741683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191667+0000","last_change":"2024-01-15T18:15:45.184794+0000","last_active":"2024-01-15T18:15:45.191667+0000","last_peered":"2024-01-15T18:15:45.191667+0000","last_clean":"2024-01-15T18:15:45.191667+0000","last_became_active":"2024-01-15T18:15:45.184528+0000","last_became_peered":"2024-01-15T18:15:45.184528+0000","last_unstale":"2024-01-15T18:15:45.191667+0000","last_undegraded":"2024-01-15T18:15:45.191667+0000","last_fullsized":"2024-01-15T18:15:45.191667+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:33.583003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.158290+0000","last_change":"2024-01-15T18:15:44.158290+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.158290+0000","last_undegraded":"2024-01-15T18:15:44.158290+0000","last_fullsized":"2024-01-15T18:15:44.158290+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.203247+0000","last_change":"2024-01-15T18:15:45.202436+0000","last_active":"2024-01-15T18:15:45.203247+0000","last_peered":"2024-01-15T18:15:45.203247+0000","last_clean":"2024-01-15T18:15:45.203247+0000","last_became_active":"2024-01-15T18:15:45.202075+0000","last_became_peered":"2024-01-15T18:15:45.202075+0000","last_unstale":"2024-01-15T18:15:45.203247+0000","last_undegraded":"2024-01-15T18:15:45.203247+0000","last_fullsized":"2024-01-15T18:15:45.203247+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:29:34.404126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198796+0000","last_change":"2024-01-15T18:15:45.197310+0000","last_active":"2024-01-15T18:15:45.198796+0000","last_peered":"2024-01-15T18:15:45.198796+0000","last_clean":"2024-01-15T18:15:45.198796+0000","last_became_active":"2024-01-15T18:15:45.196364+0000","last_became_peered":"2024-01-15T18:15:45.196364+0000","last_unstale":"2024-01-15T18:15:45.198796+0000","last_undegraded":"2024-01-15T18:15:45.198796+0000","last_fullsized":"2024-01-15T18:15:45.198796+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:28:16.596449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191310+0000","last_change":"2024-01-15T18:15:45.185683+0000","last_active":"2024-01-15T18:15:45.191310+0000","last_peered":"2024-01-15T18:15:45.191310+0000","last_clean":"2024-01-15T18:15:45.191310+0000","last_became_active":"2024-01-15T18:15:45.185303+0000","last_became_peered":"2024-01-15T18:15:45.185303+0000","last_unstale":"2024-01-15T18:15:45.191310+0000","last_undegraded":"2024-01-15T18:15:45.191310+0000","last_fullsized":"2024-01-15T18:15:45.191310+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:12:38.030672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198110+0000","last_change":"2024-01-15T18:15:45.195966+0000","last_active":"2024-01-15T18:15:45.198110+0000","last_peered":"2024-01-15T18:15:45.198110+0000","last_clean":"2024-01-15T18:15:45.198110+0000","last_became_active":"2024-01-15T18:15:45.194717+0000","last_became_peered":"2024-01-15T18:15:45.194717+0000","last_unstale":"2024-01-15T18:15:45.198110+0000","last_undegraded":"2024-01-15T18:15:45.198110+0000","last_fullsized":"2024-01-15T18:15:45.198110+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:50:32.308793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.190685+0000","last_change":"2024-01-15T18:15:45.179745+0000","last_active":"2024-01-15T18:15:45.190685+0000","last_peered":"2024-01-15T18:15:45.190685+0000","last_clean":"2024-01-15T18:15:45.190685+0000","last_became_active":"2024-01-15T18:15:45.179329+0000","last_became_peered":"2024-01-15T18:15:45.179329+0000","last_unstale":"2024-01-15T18:15:45.190685+0000","last_undegraded":"2024-01-15T18:15:45.190685+0000","last_fullsized":"2024-01-15T18:15:45.190685+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:01:10.870535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198301+0000","last_change":"2024-01-15T18:15:45.192970+0000","last_active":"2024-01-15T18:15:45.198301+0000","last_peered":"2024-01-15T18:15:45.198301+0000","last_clean":"2024-01-15T18:15:45.198301+0000","last_became_active":"2024-01-15T18:15:45.192474+0000","last_became_peered":"2024-01-15T18:15:45.192474+0000","last_unstale":"2024-01-15T18:15:45.198301+0000","last_undegraded":"2024-01-15T18:15:45.198301+0000","last_fullsized":"2024-01-15T18:15:45.198301+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:38.217871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.183678+0000","last_change":"2024-01-15T18:15:45.180211+0000","last_active":"2024-01-15T18:15:45.183678+0000","last_peered":"2024-01-15T18:15:45.183678+0000","last_clean":"2024-01-15T18:15:45.183678+0000","last_became_active":"2024-01-15T18:15:45.179566+0000","last_became_peered":"2024-01-15T18:15:45.179566+0000","last_unstale":"2024-01-15T18:15:45.183678+0000","last_undegraded":"2024-01-15T18:15:45.183678+0000","last_fullsized":"2024-01-15T18:15:45.183678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:40:13.829197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.192821+0000","last_change":"2024-01-15T18:15:45.189295+0000","last_active":"2024-01-15T18:15:45.192821+0000","last_peered":"2024-01-15T18:15:45.192821+0000","last_clean":"2024-01-15T18:15:45.192821+0000","last_became_active":"2024-01-15T18:15:45.187101+0000","last_became_peered":"2024-01-15T18:15:45.187101+0000","last_unstale":"2024-01-15T18:15:45.192821+0000","last_undegraded":"2024-01-15T18:15:45.192821+0000","last_fullsized":"2024-01-15T18:15:45.192821+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:01:13.745010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.195405+0000","last_change":"2024-01-15T18:15:45.192353+0000","last_active":"2024-01-15T18:15:45.195405+0000","last_peered":"2024-01-15T18:15:45.195405+0000","last_clean":"2024-01-15T18:15:45.195405+0000","last_became_active":"2024-01-15T18:15:45.191563+0000","last_became_peered":"2024-01-15T18:15:45.191563+0000","last_unstale":"2024-01-15T18:15:45.195405+0000","last_undegraded":"2024-01-15T18:15:45.195405+0000","last_fullsized":"2024-01-15T18:15:45.195405+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:57:43.892130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.197122+0000","last_change":"2024-01-15T18:15:45.190607+0000","last_active":"2024-01-15T18:15:45.197122+0000","last_peered":"2024-01-15T18:15:45.197122+0000","last_clean":"2024-01-15T18:15:45.197122+0000","last_became_active":"2024-01-15T18:15:45.189633+0000","last_became_peered":"2024-01-15T18:15:45.189633+0000","last_unstale":"2024-01-15T18:15:45.197122+0000","last_undegraded":"2024-01-15T18:15:45.197122+0000","last_fullsized":"2024-01-15T18:15:45.197122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:35:49.681365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.187329+0000","last_change":"2024-01-15T18:15:45.180268+0000","last_active":"2024-01-15T18:15:45.187329+0000","last_peered":"2024-01-15T18:15:45.187329+0000","last_clean":"2024-01-15T18:15:45.187329+0000","last_became_active":"2024-01-15T18:15:45.179617+0000","last_became_peered":"2024-01-15T18:15:45.179617+0000","last_unstale":"2024-01-15T18:15:45.187329+0000","last_undegraded":"2024-01-15T18:15:45.187329+0000","last_fullsized":"2024-01-15T18:15:45.187329+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:18:22.468719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.186449+0000","last_change":"2024-01-15T18:15:45.177309+0000","last_active":"2024-01-15T18:15:45.186449+0000","last_peered":"2024-01-15T18:15:45.186449+0000","last_clean":"2024-01-15T18:15:45.186449+0000","last_became_active":"2024-01-15T18:15:45.177022+0000","last_became_peered":"2024-01-15T18:15:45.177022+0000","last_unstale":"2024-01-15T18:15:45.186449+0000","last_undegraded":"2024-01-15T18:15:45.186449+0000","last_fullsized":"2024-01-15T18:15:45.186449+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:53.495194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.187438+0000","last_change":"2024-01-15T18:15:45.182964+0000","last_active":"2024-01-15T18:15:45.187438+0000","last_peered":"2024-01-15T18:15:45.187438+0000","last_clean":"2024-01-15T18:15:45.187438+0000","last_became_active":"2024-01-15T18:15:45.182690+0000","last_became_peered":"2024-01-15T18:15:45.182690+0000","last_unstale":"2024-01-15T18:15:45.187438+0000","last_undegraded":"2024-01-15T18:15:45.187438+0000","last_fullsized":"2024-01-15T18:15:45.187438+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:31.079315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.192979+0000","last_change":"2024-01-15T18:15:45.189818+0000","last_active":"2024-01-15T18:15:45.192979+0000","last_peered":"2024-01-15T18:15:45.192979+0000","last_clean":"2024-01-15T18:15:45.192979+0000","last_became_active":"2024-01-15T18:15:45.189534+0000","last_became_peered":"2024-01-15T18:15:45.189534+0000","last_unstale":"2024-01-15T18:15:45.192979+0000","last_undegraded":"2024-01-15T18:15:45.192979+0000","last_fullsized":"2024-01-15T18:15:45.192979+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:13:12.060989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"83'4","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.197594+0000","last_change":"2024-01-15T18:15:44.163810+0000","last_active":"2024-01-15T18:15:45.197594+0000","last_peered":"2024-01-15T18:15:45.197594+0000","last_clean":"2024-01-15T18:15:45.197594+0000","last_became_active":"2024-01-15T18:15:44.163128+0000","last_became_peered":"2024-01-15T18:15:44.163128+0000","last_unstale":"2024-01-15T18:15:45.197594+0000","last_undegraded":"2024-01-15T18:15:45.197594+0000","last_fullsized":"2024-01-15T18:15:45.197594+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:22:07.304644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.202605+0000","last_change":"2024-01-15T18:15:45.201966+0000","last_active":"2024-01-15T18:15:45.202605+0000","last_peered":"2024-01-15T18:15:45.202605+0000","last_clean":"2024-01-15T18:15:45.202605+0000","last_became_active":"2024-01-15T18:15:45.201568+0000","last_became_peered":"2024-01-15T18:15:45.201568+0000","last_unstale":"2024-01-15T18:15:45.202605+0000","last_undegraded":"2024-01-15T18:15:45.202605+0000","last_fullsized":"2024-01-15T18:15:45.202605+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:16.351733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.187621+0000","last_change":"2024-01-15T18:15:44.156714+0000","last_active":"2024-01-15T18:15:45.187621+0000","last_peered":"2024-01-15T18:15:45.187621+0000","last_clean":"2024-01-15T18:15:45.187621+0000","last_became_active":"2024-01-15T18:15:44.156419+0000","last_became_peered":"2024-01-15T18:15:44.156419+0000","last_unstale":"2024-01-15T18:15:45.187621+0000","last_undegraded":"2024-01-15T18:15:45.187621+0000","last_fullsized":"2024-01-15T18:15:45.187621+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:14:41.627743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.197147+0000","last_change":"2024-01-15T18:15:45.192955+0000","last_active":"2024-01-15T18:15:45.197147+0000","last_peered":"2024-01-15T18:15:45.197147+0000","last_clean":"2024-01-15T18:15:45.197147+0000","last_became_active":"2024-01-15T18:15:45.192565+0000","last_became_peered":"2024-01-15T18:15:45.192565+0000","last_unstale":"2024-01-15T18:15:45.197147+0000","last_undegraded":"2024-01-15T18:15:45.197147+0000","last_fullsized":"2024-01-15T18:15:45.197147+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:07:58.498146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.171318+0000","last_change":"2024-01-15T18:15:44.167059+0000","last_active":"2024-01-15T18:15:44.171318+0000","last_peered":"2024-01-15T18:15:44.171318+0000","last_clean":"2024-01-15T18:15:44.171318+0000","last_became_active":"2024-01-15T18:15:44.166829+0000","last_became_peered":"2024-01-15T18:15:44.166829+0000","last_unstale":"2024-01-15T18:15:44.171318+0000","last_undegraded":"2024-01-15T18:15:44.171318+0000","last_fullsized":"2024-01-15T18:15:44.171318+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:39:01.209507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.194382+0000","last_change":"2024-01-15T18:15:45.186505+0000","last_active":"2024-01-15T18:15:45.194382+0000","last_peered":"2024-01-15T18:15:45.194382+0000","last_clean":"2024-01-15T18:15:45.194382+0000","last_became_active":"2024-01-15T18:15:45.186181+0000","last_became_peered":"2024-01-15T18:15:45.186181+0000","last_unstale":"2024-01-15T18:15:45.194382+0000","last_undegraded":"2024-01-15T18:15:45.194382+0000","last_fullsized":"2024-01-15T18:15:45.194382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:03:29.579452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.170558+0000","last_change":"2024-01-15T18:15:44.166697+0000","last_active":"2024-01-15T18:15:44.170558+0000","last_peered":"2024-01-15T18:15:44.170558+0000","last_clean":"2024-01-15T18:15:44.170558+0000","last_became_active":"2024-01-15T18:15:44.166389+0000","last_became_peered":"2024-01-15T18:15:44.166389+0000","last_unstale":"2024-01-15T18:15:44.170558+0000","last_undegraded":"2024-01-15T18:15:44.170558+0000","last_fullsized":"2024-01-15T18:15:44.170558+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:19:36.045488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.187351+0000","last_change":"2024-01-15T18:15:45.180445+0000","last_active":"2024-01-15T18:15:45.187351+0000","last_peered":"2024-01-15T18:15:45.187351+0000","last_clean":"2024-01-15T18:15:45.187351+0000","last_became_active":"2024-01-15T18:15:45.179946+0000","last_became_peered":"2024-01-15T18:15:45.179946+0000","last_unstale":"2024-01-15T18:15:45.187351+0000","last_undegraded":"2024-01-15T18:15:45.187351+0000","last_fullsized":"2024-01-15T18:15:45.187351+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:50:32.308793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.181787+0000","last_change":"2024-01-15T18:15:44.157992+0000","last_active":"2024-01-15T18:15:45.181787+0000","last_peered":"2024-01-15T18:15:45.181787+0000","last_clean":"2024-01-15T18:15:45.181787+0000","last_became_active":"2024-01-15T18:15:44.157721+0000","last_became_peered":"2024-01-15T18:15:44.157721+0000","last_unstale":"2024-01-15T18:15:45.181787+0000","last_undegraded":"2024-01-15T18:15:45.181787+0000","last_fullsized":"2024-01-15T18:15:45.181787+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:27:47.584018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.185831+0000","last_change":"2024-01-15T18:15:45.178738+0000","last_active":"2024-01-15T18:15:45.185831+0000","last_peered":"2024-01-15T18:15:45.185831+0000","last_clean":"2024-01-15T18:15:45.185831+0000","last_became_active":"2024-01-15T18:15:45.178446+0000","last_became_peered":"2024-01-15T18:15:45.178446+0000","last_unstale":"2024-01-15T18:15:45.185831+0000","last_undegraded":"2024-01-15T18:15:45.185831+0000","last_fullsized":"2024-01-15T18:15:45.185831+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:59:33.238710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.187378+0000","last_change":"2024-01-15T18:15:44.154715+0000","last_active":"2024-01-15T18:15:45.187378+0000","last_peered":"2024-01-15T18:15:45.187378+0000","last_clean":"2024-01-15T18:15:45.187378+0000","last_became_active":"2024-01-15T18:15:44.154433+0000","last_became_peered":"2024-01-15T18:15:44.154433+0000","last_unstale":"2024-01-15T18:15:45.187378+0000","last_undegraded":"2024-01-15T18:15:45.187378+0000","last_fullsized":"2024-01-15T18:15:45.187378+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:42:00.915026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.202740+0000","last_change":"2024-01-15T18:15:45.202049+0000","last_active":"2024-01-15T18:15:45.202740+0000","last_peered":"2024-01-15T18:15:45.202740+0000","last_clean":"2024-01-15T18:15:45.202740+0000","last_became_active":"2024-01-15T18:15:45.201742+0000","last_became_peered":"2024-01-15T18:15:45.201742+0000","last_unstale":"2024-01-15T18:15:45.202740+0000","last_undegraded":"2024-01-15T18:15:45.202740+0000","last_fullsized":"2024-01-15T18:15:45.202740+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:56.553890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.172316+0000","last_change":"2024-01-15T18:15:44.167859+0000","last_active":"2024-01-15T18:15:44.172316+0000","last_peered":"2024-01-15T18:15:44.172316+0000","last_clean":"2024-01-15T18:15:44.172316+0000","last_became_active":"2024-01-15T18:15:44.167561+0000","last_became_peered":"2024-01-15T18:15:44.167561+0000","last_unstale":"2024-01-15T18:15:44.172316+0000","last_undegraded":"2024-01-15T18:15:44.172316+0000","last_fullsized":"2024-01-15T18:15:44.172316+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:17.332973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196319+0000","last_change":"2024-01-15T18:15:45.187853+0000","last_active":"2024-01-15T18:15:45.196319+0000","last_peered":"2024-01-15T18:15:45.196319+0000","last_clean":"2024-01-15T18:15:45.196319+0000","last_became_active":"2024-01-15T18:15:45.187552+0000","last_became_peered":"2024-01-15T18:15:45.187552+0000","last_unstale":"2024-01-15T18:15:45.196319+0000","last_undegraded":"2024-01-15T18:15:45.196319+0000","last_fullsized":"2024-01-15T18:15:45.196319+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:56.553890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.162106+0000","last_change":"2024-01-15T18:15:44.155651+0000","last_active":"2024-01-15T18:15:45.162106+0000","last_peered":"2024-01-15T18:15:45.162106+0000","last_clean":"2024-01-15T18:15:45.162106+0000","last_became_active":"2024-01-15T18:15:44.155306+0000","last_became_peered":"2024-01-15T18:15:44.155306+0000","last_unstale":"2024-01-15T18:15:45.162106+0000","last_undegraded":"2024-01-15T18:15:45.162106+0000","last_fullsized":"2024-01-15T18:15:45.162106+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:06:15.812490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.189876+0000","last_change":"2024-01-15T18:15:45.184002+0000","last_active":"2024-01-15T18:15:45.189876+0000","last_peered":"2024-01-15T18:15:45.189876+0000","last_clean":"2024-01-15T18:15:45.189876+0000","last_became_active":"2024-01-15T18:15:45.183624+0000","last_became_peered":"2024-01-15T18:15:45.183624+0000","last_unstale":"2024-01-15T18:15:45.189876+0000","last_undegraded":"2024-01-15T18:15:45.189876+0000","last_fullsized":"2024-01-15T18:15:45.189876+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:03:29.579452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.186739+0000","last_change":"2024-01-15T18:15:44.160123+0000","last_active":"2024-01-15T18:15:45.186739+0000","last_peered":"2024-01-15T18:15:45.186739+0000","last_clean":"2024-01-15T18:15:45.186739+0000","last_became_active":"2024-01-15T18:15:44.159761+0000","last_became_peered":"2024-01-15T18:15:44.159761+0000","last_unstale":"2024-01-15T18:15:45.186739+0000","last_undegraded":"2024-01-15T18:15:45.186739+0000","last_fullsized":"2024-01-15T18:15:45.186739+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:42:00.915026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.144921+0000","last_change":"2024-01-15T18:15:44.144921+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.144921+0000","last_undegraded":"2024-01-15T18:15:44.144921+0000","last_fullsized":"2024-01-15T18:15:44.144921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.187555+0000","last_change":"2024-01-15T18:15:44.166336+0000","last_active":"2024-01-15T18:15:45.187555+0000","last_peered":"2024-01-15T18:15:45.187555+0000","last_clean":"2024-01-15T18:15:45.187555+0000","last_became_active":"2024-01-15T18:15:44.165403+0000","last_became_peered":"2024-01-15T18:15:44.165403+0000","last_unstale":"2024-01-15T18:15:45.187555+0000","last_undegraded":"2024-01-15T18:15:45.187555+0000","last_fullsized":"2024-01-15T18:15:45.187555+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:42:53.652872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.155839+0000","last_change":"2024-01-15T18:15:44.155839+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.155839+0000","last_undegraded":"2024-01-15T18:15:44.155839+0000","last_fullsized":"2024-01-15T18:15:44.155839+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"83'3","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.197477+0000","last_change":"2024-01-15T18:15:44.160184+0000","last_active":"2024-01-15T18:15:45.197477+0000","last_peered":"2024-01-15T18:15:45.197477+0000","last_clean":"2024-01-15T18:15:45.197477+0000","last_became_active":"2024-01-15T18:15:44.159855+0000","last_became_peered":"2024-01-15T18:15:44.159855+0000","last_unstale":"2024-01-15T18:15:45.197477+0000","last_undegraded":"2024-01-15T18:15:45.197477+0000","last_fullsized":"2024-01-15T18:15:45.197477+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:04:08.736438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.192668+0000","last_change":"2024-01-15T18:15:45.187059+0000","last_active":"2024-01-15T18:15:45.192668+0000","last_peered":"2024-01-15T18:15:45.192668+0000","last_clean":"2024-01-15T18:15:45.192668+0000","last_became_active":"2024-01-15T18:15:45.184830+0000","last_became_peered":"2024-01-15T18:15:45.184830+0000","last_unstale":"2024-01-15T18:15:45.192668+0000","last_undegraded":"2024-01-15T18:15:45.192668+0000","last_fullsized":"2024-01-15T18:15:45.192668+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:58:35.413117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.163096+0000","last_change":"2024-01-15T18:15:44.158973+0000","last_active":"2024-01-15T18:15:44.163096+0000","last_peered":"2024-01-15T18:15:44.163096+0000","last_clean":"2024-01-15T18:15:44.163096+0000","last_became_active":"2024-01-15T18:15:44.158684+0000","last_became_peered":"2024-01-15T18:15:44.158684+0000","last_unstale":"2024-01-15T18:15:44.163096+0000","last_undegraded":"2024-01-15T18:15:44.163096+0000","last_fullsized":"2024-01-15T18:15:44.163096+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:22:27.571591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.153863+0000","last_change":"2024-01-15T18:15:44.153863+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.153863+0000","last_undegraded":"2024-01-15T18:15:44.153863+0000","last_fullsized":"2024-01-15T18:15:44.153863+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.201845+0000","last_change":"2024-01-15T18:15:44.166340+0000","last_active":"2024-01-15T18:15:45.201845+0000","last_peered":"2024-01-15T18:15:45.201845+0000","last_clean":"2024-01-15T18:15:45.201845+0000","last_became_active":"2024-01-15T18:15:44.165947+0000","last_became_peered":"2024-01-15T18:15:44.165947+0000","last_unstale":"2024-01-15T18:15:45.201845+0000","last_undegraded":"2024-01-15T18:15:45.201845+0000","last_fullsized":"2024-01-15T18:15:45.201845+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:12:37.715165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.199984+0000","last_change":"2024-01-15T18:15:45.199248+0000","last_active":"2024-01-15T18:15:45.199984+0000","last_peered":"2024-01-15T18:15:45.199984+0000","last_clean":"2024-01-15T18:15:45.199984+0000","last_became_active":"2024-01-15T18:15:45.198929+0000","last_became_peered":"2024-01-15T18:15:45.198929+0000","last_unstale":"2024-01-15T18:15:45.199984+0000","last_undegraded":"2024-01-15T18:15:45.199984+0000","last_fullsized":"2024-01-15T18:15:45.199984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:26:01.224733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.182090+0000","last_change":"2024-01-15T18:15:44.161901+0000","last_active":"2024-01-15T18:15:45.182090+0000","last_peered":"2024-01-15T18:15:45.182090+0000","last_clean":"2024-01-15T18:15:45.182090+0000","last_became_active":"2024-01-15T18:15:44.159756+0000","last_became_peered":"2024-01-15T18:15:44.159756+0000","last_unstale":"2024-01-15T18:15:45.182090+0000","last_undegraded":"2024-01-15T18:15:45.182090+0000","last_fullsized":"2024-01-15T18:15:45.182090+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:41:43.148064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.180743+0000","last_change":"2024-01-15T18:15:45.174549+0000","last_active":"2024-01-15T18:15:45.180743+0000","last_peered":"2024-01-15T18:15:45.180743+0000","last_clean":"2024-01-15T18:15:45.180743+0000","last_became_active":"2024-01-15T18:15:45.174046+0000","last_became_peered":"2024-01-15T18:15:45.174046+0000","last_unstale":"2024-01-15T18:15:45.180743+0000","last_undegraded":"2024-01-15T18:15:45.180743+0000","last_fullsized":"2024-01-15T18:15:45.180743+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:12:42.277859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.159621+0000","last_change":"2024-01-15T18:15:44.154875+0000","last_active":"2024-01-15T18:15:44.159621+0000","last_peered":"2024-01-15T18:15:44.159621+0000","last_clean":"2024-01-15T18:15:44.159621+0000","last_became_active":"2024-01-15T18:15:44.154585+0000","last_became_peered":"2024-01-15T18:15:44.154585+0000","last_unstale":"2024-01-15T18:15:44.159621+0000","last_undegraded":"2024-01-15T18:15:44.159621+0000","last_fullsized":"2024-01-15T18:15:44.159621+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:39:01.209507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198412+0000","last_change":"2024-01-15T18:15:45.193330+0000","last_active":"2024-01-15T18:15:45.198412+0000","last_peered":"2024-01-15T18:15:45.198412+0000","last_clean":"2024-01-15T18:15:45.198412+0000","last_became_active":"2024-01-15T18:15:45.192354+0000","last_became_peered":"2024-01-15T18:15:45.192354+0000","last_unstale":"2024-01-15T18:15:45.198412+0000","last_undegraded":"2024-01-15T18:15:45.198412+0000","last_fullsized":"2024-01-15T18:15:45.198412+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:12:58.057727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.172772+0000","last_change":"2024-01-15T18:15:44.168821+0000","last_active":"2024-01-15T18:15:44.172772+0000","last_peered":"2024-01-15T18:15:44.172772+0000","last_clean":"2024-01-15T18:15:44.172772+0000","last_became_active":"2024-01-15T18:15:44.168551+0000","last_became_peered":"2024-01-15T18:15:44.168551+0000","last_unstale":"2024-01-15T18:15:44.172772+0000","last_undegraded":"2024-01-15T18:15:44.172772+0000","last_fullsized":"2024-01-15T18:15:44.172772+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:14:11.832883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191678+0000","last_change":"2024-01-15T18:15:45.180405+0000","last_active":"2024-01-15T18:15:45.191678+0000","last_peered":"2024-01-15T18:15:45.191678+0000","last_clean":"2024-01-15T18:15:45.191678+0000","last_became_active":"2024-01-15T18:15:45.179918+0000","last_became_peered":"2024-01-15T18:15:45.179918+0000","last_unstale":"2024-01-15T18:15:45.191678+0000","last_undegraded":"2024-01-15T18:15:45.191678+0000","last_fullsized":"2024-01-15T18:15:45.191678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:01:49.697849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.192261+0000","last_change":"2024-01-15T18:15:44.159443+0000","last_active":"2024-01-15T18:15:45.192261+0000","last_peered":"2024-01-15T18:15:45.192261+0000","last_clean":"2024-01-15T18:15:45.192261+0000","last_became_active":"2024-01-15T18:15:44.159118+0000","last_became_peered":"2024-01-15T18:15:44.159118+0000","last_unstale":"2024-01-15T18:15:45.192261+0000","last_undegraded":"2024-01-15T18:15:45.192261+0000","last_fullsized":"2024-01-15T18:15:45.192261+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:13:11.055744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198173+0000","last_change":"2024-01-15T18:15:45.195167+0000","last_active":"2024-01-15T18:15:45.198173+0000","last_peered":"2024-01-15T18:15:45.198173+0000","last_clean":"2024-01-15T18:15:45.198173+0000","last_became_active":"2024-01-15T18:15:45.194897+0000","last_became_peered":"2024-01-15T18:15:45.194897+0000","last_unstale":"2024-01-15T18:15:45.198173+0000","last_undegraded":"2024-01-15T18:15:45.198173+0000","last_fullsized":"2024-01-15T18:15:45.198173+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:53:25.231875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.172853+0000","last_change":"2024-01-15T18:15:44.169724+0000","last_active":"2024-01-15T18:15:44.172853+0000","last_peered":"2024-01-15T18:15:44.172853+0000","last_clean":"2024-01-15T18:15:44.172853+0000","last_became_active":"2024-01-15T18:15:44.169420+0000","last_became_peered":"2024-01-15T18:15:44.169420+0000","last_unstale":"2024-01-15T18:15:44.172853+0000","last_undegraded":"2024-01-15T18:15:44.172853+0000","last_fullsized":"2024-01-15T18:15:44.172853+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:50:20.346298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.185177+0000","last_change":"2024-01-15T18:15:45.175568+0000","last_active":"2024-01-15T18:15:45.185177+0000","last_peered":"2024-01-15T18:15:45.185177+0000","last_clean":"2024-01-15T18:15:45.185177+0000","last_became_active":"2024-01-15T18:15:45.173399+0000","last_became_peered":"2024-01-15T18:15:45.173399+0000","last_unstale":"2024-01-15T18:15:45.185177+0000","last_undegraded":"2024-01-15T18:15:45.185177+0000","last_fullsized":"2024-01-15T18:15:45.185177+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:42:54.658117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.172758+0000","last_change":"2024-01-15T18:15:44.168665+0000","last_active":"2024-01-15T18:15:44.172758+0000","last_peered":"2024-01-15T18:15:44.172758+0000","last_clean":"2024-01-15T18:15:44.172758+0000","last_became_active":"2024-01-15T18:15:44.168375+0000","last_became_peered":"2024-01-15T18:15:44.168375+0000","last_unstale":"2024-01-15T18:15:44.172758+0000","last_undegraded":"2024-01-15T18:15:44.172758+0000","last_fullsized":"2024-01-15T18:15:44.172758+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:09:06.608452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191116+0000","last_change":"2024-01-15T18:15:45.179812+0000","last_active":"2024-01-15T18:15:45.191116+0000","last_peered":"2024-01-15T18:15:45.191116+0000","last_clean":"2024-01-15T18:15:45.191116+0000","last_became_active":"2024-01-15T18:15:45.179148+0000","last_became_peered":"2024-01-15T18:15:45.179148+0000","last_unstale":"2024-01-15T18:15:45.191116+0000","last_undegraded":"2024-01-15T18:15:45.191116+0000","last_fullsized":"2024-01-15T18:15:45.191116+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:47:52.953626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196980+0000","last_change":"2024-01-15T18:15:44.163128+0000","last_active":"2024-01-15T18:15:45.196980+0000","last_peered":"2024-01-15T18:15:45.196980+0000","last_clean":"2024-01-15T18:15:45.196980+0000","last_became_active":"2024-01-15T18:15:44.162839+0000","last_became_peered":"2024-01-15T18:15:44.162839+0000","last_unstale":"2024-01-15T18:15:45.196980+0000","last_undegraded":"2024-01-15T18:15:45.196980+0000","last_fullsized":"2024-01-15T18:15:45.196980+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:34:15.516676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.146439+0000","last_change":"2024-01-15T18:15:44.146439+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.146439+0000","last_undegraded":"2024-01-15T18:15:44.146439+0000","last_fullsized":"2024-01-15T18:15:44.146439+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.174831+0000","last_change":"2024-01-15T18:15:44.167859+0000","last_active":"2024-01-15T18:15:45.174831+0000","last_peered":"2024-01-15T18:15:45.174831+0000","last_clean":"2024-01-15T18:15:45.174831+0000","last_became_active":"2024-01-15T18:15:44.167592+0000","last_became_peered":"2024-01-15T18:15:44.167592+0000","last_unstale":"2024-01-15T18:15:45.174831+0000","last_undegraded":"2024-01-15T18:15:45.174831+0000","last_fullsized":"2024-01-15T18:15:45.174831+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:02:39.986413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.190664+0000","last_change":"2024-01-15T18:15:45.179572+0000","last_active":"2024-01-15T18:15:45.190664+0000","last_peered":"2024-01-15T18:15:45.190664+0000","last_clean":"2024-01-15T18:15:45.190664+0000","last_became_active":"2024-01-15T18:15:45.178802+0000","last_became_peered":"2024-01-15T18:15:45.178802+0000","last_unstale":"2024-01-15T18:15:45.190664+0000","last_undegraded":"2024-01-15T18:15:45.190664+0000","last_fullsized":"2024-01-15T18:15:45.190664+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:26:43.360415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.173615+0000","last_change":"2024-01-15T18:15:44.168997+0000","last_active":"2024-01-15T18:15:45.173615+0000","last_peered":"2024-01-15T18:15:45.173615+0000","last_clean":"2024-01-15T18:15:45.173615+0000","last_became_active":"2024-01-15T18:15:44.168678+0000","last_became_peered":"2024-01-15T18:15:44.168678+0000","last_unstale":"2024-01-15T18:15:45.173615+0000","last_undegraded":"2024-01-15T18:15:45.173615+0000","last_fullsized":"2024-01-15T18:15:45.173615+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:20:19.634573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.145752+0000","last_change":"2024-01-15T18:15:44.145752+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.145752+0000","last_undegraded":"2024-01-15T18:15:44.145752+0000","last_fullsized":"2024-01-15T18:15:44.145752+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.178682+0000","last_change":"2024-01-15T18:15:44.162077+0000","last_active":"2024-01-15T18:15:45.178682+0000","last_peered":"2024-01-15T18:15:45.178682+0000","last_clean":"2024-01-15T18:15:45.178682+0000","last_became_active":"2024-01-15T18:15:44.161794+0000","last_became_peered":"2024-01-15T18:15:44.161794+0000","last_unstale":"2024-01-15T18:15:45.178682+0000","last_undegraded":"2024-01-15T18:15:45.178682+0000","last_fullsized":"2024-01-15T18:15:45.178682+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:05:24.652003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.185502+0000","last_change":"2024-01-15T18:15:45.175862+0000","last_active":"2024-01-15T18:15:45.185502+0000","last_peered":"2024-01-15T18:15:45.185502+0000","last_clean":"2024-01-15T18:15:45.185502+0000","last_became_active":"2024-01-15T18:15:45.175569+0000","last_became_peered":"2024-01-15T18:15:45.175569+0000","last_unstale":"2024-01-15T18:15:45.185502+0000","last_undegraded":"2024-01-15T18:15:45.185502+0000","last_fullsized":"2024-01-15T18:15:45.185502+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:22:08.309890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.193674+0000","last_change":"2024-01-15T18:15:44.168092+0000","last_active":"2024-01-15T18:15:45.193674+0000","last_peered":"2024-01-15T18:15:45.193674+0000","last_clean":"2024-01-15T18:15:45.193674+0000","last_became_active":"2024-01-15T18:15:44.167828+0000","last_became_peered":"2024-01-15T18:15:44.167828+0000","last_unstale":"2024-01-15T18:15:45.193674+0000","last_undegraded":"2024-01-15T18:15:45.193674+0000","last_fullsized":"2024-01-15T18:15:45.193674+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:27:26.285133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198128+0000","last_change":"2024-01-15T18:15:45.194246+0000","last_active":"2024-01-15T18:15:45.198128+0000","last_peered":"2024-01-15T18:15:45.198128+0000","last_clean":"2024-01-15T18:15:45.198128+0000","last_became_active":"2024-01-15T18:15:45.193864+0000","last_became_peered":"2024-01-15T18:15:45.193864+0000","last_unstale":"2024-01-15T18:15:45.198128+0000","last_undegraded":"2024-01-15T18:15:45.198128+0000","last_fullsized":"2024-01-15T18:15:45.198128+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:05:25.657249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.162597+0000","last_change":"2024-01-15T18:15:44.166561+0000","last_active":"2024-01-15T18:15:45.162597+0000","last_peered":"2024-01-15T18:15:45.162597+0000","last_clean":"2024-01-15T18:15:45.162597+0000","last_became_active":"2024-01-15T18:15:44.166280+0000","last_became_peered":"2024-01-15T18:15:44.166280+0000","last_unstale":"2024-01-15T18:15:45.162597+0000","last_undegraded":"2024-01-15T18:15:45.162597+0000","last_fullsized":"2024-01-15T18:15:45.162597+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:30.074069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198142+0000","last_change":"2024-01-15T18:15:45.196412+0000","last_active":"2024-01-15T18:15:45.198142+0000","last_peered":"2024-01-15T18:15:45.198142+0000","last_clean":"2024-01-15T18:15:45.198142+0000","last_became_active":"2024-01-15T18:15:45.196022+0000","last_became_peered":"2024-01-15T18:15:45.196022+0000","last_unstale":"2024-01-15T18:15:45.198142+0000","last_undegraded":"2024-01-15T18:15:45.198142+0000","last_fullsized":"2024-01-15T18:15:45.198142+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:09:32.875279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.184853+0000","last_change":"2024-01-15T18:15:44.163120+0000","last_active":"2024-01-15T18:15:45.184853+0000","last_peered":"2024-01-15T18:15:45.184853+0000","last_clean":"2024-01-15T18:15:45.184853+0000","last_became_active":"2024-01-15T18:15:44.162805+0000","last_became_peered":"2024-01-15T18:15:44.162805+0000","last_unstale":"2024-01-15T18:15:45.184853+0000","last_undegraded":"2024-01-15T18:15:45.184853+0000","last_fullsized":"2024-01-15T18:15:45.184853+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:49.344761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191100+0000","last_change":"2024-01-15T18:15:45.185283+0000","last_active":"2024-01-15T18:15:45.191100+0000","last_peered":"2024-01-15T18:15:45.191100+0000","last_clean":"2024-01-15T18:15:45.191100+0000","last_became_active":"2024-01-15T18:15:45.185001+0000","last_became_peered":"2024-01-15T18:15:45.185001+0000","last_unstale":"2024-01-15T18:15:45.191100+0000","last_undegraded":"2024-01-15T18:15:45.191100+0000","last_fullsized":"2024-01-15T18:15:45.191100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:41:05.797910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.183836+0000","last_change":"2024-01-15T18:15:44.163840+0000","last_active":"2024-01-15T18:15:45.183836+0000","last_peered":"2024-01-15T18:15:45.183836+0000","last_clean":"2024-01-15T18:15:45.183836+0000","last_became_active":"2024-01-15T18:15:44.163441+0000","last_became_peered":"2024-01-15T18:15:44.163441+0000","last_unstale":"2024-01-15T18:15:45.183836+0000","last_undegraded":"2024-01-15T18:15:45.183836+0000","last_fullsized":"2024-01-15T18:15:45.183836+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:41:04.792664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.145185+0000","last_change":"2024-01-15T18:15:44.145185+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.145185+0000","last_undegraded":"2024-01-15T18:15:44.145185+0000","last_fullsized":"2024-01-15T18:15:44.145185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.172834+0000","last_change":"2024-01-15T18:15:44.169437+0000","last_active":"2024-01-15T18:15:44.172834+0000","last_peered":"2024-01-15T18:15:44.172834+0000","last_clean":"2024-01-15T18:15:44.172834+0000","last_became_active":"2024-01-15T18:15:44.169170+0000","last_became_peered":"2024-01-15T18:15:44.169170+0000","last_unstale":"2024-01-15T18:15:44.172834+0000","last_undegraded":"2024-01-15T18:15:44.172834+0000","last_fullsized":"2024-01-15T18:15:44.172834+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:50:41.854134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":1,"reported_epoch":82,"state":"creating+peering","last_fresh":"2024-01-15T18:15:44.156826+0000","last_change":"2024-01-15T18:15:44.156826+0000","last_active":"2024-01-15T18:15:44.136826+0000","last_peered":"2024-01-15T18:15:44.136826+0000","last_clean":"2024-01-15T18:15:44.136826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:15:44.156826+0000","last_undegraded":"2024-01-15T18:15:44.156826+0000","last_fullsized":"2024-01-15T18:15:44.156826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.164910+0000","last_change":"2024-01-15T18:15:44.154568+0000","last_active":"2024-01-15T18:15:45.164910+0000","last_peered":"2024-01-15T18:15:45.164910+0000","last_clean":"2024-01-15T18:15:45.164910+0000","last_became_active":"2024-01-15T18:15:44.154277+0000","last_became_peered":"2024-01-15T18:15:44.154277+0000","last_unstale":"2024-01-15T18:15:45.164910+0000","last_undegraded":"2024-01-15T18:15:45.164910+0000","last_fullsized":"2024-01-15T18:15:45.164910+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:39:47.546718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191500+0000","last_change":"2024-01-15T18:15:45.182784+0000","last_active":"2024-01-15T18:15:45.191500+0000","last_peered":"2024-01-15T18:15:45.191500+0000","last_clean":"2024-01-15T18:15:45.191500+0000","last_became_active":"2024-01-15T18:15:45.179168+0000","last_became_peered":"2024-01-15T18:15:45.179168+0000","last_unstale":"2024-01-15T18:15:45.191500+0000","last_undegraded":"2024-01-15T18:15:45.191500+0000","last_fullsized":"2024-01-15T18:15:45.191500+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:34:16.521921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.183666+0000","last_change":"2024-01-15T18:15:44.160891+0000","last_active":"2024-01-15T18:15:45.183666+0000","last_peered":"2024-01-15T18:15:45.183666+0000","last_clean":"2024-01-15T18:15:45.183666+0000","last_became_active":"2024-01-15T18:15:44.160625+0000","last_became_peered":"2024-01-15T18:15:44.160625+0000","last_unstale":"2024-01-15T18:15:45.183666+0000","last_undegraded":"2024-01-15T18:15:45.183666+0000","last_fullsized":"2024-01-15T18:15:45.183666+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:52.489949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196072+0000","last_change":"2024-01-15T18:15:45.192344+0000","last_active":"2024-01-15T18:15:45.196072+0000","last_peered":"2024-01-15T18:15:45.196072+0000","last_clean":"2024-01-15T18:15:45.196072+0000","last_became_active":"2024-01-15T18:15:45.191845+0000","last_became_peered":"2024-01-15T18:15:45.191845+0000","last_unstale":"2024-01-15T18:15:45.196072+0000","last_undegraded":"2024-01-15T18:15:45.196072+0000","last_fullsized":"2024-01-15T18:15:45.196072+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:01:31.532362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.163378+0000","last_change":"2024-01-15T18:15:44.156428+0000","last_active":"2024-01-15T18:15:45.163378+0000","last_peered":"2024-01-15T18:15:45.163378+0000","last_clean":"2024-01-15T18:15:45.163378+0000","last_became_active":"2024-01-15T18:15:44.156132+0000","last_became_peered":"2024-01-15T18:15:44.156132+0000","last_unstale":"2024-01-15T18:15:45.163378+0000","last_undegraded":"2024-01-15T18:15:45.163378+0000","last_fullsized":"2024-01-15T18:15:45.163378+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:49.344761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.186454+0000","last_change":"2024-01-15T18:15:45.178846+0000","last_active":"2024-01-15T18:15:45.186454+0000","last_peered":"2024-01-15T18:15:45.186454+0000","last_clean":"2024-01-15T18:15:45.186454+0000","last_became_active":"2024-01-15T18:15:45.178552+0000","last_became_peered":"2024-01-15T18:15:45.178552+0000","last_unstale":"2024-01-15T18:15:45.186454+0000","last_undegraded":"2024-01-15T18:15:45.186454+0000","last_fullsized":"2024-01-15T18:15:45.186454+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:45:31.736245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.183987+0000","last_change":"2024-01-15T18:15:44.170831+0000","last_active":"2024-01-15T18:15:45.183987+0000","last_peered":"2024-01-15T18:15:45.183987+0000","last_clean":"2024-01-15T18:15:45.183987+0000","last_became_active":"2024-01-15T18:15:44.170489+0000","last_became_peered":"2024-01-15T18:15:44.170489+0000","last_unstale":"2024-01-15T18:15:45.183987+0000","last_undegraded":"2024-01-15T18:15:45.183987+0000","last_fullsized":"2024-01-15T18:15:45.183987+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:54:03.053923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":218,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.162031+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:15:44.162031+0000","last_peered":"2024-01-15T18:15:44.162031+0000","last_clean":"2024-01-15T18:15:44.162031+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:15:44.162031+0000","last_undegraded":"2024-01-15T18:15:44.162031+0000","last_fullsized":"2024-01-15T18:15:44.162031+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":798,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509575,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77092,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294748,"statfs":{"total":96636764160,"available":96557821952,"internally_reserved":0,"allocated":1593344,"data_stored":1017889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77330890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":1.171,"15min":0.874},"min":{"1min":0.52400000000000002,"5min":0.38100000000000001,"15min":0.36199999999999999},"max":{"1min":1.0089999999999999,"5min":9.6989999999999998,"15min":10.284000000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.68799999999999994,"5min":1.1399999999999999,"15min":0.90000000000000002},"min":{"1min":0.48599999999999999,"5min":0.39100000000000001,"15min":0.378},"max":{"1min":0.83099999999999996,"5min":16.968,"15min":16.968},"last":6.6189999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":1.3129999999999999,"15min":0.91400000000000003},"min":{"1min":0.52600000000000002,"5min":0.36099999999999999,"15min":0.33800000000000002},"max":{"1min":0.90100000000000002,"5min":15.791,"15min":15.791},"last":4.944},{"interface":"front","average":{"1min":0.71099999999999997,"5min":1.109,"15min":0.90000000000000002},"min":{"1min":0.51500000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.85799999999999998,"5min":10.842000000000001,"15min":10.842000000000001},"last":4.6699999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":1.2609999999999999,"15min":0.92500000000000004},"min":{"1min":0.502,"5min":0.40400000000000003,"15min":0.378},"max":{"1min":0.98299999999999998,"5min":22.318000000000001,"15min":22.318000000000001},"last":4.4669999999999996},{"interface":"front","average":{"1min":0.76200000000000001,"5min":1.159,"15min":0.89400000000000002},"min":{"1min":0.58399999999999996,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.93700000000000006,"5min":9.7539999999999996,"15min":9.7539999999999996},"last":4.7030000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":1.387,"15min":0.98899999999999999},"min":{"1min":0.59799999999999998,"5min":0.47099999999999997,"15min":0.38100000000000001},"max":{"1min":0.93300000000000005,"5min":18.152999999999999,"15min":18.152999999999999},"last":1.456},{"interface":"front","average":{"1min":0.81599999999999995,"5min":1.274,"15min":0.98599999999999999},"min":{"1min":0.622,"5min":0.433,"15min":0.43099999999999999},"max":{"1min":0.96599999999999997,"5min":9.7959999999999994,"15min":10.34},"last":2.6909999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":1.2150000000000001,"15min":0.92700000000000005},"min":{"1min":0.44,"5min":0.314,"15min":0.314},"max":{"1min":0.85199999999999998,"5min":20.387,"15min":20.387},"last":4.4480000000000004},{"interface":"front","average":{"1min":0.623,"5min":1.353,"15min":0.98399999999999999},"min":{"1min":0.47099999999999997,"5min":0.38800000000000001,"15min":0.36099999999999999},"max":{"1min":1.032,"5min":17.215,"15min":17.215},"last":3.6309999999999998}]},{"osd":5,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":1.2410000000000001,"15min":0.94499999999999995},"min":{"1min":0.46400000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.95299999999999996,"5min":9.8450000000000006,"15min":19.959},"last":4.8639999999999999},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1.278,"15min":0.97699999999999998},"min":{"1min":0.439,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83999999999999997,"5min":19.905000000000001,"15min":19.905000000000001},"last":4.3959999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":1.4470000000000001,"15min":1.1499999999999999},"min":{"1min":0.59499999999999997,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97099999999999997,"5min":20.492000000000001,"15min":22.478999999999999},"last":4.3330000000000002},{"interface":"front","average":{"1min":0.749,"5min":1.272,"15min":1.0600000000000001},"min":{"1min":0.52200000000000002,"5min":0.44,"15min":0.38300000000000001},"max":{"1min":0.998,"5min":17.132000000000001,"15min":21.509},"last":3.573}]}]},{"osd":6,"up_from":15,"seq":64424509575,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58524,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313316,"statfs":{"total":96636764160,"available":96576835584,"internally_reserved":0,"allocated":999424,"data_stored":427521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":1.3009999999999999,"15min":0.95699999999999996},"min":{"1min":0.55700000000000005,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.96099999999999997,"5min":32.587000000000003,"15min":32.587000000000003},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.75,"5min":1.0660000000000001,"15min":0.86199999999999999},"min":{"1min":0.49199999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.97799999999999998,"5min":16.193000000000001,"15min":16.193000000000001},"last":0.69399999999999995}]},{"osd":1,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.92400000000000004,"15min":0.98699999999999999},"min":{"1min":0.439,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.90700000000000003,"5min":7.7649999999999997,"15min":53.451999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.72499999999999998,"5min":1.2090000000000001,"15min":0.92600000000000005},"min":{"1min":0.53400000000000003,"5min":0.437,"15min":0.42899999999999999},"max":{"1min":0.86299999999999999,"5min":32.512,"15min":32.512},"last":0.51700000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":1.24,"15min":0.96799999999999997},"min":{"1min":0.58399999999999996,"5min":0.41699999999999998,"15min":0.38900000000000001},"max":{"1min":0.92200000000000004,"5min":32.472999999999999,"15min":32.472999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.754,"5min":1.0649999999999999,"15min":0.88500000000000001},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.38500000000000001},"max":{"1min":1.02,"5min":16.602,"15min":16.602},"last":0.65300000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":1.1699999999999999,"15min":0.93600000000000005},"min":{"1min":0.56399999999999995,"5min":0.46000000000000002,"15min":0.41399999999999998},"max":{"1min":0.92100000000000004,"5min":16.388000000000002,"15min":16.388000000000002},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.0680000000000001,"15min":1.075},"min":{"1min":0.54400000000000004,"5min":0.432,"15min":0.41599999999999998},"max":{"1min":0.95999999999999996,"5min":9.3719999999999999,"15min":53.398000000000003},"last":0.67200000000000004}]},{"osd":4,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":1.149,"15min":0.89500000000000002},"min":{"1min":0.435,"5min":0.38100000000000001,"15min":0.34999999999999998},"max":{"1min":0.79700000000000004,"5min":32.549999999999997,"15min":32.549999999999997},"last":0.63},{"interface":"front","average":{"1min":0.59499999999999997,"5min":1.0409999999999999,"15min":0.84899999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.34499999999999997},"max":{"1min":0.96499999999999997,"5min":11.852,"15min":11.852},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":1.0720000000000001,"15min":1.034},"min":{"1min":0.42699999999999999,"5min":0.35799999999999998,"15min":0.30399999999999999},"max":{"1min":0.77300000000000002,"5min":13.087999999999999,"15min":53.229999999999997},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.68700000000000006,"5min":1.375,"15min":1.0349999999999999},"min":{"1min":0.42599999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.0009999999999999,"5min":32.404000000000003,"15min":32.404000000000003},"last":0.71899999999999997}]},{"osd":7,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":1.294,"15min":1.1200000000000001},"min":{"1min":0.52900000000000003,"5min":0.436,"15min":0.40400000000000003},"max":{"1min":0.94099999999999995,"5min":23.991,"15min":45.070999999999998},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":1.3260000000000001,"15min":1.2170000000000001},"min":{"1min":0.51200000000000001,"5min":0.40999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88200000000000001,"5min":40.972999999999999,"15min":53.338000000000001},"last":0.753}]}]},{"osd":4,"up_from":15,"seq":64424509575,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58528,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313312,"statfs":{"total":96636764160,"available":96576831488,"internally_reserved":0,"allocated":999424,"data_stored":427521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":1.111,"15min":0.88500000000000001},"min":{"1min":0.46000000000000002,"5min":0.437,"15min":0.32900000000000001},"max":{"1min":0.96199999999999997,"5min":13.492000000000001,"15min":23.318000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.92100000000000004,"15min":0.80200000000000005},"min":{"1min":0.41999999999999998,"5min":0.378,"15min":0.32600000000000001},"max":{"1min":0.875,"5min":14.723000000000001,"15min":14.723000000000001},"last":0.71099999999999997}]},{"osd":1,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":1.0049999999999999,"15min":0.78800000000000003},"min":{"1min":0.50800000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.84299999999999997,"5min":9.5700000000000003,"15min":9.5700000000000003},"last":0.63},{"interface":"front","average":{"1min":0.71999999999999997,"5min":1.155,"15min":0.91400000000000003},"min":{"1min":0.47399999999999998,"5min":0.40200000000000002,"15min":0.36899999999999999},"max":{"1min":0.84999999999999998,"5min":13.57,"15min":22.408999999999999},"last":0.751}]},{"osd":2,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":1.038,"15min":0.79400000000000004},"min":{"1min":0.53000000000000003,"5min":0.41599999999999998,"15min":0.34399999999999997},"max":{"1min":0.84199999999999997,"5min":9.7089999999999996,"15min":9.7089999999999996},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.73499999999999999,"5min":1.2949999999999999,"15min":1.014},"min":{"1min":0.439,"5min":0.439,"15min":0.39100000000000001},"max":{"1min":0.92600000000000005,"5min":14.118,"15min":22.338999999999999},"last":0.69999999999999996}]},{"osd":3,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":1.016,"15min":0.873},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.41199999999999998},"max":{"1min":0.83199999999999996,"5min":14.763999999999999,"15min":14.763999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.77300000000000002,"5min":1.1479999999999999,"15min":0.90500000000000003},"min":{"1min":0.622,"5min":0.48099999999999998,"15min":0.45500000000000002},"max":{"1min":0.89400000000000002,"5min":9.6289999999999996,"15min":9.6289999999999996},"last":0.72999999999999998}]},{"osd":5,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":1.1299999999999999,"15min":0.96299999999999997},"min":{"1min":0.39000000000000001,"5min":0.34799999999999998,"15min":0.33200000000000002},"max":{"1min":0.875,"5min":15.787000000000001,"15min":31.617999999999999},"last":0.501},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.94699999999999995,"15min":0.86699999999999999},"min":{"1min":0.44,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.92300000000000004,"5min":16.388000000000002,"15min":17.145},"last":0.59899999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.98399999999999999,"15min":0.83099999999999996},"min":{"1min":0.42799999999999999,"5min":0.37,"15min":0.35499999999999998},"max":{"1min":0.94599999999999995,"5min":11.124000000000001,"15min":11.124000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.71699999999999997,"5min":1.3580000000000001,"15min":1.226},"min":{"1min":0.53700000000000003,"5min":0.434,"15min":0.40200000000000002},"max":{"1min":1.018,"5min":32.308,"15min":32.308},"last":0.55500000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":1.002,"15min":1.0109999999999999},"min":{"1min":0.49399999999999999,"5min":0.41099999999999998,"15min":0.34799999999999998},"max":{"1min":0.89900000000000002,"5min":14.653,"15min":39.256},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.746,"5min":1.2290000000000001,"15min":1.0129999999999999},"min":{"1min":0.57799999999999996,"5min":0.46500000000000002,"15min":0.42199999999999999},"max":{"1min":0.98599999999999999,"5min":16.277000000000001,"15min":26.573},"last":0.57399999999999995}]}]},{"osd":3,"up_from":15,"seq":64424509574,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80676,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94291164,"statfs":{"total":96636764160,"available":96554151936,"internally_reserved":0,"allocated":999424,"data_stored":427521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":81590727},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.59199999999999997,"15min":0.49099999999999999},"min":{"1min":0.29299999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.751,"5min":5.8849999999999998,"15min":5.8849999999999998},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.68999999999999995,"15min":0.61599999999999999},"min":{"1min":0.38600000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.77500000000000002,"5min":4.7990000000000004,"15min":17.756},"last":0.29799999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.73099999999999998,"15min":0.59299999999999997},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.621,"5min":12.307,"15min":12.307},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.69399999999999995,"15min":0.59299999999999997},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.32200000000000001},"max":{"1min":0.83799999999999997,"5min":6.0640000000000001,"15min":6.0640000000000001},"last":0.38600000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.83099999999999996,"15min":0.69599999999999995},"min":{"1min":0.46800000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":3.633,"5min":12.468999999999999,"15min":12.468999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.85399999999999998,"15min":0.68500000000000005},"min":{"1min":0.48599999999999999,"5min":0.41599999999999998,"15min":0.34599999999999997},"max":{"1min":1.0129999999999999,"5min":10.837999999999999,"15min":10.837999999999999},"last":0.47199999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":1.292,"15min":1.028},"min":{"1min":0.59499999999999997,"5min":0.375,"15min":0.375},"max":{"1min":1.014,"5min":13.863,"15min":17.853999999999999},"last":0.874},{"interface":"front","average":{"1min":0.82899999999999996,"5min":1.2809999999999999,"15min":0.96899999999999997},"min":{"1min":0.63,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.0469999999999999,"5min":13.31,"15min":13.31},"last":0.72099999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":1.369,"15min":0.97599999999999998},"min":{"1min":0.66500000000000004,"5min":0.52100000000000002,"15min":0.51000000000000001},"max":{"1min":0.94399999999999995,"5min":17.687999999999999,"15min":17.687999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.206,"15min":0.97399999999999998},"min":{"1min":0.51700000000000002,"5min":0.438,"15min":0.438},"max":{"1min":3.2160000000000002,"5min":20.515999999999998,"15min":20.515999999999998},"last":0.82099999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":1.1499999999999999,"15min":0.98999999999999999},"min":{"1min":0.55600000000000005,"5min":0.432,"15min":0.432},"max":{"1min":1.1060000000000001,"5min":13.250999999999999,"15min":23.100999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.82099999999999995,"5min":1.2150000000000001,"15min":0.93799999999999994},"min":{"1min":0.60799999999999998,"5min":0.5,"15min":0.49399999999999999},"max":{"1min":1.0169999999999999,"5min":12.76,"15min":12.76},"last":0.93400000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.7889999999999999,"15min":1.1719999999999999},"min":{"1min":0.625,"5min":0.48599999999999999,"15min":0.48199999999999998},"max":{"1min":0.99099999999999999,"5min":24.321999999999999,"15min":24.321999999999999},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.81999999999999995,"5min":1.329,"15min":1.028},"min":{"1min":0.58399999999999996,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0569999999999999,"5min":13.385,"15min":13.385},"last":0.78100000000000003}]}]},{"osd":2,"up_from":15,"seq":64424509574,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76532,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295308,"statfs":{"total":96636764160,"available":96558395392,"internally_reserved":0,"allocated":1015808,"data_stored":434962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.51900000000000002,"15min":0.51200000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26500000000000001},"max":{"1min":0.66500000000000004,"5min":1.857,"15min":5.1849999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54200000000000004,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.252,"15min":0.252},"max":{"1min":0.89300000000000002,"5min":2.3759999999999999,"15min":7.516},"last":0.55900000000000005}]},{"osd":1,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.70599999999999996,"15min":0.61599999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.875,"5min":10.907,"15min":10.907},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.68500000000000005,"15min":0.58799999999999997},"min":{"1min":0.38100000000000001,"5min":0.28100000000000003,"15min":0.27000000000000002},"max":{"1min":0.73799999999999999,"5min":7.2119999999999997,"15min":8.9190000000000005},"last":0.53500000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.60599999999999998,"15min":0.626},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.72999999999999998,"5min":2.7290000000000001,"15min":12.185},"last":0.621},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.747,"15min":0.66100000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.34000000000000002},"max":{"1min":0.96699999999999997,"5min":7.6760000000000002,"15min":7.6760000000000002},"last":0.67900000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.89500000000000002,"15min":0.85799999999999998},"min":{"1min":0.55700000000000005,"5min":0.47499999999999998,"15min":0.45800000000000002},"max":{"1min":1.1870000000000001,"5min":4.3150000000000004,"15min":8.6609999999999996},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.755,"5min":0.878,"15min":0.90900000000000003},"min":{"1min":0.5,"5min":0.45800000000000002,"15min":0.42499999999999999},"max":{"1min":1.075,"5min":8.3190000000000008,"15min":19.146999999999998},"last":0.84199999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.80400000000000005,"15min":0.90200000000000002},"min":{"1min":0.52400000000000002,"5min":0.442,"15min":0.44},"max":{"1min":1.1599999999999999,"5min":2.7789999999999999,"15min":31.710000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.996,"15min":1},"min":{"1min":0.54400000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.155,"5min":10.460000000000001,"15min":22.539000000000001},"last":0.77300000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.99399999999999999,"15min":0.89700000000000002},"min":{"1min":0.501,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.1220000000000001,"5min":8.3580000000000005,"15min":9.4540000000000006},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.83799999999999997,"15min":0.90500000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0489999999999999,"5min":2.669,"15min":18.614999999999998},"last":0.88}]},{"osd":7,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.125,"15min":0.95399999999999996},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.47999999999999998},"max":{"1min":1.157,"5min":11.606,"15min":11.606},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.92200000000000004,"15min":0.98699999999999999},"min":{"1min":0.60399999999999998,"5min":0.51700000000000002,"15min":0.502},"max":{"1min":1.1180000000000001,"5min":3.6850000000000001,"15min":17.013000000000002},"last":0.86199999999999999}]}]},{"osd":1,"up_from":15,"seq":64424509576,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80700,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94291140,"statfs":{"total":96636764160,"available":96554127360,"internally_reserved":0,"allocated":1024000,"data_stored":434574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81590725},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.70699999999999996,"15min":0.66300000000000003},"min":{"1min":0.26300000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.745,"5min":17.396999999999998,"15min":19.577000000000002},"last":0.623},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.74099999999999999,"15min":0.76000000000000001},"min":{"1min":0.33900000000000002,"5min":0.29199999999999998,"15min":0.251},"max":{"1min":0.83999999999999997,"5min":12.378,"15min":48.277999999999999},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.61399999999999999,"15min":0.71399999999999997},"min":{"1min":0.33600000000000002,"5min":0.309,"15min":0.30299999999999999},"max":{"1min":0.82799999999999996,"5min":2.8610000000000002,"15min":40.698},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.65900000000000003,"15min":0.90100000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.317},"max":{"1min":0.89500000000000002,"5min":3.0449999999999999,"15min":48.316000000000003},"last":0.441}]},{"osd":3,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.71799999999999997,"15min":0.84499999999999997},"min":{"1min":0.434,"5min":0.39800000000000002,"15min":0.33700000000000002},"max":{"1min":1.2509999999999999,"5min":3.0899999999999999,"15min":48.234999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.83799999999999997,"15min":0.68999999999999995},"min":{"1min":0.315,"5min":0.315,"15min":0.311},"max":{"1min":4.0780000000000003,"5min":9.9719999999999995,"15min":9.9719999999999995},"last":0.46000000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":1.0980000000000001,"15min":0.88700000000000001},"min":{"1min":0.49299999999999999,"5min":0.48999999999999999,"15min":0.44500000000000001},"max":{"1min":1.446,"5min":9.9329999999999998,"15min":9.9329999999999998},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.80800000000000005,"5min":1.026,"15min":0.93600000000000005},"min":{"1min":0.47499999999999998,"5min":0.45800000000000002,"15min":0.42499999999999999},"max":{"1min":1.163,"5min":9.8610000000000007,"15min":19.829000000000001},"last":0.88300000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.95399999999999996,"15min":1.0169999999999999},"min":{"1min":0.502,"5min":0.48599999999999999,"15min":0.46100000000000002},"max":{"1min":1.425,"5min":8.5839999999999996,"15min":22.594999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.82099999999999995,"5min":1.2629999999999999,"15min":1.163},"min":{"1min":0.58499999999999996,"5min":0.443,"15min":0.443},"max":{"1min":1.109,"5min":26.748999999999999,"15min":48.198},"last":0.76900000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":1.125,"15min":1.046},"min":{"1min":0.52200000000000002,"5min":0.50600000000000001,"15min":0.47499999999999998},"max":{"1min":1.879,"5min":9.6349999999999998,"15min":35.112000000000002},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.999,"15min":0.97599999999999998},"min":{"1min":0.58599999999999997,"5min":0.53100000000000003,"15min":0.50800000000000001},"max":{"1min":1.77,"5min":5.8849999999999998,"15min":19.780999999999999},"last":0.95199999999999996}]},{"osd":7,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3340000000000001,"15min":1.169},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.97499999999999998,"5min":20.872,"15min":48.148000000000003},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.85899999999999999,"5min":1.246,"15min":0.996},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.52100000000000002},"max":{"1min":1.52,"5min":9.8640000000000008,"15min":9.8640000000000008},"last":0.873}]}]},{"osd":5,"up_from":14,"seq":60129542279,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54352,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317488,"statfs":{"total":96636764160,"available":96581107712,"internally_reserved":0,"allocated":987136,"data_stored":421162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.86699999999999999,"15min":0.80900000000000005},"min":{"1min":0.51800000000000002,"5min":0.29899999999999999,"15min":0.28999999999999998},"max":{"1min":3.8410000000000002,"5min":5.1639999999999997,"15min":21.385000000000002},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.0069999999999999,"15min":0.89600000000000002},"min":{"1min":0.57099999999999995,"5min":0.41599999999999998,"15min":0.34899999999999998},"max":{"1min":3.1080000000000001,"5min":9.359,"15min":9.359},"last":0.71099999999999997}]},{"osd":1,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":1.3120000000000001,"15min":0.89600000000000002},"min":{"1min":0.55400000000000005,"5min":0.435,"15min":0.34899999999999998},"max":{"1min":1.0469999999999999,"5min":21.120000000000001,"15min":21.120000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.2529999999999999,"15min":0.96499999999999997},"min":{"1min":0.628,"5min":0.38600000000000001,"15min":0.315},"max":{"1min":6.1310000000000002,"5min":15.843,"15min":21.5},"last":0.52900000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.98699999999999999,"15min":0.89800000000000002},"min":{"1min":0.51400000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":3.9220000000000002,"5min":6.2750000000000004,"15min":8.7750000000000004},"last":0.54500000000000004},{"interface":"front","average":{"1min":1.135,"5min":1.3640000000000001,"15min":0.998},"min":{"1min":0.50800000000000001,"5min":0.42499999999999999,"15min":0.40400000000000003},"max":{"1min":8.3529999999999998,"5min":21.201000000000001,"15min":21.201000000000001},"last":0.69499999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":0.98599999999999999,"15min":0.92200000000000004},"min":{"1min":0.60499999999999998,"5min":0.37,"15min":0.37},"max":{"1min":7.7199999999999998,"5min":7.7199999999999998,"15min":21.457000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.90300000000000002,"5min":1.034,"15min":0.97899999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.45400000000000001},"max":{"1min":3.8239999999999998,"5min":6.8650000000000002,"15min":8.7300000000000004},"last":0.47699999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":1.0740000000000001,"15min":0.88800000000000001},"min":{"1min":0.51800000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.99199999999999999,"5min":10.962,"15min":10.962},"last":0.626},{"interface":"front","average":{"1min":0.69299999999999995,"5min":1.4410000000000001,"15min":1.034},"min":{"1min":0.48399999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0189999999999999,"5min":21.724,"15min":26.146999999999998},"last":0.51300000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.83899999999999997},"min":{"1min":0.44500000000000001,"5min":0.36099999999999999,"15min":0.32000000000000001},"max":{"1min":0.95599999999999996,"5min":5.2160000000000002,"15min":21.535},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.79000000000000004,"5min":1.012,"15min":1.0129999999999999},"min":{"1min":0.52400000000000002,"5min":0.379,"15min":0.36299999999999999},"max":{"1min":1.03,"5min":5.7539999999999996,"15min":30.488},"last":0.67800000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":2.0070000000000001,"15min":1.216},"min":{"1min":0.47699999999999998,"5min":0.42599999999999999,"15min":0.39700000000000002},"max":{"1min":1.008,"5min":33.499000000000002,"15min":33.499000000000002},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.81000000000000005,"5min":1.502,"15min":1.0880000000000001},"min":{"1min":0.54300000000000004,"5min":0.42899999999999999,"15min":0.36499999999999999},"max":{"1min":1.0609999999999999,"5min":25.437999999999999,"15min":25.437999999999999},"last":0.64300000000000002}]}]},{"osd":0,"up_from":14,"seq":60129542279,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77092,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294748,"statfs":{"total":96636764160,"available":96557821952,"internally_reserved":0,"allocated":1593344,"data_stored":1017889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.90400000000000003,"15min":0.63},"min":{"1min":0.245,"5min":0.245,"15min":0.23699999999999999},"max":{"1min":0.78600000000000003,"5min":19.843,"15min":19.843},"last":0.37},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.94699999999999995,"15min":0.68000000000000005},"min":{"1min":0.29999999999999999,"5min":0.29199999999999998,"15min":0.27300000000000002},"max":{"1min":0.746,"5min":10.154,"15min":10.154},"last":0.39200000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":1.2729999999999999,"15min":0.80400000000000005},"min":{"1min":0.39500000000000002,"5min":0.34899999999999998,"15min":0.30399999999999999},"max":{"1min":0.84099999999999997,"5min":25.905000000000001,"15min":25.905000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.82699999999999996,"15min":0.61199999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.26700000000000002},"max":{"1min":0.80000000000000004,"5min":9.3829999999999991,"15min":9.3829999999999991},"last":0.51900000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":1.026,"15min":0.74199999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.80400000000000005,"5min":9.1780000000000008,"15min":9.1780000000000008},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":1.016,"15min":0.76700000000000002},"min":{"1min":0.41699999999999998,"5min":0.40000000000000002,"15min":0.36899999999999999},"max":{"1min":0.89900000000000002,"5min":10.210000000000001,"15min":10.210000000000001},"last":0.44500000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":1.1319999999999999,"15min":0.98799999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.38},"max":{"1min":1.1459999999999999,"5min":10.691000000000001,"15min":22.814},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.80400000000000005,"5min":1.395,"15min":1.042},"min":{"1min":0.50600000000000001,"5min":0.49099999999999999,"15min":0.41199999999999998},"max":{"1min":1.046,"5min":23.341999999999999,"15min":23.341999999999999},"last":0.66000000000000003}]},{"osd":5,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":1.141,"15min":0.96899999999999997},"min":{"1min":0.55700000000000005,"5min":0.46899999999999997,"15min":0.441},"max":{"1min":1.046,"5min":10.737,"15min":12.428000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.83799999999999997,"5min":1.962,"15min":1.27},"min":{"1min":0.56999999999999995,"5min":0.498,"15min":0.46300000000000002},"max":{"1min":1.3049999999999999,"5min":23.021000000000001,"15min":23.021000000000001},"last":0.68999999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":1.696,"15min":1.218},"min":{"1min":0.52700000000000002,"5min":0.46700000000000003,"15min":0.46200000000000002},"max":{"1min":1.0229999999999999,"5min":26.036000000000001,"15min":31.059000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.88800000000000001,"5min":1.083,"15min":1.0920000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.49099999999999999},"max":{"1min":1.931,"5min":6.9690000000000003,"15min":40.731000000000002},"last":0.70299999999999996}]},{"osd":7,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.8400000000000001,"15min":1.208},"min":{"1min":0.58999999999999997,"5min":0.54900000000000004,"15min":0.42499999999999999},"max":{"1min":4.1609999999999996,"5min":24.492999999999999,"15min":24.492999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.83499999999999996,"5min":1.242,"15min":1.1140000000000001},"min":{"1min":0.64500000000000002,"5min":0.58399999999999996,"15min":0.46600000000000003},"max":{"1min":1.008,"5min":9.2789999999999999,"15min":23.524000000000001},"last":0.72899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:15:50.078 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3a is not active+clean 2024-01-15T18:15:50.078 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:15:44.145723+0000', 'last_change': '2024-01-15T18:15:44.145723+0000', 'last_active': '2024-01-15T18:15:44.136826+0000', 'last_peered': '2024-01-15T18:15:44.136826+0000', 'last_clean': '2024-01-15T18:15:44.136826+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:15:44.145723+0000', 'last_undegraded': '2024-01-15T18:15:44.145723+0000', 'last_fullsized': '2024-01-15T18:15:44.145723+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'last_clean_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T18:15:50.078 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.38 is not active+clean 2024-01-15T18:15:50.078 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:15:44.144794+0000', 'last_change': '2024-01-15T18:15:44.144794+0000', 'last_active': '2024-01-15T18:15:44.136826+0000', 'last_peered': '2024-01-15T18:15:44.136826+0000', 'last_clean': '2024-01-15T18:15:44.136826+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:15:44.144794+0000', 'last_undegraded': '2024-01-15T18:15:44.144794+0000', 'last_fullsized': '2024-01-15T18:15:44.144794+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'last_clean_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T18:15:50.078 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.37 is not active+clean 2024-01-15T18:15:50.078 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:15:44.145532+0000', 'last_change': '2024-01-15T18:15:44.145532+0000', 'last_active': '2024-01-15T18:15:44.136826+0000', 'last_peered': '2024-01-15T18:15:44.136826+0000', 'last_clean': '2024-01-15T18:15:44.136826+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:15:44.145532+0000', 'last_undegraded': '2024-01-15T18:15:44.145532+0000', 'last_fullsized': '2024-01-15T18:15:44.145532+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'last_clean_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T18:15:50.078 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2d is not active+clean 2024-01-15T18:15:50.078 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:15:44.158290+0000', 'last_change': '2024-01-15T18:15:44.158290+0000', 'last_active': '2024-01-15T18:15:44.136826+0000', 'last_peered': '2024-01-15T18:15:44.136826+0000', 'last_clean': '2024-01-15T18:15:44.136826+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:15:44.158290+0000', 'last_undegraded': '2024-01-15T18:15:44.158290+0000', 'last_fullsized': '2024-01-15T18:15:44.158290+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'last_clean_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:15:50.078 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.17 is not active+clean 2024-01-15T18:15:50.078 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:15:44.144921+0000', 'last_change': '2024-01-15T18:15:44.144921+0000', 'last_active': '2024-01-15T18:15:44.136826+0000', 'last_peered': '2024-01-15T18:15:44.136826+0000', 'last_clean': '2024-01-15T18:15:44.136826+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:15:44.144921+0000', 'last_undegraded': '2024-01-15T18:15:44.144921+0000', 'last_fullsized': '2024-01-15T18:15:44.144921+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'last_clean_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T18:15:50.079 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.14 is not active+clean 2024-01-15T18:15:50.079 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:15:44.155839+0000', 'last_change': '2024-01-15T18:15:44.155839+0000', 'last_active': '2024-01-15T18:15:44.136826+0000', 'last_peered': '2024-01-15T18:15:44.136826+0000', 'last_clean': '2024-01-15T18:15:44.136826+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:15:44.155839+0000', 'last_undegraded': '2024-01-15T18:15:44.155839+0000', 'last_fullsized': '2024-01-15T18:15:44.155839+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'last_clean_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:15:50.079 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.12 is not active+clean 2024-01-15T18:15:50.079 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:15:44.153863+0000', 'last_change': '2024-01-15T18:15:44.153863+0000', 'last_active': '2024-01-15T18:15:44.136826+0000', 'last_peered': '2024-01-15T18:15:44.136826+0000', 'last_clean': '2024-01-15T18:15:44.136826+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:15:44.153863+0000', 'last_undegraded': '2024-01-15T18:15:44.153863+0000', 'last_fullsized': '2024-01-15T18:15:44.153863+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'last_clean_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:15:50.079 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.b is not active+clean 2024-01-15T18:15:50.079 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:15:44.146439+0000', 'last_change': '2024-01-15T18:15:44.146439+0000', 'last_active': '2024-01-15T18:15:44.136826+0000', 'last_peered': '2024-01-15T18:15:44.136826+0000', 'last_clean': '2024-01-15T18:15:44.136826+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:15:44.146439+0000', 'last_undegraded': '2024-01-15T18:15:44.146439+0000', 'last_fullsized': '2024-01-15T18:15:44.146439+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'last_clean_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T18:15:50.079 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.9 is not active+clean 2024-01-15T18:15:50.079 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:15:44.145752+0000', 'last_change': '2024-01-15T18:15:44.145752+0000', 'last_active': '2024-01-15T18:15:44.136826+0000', 'last_peered': '2024-01-15T18:15:44.136826+0000', 'last_clean': '2024-01-15T18:15:44.136826+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:15:44.145752+0000', 'last_undegraded': '2024-01-15T18:15:44.145752+0000', 'last_fullsized': '2024-01-15T18:15:44.145752+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'last_clean_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T18:15:50.079 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2 is not active+clean 2024-01-15T18:15:50.079 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:15:44.145185+0000', 'last_change': '2024-01-15T18:15:44.145185+0000', 'last_active': '2024-01-15T18:15:44.136826+0000', 'last_peered': '2024-01-15T18:15:44.136826+0000', 'last_clean': '2024-01-15T18:15:44.136826+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:15:44.145185+0000', 'last_undegraded': '2024-01-15T18:15:44.145185+0000', 'last_fullsized': '2024-01-15T18:15:44.145185+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'last_clean_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T18:15:50.079 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3 is not active+clean 2024-01-15T18:15:50.080 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 82, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:15:44.156826+0000', 'last_change': '2024-01-15T18:15:44.156826+0000', 'last_active': '2024-01-15T18:15:44.136826+0000', 'last_peered': '2024-01-15T18:15:44.136826+0000', 'last_clean': '2024-01-15T18:15:44.136826+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:15:44.156826+0000', 'last_undegraded': '2024-01-15T18:15:44.156826+0000', 'last_fullsized': '2024-01-15T18:15:44.156826+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'last_clean_scrub_stamp': '2024-01-15T18:15:44.136826+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:15:50.080 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-15T18:15:50.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.256+0000 7f2977c00640 1 Processor -- start 2024-01-15T18:15:50.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.256+0000 7f2977c00640 1 -- start start 2024-01-15T18:15:50.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.260+0000 7f2977c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f297015d2c0 0x7f29700c62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:50.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.260+0000 7f2977c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29700c5d00 con 0x7f297006ac00 2024-01-15T18:15:50.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.260+0000 7f2977c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f297006fc90 con 0x7f29700c6800 2024-01-15T18:15:50.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.260+0000 7f2977c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29700ce600 con 0x7f297015d2c0 2024-01-15T18:15:50.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.260+0000 7f2975c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f297006ac00 0x7f297006d390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46162/0 (socket says 172.21.15.27:46162) 2024-01-15T18:15:50.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.260+0000 7f2975c00640 1 -- 172.21.15.27:0/2417422920 learned_addr learned my addr 172.21.15.27:0/2417422920 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:50.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.260+0000 7f2975200640 1 --2- 172.21.15.27:0/2417422920 >> v2:172.21.15.93:3300/0 conn(0x7f297015d2c0 0x7f29700c62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:50.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.260+0000 7f296f400640 1 -- 172.21.15.27:0/2417422920 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122488089 0 0) 0x7f29700c5d00 con 0x7f297006ac00 2024-01-15T18:15:50.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.260+0000 7f296f400640 1 -- 172.21.15.27:0/2417422920 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2950003530 con 0x7f297006ac00 2024-01-15T18:15:50.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.260+0000 7f2975200640 1 -- 172.21.15.27:0/2417422920 >> v1:172.21.15.93:6789/0 conn(0x7f29700c6800 legacy=0x7f297006a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:50.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.260+0000 7f296f400640 1 -- 172.21.15.27:0/2417422920 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197409782 0 0) 0x7f297006fc90 con 0x7f29700c6800 2024-01-15T18:15:50.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.260+0000 7f2975200640 1 -- 172.21.15.27:0/2417422920 >> v1:172.21.15.27:6789/0 conn(0x7f297006ac00 legacy=0x7f297006d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:50.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.260+0000 7f2975200640 1 -- 172.21.15.27:0/2417422920 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29700c50f0 con 0x7f297015d2c0 2024-01-15T18:15:50.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.264+0000 7f2975200640 1 --2- 172.21.15.27:0/2417422920 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f297015d2c0 0x7f29700c62c0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f29600097c0 tx=0x7f296001c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7676997cef0cbca server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:50.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.264+0000 7f296f400640 1 -- 172.21.15.27:0/2417422920 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29600ab650 con 0x7f297015d2c0 2024-01-15T18:15:50.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.264+0000 7f296f400640 1 -- 172.21.15.27:0/2417422920 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29600ab7b0 con 0x7f297015d2c0 2024-01-15T18:15:50.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.264+0000 7f296f400640 1 -- 172.21.15.27:0/2417422920 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29600abb20 con 0x7f297015d2c0 2024-01-15T18:15:50.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.264+0000 7f2977c00640 1 -- 172.21.15.27:0/2417422920 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f297015d2c0 msgr2=0x7f29700c62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:50.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.264+0000 7f2977c00640 1 --2- 172.21.15.27:0/2417422920 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f297015d2c0 0x7f29700c62c0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f29600097c0 tx=0x7f296001c450 comp rx=0 tx=0).stop 2024-01-15T18:15:50.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.264+0000 7f2977c00640 1 -- 172.21.15.27:0/2417422920 shutdown_connections 2024-01-15T18:15:50.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.264+0000 7f2977c00640 1 --2- 172.21.15.27:0/2417422920 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f297015d2c0 0x7f29700c62c0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:50.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.264+0000 7f2977c00640 1 -- 172.21.15.27:0/2417422920 >> 172.21.15.27:0/2417422920 conn(0x7f29700cfd60 msgr2=0x7f29700d0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:50.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.264+0000 7f2977c00640 1 -- 172.21.15.27:0/2417422920 shutdown_connections 2024-01-15T18:15:50.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.264+0000 7f2977c00640 1 -- 172.21.15.27:0/2417422920 wait complete. 2024-01-15T18:15:50.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f2977c00640 1 Processor -- start 2024-01-15T18:15:50.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f2977c00640 1 -- start start 2024-01-15T18:15:50.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f2977c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f297006ac00 0x7f29701505b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:50.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f2977c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29700c6800 0x7f29700cb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:50.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f2977c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f297015d2c0 0x7f29700cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:50.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f2977c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29700adf70 con 0x7f29700c6800 2024-01-15T18:15:50.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f2977c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29700aced0 con 0x7f297015d2c0 2024-01-15T18:15:50.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f2977c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29700ad1b0 con 0x7f297006ac00 2024-01-15T18:15:50.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f296fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29700c6800 0x7f29700cb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:50.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f2975200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f297006ac00 0x7f29701505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:50.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f2975c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f297015d2c0 0x7f29700cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:50.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f296fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29700c6800 0x7f29700cb8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56636/0 (socket says 172.21.15.27:56636) 2024-01-15T18:15:50.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f2975200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f297006ac00 0x7f29701505b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:43126/0 (socket says 172.21.15.27:43126) 2024-01-15T18:15:50.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f296fe00640 1 -- 172.21.15.27:0/2177931108 learned_addr learned my addr 172.21.15.27:0/2177931108 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:50.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f2975c00640 1 -- 172.21.15.27:0/2177931108 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f297006ac00 msgr2=0x7f29701505b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:50.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f2975c00640 1 --2- 172.21.15.27:0/2177931108 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f297006ac00 0x7f29701505b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:50.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f2975c00640 1 -- 172.21.15.27:0/2177931108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29700c6800 msgr2=0x7f29700cb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:50.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f2975c00640 1 --2- 172.21.15.27:0/2177931108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29700c6800 0x7f29700cb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:50.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f2975c00640 1 -- 172.21.15.27:0/2177931108 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2960009420 con 0x7f297015d2c0 2024-01-15T18:15:50.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f296fe00640 1 --2- 172.21.15.27:0/2177931108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29700c6800 0x7f29700cb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:15:50.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f2975c00640 1 --2- 172.21.15.27:0/2177931108 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f297015d2c0 0x7f29700cbdf0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f296800ca50 tx=0x7f296800cf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:50.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f296cc00640 1 -- 172.21.15.27:0/2177931108 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296800abb0 con 0x7f297015d2c0 2024-01-15T18:15:50.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f296cc00640 1 -- 172.21.15.27:0/2177931108 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f296800ad10 con 0x7f297015d2c0 2024-01-15T18:15:50.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f2977c00640 1 -- 172.21.15.27:0/2177931108 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29700ad490 con 0x7f297015d2c0 2024-01-15T18:15:50.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f296cc00640 1 -- 172.21.15.27:0/2177931108 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2968012600 con 0x7f297015d2c0 2024-01-15T18:15:50.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.268+0000 7f2977c00640 1 -- 172.21.15.27:0/2177931108 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29700ada50 con 0x7f297015d2c0 2024-01-15T18:15:50.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.272+0000 7f2977c00640 1 -- 172.21.15.27:0/2177931108 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2970067c00 con 0x7f297015d2c0 2024-01-15T18:15:50.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.272+0000 7f296cc00640 1 -- 172.21.15.27:0/2177931108 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7f2968007070 con 0x7f297015d2c0 2024-01-15T18:15:50.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.272+0000 7f296cc00640 1 --2- 172.21.15.27:0/2177931108 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f29480674c0 0x7f2948069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:50.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.272+0000 7f296cc00640 1 -- 172.21.15.27:0/2177931108 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f2968082de0 con 0x7f297015d2c0 2024-01-15T18:15:50.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.272+0000 7f2975200640 1 --2- 172.21.15.27:0/2177931108 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f29480674c0 0x7f2948069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:50.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.272+0000 7f2975200640 1 --2- 172.21.15.27:0/2177931108 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f29480674c0 0x7f2948069980 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f29600af000 tx=0x7f29600a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:50.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.276+0000 7f296cc00640 1 -- 172.21.15.27:0/2177931108 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f29680518c0 con 0x7f297015d2c0 2024-01-15T18:15:50.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.488+0000 7f2977c00640 1 -- 172.21.15.27:0/2177931108 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f29700adc70 con 0x7f297015d2c0 2024-01-15T18:15:50.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.492+0000 7f296cc00640 1 -- 172.21.15.27:0/2177931108 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1099 (secure 0 0 0) 0x7f2968017020 con 0x7f297015d2c0 2024-01-15T18:15:50.489 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:15:50.489 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":677,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":84,"num_osds":8,"num_up_osds":8,"osd_up_since":1705341884,"num_in_osds":8,"osd_in_since":1705341875,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":86},{"state_name":"creating+peering","count":11}],"num_pgs":97,"num_pools":3,"num_objects":12,"data_bytes":591166,"bytes_used":577019904,"bytes_avail":772517093376,"bytes_total":773094113280,"inactive_pgs_ratio":0.11340206116437912,"write_bytes_sec":2169,"read_op_per_sec":0,"write_op_per_sec":3},"fsmap":{"epoch":57,"id":6,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":6,"rank":0,"name":"b","status":"up:active","gid":8391}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":16,"modified":"2024-01-15T18:13:08.281503+0000","services":{}},"progress_events":{}} 2024-01-15T18:15:50.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.492+0000 7f2977c00640 1 -- 172.21.15.27:0/2177931108 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f29480674c0 msgr2=0x7f2948069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:50.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.492+0000 7f2977c00640 1 --2- 172.21.15.27:0/2177931108 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f29480674c0 0x7f2948069980 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f29600af000 tx=0x7f29600a7000 comp rx=0 tx=0).stop 2024-01-15T18:15:50.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.492+0000 7f2977c00640 1 -- 172.21.15.27:0/2177931108 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f297015d2c0 msgr2=0x7f29700cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:50.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.492+0000 7f2977c00640 1 --2- 172.21.15.27:0/2177931108 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f297015d2c0 0x7f29700cbdf0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f296800ca50 tx=0x7f296800cf10 comp rx=0 tx=0).stop 2024-01-15T18:15:50.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.492+0000 7f2977c00640 1 -- 172.21.15.27:0/2177931108 shutdown_connections 2024-01-15T18:15:50.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.492+0000 7f2977c00640 1 --2- 172.21.15.27:0/2177931108 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f29480674c0 0x7f2948069980 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:50.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.492+0000 7f2977c00640 1 --2- 172.21.15.27:0/2177931108 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f297015d2c0 0x7f29700cbdf0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:50.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.492+0000 7f2977c00640 1 --2- 172.21.15.27:0/2177931108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29700c6800 0x7f29700cb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:50.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.492+0000 7f2977c00640 1 --2- 172.21.15.27:0/2177931108 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f297006ac00 0x7f29701505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:50.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.492+0000 7f2977c00640 1 -- 172.21.15.27:0/2177931108 >> 172.21.15.27:0/2177931108 conn(0x7f29700cfd60 msgr2=0x7f2970071290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:50.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.492+0000 7f2977c00640 1 -- 172.21.15.27:0/2177931108 shutdown_connections 2024-01-15T18:15:50.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.492+0000 7f2977c00640 1 -- 172.21.15.27:0/2177931108 wait complete. 2024-01-15T18:15:50.512 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-01-15T18:15:50.512 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:15:50.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.656+0000 7fe558c00640 1 Processor -- start 2024-01-15T18:15:50.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.656+0000 7fe558c00640 1 -- start start 2024-01-15T18:15:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.656+0000 7fe558c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe554159bf0 0x7fe554159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.656+0000 7fe558c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fe554071e40 0x7fe5540c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.656+0000 7fe558c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe554159a50 con 0x7fe55406d430 2024-01-15T18:15:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.656+0000 7fe558c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5540cce00 con 0x7fe554159bf0 2024-01-15T18:15:50.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.656+0000 7fe558c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe55415a510 con 0x7fe554071e40 2024-01-15T18:15:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.656+0000 7fe551e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe554159bf0 0x7fe554159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.656+0000 7fe551e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe554159bf0 0x7fe554159fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56638/0 (socket says 172.21.15.27:56638) 2024-01-15T18:15:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.656+0000 7fe551e00640 1 -- 172.21.15.27:0/2406076658 learned_addr learned my addr 172.21.15.27:0/2406076658 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.656+0000 7fe551400640 1 --2- 172.21.15.27:0/2406076658 >> v2:172.21.15.93:3300/0 conn(0x7fe554071e40 0x7fe5540c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.660+0000 7fe551e00640 1 -- 172.21.15.27:0/2406076658 >> v2:172.21.15.93:3300/0 conn(0x7fe554071e40 msgr2=0x7fe5540c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.660+0000 7fe551e00640 1 --2- 172.21.15.27:0/2406076658 >> v2:172.21.15.93:3300/0 conn(0x7fe554071e40 0x7fe5540c9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.660+0000 7fe551e00640 1 -- 172.21.15.27:0/2406076658 >> v1:172.21.15.93:6789/0 conn(0x7fe55406d430 legacy=0x7fe55415d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:50.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.660+0000 7fe551e00640 1 -- 172.21.15.27:0/2406076658 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55406d980 con 0x7fe554159bf0 2024-01-15T18:15:50.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.660+0000 7fe551e00640 1 --2- 172.21.15.27:0/2406076658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe554159bf0 0x7fe554159fd0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fe5540cfb10 tx=0x7fe54401fe00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c12f13baeeb77a0e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:50.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.660+0000 7fe550a00640 1 -- 172.21.15.27:0/2406076658 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5440a9b00 con 0x7fe554159bf0 2024-01-15T18:15:50.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.660+0000 7fe550a00640 1 -- 172.21.15.27:0/2406076658 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5440a9c60 con 0x7fe554159bf0 2024-01-15T18:15:50.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.660+0000 7fe550a00640 1 -- 172.21.15.27:0/2406076658 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5440b1440 con 0x7fe554159bf0 2024-01-15T18:15:50.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.664+0000 7fe558c00640 1 -- 172.21.15.27:0/2406076658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe554159bf0 msgr2=0x7fe554159fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:50.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.664+0000 7fe558c00640 1 --2- 172.21.15.27:0/2406076658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe554159bf0 0x7fe554159fd0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fe5540cfb10 tx=0x7fe54401fe00 comp rx=0 tx=0).stop 2024-01-15T18:15:50.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.664+0000 7fe558c00640 1 -- 172.21.15.27:0/2406076658 shutdown_connections 2024-01-15T18:15:50.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.664+0000 7fe558c00640 1 --2- 172.21.15.27:0/2406076658 >> v2:172.21.15.93:3300/0 conn(0x7fe554071e40 0x7fe5540c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:50.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.664+0000 7fe558c00640 1 --2- 172.21.15.27:0/2406076658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe554159bf0 0x7fe554159fd0 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:50.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.664+0000 7fe558c00640 1 -- 172.21.15.27:0/2406076658 >> 172.21.15.27:0/2406076658 conn(0x7fe5540cec60 msgr2=0x7fe5540cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:50.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.664+0000 7fe558c00640 1 -- 172.21.15.27:0/2406076658 shutdown_connections 2024-01-15T18:15:50.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.664+0000 7fe558c00640 1 -- 172.21.15.27:0/2406076658 wait complete. 2024-01-15T18:15:50.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.664+0000 7fe558c00640 1 Processor -- start 2024-01-15T18:15:50.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.664+0000 7fe558c00640 1 -- start start 2024-01-15T18:15:50.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.664+0000 7fe558c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe55406d430 0x7fe5540b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:50.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.664+0000 7fe558c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe554159bf0 0x7fe55415ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:50.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.664+0000 7fe558c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5540b1920 0x7fe5541f64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:50.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.664+0000 7fe558c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe554151960 con 0x7fe5540b1920 2024-01-15T18:15:50.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.664+0000 7fe558c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55415c290 con 0x7fe55406d430 2024-01-15T18:15:50.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.664+0000 7fe558c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe55415c570 con 0x7fe554159bf0 2024-01-15T18:15:50.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.664+0000 7fe552800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5540b1920 0x7fe5541f64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:50.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.664+0000 7fe551400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe554159bf0 0x7fe55415ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:50.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.664+0000 7fe551e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe55406d430 0x7fe5540b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:50.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.664+0000 7fe552800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5540b1920 0x7fe5541f64c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56640/0 (socket says 172.21.15.27:56640) 2024-01-15T18:15:50.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.664+0000 7fe552800640 1 -- 172.21.15.27:0/1847709629 learned_addr learned my addr 172.21.15.27:0/1847709629 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:50.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.664+0000 7fe551400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe554159bf0 0x7fe55415ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:43130/0 (socket says 172.21.15.27:43130) 2024-01-15T18:15:50.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.668+0000 7fe551e00640 1 -- 172.21.15.27:0/1847709629 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe554159bf0 msgr2=0x7fe55415ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:50.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.668+0000 7fe551e00640 1 --2- 172.21.15.27:0/1847709629 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe554159bf0 0x7fe55415ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:50.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.668+0000 7fe551e00640 1 -- 172.21.15.27:0/1847709629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5540b1920 msgr2=0x7fe5541f64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:50.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.668+0000 7fe551e00640 1 --2- 172.21.15.27:0/1847709629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5540b1920 0x7fe5541f64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:50.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.668+0000 7fe551e00640 1 -- 172.21.15.27:0/1847709629 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5440094d0 con 0x7fe55406d430 2024-01-15T18:15:50.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.668+0000 7fe551400640 1 --2- 172.21.15.27:0/1847709629 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe554159bf0 0x7fe55415ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:15:50.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.668+0000 7fe552800640 1 --2- 172.21.15.27:0/1847709629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5540b1920 0x7fe5541f64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:15:50.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.668+0000 7fe551e00640 1 --2- 172.21.15.27:0/1847709629 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe55406d430 0x7fe5540b4450 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fe5440b0000 tx=0x7fe54401cba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:50.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.668+0000 7fe54a000640 1 -- 172.21.15.27:0/1847709629 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5440b2e60 con 0x7fe55406d430 2024-01-15T18:15:50.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.668+0000 7fe54a000640 1 -- 172.21.15.27:0/1847709629 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5440bc480 con 0x7fe55406d430 2024-01-15T18:15:50.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.668+0000 7fe558c00640 1 -- 172.21.15.27:0/1847709629 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5540ac890 con 0x7fe55406d430 2024-01-15T18:15:50.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.668+0000 7fe54a000640 1 -- 172.21.15.27:0/1847709629 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5440bc810 con 0x7fe55406d430 2024-01-15T18:15:50.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.668+0000 7fe558c00640 1 -- 172.21.15.27:0/1847709629 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5540ace50 con 0x7fe55406d430 2024-01-15T18:15:50.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.668+0000 7fe558c00640 1 -- 172.21.15.27:0/1847709629 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe55404e850 con 0x7fe55406d430 2024-01-15T18:15:50.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.668+0000 7fe54a000640 1 -- 172.21.15.27:0/1847709629 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7fe5440bc970 con 0x7fe55406d430 2024-01-15T18:15:50.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.672+0000 7fe54a000640 1 --2- 172.21.15.27:0/1847709629 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe52c0674f0 0x7fe52c0699b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:50.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.672+0000 7fe54a000640 1 -- 172.21.15.27:0/1847709629 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fe544121620 con 0x7fe55406d430 2024-01-15T18:15:50.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.672+0000 7fe551400640 1 --2- 172.21.15.27:0/1847709629 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe52c0674f0 0x7fe52c0699b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:50.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.672+0000 7fe551400640 1 --2- 172.21.15.27:0/1847709629 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe52c0674f0 0x7fe52c0699b0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fe55415b000 tx=0x7fe53c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:50.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.672+0000 7fe54a000640 1 -- 172.21.15.27:0/1847709629 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe5440f0100 con 0x7fe55406d430 2024-01-15T18:15:50.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.828+0000 7fe558c00640 1 -- 172.21.15.27:0/1847709629 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe55406e220 con 0x7fe52c0674f0 2024-01-15T18:15:50.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.836+0000 7fe54a000640 1 -- 172.21.15.27:0/1847709629 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288263 (secure 0 0 0) 0x7fe53c011e60 con 0x7fe52c0674f0 2024-01-15T18:15:50.837 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:15:50.839 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:15:50.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.840+0000 7fe558c00640 1 -- 172.21.15.27:0/1847709629 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe52c0674f0 msgr2=0x7fe52c0699b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:50.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.840+0000 7fe558c00640 1 --2- 172.21.15.27:0/1847709629 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe52c0674f0 0x7fe52c0699b0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fe55415b000 tx=0x7fe53c009190 comp rx=0 tx=0).stop 2024-01-15T18:15:50.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.840+0000 7fe558c00640 1 -- 172.21.15.27:0/1847709629 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe55406d430 msgr2=0x7fe5540b4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:50.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.840+0000 7fe558c00640 1 --2- 172.21.15.27:0/1847709629 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe55406d430 0x7fe5540b4450 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fe5440b0000 tx=0x7fe54401cba0 comp rx=0 tx=0).stop 2024-01-15T18:15:50.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.840+0000 7fe558c00640 1 -- 172.21.15.27:0/1847709629 shutdown_connections 2024-01-15T18:15:50.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.840+0000 7fe558c00640 1 --2- 172.21.15.27:0/1847709629 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe52c0674f0 0x7fe52c0699b0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:50.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.840+0000 7fe558c00640 1 --2- 172.21.15.27:0/1847709629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5540b1920 0x7fe5541f64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:50.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.840+0000 7fe558c00640 1 --2- 172.21.15.27:0/1847709629 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe554159bf0 0x7fe55415ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:50.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.840+0000 7fe558c00640 1 --2- 172.21.15.27:0/1847709629 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe55406d430 0x7fe5540b4450 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:50.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.840+0000 7fe558c00640 1 -- 172.21.15.27:0/1847709629 >> 172.21.15.27:0/1847709629 conn(0x7fe5540cec60 msgr2=0x7fe5540c9290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:50.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.840+0000 7fe558c00640 1 -- 172.21.15.27:0/1847709629 shutdown_connections 2024-01-15T18:15:50.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:50.840+0000 7fe558c00640 1 -- 172.21.15.27:0/1847709629 wait complete. 2024-01-15T18:15:50.859 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":410,"stamp":"2024-01-15T18:15:50.331449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":563576,"kb_used_data":9076,"kb_used_omap":12,"kb_used_meta":554355,"kb_avail":754411144,"statfs":{"total":773094113280,"available":772517011456,"internally_reserved":0,"allocated":9293824,"data_stored":4643552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":567660080},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.190364"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.197673+0000","last_change":"2024-01-15T18:15:45.194410+0000","last_active":"2024-01-15T18:15:45.197673+0000","last_peered":"2024-01-15T18:15:45.197673+0000","last_clean":"2024-01-15T18:15:45.197673+0000","last_became_active":"2024-01-15T18:15:45.193977+0000","last_became_peered":"2024-01-15T18:15:45.193977+0000","last_unstale":"2024-01-15T18:15:45.197673+0000","last_undegraded":"2024-01-15T18:15:45.197673+0000","last_fullsized":"2024-01-15T18:15:45.197673+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:59:33.238710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.193135+0000","last_change":"2024-01-15T18:15:45.185773+0000","last_active":"2024-01-15T18:15:45.193135+0000","last_peered":"2024-01-15T18:15:45.193135+0000","last_clean":"2024-01-15T18:15:45.193135+0000","last_became_active":"2024-01-15T18:15:45.185459+0000","last_became_peered":"2024-01-15T18:15:45.185459+0000","last_unstale":"2024-01-15T18:15:45.193135+0000","last_undegraded":"2024-01-15T18:15:45.193135+0000","last_fullsized":"2024-01-15T18:15:45.193135+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:33:22.524087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196740+0000","last_change":"2024-01-15T18:15:45.190593+0000","last_active":"2024-01-15T18:15:45.196740+0000","last_peered":"2024-01-15T18:15:45.196740+0000","last_clean":"2024-01-15T18:15:45.196740+0000","last_became_active":"2024-01-15T18:15:45.190292+0000","last_became_peered":"2024-01-15T18:15:45.190292+0000","last_unstale":"2024-01-15T18:15:45.196740+0000","last_undegraded":"2024-01-15T18:15:45.196740+0000","last_fullsized":"2024-01-15T18:15:45.196740+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:41:44.153310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.197017+0000","last_change":"2024-01-15T18:15:45.192728+0000","last_active":"2024-01-15T18:15:45.197017+0000","last_peered":"2024-01-15T18:15:45.197017+0000","last_clean":"2024-01-15T18:15:45.197017+0000","last_became_active":"2024-01-15T18:15:45.192270+0000","last_became_peered":"2024-01-15T18:15:45.192270+0000","last_unstale":"2024-01-15T18:15:45.197017+0000","last_undegraded":"2024-01-15T18:15:45.197017+0000","last_fullsized":"2024-01-15T18:15:45.197017+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:10:12.400165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191097+0000","last_change":"2024-01-15T18:15:45.181959+0000","last_active":"2024-01-15T18:15:45.191097+0000","last_peered":"2024-01-15T18:15:45.191097+0000","last_clean":"2024-01-15T18:15:45.191097+0000","last_became_active":"2024-01-15T18:15:45.181526+0000","last_became_peered":"2024-01-15T18:15:45.181526+0000","last_unstale":"2024-01-15T18:15:45.191097+0000","last_undegraded":"2024-01-15T18:15:45.191097+0000","last_fullsized":"2024-01-15T18:15:45.191097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:10:10.202003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.058582+0000","last_change":"2024-01-15T18:15:45.178635+0000","last_active":"2024-01-15T18:15:47.058582+0000","last_peered":"2024-01-15T18:15:47.058582+0000","last_clean":"2024-01-15T18:15:47.058582+0000","last_became_active":"2024-01-15T18:15:45.178354+0000","last_became_peered":"2024-01-15T18:15:45.178354+0000","last_unstale":"2024-01-15T18:15:47.058582+0000","last_undegraded":"2024-01-15T18:15:47.058582+0000","last_fullsized":"2024-01-15T18:15:47.058582+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:47:05.193543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.188332+0000","last_change":"2024-01-15T18:15:45.177382+0000","last_active":"2024-01-15T18:15:45.188332+0000","last_peered":"2024-01-15T18:15:45.188332+0000","last_clean":"2024-01-15T18:15:45.188332+0000","last_became_active":"2024-01-15T18:15:45.173825+0000","last_became_peered":"2024-01-15T18:15:45.173825+0000","last_unstale":"2024-01-15T18:15:45.188332+0000","last_undegraded":"2024-01-15T18:15:45.188332+0000","last_fullsized":"2024-01-15T18:15:45.188332+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:03:55.153676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.058383+0000","last_change":"2024-01-15T18:15:45.172298+0000","last_active":"2024-01-15T18:15:47.058383+0000","last_peered":"2024-01-15T18:15:47.058383+0000","last_clean":"2024-01-15T18:15:47.058383+0000","last_became_active":"2024-01-15T18:15:45.172026+0000","last_became_peered":"2024-01-15T18:15:45.172026+0000","last_unstale":"2024-01-15T18:15:47.058383+0000","last_undegraded":"2024-01-15T18:15:47.058383+0000","last_fullsized":"2024-01-15T18:15:47.058383+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:30:49.183104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.058850+0000","last_change":"2024-01-15T18:15:45.199261+0000","last_active":"2024-01-15T18:15:47.058850+0000","last_peered":"2024-01-15T18:15:47.058850+0000","last_clean":"2024-01-15T18:15:47.058850+0000","last_became_active":"2024-01-15T18:15:45.198949+0000","last_became_peered":"2024-01-15T18:15:45.198949+0000","last_unstale":"2024-01-15T18:15:47.058850+0000","last_undegraded":"2024-01-15T18:15:47.058850+0000","last_fullsized":"2024-01-15T18:15:47.058850+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:53:53.175858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.195302+0000","last_change":"2024-01-15T18:15:45.190970+0000","last_active":"2024-01-15T18:15:45.195302+0000","last_peered":"2024-01-15T18:15:45.195302+0000","last_clean":"2024-01-15T18:15:45.195302+0000","last_became_active":"2024-01-15T18:15:45.187235+0000","last_became_peered":"2024-01-15T18:15:45.187235+0000","last_unstale":"2024-01-15T18:15:45.195302+0000","last_undegraded":"2024-01-15T18:15:45.195302+0000","last_fullsized":"2024-01-15T18:15:45.195302+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:35:49.681365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.186724+0000","last_change":"2024-01-15T18:15:45.179561+0000","last_active":"2024-01-15T18:15:45.186724+0000","last_peered":"2024-01-15T18:15:45.186724+0000","last_clean":"2024-01-15T18:15:45.186724+0000","last_became_active":"2024-01-15T18:15:45.179305+0000","last_became_peered":"2024-01-15T18:15:45.179305+0000","last_unstale":"2024-01-15T18:15:45.186724+0000","last_undegraded":"2024-01-15T18:15:45.186724+0000","last_fullsized":"2024-01-15T18:15:45.186724+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:56:26.621247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191872+0000","last_change":"2024-01-15T18:15:45.189460+0000","last_active":"2024-01-15T18:15:45.191872+0000","last_peered":"2024-01-15T18:15:45.191872+0000","last_clean":"2024-01-15T18:15:45.191872+0000","last_became_active":"2024-01-15T18:15:45.188105+0000","last_became_peered":"2024-01-15T18:15:45.188105+0000","last_unstale":"2024-01-15T18:15:45.191872+0000","last_undegraded":"2024-01-15T18:15:45.191872+0000","last_fullsized":"2024-01-15T18:15:45.191872+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:45:42.971410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196682+0000","last_change":"2024-01-15T18:15:45.190326+0000","last_active":"2024-01-15T18:15:45.196682+0000","last_peered":"2024-01-15T18:15:45.196682+0000","last_clean":"2024-01-15T18:15:45.196682+0000","last_became_active":"2024-01-15T18:15:45.188312+0000","last_became_peered":"2024-01-15T18:15:45.188312+0000","last_unstale":"2024-01-15T18:15:45.196682+0000","last_undegraded":"2024-01-15T18:15:45.196682+0000","last_fullsized":"2024-01-15T18:15:45.196682+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:28:05.260374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191135+0000","last_change":"2024-01-15T18:15:45.180022+0000","last_active":"2024-01-15T18:15:45.191135+0000","last_peered":"2024-01-15T18:15:45.191135+0000","last_clean":"2024-01-15T18:15:45.191135+0000","last_became_active":"2024-01-15T18:15:45.179537+0000","last_became_peered":"2024-01-15T18:15:45.179537+0000","last_unstale":"2024-01-15T18:15:45.191135+0000","last_undegraded":"2024-01-15T18:15:45.191135+0000","last_fullsized":"2024-01-15T18:15:45.191135+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:55.575874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196300+0000","last_change":"2024-01-15T18:15:45.185924+0000","last_active":"2024-01-15T18:15:45.196300+0000","last_peered":"2024-01-15T18:15:45.196300+0000","last_clean":"2024-01-15T18:15:45.196300+0000","last_became_active":"2024-01-15T18:15:45.185669+0000","last_became_peered":"2024-01-15T18:15:45.185669+0000","last_unstale":"2024-01-15T18:15:45.196300+0000","last_undegraded":"2024-01-15T18:15:45.196300+0000","last_fullsized":"2024-01-15T18:15:45.196300+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:08:31.073685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.184890+0000","last_change":"2024-01-15T18:15:45.176262+0000","last_active":"2024-01-15T18:15:45.184890+0000","last_peered":"2024-01-15T18:15:45.184890+0000","last_clean":"2024-01-15T18:15:45.184890+0000","last_became_active":"2024-01-15T18:15:45.175976+0000","last_became_peered":"2024-01-15T18:15:45.175976+0000","last_unstale":"2024-01-15T18:15:45.184890+0000","last_undegraded":"2024-01-15T18:15:45.184890+0000","last_fullsized":"2024-01-15T18:15:45.184890+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:16.351733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191295+0000","last_change":"2024-01-15T18:15:45.184436+0000","last_active":"2024-01-15T18:15:45.191295+0000","last_peered":"2024-01-15T18:15:45.191295+0000","last_clean":"2024-01-15T18:15:45.191295+0000","last_became_active":"2024-01-15T18:15:45.181072+0000","last_became_peered":"2024-01-15T18:15:45.181072+0000","last_unstale":"2024-01-15T18:15:45.191295+0000","last_undegraded":"2024-01-15T18:15:45.191295+0000","last_fullsized":"2024-01-15T18:15:45.191295+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:04:09.741683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191667+0000","last_change":"2024-01-15T18:15:45.184794+0000","last_active":"2024-01-15T18:15:45.191667+0000","last_peered":"2024-01-15T18:15:45.191667+0000","last_clean":"2024-01-15T18:15:45.191667+0000","last_became_active":"2024-01-15T18:15:45.184528+0000","last_became_peered":"2024-01-15T18:15:45.184528+0000","last_unstale":"2024-01-15T18:15:45.191667+0000","last_undegraded":"2024-01-15T18:15:45.191667+0000","last_fullsized":"2024-01-15T18:15:45.191667+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:33.583003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.184329+0000","last_change":"2024-01-15T18:15:45.196447+0000","last_active":"2024-01-15T18:15:46.184329+0000","last_peered":"2024-01-15T18:15:46.184329+0000","last_clean":"2024-01-15T18:15:46.184329+0000","last_became_active":"2024-01-15T18:15:45.194478+0000","last_became_peered":"2024-01-15T18:15:45.194478+0000","last_unstale":"2024-01-15T18:15:46.184329+0000","last_undegraded":"2024-01-15T18:15:46.184329+0000","last_fullsized":"2024-01-15T18:15:46.184329+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:20:05.906577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.203247+0000","last_change":"2024-01-15T18:15:45.202436+0000","last_active":"2024-01-15T18:15:45.203247+0000","last_peered":"2024-01-15T18:15:45.203247+0000","last_clean":"2024-01-15T18:15:45.203247+0000","last_became_active":"2024-01-15T18:15:45.202075+0000","last_became_peered":"2024-01-15T18:15:45.202075+0000","last_unstale":"2024-01-15T18:15:45.203247+0000","last_undegraded":"2024-01-15T18:15:45.203247+0000","last_fullsized":"2024-01-15T18:15:45.203247+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:29:34.404126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198796+0000","last_change":"2024-01-15T18:15:45.197310+0000","last_active":"2024-01-15T18:15:45.198796+0000","last_peered":"2024-01-15T18:15:45.198796+0000","last_clean":"2024-01-15T18:15:45.198796+0000","last_became_active":"2024-01-15T18:15:45.196364+0000","last_became_peered":"2024-01-15T18:15:45.196364+0000","last_unstale":"2024-01-15T18:15:45.198796+0000","last_undegraded":"2024-01-15T18:15:45.198796+0000","last_fullsized":"2024-01-15T18:15:45.198796+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:28:16.596449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191310+0000","last_change":"2024-01-15T18:15:45.185683+0000","last_active":"2024-01-15T18:15:45.191310+0000","last_peered":"2024-01-15T18:15:45.191310+0000","last_clean":"2024-01-15T18:15:45.191310+0000","last_became_active":"2024-01-15T18:15:45.185303+0000","last_became_peered":"2024-01-15T18:15:45.185303+0000","last_unstale":"2024-01-15T18:15:45.191310+0000","last_undegraded":"2024-01-15T18:15:45.191310+0000","last_fullsized":"2024-01-15T18:15:45.191310+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:12:38.030672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198110+0000","last_change":"2024-01-15T18:15:45.195966+0000","last_active":"2024-01-15T18:15:45.198110+0000","last_peered":"2024-01-15T18:15:45.198110+0000","last_clean":"2024-01-15T18:15:45.198110+0000","last_became_active":"2024-01-15T18:15:45.194717+0000","last_became_peered":"2024-01-15T18:15:45.194717+0000","last_unstale":"2024-01-15T18:15:45.198110+0000","last_undegraded":"2024-01-15T18:15:45.198110+0000","last_fullsized":"2024-01-15T18:15:45.198110+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:50:32.308793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.190685+0000","last_change":"2024-01-15T18:15:45.179745+0000","last_active":"2024-01-15T18:15:45.190685+0000","last_peered":"2024-01-15T18:15:45.190685+0000","last_clean":"2024-01-15T18:15:45.190685+0000","last_became_active":"2024-01-15T18:15:45.179329+0000","last_became_peered":"2024-01-15T18:15:45.179329+0000","last_unstale":"2024-01-15T18:15:45.190685+0000","last_undegraded":"2024-01-15T18:15:45.190685+0000","last_fullsized":"2024-01-15T18:15:45.190685+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:01:10.870535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198301+0000","last_change":"2024-01-15T18:15:45.192970+0000","last_active":"2024-01-15T18:15:45.198301+0000","last_peered":"2024-01-15T18:15:45.198301+0000","last_clean":"2024-01-15T18:15:45.198301+0000","last_became_active":"2024-01-15T18:15:45.192474+0000","last_became_peered":"2024-01-15T18:15:45.192474+0000","last_unstale":"2024-01-15T18:15:45.198301+0000","last_undegraded":"2024-01-15T18:15:45.198301+0000","last_fullsized":"2024-01-15T18:15:45.198301+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:38.217871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.183678+0000","last_change":"2024-01-15T18:15:45.180211+0000","last_active":"2024-01-15T18:15:45.183678+0000","last_peered":"2024-01-15T18:15:45.183678+0000","last_clean":"2024-01-15T18:15:45.183678+0000","last_became_active":"2024-01-15T18:15:45.179566+0000","last_became_peered":"2024-01-15T18:15:45.179566+0000","last_unstale":"2024-01-15T18:15:45.183678+0000","last_undegraded":"2024-01-15T18:15:45.183678+0000","last_fullsized":"2024-01-15T18:15:45.183678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:40:13.829197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.192821+0000","last_change":"2024-01-15T18:15:45.189295+0000","last_active":"2024-01-15T18:15:45.192821+0000","last_peered":"2024-01-15T18:15:45.192821+0000","last_clean":"2024-01-15T18:15:45.192821+0000","last_became_active":"2024-01-15T18:15:45.187101+0000","last_became_peered":"2024-01-15T18:15:45.187101+0000","last_unstale":"2024-01-15T18:15:45.192821+0000","last_undegraded":"2024-01-15T18:15:45.192821+0000","last_fullsized":"2024-01-15T18:15:45.192821+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:01:13.745010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.195405+0000","last_change":"2024-01-15T18:15:45.192353+0000","last_active":"2024-01-15T18:15:45.195405+0000","last_peered":"2024-01-15T18:15:45.195405+0000","last_clean":"2024-01-15T18:15:45.195405+0000","last_became_active":"2024-01-15T18:15:45.191563+0000","last_became_peered":"2024-01-15T18:15:45.191563+0000","last_unstale":"2024-01-15T18:15:45.195405+0000","last_undegraded":"2024-01-15T18:15:45.195405+0000","last_fullsized":"2024-01-15T18:15:45.195405+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:57:43.892130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.197122+0000","last_change":"2024-01-15T18:15:45.190607+0000","last_active":"2024-01-15T18:15:45.197122+0000","last_peered":"2024-01-15T18:15:45.197122+0000","last_clean":"2024-01-15T18:15:45.197122+0000","last_became_active":"2024-01-15T18:15:45.189633+0000","last_became_peered":"2024-01-15T18:15:45.189633+0000","last_unstale":"2024-01-15T18:15:45.197122+0000","last_undegraded":"2024-01-15T18:15:45.197122+0000","last_fullsized":"2024-01-15T18:15:45.197122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:35:49.681365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.187329+0000","last_change":"2024-01-15T18:15:45.180268+0000","last_active":"2024-01-15T18:15:45.187329+0000","last_peered":"2024-01-15T18:15:45.187329+0000","last_clean":"2024-01-15T18:15:45.187329+0000","last_became_active":"2024-01-15T18:15:45.179617+0000","last_became_peered":"2024-01-15T18:15:45.179617+0000","last_unstale":"2024-01-15T18:15:45.187329+0000","last_undegraded":"2024-01-15T18:15:45.187329+0000","last_fullsized":"2024-01-15T18:15:45.187329+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:18:22.468719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.186449+0000","last_change":"2024-01-15T18:15:45.177309+0000","last_active":"2024-01-15T18:15:45.186449+0000","last_peered":"2024-01-15T18:15:45.186449+0000","last_clean":"2024-01-15T18:15:45.186449+0000","last_became_active":"2024-01-15T18:15:45.177022+0000","last_became_peered":"2024-01-15T18:15:45.177022+0000","last_unstale":"2024-01-15T18:15:45.186449+0000","last_undegraded":"2024-01-15T18:15:45.186449+0000","last_fullsized":"2024-01-15T18:15:45.186449+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:53.495194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.187438+0000","last_change":"2024-01-15T18:15:45.182964+0000","last_active":"2024-01-15T18:15:45.187438+0000","last_peered":"2024-01-15T18:15:45.187438+0000","last_clean":"2024-01-15T18:15:45.187438+0000","last_became_active":"2024-01-15T18:15:45.182690+0000","last_became_peered":"2024-01-15T18:15:45.182690+0000","last_unstale":"2024-01-15T18:15:45.187438+0000","last_undegraded":"2024-01-15T18:15:45.187438+0000","last_fullsized":"2024-01-15T18:15:45.187438+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:31.079315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.192979+0000","last_change":"2024-01-15T18:15:45.189818+0000","last_active":"2024-01-15T18:15:45.192979+0000","last_peered":"2024-01-15T18:15:45.192979+0000","last_clean":"2024-01-15T18:15:45.192979+0000","last_became_active":"2024-01-15T18:15:45.189534+0000","last_became_peered":"2024-01-15T18:15:45.189534+0000","last_unstale":"2024-01-15T18:15:45.192979+0000","last_undegraded":"2024-01-15T18:15:45.192979+0000","last_fullsized":"2024-01-15T18:15:45.192979+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:13:12.060989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"83'4","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.197594+0000","last_change":"2024-01-15T18:15:44.163810+0000","last_active":"2024-01-15T18:15:45.197594+0000","last_peered":"2024-01-15T18:15:45.197594+0000","last_clean":"2024-01-15T18:15:45.197594+0000","last_became_active":"2024-01-15T18:15:44.163128+0000","last_became_peered":"2024-01-15T18:15:44.163128+0000","last_unstale":"2024-01-15T18:15:45.197594+0000","last_undegraded":"2024-01-15T18:15:45.197594+0000","last_fullsized":"2024-01-15T18:15:45.197594+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:22:07.304644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.202605+0000","last_change":"2024-01-15T18:15:45.201966+0000","last_active":"2024-01-15T18:15:45.202605+0000","last_peered":"2024-01-15T18:15:45.202605+0000","last_clean":"2024-01-15T18:15:45.202605+0000","last_became_active":"2024-01-15T18:15:45.201568+0000","last_became_peered":"2024-01-15T18:15:45.201568+0000","last_unstale":"2024-01-15T18:15:45.202605+0000","last_undegraded":"2024-01-15T18:15:45.202605+0000","last_fullsized":"2024-01-15T18:15:45.202605+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:16.351733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.187621+0000","last_change":"2024-01-15T18:15:44.156714+0000","last_active":"2024-01-15T18:15:45.187621+0000","last_peered":"2024-01-15T18:15:45.187621+0000","last_clean":"2024-01-15T18:15:45.187621+0000","last_became_active":"2024-01-15T18:15:44.156419+0000","last_became_peered":"2024-01-15T18:15:44.156419+0000","last_unstale":"2024-01-15T18:15:45.187621+0000","last_undegraded":"2024-01-15T18:15:45.187621+0000","last_fullsized":"2024-01-15T18:15:45.187621+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:14:41.627743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.197147+0000","last_change":"2024-01-15T18:15:45.192955+0000","last_active":"2024-01-15T18:15:45.197147+0000","last_peered":"2024-01-15T18:15:45.197147+0000","last_clean":"2024-01-15T18:15:45.197147+0000","last_became_active":"2024-01-15T18:15:45.192565+0000","last_became_peered":"2024-01-15T18:15:45.192565+0000","last_unstale":"2024-01-15T18:15:45.197147+0000","last_undegraded":"2024-01-15T18:15:45.197147+0000","last_fullsized":"2024-01-15T18:15:45.197147+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:07:58.498146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"83'2","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196747+0000","last_change":"2024-01-15T18:15:44.167059+0000","last_active":"2024-01-15T18:15:45.196747+0000","last_peered":"2024-01-15T18:15:45.196747+0000","last_clean":"2024-01-15T18:15:45.196747+0000","last_became_active":"2024-01-15T18:15:44.166829+0000","last_became_peered":"2024-01-15T18:15:44.166829+0000","last_unstale":"2024-01-15T18:15:45.196747+0000","last_undegraded":"2024-01-15T18:15:45.196747+0000","last_fullsized":"2024-01-15T18:15:45.196747+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:53:52.170612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.194382+0000","last_change":"2024-01-15T18:15:45.186505+0000","last_active":"2024-01-15T18:15:45.194382+0000","last_peered":"2024-01-15T18:15:45.194382+0000","last_clean":"2024-01-15T18:15:45.194382+0000","last_became_active":"2024-01-15T18:15:45.186181+0000","last_became_peered":"2024-01-15T18:15:45.186181+0000","last_unstale":"2024-01-15T18:15:45.194382+0000","last_undegraded":"2024-01-15T18:15:45.194382+0000","last_fullsized":"2024-01-15T18:15:45.194382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:03:29.579452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"83'2","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.199414+0000","last_change":"2024-01-15T18:15:44.166697+0000","last_active":"2024-01-15T18:15:45.199414+0000","last_peered":"2024-01-15T18:15:45.199414+0000","last_clean":"2024-01-15T18:15:45.199414+0000","last_became_active":"2024-01-15T18:15:44.166389+0000","last_became_peered":"2024-01-15T18:15:44.166389+0000","last_unstale":"2024-01-15T18:15:45.199414+0000","last_undegraded":"2024-01-15T18:15:45.199414+0000","last_fullsized":"2024-01-15T18:15:45.199414+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:56:34.685926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.187351+0000","last_change":"2024-01-15T18:15:45.180445+0000","last_active":"2024-01-15T18:15:45.187351+0000","last_peered":"2024-01-15T18:15:45.187351+0000","last_clean":"2024-01-15T18:15:45.187351+0000","last_became_active":"2024-01-15T18:15:45.179946+0000","last_became_peered":"2024-01-15T18:15:45.179946+0000","last_unstale":"2024-01-15T18:15:45.187351+0000","last_undegraded":"2024-01-15T18:15:45.187351+0000","last_fullsized":"2024-01-15T18:15:45.187351+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:50:32.308793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.181787+0000","last_change":"2024-01-15T18:15:44.157992+0000","last_active":"2024-01-15T18:15:45.181787+0000","last_peered":"2024-01-15T18:15:45.181787+0000","last_clean":"2024-01-15T18:15:45.181787+0000","last_became_active":"2024-01-15T18:15:44.157721+0000","last_became_peered":"2024-01-15T18:15:44.157721+0000","last_unstale":"2024-01-15T18:15:45.181787+0000","last_undegraded":"2024-01-15T18:15:45.181787+0000","last_fullsized":"2024-01-15T18:15:45.181787+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:27:47.584018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.185831+0000","last_change":"2024-01-15T18:15:45.178738+0000","last_active":"2024-01-15T18:15:45.185831+0000","last_peered":"2024-01-15T18:15:45.185831+0000","last_clean":"2024-01-15T18:15:45.185831+0000","last_became_active":"2024-01-15T18:15:45.178446+0000","last_became_peered":"2024-01-15T18:15:45.178446+0000","last_unstale":"2024-01-15T18:15:45.185831+0000","last_undegraded":"2024-01-15T18:15:45.185831+0000","last_fullsized":"2024-01-15T18:15:45.185831+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:59:33.238710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.187378+0000","last_change":"2024-01-15T18:15:44.154715+0000","last_active":"2024-01-15T18:15:45.187378+0000","last_peered":"2024-01-15T18:15:45.187378+0000","last_clean":"2024-01-15T18:15:45.187378+0000","last_became_active":"2024-01-15T18:15:44.154433+0000","last_became_peered":"2024-01-15T18:15:44.154433+0000","last_unstale":"2024-01-15T18:15:45.187378+0000","last_undegraded":"2024-01-15T18:15:45.187378+0000","last_fullsized":"2024-01-15T18:15:45.187378+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:42:00.915026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.202740+0000","last_change":"2024-01-15T18:15:45.202049+0000","last_active":"2024-01-15T18:15:45.202740+0000","last_peered":"2024-01-15T18:15:45.202740+0000","last_clean":"2024-01-15T18:15:45.202740+0000","last_became_active":"2024-01-15T18:15:45.201742+0000","last_became_peered":"2024-01-15T18:15:45.201742+0000","last_unstale":"2024-01-15T18:15:45.202740+0000","last_undegraded":"2024-01-15T18:15:45.202740+0000","last_fullsized":"2024-01-15T18:15:45.202740+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:56.553890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.171647+0000","last_change":"2024-01-15T18:15:44.167859+0000","last_active":"2024-01-15T18:15:45.171647+0000","last_peered":"2024-01-15T18:15:45.171647+0000","last_clean":"2024-01-15T18:15:45.171647+0000","last_became_active":"2024-01-15T18:15:44.167561+0000","last_became_peered":"2024-01-15T18:15:44.167561+0000","last_unstale":"2024-01-15T18:15:45.171647+0000","last_undegraded":"2024-01-15T18:15:45.171647+0000","last_fullsized":"2024-01-15T18:15:45.171647+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:31.305427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196319+0000","last_change":"2024-01-15T18:15:45.187853+0000","last_active":"2024-01-15T18:15:45.196319+0000","last_peered":"2024-01-15T18:15:45.196319+0000","last_clean":"2024-01-15T18:15:45.196319+0000","last_became_active":"2024-01-15T18:15:45.187552+0000","last_became_peered":"2024-01-15T18:15:45.187552+0000","last_unstale":"2024-01-15T18:15:45.196319+0000","last_undegraded":"2024-01-15T18:15:45.196319+0000","last_fullsized":"2024-01-15T18:15:45.196319+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:56.553890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.162106+0000","last_change":"2024-01-15T18:15:44.155651+0000","last_active":"2024-01-15T18:15:45.162106+0000","last_peered":"2024-01-15T18:15:45.162106+0000","last_clean":"2024-01-15T18:15:45.162106+0000","last_became_active":"2024-01-15T18:15:44.155306+0000","last_became_peered":"2024-01-15T18:15:44.155306+0000","last_unstale":"2024-01-15T18:15:45.162106+0000","last_undegraded":"2024-01-15T18:15:45.162106+0000","last_fullsized":"2024-01-15T18:15:45.162106+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:06:15.812490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.189876+0000","last_change":"2024-01-15T18:15:45.184002+0000","last_active":"2024-01-15T18:15:45.189876+0000","last_peered":"2024-01-15T18:15:45.189876+0000","last_clean":"2024-01-15T18:15:45.189876+0000","last_became_active":"2024-01-15T18:15:45.183624+0000","last_became_peered":"2024-01-15T18:15:45.183624+0000","last_unstale":"2024-01-15T18:15:45.189876+0000","last_undegraded":"2024-01-15T18:15:45.189876+0000","last_fullsized":"2024-01-15T18:15:45.189876+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:03:29.579452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.186739+0000","last_change":"2024-01-15T18:15:44.160123+0000","last_active":"2024-01-15T18:15:45.186739+0000","last_peered":"2024-01-15T18:15:45.186739+0000","last_clean":"2024-01-15T18:15:45.186739+0000","last_became_active":"2024-01-15T18:15:44.159761+0000","last_became_peered":"2024-01-15T18:15:44.159761+0000","last_unstale":"2024-01-15T18:15:45.186739+0000","last_undegraded":"2024-01-15T18:15:45.186739+0000","last_fullsized":"2024-01-15T18:15:45.186739+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:42:00.915026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.058938+0000","last_change":"2024-01-15T18:15:45.193928+0000","last_active":"2024-01-15T18:15:47.058938+0000","last_peered":"2024-01-15T18:15:47.058938+0000","last_clean":"2024-01-15T18:15:47.058938+0000","last_became_active":"2024-01-15T18:15:45.193656+0000","last_became_peered":"2024-01-15T18:15:45.193656+0000","last_unstale":"2024-01-15T18:15:47.058938+0000","last_undegraded":"2024-01-15T18:15:47.058938+0000","last_fullsized":"2024-01-15T18:15:47.058938+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:33.549022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.187555+0000","last_change":"2024-01-15T18:15:44.166336+0000","last_active":"2024-01-15T18:15:45.187555+0000","last_peered":"2024-01-15T18:15:45.187555+0000","last_clean":"2024-01-15T18:15:45.187555+0000","last_became_active":"2024-01-15T18:15:44.165403+0000","last_became_peered":"2024-01-15T18:15:44.165403+0000","last_unstale":"2024-01-15T18:15:45.187555+0000","last_undegraded":"2024-01-15T18:15:45.187555+0000","last_fullsized":"2024-01-15T18:15:45.187555+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:42:53.652872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.184288+0000","last_change":"2024-01-15T18:15:45.187664+0000","last_active":"2024-01-15T18:15:46.184288+0000","last_peered":"2024-01-15T18:15:46.184288+0000","last_clean":"2024-01-15T18:15:46.184288+0000","last_became_active":"2024-01-15T18:15:45.187401+0000","last_became_peered":"2024-01-15T18:15:45.187401+0000","last_unstale":"2024-01-15T18:15:46.184288+0000","last_undegraded":"2024-01-15T18:15:46.184288+0000","last_fullsized":"2024-01-15T18:15:46.184288+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:31:04.304785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"83'3","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.197477+0000","last_change":"2024-01-15T18:15:44.160184+0000","last_active":"2024-01-15T18:15:45.197477+0000","last_peered":"2024-01-15T18:15:45.197477+0000","last_clean":"2024-01-15T18:15:45.197477+0000","last_became_active":"2024-01-15T18:15:44.159855+0000","last_became_peered":"2024-01-15T18:15:44.159855+0000","last_unstale":"2024-01-15T18:15:45.197477+0000","last_undegraded":"2024-01-15T18:15:45.197477+0000","last_fullsized":"2024-01-15T18:15:45.197477+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:04:08.736438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.192668+0000","last_change":"2024-01-15T18:15:45.187059+0000","last_active":"2024-01-15T18:15:45.192668+0000","last_peered":"2024-01-15T18:15:45.192668+0000","last_clean":"2024-01-15T18:15:45.192668+0000","last_became_active":"2024-01-15T18:15:45.184830+0000","last_became_peered":"2024-01-15T18:15:45.184830+0000","last_unstale":"2024-01-15T18:15:45.192668+0000","last_undegraded":"2024-01-15T18:15:45.192668+0000","last_fullsized":"2024-01-15T18:15:45.192668+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:58:35.413117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"83'6","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198837+0000","last_change":"2024-01-15T18:15:44.158973+0000","last_active":"2024-01-15T18:15:45.198837+0000","last_peered":"2024-01-15T18:15:45.198837+0000","last_clean":"2024-01-15T18:15:45.198837+0000","last_became_active":"2024-01-15T18:15:44.158684+0000","last_became_peered":"2024-01-15T18:15:44.158684+0000","last_unstale":"2024-01-15T18:15:45.198837+0000","last_undegraded":"2024-01-15T18:15:45.198837+0000","last_fullsized":"2024-01-15T18:15:45.198837+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:03:51.252813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.184115+0000","last_change":"2024-01-15T18:15:45.196327+0000","last_active":"2024-01-15T18:15:46.184115+0000","last_peered":"2024-01-15T18:15:46.184115+0000","last_clean":"2024-01-15T18:15:46.184115+0000","last_became_active":"2024-01-15T18:15:45.195928+0000","last_became_peered":"2024-01-15T18:15:45.195928+0000","last_unstale":"2024-01-15T18:15:46.184115+0000","last_undegraded":"2024-01-15T18:15:46.184115+0000","last_fullsized":"2024-01-15T18:15:46.184115+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:57:47.746767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.201845+0000","last_change":"2024-01-15T18:15:44.166340+0000","last_active":"2024-01-15T18:15:45.201845+0000","last_peered":"2024-01-15T18:15:45.201845+0000","last_clean":"2024-01-15T18:15:45.201845+0000","last_became_active":"2024-01-15T18:15:44.165947+0000","last_became_peered":"2024-01-15T18:15:44.165947+0000","last_unstale":"2024-01-15T18:15:45.201845+0000","last_undegraded":"2024-01-15T18:15:45.201845+0000","last_fullsized":"2024-01-15T18:15:45.201845+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:12:37.715165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.199984+0000","last_change":"2024-01-15T18:15:45.199248+0000","last_active":"2024-01-15T18:15:45.199984+0000","last_peered":"2024-01-15T18:15:45.199984+0000","last_clean":"2024-01-15T18:15:45.199984+0000","last_became_active":"2024-01-15T18:15:45.198929+0000","last_became_peered":"2024-01-15T18:15:45.198929+0000","last_unstale":"2024-01-15T18:15:45.199984+0000","last_undegraded":"2024-01-15T18:15:45.199984+0000","last_fullsized":"2024-01-15T18:15:45.199984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:26:01.224733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.182090+0000","last_change":"2024-01-15T18:15:44.161901+0000","last_active":"2024-01-15T18:15:45.182090+0000","last_peered":"2024-01-15T18:15:45.182090+0000","last_clean":"2024-01-15T18:15:45.182090+0000","last_became_active":"2024-01-15T18:15:44.159756+0000","last_became_peered":"2024-01-15T18:15:44.159756+0000","last_unstale":"2024-01-15T18:15:45.182090+0000","last_undegraded":"2024-01-15T18:15:45.182090+0000","last_fullsized":"2024-01-15T18:15:45.182090+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:41:43.148064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.180743+0000","last_change":"2024-01-15T18:15:45.174549+0000","last_active":"2024-01-15T18:15:45.180743+0000","last_peered":"2024-01-15T18:15:45.180743+0000","last_clean":"2024-01-15T18:15:45.180743+0000","last_became_active":"2024-01-15T18:15:45.174046+0000","last_became_peered":"2024-01-15T18:15:45.174046+0000","last_unstale":"2024-01-15T18:15:45.180743+0000","last_undegraded":"2024-01-15T18:15:45.180743+0000","last_fullsized":"2024-01-15T18:15:45.180743+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:12:42.277859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.167820+0000","last_change":"2024-01-15T18:15:44.154875+0000","last_active":"2024-01-15T18:15:45.167820+0000","last_peered":"2024-01-15T18:15:45.167820+0000","last_clean":"2024-01-15T18:15:45.167820+0000","last_became_active":"2024-01-15T18:15:44.154585+0000","last_became_peered":"2024-01-15T18:15:44.154585+0000","last_unstale":"2024-01-15T18:15:45.167820+0000","last_undegraded":"2024-01-15T18:15:45.167820+0000","last_fullsized":"2024-01-15T18:15:45.167820+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:58:47.607676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198412+0000","last_change":"2024-01-15T18:15:45.193330+0000","last_active":"2024-01-15T18:15:45.198412+0000","last_peered":"2024-01-15T18:15:45.198412+0000","last_clean":"2024-01-15T18:15:45.198412+0000","last_became_active":"2024-01-15T18:15:45.192354+0000","last_became_peered":"2024-01-15T18:15:45.192354+0000","last_unstale":"2024-01-15T18:15:45.198412+0000","last_undegraded":"2024-01-15T18:15:45.198412+0000","last_fullsized":"2024-01-15T18:15:45.198412+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:12:58.057727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"83'8","reported_seq":20,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.202181+0000","last_change":"2024-01-15T18:15:44.168821+0000","last_active":"2024-01-15T18:15:45.202181+0000","last_peered":"2024-01-15T18:15:45.202181+0000","last_clean":"2024-01-15T18:15:45.202181+0000","last_became_active":"2024-01-15T18:15:44.168551+0000","last_became_peered":"2024-01-15T18:15:44.168551+0000","last_unstale":"2024-01-15T18:15:45.202181+0000","last_undegraded":"2024-01-15T18:15:45.202181+0000","last_fullsized":"2024-01-15T18:15:45.202181+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:32.543776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191678+0000","last_change":"2024-01-15T18:15:45.180405+0000","last_active":"2024-01-15T18:15:45.191678+0000","last_peered":"2024-01-15T18:15:45.191678+0000","last_clean":"2024-01-15T18:15:45.191678+0000","last_became_active":"2024-01-15T18:15:45.179918+0000","last_became_peered":"2024-01-15T18:15:45.179918+0000","last_unstale":"2024-01-15T18:15:45.191678+0000","last_undegraded":"2024-01-15T18:15:45.191678+0000","last_fullsized":"2024-01-15T18:15:45.191678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:01:49.697849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.192261+0000","last_change":"2024-01-15T18:15:44.159443+0000","last_active":"2024-01-15T18:15:45.192261+0000","last_peered":"2024-01-15T18:15:45.192261+0000","last_clean":"2024-01-15T18:15:45.192261+0000","last_became_active":"2024-01-15T18:15:44.159118+0000","last_became_peered":"2024-01-15T18:15:44.159118+0000","last_unstale":"2024-01-15T18:15:45.192261+0000","last_undegraded":"2024-01-15T18:15:45.192261+0000","last_fullsized":"2024-01-15T18:15:45.192261+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:13:11.055744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198173+0000","last_change":"2024-01-15T18:15:45.195167+0000","last_active":"2024-01-15T18:15:45.198173+0000","last_peered":"2024-01-15T18:15:45.198173+0000","last_clean":"2024-01-15T18:15:45.198173+0000","last_became_active":"2024-01-15T18:15:45.194897+0000","last_became_peered":"2024-01-15T18:15:45.194897+0000","last_unstale":"2024-01-15T18:15:45.198173+0000","last_undegraded":"2024-01-15T18:15:45.198173+0000","last_fullsized":"2024-01-15T18:15:45.198173+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:53:25.231875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"83'2","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198962+0000","last_change":"2024-01-15T18:15:44.169724+0000","last_active":"2024-01-15T18:15:45.198962+0000","last_peered":"2024-01-15T18:15:45.198962+0000","last_clean":"2024-01-15T18:15:45.198962+0000","last_became_active":"2024-01-15T18:15:44.169420+0000","last_became_peered":"2024-01-15T18:15:44.169420+0000","last_unstale":"2024-01-15T18:15:45.198962+0000","last_undegraded":"2024-01-15T18:15:45.198962+0000","last_fullsized":"2024-01-15T18:15:45.198962+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:04:34.325987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.185177+0000","last_change":"2024-01-15T18:15:45.175568+0000","last_active":"2024-01-15T18:15:45.185177+0000","last_peered":"2024-01-15T18:15:45.185177+0000","last_clean":"2024-01-15T18:15:45.185177+0000","last_became_active":"2024-01-15T18:15:45.173399+0000","last_became_peered":"2024-01-15T18:15:45.173399+0000","last_unstale":"2024-01-15T18:15:45.185177+0000","last_undegraded":"2024-01-15T18:15:45.185177+0000","last_fullsized":"2024-01-15T18:15:45.185177+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:42:54.658117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.170840+0000","last_change":"2024-01-15T18:15:44.168665+0000","last_active":"2024-01-15T18:15:45.170840+0000","last_peered":"2024-01-15T18:15:45.170840+0000","last_clean":"2024-01-15T18:15:45.170840+0000","last_became_active":"2024-01-15T18:15:44.168375+0000","last_became_peered":"2024-01-15T18:15:44.168375+0000","last_unstale":"2024-01-15T18:15:45.170840+0000","last_undegraded":"2024-01-15T18:15:45.170840+0000","last_fullsized":"2024-01-15T18:15:45.170840+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:30:48.177858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191116+0000","last_change":"2024-01-15T18:15:45.179812+0000","last_active":"2024-01-15T18:15:45.191116+0000","last_peered":"2024-01-15T18:15:45.191116+0000","last_clean":"2024-01-15T18:15:45.191116+0000","last_became_active":"2024-01-15T18:15:45.179148+0000","last_became_peered":"2024-01-15T18:15:45.179148+0000","last_unstale":"2024-01-15T18:15:45.191116+0000","last_undegraded":"2024-01-15T18:15:45.191116+0000","last_fullsized":"2024-01-15T18:15:45.191116+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:47:52.953626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196980+0000","last_change":"2024-01-15T18:15:44.163128+0000","last_active":"2024-01-15T18:15:45.196980+0000","last_peered":"2024-01-15T18:15:45.196980+0000","last_clean":"2024-01-15T18:15:45.196980+0000","last_became_active":"2024-01-15T18:15:44.162839+0000","last_became_peered":"2024-01-15T18:15:44.162839+0000","last_unstale":"2024-01-15T18:15:45.196980+0000","last_undegraded":"2024-01-15T18:15:45.196980+0000","last_fullsized":"2024-01-15T18:15:45.196980+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:34:15.516676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.058582+0000","last_change":"2024-01-15T18:15:45.188550+0000","last_active":"2024-01-15T18:15:47.058582+0000","last_peered":"2024-01-15T18:15:47.058582+0000","last_clean":"2024-01-15T18:15:47.058582+0000","last_became_active":"2024-01-15T18:15:45.188224+0000","last_became_peered":"2024-01-15T18:15:45.188224+0000","last_unstale":"2024-01-15T18:15:47.058582+0000","last_undegraded":"2024-01-15T18:15:47.058582+0000","last_fullsized":"2024-01-15T18:15:47.058582+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:04:35.331233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.174831+0000","last_change":"2024-01-15T18:15:44.167859+0000","last_active":"2024-01-15T18:15:45.174831+0000","last_peered":"2024-01-15T18:15:45.174831+0000","last_clean":"2024-01-15T18:15:45.174831+0000","last_became_active":"2024-01-15T18:15:44.167592+0000","last_became_peered":"2024-01-15T18:15:44.167592+0000","last_unstale":"2024-01-15T18:15:45.174831+0000","last_undegraded":"2024-01-15T18:15:45.174831+0000","last_fullsized":"2024-01-15T18:15:45.174831+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:02:39.986413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.190664+0000","last_change":"2024-01-15T18:15:45.179572+0000","last_active":"2024-01-15T18:15:45.190664+0000","last_peered":"2024-01-15T18:15:45.190664+0000","last_clean":"2024-01-15T18:15:45.190664+0000","last_became_active":"2024-01-15T18:15:45.178802+0000","last_became_peered":"2024-01-15T18:15:45.178802+0000","last_unstale":"2024-01-15T18:15:45.190664+0000","last_undegraded":"2024-01-15T18:15:45.190664+0000","last_fullsized":"2024-01-15T18:15:45.190664+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:26:43.360415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.173615+0000","last_change":"2024-01-15T18:15:44.168997+0000","last_active":"2024-01-15T18:15:45.173615+0000","last_peered":"2024-01-15T18:15:45.173615+0000","last_clean":"2024-01-15T18:15:45.173615+0000","last_became_active":"2024-01-15T18:15:44.168678+0000","last_became_peered":"2024-01-15T18:15:44.168678+0000","last_unstale":"2024-01-15T18:15:45.173615+0000","last_undegraded":"2024-01-15T18:15:45.173615+0000","last_fullsized":"2024-01-15T18:15:45.173615+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:20:19.634573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.058442+0000","last_change":"2024-01-15T18:15:45.180279+0000","last_active":"2024-01-15T18:15:47.058442+0000","last_peered":"2024-01-15T18:15:47.058442+0000","last_clean":"2024-01-15T18:15:47.058442+0000","last_became_active":"2024-01-15T18:15:45.177474+0000","last_became_peered":"2024-01-15T18:15:45.177474+0000","last_unstale":"2024-01-15T18:15:47.058442+0000","last_undegraded":"2024-01-15T18:15:47.058442+0000","last_fullsized":"2024-01-15T18:15:47.058442+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:08:41.639657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.178682+0000","last_change":"2024-01-15T18:15:44.162077+0000","last_active":"2024-01-15T18:15:45.178682+0000","last_peered":"2024-01-15T18:15:45.178682+0000","last_clean":"2024-01-15T18:15:45.178682+0000","last_became_active":"2024-01-15T18:15:44.161794+0000","last_became_peered":"2024-01-15T18:15:44.161794+0000","last_unstale":"2024-01-15T18:15:45.178682+0000","last_undegraded":"2024-01-15T18:15:45.178682+0000","last_fullsized":"2024-01-15T18:15:45.178682+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:05:24.652003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.185502+0000","last_change":"2024-01-15T18:15:45.175862+0000","last_active":"2024-01-15T18:15:45.185502+0000","last_peered":"2024-01-15T18:15:45.185502+0000","last_clean":"2024-01-15T18:15:45.185502+0000","last_became_active":"2024-01-15T18:15:45.175569+0000","last_became_peered":"2024-01-15T18:15:45.175569+0000","last_unstale":"2024-01-15T18:15:45.185502+0000","last_undegraded":"2024-01-15T18:15:45.185502+0000","last_fullsized":"2024-01-15T18:15:45.185502+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:22:08.309890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"83'2","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.193674+0000","last_change":"2024-01-15T18:15:44.168092+0000","last_active":"2024-01-15T18:15:45.193674+0000","last_peered":"2024-01-15T18:15:45.193674+0000","last_clean":"2024-01-15T18:15:45.193674+0000","last_became_active":"2024-01-15T18:15:44.167828+0000","last_became_peered":"2024-01-15T18:15:44.167828+0000","last_unstale":"2024-01-15T18:15:45.193674+0000","last_undegraded":"2024-01-15T18:15:45.193674+0000","last_fullsized":"2024-01-15T18:15:45.193674+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:27:26.285133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198128+0000","last_change":"2024-01-15T18:15:45.194246+0000","last_active":"2024-01-15T18:15:45.198128+0000","last_peered":"2024-01-15T18:15:45.198128+0000","last_clean":"2024-01-15T18:15:45.198128+0000","last_became_active":"2024-01-15T18:15:45.193864+0000","last_became_peered":"2024-01-15T18:15:45.193864+0000","last_unstale":"2024-01-15T18:15:45.198128+0000","last_undegraded":"2024-01-15T18:15:45.198128+0000","last_fullsized":"2024-01-15T18:15:45.198128+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:05:25.657249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.162597+0000","last_change":"2024-01-15T18:15:44.166561+0000","last_active":"2024-01-15T18:15:45.162597+0000","last_peered":"2024-01-15T18:15:45.162597+0000","last_clean":"2024-01-15T18:15:45.162597+0000","last_became_active":"2024-01-15T18:15:44.166280+0000","last_became_peered":"2024-01-15T18:15:44.166280+0000","last_unstale":"2024-01-15T18:15:45.162597+0000","last_undegraded":"2024-01-15T18:15:45.162597+0000","last_fullsized":"2024-01-15T18:15:45.162597+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:30.074069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198142+0000","last_change":"2024-01-15T18:15:45.196412+0000","last_active":"2024-01-15T18:15:45.198142+0000","last_peered":"2024-01-15T18:15:45.198142+0000","last_clean":"2024-01-15T18:15:45.198142+0000","last_became_active":"2024-01-15T18:15:45.196022+0000","last_became_peered":"2024-01-15T18:15:45.196022+0000","last_unstale":"2024-01-15T18:15:45.198142+0000","last_undegraded":"2024-01-15T18:15:45.198142+0000","last_fullsized":"2024-01-15T18:15:45.198142+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:09:32.875279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.184853+0000","last_change":"2024-01-15T18:15:44.163120+0000","last_active":"2024-01-15T18:15:45.184853+0000","last_peered":"2024-01-15T18:15:45.184853+0000","last_clean":"2024-01-15T18:15:45.184853+0000","last_became_active":"2024-01-15T18:15:44.162805+0000","last_became_peered":"2024-01-15T18:15:44.162805+0000","last_unstale":"2024-01-15T18:15:45.184853+0000","last_undegraded":"2024-01-15T18:15:45.184853+0000","last_fullsized":"2024-01-15T18:15:45.184853+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:49.344761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191100+0000","last_change":"2024-01-15T18:15:45.185283+0000","last_active":"2024-01-15T18:15:45.191100+0000","last_peered":"2024-01-15T18:15:45.191100+0000","last_clean":"2024-01-15T18:15:45.191100+0000","last_became_active":"2024-01-15T18:15:45.185001+0000","last_became_peered":"2024-01-15T18:15:45.185001+0000","last_unstale":"2024-01-15T18:15:45.191100+0000","last_undegraded":"2024-01-15T18:15:45.191100+0000","last_fullsized":"2024-01-15T18:15:45.191100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:41:05.797910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.183836+0000","last_change":"2024-01-15T18:15:44.163840+0000","last_active":"2024-01-15T18:15:45.183836+0000","last_peered":"2024-01-15T18:15:45.183836+0000","last_clean":"2024-01-15T18:15:45.183836+0000","last_became_active":"2024-01-15T18:15:44.163441+0000","last_became_peered":"2024-01-15T18:15:44.163441+0000","last_unstale":"2024-01-15T18:15:45.183836+0000","last_undegraded":"2024-01-15T18:15:45.183836+0000","last_fullsized":"2024-01-15T18:15:45.183836+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:41:04.792664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.058739+0000","last_change":"2024-01-15T18:15:45.187893+0000","last_active":"2024-01-15T18:15:47.058739+0000","last_peered":"2024-01-15T18:15:47.058739+0000","last_clean":"2024-01-15T18:15:47.058739+0000","last_became_active":"2024-01-15T18:15:45.187603+0000","last_became_peered":"2024-01-15T18:15:45.187603+0000","last_unstale":"2024-01-15T18:15:47.058739+0000","last_undegraded":"2024-01-15T18:15:47.058739+0000","last_fullsized":"2024-01-15T18:15:47.058739+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:32.310673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.170849+0000","last_change":"2024-01-15T18:15:44.169437+0000","last_active":"2024-01-15T18:15:45.170849+0000","last_peered":"2024-01-15T18:15:45.170849+0000","last_clean":"2024-01-15T18:15:45.170849+0000","last_became_active":"2024-01-15T18:15:44.169170+0000","last_became_peered":"2024-01-15T18:15:44.169170+0000","last_unstale":"2024-01-15T18:15:45.170849+0000","last_undegraded":"2024-01-15T18:15:45.170849+0000","last_fullsized":"2024-01-15T18:15:45.170849+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:08:40.634411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.184432+0000","last_change":"2024-01-15T18:15:45.183921+0000","last_active":"2024-01-15T18:15:46.184432+0000","last_peered":"2024-01-15T18:15:46.184432+0000","last_clean":"2024-01-15T18:15:46.184432+0000","last_became_active":"2024-01-15T18:15:45.183090+0000","last_became_peered":"2024-01-15T18:15:45.183090+0000","last_unstale":"2024-01-15T18:15:46.184432+0000","last_undegraded":"2024-01-15T18:15:46.184432+0000","last_fullsized":"2024-01-15T18:15:46.184432+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:21:19.894599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.164910+0000","last_change":"2024-01-15T18:15:44.154568+0000","last_active":"2024-01-15T18:15:45.164910+0000","last_peered":"2024-01-15T18:15:45.164910+0000","last_clean":"2024-01-15T18:15:45.164910+0000","last_became_active":"2024-01-15T18:15:44.154277+0000","last_became_peered":"2024-01-15T18:15:44.154277+0000","last_unstale":"2024-01-15T18:15:45.164910+0000","last_undegraded":"2024-01-15T18:15:45.164910+0000","last_fullsized":"2024-01-15T18:15:45.164910+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:39:47.546718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.191500+0000","last_change":"2024-01-15T18:15:45.182784+0000","last_active":"2024-01-15T18:15:45.191500+0000","last_peered":"2024-01-15T18:15:45.191500+0000","last_clean":"2024-01-15T18:15:45.191500+0000","last_became_active":"2024-01-15T18:15:45.179168+0000","last_became_peered":"2024-01-15T18:15:45.179168+0000","last_unstale":"2024-01-15T18:15:45.191500+0000","last_undegraded":"2024-01-15T18:15:45.191500+0000","last_fullsized":"2024-01-15T18:15:45.191500+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:34:16.521921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.183666+0000","last_change":"2024-01-15T18:15:44.160891+0000","last_active":"2024-01-15T18:15:45.183666+0000","last_peered":"2024-01-15T18:15:45.183666+0000","last_clean":"2024-01-15T18:15:45.183666+0000","last_became_active":"2024-01-15T18:15:44.160625+0000","last_became_peered":"2024-01-15T18:15:44.160625+0000","last_unstale":"2024-01-15T18:15:45.183666+0000","last_undegraded":"2024-01-15T18:15:45.183666+0000","last_fullsized":"2024-01-15T18:15:45.183666+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:52.489949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196072+0000","last_change":"2024-01-15T18:15:45.192344+0000","last_active":"2024-01-15T18:15:45.196072+0000","last_peered":"2024-01-15T18:15:45.196072+0000","last_clean":"2024-01-15T18:15:45.196072+0000","last_became_active":"2024-01-15T18:15:45.191845+0000","last_became_peered":"2024-01-15T18:15:45.191845+0000","last_unstale":"2024-01-15T18:15:45.196072+0000","last_undegraded":"2024-01-15T18:15:45.196072+0000","last_fullsized":"2024-01-15T18:15:45.196072+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:01:31.532362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.163378+0000","last_change":"2024-01-15T18:15:44.156428+0000","last_active":"2024-01-15T18:15:45.163378+0000","last_peered":"2024-01-15T18:15:45.163378+0000","last_clean":"2024-01-15T18:15:45.163378+0000","last_became_active":"2024-01-15T18:15:44.156132+0000","last_became_peered":"2024-01-15T18:15:44.156132+0000","last_unstale":"2024-01-15T18:15:45.163378+0000","last_undegraded":"2024-01-15T18:15:45.163378+0000","last_fullsized":"2024-01-15T18:15:45.163378+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:49.344761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.186454+0000","last_change":"2024-01-15T18:15:45.178846+0000","last_active":"2024-01-15T18:15:45.186454+0000","last_peered":"2024-01-15T18:15:45.186454+0000","last_clean":"2024-01-15T18:15:45.186454+0000","last_became_active":"2024-01-15T18:15:45.178552+0000","last_became_peered":"2024-01-15T18:15:45.178552+0000","last_unstale":"2024-01-15T18:15:45.186454+0000","last_undegraded":"2024-01-15T18:15:45.186454+0000","last_fullsized":"2024-01-15T18:15:45.186454+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:45:31.736245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"83'1","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.183987+0000","last_change":"2024-01-15T18:15:44.170831+0000","last_active":"2024-01-15T18:15:45.183987+0000","last_peered":"2024-01-15T18:15:45.183987+0000","last_clean":"2024-01-15T18:15:45.183987+0000","last_became_active":"2024-01-15T18:15:44.170489+0000","last_became_peered":"2024-01-15T18:15:44.170489+0000","last_unstale":"2024-01-15T18:15:45.183987+0000","last_undegraded":"2024-01-15T18:15:45.183987+0000","last_fullsized":"2024-01-15T18:15:45.183987+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:54:03.053923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":218,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-01-15T18:15:44.162031+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:15:44.162031+0000","last_peered":"2024-01-15T18:15:44.162031+0000","last_clean":"2024-01-15T18:15:44.162031+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:15:44.162031+0000","last_undegraded":"2024-01-15T18:15:44.162031+0000","last_fullsized":"2024-01-15T18:15:44.162031+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509575,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77092,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294748,"statfs":{"total":96636764160,"available":96557821952,"internally_reserved":0,"allocated":1593344,"data_stored":1017889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77330890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":1.171,"15min":0.874},"min":{"1min":0.52400000000000002,"5min":0.38100000000000001,"15min":0.36199999999999999},"max":{"1min":1.0089999999999999,"5min":9.6989999999999998,"15min":10.284000000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.68799999999999994,"5min":1.1399999999999999,"15min":0.90000000000000002},"min":{"1min":0.48599999999999999,"5min":0.39100000000000001,"15min":0.378},"max":{"1min":0.83099999999999996,"5min":16.968,"15min":16.968},"last":6.6189999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":1.3129999999999999,"15min":0.91400000000000003},"min":{"1min":0.52600000000000002,"5min":0.36099999999999999,"15min":0.33800000000000002},"max":{"1min":0.90100000000000002,"5min":15.791,"15min":15.791},"last":4.944},{"interface":"front","average":{"1min":0.71099999999999997,"5min":1.109,"15min":0.90000000000000002},"min":{"1min":0.51500000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.85799999999999998,"5min":10.842000000000001,"15min":10.842000000000001},"last":4.6699999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":1.2609999999999999,"15min":0.92500000000000004},"min":{"1min":0.502,"5min":0.40400000000000003,"15min":0.378},"max":{"1min":0.98299999999999998,"5min":22.318000000000001,"15min":22.318000000000001},"last":4.4669999999999996},{"interface":"front","average":{"1min":0.76200000000000001,"5min":1.159,"15min":0.89400000000000002},"min":{"1min":0.58399999999999996,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.93700000000000006,"5min":9.7539999999999996,"15min":9.7539999999999996},"last":4.7030000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":1.387,"15min":0.98899999999999999},"min":{"1min":0.59799999999999998,"5min":0.47099999999999997,"15min":0.38100000000000001},"max":{"1min":0.93300000000000005,"5min":18.152999999999999,"15min":18.152999999999999},"last":1.456},{"interface":"front","average":{"1min":0.81599999999999995,"5min":1.274,"15min":0.98599999999999999},"min":{"1min":0.622,"5min":0.433,"15min":0.43099999999999999},"max":{"1min":0.96599999999999997,"5min":9.7959999999999994,"15min":10.34},"last":2.6909999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":1.2150000000000001,"15min":0.92700000000000005},"min":{"1min":0.44,"5min":0.314,"15min":0.314},"max":{"1min":0.85199999999999998,"5min":20.387,"15min":20.387},"last":4.4480000000000004},{"interface":"front","average":{"1min":0.623,"5min":1.353,"15min":0.98399999999999999},"min":{"1min":0.47099999999999997,"5min":0.38800000000000001,"15min":0.36099999999999999},"max":{"1min":1.032,"5min":17.215,"15min":17.215},"last":3.6309999999999998}]},{"osd":5,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":1.2410000000000001,"15min":0.94499999999999995},"min":{"1min":0.46400000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.95299999999999996,"5min":9.8450000000000006,"15min":19.959},"last":4.8639999999999999},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1.278,"15min":0.97699999999999998},"min":{"1min":0.439,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83999999999999997,"5min":19.905000000000001,"15min":19.905000000000001},"last":4.3959999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":1.4470000000000001,"15min":1.1499999999999999},"min":{"1min":0.59499999999999997,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97099999999999997,"5min":20.492000000000001,"15min":22.478999999999999},"last":4.3330000000000002},{"interface":"front","average":{"1min":0.749,"5min":1.272,"15min":1.0600000000000001},"min":{"1min":0.52200000000000002,"5min":0.44,"15min":0.38300000000000001},"max":{"1min":0.998,"5min":17.132000000000001,"15min":21.509},"last":3.573}]}]},{"osd":6,"up_from":15,"seq":64424509575,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58524,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313316,"statfs":{"total":96636764160,"available":96576835584,"internally_reserved":0,"allocated":999424,"data_stored":427521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":1.3009999999999999,"15min":0.95699999999999996},"min":{"1min":0.55700000000000005,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.96099999999999997,"5min":32.587000000000003,"15min":32.587000000000003},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.75,"5min":1.0660000000000001,"15min":0.86199999999999999},"min":{"1min":0.49199999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.97799999999999998,"5min":16.193000000000001,"15min":16.193000000000001},"last":0.69399999999999995}]},{"osd":1,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.92400000000000004,"15min":0.98699999999999999},"min":{"1min":0.439,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.90700000000000003,"5min":7.7649999999999997,"15min":53.451999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.72499999999999998,"5min":1.2090000000000001,"15min":0.92600000000000005},"min":{"1min":0.53400000000000003,"5min":0.437,"15min":0.42899999999999999},"max":{"1min":0.86299999999999999,"5min":32.512,"15min":32.512},"last":0.51700000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":1.24,"15min":0.96799999999999997},"min":{"1min":0.58399999999999996,"5min":0.41699999999999998,"15min":0.38900000000000001},"max":{"1min":0.92200000000000004,"5min":32.472999999999999,"15min":32.472999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.754,"5min":1.0649999999999999,"15min":0.88500000000000001},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.38500000000000001},"max":{"1min":1.02,"5min":16.602,"15min":16.602},"last":0.65300000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":1.1699999999999999,"15min":0.93600000000000005},"min":{"1min":0.56399999999999995,"5min":0.46000000000000002,"15min":0.41399999999999998},"max":{"1min":0.92100000000000004,"5min":16.388000000000002,"15min":16.388000000000002},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.0680000000000001,"15min":1.075},"min":{"1min":0.54400000000000004,"5min":0.432,"15min":0.41599999999999998},"max":{"1min":0.95999999999999996,"5min":9.3719999999999999,"15min":53.398000000000003},"last":0.67200000000000004}]},{"osd":4,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":1.149,"15min":0.89500000000000002},"min":{"1min":0.435,"5min":0.38100000000000001,"15min":0.34999999999999998},"max":{"1min":0.79700000000000004,"5min":32.549999999999997,"15min":32.549999999999997},"last":0.63},{"interface":"front","average":{"1min":0.59499999999999997,"5min":1.0409999999999999,"15min":0.84899999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.34499999999999997},"max":{"1min":0.96499999999999997,"5min":11.852,"15min":11.852},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":1.0720000000000001,"15min":1.034},"min":{"1min":0.42699999999999999,"5min":0.35799999999999998,"15min":0.30399999999999999},"max":{"1min":0.77300000000000002,"5min":13.087999999999999,"15min":53.229999999999997},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.68700000000000006,"5min":1.375,"15min":1.0349999999999999},"min":{"1min":0.42599999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.0009999999999999,"5min":32.404000000000003,"15min":32.404000000000003},"last":0.71899999999999997}]},{"osd":7,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":1.294,"15min":1.1200000000000001},"min":{"1min":0.52900000000000003,"5min":0.436,"15min":0.40400000000000003},"max":{"1min":0.94099999999999995,"5min":23.991,"15min":45.070999999999998},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":1.3260000000000001,"15min":1.2170000000000001},"min":{"1min":0.51200000000000001,"5min":0.40999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88200000000000001,"5min":40.972999999999999,"15min":53.338000000000001},"last":0.753}]}]},{"osd":4,"up_from":15,"seq":64424509575,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58528,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313312,"statfs":{"total":96636764160,"available":96576831488,"internally_reserved":0,"allocated":999424,"data_stored":427521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":1.111,"15min":0.88500000000000001},"min":{"1min":0.46000000000000002,"5min":0.437,"15min":0.32900000000000001},"max":{"1min":0.96199999999999997,"5min":13.492000000000001,"15min":23.318000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.92100000000000004,"15min":0.80200000000000005},"min":{"1min":0.41999999999999998,"5min":0.378,"15min":0.32600000000000001},"max":{"1min":0.875,"5min":14.723000000000001,"15min":14.723000000000001},"last":0.71099999999999997}]},{"osd":1,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":1.0049999999999999,"15min":0.78800000000000003},"min":{"1min":0.50800000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.84299999999999997,"5min":9.5700000000000003,"15min":9.5700000000000003},"last":0.63},{"interface":"front","average":{"1min":0.71999999999999997,"5min":1.155,"15min":0.91400000000000003},"min":{"1min":0.47399999999999998,"5min":0.40200000000000002,"15min":0.36899999999999999},"max":{"1min":0.84999999999999998,"5min":13.57,"15min":22.408999999999999},"last":0.751}]},{"osd":2,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":1.038,"15min":0.79400000000000004},"min":{"1min":0.53000000000000003,"5min":0.41599999999999998,"15min":0.34399999999999997},"max":{"1min":0.84199999999999997,"5min":9.7089999999999996,"15min":9.7089999999999996},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.73499999999999999,"5min":1.2949999999999999,"15min":1.014},"min":{"1min":0.439,"5min":0.439,"15min":0.39100000000000001},"max":{"1min":0.92600000000000005,"5min":14.118,"15min":22.338999999999999},"last":0.69999999999999996}]},{"osd":3,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":1.016,"15min":0.873},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.41199999999999998},"max":{"1min":0.83199999999999996,"5min":14.763999999999999,"15min":14.763999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.77300000000000002,"5min":1.1479999999999999,"15min":0.90500000000000003},"min":{"1min":0.622,"5min":0.48099999999999998,"15min":0.45500000000000002},"max":{"1min":0.89400000000000002,"5min":9.6289999999999996,"15min":9.6289999999999996},"last":0.72999999999999998}]},{"osd":5,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":1.1299999999999999,"15min":0.96299999999999997},"min":{"1min":0.39000000000000001,"5min":0.34799999999999998,"15min":0.33200000000000002},"max":{"1min":0.875,"5min":15.787000000000001,"15min":31.617999999999999},"last":0.501},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.94699999999999995,"15min":0.86699999999999999},"min":{"1min":0.44,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.92300000000000004,"5min":16.388000000000002,"15min":17.145},"last":0.59899999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.98399999999999999,"15min":0.83099999999999996},"min":{"1min":0.42799999999999999,"5min":0.37,"15min":0.35499999999999998},"max":{"1min":0.94599999999999995,"5min":11.124000000000001,"15min":11.124000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.71699999999999997,"5min":1.3580000000000001,"15min":1.226},"min":{"1min":0.53700000000000003,"5min":0.434,"15min":0.40200000000000002},"max":{"1min":1.018,"5min":32.308,"15min":32.308},"last":0.55500000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":1.002,"15min":1.0109999999999999},"min":{"1min":0.49399999999999999,"5min":0.41099999999999998,"15min":0.34799999999999998},"max":{"1min":0.89900000000000002,"5min":14.653,"15min":39.256},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.746,"5min":1.2290000000000001,"15min":1.0129999999999999},"min":{"1min":0.57799999999999996,"5min":0.46500000000000002,"15min":0.42199999999999999},"max":{"1min":0.98599999999999999,"5min":16.277000000000001,"15min":26.573},"last":0.57399999999999995}]}]},{"osd":3,"up_from":15,"seq":64424509574,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80676,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94291164,"statfs":{"total":96636764160,"available":96554151936,"internally_reserved":0,"allocated":999424,"data_stored":427521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":81590727},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.59199999999999997,"15min":0.49099999999999999},"min":{"1min":0.29299999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.751,"5min":5.8849999999999998,"15min":5.8849999999999998},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.68999999999999995,"15min":0.61599999999999999},"min":{"1min":0.38600000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.77500000000000002,"5min":4.7990000000000004,"15min":17.756},"last":0.29799999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.73099999999999998,"15min":0.59299999999999997},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.621,"5min":12.307,"15min":12.307},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.69399999999999995,"15min":0.59299999999999997},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.32200000000000001},"max":{"1min":0.83799999999999997,"5min":6.0640000000000001,"15min":6.0640000000000001},"last":0.38600000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.83099999999999996,"15min":0.69599999999999995},"min":{"1min":0.46800000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":3.633,"5min":12.468999999999999,"15min":12.468999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.85399999999999998,"15min":0.68500000000000005},"min":{"1min":0.48599999999999999,"5min":0.41599999999999998,"15min":0.34599999999999997},"max":{"1min":1.0129999999999999,"5min":10.837999999999999,"15min":10.837999999999999},"last":0.47199999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":1.292,"15min":1.028},"min":{"1min":0.59499999999999997,"5min":0.375,"15min":0.375},"max":{"1min":1.014,"5min":13.863,"15min":17.853999999999999},"last":0.874},{"interface":"front","average":{"1min":0.82899999999999996,"5min":1.2809999999999999,"15min":0.96899999999999997},"min":{"1min":0.63,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.0469999999999999,"5min":13.31,"15min":13.31},"last":0.72099999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":1.369,"15min":0.97599999999999998},"min":{"1min":0.66500000000000004,"5min":0.52100000000000002,"15min":0.51000000000000001},"max":{"1min":0.94399999999999995,"5min":17.687999999999999,"15min":17.687999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.206,"15min":0.97399999999999998},"min":{"1min":0.51700000000000002,"5min":0.438,"15min":0.438},"max":{"1min":3.2160000000000002,"5min":20.515999999999998,"15min":20.515999999999998},"last":0.82099999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":1.1499999999999999,"15min":0.98999999999999999},"min":{"1min":0.55600000000000005,"5min":0.432,"15min":0.432},"max":{"1min":1.1060000000000001,"5min":13.250999999999999,"15min":23.100999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.82099999999999995,"5min":1.2150000000000001,"15min":0.93799999999999994},"min":{"1min":0.60799999999999998,"5min":0.5,"15min":0.49399999999999999},"max":{"1min":1.0169999999999999,"5min":12.76,"15min":12.76},"last":0.93400000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.7889999999999999,"15min":1.1719999999999999},"min":{"1min":0.625,"5min":0.48599999999999999,"15min":0.48199999999999998},"max":{"1min":0.99099999999999999,"5min":24.321999999999999,"15min":24.321999999999999},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.81999999999999995,"5min":1.329,"15min":1.028},"min":{"1min":0.58399999999999996,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0569999999999999,"5min":13.385,"15min":13.385},"last":0.78100000000000003}]}]},{"osd":2,"up_from":15,"seq":64424509574,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76532,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295308,"statfs":{"total":96636764160,"available":96558395392,"internally_reserved":0,"allocated":1015808,"data_stored":434962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.51900000000000002,"15min":0.51200000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26500000000000001},"max":{"1min":0.66500000000000004,"5min":1.857,"15min":5.1849999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54200000000000004,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.252,"15min":0.252},"max":{"1min":0.89300000000000002,"5min":2.3759999999999999,"15min":7.516},"last":0.55900000000000005}]},{"osd":1,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.70599999999999996,"15min":0.61599999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.875,"5min":10.907,"15min":10.907},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.68500000000000005,"15min":0.58799999999999997},"min":{"1min":0.38100000000000001,"5min":0.28100000000000003,"15min":0.27000000000000002},"max":{"1min":0.73799999999999999,"5min":7.2119999999999997,"15min":8.9190000000000005},"last":0.53500000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.60599999999999998,"15min":0.626},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.72999999999999998,"5min":2.7290000000000001,"15min":12.185},"last":0.621},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.747,"15min":0.66100000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.34000000000000002},"max":{"1min":0.96699999999999997,"5min":7.6760000000000002,"15min":7.6760000000000002},"last":0.67900000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.89500000000000002,"15min":0.85799999999999998},"min":{"1min":0.55700000000000005,"5min":0.47499999999999998,"15min":0.45800000000000002},"max":{"1min":1.1870000000000001,"5min":4.3150000000000004,"15min":8.6609999999999996},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.755,"5min":0.878,"15min":0.90900000000000003},"min":{"1min":0.5,"5min":0.45800000000000002,"15min":0.42499999999999999},"max":{"1min":1.075,"5min":8.3190000000000008,"15min":19.146999999999998},"last":0.84199999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.80400000000000005,"15min":0.90200000000000002},"min":{"1min":0.52400000000000002,"5min":0.442,"15min":0.44},"max":{"1min":1.1599999999999999,"5min":2.7789999999999999,"15min":31.710000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.996,"15min":1},"min":{"1min":0.54400000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.155,"5min":10.460000000000001,"15min":22.539000000000001},"last":0.77300000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.99399999999999999,"15min":0.89700000000000002},"min":{"1min":0.501,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.1220000000000001,"5min":8.3580000000000005,"15min":9.4540000000000006},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.83799999999999997,"15min":0.90500000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0489999999999999,"5min":2.669,"15min":18.614999999999998},"last":0.88}]},{"osd":7,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.125,"15min":0.95399999999999996},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.47999999999999998},"max":{"1min":1.157,"5min":11.606,"15min":11.606},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.92200000000000004,"15min":0.98699999999999999},"min":{"1min":0.60399999999999998,"5min":0.51700000000000002,"15min":0.502},"max":{"1min":1.1180000000000001,"5min":3.6850000000000001,"15min":17.013000000000002},"last":0.86199999999999999}]}]},{"osd":1,"up_from":15,"seq":64424509576,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80700,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94291140,"statfs":{"total":96636764160,"available":96554127360,"internally_reserved":0,"allocated":1024000,"data_stored":434574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81590725},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.70699999999999996,"15min":0.66300000000000003},"min":{"1min":0.26300000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.745,"5min":17.396999999999998,"15min":19.577000000000002},"last":0.623},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.74099999999999999,"15min":0.76000000000000001},"min":{"1min":0.33900000000000002,"5min":0.29199999999999998,"15min":0.251},"max":{"1min":0.83999999999999997,"5min":12.378,"15min":48.277999999999999},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.61399999999999999,"15min":0.71399999999999997},"min":{"1min":0.33600000000000002,"5min":0.309,"15min":0.30299999999999999},"max":{"1min":0.82799999999999996,"5min":2.8610000000000002,"15min":40.698},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.65900000000000003,"15min":0.90100000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.317},"max":{"1min":0.89500000000000002,"5min":3.0449999999999999,"15min":48.316000000000003},"last":0.441}]},{"osd":3,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.71799999999999997,"15min":0.84499999999999997},"min":{"1min":0.434,"5min":0.39800000000000002,"15min":0.33700000000000002},"max":{"1min":1.2509999999999999,"5min":3.0899999999999999,"15min":48.234999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.83799999999999997,"15min":0.68999999999999995},"min":{"1min":0.315,"5min":0.315,"15min":0.311},"max":{"1min":4.0780000000000003,"5min":9.9719999999999995,"15min":9.9719999999999995},"last":0.46000000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":1.0980000000000001,"15min":0.88700000000000001},"min":{"1min":0.49299999999999999,"5min":0.48999999999999999,"15min":0.44500000000000001},"max":{"1min":1.446,"5min":9.9329999999999998,"15min":9.9329999999999998},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.80800000000000005,"5min":1.026,"15min":0.93600000000000005},"min":{"1min":0.47499999999999998,"5min":0.45800000000000002,"15min":0.42499999999999999},"max":{"1min":1.163,"5min":9.8610000000000007,"15min":19.829000000000001},"last":0.88300000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.95399999999999996,"15min":1.0169999999999999},"min":{"1min":0.502,"5min":0.48599999999999999,"15min":0.46100000000000002},"max":{"1min":1.425,"5min":8.5839999999999996,"15min":22.594999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.82099999999999995,"5min":1.2629999999999999,"15min":1.163},"min":{"1min":0.58499999999999996,"5min":0.443,"15min":0.443},"max":{"1min":1.109,"5min":26.748999999999999,"15min":48.198},"last":0.76900000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":1.125,"15min":1.046},"min":{"1min":0.52200000000000002,"5min":0.50600000000000001,"15min":0.47499999999999998},"max":{"1min":1.879,"5min":9.6349999999999998,"15min":35.112000000000002},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.999,"15min":0.97599999999999998},"min":{"1min":0.58599999999999997,"5min":0.53100000000000003,"15min":0.50800000000000001},"max":{"1min":1.77,"5min":5.8849999999999998,"15min":19.780999999999999},"last":0.95199999999999996}]},{"osd":7,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3340000000000001,"15min":1.169},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.97499999999999998,"5min":20.872,"15min":48.148000000000003},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.85899999999999999,"5min":1.246,"15min":0.996},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.52100000000000002},"max":{"1min":1.52,"5min":9.8640000000000008,"15min":9.8640000000000008},"last":0.873}]}]},{"osd":5,"up_from":14,"seq":60129542280,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54404,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317436,"statfs":{"total":96636764160,"available":96581054464,"internally_reserved":0,"allocated":1040384,"data_stored":442398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.86699999999999999,"15min":0.80900000000000005},"min":{"1min":0.51800000000000002,"5min":0.29899999999999999,"15min":0.28999999999999998},"max":{"1min":3.8410000000000002,"5min":5.1639999999999997,"15min":21.385000000000002},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.0069999999999999,"15min":0.89600000000000002},"min":{"1min":0.57099999999999995,"5min":0.41599999999999998,"15min":0.34899999999999998},"max":{"1min":3.1080000000000001,"5min":9.359,"15min":9.359},"last":0.379}]},{"osd":1,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":1.3120000000000001,"15min":0.89600000000000002},"min":{"1min":0.55400000000000005,"5min":0.435,"15min":0.34899999999999998},"max":{"1min":1.0469999999999999,"5min":21.120000000000001,"15min":21.120000000000001},"last":0.502},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.2529999999999999,"15min":0.96499999999999997},"min":{"1min":0.628,"5min":0.38600000000000001,"15min":0.315},"max":{"1min":6.1310000000000002,"5min":15.843,"15min":21.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.98699999999999999,"15min":0.89800000000000002},"min":{"1min":0.51400000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":3.9220000000000002,"5min":6.2750000000000004,"15min":8.7750000000000004},"last":0.438},{"interface":"front","average":{"1min":1.135,"5min":1.3640000000000001,"15min":0.998},"min":{"1min":0.50800000000000001,"5min":0.42499999999999999,"15min":0.40400000000000003},"max":{"1min":8.3529999999999998,"5min":21.201000000000001,"15min":21.201000000000001},"last":0.57999999999999996}]},{"osd":3,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":0.98599999999999999,"15min":0.92200000000000004},"min":{"1min":0.60499999999999998,"5min":0.37,"15min":0.37},"max":{"1min":7.7199999999999998,"5min":7.7199999999999998,"15min":21.457000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.90300000000000002,"5min":1.034,"15min":0.97899999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.45400000000000001},"max":{"1min":3.8239999999999998,"5min":6.8650000000000002,"15min":8.7300000000000004},"last":0.54300000000000004}]},{"osd":4,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":1.0740000000000001,"15min":0.88800000000000001},"min":{"1min":0.51800000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.99199999999999999,"5min":10.962,"15min":10.962},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.69299999999999995,"5min":1.4410000000000001,"15min":1.034},"min":{"1min":0.48399999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0189999999999999,"5min":21.724,"15min":26.146999999999998},"last":0.46100000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.83899999999999997},"min":{"1min":0.44500000000000001,"5min":0.36099999999999999,"15min":0.32000000000000001},"max":{"1min":0.95599999999999996,"5min":5.2160000000000002,"15min":21.535},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.79000000000000004,"5min":1.012,"15min":1.0129999999999999},"min":{"1min":0.52400000000000002,"5min":0.379,"15min":0.36299999999999999},"max":{"1min":1.03,"5min":5.7539999999999996,"15min":30.488},"last":0.67500000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":2.0070000000000001,"15min":1.216},"min":{"1min":0.47699999999999998,"5min":0.42599999999999999,"15min":0.39700000000000002},"max":{"1min":1.008,"5min":33.499000000000002,"15min":33.499000000000002},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.81000000000000005,"5min":1.502,"15min":1.0880000000000001},"min":{"1min":0.54300000000000004,"5min":0.42899999999999999,"15min":0.36499999999999999},"max":{"1min":1.0609999999999999,"5min":25.437999999999999,"15min":25.437999999999999},"last":0.69399999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542280,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77120,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294720,"statfs":{"total":96636764160,"available":96557793280,"internally_reserved":0,"allocated":1622016,"data_stored":1031166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.90400000000000003,"15min":0.63},"min":{"1min":0.245,"5min":0.245,"15min":0.23699999999999999},"max":{"1min":0.78600000000000003,"5min":19.843,"15min":19.843},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.94699999999999995,"15min":0.68000000000000005},"min":{"1min":0.29999999999999999,"5min":0.29199999999999998,"15min":0.27300000000000002},"max":{"1min":0.746,"5min":10.154,"15min":10.154},"last":0.52200000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":1.2729999999999999,"15min":0.80400000000000005},"min":{"1min":0.39500000000000002,"5min":0.34899999999999998,"15min":0.30399999999999999},"max":{"1min":0.84099999999999997,"5min":25.905000000000001,"15min":25.905000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.82699999999999996,"15min":0.61199999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.26700000000000002},"max":{"1min":0.80000000000000004,"5min":9.3829999999999991,"15min":9.3829999999999991},"last":0.56799999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":1.026,"15min":0.74199999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.80400000000000005,"5min":9.1780000000000008,"15min":9.1780000000000008},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":1.016,"15min":0.76700000000000002},"min":{"1min":0.41699999999999998,"5min":0.40000000000000002,"15min":0.36899999999999999},"max":{"1min":0.89900000000000002,"5min":10.210000000000001,"15min":10.210000000000001},"last":0.49199999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":1.1319999999999999,"15min":0.98799999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.38},"max":{"1min":1.1459999999999999,"5min":10.691000000000001,"15min":22.814},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.80400000000000005,"5min":1.395,"15min":1.042},"min":{"1min":0.50600000000000001,"5min":0.49099999999999999,"15min":0.41199999999999998},"max":{"1min":1.046,"5min":23.341999999999999,"15min":23.341999999999999},"last":0.82699999999999996}]},{"osd":5,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":1.141,"15min":0.96899999999999997},"min":{"1min":0.55700000000000005,"5min":0.46899999999999997,"15min":0.441},"max":{"1min":1.046,"5min":10.737,"15min":12.428000000000001},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.83799999999999997,"5min":1.962,"15min":1.27},"min":{"1min":0.56999999999999995,"5min":0.498,"15min":0.46300000000000002},"max":{"1min":1.3049999999999999,"5min":23.021000000000001,"15min":23.021000000000001},"last":0.877}]},{"osd":6,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":1.696,"15min":1.218},"min":{"1min":0.52700000000000002,"5min":0.46700000000000003,"15min":0.46200000000000002},"max":{"1min":1.0229999999999999,"5min":26.036000000000001,"15min":31.059000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.88800000000000001,"5min":1.083,"15min":1.0920000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.49099999999999999},"max":{"1min":1.931,"5min":6.9690000000000003,"15min":40.731000000000002},"last":0.76700000000000002}]},{"osd":7,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.8400000000000001,"15min":1.208},"min":{"1min":0.58999999999999997,"5min":0.54900000000000004,"15min":0.42499999999999999},"max":{"1min":4.1609999999999996,"5min":24.492999999999999,"15min":24.492999999999999},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.83499999999999996,"5min":1.242,"15min":1.1140000000000001},"min":{"1min":0.64500000000000002,"5min":0.58399999999999996,"15min":0.46600000000000003},"max":{"1min":1.008,"5min":9.2789999999999999,"15min":23.524000000000001},"last":0.91300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:15:53.865 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:15:54.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.020+0000 7f01b2000640 1 Processor -- start 2024-01-15T18:15:54.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.020+0000 7f01b2000640 1 -- start start 2024-01-15T18:15:54.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01b2000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f01ac0c9bd0 0x7f01ac06c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:54.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01b2000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01ac0ad180 con 0x7f01ac15d0a0 2024-01-15T18:15:54.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01b2000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01ac0ac890 con 0x7f01ac0ce710 2024-01-15T18:15:54.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01b2000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01ac0cda90 con 0x7f01ac0c9bd0 2024-01-15T18:15:54.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01ab400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f01ac15d0a0 0x7f01ac0cd240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44320/0 (socket says 172.21.15.27:44320) 2024-01-15T18:15:54.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01ab400640 1 -- 172.21.15.27:0/1556309250 learned_addr learned my addr 172.21.15.27:0/1556309250 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:54.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01abe00640 1 --2- 172.21.15.27:0/1556309250 >> v2:172.21.15.93:3300/0 conn(0x7f01ac0c9bd0 0x7f01ac06c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:54.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01aa000640 1 -- 172.21.15.27:0/1556309250 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2921694027 0 0) 0x7f01ac0ad180 con 0x7f01ac15d0a0 2024-01-15T18:15:54.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01aa000640 1 -- 172.21.15.27:0/1556309250 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f018c003530 con 0x7f01ac15d0a0 2024-01-15T18:15:54.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01aa000640 1 -- 172.21.15.27:0/1556309250 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1801865037 0 0) 0x7f018c003530 con 0x7f01ac15d0a0 2024-01-15T18:15:54.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01aa000640 1 -- 172.21.15.27:0/1556309250 >> v2:172.21.15.93:3300/0 conn(0x7f01ac0c9bd0 msgr2=0x7f01ac06c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:54.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01aa000640 1 --2- 172.21.15.27:0/1556309250 >> v2:172.21.15.93:3300/0 conn(0x7f01ac0c9bd0 0x7f01ac06c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:54.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01aa000640 1 -- 172.21.15.27:0/1556309250 >> v1:172.21.15.93:6789/0 conn(0x7f01ac0ce710 legacy=0x7f01ac0c94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:54.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01aa000640 1 -- 172.21.15.27:0/1556309250 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ac0acb30 con 0x7f01ac15d0a0 2024-01-15T18:15:54.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01aa000640 1 -- 172.21.15.27:0/1556309250 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f019c001680 con 0x7f01ac15d0a0 2024-01-15T18:15:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01aa000640 1 -- 172.21.15.27:0/1556309250 >> v1:172.21.15.27:6789/0 conn(0x7f01ac15d0a0 legacy=0x7f01ac0cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01aa000640 1 --2- 172.21.15.27:0/1556309250 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f018c0039f0 0x7f018c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01aa000640 1 --2- 172.21.15.27:0/1556309250 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f018c0047d0 0x7f018c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01aa000640 1 --2- 172.21.15.27:0/1556309250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c005190 0x7f018c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01aa000640 1 -- 172.21.15.27:0/1556309250 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac0cda90 con 0x7f018c005190 2024-01-15T18:15:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01aa000640 1 -- 172.21.15.27:0/1556309250 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018c00c0c0 con 0x7f018c0047d0 2024-01-15T18:15:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01aa000640 1 -- 172.21.15.27:0/1556309250 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f018c00c280 con 0x7f018c0039f0 2024-01-15T18:15:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01abe00640 1 --2- 172.21.15.27:0/1556309250 >> v2:172.21.15.93:3300/0 conn(0x7f01ac0c9bd0 0x7f01ac06c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:15:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01abe00640 1 --2- 172.21.15.27:0/1556309250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c005190 0x7f018c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01aaa00640 1 --2- 172.21.15.27:0/1556309250 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f018c0047d0 0x7f018c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01ab400640 1 --2- 172.21.15.27:0/1556309250 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f018c0039f0 0x7f018c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01abe00640 1 -- 172.21.15.27:0/1556309250 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f018c0039f0 msgr2=0x7f018c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01abe00640 1 --2- 172.21.15.27:0/1556309250 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f018c0039f0 0x7f018c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:54.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01abe00640 1 -- 172.21.15.27:0/1556309250 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f018c0047d0 msgr2=0x7f018c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:54.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01abe00640 1 --2- 172.21.15.27:0/1556309250 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f018c0047d0 0x7f018c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:54.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.024+0000 7f01abe00640 1 -- 172.21.15.27:0/1556309250 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f018c003530 con 0x7f018c005190 2024-01-15T18:15:54.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.028+0000 7f01abe00640 1 -- 172.21.15.27:0/1556309250 reap_dead start 2024-01-15T18:15:54.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.028+0000 7f01abe00640 1 --2- 172.21.15.27:0/1556309250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c005190 0x7f018c00b990 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f01a400e320 tx=0x7f01a401ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d833635fcab84802 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:54.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.028+0000 7f01aa000640 1 -- 172.21.15.27:0/1556309250 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a40a5e10 con 0x7f018c005190 2024-01-15T18:15:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.028+0000 7f01aa000640 1 -- 172.21.15.27:0/1556309250 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a400f040 con 0x7f018c005190 2024-01-15T18:15:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.028+0000 7f01aa000640 1 -- 172.21.15.27:0/1556309250 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a4003be0 con 0x7f018c005190 2024-01-15T18:15:54.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.028+0000 7f01b2000640 1 -- 172.21.15.27:0/1556309250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c005190 msgr2=0x7f018c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:54.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.028+0000 7f01b2000640 1 --2- 172.21.15.27:0/1556309250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c005190 0x7f018c00b990 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f01a400e320 tx=0x7f01a401ce00 comp rx=0 tx=0).stop 2024-01-15T18:15:54.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.028+0000 7f01b2000640 1 -- 172.21.15.27:0/1556309250 shutdown_connections 2024-01-15T18:15:54.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.028+0000 7f01b2000640 1 --2- 172.21.15.27:0/1556309250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c005190 0x7f018c00b990 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:54.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.028+0000 7f01b2000640 1 -- 172.21.15.27:0/1556309250 >> 172.21.15.27:0/1556309250 conn(0x7f01ac0cec60 msgr2=0x7f01ac0cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:54.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.028+0000 7f01b2000640 1 -- 172.21.15.27:0/1556309250 shutdown_connections 2024-01-15T18:15:54.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.028+0000 7f01b2000640 1 -- 172.21.15.27:0/1556309250 wait complete. 2024-01-15T18:15:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.032+0000 7f01b2000640 1 Processor -- start 2024-01-15T18:15:54.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.032+0000 7f01b2000640 1 -- start start 2024-01-15T18:15:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.032+0000 7f01b2000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c005190 0x7f01ac156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.032+0000 7f01b2000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f01ac158aa0 0x7f01ac148060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.032+0000 7f01b2000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f01ac0c7f70 0x7f01ac0c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.032+0000 7f01b2000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac0c8930 con 0x7f018c005190 2024-01-15T18:15:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.032+0000 7f01b2000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac157ac0 con 0x7f01ac158aa0 2024-01-15T18:15:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.032+0000 7f01b2000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac1485a0 con 0x7f01ac0c7f70 2024-01-15T18:15:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.032+0000 7f01ab400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c005190 0x7f01ac156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.032+0000 7f01ab400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c005190 0x7f01ac156b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37274/0 (socket says 172.21.15.27:37274) 2024-01-15T18:15:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.032+0000 7f01ab400640 1 -- 172.21.15.27:0/2034269778 learned_addr learned my addr 172.21.15.27:0/2034269778 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.032+0000 7f01aaa00640 1 --2- 172.21.15.27:0/2034269778 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f01ac158aa0 0x7f01ac148060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:54.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.032+0000 7f01abe00640 1 --2- 172.21.15.27:0/2034269778 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f01ac0c7f70 0x7f01ac0c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.032+0000 7f01ab400640 1 -- 172.21.15.27:0/2034269778 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f01ac0c7f70 msgr2=0x7f01ac0c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.032+0000 7f01ab400640 1 --2- 172.21.15.27:0/2034269778 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f01ac0c7f70 0x7f01ac0c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.032+0000 7f01ab400640 1 -- 172.21.15.27:0/2034269778 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f01ac158aa0 msgr2=0x7f01ac148060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.032+0000 7f01ab400640 1 --2- 172.21.15.27:0/2034269778 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f01ac158aa0 0x7f01ac148060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.032+0000 7f01ab400640 1 -- 172.21.15.27:0/2034269778 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ac157220 con 0x7f018c005190 2024-01-15T18:15:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.032+0000 7f01ab400640 1 --2- 172.21.15.27:0/2034269778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c005190 0x7f01ac156b40 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f019c007290 tx=0x7f019c007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.032+0000 7f01a3400640 1 -- 172.21.15.27:0/2034269778 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c00f8e0 con 0x7f018c005190 2024-01-15T18:15:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.032+0000 7f01a3400640 1 -- 172.21.15.27:0/2034269778 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019c00fa60 con 0x7f018c005190 2024-01-15T18:15:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.032+0000 7f01a3400640 1 -- 172.21.15.27:0/2034269778 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c00fe10 con 0x7f018c005190 2024-01-15T18:15:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.032+0000 7f01b2000640 1 -- 172.21.15.27:0/2034269778 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01ac157500 con 0x7f018c005190 2024-01-15T18:15:54.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.032+0000 7f01b2000640 1 -- 172.21.15.27:0/2034269778 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01ac0c5e60 con 0x7f018c005190 2024-01-15T18:15:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.032+0000 7f01b2000640 1 -- 172.21.15.27:0/2034269778 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01ac04e850 con 0x7f018c005190 2024-01-15T18:15:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.036+0000 7f01a3400640 1 -- 172.21.15.27:0/2034269778 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7f019c01d460 con 0x7f018c005190 2024-01-15T18:15:54.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.036+0000 7f01a3400640 1 --2- 172.21.15.27:0/2034269778 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f01840674c0 0x7f0184069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.036+0000 7f01a3400640 1 -- 172.21.15.27:0/2034269778 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f019c083190 con 0x7f018c005190 2024-01-15T18:15:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.036+0000 7f01aaa00640 1 --2- 172.21.15.27:0/2034269778 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f01840674c0 0x7f0184069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:54.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.036+0000 7f01aaa00640 1 --2- 172.21.15.27:0/2034269778 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f01840674c0 0x7f0184069980 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f01ac158950 tx=0x7f0194009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:54.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.040+0000 7f01a3400640 1 -- 172.21.15.27:0/2034269778 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f019c051ba0 con 0x7f018c005190 2024-01-15T18:15:54.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.196+0000 7f01b2000640 1 -- 172.21.15.27:0/2034269778 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f01ac06c1e0 con 0x7f01840674c0 2024-01-15T18:15:54.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.204+0000 7f01a3400640 1 -- 172.21.15.27:0/2034269778 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288354 (secure 0 0 0) 0x7f01940014a0 con 0x7f01840674c0 2024-01-15T18:15:54.204 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:15:54.205 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:15:54.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.208+0000 7f01b2000640 1 -- 172.21.15.27:0/2034269778 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f01840674c0 msgr2=0x7f0184069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:54.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.208+0000 7f01b2000640 1 --2- 172.21.15.27:0/2034269778 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f01840674c0 0x7f0184069980 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f01ac158950 tx=0x7f0194009040 comp rx=0 tx=0).stop 2024-01-15T18:15:54.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.208+0000 7f01b2000640 1 -- 172.21.15.27:0/2034269778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c005190 msgr2=0x7f01ac156b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:54.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.208+0000 7f01b2000640 1 --2- 172.21.15.27:0/2034269778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c005190 0x7f01ac156b40 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f019c007290 tx=0x7f019c007750 comp rx=0 tx=0).stop 2024-01-15T18:15:54.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.208+0000 7f01b2000640 1 -- 172.21.15.27:0/2034269778 shutdown_connections 2024-01-15T18:15:54.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.208+0000 7f01b2000640 1 --2- 172.21.15.27:0/2034269778 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f01840674c0 0x7f0184069980 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:54.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.208+0000 7f01b2000640 1 --2- 172.21.15.27:0/2034269778 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f01ac0c7f70 0x7f01ac0c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:54.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.208+0000 7f01b2000640 1 --2- 172.21.15.27:0/2034269778 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f01ac158aa0 0x7f01ac148060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:54.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.208+0000 7f01b2000640 1 --2- 172.21.15.27:0/2034269778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f018c005190 0x7f01ac156b40 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:54.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.208+0000 7f01b2000640 1 -- 172.21.15.27:0/2034269778 >> 172.21.15.27:0/2034269778 conn(0x7f01ac0cec60 msgr2=0x7f01ac0af100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:54.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.208+0000 7f01b2000640 1 -- 172.21.15.27:0/2034269778 shutdown_connections 2024-01-15T18:15:54.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.208+0000 7f01b2000640 1 -- 172.21.15.27:0/2034269778 wait complete. 2024-01-15T18:15:54.225 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":411,"stamp":"2024-01-15T18:15:52.331872+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":190,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":563700,"kb_used_data":9224,"kb_used_omap":12,"kb_used_meta":554355,"kb_avail":754411020,"statfs":{"total":773094113280,"available":772516884480,"internally_reserved":0,"allocated":9445376,"data_stored":4711668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":567660080},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001712"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.776146+0000","last_change":"2024-01-15T18:15:45.194410+0000","last_active":"2024-01-15T18:15:46.776146+0000","last_peered":"2024-01-15T18:15:46.776146+0000","last_clean":"2024-01-15T18:15:46.776146+0000","last_became_active":"2024-01-15T18:15:45.193977+0000","last_became_peered":"2024-01-15T18:15:45.193977+0000","last_unstale":"2024-01-15T18:15:46.776146+0000","last_undegraded":"2024-01-15T18:15:46.776146+0000","last_fullsized":"2024-01-15T18:15:46.776146+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:51:46.425900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.059968+0000","last_change":"2024-01-15T18:15:45.185773+0000","last_active":"2024-01-15T18:15:47.059968+0000","last_peered":"2024-01-15T18:15:47.059968+0000","last_clean":"2024-01-15T18:15:47.059968+0000","last_became_active":"2024-01-15T18:15:45.185459+0000","last_became_peered":"2024-01-15T18:15:45.185459+0000","last_unstale":"2024-01-15T18:15:47.059968+0000","last_undegraded":"2024-01-15T18:15:47.059968+0000","last_fullsized":"2024-01-15T18:15:47.059968+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:53.495194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.184696+0000","last_change":"2024-01-15T18:15:45.190593+0000","last_active":"2024-01-15T18:15:46.184696+0000","last_peered":"2024-01-15T18:15:46.184696+0000","last_clean":"2024-01-15T18:15:46.184696+0000","last_became_active":"2024-01-15T18:15:45.190292+0000","last_became_peered":"2024-01-15T18:15:45.190292+0000","last_unstale":"2024-01-15T18:15:46.184696+0000","last_undegraded":"2024-01-15T18:15:46.184696+0000","last_fullsized":"2024-01-15T18:15:46.184696+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:39:49.161202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.059753+0000","last_change":"2024-01-15T18:15:45.192728+0000","last_active":"2024-01-15T18:15:47.059753+0000","last_peered":"2024-01-15T18:15:47.059753+0000","last_clean":"2024-01-15T18:15:47.059753+0000","last_became_active":"2024-01-15T18:15:45.192270+0000","last_became_peered":"2024-01-15T18:15:45.192270+0000","last_unstale":"2024-01-15T18:15:47.059753+0000","last_undegraded":"2024-01-15T18:15:47.059753+0000","last_fullsized":"2024-01-15T18:15:47.059753+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:32:17.629691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.058669+0000","last_change":"2024-01-15T18:15:45.181959+0000","last_active":"2024-01-15T18:15:47.058669+0000","last_peered":"2024-01-15T18:15:47.058669+0000","last_clean":"2024-01-15T18:15:47.058669+0000","last_became_active":"2024-01-15T18:15:45.181526+0000","last_became_peered":"2024-01-15T18:15:45.181526+0000","last_unstale":"2024-01-15T18:15:47.058669+0000","last_undegraded":"2024-01-15T18:15:47.058669+0000","last_fullsized":"2024-01-15T18:15:47.058669+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:41:44.153310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.058582+0000","last_change":"2024-01-15T18:15:45.178635+0000","last_active":"2024-01-15T18:15:47.058582+0000","last_peered":"2024-01-15T18:15:47.058582+0000","last_clean":"2024-01-15T18:15:47.058582+0000","last_became_active":"2024-01-15T18:15:45.178354+0000","last_became_peered":"2024-01-15T18:15:45.178354+0000","last_unstale":"2024-01-15T18:15:47.058582+0000","last_undegraded":"2024-01-15T18:15:47.058582+0000","last_fullsized":"2024-01-15T18:15:47.058582+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:47:05.193543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.059365+0000","last_change":"2024-01-15T18:15:45.177382+0000","last_active":"2024-01-15T18:15:47.059365+0000","last_peered":"2024-01-15T18:15:47.059365+0000","last_clean":"2024-01-15T18:15:47.059365+0000","last_became_active":"2024-01-15T18:15:45.173825+0000","last_became_peered":"2024-01-15T18:15:45.173825+0000","last_unstale":"2024-01-15T18:15:47.059365+0000","last_undegraded":"2024-01-15T18:15:47.059365+0000","last_fullsized":"2024-01-15T18:15:47.059365+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:56:46.648785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.058383+0000","last_change":"2024-01-15T18:15:45.172298+0000","last_active":"2024-01-15T18:15:47.058383+0000","last_peered":"2024-01-15T18:15:47.058383+0000","last_clean":"2024-01-15T18:15:47.058383+0000","last_became_active":"2024-01-15T18:15:45.172026+0000","last_became_peered":"2024-01-15T18:15:45.172026+0000","last_unstale":"2024-01-15T18:15:47.058383+0000","last_undegraded":"2024-01-15T18:15:47.058383+0000","last_fullsized":"2024-01-15T18:15:47.058383+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:30:49.183104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.058850+0000","last_change":"2024-01-15T18:15:45.199261+0000","last_active":"2024-01-15T18:15:47.058850+0000","last_peered":"2024-01-15T18:15:47.058850+0000","last_clean":"2024-01-15T18:15:47.058850+0000","last_became_active":"2024-01-15T18:15:45.198949+0000","last_became_peered":"2024-01-15T18:15:45.198949+0000","last_unstale":"2024-01-15T18:15:47.058850+0000","last_undegraded":"2024-01-15T18:15:47.058850+0000","last_fullsized":"2024-01-15T18:15:47.058850+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:53:53.175858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.776261+0000","last_change":"2024-01-15T18:15:45.190970+0000","last_active":"2024-01-15T18:15:46.776261+0000","last_peered":"2024-01-15T18:15:46.776261+0000","last_clean":"2024-01-15T18:15:46.776261+0000","last_became_active":"2024-01-15T18:15:45.187235+0000","last_became_peered":"2024-01-15T18:15:45.187235+0000","last_unstale":"2024-01-15T18:15:46.776261+0000","last_undegraded":"2024-01-15T18:15:46.776261+0000","last_fullsized":"2024-01-15T18:15:46.776261+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:05:25.657249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.059719+0000","last_change":"2024-01-15T18:15:45.179561+0000","last_active":"2024-01-15T18:15:47.059719+0000","last_peered":"2024-01-15T18:15:47.059719+0000","last_clean":"2024-01-15T18:15:47.059719+0000","last_became_active":"2024-01-15T18:15:45.179305+0000","last_became_peered":"2024-01-15T18:15:45.179305+0000","last_unstale":"2024-01-15T18:15:47.059719+0000","last_undegraded":"2024-01-15T18:15:47.059719+0000","last_fullsized":"2024-01-15T18:15:47.059719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:53:25.231875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.775747+0000","last_change":"2024-01-15T18:15:45.189460+0000","last_active":"2024-01-15T18:15:46.775747+0000","last_peered":"2024-01-15T18:15:46.775747+0000","last_clean":"2024-01-15T18:15:46.775747+0000","last_became_active":"2024-01-15T18:15:45.188105+0000","last_became_peered":"2024-01-15T18:15:45.188105+0000","last_unstale":"2024-01-15T18:15:46.775747+0000","last_undegraded":"2024-01-15T18:15:46.775747+0000","last_fullsized":"2024-01-15T18:15:46.775747+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:10:10.202003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.184509+0000","last_change":"2024-01-15T18:15:45.190326+0000","last_active":"2024-01-15T18:15:46.184509+0000","last_peered":"2024-01-15T18:15:46.184509+0000","last_clean":"2024-01-15T18:15:46.184509+0000","last_became_active":"2024-01-15T18:15:45.188312+0000","last_became_peered":"2024-01-15T18:15:45.188312+0000","last_unstale":"2024-01-15T18:15:46.184509+0000","last_undegraded":"2024-01-15T18:15:46.184509+0000","last_fullsized":"2024-01-15T18:15:46.184509+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:31:45.154198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.059861+0000","last_change":"2024-01-15T18:15:45.180022+0000","last_active":"2024-01-15T18:15:47.059861+0000","last_peered":"2024-01-15T18:15:47.059861+0000","last_clean":"2024-01-15T18:15:47.059861+0000","last_became_active":"2024-01-15T18:15:45.179537+0000","last_became_peered":"2024-01-15T18:15:45.179537+0000","last_unstale":"2024-01-15T18:15:47.059861+0000","last_undegraded":"2024-01-15T18:15:47.059861+0000","last_fullsized":"2024-01-15T18:15:47.059861+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:12:38.030672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.180223+0000","last_change":"2024-01-15T18:15:45.185924+0000","last_active":"2024-01-15T18:15:46.180223+0000","last_peered":"2024-01-15T18:15:46.180223+0000","last_clean":"2024-01-15T18:15:46.180223+0000","last_became_active":"2024-01-15T18:15:45.185669+0000","last_became_peered":"2024-01-15T18:15:45.185669+0000","last_unstale":"2024-01-15T18:15:46.180223+0000","last_undegraded":"2024-01-15T18:15:46.180223+0000","last_fullsized":"2024-01-15T18:15:46.180223+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:01:49.697849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.058131+0000","last_change":"2024-01-15T18:15:45.176262+0000","last_active":"2024-01-15T18:15:47.058131+0000","last_peered":"2024-01-15T18:15:47.058131+0000","last_clean":"2024-01-15T18:15:47.058131+0000","last_became_active":"2024-01-15T18:15:45.175976+0000","last_became_peered":"2024-01-15T18:15:45.175976+0000","last_unstale":"2024-01-15T18:15:47.058131+0000","last_undegraded":"2024-01-15T18:15:47.058131+0000","last_fullsized":"2024-01-15T18:15:47.058131+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:56.553890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.059306+0000","last_change":"2024-01-15T18:15:45.184436+0000","last_active":"2024-01-15T18:15:47.059306+0000","last_peered":"2024-01-15T18:15:47.059306+0000","last_clean":"2024-01-15T18:15:47.059306+0000","last_became_active":"2024-01-15T18:15:45.181072+0000","last_became_peered":"2024-01-15T18:15:45.181072+0000","last_unstale":"2024-01-15T18:15:47.059306+0000","last_undegraded":"2024-01-15T18:15:47.059306+0000","last_fullsized":"2024-01-15T18:15:47.059306+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:54:04.059169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.060172+0000","last_change":"2024-01-15T18:15:45.184794+0000","last_active":"2024-01-15T18:15:47.060172+0000","last_peered":"2024-01-15T18:15:47.060172+0000","last_clean":"2024-01-15T18:15:47.060172+0000","last_became_active":"2024-01-15T18:15:45.184528+0000","last_became_peered":"2024-01-15T18:15:45.184528+0000","last_unstale":"2024-01-15T18:15:47.060172+0000","last_undegraded":"2024-01-15T18:15:47.060172+0000","last_fullsized":"2024-01-15T18:15:47.060172+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:33:22.524087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.184329+0000","last_change":"2024-01-15T18:15:45.196447+0000","last_active":"2024-01-15T18:15:46.184329+0000","last_peered":"2024-01-15T18:15:46.184329+0000","last_clean":"2024-01-15T18:15:46.184329+0000","last_became_active":"2024-01-15T18:15:45.194478+0000","last_became_peered":"2024-01-15T18:15:45.194478+0000","last_unstale":"2024-01-15T18:15:46.184329+0000","last_undegraded":"2024-01-15T18:15:46.184329+0000","last_fullsized":"2024-01-15T18:15:46.184329+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:20:05.906577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.059846+0000","last_change":"2024-01-15T18:15:45.202436+0000","last_active":"2024-01-15T18:15:47.059846+0000","last_peered":"2024-01-15T18:15:47.059846+0000","last_clean":"2024-01-15T18:15:47.059846+0000","last_became_active":"2024-01-15T18:15:45.202075+0000","last_became_peered":"2024-01-15T18:15:45.202075+0000","last_unstale":"2024-01-15T18:15:47.059846+0000","last_undegraded":"2024-01-15T18:15:47.059846+0000","last_fullsized":"2024-01-15T18:15:47.059846+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:12:42.277859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.059680+0000","last_change":"2024-01-15T18:15:45.197310+0000","last_active":"2024-01-15T18:15:47.059680+0000","last_peered":"2024-01-15T18:15:47.059680+0000","last_clean":"2024-01-15T18:15:47.059680+0000","last_became_active":"2024-01-15T18:15:45.196364+0000","last_became_peered":"2024-01-15T18:15:45.196364+0000","last_unstale":"2024-01-15T18:15:47.059680+0000","last_undegraded":"2024-01-15T18:15:47.059680+0000","last_fullsized":"2024-01-15T18:15:47.059680+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:30.485073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.059914+0000","last_change":"2024-01-15T18:15:45.185683+0000","last_active":"2024-01-15T18:15:47.059914+0000","last_peered":"2024-01-15T18:15:47.059914+0000","last_clean":"2024-01-15T18:15:47.059914+0000","last_became_active":"2024-01-15T18:15:45.185303+0000","last_became_peered":"2024-01-15T18:15:45.185303+0000","last_unstale":"2024-01-15T18:15:47.059914+0000","last_undegraded":"2024-01-15T18:15:47.059914+0000","last_fullsized":"2024-01-15T18:15:47.059914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:59:33.238710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.775913+0000","last_change":"2024-01-15T18:15:45.195966+0000","last_active":"2024-01-15T18:15:46.775913+0000","last_peered":"2024-01-15T18:15:46.775913+0000","last_clean":"2024-01-15T18:15:46.775913+0000","last_became_active":"2024-01-15T18:15:45.194717+0000","last_became_peered":"2024-01-15T18:15:45.194717+0000","last_unstale":"2024-01-15T18:15:46.775913+0000","last_undegraded":"2024-01-15T18:15:46.775913+0000","last_fullsized":"2024-01-15T18:15:46.775913+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:14:42.632988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.059458+0000","last_change":"2024-01-15T18:15:45.179745+0000","last_active":"2024-01-15T18:15:47.059458+0000","last_peered":"2024-01-15T18:15:47.059458+0000","last_clean":"2024-01-15T18:15:47.059458+0000","last_became_active":"2024-01-15T18:15:45.179329+0000","last_became_peered":"2024-01-15T18:15:45.179329+0000","last_unstale":"2024-01-15T18:15:47.059458+0000","last_undegraded":"2024-01-15T18:15:47.059458+0000","last_fullsized":"2024-01-15T18:15:47.059458+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:53:36.315415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.059804+0000","last_change":"2024-01-15T18:15:45.192970+0000","last_active":"2024-01-15T18:15:47.059804+0000","last_peered":"2024-01-15T18:15:47.059804+0000","last_clean":"2024-01-15T18:15:47.059804+0000","last_became_active":"2024-01-15T18:15:45.192474+0000","last_became_peered":"2024-01-15T18:15:45.192474+0000","last_unstale":"2024-01-15T18:15:47.059804+0000","last_undegraded":"2024-01-15T18:15:47.059804+0000","last_fullsized":"2024-01-15T18:15:47.059804+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:16.351733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.184793+0000","last_change":"2024-01-15T18:15:45.180211+0000","last_active":"2024-01-15T18:15:46.184793+0000","last_peered":"2024-01-15T18:15:46.184793+0000","last_clean":"2024-01-15T18:15:46.184793+0000","last_became_active":"2024-01-15T18:15:45.179566+0000","last_became_peered":"2024-01-15T18:15:45.179566+0000","last_unstale":"2024-01-15T18:15:46.184793+0000","last_undegraded":"2024-01-15T18:15:46.184793+0000","last_fullsized":"2024-01-15T18:15:46.184793+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:03:54.512566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.058563+0000","last_change":"2024-01-15T18:15:45.189295+0000","last_active":"2024-01-15T18:15:47.058563+0000","last_peered":"2024-01-15T18:15:47.058563+0000","last_clean":"2024-01-15T18:15:47.058563+0000","last_became_active":"2024-01-15T18:15:45.187101+0000","last_became_peered":"2024-01-15T18:15:45.187101+0000","last_unstale":"2024-01-15T18:15:47.058563+0000","last_undegraded":"2024-01-15T18:15:47.058563+0000","last_fullsized":"2024-01-15T18:15:47.058563+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:03:29.579452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.775931+0000","last_change":"2024-01-15T18:15:45.192353+0000","last_active":"2024-01-15T18:15:46.775931+0000","last_peered":"2024-01-15T18:15:46.775931+0000","last_clean":"2024-01-15T18:15:46.775931+0000","last_became_active":"2024-01-15T18:15:45.191563+0000","last_became_peered":"2024-01-15T18:15:45.191563+0000","last_unstale":"2024-01-15T18:15:46.775931+0000","last_undegraded":"2024-01-15T18:15:46.775931+0000","last_fullsized":"2024-01-15T18:15:46.775931+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:01:13.745010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.059357+0000","last_change":"2024-01-15T18:15:45.190607+0000","last_active":"2024-01-15T18:15:47.059357+0000","last_peered":"2024-01-15T18:15:47.059357+0000","last_clean":"2024-01-15T18:15:47.059357+0000","last_became_active":"2024-01-15T18:15:45.189633+0000","last_became_peered":"2024-01-15T18:15:45.189633+0000","last_unstale":"2024-01-15T18:15:47.059357+0000","last_undegraded":"2024-01-15T18:15:47.059357+0000","last_fullsized":"2024-01-15T18:15:47.059357+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:57:43.892130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.060186+0000","last_change":"2024-01-15T18:15:45.180268+0000","last_active":"2024-01-15T18:15:47.060186+0000","last_peered":"2024-01-15T18:15:47.060186+0000","last_clean":"2024-01-15T18:15:47.060186+0000","last_became_active":"2024-01-15T18:15:45.179617+0000","last_became_peered":"2024-01-15T18:15:45.179617+0000","last_unstale":"2024-01-15T18:15:47.060186+0000","last_undegraded":"2024-01-15T18:15:47.060186+0000","last_fullsized":"2024-01-15T18:15:47.060186+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:41:05.797910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.058281+0000","last_change":"2024-01-15T18:15:45.177309+0000","last_active":"2024-01-15T18:15:47.058281+0000","last_peered":"2024-01-15T18:15:47.058281+0000","last_clean":"2024-01-15T18:15:47.058281+0000","last_became_active":"2024-01-15T18:15:45.177022+0000","last_became_peered":"2024-01-15T18:15:45.177022+0000","last_unstale":"2024-01-15T18:15:47.058281+0000","last_undegraded":"2024-01-15T18:15:47.058281+0000","last_fullsized":"2024-01-15T18:15:47.058281+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:08:31.073685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.775731+0000","last_change":"2024-01-15T18:15:45.182964+0000","last_active":"2024-01-15T18:15:46.775731+0000","last_peered":"2024-01-15T18:15:46.775731+0000","last_clean":"2024-01-15T18:15:46.775731+0000","last_became_active":"2024-01-15T18:15:45.182690+0000","last_became_peered":"2024-01-15T18:15:45.182690+0000","last_unstale":"2024-01-15T18:15:46.775731+0000","last_undegraded":"2024-01-15T18:15:46.775731+0000","last_fullsized":"2024-01-15T18:15:46.775731+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:39:48.551963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.059170+0000","last_change":"2024-01-15T18:15:45.189818+0000","last_active":"2024-01-15T18:15:47.059170+0000","last_peered":"2024-01-15T18:15:47.059170+0000","last_clean":"2024-01-15T18:15:47.059170+0000","last_became_active":"2024-01-15T18:15:45.189534+0000","last_became_peered":"2024-01-15T18:15:45.189534+0000","last_unstale":"2024-01-15T18:15:47.059170+0000","last_undegraded":"2024-01-15T18:15:47.059170+0000","last_fullsized":"2024-01-15T18:15:47.059170+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:28:05.260374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"83'4","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.450194+0000","last_change":"2024-01-15T18:15:44.163810+0000","last_active":"2024-01-15T18:15:45.450194+0000","last_peered":"2024-01-15T18:15:45.450194+0000","last_clean":"2024-01-15T18:15:45.450194+0000","last_became_active":"2024-01-15T18:15:44.163128+0000","last_became_peered":"2024-01-15T18:15:44.163128+0000","last_unstale":"2024-01-15T18:15:45.450194+0000","last_undegraded":"2024-01-15T18:15:45.450194+0000","last_fullsized":"2024-01-15T18:15:45.450194+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:22:07.304644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.776209+0000","last_change":"2024-01-15T18:15:45.201966+0000","last_active":"2024-01-15T18:15:46.776209+0000","last_peered":"2024-01-15T18:15:46.776209+0000","last_clean":"2024-01-15T18:15:46.776209+0000","last_became_active":"2024-01-15T18:15:45.201568+0000","last_became_peered":"2024-01-15T18:15:45.201568+0000","last_unstale":"2024-01-15T18:15:46.776209+0000","last_undegraded":"2024-01-15T18:15:46.776209+0000","last_fullsized":"2024-01-15T18:15:46.776209+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:56.553890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"83'1","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.336547+0000","last_change":"2024-01-15T18:15:44.156714+0000","last_active":"2024-01-15T18:15:45.336547+0000","last_peered":"2024-01-15T18:15:45.336547+0000","last_clean":"2024-01-15T18:15:45.336547+0000","last_became_active":"2024-01-15T18:15:44.156419+0000","last_became_peered":"2024-01-15T18:15:44.156419+0000","last_unstale":"2024-01-15T18:15:45.336547+0000","last_undegraded":"2024-01-15T18:15:45.336547+0000","last_fullsized":"2024-01-15T18:15:45.336547+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:14:41.627743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.775960+0000","last_change":"2024-01-15T18:15:45.192955+0000","last_active":"2024-01-15T18:15:46.775960+0000","last_peered":"2024-01-15T18:15:46.775960+0000","last_clean":"2024-01-15T18:15:46.775960+0000","last_became_active":"2024-01-15T18:15:45.192565+0000","last_became_peered":"2024-01-15T18:15:45.192565+0000","last_unstale":"2024-01-15T18:15:46.775960+0000","last_undegraded":"2024-01-15T18:15:46.775960+0000","last_fullsized":"2024-01-15T18:15:46.775960+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:27:48.589264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"83'2","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.196747+0000","last_change":"2024-01-15T18:15:44.167059+0000","last_active":"2024-01-15T18:15:45.196747+0000","last_peered":"2024-01-15T18:15:45.196747+0000","last_clean":"2024-01-15T18:15:45.196747+0000","last_became_active":"2024-01-15T18:15:44.166829+0000","last_became_peered":"2024-01-15T18:15:44.166829+0000","last_unstale":"2024-01-15T18:15:45.196747+0000","last_undegraded":"2024-01-15T18:15:45.196747+0000","last_fullsized":"2024-01-15T18:15:45.196747+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:53:52.170612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.184953+0000","last_change":"2024-01-15T18:15:45.186505+0000","last_active":"2024-01-15T18:15:46.184953+0000","last_peered":"2024-01-15T18:15:46.184953+0000","last_clean":"2024-01-15T18:15:46.184953+0000","last_became_active":"2024-01-15T18:15:45.186181+0000","last_became_peered":"2024-01-15T18:15:45.186181+0000","last_unstale":"2024-01-15T18:15:46.184953+0000","last_undegraded":"2024-01-15T18:15:46.184953+0000","last_fullsized":"2024-01-15T18:15:46.184953+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:23:00.725727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"83'2","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.199414+0000","last_change":"2024-01-15T18:15:44.166697+0000","last_active":"2024-01-15T18:15:45.199414+0000","last_peered":"2024-01-15T18:15:45.199414+0000","last_clean":"2024-01-15T18:15:45.199414+0000","last_became_active":"2024-01-15T18:15:44.166389+0000","last_became_peered":"2024-01-15T18:15:44.166389+0000","last_unstale":"2024-01-15T18:15:45.199414+0000","last_undegraded":"2024-01-15T18:15:45.199414+0000","last_fullsized":"2024-01-15T18:15:45.199414+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:56:34.685926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.060217+0000","last_change":"2024-01-15T18:15:45.180445+0000","last_active":"2024-01-15T18:15:47.060217+0000","last_peered":"2024-01-15T18:15:47.060217+0000","last_clean":"2024-01-15T18:15:47.060217+0000","last_became_active":"2024-01-15T18:15:45.179946+0000","last_became_peered":"2024-01-15T18:15:45.179946+0000","last_unstale":"2024-01-15T18:15:47.060217+0000","last_undegraded":"2024-01-15T18:15:47.060217+0000","last_fullsized":"2024-01-15T18:15:47.060217+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:04:09.741683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"83'1","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.336483+0000","last_change":"2024-01-15T18:15:44.157992+0000","last_active":"2024-01-15T18:15:45.336483+0000","last_peered":"2024-01-15T18:15:45.336483+0000","last_clean":"2024-01-15T18:15:45.336483+0000","last_became_active":"2024-01-15T18:15:44.157721+0000","last_became_peered":"2024-01-15T18:15:44.157721+0000","last_unstale":"2024-01-15T18:15:45.336483+0000","last_undegraded":"2024-01-15T18:15:45.336483+0000","last_fullsized":"2024-01-15T18:15:45.336483+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:27:47.584018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.058662+0000","last_change":"2024-01-15T18:15:45.178738+0000","last_active":"2024-01-15T18:15:47.058662+0000","last_peered":"2024-01-15T18:15:47.058662+0000","last_clean":"2024-01-15T18:15:47.058662+0000","last_became_active":"2024-01-15T18:15:45.178446+0000","last_became_peered":"2024-01-15T18:15:45.178446+0000","last_unstale":"2024-01-15T18:15:47.058662+0000","last_undegraded":"2024-01-15T18:15:47.058662+0000","last_fullsized":"2024-01-15T18:15:47.058662+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:05:25.657249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"83'1","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.336430+0000","last_change":"2024-01-15T18:15:44.154715+0000","last_active":"2024-01-15T18:15:45.336430+0000","last_peered":"2024-01-15T18:15:45.336430+0000","last_clean":"2024-01-15T18:15:45.336430+0000","last_became_active":"2024-01-15T18:15:44.154433+0000","last_became_peered":"2024-01-15T18:15:44.154433+0000","last_unstale":"2024-01-15T18:15:45.336430+0000","last_undegraded":"2024-01-15T18:15:45.336430+0000","last_fullsized":"2024-01-15T18:15:45.336430+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:42:00.915026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.184371+0000","last_change":"2024-01-15T18:15:45.202049+0000","last_active":"2024-01-15T18:15:46.184371+0000","last_peered":"2024-01-15T18:15:46.184371+0000","last_clean":"2024-01-15T18:15:46.184371+0000","last_became_active":"2024-01-15T18:15:45.201742+0000","last_became_peered":"2024-01-15T18:15:45.201742+0000","last_unstale":"2024-01-15T18:15:46.184371+0000","last_undegraded":"2024-01-15T18:15:46.184371+0000","last_fullsized":"2024-01-15T18:15:46.184371+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:56:00.059318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.171647+0000","last_change":"2024-01-15T18:15:44.167859+0000","last_active":"2024-01-15T18:15:45.171647+0000","last_peered":"2024-01-15T18:15:45.171647+0000","last_clean":"2024-01-15T18:15:45.171647+0000","last_became_active":"2024-01-15T18:15:44.167561+0000","last_became_peered":"2024-01-15T18:15:44.167561+0000","last_unstale":"2024-01-15T18:15:45.171647+0000","last_undegraded":"2024-01-15T18:15:45.171647+0000","last_fullsized":"2024-01-15T18:15:45.171647+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:31.305427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.180593+0000","last_change":"2024-01-15T18:15:45.187853+0000","last_active":"2024-01-15T18:15:46.180593+0000","last_peered":"2024-01-15T18:15:46.180593+0000","last_clean":"2024-01-15T18:15:46.180593+0000","last_became_active":"2024-01-15T18:15:45.187552+0000","last_became_peered":"2024-01-15T18:15:45.187552+0000","last_unstale":"2024-01-15T18:15:46.180593+0000","last_undegraded":"2024-01-15T18:15:46.180593+0000","last_fullsized":"2024-01-15T18:15:46.180593+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:28:05.260374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.430385+0000","last_change":"2024-01-15T18:15:44.155651+0000","last_active":"2024-01-15T18:15:45.430385+0000","last_peered":"2024-01-15T18:15:45.430385+0000","last_clean":"2024-01-15T18:15:45.430385+0000","last_became_active":"2024-01-15T18:15:44.155306+0000","last_became_peered":"2024-01-15T18:15:44.155306+0000","last_unstale":"2024-01-15T18:15:45.430385+0000","last_undegraded":"2024-01-15T18:15:45.430385+0000","last_fullsized":"2024-01-15T18:15:45.430385+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:06:15.812490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.180798+0000","last_change":"2024-01-15T18:15:45.184002+0000","last_active":"2024-01-15T18:15:46.180798+0000","last_peered":"2024-01-15T18:15:46.180798+0000","last_clean":"2024-01-15T18:15:46.180798+0000","last_became_active":"2024-01-15T18:15:45.183624+0000","last_became_peered":"2024-01-15T18:15:45.183624+0000","last_unstale":"2024-01-15T18:15:46.180798+0000","last_undegraded":"2024-01-15T18:15:46.180798+0000","last_fullsized":"2024-01-15T18:15:46.180798+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:26:01.224733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.450204+0000","last_change":"2024-01-15T18:15:44.160123+0000","last_active":"2024-01-15T18:15:45.450204+0000","last_peered":"2024-01-15T18:15:45.450204+0000","last_clean":"2024-01-15T18:15:45.450204+0000","last_became_active":"2024-01-15T18:15:44.159761+0000","last_became_peered":"2024-01-15T18:15:44.159761+0000","last_unstale":"2024-01-15T18:15:45.450204+0000","last_undegraded":"2024-01-15T18:15:45.450204+0000","last_fullsized":"2024-01-15T18:15:45.450204+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:42:00.915026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.058938+0000","last_change":"2024-01-15T18:15:45.193928+0000","last_active":"2024-01-15T18:15:47.058938+0000","last_peered":"2024-01-15T18:15:47.058938+0000","last_clean":"2024-01-15T18:15:47.058938+0000","last_became_active":"2024-01-15T18:15:45.193656+0000","last_became_peered":"2024-01-15T18:15:45.193656+0000","last_unstale":"2024-01-15T18:15:47.058938+0000","last_undegraded":"2024-01-15T18:15:47.058938+0000","last_fullsized":"2024-01-15T18:15:47.058938+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:33.549022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.450125+0000","last_change":"2024-01-15T18:15:44.166336+0000","last_active":"2024-01-15T18:15:45.450125+0000","last_peered":"2024-01-15T18:15:45.450125+0000","last_clean":"2024-01-15T18:15:45.450125+0000","last_became_active":"2024-01-15T18:15:44.165403+0000","last_became_peered":"2024-01-15T18:15:44.165403+0000","last_unstale":"2024-01-15T18:15:45.450125+0000","last_undegraded":"2024-01-15T18:15:45.450125+0000","last_fullsized":"2024-01-15T18:15:45.450125+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:42:53.652872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.184288+0000","last_change":"2024-01-15T18:15:45.187664+0000","last_active":"2024-01-15T18:15:46.184288+0000","last_peered":"2024-01-15T18:15:46.184288+0000","last_clean":"2024-01-15T18:15:46.184288+0000","last_became_active":"2024-01-15T18:15:45.187401+0000","last_became_peered":"2024-01-15T18:15:45.187401+0000","last_unstale":"2024-01-15T18:15:46.184288+0000","last_undegraded":"2024-01-15T18:15:46.184288+0000","last_fullsized":"2024-01-15T18:15:46.184288+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:31:04.304785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"83'3","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.450093+0000","last_change":"2024-01-15T18:15:44.160184+0000","last_active":"2024-01-15T18:15:45.450093+0000","last_peered":"2024-01-15T18:15:45.450093+0000","last_clean":"2024-01-15T18:15:45.450093+0000","last_became_active":"2024-01-15T18:15:44.159855+0000","last_became_peered":"2024-01-15T18:15:44.159855+0000","last_unstale":"2024-01-15T18:15:45.450093+0000","last_undegraded":"2024-01-15T18:15:45.450093+0000","last_fullsized":"2024-01-15T18:15:45.450093+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:04:08.736438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.058704+0000","last_change":"2024-01-15T18:15:45.187059+0000","last_active":"2024-01-15T18:15:47.058704+0000","last_peered":"2024-01-15T18:15:47.058704+0000","last_clean":"2024-01-15T18:15:47.058704+0000","last_became_active":"2024-01-15T18:15:45.184830+0000","last_became_peered":"2024-01-15T18:15:45.184830+0000","last_unstale":"2024-01-15T18:15:47.058704+0000","last_undegraded":"2024-01-15T18:15:47.058704+0000","last_fullsized":"2024-01-15T18:15:47.058704+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:34:16.521921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"83'6","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198837+0000","last_change":"2024-01-15T18:15:44.158973+0000","last_active":"2024-01-15T18:15:45.198837+0000","last_peered":"2024-01-15T18:15:45.198837+0000","last_clean":"2024-01-15T18:15:45.198837+0000","last_became_active":"2024-01-15T18:15:44.158684+0000","last_became_peered":"2024-01-15T18:15:44.158684+0000","last_unstale":"2024-01-15T18:15:45.198837+0000","last_undegraded":"2024-01-15T18:15:45.198837+0000","last_fullsized":"2024-01-15T18:15:45.198837+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:03:51.252813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.184115+0000","last_change":"2024-01-15T18:15:45.196327+0000","last_active":"2024-01-15T18:15:46.184115+0000","last_peered":"2024-01-15T18:15:46.184115+0000","last_clean":"2024-01-15T18:15:46.184115+0000","last_became_active":"2024-01-15T18:15:45.195928+0000","last_became_peered":"2024-01-15T18:15:45.195928+0000","last_unstale":"2024-01-15T18:15:46.184115+0000","last_undegraded":"2024-01-15T18:15:46.184115+0000","last_fullsized":"2024-01-15T18:15:46.184115+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:57:47.746767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"83'2","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.430482+0000","last_change":"2024-01-15T18:15:44.166340+0000","last_active":"2024-01-15T18:15:45.430482+0000","last_peered":"2024-01-15T18:15:45.430482+0000","last_clean":"2024-01-15T18:15:45.430482+0000","last_became_active":"2024-01-15T18:15:44.165947+0000","last_became_peered":"2024-01-15T18:15:44.165947+0000","last_unstale":"2024-01-15T18:15:45.430482+0000","last_undegraded":"2024-01-15T18:15:45.430482+0000","last_fullsized":"2024-01-15T18:15:45.430482+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:12:37.715165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.184718+0000","last_change":"2024-01-15T18:15:45.199248+0000","last_active":"2024-01-15T18:15:46.184718+0000","last_peered":"2024-01-15T18:15:46.184718+0000","last_clean":"2024-01-15T18:15:46.184718+0000","last_became_active":"2024-01-15T18:15:45.198929+0000","last_became_peered":"2024-01-15T18:15:45.198929+0000","last_unstale":"2024-01-15T18:15:46.184718+0000","last_undegraded":"2024-01-15T18:15:46.184718+0000","last_fullsized":"2024-01-15T18:15:46.184718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:04:37.230606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.401171+0000","last_change":"2024-01-15T18:15:44.161901+0000","last_active":"2024-01-15T18:15:45.401171+0000","last_peered":"2024-01-15T18:15:45.401171+0000","last_clean":"2024-01-15T18:15:45.401171+0000","last_became_active":"2024-01-15T18:15:44.159756+0000","last_became_peered":"2024-01-15T18:15:44.159756+0000","last_unstale":"2024-01-15T18:15:45.401171+0000","last_undegraded":"2024-01-15T18:15:45.401171+0000","last_fullsized":"2024-01-15T18:15:45.401171+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:41:43.148064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.059163+0000","last_change":"2024-01-15T18:15:45.174549+0000","last_active":"2024-01-15T18:15:47.059163+0000","last_peered":"2024-01-15T18:15:47.059163+0000","last_clean":"2024-01-15T18:15:47.059163+0000","last_became_active":"2024-01-15T18:15:45.174046+0000","last_became_peered":"2024-01-15T18:15:45.174046+0000","last_unstale":"2024-01-15T18:15:47.059163+0000","last_undegraded":"2024-01-15T18:15:47.059163+0000","last_fullsized":"2024-01-15T18:15:47.059163+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:07:58.498146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.167820+0000","last_change":"2024-01-15T18:15:44.154875+0000","last_active":"2024-01-15T18:15:45.167820+0000","last_peered":"2024-01-15T18:15:45.167820+0000","last_clean":"2024-01-15T18:15:45.167820+0000","last_became_active":"2024-01-15T18:15:44.154585+0000","last_became_peered":"2024-01-15T18:15:44.154585+0000","last_unstale":"2024-01-15T18:15:45.167820+0000","last_undegraded":"2024-01-15T18:15:45.167820+0000","last_fullsized":"2024-01-15T18:15:45.167820+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:58:47.607676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.059135+0000","last_change":"2024-01-15T18:15:45.193330+0000","last_active":"2024-01-15T18:15:47.059135+0000","last_peered":"2024-01-15T18:15:47.059135+0000","last_clean":"2024-01-15T18:15:47.059135+0000","last_became_active":"2024-01-15T18:15:45.192354+0000","last_became_peered":"2024-01-15T18:15:45.192354+0000","last_unstale":"2024-01-15T18:15:47.059135+0000","last_undegraded":"2024-01-15T18:15:47.059135+0000","last_fullsized":"2024-01-15T18:15:47.059135+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:01:31.532362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"83'8","reported_seq":20,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.202181+0000","last_change":"2024-01-15T18:15:44.168821+0000","last_active":"2024-01-15T18:15:45.202181+0000","last_peered":"2024-01-15T18:15:45.202181+0000","last_clean":"2024-01-15T18:15:45.202181+0000","last_became_active":"2024-01-15T18:15:44.168551+0000","last_became_peered":"2024-01-15T18:15:44.168551+0000","last_unstale":"2024-01-15T18:15:45.202181+0000","last_undegraded":"2024-01-15T18:15:45.202181+0000","last_fullsized":"2024-01-15T18:15:45.202181+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:32.543776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.184764+0000","last_change":"2024-01-15T18:15:45.180405+0000","last_active":"2024-01-15T18:15:46.184764+0000","last_peered":"2024-01-15T18:15:46.184764+0000","last_clean":"2024-01-15T18:15:46.184764+0000","last_became_active":"2024-01-15T18:15:45.179918+0000","last_became_peered":"2024-01-15T18:15:45.179918+0000","last_unstale":"2024-01-15T18:15:46.184764+0000","last_undegraded":"2024-01-15T18:15:46.184764+0000","last_fullsized":"2024-01-15T18:15:46.184764+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:29:58.891670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"83'1","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.450214+0000","last_change":"2024-01-15T18:15:44.159443+0000","last_active":"2024-01-15T18:15:45.450214+0000","last_peered":"2024-01-15T18:15:45.450214+0000","last_clean":"2024-01-15T18:15:45.450214+0000","last_became_active":"2024-01-15T18:15:44.159118+0000","last_became_peered":"2024-01-15T18:15:44.159118+0000","last_unstale":"2024-01-15T18:15:45.450214+0000","last_undegraded":"2024-01-15T18:15:45.450214+0000","last_fullsized":"2024-01-15T18:15:45.450214+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:13:11.055744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.776354+0000","last_change":"2024-01-15T18:15:45.195167+0000","last_active":"2024-01-15T18:15:46.776354+0000","last_peered":"2024-01-15T18:15:46.776354+0000","last_clean":"2024-01-15T18:15:46.776354+0000","last_became_active":"2024-01-15T18:15:45.194897+0000","last_became_peered":"2024-01-15T18:15:45.194897+0000","last_unstale":"2024-01-15T18:15:46.776354+0000","last_undegraded":"2024-01-15T18:15:46.776354+0000","last_fullsized":"2024-01-15T18:15:46.776354+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:41:44.153310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"83'2","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.198962+0000","last_change":"2024-01-15T18:15:44.169724+0000","last_active":"2024-01-15T18:15:45.198962+0000","last_peered":"2024-01-15T18:15:45.198962+0000","last_clean":"2024-01-15T18:15:45.198962+0000","last_became_active":"2024-01-15T18:15:44.169420+0000","last_became_peered":"2024-01-15T18:15:44.169420+0000","last_unstale":"2024-01-15T18:15:45.198962+0000","last_undegraded":"2024-01-15T18:15:45.198962+0000","last_fullsized":"2024-01-15T18:15:45.198962+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:04:34.325987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.058843+0000","last_change":"2024-01-15T18:15:45.175568+0000","last_active":"2024-01-15T18:15:47.058843+0000","last_peered":"2024-01-15T18:15:47.058843+0000","last_clean":"2024-01-15T18:15:47.058843+0000","last_became_active":"2024-01-15T18:15:45.173399+0000","last_became_peered":"2024-01-15T18:15:45.173399+0000","last_unstale":"2024-01-15T18:15:47.058843+0000","last_undegraded":"2024-01-15T18:15:47.058843+0000","last_fullsized":"2024-01-15T18:15:47.058843+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:40:13.829197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.170840+0000","last_change":"2024-01-15T18:15:44.168665+0000","last_active":"2024-01-15T18:15:45.170840+0000","last_peered":"2024-01-15T18:15:45.170840+0000","last_clean":"2024-01-15T18:15:45.170840+0000","last_became_active":"2024-01-15T18:15:44.168375+0000","last_became_peered":"2024-01-15T18:15:44.168375+0000","last_unstale":"2024-01-15T18:15:45.170840+0000","last_undegraded":"2024-01-15T18:15:45.170840+0000","last_fullsized":"2024-01-15T18:15:45.170840+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:30:48.177858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.059970+0000","last_change":"2024-01-15T18:15:45.179812+0000","last_active":"2024-01-15T18:15:47.059970+0000","last_peered":"2024-01-15T18:15:47.059970+0000","last_clean":"2024-01-15T18:15:47.059970+0000","last_became_active":"2024-01-15T18:15:45.179148+0000","last_became_peered":"2024-01-15T18:15:45.179148+0000","last_unstale":"2024-01-15T18:15:47.059970+0000","last_undegraded":"2024-01-15T18:15:47.059970+0000","last_fullsized":"2024-01-15T18:15:47.059970+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:18:22.468719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"83'2","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.401181+0000","last_change":"2024-01-15T18:15:44.163128+0000","last_active":"2024-01-15T18:15:45.401181+0000","last_peered":"2024-01-15T18:15:45.401181+0000","last_clean":"2024-01-15T18:15:45.401181+0000","last_became_active":"2024-01-15T18:15:44.162839+0000","last_became_peered":"2024-01-15T18:15:44.162839+0000","last_unstale":"2024-01-15T18:15:45.401181+0000","last_undegraded":"2024-01-15T18:15:45.401181+0000","last_fullsized":"2024-01-15T18:15:45.401181+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:34:15.516676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.058582+0000","last_change":"2024-01-15T18:15:45.188550+0000","last_active":"2024-01-15T18:15:47.058582+0000","last_peered":"2024-01-15T18:15:47.058582+0000","last_clean":"2024-01-15T18:15:47.058582+0000","last_became_active":"2024-01-15T18:15:45.188224+0000","last_became_peered":"2024-01-15T18:15:45.188224+0000","last_unstale":"2024-01-15T18:15:47.058582+0000","last_undegraded":"2024-01-15T18:15:47.058582+0000","last_fullsized":"2024-01-15T18:15:47.058582+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:04:35.331233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.406592+0000","last_change":"2024-01-15T18:15:44.167859+0000","last_active":"2024-01-15T18:15:45.406592+0000","last_peered":"2024-01-15T18:15:45.406592+0000","last_clean":"2024-01-15T18:15:45.406592+0000","last_became_active":"2024-01-15T18:15:44.167592+0000","last_became_peered":"2024-01-15T18:15:44.167592+0000","last_unstale":"2024-01-15T18:15:45.406592+0000","last_undegraded":"2024-01-15T18:15:45.406592+0000","last_fullsized":"2024-01-15T18:15:45.406592+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:02:39.986413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.059629+0000","last_change":"2024-01-15T18:15:45.179572+0000","last_active":"2024-01-15T18:15:47.059629+0000","last_peered":"2024-01-15T18:15:47.059629+0000","last_clean":"2024-01-15T18:15:47.059629+0000","last_became_active":"2024-01-15T18:15:45.178802+0000","last_became_peered":"2024-01-15T18:15:45.178802+0000","last_unstale":"2024-01-15T18:15:47.059629+0000","last_undegraded":"2024-01-15T18:15:47.059629+0000","last_fullsized":"2024-01-15T18:15:47.059629+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:54:05.847808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.406566+0000","last_change":"2024-01-15T18:15:44.168997+0000","last_active":"2024-01-15T18:15:45.406566+0000","last_peered":"2024-01-15T18:15:45.406566+0000","last_clean":"2024-01-15T18:15:45.406566+0000","last_became_active":"2024-01-15T18:15:44.168678+0000","last_became_peered":"2024-01-15T18:15:44.168678+0000","last_unstale":"2024-01-15T18:15:45.406566+0000","last_undegraded":"2024-01-15T18:15:45.406566+0000","last_fullsized":"2024-01-15T18:15:45.406566+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:20:19.634573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.058442+0000","last_change":"2024-01-15T18:15:45.180279+0000","last_active":"2024-01-15T18:15:47.058442+0000","last_peered":"2024-01-15T18:15:47.058442+0000","last_clean":"2024-01-15T18:15:47.058442+0000","last_became_active":"2024-01-15T18:15:45.177474+0000","last_became_peered":"2024-01-15T18:15:45.177474+0000","last_unstale":"2024-01-15T18:15:47.058442+0000","last_undegraded":"2024-01-15T18:15:47.058442+0000","last_fullsized":"2024-01-15T18:15:47.058442+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:08:41.639657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.401137+0000","last_change":"2024-01-15T18:15:44.162077+0000","last_active":"2024-01-15T18:15:45.401137+0000","last_peered":"2024-01-15T18:15:45.401137+0000","last_clean":"2024-01-15T18:15:45.401137+0000","last_became_active":"2024-01-15T18:15:44.161794+0000","last_became_peered":"2024-01-15T18:15:44.161794+0000","last_unstale":"2024-01-15T18:15:45.401137+0000","last_undegraded":"2024-01-15T18:15:45.401137+0000","last_fullsized":"2024-01-15T18:15:45.401137+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:05:24.652003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.059213+0000","last_change":"2024-01-15T18:15:45.175862+0000","last_active":"2024-01-15T18:15:47.059213+0000","last_peered":"2024-01-15T18:15:47.059213+0000","last_clean":"2024-01-15T18:15:47.059213+0000","last_became_active":"2024-01-15T18:15:45.175569+0000","last_became_peered":"2024-01-15T18:15:45.175569+0000","last_unstale":"2024-01-15T18:15:47.059213+0000","last_undegraded":"2024-01-15T18:15:47.059213+0000","last_fullsized":"2024-01-15T18:15:47.059213+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:26:01.224733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"83'2","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.430583+0000","last_change":"2024-01-15T18:15:44.168092+0000","last_active":"2024-01-15T18:15:45.430583+0000","last_peered":"2024-01-15T18:15:45.430583+0000","last_clean":"2024-01-15T18:15:45.430583+0000","last_became_active":"2024-01-15T18:15:44.167828+0000","last_became_peered":"2024-01-15T18:15:44.167828+0000","last_unstale":"2024-01-15T18:15:45.430583+0000","last_undegraded":"2024-01-15T18:15:45.430583+0000","last_fullsized":"2024-01-15T18:15:45.430583+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:27:26.285133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.184871+0000","last_change":"2024-01-15T18:15:45.194246+0000","last_active":"2024-01-15T18:15:46.184871+0000","last_peered":"2024-01-15T18:15:46.184871+0000","last_clean":"2024-01-15T18:15:46.184871+0000","last_became_active":"2024-01-15T18:15:45.193864+0000","last_became_peered":"2024-01-15T18:15:45.193864+0000","last_unstale":"2024-01-15T18:15:46.184871+0000","last_undegraded":"2024-01-15T18:15:46.184871+0000","last_fullsized":"2024-01-15T18:15:46.184871+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:28:14.766059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"84'1","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.059862+0000","last_change":"2024-01-15T18:15:44.166561+0000","last_active":"2024-01-15T18:15:47.059862+0000","last_peered":"2024-01-15T18:15:47.059862+0000","last_clean":"2024-01-15T18:15:47.059862+0000","last_became_active":"2024-01-15T18:15:44.166280+0000","last_became_peered":"2024-01-15T18:15:44.166280+0000","last_unstale":"2024-01-15T18:15:47.059862+0000","last_undegraded":"2024-01-15T18:15:47.059862+0000","last_fullsized":"2024-01-15T18:15:47.059862+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:46:30.074069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.776155+0000","last_change":"2024-01-15T18:15:45.196412+0000","last_active":"2024-01-15T18:15:46.776155+0000","last_peered":"2024-01-15T18:15:46.776155+0000","last_clean":"2024-01-15T18:15:46.776155+0000","last_became_active":"2024-01-15T18:15:45.196022+0000","last_became_peered":"2024-01-15T18:15:45.196022+0000","last_unstale":"2024-01-15T18:15:46.776155+0000","last_undegraded":"2024-01-15T18:15:46.776155+0000","last_fullsized":"2024-01-15T18:15:46.776155+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:08:31.073685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.450106+0000","last_change":"2024-01-15T18:15:44.163120+0000","last_active":"2024-01-15T18:15:45.450106+0000","last_peered":"2024-01-15T18:15:45.450106+0000","last_clean":"2024-01-15T18:15:45.450106+0000","last_became_active":"2024-01-15T18:15:44.162805+0000","last_became_peered":"2024-01-15T18:15:44.162805+0000","last_unstale":"2024-01-15T18:15:45.450106+0000","last_undegraded":"2024-01-15T18:15:45.450106+0000","last_fullsized":"2024-01-15T18:15:45.450106+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:49.344761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.058980+0000","last_change":"2024-01-15T18:15:45.185283+0000","last_active":"2024-01-15T18:15:47.058980+0000","last_peered":"2024-01-15T18:15:47.058980+0000","last_clean":"2024-01-15T18:15:47.058980+0000","last_became_active":"2024-01-15T18:15:45.185001+0000","last_became_peered":"2024-01-15T18:15:45.185001+0000","last_unstale":"2024-01-15T18:15:47.058980+0000","last_undegraded":"2024-01-15T18:15:47.058980+0000","last_fullsized":"2024-01-15T18:15:47.058980+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:01:49.697849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.450083+0000","last_change":"2024-01-15T18:15:44.163840+0000","last_active":"2024-01-15T18:15:45.450083+0000","last_peered":"2024-01-15T18:15:45.450083+0000","last_clean":"2024-01-15T18:15:45.450083+0000","last_became_active":"2024-01-15T18:15:44.163441+0000","last_became_peered":"2024-01-15T18:15:44.163441+0000","last_unstale":"2024-01-15T18:15:45.450083+0000","last_undegraded":"2024-01-15T18:15:45.450083+0000","last_fullsized":"2024-01-15T18:15:45.450083+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:41:04.792664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.058739+0000","last_change":"2024-01-15T18:15:45.187893+0000","last_active":"2024-01-15T18:15:47.058739+0000","last_peered":"2024-01-15T18:15:47.058739+0000","last_clean":"2024-01-15T18:15:47.058739+0000","last_became_active":"2024-01-15T18:15:45.187603+0000","last_became_peered":"2024-01-15T18:15:45.187603+0000","last_unstale":"2024-01-15T18:15:47.058739+0000","last_undegraded":"2024-01-15T18:15:47.058739+0000","last_fullsized":"2024-01-15T18:15:47.058739+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:32.310673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.170849+0000","last_change":"2024-01-15T18:15:44.169437+0000","last_active":"2024-01-15T18:15:45.170849+0000","last_peered":"2024-01-15T18:15:45.170849+0000","last_clean":"2024-01-15T18:15:45.170849+0000","last_became_active":"2024-01-15T18:15:44.169170+0000","last_became_peered":"2024-01-15T18:15:44.169170+0000","last_unstale":"2024-01-15T18:15:45.170849+0000","last_undegraded":"2024-01-15T18:15:45.170849+0000","last_fullsized":"2024-01-15T18:15:45.170849+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:08:40.634411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.184432+0000","last_change":"2024-01-15T18:15:45.183921+0000","last_active":"2024-01-15T18:15:46.184432+0000","last_peered":"2024-01-15T18:15:46.184432+0000","last_clean":"2024-01-15T18:15:46.184432+0000","last_became_active":"2024-01-15T18:15:45.183090+0000","last_became_peered":"2024-01-15T18:15:45.183090+0000","last_unstale":"2024-01-15T18:15:46.184432+0000","last_undegraded":"2024-01-15T18:15:46.184432+0000","last_fullsized":"2024-01-15T18:15:46.184432+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:21:19.894599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.336455+0000","last_change":"2024-01-15T18:15:44.154568+0000","last_active":"2024-01-15T18:15:45.336455+0000","last_peered":"2024-01-15T18:15:45.336455+0000","last_clean":"2024-01-15T18:15:45.336455+0000","last_became_active":"2024-01-15T18:15:44.154277+0000","last_became_peered":"2024-01-15T18:15:44.154277+0000","last_unstale":"2024-01-15T18:15:45.336455+0000","last_undegraded":"2024-01-15T18:15:45.336455+0000","last_fullsized":"2024-01-15T18:15:45.336455+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:39:47.546718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.184366+0000","last_change":"2024-01-15T18:15:45.182784+0000","last_active":"2024-01-15T18:15:46.184366+0000","last_peered":"2024-01-15T18:15:46.184366+0000","last_clean":"2024-01-15T18:15:46.184366+0000","last_became_active":"2024-01-15T18:15:45.179168+0000","last_became_peered":"2024-01-15T18:15:45.179168+0000","last_unstale":"2024-01-15T18:15:46.184366+0000","last_undegraded":"2024-01-15T18:15:46.184366+0000","last_fullsized":"2024-01-15T18:15:46.184366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:10:05.015678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.450035+0000","last_change":"2024-01-15T18:15:44.160891+0000","last_active":"2024-01-15T18:15:45.450035+0000","last_peered":"2024-01-15T18:15:45.450035+0000","last_clean":"2024-01-15T18:15:45.450035+0000","last_became_active":"2024-01-15T18:15:44.160625+0000","last_became_peered":"2024-01-15T18:15:44.160625+0000","last_unstale":"2024-01-15T18:15:45.450035+0000","last_undegraded":"2024-01-15T18:15:45.450035+0000","last_fullsized":"2024-01-15T18:15:45.450035+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:52.489949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:46.775948+0000","last_change":"2024-01-15T18:15:45.192344+0000","last_active":"2024-01-15T18:15:46.775948+0000","last_peered":"2024-01-15T18:15:46.775948+0000","last_clean":"2024-01-15T18:15:46.775948+0000","last_became_active":"2024-01-15T18:15:45.191845+0000","last_became_peered":"2024-01-15T18:15:45.191845+0000","last_unstale":"2024-01-15T18:15:46.775948+0000","last_undegraded":"2024-01-15T18:15:46.775948+0000","last_fullsized":"2024-01-15T18:15:46.775948+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:03:29.579452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.336410+0000","last_change":"2024-01-15T18:15:44.156428+0000","last_active":"2024-01-15T18:15:45.336410+0000","last_peered":"2024-01-15T18:15:45.336410+0000","last_clean":"2024-01-15T18:15:45.336410+0000","last_became_active":"2024-01-15T18:15:44.156132+0000","last_became_peered":"2024-01-15T18:15:44.156132+0000","last_unstale":"2024-01-15T18:15:45.336410+0000","last_undegraded":"2024-01-15T18:15:45.336410+0000","last_fullsized":"2024-01-15T18:15:45.336410+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:49.344761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":11,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-01-15T18:15:47.059570+0000","last_change":"2024-01-15T18:15:45.178846+0000","last_active":"2024-01-15T18:15:47.059570+0000","last_peered":"2024-01-15T18:15:47.059570+0000","last_clean":"2024-01-15T18:15:47.059570+0000","last_became_active":"2024-01-15T18:15:45.178552+0000","last_became_peered":"2024-01-15T18:15:45.178552+0000","last_unstale":"2024-01-15T18:15:47.059570+0000","last_undegraded":"2024-01-15T18:15:47.059570+0000","last_fullsized":"2024-01-15T18:15:47.059570+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:44.136826+0000","last_clean_scrub_stamp":"2024-01-15T18:15:44.136826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:45:42.971410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"83'1","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.385486+0000","last_change":"2024-01-15T18:15:44.170831+0000","last_active":"2024-01-15T18:15:45.385486+0000","last_peered":"2024-01-15T18:15:45.385486+0000","last_clean":"2024-01-15T18:15:45.385486+0000","last_became_active":"2024-01-15T18:15:44.170489+0000","last_became_peered":"2024-01-15T18:15:44.170489+0000","last_unstale":"2024-01-15T18:15:45.385486+0000","last_undegraded":"2024-01-15T18:15:45.385486+0000","last_fullsized":"2024-01-15T18:15:45.385486+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:15:43.131580+0000","last_clean_scrub_stamp":"2024-01-15T18:15:43.131580+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:54:03.053923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":219,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-01-15T18:15:45.170000+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:15:45.170000+0000","last_peered":"2024-01-15T18:15:45.170000+0000","last_clean":"2024-01-15T18:15:45.170000+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:15:45.170000+0000","last_undegraded":"2024-01-15T18:15:45.170000+0000","last_fullsized":"2024-01-15T18:15:45.170000+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509576,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77120,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294720,"statfs":{"total":96636764160,"available":96557793280,"internally_reserved":0,"allocated":1622016,"data_stored":1031202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77330890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":1.171,"15min":0.874},"min":{"1min":0.52400000000000002,"5min":0.38100000000000001,"15min":0.36199999999999999},"max":{"1min":1.0089999999999999,"5min":9.6989999999999998,"15min":10.284000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.68799999999999994,"5min":1.1399999999999999,"15min":0.90000000000000002},"min":{"1min":0.48599999999999999,"5min":0.39100000000000001,"15min":0.378},"max":{"1min":0.83099999999999996,"5min":16.968,"15min":16.968},"last":0.56699999999999995}]},{"osd":1,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":1.3129999999999999,"15min":0.91400000000000003},"min":{"1min":0.52600000000000002,"5min":0.36099999999999999,"15min":0.33800000000000002},"max":{"1min":0.90100000000000002,"5min":15.791,"15min":15.791},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.71099999999999997,"5min":1.109,"15min":0.90000000000000002},"min":{"1min":0.51500000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.85799999999999998,"5min":10.842000000000001,"15min":10.842000000000001},"last":0.42999999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":1.2609999999999999,"15min":0.92500000000000004},"min":{"1min":0.502,"5min":0.40400000000000003,"15min":0.378},"max":{"1min":0.98299999999999998,"5min":22.318000000000001,"15min":22.318000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.76200000000000001,"5min":1.159,"15min":0.89400000000000002},"min":{"1min":0.58399999999999996,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.93700000000000006,"5min":9.7539999999999996,"15min":9.7539999999999996},"last":0.627}]},{"osd":3,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":1.387,"15min":0.98899999999999999},"min":{"1min":0.59799999999999998,"5min":0.47099999999999997,"15min":0.38100000000000001},"max":{"1min":0.93300000000000005,"5min":18.152999999999999,"15min":18.152999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.81599999999999995,"5min":1.274,"15min":0.98599999999999999},"min":{"1min":0.622,"5min":0.433,"15min":0.43099999999999999},"max":{"1min":0.96599999999999997,"5min":9.7959999999999994,"15min":10.34},"last":0.59399999999999997}]},{"osd":4,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":1.2150000000000001,"15min":0.92700000000000005},"min":{"1min":0.44,"5min":0.314,"15min":0.314},"max":{"1min":0.85199999999999998,"5min":20.387,"15min":20.387},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.623,"5min":1.353,"15min":0.98399999999999999},"min":{"1min":0.47099999999999997,"5min":0.38800000000000001,"15min":0.36099999999999999},"max":{"1min":1.032,"5min":17.215,"15min":17.215},"last":0.39300000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":1.2410000000000001,"15min":0.94499999999999995},"min":{"1min":0.46400000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.95299999999999996,"5min":9.8450000000000006,"15min":19.959},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1.278,"15min":0.97699999999999998},"min":{"1min":0.439,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83999999999999997,"5min":19.905000000000001,"15min":19.905000000000001},"last":0.61399999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":1.4470000000000001,"15min":1.1499999999999999},"min":{"1min":0.59499999999999997,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97099999999999997,"5min":20.492000000000001,"15min":22.478999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.749,"5min":1.272,"15min":1.0600000000000001},"min":{"1min":0.52200000000000002,"5min":0.44,"15min":0.38300000000000001},"max":{"1min":0.998,"5min":17.132000000000001,"15min":21.509},"last":0.5}]}]},{"osd":6,"up_from":15,"seq":64424509576,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58560,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313280,"statfs":{"total":96636764160,"available":96576798720,"internally_reserved":0,"allocated":1036288,"data_stored":441458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":1.3009999999999999,"15min":0.95699999999999996},"min":{"1min":0.55700000000000005,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.96099999999999997,"5min":32.587000000000003,"15min":32.587000000000003},"last":0.622},{"interface":"front","average":{"1min":0.75,"5min":1.0660000000000001,"15min":0.86199999999999999},"min":{"1min":0.49199999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.97799999999999998,"5min":16.193000000000001,"15min":16.193000000000001},"last":0.73499999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.92400000000000004,"15min":0.98699999999999999},"min":{"1min":0.439,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.90700000000000003,"5min":7.7649999999999997,"15min":53.451999999999998},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.72499999999999998,"5min":1.2090000000000001,"15min":0.92600000000000005},"min":{"1min":0.53400000000000003,"5min":0.437,"15min":0.42899999999999999},"max":{"1min":0.86299999999999999,"5min":32.512,"15min":32.512},"last":0.78400000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":1.24,"15min":0.96799999999999997},"min":{"1min":0.58399999999999996,"5min":0.41699999999999998,"15min":0.38900000000000001},"max":{"1min":0.92200000000000004,"5min":32.472999999999999,"15min":32.472999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.754,"5min":1.0649999999999999,"15min":0.88500000000000001},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.38500000000000001},"max":{"1min":1.02,"5min":16.602,"15min":16.602},"last":0.65400000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":1.1699999999999999,"15min":0.93600000000000005},"min":{"1min":0.56399999999999995,"5min":0.46000000000000002,"15min":0.41399999999999998},"max":{"1min":0.92100000000000004,"5min":16.388000000000002,"15min":16.388000000000002},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.0680000000000001,"15min":1.075},"min":{"1min":0.54400000000000004,"5min":0.432,"15min":0.41599999999999998},"max":{"1min":0.95999999999999996,"5min":9.3719999999999999,"15min":53.398000000000003},"last":0.76300000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":1.149,"15min":0.89500000000000002},"min":{"1min":0.435,"5min":0.38100000000000001,"15min":0.34999999999999998},"max":{"1min":0.79700000000000004,"5min":32.549999999999997,"15min":32.549999999999997},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":1.0409999999999999,"15min":0.84899999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.34499999999999997},"max":{"1min":0.96499999999999997,"5min":11.852,"15min":11.852},"last":0.89700000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":1.0720000000000001,"15min":1.034},"min":{"1min":0.42699999999999999,"5min":0.35799999999999998,"15min":0.30399999999999999},"max":{"1min":0.77300000000000002,"5min":13.087999999999999,"15min":53.229999999999997},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.68700000000000006,"5min":1.375,"15min":1.0349999999999999},"min":{"1min":0.42599999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.0009999999999999,"5min":32.404000000000003,"15min":32.404000000000003},"last":0.93400000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":1.294,"15min":1.1200000000000001},"min":{"1min":0.52900000000000003,"5min":0.436,"15min":0.40400000000000003},"max":{"1min":0.94099999999999995,"5min":23.991,"15min":45.070999999999998},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.66100000000000003,"5min":1.3260000000000001,"15min":1.2170000000000001},"min":{"1min":0.51200000000000001,"5min":0.40999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88200000000000001,"5min":40.972999999999999,"15min":53.338000000000001},"last":0.91900000000000004}]}]},{"osd":4,"up_from":15,"seq":64424509576,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58548,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313292,"statfs":{"total":96636764160,"available":96576811008,"internally_reserved":0,"allocated":1024000,"data_stored":440764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":1.111,"15min":0.88500000000000001},"min":{"1min":0.46000000000000002,"5min":0.437,"15min":0.32900000000000001},"max":{"1min":0.96199999999999997,"5min":13.492000000000001,"15min":23.318000000000001},"last":0.623},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.92100000000000004,"15min":0.80200000000000005},"min":{"1min":0.41999999999999998,"5min":0.378,"15min":0.32600000000000001},"max":{"1min":0.875,"5min":14.723000000000001,"15min":14.723000000000001},"last":0.372}]},{"osd":1,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":1.0049999999999999,"15min":0.78800000000000003},"min":{"1min":0.50800000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.84299999999999997,"5min":9.5700000000000003,"15min":9.5700000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":1.155,"15min":0.91400000000000003},"min":{"1min":0.47399999999999998,"5min":0.40200000000000002,"15min":0.36899999999999999},"max":{"1min":0.84999999999999998,"5min":13.57,"15min":22.408999999999999},"last":0.41399999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":1.038,"15min":0.79400000000000004},"min":{"1min":0.53000000000000003,"5min":0.41599999999999998,"15min":0.34399999999999997},"max":{"1min":0.84199999999999997,"5min":9.7089999999999996,"15min":9.7089999999999996},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.73499999999999999,"5min":1.2949999999999999,"15min":1.014},"min":{"1min":0.439,"5min":0.439,"15min":0.39100000000000001},"max":{"1min":0.92600000000000005,"5min":14.118,"15min":22.338999999999999},"last":0.504}]},{"osd":3,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":1.016,"15min":0.873},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.41199999999999998},"max":{"1min":0.83199999999999996,"5min":14.763999999999999,"15min":14.763999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.77300000000000002,"5min":1.1479999999999999,"15min":0.90500000000000003},"min":{"1min":0.622,"5min":0.48099999999999998,"15min":0.45500000000000002},"max":{"1min":0.89400000000000002,"5min":9.6289999999999996,"15min":9.6289999999999996},"last":0.70999999999999996}]},{"osd":5,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":1.1299999999999999,"15min":0.96299999999999997},"min":{"1min":0.39000000000000001,"5min":0.34799999999999998,"15min":0.33200000000000002},"max":{"1min":0.875,"5min":15.787000000000001,"15min":31.617999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.94699999999999995,"15min":0.86699999999999999},"min":{"1min":0.44,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.92300000000000004,"5min":16.388000000000002,"15min":17.145},"last":0.53600000000000003}]},{"osd":6,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.98399999999999999,"15min":0.83099999999999996},"min":{"1min":0.42799999999999999,"5min":0.37,"15min":0.35499999999999998},"max":{"1min":0.94599999999999995,"5min":11.124000000000001,"15min":11.124000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.71699999999999997,"5min":1.3580000000000001,"15min":1.226},"min":{"1min":0.53700000000000003,"5min":0.434,"15min":0.40200000000000002},"max":{"1min":1.018,"5min":32.308,"15min":32.308},"last":0.68300000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":1.002,"15min":1.0109999999999999},"min":{"1min":0.49399999999999999,"5min":0.41099999999999998,"15min":0.34799999999999998},"max":{"1min":0.89900000000000002,"5min":14.653,"15min":39.256},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.746,"5min":1.2290000000000001,"15min":1.0129999999999999},"min":{"1min":0.57799999999999996,"5min":0.46500000000000002,"15min":0.42199999999999999},"max":{"1min":0.98599999999999999,"5min":16.277000000000001,"15min":26.573},"last":0.76300000000000001}]}]},{"osd":3,"up_from":15,"seq":64424509575,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80708,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94291132,"statfs":{"total":96636764160,"available":96554119168,"internally_reserved":0,"allocated":1036288,"data_stored":442376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":81590727},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.59199999999999997,"15min":0.49099999999999999},"min":{"1min":0.29299999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.751,"5min":5.8849999999999998,"15min":5.8849999999999998},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.68999999999999995,"15min":0.61599999999999999},"min":{"1min":0.38600000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.77500000000000002,"5min":4.7990000000000004,"15min":17.756},"last":0.622}]},{"osd":1,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.73099999999999998,"15min":0.59299999999999997},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.621,"5min":12.307,"15min":12.307},"last":0.437},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.69399999999999995,"15min":0.59299999999999997},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.32200000000000001},"max":{"1min":0.83799999999999997,"5min":6.0640000000000001,"15min":6.0640000000000001},"last":0.41799999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.83099999999999996,"15min":0.69599999999999995},"min":{"1min":0.46800000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":3.633,"5min":12.468999999999999,"15min":12.468999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.85399999999999998,"15min":0.68500000000000005},"min":{"1min":0.48599999999999999,"5min":0.41599999999999998,"15min":0.34599999999999997},"max":{"1min":1.0129999999999999,"5min":10.837999999999999,"15min":10.837999999999999},"last":0.54900000000000004}]},{"osd":4,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":1.292,"15min":1.028},"min":{"1min":0.59499999999999997,"5min":0.375,"15min":0.375},"max":{"1min":1.014,"5min":13.863,"15min":17.853999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.82899999999999996,"5min":1.2809999999999999,"15min":0.96899999999999997},"min":{"1min":0.63,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.0469999999999999,"5min":13.31,"15min":13.31},"last":0.69399999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":1.369,"15min":0.97599999999999998},"min":{"1min":0.66500000000000004,"5min":0.52100000000000002,"15min":0.51000000000000001},"max":{"1min":0.94399999999999995,"5min":17.687999999999999,"15min":17.687999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.206,"15min":0.97399999999999998},"min":{"1min":0.51700000000000002,"5min":0.438,"15min":0.438},"max":{"1min":3.2160000000000002,"5min":20.515999999999998,"15min":20.515999999999998},"last":0.68000000000000005}]},{"osd":6,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":1.1499999999999999,"15min":0.98999999999999999},"min":{"1min":0.55600000000000005,"5min":0.432,"15min":0.432},"max":{"1min":1.1060000000000001,"5min":13.250999999999999,"15min":23.100999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.82099999999999995,"5min":1.2150000000000001,"15min":0.93799999999999994},"min":{"1min":0.60799999999999998,"5min":0.5,"15min":0.49399999999999999},"max":{"1min":1.0169999999999999,"5min":12.76,"15min":12.76},"last":0.64900000000000002}]},{"osd":7,"last update":"Mon Jan 15 18:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.7889999999999999,"15min":1.1719999999999999},"min":{"1min":0.625,"5min":0.48599999999999999,"15min":0.48199999999999998},"max":{"1min":0.99099999999999999,"5min":24.321999999999999,"15min":24.321999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.81999999999999995,"5min":1.329,"15min":1.028},"min":{"1min":0.58399999999999996,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0569999999999999,"5min":13.385,"15min":13.385},"last":0.60599999999999998}]}]},{"osd":2,"up_from":15,"seq":64424509575,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76536,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295304,"statfs":{"total":96636764160,"available":96558391296,"internally_reserved":0,"allocated":1028096,"data_stored":441346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.51900000000000002,"15min":0.51200000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26500000000000001},"max":{"1min":0.66500000000000004,"5min":1.857,"15min":5.1849999999999996},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54200000000000004,"15min":0.53300000000000003},"min":{"1min":0.34100000000000003,"5min":0.252,"15min":0.252},"max":{"1min":0.89300000000000002,"5min":2.3759999999999999,"15min":7.516},"last":0.35299999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.70599999999999996,"15min":0.61599999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.875,"5min":10.907,"15min":10.907},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.68500000000000005,"15min":0.58799999999999997},"min":{"1min":0.38100000000000001,"5min":0.28100000000000003,"15min":0.27000000000000002},"max":{"1min":0.73799999999999999,"5min":7.2119999999999997,"15min":8.9190000000000005},"last":0.39600000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.60599999999999998,"15min":0.626},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.72999999999999998,"5min":2.7290000000000001,"15min":12.185},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.747,"15min":0.66100000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.34000000000000002},"max":{"1min":0.96699999999999997,"5min":7.6760000000000002,"15min":7.6760000000000002},"last":0.49199999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.89500000000000002,"15min":0.85799999999999998},"min":{"1min":0.55700000000000005,"5min":0.47499999999999998,"15min":0.45800000000000002},"max":{"1min":1.1870000000000001,"5min":4.3150000000000004,"15min":8.6609999999999996},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.755,"5min":0.878,"15min":0.90900000000000003},"min":{"1min":0.5,"5min":0.45800000000000002,"15min":0.42499999999999999},"max":{"1min":1.075,"5min":8.3190000000000008,"15min":19.146999999999998},"last":0.751}]},{"osd":5,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.80400000000000005,"15min":0.90200000000000002},"min":{"1min":0.52400000000000002,"5min":0.442,"15min":0.44},"max":{"1min":1.1599999999999999,"5min":2.7789999999999999,"15min":31.710000000000001},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.996,"15min":1},"min":{"1min":0.54400000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.155,"5min":10.460000000000001,"15min":22.539000000000001},"last":0.80200000000000005}]},{"osd":6,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.99399999999999999,"15min":0.89700000000000002},"min":{"1min":0.501,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.1220000000000001,"5min":8.3580000000000005,"15min":9.4540000000000006},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.83799999999999997,"15min":0.90500000000000003},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0489999999999999,"5min":2.669,"15min":18.614999999999998},"last":0.92200000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:15:02 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.125,"15min":0.95399999999999996},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.47999999999999998},"max":{"1min":1.157,"5min":11.606,"15min":11.606},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.92200000000000004,"15min":0.98699999999999999},"min":{"1min":0.60399999999999998,"5min":0.51700000000000002,"15min":0.502},"max":{"1min":1.1180000000000001,"5min":3.6850000000000001,"15min":17.013000000000002},"last":0.70299999999999996}]}]},{"osd":1,"up_from":15,"seq":64424509577,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80704,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94291136,"statfs":{"total":96636764160,"available":96554123264,"internally_reserved":0,"allocated":1036288,"data_stored":440958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81590725},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.70699999999999996,"15min":0.66300000000000003},"min":{"1min":0.26300000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.745,"5min":17.396999999999998,"15min":19.577000000000002},"last":0.623},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.74099999999999999,"15min":0.76000000000000001},"min":{"1min":0.33900000000000002,"5min":0.29199999999999998,"15min":0.251},"max":{"1min":0.83999999999999997,"5min":12.378,"15min":48.277999999999999},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.61399999999999999,"15min":0.71399999999999997},"min":{"1min":0.33600000000000002,"5min":0.309,"15min":0.30299999999999999},"max":{"1min":0.82799999999999996,"5min":2.8610000000000002,"15min":40.698},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.65900000000000003,"15min":0.90100000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.317},"max":{"1min":0.89500000000000002,"5min":3.0449999999999999,"15min":48.316000000000003},"last":0.441}]},{"osd":3,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.71799999999999997,"15min":0.84499999999999997},"min":{"1min":0.434,"5min":0.39800000000000002,"15min":0.33700000000000002},"max":{"1min":1.2509999999999999,"5min":3.0899999999999999,"15min":48.234999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.83799999999999997,"15min":0.68999999999999995},"min":{"1min":0.315,"5min":0.315,"15min":0.311},"max":{"1min":4.0780000000000003,"5min":9.9719999999999995,"15min":9.9719999999999995},"last":0.46000000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":1.0980000000000001,"15min":0.88700000000000001},"min":{"1min":0.49299999999999999,"5min":0.48999999999999999,"15min":0.44500000000000001},"max":{"1min":1.446,"5min":9.9329999999999998,"15min":9.9329999999999998},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.80800000000000005,"5min":1.026,"15min":0.93600000000000005},"min":{"1min":0.47499999999999998,"5min":0.45800000000000002,"15min":0.42499999999999999},"max":{"1min":1.163,"5min":9.8610000000000007,"15min":19.829000000000001},"last":0.88300000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.95399999999999996,"15min":1.0169999999999999},"min":{"1min":0.502,"5min":0.48599999999999999,"15min":0.46100000000000002},"max":{"1min":1.425,"5min":8.5839999999999996,"15min":22.594999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.82099999999999995,"5min":1.2629999999999999,"15min":1.163},"min":{"1min":0.58499999999999996,"5min":0.443,"15min":0.443},"max":{"1min":1.109,"5min":26.748999999999999,"15min":48.198},"last":0.76900000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":1.125,"15min":1.046},"min":{"1min":0.52200000000000002,"5min":0.50600000000000001,"15min":0.47499999999999998},"max":{"1min":1.879,"5min":9.6349999999999998,"15min":35.112000000000002},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.999,"15min":0.97599999999999998},"min":{"1min":0.58599999999999997,"5min":0.53100000000000003,"15min":0.50800000000000001},"max":{"1min":1.77,"5min":5.8849999999999998,"15min":19.780999999999999},"last":0.95199999999999996}]},{"osd":7,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3340000000000001,"15min":1.169},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.97499999999999998,"5min":20.872,"15min":48.148000000000003},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.85899999999999999,"5min":1.246,"15min":0.996},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.52100000000000002},"max":{"1min":1.52,"5min":9.8640000000000008,"15min":9.8640000000000008},"last":0.873}]}]},{"osd":5,"up_from":14,"seq":60129542280,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54404,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317436,"statfs":{"total":96636764160,"available":96581054464,"internally_reserved":0,"allocated":1040384,"data_stored":442398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.86699999999999999,"15min":0.80900000000000005},"min":{"1min":0.51800000000000002,"5min":0.29899999999999999,"15min":0.28999999999999998},"max":{"1min":3.8410000000000002,"5min":5.1639999999999997,"15min":21.385000000000002},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.0069999999999999,"15min":0.89600000000000002},"min":{"1min":0.57099999999999995,"5min":0.41599999999999998,"15min":0.34899999999999998},"max":{"1min":3.1080000000000001,"5min":9.359,"15min":9.359},"last":0.379}]},{"osd":1,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":1.3120000000000001,"15min":0.89600000000000002},"min":{"1min":0.55400000000000005,"5min":0.435,"15min":0.34899999999999998},"max":{"1min":1.0469999999999999,"5min":21.120000000000001,"15min":21.120000000000001},"last":0.502},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.2529999999999999,"15min":0.96499999999999997},"min":{"1min":0.628,"5min":0.38600000000000001,"15min":0.315},"max":{"1min":6.1310000000000002,"5min":15.843,"15min":21.5},"last":0.41299999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.98699999999999999,"15min":0.89800000000000002},"min":{"1min":0.51400000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":3.9220000000000002,"5min":6.2750000000000004,"15min":8.7750000000000004},"last":0.438},{"interface":"front","average":{"1min":1.135,"5min":1.3640000000000001,"15min":0.998},"min":{"1min":0.50800000000000001,"5min":0.42499999999999999,"15min":0.40400000000000003},"max":{"1min":8.3529999999999998,"5min":21.201000000000001,"15min":21.201000000000001},"last":0.57999999999999996}]},{"osd":3,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":0.98599999999999999,"15min":0.92200000000000004},"min":{"1min":0.60499999999999998,"5min":0.37,"15min":0.37},"max":{"1min":7.7199999999999998,"5min":7.7199999999999998,"15min":21.457000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.90300000000000002,"5min":1.034,"15min":0.97899999999999998},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.45400000000000001},"max":{"1min":3.8239999999999998,"5min":6.8650000000000002,"15min":8.7300000000000004},"last":0.54300000000000004}]},{"osd":4,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":1.0740000000000001,"15min":0.88800000000000001},"min":{"1min":0.51800000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.99199999999999999,"5min":10.962,"15min":10.962},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.69299999999999995,"5min":1.4410000000000001,"15min":1.034},"min":{"1min":0.48399999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0189999999999999,"5min":21.724,"15min":26.146999999999998},"last":0.46100000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.83899999999999997},"min":{"1min":0.44500000000000001,"5min":0.36099999999999999,"15min":0.32000000000000001},"max":{"1min":0.95599999999999996,"5min":5.2160000000000002,"15min":21.535},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.79000000000000004,"5min":1.012,"15min":1.0129999999999999},"min":{"1min":0.52400000000000002,"5min":0.379,"15min":0.36299999999999999},"max":{"1min":1.03,"5min":5.7539999999999996,"15min":30.488},"last":0.67500000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:15:14 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":2.0070000000000001,"15min":1.216},"min":{"1min":0.47699999999999998,"5min":0.42599999999999999,"15min":0.39700000000000002},"max":{"1min":1.008,"5min":33.499000000000002,"15min":33.499000000000002},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.81000000000000005,"5min":1.502,"15min":1.0880000000000001},"min":{"1min":0.54300000000000004,"5min":0.42899999999999999,"15min":0.36499999999999999},"max":{"1min":1.0609999999999999,"5min":25.437999999999999,"15min":25.437999999999999},"last":0.69399999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542280,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77120,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294720,"statfs":{"total":96636764160,"available":96557793280,"internally_reserved":0,"allocated":1622016,"data_stored":1031166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.90400000000000003,"15min":0.63},"min":{"1min":0.245,"5min":0.245,"15min":0.23699999999999999},"max":{"1min":0.78600000000000003,"5min":19.843,"15min":19.843},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.94699999999999995,"15min":0.68000000000000005},"min":{"1min":0.29999999999999999,"5min":0.29199999999999998,"15min":0.27300000000000002},"max":{"1min":0.746,"5min":10.154,"15min":10.154},"last":0.52200000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":1.2729999999999999,"15min":0.80400000000000005},"min":{"1min":0.39500000000000002,"5min":0.34899999999999998,"15min":0.30399999999999999},"max":{"1min":0.84099999999999997,"5min":25.905000000000001,"15min":25.905000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.82699999999999996,"15min":0.61199999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.26700000000000002},"max":{"1min":0.80000000000000004,"5min":9.3829999999999991,"15min":9.3829999999999991},"last":0.56799999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":1.026,"15min":0.74199999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.80400000000000005,"5min":9.1780000000000008,"15min":9.1780000000000008},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":1.016,"15min":0.76700000000000002},"min":{"1min":0.41699999999999998,"5min":0.40000000000000002,"15min":0.36899999999999999},"max":{"1min":0.89900000000000002,"5min":10.210000000000001,"15min":10.210000000000001},"last":0.49199999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":1.1319999999999999,"15min":0.98799999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.38},"max":{"1min":1.1459999999999999,"5min":10.691000000000001,"15min":22.814},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.80400000000000005,"5min":1.395,"15min":1.042},"min":{"1min":0.50600000000000001,"5min":0.49099999999999999,"15min":0.41199999999999998},"max":{"1min":1.046,"5min":23.341999999999999,"15min":23.341999999999999},"last":0.82699999999999996}]},{"osd":5,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":1.141,"15min":0.96899999999999997},"min":{"1min":0.55700000000000005,"5min":0.46899999999999997,"15min":0.441},"max":{"1min":1.046,"5min":10.737,"15min":12.428000000000001},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.83799999999999997,"5min":1.962,"15min":1.27},"min":{"1min":0.56999999999999995,"5min":0.498,"15min":0.46300000000000002},"max":{"1min":1.3049999999999999,"5min":23.021000000000001,"15min":23.021000000000001},"last":0.877}]},{"osd":6,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":1.696,"15min":1.218},"min":{"1min":0.52700000000000002,"5min":0.46700000000000003,"15min":0.46200000000000002},"max":{"1min":1.0229999999999999,"5min":26.036000000000001,"15min":31.059000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.88800000000000001,"5min":1.083,"15min":1.0920000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.49099999999999999},"max":{"1min":1.931,"5min":6.9690000000000003,"15min":40.731000000000002},"last":0.76700000000000002}]},{"osd":7,"last update":"Mon Jan 15 18:14:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.8400000000000001,"15min":1.208},"min":{"1min":0.58999999999999997,"5min":0.54900000000000004,"15min":0.42499999999999999},"max":{"1min":4.1609999999999996,"5min":24.492999999999999,"15min":24.492999999999999},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.83499999999999996,"5min":1.242,"15min":1.1140000000000001},"min":{"1min":0.64500000000000002,"5min":0.58399999999999996,"15min":0.46600000000000003},"max":{"1min":1.008,"5min":9.2789999999999999,"15min":23.524000000000001},"last":0.91300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:15:54.230 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-15T18:15:54.231 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-15T18:15:54.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.416+0000 7fd1d2000640 1 Processor -- start 2024-01-15T18:15:54.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.416+0000 7fd1d2000640 1 -- start start 2024-01-15T18:15:54.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.420+0000 7fd1d2000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd1cc0cf020 0x7fd1cc0b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:54.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.420+0000 7fd1d2000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc0712b0 con 0x7fd1cc0cec40 2024-01-15T18:15:54.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.420+0000 7fd1d2000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc0ccfb0 con 0x7fd1cc0b22f0 2024-01-15T18:15:54.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.420+0000 7fd1d2000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1cc0cd290 con 0x7fd1cc0cf020 2024-01-15T18:15:54.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.420+0000 7fd1caa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd1cc0cf020 0x7fd1cc0b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:54.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.420+0000 7fd1cb400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd1cc0cec40 0x7fd1cc070ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44332/0 (socket says 172.21.15.27:44332) 2024-01-15T18:15:54.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.420+0000 7fd1cb400640 1 -- 172.21.15.27:0/1913558332 learned_addr learned my addr 172.21.15.27:0/1913558332 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:54.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.420+0000 7fd1caa00640 1 -- 172.21.15.27:0/1913558332 >> v1:172.21.15.93:6789/0 conn(0x7fd1cc0b22f0 legacy=0x7fd1cc0ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:54.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.420+0000 7fd1caa00640 1 -- 172.21.15.27:0/1913558332 >> v1:172.21.15.27:6789/0 conn(0x7fd1cc0cec40 legacy=0x7fd1cc070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:54.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.420+0000 7fd1ca000640 1 -- 172.21.15.27:0/1913558332 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427355357 0 0) 0x7fd1cc0712b0 con 0x7fd1cc0cec40 2024-01-15T18:15:54.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.420+0000 7fd1caa00640 1 -- 172.21.15.27:0/1913558332 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1cc0cd570 con 0x7fd1cc0cf020 2024-01-15T18:15:54.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.424+0000 7fd1caa00640 1 --2- 172.21.15.27:0/1913558332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1cc0cf020 0x7fd1cc0b1db0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40097f0 tx=0x7fd1b401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea1477de234857e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:54.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.424+0000 7fd1ca000640 1 -- 172.21.15.27:0/1913558332 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b40ab600 con 0x7fd1cc0cf020 2024-01-15T18:15:54.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.424+0000 7fd1ca000640 1 -- 172.21.15.27:0/1913558332 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b40a7050 con 0x7fd1cc0cf020 2024-01-15T18:15:54.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.424+0000 7fd1ca000640 1 -- 172.21.15.27:0/1913558332 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b40ab980 con 0x7fd1cc0cf020 2024-01-15T18:15:54.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.424+0000 7fd1d2000640 1 -- 172.21.15.27:0/1913558332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1cc0cf020 msgr2=0x7fd1cc0b1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:54.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.424+0000 7fd1d2000640 1 --2- 172.21.15.27:0/1913558332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1cc0cf020 0x7fd1cc0b1db0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40097f0 tx=0x7fd1b401c3c0 comp rx=0 tx=0).stop 2024-01-15T18:15:54.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.424+0000 7fd1d2000640 1 -- 172.21.15.27:0/1913558332 shutdown_connections 2024-01-15T18:15:54.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.424+0000 7fd1d2000640 1 --2- 172.21.15.27:0/1913558332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1cc0cf020 0x7fd1cc0b1db0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:54.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.424+0000 7fd1d2000640 1 -- 172.21.15.27:0/1913558332 >> 172.21.15.27:0/1913558332 conn(0x7fd1cc0cb700 msgr2=0x7fd1cc0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:54.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.424+0000 7fd1d2000640 1 -- 172.21.15.27:0/1913558332 shutdown_connections 2024-01-15T18:15:54.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.424+0000 7fd1d2000640 1 -- 172.21.15.27:0/1913558332 wait complete. 2024-01-15T18:15:54.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.424+0000 7fd1d2000640 1 Processor -- start 2024-01-15T18:15:54.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.424+0000 7fd1d2000640 1 -- start start 2024-01-15T18:15:54.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1d2000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1cc0b22f0 0x7fd1cc158830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:54.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1d2000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd1cc0cec40 0x7fd1cc144c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:54.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1d2000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1cc0cf020 0x7fd1cc15ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:54.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1d2000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1cc157790 con 0x7fd1cc0b22f0 2024-01-15T18:15:54.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1d2000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1cc157a70 con 0x7fd1cc0cf020 2024-01-15T18:15:54.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1d2000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1cc157d50 con 0x7fd1cc0cec40 2024-01-15T18:15:54.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1cb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1cc0b22f0 0x7fd1cc158830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:54.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1cb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1cc0b22f0 0x7fd1cc158830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37292/0 (socket says 172.21.15.27:37292) 2024-01-15T18:15:54.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1cb400640 1 -- 172.21.15.27:0/3652038848 learned_addr learned my addr 172.21.15.27:0/3652038848 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:54.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1caa00640 1 --2- 172.21.15.27:0/3652038848 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd1cc0cec40 0x7fd1cc144c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:54.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1cbe00640 1 --2- 172.21.15.27:0/3652038848 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1cc0cf020 0x7fd1cc15ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:54.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1cb400640 1 -- 172.21.15.27:0/3652038848 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd1cc0cec40 msgr2=0x7fd1cc144c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:54.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1cb400640 1 --2- 172.21.15.27:0/3652038848 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd1cc0cec40 0x7fd1cc144c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:54.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1cb400640 1 -- 172.21.15.27:0/3652038848 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1cc0cf020 msgr2=0x7fd1cc15ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:54.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1cb400640 1 --2- 172.21.15.27:0/3652038848 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1cc0cf020 0x7fd1cc15ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:54.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1cb400640 1 -- 172.21.15.27:0/3652038848 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b4009420 con 0x7fd1cc0b22f0 2024-01-15T18:15:54.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1cbe00640 1 --2- 172.21.15.27:0/3652038848 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1cc0cf020 0x7fd1cc15ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:54.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1caa00640 1 --2- 172.21.15.27:0/3652038848 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd1cc0cec40 0x7fd1cc144c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:54.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1cb400640 1 --2- 172.21.15.27:0/3652038848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1cc0b22f0 0x7fd1cc158830 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc00c4a0 tx=0x7fd1bc00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:54.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1c3400640 1 -- 172.21.15.27:0/3652038848 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc0074e0 con 0x7fd1cc0b22f0 2024-01-15T18:15:54.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1c3400640 1 -- 172.21.15.27:0/3652038848 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1bc007640 con 0x7fd1cc0b22f0 2024-01-15T18:15:54.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1c3400640 1 -- 172.21.15.27:0/3652038848 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc004a30 con 0x7fd1cc0b22f0 2024-01-15T18:15:54.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1d2000640 1 -- 172.21.15.27:0/3652038848 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1cc158030 con 0x7fd1cc0b22f0 2024-01-15T18:15:54.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1d2000640 1 -- 172.21.15.27:0/3652038848 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1cc145c60 con 0x7fd1cc0b22f0 2024-01-15T18:15:54.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1d2000640 1 -- 172.21.15.27:0/3652038848 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1cc04e850 con 0x7fd1cc0b22f0 2024-01-15T18:15:54.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.428+0000 7fd1c3400640 1 -- 172.21.15.27:0/3652038848 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7fd1bc004b90 con 0x7fd1cc0b22f0 2024-01-15T18:15:54.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.432+0000 7fd1c3400640 1 --2- 172.21.15.27:0/3652038848 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd1a40674c0 0x7fd1a4069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:54.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.432+0000 7fd1c3400640 1 -- 172.21.15.27:0/3652038848 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fd1bc0822a0 con 0x7fd1cc0b22f0 2024-01-15T18:15:54.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.432+0000 7fd1caa00640 1 --2- 172.21.15.27:0/3652038848 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd1a40674c0 0x7fd1a4069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:54.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.432+0000 7fd1caa00640 1 --2- 172.21.15.27:0/3652038848 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd1a40674c0 0x7fd1a4069980 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40ad000 tx=0x7fd1b40a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:54.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.436+0000 7fd1c3400640 1 -- 172.21.15.27:0/3652038848 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd1bc050d30 con 0x7fd1cc0b22f0 2024-01-15T18:15:54.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.648+0000 7fd1d2000640 1 -- 172.21.15.27:0/3652038848 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fd1cc1584e0 con 0x7fd1cc0b22f0 2024-01-15T18:15:54.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.648+0000 7fd1c3400640 1 -- 172.21.15.27:0/3652038848 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+147 (secure 0 0 0) 0x7fd1bc018020 con 0x7fd1cc0b22f0 2024-01-15T18:15:54.648 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-01-15T18:15:54.648 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOc6VlH+DCLxAA1sKVut+3WjA/B+aYJ6/Jeg== 2024-01-15T18:15:54.648 INFO:teuthology.orchestra.run.smithi027.stdout: caps mds = "allow" 2024-01-15T18:15:54.648 INFO:teuthology.orchestra.run.smithi027.stdout: caps mon = "allow r" 2024-01-15T18:15:54.648 INFO:teuthology.orchestra.run.smithi027.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-15T18:15:54.649 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-01-15T18:15:54.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.652+0000 7fd1d2000640 1 -- 172.21.15.27:0/3652038848 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd1a40674c0 msgr2=0x7fd1a4069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:54.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.652+0000 7fd1d2000640 1 --2- 172.21.15.27:0/3652038848 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd1a40674c0 0x7fd1a4069980 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40ad000 tx=0x7fd1b40a6000 comp rx=0 tx=0).stop 2024-01-15T18:15:54.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.652+0000 7fd1d2000640 1 -- 172.21.15.27:0/3652038848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1cc0b22f0 msgr2=0x7fd1cc158830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:54.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.652+0000 7fd1d2000640 1 --2- 172.21.15.27:0/3652038848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1cc0b22f0 0x7fd1cc158830 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc00c4a0 tx=0x7fd1bc00c960 comp rx=0 tx=0).stop 2024-01-15T18:15:54.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.652+0000 7fd1d2000640 1 -- 172.21.15.27:0/3652038848 shutdown_connections 2024-01-15T18:15:54.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.652+0000 7fd1d2000640 1 --2- 172.21.15.27:0/3652038848 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd1a40674c0 0x7fd1a4069980 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:54.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.652+0000 7fd1d2000640 1 --2- 172.21.15.27:0/3652038848 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1cc0cf020 0x7fd1cc15ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:54.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.652+0000 7fd1d2000640 1 --2- 172.21.15.27:0/3652038848 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd1cc0cec40 0x7fd1cc144c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:54.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.652+0000 7fd1d2000640 1 --2- 172.21.15.27:0/3652038848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1cc0b22f0 0x7fd1cc158830 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:54.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.652+0000 7fd1d2000640 1 -- 172.21.15.27:0/3652038848 >> 172.21.15.27:0/3652038848 conn(0x7fd1cc0cb700 msgr2=0x7fd1cc0b4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:54.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.652+0000 7fd1d2000640 1 -- 172.21.15.27:0/3652038848 shutdown_connections 2024-01-15T18:15:54.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.652+0000 7fd1d2000640 1 -- 172.21.15.27:0/3652038848 wait complete. 2024-01-15T18:15:54.671 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:15:54.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.816+0000 7fe750400640 1 Processor -- start 2024-01-15T18:15:54.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.816+0000 7fe750400640 1 -- start start 2024-01-15T18:15:54.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.816+0000 7fe750400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fe7480cedc0 0x7fe7480b17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:54.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.816+0000 7fe750400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7480b38b0 con 0x7fe74815e7d0 2024-01-15T18:15:54.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.816+0000 7fe750400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7480cd550 con 0x7fe7480cf9b0 2024-01-15T18:15:54.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.816+0000 7fe750400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7480cd270 con 0x7fe7480cedc0 2024-01-15T18:15:54.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.816+0000 7fe74da00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe74815e7d0 0x7fe7480cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44342/0 (socket says 172.21.15.27:44342) 2024-01-15T18:15:54.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.816+0000 7fe74da00640 1 -- 172.21.15.27:0/1031142835 learned_addr learned my addr 172.21.15.27:0/1031142835 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:54.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.816+0000 7fe74e400640 1 --2- 172.21.15.27:0/1031142835 >> v2:172.21.15.93:3300/0 conn(0x7fe7480cedc0 0x7fe7480b17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:54.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.816+0000 7fe747e00640 1 -- 172.21.15.27:0/1031142835 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691294285 0 0) 0x7fe7480b38b0 con 0x7fe74815e7d0 2024-01-15T18:15:54.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.816+0000 7fe747e00640 1 -- 172.21.15.27:0/1031142835 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe728003530 con 0x7fe74815e7d0 2024-01-15T18:15:54.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.816+0000 7fe74e400640 1 -- 172.21.15.27:0/1031142835 >> v1:172.21.15.93:6789/0 conn(0x7fe7480cf9b0 legacy=0x7fe7480b10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:54.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.816+0000 7fe747e00640 1 -- 172.21.15.27:0/1031142835 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4217780601 0 0) 0x7fe728003530 con 0x7fe74815e7d0 2024-01-15T18:15:54.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.816+0000 7fe74e400640 1 -- 172.21.15.27:0/1031142835 >> v1:172.21.15.27:6789/0 conn(0x7fe74815e7d0 legacy=0x7fe7480cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:54.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.816+0000 7fe74e400640 1 -- 172.21.15.27:0/1031142835 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7480b35d0 con 0x7fe7480cedc0 2024-01-15T18:15:54.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.820+0000 7fe74e400640 1 --2- 172.21.15.27:0/1031142835 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7480cedc0 0x7fe7480b17d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fe740009d90 tx=0x7fe7480b2520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91f327669d753861 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:54.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.820+0000 7fe747e00640 1 -- 172.21.15.27:0/1031142835 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe738001680 con 0x7fe7480cedc0 2024-01-15T18:15:54.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.820+0000 7fe747e00640 1 -- 172.21.15.27:0/1031142835 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7400a5510 con 0x7fe7480cedc0 2024-01-15T18:15:54.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.820+0000 7fe747e00640 1 -- 172.21.15.27:0/1031142835 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7400a58a0 con 0x7fe7480cedc0 2024-01-15T18:15:54.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.820+0000 7fe750400640 1 -- 172.21.15.27:0/1031142835 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7480cedc0 msgr2=0x7fe7480b17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:54.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.820+0000 7fe750400640 1 --2- 172.21.15.27:0/1031142835 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7480cedc0 0x7fe7480b17d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fe740009d90 tx=0x7fe7480b2520 comp rx=0 tx=0).stop 2024-01-15T18:15:54.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.820+0000 7fe750400640 1 -- 172.21.15.27:0/1031142835 shutdown_connections 2024-01-15T18:15:54.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.820+0000 7fe750400640 1 --2- 172.21.15.27:0/1031142835 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7480cedc0 0x7fe7480b17d0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:54.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.820+0000 7fe750400640 1 -- 172.21.15.27:0/1031142835 >> 172.21.15.27:0/1031142835 conn(0x7fe7480b2a10 msgr2=0x7fe7480b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:54.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.820+0000 7fe750400640 1 -- 172.21.15.27:0/1031142835 shutdown_connections 2024-01-15T18:15:54.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.820+0000 7fe750400640 1 -- 172.21.15.27:0/1031142835 wait complete. 2024-01-15T18:15:54.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.824+0000 7fe750400640 1 Processor -- start 2024-01-15T18:15:54.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.824+0000 7fe750400640 1 -- start start 2024-01-15T18:15:54.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.824+0000 7fe750400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe7480cedc0 0x7fe748153c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:54.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.824+0000 7fe750400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7480cf9b0 0x7fe7481541d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:54.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.824+0000 7fe750400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe74815e7d0 0x7fe7480b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:54.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.824+0000 7fe750400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74815b150 con 0x7fe74815e7d0 2024-01-15T18:15:54.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.824+0000 7fe750400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74815ae70 con 0x7fe7480cf9b0 2024-01-15T18:15:54.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.824+0000 7fe750400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7480b5ec0 con 0x7fe7480cedc0 2024-01-15T18:15:54.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.824+0000 7fe74e400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe74815e7d0 0x7fe7480b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:54.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.824+0000 7fe74e400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe74815e7d0 0x7fe7480b3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37304/0 (socket says 172.21.15.27:37304) 2024-01-15T18:15:54.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.824+0000 7fe74e400640 1 -- 172.21.15.27:0/577301771 learned_addr learned my addr 172.21.15.27:0/577301771 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:54.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.824+0000 7fe74da00640 1 --2- 172.21.15.27:0/577301771 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe7480cedc0 0x7fe748153c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:54.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.824+0000 7fe74d000640 1 --2- 172.21.15.27:0/577301771 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7480cf9b0 0x7fe7481541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:54.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.824+0000 7fe74e400640 1 -- 172.21.15.27:0/577301771 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe7480cedc0 msgr2=0x7fe748153c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:54.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.824+0000 7fe74e400640 1 --2- 172.21.15.27:0/577301771 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe7480cedc0 0x7fe748153c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:54.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.824+0000 7fe74e400640 1 -- 172.21.15.27:0/577301771 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7480cf9b0 msgr2=0x7fe7481541d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:54.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.824+0000 7fe74e400640 1 --2- 172.21.15.27:0/577301771 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7480cf9b0 0x7fe7481541d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:54.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.824+0000 7fe74e400640 1 -- 172.21.15.27:0/577301771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe740009a70 con 0x7fe74815e7d0 2024-01-15T18:15:54.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.824+0000 7fe74da00640 1 --2- 172.21.15.27:0/577301771 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe7480cedc0 0x7fe748153c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:54.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.824+0000 7fe74e400640 1 --2- 172.21.15.27:0/577301771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe74815e7d0 0x7fe7480b3d30 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fe74001a410 tx=0x7fe740019580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:54.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.828+0000 7fe745600640 1 -- 172.21.15.27:0/577301771 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe740003ba0 con 0x7fe74815e7d0 2024-01-15T18:15:54.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.828+0000 7fe745600640 1 -- 172.21.15.27:0/577301771 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe740003d00 con 0x7fe74815e7d0 2024-01-15T18:15:54.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.828+0000 7fe745600640 1 -- 172.21.15.27:0/577301771 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe740004270 con 0x7fe74815e7d0 2024-01-15T18:15:54.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.828+0000 7fe750400640 1 -- 172.21.15.27:0/577301771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7480ae7a0 con 0x7fe74815e7d0 2024-01-15T18:15:54.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.828+0000 7fe750400640 1 -- 172.21.15.27:0/577301771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7480b61a0 con 0x7fe74815e7d0 2024-01-15T18:15:54.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.828+0000 7fe750400640 1 -- 172.21.15.27:0/577301771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7480ae990 con 0x7fe74815e7d0 2024-01-15T18:15:54.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.828+0000 7fe745600640 1 -- 172.21.15.27:0/577301771 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7fe7400b4040 con 0x7fe74815e7d0 2024-01-15T18:15:54.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.828+0000 7fe745600640 1 --2- 172.21.15.27:0/577301771 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe7200674c0 0x7fe720069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:54.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.828+0000 7fe745600640 1 -- 172.21.15.27:0/577301771 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fe74011d3d0 con 0x7fe74815e7d0 2024-01-15T18:15:54.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.828+0000 7fe74da00640 1 --2- 172.21.15.27:0/577301771 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe7200674c0 0x7fe720069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:54.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.832+0000 7fe74da00640 1 --2- 172.21.15.27:0/577301771 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe7200674c0 0x7fe720069980 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fe738000f70 tx=0x7fe738009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:54.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:54.832+0000 7fe745600640 1 -- 172.21.15.27:0/577301771 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe7400ebf20 con 0x7fe74815e7d0 2024-01-15T18:15:55.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.020+0000 7fe750400640 1 -- 172.21.15.27:0/577301771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe74804e850 con 0x7fe74815e7d0 2024-01-15T18:15:55.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.020+0000 7fe745600640 1 -- 172.21.15.27:0/577301771 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4912 (secure 0 0 0) 0x7fe7400b2090 con 0x7fe74815e7d0 2024-01-15T18:15:55.020 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:15:55.020 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":57,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8376,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/2584136338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2584136338},{"type":"v1","addr":"172.21.15.27:6833","nonce":2584136338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":8385,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2635542478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2635542478},{"type":"v1","addr":"172.21.15.27:6835","nonce":2635542478}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52},{"gid":8388,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/2822170069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":2822170069},{"type":"v1","addr":"172.21.15.93:6836","nonce":2822170069}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:15:44.541523+0000","modified":"2024-01-15T18:15:47.581737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8391},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8391":{"gid":8391,"name":"b","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.93:6837/980414657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":980414657},{"type":"v1","addr":"172.21.15.93:6837","nonce":980414657}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-01-15T18:15:55.020 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 57 2024-01-15T18:15:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.024+0000 7fe750400640 1 -- 172.21.15.27:0/577301771 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe7200674c0 msgr2=0x7fe720069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.024+0000 7fe750400640 1 --2- 172.21.15.27:0/577301771 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe7200674c0 0x7fe720069980 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fe738000f70 tx=0x7fe738009040 comp rx=0 tx=0).stop 2024-01-15T18:15:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.024+0000 7fe750400640 1 -- 172.21.15.27:0/577301771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe74815e7d0 msgr2=0x7fe7480b3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.024+0000 7fe750400640 1 --2- 172.21.15.27:0/577301771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe74815e7d0 0x7fe7480b3d30 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fe74001a410 tx=0x7fe740019580 comp rx=0 tx=0).stop 2024-01-15T18:15:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.024+0000 7fe750400640 1 -- 172.21.15.27:0/577301771 shutdown_connections 2024-01-15T18:15:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.024+0000 7fe750400640 1 --2- 172.21.15.27:0/577301771 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe7200674c0 0x7fe720069980 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.024+0000 7fe750400640 1 --2- 172.21.15.27:0/577301771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe74815e7d0 0x7fe7480b3d30 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.024+0000 7fe750400640 1 --2- 172.21.15.27:0/577301771 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7480cf9b0 0x7fe7481541d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.024+0000 7fe750400640 1 --2- 172.21.15.27:0/577301771 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe7480cedc0 0x7fe748153c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.024+0000 7fe750400640 1 -- 172.21.15.27:0/577301771 >> 172.21.15.27:0/577301771 conn(0x7fe7480b2a10 msgr2=0x7fe7480cc490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:55.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.024+0000 7fe750400640 1 -- 172.21.15.27:0/577301771 shutdown_connections 2024-01-15T18:15:55.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.024+0000 7fe750400640 1 -- 172.21.15.27:0/577301771 wait complete. 2024-01-15T18:15:55.445 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-15T18:15:44.541523+0000', 'modified': '2024-01-15T18:15:47.581737+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8391}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8391': {'gid': 8391, 'name': 'b', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.93:6837/980414657', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6835', 'nonce': 980414657}, {'type': 'v1', 'addr': '172.21.15.93:6837', 'nonce': 980414657}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-15T18:15:55.445 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-15T18:15:55.445 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-01-15T18:15:55.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.592+0000 7fc79fe00640 1 Processor -- start 2024-01-15T18:15:55.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.592+0000 7fc79fe00640 1 -- start start 2024-01-15T18:15:55.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.592+0000 7fc79fe00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fc79815b060 0x7fc79815a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.592+0000 7fc79fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc79815d9e0 0x7fc7980712e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.592+0000 7fc79fe00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc798071c70 con 0x7fc798071820 2024-01-15T18:15:55.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.592+0000 7fc79fe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc79815b440 con 0x7fc79815d9e0 2024-01-15T18:15:55.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.592+0000 7fc79fe00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc79815dfa0 con 0x7fc79815b060 2024-01-15T18:15:55.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.592+0000 7fc79ca00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc79815d9e0 0x7fc7980712e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.592+0000 7fc79ca00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc79815d9e0 0x7fc7980712e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37306/0 (socket says 172.21.15.27:37306) 2024-01-15T18:15:55.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.592+0000 7fc79ca00640 1 -- 172.21.15.27:0/2699765712 learned_addr learned my addr 172.21.15.27:0/2699765712 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:55.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.592+0000 7fc79de00640 1 --1- 172.21.15.27:0/2699765712 >> v1:172.21.15.27:6789/0 conn(0x7fc798071820 0x7fc7980701c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44354/0 (socket says 172.21.15.27:44354) 2024-01-15T18:15:55.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.592+0000 7fc79d400640 1 --2- 172.21.15.27:0/2699765712 >> v2:172.21.15.93:3300/0 conn(0x7fc79815b060 0x7fc79815a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.592+0000 7fc79ca00640 1 -- 172.21.15.27:0/2699765712 >> v2:172.21.15.93:3300/0 conn(0x7fc79815b060 msgr2=0x7fc79815a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:55.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.592+0000 7fc79ca00640 1 --2- 172.21.15.27:0/2699765712 >> v2:172.21.15.93:3300/0 conn(0x7fc79815b060 0x7fc79815a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.592+0000 7fc79ca00640 1 -- 172.21.15.27:0/2699765712 >> v1:172.21.15.27:6789/0 conn(0x7fc798071820 legacy=0x7fc7980701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:55.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.592+0000 7fc79ca00640 1 -- 172.21.15.27:0/2699765712 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79815e280 con 0x7fc79815d9e0 2024-01-15T18:15:55.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.596+0000 7fc79ca00640 1 --2- 172.21.15.27:0/2699765712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc79815d9e0 0x7fc7980712e0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fc79806bc10 tx=0x7fc78001c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1084577b7f05cecb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:55.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.596+0000 7fc797e00640 1 -- 172.21.15.27:0/2699765712 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7800ac410 con 0x7fc79815d9e0 2024-01-15T18:15:55.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.596+0000 7fc797e00640 1 -- 172.21.15.27:0/2699765712 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7800ac570 con 0x7fc79815d9e0 2024-01-15T18:15:55.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.596+0000 7fc797e00640 1 -- 172.21.15.27:0/2699765712 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7800ac900 con 0x7fc79815d9e0 2024-01-15T18:15:55.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.596+0000 7fc79fe00640 1 -- 172.21.15.27:0/2699765712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc79815d9e0 msgr2=0x7fc7980712e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:55.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.596+0000 7fc79fe00640 1 --2- 172.21.15.27:0/2699765712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc79815d9e0 0x7fc7980712e0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fc79806bc10 tx=0x7fc78001c3b0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.596+0000 7fc79fe00640 1 -- 172.21.15.27:0/2699765712 shutdown_connections 2024-01-15T18:15:55.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.596+0000 7fc79fe00640 1 --2- 172.21.15.27:0/2699765712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc79815d9e0 0x7fc7980712e0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.596+0000 7fc79fe00640 1 --2- 172.21.15.27:0/2699765712 >> v2:172.21.15.93:3300/0 conn(0x7fc79815b060 0x7fc79815a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.596+0000 7fc79fe00640 1 -- 172.21.15.27:0/2699765712 >> 172.21.15.27:0/2699765712 conn(0x7fc7980b0850 msgr2=0x7fc7980b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:55.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.596+0000 7fc79fe00640 1 -- 172.21.15.27:0/2699765712 shutdown_connections 2024-01-15T18:15:55.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.596+0000 7fc79fe00640 1 -- 172.21.15.27:0/2699765712 wait complete. 2024-01-15T18:15:55.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc79fe00640 1 Processor -- start 2024-01-15T18:15:55.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc79fe00640 1 -- start start 2024-01-15T18:15:55.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc79fe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc798071820 0x7fc7980cd740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc79fe00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc79815b060 0x7fc798150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc79fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc79815d9e0 0x7fc79814f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc79fe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7980ce550 con 0x7fc79815d9e0 2024-01-15T18:15:55.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc79fe00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7980cdf90 con 0x7fc79815b060 2024-01-15T18:15:55.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc79fe00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7980ce270 con 0x7fc798071820 2024-01-15T18:15:55.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc79de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc79815d9e0 0x7fc79814f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc79de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc79815d9e0 0x7fc79814f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37320/0 (socket says 172.21.15.27:37320) 2024-01-15T18:15:55.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc79de00640 1 -- 172.21.15.27:0/2433595796 learned_addr learned my addr 172.21.15.27:0/2433595796 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:55.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc79d400640 1 --2- 172.21.15.27:0/2433595796 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc798071820 0x7fc7980cd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc79ca00640 1 --2- 172.21.15.27:0/2433595796 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc79815b060 0x7fc798150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc79de00640 1 -- 172.21.15.27:0/2433595796 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc798071820 msgr2=0x7fc7980cd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:55.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc79de00640 1 --2- 172.21.15.27:0/2433595796 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc798071820 0x7fc7980cd740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc79de00640 1 -- 172.21.15.27:0/2433595796 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc79815b060 msgr2=0x7fc798150b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:55.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc79de00640 1 --2- 172.21.15.27:0/2433595796 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc79815b060 0x7fc798150b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc79de00640 1 -- 172.21.15.27:0/2433595796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7800094d0 con 0x7fc79815d9e0 2024-01-15T18:15:55.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc79d400640 1 --2- 172.21.15.27:0/2433595796 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc798071820 0x7fc7980cd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:55.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc79ca00640 1 --2- 172.21.15.27:0/2433595796 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc79815b060 0x7fc798150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:55.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc79de00640 1 --2- 172.21.15.27:0/2433595796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc79815d9e0 0x7fc79814f1c0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fc79000c7a0 tx=0x7fc79000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:55.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc795600640 1 -- 172.21.15.27:0/2433595796 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79000a640 con 0x7fc79815d9e0 2024-01-15T18:15:55.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc795600640 1 -- 172.21.15.27:0/2433595796 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79000a7a0 con 0x7fc79815d9e0 2024-01-15T18:15:55.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc79fe00640 1 -- 172.21.15.27:0/2433595796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc798150360 con 0x7fc79815d9e0 2024-01-15T18:15:55.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc795600640 1 -- 172.21.15.27:0/2433595796 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79000ab10 con 0x7fc79815d9e0 2024-01-15T18:15:55.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.600+0000 7fc79fe00640 1 -- 172.21.15.27:0/2433595796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7980cdc80 con 0x7fc79815d9e0 2024-01-15T18:15:55.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.604+0000 7fc795600640 1 -- 172.21.15.27:0/2433595796 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7fc79000ac70 con 0x7fc79815d9e0 2024-01-15T18:15:55.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.604+0000 7fc795600640 1 --2- 172.21.15.27:0/2433595796 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc7700674c0 0x7fc770069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.604+0000 7fc795600640 1 -- 172.21.15.27:0/2433595796 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fc790082740 con 0x7fc79815d9e0 2024-01-15T18:15:55.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.604+0000 7fc79d400640 1 --2- 172.21.15.27:0/2433595796 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc7700674c0 0x7fc770069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.604+0000 7fc7a32ea000 1 Processor -- start 2024-01-15T18:15:55.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.604+0000 7fc7a32ea000 1 -- start start 2024-01-15T18:15:55.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.604+0000 7fc79d400640 1 --2- 172.21.15.27:0/2433595796 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc7700674c0 0x7fc770069980 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fc788009ae0 tx=0x7fc788008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.604+0000 7fc7a32ea000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5608b6920c90 0x5608b6921070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.604+0000 7fc7a32ea000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b69215b0 0x5608b68d6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.604+0000 7fc7a32ea000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5608b68d7320 0x5608b68d7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.604+0000 7fc7a32ea000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608b6741130 con 0x5608b69215b0 2024-01-15T18:15:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.604+0000 7fc7a32ea000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608b672e9a0 con 0x5608b6920c90 2024-01-15T18:15:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.604+0000 7fc7a32ea000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x5608b672bed0 con 0x5608b68d7320 2024-01-15T18:15:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc79ca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b69215b0 0x5608b68d6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc79ca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b69215b0 0x5608b68d6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37332/0 (socket says 172.21.15.27:37332) 2024-01-15T18:15:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc79ca00640 1 -- 172.21.15.27:0/1147519119 learned_addr learned my addr 172.21.15.27:0/1147519119 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc79d400640 1 --2- 172.21.15.27:0/1147519119 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5608b6920c90 0x5608b6921070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc79de00640 1 --2- 172.21.15.27:0/1147519119 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5608b68d7320 0x5608b68d7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc79ca00640 1 -- 172.21.15.27:0/1147519119 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5608b68d7320 msgr2=0x5608b68d7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:55.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc79ca00640 1 --2- 172.21.15.27:0/1147519119 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5608b68d7320 0x5608b68d7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc79ca00640 1 -- 172.21.15.27:0/1147519119 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5608b6920c90 msgr2=0x5608b6921070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:55.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc79ca00640 1 --2- 172.21.15.27:0/1147519119 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5608b6920c90 0x5608b6921070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc79ca00640 1 -- 172.21.15.27:0/1147519119 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5608b641f890 con 0x5608b69215b0 2024-01-15T18:15:55.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc79de00640 1 --2- 172.21.15.27:0/1147519119 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5608b68d7320 0x5608b68d7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:55.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc79ca00640 1 --2- 172.21.15.27:0/1147519119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b69215b0 0x5608b68d6de0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fc780018fc0 tx=0x7fc780004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=880c73bb32d47c26 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:55.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc78e000640 1 -- 172.21.15.27:0/1147519119 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7800ad7f0 con 0x5608b69215b0 2024-01-15T18:15:55.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc78e000640 1 -- 172.21.15.27:0/1147519119 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7800ad950 con 0x5608b69215b0 2024-01-15T18:15:55.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc78e000640 1 -- 172.21.15.27:0/1147519119 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7800adcc0 con 0x5608b69215b0 2024-01-15T18:15:55.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc7a32ea000 1 -- 172.21.15.27:0/1147519119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b69215b0 msgr2=0x5608b68d6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:55.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc7a32ea000 1 --2- 172.21.15.27:0/1147519119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b69215b0 0x5608b68d6de0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fc780018fc0 tx=0x7fc780004740 comp rx=0 tx=0).stop 2024-01-15T18:15:55.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc7a32ea000 1 -- 172.21.15.27:0/1147519119 shutdown_connections 2024-01-15T18:15:55.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc7a32ea000 1 --2- 172.21.15.27:0/1147519119 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5608b68d7320 0x5608b68d7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc7a32ea000 1 --2- 172.21.15.27:0/1147519119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b69215b0 0x5608b68d6de0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc7a32ea000 1 --2- 172.21.15.27:0/1147519119 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5608b6920c90 0x5608b6921070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc7a32ea000 1 -- 172.21.15.27:0/1147519119 >> 172.21.15.27:0/1147519119 conn(0x5608b68d32d0 msgr2=0x5608b68d56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:55.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc7a32ea000 1 -- 172.21.15.27:0/1147519119 shutdown_connections 2024-01-15T18:15:55.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc7a32ea000 1 -- 172.21.15.27:0/1147519119 wait complete. 2024-01-15T18:15:55.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc7a32ea000 1 -- start start 2024-01-15T18:15:55.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.608+0000 7fc7a32ea000 1 Processor -- start 2024-01-15T18:15:55.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc7a32ea000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b68d7320 0x5608b68dbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc7a32ea000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5608b6920c90 0x5608b68dc280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc7a32ea000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5608b69215b0 0x5608b68ef910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc7a32ea000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608b67277c0 con 0x5608b68d7320 2024-01-15T18:15:55.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc7a32ea000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608b66dfbf0 con 0x5608b69215b0 2024-01-15T18:15:55.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc7a32ea000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x5608b66e5a90 con 0x5608b6920c90 2024-01-15T18:15:55.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc7a32ea000 10 client.0 _ll_register_callbacks cb 0x5608b68c23c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:15:55.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc79d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b68d7320 0x5608b68dbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc79d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b68d7320 0x5608b68dbd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37340/0 (socket says 172.21.15.27:37340) 2024-01-15T18:15:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc79d400640 1 -- 172.21.15.27:0/2369369700 learned_addr learned my addr 172.21.15.27:0/2369369700 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc79de00640 1 --2- 172.21.15.27:0/2369369700 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5608b69215b0 0x5608b68ef910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc79ca00640 1 --2- 172.21.15.27:0/2369369700 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5608b6920c90 0x5608b68dc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc79d400640 1 -- 172.21.15.27:0/2369369700 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5608b6920c90 msgr2=0x5608b68dc280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc79d400640 1 --2- 172.21.15.27:0/2369369700 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5608b6920c90 0x5608b68dc280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc79d400640 1 -- 172.21.15.27:0/2369369700 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5608b69215b0 msgr2=0x5608b68ef910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc79d400640 1 --2- 172.21.15.27:0/2369369700 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5608b69215b0 0x5608b68ef910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc79d400640 1 -- 172.21.15.27:0/2369369700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc79815e280 con 0x5608b68d7320 2024-01-15T18:15:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc79de00640 1 --2- 172.21.15.27:0/2369369700 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5608b69215b0 0x5608b68ef910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:15:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc79d400640 1 --2- 172.21.15.27:0/2369369700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b68d7320 0x5608b68dbd40 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fc788015a20 tx=0x7fc7880046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd02d0ed33bd0e49 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc786a00640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-15T18:15:55.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc786a00640 1 -- 172.21.15.27:0/2369369700 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc788016d30 con 0x5608b68d7320 2024-01-15T18:15:55.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc7a32ea000 1 -- 172.21.15.27:0/2369369700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5608b66e9470 con 0x5608b68d7320 2024-01-15T18:15:55.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc786a00640 1 -- 172.21.15.27:0/2369369700 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fc7880223f0 con 0x5608b68d7320 2024-01-15T18:15:55.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc786a00640 1 -- 172.21.15.27:0/2369369700 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc788016e90 con 0x5608b68d7320 2024-01-15T18:15:55.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc7a32ea000 10 client.8544 fetch_fsmap learned FSMap version 57 2024-01-15T18:15:55.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc7a32ea000 1 -- 172.21.15.27:0/2369369700 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) v3 -- 0x5608b66e8190 con 0x5608b68d7320 2024-01-15T18:15:55.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc786a00640 1 -- 172.21.15.27:0/2369369700 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5608b66e9470 con 0x5608b68d7320 2024-01-15T18:15:55.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc786a00640 1 -- 172.21.15.27:0/2369369700 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 57) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc7880225b0 con 0x5608b68d7320 2024-01-15T18:15:55.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc7a32ea000 10 client.8544 fetch_fsmap finished waiting for FSMap version 57 2024-01-15T18:15:55.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc7a32ea000 10 client.8544 resolve_mds: resolved name 'b' to daemon mds.b 2024-01-15T18:15:55.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc7a32ea000 20 client.8544 populate_metadata read hostname 'smithi027' 2024-01-15T18:15:55.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc7a32ea000 1 --2- 172.21.15.27:0/2369369700 >> [v2:172.21.15.93:6835/980414657,v1:172.21.15.93:6837/980414657] conn(0x5608b68f0320 0x5608b6932f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc7a32ea000 4 client.8544 mds_command: new command op to 8391 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-15T18:15:55.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc7a32ea000 1 -- 172.21.15.27:0/2369369700 --> [v2:172.21.15.93:6835/980414657,v1:172.21.15.93:6837/980414657] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5608b66ecb50 con 0x5608b68f0320 2024-01-15T18:15:55.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc79ca00640 1 --2- 172.21.15.27:0/2369369700 >> [v2:172.21.15.93:6835/980414657,v1:172.21.15.93:6837/980414657] conn(0x5608b68f0320 0x5608b6932f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc79ca00640 1 --2- 172.21.15.27:0/2369369700 >> [v2:172.21.15.93:6835/980414657,v1:172.21.15.93:6837/980414657] conn(0x5608b68f0320 0x5608b6932f50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8ba844ab7a01df server_cookie=5d13acf37931fef5 in_seq=0 out_seq=0 2024-01-15T18:15:55.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.612+0000 7fc786a00640 10 client.8544 ms_handle_connect on v2:172.21.15.93:6835/980414657 2024-01-15T18:15:55.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.616+0000 7fc786a00640 1 -- 172.21.15.27:0/2369369700 <== mds.0 v2:172.21.15.93:6835/980414657 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5608b66ecb50 con 0x5608b68f0320 2024-01-15T18:15:55.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.616+0000 7fc786a00640 10 client.8544 handle_command_reply: tid=0 2024-01-15T18:15:55.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.616+0000 7fc7a32ea000 1 client.8544 shutdown 2024-01-15T18:15:55.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.616+0000 7fc7a32ea000 1 -- 172.21.15.27:0/2369369700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b68d7320 msgr2=0x5608b68dbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:55.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.616+0000 7fc7a32ea000 1 --2- 172.21.15.27:0/2369369700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b68d7320 0x5608b68dbd40 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fc788015a20 tx=0x7fc7880046d0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.616+0000 7fc7a32ea000 1 -- 172.21.15.27:0/2369369700 shutdown_connections 2024-01-15T18:15:55.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.616+0000 7fc7a32ea000 1 --2- 172.21.15.27:0/2369369700 >> [v2:172.21.15.93:6835/980414657,v1:172.21.15.93:6837/980414657] conn(0x5608b68f0320 0x5608b6932f50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.616+0000 7fc7a32ea000 1 --2- 172.21.15.27:0/2369369700 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5608b69215b0 0x5608b68ef910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.616+0000 7fc7a32ea000 1 --2- 172.21.15.27:0/2369369700 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5608b6920c90 0x5608b68dc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.616+0000 7fc7a32ea000 1 --2- 172.21.15.27:0/2369369700 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b68d7320 0x5608b68dbd40 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.616+0000 7fc786a00640 0 client.8544 ms_handle_reset on v2:172.21.15.93:6835/980414657 2024-01-15T18:15:55.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.616+0000 7fc7a32ea000 1 -- 172.21.15.27:0/2369369700 >> 172.21.15.27:0/2369369700 conn(0x5608b68d32d0 msgr2=0x5608b68d56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:55.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.616+0000 7fc7a32ea000 1 -- 172.21.15.27:0/2369369700 shutdown_connections 2024-01-15T18:15:55.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.616+0000 7fc7a32ea000 1 -- 172.21.15.27:0/2369369700 wait complete. 2024-01-15T18:15:55.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.616+0000 7fc7a32ea000 20 client.8544 trim_cache size 0 max 16384 2024-01-15T18:15:55.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.632+0000 7fc7a32ea000 1 Processor -- start 2024-01-15T18:15:55.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.632+0000 7fc7a32ea000 1 -- start start 2024-01-15T18:15:55.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.632+0000 7fc7a32ea000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5608b68d7320 0x5608b691fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.632+0000 7fc7a32ea000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5608b6920c90 0x5608b691eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.632+0000 7fc7a32ea000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b69215b0 0x5608b691f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.632+0000 7fc7a32ea000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608b63e0940 con 0x5608b69215b0 2024-01-15T18:15:55.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.632+0000 7fc7a32ea000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608b68de890 con 0x5608b6920c90 2024-01-15T18:15:55.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.632+0000 7fc7a32ea000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x5608b66e6e20 con 0x5608b68d7320 2024-01-15T18:15:55.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.632+0000 7fc79de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b69215b0 0x5608b691f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.632+0000 7fc79de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b69215b0 0x5608b691f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37346/0 (socket says 172.21.15.27:37346) 2024-01-15T18:15:55.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.632+0000 7fc79de00640 1 -- 172.21.15.27:0/597885145 learned_addr learned my addr 172.21.15.27:0/597885145 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:55.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.632+0000 7fc79d400640 1 --2- 172.21.15.27:0/597885145 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5608b68d7320 0x5608b691fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.632+0000 7fc79ca00640 1 --2- 172.21.15.27:0/597885145 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5608b6920c90 0x5608b691eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.632+0000 7fc79de00640 1 -- 172.21.15.27:0/597885145 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5608b68d7320 msgr2=0x5608b691fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:55.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.632+0000 7fc79de00640 1 --2- 172.21.15.27:0/597885145 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5608b68d7320 0x5608b691fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.632+0000 7fc79de00640 1 -- 172.21.15.27:0/597885145 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5608b6920c90 msgr2=0x5608b691eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:55.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.632+0000 7fc79de00640 1 --2- 172.21.15.27:0/597885145 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5608b6920c90 0x5608b691eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.632+0000 7fc79de00640 1 -- 172.21.15.27:0/597885145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5608b66ce490 con 0x5608b69215b0 2024-01-15T18:15:55.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.632+0000 7fc79d400640 1 --2- 172.21.15.27:0/597885145 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5608b68d7320 0x5608b691fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:55.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.632+0000 7fc79de00640 1 --2- 172.21.15.27:0/597885145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b69215b0 0x5608b691f070 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fc790001340 tx=0x7fc790054f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d40ee140ecc652 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.632+0000 7fc78ea00640 1 -- 172.21.15.27:0/597885145 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc790054350 con 0x5608b69215b0 2024-01-15T18:15:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.632+0000 7fc78ea00640 1 -- 172.21.15.27:0/597885145 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7900544b0 con 0x5608b69215b0 2024-01-15T18:15:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.632+0000 7fc78ea00640 1 -- 172.21.15.27:0/597885145 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc790054620 con 0x5608b69215b0 2024-01-15T18:15:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc7a32ea000 1 -- 172.21.15.27:0/597885145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b69215b0 msgr2=0x5608b691f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc7a32ea000 1 --2- 172.21.15.27:0/597885145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b69215b0 0x5608b691f070 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fc790001340 tx=0x7fc790054f20 comp rx=0 tx=0).stop 2024-01-15T18:15:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc7a32ea000 1 -- 172.21.15.27:0/597885145 shutdown_connections 2024-01-15T18:15:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc7a32ea000 1 --2- 172.21.15.27:0/597885145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b69215b0 0x5608b691f070 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc7a32ea000 1 --2- 172.21.15.27:0/597885145 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5608b6920c90 0x5608b691eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc7a32ea000 1 --2- 172.21.15.27:0/597885145 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5608b68d7320 0x5608b691fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc7a32ea000 1 -- 172.21.15.27:0/597885145 >> 172.21.15.27:0/597885145 conn(0x5608b68d32d0 msgr2=0x5608b692e280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc7a32ea000 1 -- 172.21.15.27:0/597885145 shutdown_connections 2024-01-15T18:15:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc7a32ea000 1 -- 172.21.15.27:0/597885145 wait complete. 2024-01-15T18:15:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc7a32ea000 1 -- start start 2024-01-15T18:15:55.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc7a32ea000 1 Processor -- start 2024-01-15T18:15:55.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc7a32ea000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5608b68d7320 0x5608b6903ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc7a32ea000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b6920c90 0x5608b68d86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc7a32ea000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5608b69215b0 0x5608b68d8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc7a32ea000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608b6726fd0 con 0x5608b6920c90 2024-01-15T18:15:55.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc7a32ea000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608b66aea00 con 0x5608b69215b0 2024-01-15T18:15:55.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc7a32ea000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x5608b67235c0 con 0x5608b68d7320 2024-01-15T18:15:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc7a32ea000 10 client.0 _ll_register_callbacks cb 0x5608b6775c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:15:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc79ca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b6920c90 0x5608b68d86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc79d400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5608b68d7320 0x5608b6903ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc79ca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b6920c90 0x5608b68d86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37360/0 (socket says 172.21.15.27:37360) 2024-01-15T18:15:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc79ca00640 1 -- 172.21.15.27:0/192666286 learned_addr learned my addr 172.21.15.27:0/192666286 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc79de00640 1 --2- 172.21.15.27:0/192666286 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5608b69215b0 0x5608b68d8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc79ca00640 1 -- 172.21.15.27:0/192666286 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5608b68d7320 msgr2=0x5608b6903ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc79ca00640 1 --2- 172.21.15.27:0/192666286 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5608b68d7320 0x5608b6903ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc79ca00640 1 -- 172.21.15.27:0/192666286 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5608b69215b0 msgr2=0x5608b68d8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc79ca00640 1 --2- 172.21.15.27:0/192666286 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5608b69215b0 0x5608b68d8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc79ca00640 1 -- 172.21.15.27:0/192666286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7980cdc80 con 0x5608b6920c90 2024-01-15T18:15:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc79de00640 1 --2- 172.21.15.27:0/192666286 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5608b69215b0 0x5608b68d8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc79ca00640 1 --2- 172.21.15.27:0/192666286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b6920c90 0x5608b68d86a0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fc78001c920 tx=0x7fc7800046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63d22c9ae795a2cb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:55.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc786a00640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-15T18:15:55.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc786a00640 1 -- 172.21.15.27:0/192666286 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc780006af0 con 0x5608b6920c90 2024-01-15T18:15:55.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc786a00640 1 -- 172.21.15.27:0/192666286 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fc780004220 con 0x5608b6920c90 2024-01-15T18:15:55.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc7a32ea000 1 -- 172.21.15.27:0/192666286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5608b6726110 con 0x5608b6920c90 2024-01-15T18:15:55.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc786a00640 1 -- 172.21.15.27:0/192666286 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc7800043e0 con 0x5608b6920c90 2024-01-15T18:15:55.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc7a32ea000 10 client.8550 fetch_fsmap learned FSMap version 57 2024-01-15T18:15:55.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.636+0000 7fc7a32ea000 1 -- 172.21.15.27:0/192666286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) v3 -- 0x5608b641fcc0 con 0x5608b6920c90 2024-01-15T18:15:55.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc786a00640 1 -- 172.21.15.27:0/192666286 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5608b6726110 con 0x5608b6920c90 2024-01-15T18:15:55.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc786a00640 1 -- 172.21.15.27:0/192666286 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 57) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc7800ad770 con 0x5608b6920c90 2024-01-15T18:15:55.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc7a32ea000 10 client.8550 fetch_fsmap finished waiting for FSMap version 57 2024-01-15T18:15:55.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc7a32ea000 10 client.8550 resolve_mds: resolved name 'b' to daemon mds.b 2024-01-15T18:15:55.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc7a32ea000 20 client.8550 populate_metadata read hostname 'smithi027' 2024-01-15T18:15:55.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc7a32ea000 1 --2- 172.21.15.27:0/192666286 >> [v2:172.21.15.93:6835/980414657,v1:172.21.15.93:6837/980414657] conn(0x5608b6987e30 0x5608b698a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc7a32ea000 4 client.8550 mds_command: new command op to 8391 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-15T18:15:55.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc7a32ea000 1 -- 172.21.15.27:0/192666286 --> [v2:172.21.15.93:6835/980414657,v1:172.21.15.93:6837/980414657] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5608b67559a0 con 0x5608b6987e30 2024-01-15T18:15:55.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc79d400640 1 --2- 172.21.15.27:0/192666286 >> [v2:172.21.15.93:6835/980414657,v1:172.21.15.93:6837/980414657] conn(0x5608b6987e30 0x5608b698a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc79d400640 1 --2- 172.21.15.27:0/192666286 >> [v2:172.21.15.93:6835/980414657,v1:172.21.15.93:6837/980414657] conn(0x5608b6987e30 0x5608b698a220 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ab0e94d3e877c87 server_cookie=57b55fa0aa5e681b in_seq=0 out_seq=0 2024-01-15T18:15:55.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc786a00640 10 client.8550 ms_handle_connect on v2:172.21.15.93:6835/980414657 2024-01-15T18:15:55.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc786a00640 1 -- 172.21.15.27:0/192666286 <== mds.0 v2:172.21.15.93:6835/980414657 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+319 (crc 0 0 0) 0x5608b67559a0 con 0x5608b6987e30 2024-01-15T18:15:55.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc786a00640 10 client.8550 handle_command_reply: tid=0 2024-01-15T18:15:55.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc7a32ea000 1 client.8550 shutdown 2024-01-15T18:15:55.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc7a32ea000 1 -- 172.21.15.27:0/192666286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b6920c90 msgr2=0x5608b68d86a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:55.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc7a32ea000 1 --2- 172.21.15.27:0/192666286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b6920c90 0x5608b68d86a0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fc78001c920 tx=0x7fc7800046d0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc7a32ea000 1 -- 172.21.15.27:0/192666286 shutdown_connections 2024-01-15T18:15:55.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc7a32ea000 1 --2- 172.21.15.27:0/192666286 >> [v2:172.21.15.93:6835/980414657,v1:172.21.15.93:6837/980414657] conn(0x5608b6987e30 0x5608b698a220 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc7a32ea000 1 --2- 172.21.15.27:0/192666286 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x5608b69215b0 0x5608b68d8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc7a32ea000 1 --2- 172.21.15.27:0/192666286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5608b6920c90 0x5608b68d86a0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc7a32ea000 1 --2- 172.21.15.27:0/192666286 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x5608b68d7320 0x5608b6903ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc786a00640 0 client.8550 ms_handle_reset on v2:172.21.15.93:6835/980414657 2024-01-15T18:15:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc7a32ea000 1 -- 172.21.15.27:0/192666286 >> 172.21.15.27:0/192666286 conn(0x5608b68d32d0 msgr2=0x5608b68d56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc7a32ea000 1 -- 172.21.15.27:0/192666286 shutdown_connections 2024-01-15T18:15:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc7a32ea000 1 -- 172.21.15.27:0/192666286 wait complete. 2024-01-15T18:15:55.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc7a32ea000 20 client.8550 trim_cache size 0 max 16384 2024-01-15T18:15:55.642 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:15:55.642 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:15:55.642 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-01-15T18:15:55.642 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 8391, 2024-01-15T18:15:55.642 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-01-15T18:15:55.643 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-01-15T18:15:55.643 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-01-15T18:15:55.643 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 10.484086848, 2024-01-15T18:15:55.643 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 57, 2024-01-15T18:15:55.643 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 84, 2024-01-15T18:15:55.643 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 84, 2024-01-15T18:15:55.643 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 15.364268765 2024-01-15T18:15:55.643 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:15:55.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc79fe00640 1 -- 172.21.15.27:0/2433595796 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc7700674c0 msgr2=0x7fc770069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:55.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc79fe00640 1 --2- 172.21.15.27:0/2433595796 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc7700674c0 0x7fc770069980 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fc788009ae0 tx=0x7fc788008040 comp rx=0 tx=0).stop 2024-01-15T18:15:55.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc79fe00640 1 -- 172.21.15.27:0/2433595796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc79815d9e0 msgr2=0x7fc79814f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:55.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc79fe00640 1 --2- 172.21.15.27:0/2433595796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc79815d9e0 0x7fc79814f1c0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7fc79000c7a0 tx=0x7fc79000cc60 comp rx=0 tx=0).stop 2024-01-15T18:15:55.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc79fe00640 1 -- 172.21.15.27:0/2433595796 shutdown_connections 2024-01-15T18:15:55.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc79fe00640 1 --2- 172.21.15.27:0/2433595796 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc7700674c0 0x7fc770069980 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc79fe00640 1 --2- 172.21.15.27:0/2433595796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc79815d9e0 0x7fc79814f1c0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc79fe00640 1 --2- 172.21.15.27:0/2433595796 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc79815b060 0x7fc798150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc79fe00640 1 --2- 172.21.15.27:0/2433595796 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc798071820 0x7fc7980cd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc79fe00640 1 -- 172.21.15.27:0/2433595796 >> 172.21.15.27:0/2433595796 conn(0x7fc7980b0850 msgr2=0x7fc7980b3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:55.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc79fe00640 1 -- 172.21.15.27:0/2433595796 shutdown_connections 2024-01-15T18:15:55.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.640+0000 7fc79fe00640 1 -- 172.21.15.27:0/2433595796 wait complete. 2024-01-15T18:15:55.660 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:15:55.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff872400640 1 Processor -- start 2024-01-15T18:15:55.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff872400640 1 -- start start 2024-01-15T18:15:55.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff872400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff86c06bb80 0x7ff86c0cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff872400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff86c0b2770 con 0x7ff86c0cddc0 2024-01-15T18:15:55.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff872400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff86c06c060 con 0x7ff86c0b2390 2024-01-15T18:15:55.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff872400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff86c0ce270 con 0x7ff86c06bb80 2024-01-15T18:15:55.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86be00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff86c0cddc0 0x7ff86c0b0880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44362/0 (socket says 172.21.15.27:44362) 2024-01-15T18:15:55.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86be00640 1 -- 172.21.15.27:0/930016591 learned_addr learned my addr 172.21.15.27:0/930016591 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86aa00640 1 --2- 172.21.15.27:0/930016591 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff86c06bb80 0x7ff86c0cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86a000640 1 -- 172.21.15.27:0/930016591 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135015840 0 0) 0x7ff86c0b2770 con 0x7ff86c0cddc0 2024-01-15T18:15:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86a000640 1 -- 172.21.15.27:0/930016591 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff84c003530 con 0x7ff86c0cddc0 2024-01-15T18:15:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86a000640 1 -- 172.21.15.27:0/930016591 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1767590736 0 0) 0x7ff84c003530 con 0x7ff86c0cddc0 2024-01-15T18:15:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86a000640 1 -- 172.21.15.27:0/930016591 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff86c06bb80 msgr2=0x7ff86c0cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86a000640 1 --2- 172.21.15.27:0/930016591 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff86c06bb80 0x7ff86c0cd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86a000640 1 -- 172.21.15.27:0/930016591 >> v1:172.21.15.93:6789/0 conn(0x7ff86c0b2390 legacy=0x7ff86c0cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86a000640 1 -- 172.21.15.27:0/930016591 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86c06c340 con 0x7ff86c0cddc0 2024-01-15T18:15:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86a000640 1 -- 172.21.15.27:0/930016591 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7ff864001a00 con 0x7ff86c0cddc0 2024-01-15T18:15:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86a000640 1 -- 172.21.15.27:0/930016591 >> v1:172.21.15.27:6789/0 conn(0x7ff86c0cddc0 legacy=0x7ff86c0b0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86a000640 1 --2- 172.21.15.27:0/930016591 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff86c06bb80 0x7ff84c003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86a000640 1 --2- 172.21.15.27:0/930016591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff84c003f60 0x7ff84c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86a000640 1 --2- 172.21.15.27:0/930016591 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff84c004d10 0x7ff84c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86a000640 1 -- 172.21.15.27:0/930016591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86c0ce270 con 0x7ff84c003f60 2024-01-15T18:15:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86a000640 1 -- 172.21.15.27:0/930016591 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c00ba70 con 0x7ff84c004d10 2024-01-15T18:15:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86a000640 1 -- 172.21.15.27:0/930016591 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c00bc30 con 0x7ff86c06bb80 2024-01-15T18:15:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86aa00640 1 --2- 172.21.15.27:0/930016591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff84c003f60 0x7ff84c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86be00640 1 --2- 172.21.15.27:0/930016591 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff84c004d10 0x7ff84c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86b400640 1 --2- 172.21.15.27:0/930016591 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff86c06bb80 0x7ff84c003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86aa00640 1 -- 172.21.15.27:0/930016591 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff86c06bb80 msgr2=0x7ff84c003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86aa00640 1 --2- 172.21.15.27:0/930016591 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff86c06bb80 0x7ff84c003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86aa00640 1 -- 172.21.15.27:0/930016591 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff84c004d10 msgr2=0x7ff84c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86aa00640 1 --2- 172.21.15.27:0/930016591 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff84c004d10 0x7ff84c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.804+0000 7ff86aa00640 1 -- 172.21.15.27:0/930016591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84c003530 con 0x7ff84c003f60 2024-01-15T18:15:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.808+0000 7ff86aa00640 1 --2- 172.21.15.27:0/930016591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff84c003f60 0x7ff84c0047d0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7ff854009ae0 tx=0x7ff85401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=773b7ee56923e163 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.808+0000 7ff86a000640 1 -- 172.21.15.27:0/930016591 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8540a5640 con 0x7ff84c003f60 2024-01-15T18:15:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.808+0000 7ff86a000640 1 -- 172.21.15.27:0/930016591 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8540a57a0 con 0x7ff84c003f60 2024-01-15T18:15:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.808+0000 7ff86a000640 1 -- 172.21.15.27:0/930016591 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8540a5b50 con 0x7ff84c003f60 2024-01-15T18:15:55.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.808+0000 7ff872400640 1 -- 172.21.15.27:0/930016591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff84c003f60 msgr2=0x7ff84c0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:55.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.808+0000 7ff872400640 1 --2- 172.21.15.27:0/930016591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff84c003f60 0x7ff84c0047d0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7ff854009ae0 tx=0x7ff85401ca50 comp rx=0 tx=0).stop 2024-01-15T18:15:55.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.808+0000 7ff86be00640 1 -- 172.21.15.27:0/930016591 reap_dead start 2024-01-15T18:15:55.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.808+0000 7ff872400640 1 -- 172.21.15.27:0/930016591 shutdown_connections 2024-01-15T18:15:55.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.808+0000 7ff872400640 1 -- 172.21.15.27:0/930016591 >> 172.21.15.27:0/930016591 conn(0x7ff86c0b2a10 msgr2=0x7ff86c0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:55.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.808+0000 7ff872400640 1 -- 172.21.15.27:0/930016591 shutdown_connections 2024-01-15T18:15:55.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.808+0000 7ff872400640 1 -- 172.21.15.27:0/930016591 wait complete. 2024-01-15T18:15:55.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.812+0000 7ff872400640 1 Processor -- start 2024-01-15T18:15:55.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.812+0000 7ff872400640 1 -- start start 2024-01-15T18:15:55.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.812+0000 7ff872400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff84c003f60 0x7ff86c1f24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.812+0000 7ff872400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff86c15fa00 0x7ff86c15d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.812+0000 7ff872400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff86c15d850 0x7ff86c0adcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.812+0000 7ff872400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86c15a610 con 0x7ff86c15fa00 2024-01-15T18:15:55.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.812+0000 7ff872400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86c15fea0 con 0x7ff86c15d850 2024-01-15T18:15:55.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.812+0000 7ff872400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff86c160180 con 0x7ff84c003f60 2024-01-15T18:15:55.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.812+0000 7ff86aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff86c15fa00 0x7ff86c15d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.812+0000 7ff86aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff86c15fa00 0x7ff86c15d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37382/0 (socket says 172.21.15.27:37382) 2024-01-15T18:15:55.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.812+0000 7ff86aa00640 1 -- 172.21.15.27:0/2386401585 learned_addr learned my addr 172.21.15.27:0/2386401585 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:55.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.812+0000 7ff86b400640 1 --2- 172.21.15.27:0/2386401585 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff84c003f60 0x7ff86c1f24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.812+0000 7ff86be00640 1 --2- 172.21.15.27:0/2386401585 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff86c15d850 0x7ff86c0adcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.812+0000 7ff86aa00640 1 -- 172.21.15.27:0/2386401585 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff84c003f60 msgr2=0x7ff86c1f24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:55.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.812+0000 7ff86aa00640 1 --2- 172.21.15.27:0/2386401585 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff84c003f60 0x7ff86c1f24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.812+0000 7ff86aa00640 1 -- 172.21.15.27:0/2386401585 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff86c15d850 msgr2=0x7ff86c0adcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:55.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.812+0000 7ff86aa00640 1 --2- 172.21.15.27:0/2386401585 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff86c15d850 0x7ff86c0adcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:55.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.812+0000 7ff86aa00640 1 -- 172.21.15.27:0/2386401585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86c160460 con 0x7ff86c15fa00 2024-01-15T18:15:55.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.812+0000 7ff86aa00640 1 --2- 172.21.15.27:0/2386401585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff86c15fa00 0x7ff86c15d310 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7ff854009ae0 tx=0x7ff84c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:55.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.812+0000 7ff863400640 1 -- 172.21.15.27:0/2386401585 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff854004230 con 0x7ff86c15fa00 2024-01-15T18:15:55.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.812+0000 7ff863400640 1 -- 172.21.15.27:0/2386401585 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff854004390 con 0x7ff86c15fa00 2024-01-15T18:15:55.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.812+0000 7ff863400640 1 -- 172.21.15.27:0/2386401585 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8540ad850 con 0x7ff86c15fa00 2024-01-15T18:15:55.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.812+0000 7ff872400640 1 -- 172.21.15.27:0/2386401585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff86c15a2a0 con 0x7ff86c15fa00 2024-01-15T18:15:55.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.816+0000 7ff872400640 1 -- 172.21.15.27:0/2386401585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff86c0ae230 con 0x7ff86c15fa00 2024-01-15T18:15:55.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.816+0000 7ff872400640 1 -- 172.21.15.27:0/2386401585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff86c067a10 con 0x7ff86c15fa00 2024-01-15T18:15:55.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.816+0000 7ff863400640 1 -- 172.21.15.27:0/2386401585 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7ff8540b4040 con 0x7ff86c15fa00 2024-01-15T18:15:55.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.816+0000 7ff863400640 1 --2- 172.21.15.27:0/2386401585 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff8440674c0 0x7ff844069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:55.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.816+0000 7ff863400640 1 -- 172.21.15.27:0/2386401585 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7ff85411d6f0 con 0x7ff86c15fa00 2024-01-15T18:15:55.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.816+0000 7ff86b400640 1 --2- 172.21.15.27:0/2386401585 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff8440674c0 0x7ff844069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:55.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.820+0000 7ff86b400640 1 --2- 172.21.15.27:0/2386401585 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff8440674c0 0x7ff844069980 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff85c007fd0 tx=0x7ff85c0013f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:55.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:55.820+0000 7ff863400640 1 -- 172.21.15.27:0/2386401585 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff8540ec230 con 0x7ff86c15fa00 2024-01-15T18:15:56.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:56.012+0000 7ff872400640 1 -- 172.21.15.27:0/2386401585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff86c15a430 con 0x7ff86c15fa00 2024-01-15T18:15:56.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:56.012+0000 7ff863400640 1 -- 172.21.15.27:0/2386401585 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v57) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff8540ef850 con 0x7ff86c15fa00 2024-01-15T18:15:56.010 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:15:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:56.016+0000 7ff872400640 1 -- 172.21.15.27:0/2386401585 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff8440674c0 msgr2=0x7ff844069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:56.016+0000 7ff872400640 1 --2- 172.21.15.27:0/2386401585 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff8440674c0 0x7ff844069980 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff85c007fd0 tx=0x7ff85c0013f0 comp rx=0 tx=0).stop 2024-01-15T18:15:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:56.016+0000 7ff872400640 1 -- 172.21.15.27:0/2386401585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff86c15fa00 msgr2=0x7ff86c15d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:56.016+0000 7ff872400640 1 --2- 172.21.15.27:0/2386401585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff86c15fa00 0x7ff86c15d310 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7ff854009ae0 tx=0x7ff84c005800 comp rx=0 tx=0).stop 2024-01-15T18:15:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:56.016+0000 7ff872400640 1 -- 172.21.15.27:0/2386401585 shutdown_connections 2024-01-15T18:15:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:56.016+0000 7ff872400640 1 --2- 172.21.15.27:0/2386401585 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff8440674c0 0x7ff844069980 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:56.016+0000 7ff872400640 1 --2- 172.21.15.27:0/2386401585 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff86c15d850 0x7ff86c0adcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:56.016+0000 7ff872400640 1 --2- 172.21.15.27:0/2386401585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff86c15fa00 0x7ff86c15d310 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:56.016+0000 7ff872400640 1 --2- 172.21.15.27:0/2386401585 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff84c003f60 0x7ff86c1f24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:56.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:56.016+0000 7ff872400640 1 -- 172.21.15.27:0/2386401585 >> 172.21.15.27:0/2386401585 conn(0x7ff86c0b2a10 msgr2=0x7ff86c0cb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:56.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:56.016+0000 7ff872400640 1 -- 172.21.15.27:0/2386401585 shutdown_connections 2024-01-15T18:15:56.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:56.016+0000 7ff872400640 1 -- 172.21.15.27:0/2386401585 wait complete. 2024-01-15T18:15:56.033 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:15:56.033 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:15:56.033 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-01-15T18:15:56.033 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-15T18:15:56.033 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-15T18:15:56.033 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:15:56.033 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:15:56.033 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:15:56.033 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:15:56.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:56.034 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-01-15T18:15:56.043 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:15:56.043 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:15:56.043 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:15:56.043 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:15:56.043 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-15T18:15:56.043 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:15:56.043 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:15:56.043 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:15:56.043 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-01-15T18:15:56.043 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:15:56.044 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:15:56.044 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-01-15T18:15:56.044 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:15:56.044 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-01-15T18:15:56.044 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-01-15T18:15:56.044 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-01-15T18:15:56.044 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 41803sec preferred_lft 41803sec 2024-01-15T18:15:56.044 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-01-15T18:15:56.044 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:15:56.044 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:15:56.044 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-01-15T18:15:56.044 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-01-15T18:15:56.044 INFO:teuthology.orchestra.run.smithi027.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-01-15T18:15:56.044 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 02:42:65:20:14:72 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:15:56.045 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-01-15T18:15:56.045 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:15:56.045 INFO:teuthology.orchestra.run.smithi027.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:15:56.045 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-01-15T18:15:56.045 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:15:56.045 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:15:56.045 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-01-15T18:15:56.045 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:15:56.045 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:15:56.045 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:15:56.045 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-01-15T18:15:56.045 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:15:56.045 INFO:teuthology.orchestra.run.smithi027.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:15:56.045 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:15:56.045 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-01-15T18:15:56.046 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:15:56.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:56.047 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:15:56.106 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-01-15T18:15:56.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:56.108 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-01-15T18:15:56.165 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-01-15T18:15:56.165 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:15:56.165 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-01-15T18:15:56.165 INFO:teuthology.orchestra.run.smithi027.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-01-15T18:15:56.165 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-01-15T18:15:56.165 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:15:56.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:56.166 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-15T18:15:56.166 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-15T18:15:56.166 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-15T18:15:56.166 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-15T18:15:56.166 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-15T18:15:56.166 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-15T18:15:56.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:56.262 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-15T18:15:56.271 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-15T18:15:56.271 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-15T18:15:56.271 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:15:56.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:56.272 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-01-15T18:15:56.336 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-15T18:15:56.336 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:15:56.336 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-15T18:15:56.388 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:15:56.388 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:15:56.388 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:15:56.439 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-01-15T18:15:56.500 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:15:56.553 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-01-15T18:15:56.553 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-01-15T18:15:56.553 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-01-15T18:15:56.553 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-01-15T18:15:56.553 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-01-15T18:15:56.553 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-01-15T18:15:56.554 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-01-15T18:15:56.554 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-01-15T18:15:56.554 INFO:teuthology.orchestra.run.smithi027.stdout:/run/lock 2024-01-15T18:15:56.554 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-01-15T18:15:56.554 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-01-15T18:15:56.554 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-01-15T18:15:56.554 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:15:56.554 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-01-15T18:15:56.554 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-01-15T18:15:56.554 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-01-15T18:15:56.554 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-01-15T18:15:56.554 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/fs/nfsd 2024-01-15T18:15:56.554 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-01-15T18:15:56.554 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-01-15T18:15:56.554 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:15:56.555 INFO:teuthology.orchestra.run.smithi027.stdout:/run/rpc_pipefs 2024-01-15T18:15:56.555 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-01-15T18:15:56.555 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-01-15T18:15:56.555 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:15:56.555 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:15:56.555 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:15:56.555 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:15:56.555 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-01-15T18:15:56.555 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:15:56.555 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:15:56.555 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:15:56.555 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:15:56.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:56.556 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-01-15T18:15:56.607 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:15:56.608 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-15T18:15:56.651 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-01-15T18:15:56.672 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:15:56.725 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-01-15T18:15:56.725 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-01-15T18:15:56.725 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-01-15T18:15:56.725 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-01-15T18:15:56.726 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-01-15T18:15:56.726 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-01-15T18:15:56.726 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-01-15T18:15:56.726 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-01-15T18:15:56.726 INFO:teuthology.orchestra.run.smithi027.stdout:/run/lock 2024-01-15T18:15:56.726 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-01-15T18:15:56.726 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-01-15T18:15:56.726 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-01-15T18:15:56.726 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:15:56.726 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-01-15T18:15:56.726 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-01-15T18:15:56.726 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-01-15T18:15:56.726 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-01-15T18:15:56.726 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/fs/nfsd 2024-01-15T18:15:56.726 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-01-15T18:15:56.727 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-01-15T18:15:56.727 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:15:56.727 INFO:teuthology.orchestra.run.smithi027.stdout:/run/rpc_pipefs 2024-01-15T18:15:56.727 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-01-15T18:15:56.727 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-01-15T18:15:56.727 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:15:56.727 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:15:56.727 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:15:56.727 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:15:56.727 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-01-15T18:15:56.727 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:15:56.727 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:15:56.727 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:15:56.727 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:15:56.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:56.728 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-01-15T18:15:56.746 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:2024-01-15T18:15:56.748+0000 7f88d0c80580 -1 init, newargv = 0x5646d22e62a0 newargc=15 2024-01-15T18:15:56.746 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[48488]: starting ceph client 2024-01-15T18:15:56.753 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[48488]: starting fuse 2024-01-15T18:15:56.754 INFO:teuthology.orchestra.run.smithi027.stdout:43 2024-01-15T18:15:56.754 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-01-15T18:15:56.755 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:15:56.755 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-01-15T18:15:56.755 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-01-15T18:15:56.755 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-01-15T18:15:56.755 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-01-15T18:15:56.755 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:15:56.755 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-01-15T18:15:56.755 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-15T18:15:56.755 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-01-15T18:15:56.755 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-15T18:15:56.755 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:15:56.755 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-01-15T18:15:56.755 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-01-15T18:15:56.755 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-01-15T18:15:56.755 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-01-15T18:15:56.756 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:15:56.756 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-01-15T18:15:56.756 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:15:56.756 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:15:56.756 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:15:56.756 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-01-15T18:15:56.756 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-01-15T18:15:56.756 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-01-15T18:15:56.756 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:15:56.756 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:15:56.756 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.0")) 2024-01-15T18:15:56.756 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-01-15T18:15:56.878 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.0.48488.asok 2024-01-15T18:15:56.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:15:56.889 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.48488.asok 2024-01-15T18:15:56.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:56.890 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.48488.asok status 2024-01-15T18:15:57.062 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:15:57.062 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-01-15T18:15:57.062 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:15:57.063 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:15:57.063 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "0", 2024-01-15T18:15:57.063 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-01-15T18:15:57.063 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-15T18:15:57.063 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "48488", 2024-01-15T18:15:57.063 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-01-15T18:15:57.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:57.063 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-01-15T18:15:57.063 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-01-15T18:15:57.063 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 8565, 2024-01-15T18:15:57.063 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-01-15T18:15:57.063 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-01-15T18:15:57.063 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-01-15T18:15:57.063 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 8565 2024-01-15T18:15:57.063 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:57.063 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-01-15T18:15:57.064 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-01-15T18:15:57.064 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:15:57.064 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 770677392 2024-01-15T18:15:57.064 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:57.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:57.064 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-01-15T18:15:57.064 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-01-15T18:15:57.064 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:15:57.064 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 770677392 2024-01-15T18:15:57.064 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:15:57.064 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.8565 192.168.0.1:0/770677392", 2024-01-15T18:15:57.064 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.1:0/770677392", 2024-01-15T18:15:57.064 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-01-15T18:15:57.064 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 57, 2024-01-15T18:15:57.065 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 84, 2024-01-15T18:15:57.065 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:15:57.065 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-01-15T18:15:57.065 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-01-15T18:15:57.065 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:15:57.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:57.077 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-01-15T18:15:57.078 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-15T18:15:57.087 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-01-15T18:15:57.087 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-15T18:15:57.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:57.087 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:15:57.148 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:15:57.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.340+0000 7f5971a00640 1 Processor -- start 2024-01-15T18:15:57.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.340+0000 7f5971a00640 1 -- start start 2024-01-15T18:15:57.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.340+0000 7f5971a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f596c159bf0 0x7f596c159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:57.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.340+0000 7f5971a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f596c071e40 0x7f596c0c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:57.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.340+0000 7f5971a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f596c159a50 con 0x7f596c06d430 2024-01-15T18:15:57.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.340+0000 7f5971a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f596c0cce00 con 0x7f596c159bf0 2024-01-15T18:15:57.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.340+0000 7f5971a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f596c15a510 con 0x7f596c071e40 2024-01-15T18:15:57.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.340+0000 7f596aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f596c159bf0 0x7f596c159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:57.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.340+0000 7f596aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f596c159bf0 0x7f596c159fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37388/0 (socket says 172.21.15.27:37388) 2024-01-15T18:15:57.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.340+0000 7f596aa00640 1 -- 172.21.15.27:0/2834267003 learned_addr learned my addr 172.21.15.27:0/2834267003 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:57.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.340+0000 7f596a000640 1 --2- 172.21.15.27:0/2834267003 >> v2:172.21.15.93:3300/0 conn(0x7f596c071e40 0x7f596c0c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:57.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.340+0000 7f596aa00640 1 -- 172.21.15.27:0/2834267003 >> v2:172.21.15.93:3300/0 conn(0x7f596c071e40 msgr2=0x7f596c0c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:57.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.340+0000 7f596aa00640 1 --2- 172.21.15.27:0/2834267003 >> v2:172.21.15.93:3300/0 conn(0x7f596c071e40 0x7f596c0c9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:57.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.340+0000 7f596aa00640 1 -- 172.21.15.27:0/2834267003 >> v1:172.21.15.93:6789/0 conn(0x7f596c06d430 legacy=0x7f596c15d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:57.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.340+0000 7f596aa00640 1 -- 172.21.15.27:0/2834267003 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596c06d980 con 0x7f596c159bf0 2024-01-15T18:15:57.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.344+0000 7f596aa00640 1 --2- 172.21.15.27:0/2834267003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c159bf0 0x7f596c159fd0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f596c0cfb10 tx=0x7f595c0211a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31e69a78edb54876 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:57.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.344+0000 7f5969600640 1 -- 172.21.15.27:0/2834267003 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595c004030 con 0x7f596c159bf0 2024-01-15T18:15:57.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.344+0000 7f5969600640 1 -- 172.21.15.27:0/2834267003 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595c0ad030 con 0x7f596c159bf0 2024-01-15T18:15:57.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.344+0000 7f5969600640 1 -- 172.21.15.27:0/2834267003 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595c004410 con 0x7f596c159bf0 2024-01-15T18:15:57.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.344+0000 7f5971a00640 1 -- 172.21.15.27:0/2834267003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c159bf0 msgr2=0x7f596c159fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:57.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.344+0000 7f5971a00640 1 --2- 172.21.15.27:0/2834267003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c159bf0 0x7f596c159fd0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f596c0cfb10 tx=0x7f595c0211a0 comp rx=0 tx=0).stop 2024-01-15T18:15:57.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.344+0000 7f5971a00640 1 -- 172.21.15.27:0/2834267003 shutdown_connections 2024-01-15T18:15:57.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.344+0000 7f5971a00640 1 --2- 172.21.15.27:0/2834267003 >> v2:172.21.15.93:3300/0 conn(0x7f596c071e40 0x7f596c0c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:57.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.344+0000 7f5971a00640 1 --2- 172.21.15.27:0/2834267003 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c159bf0 0x7f596c159fd0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:57.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.344+0000 7f5971a00640 1 -- 172.21.15.27:0/2834267003 >> 172.21.15.27:0/2834267003 conn(0x7f596c0cec60 msgr2=0x7f596c0cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:57.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.344+0000 7f5971a00640 1 -- 172.21.15.27:0/2834267003 shutdown_connections 2024-01-15T18:15:57.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.344+0000 7f5971a00640 1 -- 172.21.15.27:0/2834267003 wait complete. 2024-01-15T18:15:57.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.348+0000 7f5971a00640 1 Processor -- start 2024-01-15T18:15:57.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.348+0000 7f5971a00640 1 -- start start 2024-01-15T18:15:57.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.348+0000 7f5971a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f596c06d430 0x7f596c14f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:57.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.348+0000 7f5971a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c159bf0 0x7f596c0c2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:57.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.348+0000 7f5971a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f596c0c34e0 0x7f596c1e3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:57.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.348+0000 7f5971a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c14f6a0 con 0x7f596c159bf0 2024-01-15T18:15:57.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.348+0000 7f5971a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596c0c3a60 con 0x7f596c06d430 2024-01-15T18:15:57.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.348+0000 7f5971a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596c0c3d40 con 0x7f596c0c34e0 2024-01-15T18:15:57.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.348+0000 7f596a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c159bf0 0x7f596c0c2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:57.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.348+0000 7f596a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c159bf0 0x7f596c0c2fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37390/0 (socket says 172.21.15.27:37390) 2024-01-15T18:15:57.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.348+0000 7f596a000640 1 -- 172.21.15.27:0/2412159081 learned_addr learned my addr 172.21.15.27:0/2412159081 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:57.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.348+0000 7f596aa00640 1 --2- 172.21.15.27:0/2412159081 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f596c06d430 0x7f596c14f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:57.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.348+0000 7f596b400640 1 --2- 172.21.15.27:0/2412159081 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f596c0c34e0 0x7f596c1e3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:57.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.348+0000 7f596a000640 1 -- 172.21.15.27:0/2412159081 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f596c0c34e0 msgr2=0x7f596c1e3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:57.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.348+0000 7f596a000640 1 --2- 172.21.15.27:0/2412159081 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f596c0c34e0 0x7f596c1e3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:57.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.348+0000 7f596a000640 1 -- 172.21.15.27:0/2412159081 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f596c06d430 msgr2=0x7f596c14f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:57.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.348+0000 7f596a000640 1 --2- 172.21.15.27:0/2412159081 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f596c06d430 0x7f596c14f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:57.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.348+0000 7f596a000640 1 -- 172.21.15.27:0/2412159081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f595c00b780 con 0x7f596c159bf0 2024-01-15T18:15:57.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.348+0000 7f596a000640 1 --2- 172.21.15.27:0/2412159081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c159bf0 0x7f596c0c2fa0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f595400e9b0 tx=0x7f595400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:57.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.348+0000 7f5962a00640 1 -- 172.21.15.27:0/2412159081 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59540096b0 con 0x7f596c159bf0 2024-01-15T18:15:57.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.348+0000 7f5962a00640 1 -- 172.21.15.27:0/2412159081 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595400c9a0 con 0x7f596c159bf0 2024-01-15T18:15:57.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.348+0000 7f5962a00640 1 -- 172.21.15.27:0/2412159081 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595400cd30 con 0x7f596c159bf0 2024-01-15T18:15:57.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.348+0000 7f5971a00640 1 -- 172.21.15.27:0/2412159081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f596c0c4020 con 0x7f596c159bf0 2024-01-15T18:15:57.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.348+0000 7f5971a00640 1 -- 172.21.15.27:0/2412159081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f596c0c6150 con 0x7f596c159bf0 2024-01-15T18:15:57.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.348+0000 7f5971a00640 1 -- 172.21.15.27:0/2412159081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f596c0c6400 con 0x7f596c159bf0 2024-01-15T18:15:57.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.352+0000 7f5962a00640 1 -- 172.21.15.27:0/2412159081 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7f5954013470 con 0x7f596c159bf0 2024-01-15T18:15:57.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.352+0000 7f5962a00640 1 --2- 172.21.15.27:0/2412159081 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f59440674c0 0x7f5944069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:57.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.352+0000 7f5962a00640 1 -- 172.21.15.27:0/2412159081 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f5954083250 con 0x7f596c159bf0 2024-01-15T18:15:57.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.352+0000 7f596aa00640 1 --2- 172.21.15.27:0/2412159081 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f59440674c0 0x7f5944069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:57.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.352+0000 7f596aa00640 1 --2- 172.21.15.27:0/2412159081 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f59440674c0 0x7f5944069980 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f595c0b4000 tx=0x7f595c0ac000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:57.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.356+0000 7f5962a00640 1 -- 172.21.15.27:0/2412159081 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f596c0c6400 con 0x7f596c159bf0 2024-01-15T18:15:57.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.548+0000 7f5971a00640 1 -- 172.21.15.27:0/2412159081 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f596c0c6400 con 0x7f596c159bf0 2024-01-15T18:15:57.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.548+0000 7f5962a00640 1 -- 172.21.15.27:0/2412159081 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v57) v1 ==== 53+0+73 (secure 0 0 0) 0x7f596c0c6400 con 0x7f596c159bf0 2024-01-15T18:15:57.545 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:15:57.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.548+0000 7f5971a00640 1 -- 172.21.15.27:0/2412159081 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f59440674c0 msgr2=0x7f5944069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:57.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.548+0000 7f5971a00640 1 --2- 172.21.15.27:0/2412159081 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f59440674c0 0x7f5944069980 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f595c0b4000 tx=0x7f595c0ac000 comp rx=0 tx=0).stop 2024-01-15T18:15:57.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.548+0000 7f5971a00640 1 -- 172.21.15.27:0/2412159081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c159bf0 msgr2=0x7f596c0c2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:57.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.548+0000 7f5971a00640 1 --2- 172.21.15.27:0/2412159081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c159bf0 0x7f596c0c2fa0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f595400e9b0 tx=0x7f595400ee70 comp rx=0 tx=0).stop 2024-01-15T18:15:57.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.548+0000 7f5971a00640 1 -- 172.21.15.27:0/2412159081 shutdown_connections 2024-01-15T18:15:57.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.548+0000 7f5971a00640 1 --2- 172.21.15.27:0/2412159081 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f59440674c0 0x7f5944069980 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:57.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.548+0000 7f5971a00640 1 --2- 172.21.15.27:0/2412159081 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f596c0c34e0 0x7f596c1e3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:57.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.548+0000 7f5971a00640 1 --2- 172.21.15.27:0/2412159081 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f596c159bf0 0x7f596c0c2fa0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:57.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.548+0000 7f5971a00640 1 --2- 172.21.15.27:0/2412159081 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f596c06d430 0x7f596c14f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:57.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.548+0000 7f5971a00640 1 -- 172.21.15.27:0/2412159081 >> 172.21.15.27:0/2412159081 conn(0x7f596c0cec60 msgr2=0x7f596c0c9290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:57.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.548+0000 7f5971a00640 1 -- 172.21.15.27:0/2412159081 shutdown_connections 2024-01-15T18:15:57.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:57.548+0000 7f5971a00640 1 -- 172.21.15.27:0/2412159081 wait complete. 2024-01-15T18:15:57.552 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:15:57.555 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:15:57.567 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:15:57.567 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:15:57.567 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi093.front.sepia.ceph.com 2024-01-15T18:15:57.567 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-15T18:15:57.567 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-15T18:15:57.567 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:15:57.567 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:15:57.567 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:15:57.567 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:15:57.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:57.568 DEBUG:teuthology.orchestra.run.smithi093:> ip addr 2024-01-15T18:15:57.612 INFO:teuthology.orchestra.run.smithi093.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:15:57.612 INFO:teuthology.orchestra.run.smithi093.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:15:57.612 INFO:teuthology.orchestra.run.smithi093.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:15:57.612 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:15:57.613 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 ::1/128 scope host 2024-01-15T18:15:57.613 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:15:57.613 INFO:teuthology.orchestra.run.smithi093.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:15:57.613 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:d8:da:1c brd ff:ff:ff:ff:ff:ff 2024-01-15T18:15:57.613 INFO:teuthology.orchestra.run.smithi093.stdout: altname enp7s0f0 2024-01-15T18:15:57.613 INFO:teuthology.orchestra.run.smithi093.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:15:57.613 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:d8:da:1d brd ff:ff:ff:ff:ff:ff 2024-01-15T18:15:57.613 INFO:teuthology.orchestra.run.smithi093.stdout: altname enp7s0f1 2024-01-15T18:15:57.613 INFO:teuthology.orchestra.run.smithi093.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:15:57.613 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:8f:d6:34 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:15:57.613 INFO:teuthology.orchestra.run.smithi093.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:15:57.613 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:8f:d6:35 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:15:57.613 INFO:teuthology.orchestra.run.smithi093.stdout: inet 172.21.15.93/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-01-15T18:15:57.613 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft 41781sec preferred_lft 41781sec 2024-01-15T18:15:57.614 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::ec4:7aff:fe8f:d635/64 scope link 2024-01-15T18:15:57.614 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:15:57.614 INFO:teuthology.orchestra.run.smithi093.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-01-15T18:15:57.614 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 02:42:2a:61:fc:7d brd ff:ff:ff:ff:ff:ff 2024-01-15T18:15:57.614 INFO:teuthology.orchestra.run.smithi093.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-01-15T18:15:57.614 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:15:57.614 INFO:teuthology.orchestra.run.smithi093.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:15:57.615 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-01-15T18:15:57.615 INFO:teuthology.orchestra.run.smithi093.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:15:57.615 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:15:57.615 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-01-15T18:15:57.615 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:15:57.615 INFO:teuthology.orchestra.run.smithi093.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:15:57.615 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:15:57.615 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-01-15T18:15:57.615 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:15:57.615 INFO:teuthology.orchestra.run.smithi093.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:15:57.615 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:15:57.615 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-01-15T18:15:57.615 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:15:57.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:57.616 DEBUG:teuthology.orchestra.run.smithi093:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:15:57.679 INFO:teuthology.orchestra.run.smithi093.stdout:1 2024-01-15T18:15:57.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:57.682 DEBUG:teuthology.orchestra.run.smithi093:> route 2024-01-15T18:15:57.735 INFO:teuthology.orchestra.run.smithi093.stdout:Kernel IP routing table 2024-01-15T18:15:57.735 INFO:teuthology.orchestra.run.smithi093.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:15:57.735 INFO:teuthology.orchestra.run.smithi093.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-01-15T18:15:57.735 INFO:teuthology.orchestra.run.smithi093.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T18:15:57.736 INFO:teuthology.orchestra.run.smithi093.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-01-15T18:15:57.736 INFO:teuthology.orchestra.run.smithi093.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T18:15:57.736 INFO:teuthology.orchestra.run.smithi093.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:15:57.736 INFO:teuthology.orchestra.run.smithi093.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:15:57.736 INFO:teuthology.orchestra.run.smithi093.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:15:57.736 INFO:teuthology.orchestra.run.smithi093.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:15:57.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:57.736 DEBUG:teuthology.orchestra.run.smithi093:> (cd / && exec bash -c ' 2024-01-15T18:15:57.736 DEBUG:teuthology.orchestra.run.smithi093:> set -e 2024-01-15T18:15:57.736 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T18:15:57.737 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T18:15:57.737 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T18:15:57.737 DEBUG:teuthology.orchestra.run.smithi093:> ') 2024-01-15T18:15:57.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:57.829 DEBUG:teuthology.orchestra.run.smithi093:> ip netns list 2024-01-15T18:15:57.838 INFO:teuthology.orchestra.run.smithi093.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-15T18:15:57.838 INFO:teuthology.orchestra.run.smithi093.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-15T18:15:57.838 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:15:57.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:57.839 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip link set brx.0 up 2024-01-15T18:15:57.900 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-15T18:15:57.900 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:15:57.900 DEBUG:teuthology.orchestra.run.smithi093:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-15T18:15:57.948 INFO:teuthology.orchestra.run.smithi093.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:15:57.948 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:15:57.948 DEBUG:teuthology.orchestra.run.smithi093:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:15:57.995 DEBUG:teuthology.orchestra.run.smithi093:> sudo modprobe fuse 2024-01-15T18:15:58.056 DEBUG:teuthology.orchestra.run.smithi093:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:15:58.110 INFO:teuthology.orchestra.run.smithi093.stdout:/sys 2024-01-15T18:15:58.110 INFO:teuthology.orchestra.run.smithi093.stdout:/proc 2024-01-15T18:15:58.111 INFO:teuthology.orchestra.run.smithi093.stdout:/dev 2024-01-15T18:15:58.111 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/pts 2024-01-15T18:15:58.111 INFO:teuthology.orchestra.run.smithi093.stdout:/run 2024-01-15T18:15:58.111 INFO:teuthology.orchestra.run.smithi093.stdout:/ 2024-01-15T18:15:58.111 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/security 2024-01-15T18:15:58.111 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/shm 2024-01-15T18:15:58.111 INFO:teuthology.orchestra.run.smithi093.stdout:/run/lock 2024-01-15T18:15:58.111 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/cgroup 2024-01-15T18:15:58.111 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/pstore 2024-01-15T18:15:58.111 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/bpf 2024-01-15T18:15:58.111 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:15:58.111 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/hugepages 2024-01-15T18:15:58.111 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/mqueue 2024-01-15T18:15:58.111 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/debug 2024-01-15T18:15:58.112 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/tracing 2024-01-15T18:15:58.112 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/fs/nfsd 2024-01-15T18:15:58.112 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/fuse/connections 2024-01-15T18:15:58.112 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/config 2024-01-15T18:15:58.112 INFO:teuthology.orchestra.run.smithi093.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:15:58.112 INFO:teuthology.orchestra.run.smithi093.stdout:/run/rpc_pipefs 2024-01-15T18:15:58.112 INFO:teuthology.orchestra.run.smithi093.stdout:/run/user/1000 2024-01-15T18:15:58.112 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph 2024-01-15T18:15:58.112 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:15:58.112 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:15:58.112 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:15:58.112 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:15:58.112 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns 2024-01-15T18:15:58.112 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:15:58.113 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:15:58.113 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:15:58.113 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:15:58.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:58.114 DEBUG:teuthology.orchestra.run.smithi093:> ls /sys/fs/fuse/connections 2024-01-15T18:15:58.164 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:15:58.164 DEBUG:teuthology.orchestra.run.smithi093:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-15T18:15:58.207 DEBUG:teuthology.orchestra.run.smithi093:> sudo modprobe fuse 2024-01-15T18:15:58.228 DEBUG:teuthology.orchestra.run.smithi093:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:15:58.278 INFO:teuthology.orchestra.run.smithi093.stdout:/sys 2024-01-15T18:15:58.278 INFO:teuthology.orchestra.run.smithi093.stdout:/proc 2024-01-15T18:15:58.278 INFO:teuthology.orchestra.run.smithi093.stdout:/dev 2024-01-15T18:15:58.278 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/pts 2024-01-15T18:15:58.278 INFO:teuthology.orchestra.run.smithi093.stdout:/run 2024-01-15T18:15:58.278 INFO:teuthology.orchestra.run.smithi093.stdout:/ 2024-01-15T18:15:58.278 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/security 2024-01-15T18:15:58.278 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/shm 2024-01-15T18:15:58.278 INFO:teuthology.orchestra.run.smithi093.stdout:/run/lock 2024-01-15T18:15:58.278 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/cgroup 2024-01-15T18:15:58.278 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/pstore 2024-01-15T18:15:58.278 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/bpf 2024-01-15T18:15:58.279 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:15:58.279 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/hugepages 2024-01-15T18:15:58.279 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/mqueue 2024-01-15T18:15:58.279 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/debug 2024-01-15T18:15:58.279 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/tracing 2024-01-15T18:15:58.279 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/fs/nfsd 2024-01-15T18:15:58.279 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/fuse/connections 2024-01-15T18:15:58.279 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/config 2024-01-15T18:15:58.279 INFO:teuthology.orchestra.run.smithi093.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:15:58.279 INFO:teuthology.orchestra.run.smithi093.stdout:/run/rpc_pipefs 2024-01-15T18:15:58.279 INFO:teuthology.orchestra.run.smithi093.stdout:/run/user/1000 2024-01-15T18:15:58.279 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph 2024-01-15T18:15:58.279 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:15:58.279 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:15:58.280 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:15:58.280 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:15:58.280 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns 2024-01-15T18:15:58.280 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:15:58.280 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:15:58.280 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:15:58.280 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:15:58.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:58.281 DEBUG:teuthology.orchestra.run.smithi093:> ls /sys/fs/fuse/connections 2024-01-15T18:15:58.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:2024-01-15T18:15:58.301+0000 7fe87cb4c580 -1 init, newargv = 0x56389f674590 newargc=15 2024-01-15T18:15:58.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:ceph-fuse[34346]: starting ceph client 2024-01-15T18:15:58.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:ceph-fuse[34346]: starting fuse 2024-01-15T18:15:58.310 INFO:teuthology.orchestra.run.smithi093.stdout:43 2024-01-15T18:15:58.310 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-01-15T18:15:58.310 DEBUG:teuthology.orchestra.run.smithi093:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:15:58.310 DEBUG:teuthology.orchestra.run.smithi093:> import glob 2024-01-15T18:15:58.310 DEBUG:teuthology.orchestra.run.smithi093:> import re 2024-01-15T18:15:58.310 DEBUG:teuthology.orchestra.run.smithi093:> import os 2024-01-15T18:15:58.310 DEBUG:teuthology.orchestra.run.smithi093:> import subprocess 2024-01-15T18:15:58.310 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:15:58.311 DEBUG:teuthology.orchestra.run.smithi093:> def _find_admin_socket(client_name): 2024-01-15T18:15:58.311 DEBUG:teuthology.orchestra.run.smithi093:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-15T18:15:58.311 DEBUG:teuthology.orchestra.run.smithi093:> files = glob.glob(asok_path) 2024-01-15T18:15:58.311 DEBUG:teuthology.orchestra.run.smithi093:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-15T18:15:58.311 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:15:58.311 DEBUG:teuthology.orchestra.run.smithi093:> # Given a non-glob path, it better be there 2024-01-15T18:15:58.311 DEBUG:teuthology.orchestra.run.smithi093:> if "*" not in asok_path: 2024-01-15T18:15:58.311 DEBUG:teuthology.orchestra.run.smithi093:> assert(len(files) == 1) 2024-01-15T18:15:58.311 DEBUG:teuthology.orchestra.run.smithi093:> return files[0] 2024-01-15T18:15:58.311 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:15:58.311 DEBUG:teuthology.orchestra.run.smithi093:> for f in files: 2024-01-15T18:15:58.311 DEBUG:teuthology.orchestra.run.smithi093:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:15:58.311 DEBUG:teuthology.orchestra.run.smithi093:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:15:58.311 DEBUG:teuthology.orchestra.run.smithi093:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:15:58.311 DEBUG:teuthology.orchestra.run.smithi093:> contents = proc_f.read() 2024-01-15T18:15:58.312 DEBUG:teuthology.orchestra.run.smithi093:> if mountpoint in contents: 2024-01-15T18:15:58.312 DEBUG:teuthology.orchestra.run.smithi093:> return f 2024-01-15T18:15:58.312 DEBUG:teuthology.orchestra.run.smithi093:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:15:58.312 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:15:58.312 DEBUG:teuthology.orchestra.run.smithi093:> print(_find_admin_socket("client.1")) 2024-01-15T18:15:58.312 DEBUG:teuthology.orchestra.run.smithi093:> ' 2024-01-15T18:15:58.432 INFO:teuthology.orchestra.run.smithi093.stdout:/var/run/ceph/ceph-client.1.34346.asok 2024-01-15T18:15:58.436 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:15:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:15:58.443 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.34346.asok 2024-01-15T18:15:58.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:58.444 DEBUG:teuthology.orchestra.run.smithi093:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.34346.asok status 2024-01-15T18:15:58.610 INFO:teuthology.orchestra.run.smithi093.stdout:{ 2024-01-15T18:15:58.610 INFO:teuthology.orchestra.run.smithi093.stdout: "metadata": { 2024-01-15T18:15:58.610 INFO:teuthology.orchestra.run.smithi093.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:15:58.610 INFO:teuthology.orchestra.run.smithi093.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:15:58.610 INFO:teuthology.orchestra.run.smithi093.stdout: "entity_id": "1", 2024-01-15T18:15:58.610 INFO:teuthology.orchestra.run.smithi093.stdout: "hostname": "smithi093", 2024-01-15T18:15:58.610 INFO:teuthology.orchestra.run.smithi093.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-15T18:15:58.610 INFO:teuthology.orchestra.run.smithi093.stdout: "pid": "34346", 2024-01-15T18:15:58.610 INFO:teuthology.orchestra.run.smithi093.stdout: "root": "/" 2024-01-15T18:15:58.610 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:15:58.610 INFO:teuthology.orchestra.run.smithi093.stdout: "dentry_count": 0, 2024-01-15T18:15:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "dentry_pinned_count": 0, 2024-01-15T18:15:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "id": 7972, 2024-01-15T18:15:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "inst": { 2024-01-15T18:15:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "name": { 2024-01-15T18:15:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "client", 2024-01-15T18:15:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "num": 7972 2024-01-15T18:15:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:15:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": { 2024-01-15T18:15:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "v1", 2024-01-15T18:15:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:15:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "nonce": 4232082883 2024-01-15T18:15:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: } 2024-01-15T18:15:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:15:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": { 2024-01-15T18:15:58.611 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "v1", 2024-01-15T18:15:58.612 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:15:58.612 INFO:teuthology.orchestra.run.smithi093.stdout: "nonce": 4232082883 2024-01-15T18:15:58.612 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:15:58.612 INFO:teuthology.orchestra.run.smithi093.stdout: "inst_str": "client.7972 192.168.0.1:0/4232082883", 2024-01-15T18:15:58.612 INFO:teuthology.orchestra.run.smithi093.stdout: "addr_str": "192.168.0.1:0/4232082883", 2024-01-15T18:15:58.612 INFO:teuthology.orchestra.run.smithi093.stdout: "inode_count": 1, 2024-01-15T18:15:58.612 INFO:teuthology.orchestra.run.smithi093.stdout: "mds_epoch": 57, 2024-01-15T18:15:58.612 INFO:teuthology.orchestra.run.smithi093.stdout: "osd_epoch": 84, 2024-01-15T18:15:58.612 INFO:teuthology.orchestra.run.smithi093.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:15:58.612 INFO:teuthology.orchestra.run.smithi093.stdout: "blocklisted": false, 2024-01-15T18:15:58.612 INFO:teuthology.orchestra.run.smithi093.stdout: "fs_name": "cephfs" 2024-01-15T18:15:58.612 INFO:teuthology.orchestra.run.smithi093.stdout:} 2024-01-15T18:15:58.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:58.624 DEBUG:teuthology.orchestra.run.smithi093:> stat --file-system '--printf=%T 2024-01-15T18:15:58.625 DEBUG:teuthology.orchestra.run.smithi093:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-15T18:15:58.633 INFO:teuthology.orchestra.run.smithi093.stdout:fuseblk 2024-01-15T18:15:58.633 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-15T18:15:58.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:58.634 DEBUG:teuthology.orchestra.run.smithi093:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:15:58.695 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-15T18:15:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.852+0000 7f88d0000640 1 Processor -- start 2024-01-15T18:15:58.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.856+0000 7f88d0000640 1 -- start start 2024-01-15T18:15:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.856+0000 7f88d0000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88c815d190 0x7f88c80cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.856+0000 7f88d0000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f88c806c520 0x7f88c80cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.856+0000 7f88d0000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c80b0d70 con 0x7f88c815d570 2024-01-15T18:15:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.856+0000 7f88d0000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c80d0540 con 0x7f88c806c520 2024-01-15T18:15:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.856+0000 7f88d0000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c80b0440 con 0x7f88c815d190 2024-01-15T18:15:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.856+0000 7f88ccc00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f88c815d570 0x7f88c806be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44376/0 (socket says 172.21.15.27:44376) 2024-01-15T18:15:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.856+0000 7f88ccc00640 1 -- 172.21.15.27:0/3351484904 learned_addr learned my addr 172.21.15.27:0/3351484904 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.856+0000 7f88cd600640 1 --2- 172.21.15.27:0/3351484904 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88c815d190 0x7f88c80cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:58.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.856+0000 7f88ce000640 1 --2- 172.21.15.27:0/3351484904 >> v2:172.21.15.93:3300/0 conn(0x7f88c806c520 0x7f88c80cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:58.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.856+0000 7f88c7e00640 1 -- 172.21.15.27:0/3351484904 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660734221 0 0) 0x7f88c80b0d70 con 0x7f88c815d570 2024-01-15T18:15:58.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.856+0000 7f88c7e00640 1 -- 172.21.15.27:0/3351484904 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88a8003530 con 0x7f88c815d570 2024-01-15T18:15:58.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.856+0000 7f88cd600640 1 -- 172.21.15.27:0/3351484904 >> v2:172.21.15.93:3300/0 conn(0x7f88c806c520 msgr2=0x7f88c80cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:58.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.856+0000 7f88cd600640 1 --2- 172.21.15.27:0/3351484904 >> v2:172.21.15.93:3300/0 conn(0x7f88c806c520 0x7f88c80cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:58.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.856+0000 7f88cd600640 1 -- 172.21.15.27:0/3351484904 >> v1:172.21.15.27:6789/0 conn(0x7f88c815d570 legacy=0x7f88c806be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:58.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.856+0000 7f88cd600640 1 -- 172.21.15.27:0/3351484904 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c80b0720 con 0x7f88c815d190 2024-01-15T18:15:58.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.856+0000 7f88ce000640 1 --2- 172.21.15.27:0/3351484904 >> v2:172.21.15.93:3300/0 conn(0x7f88c806c520 0x7f88c80cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:15:58.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.860+0000 7f88cd600640 1 --2- 172.21.15.27:0/3351484904 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88c815d190 0x7f88c80cfcf0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f88b80097c0 tx=0x7f88b801c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d3d5893580f74c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:58.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.860+0000 7f88c7e00640 1 -- 172.21.15.27:0/3351484904 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b80ab610 con 0x7f88c815d190 2024-01-15T18:15:58.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.860+0000 7f88c7e00640 1 -- 172.21.15.27:0/3351484904 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b80ab770 con 0x7f88c815d190 2024-01-15T18:15:58.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.860+0000 7f88c7e00640 1 -- 172.21.15.27:0/3351484904 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b80abb00 con 0x7f88c815d190 2024-01-15T18:15:58.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.860+0000 7f88d0000640 1 -- 172.21.15.27:0/3351484904 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88c815d190 msgr2=0x7f88c80cfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:58.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.860+0000 7f88d0000640 1 --2- 172.21.15.27:0/3351484904 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88c815d190 0x7f88c80cfcf0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f88b80097c0 tx=0x7f88b801c450 comp rx=0 tx=0).stop 2024-01-15T18:15:58.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.860+0000 7f88d0000640 1 -- 172.21.15.27:0/3351484904 shutdown_connections 2024-01-15T18:15:58.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.860+0000 7f88d0000640 1 --2- 172.21.15.27:0/3351484904 >> v2:172.21.15.93:3300/0 conn(0x7f88c806c520 0x7f88c80cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:58.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.860+0000 7f88d0000640 1 --2- 172.21.15.27:0/3351484904 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88c815d190 0x7f88c80cfcf0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:58.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.860+0000 7f88d0000640 1 -- 172.21.15.27:0/3351484904 >> 172.21.15.27:0/3351484904 conn(0x7f88c80d0e70 msgr2=0x7f88c80d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:58.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.860+0000 7f88d0000640 1 -- 172.21.15.27:0/3351484904 shutdown_connections 2024-01-15T18:15:58.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.860+0000 7f88d0000640 1 -- 172.21.15.27:0/3351484904 wait complete. 2024-01-15T18:15:58.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.864+0000 7f88d0000640 1 Processor -- start 2024-01-15T18:15:58.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.864+0000 7f88d0000640 1 -- start start 2024-01-15T18:15:58.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.864+0000 7f88d0000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88c806c520 0x7f88c81529f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:58.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.864+0000 7f88d0000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88c815d190 0x7f88c80b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:58.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.864+0000 7f88d0000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88c815d570 0x7f88c80c6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:58.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.864+0000 7f88d0000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c8152f30 con 0x7f88c806c520 2024-01-15T18:15:58.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.864+0000 7f88d0000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c80c6b10 con 0x7f88c815d570 2024-01-15T18:15:58.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.864+0000 7f88d0000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c80c5f00 con 0x7f88c815d190 2024-01-15T18:15:58.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.864+0000 7f88cd600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88c806c520 0x7f88c81529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:58.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.864+0000 7f88cd600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88c806c520 0x7f88c81529f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37392/0 (socket says 172.21.15.27:37392) 2024-01-15T18:15:58.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.864+0000 7f88cd600640 1 -- 172.21.15.27:0/1058255209 learned_addr learned my addr 172.21.15.27:0/1058255209 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:58.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.864+0000 7f88ccc00640 1 --2- 172.21.15.27:0/1058255209 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88c815d190 0x7f88c80b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:58.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.864+0000 7f88ce000640 1 --2- 172.21.15.27:0/1058255209 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88c815d570 0x7f88c80c6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:58.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.864+0000 7f88cd600640 1 -- 172.21.15.27:0/1058255209 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88c815d190 msgr2=0x7f88c80b4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:58.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.864+0000 7f88cd600640 1 --2- 172.21.15.27:0/1058255209 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88c815d190 0x7f88c80b4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:58.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.864+0000 7f88cd600640 1 -- 172.21.15.27:0/1058255209 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88c815d570 msgr2=0x7f88c80c6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:58.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.864+0000 7f88cd600640 1 --2- 172.21.15.27:0/1058255209 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88c815d570 0x7f88c80c6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:58.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.864+0000 7f88cd600640 1 -- 172.21.15.27:0/1058255209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c0009a70 con 0x7f88c806c520 2024-01-15T18:15:58.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.864+0000 7f88cd600640 1 --2- 172.21.15.27:0/1058255209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88c806c520 0x7f88c81529f0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f88b80ad040 tx=0x7f88b801ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.864+0000 7f88c5600640 1 -- 172.21.15.27:0/1058255209 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b80bf610 con 0x7f88c806c520 2024-01-15T18:15:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.864+0000 7f88c5600640 1 -- 172.21.15.27:0/1058255209 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b80bf7b0 con 0x7f88c806c520 2024-01-15T18:15:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.864+0000 7f88c5600640 1 -- 172.21.15.27:0/1058255209 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b80bfb60 con 0x7f88c806c520 2024-01-15T18:15:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.864+0000 7f88d0000640 1 -- 172.21.15.27:0/1058255209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b8009420 con 0x7f88c806c520 2024-01-15T18:15:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.864+0000 7f88d0000640 1 -- 172.21.15.27:0/1058255209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c80c64c0 con 0x7f88c806c520 2024-01-15T18:15:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.864+0000 7f88d0000640 1 -- 172.21.15.27:0/1058255209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c804e850 con 0x7f88c806c520 2024-01-15T18:15:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.868+0000 7f88c5600640 1 -- 172.21.15.27:0/1058255209 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7f88b80b5400 con 0x7f88c806c520 2024-01-15T18:15:58.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.868+0000 7f88c5600640 1 --2- 172.21.15.27:0/1058255209 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88a00674c0 0x7f88a0069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:58.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.868+0000 7f88c5600640 1 -- 172.21.15.27:0/1058255209 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f88b811d700 con 0x7f88c806c520 2024-01-15T18:15:58.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.868+0000 7f88ccc00640 1 --2- 172.21.15.27:0/1058255209 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88a00674c0 0x7f88a0069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:58.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.868+0000 7f88ccc00640 1 --2- 172.21.15.27:0/1058255209 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88a00674c0 0x7f88a0069980 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f88c81580b0 tx=0x7f88b0007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:58.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:58.872+0000 7f88c5600640 1 -- 172.21.15.27:0/1058255209 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f88b80ec4e0 con 0x7f88c806c520 2024-01-15T18:15:59.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.064+0000 7f88d0000640 1 -- 172.21.15.27:0/1058255209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f88c80ce420 con 0x7f88c806c520 2024-01-15T18:15:59.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.120+0000 7f88c5600640 1 -- 172.21.15.27:0/1058255209 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v58) v1 ==== 136+0+0 (secure 0 0 0) 0x7f88b80efb00 con 0x7f88c806c520 2024-01-15T18:15:59.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.120+0000 7f88d0000640 1 -- 172.21.15.27:0/1058255209 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88a00674c0 msgr2=0x7f88a0069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:59.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.120+0000 7f88d0000640 1 --2- 172.21.15.27:0/1058255209 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88a00674c0 0x7f88a0069980 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f88c81580b0 tx=0x7f88b0007d40 comp rx=0 tx=0).stop 2024-01-15T18:15:59.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.120+0000 7f88d0000640 1 -- 172.21.15.27:0/1058255209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88c806c520 msgr2=0x7f88c81529f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:59.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.120+0000 7f88d0000640 1 --2- 172.21.15.27:0/1058255209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88c806c520 0x7f88c81529f0 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f88b80ad040 tx=0x7f88b801ac90 comp rx=0 tx=0).stop 2024-01-15T18:15:59.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.120+0000 7f88d0000640 1 -- 172.21.15.27:0/1058255209 shutdown_connections 2024-01-15T18:15:59.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.120+0000 7f88d0000640 1 --2- 172.21.15.27:0/1058255209 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88a00674c0 0x7f88a0069980 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.120+0000 7f88d0000640 1 --2- 172.21.15.27:0/1058255209 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88c815d570 0x7f88c80c6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.120+0000 7f88d0000640 1 --2- 172.21.15.27:0/1058255209 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88c815d190 0x7f88c80b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.120+0000 7f88d0000640 1 --2- 172.21.15.27:0/1058255209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88c806c520 0x7f88c81529f0 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.120+0000 7f88d0000640 1 -- 172.21.15.27:0/1058255209 >> 172.21.15.27:0/1058255209 conn(0x7f88c80d0e70 msgr2=0x7f88c80aed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:59.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.124+0000 7f88d0000640 1 -- 172.21.15.27:0/1058255209 shutdown_connections 2024-01-15T18:15:59.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.124+0000 7f88d0000640 1 -- 172.21.15.27:0/1058255209 wait complete. 2024-01-15T18:15:59.140 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:15:59.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.284+0000 7fb9a3800640 1 Processor -- start 2024-01-15T18:15:59.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.284+0000 7fb9a3800640 1 -- start start 2024-01-15T18:15:59.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.284+0000 7fb9a3800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb99c15e7d0 0x7fb99c099a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:59.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.284+0000 7fb9a3800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb99c0afe10 0x7fb99c099f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:59.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.284+0000 7fb9a3800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb99c0cd000 con 0x7fb99c0b35a0 2024-01-15T18:15:59.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.284+0000 7fb9a3800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb99c0cd870 con 0x7fb99c0afe10 2024-01-15T18:15:59.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.284+0000 7fb9a3800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c09a700 con 0x7fb99c15e7d0 2024-01-15T18:15:59.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.284+0000 7fb9a1800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb99c0afe10 0x7fb99c099f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:59.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.284+0000 7fb99be00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb99c0b35a0 0x7fb99c15efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44392/0 (socket says 172.21.15.27:44392) 2024-01-15T18:15:59.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.284+0000 7fb99be00640 1 -- 172.21.15.27:0/2680150789 learned_addr learned my addr 172.21.15.27:0/2680150789 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:59.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.284+0000 7fb9a0e00640 1 --2- 172.21.15.27:0/2680150789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb99c15e7d0 0x7fb99c099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:59.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.284+0000 7fb9a1800640 1 -- 172.21.15.27:0/2680150789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb99c15e7d0 msgr2=0x7fb99c099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:59.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.284+0000 7fb9a1800640 1 --2- 172.21.15.27:0/2680150789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb99c15e7d0 0x7fb99c099a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.284+0000 7fb9a1800640 1 -- 172.21.15.27:0/2680150789 >> v1:172.21.15.27:6789/0 conn(0x7fb99c0b35a0 legacy=0x7fb99c15efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:59.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.284+0000 7fb9a1800640 1 -- 172.21.15.27:0/2680150789 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb99c0cd590 con 0x7fb99c0afe10 2024-01-15T18:15:59.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.284+0000 7fb99b400640 1 -- 172.21.15.27:0/2680150789 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654791648 0 0) 0x7fb99c0cd000 con 0x7fb99c0b35a0 2024-01-15T18:15:59.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.288+0000 7fb9a1800640 1 --2- 172.21.15.27:0/2680150789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb99c0afe10 0x7fb99c099f50 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fb994009d90 tx=0x7fb99401c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1c28bb46fc09da server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:59.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.288+0000 7fb99b400640 1 -- 172.21.15.27:0/2680150789 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9940a5920 con 0x7fb99c0afe10 2024-01-15T18:15:59.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.288+0000 7fb99b400640 1 -- 172.21.15.27:0/2680150789 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9940a5a80 con 0x7fb99c0afe10 2024-01-15T18:15:59.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.288+0000 7fb99b400640 1 -- 172.21.15.27:0/2680150789 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9940a5e10 con 0x7fb99c0afe10 2024-01-15T18:15:59.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.288+0000 7fb9a3800640 1 -- 172.21.15.27:0/2680150789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb99c0afe10 msgr2=0x7fb99c099f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:59.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.288+0000 7fb9a3800640 1 --2- 172.21.15.27:0/2680150789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb99c0afe10 0x7fb99c099f50 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fb994009d90 tx=0x7fb99401c920 comp rx=0 tx=0).stop 2024-01-15T18:15:59.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.288+0000 7fb9a3800640 1 -- 172.21.15.27:0/2680150789 shutdown_connections 2024-01-15T18:15:59.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.288+0000 7fb9a3800640 1 --2- 172.21.15.27:0/2680150789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb99c0afe10 0x7fb99c099f50 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.288+0000 7fb9a3800640 1 --2- 172.21.15.27:0/2680150789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb99c15e7d0 0x7fb99c099a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.288+0000 7fb9a3800640 1 -- 172.21.15.27:0/2680150789 >> 172.21.15.27:0/2680150789 conn(0x7fb99c0b2a80 msgr2=0x7fb99c0b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:59.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.288+0000 7fb9a3800640 1 -- 172.21.15.27:0/2680150789 shutdown_connections 2024-01-15T18:15:59.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.288+0000 7fb9a3800640 1 -- 172.21.15.27:0/2680150789 wait complete. 2024-01-15T18:15:59.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.292+0000 7fb9a3800640 1 Processor -- start 2024-01-15T18:15:59.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.292+0000 7fb9a3800640 1 -- start start 2024-01-15T18:15:59.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.292+0000 7fb9a3800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb99c0afe10 0x7fb99c158e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:59.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.292+0000 7fb9a3800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb99c0b35a0 0x7fb99c1f5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:59.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.292+0000 7fb9a3800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb99c15e7d0 0x7fb99c1f5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:59.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.292+0000 7fb9a3800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c15cbe0 con 0x7fb99c0afe10 2024-01-15T18:15:59.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.292+0000 7fb9a3800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c159490 con 0x7fb99c0b35a0 2024-01-15T18:15:59.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.292+0000 7fb9a3800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c159770 con 0x7fb99c15e7d0 2024-01-15T18:15:59.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.292+0000 7fb9a0e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb99c0afe10 0x7fb99c158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:59.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.292+0000 7fb9a0e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb99c0afe10 0x7fb99c158e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37416/0 (socket says 172.21.15.27:37416) 2024-01-15T18:15:59.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.292+0000 7fb9a0e00640 1 -- 172.21.15.27:0/2673261380 learned_addr learned my addr 172.21.15.27:0/2673261380 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:59.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.292+0000 7fb99be00640 1 --2- 172.21.15.27:0/2673261380 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb99c0b35a0 0x7fb99c1f5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:59.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.292+0000 7fb9a1800640 1 --2- 172.21.15.27:0/2673261380 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb99c15e7d0 0x7fb99c1f5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:59.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.292+0000 7fb9a0e00640 1 -- 172.21.15.27:0/2673261380 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb99c15e7d0 msgr2=0x7fb99c1f5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:59.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.292+0000 7fb9a0e00640 1 --2- 172.21.15.27:0/2673261380 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb99c15e7d0 0x7fb99c1f5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.292+0000 7fb9a0e00640 1 -- 172.21.15.27:0/2673261380 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb99c0b35a0 msgr2=0x7fb99c1f5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:59.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.292+0000 7fb9a0e00640 1 --2- 172.21.15.27:0/2673261380 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb99c0b35a0 0x7fb99c1f5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.292+0000 7fb9a0e00640 1 -- 172.21.15.27:0/2673261380 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb994009a70 con 0x7fb99c0afe10 2024-01-15T18:15:59.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.292+0000 7fb99be00640 1 --2- 172.21.15.27:0/2673261380 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb99c0b35a0 0x7fb99c1f5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:59.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.292+0000 7fb9a0e00640 1 --2- 172.21.15.27:0/2673261380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb99c0afe10 0x7fb99c158e00 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fb98c00e930 tx=0x7fb98c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:59.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.292+0000 7fb998c00640 1 -- 172.21.15.27:0/2673261380 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb98c009690 con 0x7fb99c0afe10 2024-01-15T18:15:59.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.292+0000 7fb998c00640 1 -- 172.21.15.27:0/2673261380 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb98c00ca00 con 0x7fb99c0afe10 2024-01-15T18:15:59.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.292+0000 7fb9a3800640 1 -- 172.21.15.27:0/2673261380 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb99c159a50 con 0x7fb99c0afe10 2024-01-15T18:15:59.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.292+0000 7fb998c00640 1 -- 172.21.15.27:0/2673261380 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb98c00cd70 con 0x7fb99c0afe10 2024-01-15T18:15:59.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.292+0000 7fb9a3800640 1 -- 172.21.15.27:0/2673261380 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb99c0b0f20 con 0x7fb99c0afe10 2024-01-15T18:15:59.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.296+0000 7fb9a3800640 1 -- 172.21.15.27:0/2673261380 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb99c04e850 con 0x7fb99c0afe10 2024-01-15T18:15:59.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.296+0000 7fb998c00640 1 -- 172.21.15.27:0/2673261380 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7fb98c013470 con 0x7fb99c0afe10 2024-01-15T18:15:59.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.296+0000 7fb998c00640 1 --2- 172.21.15.27:0/2673261380 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb9740674f0 0x7fb9740699b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:59.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.296+0000 7fb998c00640 1 -- 172.21.15.27:0/2673261380 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fb98c0832a0 con 0x7fb99c0afe10 2024-01-15T18:15:59.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.296+0000 7fb99be00640 1 --2- 172.21.15.27:0/2673261380 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb9740674f0 0x7fb9740699b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:59.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.300+0000 7fb99be00640 1 --2- 172.21.15.27:0/2673261380 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb9740674f0 0x7fb9740699b0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fb984007fd0 tx=0x7fb984007c20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:59.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.300+0000 7fb998c00640 1 -- 172.21.15.27:0/2673261380 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb98c089030 con 0x7fb99c0afe10 2024-01-15T18:15:59.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.476+0000 7fb9a3800640 1 -- 172.21.15.27:0/2673261380 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fb99c0cb7f0 con 0x7fb9740674f0 2024-01-15T18:15:59.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.480+0000 7fb998c00640 1 -- 172.21.15.27:0/2673261380 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fb99c0cb7f0 con 0x7fb9740674f0 2024-01-15T18:15:59.477 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:15:59.477 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:15:59.478 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-15T18:15:59.478 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:15:59.478 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:15:59.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.480+0000 7fb9a3800640 1 -- 172.21.15.27:0/2673261380 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb9740674f0 msgr2=0x7fb9740699b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:59.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.480+0000 7fb9a3800640 1 --2- 172.21.15.27:0/2673261380 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb9740674f0 0x7fb9740699b0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fb984007fd0 tx=0x7fb984007c20 comp rx=0 tx=0).stop 2024-01-15T18:15:59.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.480+0000 7fb9a3800640 1 -- 172.21.15.27:0/2673261380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb99c0afe10 msgr2=0x7fb99c158e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:59.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.480+0000 7fb9a3800640 1 --2- 172.21.15.27:0/2673261380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb99c0afe10 0x7fb99c158e00 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fb98c00e930 tx=0x7fb98c00edf0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.480+0000 7fb9a3800640 1 -- 172.21.15.27:0/2673261380 shutdown_connections 2024-01-15T18:15:59.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.480+0000 7fb9a3800640 1 --2- 172.21.15.27:0/2673261380 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb9740674f0 0x7fb9740699b0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.480+0000 7fb9a3800640 1 --2- 172.21.15.27:0/2673261380 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb99c15e7d0 0x7fb99c1f5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.480+0000 7fb9a3800640 1 --2- 172.21.15.27:0/2673261380 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb99c0b35a0 0x7fb99c1f5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.480+0000 7fb9a3800640 1 --2- 172.21.15.27:0/2673261380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb99c0afe10 0x7fb99c158e00 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.480+0000 7fb9a3800640 1 -- 172.21.15.27:0/2673261380 >> 172.21.15.27:0/2673261380 conn(0x7fb99c0b2a80 msgr2=0x7fb99c0cdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:59.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.480+0000 7fb9a3800640 1 -- 172.21.15.27:0/2673261380 shutdown_connections 2024-01-15T18:15:59.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.480+0000 7fb9a3800640 1 -- 172.21.15.27:0/2673261380 wait complete. 2024-01-15T18:15:59.500 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:15:59.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.644+0000 7f1ecfc00640 1 Processor -- start 2024-01-15T18:15:59.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.644+0000 7f1ecfc00640 1 -- start start 2024-01-15T18:15:59.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ecfc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ec815d190 0x7f1ec80cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:59.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ecfc00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f1ec806c520 0x7f1ec80cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:59.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ecfc00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec80b0d70 con 0x7f1ec815d570 2024-01-15T18:15:59.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ecfc00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec80d0540 con 0x7f1ec806c520 2024-01-15T18:15:59.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ecfc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec80b0440 con 0x7f1ec815d190 2024-01-15T18:15:59.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ec7e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1ec815d570 0x7f1ec806be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44394/0 (socket says 172.21.15.27:44394) 2024-01-15T18:15:59.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ec7e00640 1 -- 172.21.15.27:0/3767925060 learned_addr learned my addr 172.21.15.27:0/3767925060 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:59.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ecdc00640 1 --2- 172.21.15.27:0/3767925060 >> v2:172.21.15.93:3300/0 conn(0x7f1ec806c520 0x7f1ec80cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:59.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ecd200640 1 --2- 172.21.15.27:0/3767925060 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ec815d190 0x7f1ec80cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:59.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ec7400640 1 -- 172.21.15.27:0/3767925060 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548270778 0 0) 0x7f1ec80b0d70 con 0x7f1ec815d570 2024-01-15T18:15:59.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ec7400640 1 -- 172.21.15.27:0/3767925060 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea8003530 con 0x7f1ec815d570 2024-01-15T18:15:59.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ec7400640 1 -- 172.21.15.27:0/3767925060 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2682940661 0 0) 0x7f1ea8003530 con 0x7f1ec815d570 2024-01-15T18:15:59.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ec7400640 1 -- 172.21.15.27:0/3767925060 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ec815d190 msgr2=0x7f1ec80cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:59.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ec7400640 1 --2- 172.21.15.27:0/3767925060 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ec815d190 0x7f1ec80cfcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ec7400640 1 -- 172.21.15.27:0/3767925060 >> v2:172.21.15.93:3300/0 conn(0x7f1ec806c520 msgr2=0x7f1ec80cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:59.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ec7400640 1 --2- 172.21.15.27:0/3767925060 >> v2:172.21.15.93:3300/0 conn(0x7f1ec806c520 0x7f1ec80cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ec7400640 1 -- 172.21.15.27:0/3767925060 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec80b0720 con 0x7f1ec815d570 2024-01-15T18:15:59.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ec7400640 1 -- 172.21.15.27:0/3767925060 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f1eb0001680 con 0x7f1ec815d570 2024-01-15T18:15:59.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ecd200640 1 --2- 172.21.15.27:0/3767925060 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ec815d190 0x7f1ec80cfcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:15:59.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ec7400640 1 -- 172.21.15.27:0/3767925060 >> v1:172.21.15.27:6789/0 conn(0x7f1ec815d570 legacy=0x7f1ec806be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:59.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ec7400640 1 --2- 172.21.15.27:0/3767925060 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ec815d190 0x7f1ea80039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:59.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ec7400640 1 --2- 172.21.15.27:0/3767925060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ea8003f30 0x7f1ea80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:59.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ec7400640 1 --2- 172.21.15.27:0/3767925060 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ea8004d10 0x7f1ea8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:59.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ec7400640 1 -- 172.21.15.27:0/3767925060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec80d0540 con 0x7f1ea8003f30 2024-01-15T18:15:59.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ec7400640 1 -- 172.21.15.27:0/3767925060 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec80b0440 con 0x7f1ea8004d10 2024-01-15T18:15:59.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ec7400640 1 -- 172.21.15.27:0/3767925060 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea800b990 con 0x7f1ec815d190 2024-01-15T18:15:59.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ec7e00640 1 --2- 172.21.15.27:0/3767925060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ea8003f30 0x7f1ea80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:59.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ecd200640 1 --2- 172.21.15.27:0/3767925060 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ec815d190 0x7f1ea80039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:59.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ecdc00640 1 --2- 172.21.15.27:0/3767925060 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ea8004d10 0x7f1ea8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:59.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ec7e00640 1 -- 172.21.15.27:0/3767925060 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ec815d190 msgr2=0x7f1ea80039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:59.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ec7e00640 1 --2- 172.21.15.27:0/3767925060 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ec815d190 0x7f1ea80039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ec7e00640 1 -- 172.21.15.27:0/3767925060 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ea8004d10 msgr2=0x7f1ea8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:59.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ec7e00640 1 --2- 172.21.15.27:0/3767925060 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ea8004d10 0x7f1ea8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.648+0000 7f1ec7e00640 1 -- 172.21.15.27:0/3767925060 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb8009420 con 0x7f1ea8003f30 2024-01-15T18:15:59.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.652+0000 7f1ec7e00640 1 --2- 172.21.15.27:0/3767925060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ea8003f30 0x7f1ea80047d0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f1eb000c240 tx=0x7f1eb001f3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3fb35c43324302c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:59.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.652+0000 7f1ec7400640 1 -- 172.21.15.27:0/3767925060 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb0005ae0 con 0x7f1ea8003f30 2024-01-15T18:15:59.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.652+0000 7f1ec7400640 1 -- 172.21.15.27:0/3767925060 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb00ab070 con 0x7f1ea8003f30 2024-01-15T18:15:59.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.652+0000 7f1ec7400640 1 -- 172.21.15.27:0/3767925060 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb0007070 con 0x7f1ea8003f30 2024-01-15T18:15:59.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.652+0000 7f1ecfc00640 1 -- 172.21.15.27:0/3767925060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ea8003f30 msgr2=0x7f1ea80047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:59.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.652+0000 7f1ecfc00640 1 --2- 172.21.15.27:0/3767925060 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ea8003f30 0x7f1ea80047d0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f1eb000c240 tx=0x7f1eb001f3d0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.652+0000 7f1ecdc00640 1 -- 172.21.15.27:0/3767925060 reap_dead start 2024-01-15T18:15:59.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.652+0000 7f1ecfc00640 1 -- 172.21.15.27:0/3767925060 shutdown_connections 2024-01-15T18:15:59.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.652+0000 7f1ecfc00640 1 -- 172.21.15.27:0/3767925060 >> 172.21.15.27:0/3767925060 conn(0x7f1ec80d0e70 msgr2=0x7f1ec80d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:59.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.652+0000 7f1ecfc00640 1 -- 172.21.15.27:0/3767925060 shutdown_connections 2024-01-15T18:15:59.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.652+0000 7f1ecfc00640 1 -- 172.21.15.27:0/3767925060 wait complete. 2024-01-15T18:15:59.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.656+0000 7f1ecfc00640 1 Processor -- start 2024-01-15T18:15:59.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.656+0000 7f1ecfc00640 1 -- start start 2024-01-15T18:15:59.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.656+0000 7f1ecfc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ea8003f30 0x7f1ec81571c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:59.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.656+0000 7f1ecfc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ec80c4ab0 0x7f1ec80c7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:59.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.656+0000 7f1ecfc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ec80af220 0x7f1ec80af6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:59.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.656+0000 7f1ecfc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec80afbe0 con 0x7f1ec80af220 2024-01-15T18:15:59.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.656+0000 7f1ecfc00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec80b4a70 con 0x7f1ea8003f30 2024-01-15T18:15:59.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.656+0000 7f1ecfc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec8157700 con 0x7f1ec80c4ab0 2024-01-15T18:15:59.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.656+0000 7f1ecdc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ec80af220 0x7f1ec80af6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:59.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.656+0000 7f1ecdc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ec80af220 0x7f1ec80af6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37428/0 (socket says 172.21.15.27:37428) 2024-01-15T18:15:59.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.656+0000 7f1ecdc00640 1 -- 172.21.15.27:0/2030700519 learned_addr learned my addr 172.21.15.27:0/2030700519 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:59.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.656+0000 7f1ec7e00640 1 --2- 172.21.15.27:0/2030700519 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ec80c4ab0 0x7f1ec80c7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:59.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.656+0000 7f1ecd200640 1 --2- 172.21.15.27:0/2030700519 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ea8003f30 0x7f1ec81571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:59.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.656+0000 7f1ecdc00640 1 -- 172.21.15.27:0/2030700519 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ec80c4ab0 msgr2=0x7f1ec80c7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:59.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.656+0000 7f1ecdc00640 1 --2- 172.21.15.27:0/2030700519 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ec80c4ab0 0x7f1ec80c7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.656+0000 7f1ecdc00640 1 -- 172.21.15.27:0/2030700519 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ea8003f30 msgr2=0x7f1ec81571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:59.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.656+0000 7f1ecdc00640 1 --2- 172.21.15.27:0/2030700519 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ea8003f30 0x7f1ec81571c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.656+0000 7f1ecdc00640 1 -- 172.21.15.27:0/2030700519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec80b3e60 con 0x7f1ec80af220 2024-01-15T18:15:59.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.656+0000 7f1ecd200640 1 --2- 172.21.15.27:0/2030700519 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ea8003f30 0x7f1ec81571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:59.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.656+0000 7f1ecdc00640 1 --2- 172.21.15.27:0/2030700519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ec80af220 0x7f1ec80af6a0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f1ec80d0e20 tx=0x7f1ec000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:59.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.656+0000 7f1ec4c00640 1 -- 172.21.15.27:0/2030700519 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec000e040 con 0x7f1ec80af220 2024-01-15T18:15:59.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.656+0000 7f1ec4c00640 1 -- 172.21.15.27:0/2030700519 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ec000f040 con 0x7f1ec80af220 2024-01-15T18:15:59.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.656+0000 7f1ec4c00640 1 -- 172.21.15.27:0/2030700519 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec0003c00 con 0x7f1ec80af220 2024-01-15T18:15:59.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.656+0000 7f1ecfc00640 1 -- 172.21.15.27:0/2030700519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec80b4140 con 0x7f1ec80af220 2024-01-15T18:15:59.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.656+0000 7f1ecfc00640 1 -- 172.21.15.27:0/2030700519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec80b4700 con 0x7f1ec80af220 2024-01-15T18:15:59.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.660+0000 7f1ecfc00640 1 -- 172.21.15.27:0/2030700519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ec804e850 con 0x7f1ec80af220 2024-01-15T18:15:59.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.660+0000 7f1ec4c00640 1 -- 172.21.15.27:0/2030700519 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7f1ec0004030 con 0x7f1ec80af220 2024-01-15T18:15:59.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.660+0000 7f1ec4c00640 1 --2- 172.21.15.27:0/2030700519 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1ea00674f0 0x7f1ea00699b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:59.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.660+0000 7f1ec4c00640 1 -- 172.21.15.27:0/2030700519 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f1ec0083470 con 0x7f1ec80af220 2024-01-15T18:15:59.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.660+0000 7f1ecd200640 1 --2- 172.21.15.27:0/2030700519 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1ea00674f0 0x7f1ea00699b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:59.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.660+0000 7f1ecd200640 1 --2- 172.21.15.27:0/2030700519 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1ea00674f0 0x7f1ea00699b0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8009c90 tx=0x7f1eb8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:59.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.664+0000 7f1ec4c00640 1 -- 172.21.15.27:0/2030700519 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1ec0055bd0 con 0x7f1ec80af220 2024-01-15T18:15:59.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.820+0000 7f1ecfc00640 1 -- 172.21.15.27:0/2030700519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f1ec80ce420 con 0x7f1ec80af220 2024-01-15T18:15:59.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.820+0000 7f1ec4c00640 1 -- 172.21.15.27:0/2030700519 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1ec0055760 con 0x7f1ec80af220 2024-01-15T18:15:59.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.824+0000 7f1ecfc00640 1 -- 172.21.15.27:0/2030700519 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1ea00674f0 msgr2=0x7f1ea00699b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:59.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.824+0000 7f1ecfc00640 1 --2- 172.21.15.27:0/2030700519 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1ea00674f0 0x7f1ea00699b0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8009c90 tx=0x7f1eb8008040 comp rx=0 tx=0).stop 2024-01-15T18:15:59.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.824+0000 7f1ecfc00640 1 -- 172.21.15.27:0/2030700519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ec80af220 msgr2=0x7f1ec80af6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:59.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.824+0000 7f1ecfc00640 1 --2- 172.21.15.27:0/2030700519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ec80af220 0x7f1ec80af6a0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f1ec80d0e20 tx=0x7f1ec000f580 comp rx=0 tx=0).stop 2024-01-15T18:15:59.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.824+0000 7f1ecfc00640 1 -- 172.21.15.27:0/2030700519 shutdown_connections 2024-01-15T18:15:59.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.824+0000 7f1ecfc00640 1 --2- 172.21.15.27:0/2030700519 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1ea00674f0 0x7f1ea00699b0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.824+0000 7f1ecfc00640 1 --2- 172.21.15.27:0/2030700519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ec80af220 0x7f1ec80af6a0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.824+0000 7f1ecfc00640 1 --2- 172.21.15.27:0/2030700519 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1ec80c4ab0 0x7f1ec80c7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.824+0000 7f1ecfc00640 1 --2- 172.21.15.27:0/2030700519 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1ea8003f30 0x7f1ec81571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.824+0000 7f1ecfc00640 1 -- 172.21.15.27:0/2030700519 >> 172.21.15.27:0/2030700519 conn(0x7f1ec80d0e70 msgr2=0x7f1ec815aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:59.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.824+0000 7f1ecfc00640 1 -- 172.21.15.27:0/2030700519 shutdown_connections 2024-01-15T18:15:59.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.824+0000 7f1ecfc00640 1 -- 172.21.15.27:0/2030700519 wait complete. 2024-01-15T18:15:59.841 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-15T18:15:59.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.988+0000 7f9b13800640 1 Processor -- start 2024-01-15T18:15:59.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.988+0000 7f9b13800640 1 -- start start 2024-01-15T18:15:59.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.988+0000 7f9b13800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b0c15f550 0x7f9b0c0cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:59.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.988+0000 7f9b13800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9b0c0cf110 0x7f9b0c0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:59.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.988+0000 7f9b13800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b0c0b1cb0 con 0x7f9b0c158900 2024-01-15T18:15:59.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.988+0000 7f9b13800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b0c15fa00 con 0x7f9b0c0cf110 2024-01-15T18:15:59.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.988+0000 7f9b13800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c0cf540 con 0x7f9b0c15f550 2024-01-15T18:15:59.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.988+0000 7f9b10e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9b0c158900 0x7f9b0c15bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44408/0 (socket says 172.21.15.27:44408) 2024-01-15T18:15:59.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.988+0000 7f9b11800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9b0c0cf110 0x7f9b0c0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:59.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.988+0000 7f9b10e00640 1 -- 172.21.15.27:0/144038856 learned_addr learned my addr 172.21.15.27:0/144038856 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:15:59.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.988+0000 7f9b0be00640 1 --2- 172.21.15.27:0/144038856 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b0c15f550 0x7f9b0c0cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:59.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.992+0000 7f9b0b400640 1 -- 172.21.15.27:0/144038856 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975508262 0 0) 0x7f9b0c0b1cb0 con 0x7f9b0c158900 2024-01-15T18:15:59.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.992+0000 7f9b0b400640 1 -- 172.21.15.27:0/144038856 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aec003530 con 0x7f9b0c158900 2024-01-15T18:15:59.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.992+0000 7f9b11800640 1 -- 172.21.15.27:0/144038856 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b0c15f550 msgr2=0x7f9b0c0cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:59.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.992+0000 7f9b11800640 1 --2- 172.21.15.27:0/144038856 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b0c15f550 0x7f9b0c0cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.992+0000 7f9b11800640 1 -- 172.21.15.27:0/144038856 >> v1:172.21.15.27:6789/0 conn(0x7f9b0c158900 legacy=0x7f9b0c15bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:59.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.992+0000 7f9b11800640 1 -- 172.21.15.27:0/144038856 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0c0b1f00 con 0x7f9b0c0cf110 2024-01-15T18:15:59.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.992+0000 7f9b11800640 1 --2- 172.21.15.27:0/144038856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0cf110 0x7f9b0c0701e0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f9b04009d90 tx=0x7f9b0401c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82dacfe3cba4576 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:59.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.992+0000 7f9b0b400640 1 -- 172.21.15.27:0/144038856 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c070f30 con 0x7f9b0c0cf110 2024-01-15T18:15:59.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.992+0000 7f9b0b400640 1 -- 172.21.15.27:0/144038856 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b040a59e0 con 0x7f9b0c0cf110 2024-01-15T18:15:59.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.992+0000 7f9b0b400640 1 -- 172.21.15.27:0/144038856 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b040a5d70 con 0x7f9b0c0cf110 2024-01-15T18:15:59.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.992+0000 7f9b13800640 1 -- 172.21.15.27:0/144038856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0cf110 msgr2=0x7f9b0c0701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:59.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.992+0000 7f9b13800640 1 --2- 172.21.15.27:0/144038856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0cf110 0x7f9b0c0701e0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f9b04009d90 tx=0x7f9b0401c920 comp rx=0 tx=0).stop 2024-01-15T18:15:59.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.996+0000 7f9b13800640 1 -- 172.21.15.27:0/144038856 shutdown_connections 2024-01-15T18:15:59.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.996+0000 7f9b13800640 1 --2- 172.21.15.27:0/144038856 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c0cf110 0x7f9b0c0701e0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.996+0000 7f9b13800640 1 --2- 172.21.15.27:0/144038856 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b0c15f550 0x7f9b0c0cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.996+0000 7f9b13800640 1 -- 172.21.15.27:0/144038856 >> 172.21.15.27:0/144038856 conn(0x7f9b0c0cb700 msgr2=0x7f9b0c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:59.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.996+0000 7f9b13800640 1 -- 172.21.15.27:0/144038856 shutdown_connections 2024-01-15T18:15:59.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.996+0000 7f9b13800640 1 -- 172.21.15.27:0/144038856 wait complete. 2024-01-15T18:15:59.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.996+0000 7f9b13800640 1 Processor -- start 2024-01-15T18:15:59.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.996+0000 7f9b13800640 1 -- start start 2024-01-15T18:15:59.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.996+0000 7f9b13800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b0c0cf110 0x7f9b0c0ae010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:59.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.996+0000 7f9b13800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c158900 0x7f9b0c15a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:59.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.996+0000 7f9b13800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9b0c15f550 0x7f9b0c14d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:59.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.996+0000 7f9b13800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c0ccd10 con 0x7f9b0c158900 2024-01-15T18:15:59.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.996+0000 7f9b13800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c0cd190 con 0x7f9b0c15f550 2024-01-15T18:15:59.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.996+0000 7f9b13800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0c0cd470 con 0x7f9b0c0cf110 2024-01-15T18:15:59.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.996+0000 7f9b0be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c158900 0x7f9b0c15a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:59.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.996+0000 7f9b0be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c158900 0x7f9b0c15a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37454/0 (socket says 172.21.15.27:37454) 2024-01-15T18:15:59.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.996+0000 7f9b0be00640 1 -- 172.21.15.27:0/1327165276 learned_addr learned my addr 172.21.15.27:0/1327165276 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:15:59.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.996+0000 7f9b10e00640 1 --2- 172.21.15.27:0/1327165276 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b0c0cf110 0x7f9b0c0ae010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:59.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.996+0000 7f9b11800640 1 --2- 172.21.15.27:0/1327165276 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9b0c15f550 0x7f9b0c14d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:59.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.996+0000 7f9b0be00640 1 -- 172.21.15.27:0/1327165276 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b0c0cf110 msgr2=0x7f9b0c0ae010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:59.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.996+0000 7f9b0be00640 1 --2- 172.21.15.27:0/1327165276 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b0c0cf110 0x7f9b0c0ae010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.996+0000 7f9b0be00640 1 -- 172.21.15.27:0/1327165276 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9b0c15f550 msgr2=0x7f9b0c14d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:59.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.996+0000 7f9b0be00640 1 --2- 172.21.15.27:0/1327165276 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9b0c15f550 0x7f9b0c14d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:59.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:15:59.996+0000 7f9b0be00640 1 -- 172.21.15.27:0/1327165276 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b04009a70 con 0x7f9b0c158900 2024-01-15T18:15:59.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.000+0000 7f9b0be00640 1 --2- 172.21.15.27:0/1327165276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c158900 0x7f9b0c15a480 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f9af400d800 tx=0x7f9af400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:59.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.000+0000 7f9b08c00640 1 -- 172.21.15.27:0/1327165276 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af400b6c0 con 0x7f9b0c158900 2024-01-15T18:15:59.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.000+0000 7f9b08c00640 1 -- 172.21.15.27:0/1327165276 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af400b820 con 0x7f9b0c158900 2024-01-15T18:15:59.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.000+0000 7f9b08c00640 1 -- 172.21.15.27:0/1327165276 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af400bb90 con 0x7f9b0c158900 2024-01-15T18:15:59.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.000+0000 7f9b13800640 1 -- 172.21.15.27:0/1327165276 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b0c14f9c0 con 0x7f9b0c158900 2024-01-15T18:15:59.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.000+0000 7f9b13800640 1 -- 172.21.15.27:0/1327165276 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b0c14ea10 con 0x7f9b0c158900 2024-01-15T18:15:59.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.000+0000 7f9b13800640 1 -- 172.21.15.27:0/1327165276 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b0c04e850 con 0x7f9b0c158900 2024-01-15T18:15:59.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.000+0000 7f9b08c00640 1 -- 172.21.15.27:0/1327165276 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7f9af400bcf0 con 0x7f9b0c158900 2024-01-15T18:16:00.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.000+0000 7f9b08c00640 1 --2- 172.21.15.27:0/1327165276 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9ae40674c0 0x7f9ae4069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:00.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.000+0000 7f9b08c00640 1 -- 172.21.15.27:0/1327165276 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f9af4083470 con 0x7f9b0c158900 2024-01-15T18:16:00.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.004+0000 7f9b10e00640 1 --2- 172.21.15.27:0/1327165276 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9ae40674c0 0x7f9ae4069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:00.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.004+0000 7f9b10e00640 1 --2- 172.21.15.27:0/1327165276 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9ae40674c0 0x7f9ae4069980 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f9afc0011f0 tx=0x7f9afc007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:00.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.004+0000 7f9b08c00640 1 -- 172.21.15.27:0/1327165276 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9af4051f50 con 0x7f9b0c158900 2024-01-15T18:16:00.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.188+0000 7f9b13800640 1 -- 172.21.15.27:0/1327165276 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f9b0c0cd920 con 0x7f9ae40674c0 2024-01-15T18:16:00.192 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:16:00.189+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:16:00.193 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:16:00.189+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:16:00.193 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:16:00.189+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:16:00.193 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:16:00.189+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:16:00.193 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:16:00.189+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:16:00.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.228+0000 7f9b08c00640 1 -- 172.21.15.27:0/1327165276 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9b0c0cd920 con 0x7f9ae40674c0 2024-01-15T18:16:00.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.232+0000 7f9b13800640 1 -- 172.21.15.27:0/1327165276 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9ae40674c0 msgr2=0x7f9ae4069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:00.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.232+0000 7f9b13800640 1 --2- 172.21.15.27:0/1327165276 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9ae40674c0 0x7f9ae4069980 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f9afc0011f0 tx=0x7f9afc007d40 comp rx=0 tx=0).stop 2024-01-15T18:16:00.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.232+0000 7f9b13800640 1 -- 172.21.15.27:0/1327165276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c158900 msgr2=0x7f9b0c15a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:00.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.232+0000 7f9b13800640 1 --2- 172.21.15.27:0/1327165276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c158900 0x7f9b0c15a480 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f9af400d800 tx=0x7f9af400dcc0 comp rx=0 tx=0).stop 2024-01-15T18:16:00.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.232+0000 7f9b13800640 1 -- 172.21.15.27:0/1327165276 shutdown_connections 2024-01-15T18:16:00.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.232+0000 7f9b13800640 1 --2- 172.21.15.27:0/1327165276 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9ae40674c0 0x7f9ae4069980 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:00.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.232+0000 7f9b13800640 1 --2- 172.21.15.27:0/1327165276 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9b0c15f550 0x7f9b0c14d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:00.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.232+0000 7f9b13800640 1 --2- 172.21.15.27:0/1327165276 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b0c158900 0x7f9b0c15a480 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:00.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.232+0000 7f9b13800640 1 --2- 172.21.15.27:0/1327165276 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b0c0cf110 0x7f9b0c0ae010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:00.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.232+0000 7f9b13800640 1 -- 172.21.15.27:0/1327165276 >> 172.21.15.27:0/1327165276 conn(0x7f9b0c0cb700 msgr2=0x7f9b0c154870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:00.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.232+0000 7f9b13800640 1 -- 172.21.15.27:0/1327165276 shutdown_connections 2024-01-15T18:16:00.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.232+0000 7f9b13800640 1 -- 172.21.15.27:0/1327165276 wait complete. 2024-01-15T18:16:00.250 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:16:00.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.396+0000 7f913c600640 1 Processor -- start 2024-01-15T18:16:00.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.396+0000 7f913c600640 1 -- start start 2024-01-15T18:16:00.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.400+0000 7f913c600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f9134160ca0 0x7f91340d01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:00.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.400+0000 7f913c600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91340b3b40 con 0x7f91340b4000 2024-01-15T18:16:00.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.400+0000 7f913c600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91340b4770 con 0x7f91341608c0 2024-01-15T18:16:00.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.400+0000 7f913c600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91340d4070 con 0x7f9134160ca0 2024-01-15T18:16:00.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.400+0000 7f913a600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f91340b4000 0x7f91340cdec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44418/0 (socket says 172.21.15.27:44418) 2024-01-15T18:16:00.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.400+0000 7f913a600640 1 -- 172.21.15.27:0/319378978 learned_addr learned my addr 172.21.15.27:0/319378978 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:16:00.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.400+0000 7f9139200640 1 --2- 172.21.15.27:0/319378978 >> v2:172.21.15.93:3300/0 conn(0x7f9134160ca0 0x7f91340d01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:00.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.400+0000 7f9133e00640 1 -- 172.21.15.27:0/319378978 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1904591578 0 0) 0x7f91340b3b40 con 0x7f91340b4000 2024-01-15T18:16:00.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.400+0000 7f9133e00640 1 -- 172.21.15.27:0/319378978 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9114003530 con 0x7f91340b4000 2024-01-15T18:16:00.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.400+0000 7f9139200640 1 -- 172.21.15.27:0/319378978 >> v1:172.21.15.93:6789/0 conn(0x7f91341608c0 legacy=0x7f91340cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:00.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.400+0000 7f9139200640 1 -- 172.21.15.27:0/319378978 >> v1:172.21.15.27:6789/0 conn(0x7f91340b4000 legacy=0x7f91340cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:00.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.400+0000 7f9139200640 1 -- 172.21.15.27:0/319378978 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91340d3d90 con 0x7f9134160ca0 2024-01-15T18:16:00.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.404+0000 7f9139200640 1 --2- 172.21.15.27:0/319378978 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9134160ca0 0x7f91340d01f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f911c0097c0 tx=0x7f911c01c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=282341eda4ef08a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:00.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.404+0000 7f9133e00640 1 -- 172.21.15.27:0/319378978 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911c0ab650 con 0x7f9134160ca0 2024-01-15T18:16:00.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.404+0000 7f9133e00640 1 -- 172.21.15.27:0/319378978 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f911c0ab7b0 con 0x7f9134160ca0 2024-01-15T18:16:00.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.404+0000 7f9133e00640 1 -- 172.21.15.27:0/319378978 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911c0abb40 con 0x7f9134160ca0 2024-01-15T18:16:00.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.404+0000 7f913c600640 1 -- 172.21.15.27:0/319378978 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9134160ca0 msgr2=0x7f91340d01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:00.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.404+0000 7f913c600640 1 --2- 172.21.15.27:0/319378978 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9134160ca0 0x7f91340d01f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f911c0097c0 tx=0x7f911c01c450 comp rx=0 tx=0).stop 2024-01-15T18:16:00.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.404+0000 7f913c600640 1 -- 172.21.15.27:0/319378978 shutdown_connections 2024-01-15T18:16:00.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.404+0000 7f913c600640 1 --2- 172.21.15.27:0/319378978 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9134160ca0 0x7f91340d01f0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:00.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.404+0000 7f913c600640 1 -- 172.21.15.27:0/319378978 >> 172.21.15.27:0/319378978 conn(0x7f91340d30d0 msgr2=0x7f91340d34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:00.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.404+0000 7f913c600640 1 -- 172.21.15.27:0/319378978 shutdown_connections 2024-01-15T18:16:00.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.404+0000 7f913c600640 1 -- 172.21.15.27:0/319378978 wait complete. 2024-01-15T18:16:00.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.404+0000 7f913c600640 1 Processor -- start 2024-01-15T18:16:00.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.404+0000 7f913c600640 1 -- start start 2024-01-15T18:16:00.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.408+0000 7f913c600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f91340b4000 0x7f91340c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:00.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.408+0000 7f913c600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91341608c0 0x7f91340c8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:00.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.408+0000 7f913c600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9134160ca0 0x7f91340c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:00.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.408+0000 7f913c600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91341f59e0 con 0x7f91341608c0 2024-01-15T18:16:00.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.408+0000 7f913c600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91340c7140 con 0x7f9134160ca0 2024-01-15T18:16:00.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.408+0000 7f913c600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91340c73e0 con 0x7f91340b4000 2024-01-15T18:16:00.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.408+0000 7f9139200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91341608c0 0x7f91340c8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:00.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.408+0000 7f9139200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91341608c0 0x7f91340c8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37456/0 (socket says 172.21.15.27:37456) 2024-01-15T18:16:00.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.408+0000 7f9139200640 1 -- 172.21.15.27:0/3760547183 learned_addr learned my addr 172.21.15.27:0/3760547183 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:00.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.408+0000 7f9139c00640 1 --2- 172.21.15.27:0/3760547183 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f91340b4000 0x7f91340c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:00.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.408+0000 7f913a600640 1 --2- 172.21.15.27:0/3760547183 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9134160ca0 0x7f91340c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:00.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.408+0000 7f9139200640 1 -- 172.21.15.27:0/3760547183 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f91340b4000 msgr2=0x7f91340c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:00.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.408+0000 7f9139200640 1 --2- 172.21.15.27:0/3760547183 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f91340b4000 0x7f91340c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:00.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.408+0000 7f9139200640 1 -- 172.21.15.27:0/3760547183 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9134160ca0 msgr2=0x7f91340c8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:00.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.408+0000 7f9139200640 1 --2- 172.21.15.27:0/3760547183 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9134160ca0 0x7f91340c8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:00.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.408+0000 7f9139200640 1 -- 172.21.15.27:0/3760547183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f911c009420 con 0x7f91341608c0 2024-01-15T18:16:00.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.408+0000 7f9139200640 1 --2- 172.21.15.27:0/3760547183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91341608c0 0x7f91340c8520 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f911c01a080 tx=0x7f911c0191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:00.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.408+0000 7f9131600640 1 -- 172.21.15.27:0/3760547183 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911c0abe70 con 0x7f91341608c0 2024-01-15T18:16:00.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.408+0000 7f9131600640 1 -- 172.21.15.27:0/3760547183 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f911c0bf450 con 0x7f91341608c0 2024-01-15T18:16:00.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.408+0000 7f9131600640 1 -- 172.21.15.27:0/3760547183 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911c0bf820 con 0x7f91341608c0 2024-01-15T18:16:00.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.408+0000 7f913c600640 1 -- 172.21.15.27:0/3760547183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91340c76c0 con 0x7f91341608c0 2024-01-15T18:16:00.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.408+0000 7f913c600640 1 -- 172.21.15.27:0/3760547183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91340c7c80 con 0x7f91341608c0 2024-01-15T18:16:00.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.408+0000 7f913c600640 1 -- 172.21.15.27:0/3760547183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f913404e850 con 0x7f91341608c0 2024-01-15T18:16:00.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.412+0000 7f9131600640 1 -- 172.21.15.27:0/3760547183 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 30) v1 ==== 87231+0+0 (secure 0 0 0) 0x7f911c0bf980 con 0x7f91341608c0 2024-01-15T18:16:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.412+0000 7f9131600640 1 --2- 172.21.15.27:0/3760547183 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f910c0674c0 0x7f910c069980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.412+0000 7f9131600640 1 -- 172.21.15.27:0/3760547183 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f911c11d8a0 con 0x7f91341608c0 2024-01-15T18:16:00.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.412+0000 7f9139c00640 1 --2- 172.21.15.27:0/3760547183 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f910c0674c0 0x7f910c069980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:00.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.412+0000 7f9139c00640 1 --2- 172.21.15.27:0/3760547183 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f910c0674c0 0x7f910c069980 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9124007fd0 tx=0x7f9124009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:00.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.416+0000 7f9131600640 1 -- 172.21.15.27:0/3760547183 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f911c0ec380 con 0x7f91341608c0 2024-01-15T18:16:00.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.600+0000 7f913c600640 1 -- 172.21.15.27:0/3760547183 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f91340b1940 con 0x7f910c0674c0 2024-01-15T18:16:00.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.604+0000 7f9131600640 1 -- 172.21.15.27:0/3760547183 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f91340b1940 con 0x7f910c0674c0 2024-01-15T18:16:00.602 INFO:teuthology.orchestra.run.smithi027.stdout:/volumes/_nogroup/sv1/4482b5e7-74d5-4ce9-8b87-3b51c5526f5f 2024-01-15T18:16:00.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.608+0000 7f913c600640 1 -- 172.21.15.27:0/3760547183 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f910c0674c0 msgr2=0x7f910c069980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:00.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.608+0000 7f913c600640 1 --2- 172.21.15.27:0/3760547183 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f910c0674c0 0x7f910c069980 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9124007fd0 tx=0x7f9124009040 comp rx=0 tx=0).stop 2024-01-15T18:16:00.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.608+0000 7f913c600640 1 -- 172.21.15.27:0/3760547183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91341608c0 msgr2=0x7f91340c8520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:00.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.608+0000 7f913c600640 1 --2- 172.21.15.27:0/3760547183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91341608c0 0x7f91340c8520 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f911c01a080 tx=0x7f911c0191f0 comp rx=0 tx=0).stop 2024-01-15T18:16:00.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.608+0000 7f913c600640 1 -- 172.21.15.27:0/3760547183 shutdown_connections 2024-01-15T18:16:00.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.608+0000 7f913c600640 1 --2- 172.21.15.27:0/3760547183 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f910c0674c0 0x7f910c069980 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:00.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.608+0000 7f913c600640 1 --2- 172.21.15.27:0/3760547183 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9134160ca0 0x7f91340c8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:00.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.608+0000 7f913c600640 1 --2- 172.21.15.27:0/3760547183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91341608c0 0x7f91340c8520 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:00.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.608+0000 7f913c600640 1 --2- 172.21.15.27:0/3760547183 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f91340b4000 0x7f91340c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:00.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.608+0000 7f913c600640 1 -- 172.21.15.27:0/3760547183 >> 172.21.15.27:0/3760547183 conn(0x7f91340d30d0 msgr2=0x7f91340b0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:00.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.608+0000 7f913c600640 1 -- 172.21.15.27:0/3760547183 shutdown_connections 2024-01-15T18:16:00.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:00.608+0000 7f913c600640 1 -- 172.21.15.27:0/3760547183 wait complete. 2024-01-15T18:16:00.625 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/4482b5e7-74d5-4ce9-8b87-3b51c5526f5f 2024-01-15T18:16:00.625 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:16:00.626 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:16:00.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:16:00.635 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/4482b5e7-74d5-4ce9-8b87-3b51c5526f5f/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:16:07.792 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records in 2024-01-15T18:16:07.793 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records out 2024-01-15T18:16:07.793 INFO:teuthology.orchestra.run.smithi027.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.09286 s, 151 MB/s 2024-01-15T18:16:07.794 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:16:07.794 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:16:07.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:16:07.803 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/4482b5e7-74d5-4ce9-8b87-3b51c5526f5f/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:16:14.710 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records in 2024-01-15T18:16:14.711 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records out 2024-01-15T18:16:14.711 INFO:teuthology.orchestra.run.smithi027.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.85386 s, 157 MB/s 2024-01-15T18:16:14.712 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:16:14.712 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:16:14.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:16:14.721 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/4482b5e7-74d5-4ce9-8b87-3b51c5526f5f/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:16:21.652 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records in 2024-01-15T18:16:21.652 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records out 2024-01-15T18:16:21.653 INFO:teuthology.orchestra.run.smithi027.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.87225 s, 156 MB/s 2024-01-15T18:16:21.654 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 2 2024-01-15T18:16:21.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.800+0000 7fbd97000640 1 Processor -- start 2024-01-15T18:16:21.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.800+0000 7fbd97000640 1 -- start start 2024-01-15T18:16:21.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.804+0000 7fbd97000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fbd9015f550 0x7fbd900cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:21.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.804+0000 7fbd97000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbd900cf110 0x7fbd900701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:21.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.804+0000 7fbd97000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd900b1cb0 con 0x7fbd90158900 2024-01-15T18:16:21.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.804+0000 7fbd97000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd900cf580 con 0x7fbd900cf110 2024-01-15T18:16:21.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.804+0000 7fbd97000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd9015fa00 con 0x7fbd9015f550 2024-01-15T18:16:21.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.804+0000 7fbd95000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbd900cf110 0x7fbd900701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:21.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.804+0000 7fbd95000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbd900cf110 0x7fbd900701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49922/0 (socket says 172.21.15.27:49922) 2024-01-15T18:16:21.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.804+0000 7fbd95000640 1 -- 172.21.15.27:0/678918131 learned_addr learned my addr 172.21.15.27:0/678918131 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:21.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.804+0000 7fbd8f400640 1 --2- 172.21.15.27:0/678918131 >> v2:172.21.15.93:3300/0 conn(0x7fbd9015f550 0x7fbd900cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:21.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.804+0000 7fbd95000640 1 -- 172.21.15.27:0/678918131 >> v2:172.21.15.93:3300/0 conn(0x7fbd9015f550 msgr2=0x7fbd900cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:21.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.804+0000 7fbd95000640 1 --2- 172.21.15.27:0/678918131 >> v2:172.21.15.93:3300/0 conn(0x7fbd9015f550 0x7fbd900cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:21.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.804+0000 7fbd95000640 1 -- 172.21.15.27:0/678918131 >> v1:172.21.15.93:6789/0 conn(0x7fbd90158900 legacy=0x7fbd9015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:21.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.804+0000 7fbd95000640 1 -- 172.21.15.27:0/678918131 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd900b1f00 con 0x7fbd900cf110 2024-01-15T18:16:21.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.804+0000 7fbd95000640 1 --2- 172.21.15.27:0/678918131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd900cf110 0x7fbd900701e0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fbd88009e50 tx=0x7fbd8801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25c14dbed4a99e97 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:21.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.804+0000 7fbd8ea00640 1 -- 172.21.15.27:0/678918131 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd90070f30 con 0x7fbd900cf110 2024-01-15T18:16:21.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.804+0000 7fbd8ea00640 1 -- 172.21.15.27:0/678918131 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd880a5860 con 0x7fbd900cf110 2024-01-15T18:16:21.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.808+0000 7fbd8ea00640 1 -- 172.21.15.27:0/678918131 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd880a5bf0 con 0x7fbd900cf110 2024-01-15T18:16:21.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.808+0000 7fbd97000640 1 -- 172.21.15.27:0/678918131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd900cf110 msgr2=0x7fbd900701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:21.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.808+0000 7fbd97000640 1 --2- 172.21.15.27:0/678918131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd900cf110 0x7fbd900701e0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fbd88009e50 tx=0x7fbd8801c9a0 comp rx=0 tx=0).stop 2024-01-15T18:16:21.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.808+0000 7fbd97000640 1 -- 172.21.15.27:0/678918131 shutdown_connections 2024-01-15T18:16:21.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.808+0000 7fbd97000640 1 --2- 172.21.15.27:0/678918131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd900cf110 0x7fbd900701e0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:21.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.808+0000 7fbd97000640 1 --2- 172.21.15.27:0/678918131 >> v2:172.21.15.93:3300/0 conn(0x7fbd9015f550 0x7fbd900cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:21.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.808+0000 7fbd97000640 1 -- 172.21.15.27:0/678918131 >> 172.21.15.27:0/678918131 conn(0x7fbd900cb700 msgr2=0x7fbd900cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:21.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.808+0000 7fbd97000640 1 -- 172.21.15.27:0/678918131 shutdown_connections 2024-01-15T18:16:21.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.808+0000 7fbd97000640 1 -- 172.21.15.27:0/678918131 wait complete. 2024-01-15T18:16:21.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.808+0000 7fbd97000640 1 Processor -- start 2024-01-15T18:16:21.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.808+0000 7fbd97000640 1 -- start start 2024-01-15T18:16:21.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.808+0000 7fbd97000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbd900cf110 0x7fbd9014fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:21.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.808+0000 7fbd97000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbd90158900 0x7fbd9015bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:21.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.808+0000 7fbd97000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd9015f550 0x7fbd9015a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:21.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.808+0000 7fbd97000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd900cc810 con 0x7fbd9015f550 2024-01-15T18:16:21.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.808+0000 7fbd97000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd900cd020 con 0x7fbd90158900 2024-01-15T18:16:21.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.808+0000 7fbd97000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd900cca60 con 0x7fbd900cf110 2024-01-15T18:16:21.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.808+0000 7fbd95000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd9015f550 0x7fbd9015a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:21.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.808+0000 7fbd95000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd9015f550 0x7fbd9015a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49936/0 (socket says 172.21.15.27:49936) 2024-01-15T18:16:21.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.808+0000 7fbd95000640 1 -- 172.21.15.27:0/778891153 learned_addr learned my addr 172.21.15.27:0/778891153 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:21.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.812+0000 7fbd8f400640 1 --2- 172.21.15.27:0/778891153 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbd90158900 0x7fbd9015bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:21.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.812+0000 7fbd8fe00640 1 --2- 172.21.15.27:0/778891153 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbd900cf110 0x7fbd9014fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:21.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.812+0000 7fbd95000640 1 -- 172.21.15.27:0/778891153 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbd900cf110 msgr2=0x7fbd9014fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:21.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.812+0000 7fbd95000640 1 --2- 172.21.15.27:0/778891153 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbd900cf110 0x7fbd9014fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:21.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.812+0000 7fbd95000640 1 -- 172.21.15.27:0/778891153 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbd90158900 msgr2=0x7fbd9015bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:21.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.812+0000 7fbd95000640 1 --2- 172.21.15.27:0/778891153 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbd90158900 0x7fbd9015bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:21.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.812+0000 7fbd95000640 1 -- 172.21.15.27:0/778891153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd88009b30 con 0x7fbd9015f550 2024-01-15T18:16:21.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.812+0000 7fbd8f400640 1 --2- 172.21.15.27:0/778891153 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbd90158900 0x7fbd9015bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:16:21.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.812+0000 7fbd95000640 1 --2- 172.21.15.27:0/778891153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd9015f550 0x7fbd9015a1c0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fbd880af040 tx=0x7fbd88019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:21.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.812+0000 7fbd87e00640 1 -- 172.21.15.27:0/778891153 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd880b8670 con 0x7fbd9015f550 2024-01-15T18:16:21.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.812+0000 7fbd87e00640 1 -- 172.21.15.27:0/778891153 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd880b87d0 con 0x7fbd9015f550 2024-01-15T18:16:21.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.812+0000 7fbd87e00640 1 -- 172.21.15.27:0/778891153 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd880b8b60 con 0x7fbd9015f550 2024-01-15T18:16:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.812+0000 7fbd97000640 1 -- 172.21.15.27:0/778891153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd900ccd40 con 0x7fbd9015f550 2024-01-15T18:16:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.812+0000 7fbd97000640 1 -- 172.21.15.27:0/778891153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd901501f0 con 0x7fbd9015f550 2024-01-15T18:16:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.812+0000 7fbd97000640 1 -- 172.21.15.27:0/778891153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd900ac880 con 0x7fbd9015f550 2024-01-15T18:16:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.812+0000 7fbd87e00640 1 -- 172.21.15.27:0/778891153 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 31) v1 ==== 87282+0+0 (secure 0 0 0) 0x7fbd880b8cc0 con 0x7fbd9015f550 2024-01-15T18:16:21.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.812+0000 7fbd87e00640 1 --2- 172.21.15.27:0/778891153 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbd68067550 0x7fbd68069a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:21.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.816+0000 7fbd87e00640 1 -- 172.21.15.27:0/778891153 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fbd8811e7c0 con 0x7fbd9015f550 2024-01-15T18:16:21.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.816+0000 7fbd8fe00640 1 --2- 172.21.15.27:0/778891153 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbd68067550 0x7fbd68069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:21.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.816+0000 7fbd8fe00640 1 --2- 172.21.15.27:0/778891153 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbd68067550 0x7fbd68069a10 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fbd80000f70 tx=0x7fbd80009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:21.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.820+0000 7fbd87e00640 1 -- 172.21.15.27:0/778891153 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbd900ac880 con 0x7fbd9015f550 2024-01-15T18:16:21.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.976+0000 7fbd97000640 1 -- 172.21.15.27:0/778891153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7fbd900ac880 con 0x7fbd9015f550 2024-01-15T18:16:21.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.980+0000 7fbd87e00640 1 -- 172.21.15.27:0/778891153 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v31)=0 v31) v1 ==== 135+0+0 (secure 0 0 0) 0x7fbd900ac880 con 0x7fbd9015f550 2024-01-15T18:16:21.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.984+0000 7fbd97000640 1 -- 172.21.15.27:0/778891153 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbd68067550 msgr2=0x7fbd68069a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:21.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.984+0000 7fbd97000640 1 --2- 172.21.15.27:0/778891153 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbd68067550 0x7fbd68069a10 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fbd80000f70 tx=0x7fbd80009040 comp rx=0 tx=0).stop 2024-01-15T18:16:21.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.984+0000 7fbd97000640 1 -- 172.21.15.27:0/778891153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd9015f550 msgr2=0x7fbd9015a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:21.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.984+0000 7fbd97000640 1 --2- 172.21.15.27:0/778891153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd9015f550 0x7fbd9015a1c0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fbd880af040 tx=0x7fbd88019740 comp rx=0 tx=0).stop 2024-01-15T18:16:21.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.984+0000 7fbd97000640 1 -- 172.21.15.27:0/778891153 shutdown_connections 2024-01-15T18:16:21.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.984+0000 7fbd97000640 1 --2- 172.21.15.27:0/778891153 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbd68067550 0x7fbd68069a10 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:21.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.984+0000 7fbd97000640 1 --2- 172.21.15.27:0/778891153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd9015f550 0x7fbd9015a1c0 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:21.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.984+0000 7fbd97000640 1 --2- 172.21.15.27:0/778891153 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbd90158900 0x7fbd9015bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:21.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.984+0000 7fbd97000640 1 --2- 172.21.15.27:0/778891153 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbd900cf110 0x7fbd9014fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:21.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.984+0000 7fbd97000640 1 -- 172.21.15.27:0/778891153 >> 172.21.15.27:0/778891153 conn(0x7fbd900cb700 msgr2=0x7fbd901544b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:21.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.984+0000 7fbd97000640 1 -- 172.21.15.27:0/778891153 shutdown_connections 2024-01-15T18:16:21.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:21.984+0000 7fbd97000640 1 -- 172.21.15.27:0/778891153 wait complete. 2024-01-15T18:16:22.002 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-15T18:16:22.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.148+0000 7fe709600640 1 Processor -- start 2024-01-15T18:16:22.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.148+0000 7fe709600640 1 -- start start 2024-01-15T18:16:22.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.152+0000 7fe709600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe70415b310 0x7fe7040d1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:22.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.152+0000 7fe709600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe7040afd00 0x7fe7040cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:22.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.152+0000 7fe709600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7040cc770 con 0x7fe7040cddc0 2024-01-15T18:16:22.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.152+0000 7fe709600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe70415b6f0 con 0x7fe7040afd00 2024-01-15T18:16:22.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.152+0000 7fe709600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7040ce250 con 0x7fe70415b310 2024-01-15T18:16:22.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.152+0000 7fe702000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe7040afd00 0x7fe7040cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:22.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.152+0000 7fe702000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe7040afd00 0x7fe7040cd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49940/0 (socket says 172.21.15.27:49940) 2024-01-15T18:16:22.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.152+0000 7fe702000640 1 -- 172.21.15.27:0/1561512774 learned_addr learned my addr 172.21.15.27:0/1561512774 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:22.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.152+0000 7fe702a00640 1 --2- 172.21.15.27:0/1561512774 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe70415b310 0x7fe7040d1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:22.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.152+0000 7fe702000640 1 -- 172.21.15.27:0/1561512774 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe70415b310 msgr2=0x7fe7040d1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:22.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.152+0000 7fe702000640 1 --2- 172.21.15.27:0/1561512774 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe70415b310 0x7fe7040d1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:22.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.152+0000 7fe702000640 1 -- 172.21.15.27:0/1561512774 >> v1:172.21.15.93:6789/0 conn(0x7fe7040cddc0 legacy=0x7fe70415ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:22.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.152+0000 7fe702000640 1 -- 172.21.15.27:0/1561512774 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7040ceb70 con 0x7fe7040afd00 2024-01-15T18:16:22.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.156+0000 7fe702000640 1 --2- 172.21.15.27:0/1561512774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7040afd00 0x7fe7040cd880 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec009870 tx=0x7fe6ec01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=239a07831c5edcaf server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:22.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.156+0000 7fe701600640 1 -- 172.21.15.27:0/1561512774 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec0a8a30 con 0x7fe7040afd00 2024-01-15T18:16:22.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.156+0000 7fe701600640 1 -- 172.21.15.27:0/1561512774 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6ec0a8b90 con 0x7fe7040afd00 2024-01-15T18:16:22.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.156+0000 7fe701600640 1 -- 172.21.15.27:0/1561512774 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec0b0440 con 0x7fe7040afd00 2024-01-15T18:16:22.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.156+0000 7fe709600640 1 -- 172.21.15.27:0/1561512774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7040afd00 msgr2=0x7fe7040cd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:22.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.156+0000 7fe709600640 1 --2- 172.21.15.27:0/1561512774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7040afd00 0x7fe7040cd880 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec009870 tx=0x7fe6ec01ed30 comp rx=0 tx=0).stop 2024-01-15T18:16:22.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.156+0000 7fe709600640 1 -- 172.21.15.27:0/1561512774 shutdown_connections 2024-01-15T18:16:22.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.156+0000 7fe709600640 1 --2- 172.21.15.27:0/1561512774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7040afd00 0x7fe7040cd880 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:22.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.156+0000 7fe709600640 1 --2- 172.21.15.27:0/1561512774 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe70415b310 0x7fe7040d1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:22.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.156+0000 7fe709600640 1 -- 172.21.15.27:0/1561512774 >> 172.21.15.27:0/1561512774 conn(0x7fe7040d0e70 msgr2=0x7fe7040d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:22.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.156+0000 7fe709600640 1 -- 172.21.15.27:0/1561512774 shutdown_connections 2024-01-15T18:16:22.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.156+0000 7fe709600640 1 -- 172.21.15.27:0/1561512774 wait complete. 2024-01-15T18:16:22.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.156+0000 7fe709600640 1 Processor -- start 2024-01-15T18:16:22.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.156+0000 7fe709600640 1 -- start start 2024-01-15T18:16:22.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.160+0000 7fe709600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7040afd00 0x7fe70414b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:22.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.160+0000 7fe709600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7040cddc0 0x7fe7040c7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:22.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.160+0000 7fe709600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe70415b310 0x7fe70414be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:22.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.160+0000 7fe709600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe70406b9d0 con 0x7fe7040cddc0 2024-01-15T18:16:22.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.160+0000 7fe709600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe70406c830 con 0x7fe7040afd00 2024-01-15T18:16:22.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.160+0000 7fe709600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7040cae60 con 0x7fe70415b310 2024-01-15T18:16:22.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.160+0000 7fe702000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7040cddc0 0x7fe7040c7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:22.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.160+0000 7fe702000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7040cddc0 0x7fe7040c7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49944/0 (socket says 172.21.15.27:49944) 2024-01-15T18:16:22.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.160+0000 7fe702000640 1 -- 172.21.15.27:0/2001247525 learned_addr learned my addr 172.21.15.27:0/2001247525 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:22.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.160+0000 7fe702a00640 1 --2- 172.21.15.27:0/2001247525 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7040afd00 0x7fe70414b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:22.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.160+0000 7fe703400640 1 --2- 172.21.15.27:0/2001247525 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe70415b310 0x7fe70414be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:22.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.160+0000 7fe702000640 1 -- 172.21.15.27:0/2001247525 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe70415b310 msgr2=0x7fe70414be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:22.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.160+0000 7fe702000640 1 --2- 172.21.15.27:0/2001247525 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe70415b310 0x7fe70414be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:22.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.160+0000 7fe702000640 1 -- 172.21.15.27:0/2001247525 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7040afd00 msgr2=0x7fe70414b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:22.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.160+0000 7fe702000640 1 --2- 172.21.15.27:0/2001247525 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7040afd00 0x7fe70414b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:22.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.160+0000 7fe702000640 1 -- 172.21.15.27:0/2001247525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6ec0094d0 con 0x7fe7040cddc0 2024-01-15T18:16:22.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.160+0000 7fe702000640 1 --2- 172.21.15.27:0/2001247525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7040cddc0 0x7fe7040c7f40 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec01d3e0 tx=0x7fe6ec01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:22.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.160+0000 7fe6faa00640 1 -- 172.21.15.27:0/2001247525 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec0b0dd0 con 0x7fe7040cddc0 2024-01-15T18:16:22.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.160+0000 7fe6faa00640 1 -- 172.21.15.27:0/2001247525 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6ec0bb460 con 0x7fe7040cddc0 2024-01-15T18:16:22.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.160+0000 7fe709600640 1 -- 172.21.15.27:0/2001247525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe70406bc20 con 0x7fe7040cddc0 2024-01-15T18:16:22.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.160+0000 7fe6faa00640 1 -- 172.21.15.27:0/2001247525 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec0bb810 con 0x7fe7040cddc0 2024-01-15T18:16:22.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.160+0000 7fe709600640 1 -- 172.21.15.27:0/2001247525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe70406c1e0 con 0x7fe7040cddc0 2024-01-15T18:16:22.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.160+0000 7fe709600640 1 -- 172.21.15.27:0/2001247525 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe70404e850 con 0x7fe7040cddc0 2024-01-15T18:16:22.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.164+0000 7fe6faa00640 1 -- 172.21.15.27:0/2001247525 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 31) v1 ==== 87282+0+0 (secure 0 0 0) 0x7fe6ec0b83f0 con 0x7fe7040cddc0 2024-01-15T18:16:22.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.164+0000 7fe6faa00640 1 --2- 172.21.15.27:0/2001247525 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe6dc067550 0x7fe6dc069a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:22.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.164+0000 7fe6faa00640 1 -- 172.21.15.27:0/2001247525 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fe6ec120930 con 0x7fe7040cddc0 2024-01-15T18:16:22.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.164+0000 7fe702a00640 1 --2- 172.21.15.27:0/2001247525 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe6dc067550 0x7fe6dc069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:22.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.164+0000 7fe702a00640 1 --2- 172.21.15.27:0/2001247525 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe6dc067550 0x7fe6dc069a10 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4009b60 tx=0x7fe6f4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:22.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.168+0000 7fe6faa00640 1 -- 172.21.15.27:0/2001247525 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe6ec0ef370 con 0x7fe7040cddc0 2024-01-15T18:16:22.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.352+0000 7fe709600640 1 -- 172.21.15.27:0/2001247525 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7040ccf80 con 0x7fe6dc067550 2024-01-15T18:16:22.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.364+0000 7fe6faa00640 1 -- 172.21.15.27:0/2001247525 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe6f4011ac0 con 0x7fe6dc067550 2024-01-15T18:16:22.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.368+0000 7fe709600640 1 -- 172.21.15.27:0/2001247525 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe6dc067550 msgr2=0x7fe6dc069a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:22.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.368+0000 7fe709600640 1 --2- 172.21.15.27:0/2001247525 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe6dc067550 0x7fe6dc069a10 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4009b60 tx=0x7fe6f4009190 comp rx=0 tx=0).stop 2024-01-15T18:16:22.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.368+0000 7fe709600640 1 -- 172.21.15.27:0/2001247525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7040cddc0 msgr2=0x7fe7040c7f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:22.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.368+0000 7fe709600640 1 --2- 172.21.15.27:0/2001247525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7040cddc0 0x7fe7040c7f40 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec01d3e0 tx=0x7fe6ec01bad0 comp rx=0 tx=0).stop 2024-01-15T18:16:22.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.368+0000 7fe709600640 1 -- 172.21.15.27:0/2001247525 shutdown_connections 2024-01-15T18:16:22.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.368+0000 7fe709600640 1 --2- 172.21.15.27:0/2001247525 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe6dc067550 0x7fe6dc069a10 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:22.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.368+0000 7fe709600640 1 --2- 172.21.15.27:0/2001247525 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe70415b310 0x7fe70414be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:22.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.368+0000 7fe709600640 1 --2- 172.21.15.27:0/2001247525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7040cddc0 0x7fe7040c7f40 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:22.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.368+0000 7fe709600640 1 --2- 172.21.15.27:0/2001247525 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7040afd00 0x7fe70414b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:22.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.368+0000 7fe709600640 1 -- 172.21.15.27:0/2001247525 >> 172.21.15.27:0/2001247525 conn(0x7fe7040d0e70 msgr2=0x7fe7040adae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:22.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.368+0000 7fe709600640 1 -- 172.21.15.27:0/2001247525 shutdown_connections 2024-01-15T18:16:22.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.368+0000 7fe709600640 1 -- 172.21.15.27:0/2001247525 wait complete. 2024-01-15T18:16:22.385 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:16:22.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.532+0000 7f5a35800640 1 Processor -- start 2024-01-15T18:16:22.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.532+0000 7f5a35800640 1 -- start start 2024-01-15T18:16:22.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.532+0000 7f5a35800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5a3015f550 0x7f5a300cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:22.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.532+0000 7f5a35800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a300cf110 0x7f5a300701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:22.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.532+0000 7f5a35800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a300b1cb0 con 0x7f5a30158900 2024-01-15T18:16:22.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.532+0000 7f5a35800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a300cf580 con 0x7f5a3015f550 2024-01-15T18:16:22.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.532+0000 7f5a35800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3015fa00 con 0x7f5a300cf110 2024-01-15T18:16:22.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.532+0000 7f5a2e000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5a3015f550 0x7f5a300cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:22.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.532+0000 7f5a2e000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5a3015f550 0x7f5a300cebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49958/0 (socket says 172.21.15.27:49958) 2024-01-15T18:16:22.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.532+0000 7f5a2e000640 1 -- 172.21.15.27:0/1916861748 learned_addr learned my addr 172.21.15.27:0/1916861748 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:22.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.532+0000 7f5a2f400640 1 --2- 172.21.15.27:0/1916861748 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a300cf110 0x7f5a300701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:22.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.532+0000 7f5a2e000640 1 -- 172.21.15.27:0/1916861748 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a300cf110 msgr2=0x7f5a300701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:22.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.532+0000 7f5a2e000640 1 --2- 172.21.15.27:0/1916861748 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a300cf110 0x7f5a300701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:22.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.532+0000 7f5a2e000640 1 -- 172.21.15.27:0/1916861748 >> v1:172.21.15.93:6789/0 conn(0x7f5a30158900 legacy=0x7f5a3015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:22.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.532+0000 7f5a2e000640 1 -- 172.21.15.27:0/1916861748 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a300b1f00 con 0x7f5a3015f550 2024-01-15T18:16:22.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.536+0000 7f5a2e000640 1 --2- 172.21.15.27:0/1916861748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a3015f550 0x7f5a300cebd0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009870 tx=0x7f5a1801c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d38fec1bcd860b6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:22.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.536+0000 7f5a2d600640 1 -- 172.21.15.27:0/1916861748 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a30070f30 con 0x7f5a3015f550 2024-01-15T18:16:22.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.536+0000 7f5a2d600640 1 -- 172.21.15.27:0/1916861748 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a180aa670 con 0x7f5a3015f550 2024-01-15T18:16:22.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.536+0000 7f5a2d600640 1 -- 172.21.15.27:0/1916861748 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a180aaa00 con 0x7f5a3015f550 2024-01-15T18:16:22.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.536+0000 7f5a35800640 1 -- 172.21.15.27:0/1916861748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a3015f550 msgr2=0x7f5a300cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:22.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.536+0000 7f5a35800640 1 --2- 172.21.15.27:0/1916861748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a3015f550 0x7f5a300cebd0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009870 tx=0x7f5a1801c510 comp rx=0 tx=0).stop 2024-01-15T18:16:22.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.536+0000 7f5a35800640 1 -- 172.21.15.27:0/1916861748 shutdown_connections 2024-01-15T18:16:22.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.536+0000 7f5a35800640 1 --2- 172.21.15.27:0/1916861748 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a300cf110 0x7f5a300701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:22.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.536+0000 7f5a35800640 1 --2- 172.21.15.27:0/1916861748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a3015f550 0x7f5a300cebd0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:22.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.536+0000 7f5a35800640 1 -- 172.21.15.27:0/1916861748 >> 172.21.15.27:0/1916861748 conn(0x7f5a300cb700 msgr2=0x7f5a300cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:22.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.536+0000 7f5a35800640 1 -- 172.21.15.27:0/1916861748 shutdown_connections 2024-01-15T18:16:22.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.536+0000 7f5a35800640 1 -- 172.21.15.27:0/1916861748 wait complete. 2024-01-15T18:16:22.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.540+0000 7f5a35800640 1 Processor -- start 2024-01-15T18:16:22.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.540+0000 7f5a35800640 1 -- start start 2024-01-15T18:16:22.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.540+0000 7f5a35800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a300cf110 0x7f5a300adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:22.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.540+0000 7f5a35800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a30158900 0x7f5a3013ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:22.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.540+0000 7f5a35800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a3015f550 0x7f5a30151480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:22.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.540+0000 7f5a35800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3013cc00 con 0x7f5a3015f550 2024-01-15T18:16:22.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.540+0000 7f5a35800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a300ae2e0 con 0x7f5a300cf110 2024-01-15T18:16:22.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.540+0000 7f5a35800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3013d700 con 0x7f5a30158900 2024-01-15T18:16:22.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.540+0000 7f5a2f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a3015f550 0x7f5a30151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:22.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.540+0000 7f5a2ea00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a300cf110 0x7f5a300adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:22.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.540+0000 7f5a2e000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a30158900 0x7f5a3013ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:22.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.540+0000 7f5a2f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a3015f550 0x7f5a30151480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49968/0 (socket says 172.21.15.27:49968) 2024-01-15T18:16:22.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.540+0000 7f5a2f400640 1 -- 172.21.15.27:0/932133804 learned_addr learned my addr 172.21.15.27:0/932133804 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:22.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.540+0000 7f5a2f400640 1 -- 172.21.15.27:0/932133804 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a30158900 msgr2=0x7f5a3013ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:22.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.540+0000 7f5a2f400640 1 --2- 172.21.15.27:0/932133804 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a30158900 0x7f5a3013ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:22.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.540+0000 7f5a2f400640 1 -- 172.21.15.27:0/932133804 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a300cf110 msgr2=0x7f5a300adda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:22.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.540+0000 7f5a2f400640 1 --2- 172.21.15.27:0/932133804 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a300cf110 0x7f5a300adda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:22.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.540+0000 7f5a2f400640 1 -- 172.21.15.27:0/932133804 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a180094d0 con 0x7f5a3015f550 2024-01-15T18:16:22.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.540+0000 7f5a2f400640 1 --2- 172.21.15.27:0/932133804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a3015f550 0x7f5a30151480 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f5a2800efc0 tx=0x7f5a2800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:22.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.540+0000 7f5a26a00640 1 -- 172.21.15.27:0/932133804 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28014480 con 0x7f5a3015f550 2024-01-15T18:16:22.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.540+0000 7f5a26a00640 1 -- 172.21.15.27:0/932133804 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a2800f040 con 0x7f5a3015f550 2024-01-15T18:16:22.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.540+0000 7f5a26a00640 1 -- 172.21.15.27:0/932133804 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28014820 con 0x7f5a3015f550 2024-01-15T18:16:22.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.540+0000 7f5a35800640 1 -- 172.21.15.27:0/932133804 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a3013d9e0 con 0x7f5a3015f550 2024-01-15T18:16:22.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.544+0000 7f5a35800640 1 -- 172.21.15.27:0/932133804 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a3013dfa0 con 0x7f5a3015f550 2024-01-15T18:16:22.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.544+0000 7f5a35800640 1 -- 172.21.15.27:0/932133804 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a3013b670 con 0x7f5a3015f550 2024-01-15T18:16:22.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.544+0000 7f5a26a00640 1 -- 172.21.15.27:0/932133804 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 31) v1 ==== 87282+0+0 (secure 0 0 0) 0x7f5a28014980 con 0x7f5a3015f550 2024-01-15T18:16:22.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.544+0000 7f5a26a00640 1 --2- 172.21.15.27:0/932133804 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5a08067550 0x7f5a08069a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:22.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.544+0000 7f5a26a00640 1 -- 172.21.15.27:0/932133804 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f5a280839d0 con 0x7f5a3015f550 2024-01-15T18:16:22.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.544+0000 7f5a2ea00640 1 --2- 172.21.15.27:0/932133804 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5a08067550 0x7f5a08069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:22.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.548+0000 7f5a2ea00640 1 --2- 172.21.15.27:0/932133804 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5a08067550 0x7f5a08069a10 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f5a20001810 tx=0x7f5a20009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:22.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.548+0000 7f5a26a00640 1 -- 172.21.15.27:0/932133804 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5a3013b670 con 0x7f5a3015f550 2024-01-15T18:16:22.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.732+0000 7f5a35800640 1 -- 172.21.15.27:0/932133804 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a300733c0 con 0x7f5a08067550 2024-01-15T18:16:22.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.736+0000 7f5a26a00640 1 -- 172.21.15.27:0/932133804 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f5a300733c0 con 0x7f5a08067550 2024-01-15T18:16:22.734 INFO:teuthology.orchestra.run.smithi027.stdout:/volumes/_nogroup/sv1/4482b5e7-74d5-4ce9-8b87-3b51c5526f5f 2024-01-15T18:16:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.740+0000 7f5a35800640 1 -- 172.21.15.27:0/932133804 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5a08067550 msgr2=0x7f5a08069a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.740+0000 7f5a35800640 1 --2- 172.21.15.27:0/932133804 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5a08067550 0x7f5a08069a10 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f5a20001810 tx=0x7f5a20009040 comp rx=0 tx=0).stop 2024-01-15T18:16:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.740+0000 7f5a35800640 1 -- 172.21.15.27:0/932133804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a3015f550 msgr2=0x7f5a30151480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.740+0000 7f5a35800640 1 --2- 172.21.15.27:0/932133804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a3015f550 0x7f5a30151480 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f5a2800efc0 tx=0x7f5a2800c490 comp rx=0 tx=0).stop 2024-01-15T18:16:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.740+0000 7f5a35800640 1 -- 172.21.15.27:0/932133804 shutdown_connections 2024-01-15T18:16:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.740+0000 7f5a35800640 1 --2- 172.21.15.27:0/932133804 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5a08067550 0x7f5a08069a10 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.740+0000 7f5a35800640 1 --2- 172.21.15.27:0/932133804 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a3015f550 0x7f5a30151480 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.740+0000 7f5a35800640 1 --2- 172.21.15.27:0/932133804 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5a30158900 0x7f5a3013ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.740+0000 7f5a35800640 1 --2- 172.21.15.27:0/932133804 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5a300cf110 0x7f5a300adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.740+0000 7f5a35800640 1 -- 172.21.15.27:0/932133804 >> 172.21.15.27:0/932133804 conn(0x7f5a300cb700 msgr2=0x7f5a300af3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.740+0000 7f5a35800640 1 -- 172.21.15.27:0/932133804 shutdown_connections 2024-01-15T18:16:22.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:22.740+0000 7f5a35800640 1 -- 172.21.15.27:0/932133804 wait complete. 2024-01-15T18:16:22.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:16:22.756 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/4482b5e7-74d5-4ce9-8b87-3b51c5526f5f) 2024-01-15T18:16:22.770 INFO:teuthology.orchestra.run.smithi027.stdout:# file: volumes/_nogroup/sv1/4482b5e7-74d5-4ce9-8b87-3b51c5526f5f 2024-01-15T18:16:22.770 INFO:teuthology.orchestra.run.smithi027.stdout:ceph.dir.rbytes="3138256896" 2024-01-15T18:16:22.770 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:16:22.771 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-15T18:16:24.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:16:24.772 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/4482b5e7-74d5-4ce9-8b87-3b51c5526f5f) 2024-01-15T18:16:24.787 INFO:teuthology.orchestra.run.smithi027.stdout:# file: volumes/_nogroup/sv1/4482b5e7-74d5-4ce9-8b87-3b51c5526f5f 2024-01-15T18:16:24.787 INFO:teuthology.orchestra.run.smithi027.stdout:ceph.dir.rbytes="3138256896" 2024-01-15T18:16:24.787 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:16:24.788 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-15T18:16:26.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:16:26.790 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/4482b5e7-74d5-4ce9-8b87-3b51c5526f5f) 2024-01-15T18:16:26.805 INFO:teuthology.orchestra.run.smithi027.stdout:# file: volumes/_nogroup/sv1/4482b5e7-74d5-4ce9-8b87-3b51c5526f5f 2024-01-15T18:16:26.805 INFO:teuthology.orchestra.run.smithi027.stdout:ceph.dir.rbytes="3221225472" 2024-01-15T18:16:26.805 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:16:26.806 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-15T18:16:26.806 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 ss1clone1 2024-01-15T18:16:27.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.007+0000 7fe71c400640 1 Processor -- start 2024-01-15T18:16:27.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.007+0000 7fe71c400640 1 -- start start 2024-01-15T18:16:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.007+0000 7fe71c400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe7140cd070 0x7fe714168060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.007+0000 7fe71c400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7140ad920 con 0x7fe7141468d0 2024-01-15T18:16:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.007+0000 7fe71c400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7140ac880 con 0x7fe7140cf100 2024-01-15T18:16:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.007+0000 7fe71c400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7140cd4c0 con 0x7fe7140cd070 2024-01-15T18:16:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.007+0000 7fe71a400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe7140cd070 0x7fe714168060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.007+0000 7fe719a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe7141468d0 0x7fe7140ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58452/0 (socket says 172.21.15.27:58452) 2024-01-15T18:16:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.007+0000 7fe719a00640 1 -- 172.21.15.27:0/588538171 learned_addr learned my addr 172.21.15.27:0/588538171 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:16:27.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.007+0000 7fe71a400640 1 -- 172.21.15.27:0/588538171 >> v1:172.21.15.93:6789/0 conn(0x7fe7140cf100 legacy=0x7fe7140cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:27.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.007+0000 7fe71a400640 1 -- 172.21.15.27:0/588538171 >> v1:172.21.15.27:6789/0 conn(0x7fe7141468d0 legacy=0x7fe7140ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:27.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.007+0000 7fe71a400640 1 -- 172.21.15.27:0/588538171 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7140acb60 con 0x7fe7140cd070 2024-01-15T18:16:27.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.011+0000 7fe71a400640 1 --2- 172.21.15.27:0/588538171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7140cd070 0x7fe714168060 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fe70c009d90 tx=0x7fe70c01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e46f71f58f8b4deb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:27.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.011+0000 7fe713e00640 1 -- 172.21.15.27:0/588538171 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c0a59a0 con 0x7fe7140cd070 2024-01-15T18:16:27.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.011+0000 7fe713e00640 1 -- 172.21.15.27:0/588538171 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70c0a5b00 con 0x7fe7140cd070 2024-01-15T18:16:27.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.011+0000 7fe713e00640 1 -- 172.21.15.27:0/588538171 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c0a5e90 con 0x7fe7140cd070 2024-01-15T18:16:27.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.011+0000 7fe71c400640 1 -- 172.21.15.27:0/588538171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7140cd070 msgr2=0x7fe714168060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:27.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.011+0000 7fe71c400640 1 --2- 172.21.15.27:0/588538171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7140cd070 0x7fe714168060 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fe70c009d90 tx=0x7fe70c01c8e0 comp rx=0 tx=0).stop 2024-01-15T18:16:27.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.011+0000 7fe71c400640 1 -- 172.21.15.27:0/588538171 shutdown_connections 2024-01-15T18:16:27.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.011+0000 7fe71c400640 1 --2- 172.21.15.27:0/588538171 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7140cd070 0x7fe714168060 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:27.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.011+0000 7fe71c400640 1 -- 172.21.15.27:0/588538171 >> 172.21.15.27:0/588538171 conn(0x7fe7140cb700 msgr2=0x7fe7140cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.011+0000 7fe71c400640 1 -- 172.21.15.27:0/588538171 shutdown_connections 2024-01-15T18:16:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.011+0000 7fe71c400640 1 -- 172.21.15.27:0/588538171 wait complete. 2024-01-15T18:16:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe71c400640 1 Processor -- start 2024-01-15T18:16:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe71c400640 1 -- start start 2024-01-15T18:16:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe71c400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe7140cd070 0x7fe71415d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe71c400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7140cf100 0x7fe7140b0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe71c400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7141468d0 0x7fe7141419a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe71c400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71413d8e0 con 0x7fe7141468d0 2024-01-15T18:16:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe71c400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71413d590 con 0x7fe7140cf100 2024-01-15T18:16:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe71c400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe714160280 con 0x7fe7140cd070 2024-01-15T18:16:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe71a400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7141468d0 0x7fe7141419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe71a400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7141468d0 0x7fe7141419a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52960/0 (socket says 172.21.15.27:52960) 2024-01-15T18:16:27.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe71a400640 1 -- 172.21.15.27:0/3841630565 learned_addr learned my addr 172.21.15.27:0/3841630565 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:27.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe719a00640 1 --2- 172.21.15.27:0/3841630565 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe7140cd070 0x7fe71415d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:27.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe719000640 1 --2- 172.21.15.27:0/3841630565 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7140cf100 0x7fe7140b0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:27.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe71a400640 1 -- 172.21.15.27:0/3841630565 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe7140cd070 msgr2=0x7fe71415d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:27.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe71a400640 1 --2- 172.21.15.27:0/3841630565 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe7140cd070 0x7fe71415d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:27.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe71a400640 1 -- 172.21.15.27:0/3841630565 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7140cf100 msgr2=0x7fe7140b0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:27.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe71a400640 1 --2- 172.21.15.27:0/3841630565 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7140cf100 0x7fe7140b0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:27.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe71a400640 1 -- 172.21.15.27:0/3841630565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe70c009a70 con 0x7fe7141468d0 2024-01-15T18:16:27.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe719a00640 1 --2- 172.21.15.27:0/3841630565 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe7140cd070 0x7fe71415d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:16:27.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe719000640 1 --2- 172.21.15.27:0/3841630565 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7140cf100 0x7fe7140b0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:16:27.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe71a400640 1 --2- 172.21.15.27:0/3841630565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7141468d0 0x7fe7141419a0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fe70c01a510 tx=0x7fe70c019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:27.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe711600640 1 -- 172.21.15.27:0/3841630565 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c0ade40 con 0x7fe7141468d0 2024-01-15T18:16:27.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe711600640 1 -- 172.21.15.27:0/3841630565 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70c0b9450 con 0x7fe7141468d0 2024-01-15T18:16:27.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe711600640 1 -- 172.21.15.27:0/3841630565 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c0b97e0 con 0x7fe7141468d0 2024-01-15T18:16:27.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe71c400640 1 -- 172.21.15.27:0/3841630565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe714160560 con 0x7fe7141468d0 2024-01-15T18:16:27.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.015+0000 7fe71c400640 1 -- 172.21.15.27:0/3841630565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7140b1130 con 0x7fe7141468d0 2024-01-15T18:16:27.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.019+0000 7fe71c400640 1 -- 172.21.15.27:0/3841630565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe71413da50 con 0x7fe7141468d0 2024-01-15T18:16:27.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.019+0000 7fe711600640 1 -- 172.21.15.27:0/3841630565 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 31) v1 ==== 87282+0+0 (secure 0 0 0) 0x7fe70c0b9940 con 0x7fe7141468d0 2024-01-15T18:16:27.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.019+0000 7fe711600640 1 --2- 172.21.15.27:0/3841630565 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe6ec067550 0x7fe6ec069a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:27.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.019+0000 7fe711600640 1 -- 172.21.15.27:0/3841630565 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fe70c11ea80 con 0x7fe7141468d0 2024-01-15T18:16:27.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.019+0000 7fe719a00640 1 --2- 172.21.15.27:0/3841630565 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe6ec067550 0x7fe6ec069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:27.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.019+0000 7fe719a00640 1 --2- 172.21.15.27:0/3841630565 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe6ec067550 0x7fe6ec069a10 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe704005890 tx=0x7fe704009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:27.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.023+0000 7fe711600640 1 -- 172.21.15.27:0/3841630565 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe70c0ed550 con 0x7fe7141468d0 2024-01-15T18:16:27.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.215+0000 7fe71c400640 1 -- 172.21.15.27:0/3841630565 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe71413d410 con 0x7fe6ec067550 2024-01-15T18:16:27.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.271+0000 7fe711600640 1 -- 172.21.15.27:0/3841630565 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe704001c50 con 0x7fe6ec067550 2024-01-15T18:16:27.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.271+0000 7fe71c400640 1 -- 172.21.15.27:0/3841630565 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe6ec067550 msgr2=0x7fe6ec069a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:27.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.271+0000 7fe71c400640 1 --2- 172.21.15.27:0/3841630565 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe6ec067550 0x7fe6ec069a10 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe704005890 tx=0x7fe704009040 comp rx=0 tx=0).stop 2024-01-15T18:16:27.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.271+0000 7fe71c400640 1 -- 172.21.15.27:0/3841630565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7141468d0 msgr2=0x7fe7141419a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:27.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.271+0000 7fe71c400640 1 --2- 172.21.15.27:0/3841630565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7141468d0 0x7fe7141419a0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fe70c01a510 tx=0x7fe70c019680 comp rx=0 tx=0).stop 2024-01-15T18:16:27.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.271+0000 7fe71c400640 1 -- 172.21.15.27:0/3841630565 shutdown_connections 2024-01-15T18:16:27.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.271+0000 7fe71c400640 1 --2- 172.21.15.27:0/3841630565 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe6ec067550 0x7fe6ec069a10 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:27.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.271+0000 7fe71c400640 1 --2- 172.21.15.27:0/3841630565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7141468d0 0x7fe7141419a0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:27.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.271+0000 7fe71c400640 1 --2- 172.21.15.27:0/3841630565 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe7140cf100 0x7fe7140b0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:27.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.271+0000 7fe71c400640 1 --2- 172.21.15.27:0/3841630565 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe7140cd070 0x7fe71415d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:27.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.271+0000 7fe71c400640 1 -- 172.21.15.27:0/3841630565 >> 172.21.15.27:0/3841630565 conn(0x7fe7140cb700 msgr2=0x7fe71415c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:27.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.275+0000 7fe71c400640 1 -- 172.21.15.27:0/3841630565 shutdown_connections 2024-01-15T18:16:27.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.275+0000 7fe71c400640 1 -- 172.21.15.27:0/3841630565 wait complete. 2024-01-15T18:16:27.277 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:16:27.276+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:16:27.277 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:16:27.276+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:16:27.278 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:16:27.276+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:16:27.278 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:16:27.276+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:16:27.278 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:16:27.276+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:16:27.293 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-15T18:16:27.293 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:16:27.295 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:16:27.292+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:16:27.295 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:16:27.292+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:16:27.295 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:16:27.292+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:16:27.295 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:16:27.292+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:16:27.295 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:16:27.292+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:16:27.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.439+0000 7f0045800640 1 Processor -- start 2024-01-15T18:16:27.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.439+0000 7f0045800640 1 -- start start 2024-01-15T18:16:27.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.439+0000 7f0045800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f004015d3b0 0x7f00400cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:27.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.439+0000 7f0045800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f004006c3f0 0x7f00400b1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:27.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.439+0000 7f0045800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00400b04b0 con 0x7f004006b600 2024-01-15T18:16:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.439+0000 7f0045800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00400d0160 con 0x7f004006c3f0 2024-01-15T18:16:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.439+0000 7f0045800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00400b0720 con 0x7f004015d3b0 2024-01-15T18:16:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.439+0000 7f003f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f004006c3f0 0x7f00400b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.439+0000 7f003e000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f004006b600 0x7f004006bce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58454/0 (socket says 172.21.15.27:58454) 2024-01-15T18:16:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.439+0000 7f003e000640 1 -- 172.21.15.27:0/1328571184 learned_addr learned my addr 172.21.15.27:0/1328571184 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:16:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.439+0000 7f003ea00640 1 --2- 172.21.15.27:0/1328571184 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f004015d3b0 0x7f00400cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.443+0000 7f003f400640 1 -- 172.21.15.27:0/1328571184 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f004015d3b0 msgr2=0x7f00400cfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.443+0000 7f003f400640 1 --2- 172.21.15.27:0/1328571184 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f004015d3b0 0x7f00400cfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.443+0000 7f003f400640 1 -- 172.21.15.27:0/1328571184 >> v1:172.21.15.27:6789/0 conn(0x7f004006b600 legacy=0x7f004006bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.443+0000 7f003f400640 1 -- 172.21.15.27:0/1328571184 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00400b0a00 con 0x7f004006c3f0 2024-01-15T18:16:27.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.443+0000 7f003d600640 1 -- 172.21.15.27:0/1328571184 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293645304 0 0) 0x7f00400b04b0 con 0x7f004006b600 2024-01-15T18:16:27.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.443+0000 7f003f400640 1 --2- 172.21.15.27:0/1328571184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f004006c3f0 0x7f00400b1200 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f0038009d90 tx=0x7f003801c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=656efb129c90bb23 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:27.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.443+0000 7f003d600640 1 -- 172.21.15.27:0/1328571184 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00380a5980 con 0x7f004006c3f0 2024-01-15T18:16:27.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.443+0000 7f003d600640 1 -- 172.21.15.27:0/1328571184 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00380a5ae0 con 0x7f004006c3f0 2024-01-15T18:16:27.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.443+0000 7f003d600640 1 -- 172.21.15.27:0/1328571184 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00380a5e70 con 0x7f004006c3f0 2024-01-15T18:16:27.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.443+0000 7f0045800640 1 -- 172.21.15.27:0/1328571184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f004006c3f0 msgr2=0x7f00400b1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:27.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.443+0000 7f0045800640 1 --2- 172.21.15.27:0/1328571184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f004006c3f0 0x7f00400b1200 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f0038009d90 tx=0x7f003801c920 comp rx=0 tx=0).stop 2024-01-15T18:16:27.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.447+0000 7f0045800640 1 -- 172.21.15.27:0/1328571184 shutdown_connections 2024-01-15T18:16:27.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.447+0000 7f0045800640 1 --2- 172.21.15.27:0/1328571184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f004006c3f0 0x7f00400b1200 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:27.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.447+0000 7f0045800640 1 --2- 172.21.15.27:0/1328571184 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f004015d3b0 0x7f00400cfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:27.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.447+0000 7f0045800640 1 -- 172.21.15.27:0/1328571184 >> 172.21.15.27:0/1328571184 conn(0x7f00400d0ed0 msgr2=0x7f00400d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:27.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.447+0000 7f0045800640 1 -- 172.21.15.27:0/1328571184 shutdown_connections 2024-01-15T18:16:27.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.447+0000 7f0045800640 1 -- 172.21.15.27:0/1328571184 wait complete. 2024-01-15T18:16:27.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.447+0000 7f0045800640 1 Processor -- start 2024-01-15T18:16:27.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.447+0000 7f0045800640 1 -- start start 2024-01-15T18:16:27.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.447+0000 7f0045800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f004006b600 0x7f0040158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:27.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.447+0000 7f0045800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f004006c3f0 0x7f0040158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:27.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.447+0000 7f0045800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f004015d3b0 0x7f00400b5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:27.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.447+0000 7f0045800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f004009d630 con 0x7f004006b600 2024-01-15T18:16:27.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.447+0000 7f0045800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f004009dd10 con 0x7f004015d3b0 2024-01-15T18:16:27.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.447+0000 7f0045800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0040158c90 con 0x7f004006c3f0 2024-01-15T18:16:27.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.447+0000 7f003ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f004006b600 0x7f0040158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:27.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.447+0000 7f003ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f004006b600 0x7f0040158750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52964/0 (socket says 172.21.15.27:52964) 2024-01-15T18:16:27.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.447+0000 7f003ea00640 1 -- 172.21.15.27:0/2124513202 learned_addr learned my addr 172.21.15.27:0/2124513202 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:27.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.447+0000 7f003e000640 1 --2- 172.21.15.27:0/2124513202 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f004006c3f0 0x7f0040158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:27.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.447+0000 7f003f400640 1 --2- 172.21.15.27:0/2124513202 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f004015d3b0 0x7f00400b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:27.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.447+0000 7f003ea00640 1 -- 172.21.15.27:0/2124513202 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f004006c3f0 msgr2=0x7f0040158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:27.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.447+0000 7f003ea00640 1 --2- 172.21.15.27:0/2124513202 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f004006c3f0 0x7f0040158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:27.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.447+0000 7f003ea00640 1 -- 172.21.15.27:0/2124513202 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f004015d3b0 msgr2=0x7f00400b5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:27.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.447+0000 7f003ea00640 1 --2- 172.21.15.27:0/2124513202 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f004015d3b0 0x7f00400b5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:27.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.447+0000 7f003ea00640 1 -- 172.21.15.27:0/2124513202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0038009a70 con 0x7f004006b600 2024-01-15T18:16:27.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.451+0000 7f003ea00640 1 --2- 172.21.15.27:0/2124513202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f004006b600 0x7f0040158750 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f003000e870 tx=0x7f003000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:27.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.451+0000 7f0036a00640 1 -- 172.21.15.27:0/2124513202 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0030009510 con 0x7f004006b600 2024-01-15T18:16:27.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.451+0000 7f0036a00640 1 -- 172.21.15.27:0/2124513202 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f003000eec0 con 0x7f004006b600 2024-01-15T18:16:27.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.451+0000 7f0036a00640 1 -- 172.21.15.27:0/2124513202 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f003000cb40 con 0x7f004006b600 2024-01-15T18:16:27.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.451+0000 7f0045800640 1 -- 172.21.15.27:0/2124513202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00400b37e0 con 0x7f004006b600 2024-01-15T18:16:27.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.451+0000 7f0045800640 1 -- 172.21.15.27:0/2124513202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00400b3da0 con 0x7f004006b600 2024-01-15T18:16:27.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.451+0000 7f0045800640 1 -- 172.21.15.27:0/2124513202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f004006d550 con 0x7f004006b600 2024-01-15T18:16:27.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.451+0000 7f0036a00640 1 -- 172.21.15.27:0/2124513202 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 31) v1 ==== 87282+0+0 (secure 0 0 0) 0x7f003000cca0 con 0x7f004006b600 2024-01-15T18:16:27.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.451+0000 7f0036a00640 1 --2- 172.21.15.27:0/2124513202 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0018067550 0x7f0018069a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:27.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.451+0000 7f0036a00640 1 -- 172.21.15.27:0/2124513202 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f00300830f0 con 0x7f004006b600 2024-01-15T18:16:27.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.451+0000 7f003e000640 1 --2- 172.21.15.27:0/2124513202 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0018067550 0x7f0018069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:27.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.455+0000 7f003e000640 1 --2- 172.21.15.27:0/2124513202 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0018067550 0x7f0018069a10 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f00400c7f50 tx=0x7f0028007d60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:27.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.455+0000 7f0036a00640 1 -- 172.21.15.27:0/2124513202 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0030051bb0 con 0x7f004006b600 2024-01-15T18:16:27.608 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:16:27.612 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:16:27.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.647+0000 7f0045800640 1 -- 172.21.15.27:0/2124513202 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f004009de80 con 0x7f0018067550 2024-01-15T18:16:27.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.655+0000 7f0036a00640 1 -- 172.21.15.27:0/2124513202 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f004009de80 con 0x7f0018067550 2024-01-15T18:16:27.652 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:16:27.653 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:16:27.653 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "pending", 2024-01-15T18:16:27.653 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:16:27.653 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:16:27.653 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:16:27.653 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:16:27.653 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:27.653 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:27.653 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:16:27.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.655+0000 7f0045800640 1 -- 172.21.15.27:0/2124513202 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0018067550 msgr2=0x7f0018069a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:27.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.655+0000 7f0045800640 1 --2- 172.21.15.27:0/2124513202 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0018067550 0x7f0018069a10 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f00400c7f50 tx=0x7f0028007d60 comp rx=0 tx=0).stop 2024-01-15T18:16:27.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.655+0000 7f0045800640 1 -- 172.21.15.27:0/2124513202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f004006b600 msgr2=0x7f0040158750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:27.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.655+0000 7f0045800640 1 --2- 172.21.15.27:0/2124513202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f004006b600 0x7f0040158750 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f003000e870 tx=0x7f003000ec70 comp rx=0 tx=0).stop 2024-01-15T18:16:27.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.655+0000 7f0045800640 1 -- 172.21.15.27:0/2124513202 shutdown_connections 2024-01-15T18:16:27.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.655+0000 7f0045800640 1 --2- 172.21.15.27:0/2124513202 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0018067550 0x7f0018069a10 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:27.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.655+0000 7f0045800640 1 --2- 172.21.15.27:0/2124513202 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f004015d3b0 0x7f00400b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:27.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.655+0000 7f0045800640 1 --2- 172.21.15.27:0/2124513202 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f004006c3f0 0x7f0040158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:27.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.655+0000 7f0045800640 1 --2- 172.21.15.27:0/2124513202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f004006b600 0x7f0040158750 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:27.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.655+0000 7f0045800640 1 -- 172.21.15.27:0/2124513202 >> 172.21.15.27:0/2124513202 conn(0x7f00400d0ed0 msgr2=0x7f00400c5320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:27.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.655+0000 7f0045800640 1 -- 172.21.15.27:0/2124513202 shutdown_connections 2024-01-15T18:16:27.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:27.655+0000 7f0045800640 1 -- 172.21.15.27:0/2124513202 wait complete. 2024-01-15T18:16:27.675 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-15T18:16:28.676 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:16:28.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.831+0000 7fd471600640 1 Processor -- start 2024-01-15T18:16:28.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.831+0000 7fd471600640 1 -- start start 2024-01-15T18:16:28.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd471600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fd46c0cf2e0 0x7fd46c0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:28.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd471600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd46c0b1cb0 con 0x7fd46c158900 2024-01-15T18:16:28.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd471600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd46c15fa00 con 0x7fd46c15f550 2024-01-15T18:16:28.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd471600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd46c0b1f00 con 0x7fd46c0cf2e0 2024-01-15T18:16:28.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd46aa00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd46c158900 0x7fd46c15bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58462/0 (socket says 172.21.15.27:58462) 2024-01-15T18:16:28.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd46aa00640 1 -- 172.21.15.27:0/3052159895 learned_addr learned my addr 172.21.15.27:0/3052159895 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:16:28.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd46b400640 1 --2- 172.21.15.27:0/3052159895 >> v2:172.21.15.93:3300/0 conn(0x7fd46c0cf2e0 0x7fd46c0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:28.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd469600640 1 -- 172.21.15.27:0/3052159895 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251560419 0 0) 0x7fd46c0b1cb0 con 0x7fd46c158900 2024-01-15T18:16:28.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd469600640 1 -- 172.21.15.27:0/3052159895 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd44c003530 con 0x7fd46c158900 2024-01-15T18:16:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd469600640 1 -- 172.21.15.27:0/3052159895 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2950109741 0 0) 0x7fd44c003530 con 0x7fd46c158900 2024-01-15T18:16:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd469600640 1 -- 172.21.15.27:0/3052159895 >> v2:172.21.15.93:3300/0 conn(0x7fd46c0cf2e0 msgr2=0x7fd46c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd469600640 1 --2- 172.21.15.27:0/3052159895 >> v2:172.21.15.93:3300/0 conn(0x7fd46c0cf2e0 0x7fd46c0701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd469600640 1 -- 172.21.15.27:0/3052159895 >> v1:172.21.15.93:6789/0 conn(0x7fd46c15f550 legacy=0x7fd46c0cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd469600640 1 -- 172.21.15.27:0/3052159895 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd46c0b21e0 con 0x7fd46c158900 2024-01-15T18:16:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd469600640 1 -- 172.21.15.27:0/3052159895 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fd45c001680 con 0x7fd46c158900 2024-01-15T18:16:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd469600640 1 -- 172.21.15.27:0/3052159895 >> v1:172.21.15.27:6789/0 conn(0x7fd46c158900 legacy=0x7fd46c15bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd469600640 1 --2- 172.21.15.27:0/3052159895 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd44c0039f0 0x7fd44c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd469600640 1 --2- 172.21.15.27:0/3052159895 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd44c0047d0 0x7fd44c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd469600640 1 --2- 172.21.15.27:0/3052159895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd44c005190 0x7fd44c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd469600640 1 -- 172.21.15.27:0/3052159895 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c0b1f00 con 0x7fd44c005190 2024-01-15T18:16:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd469600640 1 -- 172.21.15.27:0/3052159895 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c00c010 con 0x7fd44c0047d0 2024-01-15T18:16:28.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd469600640 1 -- 172.21.15.27:0/3052159895 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c00c1d0 con 0x7fd44c0039f0 2024-01-15T18:16:28.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd46b400640 1 --2- 172.21.15.27:0/3052159895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd44c005190 0x7fd44c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:28.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd46aa00640 1 --2- 172.21.15.27:0/3052159895 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd44c0039f0 0x7fd44c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:28.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd46a000640 1 --2- 172.21.15.27:0/3052159895 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd44c0047d0 0x7fd44c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:28.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd46b400640 1 -- 172.21.15.27:0/3052159895 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd44c0039f0 msgr2=0x7fd44c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:28.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd46b400640 1 --2- 172.21.15.27:0/3052159895 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd44c0039f0 0x7fd44c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:28.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd46b400640 1 -- 172.21.15.27:0/3052159895 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd44c0047d0 msgr2=0x7fd44c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:28.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd46b400640 1 --2- 172.21.15.27:0/3052159895 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd44c0047d0 0x7fd44c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:28.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.835+0000 7fd46b400640 1 -- 172.21.15.27:0/3052159895 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd44c003530 con 0x7fd44c005190 2024-01-15T18:16:28.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.839+0000 7fd46b400640 1 -- 172.21.15.27:0/3052159895 reap_dead start 2024-01-15T18:16:28.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.839+0000 7fd46b400640 1 --2- 172.21.15.27:0/3052159895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd44c005190 0x7fd44c00b990 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fd464009f90 tx=0x7fd46401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fd666f14aa9b565 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:28.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.839+0000 7fd469600640 1 -- 172.21.15.27:0/3052159895 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4640a5c70 con 0x7fd44c005190 2024-01-15T18:16:28.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.839+0000 7fd469600640 1 -- 172.21.15.27:0/3052159895 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd46400f040 con 0x7fd44c005190 2024-01-15T18:16:28.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.839+0000 7fd469600640 1 -- 172.21.15.27:0/3052159895 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4640039a0 con 0x7fd44c005190 2024-01-15T18:16:28.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.839+0000 7fd471600640 1 -- 172.21.15.27:0/3052159895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd44c005190 msgr2=0x7fd44c00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:28.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.839+0000 7fd471600640 1 --2- 172.21.15.27:0/3052159895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd44c005190 0x7fd44c00b990 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fd464009f90 tx=0x7fd46401cc60 comp rx=0 tx=0).stop 2024-01-15T18:16:28.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.839+0000 7fd471600640 1 -- 172.21.15.27:0/3052159895 shutdown_connections 2024-01-15T18:16:28.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.839+0000 7fd471600640 1 --2- 172.21.15.27:0/3052159895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd44c005190 0x7fd44c00b990 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:28.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.839+0000 7fd471600640 1 -- 172.21.15.27:0/3052159895 >> 172.21.15.27:0/3052159895 conn(0x7fd46c0cb700 msgr2=0x7fd46c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:28.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.839+0000 7fd471600640 1 -- 172.21.15.27:0/3052159895 shutdown_connections 2024-01-15T18:16:28.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.839+0000 7fd471600640 1 -- 172.21.15.27:0/3052159895 wait complete. 2024-01-15T18:16:28.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.843+0000 7fd471600640 1 Processor -- start 2024-01-15T18:16:28.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.843+0000 7fd471600640 1 -- start start 2024-01-15T18:16:28.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.843+0000 7fd471600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd44c005190 0x7fd46c135d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:28.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.843+0000 7fd471600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd46c13df90 0x7fd46c13e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:28.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.843+0000 7fd471600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd46c13adc0 0x7fd46c13b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:28.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.843+0000 7fd471600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c15c3c0 con 0x7fd46c13df90 2024-01-15T18:16:28.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.843+0000 7fd471600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c13e950 con 0x7fd46c13adc0 2024-01-15T18:16:28.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.843+0000 7fd471600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c15b4f0 con 0x7fd44c005190 2024-01-15T18:16:28.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.843+0000 7fd46a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd46c13df90 0x7fd46c13e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:28.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.843+0000 7fd46a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd46c13df90 0x7fd46c13e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52994/0 (socket says 172.21.15.27:52994) 2024-01-15T18:16:28.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.843+0000 7fd46a000640 1 -- 172.21.15.27:0/636042924 learned_addr learned my addr 172.21.15.27:0/636042924 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:28.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.843+0000 7fd46aa00640 1 --2- 172.21.15.27:0/636042924 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd44c005190 0x7fd46c135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:28.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.843+0000 7fd46b400640 1 --2- 172.21.15.27:0/636042924 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd46c13adc0 0x7fd46c13b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:28.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.843+0000 7fd46a000640 1 -- 172.21.15.27:0/636042924 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd44c005190 msgr2=0x7fd46c135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:28.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.843+0000 7fd46a000640 1 --2- 172.21.15.27:0/636042924 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd44c005190 0x7fd46c135d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:28.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.843+0000 7fd46a000640 1 -- 172.21.15.27:0/636042924 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd46c13adc0 msgr2=0x7fd46c13b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:28.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.843+0000 7fd46a000640 1 --2- 172.21.15.27:0/636042924 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd46c13adc0 0x7fd46c13b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:28.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.843+0000 7fd46a000640 1 -- 172.21.15.27:0/636042924 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd46c13b780 con 0x7fd46c13df90 2024-01-15T18:16:28.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.843+0000 7fd46a000640 1 --2- 172.21.15.27:0/636042924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd46c13df90 0x7fd46c13e410 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fd45400f5f0 tx=0x7fd45400fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:28.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.843+0000 7fd462a00640 1 -- 172.21.15.27:0/636042924 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4540052e0 con 0x7fd46c13df90 2024-01-15T18:16:28.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.843+0000 7fd462a00640 1 -- 172.21.15.27:0/636042924 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd454005440 con 0x7fd46c13df90 2024-01-15T18:16:28.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.843+0000 7fd462a00640 1 -- 172.21.15.27:0/636042924 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd454003820 con 0x7fd46c13df90 2024-01-15T18:16:28.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.843+0000 7fd471600640 1 -- 172.21.15.27:0/636042924 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd46c15b740 con 0x7fd46c13df90 2024-01-15T18:16:28.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.843+0000 7fd471600640 1 -- 172.21.15.27:0/636042924 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd46c15bd00 con 0x7fd46c13df90 2024-01-15T18:16:28.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.847+0000 7fd471600640 1 -- 172.21.15.27:0/636042924 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd46c158280 con 0x7fd46c13df90 2024-01-15T18:16:28.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.847+0000 7fd462a00640 1 -- 172.21.15.27:0/636042924 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 31) v1 ==== 87282+0+0 (secure 0 0 0) 0x7fd454003980 con 0x7fd46c13df90 2024-01-15T18:16:28.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.847+0000 7fd462a00640 1 --2- 172.21.15.27:0/636042924 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd444067580 0x7fd444069a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:28.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.847+0000 7fd462a00640 1 -- 172.21.15.27:0/636042924 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fd454082c80 con 0x7fd46c13df90 2024-01-15T18:16:28.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.847+0000 7fd46aa00640 1 --2- 172.21.15.27:0/636042924 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd444067580 0x7fd444069a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:28.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.847+0000 7fd46aa00640 1 --2- 172.21.15.27:0/636042924 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd444067580 0x7fd444069a40 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fd45c000c10 tx=0x7fd45c008c00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:28.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:28.851+0000 7fd462a00640 1 -- 172.21.15.27:0/636042924 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd454051910 con 0x7fd46c13df90 2024-01-15T18:16:29.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:29.043+0000 7fd471600640 1 -- 172.21.15.27:0/636042924 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd46c0b4dc0 con 0x7fd444067580 2024-01-15T18:16:29.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:29.051+0000 7fd462a00640 1 -- 172.21.15.27:0/636042924 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7fd46c0b4dc0 con 0x7fd444067580 2024-01-15T18:16:29.051 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:16:29.051 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:16:29.051 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "pending", 2024-01-15T18:16:29.051 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:16:29.052 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:16:29.052 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:16:29.052 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:16:29.052 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:29.052 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:29.052 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:16:29.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:29.055+0000 7fd471600640 1 -- 172.21.15.27:0/636042924 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd444067580 msgr2=0x7fd444069a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:29.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:29.055+0000 7fd471600640 1 --2- 172.21.15.27:0/636042924 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd444067580 0x7fd444069a40 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fd45c000c10 tx=0x7fd45c008c00 comp rx=0 tx=0).stop 2024-01-15T18:16:29.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:29.055+0000 7fd471600640 1 -- 172.21.15.27:0/636042924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd46c13df90 msgr2=0x7fd46c13e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:29.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:29.055+0000 7fd471600640 1 --2- 172.21.15.27:0/636042924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd46c13df90 0x7fd46c13e410 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fd45400f5f0 tx=0x7fd45400fab0 comp rx=0 tx=0).stop 2024-01-15T18:16:29.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:29.055+0000 7fd471600640 1 -- 172.21.15.27:0/636042924 shutdown_connections 2024-01-15T18:16:29.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:29.055+0000 7fd471600640 1 --2- 172.21.15.27:0/636042924 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd444067580 0x7fd444069a40 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:29.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:29.055+0000 7fd471600640 1 --2- 172.21.15.27:0/636042924 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd46c13adc0 0x7fd46c13b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:29.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:29.055+0000 7fd471600640 1 --2- 172.21.15.27:0/636042924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd46c13df90 0x7fd46c13e410 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:29.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:29.055+0000 7fd471600640 1 --2- 172.21.15.27:0/636042924 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd44c005190 0x7fd46c135d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:29.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:29.055+0000 7fd471600640 1 -- 172.21.15.27:0/636042924 >> 172.21.15.27:0/636042924 conn(0x7fd46c0cb700 msgr2=0x7fd46c157b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:29.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:29.055+0000 7fd471600640 1 -- 172.21.15.27:0/636042924 shutdown_connections 2024-01-15T18:16:29.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:29.055+0000 7fd471600640 1 -- 172.21.15.27:0/636042924 wait complete. 2024-01-15T18:16:29.073 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-15T18:16:30.075 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:16:30.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.235+0000 7f3a8d600640 1 Processor -- start 2024-01-15T18:16:30.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.235+0000 7f3a8d600640 1 -- start start 2024-01-15T18:16:30.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.235+0000 7f3a8d600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a8806c430 0x7f3a88070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:30.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.235+0000 7f3a8d600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3a880b1db0 0x7f3a880b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:30.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.235+0000 7f3a8d600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f3a880ce050 0x7f3a880ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:30.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.235+0000 7f3a8d600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a880cea20 con 0x7f3a880b1db0 2024-01-15T18:16:30.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.235+0000 7f3a8d600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a880ccfb0 con 0x7f3a880ce050 2024-01-15T18:16:30.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.235+0000 7f3a8d600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a880cd290 con 0x7f3a8806c430 2024-01-15T18:16:30.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.235+0000 7f3a86000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3a880b1db0 0x7f3a880b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:30.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.235+0000 7f3a86000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3a880b1db0 0x7f3a880b2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53002/0 (socket says 172.21.15.27:53002) 2024-01-15T18:16:30.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.235+0000 7f3a86000640 1 -- 172.21.15.27:0/1980886521 learned_addr learned my addr 172.21.15.27:0/1980886521 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:30.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.235+0000 7f3a87400640 1 --2- 172.21.15.27:0/1980886521 >> v2:172.21.15.93:3300/0 conn(0x7f3a880ce050 0x7f3a880ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:30.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.235+0000 7f3a86a00640 1 --2- 172.21.15.27:0/1980886521 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a8806c430 0x7f3a88070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:30.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.235+0000 7f3a86000640 1 -- 172.21.15.27:0/1980886521 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a8806c430 msgr2=0x7f3a88070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:30.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.235+0000 7f3a86000640 1 --2- 172.21.15.27:0/1980886521 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a8806c430 0x7f3a88070ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:30.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.235+0000 7f3a86000640 1 -- 172.21.15.27:0/1980886521 >> v2:172.21.15.93:3300/0 conn(0x7f3a880ce050 msgr2=0x7f3a880ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:30.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.235+0000 7f3a86000640 1 --2- 172.21.15.27:0/1980886521 >> v2:172.21.15.93:3300/0 conn(0x7f3a880ce050 0x7f3a880ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:30.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.235+0000 7f3a86000640 1 -- 172.21.15.27:0/1980886521 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a880cd570 con 0x7f3a880b1db0 2024-01-15T18:16:30.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.235+0000 7f3a86a00640 1 --2- 172.21.15.27:0/1980886521 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a8806c430 0x7f3a88070ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:16:30.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.239+0000 7f3a86000640 1 --2- 172.21.15.27:0/1980886521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a880b1db0 0x7f3a880b2210 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f3a88072330 tx=0x7f3a7001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8fe3b97f1ffc311 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:30.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.239+0000 7f3a85600640 1 -- 172.21.15.27:0/1980886521 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a700ab600 con 0x7f3a880b1db0 2024-01-15T18:16:30.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.239+0000 7f3a85600640 1 -- 172.21.15.27:0/1980886521 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a700a7050 con 0x7f3a880b1db0 2024-01-15T18:16:30.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.239+0000 7f3a85600640 1 -- 172.21.15.27:0/1980886521 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a700ab980 con 0x7f3a880b1db0 2024-01-15T18:16:30.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.239+0000 7f3a8d600640 1 -- 172.21.15.27:0/1980886521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a880b1db0 msgr2=0x7f3a880b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:30.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.239+0000 7f3a8d600640 1 --2- 172.21.15.27:0/1980886521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a880b1db0 0x7f3a880b2210 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f3a88072330 tx=0x7f3a7001c470 comp rx=0 tx=0).stop 2024-01-15T18:16:30.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.239+0000 7f3a8d600640 1 -- 172.21.15.27:0/1980886521 shutdown_connections 2024-01-15T18:16:30.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.239+0000 7f3a8d600640 1 --2- 172.21.15.27:0/1980886521 >> v2:172.21.15.93:3300/0 conn(0x7f3a880ce050 0x7f3a880ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:30.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.239+0000 7f3a8d600640 1 --2- 172.21.15.27:0/1980886521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a880b1db0 0x7f3a880b2210 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:30.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.239+0000 7f3a8d600640 1 --2- 172.21.15.27:0/1980886521 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a8806c430 0x7f3a88070ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:30.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.239+0000 7f3a8d600640 1 -- 172.21.15.27:0/1980886521 >> 172.21.15.27:0/1980886521 conn(0x7f3a880cb700 msgr2=0x7f3a880cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:30.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.239+0000 7f3a8d600640 1 -- 172.21.15.27:0/1980886521 shutdown_connections 2024-01-15T18:16:30.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.239+0000 7f3a8d600640 1 -- 172.21.15.27:0/1980886521 wait complete. 2024-01-15T18:16:30.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.243+0000 7f3a8d600640 1 Processor -- start 2024-01-15T18:16:30.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.243+0000 7f3a8d600640 1 -- start start 2024-01-15T18:16:30.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.243+0000 7f3a8d600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a8806c430 0x7f3a88148450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:30.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.243+0000 7f3a8d600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3a880b1db0 0x7f3a88150b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:30.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.243+0000 7f3a8d600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a880ce050 0x7f3a88149f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:30.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.243+0000 7f3a8d600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a88150180 con 0x7f3a8806c430 2024-01-15T18:16:30.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.243+0000 7f3a8d600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a88148990 con 0x7f3a880b1db0 2024-01-15T18:16:30.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.243+0000 7f3a8d600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8814b1e0 con 0x7f3a880ce050 2024-01-15T18:16:30.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.243+0000 7f3a86a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a8806c430 0x7f3a88148450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:30.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.243+0000 7f3a86a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a8806c430 0x7f3a88148450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53008/0 (socket says 172.21.15.27:53008) 2024-01-15T18:16:30.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.243+0000 7f3a86a00640 1 -- 172.21.15.27:0/759197637 learned_addr learned my addr 172.21.15.27:0/759197637 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:30.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.243+0000 7f3a87400640 1 --2- 172.21.15.27:0/759197637 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a880ce050 0x7f3a88149f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:30.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.243+0000 7f3a86000640 1 --2- 172.21.15.27:0/759197637 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3a880b1db0 0x7f3a88150b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:30.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.243+0000 7f3a86a00640 1 -- 172.21.15.27:0/759197637 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a880ce050 msgr2=0x7f3a88149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:30.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.243+0000 7f3a86a00640 1 --2- 172.21.15.27:0/759197637 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a880ce050 0x7f3a88149f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:30.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.243+0000 7f3a86a00640 1 -- 172.21.15.27:0/759197637 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3a880b1db0 msgr2=0x7f3a88150b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:30.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.243+0000 7f3a86a00640 1 --2- 172.21.15.27:0/759197637 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3a880b1db0 0x7f3a88150b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:30.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.243+0000 7f3a86a00640 1 -- 172.21.15.27:0/759197637 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a700094d0 con 0x7f3a8806c430 2024-01-15T18:16:30.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.243+0000 7f3a86a00640 1 --2- 172.21.15.27:0/759197637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a8806c430 0x7f3a88148450 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f3a7800e810 tx=0x7f3a7800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:30.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.243+0000 7f3a7ea00640 1 -- 172.21.15.27:0/759197637 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7800c8d0 con 0x7f3a8806c430 2024-01-15T18:16:30.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.243+0000 7f3a7ea00640 1 -- 172.21.15.27:0/759197637 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a7800ca30 con 0x7f3a8806c430 2024-01-15T18:16:30.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.243+0000 7f3a7ea00640 1 -- 172.21.15.27:0/759197637 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7800cde0 con 0x7f3a8806c430 2024-01-15T18:16:30.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.243+0000 7f3a8d600640 1 -- 172.21.15.27:0/759197637 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a8814b4c0 con 0x7f3a8806c430 2024-01-15T18:16:30.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.243+0000 7f3a8d600640 1 -- 172.21.15.27:0/759197637 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a88150670 con 0x7f3a8806c430 2024-01-15T18:16:30.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.243+0000 7f3a8d600640 1 -- 172.21.15.27:0/759197637 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a8814b6b0 con 0x7f3a8806c430 2024-01-15T18:16:30.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.247+0000 7f3a7ea00640 1 -- 172.21.15.27:0/759197637 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 32) v1 ==== 87384+0+0 (secure 0 0 0) 0x7f3a78013410 con 0x7f3a8806c430 2024-01-15T18:16:30.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.247+0000 7f3a7ea00640 1 --2- 172.21.15.27:0/759197637 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a60067670 0x7f3a60069b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:30.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.247+0000 7f3a7ea00640 1 -- 172.21.15.27:0/759197637 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f3a78082ed0 con 0x7f3a8806c430 2024-01-15T18:16:30.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.247+0000 7f3a86000640 1 --2- 172.21.15.27:0/759197637 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a60067670 0x7f3a60069b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:30.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.247+0000 7f3a86000640 1 --2- 172.21.15.27:0/759197637 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a60067670 0x7f3a60069b30 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f3a700ad000 tx=0x7f3a700a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:30.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.251+0000 7f3a7ea00640 1 -- 172.21.15.27:0/759197637 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3a8814b6b0 con 0x7f3a8806c430 2024-01-15T18:16:30.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.443+0000 7f3a8d600640 1 -- 172.21.15.27:0/759197637 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a8806c810 con 0x7f3a60067670 2024-01-15T18:16:30.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.451+0000 7f3a7ea00640 1 -- 172.21.15.27:0/759197637 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f3a700ab660 con 0x7f3a60067670 2024-01-15T18:16:30.448 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:16:30.448 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:16:30.448 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "pending", 2024-01-15T18:16:30.449 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:16:30.449 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:16:30.449 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:16:30.449 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:16:30.449 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:30.449 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:30.449 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:16:30.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.451+0000 7f3a8d600640 1 -- 172.21.15.27:0/759197637 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a60067670 msgr2=0x7f3a60069b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:30.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.451+0000 7f3a8d600640 1 --2- 172.21.15.27:0/759197637 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a60067670 0x7f3a60069b30 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f3a700ad000 tx=0x7f3a700a6000 comp rx=0 tx=0).stop 2024-01-15T18:16:30.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.451+0000 7f3a8d600640 1 -- 172.21.15.27:0/759197637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a8806c430 msgr2=0x7f3a88148450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:30.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.451+0000 7f3a8d600640 1 --2- 172.21.15.27:0/759197637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a8806c430 0x7f3a88148450 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f3a7800e810 tx=0x7f3a7800ecd0 comp rx=0 tx=0).stop 2024-01-15T18:16:30.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.451+0000 7f3a8d600640 1 -- 172.21.15.27:0/759197637 shutdown_connections 2024-01-15T18:16:30.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.451+0000 7f3a8d600640 1 --2- 172.21.15.27:0/759197637 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a60067670 0x7f3a60069b30 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:30.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.451+0000 7f3a8d600640 1 --2- 172.21.15.27:0/759197637 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a880ce050 0x7f3a88149f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:30.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.451+0000 7f3a8d600640 1 --2- 172.21.15.27:0/759197637 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3a880b1db0 0x7f3a88150b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:30.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.451+0000 7f3a8d600640 1 --2- 172.21.15.27:0/759197637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a8806c430 0x7f3a88148450 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:30.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.451+0000 7f3a8d600640 1 -- 172.21.15.27:0/759197637 >> 172.21.15.27:0/759197637 conn(0x7f3a880cb700 msgr2=0x7f3a8814b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:30.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.451+0000 7f3a8d600640 1 -- 172.21.15.27:0/759197637 shutdown_connections 2024-01-15T18:16:30.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:30.451+0000 7f3a8d600640 1 -- 172.21.15.27:0/759197637 wait complete. 2024-01-15T18:16:30.470 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-15T18:16:31.471 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:16:31.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.627+0000 7fa579a00640 1 Processor -- start 2024-01-15T18:16:31.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.627+0000 7fa579a00640 1 -- start start 2024-01-15T18:16:31.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.627+0000 7fa579a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa574158900 0x7fa57415f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:31.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.627+0000 7fa579a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa57415bf00 0x7fa5740cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.627+0000 7fa579a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5740cf110 0x7fa5740701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.627+0000 7fa579a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5740b1cb0 con 0x7fa5740cf110 2024-01-15T18:16:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.627+0000 7fa579a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5740cf510 con 0x7fa57415bf00 2024-01-15T18:16:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.627+0000 7fa579a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5740b1f00 con 0x7fa574158900 2024-01-15T18:16:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.627+0000 7fa573400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5740cf110 0x7fa5740701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.627+0000 7fa573400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa5740cf110 0x7fa5740701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53018/0 (socket says 172.21.15.27:53018) 2024-01-15T18:16:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.627+0000 7fa573400640 1 -- 172.21.15.27:0/3738679544 learned_addr learned my addr 172.21.15.27:0/3738679544 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.627+0000 7fa572000640 1 --2- 172.21.15.27:0/3738679544 >> v2:172.21.15.93:3300/0 conn(0x7fa57415bf00 0x7fa5740cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.627+0000 7fa572a00640 1 --2- 172.21.15.27:0/3738679544 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa574158900 0x7fa57415f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.631+0000 7fa573400640 1 -- 172.21.15.27:0/3738679544 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa574158900 msgr2=0x7fa57415f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.631+0000 7fa573400640 1 --2- 172.21.15.27:0/3738679544 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa574158900 0x7fa57415f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.631+0000 7fa573400640 1 -- 172.21.15.27:0/3738679544 >> v2:172.21.15.93:3300/0 conn(0x7fa57415bf00 msgr2=0x7fa5740cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.631+0000 7fa573400640 1 --2- 172.21.15.27:0/3738679544 >> v2:172.21.15.93:3300/0 conn(0x7fa57415bf00 0x7fa5740cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.631+0000 7fa573400640 1 -- 172.21.15.27:0/3738679544 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5740b21e0 con 0x7fa5740cf110 2024-01-15T18:16:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.631+0000 7fa573400640 1 --2- 172.21.15.27:0/3738679544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5740cf110 0x7fa5740701e0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fa5740cead0 tx=0x7fa56c01c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6eb1aa72e9e82a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.631+0000 7fa571600640 1 -- 172.21.15.27:0/3738679544 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa574070f30 con 0x7fa5740cf110 2024-01-15T18:16:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.631+0000 7fa571600640 1 -- 172.21.15.27:0/3738679544 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56c0a5740 con 0x7fa5740cf110 2024-01-15T18:16:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.631+0000 7fa571600640 1 -- 172.21.15.27:0/3738679544 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c0a5ad0 con 0x7fa5740cf110 2024-01-15T18:16:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.631+0000 7fa579a00640 1 -- 172.21.15.27:0/3738679544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5740cf110 msgr2=0x7fa5740701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.631+0000 7fa579a00640 1 --2- 172.21.15.27:0/3738679544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5740cf110 0x7fa5740701e0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fa5740cead0 tx=0x7fa56c01c9d0 comp rx=0 tx=0).stop 2024-01-15T18:16:31.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.635+0000 7fa579a00640 1 -- 172.21.15.27:0/3738679544 shutdown_connections 2024-01-15T18:16:31.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.635+0000 7fa579a00640 1 --2- 172.21.15.27:0/3738679544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5740cf110 0x7fa5740701e0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:31.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.635+0000 7fa579a00640 1 --2- 172.21.15.27:0/3738679544 >> v2:172.21.15.93:3300/0 conn(0x7fa57415bf00 0x7fa5740cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:31.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.635+0000 7fa579a00640 1 --2- 172.21.15.27:0/3738679544 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa574158900 0x7fa57415f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:31.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.635+0000 7fa579a00640 1 -- 172.21.15.27:0/3738679544 >> 172.21.15.27:0/3738679544 conn(0x7fa5740cb700 msgr2=0x7fa5740cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:31.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.635+0000 7fa579a00640 1 -- 172.21.15.27:0/3738679544 shutdown_connections 2024-01-15T18:16:31.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.635+0000 7fa579a00640 1 -- 172.21.15.27:0/3738679544 wait complete. 2024-01-15T18:16:31.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.635+0000 7fa579a00640 1 Processor -- start 2024-01-15T18:16:31.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.635+0000 7fa579a00640 1 -- start start 2024-01-15T18:16:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.635+0000 7fa579a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5740cf110 0x7fa57414e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.635+0000 7fa579a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa574158900 0x7fa5740af2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.635+0000 7fa579a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa57415bf00 0x7fa5740af7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.635+0000 7fa579a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa574156ef0 con 0x7fa5740cf110 2024-01-15T18:16:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.635+0000 7fa579a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5740afd30 con 0x7fa57415bf00 2024-01-15T18:16:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.635+0000 7fa579a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa574157160 con 0x7fa574158900 2024-01-15T18:16:31.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.635+0000 7fa572a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5740cf110 0x7fa57414e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:31.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.635+0000 7fa572a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5740cf110 0x7fa57414e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53022/0 (socket says 172.21.15.27:53022) 2024-01-15T18:16:31.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.635+0000 7fa572a00640 1 -- 172.21.15.27:0/2445688918 learned_addr learned my addr 172.21.15.27:0/2445688918 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:31.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.635+0000 7fa573400640 1 --2- 172.21.15.27:0/2445688918 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa57415bf00 0x7fa5740af7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:31.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.635+0000 7fa572000640 1 --2- 172.21.15.27:0/2445688918 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa574158900 0x7fa5740af2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:31.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.635+0000 7fa572a00640 1 -- 172.21.15.27:0/2445688918 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa574158900 msgr2=0x7fa5740af2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:31.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.635+0000 7fa572a00640 1 --2- 172.21.15.27:0/2445688918 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa574158900 0x7fa5740af2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:31.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.635+0000 7fa572a00640 1 -- 172.21.15.27:0/2445688918 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa57415bf00 msgr2=0x7fa5740af7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:31.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.635+0000 7fa572a00640 1 --2- 172.21.15.27:0/2445688918 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa57415bf00 0x7fa5740af7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:31.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.635+0000 7fa572a00640 1 -- 172.21.15.27:0/2445688918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56c009b30 con 0x7fa5740cf110 2024-01-15T18:16:31.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.639+0000 7fa572a00640 1 --2- 172.21.15.27:0/2445688918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5740cf110 0x7fa57414e6a0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fa56400e960 tx=0x7fa56400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:31.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.639+0000 7fa56aa00640 1 -- 172.21.15.27:0/2445688918 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa564009690 con 0x7fa5740cf110 2024-01-15T18:16:31.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.639+0000 7fa56aa00640 1 -- 172.21.15.27:0/2445688918 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56400ca00 con 0x7fa5740cf110 2024-01-15T18:16:31.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.639+0000 7fa56aa00640 1 -- 172.21.15.27:0/2445688918 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56400cd70 con 0x7fa5740cf110 2024-01-15T18:16:31.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.639+0000 7fa579a00640 1 -- 172.21.15.27:0/2445688918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa574157440 con 0x7fa5740cf110 2024-01-15T18:16:31.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.639+0000 7fa579a00640 1 -- 172.21.15.27:0/2445688918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa574157a00 con 0x7fa5740cf110 2024-01-15T18:16:31.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.639+0000 7fa579a00640 1 -- 172.21.15.27:0/2445688918 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa57404e850 con 0x7fa5740cf110 2024-01-15T18:16:31.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.639+0000 7fa56aa00640 1 -- 172.21.15.27:0/2445688918 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 32) v1 ==== 87384+0+0 (secure 0 0 0) 0x7fa564013470 con 0x7fa5740cf110 2024-01-15T18:16:31.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.639+0000 7fa56aa00640 1 --2- 172.21.15.27:0/2445688918 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa54c0676a0 0x7fa54c069b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:31.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.639+0000 7fa56aa00640 1 -- 172.21.15.27:0/2445688918 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fa5640832b0 con 0x7fa5740cf110 2024-01-15T18:16:31.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.643+0000 7fa572000640 1 --2- 172.21.15.27:0/2445688918 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa54c0676a0 0x7fa54c069b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:31.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.643+0000 7fa572000640 1 --2- 172.21.15.27:0/2445688918 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa54c0676a0 0x7fa54c069b60 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fa574145d80 tx=0x7fa55c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:31.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.643+0000 7fa56aa00640 1 -- 172.21.15.27:0/2445688918 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa564051d40 con 0x7fa5740cf110 2024-01-15T18:16:31.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.839+0000 7fa579a00640 1 -- 172.21.15.27:0/2445688918 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5740cf990 con 0x7fa54c0676a0 2024-01-15T18:16:31.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.847+0000 7fa56aa00640 1 -- 172.21.15.27:0/2445688918 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7fa5740cf990 con 0x7fa54c0676a0 2024-01-15T18:16:31.845 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:16:31.845 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:16:31.845 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "pending", 2024-01-15T18:16:31.845 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:16:31.846 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:16:31.846 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:16:31.846 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:16:31.846 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:31.846 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:31.846 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:16:31.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.847+0000 7fa579a00640 1 -- 172.21.15.27:0/2445688918 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa54c0676a0 msgr2=0x7fa54c069b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:31.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.847+0000 7fa579a00640 1 --2- 172.21.15.27:0/2445688918 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa54c0676a0 0x7fa54c069b60 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fa574145d80 tx=0x7fa55c008040 comp rx=0 tx=0).stop 2024-01-15T18:16:31.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.847+0000 7fa579a00640 1 -- 172.21.15.27:0/2445688918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5740cf110 msgr2=0x7fa57414e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:31.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.847+0000 7fa579a00640 1 --2- 172.21.15.27:0/2445688918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5740cf110 0x7fa57414e6a0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7fa56400e960 tx=0x7fa56400ee20 comp rx=0 tx=0).stop 2024-01-15T18:16:31.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.847+0000 7fa579a00640 1 -- 172.21.15.27:0/2445688918 shutdown_connections 2024-01-15T18:16:31.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.847+0000 7fa579a00640 1 --2- 172.21.15.27:0/2445688918 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa54c0676a0 0x7fa54c069b60 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:31.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.847+0000 7fa579a00640 1 --2- 172.21.15.27:0/2445688918 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa57415bf00 0x7fa5740af7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:31.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.847+0000 7fa579a00640 1 --2- 172.21.15.27:0/2445688918 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa574158900 0x7fa5740af2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:31.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.847+0000 7fa579a00640 1 --2- 172.21.15.27:0/2445688918 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5740cf110 0x7fa57414e6a0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:31.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.847+0000 7fa579a00640 1 -- 172.21.15.27:0/2445688918 >> 172.21.15.27:0/2445688918 conn(0x7fa5740cb700 msgr2=0x7fa574073450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:31.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.847+0000 7fa579a00640 1 -- 172.21.15.27:0/2445688918 shutdown_connections 2024-01-15T18:16:31.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:31.847+0000 7fa579a00640 1 -- 172.21.15.27:0/2445688918 wait complete. 2024-01-15T18:16:31.868 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-15T18:16:32.869 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:16:33.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.027+0000 7f3844400640 1 Processor -- start 2024-01-15T18:16:33.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.027+0000 7f3844400640 1 -- start start 2024-01-15T18:16:33.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.027+0000 7f3844400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f383c15ad20 0x7f383c15b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:33.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.027+0000 7f3844400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f383c0715f0 0x7f383c15ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:33.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.027+0000 7f3844400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f383c15a610 con 0x7f383c15d900 2024-01-15T18:16:33.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.027+0000 7f3844400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f383c15e3c0 con 0x7f383c0715f0 2024-01-15T18:16:33.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.027+0000 7f3844400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f383c15e6a0 con 0x7f383c15ad20 2024-01-15T18:16:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.027+0000 7f3842400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f383c0715f0 0x7f383c15ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.027+0000 7f3841000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f383c15d900 0x7f383c070ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58478/0 (socket says 172.21.15.27:58478) 2024-01-15T18:16:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.027+0000 7f3841000640 1 -- 172.21.15.27:0/3225979741 learned_addr learned my addr 172.21.15.27:0/3225979741 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:16:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.027+0000 7f3841a00640 1 --2- 172.21.15.27:0/3225979741 >> v2:172.21.15.93:3300/0 conn(0x7f383c15ad20 0x7f383c15b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.027+0000 7f3842400640 1 -- 172.21.15.27:0/3225979741 >> v2:172.21.15.93:3300/0 conn(0x7f383c15ad20 msgr2=0x7f383c15b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:33.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.027+0000 7f3842400640 1 --2- 172.21.15.27:0/3225979741 >> v2:172.21.15.93:3300/0 conn(0x7f383c15ad20 0x7f383c15b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:33.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.027+0000 7f3842400640 1 -- 172.21.15.27:0/3225979741 >> v1:172.21.15.27:6789/0 conn(0x7f383c15d900 legacy=0x7f383c070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:33.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.027+0000 7f383be00640 1 -- 172.21.15.27:0/3225979741 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321349630 0 0) 0x7f383c15a610 con 0x7f383c15d900 2024-01-15T18:16:33.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.027+0000 7f3842400640 1 -- 172.21.15.27:0/3225979741 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383c15b6e0 con 0x7f383c0715f0 2024-01-15T18:16:33.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.031+0000 7f3842400640 1 --2- 172.21.15.27:0/3225979741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f383c0715f0 0x7f383c15ddb0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f3834009d90 tx=0x7f383401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=994b43485331f53e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:33.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.031+0000 7f383be00640 1 -- 172.21.15.27:0/3225979741 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38340a59c0 con 0x7f383c0715f0 2024-01-15T18:16:33.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.031+0000 7f383be00640 1 -- 172.21.15.27:0/3225979741 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38340a5b20 con 0x7f383c0715f0 2024-01-15T18:16:33.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.031+0000 7f383be00640 1 -- 172.21.15.27:0/3225979741 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38340ad440 con 0x7f383c0715f0 2024-01-15T18:16:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.031+0000 7f3844400640 1 -- 172.21.15.27:0/3225979741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f383c0715f0 msgr2=0x7f383c15ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.031+0000 7f3844400640 1 --2- 172.21.15.27:0/3225979741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f383c0715f0 0x7f383c15ddb0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f3834009d90 tx=0x7f383401c8e0 comp rx=0 tx=0).stop 2024-01-15T18:16:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.031+0000 7f3844400640 1 -- 172.21.15.27:0/3225979741 shutdown_connections 2024-01-15T18:16:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.031+0000 7f3844400640 1 --2- 172.21.15.27:0/3225979741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f383c0715f0 0x7f383c15ddb0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.031+0000 7f3844400640 1 --2- 172.21.15.27:0/3225979741 >> v2:172.21.15.93:3300/0 conn(0x7f383c15ad20 0x7f383c15b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.031+0000 7f3844400640 1 -- 172.21.15.27:0/3225979741 >> 172.21.15.27:0/3225979741 conn(0x7f383c0cb700 msgr2=0x7f383c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.031+0000 7f3844400640 1 -- 172.21.15.27:0/3225979741 shutdown_connections 2024-01-15T18:16:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.031+0000 7f3844400640 1 -- 172.21.15.27:0/3225979741 wait complete. 2024-01-15T18:16:33.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.035+0000 7f3844400640 1 Processor -- start 2024-01-15T18:16:33.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.035+0000 7f3844400640 1 -- start start 2024-01-15T18:16:33.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.035+0000 7f3844400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f383c0715f0 0x7f383c0b2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:33.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.035+0000 7f3844400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f383c15ad20 0x7f383c0c9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:33.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.035+0000 7f3844400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f383c15d900 0x7f383c0c9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:33.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.035+0000 7f3844400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c0c8e90 con 0x7f383c0715f0 2024-01-15T18:16:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.035+0000 7f3844400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c0c8030 con 0x7f383c15ad20 2024-01-15T18:16:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.035+0000 7f3844400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f383c0c8280 con 0x7f383c15d900 2024-01-15T18:16:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.035+0000 7f3841a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f383c0715f0 0x7f383c0b2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.035+0000 7f3841a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f383c0715f0 0x7f383c0b2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53040/0 (socket says 172.21.15.27:53040) 2024-01-15T18:16:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.035+0000 7f3841a00640 1 -- 172.21.15.27:0/2799712204 learned_addr learned my addr 172.21.15.27:0/2799712204 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.035+0000 7f3841000640 1 --2- 172.21.15.27:0/2799712204 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f383c15ad20 0x7f383c0c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.035+0000 7f3842400640 1 --2- 172.21.15.27:0/2799712204 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f383c15d900 0x7f383c0c9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.035+0000 7f3841a00640 1 -- 172.21.15.27:0/2799712204 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f383c15d900 msgr2=0x7f383c0c9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.035+0000 7f3841a00640 1 --2- 172.21.15.27:0/2799712204 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f383c15d900 0x7f383c0c9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.035+0000 7f3841a00640 1 -- 172.21.15.27:0/2799712204 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f383c15ad20 msgr2=0x7f383c0c9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.035+0000 7f3841a00640 1 --2- 172.21.15.27:0/2799712204 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f383c15ad20 0x7f383c0c9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.035+0000 7f3841a00640 1 -- 172.21.15.27:0/2799712204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3834009a70 con 0x7f383c0715f0 2024-01-15T18:16:33.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.035+0000 7f3841a00640 1 --2- 172.21.15.27:0/2799712204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f383c0715f0 0x7f383c0b2870 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f382c00d800 tx=0x7f382c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:33.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.035+0000 7f3839600640 1 -- 172.21.15.27:0/2799712204 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c00b6c0 con 0x7f383c0715f0 2024-01-15T18:16:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.035+0000 7f3839600640 1 -- 172.21.15.27:0/2799712204 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382c00b820 con 0x7f383c0715f0 2024-01-15T18:16:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.035+0000 7f3844400640 1 -- 172.21.15.27:0/2799712204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f383c0c8560 con 0x7f383c0715f0 2024-01-15T18:16:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.035+0000 7f3839600640 1 -- 172.21.15.27:0/2799712204 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c00bb90 con 0x7f383c0715f0 2024-01-15T18:16:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.035+0000 7f3844400640 1 -- 172.21.15.27:0/2799712204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f383c0c8b20 con 0x7f383c0715f0 2024-01-15T18:16:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.039+0000 7f3844400640 1 -- 172.21.15.27:0/2799712204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f383c04e850 con 0x7f383c0715f0 2024-01-15T18:16:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.039+0000 7f3839600640 1 -- 172.21.15.27:0/2799712204 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 32) v1 ==== 87384+0+0 (secure 0 0 0) 0x7f382c00bcf0 con 0x7f383c0715f0 2024-01-15T18:16:33.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.039+0000 7f3839600640 1 --2- 172.21.15.27:0/2799712204 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3814067670 0x7f3814069b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:33.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.039+0000 7f3839600640 1 -- 172.21.15.27:0/2799712204 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f382c0835b0 con 0x7f383c0715f0 2024-01-15T18:16:33.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.039+0000 7f3841000640 1 --2- 172.21.15.27:0/2799712204 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3814067670 0x7f3814069b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:33.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.039+0000 7f3841000640 1 --2- 172.21.15.27:0/2799712204 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3814067670 0x7f3814069b30 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f383c156c10 tx=0x7f3824007df0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:33.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.043+0000 7f3839600640 1 -- 172.21.15.27:0/2799712204 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f382c052040 con 0x7f383c0715f0 2024-01-15T18:16:33.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.247+0000 7f3844400640 1 -- 172.21.15.27:0/2799712204 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f383c0b1e20 con 0x7f3814067670 2024-01-15T18:16:33.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.255+0000 7f3839600640 1 -- 172.21.15.27:0/2799712204 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f383c0b1e20 con 0x7f3814067670 2024-01-15T18:16:33.253 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:16:33.254 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:16:33.254 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "pending", 2024-01-15T18:16:33.254 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:16:33.254 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:16:33.254 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:16:33.254 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:16:33.254 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:33.254 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:33.254 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:16:33.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.255+0000 7f3844400640 1 -- 172.21.15.27:0/2799712204 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3814067670 msgr2=0x7f3814069b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:33.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.255+0000 7f3844400640 1 --2- 172.21.15.27:0/2799712204 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3814067670 0x7f3814069b30 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f383c156c10 tx=0x7f3824007df0 comp rx=0 tx=0).stop 2024-01-15T18:16:33.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.255+0000 7f3844400640 1 -- 172.21.15.27:0/2799712204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f383c0715f0 msgr2=0x7f383c0b2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:33.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.255+0000 7f3844400640 1 --2- 172.21.15.27:0/2799712204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f383c0715f0 0x7f383c0b2870 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f382c00d800 tx=0x7f382c00dcc0 comp rx=0 tx=0).stop 2024-01-15T18:16:33.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.255+0000 7f3844400640 1 -- 172.21.15.27:0/2799712204 shutdown_connections 2024-01-15T18:16:33.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.255+0000 7f3844400640 1 --2- 172.21.15.27:0/2799712204 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3814067670 0x7f3814069b30 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:33.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.255+0000 7f3844400640 1 --2- 172.21.15.27:0/2799712204 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f383c15d900 0x7f383c0c9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:33.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.255+0000 7f3844400640 1 --2- 172.21.15.27:0/2799712204 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f383c15ad20 0x7f383c0c9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:33.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.255+0000 7f3844400640 1 --2- 172.21.15.27:0/2799712204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f383c0715f0 0x7f383c0b2870 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:33.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.255+0000 7f3844400640 1 -- 172.21.15.27:0/2799712204 >> 172.21.15.27:0/2799712204 conn(0x7f383c0cb700 msgr2=0x7f383c06dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:33.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.255+0000 7f3844400640 1 -- 172.21.15.27:0/2799712204 shutdown_connections 2024-01-15T18:16:33.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:33.255+0000 7f3844400640 1 -- 172.21.15.27:0/2799712204 wait complete. 2024-01-15T18:16:33.276 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-15T18:16:34.277 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:16:34.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.431+0000 7f8fd3200640 1 Processor -- start 2024-01-15T18:16:34.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.431+0000 7f8fd3200640 1 -- start start 2024-01-15T18:16:34.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.435+0000 7f8fd3200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8fcc0b1db0 0x7f8fcc0b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:34.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.435+0000 7f8fd3200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8fcc0ce050 0x7f8fcc0ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:34.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.435+0000 7f8fd3200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fcc0712b0 con 0x7f8fcc06c430 2024-01-15T18:16:34.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.435+0000 7f8fd3200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fcc0cea20 con 0x7f8fcc0ce050 2024-01-15T18:16:34.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.435+0000 7f8fd3200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc0ccfb0 con 0x7f8fcc0b1db0 2024-01-15T18:16:34.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.435+0000 7f8fd1200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8fcc0ce050 0x7f8fcc0ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:34.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.435+0000 7f8fcbe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8fcc06c430 0x7f8fcc070ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40576/0 (socket says 172.21.15.27:40576) 2024-01-15T18:16:34.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.435+0000 7f8fcbe00640 1 -- 172.21.15.27:0/3759930765 learned_addr learned my addr 172.21.15.27:0/3759930765 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:16:34.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.435+0000 7f8fcb400640 1 --2- 172.21.15.27:0/3759930765 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8fcc0b1db0 0x7f8fcc0b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:34.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.435+0000 7f8fcaa00640 1 -- 172.21.15.27:0/3759930765 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 500964950 0 0) 0x7f8fcc0712b0 con 0x7f8fcc06c430 2024-01-15T18:16:34.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.435+0000 7f8fcaa00640 1 -- 172.21.15.27:0/3759930765 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fac005740 con 0x7f8fcc06c430 2024-01-15T18:16:34.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.435+0000 7f8fd1200640 1 -- 172.21.15.27:0/3759930765 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8fcc0b1db0 msgr2=0x7f8fcc0b2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:34.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.435+0000 7f8fd1200640 1 --2- 172.21.15.27:0/3759930765 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8fcc0b1db0 0x7f8fcc0b2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:34.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.435+0000 7f8fd1200640 1 -- 172.21.15.27:0/3759930765 >> v1:172.21.15.27:6789/0 conn(0x7f8fcc06c430 legacy=0x7f8fcc070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:34.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.435+0000 7f8fd1200640 1 -- 172.21.15.27:0/3759930765 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fcc0cd290 con 0x7f8fcc0ce050 2024-01-15T18:16:34.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.435+0000 7f8fcb400640 1 --2- 172.21.15.27:0/3759930765 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8fcc0b1db0 0x7f8fcc0b2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:16:34.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.439+0000 7f8fd1200640 1 --2- 172.21.15.27:0/3759930765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc0ce050 0x7f8fcc0ce4e0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400aea0 tx=0x7f8fc401c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a1ca947b6ea29d8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:34.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.439+0000 7f8fcaa00640 1 -- 172.21.15.27:0/3759930765 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc400a2d0 con 0x7f8fcc0ce050 2024-01-15T18:16:34.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.439+0000 7f8fcaa00640 1 -- 172.21.15.27:0/3759930765 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc40a55f0 con 0x7f8fcc0ce050 2024-01-15T18:16:34.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.439+0000 7f8fcaa00640 1 -- 172.21.15.27:0/3759930765 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc40a5980 con 0x7f8fcc0ce050 2024-01-15T18:16:34.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.439+0000 7f8fd3200640 1 -- 172.21.15.27:0/3759930765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc0ce050 msgr2=0x7f8fcc0ce4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:34.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.439+0000 7f8fd3200640 1 --2- 172.21.15.27:0/3759930765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc0ce050 0x7f8fcc0ce4e0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400aea0 tx=0x7f8fc401c8c0 comp rx=0 tx=0).stop 2024-01-15T18:16:34.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.439+0000 7f8fd3200640 1 -- 172.21.15.27:0/3759930765 shutdown_connections 2024-01-15T18:16:34.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.439+0000 7f8fd3200640 1 --2- 172.21.15.27:0/3759930765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc0ce050 0x7f8fcc0ce4e0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:34.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.439+0000 7f8fd3200640 1 --2- 172.21.15.27:0/3759930765 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8fcc0b1db0 0x7f8fcc0b2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:34.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.439+0000 7f8fd3200640 1 -- 172.21.15.27:0/3759930765 >> 172.21.15.27:0/3759930765 conn(0x7f8fcc0cb700 msgr2=0x7f8fcc0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:34.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.439+0000 7f8fd3200640 1 -- 172.21.15.27:0/3759930765 shutdown_connections 2024-01-15T18:16:34.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.439+0000 7f8fd3200640 1 -- 172.21.15.27:0/3759930765 wait complete. 2024-01-15T18:16:34.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.439+0000 7f8fd3200640 1 Processor -- start 2024-01-15T18:16:34.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.439+0000 7f8fd3200640 1 -- start start 2024-01-15T18:16:34.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.443+0000 7f8fd3200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8fcc06c430 0x7f8fcc15b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:34.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.443+0000 7f8fd3200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8fcc0b1db0 0x7f8fcc1569b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:34.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.443+0000 7f8fd3200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc0ce050 0x7f8fcc151a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:34.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.443+0000 7f8fd3200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc15b210 con 0x7f8fcc0ce050 2024-01-15T18:16:34.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.443+0000 7f8fd3200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc156ef0 con 0x7f8fcc0b1db0 2024-01-15T18:16:34.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.443+0000 7f8fd3200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc151f60 con 0x7f8fcc06c430 2024-01-15T18:16:34.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.443+0000 7f8fd1200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc0ce050 0x7f8fcc151a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:34.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.443+0000 7f8fd1200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc0ce050 0x7f8fcc151a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53186/0 (socket says 172.21.15.27:53186) 2024-01-15T18:16:34.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.443+0000 7f8fd1200640 1 -- 172.21.15.27:0/1304880041 learned_addr learned my addr 172.21.15.27:0/1304880041 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:34.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.443+0000 7f8fcb400640 1 --2- 172.21.15.27:0/1304880041 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8fcc0b1db0 0x7f8fcc1569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:34.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.443+0000 7f8fcbe00640 1 --2- 172.21.15.27:0/1304880041 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8fcc06c430 0x7f8fcc15b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:34.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.443+0000 7f8fd1200640 1 -- 172.21.15.27:0/1304880041 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8fcc06c430 msgr2=0x7f8fcc15b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:34.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.443+0000 7f8fd1200640 1 --2- 172.21.15.27:0/1304880041 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8fcc06c430 0x7f8fcc15b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:34.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.443+0000 7f8fd1200640 1 -- 172.21.15.27:0/1304880041 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8fcc0b1db0 msgr2=0x7f8fcc1569b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:34.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.443+0000 7f8fd1200640 1 --2- 172.21.15.27:0/1304880041 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8fcc0b1db0 0x7f8fcc1569b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:34.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.443+0000 7f8fd1200640 1 -- 172.21.15.27:0/1304880041 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc400ab80 con 0x7f8fcc0ce050 2024-01-15T18:16:34.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.443+0000 7f8fd1200640 1 --2- 172.21.15.27:0/1304880041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc0ce050 0x7f8fcc151a20 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f8fc40af040 tx=0x7f8fc4019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:34.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.443+0000 7f8fc3e00640 1 -- 172.21.15.27:0/1304880041 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc40b8670 con 0x7f8fcc0ce050 2024-01-15T18:16:34.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.443+0000 7f8fc3e00640 1 -- 172.21.15.27:0/1304880041 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc40b87f0 con 0x7f8fcc0ce050 2024-01-15T18:16:34.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.443+0000 7f8fd3200640 1 -- 172.21.15.27:0/1304880041 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fcc15ab70 con 0x7f8fcc0ce050 2024-01-15T18:16:34.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.443+0000 7f8fc3e00640 1 -- 172.21.15.27:0/1304880041 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc40b8ba0 con 0x7f8fcc0ce050 2024-01-15T18:16:34.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.443+0000 7f8fd3200640 1 -- 172.21.15.27:0/1304880041 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fcc152470 con 0x7f8fcc0ce050 2024-01-15T18:16:34.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.443+0000 7f8fd3200640 1 -- 172.21.15.27:0/1304880041 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fcc15b350 con 0x7f8fcc0ce050 2024-01-15T18:16:34.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.447+0000 7f8fc3e00640 1 -- 172.21.15.27:0/1304880041 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 32) v1 ==== 87384+0+0 (secure 0 0 0) 0x7f8fc40b8d00 con 0x7f8fcc0ce050 2024-01-15T18:16:34.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.447+0000 7f8fc3e00640 1 --2- 172.21.15.27:0/1304880041 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8fa4067670 0x7f8fa4069b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:34.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.447+0000 7f8fc3e00640 1 -- 172.21.15.27:0/1304880041 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f8fc411e350 con 0x7f8fcc0ce050 2024-01-15T18:16:34.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.447+0000 7f8fcbe00640 1 --2- 172.21.15.27:0/1304880041 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8fa4067670 0x7f8fa4069b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:34.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.447+0000 7f8fcbe00640 1 --2- 172.21.15.27:0/1304880041 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8fa4067670 0x7f8fa4069b30 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc0011f0 tx=0x7f8fbc007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:34.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.451+0000 7f8fc3e00640 1 -- 172.21.15.27:0/1304880041 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8fcc15b350 con 0x7f8fcc0ce050 2024-01-15T18:16:34.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.639+0000 7f8fd3200640 1 -- 172.21.15.27:0/1304880041 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8fcc0b49c0 con 0x7f8fa4067670 2024-01-15T18:16:34.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.647+0000 7f8fc3e00640 1 -- 172.21.15.27:0/1304880041 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f8fbc006b50 con 0x7f8fa4067670 2024-01-15T18:16:34.646 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:16:34.646 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:16:34.646 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "pending", 2024-01-15T18:16:34.647 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:16:34.647 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:16:34.647 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:16:34.647 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:16:34.647 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:34.647 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:34.647 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:16:34.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.647+0000 7f8fd3200640 1 -- 172.21.15.27:0/1304880041 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8fa4067670 msgr2=0x7f8fa4069b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:34.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.647+0000 7f8fd3200640 1 --2- 172.21.15.27:0/1304880041 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8fa4067670 0x7f8fa4069b30 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc0011f0 tx=0x7f8fbc007d40 comp rx=0 tx=0).stop 2024-01-15T18:16:34.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.651+0000 7f8fd3200640 1 -- 172.21.15.27:0/1304880041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc0ce050 msgr2=0x7f8fcc151a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:34.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.651+0000 7f8fd3200640 1 --2- 172.21.15.27:0/1304880041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc0ce050 0x7f8fcc151a20 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f8fc40af040 tx=0x7f8fc4019660 comp rx=0 tx=0).stop 2024-01-15T18:16:34.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.651+0000 7f8fd3200640 1 -- 172.21.15.27:0/1304880041 shutdown_connections 2024-01-15T18:16:34.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.651+0000 7f8fd3200640 1 --2- 172.21.15.27:0/1304880041 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8fa4067670 0x7f8fa4069b30 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:34.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.651+0000 7f8fd3200640 1 --2- 172.21.15.27:0/1304880041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fcc0ce050 0x7f8fcc151a20 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:34.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.651+0000 7f8fd3200640 1 --2- 172.21.15.27:0/1304880041 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8fcc0b1db0 0x7f8fcc1569b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:34.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.651+0000 7f8fd3200640 1 --2- 172.21.15.27:0/1304880041 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8fcc06c430 0x7f8fcc15b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:34.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.651+0000 7f8fd3200640 1 -- 172.21.15.27:0/1304880041 >> 172.21.15.27:0/1304880041 conn(0x7f8fcc0cb700 msgr2=0x7f8fcc14df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:34.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.651+0000 7f8fd3200640 1 -- 172.21.15.27:0/1304880041 shutdown_connections 2024-01-15T18:16:34.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:34.651+0000 7f8fd3200640 1 -- 172.21.15.27:0/1304880041 wait complete. 2024-01-15T18:16:34.669 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-15T18:16:35.670 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:16:35.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.827+0000 7ff4cc400640 1 Processor -- start 2024-01-15T18:16:35.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.827+0000 7ff4cc400640 1 -- start start 2024-01-15T18:16:35.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.827+0000 7ff4cc400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff4c415d3b0 0x7ff4c40cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:35.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.827+0000 7ff4cc400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff4c406b600 0x7ff4c406bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:35.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.827+0000 7ff4cc400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c40b04b0 con 0x7ff4c406c220 2024-01-15T18:16:35.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.827+0000 7ff4cc400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c406c6b0 con 0x7ff4c406b600 2024-01-15T18:16:35.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.827+0000 7ff4cc400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c40d0160 con 0x7ff4c415d3b0 2024-01-15T18:16:35.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.831+0000 7ff4c9000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff4c406b600 0x7ff4c406bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:35.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.831+0000 7ff4ca400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff4c406c220 0x7ff4c4160f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40592/0 (socket says 172.21.15.27:40592) 2024-01-15T18:16:35.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.831+0000 7ff4ca400640 1 -- 172.21.15.27:0/4213732650 learned_addr learned my addr 172.21.15.27:0/4213732650 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:16:35.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.831+0000 7ff4c9a00640 1 --2- 172.21.15.27:0/4213732650 >> v2:172.21.15.93:3300/0 conn(0x7ff4c415d3b0 0x7ff4c40cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:35.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.831+0000 7ff4c3e00640 1 -- 172.21.15.27:0/4213732650 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232743963 0 0) 0x7ff4c40b04b0 con 0x7ff4c406c220 2024-01-15T18:16:35.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.831+0000 7ff4c3e00640 1 -- 172.21.15.27:0/4213732650 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4a4003530 con 0x7ff4c406c220 2024-01-15T18:16:35.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.831+0000 7ff4c9000640 1 -- 172.21.15.27:0/4213732650 >> v2:172.21.15.93:3300/0 conn(0x7ff4c415d3b0 msgr2=0x7ff4c40cfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:35.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.831+0000 7ff4c9000640 1 --2- 172.21.15.27:0/4213732650 >> v2:172.21.15.93:3300/0 conn(0x7ff4c415d3b0 0x7ff4c40cfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:35.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.831+0000 7ff4c9000640 1 -- 172.21.15.27:0/4213732650 >> v1:172.21.15.27:6789/0 conn(0x7ff4c406c220 legacy=0x7ff4c4160f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:35.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.831+0000 7ff4c9000640 1 -- 172.21.15.27:0/4213732650 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c40b0720 con 0x7ff4c406b600 2024-01-15T18:16:35.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.831+0000 7ff4c9000640 1 --2- 172.21.15.27:0/4213732650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4c406b600 0x7ff4c406bce0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4072890 tx=0x7ff4ac01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8534a14e6fef355 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:35.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.831+0000 7ff4c3e00640 1 -- 172.21.15.27:0/4213732650 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac0a7070 con 0x7ff4c406b600 2024-01-15T18:16:35.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.831+0000 7ff4c3e00640 1 -- 172.21.15.27:0/4213732650 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4ac0ab430 con 0x7ff4c406b600 2024-01-15T18:16:35.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.831+0000 7ff4c3e00640 1 -- 172.21.15.27:0/4213732650 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac0ab7c0 con 0x7ff4c406b600 2024-01-15T18:16:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.835+0000 7ff4cc400640 1 -- 172.21.15.27:0/4213732650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4c406b600 msgr2=0x7ff4c406bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.835+0000 7ff4cc400640 1 --2- 172.21.15.27:0/4213732650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4c406b600 0x7ff4c406bce0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4072890 tx=0x7ff4ac01c430 comp rx=0 tx=0).stop 2024-01-15T18:16:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.835+0000 7ff4cc400640 1 -- 172.21.15.27:0/4213732650 shutdown_connections 2024-01-15T18:16:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.835+0000 7ff4cc400640 1 --2- 172.21.15.27:0/4213732650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4c406b600 0x7ff4c406bce0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.835+0000 7ff4cc400640 1 --2- 172.21.15.27:0/4213732650 >> v2:172.21.15.93:3300/0 conn(0x7ff4c415d3b0 0x7ff4c40cfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.835+0000 7ff4cc400640 1 -- 172.21.15.27:0/4213732650 >> 172.21.15.27:0/4213732650 conn(0x7ff4c40d0ed0 msgr2=0x7ff4c40d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.835+0000 7ff4cc400640 1 -- 172.21.15.27:0/4213732650 shutdown_connections 2024-01-15T18:16:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.835+0000 7ff4cc400640 1 -- 172.21.15.27:0/4213732650 wait complete. 2024-01-15T18:16:35.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.835+0000 7ff4cc400640 1 Processor -- start 2024-01-15T18:16:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.835+0000 7ff4cc400640 1 -- start start 2024-01-15T18:16:35.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.835+0000 7ff4cc400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4c406b600 0x7ff4c40af5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:35.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.835+0000 7ff4cc400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4c406c220 0x7ff4c40c71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:35.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.835+0000 7ff4cc400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff4c415d3b0 0x7ff4c40c76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:35.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.835+0000 7ff4cc400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c4154960 con 0x7ff4c406b600 2024-01-15T18:16:35.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.835+0000 7ff4cc400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c4153ce0 con 0x7ff4c406c220 2024-01-15T18:16:35.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.835+0000 7ff4cc400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c4153fc0 con 0x7ff4c415d3b0 2024-01-15T18:16:35.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.835+0000 7ff4c9a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4c406b600 0x7ff4c40af5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:35.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.839+0000 7ff4c9a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4c406b600 0x7ff4c40af5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53218/0 (socket says 172.21.15.27:53218) 2024-01-15T18:16:35.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.839+0000 7ff4c9a00640 1 -- 172.21.15.27:0/3392417099 learned_addr learned my addr 172.21.15.27:0/3392417099 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:35.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.839+0000 7ff4c9000640 1 --2- 172.21.15.27:0/3392417099 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4c406c220 0x7ff4c40c71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:35.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.839+0000 7ff4ca400640 1 --2- 172.21.15.27:0/3392417099 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff4c415d3b0 0x7ff4c40c76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:35.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.839+0000 7ff4c9a00640 1 -- 172.21.15.27:0/3392417099 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff4c415d3b0 msgr2=0x7ff4c40c76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:35.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.839+0000 7ff4c9a00640 1 --2- 172.21.15.27:0/3392417099 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff4c415d3b0 0x7ff4c40c76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:35.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.839+0000 7ff4c9a00640 1 -- 172.21.15.27:0/3392417099 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4c406c220 msgr2=0x7ff4c40c71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:35.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.839+0000 7ff4c9a00640 1 --2- 172.21.15.27:0/3392417099 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4c406c220 0x7ff4c40c71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:35.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.839+0000 7ff4c9a00640 1 -- 172.21.15.27:0/3392417099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ac009420 con 0x7ff4c406b600 2024-01-15T18:16:35.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.839+0000 7ff4c9a00640 1 --2- 172.21.15.27:0/3392417099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4c406b600 0x7ff4c40af5e0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7ff4b400e930 tx=0x7ff4b400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:35.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.839+0000 7ff4c1600640 1 -- 172.21.15.27:0/3392417099 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4009690 con 0x7ff4c406b600 2024-01-15T18:16:35.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.839+0000 7ff4c1600640 1 -- 172.21.15.27:0/3392417099 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b400ca00 con 0x7ff4c406b600 2024-01-15T18:16:35.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.839+0000 7ff4cc400640 1 -- 172.21.15.27:0/3392417099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c41542a0 con 0x7ff4c406b600 2024-01-15T18:16:35.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.839+0000 7ff4c1600640 1 -- 172.21.15.27:0/3392417099 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b400cd70 con 0x7ff4c406b600 2024-01-15T18:16:35.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.839+0000 7ff4cc400640 1 -- 172.21.15.27:0/3392417099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c40c7c20 con 0x7ff4c406b600 2024-01-15T18:16:35.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.839+0000 7ff4cc400640 1 -- 172.21.15.27:0/3392417099 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c406d550 con 0x7ff4c406b600 2024-01-15T18:16:35.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.839+0000 7ff4c1600640 1 -- 172.21.15.27:0/3392417099 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 32) v1 ==== 87384+0+0 (secure 0 0 0) 0x7ff4b4013470 con 0x7ff4c406b600 2024-01-15T18:16:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.843+0000 7ff4c1600640 1 --2- 172.21.15.27:0/3392417099 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff49c067670 0x7ff49c069b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.843+0000 7ff4c1600640 1 -- 172.21.15.27:0/3392417099 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7ff4b4083300 con 0x7ff4c406b600 2024-01-15T18:16:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.843+0000 7ff4c9000640 1 --2- 172.21.15.27:0/3392417099 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff49c067670 0x7ff49c069b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:35.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.843+0000 7ff4c9000640 1 --2- 172.21.15.27:0/3392417099 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff49c067670 0x7ff49c069b30 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac0af000 tx=0x7ff4ac0a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:35.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:35.843+0000 7ff4c1600640 1 -- 172.21.15.27:0/3392417099 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff4b4051e50 con 0x7ff4c406b600 2024-01-15T18:16:36.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:36.031+0000 7ff4cc400640 1 -- 172.21.15.27:0/3392417099 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4c4066f40 con 0x7ff49c067670 2024-01-15T18:16:36.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:36.039+0000 7ff4c1600640 1 -- 172.21.15.27:0/3392417099 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7ff4c4066f40 con 0x7ff49c067670 2024-01-15T18:16:36.038 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:16:36.039 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:16:36.039 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "pending", 2024-01-15T18:16:36.039 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:16:36.039 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:16:36.039 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:16:36.039 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:16:36.039 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:36.039 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:36.039 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:16:36.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:36.043+0000 7ff4cc400640 1 -- 172.21.15.27:0/3392417099 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff49c067670 msgr2=0x7ff49c069b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:36.043+0000 7ff4cc400640 1 --2- 172.21.15.27:0/3392417099 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff49c067670 0x7ff49c069b30 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac0af000 tx=0x7ff4ac0a7000 comp rx=0 tx=0).stop 2024-01-15T18:16:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:36.043+0000 7ff4cc400640 1 -- 172.21.15.27:0/3392417099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4c406b600 msgr2=0x7ff4c40af5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:36.043+0000 7ff4cc400640 1 --2- 172.21.15.27:0/3392417099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4c406b600 0x7ff4c40af5e0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7ff4b400e930 tx=0x7ff4b400edf0 comp rx=0 tx=0).stop 2024-01-15T18:16:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:36.043+0000 7ff4cc400640 1 -- 172.21.15.27:0/3392417099 shutdown_connections 2024-01-15T18:16:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:36.043+0000 7ff4cc400640 1 --2- 172.21.15.27:0/3392417099 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff49c067670 0x7ff49c069b30 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:36.043+0000 7ff4cc400640 1 --2- 172.21.15.27:0/3392417099 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff4c415d3b0 0x7ff4c40c76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:36.043+0000 7ff4cc400640 1 --2- 172.21.15.27:0/3392417099 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff4c406c220 0x7ff4c40c71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:36.043+0000 7ff4cc400640 1 --2- 172.21.15.27:0/3392417099 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff4c406b600 0x7ff4c40af5e0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:36.043+0000 7ff4cc400640 1 -- 172.21.15.27:0/3392417099 >> 172.21.15.27:0/3392417099 conn(0x7ff4c40d0ed0 msgr2=0x7ff4c406fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:36.043+0000 7ff4cc400640 1 -- 172.21.15.27:0/3392417099 shutdown_connections 2024-01-15T18:16:36.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:36.043+0000 7ff4cc400640 1 -- 172.21.15.27:0/3392417099 wait complete. 2024-01-15T18:16:36.060 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-15T18:16:37.061 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:16:37.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.219+0000 7fc619c00640 1 Processor -- start 2024-01-15T18:16:37.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.219+0000 7fc619c00640 1 -- start start 2024-01-15T18:16:37.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.219+0000 7fc619c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc6140b56b0 0x7fc614168440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:37.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.219+0000 7fc619c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc61415d9e0 con 0x7fc6140d1ab0 2024-01-15T18:16:37.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.219+0000 7fc619c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc61406b9b0 con 0x7fc61415d600 2024-01-15T18:16:37.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.219+0000 7fc619c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc61406bc90 con 0x7fc6140b56b0 2024-01-15T18:16:37.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.219+0000 7fc612000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc6140d1ab0 0x7fc614161820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40608/0 (socket says 172.21.15.27:40608) 2024-01-15T18:16:37.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.219+0000 7fc612000640 1 -- 172.21.15.27:0/312430524 learned_addr learned my addr 172.21.15.27:0/312430524 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:16:37.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.219+0000 7fc613400640 1 --2- 172.21.15.27:0/312430524 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc6140b56b0 0x7fc614168440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:37.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.219+0000 7fc611600640 1 -- 172.21.15.27:0/312430524 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3218873057 0 0) 0x7fc61415d9e0 con 0x7fc6140d1ab0 2024-01-15T18:16:37.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.219+0000 7fc611600640 1 -- 172.21.15.27:0/312430524 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5f4003530 con 0x7fc6140d1ab0 2024-01-15T18:16:37.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.219+0000 7fc611600640 1 -- 172.21.15.27:0/312430524 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559577939 0 0) 0x7fc61406b9b0 con 0x7fc61415d600 2024-01-15T18:16:37.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.219+0000 7fc611600640 1 -- 172.21.15.27:0/312430524 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc61415d9e0 con 0x7fc61415d600 2024-01-15T18:16:37.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.219+0000 7fc613400640 1 -- 172.21.15.27:0/312430524 >> v1:172.21.15.93:6789/0 conn(0x7fc61415d600 legacy=0x7fc614161110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:37.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.219+0000 7fc613400640 1 -- 172.21.15.27:0/312430524 >> v1:172.21.15.27:6789/0 conn(0x7fc6140d1ab0 legacy=0x7fc614161820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:37.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.219+0000 7fc613400640 1 -- 172.21.15.27:0/312430524 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6140b3ea0 con 0x7fc6140b56b0 2024-01-15T18:16:37.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.223+0000 7fc613400640 1 --2- 172.21.15.27:0/312430524 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc6140b56b0 0x7fc614168440 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc60c009dc0 tx=0x7fc60c01c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bffbb578cca23218 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:37.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.223+0000 7fc611600640 1 -- 172.21.15.27:0/312430524 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c0a5940 con 0x7fc6140b56b0 2024-01-15T18:16:37.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.223+0000 7fc611600640 1 -- 172.21.15.27:0/312430524 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc60c0a5aa0 con 0x7fc6140b56b0 2024-01-15T18:16:37.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.223+0000 7fc611600640 1 -- 172.21.15.27:0/312430524 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c0a5e10 con 0x7fc6140b56b0 2024-01-15T18:16:37.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.223+0000 7fc619c00640 1 -- 172.21.15.27:0/312430524 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc6140b56b0 msgr2=0x7fc614168440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:37.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.223+0000 7fc619c00640 1 --2- 172.21.15.27:0/312430524 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc6140b56b0 0x7fc614168440 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc60c009dc0 tx=0x7fc60c01c970 comp rx=0 tx=0).stop 2024-01-15T18:16:37.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.223+0000 7fc619c00640 1 -- 172.21.15.27:0/312430524 shutdown_connections 2024-01-15T18:16:37.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.223+0000 7fc619c00640 1 --2- 172.21.15.27:0/312430524 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc6140b56b0 0x7fc614168440 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:37.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.223+0000 7fc619c00640 1 -- 172.21.15.27:0/312430524 >> 172.21.15.27:0/312430524 conn(0x7fc6140d0eb0 msgr2=0x7fc6140d12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:37.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.223+0000 7fc619c00640 1 -- 172.21.15.27:0/312430524 shutdown_connections 2024-01-15T18:16:37.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.223+0000 7fc619c00640 1 -- 172.21.15.27:0/312430524 wait complete. 2024-01-15T18:16:37.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.227+0000 7fc619c00640 1 Processor -- start 2024-01-15T18:16:37.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.227+0000 7fc619c00640 1 -- start start 2024-01-15T18:16:37.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.227+0000 7fc619c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6140b56b0 0x7fc6140c4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:37.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.227+0000 7fc619c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6140d1ab0 0x7fc6140cb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:37.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.227+0000 7fc619c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc61415d600 0x7fc61415a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:37.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.227+0000 7fc619c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6140cbed0 con 0x7fc6140d1ab0 2024-01-15T18:16:37.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.227+0000 7fc619c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6140cdf20 con 0x7fc6140b56b0 2024-01-15T18:16:37.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.227+0000 7fc619c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6140ce200 con 0x7fc61415d600 2024-01-15T18:16:37.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.227+0000 7fc612000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6140d1ab0 0x7fc6140cb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:37.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.227+0000 7fc612a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6140b56b0 0x7fc6140c4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:37.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.227+0000 7fc613400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc61415d600 0x7fc61415a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:37.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.227+0000 7fc612a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6140b56b0 0x7fc6140c4770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:56170/0 (socket says 172.21.15.27:56170) 2024-01-15T18:16:37.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.227+0000 7fc612a00640 1 -- 172.21.15.27:0/2613382653 learned_addr learned my addr 172.21.15.27:0/2613382653 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.227+0000 7fc612000640 1 -- 172.21.15.27:0/2613382653 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc61415d600 msgr2=0x7fc61415a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.227+0000 7fc612000640 1 --2- 172.21.15.27:0/2613382653 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc61415d600 0x7fc61415a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.227+0000 7fc612000640 1 -- 172.21.15.27:0/2613382653 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6140b56b0 msgr2=0x7fc6140c4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.227+0000 7fc612000640 1 --2- 172.21.15.27:0/2613382653 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6140b56b0 0x7fc6140c4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.227+0000 7fc612000640 1 -- 172.21.15.27:0/2613382653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc60c009a70 con 0x7fc6140d1ab0 2024-01-15T18:16:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.227+0000 7fc612a00640 1 --2- 172.21.15.27:0/2613382653 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6140b56b0 0x7fc6140c4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:16:37.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.227+0000 7fc612000640 1 --2- 172.21.15.27:0/2613382653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6140d1ab0 0x7fc6140cb6d0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc00c400 tx=0x7fc5fc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:37.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.231+0000 7fc60aa00640 1 -- 172.21.15.27:0/2613382653 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc005cf0 con 0x7fc6140d1ab0 2024-01-15T18:16:37.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.231+0000 7fc60aa00640 1 -- 172.21.15.27:0/2613382653 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5fc005e90 con 0x7fc6140d1ab0 2024-01-15T18:16:37.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.231+0000 7fc619c00640 1 -- 172.21.15.27:0/2613382653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6140cc120 con 0x7fc6140d1ab0 2024-01-15T18:16:37.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.231+0000 7fc60aa00640 1 -- 172.21.15.27:0/2613382653 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc007310 con 0x7fc6140d1ab0 2024-01-15T18:16:37.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.231+0000 7fc619c00640 1 -- 172.21.15.27:0/2613382653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc61415a940 con 0x7fc6140d1ab0 2024-01-15T18:16:37.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.231+0000 7fc619c00640 1 -- 172.21.15.27:0/2613382653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6140ce420 con 0x7fc6140d1ab0 2024-01-15T18:16:37.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.231+0000 7fc60aa00640 1 -- 172.21.15.27:0/2613382653 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 32) v1 ==== 87384+0+0 (secure 0 0 0) 0x7fc5fc007470 con 0x7fc6140d1ab0 2024-01-15T18:16:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.231+0000 7fc60aa00640 1 --2- 172.21.15.27:0/2613382653 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc5ec0676a0 0x7fc5ec069b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.231+0000 7fc60aa00640 1 -- 172.21.15.27:0/2613382653 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fc5fc082610 con 0x7fc6140d1ab0 2024-01-15T18:16:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.235+0000 7fc612a00640 1 --2- 172.21.15.27:0/2613382653 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc5ec0676a0 0x7fc5ec069b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:37.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.235+0000 7fc612a00640 1 --2- 172.21.15.27:0/2613382653 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc5ec0676a0 0x7fc5ec069b60 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fc614155b90 tx=0x7fc604009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:37.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.235+0000 7fc60aa00640 1 -- 172.21.15.27:0/2613382653 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc5fc051140 con 0x7fc6140d1ab0 2024-01-15T18:16:37.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:37.427+0000 7fc619c00640 1 -- 172.21.15.27:0/2613382653 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6140d0480 con 0x7fc5ec0676a0 2024-01-15T18:16:40.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:40.291+0000 7fc60aa00640 1 -- 172.21.15.27:0/2613382653 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7fc6140d0480 con 0x7fc5ec0676a0 2024-01-15T18:16:40.290 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:16:40.290 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:16:40.290 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:16:40.291 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:16:40.291 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:16:40.291 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:16:40.291 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:16:40.291 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:40.291 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:40.291 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:16:40.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:40.295+0000 7fc619c00640 1 -- 172.21.15.27:0/2613382653 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc5ec0676a0 msgr2=0x7fc5ec069b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:40.295+0000 7fc619c00640 1 --2- 172.21.15.27:0/2613382653 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc5ec0676a0 0x7fc5ec069b60 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fc614155b90 tx=0x7fc604009040 comp rx=0 tx=0).stop 2024-01-15T18:16:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:40.295+0000 7fc619c00640 1 -- 172.21.15.27:0/2613382653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6140d1ab0 msgr2=0x7fc6140cb6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:40.295+0000 7fc619c00640 1 --2- 172.21.15.27:0/2613382653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6140d1ab0 0x7fc6140cb6d0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc00c400 tx=0x7fc5fc00c8c0 comp rx=0 tx=0).stop 2024-01-15T18:16:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:40.295+0000 7fc619c00640 1 -- 172.21.15.27:0/2613382653 shutdown_connections 2024-01-15T18:16:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:40.295+0000 7fc619c00640 1 --2- 172.21.15.27:0/2613382653 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc5ec0676a0 0x7fc5ec069b60 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:40.295+0000 7fc619c00640 1 --2- 172.21.15.27:0/2613382653 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc61415d600 0x7fc61415a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:40.295+0000 7fc619c00640 1 --2- 172.21.15.27:0/2613382653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6140d1ab0 0x7fc6140cb6d0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:40.295+0000 7fc619c00640 1 --2- 172.21.15.27:0/2613382653 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6140b56b0 0x7fc6140c4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:40.295+0000 7fc619c00640 1 -- 172.21.15.27:0/2613382653 >> 172.21.15.27:0/2613382653 conn(0x7fc6140d0eb0 msgr2=0x7fc614153150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:40.295+0000 7fc619c00640 1 -- 172.21.15.27:0/2613382653 shutdown_connections 2024-01-15T18:16:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:40.295+0000 7fc619c00640 1 -- 172.21.15.27:0/2613382653 wait complete. 2024-01-15T18:16:42.317 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:16:42.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.503+0000 7f56ac400640 1 Processor -- start 2024-01-15T18:16:42.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.503+0000 7f56ac400640 1 -- start start 2024-01-15T18:16:42.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.503+0000 7f56ac400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56a40cf460 0x7f56a40cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:42.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.503+0000 7f56ac400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f56a40b2590 0x7f56a40cc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:42.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.503+0000 7f56ac400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a40cfd80 con 0x7f56a415b4b0 2024-01-15T18:16:42.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.503+0000 7f56ac400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56a40d19a0 con 0x7f56a40b2590 2024-01-15T18:16:42.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.503+0000 7f56ac400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56a40d1c80 con 0x7f56a40cf460 2024-01-15T18:16:42.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.507+0000 7f56a9a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56a40cf460 0x7f56a40cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:42.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.507+0000 7f56a9a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56a40cf460 0x7f56a40cf840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:34180/0 (socket says 172.21.15.27:34180) 2024-01-15T18:16:42.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.507+0000 7f56a9a00640 1 -- 172.21.15.27:0/312727699 learned_addr learned my addr 172.21.15.27:0/312727699 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:42.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.507+0000 7f56a9a00640 1 -- 172.21.15.27:0/312727699 >> v2:172.21.15.93:3300/0 conn(0x7f56a40b2590 msgr2=0x7f56a40cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:42.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.507+0000 7f56a9a00640 1 --2- 172.21.15.27:0/312727699 >> v2:172.21.15.93:3300/0 conn(0x7f56a40b2590 0x7f56a40cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:42.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.507+0000 7f56a9a00640 1 -- 172.21.15.27:0/312727699 >> v1:172.21.15.93:6789/0 conn(0x7f56a415b4b0 legacy=0x7f56a40cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:42.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.507+0000 7f56a9a00640 1 -- 172.21.15.27:0/312727699 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a40b07c0 con 0x7f56a40cf460 2024-01-15T18:16:42.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.511+0000 7f56a9a00640 1 --2- 172.21.15.27:0/312727699 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56a40cf460 0x7f56a40cf840 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5694009870 tx=0x7f569401c500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e2fa9adfd9c6d18 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:42.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.515+0000 7f56a3e00640 1 -- 172.21.15.27:0/312727699 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56940aa620 con 0x7f56a40cf460 2024-01-15T18:16:42.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.515+0000 7f56a3e00640 1 -- 172.21.15.27:0/312727699 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56940aa780 con 0x7f56a40cf460 2024-01-15T18:16:42.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.515+0000 7f56a3e00640 1 -- 172.21.15.27:0/312727699 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56940aab10 con 0x7f56a40cf460 2024-01-15T18:16:42.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.515+0000 7f56ac400640 1 -- 172.21.15.27:0/312727699 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56a40cf460 msgr2=0x7f56a40cf840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:42.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.515+0000 7f56ac400640 1 --2- 172.21.15.27:0/312727699 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56a40cf460 0x7f56a40cf840 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5694009870 tx=0x7f569401c500 comp rx=0 tx=0).stop 2024-01-15T18:16:42.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.519+0000 7f56ac400640 1 -- 172.21.15.27:0/312727699 shutdown_connections 2024-01-15T18:16:42.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.519+0000 7f56ac400640 1 --2- 172.21.15.27:0/312727699 >> v2:172.21.15.93:3300/0 conn(0x7f56a40b2590 0x7f56a40cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:42.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.519+0000 7f56ac400640 1 --2- 172.21.15.27:0/312727699 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56a40cf460 0x7f56a40cf840 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:42.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.519+0000 7f56ac400640 1 -- 172.21.15.27:0/312727699 >> 172.21.15.27:0/312727699 conn(0x7f56a40d0e80 msgr2=0x7f56a40d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:42.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.519+0000 7f56ac400640 1 -- 172.21.15.27:0/312727699 shutdown_connections 2024-01-15T18:16:42.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.519+0000 7f56ac400640 1 -- 172.21.15.27:0/312727699 wait complete. 2024-01-15T18:16:42.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.523+0000 7f56ac400640 1 Processor -- start 2024-01-15T18:16:42.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.523+0000 7f56ac400640 1 -- start start 2024-01-15T18:16:42.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.523+0000 7f56ac400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56a40b2590 0x7f56a40c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:42.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.523+0000 7f56ac400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56a40cf460 0x7f56a415e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:42.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.523+0000 7f56ac400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56a415b4b0 0x7f56a413dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:42.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.523+0000 7f56ac400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a4157b90 con 0x7f56a40cf460 2024-01-15T18:16:42.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.523+0000 7f56ac400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a4158fb0 con 0x7f56a40b2590 2024-01-15T18:16:42.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.523+0000 7f56ac400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56a41575d0 con 0x7f56a415b4b0 2024-01-15T18:16:42.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.527+0000 7f56aa400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56a415b4b0 0x7f56a413dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:42.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.527+0000 7f56aa400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56a415b4b0 0x7f56a413dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:34194/0 (socket says 172.21.15.27:34194) 2024-01-15T18:16:42.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.527+0000 7f56a9a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56a40b2590 0x7f56a40c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:42.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.527+0000 7f56a9a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56a40b2590 0x7f56a40c6c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:56184/0 (socket says 172.21.15.27:56184) 2024-01-15T18:16:42.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.527+0000 7f56a9000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56a40cf460 0x7f56a415e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:42.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.527+0000 7f56aa400640 1 -- 172.21.15.27:0/2542044825 learned_addr learned my addr 172.21.15.27:0/2542044825 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:42.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.527+0000 7f56a9a00640 1 -- 172.21.15.27:0/2542044825 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56a415b4b0 msgr2=0x7f56a413dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:42.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.527+0000 7f56a9a00640 1 --2- 172.21.15.27:0/2542044825 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56a415b4b0 0x7f56a413dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:42.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.527+0000 7f56a9a00640 1 -- 172.21.15.27:0/2542044825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56a40cf460 msgr2=0x7f56a415e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:42.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.527+0000 7f56a9a00640 1 --2- 172.21.15.27:0/2542044825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56a40cf460 0x7f56a415e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:42.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.527+0000 7f56a9a00640 1 -- 172.21.15.27:0/2542044825 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56940094d0 con 0x7f56a40b2590 2024-01-15T18:16:42.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.527+0000 7f56a9000640 1 --2- 172.21.15.27:0/2542044825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56a40cf460 0x7f56a415e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:16:42.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.527+0000 7f56a9a00640 1 --2- 172.21.15.27:0/2542044825 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56a40b2590 0x7f56a40c6c80 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f56940ab000 tx=0x7f569401ad40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:42.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.527+0000 7f56a1600640 1 -- 172.21.15.27:0/2542044825 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56940bd5f0 con 0x7f56a40b2590 2024-01-15T18:16:42.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.527+0000 7f56a1600640 1 -- 172.21.15.27:0/2542044825 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56940bd750 con 0x7f56a40b2590 2024-01-15T18:16:42.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.527+0000 7f56ac400640 1 -- 172.21.15.27:0/2542044825 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a41578b0 con 0x7f56a40b2590 2024-01-15T18:16:42.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.531+0000 7f56ac400640 1 -- 172.21.15.27:0/2542044825 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a4159530 con 0x7f56a40b2590 2024-01-15T18:16:42.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.531+0000 7f56a1600640 1 -- 172.21.15.27:0/2542044825 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56940bdb00 con 0x7f56a40b2590 2024-01-15T18:16:42.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.531+0000 7f56aa400640 1 --2- 172.21.15.27:0/2542044825 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56a415b4b0 0x7f56a413dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:16:42.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.531+0000 7f56ac400640 1 -- 172.21.15.27:0/2542044825 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5674005260 con 0x7f56a40b2590 2024-01-15T18:16:42.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.531+0000 7f56a1600640 1 -- 172.21.15.27:0/2542044825 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 32) v1 ==== 87384+0+0 (secure 0 0 0) 0x7f56940b33f0 con 0x7f56a40b2590 2024-01-15T18:16:42.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.539+0000 7f56a1600640 1 --2- 172.21.15.27:0/2542044825 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f567c0676f0 0x7f567c069bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:42.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.539+0000 7f56a1600640 1 -- 172.21.15.27:0/2542044825 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5822+0+0 (secure 0 0 0) 0x7f569411d7d0 con 0x7f56a40b2590 2024-01-15T18:16:42.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.539+0000 7f56a9000640 1 --2- 172.21.15.27:0/2542044825 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f567c0676f0 0x7f567c069bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:42.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.539+0000 7f56a1600640 1 -- 172.21.15.27:0/2542044825 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f56941261f0 con 0x7f56a40b2590 2024-01-15T18:16:42.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.543+0000 7f56a9000640 1 --2- 172.21.15.27:0/2542044825 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f567c0676f0 0x7f567c069bb0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f56a4151d60 tx=0x7f568c009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:42.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.835+0000 7f56ac400640 1 -- 172.21.15.27:0/2542044825 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5674000bd0 con 0x7f567c0676f0 2024-01-15T18:16:42.845 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:16:42.845 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:16:42.845 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:16:42.845 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:16:42.845 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:16:42.845 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:16:42.845 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:16:42.845 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:16:42.845 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:16:42.845 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 0.0, 2024-01-15T18:16:42.846 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "0 B/3 GB", 2024-01-15T18:16:42.846 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "0/3", 2024-01-15T18:16:42.846 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:16:42.846 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:16:42.846 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:42.846 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:42.846 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:16:42.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.843+0000 7f56a1600640 1 -- 172.21.15.27:0/2542044825 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f5674000bd0 con 0x7f567c0676f0 2024-01-15T18:16:42.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.847+0000 7f56ac400640 1 -- 172.21.15.27:0/2542044825 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f567c0676f0 msgr2=0x7f567c069bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:42.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.847+0000 7f56ac400640 1 --2- 172.21.15.27:0/2542044825 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f567c0676f0 0x7f567c069bb0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f56a4151d60 tx=0x7f568c009060 comp rx=0 tx=0).stop 2024-01-15T18:16:42.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.847+0000 7f56ac400640 1 -- 172.21.15.27:0/2542044825 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56a40b2590 msgr2=0x7f56a40c6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:42.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.847+0000 7f56ac400640 1 --2- 172.21.15.27:0/2542044825 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56a40b2590 0x7f56a40c6c80 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f56940ab000 tx=0x7f569401ad40 comp rx=0 tx=0).stop 2024-01-15T18:16:42.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.851+0000 7f56ac400640 1 -- 172.21.15.27:0/2542044825 shutdown_connections 2024-01-15T18:16:42.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.851+0000 7f56ac400640 1 --2- 172.21.15.27:0/2542044825 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f567c0676f0 0x7f567c069bb0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:42.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.851+0000 7f56ac400640 1 --2- 172.21.15.27:0/2542044825 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f56a415b4b0 0x7f56a413dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:42.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.851+0000 7f56ac400640 1 --2- 172.21.15.27:0/2542044825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56a40cf460 0x7f56a415e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:42.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.851+0000 7f56ac400640 1 --2- 172.21.15.27:0/2542044825 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56a40b2590 0x7f56a40c6c80 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:42.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.851+0000 7f56ac400640 1 -- 172.21.15.27:0/2542044825 >> 172.21.15.27:0/2542044825 conn(0x7f56a40d0e80 msgr2=0x7f56a40cd370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:42.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.851+0000 7f56ac400640 1 -- 172.21.15.27:0/2542044825 shutdown_connections 2024-01-15T18:16:42.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:42.855+0000 7f56ac400640 1 -- 172.21.15.27:0/2542044825 wait complete. 2024-01-15T18:16:42.881 INFO:tasks.cephfs.test_volumes:percetange completed = 0.0 2024-01-15T18:16:44.882 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:16:45.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.035+0000 7fab25200640 1 Processor -- start 2024-01-15T18:16:45.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.035+0000 7fab25200640 1 -- start start 2024-01-15T18:16:45.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.039+0000 7fab25200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fab2015f550 0x7fab200cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:45.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.039+0000 7fab25200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab200b1cb0 con 0x7fab20158900 2024-01-15T18:16:45.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.039+0000 7fab25200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab2015fa00 con 0x7fab200cf110 2024-01-15T18:16:45.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.039+0000 7fab25200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab200cf580 con 0x7fab2015f550 2024-01-15T18:16:45.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.039+0000 7fab1de00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fab20158900 0x7fab2015bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45774/0 (socket says 172.21.15.27:45774) 2024-01-15T18:16:45.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.039+0000 7fab1d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fab2015f550 0x7fab200cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:45.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.039+0000 7fab1de00640 1 -- 172.21.15.27:0/3243749837 learned_addr learned my addr 172.21.15.27:0/3243749837 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:16:45.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.039+0000 7fab1d400640 1 -- 172.21.15.27:0/3243749837 >> v1:172.21.15.93:6789/0 conn(0x7fab200cf110 legacy=0x7fab200701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:45.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.039+0000 7fab1d400640 1 -- 172.21.15.27:0/3243749837 >> v1:172.21.15.27:6789/0 conn(0x7fab20158900 legacy=0x7fab2015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:45.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.039+0000 7fab1d400640 1 -- 172.21.15.27:0/3243749837 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab200b1f00 con 0x7fab2015f550 2024-01-15T18:16:45.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.043+0000 7fab1d400640 1 --2- 172.21.15.27:0/3243749837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab2015f550 0x7fab200cebd0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fab080097f0 tx=0x7fab0801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46e0921de7b0e5f9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:45.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.043+0000 7fab1ca00640 1 -- 172.21.15.27:0/3243749837 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab080ab650 con 0x7fab2015f550 2024-01-15T18:16:45.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.043+0000 7fab1ca00640 1 -- 172.21.15.27:0/3243749837 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab080ab7b0 con 0x7fab2015f550 2024-01-15T18:16:45.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.043+0000 7fab1ca00640 1 -- 172.21.15.27:0/3243749837 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab080abb40 con 0x7fab2015f550 2024-01-15T18:16:45.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.043+0000 7fab25200640 1 -- 172.21.15.27:0/3243749837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab2015f550 msgr2=0x7fab200cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:45.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.043+0000 7fab25200640 1 --2- 172.21.15.27:0/3243749837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab2015f550 0x7fab200cebd0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fab080097f0 tx=0x7fab0801c450 comp rx=0 tx=0).stop 2024-01-15T18:16:45.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.043+0000 7fab25200640 1 -- 172.21.15.27:0/3243749837 shutdown_connections 2024-01-15T18:16:45.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.043+0000 7fab25200640 1 --2- 172.21.15.27:0/3243749837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab2015f550 0x7fab200cebd0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:45.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.043+0000 7fab25200640 1 -- 172.21.15.27:0/3243749837 >> 172.21.15.27:0/3243749837 conn(0x7fab200cb700 msgr2=0x7fab200cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:45.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.043+0000 7fab25200640 1 -- 172.21.15.27:0/3243749837 shutdown_connections 2024-01-15T18:16:45.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.043+0000 7fab25200640 1 -- 172.21.15.27:0/3243749837 wait complete. 2024-01-15T18:16:45.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.043+0000 7fab25200640 1 Processor -- start 2024-01-15T18:16:45.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.043+0000 7fab25200640 1 -- start start 2024-01-15T18:16:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.047+0000 7fab25200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab200cf110 0x7fab2014bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.047+0000 7fab25200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fab20158900 0x7fab20159e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.047+0000 7fab25200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab2015f550 0x7fab2015a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.047+0000 7fab25200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab200cd070 con 0x7fab200cf110 2024-01-15T18:16:45.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.047+0000 7fab25200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2014dc50 con 0x7fab2015f550 2024-01-15T18:16:45.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.047+0000 7fab25200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab200cc9b0 con 0x7fab20158900 2024-01-15T18:16:45.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.047+0000 7fab1de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab200cf110 0x7fab2014bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:45.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.047+0000 7fab1de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab200cf110 0x7fab2014bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55266/0 (socket says 172.21.15.27:55266) 2024-01-15T18:16:45.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.047+0000 7fab1de00640 1 -- 172.21.15.27:0/1417288004 learned_addr learned my addr 172.21.15.27:0/1417288004 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:45.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.047+0000 7fab1e800640 1 --2- 172.21.15.27:0/1417288004 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab2015f550 0x7fab2015a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:45.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.047+0000 7fab1d400640 1 --2- 172.21.15.27:0/1417288004 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fab20158900 0x7fab20159e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:45.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.047+0000 7fab1de00640 1 -- 172.21.15.27:0/1417288004 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fab20158900 msgr2=0x7fab20159e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:45.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.047+0000 7fab1de00640 1 --2- 172.21.15.27:0/1417288004 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fab20158900 0x7fab20159e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:45.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.047+0000 7fab1de00640 1 -- 172.21.15.27:0/1417288004 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab2015f550 msgr2=0x7fab2015a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:45.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.047+0000 7fab1de00640 1 --2- 172.21.15.27:0/1417288004 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab2015f550 0x7fab2015a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:45.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.047+0000 7fab1de00640 1 -- 172.21.15.27:0/1417288004 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab08009420 con 0x7fab200cf110 2024-01-15T18:16:45.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.047+0000 7fab1de00640 1 --2- 172.21.15.27:0/1417288004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab200cf110 0x7fab2014bd90 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fab1000c500 tx=0x7fab1000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:45.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.047+0000 7fab16000640 1 -- 172.21.15.27:0/1417288004 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab10005cf0 con 0x7fab200cf110 2024-01-15T18:16:45.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.047+0000 7fab16000640 1 -- 172.21.15.27:0/1417288004 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab10005e50 con 0x7fab200cf110 2024-01-15T18:16:45.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.047+0000 7fab16000640 1 -- 172.21.15.27:0/1417288004 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab100072f0 con 0x7fab200cf110 2024-01-15T18:16:45.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.047+0000 7fab25200640 1 -- 172.21.15.27:0/1417288004 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab200ccc90 con 0x7fab200cf110 2024-01-15T18:16:45.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.047+0000 7fab25200640 1 -- 172.21.15.27:0/1417288004 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab2014c310 con 0x7fab200cf110 2024-01-15T18:16:45.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.047+0000 7fab25200640 1 -- 172.21.15.27:0/1417288004 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab200cc810 con 0x7fab200cf110 2024-01-15T18:16:45.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.051+0000 7fab16000640 1 -- 172.21.15.27:0/1417288004 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 32) v1 ==== 87384+0+0 (secure 0 0 0) 0x7fab10007450 con 0x7fab200cf110 2024-01-15T18:16:45.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.051+0000 7fab16000640 1 --2- 172.21.15.27:0/1417288004 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faaf8067670 0x7faaf8069b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:45.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.051+0000 7fab16000640 1 -- 172.21.15.27:0/1417288004 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5822+0+0 (secure 0 0 0) 0x7fab10082600 con 0x7fab200cf110 2024-01-15T18:16:45.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.051+0000 7fab1d400640 1 --2- 172.21.15.27:0/1417288004 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faaf8067670 0x7faaf8069b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:45.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.051+0000 7fab1d400640 1 --2- 172.21.15.27:0/1417288004 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faaf8067670 0x7faaf8069b30 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fab080af000 tx=0x7fab080a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:45.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.055+0000 7fab16000640 1 -- 172.21.15.27:0/1417288004 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fab200cc810 con 0x7fab200cf110 2024-01-15T18:16:45.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.243+0000 7fab25200640 1 -- 172.21.15.27:0/1417288004 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fab20073570 con 0x7faaf8067670 2024-01-15T18:16:45.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.247+0000 7fab16000640 1 -- 172.21.15.27:0/1417288004 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fab20073570 con 0x7faaf8067670 2024-01-15T18:16:45.247 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:16:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:16:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:16:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:16:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:16:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:16:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:16:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:16:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:16:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 33.33333333333333, 2024-01-15T18:16:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "1 GB/3 GB", 2024-01-15T18:16:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "1/3", 2024-01-15T18:16:45.247 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:16:45.248 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:16:45.248 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:45.248 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:45.248 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:16:45.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.251+0000 7fab25200640 1 -- 172.21.15.27:0/1417288004 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faaf8067670 msgr2=0x7faaf8069b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:45.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.251+0000 7fab25200640 1 --2- 172.21.15.27:0/1417288004 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faaf8067670 0x7faaf8069b30 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fab080af000 tx=0x7fab080a7000 comp rx=0 tx=0).stop 2024-01-15T18:16:45.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.251+0000 7fab25200640 1 -- 172.21.15.27:0/1417288004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab200cf110 msgr2=0x7fab2014bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:45.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.251+0000 7fab25200640 1 --2- 172.21.15.27:0/1417288004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab200cf110 0x7fab2014bd90 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fab1000c500 tx=0x7fab1000c9c0 comp rx=0 tx=0).stop 2024-01-15T18:16:45.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.251+0000 7fab25200640 1 -- 172.21.15.27:0/1417288004 shutdown_connections 2024-01-15T18:16:45.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.251+0000 7fab25200640 1 --2- 172.21.15.27:0/1417288004 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faaf8067670 0x7faaf8069b30 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:45.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.251+0000 7fab25200640 1 --2- 172.21.15.27:0/1417288004 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fab2015f550 0x7fab2015a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:45.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.251+0000 7fab25200640 1 --2- 172.21.15.27:0/1417288004 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fab20158900 0x7fab20159e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:45.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.251+0000 7fab25200640 1 --2- 172.21.15.27:0/1417288004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fab200cf110 0x7fab2014bd90 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:45.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.251+0000 7fab25200640 1 -- 172.21.15.27:0/1417288004 >> 172.21.15.27:0/1417288004 conn(0x7fab200cb700 msgr2=0x7fab200b0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:45.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.251+0000 7fab25200640 1 -- 172.21.15.27:0/1417288004 shutdown_connections 2024-01-15T18:16:45.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:45.251+0000 7fab25200640 1 -- 172.21.15.27:0/1417288004 wait complete. 2024-01-15T18:16:45.269 INFO:tasks.cephfs.test_volumes:percetange completed = 33.33333333333333 2024-01-15T18:16:47.270 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:16:47.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.495+0000 7f694b000640 1 Processor -- start 2024-01-15T18:16:47.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.495+0000 7f694b000640 1 -- start start 2024-01-15T18:16:47.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.495+0000 7f694b000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6944071360 0x7f69440717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:47.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.495+0000 7f694b000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69440b4dd0 con 0x7f6944143a70 2024-01-15T18:16:47.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.495+0000 7f694b000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69440b50b0 con 0x7f694406d5b0 2024-01-15T18:16:47.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.495+0000 7f694b000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69440b4be0 con 0x7f6944071360 2024-01-15T18:16:47.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.495+0000 7f6943e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6944143a70 0x7f69440b2f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45778/0 (socket says 172.21.15.27:45778) 2024-01-15T18:16:47.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.495+0000 7f6943e00640 1 -- 172.21.15.27:0/3342727161 learned_addr learned my addr 172.21.15.27:0/3342727161 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:16:47.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.495+0000 7f6943400640 1 --2- 172.21.15.27:0/3342727161 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6944071360 0x7f69440717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:47.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.495+0000 7f6942a00640 1 -- 172.21.15.27:0/3342727161 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3224256287 0 0) 0x7f69440b4dd0 con 0x7f6944143a70 2024-01-15T18:16:47.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.495+0000 7f6942a00640 1 -- 172.21.15.27:0/3342727161 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6924003530 con 0x7f6944143a70 2024-01-15T18:16:47.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.495+0000 7f6942a00640 1 -- 172.21.15.27:0/3342727161 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438716990 0 0) 0x7f69440b50b0 con 0x7f694406d5b0 2024-01-15T18:16:47.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.495+0000 7f6942a00640 1 -- 172.21.15.27:0/3342727161 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69440b4dd0 con 0x7f694406d5b0 2024-01-15T18:16:47.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.495+0000 7f6943400640 1 -- 172.21.15.27:0/3342727161 >> v1:172.21.15.93:6789/0 conn(0x7f694406d5b0 legacy=0x7f694406da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:47.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.495+0000 7f6943400640 1 -- 172.21.15.27:0/3342727161 >> v1:172.21.15.27:6789/0 conn(0x7f6944143a70 legacy=0x7f69440b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:47.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.495+0000 7f6943400640 1 -- 172.21.15.27:0/3342727161 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69440b5390 con 0x7f6944071360 2024-01-15T18:16:47.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.499+0000 7f6943400640 1 --2- 172.21.15.27:0/3342727161 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6944071360 0x7f69440717e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f692c0097f0 tx=0x7f692c01c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe0f9cc4edbff6af server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:47.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.503+0000 7f6942a00640 1 -- 172.21.15.27:0/3342727161 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692c0ab600 con 0x7f6944071360 2024-01-15T18:16:47.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.503+0000 7f6942a00640 1 -- 172.21.15.27:0/3342727161 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f692c0a7050 con 0x7f6944071360 2024-01-15T18:16:47.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.503+0000 7f6942a00640 1 -- 172.21.15.27:0/3342727161 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692c0ab980 con 0x7f6944071360 2024-01-15T18:16:47.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.503+0000 7f694b000640 1 -- 172.21.15.27:0/3342727161 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6944071360 msgr2=0x7f69440717e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:47.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.503+0000 7f694b000640 1 --2- 172.21.15.27:0/3342727161 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6944071360 0x7f69440717e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f692c0097f0 tx=0x7f692c01c4c0 comp rx=0 tx=0).stop 2024-01-15T18:16:47.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.503+0000 7f694b000640 1 -- 172.21.15.27:0/3342727161 shutdown_connections 2024-01-15T18:16:47.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.503+0000 7f694b000640 1 --2- 172.21.15.27:0/3342727161 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6944071360 0x7f69440717e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:47.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.503+0000 7f694b000640 1 -- 172.21.15.27:0/3342727161 >> 172.21.15.27:0/3342727161 conn(0x7f69440b1c80 msgr2=0x7f69440b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:47.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.503+0000 7f694b000640 1 -- 172.21.15.27:0/3342727161 shutdown_connections 2024-01-15T18:16:47.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.503+0000 7f694b000640 1 -- 172.21.15.27:0/3342727161 wait complete. 2024-01-15T18:16:47.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.507+0000 7f694b000640 1 Processor -- start 2024-01-15T18:16:47.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.507+0000 7f694b000640 1 -- start start 2024-01-15T18:16:47.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.507+0000 7f694b000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f694406d5b0 0x7f694415ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:47.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.507+0000 7f694b000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6944071360 0x7f694415ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:47.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.507+0000 7f694b000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6944143a70 0x7f694415f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:47.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.507+0000 7f694b000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6944140a80 con 0x7f694406d5b0 2024-01-15T18:16:47.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.507+0000 7f694b000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6944159db0 con 0x7f6944143a70 2024-01-15T18:16:47.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.507+0000 7f694b000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f694415a090 con 0x7f6944071360 2024-01-15T18:16:47.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.507+0000 7f6943e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f694406d5b0 0x7f694415ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:47.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.507+0000 7f6943e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f694406d5b0 0x7f694415ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55278/0 (socket says 172.21.15.27:55278) 2024-01-15T18:16:47.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.507+0000 7f6943e00640 1 -- 172.21.15.27:0/4265510412 learned_addr learned my addr 172.21.15.27:0/4265510412 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:47.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.507+0000 7f6943400640 1 --2- 172.21.15.27:0/4265510412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6944071360 0x7f694415ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:47.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.507+0000 7f6943e00640 1 -- 172.21.15.27:0/4265510412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6944071360 msgr2=0x7f694415ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:47.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.507+0000 7f6949000640 1 --2- 172.21.15.27:0/4265510412 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6944143a70 0x7f694415f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:47.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.507+0000 7f6943e00640 1 --2- 172.21.15.27:0/4265510412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6944071360 0x7f694415ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:47.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.507+0000 7f6943e00640 1 -- 172.21.15.27:0/4265510412 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6944143a70 msgr2=0x7f694415f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:47.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.507+0000 7f6943e00640 1 --2- 172.21.15.27:0/4265510412 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6944143a70 0x7f694415f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:47.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.507+0000 7f6943e00640 1 -- 172.21.15.27:0/4265510412 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692c009420 con 0x7f694406d5b0 2024-01-15T18:16:47.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.507+0000 7f6949000640 1 --2- 172.21.15.27:0/4265510412 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6944143a70 0x7f694415f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:16:47.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.507+0000 7f6943e00640 1 --2- 172.21.15.27:0/4265510412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f694406d5b0 0x7f694415ae50 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f693400cbe0 tx=0x7f693400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:47.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.507+0000 7f693be00640 1 -- 172.21.15.27:0/4265510412 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693400ad40 con 0x7f694406d5b0 2024-01-15T18:16:47.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.507+0000 7f693be00640 1 -- 172.21.15.27:0/4265510412 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6934012400 con 0x7f694406d5b0 2024-01-15T18:16:47.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.507+0000 7f694b000640 1 -- 172.21.15.27:0/4265510412 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f694415a370 con 0x7f694406d5b0 2024-01-15T18:16:47.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.507+0000 7f693be00640 1 -- 172.21.15.27:0/4265510412 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693400aea0 con 0x7f694406d5b0 2024-01-15T18:16:47.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.507+0000 7f694b000640 1 -- 172.21.15.27:0/4265510412 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f694415a930 con 0x7f694406d5b0 2024-01-15T18:16:47.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.511+0000 7f694b000640 1 -- 172.21.15.27:0/4265510412 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f694415ab50 con 0x7f694406d5b0 2024-01-15T18:16:47.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.511+0000 7f693be00640 1 -- 172.21.15.27:0/4265510412 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 32) v1 ==== 87384+0+0 (secure 0 0 0) 0x7f6934007070 con 0x7f694406d5b0 2024-01-15T18:16:47.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.515+0000 7f693be00640 1 --2- 172.21.15.27:0/4265510412 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f691c0676a0 0x7f691c069b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:47.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.515+0000 7f693be00640 1 -- 172.21.15.27:0/4265510412 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5822+0+0 (secure 0 0 0) 0x7f6934082fb0 con 0x7f694406d5b0 2024-01-15T18:16:47.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.519+0000 7f6943400640 1 --2- 172.21.15.27:0/4265510412 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f691c0676a0 0x7f691c069b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:47.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.519+0000 7f693be00640 1 -- 172.21.15.27:0/4265510412 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6934051aa0 con 0x7f694406d5b0 2024-01-15T18:16:47.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.523+0000 7f6943400640 1 --2- 172.21.15.27:0/4265510412 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f691c0676a0 0x7f691c069b60 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f692c0ad000 tx=0x7f692c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:47.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.819+0000 7f694b000640 1 -- 172.21.15.27:0/4265510412 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f69440b5e70 con 0x7f691c0676a0 2024-01-15T18:16:47.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.823+0000 7f693be00640 1 -- 172.21.15.27:0/4265510412 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f692c0ab660 con 0x7f691c0676a0 2024-01-15T18:16:47.823 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:16:47.823 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:16:47.824 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:16:47.824 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:16:47.824 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:16:47.824 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:16:47.824 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:16:47.824 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:16:47.824 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:16:47.824 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 66.66666666666666, 2024-01-15T18:16:47.824 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T18:16:47.824 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "2/3", 2024-01-15T18:16:47.824 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:16:47.824 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:16:47.824 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:47.824 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:47.825 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:16:47.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.827+0000 7f694b000640 1 -- 172.21.15.27:0/4265510412 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f691c0676a0 msgr2=0x7f691c069b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:47.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.827+0000 7f694b000640 1 --2- 172.21.15.27:0/4265510412 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f691c0676a0 0x7f691c069b60 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f692c0ad000 tx=0x7f692c008040 comp rx=0 tx=0).stop 2024-01-15T18:16:47.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.827+0000 7f694b000640 1 -- 172.21.15.27:0/4265510412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f694406d5b0 msgr2=0x7f694415ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:47.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.827+0000 7f694b000640 1 --2- 172.21.15.27:0/4265510412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f694406d5b0 0x7f694415ae50 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f693400cbe0 tx=0x7f693400a430 comp rx=0 tx=0).stop 2024-01-15T18:16:47.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.827+0000 7f694b000640 1 -- 172.21.15.27:0/4265510412 shutdown_connections 2024-01-15T18:16:47.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.827+0000 7f694b000640 1 --2- 172.21.15.27:0/4265510412 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f691c0676a0 0x7f691c069b60 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:47.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.827+0000 7f694b000640 1 --2- 172.21.15.27:0/4265510412 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6944143a70 0x7f694415f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:47.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.827+0000 7f694b000640 1 --2- 172.21.15.27:0/4265510412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6944071360 0x7f694415ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:47.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.827+0000 7f694b000640 1 --2- 172.21.15.27:0/4265510412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f694406d5b0 0x7f694415ae50 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:47.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.827+0000 7f694b000640 1 -- 172.21.15.27:0/4265510412 >> 172.21.15.27:0/4265510412 conn(0x7f69440b1c80 msgr2=0x7f6944071810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:47.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.827+0000 7f694b000640 1 -- 172.21.15.27:0/4265510412 shutdown_connections 2024-01-15T18:16:47.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:47.827+0000 7f694b000640 1 -- 172.21.15.27:0/4265510412 wait complete. 2024-01-15T18:16:47.845 INFO:tasks.cephfs.test_volumes:percetange completed = 66.66666666666666 2024-01-15T18:16:49.846 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:16:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feefcc00640 1 Processor -- start 2024-01-15T18:16:50.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feefcc00640 1 -- start start 2024-01-15T18:16:50.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feefcc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feef8072360 0x7feef80b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:50.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feefcc00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef80b5980 con 0x7feef80cf570 2024-01-15T18:16:50.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feefcc00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef80ce7b0 con 0x7feef815d5c0 2024-01-15T18:16:50.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feefcc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef80b5760 con 0x7feef8072360 2024-01-15T18:16:50.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feef7200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7feef80cf570 0x7feef80b5050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:45780/0 (socket says 172.21.15.27:45780) 2024-01-15T18:16:50.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feef7200640 1 -- 172.21.15.27:0/327564244 learned_addr learned my addr 172.21.15.27:0/327564244 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:16:50.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feef4a00640 1 -- 172.21.15.27:0/327564244 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415378573 0 0) 0x7feef80b5980 con 0x7feef80cf570 2024-01-15T18:16:50.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feef4a00640 1 -- 172.21.15.27:0/327564244 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feed8003530 con 0x7feef80cf570 2024-01-15T18:16:50.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feef4a00640 1 -- 172.21.15.27:0/327564244 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3085631627 0 0) 0x7feed8003530 con 0x7feef80cf570 2024-01-15T18:16:50.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feef4a00640 1 -- 172.21.15.27:0/327564244 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feef8072360 msgr2=0x7feef80b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:50.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feef5400640 1 --2- 172.21.15.27:0/327564244 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feef8072360 0x7feef80b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:50.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feef4a00640 1 --2- 172.21.15.27:0/327564244 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feef8072360 0x7feef80b4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:50.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feef4a00640 1 -- 172.21.15.27:0/327564244 >> v1:172.21.15.93:6789/0 conn(0x7feef815d5c0 legacy=0x7feef8071c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:50.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feef4a00640 1 -- 172.21.15.27:0/327564244 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef80cea90 con 0x7feef80cf570 2024-01-15T18:16:50.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feef4a00640 1 -- 172.21.15.27:0/327564244 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7feef0001a00 con 0x7feef80cf570 2024-01-15T18:16:50.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feef5400640 1 --2- 172.21.15.27:0/327564244 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feef8072360 0x7feef80b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:16:50.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feef4a00640 1 -- 172.21.15.27:0/327564244 >> v1:172.21.15.27:6789/0 conn(0x7feef80cf570 legacy=0x7feef80b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:50.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feef4a00640 1 --2- 172.21.15.27:0/327564244 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feef8072360 0x7feed80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:50.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feef4a00640 1 --2- 172.21.15.27:0/327564244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feed8004080 0x7feed80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:50.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feef4a00640 1 --2- 172.21.15.27:0/327564244 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feed8004d60 0x7feed80051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:50.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feef4a00640 1 -- 172.21.15.27:0/327564244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef80b5760 con 0x7feed8004080 2024-01-15T18:16:50.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feef4a00640 1 -- 172.21.15.27:0/327564244 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed800bc70 con 0x7feed8004d60 2024-01-15T18:16:50.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feef4a00640 1 -- 172.21.15.27:0/327564244 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feed800be30 con 0x7feef8072360 2024-01-15T18:16:50.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.079+0000 7feef5400640 1 --2- 172.21.15.27:0/327564244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feed8004080 0x7feed80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:50.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.083+0000 7feef5e00640 1 --2- 172.21.15.27:0/327564244 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feef8072360 0x7feed80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:50.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.083+0000 7feef5400640 1 -- 172.21.15.27:0/327564244 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feef8072360 msgr2=0x7feed80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:50.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.083+0000 7feef5400640 1 --2- 172.21.15.27:0/327564244 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feef8072360 0x7feed80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:50.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.083+0000 7feef5400640 1 -- 172.21.15.27:0/327564244 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feed8004d60 msgr2=0x7feed80051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:50.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.083+0000 7feef5400640 1 --2- 172.21.15.27:0/327564244 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feed8004d60 0x7feed80051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:50.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.083+0000 7feef5400640 1 -- 172.21.15.27:0/327564244 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed8003530 con 0x7feed8004080 2024-01-15T18:16:50.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.083+0000 7feef5e00640 1 --2- 172.21.15.27:0/327564244 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feef8072360 0x7feed80039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:16:50.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.083+0000 7feef5400640 1 --2- 172.21.15.27:0/327564244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feed8004080 0x7feed80047d0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7feee000aa30 tx=0x7feee001c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ac0278b0fc9f975 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:50.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.083+0000 7feef4a00640 1 -- 172.21.15.27:0/327564244 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee00ab600 con 0x7feed8004080 2024-01-15T18:16:50.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.083+0000 7feef4a00640 1 -- 172.21.15.27:0/327564244 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee00a7050 con 0x7feed8004080 2024-01-15T18:16:50.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.083+0000 7feef4a00640 1 -- 172.21.15.27:0/327564244 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee00ab9c0 con 0x7feed8004080 2024-01-15T18:16:50.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.083+0000 7feefcc00640 1 -- 172.21.15.27:0/327564244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feed8004080 msgr2=0x7feed80047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:50.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.083+0000 7feefcc00640 1 --2- 172.21.15.27:0/327564244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feed8004080 0x7feed80047d0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7feee000aa30 tx=0x7feee001c5d0 comp rx=0 tx=0).stop 2024-01-15T18:16:50.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.083+0000 7feef7200640 1 -- 172.21.15.27:0/327564244 reap_dead start 2024-01-15T18:16:50.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.087+0000 7feefcc00640 1 -- 172.21.15.27:0/327564244 shutdown_connections 2024-01-15T18:16:50.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.087+0000 7feefcc00640 1 -- 172.21.15.27:0/327564244 >> 172.21.15.27:0/327564244 conn(0x7feef8099e10 msgr2=0x7feef80b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:50.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.087+0000 7feefcc00640 1 -- 172.21.15.27:0/327564244 shutdown_connections 2024-01-15T18:16:50.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.087+0000 7feefcc00640 1 -- 172.21.15.27:0/327564244 wait complete. 2024-01-15T18:16:50.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.087+0000 7feefcc00640 1 Processor -- start 2024-01-15T18:16:50.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.087+0000 7feefcc00640 1 -- start start 2024-01-15T18:16:50.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.091+0000 7feefcc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feed8004080 0x7feef806af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:50.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.091+0000 7feefcc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feef80b0050 0x7feef814dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:50.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.091+0000 7feefcc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feef814ff10 0x7feef81503c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:50.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.091+0000 7feefcc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef8150900 con 0x7feed8004080 2024-01-15T18:16:50.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.091+0000 7feefcc00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef80b1a20 con 0x7feef80b0050 2024-01-15T18:16:50.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.091+0000 7feefcc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef80b1d00 con 0x7feef814ff10 2024-01-15T18:16:50.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.091+0000 7feef5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feed8004080 0x7feef806af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:50.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.091+0000 7feef5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feed8004080 0x7feef806af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55304/0 (socket says 172.21.15.27:55304) 2024-01-15T18:16:50.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.091+0000 7feef5e00640 1 -- 172.21.15.27:0/2897311651 learned_addr learned my addr 172.21.15.27:0/2897311651 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:50.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.091+0000 7feef5400640 1 --2- 172.21.15.27:0/2897311651 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feef80b0050 0x7feef814dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:50.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.091+0000 7feef5e00640 1 -- 172.21.15.27:0/2897311651 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feef814ff10 msgr2=0x7feef81503c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:50.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.091+0000 7feef5e00640 1 --2- 172.21.15.27:0/2897311651 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feef814ff10 0x7feef81503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:50.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.091+0000 7feef5e00640 1 -- 172.21.15.27:0/2897311651 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feef80b0050 msgr2=0x7feef814dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:50.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.091+0000 7feef5e00640 1 --2- 172.21.15.27:0/2897311651 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feef80b0050 0x7feef814dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:50.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.091+0000 7feef5e00640 1 -- 172.21.15.27:0/2897311651 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef81576e0 con 0x7feed8004080 2024-01-15T18:16:50.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.091+0000 7feef5400640 1 --2- 172.21.15.27:0/2897311651 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feef80b0050 0x7feef814dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:16:50.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.091+0000 7feef5e00640 1 --2- 172.21.15.27:0/2897311651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feed8004080 0x7feef806af00 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7feee800d590 tx=0x7feee800da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:50.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.091+0000 7feeee000640 1 -- 172.21.15.27:0/2897311651 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee8005480 con 0x7feed8004080 2024-01-15T18:16:50.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.091+0000 7feeee000640 1 -- 172.21.15.27:0/2897311651 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee8003600 con 0x7feed8004080 2024-01-15T18:16:50.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.091+0000 7feeee000640 1 -- 172.21.15.27:0/2897311651 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee80039b0 con 0x7feed8004080 2024-01-15T18:16:50.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.091+0000 7feefcc00640 1 -- 172.21.15.27:0/2897311651 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef81579c0 con 0x7feed8004080 2024-01-15T18:16:50.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.091+0000 7feefcc00640 1 -- 172.21.15.27:0/2897311651 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef806b440 con 0x7feed8004080 2024-01-15T18:16:50.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.095+0000 7feefcc00640 1 -- 172.21.15.27:0/2897311651 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef8157bb0 con 0x7feed8004080 2024-01-15T18:16:50.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.095+0000 7feeee000640 1 -- 172.21.15.27:0/2897311651 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 32) v1 ==== 87384+0+0 (secure 0 0 0) 0x7feee8003b10 con 0x7feed8004080 2024-01-15T18:16:50.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.095+0000 7feeee000640 1 --2- 172.21.15.27:0/2897311651 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7feed00676a0 0x7feed0069b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:50.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.099+0000 7feeee000640 1 -- 172.21.15.27:0/2897311651 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5822+0+0 (secure 0 0 0) 0x7feee8083070 con 0x7feed8004080 2024-01-15T18:16:50.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.099+0000 7feef5400640 1 --2- 172.21.15.27:0/2897311651 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7feed00676a0 0x7feed0069b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:50.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.099+0000 7feef5400640 1 --2- 172.21.15.27:0/2897311651 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7feed00676a0 0x7feed0069b60 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7feee00ad000 tx=0x7feee00046d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:50.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.103+0000 7feeee000640 1 -- 172.21.15.27:0/2897311651 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7feee8051ab0 con 0x7feed8004080 2024-01-15T18:16:50.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.411+0000 7feefcc00640 1 -- 172.21.15.27:0/2897311651 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7feef809e950 con 0x7feed00676a0 2024-01-15T18:16:50.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.439+0000 7feeee000640 1 -- 172.21.15.27:0/2897311651 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7feef809e950 con 0x7feed00676a0 2024-01-15T18:16:50.437 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:16:50.437 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:16:50.437 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "in-progress", 2024-01-15T18:16:50.437 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:16:50.437 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:16:50.438 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:16:50.438 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:16:50.438 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:16:50.438 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_report": { 2024-01-15T18:16:50.438 INFO:teuthology.orchestra.run.smithi027.stdout: "percentage completed": 66.66666666666666, 2024-01-15T18:16:50.438 INFO:teuthology.orchestra.run.smithi027.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T18:16:50.438 INFO:teuthology.orchestra.run.smithi027.stdout: "regfiles copied": "2/3", 2024-01-15T18:16:50.438 INFO:teuthology.orchestra.run.smithi027.stdout: "subdirs copied": "0/1", 2024-01-15T18:16:50.438 INFO:teuthology.orchestra.run.smithi027.stdout: "symlinks copied": "0/0" 2024-01-15T18:16:50.438 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:50.438 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:50.438 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:16:50.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.439+0000 7feefcc00640 1 -- 172.21.15.27:0/2897311651 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7feed00676a0 msgr2=0x7feed0069b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:50.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.439+0000 7feefcc00640 1 --2- 172.21.15.27:0/2897311651 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7feed00676a0 0x7feed0069b60 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7feee00ad000 tx=0x7feee00046d0 comp rx=0 tx=0).stop 2024-01-15T18:16:50.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.439+0000 7feefcc00640 1 -- 172.21.15.27:0/2897311651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feed8004080 msgr2=0x7feef806af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:50.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.439+0000 7feefcc00640 1 --2- 172.21.15.27:0/2897311651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feed8004080 0x7feef806af00 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7feee800d590 tx=0x7feee800da50 comp rx=0 tx=0).stop 2024-01-15T18:16:50.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.439+0000 7feefcc00640 1 -- 172.21.15.27:0/2897311651 shutdown_connections 2024-01-15T18:16:50.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.439+0000 7feefcc00640 1 --2- 172.21.15.27:0/2897311651 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7feed00676a0 0x7feed0069b60 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:50.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.439+0000 7feefcc00640 1 --2- 172.21.15.27:0/2897311651 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feef814ff10 0x7feef81503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:50.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.439+0000 7feefcc00640 1 --2- 172.21.15.27:0/2897311651 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feef80b0050 0x7feef814dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:50.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.439+0000 7feefcc00640 1 --2- 172.21.15.27:0/2897311651 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feed8004080 0x7feef806af00 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:50.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.439+0000 7feefcc00640 1 -- 172.21.15.27:0/2897311651 >> 172.21.15.27:0/2897311651 conn(0x7feef8099e10 msgr2=0x7feef80d05b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:50.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.439+0000 7feefcc00640 1 -- 172.21.15.27:0/2897311651 shutdown_connections 2024-01-15T18:16:50.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:50.439+0000 7feefcc00640 1 -- 172.21.15.27:0/2897311651 wait complete. 2024-01-15T18:16:50.477 INFO:tasks.cephfs.test_volumes:percetange completed = 66.66666666666666 2024-01-15T18:16:52.479 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:16:52.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.635+0000 7f2684000640 1 Processor -- start 2024-01-15T18:16:52.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.635+0000 7f2684000640 1 -- start start 2024-01-15T18:16:52.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.635+0000 7f2684000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f267c0b1db0 0x7f267c0b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:52.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.635+0000 7f2684000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f267c0ce050 0x7f267c0ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:52.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.635+0000 7f2684000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f267c0ce9e0 con 0x7f267c06c430 2024-01-15T18:16:52.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.635+0000 7f2684000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f267c0712b0 con 0x7f267c0b1db0 2024-01-15T18:16:52.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.635+0000 7f2684000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f267c0ccfb0 con 0x7f267c0ce050 2024-01-15T18:16:52.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.639+0000 7f2680c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f267c0b1db0 0x7f267c0b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:52.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.639+0000 7f2680c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f267c0b1db0 0x7f267c0b2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55318/0 (socket says 172.21.15.27:55318) 2024-01-15T18:16:52.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.639+0000 7f2680c00640 1 -- 172.21.15.27:0/2349185470 learned_addr learned my addr 172.21.15.27:0/2349185470 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:52.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.639+0000 7f2682000640 1 --2- 172.21.15.27:0/2349185470 >> v2:172.21.15.93:3300/0 conn(0x7f267c0ce050 0x7f267c0ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:52.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.639+0000 7f2680c00640 1 -- 172.21.15.27:0/2349185470 >> v2:172.21.15.93:3300/0 conn(0x7f267c0ce050 msgr2=0x7f267c0ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:52.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.639+0000 7f2680c00640 1 --2- 172.21.15.27:0/2349185470 >> v2:172.21.15.93:3300/0 conn(0x7f267c0ce050 0x7f267c0ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:52.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.639+0000 7f2680c00640 1 -- 172.21.15.27:0/2349185470 >> v1:172.21.15.93:6789/0 conn(0x7f267c06c430 legacy=0x7f267c070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:52.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.639+0000 7f2680c00640 1 -- 172.21.15.27:0/2349185470 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f267c0cd290 con 0x7f267c0b1db0 2024-01-15T18:16:52.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.639+0000 7f2680c00640 1 --2- 172.21.15.27:0/2349185470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f267c0b1db0 0x7f267c0b2240 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f2664009870 tx=0x7f266401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c113646cbc21f18 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:52.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.639+0000 7f267be00640 1 -- 172.21.15.27:0/2349185470 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26640ac620 con 0x7f267c0b1db0 2024-01-15T18:16:52.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.639+0000 7f267be00640 1 -- 172.21.15.27:0/2349185470 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26640ac780 con 0x7f267c0b1db0 2024-01-15T18:16:52.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.639+0000 7f267be00640 1 -- 172.21.15.27:0/2349185470 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26640acb10 con 0x7f267c0b1db0 2024-01-15T18:16:52.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.643+0000 7f2684000640 1 -- 172.21.15.27:0/2349185470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f267c0b1db0 msgr2=0x7f267c0b2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:52.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.643+0000 7f2684000640 1 --2- 172.21.15.27:0/2349185470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f267c0b1db0 0x7f267c0b2240 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f2664009870 tx=0x7f266401c440 comp rx=0 tx=0).stop 2024-01-15T18:16:52.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.643+0000 7f2684000640 1 -- 172.21.15.27:0/2349185470 shutdown_connections 2024-01-15T18:16:52.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.643+0000 7f2684000640 1 --2- 172.21.15.27:0/2349185470 >> v2:172.21.15.93:3300/0 conn(0x7f267c0ce050 0x7f267c0ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:52.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.643+0000 7f2684000640 1 --2- 172.21.15.27:0/2349185470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f267c0b1db0 0x7f267c0b2240 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:52.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.643+0000 7f2684000640 1 -- 172.21.15.27:0/2349185470 >> 172.21.15.27:0/2349185470 conn(0x7f267c0cb700 msgr2=0x7f267c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:52.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.643+0000 7f2684000640 1 -- 172.21.15.27:0/2349185470 shutdown_connections 2024-01-15T18:16:52.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.643+0000 7f2684000640 1 -- 172.21.15.27:0/2349185470 wait complete. 2024-01-15T18:16:52.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.643+0000 7f2684000640 1 Processor -- start 2024-01-15T18:16:52.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.643+0000 7f2684000640 1 -- start start 2024-01-15T18:16:52.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.643+0000 7f2684000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f267c06c430 0x7f267c14b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:52.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.643+0000 7f2684000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f267c0b1db0 0x7f267c156190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:52.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.643+0000 7f2684000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f267c0ce050 0x7f267c1566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.643+0000 7f2684000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267c1f9c50 con 0x7f267c0ce050 2024-01-15T18:16:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.643+0000 7f2684000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267c14baa0 con 0x7f267c06c430 2024-01-15T18:16:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.643+0000 7f2684000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f267c156d50 con 0x7f267c0b1db0 2024-01-15T18:16:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.643+0000 7f2681600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f267c06c430 0x7f267c14b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.643+0000 7f2682000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f267c0ce050 0x7f267c1566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.647+0000 7f2681600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f267c06c430 0x7f267c14b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:47756/0 (socket says 172.21.15.27:47756) 2024-01-15T18:16:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.647+0000 7f2681600640 1 -- 172.21.15.27:0/953546992 learned_addr learned my addr 172.21.15.27:0/953546992 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.647+0000 7f2680c00640 1 --2- 172.21.15.27:0/953546992 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f267c0b1db0 0x7f267c156190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.647+0000 7f2682000640 1 -- 172.21.15.27:0/953546992 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f267c0b1db0 msgr2=0x7f267c156190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.647+0000 7f2682000640 1 --2- 172.21.15.27:0/953546992 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f267c0b1db0 0x7f267c156190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.647+0000 7f2682000640 1 -- 172.21.15.27:0/953546992 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f267c06c430 msgr2=0x7f267c14b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:52.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.647+0000 7f2682000640 1 --2- 172.21.15.27:0/953546992 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f267c06c430 0x7f267c14b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:52.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.647+0000 7f2682000640 1 -- 172.21.15.27:0/953546992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26640094d0 con 0x7f267c0ce050 2024-01-15T18:16:52.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.647+0000 7f2681600640 1 --2- 172.21.15.27:0/953546992 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f267c06c430 0x7f267c14b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:16:52.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.647+0000 7f2682000640 1 --2- 172.21.15.27:0/953546992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f267c0ce050 0x7f267c1566d0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f2674007cf0 tx=0x7f267400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:52.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.647+0000 7f2679600640 1 -- 172.21.15.27:0/953546992 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2674018070 con 0x7f267c0ce050 2024-01-15T18:16:52.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.647+0000 7f2679600640 1 -- 172.21.15.27:0/953546992 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f267400edd0 con 0x7f267c0ce050 2024-01-15T18:16:52.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.647+0000 7f2679600640 1 -- 172.21.15.27:0/953546992 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2674003bc0 con 0x7f267c0ce050 2024-01-15T18:16:52.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.647+0000 7f2684000640 1 -- 172.21.15.27:0/953546992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f267c1f9320 con 0x7f267c0ce050 2024-01-15T18:16:52.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.647+0000 7f2684000640 1 -- 172.21.15.27:0/953546992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f267c1f98e0 con 0x7f267c0ce050 2024-01-15T18:16:52.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.647+0000 7f2684000640 1 -- 172.21.15.27:0/953546992 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f267c1f9dc0 con 0x7f267c0ce050 2024-01-15T18:16:52.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.647+0000 7f2679600640 1 -- 172.21.15.27:0/953546992 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 32) v1 ==== 87384+0+0 (secure 0 0 0) 0x7f2674014420 con 0x7f267c0ce050 2024-01-15T18:16:52.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.651+0000 7f2679600640 1 --2- 172.21.15.27:0/953546992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f26540676a0 0x7f2654069b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:52.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.651+0000 7f2679600640 1 -- 172.21.15.27:0/953546992 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5822+0+0 (secure 0 0 0) 0x7f26740831c0 con 0x7f267c0ce050 2024-01-15T18:16:52.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.651+0000 7f2681600640 1 --2- 172.21.15.27:0/953546992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f26540676a0 0x7f2654069b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:52.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.651+0000 7f2681600640 1 --2- 172.21.15.27:0/953546992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f26540676a0 0x7f2654069b60 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f266c001ca0 tx=0x7f266c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:52.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.655+0000 7f2679600640 1 -- 172.21.15.27:0/953546992 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2674051c50 con 0x7f267c0ce050 2024-01-15T18:16:52.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:52.843+0000 7f2684000640 1 -- 172.21.15.27:0/953546992 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f267c04e850 con 0x7f26540676a0 2024-01-15T18:16:55.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.287+0000 7f2679600640 1 -- 172.21.15.27:0/953546992 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f267c04e850 con 0x7f26540676a0 2024-01-15T18:16:55.286 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:16:55.286 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:16:55.287 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "complete" 2024-01-15T18:16:55.287 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:55.287 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:16:55.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.291+0000 7f2684000640 1 -- 172.21.15.27:0/953546992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f26540676a0 msgr2=0x7f2654069b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:55.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.291+0000 7f2684000640 1 --2- 172.21.15.27:0/953546992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f26540676a0 0x7f2654069b60 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f266c001ca0 tx=0x7f266c009040 comp rx=0 tx=0).stop 2024-01-15T18:16:55.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.291+0000 7f2684000640 1 -- 172.21.15.27:0/953546992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f267c0ce050 msgr2=0x7f267c1566d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:55.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.291+0000 7f2684000640 1 --2- 172.21.15.27:0/953546992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f267c0ce050 0x7f267c1566d0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f2674007cf0 tx=0x7f267400e4b0 comp rx=0 tx=0).stop 2024-01-15T18:16:55.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.291+0000 7f2684000640 1 -- 172.21.15.27:0/953546992 shutdown_connections 2024-01-15T18:16:55.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.291+0000 7f2684000640 1 --2- 172.21.15.27:0/953546992 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f26540676a0 0x7f2654069b60 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:55.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.291+0000 7f2684000640 1 --2- 172.21.15.27:0/953546992 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f267c0ce050 0x7f267c1566d0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:55.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.291+0000 7f2684000640 1 --2- 172.21.15.27:0/953546992 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f267c0b1db0 0x7f267c156190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:55.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.291+0000 7f2684000640 1 --2- 172.21.15.27:0/953546992 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f267c06c430 0x7f267c14b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:55.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.291+0000 7f2684000640 1 -- 172.21.15.27:0/953546992 >> 172.21.15.27:0/953546992 conn(0x7f267c0cb700 msgr2=0x7f267c14e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:55.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.291+0000 7f2684000640 1 -- 172.21.15.27:0/953546992 shutdown_connections 2024-01-15T18:16:55.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.291+0000 7f2684000640 1 -- 172.21.15.27:0/953546992 wait complete. 2024-01-15T18:16:55.313 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:16:55.313 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:16:55.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb070600640 1 Processor -- start 2024-01-15T18:16:55.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb070600640 1 -- start start 2024-01-15T18:16:55.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb070600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb06815ad20 0x7fb06815b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:55.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb070600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fb0680715f0 0x7fb06815dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:55.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb070600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb06815a610 con 0x7fb06815d900 2024-01-15T18:16:55.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb070600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb06815e3c0 con 0x7fb0680715f0 2024-01-15T18:16:55.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb070600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb06815e6a0 con 0x7fb06815ad20 2024-01-15T18:16:55.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb06d200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb06815d900 0x7fb068070ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40986/0 (socket says 172.21.15.27:40986) 2024-01-15T18:16:55.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb06d200640 1 -- 172.21.15.27:0/2425746437 learned_addr learned my addr 172.21.15.27:0/2425746437 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:16:55.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb06dc00640 1 --2- 172.21.15.27:0/2425746437 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb06815ad20 0x7fb06815b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:55.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb06e600640 1 --2- 172.21.15.27:0/2425746437 >> v2:172.21.15.93:3300/0 conn(0x7fb0680715f0 0x7fb06815dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:55.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb067e00640 1 -- 172.21.15.27:0/2425746437 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 648972357 0 0) 0x7fb06815a610 con 0x7fb06815d900 2024-01-15T18:16:55.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb067e00640 1 -- 172.21.15.27:0/2425746437 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb048005740 con 0x7fb06815d900 2024-01-15T18:16:55.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb067e00640 1 -- 172.21.15.27:0/2425746437 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2238767546 0 0) 0x7fb048005740 con 0x7fb06815d900 2024-01-15T18:16:55.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb067e00640 1 -- 172.21.15.27:0/2425746437 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb06815ad20 msgr2=0x7fb06815b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:55.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb067e00640 1 --2- 172.21.15.27:0/2425746437 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb06815ad20 0x7fb06815b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:55.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb067e00640 1 -- 172.21.15.27:0/2425746437 >> v2:172.21.15.93:3300/0 conn(0x7fb0680715f0 msgr2=0x7fb06815dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:55.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb067e00640 1 --2- 172.21.15.27:0/2425746437 >> v2:172.21.15.93:3300/0 conn(0x7fb0680715f0 0x7fb06815dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:55.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb067e00640 1 -- 172.21.15.27:0/2425746437 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06815b6e0 con 0x7fb06815d900 2024-01-15T18:16:55.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb067e00640 1 -- 172.21.15.27:0/2425746437 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fb050001680 con 0x7fb06815d900 2024-01-15T18:16:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb067e00640 1 -- 172.21.15.27:0/2425746437 >> v1:172.21.15.27:6789/0 conn(0x7fb06815d900 legacy=0x7fb068070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb067e00640 1 --2- 172.21.15.27:0/2425746437 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb048005c00 0x7fb048006020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb067e00640 1 --2- 172.21.15.27:0/2425746437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0480069e0 0x7fb048006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb067e00640 1 --2- 172.21.15.27:0/2425746437 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb06815ad20 0x7fb0480073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb067e00640 1 -- 172.21.15.27:0/2425746437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06815e3c0 con 0x7fb0480069e0 2024-01-15T18:16:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb067e00640 1 -- 172.21.15.27:0/2425746437 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06815e6a0 con 0x7fb048005c00 2024-01-15T18:16:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb067e00640 1 -- 172.21.15.27:0/2425746437 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb04800dba0 con 0x7fb06815ad20 2024-01-15T18:16:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.463+0000 7fb06d200640 1 --2- 172.21.15.27:0/2425746437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0480069e0 0x7fb048006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.467+0000 7fb06dc00640 1 --2- 172.21.15.27:0/2425746437 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb048005c00 0x7fb048006020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.467+0000 7fb06e600640 1 --2- 172.21.15.27:0/2425746437 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb06815ad20 0x7fb0480073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.467+0000 7fb06d200640 1 -- 172.21.15.27:0/2425746437 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb06815ad20 msgr2=0x7fb0480073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.467+0000 7fb06d200640 1 --2- 172.21.15.27:0/2425746437 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb06815ad20 0x7fb0480073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.467+0000 7fb06d200640 1 -- 172.21.15.27:0/2425746437 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb048005c00 msgr2=0x7fb048006020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:55.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.467+0000 7fb06d200640 1 --2- 172.21.15.27:0/2425746437 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb048005c00 0x7fb048006020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:55.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.467+0000 7fb06d200640 1 -- 172.21.15.27:0/2425746437 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb048005740 con 0x7fb0480069e0 2024-01-15T18:16:55.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.467+0000 7fb06d200640 1 --2- 172.21.15.27:0/2425746437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0480069e0 0x7fb048006e60 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fb05000c270 tx=0x7fb05001f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ba98aaa80f91be0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:55.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.467+0000 7fb067e00640 1 -- 172.21.15.27:0/2425746437 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0500a8ce0 con 0x7fb0480069e0 2024-01-15T18:16:55.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.467+0000 7fb067e00640 1 -- 172.21.15.27:0/2425746437 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0500a8e40 con 0x7fb0480069e0 2024-01-15T18:16:55.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.467+0000 7fb067e00640 1 -- 172.21.15.27:0/2425746437 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb050005b00 con 0x7fb0480069e0 2024-01-15T18:16:55.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.471+0000 7fb070600640 1 -- 172.21.15.27:0/2425746437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0480069e0 msgr2=0x7fb048006e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:55.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.471+0000 7fb070600640 1 --2- 172.21.15.27:0/2425746437 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0480069e0 0x7fb048006e60 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fb05000c270 tx=0x7fb05001f3b0 comp rx=0 tx=0).stop 2024-01-15T18:16:55.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.471+0000 7fb06e600640 1 -- 172.21.15.27:0/2425746437 reap_dead start 2024-01-15T18:16:55.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.471+0000 7fb070600640 1 -- 172.21.15.27:0/2425746437 shutdown_connections 2024-01-15T18:16:55.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.471+0000 7fb070600640 1 -- 172.21.15.27:0/2425746437 >> 172.21.15.27:0/2425746437 conn(0x7fb0680cb700 msgr2=0x7fb0680cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:55.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.471+0000 7fb070600640 1 -- 172.21.15.27:0/2425746437 shutdown_connections 2024-01-15T18:16:55.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.471+0000 7fb070600640 1 -- 172.21.15.27:0/2425746437 wait complete. 2024-01-15T18:16:55.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.471+0000 7fb070600640 1 Processor -- start 2024-01-15T18:16:55.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.471+0000 7fb070600640 1 -- start start 2024-01-15T18:16:55.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.471+0000 7fb070600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0480069e0 0x7fb0680cda00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:55.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.471+0000 7fb070600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb068154900 0x7fb0681f2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:55.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.471+0000 7fb070600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0681f3230 0x7fb0681491c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:55.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.471+0000 7fb070600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb068154de0 con 0x7fb068154900 2024-01-15T18:16:55.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.471+0000 7fb070600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0680c9420 con 0x7fb0480069e0 2024-01-15T18:16:55.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.471+0000 7fb070600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0680c9700 con 0x7fb0681f3230 2024-01-15T18:16:55.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.471+0000 7fb06dc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0480069e0 0x7fb0680cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:55.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.471+0000 7fb06d200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb068154900 0x7fb0681f2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:55.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.471+0000 7fb06dc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0480069e0 0x7fb0680cda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:49554/0 (socket says 172.21.15.27:49554) 2024-01-15T18:16:55.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.471+0000 7fb06dc00640 1 -- 172.21.15.27:0/1249037106 learned_addr learned my addr 172.21.15.27:0/1249037106 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:55.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.471+0000 7fb06e600640 1 --2- 172.21.15.27:0/1249037106 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0681f3230 0x7fb0681491c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:55.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.475+0000 7fb06d200640 1 -- 172.21.15.27:0/1249037106 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0681f3230 msgr2=0x7fb0681491c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:55.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.475+0000 7fb06d200640 1 --2- 172.21.15.27:0/1249037106 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0681f3230 0x7fb0681491c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:55.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.475+0000 7fb06d200640 1 -- 172.21.15.27:0/1249037106 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0480069e0 msgr2=0x7fb0680cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:55.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.475+0000 7fb06d200640 1 --2- 172.21.15.27:0/1249037106 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0480069e0 0x7fb0680cda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:55.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.475+0000 7fb06d200640 1 -- 172.21.15.27:0/1249037106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0680c99e0 con 0x7fb068154900 2024-01-15T18:16:55.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.475+0000 7fb06d200640 1 --2- 172.21.15.27:0/1249037106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb068154900 0x7fb0681f2cf0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fb050009000 tx=0x7fb048007a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:55.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.475+0000 7fb065600640 1 -- 172.21.15.27:0/1249037106 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0500049f0 con 0x7fb068154900 2024-01-15T18:16:55.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.475+0000 7fb065600640 1 -- 172.21.15.27:0/1249037106 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0500b6070 con 0x7fb068154900 2024-01-15T18:16:55.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.475+0000 7fb065600640 1 -- 172.21.15.27:0/1249037106 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb050004d50 con 0x7fb068154900 2024-01-15T18:16:55.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.475+0000 7fb070600640 1 -- 172.21.15.27:0/1249037106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0680c9cc0 con 0x7fb068154900 2024-01-15T18:16:55.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.475+0000 7fb070600640 1 -- 172.21.15.27:0/1249037106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0680ce110 con 0x7fb068154900 2024-01-15T18:16:55.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.475+0000 7fb070600640 1 -- 172.21.15.27:0/1249037106 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb06804e850 con 0x7fb068154900 2024-01-15T18:16:55.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.475+0000 7fb065600640 1 -- 172.21.15.27:0/1249037106 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 32) v1 ==== 87384+0+0 (secure 0 0 0) 0x7fb0500070d0 con 0x7fb068154900 2024-01-15T18:16:55.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.475+0000 7fb065600640 1 --2- 172.21.15.27:0/1249037106 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb040067670 0x7fb040069b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:55.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.475+0000 7fb065600640 1 -- 172.21.15.27:0/1249037106 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5822+0+0 (secure 0 0 0) 0x7fb0500b8070 con 0x7fb068154900 2024-01-15T18:16:55.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.479+0000 7fb06dc00640 1 --2- 172.21.15.27:0/1249037106 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb040067670 0x7fb040069b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:55.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.479+0000 7fb06dc00640 1 --2- 172.21.15.27:0/1249037106 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb040067670 0x7fb040069b30 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fb058009ba0 tx=0x7fb058009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:55.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.479+0000 7fb065600640 1 -- 172.21.15.27:0/1249037106 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb0500ee920 con 0x7fb068154900 2024-01-15T18:16:55.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.671+0000 7fb070600640 1 -- 172.21.15.27:0/1249037106 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0680b47c0 con 0x7fb040067670 2024-01-15T18:16:55.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.679+0000 7fb065600640 1 -- 172.21.15.27:0/1249037106 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fb0680b47c0 con 0x7fb040067670 2024-01-15T18:16:55.678 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:16:55.678 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:16:55.678 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "complete" 2024-01-15T18:16:55.678 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:55.678 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:16:55.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.679+0000 7fb070600640 1 -- 172.21.15.27:0/1249037106 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb040067670 msgr2=0x7fb040069b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:55.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.679+0000 7fb070600640 1 --2- 172.21.15.27:0/1249037106 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb040067670 0x7fb040069b30 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fb058009ba0 tx=0x7fb058009190 comp rx=0 tx=0).stop 2024-01-15T18:16:55.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.679+0000 7fb070600640 1 -- 172.21.15.27:0/1249037106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb068154900 msgr2=0x7fb0681f2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:55.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.679+0000 7fb070600640 1 --2- 172.21.15.27:0/1249037106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb068154900 0x7fb0681f2cf0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fb050009000 tx=0x7fb048007a10 comp rx=0 tx=0).stop 2024-01-15T18:16:55.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.683+0000 7fb070600640 1 -- 172.21.15.27:0/1249037106 shutdown_connections 2024-01-15T18:16:55.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.683+0000 7fb070600640 1 --2- 172.21.15.27:0/1249037106 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb040067670 0x7fb040069b30 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:55.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.683+0000 7fb070600640 1 --2- 172.21.15.27:0/1249037106 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb0681f3230 0x7fb0681491c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:55.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.683+0000 7fb070600640 1 --2- 172.21.15.27:0/1249037106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb068154900 0x7fb0681f2cf0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:55.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.683+0000 7fb070600640 1 --2- 172.21.15.27:0/1249037106 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb0480069e0 0x7fb0680cda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:55.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.683+0000 7fb070600640 1 -- 172.21.15.27:0/1249037106 >> 172.21.15.27:0/1249037106 conn(0x7fb0680cb700 msgr2=0x7fb068153b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:55.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.683+0000 7fb070600640 1 -- 172.21.15.27:0/1249037106 shutdown_connections 2024-01-15T18:16:55.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.683+0000 7fb070600640 1 -- 172.21.15.27:0/1249037106 wait complete. 2024-01-15T18:16:55.702 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:16:55.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.847+0000 7f31dc000640 1 Processor -- start 2024-01-15T18:16:55.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.847+0000 7f31dc000640 1 -- start start 2024-01-15T18:16:55.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.847+0000 7f31dc000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f31d415b180 0x7f31d40cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:55.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.847+0000 7f31dc000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f31d40cd7e0 0x7f31d40cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:55.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.847+0000 7f31dc000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f31d40b09d0 0x7f31d40b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:55.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.847+0000 7f31dc000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31d40d1e20 con 0x7f31d415b180 2024-01-15T18:16:55.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.847+0000 7f31dc000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31d40ceed0 con 0x7f31d40cd7e0 2024-01-15T18:16:55.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.847+0000 7f31dc000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d40ce180 con 0x7f31d40b09d0 2024-01-15T18:16:55.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.847+0000 7f31d9600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f31d415b180 0x7f31d40cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:55.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.847+0000 7f31d9600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f31d415b180 0x7f31d40cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56530/0 (socket says 172.21.15.27:56530) 2024-01-15T18:16:55.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.847+0000 7f31d9600640 1 -- 172.21.15.27:0/888029187 learned_addr learned my addr 172.21.15.27:0/888029187 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:55.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.847+0000 7f31d9600640 1 -- 172.21.15.27:0/888029187 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f31d40b09d0 msgr2=0x7f31d40b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:55.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.847+0000 7f31d9600640 1 --2- 172.21.15.27:0/888029187 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f31d40b09d0 0x7f31d40b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:55.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.847+0000 7f31d9600640 1 -- 172.21.15.27:0/888029187 >> v2:172.21.15.93:3300/0 conn(0x7f31d40cd7e0 msgr2=0x7f31d40cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:55.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.847+0000 7f31d8c00640 1 --2- 172.21.15.27:0/888029187 >> v2:172.21.15.93:3300/0 conn(0x7f31d40cd7e0 0x7f31d40cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:55.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.847+0000 7f31d9600640 1 --2- 172.21.15.27:0/888029187 >> v2:172.21.15.93:3300/0 conn(0x7f31d40cd7e0 0x7f31d40cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:55.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.847+0000 7f31d9600640 1 -- 172.21.15.27:0/888029187 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d40ce7d0 con 0x7f31d415b180 2024-01-15T18:16:55.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.847+0000 7f31d8c00640 1 --2- 172.21.15.27:0/888029187 >> v2:172.21.15.93:3300/0 conn(0x7f31d40cd7e0 0x7f31d40cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:16:55.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.851+0000 7f31d9600640 1 --2- 172.21.15.27:0/888029187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d415b180 0x7f31d40cfa90 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f31d4071d00 tx=0x7f31c401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9988e84570a1d6e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:55.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.851+0000 7f31d3e00640 1 -- 172.21.15.27:0/888029187 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c40aa5f0 con 0x7f31d415b180 2024-01-15T18:16:55.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.851+0000 7f31d3e00640 1 -- 172.21.15.27:0/888029187 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c40aa750 con 0x7f31d415b180 2024-01-15T18:16:55.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.851+0000 7f31d3e00640 1 -- 172.21.15.27:0/888029187 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c40aaae0 con 0x7f31d415b180 2024-01-15T18:16:55.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.851+0000 7f31dc000640 1 -- 172.21.15.27:0/888029187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d415b180 msgr2=0x7f31d40cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:55.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.851+0000 7f31dc000640 1 --2- 172.21.15.27:0/888029187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d415b180 0x7f31d40cfa90 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f31d4071d00 tx=0x7f31c401c4d0 comp rx=0 tx=0).stop 2024-01-15T18:16:55.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.851+0000 7f31dc000640 1 -- 172.21.15.27:0/888029187 shutdown_connections 2024-01-15T18:16:55.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.851+0000 7f31dc000640 1 --2- 172.21.15.27:0/888029187 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f31d40b09d0 0x7f31d40b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:55.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.851+0000 7f31dc000640 1 --2- 172.21.15.27:0/888029187 >> v2:172.21.15.93:3300/0 conn(0x7f31d40cd7e0 0x7f31d40cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:55.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.851+0000 7f31dc000640 1 --2- 172.21.15.27:0/888029187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d415b180 0x7f31d40cfa90 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:55.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.851+0000 7f31dc000640 1 -- 172.21.15.27:0/888029187 >> 172.21.15.27:0/888029187 conn(0x7f31d40d0e80 msgr2=0x7f31d40d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:55.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.851+0000 7f31dc000640 1 -- 172.21.15.27:0/888029187 shutdown_connections 2024-01-15T18:16:55.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.851+0000 7f31dc000640 1 -- 172.21.15.27:0/888029187 wait complete. 2024-01-15T18:16:55.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.855+0000 7f31dc000640 1 Processor -- start 2024-01-15T18:16:55.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.855+0000 7f31dc000640 1 -- start start 2024-01-15T18:16:55.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.855+0000 7f31dc000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f31d40b09d0 0x7f31d414db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:55.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.855+0000 7f31dc000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f31d40cd7e0 0x7f31d40c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:55.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.855+0000 7f31dc000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d415b180 0x7f31d40c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:55.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.855+0000 7f31dc000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d40ca2b0 con 0x7f31d415b180 2024-01-15T18:16:55.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.855+0000 7f31dc000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d414e0d0 con 0x7f31d40cd7e0 2024-01-15T18:16:55.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.855+0000 7f31dc000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d40c9110 con 0x7f31d40b09d0 2024-01-15T18:16:55.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.855+0000 7f31da000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d415b180 0x7f31d40c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:55.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.855+0000 7f31da000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d415b180 0x7f31d40c96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56538/0 (socket says 172.21.15.27:56538) 2024-01-15T18:16:55.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.855+0000 7f31da000640 1 -- 172.21.15.27:0/919330202 learned_addr learned my addr 172.21.15.27:0/919330202 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:55.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.855+0000 7f31d8c00640 1 --2- 172.21.15.27:0/919330202 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f31d40cd7e0 0x7f31d40c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:55.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.855+0000 7f31d9600640 1 --2- 172.21.15.27:0/919330202 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f31d40b09d0 0x7f31d414db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:55.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.855+0000 7f31da000640 1 -- 172.21.15.27:0/919330202 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f31d40b09d0 msgr2=0x7f31d414db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.855+0000 7f31da000640 1 --2- 172.21.15.27:0/919330202 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f31d40b09d0 0x7f31d414db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.855+0000 7f31da000640 1 -- 172.21.15.27:0/919330202 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f31d40cd7e0 msgr2=0x7f31d40c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.855+0000 7f31da000640 1 --2- 172.21.15.27:0/919330202 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f31d40cd7e0 0x7f31d40c8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.855+0000 7f31da000640 1 -- 172.21.15.27:0/919330202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c40094d0 con 0x7f31d415b180 2024-01-15T18:16:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.855+0000 7f31da000640 1 --2- 172.21.15.27:0/919330202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d415b180 0x7f31d40c96d0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00b970 tx=0x7f31cc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.855+0000 7f31d1600640 1 -- 172.21.15.27:0/919330202 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31cc00cb40 con 0x7f31d415b180 2024-01-15T18:16:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.855+0000 7f31d1600640 1 -- 172.21.15.27:0/919330202 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31cc00ccc0 con 0x7f31d415b180 2024-01-15T18:16:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.855+0000 7f31d1600640 1 -- 172.21.15.27:0/919330202 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31cc014680 con 0x7f31d415b180 2024-01-15T18:16:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.855+0000 7f31dc000640 1 -- 172.21.15.27:0/919330202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d40c9fd0 con 0x7f31d415b180 2024-01-15T18:16:55.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.855+0000 7f31dc000640 1 -- 172.21.15.27:0/919330202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d40c83d0 con 0x7f31d415b180 2024-01-15T18:16:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.859+0000 7f31dc000640 1 -- 172.21.15.27:0/919330202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31d40c7f50 con 0x7f31d415b180 2024-01-15T18:16:55.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.859+0000 7f31d1600640 1 -- 172.21.15.27:0/919330202 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 32) v1 ==== 87384+0+0 (secure 0 0 0) 0x7f31cc007370 con 0x7f31d415b180 2024-01-15T18:16:55.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.859+0000 7f31d1600640 1 --2- 172.21.15.27:0/919330202 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f31ac067670 0x7f31ac069b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:55.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.859+0000 7f31d1600640 1 -- 172.21.15.27:0/919330202 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5822+0+0 (secure 0 0 0) 0x7f31cc082fd0 con 0x7f31d415b180 2024-01-15T18:16:55.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.859+0000 7f31d9600640 1 --2- 172.21.15.27:0/919330202 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f31ac067670 0x7f31ac069b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:55.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.859+0000 7f31d9600640 1 --2- 172.21.15.27:0/919330202 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f31ac067670 0x7f31ac069b30 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f31c40ae000 tx=0x7f31c40b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:55.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:55.863+0000 7f31d1600640 1 -- 172.21.15.27:0/919330202 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f31d40c7f50 con 0x7f31d415b180 2024-01-15T18:16:56.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.039+0000 7f31dc000640 1 -- 172.21.15.27:0/919330202 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f31d40cb300 con 0x7f31ac067670 2024-01-15T18:16:56.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.043+0000 7f31d1600640 1 -- 172.21.15.27:0/919330202 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f31d40cb300 con 0x7f31ac067670 2024-01-15T18:16:56.040 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:16:56.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:16:56.041 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-15T18:16:56.041 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:56.041 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:16:56.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.043+0000 7f31dc000640 1 -- 172.21.15.27:0/919330202 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f31ac067670 msgr2=0x7f31ac069b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:56.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.043+0000 7f31dc000640 1 --2- 172.21.15.27:0/919330202 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f31ac067670 0x7f31ac069b30 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f31c40ae000 tx=0x7f31c40b5040 comp rx=0 tx=0).stop 2024-01-15T18:16:56.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.043+0000 7f31dc000640 1 -- 172.21.15.27:0/919330202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d415b180 msgr2=0x7f31d40c96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:56.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.043+0000 7f31dc000640 1 --2- 172.21.15.27:0/919330202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d415b180 0x7f31d40c96d0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00b970 tx=0x7f31cc00be30 comp rx=0 tx=0).stop 2024-01-15T18:16:56.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.043+0000 7f31dc000640 1 -- 172.21.15.27:0/919330202 shutdown_connections 2024-01-15T18:16:56.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.043+0000 7f31dc000640 1 --2- 172.21.15.27:0/919330202 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f31ac067670 0x7f31ac069b30 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:56.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.043+0000 7f31dc000640 1 --2- 172.21.15.27:0/919330202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31d415b180 0x7f31d40c96d0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:56.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.043+0000 7f31dc000640 1 --2- 172.21.15.27:0/919330202 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f31d40cd7e0 0x7f31d40c8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:56.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.043+0000 7f31dc000640 1 --2- 172.21.15.27:0/919330202 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f31d40b09d0 0x7f31d414db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:56.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.043+0000 7f31dc000640 1 -- 172.21.15.27:0/919330202 >> 172.21.15.27:0/919330202 conn(0x7f31d40d0e80 msgr2=0x7f31d40ae8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:56.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.043+0000 7f31dc000640 1 -- 172.21.15.27:0/919330202 shutdown_connections 2024-01-15T18:16:56.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.043+0000 7f31dc000640 1 -- 172.21.15.27:0/919330202 wait complete. 2024-01-15T18:16:56.062 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-15T18:16:56.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.211+0000 7ff34c800640 1 Processor -- start 2024-01-15T18:16:56.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.211+0000 7ff34c800640 1 -- start start 2024-01-15T18:16:56.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.211+0000 7ff34c800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff34415d0d0 0x7ff3440b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:56.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.211+0000 7ff34c800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff3440cff40 0x7ff344161410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:56.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.211+0000 7ff34c800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3440d0400 con 0x7ff3440cfb20 2024-01-15T18:16:56.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.211+0000 7ff34c800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3440721c0 con 0x7ff3440cff40 2024-01-15T18:16:56.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.211+0000 7ff34c800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3440b4f90 con 0x7ff34415d0d0 2024-01-15T18:16:56.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.211+0000 7ff349400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff3440cfb20 0x7ff3440b4880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40994/0 (socket says 172.21.15.27:40994) 2024-01-15T18:16:56.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.211+0000 7ff34a800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff3440cff40 0x7ff344161410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:56.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.211+0000 7ff349400640 1 -- 172.21.15.27:0/2283470412 learned_addr learned my addr 172.21.15.27:0/2283470412 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:16:56.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.211+0000 7ff349e00640 1 --2- 172.21.15.27:0/2283470412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff34415d0d0 0x7ff3440b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:56.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.211+0000 7ff34a800640 1 -- 172.21.15.27:0/2283470412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff34415d0d0 msgr2=0x7ff3440b2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:56.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.211+0000 7ff34a800640 1 --2- 172.21.15.27:0/2283470412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff34415d0d0 0x7ff3440b2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:56.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.211+0000 7ff34a800640 1 -- 172.21.15.27:0/2283470412 >> v1:172.21.15.27:6789/0 conn(0x7ff3440cfb20 legacy=0x7ff3440b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:56.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.211+0000 7ff348a00640 1 -- 172.21.15.27:0/2283470412 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014234509 0 0) 0x7ff3440d0400 con 0x7ff3440cfb20 2024-01-15T18:16:56.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.211+0000 7ff34a800640 1 -- 172.21.15.27:0/2283470412 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff344071940 con 0x7ff3440cff40 2024-01-15T18:16:56.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.215+0000 7ff34a800640 1 --2- 172.21.15.27:0/2283470412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3440cff40 0x7ff344161410 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7ff33c009dc0 tx=0x7ff33c01c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acf2f7c7530974c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:56.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.215+0000 7ff348a00640 1 -- 172.21.15.27:0/2283470412 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c0a5930 con 0x7ff3440cff40 2024-01-15T18:16:56.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.215+0000 7ff348a00640 1 -- 172.21.15.27:0/2283470412 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33c0a5a90 con 0x7ff3440cff40 2024-01-15T18:16:56.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.215+0000 7ff348a00640 1 -- 172.21.15.27:0/2283470412 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33c0a5e20 con 0x7ff3440cff40 2024-01-15T18:16:56.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.215+0000 7ff34c800640 1 -- 172.21.15.27:0/2283470412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3440cff40 msgr2=0x7ff344161410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:56.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.215+0000 7ff34c800640 1 --2- 172.21.15.27:0/2283470412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3440cff40 0x7ff344161410 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7ff33c009dc0 tx=0x7ff33c01c870 comp rx=0 tx=0).stop 2024-01-15T18:16:56.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.215+0000 7ff34c800640 1 -- 172.21.15.27:0/2283470412 shutdown_connections 2024-01-15T18:16:56.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.215+0000 7ff34c800640 1 --2- 172.21.15.27:0/2283470412 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3440cff40 0x7ff344161410 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:56.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.215+0000 7ff34c800640 1 --2- 172.21.15.27:0/2283470412 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff34415d0d0 0x7ff3440b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:56.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.215+0000 7ff34c800640 1 -- 172.21.15.27:0/2283470412 >> 172.21.15.27:0/2283470412 conn(0x7ff34406b600 msgr2=0x7ff34406ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:56.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.215+0000 7ff34c800640 1 -- 172.21.15.27:0/2283470412 shutdown_connections 2024-01-15T18:16:56.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.215+0000 7ff34c800640 1 -- 172.21.15.27:0/2283470412 wait complete. 2024-01-15T18:16:56.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.219+0000 7ff34c800640 1 Processor -- start 2024-01-15T18:16:56.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.219+0000 7ff34c800640 1 -- start start 2024-01-15T18:16:56.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.219+0000 7ff34c800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3440cfb20 0x7ff344153850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:56.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.219+0000 7ff34c800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3440cff40 0x7ff3441367c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:56.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.219+0000 7ff34c800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff34415d0d0 0x7ff34414b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:56.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.219+0000 7ff34c800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff344155200 con 0x7ff3440cff40 2024-01-15T18:16:56.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.219+0000 7ff34c800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3441545f0 con 0x7ff3440cfb20 2024-01-15T18:16:56.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.219+0000 7ff34c800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3441548d0 con 0x7ff34415d0d0 2024-01-15T18:16:56.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.219+0000 7ff349400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3440cff40 0x7ff3441367c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:56.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.219+0000 7ff349400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3440cff40 0x7ff3441367c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56562/0 (socket says 172.21.15.27:56562) 2024-01-15T18:16:56.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.219+0000 7ff349400640 1 -- 172.21.15.27:0/247538948 learned_addr learned my addr 172.21.15.27:0/247538948 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:56.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.219+0000 7ff349e00640 1 --2- 172.21.15.27:0/247538948 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3440cfb20 0x7ff344153850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:56.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.219+0000 7ff34a800640 1 --2- 172.21.15.27:0/247538948 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff34415d0d0 0x7ff34414b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:56.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.219+0000 7ff349400640 1 -- 172.21.15.27:0/247538948 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff34415d0d0 msgr2=0x7ff34414b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.219+0000 7ff349400640 1 --2- 172.21.15.27:0/247538948 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff34415d0d0 0x7ff34414b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.219+0000 7ff349400640 1 -- 172.21.15.27:0/247538948 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3440cfb20 msgr2=0x7ff344153850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.219+0000 7ff349400640 1 --2- 172.21.15.27:0/247538948 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3440cfb20 0x7ff344153850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.219+0000 7ff349400640 1 -- 172.21.15.27:0/247538948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff33c009a70 con 0x7ff3440cff40 2024-01-15T18:16:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.219+0000 7ff349400640 1 --2- 172.21.15.27:0/247538948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3440cff40 0x7ff3441367c0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7ff32c00c480 tx=0x7ff32c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.219+0000 7ff342000640 1 -- 172.21.15.27:0/247538948 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c0074e0 con 0x7ff3440cff40 2024-01-15T18:16:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.219+0000 7ff342000640 1 -- 172.21.15.27:0/247538948 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32c007660 con 0x7ff3440cff40 2024-01-15T18:16:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.219+0000 7ff342000640 1 -- 172.21.15.27:0/247538948 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32c004a30 con 0x7ff3440cff40 2024-01-15T18:16:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.219+0000 7ff34c800640 1 -- 172.21.15.27:0/247538948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff344154bb0 con 0x7ff3440cff40 2024-01-15T18:16:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.219+0000 7ff34c800640 1 -- 172.21.15.27:0/247538948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff344153d90 con 0x7ff3440cff40 2024-01-15T18:16:56.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.219+0000 7ff34c800640 1 -- 172.21.15.27:0/247538948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff344155370 con 0x7ff3440cff40 2024-01-15T18:16:56.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.223+0000 7ff342000640 1 -- 172.21.15.27:0/247538948 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 32) v1 ==== 87384+0+0 (secure 0 0 0) 0x7ff32c004b90 con 0x7ff3440cff40 2024-01-15T18:16:56.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.223+0000 7ff342000640 1 --2- 172.21.15.27:0/247538948 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff31c067670 0x7ff31c069b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:56.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.223+0000 7ff342000640 1 -- 172.21.15.27:0/247538948 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5822+0+0 (secure 0 0 0) 0x7ff32c082260 con 0x7ff3440cff40 2024-01-15T18:16:56.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.223+0000 7ff349e00640 1 --2- 172.21.15.27:0/247538948 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff31c067670 0x7ff31c069b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:56.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.223+0000 7ff349e00640 1 --2- 172.21.15.27:0/247538948 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff31c067670 0x7ff31c069b30 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff334007a60 tx=0x7ff334008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:56.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.227+0000 7ff342000640 1 -- 172.21.15.27:0/247538948 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff32c050cf0 con 0x7ff3440cff40 2024-01-15T18:16:56.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.415+0000 7ff34c800640 1 -- 172.21.15.27:0/247538948 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff344161950 con 0x7ff31c067670 2024-01-15T18:16:56.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.419+0000 7ff342000640 1 -- 172.21.15.27:0/247538948 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+79 (secure 0 0 0) 0x7ff334010990 con 0x7ff31c067670 2024-01-15T18:16:56.419 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:16:56.419 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:16:56.419 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:16:56.419 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sv1" 2024-01-15T18:16:56.419 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:16:56.420 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:16:56.420 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ss1clone1" 2024-01-15T18:16:56.420 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:56.420 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:16:56.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.423+0000 7ff34c800640 1 -- 172.21.15.27:0/247538948 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff31c067670 msgr2=0x7ff31c069b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:56.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.423+0000 7ff34c800640 1 --2- 172.21.15.27:0/247538948 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff31c067670 0x7ff31c069b30 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff334007a60 tx=0x7ff334008040 comp rx=0 tx=0).stop 2024-01-15T18:16:56.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.423+0000 7ff34c800640 1 -- 172.21.15.27:0/247538948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3440cff40 msgr2=0x7ff3441367c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:56.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.423+0000 7ff34c800640 1 --2- 172.21.15.27:0/247538948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3440cff40 0x7ff3441367c0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7ff32c00c480 tx=0x7ff32c00c940 comp rx=0 tx=0).stop 2024-01-15T18:16:56.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.423+0000 7ff34c800640 1 -- 172.21.15.27:0/247538948 shutdown_connections 2024-01-15T18:16:56.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.423+0000 7ff34c800640 1 --2- 172.21.15.27:0/247538948 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff31c067670 0x7ff31c069b30 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:56.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.423+0000 7ff34c800640 1 --2- 172.21.15.27:0/247538948 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff34415d0d0 0x7ff34414b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:56.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.423+0000 7ff34c800640 1 --2- 172.21.15.27:0/247538948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3440cff40 0x7ff3441367c0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:56.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.423+0000 7ff34c800640 1 --2- 172.21.15.27:0/247538948 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3440cfb20 0x7ff344153850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:56.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.423+0000 7ff34c800640 1 -- 172.21.15.27:0/247538948 >> 172.21.15.27:0/247538948 conn(0x7ff34406b600 msgr2=0x7ff3440ad760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:56.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.423+0000 7ff34c800640 1 -- 172.21.15.27:0/247538948 shutdown_connections 2024-01-15T18:16:56.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.423+0000 7ff34c800640 1 -- 172.21.15.27:0/247538948 wait complete. 2024-01-15T18:16:56.442 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-15T18:16:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.591+0000 7fbb75200640 1 Processor -- start 2024-01-15T18:16:56.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.591+0000 7fbb75200640 1 -- start start 2024-01-15T18:16:56.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.591+0000 7fbb75200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbb700cca40 0x7fbb700d19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:56.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.591+0000 7fbb75200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb7015b6b0 con 0x7fbb700ce2c0 2024-01-15T18:16:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.591+0000 7fbb75200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb700cfdb0 con 0x7fbb700cef20 2024-01-15T18:16:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.591+0000 7fbb75200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb700cb9e0 con 0x7fbb700cca40 2024-01-15T18:16:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.591+0000 7fbb6d400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbb700ce2c0 0x7fbb700b3d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41008/0 (socket says 172.21.15.27:41008) 2024-01-15T18:16:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.591+0000 7fbb6d400640 1 -- 172.21.15.27:0/513390943 learned_addr learned my addr 172.21.15.27:0/513390943 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:16:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.591+0000 7fbb6e800640 1 --2- 172.21.15.27:0/513390943 >> v2:172.21.15.27:3300/0 conn(0x7fbb700cca40 0x7fbb700d19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:56.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.591+0000 7fbb6ca00640 1 -- 172.21.15.27:0/513390943 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1648891191 0 0) 0x7fbb7015b6b0 con 0x7fbb700ce2c0 2024-01-15T18:16:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.591+0000 7fbb6ca00640 1 -- 172.21.15.27:0/513390943 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb50003530 con 0x7fbb700ce2c0 2024-01-15T18:16:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.591+0000 7fbb6e800640 1 -- 172.21.15.27:0/513390943 >> v1:172.21.15.93:6789/0 conn(0x7fbb700cef20 legacy=0x7fbb700cf300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.591+0000 7fbb6e800640 1 -- 172.21.15.27:0/513390943 >> v1:172.21.15.27:6789/0 conn(0x7fbb700ce2c0 legacy=0x7fbb700b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:56.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.591+0000 7fbb6e800640 1 -- 172.21.15.27:0/513390943 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb700cbc80 con 0x7fbb700cca40 2024-01-15T18:16:56.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.595+0000 7fbb6e800640 1 --2- 172.21.15.27:0/513390943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb700cca40 0x7fbb700d19f0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fbb68009d90 tx=0x7fbb6801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceacbc0722295382 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:56.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.595+0000 7fbb6ca00640 1 -- 172.21.15.27:0/513390943 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb680a5900 con 0x7fbb700cca40 2024-01-15T18:16:56.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.595+0000 7fbb6ca00640 1 -- 172.21.15.27:0/513390943 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb680a5a60 con 0x7fbb700cca40 2024-01-15T18:16:56.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.595+0000 7fbb6ca00640 1 -- 172.21.15.27:0/513390943 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb680a5df0 con 0x7fbb700cca40 2024-01-15T18:16:56.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.595+0000 7fbb75200640 1 -- 172.21.15.27:0/513390943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb700cca40 msgr2=0x7fbb700d19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:56.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.595+0000 7fbb75200640 1 --2- 172.21.15.27:0/513390943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb700cca40 0x7fbb700d19f0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fbb68009d90 tx=0x7fbb6801c950 comp rx=0 tx=0).stop 2024-01-15T18:16:56.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.595+0000 7fbb75200640 1 -- 172.21.15.27:0/513390943 shutdown_connections 2024-01-15T18:16:56.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.595+0000 7fbb75200640 1 --2- 172.21.15.27:0/513390943 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb700cca40 0x7fbb700d19f0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:56.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.595+0000 7fbb75200640 1 -- 172.21.15.27:0/513390943 >> 172.21.15.27:0/513390943 conn(0x7fbb700d0ed0 msgr2=0x7fbb700d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:56.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.595+0000 7fbb75200640 1 -- 172.21.15.27:0/513390943 shutdown_connections 2024-01-15T18:16:56.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.595+0000 7fbb75200640 1 -- 172.21.15.27:0/513390943 wait complete. 2024-01-15T18:16:56.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.599+0000 7fbb75200640 1 Processor -- start 2024-01-15T18:16:56.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.599+0000 7fbb75200640 1 -- start start 2024-01-15T18:16:56.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.599+0000 7fbb75200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbb700cca40 0x7fbb701529f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:56.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.599+0000 7fbb75200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbb700ce2c0 0x7fbb700ca930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:56.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.599+0000 7fbb75200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb700cef20 0x7fbb7015a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:56.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.599+0000 7fbb75200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb700cae70 con 0x7fbb700cef20 2024-01-15T18:16:56.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.599+0000 7fbb75200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb70165890 con 0x7fbb700ce2c0 2024-01-15T18:16:56.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.599+0000 7fbb75200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb70165b70 con 0x7fbb700cca40 2024-01-15T18:16:56.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.599+0000 7fbb6e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb700cef20 0x7fbb7015a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:56.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.599+0000 7fbb6e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb700cef20 0x7fbb7015a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56574/0 (socket says 172.21.15.27:56574) 2024-01-15T18:16:56.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.599+0000 7fbb6e800640 1 -- 172.21.15.27:0/685159880 learned_addr learned my addr 172.21.15.27:0/685159880 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:56.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.599+0000 7fbb6d400640 1 --2- 172.21.15.27:0/685159880 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbb700ce2c0 0x7fbb700ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:56.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.599+0000 7fbb6de00640 1 --2- 172.21.15.27:0/685159880 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbb700cca40 0x7fbb701529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:56.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.599+0000 7fbb6e800640 1 -- 172.21.15.27:0/685159880 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbb700cca40 msgr2=0x7fbb701529f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:56.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.599+0000 7fbb6e800640 1 --2- 172.21.15.27:0/685159880 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbb700cca40 0x7fbb701529f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:56.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.599+0000 7fbb6e800640 1 -- 172.21.15.27:0/685159880 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbb700ce2c0 msgr2=0x7fbb700ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:56.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.599+0000 7fbb6e800640 1 --2- 172.21.15.27:0/685159880 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbb700ce2c0 0x7fbb700ca930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:56.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.599+0000 7fbb6e800640 1 -- 172.21.15.27:0/685159880 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb68009a70 con 0x7fbb700cef20 2024-01-15T18:16:56.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.599+0000 7fbb6d400640 1 --2- 172.21.15.27:0/685159880 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbb700ce2c0 0x7fbb700ca930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:16:56.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.599+0000 7fbb6e800640 1 --2- 172.21.15.27:0/685159880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb700cef20 0x7fbb7015a3c0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fbb6801a580 tx=0x7fbb700b4c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:56.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.599+0000 7fbb66000640 1 -- 172.21.15.27:0/685159880 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb680a5900 con 0x7fbb700cef20 2024-01-15T18:16:56.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.599+0000 7fbb66000640 1 -- 172.21.15.27:0/685159880 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb68003ba0 con 0x7fbb700cef20 2024-01-15T18:16:56.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.599+0000 7fbb75200640 1 -- 172.21.15.27:0/685159880 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb70164470 con 0x7fbb700cef20 2024-01-15T18:16:56.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.599+0000 7fbb66000640 1 -- 172.21.15.27:0/685159880 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb680b9660 con 0x7fbb700cef20 2024-01-15T18:16:56.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.599+0000 7fbb75200640 1 -- 172.21.15.27:0/685159880 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb70164a30 con 0x7fbb700cef20 2024-01-15T18:16:56.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.603+0000 7fbb75200640 1 -- 172.21.15.27:0/685159880 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb700d09f0 con 0x7fbb700cef20 2024-01-15T18:16:56.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.603+0000 7fbb66000640 1 -- 172.21.15.27:0/685159880 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 32) v1 ==== 87384+0+0 (secure 0 0 0) 0x7fbb680b97c0 con 0x7fbb700cef20 2024-01-15T18:16:56.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.603+0000 7fbb66000640 1 --2- 172.21.15.27:0/685159880 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbb48067670 0x7fbb48069b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:56.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.603+0000 7fbb66000640 1 -- 172.21.15.27:0/685159880 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5822+0+0 (secure 0 0 0) 0x7fbb6811d9f0 con 0x7fbb700cef20 2024-01-15T18:16:56.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.603+0000 7fbb6de00640 1 --2- 172.21.15.27:0/685159880 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbb48067670 0x7fbb48069b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:56.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.603+0000 7fbb6de00640 1 --2- 172.21.15.27:0/685159880 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbb48067670 0x7fbb48069b30 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fbb700d0440 tx=0x7fbb60009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:56.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.607+0000 7fbb66000640 1 -- 172.21.15.27:0/685159880 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbb680ec350 con 0x7fbb700cef20 2024-01-15T18:16:56.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.787+0000 7fbb75200640 1 -- 172.21.15.27:0/685159880 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbb700c8240 con 0x7fbb48067670 2024-01-15T18:16:56.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.803+0000 7fbb66000640 1 -- 172.21.15.27:0/685159880 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fbb700c8240 con 0x7fbb48067670 2024-01-15T18:16:56.801 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:16:56.801 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:16:56.801 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:16:56.801 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ss1" 2024-01-15T18:16:56.801 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:16:56.801 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:16:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.803+0000 7fbb75200640 1 -- 172.21.15.27:0/685159880 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbb48067670 msgr2=0x7fbb48069b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.803+0000 7fbb75200640 1 --2- 172.21.15.27:0/685159880 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbb48067670 0x7fbb48069b30 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fbb700d0440 tx=0x7fbb60009040 comp rx=0 tx=0).stop 2024-01-15T18:16:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.803+0000 7fbb75200640 1 -- 172.21.15.27:0/685159880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb700cef20 msgr2=0x7fbb7015a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.803+0000 7fbb75200640 1 --2- 172.21.15.27:0/685159880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb700cef20 0x7fbb7015a3c0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fbb6801a580 tx=0x7fbb700b4c20 comp rx=0 tx=0).stop 2024-01-15T18:16:56.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.803+0000 7fbb75200640 1 -- 172.21.15.27:0/685159880 shutdown_connections 2024-01-15T18:16:56.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.803+0000 7fbb75200640 1 --2- 172.21.15.27:0/685159880 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbb48067670 0x7fbb48069b30 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:56.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.803+0000 7fbb75200640 1 --2- 172.21.15.27:0/685159880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb700cef20 0x7fbb7015a3c0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:56.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.803+0000 7fbb75200640 1 --2- 172.21.15.27:0/685159880 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbb700ce2c0 0x7fbb700ca930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:56.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.803+0000 7fbb75200640 1 --2- 172.21.15.27:0/685159880 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbb700cca40 0x7fbb701529f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:56.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.803+0000 7fbb75200640 1 -- 172.21.15.27:0/685159880 >> 172.21.15.27:0/685159880 conn(0x7fbb700d0ed0 msgr2=0x7fbb7006fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:56.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.803+0000 7fbb75200640 1 -- 172.21.15.27:0/685159880 shutdown_connections 2024-01-15T18:16:56.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.803+0000 7fbb75200640 1 -- 172.21.15.27:0/685159880 wait complete. 2024-01-15T18:16:56.824 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:16:56.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.971+0000 7f59c1400640 1 Processor -- start 2024-01-15T18:16:56.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.971+0000 7f59c1400640 1 -- start start 2024-01-15T18:16:56.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.971+0000 7f59c1400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59bc0cec60 0x7f59bc0b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:56.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.971+0000 7f59c1400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59bc0d19a0 con 0x7f59bc15b4b0 2024-01-15T18:16:56.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.971+0000 7f59c1400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59bc0d1c80 con 0x7f59bc0cf460 2024-01-15T18:16:56.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.971+0000 7f59c1400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59bc0b07c0 con 0x7f59bc0cec60 2024-01-15T18:16:56.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.971+0000 7f59b9e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f59bc15b4b0 0x7f59bc0b1580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41012/0 (socket says 172.21.15.27:41012) 2024-01-15T18:16:56.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.971+0000 7f59b9e00640 1 -- 172.21.15.27:0/3321116462 learned_addr learned my addr 172.21.15.27:0/3321116462 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:16:56.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.971+0000 7f59bb200640 1 --2- 172.21.15.27:0/3321116462 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59bc0cec60 0x7f59bc0b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:56.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.971+0000 7f59b9400640 1 -- 172.21.15.27:0/3321116462 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869584317 0 0) 0x7f59bc0d19a0 con 0x7f59bc15b4b0 2024-01-15T18:16:56.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.971+0000 7f59b9400640 1 -- 172.21.15.27:0/3321116462 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f599c003530 con 0x7f59bc15b4b0 2024-01-15T18:16:56.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.971+0000 7f59b9400640 1 -- 172.21.15.27:0/3321116462 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3803025273 0 0) 0x7f59bc0d1c80 con 0x7f59bc0cf460 2024-01-15T18:16:56.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.971+0000 7f59b9400640 1 -- 172.21.15.27:0/3321116462 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59bc0d19a0 con 0x7f59bc0cf460 2024-01-15T18:16:56.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.975+0000 7f59b9400640 1 -- 172.21.15.27:0/3321116462 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1684619321 0 0) 0x7f599c003530 con 0x7f59bc15b4b0 2024-01-15T18:16:56.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.975+0000 7f59bb200640 1 -- 172.21.15.27:0/3321116462 >> v1:172.21.15.93:6789/0 conn(0x7f59bc0cf460 legacy=0x7f59bc0cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:56.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.975+0000 7f59bb200640 1 -- 172.21.15.27:0/3321116462 >> v1:172.21.15.27:6789/0 conn(0x7f59bc15b4b0 legacy=0x7f59bc0b1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:56.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.975+0000 7f59bb200640 1 -- 172.21.15.27:0/3321116462 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59bc0b0aa0 con 0x7f59bc0cec60 2024-01-15T18:16:56.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.975+0000 7f59bb200640 1 --2- 172.21.15.27:0/3321116462 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59bc0cec60 0x7f59bc0b1c90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f59b4009dc0 tx=0x7f59b401c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=188194fb64b21dfd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:56.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.975+0000 7f59b9400640 1 -- 172.21.15.27:0/3321116462 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a4001680 con 0x7f59bc0cec60 2024-01-15T18:16:56.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.975+0000 7f59b9400640 1 -- 172.21.15.27:0/3321116462 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b40a56d0 con 0x7f59bc0cec60 2024-01-15T18:16:56.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.975+0000 7f59b9400640 1 -- 172.21.15.27:0/3321116462 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b40a5a60 con 0x7f59bc0cec60 2024-01-15T18:16:56.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.975+0000 7f59c1400640 1 -- 172.21.15.27:0/3321116462 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59bc0cec60 msgr2=0x7f59bc0b1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:56.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.975+0000 7f59c1400640 1 --2- 172.21.15.27:0/3321116462 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59bc0cec60 0x7f59bc0b1c90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f59b4009dc0 tx=0x7f59b401c890 comp rx=0 tx=0).stop 2024-01-15T18:16:56.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.979+0000 7f59c1400640 1 -- 172.21.15.27:0/3321116462 shutdown_connections 2024-01-15T18:16:56.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.979+0000 7f59c1400640 1 --2- 172.21.15.27:0/3321116462 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59bc0cec60 0x7f59bc0b1c90 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:56.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.979+0000 7f59c1400640 1 -- 172.21.15.27:0/3321116462 >> 172.21.15.27:0/3321116462 conn(0x7f59bc0d0e80 msgr2=0x7f59bc0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:56.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.979+0000 7f59c1400640 1 -- 172.21.15.27:0/3321116462 shutdown_connections 2024-01-15T18:16:56.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.979+0000 7f59c1400640 1 -- 172.21.15.27:0/3321116462 wait complete. 2024-01-15T18:16:56.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.979+0000 7f59c1400640 1 Processor -- start 2024-01-15T18:16:56.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.979+0000 7f59c1400640 1 -- start start 2024-01-15T18:16:56.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.979+0000 7f59c1400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59bc0cec60 0x7f59bc0ae820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:56.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.979+0000 7f59c1400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59bc0cf460 0x7f59bc0aed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:56.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.979+0000 7f59c1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59bc15b4b0 0x7f59bc14a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:56.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.979+0000 7f59c1400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59bc157ce0 con 0x7f59bc15b4b0 2024-01-15T18:16:56.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.979+0000 7f59c1400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59bc0ca110 con 0x7f59bc0cf460 2024-01-15T18:16:56.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.979+0000 7f59c1400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59bc0ca3f0 con 0x7f59bc0cec60 2024-01-15T18:16:56.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.979+0000 7f59bb200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59bc15b4b0 0x7f59bc14a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:56.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.979+0000 7f59bb200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59bc15b4b0 0x7f59bc14a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56576/0 (socket says 172.21.15.27:56576) 2024-01-15T18:16:56.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.979+0000 7f59bb200640 1 -- 172.21.15.27:0/2254692185 learned_addr learned my addr 172.21.15.27:0/2254692185 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:56.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.979+0000 7f59ba800640 1 --2- 172.21.15.27:0/2254692185 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59bc0cec60 0x7f59bc0ae820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:56.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.979+0000 7f59b9e00640 1 --2- 172.21.15.27:0/2254692185 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59bc0cf460 0x7f59bc0aed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:56.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.979+0000 7f59bb200640 1 -- 172.21.15.27:0/2254692185 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59bc0cec60 msgr2=0x7f59bc0ae820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:56.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.979+0000 7f59bb200640 1 --2- 172.21.15.27:0/2254692185 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59bc0cec60 0x7f59bc0ae820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:56.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.979+0000 7f59bb200640 1 -- 172.21.15.27:0/2254692185 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59bc0cf460 msgr2=0x7f59bc0aed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:56.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.979+0000 7f59bb200640 1 --2- 172.21.15.27:0/2254692185 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59bc0cf460 0x7f59bc0aed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:56.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.983+0000 7f59bb200640 1 -- 172.21.15.27:0/2254692185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b4009a70 con 0x7f59bc15b4b0 2024-01-15T18:16:56.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.983+0000 7f59bb200640 1 --2- 172.21.15.27:0/2254692185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59bc15b4b0 0x7f59bc14a860 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f59b401cd70 tx=0x7f59b4019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:56.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.983+0000 7f59b2a00640 1 -- 172.21.15.27:0/2254692185 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b40a5ea0 con 0x7f59bc15b4b0 2024-01-15T18:16:56.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.983+0000 7f59b2a00640 1 -- 172.21.15.27:0/2254692185 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b40b9860 con 0x7f59bc15b4b0 2024-01-15T18:16:56.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.983+0000 7f59b2a00640 1 -- 172.21.15.27:0/2254692185 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59b40b9c10 con 0x7f59bc15b4b0 2024-01-15T18:16:56.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.983+0000 7f59c1400640 1 -- 172.21.15.27:0/2254692185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59bc0ca6d0 con 0x7f59bc15b4b0 2024-01-15T18:16:56.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.983+0000 7f59c1400640 1 -- 172.21.15.27:0/2254692185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59bc154950 con 0x7f59bc15b4b0 2024-01-15T18:16:56.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.983+0000 7f59c1400640 1 -- 172.21.15.27:0/2254692185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59bc0afd00 con 0x7f59bc15b4b0 2024-01-15T18:16:56.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.983+0000 7f59b2a00640 1 -- 172.21.15.27:0/2254692185 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 32) v1 ==== 87384+0+0 (secure 0 0 0) 0x7f59b40c4460 con 0x7f59bc15b4b0 2024-01-15T18:16:56.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.983+0000 7f59b2a00640 1 --2- 172.21.15.27:0/2254692185 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5994067670 0x7f5994069b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:56.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.983+0000 7f59b2a00640 1 -- 172.21.15.27:0/2254692185 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5822+0+0 (secure 0 0 0) 0x7f59b411e070 con 0x7f59bc15b4b0 2024-01-15T18:16:56.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.983+0000 7f59ba800640 1 --2- 172.21.15.27:0/2254692185 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5994067670 0x7f5994069b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:56.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.987+0000 7f59ba800640 1 --2- 172.21.15.27:0/2254692185 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5994067670 0x7f5994069b30 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f59ac0011f0 tx=0x7f59ac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:56.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:56.987+0000 7f59b2a00640 1 -- 172.21.15.27:0/2254692185 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f59b40ecb00 con 0x7f59bc15b4b0 2024-01-15T18:16:57.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.175+0000 7f59c1400640 1 -- 172.21.15.27:0/2254692185 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f59bc0ca280 con 0x7f5994067670 2024-01-15T18:16:57.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.203+0000 7f59b2a00640 1 -- 172.21.15.27:0/2254692185 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f59bc0ca280 con 0x7f5994067670 2024-01-15T18:16:57.202 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:16:57.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.207+0000 7f59c1400640 1 -- 172.21.15.27:0/2254692185 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5994067670 msgr2=0x7f5994069b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:57.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.207+0000 7f59c1400640 1 --2- 172.21.15.27:0/2254692185 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5994067670 0x7f5994069b30 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f59ac0011f0 tx=0x7f59ac009040 comp rx=0 tx=0).stop 2024-01-15T18:16:57.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.207+0000 7f59c1400640 1 -- 172.21.15.27:0/2254692185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59bc15b4b0 msgr2=0x7f59bc14a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:57.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.207+0000 7f59c1400640 1 --2- 172.21.15.27:0/2254692185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59bc15b4b0 0x7f59bc14a860 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f59b401cd70 tx=0x7f59b4019630 comp rx=0 tx=0).stop 2024-01-15T18:16:57.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.207+0000 7f59c1400640 1 -- 172.21.15.27:0/2254692185 shutdown_connections 2024-01-15T18:16:57.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.207+0000 7f59c1400640 1 --2- 172.21.15.27:0/2254692185 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5994067670 0x7f5994069b30 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:57.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.207+0000 7f59c1400640 1 --2- 172.21.15.27:0/2254692185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59bc15b4b0 0x7f59bc14a860 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:57.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.207+0000 7f59c1400640 1 --2- 172.21.15.27:0/2254692185 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59bc0cf460 0x7f59bc0aed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:57.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.207+0000 7f59c1400640 1 --2- 172.21.15.27:0/2254692185 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59bc0cec60 0x7f59bc0ae820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:57.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.207+0000 7f59c1400640 1 -- 172.21.15.27:0/2254692185 >> 172.21.15.27:0/2254692185 conn(0x7f59bc0d0e80 msgr2=0x7f59bc154d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:57.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.207+0000 7f59c1400640 1 -- 172.21.15.27:0/2254692185 shutdown_connections 2024-01-15T18:16:57.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.207+0000 7f59c1400640 1 -- 172.21.15.27:0/2254692185 wait complete. 2024-01-15T18:16:57.224 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-15T18:16:57.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.371+0000 7f2b03200640 1 Processor -- start 2024-01-15T18:16:57.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.371+0000 7f2b03200640 1 -- start start 2024-01-15T18:16:57.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.371+0000 7f2b03200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2afc0ce760 0x7f2afc0b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:57.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.371+0000 7f2b03200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2afc0712b0 con 0x7f2afc06c430 2024-01-15T18:16:57.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.371+0000 7f2b03200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2afc0ccfb0 con 0x7f2afc0b1db0 2024-01-15T18:16:57.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.371+0000 7f2b03200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2afc0cd290 con 0x7f2afc0ce760 2024-01-15T18:16:57.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.371+0000 7f2afbe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2afc06c430 0x7f2afc070ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41022/0 (socket says 172.21.15.27:41022) 2024-01-15T18:16:57.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.371+0000 7f2afbe00640 1 -- 172.21.15.27:0/3095749769 learned_addr learned my addr 172.21.15.27:0/3095749769 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:16:57.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.371+0000 7f2b01200640 1 --2- 172.21.15.27:0/3095749769 >> v2:172.21.15.27:3300/0 conn(0x7f2afc0ce760 0x7f2afc0b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:57.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.371+0000 7f2afaa00640 1 -- 172.21.15.27:0/3095749769 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275828333 0 0) 0x7f2afc0712b0 con 0x7f2afc06c430 2024-01-15T18:16:57.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.371+0000 7f2afaa00640 1 -- 172.21.15.27:0/3095749769 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2adc003450 con 0x7f2afc06c430 2024-01-15T18:16:57.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.371+0000 7f2b01200640 1 -- 172.21.15.27:0/3095749769 >> v1:172.21.15.93:6789/0 conn(0x7f2afc0b1db0 legacy=0x7f2afc0ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:57.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.371+0000 7f2b01200640 1 -- 172.21.15.27:0/3095749769 >> v1:172.21.15.27:6789/0 conn(0x7f2afc06c430 legacy=0x7f2afc070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:57.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.371+0000 7f2b01200640 1 -- 172.21.15.27:0/3095749769 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2afc0cd570 con 0x7f2afc0ce760 2024-01-15T18:16:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.375+0000 7f2b01200640 1 --2- 172.21.15.27:0/3095749769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2afc0ce760 0x7f2afc0b2240 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f2af4009dc0 tx=0x7f2af401c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e8e5949acd70e63 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:57.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.375+0000 7f2afaa00640 1 -- 172.21.15.27:0/3095749769 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af40a58b0 con 0x7f2afc0ce760 2024-01-15T18:16:57.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.375+0000 7f2afaa00640 1 -- 172.21.15.27:0/3095749769 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af40a5a10 con 0x7f2afc0ce760 2024-01-15T18:16:57.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.375+0000 7f2afaa00640 1 -- 172.21.15.27:0/3095749769 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af40a5da0 con 0x7f2afc0ce760 2024-01-15T18:16:57.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.375+0000 7f2b03200640 1 -- 172.21.15.27:0/3095749769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2afc0ce760 msgr2=0x7f2afc0b2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:57.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.375+0000 7f2b03200640 1 --2- 172.21.15.27:0/3095749769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2afc0ce760 0x7f2afc0b2240 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f2af4009dc0 tx=0x7f2af401c870 comp rx=0 tx=0).stop 2024-01-15T18:16:57.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.375+0000 7f2b03200640 1 -- 172.21.15.27:0/3095749769 shutdown_connections 2024-01-15T18:16:57.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.375+0000 7f2b03200640 1 --2- 172.21.15.27:0/3095749769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2afc0ce760 0x7f2afc0b2240 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:57.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.375+0000 7f2b03200640 1 -- 172.21.15.27:0/3095749769 >> 172.21.15.27:0/3095749769 conn(0x7f2afc0cb700 msgr2=0x7f2afc0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:57.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.375+0000 7f2b03200640 1 -- 172.21.15.27:0/3095749769 shutdown_connections 2024-01-15T18:16:57.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.375+0000 7f2b03200640 1 -- 172.21.15.27:0/3095749769 wait complete. 2024-01-15T18:16:57.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.379+0000 7f2b03200640 1 Processor -- start 2024-01-15T18:16:57.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.379+0000 7f2b03200640 1 -- start start 2024-01-15T18:16:57.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.379+0000 7f2b03200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2afc06c430 0x7f2afc14d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.379+0000 7f2b03200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2afc0b1db0 0x7f2afc146e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.379+0000 7f2b03200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2afc0ce760 0x7f2afc144690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.379+0000 7f2b03200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc1593c0 con 0x7f2afc06c430 2024-01-15T18:16:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.379+0000 7f2b03200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc145400 con 0x7f2afc0b1db0 2024-01-15T18:16:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.379+0000 7f2b03200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc1456a0 con 0x7f2afc0ce760 2024-01-15T18:16:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.379+0000 7f2afbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2afc06c430 0x7f2afc14d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.379+0000 7f2afbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2afc06c430 0x7f2afc14d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56596/0 (socket says 172.21.15.27:56596) 2024-01-15T18:16:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.379+0000 7f2afbe00640 1 -- 172.21.15.27:0/657913526 learned_addr learned my addr 172.21.15.27:0/657913526 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.379+0000 7f2b01200640 1 --2- 172.21.15.27:0/657913526 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2afc0ce760 0x7f2afc144690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.379+0000 7f2afb400640 1 --2- 172.21.15.27:0/657913526 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2afc0b1db0 0x7f2afc146e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.379+0000 7f2afbe00640 1 -- 172.21.15.27:0/657913526 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2afc0ce760 msgr2=0x7f2afc144690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.379+0000 7f2afbe00640 1 --2- 172.21.15.27:0/657913526 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2afc0ce760 0x7f2afc144690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.379+0000 7f2afbe00640 1 -- 172.21.15.27:0/657913526 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2afc0b1db0 msgr2=0x7f2afc146e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.379+0000 7f2afbe00640 1 --2- 172.21.15.27:0/657913526 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2afc0b1db0 0x7f2afc146e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.379+0000 7f2afbe00640 1 -- 172.21.15.27:0/657913526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af4009a70 con 0x7f2afc06c430 2024-01-15T18:16:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.379+0000 7f2afbe00640 1 --2- 172.21.15.27:0/657913526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2afc06c430 0x7f2afc14d070 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f2aec00c4d0 tx=0x7f2aec00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.379+0000 7f2af3e00640 1 -- 172.21.15.27:0/657913526 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aec005cf0 con 0x7f2afc06c430 2024-01-15T18:16:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.379+0000 7f2af3e00640 1 -- 172.21.15.27:0/657913526 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aec005e50 con 0x7f2afc06c430 2024-01-15T18:16:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.379+0000 7f2af3e00640 1 -- 172.21.15.27:0/657913526 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aec007270 con 0x7f2afc06c430 2024-01-15T18:16:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.379+0000 7f2b03200640 1 -- 172.21.15.27:0/657913526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2afc145980 con 0x7f2afc06c430 2024-01-15T18:16:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.379+0000 7f2b03200640 1 -- 172.21.15.27:0/657913526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2afc145f40 con 0x7f2afc06c430 2024-01-15T18:16:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.383+0000 7f2b03200640 1 -- 172.21.15.27:0/657913526 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2afc073c10 con 0x7f2afc06c430 2024-01-15T18:16:57.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.383+0000 7f2af3e00640 1 -- 172.21.15.27:0/657913526 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 32) v1 ==== 87384+0+0 (secure 0 0 0) 0x7f2aec0073d0 con 0x7f2afc06c430 2024-01-15T18:16:57.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.383+0000 7f2af3e00640 1 --2- 172.21.15.27:0/657913526 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2ad40676a0 0x7f2ad4069b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:57.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.383+0000 7f2af3e00640 1 -- 172.21.15.27:0/657913526 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5822+0+0 (secure 0 0 0) 0x7f2aec082570 con 0x7f2afc06c430 2024-01-15T18:16:57.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.383+0000 7f2afb400640 1 --2- 172.21.15.27:0/657913526 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2ad40676a0 0x7f2ad4069b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:57.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.383+0000 7f2afb400640 1 --2- 172.21.15.27:0/657913526 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2ad40676a0 0x7f2ad4069b60 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f2afc146b40 tx=0x7f2ae4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:57.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.387+0000 7f2af3e00640 1 -- 172.21.15.27:0/657913526 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2aec051060 con 0x7f2afc06c430 2024-01-15T18:16:57.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.567+0000 7f2b03200640 1 -- 172.21.15.27:0/657913526 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f2afc15ced0 con 0x7f2ad40676a0 2024-01-15T18:16:57.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.587+0000 7f2af3e00640 1 -- 172.21.15.27:0/657913526 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2afc15ced0 con 0x7f2ad40676a0 2024-01-15T18:16:57.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.591+0000 7f2b03200640 1 -- 172.21.15.27:0/657913526 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2ad40676a0 msgr2=0x7f2ad4069b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:57.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.591+0000 7f2b03200640 1 --2- 172.21.15.27:0/657913526 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2ad40676a0 0x7f2ad4069b60 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f2afc146b40 tx=0x7f2ae4009040 comp rx=0 tx=0).stop 2024-01-15T18:16:57.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.591+0000 7f2b03200640 1 -- 172.21.15.27:0/657913526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2afc06c430 msgr2=0x7f2afc14d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:57.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.591+0000 7f2b03200640 1 --2- 172.21.15.27:0/657913526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2afc06c430 0x7f2afc14d070 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f2aec00c4d0 tx=0x7f2aec00c990 comp rx=0 tx=0).stop 2024-01-15T18:16:57.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.591+0000 7f2b03200640 1 -- 172.21.15.27:0/657913526 shutdown_connections 2024-01-15T18:16:57.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.591+0000 7f2b03200640 1 --2- 172.21.15.27:0/657913526 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2ad40676a0 0x7f2ad4069b60 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:57.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.591+0000 7f2b03200640 1 --2- 172.21.15.27:0/657913526 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2afc0ce760 0x7f2afc144690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:57.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.591+0000 7f2b03200640 1 --2- 172.21.15.27:0/657913526 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2afc0b1db0 0x7f2afc146e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:57.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.591+0000 7f2b03200640 1 --2- 172.21.15.27:0/657913526 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2afc06c430 0x7f2afc14d070 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:57.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.591+0000 7f2b03200640 1 -- 172.21.15.27:0/657913526 >> 172.21.15.27:0/657913526 conn(0x7f2afc0cb700 msgr2=0x7f2afc14d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:57.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.591+0000 7f2b03200640 1 -- 172.21.15.27:0/657913526 shutdown_connections 2024-01-15T18:16:57.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.591+0000 7f2b03200640 1 -- 172.21.15.27:0/657913526 wait complete. 2024-01-15T18:16:57.592 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:16:57.588+0000 7f59f4c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:16:57.592 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:16:57.588+0000 7f59f4c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:16:57.592 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:16:57.588+0000 7f59f4c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:16:57.592 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:16:57.588+0000 7f59f4c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:16:57.592 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:16:57.588+0000 7f59f4c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:16:57.610 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:16:57.608+0000 7f59f6000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:16:57.610 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:16:57.608+0000 7f59f6000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:16:57.610 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:16:57.608+0000 7f59f6000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:16:57.610 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:16:57.608+0000 7f59f6000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:16:57.610 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:16:57.608+0000 7f59f6000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:16:57.612 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs ss1clone1 2024-01-15T18:16:57.669 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:16:57.675 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:16:57.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.763+0000 7f3291800640 1 Processor -- start 2024-01-15T18:16:57.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.763+0000 7f3291800640 1 -- start start 2024-01-15T18:16:57.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.763+0000 7f3291800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f328c0cf460 0x7f328c0cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:57.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.763+0000 7f3291800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f328c0b2590 0x7f328c0cc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:57.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.763+0000 7f3291800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328c0cfd80 con 0x7f328c15b4b0 2024-01-15T18:16:57.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.763+0000 7f3291800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f328c0d19a0 con 0x7f328c0b2590 2024-01-15T18:16:57.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.763+0000 7f3291800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f328c0d1c80 con 0x7f328c0cf460 2024-01-15T18:16:57.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.763+0000 7f328b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f328c0b2590 0x7f328c0cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:57.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.763+0000 7f328a000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f328c15b4b0 0x7f328c0cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41024/0 (socket says 172.21.15.27:41024) 2024-01-15T18:16:57.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.763+0000 7f328a000640 1 -- 172.21.15.27:0/84290301 learned_addr learned my addr 172.21.15.27:0/84290301 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:16:57.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.763+0000 7f328aa00640 1 --2- 172.21.15.27:0/84290301 >> v2:172.21.15.93:3300/0 conn(0x7f328c0cf460 0x7f328c0cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:57.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.763+0000 7f328b400640 1 -- 172.21.15.27:0/84290301 >> v2:172.21.15.93:3300/0 conn(0x7f328c0cf460 msgr2=0x7f328c0cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:16:57.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.763+0000 7f328b400640 1 --2- 172.21.15.27:0/84290301 >> v2:172.21.15.93:3300/0 conn(0x7f328c0cf460 0x7f328c0cf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:57.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.763+0000 7f328b400640 1 -- 172.21.15.27:0/84290301 >> v1:172.21.15.27:6789/0 conn(0x7f328c15b4b0 legacy=0x7f328c0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:57.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.763+0000 7f328b400640 1 -- 172.21.15.27:0/84290301 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328c0b07c0 con 0x7f328c0b2590 2024-01-15T18:16:57.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.763+0000 7f328aa00640 1 --2- 172.21.15.27:0/84290301 >> v2:172.21.15.93:3300/0 conn(0x7f328c0cf460 0x7f328c0cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:16:57.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.767+0000 7f328b400640 1 --2- 172.21.15.27:0/84290301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f328c0b2590 0x7f328c0cc130 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f328c0681f0 tx=0x7f328401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbd5e3843f435faa server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:57.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.767+0000 7f3289600640 1 -- 172.21.15.27:0/84290301 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32840a59d0 con 0x7f328c0b2590 2024-01-15T18:16:57.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.767+0000 7f3289600640 1 -- 172.21.15.27:0/84290301 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32840a5b30 con 0x7f328c0b2590 2024-01-15T18:16:57.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.767+0000 7f3289600640 1 -- 172.21.15.27:0/84290301 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32840ad440 con 0x7f328c0b2590 2024-01-15T18:16:57.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.767+0000 7f3291800640 1 -- 172.21.15.27:0/84290301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f328c0b2590 msgr2=0x7f328c0cc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:57.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.767+0000 7f3291800640 1 --2- 172.21.15.27:0/84290301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f328c0b2590 0x7f328c0cc130 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f328c0681f0 tx=0x7f328401c910 comp rx=0 tx=0).stop 2024-01-15T18:16:57.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.767+0000 7f3291800640 1 -- 172.21.15.27:0/84290301 shutdown_connections 2024-01-15T18:16:57.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.767+0000 7f3291800640 1 --2- 172.21.15.27:0/84290301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f328c0b2590 0x7f328c0cc130 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:57.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.767+0000 7f3291800640 1 --2- 172.21.15.27:0/84290301 >> v2:172.21.15.93:3300/0 conn(0x7f328c0cf460 0x7f328c0cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:57.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.767+0000 7f3291800640 1 -- 172.21.15.27:0/84290301 >> 172.21.15.27:0/84290301 conn(0x7f328c0d0e80 msgr2=0x7f328c0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:57.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.767+0000 7f3291800640 1 -- 172.21.15.27:0/84290301 shutdown_connections 2024-01-15T18:16:57.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.767+0000 7f3291800640 1 -- 172.21.15.27:0/84290301 wait complete. 2024-01-15T18:16:57.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.771+0000 7f3291800640 1 Processor -- start 2024-01-15T18:16:57.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.771+0000 7f3291800640 1 -- start start 2024-01-15T18:16:57.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.771+0000 7f3291800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f328c0b2590 0x7f328c0c7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:57.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.771+0000 7f3291800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f328c0cf460 0x7f328c0c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:57.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.771+0000 7f3291800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f328c15b4b0 0x7f328c0ca520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:57.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.771+0000 7f3291800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328c15c7e0 con 0x7f328c0b2590 2024-01-15T18:16:57.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.771+0000 7f3291800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328c15cac0 con 0x7f328c0cf460 2024-01-15T18:16:57.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.771+0000 7f3291800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f328c15cda0 con 0x7f328c15b4b0 2024-01-15T18:16:57.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.771+0000 7f328aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f328c0b2590 0x7f328c0c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:57.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.771+0000 7f328aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f328c0b2590 0x7f328c0c7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56610/0 (socket says 172.21.15.27:56610) 2024-01-15T18:16:57.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.771+0000 7f328aa00640 1 -- 172.21.15.27:0/1886427212 learned_addr learned my addr 172.21.15.27:0/1886427212 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:16:57.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.771+0000 7f328b400640 1 --2- 172.21.15.27:0/1886427212 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f328c15b4b0 0x7f328c0ca520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:57.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.771+0000 7f328a000640 1 --2- 172.21.15.27:0/1886427212 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f328c0cf460 0x7f328c0c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:57.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.771+0000 7f328aa00640 1 -- 172.21.15.27:0/1886427212 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f328c15b4b0 msgr2=0x7f328c0ca520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:57.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.771+0000 7f328aa00640 1 --2- 172.21.15.27:0/1886427212 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f328c15b4b0 0x7f328c0ca520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:57.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.771+0000 7f328aa00640 1 -- 172.21.15.27:0/1886427212 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f328c0cf460 msgr2=0x7f328c0c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:57.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.771+0000 7f328aa00640 1 --2- 172.21.15.27:0/1886427212 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f328c0cf460 0x7f328c0c9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:57.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.771+0000 7f328aa00640 1 -- 172.21.15.27:0/1886427212 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3284009a70 con 0x7f328c0b2590 2024-01-15T18:16:57.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.771+0000 7f328aa00640 1 --2- 172.21.15.27:0/1886427212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f328c0b2590 0x7f328c0c7350 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f327c00c7c0 tx=0x7f327c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:57.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.771+0000 7f3282a00640 1 -- 172.21.15.27:0/1886427212 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c007a30 con 0x7f328c0b2590 2024-01-15T18:16:57.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.771+0000 7f3282a00640 1 -- 172.21.15.27:0/1886427212 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f327c007b90 con 0x7f328c0b2590 2024-01-15T18:16:57.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.771+0000 7f3282a00640 1 -- 172.21.15.27:0/1886427212 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c00ce70 con 0x7f328c0b2590 2024-01-15T18:16:57.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.771+0000 7f3291800640 1 -- 172.21.15.27:0/1886427212 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f328c15d080 con 0x7f328c0b2590 2024-01-15T18:16:57.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.771+0000 7f3291800640 1 -- 172.21.15.27:0/1886427212 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f328c150c70 con 0x7f328c0b2590 2024-01-15T18:16:57.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.775+0000 7f3291800640 1 -- 172.21.15.27:0/1886427212 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f328c15d2c0 con 0x7f328c0b2590 2024-01-15T18:16:57.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.775+0000 7f3282a00640 1 -- 172.21.15.27:0/1886427212 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 32) v1 ==== 87384+0+0 (secure 0 0 0) 0x7f327c0113f0 con 0x7f328c0b2590 2024-01-15T18:16:57.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.775+0000 7f3282a00640 1 --2- 172.21.15.27:0/1886427212 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3264067670 0x7f3264069b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:16:57.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.775+0000 7f3282a00640 1 -- 172.21.15.27:0/1886427212 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5822+0+0 (secure 0 0 0) 0x7f327c0834d0 con 0x7f328c0b2590 2024-01-15T18:16:57.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.775+0000 7f328a000640 1 --2- 172.21.15.27:0/1886427212 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3264067670 0x7f3264069b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:16:57.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.775+0000 7f328a000640 1 --2- 172.21.15.27:0/1886427212 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3264067670 0x7f3264069b30 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f328c0afee0 tx=0x7f3274009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:16:57.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.779+0000 7f3282a00640 1 -- 172.21.15.27:0/1886427212 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f328c15d2c0 con 0x7f328c0b2590 2024-01-15T18:16:57.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.963+0000 7f3291800640 1 -- 172.21.15.27:0/1886427212 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f328c15d2c0 con 0x7f3264067670 2024-01-15T18:16:57.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.983+0000 7f3282a00640 1 -- 172.21.15.27:0/1886427212 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f328c15d2c0 con 0x7f3264067670 2024-01-15T18:16:57.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.983+0000 7f3291800640 1 -- 172.21.15.27:0/1886427212 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3264067670 msgr2=0x7f3264069b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:57.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.983+0000 7f3291800640 1 --2- 172.21.15.27:0/1886427212 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3264067670 0x7f3264069b30 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f328c0afee0 tx=0x7f3274009040 comp rx=0 tx=0).stop 2024-01-15T18:16:57.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.983+0000 7f3291800640 1 -- 172.21.15.27:0/1886427212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f328c0b2590 msgr2=0x7f328c0c7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:16:57.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.983+0000 7f3291800640 1 --2- 172.21.15.27:0/1886427212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f328c0b2590 0x7f328c0c7350 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f327c00c7c0 tx=0x7f327c00cc80 comp rx=0 tx=0).stop 2024-01-15T18:16:57.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.983+0000 7f3291800640 1 -- 172.21.15.27:0/1886427212 shutdown_connections 2024-01-15T18:16:57.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.983+0000 7f3291800640 1 --2- 172.21.15.27:0/1886427212 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3264067670 0x7f3264069b30 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:57.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.983+0000 7f3291800640 1 --2- 172.21.15.27:0/1886427212 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f328c15b4b0 0x7f328c0ca520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:57.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.983+0000 7f3291800640 1 --2- 172.21.15.27:0/1886427212 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f328c0cf460 0x7f328c0c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:57.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.983+0000 7f3291800640 1 --2- 172.21.15.27:0/1886427212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f328c0b2590 0x7f328c0c7350 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:16:57.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.983+0000 7f3291800640 1 -- 172.21.15.27:0/1886427212 >> 172.21.15.27:0/1886427212 conn(0x7f328c0d0e80 msgr2=0x7f328c0adb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:16:57.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.983+0000 7f3291800640 1 -- 172.21.15.27:0/1886427212 shutdown_connections 2024-01-15T18:16:57.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:16:57.983+0000 7f3291800640 1 -- 172.21.15.27:0/1886427212 wait complete. 2024-01-15T18:16:58.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:16:58.003 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-15T18:17:00.288 INFO:teuthology.orchestra.run.smithi027.stdout:2 2024-01-15T18:17:00.290 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-15T18:17:00.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.531+0000 7fbfdfe00640 1 Processor -- start 2024-01-15T18:17:00.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.531+0000 7fbfdfe00640 1 -- start start 2024-01-15T18:17:00.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.531+0000 7fbfdfe00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fbfd81617f0 0x7fbfd806e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:00.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.535+0000 7fbfdfe00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfd806ea50 con 0x7fbfd806dad0 2024-01-15T18:17:00.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.535+0000 7fbfdfe00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfd80cf680 con 0x7fbfd8072360 2024-01-15T18:17:00.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.535+0000 7fbfdfe00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfd8161bd0 con 0x7fbfd81617f0 2024-01-15T18:17:00.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.535+0000 7fbfdca00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbfd806dad0 0x7fbfd8071c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41038/0 (socket says 172.21.15.27:41038) 2024-01-15T18:17:00.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.535+0000 7fbfdca00640 1 -- 172.21.15.27:0/2953457747 learned_addr learned my addr 172.21.15.27:0/2953457747 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:00.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.535+0000 7fbfdd400640 1 --2- 172.21.15.27:0/2953457747 >> v2:172.21.15.93:3300/0 conn(0x7fbfd81617f0 0x7fbfd806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:00.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.535+0000 7fbfd7e00640 1 -- 172.21.15.27:0/2953457747 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623966642 0 0) 0x7fbfd806ea50 con 0x7fbfd806dad0 2024-01-15T18:17:00.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.535+0000 7fbfd7e00640 1 -- 172.21.15.27:0/2953457747 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfb8003530 con 0x7fbfd806dad0 2024-01-15T18:17:00.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.535+0000 7fbfdd400640 1 -- 172.21.15.27:0/2953457747 >> v1:172.21.15.93:6789/0 conn(0x7fbfd8072360 legacy=0x7fbfd80b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:00.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.535+0000 7fbfdd400640 1 -- 172.21.15.27:0/2953457747 >> v1:172.21.15.27:6789/0 conn(0x7fbfd806dad0 legacy=0x7fbfd8071c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:00.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.535+0000 7fbfdd400640 1 -- 172.21.15.27:0/2953457747 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd80cf960 con 0x7fbfd81617f0 2024-01-15T18:17:00.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.539+0000 7fbfdd400640 1 --2- 172.21.15.27:0/2953457747 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfd81617f0 0x7fbfd806e350 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fbfc80097c0 tx=0x7fbfc801c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b988f1543516cb6e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:00.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.543+0000 7fbfd7e00640 1 -- 172.21.15.27:0/2953457747 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc80ab680 con 0x7fbfd81617f0 2024-01-15T18:17:00.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.543+0000 7fbfd7e00640 1 -- 172.21.15.27:0/2953457747 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc80ab7e0 con 0x7fbfd81617f0 2024-01-15T18:17:00.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.543+0000 7fbfd7e00640 1 -- 172.21.15.27:0/2953457747 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc80abb70 con 0x7fbfd81617f0 2024-01-15T18:17:00.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.543+0000 7fbfdfe00640 1 -- 172.21.15.27:0/2953457747 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfd81617f0 msgr2=0x7fbfd806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:00.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.543+0000 7fbfdfe00640 1 --2- 172.21.15.27:0/2953457747 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfd81617f0 0x7fbfd806e350 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fbfc80097c0 tx=0x7fbfc801c480 comp rx=0 tx=0).stop 2024-01-15T18:17:00.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.543+0000 7fbfdfe00640 1 -- 172.21.15.27:0/2953457747 shutdown_connections 2024-01-15T18:17:00.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.543+0000 7fbfdfe00640 1 --2- 172.21.15.27:0/2953457747 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfd81617f0 0x7fbfd806e350 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:00.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.543+0000 7fbfdfe00640 1 -- 172.21.15.27:0/2953457747 >> 172.21.15.27:0/2953457747 conn(0x7fbfd8099e10 msgr2=0x7fbfd80b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:00.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.543+0000 7fbfdfe00640 1 -- 172.21.15.27:0/2953457747 shutdown_connections 2024-01-15T18:17:00.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.543+0000 7fbfdfe00640 1 -- 172.21.15.27:0/2953457747 wait complete. 2024-01-15T18:17:00.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.547+0000 7fbfdfe00640 1 Processor -- start 2024-01-15T18:17:00.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.547+0000 7fbfdfe00640 1 -- start start 2024-01-15T18:17:00.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.551+0000 7fbfdfe00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfd806dad0 0x7fbfd8164800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:00.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.551+0000 7fbfdfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfd8072360 0x7fbfd80cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:00.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.551+0000 7fbfdfe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbfd81617f0 0x7fbfd8157930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:00.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.551+0000 7fbfdfe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd815f500 con 0x7fbfd8072360 2024-01-15T18:17:00.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.551+0000 7fbfdfe00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd80cde00 con 0x7fbfd806dad0 2024-01-15T18:17:00.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.551+0000 7fbfdfe00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd8162d30 con 0x7fbfd81617f0 2024-01-15T18:17:00.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.551+0000 7fbfdca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfd8072360 0x7fbfd80cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:00.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.551+0000 7fbfdca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfd8072360 0x7fbfd80cd8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56616/0 (socket says 172.21.15.27:56616) 2024-01-15T18:17:00.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.551+0000 7fbfdca00640 1 -- 172.21.15.27:0/380688936 learned_addr learned my addr 172.21.15.27:0/380688936 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:00.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.551+0000 7fbfdd400640 1 --2- 172.21.15.27:0/380688936 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfd806dad0 0x7fbfd8164800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:00.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.551+0000 7fbfdca00640 1 -- 172.21.15.27:0/380688936 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbfd81617f0 msgr2=0x7fbfd8157930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:00.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.551+0000 7fbfdca00640 1 --2- 172.21.15.27:0/380688936 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbfd81617f0 0x7fbfd8157930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:00.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.551+0000 7fbfdca00640 1 -- 172.21.15.27:0/380688936 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfd806dad0 msgr2=0x7fbfd8164800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:00.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.551+0000 7fbfdca00640 1 --2- 172.21.15.27:0/380688936 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfd806dad0 0x7fbfd8164800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:00.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.551+0000 7fbfdca00640 1 -- 172.21.15.27:0/380688936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc8009420 con 0x7fbfd8072360 2024-01-15T18:17:00.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.551+0000 7fbfdd400640 1 --2- 172.21.15.27:0/380688936 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfd806dad0 0x7fbfd8164800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:17:00.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.551+0000 7fbfdca00640 1 --2- 172.21.15.27:0/380688936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfd8072360 0x7fbfd80cd8c0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fbfc000c500 tx=0x7fbfc000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:00.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.551+0000 7fbfd5600640 1 -- 172.21.15.27:0/380688936 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc0005cf0 con 0x7fbfd8072360 2024-01-15T18:17:00.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.551+0000 7fbfd5600640 1 -- 172.21.15.27:0/380688936 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfc0005e50 con 0x7fbfd8072360 2024-01-15T18:17:00.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.551+0000 7fbfd5600640 1 -- 172.21.15.27:0/380688936 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfc00072f0 con 0x7fbfd8072360 2024-01-15T18:17:00.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.555+0000 7fbfdfe00640 1 -- 172.21.15.27:0/380688936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd80d05c0 con 0x7fbfd8072360 2024-01-15T18:17:00.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.555+0000 7fbfdfe00640 1 -- 172.21.15.27:0/380688936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd8157ed0 con 0x7fbfd8072360 2024-01-15T18:17:00.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.555+0000 7fbfcea00640 1 -- 172.21.15.27:0/380688936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfd809e950 con 0x7fbfd8072360 2024-01-15T18:17:00.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.559+0000 7fbfd5600640 1 -- 172.21.15.27:0/380688936 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 33) v1 ==== 87486+0+0 (secure 0 0 0) 0x7fbfc0007450 con 0x7fbfd8072360 2024-01-15T18:17:00.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.559+0000 7fbfd5600640 1 --2- 172.21.15.27:0/380688936 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbfb0067860 0x7fbfb0069d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:00.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.559+0000 7fbfd5600640 1 -- 172.21.15.27:0/380688936 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5822+0+0 (secure 0 0 0) 0x7fbfc0082600 con 0x7fbfd8072360 2024-01-15T18:17:00.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.563+0000 7fbfdd400640 1 --2- 172.21.15.27:0/380688936 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbfb0067860 0x7fbfb0069d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:00.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.563+0000 7fbfdd400640 1 --2- 172.21.15.27:0/380688936 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbfb0067860 0x7fbfb0069d20 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fbfc80af000 tx=0x7fbfc80a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:00.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.571+0000 7fbfd5600640 1 -- 172.21.15.27:0/380688936 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbfc0051060 con 0x7fbfd8072360 2024-01-15T18:17:00.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:00.943+0000 7fbfcea00640 1 -- 172.21.15.27:0/380688936 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7fbfd80d0440 con 0x7fbfb0067860 2024-01-15T18:17:00.991 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-01-15T18:17:00.991+0000 7f053dc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-15T18:17:01.087 INFO:tasks.ceph.mds.b.smithi093.stdout:starting mds.b at 2024-01-15T18:17:03.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:03.163+0000 7fbfd5600640 1 -- 172.21.15.27:0/380688936 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7fbfc0051430 con 0x7fbfd8072360 2024-01-15T18:17:04.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.031+0000 7fbfd5600640 1 -- 172.21.15.27:0/380688936 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fbfc80ab950 con 0x7fbfb0067860 2024-01-15T18:17:04.030 INFO:teuthology.orchestra.run.smithi027.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-15T18:17:04.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.035+0000 7fbfcea00640 1 -- 172.21.15.27:0/380688936 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbfb0067860 msgr2=0x7fbfb0069d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:04.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.035+0000 7fbfcea00640 1 --2- 172.21.15.27:0/380688936 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbfb0067860 0x7fbfb0069d20 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fbfc80af000 tx=0x7fbfc80a7000 comp rx=0 tx=0).stop 2024-01-15T18:17:04.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.035+0000 7fbfcea00640 1 -- 172.21.15.27:0/380688936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfd8072360 msgr2=0x7fbfd80cd8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:04.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.035+0000 7fbfcea00640 1 --2- 172.21.15.27:0/380688936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfd8072360 0x7fbfd80cd8c0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fbfc000c500 tx=0x7fbfc000c9c0 comp rx=0 tx=0).stop 2024-01-15T18:17:04.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.035+0000 7fbfcea00640 1 -- 172.21.15.27:0/380688936 shutdown_connections 2024-01-15T18:17:04.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.035+0000 7fbfcea00640 1 --2- 172.21.15.27:0/380688936 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbfb0067860 0x7fbfb0069d20 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:04.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.035+0000 7fbfcea00640 1 --2- 172.21.15.27:0/380688936 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbfd81617f0 0x7fbfd8157930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:04.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.035+0000 7fbfcea00640 1 --2- 172.21.15.27:0/380688936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbfd8072360 0x7fbfd80cd8c0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:04.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.035+0000 7fbfcea00640 1 --2- 172.21.15.27:0/380688936 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbfd806dad0 0x7fbfd8164800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:04.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.035+0000 7fbfcea00640 1 -- 172.21.15.27:0/380688936 >> 172.21.15.27:0/380688936 conn(0x7fbfd8099e10 msgr2=0x7fbfd80af5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:04.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.035+0000 7fbfcea00640 1 -- 172.21.15.27:0/380688936 shutdown_connections 2024-01-15T18:17:04.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.035+0000 7fbfcea00640 1 -- 172.21.15.27:0/380688936 wait complete. 2024-01-15T18:17:04.071 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T18:17:04.072 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:17:04.170 DEBUG:teuthology.orchestra.run.smithi093:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:17:04.266 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T18:17:04.267 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:17:04.267 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:17:04.277 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:17:04.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.527+0000 7f594a600640 1 Processor -- start 2024-01-15T18:17:04.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.527+0000 7f594a600640 1 -- start start 2024-01-15T18:17:04.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.527+0000 7f594a600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f594415e7d0 0x7f59440cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:04.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.527+0000 7f594a600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59440b38b0 con 0x7f59440cedc0 2024-01-15T18:17:04.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.527+0000 7f594a600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59440cd550 con 0x7f59440b20d0 2024-01-15T18:17:04.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.527+0000 7f594a600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59440cd270 con 0x7f594415e7d0 2024-01-15T18:17:04.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.527+0000 7f5942a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f59440cedc0 0x7f59440cba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47020/0 (socket says 172.21.15.27:47020) 2024-01-15T18:17:04.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.527+0000 7f5942a00640 1 -- 172.21.15.27:0/4258076623 learned_addr learned my addr 172.21.15.27:0/4258076623 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:04.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.527+0000 7f5943400640 1 --2- 172.21.15.27:0/4258076623 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f594415e7d0 0x7f59440cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:04.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.527+0000 7f5942000640 1 -- 172.21.15.27:0/4258076623 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1141000976 0 0) 0x7f59440b38b0 con 0x7f59440cedc0 2024-01-15T18:17:04.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.527+0000 7f5942000640 1 -- 172.21.15.27:0/4258076623 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5924003530 con 0x7f59440cedc0 2024-01-15T18:17:04.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.527+0000 7f5942000640 1 -- 172.21.15.27:0/4258076623 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 329315846 0 0) 0x7f59440cd550 con 0x7f59440b20d0 2024-01-15T18:17:04.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.527+0000 7f5942000640 1 -- 172.21.15.27:0/4258076623 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59440b38b0 con 0x7f59440b20d0 2024-01-15T18:17:04.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.527+0000 7f5943400640 1 -- 172.21.15.27:0/4258076623 >> v1:172.21.15.93:6789/0 conn(0x7f59440b20d0 legacy=0x7f59440cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:04.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.527+0000 7f5943400640 1 -- 172.21.15.27:0/4258076623 >> v1:172.21.15.27:6789/0 conn(0x7f59440cedc0 legacy=0x7f59440cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:04.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.527+0000 7f5942000640 1 -- 172.21.15.27:0/4258076623 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4279259336 0 0) 0x7f5924003530 con 0x7f59440cedc0 2024-01-15T18:17:04.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.527+0000 7f5943400640 1 -- 172.21.15.27:0/4258076623 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59440b35d0 con 0x7f594415e7d0 2024-01-15T18:17:04.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.531+0000 7f5943400640 1 --2- 172.21.15.27:0/4258076623 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f594415e7d0 0x7f59440cf2a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f59340097f0 tx=0x7f593401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f60aa58c91e93a1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:04.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.531+0000 7f5942000640 1 -- 172.21.15.27:0/4258076623 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c001680 con 0x7f594415e7d0 2024-01-15T18:17:04.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.531+0000 7f5942000640 1 -- 172.21.15.27:0/4258076623 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59340a7050 con 0x7f594415e7d0 2024-01-15T18:17:04.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.531+0000 7f5942000640 1 -- 172.21.15.27:0/4258076623 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59340ab620 con 0x7f594415e7d0 2024-01-15T18:17:04.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.531+0000 7f594a600640 1 -- 172.21.15.27:0/4258076623 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f594415e7d0 msgr2=0x7f59440cf2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:04.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.531+0000 7f594a600640 1 --2- 172.21.15.27:0/4258076623 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f594415e7d0 0x7f59440cf2a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f59340097f0 tx=0x7f593401c3e0 comp rx=0 tx=0).stop 2024-01-15T18:17:04.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.535+0000 7f594a600640 1 -- 172.21.15.27:0/4258076623 shutdown_connections 2024-01-15T18:17:04.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.535+0000 7f594a600640 1 --2- 172.21.15.27:0/4258076623 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f594415e7d0 0x7f59440cf2a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:04.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.535+0000 7f594a600640 1 -- 172.21.15.27:0/4258076623 >> 172.21.15.27:0/4258076623 conn(0x7f59440b2a10 msgr2=0x7f59440b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:04.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.535+0000 7f594a600640 1 -- 172.21.15.27:0/4258076623 shutdown_connections 2024-01-15T18:17:04.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.535+0000 7f594a600640 1 -- 172.21.15.27:0/4258076623 wait complete. 2024-01-15T18:17:04.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.535+0000 7f594a600640 1 Processor -- start 2024-01-15T18:17:04.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.535+0000 7f594a600640 1 -- start start 2024-01-15T18:17:04.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.535+0000 7f594a600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59440b20d0 0x7f59440c93f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:04.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.535+0000 7f594a600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59440cedc0 0x7f59441571c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:04.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.535+0000 7f594a600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f594415e7d0 0x7f5944157700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:04.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.535+0000 7f594a600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59440c8fd0 con 0x7f594415e7d0 2024-01-15T18:17:04.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.535+0000 7f594a600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5944157c40 con 0x7f59440cedc0 2024-01-15T18:17:04.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.535+0000 7f594a600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59440c5d10 con 0x7f59440b20d0 2024-01-15T18:17:04.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.535+0000 7f5943e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f594415e7d0 0x7f5944157700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:04.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.535+0000 7f5942a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59440cedc0 0x7f59441571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:04.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.535+0000 7f5943400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59440b20d0 0x7f59440c93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:04.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.535+0000 7f5942a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59440cedc0 0x7f59441571c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:46490/0 (socket says 172.21.15.27:46490) 2024-01-15T18:17:04.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.535+0000 7f5942a00640 1 -- 172.21.15.27:0/1257974802 learned_addr learned my addr 172.21.15.27:0/1257974802 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:04.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.535+0000 7f5942a00640 1 -- 172.21.15.27:0/1257974802 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59440b20d0 msgr2=0x7f59440c93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:04.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.535+0000 7f5942a00640 1 --2- 172.21.15.27:0/1257974802 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59440b20d0 0x7f59440c93f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:04.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.535+0000 7f5942a00640 1 -- 172.21.15.27:0/1257974802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f594415e7d0 msgr2=0x7f5944157700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:04.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.535+0000 7f5942a00640 1 --2- 172.21.15.27:0/1257974802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f594415e7d0 0x7f5944157700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:04.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.535+0000 7f5942a00640 1 -- 172.21.15.27:0/1257974802 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5934009420 con 0x7f59440cedc0 2024-01-15T18:17:04.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.539+0000 7f5942a00640 1 --2- 172.21.15.27:0/1257974802 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59440cedc0 0x7f59441571c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f592c0011f0 tx=0x7f592c00cbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:04.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.539+0000 7f593b400640 1 -- 172.21.15.27:0/1257974802 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c005ae0 con 0x7f59440cedc0 2024-01-15T18:17:04.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.539+0000 7f593b400640 1 -- 172.21.15.27:0/1257974802 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f592c005c40 con 0x7f59440cedc0 2024-01-15T18:17:04.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.539+0000 7f593b400640 1 -- 172.21.15.27:0/1257974802 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c007070 con 0x7f59440cedc0 2024-01-15T18:17:04.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.539+0000 7f594a600640 1 -- 172.21.15.27:0/1257974802 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59440c9930 con 0x7f59440cedc0 2024-01-15T18:17:04.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.539+0000 7f594a600640 1 -- 172.21.15.27:0/1257974802 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59440cad60 con 0x7f59440cedc0 2024-01-15T18:17:04.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.539+0000 7f594a600640 1 -- 172.21.15.27:0/1257974802 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59440c9110 con 0x7f59440cedc0 2024-01-15T18:17:04.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.539+0000 7f593b400640 1 -- 172.21.15.27:0/1257974802 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f592c0071f0 con 0x7f59440cedc0 2024-01-15T18:17:04.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.539+0000 7f593b400640 1 --2- 172.21.15.27:0/1257974802 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f591c067700 0x7f591c069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:04.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.539+0000 7f593b400640 1 -- 172.21.15.27:0/1257974802 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f592c082530 con 0x7f59440cedc0 2024-01-15T18:17:04.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.539+0000 7f5943400640 1 --2- 172.21.15.27:0/1257974802 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f591c067700 0x7f591c069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:04.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.543+0000 7f5943400640 1 --2- 172.21.15.27:0/1257974802 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f591c067700 0x7f591c069bc0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f59340ad000 tx=0x7f59340a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.543+0000 7f593b400640 1 -- 172.21.15.27:0/1257974802 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f59440c9110 con 0x7f59440cedc0 2024-01-15T18:17:04.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.711+0000 7f594a600640 1 -- 172.21.15.27:0/1257974802 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59440c9930 con 0x7f59440cedc0 2024-01-15T18:17:04.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.715+0000 7f593b400640 1 -- 172.21.15.27:0/1257974802 <== mon.1 v2:172.21.15.93:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+13066 (secure 0 0 0) 0x7f59440c9930 con 0x7f59440cedc0 2024-01-15T18:17:04.713 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:17:04.713 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:17:04.008565+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.93:6837/980414657":"2024-01-16T18:17:00.989983+0000","172.21.15.93:6835/980414657":"2024-01-16T18:17:00.989983+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:17:04.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.715+0000 7f594a600640 1 -- 172.21.15.27:0/1257974802 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f591c067700 msgr2=0x7f591c069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:04.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.715+0000 7f594a600640 1 --2- 172.21.15.27:0/1257974802 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f591c067700 0x7f591c069bc0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f59340ad000 tx=0x7f59340a6000 comp rx=0 tx=0).stop 2024-01-15T18:17:04.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.715+0000 7f594a600640 1 -- 172.21.15.27:0/1257974802 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59440cedc0 msgr2=0x7f59441571c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:04.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.715+0000 7f594a600640 1 --2- 172.21.15.27:0/1257974802 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59440cedc0 0x7f59441571c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f592c0011f0 tx=0x7f592c00cbe0 comp rx=0 tx=0).stop 2024-01-15T18:17:04.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.715+0000 7f594a600640 1 -- 172.21.15.27:0/1257974802 shutdown_connections 2024-01-15T18:17:04.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.715+0000 7f594a600640 1 --2- 172.21.15.27:0/1257974802 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f591c067700 0x7f591c069bc0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:04.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.715+0000 7f594a600640 1 --2- 172.21.15.27:0/1257974802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f594415e7d0 0x7f5944157700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:04.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.715+0000 7f594a600640 1 --2- 172.21.15.27:0/1257974802 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59440cedc0 0x7f59441571c0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:04.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.715+0000 7f594a600640 1 --2- 172.21.15.27:0/1257974802 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59440b20d0 0x7f59440c93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:04.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.715+0000 7f594a600640 1 -- 172.21.15.27:0/1257974802 >> 172.21.15.27:0/1257974802 conn(0x7f59440b2a10 msgr2=0x7f59440afd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:04.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.715+0000 7f594a600640 1 -- 172.21.15.27:0/1257974802 shutdown_connections 2024-01-15T18:17:04.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.715+0000 7f594a600640 1 -- 172.21.15.27:0/1257974802 wait complete. 2024-01-15T18:17:04.735 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:17:04.735 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:17:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.879+0000 7fdba2c00640 1 Processor -- start 2024-01-15T18:17:04.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.879+0000 7fdba2c00640 1 -- start start 2024-01-15T18:17:04.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.883+0000 7fdba2c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb9c15d190 0x7fdb9c0cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:04.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.883+0000 7fdba2c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb9c0b0d70 con 0x7fdb9c06c520 2024-01-15T18:17:04.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.883+0000 7fdba2c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb9c0d0540 con 0x7fdb9c15d570 2024-01-15T18:17:04.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.883+0000 7fdba2c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9c0b0440 con 0x7fdb9c15d190 2024-01-15T18:17:04.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.883+0000 7fdba0c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdb9c06c520 0x7fdb9c0cd5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47028/0 (socket says 172.21.15.27:47028) 2024-01-15T18:17:04.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.883+0000 7fdba0c00640 1 -- 172.21.15.27:0/3854385868 learned_addr learned my addr 172.21.15.27:0/3854385868 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:04.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.883+0000 7fdb9be00640 1 --2- 172.21.15.27:0/3854385868 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb9c15d190 0x7fdb9c0cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:04.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.883+0000 7fdb9aa00640 1 -- 172.21.15.27:0/3854385868 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3671523923 0 0) 0x7fdb9c0d0540 con 0x7fdb9c15d570 2024-01-15T18:17:04.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.883+0000 7fdb9aa00640 1 -- 172.21.15.27:0/3854385868 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb7c003530 con 0x7fdb9c15d570 2024-01-15T18:17:04.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.883+0000 7fdb9aa00640 1 -- 172.21.15.27:0/3854385868 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 673298527 0 0) 0x7fdb9c0b0d70 con 0x7fdb9c06c520 2024-01-15T18:17:04.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.883+0000 7fdb9aa00640 1 -- 172.21.15.27:0/3854385868 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb9c0d0540 con 0x7fdb9c06c520 2024-01-15T18:17:04.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.883+0000 7fdb9be00640 1 -- 172.21.15.27:0/3854385868 >> v1:172.21.15.93:6789/0 conn(0x7fdb9c15d570 legacy=0x7fdb9c06be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:04.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.883+0000 7fdb9be00640 1 -- 172.21.15.27:0/3854385868 >> v1:172.21.15.27:6789/0 conn(0x7fdb9c06c520 legacy=0x7fdb9c0cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:04.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.883+0000 7fdb9be00640 1 -- 172.21.15.27:0/3854385868 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb9c0b0720 con 0x7fdb9c15d190 2024-01-15T18:17:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.887+0000 7fdb9be00640 1 --2- 172.21.15.27:0/3854385868 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb9c15d190 0x7fdb9c0cfcf0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c0097f0 tx=0x7fdb8c01c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edf95cbf648e9418 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.887+0000 7fdb9aa00640 1 -- 172.21.15.27:0/3854385868 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c0ab650 con 0x7fdb9c15d190 2024-01-15T18:17:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.887+0000 7fdb9aa00640 1 -- 172.21.15.27:0/3854385868 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb8c0ab7b0 con 0x7fdb9c15d190 2024-01-15T18:17:04.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.887+0000 7fdb9aa00640 1 -- 172.21.15.27:0/3854385868 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c0abb40 con 0x7fdb9c15d190 2024-01-15T18:17:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.887+0000 7fdba2c00640 1 -- 172.21.15.27:0/3854385868 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb9c15d190 msgr2=0x7fdb9c0cfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.887+0000 7fdba2c00640 1 --2- 172.21.15.27:0/3854385868 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb9c15d190 0x7fdb9c0cfcf0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c0097f0 tx=0x7fdb8c01c490 comp rx=0 tx=0).stop 2024-01-15T18:17:04.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.887+0000 7fdba2c00640 1 -- 172.21.15.27:0/3854385868 shutdown_connections 2024-01-15T18:17:04.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.887+0000 7fdba2c00640 1 --2- 172.21.15.27:0/3854385868 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb9c15d190 0x7fdb9c0cfcf0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:04.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.887+0000 7fdba2c00640 1 -- 172.21.15.27:0/3854385868 >> 172.21.15.27:0/3854385868 conn(0x7fdb9c0d0e70 msgr2=0x7fdb9c0d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:04.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.887+0000 7fdba2c00640 1 -- 172.21.15.27:0/3854385868 shutdown_connections 2024-01-15T18:17:04.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.887+0000 7fdba2c00640 1 -- 172.21.15.27:0/3854385868 wait complete. 2024-01-15T18:17:04.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.887+0000 7fdba2c00640 1 Processor -- start 2024-01-15T18:17:04.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.887+0000 7fdba2c00640 1 -- start start 2024-01-15T18:17:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdba2c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdb9c06c520 0x7fdb9c14e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdba2c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb9c15d190 0x7fdb9c14b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdba2c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb9c15d570 0x7fdb9c06eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdba2c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9c0b6da0 con 0x7fdb9c15d570 2024-01-15T18:17:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdba2c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9c0c7340 con 0x7fdb9c06c520 2024-01-15T18:17:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdba2c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9c0b6ac0 con 0x7fdb9c15d190 2024-01-15T18:17:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdba0c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb9c15d570 0x7fdb9c06eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:04.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdb9be00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdb9c06c520 0x7fdb9c14e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdb9b400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb9c15d190 0x7fdb9c14b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdba0c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb9c15d570 0x7fdb9c06eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48352/0 (socket says 172.21.15.27:48352) 2024-01-15T18:17:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdba0c00640 1 -- 172.21.15.27:0/3455902991 learned_addr learned my addr 172.21.15.27:0/3455902991 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdb9be00640 1 --2- 172.21.15.27:0/3455902991 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdb9c06c520 0x7fdb9c14e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:46498/0 (socket says 172.21.15.27:46498) 2024-01-15T18:17:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdba0c00640 1 -- 172.21.15.27:0/3455902991 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb9c15d190 msgr2=0x7fdb9c14b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdba0c00640 1 --2- 172.21.15.27:0/3455902991 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb9c15d190 0x7fdb9c14b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdba0c00640 1 -- 172.21.15.27:0/3455902991 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdb9c06c520 msgr2=0x7fdb9c14e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdba0c00640 1 --2- 172.21.15.27:0/3455902991 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdb9c06c520 0x7fdb9c14e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdba0c00640 1 -- 172.21.15.27:0/3455902991 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8c009420 con 0x7fdb9c15d570 2024-01-15T18:17:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdb9be00640 1 --2- 172.21.15.27:0/3455902991 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdb9c06c520 0x7fdb9c14e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:17:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdba0c00640 1 --2- 172.21.15.27:0/3455902991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb9c15d570 0x7fdb9c06eb70 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fdb9400cbc0 tx=0x7fdb9400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdb93e00640 1 -- 172.21.15.27:0/3455902991 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9400aca0 con 0x7fdb9c15d570 2024-01-15T18:17:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdb93e00640 1 -- 172.21.15.27:0/3455902991 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9400ae00 con 0x7fdb9c15d570 2024-01-15T18:17:04.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdba2c00640 1 -- 172.21.15.27:0/3455902991 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb9c0c6cf0 con 0x7fdb9c15d570 2024-01-15T18:17:04.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdb93e00640 1 -- 172.21.15.27:0/3455902991 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb94012640 con 0x7fdb9c15d570 2024-01-15T18:17:04.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdba2c00640 1 -- 172.21.15.27:0/3455902991 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb9c14b970 con 0x7fdb9c15d570 2024-01-15T18:17:04.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdba2c00640 1 -- 172.21.15.27:0/3455902991 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb9c0c6b50 con 0x7fdb9c15d570 2024-01-15T18:17:04.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.891+0000 7fdb93e00640 1 -- 172.21.15.27:0/3455902991 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7fdb94007070 con 0x7fdb9c15d570 2024-01-15T18:17:04.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.895+0000 7fdb93e00640 1 --2- 172.21.15.27:0/3455902991 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdb74067700 0x7fdb74069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:04.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.895+0000 7fdb93e00640 1 -- 172.21.15.27:0/3455902991 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5055+0+0 (secure 0 0 0) 0x7fdb94083070 con 0x7fdb9c15d570 2024-01-15T18:17:04.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.895+0000 7fdb9be00640 1 --2- 172.21.15.27:0/3455902991 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdb74067700 0x7fdb74069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:04.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.895+0000 7fdb9be00640 1 --2- 172.21.15.27:0/3455902991 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdb74067700 0x7fdb74069bc0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c0af000 tx=0x7fdb8c0a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:04.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:04.899+0000 7fdb93e00640 1 -- 172.21.15.27:0/3455902991 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fdb9c0c6b50 con 0x7fdb9c15d570 2024-01-15T18:17:05.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.055+0000 7fdba2c00640 1 -- 172.21.15.27:0/3455902991 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdb9c0c6b50 con 0x7fdb9c15d570 2024-01-15T18:17:05.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.059+0000 7fdb93e00640 1 -- 172.21.15.27:0/3455902991 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdb9c0c6b50 con 0x7fdb9c15d570 2024-01-15T18:17:05.056 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:17:05.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.059+0000 7fdba2c00640 1 -- 172.21.15.27:0/3455902991 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdb74067700 msgr2=0x7fdb74069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:05.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.059+0000 7fdba2c00640 1 --2- 172.21.15.27:0/3455902991 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdb74067700 0x7fdb74069bc0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c0af000 tx=0x7fdb8c0a7000 comp rx=0 tx=0).stop 2024-01-15T18:17:05.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.059+0000 7fdba2c00640 1 -- 172.21.15.27:0/3455902991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb9c15d570 msgr2=0x7fdb9c06eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:05.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.059+0000 7fdba2c00640 1 --2- 172.21.15.27:0/3455902991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb9c15d570 0x7fdb9c06eb70 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fdb9400cbc0 tx=0x7fdb9400a060 comp rx=0 tx=0).stop 2024-01-15T18:17:05.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.059+0000 7fdba2c00640 1 -- 172.21.15.27:0/3455902991 shutdown_connections 2024-01-15T18:17:05.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.059+0000 7fdba2c00640 1 --2- 172.21.15.27:0/3455902991 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdb74067700 0x7fdb74069bc0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:05.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.059+0000 7fdba2c00640 1 --2- 172.21.15.27:0/3455902991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb9c15d570 0x7fdb9c06eb70 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:05.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.059+0000 7fdba2c00640 1 --2- 172.21.15.27:0/3455902991 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdb9c15d190 0x7fdb9c14b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:05.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.059+0000 7fdba2c00640 1 --2- 172.21.15.27:0/3455902991 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdb9c06c520 0x7fdb9c14e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:05.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.059+0000 7fdba2c00640 1 -- 172.21.15.27:0/3455902991 >> 172.21.15.27:0/3455902991 conn(0x7fdb9c0d0e70 msgr2=0x7fdb9c1548b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:05.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.059+0000 7fdba2c00640 1 -- 172.21.15.27:0/3455902991 shutdown_connections 2024-01-15T18:17:05.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.059+0000 7fdba2c00640 1 -- 172.21.15.27:0/3455902991 wait complete. 2024-01-15T18:17:05.078 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:17:05.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.219+0000 7fe463600640 1 Processor -- start 2024-01-15T18:17:05.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.219+0000 7fe463600640 1 -- start start 2024-01-15T18:17:05.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.223+0000 7fe463600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fe45c143da0 0x7fe45c06d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:05.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.223+0000 7fe463600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe45c0cad40 0x7fe45c0cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:05.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.223+0000 7fe463600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe45c06c750 con 0x7fe45c06c200 2024-01-15T18:17:05.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.223+0000 7fe463600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe45c0cb710 con 0x7fe45c143da0 2024-01-15T18:17:05.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.223+0000 7fe463600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c144180 con 0x7fe45c0cad40 2024-01-15T18:17:05.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.223+0000 7fe461600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe45c0cad40 0x7fe45c0cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:05.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.223+0000 7fe45be00640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7fe45c06c200 0x7fe45c0cb9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:37734/0 (socket says 172.21.15.27:37734) 2024-01-15T18:17:05.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.223+0000 7fe45be00640 1 -- 172.21.15.27:0/1409466846 learned_addr learned my addr 172.21.15.27:0/1409466846 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:05.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.223+0000 7fe460c00640 1 --2- 172.21.15.27:0/1409466846 >> v2:172.21.15.93:3300/0 conn(0x7fe45c143da0 0x7fe45c06d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:05.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.223+0000 7fe461600640 1 -- 172.21.15.27:0/1409466846 >> v2:172.21.15.93:3300/0 conn(0x7fe45c143da0 msgr2=0x7fe45c06d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:05.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.223+0000 7fe461600640 1 --2- 172.21.15.27:0/1409466846 >> v2:172.21.15.93:3300/0 conn(0x7fe45c143da0 0x7fe45c06d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:05.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.223+0000 7fe461600640 1 -- 172.21.15.27:0/1409466846 >> v1:172.21.15.93:6789/0 conn(0x7fe45c06c200 legacy=0x7fe45c0cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:05.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.223+0000 7fe461600640 1 -- 172.21.15.27:0/1409466846 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe45c06ca30 con 0x7fe45c0cad40 2024-01-15T18:17:05.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.223+0000 7fe460c00640 1 --2- 172.21.15.27:0/1409466846 >> v2:172.21.15.93:3300/0 conn(0x7fe45c143da0 0x7fe45c06d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:17:05.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.227+0000 7fe461600640 1 --2- 172.21.15.27:0/1409466846 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe45c0cad40 0x7fe45c0cb1d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe454009d90 tx=0x7fe45401c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d106cef4e7f71ca server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:05.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.227+0000 7fe45b400640 1 -- 172.21.15.27:0/1409466846 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4540a5900 con 0x7fe45c0cad40 2024-01-15T18:17:05.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.227+0000 7fe45b400640 1 -- 172.21.15.27:0/1409466846 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4540a5a60 con 0x7fe45c0cad40 2024-01-15T18:17:05.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.227+0000 7fe45b400640 1 -- 172.21.15.27:0/1409466846 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4540a5df0 con 0x7fe45c0cad40 2024-01-15T18:17:05.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.227+0000 7fe463600640 1 -- 172.21.15.27:0/1409466846 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe45c0cad40 msgr2=0x7fe45c0cb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:05.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.227+0000 7fe463600640 1 --2- 172.21.15.27:0/1409466846 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe45c0cad40 0x7fe45c0cb1d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe454009d90 tx=0x7fe45401c950 comp rx=0 tx=0).stop 2024-01-15T18:17:05.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.227+0000 7fe463600640 1 -- 172.21.15.27:0/1409466846 shutdown_connections 2024-01-15T18:17:05.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.227+0000 7fe463600640 1 --2- 172.21.15.27:0/1409466846 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe45c0cad40 0x7fe45c0cb1d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:05.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.227+0000 7fe463600640 1 --2- 172.21.15.27:0/1409466846 >> v2:172.21.15.93:3300/0 conn(0x7fe45c143da0 0x7fe45c06d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:05.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.227+0000 7fe463600640 1 -- 172.21.15.27:0/1409466846 >> 172.21.15.27:0/1409466846 conn(0x7fe45c0b0850 msgr2=0x7fe45c0b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:05.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.227+0000 7fe463600640 1 -- 172.21.15.27:0/1409466846 shutdown_connections 2024-01-15T18:17:05.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.227+0000 7fe463600640 1 -- 172.21.15.27:0/1409466846 wait complete. 2024-01-15T18:17:05.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.231+0000 7fe463600640 1 Processor -- start 2024-01-15T18:17:05.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.231+0000 7fe463600640 1 -- start start 2024-01-15T18:17:05.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.231+0000 7fe463600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe45c06c200 0x7fe45c15d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:05.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.231+0000 7fe463600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe45c0cad40 0x7fe45c15d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:05.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.231+0000 7fe463600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe45c143da0 0x7fe45c136d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:05.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.231+0000 7fe463600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c15bd40 con 0x7fe45c143da0 2024-01-15T18:17:05.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.231+0000 7fe463600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c0ae290 con 0x7fe45c06c200 2024-01-15T18:17:05.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.231+0000 7fe463600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c15b3a0 con 0x7fe45c0cad40 2024-01-15T18:17:05.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.231+0000 7fe461600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe45c143da0 0x7fe45c136d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:05.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.231+0000 7fe461600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe45c143da0 0x7fe45c136d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48358/0 (socket says 172.21.15.27:48358) 2024-01-15T18:17:05.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.231+0000 7fe461600640 1 -- 172.21.15.27:0/1062144094 learned_addr learned my addr 172.21.15.27:0/1062144094 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:05.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.231+0000 7fe460c00640 1 --2- 172.21.15.27:0/1062144094 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe45c06c200 0x7fe45c15d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:05.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.231+0000 7fe45be00640 1 --2- 172.21.15.27:0/1062144094 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe45c0cad40 0x7fe45c15d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:05.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.231+0000 7fe461600640 1 -- 172.21.15.27:0/1062144094 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe45c0cad40 msgr2=0x7fe45c15d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:05.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.231+0000 7fe461600640 1 --2- 172.21.15.27:0/1062144094 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe45c0cad40 0x7fe45c15d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:05.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.231+0000 7fe461600640 1 -- 172.21.15.27:0/1062144094 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe45c06c200 msgr2=0x7fe45c15d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:05.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.231+0000 7fe461600640 1 --2- 172.21.15.27:0/1062144094 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe45c06c200 0x7fe45c15d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:05.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.231+0000 7fe461600640 1 -- 172.21.15.27:0/1062144094 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe44c009420 con 0x7fe45c143da0 2024-01-15T18:17:05.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.231+0000 7fe461600640 1 --2- 172.21.15.27:0/1062144094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe45c143da0 0x7fe45c136d60 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fe4540b0040 tx=0x7fe4540196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:05.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.231+0000 7fe458c00640 1 -- 172.21.15.27:0/1062144094 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4540b9860 con 0x7fe45c143da0 2024-01-15T18:17:05.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.231+0000 7fe458c00640 1 -- 172.21.15.27:0/1062144094 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4540a58f0 con 0x7fe45c143da0 2024-01-15T18:17:05.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.231+0000 7fe458c00640 1 -- 172.21.15.27:0/1062144094 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4540b9c00 con 0x7fe45c143da0 2024-01-15T18:17:05.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.231+0000 7fe463600640 1 -- 172.21.15.27:0/1062144094 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe454009a70 con 0x7fe45c143da0 2024-01-15T18:17:05.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.231+0000 7fe463600640 1 -- 172.21.15.27:0/1062144094 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe45c15b960 con 0x7fe45c143da0 2024-01-15T18:17:05.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.231+0000 7fe463600640 1 -- 172.21.15.27:0/1062144094 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe45c15be80 con 0x7fe45c143da0 2024-01-15T18:17:05.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.235+0000 7fe458c00640 1 -- 172.21.15.27:0/1062144094 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7fe4540ad440 con 0x7fe45c143da0 2024-01-15T18:17:05.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.235+0000 7fe458c00640 1 --2- 172.21.15.27:0/1062144094 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe434067700 0x7fe434069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:05.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.235+0000 7fe458c00640 1 -- 172.21.15.27:0/1062144094 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5055+0+0 (secure 0 0 0) 0x7fe45411e110 con 0x7fe45c143da0 2024-01-15T18:17:05.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.235+0000 7fe460c00640 1 --2- 172.21.15.27:0/1062144094 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe434067700 0x7fe434069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:05.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.235+0000 7fe460c00640 1 --2- 172.21.15.27:0/1062144094 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe434067700 0x7fe434069bc0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fe45c159ac0 tx=0x7fe44c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:05.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.239+0000 7fe458c00640 1 -- 172.21.15.27:0/1062144094 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe4540eceb0 con 0x7fe45c143da0 2024-01-15T18:17:05.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.407+0000 7fe463600640 1 -- 172.21.15.27:0/1062144094 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe45c04e850 con 0x7fe45c143da0 2024-01-15T18:17:05.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.407+0000 7fe458c00640 1 -- 172.21.15.27:0/1062144094 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v89) v1 ==== 80+0+120 (secure 0 0 0) 0x7fe4540f04d0 con 0x7fe45c143da0 2024-01-15T18:17:05.406 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.93:6837/980414657 2024-01-16T18:17:00.989983+0000 2024-01-15T18:17:05.406 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.93:6835/980414657 2024-01-16T18:17:00.989983+0000 2024-01-15T18:17:05.406 INFO:teuthology.orchestra.run.smithi027.stderr:listed 2 entries 2024-01-15T18:17:05.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.407+0000 7fe463600640 1 -- 172.21.15.27:0/1062144094 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe434067700 msgr2=0x7fe434069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:05.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.407+0000 7fe463600640 1 --2- 172.21.15.27:0/1062144094 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe434067700 0x7fe434069bc0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fe45c159ac0 tx=0x7fe44c009190 comp rx=0 tx=0).stop 2024-01-15T18:17:05.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.407+0000 7fe463600640 1 -- 172.21.15.27:0/1062144094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe45c143da0 msgr2=0x7fe45c136d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:05.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.407+0000 7fe463600640 1 --2- 172.21.15.27:0/1062144094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe45c143da0 0x7fe45c136d60 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fe4540b0040 tx=0x7fe4540196f0 comp rx=0 tx=0).stop 2024-01-15T18:17:05.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.411+0000 7fe463600640 1 -- 172.21.15.27:0/1062144094 shutdown_connections 2024-01-15T18:17:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.411+0000 7fe463600640 1 --2- 172.21.15.27:0/1062144094 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe434067700 0x7fe434069bc0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.411+0000 7fe463600640 1 --2- 172.21.15.27:0/1062144094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe45c143da0 0x7fe45c136d60 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.411+0000 7fe463600640 1 --2- 172.21.15.27:0/1062144094 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe45c0cad40 0x7fe45c15d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.411+0000 7fe463600640 1 --2- 172.21.15.27:0/1062144094 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe45c06c200 0x7fe45c15d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.411+0000 7fe463600640 1 -- 172.21.15.27:0/1062144094 >> 172.21.15.27:0/1062144094 conn(0x7fe45c0b0850 msgr2=0x7fe45c13b320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.411+0000 7fe463600640 1 -- 172.21.15.27:0/1062144094 shutdown_connections 2024-01-15T18:17:05.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.411+0000 7fe463600640 1 -- 172.21.15.27:0/1062144094 wait complete. 2024-01-15T18:17:05.428 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-01-15T18:17:05.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:05.428 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-15T18:17:05.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[48488]: fuse finished with error 0 and tester_r 0 2024-01-15T18:17:05.449 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T18:17:05.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:05.449 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-15T18:17:05.635 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:17:05.635 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:17:05.645 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:17:05.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.827+0000 7ff9e8e00640 1 Processor -- start 2024-01-15T18:17:05.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.827+0000 7ff9e8e00640 1 -- start start 2024-01-15T18:17:05.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.827+0000 7ff9e8e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff9e40701a0 0x7ff9e4070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:05.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.827+0000 7ff9e8e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e406c4a0 con 0x7ff9e415c130 2024-01-15T18:17:05.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.827+0000 7ff9e8e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e4070ac0 con 0x7ff9e40c92b0 2024-01-15T18:17:05.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.827+0000 7ff9e8e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e415c660 con 0x7ff9e40701a0 2024-01-15T18:17:05.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.827+0000 7ff9e1e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff9e40701a0 0x7ff9e4070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:05.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.827+0000 7ff9e2800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff9e415c130 0x7ff9e415d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47030/0 (socket says 172.21.15.27:47030) 2024-01-15T18:17:05.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.827+0000 7ff9e1e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff9e40701a0 0x7ff9e4070580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48364/0 (socket says 172.21.15.27:48364) 2024-01-15T18:17:05.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.827+0000 7ff9e2800640 1 -- 172.21.15.27:0/4158801806 learned_addr learned my addr 172.21.15.27:0/4158801806 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:05.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.827+0000 7ff9e1e00640 1 -- 172.21.15.27:0/4158801806 >> v1:172.21.15.93:6789/0 conn(0x7ff9e40c92b0 legacy=0x7ff9e406dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:05.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.827+0000 7ff9e1e00640 1 -- 172.21.15.27:0/4158801806 >> v1:172.21.15.27:6789/0 conn(0x7ff9e415c130 legacy=0x7ff9e415d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:05.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.827+0000 7ff9e1e00640 1 -- 172.21.15.27:0/4158801806 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e415c940 con 0x7ff9e40701a0 2024-01-15T18:17:05.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.827+0000 7ff9e0a00640 1 -- 172.21.15.27:0/4158801806 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205491497 0 0) 0x7ff9e406c4a0 con 0x7ff9e415c130 2024-01-15T18:17:05.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.831+0000 7ff9e1e00640 1 --2- 172.21.15.27:0/4158801806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9e40701a0 0x7ff9e4070580 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40097c0 tx=0x7ff9d401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c494f1b00abe4702 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:05.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.831+0000 7ff9e0a00640 1 -- 172.21.15.27:0/4158801806 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d40ab650 con 0x7ff9e40701a0 2024-01-15T18:17:05.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.831+0000 7ff9e0a00640 1 -- 172.21.15.27:0/4158801806 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d40ab7b0 con 0x7ff9e40701a0 2024-01-15T18:17:05.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.831+0000 7ff9e0a00640 1 -- 172.21.15.27:0/4158801806 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d40abb40 con 0x7ff9e40701a0 2024-01-15T18:17:05.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.831+0000 7ff9e8e00640 1 -- 172.21.15.27:0/4158801806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9e40701a0 msgr2=0x7ff9e4070580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:05.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.831+0000 7ff9e8e00640 1 --2- 172.21.15.27:0/4158801806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9e40701a0 0x7ff9e4070580 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40097c0 tx=0x7ff9d401c450 comp rx=0 tx=0).stop 2024-01-15T18:17:05.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.831+0000 7ff9e8e00640 1 -- 172.21.15.27:0/4158801806 shutdown_connections 2024-01-15T18:17:05.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.831+0000 7ff9e8e00640 1 --2- 172.21.15.27:0/4158801806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9e40701a0 0x7ff9e4070580 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:05.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.831+0000 7ff9e8e00640 1 -- 172.21.15.27:0/4158801806 >> 172.21.15.27:0/4158801806 conn(0x7ff9e40cec60 msgr2=0x7ff9e40cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:05.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.831+0000 7ff9e8e00640 1 -- 172.21.15.27:0/4158801806 shutdown_connections 2024-01-15T18:17:05.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.831+0000 7ff9e8e00640 1 -- 172.21.15.27:0/4158801806 wait complete. 2024-01-15T18:17:05.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9e8e00640 1 Processor -- start 2024-01-15T18:17:05.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9e8e00640 1 -- start start 2024-01-15T18:17:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9e8e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9e40701a0 0x7ff9e40cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9e8e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff9e40c92b0 0x7ff9e4151690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9e8e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9e415c130 0x7ff9e40ae090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9e8e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e40ccf20 con 0x7ff9e415c130 2024-01-15T18:17:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9e8e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e40aed50 con 0x7ff9e40c92b0 2024-01-15T18:17:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9e8e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e40aea70 con 0x7ff9e40701a0 2024-01-15T18:17:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9e2800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9e415c130 0x7ff9e40ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9e2800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9e415c130 0x7ff9e40ae090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48378/0 (socket says 172.21.15.27:48378) 2024-01-15T18:17:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9e2800640 1 -- 172.21.15.27:0/1123291783 learned_addr learned my addr 172.21.15.27:0/1123291783 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:05.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9e1400640 1 --2- 172.21.15.27:0/1123291783 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff9e40c92b0 0x7ff9e4151690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9e1e00640 1 --2- 172.21.15.27:0/1123291783 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9e40701a0 0x7ff9e40cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9e2800640 1 -- 172.21.15.27:0/1123291783 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9e40701a0 msgr2=0x7ff9e40cd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9e2800640 1 --2- 172.21.15.27:0/1123291783 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9e40701a0 0x7ff9e40cd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9e2800640 1 -- 172.21.15.27:0/1123291783 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff9e40c92b0 msgr2=0x7ff9e4151690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9e2800640 1 --2- 172.21.15.27:0/1123291783 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff9e40c92b0 0x7ff9e4151690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9e2800640 1 -- 172.21.15.27:0/1123291783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d4009420 con 0x7ff9e415c130 2024-01-15T18:17:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9e1e00640 1 --2- 172.21.15.27:0/1123291783 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9e40701a0 0x7ff9e40cd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:17:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9e1400640 1 --2- 172.21.15.27:0/1123291783 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff9e40c92b0 0x7ff9e4151690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:17:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9e2800640 1 --2- 172.21.15.27:0/1123291783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9e415c130 0x7ff9e40ae090 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc00cac0 tx=0x7ff9dc00cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9da000640 1 -- 172.21.15.27:0/1123291783 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc00abd0 con 0x7ff9e415c130 2024-01-15T18:17:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9da000640 1 -- 172.21.15.27:0/1123291783 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9dc00ad30 con 0x7ff9e415c130 2024-01-15T18:17:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9da000640 1 -- 172.21.15.27:0/1123291783 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc012620 con 0x7ff9e415c130 2024-01-15T18:17:05.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9e8e00640 1 -- 172.21.15.27:0/1123291783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e40cc8d0 con 0x7ff9e415c130 2024-01-15T18:17:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.835+0000 7ff9e8e00640 1 -- 172.21.15.27:0/1123291783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e4151bd0 con 0x7ff9e415c130 2024-01-15T18:17:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.839+0000 7ff9e8e00640 1 -- 172.21.15.27:0/1123291783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e40cd090 con 0x7ff9e415c130 2024-01-15T18:17:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.839+0000 7ff9da000640 1 -- 172.21.15.27:0/1123291783 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7ff9dc007070 con 0x7ff9e415c130 2024-01-15T18:17:05.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.839+0000 7ff9da000640 1 --2- 172.21.15.27:0/1123291783 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff9bc067700 0x7ff9bc069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:05.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.839+0000 7ff9da000640 1 -- 172.21.15.27:0/1123291783 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5055+0+0 (secure 0 0 0) 0x7ff9dc083100 con 0x7ff9e415c130 2024-01-15T18:17:05.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.839+0000 7ff9e1e00640 1 --2- 172.21.15.27:0/1123291783 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff9bc067700 0x7ff9bc069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:05.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.839+0000 7ff9e1e00640 1 --2- 172.21.15.27:0/1123291783 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff9bc067700 0x7ff9bc069bc0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40af000 tx=0x7ff9d4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:05.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:05.843+0000 7ff9da000640 1 -- 172.21.15.27:0/1123291783 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff9dc051ad0 con 0x7ff9e415c130 2024-01-15T18:17:06.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.003+0000 7ff9e8e00640 1 -- 172.21.15.27:0/1123291783 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9e406e390 con 0x7ff9e415c130 2024-01-15T18:17:06.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.003+0000 7ff9da000640 1 -- 172.21.15.27:0/1123291783 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+13066 (secure 0 0 0) 0x7ff9dc017020 con 0x7ff9e415c130 2024-01-15T18:17:06.003 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:17:06.003 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:17:04.008565+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.93:6837/980414657":"2024-01-16T18:17:00.989983+0000","172.21.15.93:6835/980414657":"2024-01-16T18:17:00.989983+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:17:06.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.007+0000 7ff9e8e00640 1 -- 172.21.15.27:0/1123291783 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff9bc067700 msgr2=0x7ff9bc069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:06.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.007+0000 7ff9e8e00640 1 --2- 172.21.15.27:0/1123291783 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff9bc067700 0x7ff9bc069bc0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40af000 tx=0x7ff9d4008040 comp rx=0 tx=0).stop 2024-01-15T18:17:06.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.007+0000 7ff9e8e00640 1 -- 172.21.15.27:0/1123291783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9e415c130 msgr2=0x7ff9e40ae090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:06.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.007+0000 7ff9e8e00640 1 --2- 172.21.15.27:0/1123291783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9e415c130 0x7ff9e40ae090 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc00cac0 tx=0x7ff9dc00cf80 comp rx=0 tx=0).stop 2024-01-15T18:17:06.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.007+0000 7ff9e8e00640 1 -- 172.21.15.27:0/1123291783 shutdown_connections 2024-01-15T18:17:06.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.007+0000 7ff9e8e00640 1 --2- 172.21.15.27:0/1123291783 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff9bc067700 0x7ff9bc069bc0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:06.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.007+0000 7ff9e8e00640 1 --2- 172.21.15.27:0/1123291783 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9e415c130 0x7ff9e40ae090 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:06.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.007+0000 7ff9e8e00640 1 --2- 172.21.15.27:0/1123291783 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff9e40c92b0 0x7ff9e4151690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:06.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.007+0000 7ff9e8e00640 1 --2- 172.21.15.27:0/1123291783 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff9e40701a0 0x7ff9e40cd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:06.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.007+0000 7ff9e8e00640 1 -- 172.21.15.27:0/1123291783 >> 172.21.15.27:0/1123291783 conn(0x7ff9e40cec60 msgr2=0x7ff9e41564b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:06.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.007+0000 7ff9e8e00640 1 -- 172.21.15.27:0/1123291783 shutdown_connections 2024-01-15T18:17:06.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.007+0000 7ff9e8e00640 1 -- 172.21.15.27:0/1123291783 wait complete. 2024-01-15T18:17:06.025 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:17:06.025 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:17:06.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.171+0000 7f6570200640 1 Processor -- start 2024-01-15T18:17:06.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.171+0000 7f6570200640 1 -- start start 2024-01-15T18:17:06.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.171+0000 7f6570200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f656815f550 0x7f65680cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:06.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.171+0000 7f6570200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65680b1cb0 con 0x7f6568158900 2024-01-15T18:17:06.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.171+0000 7f6570200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f656815fa00 con 0x7f65680cf110 2024-01-15T18:17:06.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.171+0000 7f6570200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65680cf580 con 0x7f656815f550 2024-01-15T18:17:06.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.171+0000 7f656ce00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f656815f550 0x7f65680cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:06.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.171+0000 7f656d800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6568158900 0x7f656815bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47046/0 (socket says 172.21.15.27:47046) 2024-01-15T18:17:06.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.171+0000 7f656d800640 1 -- 172.21.15.27:0/273154741 learned_addr learned my addr 172.21.15.27:0/273154741 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:06.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.171+0000 7f656ce00640 1 -- 172.21.15.27:0/273154741 >> v1:172.21.15.93:6789/0 conn(0x7f65680cf110 legacy=0x7f65680701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:06.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.171+0000 7f656ce00640 1 -- 172.21.15.27:0/273154741 >> v1:172.21.15.27:6789/0 conn(0x7f6568158900 legacy=0x7f656815bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:06.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.171+0000 7f6567e00640 1 -- 172.21.15.27:0/273154741 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510729524 0 0) 0x7f65680b1cb0 con 0x7f6568158900 2024-01-15T18:17:06.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.171+0000 7f656ce00640 1 -- 172.21.15.27:0/273154741 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65680b1f00 con 0x7f656815f550 2024-01-15T18:17:06.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.175+0000 7f656ce00640 1 --2- 172.21.15.27:0/273154741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f656815f550 0x7f65680cebd0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f65500097f0 tx=0x7f655001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb632299e560a283 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:06.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.175+0000 7f6567e00640 1 -- 172.21.15.27:0/273154741 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65500ab610 con 0x7f656815f550 2024-01-15T18:17:06.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.175+0000 7f6567e00640 1 -- 172.21.15.27:0/273154741 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65500ab770 con 0x7f656815f550 2024-01-15T18:17:06.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.175+0000 7f6567e00640 1 -- 172.21.15.27:0/273154741 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65500abb00 con 0x7f656815f550 2024-01-15T18:17:06.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.175+0000 7f6570200640 1 -- 172.21.15.27:0/273154741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f656815f550 msgr2=0x7f65680cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:06.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.175+0000 7f6570200640 1 --2- 172.21.15.27:0/273154741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f656815f550 0x7f65680cebd0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f65500097f0 tx=0x7f655001c410 comp rx=0 tx=0).stop 2024-01-15T18:17:06.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.179+0000 7f6570200640 1 -- 172.21.15.27:0/273154741 shutdown_connections 2024-01-15T18:17:06.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.179+0000 7f6570200640 1 --2- 172.21.15.27:0/273154741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f656815f550 0x7f65680cebd0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:06.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.179+0000 7f6570200640 1 -- 172.21.15.27:0/273154741 >> 172.21.15.27:0/273154741 conn(0x7f65680cb700 msgr2=0x7f65680cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:06.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.179+0000 7f6570200640 1 -- 172.21.15.27:0/273154741 shutdown_connections 2024-01-15T18:17:06.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.179+0000 7f6570200640 1 -- 172.21.15.27:0/273154741 wait complete. 2024-01-15T18:17:06.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.179+0000 7f6570200640 1 Processor -- start 2024-01-15T18:17:06.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.179+0000 7f6570200640 1 -- start start 2024-01-15T18:17:06.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.179+0000 7f6570200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f65680cf110 0x7f65681458e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:06.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.179+0000 7f6570200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6568158900 0x7f6568152010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:06.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.179+0000 7f6570200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f656815f550 0x7f6568152550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:06.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.179+0000 7f6570200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6568152fa0 con 0x7f656815f550 2024-01-15T18:17:06.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.179+0000 7f6570200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6568152a90 con 0x7f6568158900 2024-01-15T18:17:06.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.179+0000 7f6570200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f656815ae50 con 0x7f65680cf110 2024-01-15T18:17:06.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.179+0000 7f656e200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f656815f550 0x7f6568152550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:06.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.179+0000 7f656e200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f656815f550 0x7f6568152550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48406/0 (socket says 172.21.15.27:48406) 2024-01-15T18:17:06.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.179+0000 7f656e200640 1 -- 172.21.15.27:0/653649447 learned_addr learned my addr 172.21.15.27:0/653649447 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:06.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.179+0000 7f656d800640 1 --2- 172.21.15.27:0/653649447 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f65680cf110 0x7f65681458e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:06.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.179+0000 7f656ce00640 1 --2- 172.21.15.27:0/653649447 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6568158900 0x7f6568152010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:06.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.179+0000 7f656e200640 1 -- 172.21.15.27:0/653649447 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f65680cf110 msgr2=0x7f65681458e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:06.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.179+0000 7f656e200640 1 --2- 172.21.15.27:0/653649447 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f65680cf110 0x7f65681458e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:06.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.179+0000 7f656e200640 1 -- 172.21.15.27:0/653649447 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6568158900 msgr2=0x7f6568152010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:06.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.179+0000 7f656e200640 1 --2- 172.21.15.27:0/653649447 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6568158900 0x7f6568152010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:06.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.179+0000 7f656e200640 1 -- 172.21.15.27:0/653649447 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6550009420 con 0x7f656815f550 2024-01-15T18:17:06.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.179+0000 7f656e200640 1 --2- 172.21.15.27:0/653649447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f656815f550 0x7f6568152550 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f656000c780 tx=0x7f656000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:06.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.183+0000 7f6565600640 1 -- 172.21.15.27:0/653649447 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6560004b70 con 0x7f656815f550 2024-01-15T18:17:06.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.183+0000 7f6565600640 1 -- 172.21.15.27:0/653649447 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656000ce90 con 0x7f656815f550 2024-01-15T18:17:06.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.183+0000 7f6565600640 1 -- 172.21.15.27:0/653649447 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656000a850 con 0x7f656815f550 2024-01-15T18:17:06.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.183+0000 7f6570200640 1 -- 172.21.15.27:0/653649447 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6568145e20 con 0x7f656815f550 2024-01-15T18:17:06.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.183+0000 7f6570200640 1 -- 172.21.15.27:0/653649447 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65680cdbb0 con 0x7f656815f550 2024-01-15T18:17:06.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.183+0000 7f6570200640 1 -- 172.21.15.27:0/653649447 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f656804e850 con 0x7f656815f550 2024-01-15T18:17:06.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.183+0000 7f6565600640 1 -- 172.21.15.27:0/653649447 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f656000a9b0 con 0x7f656815f550 2024-01-15T18:17:06.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.183+0000 7f6565600640 1 --2- 172.21.15.27:0/653649447 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6540067700 0x7f6540069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:06.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.183+0000 7f6565600640 1 -- 172.21.15.27:0/653649447 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f6560083310 con 0x7f656815f550 2024-01-15T18:17:06.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.183+0000 7f656d800640 1 --2- 172.21.15.27:0/653649447 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6540067700 0x7f6540069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:06.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.187+0000 7f656d800640 1 --2- 172.21.15.27:0/653649447 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6540067700 0x7f6540069bc0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f6558000f70 tx=0x7f6558007b80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:06.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.187+0000 7f6565600640 1 -- 172.21.15.27:0/653649447 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6560051ce0 con 0x7f656815f550 2024-01-15T18:17:06.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.347+0000 7f6570200640 1 -- 172.21.15.27:0/653649447 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f65680cd9a0 con 0x7f656815f550 2024-01-15T18:17:06.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.347+0000 7f6565600640 1 -- 172.21.15.27:0/653649447 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6560055300 con 0x7f656815f550 2024-01-15T18:17:06.346 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:17:06.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.351+0000 7f6570200640 1 -- 172.21.15.27:0/653649447 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6540067700 msgr2=0x7f6540069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:06.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.351+0000 7f6570200640 1 --2- 172.21.15.27:0/653649447 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6540067700 0x7f6540069bc0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f6558000f70 tx=0x7f6558007b80 comp rx=0 tx=0).stop 2024-01-15T18:17:06.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.351+0000 7f6570200640 1 -- 172.21.15.27:0/653649447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f656815f550 msgr2=0x7f6568152550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:06.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.351+0000 7f6570200640 1 --2- 172.21.15.27:0/653649447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f656815f550 0x7f6568152550 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f656000c780 tx=0x7f656000cc40 comp rx=0 tx=0).stop 2024-01-15T18:17:06.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.351+0000 7f6570200640 1 -- 172.21.15.27:0/653649447 shutdown_connections 2024-01-15T18:17:06.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.351+0000 7f6570200640 1 --2- 172.21.15.27:0/653649447 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6540067700 0x7f6540069bc0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:06.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.351+0000 7f6570200640 1 --2- 172.21.15.27:0/653649447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f656815f550 0x7f6568152550 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:06.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.351+0000 7f6570200640 1 --2- 172.21.15.27:0/653649447 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6568158900 0x7f6568152010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:06.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.351+0000 7f6570200640 1 --2- 172.21.15.27:0/653649447 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f65680cf110 0x7f65681458e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:06.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.351+0000 7f6570200640 1 -- 172.21.15.27:0/653649447 >> 172.21.15.27:0/653649447 conn(0x7f65680cb700 msgr2=0x7f656815df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:06.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.351+0000 7f6570200640 1 -- 172.21.15.27:0/653649447 shutdown_connections 2024-01-15T18:17:06.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.351+0000 7f6570200640 1 -- 172.21.15.27:0/653649447 wait complete. 2024-01-15T18:17:06.369 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:17:06.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.511+0000 7fa1f2e00640 1 Processor -- start 2024-01-15T18:17:06.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.511+0000 7fa1f2e00640 1 -- start start 2024-01-15T18:17:06.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.515+0000 7fa1f2e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa1ec0b4da0 0x7fa1ec0b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:06.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.515+0000 7fa1f2e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ec0d03a0 con 0x7fa1ec0b2190 2024-01-15T18:17:06.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.515+0000 7fa1f2e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ec0b2650 con 0x7fa1ec161390 2024-01-15T18:17:06.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.515+0000 7fa1f2e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec0cfb70 con 0x7fa1ec0b4da0 2024-01-15T18:17:06.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.515+0000 7fa1eb400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa1ec0b2190 0x7fa1ec0b0f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47060/0 (socket says 172.21.15.27:47060) 2024-01-15T18:17:06.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.515+0000 7fa1eb400640 1 -- 172.21.15.27:0/3597893120 learned_addr learned my addr 172.21.15.27:0/3597893120 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:06.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.515+0000 7fa1f0e00640 1 --2- 172.21.15.27:0/3597893120 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa1ec0b4da0 0x7fa1ec0b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:06.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.515+0000 7fa1eaa00640 1 -- 172.21.15.27:0/3597893120 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632266494 0 0) 0x7fa1ec0d03a0 con 0x7fa1ec0b2190 2024-01-15T18:17:06.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.515+0000 7fa1eaa00640 1 -- 172.21.15.27:0/3597893120 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1cc003530 con 0x7fa1ec0b2190 2024-01-15T18:17:06.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.515+0000 7fa1eaa00640 1 -- 172.21.15.27:0/3597893120 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209262600 0 0) 0x7fa1ec0b2650 con 0x7fa1ec161390 2024-01-15T18:17:06.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.515+0000 7fa1eaa00640 1 -- 172.21.15.27:0/3597893120 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1ec0d03a0 con 0x7fa1ec161390 2024-01-15T18:17:06.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.515+0000 7fa1f0e00640 1 -- 172.21.15.27:0/3597893120 >> v1:172.21.15.93:6789/0 conn(0x7fa1ec161390 legacy=0x7fa1ec161770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:06.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.515+0000 7fa1f0e00640 1 -- 172.21.15.27:0/3597893120 >> v1:172.21.15.27:6789/0 conn(0x7fa1ec0b2190 legacy=0x7fa1ec0b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:06.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.515+0000 7fa1eaa00640 1 -- 172.21.15.27:0/3597893120 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3706136197 0 0) 0x7fa1cc003530 con 0x7fa1ec0b2190 2024-01-15T18:17:06.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.515+0000 7fa1f0e00640 1 -- 172.21.15.27:0/3597893120 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ec0cfd50 con 0x7fa1ec0b4da0 2024-01-15T18:17:06.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.519+0000 7fa1f0e00640 1 --2- 172.21.15.27:0/3597893120 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa1ec0b4da0 0x7fa1ec0b1660 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009dc0 tx=0x7fa1e401c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a754f8a8d9f371a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:06.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.519+0000 7fa1eaa00640 1 -- 172.21.15.27:0/3597893120 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d4001680 con 0x7fa1ec0b4da0 2024-01-15T18:17:06.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.519+0000 7fa1eaa00640 1 -- 172.21.15.27:0/3597893120 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e40a5630 con 0x7fa1ec0b4da0 2024-01-15T18:17:06.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.519+0000 7fa1eaa00640 1 -- 172.21.15.27:0/3597893120 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e40a59c0 con 0x7fa1ec0b4da0 2024-01-15T18:17:06.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.519+0000 7fa1f2e00640 1 -- 172.21.15.27:0/3597893120 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa1ec0b4da0 msgr2=0x7fa1ec0b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:06.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.519+0000 7fa1f2e00640 1 --2- 172.21.15.27:0/3597893120 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa1ec0b4da0 0x7fa1ec0b1660 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009dc0 tx=0x7fa1e401c7d0 comp rx=0 tx=0).stop 2024-01-15T18:17:06.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.519+0000 7fa1f2e00640 1 -- 172.21.15.27:0/3597893120 shutdown_connections 2024-01-15T18:17:06.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.519+0000 7fa1f2e00640 1 --2- 172.21.15.27:0/3597893120 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa1ec0b4da0 0x7fa1ec0b1660 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:06.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.519+0000 7fa1f2e00640 1 -- 172.21.15.27:0/3597893120 >> 172.21.15.27:0/3597893120 conn(0x7fa1ec0d0f40 msgr2=0x7fa1ec0d1350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:06.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.519+0000 7fa1f2e00640 1 -- 172.21.15.27:0/3597893120 shutdown_connections 2024-01-15T18:17:06.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.519+0000 7fa1f2e00640 1 -- 172.21.15.27:0/3597893120 wait complete. 2024-01-15T18:17:06.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.523+0000 7fa1f2e00640 1 Processor -- start 2024-01-15T18:17:06.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.523+0000 7fa1f2e00640 1 -- start start 2024-01-15T18:17:06.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.523+0000 7fa1f2e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa1ec0b2190 0x7fa1ec0aea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:06.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.523+0000 7fa1f2e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1ec0b4da0 0x7fa1ec0b05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:06.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.523+0000 7fa1f2e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1ec161390 0x7fa1ec153810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:06.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.523+0000 7fa1f2e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec0b0b10 con 0x7fa1ec161390 2024-01-15T18:17:06.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.523+0000 7fa1f2e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec1543a0 con 0x7fa1ec0b4da0 2024-01-15T18:17:06.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.523+0000 7fa1f2e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec15bba0 con 0x7fa1ec0b2190 2024-01-15T18:17:06.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.523+0000 7fa1f0e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1ec161390 0x7fa1ec153810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:06.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.523+0000 7fa1f0e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1ec161390 0x7fa1ec153810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48420/0 (socket says 172.21.15.27:48420) 2024-01-15T18:17:06.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.523+0000 7fa1f0e00640 1 -- 172.21.15.27:0/3410008449 learned_addr learned my addr 172.21.15.27:0/3410008449 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:06.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.523+0000 7fa1eb400640 1 --2- 172.21.15.27:0/3410008449 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1ec0b4da0 0x7fa1ec0b05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:06.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.523+0000 7fa1ebe00640 1 --2- 172.21.15.27:0/3410008449 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa1ec0b2190 0x7fa1ec0aea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:06.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.523+0000 7fa1f0e00640 1 -- 172.21.15.27:0/3410008449 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa1ec0b2190 msgr2=0x7fa1ec0aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:06.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.523+0000 7fa1f0e00640 1 --2- 172.21.15.27:0/3410008449 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa1ec0b2190 0x7fa1ec0aea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:06.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.523+0000 7fa1f0e00640 1 -- 172.21.15.27:0/3410008449 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1ec0b4da0 msgr2=0x7fa1ec0b05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:06.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.523+0000 7fa1f0e00640 1 --2- 172.21.15.27:0/3410008449 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1ec0b4da0 0x7fa1ec0b05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:06.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.523+0000 7fa1f0e00640 1 -- 172.21.15.27:0/3410008449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e4009a70 con 0x7fa1ec161390 2024-01-15T18:17:06.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.523+0000 7fa1f0e00640 1 --2- 172.21.15.27:0/3410008449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1ec161390 0x7fa1ec153810 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fa1e40b0040 tx=0x7fa1e4019570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:06.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.523+0000 7fa1e3e00640 1 -- 172.21.15.27:0/3410008449 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e40b9860 con 0x7fa1ec161390 2024-01-15T18:17:06.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.523+0000 7fa1e3e00640 1 -- 172.21.15.27:0/3410008449 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e40b99c0 con 0x7fa1ec161390 2024-01-15T18:17:06.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.523+0000 7fa1f2e00640 1 -- 172.21.15.27:0/3410008449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ec15be80 con 0x7fa1ec161390 2024-01-15T18:17:06.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.523+0000 7fa1e3e00640 1 -- 172.21.15.27:0/3410008449 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e40b9d50 con 0x7fa1ec161390 2024-01-15T18:17:06.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.523+0000 7fa1f2e00640 1 -- 172.21.15.27:0/3410008449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ec15c440 con 0x7fa1ec161390 2024-01-15T18:17:06.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.527+0000 7fa1f2e00640 1 -- 172.21.15.27:0/3410008449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ec04e850 con 0x7fa1ec161390 2024-01-15T18:17:06.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.527+0000 7fa1e3e00640 1 -- 172.21.15.27:0/3410008449 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7fa1e40ad440 con 0x7fa1ec161390 2024-01-15T18:17:06.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.527+0000 7fa1e3e00640 1 --2- 172.21.15.27:0/3410008449 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa1c4067700 0x7fa1c4069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:06.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.527+0000 7fa1e3e00640 1 -- 172.21.15.27:0/3410008449 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5055+0+0 (secure 0 0 0) 0x7fa1e411dfb0 con 0x7fa1ec161390 2024-01-15T18:17:06.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.527+0000 7fa1ebe00640 1 --2- 172.21.15.27:0/3410008449 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa1c4067700 0x7fa1c4069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:06.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.527+0000 7fa1ebe00640 1 --2- 172.21.15.27:0/3410008449 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa1c4067700 0x7fa1c4069bc0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc0011f0 tx=0x7fa1dc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:06.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.531+0000 7fa1e3e00640 1 -- 172.21.15.27:0/3410008449 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa1e40eccf0 con 0x7fa1ec161390 2024-01-15T18:17:06.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.699+0000 7fa1f2e00640 1 -- 172.21.15.27:0/3410008449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa1ec0b6ec0 con 0x7fa1ec161390 2024-01-15T18:17:06.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.699+0000 7fa1e3e00640 1 -- 172.21.15.27:0/3410008449 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v89) v1 ==== 80+0+120 (secure 0 0 0) 0x7fa1e40bd090 con 0x7fa1ec161390 2024-01-15T18:17:06.698 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.93:6837/980414657 2024-01-16T18:17:00.989983+0000 2024-01-15T18:17:06.698 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.93:6835/980414657 2024-01-16T18:17:00.989983+0000 2024-01-15T18:17:06.698 INFO:teuthology.orchestra.run.smithi027.stderr:listed 2 entries 2024-01-15T18:17:06.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.699+0000 7fa1f2e00640 1 -- 172.21.15.27:0/3410008449 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa1c4067700 msgr2=0x7fa1c4069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:06.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.699+0000 7fa1f2e00640 1 --2- 172.21.15.27:0/3410008449 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa1c4067700 0x7fa1c4069bc0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc0011f0 tx=0x7fa1dc009040 comp rx=0 tx=0).stop 2024-01-15T18:17:06.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.699+0000 7fa1f2e00640 1 -- 172.21.15.27:0/3410008449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1ec161390 msgr2=0x7fa1ec153810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:06.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.699+0000 7fa1f2e00640 1 --2- 172.21.15.27:0/3410008449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1ec161390 0x7fa1ec153810 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fa1e40b0040 tx=0x7fa1e4019570 comp rx=0 tx=0).stop 2024-01-15T18:17:06.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.703+0000 7fa1f2e00640 1 -- 172.21.15.27:0/3410008449 shutdown_connections 2024-01-15T18:17:06.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.703+0000 7fa1f2e00640 1 --2- 172.21.15.27:0/3410008449 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa1c4067700 0x7fa1c4069bc0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:06.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.703+0000 7fa1f2e00640 1 --2- 172.21.15.27:0/3410008449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1ec161390 0x7fa1ec153810 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:06.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.703+0000 7fa1f2e00640 1 --2- 172.21.15.27:0/3410008449 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa1ec0b4da0 0x7fa1ec0b05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:06.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.703+0000 7fa1f2e00640 1 --2- 172.21.15.27:0/3410008449 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa1ec0b2190 0x7fa1ec0aea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:06.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.703+0000 7fa1f2e00640 1 -- 172.21.15.27:0/3410008449 >> 172.21.15.27:0/3410008449 conn(0x7fa1ec0d0f40 msgr2=0x7fa1ec06e7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:06.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.703+0000 7fa1f2e00640 1 -- 172.21.15.27:0/3410008449 shutdown_connections 2024-01-15T18:17:06.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:06.703+0000 7fa1f2e00640 1 -- 172.21.15.27:0/3410008449 wait complete. 2024-01-15T18:17:06.721 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi093.front.sepia.ceph.com... 2024-01-15T18:17:06.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:06.721 DEBUG:teuthology.orchestra.run.smithi093:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-15T18:17:06.741 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:ceph-fuse[34346]: fuse finished with error 0 and tester_r 0 2024-01-15T18:17:06.744 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T18:17:06.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:06.744 DEBUG:teuthology.orchestra.run.smithi093:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-15T18:17:06.918 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:17:06.918 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:17:06.927 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354580k,nr_inodes=4088645,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9249 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:17:06.927 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T18:17:06.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:06.927 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-15T18:17:06.979 INFO:teuthology.orchestra.run.smithi027.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-15T18:17:06.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:17:06.980 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:17:06.980 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:17:06.989 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354484k,nr_inodes=4088621,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8403 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:17:06.989 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T18:17:06.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:06.990 DEBUG:teuthology.orchestra.run.smithi093:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-15T18:17:07.041 INFO:teuthology.orchestra.run.smithi093.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-15T18:17:07.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:17:07.042 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:17:07.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.183+0000 7fe01e400640 1 Processor -- start 2024-01-15T18:17:07.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.183+0000 7fe01e400640 1 -- start start 2024-01-15T18:17:07.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.187+0000 7fe01e400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe0180afe70 0x7fe0180cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:07.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.187+0000 7fe01e400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe0180cb7b0 0x7fe0180cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:07.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.187+0000 7fe01e400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0180d05b0 con 0x7fe018160b10 2024-01-15T18:17:07.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.187+0000 7fe01e400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0180ce7d0 con 0x7fe0180cb7b0 2024-01-15T18:17:07.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.187+0000 7fe01e400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0180cfc80 con 0x7fe0180afe70 2024-01-15T18:17:07.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.187+0000 7fe017e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe0180cb7b0 0x7fe0180cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:07.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.187+0000 7fe017400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe018160b10 0x7fe0180cd7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47066/0 (socket says 172.21.15.27:47066) 2024-01-15T18:17:07.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.187+0000 7fe017e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe0180cb7b0 0x7fe0180cbc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48430/0 (socket says 172.21.15.27:48430) 2024-01-15T18:17:07.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.187+0000 7fe017400640 1 -- 172.21.15.27:0/43451132 learned_addr learned my addr 172.21.15.27:0/43451132 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:07.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.187+0000 7fe016a00640 1 --2- 172.21.15.27:0/43451132 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe0180afe70 0x7fe0180cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:07.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.187+0000 7fe017e00640 1 -- 172.21.15.27:0/43451132 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe0180afe70 msgr2=0x7fe0180cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:07.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.187+0000 7fe017e00640 1 --2- 172.21.15.27:0/43451132 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe0180afe70 0x7fe0180cdeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:07.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.187+0000 7fe017e00640 1 -- 172.21.15.27:0/43451132 >> v1:172.21.15.27:6789/0 conn(0x7fe018160b10 legacy=0x7fe0180cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:07.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.187+0000 7fe017e00640 1 -- 172.21.15.27:0/43451132 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0180cff60 con 0x7fe0180cb7b0 2024-01-15T18:17:07.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.187+0000 7fe016a00640 1 --2- 172.21.15.27:0/43451132 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe0180afe70 0x7fe0180cdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:17:07.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.187+0000 7fe017e00640 1 --2- 172.21.15.27:0/43451132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0180cb7b0 0x7fe0180cbc40 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fe010009d90 tx=0x7fe01001c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6852f7399c7f9660 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:07.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.191+0000 7fe016000640 1 -- 172.21.15.27:0/43451132 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0100a5860 con 0x7fe0180cb7b0 2024-01-15T18:17:07.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.191+0000 7fe016000640 1 -- 172.21.15.27:0/43451132 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0100a59c0 con 0x7fe0180cb7b0 2024-01-15T18:17:07.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.191+0000 7fe016000640 1 -- 172.21.15.27:0/43451132 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0100a5d50 con 0x7fe0180cb7b0 2024-01-15T18:17:07.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.191+0000 7fe01e400640 1 -- 172.21.15.27:0/43451132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0180cb7b0 msgr2=0x7fe0180cbc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:07.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.191+0000 7fe01e400640 1 --2- 172.21.15.27:0/43451132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0180cb7b0 0x7fe0180cbc40 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fe010009d90 tx=0x7fe01001c890 comp rx=0 tx=0).stop 2024-01-15T18:17:07.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.191+0000 7fe01e400640 1 -- 172.21.15.27:0/43451132 shutdown_connections 2024-01-15T18:17:07.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.191+0000 7fe01e400640 1 --2- 172.21.15.27:0/43451132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0180cb7b0 0x7fe0180cbc40 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:07.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.191+0000 7fe01e400640 1 --2- 172.21.15.27:0/43451132 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe0180afe70 0x7fe0180cdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:07.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.191+0000 7fe01e400640 1 -- 172.21.15.27:0/43451132 >> 172.21.15.27:0/43451132 conn(0x7fe0180d0f20 msgr2=0x7fe0180d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:07.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.191+0000 7fe01e400640 1 -- 172.21.15.27:0/43451132 shutdown_connections 2024-01-15T18:17:07.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.191+0000 7fe01e400640 1 -- 172.21.15.27:0/43451132 wait complete. 2024-01-15T18:17:07.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.191+0000 7fe01e400640 1 Processor -- start 2024-01-15T18:17:07.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.191+0000 7fe01e400640 1 -- start start 2024-01-15T18:17:07.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.191+0000 7fe01e400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0180afe70 0x7fe018159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:07.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.191+0000 7fe01e400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0180cb7b0 0x7fe018156050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:07.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.191+0000 7fe01e400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe018160b10 0x7fe0180c76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:07.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.191+0000 7fe01e400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01809c230 con 0x7fe0180afe70 2024-01-15T18:17:07.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.191+0000 7fe01e400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe018156590 con 0x7fe0180cb7b0 2024-01-15T18:17:07.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.191+0000 7fe01e400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe018159e30 con 0x7fe018160b10 2024-01-15T18:17:07.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.195+0000 7fe017400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0180afe70 0x7fe018159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:07.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.195+0000 7fe017400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0180afe70 0x7fe018159790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48438/0 (socket says 172.21.15.27:48438) 2024-01-15T18:17:07.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.195+0000 7fe017400640 1 -- 172.21.15.27:0/2075068204 learned_addr learned my addr 172.21.15.27:0/2075068204 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:07.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.195+0000 7fe017e00640 1 --2- 172.21.15.27:0/2075068204 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe018160b10 0x7fe0180c76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:07.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.195+0000 7fe016a00640 1 --2- 172.21.15.27:0/2075068204 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0180cb7b0 0x7fe018156050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:07.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.195+0000 7fe017400640 1 -- 172.21.15.27:0/2075068204 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe018160b10 msgr2=0x7fe0180c76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:07.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.195+0000 7fe017400640 1 --2- 172.21.15.27:0/2075068204 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe018160b10 0x7fe0180c76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:07.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.195+0000 7fe017400640 1 -- 172.21.15.27:0/2075068204 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0180cb7b0 msgr2=0x7fe018156050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:07.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.195+0000 7fe017400640 1 --2- 172.21.15.27:0/2075068204 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0180cb7b0 0x7fe018156050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:07.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.195+0000 7fe017400640 1 -- 172.21.15.27:0/2075068204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe010009a70 con 0x7fe0180afe70 2024-01-15T18:17:07.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.195+0000 7fe016a00640 1 --2- 172.21.15.27:0/2075068204 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0180cb7b0 0x7fe018156050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:17:07.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.195+0000 7fe017400640 1 --2- 172.21.15.27:0/2075068204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0180afe70 0x7fe018159790 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fe00800c500 tx=0x7fe00800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:07.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.195+0000 7fe00f400640 1 -- 172.21.15.27:0/2075068204 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe008005cf0 con 0x7fe0180afe70 2024-01-15T18:17:07.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.195+0000 7fe00f400640 1 -- 172.21.15.27:0/2075068204 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe008005e50 con 0x7fe0180afe70 2024-01-15T18:17:07.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.195+0000 7fe01e400640 1 -- 172.21.15.27:0/2075068204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe01815a110 con 0x7fe0180afe70 2024-01-15T18:17:07.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.195+0000 7fe00f400640 1 -- 172.21.15.27:0/2075068204 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0080072f0 con 0x7fe0180afe70 2024-01-15T18:17:07.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.195+0000 7fe01e400640 1 -- 172.21.15.27:0/2075068204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe01815a6d0 con 0x7fe0180afe70 2024-01-15T18:17:07.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.195+0000 7fe01e400640 1 -- 172.21.15.27:0/2075068204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe01809c3a0 con 0x7fe0180afe70 2024-01-15T18:17:07.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.195+0000 7fe00f400640 1 -- 172.21.15.27:0/2075068204 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7fe008007450 con 0x7fe0180afe70 2024-01-15T18:17:07.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.199+0000 7fe00f400640 1 --2- 172.21.15.27:0/2075068204 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdff0067730 0x7fdff0069bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:07.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.199+0000 7fe00f400640 1 -- 172.21.15.27:0/2075068204 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5055+0+0 (secure 0 0 0) 0x7fe008082830 con 0x7fe0180afe70 2024-01-15T18:17:07.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.199+0000 7fe016a00640 1 --2- 172.21.15.27:0/2075068204 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdff0067730 0x7fdff0069bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:07.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.199+0000 7fe016a00640 1 --2- 172.21.15.27:0/2075068204 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdff0067730 0x7fdff0069bf0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fe01815b130 tx=0x7fe000008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:07.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.199+0000 7fe00f400640 1 -- 172.21.15.27:0/2075068204 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe01809c3a0 con 0x7fe0180afe70 2024-01-15T18:17:07.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.395+0000 7fe01e400640 1 -- 172.21.15.27:0/2075068204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0180702c0 con 0x7fe0180afe70 2024-01-15T18:17:07.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.395+0000 7fe00f400640 1 -- 172.21.15.27:0/2075068204 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+3699 (secure 0 0 0) 0x7fe008051130 con 0x7fe0180afe70 2024-01-15T18:17:07.395 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:17:07.395 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":60,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8167,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/3555344514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":3555344514},{"type":"v1","addr":"172.21.15.93:6837","nonce":3555344514}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":8376,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/2584136338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2584136338},{"type":"v1","addr":"172.21.15.27:6833","nonce":2584136338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":8385,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2635542478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2635542478},{"type":"v1","addr":"172.21.15.27:6835","nonce":2635542478}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52},{"gid":8388,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/2822170069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":2822170069},{"type":"v1","addr":"172.21.15.93:6836","nonce":2822170069}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52}],"filesystems":[]} 2024-01-15T18:17:07.396 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 60 2024-01-15T18:17:07.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.399+0000 7fe01e400640 1 -- 172.21.15.27:0/2075068204 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdff0067730 msgr2=0x7fdff0069bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:07.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.399+0000 7fe01e400640 1 --2- 172.21.15.27:0/2075068204 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdff0067730 0x7fdff0069bf0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fe01815b130 tx=0x7fe000008040 comp rx=0 tx=0).stop 2024-01-15T18:17:07.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.399+0000 7fe01e400640 1 -- 172.21.15.27:0/2075068204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0180afe70 msgr2=0x7fe018159790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:07.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.399+0000 7fe01e400640 1 --2- 172.21.15.27:0/2075068204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0180afe70 0x7fe018159790 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fe00800c500 tx=0x7fe00800c9c0 comp rx=0 tx=0).stop 2024-01-15T18:17:07.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.399+0000 7fe01e400640 1 -- 172.21.15.27:0/2075068204 shutdown_connections 2024-01-15T18:17:07.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.399+0000 7fe01e400640 1 --2- 172.21.15.27:0/2075068204 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdff0067730 0x7fdff0069bf0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:07.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.399+0000 7fe01e400640 1 --2- 172.21.15.27:0/2075068204 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe018160b10 0x7fe0180c76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:07.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.399+0000 7fe01e400640 1 --2- 172.21.15.27:0/2075068204 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe0180cb7b0 0x7fe018156050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:07.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.399+0000 7fe01e400640 1 --2- 172.21.15.27:0/2075068204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe0180afe70 0x7fe018159790 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:07.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.399+0000 7fe01e400640 1 -- 172.21.15.27:0/2075068204 >> 172.21.15.27:0/2075068204 conn(0x7fe0180d0f20 msgr2=0x7fe01806fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:07.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.399+0000 7fe01e400640 1 -- 172.21.15.27:0/2075068204 shutdown_connections 2024-01-15T18:17:07.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.399+0000 7fe01e400640 1 -- 172.21.15.27:0/2075068204 wait complete. 2024-01-15T18:17:07.418 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-15T18:17:07.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.563+0000 7f89ee000640 1 Processor -- start 2024-01-15T18:17:07.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.563+0000 7f89ee000640 1 -- start start 2024-01-15T18:17:07.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.563+0000 7f89ee000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f89e80ce760 0x7f89e80b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:07.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.563+0000 7f89ee000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e80712b0 con 0x7f89e80b1db0 2024-01-15T18:17:07.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.563+0000 7f89ee000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89e80ccfb0 con 0x7f89e806c430 2024-01-15T18:17:07.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.563+0000 7f89ee000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89e80cd290 con 0x7f89e80ce760 2024-01-15T18:17:07.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.563+0000 7f89e7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f89e80ce760 0x7f89e80b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:07.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.563+0000 7f89e6a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f89e80b1db0 0x7f89e80ce050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47078/0 (socket says 172.21.15.27:47078) 2024-01-15T18:17:07.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.563+0000 7f89e6a00640 1 -- 172.21.15.27:0/65649561 learned_addr learned my addr 172.21.15.27:0/65649561 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:07.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.563+0000 7f89e7e00640 1 -- 172.21.15.27:0/65649561 >> v1:172.21.15.93:6789/0 conn(0x7f89e806c430 legacy=0x7f89e8070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:07.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.563+0000 7f89e7e00640 1 -- 172.21.15.27:0/65649561 >> v1:172.21.15.27:6789/0 conn(0x7f89e80b1db0 legacy=0x7f89e80ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:07.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.563+0000 7f89e7e00640 1 -- 172.21.15.27:0/65649561 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e80cd570 con 0x7f89e80ce760 2024-01-15T18:17:07.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.567+0000 7f89e7e00640 1 --2- 172.21.15.27:0/65649561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89e80ce760 0x7f89e80b2240 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f89e0009dc0 tx=0x7f89e001d980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8b689fb6d62f392 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:07.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.567+0000 7f89e6000640 1 -- 172.21.15.27:0/65649561 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e00a69c0 con 0x7f89e80ce760 2024-01-15T18:17:07.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.567+0000 7f89e6000640 1 -- 172.21.15.27:0/65649561 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e00a6b20 con 0x7f89e80ce760 2024-01-15T18:17:07.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.567+0000 7f89e6000640 1 -- 172.21.15.27:0/65649561 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e00ae440 con 0x7f89e80ce760 2024-01-15T18:17:07.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.567+0000 7f89ee000640 1 -- 172.21.15.27:0/65649561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89e80ce760 msgr2=0x7f89e80b2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:07.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.567+0000 7f89ee000640 1 --2- 172.21.15.27:0/65649561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89e80ce760 0x7f89e80b2240 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f89e0009dc0 tx=0x7f89e001d980 comp rx=0 tx=0).stop 2024-01-15T18:17:07.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.567+0000 7f89ee000640 1 -- 172.21.15.27:0/65649561 shutdown_connections 2024-01-15T18:17:07.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.567+0000 7f89ee000640 1 --2- 172.21.15.27:0/65649561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89e80ce760 0x7f89e80b2240 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:07.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.567+0000 7f89ee000640 1 -- 172.21.15.27:0/65649561 >> 172.21.15.27:0/65649561 conn(0x7f89e80cb700 msgr2=0x7f89e80cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:07.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.567+0000 7f89ee000640 1 -- 172.21.15.27:0/65649561 shutdown_connections 2024-01-15T18:17:07.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.567+0000 7f89ee000640 1 -- 172.21.15.27:0/65649561 wait complete. 2024-01-15T18:17:07.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89ee000640 1 Processor -- start 2024-01-15T18:17:07.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89ee000640 1 -- start start 2024-01-15T18:17:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89ee000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f89e806c430 0x7f89e8140440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89ee000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89e80b1db0 0x7f89e8148260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89ee000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89e80ce760 0x7f89e815a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89ee000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e814ba40 con 0x7f89e80ce760 2024-01-15T18:17:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89ee000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e8140980 con 0x7f89e806c430 2024-01-15T18:17:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89ee000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89e8148b60 con 0x7f89e80b1db0 2024-01-15T18:17:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89e7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89e80ce760 0x7f89e815a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89e7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89e80ce760 0x7f89e815a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48464/0 (socket says 172.21.15.27:48464) 2024-01-15T18:17:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89e7e00640 1 -- 172.21.15.27:0/1913235260 learned_addr learned my addr 172.21.15.27:0/1913235260 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89e6a00640 1 --2- 172.21.15.27:0/1913235260 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89e80b1db0 0x7f89e8148260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89e7e00640 1 -- 172.21.15.27:0/1913235260 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89e80b1db0 msgr2=0x7f89e8148260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89e7400640 1 --2- 172.21.15.27:0/1913235260 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f89e806c430 0x7f89e8140440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89e7e00640 1 --2- 172.21.15.27:0/1913235260 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89e80b1db0 0x7f89e8148260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89e7e00640 1 -- 172.21.15.27:0/1913235260 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f89e806c430 msgr2=0x7f89e8140440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:07.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89e7e00640 1 --2- 172.21.15.27:0/1913235260 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f89e806c430 0x7f89e8140440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89e7e00640 1 -- 172.21.15.27:0/1913235260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89e0009a70 con 0x7f89e80ce760 2024-01-15T18:17:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89e6a00640 1 --2- 172.21.15.27:0/1913235260 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89e80b1db0 0x7f89e8148260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:17:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89e7400640 1 --2- 172.21.15.27:0/1913235260 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f89e806c430 0x7f89e8140440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:17:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89e7e00640 1 --2- 172.21.15.27:0/1913235260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89e80ce760 0x7f89e815a530 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f89e001b5b0 tx=0x7f89e001c1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89df400640 1 -- 172.21.15.27:0/1913235260 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e00aedc0 con 0x7f89e80ce760 2024-01-15T18:17:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89df400640 1 -- 172.21.15.27:0/1913235260 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e00b9470 con 0x7f89e80ce760 2024-01-15T18:17:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89df400640 1 -- 172.21.15.27:0/1913235260 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e00b97e0 con 0x7f89e80ce760 2024-01-15T18:17:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89ee000640 1 -- 172.21.15.27:0/1913235260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e8148e40 con 0x7f89e80ce760 2024-01-15T18:17:07.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89ee000640 1 -- 172.21.15.27:0/1913235260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e814b660 con 0x7f89e80ce760 2024-01-15T18:17:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.571+0000 7f89ee000640 1 -- 172.21.15.27:0/1913235260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89e814bb80 con 0x7f89e80ce760 2024-01-15T18:17:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.575+0000 7f89df400640 1 -- 172.21.15.27:0/1913235260 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f89e00b5070 con 0x7f89e80ce760 2024-01-15T18:17:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.575+0000 7f89df400640 1 --2- 172.21.15.27:0/1913235260 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f89c0067730 0x7f89c0069bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:07.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.575+0000 7f89df400640 1 -- 172.21.15.27:0/1913235260 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f89e011e520 con 0x7f89e80ce760 2024-01-15T18:17:07.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.575+0000 7f89e7400640 1 --2- 172.21.15.27:0/1913235260 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f89c0067730 0x7f89c0069bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:07.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.575+0000 7f89e7400640 1 --2- 172.21.15.27:0/1913235260 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f89c0067730 0x7f89c0069bf0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f89d8000f70 tx=0x7f89d8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:07.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.579+0000 7f89df400640 1 -- 172.21.15.27:0/1913235260 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f89e814bb80 con 0x7f89e80ce760 2024-01-15T18:17:07.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.735+0000 7f89ee000640 1 -- 172.21.15.27:0/1913235260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7f89e814bb80 con 0x7f89e80ce760 2024-01-15T18:17:07.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.739+0000 7f89df400640 1 -- 172.21.15.27:0/1913235260 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v32) v1 ==== 120+0+0 (secure 0 0 0) 0x7f89e814bb80 con 0x7f89e80ce760 2024-01-15T18:17:07.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.743+0000 7f89ee000640 1 -- 172.21.15.27:0/1913235260 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f89c0067730 msgr2=0x7f89c0069bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:07.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.743+0000 7f89ee000640 1 --2- 172.21.15.27:0/1913235260 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f89c0067730 0x7f89c0069bf0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f89d8000f70 tx=0x7f89d8009040 comp rx=0 tx=0).stop 2024-01-15T18:17:07.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.743+0000 7f89ee000640 1 -- 172.21.15.27:0/1913235260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89e80ce760 msgr2=0x7f89e815a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:07.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.743+0000 7f89ee000640 1 --2- 172.21.15.27:0/1913235260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89e80ce760 0x7f89e815a530 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f89e001b5b0 tx=0x7f89e001c1c0 comp rx=0 tx=0).stop 2024-01-15T18:17:07.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.743+0000 7f89ee000640 1 -- 172.21.15.27:0/1913235260 shutdown_connections 2024-01-15T18:17:07.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.743+0000 7f89ee000640 1 --2- 172.21.15.27:0/1913235260 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f89c0067730 0x7f89c0069bf0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:07.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.743+0000 7f89ee000640 1 --2- 172.21.15.27:0/1913235260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f89e80ce760 0x7f89e815a530 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:07.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.743+0000 7f89ee000640 1 --2- 172.21.15.27:0/1913235260 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89e80b1db0 0x7f89e8148260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:07.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.743+0000 7f89ee000640 1 --2- 172.21.15.27:0/1913235260 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f89e806c430 0x7f89e8140440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:07.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.743+0000 7f89ee000640 1 -- 172.21.15.27:0/1913235260 >> 172.21.15.27:0/1913235260 conn(0x7f89e80cb700 msgr2=0x7f89e80b3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:07.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.743+0000 7f89ee000640 1 -- 172.21.15.27:0/1913235260 shutdown_connections 2024-01-15T18:17:07.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.743+0000 7f89ee000640 1 -- 172.21.15.27:0/1913235260 wait complete. 2024-01-15T18:17:07.761 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-15T18:17:07.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.907+0000 7f5af9400640 1 Processor -- start 2024-01-15T18:17:07.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.907+0000 7f5af9400640 1 -- start start 2024-01-15T18:17:07.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.907+0000 7f5af9400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5af40cf460 0x7f5af40cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:07.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.907+0000 7f5af9400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f5af40b2590 0x7f5af40cc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:07.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.907+0000 7f5af9400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af40cfd80 con 0x7f5af415b4b0 2024-01-15T18:17:07.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.907+0000 7f5af9400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5af40d19a0 con 0x7f5af40cf460 2024-01-15T18:17:07.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.907+0000 7f5af9400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5af40d1c80 con 0x7f5af40b2590 2024-01-15T18:17:07.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.907+0000 7f5af2800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5af40cf460 0x7f5af40cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:07.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.907+0000 7f5af2800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5af40cf460 0x7f5af40cf840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48480/0 (socket says 172.21.15.27:48480) 2024-01-15T18:17:07.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.907+0000 7f5af2800640 1 -- 172.21.15.27:0/2710680462 learned_addr learned my addr 172.21.15.27:0/2710680462 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:07.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.907+0000 7f5af1e00640 1 --1- 172.21.15.27:0/2710680462 >> v1:172.21.15.27:6789/0 conn(0x7f5af415b4b0 0x7f5af40cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47088/0 (socket says 172.21.15.27:47088) 2024-01-15T18:17:07.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.907+0000 7f5af3200640 1 --2- 172.21.15.27:0/2710680462 >> v2:172.21.15.93:3300/0 conn(0x7f5af40b2590 0x7f5af40cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:07.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.907+0000 7f5af2800640 1 -- 172.21.15.27:0/2710680462 >> v2:172.21.15.93:3300/0 conn(0x7f5af40b2590 msgr2=0x7f5af40cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:07.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.907+0000 7f5af2800640 1 --2- 172.21.15.27:0/2710680462 >> v2:172.21.15.93:3300/0 conn(0x7f5af40b2590 0x7f5af40cc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:07.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.907+0000 7f5af2800640 1 -- 172.21.15.27:0/2710680462 >> v1:172.21.15.27:6789/0 conn(0x7f5af415b4b0 legacy=0x7f5af40cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:07.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.907+0000 7f5af2800640 1 -- 172.21.15.27:0/2710680462 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af40b07c0 con 0x7f5af40cf460 2024-01-15T18:17:07.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.911+0000 7f5af2800640 1 --2- 172.21.15.27:0/2710680462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5af40cf460 0x7f5af40cf840 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f5af40d08b0 tx=0x7f5ae401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d39d5e44b2c83a7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:07.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.911+0000 7f5af1400640 1 -- 172.21.15.27:0/2710680462 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40aa660 con 0x7f5af40cf460 2024-01-15T18:17:07.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.911+0000 7f5af1400640 1 -- 172.21.15.27:0/2710680462 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae40aa7c0 con 0x7f5af40cf460 2024-01-15T18:17:07.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.911+0000 7f5af1400640 1 -- 172.21.15.27:0/2710680462 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40aab50 con 0x7f5af40cf460 2024-01-15T18:17:07.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.911+0000 7f5af9400640 1 -- 172.21.15.27:0/2710680462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5af40cf460 msgr2=0x7f5af40cf840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:07.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.911+0000 7f5af9400640 1 --2- 172.21.15.27:0/2710680462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5af40cf460 0x7f5af40cf840 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f5af40d08b0 tx=0x7f5ae401c500 comp rx=0 tx=0).stop 2024-01-15T18:17:07.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.911+0000 7f5af9400640 1 -- 172.21.15.27:0/2710680462 shutdown_connections 2024-01-15T18:17:07.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.911+0000 7f5af9400640 1 --2- 172.21.15.27:0/2710680462 >> v2:172.21.15.93:3300/0 conn(0x7f5af40b2590 0x7f5af40cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:07.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.911+0000 7f5af9400640 1 --2- 172.21.15.27:0/2710680462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5af40cf460 0x7f5af40cf840 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:07.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.911+0000 7f5af9400640 1 -- 172.21.15.27:0/2710680462 >> 172.21.15.27:0/2710680462 conn(0x7f5af40d0e80 msgr2=0x7f5af40d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:07.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.911+0000 7f5af9400640 1 -- 172.21.15.27:0/2710680462 shutdown_connections 2024-01-15T18:17:07.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.911+0000 7f5af9400640 1 -- 172.21.15.27:0/2710680462 wait complete. 2024-01-15T18:17:07.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.915+0000 7f5af9400640 1 Processor -- start 2024-01-15T18:17:07.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.915+0000 7f5af9400640 1 -- start start 2024-01-15T18:17:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.915+0000 7f5af9400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5af40b2590 0x7f5af415a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.915+0000 7f5af9400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5af40cf460 0x7f5af4150a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.915+0000 7f5af9400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5af415b4b0 0x7f5af41528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.915+0000 7f5af9400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af4153120 con 0x7f5af415b4b0 2024-01-15T18:17:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.915+0000 7f5af9400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af4159310 con 0x7f5af40b2590 2024-01-15T18:17:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.915+0000 7f5af9400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af41595f0 con 0x7f5af40cf460 2024-01-15T18:17:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.915+0000 7f5af3200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5af415b4b0 0x7f5af41528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.915+0000 7f5af3200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5af415b4b0 0x7f5af41528f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48490/0 (socket says 172.21.15.27:48490) 2024-01-15T18:17:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.915+0000 7f5af3200640 1 -- 172.21.15.27:0/3367370907 learned_addr learned my addr 172.21.15.27:0/3367370907 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.915+0000 7f5af1e00640 1 --2- 172.21.15.27:0/3367370907 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5af40cf460 0x7f5af4150a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.915+0000 7f5af2800640 1 --2- 172.21.15.27:0/3367370907 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5af40b2590 0x7f5af415a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.915+0000 7f5af3200640 1 -- 172.21.15.27:0/3367370907 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5af40cf460 msgr2=0x7f5af4150a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:07.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.915+0000 7f5af3200640 1 --2- 172.21.15.27:0/3367370907 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5af40cf460 0x7f5af4150a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:07.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.915+0000 7f5af3200640 1 -- 172.21.15.27:0/3367370907 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5af40b2590 msgr2=0x7f5af415a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:07.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.915+0000 7f5af3200640 1 --2- 172.21.15.27:0/3367370907 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5af40b2590 0x7f5af415a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:07.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.915+0000 7f5af3200640 1 -- 172.21.15.27:0/3367370907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae40094d0 con 0x7f5af415b4b0 2024-01-15T18:17:07.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.915+0000 7f5af3200640 1 --2- 172.21.15.27:0/3367370907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5af415b4b0 0x7f5af41528f0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f5aec007e30 tx=0x7f5aec00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:07.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.915+0000 7f5aeaa00640 1 -- 172.21.15.27:0/3367370907 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec014620 con 0x7f5af415b4b0 2024-01-15T18:17:07.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.915+0000 7f5aeaa00640 1 -- 172.21.15.27:0/3367370907 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aec00ee90 con 0x7f5af415b4b0 2024-01-15T18:17:07.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.915+0000 7f5aeaa00640 1 -- 172.21.15.27:0/3367370907 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec0149c0 con 0x7f5af415b4b0 2024-01-15T18:17:07.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.915+0000 7f5af9400640 1 -- 172.21.15.27:0/3367370907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af41598d0 con 0x7f5af415b4b0 2024-01-15T18:17:07.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.915+0000 7f5af9400640 1 -- 172.21.15.27:0/3367370907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af4152e30 con 0x7f5af415b4b0 2024-01-15T18:17:07.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.919+0000 7f5af9400640 1 -- 172.21.15.27:0/3367370907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af404e850 con 0x7f5af415b4b0 2024-01-15T18:17:07.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.919+0000 7f5aeaa00640 1 -- 172.21.15.27:0/3367370907 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f5aec014b20 con 0x7f5af415b4b0 2024-01-15T18:17:07.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.919+0000 7f5aeaa00640 1 --2- 172.21.15.27:0/3367370907 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5acc067700 0x7f5acc069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:07.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.919+0000 7f5aeaa00640 1 -- 172.21.15.27:0/3367370907 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f5aec083a00 con 0x7f5af415b4b0 2024-01-15T18:17:07.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.919+0000 7f5af2800640 1 --2- 172.21.15.27:0/3367370907 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5acc067700 0x7f5acc069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:07.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.919+0000 7f5af2800640 1 --2- 172.21.15.27:0/3367370907 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5acc067700 0x7f5acc069bc0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f5ae40ae000 tx=0x7f5ae40b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:07.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:07.923+0000 7f5aeaa00640 1 -- 172.21.15.27:0/3367370907 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5aec052480 con 0x7f5af415b4b0 2024-01-15T18:17:08.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.083+0000 7f5af9400640 1 -- 172.21.15.27:0/3367370907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f5af4159d80 con 0x7f5af415b4b0 2024-01-15T18:17:08.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.087+0000 7f5aeaa00640 1 -- 172.21.15.27:0/3367370907 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5aec055aa0 con 0x7f5af415b4b0 2024-01-15T18:17:08.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.087+0000 7f5af9400640 1 -- 172.21.15.27:0/3367370907 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5acc067700 msgr2=0x7f5acc069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:08.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.087+0000 7f5af9400640 1 --2- 172.21.15.27:0/3367370907 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5acc067700 0x7f5acc069bc0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f5ae40ae000 tx=0x7f5ae40b5040 comp rx=0 tx=0).stop 2024-01-15T18:17:08.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.087+0000 7f5af9400640 1 -- 172.21.15.27:0/3367370907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5af415b4b0 msgr2=0x7f5af41528f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:08.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.087+0000 7f5af9400640 1 --2- 172.21.15.27:0/3367370907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5af415b4b0 0x7f5af41528f0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f5aec007e30 tx=0x7f5aec00e4b0 comp rx=0 tx=0).stop 2024-01-15T18:17:08.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.091+0000 7f5af9400640 1 -- 172.21.15.27:0/3367370907 shutdown_connections 2024-01-15T18:17:08.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.091+0000 7f5af9400640 1 --2- 172.21.15.27:0/3367370907 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5acc067700 0x7f5acc069bc0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:08.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.091+0000 7f5af9400640 1 --2- 172.21.15.27:0/3367370907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5af415b4b0 0x7f5af41528f0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:08.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.091+0000 7f5af9400640 1 --2- 172.21.15.27:0/3367370907 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5af40cf460 0x7f5af4150a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:08.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.091+0000 7f5af9400640 1 --2- 172.21.15.27:0/3367370907 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5af40b2590 0x7f5af415a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:08.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.091+0000 7f5af9400640 1 -- 172.21.15.27:0/3367370907 >> 172.21.15.27:0/3367370907 conn(0x7f5af40d0e80 msgr2=0x7f5af40ada70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:08.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.091+0000 7f5af9400640 1 -- 172.21.15.27:0/3367370907 shutdown_connections 2024-01-15T18:17:08.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.091+0000 7f5af9400640 1 -- 172.21.15.27:0/3367370907 wait complete. 2024-01-15T18:17:08.107 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_stats_are_printed' 2024-01-15T18:17:08.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.251+0000 7f0558400640 1 Processor -- start 2024-01-15T18:17:08.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.251+0000 7f0558400640 1 -- start start 2024-01-15T18:17:08.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.251+0000 7f0558400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f055006db80 0x7f055006df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:08.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.251+0000 7f0558400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f055015e730 0x7f05500cd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:08.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.251+0000 7f0558400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05500ce920 con 0x7f05500cdd10 2024-01-15T18:17:08.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.251+0000 7f0558400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f055006e4a0 con 0x7f055015e730 2024-01-15T18:17:08.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.251+0000 7f0558400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05500ce640 con 0x7f055006db80 2024-01-15T18:17:08.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.251+0000 7f0556400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f05500cdd10 0x7f0550070c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47096/0 (socket says 172.21.15.27:47096) 2024-01-15T18:17:08.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.251+0000 7f0556400640 1 -- 172.21.15.27:0/654561791 learned_addr learned my addr 172.21.15.27:0/654561791 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:08.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.251+0000 7f0555a00640 1 --2- 172.21.15.27:0/654561791 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f055006db80 0x7f055006df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:08.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.251+0000 7f0555000640 1 --2- 172.21.15.27:0/654561791 >> v2:172.21.15.93:3300/0 conn(0x7f055015e730 0x7f05500cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:08.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.251+0000 7f054fe00640 1 -- 172.21.15.27:0/654561791 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 278349245 0 0) 0x7f05500ce920 con 0x7f05500cdd10 2024-01-15T18:17:08.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.251+0000 7f054fe00640 1 -- 172.21.15.27:0/654561791 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0530005740 con 0x7f05500cdd10 2024-01-15T18:17:08.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.251+0000 7f0555a00640 1 -- 172.21.15.27:0/654561791 >> v2:172.21.15.93:3300/0 conn(0x7f055015e730 msgr2=0x7f05500cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:08.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.251+0000 7f0555a00640 1 --2- 172.21.15.27:0/654561791 >> v2:172.21.15.93:3300/0 conn(0x7f055015e730 0x7f05500cd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:08.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.251+0000 7f0555a00640 1 -- 172.21.15.27:0/654561791 >> v1:172.21.15.27:6789/0 conn(0x7f05500cdd10 legacy=0x7f0550070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:08.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.251+0000 7f0555a00640 1 -- 172.21.15.27:0/654561791 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05500cc940 con 0x7f055006db80 2024-01-15T18:17:08.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.255+0000 7f0555a00640 1 --2- 172.21.15.27:0/654561791 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f055006db80 0x7f055006df60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f05400097c0 tx=0x7f054001c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18c8cf51ec233ec7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:08.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.255+0000 7f054fe00640 1 -- 172.21.15.27:0/654561791 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0540004230 con 0x7f055006db80 2024-01-15T18:17:08.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.255+0000 7f054fe00640 1 -- 172.21.15.27:0/654561791 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05400a7050 con 0x7f055006db80 2024-01-15T18:17:08.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.255+0000 7f054fe00640 1 -- 172.21.15.27:0/654561791 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0540003620 con 0x7f055006db80 2024-01-15T18:17:08.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0558400640 1 -- 172.21.15.27:0/654561791 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f055006db80 msgr2=0x7f055006df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:08.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0558400640 1 --2- 172.21.15.27:0/654561791 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f055006db80 0x7f055006df60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f05400097c0 tx=0x7f054001c450 comp rx=0 tx=0).stop 2024-01-15T18:17:08.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0558400640 1 -- 172.21.15.27:0/654561791 shutdown_connections 2024-01-15T18:17:08.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0558400640 1 --2- 172.21.15.27:0/654561791 >> v2:172.21.15.93:3300/0 conn(0x7f055015e730 0x7f05500cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:08.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0558400640 1 --2- 172.21.15.27:0/654561791 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f055006db80 0x7f055006df60 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:08.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0558400640 1 -- 172.21.15.27:0/654561791 >> 172.21.15.27:0/654561791 conn(0x7f05500cec60 msgr2=0x7f05500cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:08.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0558400640 1 -- 172.21.15.27:0/654561791 shutdown_connections 2024-01-15T18:17:08.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0558400640 1 -- 172.21.15.27:0/654561791 wait complete. 2024-01-15T18:17:08.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0558400640 1 Processor -- start 2024-01-15T18:17:08.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0558400640 1 -- start start 2024-01-15T18:17:08.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0558400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f055006db80 0x7f0550154ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:08.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0558400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05500cdd10 0x7f05500c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:08.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0558400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f055015e730 0x7f05501640b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:08.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0558400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0550163c90 con 0x7f05500cdd10 2024-01-15T18:17:08.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0558400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0550155010 con 0x7f055006db80 2024-01-15T18:17:08.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0558400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0550163010 con 0x7f055015e730 2024-01-15T18:17:08.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0555000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05500cdd10 0x7f05500c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:08.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0555000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05500cdd10 0x7f05500c54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48502/0 (socket says 172.21.15.27:48502) 2024-01-15T18:17:08.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0555000640 1 -- 172.21.15.27:0/4101754469 learned_addr learned my addr 172.21.15.27:0/4101754469 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:08.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0556400640 1 --2- 172.21.15.27:0/4101754469 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f055015e730 0x7f05501640b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:08.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0555a00640 1 --2- 172.21.15.27:0/4101754469 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f055006db80 0x7f0550154ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:08.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0555000640 1 -- 172.21.15.27:0/4101754469 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f055015e730 msgr2=0x7f05501640b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:08.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0555000640 1 --2- 172.21.15.27:0/4101754469 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f055015e730 0x7f05501640b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:08.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0555000640 1 -- 172.21.15.27:0/4101754469 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f055006db80 msgr2=0x7f0550154ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:08.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0555000640 1 --2- 172.21.15.27:0/4101754469 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f055006db80 0x7f0550154ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:08.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.259+0000 7f0555000640 1 -- 172.21.15.27:0/4101754469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0540009420 con 0x7f05500cdd10 2024-01-15T18:17:08.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.263+0000 7f0555000640 1 --2- 172.21.15.27:0/4101754469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05500cdd10 0x7f05500c54a0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f053800e930 tx=0x7f053800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:08.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.263+0000 7f054d600640 1 -- 172.21.15.27:0/4101754469 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0538009690 con 0x7f05500cdd10 2024-01-15T18:17:08.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.263+0000 7f054d600640 1 -- 172.21.15.27:0/4101754469 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f053800ca00 con 0x7f05500cdd10 2024-01-15T18:17:08.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.263+0000 7f0558400640 1 -- 172.21.15.27:0/4101754469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05501632f0 con 0x7f05500cdd10 2024-01-15T18:17:08.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.263+0000 7f054d600640 1 -- 172.21.15.27:0/4101754469 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053800cd70 con 0x7f05500cdd10 2024-01-15T18:17:08.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.263+0000 7f0558400640 1 -- 172.21.15.27:0/4101754469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05501638b0 con 0x7f05500cdd10 2024-01-15T18:17:08.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.263+0000 7f0558400640 1 -- 172.21.15.27:0/4101754469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0550163dd0 con 0x7f05500cdd10 2024-01-15T18:17:08.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.263+0000 7f054d600640 1 -- 172.21.15.27:0/4101754469 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f0538013470 con 0x7f05500cdd10 2024-01-15T18:17:08.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.263+0000 7f054d600640 1 --2- 172.21.15.27:0/4101754469 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0528067700 0x7f0528069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:08.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.263+0000 7f054d600640 1 -- 172.21.15.27:0/4101754469 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f05380833f0 con 0x7f05500cdd10 2024-01-15T18:17:08.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.263+0000 7f0555a00640 1 --2- 172.21.15.27:0/4101754469 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0528067700 0x7f0528069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:08.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.267+0000 7f0555a00640 1 --2- 172.21.15.27:0/4101754469 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0528067700 0x7f0528069bc0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f05400ad000 tx=0x7f05400a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:08.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.267+0000 7f054d600640 1 -- 172.21.15.27:0/4101754469 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0550163dd0 con 0x7f05500cdd10 2024-01-15T18:17:08.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:08.479+0000 7f0558400640 1 -- 172.21.15.27:0/4101754469 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_stats_are_printed"]} v 0) v1 -- 0x7f0550163dd0 con 0x7f05500cdd10 2024-01-15T18:17:09.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.043+0000 7f054d600640 1 -- 172.21.15.27:0/4101754469 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_stats_are_printed"]}]=0 v568) v1 ==== 145+0+0 (secure 0 0 0) 0x7f0550163dd0 con 0x7f05500cdd10 2024-01-15T18:17:09.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.043+0000 7f0558400640 1 -- 172.21.15.27:0/4101754469 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0528067700 msgr2=0x7f0528069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:09.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.043+0000 7f0558400640 1 --2- 172.21.15.27:0/4101754469 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0528067700 0x7f0528069bc0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f05400ad000 tx=0x7f05400a6000 comp rx=0 tx=0).stop 2024-01-15T18:17:09.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.043+0000 7f0558400640 1 -- 172.21.15.27:0/4101754469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05500cdd10 msgr2=0x7f05500c54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:09.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.043+0000 7f0558400640 1 --2- 172.21.15.27:0/4101754469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05500cdd10 0x7f05500c54a0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f053800e930 tx=0x7f053800edf0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.043+0000 7f0558400640 1 -- 172.21.15.27:0/4101754469 shutdown_connections 2024-01-15T18:17:09.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.043+0000 7f0558400640 1 --2- 172.21.15.27:0/4101754469 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0528067700 0x7f0528069bc0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.043+0000 7f0558400640 1 --2- 172.21.15.27:0/4101754469 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f055015e730 0x7f05501640b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.043+0000 7f0558400640 1 --2- 172.21.15.27:0/4101754469 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05500cdd10 0x7f05500c54a0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.043+0000 7f0558400640 1 --2- 172.21.15.27:0/4101754469 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f055006db80 0x7f0550154ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.043+0000 7f0558400640 1 -- 172.21.15.27:0/4101754469 >> 172.21.15.27:0/4101754469 conn(0x7f05500cec60 msgr2=0x7f05501591a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:09.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.043+0000 7f0558400640 1 -- 172.21.15.27:0/4101754469 shutdown_connections 2024-01-15T18:17:09.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.043+0000 7f0558400640 1 -- 172.21.15.27:0/4101754469 wait complete. 2024-01-15T18:17:09.066 INFO:tasks.cephfs_test_runner:Test that the command "ceph fs clone status" prints progress stats ... ok 2024-01-15T18:17:09.066 INFO:tasks.cephfs_test_runner:Starting test: test_when_1_clone_is_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) Test that the progress bars for ongoing clones is removed from the 2024-01-15T18:17:09.066 INFO:tasks.cephfs_test_runner:test_when_1_clone_is_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T18:17:09.067 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:17:09.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.211+0000 7f208b600640 1 Processor -- start 2024-01-15T18:17:09.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.211+0000 7f208b600640 1 -- start start 2024-01-15T18:17:09.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.215+0000 7f208b600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f208415e830 0x7f20840ceaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:09.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.215+0000 7f208b600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f20840d1ab0 0x7f20840cf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:09.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.215+0000 7f208b600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20840c4ba0 con 0x7f20840cf570 2024-01-15T18:17:09.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.215+0000 7f208b600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20840cc5a0 con 0x7f208415e830 2024-01-15T18:17:09.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.215+0000 7f208b600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20840cc840 con 0x7f20840d1ab0 2024-01-15T18:17:09.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.215+0000 7f2088c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f208415e830 0x7f20840ceaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:09.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.215+0000 7f2088c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f208415e830 0x7f20840ceaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48516/0 (socket says 172.21.15.27:48516) 2024-01-15T18:17:09.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.215+0000 7f2088c00640 1 -- 172.21.15.27:0/454283349 learned_addr learned my addr 172.21.15.27:0/454283349 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:09.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.215+0000 7f2083e00640 1 --2- 172.21.15.27:0/454283349 >> v2:172.21.15.93:3300/0 conn(0x7f20840d1ab0 0x7f20840cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:09.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.215+0000 7f2088c00640 1 -- 172.21.15.27:0/454283349 >> v2:172.21.15.93:3300/0 conn(0x7f20840d1ab0 msgr2=0x7f20840cf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:09.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.215+0000 7f2088c00640 1 --2- 172.21.15.27:0/454283349 >> v2:172.21.15.93:3300/0 conn(0x7f20840d1ab0 0x7f20840cf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.215+0000 7f2088c00640 1 -- 172.21.15.27:0/454283349 >> v1:172.21.15.93:6789/0 conn(0x7f20840cf570 legacy=0x7f20840cd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:09.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.215+0000 7f2088c00640 1 -- 172.21.15.27:0/454283349 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20840ccb20 con 0x7f208415e830 2024-01-15T18:17:09.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.219+0000 7f2088c00640 1 --2- 172.21.15.27:0/454283349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f208415e830 0x7f20840ceaf0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f2074009870 tx=0x7f207401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd4b99fc67aa8e71 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:09.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.219+0000 7f2083400640 1 -- 172.21.15.27:0/454283349 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20740ab600 con 0x7f208415e830 2024-01-15T18:17:09.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.219+0000 7f2083400640 1 -- 172.21.15.27:0/454283349 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20740a7050 con 0x7f208415e830 2024-01-15T18:17:09.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.219+0000 7f2083400640 1 -- 172.21.15.27:0/454283349 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20740ab980 con 0x7f208415e830 2024-01-15T18:17:09.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.219+0000 7f208b600640 1 -- 172.21.15.27:0/454283349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f208415e830 msgr2=0x7f20840ceaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:09.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.219+0000 7f208b600640 1 --2- 172.21.15.27:0/454283349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f208415e830 0x7f20840ceaf0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f2074009870 tx=0x7f207401c430 comp rx=0 tx=0).stop 2024-01-15T18:17:09.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.219+0000 7f208b600640 1 -- 172.21.15.27:0/454283349 shutdown_connections 2024-01-15T18:17:09.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.219+0000 7f208b600640 1 --2- 172.21.15.27:0/454283349 >> v2:172.21.15.93:3300/0 conn(0x7f20840d1ab0 0x7f20840cf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.219+0000 7f208b600640 1 --2- 172.21.15.27:0/454283349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f208415e830 0x7f20840ceaf0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.219+0000 7f208b600640 1 -- 172.21.15.27:0/454283349 >> 172.21.15.27:0/454283349 conn(0x7f20840d0f20 msgr2=0x7f20840d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:09.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.219+0000 7f208b600640 1 -- 172.21.15.27:0/454283349 shutdown_connections 2024-01-15T18:17:09.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.219+0000 7f208b600640 1 -- 172.21.15.27:0/454283349 wait complete. 2024-01-15T18:17:09.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.219+0000 7f208b600640 1 Processor -- start 2024-01-15T18:17:09.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.219+0000 7f208b600640 1 -- start start 2024-01-15T18:17:09.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.223+0000 7f208b600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20840cf570 0x7f208415b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:09.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.223+0000 7f208b600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20840d1ab0 0x7f20840cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:09.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.223+0000 7f208b600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f208415e830 0x7f20840cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:09.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.223+0000 7f208b600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208415b890 con 0x7f20840d1ab0 2024-01-15T18:17:09.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.223+0000 7f208b600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208415cbb0 con 0x7f20840cf570 2024-01-15T18:17:09.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.223+0000 7f208b600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f208415bb10 con 0x7f208415e830 2024-01-15T18:17:09.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.223+0000 7f2083e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20840d1ab0 0x7f20840cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:09.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.223+0000 7f2083e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20840d1ab0 0x7f20840cb9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48526/0 (socket says 172.21.15.27:48526) 2024-01-15T18:17:09.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.223+0000 7f2083e00640 1 -- 172.21.15.27:0/1097272063 learned_addr learned my addr 172.21.15.27:0/1097272063 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:09.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.223+0000 7f2088c00640 1 --2- 172.21.15.27:0/1097272063 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20840cf570 0x7f208415b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:09.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.223+0000 7f2089600640 1 --2- 172.21.15.27:0/1097272063 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f208415e830 0x7f20840cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:09.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.223+0000 7f2083e00640 1 -- 172.21.15.27:0/1097272063 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f208415e830 msgr2=0x7f20840cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:09.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.223+0000 7f2083e00640 1 --2- 172.21.15.27:0/1097272063 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f208415e830 0x7f20840cbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.223+0000 7f2083e00640 1 -- 172.21.15.27:0/1097272063 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20840cf570 msgr2=0x7f208415b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:09.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.223+0000 7f2083e00640 1 --2- 172.21.15.27:0/1097272063 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20840cf570 0x7f208415b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.223+0000 7f2083e00640 1 -- 172.21.15.27:0/1097272063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20740094d0 con 0x7f20840d1ab0 2024-01-15T18:17:09.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.223+0000 7f2083e00640 1 --2- 172.21.15.27:0/1097272063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20840d1ab0 0x7f20840cb9b0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f206c00d800 tx=0x7f206c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:09.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.223+0000 7f2080c00640 1 -- 172.21.15.27:0/1097272063 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f206c00b6c0 con 0x7f20840d1ab0 2024-01-15T18:17:09.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.223+0000 7f2080c00640 1 -- 172.21.15.27:0/1097272063 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f206c00b820 con 0x7f20840d1ab0 2024-01-15T18:17:09.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.223+0000 7f2080c00640 1 -- 172.21.15.27:0/1097272063 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f206c00bb90 con 0x7f20840d1ab0 2024-01-15T18:17:09.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.223+0000 7f208b600640 1 -- 172.21.15.27:0/1097272063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f208415bdf0 con 0x7f20840d1ab0 2024-01-15T18:17:09.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.223+0000 7f208b600640 1 -- 172.21.15.27:0/1097272063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f208415c3b0 con 0x7f20840d1ab0 2024-01-15T18:17:09.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.223+0000 7f208b600640 1 -- 172.21.15.27:0/1097272063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f208404e850 con 0x7f20840d1ab0 2024-01-15T18:17:09.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.227+0000 7f2080c00640 1 -- 172.21.15.27:0/1097272063 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f206c00bcf0 con 0x7f20840d1ab0 2024-01-15T18:17:09.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.227+0000 7f2080c00640 1 --2- 172.21.15.27:0/1097272063 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f205c067700 0x7f205c069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:09.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.227+0000 7f2080c00640 1 -- 172.21.15.27:0/1097272063 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f206c083830 con 0x7f20840d1ab0 2024-01-15T18:17:09.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.227+0000 7f2088c00640 1 --2- 172.21.15.27:0/1097272063 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f205c067700 0x7f205c069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:09.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.227+0000 7f2088c00640 1 --2- 172.21.15.27:0/1097272063 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f205c067700 0x7f205c069bc0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f20740ad000 tx=0x7f20740a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:09.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.231+0000 7f2080c00640 1 -- 172.21.15.27:0/1097272063 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f206c052100 con 0x7f20840d1ab0 2024-01-15T18:17:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.387+0000 7f208b600640 1 -- 172.21.15.27:0/1097272063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f208406f7d0 con 0x7f20840d1ab0 2024-01-15T18:17:09.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.387+0000 7f2080c00640 1 -- 172.21.15.27:0/1097272063 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+13066 (secure 0 0 0) 0x7f206c011340 con 0x7f20840d1ab0 2024-01-15T18:17:09.386 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:17:09.387 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":89,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:17:04.008565+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.93:6837/980414657":"2024-01-16T18:17:00.989983+0000","172.21.15.93:6835/980414657":"2024-01-16T18:17:00.989983+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:17:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.391+0000 7f208b600640 1 -- 172.21.15.27:0/1097272063 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f205c067700 msgr2=0x7f205c069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.391+0000 7f208b600640 1 --2- 172.21.15.27:0/1097272063 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f205c067700 0x7f205c069bc0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f20740ad000 tx=0x7f20740a6000 comp rx=0 tx=0).stop 2024-01-15T18:17:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.391+0000 7f208b600640 1 -- 172.21.15.27:0/1097272063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20840d1ab0 msgr2=0x7f20840cb9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.391+0000 7f208b600640 1 --2- 172.21.15.27:0/1097272063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20840d1ab0 0x7f20840cb9b0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f206c00d800 tx=0x7f206c00dcc0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.391+0000 7f208b600640 1 -- 172.21.15.27:0/1097272063 shutdown_connections 2024-01-15T18:17:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.391+0000 7f208b600640 1 --2- 172.21.15.27:0/1097272063 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f205c067700 0x7f205c069bc0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.391+0000 7f208b600640 1 --2- 172.21.15.27:0/1097272063 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f208415e830 0x7f20840cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.391+0000 7f208b600640 1 --2- 172.21.15.27:0/1097272063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20840d1ab0 0x7f20840cb9b0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.391+0000 7f208b600640 1 --2- 172.21.15.27:0/1097272063 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20840cf570 0x7f208415b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.391+0000 7f208b600640 1 -- 172.21.15.27:0/1097272063 >> 172.21.15.27:0/1097272063 conn(0x7f20840d0f20 msgr2=0x7f208415dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.391+0000 7f208b600640 1 -- 172.21.15.27:0/1097272063 shutdown_connections 2024-01-15T18:17:09.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.391+0000 7f208b600640 1 -- 172.21.15.27:0/1097272063 wait complete. 2024-01-15T18:17:09.409 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:17:09.410 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:17:09.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.555+0000 7f7540600640 1 Processor -- start 2024-01-15T18:17:09.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.555+0000 7f7540600640 1 -- start start 2024-01-15T18:17:09.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.555+0000 7f7540600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f753815d3b0 0x7f75380cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:09.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.555+0000 7f7540600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f753806b600 0x7f753806bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:09.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.555+0000 7f7540600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f753806c220 0x7f7538160f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:09.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.555+0000 7f7540600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f753806c6b0 con 0x7f753815d3b0 2024-01-15T18:17:09.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.555+0000 7f7540600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75380b04b0 con 0x7f753806c220 2024-01-15T18:17:09.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.555+0000 7f7540600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75380d0160 con 0x7f753806b600 2024-01-15T18:17:09.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.555+0000 7f753dc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f753815d3b0 0x7f75380cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:09.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.555+0000 7f753dc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f753815d3b0 0x7f75380cfc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48538/0 (socket says 172.21.15.27:48538) 2024-01-15T18:17:09.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.555+0000 7f753dc00640 1 -- 172.21.15.27:0/3507169800 learned_addr learned my addr 172.21.15.27:0/3507169800 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:09.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.555+0000 7f753e600640 1 --2- 172.21.15.27:0/3507169800 >> v2:172.21.15.93:3300/0 conn(0x7f753806c220 0x7f7538160f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:09.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.555+0000 7f753d200640 1 --2- 172.21.15.27:0/3507169800 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f753806b600 0x7f753806bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:09.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.555+0000 7f753dc00640 1 -- 172.21.15.27:0/3507169800 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f753806b600 msgr2=0x7f753806bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:09.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.555+0000 7f753dc00640 1 --2- 172.21.15.27:0/3507169800 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f753806b600 0x7f753806bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.555+0000 7f753dc00640 1 -- 172.21.15.27:0/3507169800 >> v2:172.21.15.93:3300/0 conn(0x7f753806c220 msgr2=0x7f7538160f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:09.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.555+0000 7f753dc00640 1 --2- 172.21.15.27:0/3507169800 >> v2:172.21.15.93:3300/0 conn(0x7f753806c220 0x7f7538160f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.555+0000 7f753dc00640 1 -- 172.21.15.27:0/3507169800 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75380b0720 con 0x7f753815d3b0 2024-01-15T18:17:09.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.555+0000 7f753d200640 1 --2- 172.21.15.27:0/3507169800 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f753806b600 0x7f753806bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:17:09.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.559+0000 7f753dc00640 1 --2- 172.21.15.27:0/3507169800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f753815d3b0 0x7f75380cfc20 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f7528009870 tx=0x7f752801c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6649f53e69e190bc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:09.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.559+0000 7f7537e00640 1 -- 172.21.15.27:0/3507169800 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75280ab600 con 0x7f753815d3b0 2024-01-15T18:17:09.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.559+0000 7f7537e00640 1 -- 172.21.15.27:0/3507169800 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75280a7050 con 0x7f753815d3b0 2024-01-15T18:17:09.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.559+0000 7f7537e00640 1 -- 172.21.15.27:0/3507169800 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75280ab980 con 0x7f753815d3b0 2024-01-15T18:17:09.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.559+0000 7f7540600640 1 -- 172.21.15.27:0/3507169800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f753815d3b0 msgr2=0x7f75380cfc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:09.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.559+0000 7f7540600640 1 --2- 172.21.15.27:0/3507169800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f753815d3b0 0x7f75380cfc20 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f7528009870 tx=0x7f752801c210 comp rx=0 tx=0).stop 2024-01-15T18:17:09.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.559+0000 7f7540600640 1 -- 172.21.15.27:0/3507169800 shutdown_connections 2024-01-15T18:17:09.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.559+0000 7f7540600640 1 --2- 172.21.15.27:0/3507169800 >> v2:172.21.15.93:3300/0 conn(0x7f753806c220 0x7f7538160f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.559+0000 7f7540600640 1 --2- 172.21.15.27:0/3507169800 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f753806b600 0x7f753806bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.559+0000 7f7540600640 1 --2- 172.21.15.27:0/3507169800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f753815d3b0 0x7f75380cfc20 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.559+0000 7f7540600640 1 -- 172.21.15.27:0/3507169800 >> 172.21.15.27:0/3507169800 conn(0x7f75380d0ed0 msgr2=0x7f75380d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:09.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.559+0000 7f7540600640 1 -- 172.21.15.27:0/3507169800 shutdown_connections 2024-01-15T18:17:09.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.559+0000 7f7540600640 1 -- 172.21.15.27:0/3507169800 wait complete. 2024-01-15T18:17:09.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.563+0000 7f7540600640 1 Processor -- start 2024-01-15T18:17:09.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.563+0000 7f7540600640 1 -- start start 2024-01-15T18:17:09.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.563+0000 7f7540600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f753806b600 0x7f7538154fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:09.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.563+0000 7f7540600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f753806c220 0x7f753809d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:09.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.563+0000 7f7540600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f753815d3b0 0x7f753809da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:09.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.563+0000 7f7540600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f753815e9f0 con 0x7f753806c220 2024-01-15T18:17:09.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.563+0000 7f7540600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f753815ecd0 con 0x7f753806b600 2024-01-15T18:17:09.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.563+0000 7f7540600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f753815efb0 con 0x7f753815d3b0 2024-01-15T18:17:09.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.563+0000 7f753d200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f753806c220 0x7f753809d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:09.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.563+0000 7f753d200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f753806c220 0x7f753809d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48550/0 (socket says 172.21.15.27:48550) 2024-01-15T18:17:09.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.563+0000 7f753d200640 1 -- 172.21.15.27:0/2551335802 learned_addr learned my addr 172.21.15.27:0/2551335802 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:09.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.563+0000 7f753dc00640 1 --2- 172.21.15.27:0/2551335802 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f753806b600 0x7f7538154fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:09.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.563+0000 7f753e600640 1 --2- 172.21.15.27:0/2551335802 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f753815d3b0 0x7f753809da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:09.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.563+0000 7f753d200640 1 -- 172.21.15.27:0/2551335802 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f753815d3b0 msgr2=0x7f753809da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:09.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.563+0000 7f753d200640 1 --2- 172.21.15.27:0/2551335802 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f753815d3b0 0x7f753809da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.563+0000 7f753d200640 1 -- 172.21.15.27:0/2551335802 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f753806b600 msgr2=0x7f7538154fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:09.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.563+0000 7f753d200640 1 --2- 172.21.15.27:0/2551335802 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f753806b600 0x7f7538154fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.563+0000 7f753d200640 1 -- 172.21.15.27:0/2551335802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75280094d0 con 0x7f753806c220 2024-01-15T18:17:09.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.563+0000 7f753e600640 1 --2- 172.21.15.27:0/2551335802 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f753815d3b0 0x7f753809da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:17:09.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.563+0000 7f753d200640 1 --2- 172.21.15.27:0/2551335802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f753806c220 0x7f753809d540 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f752000c7c0 tx=0x7f752000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:09.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.563+0000 7f7535600640 1 -- 172.21.15.27:0/2551335802 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7520007a30 con 0x7f753806c220 2024-01-15T18:17:09.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.563+0000 7f7535600640 1 -- 172.21.15.27:0/2551335802 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7520007b90 con 0x7f753806c220 2024-01-15T18:17:09.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.563+0000 7f7535600640 1 -- 172.21.15.27:0/2551335802 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f752000ce70 con 0x7f753806c220 2024-01-15T18:17:09.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.563+0000 7f7540600640 1 -- 172.21.15.27:0/2551335802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f753815f290 con 0x7f753806c220 2024-01-15T18:17:09.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.563+0000 7f7540600640 1 -- 172.21.15.27:0/2551335802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f753815f850 con 0x7f753806c220 2024-01-15T18:17:09.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.567+0000 7f7540600640 1 -- 172.21.15.27:0/2551335802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f753806d550 con 0x7f753806c220 2024-01-15T18:17:09.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.567+0000 7f7535600640 1 -- 172.21.15.27:0/2551335802 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f75200113f0 con 0x7f753806c220 2024-01-15T18:17:09.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.567+0000 7f7535600640 1 --2- 172.21.15.27:0/2551335802 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7510067700 0x7f7510069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:09.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.567+0000 7f7535600640 1 -- 172.21.15.27:0/2551335802 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f7520083230 con 0x7f753806c220 2024-01-15T18:17:09.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.567+0000 7f753dc00640 1 --2- 172.21.15.27:0/2551335802 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7510067700 0x7f7510069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:09.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.567+0000 7f753dc00640 1 --2- 172.21.15.27:0/2551335802 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7510067700 0x7f7510069bc0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f75280ad000 tx=0x7f7528008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:09.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.571+0000 7f7535600640 1 -- 172.21.15.27:0/2551335802 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7520051cb0 con 0x7f753806c220 2024-01-15T18:17:09.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.727+0000 7f7540600640 1 -- 172.21.15.27:0/2551335802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f753815fa70 con 0x7f753806c220 2024-01-15T18:17:09.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.727+0000 7f7535600640 1 -- 172.21.15.27:0/2551335802 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f75200552d0 con 0x7f753806c220 2024-01-15T18:17:09.727 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:17:09.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.731+0000 7f7540600640 1 -- 172.21.15.27:0/2551335802 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7510067700 msgr2=0x7f7510069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:09.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.731+0000 7f7540600640 1 --2- 172.21.15.27:0/2551335802 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7510067700 0x7f7510069bc0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f75280ad000 tx=0x7f7528008040 comp rx=0 tx=0).stop 2024-01-15T18:17:09.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.731+0000 7f7540600640 1 -- 172.21.15.27:0/2551335802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f753806c220 msgr2=0x7f753809d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:09.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.731+0000 7f7540600640 1 --2- 172.21.15.27:0/2551335802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f753806c220 0x7f753809d540 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f752000c7c0 tx=0x7f752000cc80 comp rx=0 tx=0).stop 2024-01-15T18:17:09.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.731+0000 7f7540600640 1 -- 172.21.15.27:0/2551335802 shutdown_connections 2024-01-15T18:17:09.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.731+0000 7f7540600640 1 --2- 172.21.15.27:0/2551335802 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7510067700 0x7f7510069bc0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.731+0000 7f7540600640 1 --2- 172.21.15.27:0/2551335802 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f753815d3b0 0x7f753809da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.731+0000 7f7540600640 1 --2- 172.21.15.27:0/2551335802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f753806c220 0x7f753809d540 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.731+0000 7f7540600640 1 --2- 172.21.15.27:0/2551335802 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f753806b600 0x7f7538154fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.731+0000 7f7540600640 1 -- 172.21.15.27:0/2551335802 >> 172.21.15.27:0/2551335802 conn(0x7f75380d0ed0 msgr2=0x7f753806f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:09.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.731+0000 7f7540600640 1 -- 172.21.15.27:0/2551335802 shutdown_connections 2024-01-15T18:17:09.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.731+0000 7f7540600640 1 -- 172.21.15.27:0/2551335802 wait complete. 2024-01-15T18:17:09.749 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled' 2024-01-15T18:17:09.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.891+0000 7f5956200640 1 Processor -- start 2024-01-15T18:17:09.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.891+0000 7f5956200640 1 -- start start 2024-01-15T18:17:09.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.891+0000 7f5956200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f595015d0a0 0x7f595006c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:09.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.891+0000 7f5956200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f59500ce710 0x7f59500cd240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:09.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.891+0000 7f5956200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59500ad180 con 0x7f59500c94c0 2024-01-15T18:17:09.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.891+0000 7f5956200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59500ac890 con 0x7f59500ce710 2024-01-15T18:17:09.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.891+0000 7f5956200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59500cda90 con 0x7f595015d0a0 2024-01-15T18:17:09.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.891+0000 7f594ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f59500ce710 0x7f59500cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:09.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.891+0000 7f594fe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f59500c94c0 0x7f59500ad610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47104/0 (socket says 172.21.15.27:47104) 2024-01-15T18:17:09.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.891+0000 7f594ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f59500ce710 0x7f59500cd240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48566/0 (socket says 172.21.15.27:48566) 2024-01-15T18:17:09.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.891+0000 7f594fe00640 1 -- 172.21.15.27:0/602860252 learned_addr learned my addr 172.21.15.27:0/602860252 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:09.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.895+0000 7f594f400640 1 --2- 172.21.15.27:0/602860252 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f595015d0a0 0x7f595006c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:09.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.895+0000 7f594ea00640 1 -- 172.21.15.27:0/602860252 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f595015d0a0 msgr2=0x7f595006c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:09.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.895+0000 7f594ea00640 1 --2- 172.21.15.27:0/602860252 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f595015d0a0 0x7f595006c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.895+0000 7f594ea00640 1 -- 172.21.15.27:0/602860252 >> v1:172.21.15.27:6789/0 conn(0x7f59500c94c0 legacy=0x7f59500ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:09.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.895+0000 7f594ea00640 1 -- 172.21.15.27:0/602860252 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59500acb30 con 0x7f59500ce710 2024-01-15T18:17:09.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.895+0000 7f594e000640 1 -- 172.21.15.27:0/602860252 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1030278915 0 0) 0x7f59500ad180 con 0x7f59500c94c0 2024-01-15T18:17:09.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.895+0000 7f594f400640 1 --2- 172.21.15.27:0/602860252 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f595015d0a0 0x7f595006c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:17:09.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.895+0000 7f594ea00640 1 --2- 172.21.15.27:0/602860252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59500ce710 0x7f59500cd240 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f59380097c0 tx=0x7f593801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6b97bb01beb8915 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:09.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.895+0000 7f594e000640 1 -- 172.21.15.27:0/602860252 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59380ab610 con 0x7f59500ce710 2024-01-15T18:17:09.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.895+0000 7f594e000640 1 -- 172.21.15.27:0/602860252 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59380ab770 con 0x7f59500ce710 2024-01-15T18:17:09.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.895+0000 7f594e000640 1 -- 172.21.15.27:0/602860252 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59380abb00 con 0x7f59500ce710 2024-01-15T18:17:09.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.899+0000 7f5956200640 1 -- 172.21.15.27:0/602860252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59500ce710 msgr2=0x7f59500cd240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:09.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.899+0000 7f5956200640 1 --2- 172.21.15.27:0/602860252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59500ce710 0x7f59500cd240 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f59380097c0 tx=0x7f593801c450 comp rx=0 tx=0).stop 2024-01-15T18:17:09.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.899+0000 7f5956200640 1 -- 172.21.15.27:0/602860252 shutdown_connections 2024-01-15T18:17:09.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.899+0000 7f5956200640 1 --2- 172.21.15.27:0/602860252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59500ce710 0x7f59500cd240 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.899+0000 7f5956200640 1 --2- 172.21.15.27:0/602860252 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f595015d0a0 0x7f595006c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.899+0000 7f5956200640 1 -- 172.21.15.27:0/602860252 >> 172.21.15.27:0/602860252 conn(0x7f59500cec60 msgr2=0x7f59500cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:09.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.899+0000 7f5956200640 1 -- 172.21.15.27:0/602860252 shutdown_connections 2024-01-15T18:17:09.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.899+0000 7f5956200640 1 -- 172.21.15.27:0/602860252 wait complete. 2024-01-15T18:17:09.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.899+0000 7f5956200640 1 Processor -- start 2024-01-15T18:17:09.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.899+0000 7f5956200640 1 -- start start 2024-01-15T18:17:09.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.899+0000 7f5956200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59500c94c0 0x7f59501593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:09.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.899+0000 7f5956200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59500ce710 0x7f595015ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:09.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.899+0000 7f5956200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f595015d0a0 0x7f595015b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:09.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.899+0000 7f5956200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5950152e80 con 0x7f59500ce710 2024-01-15T18:17:09.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.899+0000 7f5956200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595014fc70 con 0x7f59500c94c0 2024-01-15T18:17:09.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.899+0000 7f5956200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f595014d880 con 0x7f595015d0a0 2024-01-15T18:17:09.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.899+0000 7f594ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59500ce710 0x7f595015ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:09.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.899+0000 7f594ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59500ce710 0x7f595015ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48576/0 (socket says 172.21.15.27:48576) 2024-01-15T18:17:09.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.899+0000 7f594ea00640 1 -- 172.21.15.27:0/2567174119 learned_addr learned my addr 172.21.15.27:0/2567174119 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:09.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.903+0000 7f594f400640 1 --2- 172.21.15.27:0/2567174119 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59500c94c0 0x7f59501593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:09.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.903+0000 7f594fe00640 1 --2- 172.21.15.27:0/2567174119 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f595015d0a0 0x7f595015b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:09.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.903+0000 7f594ea00640 1 -- 172.21.15.27:0/2567174119 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f595015d0a0 msgr2=0x7f595015b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:09.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.903+0000 7f594ea00640 1 --2- 172.21.15.27:0/2567174119 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f595015d0a0 0x7f595015b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.903+0000 7f594ea00640 1 -- 172.21.15.27:0/2567174119 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59500c94c0 msgr2=0x7f59501593b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:09.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.903+0000 7f594ea00640 1 --2- 172.21.15.27:0/2567174119 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59500c94c0 0x7f59501593b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:09.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.903+0000 7f594ea00640 1 -- 172.21.15.27:0/2567174119 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5938009420 con 0x7f59500ce710 2024-01-15T18:17:09.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.903+0000 7f594fe00640 1 --2- 172.21.15.27:0/2567174119 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f595015d0a0 0x7f595015b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:17:09.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.903+0000 7f594ea00640 1 --2- 172.21.15.27:0/2567174119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59500ce710 0x7f595015ad40 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f593801a080 tx=0x7f593801ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:09.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.903+0000 7f5947400640 1 -- 172.21.15.27:0/2567174119 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59380b5810 con 0x7f59500ce710 2024-01-15T18:17:09.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.903+0000 7f5947400640 1 -- 172.21.15.27:0/2567174119 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59380b5990 con 0x7f59500ce710 2024-01-15T18:17:09.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.903+0000 7f5947400640 1 -- 172.21.15.27:0/2567174119 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59380b5d40 con 0x7f59500ce710 2024-01-15T18:17:09.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.903+0000 7f5956200640 1 -- 172.21.15.27:0/2567174119 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59500ae2b0 con 0x7f59500ce710 2024-01-15T18:17:09.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.903+0000 7f5956200640 1 -- 172.21.15.27:0/2567174119 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f595014de20 con 0x7f59500ce710 2024-01-15T18:17:09.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.903+0000 7f5956200640 1 -- 172.21.15.27:0/2567174119 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f595004e850 con 0x7f59500ce710 2024-01-15T18:17:09.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.903+0000 7f5947400640 1 -- 172.21.15.27:0/2567174119 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f59380bf410 con 0x7f59500ce710 2024-01-15T18:17:09.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.903+0000 7f5947400640 1 --2- 172.21.15.27:0/2567174119 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5928067700 0x7f5928069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:09.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.907+0000 7f5947400640 1 -- 172.21.15.27:0/2567174119 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f593811da20 con 0x7f59500ce710 2024-01-15T18:17:09.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.907+0000 7f594f400640 1 --2- 172.21.15.27:0/2567174119 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5928067700 0x7f5928069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:09.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.907+0000 7f594f400640 1 --2- 172.21.15.27:0/2567174119 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5928067700 0x7f5928069bc0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f5940006fd0 tx=0x7f5940008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:09.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:09.907+0000 7f5947400640 1 -- 172.21.15.27:0/2567174119 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f59380ec5f0 con 0x7f59500ce710 2024-01-15T18:17:10.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:10.123+0000 7f5956200640 1 -- 172.21.15.27:0/2567174119 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled"]} v 0) v1 -- 0x7f5950070e00 con 0x7f59500ce710 2024-01-15T18:17:11.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.051+0000 7f5947400640 1 -- 172.21.15.27:0/2567174119 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled"]}]=0 v570) v1 ==== 156+0+0 (secure 0 0 0) 0x7f59380efc10 con 0x7f59500ce710 2024-01-15T18:17:11.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.055+0000 7f5956200640 1 -- 172.21.15.27:0/2567174119 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5928067700 msgr2=0x7f5928069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:11.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.055+0000 7f5956200640 1 --2- 172.21.15.27:0/2567174119 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5928067700 0x7f5928069bc0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f5940006fd0 tx=0x7f5940008040 comp rx=0 tx=0).stop 2024-01-15T18:17:11.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.055+0000 7f5956200640 1 -- 172.21.15.27:0/2567174119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59500ce710 msgr2=0x7f595015ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:11.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.055+0000 7f5956200640 1 --2- 172.21.15.27:0/2567174119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59500ce710 0x7f595015ad40 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f593801a080 tx=0x7f593801ac90 comp rx=0 tx=0).stop 2024-01-15T18:17:11.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.055+0000 7f5956200640 1 -- 172.21.15.27:0/2567174119 shutdown_connections 2024-01-15T18:17:11.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.055+0000 7f5956200640 1 --2- 172.21.15.27:0/2567174119 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5928067700 0x7f5928069bc0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:11.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.055+0000 7f5956200640 1 --2- 172.21.15.27:0/2567174119 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f595015d0a0 0x7f595015b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:11.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.055+0000 7f5956200640 1 --2- 172.21.15.27:0/2567174119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59500ce710 0x7f595015ad40 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:11.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.055+0000 7f5956200640 1 --2- 172.21.15.27:0/2567174119 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59500c94c0 0x7f59501593b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:11.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.055+0000 7f5956200640 1 -- 172.21.15.27:0/2567174119 >> 172.21.15.27:0/2567174119 conn(0x7f59500cec60 msgr2=0x7f5950070180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:11.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.055+0000 7f5956200640 1 -- 172.21.15.27:0/2567174119 shutdown_connections 2024-01-15T18:17:11.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.055+0000 7f5956200640 1 -- 172.21.15.27:0/2567174119 wait complete. 2024-01-15T18:17:11.079 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:17:11.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa329000640 1 Processor -- start 2024-01-15T18:17:11.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa329000640 1 -- start start 2024-01-15T18:17:11.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa329000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa3240cdab0 0x7fa3240b1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:11.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa329000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3240cc610 con 0x7fa32406bd10 2024-01-15T18:17:11.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa329000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa32406c380 con 0x7fa324160bc0 2024-01-15T18:17:11.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa329000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa324160fa0 con 0x7fa3240cdab0 2024-01-15T18:17:11.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa322800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa32406bd10 0x7fa3240cf230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47106/0 (socket says 172.21.15.27:47106) 2024-01-15T18:17:11.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa322800640 1 -- 172.21.15.27:0/3405155817 learned_addr learned my addr 172.21.15.27:0/3405155817 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:11.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa321400640 1 --2- 172.21.15.27:0/3405155817 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa3240cdab0 0x7fa3240b1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:11.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa320a00640 1 -- 172.21.15.27:0/3405155817 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442079495 0 0) 0x7fa3240cc610 con 0x7fa32406bd10 2024-01-15T18:17:11.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa320a00640 1 -- 172.21.15.27:0/3405155817 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa304003530 con 0x7fa32406bd10 2024-01-15T18:17:11.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa320a00640 1 -- 172.21.15.27:0/3405155817 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911176189 0 0) 0x7fa32406c380 con 0x7fa324160bc0 2024-01-15T18:17:11.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa320a00640 1 -- 172.21.15.27:0/3405155817 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3240cc610 con 0x7fa324160bc0 2024-01-15T18:17:11.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa320a00640 1 -- 172.21.15.27:0/3405155817 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1414480388 0 0) 0x7fa304003530 con 0x7fa32406bd10 2024-01-15T18:17:11.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa320a00640 1 -- 172.21.15.27:0/3405155817 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa3240cdab0 msgr2=0x7fa3240b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:11.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa320a00640 1 --2- 172.21.15.27:0/3405155817 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa3240cdab0 0x7fa3240b1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:11.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa320a00640 1 -- 172.21.15.27:0/3405155817 >> v1:172.21.15.93:6789/0 conn(0x7fa324160bc0 legacy=0x7fa32406b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:11.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa320a00640 1 -- 172.21.15.27:0/3405155817 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32406c660 con 0x7fa32406bd10 2024-01-15T18:17:11.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa320a00640 1 -- 172.21.15.27:0/3405155817 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fa31c001a00 con 0x7fa32406bd10 2024-01-15T18:17:11.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa320a00640 1 -- 172.21.15.27:0/3405155817 >> v1:172.21.15.27:6789/0 conn(0x7fa32406bd10 legacy=0x7fa3240cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:11.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa320a00640 1 --2- 172.21.15.27:0/3405155817 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa304003b40 0x7fa304003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:11.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa320a00640 1 --2- 172.21.15.27:0/3405155817 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa3240cdab0 0x7fa3040047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:11.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa320a00640 1 --2- 172.21.15.27:0/3405155817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa304004d10 0x7fa304005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:11.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa320a00640 1 -- 172.21.15.27:0/3405155817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa324160fa0 con 0x7fa304004d10 2024-01-15T18:17:11.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa320a00640 1 -- 172.21.15.27:0/3405155817 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30400ba40 con 0x7fa304003b40 2024-01-15T18:17:11.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa320a00640 1 -- 172.21.15.27:0/3405155817 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa30400bc00 con 0x7fa3240cdab0 2024-01-15T18:17:11.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa322800640 1 --2- 172.21.15.27:0/3405155817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa304004d10 0x7fa304005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:11.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa321e00640 1 --2- 172.21.15.27:0/3405155817 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa304003b40 0x7fa304003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:11.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa321400640 1 --2- 172.21.15.27:0/3405155817 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa3240cdab0 0x7fa3040047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:11.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa322800640 1 -- 172.21.15.27:0/3405155817 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa3240cdab0 msgr2=0x7fa3040047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:11.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa322800640 1 --2- 172.21.15.27:0/3405155817 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa3240cdab0 0x7fa3040047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:11.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa322800640 1 -- 172.21.15.27:0/3405155817 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa304003b40 msgr2=0x7fa304003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:11.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa322800640 1 --2- 172.21.15.27:0/3405155817 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa304003b40 0x7fa304003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:11.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.231+0000 7fa322800640 1 -- 172.21.15.27:0/3405155817 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa304003530 con 0x7fa304004d10 2024-01-15T18:17:11.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.235+0000 7fa322800640 1 --2- 172.21.15.27:0/3405155817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa304004d10 0x7fa304005190 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fa31c00a710 tx=0x7fa31c01f960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da208bab34e4924f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:11.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.235+0000 7fa320a00640 1 -- 172.21.15.27:0/3405155817 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa31c0ac610 con 0x7fa304004d10 2024-01-15T18:17:11.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.235+0000 7fa320a00640 1 -- 172.21.15.27:0/3405155817 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa31c0ac770 con 0x7fa304004d10 2024-01-15T18:17:11.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.235+0000 7fa320a00640 1 -- 172.21.15.27:0/3405155817 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa31c0acb60 con 0x7fa304004d10 2024-01-15T18:17:11.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.235+0000 7fa329000640 1 -- 172.21.15.27:0/3405155817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa304004d10 msgr2=0x7fa304005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:11.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.235+0000 7fa329000640 1 --2- 172.21.15.27:0/3405155817 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa304004d10 0x7fa304005190 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fa31c00a710 tx=0x7fa31c01f960 comp rx=0 tx=0).stop 2024-01-15T18:17:11.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.235+0000 7fa322800640 1 -- 172.21.15.27:0/3405155817 reap_dead start 2024-01-15T18:17:11.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.235+0000 7fa329000640 1 -- 172.21.15.27:0/3405155817 shutdown_connections 2024-01-15T18:17:11.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.235+0000 7fa329000640 1 -- 172.21.15.27:0/3405155817 >> 172.21.15.27:0/3405155817 conn(0x7fa324099a10 msgr2=0x7fa324099e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:11.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.235+0000 7fa329000640 1 -- 172.21.15.27:0/3405155817 shutdown_connections 2024-01-15T18:17:11.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.235+0000 7fa329000640 1 -- 172.21.15.27:0/3405155817 wait complete. 2024-01-15T18:17:11.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.239+0000 7fa329000640 1 Processor -- start 2024-01-15T18:17:11.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.239+0000 7fa329000640 1 -- start start 2024-01-15T18:17:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.239+0000 7fa329000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa304004d10 0x7fa3241f2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.239+0000 7fa329000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa3240c4cc0 0x7fa3240c5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.239+0000 7fa329000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa32415ac90 0x7fa32415b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.239+0000 7fa329000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32406c1a0 con 0x7fa32415ac90 2024-01-15T18:17:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.239+0000 7fa329000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3240c5680 con 0x7fa3240c4cc0 2024-01-15T18:17:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.239+0000 7fa329000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa32415b950 con 0x7fa304004d10 2024-01-15T18:17:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.239+0000 7fa322800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa32415ac90 0x7fa32415b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.239+0000 7fa322800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa32415ac90 0x7fa32415b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48594/0 (socket says 172.21.15.27:48594) 2024-01-15T18:17:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.239+0000 7fa322800640 1 -- 172.21.15.27:0/403428867 learned_addr learned my addr 172.21.15.27:0/403428867 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:11.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.239+0000 7fa321e00640 1 --2- 172.21.15.27:0/403428867 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa304004d10 0x7fa3241f2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:11.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.239+0000 7fa321400640 1 --2- 172.21.15.27:0/403428867 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa3240c4cc0 0x7fa3240c5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:11.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.239+0000 7fa322800640 1 -- 172.21.15.27:0/403428867 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa304004d10 msgr2=0x7fa3241f2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:11.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.239+0000 7fa322800640 1 --2- 172.21.15.27:0/403428867 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa304004d10 0x7fa3241f2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:11.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.239+0000 7fa322800640 1 -- 172.21.15.27:0/403428867 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa3240c4cc0 msgr2=0x7fa3240c5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:11.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.239+0000 7fa322800640 1 --2- 172.21.15.27:0/403428867 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa3240c4cc0 0x7fa3240c5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:11.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.239+0000 7fa322800640 1 -- 172.21.15.27:0/403428867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32415bc30 con 0x7fa32415ac90 2024-01-15T18:17:11.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.239+0000 7fa322800640 1 --2- 172.21.15.27:0/403428867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa32415ac90 0x7fa32415b110 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fa31c00a710 tx=0x7fa304005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:11.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.239+0000 7fa31a000640 1 -- 172.21.15.27:0/403428867 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa31c0ac610 con 0x7fa32415ac90 2024-01-15T18:17:11.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.239+0000 7fa31a000640 1 -- 172.21.15.27:0/403428867 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa31c0011d0 con 0x7fa32415ac90 2024-01-15T18:17:11.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.239+0000 7fa31a000640 1 -- 172.21.15.27:0/403428867 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa31c004d70 con 0x7fa32415ac90 2024-01-15T18:17:11.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.239+0000 7fa329000640 1 -- 172.21.15.27:0/403428867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa32415bf10 con 0x7fa32415ac90 2024-01-15T18:17:11.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.239+0000 7fa329000640 1 -- 172.21.15.27:0/403428867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa32415c4d0 con 0x7fa32415ac90 2024-01-15T18:17:11.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.243+0000 7fa329000640 1 -- 172.21.15.27:0/403428867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa32404e850 con 0x7fa32415ac90 2024-01-15T18:17:11.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.243+0000 7fa31a000640 1 -- 172.21.15.27:0/403428867 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7fa31c0be070 con 0x7fa32415ac90 2024-01-15T18:17:11.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.243+0000 7fa31a000640 1 --2- 172.21.15.27:0/403428867 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa2fc067730 0x7fa2fc069bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:11.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.243+0000 7fa31a000640 1 -- 172.21.15.27:0/403428867 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5055+0+0 (secure 0 0 0) 0x7fa31c120f80 con 0x7fa32415ac90 2024-01-15T18:17:11.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.243+0000 7fa321e00640 1 --2- 172.21.15.27:0/403428867 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa2fc067730 0x7fa2fc069bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:11.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.247+0000 7fa321e00640 1 --2- 172.21.15.27:0/403428867 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa2fc067730 0x7fa2fc069bf0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fa314007fd0 tx=0x7fa314009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:11.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.247+0000 7fa31a000640 1 -- 172.21.15.27:0/403428867 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa31c0efd60 con 0x7fa32415ac90 2024-01-15T18:17:11.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.407+0000 7fa329000640 1 -- 172.21.15.27:0/403428867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa32406ef60 con 0x7fa32415ac90 2024-01-15T18:17:11.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.411+0000 7fa31a000640 1 -- 172.21.15.27:0/403428867 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa31c0f3380 con 0x7fa32415ac90 2024-01-15T18:17:11.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.415+0000 7fa329000640 1 -- 172.21.15.27:0/403428867 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa2fc067730 msgr2=0x7fa2fc069bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:11.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.415+0000 7fa329000640 1 --2- 172.21.15.27:0/403428867 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa2fc067730 0x7fa2fc069bf0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fa314007fd0 tx=0x7fa314009040 comp rx=0 tx=0).stop 2024-01-15T18:17:11.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.415+0000 7fa329000640 1 -- 172.21.15.27:0/403428867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa32415ac90 msgr2=0x7fa32415b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:11.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.415+0000 7fa329000640 1 --2- 172.21.15.27:0/403428867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa32415ac90 0x7fa32415b110 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fa31c00a710 tx=0x7fa304005800 comp rx=0 tx=0).stop 2024-01-15T18:17:11.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.415+0000 7fa329000640 1 -- 172.21.15.27:0/403428867 shutdown_connections 2024-01-15T18:17:11.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.415+0000 7fa329000640 1 --2- 172.21.15.27:0/403428867 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa2fc067730 0x7fa2fc069bf0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:11.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.415+0000 7fa329000640 1 --2- 172.21.15.27:0/403428867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa32415ac90 0x7fa32415b110 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:11.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.415+0000 7fa329000640 1 --2- 172.21.15.27:0/403428867 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa3240c4cc0 0x7fa3240c5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:11.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.415+0000 7fa329000640 1 --2- 172.21.15.27:0/403428867 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa304004d10 0x7fa3241f2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:11.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.415+0000 7fa329000640 1 -- 172.21.15.27:0/403428867 >> 172.21.15.27:0/403428867 conn(0x7fa324099a10 msgr2=0x7fa32406e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:11.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.415+0000 7fa329000640 1 -- 172.21.15.27:0/403428867 shutdown_connections 2024-01-15T18:17:11.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.415+0000 7fa329000640 1 -- 172.21.15.27:0/403428867 wait complete. 2024-01-15T18:17:11.432 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T18:17:11.432 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:17:11.518 DEBUG:teuthology.orchestra.run.smithi093:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:17:11.590 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T18:17:11.591 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:17:11.591 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:17:11.600 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354580k,nr_inodes=4088645,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9249 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:17:11.600 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:17:11.600 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:17:11.610 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354484k,nr_inodes=4088621,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8403 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:17:11.610 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:17:11.610 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:17:11.652 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354580k,nr_inodes=4088645,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9249 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:17:11.652 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:17:11.652 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:17:11.661 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354484k,nr_inodes=4088621,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8403 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:17:11.661 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-15T18:17:11.661 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:17:11.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.839+0000 7f17bce00640 1 Processor -- start 2024-01-15T18:17:11.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.839+0000 7f17bce00640 1 -- start start 2024-01-15T18:17:11.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.839+0000 7f17bce00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f17b815e730 0x7f17b80cd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:11.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.839+0000 7f17bce00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b80ce920 con 0x7f17b80cdd10 2024-01-15T18:17:11.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.839+0000 7f17bce00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b80ce640 con 0x7f17b806db80 2024-01-15T18:17:11.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.839+0000 7f17bce00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17b80cc940 con 0x7f17b815e730 2024-01-15T18:17:11.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.839+0000 7f17b6800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f17b80cdd10 0x7f17b8070c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47120/0 (socket says 172.21.15.27:47120) 2024-01-15T18:17:11.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.839+0000 7f17b6800640 1 -- 172.21.15.27:0/2490966388 learned_addr learned my addr 172.21.15.27:0/2490966388 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:11.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.839+0000 7f17b5400640 1 --2- 172.21.15.27:0/2490966388 >> v2:172.21.15.93:3300/0 conn(0x7f17b815e730 0x7f17b80cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:11.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.839+0000 7f17b4a00640 1 -- 172.21.15.27:0/2490966388 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013914107 0 0) 0x7f17b80ce920 con 0x7f17b80cdd10 2024-01-15T18:17:11.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.839+0000 7f17b4a00640 1 -- 172.21.15.27:0/2490966388 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1798003530 con 0x7f17b80cdd10 2024-01-15T18:17:11.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.839+0000 7f17b4a00640 1 -- 172.21.15.27:0/2490966388 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 343685353 0 0) 0x7f1798003530 con 0x7f17b80cdd10 2024-01-15T18:17:11.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.839+0000 7f17b5400640 1 -- 172.21.15.27:0/2490966388 >> v1:172.21.15.93:6789/0 conn(0x7f17b806db80 legacy=0x7f17b806df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:11.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.839+0000 7f17b5400640 1 -- 172.21.15.27:0/2490966388 >> v1:172.21.15.27:6789/0 conn(0x7f17b80cdd10 legacy=0x7f17b8070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:11.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.839+0000 7f17b5400640 1 -- 172.21.15.27:0/2490966388 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b80ccc20 con 0x7f17b815e730 2024-01-15T18:17:11.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.843+0000 7f17b5400640 1 --2- 172.21.15.27:0/2490966388 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17b815e730 0x7f17b80cd420 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f17a00097c0 tx=0x7f17a001c220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f09860a205829dc1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:11.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.843+0000 7f17b4a00640 1 -- 172.21.15.27:0/2490966388 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b0001a00 con 0x7f17b815e730 2024-01-15T18:17:11.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.843+0000 7f17b4a00640 1 -- 172.21.15.27:0/2490966388 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17a00a7050 con 0x7f17b815e730 2024-01-15T18:17:11.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.843+0000 7f17b4a00640 1 -- 172.21.15.27:0/2490966388 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a00ab620 con 0x7f17b815e730 2024-01-15T18:17:11.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.843+0000 7f17bce00640 1 -- 172.21.15.27:0/2490966388 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17b815e730 msgr2=0x7f17b80cd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:11.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.843+0000 7f17bce00640 1 --2- 172.21.15.27:0/2490966388 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17b815e730 0x7f17b80cd420 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f17a00097c0 tx=0x7f17a001c220 comp rx=0 tx=0).stop 2024-01-15T18:17:11.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.843+0000 7f17bce00640 1 -- 172.21.15.27:0/2490966388 shutdown_connections 2024-01-15T18:17:11.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.843+0000 7f17bce00640 1 --2- 172.21.15.27:0/2490966388 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17b815e730 0x7f17b80cd420 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:11.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.843+0000 7f17bce00640 1 -- 172.21.15.27:0/2490966388 >> 172.21.15.27:0/2490966388 conn(0x7f17b80cec60 msgr2=0x7f17b80cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:11.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.843+0000 7f17bce00640 1 -- 172.21.15.27:0/2490966388 shutdown_connections 2024-01-15T18:17:11.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.843+0000 7f17bce00640 1 -- 172.21.15.27:0/2490966388 wait complete. 2024-01-15T18:17:11.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.847+0000 7f17bce00640 1 Processor -- start 2024-01-15T18:17:11.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.847+0000 7f17bce00640 1 -- start start 2024-01-15T18:17:11.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.847+0000 7f17bce00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17b806db80 0x7f17b80c6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:11.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.847+0000 7f17bce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17b80cdd10 0x7f17b8155e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:11.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.847+0000 7f17bce00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17b815e730 0x7f17b814fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:11.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.847+0000 7f17bce00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b81563d0 con 0x7f17b80cdd10 2024-01-15T18:17:11.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.847+0000 7f17bce00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b8150860 con 0x7f17b815e730 2024-01-15T18:17:11.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.847+0000 7f17bce00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17b80c6dd0 con 0x7f17b806db80 2024-01-15T18:17:11.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.847+0000 7f17b5400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17b80cdd10 0x7f17b8155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:11.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.847+0000 7f17b5400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17b80cdd10 0x7f17b8155e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48610/0 (socket says 172.21.15.27:48610) 2024-01-15T18:17:11.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.847+0000 7f17b5400640 1 -- 172.21.15.27:0/3442982462 learned_addr learned my addr 172.21.15.27:0/3442982462 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:11.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.847+0000 7f17b6800640 1 --2- 172.21.15.27:0/3442982462 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17b815e730 0x7f17b814fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:11.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.847+0000 7f17b5e00640 1 --2- 172.21.15.27:0/3442982462 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17b806db80 0x7f17b80c6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:11.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.847+0000 7f17b5400640 1 -- 172.21.15.27:0/3442982462 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17b806db80 msgr2=0x7f17b80c6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:11.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.847+0000 7f17b5400640 1 --2- 172.21.15.27:0/3442982462 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17b806db80 0x7f17b80c6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:11.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.847+0000 7f17b5400640 1 -- 172.21.15.27:0/3442982462 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17b815e730 msgr2=0x7f17b814fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:11.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.847+0000 7f17b5400640 1 --2- 172.21.15.27:0/3442982462 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17b815e730 0x7f17b814fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:11.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.847+0000 7f17b5400640 1 -- 172.21.15.27:0/3442982462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a0009420 con 0x7f17b80cdd10 2024-01-15T18:17:11.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.847+0000 7f17b5400640 1 --2- 172.21.15.27:0/3442982462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17b80cdd10 0x7f17b8155e90 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f17a00097c0 tx=0x7f17a001aa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:11.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.847+0000 7f17ae000640 1 -- 172.21.15.27:0/3442982462 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a00abd80 con 0x7f17b80cdd10 2024-01-15T18:17:11.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.847+0000 7f17ae000640 1 -- 172.21.15.27:0/3442982462 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17a00a4e90 con 0x7f17b80cdd10 2024-01-15T18:17:11.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.847+0000 7f17ae000640 1 -- 172.21.15.27:0/3442982462 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a0004260 con 0x7f17b80cdd10 2024-01-15T18:17:11.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.847+0000 7f17bce00640 1 -- 172.21.15.27:0/3442982462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17b8154e70 con 0x7f17b80cdd10 2024-01-15T18:17:11.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.847+0000 7f17bce00640 1 -- 172.21.15.27:0/3442982462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17b8154820 con 0x7f17b80cdd10 2024-01-15T18:17:11.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.851+0000 7f17bce00640 1 -- 172.21.15.27:0/3442982462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17b8155000 con 0x7f17b80cdd10 2024-01-15T18:17:11.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.851+0000 7f17ae000640 1 -- 172.21.15.27:0/3442982462 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f17a00043c0 con 0x7f17b80cdd10 2024-01-15T18:17:11.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.851+0000 7f17ae000640 1 --2- 172.21.15.27:0/3442982462 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1790067780 0x7f1790069c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:11.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.851+0000 7f17ae000640 1 -- 172.21.15.27:0/3442982462 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f17a011cc90 con 0x7f17b80cdd10 2024-01-15T18:17:11.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.851+0000 7f17b5e00640 1 --2- 172.21.15.27:0/3442982462 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1790067780 0x7f1790069c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:11.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.851+0000 7f17b5e00640 1 --2- 172.21.15.27:0/3442982462 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1790067780 0x7f1790069c40 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f17a8000c10 tx=0x7f17a8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:11.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:11.855+0000 7f17ae000640 1 -- 172.21.15.27:0/3442982462 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f17a00eb9d0 con 0x7f17b80cdd10 2024-01-15T18:17:12.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.043+0000 7f17bce00640 1 -- 172.21.15.27:0/3442982462 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17b804e850 con 0x7f17b80cdd10 2024-01-15T18:17:12.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.043+0000 7f17ae000640 1 -- 172.21.15.27:0/3442982462 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f17a00bbe90 con 0x7f17b80cdd10 2024-01-15T18:17:12.043 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:17:12.043 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":60,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8167,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/3555344514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":3555344514},{"type":"v1","addr":"172.21.15.93:6837","nonce":3555344514}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":8376,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6833/2584136338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":2584136338},{"type":"v1","addr":"172.21.15.27:6833","nonce":2584136338}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":8385,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/2635542478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6834","nonce":2635542478},{"type":"v1","addr":"172.21.15.27:6835","nonce":2635542478}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52},{"gid":8388,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/2822170069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":2822170069},{"type":"v1","addr":"172.21.15.93:6836","nonce":2822170069}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52}],"filesystems":[]} 2024-01-15T18:17:12.043 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 60 2024-01-15T18:17:12.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.047+0000 7f17bce00640 1 -- 172.21.15.27:0/3442982462 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1790067780 msgr2=0x7f1790069c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:12.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.047+0000 7f17bce00640 1 --2- 172.21.15.27:0/3442982462 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1790067780 0x7f1790069c40 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f17a8000c10 tx=0x7f17a8009040 comp rx=0 tx=0).stop 2024-01-15T18:17:12.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.047+0000 7f17bce00640 1 -- 172.21.15.27:0/3442982462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17b80cdd10 msgr2=0x7f17b8155e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:12.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.047+0000 7f17bce00640 1 --2- 172.21.15.27:0/3442982462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17b80cdd10 0x7f17b8155e90 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f17a00097c0 tx=0x7f17a001aa60 comp rx=0 tx=0).stop 2024-01-15T18:17:12.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.047+0000 7f17bce00640 1 -- 172.21.15.27:0/3442982462 shutdown_connections 2024-01-15T18:17:12.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.047+0000 7f17bce00640 1 --2- 172.21.15.27:0/3442982462 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1790067780 0x7f1790069c40 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:12.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.047+0000 7f17bce00640 1 --2- 172.21.15.27:0/3442982462 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f17b815e730 0x7f17b814fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:12.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.047+0000 7f17bce00640 1 --2- 172.21.15.27:0/3442982462 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17b80cdd10 0x7f17b8155e90 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:12.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.047+0000 7f17bce00640 1 --2- 172.21.15.27:0/3442982462 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f17b806db80 0x7f17b80c6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:12.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.047+0000 7f17bce00640 1 -- 172.21.15.27:0/3442982462 >> 172.21.15.27:0/3442982462 conn(0x7f17b80cec60 msgr2=0x7f17b80ca410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:12.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.047+0000 7f17bce00640 1 -- 172.21.15.27:0/3442982462 shutdown_connections 2024-01-15T18:17:12.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.047+0000 7f17bce00640 1 -- 172.21.15.27:0/3442982462 wait complete. 2024-01-15T18:17:12.066 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-15T18:17:12.067 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-15T18:17:12.067 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-15T18:17:12.067 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:17:12.067 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-15T18:17:12.067 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-15T18:17:12.068 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-15T18:17:12.068 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:17:12.068 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-15T18:17:12.068 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-15T18:17:12.068 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-15T18:17:12.069 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:17:12.069 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-15T18:17:12.069 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-15T18:17:12.069 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-15T18:17:12.069 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:17:12.079 INFO:tasks.ceph.mds.a:Stopped 2024-01-15T18:17:12.080 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-15T18:17:12.082 INFO:tasks.ceph.mds.c:Stopped 2024-01-15T18:17:12.082 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-15T18:17:12.084 INFO:tasks.ceph.mds.b:Stopped 2024-01-15T18:17:12.085 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-15T18:17:12.085 INFO:tasks.ceph.mds.d:Stopped 2024-01-15T18:17:12.086 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-15T18:17:12.126 INFO:tasks.ceph.mds.a:Started 2024-01-15T18:17:12.126 DEBUG:teuthology.parallel:result is None 2024-01-15T18:17:12.129 INFO:tasks.ceph.mds.c:Started 2024-01-15T18:17:12.129 DEBUG:teuthology.parallel:result is None 2024-01-15T18:17:12.131 INFO:tasks.ceph.mds.b:Started 2024-01-15T18:17:12.131 DEBUG:teuthology.parallel:result is None 2024-01-15T18:17:12.133 INFO:tasks.ceph.mds.d:Started 2024-01-15T18:17:12.133 DEBUG:teuthology.parallel:result is None 2024-01-15T18:17:12.134 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-15T18:17:12.212 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-15T18:17:12.213 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-15T18:17:12.214 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-01-15T18:17:12.215+0000 7fedd1400640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-01-15T18:17:12.215 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-01-15T18:17:12.215+0000 7fedd1400640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6834/0: (98) Address already in use 2024-01-15T18:17:12.222 INFO:tasks.ceph.mds.b.smithi093.stdout:starting mds.b at 2024-01-15T18:17:12.223 INFO:tasks.ceph.mds.d.smithi093.stdout:starting mds.d at 2024-01-15T18:17:12.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.283+0000 7fb218c00640 1 Processor -- start 2024-01-15T18:17:12.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.283+0000 7fb218c00640 1 -- start start 2024-01-15T18:17:12.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.283+0000 7fb218c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb214153c80 0x7fb214071020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:12.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.283+0000 7fb218c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fb21406ca90 0x7fb21406cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:12.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.283+0000 7fb218c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb214159c10 0x7fb21415a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:12.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.283+0000 7fb218c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb21406d430 con 0x7fb214159c10 2024-01-15T18:17:12.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.283+0000 7fb218c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb214070b90 con 0x7fb21406ca90 2024-01-15T18:17:12.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.283+0000 7fb218c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21415a980 con 0x7fb214153c80 2024-01-15T18:17:12.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.283+0000 7fb212800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb214159c10 0x7fb21415a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:12.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.283+0000 7fb212800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb214159c10 0x7fb21415a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48650/0 (socket says 172.21.15.27:48650) 2024-01-15T18:17:12.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.283+0000 7fb212800640 1 -- 172.21.15.27:0/3392190371 learned_addr learned my addr 172.21.15.27:0/3392190371 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:12.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.283+0000 7fb211e00640 1 --2- 172.21.15.27:0/3392190371 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb214153c80 0x7fb214071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:12.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.283+0000 7fb211400640 1 --2- 172.21.15.27:0/3392190371 >> v2:172.21.15.93:3300/0 conn(0x7fb21406ca90 0x7fb21406cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:12.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.283+0000 7fb212800640 1 -- 172.21.15.27:0/3392190371 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb214153c80 msgr2=0x7fb214071020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:12.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.283+0000 7fb212800640 1 --2- 172.21.15.27:0/3392190371 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb214153c80 0x7fb214071020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:12.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.283+0000 7fb212800640 1 -- 172.21.15.27:0/3392190371 >> v2:172.21.15.93:3300/0 conn(0x7fb21406ca90 msgr2=0x7fb21406cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:12.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.283+0000 7fb212800640 1 --2- 172.21.15.27:0/3392190371 >> v2:172.21.15.93:3300/0 conn(0x7fb21406ca90 0x7fb21406cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:12.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.283+0000 7fb212800640 1 -- 172.21.15.27:0/3392190371 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb214070260 con 0x7fb214159c10 2024-01-15T18:17:12.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.287+0000 7fb212800640 1 --2- 172.21.15.27:0/3392190371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb214159c10 0x7fb21415a080 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fb2140cfca0 tx=0x7fb20c01da10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82484afb27466ebc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:12.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.287+0000 7fb210a00640 1 -- 172.21.15.27:0/3392190371 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20c00a2d0 con 0x7fb214159c10 2024-01-15T18:17:12.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.287+0000 7fb210a00640 1 -- 172.21.15.27:0/3392190371 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20c0a6810 con 0x7fb214159c10 2024-01-15T18:17:12.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.287+0000 7fb210a00640 1 -- 172.21.15.27:0/3392190371 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20c0a6ba0 con 0x7fb214159c10 2024-01-15T18:17:12.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.287+0000 7fb218c00640 1 -- 172.21.15.27:0/3392190371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb214159c10 msgr2=0x7fb21415a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:12.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.287+0000 7fb218c00640 1 --2- 172.21.15.27:0/3392190371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb214159c10 0x7fb21415a080 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fb2140cfca0 tx=0x7fb20c01da10 comp rx=0 tx=0).stop 2024-01-15T18:17:12.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.287+0000 7fb218c00640 1 -- 172.21.15.27:0/3392190371 shutdown_connections 2024-01-15T18:17:12.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.287+0000 7fb218c00640 1 --2- 172.21.15.27:0/3392190371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb214159c10 0x7fb21415a080 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:12.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.287+0000 7fb218c00640 1 --2- 172.21.15.27:0/3392190371 >> v2:172.21.15.93:3300/0 conn(0x7fb21406ca90 0x7fb21406cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:12.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.287+0000 7fb218c00640 1 --2- 172.21.15.27:0/3392190371 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb214153c80 0x7fb214071020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:12.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.287+0000 7fb218c00640 1 -- 172.21.15.27:0/3392190371 >> 172.21.15.27:0/3392190371 conn(0x7fb2140b0850 msgr2=0x7fb2140b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:12.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.287+0000 7fb218c00640 1 -- 172.21.15.27:0/3392190371 shutdown_connections 2024-01-15T18:17:12.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.287+0000 7fb218c00640 1 -- 172.21.15.27:0/3392190371 wait complete. 2024-01-15T18:17:12.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb218c00640 1 Processor -- start 2024-01-15T18:17:12.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb218c00640 1 -- start start 2024-01-15T18:17:12.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb218c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb21406ca90 0x7fb2141f5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:12.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb218c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb214153c80 0x7fb214154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:12.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb218c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb214159c10 0x7fb214147370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:12.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb218c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21413f560 con 0x7fb214159c10 2024-01-15T18:17:12.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb218c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb214156c80 con 0x7fb214153c80 2024-01-15T18:17:12.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb218c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21413e1e0 con 0x7fb21406ca90 2024-01-15T18:17:12.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb212800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb214159c10 0x7fb214147370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:12.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb211e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb21406ca90 0x7fb2141f5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:12.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb212800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb214159c10 0x7fb214147370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48656/0 (socket says 172.21.15.27:48656) 2024-01-15T18:17:12.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb212800640 1 -- 172.21.15.27:0/2585404987 learned_addr learned my addr 172.21.15.27:0/2585404987 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb211400640 1 --2- 172.21.15.27:0/2585404987 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb214153c80 0x7fb214154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb211e00640 1 -- 172.21.15.27:0/2585404987 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb214153c80 msgr2=0x7fb214154060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb211e00640 1 --2- 172.21.15.27:0/2585404987 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb214153c80 0x7fb214154060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb211e00640 1 -- 172.21.15.27:0/2585404987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb214159c10 msgr2=0x7fb214147370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb211e00640 1 --2- 172.21.15.27:0/2585404987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb214159c10 0x7fb214147370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb211e00640 1 -- 172.21.15.27:0/2585404987 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20c00ac40 con 0x7fb21406ca90 2024-01-15T18:17:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb212800640 1 --2- 172.21.15.27:0/2585404987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb214159c10 0x7fb214147370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:17:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb211e00640 1 --2- 172.21.15.27:0/2585404987 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb21406ca90 0x7fb2141f5a00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb20400e870 tx=0x7fb20400ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb20a000640 1 -- 172.21.15.27:0/2585404987 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2040095d0 con 0x7fb21406ca90 2024-01-15T18:17:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb20a000640 1 -- 172.21.15.27:0/2585404987 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20400c940 con 0x7fb21406ca90 2024-01-15T18:17:12.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb20a000640 1 -- 172.21.15.27:0/2585404987 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20400ccb0 con 0x7fb21406ca90 2024-01-15T18:17:12.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb218c00640 1 -- 172.21.15.27:0/2585404987 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb21413e4c0 con 0x7fb21406ca90 2024-01-15T18:17:12.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb218c00640 1 -- 172.21.15.27:0/2585404987 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb21413ea80 con 0x7fb21406ca90 2024-01-15T18:17:12.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.291+0000 7fb218c00640 1 -- 172.21.15.27:0/2585404987 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb21404e850 con 0x7fb21406ca90 2024-01-15T18:17:12.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.295+0000 7fb20a000640 1 -- 172.21.15.27:0/2585404987 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7fb204013470 con 0x7fb21406ca90 2024-01-15T18:17:12.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.295+0000 7fb20a000640 1 --2- 172.21.15.27:0/2585404987 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb1ec067700 0x7fb1ec069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:12.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.295+0000 7fb20a000640 1 -- 172.21.15.27:0/2585404987 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5055+0+0 (secure 0 0 0) 0x7fb204083280 con 0x7fb21406ca90 2024-01-15T18:17:12.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.295+0000 7fb211400640 1 --2- 172.21.15.27:0/2585404987 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb1ec067700 0x7fb1ec069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:12.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.295+0000 7fb211400640 1 --2- 172.21.15.27:0/2585404987 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb1ec067700 0x7fb1ec069bc0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb21415cc40 tx=0x7fb1fc009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:12.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.299+0000 7fb20a000640 1 -- 172.21.15.27:0/2585404987 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb204051d00 con 0x7fb21406ca90 2024-01-15T18:17:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:12.471+0000 7fb218c00640 1 -- 172.21.15.27:0/2585404987 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fb21406d650 con 0x7fb21406ca90 2024-01-15T18:17:13.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.071+0000 7fb20a000640 1 -- 172.21.15.27:0/2585404987 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v91) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb204017080 con 0x7fb21406ca90 2024-01-15T18:17:13.071 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-01-15T18:17:13.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.075+0000 7fb218c00640 1 -- 172.21.15.27:0/2585404987 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb1ec067700 msgr2=0x7fb1ec069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:13.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.075+0000 7fb218c00640 1 --2- 172.21.15.27:0/2585404987 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb1ec067700 0x7fb1ec069bc0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb21415cc40 tx=0x7fb1fc009190 comp rx=0 tx=0).stop 2024-01-15T18:17:13.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.075+0000 7fb218c00640 1 -- 172.21.15.27:0/2585404987 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb21406ca90 msgr2=0x7fb2141f5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:13.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.075+0000 7fb218c00640 1 --2- 172.21.15.27:0/2585404987 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb21406ca90 0x7fb2141f5a00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb20400e870 tx=0x7fb20400ed30 comp rx=0 tx=0).stop 2024-01-15T18:17:13.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.075+0000 7fb218c00640 1 -- 172.21.15.27:0/2585404987 shutdown_connections 2024-01-15T18:17:13.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.075+0000 7fb218c00640 1 --2- 172.21.15.27:0/2585404987 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb1ec067700 0x7fb1ec069bc0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.075+0000 7fb218c00640 1 --2- 172.21.15.27:0/2585404987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb214159c10 0x7fb214147370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.075+0000 7fb218c00640 1 --2- 172.21.15.27:0/2585404987 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb214153c80 0x7fb214154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.075+0000 7fb218c00640 1 --2- 172.21.15.27:0/2585404987 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb21406ca90 0x7fb2141f5a00 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.075+0000 7fb218c00640 1 -- 172.21.15.27:0/2585404987 >> 172.21.15.27:0/2585404987 conn(0x7fb2140b0850 msgr2=0x7fb2140cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:13.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.075+0000 7fb218c00640 1 -- 172.21.15.27:0/2585404987 shutdown_connections 2024-01-15T18:17:13.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.075+0000 7fb218c00640 1 -- 172.21.15.27:0/2585404987 wait complete. 2024-01-15T18:17:13.097 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-15T18:17:13.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f6107800640 1 Processor -- start 2024-01-15T18:17:13.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f6107800640 1 -- start start 2024-01-15T18:17:13.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f6107800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61001614d0 0x7f61000b4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:13.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f6107800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f6100072380 0x7f610006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:13.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f6107800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f610006a620 con 0x7f61001618b0 2024-01-15T18:17:13.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f6107800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f610006a900 con 0x7f6100072380 2024-01-15T18:17:13.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f6107800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f610006abe0 con 0x7f61001614d0 2024-01-15T18:17:13.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f60ffe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f61001618b0 0x7f6100071c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47130/0 (socket says 172.21.15.27:47130) 2024-01-15T18:17:13.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f60ffe00640 1 -- 172.21.15.27:0/1464483374 learned_addr learned my addr 172.21.15.27:0/1464483374 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:13.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f6104e00640 1 --2- 172.21.15.27:0/1464483374 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61001614d0 0x7f61000b4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:13.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f60ff400640 1 -- 172.21.15.27:0/1464483374 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 533434906 0 0) 0x7f610006a620 con 0x7f61001618b0 2024-01-15T18:17:13.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f60ff400640 1 -- 172.21.15.27:0/1464483374 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60e0003530 con 0x7f61001618b0 2024-01-15T18:17:13.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f6106200640 1 --2- 172.21.15.27:0/1464483374 >> v2:172.21.15.93:3300/0 conn(0x7f6100072380 0x7f610006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:13.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f60ff400640 1 -- 172.21.15.27:0/1464483374 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2763944487 0 0) 0x7f60e0003530 con 0x7f61001618b0 2024-01-15T18:17:13.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f60ff400640 1 -- 172.21.15.27:0/1464483374 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61001614d0 msgr2=0x7f61000b4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:13.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f60ff400640 1 --2- 172.21.15.27:0/1464483374 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61001614d0 0x7f61000b4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f60ff400640 1 -- 172.21.15.27:0/1464483374 >> v2:172.21.15.93:3300/0 conn(0x7f6100072380 msgr2=0x7f610006dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:13.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f60ff400640 1 --2- 172.21.15.27:0/1464483374 >> v2:172.21.15.93:3300/0 conn(0x7f6100072380 0x7f610006dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f60ff400640 1 -- 172.21.15.27:0/1464483374 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f610006aec0 con 0x7f61001618b0 2024-01-15T18:17:13.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f60ff400640 1 -- 172.21.15.27:0/1464483374 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f60e8001680 con 0x7f61001618b0 2024-01-15T18:17:13.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f6104e00640 1 --2- 172.21.15.27:0/1464483374 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61001614d0 0x7f61000b4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:17:13.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f60ff400640 1 -- 172.21.15.27:0/1464483374 >> v1:172.21.15.27:6789/0 conn(0x7f61001618b0 legacy=0x7f6100071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:13.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f60ff400640 1 --2- 172.21.15.27:0/1464483374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60e0003980 0x7f60e0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:13.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f60ff400640 1 --2- 172.21.15.27:0/1464483374 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61001614d0 0x7f60e00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:13.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f60ff400640 1 --2- 172.21.15.27:0/1464483374 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f60e0004d10 0x7f60e0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:13.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f60ff400640 1 -- 172.21.15.27:0/1464483374 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610006a900 con 0x7f60e0003980 2024-01-15T18:17:13.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f60ff400640 1 -- 172.21.15.27:0/1464483374 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610006abe0 con 0x7f60e0004d10 2024-01-15T18:17:13.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f60ff400640 1 -- 172.21.15.27:0/1464483374 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e000b990 con 0x7f61001614d0 2024-01-15T18:17:13.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f6104e00640 1 --2- 172.21.15.27:0/1464483374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60e0003980 0x7f60e0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:13.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f60ffe00640 1 --2- 172.21.15.27:0/1464483374 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61001614d0 0x7f60e00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:13.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f6106200640 1 --2- 172.21.15.27:0/1464483374 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f60e0004d10 0x7f60e0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:13.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f6104e00640 1 -- 172.21.15.27:0/1464483374 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61001614d0 msgr2=0x7f60e00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:13.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f6104e00640 1 --2- 172.21.15.27:0/1464483374 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61001614d0 0x7f60e00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f6104e00640 1 -- 172.21.15.27:0/1464483374 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f60e0004d10 msgr2=0x7f60e0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:13.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f6104e00640 1 --2- 172.21.15.27:0/1464483374 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f60e0004d10 0x7f60e0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f6104e00640 1 -- 172.21.15.27:0/1464483374 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f0009420 con 0x7f60e0003980 2024-01-15T18:17:13.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f6106200640 1 --2- 172.21.15.27:0/1464483374 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f60e0004d10 0x7f60e0005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:17:13.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.243+0000 7f60ffe00640 1 --2- 172.21.15.27:0/1464483374 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61001614d0 0x7f60e00047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:17:13.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.247+0000 7f6104e00640 1 --2- 172.21.15.27:0/1464483374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60e0003980 0x7f60e0003da0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009cb0 tx=0x7f60f001cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b10d6d9b8cfb0440 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:13.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.247+0000 7f60ff400640 1 -- 172.21.15.27:0/1464483374 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f00a5880 con 0x7f60e0003980 2024-01-15T18:17:13.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.247+0000 7f60ff400640 1 -- 172.21.15.27:0/1464483374 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f00a59e0 con 0x7f60e0003980 2024-01-15T18:17:13.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.247+0000 7f60ff400640 1 -- 172.21.15.27:0/1464483374 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f00a5d90 con 0x7f60e0003980 2024-01-15T18:17:13.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.247+0000 7f6107800640 1 -- 172.21.15.27:0/1464483374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60e0003980 msgr2=0x7f60e0003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:13.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.247+0000 7f6107800640 1 --2- 172.21.15.27:0/1464483374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60e0003980 0x7f60e0003da0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009cb0 tx=0x7f60f001cb60 comp rx=0 tx=0).stop 2024-01-15T18:17:13.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.247+0000 7f6106200640 1 -- 172.21.15.27:0/1464483374 reap_dead start 2024-01-15T18:17:13.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.247+0000 7f6107800640 1 -- 172.21.15.27:0/1464483374 shutdown_connections 2024-01-15T18:17:13.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.247+0000 7f6107800640 1 -- 172.21.15.27:0/1464483374 >> 172.21.15.27:0/1464483374 conn(0x7f61000b2360 msgr2=0x7f61000b2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:13.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.251+0000 7f6107800640 1 -- 172.21.15.27:0/1464483374 shutdown_connections 2024-01-15T18:17:13.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.251+0000 7f6107800640 1 -- 172.21.15.27:0/1464483374 wait complete. 2024-01-15T18:17:13.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.251+0000 7f6107800640 1 Processor -- start 2024-01-15T18:17:13.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.251+0000 7f6107800640 1 -- start start 2024-01-15T18:17:13.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.251+0000 7f6107800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60e0003980 0x7f610009df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:13.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.251+0000 7f6107800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f610015bf10 0x7f61000cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:13.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.251+0000 7f6107800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f610015b3c0 0x7f610015b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:13.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.251+0000 7f6107800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6100161f10 con 0x7f60e0003980 2024-01-15T18:17:13.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.251+0000 7f6107800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610009e470 con 0x7f610015bf10 2024-01-15T18:17:13.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.251+0000 7f6107800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f610015bd80 con 0x7f610015b3c0 2024-01-15T18:17:13.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.251+0000 7f6104e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60e0003980 0x7f610009df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:13.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.251+0000 7f6104e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60e0003980 0x7f610009df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48686/0 (socket says 172.21.15.27:48686) 2024-01-15T18:17:13.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.251+0000 7f6104e00640 1 -- 172.21.15.27:0/4090814673 learned_addr learned my addr 172.21.15.27:0/4090814673 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:13.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.251+0000 7f6106200640 1 --2- 172.21.15.27:0/4090814673 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f610015b3c0 0x7f610015b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:13.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.251+0000 7f60ffe00640 1 --2- 172.21.15.27:0/4090814673 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f610015bf10 0x7f61000cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:13.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.251+0000 7f6104e00640 1 -- 172.21.15.27:0/4090814673 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f610015b3c0 msgr2=0x7f610015b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:13.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.251+0000 7f6104e00640 1 --2- 172.21.15.27:0/4090814673 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f610015b3c0 0x7f610015b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.251+0000 7f6104e00640 1 -- 172.21.15.27:0/4090814673 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f610015bf10 msgr2=0x7f61000cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:13.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.251+0000 7f6104e00640 1 --2- 172.21.15.27:0/4090814673 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f610015bf10 0x7f61000cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.251+0000 7f6104e00640 1 -- 172.21.15.27:0/4090814673 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6100164ba0 con 0x7f60e0003980 2024-01-15T18:17:13.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.251+0000 7f6104e00640 1 --2- 172.21.15.27:0/4090814673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60e0003980 0x7f610009df30 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009cb0 tx=0x7f60e0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:13.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.255+0000 7f60fcc00640 1 -- 172.21.15.27:0/4090814673 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f00ad440 con 0x7f60e0003980 2024-01-15T18:17:13.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.255+0000 7f60fcc00640 1 -- 172.21.15.27:0/4090814673 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f00ad5a0 con 0x7f60e0003980 2024-01-15T18:17:13.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.255+0000 7f6107800640 1 -- 172.21.15.27:0/4090814673 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61001645e0 con 0x7f60e0003980 2024-01-15T18:17:13.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.255+0000 7f60fcc00640 1 -- 172.21.15.27:0/4090814673 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f00ad950 con 0x7f60e0003980 2024-01-15T18:17:13.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.255+0000 7f6107800640 1 -- 172.21.15.27:0/4090814673 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61000cf830 con 0x7f60e0003980 2024-01-15T18:17:13.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.255+0000 7f6107800640 1 -- 172.21.15.27:0/4090814673 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6100067a10 con 0x7f60e0003980 2024-01-15T18:17:13.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.255+0000 7f60fcc00640 1 -- 172.21.15.27:0/4090814673 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f60f00b4040 con 0x7f60e0003980 2024-01-15T18:17:13.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.255+0000 7f60fcc00640 1 --2- 172.21.15.27:0/4090814673 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f60d8067730 0x7f60d8069bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:13.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.255+0000 7f60fcc00640 1 -- 172.21.15.27:0/4090814673 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4969+0+0 (secure 0 0 0) 0x7f60f011d400 con 0x7f60e0003980 2024-01-15T18:17:13.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.255+0000 7f60ffe00640 1 --2- 172.21.15.27:0/4090814673 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f60d8067730 0x7f60d8069bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:13.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.259+0000 7f60ffe00640 1 --2- 172.21.15.27:0/4090814673 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f60d8067730 0x7f60d8069bf0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f61001643d0 tx=0x7f60e8007040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:13.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.259+0000 7f60fcc00640 1 -- 172.21.15.27:0/4090814673 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f60f00ec260 con 0x7f60e0003980 2024-01-15T18:17:13.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.471+0000 7f6107800640 1 -- 172.21.15.27:0/4090814673 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f61000cfa20 con 0x7f60e0003980 2024-01-15T18:17:13.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.471+0000 7f60fcc00640 1 -- 172.21.15.27:0/4090814673 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f61000cfa20 con 0x7f60e0003980 2024-01-15T18:17:13.469 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:17:13.470 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:17:13.470 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-01-15T18:17:13.470 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:17:13.470 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-01-15T18:17:13.470 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlbMZIGBAAhSU4y2DGJd3MH0v6v3LsDg==", 2024-01-15T18:17:13.470 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:17:13.470 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:17:13.470 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:17:13.470 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:17:13.470 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:17:13.470 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:17:13.470 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:17:13.470 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:17:13.471 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-01-15T18:17:13.471 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlC7B9JBAAsXmBjV4KwPZejOmowXO2Zg==", 2024-01-15T18:17:13.471 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:17:13.471 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:17:13.471 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:17:13.471 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:17:13.471 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:17:13.471 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:17:13.471 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:17:13.471 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:17:13.471 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-01-15T18:17:13.471 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6Vla6a3HxAA/AHc6FSTccj0JGwiR4Vl7g==", 2024-01-15T18:17:13.471 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:17:13.472 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:17:13.472 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:17:13.472 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:17:13.472 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:17:13.472 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:17:13.472 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:17:13.472 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:17:13.472 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-01-15T18:17:13.472 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6Vl9nqzKxAAx2aDA8UwidAUrU9DBxGldA==", 2024-01-15T18:17:13.472 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:17:13.472 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:17:13.472 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:17:13.472 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:17:13.473 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:17:13.473 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:17:13.473 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:17:13.473 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:17:13.473 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-01-15T18:17:13.473 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCSc6Vly4TACRAAYBmMIcsZsRp4CZ5a3xCWgA==", 2024-01-15T18:17:13.473 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:17:13.473 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:17:13.473 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:17:13.473 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:17:13.473 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:17:13.473 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:17:13.473 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:17:13.473 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-01-15T18:17:13.474 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCVc6VlW/HEERAAOBgPz+eGpHwNXDe8fkYo7w==", 2024-01-15T18:17:13.474 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:17:13.474 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:17:13.474 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:17:13.474 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:17:13.474 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:17:13.474 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:17:13.474 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:17:13.474 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-01-15T18:17:13.474 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCYc6Vl+Ec4GBAAWRe/6u7u4RXApd57H8ihng==", 2024-01-15T18:17:13.474 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:17:13.474 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:17:13.474 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:17:13.475 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:17:13.475 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:17:13.475 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:17:13.475 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:17:13.475 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-01-15T18:17:13.475 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCbc6VlNdMPHxAAHzJvzLltsTl1wrEnI/iyrQ==", 2024-01-15T18:17:13.475 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:17:13.475 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:17:13.475 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:17:13.475 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:17:13.475 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:17:13.475 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:17:13.475 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:17:13.476 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-01-15T18:17:13.476 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQChc6Vlg8eVKBAAwZWAncCE2Lra+CxmSIxZkA==", 2024-01-15T18:17:13.476 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:17:13.476 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:17:13.476 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:17:13.476 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:17:13.476 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:17:13.476 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:17:13.476 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:17:13.476 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-01-15T18:17:13.476 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCkc6VlIXDOLxAAFISn6ILqq/aNFK07RxNnaw==", 2024-01-15T18:17:13.476 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:17:13.476 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:17:13.477 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:17:13.477 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:17:13.477 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:17:13.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:17:13.477 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:17:13.477 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-01-15T18:17:13.477 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCnc6Vl9Ru2NRAAl996VBDudw23SGiFPusIcw==", 2024-01-15T18:17:13.477 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:17:13.477 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:17:13.477 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:17:13.477 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:17:13.477 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:17:13.477 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:17:13.478 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:17:13.478 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-01-15T18:17:13.478 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCrc6VlHoAsARAAIoPMridLsIcpU3I9Hr5XLQ==", 2024-01-15T18:17:13.478 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:17:13.478 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:17:13.478 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:17:13.478 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:17:13.478 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:17:13.478 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:17:13.478 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:17:13.478 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-01-15T18:17:13.478 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlH+DCLxAA1sKVut+3WjA/B+aYJ6/Jeg==", 2024-01-15T18:17:13.478 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:17:13.479 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:17:13.479 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-01-15T18:17:13.479 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-15T18:17:13.479 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:17:13.479 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:17:13.479 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:17:13.479 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-01-15T18:17:13.479 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlaRBaORAAijn3UJLHbWm8ol2NiKNSaw==", 2024-01-15T18:17:13.479 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:17:13.479 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:17:13.479 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-15T18:17:13.479 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-15T18:17:13.479 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-15T18:17:13.480 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:17:13.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:17:13.480 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:17:13.480 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-01-15T18:17:13.480 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlDbLQNRAA82Lwz+pKY+3nHPAr1DichA==", 2024-01-15T18:17:13.480 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:17:13.480 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:17:13.480 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-15T18:17:13.480 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-15T18:17:13.480 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-15T18:17:13.480 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:17:13.480 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:17:13.480 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:17:13.481 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-01-15T18:17:13.481 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCPc6VlOyDaAxAADea0K1OEDTfrzsM7VEkixg==", 2024-01-15T18:17:13.481 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:17:13.481 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:17:13.481 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-15T18:17:13.481 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-15T18:17:13.481 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-15T18:17:13.481 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:17:13.481 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:17:13.481 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:17:13.481 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-01-15T18:17:13.481 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNc6VlK70BNxAAgYXkW/pMlzy8GZoZhX9NMA==", 2024-01-15T18:17:13.481 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:17:13.481 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-15T18:17:13.482 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:17:13.482 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:17:13.482 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:17:13.482 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:17:13.482 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:17:13.482 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:17:13.482 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-01-15T18:17:13.482 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlQbinExAAWtUdgkUNR3cCKV+ytcbUuQ==", 2024-01-15T18:17:13.482 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:17:13.482 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-15T18:17:13.482 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-15T18:17:13.482 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:17:13.482 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:17:13.483 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:17:13.483 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:17:13.483 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-01-15T18:17:13.483 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlzafrDxAAVVEBuX8BDT3cB9NfD2xTdQ==", 2024-01-15T18:17:13.483 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:17:13.483 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-15T18:17:13.483 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-15T18:17:13.483 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:17:13.483 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:17:13.483 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:17:13.483 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-01-15T18:17:13.483 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:17:13.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.471+0000 7f6107800640 1 -- 172.21.15.27:0/4090814673 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f60d8067730 msgr2=0x7f60d8069bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:13.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.471+0000 7f6107800640 1 --2- 172.21.15.27:0/4090814673 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f60d8067730 0x7f60d8069bf0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f61001643d0 tx=0x7f60e8007040 comp rx=0 tx=0).stop 2024-01-15T18:17:13.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.471+0000 7f6107800640 1 -- 172.21.15.27:0/4090814673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60e0003980 msgr2=0x7f610009df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:13.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.471+0000 7f6107800640 1 --2- 172.21.15.27:0/4090814673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60e0003980 0x7f610009df30 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009cb0 tx=0x7f60e0005800 comp rx=0 tx=0).stop 2024-01-15T18:17:13.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.471+0000 7f6107800640 1 -- 172.21.15.27:0/4090814673 shutdown_connections 2024-01-15T18:17:13.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.471+0000 7f6107800640 1 --2- 172.21.15.27:0/4090814673 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f60d8067730 0x7f60d8069bf0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.471+0000 7f6107800640 1 --2- 172.21.15.27:0/4090814673 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f610015b3c0 0x7f610015b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.471+0000 7f6107800640 1 --2- 172.21.15.27:0/4090814673 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f610015bf10 0x7f61000cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.471+0000 7f6107800640 1 --2- 172.21.15.27:0/4090814673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f60e0003980 0x7f610009df30 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.471+0000 7f6107800640 1 -- 172.21.15.27:0/4090814673 >> 172.21.15.27:0/4090814673 conn(0x7f61000b2360 msgr2=0x7f61000b1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:13.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.471+0000 7f6107800640 1 -- 172.21.15.27:0/4090814673 shutdown_connections 2024-01-15T18:17:13.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.471+0000 7f6107800640 1 -- 172.21.15.27:0/4090814673 wait complete. 2024-01-15T18:17:13.492 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:17:13.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.635+0000 7f3b08400640 1 Processor -- start 2024-01-15T18:17:13.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.635+0000 7f3b08400640 1 -- start start 2024-01-15T18:17:13.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.635+0000 7f3b08400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3b0006cc70 0x7f3b0006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:13.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.635+0000 7f3b08400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3b00070d10 0x7f3b000711a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:13.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.635+0000 7f3b08400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b00070880 con 0x7f3b000b3c70 2024-01-15T18:17:13.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.635+0000 7f3b08400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b00070020 con 0x7f3b00070d10 2024-01-15T18:17:13.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.635+0000 7f3b08400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0006d630 con 0x7f3b0006cc70 2024-01-15T18:17:13.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.635+0000 7f3b06400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3b00070d10 0x7f3b000711a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:13.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.635+0000 7f3b05a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3b000b3c70 0x7f3b0015ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48788/0 (socket says 172.21.15.27:48788) 2024-01-15T18:17:13.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.635+0000 7f3b05a00640 1 -- 172.21.15.27:0/3050115650 learned_addr learned my addr 172.21.15.27:0/3050115650 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:13.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.635+0000 7f3b05000640 1 --2- 172.21.15.27:0/3050115650 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3b0006cc70 0x7f3b0006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:13.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.639+0000 7f3b06400640 1 -- 172.21.15.27:0/3050115650 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3b0006cc70 msgr2=0x7f3b0006d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:13.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.639+0000 7f3b06400640 1 --2- 172.21.15.27:0/3050115650 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3b0006cc70 0x7f3b0006d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.639+0000 7f3b06400640 1 -- 172.21.15.27:0/3050115650 >> v1:172.21.15.27:6789/0 conn(0x7f3b000b3c70 legacy=0x7f3b0015ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:13.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.639+0000 7f3b06400640 1 -- 172.21.15.27:0/3050115650 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0015a8b0 con 0x7f3b00070d10 2024-01-15T18:17:13.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.639+0000 7f3affe00640 1 -- 172.21.15.27:0/3050115650 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2209181470 0 0) 0x7f3b00070880 con 0x7f3b000b3c70 2024-01-15T18:17:13.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.639+0000 7f3b06400640 1 --2- 172.21.15.27:0/3050115650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b00070d10 0x7f3b000711a0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f3af8009d90 tx=0x7f3af801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62dbc43188edb875 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:13.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.639+0000 7f3affe00640 1 -- 172.21.15.27:0/3050115650 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af80a5880 con 0x7f3b00070d10 2024-01-15T18:17:13.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.639+0000 7f3affe00640 1 -- 172.21.15.27:0/3050115650 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af80a59e0 con 0x7f3b00070d10 2024-01-15T18:17:13.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.639+0000 7f3affe00640 1 -- 172.21.15.27:0/3050115650 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af80a5d70 con 0x7f3b00070d10 2024-01-15T18:17:13.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b08400640 1 -- 172.21.15.27:0/3050115650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b00070d10 msgr2=0x7f3b000711a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:13.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b08400640 1 --2- 172.21.15.27:0/3050115650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b00070d10 0x7f3b000711a0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f3af8009d90 tx=0x7f3af801c950 comp rx=0 tx=0).stop 2024-01-15T18:17:13.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b08400640 1 -- 172.21.15.27:0/3050115650 shutdown_connections 2024-01-15T18:17:13.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b08400640 1 --2- 172.21.15.27:0/3050115650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b00070d10 0x7f3b000711a0 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b08400640 1 --2- 172.21.15.27:0/3050115650 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3b0006cc70 0x7f3b0006d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b08400640 1 -- 172.21.15.27:0/3050115650 >> 172.21.15.27:0/3050115650 conn(0x7f3b0006b0c0 msgr2=0x7f3b000cec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:13.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b08400640 1 -- 172.21.15.27:0/3050115650 shutdown_connections 2024-01-15T18:17:13.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b08400640 1 -- 172.21.15.27:0/3050115650 wait complete. 2024-01-15T18:17:13.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b08400640 1 Processor -- start 2024-01-15T18:17:13.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b08400640 1 -- start start 2024-01-15T18:17:13.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b08400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3b0006cc70 0x7f3b00148bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:13.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b08400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3b00070d10 0x7f3b001474a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:13.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b08400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b000b3c70 0x7f3b00155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:13.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b08400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b00155b40 con 0x7f3b000b3c70 2024-01-15T18:17:13.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b08400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b00145f40 con 0x7f3b0006cc70 2024-01-15T18:17:13.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b08400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b00147b30 con 0x7f3b00070d10 2024-01-15T18:17:13.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b06400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b000b3c70 0x7f3b00155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:13.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b06400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b000b3c70 0x7f3b00155600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59492/0 (socket says 172.21.15.27:59492) 2024-01-15T18:17:13.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b06400640 1 -- 172.21.15.27:0/4019850350 learned_addr learned my addr 172.21.15.27:0/4019850350 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:13.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b05000640 1 --2- 172.21.15.27:0/4019850350 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3b00070d10 0x7f3b001474a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:13.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b06400640 1 -- 172.21.15.27:0/4019850350 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3b00070d10 msgr2=0x7f3b001474a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:13.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b05a00640 1 --2- 172.21.15.27:0/4019850350 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3b0006cc70 0x7f3b00148bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:13.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b06400640 1 --2- 172.21.15.27:0/4019850350 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3b00070d10 0x7f3b001474a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b06400640 1 -- 172.21.15.27:0/4019850350 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3b0006cc70 msgr2=0x7f3b00148bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:13.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b06400640 1 --2- 172.21.15.27:0/4019850350 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3b0006cc70 0x7f3b00148bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b06400640 1 -- 172.21.15.27:0/4019850350 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af8009a70 con 0x7f3b000b3c70 2024-01-15T18:17:13.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.643+0000 7f3b05000640 1 --2- 172.21.15.27:0/4019850350 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3b00070d10 0x7f3b001474a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:17:13.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.647+0000 7f3b06400640 1 --2- 172.21.15.27:0/4019850350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b000b3c70 0x7f3b00155600 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f3af801b000 tx=0x7f3af80196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:13.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.647+0000 7f3afd600640 1 -- 172.21.15.27:0/4019850350 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af80ad850 con 0x7f3b000b3c70 2024-01-15T18:17:13.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.647+0000 7f3afd600640 1 -- 172.21.15.27:0/4019850350 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af80ad9b0 con 0x7f3b000b3c70 2024-01-15T18:17:13.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.647+0000 7f3b08400640 1 -- 172.21.15.27:0/4019850350 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b00147e10 con 0x7f3b000b3c70 2024-01-15T18:17:13.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.647+0000 7f3afd600640 1 -- 172.21.15.27:0/4019850350 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af80add20 con 0x7f3b000b3c70 2024-01-15T18:17:13.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.647+0000 7f3b08400640 1 -- 172.21.15.27:0/4019850350 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b001483d0 con 0x7f3b000b3c70 2024-01-15T18:17:13.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.647+0000 7f3b08400640 1 -- 172.21.15.27:0/4019850350 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b0004e850 con 0x7f3b000b3c70 2024-01-15T18:17:13.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.647+0000 7f3afd600640 1 -- 172.21.15.27:0/4019850350 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f3af80b4050 con 0x7f3b000b3c70 2024-01-15T18:17:13.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.647+0000 7f3afd600640 1 --2- 172.21.15.27:0/4019850350 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3ad8067700 0x7f3ad8069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:13.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.647+0000 7f3afd600640 1 -- 172.21.15.27:0/4019850350 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4969+0+0 (secure 0 0 0) 0x7f3af811dd10 con 0x7f3b000b3c70 2024-01-15T18:17:13.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.651+0000 7f3b05a00640 1 --2- 172.21.15.27:0/4019850350 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3ad8067700 0x7f3ad8069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:13.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.651+0000 7f3b05a00640 1 --2- 172.21.15.27:0/4019850350 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3ad8067700 0x7f3ad8069bc0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f3af0007fd0 tx=0x7f3af0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:13.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.651+0000 7f3afd600640 1 -- 172.21.15.27:0/4019850350 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3af80ecac0 con 0x7f3b000b3c70 2024-01-15T18:17:13.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.811+0000 7f3b08400640 1 -- 172.21.15.27:0/4019850350 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b0006b500 con 0x7f3b000b3c70 2024-01-15T18:17:13.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.811+0000 7f3afd600640 1 -- 172.21.15.27:0/4019850350 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+12939 (secure 0 0 0) 0x7f3af80b2090 con 0x7f3b000b3c70 2024-01-15T18:17:13.810 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:17:13.810 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":93,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:17:13.080395+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:17:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.811+0000 7f3b08400640 1 -- 172.21.15.27:0/4019850350 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3ad8067700 msgr2=0x7f3ad8069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.811+0000 7f3b08400640 1 --2- 172.21.15.27:0/4019850350 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3ad8067700 0x7f3ad8069bc0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f3af0007fd0 tx=0x7f3af0009040 comp rx=0 tx=0).stop 2024-01-15T18:17:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.811+0000 7f3b08400640 1 -- 172.21.15.27:0/4019850350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b000b3c70 msgr2=0x7f3b00155600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.811+0000 7f3b08400640 1 --2- 172.21.15.27:0/4019850350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b000b3c70 0x7f3b00155600 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f3af801b000 tx=0x7f3af80196f0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.815+0000 7f3b08400640 1 -- 172.21.15.27:0/4019850350 shutdown_connections 2024-01-15T18:17:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.815+0000 7f3b08400640 1 --2- 172.21.15.27:0/4019850350 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3ad8067700 0x7f3ad8069bc0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.815+0000 7f3b08400640 1 --2- 172.21.15.27:0/4019850350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b000b3c70 0x7f3b00155600 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.815+0000 7f3b08400640 1 --2- 172.21.15.27:0/4019850350 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3b00070d10 0x7f3b001474a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.815+0000 7f3b08400640 1 --2- 172.21.15.27:0/4019850350 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3b0006cc70 0x7f3b00148bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.815+0000 7f3b08400640 1 -- 172.21.15.27:0/4019850350 >> 172.21.15.27:0/4019850350 conn(0x7f3b0006b0c0 msgr2=0x7f3b000b01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.815+0000 7f3b08400640 1 -- 172.21.15.27:0/4019850350 shutdown_connections 2024-01-15T18:17:13.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.815+0000 7f3b08400640 1 -- 172.21.15.27:0/4019850350 wait complete. 2024-01-15T18:17:13.832 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:17:13.833 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:17:13.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.979+0000 7fd537a00640 1 Processor -- start 2024-01-15T18:17:13.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.979+0000 7fd537a00640 1 -- start start 2024-01-15T18:17:13.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.979+0000 7fd537a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fd5300cf2e0 0x7fd5300701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:13.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.979+0000 7fd537a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5300b1cb0 con 0x7fd530158900 2024-01-15T18:17:13.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.979+0000 7fd537a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd53015fa00 con 0x7fd53015f550 2024-01-15T18:17:13.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.979+0000 7fd537a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5300b1f00 con 0x7fd5300cf2e0 2024-01-15T18:17:13.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.979+0000 7fd535000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd530158900 0x7fd53015bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48792/0 (socket says 172.21.15.27:48792) 2024-01-15T18:17:13.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.979+0000 7fd535000640 1 -- 172.21.15.27:0/1896944778 learned_addr learned my addr 172.21.15.27:0/1896944778 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:13.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.979+0000 7fd535a00640 1 --2- 172.21.15.27:0/1896944778 >> v2:172.21.15.93:3300/0 conn(0x7fd5300cf2e0 0x7fd5300701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:13.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.979+0000 7fd52f400640 1 -- 172.21.15.27:0/1896944778 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1607068247 0 0) 0x7fd5300b1cb0 con 0x7fd530158900 2024-01-15T18:17:13.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.979+0000 7fd52f400640 1 -- 172.21.15.27:0/1896944778 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd510003530 con 0x7fd530158900 2024-01-15T18:17:13.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.979+0000 7fd52f400640 1 -- 172.21.15.27:0/1896944778 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3428378488 0 0) 0x7fd53015fa00 con 0x7fd53015f550 2024-01-15T18:17:13.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.979+0000 7fd52f400640 1 -- 172.21.15.27:0/1896944778 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5300b1cb0 con 0x7fd53015f550 2024-01-15T18:17:13.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.979+0000 7fd52f400640 1 -- 172.21.15.27:0/1896944778 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4095302716 0 0) 0x7fd510003530 con 0x7fd530158900 2024-01-15T18:17:13.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.979+0000 7fd535a00640 1 -- 172.21.15.27:0/1896944778 >> v1:172.21.15.93:6789/0 conn(0x7fd53015f550 legacy=0x7fd5300cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:13.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.979+0000 7fd535a00640 1 -- 172.21.15.27:0/1896944778 >> v1:172.21.15.27:6789/0 conn(0x7fd530158900 legacy=0x7fd53015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:13.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.979+0000 7fd535a00640 1 -- 172.21.15.27:0/1896944778 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5300b21e0 con 0x7fd5300cf2e0 2024-01-15T18:17:13.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.983+0000 7fd535a00640 1 --2- 172.21.15.27:0/1896944778 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5300cf2e0 0x7fd5300701e0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd528009d90 tx=0x7fd52801c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=489feaced362dc48 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:13.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.983+0000 7fd52f400640 1 -- 172.21.15.27:0/1896944778 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd530070f30 con 0x7fd5300cf2e0 2024-01-15T18:17:13.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.983+0000 7fd52f400640 1 -- 172.21.15.27:0/1896944778 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5280a5680 con 0x7fd5300cf2e0 2024-01-15T18:17:13.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.983+0000 7fd52f400640 1 -- 172.21.15.27:0/1896944778 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd520001680 con 0x7fd5300cf2e0 2024-01-15T18:17:13.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.983+0000 7fd537a00640 1 -- 172.21.15.27:0/1896944778 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5300cf2e0 msgr2=0x7fd5300701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:13.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.983+0000 7fd537a00640 1 --2- 172.21.15.27:0/1896944778 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5300cf2e0 0x7fd5300701e0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd528009d90 tx=0x7fd52801c840 comp rx=0 tx=0).stop 2024-01-15T18:17:13.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.983+0000 7fd537a00640 1 -- 172.21.15.27:0/1896944778 shutdown_connections 2024-01-15T18:17:13.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.983+0000 7fd537a00640 1 --2- 172.21.15.27:0/1896944778 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd5300cf2e0 0x7fd5300701e0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.983+0000 7fd537a00640 1 -- 172.21.15.27:0/1896944778 >> 172.21.15.27:0/1896944778 conn(0x7fd5300cb700 msgr2=0x7fd5300cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:13.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.983+0000 7fd537a00640 1 -- 172.21.15.27:0/1896944778 shutdown_connections 2024-01-15T18:17:13.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.983+0000 7fd537a00640 1 -- 172.21.15.27:0/1896944778 wait complete. 2024-01-15T18:17:13.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.987+0000 7fd537a00640 1 Processor -- start 2024-01-15T18:17:13.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.987+0000 7fd537a00640 1 -- start start 2024-01-15T18:17:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.987+0000 7fd537a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd5300cf2e0 0x7fd5300b0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.987+0000 7fd537a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd530158900 0x7fd53015cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.987+0000 7fd537a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd53015f550 0x7fd53015d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.987+0000 7fd537a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5300cd5c0 con 0x7fd530158900 2024-01-15T18:17:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.987+0000 7fd537a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5300b06a0 con 0x7fd53015f550 2024-01-15T18:17:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.987+0000 7fd537a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5300cc940 con 0x7fd5300cf2e0 2024-01-15T18:17:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.987+0000 7fd52fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd530158900 0x7fd53015cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.987+0000 7fd52fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd530158900 0x7fd53015cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59498/0 (socket says 172.21.15.27:59498) 2024-01-15T18:17:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.987+0000 7fd52fe00640 1 -- 172.21.15.27:0/1263582104 learned_addr learned my addr 172.21.15.27:0/1263582104 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.987+0000 7fd535000640 1 --2- 172.21.15.27:0/1263582104 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd5300cf2e0 0x7fd5300b0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.987+0000 7fd535a00640 1 --2- 172.21.15.27:0/1263582104 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd53015f550 0x7fd53015d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:13.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.987+0000 7fd52fe00640 1 -- 172.21.15.27:0/1263582104 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd5300cf2e0 msgr2=0x7fd5300b0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:13.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.987+0000 7fd52fe00640 1 --2- 172.21.15.27:0/1263582104 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd5300cf2e0 0x7fd5300b0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.987+0000 7fd52fe00640 1 -- 172.21.15.27:0/1263582104 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd53015f550 msgr2=0x7fd53015d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:13.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.987+0000 7fd52fe00640 1 --2- 172.21.15.27:0/1263582104 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd53015f550 0x7fd53015d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:13.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.987+0000 7fd52fe00640 1 -- 172.21.15.27:0/1263582104 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd528009a70 con 0x7fd530158900 2024-01-15T18:17:13.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.987+0000 7fd535a00640 1 --2- 172.21.15.27:0/1263582104 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd53015f550 0x7fd53015d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:17:13.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.987+0000 7fd52fe00640 1 --2- 172.21.15.27:0/1263582104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd530158900 0x7fd53015cdd0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fd51800c490 tx=0x7fd51800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:13.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.987+0000 7fd52cc00640 1 -- 172.21.15.27:0/1263582104 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5180052d0 con 0x7fd530158900 2024-01-15T18:17:13.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.987+0000 7fd52cc00640 1 -- 172.21.15.27:0/1263582104 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd518005430 con 0x7fd530158900 2024-01-15T18:17:13.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.987+0000 7fd52cc00640 1 -- 172.21.15.27:0/1263582104 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd518012870 con 0x7fd530158900 2024-01-15T18:17:13.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.987+0000 7fd537a00640 1 -- 172.21.15.27:0/1263582104 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5300ccc20 con 0x7fd530158900 2024-01-15T18:17:13.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.987+0000 7fd537a00640 1 -- 172.21.15.27:0/1263582104 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5300cd1e0 con 0x7fd530158900 2024-01-15T18:17:13.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.991+0000 7fd537a00640 1 -- 172.21.15.27:0/1263582104 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5300cd700 con 0x7fd530158900 2024-01-15T18:17:13.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.991+0000 7fd52cc00640 1 -- 172.21.15.27:0/1263582104 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7fd5180129d0 con 0x7fd530158900 2024-01-15T18:17:13.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.991+0000 7fd52cc00640 1 --2- 172.21.15.27:0/1263582104 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd508067730 0x7fd508069bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:13.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.991+0000 7fd52cc00640 1 -- 172.21.15.27:0/1263582104 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4969+0+0 (secure 0 0 0) 0x7fd5180835d0 con 0x7fd530158900 2024-01-15T18:17:13.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.991+0000 7fd535000640 1 --2- 172.21.15.27:0/1263582104 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd508067730 0x7fd508069bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:13.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.991+0000 7fd535000640 1 --2- 172.21.15.27:0/1263582104 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd508067730 0x7fd508069bf0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd520000f70 tx=0x7fd5200016c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:13.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:13.995+0000 7fd52cc00640 1 -- 172.21.15.27:0/1263582104 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd518051fa0 con 0x7fd530158900 2024-01-15T18:17:14.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.155+0000 7fd537a00640 1 -- 172.21.15.27:0/1263582104 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd5300cd700 con 0x7fd530158900 2024-01-15T18:17:14.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.155+0000 7fd52cc00640 1 -- 172.21.15.27:0/1263582104 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd5300cd700 con 0x7fd530158900 2024-01-15T18:17:14.153 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:17:14.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.155+0000 7fd537a00640 1 -- 172.21.15.27:0/1263582104 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd508067730 msgr2=0x7fd508069bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:14.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.155+0000 7fd537a00640 1 --2- 172.21.15.27:0/1263582104 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd508067730 0x7fd508069bf0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd520000f70 tx=0x7fd5200016c0 comp rx=0 tx=0).stop 2024-01-15T18:17:14.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.155+0000 7fd537a00640 1 -- 172.21.15.27:0/1263582104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd530158900 msgr2=0x7fd53015cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:14.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.155+0000 7fd537a00640 1 --2- 172.21.15.27:0/1263582104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd530158900 0x7fd53015cdd0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fd51800c490 tx=0x7fd51800c950 comp rx=0 tx=0).stop 2024-01-15T18:17:14.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.155+0000 7fd537a00640 1 -- 172.21.15.27:0/1263582104 shutdown_connections 2024-01-15T18:17:14.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.155+0000 7fd537a00640 1 --2- 172.21.15.27:0/1263582104 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd508067730 0x7fd508069bf0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:14.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.155+0000 7fd537a00640 1 --2- 172.21.15.27:0/1263582104 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd53015f550 0x7fd53015d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:14.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.155+0000 7fd537a00640 1 --2- 172.21.15.27:0/1263582104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd530158900 0x7fd53015cdd0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:14.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.155+0000 7fd537a00640 1 --2- 172.21.15.27:0/1263582104 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd5300cf2e0 0x7fd5300b0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:14.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.155+0000 7fd537a00640 1 -- 172.21.15.27:0/1263582104 >> 172.21.15.27:0/1263582104 conn(0x7fd5300cb700 msgr2=0x7fd5300cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:14.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.155+0000 7fd537a00640 1 -- 172.21.15.27:0/1263582104 shutdown_connections 2024-01-15T18:17:14.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.155+0000 7fd537a00640 1 -- 172.21.15.27:0/1263582104 wait complete. 2024-01-15T18:17:14.175 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-15T18:17:14.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.319+0000 7f6593e00640 1 Processor -- start 2024-01-15T18:17:14.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.319+0000 7f6593e00640 1 -- start start 2024-01-15T18:17:14.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.319+0000 7f6593e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f658c0cf9b0 0x7f658c0b10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:14.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.319+0000 7f6593e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f658c0cedc0 0x7f658c0b1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:14.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.319+0000 7f6593e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658c0b38b0 con 0x7f658c15e7d0 2024-01-15T18:17:14.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.319+0000 7f6593e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f658c0cd550 con 0x7f658c0cedc0 2024-01-15T18:17:14.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.319+0000 7f6593e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658c0cd270 con 0x7f658c0cf9b0 2024-01-15T18:17:14.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.319+0000 7f6591400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f658c15e7d0 0x7f658c0cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48802/0 (socket says 172.21.15.27:48802) 2024-01-15T18:17:14.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.319+0000 7f6591400640 1 -- 172.21.15.27:0/2604716390 learned_addr learned my addr 172.21.15.27:0/2604716390 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:14.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.319+0000 7f6591e00640 1 --2- 172.21.15.27:0/2604716390 >> v2:172.21.15.93:3300/0 conn(0x7f658c0cedc0 0x7f658c0b1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:14.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.319+0000 7f6590a00640 1 --2- 172.21.15.27:0/2604716390 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f658c0cf9b0 0x7f658c0b10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:14.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.319+0000 7f658be00640 1 -- 172.21.15.27:0/2604716390 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1665188164 0 0) 0x7f658c0b38b0 con 0x7f658c15e7d0 2024-01-15T18:17:14.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.319+0000 7f658be00640 1 -- 172.21.15.27:0/2604716390 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f656c003530 con 0x7f658c15e7d0 2024-01-15T18:17:14.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.319+0000 7f6591e00640 1 -- 172.21.15.27:0/2604716390 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f658c0cf9b0 msgr2=0x7f658c0b10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:14.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.319+0000 7f6591e00640 1 --2- 172.21.15.27:0/2604716390 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f658c0cf9b0 0x7f658c0b10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:14.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.319+0000 7f6591e00640 1 -- 172.21.15.27:0/2604716390 >> v1:172.21.15.27:6789/0 conn(0x7f658c15e7d0 legacy=0x7f658c0cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:14.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.319+0000 7f6591e00640 1 -- 172.21.15.27:0/2604716390 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658c0b35d0 con 0x7f658c0cedc0 2024-01-15T18:17:14.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.319+0000 7f6590a00640 1 --2- 172.21.15.27:0/2604716390 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f658c0cf9b0 0x7f658c0b10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:17:14.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.323+0000 7f6591e00640 1 --2- 172.21.15.27:0/2604716390 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f658c0cedc0 0x7f658c0b1600 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f6584009d90 tx=0x7f658401c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a9d6061f9d393f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:14.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.323+0000 7f658be00640 1 -- 172.21.15.27:0/2604716390 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65840a5900 con 0x7f658c0cedc0 2024-01-15T18:17:14.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.323+0000 7f658be00640 1 -- 172.21.15.27:0/2604716390 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65840a5a60 con 0x7f658c0cedc0 2024-01-15T18:17:14.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.323+0000 7f658be00640 1 -- 172.21.15.27:0/2604716390 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65840a5df0 con 0x7f658c0cedc0 2024-01-15T18:17:14.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.323+0000 7f6593e00640 1 -- 172.21.15.27:0/2604716390 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f658c0cedc0 msgr2=0x7f658c0b1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:14.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.323+0000 7f6593e00640 1 --2- 172.21.15.27:0/2604716390 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f658c0cedc0 0x7f658c0b1600 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f6584009d90 tx=0x7f658401c950 comp rx=0 tx=0).stop 2024-01-15T18:17:14.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.323+0000 7f6593e00640 1 -- 172.21.15.27:0/2604716390 shutdown_connections 2024-01-15T18:17:14.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.323+0000 7f6593e00640 1 --2- 172.21.15.27:0/2604716390 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f658c0cf9b0 0x7f658c0b10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:14.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.323+0000 7f6593e00640 1 --2- 172.21.15.27:0/2604716390 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f658c0cedc0 0x7f658c0b1600 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:14.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.323+0000 7f6593e00640 1 -- 172.21.15.27:0/2604716390 >> 172.21.15.27:0/2604716390 conn(0x7f658c0b2a10 msgr2=0x7f658c0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:14.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.323+0000 7f6593e00640 1 -- 172.21.15.27:0/2604716390 shutdown_connections 2024-01-15T18:17:14.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.323+0000 7f6593e00640 1 -- 172.21.15.27:0/2604716390 wait complete. 2024-01-15T18:17:14.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.327+0000 7f6593e00640 1 Processor -- start 2024-01-15T18:17:14.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.327+0000 7f6593e00640 1 -- start start 2024-01-15T18:17:14.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.327+0000 7f6593e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f658c0cedc0 0x7f658c164270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:14.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.327+0000 7f6593e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f658c0cf9b0 0x7f658c154610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:14.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.327+0000 7f6593e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f658c15e7d0 0x7f658c157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:14.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.327+0000 7f6593e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658c163de0 con 0x7f658c15e7d0 2024-01-15T18:17:14.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.327+0000 7f6593e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658c1631d0 con 0x7f658c0cedc0 2024-01-15T18:17:14.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.327+0000 7f6593e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658c1634b0 con 0x7f658c0cf9b0 2024-01-15T18:17:14.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.327+0000 7f6591e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f658c15e7d0 0x7f658c157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:14.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.327+0000 7f6591e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f658c15e7d0 0x7f658c157ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59508/0 (socket says 172.21.15.27:59508) 2024-01-15T18:17:14.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.327+0000 7f6591e00640 1 -- 172.21.15.27:0/1104148717 learned_addr learned my addr 172.21.15.27:0/1104148717 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:14.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.327+0000 7f6591400640 1 --2- 172.21.15.27:0/1104148717 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f658c0cedc0 0x7f658c164270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:14.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.327+0000 7f6590a00640 1 --2- 172.21.15.27:0/1104148717 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f658c0cf9b0 0x7f658c154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:14.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.327+0000 7f6591e00640 1 -- 172.21.15.27:0/1104148717 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f658c0cf9b0 msgr2=0x7f658c154610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:14.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.327+0000 7f6591e00640 1 --2- 172.21.15.27:0/1104148717 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f658c0cf9b0 0x7f658c154610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:14.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.327+0000 7f6591e00640 1 -- 172.21.15.27:0/1104148717 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f658c0cedc0 msgr2=0x7f658c164270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:14.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.327+0000 7f6591e00640 1 --2- 172.21.15.27:0/1104148717 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f658c0cedc0 0x7f658c164270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:14.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.327+0000 7f6591e00640 1 -- 172.21.15.27:0/1104148717 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6574009420 con 0x7f658c15e7d0 2024-01-15T18:17:14.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.327+0000 7f6590a00640 1 --2- 172.21.15.27:0/1104148717 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f658c0cf9b0 0x7f658c154610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:17:14.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.327+0000 7f6591e00640 1 --2- 172.21.15.27:0/1104148717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f658c15e7d0 0x7f658c157ff0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f658401b000 tx=0x7f65840196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:14.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.327+0000 7f6589600640 1 -- 172.21.15.27:0/1104148717 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65840b9650 con 0x7f658c15e7d0 2024-01-15T18:17:14.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.327+0000 7f6589600640 1 -- 172.21.15.27:0/1104148717 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65840b97b0 con 0x7f658c15e7d0 2024-01-15T18:17:14.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.327+0000 7f6589600640 1 -- 172.21.15.27:0/1104148717 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65840b9b20 con 0x7f658c15e7d0 2024-01-15T18:17:14.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.327+0000 7f6593e00640 1 -- 172.21.15.27:0/1104148717 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6584009a70 con 0x7f658c15e7d0 2024-01-15T18:17:14.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.327+0000 7f6593e00640 1 -- 172.21.15.27:0/1104148717 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f658c163a70 con 0x7f658c15e7d0 2024-01-15T18:17:14.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.331+0000 7f6593e00640 1 -- 172.21.15.27:0/1104148717 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f658c163f50 con 0x7f658c15e7d0 2024-01-15T18:17:14.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.331+0000 7f6589600640 1 -- 172.21.15.27:0/1104148717 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f65840b9c80 con 0x7f658c15e7d0 2024-01-15T18:17:14.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.331+0000 7f6589600640 1 --2- 172.21.15.27:0/1104148717 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6564067700 0x7f6564069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:14.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.331+0000 7f6589600640 1 -- 172.21.15.27:0/1104148717 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4969+0+0 (secure 0 0 0) 0x7f658411e200 con 0x7f658c15e7d0 2024-01-15T18:17:14.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.331+0000 7f6591400640 1 --2- 172.21.15.27:0/1104148717 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6564067700 0x7f6564069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:14.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.331+0000 7f6591400640 1 --2- 172.21.15.27:0/1104148717 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6564067700 0x7f6564069bc0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f657c0011f0 tx=0x7f657c007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:14.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.335+0000 7f6589600640 1 -- 172.21.15.27:0/1104148717 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f65840ecfb0 con 0x7f658c15e7d0 2024-01-15T18:17:14.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.495+0000 7f6593e00640 1 -- 172.21.15.27:0/1104148717 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f658c04e850 con 0x7f658c15e7d0 2024-01-15T18:17:14.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.495+0000 7f6589600640 1 -- 172.21.15.27:0/1104148717 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v93) v1 ==== 84+0+65 (secure 0 0 0) 0x7f65840f05d0 con 0x7f658c15e7d0 2024-01-15T18:17:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:17:14.494 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:17:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:17:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-01-15T18:17:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-01-15T18:17:14.494 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:17:14.494 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:17:14.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.495+0000 7f6593e00640 1 -- 172.21.15.27:0/1104148717 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6564067700 msgr2=0x7f6564069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:14.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.495+0000 7f6593e00640 1 --2- 172.21.15.27:0/1104148717 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6564067700 0x7f6564069bc0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f657c0011f0 tx=0x7f657c007d40 comp rx=0 tx=0).stop 2024-01-15T18:17:14.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.495+0000 7f6593e00640 1 -- 172.21.15.27:0/1104148717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f658c15e7d0 msgr2=0x7f658c157ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:14.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.495+0000 7f6593e00640 1 --2- 172.21.15.27:0/1104148717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f658c15e7d0 0x7f658c157ff0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f658401b000 tx=0x7f65840196f0 comp rx=0 tx=0).stop 2024-01-15T18:17:14.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.495+0000 7f6593e00640 1 -- 172.21.15.27:0/1104148717 shutdown_connections 2024-01-15T18:17:14.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.495+0000 7f6593e00640 1 --2- 172.21.15.27:0/1104148717 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6564067700 0x7f6564069bc0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:14.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.495+0000 7f6593e00640 1 --2- 172.21.15.27:0/1104148717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f658c15e7d0 0x7f658c157ff0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:14.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.495+0000 7f6593e00640 1 --2- 172.21.15.27:0/1104148717 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f658c0cf9b0 0x7f658c154610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:14.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.495+0000 7f6593e00640 1 --2- 172.21.15.27:0/1104148717 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f658c0cedc0 0x7f658c164270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:14.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.495+0000 7f6593e00640 1 -- 172.21.15.27:0/1104148717 >> 172.21.15.27:0/1104148717 conn(0x7f658c0b2a10 msgr2=0x7f658c0cc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:14.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.499+0000 7f6593e00640 1 -- 172.21.15.27:0/1104148717 shutdown_connections 2024-01-15T18:17:14.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.499+0000 7f6593e00640 1 -- 172.21.15.27:0/1104148717 wait complete. 2024-01-15T18:17:14.517 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-15T18:17:14.517 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-15T18:17:14.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.663+0000 7f63b8000640 1 Processor -- start 2024-01-15T18:17:14.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.663+0000 7f63b8000640 1 -- start start 2024-01-15T18:17:14.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.663+0000 7f63b8000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f63b00ce2c0 0x7f63b00cca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:14.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.663+0000 7f63b8000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f63b00d19f0 0x7f63b00b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:14.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.663+0000 7f63b8000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b015b6b0 con 0x7f63b00cef20 2024-01-15T18:17:14.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.663+0000 7f63b8000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63b00d1ea0 con 0x7f63b00d19f0 2024-01-15T18:17:14.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.663+0000 7f63b8000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63b00cb9e0 con 0x7f63b00ce2c0 2024-01-15T18:17:14.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.663+0000 7f63b6000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f63b00d19f0 0x7f63b00b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:14.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.663+0000 7f63b5600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f63b00cef20 0x7f63b00cf300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48814/0 (socket says 172.21.15.27:48814) 2024-01-15T18:17:14.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.663+0000 7f63b6000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f63b00d19f0 0x7f63b00b3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59518/0 (socket says 172.21.15.27:59518) 2024-01-15T18:17:14.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.663+0000 7f63b5600640 1 -- 172.21.15.27:0/2796287671 learned_addr learned my addr 172.21.15.27:0/2796287671 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:14.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.663+0000 7f63b4c00640 1 --2- 172.21.15.27:0/2796287671 >> v2:172.21.15.93:3300/0 conn(0x7f63b00ce2c0 0x7f63b00cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:14.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.663+0000 7f63b6000640 1 -- 172.21.15.27:0/2796287671 >> v2:172.21.15.93:3300/0 conn(0x7f63b00ce2c0 msgr2=0x7f63b00cca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:14.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.663+0000 7f63b6000640 1 --2- 172.21.15.27:0/2796287671 >> v2:172.21.15.93:3300/0 conn(0x7f63b00ce2c0 0x7f63b00cca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:14.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.663+0000 7f63b6000640 1 -- 172.21.15.27:0/2796287671 >> v1:172.21.15.27:6789/0 conn(0x7f63b00cef20 legacy=0x7f63b00cf300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:14.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.663+0000 7f63b6000640 1 -- 172.21.15.27:0/2796287671 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b00cfdb0 con 0x7f63b00d19f0 2024-01-15T18:17:14.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.667+0000 7f63b6000640 1 --2- 172.21.15.27:0/2796287671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b00d19f0 0x7f63b00b3d00 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f63b0070e80 tx=0x7f63a801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8214a5b02af7d7d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:14.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.667+0000 7f63afe00640 1 -- 172.21.15.27:0/2796287671 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a80a58c0 con 0x7f63b00d19f0 2024-01-15T18:17:14.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.667+0000 7f63afe00640 1 -- 172.21.15.27:0/2796287671 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63a80a5a20 con 0x7f63b00d19f0 2024-01-15T18:17:14.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.667+0000 7f63afe00640 1 -- 172.21.15.27:0/2796287671 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a80a5db0 con 0x7f63b00d19f0 2024-01-15T18:17:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.667+0000 7f63b8000640 1 -- 172.21.15.27:0/2796287671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b00d19f0 msgr2=0x7f63b00b3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.667+0000 7f63b8000640 1 --2- 172.21.15.27:0/2796287671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b00d19f0 0x7f63b00b3d00 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f63b0070e80 tx=0x7f63a801c910 comp rx=0 tx=0).stop 2024-01-15T18:17:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.667+0000 7f63b8000640 1 -- 172.21.15.27:0/2796287671 shutdown_connections 2024-01-15T18:17:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.667+0000 7f63b8000640 1 --2- 172.21.15.27:0/2796287671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b00d19f0 0x7f63b00b3d00 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.667+0000 7f63b8000640 1 --2- 172.21.15.27:0/2796287671 >> v2:172.21.15.93:3300/0 conn(0x7f63b00ce2c0 0x7f63b00cca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.667+0000 7f63b8000640 1 -- 172.21.15.27:0/2796287671 >> 172.21.15.27:0/2796287671 conn(0x7f63b00d0ed0 msgr2=0x7f63b00d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.667+0000 7f63b8000640 1 -- 172.21.15.27:0/2796287671 shutdown_connections 2024-01-15T18:17:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.667+0000 7f63b8000640 1 -- 172.21.15.27:0/2796287671 wait complete. 2024-01-15T18:17:14.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.671+0000 7f63b8000640 1 Processor -- start 2024-01-15T18:17:14.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.671+0000 7f63b8000640 1 -- start start 2024-01-15T18:17:14.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.671+0000 7f63b8000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b00ce2c0 0x7f63b014a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:14.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.671+0000 7f63b8000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f63b00cef20 0x7f63b015a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:14.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.671+0000 7f63b8000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63b00d19f0 0x7f63b0148580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:14.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.671+0000 7f63b8000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b014a810 con 0x7f63b00ce2c0 2024-01-15T18:17:14.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.671+0000 7f63b8000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b015a6c0 con 0x7f63b00d19f0 2024-01-15T18:17:14.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.671+0000 7f63b8000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b014e1a0 con 0x7f63b00cef20 2024-01-15T18:17:14.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.671+0000 7f63b5600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b00ce2c0 0x7f63b014a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:14.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.671+0000 7f63b5600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b00ce2c0 0x7f63b014a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59530/0 (socket says 172.21.15.27:59530) 2024-01-15T18:17:14.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.671+0000 7f63b5600640 1 -- 172.21.15.27:0/610520007 learned_addr learned my addr 172.21.15.27:0/610520007 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:14.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.671+0000 7f63b6000640 1 --2- 172.21.15.27:0/610520007 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63b00d19f0 0x7f63b0148580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:14.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.671+0000 7f63b4c00640 1 --2- 172.21.15.27:0/610520007 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f63b00cef20 0x7f63b015a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:14.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.671+0000 7f63b5600640 1 -- 172.21.15.27:0/610520007 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f63b00cef20 msgr2=0x7f63b015a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:14.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.671+0000 7f63b5600640 1 --2- 172.21.15.27:0/610520007 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f63b00cef20 0x7f63b015a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:14.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.671+0000 7f63b5600640 1 -- 172.21.15.27:0/610520007 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63b00d19f0 msgr2=0x7f63b0148580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:14.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.671+0000 7f63b5600640 1 --2- 172.21.15.27:0/610520007 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63b00d19f0 0x7f63b0148580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:14.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.671+0000 7f63b5600640 1 -- 172.21.15.27:0/610520007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a8009a70 con 0x7f63b00ce2c0 2024-01-15T18:17:14.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.671+0000 7f63b4c00640 1 --2- 172.21.15.27:0/610520007 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f63b00cef20 0x7f63b015a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:17:14.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.671+0000 7f63b5600640 1 --2- 172.21.15.27:0/610520007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b00ce2c0 0x7f63b014a2d0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f63a000c500 tx=0x7f63a000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:14.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.671+0000 7f63ad600640 1 -- 172.21.15.27:0/610520007 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a0005cf0 con 0x7f63b00ce2c0 2024-01-15T18:17:14.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.671+0000 7f63ad600640 1 -- 172.21.15.27:0/610520007 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63a0005e50 con 0x7f63b00ce2c0 2024-01-15T18:17:14.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.671+0000 7f63b8000640 1 -- 172.21.15.27:0/610520007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63b014e480 con 0x7f63b00ce2c0 2024-01-15T18:17:14.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.671+0000 7f63ad600640 1 -- 172.21.15.27:0/610520007 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a00072f0 con 0x7f63b00ce2c0 2024-01-15T18:17:14.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.671+0000 7f63b8000640 1 -- 172.21.15.27:0/610520007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b0149ad0 con 0x7f63b00ce2c0 2024-01-15T18:17:14.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.675+0000 7f63b8000640 1 -- 172.21.15.27:0/610520007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b00d09f0 con 0x7f63b00ce2c0 2024-01-15T18:17:14.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.675+0000 7f63ad600640 1 -- 172.21.15.27:0/610520007 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f63a0007450 con 0x7f63b00ce2c0 2024-01-15T18:17:14.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.675+0000 7f63ad600640 1 --2- 172.21.15.27:0/610520007 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6388067700 0x7f6388069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:14.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.675+0000 7f63ad600640 1 -- 172.21.15.27:0/610520007 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 4969+0+0 (secure 0 0 0) 0x7f63a0082760 con 0x7f63b00ce2c0 2024-01-15T18:17:14.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.675+0000 7f63b4c00640 1 --2- 172.21.15.27:0/610520007 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6388067700 0x7f6388069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:14.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.675+0000 7f63b4c00640 1 --2- 172.21.15.27:0/610520007 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6388067700 0x7f6388069bc0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f63b0151e20 tx=0x7f6398008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:14.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.679+0000 7f63ad600640 1 -- 172.21.15.27:0/610520007 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f63a00b6280 con 0x7f63b00ce2c0 2024-01-15T18:17:14.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:14.835+0000 7f63b8000640 1 -- 172.21.15.27:0/610520007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f63b004e850 con 0x7f63b00ce2c0 2024-01-15T18:17:15.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.083+0000 7f63ad600640 1 -- 172.21.15.27:0/610520007 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v94) v1 ==== 138+0+0 (secure 0 0 0) 0x7f63a0017020 con 0x7f63b00ce2c0 2024-01-15T18:17:15.081 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-01-15T18:17:15.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.083+0000 7f63b8000640 1 -- 172.21.15.27:0/610520007 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6388067700 msgr2=0x7f6388069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:15.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.083+0000 7f63b8000640 1 --2- 172.21.15.27:0/610520007 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6388067700 0x7f6388069bc0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f63b0151e20 tx=0x7f6398008040 comp rx=0 tx=0).stop 2024-01-15T18:17:15.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.083+0000 7f63b8000640 1 -- 172.21.15.27:0/610520007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b00ce2c0 msgr2=0x7f63b014a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:15.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.083+0000 7f63b8000640 1 --2- 172.21.15.27:0/610520007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b00ce2c0 0x7f63b014a2d0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f63a000c500 tx=0x7f63a000c9c0 comp rx=0 tx=0).stop 2024-01-15T18:17:15.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.083+0000 7f63b8000640 1 -- 172.21.15.27:0/610520007 shutdown_connections 2024-01-15T18:17:15.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.083+0000 7f63b8000640 1 --2- 172.21.15.27:0/610520007 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6388067700 0x7f6388069bc0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:15.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.083+0000 7f63b8000640 1 --2- 172.21.15.27:0/610520007 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f63b00d19f0 0x7f63b0148580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:15.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.083+0000 7f63b8000640 1 --2- 172.21.15.27:0/610520007 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f63b00cef20 0x7f63b015a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:15.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.083+0000 7f63b8000640 1 --2- 172.21.15.27:0/610520007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63b00ce2c0 0x7f63b014a2d0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:15.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.083+0000 7f63b8000640 1 -- 172.21.15.27:0/610520007 >> 172.21.15.27:0/610520007 conn(0x7f63b00d0ed0 msgr2=0x7f63b00b31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:15.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.087+0000 7f63b8000640 1 -- 172.21.15.27:0/610520007 shutdown_connections 2024-01-15T18:17:15.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.087+0000 7f63b8000640 1 -- 172.21.15.27:0/610520007 wait complete. 2024-01-15T18:17:15.109 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-15T18:17:15.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.259+0000 7f9ee1c00640 1 Processor -- start 2024-01-15T18:17:15.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.259+0000 7f9ee1c00640 1 -- start start 2024-01-15T18:17:15.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.259+0000 7f9ee1c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9edc0cec60 0x7f9edc0b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:15.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.259+0000 7f9ee1c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9edc0d19a0 con 0x7f9edc0cf460 2024-01-15T18:17:15.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.259+0000 7f9ee1c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9edc0d1c80 con 0x7f9edc15b4b0 2024-01-15T18:17:15.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.259+0000 7f9ee1c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc0b07c0 con 0x7f9edc0cec60 2024-01-15T18:17:15.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.259+0000 7f9edaa00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9edc0cf460 0x7f9edc0cf840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48824/0 (socket says 172.21.15.27:48824) 2024-01-15T18:17:15.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.259+0000 7f9edaa00640 1 -- 172.21.15.27:0/311499074 learned_addr learned my addr 172.21.15.27:0/311499074 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:15.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.259+0000 7f9edb400640 1 --2- 172.21.15.27:0/311499074 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9edc0cec60 0x7f9edc0b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:15.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.259+0000 7f9ed9600640 1 -- 172.21.15.27:0/311499074 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232620202 0 0) 0x7f9edc0d19a0 con 0x7f9edc0cf460 2024-01-15T18:17:15.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.259+0000 7f9ed9600640 1 -- 172.21.15.27:0/311499074 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ebc003530 con 0x7f9edc0cf460 2024-01-15T18:17:15.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.259+0000 7f9ed9600640 1 -- 172.21.15.27:0/311499074 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972381759 0 0) 0x7f9edc0d1c80 con 0x7f9edc15b4b0 2024-01-15T18:17:15.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.259+0000 7f9ed9600640 1 -- 172.21.15.27:0/311499074 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9edc0d19a0 con 0x7f9edc15b4b0 2024-01-15T18:17:15.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.259+0000 7f9edb400640 1 -- 172.21.15.27:0/311499074 >> v1:172.21.15.93:6789/0 conn(0x7f9edc15b4b0 legacy=0x7f9edc0b1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:15.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.259+0000 7f9edb400640 1 -- 172.21.15.27:0/311499074 >> v1:172.21.15.27:6789/0 conn(0x7f9edc0cf460 legacy=0x7f9edc0cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:15.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.259+0000 7f9edb400640 1 -- 172.21.15.27:0/311499074 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9edc0b0aa0 con 0x7f9edc0cec60 2024-01-15T18:17:15.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.263+0000 7f9edb400640 1 --2- 172.21.15.27:0/311499074 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9edc0cec60 0x7f9edc0b1c90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4009dc0 tx=0x7f9ed401c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bb31a5de844fd1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:15.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.263+0000 7f9ed9600640 1 -- 172.21.15.27:0/311499074 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed40a5a30 con 0x7f9edc0cec60 2024-01-15T18:17:15.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.263+0000 7f9ed9600640 1 -- 172.21.15.27:0/311499074 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed40a5b90 con 0x7f9edc0cec60 2024-01-15T18:17:15.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.263+0000 7f9ed9600640 1 -- 172.21.15.27:0/311499074 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed40ad440 con 0x7f9edc0cec60 2024-01-15T18:17:15.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.263+0000 7f9ee1c00640 1 -- 172.21.15.27:0/311499074 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9edc0cec60 msgr2=0x7f9edc0b1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:15.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.263+0000 7f9ee1c00640 1 --2- 172.21.15.27:0/311499074 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9edc0cec60 0x7f9edc0b1c90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4009dc0 tx=0x7f9ed401c970 comp rx=0 tx=0).stop 2024-01-15T18:17:15.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.263+0000 7f9ee1c00640 1 -- 172.21.15.27:0/311499074 shutdown_connections 2024-01-15T18:17:15.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.263+0000 7f9ee1c00640 1 --2- 172.21.15.27:0/311499074 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9edc0cec60 0x7f9edc0b1c90 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:15.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.263+0000 7f9ee1c00640 1 -- 172.21.15.27:0/311499074 >> 172.21.15.27:0/311499074 conn(0x7f9edc0d0e80 msgr2=0x7f9edc0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:15.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.267+0000 7f9ee1c00640 1 -- 172.21.15.27:0/311499074 shutdown_connections 2024-01-15T18:17:15.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.267+0000 7f9ee1c00640 1 -- 172.21.15.27:0/311499074 wait complete. 2024-01-15T18:17:15.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.267+0000 7f9ee1c00640 1 Processor -- start 2024-01-15T18:17:15.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.267+0000 7f9ee1c00640 1 -- start start 2024-01-15T18:17:15.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.267+0000 7f9ee1c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9edc0cec60 0x7f9edc14f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:15.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.267+0000 7f9ee1c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9edc0cf460 0x7f9edc0c8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:15.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.267+0000 7f9ee1c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9edc15b4b0 0x7f9edc0c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:15.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.267+0000 7f9ee1c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc15b890 con 0x7f9edc0cec60 2024-01-15T18:17:15.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.267+0000 7f9ee1c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc15be50 con 0x7f9edc0cf460 2024-01-15T18:17:15.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.267+0000 7f9ee1c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc14f8a0 con 0x7f9edc15b4b0 2024-01-15T18:17:15.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.267+0000 7f9edaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9edc0cec60 0x7f9edc14f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:15.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.267+0000 7f9edaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9edc0cec60 0x7f9edc14f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59532/0 (socket says 172.21.15.27:59532) 2024-01-15T18:17:15.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.267+0000 7f9edaa00640 1 -- 172.21.15.27:0/3520998231 learned_addr learned my addr 172.21.15.27:0/3520998231 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:15.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.267+0000 7f9edb400640 1 --2- 172.21.15.27:0/3520998231 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9edc15b4b0 0x7f9edc0c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:15.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.267+0000 7f9eda000640 1 --2- 172.21.15.27:0/3520998231 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9edc0cf460 0x7f9edc0c8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:15.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.267+0000 7f9edaa00640 1 -- 172.21.15.27:0/3520998231 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9edc15b4b0 msgr2=0x7f9edc0c94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:15.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.267+0000 7f9edaa00640 1 --2- 172.21.15.27:0/3520998231 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9edc15b4b0 0x7f9edc0c94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:15.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.267+0000 7f9edaa00640 1 -- 172.21.15.27:0/3520998231 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9edc0cf460 msgr2=0x7f9edc0c8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:15.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.267+0000 7f9edaa00640 1 --2- 172.21.15.27:0/3520998231 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9edc0cf460 0x7f9edc0c8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:15.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.267+0000 7f9edaa00640 1 -- 172.21.15.27:0/3520998231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed4009a70 con 0x7f9edc0cec60 2024-01-15T18:17:15.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.267+0000 7f9eda000640 1 --2- 172.21.15.27:0/3520998231 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9edc0cf460 0x7f9edc0c8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:17:15.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.267+0000 7f9edaa00640 1 --2- 172.21.15.27:0/3520998231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9edc0cec60 0x7f9edc14f360 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc00c500 tx=0x7f9ecc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:15.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.271+0000 7f9ed2a00640 1 -- 172.21.15.27:0/3520998231 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc005cf0 con 0x7f9edc0cec60 2024-01-15T18:17:15.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.271+0000 7f9ed2a00640 1 -- 172.21.15.27:0/3520998231 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ecc005e50 con 0x7f9edc0cec60 2024-01-15T18:17:15.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.271+0000 7f9ed2a00640 1 -- 172.21.15.27:0/3520998231 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc007330 con 0x7f9edc0cec60 2024-01-15T18:17:15.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.271+0000 7f9ee1c00640 1 -- 172.21.15.27:0/3520998231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9edc06d190 con 0x7f9edc0cec60 2024-01-15T18:17:15.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.271+0000 7f9ee1c00640 1 -- 172.21.15.27:0/3520998231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9edc15bae0 con 0x7f9edc0cec60 2024-01-15T18:17:15.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.271+0000 7f9ee1c00640 1 -- 172.21.15.27:0/3520998231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9edc04e850 con 0x7f9edc0cec60 2024-01-15T18:17:15.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.271+0000 7f9ed2a00640 1 -- 172.21.15.27:0/3520998231 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f9ecc007490 con 0x7f9edc0cec60 2024-01-15T18:17:15.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.275+0000 7f9ed2a00640 1 --2- 172.21.15.27:0/3520998231 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9eb4067700 0x7f9eb4069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:15.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.275+0000 7f9ed2a00640 1 -- 172.21.15.27:0/3520998231 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f9ecc082be0 con 0x7f9edc0cec60 2024-01-15T18:17:15.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.275+0000 7f9eda000640 1 --2- 172.21.15.27:0/3520998231 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9eb4067700 0x7f9eb4069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:15.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.275+0000 7f9eda000640 1 --2- 172.21.15.27:0/3520998231 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9eb4067700 0x7f9eb4069bc0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f9edc166df0 tx=0x7f9ec4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:15.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.279+0000 7f9ed2a00640 1 -- 172.21.15.27:0/3520998231 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9ecc0516a0 con 0x7f9edc0cec60 2024-01-15T18:17:15.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:15.435+0000 7f9ee1c00640 1 -- 172.21.15.27:0/3520998231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f9edc15eaf0 con 0x7f9edc0cec60 2024-01-15T18:17:16.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.086+0000 7f9ed2a00640 1 -- 172.21.15.27:0/3520998231 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v95) v1 ==== 170+0+0 (secure 0 0 0) 0x7f9ecc017020 con 0x7f9edc0cec60 2024-01-15T18:17:16.086 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-01-15T18:17:16.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.090+0000 7f9ee1c00640 1 -- 172.21.15.27:0/3520998231 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9eb4067700 msgr2=0x7f9eb4069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:16.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.090+0000 7f9ee1c00640 1 --2- 172.21.15.27:0/3520998231 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9eb4067700 0x7f9eb4069bc0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f9edc166df0 tx=0x7f9ec4009040 comp rx=0 tx=0).stop 2024-01-15T18:17:16.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.090+0000 7f9ee1c00640 1 -- 172.21.15.27:0/3520998231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9edc0cec60 msgr2=0x7f9edc14f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:16.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.090+0000 7f9ee1c00640 1 --2- 172.21.15.27:0/3520998231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9edc0cec60 0x7f9edc14f360 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc00c500 tx=0x7f9ecc00c9c0 comp rx=0 tx=0).stop 2024-01-15T18:17:16.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.090+0000 7f9ee1c00640 1 -- 172.21.15.27:0/3520998231 shutdown_connections 2024-01-15T18:17:16.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.090+0000 7f9ee1c00640 1 --2- 172.21.15.27:0/3520998231 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9eb4067700 0x7f9eb4069bc0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:16.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.090+0000 7f9ee1c00640 1 --2- 172.21.15.27:0/3520998231 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9edc15b4b0 0x7f9edc0c94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:16.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.090+0000 7f9ee1c00640 1 --2- 172.21.15.27:0/3520998231 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9edc0cf460 0x7f9edc0c8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:16.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.090+0000 7f9ee1c00640 1 --2- 172.21.15.27:0/3520998231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9edc0cec60 0x7f9edc14f360 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:16.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.090+0000 7f9ee1c00640 1 -- 172.21.15.27:0/3520998231 >> 172.21.15.27:0/3520998231 conn(0x7f9edc0d0e80 msgr2=0x7f9edc0cf840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:16.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.090+0000 7f9ee1c00640 1 -- 172.21.15.27:0/3520998231 shutdown_connections 2024-01-15T18:17:16.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.090+0000 7f9ee1c00640 1 -- 172.21.15.27:0/3520998231 wait complete. 2024-01-15T18:17:16.131 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-15T18:17:16.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.278+0000 7f2efc200640 1 Processor -- start 2024-01-15T18:17:16.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.278+0000 7f2efc200640 1 -- start start 2024-01-15T18:17:16.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.282+0000 7f2efc200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f2ef415b100 0x7f2ef4070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:16.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.282+0000 7f2efc200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2ef4071420 0x7f2ef415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:16.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.282+0000 7f2efc200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef4071860 con 0x7f2ef415ad20 2024-01-15T18:17:16.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.282+0000 7f2efc200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ef415a610 con 0x7f2ef4071420 2024-01-15T18:17:16.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.282+0000 7f2efc200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ef415e3c0 con 0x7f2ef415b100 2024-01-15T18:17:16.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.282+0000 7f2efa200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2ef4071420 0x7f2ef415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:16.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.282+0000 7f2efa200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2ef4071420 0x7f2ef415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59544/0 (socket says 172.21.15.27:59544) 2024-01-15T18:17:16.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.282+0000 7f2efa200640 1 -- 172.21.15.27:0/2268236176 learned_addr learned my addr 172.21.15.27:0/2268236176 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:16.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.282+0000 7f2ef8e00640 1 --2- 172.21.15.27:0/2268236176 >> v2:172.21.15.93:3300/0 conn(0x7f2ef415b100 0x7f2ef4070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:16.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.282+0000 7f2efa200640 1 -- 172.21.15.27:0/2268236176 >> v2:172.21.15.93:3300/0 conn(0x7f2ef415b100 msgr2=0x7f2ef4070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:16.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.282+0000 7f2efa200640 1 --2- 172.21.15.27:0/2268236176 >> v2:172.21.15.93:3300/0 conn(0x7f2ef415b100 0x7f2ef4070ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:16.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.282+0000 7f2efa200640 1 -- 172.21.15.27:0/2268236176 >> v1:172.21.15.93:6789/0 conn(0x7f2ef415ad20 legacy=0x7f2ef415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:16.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.282+0000 7f2efa200640 1 -- 172.21.15.27:0/2268236176 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef415e6a0 con 0x7f2ef4071420 2024-01-15T18:17:16.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.282+0000 7f2efa200640 1 --2- 172.21.15.27:0/2268236176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ef4071420 0x7f2ef415bee0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f2eec009e80 tx=0x7f2eec01c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96a2cf615b313cbe server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:16.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.286+0000 7f2ef3e00640 1 -- 172.21.15.27:0/2268236176 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec0a57b0 con 0x7f2ef4071420 2024-01-15T18:17:16.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.286+0000 7f2ef3e00640 1 -- 172.21.15.27:0/2268236176 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eec0a5910 con 0x7f2ef4071420 2024-01-15T18:17:16.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.286+0000 7f2ef3e00640 1 -- 172.21.15.27:0/2268236176 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec0a5ca0 con 0x7f2ef4071420 2024-01-15T18:17:16.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.286+0000 7f2efc200640 1 -- 172.21.15.27:0/2268236176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ef4071420 msgr2=0x7f2ef415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:16.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.286+0000 7f2efc200640 1 --2- 172.21.15.27:0/2268236176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ef4071420 0x7f2ef415bee0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f2eec009e80 tx=0x7f2eec01c8f0 comp rx=0 tx=0).stop 2024-01-15T18:17:16.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.286+0000 7f2efc200640 1 -- 172.21.15.27:0/2268236176 shutdown_connections 2024-01-15T18:17:16.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.286+0000 7f2efc200640 1 --2- 172.21.15.27:0/2268236176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ef4071420 0x7f2ef415bee0 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:16.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.286+0000 7f2efc200640 1 --2- 172.21.15.27:0/2268236176 >> v2:172.21.15.93:3300/0 conn(0x7f2ef415b100 0x7f2ef4070ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:16.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.286+0000 7f2efc200640 1 -- 172.21.15.27:0/2268236176 >> 172.21.15.27:0/2268236176 conn(0x7f2ef40cb700 msgr2=0x7f2ef40cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:16.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.286+0000 7f2efc200640 1 -- 172.21.15.27:0/2268236176 shutdown_connections 2024-01-15T18:17:16.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.286+0000 7f2efc200640 1 -- 172.21.15.27:0/2268236176 wait complete. 2024-01-15T18:17:16.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.286+0000 7f2efc200640 1 Processor -- start 2024-01-15T18:17:16.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.286+0000 7f2efc200640 1 -- start start 2024-01-15T18:17:16.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.286+0000 7f2efc200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ef4071420 0x7f2ef4149550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:16.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.286+0000 7f2efc200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2ef415ad20 0x7f2ef406c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:16.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.286+0000 7f2efc200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ef415b100 0x7f2ef4148b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:16.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.286+0000 7f2efc200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef414f850 con 0x7f2ef415b100 2024-01-15T18:17:16.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.286+0000 7f2efc200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef4148700 con 0x7f2ef4071420 2024-01-15T18:17:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.286+0000 7f2efc200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef4164550 con 0x7f2ef415ad20 2024-01-15T18:17:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.290+0000 7f2efa200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ef415b100 0x7f2ef4148b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.290+0000 7f2efa200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ef415b100 0x7f2ef4148b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59552/0 (socket says 172.21.15.27:59552) 2024-01-15T18:17:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.290+0000 7f2efa200640 1 -- 172.21.15.27:0/217896386 learned_addr learned my addr 172.21.15.27:0/217896386 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.290+0000 7f2ef9800640 1 --2- 172.21.15.27:0/217896386 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ef4071420 0x7f2ef4149550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.290+0000 7f2ef8e00640 1 --2- 172.21.15.27:0/217896386 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2ef415ad20 0x7f2ef406c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.290+0000 7f2efa200640 1 -- 172.21.15.27:0/217896386 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2ef415ad20 msgr2=0x7f2ef406c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.290+0000 7f2efa200640 1 --2- 172.21.15.27:0/217896386 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2ef415ad20 0x7f2ef406c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.290+0000 7f2efa200640 1 -- 172.21.15.27:0/217896386 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ef4071420 msgr2=0x7f2ef4149550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.290+0000 7f2efa200640 1 --2- 172.21.15.27:0/217896386 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ef4071420 0x7f2ef4149550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.290+0000 7f2efa200640 1 -- 172.21.15.27:0/217896386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eec009b30 con 0x7f2ef415b100 2024-01-15T18:17:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.290+0000 7f2efa200640 1 --2- 172.21.15.27:0/217896386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ef415b100 0x7f2ef4148b90 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f2eec009e80 tx=0x7f2eec019690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:16.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.290+0000 7f2ef1600640 1 -- 172.21.15.27:0/217896386 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec0b8670 con 0x7f2ef415b100 2024-01-15T18:17:16.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.290+0000 7f2ef1600640 1 -- 172.21.15.27:0/217896386 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eec0b87d0 con 0x7f2ef415b100 2024-01-15T18:17:16.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.290+0000 7f2efc200640 1 -- 172.21.15.27:0/217896386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef4164830 con 0x7f2ef415b100 2024-01-15T18:17:16.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.290+0000 7f2ef1600640 1 -- 172.21.15.27:0/217896386 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec0b8b60 con 0x7f2ef415b100 2024-01-15T18:17:16.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.290+0000 7f2efc200640 1 -- 172.21.15.27:0/217896386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef4148390 con 0x7f2ef415b100 2024-01-15T18:17:16.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.290+0000 7f2efc200640 1 -- 172.21.15.27:0/217896386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef41488b0 con 0x7f2ef415b100 2024-01-15T18:17:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.294+0000 7f2ef1600640 1 -- 172.21.15.27:0/217896386 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f2eec0b8cc0 con 0x7f2ef415b100 2024-01-15T18:17:16.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.294+0000 7f2ef1600640 1 --2- 172.21.15.27:0/217896386 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2ecc067700 0x7f2ecc069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:16.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.294+0000 7f2ef1600640 1 -- 172.21.15.27:0/217896386 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f2eec11e6d0 con 0x7f2ef415b100 2024-01-15T18:17:16.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.294+0000 7f2ef9800640 1 --2- 172.21.15.27:0/217896386 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2ecc067700 0x7f2ecc069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:16.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.294+0000 7f2ef9800640 1 --2- 172.21.15.27:0/217896386 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2ecc067700 0x7f2ecc069bc0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f2ef414c400 tx=0x7f2ee4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:16.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.298+0000 7f2ef1600640 1 -- 172.21.15.27:0/217896386 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2eec0ed570 con 0x7f2ef415b100 2024-01-15T18:17:16.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:16.486+0000 7f2efc200640 1 -- 172.21.15.27:0/217896386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f2ef41488b0 con 0x7f2ef415b100 2024-01-15T18:17:17.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.114+0000 7f2ef1600640 1 -- 172.21.15.27:0/217896386 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v63) v1 ==== 174+0+0 (secure 0 0 0) 0x7f2ef41488b0 con 0x7f2ef415b100 2024-01-15T18:17:17.118 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 14 and data pool 15 2024-01-15T18:17:17.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.126+0000 7f2efc200640 1 -- 172.21.15.27:0/217896386 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2ecc067700 msgr2=0x7f2ecc069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:17.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.126+0000 7f2efc200640 1 --2- 172.21.15.27:0/217896386 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2ecc067700 0x7f2ecc069bc0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f2ef414c400 tx=0x7f2ee4009040 comp rx=0 tx=0).stop 2024-01-15T18:17:17.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.126+0000 7f2efc200640 1 -- 172.21.15.27:0/217896386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ef415b100 msgr2=0x7f2ef4148b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:17.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.126+0000 7f2efc200640 1 --2- 172.21.15.27:0/217896386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ef415b100 0x7f2ef4148b90 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f2eec009e80 tx=0x7f2eec019690 comp rx=0 tx=0).stop 2024-01-15T18:17:17.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.130+0000 7f2efc200640 1 -- 172.21.15.27:0/217896386 shutdown_connections 2024-01-15T18:17:17.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.130+0000 7f2efc200640 1 --2- 172.21.15.27:0/217896386 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2ecc067700 0x7f2ecc069bc0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:17.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.130+0000 7f2efc200640 1 --2- 172.21.15.27:0/217896386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ef415b100 0x7f2ef4148b90 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:17.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.130+0000 7f2efc200640 1 --2- 172.21.15.27:0/217896386 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2ef415ad20 0x7f2ef406c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:17.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.130+0000 7f2efc200640 1 --2- 172.21.15.27:0/217896386 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ef4071420 0x7f2ef4149550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:17.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.130+0000 7f2efc200640 1 -- 172.21.15.27:0/217896386 >> 172.21.15.27:0/217896386 conn(0x7f2ef40cb700 msgr2=0x7f2ef406dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:17.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.134+0000 7f2efc200640 1 -- 172.21.15.27:0/217896386 shutdown_connections 2024-01-15T18:17:17.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.134+0000 7f2efc200640 1 -- 172.21.15.27:0/217896386 wait complete. 2024-01-15T18:17:17.162 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:17:17.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.306+0000 7fbdd7200640 1 Processor -- start 2024-01-15T18:17:17.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.306+0000 7fbdd7200640 1 -- start start 2024-01-15T18:17:17.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.306+0000 7fbdd7200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fbdd00cdd10 0x7fbdd0070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:17.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.306+0000 7fbdd7200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd00ce920 con 0x7fbdd015e730 2024-01-15T18:17:17.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.306+0000 7fbdd7200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd00ce640 con 0x7fbdd006db80 2024-01-15T18:17:17.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.306+0000 7fbdd7200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdd00cdb30 con 0x7fbdd00cdd10 2024-01-15T18:17:17.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.306+0000 7fbdcf400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbdd015e730 0x7fbdd00cd420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48832/0 (socket says 172.21.15.27:48832) 2024-01-15T18:17:17.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.306+0000 7fbdcf400640 1 -- 172.21.15.27:0/4148834671 learned_addr learned my addr 172.21.15.27:0/4148834671 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:17.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.306+0000 7fbdd5200640 1 --2- 172.21.15.27:0/4148834671 >> v2:172.21.15.93:3300/0 conn(0x7fbdd00cdd10 0x7fbdd0070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:17.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.306+0000 7fbdcea00640 1 -- 172.21.15.27:0/4148834671 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432898362 0 0) 0x7fbdd00ce920 con 0x7fbdd015e730 2024-01-15T18:17:17.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.306+0000 7fbdcea00640 1 -- 172.21.15.27:0/4148834671 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdb0003530 con 0x7fbdd015e730 2024-01-15T18:17:17.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.306+0000 7fbdd5200640 1 -- 172.21.15.27:0/4148834671 >> v1:172.21.15.93:6789/0 conn(0x7fbdd006db80 legacy=0x7fbdd006df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:17.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.306+0000 7fbdd5200640 1 -- 172.21.15.27:0/4148834671 >> v1:172.21.15.27:6789/0 conn(0x7fbdd015e730 legacy=0x7fbdd00cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:17.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.306+0000 7fbdcea00640 1 -- 172.21.15.27:0/4148834671 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3585641290 0 0) 0x7fbdb0003530 con 0x7fbdd015e730 2024-01-15T18:17:17.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.306+0000 7fbdd5200640 1 -- 172.21.15.27:0/4148834671 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd00cc940 con 0x7fbdd00cdd10 2024-01-15T18:17:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.310+0000 7fbdd5200640 1 --2- 172.21.15.27:0/4148834671 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbdd00cdd10 0x7fbdd0070c20 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8009d90 tx=0x7fbdc801c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd4e1d1331783182 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.310+0000 7fbdcea00640 1 -- 172.21.15.27:0/4148834671 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb8001680 con 0x7fbdd00cdd10 2024-01-15T18:17:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.310+0000 7fbdcea00640 1 -- 172.21.15.27:0/4148834671 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc80a5650 con 0x7fbdd00cdd10 2024-01-15T18:17:17.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.310+0000 7fbdcea00640 1 -- 172.21.15.27:0/4148834671 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc80a59c0 con 0x7fbdd00cdd10 2024-01-15T18:17:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.310+0000 7fbdd7200640 1 -- 172.21.15.27:0/4148834671 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbdd00cdd10 msgr2=0x7fbdd0070c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.310+0000 7fbdd7200640 1 --2- 172.21.15.27:0/4148834671 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbdd00cdd10 0x7fbdd0070c20 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8009d90 tx=0x7fbdc801c840 comp rx=0 tx=0).stop 2024-01-15T18:17:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.310+0000 7fbdd7200640 1 -- 172.21.15.27:0/4148834671 shutdown_connections 2024-01-15T18:17:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.310+0000 7fbdd7200640 1 --2- 172.21.15.27:0/4148834671 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbdd00cdd10 0x7fbdd0070c20 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.310+0000 7fbdd7200640 1 -- 172.21.15.27:0/4148834671 >> 172.21.15.27:0/4148834671 conn(0x7fbdd00cec60 msgr2=0x7fbdd00cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.310+0000 7fbdd7200640 1 -- 172.21.15.27:0/4148834671 shutdown_connections 2024-01-15T18:17:17.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.310+0000 7fbdd7200640 1 -- 172.21.15.27:0/4148834671 wait complete. 2024-01-15T18:17:17.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.314+0000 7fbdd7200640 1 Processor -- start 2024-01-15T18:17:17.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.314+0000 7fbdd7200640 1 -- start start 2024-01-15T18:17:17.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.314+0000 7fbdd7200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbdd006db80 0x7fbdd0156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:17.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.314+0000 7fbdd7200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdd00cdd10 0x7fbdd0155890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:17.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.314+0000 7fbdd7200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbdd015e730 0x7fbdd0155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:17.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.314+0000 7fbdd7200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd0157420 con 0x7fbdd00cdd10 2024-01-15T18:17:17.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.314+0000 7fbdd7200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd0157700 con 0x7fbdd006db80 2024-01-15T18:17:17.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.314+0000 7fbdd7200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd01579e0 con 0x7fbdd015e730 2024-01-15T18:17:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.314+0000 7fbdcf400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdd00cdd10 0x7fbdd0155890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.314+0000 7fbdcf400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdd00cdd10 0x7fbdd0155890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59560/0 (socket says 172.21.15.27:59560) 2024-01-15T18:17:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.314+0000 7fbdcf400640 1 -- 172.21.15.27:0/6917335 learned_addr learned my addr 172.21.15.27:0/6917335 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.314+0000 7fbdd5200640 1 --2- 172.21.15.27:0/6917335 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbdd015e730 0x7fbdd0155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.314+0000 7fbdcfe00640 1 --2- 172.21.15.27:0/6917335 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbdd006db80 0x7fbdd0156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.314+0000 7fbdcf400640 1 -- 172.21.15.27:0/6917335 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbdd015e730 msgr2=0x7fbdd0155dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.314+0000 7fbdcf400640 1 --2- 172.21.15.27:0/6917335 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbdd015e730 0x7fbdd0155dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.314+0000 7fbdcf400640 1 -- 172.21.15.27:0/6917335 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbdd006db80 msgr2=0x7fbdd0156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.314+0000 7fbdcf400640 1 --2- 172.21.15.27:0/6917335 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbdd006db80 0x7fbdd0156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.314+0000 7fbdcf400640 1 -- 172.21.15.27:0/6917335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc8009a70 con 0x7fbdd00cdd10 2024-01-15T18:17:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.314+0000 7fbdcfe00640 1 --2- 172.21.15.27:0/6917335 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbdd006db80 0x7fbdd0156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:17:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.314+0000 7fbdcf400640 1 --2- 172.21.15.27:0/6917335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdd00cdd10 0x7fbdd0155890 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fbdb80011f0 tx=0x7fbdb800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.314+0000 7fbdc7e00640 1 -- 172.21.15.27:0/6917335 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb8005ae0 con 0x7fbdd00cdd10 2024-01-15T18:17:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.314+0000 7fbdc7e00640 1 -- 172.21.15.27:0/6917335 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdb8005c60 con 0x7fbdd00cdd10 2024-01-15T18:17:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.314+0000 7fbdd7200640 1 -- 172.21.15.27:0/6917335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd0157cc0 con 0x7fbdd00cdd10 2024-01-15T18:17:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.314+0000 7fbdc7e00640 1 -- 172.21.15.27:0/6917335 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb80072b0 con 0x7fbdd00cdd10 2024-01-15T18:17:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.314+0000 7fbdd7200640 1 -- 172.21.15.27:0/6917335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd00b1f70 con 0x7fbdd00cdd10 2024-01-15T18:17:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.318+0000 7fbdd7200640 1 -- 172.21.15.27:0/6917335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdd004e850 con 0x7fbdd00cdd10 2024-01-15T18:17:17.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.318+0000 7fbdc7e00640 1 -- 172.21.15.27:0/6917335 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7fbdb8007410 con 0x7fbdd00cdd10 2024-01-15T18:17:17.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.318+0000 7fbdc7e00640 1 --2- 172.21.15.27:0/6917335 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbda8067700 0x7fbda8069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:17.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.318+0000 7fbdc7e00640 1 -- 172.21.15.27:0/6917335 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fbdb8082870 con 0x7fbdd00cdd10 2024-01-15T18:17:17.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.318+0000 7fbdcfe00640 1 --2- 172.21.15.27:0/6917335 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbda8067700 0x7fbda8069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:17.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.318+0000 7fbdcfe00640 1 --2- 172.21.15.27:0/6917335 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbda8067700 0x7fbda8069bc0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0000f70 tx=0x7fbdc0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:17.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.322+0000 7fbdc7e00640 1 -- 172.21.15.27:0/6917335 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbdb8051240 con 0x7fbdd00cdd10 2024-01-15T18:17:17.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.478+0000 7fbdd7200640 1 -- 172.21.15.27:0/6917335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbdd006c800 con 0x7fbdd00cdd10 2024-01-15T18:17:17.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.478+0000 7fbdc7e00640 1 -- 172.21.15.27:0/6917335 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16602 (secure 0 0 0) 0x7fbdb8054860 con 0x7fbdd00cdd10 2024-01-15T18:17:17.480 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:17:17.480 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":96,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:17:16.491637+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:17:14.885774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-15T18:17:15.483714+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:17:17.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.482+0000 7fbdd7200640 1 -- 172.21.15.27:0/6917335 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbda8067700 msgr2=0x7fbda8069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:17.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.482+0000 7fbdd7200640 1 --2- 172.21.15.27:0/6917335 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbda8067700 0x7fbda8069bc0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0000f70 tx=0x7fbdc0009040 comp rx=0 tx=0).stop 2024-01-15T18:17:17.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.482+0000 7fbdd7200640 1 -- 172.21.15.27:0/6917335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdd00cdd10 msgr2=0x7fbdd0155890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:17.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.482+0000 7fbdd7200640 1 --2- 172.21.15.27:0/6917335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdd00cdd10 0x7fbdd0155890 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fbdb80011f0 tx=0x7fbdb800cb90 comp rx=0 tx=0).stop 2024-01-15T18:17:17.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.482+0000 7fbdd7200640 1 -- 172.21.15.27:0/6917335 shutdown_connections 2024-01-15T18:17:17.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.482+0000 7fbdd7200640 1 --2- 172.21.15.27:0/6917335 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbda8067700 0x7fbda8069bc0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:17.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.482+0000 7fbdd7200640 1 --2- 172.21.15.27:0/6917335 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbdd015e730 0x7fbdd0155dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:17.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.482+0000 7fbdd7200640 1 --2- 172.21.15.27:0/6917335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdd00cdd10 0x7fbdd0155890 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:17.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.482+0000 7fbdd7200640 1 --2- 172.21.15.27:0/6917335 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbdd006db80 0x7fbdd0156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:17.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.482+0000 7fbdd7200640 1 -- 172.21.15.27:0/6917335 >> 172.21.15.27:0/6917335 conn(0x7fbdd00cec60 msgr2=0x7fbdd00ad050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:17.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.482+0000 7fbdd7200640 1 -- 172.21.15.27:0/6917335 shutdown_connections 2024-01-15T18:17:17.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.482+0000 7fbdd7200640 1 -- 172.21.15.27:0/6917335 wait complete. 2024-01-15T18:17:17.502 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:17:17.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.642+0000 7f0333c00640 1 Processor -- start 2024-01-15T18:17:17.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.642+0000 7f0333c00640 1 -- start start 2024-01-15T18:17:17.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.642+0000 7f0333c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f032c0c92b0 0x7f032c15d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:17.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.642+0000 7f0333c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032c06c4a0 con 0x7f032c0701a0 2024-01-15T18:17:17.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.642+0000 7f0333c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032c15c660 con 0x7f032c15c130 2024-01-15T18:17:17.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.642+0000 7f0333c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f032c15c940 con 0x7f032c0c92b0 2024-01-15T18:17:17.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.646+0000 7f0331c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f032c0c92b0 0x7f032c15d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:17.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.646+0000 7f0331200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f032c0701a0 0x7f032c070580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48848/0 (socket says 172.21.15.27:48848) 2024-01-15T18:17:17.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.646+0000 7f0331200640 1 -- 172.21.15.27:0/3594685520 learned_addr learned my addr 172.21.15.27:0/3594685520 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:17.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.646+0000 7f0331c00640 1 -- 172.21.15.27:0/3594685520 >> v1:172.21.15.93:6789/0 conn(0x7f032c15c130 legacy=0x7f032c06dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:17.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.646+0000 7f0331c00640 1 -- 172.21.15.27:0/3594685520 >> v1:172.21.15.27:6789/0 conn(0x7f032c0701a0 legacy=0x7f032c070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:17.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.646+0000 7f0331c00640 1 -- 172.21.15.27:0/3594685520 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032c15cc20 con 0x7f032c0c92b0 2024-01-15T18:17:17.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.646+0000 7f032b400640 1 -- 172.21.15.27:0/3594685520 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2821607561 0 0) 0x7f032c06c4a0 con 0x7f032c0701a0 2024-01-15T18:17:17.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.646+0000 7f0331c00640 1 --2- 172.21.15.27:0/3594685520 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f032c0c92b0 0x7f032c15d9e0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f0324009d90 tx=0x7f032401c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58f037f3221328b7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:17.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.646+0000 7f032b400640 1 -- 172.21.15.27:0/3594685520 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03240a5950 con 0x7f032c0c92b0 2024-01-15T18:17:17.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.646+0000 7f032b400640 1 -- 172.21.15.27:0/3594685520 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03240a5ab0 con 0x7f032c0c92b0 2024-01-15T18:17:17.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.646+0000 7f032b400640 1 -- 172.21.15.27:0/3594685520 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03240a5e40 con 0x7f032c0c92b0 2024-01-15T18:17:17.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.650+0000 7f0333c00640 1 -- 172.21.15.27:0/3594685520 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f032c0c92b0 msgr2=0x7f032c15d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:17.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.650+0000 7f0333c00640 1 --2- 172.21.15.27:0/3594685520 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f032c0c92b0 0x7f032c15d9e0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f0324009d90 tx=0x7f032401c890 comp rx=0 tx=0).stop 2024-01-15T18:17:17.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.650+0000 7f0333c00640 1 -- 172.21.15.27:0/3594685520 shutdown_connections 2024-01-15T18:17:17.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.650+0000 7f0333c00640 1 --2- 172.21.15.27:0/3594685520 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f032c0c92b0 0x7f032c15d9e0 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:17.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.650+0000 7f0333c00640 1 -- 172.21.15.27:0/3594685520 >> 172.21.15.27:0/3594685520 conn(0x7f032c0cec60 msgr2=0x7f032c0cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:17.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.650+0000 7f0333c00640 1 -- 172.21.15.27:0/3594685520 shutdown_connections 2024-01-15T18:17:17.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.650+0000 7f0333c00640 1 -- 172.21.15.27:0/3594685520 wait complete. 2024-01-15T18:17:17.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.650+0000 7f0333c00640 1 Processor -- start 2024-01-15T18:17:17.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.650+0000 7f0333c00640 1 -- start start 2024-01-15T18:17:17.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.650+0000 7f0333c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f032c0701a0 0x7f032c153c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:17.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.650+0000 7f0333c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f032c0c92b0 0x7f032c1f8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:17.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.650+0000 7f0333c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f032c15c130 0x7f032c154260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:17.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.650+0000 7f0333c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032c154a90 con 0x7f032c15c130 2024-01-15T18:17:17.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.650+0000 7f0333c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032c1f75c0 con 0x7f032c0c92b0 2024-01-15T18:17:17.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.650+0000 7f0333c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f032c0ce460 con 0x7f032c0701a0 2024-01-15T18:17:17.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.650+0000 7f0331c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f032c15c130 0x7f032c154260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:17.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.650+0000 7f0331c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f032c15c130 0x7f032c154260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59574/0 (socket says 172.21.15.27:59574) 2024-01-15T18:17:17.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.650+0000 7f0331c00640 1 -- 172.21.15.27:0/712215470 learned_addr learned my addr 172.21.15.27:0/712215470 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:17.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.654+0000 7f032be00640 1 --2- 172.21.15.27:0/712215470 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f032c0c92b0 0x7f032c1f8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:17.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.654+0000 7f0331200640 1 --2- 172.21.15.27:0/712215470 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f032c0701a0 0x7f032c153c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:17.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.654+0000 7f0331c00640 1 -- 172.21.15.27:0/712215470 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f032c0701a0 msgr2=0x7f032c153c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:17.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.654+0000 7f0331c00640 1 --2- 172.21.15.27:0/712215470 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f032c0701a0 0x7f032c153c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:17.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.654+0000 7f0331c00640 1 -- 172.21.15.27:0/712215470 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f032c0c92b0 msgr2=0x7f032c1f8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:17.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.654+0000 7f0331c00640 1 --2- 172.21.15.27:0/712215470 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f032c0c92b0 0x7f032c1f8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:17.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.654+0000 7f0331c00640 1 -- 172.21.15.27:0/712215470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0324009a70 con 0x7f032c15c130 2024-01-15T18:17:17.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.654+0000 7f0331c00640 1 --2- 172.21.15.27:0/712215470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f032c15c130 0x7f032c154260 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f0324009d90 tx=0x7f0324019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.654+0000 7f0328c00640 1 -- 172.21.15.27:0/712215470 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03240ade40 con 0x7f032c15c130 2024-01-15T18:17:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.654+0000 7f0328c00640 1 -- 172.21.15.27:0/712215470 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03240b7410 con 0x7f032c15c130 2024-01-15T18:17:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.654+0000 7f0328c00640 1 -- 172.21.15.27:0/712215470 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03240b77c0 con 0x7f032c15c130 2024-01-15T18:17:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.654+0000 7f0333c00640 1 -- 172.21.15.27:0/712215470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f032c1f9460 con 0x7f032c15c130 2024-01-15T18:17:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.654+0000 7f0333c00640 1 -- 172.21.15.27:0/712215470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f032c0cde10 con 0x7f032c15c130 2024-01-15T18:17:17.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.654+0000 7f0333c00640 1 -- 172.21.15.27:0/712215470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f032c0ce5f0 con 0x7f032c15c130 2024-01-15T18:17:17.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.654+0000 7f0328c00640 1 -- 172.21.15.27:0/712215470 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f03240b4050 con 0x7f032c15c130 2024-01-15T18:17:17.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.658+0000 7f0328c00640 1 --2- 172.21.15.27:0/712215470 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0304067700 0x7f0304069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:17.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.658+0000 7f0328c00640 1 -- 172.21.15.27:0/712215470 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f032411e3b0 con 0x7f032c15c130 2024-01-15T18:17:17.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.658+0000 7f0331200640 1 --2- 172.21.15.27:0/712215470 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0304067700 0x7f0304069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:17.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.658+0000 7f0331200640 1 --2- 172.21.15.27:0/712215470 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0304067700 0x7f0304069bc0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f031c007fd0 tx=0x7f031c007ba0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:17.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.658+0000 7f0328c00640 1 -- 172.21.15.27:0/712215470 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f03240ed220 con 0x7f032c15c130 2024-01-15T18:17:17.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.818+0000 7f0333c00640 1 -- 172.21.15.27:0/712215470 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f032c04e850 con 0x7f032c15c130 2024-01-15T18:17:17.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.818+0000 7f0328c00640 1 -- 172.21.15.27:0/712215470 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16602 (secure 0 0 0) 0x7f03240b2090 con 0x7f032c15c130 2024-01-15T18:17:17.820 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:17:17.820 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":96,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:17:16.491637+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:17:14.885774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-15T18:17:15.483714+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:17:17.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.822+0000 7f0333c00640 1 -- 172.21.15.27:0/712215470 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0304067700 msgr2=0x7f0304069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:17.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.822+0000 7f0333c00640 1 --2- 172.21.15.27:0/712215470 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0304067700 0x7f0304069bc0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f031c007fd0 tx=0x7f031c007ba0 comp rx=0 tx=0).stop 2024-01-15T18:17:17.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.822+0000 7f0333c00640 1 -- 172.21.15.27:0/712215470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f032c15c130 msgr2=0x7f032c154260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:17.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.822+0000 7f0333c00640 1 --2- 172.21.15.27:0/712215470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f032c15c130 0x7f032c154260 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f0324009d90 tx=0x7f0324019630 comp rx=0 tx=0).stop 2024-01-15T18:17:17.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.822+0000 7f0333c00640 1 -- 172.21.15.27:0/712215470 shutdown_connections 2024-01-15T18:17:17.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.822+0000 7f0333c00640 1 --2- 172.21.15.27:0/712215470 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0304067700 0x7f0304069bc0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:17.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.822+0000 7f0333c00640 1 --2- 172.21.15.27:0/712215470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f032c15c130 0x7f032c154260 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:17.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.822+0000 7f0333c00640 1 --2- 172.21.15.27:0/712215470 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f032c0c92b0 0x7f032c1f8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:17.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.822+0000 7f0333c00640 1 --2- 172.21.15.27:0/712215470 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f032c0701a0 0x7f032c153c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:17.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.822+0000 7f0333c00640 1 -- 172.21.15.27:0/712215470 >> 172.21.15.27:0/712215470 conn(0x7f032c0cec60 msgr2=0x7f032c154c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:17.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.822+0000 7f0333c00640 1 -- 172.21.15.27:0/712215470 shutdown_connections 2024-01-15T18:17:17.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.822+0000 7f0333c00640 1 -- 172.21.15.27:0/712215470 wait complete. 2024-01-15T18:17:17.842 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-15T18:17:17.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.986+0000 7fe662200640 1 Processor -- start 2024-01-15T18:17:17.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.986+0000 7fe662200640 1 -- start start 2024-01-15T18:17:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe662200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fe65c06c430 0x7fe65c070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe662200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe65c0ccfb0 con 0x7fe65c0ce760 2024-01-15T18:17:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe662200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe65c0cd290 con 0x7fe65c0b1db0 2024-01-15T18:17:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe662200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe65c0cd570 con 0x7fe65c06c430 2024-01-15T18:17:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65be00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe65c0ce760 0x7fe65c06c810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48858/0 (socket says 172.21.15.27:48858) 2024-01-15T18:17:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65be00640 1 -- 172.21.15.27:0/1950375996 learned_addr learned my addr 172.21.15.27:0/1950375996 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65b400640 1 --2- 172.21.15.27:0/1950375996 >> v2:172.21.15.93:3300/0 conn(0x7fe65c06c430 0x7fe65c070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65a000640 1 -- 172.21.15.27:0/1950375996 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416029475 0 0) 0x7fe65c0ccfb0 con 0x7fe65c0ce760 2024-01-15T18:17:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65a000640 1 -- 172.21.15.27:0/1950375996 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe63c003530 con 0x7fe65c0ce760 2024-01-15T18:17:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65a000640 1 -- 172.21.15.27:0/1950375996 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 148517894 0 0) 0x7fe65c0cd290 con 0x7fe65c0b1db0 2024-01-15T18:17:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65a000640 1 -- 172.21.15.27:0/1950375996 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe65c0ccfb0 con 0x7fe65c0b1db0 2024-01-15T18:17:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65a000640 1 -- 172.21.15.27:0/1950375996 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 63319650 0 0) 0x7fe63c003530 con 0x7fe65c0ce760 2024-01-15T18:17:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65a000640 1 -- 172.21.15.27:0/1950375996 >> v2:172.21.15.93:3300/0 conn(0x7fe65c06c430 msgr2=0x7fe65c070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65a000640 1 --2- 172.21.15.27:0/1950375996 >> v2:172.21.15.93:3300/0 conn(0x7fe65c06c430 0x7fe65c070ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65a000640 1 -- 172.21.15.27:0/1950375996 >> v1:172.21.15.93:6789/0 conn(0x7fe65c0b1db0 legacy=0x7fe65c0ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65a000640 1 -- 172.21.15.27:0/1950375996 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe65c0cd850 con 0x7fe65c0ce760 2024-01-15T18:17:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65a000640 1 -- 172.21.15.27:0/1950375996 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fe654001a00 con 0x7fe65c0ce760 2024-01-15T18:17:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65b400640 1 --2- 172.21.15.27:0/1950375996 >> v2:172.21.15.93:3300/0 conn(0x7fe65c06c430 0x7fe65c070ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:17:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65a000640 1 -- 172.21.15.27:0/1950375996 >> v1:172.21.15.27:6789/0 conn(0x7fe65c0ce760 legacy=0x7fe65c06c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65a000640 1 --2- 172.21.15.27:0/1950375996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe63c003b80 0x7fe63c003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65a000640 1 --2- 172.21.15.27:0/1950375996 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe63c0047d0 0x7fe63c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65a000640 1 --2- 172.21.15.27:0/1950375996 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe63c0051e0 0x7fe63c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65a000640 1 -- 172.21.15.27:0/1950375996 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c0cd570 con 0x7fe63c003b80 2024-01-15T18:17:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65a000640 1 -- 172.21.15.27:0/1950375996 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63c00c130 con 0x7fe63c0047d0 2024-01-15T18:17:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65a000640 1 -- 172.21.15.27:0/1950375996 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe63c00c2f0 con 0x7fe63c0051e0 2024-01-15T18:17:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65b400640 1 --2- 172.21.15.27:0/1950375996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe63c003b80 0x7fe63c003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65be00640 1 --2- 172.21.15.27:0/1950375996 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe63c0051e0 0x7fe63c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65aa00640 1 --2- 172.21.15.27:0/1950375996 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe63c0047d0 0x7fe63c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65b400640 1 -- 172.21.15.27:0/1950375996 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe63c0051e0 msgr2=0x7fe63c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:17.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65b400640 1 --2- 172.21.15.27:0/1950375996 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe63c0051e0 0x7fe63c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:17.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65b400640 1 -- 172.21.15.27:0/1950375996 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe63c0047d0 msgr2=0x7fe63c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:17.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65b400640 1 --2- 172.21.15.27:0/1950375996 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe63c0047d0 0x7fe63c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:17.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65b400640 1 -- 172.21.15.27:0/1950375996 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63c003530 con 0x7fe63c003b80 2024-01-15T18:17:17.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.990+0000 7fe65be00640 1 -- 172.21.15.27:0/1950375996 reap_dead start 2024-01-15T18:17:17.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.994+0000 7fe65b400640 1 --2- 172.21.15.27:0/1950375996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe63c003b80 0x7fe63c003f80 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fe64c00ac80 tx=0x7fe64c01e0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88b5818fe5c67b66 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:17.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.994+0000 7fe65a000640 1 -- 172.21.15.27:0/1950375996 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64c0ad600 con 0x7fe63c003b80 2024-01-15T18:17:17.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.994+0000 7fe65a000640 1 -- 172.21.15.27:0/1950375996 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64c0a9050 con 0x7fe63c003b80 2024-01-15T18:17:17.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.994+0000 7fe65a000640 1 -- 172.21.15.27:0/1950375996 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64c0ad9e0 con 0x7fe63c003b80 2024-01-15T18:17:17.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.994+0000 7fe662200640 1 -- 172.21.15.27:0/1950375996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe63c003b80 msgr2=0x7fe63c003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:17.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.994+0000 7fe662200640 1 --2- 172.21.15.27:0/1950375996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe63c003b80 0x7fe63c003f80 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fe64c00ac80 tx=0x7fe64c01e0b0 comp rx=0 tx=0).stop 2024-01-15T18:17:17.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.994+0000 7fe662200640 1 -- 172.21.15.27:0/1950375996 shutdown_connections 2024-01-15T18:17:17.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.994+0000 7fe662200640 1 --2- 172.21.15.27:0/1950375996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe63c003b80 0x7fe63c003f80 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:17.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.994+0000 7fe662200640 1 -- 172.21.15.27:0/1950375996 >> 172.21.15.27:0/1950375996 conn(0x7fe65c0cb700 msgr2=0x7fe65c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:17.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.994+0000 7fe662200640 1 -- 172.21.15.27:0/1950375996 shutdown_connections 2024-01-15T18:17:17.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.994+0000 7fe662200640 1 -- 172.21.15.27:0/1950375996 wait complete. 2024-01-15T18:17:17.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.998+0000 7fe662200640 1 Processor -- start 2024-01-15T18:17:17.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.998+0000 7fe662200640 1 -- start start 2024-01-15T18:17:17.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.998+0000 7fe662200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe63c003b80 0x7fe65c154360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:17.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.998+0000 7fe662200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe65c1e4890 0x7fe65c0ac880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:17.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.998+0000 7fe662200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe65c1e4d10 0x7fe65c0acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:17.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.998+0000 7fe662200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c157180 con 0x7fe65c1e4d10 2024-01-15T18:17:17.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.998+0000 7fe662200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c154f20 con 0x7fe65c1e4890 2024-01-15T18:17:17.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.998+0000 7fe662200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c0adde0 con 0x7fe63c003b80 2024-01-15T18:17:17.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.998+0000 7fe65be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe65c1e4d10 0x7fe65c0acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:17.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.998+0000 7fe65be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe65c1e4d10 0x7fe65c0acdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59606/0 (socket says 172.21.15.27:59606) 2024-01-15T18:17:17.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.998+0000 7fe65be00640 1 -- 172.21.15.27:0/698806010 learned_addr learned my addr 172.21.15.27:0/698806010 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:17.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.998+0000 7fe65b400640 1 --2- 172.21.15.27:0/698806010 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe63c003b80 0x7fe65c154360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:18.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.998+0000 7fe65aa00640 1 --2- 172.21.15.27:0/698806010 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe65c1e4890 0x7fe65c0ac880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:18.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.998+0000 7fe65be00640 1 -- 172.21.15.27:0/698806010 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe63c003b80 msgr2=0x7fe65c154360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:18.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.998+0000 7fe65be00640 1 --2- 172.21.15.27:0/698806010 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe63c003b80 0x7fe65c154360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:18.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.998+0000 7fe65be00640 1 -- 172.21.15.27:0/698806010 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe65c1e4890 msgr2=0x7fe65c0ac880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:18.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.998+0000 7fe65be00640 1 --2- 172.21.15.27:0/698806010 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe65c1e4890 0x7fe65c0ac880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:18.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.998+0000 7fe65be00640 1 -- 172.21.15.27:0/698806010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe65c0ae0c0 con 0x7fe65c1e4d10 2024-01-15T18:17:18.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.998+0000 7fe65aa00640 1 --2- 172.21.15.27:0/698806010 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe65c1e4890 0x7fe65c0ac880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:17:18.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.998+0000 7fe65be00640 1 --2- 172.21.15.27:0/698806010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe65c1e4d10 0x7fe65c0acdc0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fe65c0cc6a0 tx=0x7fe654007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:18.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.998+0000 7fe653400640 1 -- 172.21.15.27:0/698806010 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65400a9b0 con 0x7fe65c1e4d10 2024-01-15T18:17:18.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.998+0000 7fe653400640 1 -- 172.21.15.27:0/698806010 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65400ab30 con 0x7fe65c1e4d10 2024-01-15T18:17:18.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.998+0000 7fe653400640 1 -- 172.21.15.27:0/698806010 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6540133f0 con 0x7fe65c1e4d10 2024-01-15T18:17:18.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.998+0000 7fe662200640 1 -- 172.21.15.27:0/698806010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe65c1548d0 con 0x7fe65c1e4d10 2024-01-15T18:17:18.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:17.998+0000 7fe662200640 1 -- 172.21.15.27:0/698806010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe65c0ada40 con 0x7fe65c1e4d10 2024-01-15T18:17:18.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:18.002+0000 7fe662200640 1 -- 172.21.15.27:0/698806010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe65c04e850 con 0x7fe65c1e4d10 2024-01-15T18:17:18.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:18.002+0000 7fe653400640 1 -- 172.21.15.27:0/698806010 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7fe654013550 con 0x7fe65c1e4d10 2024-01-15T18:17:18.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:18.002+0000 7fe653400640 1 --2- 172.21.15.27:0/698806010 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe634067780 0x7fe634069c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:18.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:18.002+0000 7fe653400640 1 -- 172.21.15.27:0/698806010 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fe654083690 con 0x7fe65c1e4d10 2024-01-15T18:17:18.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:18.002+0000 7fe65b400640 1 --2- 172.21.15.27:0/698806010 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe634067780 0x7fe634069c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:18.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:18.002+0000 7fe65b400640 1 --2- 172.21.15.27:0/698806010 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe634067780 0x7fe634069c40 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fe64c0af000 tx=0x7fe64c0046d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:18.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:18.006+0000 7fe653400640 1 -- 172.21.15.27:0/698806010 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe654052020 con 0x7fe65c1e4d10 2024-01-15T18:17:18.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:18.198+0000 7fe662200640 1 -- 172.21.15.27:0/698806010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe65c155060 con 0x7fe65c1e4d10 2024-01-15T18:17:19.079 INFO:tasks.ceph.mds.a.smithi027.stderr:2024-01-15T18:17:19.082+0000 7fedc9c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-15T18:17:19.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.130+0000 7fe653400640 1 -- 172.21.15.27:0/698806010 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v66) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe654055640 con 0x7fe65c1e4d10 2024-01-15T18:17:19.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.130+0000 7fe662200640 1 -- 172.21.15.27:0/698806010 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe634067780 msgr2=0x7fe634069c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:19.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.130+0000 7fe662200640 1 --2- 172.21.15.27:0/698806010 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe634067780 0x7fe634069c40 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fe64c0af000 tx=0x7fe64c0046d0 comp rx=0 tx=0).stop 2024-01-15T18:17:19.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.130+0000 7fe662200640 1 -- 172.21.15.27:0/698806010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe65c1e4d10 msgr2=0x7fe65c0acdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:19.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.130+0000 7fe662200640 1 --2- 172.21.15.27:0/698806010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe65c1e4d10 0x7fe65c0acdc0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fe65c0cc6a0 tx=0x7fe654007a20 comp rx=0 tx=0).stop 2024-01-15T18:17:19.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.130+0000 7fe662200640 1 -- 172.21.15.27:0/698806010 shutdown_connections 2024-01-15T18:17:19.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.130+0000 7fe662200640 1 --2- 172.21.15.27:0/698806010 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe634067780 0x7fe634069c40 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:19.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.130+0000 7fe662200640 1 --2- 172.21.15.27:0/698806010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe65c1e4d10 0x7fe65c0acdc0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:19.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.130+0000 7fe662200640 1 --2- 172.21.15.27:0/698806010 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe65c1e4890 0x7fe65c0ac880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:19.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.130+0000 7fe662200640 1 --2- 172.21.15.27:0/698806010 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe63c003b80 0x7fe65c154360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:19.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.130+0000 7fe662200640 1 -- 172.21.15.27:0/698806010 >> 172.21.15.27:0/698806010 conn(0x7fe65c0cb700 msgr2=0x7fe65c0caea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:19.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.130+0000 7fe662200640 1 -- 172.21.15.27:0/698806010 shutdown_connections 2024-01-15T18:17:19.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.130+0000 7fe662200640 1 -- 172.21.15.27:0/698806010 wait complete. 2024-01-15T18:17:19.156 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-15T18:17:19.156 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-15T18:17:19.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.306+0000 7ff2cca00640 1 Processor -- start 2024-01-15T18:17:19.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.306+0000 7ff2cca00640 1 -- start start 2024-01-15T18:17:19.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.306+0000 7ff2cca00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff2c40b1db0 0x7ff2c40b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:19.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.306+0000 7ff2cca00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff2c40ce050 0x7ff2c40ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:19.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.306+0000 7ff2cca00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2c40ce9e0 con 0x7ff2c406c430 2024-01-15T18:17:19.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.306+0000 7ff2cca00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2c40712b0 con 0x7ff2c40ce050 2024-01-15T18:17:19.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.306+0000 7ff2cca00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2c40ccfb0 con 0x7ff2c40b1db0 2024-01-15T18:17:19.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.306+0000 7ff2caa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff2c40ce050 0x7ff2c40ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:19.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.306+0000 7ff2caa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff2c40ce050 0x7ff2c40ce4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59616/0 (socket says 172.21.15.27:59616) 2024-01-15T18:17:19.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.306+0000 7ff2caa00640 1 -- 172.21.15.27:0/1955857845 learned_addr learned my addr 172.21.15.27:0/1955857845 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:19.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.306+0000 7ff2c9600640 1 --2- 172.21.15.27:0/1955857845 >> v2:172.21.15.93:3300/0 conn(0x7ff2c40b1db0 0x7ff2c40b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:19.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.306+0000 7ff2caa00640 1 -- 172.21.15.27:0/1955857845 >> v2:172.21.15.93:3300/0 conn(0x7ff2c40b1db0 msgr2=0x7ff2c40b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:19.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.306+0000 7ff2caa00640 1 --2- 172.21.15.27:0/1955857845 >> v2:172.21.15.93:3300/0 conn(0x7ff2c40b1db0 0x7ff2c40b2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:19.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.306+0000 7ff2caa00640 1 -- 172.21.15.27:0/1955857845 >> v1:172.21.15.93:6789/0 conn(0x7ff2c406c430 legacy=0x7ff2c4070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:19.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.306+0000 7ff2caa00640 1 -- 172.21.15.27:0/1955857845 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c40cd290 con 0x7ff2c40ce050 2024-01-15T18:17:19.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.310+0000 7ff2caa00640 1 --2- 172.21.15.27:0/1955857845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c40ce050 0x7ff2c40ce4a0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc009e50 tx=0x7ff2bc01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efa55b0663646b55 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:19.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.310+0000 7ff2c8c00640 1 -- 172.21.15.27:0/1955857845 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc0a5640 con 0x7ff2c40ce050 2024-01-15T18:17:19.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.310+0000 7ff2c8c00640 1 -- 172.21.15.27:0/1955857845 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2bc0a57a0 con 0x7ff2c40ce050 2024-01-15T18:17:19.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.310+0000 7ff2c8c00640 1 -- 172.21.15.27:0/1955857845 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc0a5b30 con 0x7ff2c40ce050 2024-01-15T18:17:19.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.310+0000 7ff2cca00640 1 -- 172.21.15.27:0/1955857845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c40ce050 msgr2=0x7ff2c40ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:19.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.310+0000 7ff2cca00640 1 --2- 172.21.15.27:0/1955857845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c40ce050 0x7ff2c40ce4a0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc009e50 tx=0x7ff2bc01c910 comp rx=0 tx=0).stop 2024-01-15T18:17:19.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.310+0000 7ff2cca00640 1 -- 172.21.15.27:0/1955857845 shutdown_connections 2024-01-15T18:17:19.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.310+0000 7ff2cca00640 1 --2- 172.21.15.27:0/1955857845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c40ce050 0x7ff2c40ce4a0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:19.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.310+0000 7ff2cca00640 1 --2- 172.21.15.27:0/1955857845 >> v2:172.21.15.93:3300/0 conn(0x7ff2c40b1db0 0x7ff2c40b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:19.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.310+0000 7ff2cca00640 1 -- 172.21.15.27:0/1955857845 >> 172.21.15.27:0/1955857845 conn(0x7ff2c40cb700 msgr2=0x7ff2c40cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:19.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.310+0000 7ff2cca00640 1 -- 172.21.15.27:0/1955857845 shutdown_connections 2024-01-15T18:17:19.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.310+0000 7ff2cca00640 1 -- 172.21.15.27:0/1955857845 wait complete. 2024-01-15T18:17:19.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.314+0000 7ff2cca00640 1 Processor -- start 2024-01-15T18:17:19.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.314+0000 7ff2cca00640 1 -- start start 2024-01-15T18:17:19.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.314+0000 7ff2cca00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2c406c430 0x7ff2c41662c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:19.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.314+0000 7ff2cca00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2c40b1db0 0x7ff2c4167e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:19.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.314+0000 7ff2cca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c40ce050 0x7ff2c4148600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:19.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.314+0000 7ff2cca00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c40aca20 con 0x7ff2c40ce050 2024-01-15T18:17:19.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.314+0000 7ff2cca00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c41439a0 con 0x7ff2c40b1db0 2024-01-15T18:17:19.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.314+0000 7ff2cca00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c4145a30 con 0x7ff2c406c430 2024-01-15T18:17:19.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.314+0000 7ff2caa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c40ce050 0x7ff2c4148600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:19.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.314+0000 7ff2caa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c40ce050 0x7ff2c4148600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59632/0 (socket says 172.21.15.27:59632) 2024-01-15T18:17:19.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.314+0000 7ff2caa00640 1 -- 172.21.15.27:0/774987655 learned_addr learned my addr 172.21.15.27:0/774987655 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:19.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.314+0000 7ff2c9600640 1 --2- 172.21.15.27:0/774987655 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2c40b1db0 0x7ff2c4167e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:19.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.314+0000 7ff2ca000640 1 --2- 172.21.15.27:0/774987655 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2c406c430 0x7ff2c41662c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:19.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.314+0000 7ff2caa00640 1 -- 172.21.15.27:0/774987655 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2c406c430 msgr2=0x7ff2c41662c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:19.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.314+0000 7ff2caa00640 1 --2- 172.21.15.27:0/774987655 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2c406c430 0x7ff2c41662c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:19.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.314+0000 7ff2caa00640 1 -- 172.21.15.27:0/774987655 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2c40b1db0 msgr2=0x7ff2c4167e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:19.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.314+0000 7ff2caa00640 1 --2- 172.21.15.27:0/774987655 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2c40b1db0 0x7ff2c4167e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:19.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.314+0000 7ff2caa00640 1 -- 172.21.15.27:0/774987655 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2bc009b30 con 0x7ff2c40ce050 2024-01-15T18:17:19.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.314+0000 7ff2caa00640 1 --2- 172.21.15.27:0/774987655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c40ce050 0x7ff2c4148600 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc019520 tx=0x7ff2bc0196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:19.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.314+0000 7ff2c2000640 1 -- 172.21.15.27:0/774987655 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc0a5640 con 0x7ff2c40ce050 2024-01-15T18:17:19.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.314+0000 7ff2c2000640 1 -- 172.21.15.27:0/774987655 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2bc0b8670 con 0x7ff2c40ce050 2024-01-15T18:17:19.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.314+0000 7ff2cca00640 1 -- 172.21.15.27:0/774987655 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2c4145cd0 con 0x7ff2c40ce050 2024-01-15T18:17:19.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.314+0000 7ff2c2000640 1 -- 172.21.15.27:0/774987655 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc0b89e0 con 0x7ff2c40ce050 2024-01-15T18:17:19.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.314+0000 7ff2cca00640 1 -- 172.21.15.27:0/774987655 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2c41668a0 con 0x7ff2c40ce050 2024-01-15T18:17:19.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.318+0000 7ff2cca00640 1 -- 172.21.15.27:0/774987655 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2c40acb60 con 0x7ff2c40ce050 2024-01-15T18:17:19.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.318+0000 7ff2c2000640 1 -- 172.21.15.27:0/774987655 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7ff2bc0b8b40 con 0x7ff2c40ce050 2024-01-15T18:17:19.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.318+0000 7ff2c2000640 1 --2- 172.21.15.27:0/774987655 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff29c067700 0x7ff29c069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:19.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.318+0000 7ff2c2000640 1 -- 172.21.15.27:0/774987655 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7ff2bc11e130 con 0x7ff2c40ce050 2024-01-15T18:17:19.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.318+0000 7ff2ca000640 1 --2- 172.21.15.27:0/774987655 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff29c067700 0x7ff29c069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:19.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.318+0000 7ff2ca000640 1 --2- 172.21.15.27:0/774987655 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff29c067700 0x7ff29c069bc0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7ff2c41f9660 tx=0x7ff2b4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:19.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.322+0000 7ff2c2000640 1 -- 172.21.15.27:0/774987655 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff2c40acb60 con 0x7ff2c40ce050 2024-01-15T18:17:19.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:19.510+0000 7ff2cca00640 1 -- 172.21.15.27:0/774987655 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff2c40acb60 con 0x7ff2c40ce050 2024-01-15T18:17:20.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.138+0000 7ff2c2000640 1 -- 172.21.15.27:0/774987655 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v67) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff2c40acb60 con 0x7ff2c40ce050 2024-01-15T18:17:20.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.138+0000 7ff2cca00640 1 -- 172.21.15.27:0/774987655 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff29c067700 msgr2=0x7ff29c069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:20.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.138+0000 7ff2cca00640 1 --2- 172.21.15.27:0/774987655 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff29c067700 0x7ff29c069bc0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7ff2c41f9660 tx=0x7ff2b4009040 comp rx=0 tx=0).stop 2024-01-15T18:17:20.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.138+0000 7ff2cca00640 1 -- 172.21.15.27:0/774987655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c40ce050 msgr2=0x7ff2c4148600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:20.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.138+0000 7ff2cca00640 1 --2- 172.21.15.27:0/774987655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c40ce050 0x7ff2c4148600 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc019520 tx=0x7ff2bc0196b0 comp rx=0 tx=0).stop 2024-01-15T18:17:20.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.138+0000 7ff2cca00640 1 -- 172.21.15.27:0/774987655 shutdown_connections 2024-01-15T18:17:20.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.138+0000 7ff2cca00640 1 --2- 172.21.15.27:0/774987655 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff29c067700 0x7ff29c069bc0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:20.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.138+0000 7ff2cca00640 1 --2- 172.21.15.27:0/774987655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c40ce050 0x7ff2c4148600 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:20.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.138+0000 7ff2cca00640 1 --2- 172.21.15.27:0/774987655 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2c40b1db0 0x7ff2c4167e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:20.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.138+0000 7ff2cca00640 1 --2- 172.21.15.27:0/774987655 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2c406c430 0x7ff2c41662c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:20.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.138+0000 7ff2cca00640 1 -- 172.21.15.27:0/774987655 >> 172.21.15.27:0/774987655 conn(0x7ff2c40cb700 msgr2=0x7ff2c40b4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:20.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.138+0000 7ff2cca00640 1 -- 172.21.15.27:0/774987655 shutdown_connections 2024-01-15T18:17:20.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.138+0000 7ff2cca00640 1 -- 172.21.15.27:0/774987655 wait complete. 2024-01-15T18:17:20.162 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-15T18:17:20.163 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:17:20.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.310+0000 7f8314000640 1 Processor -- start 2024-01-15T18:17:20.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.310+0000 7f8314000640 1 -- start start 2024-01-15T18:17:20.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.310+0000 7f8314000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f830c071e40 0x7f830c0c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:20.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.310+0000 7f8314000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f830c06d430 0x7f830c0cd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:20.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.310+0000 7f8314000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f830c159a50 con 0x7f830c159bf0 2024-01-15T18:17:20.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.310+0000 7f8314000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f830c0cce00 con 0x7f830c071e40 2024-01-15T18:17:20.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.310+0000 7f8314000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f830c06d980 con 0x7f830c06d430 2024-01-15T18:17:20.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.310+0000 7f8310c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f830c071e40 0x7f830c0c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:20.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.310+0000 7f8311600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f830c159bf0 0x7f830c159fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48866/0 (socket says 172.21.15.27:48866) 2024-01-15T18:17:20.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.310+0000 7f8311600640 1 -- 172.21.15.27:0/3039107695 learned_addr learned my addr 172.21.15.27:0/3039107695 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:20.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.310+0000 7f8312000640 1 --2- 172.21.15.27:0/3039107695 >> v2:172.21.15.93:3300/0 conn(0x7f830c06d430 0x7f830c0cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:20.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.310+0000 7f8310c00640 1 -- 172.21.15.27:0/3039107695 >> v2:172.21.15.93:3300/0 conn(0x7f830c06d430 msgr2=0x7f830c0cd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:20.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.310+0000 7f8310c00640 1 --2- 172.21.15.27:0/3039107695 >> v2:172.21.15.93:3300/0 conn(0x7f830c06d430 0x7f830c0cd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:20.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.310+0000 7f8310c00640 1 -- 172.21.15.27:0/3039107695 >> v1:172.21.15.27:6789/0 conn(0x7f830c159bf0 legacy=0x7f830c159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:20.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.310+0000 7f8310c00640 1 -- 172.21.15.27:0/3039107695 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830c06dc60 con 0x7f830c071e40 2024-01-15T18:17:20.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.314+0000 7f8310c00640 1 --2- 172.21.15.27:0/3039107695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f830c071e40 0x7f830c0c9be0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f830c15d090 tx=0x7f82f401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc7a256a1c7770df server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:20.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.314+0000 7f830be00640 1 -- 172.21.15.27:0/3039107695 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f40a8950 con 0x7f830c071e40 2024-01-15T18:17:20.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.314+0000 7f830be00640 1 -- 172.21.15.27:0/3039107695 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f40a8ab0 con 0x7f830c071e40 2024-01-15T18:17:20.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.314+0000 7f830be00640 1 -- 172.21.15.27:0/3039107695 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f40a8e40 con 0x7f830c071e40 2024-01-15T18:17:20.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.314+0000 7f8314000640 1 -- 172.21.15.27:0/3039107695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f830c071e40 msgr2=0x7f830c0c9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:20.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.314+0000 7f8314000640 1 --2- 172.21.15.27:0/3039107695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f830c071e40 0x7f830c0c9be0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f830c15d090 tx=0x7f82f401ec50 comp rx=0 tx=0).stop 2024-01-15T18:17:20.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.314+0000 7f8314000640 1 -- 172.21.15.27:0/3039107695 shutdown_connections 2024-01-15T18:17:20.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.314+0000 7f8314000640 1 --2- 172.21.15.27:0/3039107695 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f830c071e40 0x7f830c0c9be0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:20.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.314+0000 7f8314000640 1 --2- 172.21.15.27:0/3039107695 >> v2:172.21.15.93:3300/0 conn(0x7f830c06d430 0x7f830c0cd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:20.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.314+0000 7f8314000640 1 -- 172.21.15.27:0/3039107695 >> 172.21.15.27:0/3039107695 conn(0x7f830c0cec60 msgr2=0x7f830c0cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:20.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.314+0000 7f8314000640 1 -- 172.21.15.27:0/3039107695 shutdown_connections 2024-01-15T18:17:20.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.314+0000 7f8314000640 1 -- 172.21.15.27:0/3039107695 wait complete. 2024-01-15T18:17:20.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8314000640 1 Processor -- start 2024-01-15T18:17:20.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8314000640 1 -- start start 2024-01-15T18:17:20.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8314000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f830c06d430 0x7f830c158110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:20.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8314000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f830c159bf0 0x7f830c0b2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:20.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8314000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f830c0b2870 0x7f830c0c62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:20.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8314000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830c0b2d10 con 0x7f830c0b2870 2024-01-15T18:17:20.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8314000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830c0b2ff0 con 0x7f830c159bf0 2024-01-15T18:17:20.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8314000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f830c0b32d0 con 0x7f830c06d430 2024-01-15T18:17:20.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8312000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f830c0b2870 0x7f830c0c62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:20.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8312000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f830c0b2870 0x7f830c0c62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59650/0 (socket says 172.21.15.27:59650) 2024-01-15T18:17:20.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8312000640 1 -- 172.21.15.27:0/2716381399 learned_addr learned my addr 172.21.15.27:0/2716381399 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:20.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8311600640 1 --2- 172.21.15.27:0/2716381399 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f830c06d430 0x7f830c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:20.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8310c00640 1 --2- 172.21.15.27:0/2716381399 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f830c159bf0 0x7f830c0b2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:20.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8312000640 1 -- 172.21.15.27:0/2716381399 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f830c06d430 msgr2=0x7f830c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:20.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8312000640 1 --2- 172.21.15.27:0/2716381399 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f830c06d430 0x7f830c158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:20.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8312000640 1 -- 172.21.15.27:0/2716381399 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f830c159bf0 msgr2=0x7f830c0b2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:20.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8312000640 1 --2- 172.21.15.27:0/2716381399 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f830c159bf0 0x7f830c0b2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:20.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8312000640 1 -- 172.21.15.27:0/2716381399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f4009420 con 0x7f830c0b2870 2024-01-15T18:17:20.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8310c00640 1 --2- 172.21.15.27:0/2716381399 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f830c159bf0 0x7f830c0b2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:17:20.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8311600640 1 --2- 172.21.15.27:0/2716381399 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f830c06d430 0x7f830c158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:17:20.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8312000640 1 --2- 172.21.15.27:0/2716381399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f830c0b2870 0x7f830c0c62c0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f8304007d80 tx=0x7f830400d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:20.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8309600640 1 -- 172.21.15.27:0/2716381399 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f830400f040 con 0x7f830c0b2870 2024-01-15T18:17:20.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8309600640 1 -- 172.21.15.27:0/2716381399 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f830400dda0 con 0x7f830c0b2870 2024-01-15T18:17:20.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8309600640 1 -- 172.21.15.27:0/2716381399 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8304003be0 con 0x7f830c0b2870 2024-01-15T18:17:20.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8314000640 1 -- 172.21.15.27:0/2716381399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f830c0b35b0 con 0x7f830c0b2870 2024-01-15T18:17:20.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8314000640 1 -- 172.21.15.27:0/2716381399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f830c158650 con 0x7f830c0b2870 2024-01-15T18:17:20.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.318+0000 7f8314000640 1 -- 172.21.15.27:0/2716381399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f830c0ca0a0 con 0x7f830c0b2870 2024-01-15T18:17:20.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.322+0000 7f8309600640 1 -- 172.21.15.27:0/2716381399 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f8304003d40 con 0x7f830c0b2870 2024-01-15T18:17:20.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.322+0000 7f8309600640 1 --2- 172.21.15.27:0/2716381399 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f82e4067700 0x7f82e4069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:20.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.322+0000 7f8309600640 1 -- 172.21.15.27:0/2716381399 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f83040832c0 con 0x7f830c0b2870 2024-01-15T18:17:20.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.322+0000 7f8311600640 1 --2- 172.21.15.27:0/2716381399 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f82e4067700 0x7f82e4069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:20.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.322+0000 7f8311600640 1 --2- 172.21.15.27:0/2716381399 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f82e4067700 0x7f82e4069bc0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f82fc005890 tx=0x7f82fc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:20.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.326+0000 7f8309600640 1 -- 172.21.15.27:0/2716381399 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8304051c90 con 0x7f830c0b2870 2024-01-15T18:17:20.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.514+0000 7f8314000640 1 -- 172.21.15.27:0/2716381399 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f830c04e850 con 0x7f830c0b2870 2024-01-15T18:17:20.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.514+0000 7f8309600640 1 -- 172.21.15.27:0/2716381399 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4912 (secure 0 0 0) 0x7f83040552b0 con 0x7f830c0b2870 2024-01-15T18:17:20.514 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:17:20.515 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/1905230324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":1905230324},{"type":"v1","addr":"172.21.15.93:6837","nonce":1905230324}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":8269,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/2926806399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":2926806399},{"type":"v1","addr":"172.21.15.93:6836","nonce":2926806399}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":8934,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1107490361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1107490361},{"type":"v1","addr":"172.21.15.27:6834","nonce":1107490361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:17:16.491967+0000","modified":"2024-01-15T18:17:19.516616+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8937":{"gid":8937,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/259930036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":259930036},{"type":"v1","addr":"172.21.15.27:6835","nonce":259930036}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-01-15T18:17:20.515 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 67 2024-01-15T18:17:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.518+0000 7f8314000640 1 -- 172.21.15.27:0/2716381399 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f82e4067700 msgr2=0x7f82e4069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.518+0000 7f8314000640 1 --2- 172.21.15.27:0/2716381399 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f82e4067700 0x7f82e4069bc0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f82fc005890 tx=0x7f82fc009040 comp rx=0 tx=0).stop 2024-01-15T18:17:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.518+0000 7f8314000640 1 -- 172.21.15.27:0/2716381399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f830c0b2870 msgr2=0x7f830c0c62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.518+0000 7f8314000640 1 --2- 172.21.15.27:0/2716381399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f830c0b2870 0x7f830c0c62c0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f8304007d80 tx=0x7f830400d490 comp rx=0 tx=0).stop 2024-01-15T18:17:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.518+0000 7f8314000640 1 -- 172.21.15.27:0/2716381399 shutdown_connections 2024-01-15T18:17:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.518+0000 7f8314000640 1 --2- 172.21.15.27:0/2716381399 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f82e4067700 0x7f82e4069bc0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.518+0000 7f8314000640 1 --2- 172.21.15.27:0/2716381399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f830c0b2870 0x7f830c0c62c0 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.518+0000 7f8314000640 1 --2- 172.21.15.27:0/2716381399 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f830c159bf0 0x7f830c0b2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.518+0000 7f8314000640 1 --2- 172.21.15.27:0/2716381399 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f830c06d430 0x7f830c158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:20.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.518+0000 7f8314000640 1 -- 172.21.15.27:0/2716381399 >> 172.21.15.27:0/2716381399 conn(0x7f830c0cec60 msgr2=0x7f830c0ce170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:20.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.518+0000 7f8314000640 1 -- 172.21.15.27:0/2716381399 shutdown_connections 2024-01-15T18:17:20.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.518+0000 7f8314000640 1 -- 172.21.15.27:0/2716381399 wait complete. 2024-01-15T18:17:20.537 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:17:20.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.678+0000 7f71a8c00640 1 Processor -- start 2024-01-15T18:17:20.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.678+0000 7f71a8c00640 1 -- start start 2024-01-15T18:17:20.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.678+0000 7f71a8c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71a40c92b0 0x7f71a415d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:20.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.678+0000 7f71a8c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71a406c4a0 con 0x7f71a40701a0 2024-01-15T18:17:20.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.678+0000 7f71a8c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71a415c660 con 0x7f71a415c130 2024-01-15T18:17:20.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.678+0000 7f71a8c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71a415c940 con 0x7f71a40c92b0 2024-01-15T18:17:20.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.682+0000 7f71a1e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f71a40701a0 0x7f71a4070580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48874/0 (socket says 172.21.15.27:48874) 2024-01-15T18:17:20.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.682+0000 7f71a1e00640 1 -- 172.21.15.27:0/1608504057 learned_addr learned my addr 172.21.15.27:0/1608504057 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:20.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.682+0000 7f71a2800640 1 --2- 172.21.15.27:0/1608504057 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71a40c92b0 0x7f71a415d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:20.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.682+0000 7f71a0a00640 1 -- 172.21.15.27:0/1608504057 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1640867724 0 0) 0x7f71a406c4a0 con 0x7f71a40701a0 2024-01-15T18:17:20.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.682+0000 7f71a0a00640 1 -- 172.21.15.27:0/1608504057 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7184003530 con 0x7f71a40701a0 2024-01-15T18:17:20.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.682+0000 7f71a0a00640 1 -- 172.21.15.27:0/1608504057 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266152324 0 0) 0x7f71a415c660 con 0x7f71a415c130 2024-01-15T18:17:20.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.682+0000 7f71a0a00640 1 -- 172.21.15.27:0/1608504057 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71a406c4a0 con 0x7f71a415c130 2024-01-15T18:17:20.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.682+0000 7f71a0a00640 1 -- 172.21.15.27:0/1608504057 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1962894521 0 0) 0x7f7184003530 con 0x7f71a40701a0 2024-01-15T18:17:20.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.682+0000 7f71a2800640 1 -- 172.21.15.27:0/1608504057 >> v1:172.21.15.93:6789/0 conn(0x7f71a415c130 legacy=0x7f71a406dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:20.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.682+0000 7f71a2800640 1 -- 172.21.15.27:0/1608504057 >> v1:172.21.15.27:6789/0 conn(0x7f71a40701a0 legacy=0x7f71a4070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:20.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.682+0000 7f71a2800640 1 -- 172.21.15.27:0/1608504057 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a415cc20 con 0x7f71a40c92b0 2024-01-15T18:17:20.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.682+0000 7f71a2800640 1 --2- 172.21.15.27:0/1608504057 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71a40c92b0 0x7f71a415d9e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f719c009dc0 tx=0x7f719c01c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d81e6f8b8be40eff server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:20.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.686+0000 7f71a0a00640 1 -- 172.21.15.27:0/1608504057 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7194001680 con 0x7f71a40c92b0 2024-01-15T18:17:20.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.686+0000 7f71a0a00640 1 -- 172.21.15.27:0/1608504057 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f719c0a56a0 con 0x7f71a40c92b0 2024-01-15T18:17:20.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.686+0000 7f71a0a00640 1 -- 172.21.15.27:0/1608504057 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719c0a5a30 con 0x7f71a40c92b0 2024-01-15T18:17:20.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.686+0000 7f71a8c00640 1 -- 172.21.15.27:0/1608504057 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71a40c92b0 msgr2=0x7f71a415d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:20.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.686+0000 7f71a8c00640 1 --2- 172.21.15.27:0/1608504057 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71a40c92b0 0x7f71a415d9e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f719c009dc0 tx=0x7f719c01c860 comp rx=0 tx=0).stop 2024-01-15T18:17:20.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.686+0000 7f71a8c00640 1 -- 172.21.15.27:0/1608504057 shutdown_connections 2024-01-15T18:17:20.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.686+0000 7f71a8c00640 1 --2- 172.21.15.27:0/1608504057 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71a40c92b0 0x7f71a415d9e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:20.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.686+0000 7f71a8c00640 1 -- 172.21.15.27:0/1608504057 >> 172.21.15.27:0/1608504057 conn(0x7f71a40cec60 msgr2=0x7f71a40cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:20.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.686+0000 7f71a8c00640 1 -- 172.21.15.27:0/1608504057 shutdown_connections 2024-01-15T18:17:20.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.686+0000 7f71a8c00640 1 -- 172.21.15.27:0/1608504057 wait complete. 2024-01-15T18:17:20.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.686+0000 7f71a8c00640 1 Processor -- start 2024-01-15T18:17:20.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.686+0000 7f71a8c00640 1 -- start start 2024-01-15T18:17:20.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.686+0000 7f71a8c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f71a40701a0 0x7f71a4153350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:20.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.690+0000 7f71a8c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71a40c92b0 0x7f71a40c2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:20.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.690+0000 7f71a8c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71a415c130 0x7f71a4153b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:20.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.690+0000 7f71a8c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a40af290 con 0x7f71a40c92b0 2024-01-15T18:17:20.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.690+0000 7f71a8c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a40ae680 con 0x7f71a40701a0 2024-01-15T18:17:20.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.690+0000 7f71a8c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71a40ae960 con 0x7f71a415c130 2024-01-15T18:17:20.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.690+0000 7f71a1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71a40c92b0 0x7f71a40c2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:20.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.690+0000 7f71a1400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71a40c92b0 0x7f71a40c2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59660/0 (socket says 172.21.15.27:59660) 2024-01-15T18:17:20.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.690+0000 7f71a1400640 1 -- 172.21.15.27:0/4192458964 learned_addr learned my addr 172.21.15.27:0/4192458964 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:20.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.690+0000 7f71a2800640 1 --2- 172.21.15.27:0/4192458964 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71a415c130 0x7f71a4153b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:20.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.690+0000 7f71a1e00640 1 --2- 172.21.15.27:0/4192458964 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f71a40701a0 0x7f71a4153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:20.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.690+0000 7f71a1400640 1 -- 172.21.15.27:0/4192458964 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71a415c130 msgr2=0x7f71a4153b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:20.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.690+0000 7f71a1400640 1 --2- 172.21.15.27:0/4192458964 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71a415c130 0x7f71a4153b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:20.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.690+0000 7f71a1400640 1 -- 172.21.15.27:0/4192458964 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f71a40701a0 msgr2=0x7f71a4153350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:20.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.690+0000 7f71a1400640 1 --2- 172.21.15.27:0/4192458964 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f71a40701a0 0x7f71a4153350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:20.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.690+0000 7f71a1400640 1 -- 172.21.15.27:0/4192458964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719c009a70 con 0x7f71a40c92b0 2024-01-15T18:17:20.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.690+0000 7f71a1e00640 1 --2- 172.21.15.27:0/4192458964 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f71a40701a0 0x7f71a4153350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:17:20.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.690+0000 7f71a1400640 1 --2- 172.21.15.27:0/4192458964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71a40c92b0 0x7f71a40c2670 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f718c00c430 tx=0x7f718c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:20.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.690+0000 7f719a000640 1 -- 172.21.15.27:0/4192458964 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718c0052d0 con 0x7f71a40c92b0 2024-01-15T18:17:20.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.690+0000 7f719a000640 1 -- 172.21.15.27:0/4192458964 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f718c005450 con 0x7f71a40c92b0 2024-01-15T18:17:20.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.690+0000 7f719a000640 1 -- 172.21.15.27:0/4192458964 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718c003840 con 0x7f71a40c92b0 2024-01-15T18:17:20.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.690+0000 7f71a8c00640 1 -- 172.21.15.27:0/4192458964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71a40aec40 con 0x7f71a40c92b0 2024-01-15T18:17:20.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.690+0000 7f71a8c00640 1 -- 172.21.15.27:0/4192458964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71a4153890 con 0x7f71a40c92b0 2024-01-15T18:17:20.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.690+0000 7f71a8c00640 1 -- 172.21.15.27:0/4192458964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71a404e850 con 0x7f71a40c92b0 2024-01-15T18:17:20.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.690+0000 7f719a000640 1 -- 172.21.15.27:0/4192458964 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f718c0039a0 con 0x7f71a40c92b0 2024-01-15T18:17:20.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.694+0000 7f719a000640 1 --2- 172.21.15.27:0/4192458964 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f717c067730 0x7f717c069bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:20.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.694+0000 7f719a000640 1 -- 172.21.15.27:0/4192458964 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f718c082780 con 0x7f71a40c92b0 2024-01-15T18:17:20.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.694+0000 7f71a1e00640 1 --2- 172.21.15.27:0/4192458964 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f717c067730 0x7f717c069bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:20.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.694+0000 7f71a1e00640 1 --2- 172.21.15.27:0/4192458964 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f717c067730 0x7f717c069bf0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f7194000f70 tx=0x7f7194009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:20.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.698+0000 7f719a000640 1 -- 172.21.15.27:0/4192458964 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f718c051180 con 0x7f71a40c92b0 2024-01-15T18:17:20.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.854+0000 7f71a8c00640 1 -- 172.21.15.27:0/4192458964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71a4158fa0 con 0x7f71a40c92b0 2024-01-15T18:17:20.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.854+0000 7f719a000640 1 -- 172.21.15.27:0/4192458964 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16589 (secure 0 0 0) 0x7f718c0547a0 con 0x7f71a40c92b0 2024-01-15T18:17:20.855 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:17:20.855 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":97,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:17:18.115138+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:17:14.885774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-15T18:17:15.483714+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:17:20.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.858+0000 7f71a8c00640 1 -- 172.21.15.27:0/4192458964 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f717c067730 msgr2=0x7f717c069bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:20.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.858+0000 7f71a8c00640 1 --2- 172.21.15.27:0/4192458964 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f717c067730 0x7f717c069bf0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f7194000f70 tx=0x7f7194009040 comp rx=0 tx=0).stop 2024-01-15T18:17:20.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.858+0000 7f71a8c00640 1 -- 172.21.15.27:0/4192458964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71a40c92b0 msgr2=0x7f71a40c2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:20.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.858+0000 7f71a8c00640 1 --2- 172.21.15.27:0/4192458964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71a40c92b0 0x7f71a40c2670 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f718c00c430 tx=0x7f718c00c8f0 comp rx=0 tx=0).stop 2024-01-15T18:17:20.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.858+0000 7f71a8c00640 1 -- 172.21.15.27:0/4192458964 shutdown_connections 2024-01-15T18:17:20.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.858+0000 7f71a8c00640 1 --2- 172.21.15.27:0/4192458964 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f717c067730 0x7f717c069bf0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:20.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.858+0000 7f71a8c00640 1 --2- 172.21.15.27:0/4192458964 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71a415c130 0x7f71a4153b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:20.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.858+0000 7f71a8c00640 1 --2- 172.21.15.27:0/4192458964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71a40c92b0 0x7f71a40c2670 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:20.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.858+0000 7f71a8c00640 1 --2- 172.21.15.27:0/4192458964 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f71a40701a0 0x7f71a4153350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:20.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.858+0000 7f71a8c00640 1 -- 172.21.15.27:0/4192458964 >> 172.21.15.27:0/4192458964 conn(0x7f71a40cec60 msgr2=0x7f71a4155340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:20.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.858+0000 7f71a8c00640 1 -- 172.21.15.27:0/4192458964 shutdown_connections 2024-01-15T18:17:20.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:20.858+0000 7f71a8c00640 1 -- 172.21.15.27:0/4192458964 wait complete. 2024-01-15T18:17:20.877 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-15T18:17:20.877 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:17:21.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.018+0000 7fd0c4e00640 1 Processor -- start 2024-01-15T18:17:21.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.018+0000 7fd0c4e00640 1 -- start start 2024-01-15T18:17:21.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.022+0000 7fd0c4e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd0c015c130 0x7fd0c006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:21.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.022+0000 7fd0c4e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0c006c4a0 con 0x7fd0c006e240 2024-01-15T18:17:21.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.022+0000 7fd0c4e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0c015c660 con 0x7fd0c00701a0 2024-01-15T18:17:21.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.022+0000 7fd0c4e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0c015c940 con 0x7fd0c015c130 2024-01-15T18:17:21.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.022+0000 7fd0bd400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd0c015c130 0x7fd0c006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:21.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.022+0000 7fd0be800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd0c006e240 0x7fd0c015d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48886/0 (socket says 172.21.15.27:48886) 2024-01-15T18:17:21.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.022+0000 7fd0be800640 1 -- 172.21.15.27:0/666000186 learned_addr learned my addr 172.21.15.27:0/666000186 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:21.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.022+0000 7fd0bd400640 1 -- 172.21.15.27:0/666000186 >> v1:172.21.15.93:6789/0 conn(0x7fd0c00701a0 legacy=0x7fd0c0070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:21.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.022+0000 7fd0bd400640 1 -- 172.21.15.27:0/666000186 >> v1:172.21.15.27:6789/0 conn(0x7fd0c006e240 legacy=0x7fd0c015d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:21.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.022+0000 7fd0bd400640 1 -- 172.21.15.27:0/666000186 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c015cc20 con 0x7fd0c015c130 2024-01-15T18:17:21.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.022+0000 7fd0bca00640 1 -- 172.21.15.27:0/666000186 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1748594660 0 0) 0x7fd0c006c4a0 con 0x7fd0c006e240 2024-01-15T18:17:21.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.022+0000 7fd0bd400640 1 --2- 172.21.15.27:0/666000186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0c015c130 0x7fd0c006dd00 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fd0a80097c0 tx=0x7fd0a801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38b80f76951ed93b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:21.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.026+0000 7fd0bca00640 1 -- 172.21.15.27:0/666000186 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a80ab610 con 0x7fd0c015c130 2024-01-15T18:17:21.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.026+0000 7fd0bca00640 1 -- 172.21.15.27:0/666000186 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a80ab770 con 0x7fd0c015c130 2024-01-15T18:17:21.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.026+0000 7fd0bca00640 1 -- 172.21.15.27:0/666000186 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a80abb00 con 0x7fd0c015c130 2024-01-15T18:17:21.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.026+0000 7fd0c4e00640 1 -- 172.21.15.27:0/666000186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0c015c130 msgr2=0x7fd0c006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:21.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.026+0000 7fd0c4e00640 1 --2- 172.21.15.27:0/666000186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0c015c130 0x7fd0c006dd00 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fd0a80097c0 tx=0x7fd0a801c410 comp rx=0 tx=0).stop 2024-01-15T18:17:21.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.026+0000 7fd0c4e00640 1 -- 172.21.15.27:0/666000186 shutdown_connections 2024-01-15T18:17:21.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.026+0000 7fd0c4e00640 1 --2- 172.21.15.27:0/666000186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0c015c130 0x7fd0c006dd00 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.026+0000 7fd0c4e00640 1 -- 172.21.15.27:0/666000186 >> 172.21.15.27:0/666000186 conn(0x7fd0c00cec60 msgr2=0x7fd0c00cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:21.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.026+0000 7fd0c4e00640 1 -- 172.21.15.27:0/666000186 shutdown_connections 2024-01-15T18:17:21.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.026+0000 7fd0c4e00640 1 -- 172.21.15.27:0/666000186 wait complete. 2024-01-15T18:17:21.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.026+0000 7fd0c4e00640 1 Processor -- start 2024-01-15T18:17:21.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.026+0000 7fd0c4e00640 1 -- start start 2024-01-15T18:17:21.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.030+0000 7fd0c4e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0c006e240 0x7fd0c0154340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:21.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.030+0000 7fd0c4e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd0c00701a0 0x7fd0c00ad7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:21.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.030+0000 7fd0c4e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd0c015c130 0x7fd0c00c32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:21.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.030+0000 7fd0c4e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c00ccbe0 con 0x7fd0c006e240 2024-01-15T18:17:21.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.030+0000 7fd0c4e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c00ae000 con 0x7fd0c00701a0 2024-01-15T18:17:21.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.030+0000 7fd0c4e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c00cce30 con 0x7fd0c015c130 2024-01-15T18:17:21.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.030+0000 7fd0bde00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0c006e240 0x7fd0c0154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:21.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.030+0000 7fd0bde00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0c006e240 0x7fd0c0154340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59684/0 (socket says 172.21.15.27:59684) 2024-01-15T18:17:21.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.030+0000 7fd0bde00640 1 -- 172.21.15.27:0/3989307564 learned_addr learned my addr 172.21.15.27:0/3989307564 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:21.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.030+0000 7fd0be800640 1 --2- 172.21.15.27:0/3989307564 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd0c015c130 0x7fd0c00c32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:21.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.030+0000 7fd0bd400640 1 --2- 172.21.15.27:0/3989307564 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd0c00701a0 0x7fd0c00ad7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:21.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.030+0000 7fd0bde00640 1 -- 172.21.15.27:0/3989307564 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd0c015c130 msgr2=0x7fd0c00c32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:21.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.030+0000 7fd0bde00640 1 --2- 172.21.15.27:0/3989307564 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd0c015c130 0x7fd0c00c32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.030+0000 7fd0bde00640 1 -- 172.21.15.27:0/3989307564 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd0c00701a0 msgr2=0x7fd0c00ad7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:21.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.030+0000 7fd0bde00640 1 --2- 172.21.15.27:0/3989307564 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd0c00701a0 0x7fd0c00ad7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.030+0000 7fd0bde00640 1 -- 172.21.15.27:0/3989307564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a8009420 con 0x7fd0c006e240 2024-01-15T18:17:21.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.030+0000 7fd0bde00640 1 --2- 172.21.15.27:0/3989307564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0c006e240 0x7fd0c0154340 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fd0b000c450 tx=0x7fd0b000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:21.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.030+0000 7fd0b6000640 1 -- 172.21.15.27:0/3989307564 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b0012830 con 0x7fd0c006e240 2024-01-15T18:17:21.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.030+0000 7fd0b6000640 1 -- 172.21.15.27:0/3989307564 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b00129b0 con 0x7fd0c006e240 2024-01-15T18:17:21.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.030+0000 7fd0c4e00640 1 -- 172.21.15.27:0/3989307564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0c00cd110 con 0x7fd0c006e240 2024-01-15T18:17:21.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.030+0000 7fd0b6000640 1 -- 172.21.15.27:0/3989307564 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b0012d60 con 0x7fd0c006e240 2024-01-15T18:17:21.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.030+0000 7fd0c4e00640 1 -- 172.21.15.27:0/3989307564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0c00adcf0 con 0x7fd0c006e240 2024-01-15T18:17:21.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.030+0000 7fd0c4e00640 1 -- 172.21.15.27:0/3989307564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0c004e850 con 0x7fd0c006e240 2024-01-15T18:17:21.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.030+0000 7fd0b6000640 1 -- 172.21.15.27:0/3989307564 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7fd0b001c460 con 0x7fd0c006e240 2024-01-15T18:17:21.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.034+0000 7fd0b6000640 1 --2- 172.21.15.27:0/3989307564 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd098067700 0x7fd098069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:21.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.034+0000 7fd0b6000640 1 -- 172.21.15.27:0/3989307564 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fd0b0082e80 con 0x7fd0c006e240 2024-01-15T18:17:21.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.034+0000 7fd0bd400640 1 --2- 172.21.15.27:0/3989307564 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd098067700 0x7fd098069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:21.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.034+0000 7fd0bd400640 1 --2- 172.21.15.27:0/3989307564 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd098067700 0x7fd098069bc0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fd0a80af000 tx=0x7fd0a80a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:21.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.038+0000 7fd0b6000640 1 -- 172.21.15.27:0/3989307564 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd0b0051960 con 0x7fd0c006e240 2024-01-15T18:17:21.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.202+0000 7fd0c4e00640 1 -- 172.21.15.27:0/3989307564 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd0c00b4b90 con 0x7fd098067700 2024-01-15T18:17:21.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.210+0000 7fd0b6000640 1 -- 172.21.15.27:0/3989307564 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+281032 (secure 0 0 0) 0x7fd0c00b4b90 con 0x7fd098067700 2024-01-15T18:17:21.214 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:17:21.217 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:17:21.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.214+0000 7fd0c4e00640 1 -- 172.21.15.27:0/3989307564 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd098067700 msgr2=0x7fd098069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:21.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.214+0000 7fd0c4e00640 1 --2- 172.21.15.27:0/3989307564 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd098067700 0x7fd098069bc0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fd0a80af000 tx=0x7fd0a80a7000 comp rx=0 tx=0).stop 2024-01-15T18:17:21.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.214+0000 7fd0c4e00640 1 -- 172.21.15.27:0/3989307564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0c006e240 msgr2=0x7fd0c0154340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:21.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.214+0000 7fd0c4e00640 1 --2- 172.21.15.27:0/3989307564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0c006e240 0x7fd0c0154340 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fd0b000c450 tx=0x7fd0b000c910 comp rx=0 tx=0).stop 2024-01-15T18:17:21.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.214+0000 7fd0c4e00640 1 -- 172.21.15.27:0/3989307564 shutdown_connections 2024-01-15T18:17:21.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.214+0000 7fd0c4e00640 1 --2- 172.21.15.27:0/3989307564 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd098067700 0x7fd098069bc0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.214+0000 7fd0c4e00640 1 --2- 172.21.15.27:0/3989307564 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd0c015c130 0x7fd0c00c32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.214+0000 7fd0c4e00640 1 --2- 172.21.15.27:0/3989307564 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd0c00701a0 0x7fd0c00ad7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.214+0000 7fd0c4e00640 1 --2- 172.21.15.27:0/3989307564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0c006e240 0x7fd0c0154340 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.214+0000 7fd0c4e00640 1 -- 172.21.15.27:0/3989307564 >> 172.21.15.27:0/3989307564 conn(0x7fd0c00cec60 msgr2=0x7fd0c00afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:21.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.214+0000 7fd0c4e00640 1 -- 172.21.15.27:0/3989307564 shutdown_connections 2024-01-15T18:17:21.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.214+0000 7fd0c4e00640 1 -- 172.21.15.27:0/3989307564 wait complete. 2024-01-15T18:17:21.234 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-01-15T18:17:20.354630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592562,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":182,"num_write_kb":2420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138,"ondisk_log_size":138,"up":104,"acting":104,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":71,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":542496,"kb_used_data":10272,"kb_used_omap":12,"kb_used_meta":532083,"kb_avail":754432224,"statfs":{"total":773094113280,"available":772538597376,"internally_reserved":0,"allocated":10518528,"data_stored":5203370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":544853552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2194,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.265190"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440653+0000","last_change":"2024-01-15T18:17:17.132525+0000","last_active":"2024-01-15T18:17:18.440653+0000","last_peered":"2024-01-15T18:17:18.440653+0000","last_clean":"2024-01-15T18:17:18.440653+0000","last_became_active":"2024-01-15T18:17:17.132261+0000","last_became_peered":"2024-01-15T18:17:17.132261+0000","last_unstale":"2024-01-15T18:17:18.440653+0000","last_undegraded":"2024-01-15T18:17:18.440653+0000","last_fullsized":"2024-01-15T18:17:18.440653+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:07.042399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439412+0000","last_change":"2024-01-15T18:17:17.129139+0000","last_active":"2024-01-15T18:17:18.439412+0000","last_peered":"2024-01-15T18:17:18.439412+0000","last_clean":"2024-01-15T18:17:18.439412+0000","last_became_active":"2024-01-15T18:17:17.126922+0000","last_became_peered":"2024-01-15T18:17:17.126922+0000","last_unstale":"2024-01-15T18:17:18.439412+0000","last_undegraded":"2024-01-15T18:17:18.439412+0000","last_fullsized":"2024-01-15T18:17:18.439412+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:40:32.654203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.132988+0000","last_change":"2024-01-15T18:17:17.117455+0000","last_active":"2024-01-15T18:17:18.132988+0000","last_peered":"2024-01-15T18:17:18.132988+0000","last_clean":"2024-01-15T18:17:18.132988+0000","last_became_active":"2024-01-15T18:17:17.117259+0000","last_became_peered":"2024-01-15T18:17:17.117259+0000","last_unstale":"2024-01-15T18:17:18.132988+0000","last_undegraded":"2024-01-15T18:17:18.132988+0000","last_fullsized":"2024-01-15T18:17:18.132988+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:23.944086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.132977+0000","last_change":"2024-01-15T18:17:17.126892+0000","last_active":"2024-01-15T18:17:18.132977+0000","last_peered":"2024-01-15T18:17:18.132977+0000","last_clean":"2024-01-15T18:17:18.132977+0000","last_became_active":"2024-01-15T18:17:17.125684+0000","last_became_peered":"2024-01-15T18:17:17.125684+0000","last_unstale":"2024-01-15T18:17:18.132977+0000","last_undegraded":"2024-01-15T18:17:18.132977+0000","last_fullsized":"2024-01-15T18:17:18.132977+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:54.730338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440809+0000","last_change":"2024-01-15T18:17:17.125211+0000","last_active":"2024-01-15T18:17:18.440809+0000","last_peered":"2024-01-15T18:17:18.440809+0000","last_clean":"2024-01-15T18:17:18.440809+0000","last_became_active":"2024-01-15T18:17:17.119327+0000","last_became_peered":"2024-01-15T18:17:17.119327+0000","last_unstale":"2024-01-15T18:17:18.440809+0000","last_undegraded":"2024-01-15T18:17:18.440809+0000","last_fullsized":"2024-01-15T18:17:18.440809+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:57:35.828970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.133012+0000","last_change":"2024-01-15T18:17:17.117786+0000","last_active":"2024-01-15T18:17:18.133012+0000","last_peered":"2024-01-15T18:17:18.133012+0000","last_clean":"2024-01-15T18:17:18.133012+0000","last_became_active":"2024-01-15T18:17:17.117505+0000","last_became_peered":"2024-01-15T18:17:17.117505+0000","last_unstale":"2024-01-15T18:17:18.133012+0000","last_undegraded":"2024-01-15T18:17:18.133012+0000","last_fullsized":"2024-01-15T18:17:18.133012+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:39:09.922990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440211+0000","last_change":"2024-01-15T18:17:17.125005+0000","last_active":"2024-01-15T18:17:18.440211+0000","last_peered":"2024-01-15T18:17:18.440211+0000","last_clean":"2024-01-15T18:17:18.440211+0000","last_became_active":"2024-01-15T18:17:17.124717+0000","last_became_peered":"2024-01-15T18:17:17.124717+0000","last_unstale":"2024-01-15T18:17:18.440211+0000","last_undegraded":"2024-01-15T18:17:18.440211+0000","last_fullsized":"2024-01-15T18:17:18.440211+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:48:30.181901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440100+0000","last_change":"2024-01-15T18:17:17.130054+0000","last_active":"2024-01-15T18:17:18.440100+0000","last_peered":"2024-01-15T18:17:18.440100+0000","last_clean":"2024-01-15T18:17:18.440100+0000","last_became_active":"2024-01-15T18:17:17.129779+0000","last_became_peered":"2024-01-15T18:17:17.129779+0000","last_unstale":"2024-01-15T18:17:18.440100+0000","last_undegraded":"2024-01-15T18:17:18.440100+0000","last_fullsized":"2024-01-15T18:17:18.440100+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:42:04.787652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440698+0000","last_change":"2024-01-15T18:17:17.132156+0000","last_active":"2024-01-15T18:17:18.440698+0000","last_peered":"2024-01-15T18:17:18.440698+0000","last_clean":"2024-01-15T18:17:18.440698+0000","last_became_active":"2024-01-15T18:17:17.131822+0000","last_became_peered":"2024-01-15T18:17:17.131822+0000","last_unstale":"2024-01-15T18:17:18.440698+0000","last_undegraded":"2024-01-15T18:17:18.440698+0000","last_fullsized":"2024-01-15T18:17:18.440698+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:33:59.696616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440705+0000","last_change":"2024-01-15T18:17:17.127796+0000","last_active":"2024-01-15T18:17:18.440705+0000","last_peered":"2024-01-15T18:17:18.440705+0000","last_clean":"2024-01-15T18:17:18.440705+0000","last_became_active":"2024-01-15T18:17:17.127390+0000","last_became_peered":"2024-01-15T18:17:17.127390+0000","last_unstale":"2024-01-15T18:17:18.440705+0000","last_undegraded":"2024-01-15T18:17:18.440705+0000","last_fullsized":"2024-01-15T18:17:18.440705+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:03:11.357766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1f","version":"96'4","reported_seq":16,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440784+0000","last_change":"2024-01-15T18:17:16.117003+0000","last_active":"2024-01-15T18:17:18.440784+0000","last_peered":"2024-01-15T18:17:18.440784+0000","last_clean":"2024-01-15T18:17:18.440784+0000","last_became_active":"2024-01-15T18:17:16.116625+0000","last_became_peered":"2024-01-15T18:17:16.116625+0000","last_unstale":"2024-01-15T18:17:18.440784+0000","last_undegraded":"2024-01-15T18:17:18.440784+0000","last_fullsized":"2024-01-15T18:17:18.440784+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:11:58.418627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.096546+0000","last_change":"2024-01-15T18:17:15.096546+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.096546+0000","last_undegraded":"2024-01-15T18:17:15.096546+0000","last_fullsized":"2024-01-15T18:17:15.096546+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1d","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440270+0000","last_change":"2024-01-15T18:17:16.115473+0000","last_active":"2024-01-15T18:17:18.440270+0000","last_peered":"2024-01-15T18:17:18.440270+0000","last_clean":"2024-01-15T18:17:18.440270+0000","last_became_active":"2024-01-15T18:17:16.115189+0000","last_became_peered":"2024-01-15T18:17:16.115189+0000","last_unstale":"2024-01-15T18:17:18.440270+0000","last_undegraded":"2024-01-15T18:17:18.440270+0000","last_fullsized":"2024-01-15T18:17:18.440270+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:58:47.776409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.133184+0000","last_change":"2024-01-15T18:17:17.127577+0000","last_active":"2024-01-15T18:17:18.133184+0000","last_peered":"2024-01-15T18:17:18.133184+0000","last_clean":"2024-01-15T18:17:18.133184+0000","last_became_active":"2024-01-15T18:17:17.125488+0000","last_became_peered":"2024-01-15T18:17:17.125488+0000","last_unstale":"2024-01-15T18:17:18.133184+0000","last_undegraded":"2024-01-15T18:17:18.133184+0000","last_fullsized":"2024-01-15T18:17:18.133184+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:23.103323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440326+0000","last_change":"2024-01-15T18:17:16.123895+0000","last_active":"2024-01-15T18:17:18.440326+0000","last_peered":"2024-01-15T18:17:18.440326+0000","last_clean":"2024-01-15T18:17:18.440326+0000","last_became_active":"2024-01-15T18:17:16.123325+0000","last_became_peered":"2024-01-15T18:17:16.123325+0000","last_unstale":"2024-01-15T18:17:18.440326+0000","last_undegraded":"2024-01-15T18:17:18.440326+0000","last_fullsized":"2024-01-15T18:17:18.440326+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:13:09.312116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.131438+0000","last_change":"2024-01-15T18:17:17.118290+0000","last_active":"2024-01-15T18:17:18.131438+0000","last_peered":"2024-01-15T18:17:18.131438+0000","last_clean":"2024-01-15T18:17:18.131438+0000","last_became_active":"2024-01-15T18:17:17.117943+0000","last_became_peered":"2024-01-15T18:17:17.117943+0000","last_unstale":"2024-01-15T18:17:18.131438+0000","last_undegraded":"2024-01-15T18:17:18.131438+0000","last_fullsized":"2024-01-15T18:17:18.131438+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:50:04.755553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.093523+0000","last_change":"2024-01-15T18:17:15.093523+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.093523+0000","last_undegraded":"2024-01-15T18:17:15.093523+0000","last_fullsized":"2024-01-15T18:17:15.093523+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1a","version":"96'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440314+0000","last_change":"2024-01-15T18:17:16.121483+0000","last_active":"2024-01-15T18:17:18.440314+0000","last_peered":"2024-01-15T18:17:18.440314+0000","last_clean":"2024-01-15T18:17:18.440314+0000","last_became_active":"2024-01-15T18:17:16.121199+0000","last_became_peered":"2024-01-15T18:17:16.121199+0000","last_unstale":"2024-01-15T18:17:18.440314+0000","last_undegraded":"2024-01-15T18:17:18.440314+0000","last_fullsized":"2024-01-15T18:17:18.440314+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:51:29.086954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.095582+0000","last_change":"2024-01-15T18:17:15.095582+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.095582+0000","last_undegraded":"2024-01-15T18:17:15.095582+0000","last_fullsized":"2024-01-15T18:17:15.095582+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.132487+0000","last_change":"2024-01-15T18:17:17.119962+0000","last_active":"2024-01-15T18:17:18.132487+0000","last_peered":"2024-01-15T18:17:18.132487+0000","last_clean":"2024-01-15T18:17:18.132487+0000","last_became_active":"2024-01-15T18:17:17.118612+0000","last_became_peered":"2024-01-15T18:17:17.118612+0000","last_unstale":"2024-01-15T18:17:18.132487+0000","last_undegraded":"2024-01-15T18:17:18.132487+0000","last_fullsized":"2024-01-15T18:17:18.132487+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:45:26.435028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.093925+0000","last_change":"2024-01-15T18:17:15.093925+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.093925+0000","last_undegraded":"2024-01-15T18:17:15.093925+0000","last_fullsized":"2024-01-15T18:17:15.093925+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440790+0000","last_change":"2024-01-15T18:17:17.124377+0000","last_active":"2024-01-15T18:17:18.440790+0000","last_peered":"2024-01-15T18:17:18.440790+0000","last_clean":"2024-01-15T18:17:18.440790+0000","last_became_active":"2024-01-15T18:17:17.124096+0000","last_became_peered":"2024-01-15T18:17:17.124096+0000","last_unstale":"2024-01-15T18:17:18.440790+0000","last_undegraded":"2024-01-15T18:17:18.440790+0000","last_fullsized":"2024-01-15T18:17:18.440790+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:56:21.720071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.093475+0000","last_change":"2024-01-15T18:17:15.093475+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.093475+0000","last_undegraded":"2024-01-15T18:17:15.093475+0000","last_fullsized":"2024-01-15T18:17:15.093475+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.133177+0000","last_change":"2024-01-15T18:17:17.122838+0000","last_active":"2024-01-15T18:17:18.133177+0000","last_peered":"2024-01-15T18:17:18.133177+0000","last_clean":"2024-01-15T18:17:18.133177+0000","last_became_active":"2024-01-15T18:17:17.122579+0000","last_became_peered":"2024-01-15T18:17:17.122579+0000","last_unstale":"2024-01-15T18:17:18.133177+0000","last_undegraded":"2024-01-15T18:17:18.133177+0000","last_fullsized":"2024-01-15T18:17:18.133177+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:13:53.357957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.095406+0000","last_change":"2024-01-15T18:17:15.095406+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.095406+0000","last_undegraded":"2024-01-15T18:17:15.095406+0000","last_fullsized":"2024-01-15T18:17:15.095406+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.096252+0000","last_change":"2024-01-15T18:17:15.096252+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.096252+0000","last_undegraded":"2024-01-15T18:17:15.096252+0000","last_fullsized":"2024-01-15T18:17:15.096252+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.14","version":"96'6","reported_seq":18,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.148952+0000","last_change":"2024-01-15T18:17:16.120030+0000","last_active":"2024-01-15T18:17:17.148952+0000","last_peered":"2024-01-15T18:17:17.148952+0000","last_clean":"2024-01-15T18:17:17.148952+0000","last_became_active":"2024-01-15T18:17:16.119705+0000","last_became_peered":"2024-01-15T18:17:16.119705+0000","last_unstale":"2024-01-15T18:17:17.148952+0000","last_undegraded":"2024-01-15T18:17:17.148952+0000","last_fullsized":"2024-01-15T18:17:17.148952+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:47:48.944557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.094637+0000","last_change":"2024-01-15T18:17:15.094637+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.094637+0000","last_undegraded":"2024-01-15T18:17:15.094637+0000","last_fullsized":"2024-01-15T18:17:15.094637+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.094404+0000","last_change":"2024-01-15T18:17:15.094404+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.094404+0000","last_undegraded":"2024-01-15T18:17:15.094404+0000","last_fullsized":"2024-01-15T18:17:15.094404+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.132434+0000","last_change":"2024-01-15T18:17:17.121456+0000","last_active":"2024-01-15T18:17:18.132434+0000","last_peered":"2024-01-15T18:17:18.132434+0000","last_clean":"2024-01-15T18:17:18.132434+0000","last_became_active":"2024-01-15T18:17:17.121179+0000","last_became_peered":"2024-01-15T18:17:17.121179+0000","last_unstale":"2024-01-15T18:17:18.132434+0000","last_undegraded":"2024-01-15T18:17:18.132434+0000","last_fullsized":"2024-01-15T18:17:18.132434+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:53:26.915291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.111628+0000","last_change":"2024-01-15T18:17:16.113394+0000","last_active":"2024-01-15T18:17:17.111628+0000","last_peered":"2024-01-15T18:17:17.111628+0000","last_clean":"2024-01-15T18:17:17.111628+0000","last_became_active":"2024-01-15T18:17:16.113105+0000","last_became_peered":"2024-01-15T18:17:16.113105+0000","last_unstale":"2024-01-15T18:17:17.111628+0000","last_undegraded":"2024-01-15T18:17:17.111628+0000","last_fullsized":"2024-01-15T18:17:17.111628+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:41:23.459232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.093447+0000","last_change":"2024-01-15T18:17:15.093447+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.093447+0000","last_undegraded":"2024-01-15T18:17:15.093447+0000","last_fullsized":"2024-01-15T18:17:15.093447+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.e","version":"96'2","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.143716+0000","last_change":"2024-01-15T18:17:16.108046+0000","last_active":"2024-01-15T18:17:17.143716+0000","last_peered":"2024-01-15T18:17:17.143716+0000","last_clean":"2024-01-15T18:17:17.143716+0000","last_became_active":"2024-01-15T18:17:16.107750+0000","last_became_peered":"2024-01-15T18:17:16.107750+0000","last_unstale":"2024-01-15T18:17:17.143716+0000","last_undegraded":"2024-01-15T18:17:17.143716+0000","last_fullsized":"2024-01-15T18:17:17.143716+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:14.320865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.131540+0000","last_change":"2024-01-15T18:17:17.123658+0000","last_active":"2024-01-15T18:17:18.131540+0000","last_peered":"2024-01-15T18:17:18.131540+0000","last_clean":"2024-01-15T18:17:18.131540+0000","last_became_active":"2024-01-15T18:17:17.123351+0000","last_became_peered":"2024-01-15T18:17:17.123351+0000","last_unstale":"2024-01-15T18:17:18.131540+0000","last_undegraded":"2024-01-15T18:17:18.131540+0000","last_fullsized":"2024-01-15T18:17:18.131540+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:38:11.462593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.095378+0000","last_change":"2024-01-15T18:17:15.095378+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.095378+0000","last_undegraded":"2024-01-15T18:17:15.095378+0000","last_fullsized":"2024-01-15T18:17:15.095378+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440407+0000","last_change":"2024-01-15T18:17:16.125410+0000","last_active":"2024-01-15T18:17:18.440407+0000","last_peered":"2024-01-15T18:17:18.440407+0000","last_clean":"2024-01-15T18:17:18.440407+0000","last_became_active":"2024-01-15T18:17:16.124440+0000","last_became_peered":"2024-01-15T18:17:16.124440+0000","last_unstale":"2024-01-15T18:17:18.440407+0000","last_undegraded":"2024-01-15T18:17:18.440407+0000","last_fullsized":"2024-01-15T18:17:18.440407+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:37:51.051323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.b","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440507+0000","last_change":"2024-01-15T18:17:16.120398+0000","last_active":"2024-01-15T18:17:18.440507+0000","last_peered":"2024-01-15T18:17:18.440507+0000","last_clean":"2024-01-15T18:17:18.440507+0000","last_became_active":"2024-01-15T18:17:16.119944+0000","last_became_peered":"2024-01-15T18:17:16.119944+0000","last_unstale":"2024-01-15T18:17:18.440507+0000","last_undegraded":"2024-01-15T18:17:18.440507+0000","last_fullsized":"2024-01-15T18:17:18.440507+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:54:02.313160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.131461+0000","last_change":"2024-01-15T18:17:16.102303+0000","last_active":"2024-01-15T18:17:18.131461+0000","last_peered":"2024-01-15T18:17:18.131461+0000","last_clean":"2024-01-15T18:17:18.131461+0000","last_became_active":"2024-01-15T18:17:16.099773+0000","last_became_peered":"2024-01-15T18:17:16.099773+0000","last_unstale":"2024-01-15T18:17:18.131461+0000","last_undegraded":"2024-01-15T18:17:18.131461+0000","last_fullsized":"2024-01-15T18:17:18.131461+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:40:51.210210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.131385+0000","last_change":"2024-01-15T18:17:16.119288+0000","last_active":"2024-01-15T18:17:18.131385+0000","last_peered":"2024-01-15T18:17:18.131385+0000","last_clean":"2024-01-15T18:17:18.131385+0000","last_became_active":"2024-01-15T18:17:16.118972+0000","last_became_peered":"2024-01-15T18:17:16.118972+0000","last_unstale":"2024-01-15T18:17:18.131385+0000","last_undegraded":"2024-01-15T18:17:18.131385+0000","last_fullsized":"2024-01-15T18:17:18.131385+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:14:29.755549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439494+0000","last_change":"2024-01-15T18:17:16.127573+0000","last_active":"2024-01-15T18:17:18.439494+0000","last_peered":"2024-01-15T18:17:18.439494+0000","last_clean":"2024-01-15T18:17:18.439494+0000","last_became_active":"2024-01-15T18:17:16.127194+0000","last_became_peered":"2024-01-15T18:17:16.127194+0000","last_unstale":"2024-01-15T18:17:18.439494+0000","last_undegraded":"2024-01-15T18:17:18.439494+0000","last_fullsized":"2024-01-15T18:17:18.439494+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:48:07.276946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440832+0000","last_change":"2024-01-15T18:17:17.132695+0000","last_active":"2024-01-15T18:17:18.440832+0000","last_peered":"2024-01-15T18:17:18.440832+0000","last_clean":"2024-01-15T18:17:18.440832+0000","last_became_active":"2024-01-15T18:17:17.132430+0000","last_became_peered":"2024-01-15T18:17:17.132430+0000","last_unstale":"2024-01-15T18:17:18.440832+0000","last_undegraded":"2024-01-15T18:17:18.440832+0000","last_fullsized":"2024-01-15T18:17:18.440832+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:14:02.211346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.132130+0000","last_change":"2024-01-15T18:17:16.102078+0000","last_active":"2024-01-15T18:17:18.132130+0000","last_peered":"2024-01-15T18:17:18.132130+0000","last_clean":"2024-01-15T18:17:18.132130+0000","last_became_active":"2024-01-15T18:17:16.101772+0000","last_became_peered":"2024-01-15T18:17:16.101772+0000","last_unstale":"2024-01-15T18:17:18.132130+0000","last_undegraded":"2024-01-15T18:17:18.132130+0000","last_fullsized":"2024-01-15T18:17:18.132130+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:28:05.931436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440855+0000","last_change":"2024-01-15T18:17:16.121035+0000","last_active":"2024-01-15T18:17:18.440855+0000","last_peered":"2024-01-15T18:17:18.440855+0000","last_clean":"2024-01-15T18:17:18.440855+0000","last_became_active":"2024-01-15T18:17:16.120714+0000","last_became_peered":"2024-01-15T18:17:16.120714+0000","last_unstale":"2024-01-15T18:17:18.440855+0000","last_undegraded":"2024-01-15T18:17:18.440855+0000","last_fullsized":"2024-01-15T18:17:18.440855+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:30:03.158862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440319+0000","last_change":"2024-01-15T18:17:16.114051+0000","last_active":"2024-01-15T18:17:18.440319+0000","last_peered":"2024-01-15T18:17:18.440319+0000","last_clean":"2024-01-15T18:17:18.440319+0000","last_became_active":"2024-01-15T18:17:16.113079+0000","last_became_peered":"2024-01-15T18:17:16.113079+0000","last_unstale":"2024-01-15T18:17:18.440319+0000","last_undegraded":"2024-01-15T18:17:18.440319+0000","last_fullsized":"2024-01-15T18:17:18.440319+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:25:22.584736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.095295+0000","last_change":"2024-01-15T18:17:15.095295+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.095295+0000","last_undegraded":"2024-01-15T18:17:15.095295+0000","last_fullsized":"2024-01-15T18:17:15.095295+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440562+0000","last_change":"2024-01-15T18:17:16.121786+0000","last_active":"2024-01-15T18:17:18.440562+0000","last_peered":"2024-01-15T18:17:18.440562+0000","last_clean":"2024-01-15T18:17:18.440562+0000","last_became_active":"2024-01-15T18:17:16.121508+0000","last_became_peered":"2024-01-15T18:17:16.121508+0000","last_unstale":"2024-01-15T18:17:18.440562+0000","last_undegraded":"2024-01-15T18:17:18.440562+0000","last_fullsized":"2024-01-15T18:17:18.440562+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:51:43.523481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.133059+0000","last_change":"2024-01-15T18:17:17.126022+0000","last_active":"2024-01-15T18:17:18.133059+0000","last_peered":"2024-01-15T18:17:18.133059+0000","last_clean":"2024-01-15T18:17:18.133059+0000","last_became_active":"2024-01-15T18:17:17.125366+0000","last_became_peered":"2024-01-15T18:17:17.125366+0000","last_unstale":"2024-01-15T18:17:18.133059+0000","last_undegraded":"2024-01-15T18:17:18.133059+0000","last_fullsized":"2024-01-15T18:17:18.133059+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:27:59.524326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.095318+0000","last_change":"2024-01-15T18:17:15.095318+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.095318+0000","last_undegraded":"2024-01-15T18:17:15.095318+0000","last_fullsized":"2024-01-15T18:17:15.095318+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.095327+0000","last_change":"2024-01-15T18:17:15.095327+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.095327+0000","last_undegraded":"2024-01-15T18:17:15.095327+0000","last_fullsized":"2024-01-15T18:17:15.095327+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.094409+0000","last_change":"2024-01-15T18:17:15.094409+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.094409+0000","last_undegraded":"2024-01-15T18:17:15.094409+0000","last_fullsized":"2024-01-15T18:17:15.094409+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.096380+0000","last_change":"2024-01-15T18:17:15.096380+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.096380+0000","last_undegraded":"2024-01-15T18:17:15.096380+0000","last_fullsized":"2024-01-15T18:17:15.096380+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":227,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-01-15T18:17:13.089307+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:17:13.089307+0000","last_peered":"2024-01-15T18:17:13.089307+0000","last_clean":"2024-01-15T18:17:13.089307+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:17:13.089307+0000","last_undegraded":"2024-01-15T18:17:13.089307+0000","last_fullsized":"2024-01-15T18:17:13.089307+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":38,"acting":38,"num_store_stats":3},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2194,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21,"ondisk_log_size":21,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509593,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64748,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307092,"statfs":{"total":96636764160,"available":96570462208,"internally_reserved":0,"allocated":1736704,"data_stored":1082552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64551370},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.87,"15min":0.871},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.90800000000000003,"5min":9.3040000000000003,"15min":10.284000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.83899999999999997,"15min":0.91600000000000004},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.378},"max":{"1min":6.6189999999999998,"5min":6.6189999999999998,"15min":16.968},"last":0.68100000000000005}]},{"osd":1,"last update":"Mon Jan 15 18:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.93200000000000005,"15min":0.92700000000000005},"min":{"1min":0.46000000000000002,"5min":0.36099999999999999,"15min":0.33800000000000002},"max":{"1min":4.944,"5min":9.3829999999999991,"15min":15.791},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.84499999999999997,"15min":0.90800000000000003},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.371},"max":{"1min":4.6699999999999999,"5min":5.0899999999999999,"15min":10.842000000000001},"last":0.72799999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":1.159,"15min":0.93500000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.378},"max":{"1min":4.4669999999999996,"5min":22.318000000000001,"15min":22.318000000000001},"last":0.63},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.92500000000000004,"15min":0.90600000000000003},"min":{"1min":0.433,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":4.7030000000000003,"5min":9.5169999999999995,"15min":9.7539999999999996},"last":0.59899999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":1.163,"15min":0.98999999999999999},"min":{"1min":0.50900000000000001,"5min":0.503,"15min":0.38100000000000001},"max":{"1min":1.456,"5min":18.152999999999999,"15min":18.152999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.95899999999999996,"15min":0.98499999999999999},"min":{"1min":0.49199999999999999,"5min":0.433,"15min":0.43099999999999999},"max":{"1min":2.6909999999999998,"5min":9.75,"15min":10.34},"last":0.70599999999999996}]},{"osd":4,"last update":"Mon Jan 15 18:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.84899999999999998,"15min":0.93500000000000005},"min":{"1min":0.38900000000000001,"5min":0.314,"15min":0.314},"max":{"1min":4.4480000000000004,"5min":8.4420000000000002,"15min":20.387},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.73199999999999998,"5min":1.1699999999999999,"15min":0.98799999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":3.6309999999999998,"5min":17.215,"15min":17.215},"last":0.58499999999999996}]},{"osd":5,"last update":"Mon Jan 15 18:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.94499999999999995,"15min":0.95099999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":4.8639999999999999,"5min":9.6630000000000003,"15min":19.959},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.91300000000000003,"15min":0.98099999999999998},"min":{"1min":0.39400000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":4.3959999999999999,"5min":7.6980000000000004,"15min":19.905000000000001},"last":0.79300000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.99099999999999999,"15min":1.155},"min":{"1min":0.505,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":4.3330000000000002,"5min":8.6609999999999996,"15min":22.478999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.82399999999999995,"5min":1.1579999999999999,"15min":1.0660000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.38300000000000001},"max":{"1min":3.573,"5min":17.132000000000001,"15min":21.509},"last":0.69799999999999995}]}]},{"osd":6,"up_from":15,"seq":64424509593,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80812,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94291028,"statfs":{"total":96636764160,"available":96554012672,"internally_reserved":0,"allocated":1142784,"data_stored":492184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81590726},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":1.0529999999999999,"15min":0.97799999999999998},"min":{"1min":0.40000000000000002,"5min":0.32800000000000001,"15min":0.29099999999999998},"max":{"1min":0.86699999999999999,"5min":32.587000000000003,"15min":32.587000000000003},"last":0.873},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.91600000000000004,"15min":0.86299999999999999},"min":{"1min":0.41999999999999998,"5min":0.38500000000000001,"15min":0.35399999999999998},"max":{"1min":0.88400000000000001,"5min":16.193000000000001,"15min":16.193000000000001},"last":0.59399999999999997}]},{"osd":1,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.82499999999999996,"15min":0.99399999999999999},"min":{"1min":0.40600000000000003,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95299999999999996,"5min":7.7649999999999997,"15min":53.451999999999998},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1,"15min":0.91800000000000004},"min":{"1min":0.49299999999999999,"5min":0.437,"15min":0.42899999999999999},"max":{"1min":0.89900000000000002,"5min":32.512,"15min":32.512},"last":0.64400000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":1.087,"15min":0.97099999999999997},"min":{"1min":0.48299999999999998,"5min":0.41699999999999998,"15min":0.38900000000000001},"max":{"1min":0.90100000000000002,"5min":32.472999999999999,"15min":32.472999999999999},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.997,"15min":0.89600000000000002},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.38500000000000001},"max":{"1min":0.97899999999999998,"5min":16.602,"15min":16.602},"last":0.53500000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.97999999999999998,"15min":0.93100000000000005},"min":{"1min":0.50700000000000001,"5min":0.49399999999999999,"15min":0.41399999999999998},"max":{"1min":0.94499999999999995,"5min":16.388000000000002,"15min":16.388000000000002},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.879,"15min":1.079},"min":{"1min":0.52300000000000002,"5min":0.432,"15min":0.41599999999999998},"max":{"1min":0.996,"5min":5.9740000000000002,"15min":53.398000000000003},"last":0.85299999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.98799999999999999,"15min":0.89100000000000001},"min":{"1min":0.42999999999999999,"5min":0.38100000000000001,"15min":0.34999999999999998},"max":{"1min":0.92200000000000004,"5min":32.549999999999997,"15min":32.549999999999997},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.878,"15min":0.83999999999999997},"min":{"1min":0.42299999999999999,"5min":0.38700000000000001,"15min":0.34499999999999997},"max":{"1min":1.087,"5min":11.852,"15min":11.852},"last":0.75600000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.82699999999999996,"15min":1.0369999999999999},"min":{"1min":0.36199999999999999,"5min":0.35799999999999998,"15min":0.30399999999999999},"max":{"1min":1.1759999999999999,"5min":7.5650000000000004,"15min":53.229999999999997},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.081,"15min":1.028},"min":{"1min":0.432,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.228,"5min":32.404000000000003,"15min":32.404000000000003},"last":0.92600000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.93899999999999995,"15min":1.105},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.1220000000000001,"5min":16.565000000000001,"15min":45.070999999999998},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.68799999999999994,"5min":1.161,"15min":1.21},"min":{"1min":0.46300000000000002,"5min":0.40999999999999998,"15min":0.34999999999999998},"max":{"1min":1.032,"5min":40.972999999999999,"15min":53.338000000000001},"last":0.90700000000000003}]}]},{"osd":4,"up_from":15,"seq":64424509594,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76724,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295116,"statfs":{"total":96636764160,"available":96558198784,"internally_reserved":0,"allocated":1212416,"data_stored":519535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.82499999999999996,"15min":0.88400000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.32900000000000001},"max":{"1min":0.91800000000000004,"5min":7.3380000000000001,"15min":23.318000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.70199999999999996,"15min":0.79300000000000004},"min":{"1min":0.372,"5min":0.36399999999999999,"15min":0.32600000000000001},"max":{"1min":0.82699999999999996,"5min":6.444,"15min":14.723000000000001},"last":0.72399999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.82699999999999996,"15min":0.78800000000000003},"min":{"1min":0.39600000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.2,"5min":9.5700000000000003,"15min":9.5700000000000003},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.92100000000000004,"15min":0.91600000000000004},"min":{"1min":0.45200000000000001,"5min":0.40200000000000002,"15min":0.36899999999999999},"max":{"1min":0.84099999999999997,"5min":7.9980000000000002,"15min":22.408999999999999},"last":0.65600000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.85399999999999998,"15min":0.79900000000000004},"min":{"1min":0.439,"5min":0.439,"15min":0.34399999999999997},"max":{"1min":1.266,"5min":9.7089999999999996,"15min":9.7089999999999996},"last":0.629},{"interface":"front","average":{"1min":0.73699999999999999,"5min":1.0649999999999999,"15min":1.0149999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.39100000000000001},"max":{"1min":1.714,"5min":14.118,"15min":22.338999999999999},"last":0.77000000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.77800000000000002,"15min":0.872},"min":{"1min":0.437,"5min":0.437,"15min":0.41199999999999998},"max":{"1min":0.88800000000000001,"5min":6.7930000000000001,"15min":14.763999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.94199999999999995,"15min":0.90600000000000003},"min":{"1min":0.504,"5min":0.48099999999999998,"15min":0.45500000000000002},"max":{"1min":0.93600000000000005,"5min":9.6289999999999996,"15min":9.6289999999999996},"last":0.81399999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.94599999999999995,"15min":0.95699999999999996},"min":{"1min":0.40000000000000002,"5min":0.34799999999999998,"15min":0.33200000000000002},"max":{"1min":0.94099999999999995,"5min":15.787000000000001,"15min":31.617999999999999},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.75,"15min":0.86399999999999999},"min":{"1min":0.42599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95299999999999996,"5min":6.7169999999999996,"15min":17.145},"last":0.79600000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.84899999999999998,"15min":0.82799999999999996},"min":{"1min":0.42299999999999999,"5min":0.39800000000000002,"15min":0.35499999999999998},"max":{"1min":0.97799999999999998,"5min":9.6709999999999994,"15min":11.124000000000001},"last":0.751},{"interface":"front","average":{"1min":0.69799999999999995,"5min":1.1699999999999999,"15min":1.2230000000000001},"min":{"1min":0.56599999999999995,"5min":0.438,"15min":0.40200000000000002},"max":{"1min":0.90200000000000002,"5min":32.308,"15min":32.308},"last":0.82299999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.749,"15min":1.0029999999999999},"min":{"1min":0.434,"5min":0.41099999999999998,"15min":0.34799999999999998},"max":{"1min":0.91900000000000004,"5min":6.8390000000000004,"15min":39.256},"last":0.876},{"interface":"front","average":{"1min":0.70399999999999996,"5min":1.1160000000000001,"15min":1.004},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.42199999999999999},"max":{"1min":0.86199999999999999,"5min":16.277000000000001,"15min":26.573},"last":0.86299999999999999}]}]},{"osd":3,"up_from":15,"seq":64424509592,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64172,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307668,"statfs":{"total":96636764160,"available":96571052032,"internally_reserved":0,"allocated":1142784,"data_stored":492184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64551367},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.44900000000000001,"15min":0.48899999999999999},"min":{"1min":0.27300000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.624,"5min":2.1160000000000001,"15min":5.8849999999999998},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.54100000000000004,"15min":0.61599999999999999},"min":{"1min":0.312,"5min":0.29799999999999999,"15min":0.29499999999999998},"max":{"1min":0.66600000000000004,"5min":3.0649999999999999,"15min":17.756},"last":0.51800000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.495,"15min":0.58499999999999996},"min":{"1min":0.28899999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68700000000000006,"5min":3.6389999999999998,"15min":12.307},"last":0.435},{"interface":"front","average":{"1min":0.503,"5min":0.55500000000000005,"15min":0.58499999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.70299999999999996,"5min":3.738,"15min":6.0640000000000001},"last":0.41399999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.65400000000000003,"15min":0.68999999999999995},"min":{"1min":0.33700000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69999999999999996,"5min":5.431,"15min":12.468999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.70299999999999996,"15min":0.68300000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.34599999999999997},"max":{"1min":0.77800000000000002,"5min":10.837999999999999,"15min":10.837999999999999},"last":0.748}]},{"osd":4,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":1.0609999999999999,"15min":1.014},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.375},"max":{"1min":1.024,"5min":13.863,"15min":17.853999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.94999999999999996,"15min":0.96199999999999997},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.47099999999999997},"max":{"1min":0.94599999999999995,"5min":6.8289999999999997,"15min":13.31},"last":0.64100000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.153,"15min":0.97399999999999998},"min":{"1min":0.54200000000000004,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1020000000000001,"5min":17.687999999999999,"15min":17.687999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.88500000000000001,"15min":0.96999999999999997},"min":{"1min":0.45500000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.085,"5min":5.0949999999999998,"15min":20.515999999999998},"last":0.86899999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.80200000000000005,"15min":0.97699999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.432},"max":{"1min":1.05,"5min":4.0910000000000002,"15min":23.100999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.90000000000000002,"15min":0.93600000000000005},"min":{"1min":0.59399999999999997,"5min":0.53600000000000003,"15min":0.49399999999999999},"max":{"1min":1.0069999999999999,"5min":6.7220000000000004,"15min":12.76},"last":0.70399999999999996}]},{"osd":7,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":1.331,"15min":1.1659999999999999},"min":{"1min":0.503,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.0089999999999999,"5min":24.321999999999999,"15min":24.321999999999999},"last":0.90900000000000003},{"interface":"front","average":{"1min":2.0510000000000002,"5min":1.139,"15min":1.1020000000000001},"min":{"1min":0.54700000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"last":0.60999999999999999}]}]},{"osd":2,"up_from":15,"seq":64424509592,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64184,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307656,"statfs":{"total":96636764160,"available":96571039744,"internally_reserved":0,"allocated":1155072,"data_stored":498113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.45500000000000002,"15min":0.50900000000000001},"min":{"1min":0.26500000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.71299999999999997,"5min":0.98199999999999998,"15min":5.1849999999999996},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.49199999999999999,"15min":0.53200000000000003},"min":{"1min":0.315,"5min":0.252,"15min":0.252},"max":{"1min":0.68899999999999995,"5min":1.1579999999999999,"15min":7.516},"last":0.51900000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.55800000000000005,"15min":0.61599999999999999},"min":{"1min":0.39800000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.83299999999999996,"5min":0.95799999999999996,"15min":10.907},"last":0.371},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.50800000000000001,"15min":0.59199999999999997},"min":{"1min":0.38200000000000001,"5min":0.28100000000000003,"15min":0.27000000000000002},"max":{"1min":0.77500000000000002,"5min":1.2849999999999999,"15min":8.9190000000000005},"last":0.45900000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55500000000000005,"15min":0.623},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0249999999999999,"5min":1.133,"15min":12.185},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.60199999999999998,"15min":0.66400000000000003},"min":{"1min":0.40600000000000003,"5min":0.373,"15min":0.34000000000000002},"max":{"1min":0.79000000000000004,"5min":0.96699999999999997,"15min":7.6760000000000002},"last":0.439}]},{"osd":4,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.88800000000000001,"15min":0.875},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":4.0190000000000001,"5min":4.3150000000000004,"15min":8.6609999999999996},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.76200000000000001,"15min":0.90300000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.5860000000000001,"5min":2.468,"15min":19.146999999999998},"last":0.56599999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":0.79700000000000004,"15min":0.90800000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":5.0309999999999997,"5min":5.0309999999999997,"15min":31.710000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.83999999999999997,"15min":0.999},"min":{"1min":0.51100000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.3660000000000001,"5min":3.6459999999999999,"15min":22.539000000000001},"last":0.60999999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.84099999999999997,"15min":0.90600000000000003},"min":{"1min":0.499,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":4.1959999999999997,"5min":4.1959999999999997,"15min":9.4540000000000006},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.79200000000000004,"15min":0.90600000000000003},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.633,"5min":1.633,"15min":18.614999999999998},"last":0.68899999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.97799999999999998,"15min":0.96199999999999997},"min":{"1min":0.52100000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":4.3170000000000002,"5min":11.606,"15min":11.606},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.159,"5min":0.92900000000000005,"15min":1.004},"min":{"1min":0.54900000000000004,"5min":0.52500000000000002,"15min":0.502},"max":{"1min":6.1260000000000003,"5min":6.1260000000000003,"15min":17.013000000000002},"last":0.70599999999999996}]}]},{"osd":1,"up_from":15,"seq":64424509594,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68332,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303508,"statfs":{"total":96636764160,"available":96566792192,"internally_reserved":0,"allocated":1142784,"data_stored":492184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.51700000000000002,"15min":0.65700000000000003},"min":{"1min":0.28399999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.6140000000000001,"5min":4.1550000000000002,"15min":19.577000000000002},"last":0.374},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.623,"15min":0.76000000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.251},"max":{"1min":0.83899999999999997,"5min":12.378,"15min":48.277999999999999},"last":0.67400000000000004}]},{"osd":2,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55400000000000005,"15min":0.70799999999999996},"min":{"1min":0.32800000000000001,"5min":0.309,"15min":0.30299999999999999},"max":{"1min":1.8049999999999999,"5min":2.8610000000000002,"15min":40.698},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59099999999999997,"15min":0.877},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.317},"max":{"1min":1.978,"5min":2.5139999999999998,"15min":48.316000000000003},"last":0.63800000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.623,"15min":0.82999999999999996},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.33700000000000002},"max":{"1min":0.94699999999999995,"5min":2.8620000000000001,"15min":48.234999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.69299999999999995,"15min":0.67700000000000005},"min":{"1min":0.33400000000000002,"5min":0.315,"15min":0.311},"max":{"1min":1.0780000000000001,"5min":9.9719999999999995,"15min":9.9719999999999995},"last":0.499}]},{"osd":4,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.90300000000000002,"15min":0.88800000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.44500000000000001},"max":{"1min":0.97499999999999998,"5min":9.9329999999999998,"15min":9.9329999999999998},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.86199999999999999,"15min":0.93300000000000005},"min":{"1min":0.502,"5min":0.45800000000000002,"15min":0.42499999999999999},"max":{"1min":2.3900000000000001,"5min":9.8610000000000007,"15min":19.829000000000001},"last":0.85599999999999998}]},{"osd":5,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.82899999999999996,"15min":1.0129999999999999},"min":{"1min":0.53700000000000003,"5min":0.48599999999999999,"15min":0.46100000000000002},"max":{"1min":2.5960000000000001,"5min":8.5839999999999996,"15min":22.594999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.80600000000000005,"15min":1.157},"min":{"1min":0.44400000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.6879999999999999,"5min":2.9609999999999999,"15min":48.198},"last":0.75700000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.90800000000000003,"15min":1.04},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0189999999999999,"5min":9.3989999999999991,"15min":35.112000000000002},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.91300000000000003,"15min":0.96899999999999997},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.50800000000000001},"max":{"1min":2.512,"5min":5.8849999999999998,"15min":19.780999999999999},"last":0.94399999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.89200000000000002,"15min":1.1619999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.93300000000000005,"5min":11.039,"15min":48.148000000000003},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.748,"5min":0.97899999999999998,"15min":0.98999999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.52100000000000002},"max":{"1min":1.0609999999999999,"5min":9.8640000000000008,"15min":9.8640000000000008},"last":0.95899999999999996}]}]},{"osd":5,"up_from":14,"seq":60129542298,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58724,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313116,"statfs":{"total":96636764160,"available":96576630784,"internally_reserved":0,"allocated":1196032,"data_stored":517869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.81200000000000006,"15min":0.80100000000000005},"min":{"1min":0.375,"5min":0.29899999999999999,"15min":0.28999999999999998},"max":{"1min":1.383,"5min":5.1639999999999997,"15min":21.385000000000002},"last":0.874},{"interface":"front","average":{"1min":0.748,"5min":0.78700000000000003,"15min":0.877},"min":{"1min":0.57999999999999996,"5min":0.378,"15min":0.34899999999999998},"max":{"1min":1.3029999999999999,"5min":5.3869999999999996,"15min":9.359},"last":0.67100000000000004}]},{"osd":1,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.82199999999999995,"15min":0.89500000000000002},"min":{"1min":0.44400000000000001,"5min":0.375,"15min":0.34899999999999998},"max":{"1min":1.2070000000000001,"5min":3.7429999999999999,"15min":21.120000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.78500000000000003,"5min":1.1850000000000001,"15min":0.96999999999999997},"min":{"1min":0.48999999999999999,"5min":0.38600000000000001,"15min":0.315},"max":{"1min":1.532,"5min":15.843,"15min":21.5},"last":0.90000000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.83799999999999997,"15min":0.89800000000000002},"min":{"1min":0.47699999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.421,"5min":5.516,"15min":8.7750000000000004},"last":0.622},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.90800000000000003,"15min":0.98799999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.347,"5min":8.3529999999999998,"15min":21.201000000000001},"last":0.68700000000000006}]},{"osd":3,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.93200000000000005,"15min":0.91300000000000003},"min":{"1min":0.58499999999999996,"5min":0.37,"15min":0.37},"max":{"1min":1.478,"5min":7.7199999999999998,"15min":21.457000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.85499999999999998,"15min":0.96299999999999997},"min":{"1min":0.57199999999999995,"5min":0.47699999999999998,"15min":0.45400000000000001},"max":{"1min":1.5,"5min":5.6109999999999998,"15min":8.7300000000000004},"last":0.78900000000000003}]},{"osd":4,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.93500000000000005,"15min":0.88800000000000001},"min":{"1min":0.376,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":4.8650000000000002,"5min":10.962,"15min":10.962},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.78800000000000003,"15min":1.018},"min":{"1min":0.442,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.464,"5min":4.6639999999999997,"15min":26.146999999999998},"last":0.80700000000000005}]},{"osd":6,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.86599999999999999,"15min":0.83799999999999997},"min":{"1min":0.44700000000000001,"5min":0.36099999999999999,"15min":0.32000000000000001},"max":{"1min":2.8940000000000001,"5min":5.2160000000000002,"15min":21.535},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.85899999999999999,"15min":1.0129999999999999},"min":{"1min":0.432,"5min":0.379,"15min":0.36299999999999999},"max":{"1min":4.9139999999999997,"5min":4.9139999999999997,"15min":30.488},"last":0.73899999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.276,"15min":1.2290000000000001},"min":{"1min":0.51800000000000002,"5min":0.432,"15min":0.39700000000000002},"max":{"1min":8.7539999999999996,"5min":33.499000000000002,"15min":33.499000000000002},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.98599999999999999,"15min":1.0900000000000001},"min":{"1min":0.46200000000000002,"5min":0.441,"15min":0.36499999999999999},"max":{"1min":4.3120000000000003,"5min":6.306,"15min":25.437999999999999},"last":0.76200000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542298,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64800,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307040,"statfs":{"total":96636764160,"available":96570408960,"internally_reserved":0,"allocated":1789952,"data_stored":1108749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.52400000000000002,"15min":0.627},"min":{"1min":0.26100000000000001,"5min":0.245,"15min":0.23699999999999999},"max":{"1min":0.93500000000000005,"5min":7.907,"15min":19.843},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58199999999999996,"15min":0.67400000000000004},"min":{"1min":0.35599999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.8240000000000001,"5min":3.512,"15min":10.154},"last":0.33100000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.71599999999999997,"15min":0.80100000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.30399999999999999},"max":{"1min":1.4410000000000001,"5min":7.0810000000000004,"15min":25.905000000000001},"last":0.379},{"interface":"front","average":{"1min":0.498,"5min":0.57499999999999996,"15min":0.61399999999999999},"min":{"1min":0.32700000000000001,"5min":0.27800000000000002,"15min":0.26700000000000002},"max":{"1min":0.79300000000000004,"5min":3.8500000000000001,"15min":9.3829999999999991},"last":0.39600000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.69399999999999995,"15min":0.74199999999999999},"min":{"1min":0.34699999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":2.3519999999999999,"5min":6.2850000000000001,"15min":9.1780000000000008},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.70199999999999996,"15min":0.76200000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.36899999999999999},"max":{"1min":1.327,"5min":6.5979999999999999,"15min":10.210000000000001},"last":0.44400000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.84199999999999997,"15min":0.98399999999999999},"min":{"1min":0.47699999999999998,"5min":0.39100000000000001,"15min":0.38},"max":{"1min":1.2929999999999999,"5min":4.5179999999999998,"15min":22.814},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.83399999999999996,"15min":1.0289999999999999},"min":{"1min":0.502,"5min":0.502,"15min":0.41199999999999998},"max":{"1min":1.2749999999999999,"5min":4.2329999999999997,"15min":23.341999999999999},"last":0.57999999999999996}]},{"osd":5,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.86399999999999999,"15min":0.96999999999999997},"min":{"1min":0.54400000000000004,"5min":0.51600000000000001,"15min":0.441},"max":{"1min":2.6190000000000002,"5min":4.008,"15min":12.428000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.79900000000000004,"5min":1.252,"15min":1.258},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.3120000000000001,"5min":23.021000000000001,"15min":23.021000000000001},"last":0.66100000000000003}]},{"osd":6,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.92000000000000004,"15min":1.212},"min":{"1min":0.53500000000000003,"5min":0.48399999999999999,"15min":0.46200000000000002},"max":{"1min":1.6359999999999999,"5min":6.5250000000000004,"15min":31.059000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.94799999999999995,"15min":1.089},"min":{"1min":0.59399999999999997,"5min":0.52700000000000002,"15min":0.49099999999999999},"max":{"1min":2.5659999999999998,"5min":6.9690000000000003,"15min":40.731000000000002},"last":0.623}]},{"osd":7,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":1.107,"15min":1.2},"min":{"1min":0.59499999999999997,"5min":0.499,"15min":0.42499999999999999},"max":{"1min":2.6539999999999999,"5min":10.497999999999999,"15min":24.492999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.88300000000000001,"15min":1.103},"min":{"1min":0.55300000000000005,"5min":0.495,"15min":0.46600000000000003},"max":{"1min":1.3740000000000001,"5min":4.6600000000000001,"15min":23.524000000000001},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:17:21.241 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:17:21.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f659f400640 1 Processor -- start 2024-01-15T18:17:21.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f659f400640 1 -- start start 2024-01-15T18:17:21.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f659f400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f65980cf9b0 0x7f65980b10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:21.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f659f400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f65980cedc0 0x7f65980b1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:21.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f659f400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65980b38b0 con 0x7f659815e7d0 2024-01-15T18:17:21.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f659f400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65980cd550 con 0x7f65980cf9b0 2024-01-15T18:17:21.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f659f400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65980cd270 con 0x7f65980cedc0 2024-01-15T18:17:21.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f659ca00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f659815e7d0 0x7f65980cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48898/0 (socket says 172.21.15.27:48898) 2024-01-15T18:17:21.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f659ca00640 1 -- 172.21.15.27:0/1605869225 learned_addr learned my addr 172.21.15.27:0/1605869225 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:21.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f6597e00640 1 --2- 172.21.15.27:0/1605869225 >> v2:172.21.15.93:3300/0 conn(0x7f65980cf9b0 0x7f65980b10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:21.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f659d400640 1 --2- 172.21.15.27:0/1605869225 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f65980cedc0 0x7f65980b1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:21.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f6597400640 1 -- 172.21.15.27:0/1605869225 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299702891 0 0) 0x7f65980b38b0 con 0x7f659815e7d0 2024-01-15T18:17:21.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f6597400640 1 -- 172.21.15.27:0/1605869225 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6578003530 con 0x7f659815e7d0 2024-01-15T18:17:21.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f6597400640 1 -- 172.21.15.27:0/1605869225 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2901649853 0 0) 0x7f6578003530 con 0x7f659815e7d0 2024-01-15T18:17:21.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f6597400640 1 -- 172.21.15.27:0/1605869225 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f65980cedc0 msgr2=0x7f65980b1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f6597400640 1 --2- 172.21.15.27:0/1605869225 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f65980cedc0 0x7f65980b1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f6597400640 1 -- 172.21.15.27:0/1605869225 >> v2:172.21.15.93:3300/0 conn(0x7f65980cf9b0 msgr2=0x7f65980b10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f6597400640 1 --2- 172.21.15.27:0/1605869225 >> v2:172.21.15.93:3300/0 conn(0x7f65980cf9b0 0x7f65980b10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f6597400640 1 -- 172.21.15.27:0/1605869225 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65980b35d0 con 0x7f659815e7d0 2024-01-15T18:17:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f6597400640 1 -- 172.21.15.27:0/1605869225 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f6588001680 con 0x7f659815e7d0 2024-01-15T18:17:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f6597e00640 1 --2- 172.21.15.27:0/1605869225 >> v2:172.21.15.93:3300/0 conn(0x7f65980cf9b0 0x7f65980b10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:17:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f6597400640 1 -- 172.21.15.27:0/1605869225 >> v1:172.21.15.27:6789/0 conn(0x7f659815e7d0 legacy=0x7f65980cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f6597400640 1 --2- 172.21.15.27:0/1605869225 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f65780039f0 0x7f6578003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f6597400640 1 --2- 172.21.15.27:0/1605869225 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f65980cedc0 0x7f65780047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f6597400640 1 --2- 172.21.15.27:0/1605869225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6578004d10 0x7f6578005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:21.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f6597400640 1 -- 172.21.15.27:0/1605869225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65980cd550 con 0x7f6578004d10 2024-01-15T18:17:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f6597400640 1 -- 172.21.15.27:0/1605869225 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65980cd270 con 0x7f65780039f0 2024-01-15T18:17:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f6597400640 1 -- 172.21.15.27:0/1605869225 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657800b990 con 0x7f65980cedc0 2024-01-15T18:17:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f659d400640 1 --2- 172.21.15.27:0/1605869225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6578004d10 0x7f6578005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f6597e00640 1 --2- 172.21.15.27:0/1605869225 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f65980cedc0 0x7f65780047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.418+0000 7f659ca00640 1 --2- 172.21.15.27:0/1605869225 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f65780039f0 0x7f6578003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.422+0000 7f659d400640 1 -- 172.21.15.27:0/1605869225 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f65980cedc0 msgr2=0x7f65780047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.422+0000 7f659d400640 1 --2- 172.21.15.27:0/1605869225 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f65980cedc0 0x7f65780047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.422+0000 7f659d400640 1 -- 172.21.15.27:0/1605869225 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f65780039f0 msgr2=0x7f6578003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.422+0000 7f659d400640 1 --2- 172.21.15.27:0/1605869225 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f65780039f0 0x7f6578003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.422+0000 7f659d400640 1 -- 172.21.15.27:0/1605869225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6578003530 con 0x7f6578004d10 2024-01-15T18:17:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.422+0000 7f659ca00640 1 --2- 172.21.15.27:0/1605869225 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f65780039f0 0x7f6578003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:17:21.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.422+0000 7f659d400640 1 --2- 172.21.15.27:0/1605869225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6578004d10 0x7f6578005190 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f6590009fc0 tx=0x7f659001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a21ef3c6b3c800a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:21.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.422+0000 7f6597400640 1 -- 172.21.15.27:0/1605869225 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65900a5c70 con 0x7f6578004d10 2024-01-15T18:17:21.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.422+0000 7f6597400640 1 -- 172.21.15.27:0/1605869225 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f659000f040 con 0x7f6578004d10 2024-01-15T18:17:21.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.422+0000 7f6597400640 1 -- 172.21.15.27:0/1605869225 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65900ad440 con 0x7f6578004d10 2024-01-15T18:17:21.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.422+0000 7f659f400640 1 -- 172.21.15.27:0/1605869225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6578004d10 msgr2=0x7f6578005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:21.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.422+0000 7f659f400640 1 --2- 172.21.15.27:0/1605869225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6578004d10 0x7f6578005190 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f6590009fc0 tx=0x7f659001cc60 comp rx=0 tx=0).stop 2024-01-15T18:17:21.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.422+0000 7f659d400640 1 -- 172.21.15.27:0/1605869225 reap_dead start 2024-01-15T18:17:21.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.426+0000 7f659f400640 1 -- 172.21.15.27:0/1605869225 shutdown_connections 2024-01-15T18:17:21.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.426+0000 7f659f400640 1 -- 172.21.15.27:0/1605869225 >> 172.21.15.27:0/1605869225 conn(0x7f65980b2a10 msgr2=0x7f65980b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:21.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.426+0000 7f659f400640 1 -- 172.21.15.27:0/1605869225 shutdown_connections 2024-01-15T18:17:21.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.426+0000 7f659f400640 1 -- 172.21.15.27:0/1605869225 wait complete. 2024-01-15T18:17:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.426+0000 7f659f400640 1 Processor -- start 2024-01-15T18:17:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.426+0000 7f659f400640 1 -- start start 2024-01-15T18:17:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.426+0000 7f659f400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6578004d10 0x7f65980c5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.426+0000 7f659f400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f65980caf40 0x7f659815a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:21.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.426+0000 7f659f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f659815bc20 0x7f659815c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:21.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.426+0000 7f659f400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659806bad0 con 0x7f659815bc20 2024-01-15T18:17:21.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.426+0000 7f659f400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659815c5f0 con 0x7f6578004d10 2024-01-15T18:17:21.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.426+0000 7f659f400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65980c50a0 con 0x7f65980caf40 2024-01-15T18:17:21.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.426+0000 7f659d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f659815bc20 0x7f659815c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:21.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.426+0000 7f659d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f659815bc20 0x7f659815c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59702/0 (socket says 172.21.15.27:59702) 2024-01-15T18:17:21.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.426+0000 7f659d400640 1 -- 172.21.15.27:0/3394610723 learned_addr learned my addr 172.21.15.27:0/3394610723 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:21.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.426+0000 7f6597e00640 1 --2- 172.21.15.27:0/3394610723 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f65980caf40 0x7f659815a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:21.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.426+0000 7f659ca00640 1 --2- 172.21.15.27:0/3394610723 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6578004d10 0x7f65980c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:21.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.426+0000 7f659d400640 1 -- 172.21.15.27:0/3394610723 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f65980caf40 msgr2=0x7f659815a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:21.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.426+0000 7f659d400640 1 --2- 172.21.15.27:0/3394610723 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f65980caf40 0x7f659815a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.426+0000 7f659d400640 1 -- 172.21.15.27:0/3394610723 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6578004d10 msgr2=0x7f65980c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:21.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.426+0000 7f659d400640 1 --2- 172.21.15.27:0/3394610723 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6578004d10 0x7f65980c5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.426+0000 7f659d400640 1 -- 172.21.15.27:0/3394610723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65980c4b20 con 0x7f659815bc20 2024-01-15T18:17:21.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.430+0000 7f659d400640 1 --2- 172.21.15.27:0/3394610723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f659815bc20 0x7f659815c0b0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f65900b0040 tx=0x7f6590009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:21.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.430+0000 7f6594c00640 1 -- 172.21.15.27:0/3394610723 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65900bc070 con 0x7f659815bc20 2024-01-15T18:17:21.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.430+0000 7f6594c00640 1 -- 172.21.15.27:0/3394610723 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65900039a0 con 0x7f659815bc20 2024-01-15T18:17:21.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.430+0000 7f6594c00640 1 -- 172.21.15.27:0/3394610723 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6590003b30 con 0x7f659815bc20 2024-01-15T18:17:21.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.430+0000 7f659f400640 1 -- 172.21.15.27:0/3394610723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f659806b730 con 0x7f659815bc20 2024-01-15T18:17:21.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.430+0000 7f659f400640 1 -- 172.21.15.27:0/3394610723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65980c5940 con 0x7f659815bc20 2024-01-15T18:17:21.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.430+0000 7f659f400640 1 -- 172.21.15.27:0/3394610723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f659806bc10 con 0x7f659815bc20 2024-01-15T18:17:21.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.430+0000 7f6594c00640 1 -- 172.21.15.27:0/3394610723 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f65900b4040 con 0x7f659815bc20 2024-01-15T18:17:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.430+0000 7f6594c00640 1 --2- 172.21.15.27:0/3394610723 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6570067730 0x7f6570069bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.430+0000 7f6594c00640 1 -- 172.21.15.27:0/3394610723 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f659011d040 con 0x7f659815bc20 2024-01-15T18:17:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.434+0000 7f659ca00640 1 --2- 172.21.15.27:0/3394610723 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6570067730 0x7f6570069bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:21.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.434+0000 7f659ca00640 1 --2- 172.21.15.27:0/3394610723 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6570067730 0x7f6570069bf0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f6588001ec0 tx=0x7f6588007040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:21.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.434+0000 7f6594c00640 1 -- 172.21.15.27:0/3394610723 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f659806bc10 con 0x7f659815bc20 2024-01-15T18:17:21.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.590+0000 7f659f400640 1 -- 172.21.15.27:0/3394610723 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f65980cf2a0 con 0x7f6570067730 2024-01-15T18:17:21.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.598+0000 7f6594c00640 1 -- 172.21.15.27:0/3394610723 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+281032 (secure 0 0 0) 0x7f65980cf2a0 con 0x7f6570067730 2024-01-15T18:17:21.600 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:17:21.602 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:17:21.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.602+0000 7f659f400640 1 -- 172.21.15.27:0/3394610723 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6570067730 msgr2=0x7f6570069bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:21.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.602+0000 7f659f400640 1 --2- 172.21.15.27:0/3394610723 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6570067730 0x7f6570069bf0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f6588001ec0 tx=0x7f6588007040 comp rx=0 tx=0).stop 2024-01-15T18:17:21.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.602+0000 7f659f400640 1 -- 172.21.15.27:0/3394610723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f659815bc20 msgr2=0x7f659815c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:21.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.602+0000 7f659f400640 1 --2- 172.21.15.27:0/3394610723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f659815bc20 0x7f659815c0b0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f65900b0040 tx=0x7f6590009750 comp rx=0 tx=0).stop 2024-01-15T18:17:21.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.602+0000 7f659f400640 1 -- 172.21.15.27:0/3394610723 shutdown_connections 2024-01-15T18:17:21.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.602+0000 7f659f400640 1 --2- 172.21.15.27:0/3394610723 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6570067730 0x7f6570069bf0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.602+0000 7f659f400640 1 --2- 172.21.15.27:0/3394610723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f659815bc20 0x7f659815c0b0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.602+0000 7f659f400640 1 --2- 172.21.15.27:0/3394610723 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f65980caf40 0x7f659815a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.602+0000 7f659f400640 1 --2- 172.21.15.27:0/3394610723 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6578004d10 0x7f65980c5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.602+0000 7f659f400640 1 -- 172.21.15.27:0/3394610723 >> 172.21.15.27:0/3394610723 conn(0x7f65980b2a10 msgr2=0x7f65980b2df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:21.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.602+0000 7f659f400640 1 -- 172.21.15.27:0/3394610723 shutdown_connections 2024-01-15T18:17:21.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.602+0000 7f659f400640 1 -- 172.21.15.27:0/3394610723 wait complete. 2024-01-15T18:17:21.621 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-01-15T18:17:20.354630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592562,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":182,"num_write_kb":2420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138,"ondisk_log_size":138,"up":104,"acting":104,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":71,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":542496,"kb_used_data":10272,"kb_used_omap":12,"kb_used_meta":532083,"kb_avail":754432224,"statfs":{"total":773094113280,"available":772538597376,"internally_reserved":0,"allocated":10518528,"data_stored":5203370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":544853552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2194,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.265190"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440653+0000","last_change":"2024-01-15T18:17:17.132525+0000","last_active":"2024-01-15T18:17:18.440653+0000","last_peered":"2024-01-15T18:17:18.440653+0000","last_clean":"2024-01-15T18:17:18.440653+0000","last_became_active":"2024-01-15T18:17:17.132261+0000","last_became_peered":"2024-01-15T18:17:17.132261+0000","last_unstale":"2024-01-15T18:17:18.440653+0000","last_undegraded":"2024-01-15T18:17:18.440653+0000","last_fullsized":"2024-01-15T18:17:18.440653+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:07.042399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439412+0000","last_change":"2024-01-15T18:17:17.129139+0000","last_active":"2024-01-15T18:17:18.439412+0000","last_peered":"2024-01-15T18:17:18.439412+0000","last_clean":"2024-01-15T18:17:18.439412+0000","last_became_active":"2024-01-15T18:17:17.126922+0000","last_became_peered":"2024-01-15T18:17:17.126922+0000","last_unstale":"2024-01-15T18:17:18.439412+0000","last_undegraded":"2024-01-15T18:17:18.439412+0000","last_fullsized":"2024-01-15T18:17:18.439412+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:40:32.654203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.132988+0000","last_change":"2024-01-15T18:17:17.117455+0000","last_active":"2024-01-15T18:17:18.132988+0000","last_peered":"2024-01-15T18:17:18.132988+0000","last_clean":"2024-01-15T18:17:18.132988+0000","last_became_active":"2024-01-15T18:17:17.117259+0000","last_became_peered":"2024-01-15T18:17:17.117259+0000","last_unstale":"2024-01-15T18:17:18.132988+0000","last_undegraded":"2024-01-15T18:17:18.132988+0000","last_fullsized":"2024-01-15T18:17:18.132988+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:23.944086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.132977+0000","last_change":"2024-01-15T18:17:17.126892+0000","last_active":"2024-01-15T18:17:18.132977+0000","last_peered":"2024-01-15T18:17:18.132977+0000","last_clean":"2024-01-15T18:17:18.132977+0000","last_became_active":"2024-01-15T18:17:17.125684+0000","last_became_peered":"2024-01-15T18:17:17.125684+0000","last_unstale":"2024-01-15T18:17:18.132977+0000","last_undegraded":"2024-01-15T18:17:18.132977+0000","last_fullsized":"2024-01-15T18:17:18.132977+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:54.730338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440809+0000","last_change":"2024-01-15T18:17:17.125211+0000","last_active":"2024-01-15T18:17:18.440809+0000","last_peered":"2024-01-15T18:17:18.440809+0000","last_clean":"2024-01-15T18:17:18.440809+0000","last_became_active":"2024-01-15T18:17:17.119327+0000","last_became_peered":"2024-01-15T18:17:17.119327+0000","last_unstale":"2024-01-15T18:17:18.440809+0000","last_undegraded":"2024-01-15T18:17:18.440809+0000","last_fullsized":"2024-01-15T18:17:18.440809+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:57:35.828970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.133012+0000","last_change":"2024-01-15T18:17:17.117786+0000","last_active":"2024-01-15T18:17:18.133012+0000","last_peered":"2024-01-15T18:17:18.133012+0000","last_clean":"2024-01-15T18:17:18.133012+0000","last_became_active":"2024-01-15T18:17:17.117505+0000","last_became_peered":"2024-01-15T18:17:17.117505+0000","last_unstale":"2024-01-15T18:17:18.133012+0000","last_undegraded":"2024-01-15T18:17:18.133012+0000","last_fullsized":"2024-01-15T18:17:18.133012+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:39:09.922990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440211+0000","last_change":"2024-01-15T18:17:17.125005+0000","last_active":"2024-01-15T18:17:18.440211+0000","last_peered":"2024-01-15T18:17:18.440211+0000","last_clean":"2024-01-15T18:17:18.440211+0000","last_became_active":"2024-01-15T18:17:17.124717+0000","last_became_peered":"2024-01-15T18:17:17.124717+0000","last_unstale":"2024-01-15T18:17:18.440211+0000","last_undegraded":"2024-01-15T18:17:18.440211+0000","last_fullsized":"2024-01-15T18:17:18.440211+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:48:30.181901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440100+0000","last_change":"2024-01-15T18:17:17.130054+0000","last_active":"2024-01-15T18:17:18.440100+0000","last_peered":"2024-01-15T18:17:18.440100+0000","last_clean":"2024-01-15T18:17:18.440100+0000","last_became_active":"2024-01-15T18:17:17.129779+0000","last_became_peered":"2024-01-15T18:17:17.129779+0000","last_unstale":"2024-01-15T18:17:18.440100+0000","last_undegraded":"2024-01-15T18:17:18.440100+0000","last_fullsized":"2024-01-15T18:17:18.440100+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:42:04.787652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440698+0000","last_change":"2024-01-15T18:17:17.132156+0000","last_active":"2024-01-15T18:17:18.440698+0000","last_peered":"2024-01-15T18:17:18.440698+0000","last_clean":"2024-01-15T18:17:18.440698+0000","last_became_active":"2024-01-15T18:17:17.131822+0000","last_became_peered":"2024-01-15T18:17:17.131822+0000","last_unstale":"2024-01-15T18:17:18.440698+0000","last_undegraded":"2024-01-15T18:17:18.440698+0000","last_fullsized":"2024-01-15T18:17:18.440698+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:33:59.696616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440705+0000","last_change":"2024-01-15T18:17:17.127796+0000","last_active":"2024-01-15T18:17:18.440705+0000","last_peered":"2024-01-15T18:17:18.440705+0000","last_clean":"2024-01-15T18:17:18.440705+0000","last_became_active":"2024-01-15T18:17:17.127390+0000","last_became_peered":"2024-01-15T18:17:17.127390+0000","last_unstale":"2024-01-15T18:17:18.440705+0000","last_undegraded":"2024-01-15T18:17:18.440705+0000","last_fullsized":"2024-01-15T18:17:18.440705+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:03:11.357766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1f","version":"96'4","reported_seq":16,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440784+0000","last_change":"2024-01-15T18:17:16.117003+0000","last_active":"2024-01-15T18:17:18.440784+0000","last_peered":"2024-01-15T18:17:18.440784+0000","last_clean":"2024-01-15T18:17:18.440784+0000","last_became_active":"2024-01-15T18:17:16.116625+0000","last_became_peered":"2024-01-15T18:17:16.116625+0000","last_unstale":"2024-01-15T18:17:18.440784+0000","last_undegraded":"2024-01-15T18:17:18.440784+0000","last_fullsized":"2024-01-15T18:17:18.440784+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:11:58.418627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.096546+0000","last_change":"2024-01-15T18:17:15.096546+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.096546+0000","last_undegraded":"2024-01-15T18:17:15.096546+0000","last_fullsized":"2024-01-15T18:17:15.096546+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1d","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440270+0000","last_change":"2024-01-15T18:17:16.115473+0000","last_active":"2024-01-15T18:17:18.440270+0000","last_peered":"2024-01-15T18:17:18.440270+0000","last_clean":"2024-01-15T18:17:18.440270+0000","last_became_active":"2024-01-15T18:17:16.115189+0000","last_became_peered":"2024-01-15T18:17:16.115189+0000","last_unstale":"2024-01-15T18:17:18.440270+0000","last_undegraded":"2024-01-15T18:17:18.440270+0000","last_fullsized":"2024-01-15T18:17:18.440270+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:58:47.776409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.133184+0000","last_change":"2024-01-15T18:17:17.127577+0000","last_active":"2024-01-15T18:17:18.133184+0000","last_peered":"2024-01-15T18:17:18.133184+0000","last_clean":"2024-01-15T18:17:18.133184+0000","last_became_active":"2024-01-15T18:17:17.125488+0000","last_became_peered":"2024-01-15T18:17:17.125488+0000","last_unstale":"2024-01-15T18:17:18.133184+0000","last_undegraded":"2024-01-15T18:17:18.133184+0000","last_fullsized":"2024-01-15T18:17:18.133184+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:23.103323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440326+0000","last_change":"2024-01-15T18:17:16.123895+0000","last_active":"2024-01-15T18:17:18.440326+0000","last_peered":"2024-01-15T18:17:18.440326+0000","last_clean":"2024-01-15T18:17:18.440326+0000","last_became_active":"2024-01-15T18:17:16.123325+0000","last_became_peered":"2024-01-15T18:17:16.123325+0000","last_unstale":"2024-01-15T18:17:18.440326+0000","last_undegraded":"2024-01-15T18:17:18.440326+0000","last_fullsized":"2024-01-15T18:17:18.440326+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:13:09.312116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.131438+0000","last_change":"2024-01-15T18:17:17.118290+0000","last_active":"2024-01-15T18:17:18.131438+0000","last_peered":"2024-01-15T18:17:18.131438+0000","last_clean":"2024-01-15T18:17:18.131438+0000","last_became_active":"2024-01-15T18:17:17.117943+0000","last_became_peered":"2024-01-15T18:17:17.117943+0000","last_unstale":"2024-01-15T18:17:18.131438+0000","last_undegraded":"2024-01-15T18:17:18.131438+0000","last_fullsized":"2024-01-15T18:17:18.131438+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:50:04.755553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.093523+0000","last_change":"2024-01-15T18:17:15.093523+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.093523+0000","last_undegraded":"2024-01-15T18:17:15.093523+0000","last_fullsized":"2024-01-15T18:17:15.093523+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1a","version":"96'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440314+0000","last_change":"2024-01-15T18:17:16.121483+0000","last_active":"2024-01-15T18:17:18.440314+0000","last_peered":"2024-01-15T18:17:18.440314+0000","last_clean":"2024-01-15T18:17:18.440314+0000","last_became_active":"2024-01-15T18:17:16.121199+0000","last_became_peered":"2024-01-15T18:17:16.121199+0000","last_unstale":"2024-01-15T18:17:18.440314+0000","last_undegraded":"2024-01-15T18:17:18.440314+0000","last_fullsized":"2024-01-15T18:17:18.440314+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:51:29.086954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.095582+0000","last_change":"2024-01-15T18:17:15.095582+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.095582+0000","last_undegraded":"2024-01-15T18:17:15.095582+0000","last_fullsized":"2024-01-15T18:17:15.095582+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.132487+0000","last_change":"2024-01-15T18:17:17.119962+0000","last_active":"2024-01-15T18:17:18.132487+0000","last_peered":"2024-01-15T18:17:18.132487+0000","last_clean":"2024-01-15T18:17:18.132487+0000","last_became_active":"2024-01-15T18:17:17.118612+0000","last_became_peered":"2024-01-15T18:17:17.118612+0000","last_unstale":"2024-01-15T18:17:18.132487+0000","last_undegraded":"2024-01-15T18:17:18.132487+0000","last_fullsized":"2024-01-15T18:17:18.132487+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:45:26.435028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.093925+0000","last_change":"2024-01-15T18:17:15.093925+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.093925+0000","last_undegraded":"2024-01-15T18:17:15.093925+0000","last_fullsized":"2024-01-15T18:17:15.093925+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440790+0000","last_change":"2024-01-15T18:17:17.124377+0000","last_active":"2024-01-15T18:17:18.440790+0000","last_peered":"2024-01-15T18:17:18.440790+0000","last_clean":"2024-01-15T18:17:18.440790+0000","last_became_active":"2024-01-15T18:17:17.124096+0000","last_became_peered":"2024-01-15T18:17:17.124096+0000","last_unstale":"2024-01-15T18:17:18.440790+0000","last_undegraded":"2024-01-15T18:17:18.440790+0000","last_fullsized":"2024-01-15T18:17:18.440790+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:56:21.720071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.093475+0000","last_change":"2024-01-15T18:17:15.093475+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.093475+0000","last_undegraded":"2024-01-15T18:17:15.093475+0000","last_fullsized":"2024-01-15T18:17:15.093475+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.133177+0000","last_change":"2024-01-15T18:17:17.122838+0000","last_active":"2024-01-15T18:17:18.133177+0000","last_peered":"2024-01-15T18:17:18.133177+0000","last_clean":"2024-01-15T18:17:18.133177+0000","last_became_active":"2024-01-15T18:17:17.122579+0000","last_became_peered":"2024-01-15T18:17:17.122579+0000","last_unstale":"2024-01-15T18:17:18.133177+0000","last_undegraded":"2024-01-15T18:17:18.133177+0000","last_fullsized":"2024-01-15T18:17:18.133177+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:13:53.357957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.095406+0000","last_change":"2024-01-15T18:17:15.095406+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.095406+0000","last_undegraded":"2024-01-15T18:17:15.095406+0000","last_fullsized":"2024-01-15T18:17:15.095406+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.096252+0000","last_change":"2024-01-15T18:17:15.096252+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.096252+0000","last_undegraded":"2024-01-15T18:17:15.096252+0000","last_fullsized":"2024-01-15T18:17:15.096252+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.14","version":"96'6","reported_seq":18,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.148952+0000","last_change":"2024-01-15T18:17:16.120030+0000","last_active":"2024-01-15T18:17:17.148952+0000","last_peered":"2024-01-15T18:17:17.148952+0000","last_clean":"2024-01-15T18:17:17.148952+0000","last_became_active":"2024-01-15T18:17:16.119705+0000","last_became_peered":"2024-01-15T18:17:16.119705+0000","last_unstale":"2024-01-15T18:17:17.148952+0000","last_undegraded":"2024-01-15T18:17:17.148952+0000","last_fullsized":"2024-01-15T18:17:17.148952+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:47:48.944557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.094637+0000","last_change":"2024-01-15T18:17:15.094637+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.094637+0000","last_undegraded":"2024-01-15T18:17:15.094637+0000","last_fullsized":"2024-01-15T18:17:15.094637+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.094404+0000","last_change":"2024-01-15T18:17:15.094404+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.094404+0000","last_undegraded":"2024-01-15T18:17:15.094404+0000","last_fullsized":"2024-01-15T18:17:15.094404+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.132434+0000","last_change":"2024-01-15T18:17:17.121456+0000","last_active":"2024-01-15T18:17:18.132434+0000","last_peered":"2024-01-15T18:17:18.132434+0000","last_clean":"2024-01-15T18:17:18.132434+0000","last_became_active":"2024-01-15T18:17:17.121179+0000","last_became_peered":"2024-01-15T18:17:17.121179+0000","last_unstale":"2024-01-15T18:17:18.132434+0000","last_undegraded":"2024-01-15T18:17:18.132434+0000","last_fullsized":"2024-01-15T18:17:18.132434+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:53:26.915291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.111628+0000","last_change":"2024-01-15T18:17:16.113394+0000","last_active":"2024-01-15T18:17:17.111628+0000","last_peered":"2024-01-15T18:17:17.111628+0000","last_clean":"2024-01-15T18:17:17.111628+0000","last_became_active":"2024-01-15T18:17:16.113105+0000","last_became_peered":"2024-01-15T18:17:16.113105+0000","last_unstale":"2024-01-15T18:17:17.111628+0000","last_undegraded":"2024-01-15T18:17:17.111628+0000","last_fullsized":"2024-01-15T18:17:17.111628+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:41:23.459232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.093447+0000","last_change":"2024-01-15T18:17:15.093447+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.093447+0000","last_undegraded":"2024-01-15T18:17:15.093447+0000","last_fullsized":"2024-01-15T18:17:15.093447+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.e","version":"96'2","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.143716+0000","last_change":"2024-01-15T18:17:16.108046+0000","last_active":"2024-01-15T18:17:17.143716+0000","last_peered":"2024-01-15T18:17:17.143716+0000","last_clean":"2024-01-15T18:17:17.143716+0000","last_became_active":"2024-01-15T18:17:16.107750+0000","last_became_peered":"2024-01-15T18:17:16.107750+0000","last_unstale":"2024-01-15T18:17:17.143716+0000","last_undegraded":"2024-01-15T18:17:17.143716+0000","last_fullsized":"2024-01-15T18:17:17.143716+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:14.320865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.131540+0000","last_change":"2024-01-15T18:17:17.123658+0000","last_active":"2024-01-15T18:17:18.131540+0000","last_peered":"2024-01-15T18:17:18.131540+0000","last_clean":"2024-01-15T18:17:18.131540+0000","last_became_active":"2024-01-15T18:17:17.123351+0000","last_became_peered":"2024-01-15T18:17:17.123351+0000","last_unstale":"2024-01-15T18:17:18.131540+0000","last_undegraded":"2024-01-15T18:17:18.131540+0000","last_fullsized":"2024-01-15T18:17:18.131540+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:38:11.462593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.095378+0000","last_change":"2024-01-15T18:17:15.095378+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.095378+0000","last_undegraded":"2024-01-15T18:17:15.095378+0000","last_fullsized":"2024-01-15T18:17:15.095378+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440407+0000","last_change":"2024-01-15T18:17:16.125410+0000","last_active":"2024-01-15T18:17:18.440407+0000","last_peered":"2024-01-15T18:17:18.440407+0000","last_clean":"2024-01-15T18:17:18.440407+0000","last_became_active":"2024-01-15T18:17:16.124440+0000","last_became_peered":"2024-01-15T18:17:16.124440+0000","last_unstale":"2024-01-15T18:17:18.440407+0000","last_undegraded":"2024-01-15T18:17:18.440407+0000","last_fullsized":"2024-01-15T18:17:18.440407+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:37:51.051323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.b","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440507+0000","last_change":"2024-01-15T18:17:16.120398+0000","last_active":"2024-01-15T18:17:18.440507+0000","last_peered":"2024-01-15T18:17:18.440507+0000","last_clean":"2024-01-15T18:17:18.440507+0000","last_became_active":"2024-01-15T18:17:16.119944+0000","last_became_peered":"2024-01-15T18:17:16.119944+0000","last_unstale":"2024-01-15T18:17:18.440507+0000","last_undegraded":"2024-01-15T18:17:18.440507+0000","last_fullsized":"2024-01-15T18:17:18.440507+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:54:02.313160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.131461+0000","last_change":"2024-01-15T18:17:16.102303+0000","last_active":"2024-01-15T18:17:18.131461+0000","last_peered":"2024-01-15T18:17:18.131461+0000","last_clean":"2024-01-15T18:17:18.131461+0000","last_became_active":"2024-01-15T18:17:16.099773+0000","last_became_peered":"2024-01-15T18:17:16.099773+0000","last_unstale":"2024-01-15T18:17:18.131461+0000","last_undegraded":"2024-01-15T18:17:18.131461+0000","last_fullsized":"2024-01-15T18:17:18.131461+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:40:51.210210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.131385+0000","last_change":"2024-01-15T18:17:16.119288+0000","last_active":"2024-01-15T18:17:18.131385+0000","last_peered":"2024-01-15T18:17:18.131385+0000","last_clean":"2024-01-15T18:17:18.131385+0000","last_became_active":"2024-01-15T18:17:16.118972+0000","last_became_peered":"2024-01-15T18:17:16.118972+0000","last_unstale":"2024-01-15T18:17:18.131385+0000","last_undegraded":"2024-01-15T18:17:18.131385+0000","last_fullsized":"2024-01-15T18:17:18.131385+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:14:29.755549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439494+0000","last_change":"2024-01-15T18:17:16.127573+0000","last_active":"2024-01-15T18:17:18.439494+0000","last_peered":"2024-01-15T18:17:18.439494+0000","last_clean":"2024-01-15T18:17:18.439494+0000","last_became_active":"2024-01-15T18:17:16.127194+0000","last_became_peered":"2024-01-15T18:17:16.127194+0000","last_unstale":"2024-01-15T18:17:18.439494+0000","last_undegraded":"2024-01-15T18:17:18.439494+0000","last_fullsized":"2024-01-15T18:17:18.439494+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:48:07.276946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440832+0000","last_change":"2024-01-15T18:17:17.132695+0000","last_active":"2024-01-15T18:17:18.440832+0000","last_peered":"2024-01-15T18:17:18.440832+0000","last_clean":"2024-01-15T18:17:18.440832+0000","last_became_active":"2024-01-15T18:17:17.132430+0000","last_became_peered":"2024-01-15T18:17:17.132430+0000","last_unstale":"2024-01-15T18:17:18.440832+0000","last_undegraded":"2024-01-15T18:17:18.440832+0000","last_fullsized":"2024-01-15T18:17:18.440832+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:14:02.211346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.132130+0000","last_change":"2024-01-15T18:17:16.102078+0000","last_active":"2024-01-15T18:17:18.132130+0000","last_peered":"2024-01-15T18:17:18.132130+0000","last_clean":"2024-01-15T18:17:18.132130+0000","last_became_active":"2024-01-15T18:17:16.101772+0000","last_became_peered":"2024-01-15T18:17:16.101772+0000","last_unstale":"2024-01-15T18:17:18.132130+0000","last_undegraded":"2024-01-15T18:17:18.132130+0000","last_fullsized":"2024-01-15T18:17:18.132130+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:28:05.931436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440855+0000","last_change":"2024-01-15T18:17:16.121035+0000","last_active":"2024-01-15T18:17:18.440855+0000","last_peered":"2024-01-15T18:17:18.440855+0000","last_clean":"2024-01-15T18:17:18.440855+0000","last_became_active":"2024-01-15T18:17:16.120714+0000","last_became_peered":"2024-01-15T18:17:16.120714+0000","last_unstale":"2024-01-15T18:17:18.440855+0000","last_undegraded":"2024-01-15T18:17:18.440855+0000","last_fullsized":"2024-01-15T18:17:18.440855+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:30:03.158862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440319+0000","last_change":"2024-01-15T18:17:16.114051+0000","last_active":"2024-01-15T18:17:18.440319+0000","last_peered":"2024-01-15T18:17:18.440319+0000","last_clean":"2024-01-15T18:17:18.440319+0000","last_became_active":"2024-01-15T18:17:16.113079+0000","last_became_peered":"2024-01-15T18:17:16.113079+0000","last_unstale":"2024-01-15T18:17:18.440319+0000","last_undegraded":"2024-01-15T18:17:18.440319+0000","last_fullsized":"2024-01-15T18:17:18.440319+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:25:22.584736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.095295+0000","last_change":"2024-01-15T18:17:15.095295+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.095295+0000","last_undegraded":"2024-01-15T18:17:15.095295+0000","last_fullsized":"2024-01-15T18:17:15.095295+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440562+0000","last_change":"2024-01-15T18:17:16.121786+0000","last_active":"2024-01-15T18:17:18.440562+0000","last_peered":"2024-01-15T18:17:18.440562+0000","last_clean":"2024-01-15T18:17:18.440562+0000","last_became_active":"2024-01-15T18:17:16.121508+0000","last_became_peered":"2024-01-15T18:17:16.121508+0000","last_unstale":"2024-01-15T18:17:18.440562+0000","last_undegraded":"2024-01-15T18:17:18.440562+0000","last_fullsized":"2024-01-15T18:17:18.440562+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:51:43.523481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.133059+0000","last_change":"2024-01-15T18:17:17.126022+0000","last_active":"2024-01-15T18:17:18.133059+0000","last_peered":"2024-01-15T18:17:18.133059+0000","last_clean":"2024-01-15T18:17:18.133059+0000","last_became_active":"2024-01-15T18:17:17.125366+0000","last_became_peered":"2024-01-15T18:17:17.125366+0000","last_unstale":"2024-01-15T18:17:18.133059+0000","last_undegraded":"2024-01-15T18:17:18.133059+0000","last_fullsized":"2024-01-15T18:17:18.133059+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:27:59.524326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.095318+0000","last_change":"2024-01-15T18:17:15.095318+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.095318+0000","last_undegraded":"2024-01-15T18:17:15.095318+0000","last_fullsized":"2024-01-15T18:17:15.095318+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.095327+0000","last_change":"2024-01-15T18:17:15.095327+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.095327+0000","last_undegraded":"2024-01-15T18:17:15.095327+0000","last_fullsized":"2024-01-15T18:17:15.095327+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.094409+0000","last_change":"2024-01-15T18:17:15.094409+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.094409+0000","last_undegraded":"2024-01-15T18:17:15.094409+0000","last_fullsized":"2024-01-15T18:17:15.094409+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.096380+0000","last_change":"2024-01-15T18:17:15.096380+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.096380+0000","last_undegraded":"2024-01-15T18:17:15.096380+0000","last_fullsized":"2024-01-15T18:17:15.096380+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":227,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-01-15T18:17:13.089307+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:17:13.089307+0000","last_peered":"2024-01-15T18:17:13.089307+0000","last_clean":"2024-01-15T18:17:13.089307+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:17:13.089307+0000","last_undegraded":"2024-01-15T18:17:13.089307+0000","last_fullsized":"2024-01-15T18:17:13.089307+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":38,"acting":38,"num_store_stats":3},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2194,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21,"ondisk_log_size":21,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509593,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64748,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307092,"statfs":{"total":96636764160,"available":96570462208,"internally_reserved":0,"allocated":1736704,"data_stored":1082552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64551370},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.87,"15min":0.871},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.90800000000000003,"5min":9.3040000000000003,"15min":10.284000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.83899999999999997,"15min":0.91600000000000004},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.378},"max":{"1min":6.6189999999999998,"5min":6.6189999999999998,"15min":16.968},"last":0.68100000000000005}]},{"osd":1,"last update":"Mon Jan 15 18:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.93200000000000005,"15min":0.92700000000000005},"min":{"1min":0.46000000000000002,"5min":0.36099999999999999,"15min":0.33800000000000002},"max":{"1min":4.944,"5min":9.3829999999999991,"15min":15.791},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.84499999999999997,"15min":0.90800000000000003},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.371},"max":{"1min":4.6699999999999999,"5min":5.0899999999999999,"15min":10.842000000000001},"last":0.72799999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":1.159,"15min":0.93500000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.378},"max":{"1min":4.4669999999999996,"5min":22.318000000000001,"15min":22.318000000000001},"last":0.63},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.92500000000000004,"15min":0.90600000000000003},"min":{"1min":0.433,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":4.7030000000000003,"5min":9.5169999999999995,"15min":9.7539999999999996},"last":0.59899999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":1.163,"15min":0.98999999999999999},"min":{"1min":0.50900000000000001,"5min":0.503,"15min":0.38100000000000001},"max":{"1min":1.456,"5min":18.152999999999999,"15min":18.152999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.95899999999999996,"15min":0.98499999999999999},"min":{"1min":0.49199999999999999,"5min":0.433,"15min":0.43099999999999999},"max":{"1min":2.6909999999999998,"5min":9.75,"15min":10.34},"last":0.70599999999999996}]},{"osd":4,"last update":"Mon Jan 15 18:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.84899999999999998,"15min":0.93500000000000005},"min":{"1min":0.38900000000000001,"5min":0.314,"15min":0.314},"max":{"1min":4.4480000000000004,"5min":8.4420000000000002,"15min":20.387},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.73199999999999998,"5min":1.1699999999999999,"15min":0.98799999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":3.6309999999999998,"5min":17.215,"15min":17.215},"last":0.58499999999999996}]},{"osd":5,"last update":"Mon Jan 15 18:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.94499999999999995,"15min":0.95099999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":4.8639999999999999,"5min":9.6630000000000003,"15min":19.959},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.91300000000000003,"15min":0.98099999999999998},"min":{"1min":0.39400000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":4.3959999999999999,"5min":7.6980000000000004,"15min":19.905000000000001},"last":0.79300000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.99099999999999999,"15min":1.155},"min":{"1min":0.505,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":4.3330000000000002,"5min":8.6609999999999996,"15min":22.478999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.82399999999999995,"5min":1.1579999999999999,"15min":1.0660000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.38300000000000001},"max":{"1min":3.573,"5min":17.132000000000001,"15min":21.509},"last":0.69799999999999995}]}]},{"osd":6,"up_from":15,"seq":64424509593,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80812,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94291028,"statfs":{"total":96636764160,"available":96554012672,"internally_reserved":0,"allocated":1142784,"data_stored":492184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81590726},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":1.0529999999999999,"15min":0.97799999999999998},"min":{"1min":0.40000000000000002,"5min":0.32800000000000001,"15min":0.29099999999999998},"max":{"1min":0.86699999999999999,"5min":32.587000000000003,"15min":32.587000000000003},"last":0.873},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.91600000000000004,"15min":0.86299999999999999},"min":{"1min":0.41999999999999998,"5min":0.38500000000000001,"15min":0.35399999999999998},"max":{"1min":0.88400000000000001,"5min":16.193000000000001,"15min":16.193000000000001},"last":0.59399999999999997}]},{"osd":1,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.82499999999999996,"15min":0.99399999999999999},"min":{"1min":0.40600000000000003,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95299999999999996,"5min":7.7649999999999997,"15min":53.451999999999998},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1,"15min":0.91800000000000004},"min":{"1min":0.49299999999999999,"5min":0.437,"15min":0.42899999999999999},"max":{"1min":0.89900000000000002,"5min":32.512,"15min":32.512},"last":0.64400000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":1.087,"15min":0.97099999999999997},"min":{"1min":0.48299999999999998,"5min":0.41699999999999998,"15min":0.38900000000000001},"max":{"1min":0.90100000000000002,"5min":32.472999999999999,"15min":32.472999999999999},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.997,"15min":0.89600000000000002},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.38500000000000001},"max":{"1min":0.97899999999999998,"5min":16.602,"15min":16.602},"last":0.53500000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.97999999999999998,"15min":0.93100000000000005},"min":{"1min":0.50700000000000001,"5min":0.49399999999999999,"15min":0.41399999999999998},"max":{"1min":0.94499999999999995,"5min":16.388000000000002,"15min":16.388000000000002},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.879,"15min":1.079},"min":{"1min":0.52300000000000002,"5min":0.432,"15min":0.41599999999999998},"max":{"1min":0.996,"5min":5.9740000000000002,"15min":53.398000000000003},"last":0.85299999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.98799999999999999,"15min":0.89100000000000001},"min":{"1min":0.42999999999999999,"5min":0.38100000000000001,"15min":0.34999999999999998},"max":{"1min":0.92200000000000004,"5min":32.549999999999997,"15min":32.549999999999997},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.878,"15min":0.83999999999999997},"min":{"1min":0.42299999999999999,"5min":0.38700000000000001,"15min":0.34499999999999997},"max":{"1min":1.087,"5min":11.852,"15min":11.852},"last":0.75600000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.82699999999999996,"15min":1.0369999999999999},"min":{"1min":0.36199999999999999,"5min":0.35799999999999998,"15min":0.30399999999999999},"max":{"1min":1.1759999999999999,"5min":7.5650000000000004,"15min":53.229999999999997},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.081,"15min":1.028},"min":{"1min":0.432,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.228,"5min":32.404000000000003,"15min":32.404000000000003},"last":0.92600000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.93899999999999995,"15min":1.105},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.1220000000000001,"5min":16.565000000000001,"15min":45.070999999999998},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.68799999999999994,"5min":1.161,"15min":1.21},"min":{"1min":0.46300000000000002,"5min":0.40999999999999998,"15min":0.34999999999999998},"max":{"1min":1.032,"5min":40.972999999999999,"15min":53.338000000000001},"last":0.90700000000000003}]}]},{"osd":4,"up_from":15,"seq":64424509594,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76724,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295116,"statfs":{"total":96636764160,"available":96558198784,"internally_reserved":0,"allocated":1212416,"data_stored":519535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.82499999999999996,"15min":0.88400000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.32900000000000001},"max":{"1min":0.91800000000000004,"5min":7.3380000000000001,"15min":23.318000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.70199999999999996,"15min":0.79300000000000004},"min":{"1min":0.372,"5min":0.36399999999999999,"15min":0.32600000000000001},"max":{"1min":0.82699999999999996,"5min":6.444,"15min":14.723000000000001},"last":0.72399999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.82699999999999996,"15min":0.78800000000000003},"min":{"1min":0.39600000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.2,"5min":9.5700000000000003,"15min":9.5700000000000003},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.92100000000000004,"15min":0.91600000000000004},"min":{"1min":0.45200000000000001,"5min":0.40200000000000002,"15min":0.36899999999999999},"max":{"1min":0.84099999999999997,"5min":7.9980000000000002,"15min":22.408999999999999},"last":0.65600000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.85399999999999998,"15min":0.79900000000000004},"min":{"1min":0.439,"5min":0.439,"15min":0.34399999999999997},"max":{"1min":1.266,"5min":9.7089999999999996,"15min":9.7089999999999996},"last":0.629},{"interface":"front","average":{"1min":0.73699999999999999,"5min":1.0649999999999999,"15min":1.0149999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.39100000000000001},"max":{"1min":1.714,"5min":14.118,"15min":22.338999999999999},"last":0.77000000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.77800000000000002,"15min":0.872},"min":{"1min":0.437,"5min":0.437,"15min":0.41199999999999998},"max":{"1min":0.88800000000000001,"5min":6.7930000000000001,"15min":14.763999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.94199999999999995,"15min":0.90600000000000003},"min":{"1min":0.504,"5min":0.48099999999999998,"15min":0.45500000000000002},"max":{"1min":0.93600000000000005,"5min":9.6289999999999996,"15min":9.6289999999999996},"last":0.81399999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.94599999999999995,"15min":0.95699999999999996},"min":{"1min":0.40000000000000002,"5min":0.34799999999999998,"15min":0.33200000000000002},"max":{"1min":0.94099999999999995,"5min":15.787000000000001,"15min":31.617999999999999},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.75,"15min":0.86399999999999999},"min":{"1min":0.42599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95299999999999996,"5min":6.7169999999999996,"15min":17.145},"last":0.79600000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.84899999999999998,"15min":0.82799999999999996},"min":{"1min":0.42299999999999999,"5min":0.39800000000000002,"15min":0.35499999999999998},"max":{"1min":0.97799999999999998,"5min":9.6709999999999994,"15min":11.124000000000001},"last":0.751},{"interface":"front","average":{"1min":0.69799999999999995,"5min":1.1699999999999999,"15min":1.2230000000000001},"min":{"1min":0.56599999999999995,"5min":0.438,"15min":0.40200000000000002},"max":{"1min":0.90200000000000002,"5min":32.308,"15min":32.308},"last":0.82299999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.749,"15min":1.0029999999999999},"min":{"1min":0.434,"5min":0.41099999999999998,"15min":0.34799999999999998},"max":{"1min":0.91900000000000004,"5min":6.8390000000000004,"15min":39.256},"last":0.876},{"interface":"front","average":{"1min":0.70399999999999996,"5min":1.1160000000000001,"15min":1.004},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.42199999999999999},"max":{"1min":0.86199999999999999,"5min":16.277000000000001,"15min":26.573},"last":0.86299999999999999}]}]},{"osd":3,"up_from":15,"seq":64424509592,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64172,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307668,"statfs":{"total":96636764160,"available":96571052032,"internally_reserved":0,"allocated":1142784,"data_stored":492184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64551367},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.44900000000000001,"15min":0.48899999999999999},"min":{"1min":0.27300000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.624,"5min":2.1160000000000001,"15min":5.8849999999999998},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.54100000000000004,"15min":0.61599999999999999},"min":{"1min":0.312,"5min":0.29799999999999999,"15min":0.29499999999999998},"max":{"1min":0.66600000000000004,"5min":3.0649999999999999,"15min":17.756},"last":0.51800000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.495,"15min":0.58499999999999996},"min":{"1min":0.28899999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68700000000000006,"5min":3.6389999999999998,"15min":12.307},"last":0.435},{"interface":"front","average":{"1min":0.503,"5min":0.55500000000000005,"15min":0.58499999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.70299999999999996,"5min":3.738,"15min":6.0640000000000001},"last":0.41399999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.65400000000000003,"15min":0.68999999999999995},"min":{"1min":0.33700000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69999999999999996,"5min":5.431,"15min":12.468999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.70299999999999996,"15min":0.68300000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.34599999999999997},"max":{"1min":0.77800000000000002,"5min":10.837999999999999,"15min":10.837999999999999},"last":0.748}]},{"osd":4,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":1.0609999999999999,"15min":1.014},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.375},"max":{"1min":1.024,"5min":13.863,"15min":17.853999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.94999999999999996,"15min":0.96199999999999997},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.47099999999999997},"max":{"1min":0.94599999999999995,"5min":6.8289999999999997,"15min":13.31},"last":0.64100000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.153,"15min":0.97399999999999998},"min":{"1min":0.54200000000000004,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1020000000000001,"5min":17.687999999999999,"15min":17.687999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.88500000000000001,"15min":0.96999999999999997},"min":{"1min":0.45500000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.085,"5min":5.0949999999999998,"15min":20.515999999999998},"last":0.86899999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.80200000000000005,"15min":0.97699999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.432},"max":{"1min":1.05,"5min":4.0910000000000002,"15min":23.100999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.90000000000000002,"15min":0.93600000000000005},"min":{"1min":0.59399999999999997,"5min":0.53600000000000003,"15min":0.49399999999999999},"max":{"1min":1.0069999999999999,"5min":6.7220000000000004,"15min":12.76},"last":0.70399999999999996}]},{"osd":7,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":1.331,"15min":1.1659999999999999},"min":{"1min":0.503,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.0089999999999999,"5min":24.321999999999999,"15min":24.321999999999999},"last":0.90900000000000003},{"interface":"front","average":{"1min":2.0510000000000002,"5min":1.139,"15min":1.1020000000000001},"min":{"1min":0.54700000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"last":0.60999999999999999}]}]},{"osd":2,"up_from":15,"seq":64424509592,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64184,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307656,"statfs":{"total":96636764160,"available":96571039744,"internally_reserved":0,"allocated":1155072,"data_stored":498113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.45500000000000002,"15min":0.50900000000000001},"min":{"1min":0.26500000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.71299999999999997,"5min":0.98199999999999998,"15min":5.1849999999999996},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.49199999999999999,"15min":0.53200000000000003},"min":{"1min":0.315,"5min":0.252,"15min":0.252},"max":{"1min":0.68899999999999995,"5min":1.1579999999999999,"15min":7.516},"last":0.51900000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.55800000000000005,"15min":0.61599999999999999},"min":{"1min":0.39800000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.83299999999999996,"5min":0.95799999999999996,"15min":10.907},"last":0.371},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.50800000000000001,"15min":0.59199999999999997},"min":{"1min":0.38200000000000001,"5min":0.28100000000000003,"15min":0.27000000000000002},"max":{"1min":0.77500000000000002,"5min":1.2849999999999999,"15min":8.9190000000000005},"last":0.45900000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55500000000000005,"15min":0.623},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0249999999999999,"5min":1.133,"15min":12.185},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.60199999999999998,"15min":0.66400000000000003},"min":{"1min":0.40600000000000003,"5min":0.373,"15min":0.34000000000000002},"max":{"1min":0.79000000000000004,"5min":0.96699999999999997,"15min":7.6760000000000002},"last":0.439}]},{"osd":4,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.88800000000000001,"15min":0.875},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":4.0190000000000001,"5min":4.3150000000000004,"15min":8.6609999999999996},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.76200000000000001,"15min":0.90300000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.5860000000000001,"5min":2.468,"15min":19.146999999999998},"last":0.56599999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":0.79700000000000004,"15min":0.90800000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":5.0309999999999997,"5min":5.0309999999999997,"15min":31.710000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.83999999999999997,"15min":0.999},"min":{"1min":0.51100000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.3660000000000001,"5min":3.6459999999999999,"15min":22.539000000000001},"last":0.60999999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.84099999999999997,"15min":0.90600000000000003},"min":{"1min":0.499,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":4.1959999999999997,"5min":4.1959999999999997,"15min":9.4540000000000006},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.79200000000000004,"15min":0.90600000000000003},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.633,"5min":1.633,"15min":18.614999999999998},"last":0.68899999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.97799999999999998,"15min":0.96199999999999997},"min":{"1min":0.52100000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":4.3170000000000002,"5min":11.606,"15min":11.606},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.159,"5min":0.92900000000000005,"15min":1.004},"min":{"1min":0.54900000000000004,"5min":0.52500000000000002,"15min":0.502},"max":{"1min":6.1260000000000003,"5min":6.1260000000000003,"15min":17.013000000000002},"last":0.70599999999999996}]}]},{"osd":1,"up_from":15,"seq":64424509594,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68332,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303508,"statfs":{"total":96636764160,"available":96566792192,"internally_reserved":0,"allocated":1142784,"data_stored":492184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.51700000000000002,"15min":0.65700000000000003},"min":{"1min":0.28399999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.6140000000000001,"5min":4.1550000000000002,"15min":19.577000000000002},"last":0.374},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.623,"15min":0.76000000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.251},"max":{"1min":0.83899999999999997,"5min":12.378,"15min":48.277999999999999},"last":0.67400000000000004}]},{"osd":2,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55400000000000005,"15min":0.70799999999999996},"min":{"1min":0.32800000000000001,"5min":0.309,"15min":0.30299999999999999},"max":{"1min":1.8049999999999999,"5min":2.8610000000000002,"15min":40.698},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59099999999999997,"15min":0.877},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.317},"max":{"1min":1.978,"5min":2.5139999999999998,"15min":48.316000000000003},"last":0.63800000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.623,"15min":0.82999999999999996},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.33700000000000002},"max":{"1min":0.94699999999999995,"5min":2.8620000000000001,"15min":48.234999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.69299999999999995,"15min":0.67700000000000005},"min":{"1min":0.33400000000000002,"5min":0.315,"15min":0.311},"max":{"1min":1.0780000000000001,"5min":9.9719999999999995,"15min":9.9719999999999995},"last":0.499}]},{"osd":4,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.90300000000000002,"15min":0.88800000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.44500000000000001},"max":{"1min":0.97499999999999998,"5min":9.9329999999999998,"15min":9.9329999999999998},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.86199999999999999,"15min":0.93300000000000005},"min":{"1min":0.502,"5min":0.45800000000000002,"15min":0.42499999999999999},"max":{"1min":2.3900000000000001,"5min":9.8610000000000007,"15min":19.829000000000001},"last":0.85599999999999998}]},{"osd":5,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.82899999999999996,"15min":1.0129999999999999},"min":{"1min":0.53700000000000003,"5min":0.48599999999999999,"15min":0.46100000000000002},"max":{"1min":2.5960000000000001,"5min":8.5839999999999996,"15min":22.594999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.80600000000000005,"15min":1.157},"min":{"1min":0.44400000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.6879999999999999,"5min":2.9609999999999999,"15min":48.198},"last":0.75700000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.90800000000000003,"15min":1.04},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0189999999999999,"5min":9.3989999999999991,"15min":35.112000000000002},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.91300000000000003,"15min":0.96899999999999997},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.50800000000000001},"max":{"1min":2.512,"5min":5.8849999999999998,"15min":19.780999999999999},"last":0.94399999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.89200000000000002,"15min":1.1619999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.93300000000000005,"5min":11.039,"15min":48.148000000000003},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.748,"5min":0.97899999999999998,"15min":0.98999999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.52100000000000002},"max":{"1min":1.0609999999999999,"5min":9.8640000000000008,"15min":9.8640000000000008},"last":0.95899999999999996}]}]},{"osd":5,"up_from":14,"seq":60129542298,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58724,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313116,"statfs":{"total":96636764160,"available":96576630784,"internally_reserved":0,"allocated":1196032,"data_stored":517869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.81200000000000006,"15min":0.80100000000000005},"min":{"1min":0.375,"5min":0.29899999999999999,"15min":0.28999999999999998},"max":{"1min":1.383,"5min":5.1639999999999997,"15min":21.385000000000002},"last":0.874},{"interface":"front","average":{"1min":0.748,"5min":0.78700000000000003,"15min":0.877},"min":{"1min":0.57999999999999996,"5min":0.378,"15min":0.34899999999999998},"max":{"1min":1.3029999999999999,"5min":5.3869999999999996,"15min":9.359},"last":0.67100000000000004}]},{"osd":1,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.82199999999999995,"15min":0.89500000000000002},"min":{"1min":0.44400000000000001,"5min":0.375,"15min":0.34899999999999998},"max":{"1min":1.2070000000000001,"5min":3.7429999999999999,"15min":21.120000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.78500000000000003,"5min":1.1850000000000001,"15min":0.96999999999999997},"min":{"1min":0.48999999999999999,"5min":0.38600000000000001,"15min":0.315},"max":{"1min":1.532,"5min":15.843,"15min":21.5},"last":0.90000000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.83799999999999997,"15min":0.89800000000000002},"min":{"1min":0.47699999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.421,"5min":5.516,"15min":8.7750000000000004},"last":0.622},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.90800000000000003,"15min":0.98799999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.347,"5min":8.3529999999999998,"15min":21.201000000000001},"last":0.68700000000000006}]},{"osd":3,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.93200000000000005,"15min":0.91300000000000003},"min":{"1min":0.58499999999999996,"5min":0.37,"15min":0.37},"max":{"1min":1.478,"5min":7.7199999999999998,"15min":21.457000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.85499999999999998,"15min":0.96299999999999997},"min":{"1min":0.57199999999999995,"5min":0.47699999999999998,"15min":0.45400000000000001},"max":{"1min":1.5,"5min":5.6109999999999998,"15min":8.7300000000000004},"last":0.78900000000000003}]},{"osd":4,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.93500000000000005,"15min":0.88800000000000001},"min":{"1min":0.376,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":4.8650000000000002,"5min":10.962,"15min":10.962},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.78800000000000003,"15min":1.018},"min":{"1min":0.442,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.464,"5min":4.6639999999999997,"15min":26.146999999999998},"last":0.80700000000000005}]},{"osd":6,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.86599999999999999,"15min":0.83799999999999997},"min":{"1min":0.44700000000000001,"5min":0.36099999999999999,"15min":0.32000000000000001},"max":{"1min":2.8940000000000001,"5min":5.2160000000000002,"15min":21.535},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.85899999999999999,"15min":1.0129999999999999},"min":{"1min":0.432,"5min":0.379,"15min":0.36299999999999999},"max":{"1min":4.9139999999999997,"5min":4.9139999999999997,"15min":30.488},"last":0.73899999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.276,"15min":1.2290000000000001},"min":{"1min":0.51800000000000002,"5min":0.432,"15min":0.39700000000000002},"max":{"1min":8.7539999999999996,"5min":33.499000000000002,"15min":33.499000000000002},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.98599999999999999,"15min":1.0900000000000001},"min":{"1min":0.46200000000000002,"5min":0.441,"15min":0.36499999999999999},"max":{"1min":4.3120000000000003,"5min":6.306,"15min":25.437999999999999},"last":0.76200000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542298,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64800,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307040,"statfs":{"total":96636764160,"available":96570408960,"internally_reserved":0,"allocated":1789952,"data_stored":1108749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.52400000000000002,"15min":0.627},"min":{"1min":0.26100000000000001,"5min":0.245,"15min":0.23699999999999999},"max":{"1min":0.93500000000000005,"5min":7.907,"15min":19.843},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58199999999999996,"15min":0.67400000000000004},"min":{"1min":0.35599999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.8240000000000001,"5min":3.512,"15min":10.154},"last":0.33100000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.71599999999999997,"15min":0.80100000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.30399999999999999},"max":{"1min":1.4410000000000001,"5min":7.0810000000000004,"15min":25.905000000000001},"last":0.379},{"interface":"front","average":{"1min":0.498,"5min":0.57499999999999996,"15min":0.61399999999999999},"min":{"1min":0.32700000000000001,"5min":0.27800000000000002,"15min":0.26700000000000002},"max":{"1min":0.79300000000000004,"5min":3.8500000000000001,"15min":9.3829999999999991},"last":0.39600000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.69399999999999995,"15min":0.74199999999999999},"min":{"1min":0.34699999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":2.3519999999999999,"5min":6.2850000000000001,"15min":9.1780000000000008},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.70199999999999996,"15min":0.76200000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.36899999999999999},"max":{"1min":1.327,"5min":6.5979999999999999,"15min":10.210000000000001},"last":0.44400000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.84199999999999997,"15min":0.98399999999999999},"min":{"1min":0.47699999999999998,"5min":0.39100000000000001,"15min":0.38},"max":{"1min":1.2929999999999999,"5min":4.5179999999999998,"15min":22.814},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.83399999999999996,"15min":1.0289999999999999},"min":{"1min":0.502,"5min":0.502,"15min":0.41199999999999998},"max":{"1min":1.2749999999999999,"5min":4.2329999999999997,"15min":23.341999999999999},"last":0.57999999999999996}]},{"osd":5,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.86399999999999999,"15min":0.96999999999999997},"min":{"1min":0.54400000000000004,"5min":0.51600000000000001,"15min":0.441},"max":{"1min":2.6190000000000002,"5min":4.008,"15min":12.428000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.79900000000000004,"5min":1.252,"15min":1.258},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.3120000000000001,"5min":23.021000000000001,"15min":23.021000000000001},"last":0.66100000000000003}]},{"osd":6,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.92000000000000004,"15min":1.212},"min":{"1min":0.53500000000000003,"5min":0.48399999999999999,"15min":0.46200000000000002},"max":{"1min":1.6359999999999999,"5min":6.5250000000000004,"15min":31.059000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.94799999999999995,"15min":1.089},"min":{"1min":0.59399999999999997,"5min":0.52700000000000002,"15min":0.49099999999999999},"max":{"1min":2.5659999999999998,"5min":6.9690000000000003,"15min":40.731000000000002},"last":0.623}]},{"osd":7,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":1.107,"15min":1.2},"min":{"1min":0.59499999999999997,"5min":0.499,"15min":0.42499999999999999},"max":{"1min":2.6539999999999999,"5min":10.497999999999999,"15min":24.492999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.88300000000000001,"15min":1.103},"min":{"1min":0.55300000000000005,"5min":0.495,"15min":0.46600000000000003},"max":{"1min":1.3740000000000001,"5min":4.6600000000000001,"15min":23.524000000000001},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:17:21.626 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:17:21.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.810+0000 7efc9ba00640 1 Processor -- start 2024-01-15T18:17:21.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.810+0000 7efc9ba00640 1 -- start start 2024-01-15T18:17:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.810+0000 7efc9ba00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7efc9406db80 0x7efc9406df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.810+0000 7efc9ba00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efc940cdd10 0x7efc94070c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.810+0000 7efc9ba00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc940ce920 con 0x7efc9415e730 2024-01-15T18:17:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.810+0000 7efc9ba00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc940cdb30 con 0x7efc940cdd10 2024-01-15T18:17:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.810+0000 7efc9ba00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc9406e4a0 con 0x7efc9406db80 2024-01-15T18:17:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.810+0000 7efc99a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efc940cdd10 0x7efc94070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.810+0000 7efc99a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efc940cdd10 0x7efc94070c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59704/0 (socket says 172.21.15.27:59704) 2024-01-15T18:17:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.810+0000 7efc99a00640 1 -- 172.21.15.27:0/2714658728 learned_addr learned my addr 172.21.15.27:0/2714658728 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.810+0000 7efc99000640 1 --2- 172.21.15.27:0/2714658728 >> v2:172.21.15.93:3300/0 conn(0x7efc9406db80 0x7efc9406df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:21.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.810+0000 7efc99a00640 1 -- 172.21.15.27:0/2714658728 >> v2:172.21.15.93:3300/0 conn(0x7efc9406db80 msgr2=0x7efc9406df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:21.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.810+0000 7efc99a00640 1 --2- 172.21.15.27:0/2714658728 >> v2:172.21.15.93:3300/0 conn(0x7efc9406db80 0x7efc9406df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.810+0000 7efc99a00640 1 -- 172.21.15.27:0/2714658728 >> v1:172.21.15.93:6789/0 conn(0x7efc9415e730 legacy=0x7efc940cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:21.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.810+0000 7efc99a00640 1 -- 172.21.15.27:0/2714658728 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc940ce640 con 0x7efc940cdd10 2024-01-15T18:17:21.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.814+0000 7efc99a00640 1 --2- 172.21.15.27:0/2714658728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc940cdd10 0x7efc94070c20 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7efc9406be30 tx=0x7efc8c01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36e848d53c6b09d6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:21.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.814+0000 7efc93400640 1 -- 172.21.15.27:0/2714658728 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8c0a5830 con 0x7efc940cdd10 2024-01-15T18:17:21.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.814+0000 7efc93400640 1 -- 172.21.15.27:0/2714658728 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc8c0a5990 con 0x7efc940cdd10 2024-01-15T18:17:21.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.814+0000 7efc93400640 1 -- 172.21.15.27:0/2714658728 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8c0a5d20 con 0x7efc940cdd10 2024-01-15T18:17:21.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.814+0000 7efc9ba00640 1 -- 172.21.15.27:0/2714658728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc940cdd10 msgr2=0x7efc94070c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:21.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.814+0000 7efc9ba00640 1 --2- 172.21.15.27:0/2714658728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc940cdd10 0x7efc94070c20 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7efc9406be30 tx=0x7efc8c01c9a0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.814+0000 7efc9ba00640 1 -- 172.21.15.27:0/2714658728 shutdown_connections 2024-01-15T18:17:21.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.814+0000 7efc9ba00640 1 --2- 172.21.15.27:0/2714658728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc940cdd10 0x7efc94070c20 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.814+0000 7efc9ba00640 1 --2- 172.21.15.27:0/2714658728 >> v2:172.21.15.93:3300/0 conn(0x7efc9406db80 0x7efc9406df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.814+0000 7efc9ba00640 1 -- 172.21.15.27:0/2714658728 >> 172.21.15.27:0/2714658728 conn(0x7efc940cec60 msgr2=0x7efc940cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:21.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.814+0000 7efc9ba00640 1 -- 172.21.15.27:0/2714658728 shutdown_connections 2024-01-15T18:17:21.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.814+0000 7efc9ba00640 1 -- 172.21.15.27:0/2714658728 wait complete. 2024-01-15T18:17:21.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.818+0000 7efc9ba00640 1 Processor -- start 2024-01-15T18:17:21.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.818+0000 7efc9ba00640 1 -- start start 2024-01-15T18:17:21.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.818+0000 7efc9ba00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efc9406db80 0x7efc941671a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:21.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.818+0000 7efc9ba00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc940cdd10 0x7efc94159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:21.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.818+0000 7efc9ba00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efc9415e730 0x7efc94156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:21.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.818+0000 7efc9ba00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc940c8ca0 con 0x7efc940cdd10 2024-01-15T18:17:21.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.818+0000 7efc9ba00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc940c8120 con 0x7efc9406db80 2024-01-15T18:17:21.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.818+0000 7efc9ba00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc940c8400 con 0x7efc9415e730 2024-01-15T18:17:21.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.818+0000 7efc93e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc940cdd10 0x7efc94159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:21.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.818+0000 7efc93e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc940cdd10 0x7efc94159250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59714/0 (socket says 172.21.15.27:59714) 2024-01-15T18:17:21.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.818+0000 7efc93e00640 1 -- 172.21.15.27:0/1664635894 learned_addr learned my addr 172.21.15.27:0/1664635894 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:21.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.818+0000 7efc99a00640 1 --2- 172.21.15.27:0/1664635894 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efc9415e730 0x7efc94156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:21.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.818+0000 7efc99000640 1 --2- 172.21.15.27:0/1664635894 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efc9406db80 0x7efc941671a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:21.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.818+0000 7efc93e00640 1 -- 172.21.15.27:0/1664635894 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efc9415e730 msgr2=0x7efc94156ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:21.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.818+0000 7efc93e00640 1 --2- 172.21.15.27:0/1664635894 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efc9415e730 0x7efc94156ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.818+0000 7efc93e00640 1 -- 172.21.15.27:0/1664635894 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efc9406db80 msgr2=0x7efc941671a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:21.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.818+0000 7efc93e00640 1 --2- 172.21.15.27:0/1664635894 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efc9406db80 0x7efc941671a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.818+0000 7efc93e00640 1 -- 172.21.15.27:0/1664635894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8c009b30 con 0x7efc940cdd10 2024-01-15T18:17:21.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.818+0000 7efc93e00640 1 --2- 172.21.15.27:0/1664635894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc940cdd10 0x7efc94159250 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7efc7c00c3c0 tx=0x7efc7c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:21.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.818+0000 7efc90c00640 1 -- 172.21.15.27:0/1664635894 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7c0052d0 con 0x7efc940cdd10 2024-01-15T18:17:21.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.818+0000 7efc90c00640 1 -- 172.21.15.27:0/1664635894 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc7c005430 con 0x7efc940cdd10 2024-01-15T18:17:21.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.818+0000 7efc90c00640 1 -- 172.21.15.27:0/1664635894 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7c003840 con 0x7efc940cdd10 2024-01-15T18:17:21.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.818+0000 7efc9ba00640 1 -- 172.21.15.27:0/1664635894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc940c86e0 con 0x7efc940cdd10 2024-01-15T18:17:21.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.818+0000 7efc9ba00640 1 -- 172.21.15.27:0/1664635894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc941676e0 con 0x7efc940cdd10 2024-01-15T18:17:21.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.822+0000 7efc9ba00640 1 -- 172.21.15.27:0/1664635894 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc94165230 con 0x7efc940cdd10 2024-01-15T18:17:21.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.822+0000 7efc90c00640 1 -- 172.21.15.27:0/1664635894 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7efc7c0039a0 con 0x7efc940cdd10 2024-01-15T18:17:21.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.822+0000 7efc90c00640 1 --2- 172.21.15.27:0/1664635894 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efc6c067700 0x7efc6c069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:21.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.822+0000 7efc90c00640 1 -- 172.21.15.27:0/1664635894 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7efc7c082630 con 0x7efc940cdd10 2024-01-15T18:17:21.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.822+0000 7efc99000640 1 --2- 172.21.15.27:0/1664635894 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efc6c067700 0x7efc6c069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:21.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.826+0000 7efc99000640 1 --2- 172.21.15.27:0/1664635894 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efc6c067700 0x7efc6c069bc0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7efc84009b60 tx=0x7efc84009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:21.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.826+0000 7efc90c00640 1 -- 172.21.15.27:0/1664635894 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7efc94165230 con 0x7efc940cdd10 2024-01-15T18:17:21.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.986+0000 7efc9ba00640 1 -- 172.21.15.27:0/1664635894 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efc94165230 con 0x7efc6c067700 2024-01-15T18:17:21.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.994+0000 7efc90c00640 1 -- 172.21.15.27:0/1664635894 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+281032 (secure 0 0 0) 0x7efc94165230 con 0x7efc6c067700 2024-01-15T18:17:21.995 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:17:21.997 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:17:21.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.994+0000 7efc9ba00640 1 -- 172.21.15.27:0/1664635894 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efc6c067700 msgr2=0x7efc6c069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:21.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.994+0000 7efc9ba00640 1 --2- 172.21.15.27:0/1664635894 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efc6c067700 0x7efc6c069bc0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7efc84009b60 tx=0x7efc84009190 comp rx=0 tx=0).stop 2024-01-15T18:17:21.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.994+0000 7efc9ba00640 1 -- 172.21.15.27:0/1664635894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc940cdd10 msgr2=0x7efc94159250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:21.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.994+0000 7efc9ba00640 1 --2- 172.21.15.27:0/1664635894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc940cdd10 0x7efc94159250 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7efc7c00c3c0 tx=0x7efc7c00c880 comp rx=0 tx=0).stop 2024-01-15T18:17:21.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.998+0000 7efc9ba00640 1 -- 172.21.15.27:0/1664635894 shutdown_connections 2024-01-15T18:17:21.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.998+0000 7efc9ba00640 1 --2- 172.21.15.27:0/1664635894 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efc6c067700 0x7efc6c069bc0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.998+0000 7efc9ba00640 1 --2- 172.21.15.27:0/1664635894 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efc9415e730 0x7efc94156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.998+0000 7efc9ba00640 1 --2- 172.21.15.27:0/1664635894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc940cdd10 0x7efc94159250 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.998+0000 7efc9ba00640 1 --2- 172.21.15.27:0/1664635894 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efc9406db80 0x7efc941671a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:21.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.998+0000 7efc9ba00640 1 -- 172.21.15.27:0/1664635894 >> 172.21.15.27:0/1664635894 conn(0x7efc940cec60 msgr2=0x7efc9406d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:21.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.998+0000 7efc9ba00640 1 -- 172.21.15.27:0/1664635894 shutdown_connections 2024-01-15T18:17:21.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:21.998+0000 7efc9ba00640 1 -- 172.21.15.27:0/1664635894 wait complete. 2024-01-15T18:17:22.016 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-01-15T18:17:20.354630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592562,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":182,"num_write_kb":2420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138,"ondisk_log_size":138,"up":104,"acting":104,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":71,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":542496,"kb_used_data":10272,"kb_used_omap":12,"kb_used_meta":532083,"kb_avail":754432224,"statfs":{"total":773094113280,"available":772538597376,"internally_reserved":0,"allocated":10518528,"data_stored":5203370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":544853552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2194,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.265190"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440653+0000","last_change":"2024-01-15T18:17:17.132525+0000","last_active":"2024-01-15T18:17:18.440653+0000","last_peered":"2024-01-15T18:17:18.440653+0000","last_clean":"2024-01-15T18:17:18.440653+0000","last_became_active":"2024-01-15T18:17:17.132261+0000","last_became_peered":"2024-01-15T18:17:17.132261+0000","last_unstale":"2024-01-15T18:17:18.440653+0000","last_undegraded":"2024-01-15T18:17:18.440653+0000","last_fullsized":"2024-01-15T18:17:18.440653+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:07.042399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439412+0000","last_change":"2024-01-15T18:17:17.129139+0000","last_active":"2024-01-15T18:17:18.439412+0000","last_peered":"2024-01-15T18:17:18.439412+0000","last_clean":"2024-01-15T18:17:18.439412+0000","last_became_active":"2024-01-15T18:17:17.126922+0000","last_became_peered":"2024-01-15T18:17:17.126922+0000","last_unstale":"2024-01-15T18:17:18.439412+0000","last_undegraded":"2024-01-15T18:17:18.439412+0000","last_fullsized":"2024-01-15T18:17:18.439412+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:40:32.654203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.132988+0000","last_change":"2024-01-15T18:17:17.117455+0000","last_active":"2024-01-15T18:17:18.132988+0000","last_peered":"2024-01-15T18:17:18.132988+0000","last_clean":"2024-01-15T18:17:18.132988+0000","last_became_active":"2024-01-15T18:17:17.117259+0000","last_became_peered":"2024-01-15T18:17:17.117259+0000","last_unstale":"2024-01-15T18:17:18.132988+0000","last_undegraded":"2024-01-15T18:17:18.132988+0000","last_fullsized":"2024-01-15T18:17:18.132988+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:23.944086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.132977+0000","last_change":"2024-01-15T18:17:17.126892+0000","last_active":"2024-01-15T18:17:18.132977+0000","last_peered":"2024-01-15T18:17:18.132977+0000","last_clean":"2024-01-15T18:17:18.132977+0000","last_became_active":"2024-01-15T18:17:17.125684+0000","last_became_peered":"2024-01-15T18:17:17.125684+0000","last_unstale":"2024-01-15T18:17:18.132977+0000","last_undegraded":"2024-01-15T18:17:18.132977+0000","last_fullsized":"2024-01-15T18:17:18.132977+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:54.730338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440809+0000","last_change":"2024-01-15T18:17:17.125211+0000","last_active":"2024-01-15T18:17:18.440809+0000","last_peered":"2024-01-15T18:17:18.440809+0000","last_clean":"2024-01-15T18:17:18.440809+0000","last_became_active":"2024-01-15T18:17:17.119327+0000","last_became_peered":"2024-01-15T18:17:17.119327+0000","last_unstale":"2024-01-15T18:17:18.440809+0000","last_undegraded":"2024-01-15T18:17:18.440809+0000","last_fullsized":"2024-01-15T18:17:18.440809+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:57:35.828970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.133012+0000","last_change":"2024-01-15T18:17:17.117786+0000","last_active":"2024-01-15T18:17:18.133012+0000","last_peered":"2024-01-15T18:17:18.133012+0000","last_clean":"2024-01-15T18:17:18.133012+0000","last_became_active":"2024-01-15T18:17:17.117505+0000","last_became_peered":"2024-01-15T18:17:17.117505+0000","last_unstale":"2024-01-15T18:17:18.133012+0000","last_undegraded":"2024-01-15T18:17:18.133012+0000","last_fullsized":"2024-01-15T18:17:18.133012+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:39:09.922990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440211+0000","last_change":"2024-01-15T18:17:17.125005+0000","last_active":"2024-01-15T18:17:18.440211+0000","last_peered":"2024-01-15T18:17:18.440211+0000","last_clean":"2024-01-15T18:17:18.440211+0000","last_became_active":"2024-01-15T18:17:17.124717+0000","last_became_peered":"2024-01-15T18:17:17.124717+0000","last_unstale":"2024-01-15T18:17:18.440211+0000","last_undegraded":"2024-01-15T18:17:18.440211+0000","last_fullsized":"2024-01-15T18:17:18.440211+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:48:30.181901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440100+0000","last_change":"2024-01-15T18:17:17.130054+0000","last_active":"2024-01-15T18:17:18.440100+0000","last_peered":"2024-01-15T18:17:18.440100+0000","last_clean":"2024-01-15T18:17:18.440100+0000","last_became_active":"2024-01-15T18:17:17.129779+0000","last_became_peered":"2024-01-15T18:17:17.129779+0000","last_unstale":"2024-01-15T18:17:18.440100+0000","last_undegraded":"2024-01-15T18:17:18.440100+0000","last_fullsized":"2024-01-15T18:17:18.440100+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:42:04.787652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440698+0000","last_change":"2024-01-15T18:17:17.132156+0000","last_active":"2024-01-15T18:17:18.440698+0000","last_peered":"2024-01-15T18:17:18.440698+0000","last_clean":"2024-01-15T18:17:18.440698+0000","last_became_active":"2024-01-15T18:17:17.131822+0000","last_became_peered":"2024-01-15T18:17:17.131822+0000","last_unstale":"2024-01-15T18:17:18.440698+0000","last_undegraded":"2024-01-15T18:17:18.440698+0000","last_fullsized":"2024-01-15T18:17:18.440698+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:33:59.696616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440705+0000","last_change":"2024-01-15T18:17:17.127796+0000","last_active":"2024-01-15T18:17:18.440705+0000","last_peered":"2024-01-15T18:17:18.440705+0000","last_clean":"2024-01-15T18:17:18.440705+0000","last_became_active":"2024-01-15T18:17:17.127390+0000","last_became_peered":"2024-01-15T18:17:17.127390+0000","last_unstale":"2024-01-15T18:17:18.440705+0000","last_undegraded":"2024-01-15T18:17:18.440705+0000","last_fullsized":"2024-01-15T18:17:18.440705+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:03:11.357766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1f","version":"96'4","reported_seq":16,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440784+0000","last_change":"2024-01-15T18:17:16.117003+0000","last_active":"2024-01-15T18:17:18.440784+0000","last_peered":"2024-01-15T18:17:18.440784+0000","last_clean":"2024-01-15T18:17:18.440784+0000","last_became_active":"2024-01-15T18:17:16.116625+0000","last_became_peered":"2024-01-15T18:17:16.116625+0000","last_unstale":"2024-01-15T18:17:18.440784+0000","last_undegraded":"2024-01-15T18:17:18.440784+0000","last_fullsized":"2024-01-15T18:17:18.440784+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:11:58.418627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.096546+0000","last_change":"2024-01-15T18:17:15.096546+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.096546+0000","last_undegraded":"2024-01-15T18:17:15.096546+0000","last_fullsized":"2024-01-15T18:17:15.096546+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1d","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440270+0000","last_change":"2024-01-15T18:17:16.115473+0000","last_active":"2024-01-15T18:17:18.440270+0000","last_peered":"2024-01-15T18:17:18.440270+0000","last_clean":"2024-01-15T18:17:18.440270+0000","last_became_active":"2024-01-15T18:17:16.115189+0000","last_became_peered":"2024-01-15T18:17:16.115189+0000","last_unstale":"2024-01-15T18:17:18.440270+0000","last_undegraded":"2024-01-15T18:17:18.440270+0000","last_fullsized":"2024-01-15T18:17:18.440270+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:58:47.776409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.133184+0000","last_change":"2024-01-15T18:17:17.127577+0000","last_active":"2024-01-15T18:17:18.133184+0000","last_peered":"2024-01-15T18:17:18.133184+0000","last_clean":"2024-01-15T18:17:18.133184+0000","last_became_active":"2024-01-15T18:17:17.125488+0000","last_became_peered":"2024-01-15T18:17:17.125488+0000","last_unstale":"2024-01-15T18:17:18.133184+0000","last_undegraded":"2024-01-15T18:17:18.133184+0000","last_fullsized":"2024-01-15T18:17:18.133184+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:23.103323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440326+0000","last_change":"2024-01-15T18:17:16.123895+0000","last_active":"2024-01-15T18:17:18.440326+0000","last_peered":"2024-01-15T18:17:18.440326+0000","last_clean":"2024-01-15T18:17:18.440326+0000","last_became_active":"2024-01-15T18:17:16.123325+0000","last_became_peered":"2024-01-15T18:17:16.123325+0000","last_unstale":"2024-01-15T18:17:18.440326+0000","last_undegraded":"2024-01-15T18:17:18.440326+0000","last_fullsized":"2024-01-15T18:17:18.440326+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:13:09.312116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.131438+0000","last_change":"2024-01-15T18:17:17.118290+0000","last_active":"2024-01-15T18:17:18.131438+0000","last_peered":"2024-01-15T18:17:18.131438+0000","last_clean":"2024-01-15T18:17:18.131438+0000","last_became_active":"2024-01-15T18:17:17.117943+0000","last_became_peered":"2024-01-15T18:17:17.117943+0000","last_unstale":"2024-01-15T18:17:18.131438+0000","last_undegraded":"2024-01-15T18:17:18.131438+0000","last_fullsized":"2024-01-15T18:17:18.131438+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:50:04.755553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.093523+0000","last_change":"2024-01-15T18:17:15.093523+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.093523+0000","last_undegraded":"2024-01-15T18:17:15.093523+0000","last_fullsized":"2024-01-15T18:17:15.093523+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1a","version":"96'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440314+0000","last_change":"2024-01-15T18:17:16.121483+0000","last_active":"2024-01-15T18:17:18.440314+0000","last_peered":"2024-01-15T18:17:18.440314+0000","last_clean":"2024-01-15T18:17:18.440314+0000","last_became_active":"2024-01-15T18:17:16.121199+0000","last_became_peered":"2024-01-15T18:17:16.121199+0000","last_unstale":"2024-01-15T18:17:18.440314+0000","last_undegraded":"2024-01-15T18:17:18.440314+0000","last_fullsized":"2024-01-15T18:17:18.440314+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:51:29.086954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.095582+0000","last_change":"2024-01-15T18:17:15.095582+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.095582+0000","last_undegraded":"2024-01-15T18:17:15.095582+0000","last_fullsized":"2024-01-15T18:17:15.095582+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.132487+0000","last_change":"2024-01-15T18:17:17.119962+0000","last_active":"2024-01-15T18:17:18.132487+0000","last_peered":"2024-01-15T18:17:18.132487+0000","last_clean":"2024-01-15T18:17:18.132487+0000","last_became_active":"2024-01-15T18:17:17.118612+0000","last_became_peered":"2024-01-15T18:17:17.118612+0000","last_unstale":"2024-01-15T18:17:18.132487+0000","last_undegraded":"2024-01-15T18:17:18.132487+0000","last_fullsized":"2024-01-15T18:17:18.132487+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:45:26.435028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.093925+0000","last_change":"2024-01-15T18:17:15.093925+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.093925+0000","last_undegraded":"2024-01-15T18:17:15.093925+0000","last_fullsized":"2024-01-15T18:17:15.093925+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440790+0000","last_change":"2024-01-15T18:17:17.124377+0000","last_active":"2024-01-15T18:17:18.440790+0000","last_peered":"2024-01-15T18:17:18.440790+0000","last_clean":"2024-01-15T18:17:18.440790+0000","last_became_active":"2024-01-15T18:17:17.124096+0000","last_became_peered":"2024-01-15T18:17:17.124096+0000","last_unstale":"2024-01-15T18:17:18.440790+0000","last_undegraded":"2024-01-15T18:17:18.440790+0000","last_fullsized":"2024-01-15T18:17:18.440790+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:56:21.720071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.093475+0000","last_change":"2024-01-15T18:17:15.093475+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.093475+0000","last_undegraded":"2024-01-15T18:17:15.093475+0000","last_fullsized":"2024-01-15T18:17:15.093475+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.133177+0000","last_change":"2024-01-15T18:17:17.122838+0000","last_active":"2024-01-15T18:17:18.133177+0000","last_peered":"2024-01-15T18:17:18.133177+0000","last_clean":"2024-01-15T18:17:18.133177+0000","last_became_active":"2024-01-15T18:17:17.122579+0000","last_became_peered":"2024-01-15T18:17:17.122579+0000","last_unstale":"2024-01-15T18:17:18.133177+0000","last_undegraded":"2024-01-15T18:17:18.133177+0000","last_fullsized":"2024-01-15T18:17:18.133177+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:13:53.357957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.095406+0000","last_change":"2024-01-15T18:17:15.095406+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.095406+0000","last_undegraded":"2024-01-15T18:17:15.095406+0000","last_fullsized":"2024-01-15T18:17:15.095406+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.096252+0000","last_change":"2024-01-15T18:17:15.096252+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.096252+0000","last_undegraded":"2024-01-15T18:17:15.096252+0000","last_fullsized":"2024-01-15T18:17:15.096252+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.14","version":"96'6","reported_seq":18,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.148952+0000","last_change":"2024-01-15T18:17:16.120030+0000","last_active":"2024-01-15T18:17:17.148952+0000","last_peered":"2024-01-15T18:17:17.148952+0000","last_clean":"2024-01-15T18:17:17.148952+0000","last_became_active":"2024-01-15T18:17:16.119705+0000","last_became_peered":"2024-01-15T18:17:16.119705+0000","last_unstale":"2024-01-15T18:17:17.148952+0000","last_undegraded":"2024-01-15T18:17:17.148952+0000","last_fullsized":"2024-01-15T18:17:17.148952+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:47:48.944557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.094637+0000","last_change":"2024-01-15T18:17:15.094637+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.094637+0000","last_undegraded":"2024-01-15T18:17:15.094637+0000","last_fullsized":"2024-01-15T18:17:15.094637+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.094404+0000","last_change":"2024-01-15T18:17:15.094404+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.094404+0000","last_undegraded":"2024-01-15T18:17:15.094404+0000","last_fullsized":"2024-01-15T18:17:15.094404+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.132434+0000","last_change":"2024-01-15T18:17:17.121456+0000","last_active":"2024-01-15T18:17:18.132434+0000","last_peered":"2024-01-15T18:17:18.132434+0000","last_clean":"2024-01-15T18:17:18.132434+0000","last_became_active":"2024-01-15T18:17:17.121179+0000","last_became_peered":"2024-01-15T18:17:17.121179+0000","last_unstale":"2024-01-15T18:17:18.132434+0000","last_undegraded":"2024-01-15T18:17:18.132434+0000","last_fullsized":"2024-01-15T18:17:18.132434+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:53:26.915291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.111628+0000","last_change":"2024-01-15T18:17:16.113394+0000","last_active":"2024-01-15T18:17:17.111628+0000","last_peered":"2024-01-15T18:17:17.111628+0000","last_clean":"2024-01-15T18:17:17.111628+0000","last_became_active":"2024-01-15T18:17:16.113105+0000","last_became_peered":"2024-01-15T18:17:16.113105+0000","last_unstale":"2024-01-15T18:17:17.111628+0000","last_undegraded":"2024-01-15T18:17:17.111628+0000","last_fullsized":"2024-01-15T18:17:17.111628+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:41:23.459232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.093447+0000","last_change":"2024-01-15T18:17:15.093447+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.093447+0000","last_undegraded":"2024-01-15T18:17:15.093447+0000","last_fullsized":"2024-01-15T18:17:15.093447+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.e","version":"96'2","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.143716+0000","last_change":"2024-01-15T18:17:16.108046+0000","last_active":"2024-01-15T18:17:17.143716+0000","last_peered":"2024-01-15T18:17:17.143716+0000","last_clean":"2024-01-15T18:17:17.143716+0000","last_became_active":"2024-01-15T18:17:16.107750+0000","last_became_peered":"2024-01-15T18:17:16.107750+0000","last_unstale":"2024-01-15T18:17:17.143716+0000","last_undegraded":"2024-01-15T18:17:17.143716+0000","last_fullsized":"2024-01-15T18:17:17.143716+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:14.320865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.131540+0000","last_change":"2024-01-15T18:17:17.123658+0000","last_active":"2024-01-15T18:17:18.131540+0000","last_peered":"2024-01-15T18:17:18.131540+0000","last_clean":"2024-01-15T18:17:18.131540+0000","last_became_active":"2024-01-15T18:17:17.123351+0000","last_became_peered":"2024-01-15T18:17:17.123351+0000","last_unstale":"2024-01-15T18:17:18.131540+0000","last_undegraded":"2024-01-15T18:17:18.131540+0000","last_fullsized":"2024-01-15T18:17:18.131540+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:38:11.462593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.095378+0000","last_change":"2024-01-15T18:17:15.095378+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.095378+0000","last_undegraded":"2024-01-15T18:17:15.095378+0000","last_fullsized":"2024-01-15T18:17:15.095378+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440407+0000","last_change":"2024-01-15T18:17:16.125410+0000","last_active":"2024-01-15T18:17:18.440407+0000","last_peered":"2024-01-15T18:17:18.440407+0000","last_clean":"2024-01-15T18:17:18.440407+0000","last_became_active":"2024-01-15T18:17:16.124440+0000","last_became_peered":"2024-01-15T18:17:16.124440+0000","last_unstale":"2024-01-15T18:17:18.440407+0000","last_undegraded":"2024-01-15T18:17:18.440407+0000","last_fullsized":"2024-01-15T18:17:18.440407+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:37:51.051323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.b","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440507+0000","last_change":"2024-01-15T18:17:16.120398+0000","last_active":"2024-01-15T18:17:18.440507+0000","last_peered":"2024-01-15T18:17:18.440507+0000","last_clean":"2024-01-15T18:17:18.440507+0000","last_became_active":"2024-01-15T18:17:16.119944+0000","last_became_peered":"2024-01-15T18:17:16.119944+0000","last_unstale":"2024-01-15T18:17:18.440507+0000","last_undegraded":"2024-01-15T18:17:18.440507+0000","last_fullsized":"2024-01-15T18:17:18.440507+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:54:02.313160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.131461+0000","last_change":"2024-01-15T18:17:16.102303+0000","last_active":"2024-01-15T18:17:18.131461+0000","last_peered":"2024-01-15T18:17:18.131461+0000","last_clean":"2024-01-15T18:17:18.131461+0000","last_became_active":"2024-01-15T18:17:16.099773+0000","last_became_peered":"2024-01-15T18:17:16.099773+0000","last_unstale":"2024-01-15T18:17:18.131461+0000","last_undegraded":"2024-01-15T18:17:18.131461+0000","last_fullsized":"2024-01-15T18:17:18.131461+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:40:51.210210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.131385+0000","last_change":"2024-01-15T18:17:16.119288+0000","last_active":"2024-01-15T18:17:18.131385+0000","last_peered":"2024-01-15T18:17:18.131385+0000","last_clean":"2024-01-15T18:17:18.131385+0000","last_became_active":"2024-01-15T18:17:16.118972+0000","last_became_peered":"2024-01-15T18:17:16.118972+0000","last_unstale":"2024-01-15T18:17:18.131385+0000","last_undegraded":"2024-01-15T18:17:18.131385+0000","last_fullsized":"2024-01-15T18:17:18.131385+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:14:29.755549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439494+0000","last_change":"2024-01-15T18:17:16.127573+0000","last_active":"2024-01-15T18:17:18.439494+0000","last_peered":"2024-01-15T18:17:18.439494+0000","last_clean":"2024-01-15T18:17:18.439494+0000","last_became_active":"2024-01-15T18:17:16.127194+0000","last_became_peered":"2024-01-15T18:17:16.127194+0000","last_unstale":"2024-01-15T18:17:18.439494+0000","last_undegraded":"2024-01-15T18:17:18.439494+0000","last_fullsized":"2024-01-15T18:17:18.439494+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:48:07.276946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440832+0000","last_change":"2024-01-15T18:17:17.132695+0000","last_active":"2024-01-15T18:17:18.440832+0000","last_peered":"2024-01-15T18:17:18.440832+0000","last_clean":"2024-01-15T18:17:18.440832+0000","last_became_active":"2024-01-15T18:17:17.132430+0000","last_became_peered":"2024-01-15T18:17:17.132430+0000","last_unstale":"2024-01-15T18:17:18.440832+0000","last_undegraded":"2024-01-15T18:17:18.440832+0000","last_fullsized":"2024-01-15T18:17:18.440832+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:14:02.211346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.132130+0000","last_change":"2024-01-15T18:17:16.102078+0000","last_active":"2024-01-15T18:17:18.132130+0000","last_peered":"2024-01-15T18:17:18.132130+0000","last_clean":"2024-01-15T18:17:18.132130+0000","last_became_active":"2024-01-15T18:17:16.101772+0000","last_became_peered":"2024-01-15T18:17:16.101772+0000","last_unstale":"2024-01-15T18:17:18.132130+0000","last_undegraded":"2024-01-15T18:17:18.132130+0000","last_fullsized":"2024-01-15T18:17:18.132130+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:28:05.931436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440855+0000","last_change":"2024-01-15T18:17:16.121035+0000","last_active":"2024-01-15T18:17:18.440855+0000","last_peered":"2024-01-15T18:17:18.440855+0000","last_clean":"2024-01-15T18:17:18.440855+0000","last_became_active":"2024-01-15T18:17:16.120714+0000","last_became_peered":"2024-01-15T18:17:16.120714+0000","last_unstale":"2024-01-15T18:17:18.440855+0000","last_undegraded":"2024-01-15T18:17:18.440855+0000","last_fullsized":"2024-01-15T18:17:18.440855+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:30:03.158862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440319+0000","last_change":"2024-01-15T18:17:16.114051+0000","last_active":"2024-01-15T18:17:18.440319+0000","last_peered":"2024-01-15T18:17:18.440319+0000","last_clean":"2024-01-15T18:17:18.440319+0000","last_became_active":"2024-01-15T18:17:16.113079+0000","last_became_peered":"2024-01-15T18:17:16.113079+0000","last_unstale":"2024-01-15T18:17:18.440319+0000","last_undegraded":"2024-01-15T18:17:18.440319+0000","last_fullsized":"2024-01-15T18:17:18.440319+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:25:22.584736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.095295+0000","last_change":"2024-01-15T18:17:15.095295+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.095295+0000","last_undegraded":"2024-01-15T18:17:15.095295+0000","last_fullsized":"2024-01-15T18:17:15.095295+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440562+0000","last_change":"2024-01-15T18:17:16.121786+0000","last_active":"2024-01-15T18:17:18.440562+0000","last_peered":"2024-01-15T18:17:18.440562+0000","last_clean":"2024-01-15T18:17:18.440562+0000","last_became_active":"2024-01-15T18:17:16.121508+0000","last_became_peered":"2024-01-15T18:17:16.121508+0000","last_unstale":"2024-01-15T18:17:18.440562+0000","last_undegraded":"2024-01-15T18:17:18.440562+0000","last_fullsized":"2024-01-15T18:17:18.440562+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:51:43.523481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.133059+0000","last_change":"2024-01-15T18:17:17.126022+0000","last_active":"2024-01-15T18:17:18.133059+0000","last_peered":"2024-01-15T18:17:18.133059+0000","last_clean":"2024-01-15T18:17:18.133059+0000","last_became_active":"2024-01-15T18:17:17.125366+0000","last_became_peered":"2024-01-15T18:17:17.125366+0000","last_unstale":"2024-01-15T18:17:18.133059+0000","last_undegraded":"2024-01-15T18:17:18.133059+0000","last_fullsized":"2024-01-15T18:17:18.133059+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:27:59.524326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.095318+0000","last_change":"2024-01-15T18:17:15.095318+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.095318+0000","last_undegraded":"2024-01-15T18:17:15.095318+0000","last_fullsized":"2024-01-15T18:17:15.095318+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.095327+0000","last_change":"2024-01-15T18:17:15.095327+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.095327+0000","last_undegraded":"2024-01-15T18:17:15.095327+0000","last_fullsized":"2024-01-15T18:17:15.095327+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.094409+0000","last_change":"2024-01-15T18:17:15.094409+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.094409+0000","last_undegraded":"2024-01-15T18:17:15.094409+0000","last_fullsized":"2024-01-15T18:17:15.094409+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:17:16.086114+0000","last_change":"2024-01-15T18:17:16.086114+0000","last_active":"2024-01-15T18:17:16.086114+0000","last_peered":"2024-01-15T18:17:16.086114+0000","last_clean":"2024-01-15T18:17:16.086114+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:16.086114+0000","last_undegraded":"2024-01-15T18:17:16.086114+0000","last_fullsized":"2024-01-15T18:17:16.086114+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":94,"state":"creating+peering","last_fresh":"2024-01-15T18:17:15.096380+0000","last_change":"2024-01-15T18:17:15.096380+0000","last_active":"2024-01-15T18:17:15.077857+0000","last_peered":"2024-01-15T18:17:15.077857+0000","last_clean":"2024-01-15T18:17:15.077857+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:17:15.096380+0000","last_undegraded":"2024-01-15T18:17:15.096380+0000","last_fullsized":"2024-01-15T18:17:15.096380+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":227,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-01-15T18:17:13.089307+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:17:13.089307+0000","last_peered":"2024-01-15T18:17:13.089307+0000","last_clean":"2024-01-15T18:17:13.089307+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:17:13.089307+0000","last_undegraded":"2024-01-15T18:17:13.089307+0000","last_fullsized":"2024-01-15T18:17:13.089307+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":38,"acting":38,"num_store_stats":3},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2194,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21,"ondisk_log_size":21,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509593,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64748,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307092,"statfs":{"total":96636764160,"available":96570462208,"internally_reserved":0,"allocated":1736704,"data_stored":1082552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64551370},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.87,"15min":0.871},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.90800000000000003,"5min":9.3040000000000003,"15min":10.284000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.83899999999999997,"15min":0.91600000000000004},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.378},"max":{"1min":6.6189999999999998,"5min":6.6189999999999998,"15min":16.968},"last":0.68100000000000005}]},{"osd":1,"last update":"Mon Jan 15 18:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.93200000000000005,"15min":0.92700000000000005},"min":{"1min":0.46000000000000002,"5min":0.36099999999999999,"15min":0.33800000000000002},"max":{"1min":4.944,"5min":9.3829999999999991,"15min":15.791},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.84499999999999997,"15min":0.90800000000000003},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.371},"max":{"1min":4.6699999999999999,"5min":5.0899999999999999,"15min":10.842000000000001},"last":0.72799999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":1.159,"15min":0.93500000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.378},"max":{"1min":4.4669999999999996,"5min":22.318000000000001,"15min":22.318000000000001},"last":0.63},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.92500000000000004,"15min":0.90600000000000003},"min":{"1min":0.433,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":4.7030000000000003,"5min":9.5169999999999995,"15min":9.7539999999999996},"last":0.59899999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":1.163,"15min":0.98999999999999999},"min":{"1min":0.50900000000000001,"5min":0.503,"15min":0.38100000000000001},"max":{"1min":1.456,"5min":18.152999999999999,"15min":18.152999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.95899999999999996,"15min":0.98499999999999999},"min":{"1min":0.49199999999999999,"5min":0.433,"15min":0.43099999999999999},"max":{"1min":2.6909999999999998,"5min":9.75,"15min":10.34},"last":0.70599999999999996}]},{"osd":4,"last update":"Mon Jan 15 18:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.84899999999999998,"15min":0.93500000000000005},"min":{"1min":0.38900000000000001,"5min":0.314,"15min":0.314},"max":{"1min":4.4480000000000004,"5min":8.4420000000000002,"15min":20.387},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.73199999999999998,"5min":1.1699999999999999,"15min":0.98799999999999999},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":3.6309999999999998,"5min":17.215,"15min":17.215},"last":0.58499999999999996}]},{"osd":5,"last update":"Mon Jan 15 18:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.94499999999999995,"15min":0.95099999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":4.8639999999999999,"5min":9.6630000000000003,"15min":19.959},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.91300000000000003,"15min":0.98099999999999998},"min":{"1min":0.39400000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":4.3959999999999999,"5min":7.6980000000000004,"15min":19.905000000000001},"last":0.79300000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:16:14 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.99099999999999999,"15min":1.155},"min":{"1min":0.505,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":4.3330000000000002,"5min":8.6609999999999996,"15min":22.478999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.82399999999999995,"5min":1.1579999999999999,"15min":1.0660000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.38300000000000001},"max":{"1min":3.573,"5min":17.132000000000001,"15min":21.509},"last":0.69799999999999995}]}]},{"osd":6,"up_from":15,"seq":64424509593,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80812,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94291028,"statfs":{"total":96636764160,"available":96554012672,"internally_reserved":0,"allocated":1142784,"data_stored":492184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":81590726},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":1.0529999999999999,"15min":0.97799999999999998},"min":{"1min":0.40000000000000002,"5min":0.32800000000000001,"15min":0.29099999999999998},"max":{"1min":0.86699999999999999,"5min":32.587000000000003,"15min":32.587000000000003},"last":0.873},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.91600000000000004,"15min":0.86299999999999999},"min":{"1min":0.41999999999999998,"5min":0.38500000000000001,"15min":0.35399999999999998},"max":{"1min":0.88400000000000001,"5min":16.193000000000001,"15min":16.193000000000001},"last":0.59399999999999997}]},{"osd":1,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.82499999999999996,"15min":0.99399999999999999},"min":{"1min":0.40600000000000003,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95299999999999996,"5min":7.7649999999999997,"15min":53.451999999999998},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1,"15min":0.91800000000000004},"min":{"1min":0.49299999999999999,"5min":0.437,"15min":0.42899999999999999},"max":{"1min":0.89900000000000002,"5min":32.512,"15min":32.512},"last":0.64400000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":1.087,"15min":0.97099999999999997},"min":{"1min":0.48299999999999998,"5min":0.41699999999999998,"15min":0.38900000000000001},"max":{"1min":0.90100000000000002,"5min":32.472999999999999,"15min":32.472999999999999},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.997,"15min":0.89600000000000002},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.38500000000000001},"max":{"1min":0.97899999999999998,"5min":16.602,"15min":16.602},"last":0.53500000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.97999999999999998,"15min":0.93100000000000005},"min":{"1min":0.50700000000000001,"5min":0.49399999999999999,"15min":0.41399999999999998},"max":{"1min":0.94499999999999995,"5min":16.388000000000002,"15min":16.388000000000002},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.879,"15min":1.079},"min":{"1min":0.52300000000000002,"5min":0.432,"15min":0.41599999999999998},"max":{"1min":0.996,"5min":5.9740000000000002,"15min":53.398000000000003},"last":0.85299999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.98799999999999999,"15min":0.89100000000000001},"min":{"1min":0.42999999999999999,"5min":0.38100000000000001,"15min":0.34999999999999998},"max":{"1min":0.92200000000000004,"5min":32.549999999999997,"15min":32.549999999999997},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.878,"15min":0.83999999999999997},"min":{"1min":0.42299999999999999,"5min":0.38700000000000001,"15min":0.34499999999999997},"max":{"1min":1.087,"5min":11.852,"15min":11.852},"last":0.75600000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.82699999999999996,"15min":1.0369999999999999},"min":{"1min":0.36199999999999999,"5min":0.35799999999999998,"15min":0.30399999999999999},"max":{"1min":1.1759999999999999,"5min":7.5650000000000004,"15min":53.229999999999997},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.081,"15min":1.028},"min":{"1min":0.432,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.228,"5min":32.404000000000003,"15min":32.404000000000003},"last":0.92600000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.93899999999999995,"15min":1.105},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.1220000000000001,"5min":16.565000000000001,"15min":45.070999999999998},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.68799999999999994,"5min":1.161,"15min":1.21},"min":{"1min":0.46300000000000002,"5min":0.40999999999999998,"15min":0.34999999999999998},"max":{"1min":1.032,"5min":40.972999999999999,"15min":53.338000000000001},"last":0.90700000000000003}]}]},{"osd":4,"up_from":15,"seq":64424509594,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76724,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295116,"statfs":{"total":96636764160,"available":96558198784,"internally_reserved":0,"allocated":1212416,"data_stored":519535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.82499999999999996,"15min":0.88400000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.32900000000000001},"max":{"1min":0.91800000000000004,"5min":7.3380000000000001,"15min":23.318000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.70199999999999996,"15min":0.79300000000000004},"min":{"1min":0.372,"5min":0.36399999999999999,"15min":0.32600000000000001},"max":{"1min":0.82699999999999996,"5min":6.444,"15min":14.723000000000001},"last":0.72399999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.82699999999999996,"15min":0.78800000000000003},"min":{"1min":0.39600000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.2,"5min":9.5700000000000003,"15min":9.5700000000000003},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.92100000000000004,"15min":0.91600000000000004},"min":{"1min":0.45200000000000001,"5min":0.40200000000000002,"15min":0.36899999999999999},"max":{"1min":0.84099999999999997,"5min":7.9980000000000002,"15min":22.408999999999999},"last":0.65600000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.85399999999999998,"15min":0.79900000000000004},"min":{"1min":0.439,"5min":0.439,"15min":0.34399999999999997},"max":{"1min":1.266,"5min":9.7089999999999996,"15min":9.7089999999999996},"last":0.629},{"interface":"front","average":{"1min":0.73699999999999999,"5min":1.0649999999999999,"15min":1.0149999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.39100000000000001},"max":{"1min":1.714,"5min":14.118,"15min":22.338999999999999},"last":0.77000000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.77800000000000002,"15min":0.872},"min":{"1min":0.437,"5min":0.437,"15min":0.41199999999999998},"max":{"1min":0.88800000000000001,"5min":6.7930000000000001,"15min":14.763999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.94199999999999995,"15min":0.90600000000000003},"min":{"1min":0.504,"5min":0.48099999999999998,"15min":0.45500000000000002},"max":{"1min":0.93600000000000005,"5min":9.6289999999999996,"15min":9.6289999999999996},"last":0.81399999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.94599999999999995,"15min":0.95699999999999996},"min":{"1min":0.40000000000000002,"5min":0.34799999999999998,"15min":0.33200000000000002},"max":{"1min":0.94099999999999995,"5min":15.787000000000001,"15min":31.617999999999999},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.75,"15min":0.86399999999999999},"min":{"1min":0.42599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95299999999999996,"5min":6.7169999999999996,"15min":17.145},"last":0.79600000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.84899999999999998,"15min":0.82799999999999996},"min":{"1min":0.42299999999999999,"5min":0.39800000000000002,"15min":0.35499999999999998},"max":{"1min":0.97799999999999998,"5min":9.6709999999999994,"15min":11.124000000000001},"last":0.751},{"interface":"front","average":{"1min":0.69799999999999995,"5min":1.1699999999999999,"15min":1.2230000000000001},"min":{"1min":0.56599999999999995,"5min":0.438,"15min":0.40200000000000002},"max":{"1min":0.90200000000000002,"5min":32.308,"15min":32.308},"last":0.82299999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.749,"15min":1.0029999999999999},"min":{"1min":0.434,"5min":0.41099999999999998,"15min":0.34799999999999998},"max":{"1min":0.91900000000000004,"5min":6.8390000000000004,"15min":39.256},"last":0.876},{"interface":"front","average":{"1min":0.70399999999999996,"5min":1.1160000000000001,"15min":1.004},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.42199999999999999},"max":{"1min":0.86199999999999999,"5min":16.277000000000001,"15min":26.573},"last":0.86299999999999999}]}]},{"osd":3,"up_from":15,"seq":64424509592,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64172,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307668,"statfs":{"total":96636764160,"available":96571052032,"internally_reserved":0,"allocated":1142784,"data_stored":492184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64551367},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.44900000000000001,"15min":0.48899999999999999},"min":{"1min":0.27300000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.624,"5min":2.1160000000000001,"15min":5.8849999999999998},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.54100000000000004,"15min":0.61599999999999999},"min":{"1min":0.312,"5min":0.29799999999999999,"15min":0.29499999999999998},"max":{"1min":0.66600000000000004,"5min":3.0649999999999999,"15min":17.756},"last":0.51800000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.495,"15min":0.58499999999999996},"min":{"1min":0.28899999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68700000000000006,"5min":3.6389999999999998,"15min":12.307},"last":0.435},{"interface":"front","average":{"1min":0.503,"5min":0.55500000000000005,"15min":0.58499999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.70299999999999996,"5min":3.738,"15min":6.0640000000000001},"last":0.41399999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.65400000000000003,"15min":0.68999999999999995},"min":{"1min":0.33700000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69999999999999996,"5min":5.431,"15min":12.468999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.70299999999999996,"15min":0.68300000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.34599999999999997},"max":{"1min":0.77800000000000002,"5min":10.837999999999999,"15min":10.837999999999999},"last":0.748}]},{"osd":4,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":1.0609999999999999,"15min":1.014},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.375},"max":{"1min":1.024,"5min":13.863,"15min":17.853999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.94999999999999996,"15min":0.96199999999999997},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.47099999999999997},"max":{"1min":0.94599999999999995,"5min":6.8289999999999997,"15min":13.31},"last":0.64100000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.153,"15min":0.97399999999999998},"min":{"1min":0.54200000000000004,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1020000000000001,"5min":17.687999999999999,"15min":17.687999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.88500000000000001,"15min":0.96999999999999997},"min":{"1min":0.45500000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.085,"5min":5.0949999999999998,"15min":20.515999999999998},"last":0.86899999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.80200000000000005,"15min":0.97699999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.432},"max":{"1min":1.05,"5min":4.0910000000000002,"15min":23.100999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.90000000000000002,"15min":0.93600000000000005},"min":{"1min":0.59399999999999997,"5min":0.53600000000000003,"15min":0.49399999999999999},"max":{"1min":1.0069999999999999,"5min":6.7220000000000004,"15min":12.76},"last":0.70399999999999996}]},{"osd":7,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":1.331,"15min":1.1659999999999999},"min":{"1min":0.503,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.0089999999999999,"5min":24.321999999999999,"15min":24.321999999999999},"last":0.90900000000000003},{"interface":"front","average":{"1min":2.0510000000000002,"5min":1.139,"15min":1.1020000000000001},"min":{"1min":0.54700000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"last":0.60999999999999999}]}]},{"osd":2,"up_from":15,"seq":64424509592,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64184,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307656,"statfs":{"total":96636764160,"available":96571039744,"internally_reserved":0,"allocated":1155072,"data_stored":498113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.45500000000000002,"15min":0.50900000000000001},"min":{"1min":0.26500000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.71299999999999997,"5min":0.98199999999999998,"15min":5.1849999999999996},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.49199999999999999,"15min":0.53200000000000003},"min":{"1min":0.315,"5min":0.252,"15min":0.252},"max":{"1min":0.68899999999999995,"5min":1.1579999999999999,"15min":7.516},"last":0.51900000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.55800000000000005,"15min":0.61599999999999999},"min":{"1min":0.39800000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.83299999999999996,"5min":0.95799999999999996,"15min":10.907},"last":0.371},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.50800000000000001,"15min":0.59199999999999997},"min":{"1min":0.38200000000000001,"5min":0.28100000000000003,"15min":0.27000000000000002},"max":{"1min":0.77500000000000002,"5min":1.2849999999999999,"15min":8.9190000000000005},"last":0.45900000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55500000000000005,"15min":0.623},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0249999999999999,"5min":1.133,"15min":12.185},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.60199999999999998,"15min":0.66400000000000003},"min":{"1min":0.40600000000000003,"5min":0.373,"15min":0.34000000000000002},"max":{"1min":0.79000000000000004,"5min":0.96699999999999997,"15min":7.6760000000000002},"last":0.439}]},{"osd":4,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.88800000000000001,"15min":0.875},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":4.0190000000000001,"5min":4.3150000000000004,"15min":8.6609999999999996},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.76200000000000001,"15min":0.90300000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.5860000000000001,"5min":2.468,"15min":19.146999999999998},"last":0.56599999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":0.79700000000000004,"15min":0.90800000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":5.0309999999999997,"5min":5.0309999999999997,"15min":31.710000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.83999999999999997,"15min":0.999},"min":{"1min":0.51100000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.3660000000000001,"5min":3.6459999999999999,"15min":22.539000000000001},"last":0.60999999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.84099999999999997,"15min":0.90600000000000003},"min":{"1min":0.499,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":4.1959999999999997,"5min":4.1959999999999997,"15min":9.4540000000000006},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.79200000000000004,"15min":0.90600000000000003},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.633,"5min":1.633,"15min":18.614999999999998},"last":0.68899999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.97799999999999998,"15min":0.96199999999999997},"min":{"1min":0.52100000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":4.3170000000000002,"5min":11.606,"15min":11.606},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.159,"5min":0.92900000000000005,"15min":1.004},"min":{"1min":0.54900000000000004,"5min":0.52500000000000002,"15min":0.502},"max":{"1min":6.1260000000000003,"5min":6.1260000000000003,"15min":17.013000000000002},"last":0.70599999999999996}]}]},{"osd":1,"up_from":15,"seq":64424509594,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68332,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303508,"statfs":{"total":96636764160,"available":96566792192,"internally_reserved":0,"allocated":1142784,"data_stored":492184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.51700000000000002,"15min":0.65700000000000003},"min":{"1min":0.28399999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.6140000000000001,"5min":4.1550000000000002,"15min":19.577000000000002},"last":0.374},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.623,"15min":0.76000000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.251},"max":{"1min":0.83899999999999997,"5min":12.378,"15min":48.277999999999999},"last":0.67400000000000004}]},{"osd":2,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55400000000000005,"15min":0.70799999999999996},"min":{"1min":0.32800000000000001,"5min":0.309,"15min":0.30299999999999999},"max":{"1min":1.8049999999999999,"5min":2.8610000000000002,"15min":40.698},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59099999999999997,"15min":0.877},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.317},"max":{"1min":1.978,"5min":2.5139999999999998,"15min":48.316000000000003},"last":0.63800000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.623,"15min":0.82999999999999996},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.33700000000000002},"max":{"1min":0.94699999999999995,"5min":2.8620000000000001,"15min":48.234999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.69299999999999995,"15min":0.67700000000000005},"min":{"1min":0.33400000000000002,"5min":0.315,"15min":0.311},"max":{"1min":1.0780000000000001,"5min":9.9719999999999995,"15min":9.9719999999999995},"last":0.499}]},{"osd":4,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.90300000000000002,"15min":0.88800000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.44500000000000001},"max":{"1min":0.97499999999999998,"5min":9.9329999999999998,"15min":9.9329999999999998},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.86199999999999999,"15min":0.93300000000000005},"min":{"1min":0.502,"5min":0.45800000000000002,"15min":0.42499999999999999},"max":{"1min":2.3900000000000001,"5min":9.8610000000000007,"15min":19.829000000000001},"last":0.85599999999999998}]},{"osd":5,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.82899999999999996,"15min":1.0129999999999999},"min":{"1min":0.53700000000000003,"5min":0.48599999999999999,"15min":0.46100000000000002},"max":{"1min":2.5960000000000001,"5min":8.5839999999999996,"15min":22.594999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.80600000000000005,"15min":1.157},"min":{"1min":0.44400000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.6879999999999999,"5min":2.9609999999999999,"15min":48.198},"last":0.75700000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.90800000000000003,"15min":1.04},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0189999999999999,"5min":9.3989999999999991,"15min":35.112000000000002},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.91300000000000003,"15min":0.96899999999999997},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.50800000000000001},"max":{"1min":2.512,"5min":5.8849999999999998,"15min":19.780999999999999},"last":0.94399999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.89200000000000002,"15min":1.1619999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.93300000000000005,"5min":11.039,"15min":48.148000000000003},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.748,"5min":0.97899999999999998,"15min":0.98999999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.52100000000000002},"max":{"1min":1.0609999999999999,"5min":9.8640000000000008,"15min":9.8640000000000008},"last":0.95899999999999996}]}]},{"osd":5,"up_from":14,"seq":60129542298,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58724,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313116,"statfs":{"total":96636764160,"available":96576630784,"internally_reserved":0,"allocated":1196032,"data_stored":517869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.81200000000000006,"15min":0.80100000000000005},"min":{"1min":0.375,"5min":0.29899999999999999,"15min":0.28999999999999998},"max":{"1min":1.383,"5min":5.1639999999999997,"15min":21.385000000000002},"last":0.874},{"interface":"front","average":{"1min":0.748,"5min":0.78700000000000003,"15min":0.877},"min":{"1min":0.57999999999999996,"5min":0.378,"15min":0.34899999999999998},"max":{"1min":1.3029999999999999,"5min":5.3869999999999996,"15min":9.359},"last":0.67100000000000004}]},{"osd":1,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.82199999999999995,"15min":0.89500000000000002},"min":{"1min":0.44400000000000001,"5min":0.375,"15min":0.34899999999999998},"max":{"1min":1.2070000000000001,"5min":3.7429999999999999,"15min":21.120000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.78500000000000003,"5min":1.1850000000000001,"15min":0.96999999999999997},"min":{"1min":0.48999999999999999,"5min":0.38600000000000001,"15min":0.315},"max":{"1min":1.532,"5min":15.843,"15min":21.5},"last":0.90000000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.83799999999999997,"15min":0.89800000000000002},"min":{"1min":0.47699999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.421,"5min":5.516,"15min":8.7750000000000004},"last":0.622},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.90800000000000003,"15min":0.98799999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.347,"5min":8.3529999999999998,"15min":21.201000000000001},"last":0.68700000000000006}]},{"osd":3,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.93200000000000005,"15min":0.91300000000000003},"min":{"1min":0.58499999999999996,"5min":0.37,"15min":0.37},"max":{"1min":1.478,"5min":7.7199999999999998,"15min":21.457000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.85499999999999998,"15min":0.96299999999999997},"min":{"1min":0.57199999999999995,"5min":0.47699999999999998,"15min":0.45400000000000001},"max":{"1min":1.5,"5min":5.6109999999999998,"15min":8.7300000000000004},"last":0.78900000000000003}]},{"osd":4,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.93500000000000005,"15min":0.88800000000000001},"min":{"1min":0.376,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":4.8650000000000002,"5min":10.962,"15min":10.962},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.78800000000000003,"15min":1.018},"min":{"1min":0.442,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.464,"5min":4.6639999999999997,"15min":26.146999999999998},"last":0.80700000000000005}]},{"osd":6,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.86599999999999999,"15min":0.83799999999999997},"min":{"1min":0.44700000000000001,"5min":0.36099999999999999,"15min":0.32000000000000001},"max":{"1min":2.8940000000000001,"5min":5.2160000000000002,"15min":21.535},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.85899999999999999,"15min":1.0129999999999999},"min":{"1min":0.432,"5min":0.379,"15min":0.36299999999999999},"max":{"1min":4.9139999999999997,"5min":4.9139999999999997,"15min":30.488},"last":0.73899999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.276,"15min":1.2290000000000001},"min":{"1min":0.51800000000000002,"5min":0.432,"15min":0.39700000000000002},"max":{"1min":8.7539999999999996,"5min":33.499000000000002,"15min":33.499000000000002},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.98599999999999999,"15min":1.0900000000000001},"min":{"1min":0.46200000000000002,"5min":0.441,"15min":0.36499999999999999},"max":{"1min":4.3120000000000003,"5min":6.306,"15min":25.437999999999999},"last":0.76200000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542298,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64800,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307040,"statfs":{"total":96636764160,"available":96570408960,"internally_reserved":0,"allocated":1789952,"data_stored":1108749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.52400000000000002,"15min":0.627},"min":{"1min":0.26100000000000001,"5min":0.245,"15min":0.23699999999999999},"max":{"1min":0.93500000000000005,"5min":7.907,"15min":19.843},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58199999999999996,"15min":0.67400000000000004},"min":{"1min":0.35599999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.8240000000000001,"5min":3.512,"15min":10.154},"last":0.33100000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.71599999999999997,"15min":0.80100000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.30399999999999999},"max":{"1min":1.4410000000000001,"5min":7.0810000000000004,"15min":25.905000000000001},"last":0.379},{"interface":"front","average":{"1min":0.498,"5min":0.57499999999999996,"15min":0.61399999999999999},"min":{"1min":0.32700000000000001,"5min":0.27800000000000002,"15min":0.26700000000000002},"max":{"1min":0.79300000000000004,"5min":3.8500000000000001,"15min":9.3829999999999991},"last":0.39600000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.69399999999999995,"15min":0.74199999999999999},"min":{"1min":0.34699999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":2.3519999999999999,"5min":6.2850000000000001,"15min":9.1780000000000008},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.70199999999999996,"15min":0.76200000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.36899999999999999},"max":{"1min":1.327,"5min":6.5979999999999999,"15min":10.210000000000001},"last":0.44400000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.84199999999999997,"15min":0.98399999999999999},"min":{"1min":0.47699999999999998,"5min":0.39100000000000001,"15min":0.38},"max":{"1min":1.2929999999999999,"5min":4.5179999999999998,"15min":22.814},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.83399999999999996,"15min":1.0289999999999999},"min":{"1min":0.502,"5min":0.502,"15min":0.41199999999999998},"max":{"1min":1.2749999999999999,"5min":4.2329999999999997,"15min":23.341999999999999},"last":0.57999999999999996}]},{"osd":5,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.86399999999999999,"15min":0.96999999999999997},"min":{"1min":0.54400000000000004,"5min":0.51600000000000001,"15min":0.441},"max":{"1min":2.6190000000000002,"5min":4.008,"15min":12.428000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.79900000000000004,"5min":1.252,"15min":1.258},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.3120000000000001,"5min":23.021000000000001,"15min":23.021000000000001},"last":0.66100000000000003}]},{"osd":6,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.92000000000000004,"15min":1.212},"min":{"1min":0.53500000000000003,"5min":0.48399999999999999,"15min":0.46200000000000002},"max":{"1min":1.6359999999999999,"5min":6.5250000000000004,"15min":31.059000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.94799999999999995,"15min":1.089},"min":{"1min":0.59399999999999997,"5min":0.52700000000000002,"15min":0.49099999999999999},"max":{"1min":2.5659999999999998,"5min":6.9690000000000003,"15min":40.731000000000002},"last":0.623}]},{"osd":7,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":1.107,"15min":1.2},"min":{"1min":0.59499999999999997,"5min":0.499,"15min":0.42499999999999999},"max":{"1min":2.6539999999999999,"5min":10.497999999999999,"15min":24.492999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.88300000000000001,"15min":1.103},"min":{"1min":0.55300000000000005,"5min":0.495,"15min":0.46600000000000003},"max":{"1min":1.3740000000000001,"5min":4.6600000000000001,"15min":23.524000000000001},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:17:22.020 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.3f is not active+clean 2024-01-15T18:17:22.021 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.021 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.3d is not active+clean 2024-01-15T18:17:22.021 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.021 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.3c is not active+clean 2024-01-15T18:17:22.021 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.021 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.3b is not active+clean 2024-01-15T18:17:22.021 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.021 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.3a is not active+clean 2024-01-15T18:17:22.021 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.021 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.39 is not active+clean 2024-01-15T18:17:22.021 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.022 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.36 is not active+clean 2024-01-15T18:17:22.022 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.022 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.35 is not active+clean 2024-01-15T18:17:22.022 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.022 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.33 is not active+clean 2024-01-15T18:17:22.022 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.022 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.32 is not active+clean 2024-01-15T18:17:22.022 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.022 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.31 is not active+clean 2024-01-15T18:17:22.022 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.022 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.30 is not active+clean 2024-01-15T18:17:22.022 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.023 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.2e is not active+clean 2024-01-15T18:17:22.023 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.023 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.2d is not active+clean 2024-01-15T18:17:22.023 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.023 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.2b is not active+clean 2024-01-15T18:17:22.023 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.023 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.28 is not active+clean 2024-01-15T18:17:22.023 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.023 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.27 is not active+clean 2024-01-15T18:17:22.023 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.023 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.25 is not active+clean 2024-01-15T18:17:22.023 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.023 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.24 is not active+clean 2024-01-15T18:17:22.024 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.024 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.22 is not active+clean 2024-01-15T18:17:22.024 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.024 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.21 is not active+clean 2024-01-15T18:17:22.024 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.024 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.20 is not active+clean 2024-01-15T18:17:22.024 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.024 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.1e is not active+clean 2024-01-15T18:17:22.024 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.024 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.1f is not active+clean 2024-01-15T18:17:22.024 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.025 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1e is not active+clean 2024-01-15T18:17:22.025 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 94, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:17:15.096546+0000', 'last_change': '2024-01-15T18:17:15.096546+0000', 'last_active': '2024-01-15T18:17:15.077857+0000', 'last_peered': '2024-01-15T18:17:15.077857+0000', 'last_clean': '2024-01-15T18:17:15.077857+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:15.096546+0000', 'last_undegraded': '2024-01-15T18:17:15.096546+0000', 'last_fullsized': '2024-01-15T18:17:15.096546+0000', 'mapping_epoch': 94, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 94, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-15T18:17:22.025 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.1c is not active+clean 2024-01-15T18:17:22.025 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.025 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1b is not active+clean 2024-01-15T18:17:22.025 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 94, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:17:15.093523+0000', 'last_change': '2024-01-15T18:17:15.093523+0000', 'last_active': '2024-01-15T18:17:15.077857+0000', 'last_peered': '2024-01-15T18:17:15.077857+0000', 'last_clean': '2024-01-15T18:17:15.077857+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:15.093523+0000', 'last_undegraded': '2024-01-15T18:17:15.093523+0000', 'last_fullsized': '2024-01-15T18:17:15.093523+0000', 'mapping_epoch': 94, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 94, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:17:22.025 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.1b is not active+clean 2024-01-15T18:17:22.025 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.025 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.18 is not active+clean 2024-01-15T18:17:22.025 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.025 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.19 is not active+clean 2024-01-15T18:17:22.025 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 94, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:17:15.095582+0000', 'last_change': '2024-01-15T18:17:15.095582+0000', 'last_active': '2024-01-15T18:17:15.077857+0000', 'last_peered': '2024-01-15T18:17:15.077857+0000', 'last_clean': '2024-01-15T18:17:15.077857+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:15.095582+0000', 'last_undegraded': '2024-01-15T18:17:15.095582+0000', 'last_fullsized': '2024-01-15T18:17:15.095582+0000', 'mapping_epoch': 94, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 94, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:17:22.026 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.18 is not active+clean 2024-01-15T18:17:22.026 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 94, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:17:15.093925+0000', 'last_change': '2024-01-15T18:17:15.093925+0000', 'last_active': '2024-01-15T18:17:15.077857+0000', 'last_peered': '2024-01-15T18:17:15.077857+0000', 'last_clean': '2024-01-15T18:17:15.077857+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:15.093925+0000', 'last_undegraded': '2024-01-15T18:17:15.093925+0000', 'last_fullsized': '2024-01-15T18:17:15.093925+0000', 'mapping_epoch': 94, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 94, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:17:22.026 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.17 is not active+clean 2024-01-15T18:17:22.026 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 94, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:17:15.093475+0000', 'last_change': '2024-01-15T18:17:15.093475+0000', 'last_active': '2024-01-15T18:17:15.077857+0000', 'last_peered': '2024-01-15T18:17:15.077857+0000', 'last_clean': '2024-01-15T18:17:15.077857+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:15.093475+0000', 'last_undegraded': '2024-01-15T18:17:15.093475+0000', 'last_fullsized': '2024-01-15T18:17:15.093475+0000', 'mapping_epoch': 94, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 94, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:17:22.026 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.16 is not active+clean 2024-01-15T18:17:22.026 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 94, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:17:15.095406+0000', 'last_change': '2024-01-15T18:17:15.095406+0000', 'last_active': '2024-01-15T18:17:15.077857+0000', 'last_peered': '2024-01-15T18:17:15.077857+0000', 'last_clean': '2024-01-15T18:17:15.077857+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:15.095406+0000', 'last_undegraded': '2024-01-15T18:17:15.095406+0000', 'last_fullsized': '2024-01-15T18:17:15.095406+0000', 'mapping_epoch': 94, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 94, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:17:22.026 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.14 is not active+clean 2024-01-15T18:17:22.026 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.026 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.15 is not active+clean 2024-01-15T18:17:22.026 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 94, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:17:15.096252+0000', 'last_change': '2024-01-15T18:17:15.096252+0000', 'last_active': '2024-01-15T18:17:15.077857+0000', 'last_peered': '2024-01-15T18:17:15.077857+0000', 'last_clean': '2024-01-15T18:17:15.077857+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:15.096252+0000', 'last_undegraded': '2024-01-15T18:17:15.096252+0000', 'last_fullsized': '2024-01-15T18:17:15.096252+0000', 'mapping_epoch': 94, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 94, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-15T18:17:22.026 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.15 is not active+clean 2024-01-15T18:17:22.027 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.027 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.12 is not active+clean 2024-01-15T18:17:22.027 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.027 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.13 is not active+clean 2024-01-15T18:17:22.027 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 94, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:17:15.094637+0000', 'last_change': '2024-01-15T18:17:15.094637+0000', 'last_active': '2024-01-15T18:17:15.077857+0000', 'last_peered': '2024-01-15T18:17:15.077857+0000', 'last_clean': '2024-01-15T18:17:15.077857+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:15.094637+0000', 'last_undegraded': '2024-01-15T18:17:15.094637+0000', 'last_fullsized': '2024-01-15T18:17:15.094637+0000', 'mapping_epoch': 94, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 94, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:17:22.027 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.13 is not active+clean 2024-01-15T18:17:22.027 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.027 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.12 is not active+clean 2024-01-15T18:17:22.027 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 94, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:17:15.094404+0000', 'last_change': '2024-01-15T18:17:15.094404+0000', 'last_active': '2024-01-15T18:17:15.077857+0000', 'last_peered': '2024-01-15T18:17:15.077857+0000', 'last_clean': '2024-01-15T18:17:15.077857+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:15.094404+0000', 'last_undegraded': '2024-01-15T18:17:15.094404+0000', 'last_fullsized': '2024-01-15T18:17:15.094404+0000', 'mapping_epoch': 94, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 94, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:17:22.027 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.11 is not active+clean 2024-01-15T18:17:22.027 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.027 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.10 is not active+clean 2024-01-15T18:17:22.028 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 94, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:17:15.093447+0000', 'last_change': '2024-01-15T18:17:15.093447+0000', 'last_active': '2024-01-15T18:17:15.077857+0000', 'last_peered': '2024-01-15T18:17:15.077857+0000', 'last_clean': '2024-01-15T18:17:15.077857+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:15.093447+0000', 'last_undegraded': '2024-01-15T18:17:15.093447+0000', 'last_fullsized': '2024-01-15T18:17:15.093447+0000', 'mapping_epoch': 94, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 94, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:17:22.028 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.f is not active+clean 2024-01-15T18:17:22.028 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.028 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.d is not active+clean 2024-01-15T18:17:22.028 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 94, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:17:15.095378+0000', 'last_change': '2024-01-15T18:17:15.095378+0000', 'last_active': '2024-01-15T18:17:15.077857+0000', 'last_peered': '2024-01-15T18:17:15.077857+0000', 'last_clean': '2024-01-15T18:17:15.077857+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:15.095378+0000', 'last_undegraded': '2024-01-15T18:17:15.095378+0000', 'last_fullsized': '2024-01-15T18:17:15.095378+0000', 'mapping_epoch': 94, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 94, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:17:22.028 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.d is not active+clean 2024-01-15T18:17:22.028 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.028 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.a is not active+clean 2024-01-15T18:17:22.028 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.028 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.b is not active+clean 2024-01-15T18:17:22.028 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.029 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.8 is not active+clean 2024-01-15T18:17:22.029 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.029 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.9 is not active+clean 2024-01-15T18:17:22.029 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.029 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.7 is not active+clean 2024-01-15T18:17:22.029 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.029 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.4 is not active+clean 2024-01-15T18:17:22.029 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.029 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.5 is not active+clean 2024-01-15T18:17:22.029 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.029 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.4 is not active+clean 2024-01-15T18:17:22.029 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 94, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:17:15.095295+0000', 'last_change': '2024-01-15T18:17:15.095295+0000', 'last_active': '2024-01-15T18:17:15.077857+0000', 'last_peered': '2024-01-15T18:17:15.077857+0000', 'last_clean': '2024-01-15T18:17:15.077857+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:15.095295+0000', 'last_undegraded': '2024-01-15T18:17:15.095295+0000', 'last_fullsized': '2024-01-15T18:17:15.095295+0000', 'mapping_epoch': 94, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 94, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:17:22.029 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.2 is not active+clean 2024-01-15T18:17:22.030 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.030 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2 is not active+clean 2024-01-15T18:17:22.030 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 94, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:17:15.095318+0000', 'last_change': '2024-01-15T18:17:15.095318+0000', 'last_active': '2024-01-15T18:17:15.077857+0000', 'last_peered': '2024-01-15T18:17:15.077857+0000', 'last_clean': '2024-01-15T18:17:15.077857+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:15.095318+0000', 'last_undegraded': '2024-01-15T18:17:15.095318+0000', 'last_fullsized': '2024-01-15T18:17:15.095318+0000', 'mapping_epoch': 94, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 94, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:17:22.030 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.0 is not active+clean 2024-01-15T18:17:22.030 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.030 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1 is not active+clean 2024-01-15T18:17:22.030 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 94, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:17:15.095327+0000', 'last_change': '2024-01-15T18:17:15.095327+0000', 'last_active': '2024-01-15T18:17:15.077857+0000', 'last_peered': '2024-01-15T18:17:15.077857+0000', 'last_clean': '2024-01-15T18:17:15.077857+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:15.095327+0000', 'last_undegraded': '2024-01-15T18:17:15.095327+0000', 'last_fullsized': '2024-01-15T18:17:15.095327+0000', 'mapping_epoch': 94, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 94, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:17:22.030 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.1 is not active+clean 2024-01-15T18:17:22.030 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.030 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.0 is not active+clean 2024-01-15T18:17:22.030 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 94, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:17:15.094409+0000', 'last_change': '2024-01-15T18:17:15.094409+0000', 'last_active': '2024-01-15T18:17:15.077857+0000', 'last_peered': '2024-01-15T18:17:15.077857+0000', 'last_clean': '2024-01-15T18:17:15.077857+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:15.094409+0000', 'last_undegraded': '2024-01-15T18:17:15.094409+0000', 'last_fullsized': '2024-01-15T18:17:15.094409+0000', 'mapping_epoch': 94, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 94, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:17:22.031 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.e is not active+clean 2024-01-15T18:17:22.031 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-01-15T18:17:16.086114+0000', 'last_change': '2024-01-15T18:17:16.086114+0000', 'last_active': '2024-01-15T18:17:16.086114+0000', 'last_peered': '2024-01-15T18:17:16.086114+0000', 'last_clean': '2024-01-15T18:17:16.086114+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:16.086114+0000', 'last_undegraded': '2024-01-15T18:17:16.086114+0000', 'last_fullsized': '2024-01-15T18:17:16.086114+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:16.086114+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-01-15T18:17:22.031 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.f is not active+clean 2024-01-15T18:17:22.031 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 94, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:17:15.096380+0000', 'last_change': '2024-01-15T18:17:15.096380+0000', 'last_active': '2024-01-15T18:17:15.077857+0000', 'last_peered': '2024-01-15T18:17:15.077857+0000', 'last_clean': '2024-01-15T18:17:15.077857+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:17:15.096380+0000', 'last_undegraded': '2024-01-15T18:17:15.096380+0000', 'last_fullsized': '2024-01-15T18:17:15.096380+0000', 'mapping_epoch': 94, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 94, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'last_clean_scrub_stamp': '2024-01-15T18:17:15.077857+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-15T18:17:22.031 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-15T18:17:22.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.202+0000 7f26bb400640 1 Processor -- start 2024-01-15T18:17:22.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.206+0000 7f26bb400640 1 -- start start 2024-01-15T18:17:22.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.206+0000 7f26bb400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26b415b2c0 0x7f26b406a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:22.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.206+0000 7f26bb400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b4072150 con 0x7f26b40ce3f0 2024-01-15T18:17:22.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.206+0000 7f26bb400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b40cc9e0 con 0x7f26b40cdd30 2024-01-15T18:17:22.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.206+0000 7f26bb400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b40cd660 con 0x7f26b415b2c0 2024-01-15T18:17:22.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.206+0000 7f26b9400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f26b40ce3f0 0x7f26b4071a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:48902/0 (socket says 172.21.15.27:48902) 2024-01-15T18:17:22.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.206+0000 7f26b9400640 1 -- 172.21.15.27:0/1755484797 learned_addr learned my addr 172.21.15.27:0/1755484797 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:22.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.206+0000 7f26b8a00640 1 --2- 172.21.15.27:0/1755484797 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26b415b2c0 0x7f26b406a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:22.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.206+0000 7f26b3400640 1 -- 172.21.15.27:0/1755484797 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868991284 0 0) 0x7f26b4072150 con 0x7f26b40ce3f0 2024-01-15T18:17:22.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.206+0000 7f26b3400640 1 -- 172.21.15.27:0/1755484797 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2694003530 con 0x7f26b40ce3f0 2024-01-15T18:17:22.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.206+0000 7f26b3400640 1 -- 172.21.15.27:0/1755484797 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564947086 0 0) 0x7f26b40cc9e0 con 0x7f26b40cdd30 2024-01-15T18:17:22.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.206+0000 7f26b3400640 1 -- 172.21.15.27:0/1755484797 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26b4072150 con 0x7f26b40cdd30 2024-01-15T18:17:22.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.206+0000 7f26b3400640 1 -- 172.21.15.27:0/1755484797 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 656137684 0 0) 0x7f2694003530 con 0x7f26b40ce3f0 2024-01-15T18:17:22.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.206+0000 7f26b8a00640 1 -- 172.21.15.27:0/1755484797 >> v1:172.21.15.93:6789/0 conn(0x7f26b40cdd30 legacy=0x7f26b4071330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:22.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.206+0000 7f26b8a00640 1 -- 172.21.15.27:0/1755484797 >> v1:172.21.15.27:6789/0 conn(0x7f26b40ce3f0 legacy=0x7f26b4071a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:22.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.206+0000 7f26b8a00640 1 -- 172.21.15.27:0/1755484797 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b40cccc0 con 0x7f26b415b2c0 2024-01-15T18:17:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.210+0000 7f26b8a00640 1 --2- 172.21.15.27:0/1755484797 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26b415b2c0 0x7f26b406a4f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f26a4009820 tx=0x7f26a401c390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab1112962f74de05 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.210+0000 7f26b3400640 1 -- 172.21.15.27:0/1755484797 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ac001a00 con 0x7f26b415b2c0 2024-01-15T18:17:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.210+0000 7f26b3400640 1 -- 172.21.15.27:0/1755484797 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a40a7050 con 0x7f26b415b2c0 2024-01-15T18:17:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.210+0000 7f26b3400640 1 -- 172.21.15.27:0/1755484797 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a4004250 con 0x7f26b415b2c0 2024-01-15T18:17:22.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.210+0000 7f26bb400640 1 -- 172.21.15.27:0/1755484797 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26b415b2c0 msgr2=0x7f26b406a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:22.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.210+0000 7f26bb400640 1 --2- 172.21.15.27:0/1755484797 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26b415b2c0 0x7f26b406a4f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f26a4009820 tx=0x7f26a401c390 comp rx=0 tx=0).stop 2024-01-15T18:17:22.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.210+0000 7f26bb400640 1 -- 172.21.15.27:0/1755484797 shutdown_connections 2024-01-15T18:17:22.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.210+0000 7f26bb400640 1 --2- 172.21.15.27:0/1755484797 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26b415b2c0 0x7f26b406a4f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:22.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.210+0000 7f26bb400640 1 -- 172.21.15.27:0/1755484797 >> 172.21.15.27:0/1755484797 conn(0x7f26b40b1ae0 msgr2=0x7f26b40b1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:22.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.210+0000 7f26bb400640 1 -- 172.21.15.27:0/1755484797 shutdown_connections 2024-01-15T18:17:22.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.210+0000 7f26bb400640 1 -- 172.21.15.27:0/1755484797 wait complete. 2024-01-15T18:17:22.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.214+0000 7f26bb400640 1 Processor -- start 2024-01-15T18:17:22.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.214+0000 7f26bb400640 1 -- start start 2024-01-15T18:17:22.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.214+0000 7f26bb400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26b40cdd30 0x7f26b41532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:22.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.214+0000 7f26bb400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f26b40ce3f0 0x7f26b415b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:22.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.214+0000 7f26bb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b415b2c0 0x7f26b41f6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:22.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.214+0000 7f26bb400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b415bc70 con 0x7f26b415b2c0 2024-01-15T18:17:22.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.214+0000 7f26bb400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b415bea0 con 0x7f26b40ce3f0 2024-01-15T18:17:22.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.214+0000 7f26bb400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b41514a0 con 0x7f26b40cdd30 2024-01-15T18:17:22.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.214+0000 7f26b9400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b415b2c0 0x7f26b41f6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.214+0000 7f26b8a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26b40cdd30 0x7f26b41532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.214+0000 7f26b9400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b415b2c0 0x7f26b41f6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59730/0 (socket says 172.21.15.27:59730) 2024-01-15T18:17:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.214+0000 7f26b9400640 1 -- 172.21.15.27:0/2961061125 learned_addr learned my addr 172.21.15.27:0/2961061125 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.214+0000 7f26b3e00640 1 --2- 172.21.15.27:0/2961061125 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f26b40ce3f0 0x7f26b415b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.214+0000 7f26b9400640 1 -- 172.21.15.27:0/2961061125 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26b40cdd30 msgr2=0x7f26b41532f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.214+0000 7f26b9400640 1 --2- 172.21.15.27:0/2961061125 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26b40cdd30 0x7f26b41532f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.214+0000 7f26b9400640 1 -- 172.21.15.27:0/2961061125 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f26b40ce3f0 msgr2=0x7f26b415b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.214+0000 7f26b9400640 1 --2- 172.21.15.27:0/2961061125 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f26b40ce3f0 0x7f26b415b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.214+0000 7f26b9400640 1 -- 172.21.15.27:0/2961061125 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a4009420 con 0x7f26b415b2c0 2024-01-15T18:17:22.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.214+0000 7f26b9400640 1 --2- 172.21.15.27:0/2961061125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b415b2c0 0x7f26b41f6890 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f26ac002000 tx=0x7f26ac00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:22.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.214+0000 7f26b0c00640 1 -- 172.21.15.27:0/2961061125 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ac012400 con 0x7f26b415b2c0 2024-01-15T18:17:22.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.214+0000 7f26b0c00640 1 -- 172.21.15.27:0/2961061125 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26ac012580 con 0x7f26b415b2c0 2024-01-15T18:17:22.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.214+0000 7f26b0c00640 1 -- 172.21.15.27:0/2961061125 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ac012930 con 0x7f26b415b2c0 2024-01-15T18:17:22.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.214+0000 7f26bb400640 1 -- 172.21.15.27:0/2961061125 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26b415c1f0 con 0x7f26b415b2c0 2024-01-15T18:17:22.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.214+0000 7f26bb400640 1 -- 172.21.15.27:0/2961061125 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b41f6dd0 con 0x7f26b415b2c0 2024-01-15T18:17:22.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.218+0000 7f26bb400640 1 -- 172.21.15.27:0/2961061125 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26b4067a10 con 0x7f26b415b2c0 2024-01-15T18:17:22.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.218+0000 7f26b0c00640 1 -- 172.21.15.27:0/2961061125 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f26ac007070 con 0x7f26b415b2c0 2024-01-15T18:17:22.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.218+0000 7f26b0c00640 1 --2- 172.21.15.27:0/2961061125 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f268c067700 0x7f268c069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:22.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.218+0000 7f26b0c00640 1 -- 172.21.15.27:0/2961061125 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f26ac0832f0 con 0x7f26b415b2c0 2024-01-15T18:17:22.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.218+0000 7f26b8a00640 1 --2- 172.21.15.27:0/2961061125 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f268c067700 0x7f268c069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:22.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.218+0000 7f26b8a00640 1 --2- 172.21.15.27:0/2961061125 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f268c067700 0x7f268c069bc0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f26a40ad000 tx=0x7f26a4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:22.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.222+0000 7f26b0c00640 1 -- 172.21.15.27:0/2961061125 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f26ac0b7280 con 0x7f26b415b2c0 2024-01-15T18:17:22.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.434+0000 7f26bb400640 1 -- 172.21.15.27:0/2961061125 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f26b404e850 con 0x7f26b415b2c0 2024-01-15T18:17:22.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.434+0000 7f26b0c00640 1 -- 172.21.15.27:0/2961061125 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1174 (secure 0 0 0) 0x7f26ac017080 con 0x7f26b415b2c0 2024-01-15T18:17:22.434 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:17:22.434 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":769,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":97,"num_osds":8,"num_up_osds":8,"osd_up_since":1705341884,"num_in_osds":8,"osd_in_since":1705341875,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":45},{"state_name":"active+clean","count":36},{"state_name":"creating+peering","count":16}],"num_pgs":97,"num_pools":3,"num_objects":13,"data_bytes":592562,"bytes_used":555515904,"bytes_avail":772538597376,"bytes_total":773094113280,"unknown_pgs_ratio":0.46391752362251282,"inactive_pgs_ratio":0.16494844853878021,"write_bytes_sec":777,"read_op_per_sec":0,"write_op_per_sec":4},"fsmap":{"epoch":67,"id":7,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":7,"rank":0,"name":"a","status":"up:active","gid":8937}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":18,"modified":"2024-01-15T18:16:52.347508+0000","services":{}},"progress_events":{}} 2024-01-15T18:17:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.438+0000 7f26bb400640 1 -- 172.21.15.27:0/2961061125 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f268c067700 msgr2=0x7f268c069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.438+0000 7f26bb400640 1 --2- 172.21.15.27:0/2961061125 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f268c067700 0x7f268c069bc0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f26a40ad000 tx=0x7f26a4008040 comp rx=0 tx=0).stop 2024-01-15T18:17:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.438+0000 7f26bb400640 1 -- 172.21.15.27:0/2961061125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b415b2c0 msgr2=0x7f26b41f6890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.438+0000 7f26bb400640 1 --2- 172.21.15.27:0/2961061125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b415b2c0 0x7f26b41f6890 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f26ac002000 tx=0x7f26ac00a430 comp rx=0 tx=0).stop 2024-01-15T18:17:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.438+0000 7f26bb400640 1 -- 172.21.15.27:0/2961061125 shutdown_connections 2024-01-15T18:17:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.438+0000 7f26bb400640 1 --2- 172.21.15.27:0/2961061125 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f268c067700 0x7f268c069bc0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.438+0000 7f26bb400640 1 --2- 172.21.15.27:0/2961061125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f26b415b2c0 0x7f26b41f6890 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.438+0000 7f26bb400640 1 --2- 172.21.15.27:0/2961061125 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f26b40ce3f0 0x7f26b415b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.438+0000 7f26bb400640 1 --2- 172.21.15.27:0/2961061125 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f26b40cdd30 0x7f26b41532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.438+0000 7f26bb400640 1 -- 172.21.15.27:0/2961061125 >> 172.21.15.27:0/2961061125 conn(0x7f26b40b1ae0 msgr2=0x7f26b40b12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.438+0000 7f26bb400640 1 -- 172.21.15.27:0/2961061125 shutdown_connections 2024-01-15T18:17:22.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.438+0000 7f26bb400640 1 -- 172.21.15.27:0/2961061125 wait complete. 2024-01-15T18:17:22.456 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-01-15T18:17:22.457 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:17:22.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.598+0000 7f2673200640 1 Processor -- start 2024-01-15T18:17:22.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.598+0000 7f2673200640 1 -- start start 2024-01-15T18:17:22.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.598+0000 7f2673200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f266c0b2390 0x7f266c0cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:22.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.598+0000 7f2673200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f266c06bb80 0x7f266c0cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:22.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.598+0000 7f2673200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f266c0b2770 con 0x7f266c0cfcc0 2024-01-15T18:17:22.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.598+0000 7f2673200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f266c06c060 con 0x7f266c0b2390 2024-01-15T18:17:22.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.598+0000 7f2673200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f266c06c340 con 0x7f266c06bb80 2024-01-15T18:17:22.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.598+0000 7f266be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f266c0b2390 0x7f266c0cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:22.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.598+0000 7f266be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f266c0b2390 0x7f266c0cf780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59744/0 (socket says 172.21.15.27:59744) 2024-01-15T18:17:22.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.598+0000 7f266be00640 1 -- 172.21.15.27:0/894118951 learned_addr learned my addr 172.21.15.27:0/894118951 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:22.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.598+0000 7f266b400640 1 --2- 172.21.15.27:0/894118951 >> v2:172.21.15.93:3300/0 conn(0x7f266c06bb80 0x7f266c0cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:22.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.598+0000 7f266be00640 1 -- 172.21.15.27:0/894118951 >> v2:172.21.15.93:3300/0 conn(0x7f266c06bb80 msgr2=0x7f266c0cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:22.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.598+0000 7f266be00640 1 --2- 172.21.15.27:0/894118951 >> v2:172.21.15.93:3300/0 conn(0x7f266c06bb80 0x7f266c0cd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:22.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.598+0000 7f266be00640 1 -- 172.21.15.27:0/894118951 >> v1:172.21.15.93:6789/0 conn(0x7f266c0cfcc0 legacy=0x7f266c161120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:22.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.598+0000 7f266be00640 1 -- 172.21.15.27:0/894118951 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f266c06c620 con 0x7f266c0b2390 2024-01-15T18:17:22.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.602+0000 7f266be00640 1 --2- 172.21.15.27:0/894118951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f266c0b2390 0x7f266c0cf780 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f266c070ea0 tx=0x7f265c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84ee9caa07269b0a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:22.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.602+0000 7f266aa00640 1 -- 172.21.15.27:0/894118951 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265c0aa5f0 con 0x7f266c0b2390 2024-01-15T18:17:22.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.602+0000 7f266aa00640 1 -- 172.21.15.27:0/894118951 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f265c0aa750 con 0x7f266c0b2390 2024-01-15T18:17:22.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.602+0000 7f266aa00640 1 -- 172.21.15.27:0/894118951 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265c0aaae0 con 0x7f266c0b2390 2024-01-15T18:17:22.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.602+0000 7f2673200640 1 -- 172.21.15.27:0/894118951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f266c0b2390 msgr2=0x7f266c0cf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:22.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.602+0000 7f2673200640 1 --2- 172.21.15.27:0/894118951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f266c0b2390 0x7f266c0cf780 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f266c070ea0 tx=0x7f265c01c490 comp rx=0 tx=0).stop 2024-01-15T18:17:22.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.602+0000 7f2673200640 1 -- 172.21.15.27:0/894118951 shutdown_connections 2024-01-15T18:17:22.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.602+0000 7f2673200640 1 --2- 172.21.15.27:0/894118951 >> v2:172.21.15.93:3300/0 conn(0x7f266c06bb80 0x7f266c0cd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:22.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.602+0000 7f2673200640 1 --2- 172.21.15.27:0/894118951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f266c0b2390 0x7f266c0cf780 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:22.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.602+0000 7f2673200640 1 -- 172.21.15.27:0/894118951 >> 172.21.15.27:0/894118951 conn(0x7f266c0b2a10 msgr2=0x7f266c0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:22.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.606+0000 7f2673200640 1 -- 172.21.15.27:0/894118951 shutdown_connections 2024-01-15T18:17:22.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.606+0000 7f2673200640 1 -- 172.21.15.27:0/894118951 wait complete. 2024-01-15T18:17:22.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.606+0000 7f2673200640 1 Processor -- start 2024-01-15T18:17:22.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.606+0000 7f2673200640 1 -- start start 2024-01-15T18:17:22.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.606+0000 7f2673200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f266c06bb80 0x7f266c0b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:22.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.606+0000 7f2673200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f266c0b2390 0x7f266c159be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:22.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.606+0000 7f2673200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f266c0cfcc0 0x7f266c15a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:22.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.606+0000 7f2673200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c1556b0 con 0x7f266c06bb80 2024-01-15T18:17:22.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.606+0000 7f2673200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c15a660 con 0x7f266c0b2390 2024-01-15T18:17:22.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.606+0000 7f2673200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f266c154a30 con 0x7f266c0cfcc0 2024-01-15T18:17:22.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.606+0000 7f266be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f266c06bb80 0x7f266c0b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:22.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.606+0000 7f266be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f266c06bb80 0x7f266c0b61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59758/0 (socket says 172.21.15.27:59758) 2024-01-15T18:17:22.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.606+0000 7f266be00640 1 -- 172.21.15.27:0/3735293516 learned_addr learned my addr 172.21.15.27:0/3735293516 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:22.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.606+0000 7f2671200640 1 --2- 172.21.15.27:0/3735293516 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f266c0cfcc0 0x7f266c15a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:22.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.606+0000 7f266b400640 1 --2- 172.21.15.27:0/3735293516 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f266c0b2390 0x7f266c159be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:22.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.606+0000 7f266be00640 1 -- 172.21.15.27:0/3735293516 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f266c0cfcc0 msgr2=0x7f266c15a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:22.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.606+0000 7f266be00640 1 --2- 172.21.15.27:0/3735293516 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f266c0cfcc0 0x7f266c15a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:22.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.606+0000 7f266be00640 1 -- 172.21.15.27:0/3735293516 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f266c0b2390 msgr2=0x7f266c159be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:22.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.606+0000 7f266be00640 1 --2- 172.21.15.27:0/3735293516 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f266c0b2390 0x7f266c159be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:22.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.606+0000 7f266be00640 1 -- 172.21.15.27:0/3735293516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f265c0094d0 con 0x7f266c06bb80 2024-01-15T18:17:22.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.606+0000 7f266be00640 1 --2- 172.21.15.27:0/3735293516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f266c06bb80 0x7f266c0b61e0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f265c01a0c0 tx=0x7f265c01acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:22.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.606+0000 7f2663e00640 1 -- 172.21.15.27:0/3735293516 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265c0b3800 con 0x7f266c06bb80 2024-01-15T18:17:22.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.610+0000 7f2663e00640 1 -- 172.21.15.27:0/3735293516 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f265c0b3980 con 0x7f266c06bb80 2024-01-15T18:17:22.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.610+0000 7f2663e00640 1 -- 172.21.15.27:0/3735293516 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265c0b3d30 con 0x7f266c06bb80 2024-01-15T18:17:22.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.610+0000 7f2673200640 1 -- 172.21.15.27:0/3735293516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f266c154d10 con 0x7f266c06bb80 2024-01-15T18:17:22.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.610+0000 7f2673200640 1 -- 172.21.15.27:0/3735293516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f266c1552d0 con 0x7f266c06bb80 2024-01-15T18:17:22.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.610+0000 7f2673200640 1 -- 172.21.15.27:0/3735293516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f266c0d07c0 con 0x7f266c06bb80 2024-01-15T18:17:22.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.610+0000 7f2663e00640 1 -- 172.21.15.27:0/3735293516 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f265c0c6440 con 0x7f266c06bb80 2024-01-15T18:17:22.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.610+0000 7f2663e00640 1 --2- 172.21.15.27:0/3735293516 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2644067700 0x7f2644069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:22.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.610+0000 7f2663e00640 1 -- 172.21.15.27:0/3735293516 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f265c0a7070 con 0x7f266c06bb80 2024-01-15T18:17:22.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.610+0000 7f266b400640 1 --2- 172.21.15.27:0/3735293516 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2644067700 0x7f2644069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:22.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.614+0000 7f266b400640 1 --2- 172.21.15.27:0/3735293516 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2644067700 0x7f2644069bc0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f266c09ef80 tx=0x7f2654008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:22.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.614+0000 7f2663e00640 1 -- 172.21.15.27:0/3735293516 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f265c0ec6d0 con 0x7f266c06bb80 2024-01-15T18:17:22.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.770+0000 7f2673200640 1 -- 172.21.15.27:0/3735293516 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f266c0ce1d0 con 0x7f2644067700 2024-01-15T18:17:22.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.778+0000 7f2663e00640 1 -- 172.21.15.27:0/3735293516 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288781 (secure 0 0 0) 0x7f26540108d0 con 0x7f2644067700 2024-01-15T18:17:22.782 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:17:22.784 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:17:22.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.782+0000 7f2673200640 1 -- 172.21.15.27:0/3735293516 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2644067700 msgr2=0x7f2644069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:22.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.782+0000 7f2673200640 1 --2- 172.21.15.27:0/3735293516 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2644067700 0x7f2644069bc0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f266c09ef80 tx=0x7f2654008040 comp rx=0 tx=0).stop 2024-01-15T18:17:22.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.782+0000 7f2673200640 1 -- 172.21.15.27:0/3735293516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f266c06bb80 msgr2=0x7f266c0b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:22.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.782+0000 7f2673200640 1 --2- 172.21.15.27:0/3735293516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f266c06bb80 0x7f266c0b61e0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f265c01a0c0 tx=0x7f265c01acd0 comp rx=0 tx=0).stop 2024-01-15T18:17:22.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.786+0000 7f2673200640 1 -- 172.21.15.27:0/3735293516 shutdown_connections 2024-01-15T18:17:22.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.786+0000 7f2673200640 1 --2- 172.21.15.27:0/3735293516 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2644067700 0x7f2644069bc0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:22.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.786+0000 7f2673200640 1 --2- 172.21.15.27:0/3735293516 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f266c0cfcc0 0x7f266c15a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:22.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.786+0000 7f2673200640 1 --2- 172.21.15.27:0/3735293516 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f266c0b2390 0x7f266c159be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:22.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.786+0000 7f2673200640 1 --2- 172.21.15.27:0/3735293516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f266c06bb80 0x7f266c0b61e0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:22.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.786+0000 7f2673200640 1 -- 172.21.15.27:0/3735293516 >> 172.21.15.27:0/3735293516 conn(0x7f266c0b2a10 msgr2=0x7f266c0b0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:22.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.786+0000 7f2673200640 1 -- 172.21.15.27:0/3735293516 shutdown_connections 2024-01-15T18:17:22.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:22.786+0000 7f2673200640 1 -- 172.21.15.27:0/3735293516 wait complete. 2024-01-15T18:17:22.805 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":469,"stamp":"2024-01-15T18:17:22.355155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":188,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":526120,"kb_used_data":10536,"kb_used_omap":12,"kb_used_meta":515443,"kb_avail":754448600,"statfs":{"total":773094113280,"available":772555366400,"internally_reserved":0,"allocated":10788864,"data_stored":5327948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":527814192},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.263290"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.127754+0000","last_change":"2024-01-15T18:17:17.134979+0000","last_active":"2024-01-15T18:17:18.127754+0000","last_peered":"2024-01-15T18:17:18.127754+0000","last_clean":"2024-01-15T18:17:18.127754+0000","last_became_active":"2024-01-15T18:17:17.134592+0000","last_became_peered":"2024-01-15T18:17:17.134592+0000","last_unstale":"2024-01-15T18:17:18.127754+0000","last_undegraded":"2024-01-15T18:17:18.127754+0000","last_fullsized":"2024-01-15T18:17:18.127754+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:02:20.596146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440653+0000","last_change":"2024-01-15T18:17:17.132525+0000","last_active":"2024-01-15T18:17:18.440653+0000","last_peered":"2024-01-15T18:17:18.440653+0000","last_clean":"2024-01-15T18:17:18.440653+0000","last_became_active":"2024-01-15T18:17:17.132261+0000","last_became_peered":"2024-01-15T18:17:17.132261+0000","last_unstale":"2024-01-15T18:17:18.440653+0000","last_undegraded":"2024-01-15T18:17:18.440653+0000","last_fullsized":"2024-01-15T18:17:18.440653+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:07.042399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.127694+0000","last_change":"2024-01-15T18:17:17.132080+0000","last_active":"2024-01-15T18:17:18.127694+0000","last_peered":"2024-01-15T18:17:18.127694+0000","last_clean":"2024-01-15T18:17:18.127694+0000","last_became_active":"2024-01-15T18:17:17.131696+0000","last_became_peered":"2024-01-15T18:17:17.131696+0000","last_unstale":"2024-01-15T18:17:18.127694+0000","last_undegraded":"2024-01-15T18:17:18.127694+0000","last_fullsized":"2024-01-15T18:17:18.127694+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:26:02.276866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439558+0000","last_change":"2024-01-15T18:17:17.123236+0000","last_active":"2024-01-15T18:17:18.439558+0000","last_peered":"2024-01-15T18:17:18.439558+0000","last_clean":"2024-01-15T18:17:18.439558+0000","last_became_active":"2024-01-15T18:17:17.122924+0000","last_became_peered":"2024-01-15T18:17:17.122924+0000","last_unstale":"2024-01-15T18:17:18.439558+0000","last_undegraded":"2024-01-15T18:17:18.439558+0000","last_fullsized":"2024-01-15T18:17:18.439558+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:30:09.436810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439733+0000","last_change":"2024-01-15T18:17:17.129419+0000","last_active":"2024-01-15T18:17:18.439733+0000","last_peered":"2024-01-15T18:17:18.439733+0000","last_clean":"2024-01-15T18:17:18.439733+0000","last_became_active":"2024-01-15T18:17:17.128570+0000","last_became_peered":"2024-01-15T18:17:17.128570+0000","last_unstale":"2024-01-15T18:17:18.439733+0000","last_undegraded":"2024-01-15T18:17:18.439733+0000","last_fullsized":"2024-01-15T18:17:18.439733+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:39:09.922990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439439+0000","last_change":"2024-01-15T18:17:17.129884+0000","last_active":"2024-01-15T18:17:18.439439+0000","last_peered":"2024-01-15T18:17:18.439439+0000","last_clean":"2024-01-15T18:17:18.439439+0000","last_became_active":"2024-01-15T18:17:17.129579+0000","last_became_peered":"2024-01-15T18:17:17.129579+0000","last_unstale":"2024-01-15T18:17:18.439439+0000","last_undegraded":"2024-01-15T18:17:18.439439+0000","last_fullsized":"2024-01-15T18:17:18.439439+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:07:30.777445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439255+0000","last_change":"2024-01-15T18:17:17.126726+0000","last_active":"2024-01-15T18:17:18.439255+0000","last_peered":"2024-01-15T18:17:18.439255+0000","last_clean":"2024-01-15T18:17:18.439255+0000","last_became_active":"2024-01-15T18:17:17.126431+0000","last_became_peered":"2024-01-15T18:17:17.126431+0000","last_unstale":"2024-01-15T18:17:18.439255+0000","last_undegraded":"2024-01-15T18:17:18.439255+0000","last_fullsized":"2024-01-15T18:17:18.439255+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:58:11.451418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439412+0000","last_change":"2024-01-15T18:17:17.129139+0000","last_active":"2024-01-15T18:17:18.439412+0000","last_peered":"2024-01-15T18:17:18.439412+0000","last_clean":"2024-01-15T18:17:18.439412+0000","last_became_active":"2024-01-15T18:17:17.126922+0000","last_became_peered":"2024-01-15T18:17:17.126922+0000","last_unstale":"2024-01-15T18:17:18.439412+0000","last_undegraded":"2024-01-15T18:17:18.439412+0000","last_fullsized":"2024-01-15T18:17:18.439412+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:40:32.654203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.132988+0000","last_change":"2024-01-15T18:17:17.117455+0000","last_active":"2024-01-15T18:17:18.132988+0000","last_peered":"2024-01-15T18:17:18.132988+0000","last_clean":"2024-01-15T18:17:18.132988+0000","last_became_active":"2024-01-15T18:17:17.117259+0000","last_became_peered":"2024-01-15T18:17:17.117259+0000","last_unstale":"2024-01-15T18:17:18.132988+0000","last_undegraded":"2024-01-15T18:17:18.132988+0000","last_fullsized":"2024-01-15T18:17:18.132988+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:23.944086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439695+0000","last_change":"2024-01-15T18:17:17.132515+0000","last_active":"2024-01-15T18:17:18.439695+0000","last_peered":"2024-01-15T18:17:18.439695+0000","last_clean":"2024-01-15T18:17:18.439695+0000","last_became_active":"2024-01-15T18:17:17.130433+0000","last_became_peered":"2024-01-15T18:17:17.130433+0000","last_unstale":"2024-01-15T18:17:18.439695+0000","last_undegraded":"2024-01-15T18:17:18.439695+0000","last_fullsized":"2024-01-15T18:17:18.439695+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:23.944086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439546+0000","last_change":"2024-01-15T18:17:17.134352+0000","last_active":"2024-01-15T18:17:18.439546+0000","last_peered":"2024-01-15T18:17:18.439546+0000","last_clean":"2024-01-15T18:17:18.439546+0000","last_became_active":"2024-01-15T18:17:17.134076+0000","last_became_peered":"2024-01-15T18:17:17.134076+0000","last_unstale":"2024-01-15T18:17:18.439546+0000","last_undegraded":"2024-01-15T18:17:18.439546+0000","last_fullsized":"2024-01-15T18:17:18.439546+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:59:53.121905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.132977+0000","last_change":"2024-01-15T18:17:17.126892+0000","last_active":"2024-01-15T18:17:18.132977+0000","last_peered":"2024-01-15T18:17:18.132977+0000","last_clean":"2024-01-15T18:17:18.132977+0000","last_became_active":"2024-01-15T18:17:17.125684+0000","last_became_peered":"2024-01-15T18:17:17.125684+0000","last_unstale":"2024-01-15T18:17:18.132977+0000","last_undegraded":"2024-01-15T18:17:18.132977+0000","last_fullsized":"2024-01-15T18:17:18.132977+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:54.730338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439385+0000","last_change":"2024-01-15T18:17:17.125419+0000","last_active":"2024-01-15T18:17:18.439385+0000","last_peered":"2024-01-15T18:17:18.439385+0000","last_clean":"2024-01-15T18:17:18.439385+0000","last_became_active":"2024-01-15T18:17:17.125149+0000","last_became_peered":"2024-01-15T18:17:17.125149+0000","last_unstale":"2024-01-15T18:17:18.439385+0000","last_undegraded":"2024-01-15T18:17:18.439385+0000","last_fullsized":"2024-01-15T18:17:18.439385+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:03.728057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439288+0000","last_change":"2024-01-15T18:17:17.124265+0000","last_active":"2024-01-15T18:17:18.439288+0000","last_peered":"2024-01-15T18:17:18.439288+0000","last_clean":"2024-01-15T18:17:18.439288+0000","last_became_active":"2024-01-15T18:17:17.123974+0000","last_became_peered":"2024-01-15T18:17:17.123974+0000","last_unstale":"2024-01-15T18:17:18.439288+0000","last_undegraded":"2024-01-15T18:17:18.439288+0000","last_fullsized":"2024-01-15T18:17:18.439288+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:56:04.498774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440333+0000","last_change":"2024-01-15T18:17:17.125411+0000","last_active":"2024-01-15T18:17:18.440333+0000","last_peered":"2024-01-15T18:17:18.440333+0000","last_clean":"2024-01-15T18:17:18.440333+0000","last_became_active":"2024-01-15T18:17:17.122916+0000","last_became_peered":"2024-01-15T18:17:17.122916+0000","last_unstale":"2024-01-15T18:17:18.440333+0000","last_undegraded":"2024-01-15T18:17:18.440333+0000","last_fullsized":"2024-01-15T18:17:18.440333+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:47:49.952814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439054+0000","last_change":"2024-01-15T18:17:17.133691+0000","last_active":"2024-01-15T18:17:18.439054+0000","last_peered":"2024-01-15T18:17:18.439054+0000","last_clean":"2024-01-15T18:17:18.439054+0000","last_became_active":"2024-01-15T18:17:17.132756+0000","last_became_peered":"2024-01-15T18:17:17.132756+0000","last_unstale":"2024-01-15T18:17:18.439054+0000","last_undegraded":"2024-01-15T18:17:18.439054+0000","last_fullsized":"2024-01-15T18:17:18.439054+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:11:55.311668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440809+0000","last_change":"2024-01-15T18:17:17.125211+0000","last_active":"2024-01-15T18:17:18.440809+0000","last_peered":"2024-01-15T18:17:18.440809+0000","last_clean":"2024-01-15T18:17:18.440809+0000","last_became_active":"2024-01-15T18:17:17.119327+0000","last_became_peered":"2024-01-15T18:17:17.119327+0000","last_unstale":"2024-01-15T18:17:18.440809+0000","last_undegraded":"2024-01-15T18:17:18.440809+0000","last_fullsized":"2024-01-15T18:17:18.440809+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:57:35.828970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439426+0000","last_change":"2024-01-15T18:17:17.122901+0000","last_active":"2024-01-15T18:17:18.439426+0000","last_peered":"2024-01-15T18:17:18.439426+0000","last_clean":"2024-01-15T18:17:18.439426+0000","last_became_active":"2024-01-15T18:17:17.122470+0000","last_became_peered":"2024-01-15T18:17:17.122470+0000","last_unstale":"2024-01-15T18:17:18.439426+0000","last_undegraded":"2024-01-15T18:17:18.439426+0000","last_fullsized":"2024-01-15T18:17:18.439426+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:03:06.032027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440565+0000","last_change":"2024-01-15T18:17:17.127686+0000","last_active":"2024-01-15T18:17:18.440565+0000","last_peered":"2024-01-15T18:17:18.440565+0000","last_clean":"2024-01-15T18:17:18.440565+0000","last_became_active":"2024-01-15T18:17:17.127251+0000","last_became_peered":"2024-01-15T18:17:17.127251+0000","last_unstale":"2024-01-15T18:17:18.440565+0000","last_undegraded":"2024-01-15T18:17:18.440565+0000","last_fullsized":"2024-01-15T18:17:18.440565+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:28:31.116794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.133012+0000","last_change":"2024-01-15T18:17:17.117786+0000","last_active":"2024-01-15T18:17:18.133012+0000","last_peered":"2024-01-15T18:17:18.133012+0000","last_clean":"2024-01-15T18:17:18.133012+0000","last_became_active":"2024-01-15T18:17:17.117505+0000","last_became_peered":"2024-01-15T18:17:17.117505+0000","last_unstale":"2024-01-15T18:17:18.133012+0000","last_undegraded":"2024-01-15T18:17:18.133012+0000","last_fullsized":"2024-01-15T18:17:18.133012+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:39:09.922990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439938+0000","last_change":"2024-01-15T18:17:17.127902+0000","last_active":"2024-01-15T18:17:18.439938+0000","last_peered":"2024-01-15T18:17:18.439938+0000","last_clean":"2024-01-15T18:17:18.439938+0000","last_became_active":"2024-01-15T18:17:17.127618+0000","last_became_peered":"2024-01-15T18:17:17.127618+0000","last_unstale":"2024-01-15T18:17:18.439938+0000","last_undegraded":"2024-01-15T18:17:18.439938+0000","last_fullsized":"2024-01-15T18:17:18.439938+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:09.950411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440211+0000","last_change":"2024-01-15T18:17:17.125005+0000","last_active":"2024-01-15T18:17:18.440211+0000","last_peered":"2024-01-15T18:17:18.440211+0000","last_clean":"2024-01-15T18:17:18.440211+0000","last_became_active":"2024-01-15T18:17:17.124717+0000","last_became_peered":"2024-01-15T18:17:17.124717+0000","last_unstale":"2024-01-15T18:17:18.440211+0000","last_undegraded":"2024-01-15T18:17:18.440211+0000","last_fullsized":"2024-01-15T18:17:18.440211+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:48:30.181901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440100+0000","last_change":"2024-01-15T18:17:17.130054+0000","last_active":"2024-01-15T18:17:18.440100+0000","last_peered":"2024-01-15T18:17:18.440100+0000","last_clean":"2024-01-15T18:17:18.440100+0000","last_became_active":"2024-01-15T18:17:17.129779+0000","last_became_peered":"2024-01-15T18:17:17.129779+0000","last_unstale":"2024-01-15T18:17:18.440100+0000","last_undegraded":"2024-01-15T18:17:18.440100+0000","last_fullsized":"2024-01-15T18:17:18.440100+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:42:04.787652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439325+0000","last_change":"2024-01-15T18:17:17.137320+0000","last_active":"2024-01-15T18:17:18.439325+0000","last_peered":"2024-01-15T18:17:18.439325+0000","last_clean":"2024-01-15T18:17:18.439325+0000","last_became_active":"2024-01-15T18:17:17.137071+0000","last_became_peered":"2024-01-15T18:17:17.137071+0000","last_unstale":"2024-01-15T18:17:18.439325+0000","last_undegraded":"2024-01-15T18:17:18.439325+0000","last_fullsized":"2024-01-15T18:17:18.439325+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:56:59.821700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439816+0000","last_change":"2024-01-15T18:17:17.137076+0000","last_active":"2024-01-15T18:17:18.439816+0000","last_peered":"2024-01-15T18:17:18.439816+0000","last_clean":"2024-01-15T18:17:18.439816+0000","last_became_active":"2024-01-15T18:17:17.136800+0000","last_became_peered":"2024-01-15T18:17:17.136800+0000","last_unstale":"2024-01-15T18:17:18.439816+0000","last_undegraded":"2024-01-15T18:17:18.439816+0000","last_fullsized":"2024-01-15T18:17:18.439816+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:27:59.524326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440698+0000","last_change":"2024-01-15T18:17:17.132156+0000","last_active":"2024-01-15T18:17:18.440698+0000","last_peered":"2024-01-15T18:17:18.440698+0000","last_clean":"2024-01-15T18:17:18.440698+0000","last_became_active":"2024-01-15T18:17:17.131822+0000","last_became_peered":"2024-01-15T18:17:17.131822+0000","last_unstale":"2024-01-15T18:17:18.440698+0000","last_undegraded":"2024-01-15T18:17:18.440698+0000","last_fullsized":"2024-01-15T18:17:18.440698+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:33:59.696616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439694+0000","last_change":"2024-01-15T18:17:17.136720+0000","last_active":"2024-01-15T18:17:18.439694+0000","last_peered":"2024-01-15T18:17:18.439694+0000","last_clean":"2024-01-15T18:17:18.439694+0000","last_became_active":"2024-01-15T18:17:17.135986+0000","last_became_peered":"2024-01-15T18:17:17.135986+0000","last_unstale":"2024-01-15T18:17:18.439694+0000","last_undegraded":"2024-01-15T18:17:18.439694+0000","last_fullsized":"2024-01-15T18:17:18.439694+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:52:52.621767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440073+0000","last_change":"2024-01-15T18:17:17.129422+0000","last_active":"2024-01-15T18:17:18.440073+0000","last_peered":"2024-01-15T18:17:18.440073+0000","last_clean":"2024-01-15T18:17:18.440073+0000","last_became_active":"2024-01-15T18:17:17.129128+0000","last_became_peered":"2024-01-15T18:17:17.129128+0000","last_unstale":"2024-01-15T18:17:18.440073+0000","last_undegraded":"2024-01-15T18:17:18.440073+0000","last_fullsized":"2024-01-15T18:17:18.440073+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:47:09.879357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440705+0000","last_change":"2024-01-15T18:17:17.127796+0000","last_active":"2024-01-15T18:17:18.440705+0000","last_peered":"2024-01-15T18:17:18.440705+0000","last_clean":"2024-01-15T18:17:18.440705+0000","last_became_active":"2024-01-15T18:17:17.127390+0000","last_became_peered":"2024-01-15T18:17:17.127390+0000","last_unstale":"2024-01-15T18:17:18.440705+0000","last_undegraded":"2024-01-15T18:17:18.440705+0000","last_fullsized":"2024-01-15T18:17:18.440705+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:03:11.357766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439414+0000","last_change":"2024-01-15T18:17:17.120389+0000","last_active":"2024-01-15T18:17:18.439414+0000","last_peered":"2024-01-15T18:17:18.439414+0000","last_clean":"2024-01-15T18:17:18.439414+0000","last_became_active":"2024-01-15T18:17:17.119695+0000","last_became_peered":"2024-01-15T18:17:17.119695+0000","last_unstale":"2024-01-15T18:17:18.439414+0000","last_undegraded":"2024-01-15T18:17:18.439414+0000","last_fullsized":"2024-01-15T18:17:18.439414+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:22:00.599199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440173+0000","last_change":"2024-01-15T18:17:17.130622+0000","last_active":"2024-01-15T18:17:18.440173+0000","last_peered":"2024-01-15T18:17:18.440173+0000","last_clean":"2024-01-15T18:17:18.440173+0000","last_became_active":"2024-01-15T18:17:17.130348+0000","last_became_peered":"2024-01-15T18:17:17.130348+0000","last_unstale":"2024-01-15T18:17:18.440173+0000","last_undegraded":"2024-01-15T18:17:18.440173+0000","last_fullsized":"2024-01-15T18:17:18.440173+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:41:24.467489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.127304+0000","last_change":"2024-01-15T18:17:17.136784+0000","last_active":"2024-01-15T18:17:18.127304+0000","last_peered":"2024-01-15T18:17:18.127304+0000","last_clean":"2024-01-15T18:17:18.127304+0000","last_became_active":"2024-01-15T18:17:17.136510+0000","last_became_peered":"2024-01-15T18:17:17.136510+0000","last_unstale":"2024-01-15T18:17:18.127304+0000","last_undegraded":"2024-01-15T18:17:18.127304+0000","last_fullsized":"2024-01-15T18:17:18.127304+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:45:46.354374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440893+0000","last_change":"2024-01-15T18:17:17.128149+0000","last_active":"2024-01-15T18:17:18.440893+0000","last_peered":"2024-01-15T18:17:18.440893+0000","last_clean":"2024-01-15T18:17:18.440893+0000","last_became_active":"2024-01-15T18:17:17.124768+0000","last_became_peered":"2024-01-15T18:17:17.124768+0000","last_unstale":"2024-01-15T18:17:18.440893+0000","last_undegraded":"2024-01-15T18:17:18.440893+0000","last_fullsized":"2024-01-15T18:17:18.440893+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:28:08.446574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"96'4","reported_seq":16,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440784+0000","last_change":"2024-01-15T18:17:16.117003+0000","last_active":"2024-01-15T18:17:18.440784+0000","last_peered":"2024-01-15T18:17:18.440784+0000","last_clean":"2024-01-15T18:17:18.440784+0000","last_became_active":"2024-01-15T18:17:16.116625+0000","last_became_peered":"2024-01-15T18:17:16.116625+0000","last_unstale":"2024-01-15T18:17:18.440784+0000","last_undegraded":"2024-01-15T18:17:18.440784+0000","last_fullsized":"2024-01-15T18:17:18.440784+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:11:58.418627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439809+0000","last_change":"2024-01-15T18:17:17.126785+0000","last_active":"2024-01-15T18:17:18.439809+0000","last_peered":"2024-01-15T18:17:18.439809+0000","last_clean":"2024-01-15T18:17:18.439809+0000","last_became_active":"2024-01-15T18:17:17.126456+0000","last_became_peered":"2024-01-15T18:17:17.126456+0000","last_unstale":"2024-01-15T18:17:18.439809+0000","last_undegraded":"2024-01-15T18:17:18.439809+0000","last_fullsized":"2024-01-15T18:17:18.439809+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:36:57.592354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"96'1","reported_seq":13,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.142321+0000","last_change":"2024-01-15T18:17:16.120557+0000","last_active":"2024-01-15T18:17:17.142321+0000","last_peered":"2024-01-15T18:17:17.142321+0000","last_clean":"2024-01-15T18:17:17.142321+0000","last_became_active":"2024-01-15T18:17:16.120207+0000","last_became_peered":"2024-01-15T18:17:16.120207+0000","last_unstale":"2024-01-15T18:17:17.142321+0000","last_undegraded":"2024-01-15T18:17:17.142321+0000","last_fullsized":"2024-01-15T18:17:17.142321+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:39:08.914733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.127613+0000","last_change":"2024-01-15T18:17:17.125614+0000","last_active":"2024-01-15T18:17:18.127613+0000","last_peered":"2024-01-15T18:17:18.127613+0000","last_clean":"2024-01-15T18:17:18.127613+0000","last_became_active":"2024-01-15T18:17:17.125330+0000","last_became_peered":"2024-01-15T18:17:17.125330+0000","last_unstale":"2024-01-15T18:17:18.127613+0000","last_undegraded":"2024-01-15T18:17:18.127613+0000","last_fullsized":"2024-01-15T18:17:18.127613+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:13:27.502386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440270+0000","last_change":"2024-01-15T18:17:16.115473+0000","last_active":"2024-01-15T18:17:18.440270+0000","last_peered":"2024-01-15T18:17:18.440270+0000","last_clean":"2024-01-15T18:17:18.440270+0000","last_became_active":"2024-01-15T18:17:16.115189+0000","last_became_peered":"2024-01-15T18:17:16.115189+0000","last_unstale":"2024-01-15T18:17:18.440270+0000","last_undegraded":"2024-01-15T18:17:18.440270+0000","last_fullsized":"2024-01-15T18:17:18.440270+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:58:47.776409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.133184+0000","last_change":"2024-01-15T18:17:17.127577+0000","last_active":"2024-01-15T18:17:18.133184+0000","last_peered":"2024-01-15T18:17:18.133184+0000","last_clean":"2024-01-15T18:17:18.133184+0000","last_became_active":"2024-01-15T18:17:17.125488+0000","last_became_peered":"2024-01-15T18:17:17.125488+0000","last_unstale":"2024-01-15T18:17:18.133184+0000","last_undegraded":"2024-01-15T18:17:18.133184+0000","last_fullsized":"2024-01-15T18:17:18.133184+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:23.103323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440326+0000","last_change":"2024-01-15T18:17:16.123895+0000","last_active":"2024-01-15T18:17:18.440326+0000","last_peered":"2024-01-15T18:17:18.440326+0000","last_clean":"2024-01-15T18:17:18.440326+0000","last_became_active":"2024-01-15T18:17:16.123325+0000","last_became_peered":"2024-01-15T18:17:16.123325+0000","last_unstale":"2024-01-15T18:17:18.440326+0000","last_undegraded":"2024-01-15T18:17:18.440326+0000","last_fullsized":"2024-01-15T18:17:18.440326+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:13:09.312116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.131438+0000","last_change":"2024-01-15T18:17:17.118290+0000","last_active":"2024-01-15T18:17:18.131438+0000","last_peered":"2024-01-15T18:17:18.131438+0000","last_clean":"2024-01-15T18:17:18.131438+0000","last_became_active":"2024-01-15T18:17:17.117943+0000","last_became_peered":"2024-01-15T18:17:17.117943+0000","last_unstale":"2024-01-15T18:17:18.131438+0000","last_undegraded":"2024-01-15T18:17:18.131438+0000","last_fullsized":"2024-01-15T18:17:18.131438+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:50:04.755553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"96'1","reported_seq":13,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.144016+0000","last_change":"2024-01-15T18:17:16.115057+0000","last_active":"2024-01-15T18:17:17.144016+0000","last_peered":"2024-01-15T18:17:17.144016+0000","last_clean":"2024-01-15T18:17:17.144016+0000","last_became_active":"2024-01-15T18:17:16.114788+0000","last_became_peered":"2024-01-15T18:17:16.114788+0000","last_unstale":"2024-01-15T18:17:17.144016+0000","last_undegraded":"2024-01-15T18:17:17.144016+0000","last_fullsized":"2024-01-15T18:17:17.144016+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:50:19.871865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439140+0000","last_change":"2024-01-15T18:17:17.127229+0000","last_active":"2024-01-15T18:17:18.439140+0000","last_peered":"2024-01-15T18:17:18.439140+0000","last_clean":"2024-01-15T18:17:18.439140+0000","last_became_active":"2024-01-15T18:17:17.126951+0000","last_became_peered":"2024-01-15T18:17:17.126951+0000","last_unstale":"2024-01-15T18:17:18.439140+0000","last_undegraded":"2024-01-15T18:17:18.439140+0000","last_fullsized":"2024-01-15T18:17:18.439140+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:34:39.562336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"96'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440314+0000","last_change":"2024-01-15T18:17:16.121483+0000","last_active":"2024-01-15T18:17:18.440314+0000","last_peered":"2024-01-15T18:17:18.440314+0000","last_clean":"2024-01-15T18:17:18.440314+0000","last_became_active":"2024-01-15T18:17:16.121199+0000","last_became_peered":"2024-01-15T18:17:16.121199+0000","last_unstale":"2024-01-15T18:17:18.440314+0000","last_undegraded":"2024-01-15T18:17:18.440314+0000","last_fullsized":"2024-01-15T18:17:18.440314+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:51:29.086954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439898+0000","last_change":"2024-01-15T18:17:17.121762+0000","last_active":"2024-01-15T18:17:18.439898+0000","last_peered":"2024-01-15T18:17:18.439898+0000","last_clean":"2024-01-15T18:17:18.439898+0000","last_became_active":"2024-01-15T18:17:17.118184+0000","last_became_peered":"2024-01-15T18:17:17.118184+0000","last_unstale":"2024-01-15T18:17:18.439898+0000","last_undegraded":"2024-01-15T18:17:18.439898+0000","last_fullsized":"2024-01-15T18:17:18.439898+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:15.329122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.118257+0000","last_change":"2024-01-15T18:17:16.120590+0000","last_active":"2024-01-15T18:17:17.118257+0000","last_peered":"2024-01-15T18:17:17.118257+0000","last_clean":"2024-01-15T18:17:17.118257+0000","last_became_active":"2024-01-15T18:17:16.120243+0000","last_became_peered":"2024-01-15T18:17:16.120243+0000","last_unstale":"2024-01-15T18:17:17.118257+0000","last_undegraded":"2024-01-15T18:17:17.118257+0000","last_fullsized":"2024-01-15T18:17:17.118257+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:36:56.584097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.132487+0000","last_change":"2024-01-15T18:17:17.119962+0000","last_active":"2024-01-15T18:17:18.132487+0000","last_peered":"2024-01-15T18:17:18.132487+0000","last_clean":"2024-01-15T18:17:18.132487+0000","last_became_active":"2024-01-15T18:17:17.118612+0000","last_became_peered":"2024-01-15T18:17:17.118612+0000","last_unstale":"2024-01-15T18:17:18.132487+0000","last_undegraded":"2024-01-15T18:17:18.132487+0000","last_fullsized":"2024-01-15T18:17:18.132487+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:45:26.435028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.117729+0000","last_change":"2024-01-15T18:17:16.117768+0000","last_active":"2024-01-15T18:17:17.117729+0000","last_peered":"2024-01-15T18:17:17.117729+0000","last_clean":"2024-01-15T18:17:17.117729+0000","last_became_active":"2024-01-15T18:17:16.117492+0000","last_became_peered":"2024-01-15T18:17:16.117492+0000","last_unstale":"2024-01-15T18:17:17.117729+0000","last_undegraded":"2024-01-15T18:17:17.117729+0000","last_fullsized":"2024-01-15T18:17:17.117729+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:06:44.491748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440790+0000","last_change":"2024-01-15T18:17:17.124377+0000","last_active":"2024-01-15T18:17:18.440790+0000","last_peered":"2024-01-15T18:17:18.440790+0000","last_clean":"2024-01-15T18:17:18.440790+0000","last_became_active":"2024-01-15T18:17:17.124096+0000","last_became_peered":"2024-01-15T18:17:17.124096+0000","last_unstale":"2024-01-15T18:17:18.440790+0000","last_undegraded":"2024-01-15T18:17:18.440790+0000","last_fullsized":"2024-01-15T18:17:18.440790+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:56:21.720071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.122307+0000","last_change":"2024-01-15T18:17:16.102263+0000","last_active":"2024-01-15T18:17:17.122307+0000","last_peered":"2024-01-15T18:17:17.122307+0000","last_clean":"2024-01-15T18:17:17.122307+0000","last_became_active":"2024-01-15T18:17:16.101797+0000","last_became_peered":"2024-01-15T18:17:16.101797+0000","last_unstale":"2024-01-15T18:17:17.122307+0000","last_undegraded":"2024-01-15T18:17:17.122307+0000","last_fullsized":"2024-01-15T18:17:17.122307+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:36:25.432888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.133177+0000","last_change":"2024-01-15T18:17:17.122838+0000","last_active":"2024-01-15T18:17:18.133177+0000","last_peered":"2024-01-15T18:17:18.133177+0000","last_clean":"2024-01-15T18:17:18.133177+0000","last_became_active":"2024-01-15T18:17:17.122579+0000","last_became_peered":"2024-01-15T18:17:17.122579+0000","last_unstale":"2024-01-15T18:17:18.133177+0000","last_undegraded":"2024-01-15T18:17:18.133177+0000","last_fullsized":"2024-01-15T18:17:18.133177+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:13:53.357957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.113153+0000","last_change":"2024-01-15T18:17:16.122691+0000","last_active":"2024-01-15T18:17:17.113153+0000","last_peered":"2024-01-15T18:17:17.113153+0000","last_clean":"2024-01-15T18:17:17.113153+0000","last_became_active":"2024-01-15T18:17:16.122426+0000","last_became_peered":"2024-01-15T18:17:16.122426+0000","last_unstale":"2024-01-15T18:17:17.113153+0000","last_undegraded":"2024-01-15T18:17:17.113153+0000","last_fullsized":"2024-01-15T18:17:17.113153+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:37:13.270621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440113+0000","last_change":"2024-01-15T18:17:17.129556+0000","last_active":"2024-01-15T18:17:18.440113+0000","last_peered":"2024-01-15T18:17:18.440113+0000","last_clean":"2024-01-15T18:17:18.440113+0000","last_became_active":"2024-01-15T18:17:17.129237+0000","last_became_peered":"2024-01-15T18:17:17.129237+0000","last_unstale":"2024-01-15T18:17:18.440113+0000","last_undegraded":"2024-01-15T18:17:18.440113+0000","last_fullsized":"2024-01-15T18:17:18.440113+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:15:43.895396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"96'3","reported_seq":15,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.142974+0000","last_change":"2024-01-15T18:17:16.114433+0000","last_active":"2024-01-15T18:17:17.142974+0000","last_peered":"2024-01-15T18:17:17.142974+0000","last_clean":"2024-01-15T18:17:17.142974+0000","last_became_active":"2024-01-15T18:17:16.113351+0000","last_became_peered":"2024-01-15T18:17:16.113351+0000","last_unstale":"2024-01-15T18:17:17.142974+0000","last_undegraded":"2024-01-15T18:17:17.142974+0000","last_fullsized":"2024-01-15T18:17:17.142974+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:47:08.871100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439693+0000","last_change":"2024-01-15T18:17:17.140004+0000","last_active":"2024-01-15T18:17:18.439693+0000","last_peered":"2024-01-15T18:17:18.439693+0000","last_clean":"2024-01-15T18:17:18.439693+0000","last_became_active":"2024-01-15T18:17:17.139718+0000","last_became_peered":"2024-01-15T18:17:17.139718+0000","last_unstale":"2024-01-15T18:17:18.439693+0000","last_undegraded":"2024-01-15T18:17:18.439693+0000","last_fullsized":"2024-01-15T18:17:18.439693+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:56:19.086576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"96'6","reported_seq":18,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.148952+0000","last_change":"2024-01-15T18:17:16.120030+0000","last_active":"2024-01-15T18:17:17.148952+0000","last_peered":"2024-01-15T18:17:17.148952+0000","last_clean":"2024-01-15T18:17:17.148952+0000","last_became_active":"2024-01-15T18:17:16.119705+0000","last_became_peered":"2024-01-15T18:17:16.119705+0000","last_unstale":"2024-01-15T18:17:17.148952+0000","last_undegraded":"2024-01-15T18:17:17.148952+0000","last_fullsized":"2024-01-15T18:17:17.148952+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:47:48.944557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439304+0000","last_change":"2024-01-15T18:17:17.128762+0000","last_active":"2024-01-15T18:17:18.439304+0000","last_peered":"2024-01-15T18:17:18.439304+0000","last_clean":"2024-01-15T18:17:18.439304+0000","last_became_active":"2024-01-15T18:17:17.125747+0000","last_became_peered":"2024-01-15T18:17:17.125747+0000","last_unstale":"2024-01-15T18:17:18.439304+0000","last_undegraded":"2024-01-15T18:17:18.439304+0000","last_fullsized":"2024-01-15T18:17:18.439304+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:30:58.858568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"96'2","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.146621+0000","last_change":"2024-01-15T18:17:16.108415+0000","last_active":"2024-01-15T18:17:17.146621+0000","last_peered":"2024-01-15T18:17:17.146621+0000","last_clean":"2024-01-15T18:17:17.146621+0000","last_became_active":"2024-01-15T18:17:16.108103+0000","last_became_peered":"2024-01-15T18:17:16.108103+0000","last_unstale":"2024-01-15T18:17:17.146621+0000","last_undegraded":"2024-01-15T18:17:17.146621+0000","last_fullsized":"2024-01-15T18:17:17.146621+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:31:29.832701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.127552+0000","last_change":"2024-01-15T18:17:17.128568+0000","last_active":"2024-01-15T18:17:18.127552+0000","last_peered":"2024-01-15T18:17:18.127552+0000","last_clean":"2024-01-15T18:17:18.127552+0000","last_became_active":"2024-01-15T18:17:17.128231+0000","last_became_peered":"2024-01-15T18:17:17.128231+0000","last_unstale":"2024-01-15T18:17:18.127552+0000","last_undegraded":"2024-01-15T18:17:18.127552+0000","last_fullsized":"2024-01-15T18:17:18.127552+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:14:46.363468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.106398+0000","last_change":"2024-01-15T18:17:16.115506+0000","last_active":"2024-01-15T18:17:17.106398+0000","last_peered":"2024-01-15T18:17:17.106398+0000","last_clean":"2024-01-15T18:17:17.106398+0000","last_became_active":"2024-01-15T18:17:16.114999+0000","last_became_peered":"2024-01-15T18:17:16.114999+0000","last_unstale":"2024-01-15T18:17:17.106398+0000","last_undegraded":"2024-01-15T18:17:17.106398+0000","last_fullsized":"2024-01-15T18:17:17.106398+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:41:20.102233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.132434+0000","last_change":"2024-01-15T18:17:17.121456+0000","last_active":"2024-01-15T18:17:18.132434+0000","last_peered":"2024-01-15T18:17:18.132434+0000","last_clean":"2024-01-15T18:17:18.132434+0000","last_became_active":"2024-01-15T18:17:17.121179+0000","last_became_peered":"2024-01-15T18:17:17.121179+0000","last_unstale":"2024-01-15T18:17:18.132434+0000","last_undegraded":"2024-01-15T18:17:18.132434+0000","last_fullsized":"2024-01-15T18:17:18.132434+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:53:26.915291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.111628+0000","last_change":"2024-01-15T18:17:16.113394+0000","last_active":"2024-01-15T18:17:17.111628+0000","last_peered":"2024-01-15T18:17:17.111628+0000","last_clean":"2024-01-15T18:17:17.111628+0000","last_became_active":"2024-01-15T18:17:16.113105+0000","last_became_peered":"2024-01-15T18:17:16.113105+0000","last_unstale":"2024-01-15T18:17:17.111628+0000","last_undegraded":"2024-01-15T18:17:17.111628+0000","last_fullsized":"2024-01-15T18:17:17.111628+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:41:23.459232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439139+0000","last_change":"2024-01-15T18:17:17.116240+0000","last_active":"2024-01-15T18:17:18.439139+0000","last_peered":"2024-01-15T18:17:18.439139+0000","last_clean":"2024-01-15T18:17:18.439139+0000","last_became_active":"2024-01-15T18:17:17.115648+0000","last_became_peered":"2024-01-15T18:17:17.115648+0000","last_unstale":"2024-01-15T18:17:18.439139+0000","last_undegraded":"2024-01-15T18:17:18.439139+0000","last_fullsized":"2024-01-15T18:17:18.439139+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:37:55.014002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"96'8","reported_seq":20,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.150103+0000","last_change":"2024-01-15T18:17:16.112433+0000","last_active":"2024-01-15T18:17:17.150103+0000","last_peered":"2024-01-15T18:17:17.150103+0000","last_clean":"2024-01-15T18:17:17.150103+0000","last_became_active":"2024-01-15T18:17:16.111940+0000","last_became_peered":"2024-01-15T18:17:16.111940+0000","last_unstale":"2024-01-15T18:17:17.150103+0000","last_undegraded":"2024-01-15T18:17:17.150103+0000","last_fullsized":"2024-01-15T18:17:17.150103+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:35:28.258210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439907+0000","last_change":"2024-01-15T18:17:17.133095+0000","last_active":"2024-01-15T18:17:18.439907+0000","last_peered":"2024-01-15T18:17:18.439907+0000","last_clean":"2024-01-15T18:17:18.439907+0000","last_became_active":"2024-01-15T18:17:17.130768+0000","last_became_peered":"2024-01-15T18:17:17.130768+0000","last_unstale":"2024-01-15T18:17:18.439907+0000","last_undegraded":"2024-01-15T18:17:18.439907+0000","last_fullsized":"2024-01-15T18:17:18.439907+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:23.103323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"96'2","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.143716+0000","last_change":"2024-01-15T18:17:16.108046+0000","last_active":"2024-01-15T18:17:17.143716+0000","last_peered":"2024-01-15T18:17:17.143716+0000","last_clean":"2024-01-15T18:17:17.143716+0000","last_became_active":"2024-01-15T18:17:16.107750+0000","last_became_peered":"2024-01-15T18:17:16.107750+0000","last_unstale":"2024-01-15T18:17:17.143716+0000","last_undegraded":"2024-01-15T18:17:17.143716+0000","last_fullsized":"2024-01-15T18:17:17.143716+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:14.320865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.131540+0000","last_change":"2024-01-15T18:17:17.123658+0000","last_active":"2024-01-15T18:17:18.131540+0000","last_peered":"2024-01-15T18:17:18.131540+0000","last_clean":"2024-01-15T18:17:18.131540+0000","last_became_active":"2024-01-15T18:17:17.123351+0000","last_became_peered":"2024-01-15T18:17:17.123351+0000","last_unstale":"2024-01-15T18:17:18.131540+0000","last_undegraded":"2024-01-15T18:17:18.131540+0000","last_fullsized":"2024-01-15T18:17:18.131540+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:38:11.462593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"96'1","reported_seq":13,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.145891+0000","last_change":"2024-01-15T18:17:16.110317+0000","last_active":"2024-01-15T18:17:17.145891+0000","last_peered":"2024-01-15T18:17:17.145891+0000","last_clean":"2024-01-15T18:17:17.145891+0000","last_became_active":"2024-01-15T18:17:16.110007+0000","last_became_peered":"2024-01-15T18:17:16.110007+0000","last_unstale":"2024-01-15T18:17:17.145891+0000","last_undegraded":"2024-01-15T18:17:17.145891+0000","last_fullsized":"2024-01-15T18:17:17.145891+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:41:23.459232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439844+0000","last_change":"2024-01-15T18:17:17.134428+0000","last_active":"2024-01-15T18:17:18.439844+0000","last_peered":"2024-01-15T18:17:18.439844+0000","last_clean":"2024-01-15T18:17:18.439844+0000","last_became_active":"2024-01-15T18:17:17.134096+0000","last_became_peered":"2024-01-15T18:17:17.134096+0000","last_unstale":"2024-01-15T18:17:18.439844+0000","last_undegraded":"2024-01-15T18:17:18.439844+0000","last_fullsized":"2024-01-15T18:17:18.439844+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:08:51.381274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440407+0000","last_change":"2024-01-15T18:17:16.125410+0000","last_active":"2024-01-15T18:17:18.440407+0000","last_peered":"2024-01-15T18:17:18.440407+0000","last_clean":"2024-01-15T18:17:18.440407+0000","last_became_active":"2024-01-15T18:17:16.124440+0000","last_became_peered":"2024-01-15T18:17:16.124440+0000","last_unstale":"2024-01-15T18:17:18.440407+0000","last_undegraded":"2024-01-15T18:17:18.440407+0000","last_fullsized":"2024-01-15T18:17:18.440407+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:37:51.051323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439536+0000","last_change":"2024-01-15T18:17:17.120591+0000","last_active":"2024-01-15T18:17:18.439536+0000","last_peered":"2024-01-15T18:17:18.439536+0000","last_clean":"2024-01-15T18:17:18.439536+0000","last_became_active":"2024-01-15T18:17:17.120092+0000","last_became_peered":"2024-01-15T18:17:17.120092+0000","last_unstale":"2024-01-15T18:17:18.439536+0000","last_undegraded":"2024-01-15T18:17:18.439536+0000","last_fullsized":"2024-01-15T18:17:18.439536+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:05:05.535293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440507+0000","last_change":"2024-01-15T18:17:16.120398+0000","last_active":"2024-01-15T18:17:18.440507+0000","last_peered":"2024-01-15T18:17:18.440507+0000","last_clean":"2024-01-15T18:17:18.440507+0000","last_became_active":"2024-01-15T18:17:16.119944+0000","last_became_peered":"2024-01-15T18:17:16.119944+0000","last_unstale":"2024-01-15T18:17:18.440507+0000","last_undegraded":"2024-01-15T18:17:18.440507+0000","last_fullsized":"2024-01-15T18:17:18.440507+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:54:02.313160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440061+0000","last_change":"2024-01-15T18:17:17.125305+0000","last_active":"2024-01-15T18:17:18.440061+0000","last_peered":"2024-01-15T18:17:18.440061+0000","last_clean":"2024-01-15T18:17:18.440061+0000","last_became_active":"2024-01-15T18:17:17.125035+0000","last_became_peered":"2024-01-15T18:17:17.125035+0000","last_unstale":"2024-01-15T18:17:18.440061+0000","last_undegraded":"2024-01-15T18:17:18.440061+0000","last_fullsized":"2024-01-15T18:17:18.440061+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:24:25.383229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.131461+0000","last_change":"2024-01-15T18:17:16.102303+0000","last_active":"2024-01-15T18:17:18.131461+0000","last_peered":"2024-01-15T18:17:18.131461+0000","last_clean":"2024-01-15T18:17:18.131461+0000","last_became_active":"2024-01-15T18:17:16.099773+0000","last_became_peered":"2024-01-15T18:17:16.099773+0000","last_unstale":"2024-01-15T18:17:18.131461+0000","last_undegraded":"2024-01-15T18:17:18.131461+0000","last_fullsized":"2024-01-15T18:17:18.131461+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:40:51.210210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439104+0000","last_change":"2024-01-15T18:17:17.118751+0000","last_active":"2024-01-15T18:17:18.439104+0000","last_peered":"2024-01-15T18:17:18.439104+0000","last_clean":"2024-01-15T18:17:18.439104+0000","last_became_active":"2024-01-15T18:17:17.118486+0000","last_became_peered":"2024-01-15T18:17:17.118486+0000","last_unstale":"2024-01-15T18:17:18.439104+0000","last_undegraded":"2024-01-15T18:17:18.439104+0000","last_fullsized":"2024-01-15T18:17:18.439104+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:24:32.675015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.131385+0000","last_change":"2024-01-15T18:17:16.119288+0000","last_active":"2024-01-15T18:17:18.131385+0000","last_peered":"2024-01-15T18:17:18.131385+0000","last_clean":"2024-01-15T18:17:18.131385+0000","last_became_active":"2024-01-15T18:17:16.118972+0000","last_became_peered":"2024-01-15T18:17:16.118972+0000","last_unstale":"2024-01-15T18:17:18.131385+0000","last_undegraded":"2024-01-15T18:17:18.131385+0000","last_fullsized":"2024-01-15T18:17:18.131385+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:14:29.755549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439162+0000","last_change":"2024-01-15T18:17:17.131302+0000","last_active":"2024-01-15T18:17:18.439162+0000","last_peered":"2024-01-15T18:17:18.439162+0000","last_clean":"2024-01-15T18:17:18.439162+0000","last_became_active":"2024-01-15T18:17:17.130944+0000","last_became_peered":"2024-01-15T18:17:17.130944+0000","last_unstale":"2024-01-15T18:17:18.439162+0000","last_undegraded":"2024-01-15T18:17:18.439162+0000","last_fullsized":"2024-01-15T18:17:18.439162+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:45:26.435028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439494+0000","last_change":"2024-01-15T18:17:16.127573+0000","last_active":"2024-01-15T18:17:18.439494+0000","last_peered":"2024-01-15T18:17:18.439494+0000","last_clean":"2024-01-15T18:17:18.439494+0000","last_became_active":"2024-01-15T18:17:16.127194+0000","last_became_peered":"2024-01-15T18:17:16.127194+0000","last_unstale":"2024-01-15T18:17:18.439494+0000","last_undegraded":"2024-01-15T18:17:18.439494+0000","last_fullsized":"2024-01-15T18:17:18.439494+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:48:07.276946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440832+0000","last_change":"2024-01-15T18:17:17.132695+0000","last_active":"2024-01-15T18:17:18.440832+0000","last_peered":"2024-01-15T18:17:18.440832+0000","last_clean":"2024-01-15T18:17:18.440832+0000","last_became_active":"2024-01-15T18:17:17.132430+0000","last_became_peered":"2024-01-15T18:17:17.132430+0000","last_unstale":"2024-01-15T18:17:18.440832+0000","last_undegraded":"2024-01-15T18:17:18.440832+0000","last_fullsized":"2024-01-15T18:17:18.440832+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:14:02.211346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.132130+0000","last_change":"2024-01-15T18:17:16.102078+0000","last_active":"2024-01-15T18:17:18.132130+0000","last_peered":"2024-01-15T18:17:18.132130+0000","last_clean":"2024-01-15T18:17:18.132130+0000","last_became_active":"2024-01-15T18:17:16.101772+0000","last_became_peered":"2024-01-15T18:17:16.101772+0000","last_unstale":"2024-01-15T18:17:18.132130+0000","last_undegraded":"2024-01-15T18:17:18.132130+0000","last_fullsized":"2024-01-15T18:17:18.132130+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:28:05.931436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.127924+0000","last_change":"2024-01-15T18:17:17.126658+0000","last_active":"2024-01-15T18:17:18.127924+0000","last_peered":"2024-01-15T18:17:18.127924+0000","last_clean":"2024-01-15T18:17:18.127924+0000","last_became_active":"2024-01-15T18:17:17.126383+0000","last_became_peered":"2024-01-15T18:17:17.126383+0000","last_unstale":"2024-01-15T18:17:18.127924+0000","last_undegraded":"2024-01-15T18:17:18.127924+0000","last_fullsized":"2024-01-15T18:17:18.127924+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:50:07.301222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440855+0000","last_change":"2024-01-15T18:17:16.121035+0000","last_active":"2024-01-15T18:17:18.440855+0000","last_peered":"2024-01-15T18:17:18.440855+0000","last_clean":"2024-01-15T18:17:18.440855+0000","last_became_active":"2024-01-15T18:17:16.120714+0000","last_became_peered":"2024-01-15T18:17:16.120714+0000","last_unstale":"2024-01-15T18:17:18.440855+0000","last_undegraded":"2024-01-15T18:17:18.440855+0000","last_fullsized":"2024-01-15T18:17:18.440855+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:30:03.158862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.127748+0000","last_change":"2024-01-15T18:17:17.124340+0000","last_active":"2024-01-15T18:17:18.127748+0000","last_peered":"2024-01-15T18:17:18.127748+0000","last_clean":"2024-01-15T18:17:18.127748+0000","last_became_active":"2024-01-15T18:17:17.124040+0000","last_became_peered":"2024-01-15T18:17:17.124040+0000","last_unstale":"2024-01-15T18:17:18.127748+0000","last_undegraded":"2024-01-15T18:17:18.127748+0000","last_fullsized":"2024-01-15T18:17:18.127748+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:30:47.380860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440319+0000","last_change":"2024-01-15T18:17:16.114051+0000","last_active":"2024-01-15T18:17:18.440319+0000","last_peered":"2024-01-15T18:17:18.440319+0000","last_clean":"2024-01-15T18:17:18.440319+0000","last_became_active":"2024-01-15T18:17:16.113079+0000","last_became_peered":"2024-01-15T18:17:16.113079+0000","last_unstale":"2024-01-15T18:17:18.440319+0000","last_undegraded":"2024-01-15T18:17:18.440319+0000","last_fullsized":"2024-01-15T18:17:18.440319+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:25:22.584736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439887+0000","last_change":"2024-01-15T18:17:17.131965+0000","last_active":"2024-01-15T18:17:18.439887+0000","last_peered":"2024-01-15T18:17:18.439887+0000","last_clean":"2024-01-15T18:17:18.439887+0000","last_became_active":"2024-01-15T18:17:17.131693+0000","last_became_peered":"2024-01-15T18:17:17.131693+0000","last_unstale":"2024-01-15T18:17:18.439887+0000","last_undegraded":"2024-01-15T18:17:18.439887+0000","last_fullsized":"2024-01-15T18:17:18.439887+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:13:53.357957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.118968+0000","last_change":"2024-01-15T18:17:16.114737+0000","last_active":"2024-01-15T18:17:17.118968+0000","last_peered":"2024-01-15T18:17:17.118968+0000","last_clean":"2024-01-15T18:17:17.118968+0000","last_became_active":"2024-01-15T18:17:16.114426+0000","last_became_peered":"2024-01-15T18:17:16.114426+0000","last_unstale":"2024-01-15T18:17:17.118968+0000","last_undegraded":"2024-01-15T18:17:17.118968+0000","last_fullsized":"2024-01-15T18:17:17.118968+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:14.320865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439682+0000","last_change":"2024-01-15T18:17:17.115395+0000","last_active":"2024-01-15T18:17:18.439682+0000","last_peered":"2024-01-15T18:17:18.439682+0000","last_clean":"2024-01-15T18:17:18.439682+0000","last_became_active":"2024-01-15T18:17:17.115088+0000","last_became_peered":"2024-01-15T18:17:17.115088+0000","last_unstale":"2024-01-15T18:17:18.439682+0000","last_undegraded":"2024-01-15T18:17:18.439682+0000","last_fullsized":"2024-01-15T18:17:18.439682+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:38:55.775065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440562+0000","last_change":"2024-01-15T18:17:16.121786+0000","last_active":"2024-01-15T18:17:18.440562+0000","last_peered":"2024-01-15T18:17:18.440562+0000","last_clean":"2024-01-15T18:17:18.440562+0000","last_became_active":"2024-01-15T18:17:16.121508+0000","last_became_peered":"2024-01-15T18:17:16.121508+0000","last_unstale":"2024-01-15T18:17:18.440562+0000","last_undegraded":"2024-01-15T18:17:18.440562+0000","last_fullsized":"2024-01-15T18:17:18.440562+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:51:43.523481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.133059+0000","last_change":"2024-01-15T18:17:17.126022+0000","last_active":"2024-01-15T18:17:18.133059+0000","last_peered":"2024-01-15T18:17:18.133059+0000","last_clean":"2024-01-15T18:17:18.133059+0000","last_became_active":"2024-01-15T18:17:17.125366+0000","last_became_peered":"2024-01-15T18:17:17.125366+0000","last_unstale":"2024-01-15T18:17:18.133059+0000","last_undegraded":"2024-01-15T18:17:18.133059+0000","last_fullsized":"2024-01-15T18:17:18.133059+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:27:59.524326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.117607+0000","last_change":"2024-01-15T18:17:16.112593+0000","last_active":"2024-01-15T18:17:17.117607+0000","last_peered":"2024-01-15T18:17:17.117607+0000","last_clean":"2024-01-15T18:17:17.117607+0000","last_became_active":"2024-01-15T18:17:16.112226+0000","last_became_peered":"2024-01-15T18:17:16.112226+0000","last_unstale":"2024-01-15T18:17:17.117607+0000","last_undegraded":"2024-01-15T18:17:17.117607+0000","last_fullsized":"2024-01-15T18:17:17.117607+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:30:08.428553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439756+0000","last_change":"2024-01-15T18:17:17.114883+0000","last_active":"2024-01-15T18:17:18.439756+0000","last_peered":"2024-01-15T18:17:18.439756+0000","last_clean":"2024-01-15T18:17:18.439756+0000","last_became_active":"2024-01-15T18:17:17.114495+0000","last_became_peered":"2024-01-15T18:17:17.114495+0000","last_unstale":"2024-01-15T18:17:18.439756+0000","last_undegraded":"2024-01-15T18:17:18.439756+0000","last_fullsized":"2024-01-15T18:17:18.439756+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:16:26.664356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.112868+0000","last_change":"2024-01-15T18:17:16.122610+0000","last_active":"2024-01-15T18:17:17.112868+0000","last_peered":"2024-01-15T18:17:17.112868+0000","last_clean":"2024-01-15T18:17:17.112868+0000","last_became_active":"2024-01-15T18:17:16.122043+0000","last_became_peered":"2024-01-15T18:17:16.122043+0000","last_unstale":"2024-01-15T18:17:17.112868+0000","last_undegraded":"2024-01-15T18:17:17.112868+0000","last_fullsized":"2024-01-15T18:17:17.112868+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:26:13.337246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439370+0000","last_change":"2024-01-15T18:17:17.133427+0000","last_active":"2024-01-15T18:17:18.439370+0000","last_peered":"2024-01-15T18:17:18.439370+0000","last_clean":"2024-01-15T18:17:18.439370+0000","last_became_active":"2024-01-15T18:17:17.133128+0000","last_became_peered":"2024-01-15T18:17:17.133128+0000","last_unstale":"2024-01-15T18:17:18.439370+0000","last_undegraded":"2024-01-15T18:17:18.439370+0000","last_fullsized":"2024-01-15T18:17:18.439370+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:54.730338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.106040+0000","last_change":"2024-01-15T18:17:16.120473+0000","last_active":"2024-01-15T18:17:17.106040+0000","last_peered":"2024-01-15T18:17:17.106040+0000","last_clean":"2024-01-15T18:17:17.106040+0000","last_became_active":"2024-01-15T18:17:16.119522+0000","last_became_peered":"2024-01-15T18:17:16.119522+0000","last_unstale":"2024-01-15T18:17:17.106040+0000","last_undegraded":"2024-01-15T18:17:17.106040+0000","last_fullsized":"2024-01-15T18:17:17.106040+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:55:35.000200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440906+0000","last_change":"2024-01-15T18:17:17.121210+0000","last_active":"2024-01-15T18:17:18.440906+0000","last_peered":"2024-01-15T18:17:18.440906+0000","last_clean":"2024-01-15T18:17:18.440906+0000","last_became_active":"2024-01-15T18:17:17.120931+0000","last_became_peered":"2024-01-15T18:17:17.120931+0000","last_unstale":"2024-01-15T18:17:18.440906+0000","last_undegraded":"2024-01-15T18:17:18.440906+0000","last_fullsized":"2024-01-15T18:17:18.440906+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:00:27.672303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"96'1","reported_seq":13,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.141961+0000","last_change":"2024-01-15T18:17:16.119481+0000","last_active":"2024-01-15T18:17:17.141961+0000","last_peered":"2024-01-15T18:17:17.141961+0000","last_clean":"2024-01-15T18:17:17.141961+0000","last_became_active":"2024-01-15T18:17:16.119190+0000","last_became_peered":"2024-01-15T18:17:16.119190+0000","last_unstale":"2024-01-15T18:17:17.141961+0000","last_undegraded":"2024-01-15T18:17:17.141961+0000","last_fullsized":"2024-01-15T18:17:17.141961+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:22.935829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":230,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.115466+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:17:17.115466+0000","last_peered":"2024-01-15T18:17:17.115466+0000","last_clean":"2024-01-15T18:17:17.115466+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:17:17.115466+0000","last_undegraded":"2024-01-15T18:17:17.115466+0000","last_fullsized":"2024-01-15T18:17:17.115466+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509594,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64804,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307036,"statfs":{"total":96636764160,"available":96570404864,"internally_reserved":0,"allocated":1794048,"data_stored":1108279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64551370},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.85299999999999998,"15min":0.874},"min":{"1min":0.38700000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92900000000000005,"5min":9.3040000000000003,"15min":10.284000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.86699999999999999,"15min":0.92700000000000005},"min":{"1min":0.42599999999999999,"5min":0.38400000000000001,"15min":0.378},"max":{"1min":2.145,"5min":6.6189999999999998,"15min":16.968},"last":0.85699999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.91000000000000003,"15min":0.92800000000000005},"min":{"1min":0.46800000000000003,"5min":0.36099999999999999,"15min":0.33800000000000002},"max":{"1min":0.90000000000000002,"5min":9.3829999999999991,"15min":15.791},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.86599999999999999,"15min":0.91800000000000004},"min":{"1min":0.505,"5min":0.371,"15min":0.371},"max":{"1min":2.1890000000000001,"5min":5.0899999999999999,"15min":10.842000000000001},"last":0.75800000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":1.1539999999999999,"15min":0.93799999999999994},"min":{"1min":0.51500000000000001,"5min":0.40400000000000003,"15min":0.378},"max":{"1min":1.1619999999999999,"5min":22.318000000000001,"15min":22.318000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.91300000000000003,"15min":0.90600000000000003},"min":{"1min":0.51400000000000001,"5min":0.433,"15min":0.39100000000000001},"max":{"1min":0.95499999999999996,"5min":9.5169999999999995,"15min":9.7539999999999996},"last":0.73199999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":1.151,"15min":0.99199999999999999},"min":{"1min":0.53800000000000003,"5min":0.50900000000000001,"15min":0.38100000000000001},"max":{"1min":1.1160000000000001,"5min":18.152999999999999,"15min":18.152999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.95499999999999996,"15min":0.98399999999999999},"min":{"1min":0.54300000000000004,"5min":0.48499999999999999,"15min":0.43099999999999999},"max":{"1min":1.0860000000000001,"5min":9.75,"15min":10.34},"last":0.80100000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.879,"15min":0.94299999999999995},"min":{"1min":0.38900000000000001,"5min":0.34599999999999997,"15min":0.314},"max":{"1min":2.077,"5min":8.4420000000000002,"15min":20.387},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":1.1639999999999999,"15min":0.98399999999999999},"min":{"1min":0.40899999999999997,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.91300000000000003,"5min":17.215,"15min":17.215},"last":0.70099999999999996}]},{"osd":5,"last update":"Mon Jan 15 18:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.93200000000000005,"15min":0.94399999999999995},"min":{"1min":0.41499999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.89800000000000002,"5min":9.6630000000000003,"15min":19.959},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.92900000000000005,"15min":0.98499999999999999},"min":{"1min":0.45300000000000001,"5min":0.39400000000000002,"15min":0.35799999999999998},"max":{"1min":2.105,"5min":7.6980000000000004,"15min":19.905000000000001},"last":0.64200000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":1.004,"15min":1.159},"min":{"1min":0.50800000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":2.2469999999999999,"5min":8.6609999999999996,"15min":22.478999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.69699999999999995,"5min":1.1499999999999999,"15min":1.0629999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.38300000000000001},"max":{"1min":1.032,"5min":17.132000000000001,"15min":21.509},"last":0.78200000000000003}]}]},{"osd":6,"up_from":15,"seq":64424509594,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64228,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307612,"statfs":{"total":96636764160,"available":96570994688,"internally_reserved":0,"allocated":1200128,"data_stored":518451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":1.0529999999999999,"15min":0.97799999999999998},"min":{"1min":0.40000000000000002,"5min":0.32800000000000001,"15min":0.29099999999999998},"max":{"1min":0.86699999999999999,"5min":32.587000000000003,"15min":32.587000000000003},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.91600000000000004,"15min":0.86299999999999999},"min":{"1min":0.41999999999999998,"5min":0.38500000000000001,"15min":0.35399999999999998},"max":{"1min":0.88400000000000001,"5min":16.193000000000001,"15min":16.193000000000001},"last":0.76200000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.82499999999999996,"15min":0.99399999999999999},"min":{"1min":0.40600000000000003,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95299999999999996,"5min":7.7649999999999997,"15min":53.451999999999998},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1,"15min":0.91800000000000004},"min":{"1min":0.49299999999999999,"5min":0.437,"15min":0.42899999999999999},"max":{"1min":0.89900000000000002,"5min":32.512,"15min":32.512},"last":0.56499999999999995}]},{"osd":2,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":1.087,"15min":0.97099999999999997},"min":{"1min":0.48299999999999998,"5min":0.41699999999999998,"15min":0.38900000000000001},"max":{"1min":0.90100000000000002,"5min":32.472999999999999,"15min":32.472999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.997,"15min":0.89600000000000002},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.38500000000000001},"max":{"1min":0.97899999999999998,"5min":16.602,"15min":16.602},"last":0.63400000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.97999999999999998,"15min":0.93100000000000005},"min":{"1min":0.50700000000000001,"5min":0.49399999999999999,"15min":0.41399999999999998},"max":{"1min":0.94499999999999995,"5min":16.388000000000002,"15min":16.388000000000002},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.879,"15min":1.079},"min":{"1min":0.52300000000000002,"5min":0.432,"15min":0.41599999999999998},"max":{"1min":0.996,"5min":5.9740000000000002,"15min":53.398000000000003},"last":0.52800000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.98799999999999999,"15min":0.89100000000000001},"min":{"1min":0.42999999999999999,"5min":0.38100000000000001,"15min":0.34999999999999998},"max":{"1min":0.92200000000000004,"5min":32.549999999999997,"15min":32.549999999999997},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.878,"15min":0.83999999999999997},"min":{"1min":0.42299999999999999,"5min":0.38700000000000001,"15min":0.34499999999999997},"max":{"1min":1.087,"5min":11.852,"15min":11.852},"last":0.68400000000000005}]},{"osd":5,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.82699999999999996,"15min":1.0369999999999999},"min":{"1min":0.36199999999999999,"5min":0.35799999999999998,"15min":0.30399999999999999},"max":{"1min":1.1759999999999999,"5min":7.5650000000000004,"15min":53.229999999999997},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.081,"15min":1.028},"min":{"1min":0.432,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.228,"5min":32.404000000000003,"15min":32.404000000000003},"last":0.83699999999999997}]},{"osd":7,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.93899999999999995,"15min":1.105},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.1220000000000001,"5min":16.565000000000001,"15min":45.070999999999998},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":1.161,"15min":1.21},"min":{"1min":0.46300000000000002,"5min":0.40999999999999998,"15min":0.34999999999999998},"max":{"1min":1.032,"5min":40.972999999999999,"15min":53.338000000000001},"last":0.746}]}]},{"osd":4,"up_from":15,"seq":64424509594,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76724,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295116,"statfs":{"total":96636764160,"available":96558198784,"internally_reserved":0,"allocated":1212416,"data_stored":519535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.82499999999999996,"15min":0.88400000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.32900000000000001},"max":{"1min":0.91800000000000004,"5min":7.3380000000000001,"15min":23.318000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.70199999999999996,"15min":0.79300000000000004},"min":{"1min":0.372,"5min":0.36399999999999999,"15min":0.32600000000000001},"max":{"1min":0.82699999999999996,"5min":6.444,"15min":14.723000000000001},"last":0.72399999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.82699999999999996,"15min":0.78800000000000003},"min":{"1min":0.39600000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.2,"5min":9.5700000000000003,"15min":9.5700000000000003},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.92100000000000004,"15min":0.91600000000000004},"min":{"1min":0.45200000000000001,"5min":0.40200000000000002,"15min":0.36899999999999999},"max":{"1min":0.84099999999999997,"5min":7.9980000000000002,"15min":22.408999999999999},"last":0.65600000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.85399999999999998,"15min":0.79900000000000004},"min":{"1min":0.439,"5min":0.439,"15min":0.34399999999999997},"max":{"1min":1.266,"5min":9.7089999999999996,"15min":9.7089999999999996},"last":0.629},{"interface":"front","average":{"1min":0.73699999999999999,"5min":1.0649999999999999,"15min":1.0149999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.39100000000000001},"max":{"1min":1.714,"5min":14.118,"15min":22.338999999999999},"last":0.77000000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.77800000000000002,"15min":0.872},"min":{"1min":0.437,"5min":0.437,"15min":0.41199999999999998},"max":{"1min":0.88800000000000001,"5min":6.7930000000000001,"15min":14.763999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.94199999999999995,"15min":0.90600000000000003},"min":{"1min":0.504,"5min":0.48099999999999998,"15min":0.45500000000000002},"max":{"1min":0.93600000000000005,"5min":9.6289999999999996,"15min":9.6289999999999996},"last":0.81399999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.94599999999999995,"15min":0.95699999999999996},"min":{"1min":0.40000000000000002,"5min":0.34799999999999998,"15min":0.33200000000000002},"max":{"1min":0.94099999999999995,"5min":15.787000000000001,"15min":31.617999999999999},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.75,"15min":0.86399999999999999},"min":{"1min":0.42599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95299999999999996,"5min":6.7169999999999996,"15min":17.145},"last":0.79600000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.84899999999999998,"15min":0.82799999999999996},"min":{"1min":0.42299999999999999,"5min":0.39800000000000002,"15min":0.35499999999999998},"max":{"1min":0.97799999999999998,"5min":9.6709999999999994,"15min":11.124000000000001},"last":0.751},{"interface":"front","average":{"1min":0.69799999999999995,"5min":1.1699999999999999,"15min":1.2230000000000001},"min":{"1min":0.56599999999999995,"5min":0.438,"15min":0.40200000000000002},"max":{"1min":0.90200000000000002,"5min":32.308,"15min":32.308},"last":0.82299999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.749,"15min":1.0029999999999999},"min":{"1min":0.434,"5min":0.41099999999999998,"15min":0.34799999999999998},"max":{"1min":0.91900000000000004,"5min":6.8390000000000004,"15min":39.256},"last":0.876},{"interface":"front","average":{"1min":0.70399999999999996,"5min":1.1160000000000001,"15min":1.004},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.42199999999999999},"max":{"1min":0.86199999999999999,"5min":16.277000000000001,"15min":26.573},"last":0.86299999999999999}]}]},{"osd":3,"up_from":15,"seq":64424509593,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64236,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307604,"statfs":{"total":96636764160,"available":96570986496,"internally_reserved":0,"allocated":1208320,"data_stored":519433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64551367},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.44900000000000001,"15min":0.48899999999999999},"min":{"1min":0.27300000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.624,"5min":2.1160000000000001,"15min":5.8849999999999998},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.54100000000000004,"15min":0.61599999999999999},"min":{"1min":0.312,"5min":0.29799999999999999,"15min":0.29499999999999998},"max":{"1min":0.66600000000000004,"5min":3.0649999999999999,"15min":17.756},"last":0.42899999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.495,"15min":0.58499999999999996},"min":{"1min":0.28899999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68700000000000006,"5min":3.6389999999999998,"15min":12.307},"last":0.374},{"interface":"front","average":{"1min":0.503,"5min":0.55500000000000005,"15min":0.58499999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.70299999999999996,"5min":3.738,"15min":6.0640000000000001},"last":0.49099999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.65400000000000003,"15min":0.68999999999999995},"min":{"1min":0.33700000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69999999999999996,"5min":5.431,"15min":12.468999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.70299999999999996,"15min":0.68300000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.34599999999999997},"max":{"1min":0.77800000000000002,"5min":10.837999999999999,"15min":10.837999999999999},"last":0.45200000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":1.0609999999999999,"15min":1.014},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.375},"max":{"1min":1.024,"5min":13.863,"15min":17.853999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.94999999999999996,"15min":0.96199999999999997},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.47099999999999997},"max":{"1min":0.94599999999999995,"5min":6.8289999999999997,"15min":13.31},"last":0.70999999999999996}]},{"osd":5,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.153,"15min":0.97399999999999998},"min":{"1min":0.54200000000000004,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1020000000000001,"5min":17.687999999999999,"15min":17.687999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.88500000000000001,"15min":0.96999999999999997},"min":{"1min":0.45500000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.085,"5min":5.0949999999999998,"15min":20.515999999999998},"last":0.64000000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.80200000000000005,"15min":0.97699999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.432},"max":{"1min":1.05,"5min":4.0910000000000002,"15min":23.100999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.90000000000000002,"15min":0.93600000000000005},"min":{"1min":0.59399999999999997,"5min":0.53600000000000003,"15min":0.49399999999999999},"max":{"1min":1.0069999999999999,"5min":6.7220000000000004,"15min":12.76},"last":0.73499999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":1.331,"15min":1.1659999999999999},"min":{"1min":0.503,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.0089999999999999,"5min":24.321999999999999,"15min":24.321999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":2.0510000000000002,"5min":1.139,"15min":1.1020000000000001},"min":{"1min":0.54700000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"last":0.67700000000000005}]}]},{"osd":2,"up_from":15,"seq":64424509593,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64224,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307616,"statfs":{"total":96636764160,"available":96570998784,"internally_reserved":0,"allocated":1196032,"data_stored":517833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.45500000000000002,"15min":0.50900000000000001},"min":{"1min":0.26500000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.71299999999999997,"5min":0.98199999999999998,"15min":5.1849999999999996},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.49199999999999999,"15min":0.53200000000000003},"min":{"1min":0.315,"5min":0.252,"15min":0.252},"max":{"1min":0.68899999999999995,"5min":1.1579999999999999,"15min":7.516},"last":0.36499999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.55800000000000005,"15min":0.61599999999999999},"min":{"1min":0.39800000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.83299999999999996,"5min":0.95799999999999996,"15min":10.907},"last":0.504},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.50800000000000001,"15min":0.59199999999999997},"min":{"1min":0.38200000000000001,"5min":0.28100000000000003,"15min":0.27000000000000002},"max":{"1min":0.77500000000000002,"5min":1.2849999999999999,"15min":8.9190000000000005},"last":0.41899999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55500000000000005,"15min":0.623},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0249999999999999,"5min":1.133,"15min":12.185},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.60199999999999998,"15min":0.66400000000000003},"min":{"1min":0.40600000000000003,"5min":0.373,"15min":0.34000000000000002},"max":{"1min":0.79000000000000004,"5min":0.96699999999999997,"15min":7.6760000000000002},"last":0.46300000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.88800000000000001,"15min":0.875},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":4.0190000000000001,"5min":4.3150000000000004,"15min":8.6609999999999996},"last":0.752},{"interface":"front","average":{"1min":0.745,"5min":0.76200000000000001,"15min":0.90300000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.5860000000000001,"5min":2.468,"15min":19.146999999999998},"last":0.64900000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":0.79700000000000004,"15min":0.90800000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":5.0309999999999997,"5min":5.0309999999999997,"15min":31.710000000000001},"last":0.876},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.83999999999999997,"15min":0.999},"min":{"1min":0.51100000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.3660000000000001,"5min":3.6459999999999999,"15min":22.539000000000001},"last":0.70699999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.84099999999999997,"15min":0.90600000000000003},"min":{"1min":0.499,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":4.1959999999999997,"5min":4.1959999999999997,"15min":9.4540000000000006},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.79200000000000004,"15min":0.90600000000000003},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.633,"5min":1.633,"15min":18.614999999999998},"last":0.81899999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.97799999999999998,"15min":0.96199999999999997},"min":{"1min":0.52100000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":4.3170000000000002,"5min":11.606,"15min":11.606},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.159,"5min":0.92900000000000005,"15min":1.004},"min":{"1min":0.54900000000000004,"5min":0.52500000000000002,"15min":0.502},"max":{"1min":6.1260000000000003,"5min":6.1260000000000003,"15min":17.013000000000002},"last":0.89600000000000002}]}]},{"osd":1,"up_from":15,"seq":64424509595,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68380,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303460,"statfs":{"total":96636764160,"available":96566743040,"internally_reserved":0,"allocated":1191936,"data_stored":517799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.51700000000000002,"15min":0.65700000000000003},"min":{"1min":0.28399999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.6140000000000001,"5min":4.1550000000000002,"15min":19.577000000000002},"last":0.375},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.623,"15min":0.76000000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.251},"max":{"1min":0.83899999999999997,"5min":12.378,"15min":48.277999999999999},"last":0.40000000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55400000000000005,"15min":0.70799999999999996},"min":{"1min":0.32800000000000001,"5min":0.309,"15min":0.30299999999999999},"max":{"1min":1.8049999999999999,"5min":2.8610000000000002,"15min":40.698},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59099999999999997,"15min":0.877},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.317},"max":{"1min":1.978,"5min":2.5139999999999998,"15min":48.316000000000003},"last":0.58299999999999996}]},{"osd":3,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.623,"15min":0.82999999999999996},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.33700000000000002},"max":{"1min":0.94699999999999995,"5min":2.8620000000000001,"15min":48.234999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.69299999999999995,"15min":0.67700000000000005},"min":{"1min":0.33400000000000002,"5min":0.315,"15min":0.311},"max":{"1min":1.0780000000000001,"5min":9.9719999999999995,"15min":9.9719999999999995},"last":0.55900000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.90300000000000002,"15min":0.88800000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.44500000000000001},"max":{"1min":0.97499999999999998,"5min":9.9329999999999998,"15min":9.9329999999999998},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.86199999999999999,"15min":0.93300000000000005},"min":{"1min":0.502,"5min":0.45800000000000002,"15min":0.42499999999999999},"max":{"1min":2.3900000000000001,"5min":9.8610000000000007,"15min":19.829000000000001},"last":1.012}]},{"osd":5,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.82899999999999996,"15min":1.0129999999999999},"min":{"1min":0.53700000000000003,"5min":0.48599999999999999,"15min":0.46100000000000002},"max":{"1min":2.5960000000000001,"5min":8.5839999999999996,"15min":22.594999999999999},"last":1.212},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.80600000000000005,"15min":1.157},"min":{"1min":0.44400000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.6879999999999999,"5min":2.9609999999999999,"15min":48.198},"last":1.1040000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.90800000000000003,"15min":1.04},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0189999999999999,"5min":9.3989999999999991,"15min":35.112000000000002},"last":1.2350000000000001},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.91300000000000003,"15min":0.96899999999999997},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.50800000000000001},"max":{"1min":2.512,"5min":5.8849999999999998,"15min":19.780999999999999},"last":0.91300000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.89200000000000002,"15min":1.1619999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.93300000000000005,"5min":11.039,"15min":48.148000000000003},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.748,"5min":0.97899999999999998,"15min":0.98999999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.52100000000000002},"max":{"1min":1.0609999999999999,"5min":9.8640000000000008,"15min":9.8640000000000008},"last":1.1559999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542298,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58724,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313116,"statfs":{"total":96636764160,"available":96576630784,"internally_reserved":0,"allocated":1196032,"data_stored":517869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.81200000000000006,"15min":0.80100000000000005},"min":{"1min":0.375,"5min":0.29899999999999999,"15min":0.28999999999999998},"max":{"1min":1.383,"5min":5.1639999999999997,"15min":21.385000000000002},"last":0.874},{"interface":"front","average":{"1min":0.748,"5min":0.78700000000000003,"15min":0.877},"min":{"1min":0.57999999999999996,"5min":0.378,"15min":0.34899999999999998},"max":{"1min":1.3029999999999999,"5min":5.3869999999999996,"15min":9.359},"last":0.67100000000000004}]},{"osd":1,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.82199999999999995,"15min":0.89500000000000002},"min":{"1min":0.44400000000000001,"5min":0.375,"15min":0.34899999999999998},"max":{"1min":1.2070000000000001,"5min":3.7429999999999999,"15min":21.120000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.78500000000000003,"5min":1.1850000000000001,"15min":0.96999999999999997},"min":{"1min":0.48999999999999999,"5min":0.38600000000000001,"15min":0.315},"max":{"1min":1.532,"5min":15.843,"15min":21.5},"last":0.90000000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.83799999999999997,"15min":0.89800000000000002},"min":{"1min":0.47699999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.421,"5min":5.516,"15min":8.7750000000000004},"last":0.622},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.90800000000000003,"15min":0.98799999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.347,"5min":8.3529999999999998,"15min":21.201000000000001},"last":0.68700000000000006}]},{"osd":3,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.93200000000000005,"15min":0.91300000000000003},"min":{"1min":0.58499999999999996,"5min":0.37,"15min":0.37},"max":{"1min":1.478,"5min":7.7199999999999998,"15min":21.457000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.85499999999999998,"15min":0.96299999999999997},"min":{"1min":0.57199999999999995,"5min":0.47699999999999998,"15min":0.45400000000000001},"max":{"1min":1.5,"5min":5.6109999999999998,"15min":8.7300000000000004},"last":0.78900000000000003}]},{"osd":4,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.93500000000000005,"15min":0.88800000000000001},"min":{"1min":0.376,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":4.8650000000000002,"5min":10.962,"15min":10.962},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.78800000000000003,"15min":1.018},"min":{"1min":0.442,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.464,"5min":4.6639999999999997,"15min":26.146999999999998},"last":0.80700000000000005}]},{"osd":6,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.86599999999999999,"15min":0.83799999999999997},"min":{"1min":0.44700000000000001,"5min":0.36099999999999999,"15min":0.32000000000000001},"max":{"1min":2.8940000000000001,"5min":5.2160000000000002,"15min":21.535},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.85899999999999999,"15min":1.0129999999999999},"min":{"1min":0.432,"5min":0.379,"15min":0.36299999999999999},"max":{"1min":4.9139999999999997,"5min":4.9139999999999997,"15min":30.488},"last":0.73899999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.276,"15min":1.2290000000000001},"min":{"1min":0.51800000000000002,"5min":0.432,"15min":0.39700000000000002},"max":{"1min":8.7539999999999996,"5min":33.499000000000002,"15min":33.499000000000002},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.98599999999999999,"15min":1.0900000000000001},"min":{"1min":0.46200000000000002,"5min":0.441,"15min":0.36499999999999999},"max":{"1min":4.3120000000000003,"5min":6.306,"15min":25.437999999999999},"last":0.76200000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542298,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64800,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307040,"statfs":{"total":96636764160,"available":96570408960,"internally_reserved":0,"allocated":1789952,"data_stored":1108749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.52400000000000002,"15min":0.627},"min":{"1min":0.26100000000000001,"5min":0.245,"15min":0.23699999999999999},"max":{"1min":0.93500000000000005,"5min":7.907,"15min":19.843},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58199999999999996,"15min":0.67400000000000004},"min":{"1min":0.35599999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.8240000000000001,"5min":3.512,"15min":10.154},"last":0.33100000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.71599999999999997,"15min":0.80100000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.30399999999999999},"max":{"1min":1.4410000000000001,"5min":7.0810000000000004,"15min":25.905000000000001},"last":0.379},{"interface":"front","average":{"1min":0.498,"5min":0.57499999999999996,"15min":0.61399999999999999},"min":{"1min":0.32700000000000001,"5min":0.27800000000000002,"15min":0.26700000000000002},"max":{"1min":0.79300000000000004,"5min":3.8500000000000001,"15min":9.3829999999999991},"last":0.39600000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.69399999999999995,"15min":0.74199999999999999},"min":{"1min":0.34699999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":2.3519999999999999,"5min":6.2850000000000001,"15min":9.1780000000000008},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.70199999999999996,"15min":0.76200000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.36899999999999999},"max":{"1min":1.327,"5min":6.5979999999999999,"15min":10.210000000000001},"last":0.44400000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.84199999999999997,"15min":0.98399999999999999},"min":{"1min":0.47699999999999998,"5min":0.39100000000000001,"15min":0.38},"max":{"1min":1.2929999999999999,"5min":4.5179999999999998,"15min":22.814},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.83399999999999996,"15min":1.0289999999999999},"min":{"1min":0.502,"5min":0.502,"15min":0.41199999999999998},"max":{"1min":1.2749999999999999,"5min":4.2329999999999997,"15min":23.341999999999999},"last":0.57999999999999996}]},{"osd":5,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.86399999999999999,"15min":0.96999999999999997},"min":{"1min":0.54400000000000004,"5min":0.51600000000000001,"15min":0.441},"max":{"1min":2.6190000000000002,"5min":4.008,"15min":12.428000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.79900000000000004,"5min":1.252,"15min":1.258},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.3120000000000001,"5min":23.021000000000001,"15min":23.021000000000001},"last":0.66100000000000003}]},{"osd":6,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.92000000000000004,"15min":1.212},"min":{"1min":0.53500000000000003,"5min":0.48399999999999999,"15min":0.46200000000000002},"max":{"1min":1.6359999999999999,"5min":6.5250000000000004,"15min":31.059000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.94799999999999995,"15min":1.089},"min":{"1min":0.59399999999999997,"5min":0.52700000000000002,"15min":0.49099999999999999},"max":{"1min":2.5659999999999998,"5min":6.9690000000000003,"15min":40.731000000000002},"last":0.623}]},{"osd":7,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":1.107,"15min":1.2},"min":{"1min":0.59499999999999997,"5min":0.499,"15min":0.42499999999999999},"max":{"1min":2.6539999999999999,"5min":10.497999999999999,"15min":24.492999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.88300000000000001,"15min":1.103},"min":{"1min":0.55300000000000005,"5min":0.495,"15min":0.46600000000000003},"max":{"1min":1.3740000000000001,"5min":4.6600000000000001,"15min":23.524000000000001},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:17:25.810 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:17:25.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.962+0000 7fe603400640 1 Processor -- start 2024-01-15T18:17:25.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.966+0000 7fe603400640 1 -- start start 2024-01-15T18:17:25.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.966+0000 7fe603400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fe5fc15c130 0x7fe5fc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:25.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.966+0000 7fe603400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5fc06c4a0 con 0x7fe5fc0701a0 2024-01-15T18:17:25.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.966+0000 7fe603400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5fc15c660 con 0x7fe5fc06e240 2024-01-15T18:17:25.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.966+0000 7fe603400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5fc15c940 con 0x7fe5fc15c130 2024-01-15T18:17:25.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.966+0000 7fe600a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe5fc0701a0 0x7fe5fc070580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36412/0 (socket says 172.21.15.27:36412) 2024-01-15T18:17:25.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.966+0000 7fe600a00640 1 -- 172.21.15.27:0/251125687 learned_addr learned my addr 172.21.15.27:0/251125687 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:25.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.966+0000 7fe5fbe00640 1 --2- 172.21.15.27:0/251125687 >> v2:172.21.15.93:3300/0 conn(0x7fe5fc15c130 0x7fe5fc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:25.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.966+0000 7fe5fb400640 1 -- 172.21.15.27:0/251125687 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561520611 0 0) 0x7fe5fc06c4a0 con 0x7fe5fc0701a0 2024-01-15T18:17:25.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.966+0000 7fe5fb400640 1 -- 172.21.15.27:0/251125687 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5dc003530 con 0x7fe5fc0701a0 2024-01-15T18:17:25.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.966+0000 7fe5fbe00640 1 -- 172.21.15.27:0/251125687 >> v1:172.21.15.93:6789/0 conn(0x7fe5fc06e240 legacy=0x7fe5fc15d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:25.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.966+0000 7fe5fbe00640 1 -- 172.21.15.27:0/251125687 >> v1:172.21.15.27:6789/0 conn(0x7fe5fc0701a0 legacy=0x7fe5fc070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:25.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.966+0000 7fe5fb400640 1 -- 172.21.15.27:0/251125687 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1905181993 0 0) 0x7fe5dc003530 con 0x7fe5fc0701a0 2024-01-15T18:17:25.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.966+0000 7fe5fbe00640 1 -- 172.21.15.27:0/251125687 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5fc15cc20 con 0x7fe5fc15c130 2024-01-15T18:17:25.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.970+0000 7fe5fbe00640 1 --2- 172.21.15.27:0/251125687 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe5fc15c130 0x7fe5fc06dd00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fe5e40097c0 tx=0x7fe5e401c370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1b06c92a49223e5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.970+0000 7fe5fb400640 1 -- 172.21.15.27:0/251125687 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec001680 con 0x7fe5fc15c130 2024-01-15T18:17:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.970+0000 7fe5fb400640 1 -- 172.21.15.27:0/251125687 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e40a7050 con 0x7fe5fc15c130 2024-01-15T18:17:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.970+0000 7fe5fb400640 1 -- 172.21.15.27:0/251125687 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e40ab600 con 0x7fe5fc15c130 2024-01-15T18:17:25.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.970+0000 7fe603400640 1 -- 172.21.15.27:0/251125687 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe5fc15c130 msgr2=0x7fe5fc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:25.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.970+0000 7fe603400640 1 --2- 172.21.15.27:0/251125687 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe5fc15c130 0x7fe5fc06dd00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fe5e40097c0 tx=0x7fe5e401c370 comp rx=0 tx=0).stop 2024-01-15T18:17:25.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.970+0000 7fe603400640 1 -- 172.21.15.27:0/251125687 shutdown_connections 2024-01-15T18:17:25.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.970+0000 7fe603400640 1 --2- 172.21.15.27:0/251125687 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe5fc15c130 0x7fe5fc06dd00 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:25.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.970+0000 7fe603400640 1 -- 172.21.15.27:0/251125687 >> 172.21.15.27:0/251125687 conn(0x7fe5fc0cec60 msgr2=0x7fe5fc0cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:25.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.970+0000 7fe603400640 1 -- 172.21.15.27:0/251125687 shutdown_connections 2024-01-15T18:17:25.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.970+0000 7fe603400640 1 -- 172.21.15.27:0/251125687 wait complete. 2024-01-15T18:17:25.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.974+0000 7fe603400640 1 Processor -- start 2024-01-15T18:17:25.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.974+0000 7fe603400640 1 -- start start 2024-01-15T18:17:25.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.974+0000 7fe603400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe5fc06e240 0x7fe5fc0ce360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:25.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.974+0000 7fe603400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe5fc0701a0 0x7fe5fc152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:25.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.974+0000 7fe603400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5fc15c130 0x7fe5fc0c7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:25.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.974+0000 7fe603400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5fc0ce8a0 con 0x7fe5fc15c130 2024-01-15T18:17:25.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.974+0000 7fe603400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5fc0cde00 con 0x7fe5fc0701a0 2024-01-15T18:17:25.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.974+0000 7fe603400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5fc0cccf0 con 0x7fe5fc06e240 2024-01-15T18:17:25.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.974+0000 7fe601400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5fc15c130 0x7fe5fc0c7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:25.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.974+0000 7fe601400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5fc15c130 0x7fe5fc0c7f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55338/0 (socket says 172.21.15.27:55338) 2024-01-15T18:17:25.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.974+0000 7fe601400640 1 -- 172.21.15.27:0/4040444853 learned_addr learned my addr 172.21.15.27:0/4040444853 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:25.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.974+0000 7fe5fbe00640 1 --2- 172.21.15.27:0/4040444853 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe5fc0701a0 0x7fe5fc152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:25.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.974+0000 7fe600a00640 1 --2- 172.21.15.27:0/4040444853 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe5fc06e240 0x7fe5fc0ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:25.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.974+0000 7fe601400640 1 -- 172.21.15.27:0/4040444853 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe5fc06e240 msgr2=0x7fe5fc0ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:25.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.974+0000 7fe601400640 1 --2- 172.21.15.27:0/4040444853 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe5fc06e240 0x7fe5fc0ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:25.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.974+0000 7fe601400640 1 -- 172.21.15.27:0/4040444853 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe5fc0701a0 msgr2=0x7fe5fc152090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:25.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.974+0000 7fe601400640 1 --2- 172.21.15.27:0/4040444853 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe5fc0701a0 0x7fe5fc152090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:25.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.974+0000 7fe601400640 1 -- 172.21.15.27:0/4040444853 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e4009420 con 0x7fe5fc15c130 2024-01-15T18:17:25.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.974+0000 7fe601400640 1 --2- 172.21.15.27:0/4040444853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5fc15c130 0x7fe5fc0c7f00 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fe5f400c770 tx=0x7fe5f400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:25.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.974+0000 7fe5f8c00640 1 -- 172.21.15.27:0/4040444853 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f4003b80 con 0x7fe5fc15c130 2024-01-15T18:17:25.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.974+0000 7fe5f8c00640 1 -- 172.21.15.27:0/4040444853 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f4003ce0 con 0x7fe5fc15c130 2024-01-15T18:17:25.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.974+0000 7fe5f8c00640 1 -- 172.21.15.27:0/4040444853 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f400aa90 con 0x7fe5fc15c130 2024-01-15T18:17:25.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.974+0000 7fe603400640 1 -- 172.21.15.27:0/4040444853 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5fc0ccfd0 con 0x7fe5fc15c130 2024-01-15T18:17:25.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.974+0000 7fe603400640 1 -- 172.21.15.27:0/4040444853 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5fc0cd590 con 0x7fe5fc15c130 2024-01-15T18:17:25.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.978+0000 7fe603400640 1 -- 172.21.15.27:0/4040444853 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5fc0ca8c0 con 0x7fe5fc15c130 2024-01-15T18:17:25.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.978+0000 7fe5f8c00640 1 -- 172.21.15.27:0/4040444853 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7fe5f400abf0 con 0x7fe5fc15c130 2024-01-15T18:17:25.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.978+0000 7fe5f8c00640 1 --2- 172.21.15.27:0/4040444853 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe5d4067700 0x7fe5d4069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:25.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.978+0000 7fe5f8c00640 1 -- 172.21.15.27:0/4040444853 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fe5f4082e90 con 0x7fe5fc15c130 2024-01-15T18:17:25.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.978+0000 7fe600a00640 1 --2- 172.21.15.27:0/4040444853 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe5d4067700 0x7fe5d4069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:25.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.982+0000 7fe600a00640 1 --2- 172.21.15.27:0/4040444853 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe5d4067700 0x7fe5d4069bc0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec005890 tx=0x7fe5ec001db0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:25.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:25.982+0000 7fe5f8c00640 1 -- 172.21.15.27:0/4040444853 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe5f4051910 con 0x7fe5fc15c130 2024-01-15T18:17:26.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.138+0000 7fe603400640 1 -- 172.21.15.27:0/4040444853 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe5fc06e030 con 0x7fe5d4067700 2024-01-15T18:17:26.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.146+0000 7fe5f8c00640 1 -- 172.21.15.27:0/4040444853 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288769 (secure 0 0 0) 0x7fe5fc06e030 con 0x7fe5d4067700 2024-01-15T18:17:26.148 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:17:26.149 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:17:26.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.150+0000 7fe603400640 1 -- 172.21.15.27:0/4040444853 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe5d4067700 msgr2=0x7fe5d4069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:26.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.150+0000 7fe603400640 1 --2- 172.21.15.27:0/4040444853 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe5d4067700 0x7fe5d4069bc0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec005890 tx=0x7fe5ec001db0 comp rx=0 tx=0).stop 2024-01-15T18:17:26.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.150+0000 7fe603400640 1 -- 172.21.15.27:0/4040444853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5fc15c130 msgr2=0x7fe5fc0c7f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:26.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.150+0000 7fe603400640 1 --2- 172.21.15.27:0/4040444853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5fc15c130 0x7fe5fc0c7f00 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fe5f400c770 tx=0x7fe5f400cc30 comp rx=0 tx=0).stop 2024-01-15T18:17:26.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.150+0000 7fe603400640 1 -- 172.21.15.27:0/4040444853 shutdown_connections 2024-01-15T18:17:26.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.150+0000 7fe603400640 1 --2- 172.21.15.27:0/4040444853 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe5d4067700 0x7fe5d4069bc0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:26.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.150+0000 7fe603400640 1 --2- 172.21.15.27:0/4040444853 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5fc15c130 0x7fe5fc0c7f00 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:26.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.150+0000 7fe603400640 1 --2- 172.21.15.27:0/4040444853 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe5fc0701a0 0x7fe5fc152090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:26.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.150+0000 7fe603400640 1 --2- 172.21.15.27:0/4040444853 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe5fc06e240 0x7fe5fc0ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:26.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.150+0000 7fe603400640 1 -- 172.21.15.27:0/4040444853 >> 172.21.15.27:0/4040444853 conn(0x7fe5fc0cec60 msgr2=0x7fe5fc0c9720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:26.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.150+0000 7fe603400640 1 -- 172.21.15.27:0/4040444853 shutdown_connections 2024-01-15T18:17:26.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.150+0000 7fe603400640 1 -- 172.21.15.27:0/4040444853 wait complete. 2024-01-15T18:17:26.169 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-01-15T18:17:24.355783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":188,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":526120,"kb_used_data":10536,"kb_used_omap":12,"kb_used_meta":515443,"kb_avail":754448600,"statfs":{"total":773094113280,"available":772555366400,"internally_reserved":0,"allocated":10788864,"data_stored":5327948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":527814192},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002022"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.127754+0000","last_change":"2024-01-15T18:17:17.134979+0000","last_active":"2024-01-15T18:17:18.127754+0000","last_peered":"2024-01-15T18:17:18.127754+0000","last_clean":"2024-01-15T18:17:18.127754+0000","last_became_active":"2024-01-15T18:17:17.134592+0000","last_became_peered":"2024-01-15T18:17:17.134592+0000","last_unstale":"2024-01-15T18:17:18.127754+0000","last_undegraded":"2024-01-15T18:17:18.127754+0000","last_fullsized":"2024-01-15T18:17:18.127754+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:02:20.596146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440653+0000","last_change":"2024-01-15T18:17:17.132525+0000","last_active":"2024-01-15T18:17:18.440653+0000","last_peered":"2024-01-15T18:17:18.440653+0000","last_clean":"2024-01-15T18:17:18.440653+0000","last_became_active":"2024-01-15T18:17:17.132261+0000","last_became_peered":"2024-01-15T18:17:17.132261+0000","last_unstale":"2024-01-15T18:17:18.440653+0000","last_undegraded":"2024-01-15T18:17:18.440653+0000","last_fullsized":"2024-01-15T18:17:18.440653+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:07.042399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.127694+0000","last_change":"2024-01-15T18:17:17.132080+0000","last_active":"2024-01-15T18:17:18.127694+0000","last_peered":"2024-01-15T18:17:18.127694+0000","last_clean":"2024-01-15T18:17:18.127694+0000","last_became_active":"2024-01-15T18:17:17.131696+0000","last_became_peered":"2024-01-15T18:17:17.131696+0000","last_unstale":"2024-01-15T18:17:18.127694+0000","last_undegraded":"2024-01-15T18:17:18.127694+0000","last_fullsized":"2024-01-15T18:17:18.127694+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:26:02.276866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439558+0000","last_change":"2024-01-15T18:17:17.123236+0000","last_active":"2024-01-15T18:17:18.439558+0000","last_peered":"2024-01-15T18:17:18.439558+0000","last_clean":"2024-01-15T18:17:18.439558+0000","last_became_active":"2024-01-15T18:17:17.122924+0000","last_became_peered":"2024-01-15T18:17:17.122924+0000","last_unstale":"2024-01-15T18:17:18.439558+0000","last_undegraded":"2024-01-15T18:17:18.439558+0000","last_fullsized":"2024-01-15T18:17:18.439558+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:30:09.436810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439733+0000","last_change":"2024-01-15T18:17:17.129419+0000","last_active":"2024-01-15T18:17:18.439733+0000","last_peered":"2024-01-15T18:17:18.439733+0000","last_clean":"2024-01-15T18:17:18.439733+0000","last_became_active":"2024-01-15T18:17:17.128570+0000","last_became_peered":"2024-01-15T18:17:17.128570+0000","last_unstale":"2024-01-15T18:17:18.439733+0000","last_undegraded":"2024-01-15T18:17:18.439733+0000","last_fullsized":"2024-01-15T18:17:18.439733+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:39:09.922990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439439+0000","last_change":"2024-01-15T18:17:17.129884+0000","last_active":"2024-01-15T18:17:18.439439+0000","last_peered":"2024-01-15T18:17:18.439439+0000","last_clean":"2024-01-15T18:17:18.439439+0000","last_became_active":"2024-01-15T18:17:17.129579+0000","last_became_peered":"2024-01-15T18:17:17.129579+0000","last_unstale":"2024-01-15T18:17:18.439439+0000","last_undegraded":"2024-01-15T18:17:18.439439+0000","last_fullsized":"2024-01-15T18:17:18.439439+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:07:30.777445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439255+0000","last_change":"2024-01-15T18:17:17.126726+0000","last_active":"2024-01-15T18:17:18.439255+0000","last_peered":"2024-01-15T18:17:18.439255+0000","last_clean":"2024-01-15T18:17:18.439255+0000","last_became_active":"2024-01-15T18:17:17.126431+0000","last_became_peered":"2024-01-15T18:17:17.126431+0000","last_unstale":"2024-01-15T18:17:18.439255+0000","last_undegraded":"2024-01-15T18:17:18.439255+0000","last_fullsized":"2024-01-15T18:17:18.439255+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:58:11.451418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439412+0000","last_change":"2024-01-15T18:17:17.129139+0000","last_active":"2024-01-15T18:17:18.439412+0000","last_peered":"2024-01-15T18:17:18.439412+0000","last_clean":"2024-01-15T18:17:18.439412+0000","last_became_active":"2024-01-15T18:17:17.126922+0000","last_became_peered":"2024-01-15T18:17:17.126922+0000","last_unstale":"2024-01-15T18:17:18.439412+0000","last_undegraded":"2024-01-15T18:17:18.439412+0000","last_fullsized":"2024-01-15T18:17:18.439412+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:40:32.654203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.132988+0000","last_change":"2024-01-15T18:17:17.117455+0000","last_active":"2024-01-15T18:17:18.132988+0000","last_peered":"2024-01-15T18:17:18.132988+0000","last_clean":"2024-01-15T18:17:18.132988+0000","last_became_active":"2024-01-15T18:17:17.117259+0000","last_became_peered":"2024-01-15T18:17:17.117259+0000","last_unstale":"2024-01-15T18:17:18.132988+0000","last_undegraded":"2024-01-15T18:17:18.132988+0000","last_fullsized":"2024-01-15T18:17:18.132988+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:23.944086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439695+0000","last_change":"2024-01-15T18:17:17.132515+0000","last_active":"2024-01-15T18:17:18.439695+0000","last_peered":"2024-01-15T18:17:18.439695+0000","last_clean":"2024-01-15T18:17:18.439695+0000","last_became_active":"2024-01-15T18:17:17.130433+0000","last_became_peered":"2024-01-15T18:17:17.130433+0000","last_unstale":"2024-01-15T18:17:18.439695+0000","last_undegraded":"2024-01-15T18:17:18.439695+0000","last_fullsized":"2024-01-15T18:17:18.439695+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:23.944086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439546+0000","last_change":"2024-01-15T18:17:17.134352+0000","last_active":"2024-01-15T18:17:18.439546+0000","last_peered":"2024-01-15T18:17:18.439546+0000","last_clean":"2024-01-15T18:17:18.439546+0000","last_became_active":"2024-01-15T18:17:17.134076+0000","last_became_peered":"2024-01-15T18:17:17.134076+0000","last_unstale":"2024-01-15T18:17:18.439546+0000","last_undegraded":"2024-01-15T18:17:18.439546+0000","last_fullsized":"2024-01-15T18:17:18.439546+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:59:53.121905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.132977+0000","last_change":"2024-01-15T18:17:17.126892+0000","last_active":"2024-01-15T18:17:18.132977+0000","last_peered":"2024-01-15T18:17:18.132977+0000","last_clean":"2024-01-15T18:17:18.132977+0000","last_became_active":"2024-01-15T18:17:17.125684+0000","last_became_peered":"2024-01-15T18:17:17.125684+0000","last_unstale":"2024-01-15T18:17:18.132977+0000","last_undegraded":"2024-01-15T18:17:18.132977+0000","last_fullsized":"2024-01-15T18:17:18.132977+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:54.730338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439385+0000","last_change":"2024-01-15T18:17:17.125419+0000","last_active":"2024-01-15T18:17:18.439385+0000","last_peered":"2024-01-15T18:17:18.439385+0000","last_clean":"2024-01-15T18:17:18.439385+0000","last_became_active":"2024-01-15T18:17:17.125149+0000","last_became_peered":"2024-01-15T18:17:17.125149+0000","last_unstale":"2024-01-15T18:17:18.439385+0000","last_undegraded":"2024-01-15T18:17:18.439385+0000","last_fullsized":"2024-01-15T18:17:18.439385+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:03.728057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439288+0000","last_change":"2024-01-15T18:17:17.124265+0000","last_active":"2024-01-15T18:17:18.439288+0000","last_peered":"2024-01-15T18:17:18.439288+0000","last_clean":"2024-01-15T18:17:18.439288+0000","last_became_active":"2024-01-15T18:17:17.123974+0000","last_became_peered":"2024-01-15T18:17:17.123974+0000","last_unstale":"2024-01-15T18:17:18.439288+0000","last_undegraded":"2024-01-15T18:17:18.439288+0000","last_fullsized":"2024-01-15T18:17:18.439288+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:56:04.498774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440333+0000","last_change":"2024-01-15T18:17:17.125411+0000","last_active":"2024-01-15T18:17:18.440333+0000","last_peered":"2024-01-15T18:17:18.440333+0000","last_clean":"2024-01-15T18:17:18.440333+0000","last_became_active":"2024-01-15T18:17:17.122916+0000","last_became_peered":"2024-01-15T18:17:17.122916+0000","last_unstale":"2024-01-15T18:17:18.440333+0000","last_undegraded":"2024-01-15T18:17:18.440333+0000","last_fullsized":"2024-01-15T18:17:18.440333+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:47:49.952814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439054+0000","last_change":"2024-01-15T18:17:17.133691+0000","last_active":"2024-01-15T18:17:18.439054+0000","last_peered":"2024-01-15T18:17:18.439054+0000","last_clean":"2024-01-15T18:17:18.439054+0000","last_became_active":"2024-01-15T18:17:17.132756+0000","last_became_peered":"2024-01-15T18:17:17.132756+0000","last_unstale":"2024-01-15T18:17:18.439054+0000","last_undegraded":"2024-01-15T18:17:18.439054+0000","last_fullsized":"2024-01-15T18:17:18.439054+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:11:55.311668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440809+0000","last_change":"2024-01-15T18:17:17.125211+0000","last_active":"2024-01-15T18:17:18.440809+0000","last_peered":"2024-01-15T18:17:18.440809+0000","last_clean":"2024-01-15T18:17:18.440809+0000","last_became_active":"2024-01-15T18:17:17.119327+0000","last_became_peered":"2024-01-15T18:17:17.119327+0000","last_unstale":"2024-01-15T18:17:18.440809+0000","last_undegraded":"2024-01-15T18:17:18.440809+0000","last_fullsized":"2024-01-15T18:17:18.440809+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:57:35.828970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439426+0000","last_change":"2024-01-15T18:17:17.122901+0000","last_active":"2024-01-15T18:17:18.439426+0000","last_peered":"2024-01-15T18:17:18.439426+0000","last_clean":"2024-01-15T18:17:18.439426+0000","last_became_active":"2024-01-15T18:17:17.122470+0000","last_became_peered":"2024-01-15T18:17:17.122470+0000","last_unstale":"2024-01-15T18:17:18.439426+0000","last_undegraded":"2024-01-15T18:17:18.439426+0000","last_fullsized":"2024-01-15T18:17:18.439426+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:03:06.032027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440565+0000","last_change":"2024-01-15T18:17:17.127686+0000","last_active":"2024-01-15T18:17:18.440565+0000","last_peered":"2024-01-15T18:17:18.440565+0000","last_clean":"2024-01-15T18:17:18.440565+0000","last_became_active":"2024-01-15T18:17:17.127251+0000","last_became_peered":"2024-01-15T18:17:17.127251+0000","last_unstale":"2024-01-15T18:17:18.440565+0000","last_undegraded":"2024-01-15T18:17:18.440565+0000","last_fullsized":"2024-01-15T18:17:18.440565+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:28:31.116794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.133012+0000","last_change":"2024-01-15T18:17:17.117786+0000","last_active":"2024-01-15T18:17:18.133012+0000","last_peered":"2024-01-15T18:17:18.133012+0000","last_clean":"2024-01-15T18:17:18.133012+0000","last_became_active":"2024-01-15T18:17:17.117505+0000","last_became_peered":"2024-01-15T18:17:17.117505+0000","last_unstale":"2024-01-15T18:17:18.133012+0000","last_undegraded":"2024-01-15T18:17:18.133012+0000","last_fullsized":"2024-01-15T18:17:18.133012+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:39:09.922990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439938+0000","last_change":"2024-01-15T18:17:17.127902+0000","last_active":"2024-01-15T18:17:18.439938+0000","last_peered":"2024-01-15T18:17:18.439938+0000","last_clean":"2024-01-15T18:17:18.439938+0000","last_became_active":"2024-01-15T18:17:17.127618+0000","last_became_peered":"2024-01-15T18:17:17.127618+0000","last_unstale":"2024-01-15T18:17:18.439938+0000","last_undegraded":"2024-01-15T18:17:18.439938+0000","last_fullsized":"2024-01-15T18:17:18.439938+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:09.950411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440211+0000","last_change":"2024-01-15T18:17:17.125005+0000","last_active":"2024-01-15T18:17:18.440211+0000","last_peered":"2024-01-15T18:17:18.440211+0000","last_clean":"2024-01-15T18:17:18.440211+0000","last_became_active":"2024-01-15T18:17:17.124717+0000","last_became_peered":"2024-01-15T18:17:17.124717+0000","last_unstale":"2024-01-15T18:17:18.440211+0000","last_undegraded":"2024-01-15T18:17:18.440211+0000","last_fullsized":"2024-01-15T18:17:18.440211+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:48:30.181901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440100+0000","last_change":"2024-01-15T18:17:17.130054+0000","last_active":"2024-01-15T18:17:18.440100+0000","last_peered":"2024-01-15T18:17:18.440100+0000","last_clean":"2024-01-15T18:17:18.440100+0000","last_became_active":"2024-01-15T18:17:17.129779+0000","last_became_peered":"2024-01-15T18:17:17.129779+0000","last_unstale":"2024-01-15T18:17:18.440100+0000","last_undegraded":"2024-01-15T18:17:18.440100+0000","last_fullsized":"2024-01-15T18:17:18.440100+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:42:04.787652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439325+0000","last_change":"2024-01-15T18:17:17.137320+0000","last_active":"2024-01-15T18:17:18.439325+0000","last_peered":"2024-01-15T18:17:18.439325+0000","last_clean":"2024-01-15T18:17:18.439325+0000","last_became_active":"2024-01-15T18:17:17.137071+0000","last_became_peered":"2024-01-15T18:17:17.137071+0000","last_unstale":"2024-01-15T18:17:18.439325+0000","last_undegraded":"2024-01-15T18:17:18.439325+0000","last_fullsized":"2024-01-15T18:17:18.439325+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:56:59.821700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439816+0000","last_change":"2024-01-15T18:17:17.137076+0000","last_active":"2024-01-15T18:17:18.439816+0000","last_peered":"2024-01-15T18:17:18.439816+0000","last_clean":"2024-01-15T18:17:18.439816+0000","last_became_active":"2024-01-15T18:17:17.136800+0000","last_became_peered":"2024-01-15T18:17:17.136800+0000","last_unstale":"2024-01-15T18:17:18.439816+0000","last_undegraded":"2024-01-15T18:17:18.439816+0000","last_fullsized":"2024-01-15T18:17:18.439816+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:27:59.524326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440698+0000","last_change":"2024-01-15T18:17:17.132156+0000","last_active":"2024-01-15T18:17:18.440698+0000","last_peered":"2024-01-15T18:17:18.440698+0000","last_clean":"2024-01-15T18:17:18.440698+0000","last_became_active":"2024-01-15T18:17:17.131822+0000","last_became_peered":"2024-01-15T18:17:17.131822+0000","last_unstale":"2024-01-15T18:17:18.440698+0000","last_undegraded":"2024-01-15T18:17:18.440698+0000","last_fullsized":"2024-01-15T18:17:18.440698+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:33:59.696616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439694+0000","last_change":"2024-01-15T18:17:17.136720+0000","last_active":"2024-01-15T18:17:18.439694+0000","last_peered":"2024-01-15T18:17:18.439694+0000","last_clean":"2024-01-15T18:17:18.439694+0000","last_became_active":"2024-01-15T18:17:17.135986+0000","last_became_peered":"2024-01-15T18:17:17.135986+0000","last_unstale":"2024-01-15T18:17:18.439694+0000","last_undegraded":"2024-01-15T18:17:18.439694+0000","last_fullsized":"2024-01-15T18:17:18.439694+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:52:52.621767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440073+0000","last_change":"2024-01-15T18:17:17.129422+0000","last_active":"2024-01-15T18:17:18.440073+0000","last_peered":"2024-01-15T18:17:18.440073+0000","last_clean":"2024-01-15T18:17:18.440073+0000","last_became_active":"2024-01-15T18:17:17.129128+0000","last_became_peered":"2024-01-15T18:17:17.129128+0000","last_unstale":"2024-01-15T18:17:18.440073+0000","last_undegraded":"2024-01-15T18:17:18.440073+0000","last_fullsized":"2024-01-15T18:17:18.440073+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:47:09.879357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440705+0000","last_change":"2024-01-15T18:17:17.127796+0000","last_active":"2024-01-15T18:17:18.440705+0000","last_peered":"2024-01-15T18:17:18.440705+0000","last_clean":"2024-01-15T18:17:18.440705+0000","last_became_active":"2024-01-15T18:17:17.127390+0000","last_became_peered":"2024-01-15T18:17:17.127390+0000","last_unstale":"2024-01-15T18:17:18.440705+0000","last_undegraded":"2024-01-15T18:17:18.440705+0000","last_fullsized":"2024-01-15T18:17:18.440705+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:03:11.357766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439414+0000","last_change":"2024-01-15T18:17:17.120389+0000","last_active":"2024-01-15T18:17:18.439414+0000","last_peered":"2024-01-15T18:17:18.439414+0000","last_clean":"2024-01-15T18:17:18.439414+0000","last_became_active":"2024-01-15T18:17:17.119695+0000","last_became_peered":"2024-01-15T18:17:17.119695+0000","last_unstale":"2024-01-15T18:17:18.439414+0000","last_undegraded":"2024-01-15T18:17:18.439414+0000","last_fullsized":"2024-01-15T18:17:18.439414+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:22:00.599199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440173+0000","last_change":"2024-01-15T18:17:17.130622+0000","last_active":"2024-01-15T18:17:18.440173+0000","last_peered":"2024-01-15T18:17:18.440173+0000","last_clean":"2024-01-15T18:17:18.440173+0000","last_became_active":"2024-01-15T18:17:17.130348+0000","last_became_peered":"2024-01-15T18:17:17.130348+0000","last_unstale":"2024-01-15T18:17:18.440173+0000","last_undegraded":"2024-01-15T18:17:18.440173+0000","last_fullsized":"2024-01-15T18:17:18.440173+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:41:24.467489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.127304+0000","last_change":"2024-01-15T18:17:17.136784+0000","last_active":"2024-01-15T18:17:18.127304+0000","last_peered":"2024-01-15T18:17:18.127304+0000","last_clean":"2024-01-15T18:17:18.127304+0000","last_became_active":"2024-01-15T18:17:17.136510+0000","last_became_peered":"2024-01-15T18:17:17.136510+0000","last_unstale":"2024-01-15T18:17:18.127304+0000","last_undegraded":"2024-01-15T18:17:18.127304+0000","last_fullsized":"2024-01-15T18:17:18.127304+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:45:46.354374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440893+0000","last_change":"2024-01-15T18:17:17.128149+0000","last_active":"2024-01-15T18:17:18.440893+0000","last_peered":"2024-01-15T18:17:18.440893+0000","last_clean":"2024-01-15T18:17:18.440893+0000","last_became_active":"2024-01-15T18:17:17.124768+0000","last_became_peered":"2024-01-15T18:17:17.124768+0000","last_unstale":"2024-01-15T18:17:18.440893+0000","last_undegraded":"2024-01-15T18:17:18.440893+0000","last_fullsized":"2024-01-15T18:17:18.440893+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:28:08.446574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"96'4","reported_seq":16,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440784+0000","last_change":"2024-01-15T18:17:16.117003+0000","last_active":"2024-01-15T18:17:18.440784+0000","last_peered":"2024-01-15T18:17:18.440784+0000","last_clean":"2024-01-15T18:17:18.440784+0000","last_became_active":"2024-01-15T18:17:16.116625+0000","last_became_peered":"2024-01-15T18:17:16.116625+0000","last_unstale":"2024-01-15T18:17:18.440784+0000","last_undegraded":"2024-01-15T18:17:18.440784+0000","last_fullsized":"2024-01-15T18:17:18.440784+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:11:58.418627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439809+0000","last_change":"2024-01-15T18:17:17.126785+0000","last_active":"2024-01-15T18:17:18.439809+0000","last_peered":"2024-01-15T18:17:18.439809+0000","last_clean":"2024-01-15T18:17:18.439809+0000","last_became_active":"2024-01-15T18:17:17.126456+0000","last_became_peered":"2024-01-15T18:17:17.126456+0000","last_unstale":"2024-01-15T18:17:18.439809+0000","last_undegraded":"2024-01-15T18:17:18.439809+0000","last_fullsized":"2024-01-15T18:17:18.439809+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:36:57.592354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"96'1","reported_seq":13,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.142321+0000","last_change":"2024-01-15T18:17:16.120557+0000","last_active":"2024-01-15T18:17:17.142321+0000","last_peered":"2024-01-15T18:17:17.142321+0000","last_clean":"2024-01-15T18:17:17.142321+0000","last_became_active":"2024-01-15T18:17:16.120207+0000","last_became_peered":"2024-01-15T18:17:16.120207+0000","last_unstale":"2024-01-15T18:17:17.142321+0000","last_undegraded":"2024-01-15T18:17:17.142321+0000","last_fullsized":"2024-01-15T18:17:17.142321+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:39:08.914733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.127613+0000","last_change":"2024-01-15T18:17:17.125614+0000","last_active":"2024-01-15T18:17:18.127613+0000","last_peered":"2024-01-15T18:17:18.127613+0000","last_clean":"2024-01-15T18:17:18.127613+0000","last_became_active":"2024-01-15T18:17:17.125330+0000","last_became_peered":"2024-01-15T18:17:17.125330+0000","last_unstale":"2024-01-15T18:17:18.127613+0000","last_undegraded":"2024-01-15T18:17:18.127613+0000","last_fullsized":"2024-01-15T18:17:18.127613+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:13:27.502386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440270+0000","last_change":"2024-01-15T18:17:16.115473+0000","last_active":"2024-01-15T18:17:18.440270+0000","last_peered":"2024-01-15T18:17:18.440270+0000","last_clean":"2024-01-15T18:17:18.440270+0000","last_became_active":"2024-01-15T18:17:16.115189+0000","last_became_peered":"2024-01-15T18:17:16.115189+0000","last_unstale":"2024-01-15T18:17:18.440270+0000","last_undegraded":"2024-01-15T18:17:18.440270+0000","last_fullsized":"2024-01-15T18:17:18.440270+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:58:47.776409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.133184+0000","last_change":"2024-01-15T18:17:17.127577+0000","last_active":"2024-01-15T18:17:18.133184+0000","last_peered":"2024-01-15T18:17:18.133184+0000","last_clean":"2024-01-15T18:17:18.133184+0000","last_became_active":"2024-01-15T18:17:17.125488+0000","last_became_peered":"2024-01-15T18:17:17.125488+0000","last_unstale":"2024-01-15T18:17:18.133184+0000","last_undegraded":"2024-01-15T18:17:18.133184+0000","last_fullsized":"2024-01-15T18:17:18.133184+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:23.103323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440326+0000","last_change":"2024-01-15T18:17:16.123895+0000","last_active":"2024-01-15T18:17:18.440326+0000","last_peered":"2024-01-15T18:17:18.440326+0000","last_clean":"2024-01-15T18:17:18.440326+0000","last_became_active":"2024-01-15T18:17:16.123325+0000","last_became_peered":"2024-01-15T18:17:16.123325+0000","last_unstale":"2024-01-15T18:17:18.440326+0000","last_undegraded":"2024-01-15T18:17:18.440326+0000","last_fullsized":"2024-01-15T18:17:18.440326+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:13:09.312116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.131438+0000","last_change":"2024-01-15T18:17:17.118290+0000","last_active":"2024-01-15T18:17:18.131438+0000","last_peered":"2024-01-15T18:17:18.131438+0000","last_clean":"2024-01-15T18:17:18.131438+0000","last_became_active":"2024-01-15T18:17:17.117943+0000","last_became_peered":"2024-01-15T18:17:17.117943+0000","last_unstale":"2024-01-15T18:17:18.131438+0000","last_undegraded":"2024-01-15T18:17:18.131438+0000","last_fullsized":"2024-01-15T18:17:18.131438+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:50:04.755553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"96'1","reported_seq":13,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.144016+0000","last_change":"2024-01-15T18:17:16.115057+0000","last_active":"2024-01-15T18:17:17.144016+0000","last_peered":"2024-01-15T18:17:17.144016+0000","last_clean":"2024-01-15T18:17:17.144016+0000","last_became_active":"2024-01-15T18:17:16.114788+0000","last_became_peered":"2024-01-15T18:17:16.114788+0000","last_unstale":"2024-01-15T18:17:17.144016+0000","last_undegraded":"2024-01-15T18:17:17.144016+0000","last_fullsized":"2024-01-15T18:17:17.144016+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:50:19.871865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439140+0000","last_change":"2024-01-15T18:17:17.127229+0000","last_active":"2024-01-15T18:17:18.439140+0000","last_peered":"2024-01-15T18:17:18.439140+0000","last_clean":"2024-01-15T18:17:18.439140+0000","last_became_active":"2024-01-15T18:17:17.126951+0000","last_became_peered":"2024-01-15T18:17:17.126951+0000","last_unstale":"2024-01-15T18:17:18.439140+0000","last_undegraded":"2024-01-15T18:17:18.439140+0000","last_fullsized":"2024-01-15T18:17:18.439140+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:34:39.562336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"96'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440314+0000","last_change":"2024-01-15T18:17:16.121483+0000","last_active":"2024-01-15T18:17:18.440314+0000","last_peered":"2024-01-15T18:17:18.440314+0000","last_clean":"2024-01-15T18:17:18.440314+0000","last_became_active":"2024-01-15T18:17:16.121199+0000","last_became_peered":"2024-01-15T18:17:16.121199+0000","last_unstale":"2024-01-15T18:17:18.440314+0000","last_undegraded":"2024-01-15T18:17:18.440314+0000","last_fullsized":"2024-01-15T18:17:18.440314+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:51:29.086954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439898+0000","last_change":"2024-01-15T18:17:17.121762+0000","last_active":"2024-01-15T18:17:18.439898+0000","last_peered":"2024-01-15T18:17:18.439898+0000","last_clean":"2024-01-15T18:17:18.439898+0000","last_became_active":"2024-01-15T18:17:17.118184+0000","last_became_peered":"2024-01-15T18:17:17.118184+0000","last_unstale":"2024-01-15T18:17:18.439898+0000","last_undegraded":"2024-01-15T18:17:18.439898+0000","last_fullsized":"2024-01-15T18:17:18.439898+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:15.329122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.118257+0000","last_change":"2024-01-15T18:17:16.120590+0000","last_active":"2024-01-15T18:17:17.118257+0000","last_peered":"2024-01-15T18:17:17.118257+0000","last_clean":"2024-01-15T18:17:17.118257+0000","last_became_active":"2024-01-15T18:17:16.120243+0000","last_became_peered":"2024-01-15T18:17:16.120243+0000","last_unstale":"2024-01-15T18:17:17.118257+0000","last_undegraded":"2024-01-15T18:17:17.118257+0000","last_fullsized":"2024-01-15T18:17:17.118257+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:36:56.584097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.132487+0000","last_change":"2024-01-15T18:17:17.119962+0000","last_active":"2024-01-15T18:17:18.132487+0000","last_peered":"2024-01-15T18:17:18.132487+0000","last_clean":"2024-01-15T18:17:18.132487+0000","last_became_active":"2024-01-15T18:17:17.118612+0000","last_became_peered":"2024-01-15T18:17:17.118612+0000","last_unstale":"2024-01-15T18:17:18.132487+0000","last_undegraded":"2024-01-15T18:17:18.132487+0000","last_fullsized":"2024-01-15T18:17:18.132487+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:45:26.435028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.117729+0000","last_change":"2024-01-15T18:17:16.117768+0000","last_active":"2024-01-15T18:17:17.117729+0000","last_peered":"2024-01-15T18:17:17.117729+0000","last_clean":"2024-01-15T18:17:17.117729+0000","last_became_active":"2024-01-15T18:17:16.117492+0000","last_became_peered":"2024-01-15T18:17:16.117492+0000","last_unstale":"2024-01-15T18:17:17.117729+0000","last_undegraded":"2024-01-15T18:17:17.117729+0000","last_fullsized":"2024-01-15T18:17:17.117729+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:06:44.491748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440790+0000","last_change":"2024-01-15T18:17:17.124377+0000","last_active":"2024-01-15T18:17:18.440790+0000","last_peered":"2024-01-15T18:17:18.440790+0000","last_clean":"2024-01-15T18:17:18.440790+0000","last_became_active":"2024-01-15T18:17:17.124096+0000","last_became_peered":"2024-01-15T18:17:17.124096+0000","last_unstale":"2024-01-15T18:17:18.440790+0000","last_undegraded":"2024-01-15T18:17:18.440790+0000","last_fullsized":"2024-01-15T18:17:18.440790+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:56:21.720071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.122307+0000","last_change":"2024-01-15T18:17:16.102263+0000","last_active":"2024-01-15T18:17:17.122307+0000","last_peered":"2024-01-15T18:17:17.122307+0000","last_clean":"2024-01-15T18:17:17.122307+0000","last_became_active":"2024-01-15T18:17:16.101797+0000","last_became_peered":"2024-01-15T18:17:16.101797+0000","last_unstale":"2024-01-15T18:17:17.122307+0000","last_undegraded":"2024-01-15T18:17:17.122307+0000","last_fullsized":"2024-01-15T18:17:17.122307+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:36:25.432888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.133177+0000","last_change":"2024-01-15T18:17:17.122838+0000","last_active":"2024-01-15T18:17:18.133177+0000","last_peered":"2024-01-15T18:17:18.133177+0000","last_clean":"2024-01-15T18:17:18.133177+0000","last_became_active":"2024-01-15T18:17:17.122579+0000","last_became_peered":"2024-01-15T18:17:17.122579+0000","last_unstale":"2024-01-15T18:17:18.133177+0000","last_undegraded":"2024-01-15T18:17:18.133177+0000","last_fullsized":"2024-01-15T18:17:18.133177+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:13:53.357957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.113153+0000","last_change":"2024-01-15T18:17:16.122691+0000","last_active":"2024-01-15T18:17:17.113153+0000","last_peered":"2024-01-15T18:17:17.113153+0000","last_clean":"2024-01-15T18:17:17.113153+0000","last_became_active":"2024-01-15T18:17:16.122426+0000","last_became_peered":"2024-01-15T18:17:16.122426+0000","last_unstale":"2024-01-15T18:17:17.113153+0000","last_undegraded":"2024-01-15T18:17:17.113153+0000","last_fullsized":"2024-01-15T18:17:17.113153+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:37:13.270621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440113+0000","last_change":"2024-01-15T18:17:17.129556+0000","last_active":"2024-01-15T18:17:18.440113+0000","last_peered":"2024-01-15T18:17:18.440113+0000","last_clean":"2024-01-15T18:17:18.440113+0000","last_became_active":"2024-01-15T18:17:17.129237+0000","last_became_peered":"2024-01-15T18:17:17.129237+0000","last_unstale":"2024-01-15T18:17:18.440113+0000","last_undegraded":"2024-01-15T18:17:18.440113+0000","last_fullsized":"2024-01-15T18:17:18.440113+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:15:43.895396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"96'3","reported_seq":15,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.142974+0000","last_change":"2024-01-15T18:17:16.114433+0000","last_active":"2024-01-15T18:17:17.142974+0000","last_peered":"2024-01-15T18:17:17.142974+0000","last_clean":"2024-01-15T18:17:17.142974+0000","last_became_active":"2024-01-15T18:17:16.113351+0000","last_became_peered":"2024-01-15T18:17:16.113351+0000","last_unstale":"2024-01-15T18:17:17.142974+0000","last_undegraded":"2024-01-15T18:17:17.142974+0000","last_fullsized":"2024-01-15T18:17:17.142974+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:47:08.871100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439693+0000","last_change":"2024-01-15T18:17:17.140004+0000","last_active":"2024-01-15T18:17:18.439693+0000","last_peered":"2024-01-15T18:17:18.439693+0000","last_clean":"2024-01-15T18:17:18.439693+0000","last_became_active":"2024-01-15T18:17:17.139718+0000","last_became_peered":"2024-01-15T18:17:17.139718+0000","last_unstale":"2024-01-15T18:17:18.439693+0000","last_undegraded":"2024-01-15T18:17:18.439693+0000","last_fullsized":"2024-01-15T18:17:18.439693+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:56:19.086576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"96'6","reported_seq":18,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.148952+0000","last_change":"2024-01-15T18:17:16.120030+0000","last_active":"2024-01-15T18:17:17.148952+0000","last_peered":"2024-01-15T18:17:17.148952+0000","last_clean":"2024-01-15T18:17:17.148952+0000","last_became_active":"2024-01-15T18:17:16.119705+0000","last_became_peered":"2024-01-15T18:17:16.119705+0000","last_unstale":"2024-01-15T18:17:17.148952+0000","last_undegraded":"2024-01-15T18:17:17.148952+0000","last_fullsized":"2024-01-15T18:17:17.148952+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:47:48.944557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439304+0000","last_change":"2024-01-15T18:17:17.128762+0000","last_active":"2024-01-15T18:17:18.439304+0000","last_peered":"2024-01-15T18:17:18.439304+0000","last_clean":"2024-01-15T18:17:18.439304+0000","last_became_active":"2024-01-15T18:17:17.125747+0000","last_became_peered":"2024-01-15T18:17:17.125747+0000","last_unstale":"2024-01-15T18:17:18.439304+0000","last_undegraded":"2024-01-15T18:17:18.439304+0000","last_fullsized":"2024-01-15T18:17:18.439304+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:30:58.858568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"96'2","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.146621+0000","last_change":"2024-01-15T18:17:16.108415+0000","last_active":"2024-01-15T18:17:17.146621+0000","last_peered":"2024-01-15T18:17:17.146621+0000","last_clean":"2024-01-15T18:17:17.146621+0000","last_became_active":"2024-01-15T18:17:16.108103+0000","last_became_peered":"2024-01-15T18:17:16.108103+0000","last_unstale":"2024-01-15T18:17:17.146621+0000","last_undegraded":"2024-01-15T18:17:17.146621+0000","last_fullsized":"2024-01-15T18:17:17.146621+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:31:29.832701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.127552+0000","last_change":"2024-01-15T18:17:17.128568+0000","last_active":"2024-01-15T18:17:18.127552+0000","last_peered":"2024-01-15T18:17:18.127552+0000","last_clean":"2024-01-15T18:17:18.127552+0000","last_became_active":"2024-01-15T18:17:17.128231+0000","last_became_peered":"2024-01-15T18:17:17.128231+0000","last_unstale":"2024-01-15T18:17:18.127552+0000","last_undegraded":"2024-01-15T18:17:18.127552+0000","last_fullsized":"2024-01-15T18:17:18.127552+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:14:46.363468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.106398+0000","last_change":"2024-01-15T18:17:16.115506+0000","last_active":"2024-01-15T18:17:17.106398+0000","last_peered":"2024-01-15T18:17:17.106398+0000","last_clean":"2024-01-15T18:17:17.106398+0000","last_became_active":"2024-01-15T18:17:16.114999+0000","last_became_peered":"2024-01-15T18:17:16.114999+0000","last_unstale":"2024-01-15T18:17:17.106398+0000","last_undegraded":"2024-01-15T18:17:17.106398+0000","last_fullsized":"2024-01-15T18:17:17.106398+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:41:20.102233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.132434+0000","last_change":"2024-01-15T18:17:17.121456+0000","last_active":"2024-01-15T18:17:18.132434+0000","last_peered":"2024-01-15T18:17:18.132434+0000","last_clean":"2024-01-15T18:17:18.132434+0000","last_became_active":"2024-01-15T18:17:17.121179+0000","last_became_peered":"2024-01-15T18:17:17.121179+0000","last_unstale":"2024-01-15T18:17:18.132434+0000","last_undegraded":"2024-01-15T18:17:18.132434+0000","last_fullsized":"2024-01-15T18:17:18.132434+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:53:26.915291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.111628+0000","last_change":"2024-01-15T18:17:16.113394+0000","last_active":"2024-01-15T18:17:17.111628+0000","last_peered":"2024-01-15T18:17:17.111628+0000","last_clean":"2024-01-15T18:17:17.111628+0000","last_became_active":"2024-01-15T18:17:16.113105+0000","last_became_peered":"2024-01-15T18:17:16.113105+0000","last_unstale":"2024-01-15T18:17:17.111628+0000","last_undegraded":"2024-01-15T18:17:17.111628+0000","last_fullsized":"2024-01-15T18:17:17.111628+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:41:23.459232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439139+0000","last_change":"2024-01-15T18:17:17.116240+0000","last_active":"2024-01-15T18:17:18.439139+0000","last_peered":"2024-01-15T18:17:18.439139+0000","last_clean":"2024-01-15T18:17:18.439139+0000","last_became_active":"2024-01-15T18:17:17.115648+0000","last_became_peered":"2024-01-15T18:17:17.115648+0000","last_unstale":"2024-01-15T18:17:18.439139+0000","last_undegraded":"2024-01-15T18:17:18.439139+0000","last_fullsized":"2024-01-15T18:17:18.439139+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:37:55.014002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"96'8","reported_seq":20,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.150103+0000","last_change":"2024-01-15T18:17:16.112433+0000","last_active":"2024-01-15T18:17:17.150103+0000","last_peered":"2024-01-15T18:17:17.150103+0000","last_clean":"2024-01-15T18:17:17.150103+0000","last_became_active":"2024-01-15T18:17:16.111940+0000","last_became_peered":"2024-01-15T18:17:16.111940+0000","last_unstale":"2024-01-15T18:17:17.150103+0000","last_undegraded":"2024-01-15T18:17:17.150103+0000","last_fullsized":"2024-01-15T18:17:17.150103+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:35:28.258210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439907+0000","last_change":"2024-01-15T18:17:17.133095+0000","last_active":"2024-01-15T18:17:18.439907+0000","last_peered":"2024-01-15T18:17:18.439907+0000","last_clean":"2024-01-15T18:17:18.439907+0000","last_became_active":"2024-01-15T18:17:17.130768+0000","last_became_peered":"2024-01-15T18:17:17.130768+0000","last_unstale":"2024-01-15T18:17:18.439907+0000","last_undegraded":"2024-01-15T18:17:18.439907+0000","last_fullsized":"2024-01-15T18:17:18.439907+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:23.103323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"96'2","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.143716+0000","last_change":"2024-01-15T18:17:16.108046+0000","last_active":"2024-01-15T18:17:17.143716+0000","last_peered":"2024-01-15T18:17:17.143716+0000","last_clean":"2024-01-15T18:17:17.143716+0000","last_became_active":"2024-01-15T18:17:16.107750+0000","last_became_peered":"2024-01-15T18:17:16.107750+0000","last_unstale":"2024-01-15T18:17:17.143716+0000","last_undegraded":"2024-01-15T18:17:17.143716+0000","last_fullsized":"2024-01-15T18:17:17.143716+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:14.320865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.131540+0000","last_change":"2024-01-15T18:17:17.123658+0000","last_active":"2024-01-15T18:17:18.131540+0000","last_peered":"2024-01-15T18:17:18.131540+0000","last_clean":"2024-01-15T18:17:18.131540+0000","last_became_active":"2024-01-15T18:17:17.123351+0000","last_became_peered":"2024-01-15T18:17:17.123351+0000","last_unstale":"2024-01-15T18:17:18.131540+0000","last_undegraded":"2024-01-15T18:17:18.131540+0000","last_fullsized":"2024-01-15T18:17:18.131540+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:38:11.462593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"96'1","reported_seq":13,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.145891+0000","last_change":"2024-01-15T18:17:16.110317+0000","last_active":"2024-01-15T18:17:17.145891+0000","last_peered":"2024-01-15T18:17:17.145891+0000","last_clean":"2024-01-15T18:17:17.145891+0000","last_became_active":"2024-01-15T18:17:16.110007+0000","last_became_peered":"2024-01-15T18:17:16.110007+0000","last_unstale":"2024-01-15T18:17:17.145891+0000","last_undegraded":"2024-01-15T18:17:17.145891+0000","last_fullsized":"2024-01-15T18:17:17.145891+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:41:23.459232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439844+0000","last_change":"2024-01-15T18:17:17.134428+0000","last_active":"2024-01-15T18:17:18.439844+0000","last_peered":"2024-01-15T18:17:18.439844+0000","last_clean":"2024-01-15T18:17:18.439844+0000","last_became_active":"2024-01-15T18:17:17.134096+0000","last_became_peered":"2024-01-15T18:17:17.134096+0000","last_unstale":"2024-01-15T18:17:18.439844+0000","last_undegraded":"2024-01-15T18:17:18.439844+0000","last_fullsized":"2024-01-15T18:17:18.439844+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:08:51.381274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440407+0000","last_change":"2024-01-15T18:17:16.125410+0000","last_active":"2024-01-15T18:17:18.440407+0000","last_peered":"2024-01-15T18:17:18.440407+0000","last_clean":"2024-01-15T18:17:18.440407+0000","last_became_active":"2024-01-15T18:17:16.124440+0000","last_became_peered":"2024-01-15T18:17:16.124440+0000","last_unstale":"2024-01-15T18:17:18.440407+0000","last_undegraded":"2024-01-15T18:17:18.440407+0000","last_fullsized":"2024-01-15T18:17:18.440407+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:37:51.051323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439536+0000","last_change":"2024-01-15T18:17:17.120591+0000","last_active":"2024-01-15T18:17:18.439536+0000","last_peered":"2024-01-15T18:17:18.439536+0000","last_clean":"2024-01-15T18:17:18.439536+0000","last_became_active":"2024-01-15T18:17:17.120092+0000","last_became_peered":"2024-01-15T18:17:17.120092+0000","last_unstale":"2024-01-15T18:17:18.439536+0000","last_undegraded":"2024-01-15T18:17:18.439536+0000","last_fullsized":"2024-01-15T18:17:18.439536+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:05:05.535293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440507+0000","last_change":"2024-01-15T18:17:16.120398+0000","last_active":"2024-01-15T18:17:18.440507+0000","last_peered":"2024-01-15T18:17:18.440507+0000","last_clean":"2024-01-15T18:17:18.440507+0000","last_became_active":"2024-01-15T18:17:16.119944+0000","last_became_peered":"2024-01-15T18:17:16.119944+0000","last_unstale":"2024-01-15T18:17:18.440507+0000","last_undegraded":"2024-01-15T18:17:18.440507+0000","last_fullsized":"2024-01-15T18:17:18.440507+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:54:02.313160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440061+0000","last_change":"2024-01-15T18:17:17.125305+0000","last_active":"2024-01-15T18:17:18.440061+0000","last_peered":"2024-01-15T18:17:18.440061+0000","last_clean":"2024-01-15T18:17:18.440061+0000","last_became_active":"2024-01-15T18:17:17.125035+0000","last_became_peered":"2024-01-15T18:17:17.125035+0000","last_unstale":"2024-01-15T18:17:18.440061+0000","last_undegraded":"2024-01-15T18:17:18.440061+0000","last_fullsized":"2024-01-15T18:17:18.440061+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:24:25.383229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.131461+0000","last_change":"2024-01-15T18:17:16.102303+0000","last_active":"2024-01-15T18:17:18.131461+0000","last_peered":"2024-01-15T18:17:18.131461+0000","last_clean":"2024-01-15T18:17:18.131461+0000","last_became_active":"2024-01-15T18:17:16.099773+0000","last_became_peered":"2024-01-15T18:17:16.099773+0000","last_unstale":"2024-01-15T18:17:18.131461+0000","last_undegraded":"2024-01-15T18:17:18.131461+0000","last_fullsized":"2024-01-15T18:17:18.131461+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:40:51.210210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439104+0000","last_change":"2024-01-15T18:17:17.118751+0000","last_active":"2024-01-15T18:17:18.439104+0000","last_peered":"2024-01-15T18:17:18.439104+0000","last_clean":"2024-01-15T18:17:18.439104+0000","last_became_active":"2024-01-15T18:17:17.118486+0000","last_became_peered":"2024-01-15T18:17:17.118486+0000","last_unstale":"2024-01-15T18:17:18.439104+0000","last_undegraded":"2024-01-15T18:17:18.439104+0000","last_fullsized":"2024-01-15T18:17:18.439104+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:24:32.675015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.131385+0000","last_change":"2024-01-15T18:17:16.119288+0000","last_active":"2024-01-15T18:17:18.131385+0000","last_peered":"2024-01-15T18:17:18.131385+0000","last_clean":"2024-01-15T18:17:18.131385+0000","last_became_active":"2024-01-15T18:17:16.118972+0000","last_became_peered":"2024-01-15T18:17:16.118972+0000","last_unstale":"2024-01-15T18:17:18.131385+0000","last_undegraded":"2024-01-15T18:17:18.131385+0000","last_fullsized":"2024-01-15T18:17:18.131385+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:14:29.755549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439162+0000","last_change":"2024-01-15T18:17:17.131302+0000","last_active":"2024-01-15T18:17:18.439162+0000","last_peered":"2024-01-15T18:17:18.439162+0000","last_clean":"2024-01-15T18:17:18.439162+0000","last_became_active":"2024-01-15T18:17:17.130944+0000","last_became_peered":"2024-01-15T18:17:17.130944+0000","last_unstale":"2024-01-15T18:17:18.439162+0000","last_undegraded":"2024-01-15T18:17:18.439162+0000","last_fullsized":"2024-01-15T18:17:18.439162+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:45:26.435028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439494+0000","last_change":"2024-01-15T18:17:16.127573+0000","last_active":"2024-01-15T18:17:18.439494+0000","last_peered":"2024-01-15T18:17:18.439494+0000","last_clean":"2024-01-15T18:17:18.439494+0000","last_became_active":"2024-01-15T18:17:16.127194+0000","last_became_peered":"2024-01-15T18:17:16.127194+0000","last_unstale":"2024-01-15T18:17:18.439494+0000","last_undegraded":"2024-01-15T18:17:18.439494+0000","last_fullsized":"2024-01-15T18:17:18.439494+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:48:07.276946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440832+0000","last_change":"2024-01-15T18:17:17.132695+0000","last_active":"2024-01-15T18:17:18.440832+0000","last_peered":"2024-01-15T18:17:18.440832+0000","last_clean":"2024-01-15T18:17:18.440832+0000","last_became_active":"2024-01-15T18:17:17.132430+0000","last_became_peered":"2024-01-15T18:17:17.132430+0000","last_unstale":"2024-01-15T18:17:18.440832+0000","last_undegraded":"2024-01-15T18:17:18.440832+0000","last_fullsized":"2024-01-15T18:17:18.440832+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:14:02.211346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"96'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.132130+0000","last_change":"2024-01-15T18:17:16.102078+0000","last_active":"2024-01-15T18:17:18.132130+0000","last_peered":"2024-01-15T18:17:18.132130+0000","last_clean":"2024-01-15T18:17:18.132130+0000","last_became_active":"2024-01-15T18:17:16.101772+0000","last_became_peered":"2024-01-15T18:17:16.101772+0000","last_unstale":"2024-01-15T18:17:18.132130+0000","last_undegraded":"2024-01-15T18:17:18.132130+0000","last_fullsized":"2024-01-15T18:17:18.132130+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:28:05.931436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.127924+0000","last_change":"2024-01-15T18:17:17.126658+0000","last_active":"2024-01-15T18:17:18.127924+0000","last_peered":"2024-01-15T18:17:18.127924+0000","last_clean":"2024-01-15T18:17:18.127924+0000","last_became_active":"2024-01-15T18:17:17.126383+0000","last_became_peered":"2024-01-15T18:17:17.126383+0000","last_unstale":"2024-01-15T18:17:18.127924+0000","last_undegraded":"2024-01-15T18:17:18.127924+0000","last_fullsized":"2024-01-15T18:17:18.127924+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:50:07.301222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440855+0000","last_change":"2024-01-15T18:17:16.121035+0000","last_active":"2024-01-15T18:17:18.440855+0000","last_peered":"2024-01-15T18:17:18.440855+0000","last_clean":"2024-01-15T18:17:18.440855+0000","last_became_active":"2024-01-15T18:17:16.120714+0000","last_became_peered":"2024-01-15T18:17:16.120714+0000","last_unstale":"2024-01-15T18:17:18.440855+0000","last_undegraded":"2024-01-15T18:17:18.440855+0000","last_fullsized":"2024-01-15T18:17:18.440855+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:30:03.158862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.127748+0000","last_change":"2024-01-15T18:17:17.124340+0000","last_active":"2024-01-15T18:17:18.127748+0000","last_peered":"2024-01-15T18:17:18.127748+0000","last_clean":"2024-01-15T18:17:18.127748+0000","last_became_active":"2024-01-15T18:17:17.124040+0000","last_became_peered":"2024-01-15T18:17:17.124040+0000","last_unstale":"2024-01-15T18:17:18.127748+0000","last_undegraded":"2024-01-15T18:17:18.127748+0000","last_fullsized":"2024-01-15T18:17:18.127748+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:30:47.380860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440319+0000","last_change":"2024-01-15T18:17:16.114051+0000","last_active":"2024-01-15T18:17:18.440319+0000","last_peered":"2024-01-15T18:17:18.440319+0000","last_clean":"2024-01-15T18:17:18.440319+0000","last_became_active":"2024-01-15T18:17:16.113079+0000","last_became_peered":"2024-01-15T18:17:16.113079+0000","last_unstale":"2024-01-15T18:17:18.440319+0000","last_undegraded":"2024-01-15T18:17:18.440319+0000","last_fullsized":"2024-01-15T18:17:18.440319+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:25:22.584736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439887+0000","last_change":"2024-01-15T18:17:17.131965+0000","last_active":"2024-01-15T18:17:18.439887+0000","last_peered":"2024-01-15T18:17:18.439887+0000","last_clean":"2024-01-15T18:17:18.439887+0000","last_became_active":"2024-01-15T18:17:17.131693+0000","last_became_peered":"2024-01-15T18:17:17.131693+0000","last_unstale":"2024-01-15T18:17:18.439887+0000","last_undegraded":"2024-01-15T18:17:18.439887+0000","last_fullsized":"2024-01-15T18:17:18.439887+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:13:53.357957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.118968+0000","last_change":"2024-01-15T18:17:16.114737+0000","last_active":"2024-01-15T18:17:17.118968+0000","last_peered":"2024-01-15T18:17:17.118968+0000","last_clean":"2024-01-15T18:17:17.118968+0000","last_became_active":"2024-01-15T18:17:16.114426+0000","last_became_peered":"2024-01-15T18:17:16.114426+0000","last_unstale":"2024-01-15T18:17:17.118968+0000","last_undegraded":"2024-01-15T18:17:17.118968+0000","last_fullsized":"2024-01-15T18:17:17.118968+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:14.320865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439682+0000","last_change":"2024-01-15T18:17:17.115395+0000","last_active":"2024-01-15T18:17:18.439682+0000","last_peered":"2024-01-15T18:17:18.439682+0000","last_clean":"2024-01-15T18:17:18.439682+0000","last_became_active":"2024-01-15T18:17:17.115088+0000","last_became_peered":"2024-01-15T18:17:17.115088+0000","last_unstale":"2024-01-15T18:17:18.439682+0000","last_undegraded":"2024-01-15T18:17:18.439682+0000","last_fullsized":"2024-01-15T18:17:18.439682+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:38:55.775065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440562+0000","last_change":"2024-01-15T18:17:16.121786+0000","last_active":"2024-01-15T18:17:18.440562+0000","last_peered":"2024-01-15T18:17:18.440562+0000","last_clean":"2024-01-15T18:17:18.440562+0000","last_became_active":"2024-01-15T18:17:16.121508+0000","last_became_peered":"2024-01-15T18:17:16.121508+0000","last_unstale":"2024-01-15T18:17:18.440562+0000","last_undegraded":"2024-01-15T18:17:18.440562+0000","last_fullsized":"2024-01-15T18:17:18.440562+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:51:43.523481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.133059+0000","last_change":"2024-01-15T18:17:17.126022+0000","last_active":"2024-01-15T18:17:18.133059+0000","last_peered":"2024-01-15T18:17:18.133059+0000","last_clean":"2024-01-15T18:17:18.133059+0000","last_became_active":"2024-01-15T18:17:17.125366+0000","last_became_peered":"2024-01-15T18:17:17.125366+0000","last_unstale":"2024-01-15T18:17:18.133059+0000","last_undegraded":"2024-01-15T18:17:18.133059+0000","last_fullsized":"2024-01-15T18:17:18.133059+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:27:59.524326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.117607+0000","last_change":"2024-01-15T18:17:16.112593+0000","last_active":"2024-01-15T18:17:17.117607+0000","last_peered":"2024-01-15T18:17:17.117607+0000","last_clean":"2024-01-15T18:17:17.117607+0000","last_became_active":"2024-01-15T18:17:16.112226+0000","last_became_peered":"2024-01-15T18:17:16.112226+0000","last_unstale":"2024-01-15T18:17:17.117607+0000","last_undegraded":"2024-01-15T18:17:17.117607+0000","last_fullsized":"2024-01-15T18:17:17.117607+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:30:08.428553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439756+0000","last_change":"2024-01-15T18:17:17.114883+0000","last_active":"2024-01-15T18:17:18.439756+0000","last_peered":"2024-01-15T18:17:18.439756+0000","last_clean":"2024-01-15T18:17:18.439756+0000","last_became_active":"2024-01-15T18:17:17.114495+0000","last_became_peered":"2024-01-15T18:17:17.114495+0000","last_unstale":"2024-01-15T18:17:18.439756+0000","last_undegraded":"2024-01-15T18:17:18.439756+0000","last_fullsized":"2024-01-15T18:17:18.439756+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:16:26.664356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.112868+0000","last_change":"2024-01-15T18:17:16.122610+0000","last_active":"2024-01-15T18:17:17.112868+0000","last_peered":"2024-01-15T18:17:17.112868+0000","last_clean":"2024-01-15T18:17:17.112868+0000","last_became_active":"2024-01-15T18:17:16.122043+0000","last_became_peered":"2024-01-15T18:17:16.122043+0000","last_unstale":"2024-01-15T18:17:17.112868+0000","last_undegraded":"2024-01-15T18:17:17.112868+0000","last_fullsized":"2024-01-15T18:17:17.112868+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:26:13.337246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.439370+0000","last_change":"2024-01-15T18:17:17.133427+0000","last_active":"2024-01-15T18:17:18.439370+0000","last_peered":"2024-01-15T18:17:18.439370+0000","last_clean":"2024-01-15T18:17:18.439370+0000","last_became_active":"2024-01-15T18:17:17.133128+0000","last_became_peered":"2024-01-15T18:17:17.133128+0000","last_unstale":"2024-01-15T18:17:18.439370+0000","last_undegraded":"2024-01-15T18:17:18.439370+0000","last_fullsized":"2024-01-15T18:17:18.439370+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:54.730338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.106040+0000","last_change":"2024-01-15T18:17:16.120473+0000","last_active":"2024-01-15T18:17:17.106040+0000","last_peered":"2024-01-15T18:17:17.106040+0000","last_clean":"2024-01-15T18:17:17.106040+0000","last_became_active":"2024-01-15T18:17:16.119522+0000","last_became_peered":"2024-01-15T18:17:16.119522+0000","last_unstale":"2024-01-15T18:17:17.106040+0000","last_undegraded":"2024-01-15T18:17:17.106040+0000","last_fullsized":"2024-01-15T18:17:17.106040+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:55:35.000200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":11,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-15T18:17:18.440906+0000","last_change":"2024-01-15T18:17:17.121210+0000","last_active":"2024-01-15T18:17:18.440906+0000","last_peered":"2024-01-15T18:17:18.440906+0000","last_clean":"2024-01-15T18:17:18.440906+0000","last_became_active":"2024-01-15T18:17:17.120931+0000","last_became_peered":"2024-01-15T18:17:17.120931+0000","last_unstale":"2024-01-15T18:17:18.440906+0000","last_undegraded":"2024-01-15T18:17:18.440906+0000","last_fullsized":"2024-01-15T18:17:18.440906+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:16.086114+0000","last_clean_scrub_stamp":"2024-01-15T18:17:16.086114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:00:27.672303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"96'1","reported_seq":13,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.141961+0000","last_change":"2024-01-15T18:17:16.119481+0000","last_active":"2024-01-15T18:17:17.141961+0000","last_peered":"2024-01-15T18:17:17.141961+0000","last_clean":"2024-01-15T18:17:17.141961+0000","last_became_active":"2024-01-15T18:17:16.119190+0000","last_became_peered":"2024-01-15T18:17:16.119190+0000","last_unstale":"2024-01-15T18:17:17.141961+0000","last_undegraded":"2024-01-15T18:17:17.141961+0000","last_fullsized":"2024-01-15T18:17:17.141961+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:17:15.077857+0000","last_clean_scrub_stamp":"2024-01-15T18:17:15.077857+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:22.935829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":230,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-01-15T18:17:17.115466+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:17:17.115466+0000","last_peered":"2024-01-15T18:17:17.115466+0000","last_clean":"2024-01-15T18:17:17.115466+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:17:17.115466+0000","last_undegraded":"2024-01-15T18:17:17.115466+0000","last_fullsized":"2024-01-15T18:17:17.115466+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509594,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64804,"kb_used_data":1752,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307036,"statfs":{"total":96636764160,"available":96570404864,"internally_reserved":0,"allocated":1794048,"data_stored":1108279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64551370},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.85299999999999998,"15min":0.874},"min":{"1min":0.38700000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92900000000000005,"5min":9.3040000000000003,"15min":10.284000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.86699999999999999,"15min":0.92700000000000005},"min":{"1min":0.42599999999999999,"5min":0.38400000000000001,"15min":0.378},"max":{"1min":2.145,"5min":6.6189999999999998,"15min":16.968},"last":0.85699999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.91000000000000003,"15min":0.92800000000000005},"min":{"1min":0.46800000000000003,"5min":0.36099999999999999,"15min":0.33800000000000002},"max":{"1min":0.90000000000000002,"5min":9.3829999999999991,"15min":15.791},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.86599999999999999,"15min":0.91800000000000004},"min":{"1min":0.505,"5min":0.371,"15min":0.371},"max":{"1min":2.1890000000000001,"5min":5.0899999999999999,"15min":10.842000000000001},"last":0.75800000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":1.1539999999999999,"15min":0.93799999999999994},"min":{"1min":0.51500000000000001,"5min":0.40400000000000003,"15min":0.378},"max":{"1min":1.1619999999999999,"5min":22.318000000000001,"15min":22.318000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.91300000000000003,"15min":0.90600000000000003},"min":{"1min":0.51400000000000001,"5min":0.433,"15min":0.39100000000000001},"max":{"1min":0.95499999999999996,"5min":9.5169999999999995,"15min":9.7539999999999996},"last":0.73199999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":1.151,"15min":0.99199999999999999},"min":{"1min":0.53800000000000003,"5min":0.50900000000000001,"15min":0.38100000000000001},"max":{"1min":1.1160000000000001,"5min":18.152999999999999,"15min":18.152999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.95499999999999996,"15min":0.98399999999999999},"min":{"1min":0.54300000000000004,"5min":0.48499999999999999,"15min":0.43099999999999999},"max":{"1min":1.0860000000000001,"5min":9.75,"15min":10.34},"last":0.80100000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.879,"15min":0.94299999999999995},"min":{"1min":0.38900000000000001,"5min":0.34599999999999997,"15min":0.314},"max":{"1min":2.077,"5min":8.4420000000000002,"15min":20.387},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":1.1639999999999999,"15min":0.98399999999999999},"min":{"1min":0.40899999999999997,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.91300000000000003,"5min":17.215,"15min":17.215},"last":0.70099999999999996}]},{"osd":5,"last update":"Mon Jan 15 18:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.93200000000000005,"15min":0.94399999999999995},"min":{"1min":0.41499999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.89800000000000002,"5min":9.6630000000000003,"15min":19.959},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.92900000000000005,"15min":0.98499999999999999},"min":{"1min":0.45300000000000001,"5min":0.39400000000000002,"15min":0.35799999999999998},"max":{"1min":2.105,"5min":7.6980000000000004,"15min":19.905000000000001},"last":0.64200000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:17:17 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":1.004,"15min":1.159},"min":{"1min":0.50800000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":2.2469999999999999,"5min":8.6609999999999996,"15min":22.478999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.69699999999999995,"5min":1.1499999999999999,"15min":1.0629999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.38300000000000001},"max":{"1min":1.032,"5min":17.132000000000001,"15min":21.509},"last":0.78200000000000003}]}]},{"osd":6,"up_from":15,"seq":64424509594,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64228,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307612,"statfs":{"total":96636764160,"available":96570994688,"internally_reserved":0,"allocated":1200128,"data_stored":518451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":1.0529999999999999,"15min":0.97799999999999998},"min":{"1min":0.40000000000000002,"5min":0.32800000000000001,"15min":0.29099999999999998},"max":{"1min":0.86699999999999999,"5min":32.587000000000003,"15min":32.587000000000003},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.91600000000000004,"15min":0.86299999999999999},"min":{"1min":0.41999999999999998,"5min":0.38500000000000001,"15min":0.35399999999999998},"max":{"1min":0.88400000000000001,"5min":16.193000000000001,"15min":16.193000000000001},"last":0.76200000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.82499999999999996,"15min":0.99399999999999999},"min":{"1min":0.40600000000000003,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95299999999999996,"5min":7.7649999999999997,"15min":53.451999999999998},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1,"15min":0.91800000000000004},"min":{"1min":0.49299999999999999,"5min":0.437,"15min":0.42899999999999999},"max":{"1min":0.89900000000000002,"5min":32.512,"15min":32.512},"last":0.56499999999999995}]},{"osd":2,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":1.087,"15min":0.97099999999999997},"min":{"1min":0.48299999999999998,"5min":0.41699999999999998,"15min":0.38900000000000001},"max":{"1min":0.90100000000000002,"5min":32.472999999999999,"15min":32.472999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.997,"15min":0.89600000000000002},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.38500000000000001},"max":{"1min":0.97899999999999998,"5min":16.602,"15min":16.602},"last":0.63400000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.97999999999999998,"15min":0.93100000000000005},"min":{"1min":0.50700000000000001,"5min":0.49399999999999999,"15min":0.41399999999999998},"max":{"1min":0.94499999999999995,"5min":16.388000000000002,"15min":16.388000000000002},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.879,"15min":1.079},"min":{"1min":0.52300000000000002,"5min":0.432,"15min":0.41599999999999998},"max":{"1min":0.996,"5min":5.9740000000000002,"15min":53.398000000000003},"last":0.52800000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.98799999999999999,"15min":0.89100000000000001},"min":{"1min":0.42999999999999999,"5min":0.38100000000000001,"15min":0.34999999999999998},"max":{"1min":0.92200000000000004,"5min":32.549999999999997,"15min":32.549999999999997},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.878,"15min":0.83999999999999997},"min":{"1min":0.42299999999999999,"5min":0.38700000000000001,"15min":0.34499999999999997},"max":{"1min":1.087,"5min":11.852,"15min":11.852},"last":0.68400000000000005}]},{"osd":5,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.82699999999999996,"15min":1.0369999999999999},"min":{"1min":0.36199999999999999,"5min":0.35799999999999998,"15min":0.30399999999999999},"max":{"1min":1.1759999999999999,"5min":7.5650000000000004,"15min":53.229999999999997},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.081,"15min":1.028},"min":{"1min":0.432,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.228,"5min":32.404000000000003,"15min":32.404000000000003},"last":0.83699999999999997}]},{"osd":7,"last update":"Mon Jan 15 18:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.93899999999999995,"15min":1.105},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.1220000000000001,"5min":16.565000000000001,"15min":45.070999999999998},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":1.161,"15min":1.21},"min":{"1min":0.46300000000000002,"5min":0.40999999999999998,"15min":0.34999999999999998},"max":{"1min":1.032,"5min":40.972999999999999,"15min":53.338000000000001},"last":0.746}]}]},{"osd":4,"up_from":15,"seq":64424509594,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76724,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295116,"statfs":{"total":96636764160,"available":96558198784,"internally_reserved":0,"allocated":1212416,"data_stored":519535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.82499999999999996,"15min":0.88400000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.32900000000000001},"max":{"1min":0.91800000000000004,"5min":7.3380000000000001,"15min":23.318000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.70199999999999996,"15min":0.79300000000000004},"min":{"1min":0.372,"5min":0.36399999999999999,"15min":0.32600000000000001},"max":{"1min":0.82699999999999996,"5min":6.444,"15min":14.723000000000001},"last":0.72399999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.82699999999999996,"15min":0.78800000000000003},"min":{"1min":0.39600000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.2,"5min":9.5700000000000003,"15min":9.5700000000000003},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.92100000000000004,"15min":0.91600000000000004},"min":{"1min":0.45200000000000001,"5min":0.40200000000000002,"15min":0.36899999999999999},"max":{"1min":0.84099999999999997,"5min":7.9980000000000002,"15min":22.408999999999999},"last":0.65600000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.85399999999999998,"15min":0.79900000000000004},"min":{"1min":0.439,"5min":0.439,"15min":0.34399999999999997},"max":{"1min":1.266,"5min":9.7089999999999996,"15min":9.7089999999999996},"last":0.629},{"interface":"front","average":{"1min":0.73699999999999999,"5min":1.0649999999999999,"15min":1.0149999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.39100000000000001},"max":{"1min":1.714,"5min":14.118,"15min":22.338999999999999},"last":0.77000000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.77800000000000002,"15min":0.872},"min":{"1min":0.437,"5min":0.437,"15min":0.41199999999999998},"max":{"1min":0.88800000000000001,"5min":6.7930000000000001,"15min":14.763999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.94199999999999995,"15min":0.90600000000000003},"min":{"1min":0.504,"5min":0.48099999999999998,"15min":0.45500000000000002},"max":{"1min":0.93600000000000005,"5min":9.6289999999999996,"15min":9.6289999999999996},"last":0.81399999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.94599999999999995,"15min":0.95699999999999996},"min":{"1min":0.40000000000000002,"5min":0.34799999999999998,"15min":0.33200000000000002},"max":{"1min":0.94099999999999995,"5min":15.787000000000001,"15min":31.617999999999999},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.75,"15min":0.86399999999999999},"min":{"1min":0.42599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.95299999999999996,"5min":6.7169999999999996,"15min":17.145},"last":0.79600000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.84899999999999998,"15min":0.82799999999999996},"min":{"1min":0.42299999999999999,"5min":0.39800000000000002,"15min":0.35499999999999998},"max":{"1min":0.97799999999999998,"5min":9.6709999999999994,"15min":11.124000000000001},"last":0.751},{"interface":"front","average":{"1min":0.69799999999999995,"5min":1.1699999999999999,"15min":1.2230000000000001},"min":{"1min":0.56599999999999995,"5min":0.438,"15min":0.40200000000000002},"max":{"1min":0.90200000000000002,"5min":32.308,"15min":32.308},"last":0.82299999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.749,"15min":1.0029999999999999},"min":{"1min":0.434,"5min":0.41099999999999998,"15min":0.34799999999999998},"max":{"1min":0.91900000000000004,"5min":6.8390000000000004,"15min":39.256},"last":0.876},{"interface":"front","average":{"1min":0.70399999999999996,"5min":1.1160000000000001,"15min":1.004},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.42199999999999999},"max":{"1min":0.86199999999999999,"5min":16.277000000000001,"15min":26.573},"last":0.86299999999999999}]}]},{"osd":3,"up_from":15,"seq":64424509593,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64236,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307604,"statfs":{"total":96636764160,"available":96570986496,"internally_reserved":0,"allocated":1208320,"data_stored":519433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64551367},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.44900000000000001,"15min":0.48899999999999999},"min":{"1min":0.27300000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.624,"5min":2.1160000000000001,"15min":5.8849999999999998},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.54100000000000004,"15min":0.61599999999999999},"min":{"1min":0.312,"5min":0.29799999999999999,"15min":0.29499999999999998},"max":{"1min":0.66600000000000004,"5min":3.0649999999999999,"15min":17.756},"last":0.42899999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.495,"15min":0.58499999999999996},"min":{"1min":0.28899999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68700000000000006,"5min":3.6389999999999998,"15min":12.307},"last":0.374},{"interface":"front","average":{"1min":0.503,"5min":0.55500000000000005,"15min":0.58499999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.70299999999999996,"5min":3.738,"15min":6.0640000000000001},"last":0.49099999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.65400000000000003,"15min":0.68999999999999995},"min":{"1min":0.33700000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.69999999999999996,"5min":5.431,"15min":12.468999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.70299999999999996,"15min":0.68300000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.34599999999999997},"max":{"1min":0.77800000000000002,"5min":10.837999999999999,"15min":10.837999999999999},"last":0.45200000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":1.0609999999999999,"15min":1.014},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.375},"max":{"1min":1.024,"5min":13.863,"15min":17.853999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.94999999999999996,"15min":0.96199999999999997},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.47099999999999997},"max":{"1min":0.94599999999999995,"5min":6.8289999999999997,"15min":13.31},"last":0.70999999999999996}]},{"osd":5,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.153,"15min":0.97399999999999998},"min":{"1min":0.54200000000000004,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1020000000000001,"5min":17.687999999999999,"15min":17.687999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.88500000000000001,"15min":0.96999999999999997},"min":{"1min":0.45500000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.085,"5min":5.0949999999999998,"15min":20.515999999999998},"last":0.64000000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.80200000000000005,"15min":0.97699999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.432},"max":{"1min":1.05,"5min":4.0910000000000002,"15min":23.100999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.90000000000000002,"15min":0.93600000000000005},"min":{"1min":0.59399999999999997,"5min":0.53600000000000003,"15min":0.49399999999999999},"max":{"1min":1.0069999999999999,"5min":6.7220000000000004,"15min":12.76},"last":0.73499999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":1.331,"15min":1.1659999999999999},"min":{"1min":0.503,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.0089999999999999,"5min":24.321999999999999,"15min":24.321999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":2.0510000000000002,"5min":1.139,"15min":1.1020000000000001},"min":{"1min":0.54700000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":23.013000000000002,"5min":23.013000000000002,"15min":23.013000000000002},"last":0.67700000000000005}]}]},{"osd":2,"up_from":15,"seq":64424509593,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64224,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307616,"statfs":{"total":96636764160,"available":96570998784,"internally_reserved":0,"allocated":1196032,"data_stored":517833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.45500000000000002,"15min":0.50900000000000001},"min":{"1min":0.26500000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.71299999999999997,"5min":0.98199999999999998,"15min":5.1849999999999996},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.49199999999999999,"15min":0.53200000000000003},"min":{"1min":0.315,"5min":0.252,"15min":0.252},"max":{"1min":0.68899999999999995,"5min":1.1579999999999999,"15min":7.516},"last":0.36499999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.55800000000000005,"15min":0.61599999999999999},"min":{"1min":0.39800000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.83299999999999996,"5min":0.95799999999999996,"15min":10.907},"last":0.504},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.50800000000000001,"15min":0.59199999999999997},"min":{"1min":0.38200000000000001,"5min":0.28100000000000003,"15min":0.27000000000000002},"max":{"1min":0.77500000000000002,"5min":1.2849999999999999,"15min":8.9190000000000005},"last":0.41899999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55500000000000005,"15min":0.623},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0249999999999999,"5min":1.133,"15min":12.185},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.60199999999999998,"15min":0.66400000000000003},"min":{"1min":0.40600000000000003,"5min":0.373,"15min":0.34000000000000002},"max":{"1min":0.79000000000000004,"5min":0.96699999999999997,"15min":7.6760000000000002},"last":0.46300000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.88800000000000001,"15min":0.875},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":4.0190000000000001,"5min":4.3150000000000004,"15min":8.6609999999999996},"last":0.752},{"interface":"front","average":{"1min":0.745,"5min":0.76200000000000001,"15min":0.90300000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.5860000000000001,"5min":2.468,"15min":19.146999999999998},"last":0.64900000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":0.79700000000000004,"15min":0.90800000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":5.0309999999999997,"5min":5.0309999999999997,"15min":31.710000000000001},"last":0.876},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.83999999999999997,"15min":0.999},"min":{"1min":0.51100000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.3660000000000001,"5min":3.6459999999999999,"15min":22.539000000000001},"last":0.70699999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.84099999999999997,"15min":0.90600000000000003},"min":{"1min":0.499,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":4.1959999999999997,"5min":4.1959999999999997,"15min":9.4540000000000006},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.79200000000000004,"15min":0.90600000000000003},"min":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.633,"5min":1.633,"15min":18.614999999999998},"last":0.81899999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.97799999999999998,"15min":0.96199999999999997},"min":{"1min":0.52100000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":4.3170000000000002,"5min":11.606,"15min":11.606},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.159,"5min":0.92900000000000005,"15min":1.004},"min":{"1min":0.54900000000000004,"5min":0.52500000000000002,"15min":0.502},"max":{"1min":6.1260000000000003,"5min":6.1260000000000003,"15min":17.013000000000002},"last":0.89600000000000002}]}]},{"osd":1,"up_from":15,"seq":64424509595,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68380,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303460,"statfs":{"total":96636764160,"available":96566743040,"internally_reserved":0,"allocated":1191936,"data_stored":517799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.51700000000000002,"15min":0.65700000000000003},"min":{"1min":0.28399999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.6140000000000001,"5min":4.1550000000000002,"15min":19.577000000000002},"last":0.375},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.623,"15min":0.76000000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.251},"max":{"1min":0.83899999999999997,"5min":12.378,"15min":48.277999999999999},"last":0.40000000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55400000000000005,"15min":0.70799999999999996},"min":{"1min":0.32800000000000001,"5min":0.309,"15min":0.30299999999999999},"max":{"1min":1.8049999999999999,"5min":2.8610000000000002,"15min":40.698},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59099999999999997,"15min":0.877},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.317},"max":{"1min":1.978,"5min":2.5139999999999998,"15min":48.316000000000003},"last":0.58299999999999996}]},{"osd":3,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.623,"15min":0.82999999999999996},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.33700000000000002},"max":{"1min":0.94699999999999995,"5min":2.8620000000000001,"15min":48.234999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.69299999999999995,"15min":0.67700000000000005},"min":{"1min":0.33400000000000002,"5min":0.315,"15min":0.311},"max":{"1min":1.0780000000000001,"5min":9.9719999999999995,"15min":9.9719999999999995},"last":0.55900000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.90300000000000002,"15min":0.88800000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.44500000000000001},"max":{"1min":0.97499999999999998,"5min":9.9329999999999998,"15min":9.9329999999999998},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.86199999999999999,"15min":0.93300000000000005},"min":{"1min":0.502,"5min":0.45800000000000002,"15min":0.42499999999999999},"max":{"1min":2.3900000000000001,"5min":9.8610000000000007,"15min":19.829000000000001},"last":1.012}]},{"osd":5,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.82899999999999996,"15min":1.0129999999999999},"min":{"1min":0.53700000000000003,"5min":0.48599999999999999,"15min":0.46100000000000002},"max":{"1min":2.5960000000000001,"5min":8.5839999999999996,"15min":22.594999999999999},"last":1.212},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.80600000000000005,"15min":1.157},"min":{"1min":0.44400000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.6879999999999999,"5min":2.9609999999999999,"15min":48.198},"last":1.1040000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.90800000000000003,"15min":1.04},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0189999999999999,"5min":9.3989999999999991,"15min":35.112000000000002},"last":1.2350000000000001},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.91300000000000003,"15min":0.96899999999999997},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.50800000000000001},"max":{"1min":2.512,"5min":5.8849999999999998,"15min":19.780999999999999},"last":0.91300000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.89200000000000002,"15min":1.1619999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.93300000000000005,"5min":11.039,"15min":48.148000000000003},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.748,"5min":0.97899999999999998,"15min":0.98999999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.52100000000000002},"max":{"1min":1.0609999999999999,"5min":9.8640000000000008,"15min":9.8640000000000008},"last":1.1559999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542299,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58724,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313116,"statfs":{"total":96636764160,"available":96576630784,"internally_reserved":0,"allocated":1196032,"data_stored":517869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.81200000000000006,"15min":0.80100000000000005},"min":{"1min":0.375,"5min":0.29899999999999999,"15min":0.28999999999999998},"max":{"1min":1.383,"5min":5.1639999999999997,"15min":21.385000000000002},"last":0.502},{"interface":"front","average":{"1min":0.748,"5min":0.78700000000000003,"15min":0.877},"min":{"1min":0.57999999999999996,"5min":0.378,"15min":0.34899999999999998},"max":{"1min":1.3029999999999999,"5min":5.3869999999999996,"15min":9.359},"last":0.71399999999999997}]},{"osd":1,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.82199999999999995,"15min":0.89500000000000002},"min":{"1min":0.44400000000000001,"5min":0.375,"15min":0.34899999999999998},"max":{"1min":1.2070000000000001,"5min":3.7429999999999999,"15min":21.120000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.78500000000000003,"5min":1.1850000000000001,"15min":0.96999999999999997},"min":{"1min":0.48999999999999999,"5min":0.38600000000000001,"15min":0.315},"max":{"1min":1.532,"5min":15.843,"15min":21.5},"last":0.69299999999999995}]},{"osd":2,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.83799999999999997,"15min":0.89800000000000002},"min":{"1min":0.47699999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.421,"5min":5.516,"15min":8.7750000000000004},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.90800000000000003,"15min":0.98799999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.347,"5min":8.3529999999999998,"15min":21.201000000000001},"last":0.46000000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.93200000000000005,"15min":0.91300000000000003},"min":{"1min":0.58499999999999996,"5min":0.37,"15min":0.37},"max":{"1min":1.478,"5min":7.7199999999999998,"15min":21.457000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.85499999999999998,"15min":0.96299999999999997},"min":{"1min":0.57199999999999995,"5min":0.47699999999999998,"15min":0.45400000000000001},"max":{"1min":1.5,"5min":5.6109999999999998,"15min":8.7300000000000004},"last":0.54600000000000004}]},{"osd":4,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.93500000000000005,"15min":0.88800000000000001},"min":{"1min":0.376,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":4.8650000000000002,"5min":10.962,"15min":10.962},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.78800000000000003,"15min":1.018},"min":{"1min":0.442,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.464,"5min":4.6639999999999997,"15min":26.146999999999998},"last":0.63900000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.86599999999999999,"15min":0.83799999999999997},"min":{"1min":0.44700000000000001,"5min":0.36099999999999999,"15min":0.32000000000000001},"max":{"1min":2.8940000000000001,"5min":5.2160000000000002,"15min":21.535},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.85899999999999999,"15min":1.0129999999999999},"min":{"1min":0.432,"5min":0.379,"15min":0.36299999999999999},"max":{"1min":4.9139999999999997,"5min":4.9139999999999997,"15min":30.488},"last":0.748}]},{"osd":7,"last update":"Mon Jan 15 18:17:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.276,"15min":1.2290000000000001},"min":{"1min":0.51800000000000002,"5min":0.432,"15min":0.39700000000000002},"max":{"1min":8.7539999999999996,"5min":33.499000000000002,"15min":33.499000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.98599999999999999,"15min":1.0900000000000001},"min":{"1min":0.46200000000000002,"5min":0.441,"15min":0.36499999999999999},"max":{"1min":4.3120000000000003,"5min":6.306,"15min":25.437999999999999},"last":0.65900000000000003}]}]},{"osd":0,"up_from":14,"seq":60129542298,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64800,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307040,"statfs":{"total":96636764160,"available":96570408960,"internally_reserved":0,"allocated":1789952,"data_stored":1108749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.52400000000000002,"15min":0.627},"min":{"1min":0.26100000000000001,"5min":0.245,"15min":0.23699999999999999},"max":{"1min":0.93500000000000005,"5min":7.907,"15min":19.843},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58199999999999996,"15min":0.67400000000000004},"min":{"1min":0.35599999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.8240000000000001,"5min":3.512,"15min":10.154},"last":0.33100000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.71599999999999997,"15min":0.80100000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.30399999999999999},"max":{"1min":1.4410000000000001,"5min":7.0810000000000004,"15min":25.905000000000001},"last":0.379},{"interface":"front","average":{"1min":0.498,"5min":0.57499999999999996,"15min":0.61399999999999999},"min":{"1min":0.32700000000000001,"5min":0.27800000000000002,"15min":0.26700000000000002},"max":{"1min":0.79300000000000004,"5min":3.8500000000000001,"15min":9.3829999999999991},"last":0.39600000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.69399999999999995,"15min":0.74199999999999999},"min":{"1min":0.34699999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":2.3519999999999999,"5min":6.2850000000000001,"15min":9.1780000000000008},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.70199999999999996,"15min":0.76200000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.36899999999999999},"max":{"1min":1.327,"5min":6.5979999999999999,"15min":10.210000000000001},"last":0.44400000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.84199999999999997,"15min":0.98399999999999999},"min":{"1min":0.47699999999999998,"5min":0.39100000000000001,"15min":0.38},"max":{"1min":1.2929999999999999,"5min":4.5179999999999998,"15min":22.814},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.83399999999999996,"15min":1.0289999999999999},"min":{"1min":0.502,"5min":0.502,"15min":0.41199999999999998},"max":{"1min":1.2749999999999999,"5min":4.2329999999999997,"15min":23.341999999999999},"last":0.57999999999999996}]},{"osd":5,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.86399999999999999,"15min":0.96999999999999997},"min":{"1min":0.54400000000000004,"5min":0.51600000000000001,"15min":0.441},"max":{"1min":2.6190000000000002,"5min":4.008,"15min":12.428000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.79900000000000004,"5min":1.252,"15min":1.258},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.3120000000000001,"5min":23.021000000000001,"15min":23.021000000000001},"last":0.66100000000000003}]},{"osd":6,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.92000000000000004,"15min":1.212},"min":{"1min":0.53500000000000003,"5min":0.48399999999999999,"15min":0.46200000000000002},"max":{"1min":1.6359999999999999,"5min":6.5250000000000004,"15min":31.059000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.94799999999999995,"15min":1.089},"min":{"1min":0.59399999999999997,"5min":0.52700000000000002,"15min":0.49099999999999999},"max":{"1min":2.5659999999999998,"5min":6.9690000000000003,"15min":40.731000000000002},"last":0.623}]},{"osd":7,"last update":"Mon Jan 15 18:17:05 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":1.107,"15min":1.2},"min":{"1min":0.59499999999999997,"5min":0.499,"15min":0.42499999999999999},"max":{"1min":2.6539999999999999,"5min":10.497999999999999,"15min":24.492999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.88300000000000001,"15min":1.103},"min":{"1min":0.55300000000000005,"5min":0.495,"15min":0.46600000000000003},"max":{"1min":1.3740000000000001,"5min":4.6600000000000001,"15min":23.524000000000001},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:17:26.174 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-15T18:17:26.174 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-15T18:17:26.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.362+0000 7f661e200640 1 Processor -- start 2024-01-15T18:17:26.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.362+0000 7f661e200640 1 -- start start 2024-01-15T18:17:26.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.366+0000 7f661e200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f661815ad20 0x7f661815b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:26.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.366+0000 7f661e200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f661815d900 0x7f661815ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:26.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.366+0000 7f661e200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661815a610 con 0x7f6618070ee0 2024-01-15T18:17:26.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.366+0000 7f661e200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f661815e3c0 con 0x7f661815d900 2024-01-15T18:17:26.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.366+0000 7f661e200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f661815e6a0 con 0x7f661815ad20 2024-01-15T18:17:26.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.366+0000 7f6616a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f661815d900 0x7f661815ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:26.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.366+0000 7f6616a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f661815d900 0x7f661815ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55348/0 (socket says 172.21.15.27:55348) 2024-01-15T18:17:26.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.366+0000 7f6616a00640 1 -- 172.21.15.27:0/455694991 learned_addr learned my addr 172.21.15.27:0/455694991 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:26.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.366+0000 7f6617400640 1 --2- 172.21.15.27:0/455694991 >> v2:172.21.15.93:3300/0 conn(0x7f661815ad20 0x7f661815b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:26.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.366+0000 7f6616a00640 1 -- 172.21.15.27:0/455694991 >> v2:172.21.15.93:3300/0 conn(0x7f661815ad20 msgr2=0x7f661815b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:26.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.366+0000 7f6616a00640 1 --2- 172.21.15.27:0/455694991 >> v2:172.21.15.93:3300/0 conn(0x7f661815ad20 0x7f661815b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:26.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.366+0000 7f6616a00640 1 -- 172.21.15.27:0/455694991 >> v1:172.21.15.93:6789/0 conn(0x7f6618070ee0 legacy=0x7f661806d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:26.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.366+0000 7f6616a00640 1 -- 172.21.15.27:0/455694991 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661815b6e0 con 0x7f661815d900 2024-01-15T18:17:26.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.370+0000 7f6616a00640 1 --2- 172.21.15.27:0/455694991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661815d900 0x7f661815ddb0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f660000bb10 tx=0x7f660001eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e2092d6208ef888 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:26.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.370+0000 7f6616000640 1 -- 172.21.15.27:0/455694991 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66000a7c50 con 0x7f661815d900 2024-01-15T18:17:26.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.370+0000 7f6616000640 1 -- 172.21.15.27:0/455694991 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66000a7db0 con 0x7f661815d900 2024-01-15T18:17:26.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.370+0000 7f6616000640 1 -- 172.21.15.27:0/455694991 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66000af660 con 0x7f661815d900 2024-01-15T18:17:26.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.370+0000 7f661e200640 1 -- 172.21.15.27:0/455694991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661815d900 msgr2=0x7f661815ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:26.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.370+0000 7f661e200640 1 --2- 172.21.15.27:0/455694991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661815d900 0x7f661815ddb0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f660000bb10 tx=0x7f660001eb60 comp rx=0 tx=0).stop 2024-01-15T18:17:26.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.370+0000 7f661e200640 1 -- 172.21.15.27:0/455694991 shutdown_connections 2024-01-15T18:17:26.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.370+0000 7f661e200640 1 --2- 172.21.15.27:0/455694991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661815d900 0x7f661815ddb0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:26.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.370+0000 7f661e200640 1 --2- 172.21.15.27:0/455694991 >> v2:172.21.15.93:3300/0 conn(0x7f661815ad20 0x7f661815b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:26.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.370+0000 7f661e200640 1 -- 172.21.15.27:0/455694991 >> 172.21.15.27:0/455694991 conn(0x7f66180cb700 msgr2=0x7f66180cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:26.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.370+0000 7f661e200640 1 -- 172.21.15.27:0/455694991 shutdown_connections 2024-01-15T18:17:26.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.370+0000 7f661e200640 1 -- 172.21.15.27:0/455694991 wait complete. 2024-01-15T18:17:26.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.370+0000 7f661e200640 1 Processor -- start 2024-01-15T18:17:26.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.370+0000 7f661e200640 1 -- start start 2024-01-15T18:17:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.370+0000 7f661e200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6618070ee0 0x7f6618163830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.374+0000 7f661e200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f661815ad20 0x7f66180ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.374+0000 7f661e200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661815d900 0x7f6618153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.374+0000 7f661e200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6618156030 con 0x7f661815d900 2024-01-15T18:17:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.374+0000 7f661e200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6618155420 con 0x7f661815ad20 2024-01-15T18:17:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.374+0000 7f661e200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6618155700 con 0x7f6618070ee0 2024-01-15T18:17:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.374+0000 7f6617e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661815d900 0x7f6618153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.374+0000 7f6617e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661815d900 0x7f6618153570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55364/0 (socket says 172.21.15.27:55364) 2024-01-15T18:17:26.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.374+0000 7f6617e00640 1 -- 172.21.15.27:0/1972986120 learned_addr learned my addr 172.21.15.27:0/1972986120 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.374+0000 7f6617400640 1 --2- 172.21.15.27:0/1972986120 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6618070ee0 0x7f6618163830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:26.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.374+0000 7f6616a00640 1 --2- 172.21.15.27:0/1972986120 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f661815ad20 0x7f66180ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.374+0000 7f6617e00640 1 -- 172.21.15.27:0/1972986120 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6618070ee0 msgr2=0x7f6618163830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.374+0000 7f6617e00640 1 --2- 172.21.15.27:0/1972986120 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6618070ee0 0x7f6618163830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.374+0000 7f6617e00640 1 -- 172.21.15.27:0/1972986120 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f661815ad20 msgr2=0x7f66180ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.374+0000 7f6617e00640 1 --2- 172.21.15.27:0/1972986120 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f661815ad20 0x7f66180ca390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.374+0000 7f6617e00640 1 -- 172.21.15.27:0/1972986120 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f660000b770 con 0x7f661815d900 2024-01-15T18:17:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.374+0000 7f6617e00640 1 --2- 172.21.15.27:0/1972986120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661815d900 0x7f6618153570 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f661000c780 tx=0x7f661000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.374+0000 7f660f400640 1 -- 172.21.15.27:0/1972986120 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6610004b70 con 0x7f661815d900 2024-01-15T18:17:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.374+0000 7f660f400640 1 -- 172.21.15.27:0/1972986120 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f661000ce90 con 0x7f661815d900 2024-01-15T18:17:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.374+0000 7f660f400640 1 -- 172.21.15.27:0/1972986120 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661000a850 con 0x7f661815d900 2024-01-15T18:17:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.374+0000 7f661e200640 1 -- 172.21.15.27:0/1972986120 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66181559e0 con 0x7f661815d900 2024-01-15T18:17:26.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.374+0000 7f661e200640 1 -- 172.21.15.27:0/1972986120 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6618163d70 con 0x7f661815d900 2024-01-15T18:17:26.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.374+0000 7f661e200640 1 -- 172.21.15.27:0/1972986120 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66181561a0 con 0x7f661815d900 2024-01-15T18:17:26.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.374+0000 7f660f400640 1 -- 172.21.15.27:0/1972986120 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f661000a9b0 con 0x7f661815d900 2024-01-15T18:17:26.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.378+0000 7f660f400640 1 --2- 172.21.15.27:0/1972986120 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f65f0067730 0x7f65f0069bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:26.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.378+0000 7f660f400640 1 -- 172.21.15.27:0/1972986120 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f6610083320 con 0x7f661815d900 2024-01-15T18:17:26.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.378+0000 7f6617400640 1 --2- 172.21.15.27:0/1972986120 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f65f0067730 0x7f65f0069bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:26.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.378+0000 7f6617400640 1 --2- 172.21.15.27:0/1972986120 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f65f0067730 0x7f65f0069bf0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f660800b9d0 tx=0x7f6608008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:26.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.382+0000 7f660f400640 1 -- 172.21.15.27:0/1972986120 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6610051cf0 con 0x7f661815d900 2024-01-15T18:17:26.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.590+0000 7f661e200640 1 -- 172.21.15.27:0/1972986120 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f661804e850 con 0x7f661815d900 2024-01-15T18:17:26.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.594+0000 7f660f400640 1 -- 172.21.15.27:0/1972986120 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7f6610055310 con 0x7f661815d900 2024-01-15T18:17:26.632 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-01-15T18:17:26.632 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOc6VlH+DCLxAA1sKVut+3WjA/B+aYJ6/Jeg== 2024-01-15T18:17:26.632 INFO:teuthology.orchestra.run.smithi027.stdout: caps mds = "allow" 2024-01-15T18:17:26.633 INFO:teuthology.orchestra.run.smithi027.stdout: caps mon = "allow r" 2024-01-15T18:17:26.633 INFO:teuthology.orchestra.run.smithi027.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-15T18:17:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-01-15T18:17:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.598+0000 7f661e200640 1 -- 172.21.15.27:0/1972986120 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f65f0067730 msgr2=0x7f65f0069bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.598+0000 7f661e200640 1 --2- 172.21.15.27:0/1972986120 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f65f0067730 0x7f65f0069bf0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f660800b9d0 tx=0x7f6608008040 comp rx=0 tx=0).stop 2024-01-15T18:17:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.598+0000 7f661e200640 1 -- 172.21.15.27:0/1972986120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661815d900 msgr2=0x7f6618153570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.598+0000 7f661e200640 1 --2- 172.21.15.27:0/1972986120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661815d900 0x7f6618153570 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f661000c780 tx=0x7f661000cc40 comp rx=0 tx=0).stop 2024-01-15T18:17:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.598+0000 7f661e200640 1 -- 172.21.15.27:0/1972986120 shutdown_connections 2024-01-15T18:17:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.598+0000 7f661e200640 1 --2- 172.21.15.27:0/1972986120 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f65f0067730 0x7f65f0069bf0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:26.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.598+0000 7f661e200640 1 --2- 172.21.15.27:0/1972986120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f661815d900 0x7f6618153570 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.598+0000 7f661e200640 1 --2- 172.21.15.27:0/1972986120 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f661815ad20 0x7f66180ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.598+0000 7f661e200640 1 --2- 172.21.15.27:0/1972986120 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6618070ee0 0x7f6618163830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.598+0000 7f661e200640 1 -- 172.21.15.27:0/1972986120 >> 172.21.15.27:0/1972986120 conn(0x7f66180cb700 msgr2=0x7f66180b31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.598+0000 7f661e200640 1 -- 172.21.15.27:0/1972986120 shutdown_connections 2024-01-15T18:17:26.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.598+0000 7f661e200640 1 -- 172.21.15.27:0/1972986120 wait complete. 2024-01-15T18:17:26.635 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:17:26.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.778+0000 7fce4be00640 1 Processor -- start 2024-01-15T18:17:26.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.778+0000 7fce4be00640 1 -- start start 2024-01-15T18:17:26.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.778+0000 7fce4be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fce440c3e90 0x7fce440b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:26.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.778+0000 7fce4be00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce4406b600 0x7fce440cda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:26.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.778+0000 7fce4be00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce44161d10 con 0x7fce440b23e0 2024-01-15T18:17:26.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.778+0000 7fce4be00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce44165de0 con 0x7fce440c3e90 2024-01-15T18:17:26.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.778+0000 7fce4be00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce440cf260 con 0x7fce4406b600 2024-01-15T18:17:26.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.778+0000 7fce48a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fce440b23e0 0x7fce440c4270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36426/0 (socket says 172.21.15.27:36426) 2024-01-15T18:17:26.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.778+0000 7fce48a00640 1 -- 172.21.15.27:0/2672885546 learned_addr learned my addr 172.21.15.27:0/2672885546 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:26.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.778+0000 7fce49400640 1 --2- 172.21.15.27:0/2672885546 >> v2:172.21.15.27:3300/0 conn(0x7fce440c3e90 0x7fce440b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:26.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.778+0000 7fce49e00640 1 --2- 172.21.15.27:0/2672885546 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce4406b600 0x7fce440cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:26.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.778+0000 7fce43e00640 1 -- 172.21.15.27:0/2672885546 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652958310 0 0) 0x7fce2c001120 con 0x7fce440b23e0 2024-01-15T18:17:26.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.778+0000 7fce43e00640 1 -- 172.21.15.27:0/2672885546 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce24003530 con 0x7fce440b23e0 2024-01-15T18:17:26.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.778+0000 7fce49400640 1 -- 172.21.15.27:0/2672885546 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce4406b600 msgr2=0x7fce440cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:26.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.778+0000 7fce49400640 1 --2- 172.21.15.27:0/2672885546 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce4406b600 0x7fce440cda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:26.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.778+0000 7fce49400640 1 -- 172.21.15.27:0/2672885546 >> v1:172.21.15.27:6789/0 conn(0x7fce440b23e0 legacy=0x7fce440c4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:26.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.778+0000 7fce49400640 1 -- 172.21.15.27:0/2672885546 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce44166310 con 0x7fce440c3e90 2024-01-15T18:17:26.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.782+0000 7fce49400640 1 --2- 172.21.15.27:0/2672885546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce440c3e90 0x7fce440b1e50 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fce340097c0 tx=0x7fce3401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97aad4f1c25bcb12 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:26.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.782+0000 7fce43e00640 1 -- 172.21.15.27:0/2672885546 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce340ab400 con 0x7fce440c3e90 2024-01-15T18:17:26.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.782+0000 7fce43e00640 1 -- 172.21.15.27:0/2672885546 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce340a7050 con 0x7fce440c3e90 2024-01-15T18:17:26.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.782+0000 7fce43e00640 1 -- 172.21.15.27:0/2672885546 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce340ab780 con 0x7fce440c3e90 2024-01-15T18:17:26.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.782+0000 7fce4be00640 1 -- 172.21.15.27:0/2672885546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce440c3e90 msgr2=0x7fce440b1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:26.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.782+0000 7fce4be00640 1 --2- 172.21.15.27:0/2672885546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce440c3e90 0x7fce440b1e50 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fce340097c0 tx=0x7fce3401c3c0 comp rx=0 tx=0).stop 2024-01-15T18:17:26.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.782+0000 7fce4be00640 1 -- 172.21.15.27:0/2672885546 shutdown_connections 2024-01-15T18:17:26.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.782+0000 7fce4be00640 1 --2- 172.21.15.27:0/2672885546 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce4406b600 0x7fce440cda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:26.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.782+0000 7fce4be00640 1 --2- 172.21.15.27:0/2672885546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce440c3e90 0x7fce440b1e50 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:26.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.782+0000 7fce4be00640 1 -- 172.21.15.27:0/2672885546 >> 172.21.15.27:0/2672885546 conn(0x7fce440b2a10 msgr2=0x7fce440b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:26.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.782+0000 7fce4be00640 1 -- 172.21.15.27:0/2672885546 shutdown_connections 2024-01-15T18:17:26.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.782+0000 7fce4be00640 1 -- 172.21.15.27:0/2672885546 wait complete. 2024-01-15T18:17:26.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.786+0000 7fce4be00640 1 Processor -- start 2024-01-15T18:17:26.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.786+0000 7fce4be00640 1 -- start start 2024-01-15T18:17:26.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.786+0000 7fce4be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce4406b600 0x7fce440c6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:26.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.786+0000 7fce4be00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce440b23e0 0x7fce4415c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:26.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.786+0000 7fce4be00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce440c3e90 0x7fce440b5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:26.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.786+0000 7fce4be00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce440c69c0 con 0x7fce4406b600 2024-01-15T18:17:26.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.786+0000 7fce4be00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce440c7390 con 0x7fce440c3e90 2024-01-15T18:17:26.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.786+0000 7fce4be00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce440c5db0 con 0x7fce440b23e0 2024-01-15T18:17:26.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.786+0000 7fce49400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce4406b600 0x7fce440c6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:26.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.786+0000 7fce49400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce4406b600 0x7fce440c6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55378/0 (socket says 172.21.15.27:55378) 2024-01-15T18:17:26.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.786+0000 7fce49400640 1 -- 172.21.15.27:0/2726937007 learned_addr learned my addr 172.21.15.27:0/2726937007 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:26.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.786+0000 7fce48a00640 1 --2- 172.21.15.27:0/2726937007 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce440b23e0 0x7fce4415c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:26.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.786+0000 7fce49e00640 1 --2- 172.21.15.27:0/2726937007 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce440c3e90 0x7fce440b5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:26.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.786+0000 7fce49400640 1 -- 172.21.15.27:0/2726937007 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce440b23e0 msgr2=0x7fce4415c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:26.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.786+0000 7fce49400640 1 --2- 172.21.15.27:0/2726937007 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce440b23e0 0x7fce4415c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:26.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.786+0000 7fce49400640 1 -- 172.21.15.27:0/2726937007 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce440c3e90 msgr2=0x7fce440b5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:26.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.786+0000 7fce49400640 1 --2- 172.21.15.27:0/2726937007 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce440c3e90 0x7fce440b5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:26.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.786+0000 7fce49400640 1 -- 172.21.15.27:0/2726937007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce34009420 con 0x7fce4406b600 2024-01-15T18:17:26.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.786+0000 7fce49400640 1 --2- 172.21.15.27:0/2726937007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce4406b600 0x7fce440c6e50 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fce340ae040 tx=0x7fce3401ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:26.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.786+0000 7fce41600640 1 -- 172.21.15.27:0/2726937007 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce340b7450 con 0x7fce4406b600 2024-01-15T18:17:26.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.786+0000 7fce41600640 1 -- 172.21.15.27:0/2726937007 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce340b75d0 con 0x7fce4406b600 2024-01-15T18:17:26.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.786+0000 7fce4be00640 1 -- 172.21.15.27:0/2726937007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce440c6090 con 0x7fce4406b600 2024-01-15T18:17:26.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.786+0000 7fce41600640 1 -- 172.21.15.27:0/2726937007 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce340b7980 con 0x7fce4406b600 2024-01-15T18:17:26.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.786+0000 7fce4be00640 1 -- 172.21.15.27:0/2726937007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce440c6650 con 0x7fce4406b600 2024-01-15T18:17:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.790+0000 7fce4be00640 1 -- 172.21.15.27:0/2726937007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce4404e850 con 0x7fce4406b600 2024-01-15T18:17:26.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.790+0000 7fce41600640 1 -- 172.21.15.27:0/2726937007 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7fce340b7ae0 con 0x7fce4406b600 2024-01-15T18:17:26.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.790+0000 7fce41600640 1 --2- 172.21.15.27:0/2726937007 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce1c067730 0x7fce1c069bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:26.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.790+0000 7fce41600640 1 -- 172.21.15.27:0/2726937007 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fce3411d480 con 0x7fce4406b600 2024-01-15T18:17:26.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.790+0000 7fce48a00640 1 --2- 172.21.15.27:0/2726937007 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce1c067730 0x7fce1c069bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:26.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.790+0000 7fce48a00640 1 --2- 172.21.15.27:0/2726937007 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce1c067730 0x7fce1c069bf0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fce440d1a30 tx=0x7fce2c007bc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:26.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.794+0000 7fce41600640 1 -- 172.21.15.27:0/2726937007 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fce340ec1c0 con 0x7fce4406b600 2024-01-15T18:17:26.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.986+0000 7fce4be00640 1 -- 172.21.15.27:0/2726937007 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce440b6e40 con 0x7fce4406b600 2024-01-15T18:17:26.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.986+0000 7fce41600640 1 -- 172.21.15.27:0/2726937007 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4912 (secure 0 0 0) 0x7fce340b0090 con 0x7fce4406b600 2024-01-15T18:17:26.986 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:17:26.986 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":67,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/1905230324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":1905230324},{"type":"v1","addr":"172.21.15.93:6837","nonce":1905230324}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":8269,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/2926806399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":2926806399},{"type":"v1","addr":"172.21.15.93:6836","nonce":2926806399}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":8934,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1107490361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1107490361},{"type":"v1","addr":"172.21.15.27:6834","nonce":1107490361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:17:16.491967+0000","modified":"2024-01-15T18:17:19.516616+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8937},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8937":{"gid":8937,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.27:6835/259930036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":259930036},{"type":"v1","addr":"172.21.15.27:6835","nonce":259930036}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-01-15T18:17:26.986 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 67 2024-01-15T18:17:26.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.986+0000 7fce4be00640 1 -- 172.21.15.27:0/2726937007 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce1c067730 msgr2=0x7fce1c069bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:26.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.986+0000 7fce4be00640 1 --2- 172.21.15.27:0/2726937007 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce1c067730 0x7fce1c069bf0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fce440d1a30 tx=0x7fce2c007bc0 comp rx=0 tx=0).stop 2024-01-15T18:17:26.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.986+0000 7fce4be00640 1 -- 172.21.15.27:0/2726937007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce4406b600 msgr2=0x7fce440c6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:26.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.986+0000 7fce4be00640 1 --2- 172.21.15.27:0/2726937007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce4406b600 0x7fce440c6e50 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fce340ae040 tx=0x7fce3401ac00 comp rx=0 tx=0).stop 2024-01-15T18:17:26.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.990+0000 7fce4be00640 1 -- 172.21.15.27:0/2726937007 shutdown_connections 2024-01-15T18:17:26.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.990+0000 7fce4be00640 1 --2- 172.21.15.27:0/2726937007 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce1c067730 0x7fce1c069bf0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:26.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.990+0000 7fce4be00640 1 --2- 172.21.15.27:0/2726937007 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce440c3e90 0x7fce440b5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:26.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.990+0000 7fce4be00640 1 --2- 172.21.15.27:0/2726937007 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce440b23e0 0x7fce4415c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:26.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.990+0000 7fce4be00640 1 --2- 172.21.15.27:0/2726937007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce4406b600 0x7fce440c6e50 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:26.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.990+0000 7fce4be00640 1 -- 172.21.15.27:0/2726937007 >> 172.21.15.27:0/2726937007 conn(0x7fce440b2a10 msgr2=0x7fce440cbbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:26.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.990+0000 7fce4be00640 1 -- 172.21.15.27:0/2726937007 shutdown_connections 2024-01-15T18:17:26.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:26.990+0000 7fce4be00640 1 -- 172.21.15.27:0/2726937007 wait complete. 2024-01-15T18:17:27.410 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-15T18:17:16.491967+0000', 'modified': '2024-01-15T18:17:19.516616+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8937}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8937': {'gid': 8937, 'name': 'a', 'rank': 0, 'incarnation': 63, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.27:6835/259930036', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.27:6833', 'nonce': 259930036}, {'type': 'v1', 'addr': '172.21.15.27:6835', 'nonce': 259930036}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-15T18:17:27.410 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-15T18:17:27.410 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-15T18:17:27.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.554+0000 7f0d74800640 1 Processor -- start 2024-01-15T18:17:27.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.554+0000 7f0d74800640 1 -- start start 2024-01-15T18:17:27.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.554+0000 7f0d74800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0d6c15e7d0 0x7f0d6c099a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:27.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.554+0000 7f0d74800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d6c0cd000 con 0x7f0d6c0afe10 2024-01-15T18:17:27.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.554+0000 7f0d74800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d6c15fdf0 con 0x7f0d6c0b35a0 2024-01-15T18:17:27.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.554+0000 7f0d74800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d6c0cd870 con 0x7f0d6c15e7d0 2024-01-15T18:17:27.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.558+0000 7f0d72800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0d6c0afe10 0x7f0d6c15f6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36432/0 (socket says 172.21.15.27:36432) 2024-01-15T18:17:27.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.558+0000 7f0d71e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0d6c15e7d0 0x7f0d6c099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:27.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.558+0000 7f0d72800640 1 -- 172.21.15.27:0/3445384430 learned_addr learned my addr 172.21.15.27:0/3445384430 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:27.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.558+0000 7f0d71e00640 1 -- 172.21.15.27:0/3445384430 >> v1:172.21.15.93:6789/0 conn(0x7f0d6c0b35a0 legacy=0x7f0d6c15efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:27.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.558+0000 7f0d71e00640 1 -- 172.21.15.27:0/3445384430 >> v1:172.21.15.27:6789/0 conn(0x7f0d6c0afe10 legacy=0x7f0d6c15f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:27.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.558+0000 7f0d71e00640 1 -- 172.21.15.27:0/3445384430 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6c09a700 con 0x7f0d6c15e7d0 2024-01-15T18:17:27.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.558+0000 7f0d70a00640 1 -- 172.21.15.27:0/3445384430 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623803016 0 0) 0x7f0d6c0cd000 con 0x7f0d6c0afe10 2024-01-15T18:17:27.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.558+0000 7f0d71e00640 1 --2- 172.21.15.27:0/3445384430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d6c15e7d0 0x7f0d6c099a10 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c0097c0 tx=0x7f0d5c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b62074e741418e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:27.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.558+0000 7f0d70a00640 1 -- 172.21.15.27:0/3445384430 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c0ab670 con 0x7f0d6c15e7d0 2024-01-15T18:17:27.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.558+0000 7f0d70a00640 1 -- 172.21.15.27:0/3445384430 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d5c0ab7d0 con 0x7f0d6c15e7d0 2024-01-15T18:17:27.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.558+0000 7f0d70a00640 1 -- 172.21.15.27:0/3445384430 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c0abb60 con 0x7f0d6c15e7d0 2024-01-15T18:17:27.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.562+0000 7f0d74800640 1 -- 172.21.15.27:0/3445384430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d6c15e7d0 msgr2=0x7f0d6c099a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:27.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.562+0000 7f0d74800640 1 --2- 172.21.15.27:0/3445384430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d6c15e7d0 0x7f0d6c099a10 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c0097c0 tx=0x7f0d5c01c450 comp rx=0 tx=0).stop 2024-01-15T18:17:27.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.562+0000 7f0d74800640 1 -- 172.21.15.27:0/3445384430 shutdown_connections 2024-01-15T18:17:27.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.562+0000 7f0d74800640 1 --2- 172.21.15.27:0/3445384430 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d6c15e7d0 0x7f0d6c099a10 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.562+0000 7f0d74800640 1 -- 172.21.15.27:0/3445384430 >> 172.21.15.27:0/3445384430 conn(0x7f0d6c0b2a80 msgr2=0x7f0d6c0b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:27.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.562+0000 7f0d74800640 1 -- 172.21.15.27:0/3445384430 shutdown_connections 2024-01-15T18:17:27.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.562+0000 7f0d74800640 1 -- 172.21.15.27:0/3445384430 wait complete. 2024-01-15T18:17:27.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.562+0000 7f0d74800640 1 Processor -- start 2024-01-15T18:17:27.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.562+0000 7f0d74800640 1 -- start start 2024-01-15T18:17:27.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.562+0000 7f0d74800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d6c0afe10 0x7f0d6c14f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:27.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.562+0000 7f0d74800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d6c0b35a0 0x7f0d6c155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:27.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.562+0000 7f0d74800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d6c15e7d0 0x7f0d6c14d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:27.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.562+0000 7f0d74800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c14eae0 con 0x7f0d6c0afe10 2024-01-15T18:17:27.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.562+0000 7f0d74800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c14ef00 con 0x7f0d6c0b35a0 2024-01-15T18:17:27.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.562+0000 7f0d74800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c14dc30 con 0x7f0d6c15e7d0 2024-01-15T18:17:27.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.566+0000 7f0d71e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d6c0afe10 0x7f0d6c14f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:27.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.566+0000 7f0d71e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d6c0afe10 0x7f0d6c14f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55400/0 (socket says 172.21.15.27:55400) 2024-01-15T18:17:27.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.566+0000 7f0d71e00640 1 -- 172.21.15.27:0/2204743407 learned_addr learned my addr 172.21.15.27:0/2204743407 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:27.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.566+0000 7f0d72800640 1 --2- 172.21.15.27:0/2204743407 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d6c15e7d0 0x7f0d6c14d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:27.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.566+0000 7f0d71400640 1 --2- 172.21.15.27:0/2204743407 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d6c0b35a0 0x7f0d6c155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:27.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.566+0000 7f0d71e00640 1 -- 172.21.15.27:0/2204743407 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d6c15e7d0 msgr2=0x7f0d6c14d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:27.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.566+0000 7f0d71e00640 1 --2- 172.21.15.27:0/2204743407 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d6c15e7d0 0x7f0d6c14d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.566+0000 7f0d71e00640 1 -- 172.21.15.27:0/2204743407 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d6c0b35a0 msgr2=0x7f0d6c155f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:27.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.566+0000 7f0d71e00640 1 --2- 172.21.15.27:0/2204743407 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d6c0b35a0 0x7f0d6c155f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.566+0000 7f0d71e00640 1 -- 172.21.15.27:0/2204743407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d5c009420 con 0x7f0d6c0afe10 2024-01-15T18:17:27.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.566+0000 7f0d71400640 1 --2- 172.21.15.27:0/2204743407 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d6c0b35a0 0x7f0d6c155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:17:27.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.566+0000 7f0d72800640 1 --2- 172.21.15.27:0/2204743407 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d6c15e7d0 0x7f0d6c14d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:17:27.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.566+0000 7f0d71e00640 1 --2- 172.21.15.27:0/2204743407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d6c0afe10 0x7f0d6c14f530 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c0ad040 tx=0x7f0d5c01ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:27.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.566+0000 7f0d6a000640 1 -- 172.21.15.27:0/2204743407 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c0b5810 con 0x7f0d6c0afe10 2024-01-15T18:17:27.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.566+0000 7f0d6a000640 1 -- 172.21.15.27:0/2204743407 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d5c0b59b0 con 0x7f0d6c0afe10 2024-01-15T18:17:27.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.566+0000 7f0d74800640 1 -- 172.21.15.27:0/2204743407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d6c14de60 con 0x7f0d6c0afe10 2024-01-15T18:17:27.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.566+0000 7f0d6a000640 1 -- 172.21.15.27:0/2204743407 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c0b5d60 con 0x7f0d6c0afe10 2024-01-15T18:17:27.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.566+0000 7f0d74800640 1 -- 172.21.15.27:0/2204743407 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d6c14e420 con 0x7f0d6c0afe10 2024-01-15T18:17:27.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.566+0000 7f0d6a000640 1 -- 172.21.15.27:0/2204743407 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f0d5c0c83f0 con 0x7f0d6c0afe10 2024-01-15T18:17:27.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d6a000640 1 --2- 172.21.15.27:0/2204743407 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0d44067730 0x7f0d44069bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:27.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d6a000640 1 -- 172.21.15.27:0/2204743407 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f0d5c11dc40 con 0x7f0d6c0afe10 2024-01-15T18:17:27.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d71400640 1 --2- 172.21.15.27:0/2204743407 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0d44067730 0x7f0d44069bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:27.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d77c17000 1 Processor -- start 2024-01-15T18:17:27.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d77c17000 1 -- start start 2024-01-15T18:17:27.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d71400640 1 --2- 172.21.15.27:0/2204743407 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0d44067730 0x7f0d44069bf0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c0c4a10 tx=0x7f0d54009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:27.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d77c17000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55ca1b5bdc90 0x55ca1b5be070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:27.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d77c17000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55ca1b5be5b0 0x55ca1b573de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:27.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d77c17000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b574320 0x55ca1b574700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:27.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d77c17000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca1b3de130 con 0x55ca1b574320 2024-01-15T18:17:27.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d77c17000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca1b3cb9a0 con 0x55ca1b5be5b0 2024-01-15T18:17:27.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d77c17000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ca1b3c8ed0 con 0x55ca1b5bdc90 2024-01-15T18:17:27.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d72800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b574320 0x55ca1b574700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:27.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d71e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55ca1b5bdc90 0x55ca1b5be070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:27.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d71400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55ca1b5be5b0 0x55ca1b573de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:27.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d72800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b574320 0x55ca1b574700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55416/0 (socket says 172.21.15.27:55416) 2024-01-15T18:17:27.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d71e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55ca1b5bdc90 0x55ca1b5be070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:40308/0 (socket says 172.21.15.27:40308) 2024-01-15T18:17:27.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d72800640 1 -- 172.21.15.27:0/2175031111 learned_addr learned my addr 172.21.15.27:0/2175031111 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:27.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d72800640 1 -- 172.21.15.27:0/2175031111 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55ca1b5bdc90 msgr2=0x55ca1b5be070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:27.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d72800640 1 --2- 172.21.15.27:0/2175031111 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55ca1b5bdc90 0x55ca1b5be070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d72800640 1 -- 172.21.15.27:0/2175031111 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55ca1b5be5b0 msgr2=0x55ca1b573de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:27.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d72800640 1 --2- 172.21.15.27:0/2175031111 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55ca1b5be5b0 0x55ca1b573de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d72800640 1 -- 172.21.15.27:0/2175031111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ca1b0bc890 con 0x55ca1b574320 2024-01-15T18:17:27.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d72800640 1 --2- 172.21.15.27:0/2175031111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b574320 0x55ca1b574700 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f0d64001f00 tx=0x7f0d6400d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5889fd8f34a51b66 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:27.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d62a00640 1 -- 172.21.15.27:0/2175031111 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d64002110 con 0x55ca1b574320 2024-01-15T18:17:27.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d62a00640 1 -- 172.21.15.27:0/2175031111 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d64004d70 con 0x55ca1b574320 2024-01-15T18:17:27.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.570+0000 7f0d62a00640 1 -- 172.21.15.27:0/2175031111 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d640072d0 con 0x55ca1b574320 2024-01-15T18:17:27.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d77c17000 1 -- 172.21.15.27:0/2175031111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b574320 msgr2=0x55ca1b574700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:27.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d77c17000 1 --2- 172.21.15.27:0/2175031111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b574320 0x55ca1b574700 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f0d64001f00 tx=0x7f0d6400d520 comp rx=0 tx=0).stop 2024-01-15T18:17:27.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d77c17000 1 -- 172.21.15.27:0/2175031111 shutdown_connections 2024-01-15T18:17:27.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d77c17000 1 --2- 172.21.15.27:0/2175031111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b574320 0x55ca1b574700 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d77c17000 1 --2- 172.21.15.27:0/2175031111 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55ca1b5be5b0 0x55ca1b573de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d77c17000 1 --2- 172.21.15.27:0/2175031111 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55ca1b5bdc90 0x55ca1b5be070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d77c17000 1 -- 172.21.15.27:0/2175031111 >> 172.21.15.27:0/2175031111 conn(0x55ca1b5702d0 msgr2=0x55ca1b5726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:27.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d77c17000 1 -- 172.21.15.27:0/2175031111 shutdown_connections 2024-01-15T18:17:27.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d77c17000 1 -- 172.21.15.27:0/2175031111 wait complete. 2024-01-15T18:17:27.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d77c17000 1 -- start start 2024-01-15T18:17:27.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d77c17000 1 Processor -- start 2024-01-15T18:17:27.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d77c17000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55ca1b574320 0x55ca1b578d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:27.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d77c17000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55ca1b5bdc90 0x55ca1b579280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:27.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d77c17000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b5be5b0 0x55ca1b58c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:27.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d77c17000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca1b3c47c0 con 0x55ca1b5be5b0 2024-01-15T18:17:27.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d77c17000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca1b37cbf0 con 0x55ca1b5bdc90 2024-01-15T18:17:27.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d77c17000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ca1b382a90 con 0x55ca1b574320 2024-01-15T18:17:27.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d77c17000 10 client.0 _ll_register_callbacks cb 0x55ca1b55f3c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:17:27.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d72800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b5be5b0 0x55ca1b58c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:27.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d72800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b5be5b0 0x55ca1b58c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55432/0 (socket says 172.21.15.27:55432) 2024-01-15T18:17:27.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d72800640 1 -- 172.21.15.27:0/4080574205 learned_addr learned my addr 172.21.15.27:0/4080574205 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:27.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d71e00640 1 --2- 172.21.15.27:0/4080574205 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55ca1b574320 0x55ca1b578d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:27.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d71400640 1 --2- 172.21.15.27:0/4080574205 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55ca1b5bdc90 0x55ca1b579280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:27.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d72800640 1 -- 172.21.15.27:0/4080574205 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55ca1b574320 msgr2=0x55ca1b578d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:27.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d72800640 1 --2- 172.21.15.27:0/4080574205 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55ca1b574320 0x55ca1b578d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d72800640 1 -- 172.21.15.27:0/4080574205 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55ca1b5bdc90 msgr2=0x55ca1b579280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:27.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d72800640 1 --2- 172.21.15.27:0/4080574205 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55ca1b5bdc90 0x55ca1b579280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d72800640 1 -- 172.21.15.27:0/4080574205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d64001b30 con 0x55ca1b5be5b0 2024-01-15T18:17:27.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d72800640 1 --2- 172.21.15.27:0/4080574205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b5be5b0 0x55ca1b58c910 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f0d6400da20 tx=0x7f0d64001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31657ff137fa130 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:27.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d5b400640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-15T18:17:27.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d5b400640 1 -- 172.21.15.27:0/4080574205 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d64002110 con 0x55ca1b5be5b0 2024-01-15T18:17:27.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d77c17000 1 -- 172.21.15.27:0/4080574205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ca1b386470 con 0x55ca1b5be5b0 2024-01-15T18:17:27.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d5b400640 1 -- 172.21.15.27:0/4080574205 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f0d6400de40 con 0x55ca1b5be5b0 2024-01-15T18:17:27.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d5b400640 1 -- 172.21.15.27:0/4080574205 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d64007c00 con 0x55ca1b5be5b0 2024-01-15T18:17:27.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d77c17000 10 client.9114 fetch_fsmap learned FSMap version 67 2024-01-15T18:17:27.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.574+0000 7f0d77c17000 1 -- 172.21.15.27:0/4080574205 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) v3 -- 0x55ca1b385190 con 0x55ca1b5be5b0 2024-01-15T18:17:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d5b400640 1 -- 172.21.15.27:0/4080574205 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ca1b386470 con 0x55ca1b5be5b0 2024-01-15T18:17:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d5b400640 1 -- 172.21.15.27:0/4080574205 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 67) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0d64007750 con 0x55ca1b5be5b0 2024-01-15T18:17:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d77c17000 10 client.9114 fetch_fsmap finished waiting for FSMap version 67 2024-01-15T18:17:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d77c17000 10 client.9114 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-15T18:17:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d77c17000 20 client.9114 populate_metadata read hostname 'smithi027' 2024-01-15T18:17:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d77c17000 1 --2- 172.21.15.27:0/4080574205 >> [v2:172.21.15.27:6833/259930036,v1:172.21.15.27:6835/259930036] conn(0x55ca1b58d320 0x55ca1b5cff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d77c17000 4 client.9114 mds_command: new command op to 8937 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-15T18:17:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d77c17000 1 -- 172.21.15.27:0/4080574205 --> [v2:172.21.15.27:6833/259930036,v1:172.21.15.27:6835/259930036] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ca1b389b50 con 0x55ca1b58d320 2024-01-15T18:17:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d71e00640 1 --2- 172.21.15.27:0/4080574205 >> [v2:172.21.15.27:6833/259930036,v1:172.21.15.27:6835/259930036] conn(0x55ca1b58d320 0x55ca1b5cff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d71e00640 1 --2- 172.21.15.27:0/4080574205 >> [v2:172.21.15.27:6833/259930036,v1:172.21.15.27:6835/259930036] conn(0x55ca1b58d320 0x55ca1b5cff50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3ef0a2a55507c23 server_cookie=1e01b5a08c7043f9 in_seq=0 out_seq=0 2024-01-15T18:17:27.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d5b400640 10 client.9114 ms_handle_connect on v2:172.21.15.27:6833/259930036 2024-01-15T18:17:27.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d5b400640 1 -- 172.21.15.27:0/4080574205 <== mds.0 v2:172.21.15.27:6833/259930036 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ca1b389b50 con 0x55ca1b58d320 2024-01-15T18:17:27.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d5b400640 10 client.9114 handle_command_reply: tid=0 2024-01-15T18:17:27.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d77c17000 1 client.9114 shutdown 2024-01-15T18:17:27.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d77c17000 1 -- 172.21.15.27:0/4080574205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b5be5b0 msgr2=0x55ca1b58c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:27.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d77c17000 1 --2- 172.21.15.27:0/4080574205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b5be5b0 0x55ca1b58c910 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f0d6400da20 tx=0x7f0d64001260 comp rx=0 tx=0).stop 2024-01-15T18:17:27.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d77c17000 1 -- 172.21.15.27:0/4080574205 shutdown_connections 2024-01-15T18:17:27.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d77c17000 1 --2- 172.21.15.27:0/4080574205 >> [v2:172.21.15.27:6833/259930036,v1:172.21.15.27:6835/259930036] conn(0x55ca1b58d320 0x55ca1b5cff50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d77c17000 1 --2- 172.21.15.27:0/4080574205 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b5be5b0 0x55ca1b58c910 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d77c17000 1 --2- 172.21.15.27:0/4080574205 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55ca1b5bdc90 0x55ca1b579280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d5b400640 0 client.9114 ms_handle_reset on v2:172.21.15.27:6833/259930036 2024-01-15T18:17:27.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d77c17000 1 --2- 172.21.15.27:0/4080574205 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55ca1b574320 0x55ca1b578d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d77c17000 1 -- 172.21.15.27:0/4080574205 >> 172.21.15.27:0/4080574205 conn(0x55ca1b5702d0 msgr2=0x55ca1b5726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:27.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d77c17000 1 -- 172.21.15.27:0/4080574205 shutdown_connections 2024-01-15T18:17:27.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d77c17000 1 -- 172.21.15.27:0/4080574205 wait complete. 2024-01-15T18:17:27.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.578+0000 7f0d77c17000 20 client.9114 trim_cache size 0 max 16384 2024-01-15T18:17:27.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.594+0000 7f0d77c17000 1 Processor -- start 2024-01-15T18:17:27.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.594+0000 7f0d77c17000 1 -- start start 2024-01-15T18:17:27.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.594+0000 7f0d77c17000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55ca1b574320 0x55ca1b5bca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:27.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.594+0000 7f0d77c17000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55ca1b5bdc90 0x55ca1b5bbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:27.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.594+0000 7f0d77c17000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b5be5b0 0x55ca1b5bc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:27.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.594+0000 7f0d77c17000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca1b07d940 con 0x55ca1b5be5b0 2024-01-15T18:17:27.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.594+0000 7f0d77c17000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca1b3c1bc0 con 0x55ca1b5bdc90 2024-01-15T18:17:27.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.594+0000 7f0d77c17000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ca1b58c8c0 con 0x55ca1b574320 2024-01-15T18:17:27.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.594+0000 7f0d72800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b5be5b0 0x55ca1b5bc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:27.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.594+0000 7f0d72800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b5be5b0 0x55ca1b5bc070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55434/0 (socket says 172.21.15.27:55434) 2024-01-15T18:17:27.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.594+0000 7f0d72800640 1 -- 172.21.15.27:0/4275791945 learned_addr learned my addr 172.21.15.27:0/4275791945 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:27.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d71e00640 1 --2- 172.21.15.27:0/4275791945 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55ca1b574320 0x55ca1b5bca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:27.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d71400640 1 --2- 172.21.15.27:0/4275791945 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55ca1b5bdc90 0x55ca1b5bbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:27.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d72800640 1 -- 172.21.15.27:0/4275791945 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55ca1b574320 msgr2=0x55ca1b5bca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:27.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d72800640 1 --2- 172.21.15.27:0/4275791945 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55ca1b574320 0x55ca1b5bca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d72800640 1 -- 172.21.15.27:0/4275791945 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55ca1b5bdc90 msgr2=0x55ca1b5bbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:27.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d72800640 1 --2- 172.21.15.27:0/4275791945 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55ca1b5bdc90 0x55ca1b5bbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d72800640 1 -- 172.21.15.27:0/4275791945 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ca1b36b490 con 0x55ca1b5be5b0 2024-01-15T18:17:27.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d71e00640 1 --2- 172.21.15.27:0/4275791945 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55ca1b574320 0x55ca1b5bca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:17:27.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d72800640 1 --2- 172.21.15.27:0/4275791945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b5be5b0 0x55ca1b5bc070 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f0d64007b70 tx=0x7f0d6401e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=382eef6442125d7b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:27.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d63400640 1 -- 172.21.15.27:0/4275791945 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d64002110 con 0x55ca1b5be5b0 2024-01-15T18:17:27.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d63400640 1 -- 172.21.15.27:0/4275791945 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d64007390 con 0x55ca1b5be5b0 2024-01-15T18:17:27.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d63400640 1 -- 172.21.15.27:0/4275791945 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6401f6b0 con 0x55ca1b5be5b0 2024-01-15T18:17:27.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d77c17000 1 -- 172.21.15.27:0/4275791945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b5be5b0 msgr2=0x55ca1b5bc070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:27.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d77c17000 1 --2- 172.21.15.27:0/4275791945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b5be5b0 0x55ca1b5bc070 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f0d64007b70 tx=0x7f0d6401e000 comp rx=0 tx=0).stop 2024-01-15T18:17:27.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d77c17000 1 -- 172.21.15.27:0/4275791945 shutdown_connections 2024-01-15T18:17:27.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d77c17000 1 --2- 172.21.15.27:0/4275791945 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b5be5b0 0x55ca1b5bc070 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d77c17000 1 --2- 172.21.15.27:0/4275791945 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55ca1b5bdc90 0x55ca1b5bbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d77c17000 1 --2- 172.21.15.27:0/4275791945 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55ca1b574320 0x55ca1b5bca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d77c17000 1 -- 172.21.15.27:0/4275791945 >> 172.21.15.27:0/4275791945 conn(0x55ca1b5702d0 msgr2=0x55ca1b5726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:27.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d77c17000 1 -- 172.21.15.27:0/4275791945 shutdown_connections 2024-01-15T18:17:27.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d77c17000 1 -- 172.21.15.27:0/4275791945 wait complete. 2024-01-15T18:17:27.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d77c17000 1 -- start start 2024-01-15T18:17:27.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d77c17000 1 Processor -- start 2024-01-15T18:17:27.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d77c17000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55ca1b574320 0x55ca1b5d22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:27.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d77c17000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55ca1b5bdc90 0x55ca1b5d0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:27.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d77c17000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b5be5b0 0x55ca1b5d14e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:27.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d77c17000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca1b406140 con 0x55ca1b5be5b0 2024-01-15T18:17:27.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d77c17000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca1b3c5070 con 0x55ca1b5bdc90 2024-01-15T18:17:27.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d77c17000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ca1b3cc0c0 con 0x55ca1b574320 2024-01-15T18:17:27.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d77c17000 10 client.0 _ll_register_callbacks cb 0x55ca1b412c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:17:27.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d72800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b5be5b0 0x55ca1b5d14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:27.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d71400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55ca1b5bdc90 0x55ca1b5d0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:27.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d72800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b5be5b0 0x55ca1b5d14e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55446/0 (socket says 172.21.15.27:55446) 2024-01-15T18:17:27.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d72800640 1 -- 172.21.15.27:0/1946850615 learned_addr learned my addr 172.21.15.27:0/1946850615 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:27.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.598+0000 7f0d71e00640 1 --2- 172.21.15.27:0/1946850615 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55ca1b574320 0x55ca1b5d22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:27.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d72800640 1 -- 172.21.15.27:0/1946850615 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55ca1b574320 msgr2=0x55ca1b5d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:27.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d72800640 1 --2- 172.21.15.27:0/1946850615 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55ca1b574320 0x55ca1b5d22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d72800640 1 -- 172.21.15.27:0/1946850615 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55ca1b5bdc90 msgr2=0x55ca1b5d0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:27.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d72800640 1 --2- 172.21.15.27:0/1946850615 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55ca1b5bdc90 0x55ca1b5d0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d72800640 1 -- 172.21.15.27:0/1946850615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ca1b385190 con 0x55ca1b5be5b0 2024-01-15T18:17:27.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d72800640 1 --2- 172.21.15.27:0/1946850615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b5be5b0 0x55ca1b5d14e0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f0d64009010 tx=0x7f0d6400b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eba967f5cf7be4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:27.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d5b400640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-15T18:17:27.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d5b400640 1 -- 172.21.15.27:0/1946850615 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d64002110 con 0x55ca1b5be5b0 2024-01-15T18:17:27.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d5b400640 1 -- 172.21.15.27:0/1946850615 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f0d6400de40 con 0x55ca1b5be5b0 2024-01-15T18:17:27.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d77c17000 1 -- 172.21.15.27:0/1946850615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ca1b3c3110 con 0x55ca1b5be5b0 2024-01-15T18:17:27.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d5b400640 1 -- 172.21.15.27:0/1946850615 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d64007bc0 con 0x55ca1b5be5b0 2024-01-15T18:17:27.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d77c17000 10 client.9120 fetch_fsmap learned FSMap version 67 2024-01-15T18:17:27.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d77c17000 1 -- 172.21.15.27:0/1946850615 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) v3 -- 0x55ca1b0bccc0 con 0x55ca1b5be5b0 2024-01-15T18:17:27.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d5b400640 1 -- 172.21.15.27:0/1946850615 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ca1b3c3110 con 0x55ca1b5be5b0 2024-01-15T18:17:27.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d5b400640 1 -- 172.21.15.27:0/1946850615 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 67) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0d64039070 con 0x55ca1b5be5b0 2024-01-15T18:17:27.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d77c17000 10 client.9120 fetch_fsmap finished waiting for FSMap version 67 2024-01-15T18:17:27.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d77c17000 10 client.9120 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-15T18:17:27.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d77c17000 20 client.9120 populate_metadata read hostname 'smithi027' 2024-01-15T18:17:27.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d77c17000 1 --2- 172.21.15.27:0/1946850615 >> [v2:172.21.15.27:6833/259930036,v1:172.21.15.27:6835/259930036] conn(0x55ca1b5dbe60 0x55ca1b5dc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:27.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d77c17000 4 client.9120 mds_command: new command op to 8937 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-15T18:17:27.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d77c17000 1 -- 172.21.15.27:0/1946850615 --> [v2:172.21.15.27:6833/259930036,v1:172.21.15.27:6835/259930036] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ca1b3f29a0 con 0x55ca1b5dbe60 2024-01-15T18:17:27.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d71e00640 1 --2- 172.21.15.27:0/1946850615 >> [v2:172.21.15.27:6833/259930036,v1:172.21.15.27:6835/259930036] conn(0x55ca1b5dbe60 0x55ca1b5dc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:27.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d71e00640 1 --2- 172.21.15.27:0/1946850615 >> [v2:172.21.15.27:6833/259930036,v1:172.21.15.27:6835/259930036] conn(0x55ca1b5dbe60 0x55ca1b5dc240 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1110e32c132181e0 server_cookie=529a4f12d712efd2 in_seq=0 out_seq=0 2024-01-15T18:17:27.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d5b400640 10 client.9120 ms_handle_connect on v2:172.21.15.27:6833/259930036 2024-01-15T18:17:27.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d5b400640 1 -- 172.21.15.27:0/1946850615 <== mds.0 v2:172.21.15.27:6833/259930036 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55ca1b3f29a0 con 0x55ca1b5dbe60 2024-01-15T18:17:27.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d5b400640 10 client.9120 handle_command_reply: tid=0 2024-01-15T18:17:27.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d77c17000 1 client.9120 shutdown 2024-01-15T18:17:27.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d77c17000 1 -- 172.21.15.27:0/1946850615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b5be5b0 msgr2=0x55ca1b5d14e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:27.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d77c17000 1 --2- 172.21.15.27:0/1946850615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b5be5b0 0x55ca1b5d14e0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f0d64009010 tx=0x7f0d6400b000 comp rx=0 tx=0).stop 2024-01-15T18:17:27.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d77c17000 1 -- 172.21.15.27:0/1946850615 shutdown_connections 2024-01-15T18:17:27.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d77c17000 1 --2- 172.21.15.27:0/1946850615 >> [v2:172.21.15.27:6833/259930036,v1:172.21.15.27:6835/259930036] conn(0x55ca1b5dbe60 0x55ca1b5dc240 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d77c17000 1 --2- 172.21.15.27:0/1946850615 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ca1b5be5b0 0x55ca1b5d14e0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d77c17000 1 --2- 172.21.15.27:0/1946850615 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55ca1b5bdc90 0x55ca1b5d0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d77c17000 1 --2- 172.21.15.27:0/1946850615 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55ca1b574320 0x55ca1b5d22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d5b400640 0 client.9120 ms_handle_reset on v2:172.21.15.27:6833/259930036 2024-01-15T18:17:27.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d77c17000 1 -- 172.21.15.27:0/1946850615 >> 172.21.15.27:0/1946850615 conn(0x55ca1b5702d0 msgr2=0x55ca1b5cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:27.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d77c17000 1 -- 172.21.15.27:0/1946850615 shutdown_connections 2024-01-15T18:17:27.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d77c17000 1 -- 172.21.15.27:0/1946850615 wait complete. 2024-01-15T18:17:27.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.602+0000 7f0d77c17000 20 client.9120 trim_cache size 0 max 16384 2024-01-15T18:17:27.607 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:17:27.607 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:17:27.607 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-01-15T18:17:27.607 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 8937, 2024-01-15T18:17:27.607 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-01-15T18:17:27.607 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-01-15T18:17:27.607 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-01-15T18:17:27.607 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 10.488765597, 2024-01-15T18:17:27.607 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 67, 2024-01-15T18:17:27.607 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 97, 2024-01-15T18:17:27.608 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 97, 2024-01-15T18:17:27.608 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 15.386242251000001 2024-01-15T18:17:27.608 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:17:27.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.606+0000 7f0d74800640 1 -- 172.21.15.27:0/2204743407 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0d44067730 msgr2=0x7f0d44069bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:27.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.606+0000 7f0d74800640 1 --2- 172.21.15.27:0/2204743407 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0d44067730 0x7f0d44069bf0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c0c4a10 tx=0x7f0d54009040 comp rx=0 tx=0).stop 2024-01-15T18:17:27.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.606+0000 7f0d74800640 1 -- 172.21.15.27:0/2204743407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d6c0afe10 msgr2=0x7f0d6c14f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:27.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.606+0000 7f0d74800640 1 --2- 172.21.15.27:0/2204743407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d6c0afe10 0x7f0d6c14f530 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c0ad040 tx=0x7f0d5c01ac90 comp rx=0 tx=0).stop 2024-01-15T18:17:27.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.606+0000 7f0d74800640 1 -- 172.21.15.27:0/2204743407 shutdown_connections 2024-01-15T18:17:27.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.606+0000 7f0d74800640 1 --2- 172.21.15.27:0/2204743407 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0d44067730 0x7f0d44069bf0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.606+0000 7f0d74800640 1 --2- 172.21.15.27:0/2204743407 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d6c15e7d0 0x7f0d6c14d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.606+0000 7f0d74800640 1 --2- 172.21.15.27:0/2204743407 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d6c0b35a0 0x7f0d6c155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.606+0000 7f0d74800640 1 --2- 172.21.15.27:0/2204743407 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d6c0afe10 0x7f0d6c14f530 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.606+0000 7f0d74800640 1 -- 172.21.15.27:0/2204743407 >> 172.21.15.27:0/2204743407 conn(0x7f0d6c0b2a80 msgr2=0x7f0d6c0cbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:27.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.606+0000 7f0d74800640 1 -- 172.21.15.27:0/2204743407 shutdown_connections 2024-01-15T18:17:27.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.606+0000 7f0d74800640 1 -- 172.21.15.27:0/2204743407 wait complete. 2024-01-15T18:17:27.624 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:17:27.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.766+0000 7fb951400640 1 Processor -- start 2024-01-15T18:17:27.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.766+0000 7fb951400640 1 -- start start 2024-01-15T18:17:27.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.770+0000 7fb951400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb94c0b1f80 0x7fb94c0cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:27.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.770+0000 7fb951400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94c0d1a70 con 0x7fb94c0ce420 2024-01-15T18:17:27.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.770+0000 7fb951400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94c160e00 con 0x7fb94c160a20 2024-01-15T18:17:27.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.770+0000 7fb951400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c0d1cc0 con 0x7fb94c0b1f80 2024-01-15T18:17:27.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.770+0000 7fb94b200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb94c0ce420 0x7fb94c0cb880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36436/0 (socket says 172.21.15.27:36436) 2024-01-15T18:17:27.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.770+0000 7fb94b200640 1 -- 172.21.15.27:0/2126460572 learned_addr learned my addr 172.21.15.27:0/2126460572 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:27.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.770+0000 7fb949e00640 1 --2- 172.21.15.27:0/2126460572 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb94c0b1f80 0x7fb94c0cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:27.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.770+0000 7fb949400640 1 -- 172.21.15.27:0/2126460572 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 428759035 0 0) 0x7fb94c0d1a70 con 0x7fb94c0ce420 2024-01-15T18:17:27.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.770+0000 7fb949400640 1 -- 172.21.15.27:0/2126460572 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb92c003530 con 0x7fb94c0ce420 2024-01-15T18:17:27.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.770+0000 7fb949400640 1 -- 172.21.15.27:0/2126460572 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3234008894 0 0) 0x7fb94c160e00 con 0x7fb94c160a20 2024-01-15T18:17:27.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.770+0000 7fb949400640 1 -- 172.21.15.27:0/2126460572 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb94c0d1a70 con 0x7fb94c160a20 2024-01-15T18:17:27.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.770+0000 7fb949e00640 1 -- 172.21.15.27:0/2126460572 >> v1:172.21.15.93:6789/0 conn(0x7fb94c160a20 legacy=0x7fb94c0cd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:27.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.770+0000 7fb949e00640 1 -- 172.21.15.27:0/2126460572 >> v1:172.21.15.27:6789/0 conn(0x7fb94c0ce420 legacy=0x7fb94c0cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:27.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.770+0000 7fb949e00640 1 -- 172.21.15.27:0/2126460572 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94c06c010 con 0x7fb94c0b1f80 2024-01-15T18:17:27.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.770+0000 7fb949e00640 1 --2- 172.21.15.27:0/2126460572 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb94c0b1f80 0x7fb94c0cdee0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb9340097f0 tx=0x7fb93401c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39e32343f6b4bc55 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:27.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.774+0000 7fb949400640 1 -- 172.21.15.27:0/2126460572 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9340ac410 con 0x7fb94c0b1f80 2024-01-15T18:17:27.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.774+0000 7fb949400640 1 -- 172.21.15.27:0/2126460572 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9340ac570 con 0x7fb94c0b1f80 2024-01-15T18:17:27.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.774+0000 7fb949400640 1 -- 172.21.15.27:0/2126460572 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9340ac900 con 0x7fb94c0b1f80 2024-01-15T18:17:27.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.774+0000 7fb951400640 1 -- 172.21.15.27:0/2126460572 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb94c0b1f80 msgr2=0x7fb94c0cdee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:27.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.774+0000 7fb951400640 1 --2- 172.21.15.27:0/2126460572 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb94c0b1f80 0x7fb94c0cdee0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb9340097f0 tx=0x7fb93401c4c0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.774+0000 7fb951400640 1 -- 172.21.15.27:0/2126460572 shutdown_connections 2024-01-15T18:17:27.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.774+0000 7fb951400640 1 --2- 172.21.15.27:0/2126460572 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb94c0b1f80 0x7fb94c0cdee0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.774+0000 7fb951400640 1 -- 172.21.15.27:0/2126460572 >> 172.21.15.27:0/2126460572 conn(0x7fb94c0d0f50 msgr2=0x7fb94c0d1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:27.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.774+0000 7fb951400640 1 -- 172.21.15.27:0/2126460572 shutdown_connections 2024-01-15T18:17:27.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.774+0000 7fb951400640 1 -- 172.21.15.27:0/2126460572 wait complete. 2024-01-15T18:17:27.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.774+0000 7fb951400640 1 Processor -- start 2024-01-15T18:17:27.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.774+0000 7fb951400640 1 -- start start 2024-01-15T18:17:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.774+0000 7fb951400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb94c0b1f80 0x7fb94c0c42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.774+0000 7fb951400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb94c0ce420 0x7fb94c14b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.774+0000 7fb951400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb94c160a20 0x7fb94c0c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.774+0000 7fb951400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c0c47f0 con 0x7fb94c0b1f80 2024-01-15T18:17:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.774+0000 7fb951400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c0c6490 con 0x7fb94c0ce420 2024-01-15T18:17:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.774+0000 7fb951400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94c14bea0 con 0x7fb94c160a20 2024-01-15T18:17:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.778+0000 7fb94a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb94c0b1f80 0x7fb94c0c42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.778+0000 7fb94a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb94c0b1f80 0x7fb94c0c42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55454/0 (socket says 172.21.15.27:55454) 2024-01-15T18:17:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.778+0000 7fb94a800640 1 -- 172.21.15.27:0/3609856664 learned_addr learned my addr 172.21.15.27:0/3609856664 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.778+0000 7fb94b200640 1 --2- 172.21.15.27:0/3609856664 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb94c160a20 0x7fb94c0c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.778+0000 7fb949e00640 1 --2- 172.21.15.27:0/3609856664 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb94c0ce420 0x7fb94c14b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.778+0000 7fb94a800640 1 -- 172.21.15.27:0/3609856664 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb94c160a20 msgr2=0x7fb94c0c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:27.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.778+0000 7fb94a800640 1 --2- 172.21.15.27:0/3609856664 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb94c160a20 0x7fb94c0c4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.778+0000 7fb94a800640 1 -- 172.21.15.27:0/3609856664 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb94c0ce420 msgr2=0x7fb94c14b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:27.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.778+0000 7fb94a800640 1 --2- 172.21.15.27:0/3609856664 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb94c0ce420 0x7fb94c14b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.778+0000 7fb94a800640 1 -- 172.21.15.27:0/3609856664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb934009420 con 0x7fb94c0b1f80 2024-01-15T18:17:27.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.778+0000 7fb94a800640 1 --2- 172.21.15.27:0/3609856664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb94c0b1f80 0x7fb94c0c42b0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fb93c00c490 tx=0x7fb93c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:27.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.778+0000 7fb942a00640 1 -- 172.21.15.27:0/3609856664 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c0052d0 con 0x7fb94c0b1f80 2024-01-15T18:17:27.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.778+0000 7fb942a00640 1 -- 172.21.15.27:0/3609856664 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93c005430 con 0x7fb94c0b1f80 2024-01-15T18:17:27.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.778+0000 7fb951400640 1 -- 172.21.15.27:0/3609856664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb94c154260 con 0x7fb94c0b1f80 2024-01-15T18:17:27.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.778+0000 7fb942a00640 1 -- 172.21.15.27:0/3609856664 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c012870 con 0x7fb94c0b1f80 2024-01-15T18:17:27.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.778+0000 7fb951400640 1 -- 172.21.15.27:0/3609856664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb94c154820 con 0x7fb94c0b1f80 2024-01-15T18:17:27.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.778+0000 7fb951400640 1 -- 172.21.15.27:0/3609856664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb94c04e850 con 0x7fb94c0b1f80 2024-01-15T18:17:27.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.778+0000 7fb942a00640 1 -- 172.21.15.27:0/3609856664 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7fb93c0129d0 con 0x7fb94c0b1f80 2024-01-15T18:17:27.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.782+0000 7fb942a00640 1 --2- 172.21.15.27:0/3609856664 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb924067700 0x7fb924069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:27.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.782+0000 7fb942a00640 1 -- 172.21.15.27:0/3609856664 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fb93c082b80 con 0x7fb94c0b1f80 2024-01-15T18:17:27.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.782+0000 7fb949e00640 1 --2- 172.21.15.27:0/3609856664 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb924067700 0x7fb924069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:27.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.782+0000 7fb949e00640 1 --2- 172.21.15.27:0/3609856664 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb924067700 0x7fb924069bc0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fb9340ae000 tx=0x7fb934008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:27.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.786+0000 7fb942a00640 1 -- 172.21.15.27:0/3609856664 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb93c051660 con 0x7fb94c0b1f80 2024-01-15T18:17:27.801 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:17:27.804 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:17:27.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.974+0000 7fb951400640 1 -- 172.21.15.27:0/3609856664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb94c14cb10 con 0x7fb94c0b1f80 2024-01-15T18:17:27.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.974+0000 7fb942a00640 1 -- 172.21.15.27:0/3609856664 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v67) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb93c054c80 con 0x7fb94c0b1f80 2024-01-15T18:17:27.975 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:17:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.978+0000 7fb951400640 1 -- 172.21.15.27:0/3609856664 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb924067700 msgr2=0x7fb924069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.978+0000 7fb951400640 1 --2- 172.21.15.27:0/3609856664 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb924067700 0x7fb924069bc0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fb9340ae000 tx=0x7fb934008040 comp rx=0 tx=0).stop 2024-01-15T18:17:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.978+0000 7fb951400640 1 -- 172.21.15.27:0/3609856664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb94c0b1f80 msgr2=0x7fb94c0c42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.978+0000 7fb951400640 1 --2- 172.21.15.27:0/3609856664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb94c0b1f80 0x7fb94c0c42b0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fb93c00c490 tx=0x7fb93c00c950 comp rx=0 tx=0).stop 2024-01-15T18:17:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.978+0000 7fb951400640 1 -- 172.21.15.27:0/3609856664 shutdown_connections 2024-01-15T18:17:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.978+0000 7fb951400640 1 --2- 172.21.15.27:0/3609856664 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb924067700 0x7fb924069bc0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.978+0000 7fb951400640 1 --2- 172.21.15.27:0/3609856664 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb94c160a20 0x7fb94c0c4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.978+0000 7fb951400640 1 --2- 172.21.15.27:0/3609856664 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb94c0ce420 0x7fb94c14b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.978+0000 7fb951400640 1 --2- 172.21.15.27:0/3609856664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb94c0b1f80 0x7fb94c0c42b0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:27.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.978+0000 7fb951400640 1 -- 172.21.15.27:0/3609856664 >> 172.21.15.27:0/3609856664 conn(0x7fb94c0d0f50 msgr2=0x7fb94c0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:27.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.978+0000 7fb951400640 1 -- 172.21.15.27:0/3609856664 shutdown_connections 2024-01-15T18:17:27.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:27.978+0000 7fb951400640 1 -- 172.21.15.27:0/3609856664 wait complete. 2024-01-15T18:17:27.997 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:17:27.998 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:17:27.998 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-01-15T18:17:27.998 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-15T18:17:27.998 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-15T18:17:27.998 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:17:27.998 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:17:27.998 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:17:27.998 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:17:27.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:27.998 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-01-15T18:17:28.007 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:17:28.007 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:17:28.008 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:17:28.008 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:17:28.008 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-15T18:17:28.008 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:17:28.008 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:17:28.008 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:17:28.008 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-01-15T18:17:28.008 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:17:28.008 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:17:28.008 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-01-15T18:17:28.008 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:17:28.009 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-01-15T18:17:28.009 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-01-15T18:17:28.009 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-01-15T18:17:28.009 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 41711sec preferred_lft 41711sec 2024-01-15T18:17:28.009 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-01-15T18:17:28.009 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:17:28.009 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:17:28.009 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-01-15T18:17:28.009 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-01-15T18:17:28.009 INFO:teuthology.orchestra.run.smithi027.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-01-15T18:17:28.009 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 02:42:65:20:14:72 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:17:28.009 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-01-15T18:17:28.009 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:17:28.009 INFO:teuthology.orchestra.run.smithi027.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:17:28.009 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-01-15T18:17:28.010 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:17:28.010 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:17:28.010 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-01-15T18:17:28.010 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:17:28.010 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:17:28.010 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:17:28.010 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-01-15T18:17:28.010 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:17:28.010 INFO:teuthology.orchestra.run.smithi027.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:17:28.010 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:17:28.010 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-01-15T18:17:28.010 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:17:28.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:28.011 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:17:28.078 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-01-15T18:17:28.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:28.081 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-01-15T18:17:28.137 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-01-15T18:17:28.137 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:17:28.137 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-01-15T18:17:28.137 INFO:teuthology.orchestra.run.smithi027.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-01-15T18:17:28.137 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-01-15T18:17:28.137 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:17:28.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:28.137 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-15T18:17:28.137 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-15T18:17:28.137 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-15T18:17:28.138 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-15T18:17:28.138 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-15T18:17:28.138 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-15T18:17:28.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:28.230 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-15T18:17:28.239 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-15T18:17:28.239 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-15T18:17:28.239 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:17:28.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:28.239 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-01-15T18:17:28.299 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-15T18:17:28.299 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:17:28.300 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-15T18:17:28.351 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:17:28.351 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:17:28.352 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:17:28.403 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-01-15T18:17:28.463 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:17:28.521 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-01-15T18:17:28.521 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-01-15T18:17:28.521 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-01-15T18:17:28.521 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-01-15T18:17:28.521 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-01-15T18:17:28.521 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-01-15T18:17:28.522 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-01-15T18:17:28.522 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-01-15T18:17:28.522 INFO:teuthology.orchestra.run.smithi027.stdout:/run/lock 2024-01-15T18:17:28.522 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-01-15T18:17:28.522 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-01-15T18:17:28.522 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-01-15T18:17:28.522 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:17:28.522 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-01-15T18:17:28.522 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-01-15T18:17:28.522 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-01-15T18:17:28.522 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-01-15T18:17:28.522 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/fs/nfsd 2024-01-15T18:17:28.523 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-01-15T18:17:28.523 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-01-15T18:17:28.523 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:17:28.523 INFO:teuthology.orchestra.run.smithi027.stdout:/run/rpc_pipefs 2024-01-15T18:17:28.523 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-01-15T18:17:28.523 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-01-15T18:17:28.523 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:17:28.523 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:17:28.523 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:17:28.523 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:17:28.523 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-01-15T18:17:28.523 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:17:28.523 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:17:28.523 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:17:28.524 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:17:28.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:28.525 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-01-15T18:17:28.575 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:17:28.576 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-15T18:17:28.622 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-01-15T18:17:28.643 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:17:28.701 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-01-15T18:17:28.701 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-01-15T18:17:28.701 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-01-15T18:17:28.701 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-01-15T18:17:28.701 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-01-15T18:17:28.701 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-01-15T18:17:28.701 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-01-15T18:17:28.702 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-01-15T18:17:28.702 INFO:teuthology.orchestra.run.smithi027.stdout:/run/lock 2024-01-15T18:17:28.702 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-01-15T18:17:28.702 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-01-15T18:17:28.702 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-01-15T18:17:28.702 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:17:28.702 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-01-15T18:17:28.702 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-01-15T18:17:28.702 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-01-15T18:17:28.702 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-01-15T18:17:28.702 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/fs/nfsd 2024-01-15T18:17:28.702 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-01-15T18:17:28.702 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-01-15T18:17:28.702 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:17:28.702 INFO:teuthology.orchestra.run.smithi027.stdout:/run/rpc_pipefs 2024-01-15T18:17:28.703 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-01-15T18:17:28.703 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-01-15T18:17:28.703 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:17:28.703 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:17:28.703 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:17:28.703 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:17:28.703 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-01-15T18:17:28.703 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:17:28.703 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:17:28.703 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:17:28.703 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:17:28.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:28.704 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-01-15T18:17:28.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:2024-01-15T18:17:28.718+0000 7fc90fc2b580 -1 init, newargv = 0x5614b7949280 newargc=15 2024-01-15T18:17:28.719 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[50522]: starting ceph client 2024-01-15T18:17:28.725 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[50522]: starting fuse 2024-01-15T18:17:28.727 INFO:teuthology.orchestra.run.smithi027.stdout:43 2024-01-15T18:17:28.727 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-01-15T18:17:28.727 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:17:28.727 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-01-15T18:17:28.727 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-01-15T18:17:28.727 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-01-15T18:17:28.727 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-01-15T18:17:28.727 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:17:28.727 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-01-15T18:17:28.727 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-15T18:17:28.727 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-01-15T18:17:28.728 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-15T18:17:28.728 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:17:28.728 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-01-15T18:17:28.728 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-01-15T18:17:28.728 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-01-15T18:17:28.728 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-01-15T18:17:28.728 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:17:28.728 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-01-15T18:17:28.728 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:17:28.728 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:17:28.728 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:17:28.728 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-01-15T18:17:28.728 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-01-15T18:17:28.728 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-01-15T18:17:28.728 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:17:28.729 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:17:28.729 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.0")) 2024-01-15T18:17:28.729 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-01-15T18:17:28.850 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.0.50522.asok 2024-01-15T18:17:28.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:17:28.862 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.50522.asok 2024-01-15T18:17:28.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:28.862 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.50522.asok status 2024-01-15T18:17:29.035 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:17:29.035 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-01-15T18:17:29.036 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:17:29.036 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:17:29.036 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "0", 2024-01-15T18:17:29.036 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-01-15T18:17:29.036 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-15T18:17:29.036 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "50522", 2024-01-15T18:17:29.036 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-01-15T18:17:29.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:17:29.036 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-01-15T18:17:29.036 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-01-15T18:17:29.036 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 9135, 2024-01-15T18:17:29.036 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-01-15T18:17:29.036 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-01-15T18:17:29.036 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-01-15T18:17:29.037 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 9135 2024-01-15T18:17:29.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:17:29.037 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-01-15T18:17:29.037 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-01-15T18:17:29.037 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:17:29.037 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 5443726 2024-01-15T18:17:29.037 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:17:29.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:17:29.037 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-01-15T18:17:29.037 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-01-15T18:17:29.037 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:17:29.037 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 5443726 2024-01-15T18:17:29.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:17:29.037 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.9135 192.168.0.1:0/5443726", 2024-01-15T18:17:29.037 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.1:0/5443726", 2024-01-15T18:17:29.038 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-01-15T18:17:29.038 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 67, 2024-01-15T18:17:29.038 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 97, 2024-01-15T18:17:29.038 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:17:29.038 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-01-15T18:17:29.038 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-01-15T18:17:29.038 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:17:29.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:29.051 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-01-15T18:17:29.051 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-15T18:17:29.061 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-01-15T18:17:29.061 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-15T18:17:29.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:29.061 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:17:29.123 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:17:29.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.310+0000 7f8d84000640 1 Processor -- start 2024-01-15T18:17:29.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.310+0000 7f8d84000640 1 -- start start 2024-01-15T18:17:29.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.310+0000 7f8d84000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f8d7c15e7d0 0x7f8d7c0cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:29.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.310+0000 7f8d84000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8d7c0b20d0 0x7f8d7c0cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:29.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.310+0000 7f8d84000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7c0b38b0 con 0x7f8d7c0cedc0 2024-01-15T18:17:29.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.310+0000 7f8d84000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d7c0cd550 con 0x7f8d7c0b20d0 2024-01-15T18:17:29.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.310+0000 7f8d84000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d7c0cd270 con 0x7f8d7c15e7d0 2024-01-15T18:17:29.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.310+0000 7f8d82000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8d7c0b20d0 0x7f8d7c0cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:29.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.310+0000 7f8d80c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8d7c0cedc0 0x7f8d7c0cba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36452/0 (socket says 172.21.15.27:36452) 2024-01-15T18:17:29.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.310+0000 7f8d80c00640 1 -- 172.21.15.27:0/3306917252 learned_addr learned my addr 172.21.15.27:0/3306917252 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:29.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.310+0000 7f8d81600640 1 --2- 172.21.15.27:0/3306917252 >> v2:172.21.15.93:3300/0 conn(0x7f8d7c15e7d0 0x7f8d7c0cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:29.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.310+0000 7f8d82000640 1 -- 172.21.15.27:0/3306917252 >> v2:172.21.15.93:3300/0 conn(0x7f8d7c15e7d0 msgr2=0x7f8d7c0cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:29.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.310+0000 7f8d82000640 1 --2- 172.21.15.27:0/3306917252 >> v2:172.21.15.93:3300/0 conn(0x7f8d7c15e7d0 0x7f8d7c0cf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:29.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.310+0000 7f8d82000640 1 -- 172.21.15.27:0/3306917252 >> v1:172.21.15.27:6789/0 conn(0x7f8d7c0cedc0 legacy=0x7f8d7c0cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:29.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.310+0000 7f8d82000640 1 -- 172.21.15.27:0/3306917252 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7c0b35d0 con 0x7f8d7c0b20d0 2024-01-15T18:17:29.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.310+0000 7f8d7be00640 1 -- 172.21.15.27:0/3306917252 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2370007440 0 0) 0x7f8d7c0b38b0 con 0x7f8d7c0cedc0 2024-01-15T18:17:29.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.314+0000 7f8d82000640 1 --2- 172.21.15.27:0/3306917252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7c0b20d0 0x7f8d7c0cf7e0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c071b00 tx=0x7f8d7401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ee54f59e5e4b16b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:29.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.314+0000 7f8d7be00640 1 -- 172.21.15.27:0/3306917252 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d740a58c0 con 0x7f8d7c0b20d0 2024-01-15T18:17:29.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.314+0000 7f8d7be00640 1 -- 172.21.15.27:0/3306917252 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d740a5a20 con 0x7f8d7c0b20d0 2024-01-15T18:17:29.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.314+0000 7f8d7be00640 1 -- 172.21.15.27:0/3306917252 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d740a5db0 con 0x7f8d7c0b20d0 2024-01-15T18:17:29.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.314+0000 7f8d84000640 1 -- 172.21.15.27:0/3306917252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7c0b20d0 msgr2=0x7f8d7c0cf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:29.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.314+0000 7f8d84000640 1 --2- 172.21.15.27:0/3306917252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7c0b20d0 0x7f8d7c0cf7e0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c071b00 tx=0x7f8d7401c910 comp rx=0 tx=0).stop 2024-01-15T18:17:29.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.314+0000 7f8d84000640 1 -- 172.21.15.27:0/3306917252 shutdown_connections 2024-01-15T18:17:29.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.314+0000 7f8d84000640 1 --2- 172.21.15.27:0/3306917252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7c0b20d0 0x7f8d7c0cf7e0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:29.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.314+0000 7f8d84000640 1 --2- 172.21.15.27:0/3306917252 >> v2:172.21.15.93:3300/0 conn(0x7f8d7c15e7d0 0x7f8d7c0cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:29.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.314+0000 7f8d84000640 1 -- 172.21.15.27:0/3306917252 >> 172.21.15.27:0/3306917252 conn(0x7f8d7c0b2a10 msgr2=0x7f8d7c0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:29.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.314+0000 7f8d84000640 1 -- 172.21.15.27:0/3306917252 shutdown_connections 2024-01-15T18:17:29.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.314+0000 7f8d84000640 1 -- 172.21.15.27:0/3306917252 wait complete. 2024-01-15T18:17:29.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.318+0000 7f8d84000640 1 Processor -- start 2024-01-15T18:17:29.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.318+0000 7f8d84000640 1 -- start start 2024-01-15T18:17:29.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.318+0000 7f8d84000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8d7c0b20d0 0x7f8d7c14e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:29.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.318+0000 7f8d84000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8d7c0cedc0 0x7f8d7c0cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:29.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.318+0000 7f8d84000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7c15e7d0 0x7f8d7c0cf730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:29.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.318+0000 7f8d84000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c15ca40 con 0x7f8d7c15e7d0 2024-01-15T18:17:29.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.318+0000 7f8d84000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c1657b0 con 0x7f8d7c0b20d0 2024-01-15T18:17:29.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.318+0000 7f8d84000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c0ade70 con 0x7f8d7c0cedc0 2024-01-15T18:17:29.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.318+0000 7f8d82000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7c15e7d0 0x7f8d7c0cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:29.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.318+0000 7f8d82000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7c15e7d0 0x7f8d7c0cf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55484/0 (socket says 172.21.15.27:55484) 2024-01-15T18:17:29.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.318+0000 7f8d82000640 1 -- 172.21.15.27:0/2964679845 learned_addr learned my addr 172.21.15.27:0/2964679845 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:29.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.318+0000 7f8d81600640 1 --2- 172.21.15.27:0/2964679845 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8d7c0b20d0 0x7f8d7c14e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:29.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.318+0000 7f8d80c00640 1 --2- 172.21.15.27:0/2964679845 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8d7c0cedc0 0x7f8d7c0cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:29.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.318+0000 7f8d82000640 1 -- 172.21.15.27:0/2964679845 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8d7c0cedc0 msgr2=0x7f8d7c0cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:29.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.318+0000 7f8d82000640 1 --2- 172.21.15.27:0/2964679845 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8d7c0cedc0 0x7f8d7c0cf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:29.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.318+0000 7f8d82000640 1 -- 172.21.15.27:0/2964679845 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8d7c0b20d0 msgr2=0x7f8d7c14e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:29.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.318+0000 7f8d82000640 1 --2- 172.21.15.27:0/2964679845 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8d7c0b20d0 0x7f8d7c14e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:29.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.318+0000 7f8d82000640 1 -- 172.21.15.27:0/2964679845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d74009a70 con 0x7f8d7c15e7d0 2024-01-15T18:17:29.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.318+0000 7f8d81600640 1 --2- 172.21.15.27:0/2964679845 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8d7c0b20d0 0x7f8d7c14e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:17:29.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.318+0000 7f8d82000640 1 --2- 172.21.15.27:0/2964679845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7c15e7d0 0x7f8d7c0cf730 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f8d74019520 tx=0x7f8d740196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:29.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.318+0000 7f8d79600640 1 -- 172.21.15.27:0/2964679845 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d740b9860 con 0x7f8d7c15e7d0 2024-01-15T18:17:29.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.318+0000 7f8d79600640 1 -- 172.21.15.27:0/2964679845 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d740b99c0 con 0x7f8d7c15e7d0 2024-01-15T18:17:29.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.318+0000 7f8d84000640 1 -- 172.21.15.27:0/2964679845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d7c0ae150 con 0x7f8d7c15e7d0 2024-01-15T18:17:29.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.318+0000 7f8d79600640 1 -- 172.21.15.27:0/2964679845 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d740b9d70 con 0x7f8d7c15e7d0 2024-01-15T18:17:29.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.318+0000 7f8d84000640 1 -- 172.21.15.27:0/2964679845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d7c14e860 con 0x7f8d7c15e7d0 2024-01-15T18:17:29.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.322+0000 7f8d84000640 1 -- 172.21.15.27:0/2964679845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d7c0ae670 con 0x7f8d7c15e7d0 2024-01-15T18:17:29.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.322+0000 7f8d79600640 1 -- 172.21.15.27:0/2964679845 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f8d740c4460 con 0x7f8d7c15e7d0 2024-01-15T18:17:29.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.322+0000 7f8d79600640 1 --2- 172.21.15.27:0/2964679845 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8d54067700 0x7f8d54069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:29.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.322+0000 7f8d79600640 1 -- 172.21.15.27:0/2964679845 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f8d7411ec40 con 0x7f8d7c15e7d0 2024-01-15T18:17:29.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.322+0000 7f8d81600640 1 --2- 172.21.15.27:0/2964679845 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8d54067700 0x7f8d54069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:29.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.326+0000 7f8d81600640 1 --2- 172.21.15.27:0/2964679845 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8d54067700 0x7f8d54069bc0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c00b9f0 tx=0x7f8d6c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:29.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.326+0000 7f8d79600640 1 -- 172.21.15.27:0/2964679845 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8d740ed910 con 0x7f8d7c15e7d0 2024-01-15T18:17:29.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.518+0000 7f8d84000640 1 -- 172.21.15.27:0/2964679845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8d7c0ae670 con 0x7f8d7c15e7d0 2024-01-15T18:17:29.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.518+0000 7f8d79600640 1 -- 172.21.15.27:0/2964679845 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v67) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8d7c0ae670 con 0x7f8d7c15e7d0 2024-01-15T18:17:29.517 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:17:29.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.518+0000 7f8d84000640 1 -- 172.21.15.27:0/2964679845 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8d54067700 msgr2=0x7f8d54069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:29.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.518+0000 7f8d84000640 1 --2- 172.21.15.27:0/2964679845 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8d54067700 0x7f8d54069bc0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c00b9f0 tx=0x7f8d6c008040 comp rx=0 tx=0).stop 2024-01-15T18:17:29.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.518+0000 7f8d84000640 1 -- 172.21.15.27:0/2964679845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7c15e7d0 msgr2=0x7f8d7c0cf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:29.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.518+0000 7f8d84000640 1 --2- 172.21.15.27:0/2964679845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7c15e7d0 0x7f8d7c0cf730 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f8d74019520 tx=0x7f8d740196b0 comp rx=0 tx=0).stop 2024-01-15T18:17:29.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.518+0000 7f8d84000640 1 -- 172.21.15.27:0/2964679845 shutdown_connections 2024-01-15T18:17:29.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.518+0000 7f8d84000640 1 --2- 172.21.15.27:0/2964679845 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8d54067700 0x7f8d54069bc0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:29.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.518+0000 7f8d84000640 1 --2- 172.21.15.27:0/2964679845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7c15e7d0 0x7f8d7c0cf730 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:29.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.518+0000 7f8d84000640 1 --2- 172.21.15.27:0/2964679845 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8d7c0cedc0 0x7f8d7c0cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:29.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.518+0000 7f8d84000640 1 --2- 172.21.15.27:0/2964679845 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8d7c0b20d0 0x7f8d7c14e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:29.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.518+0000 7f8d84000640 1 -- 172.21.15.27:0/2964679845 >> 172.21.15.27:0/2964679845 conn(0x7f8d7c0b2a10 msgr2=0x7f8d7c15d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:29.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.518+0000 7f8d84000640 1 -- 172.21.15.27:0/2964679845 shutdown_connections 2024-01-15T18:17:29.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:29.518+0000 7f8d84000640 1 -- 172.21.15.27:0/2964679845 wait complete. 2024-01-15T18:17:29.539 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:17:29.539 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:17:29.539 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi093.front.sepia.ceph.com 2024-01-15T18:17:29.539 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-15T18:17:29.540 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-15T18:17:29.540 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:17:29.540 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:17:29.540 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:17:29.540 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:17:29.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:29.540 DEBUG:teuthology.orchestra.run.smithi093:> ip addr 2024-01-15T18:17:29.551 INFO:teuthology.orchestra.run.smithi093.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:17:29.551 INFO:teuthology.orchestra.run.smithi093.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:17:29.551 INFO:teuthology.orchestra.run.smithi093.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:17:29.552 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:17:29.552 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 ::1/128 scope host 2024-01-15T18:17:29.552 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:17:29.552 INFO:teuthology.orchestra.run.smithi093.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:17:29.552 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:d8:da:1c brd ff:ff:ff:ff:ff:ff 2024-01-15T18:17:29.552 INFO:teuthology.orchestra.run.smithi093.stdout: altname enp7s0f0 2024-01-15T18:17:29.552 INFO:teuthology.orchestra.run.smithi093.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:17:29.552 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:d8:da:1d brd ff:ff:ff:ff:ff:ff 2024-01-15T18:17:29.552 INFO:teuthology.orchestra.run.smithi093.stdout: altname enp7s0f1 2024-01-15T18:17:29.552 INFO:teuthology.orchestra.run.smithi093.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:17:29.553 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:8f:d6:34 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:17:29.553 INFO:teuthology.orchestra.run.smithi093.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:17:29.553 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:8f:d6:35 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:17:29.553 INFO:teuthology.orchestra.run.smithi093.stdout: inet 172.21.15.93/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-01-15T18:17:29.553 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft 41689sec preferred_lft 41689sec 2024-01-15T18:17:29.553 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::ec4:7aff:fe8f:d635/64 scope link 2024-01-15T18:17:29.553 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:17:29.553 INFO:teuthology.orchestra.run.smithi093.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-01-15T18:17:29.553 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 02:42:2a:61:fc:7d brd ff:ff:ff:ff:ff:ff 2024-01-15T18:17:29.553 INFO:teuthology.orchestra.run.smithi093.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-01-15T18:17:29.553 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:17:29.554 INFO:teuthology.orchestra.run.smithi093.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:17:29.554 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-01-15T18:17:29.554 INFO:teuthology.orchestra.run.smithi093.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:17:29.554 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:17:29.554 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-01-15T18:17:29.554 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:17:29.554 INFO:teuthology.orchestra.run.smithi093.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:17:29.554 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:17:29.554 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-01-15T18:17:29.555 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:17:29.555 INFO:teuthology.orchestra.run.smithi093.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:17:29.555 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:17:29.555 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-01-15T18:17:29.555 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:17:29.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:29.555 DEBUG:teuthology.orchestra.run.smithi093:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:17:29.615 INFO:teuthology.orchestra.run.smithi093.stdout:1 2024-01-15T18:17:29.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:29.617 DEBUG:teuthology.orchestra.run.smithi093:> route 2024-01-15T18:17:29.670 INFO:teuthology.orchestra.run.smithi093.stdout:Kernel IP routing table 2024-01-15T18:17:29.670 INFO:teuthology.orchestra.run.smithi093.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:17:29.671 INFO:teuthology.orchestra.run.smithi093.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-01-15T18:17:29.671 INFO:teuthology.orchestra.run.smithi093.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T18:17:29.671 INFO:teuthology.orchestra.run.smithi093.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-01-15T18:17:29.671 INFO:teuthology.orchestra.run.smithi093.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T18:17:29.671 INFO:teuthology.orchestra.run.smithi093.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:17:29.671 INFO:teuthology.orchestra.run.smithi093.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:17:29.671 INFO:teuthology.orchestra.run.smithi093.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:17:29.671 INFO:teuthology.orchestra.run.smithi093.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:17:29.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:29.672 DEBUG:teuthology.orchestra.run.smithi093:> (cd / && exec bash -c ' 2024-01-15T18:17:29.672 DEBUG:teuthology.orchestra.run.smithi093:> set -e 2024-01-15T18:17:29.672 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T18:17:29.672 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T18:17:29.672 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T18:17:29.672 DEBUG:teuthology.orchestra.run.smithi093:> ') 2024-01-15T18:17:29.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:29.764 DEBUG:teuthology.orchestra.run.smithi093:> ip netns list 2024-01-15T18:17:29.773 INFO:teuthology.orchestra.run.smithi093.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-15T18:17:29.773 INFO:teuthology.orchestra.run.smithi093.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-15T18:17:29.774 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:17:29.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:29.774 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip link set brx.0 up 2024-01-15T18:17:29.836 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-15T18:17:29.836 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:17:29.836 DEBUG:teuthology.orchestra.run.smithi093:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-15T18:17:29.887 INFO:teuthology.orchestra.run.smithi093.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:17:29.887 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:17:29.887 DEBUG:teuthology.orchestra.run.smithi093:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:17:29.939 DEBUG:teuthology.orchestra.run.smithi093:> sudo modprobe fuse 2024-01-15T18:17:30.000 DEBUG:teuthology.orchestra.run.smithi093:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:17:30.057 INFO:teuthology.orchestra.run.smithi093.stdout:/sys 2024-01-15T18:17:30.057 INFO:teuthology.orchestra.run.smithi093.stdout:/proc 2024-01-15T18:17:30.057 INFO:teuthology.orchestra.run.smithi093.stdout:/dev 2024-01-15T18:17:30.057 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/pts 2024-01-15T18:17:30.057 INFO:teuthology.orchestra.run.smithi093.stdout:/run 2024-01-15T18:17:30.058 INFO:teuthology.orchestra.run.smithi093.stdout:/ 2024-01-15T18:17:30.058 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/security 2024-01-15T18:17:30.058 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/shm 2024-01-15T18:17:30.058 INFO:teuthology.orchestra.run.smithi093.stdout:/run/lock 2024-01-15T18:17:30.058 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/cgroup 2024-01-15T18:17:30.058 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/pstore 2024-01-15T18:17:30.058 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/bpf 2024-01-15T18:17:30.058 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:17:30.058 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/hugepages 2024-01-15T18:17:30.058 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/mqueue 2024-01-15T18:17:30.058 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/debug 2024-01-15T18:17:30.058 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/tracing 2024-01-15T18:17:30.058 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/fs/nfsd 2024-01-15T18:17:30.058 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/fuse/connections 2024-01-15T18:17:30.058 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/config 2024-01-15T18:17:30.059 INFO:teuthology.orchestra.run.smithi093.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:17:30.059 INFO:teuthology.orchestra.run.smithi093.stdout:/run/rpc_pipefs 2024-01-15T18:17:30.059 INFO:teuthology.orchestra.run.smithi093.stdout:/run/user/1000 2024-01-15T18:17:30.059 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph 2024-01-15T18:17:30.059 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:17:30.059 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:17:30.059 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:17:30.059 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:17:30.059 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns 2024-01-15T18:17:30.059 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:17:30.059 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:17:30.059 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:17:30.059 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:17:30.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:30.060 DEBUG:teuthology.orchestra.run.smithi093:> ls /sys/fs/fuse/connections 2024-01-15T18:17:30.111 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:17:30.111 DEBUG:teuthology.orchestra.run.smithi093:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-15T18:17:30.158 DEBUG:teuthology.orchestra.run.smithi093:> sudo modprobe fuse 2024-01-15T18:17:30.179 DEBUG:teuthology.orchestra.run.smithi093:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:17:30.236 INFO:teuthology.orchestra.run.smithi093.stdout:/sys 2024-01-15T18:17:30.237 INFO:teuthology.orchestra.run.smithi093.stdout:/proc 2024-01-15T18:17:30.237 INFO:teuthology.orchestra.run.smithi093.stdout:/dev 2024-01-15T18:17:30.237 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/pts 2024-01-15T18:17:30.237 INFO:teuthology.orchestra.run.smithi093.stdout:/run 2024-01-15T18:17:30.237 INFO:teuthology.orchestra.run.smithi093.stdout:/ 2024-01-15T18:17:30.237 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/security 2024-01-15T18:17:30.237 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/shm 2024-01-15T18:17:30.237 INFO:teuthology.orchestra.run.smithi093.stdout:/run/lock 2024-01-15T18:17:30.237 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/cgroup 2024-01-15T18:17:30.237 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/pstore 2024-01-15T18:17:30.237 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/bpf 2024-01-15T18:17:30.237 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:17:30.237 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/hugepages 2024-01-15T18:17:30.237 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/mqueue 2024-01-15T18:17:30.238 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/debug 2024-01-15T18:17:30.238 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/tracing 2024-01-15T18:17:30.238 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/fs/nfsd 2024-01-15T18:17:30.238 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/fuse/connections 2024-01-15T18:17:30.238 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/config 2024-01-15T18:17:30.238 INFO:teuthology.orchestra.run.smithi093.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:17:30.238 INFO:teuthology.orchestra.run.smithi093.stdout:/run/rpc_pipefs 2024-01-15T18:17:30.238 INFO:teuthology.orchestra.run.smithi093.stdout:/run/user/1000 2024-01-15T18:17:30.238 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph 2024-01-15T18:17:30.238 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:17:30.238 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:17:30.238 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:17:30.238 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:17:30.238 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns 2024-01-15T18:17:30.238 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:17:30.238 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:17:30.239 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:17:30.239 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:17:30.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:30.240 DEBUG:teuthology.orchestra.run.smithi093:> ls /sys/fs/fuse/connections 2024-01-15T18:17:30.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:2024-01-15T18:17:30.260+0000 7f1406218580 -1 init, newargv = 0x55e0e986b490 newargc=15 2024-01-15T18:17:30.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:ceph-fuse[34581]: starting ceph client 2024-01-15T18:17:30.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:ceph-fuse[34581]: starting fuse 2024-01-15T18:17:31.366 DEBUG:teuthology.orchestra.run.smithi093:> sudo modprobe fuse 2024-01-15T18:17:31.388 DEBUG:teuthology.orchestra.run.smithi093:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:17:31.441 INFO:teuthology.orchestra.run.smithi093.stdout:/sys 2024-01-15T18:17:31.441 INFO:teuthology.orchestra.run.smithi093.stdout:/proc 2024-01-15T18:17:31.441 INFO:teuthology.orchestra.run.smithi093.stdout:/dev 2024-01-15T18:17:31.441 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/pts 2024-01-15T18:17:31.441 INFO:teuthology.orchestra.run.smithi093.stdout:/run 2024-01-15T18:17:31.441 INFO:teuthology.orchestra.run.smithi093.stdout:/ 2024-01-15T18:17:31.441 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/security 2024-01-15T18:17:31.441 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/shm 2024-01-15T18:17:31.442 INFO:teuthology.orchestra.run.smithi093.stdout:/run/lock 2024-01-15T18:17:31.442 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/cgroup 2024-01-15T18:17:31.442 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/pstore 2024-01-15T18:17:31.442 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/bpf 2024-01-15T18:17:31.442 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:17:31.442 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/hugepages 2024-01-15T18:17:31.442 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/mqueue 2024-01-15T18:17:31.442 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/debug 2024-01-15T18:17:31.442 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/tracing 2024-01-15T18:17:31.442 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/fs/nfsd 2024-01-15T18:17:31.442 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/fuse/connections 2024-01-15T18:17:31.442 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/config 2024-01-15T18:17:31.442 INFO:teuthology.orchestra.run.smithi093.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:17:31.442 INFO:teuthology.orchestra.run.smithi093.stdout:/run/rpc_pipefs 2024-01-15T18:17:31.442 INFO:teuthology.orchestra.run.smithi093.stdout:/run/user/1000 2024-01-15T18:17:31.443 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph 2024-01-15T18:17:31.443 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:17:31.443 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:17:31.443 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:17:31.443 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:17:31.443 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns 2024-01-15T18:17:31.443 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:17:31.443 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:17:31.443 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:17:31.443 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:17:31.443 INFO:teuthology.orchestra.run.smithi093.stdout:/home/ubuntu/cephtest/mnt.1 2024-01-15T18:17:31.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:31.444 DEBUG:teuthology.orchestra.run.smithi093:> ls /sys/fs/fuse/connections 2024-01-15T18:17:31.495 INFO:teuthology.orchestra.run.smithi093.stdout:43 2024-01-15T18:17:31.495 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-01-15T18:17:31.495 DEBUG:teuthology.orchestra.run.smithi093:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:17:31.495 DEBUG:teuthology.orchestra.run.smithi093:> import glob 2024-01-15T18:17:31.495 DEBUG:teuthology.orchestra.run.smithi093:> import re 2024-01-15T18:17:31.495 DEBUG:teuthology.orchestra.run.smithi093:> import os 2024-01-15T18:17:31.495 DEBUG:teuthology.orchestra.run.smithi093:> import subprocess 2024-01-15T18:17:31.496 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:17:31.496 DEBUG:teuthology.orchestra.run.smithi093:> def _find_admin_socket(client_name): 2024-01-15T18:17:31.496 DEBUG:teuthology.orchestra.run.smithi093:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-15T18:17:31.496 DEBUG:teuthology.orchestra.run.smithi093:> files = glob.glob(asok_path) 2024-01-15T18:17:31.496 DEBUG:teuthology.orchestra.run.smithi093:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-15T18:17:31.496 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:17:31.496 DEBUG:teuthology.orchestra.run.smithi093:> # Given a non-glob path, it better be there 2024-01-15T18:17:31.496 DEBUG:teuthology.orchestra.run.smithi093:> if "*" not in asok_path: 2024-01-15T18:17:31.496 DEBUG:teuthology.orchestra.run.smithi093:> assert(len(files) == 1) 2024-01-15T18:17:31.496 DEBUG:teuthology.orchestra.run.smithi093:> return files[0] 2024-01-15T18:17:31.496 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:17:31.496 DEBUG:teuthology.orchestra.run.smithi093:> for f in files: 2024-01-15T18:17:31.496 DEBUG:teuthology.orchestra.run.smithi093:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:17:31.496 DEBUG:teuthology.orchestra.run.smithi093:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:17:31.496 DEBUG:teuthology.orchestra.run.smithi093:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:17:31.496 DEBUG:teuthology.orchestra.run.smithi093:> contents = proc_f.read() 2024-01-15T18:17:31.496 DEBUG:teuthology.orchestra.run.smithi093:> if mountpoint in contents: 2024-01-15T18:17:31.497 DEBUG:teuthology.orchestra.run.smithi093:> return f 2024-01-15T18:17:31.497 DEBUG:teuthology.orchestra.run.smithi093:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:17:31.497 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:17:31.497 DEBUG:teuthology.orchestra.run.smithi093:> print(_find_admin_socket("client.1")) 2024-01-15T18:17:31.497 DEBUG:teuthology.orchestra.run.smithi093:> ' 2024-01-15T18:17:31.620 INFO:teuthology.orchestra.run.smithi093.stdout:/var/run/ceph/ceph-client.1.34581.asok 2024-01-15T18:17:31.624 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:17:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:17:31.632 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.34581.asok 2024-01-15T18:17:31.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:31.632 DEBUG:teuthology.orchestra.run.smithi093:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.34581.asok status 2024-01-15T18:17:31.807 INFO:teuthology.orchestra.run.smithi093.stdout:{ 2024-01-15T18:17:31.807 INFO:teuthology.orchestra.run.smithi093.stdout: "metadata": { 2024-01-15T18:17:31.807 INFO:teuthology.orchestra.run.smithi093.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:17:31.807 INFO:teuthology.orchestra.run.smithi093.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:17:31.807 INFO:teuthology.orchestra.run.smithi093.stdout: "entity_id": "1", 2024-01-15T18:17:31.808 INFO:teuthology.orchestra.run.smithi093.stdout: "hostname": "smithi093", 2024-01-15T18:17:31.808 INFO:teuthology.orchestra.run.smithi093.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-15T18:17:31.808 INFO:teuthology.orchestra.run.smithi093.stdout: "pid": "34581", 2024-01-15T18:17:31.808 INFO:teuthology.orchestra.run.smithi093.stdout: "root": "/" 2024-01-15T18:17:31.808 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:17:31.808 INFO:teuthology.orchestra.run.smithi093.stdout: "dentry_count": 0, 2024-01-15T18:17:31.808 INFO:teuthology.orchestra.run.smithi093.stdout: "dentry_pinned_count": 0, 2024-01-15T18:17:31.808 INFO:teuthology.orchestra.run.smithi093.stdout: "id": 7514, 2024-01-15T18:17:31.808 INFO:teuthology.orchestra.run.smithi093.stdout: "inst": { 2024-01-15T18:17:31.808 INFO:teuthology.orchestra.run.smithi093.stdout: "name": { 2024-01-15T18:17:31.808 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "client", 2024-01-15T18:17:31.808 INFO:teuthology.orchestra.run.smithi093.stdout: "num": 7514 2024-01-15T18:17:31.808 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:17:31.808 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": { 2024-01-15T18:17:31.809 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "v1", 2024-01-15T18:17:31.809 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:17:31.809 INFO:teuthology.orchestra.run.smithi093.stdout: "nonce": 2952365219 2024-01-15T18:17:31.809 INFO:teuthology.orchestra.run.smithi093.stdout: } 2024-01-15T18:17:31.809 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:17:31.809 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": { 2024-01-15T18:17:31.809 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "v1", 2024-01-15T18:17:31.809 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:17:31.809 INFO:teuthology.orchestra.run.smithi093.stdout: "nonce": 2952365219 2024-01-15T18:17:31.809 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:17:31.809 INFO:teuthology.orchestra.run.smithi093.stdout: "inst_str": "client.7514 192.168.0.1:0/2952365219", 2024-01-15T18:17:31.809 INFO:teuthology.orchestra.run.smithi093.stdout: "addr_str": "192.168.0.1:0/2952365219", 2024-01-15T18:17:31.809 INFO:teuthology.orchestra.run.smithi093.stdout: "inode_count": 1, 2024-01-15T18:17:31.809 INFO:teuthology.orchestra.run.smithi093.stdout: "mds_epoch": 67, 2024-01-15T18:17:31.809 INFO:teuthology.orchestra.run.smithi093.stdout: "osd_epoch": 97, 2024-01-15T18:17:31.810 INFO:teuthology.orchestra.run.smithi093.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:17:31.810 INFO:teuthology.orchestra.run.smithi093.stdout: "blocklisted": false, 2024-01-15T18:17:31.810 INFO:teuthology.orchestra.run.smithi093.stdout: "fs_name": "cephfs" 2024-01-15T18:17:31.810 INFO:teuthology.orchestra.run.smithi093.stdout:} 2024-01-15T18:17:31.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:31.822 DEBUG:teuthology.orchestra.run.smithi093:> stat --file-system '--printf=%T 2024-01-15T18:17:31.822 DEBUG:teuthology.orchestra.run.smithi093:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-15T18:17:31.831 INFO:teuthology.orchestra.run.smithi093.stdout:fuseblk 2024-01-15T18:17:31.831 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-15T18:17:31.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:31.831 DEBUG:teuthology.orchestra.run.smithi093:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:17:31.898 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-15T18:17:32.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.054+0000 7f467a600640 1 Processor -- start 2024-01-15T18:17:32.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.054+0000 7f467a600640 1 -- start start 2024-01-15T18:17:32.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.058+0000 7f467a600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f467406b600 0x7f467406bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:32.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.058+0000 7f467a600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46740b04b0 con 0x7f467415d3b0 2024-01-15T18:17:32.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.058+0000 7f467a600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46740b0720 con 0x7f467406c220 2024-01-15T18:17:32.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.058+0000 7f467a600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f467406c6b0 con 0x7f467406b600 2024-01-15T18:17:32.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.058+0000 7f4673400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f467415d3b0 0x7f46740cfc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36466/0 (socket says 172.21.15.27:36466) 2024-01-15T18:17:32.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.058+0000 7f4673400640 1 -- 172.21.15.27:0/609632606 learned_addr learned my addr 172.21.15.27:0/609632606 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:32.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.058+0000 7f4672a00640 1 --2- 172.21.15.27:0/609632606 >> v2:172.21.15.93:3300/0 conn(0x7f467406b600 0x7f467406bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:32.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.058+0000 7f4672000640 1 -- 172.21.15.27:0/609632606 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86143512 0 0) 0x7f46740b04b0 con 0x7f467415d3b0 2024-01-15T18:17:32.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.058+0000 7f4672000640 1 -- 172.21.15.27:0/609632606 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4654003530 con 0x7f467415d3b0 2024-01-15T18:17:32.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.058+0000 7f4672000640 1 -- 172.21.15.27:0/609632606 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783398603 0 0) 0x7f46740b0720 con 0x7f467406c220 2024-01-15T18:17:32.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.058+0000 7f4672000640 1 -- 172.21.15.27:0/609632606 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46740b04b0 con 0x7f467406c220 2024-01-15T18:17:32.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.058+0000 7f4672a00640 1 -- 172.21.15.27:0/609632606 >> v1:172.21.15.93:6789/0 conn(0x7f467406c220 legacy=0x7f46740b1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:32.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.058+0000 7f4672a00640 1 -- 172.21.15.27:0/609632606 >> v1:172.21.15.27:6789/0 conn(0x7f467415d3b0 legacy=0x7f46740cfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:32.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.058+0000 7f4672a00640 1 -- 172.21.15.27:0/609632606 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46740b0a00 con 0x7f467406b600 2024-01-15T18:17:32.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.062+0000 7f4672a00640 1 --2- 172.21.15.27:0/609632606 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f467406b600 0x7f467406bce0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f465c0097c0 tx=0x7f465c01c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9eca7c0103fec9b6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:32.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.062+0000 7f4672000640 1 -- 172.21.15.27:0/609632606 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465c0ab5f0 con 0x7f467406b600 2024-01-15T18:17:32.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.062+0000 7f4672000640 1 -- 172.21.15.27:0/609632606 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f465c0ab750 con 0x7f467406b600 2024-01-15T18:17:32.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.062+0000 7f4672000640 1 -- 172.21.15.27:0/609632606 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465c0abae0 con 0x7f467406b600 2024-01-15T18:17:32.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.062+0000 7f467a600640 1 -- 172.21.15.27:0/609632606 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f467406b600 msgr2=0x7f467406bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:32.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.062+0000 7f467a600640 1 --2- 172.21.15.27:0/609632606 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f467406b600 0x7f467406bce0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f465c0097c0 tx=0x7f465c01c450 comp rx=0 tx=0).stop 2024-01-15T18:17:32.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.062+0000 7f467a600640 1 -- 172.21.15.27:0/609632606 shutdown_connections 2024-01-15T18:17:32.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.062+0000 7f467a600640 1 --2- 172.21.15.27:0/609632606 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f467406b600 0x7f467406bce0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:32.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.062+0000 7f467a600640 1 -- 172.21.15.27:0/609632606 >> 172.21.15.27:0/609632606 conn(0x7f46740d0ed0 msgr2=0x7f46740d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:32.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.062+0000 7f467a600640 1 -- 172.21.15.27:0/609632606 shutdown_connections 2024-01-15T18:17:32.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.062+0000 7f467a600640 1 -- 172.21.15.27:0/609632606 wait complete. 2024-01-15T18:17:32.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.062+0000 7f467a600640 1 Processor -- start 2024-01-15T18:17:32.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.062+0000 7f467a600640 1 -- start start 2024-01-15T18:17:32.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.066+0000 7f467a600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f467406b600 0x7f467414aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:32.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.066+0000 7f467a600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f467406c220 0x7f4674156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:32.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.066+0000 7f467a600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f467415d3b0 0x7f46741572d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:32.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.066+0000 7f467a600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467414deb0 con 0x7f467415d3b0 2024-01-15T18:17:32.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.066+0000 7f467a600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467414d2a0 con 0x7f467406c220 2024-01-15T18:17:32.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.066+0000 7f467a600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f467414d580 con 0x7f467406b600 2024-01-15T18:17:32.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.066+0000 7f4673e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f467415d3b0 0x7f46741572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:32.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.066+0000 7f4673e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f467415d3b0 0x7f46741572d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55490/0 (socket says 172.21.15.27:55490) 2024-01-15T18:17:32.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.066+0000 7f4673e00640 1 -- 172.21.15.27:0/212070964 learned_addr learned my addr 172.21.15.27:0/212070964 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:32.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.066+0000 7f4672a00640 1 --2- 172.21.15.27:0/212070964 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f467406c220 0x7f4674156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:32.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.066+0000 7f4673400640 1 --2- 172.21.15.27:0/212070964 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f467406b600 0x7f467414aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:32.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.066+0000 7f4673e00640 1 -- 172.21.15.27:0/212070964 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f467406b600 msgr2=0x7f467414aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:32.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.066+0000 7f4673e00640 1 --2- 172.21.15.27:0/212070964 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f467406b600 0x7f467414aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:32.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.066+0000 7f4673e00640 1 -- 172.21.15.27:0/212070964 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f467406c220 msgr2=0x7f4674156d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:32.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.066+0000 7f4673e00640 1 --2- 172.21.15.27:0/212070964 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f467406c220 0x7f4674156d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:32.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.066+0000 7f4673e00640 1 -- 172.21.15.27:0/212070964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f465c009420 con 0x7f467415d3b0 2024-01-15T18:17:32.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.066+0000 7f4672a00640 1 --2- 172.21.15.27:0/212070964 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f467406c220 0x7f4674156d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:17:32.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.066+0000 7f4673e00640 1 --2- 172.21.15.27:0/212070964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f467415d3b0 0x7f46741572d0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f466c00c730 tx=0x7f466c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:32.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.066+0000 7f466b400640 1 -- 172.21.15.27:0/212070964 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466c003b80 con 0x7f467415d3b0 2024-01-15T18:17:32.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.066+0000 7f466b400640 1 -- 172.21.15.27:0/212070964 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466c003ce0 con 0x7f467415d3b0 2024-01-15T18:17:32.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.066+0000 7f466b400640 1 -- 172.21.15.27:0/212070964 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466c00aa90 con 0x7f467415d3b0 2024-01-15T18:17:32.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.066+0000 7f467a600640 1 -- 172.21.15.27:0/212070964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f467414d860 con 0x7f467415d3b0 2024-01-15T18:17:32.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.066+0000 7f467a600640 1 -- 172.21.15.27:0/212070964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f467414b530 con 0x7f467415d3b0 2024-01-15T18:17:32.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.066+0000 7f467a600640 1 -- 172.21.15.27:0/212070964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f467404e850 con 0x7f467415d3b0 2024-01-15T18:17:32.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.070+0000 7f466b400640 1 -- 172.21.15.27:0/212070964 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f466c00abf0 con 0x7f467415d3b0 2024-01-15T18:17:32.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.070+0000 7f466b400640 1 --2- 172.21.15.27:0/212070964 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f464c067730 0x7f464c069bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.070+0000 7f466b400640 1 -- 172.21.15.27:0/212070964 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f466c082e20 con 0x7f467415d3b0 2024-01-15T18:17:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.070+0000 7f4673400640 1 --2- 172.21.15.27:0/212070964 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f464c067730 0x7f464c069bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:32.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.070+0000 7f4673400640 1 --2- 172.21.15.27:0/212070964 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f464c067730 0x7f464c069bf0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f46640011f0 tx=0x7f4664007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:32.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.074+0000 7f466b400640 1 -- 172.21.15.27:0/212070964 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f466c0518a0 con 0x7f467415d3b0 2024-01-15T18:17:32.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:32.266+0000 7f467a600640 1 -- 172.21.15.27:0/212070964 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f46740b6e50 con 0x7f467415d3b0 2024-01-15T18:17:33.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.190+0000 7f466b400640 1 -- 172.21.15.27:0/212070964 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v68) v1 ==== 136+0+0 (secure 0 0 0) 0x7f466c054ec0 con 0x7f467415d3b0 2024-01-15T18:17:33.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.190+0000 7f467a600640 1 -- 172.21.15.27:0/212070964 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f464c067730 msgr2=0x7f464c069bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:33.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.190+0000 7f467a600640 1 --2- 172.21.15.27:0/212070964 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f464c067730 0x7f464c069bf0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f46640011f0 tx=0x7f4664007d40 comp rx=0 tx=0).stop 2024-01-15T18:17:33.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.190+0000 7f467a600640 1 -- 172.21.15.27:0/212070964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f467415d3b0 msgr2=0x7f46741572d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:33.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.190+0000 7f467a600640 1 --2- 172.21.15.27:0/212070964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f467415d3b0 0x7f46741572d0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f466c00c730 tx=0x7f466c00cbf0 comp rx=0 tx=0).stop 2024-01-15T18:17:33.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.190+0000 7f467a600640 1 -- 172.21.15.27:0/212070964 shutdown_connections 2024-01-15T18:17:33.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.190+0000 7f467a600640 1 --2- 172.21.15.27:0/212070964 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f464c067730 0x7f464c069bf0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:33.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.190+0000 7f467a600640 1 --2- 172.21.15.27:0/212070964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f467415d3b0 0x7f46741572d0 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:33.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.190+0000 7f467a600640 1 --2- 172.21.15.27:0/212070964 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f467406c220 0x7f4674156d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:33.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.190+0000 7f467a600640 1 --2- 172.21.15.27:0/212070964 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f467406b600 0x7f467414aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:33.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.190+0000 7f467a600640 1 -- 172.21.15.27:0/212070964 >> 172.21.15.27:0/212070964 conn(0x7f46740d0ed0 msgr2=0x7f46740afa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:33.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.190+0000 7f467a600640 1 -- 172.21.15.27:0/212070964 shutdown_connections 2024-01-15T18:17:33.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.190+0000 7f467a600640 1 -- 172.21.15.27:0/212070964 wait complete. 2024-01-15T18:17:33.216 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:17:33.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.362+0000 7f7becc00640 1 Processor -- start 2024-01-15T18:17:33.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.362+0000 7f7becc00640 1 -- start start 2024-01-15T18:17:33.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.366+0000 7f7becc00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f7be8143da0 0x7f7be806d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:33.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.366+0000 7f7becc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7be806c200 0x7f7be80cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:33.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.366+0000 7f7becc00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7be806c750 con 0x7f7be80cad40 2024-01-15T18:17:33.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.366+0000 7f7becc00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7be8144180 con 0x7f7be806c200 2024-01-15T18:17:33.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.366+0000 7f7becc00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7be806ca30 con 0x7f7be8143da0 2024-01-15T18:17:33.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.366+0000 7f7be5400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7be806c200 0x7f7be80cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:33.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.366+0000 7f7be5400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7be806c200 0x7f7be80cb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55506/0 (socket says 172.21.15.27:55506) 2024-01-15T18:17:33.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.366+0000 7f7be5400640 1 -- 172.21.15.27:0/1765379557 learned_addr learned my addr 172.21.15.27:0/1765379557 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:33.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.366+0000 7f7be5e00640 1 --2- 172.21.15.27:0/1765379557 >> v2:172.21.15.93:3300/0 conn(0x7f7be8143da0 0x7f7be806d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:33.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.366+0000 7f7be5400640 1 -- 172.21.15.27:0/1765379557 >> v2:172.21.15.93:3300/0 conn(0x7f7be8143da0 msgr2=0x7f7be806d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:33.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.366+0000 7f7be5400640 1 --2- 172.21.15.27:0/1765379557 >> v2:172.21.15.93:3300/0 conn(0x7f7be8143da0 0x7f7be806d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:33.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.366+0000 7f7be5400640 1 -- 172.21.15.27:0/1765379557 >> v1:172.21.15.93:6789/0 conn(0x7f7be80cad40 legacy=0x7f7be80701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:33.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.366+0000 7f7be5400640 1 -- 172.21.15.27:0/1765379557 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be806cd10 con 0x7f7be806c200 2024-01-15T18:17:33.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.366+0000 7f7be5e00640 1 --2- 172.21.15.27:0/1765379557 >> v2:172.21.15.93:3300/0 conn(0x7f7be8143da0 0x7f7be806d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:17:33.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.370+0000 7f7be5400640 1 --2- 172.21.15.27:0/1765379557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7be806c200 0x7f7be80cb9c0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f7be806bc10 tx=0x7f7bd001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e01b904b62d1ceb1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:33.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.370+0000 7f7be4a00640 1 -- 172.21.15.27:0/1765379557 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd00aa5f0 con 0x7f7be806c200 2024-01-15T18:17:33.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.370+0000 7f7be4a00640 1 -- 172.21.15.27:0/1765379557 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd00aa750 con 0x7f7be806c200 2024-01-15T18:17:33.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.370+0000 7f7be4a00640 1 -- 172.21.15.27:0/1765379557 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd00aaae0 con 0x7f7be806c200 2024-01-15T18:17:33.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.370+0000 7f7becc00640 1 -- 172.21.15.27:0/1765379557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7be806c200 msgr2=0x7f7be80cb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:33.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.370+0000 7f7becc00640 1 --2- 172.21.15.27:0/1765379557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7be806c200 0x7f7be80cb9c0 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f7be806bc10 tx=0x7f7bd001c490 comp rx=0 tx=0).stop 2024-01-15T18:17:33.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.370+0000 7f7becc00640 1 -- 172.21.15.27:0/1765379557 shutdown_connections 2024-01-15T18:17:33.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.370+0000 7f7becc00640 1 --2- 172.21.15.27:0/1765379557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7be806c200 0x7f7be80cb9c0 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:33.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.370+0000 7f7becc00640 1 --2- 172.21.15.27:0/1765379557 >> v2:172.21.15.93:3300/0 conn(0x7f7be8143da0 0x7f7be806d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:33.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.370+0000 7f7becc00640 1 -- 172.21.15.27:0/1765379557 >> 172.21.15.27:0/1765379557 conn(0x7f7be80b0850 msgr2=0x7f7be80b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:33.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.370+0000 7f7becc00640 1 -- 172.21.15.27:0/1765379557 shutdown_connections 2024-01-15T18:17:33.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.370+0000 7f7becc00640 1 -- 172.21.15.27:0/1765379557 wait complete. 2024-01-15T18:17:33.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.370+0000 7f7becc00640 1 Processor -- start 2024-01-15T18:17:33.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.370+0000 7f7becc00640 1 -- start start 2024-01-15T18:17:33.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.374+0000 7f7becc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7be806c200 0x7f7be815c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:33.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.374+0000 7f7becc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7be80cad40 0x7f7be8156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:33.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.374+0000 7f7becc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7be8143da0 0x7f7be8136c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:33.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.374+0000 7f7becc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be815bcc0 con 0x7f7be806c200 2024-01-15T18:17:33.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.374+0000 7f7becc00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be8158ad0 con 0x7f7be80cad40 2024-01-15T18:17:33.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.374+0000 7f7becc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7be815c620 con 0x7f7be8143da0 2024-01-15T18:17:33.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.374+0000 7f7be5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7be806c200 0x7f7be815c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:33.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.374+0000 7f7be5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7be806c200 0x7f7be815c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55522/0 (socket says 172.21.15.27:55522) 2024-01-15T18:17:33.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.374+0000 7f7be5e00640 1 -- 172.21.15.27:0/3070639833 learned_addr learned my addr 172.21.15.27:0/3070639833 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:33.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.374+0000 7f7be5400640 1 --2- 172.21.15.27:0/3070639833 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7be80cad40 0x7f7be8156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:33.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.374+0000 7f7be6800640 1 --2- 172.21.15.27:0/3070639833 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7be8143da0 0x7f7be8136c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:33.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.374+0000 7f7be5e00640 1 -- 172.21.15.27:0/3070639833 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7be8143da0 msgr2=0x7f7be8136c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:33.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.374+0000 7f7be5e00640 1 --2- 172.21.15.27:0/3070639833 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7be8143da0 0x7f7be8136c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:33.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.374+0000 7f7be5e00640 1 -- 172.21.15.27:0/3070639833 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7be80cad40 msgr2=0x7f7be8156980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:33.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.374+0000 7f7be5e00640 1 --2- 172.21.15.27:0/3070639833 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7be80cad40 0x7f7be8156980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:33.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.374+0000 7f7be5e00640 1 -- 172.21.15.27:0/3070639833 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd00094d0 con 0x7f7be806c200 2024-01-15T18:17:33.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.374+0000 7f7be5e00640 1 --2- 172.21.15.27:0/3070639833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7be806c200 0x7f7be815c0e0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f7bd800c820 tx=0x7f7bd800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.374+0000 7f7bde000640 1 -- 172.21.15.27:0/3070639833 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8007a30 con 0x7f7be806c200 2024-01-15T18:17:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.374+0000 7f7bde000640 1 -- 172.21.15.27:0/3070639833 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd8007b90 con 0x7f7be806c200 2024-01-15T18:17:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.374+0000 7f7bde000640 1 -- 172.21.15.27:0/3070639833 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd800ce10 con 0x7f7be806c200 2024-01-15T18:17:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.374+0000 7f7becc00640 1 -- 172.21.15.27:0/3070639833 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7be813d270 con 0x7f7be806c200 2024-01-15T18:17:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.374+0000 7f7becc00640 1 -- 172.21.15.27:0/3070639833 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7be815b600 con 0x7f7be806c200 2024-01-15T18:17:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.374+0000 7f7becc00640 1 -- 172.21.15.27:0/3070639833 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7be804e850 con 0x7f7be806c200 2024-01-15T18:17:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.374+0000 7f7bde000640 1 -- 172.21.15.27:0/3070639833 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f7bd80113f0 con 0x7f7be806c200 2024-01-15T18:17:33.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.378+0000 7f7bde000640 1 --2- 172.21.15.27:0/3070639833 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7bc0067700 0x7f7bc0069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:33.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.378+0000 7f7bde000640 1 -- 172.21.15.27:0/3070639833 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f7bd80831b0 con 0x7f7be806c200 2024-01-15T18:17:33.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.378+0000 7f7be5400640 1 --2- 172.21.15.27:0/3070639833 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7bc0067700 0x7f7bc0069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:33.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.378+0000 7f7be5400640 1 --2- 172.21.15.27:0/3070639833 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7bc0067700 0x7f7bc0069bc0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f7bd00ae000 tx=0x7f7bd00b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:33.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.382+0000 7f7bde000640 1 -- 172.21.15.27:0/3070639833 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7bd80b7280 con 0x7f7be806c200 2024-01-15T18:17:33.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.558+0000 7f7becc00640 1 -- 172.21.15.27:0/3070639833 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f7be8167b90 con 0x7f7bc0067700 2024-01-15T18:17:33.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.562+0000 7f7bde000640 1 -- 172.21.15.27:0/3070639833 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f7bd00aa940 con 0x7f7bc0067700 2024-01-15T18:17:33.560 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:17:33.560 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:17:33.560 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-15T18:17:33.560 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:17:33.561 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:17:33.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.562+0000 7f7becc00640 1 -- 172.21.15.27:0/3070639833 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7bc0067700 msgr2=0x7f7bc0069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:33.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.562+0000 7f7becc00640 1 --2- 172.21.15.27:0/3070639833 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7bc0067700 0x7f7bc0069bc0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f7bd00ae000 tx=0x7f7bd00b5040 comp rx=0 tx=0).stop 2024-01-15T18:17:33.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.562+0000 7f7becc00640 1 -- 172.21.15.27:0/3070639833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7be806c200 msgr2=0x7f7be815c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:33.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.562+0000 7f7becc00640 1 --2- 172.21.15.27:0/3070639833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7be806c200 0x7f7be815c0e0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f7bd800c820 tx=0x7f7bd800cce0 comp rx=0 tx=0).stop 2024-01-15T18:17:33.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.562+0000 7f7becc00640 1 -- 172.21.15.27:0/3070639833 shutdown_connections 2024-01-15T18:17:33.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.562+0000 7f7becc00640 1 --2- 172.21.15.27:0/3070639833 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7bc0067700 0x7f7bc0069bc0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:33.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.562+0000 7f7becc00640 1 --2- 172.21.15.27:0/3070639833 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7be8143da0 0x7f7be8136c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:33.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.562+0000 7f7becc00640 1 --2- 172.21.15.27:0/3070639833 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7be80cad40 0x7f7be8156980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:33.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.562+0000 7f7becc00640 1 --2- 172.21.15.27:0/3070639833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7be806c200 0x7f7be815c0e0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:33.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.562+0000 7f7becc00640 1 -- 172.21.15.27:0/3070639833 >> 172.21.15.27:0/3070639833 conn(0x7f7be80b0850 msgr2=0x7f7be8137f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:33.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.562+0000 7f7becc00640 1 -- 172.21.15.27:0/3070639833 shutdown_connections 2024-01-15T18:17:33.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.562+0000 7f7becc00640 1 -- 172.21.15.27:0/3070639833 wait complete. 2024-01-15T18:17:33.583 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:17:33.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.730+0000 7fecaa000640 1 Processor -- start 2024-01-15T18:17:33.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.730+0000 7fecaa000640 1 -- start start 2024-01-15T18:17:33.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.730+0000 7fecaa000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7feca40ce760 0x7feca40b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:33.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.730+0000 7fecaa000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca40712b0 con 0x7feca40b1db0 2024-01-15T18:17:33.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.730+0000 7fecaa000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca40ccfb0 con 0x7feca406c430 2024-01-15T18:17:33.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.730+0000 7fecaa000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feca40cd290 con 0x7feca40ce760 2024-01-15T18:17:33.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.730+0000 7feca3e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7feca40ce760 0x7feca40b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:33.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.730+0000 7feca3e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7feca40ce760 0x7feca40b2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49518/0 (socket says 172.21.15.27:49518) 2024-01-15T18:17:33.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.730+0000 7feca3e00640 1 -- 172.21.15.27:0/3046415568 learned_addr learned my addr 172.21.15.27:0/3046415568 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:33.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.730+0000 7feca2a00640 1 --1- 172.21.15.27:0/3046415568 >> v1:172.21.15.27:6789/0 conn(0x7feca40b1db0 0x7feca40ce050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35428/0 (socket says 172.21.15.27:35428) 2024-01-15T18:17:33.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.730+0000 7feca3e00640 1 -- 172.21.15.27:0/3046415568 >> v1:172.21.15.93:6789/0 conn(0x7feca406c430 legacy=0x7feca4070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:33.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.730+0000 7feca3e00640 1 -- 172.21.15.27:0/3046415568 >> v1:172.21.15.27:6789/0 conn(0x7feca40b1db0 legacy=0x7feca40ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:33.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.730+0000 7feca2000640 1 -- 172.21.15.27:0/3046415568 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 819614701 0 0) 0x7feca40712b0 con 0x7feca40b1db0 2024-01-15T18:17:33.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.730+0000 7feca3e00640 1 -- 172.21.15.27:0/3046415568 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca40cd570 con 0x7feca40ce760 2024-01-15T18:17:33.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.734+0000 7feca3e00640 1 --2- 172.21.15.27:0/3046415568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca40ce760 0x7feca40b2240 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fec9c00d180 tx=0x7fec9c01eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4008cf8104f11a3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:33.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.734+0000 7feca2000640 1 -- 172.21.15.27:0/3046415568 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c00b2d0 con 0x7feca40ce760 2024-01-15T18:17:33.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.734+0000 7feca2000640 1 -- 172.21.15.27:0/3046415568 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9c00b430 con 0x7feca40ce760 2024-01-15T18:17:33.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.734+0000 7feca2000640 1 -- 172.21.15.27:0/3046415568 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c0a7ba0 con 0x7feca40ce760 2024-01-15T18:17:33.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.734+0000 7fecaa000640 1 -- 172.21.15.27:0/3046415568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca40ce760 msgr2=0x7feca40b2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:33.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.734+0000 7fecaa000640 1 --2- 172.21.15.27:0/3046415568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca40ce760 0x7feca40b2240 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fec9c00d180 tx=0x7fec9c01eb30 comp rx=0 tx=0).stop 2024-01-15T18:17:33.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.734+0000 7fecaa000640 1 -- 172.21.15.27:0/3046415568 shutdown_connections 2024-01-15T18:17:33.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.734+0000 7fecaa000640 1 --2- 172.21.15.27:0/3046415568 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca40ce760 0x7feca40b2240 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:33.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.734+0000 7fecaa000640 1 -- 172.21.15.27:0/3046415568 >> 172.21.15.27:0/3046415568 conn(0x7feca40cb700 msgr2=0x7feca40cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:33.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.734+0000 7fecaa000640 1 -- 172.21.15.27:0/3046415568 shutdown_connections 2024-01-15T18:17:33.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.734+0000 7fecaa000640 1 -- 172.21.15.27:0/3046415568 wait complete. 2024-01-15T18:17:33.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7fecaa000640 1 Processor -- start 2024-01-15T18:17:33.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7fecaa000640 1 -- start start 2024-01-15T18:17:33.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7fecaa000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feca406c430 0x7feca4155a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7fecaa000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feca40b1db0 0x7feca414fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7fecaa000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca40ce760 0x7feca40b2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7fecaa000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca40aded0 con 0x7feca40ce760 2024-01-15T18:17:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7fecaa000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca4155f80 con 0x7feca40b1db0 2024-01-15T18:17:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7fecaa000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feca40ad2c0 con 0x7feca406c430 2024-01-15T18:17:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7feca3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca40ce760 0x7feca40b2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7feca3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca40ce760 0x7feca40b2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49524/0 (socket says 172.21.15.27:49524) 2024-01-15T18:17:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7feca3e00640 1 -- 172.21.15.27:0/3814325818 learned_addr learned my addr 172.21.15.27:0/3814325818 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7feca2a00640 1 --2- 172.21.15.27:0/3814325818 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feca40b1db0 0x7feca414fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7feca3400640 1 --2- 172.21.15.27:0/3814325818 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feca406c430 0x7feca4155a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7feca3e00640 1 -- 172.21.15.27:0/3814325818 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feca406c430 msgr2=0x7feca4155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:33.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7feca3e00640 1 --2- 172.21.15.27:0/3814325818 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feca406c430 0x7feca4155a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7feca3e00640 1 -- 172.21.15.27:0/3814325818 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feca40b1db0 msgr2=0x7feca414fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7feca3e00640 1 --2- 172.21.15.27:0/3814325818 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feca40b1db0 0x7feca414fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7feca3e00640 1 -- 172.21.15.27:0/3814325818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9c00bd40 con 0x7feca40ce760 2024-01-15T18:17:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7feca2a00640 1 --2- 172.21.15.27:0/3814325818 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feca40b1db0 0x7feca414fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:17:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7feca3400640 1 --2- 172.21.15.27:0/3814325818 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feca406c430 0x7feca4155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:17:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7feca3e00640 1 --2- 172.21.15.27:0/3814325818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca40ce760 0x7feca40b2f40 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fec9c01b740 tx=0x7fec9c01b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7fec9b400640 1 -- 172.21.15.27:0/3814325818 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c009800 con 0x7feca40ce760 2024-01-15T18:17:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7fec9b400640 1 -- 172.21.15.27:0/3814325818 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9c009960 con 0x7feca40ce760 2024-01-15T18:17:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7fec9b400640 1 -- 172.21.15.27:0/3814325818 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9c009cf0 con 0x7feca40ce760 2024-01-15T18:17:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7fecaa000640 1 -- 172.21.15.27:0/3814325818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca40ad5a0 con 0x7feca40ce760 2024-01-15T18:17:33.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.738+0000 7fecaa000640 1 -- 172.21.15.27:0/3814325818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feca40adb60 con 0x7feca40ce760 2024-01-15T18:17:33.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.742+0000 7fecaa000640 1 -- 172.21.15.27:0/3814325818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca40ae040 con 0x7feca40ce760 2024-01-15T18:17:33.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.742+0000 7fec9b400640 1 -- 172.21.15.27:0/3814325818 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7fec9c0c3460 con 0x7feca40ce760 2024-01-15T18:17:33.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.742+0000 7fec9b400640 1 --2- 172.21.15.27:0/3814325818 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fec7c067700 0x7fec7c069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:33.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.742+0000 7fec9b400640 1 -- 172.21.15.27:0/3814325818 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fec9c11e6c0 con 0x7feca40ce760 2024-01-15T18:17:33.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.742+0000 7feca3400640 1 --2- 172.21.15.27:0/3814325818 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fec7c067700 0x7fec7c069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:33.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.742+0000 7feca3400640 1 --2- 172.21.15.27:0/3814325818 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fec7c067700 0x7fec7c069bc0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fec94000f70 tx=0x7fec94009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:33.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.746+0000 7fec9b400640 1 -- 172.21.15.27:0/3814325818 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fec9c152280 con 0x7feca40ce760 2024-01-15T18:17:33.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.906+0000 7fecaa000640 1 -- 172.21.15.27:0/3814325818 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7feca404e850 con 0x7feca40ce760 2024-01-15T18:17:33.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.906+0000 7fec9b400640 1 -- 172.21.15.27:0/3814325818 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7fec9c0ed350 con 0x7feca40ce760 2024-01-15T18:17:33.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.906+0000 7fecaa000640 1 -- 172.21.15.27:0/3814325818 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fec7c067700 msgr2=0x7fec7c069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:33.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.906+0000 7fecaa000640 1 --2- 172.21.15.27:0/3814325818 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fec7c067700 0x7fec7c069bc0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fec94000f70 tx=0x7fec94009040 comp rx=0 tx=0).stop 2024-01-15T18:17:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.906+0000 7fecaa000640 1 -- 172.21.15.27:0/3814325818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca40ce760 msgr2=0x7feca40b2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.906+0000 7fecaa000640 1 --2- 172.21.15.27:0/3814325818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca40ce760 0x7feca40b2f40 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fec9c01b740 tx=0x7fec9c01b8d0 comp rx=0 tx=0).stop 2024-01-15T18:17:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.906+0000 7fecaa000640 1 -- 172.21.15.27:0/3814325818 shutdown_connections 2024-01-15T18:17:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.906+0000 7fecaa000640 1 --2- 172.21.15.27:0/3814325818 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fec7c067700 0x7fec7c069bc0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.906+0000 7fecaa000640 1 --2- 172.21.15.27:0/3814325818 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feca40ce760 0x7feca40b2f40 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.906+0000 7fecaa000640 1 --2- 172.21.15.27:0/3814325818 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feca40b1db0 0x7feca414fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.906+0000 7fecaa000640 1 --2- 172.21.15.27:0/3814325818 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feca406c430 0x7feca4155a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.906+0000 7fecaa000640 1 -- 172.21.15.27:0/3814325818 >> 172.21.15.27:0/3814325818 conn(0x7feca40cb700 msgr2=0x7feca4156860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.906+0000 7fecaa000640 1 -- 172.21.15.27:0/3814325818 shutdown_connections 2024-01-15T18:17:33.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:33.906+0000 7fecaa000640 1 -- 172.21.15.27:0/3814325818 wait complete. 2024-01-15T18:17:33.928 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 10 2024-01-15T18:17:34.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.074+0000 7fe9f7e00640 1 Processor -- start 2024-01-15T18:17:34.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.074+0000 7fe9f7e00640 1 -- start start 2024-01-15T18:17:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.074+0000 7fe9f7e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9f006c430 0x7fe9f0070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.074+0000 7fe9f7e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fe9f00b1db0 0x7fe9f00b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.074+0000 7fe9f7e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9f00ce050 0x7fe9f00ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.074+0000 7fe9f7e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9f00cea20 con 0x7fe9f006c430 2024-01-15T18:17:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.074+0000 7fe9f7e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9f00ccfb0 con 0x7fe9f00b1db0 2024-01-15T18:17:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.074+0000 7fe9f7e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f00cd290 con 0x7fe9f00ce050 2024-01-15T18:17:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.074+0000 7fe9f5400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9f006c430 0x7fe9f0070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.074+0000 7fe9f5400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9f006c430 0x7fe9f0070ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49526/0 (socket says 172.21.15.27:49526) 2024-01-15T18:17:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.074+0000 7fe9f5400640 1 -- 172.21.15.27:0/4044416391 learned_addr learned my addr 172.21.15.27:0/4044416391 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.074+0000 7fe9f5e00640 1 --2- 172.21.15.27:0/4044416391 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9f00ce050 0x7fe9f00ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:34.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.074+0000 7fe9f4a00640 1 --2- 172.21.15.27:0/4044416391 >> v2:172.21.15.93:3300/0 conn(0x7fe9f00b1db0 0x7fe9f00b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:34.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.074+0000 7fe9f5400640 1 -- 172.21.15.27:0/4044416391 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9f00ce050 msgr2=0x7fe9f00ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:34.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.074+0000 7fe9f5400640 1 --2- 172.21.15.27:0/4044416391 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9f00ce050 0x7fe9f00ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.074+0000 7fe9f5400640 1 -- 172.21.15.27:0/4044416391 >> v2:172.21.15.93:3300/0 conn(0x7fe9f00b1db0 msgr2=0x7fe9f00b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:34.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.074+0000 7fe9f5400640 1 --2- 172.21.15.27:0/4044416391 >> v2:172.21.15.93:3300/0 conn(0x7fe9f00b1db0 0x7fe9f00b2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.074+0000 7fe9f5400640 1 -- 172.21.15.27:0/4044416391 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f00cd570 con 0x7fe9f006c430 2024-01-15T18:17:34.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.078+0000 7fe9f5400640 1 --2- 172.21.15.27:0/4044416391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f006c430 0x7fe9f0070ba0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0009870 tx=0x7fe9e001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d91ef20b98d6286 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:34.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.078+0000 7fe9efe00640 1 -- 172.21.15.27:0/4044416391 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e00ad680 con 0x7fe9f006c430 2024-01-15T18:17:34.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.078+0000 7fe9efe00640 1 -- 172.21.15.27:0/4044416391 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e00ad7e0 con 0x7fe9f006c430 2024-01-15T18:17:34.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.078+0000 7fe9efe00640 1 -- 172.21.15.27:0/4044416391 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e00adb70 con 0x7fe9f006c430 2024-01-15T18:17:34.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.078+0000 7fe9f7e00640 1 -- 172.21.15.27:0/4044416391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f006c430 msgr2=0x7fe9f0070ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:34.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.078+0000 7fe9f7e00640 1 --2- 172.21.15.27:0/4044416391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f006c430 0x7fe9f0070ba0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0009870 tx=0x7fe9e001c440 comp rx=0 tx=0).stop 2024-01-15T18:17:34.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.078+0000 7fe9f7e00640 1 -- 172.21.15.27:0/4044416391 shutdown_connections 2024-01-15T18:17:34.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.078+0000 7fe9f7e00640 1 --2- 172.21.15.27:0/4044416391 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9f00ce050 0x7fe9f00ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.078+0000 7fe9f7e00640 1 --2- 172.21.15.27:0/4044416391 >> v2:172.21.15.93:3300/0 conn(0x7fe9f00b1db0 0x7fe9f00b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.078+0000 7fe9f7e00640 1 --2- 172.21.15.27:0/4044416391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f006c430 0x7fe9f0070ba0 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.078+0000 7fe9f7e00640 1 -- 172.21.15.27:0/4044416391 >> 172.21.15.27:0/4044416391 conn(0x7fe9f00cb700 msgr2=0x7fe9f00cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:34.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.078+0000 7fe9f7e00640 1 -- 172.21.15.27:0/4044416391 shutdown_connections 2024-01-15T18:17:34.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.078+0000 7fe9f7e00640 1 -- 172.21.15.27:0/4044416391 wait complete. 2024-01-15T18:17:34.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.082+0000 7fe9f7e00640 1 Processor -- start 2024-01-15T18:17:34.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.082+0000 7fe9f7e00640 1 -- start start 2024-01-15T18:17:34.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.082+0000 7fe9f7e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9f006c430 0x7fe9f014e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:34.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.082+0000 7fe9f7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f00b1db0 0x7fe9f014c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:34.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.082+0000 7fe9f7e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe9f00ce050 0x7fe9f015b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:34.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.082+0000 7fe9f7e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f015ca80 con 0x7fe9f00b1db0 2024-01-15T18:17:34.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.082+0000 7fe9f7e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f014dbc0 con 0x7fe9f00ce050 2024-01-15T18:17:34.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.082+0000 7fe9f7e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f014dea0 con 0x7fe9f006c430 2024-01-15T18:17:34.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.082+0000 7fe9f4a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f00b1db0 0x7fe9f014c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:34.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.082+0000 7fe9f4a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f00b1db0 0x7fe9f014c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49528/0 (socket says 172.21.15.27:49528) 2024-01-15T18:17:34.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.082+0000 7fe9f4a00640 1 -- 172.21.15.27:0/2459222289 learned_addr learned my addr 172.21.15.27:0/2459222289 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:34.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.082+0000 7fe9f5400640 1 --2- 172.21.15.27:0/2459222289 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9f006c430 0x7fe9f014e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:34.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.082+0000 7fe9f5e00640 1 --2- 172.21.15.27:0/2459222289 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe9f00ce050 0x7fe9f015b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:34.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.082+0000 7fe9f4a00640 1 -- 172.21.15.27:0/2459222289 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9f006c430 msgr2=0x7fe9f014e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:34.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.082+0000 7fe9f4a00640 1 --2- 172.21.15.27:0/2459222289 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9f006c430 0x7fe9f014e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.082+0000 7fe9f4a00640 1 -- 172.21.15.27:0/2459222289 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe9f00ce050 msgr2=0x7fe9f015b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:34.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.082+0000 7fe9f4a00640 1 --2- 172.21.15.27:0/2459222289 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe9f00ce050 0x7fe9f015b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.082+0000 7fe9f4a00640 1 -- 172.21.15.27:0/2459222289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e00094d0 con 0x7fe9f00b1db0 2024-01-15T18:17:34.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.082+0000 7fe9f4a00640 1 --2- 172.21.15.27:0/2459222289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f00b1db0 0x7fe9f014c570 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fe9d800d800 tx=0x7fe9d800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:34.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.082+0000 7fe9ed600640 1 -- 172.21.15.27:0/2459222289 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d800b6c0 con 0x7fe9f00b1db0 2024-01-15T18:17:34.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.082+0000 7fe9ed600640 1 -- 172.21.15.27:0/2459222289 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d800b820 con 0x7fe9f00b1db0 2024-01-15T18:17:34.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.082+0000 7fe9ed600640 1 -- 172.21.15.27:0/2459222289 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d800bb90 con 0x7fe9f00b1db0 2024-01-15T18:17:34.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.082+0000 7fe9f7e00640 1 -- 172.21.15.27:0/2459222289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9f014e180 con 0x7fe9f00b1db0 2024-01-15T18:17:34.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.082+0000 7fe9f7e00640 1 -- 172.21.15.27:0/2459222289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9f015c2a0 con 0x7fe9f00b1db0 2024-01-15T18:17:34.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.086+0000 7fe9f7e00640 1 -- 172.21.15.27:0/2459222289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9f015baa0 con 0x7fe9f00b1db0 2024-01-15T18:17:34.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.086+0000 7fe9ed600640 1 -- 172.21.15.27:0/2459222289 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7fe9d800bcf0 con 0x7fe9f00b1db0 2024-01-15T18:17:34.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.086+0000 7fe9ed600640 1 --2- 172.21.15.27:0/2459222289 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe9c8067730 0x7fe9c8069bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:34.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.086+0000 7fe9ed600640 1 -- 172.21.15.27:0/2459222289 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fe9d8083530 con 0x7fe9f00b1db0 2024-01-15T18:17:34.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.086+0000 7fe9f5400640 1 --2- 172.21.15.27:0/2459222289 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe9c8067730 0x7fe9c8069bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:34.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.086+0000 7fe9f5400640 1 --2- 172.21.15.27:0/2459222289 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe9c8067730 0x7fe9c8069bf0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fe9e00af000 tx=0x7fe9e00a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:34.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.090+0000 7fe9ed600640 1 -- 172.21.15.27:0/2459222289 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe9d8051fb0 con 0x7fe9f00b1db0 2024-01-15T18:17:34.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.250+0000 7fe9f7e00640 1 -- 172.21.15.27:0/2459222289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7fe9f004e850 con 0x7fe9f00b1db0 2024-01-15T18:17:34.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.254+0000 7fe9ed600640 1 -- 172.21.15.27:0/2459222289 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v35)=0 v35) v1 ==== 131+0+0 (secure 0 0 0) 0x7fe9d80555d0 con 0x7fe9f00b1db0 2024-01-15T18:17:34.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.258+0000 7fe9f7e00640 1 -- 172.21.15.27:0/2459222289 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe9c8067730 msgr2=0x7fe9c8069bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:34.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.258+0000 7fe9f7e00640 1 --2- 172.21.15.27:0/2459222289 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe9c8067730 0x7fe9c8069bf0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fe9e00af000 tx=0x7fe9e00a7000 comp rx=0 tx=0).stop 2024-01-15T18:17:34.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.258+0000 7fe9f7e00640 1 -- 172.21.15.27:0/2459222289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f00b1db0 msgr2=0x7fe9f014c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:34.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.258+0000 7fe9f7e00640 1 --2- 172.21.15.27:0/2459222289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f00b1db0 0x7fe9f014c570 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fe9d800d800 tx=0x7fe9d800dcc0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.258+0000 7fe9f7e00640 1 -- 172.21.15.27:0/2459222289 shutdown_connections 2024-01-15T18:17:34.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.258+0000 7fe9f7e00640 1 --2- 172.21.15.27:0/2459222289 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe9c8067730 0x7fe9c8069bf0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.258+0000 7fe9f7e00640 1 --2- 172.21.15.27:0/2459222289 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe9f00ce050 0x7fe9f015b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.258+0000 7fe9f7e00640 1 --2- 172.21.15.27:0/2459222289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9f00b1db0 0x7fe9f014c570 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.258+0000 7fe9f7e00640 1 --2- 172.21.15.27:0/2459222289 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9f006c430 0x7fe9f014e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.258+0000 7fe9f7e00640 1 -- 172.21.15.27:0/2459222289 >> 172.21.15.27:0/2459222289 conn(0x7fe9f00cb700 msgr2=0x7fe9f00afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:34.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.258+0000 7fe9f7e00640 1 -- 172.21.15.27:0/2459222289 shutdown_connections 2024-01-15T18:17:34.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.258+0000 7fe9f7e00640 1 -- 172.21.15.27:0/2459222289 wait complete. 2024-01-15T18:17:34.277 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 120 2024-01-15T18:17:34.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.422+0000 7f76e8a00640 1 Processor -- start 2024-01-15T18:17:34.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.422+0000 7f76e8a00640 1 -- start start 2024-01-15T18:17:34.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.422+0000 7f76e8a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f76e015d3b0 0x7f76e00cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:34.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.422+0000 7f76e8a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f76e006b600 0x7f76e006bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:34.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.422+0000 7f76e8a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f76e006c220 0x7f76e0160f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:34.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.422+0000 7f76e8a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76e006c6b0 con 0x7f76e015d3b0 2024-01-15T18:17:34.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.422+0000 7f76e8a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76e00b04b0 con 0x7f76e006c220 2024-01-15T18:17:34.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.422+0000 7f76e8a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76e00d0160 con 0x7f76e006b600 2024-01-15T18:17:34.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.422+0000 7f76e6000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f76e015d3b0 0x7f76e00cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:34.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.422+0000 7f76e6000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f76e015d3b0 0x7f76e00cfc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49536/0 (socket says 172.21.15.27:49536) 2024-01-15T18:17:34.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.422+0000 7f76e6000640 1 -- 172.21.15.27:0/1471862158 learned_addr learned my addr 172.21.15.27:0/1471862158 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:34.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.422+0000 7f76e6a00640 1 --2- 172.21.15.27:0/1471862158 >> v2:172.21.15.93:3300/0 conn(0x7f76e006c220 0x7f76e0160f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:34.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.422+0000 7f76e5600640 1 --2- 172.21.15.27:0/1471862158 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f76e006b600 0x7f76e006bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:34.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.422+0000 7f76e6000640 1 -- 172.21.15.27:0/1471862158 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f76e006b600 msgr2=0x7f76e006bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:34.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.422+0000 7f76e6000640 1 --2- 172.21.15.27:0/1471862158 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f76e006b600 0x7f76e006bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.422+0000 7f76e6000640 1 -- 172.21.15.27:0/1471862158 >> v2:172.21.15.93:3300/0 conn(0x7f76e006c220 msgr2=0x7f76e0160f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:34.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.422+0000 7f76e6000640 1 --2- 172.21.15.27:0/1471862158 >> v2:172.21.15.93:3300/0 conn(0x7f76e006c220 0x7f76e0160f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.422+0000 7f76e6000640 1 -- 172.21.15.27:0/1471862158 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e00b0720 con 0x7f76e015d3b0 2024-01-15T18:17:34.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.426+0000 7f76e6000640 1 --2- 172.21.15.27:0/1471862158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76e015d3b0 0x7f76e00cfc20 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f76d0009870 tx=0x7f76d001c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a1b890f031aa48b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:34.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.426+0000 7f76e4c00640 1 -- 172.21.15.27:0/1471862158 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d00ab600 con 0x7f76e015d3b0 2024-01-15T18:17:34.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.426+0000 7f76e4c00640 1 -- 172.21.15.27:0/1471862158 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76d00a7050 con 0x7f76e015d3b0 2024-01-15T18:17:34.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.426+0000 7f76e4c00640 1 -- 172.21.15.27:0/1471862158 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d00ab980 con 0x7f76e015d3b0 2024-01-15T18:17:34.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.426+0000 7f76e8a00640 1 -- 172.21.15.27:0/1471862158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76e015d3b0 msgr2=0x7f76e00cfc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:34.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.426+0000 7f76e8a00640 1 --2- 172.21.15.27:0/1471862158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76e015d3b0 0x7f76e00cfc20 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f76d0009870 tx=0x7f76d001c210 comp rx=0 tx=0).stop 2024-01-15T18:17:34.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.426+0000 7f76e8a00640 1 -- 172.21.15.27:0/1471862158 shutdown_connections 2024-01-15T18:17:34.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.426+0000 7f76e8a00640 1 --2- 172.21.15.27:0/1471862158 >> v2:172.21.15.93:3300/0 conn(0x7f76e006c220 0x7f76e0160f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.426+0000 7f76e8a00640 1 --2- 172.21.15.27:0/1471862158 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f76e006b600 0x7f76e006bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.426+0000 7f76e8a00640 1 --2- 172.21.15.27:0/1471862158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76e015d3b0 0x7f76e00cfc20 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.426+0000 7f76e8a00640 1 -- 172.21.15.27:0/1471862158 >> 172.21.15.27:0/1471862158 conn(0x7f76e00d0ed0 msgr2=0x7f76e00d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:34.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.426+0000 7f76e8a00640 1 -- 172.21.15.27:0/1471862158 shutdown_connections 2024-01-15T18:17:34.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.426+0000 7f76e8a00640 1 -- 172.21.15.27:0/1471862158 wait complete. 2024-01-15T18:17:34.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.430+0000 7f76e8a00640 1 Processor -- start 2024-01-15T18:17:34.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.430+0000 7f76e8a00640 1 -- start start 2024-01-15T18:17:34.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.430+0000 7f76e8a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76e006b600 0x7f76e0154f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:34.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.430+0000 7f76e8a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f76e006c220 0x7f76e009d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:34.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.430+0000 7f76e8a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76e015d3b0 0x7f76e00c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:34.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.430+0000 7f76e8a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e009c6c0 con 0x7f76e006b600 2024-01-15T18:17:34.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.430+0000 7f76e8a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e009c960 con 0x7f76e015d3b0 2024-01-15T18:17:34.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.430+0000 7f76e8a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76e009cc40 con 0x7f76e006c220 2024-01-15T18:17:34.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.430+0000 7f76e6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76e006b600 0x7f76e0154f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:34.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.430+0000 7f76e6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76e006b600 0x7f76e0154f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49546/0 (socket says 172.21.15.27:49546) 2024-01-15T18:17:34.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.430+0000 7f76e6000640 1 -- 172.21.15.27:0/2380511955 learned_addr learned my addr 172.21.15.27:0/2380511955 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:34.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.430+0000 7f76e5600640 1 --2- 172.21.15.27:0/2380511955 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f76e006c220 0x7f76e009d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:34.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.430+0000 7f76e6a00640 1 --2- 172.21.15.27:0/2380511955 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76e015d3b0 0x7f76e00c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:34.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.430+0000 7f76e6000640 1 -- 172.21.15.27:0/2380511955 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f76e006c220 msgr2=0x7f76e009d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:34.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.430+0000 7f76e6000640 1 --2- 172.21.15.27:0/2380511955 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f76e006c220 0x7f76e009d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.430+0000 7f76e6000640 1 -- 172.21.15.27:0/2380511955 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76e015d3b0 msgr2=0x7f76e00c6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:34.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.430+0000 7f76e6000640 1 --2- 172.21.15.27:0/2380511955 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76e015d3b0 0x7f76e00c6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.430+0000 7f76e6000640 1 -- 172.21.15.27:0/2380511955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d00094d0 con 0x7f76e006b600 2024-01-15T18:17:34.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.430+0000 7f76e5600640 1 --2- 172.21.15.27:0/2380511955 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f76e006c220 0x7f76e009d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:17:34.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.430+0000 7f76e6000640 1 --2- 172.21.15.27:0/2380511955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76e006b600 0x7f76e0154f50 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f76d00ae040 tx=0x7f76d0018fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:34.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.430+0000 7f76de000640 1 -- 172.21.15.27:0/2380511955 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d00b4420 con 0x7f76e006b600 2024-01-15T18:17:34.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.430+0000 7f76de000640 1 -- 172.21.15.27:0/2380511955 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76d00b45a0 con 0x7f76e006b600 2024-01-15T18:17:34.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.430+0000 7f76de000640 1 -- 172.21.15.27:0/2380511955 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d00b4950 con 0x7f76e006b600 2024-01-15T18:17:34.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.430+0000 7f76e8a00640 1 -- 172.21.15.27:0/2380511955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76e009cf20 con 0x7f76e006b600 2024-01-15T18:17:34.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.430+0000 7f76e8a00640 1 -- 172.21.15.27:0/2380511955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76e0155490 con 0x7f76e006b600 2024-01-15T18:17:34.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.434+0000 7f76e8a00640 1 -- 172.21.15.27:0/2380511955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76e006d550 con 0x7f76e006b600 2024-01-15T18:17:34.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.434+0000 7f76de000640 1 -- 172.21.15.27:0/2380511955 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f76d00b2070 con 0x7f76e006b600 2024-01-15T18:17:34.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.434+0000 7f76de000640 1 --2- 172.21.15.27:0/2380511955 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f76b8067700 0x7f76b8069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:34.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.434+0000 7f76de000640 1 -- 172.21.15.27:0/2380511955 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f76d011be20 con 0x7f76e006b600 2024-01-15T18:17:34.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.434+0000 7f76e5600640 1 --2- 172.21.15.27:0/2380511955 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f76b8067700 0x7f76b8069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:34.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.434+0000 7f76e5600640 1 --2- 172.21.15.27:0/2380511955 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f76b8067700 0x7f76b8069bc0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f76c8004660 tx=0x7f76c8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:34.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.438+0000 7f76de000640 1 -- 172.21.15.27:0/2380511955 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f76d00eaad0 con 0x7f76e006b600 2024-01-15T18:17:34.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.594+0000 7f76e8a00640 1 -- 172.21.15.27:0/2380511955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f76e004e850 con 0x7f76e006b600 2024-01-15T18:17:34.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.602+0000 7f76de000640 1 -- 172.21.15.27:0/2380511955 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v36)=0 v36) v1 ==== 137+0+0 (secure 0 0 0) 0x7f76d00ba020 con 0x7f76e006b600 2024-01-15T18:17:34.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.602+0000 7f76e8a00640 1 -- 172.21.15.27:0/2380511955 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f76b8067700 msgr2=0x7f76b8069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:34.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.602+0000 7f76e8a00640 1 --2- 172.21.15.27:0/2380511955 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f76b8067700 0x7f76b8069bc0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f76c8004660 tx=0x7f76c8008040 comp rx=0 tx=0).stop 2024-01-15T18:17:34.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.602+0000 7f76e8a00640 1 -- 172.21.15.27:0/2380511955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76e006b600 msgr2=0x7f76e0154f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:34.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.602+0000 7f76e8a00640 1 --2- 172.21.15.27:0/2380511955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76e006b600 0x7f76e0154f50 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f76d00ae040 tx=0x7f76d0018fb0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.606+0000 7f76e8a00640 1 -- 172.21.15.27:0/2380511955 shutdown_connections 2024-01-15T18:17:34.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.606+0000 7f76e8a00640 1 --2- 172.21.15.27:0/2380511955 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f76b8067700 0x7f76b8069bc0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.606+0000 7f76e8a00640 1 --2- 172.21.15.27:0/2380511955 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76e015d3b0 0x7f76e00c6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.606+0000 7f76e8a00640 1 --2- 172.21.15.27:0/2380511955 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f76e006c220 0x7f76e009d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.606+0000 7f76e8a00640 1 --2- 172.21.15.27:0/2380511955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76e006b600 0x7f76e0154f50 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.606+0000 7f76e8a00640 1 -- 172.21.15.27:0/2380511955 >> 172.21.15.27:0/2380511955 conn(0x7f76e00d0ed0 msgr2=0x7f76e006f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:34.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.606+0000 7f76e8a00640 1 -- 172.21.15.27:0/2380511955 shutdown_connections 2024-01-15T18:17:34.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.606+0000 7f76e8a00640 1 -- 172.21.15.27:0/2380511955 wait complete. 2024-01-15T18:17:34.623 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-15T18:17:34.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.770+0000 7fb7d5600640 1 Processor -- start 2024-01-15T18:17:34.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.770+0000 7fb7d5600640 1 -- start start 2024-01-15T18:17:34.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.770+0000 7fb7d5600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb7d00cba60 0x7fb7d00b0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:34.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.770+0000 7fb7d5600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d00cefa0 con 0x7fb7d00d1a40 2024-01-15T18:17:34.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.770+0000 7fb7d5600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d00ce240 con 0x7fb7d00cf450 2024-01-15T18:17:34.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.770+0000 7fb7d5600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7d00ccd70 con 0x7fb7d00cba60 2024-01-15T18:17:34.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.770+0000 7fb7cf400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb7d00cba60 0x7fb7d00b0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:34.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.770+0000 7fb7cf400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb7d00cba60 0x7fb7d00b0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49558/0 (socket says 172.21.15.27:49558) 2024-01-15T18:17:34.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.770+0000 7fb7cf400640 1 -- 172.21.15.27:0/507277164 learned_addr learned my addr 172.21.15.27:0/507277164 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:34.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.770+0000 7fb7ce000640 1 --1- 172.21.15.27:0/507277164 >> v1:172.21.15.27:6789/0 conn(0x7fb7d00d1a40 0x7fb7d00b0650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35438/0 (socket says 172.21.15.27:35438) 2024-01-15T18:17:34.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.770+0000 7fb7cf400640 1 -- 172.21.15.27:0/507277164 >> v1:172.21.15.93:6789/0 conn(0x7fb7d00cf450 legacy=0x7fb7d00cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:34.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.770+0000 7fb7cf400640 1 -- 172.21.15.27:0/507277164 >> v1:172.21.15.27:6789/0 conn(0x7fb7d00d1a40 legacy=0x7fb7d00b0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:34.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.770+0000 7fb7cd600640 1 -- 172.21.15.27:0/507277164 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 650497687 0 0) 0x7fb7d00cefa0 con 0x7fb7d00d1a40 2024-01-15T18:17:34.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.770+0000 7fb7cf400640 1 -- 172.21.15.27:0/507277164 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d00cc1f0 con 0x7fb7d00cba60 2024-01-15T18:17:34.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.774+0000 7fb7cf400640 1 --2- 172.21.15.27:0/507277164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d00cba60 0x7fb7d00b0d60 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8009e50 tx=0x7fb7c801ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16d19ee25ce62e08 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:34.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.774+0000 7fb7cd600640 1 -- 172.21.15.27:0/507277164 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c80a56e0 con 0x7fb7d00cba60 2024-01-15T18:17:34.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.774+0000 7fb7cd600640 1 -- 172.21.15.27:0/507277164 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c80a5840 con 0x7fb7d00cba60 2024-01-15T18:17:34.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.774+0000 7fb7cd600640 1 -- 172.21.15.27:0/507277164 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c80a5bd0 con 0x7fb7d00cba60 2024-01-15T18:17:34.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.774+0000 7fb7d5600640 1 -- 172.21.15.27:0/507277164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d00cba60 msgr2=0x7fb7d00b0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:34.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.774+0000 7fb7d5600640 1 --2- 172.21.15.27:0/507277164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d00cba60 0x7fb7d00b0d60 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8009e50 tx=0x7fb7c801ca10 comp rx=0 tx=0).stop 2024-01-15T18:17:34.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.774+0000 7fb7d5600640 1 -- 172.21.15.27:0/507277164 shutdown_connections 2024-01-15T18:17:34.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.774+0000 7fb7d5600640 1 --2- 172.21.15.27:0/507277164 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d00cba60 0x7fb7d00b0d60 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.774+0000 7fb7d5600640 1 -- 172.21.15.27:0/507277164 >> 172.21.15.27:0/507277164 conn(0x7fb7d00d0e70 msgr2=0x7fb7d00d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:34.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.774+0000 7fb7d5600640 1 -- 172.21.15.27:0/507277164 shutdown_connections 2024-01-15T18:17:34.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.774+0000 7fb7d5600640 1 -- 172.21.15.27:0/507277164 wait complete. 2024-01-15T18:17:34.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.778+0000 7fb7d5600640 1 Processor -- start 2024-01-15T18:17:34.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.778+0000 7fb7d5600640 1 -- start start 2024-01-15T18:17:34.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.778+0000 7fb7d5600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d00cba60 0x7fb7d006f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:34.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.778+0000 7fb7d5600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb7d00cf450 0x7fb7d006f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:34.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.778+0000 7fb7d5600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb7d00d1a40 0x7fb7d0158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:34.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.778+0000 7fb7d5600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d00b3cc0 con 0x7fb7d00cba60 2024-01-15T18:17:34.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.778+0000 7fb7d5600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d00b3f10 con 0x7fb7d00cf450 2024-01-15T18:17:34.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.778+0000 7fb7d5600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d00b41f0 con 0x7fb7d00d1a40 2024-01-15T18:17:34.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.778+0000 7fb7cea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d00cba60 0x7fb7d006f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:34.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.778+0000 7fb7cea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d00cba60 0x7fb7d006f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49562/0 (socket says 172.21.15.27:49562) 2024-01-15T18:17:34.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.778+0000 7fb7cea00640 1 -- 172.21.15.27:0/2412769868 learned_addr learned my addr 172.21.15.27:0/2412769868 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:34.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.778+0000 7fb7ce000640 1 --2- 172.21.15.27:0/2412769868 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb7d00cf450 0x7fb7d006f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:34.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.778+0000 7fb7cf400640 1 --2- 172.21.15.27:0/2412769868 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb7d00d1a40 0x7fb7d0158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:34.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.778+0000 7fb7cea00640 1 -- 172.21.15.27:0/2412769868 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb7d00d1a40 msgr2=0x7fb7d0158aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:34.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.778+0000 7fb7cea00640 1 --2- 172.21.15.27:0/2412769868 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb7d00d1a40 0x7fb7d0158aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.778+0000 7fb7cea00640 1 -- 172.21.15.27:0/2412769868 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb7d00cf450 msgr2=0x7fb7d006f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:34.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.778+0000 7fb7cea00640 1 --2- 172.21.15.27:0/2412769868 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb7d00cf450 0x7fb7d006f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:34.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.778+0000 7fb7cea00640 1 -- 172.21.15.27:0/2412769868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c8009b30 con 0x7fb7d00cba60 2024-01-15T18:17:34.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.778+0000 7fb7cea00640 1 --2- 172.21.15.27:0/2412769868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d00cba60 0x7fb7d006f330 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fb7c000c500 tx=0x7fb7c000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:34.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.778+0000 7fb7c6a00640 1 -- 172.21.15.27:0/2412769868 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c0005220 con 0x7fb7d00cba60 2024-01-15T18:17:34.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.778+0000 7fb7c6a00640 1 -- 172.21.15.27:0/2412769868 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c0005380 con 0x7fb7d00cba60 2024-01-15T18:17:34.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.778+0000 7fb7d5600640 1 -- 172.21.15.27:0/2412769868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d00b44d0 con 0x7fb7d00cba60 2024-01-15T18:17:34.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.778+0000 7fb7c6a00640 1 -- 172.21.15.27:0/2412769868 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c0007270 con 0x7fb7d00cba60 2024-01-15T18:17:34.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.778+0000 7fb7d5600640 1 -- 172.21.15.27:0/2412769868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d0158fe0 con 0x7fb7d00cba60 2024-01-15T18:17:34.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.782+0000 7fb7d5600640 1 -- 172.21.15.27:0/2412769868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d004e850 con 0x7fb7d00cba60 2024-01-15T18:17:34.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.782+0000 7fb7c6a00640 1 -- 172.21.15.27:0/2412769868 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7fb7c00073d0 con 0x7fb7d00cba60 2024-01-15T18:17:34.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.782+0000 7fb7c6a00640 1 --2- 172.21.15.27:0/2412769868 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb7a8067700 0x7fb7a8069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:34.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.782+0000 7fb7c6a00640 1 -- 172.21.15.27:0/2412769868 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fb7c00829e0 con 0x7fb7d00cba60 2024-01-15T18:17:34.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.782+0000 7fb7ce000640 1 --2- 172.21.15.27:0/2412769868 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb7a8067700 0x7fb7a8069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:34.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.786+0000 7fb7ce000640 1 --2- 172.21.15.27:0/2412769868 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb7a8067700 0x7fb7a8069bc0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fb7d006fe80 tx=0x7fb7b8007240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:34.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.786+0000 7fb7c6a00640 1 -- 172.21.15.27:0/2412769868 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb7c00513e0 con 0x7fb7d00cba60 2024-01-15T18:17:34.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:34.966+0000 7fb7d5600640 1 -- 172.21.15.27:0/2412769868 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7d00cbe90 con 0x7fb7a8067700 2024-01-15T18:17:34.973 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:17:34.976+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:17:34.973 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:17:34.976+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:17:34.973 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:17:34.976+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:17:34.973 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:17:34.976+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:17:34.973 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:17:34.976+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:17:35.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.006+0000 7fb7c6a00640 1 -- 172.21.15.27:0/2412769868 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb7b80073d0 con 0x7fb7a8067700 2024-01-15T18:17:35.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.010+0000 7fb7d5600640 1 -- 172.21.15.27:0/2412769868 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb7a8067700 msgr2=0x7fb7a8069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:35.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.010+0000 7fb7d5600640 1 --2- 172.21.15.27:0/2412769868 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb7a8067700 0x7fb7a8069bc0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fb7d006fe80 tx=0x7fb7b8007240 comp rx=0 tx=0).stop 2024-01-15T18:17:35.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.010+0000 7fb7d5600640 1 -- 172.21.15.27:0/2412769868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d00cba60 msgr2=0x7fb7d006f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:35.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.010+0000 7fb7d5600640 1 --2- 172.21.15.27:0/2412769868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d00cba60 0x7fb7d006f330 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fb7c000c500 tx=0x7fb7c000c9c0 comp rx=0 tx=0).stop 2024-01-15T18:17:35.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.010+0000 7fb7d5600640 1 -- 172.21.15.27:0/2412769868 shutdown_connections 2024-01-15T18:17:35.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.010+0000 7fb7d5600640 1 --2- 172.21.15.27:0/2412769868 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb7a8067700 0x7fb7a8069bc0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:35.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.010+0000 7fb7d5600640 1 --2- 172.21.15.27:0/2412769868 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb7d00d1a40 0x7fb7d0158aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:35.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.010+0000 7fb7d5600640 1 --2- 172.21.15.27:0/2412769868 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb7d00cf450 0x7fb7d006f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:35.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.010+0000 7fb7d5600640 1 --2- 172.21.15.27:0/2412769868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7d00cba60 0x7fb7d006f330 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:35.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.010+0000 7fb7d5600640 1 -- 172.21.15.27:0/2412769868 >> 172.21.15.27:0/2412769868 conn(0x7fb7d00d0e70 msgr2=0x7fb7d00b0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:35.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.010+0000 7fb7d5600640 1 -- 172.21.15.27:0/2412769868 shutdown_connections 2024-01-15T18:17:35.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.010+0000 7fb7d5600640 1 -- 172.21.15.27:0/2412769868 wait complete. 2024-01-15T18:17:35.029 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:17:35.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f64200640 1 Processor -- start 2024-01-15T18:17:35.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f64200640 1 -- start start 2024-01-15T18:17:35.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f64200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f8f5c0cd070 0x7f8f5c168060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:35.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f64200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f5c0ad920 con 0x7f8f5c0cf100 2024-01-15T18:17:35.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f64200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f5c0ac880 con 0x7f8f5c1468d0 2024-01-15T18:17:35.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f64200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f5c0cd4c0 con 0x7f8f5c0cd070 2024-01-15T18:17:35.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f60e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8f5c0cf100 0x7f8f5c0cc960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35450/0 (socket says 172.21.15.27:35450) 2024-01-15T18:17:35.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f60e00640 1 -- 172.21.15.27:0/514810687 learned_addr learned my addr 172.21.15.27:0/514810687 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:35.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f62200640 1 --2- 172.21.15.27:0/514810687 >> v2:172.21.15.93:3300/0 conn(0x7f8f5c0cd070 0x7f8f5c168060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:35.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f5be00640 1 -- 172.21.15.27:0/514810687 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943447495 0 0) 0x7f8f44001120 con 0x7f8f5c0cf100 2024-01-15T18:17:35.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f5be00640 1 -- 172.21.15.27:0/514810687 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f3c003530 con 0x7f8f5c0cf100 2024-01-15T18:17:35.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f5be00640 1 -- 172.21.15.27:0/514810687 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1337269168 0 0) 0x7f8f3c003530 con 0x7f8f5c0cf100 2024-01-15T18:17:35.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f5be00640 1 -- 172.21.15.27:0/514810687 >> v2:172.21.15.93:3300/0 conn(0x7f8f5c0cd070 msgr2=0x7f8f5c168060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:35.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f5be00640 1 --2- 172.21.15.27:0/514810687 >> v2:172.21.15.93:3300/0 conn(0x7f8f5c0cd070 0x7f8f5c168060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:35.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f5be00640 1 -- 172.21.15.27:0/514810687 >> v1:172.21.15.93:6789/0 conn(0x7f8f5c1468d0 legacy=0x7f8f5c0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:35.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f5be00640 1 -- 172.21.15.27:0/514810687 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5c0acb60 con 0x7f8f5c0cf100 2024-01-15T18:17:35.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f5be00640 1 -- 172.21.15.27:0/514810687 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f8f44001810 con 0x7f8f5c0cf100 2024-01-15T18:17:35.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f5be00640 1 -- 172.21.15.27:0/514810687 >> v1:172.21.15.27:6789/0 conn(0x7f8f5c0cf100 legacy=0x7f8f5c0cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:35.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f5be00640 1 --2- 172.21.15.27:0/514810687 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f3c0039f0 0x7f8f3c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:35.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f5be00640 1 --2- 172.21.15.27:0/514810687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f3c0047d0 0x7f8f3c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:35.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f5be00640 1 --2- 172.21.15.27:0/514810687 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8f3c005190 0x7f8f3c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:35.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f5be00640 1 -- 172.21.15.27:0/514810687 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c0cd4c0 con 0x7f8f3c0047d0 2024-01-15T18:17:35.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f5be00640 1 -- 172.21.15.27:0/514810687 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3c00c060 con 0x7f8f3c0039f0 2024-01-15T18:17:35.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f5be00640 1 -- 172.21.15.27:0/514810687 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3c00c220 con 0x7f8f3c005190 2024-01-15T18:17:35.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f60e00640 1 --2- 172.21.15.27:0/514810687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f3c0047d0 0x7f8f3c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:35.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f62200640 1 --2- 172.21.15.27:0/514810687 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8f3c005190 0x7f8f3c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.174+0000 7f8f61800640 1 --2- 172.21.15.27:0/514810687 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f3c0039f0 0x7f8f3c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.178+0000 7f8f60e00640 1 -- 172.21.15.27:0/514810687 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8f3c005190 msgr2=0x7f8f3c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.178+0000 7f8f60e00640 1 --2- 172.21.15.27:0/514810687 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8f3c005190 0x7f8f3c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.178+0000 7f8f60e00640 1 -- 172.21.15.27:0/514810687 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f3c0039f0 msgr2=0x7f8f3c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.178+0000 7f8f60e00640 1 --2- 172.21.15.27:0/514810687 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f3c0039f0 0x7f8f3c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.178+0000 7f8f60e00640 1 -- 172.21.15.27:0/514810687 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3c003530 con 0x7f8f3c0047d0 2024-01-15T18:17:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.178+0000 7f8f61800640 1 --2- 172.21.15.27:0/514810687 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f3c0039f0 0x7f8f3c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:17:35.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.178+0000 7f8f62200640 1 -- 172.21.15.27:0/514810687 reap_dead start 2024-01-15T18:17:35.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.178+0000 7f8f60e00640 1 --2- 172.21.15.27:0/514810687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f3c0047d0 0x7f8f3c004c50 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f8f4400c110 tx=0x7f8f4401cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e52f57e0f9ab4743 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:35.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.178+0000 7f8f5be00640 1 -- 172.21.15.27:0/514810687 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f44005ae0 con 0x7f8f3c0047d0 2024-01-15T18:17:35.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.178+0000 7f8f5be00640 1 -- 172.21.15.27:0/514810687 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f44005c40 con 0x7f8f3c0047d0 2024-01-15T18:17:35.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.178+0000 7f8f5be00640 1 -- 172.21.15.27:0/514810687 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f440a5800 con 0x7f8f3c0047d0 2024-01-15T18:17:35.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.182+0000 7f8f64200640 1 -- 172.21.15.27:0/514810687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f3c0047d0 msgr2=0x7f8f3c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:35.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.182+0000 7f8f64200640 1 --2- 172.21.15.27:0/514810687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f3c0047d0 0x7f8f3c004c50 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f8f4400c110 tx=0x7f8f4401cb00 comp rx=0 tx=0).stop 2024-01-15T18:17:35.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.182+0000 7f8f64200640 1 -- 172.21.15.27:0/514810687 shutdown_connections 2024-01-15T18:17:35.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.182+0000 7f8f64200640 1 --2- 172.21.15.27:0/514810687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f3c0047d0 0x7f8f3c004c50 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:35.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.182+0000 7f8f64200640 1 -- 172.21.15.27:0/514810687 >> 172.21.15.27:0/514810687 conn(0x7f8f5c0cb700 msgr2=0x7f8f5c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:35.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.182+0000 7f8f64200640 1 -- 172.21.15.27:0/514810687 shutdown_connections 2024-01-15T18:17:35.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.182+0000 7f8f64200640 1 -- 172.21.15.27:0/514810687 wait complete. 2024-01-15T18:17:35.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.182+0000 7f8f64200640 1 Processor -- start 2024-01-15T18:17:35.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.182+0000 7f8f64200640 1 -- start start 2024-01-15T18:17:35.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.182+0000 7f8f64200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8f3c0047d0 0x7f8f5c15aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:35.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.182+0000 7f8f64200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f5c070bc0 0x7f8f5c15c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:35.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.182+0000 7f8f64200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f5c071040 0x7f8f5c15c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:35.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.182+0000 7f8f64200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c15b000 con 0x7f8f5c070bc0 2024-01-15T18:17:35.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.182+0000 7f8f64200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c070730 con 0x7f8f5c071040 2024-01-15T18:17:35.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.182+0000 7f8f64200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c072150 con 0x7f8f3c0047d0 2024-01-15T18:17:35.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.182+0000 7f8f60e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f5c070bc0 0x7f8f5c15c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:35.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.182+0000 7f8f60e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f5c070bc0 0x7f8f5c15c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49578/0 (socket says 172.21.15.27:49578) 2024-01-15T18:17:35.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.182+0000 7f8f60e00640 1 -- 172.21.15.27:0/3240275293 learned_addr learned my addr 172.21.15.27:0/3240275293 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:35.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.182+0000 7f8f62200640 1 --2- 172.21.15.27:0/3240275293 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f5c071040 0x7f8f5c15c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:35.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.182+0000 7f8f61800640 1 --2- 172.21.15.27:0/3240275293 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8f3c0047d0 0x7f8f5c15aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:35.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.186+0000 7f8f60e00640 1 -- 172.21.15.27:0/3240275293 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8f3c0047d0 msgr2=0x7f8f5c15aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:35.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.186+0000 7f8f60e00640 1 --2- 172.21.15.27:0/3240275293 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8f3c0047d0 0x7f8f5c15aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:35.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.186+0000 7f8f60e00640 1 -- 172.21.15.27:0/3240275293 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f5c071040 msgr2=0x7f8f5c15c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:35.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.186+0000 7f8f60e00640 1 --2- 172.21.15.27:0/3240275293 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f5c071040 0x7f8f5c15c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:35.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.186+0000 7f8f60e00640 1 -- 172.21.15.27:0/3240275293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5c072430 con 0x7f8f5c070bc0 2024-01-15T18:17:35.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.186+0000 7f8f60e00640 1 --2- 172.21.15.27:0/3240275293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f5c070bc0 0x7f8f5c15c350 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f8f4401c570 tx=0x7f8f3c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:35.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.186+0000 7f8f59600640 1 -- 172.21.15.27:0/3240275293 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f440b4070 con 0x7f8f5c070bc0 2024-01-15T18:17:35.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.186+0000 7f8f59600640 1 -- 172.21.15.27:0/3240275293 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f440a5800 con 0x7f8f5c070bc0 2024-01-15T18:17:35.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.186+0000 7f8f59600640 1 -- 172.21.15.27:0/3240275293 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f440a5b90 con 0x7f8f5c070bc0 2024-01-15T18:17:35.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.186+0000 7f8f64200640 1 -- 172.21.15.27:0/3240275293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f5c0700e0 con 0x7f8f5c070bc0 2024-01-15T18:17:35.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.186+0000 7f8f64200640 1 -- 172.21.15.27:0/3240275293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f5c071490 con 0x7f8f5c070bc0 2024-01-15T18:17:35.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.186+0000 7f8f64200640 1 -- 172.21.15.27:0/3240275293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f5c04e850 con 0x7f8f5c070bc0 2024-01-15T18:17:35.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.186+0000 7f8f59600640 1 -- 172.21.15.27:0/3240275293 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f8f440a5cf0 con 0x7f8f5c070bc0 2024-01-15T18:17:35.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.186+0000 7f8f59600640 1 --2- 172.21.15.27:0/3240275293 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8f34067700 0x7f8f34069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:35.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.186+0000 7f8f59600640 1 -- 172.21.15.27:0/3240275293 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f8f4411c810 con 0x7f8f5c070bc0 2024-01-15T18:17:35.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.186+0000 7f8f61800640 1 --2- 172.21.15.27:0/3240275293 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8f34067700 0x7f8f34069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:35.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.190+0000 7f8f61800640 1 --2- 172.21.15.27:0/3240275293 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8f34067700 0x7f8f34069bc0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c069290 tx=0x7f8f4c00aa20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:35.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.190+0000 7f8f59600640 1 -- 172.21.15.27:0/3240275293 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8f440eb550 con 0x7f8f5c070bc0 2024-01-15T18:17:35.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.378+0000 7f8f64200640 1 -- 172.21.15.27:0/3240275293 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f5c0ad400 con 0x7f8f34067700 2024-01-15T18:17:35.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.382+0000 7f8f59600640 1 -- 172.21.15.27:0/3240275293 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f8f4c00d040 con 0x7f8f34067700 2024-01-15T18:17:35.382 INFO:teuthology.orchestra.run.smithi027.stdout:/volumes/_nogroup/sv1/6c8b5da2-f8d6-43d8-942c-f8573173e9a0 2024-01-15T18:17:35.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.382+0000 7f8f64200640 1 -- 172.21.15.27:0/3240275293 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8f34067700 msgr2=0x7f8f34069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:35.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.382+0000 7f8f64200640 1 --2- 172.21.15.27:0/3240275293 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8f34067700 0x7f8f34069bc0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c069290 tx=0x7f8f4c00aa20 comp rx=0 tx=0).stop 2024-01-15T18:17:35.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.386+0000 7f8f64200640 1 -- 172.21.15.27:0/3240275293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f5c070bc0 msgr2=0x7f8f5c15c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:35.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.386+0000 7f8f64200640 1 --2- 172.21.15.27:0/3240275293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f5c070bc0 0x7f8f5c15c350 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f8f4401c570 tx=0x7f8f3c005800 comp rx=0 tx=0).stop 2024-01-15T18:17:35.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.386+0000 7f8f64200640 1 -- 172.21.15.27:0/3240275293 shutdown_connections 2024-01-15T18:17:35.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.386+0000 7f8f64200640 1 --2- 172.21.15.27:0/3240275293 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8f34067700 0x7f8f34069bc0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:35.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.386+0000 7f8f64200640 1 --2- 172.21.15.27:0/3240275293 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8f5c071040 0x7f8f5c15c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:35.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.386+0000 7f8f64200640 1 --2- 172.21.15.27:0/3240275293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f5c070bc0 0x7f8f5c15c350 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:35.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.386+0000 7f8f64200640 1 --2- 172.21.15.27:0/3240275293 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8f3c0047d0 0x7f8f5c15aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:35.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.386+0000 7f8f64200640 1 -- 172.21.15.27:0/3240275293 >> 172.21.15.27:0/3240275293 conn(0x7f8f5c0cb700 msgr2=0x7f8f5c1614d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:35.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.386+0000 7f8f64200640 1 -- 172.21.15.27:0/3240275293 shutdown_connections 2024-01-15T18:17:35.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.386+0000 7f8f64200640 1 -- 172.21.15.27:0/3240275293 wait complete. 2024-01-15T18:17:35.404 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:17:35.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.546+0000 7f28e0000640 1 Processor -- start 2024-01-15T18:17:35.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.550+0000 7f28e0000640 1 -- start start 2024-01-15T18:17:35.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.550+0000 7f28e0000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28d8158900 0x7f28d815f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:35.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.550+0000 7f28e0000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f28d80cf2e0 0x7f28d80701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:35.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.550+0000 7f28e0000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d80b1cb0 con 0x7f28d815bf00 2024-01-15T18:17:35.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.550+0000 7f28e0000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28d80b1f00 con 0x7f28d80cf2e0 2024-01-15T18:17:35.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.550+0000 7f28e0000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d80b21e0 con 0x7f28d8158900 2024-01-15T18:17:35.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.550+0000 7f28dd600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28d8158900 0x7f28d815f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:35.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.550+0000 7f28dd600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28d8158900 0x7f28d815f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:33266/0 (socket says 172.21.15.27:33266) 2024-01-15T18:17:35.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.550+0000 7f28dd600640 1 -- 172.21.15.27:0/1247858771 learned_addr learned my addr 172.21.15.27:0/1247858771 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:35.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.550+0000 7f28de000640 1 --2- 172.21.15.27:0/1247858771 >> v2:172.21.15.93:3300/0 conn(0x7f28d80cf2e0 0x7f28d80701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:35.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.550+0000 7f28dcc00640 1 --1- 172.21.15.27:0/1247858771 >> v1:172.21.15.93:6789/0 conn(0x7f28d815bf00 0x7f28d80cebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:60250/0 (socket says 172.21.15.27:60250) 2024-01-15T18:17:35.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.550+0000 7f28dd600640 1 -- 172.21.15.27:0/1247858771 >> v2:172.21.15.93:3300/0 conn(0x7f28d80cf2e0 msgr2=0x7f28d80701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:35.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.550+0000 7f28dd600640 1 --2- 172.21.15.27:0/1247858771 >> v2:172.21.15.93:3300/0 conn(0x7f28d80cf2e0 0x7f28d80701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:35.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.550+0000 7f28dd600640 1 -- 172.21.15.27:0/1247858771 >> v1:172.21.15.93:6789/0 conn(0x7f28d815bf00 legacy=0x7f28d80cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:35.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.550+0000 7f28dd600640 1 -- 172.21.15.27:0/1247858771 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d815ea70 con 0x7f28d8158900 2024-01-15T18:17:35.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.554+0000 7f28dd600640 1 --2- 172.21.15.27:0/1247858771 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28d8158900 0x7f28d815f550 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f28c8009870 tx=0x7f28c801c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef1d037a772e7851 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:35.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.554+0000 7f28d7e00640 1 -- 172.21.15.27:0/1247858771 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c80a7070 con 0x7f28d8158900 2024-01-15T18:17:35.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.554+0000 7f28d7e00640 1 -- 172.21.15.27:0/1247858771 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c80ab430 con 0x7f28d8158900 2024-01-15T18:17:35.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.554+0000 7f28d7e00640 1 -- 172.21.15.27:0/1247858771 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d8070f30 con 0x7f28d8158900 2024-01-15T18:17:35.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.554+0000 7f28e0000640 1 -- 172.21.15.27:0/1247858771 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28d8158900 msgr2=0x7f28d815f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:35.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.554+0000 7f28e0000640 1 --2- 172.21.15.27:0/1247858771 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28d8158900 0x7f28d815f550 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f28c8009870 tx=0x7f28c801c540 comp rx=0 tx=0).stop 2024-01-15T18:17:35.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.554+0000 7f28e0000640 1 -- 172.21.15.27:0/1247858771 shutdown_connections 2024-01-15T18:17:35.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.554+0000 7f28e0000640 1 --2- 172.21.15.27:0/1247858771 >> v2:172.21.15.93:3300/0 conn(0x7f28d80cf2e0 0x7f28d80701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:35.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.554+0000 7f28e0000640 1 --2- 172.21.15.27:0/1247858771 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28d8158900 0x7f28d815f550 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:35.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.554+0000 7f28e0000640 1 -- 172.21.15.27:0/1247858771 >> 172.21.15.27:0/1247858771 conn(0x7f28d80cb700 msgr2=0x7f28d80cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:35.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.554+0000 7f28e0000640 1 -- 172.21.15.27:0/1247858771 shutdown_connections 2024-01-15T18:17:35.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.554+0000 7f28e0000640 1 -- 172.21.15.27:0/1247858771 wait complete. 2024-01-15T18:17:35.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28e0000640 1 Processor -- start 2024-01-15T18:17:35.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28e0000640 1 -- start start 2024-01-15T18:17:35.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28e0000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28d80cf2e0 0x7f28d814e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:35.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28e0000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28d8158900 0x7f28d8149340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:35.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28e0000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d815bf00 0x7f28d8151670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:35.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28e0000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d814eb20 con 0x7f28d815bf00 2024-01-15T18:17:35.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28e0000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d80b0160 con 0x7f28d80cf2e0 2024-01-15T18:17:35.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28e0000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d80ccba0 con 0x7f28d8158900 2024-01-15T18:17:35.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28de000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d815bf00 0x7f28d8151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:35.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28de000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d815bf00 0x7f28d8151670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49594/0 (socket says 172.21.15.27:49594) 2024-01-15T18:17:35.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28de000640 1 -- 172.21.15.27:0/818778294 learned_addr learned my addr 172.21.15.27:0/818778294 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:35.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28dd600640 1 --2- 172.21.15.27:0/818778294 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28d80cf2e0 0x7f28d814e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:35.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28de000640 1 -- 172.21.15.27:0/818778294 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28d8158900 msgr2=0x7f28d8149340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:35.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28dcc00640 1 --2- 172.21.15.27:0/818778294 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28d8158900 0x7f28d8149340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:35.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28de000640 1 --2- 172.21.15.27:0/818778294 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28d8158900 0x7f28d8149340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:35.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28de000640 1 -- 172.21.15.27:0/818778294 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28d80cf2e0 msgr2=0x7f28d814e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:35.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28de000640 1 --2- 172.21.15.27:0/818778294 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28d80cf2e0 0x7f28d814e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:35.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28de000640 1 -- 172.21.15.27:0/818778294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c80094d0 con 0x7f28d815bf00 2024-01-15T18:17:35.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28dcc00640 1 --2- 172.21.15.27:0/818778294 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28d8158900 0x7f28d8149340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:17:35.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28dd600640 1 --2- 172.21.15.27:0/818778294 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28d80cf2e0 0x7f28d814e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:17:35.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28de000640 1 --2- 172.21.15.27:0/818778294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d815bf00 0x7f28d8151670 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f28d000efc0 tx=0x7f28d000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:35.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28d5600640 1 -- 172.21.15.27:0/818778294 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d0009ba0 con 0x7f28d815bf00 2024-01-15T18:17:35.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28d5600640 1 -- 172.21.15.27:0/818778294 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d000f040 con 0x7f28d815bf00 2024-01-15T18:17:35.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28d5600640 1 -- 172.21.15.27:0/818778294 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d00146c0 con 0x7f28d815bf00 2024-01-15T18:17:35.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28e0000640 1 -- 172.21.15.27:0/818778294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28d80cce80 con 0x7f28d815bf00 2024-01-15T18:17:35.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.558+0000 7f28e0000640 1 -- 172.21.15.27:0/818778294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d80cd440 con 0x7f28d815bf00 2024-01-15T18:17:35.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.562+0000 7f28e0000640 1 -- 172.21.15.27:0/818778294 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28d80cd630 con 0x7f28d815bf00 2024-01-15T18:17:35.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.562+0000 7f28d5600640 1 -- 172.21.15.27:0/818778294 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 34) v1 ==== 87435+0+0 (secure 0 0 0) 0x7f28d0014820 con 0x7f28d815bf00 2024-01-15T18:17:35.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.562+0000 7f28d5600640 1 --2- 172.21.15.27:0/818778294 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f28b0067700 0x7f28b0069bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:35.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.562+0000 7f28d5600640 1 -- 172.21.15.27:0/818778294 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f28d0083f20 con 0x7f28d815bf00 2024-01-15T18:17:35.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.562+0000 7f28dd600640 1 --2- 172.21.15.27:0/818778294 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f28b0067700 0x7f28b0069bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:35.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.562+0000 7f28dd600640 1 --2- 172.21.15.27:0/818778294 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f28b0067700 0x7f28b0069bc0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f28c80af000 tx=0x7f28c8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:35.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.566+0000 7f28d5600640 1 -- 172.21.15.27:0/818778294 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f28d00528f0 con 0x7f28d815bf00 2024-01-15T18:17:35.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.750+0000 7f28e0000640 1 -- 172.21.15.27:0/818778294 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f28d806e170 con 0x7f28b0067700 2024-01-15T18:17:35.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.754+0000 7f28d5600640 1 -- 172.21.15.27:0/818778294 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f28c80ba0b0 con 0x7f28b0067700 2024-01-15T18:17:35.754 INFO:teuthology.orchestra.run.smithi027.stdout:/volumes/_nogroup/sv1/6c8b5da2-f8d6-43d8-942c-f8573173e9a0 2024-01-15T18:17:35.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.758+0000 7f28e0000640 1 -- 172.21.15.27:0/818778294 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f28b0067700 msgr2=0x7f28b0069bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:35.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.758+0000 7f28e0000640 1 --2- 172.21.15.27:0/818778294 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f28b0067700 0x7f28b0069bc0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f28c80af000 tx=0x7f28c8008040 comp rx=0 tx=0).stop 2024-01-15T18:17:35.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.758+0000 7f28e0000640 1 -- 172.21.15.27:0/818778294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d815bf00 msgr2=0x7f28d8151670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:35.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.758+0000 7f28e0000640 1 --2- 172.21.15.27:0/818778294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d815bf00 0x7f28d8151670 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f28d000efc0 tx=0x7f28d000c490 comp rx=0 tx=0).stop 2024-01-15T18:17:35.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.758+0000 7f28e0000640 1 -- 172.21.15.27:0/818778294 shutdown_connections 2024-01-15T18:17:35.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.758+0000 7f28e0000640 1 --2- 172.21.15.27:0/818778294 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f28b0067700 0x7f28b0069bc0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:35.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.758+0000 7f28e0000640 1 --2- 172.21.15.27:0/818778294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28d815bf00 0x7f28d8151670 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:35.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.758+0000 7f28e0000640 1 --2- 172.21.15.27:0/818778294 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f28d8158900 0x7f28d8149340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:35.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.758+0000 7f28e0000640 1 --2- 172.21.15.27:0/818778294 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f28d80cf2e0 0x7f28d814e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:35.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.758+0000 7f28e0000640 1 -- 172.21.15.27:0/818778294 >> 172.21.15.27:0/818778294 conn(0x7f28d80cb700 msgr2=0x7f28d80b1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:35.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.758+0000 7f28e0000640 1 -- 172.21.15.27:0/818778294 shutdown_connections 2024-01-15T18:17:35.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:35.758+0000 7f28e0000640 1 -- 172.21.15.27:0/818778294 wait complete. 2024-01-15T18:17:35.776 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/6c8b5da2-f8d6-43d8-942c-f8573173e9a0 2024-01-15T18:17:35.776 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:17:35.777 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:17:35.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:35.786 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/6c8b5da2-f8d6-43d8-942c-f8573173e9a0/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:17:42.982 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records in 2024-01-15T18:17:42.983 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records out 2024-01-15T18:17:42.983 INFO:teuthology.orchestra.run.smithi027.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.13516 s, 150 MB/s 2024-01-15T18:17:42.984 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:17:42.984 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:17:42.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:42.995 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/6c8b5da2-f8d6-43d8-942c-f8573173e9a0/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:17:49.936 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records in 2024-01-15T18:17:49.936 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records out 2024-01-15T18:17:49.936 INFO:teuthology.orchestra.run.smithi027.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.88387 s, 156 MB/s 2024-01-15T18:17:49.938 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:17:49.938 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:17:49.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:49.947 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/6c8b5da2-f8d6-43d8-942c-f8573173e9a0/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:17:56.838 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records in 2024-01-15T18:17:56.838 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records out 2024-01-15T18:17:56.838 INFO:teuthology.orchestra.run.smithi027.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.8343 s, 157 MB/s 2024-01-15T18:17:56.839 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-15T18:17:56.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.986+0000 7fac7da00640 1 Processor -- start 2024-01-15T18:17:56.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.986+0000 7fac7da00640 1 -- start start 2024-01-15T18:17:56.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.986+0000 7fac7da00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fac7815d3b0 0x7fac780cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:56.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.986+0000 7fac7da00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fac7806b600 0x7fac7806bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:56.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.986+0000 7fac7da00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac780b04b0 con 0x7fac7806c220 2024-01-15T18:17:56.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.986+0000 7fac7da00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac7806c6b0 con 0x7fac7815d3b0 2024-01-15T18:17:56.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.986+0000 7fac7da00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac780d0160 con 0x7fac7806b600 2024-01-15T18:17:56.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.986+0000 7fac76a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fac7815d3b0 0x7fac780cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:56.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.986+0000 7fac76a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fac7815d3b0 0x7fac780cfc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43206/0 (socket says 172.21.15.27:43206) 2024-01-15T18:17:56.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.986+0000 7fac76a00640 1 -- 172.21.15.27:0/1090548158 learned_addr learned my addr 172.21.15.27:0/1090548158 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:56.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.986+0000 7fac76000640 1 --2- 172.21.15.27:0/1090548158 >> v2:172.21.15.93:3300/0 conn(0x7fac7806b600 0x7fac7806bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:56.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.986+0000 7fac76a00640 1 -- 172.21.15.27:0/1090548158 >> v2:172.21.15.93:3300/0 conn(0x7fac7806b600 msgr2=0x7fac7806bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:56.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.986+0000 7fac76a00640 1 --2- 172.21.15.27:0/1090548158 >> v2:172.21.15.93:3300/0 conn(0x7fac7806b600 0x7fac7806bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:56.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.986+0000 7fac76a00640 1 -- 172.21.15.27:0/1090548158 >> v1:172.21.15.93:6789/0 conn(0x7fac7806c220 legacy=0x7fac78160f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:56.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.986+0000 7fac76a00640 1 -- 172.21.15.27:0/1090548158 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac780b0720 con 0x7fac7815d3b0 2024-01-15T18:17:56.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.986+0000 7fac76000640 1 --2- 172.21.15.27:0/1090548158 >> v2:172.21.15.93:3300/0 conn(0x7fac7806b600 0x7fac7806bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:17:56.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.990+0000 7fac76a00640 1 --2- 172.21.15.27:0/1090548158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac7815d3b0 0x7fac780cfc20 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fac78070e80 tx=0x7fac6801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bb6299db9f54658 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:56.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.990+0000 7fac75600640 1 -- 172.21.15.27:0/1090548158 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac680aa430 con 0x7fac7815d3b0 2024-01-15T18:17:56.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.990+0000 7fac75600640 1 -- 172.21.15.27:0/1090548158 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac680aa590 con 0x7fac7815d3b0 2024-01-15T18:17:56.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.990+0000 7fac75600640 1 -- 172.21.15.27:0/1090548158 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac680aa920 con 0x7fac7815d3b0 2024-01-15T18:17:56.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.990+0000 7fac7da00640 1 -- 172.21.15.27:0/1090548158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac7815d3b0 msgr2=0x7fac780cfc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:56.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.990+0000 7fac7da00640 1 --2- 172.21.15.27:0/1090548158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac7815d3b0 0x7fac780cfc20 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fac78070e80 tx=0x7fac6801c4d0 comp rx=0 tx=0).stop 2024-01-15T18:17:56.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.990+0000 7fac7da00640 1 -- 172.21.15.27:0/1090548158 shutdown_connections 2024-01-15T18:17:56.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.990+0000 7fac7da00640 1 --2- 172.21.15.27:0/1090548158 >> v2:172.21.15.93:3300/0 conn(0x7fac7806b600 0x7fac7806bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:56.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.990+0000 7fac7da00640 1 --2- 172.21.15.27:0/1090548158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac7815d3b0 0x7fac780cfc20 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:56.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.990+0000 7fac7da00640 1 -- 172.21.15.27:0/1090548158 >> 172.21.15.27:0/1090548158 conn(0x7fac780d0ed0 msgr2=0x7fac780d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:56.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.990+0000 7fac7da00640 1 -- 172.21.15.27:0/1090548158 shutdown_connections 2024-01-15T18:17:56.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.990+0000 7fac7da00640 1 -- 172.21.15.27:0/1090548158 wait complete. 2024-01-15T18:17:56.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.994+0000 7fac7da00640 1 Processor -- start 2024-01-15T18:17:56.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.994+0000 7fac7da00640 1 -- start start 2024-01-15T18:17:56.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.994+0000 7fac7da00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fac7806b600 0x7fac780c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:56.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.994+0000 7fac7da00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac7806c220 0x7fac78159460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:56.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.994+0000 7fac7da00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac7815d3b0 0x7fac781547c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:56.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.994+0000 7fac7da00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac781543a0 con 0x7fac7806c220 2024-01-15T18:17:56.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.994+0000 7fac7da00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac780c5120 con 0x7fac7806b600 2024-01-15T18:17:56.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.994+0000 7fac7da00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac780c7d00 con 0x7fac7815d3b0 2024-01-15T18:17:56.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.994+0000 7fac76000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac7806c220 0x7fac78159460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:56.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.994+0000 7fac76000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac7806c220 0x7fac78159460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43208/0 (socket says 172.21.15.27:43208) 2024-01-15T18:17:56.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.994+0000 7fac76000640 1 -- 172.21.15.27:0/60073799 learned_addr learned my addr 172.21.15.27:0/60073799 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:56.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.994+0000 7fac76a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fac7806b600 0x7fac780c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:56.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.994+0000 7fac77400640 1 --2- 172.21.15.27:0/60073799 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac7815d3b0 0x7fac781547c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:56.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.994+0000 7fac76000640 1 -- 172.21.15.27:0/60073799 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac7815d3b0 msgr2=0x7fac781547c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:56.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.994+0000 7fac76000640 1 --2- 172.21.15.27:0/60073799 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac7815d3b0 0x7fac781547c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:56.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.994+0000 7fac76000640 1 -- 172.21.15.27:0/60073799 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fac7806b600 msgr2=0x7fac780c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:56.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.994+0000 7fac76000640 1 --2- 172.21.15.27:0/60073799 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fac7806b600 0x7fac780c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:56.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.994+0000 7fac76000640 1 -- 172.21.15.27:0/60073799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac680094d0 con 0x7fac7806c220 2024-01-15T18:17:56.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.994+0000 7fac76000640 1 --2- 172.21.15.27:0/60073799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac7806c220 0x7fac78159460 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fac6000d800 tx=0x7fac6000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:56.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.994+0000 7fac6ea00640 1 -- 172.21.15.27:0/60073799 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac600096c0 con 0x7fac7806c220 2024-01-15T18:17:56.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.994+0000 7fac6ea00640 1 -- 172.21.15.27:0/60073799 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac60009820 con 0x7fac7806c220 2024-01-15T18:17:56.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.994+0000 7fac6ea00640 1 -- 172.21.15.27:0/60073799 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac60009bd0 con 0x7fac7806c220 2024-01-15T18:17:56.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.994+0000 7fac7da00640 1 -- 172.21.15.27:0/60073799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac780c7a20 con 0x7fac7806c220 2024-01-15T18:17:56.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.994+0000 7fac7da00640 1 -- 172.21.15.27:0/60073799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac78153fc0 con 0x7fac7806c220 2024-01-15T18:17:56.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.998+0000 7fac7da00640 1 -- 172.21.15.27:0/60073799 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac7806d550 con 0x7fac7806c220 2024-01-15T18:17:56.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.998+0000 7fac6ea00640 1 -- 172.21.15.27:0/60073799 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 35) v1 ==== 87486+0+0 (secure 0 0 0) 0x7fac60009d30 con 0x7fac7806c220 2024-01-15T18:17:56.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.998+0000 7fac6ea00640 1 --2- 172.21.15.27:0/60073799 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fac500677c0 0x7fac50069c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:56.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.998+0000 7fac6ea00640 1 -- 172.21.15.27:0/60073799 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fac600837b0 con 0x7fac7806c220 2024-01-15T18:17:56.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.998+0000 7fac76a00640 1 --2- 172.21.15.27:0/60073799 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fac500677c0 0x7fac50069c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:56.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:56.998+0000 7fac76a00640 1 --2- 172.21.15.27:0/60073799 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fac500677c0 0x7fac50069c80 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fac680ae000 tx=0x7fac680b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:57.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.002+0000 7fac6ea00640 1 -- 172.21.15.27:0/60073799 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fac60052210 con 0x7fac7806c220 2024-01-15T18:17:57.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.190+0000 7fac7da00640 1 -- 172.21.15.27:0/60073799 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7fac780ced60 con 0x7fac500677c0 2024-01-15T18:17:57.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.202+0000 7fac6ea00640 1 -- 172.21.15.27:0/60073799 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fac780ced60 con 0x7fac500677c0 2024-01-15T18:17:57.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.206+0000 7fac7da00640 1 -- 172.21.15.27:0/60073799 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fac500677c0 msgr2=0x7fac50069c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:57.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.206+0000 7fac7da00640 1 --2- 172.21.15.27:0/60073799 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fac500677c0 0x7fac50069c80 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fac680ae000 tx=0x7fac680b5040 comp rx=0 tx=0).stop 2024-01-15T18:17:57.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.206+0000 7fac7da00640 1 -- 172.21.15.27:0/60073799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac7806c220 msgr2=0x7fac78159460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:57.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.206+0000 7fac7da00640 1 --2- 172.21.15.27:0/60073799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac7806c220 0x7fac78159460 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fac6000d800 tx=0x7fac6000dcc0 comp rx=0 tx=0).stop 2024-01-15T18:17:57.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.206+0000 7fac7da00640 1 -- 172.21.15.27:0/60073799 shutdown_connections 2024-01-15T18:17:57.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.206+0000 7fac7da00640 1 --2- 172.21.15.27:0/60073799 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fac500677c0 0x7fac50069c80 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:57.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.206+0000 7fac7da00640 1 --2- 172.21.15.27:0/60073799 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac7815d3b0 0x7fac781547c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:57.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.206+0000 7fac7da00640 1 --2- 172.21.15.27:0/60073799 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac7806c220 0x7fac78159460 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:57.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.206+0000 7fac7da00640 1 --2- 172.21.15.27:0/60073799 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fac7806b600 0x7fac780c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:57.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.206+0000 7fac7da00640 1 -- 172.21.15.27:0/60073799 >> 172.21.15.27:0/60073799 conn(0x7fac780d0ed0 msgr2=0x7fac7806fcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:57.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.206+0000 7fac7da00640 1 -- 172.21.15.27:0/60073799 shutdown_connections 2024-01-15T18:17:57.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.206+0000 7fac7da00640 1 -- 172.21.15.27:0/60073799 wait complete. 2024-01-15T18:17:57.226 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:17:57.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.370+0000 7fa591200640 1 Processor -- start 2024-01-15T18:17:57.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.370+0000 7fa591200640 1 -- start start 2024-01-15T18:17:57.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.370+0000 7fa591200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa58c15be40 0x7fa58c0cf940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:57.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.370+0000 7fa591200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa58c0cebf0 con 0x7fa58c072360 2024-01-15T18:17:57.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.370+0000 7fa591200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa58c0b46e0 con 0x7fa58c0b4300 2024-01-15T18:17:57.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.370+0000 7fa591200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa58c06a4f0 con 0x7fa58c15be40 2024-01-15T18:17:57.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.370+0000 7fa589400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa58c072360 0x7fa58c15b730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58180/0 (socket says 172.21.15.27:58180) 2024-01-15T18:17:57.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.370+0000 7fa589400640 1 -- 172.21.15.27:0/3003934829 learned_addr learned my addr 172.21.15.27:0/3003934829 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:57.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.370+0000 7fa58b200640 1 --2- 172.21.15.27:0/3003934829 >> v2:172.21.15.93:3300/0 conn(0x7fa58c15be40 0x7fa58c0cf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:57.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.370+0000 7fa588a00640 1 -- 172.21.15.27:0/3003934829 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 500716635 0 0) 0x7fa58c0cebf0 con 0x7fa58c072360 2024-01-15T18:17:57.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.370+0000 7fa588a00640 1 -- 172.21.15.27:0/3003934829 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa56c003530 con 0x7fa58c072360 2024-01-15T18:17:57.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.370+0000 7fa588a00640 1 -- 172.21.15.27:0/3003934829 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 380888626 0 0) 0x7fa58c0b46e0 con 0x7fa58c0b4300 2024-01-15T18:17:57.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.374+0000 7fa588a00640 1 -- 172.21.15.27:0/3003934829 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa58c0cebf0 con 0x7fa58c0b4300 2024-01-15T18:17:57.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.374+0000 7fa58b200640 1 -- 172.21.15.27:0/3003934829 >> v1:172.21.15.93:6789/0 conn(0x7fa58c0b4300 legacy=0x7fa58c071c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:57.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.374+0000 7fa58b200640 1 -- 172.21.15.27:0/3003934829 >> v1:172.21.15.27:6789/0 conn(0x7fa58c072360 legacy=0x7fa58c15b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:57.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.374+0000 7fa58b200640 1 -- 172.21.15.27:0/3003934829 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa58c0cee60 con 0x7fa58c15be40 2024-01-15T18:17:57.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.374+0000 7fa58b200640 1 --2- 172.21.15.27:0/3003934829 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa58c15be40 0x7fa58c0cf940 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fa584009d90 tx=0x7fa58401f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da12ffa34251fff1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:57.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.374+0000 7fa588a00640 1 -- 172.21.15.27:0/3003934829 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5840af600 con 0x7fa58c15be40 2024-01-15T18:17:57.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.374+0000 7fa588a00640 1 -- 172.21.15.27:0/3003934829 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5840ab050 con 0x7fa58c15be40 2024-01-15T18:17:57.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.374+0000 7fa588a00640 1 -- 172.21.15.27:0/3003934829 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5840af980 con 0x7fa58c15be40 2024-01-15T18:17:57.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.374+0000 7fa591200640 1 -- 172.21.15.27:0/3003934829 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa58c15be40 msgr2=0x7fa58c0cf940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:57.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.374+0000 7fa591200640 1 --2- 172.21.15.27:0/3003934829 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa58c15be40 0x7fa58c0cf940 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fa584009d90 tx=0x7fa58401f130 comp rx=0 tx=0).stop 2024-01-15T18:17:57.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.378+0000 7fa591200640 1 -- 172.21.15.27:0/3003934829 shutdown_connections 2024-01-15T18:17:57.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.378+0000 7fa591200640 1 --2- 172.21.15.27:0/3003934829 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa58c15be40 0x7fa58c0cf940 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:57.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.378+0000 7fa591200640 1 -- 172.21.15.27:0/3003934829 >> 172.21.15.27:0/3003934829 conn(0x7fa58c099e10 msgr2=0x7fa58c0b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:57.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.378+0000 7fa591200640 1 -- 172.21.15.27:0/3003934829 shutdown_connections 2024-01-15T18:17:57.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.378+0000 7fa591200640 1 -- 172.21.15.27:0/3003934829 wait complete. 2024-01-15T18:17:57.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.378+0000 7fa591200640 1 Processor -- start 2024-01-15T18:17:57.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.378+0000 7fa591200640 1 -- start start 2024-01-15T18:17:57.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.378+0000 7fa591200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa58c072360 0x7fa58c1418c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:57.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.378+0000 7fa591200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa58c0b4300 0x7fa58c0c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:57.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.378+0000 7fa591200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa58c15be40 0x7fa58c15d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:57.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.378+0000 7fa591200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c1f6790 con 0x7fa58c15be40 2024-01-15T18:17:57.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.378+0000 7fa591200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c1f6a70 con 0x7fa58c072360 2024-01-15T18:17:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.378+0000 7fa591200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c1f6d50 con 0x7fa58c0b4300 2024-01-15T18:17:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.378+0000 7fa58b200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa58c15be40 0x7fa58c15d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.378+0000 7fa58b200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa58c15be40 0x7fa58c15d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43224/0 (socket says 172.21.15.27:43224) 2024-01-15T18:17:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.378+0000 7fa58b200640 1 -- 172.21.15.27:0/2116017029 learned_addr learned my addr 172.21.15.27:0/2116017029 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.378+0000 7fa589400640 1 --2- 172.21.15.27:0/2116017029 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa58c0b4300 0x7fa58c0c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.378+0000 7fa589e00640 1 --2- 172.21.15.27:0/2116017029 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa58c072360 0x7fa58c1418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.378+0000 7fa58b200640 1 -- 172.21.15.27:0/2116017029 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa58c0b4300 msgr2=0x7fa58c0c99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.378+0000 7fa58b200640 1 --2- 172.21.15.27:0/2116017029 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa58c0b4300 0x7fa58c0c99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.378+0000 7fa58b200640 1 -- 172.21.15.27:0/2116017029 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa58c072360 msgr2=0x7fa58c1418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.378+0000 7fa58b200640 1 --2- 172.21.15.27:0/2116017029 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa58c072360 0x7fa58c1418c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.378+0000 7fa58b200640 1 -- 172.21.15.27:0/2116017029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa584009a70 con 0x7fa58c15be40 2024-01-15T18:17:57.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.378+0000 7fa589e00640 1 --2- 172.21.15.27:0/2116017029 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa58c072360 0x7fa58c1418c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:17:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.382+0000 7fa58b200640 1 --2- 172.21.15.27:0/2116017029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa58c15be40 0x7fa58c15d710 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fa5840b2040 tx=0x7fa58401bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.382+0000 7fa582000640 1 -- 172.21.15.27:0/2116017029 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa584009450 con 0x7fa58c15be40 2024-01-15T18:17:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.382+0000 7fa582000640 1 -- 172.21.15.27:0/2116017029 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5840b9410 con 0x7fa58c15be40 2024-01-15T18:17:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.382+0000 7fa591200640 1 -- 172.21.15.27:0/2116017029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa58c1f7030 con 0x7fa58c15be40 2024-01-15T18:17:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.382+0000 7fa582000640 1 -- 172.21.15.27:0/2116017029 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5840b97c0 con 0x7fa58c15be40 2024-01-15T18:17:57.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.382+0000 7fa591200640 1 -- 172.21.15.27:0/2116017029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa58c1421f0 con 0x7fa58c15be40 2024-01-15T18:17:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.382+0000 7fa591200640 1 -- 172.21.15.27:0/2116017029 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa58c142440 con 0x7fa58c15be40 2024-01-15T18:17:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.382+0000 7fa582000640 1 -- 172.21.15.27:0/2116017029 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 35) v1 ==== 87486+0+0 (secure 0 0 0) 0x7fa5840b6050 con 0x7fa58c15be40 2024-01-15T18:17:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.382+0000 7fa582000640 1 --2- 172.21.15.27:0/2116017029 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa5640677c0 0x7fa564069c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.382+0000 7fa582000640 1 -- 172.21.15.27:0/2116017029 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fa584120e80 con 0x7fa58c15be40 2024-01-15T18:17:57.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.382+0000 7fa589e00640 1 --2- 172.21.15.27:0/2116017029 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa5640677c0 0x7fa564069c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:57.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.386+0000 7fa589e00640 1 --2- 172.21.15.27:0/2116017029 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa5640677c0 0x7fa564069c80 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fa57c0011f0 tx=0x7fa57c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:57.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.386+0000 7fa582000640 1 -- 172.21.15.27:0/2116017029 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa5840ef830 con 0x7fa58c15be40 2024-01-15T18:17:57.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.574+0000 7fa591200640 1 -- 172.21.15.27:0/2116017029 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa58c071ce0 con 0x7fa5640677c0 2024-01-15T18:17:57.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.578+0000 7fa582000640 1 -- 172.21.15.27:0/2116017029 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fa58c071ce0 con 0x7fa5640677c0 2024-01-15T18:17:57.577 INFO:teuthology.orchestra.run.smithi027.stdout:/volumes/_nogroup/sv1/6c8b5da2-f8d6-43d8-942c-f8573173e9a0 2024-01-15T18:17:57.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.578+0000 7fa591200640 1 -- 172.21.15.27:0/2116017029 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa5640677c0 msgr2=0x7fa564069c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:57.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.578+0000 7fa591200640 1 --2- 172.21.15.27:0/2116017029 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa5640677c0 0x7fa564069c80 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fa57c0011f0 tx=0x7fa57c009040 comp rx=0 tx=0).stop 2024-01-15T18:17:57.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.578+0000 7fa591200640 1 -- 172.21.15.27:0/2116017029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa58c15be40 msgr2=0x7fa58c15d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:57.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.578+0000 7fa591200640 1 --2- 172.21.15.27:0/2116017029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa58c15be40 0x7fa58c15d710 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fa5840b2040 tx=0x7fa58401bed0 comp rx=0 tx=0).stop 2024-01-15T18:17:57.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.578+0000 7fa591200640 1 -- 172.21.15.27:0/2116017029 shutdown_connections 2024-01-15T18:17:57.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.578+0000 7fa591200640 1 --2- 172.21.15.27:0/2116017029 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa5640677c0 0x7fa564069c80 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:57.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.578+0000 7fa591200640 1 --2- 172.21.15.27:0/2116017029 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa58c15be40 0x7fa58c15d710 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:57.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.578+0000 7fa591200640 1 --2- 172.21.15.27:0/2116017029 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa58c0b4300 0x7fa58c0c99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:57.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.578+0000 7fa591200640 1 --2- 172.21.15.27:0/2116017029 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa58c072360 0x7fa58c1418c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:57.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.578+0000 7fa591200640 1 -- 172.21.15.27:0/2116017029 >> 172.21.15.27:0/2116017029 conn(0x7fa58c099e10 msgr2=0x7fa58c15ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:57.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.578+0000 7fa591200640 1 -- 172.21.15.27:0/2116017029 shutdown_connections 2024-01-15T18:17:57.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.578+0000 7fa591200640 1 -- 172.21.15.27:0/2116017029 wait complete. 2024-01-15T18:17:57.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:17:57.599 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/6c8b5da2-f8d6-43d8-942c-f8573173e9a0) 2024-01-15T18:17:57.612 INFO:teuthology.orchestra.run.smithi027.stdout:# file: volumes/_nogroup/sv1/6c8b5da2-f8d6-43d8-942c-f8573173e9a0 2024-01-15T18:17:57.612 INFO:teuthology.orchestra.run.smithi027.stdout:ceph.dir.rbytes="3221225472" 2024-01-15T18:17:57.612 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:17:57.613 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-15T18:17:57.614 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 ss1clone1 2024-01-15T18:17:57.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.802+0000 7fce90400640 1 Processor -- start 2024-01-15T18:17:57.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.802+0000 7fce90400640 1 -- start start 2024-01-15T18:17:57.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.802+0000 7fce90400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fce8815d570 0x7fce8806be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:57.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.802+0000 7fce90400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce880b0d70 con 0x7fce8815d190 2024-01-15T18:17:57.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.802+0000 7fce90400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce880d0540 con 0x7fce8806b810 2024-01-15T18:17:57.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.802+0000 7fce90400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce880b0440 con 0x7fce8815d570 2024-01-15T18:17:57.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.802+0000 7fce8da00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fce8815d190 0x7fce880cfcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58188/0 (socket says 172.21.15.27:58188) 2024-01-15T18:17:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.802+0000 7fce8da00640 1 -- 172.21.15.27:0/3949123901 learned_addr learned my addr 172.21.15.27:0/3949123901 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.802+0000 7fce8d000640 1 --2- 172.21.15.27:0/3949123901 >> v2:172.21.15.93:3300/0 conn(0x7fce8815d570 0x7fce8806be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.802+0000 7fce87e00640 1 -- 172.21.15.27:0/3949123901 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2357704349 0 0) 0x7fce880b0d70 con 0x7fce8815d190 2024-01-15T18:17:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.802+0000 7fce87e00640 1 -- 172.21.15.27:0/3949123901 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce68003530 con 0x7fce8815d190 2024-01-15T18:17:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.802+0000 7fce8d000640 1 -- 172.21.15.27:0/3949123901 >> v1:172.21.15.93:6789/0 conn(0x7fce8806b810 legacy=0x7fce880cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.802+0000 7fce87e00640 1 -- 172.21.15.27:0/3949123901 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 201565947 0 0) 0x7fce68003530 con 0x7fce8815d190 2024-01-15T18:17:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.802+0000 7fce8d000640 1 -- 172.21.15.27:0/3949123901 >> v1:172.21.15.27:6789/0 conn(0x7fce8815d190 legacy=0x7fce880cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:57.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.802+0000 7fce8d000640 1 -- 172.21.15.27:0/3949123901 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce880b0720 con 0x7fce8815d570 2024-01-15T18:17:57.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.806+0000 7fce8d000640 1 --2- 172.21.15.27:0/3949123901 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce8815d570 0x7fce8806be10 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fce700097c0 tx=0x7fce7001c370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54684ffdc717769a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:57.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.806+0000 7fce87e00640 1 -- 172.21.15.27:0/3949123901 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce78001680 con 0x7fce8815d570 2024-01-15T18:17:57.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.806+0000 7fce87e00640 1 -- 172.21.15.27:0/3949123901 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce700ac440 con 0x7fce8815d570 2024-01-15T18:17:57.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.806+0000 7fce87e00640 1 -- 172.21.15.27:0/3949123901 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce700ac7d0 con 0x7fce8815d570 2024-01-15T18:17:57.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.806+0000 7fce90400640 1 -- 172.21.15.27:0/3949123901 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce8815d570 msgr2=0x7fce8806be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:57.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.806+0000 7fce90400640 1 --2- 172.21.15.27:0/3949123901 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce8815d570 0x7fce8806be10 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fce700097c0 tx=0x7fce7001c370 comp rx=0 tx=0).stop 2024-01-15T18:17:57.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.806+0000 7fce90400640 1 -- 172.21.15.27:0/3949123901 shutdown_connections 2024-01-15T18:17:57.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.806+0000 7fce90400640 1 --2- 172.21.15.27:0/3949123901 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce8815d570 0x7fce8806be10 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:57.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.806+0000 7fce90400640 1 -- 172.21.15.27:0/3949123901 >> 172.21.15.27:0/3949123901 conn(0x7fce880d0e70 msgr2=0x7fce880d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:57.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.810+0000 7fce90400640 1 -- 172.21.15.27:0/3949123901 shutdown_connections 2024-01-15T18:17:57.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.810+0000 7fce90400640 1 -- 172.21.15.27:0/3949123901 wait complete. 2024-01-15T18:17:57.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.810+0000 7fce90400640 1 Processor -- start 2024-01-15T18:17:57.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.810+0000 7fce90400640 1 -- start start 2024-01-15T18:17:57.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.810+0000 7fce90400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce8806b810 0x7fce8814b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:57.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.810+0000 7fce90400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce8815d190 0x7fce880b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:57.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.810+0000 7fce90400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce8815d570 0x7fce880b5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:57.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.810+0000 7fce90400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce880b6d30 con 0x7fce8815d570 2024-01-15T18:17:57.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.810+0000 7fce90400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce880b6530 con 0x7fce8815d190 2024-01-15T18:17:57.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.810+0000 7fce90400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce8814c530 con 0x7fce8806b810 2024-01-15T18:17:57.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.810+0000 7fce8e400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce8815d570 0x7fce880b5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:57.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.810+0000 7fce8e400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce8815d570 0x7fce880b5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43236/0 (socket says 172.21.15.27:43236) 2024-01-15T18:17:57.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.810+0000 7fce8e400640 1 -- 172.21.15.27:0/674254567 learned_addr learned my addr 172.21.15.27:0/674254567 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:57.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.810+0000 7fce8d000640 1 --2- 172.21.15.27:0/674254567 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce8815d190 0x7fce880b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:57.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.810+0000 7fce8da00640 1 --2- 172.21.15.27:0/674254567 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce8806b810 0x7fce8814b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:57.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.810+0000 7fce8e400640 1 -- 172.21.15.27:0/674254567 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce8806b810 msgr2=0x7fce8814b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:57.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.810+0000 7fce8e400640 1 --2- 172.21.15.27:0/674254567 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce8806b810 0x7fce8814b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:57.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.810+0000 7fce8e400640 1 -- 172.21.15.27:0/674254567 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce8815d190 msgr2=0x7fce880b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:57.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.810+0000 7fce8e400640 1 --2- 172.21.15.27:0/674254567 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce8815d190 0x7fce880b5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:57.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.810+0000 7fce8e400640 1 -- 172.21.15.27:0/674254567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce70009420 con 0x7fce8815d570 2024-01-15T18:17:57.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.810+0000 7fce8d000640 1 --2- 172.21.15.27:0/674254567 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce8815d190 0x7fce880b5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:17:57.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.810+0000 7fce8e400640 1 --2- 172.21.15.27:0/674254567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce8815d570 0x7fce880b5ff0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fce8000c7a0 tx=0x7fce8000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:57.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.814+0000 7fce85600640 1 -- 172.21.15.27:0/674254567 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce80003b80 con 0x7fce8815d570 2024-01-15T18:17:57.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.814+0000 7fce85600640 1 -- 172.21.15.27:0/674254567 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce80003ce0 con 0x7fce8815d570 2024-01-15T18:17:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.814+0000 7fce90400640 1 -- 172.21.15.27:0/674254567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce880b6770 con 0x7fce8815d570 2024-01-15T18:17:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.814+0000 7fce85600640 1 -- 172.21.15.27:0/674254567 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8000aa30 con 0x7fce8815d570 2024-01-15T18:17:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.814+0000 7fce90400640 1 -- 172.21.15.27:0/674254567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce8814c1c0 con 0x7fce8815d570 2024-01-15T18:17:57.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.814+0000 7fce90400640 1 -- 172.21.15.27:0/674254567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce8814c020 con 0x7fce8815d570 2024-01-15T18:17:57.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.814+0000 7fce85600640 1 -- 172.21.15.27:0/674254567 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 35) v1 ==== 87486+0+0 (secure 0 0 0) 0x7fce8000ab90 con 0x7fce8815d570 2024-01-15T18:17:57.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.814+0000 7fce85600640 1 --2- 172.21.15.27:0/674254567 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce600677c0 0x7fce60069c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:57.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.814+0000 7fce85600640 1 -- 172.21.15.27:0/674254567 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fce80082b90 con 0x7fce8815d570 2024-01-15T18:17:57.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.814+0000 7fce8da00640 1 --2- 172.21.15.27:0/674254567 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce600677c0 0x7fce60069c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:57.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.818+0000 7fce8da00640 1 --2- 172.21.15.27:0/674254567 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce600677c0 0x7fce60069c80 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fce78000f70 tx=0x7fce78001e10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:57.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:57.818+0000 7fce85600640 1 -- 172.21.15.27:0/674254567 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fce8814c020 con 0x7fce8815d570 2024-01-15T18:17:57.861 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:17:57.864 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:17:58.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.010+0000 7fce90400640 1 -- 172.21.15.27:0/674254567 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fce8814c020 con 0x7fce600677c0 2024-01-15T18:17:58.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.058+0000 7fce85600640 1 -- 172.21.15.27:0/674254567 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fce8814c020 con 0x7fce600677c0 2024-01-15T18:17:58.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.062+0000 7fce90400640 1 -- 172.21.15.27:0/674254567 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce600677c0 msgr2=0x7fce60069c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:58.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.062+0000 7fce90400640 1 --2- 172.21.15.27:0/674254567 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce600677c0 0x7fce60069c80 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fce78000f70 tx=0x7fce78001e10 comp rx=0 tx=0).stop 2024-01-15T18:17:58.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.062+0000 7fce90400640 1 -- 172.21.15.27:0/674254567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce8815d570 msgr2=0x7fce880b5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:58.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.062+0000 7fce90400640 1 --2- 172.21.15.27:0/674254567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce8815d570 0x7fce880b5ff0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fce8000c7a0 tx=0x7fce8000cc60 comp rx=0 tx=0).stop 2024-01-15T18:17:58.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.062+0000 7fce90400640 1 -- 172.21.15.27:0/674254567 shutdown_connections 2024-01-15T18:17:58.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.062+0000 7fce90400640 1 --2- 172.21.15.27:0/674254567 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce600677c0 0x7fce60069c80 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:58.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.062+0000 7fce90400640 1 --2- 172.21.15.27:0/674254567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce8815d570 0x7fce880b5ff0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:58.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.062+0000 7fce90400640 1 --2- 172.21.15.27:0/674254567 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce8815d190 0x7fce880b5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:58.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.062+0000 7fce90400640 1 --2- 172.21.15.27:0/674254567 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce8806b810 0x7fce8814b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:58.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.062+0000 7fce90400640 1 -- 172.21.15.27:0/674254567 >> 172.21.15.27:0/674254567 conn(0x7fce880d0e70 msgr2=0x7fce880cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:58.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.062+0000 7fce90400640 1 -- 172.21.15.27:0/674254567 shutdown_connections 2024-01-15T18:17:58.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.062+0000 7fce90400640 1 -- 172.21.15.27:0/674254567 wait complete. 2024-01-15T18:17:58.064 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:17:58.067+0000 7f59f9c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:17:58.064 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:17:58.067+0000 7f59f9c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:17:58.064 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:17:58.067+0000 7f59f9c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:17:58.064 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:17:58.067+0000 7f59f9c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:17:58.064 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:17:58.067+0000 7f59f9c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:17:58.079 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:17:58.083+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:17:58.080 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:17:58.083+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:17:58.080 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:17:58.083+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:17:58.080 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:17:58.083+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:17:58.080 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:17:58.083+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:17:58.082 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-15T18:17:58.082 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:17:58.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.230+0000 7f2104a00640 1 Processor -- start 2024-01-15T18:17:58.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.230+0000 7f2104a00640 1 -- start start 2024-01-15T18:17:58.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.230+0000 7f2104a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f20fc0d0350 0x7f20fc167f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:58.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.230+0000 7f2104a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20fc0cebd0 con 0x7f20fc15d3e0 2024-01-15T18:17:58.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.230+0000 7f2104a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20fc0ceeb0 con 0x7f20fc0afc30 2024-01-15T18:17:58.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.230+0000 7f2104a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20fc0b39d0 con 0x7f20fc0d0350 2024-01-15T18:17:58.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.230+0000 7f2102000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f20fc15d3e0 0x7f20fc0cd130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58194/0 (socket says 172.21.15.27:58194) 2024-01-15T18:17:58.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.230+0000 7f2102000640 1 -- 172.21.15.27:0/2836424325 learned_addr learned my addr 172.21.15.27:0/2836424325 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:58.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.230+0000 7f2103400640 1 --2- 172.21.15.27:0/2836424325 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f20fc0d0350 0x7f20fc167f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:58.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.230+0000 7f2100c00640 1 -- 172.21.15.27:0/2836424325 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 718277328 0 0) 0x7f20fc0cebd0 con 0x7f20fc15d3e0 2024-01-15T18:17:58.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.230+0000 7f2100c00640 1 -- 172.21.15.27:0/2836424325 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20dc003530 con 0x7f20fc15d3e0 2024-01-15T18:17:58.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.230+0000 7f2100c00640 1 -- 172.21.15.27:0/2836424325 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612397474 0 0) 0x7f20fc0ceeb0 con 0x7f20fc0afc30 2024-01-15T18:17:58.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.230+0000 7f2100c00640 1 -- 172.21.15.27:0/2836424325 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20fc0cebd0 con 0x7f20fc0afc30 2024-01-15T18:17:58.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.230+0000 7f2103400640 1 -- 172.21.15.27:0/2836424325 >> v1:172.21.15.93:6789/0 conn(0x7f20fc0afc30 legacy=0x7f20fc071c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:58.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.230+0000 7f2103400640 1 -- 172.21.15.27:0/2836424325 >> v1:172.21.15.27:6789/0 conn(0x7f20fc15d3e0 legacy=0x7f20fc0cd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:58.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.230+0000 7f2103400640 1 -- 172.21.15.27:0/2836424325 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20fc0b0070 con 0x7f20fc0d0350 2024-01-15T18:17:58.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.234+0000 7f2103400640 1 --2- 172.21.15.27:0/2836424325 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f20fc0d0350 0x7f20fc167f90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f20f4009dc0 tx=0x7f20f401c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8e8eb6683cc3994 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:58.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.234+0000 7f2100c00640 1 -- 172.21.15.27:0/2836424325 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f40a59f0 con 0x7f20fc0d0350 2024-01-15T18:17:58.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.234+0000 7f2100c00640 1 -- 172.21.15.27:0/2836424325 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20f40a5b50 con 0x7f20fc0d0350 2024-01-15T18:17:58.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.234+0000 7f2100c00640 1 -- 172.21.15.27:0/2836424325 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f40ad440 con 0x7f20fc0d0350 2024-01-15T18:17:58.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.234+0000 7f2104a00640 1 -- 172.21.15.27:0/2836424325 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f20fc0d0350 msgr2=0x7f20fc167f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:58.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.234+0000 7f2104a00640 1 --2- 172.21.15.27:0/2836424325 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f20fc0d0350 0x7f20fc167f90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f20f4009dc0 tx=0x7f20f401c930 comp rx=0 tx=0).stop 2024-01-15T18:17:58.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.234+0000 7f2104a00640 1 -- 172.21.15.27:0/2836424325 shutdown_connections 2024-01-15T18:17:58.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.234+0000 7f2104a00640 1 --2- 172.21.15.27:0/2836424325 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f20fc0d0350 0x7f20fc167f90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:58.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.234+0000 7f2104a00640 1 -- 172.21.15.27:0/2836424325 >> 172.21.15.27:0/2836424325 conn(0x7f20fc099e10 msgr2=0x7f20fc0b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:58.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.234+0000 7f2104a00640 1 -- 172.21.15.27:0/2836424325 shutdown_connections 2024-01-15T18:17:58.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.234+0000 7f2104a00640 1 -- 172.21.15.27:0/2836424325 wait complete. 2024-01-15T18:17:58.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.238+0000 7f2104a00640 1 Processor -- start 2024-01-15T18:17:58.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.238+0000 7f2104a00640 1 -- start start 2024-01-15T18:17:58.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.238+0000 7f2104a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20fc0afc30 0x7f20fc14ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:58.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.238+0000 7f2104a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20fc0d0350 0x7f20fc0b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:58.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.238+0000 7f2104a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f20fc15d3e0 0x7f20fc148dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:58.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.238+0000 7f2104a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20fc0cc720 con 0x7f20fc0d0350 2024-01-15T18:17:58.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.238+0000 7f2104a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20fc0cc440 con 0x7f20fc0afc30 2024-01-15T18:17:58.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.238+0000 7f2104a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20fc0c9ad0 con 0x7f20fc15d3e0 2024-01-15T18:17:58.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.238+0000 7f2101600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20fc0d0350 0x7f20fc0b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:58.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.238+0000 7f2102000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20fc0afc30 0x7f20fc14ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:58.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.238+0000 7f2103400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f20fc15d3e0 0x7f20fc148dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:58.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.238+0000 7f2101600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20fc0d0350 0x7f20fc0b13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43244/0 (socket says 172.21.15.27:43244) 2024-01-15T18:17:58.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.238+0000 7f2101600640 1 -- 172.21.15.27:0/3616829301 learned_addr learned my addr 172.21.15.27:0/3616829301 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:58.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.238+0000 7f2101600640 1 -- 172.21.15.27:0/3616829301 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f20fc15d3e0 msgr2=0x7f20fc148dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:58.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.238+0000 7f2101600640 1 --2- 172.21.15.27:0/3616829301 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f20fc15d3e0 0x7f20fc148dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:58.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.238+0000 7f2101600640 1 -- 172.21.15.27:0/3616829301 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20fc0afc30 msgr2=0x7f20fc14ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:58.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.238+0000 7f2101600640 1 --2- 172.21.15.27:0/3616829301 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20fc0afc30 0x7f20fc14ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:58.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.238+0000 7f2101600640 1 -- 172.21.15.27:0/3616829301 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f4009a70 con 0x7f20fc0d0350 2024-01-15T18:17:58.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.238+0000 7f2103400640 1 --2- 172.21.15.27:0/3616829301 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f20fc15d3e0 0x7f20fc148dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:17:58.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.238+0000 7f2101600640 1 --2- 172.21.15.27:0/3616829301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20fc0d0350 0x7f20fc0b13e0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f20e400c470 tx=0x7f20e400c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:58.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.238+0000 7f20fa000640 1 -- 172.21.15.27:0/3616829301 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e4005480 con 0x7f20fc0d0350 2024-01-15T18:17:58.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.238+0000 7f20fa000640 1 -- 172.21.15.27:0/3616829301 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e4003600 con 0x7f20fc0d0350 2024-01-15T18:17:58.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.238+0000 7f20fa000640 1 -- 172.21.15.27:0/3616829301 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e40039b0 con 0x7f20fc0d0350 2024-01-15T18:17:58.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.238+0000 7f2104a00640 1 -- 172.21.15.27:0/3616829301 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20fc149300 con 0x7f20fc0d0350 2024-01-15T18:17:58.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.238+0000 7f2104a00640 1 -- 172.21.15.27:0/3616829301 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20fc0cbdf0 con 0x7f20fc0d0350 2024-01-15T18:17:58.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.242+0000 7f2104a00640 1 -- 172.21.15.27:0/3616829301 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20fc09e950 con 0x7f20fc0d0350 2024-01-15T18:17:58.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.242+0000 7f20fa000640 1 -- 172.21.15.27:0/3616829301 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 35) v1 ==== 87486+0+0 (secure 0 0 0) 0x7f20e4003b10 con 0x7f20fc0d0350 2024-01-15T18:17:58.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.242+0000 7f20fa000640 1 --2- 172.21.15.27:0/3616829301 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f20d4067790 0x7f20d4069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:58.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.242+0000 7f20fa000640 1 -- 172.21.15.27:0/3616829301 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f20e40826f0 con 0x7f20fc0d0350 2024-01-15T18:17:58.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.242+0000 7f2102000640 1 --2- 172.21.15.27:0/3616829301 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f20d4067790 0x7f20d4069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:58.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.242+0000 7f2102000640 1 --2- 172.21.15.27:0/3616829301 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f20d4067790 0x7f20d4069c50 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f20ec0011f0 tx=0x7f20ec007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:58.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.246+0000 7f20fa000640 1 -- 172.21.15.27:0/3616829301 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f20e4051090 con 0x7f20fc0d0350 2024-01-15T18:17:58.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.438+0000 7f2104a00640 1 -- 172.21.15.27:0/3616829301 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f20fc0cc860 con 0x7f20d4067790 2024-01-15T18:17:58.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.442+0000 7f20fa000640 1 -- 172.21.15.27:0/3616829301 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f20ec006b70 con 0x7f20d4067790 2024-01-15T18:17:58.443 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:17:58.443 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:17:58.443 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "pending", 2024-01-15T18:17:58.443 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:17:58.443 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:17:58.443 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:17:58.443 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:17:58.444 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:17:58.444 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:17:58.444 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:17:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.446+0000 7f2104a00640 1 -- 172.21.15.27:0/3616829301 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f20d4067790 msgr2=0x7f20d4069c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.446+0000 7f2104a00640 1 --2- 172.21.15.27:0/3616829301 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f20d4067790 0x7f20d4069c50 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f20ec0011f0 tx=0x7f20ec007d40 comp rx=0 tx=0).stop 2024-01-15T18:17:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.446+0000 7f2104a00640 1 -- 172.21.15.27:0/3616829301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20fc0d0350 msgr2=0x7f20fc0b13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.446+0000 7f2104a00640 1 --2- 172.21.15.27:0/3616829301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20fc0d0350 0x7f20fc0b13e0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f20e400c470 tx=0x7f20e400c930 comp rx=0 tx=0).stop 2024-01-15T18:17:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.446+0000 7f2104a00640 1 -- 172.21.15.27:0/3616829301 shutdown_connections 2024-01-15T18:17:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.446+0000 7f2104a00640 1 --2- 172.21.15.27:0/3616829301 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f20d4067790 0x7f20d4069c50 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.446+0000 7f2104a00640 1 --2- 172.21.15.27:0/3616829301 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f20fc15d3e0 0x7f20fc148dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.446+0000 7f2104a00640 1 --2- 172.21.15.27:0/3616829301 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20fc0d0350 0x7f20fc0b13e0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.446+0000 7f2104a00640 1 --2- 172.21.15.27:0/3616829301 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f20fc0afc30 0x7f20fc14ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.446+0000 7f2104a00640 1 -- 172.21.15.27:0/3616829301 >> 172.21.15.27:0/3616829301 conn(0x7f20fc099e10 msgr2=0x7f20fc0aeee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.446+0000 7f2104a00640 1 -- 172.21.15.27:0/3616829301 shutdown_connections 2024-01-15T18:17:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.446+0000 7f2104a00640 1 -- 172.21.15.27:0/3616829301 wait complete. 2024-01-15T18:17:58.465 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs ss1clone1 2024-01-15T18:17:58.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.610+0000 7f2230800640 1 Processor -- start 2024-01-15T18:17:58.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.610+0000 7f2230800640 1 -- start start 2024-01-15T18:17:58.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.610+0000 7f2230800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f22280cf460 0x7f22280cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:58.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.610+0000 7f2230800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22280cfd80 con 0x7f222815b4b0 2024-01-15T18:17:58.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.610+0000 7f2230800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22280d19a0 con 0x7f22280b2590 2024-01-15T18:17:58.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.610+0000 7f2230800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22280d1c80 con 0x7f22280cf460 2024-01-15T18:17:58.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.610+0000 7f222de00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f22280cf460 0x7f22280cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:58.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.610+0000 7f222d400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f222815b4b0 0x7f22280cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58202/0 (socket says 172.21.15.27:58202) 2024-01-15T18:17:58.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.610+0000 7f222d400640 1 -- 172.21.15.27:0/1785444870 learned_addr learned my addr 172.21.15.27:0/1785444870 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:17:58.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.610+0000 7f222de00640 1 -- 172.21.15.27:0/1785444870 >> v1:172.21.15.93:6789/0 conn(0x7f22280b2590 legacy=0x7f22280cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:58.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.610+0000 7f222de00640 1 -- 172.21.15.27:0/1785444870 >> v1:172.21.15.27:6789/0 conn(0x7f222815b4b0 legacy=0x7f22280cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:58.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.610+0000 7f222de00640 1 -- 172.21.15.27:0/1785444870 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22280b07c0 con 0x7f22280cf460 2024-01-15T18:17:58.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.614+0000 7f222de00640 1 --2- 172.21.15.27:0/1785444870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22280cf460 0x7f22280cf840 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f22180097c0 tx=0x7f221801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba2e4fff045568e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:58.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.614+0000 7f222ca00640 1 -- 172.21.15.27:0/1785444870 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22180ab680 con 0x7f22280cf460 2024-01-15T18:17:58.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.614+0000 7f222ca00640 1 -- 172.21.15.27:0/1785444870 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22180ab7e0 con 0x7f22280cf460 2024-01-15T18:17:58.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.614+0000 7f222ca00640 1 -- 172.21.15.27:0/1785444870 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22180abb70 con 0x7f22280cf460 2024-01-15T18:17:58.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.614+0000 7f2230800640 1 -- 172.21.15.27:0/1785444870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22280cf460 msgr2=0x7f22280cf840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:58.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.614+0000 7f2230800640 1 --2- 172.21.15.27:0/1785444870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22280cf460 0x7f22280cf840 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f22180097c0 tx=0x7f221801c480 comp rx=0 tx=0).stop 2024-01-15T18:17:58.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.614+0000 7f2230800640 1 -- 172.21.15.27:0/1785444870 shutdown_connections 2024-01-15T18:17:58.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.614+0000 7f2230800640 1 --2- 172.21.15.27:0/1785444870 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22280cf460 0x7f22280cf840 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:58.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.614+0000 7f2230800640 1 -- 172.21.15.27:0/1785444870 >> 172.21.15.27:0/1785444870 conn(0x7f22280d0e80 msgr2=0x7f22280d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:58.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.614+0000 7f2230800640 1 -- 172.21.15.27:0/1785444870 shutdown_connections 2024-01-15T18:17:58.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.614+0000 7f2230800640 1 -- 172.21.15.27:0/1785444870 wait complete. 2024-01-15T18:17:58.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.618+0000 7f2230800640 1 Processor -- start 2024-01-15T18:17:58.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.618+0000 7f2230800640 1 -- start start 2024-01-15T18:17:58.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.618+0000 7f2230800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22280b2590 0x7f2228154480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:58.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.618+0000 7f2230800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f22280cf460 0x7f22280aed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:58.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.618+0000 7f2230800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f222815b4b0 0x7f2228149810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:58.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.618+0000 7f2230800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22280c7f50 con 0x7f22280b2590 2024-01-15T18:17:58.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.618+0000 7f2230800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22280c8230 con 0x7f222815b4b0 2024-01-15T18:17:58.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.618+0000 7f2230800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22280c8510 con 0x7f22280cf460 2024-01-15T18:17:58.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.618+0000 7f222de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22280b2590 0x7f2228154480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:58.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.618+0000 7f222de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22280b2590 0x7f2228154480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43264/0 (socket says 172.21.15.27:43264) 2024-01-15T18:17:58.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.618+0000 7f222de00640 1 -- 172.21.15.27:0/3654514653 learned_addr learned my addr 172.21.15.27:0/3654514653 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:17:58.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.618+0000 7f222d400640 1 --2- 172.21.15.27:0/3654514653 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f22280cf460 0x7f22280aed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:58.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.618+0000 7f222e800640 1 --2- 172.21.15.27:0/3654514653 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f222815b4b0 0x7f2228149810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:58.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.618+0000 7f222de00640 1 -- 172.21.15.27:0/3654514653 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f22280cf460 msgr2=0x7f22280aed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:58.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.618+0000 7f222de00640 1 --2- 172.21.15.27:0/3654514653 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f22280cf460 0x7f22280aed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:58.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.618+0000 7f222de00640 1 -- 172.21.15.27:0/3654514653 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f222815b4b0 msgr2=0x7f2228149810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:58.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.618+0000 7f222de00640 1 --2- 172.21.15.27:0/3654514653 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f222815b4b0 0x7f2228149810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:58.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.618+0000 7f222de00640 1 -- 172.21.15.27:0/3654514653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2218009420 con 0x7f22280b2590 2024-01-15T18:17:58.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.618+0000 7f222de00640 1 --2- 172.21.15.27:0/3654514653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22280b2590 0x7f2228154480 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f22180ad040 tx=0x7f221801acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:58.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.618+0000 7f2226000640 1 -- 172.21.15.27:0/3654514653 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22180b5810 con 0x7f22280b2590 2024-01-15T18:17:58.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.618+0000 7f2226000640 1 -- 172.21.15.27:0/3654514653 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22180b59b0 con 0x7f22280b2590 2024-01-15T18:17:58.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.618+0000 7f2226000640 1 -- 172.21.15.27:0/3654514653 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22180b5d60 con 0x7f22280b2590 2024-01-15T18:17:58.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.618+0000 7f2230800640 1 -- 172.21.15.27:0/3654514653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22280c87f0 con 0x7f22280b2590 2024-01-15T18:17:58.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.618+0000 7f2230800640 1 -- 172.21.15.27:0/3654514653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22281549c0 con 0x7f22280b2590 2024-01-15T18:17:58.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.622+0000 7f2230800640 1 -- 172.21.15.27:0/3654514653 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f222804e850 con 0x7f22280b2590 2024-01-15T18:17:58.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.622+0000 7f2226000640 1 -- 172.21.15.27:0/3654514653 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 35) v1 ==== 87486+0+0 (secure 0 0 0) 0x7f22180c93f0 con 0x7f22280b2590 2024-01-15T18:17:58.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.622+0000 7f2226000640 1 --2- 172.21.15.27:0/3654514653 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2200067790 0x7f2200069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:58.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.622+0000 7f2226000640 1 -- 172.21.15.27:0/3654514653 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f221811e270 con 0x7f22280b2590 2024-01-15T18:17:58.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.622+0000 7f222d400640 1 --2- 172.21.15.27:0/3654514653 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2200067790 0x7f2200069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:58.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.622+0000 7f222d400640 1 --2- 172.21.15.27:0/3654514653 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2200067790 0x7f2200069c50 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f2210005890 tx=0x7f2210009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:58.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.626+0000 7f2226000640 1 -- 172.21.15.27:0/3654514653 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f22180eccd0 con 0x7f22280b2590 2024-01-15T18:17:58.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:58.818+0000 7f2230800640 1 -- 172.21.15.27:0/3654514653 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f22280cc840 con 0x7f2200067790 2024-01-15T18:17:59.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:17:59.146+0000 7f2226000640 1 -- 172.21.15.27:0/3654514653 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 36) v1 ==== 87588+0+0 (secure 0 0 0) 0x7f22180ed0a0 con 0x7f22280b2590 2024-01-15T18:18:12.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.289+0000 7f2226000640 1 -- 172.21.15.27:0/3654514653 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2210001d80 con 0x7f2200067790 2024-01-15T18:18:12.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.293+0000 7f2230800640 1 -- 172.21.15.27:0/3654514653 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2200067790 msgr2=0x7f2200069c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.293+0000 7f2230800640 1 --2- 172.21.15.27:0/3654514653 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2200067790 0x7f2200069c50 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f2210005890 tx=0x7f2210009040 comp rx=0 tx=0).stop 2024-01-15T18:18:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.293+0000 7f2230800640 1 -- 172.21.15.27:0/3654514653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22280b2590 msgr2=0x7f2228154480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.293+0000 7f2230800640 1 --2- 172.21.15.27:0/3654514653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22280b2590 0x7f2228154480 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f22180ad040 tx=0x7f221801acc0 comp rx=0 tx=0).stop 2024-01-15T18:18:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.293+0000 7f2230800640 1 -- 172.21.15.27:0/3654514653 shutdown_connections 2024-01-15T18:18:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.293+0000 7f2230800640 1 --2- 172.21.15.27:0/3654514653 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2200067790 0x7f2200069c50 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.293+0000 7f2230800640 1 --2- 172.21.15.27:0/3654514653 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f222815b4b0 0x7f2228149810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.293+0000 7f2230800640 1 --2- 172.21.15.27:0/3654514653 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f22280cf460 0x7f22280aed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.293+0000 7f2230800640 1 --2- 172.21.15.27:0/3654514653 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22280b2590 0x7f2228154480 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.293+0000 7f2230800640 1 -- 172.21.15.27:0/3654514653 >> 172.21.15.27:0/3654514653 conn(0x7f22280d0e80 msgr2=0x7f22280cc030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:12.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.293+0000 7f2230800640 1 -- 172.21.15.27:0/3654514653 shutdown_connections 2024-01-15T18:18:12.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.293+0000 7f2230800640 1 -- 172.21.15.27:0/3654514653 wait complete. 2024-01-15T18:18:12.315 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:18:12.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.461+0000 7f3d98600640 1 Processor -- start 2024-01-15T18:18:12.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.461+0000 7f3d98600640 1 -- start start 2024-01-15T18:18:12.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.465+0000 7f3d98600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f3d9015f550 0x7f3d900cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:12.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.465+0000 7f3d98600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3d900cf110 0x7f3d900701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:12.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.465+0000 7f3d98600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d900b1cb0 con 0x7f3d90158900 2024-01-15T18:18:12.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.465+0000 7f3d98600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d900cf580 con 0x7f3d900cf110 2024-01-15T18:18:12.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.465+0000 7f3d98600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d9015fa00 con 0x7f3d9015f550 2024-01-15T18:18:12.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.465+0000 7f3d96600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3d900cf110 0x7f3d900701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:12.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.465+0000 7f3d95c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3d90158900 0x7f3d9015bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50308/0 (socket says 172.21.15.27:50308) 2024-01-15T18:18:12.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.465+0000 7f3d95c00640 1 -- 172.21.15.27:0/3012413337 learned_addr learned my addr 172.21.15.27:0/3012413337 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:12.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.465+0000 7f3d95200640 1 --2- 172.21.15.27:0/3012413337 >> v2:172.21.15.93:3300/0 conn(0x7f3d9015f550 0x7f3d900cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:12.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.465+0000 7f3d96600640 1 -- 172.21.15.27:0/3012413337 >> v2:172.21.15.93:3300/0 conn(0x7f3d9015f550 msgr2=0x7f3d900cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:12.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.465+0000 7f3d96600640 1 --2- 172.21.15.27:0/3012413337 >> v2:172.21.15.93:3300/0 conn(0x7f3d9015f550 0x7f3d900cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:12.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.465+0000 7f3d96600640 1 -- 172.21.15.27:0/3012413337 >> v1:172.21.15.27:6789/0 conn(0x7f3d90158900 legacy=0x7f3d9015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:12.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.465+0000 7f3d8fe00640 1 -- 172.21.15.27:0/3012413337 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1154955046 0 0) 0x7f3d900b1cb0 con 0x7f3d90158900 2024-01-15T18:18:12.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.465+0000 7f3d96600640 1 -- 172.21.15.27:0/3012413337 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d900b1f00 con 0x7f3d900cf110 2024-01-15T18:18:12.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.465+0000 7f3d96600640 1 --2- 172.21.15.27:0/3012413337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d900cf110 0x7f3d900701e0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f3d88009d90 tx=0x7f3d8801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48cab6ee4635fc0f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:12.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.465+0000 7f3d8fe00640 1 -- 172.21.15.27:0/3012413337 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d90070f30 con 0x7f3d900cf110 2024-01-15T18:18:12.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.465+0000 7f3d8fe00640 1 -- 172.21.15.27:0/3012413337 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d880a59a0 con 0x7f3d900cf110 2024-01-15T18:18:12.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.469+0000 7f3d8fe00640 1 -- 172.21.15.27:0/3012413337 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d880a5d30 con 0x7f3d900cf110 2024-01-15T18:18:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.469+0000 7f3d98600640 1 -- 172.21.15.27:0/3012413337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d900cf110 msgr2=0x7f3d900701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.469+0000 7f3d98600640 1 --2- 172.21.15.27:0/3012413337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d900cf110 0x7f3d900701e0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f3d88009d90 tx=0x7f3d8801c8e0 comp rx=0 tx=0).stop 2024-01-15T18:18:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.469+0000 7f3d98600640 1 -- 172.21.15.27:0/3012413337 shutdown_connections 2024-01-15T18:18:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.469+0000 7f3d98600640 1 --2- 172.21.15.27:0/3012413337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d900cf110 0x7f3d900701e0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.469+0000 7f3d98600640 1 --2- 172.21.15.27:0/3012413337 >> v2:172.21.15.93:3300/0 conn(0x7f3d9015f550 0x7f3d900cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.469+0000 7f3d98600640 1 -- 172.21.15.27:0/3012413337 >> 172.21.15.27:0/3012413337 conn(0x7f3d900cb700 msgr2=0x7f3d900cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.469+0000 7f3d98600640 1 -- 172.21.15.27:0/3012413337 shutdown_connections 2024-01-15T18:18:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.469+0000 7f3d98600640 1 -- 172.21.15.27:0/3012413337 wait complete. 2024-01-15T18:18:12.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.469+0000 7f3d98600640 1 Processor -- start 2024-01-15T18:18:12.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.469+0000 7f3d98600640 1 -- start start 2024-01-15T18:18:12.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.469+0000 7f3d98600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d900cf110 0x7f3d900af750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:12.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.469+0000 7f3d98600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3d90158900 0x7f3d901f5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:12.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.469+0000 7f3d98600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3d9015f550 0x7f3d900ce280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:12.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.469+0000 7f3d98600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d90151380 con 0x7f3d900cf110 2024-01-15T18:18:12.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.469+0000 7f3d98600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d901f5770 con 0x7f3d9015f550 2024-01-15T18:18:12.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.469+0000 7f3d98600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d900afd30 con 0x7f3d90158900 2024-01-15T18:18:12.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.473+0000 7f3d95c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d900cf110 0x7f3d900af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:12.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.473+0000 7f3d95c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d900cf110 0x7f3d900af750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37254/0 (socket says 172.21.15.27:37254) 2024-01-15T18:18:12.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.473+0000 7f3d95c00640 1 -- 172.21.15.27:0/2193088356 learned_addr learned my addr 172.21.15.27:0/2193088356 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:12.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.473+0000 7f3d95200640 1 --2- 172.21.15.27:0/2193088356 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3d90158900 0x7f3d901f5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:12.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.473+0000 7f3d96600640 1 --2- 172.21.15.27:0/2193088356 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3d9015f550 0x7f3d900ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:12.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.473+0000 7f3d95c00640 1 -- 172.21.15.27:0/2193088356 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3d90158900 msgr2=0x7f3d901f5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:12.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.473+0000 7f3d95c00640 1 --2- 172.21.15.27:0/2193088356 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3d90158900 0x7f3d901f5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:12.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.473+0000 7f3d95c00640 1 -- 172.21.15.27:0/2193088356 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3d9015f550 msgr2=0x7f3d900ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:12.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.473+0000 7f3d95c00640 1 --2- 172.21.15.27:0/2193088356 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3d9015f550 0x7f3d900ce280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:12.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.473+0000 7f3d95c00640 1 -- 172.21.15.27:0/2193088356 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d88009a70 con 0x7f3d900cf110 2024-01-15T18:18:12.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.473+0000 7f3d95c00640 1 --2- 172.21.15.27:0/2193088356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d900cf110 0x7f3d900af750 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f3d800011f0 tx=0x7f3d8000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:12.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.473+0000 7f3d8d600640 1 -- 172.21.15.27:0/2193088356 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d800074e0 con 0x7f3d900cf110 2024-01-15T18:18:12.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.473+0000 7f3d8d600640 1 -- 172.21.15.27:0/2193088356 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d80007640 con 0x7f3d900cf110 2024-01-15T18:18:12.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.473+0000 7f3d98600640 1 -- 172.21.15.27:0/2193088356 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d900b0010 con 0x7f3d900cf110 2024-01-15T18:18:12.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.473+0000 7f3d8d600640 1 -- 172.21.15.27:0/2193088356 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d80004a10 con 0x7f3d900cf110 2024-01-15T18:18:12.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.473+0000 7f3d98600640 1 -- 172.21.15.27:0/2193088356 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d900ce7c0 con 0x7f3d900cf110 2024-01-15T18:18:12.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.473+0000 7f3d98600640 1 -- 172.21.15.27:0/2193088356 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d9004e850 con 0x7f3d900cf110 2024-01-15T18:18:12.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.473+0000 7f3d8d600640 1 -- 172.21.15.27:0/2193088356 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 36) v1 ==== 87588+0+0 (secure 0 0 0) 0x7f3d80004b70 con 0x7f3d900cf110 2024-01-15T18:18:12.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.477+0000 7f3d8d600640 1 --2- 172.21.15.27:0/2193088356 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3d680678b0 0x7f3d68069d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:12.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.477+0000 7f3d8d600640 1 -- 172.21.15.27:0/2193088356 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f3d80082540 con 0x7f3d900cf110 2024-01-15T18:18:12.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.477+0000 7f3d95200640 1 --2- 172.21.15.27:0/2193088356 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3d680678b0 0x7f3d68069d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:12.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.477+0000 7f3d95200640 1 --2- 172.21.15.27:0/2193088356 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3d680678b0 0x7f3d68069d70 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f3d90150db0 tx=0x7f3d78009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:12.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.481+0000 7f3d8d600640 1 -- 172.21.15.27:0/2193088356 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3d80050ed0 con 0x7f3d900cf110 2024-01-15T18:18:12.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:12.669+0000 7f3d98600640 1 -- 172.21.15.27:0/2193088356 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d9015f030 con 0x7f3d680678b0 2024-01-15T18:18:13.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.301+0000 7f3d8d600640 1 -- 172.21.15.27:0/2193088356 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f3d780119d0 con 0x7f3d680678b0 2024-01-15T18:18:13.303 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:18:13.303 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:18:13.303 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "canceled", 2024-01-15T18:18:13.303 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:18:13.303 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:18:13.303 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:18:13.303 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:18:13.303 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:13.303 INFO:teuthology.orchestra.run.smithi027.stdout: "failure": { 2024-01-15T18:18:13.304 INFO:teuthology.orchestra.run.smithi027.stdout: "errno": "4", 2024-01-15T18:18:13.304 INFO:teuthology.orchestra.run.smithi027.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:18:13.304 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:13.304 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:13.304 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:18:13.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.305+0000 7f3d98600640 1 -- 172.21.15.27:0/2193088356 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3d680678b0 msgr2=0x7f3d68069d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:13.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.305+0000 7f3d98600640 1 --2- 172.21.15.27:0/2193088356 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3d680678b0 0x7f3d68069d70 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f3d90150db0 tx=0x7f3d78009190 comp rx=0 tx=0).stop 2024-01-15T18:18:13.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.305+0000 7f3d98600640 1 -- 172.21.15.27:0/2193088356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d900cf110 msgr2=0x7f3d900af750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:13.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.305+0000 7f3d98600640 1 --2- 172.21.15.27:0/2193088356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d900cf110 0x7f3d900af750 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f3d800011f0 tx=0x7f3d8000c9b0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.305+0000 7f3d98600640 1 -- 172.21.15.27:0/2193088356 shutdown_connections 2024-01-15T18:18:13.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.305+0000 7f3d98600640 1 --2- 172.21.15.27:0/2193088356 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3d680678b0 0x7f3d68069d70 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.305+0000 7f3d98600640 1 --2- 172.21.15.27:0/2193088356 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3d9015f550 0x7f3d900ce280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.305+0000 7f3d98600640 1 --2- 172.21.15.27:0/2193088356 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3d90158900 0x7f3d901f5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.305+0000 7f3d98600640 1 --2- 172.21.15.27:0/2193088356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d900cf110 0x7f3d900af750 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.305+0000 7f3d98600640 1 -- 172.21.15.27:0/2193088356 >> 172.21.15.27:0/2193088356 conn(0x7f3d900cb700 msgr2=0x7f3d900b3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:13.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.305+0000 7f3d98600640 1 -- 172.21.15.27:0/2193088356 shutdown_connections 2024-01-15T18:18:13.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.305+0000 7f3d98600640 1 -- 172.21.15.27:0/2193088356 wait complete. 2024-01-15T18:18:13.326 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:18:13.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.465+0000 7fbc92a00640 1 Processor -- start 2024-01-15T18:18:13.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.465+0000 7fbc92a00640 1 -- start start 2024-01-15T18:18:13.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.465+0000 7fbc92a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc8c159bf0 0x7fbc8c159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:13.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.465+0000 7fbc92a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbc8c071e40 0x7fbc8c0c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:13.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.465+0000 7fbc92a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fbc8c06d430 0x7fbc8c15d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:13.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.465+0000 7fbc92a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc8c0cce00 con 0x7fbc8c071e40 2024-01-15T18:18:13.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.465+0000 7fbc92a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc8c159a50 con 0x7fbc8c06d430 2024-01-15T18:18:13.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.465+0000 7fbc92a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c15a510 con 0x7fbc8c159bf0 2024-01-15T18:18:13.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.465+0000 7fbc8b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbc8c071e40 0x7fbc8c0c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:13.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.469+0000 7fbc8b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbc8c071e40 0x7fbc8c0c9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37266/0 (socket says 172.21.15.27:37266) 2024-01-15T18:18:13.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.469+0000 7fbc8b400640 1 -- 172.21.15.27:0/1649545564 learned_addr learned my addr 172.21.15.27:0/1649545564 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:13.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.469+0000 7fbc90a00640 1 --2- 172.21.15.27:0/1649545564 >> v2:172.21.15.93:3300/0 conn(0x7fbc8c06d430 0x7fbc8c15d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:13.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.469+0000 7fbc8be00640 1 --2- 172.21.15.27:0/1649545564 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc8c159bf0 0x7fbc8c159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:13.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.469+0000 7fbc8b400640 1 -- 172.21.15.27:0/1649545564 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc8c159bf0 msgr2=0x7fbc8c159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:13.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.469+0000 7fbc8b400640 1 --2- 172.21.15.27:0/1649545564 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc8c159bf0 0x7fbc8c159fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.469+0000 7fbc8b400640 1 -- 172.21.15.27:0/1649545564 >> v2:172.21.15.93:3300/0 conn(0x7fbc8c06d430 msgr2=0x7fbc8c15d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:13.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.469+0000 7fbc8b400640 1 --2- 172.21.15.27:0/1649545564 >> v2:172.21.15.93:3300/0 conn(0x7fbc8c06d430 0x7fbc8c15d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.469+0000 7fbc8b400640 1 -- 172.21.15.27:0/1649545564 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc8c06d980 con 0x7fbc8c071e40 2024-01-15T18:18:13.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.469+0000 7fbc90a00640 1 --2- 172.21.15.27:0/1649545564 >> v2:172.21.15.93:3300/0 conn(0x7fbc8c06d430 0x7fbc8c15d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:18:13.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.469+0000 7fbc8be00640 1 --2- 172.21.15.27:0/1649545564 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc8c159bf0 0x7fbc8c159fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:18:13.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.469+0000 7fbc8b400640 1 --2- 172.21.15.27:0/1649545564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc8c071e40 0x7fbc8c0c9be0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c0cfa80 tx=0x7fbc740200c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e885813d87c7539 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:13.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.469+0000 7fbc8aa00640 1 -- 172.21.15.27:0/1649545564 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc740a9e90 con 0x7fbc8c071e40 2024-01-15T18:18:13.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.469+0000 7fbc8aa00640 1 -- 172.21.15.27:0/1649545564 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc740b1440 con 0x7fbc8c071e40 2024-01-15T18:18:13.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.469+0000 7fbc8aa00640 1 -- 172.21.15.27:0/1649545564 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc740b17d0 con 0x7fbc8c071e40 2024-01-15T18:18:13.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.473+0000 7fbc92a00640 1 -- 172.21.15.27:0/1649545564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc8c071e40 msgr2=0x7fbc8c0c9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:13.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.473+0000 7fbc92a00640 1 --2- 172.21.15.27:0/1649545564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc8c071e40 0x7fbc8c0c9be0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c0cfa80 tx=0x7fbc740200c0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.473+0000 7fbc92a00640 1 -- 172.21.15.27:0/1649545564 shutdown_connections 2024-01-15T18:18:13.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.473+0000 7fbc92a00640 1 --2- 172.21.15.27:0/1649545564 >> v2:172.21.15.93:3300/0 conn(0x7fbc8c06d430 0x7fbc8c15d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.473+0000 7fbc92a00640 1 --2- 172.21.15.27:0/1649545564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc8c071e40 0x7fbc8c0c9be0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.473+0000 7fbc92a00640 1 --2- 172.21.15.27:0/1649545564 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc8c159bf0 0x7fbc8c159fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.473+0000 7fbc92a00640 1 -- 172.21.15.27:0/1649545564 >> 172.21.15.27:0/1649545564 conn(0x7fbc8c0cec60 msgr2=0x7fbc8c0cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:13.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.473+0000 7fbc92a00640 1 -- 172.21.15.27:0/1649545564 shutdown_connections 2024-01-15T18:18:13.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.473+0000 7fbc92a00640 1 -- 172.21.15.27:0/1649545564 wait complete. 2024-01-15T18:18:13.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.473+0000 7fbc92a00640 1 Processor -- start 2024-01-15T18:18:13.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.473+0000 7fbc92a00640 1 -- start start 2024-01-15T18:18:13.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.473+0000 7fbc92a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc8c06d430 0x7fbc8c150cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:13.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.473+0000 7fbc92a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc8c159bf0 0x7fbc8c151230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:13.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.473+0000 7fbc92a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc8c0b21c0 0x7fbc8c1f6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:13.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.473+0000 7fbc92a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c0b2780 con 0x7fbc8c06d430 2024-01-15T18:18:13.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.473+0000 7fbc92a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c0b2a60 con 0x7fbc8c159bf0 2024-01-15T18:18:13.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.473+0000 7fbc92a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8c0b2d40 con 0x7fbc8c0b21c0 2024-01-15T18:18:13.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.473+0000 7fbc8be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc8c06d430 0x7fbc8c150cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:13.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.473+0000 7fbc8be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc8c06d430 0x7fbc8c150cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37274/0 (socket says 172.21.15.27:37274) 2024-01-15T18:18:13.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.473+0000 7fbc8be00640 1 -- 172.21.15.27:0/2981583297 learned_addr learned my addr 172.21.15.27:0/2981583297 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:13.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.473+0000 7fbc8b400640 1 --2- 172.21.15.27:0/2981583297 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc8c159bf0 0x7fbc8c151230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:13.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.473+0000 7fbc90a00640 1 --2- 172.21.15.27:0/2981583297 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc8c0b21c0 0x7fbc8c1f6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:13.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.477+0000 7fbc8be00640 1 -- 172.21.15.27:0/2981583297 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc8c0b21c0 msgr2=0x7fbc8c1f6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:13.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.477+0000 7fbc8be00640 1 --2- 172.21.15.27:0/2981583297 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc8c0b21c0 0x7fbc8c1f6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.477+0000 7fbc8be00640 1 -- 172.21.15.27:0/2981583297 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc8c159bf0 msgr2=0x7fbc8c151230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:13.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.477+0000 7fbc8be00640 1 --2- 172.21.15.27:0/2981583297 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc8c159bf0 0x7fbc8c151230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.477+0000 7fbc8be00640 1 -- 172.21.15.27:0/2981583297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc7400a670 con 0x7fbc8c06d430 2024-01-15T18:18:13.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.477+0000 7fbc8be00640 1 --2- 172.21.15.27:0/2981583297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc8c06d430 0x7fbc8c150cf0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c00c7f0 tx=0x7fbc7c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:13.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.477+0000 7fbc83e00640 1 -- 172.21.15.27:0/2981583297 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c007820 con 0x7fbc8c06d430 2024-01-15T18:18:13.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.477+0000 7fbc83e00640 1 -- 172.21.15.27:0/2981583297 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc7c007980 con 0x7fbc8c06d430 2024-01-15T18:18:13.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.477+0000 7fbc83e00640 1 -- 172.21.15.27:0/2981583297 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c007d30 con 0x7fbc8c06d430 2024-01-15T18:18:13.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.477+0000 7fbc92a00640 1 -- 172.21.15.27:0/2981583297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc8c0b3020 con 0x7fbc8c06d430 2024-01-15T18:18:13.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.477+0000 7fbc92a00640 1 -- 172.21.15.27:0/2981583297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc8c1519b0 con 0x7fbc8c06d430 2024-01-15T18:18:13.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.477+0000 7fbc92a00640 1 -- 172.21.15.27:0/2981583297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc8c04e850 con 0x7fbc8c06d430 2024-01-15T18:18:13.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.477+0000 7fbc83e00640 1 -- 172.21.15.27:0/2981583297 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 36) v1 ==== 87588+0+0 (secure 0 0 0) 0x7fbc7c0113f0 con 0x7fbc8c06d430 2024-01-15T18:18:13.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.477+0000 7fbc83e00640 1 --2- 172.21.15.27:0/2981583297 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbc640678e0 0x7fbc64069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:13.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.481+0000 7fbc83e00640 1 -- 172.21.15.27:0/2981583297 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fbc7c083550 con 0x7fbc8c06d430 2024-01-15T18:18:13.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.481+0000 7fbc8b400640 1 --2- 172.21.15.27:0/2981583297 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbc640678e0 0x7fbc64069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:13.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.481+0000 7fbc8b400640 1 --2- 172.21.15.27:0/2981583297 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbc640678e0 0x7fbc64069da0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fbc740b3000 tx=0x7fbc74008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:13.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.485+0000 7fbc83e00640 1 -- 172.21.15.27:0/2981583297 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbc7c051e40 con 0x7fbc8c06d430 2024-01-15T18:18:13.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.693+0000 7fbc92a00640 1 -- 172.21.15.27:0/2981583297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbc8c0cc890 con 0x7fbc8c06d430 2024-01-15T18:18:13.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.693+0000 7fbc83e00640 1 -- 172.21.15.27:0/2981583297 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1856 (secure 0 0 0) 0x7fbc7c01f090 con 0x7fbc8c06d430 2024-01-15T18:18:13.693 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:18:13.694 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:18:13.694 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:18:13.694 INFO:teuthology.orchestra.run.smithi027.stdout: "health": { 2024-01-15T18:18:13.694 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "HEALTH_OK", 2024-01-15T18:18:13.694 INFO:teuthology.orchestra.run.smithi027.stdout: "checks": {}, 2024-01-15T18:18:13.694 INFO:teuthology.orchestra.run.smithi027.stdout: "mutes": [] 2024-01-15T18:18:13.694 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:13.694 INFO:teuthology.orchestra.run.smithi027.stdout: "election_epoch": 4, 2024-01-15T18:18:13.694 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum": [ 2024-01-15T18:18:13.694 INFO:teuthology.orchestra.run.smithi027.stdout: 0, 2024-01-15T18:18:13.694 INFO:teuthology.orchestra.run.smithi027.stdout: 1, 2024-01-15T18:18:13.695 INFO:teuthology.orchestra.run.smithi027.stdout: 2 2024-01-15T18:18:13.695 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:18:13.695 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_names": [ 2024-01-15T18:18:13.695 INFO:teuthology.orchestra.run.smithi027.stdout: "a", 2024-01-15T18:18:13.695 INFO:teuthology.orchestra.run.smithi027.stdout: "b", 2024-01-15T18:18:13.695 INFO:teuthology.orchestra.run.smithi027.stdout: "c" 2024-01-15T18:18:13.695 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:18:13.695 INFO:teuthology.orchestra.run.smithi027.stdout: "quorum_age": 820, 2024-01-15T18:18:13.695 INFO:teuthology.orchestra.run.smithi027.stdout: "monmap": { 2024-01-15T18:18:13.695 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 1, 2024-01-15T18:18:13.695 INFO:teuthology.orchestra.run.smithi027.stdout: "min_mon_release_name": "reef", 2024-01-15T18:18:13.695 INFO:teuthology.orchestra.run.smithi027.stdout: "num_mons": 3 2024-01-15T18:18:13.696 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:13.696 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap": { 2024-01-15T18:18:13.696 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 97, 2024-01-15T18:18:13.696 INFO:teuthology.orchestra.run.smithi027.stdout: "num_osds": 8, 2024-01-15T18:18:13.696 INFO:teuthology.orchestra.run.smithi027.stdout: "num_up_osds": 8, 2024-01-15T18:18:13.696 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_up_since": 1705341884, 2024-01-15T18:18:13.696 INFO:teuthology.orchestra.run.smithi027.stdout: "num_in_osds": 8, 2024-01-15T18:18:13.696 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_in_since": 1705341875, 2024-01-15T18:18:13.696 INFO:teuthology.orchestra.run.smithi027.stdout: "num_remapped_pgs": 0 2024-01-15T18:18:13.696 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:13.696 INFO:teuthology.orchestra.run.smithi027.stdout: "pgmap": { 2024-01-15T18:18:13.696 INFO:teuthology.orchestra.run.smithi027.stdout: "pgs_by_state": [ 2024-01-15T18:18:13.697 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:13.697 INFO:teuthology.orchestra.run.smithi027.stdout: "state_name": "active+clean", 2024-01-15T18:18:13.697 INFO:teuthology.orchestra.run.smithi027.stdout: "count": 97 2024-01-15T18:18:13.697 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:13.697 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:18:13.697 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pgs": 97, 2024-01-15T18:18:13.697 INFO:teuthology.orchestra.run.smithi027.stdout: "num_pools": 3, 2024-01-15T18:18:13.697 INFO:teuthology.orchestra.run.smithi027.stdout: "num_objects": 797, 2024-01-15T18:18:13.697 INFO:teuthology.orchestra.run.smithi027.stdout: "data_bytes": 3222080388, 2024-01-15T18:18:13.697 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_used": 6973169664, 2024-01-15T18:18:13.697 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_avail": 766120943616, 2024-01-15T18:18:13.697 INFO:teuthology.orchestra.run.smithi027.stdout: "bytes_total": 773094113280, 2024-01-15T18:18:13.697 INFO:teuthology.orchestra.run.smithi027.stdout: "read_bytes_sec": 170, 2024-01-15T18:18:13.697 INFO:teuthology.orchestra.run.smithi027.stdout: "write_bytes_sec": 13630863, 2024-01-15T18:18:13.698 INFO:teuthology.orchestra.run.smithi027.stdout: "read_op_per_sec": 0, 2024-01-15T18:18:13.698 INFO:teuthology.orchestra.run.smithi027.stdout: "write_op_per_sec": 4 2024-01-15T18:18:13.698 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:13.698 INFO:teuthology.orchestra.run.smithi027.stdout: "fsmap": { 2024-01-15T18:18:13.698 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 68, 2024-01-15T18:18:13.698 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 7, 2024-01-15T18:18:13.698 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-01-15T18:18:13.698 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-01-15T18:18:13.698 INFO:teuthology.orchestra.run.smithi027.stdout: "max": 1, 2024-01-15T18:18:13.698 INFO:teuthology.orchestra.run.smithi027.stdout: "by_rank": [ 2024-01-15T18:18:13.698 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:13.698 INFO:teuthology.orchestra.run.smithi027.stdout: "filesystem_id": 7, 2024-01-15T18:18:13.699 INFO:teuthology.orchestra.run.smithi027.stdout: "rank": 0, 2024-01-15T18:18:13.699 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "a", 2024-01-15T18:18:13.699 INFO:teuthology.orchestra.run.smithi027.stdout: "status": "up:active", 2024-01-15T18:18:13.699 INFO:teuthology.orchestra.run.smithi027.stdout: "gid": 8937 2024-01-15T18:18:13.699 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:13.699 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:18:13.699 INFO:teuthology.orchestra.run.smithi027.stdout: "up:standby": 3 2024-01-15T18:18:13.699 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:13.699 INFO:teuthology.orchestra.run.smithi027.stdout: "mgrmap": { 2024-01-15T18:18:13.699 INFO:teuthology.orchestra.run.smithi027.stdout: "available": true, 2024-01-15T18:18:13.699 INFO:teuthology.orchestra.run.smithi027.stdout: "num_standbys": 1, 2024-01-15T18:18:13.699 INFO:teuthology.orchestra.run.smithi027.stdout: "modules": [ 2024-01-15T18:18:13.699 INFO:teuthology.orchestra.run.smithi027.stdout: "iostat", 2024-01-15T18:18:13.700 INFO:teuthology.orchestra.run.smithi027.stdout: "nfs", 2024-01-15T18:18:13.700 INFO:teuthology.orchestra.run.smithi027.stdout: "restful" 2024-01-15T18:18:13.700 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-01-15T18:18:13.700 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:18:13.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:13.700 INFO:teuthology.orchestra.run.smithi027.stdout: "servicemap": { 2024-01-15T18:18:13.700 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 18, 2024-01-15T18:18:13.700 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-01-15T18:16:52.347508+0000", 2024-01-15T18:18:13.700 INFO:teuthology.orchestra.run.smithi027.stdout: "services": {} 2024-01-15T18:18:13.700 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:13.700 INFO:teuthology.orchestra.run.smithi027.stdout: "progress_events": {} 2024-01-15T18:18:13.700 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:18:13.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.693+0000 7fbc92a00640 1 -- 172.21.15.27:0/2981583297 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbc640678e0 msgr2=0x7fbc64069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:13.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.693+0000 7fbc92a00640 1 --2- 172.21.15.27:0/2981583297 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbc640678e0 0x7fbc64069da0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fbc740b3000 tx=0x7fbc74008040 comp rx=0 tx=0).stop 2024-01-15T18:18:13.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.693+0000 7fbc92a00640 1 -- 172.21.15.27:0/2981583297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc8c06d430 msgr2=0x7fbc8c150cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:13.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.693+0000 7fbc92a00640 1 --2- 172.21.15.27:0/2981583297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc8c06d430 0x7fbc8c150cf0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c00c7f0 tx=0x7fbc7c00ccb0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.697+0000 7fbc92a00640 1 -- 172.21.15.27:0/2981583297 shutdown_connections 2024-01-15T18:18:13.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.697+0000 7fbc92a00640 1 --2- 172.21.15.27:0/2981583297 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbc640678e0 0x7fbc64069da0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.697+0000 7fbc92a00640 1 --2- 172.21.15.27:0/2981583297 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc8c0b21c0 0x7fbc8c1f6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.697+0000 7fbc92a00640 1 --2- 172.21.15.27:0/2981583297 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc8c159bf0 0x7fbc8c151230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.697+0000 7fbc92a00640 1 --2- 172.21.15.27:0/2981583297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc8c06d430 0x7fbc8c150cf0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.697+0000 7fbc92a00640 1 -- 172.21.15.27:0/2981583297 >> 172.21.15.27:0/2981583297 conn(0x7fbc8c0cec60 msgr2=0x7fbc8c158d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:13.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.697+0000 7fbc92a00640 1 -- 172.21.15.27:0/2981583297 shutdown_connections 2024-01-15T18:18:13.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.697+0000 7fbc92a00640 1 -- 172.21.15.27:0/2981583297 wait complete. 2024-01-15T18:18:13.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:13.716 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -lh volumes/_nogroup/ss1clone1/6c8b5da2-f8d6-43d8-942c-f8573173e9a0) 2024-01-15T18:18:13.728 INFO:teuthology.orchestra.run.smithi027.stderr:ls: cannot access 'volumes/_nogroup/ss1clone1/6c8b5da2-f8d6-43d8-942c-f8573173e9a0': No such file or directory 2024-01-15T18:18:13.729 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-15T18:18:13.730 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:18:13.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.917+0000 7fbf6e000640 1 Processor -- start 2024-01-15T18:18:13.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.917+0000 7fbf6e000640 1 -- start start 2024-01-15T18:18:13.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.917+0000 7fbf6e000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbf6815d7c0 0x7fbf680b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:13.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.917+0000 7fbf6e000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fbf6809a120 0x7fbf680cf180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:13.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.917+0000 7fbf6e000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf680b1aa0 con 0x7fbf680b1d90 2024-01-15T18:18:13.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.917+0000 7fbf6e000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf680b1300 con 0x7fbf6815d7c0 2024-01-15T18:18:13.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.917+0000 7fbf6e000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf680b25a0 con 0x7fbf6809a120 2024-01-15T18:18:13.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.917+0000 7fbf67400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbf6815d7c0 0x7fbf680b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:13.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.917+0000 7fbf66a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbf680b1d90 0x7fbf68099a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44184/0 (socket says 172.21.15.27:44184) 2024-01-15T18:18:13.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.917+0000 7fbf66a00640 1 -- 172.21.15.27:0/4051952831 learned_addr learned my addr 172.21.15.27:0/4051952831 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:13.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.917+0000 7fbf67400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbf6815d7c0 0x7fbf680b35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33662/0 (socket says 172.21.15.27:33662) 2024-01-15T18:18:13.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.917+0000 7fbf67e00640 1 --2- 172.21.15.27:0/4051952831 >> v2:172.21.15.93:3300/0 conn(0x7fbf6809a120 0x7fbf680cf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:13.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.917+0000 7fbf67400640 1 -- 172.21.15.27:0/4051952831 >> v2:172.21.15.93:3300/0 conn(0x7fbf6809a120 msgr2=0x7fbf680cf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:13.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.917+0000 7fbf67400640 1 --2- 172.21.15.27:0/4051952831 >> v2:172.21.15.93:3300/0 conn(0x7fbf6809a120 0x7fbf680cf180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.917+0000 7fbf67400640 1 -- 172.21.15.27:0/4051952831 >> v1:172.21.15.27:6789/0 conn(0x7fbf680b1d90 legacy=0x7fbf68099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:13.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.917+0000 7fbf67400640 1 -- 172.21.15.27:0/4051952831 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf680b14e0 con 0x7fbf6815d7c0 2024-01-15T18:18:13.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.921+0000 7fbf67400640 1 --2- 172.21.15.27:0/4051952831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6815d7c0 0x7fbf680b35a0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fbf6806e450 tx=0x7fbf5801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57627eb92952ee5b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:13.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.921+0000 7fbf66000640 1 -- 172.21.15.27:0/4051952831 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf580ab600 con 0x7fbf6815d7c0 2024-01-15T18:18:13.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.921+0000 7fbf66000640 1 -- 172.21.15.27:0/4051952831 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf580a7050 con 0x7fbf6815d7c0 2024-01-15T18:18:13.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.921+0000 7fbf66000640 1 -- 172.21.15.27:0/4051952831 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf580ab980 con 0x7fbf6815d7c0 2024-01-15T18:18:13.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.921+0000 7fbf6e000640 1 -- 172.21.15.27:0/4051952831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6815d7c0 msgr2=0x7fbf680b35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:13.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.921+0000 7fbf6e000640 1 --2- 172.21.15.27:0/4051952831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6815d7c0 0x7fbf680b35a0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fbf6806e450 tx=0x7fbf5801c3d0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.921+0000 7fbf6e000640 1 -- 172.21.15.27:0/4051952831 shutdown_connections 2024-01-15T18:18:13.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.921+0000 7fbf6e000640 1 --2- 172.21.15.27:0/4051952831 >> v2:172.21.15.93:3300/0 conn(0x7fbf6809a120 0x7fbf680cf180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.921+0000 7fbf6e000640 1 --2- 172.21.15.27:0/4051952831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6815d7c0 0x7fbf680b35a0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.921+0000 7fbf6e000640 1 -- 172.21.15.27:0/4051952831 >> 172.21.15.27:0/4051952831 conn(0x7fbf680b2a80 msgr2=0x7fbf680b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:13.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.925+0000 7fbf6e000640 1 -- 172.21.15.27:0/4051952831 shutdown_connections 2024-01-15T18:18:13.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.925+0000 7fbf6e000640 1 -- 172.21.15.27:0/4051952831 wait complete. 2024-01-15T18:18:13.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.925+0000 7fbf6e000640 1 Processor -- start 2024-01-15T18:18:13.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.925+0000 7fbf6e000640 1 -- start start 2024-01-15T18:18:13.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.925+0000 7fbf6e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6809a120 0x7fbf68150f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:13.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.925+0000 7fbf6e000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf680b1d90 0x7fbf68151730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:13.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.925+0000 7fbf6e000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf6815d7c0 0x7fbf68151c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:13.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.925+0000 7fbf6e000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6815dc40 con 0x7fbf6809a120 2024-01-15T18:18:13.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.925+0000 7fbf6e000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6815df20 con 0x7fbf6815d7c0 2024-01-15T18:18:13.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.925+0000 7fbf6e000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf680c9ff0 con 0x7fbf680b1d90 2024-01-15T18:18:13.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.925+0000 7fbf67400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6809a120 0x7fbf68150f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:13.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.925+0000 7fbf67400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6809a120 0x7fbf68150f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33672/0 (socket says 172.21.15.27:33672) 2024-01-15T18:18:13.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.925+0000 7fbf67400640 1 -- 172.21.15.27:0/2881900383 learned_addr learned my addr 172.21.15.27:0/2881900383 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:13.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.925+0000 7fbf66a00640 1 --2- 172.21.15.27:0/2881900383 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf680b1d90 0x7fbf68151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:13.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.925+0000 7fbf67e00640 1 --2- 172.21.15.27:0/2881900383 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf6815d7c0 0x7fbf68151c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:13.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.925+0000 7fbf67400640 1 -- 172.21.15.27:0/2881900383 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf680b1d90 msgr2=0x7fbf68151730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:13.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.925+0000 7fbf67400640 1 --2- 172.21.15.27:0/2881900383 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf680b1d90 0x7fbf68151730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.925+0000 7fbf67400640 1 -- 172.21.15.27:0/2881900383 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf6815d7c0 msgr2=0x7fbf68151c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:13.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.925+0000 7fbf67400640 1 --2- 172.21.15.27:0/2881900383 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf6815d7c0 0x7fbf68151c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.925+0000 7fbf67400640 1 -- 172.21.15.27:0/2881900383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf58009420 con 0x7fbf6809a120 2024-01-15T18:18:13.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.925+0000 7fbf67400640 1 --2- 172.21.15.27:0/2881900383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6809a120 0x7fbf68150f90 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fbf5801a000 tx=0x7fbf5801ac10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:13.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.929+0000 7fbf5f400640 1 -- 172.21.15.27:0/2881900383 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf580b7450 con 0x7fbf6809a120 2024-01-15T18:18:13.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.929+0000 7fbf5f400640 1 -- 172.21.15.27:0/2881900383 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf580b75d0 con 0x7fbf6809a120 2024-01-15T18:18:13.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.929+0000 7fbf6e000640 1 -- 172.21.15.27:0/2881900383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf681522a0 con 0x7fbf6809a120 2024-01-15T18:18:13.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.929+0000 7fbf5f400640 1 -- 172.21.15.27:0/2881900383 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf580b7980 con 0x7fbf6809a120 2024-01-15T18:18:13.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.929+0000 7fbf6e000640 1 -- 172.21.15.27:0/2881900383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf68152860 con 0x7fbf6809a120 2024-01-15T18:18:13.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.929+0000 7fbf6e000640 1 -- 172.21.15.27:0/2881900383 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf6804e850 con 0x7fbf6809a120 2024-01-15T18:18:13.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.929+0000 7fbf5f400640 1 -- 172.21.15.27:0/2881900383 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 36) v1 ==== 87588+0+0 (secure 0 0 0) 0x7fbf580b7ae0 con 0x7fbf6809a120 2024-01-15T18:18:13.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.929+0000 7fbf5f400640 1 --2- 172.21.15.27:0/2881900383 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbf400678b0 0x7fbf40069d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:13.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.929+0000 7fbf5f400640 1 -- 172.21.15.27:0/2881900383 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fbf5811dac0 con 0x7fbf6809a120 2024-01-15T18:18:13.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.929+0000 7fbf66a00640 1 --2- 172.21.15.27:0/2881900383 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbf400678b0 0x7fbf40069d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:13.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.933+0000 7fbf66a00640 1 --2- 172.21.15.27:0/2881900383 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbf400678b0 0x7fbf40069d70 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fbf6809e730 tx=0x7fbf50009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:13.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:13.933+0000 7fbf5f400640 1 -- 172.21.15.27:0/2881900383 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbf580ec610 con 0x7fbf6809a120 2024-01-15T18:18:14.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.113+0000 7fbf6e000640 1 -- 172.21.15.27:0/2881900383 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf6815e060 con 0x7fbf400678b0 2024-01-15T18:18:14.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.117+0000 7fbf5f400640 1 -- 172.21.15.27:0/2881900383 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fbf50001490 con 0x7fbf400678b0 2024-01-15T18:18:14.117 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:18:14.117 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:14.117 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-15T18:18:14.117 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:14.117 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:18:14.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.117+0000 7fbf6e000640 1 -- 172.21.15.27:0/2881900383 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbf400678b0 msgr2=0x7fbf40069d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:14.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.117+0000 7fbf6e000640 1 --2- 172.21.15.27:0/2881900383 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbf400678b0 0x7fbf40069d70 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fbf6809e730 tx=0x7fbf50009040 comp rx=0 tx=0).stop 2024-01-15T18:18:14.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.117+0000 7fbf6e000640 1 -- 172.21.15.27:0/2881900383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6809a120 msgr2=0x7fbf68150f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:14.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.117+0000 7fbf6e000640 1 --2- 172.21.15.27:0/2881900383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6809a120 0x7fbf68150f90 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fbf5801a000 tx=0x7fbf5801ac10 comp rx=0 tx=0).stop 2024-01-15T18:18:14.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.117+0000 7fbf6e000640 1 -- 172.21.15.27:0/2881900383 shutdown_connections 2024-01-15T18:18:14.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.117+0000 7fbf6e000640 1 --2- 172.21.15.27:0/2881900383 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbf400678b0 0x7fbf40069d70 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:14.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.117+0000 7fbf6e000640 1 --2- 172.21.15.27:0/2881900383 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbf6815d7c0 0x7fbf68151c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:14.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.117+0000 7fbf6e000640 1 --2- 172.21.15.27:0/2881900383 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbf680b1d90 0x7fbf68151730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:14.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.117+0000 7fbf6e000640 1 --2- 172.21.15.27:0/2881900383 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf6809a120 0x7fbf68150f90 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:14.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.117+0000 7fbf6e000640 1 -- 172.21.15.27:0/2881900383 >> 172.21.15.27:0/2881900383 conn(0x7fbf680b2a80 msgr2=0x7fbf680cb450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:14.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.121+0000 7fbf6e000640 1 -- 172.21.15.27:0/2881900383 shutdown_connections 2024-01-15T18:18:14.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.121+0000 7fbf6e000640 1 -- 172.21.15.27:0/2881900383 wait complete. 2024-01-15T18:18:14.139 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-15T18:18:14.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.285+0000 7f2359800640 1 Processor -- start 2024-01-15T18:18:14.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.285+0000 7f2359800640 1 -- start start 2024-01-15T18:18:14.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.285+0000 7f2359800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f235415ad20 0x7f235415b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:14.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.285+0000 7f2359800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f23540715f0 0x7f235415ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:14.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.285+0000 7f2359800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f235415a610 con 0x7f235415d900 2024-01-15T18:18:14.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.285+0000 7f2359800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f235415e3c0 con 0x7f235415ad20 2024-01-15T18:18:14.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.285+0000 7f2359800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f235415e6a0 con 0x7f23540715f0 2024-01-15T18:18:14.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.285+0000 7f2353400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f23540715f0 0x7f235415ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:14.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.285+0000 7f2353400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f23540715f0 0x7f235415ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:49834/0 (socket says 172.21.15.27:49834) 2024-01-15T18:18:14.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.285+0000 7f2353400640 1 -- 172.21.15.27:0/3342857861 learned_addr learned my addr 172.21.15.27:0/3342857861 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:14.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.285+0000 7f2352a00640 1 --2- 172.21.15.27:0/3342857861 >> v2:172.21.15.93:3300/0 conn(0x7f235415ad20 0x7f235415b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:14.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.285+0000 7f2353400640 1 -- 172.21.15.27:0/3342857861 >> v2:172.21.15.93:3300/0 conn(0x7f235415ad20 msgr2=0x7f235415b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:14.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.285+0000 7f2353400640 1 --2- 172.21.15.27:0/3342857861 >> v2:172.21.15.93:3300/0 conn(0x7f235415ad20 0x7f235415b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:14.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.285+0000 7f2353400640 1 -- 172.21.15.27:0/3342857861 >> v1:172.21.15.93:6789/0 conn(0x7f235415d900 legacy=0x7f2354070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:14.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.285+0000 7f2353400640 1 -- 172.21.15.27:0/3342857861 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f235415b6e0 con 0x7f23540715f0 2024-01-15T18:18:14.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.289+0000 7f2353400640 1 --2- 172.21.15.27:0/3342857861 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f23540715f0 0x7f235415ddb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f234c00d180 tx=0x7f234c01ebc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8b8cc51d5fc84e8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:14.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.289+0000 7f2351600640 1 -- 172.21.15.27:0/3342857861 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f234c00b2d0 con 0x7f23540715f0 2024-01-15T18:18:14.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.289+0000 7f2351600640 1 -- 172.21.15.27:0/3342857861 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f234c00b430 con 0x7f23540715f0 2024-01-15T18:18:14.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.289+0000 7f2351600640 1 -- 172.21.15.27:0/3342857861 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f234c0a7bd0 con 0x7f23540715f0 2024-01-15T18:18:14.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.289+0000 7f2359800640 1 -- 172.21.15.27:0/3342857861 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f23540715f0 msgr2=0x7f235415ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:14.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.289+0000 7f2359800640 1 --2- 172.21.15.27:0/3342857861 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f23540715f0 0x7f235415ddb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f234c00d180 tx=0x7f234c01ebc0 comp rx=0 tx=0).stop 2024-01-15T18:18:14.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.289+0000 7f2359800640 1 -- 172.21.15.27:0/3342857861 shutdown_connections 2024-01-15T18:18:14.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.289+0000 7f2359800640 1 --2- 172.21.15.27:0/3342857861 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f23540715f0 0x7f235415ddb0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:14.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.289+0000 7f2359800640 1 --2- 172.21.15.27:0/3342857861 >> v2:172.21.15.93:3300/0 conn(0x7f235415ad20 0x7f235415b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:14.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.289+0000 7f2359800640 1 -- 172.21.15.27:0/3342857861 >> 172.21.15.27:0/3342857861 conn(0x7f23540cb700 msgr2=0x7f23540cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:14.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.289+0000 7f2359800640 1 -- 172.21.15.27:0/3342857861 shutdown_connections 2024-01-15T18:18:14.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.289+0000 7f2359800640 1 -- 172.21.15.27:0/3342857861 wait complete. 2024-01-15T18:18:14.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.293+0000 7f2359800640 1 Processor -- start 2024-01-15T18:18:14.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.293+0000 7f2359800640 1 -- start start 2024-01-15T18:18:14.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.293+0000 7f2359800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f23540715f0 0x7f235414dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:14.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.293+0000 7f2359800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f235415ad20 0x7f235414bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:14.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.293+0000 7f2359800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f235415d900 0x7f23540ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:14.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.293+0000 7f2359800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f235414c1c0 con 0x7f235415ad20 2024-01-15T18:18:14.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.293+0000 7f2359800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f235414cef0 con 0x7f235415d900 2024-01-15T18:18:14.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.293+0000 7f2359800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f235414d1d0 con 0x7f23540715f0 2024-01-15T18:18:14.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.293+0000 7f2352000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f235415ad20 0x7f235414bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:14.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.293+0000 7f2352000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f235415ad20 0x7f235414bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33684/0 (socket says 172.21.15.27:33684) 2024-01-15T18:18:14.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.293+0000 7f2352000640 1 -- 172.21.15.27:0/3440041086 learned_addr learned my addr 172.21.15.27:0/3440041086 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:14.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.293+0000 7f2353400640 1 --2- 172.21.15.27:0/3440041086 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f235415d900 0x7f23540ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:14.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.293+0000 7f2352a00640 1 --2- 172.21.15.27:0/3440041086 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f23540715f0 0x7f235414dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:14.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.293+0000 7f2352000640 1 -- 172.21.15.27:0/3440041086 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f23540715f0 msgr2=0x7f235414dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:14.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.293+0000 7f2352000640 1 --2- 172.21.15.27:0/3440041086 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f23540715f0 0x7f235414dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:14.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.293+0000 7f2352000640 1 -- 172.21.15.27:0/3440041086 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f235415d900 msgr2=0x7f23540ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:14.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.293+0000 7f2352000640 1 --2- 172.21.15.27:0/3440041086 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f235415d900 0x7f23540ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:14.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.293+0000 7f2352000640 1 -- 172.21.15.27:0/3440041086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f234c00bd40 con 0x7f235415ad20 2024-01-15T18:18:14.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.293+0000 7f2352000640 1 --2- 172.21.15.27:0/3440041086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f235415ad20 0x7f235414bb30 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f233c00c3c0 tx=0x7f233c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:14.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.293+0000 7f234aa00640 1 -- 172.21.15.27:0/3440041086 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233c0052d0 con 0x7f235415ad20 2024-01-15T18:18:14.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.293+0000 7f234aa00640 1 -- 172.21.15.27:0/3440041086 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f233c005430 con 0x7f235415ad20 2024-01-15T18:18:14.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.293+0000 7f2359800640 1 -- 172.21.15.27:0/3440041086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f235414d4b0 con 0x7f235415ad20 2024-01-15T18:18:14.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.293+0000 7f234aa00640 1 -- 172.21.15.27:0/3440041086 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233c003840 con 0x7f235415ad20 2024-01-15T18:18:14.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.293+0000 7f2359800640 1 -- 172.21.15.27:0/3440041086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f235414cc10 con 0x7f235415ad20 2024-01-15T18:18:14.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.297+0000 7f2359800640 1 -- 172.21.15.27:0/3440041086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f235414d9d0 con 0x7f235415ad20 2024-01-15T18:18:14.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.297+0000 7f234aa00640 1 -- 172.21.15.27:0/3440041086 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 36) v1 ==== 87588+0+0 (secure 0 0 0) 0x7f233c0039a0 con 0x7f235415ad20 2024-01-15T18:18:14.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.297+0000 7f234aa00640 1 --2- 172.21.15.27:0/3440041086 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f232c0678b0 0x7f232c069d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:14.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.297+0000 7f234aa00640 1 -- 172.21.15.27:0/3440041086 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f233c0824e0 con 0x7f235415ad20 2024-01-15T18:18:14.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.297+0000 7f2352a00640 1 --2- 172.21.15.27:0/3440041086 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f232c0678b0 0x7f232c069d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:14.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.297+0000 7f2352a00640 1 --2- 172.21.15.27:0/3440041086 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f232c0678b0 0x7f232c069d70 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f23540b1cc0 tx=0x7f2344009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:14.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.301+0000 7f234aa00640 1 -- 172.21.15.27:0/3440041086 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f235414d9d0 con 0x7f235415ad20 2024-01-15T18:18:14.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.485+0000 7f2359800640 1 -- 172.21.15.27:0/3440041086 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f235414d9d0 con 0x7f232c0678b0 2024-01-15T18:18:14.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.489+0000 7f234aa00640 1 -- 172.21.15.27:0/3440041086 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+79 (secure 0 0 0) 0x7f235414d9d0 con 0x7f232c0678b0 2024-01-15T18:18:14.490 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:18:14.490 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:18:14.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:14.490 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sv1" 2024-01-15T18:18:14.490 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:14.490 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:14.490 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ss1clone1" 2024-01-15T18:18:14.490 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:14.491 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:18:14.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.493+0000 7f2359800640 1 -- 172.21.15.27:0/3440041086 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f232c0678b0 msgr2=0x7f232c069d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:14.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.493+0000 7f2359800640 1 --2- 172.21.15.27:0/3440041086 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f232c0678b0 0x7f232c069d70 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f23540b1cc0 tx=0x7f2344009190 comp rx=0 tx=0).stop 2024-01-15T18:18:14.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.493+0000 7f2359800640 1 -- 172.21.15.27:0/3440041086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f235415ad20 msgr2=0x7f235414bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:14.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.493+0000 7f2359800640 1 --2- 172.21.15.27:0/3440041086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f235415ad20 0x7f235414bb30 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f233c00c3c0 tx=0x7f233c00c880 comp rx=0 tx=0).stop 2024-01-15T18:18:14.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.493+0000 7f2359800640 1 -- 172.21.15.27:0/3440041086 shutdown_connections 2024-01-15T18:18:14.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.493+0000 7f2359800640 1 --2- 172.21.15.27:0/3440041086 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f232c0678b0 0x7f232c069d70 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:14.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.493+0000 7f2359800640 1 --2- 172.21.15.27:0/3440041086 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f235415d900 0x7f23540ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:14.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.493+0000 7f2359800640 1 --2- 172.21.15.27:0/3440041086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f235415ad20 0x7f235414bb30 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:14.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.493+0000 7f2359800640 1 --2- 172.21.15.27:0/3440041086 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f23540715f0 0x7f235414dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:14.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.493+0000 7f2359800640 1 -- 172.21.15.27:0/3440041086 >> 172.21.15.27:0/3440041086 conn(0x7f23540cb700 msgr2=0x7f23540b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:14.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.493+0000 7f2359800640 1 -- 172.21.15.27:0/3440041086 shutdown_connections 2024-01-15T18:18:14.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.493+0000 7f2359800640 1 -- 172.21.15.27:0/3440041086 wait complete. 2024-01-15T18:18:14.512 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-15T18:18:14.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.661+0000 7fcadda00640 1 Processor -- start 2024-01-15T18:18:14.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.661+0000 7fcadda00640 1 -- start start 2024-01-15T18:18:14.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.661+0000 7fcadda00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcad80cf450 0x7fcad80d1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:14.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.661+0000 7fcadda00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fcad80cba60 0x7fcad80cf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:14.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.661+0000 7fcadda00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad80cefa0 con 0x7fcad815ee30 2024-01-15T18:18:14.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.661+0000 7fcadda00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcad80ce240 con 0x7fcad80cf450 2024-01-15T18:18:14.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.661+0000 7fcadda00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcad80cfd70 con 0x7fcad80cba60 2024-01-15T18:18:14.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.661+0000 7fcad6a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcad80cf450 0x7fcad80d1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:14.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.661+0000 7fcad7400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fcad815ee30 0x7fcad815f240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44194/0 (socket says 172.21.15.27:44194) 2024-01-15T18:18:14.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.661+0000 7fcad6a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcad80cf450 0x7fcad80d1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33690/0 (socket says 172.21.15.27:33690) 2024-01-15T18:18:14.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.661+0000 7fcad7400640 1 -- 172.21.15.27:0/350283976 learned_addr learned my addr 172.21.15.27:0/350283976 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:14.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.661+0000 7fcad6000640 1 --2- 172.21.15.27:0/350283976 >> v2:172.21.15.93:3300/0 conn(0x7fcad80cba60 0x7fcad80cf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:14.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.661+0000 7fcad6a00640 1 -- 172.21.15.27:0/350283976 >> v2:172.21.15.93:3300/0 conn(0x7fcad80cba60 msgr2=0x7fcad80cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:14.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.661+0000 7fcad6a00640 1 --2- 172.21.15.27:0/350283976 >> v2:172.21.15.93:3300/0 conn(0x7fcad80cba60 0x7fcad80cf830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:14.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.661+0000 7fcad6a00640 1 -- 172.21.15.27:0/350283976 >> v1:172.21.15.27:6789/0 conn(0x7fcad815ee30 legacy=0x7fcad815f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:14.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.661+0000 7fcad6a00640 1 -- 172.21.15.27:0/350283976 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad80ccd70 con 0x7fcad80cf450 2024-01-15T18:18:14.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.661+0000 7fcad5600640 1 -- 172.21.15.27:0/350283976 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101731413 0 0) 0x7fcad80cefa0 con 0x7fcad815ee30 2024-01-15T18:18:14.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.665+0000 7fcad6a00640 1 --2- 172.21.15.27:0/350283976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcad80cf450 0x7fcad80d1a40 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fcad8071670 tx=0x7fcac801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da0e19223ef35ba2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:14.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.665+0000 7fcad5600640 1 -- 172.21.15.27:0/350283976 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac8004230 con 0x7fcad80cf450 2024-01-15T18:18:14.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.665+0000 7fcad5600640 1 -- 172.21.15.27:0/350283976 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac80a7050 con 0x7fcad80cf450 2024-01-15T18:18:14.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.665+0000 7fcad5600640 1 -- 172.21.15.27:0/350283976 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac8003620 con 0x7fcad80cf450 2024-01-15T18:18:14.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.665+0000 7fcadda00640 1 -- 172.21.15.27:0/350283976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcad80cf450 msgr2=0x7fcad80d1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:14.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.665+0000 7fcadda00640 1 --2- 172.21.15.27:0/350283976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcad80cf450 0x7fcad80d1a40 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fcad8071670 tx=0x7fcac801c460 comp rx=0 tx=0).stop 2024-01-15T18:18:14.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.665+0000 7fcadda00640 1 -- 172.21.15.27:0/350283976 shutdown_connections 2024-01-15T18:18:14.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.665+0000 7fcadda00640 1 --2- 172.21.15.27:0/350283976 >> v2:172.21.15.93:3300/0 conn(0x7fcad80cba60 0x7fcad80cf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:14.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.665+0000 7fcadda00640 1 --2- 172.21.15.27:0/350283976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcad80cf450 0x7fcad80d1a40 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:14.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.665+0000 7fcadda00640 1 -- 172.21.15.27:0/350283976 >> 172.21.15.27:0/350283976 conn(0x7fcad80d0e70 msgr2=0x7fcad80d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:14.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.665+0000 7fcadda00640 1 -- 172.21.15.27:0/350283976 shutdown_connections 2024-01-15T18:18:14.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.665+0000 7fcadda00640 1 -- 172.21.15.27:0/350283976 wait complete. 2024-01-15T18:18:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.669+0000 7fcadda00640 1 Processor -- start 2024-01-15T18:18:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.669+0000 7fcadda00640 1 -- start start 2024-01-15T18:18:14.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.669+0000 7fcadda00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcad80cba60 0x7fcad815c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:14.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.669+0000 7fcadda00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcad80cf450 0x7fcad80c81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:14.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.669+0000 7fcadda00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcad815ee30 0x7fcad8159f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:14.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.669+0000 7fcadda00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad815c5a0 con 0x7fcad80cf450 2024-01-15T18:18:14.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.669+0000 7fcadda00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad815b990 con 0x7fcad815ee30 2024-01-15T18:18:14.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.669+0000 7fcadda00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad815bc70 con 0x7fcad80cba60 2024-01-15T18:18:14.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.669+0000 7fcad6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcad80cf450 0x7fcad80c81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:14.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.669+0000 7fcad6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcad80cf450 0x7fcad80c81a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33692/0 (socket says 172.21.15.27:33692) 2024-01-15T18:18:14.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.669+0000 7fcad6000640 1 -- 172.21.15.27:0/1695721732 learned_addr learned my addr 172.21.15.27:0/1695721732 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:14.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.669+0000 7fcad7400640 1 --2- 172.21.15.27:0/1695721732 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcad815ee30 0x7fcad8159f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:14.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.669+0000 7fcad6a00640 1 --2- 172.21.15.27:0/1695721732 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcad80cba60 0x7fcad815c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:14.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.669+0000 7fcad6000640 1 -- 172.21.15.27:0/1695721732 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcad80cba60 msgr2=0x7fcad815c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:14.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.669+0000 7fcad6000640 1 --2- 172.21.15.27:0/1695721732 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcad80cba60 0x7fcad815c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:14.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.669+0000 7fcad6000640 1 -- 172.21.15.27:0/1695721732 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcad815ee30 msgr2=0x7fcad8159f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:14.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.669+0000 7fcad6000640 1 --2- 172.21.15.27:0/1695721732 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcad815ee30 0x7fcad8159f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:14.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.669+0000 7fcad6000640 1 -- 172.21.15.27:0/1695721732 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac8009420 con 0x7fcad80cf450 2024-01-15T18:18:14.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.669+0000 7fcad6000640 1 --2- 172.21.15.27:0/1695721732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcad80cf450 0x7fcad80c81a0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fcac000d5f0 tx=0x7fcac000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:14.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.669+0000 7fcacea00640 1 -- 172.21.15.27:0/1695721732 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac00125f0 con 0x7fcad80cf450 2024-01-15T18:18:14.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.669+0000 7fcacea00640 1 -- 172.21.15.27:0/1695721732 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac0012750 con 0x7fcad80cf450 2024-01-15T18:18:14.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.669+0000 7fcadda00640 1 -- 172.21.15.27:0/1695721732 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad815bf50 con 0x7fcad80cf450 2024-01-15T18:18:14.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.669+0000 7fcacea00640 1 -- 172.21.15.27:0/1695721732 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac0012ae0 con 0x7fcad80cf450 2024-01-15T18:18:14.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.669+0000 7fcadda00640 1 -- 172.21.15.27:0/1695721732 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad815b5d0 con 0x7fcad80cf450 2024-01-15T18:18:14.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.673+0000 7fcadda00640 1 -- 172.21.15.27:0/1695721732 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcad815c750 con 0x7fcad80cf450 2024-01-15T18:18:14.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.673+0000 7fcacea00640 1 -- 172.21.15.27:0/1695721732 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 36) v1 ==== 87588+0+0 (secure 0 0 0) 0x7fcac0012c40 con 0x7fcad80cf450 2024-01-15T18:18:14.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.673+0000 7fcacea00640 1 --2- 172.21.15.27:0/1695721732 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcab00678e0 0x7fcab0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:14.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.673+0000 7fcacea00640 1 -- 172.21.15.27:0/1695721732 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fcac0083230 con 0x7fcad80cf450 2024-01-15T18:18:14.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.673+0000 7fcad6a00640 1 --2- 172.21.15.27:0/1695721732 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcab00678e0 0x7fcab0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:14.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.673+0000 7fcad6a00640 1 --2- 172.21.15.27:0/1695721732 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcab00678e0 0x7fcab0069da0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fcac80ad000 tx=0x7fcac80a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:14.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.677+0000 7fcacea00640 1 -- 172.21.15.27:0/1695721732 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fcad815c750 con 0x7fcad80cf450 2024-01-15T18:18:14.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.861+0000 7fcadda00640 1 -- 172.21.15.27:0/1695721732 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcad80cbe70 con 0x7fcab00678e0 2024-01-15T18:18:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.873+0000 7fcacea00640 1 -- 172.21.15.27:0/1695721732 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fcac8004400 con 0x7fcab00678e0 2024-01-15T18:18:14.874 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:18:14.874 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:18:14.874 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:14.874 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ss1" 2024-01-15T18:18:14.874 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:14.874 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:18:14.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.877+0000 7fcadda00640 1 -- 172.21.15.27:0/1695721732 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcab00678e0 msgr2=0x7fcab0069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:14.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.877+0000 7fcadda00640 1 --2- 172.21.15.27:0/1695721732 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcab00678e0 0x7fcab0069da0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fcac80ad000 tx=0x7fcac80a6000 comp rx=0 tx=0).stop 2024-01-15T18:18:14.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.877+0000 7fcadda00640 1 -- 172.21.15.27:0/1695721732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcad80cf450 msgr2=0x7fcad80c81a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:14.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.877+0000 7fcadda00640 1 --2- 172.21.15.27:0/1695721732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcad80cf450 0x7fcad80c81a0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fcac000d5f0 tx=0x7fcac000dab0 comp rx=0 tx=0).stop 2024-01-15T18:18:14.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.877+0000 7fcadda00640 1 -- 172.21.15.27:0/1695721732 shutdown_connections 2024-01-15T18:18:14.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.877+0000 7fcadda00640 1 --2- 172.21.15.27:0/1695721732 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fcab00678e0 0x7fcab0069da0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:14.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.877+0000 7fcadda00640 1 --2- 172.21.15.27:0/1695721732 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fcad815ee30 0x7fcad8159f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:14.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.877+0000 7fcadda00640 1 --2- 172.21.15.27:0/1695721732 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcad80cf450 0x7fcad80c81a0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:14.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.877+0000 7fcadda00640 1 --2- 172.21.15.27:0/1695721732 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fcad80cba60 0x7fcad815c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:14.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.877+0000 7fcadda00640 1 -- 172.21.15.27:0/1695721732 >> 172.21.15.27:0/1695721732 conn(0x7fcad80d0e70 msgr2=0x7fcad8165750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:14.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.877+0000 7fcadda00640 1 -- 172.21.15.27:0/1695721732 shutdown_connections 2024-01-15T18:18:14.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:14.877+0000 7fcadda00640 1 -- 172.21.15.27:0/1695721732 wait complete. 2024-01-15T18:18:14.896 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:18:15.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.041+0000 7f8649a00640 1 Processor -- start 2024-01-15T18:18:15.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.041+0000 7f8649a00640 1 -- start start 2024-01-15T18:18:15.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.045+0000 7f8649a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f864406c430 0x7f8644070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:15.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.045+0000 7f8649a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86440ccfb0 con 0x7f86440b1db0 2024-01-15T18:18:15.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.045+0000 7f8649a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86440cd290 con 0x7f86440ce760 2024-01-15T18:18:15.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.045+0000 7f8649a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86440cd570 con 0x7f864406c430 2024-01-15T18:18:15.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.045+0000 7f8642a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f864406c430 0x7f8644070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:15.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.045+0000 7f8642000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f86440b1db0 0x7f86440ce050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44202/0 (socket says 172.21.15.27:44202) 2024-01-15T18:18:15.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.045+0000 7f8642000640 1 -- 172.21.15.27:0/1051562697 learned_addr learned my addr 172.21.15.27:0/1051562697 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:15.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.045+0000 7f8642a00640 1 -- 172.21.15.27:0/1051562697 >> v1:172.21.15.93:6789/0 conn(0x7f86440ce760 legacy=0x7f864406c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:15.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.045+0000 7f8642a00640 1 -- 172.21.15.27:0/1051562697 >> v1:172.21.15.27:6789/0 conn(0x7f86440b1db0 legacy=0x7f86440ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:15.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.045+0000 7f8642a00640 1 -- 172.21.15.27:0/1051562697 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86440cd850 con 0x7f864406c430 2024-01-15T18:18:15.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.049+0000 7f8642a00640 1 --2- 172.21.15.27:0/1051562697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864406c430 0x7f8644070ba0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f86340097f0 tx=0x7f863401d4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d31c0e99e1986f8d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:15.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.049+0000 7f8641600640 1 -- 172.21.15.27:0/1051562697 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86340ac5f0 con 0x7f864406c430 2024-01-15T18:18:15.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.049+0000 7f8641600640 1 -- 172.21.15.27:0/1051562697 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86340ac750 con 0x7f864406c430 2024-01-15T18:18:15.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.049+0000 7f8641600640 1 -- 172.21.15.27:0/1051562697 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86340acae0 con 0x7f864406c430 2024-01-15T18:18:15.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.049+0000 7f8649a00640 1 -- 172.21.15.27:0/1051562697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864406c430 msgr2=0x7f8644070ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:15.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.049+0000 7f8649a00640 1 --2- 172.21.15.27:0/1051562697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864406c430 0x7f8644070ba0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f86340097f0 tx=0x7f863401d4d0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.049+0000 7f8649a00640 1 -- 172.21.15.27:0/1051562697 shutdown_connections 2024-01-15T18:18:15.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.049+0000 7f8649a00640 1 --2- 172.21.15.27:0/1051562697 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864406c430 0x7f8644070ba0 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.049+0000 7f8649a00640 1 -- 172.21.15.27:0/1051562697 >> 172.21.15.27:0/1051562697 conn(0x7f86440cb700 msgr2=0x7f86440cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:15.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.049+0000 7f8649a00640 1 -- 172.21.15.27:0/1051562697 shutdown_connections 2024-01-15T18:18:15.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.049+0000 7f8649a00640 1 -- 172.21.15.27:0/1051562697 wait complete. 2024-01-15T18:18:15.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.049+0000 7f8649a00640 1 Processor -- start 2024-01-15T18:18:15.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.049+0000 7f8649a00640 1 -- start start 2024-01-15T18:18:15.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.053+0000 7f8649a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864406c430 0x7f8644152dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:15.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.053+0000 7f8649a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f86440b1db0 0x7f8644150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:15.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.053+0000 7f8649a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f86440ce760 0x7f86441424a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:15.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.053+0000 7f8649a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8644149fa0 con 0x7f864406c430 2024-01-15T18:18:15.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.053+0000 7f8649a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864414a280 con 0x7f86440ce760 2024-01-15T18:18:15.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.053+0000 7f8649a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864414a560 con 0x7f86440b1db0 2024-01-15T18:18:15.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.053+0000 7f8642a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864406c430 0x7f8644152dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:15.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.053+0000 7f8642a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864406c430 0x7f8644152dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33720/0 (socket says 172.21.15.27:33720) 2024-01-15T18:18:15.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.053+0000 7f8642a00640 1 -- 172.21.15.27:0/4217126298 learned_addr learned my addr 172.21.15.27:0/4217126298 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:15.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.053+0000 7f8642000640 1 --2- 172.21.15.27:0/4217126298 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f86440b1db0 0x7f8644150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:15.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.053+0000 7f8643400640 1 --2- 172.21.15.27:0/4217126298 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f86440ce760 0x7f86441424a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:15.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.053+0000 7f8642a00640 1 -- 172.21.15.27:0/4217126298 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f86440b1db0 msgr2=0x7f8644150e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:15.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.053+0000 7f8642a00640 1 --2- 172.21.15.27:0/4217126298 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f86440b1db0 0x7f8644150e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.053+0000 7f8642a00640 1 -- 172.21.15.27:0/4217126298 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f86440ce760 msgr2=0x7f86441424a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:15.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.053+0000 7f8642a00640 1 --2- 172.21.15.27:0/4217126298 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f86440ce760 0x7f86441424a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.053+0000 7f8642a00640 1 -- 172.21.15.27:0/4217126298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8634009420 con 0x7f864406c430 2024-01-15T18:18:15.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.053+0000 7f8642000640 1 --2- 172.21.15.27:0/4217126298 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f86440b1db0 0x7f8644150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:18:15.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.053+0000 7f8642a00640 1 --2- 172.21.15.27:0/4217126298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864406c430 0x7f8644152dd0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f86340ae040 tx=0x7f863401a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:15.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.053+0000 7f863aa00640 1 -- 172.21.15.27:0/4217126298 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86340b5800 con 0x7f864406c430 2024-01-15T18:18:15.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.053+0000 7f863aa00640 1 -- 172.21.15.27:0/4217126298 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86340b5980 con 0x7f864406c430 2024-01-15T18:18:15.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.053+0000 7f8649a00640 1 -- 172.21.15.27:0/4217126298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f864414a840 con 0x7f864406c430 2024-01-15T18:18:15.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.053+0000 7f863aa00640 1 -- 172.21.15.27:0/4217126298 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86340b5d30 con 0x7f864406c430 2024-01-15T18:18:15.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.053+0000 7f8649a00640 1 -- 172.21.15.27:0/4217126298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8644149be0 con 0x7f864406c430 2024-01-15T18:18:15.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.053+0000 7f8649a00640 1 -- 172.21.15.27:0/4217126298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f864404e850 con 0x7f864406c430 2024-01-15T18:18:15.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.057+0000 7f863aa00640 1 -- 172.21.15.27:0/4217126298 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 36) v1 ==== 87588+0+0 (secure 0 0 0) 0x7f86340c6450 con 0x7f864406c430 2024-01-15T18:18:15.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.057+0000 7f863aa00640 1 --2- 172.21.15.27:0/4217126298 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f861c0678e0 0x7f861c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:15.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.057+0000 7f863aa00640 1 -- 172.21.15.27:0/4217126298 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f86340b9070 con 0x7f864406c430 2024-01-15T18:18:15.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.057+0000 7f8642000640 1 --2- 172.21.15.27:0/4217126298 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f861c0678e0 0x7f861c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:15.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.057+0000 7f8642000640 1 --2- 172.21.15.27:0/4217126298 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f861c0678e0 0x7f861c069da0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f864415b6c0 tx=0x7f862c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:15.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.061+0000 7f863aa00640 1 -- 172.21.15.27:0/4217126298 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f86340ec5f0 con 0x7f864406c430 2024-01-15T18:18:15.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.249+0000 7f8649a00640 1 -- 172.21.15.27:0/4217126298 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f86441671c0 con 0x7f861c0678e0 2024-01-15T18:18:15.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.273+0000 7f863aa00640 1 -- 172.21.15.27:0/4217126298 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f862c001430 con 0x7f861c0678e0 2024-01-15T18:18:15.272 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:18:15.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.273+0000 7f8649a00640 1 -- 172.21.15.27:0/4217126298 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f861c0678e0 msgr2=0x7f861c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:15.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.273+0000 7f8649a00640 1 --2- 172.21.15.27:0/4217126298 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f861c0678e0 0x7f861c069da0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f864415b6c0 tx=0x7f862c009040 comp rx=0 tx=0).stop 2024-01-15T18:18:15.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.273+0000 7f8649a00640 1 -- 172.21.15.27:0/4217126298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864406c430 msgr2=0x7f8644152dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:15.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.273+0000 7f8649a00640 1 --2- 172.21.15.27:0/4217126298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864406c430 0x7f8644152dd0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f86340ae040 tx=0x7f863401a270 comp rx=0 tx=0).stop 2024-01-15T18:18:15.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.273+0000 7f8649a00640 1 -- 172.21.15.27:0/4217126298 shutdown_connections 2024-01-15T18:18:15.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.273+0000 7f8649a00640 1 --2- 172.21.15.27:0/4217126298 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f861c0678e0 0x7f861c069da0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.273+0000 7f8649a00640 1 --2- 172.21.15.27:0/4217126298 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f86440ce760 0x7f86441424a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.273+0000 7f8649a00640 1 --2- 172.21.15.27:0/4217126298 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f86440b1db0 0x7f8644150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.273+0000 7f8649a00640 1 --2- 172.21.15.27:0/4217126298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864406c430 0x7f8644152dd0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.273+0000 7f8649a00640 1 -- 172.21.15.27:0/4217126298 >> 172.21.15.27:0/4217126298 conn(0x7f86440cb700 msgr2=0x7f86440aef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:15.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.273+0000 7f8649a00640 1 -- 172.21.15.27:0/4217126298 shutdown_connections 2024-01-15T18:18:15.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.273+0000 7f8649a00640 1 -- 172.21.15.27:0/4217126298 wait complete. 2024-01-15T18:18:15.294 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-15T18:18:15.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.437+0000 7f5b08c00640 1 Processor -- start 2024-01-15T18:18:15.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.437+0000 7f5b08c00640 1 -- start start 2024-01-15T18:18:15.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.441+0000 7f5b08c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5b0415d190 0x7f5b040cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:15.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.441+0000 7f5b08c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f5b0415d570 0x7f5b0406be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:15.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.441+0000 7f5b08c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b0406b810 0x7f5b0406c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:15.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.441+0000 7f5b08c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b040b0d70 con 0x7f5b0415d190 2024-01-15T18:18:15.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.441+0000 7f5b08c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b040d0540 con 0x7f5b0415d570 2024-01-15T18:18:15.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.441+0000 7f5b08c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b040b0440 con 0x7f5b0406b810 2024-01-15T18:18:15.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.441+0000 7f5b01e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5b0415d190 0x7f5b040cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:15.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.441+0000 7f5b01e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5b0415d190 0x7f5b040cfcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33722/0 (socket says 172.21.15.27:33722) 2024-01-15T18:18:15.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.441+0000 7f5b01e00640 1 -- 172.21.15.27:0/2343362209 learned_addr learned my addr 172.21.15.27:0/2343362209 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:15.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.441+0000 7f5b01400640 1 --2- 172.21.15.27:0/2343362209 >> v2:172.21.15.93:3300/0 conn(0x7f5b0415d570 0x7f5b0406be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:15.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.441+0000 7f5b02800640 1 --2- 172.21.15.27:0/2343362209 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b0406b810 0x7f5b0406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:15.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.441+0000 7f5b01e00640 1 -- 172.21.15.27:0/2343362209 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b0406b810 msgr2=0x7f5b0406c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:15.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.441+0000 7f5b01e00640 1 --2- 172.21.15.27:0/2343362209 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b0406b810 0x7f5b0406c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.441+0000 7f5b01e00640 1 -- 172.21.15.27:0/2343362209 >> v2:172.21.15.93:3300/0 conn(0x7f5b0415d570 msgr2=0x7f5b0406be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:15.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.441+0000 7f5b01e00640 1 --2- 172.21.15.27:0/2343362209 >> v2:172.21.15.93:3300/0 conn(0x7f5b0415d570 0x7f5b0406be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.441+0000 7f5b01e00640 1 -- 172.21.15.27:0/2343362209 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b040b0720 con 0x7f5b0415d190 2024-01-15T18:18:15.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.441+0000 7f5b01400640 1 --2- 172.21.15.27:0/2343362209 >> v2:172.21.15.93:3300/0 conn(0x7f5b0415d570 0x7f5b0406be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:18:15.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.441+0000 7f5b01e00640 1 --2- 172.21.15.27:0/2343362209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b0415d190 0x7f5b040cfcf0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f5b040713a0 tx=0x7f5af401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7138f528dfec953c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:15.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.441+0000 7f5b00a00640 1 -- 172.21.15.27:0/2343362209 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af40ab650 con 0x7f5b0415d190 2024-01-15T18:18:15.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.441+0000 7f5b00a00640 1 -- 172.21.15.27:0/2343362209 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af40ab7b0 con 0x7f5b0415d190 2024-01-15T18:18:15.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.441+0000 7f5b00a00640 1 -- 172.21.15.27:0/2343362209 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af40abb40 con 0x7f5b0415d190 2024-01-15T18:18:15.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.445+0000 7f5b08c00640 1 -- 172.21.15.27:0/2343362209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b0415d190 msgr2=0x7f5b040cfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:15.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.445+0000 7f5b08c00640 1 --2- 172.21.15.27:0/2343362209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b0415d190 0x7f5b040cfcf0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f5b040713a0 tx=0x7f5af401c4d0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.445+0000 7f5b08c00640 1 -- 172.21.15.27:0/2343362209 shutdown_connections 2024-01-15T18:18:15.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.445+0000 7f5b08c00640 1 --2- 172.21.15.27:0/2343362209 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b0406b810 0x7f5b0406c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.445+0000 7f5b08c00640 1 --2- 172.21.15.27:0/2343362209 >> v2:172.21.15.93:3300/0 conn(0x7f5b0415d570 0x7f5b0406be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.445+0000 7f5b08c00640 1 --2- 172.21.15.27:0/2343362209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b0415d190 0x7f5b040cfcf0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.445+0000 7f5b08c00640 1 -- 172.21.15.27:0/2343362209 >> 172.21.15.27:0/2343362209 conn(0x7f5b040d0e70 msgr2=0x7f5b040d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:15.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.445+0000 7f5b08c00640 1 -- 172.21.15.27:0/2343362209 shutdown_connections 2024-01-15T18:18:15.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.445+0000 7f5b08c00640 1 -- 172.21.15.27:0/2343362209 wait complete. 2024-01-15T18:18:15.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.445+0000 7f5b08c00640 1 Processor -- start 2024-01-15T18:18:15.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.445+0000 7f5b08c00640 1 -- start start 2024-01-15T18:18:15.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.445+0000 7f5b08c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b0406b810 0x7f5b0414dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:15.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.445+0000 7f5b08c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b0415d190 0x7f5b040b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:15.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.445+0000 7f5b08c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b0415d570 0x7f5b0415f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:15.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.445+0000 7f5b08c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0414dba0 con 0x7f5b0406b810 2024-01-15T18:18:15.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.445+0000 7f5b08c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0414e250 con 0x7f5b0415d570 2024-01-15T18:18:15.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.445+0000 7f5b08c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b040c5bf0 con 0x7f5b0415d190 2024-01-15T18:18:15.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.449+0000 7f5b01e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b0406b810 0x7f5b0414dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:15.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.449+0000 7f5b01e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b0406b810 0x7f5b0414dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33734/0 (socket says 172.21.15.27:33734) 2024-01-15T18:18:15.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.449+0000 7f5b01e00640 1 -- 172.21.15.27:0/1469509803 learned_addr learned my addr 172.21.15.27:0/1469509803 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:15.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.449+0000 7f5b01400640 1 --2- 172.21.15.27:0/1469509803 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b0415d190 0x7f5b040b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:15.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.449+0000 7f5b02800640 1 --2- 172.21.15.27:0/1469509803 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b0415d570 0x7f5b0415f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:15.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.449+0000 7f5b01e00640 1 -- 172.21.15.27:0/1469509803 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b0415d190 msgr2=0x7f5b040b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:15.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.449+0000 7f5b01e00640 1 --2- 172.21.15.27:0/1469509803 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b0415d190 0x7f5b040b4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.449+0000 7f5b01e00640 1 -- 172.21.15.27:0/1469509803 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b0415d570 msgr2=0x7f5b0415f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:15.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.449+0000 7f5b01e00640 1 --2- 172.21.15.27:0/1469509803 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b0415d570 0x7f5b0415f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.449+0000 7f5b01e00640 1 -- 172.21.15.27:0/1469509803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af40094d0 con 0x7f5b0406b810 2024-01-15T18:18:15.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.449+0000 7f5b01400640 1 --2- 172.21.15.27:0/1469509803 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b0415d190 0x7f5b040b4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:18:15.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.449+0000 7f5b01e00640 1 --2- 172.21.15.27:0/1469509803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b0406b810 0x7f5b0414dd10 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f5af401ab80 tx=0x7f5af401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:15.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.449+0000 7f5afa000640 1 -- 172.21.15.27:0/1469509803 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af40b4800 con 0x7f5b0406b810 2024-01-15T18:18:15.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.449+0000 7f5afa000640 1 -- 172.21.15.27:0/1469509803 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af40b4960 con 0x7f5b0406b810 2024-01-15T18:18:15.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.449+0000 7f5afa000640 1 -- 172.21.15.27:0/1469509803 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af40b4cf0 con 0x7f5b0406b810 2024-01-15T18:18:15.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.449+0000 7f5b08c00640 1 -- 172.21.15.27:0/1469509803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b040c6770 con 0x7f5b0406b810 2024-01-15T18:18:15.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.449+0000 7f5b08c00640 1 -- 172.21.15.27:0/1469509803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b040c6120 con 0x7f5b0406b810 2024-01-15T18:18:15.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.449+0000 7f5b08c00640 1 -- 172.21.15.27:0/1469509803 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b0404e850 con 0x7f5b0406b810 2024-01-15T18:18:15.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.449+0000 7f5afa000640 1 -- 172.21.15.27:0/1469509803 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 36) v1 ==== 87588+0+0 (secure 0 0 0) 0x7f5af40b3380 con 0x7f5b0406b810 2024-01-15T18:18:15.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.449+0000 7f5afa000640 1 --2- 172.21.15.27:0/1469509803 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5adc0678b0 0x7f5adc069d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:15.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.453+0000 7f5afa000640 1 -- 172.21.15.27:0/1469509803 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f5af411d710 con 0x7f5b0406b810 2024-01-15T18:18:15.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.453+0000 7f5b01400640 1 --2- 172.21.15.27:0/1469509803 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5adc0678b0 0x7f5adc069d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:15.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.453+0000 7f5b01400640 1 --2- 172.21.15.27:0/1469509803 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5adc0678b0 0x7f5adc069d70 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f5b0406cd00 tx=0x7f5aec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:15.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.457+0000 7f5afa000640 1 -- 172.21.15.27:0/1469509803 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5af40ec1a0 con 0x7f5b0406b810 2024-01-15T18:18:15.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.633+0000 7f5b08c00640 1 -- 172.21.15.27:0/1469509803 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b04160ef0 con 0x7f5adc0678b0 2024-01-15T18:18:15.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.653+0000 7f5afa000640 1 -- 172.21.15.27:0/1469509803 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5aec0071e0 con 0x7f5adc0678b0 2024-01-15T18:18:15.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.657+0000 7f5b08c00640 1 -- 172.21.15.27:0/1469509803 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5adc0678b0 msgr2=0x7f5adc069d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:15.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.657+0000 7f5b08c00640 1 --2- 172.21.15.27:0/1469509803 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5adc0678b0 0x7f5adc069d70 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f5b0406cd00 tx=0x7f5aec008040 comp rx=0 tx=0).stop 2024-01-15T18:18:15.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.657+0000 7f5b08c00640 1 -- 172.21.15.27:0/1469509803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b0406b810 msgr2=0x7f5b0414dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:15.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.657+0000 7f5b08c00640 1 --2- 172.21.15.27:0/1469509803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b0406b810 0x7f5b0414dd10 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f5af401ab80 tx=0x7f5af401ad10 comp rx=0 tx=0).stop 2024-01-15T18:18:15.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.657+0000 7f5b08c00640 1 -- 172.21.15.27:0/1469509803 shutdown_connections 2024-01-15T18:18:15.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.657+0000 7f5b08c00640 1 --2- 172.21.15.27:0/1469509803 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5adc0678b0 0x7f5adc069d70 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.657+0000 7f5b08c00640 1 --2- 172.21.15.27:0/1469509803 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b0415d570 0x7f5b0415f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.657+0000 7f5b08c00640 1 --2- 172.21.15.27:0/1469509803 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b0415d190 0x7f5b040b4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.657+0000 7f5b08c00640 1 --2- 172.21.15.27:0/1469509803 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b0406b810 0x7f5b0414dd10 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.657+0000 7f5b08c00640 1 -- 172.21.15.27:0/1469509803 >> 172.21.15.27:0/1469509803 conn(0x7f5b040d0e70 msgr2=0x7f5b040afaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:15.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.657+0000 7f5b08c00640 1 -- 172.21.15.27:0/1469509803 shutdown_connections 2024-01-15T18:18:15.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.657+0000 7f5b08c00640 1 -- 172.21.15.27:0/1469509803 wait complete. 2024-01-15T18:18:15.660 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:18:15.663+0000 7f59f6000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:18:15.660 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:18:15.663+0000 7f59f6000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:18:15.660 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:18:15.663+0000 7f59f6000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:18:15.660 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:18:15.663+0000 7f59f6000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:18:15.660 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:18:15.663+0000 7f59f6000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:18:15.678 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:18:15.679+0000 7f59f5600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:18:15.678 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:18:15.679+0000 7f59f5600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:18:15.678 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:18:15.679+0000 7f59f5600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:18:15.678 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:18:15.679+0000 7f59f5600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:18:15.678 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:18:15.679+0000 7f59f5600640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:18:15.679 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs ss1clone1 2024-01-15T18:18:15.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.829+0000 7f7e9b800640 1 Processor -- start 2024-01-15T18:18:15.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.829+0000 7f7e9b800640 1 -- start start 2024-01-15T18:18:15.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.829+0000 7f7e9b800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7e9415d680 0x7f7e940cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:15.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.829+0000 7f7e9b800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f7e940cfc30 0x7f7e940cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:15.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.829+0000 7f7e9b800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7e940d19a0 0x7f7e9406b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:15.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.829+0000 7f7e9b800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e940cb4d0 con 0x7f7e940d19a0 2024-01-15T18:18:15.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.829+0000 7f7e9b800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e940d0120 con 0x7f7e940cfc30 2024-01-15T18:18:15.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.829+0000 7f7e9b800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e940d0400 con 0x7f7e9415d680 2024-01-15T18:18:15.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.829+0000 7f7e99800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7e940d19a0 0x7f7e9406b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:15.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.829+0000 7f7e99800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7e940d19a0 0x7f7e9406b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33746/0 (socket says 172.21.15.27:33746) 2024-01-15T18:18:15.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.829+0000 7f7e99800640 1 -- 172.21.15.27:0/3455147624 learned_addr learned my addr 172.21.15.27:0/3455147624 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:15.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.833+0000 7f7e98e00640 1 --2- 172.21.15.27:0/3455147624 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7e9415d680 0x7f7e940cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:15.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.833+0000 7f7e99800640 1 -- 172.21.15.27:0/3455147624 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7e9415d680 msgr2=0x7f7e940cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:15.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.833+0000 7f7e99800640 1 --2- 172.21.15.27:0/3455147624 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7e9415d680 0x7f7e940cd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.833+0000 7f7e99800640 1 -- 172.21.15.27:0/3455147624 >> v2:172.21.15.93:3300/0 conn(0x7f7e940cfc30 msgr2=0x7f7e940cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:15.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.833+0000 7f7e99800640 1 --2- 172.21.15.27:0/3455147624 >> v2:172.21.15.93:3300/0 conn(0x7f7e940cfc30 0x7f7e940cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.833+0000 7f7e99800640 1 -- 172.21.15.27:0/3455147624 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e940cb770 con 0x7f7e940d19a0 2024-01-15T18:18:15.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.833+0000 7f7e99800640 1 --2- 172.21.15.27:0/3455147624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e940d19a0 0x7f7e9406b930 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f7e94067280 tx=0x7f7e8c01c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a53d984048bc8d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:15.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.833+0000 7f7e93400640 1 -- 172.21.15.27:0/3455147624 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e940ce850 con 0x7f7e940d19a0 2024-01-15T18:18:15.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.833+0000 7f7e93400640 1 -- 172.21.15.27:0/3455147624 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8c0a6500 con 0x7f7e940d19a0 2024-01-15T18:18:15.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.833+0000 7f7e93400640 1 -- 172.21.15.27:0/3455147624 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c0a6890 con 0x7f7e940d19a0 2024-01-15T18:18:15.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.837+0000 7f7e9b800640 1 -- 172.21.15.27:0/3455147624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e940d19a0 msgr2=0x7f7e9406b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:15.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.837+0000 7f7e9b800640 1 --2- 172.21.15.27:0/3455147624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e940d19a0 0x7f7e9406b930 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f7e94067280 tx=0x7f7e8c01c650 comp rx=0 tx=0).stop 2024-01-15T18:18:15.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.837+0000 7f7e9b800640 1 -- 172.21.15.27:0/3455147624 shutdown_connections 2024-01-15T18:18:15.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.837+0000 7f7e9b800640 1 --2- 172.21.15.27:0/3455147624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e940d19a0 0x7f7e9406b930 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.837+0000 7f7e9b800640 1 --2- 172.21.15.27:0/3455147624 >> v2:172.21.15.93:3300/0 conn(0x7f7e940cfc30 0x7f7e940cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.837+0000 7f7e9b800640 1 --2- 172.21.15.27:0/3455147624 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7e9415d680 0x7f7e940cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.837+0000 7f7e9b800640 1 -- 172.21.15.27:0/3455147624 >> 172.21.15.27:0/3455147624 conn(0x7f7e940d0e80 msgr2=0x7f7e940d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:15.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.837+0000 7f7e9b800640 1 -- 172.21.15.27:0/3455147624 shutdown_connections 2024-01-15T18:18:15.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.837+0000 7f7e9b800640 1 -- 172.21.15.27:0/3455147624 wait complete. 2024-01-15T18:18:15.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.837+0000 7f7e9b800640 1 Processor -- start 2024-01-15T18:18:15.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.837+0000 7f7e9b800640 1 -- start start 2024-01-15T18:18:15.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.837+0000 7f7e9b800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7e940cfc30 0x7f7e94155150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:15.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.837+0000 7f7e9b800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e940d19a0 0x7f7e94158930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:15.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.837+0000 7f7e9b800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e9415d680 0x7f7e940c6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:15.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.837+0000 7f7e9b800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e94151620 con 0x7f7e9415d680 2024-01-15T18:18:15.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.837+0000 7f7e9b800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e940c62b0 con 0x7f7e940d19a0 2024-01-15T18:18:15.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.837+0000 7f7e9b800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e940c6550 con 0x7f7e940cfc30 2024-01-15T18:18:15.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.837+0000 7f7e99800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e9415d680 0x7f7e940c6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:15.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.841+0000 7f7e98e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7e940cfc30 0x7f7e94155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:15.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.841+0000 7f7e99800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e9415d680 0x7f7e940c6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33756/0 (socket says 172.21.15.27:33756) 2024-01-15T18:18:15.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.841+0000 7f7e99800640 1 -- 172.21.15.27:0/3428800499 learned_addr learned my addr 172.21.15.27:0/3428800499 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:15.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.841+0000 7f7e93e00640 1 --2- 172.21.15.27:0/3428800499 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e940d19a0 0x7f7e94158930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:15.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.841+0000 7f7e99800640 1 -- 172.21.15.27:0/3428800499 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7e940cfc30 msgr2=0x7f7e94155150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:15.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.841+0000 7f7e99800640 1 --2- 172.21.15.27:0/3428800499 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7e940cfc30 0x7f7e94155150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.841+0000 7f7e99800640 1 -- 172.21.15.27:0/3428800499 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e940d19a0 msgr2=0x7f7e94158930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:15.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.841+0000 7f7e99800640 1 --2- 172.21.15.27:0/3428800499 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e940d19a0 0x7f7e94158930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:15.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.841+0000 7f7e99800640 1 -- 172.21.15.27:0/3428800499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8c009b30 con 0x7f7e9415d680 2024-01-15T18:18:15.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.841+0000 7f7e98e00640 1 --2- 172.21.15.27:0/3428800499 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7e940cfc30 0x7f7e94155150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:18:15.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.841+0000 7f7e99800640 1 --2- 172.21.15.27:0/3428800499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e9415d680 0x7f7e940c6d10 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c01ad00 tx=0x7f7e8c0193f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:15.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.841+0000 7f7e90c00640 1 -- 172.21.15.27:0/3428800499 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c0b4070 con 0x7f7e9415d680 2024-01-15T18:18:15.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.841+0000 7f7e90c00640 1 -- 172.21.15.27:0/3428800499 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8c0a6670 con 0x7f7e9415d680 2024-01-15T18:18:15.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.841+0000 7f7e90c00640 1 -- 172.21.15.27:0/3428800499 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c0a6a20 con 0x7f7e9415d680 2024-01-15T18:18:15.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.841+0000 7f7e9b800640 1 -- 172.21.15.27:0/3428800499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e940c4e10 con 0x7f7e9415d680 2024-01-15T18:18:15.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.841+0000 7f7e9b800640 1 -- 172.21.15.27:0/3428800499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e940c67f0 con 0x7f7e9415d680 2024-01-15T18:18:15.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.841+0000 7f7e9b800640 1 -- 172.21.15.27:0/3428800499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e940c69e0 con 0x7f7e9415d680 2024-01-15T18:18:15.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.845+0000 7f7e90c00640 1 -- 172.21.15.27:0/3428800499 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 36) v1 ==== 87588+0+0 (secure 0 0 0) 0x7f7e8c0b8070 con 0x7f7e9415d680 2024-01-15T18:18:15.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.845+0000 7f7e90c00640 1 --2- 172.21.15.27:0/3428800499 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7e6c0678e0 0x7f7e6c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:15.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.845+0000 7f7e90c00640 1 -- 172.21.15.27:0/3428800499 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f7e8c0aa070 con 0x7f7e9415d680 2024-01-15T18:18:15.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.845+0000 7f7e98e00640 1 --2- 172.21.15.27:0/3428800499 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7e6c0678e0 0x7f7e6c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:15.846 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.845+0000 7f7e98e00640 1 --2- 172.21.15.27:0/3428800499 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7e6c0678e0 0x7f7e6c069da0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f7e94159900 tx=0x7f7e84009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:15.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:15.849+0000 7f7e90c00640 1 -- 172.21.15.27:0/3428800499 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7e8c0eb760 con 0x7f7e9415d680 2024-01-15T18:18:16.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.029+0000 7f7e9b800640 1 -- 172.21.15.27:0/3428800499 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e9404e850 con 0x7f7e6c0678e0 2024-01-15T18:18:16.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.049+0000 7f7e90c00640 1 -- 172.21.15.27:0/3428800499 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7e9404e850 con 0x7f7e6c0678e0 2024-01-15T18:18:16.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.053+0000 7f7e9b800640 1 -- 172.21.15.27:0/3428800499 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7e6c0678e0 msgr2=0x7f7e6c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:16.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.053+0000 7f7e9b800640 1 --2- 172.21.15.27:0/3428800499 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7e6c0678e0 0x7f7e6c069da0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f7e94159900 tx=0x7f7e84009190 comp rx=0 tx=0).stop 2024-01-15T18:18:16.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.053+0000 7f7e9b800640 1 -- 172.21.15.27:0/3428800499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e9415d680 msgr2=0x7f7e940c6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:16.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.053+0000 7f7e9b800640 1 --2- 172.21.15.27:0/3428800499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e9415d680 0x7f7e940c6d10 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c01ad00 tx=0x7f7e8c0193f0 comp rx=0 tx=0).stop 2024-01-15T18:18:16.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.053+0000 7f7e9b800640 1 -- 172.21.15.27:0/3428800499 shutdown_connections 2024-01-15T18:18:16.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.053+0000 7f7e9b800640 1 --2- 172.21.15.27:0/3428800499 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7e6c0678e0 0x7f7e6c069da0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:16.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.053+0000 7f7e9b800640 1 --2- 172.21.15.27:0/3428800499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7e9415d680 0x7f7e940c6d10 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:16.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.053+0000 7f7e9b800640 1 --2- 172.21.15.27:0/3428800499 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7e940d19a0 0x7f7e94158930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:16.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.053+0000 7f7e9b800640 1 --2- 172.21.15.27:0/3428800499 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7e940cfc30 0x7f7e94155150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:16.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.053+0000 7f7e9b800640 1 -- 172.21.15.27:0/3428800499 >> 172.21.15.27:0/3428800499 conn(0x7f7e940d0e80 msgr2=0x7f7e940d1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:16.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.053+0000 7f7e9b800640 1 -- 172.21.15.27:0/3428800499 shutdown_connections 2024-01-15T18:18:16.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.053+0000 7f7e9b800640 1 -- 172.21.15.27:0/3428800499 wait complete. 2024-01-15T18:18:16.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:16.074 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-15T18:18:16.089 INFO:teuthology.orchestra.run.smithi027.stdout:2 2024-01-15T18:18:16.090 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-15T18:18:16.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.281+0000 7fecc9a00640 1 Processor -- start 2024-01-15T18:18:16.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.281+0000 7fecc9a00640 1 -- start start 2024-01-15T18:18:16.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.281+0000 7fecc9a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fecc40cf460 0x7fecc40cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:16.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.281+0000 7fecc9a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc40cfd80 con 0x7fecc40b2590 2024-01-15T18:18:16.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.281+0000 7fecc9a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc40d19a0 con 0x7fecc415b4b0 2024-01-15T18:18:16.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.281+0000 7fecc9a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecc40d1c80 con 0x7fecc40cf460 2024-01-15T18:18:16.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.281+0000 7fecc3400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fecc40b2590 0x7fecc40cc130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44210/0 (socket says 172.21.15.27:44210) 2024-01-15T18:18:16.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.281+0000 7fecc3400640 1 -- 172.21.15.27:0/3268702351 learned_addr learned my addr 172.21.15.27:0/3268702351 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:16.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.281+0000 7fecc2a00640 1 --2- 172.21.15.27:0/3268702351 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fecc40cf460 0x7fecc40cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:16.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.281+0000 7fecc1600640 1 -- 172.21.15.27:0/3268702351 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2279049291 0 0) 0x7fecc40cfd80 con 0x7fecc40b2590 2024-01-15T18:18:16.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.281+0000 7fecc1600640 1 -- 172.21.15.27:0/3268702351 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feca4003530 con 0x7fecc40b2590 2024-01-15T18:18:16.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.281+0000 7fecc1600640 1 -- 172.21.15.27:0/3268702351 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2505088124 0 0) 0x7fecc40d19a0 con 0x7fecc415b4b0 2024-01-15T18:18:16.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.281+0000 7fecc1600640 1 -- 172.21.15.27:0/3268702351 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc40cfd80 con 0x7fecc415b4b0 2024-01-15T18:18:16.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.281+0000 7fecc2a00640 1 -- 172.21.15.27:0/3268702351 >> v1:172.21.15.93:6789/0 conn(0x7fecc415b4b0 legacy=0x7fecc40cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:16.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.281+0000 7fecc1600640 1 -- 172.21.15.27:0/3268702351 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3503606331 0 0) 0x7feca4003530 con 0x7fecc40b2590 2024-01-15T18:18:16.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.281+0000 7fecc2a00640 1 -- 172.21.15.27:0/3268702351 >> v1:172.21.15.27:6789/0 conn(0x7fecc40b2590 legacy=0x7fecc40cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:16.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.281+0000 7fecc2a00640 1 -- 172.21.15.27:0/3268702351 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc40b07c0 con 0x7fecc40cf460 2024-01-15T18:18:16.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.285+0000 7fecc2a00640 1 --2- 172.21.15.27:0/3268702351 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fecc40cf460 0x7fecc40cf840 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fecb40097f0 tx=0x7fecb401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28632fe5ee2f3736 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:16.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.285+0000 7fecc1600640 1 -- 172.21.15.27:0/3268702351 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecbc001a00 con 0x7fecc40cf460 2024-01-15T18:18:16.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.285+0000 7fecc1600640 1 -- 172.21.15.27:0/3268702351 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecb40ad460 con 0x7fecc40cf460 2024-01-15T18:18:16.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.285+0000 7fecc1600640 1 -- 172.21.15.27:0/3268702351 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb40ad7f0 con 0x7fecc40cf460 2024-01-15T18:18:16.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.285+0000 7fecc9a00640 1 -- 172.21.15.27:0/3268702351 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fecc40cf460 msgr2=0x7fecc40cf840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:16.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.285+0000 7fecc9a00640 1 --2- 172.21.15.27:0/3268702351 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fecc40cf460 0x7fecc40cf840 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fecb40097f0 tx=0x7fecb401c3e0 comp rx=0 tx=0).stop 2024-01-15T18:18:16.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.285+0000 7fecc9a00640 1 -- 172.21.15.27:0/3268702351 shutdown_connections 2024-01-15T18:18:16.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.285+0000 7fecc9a00640 1 --2- 172.21.15.27:0/3268702351 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fecc40cf460 0x7fecc40cf840 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:16.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.285+0000 7fecc9a00640 1 -- 172.21.15.27:0/3268702351 >> 172.21.15.27:0/3268702351 conn(0x7fecc40d0e80 msgr2=0x7fecc40d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:16.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.289+0000 7fecc9a00640 1 -- 172.21.15.27:0/3268702351 shutdown_connections 2024-01-15T18:18:16.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.289+0000 7fecc9a00640 1 -- 172.21.15.27:0/3268702351 wait complete. 2024-01-15T18:18:16.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.289+0000 7fecc9a00640 1 Processor -- start 2024-01-15T18:18:16.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.289+0000 7fecc9a00640 1 -- start start 2024-01-15T18:18:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.289+0000 7fecc9a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecc40b2590 0x7fecc406f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.289+0000 7fecc9a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fecc40cf460 0x7fecc4148dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.289+0000 7fecc9a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fecc415b4b0 0x7fecc40c8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.289+0000 7fecc9a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc40caed0 con 0x7fecc40b2590 2024-01-15T18:18:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.289+0000 7fecc9a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc40ca250 con 0x7fecc415b4b0 2024-01-15T18:18:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.289+0000 7fecc9a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecc40ca530 con 0x7fecc40cf460 2024-01-15T18:18:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.289+0000 7fecc2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecc40b2590 0x7fecc406f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.289+0000 7fecc2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecc40b2590 0x7fecc406f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33758/0 (socket says 172.21.15.27:33758) 2024-01-15T18:18:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.289+0000 7fecc2a00640 1 -- 172.21.15.27:0/3812554527 learned_addr learned my addr 172.21.15.27:0/3812554527 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:16.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.289+0000 7fecc2000640 1 --2- 172.21.15.27:0/3812554527 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fecc40cf460 0x7fecc4148dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:16.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.289+0000 7fecc3400640 1 --2- 172.21.15.27:0/3812554527 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fecc415b4b0 0x7fecc40c8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.289+0000 7fecc2a00640 1 -- 172.21.15.27:0/3812554527 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fecc40cf460 msgr2=0x7fecc4148dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.289+0000 7fecc2a00640 1 --2- 172.21.15.27:0/3812554527 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fecc40cf460 0x7fecc4148dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.289+0000 7fecc2a00640 1 -- 172.21.15.27:0/3812554527 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fecc415b4b0 msgr2=0x7fecc40c8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.289+0000 7fecc2a00640 1 --2- 172.21.15.27:0/3812554527 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fecc415b4b0 0x7fecc40c8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.289+0000 7fecc2a00640 1 -- 172.21.15.27:0/3812554527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb4009420 con 0x7fecc40b2590 2024-01-15T18:18:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.289+0000 7fecc2a00640 1 --2- 172.21.15.27:0/3812554527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecc40b2590 0x7fecc406f860 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7fecb40b0040 tx=0x7fecb4019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.293+0000 7fecbaa00640 1 -- 172.21.15.27:0/3812554527 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb4004020 con 0x7fecc40b2590 2024-01-15T18:18:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.293+0000 7fecbaa00640 1 -- 172.21.15.27:0/3812554527 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecb40041a0 con 0x7fecc40b2590 2024-01-15T18:18:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.293+0000 7fecbaa00640 1 -- 172.21.15.27:0/3812554527 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb4003620 con 0x7fecc40b2590 2024-01-15T18:18:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.293+0000 7fecc9a00640 1 -- 172.21.15.27:0/3812554527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc40ca810 con 0x7fecc40b2590 2024-01-15T18:18:16.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.293+0000 7fecc9a00640 1 -- 172.21.15.27:0/3812554527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc40c6d70 con 0x7fecc40b2590 2024-01-15T18:18:16.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.293+0000 7fecc9a00640 1 -- 172.21.15.27:0/3812554527 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecc40cb010 con 0x7fecc40b2590 2024-01-15T18:18:16.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.293+0000 7fecbaa00640 1 -- 172.21.15.27:0/3812554527 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 36) v1 ==== 87588+0+0 (secure 0 0 0) 0x7fecb40b4040 con 0x7fecc40b2590 2024-01-15T18:18:16.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.293+0000 7fecbaa00640 1 --2- 172.21.15.27:0/3812554527 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fec9c0678b0 0x7fec9c069d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:16.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.293+0000 7fecbaa00640 1 -- 172.21.15.27:0/3812554527 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fecb411d250 con 0x7fecc40b2590 2024-01-15T18:18:16.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.297+0000 7fecc2000640 1 --2- 172.21.15.27:0/3812554527 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fec9c0678b0 0x7fec9c069d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:16.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.297+0000 7fecc2000640 1 --2- 172.21.15.27:0/3812554527 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fec9c0678b0 0x7fec9c069d70 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fecc4157bc0 tx=0x7fecac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:16.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.301+0000 7fecbaa00640 1 -- 172.21.15.27:0/3812554527 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fecc40cb010 con 0x7fecc40b2590 2024-01-15T18:18:16.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:16.477+0000 7fecc9a00640 1 -- 172.21.15.27:0/3812554527 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7fecc4067f60 con 0x7fec9c0678b0 2024-01-15T18:18:17.042 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-01-15T18:18:17.041+0000 7f053dc00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-15T18:18:17.114 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-15T18:18:17.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:17.197+0000 7fecbaa00640 1 -- 172.21.15.27:0/3812554527 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 37) v1 ==== 87690+0+0 (secure 0 0 0) 0x7fecb4123030 con 0x7fecc40b2590 2024-01-15T18:18:19.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:19.205+0000 7fecbaa00640 1 -- 172.21.15.27:0/3812554527 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7fecb40ec050 con 0x7fecc40b2590 2024-01-15T18:18:20.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.069+0000 7fecbaa00640 1 -- 172.21.15.27:0/3812554527 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fecc4067f60 con 0x7fec9c0678b0 2024-01-15T18:18:20.074 INFO:teuthology.orchestra.run.smithi027.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-15T18:18:20.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.073+0000 7fecc9a00640 1 -- 172.21.15.27:0/3812554527 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fec9c0678b0 msgr2=0x7fec9c069d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:20.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.073+0000 7fecc9a00640 1 --2- 172.21.15.27:0/3812554527 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fec9c0678b0 0x7fec9c069d70 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fecc4157bc0 tx=0x7fecac009040 comp rx=0 tx=0).stop 2024-01-15T18:18:20.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.073+0000 7fecc9a00640 1 -- 172.21.15.27:0/3812554527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecc40b2590 msgr2=0x7fecc406f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:20.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.073+0000 7fecc9a00640 1 --2- 172.21.15.27:0/3812554527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecc40b2590 0x7fecc406f860 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7fecb40b0040 tx=0x7fecb4019180 comp rx=0 tx=0).stop 2024-01-15T18:18:20.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.073+0000 7fecc9a00640 1 -- 172.21.15.27:0/3812554527 shutdown_connections 2024-01-15T18:18:20.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.073+0000 7fecc9a00640 1 --2- 172.21.15.27:0/3812554527 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fec9c0678b0 0x7fec9c069d70 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:20.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.073+0000 7fecc9a00640 1 --2- 172.21.15.27:0/3812554527 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fecc415b4b0 0x7fecc40c8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:20.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.073+0000 7fecc9a00640 1 --2- 172.21.15.27:0/3812554527 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fecc40cf460 0x7fecc4148dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:20.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.077+0000 7fecc9a00640 1 --2- 172.21.15.27:0/3812554527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fecc40b2590 0x7fecc406f860 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:20.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.077+0000 7fecc9a00640 1 -- 172.21.15.27:0/3812554527 >> 172.21.15.27:0/3812554527 conn(0x7fecc40d0e80 msgr2=0x7fecc40adb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:20.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.077+0000 7fecc9a00640 1 -- 172.21.15.27:0/3812554527 shutdown_connections 2024-01-15T18:18:20.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.077+0000 7fecc9a00640 1 -- 172.21.15.27:0/3812554527 wait complete. 2024-01-15T18:18:20.115 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T18:18:20.115 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:18:20.186 DEBUG:teuthology.orchestra.run.smithi093:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:18:20.309 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T18:18:20.310 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:18:20.310 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:18:20.319 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:18:20.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.505+0000 7f9717800640 1 Processor -- start 2024-01-15T18:18:20.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.505+0000 7f9717800640 1 -- start start 2024-01-15T18:18:20.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.509+0000 7f9717800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f971015b1f0 0x7f97100cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:20.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.509+0000 7f9717800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f9710072190 0x7f971006f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:20.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.509+0000 7f9717800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97100cf930 con 0x7f97100d08f0 2024-01-15T18:18:20.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.509+0000 7f9717800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97100af490 con 0x7f9710072190 2024-01-15T18:18:20.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.509+0000 7f9717800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f971015b5d0 con 0x7f971015b1f0 2024-01-15T18:18:20.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.509+0000 7f970fe00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f9710072190 0x7f971006f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:20.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.509+0000 7f9714e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f971015b1f0 0x7f97100cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:20.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.509+0000 7f9714e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f971015b1f0 0x7f97100cf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:49920/0 (socket says 172.21.15.27:49920) 2024-01-15T18:18:20.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.509+0000 7f970fe00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f9710072190 0x7f971006f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:58098/0 (socket says 172.21.15.27:58098) 2024-01-15T18:18:20.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.509+0000 7f9714e00640 1 -- 172.21.15.27:0/3729713204 learned_addr learned my addr 172.21.15.27:0/3729713204 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:20.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.509+0000 7f970fe00640 1 -- 172.21.15.27:0/3729713204 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f971015b1f0 msgr2=0x7f97100cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:20.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.509+0000 7f970fe00640 1 --2- 172.21.15.27:0/3729713204 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f971015b1f0 0x7f97100cf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:20.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.509+0000 7f970fe00640 1 -- 172.21.15.27:0/3729713204 >> v1:172.21.15.93:6789/0 conn(0x7f97100d08f0 legacy=0x7f971015ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:20.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.509+0000 7f970fe00640 1 -- 172.21.15.27:0/3729713204 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97100cfbb0 con 0x7f9710072190 2024-01-15T18:18:20.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.513+0000 7f970fe00640 1 --2- 172.21.15.27:0/3729713204 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9710072190 0x7f971006f3c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f96f80097c0 tx=0x7f96f801c440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88769e36a541f6c6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:20.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.517+0000 7f970f400640 1 -- 172.21.15.27:0/3729713204 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f80ab400 con 0x7f9710072190 2024-01-15T18:18:20.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.517+0000 7f970f400640 1 -- 172.21.15.27:0/3729713204 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f80a7050 con 0x7f9710072190 2024-01-15T18:18:20.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.517+0000 7f970f400640 1 -- 172.21.15.27:0/3729713204 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f80ab780 con 0x7f9710072190 2024-01-15T18:18:20.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.517+0000 7f9717800640 1 -- 172.21.15.27:0/3729713204 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9710072190 msgr2=0x7f971006f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:20.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.517+0000 7f9717800640 1 --2- 172.21.15.27:0/3729713204 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9710072190 0x7f971006f3c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f96f80097c0 tx=0x7f96f801c440 comp rx=0 tx=0).stop 2024-01-15T18:18:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.517+0000 7f9717800640 1 -- 172.21.15.27:0/3729713204 shutdown_connections 2024-01-15T18:18:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.517+0000 7f9717800640 1 --2- 172.21.15.27:0/3729713204 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9710072190 0x7f971006f3c0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.517+0000 7f9717800640 1 --2- 172.21.15.27:0/3729713204 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f971015b1f0 0x7f97100cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.517+0000 7f9717800640 1 -- 172.21.15.27:0/3729713204 >> 172.21.15.27:0/3729713204 conn(0x7f97100cfdd0 msgr2=0x7f97100d01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:20.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.517+0000 7f9717800640 1 -- 172.21.15.27:0/3729713204 shutdown_connections 2024-01-15T18:18:20.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.517+0000 7f9717800640 1 -- 172.21.15.27:0/3729713204 wait complete. 2024-01-15T18:18:20.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.521+0000 7f9717800640 1 Processor -- start 2024-01-15T18:18:20.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.521+0000 7f9717800640 1 -- start start 2024-01-15T18:18:20.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.521+0000 7f9717800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9710072190 0x7f97101fdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:20.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.521+0000 7f9717800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f97100d08f0 0x7f971014dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:20.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.521+0000 7f9717800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f971015b1f0 0x7f971014f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:20.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.521+0000 7f9717800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97101feb00 con 0x7f9710072190 2024-01-15T18:18:20.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.521+0000 7f9717800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97101fa5e0 con 0x7f971015b1f0 2024-01-15T18:18:20.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.521+0000 7f9717800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f971015db10 con 0x7f97100d08f0 2024-01-15T18:18:20.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.525+0000 7f9714e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9710072190 0x7f97101fdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:20.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.525+0000 7f9714e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9710072190 0x7f97101fdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33770/0 (socket says 172.21.15.27:33770) 2024-01-15T18:18:20.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.525+0000 7f9714e00640 1 -- 172.21.15.27:0/1963342157 learned_addr learned my addr 172.21.15.27:0/1963342157 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:20.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.525+0000 7f970fe00640 1 --2- 172.21.15.27:0/1963342157 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f97100d08f0 0x7f971014dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:20.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.525+0000 7f9715800640 1 --2- 172.21.15.27:0/1963342157 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f971015b1f0 0x7f971014f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:20.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.525+0000 7f9714e00640 1 -- 172.21.15.27:0/1963342157 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f97100d08f0 msgr2=0x7f971014dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:20.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.525+0000 7f9714e00640 1 --2- 172.21.15.27:0/1963342157 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f97100d08f0 0x7f971014dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:20.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.525+0000 7f9714e00640 1 -- 172.21.15.27:0/1963342157 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f971015b1f0 msgr2=0x7f971014f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:20.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.525+0000 7f9714e00640 1 --2- 172.21.15.27:0/1963342157 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f971015b1f0 0x7f971014f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:20.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.525+0000 7f9714e00640 1 -- 172.21.15.27:0/1963342157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f8009420 con 0x7f9710072190 2024-01-15T18:18:20.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.525+0000 7f9715800640 1 --2- 172.21.15.27:0/1963342157 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f971015b1f0 0x7f971014f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:18:20.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.525+0000 7f9714e00640 1 --2- 172.21.15.27:0/1963342157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9710072190 0x7f97101fdbd0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f970000baf0 tx=0x7f970000f0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:20.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.525+0000 7f970cc00640 1 -- 172.21.15.27:0/1963342157 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970000dac0 con 0x7f9710072190 2024-01-15T18:18:20.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.525+0000 7f970cc00640 1 -- 172.21.15.27:0/1963342157 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f970000f9c0 con 0x7f9710072190 2024-01-15T18:18:20.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.525+0000 7f9717800640 1 -- 172.21.15.27:0/1963342157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97101fa880 con 0x7f9710072190 2024-01-15T18:18:20.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.525+0000 7f9717800640 1 -- 172.21.15.27:0/1963342157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97101fe260 con 0x7f9710072190 2024-01-15T18:18:20.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.525+0000 7f970cc00640 1 -- 172.21.15.27:0/1963342157 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970000fd70 con 0x7f9710072190 2024-01-15T18:18:20.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.529+0000 7f9717800640 1 -- 172.21.15.27:0/1963342157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96e0005260 con 0x7f9710072190 2024-01-15T18:18:20.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.529+0000 7f970cc00640 1 -- 172.21.15.27:0/1963342157 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f9700014070 con 0x7f9710072190 2024-01-15T18:18:20.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.533+0000 7f970cc00640 1 --2- 172.21.15.27:0/1963342157 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f96e8067940 0x7f96e8069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:20.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.533+0000 7f970cc00640 1 -- 172.21.15.27:0/1963342157 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f9700082890 con 0x7f9710072190 2024-01-15T18:18:20.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.537+0000 7f970fe00640 1 --2- 172.21.15.27:0/1963342157 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f96e8067940 0x7f96e8069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:20.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.537+0000 7f970fe00640 1 --2- 172.21.15.27:0/1963342157 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f96e8067940 0x7f96e8069e00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f96f80ad000 tx=0x7f96f80a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:20.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.541+0000 7f970cc00640 1 -- 172.21.15.27:0/1963342157 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f97000512b0 con 0x7f9710072190 2024-01-15T18:18:20.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.805+0000 7f9717800640 1 -- 172.21.15.27:0/1963342157 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96e00057e0 con 0x7f9710072190 2024-01-15T18:18:20.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.805+0000 7f970cc00640 1 -- 172.21.15.27:0/1963342157 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+13067 (secure 0 0 0) 0x7f97000548d0 con 0x7f9710072190 2024-01-15T18:18:20.807 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:18:20.807 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":100,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:18:20.055597+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6835/259930036":"2024-01-16T18:18:17.046652+0000","172.21.15.27:6833/259930036":"2024-01-16T18:18:17.046652+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:18:20.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.809+0000 7f9717800640 1 -- 172.21.15.27:0/1963342157 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f96e8067940 msgr2=0x7f96e8069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.809+0000 7f9717800640 1 --2- 172.21.15.27:0/1963342157 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f96e8067940 0x7f96e8069e00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f96f80ad000 tx=0x7f96f80a6000 comp rx=0 tx=0).stop 2024-01-15T18:18:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.809+0000 7f9717800640 1 -- 172.21.15.27:0/1963342157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9710072190 msgr2=0x7f97101fdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.809+0000 7f9717800640 1 --2- 172.21.15.27:0/1963342157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9710072190 0x7f97101fdbd0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f970000baf0 tx=0x7f970000f0b0 comp rx=0 tx=0).stop 2024-01-15T18:18:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.809+0000 7f9717800640 1 -- 172.21.15.27:0/1963342157 shutdown_connections 2024-01-15T18:18:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.809+0000 7f9717800640 1 --2- 172.21.15.27:0/1963342157 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f96e8067940 0x7f96e8069e00 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.809+0000 7f9717800640 1 --2- 172.21.15.27:0/1963342157 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f971015b1f0 0x7f971014f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.809+0000 7f9717800640 1 --2- 172.21.15.27:0/1963342157 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f97100d08f0 0x7f971014dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.809+0000 7f9717800640 1 --2- 172.21.15.27:0/1963342157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9710072190 0x7f97101fdbd0 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:20.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.809+0000 7f9717800640 1 -- 172.21.15.27:0/1963342157 >> 172.21.15.27:0/1963342157 conn(0x7f97100cfdd0 msgr2=0x7f97100b0020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:20.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.809+0000 7f9717800640 1 -- 172.21.15.27:0/1963342157 shutdown_connections 2024-01-15T18:18:20.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.809+0000 7f9717800640 1 -- 172.21.15.27:0/1963342157 wait complete. 2024-01-15T18:18:20.829 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:18:20.830 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:18:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.973+0000 7f10b2c00640 1 Processor -- start 2024-01-15T18:18:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.973+0000 7f10b2c00640 1 -- start start 2024-01-15T18:18:20.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.973+0000 7f10b2c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f10ac099a10 0x7f10ac099e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:20.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.977+0000 7f10b2c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10ac09a360 con 0x7f10ac15b350 2024-01-15T18:18:20.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.977+0000 7f10b2c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10ac0ccd50 con 0x7f10ac09a5b0 2024-01-15T18:18:20.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.977+0000 7f10b2c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10ac0cebc0 con 0x7f10ac099a10 2024-01-15T18:18:20.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.977+0000 7f10abe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f10ac15b350 0x7f10ac0cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44222/0 (socket says 172.21.15.27:44222) 2024-01-15T18:18:20.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.977+0000 7f10abe00640 1 -- 172.21.15.27:0/2014298819 learned_addr learned my addr 172.21.15.27:0/2014298819 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:20.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.977+0000 7f10b0c00640 1 --2- 172.21.15.27:0/2014298819 >> v2:172.21.15.93:3300/0 conn(0x7f10ac099a10 0x7f10ac099e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:20.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.977+0000 7f10aaa00640 1 -- 172.21.15.27:0/2014298819 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3732809529 0 0) 0x7f10ac09a360 con 0x7f10ac15b350 2024-01-15T18:18:20.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.977+0000 7f10aaa00640 1 -- 172.21.15.27:0/2014298819 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f108c003530 con 0x7f10ac15b350 2024-01-15T18:18:20.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.977+0000 7f10b0c00640 1 -- 172.21.15.27:0/2014298819 >> v1:172.21.15.93:6789/0 conn(0x7f10ac09a5b0 legacy=0x7f10ac0cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:20.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.977+0000 7f10b0c00640 1 -- 172.21.15.27:0/2014298819 >> v1:172.21.15.27:6789/0 conn(0x7f10ac15b350 legacy=0x7f10ac0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:20.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.977+0000 7f10b0c00640 1 -- 172.21.15.27:0/2014298819 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ac0ceea0 con 0x7f10ac099a10 2024-01-15T18:18:20.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.977+0000 7f10b0c00640 1 --2- 172.21.15.27:0/2014298819 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f10ac099a10 0x7f10ac099e20 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f10a4009d90 tx=0x7f10a401c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5bc788f7a2f0f96 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:20.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.981+0000 7f10aaa00640 1 -- 172.21.15.27:0/2014298819 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a40a5970 con 0x7f10ac099a10 2024-01-15T18:18:20.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.981+0000 7f10aaa00640 1 -- 172.21.15.27:0/2014298819 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a40a5ad0 con 0x7f10ac099a10 2024-01-15T18:18:20.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.981+0000 7f10aaa00640 1 -- 172.21.15.27:0/2014298819 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a40a5e60 con 0x7f10ac099a10 2024-01-15T18:18:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.981+0000 7f10b2c00640 1 -- 172.21.15.27:0/2014298819 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f10ac099a10 msgr2=0x7f10ac099e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.981+0000 7f10b2c00640 1 --2- 172.21.15.27:0/2014298819 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f10ac099a10 0x7f10ac099e20 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f10a4009d90 tx=0x7f10a401c890 comp rx=0 tx=0).stop 2024-01-15T18:18:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.981+0000 7f10b2c00640 1 -- 172.21.15.27:0/2014298819 shutdown_connections 2024-01-15T18:18:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.981+0000 7f10b2c00640 1 --2- 172.21.15.27:0/2014298819 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f10ac099a10 0x7f10ac099e20 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.981+0000 7f10b2c00640 1 -- 172.21.15.27:0/2014298819 >> 172.21.15.27:0/2014298819 conn(0x7f10ac0b2a80 msgr2=0x7f10ac0b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.981+0000 7f10b2c00640 1 -- 172.21.15.27:0/2014298819 shutdown_connections 2024-01-15T18:18:20.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.981+0000 7f10b2c00640 1 -- 172.21.15.27:0/2014298819 wait complete. 2024-01-15T18:18:20.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.981+0000 7f10b2c00640 1 Processor -- start 2024-01-15T18:18:20.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.981+0000 7f10b2c00640 1 -- start start 2024-01-15T18:18:20.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.981+0000 7f10b2c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f10ac099a10 0x7f10ac14e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:20.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.981+0000 7f10b2c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10ac09a5b0 0x7f10ac1570b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:20.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.981+0000 7f10b2c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f10ac15b350 0x7f10ac1575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:20.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.981+0000 7f10b2c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ac06d440 con 0x7f10ac09a5b0 2024-01-15T18:18:20.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.981+0000 7f10b2c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ac153a00 con 0x7f10ac099a10 2024-01-15T18:18:20.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.985+0000 7f10b2c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10ac153ce0 con 0x7f10ac15b350 2024-01-15T18:18:20.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.985+0000 7f10ab400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10ac09a5b0 0x7f10ac1570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:20.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.985+0000 7f10ab400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10ac09a5b0 0x7f10ac1570b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33774/0 (socket says 172.21.15.27:33774) 2024-01-15T18:18:20.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.985+0000 7f10ab400640 1 -- 172.21.15.27:0/654339842 learned_addr learned my addr 172.21.15.27:0/654339842 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:20.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.985+0000 7f10abe00640 1 --2- 172.21.15.27:0/654339842 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f10ac099a10 0x7f10ac14e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:20.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.985+0000 7f10b0c00640 1 --2- 172.21.15.27:0/654339842 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f10ac15b350 0x7f10ac1575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:20.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.985+0000 7f10ab400640 1 -- 172.21.15.27:0/654339842 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f10ac15b350 msgr2=0x7f10ac1575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:20.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.985+0000 7f10ab400640 1 --2- 172.21.15.27:0/654339842 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f10ac15b350 0x7f10ac1575f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:20.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.985+0000 7f10ab400640 1 -- 172.21.15.27:0/654339842 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f10ac099a10 msgr2=0x7f10ac14e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:20.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.985+0000 7f10ab400640 1 --2- 172.21.15.27:0/654339842 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f10ac099a10 0x7f10ac14e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:20.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.985+0000 7f10ab400640 1 -- 172.21.15.27:0/654339842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a4009a70 con 0x7f10ac09a5b0 2024-01-15T18:18:20.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.985+0000 7f10ab400640 1 --2- 172.21.15.27:0/654339842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10ac09a5b0 0x7f10ac1570b0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f109400c500 tx=0x7f109400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:20.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.985+0000 7f10a3e00640 1 -- 172.21.15.27:0/654339842 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1094005220 con 0x7f10ac09a5b0 2024-01-15T18:18:20.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.985+0000 7f10a3e00640 1 -- 172.21.15.27:0/654339842 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1094005380 con 0x7f10ac09a5b0 2024-01-15T18:18:20.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.985+0000 7f10b2c00640 1 -- 172.21.15.27:0/654339842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10ac153fc0 con 0x7f10ac09a5b0 2024-01-15T18:18:20.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.985+0000 7f10a3e00640 1 -- 172.21.15.27:0/654339842 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1094007270 con 0x7f10ac09a5b0 2024-01-15T18:18:20.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.985+0000 7f10b2c00640 1 -- 172.21.15.27:0/654339842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10ac14eb70 con 0x7f10ac09a5b0 2024-01-15T18:18:20.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.985+0000 7f10b2c00640 1 -- 172.21.15.27:0/654339842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10ac0cc370 con 0x7f10ac09a5b0 2024-01-15T18:18:20.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.985+0000 7f10a3e00640 1 -- 172.21.15.27:0/654339842 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f10940073d0 con 0x7f10ac09a5b0 2024-01-15T18:18:20.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.989+0000 7f10a3e00640 1 --2- 172.21.15.27:0/654339842 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1084067940 0x7f1084069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:20.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.989+0000 7f10a3e00640 1 -- 172.21.15.27:0/654339842 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f1094082be0 con 0x7f10ac09a5b0 2024-01-15T18:18:20.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.989+0000 7f10abe00640 1 --2- 172.21.15.27:0/654339842 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1084067940 0x7f1084069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:20.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.989+0000 7f10abe00640 1 --2- 172.21.15.27:0/654339842 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1084067940 0x7f1084069e00 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f109c0011f0 tx=0x7f109c007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:20.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:20.993+0000 7f10a3e00640 1 -- 172.21.15.27:0/654339842 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f10940b7280 con 0x7f10ac09a5b0 2024-01-15T18:18:21.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.149+0000 7f10b2c00640 1 -- 172.21.15.27:0/654339842 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f10ac04e850 con 0x7f10ac09a5b0 2024-01-15T18:18:21.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.153+0000 7f10a3e00640 1 -- 172.21.15.27:0/654339842 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1094051580 con 0x7f10ac09a5b0 2024-01-15T18:18:21.152 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:18:21.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.153+0000 7f10b2c00640 1 -- 172.21.15.27:0/654339842 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1084067940 msgr2=0x7f1084069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:21.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.153+0000 7f10b2c00640 1 --2- 172.21.15.27:0/654339842 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1084067940 0x7f1084069e00 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f109c0011f0 tx=0x7f109c007d40 comp rx=0 tx=0).stop 2024-01-15T18:18:21.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.153+0000 7f10b2c00640 1 -- 172.21.15.27:0/654339842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10ac09a5b0 msgr2=0x7f10ac1570b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:21.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.153+0000 7f10b2c00640 1 --2- 172.21.15.27:0/654339842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10ac09a5b0 0x7f10ac1570b0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f109400c500 tx=0x7f109400c9c0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.153+0000 7f10b2c00640 1 -- 172.21.15.27:0/654339842 shutdown_connections 2024-01-15T18:18:21.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.153+0000 7f10b2c00640 1 --2- 172.21.15.27:0/654339842 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1084067940 0x7f1084069e00 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.153+0000 7f10b2c00640 1 --2- 172.21.15.27:0/654339842 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f10ac15b350 0x7f10ac1575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.153+0000 7f10b2c00640 1 --2- 172.21.15.27:0/654339842 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f10ac09a5b0 0x7f10ac1570b0 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.153+0000 7f10b2c00640 1 --2- 172.21.15.27:0/654339842 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f10ac099a10 0x7f10ac14e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.153+0000 7f10b2c00640 1 -- 172.21.15.27:0/654339842 >> 172.21.15.27:0/654339842 conn(0x7f10ac0b2a80 msgr2=0x7f10ac1595c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:21.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.153+0000 7f10b2c00640 1 -- 172.21.15.27:0/654339842 shutdown_connections 2024-01-15T18:18:21.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.153+0000 7f10b2c00640 1 -- 172.21.15.27:0/654339842 wait complete. 2024-01-15T18:18:21.175 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:18:21.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.317+0000 7f24ac200640 1 Processor -- start 2024-01-15T18:18:21.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.317+0000 7f24ac200640 1 -- start start 2024-01-15T18:18:21.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.317+0000 7f24ac200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f24a409a120 0x7f24a40cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:21.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.317+0000 7f24ac200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f24a40b35a0 0x7f24a40cc530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:21.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.317+0000 7f24ac200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a409a700 con 0x7f24a415e7d0 2024-01-15T18:18:21.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.317+0000 7f24ac200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24a40cd870 con 0x7f24a409a120 2024-01-15T18:18:21.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.317+0000 7f24ac200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24a40cd590 con 0x7f24a40b35a0 2024-01-15T18:18:21.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.317+0000 7f24a8e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f24a409a120 0x7f24a40cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:21.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.317+0000 7f24a8e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f24a409a120 0x7f24a40cbff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33784/0 (socket says 172.21.15.27:33784) 2024-01-15T18:18:21.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.317+0000 7f24a8e00640 1 -- 172.21.15.27:0/3103268647 learned_addr learned my addr 172.21.15.27:0/3103268647 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:21.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.317+0000 7f24aa200640 1 --2- 172.21.15.27:0/3103268647 >> v2:172.21.15.93:3300/0 conn(0x7f24a40b35a0 0x7f24a40cc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:21.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.317+0000 7f24a8e00640 1 -- 172.21.15.27:0/3103268647 >> v2:172.21.15.93:3300/0 conn(0x7f24a40b35a0 msgr2=0x7f24a40cc530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:21.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.317+0000 7f24a8e00640 1 --2- 172.21.15.27:0/3103268647 >> v2:172.21.15.93:3300/0 conn(0x7f24a40b35a0 0x7f24a40cc530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.317+0000 7f24a8e00640 1 -- 172.21.15.27:0/3103268647 >> v1:172.21.15.93:6789/0 conn(0x7f24a415e7d0 legacy=0x7f24a4099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:21.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.317+0000 7f24a8e00640 1 -- 172.21.15.27:0/3103268647 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a40cfa00 con 0x7f24a409a120 2024-01-15T18:18:21.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.317+0000 7f24aa200640 1 --2- 172.21.15.27:0/3103268647 >> v2:172.21.15.93:3300/0 conn(0x7f24a40b35a0 0x7f24a40cc530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:18:21.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.321+0000 7f24a8e00640 1 --2- 172.21.15.27:0/3103268647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a409a120 0x7f24a40cbff0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f24a40671e0 tx=0x7f248c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88c4cb19bcd319b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:21.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.321+0000 7f24a3e00640 1 -- 172.21.15.27:0/3103268647 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c0aa650 con 0x7f24a409a120 2024-01-15T18:18:21.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.321+0000 7f24a3e00640 1 -- 172.21.15.27:0/3103268647 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f248c0aa7b0 con 0x7f24a409a120 2024-01-15T18:18:21.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.321+0000 7f24a3e00640 1 -- 172.21.15.27:0/3103268647 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f248c0aab40 con 0x7f24a409a120 2024-01-15T18:18:21.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.321+0000 7f24ac200640 1 -- 172.21.15.27:0/3103268647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a409a120 msgr2=0x7f24a40cbff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:21.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.325+0000 7f24ac200640 1 --2- 172.21.15.27:0/3103268647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a409a120 0x7f24a40cbff0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f24a40671e0 tx=0x7f248c01c4d0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.325+0000 7f24ac200640 1 -- 172.21.15.27:0/3103268647 shutdown_connections 2024-01-15T18:18:21.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.325+0000 7f24ac200640 1 --2- 172.21.15.27:0/3103268647 >> v2:172.21.15.93:3300/0 conn(0x7f24a40b35a0 0x7f24a40cc530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.325+0000 7f24ac200640 1 --2- 172.21.15.27:0/3103268647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a409a120 0x7f24a40cbff0 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.325+0000 7f24ac200640 1 -- 172.21.15.27:0/3103268647 >> 172.21.15.27:0/3103268647 conn(0x7f24a40b2a80 msgr2=0x7f24a40b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:21.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.325+0000 7f24ac200640 1 -- 172.21.15.27:0/3103268647 shutdown_connections 2024-01-15T18:18:21.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.325+0000 7f24ac200640 1 -- 172.21.15.27:0/3103268647 wait complete. 2024-01-15T18:18:21.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.325+0000 7f24ac200640 1 Processor -- start 2024-01-15T18:18:21.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.325+0000 7f24ac200640 1 -- start start 2024-01-15T18:18:21.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.325+0000 7f24ac200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f24a409a120 0x7f24a4159a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:21.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.325+0000 7f24ac200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f24a40b35a0 0x7f24a4157490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:21.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.325+0000 7f24ac200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a415e7d0 0x7f24a4153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:21.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.325+0000 7f24ac200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a41579d0 con 0x7f24a415e7d0 2024-01-15T18:18:21.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.325+0000 7f24ac200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a4153940 con 0x7f24a40b35a0 2024-01-15T18:18:21.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.325+0000 7f24ac200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a4157c00 con 0x7f24a409a120 2024-01-15T18:18:21.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.325+0000 7f24aa200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a415e7d0 0x7f24a4153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:21.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.325+0000 7f24aa200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a415e7d0 0x7f24a4153400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33786/0 (socket says 172.21.15.27:33786) 2024-01-15T18:18:21.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.325+0000 7f24aa200640 1 -- 172.21.15.27:0/602625781 learned_addr learned my addr 172.21.15.27:0/602625781 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:21.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.325+0000 7f24a8e00640 1 --2- 172.21.15.27:0/602625781 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f24a40b35a0 0x7f24a4157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:21.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.325+0000 7f24a9800640 1 --2- 172.21.15.27:0/602625781 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f24a409a120 0x7f24a4159a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:21.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.325+0000 7f24aa200640 1 -- 172.21.15.27:0/602625781 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f24a409a120 msgr2=0x7f24a4159a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:21.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.325+0000 7f24aa200640 1 --2- 172.21.15.27:0/602625781 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f24a409a120 0x7f24a4159a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.325+0000 7f24aa200640 1 -- 172.21.15.27:0/602625781 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f24a40b35a0 msgr2=0x7f24a4157490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:21.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.325+0000 7f24aa200640 1 --2- 172.21.15.27:0/602625781 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f24a40b35a0 0x7f24a4157490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.329+0000 7f24aa200640 1 -- 172.21.15.27:0/602625781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248c0094d0 con 0x7f24a415e7d0 2024-01-15T18:18:21.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.329+0000 7f24aa200640 1 --2- 172.21.15.27:0/602625781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a415e7d0 0x7f24a4153400 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f249c007ef0 tx=0x7f249c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:21.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.329+0000 7f24a1600640 1 -- 172.21.15.27:0/602625781 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c003ba0 con 0x7f24a415e7d0 2024-01-15T18:18:21.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.329+0000 7f24a1600640 1 -- 172.21.15.27:0/602625781 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f249c00f040 con 0x7f24a415e7d0 2024-01-15T18:18:21.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.329+0000 7f24a1600640 1 -- 172.21.15.27:0/602625781 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c00de80 con 0x7f24a415e7d0 2024-01-15T18:18:21.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.329+0000 7f24ac200640 1 -- 172.21.15.27:0/602625781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a4157ee0 con 0x7f24a415e7d0 2024-01-15T18:18:21.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.329+0000 7f24ac200640 1 -- 172.21.15.27:0/602625781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a406b8e0 con 0x7f24a415e7d0 2024-01-15T18:18:21.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.329+0000 7f24ac200640 1 -- 172.21.15.27:0/602625781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a404e850 con 0x7f24a415e7d0 2024-01-15T18:18:21.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.329+0000 7f24a1600640 1 -- 172.21.15.27:0/602625781 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f249c004030 con 0x7f24a415e7d0 2024-01-15T18:18:21.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.329+0000 7f24a1600640 1 --2- 172.21.15.27:0/602625781 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f247c067940 0x7f247c069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:21.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.329+0000 7f24a1600640 1 -- 172.21.15.27:0/602625781 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f249c0834d0 con 0x7f24a415e7d0 2024-01-15T18:18:21.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.329+0000 7f24a9800640 1 --2- 172.21.15.27:0/602625781 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f247c067940 0x7f247c069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:21.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.333+0000 7f24a9800640 1 --2- 172.21.15.27:0/602625781 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f247c067940 0x7f247c069e00 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f2494007fd0 tx=0x7f2494009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:21.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.333+0000 7f24a1600640 1 -- 172.21.15.27:0/602625781 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f249c051ef0 con 0x7f24a415e7d0 2024-01-15T18:18:21.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.505+0000 7f24ac200640 1 -- 172.21.15.27:0/602625781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f24a40b6ed0 con 0x7f24a415e7d0 2024-01-15T18:18:21.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.505+0000 7f24a1600640 1 -- 172.21.15.27:0/602625781 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v100) v1 ==== 80+0+120 (secure 0 0 0) 0x7f249c01a020 con 0x7f24a415e7d0 2024-01-15T18:18:21.505 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6835/259930036 2024-01-16T18:18:17.046652+0000 2024-01-15T18:18:21.505 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6833/259930036 2024-01-16T18:18:17.046652+0000 2024-01-15T18:18:21.506 INFO:teuthology.orchestra.run.smithi027.stderr:listed 2 entries 2024-01-15T18:18:21.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.505+0000 7f24ac200640 1 -- 172.21.15.27:0/602625781 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f247c067940 msgr2=0x7f247c069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:21.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.505+0000 7f24ac200640 1 --2- 172.21.15.27:0/602625781 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f247c067940 0x7f247c069e00 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f2494007fd0 tx=0x7f2494009040 comp rx=0 tx=0).stop 2024-01-15T18:18:21.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.505+0000 7f24ac200640 1 -- 172.21.15.27:0/602625781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a415e7d0 msgr2=0x7f24a4153400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:21.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.505+0000 7f24ac200640 1 --2- 172.21.15.27:0/602625781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a415e7d0 0x7f24a4153400 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f249c007ef0 tx=0x7f249c00d4a0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.509+0000 7f24ac200640 1 -- 172.21.15.27:0/602625781 shutdown_connections 2024-01-15T18:18:21.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.509+0000 7f24ac200640 1 --2- 172.21.15.27:0/602625781 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f247c067940 0x7f247c069e00 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.509+0000 7f24ac200640 1 --2- 172.21.15.27:0/602625781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a415e7d0 0x7f24a4153400 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.509+0000 7f24ac200640 1 --2- 172.21.15.27:0/602625781 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f24a40b35a0 0x7f24a4157490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.509+0000 7f24ac200640 1 --2- 172.21.15.27:0/602625781 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f24a409a120 0x7f24a4159a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.509+0000 7f24ac200640 1 -- 172.21.15.27:0/602625781 >> 172.21.15.27:0/602625781 conn(0x7f24a40b2a80 msgr2=0x7f24a40cc340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:21.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.509+0000 7f24ac200640 1 -- 172.21.15.27:0/602625781 shutdown_connections 2024-01-15T18:18:21.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.509+0000 7f24ac200640 1 -- 172.21.15.27:0/602625781 wait complete. 2024-01-15T18:18:21.528 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-01-15T18:18:21.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:21.528 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-15T18:18:21.549 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T18:18:21.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:21.549 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-15T18:18:21.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[50522]: fuse finished with error 0 and tester_r 0 2024-01-15T18:18:21.695 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:18:21.695 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:18:21.705 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:18:21.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.881+0000 7f03d3400640 1 Processor -- start 2024-01-15T18:18:21.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.881+0000 7f03d3400640 1 -- start start 2024-01-15T18:18:21.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03d3400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f03cc0c9bd0 0x7f03cc06c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:21.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03d3400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03cc0ad180 con 0x7f03cc0ce710 2024-01-15T18:18:21.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03d3400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03cc0ac890 con 0x7f03cc15d0a0 2024-01-15T18:18:21.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03d3400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03cc0cda90 con 0x7f03cc0c9bd0 2024-01-15T18:18:21.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03cbe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f03cc0ce710 0x7f03cc0c94c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44238/0 (socket says 172.21.15.27:44238) 2024-01-15T18:18:21.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03cbe00640 1 -- 172.21.15.27:0/3109544986 learned_addr learned my addr 172.21.15.27:0/3109544986 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:21.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03d1400640 1 --2- 172.21.15.27:0/3109544986 >> v2:172.21.15.93:3300/0 conn(0x7f03cc0c9bd0 0x7f03cc06c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:21.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03cb400640 1 -- 172.21.15.27:0/3109544986 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034538938 0 0) 0x7f03cc0ad180 con 0x7f03cc0ce710 2024-01-15T18:18:21.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03cb400640 1 -- 172.21.15.27:0/3109544986 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03ac003530 con 0x7f03cc0ce710 2024-01-15T18:18:21.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03cb400640 1 -- 172.21.15.27:0/3109544986 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209719650 0 0) 0x7f03cc0ac890 con 0x7f03cc15d0a0 2024-01-15T18:18:21.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03cb400640 1 -- 172.21.15.27:0/3109544986 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03cc0ad180 con 0x7f03cc15d0a0 2024-01-15T18:18:21.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03cb400640 1 -- 172.21.15.27:0/3109544986 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2584350171 0 0) 0x7f03ac003530 con 0x7f03cc0ce710 2024-01-15T18:18:21.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03cb400640 1 -- 172.21.15.27:0/3109544986 >> v2:172.21.15.93:3300/0 conn(0x7f03cc0c9bd0 msgr2=0x7f03cc06c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:21.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03cb400640 1 --2- 172.21.15.27:0/3109544986 >> v2:172.21.15.93:3300/0 conn(0x7f03cc0c9bd0 0x7f03cc06c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03cb400640 1 -- 172.21.15.27:0/3109544986 >> v1:172.21.15.93:6789/0 conn(0x7f03cc15d0a0 legacy=0x7f03cc0cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:21.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03cb400640 1 -- 172.21.15.27:0/3109544986 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03cc0acb30 con 0x7f03cc0ce710 2024-01-15T18:18:21.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03cb400640 1 -- 172.21.15.27:0/3109544986 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f03b4001680 con 0x7f03cc0ce710 2024-01-15T18:18:21.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03d1400640 1 --2- 172.21.15.27:0/3109544986 >> v2:172.21.15.93:3300/0 conn(0x7f03cc0c9bd0 0x7f03cc06c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:18:21.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03cb400640 1 -- 172.21.15.27:0/3109544986 >> v1:172.21.15.27:6789/0 conn(0x7f03cc0ce710 legacy=0x7f03cc0c94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:21.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03cb400640 1 --2- 172.21.15.27:0/3109544986 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03ac003bd0 0x7f03ac003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:21.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03cb400640 1 --2- 172.21.15.27:0/3109544986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03ac0047d0 0x7f03ac004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:21.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03cb400640 1 --2- 172.21.15.27:0/3109544986 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03ac005190 0x7f03ac00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:21.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03cb400640 1 -- 172.21.15.27:0/3109544986 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc0cda90 con 0x7f03ac0047d0 2024-01-15T18:18:21.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03cb400640 1 -- 172.21.15.27:0/3109544986 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03ac00c130 con 0x7f03ac003bd0 2024-01-15T18:18:21.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03cb400640 1 -- 172.21.15.27:0/3109544986 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03ac00c2f0 con 0x7f03ac005190 2024-01-15T18:18:21.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03cbe00640 1 --2- 172.21.15.27:0/3109544986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03ac0047d0 0x7f03ac004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:21.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03d1400640 1 --2- 172.21.15.27:0/3109544986 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03ac005190 0x7f03ac00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:21.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03d0a00640 1 --2- 172.21.15.27:0/3109544986 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03ac003bd0 0x7f03ac003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:21.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03cbe00640 1 -- 172.21.15.27:0/3109544986 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03ac005190 msgr2=0x7f03ac00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:21.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03cbe00640 1 --2- 172.21.15.27:0/3109544986 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03ac005190 0x7f03ac00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03cbe00640 1 -- 172.21.15.27:0/3109544986 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03ac003bd0 msgr2=0x7f03ac003fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:21.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03cbe00640 1 --2- 172.21.15.27:0/3109544986 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03ac003bd0 0x7f03ac003fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03cbe00640 1 -- 172.21.15.27:0/3109544986 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ac003530 con 0x7f03ac0047d0 2024-01-15T18:18:21.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03d0a00640 1 --2- 172.21.15.27:0/3109544986 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03ac003bd0 0x7f03ac003fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:18:21.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.885+0000 7f03d1400640 1 -- 172.21.15.27:0/3109544986 reap_dead start 2024-01-15T18:18:21.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.889+0000 7f03cbe00640 1 --2- 172.21.15.27:0/3109544986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03ac0047d0 0x7f03ac004c50 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f03b4007fb0 tx=0x7f03b401caa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd983ce6b52fa609 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:21.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.889+0000 7f03cb400640 1 -- 172.21.15.27:0/3109544986 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b4005ae0 con 0x7f03ac0047d0 2024-01-15T18:18:21.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.889+0000 7f03cb400640 1 -- 172.21.15.27:0/3109544986 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b4005c60 con 0x7f03ac0047d0 2024-01-15T18:18:21.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.889+0000 7f03cb400640 1 -- 172.21.15.27:0/3109544986 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b40a59d0 con 0x7f03ac0047d0 2024-01-15T18:18:21.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.889+0000 7f03d3400640 1 -- 172.21.15.27:0/3109544986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03ac0047d0 msgr2=0x7f03ac004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:21.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.889+0000 7f03d3400640 1 --2- 172.21.15.27:0/3109544986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03ac0047d0 0x7f03ac004c50 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f03b4007fb0 tx=0x7f03b401caa0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.889+0000 7f03d3400640 1 -- 172.21.15.27:0/3109544986 shutdown_connections 2024-01-15T18:18:21.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.889+0000 7f03d3400640 1 --2- 172.21.15.27:0/3109544986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03ac0047d0 0x7f03ac004c50 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.889+0000 7f03d3400640 1 -- 172.21.15.27:0/3109544986 >> 172.21.15.27:0/3109544986 conn(0x7f03cc0cec60 msgr2=0x7f03cc0cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:21.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.889+0000 7f03d3400640 1 -- 172.21.15.27:0/3109544986 shutdown_connections 2024-01-15T18:18:21.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.889+0000 7f03d3400640 1 -- 172.21.15.27:0/3109544986 wait complete. 2024-01-15T18:18:21.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.893+0000 7f03d3400640 1 Processor -- start 2024-01-15T18:18:21.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.893+0000 7f03d3400640 1 -- start start 2024-01-15T18:18:21.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.893+0000 7f03d3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03ac0047d0 0x7f03cc156880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:21.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.893+0000 7f03d3400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03cc0c5480 0x7f03cc0c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:21.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.893+0000 7f03d3400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03cc0c58e0 0x7f03cc0c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:21.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.893+0000 7f03d3400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc143700 con 0x7f03ac0047d0 2024-01-15T18:18:21.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.893+0000 7f03d3400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc142850 con 0x7f03cc0c58e0 2024-01-15T18:18:21.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.893+0000 7f03d3400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03cc142af0 con 0x7f03cc0c5480 2024-01-15T18:18:21.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.893+0000 7f03d0a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03ac0047d0 0x7f03cc156880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:21.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.893+0000 7f03d0a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03ac0047d0 0x7f03cc156880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33812/0 (socket says 172.21.15.27:33812) 2024-01-15T18:18:21.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.893+0000 7f03d0a00640 1 -- 172.21.15.27:0/3784683082 learned_addr learned my addr 172.21.15.27:0/3784683082 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:21.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.893+0000 7f03cbe00640 1 --2- 172.21.15.27:0/3784683082 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03cc0c5480 0x7f03cc0c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:21.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.893+0000 7f03d1400640 1 --2- 172.21.15.27:0/3784683082 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03cc0c58e0 0x7f03cc0c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:21.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.893+0000 7f03d0a00640 1 -- 172.21.15.27:0/3784683082 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03cc0c5480 msgr2=0x7f03cc0c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:21.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.893+0000 7f03d0a00640 1 --2- 172.21.15.27:0/3784683082 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03cc0c5480 0x7f03cc0c77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.893+0000 7f03d0a00640 1 -- 172.21.15.27:0/3784683082 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03cc0c58e0 msgr2=0x7f03cc0c7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:21.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.893+0000 7f03d0a00640 1 --2- 172.21.15.27:0/3784683082 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03cc0c58e0 0x7f03cc0c7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.893+0000 7f03d0a00640 1 -- 172.21.15.27:0/3784683082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03cc142dd0 con 0x7f03ac0047d0 2024-01-15T18:18:21.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.893+0000 7f03d0a00640 1 --2- 172.21.15.27:0/3784683082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03ac0047d0 0x7f03cc156880 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f03bc011d50 tx=0x7f03bc00f540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:21.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.893+0000 7f03c8c00640 1 -- 172.21.15.27:0/3784683082 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc015410 con 0x7f03ac0047d0 2024-01-15T18:18:21.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.893+0000 7f03c8c00640 1 -- 172.21.15.27:0/3784683082 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03bc015590 con 0x7f03ac0047d0 2024-01-15T18:18:21.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.893+0000 7f03c8c00640 1 -- 172.21.15.27:0/3784683082 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc015940 con 0x7f03ac0047d0 2024-01-15T18:18:21.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.893+0000 7f03d3400640 1 -- 172.21.15.27:0/3784683082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03cc1430b0 con 0x7f03ac0047d0 2024-01-15T18:18:21.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.893+0000 7f03d3400640 1 -- 172.21.15.27:0/3784683082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03cc156dc0 con 0x7f03ac0047d0 2024-01-15T18:18:21.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.897+0000 7f03d3400640 1 -- 172.21.15.27:0/3784683082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03cc04e850 con 0x7f03ac0047d0 2024-01-15T18:18:21.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.897+0000 7f03c8c00640 1 -- 172.21.15.27:0/3784683082 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f03bc015aa0 con 0x7f03ac0047d0 2024-01-15T18:18:21.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.897+0000 7f03c8c00640 1 --2- 172.21.15.27:0/3784683082 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03a4067940 0x7f03a4069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:21.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.897+0000 7f03c8c00640 1 -- 172.21.15.27:0/3784683082 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f03bc086b40 con 0x7f03ac0047d0 2024-01-15T18:18:21.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.897+0000 7f03cbe00640 1 --2- 172.21.15.27:0/3784683082 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03a4067940 0x7f03a4069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:21.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.897+0000 7f03cbe00640 1 --2- 172.21.15.27:0/3784683082 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03a4067940 0x7f03a4069e00 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f03b40ab000 tx=0x7f03b40b1040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:21.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:21.901+0000 7f03c8c00640 1 -- 172.21.15.27:0/3784683082 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f03bc055480 con 0x7f03ac0047d0 2024-01-15T18:18:22.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.057+0000 7f03d3400640 1 -- 172.21.15.27:0/3784683082 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03cc0c9730 con 0x7f03ac0047d0 2024-01-15T18:18:22.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.061+0000 7f03c8c00640 1 -- 172.21.15.27:0/3784683082 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+13067 (secure 0 0 0) 0x7f03bc058aa0 con 0x7f03ac0047d0 2024-01-15T18:18:22.060 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:18:22.060 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":100,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:18:20.055597+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6835/259930036":"2024-01-16T18:18:17.046652+0000","172.21.15.27:6833/259930036":"2024-01-16T18:18:17.046652+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:18:22.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.061+0000 7f03d3400640 1 -- 172.21.15.27:0/3784683082 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03a4067940 msgr2=0x7f03a4069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:22.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.061+0000 7f03d3400640 1 --2- 172.21.15.27:0/3784683082 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03a4067940 0x7f03a4069e00 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f03b40ab000 tx=0x7f03b40b1040 comp rx=0 tx=0).stop 2024-01-15T18:18:22.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.061+0000 7f03d3400640 1 -- 172.21.15.27:0/3784683082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03ac0047d0 msgr2=0x7f03cc156880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:22.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.061+0000 7f03d3400640 1 --2- 172.21.15.27:0/3784683082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03ac0047d0 0x7f03cc156880 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f03bc011d50 tx=0x7f03bc00f540 comp rx=0 tx=0).stop 2024-01-15T18:18:22.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.061+0000 7f03d3400640 1 -- 172.21.15.27:0/3784683082 shutdown_connections 2024-01-15T18:18:22.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.061+0000 7f03d3400640 1 --2- 172.21.15.27:0/3784683082 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f03a4067940 0x7f03a4069e00 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:22.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.061+0000 7f03d3400640 1 --2- 172.21.15.27:0/3784683082 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f03cc0c58e0 0x7f03cc0c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:22.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.061+0000 7f03d3400640 1 --2- 172.21.15.27:0/3784683082 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f03cc0c5480 0x7f03cc0c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:22.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.061+0000 7f03d3400640 1 --2- 172.21.15.27:0/3784683082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03ac0047d0 0x7f03cc156880 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:22.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.061+0000 7f03d3400640 1 -- 172.21.15.27:0/3784683082 >> 172.21.15.27:0/3784683082 conn(0x7f03cc0cec60 msgr2=0x7f03cc0cce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:22.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.061+0000 7f03d3400640 1 -- 172.21.15.27:0/3784683082 shutdown_connections 2024-01-15T18:18:22.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.061+0000 7f03d3400640 1 -- 172.21.15.27:0/3784683082 wait complete. 2024-01-15T18:18:22.083 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:18:22.083 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:18:22.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.225+0000 7efe87e00640 1 Processor -- start 2024-01-15T18:18:22.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.225+0000 7efe87e00640 1 -- start start 2024-01-15T18:18:22.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.229+0000 7efe87e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7efe80146a60 0x7efe800ac880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:22.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.229+0000 7efe87e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe80160cb0 0x7efe801610f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:22.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.229+0000 7efe87e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efe8006a4f0 0x7efe8006a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:22.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.229+0000 7efe87e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe80161630 con 0x7efe8006a4f0 2024-01-15T18:18:22.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.229+0000 7efe87e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe8006aec0 con 0x7efe80146a60 2024-01-15T18:18:22.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.229+0000 7efe87e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe800cde30 con 0x7efe80160cb0 2024-01-15T18:18:22.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.229+0000 7efe85e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efe8006a4f0 0x7efe8006a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:22.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.229+0000 7efe85e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efe8006a4f0 0x7efe8006a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33816/0 (socket says 172.21.15.27:33816) 2024-01-15T18:18:22.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.229+0000 7efe85e00640 1 -- 172.21.15.27:0/3306667932 learned_addr learned my addr 172.21.15.27:0/3306667932 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:22.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.229+0000 7efe85400640 1 --2- 172.21.15.27:0/3306667932 >> v2:172.21.15.93:3300/0 conn(0x7efe80146a60 0x7efe800ac880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:22.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.229+0000 7efe84a00640 1 --2- 172.21.15.27:0/3306667932 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe80160cb0 0x7efe801610f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:22.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.229+0000 7efe85e00640 1 -- 172.21.15.27:0/3306667932 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe80160cb0 msgr2=0x7efe801610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:22.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.229+0000 7efe85e00640 1 --2- 172.21.15.27:0/3306667932 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe80160cb0 0x7efe801610f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:22.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.229+0000 7efe85e00640 1 -- 172.21.15.27:0/3306667932 >> v2:172.21.15.93:3300/0 conn(0x7efe80146a60 msgr2=0x7efe800ac880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:22.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.229+0000 7efe85e00640 1 --2- 172.21.15.27:0/3306667932 >> v2:172.21.15.93:3300/0 conn(0x7efe80146a60 0x7efe800ac880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:22.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.229+0000 7efe85e00640 1 -- 172.21.15.27:0/3306667932 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe8006fa50 con 0x7efe8006a4f0 2024-01-15T18:18:22.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.229+0000 7efe84a00640 1 --2- 172.21.15.27:0/3306667932 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe80160cb0 0x7efe801610f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:18:22.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.233+0000 7efe85e00640 1 --2- 172.21.15.27:0/3306667932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe8006a4f0 0x7efe8006a980 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7efe800cd7a0 tx=0x7efe7801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15400a53e9cc4fbb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:22.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.233+0000 7efe7fe00640 1 -- 172.21.15.27:0/3306667932 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe780a5820 con 0x7efe8006a4f0 2024-01-15T18:18:22.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.233+0000 7efe7fe00640 1 -- 172.21.15.27:0/3306667932 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe780a5980 con 0x7efe8006a4f0 2024-01-15T18:18:22.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.233+0000 7efe7fe00640 1 -- 172.21.15.27:0/3306667932 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe780a5d10 con 0x7efe8006a4f0 2024-01-15T18:18:22.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.233+0000 7efe87e00640 1 -- 172.21.15.27:0/3306667932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe8006a4f0 msgr2=0x7efe8006a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:22.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.233+0000 7efe87e00640 1 --2- 172.21.15.27:0/3306667932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe8006a4f0 0x7efe8006a980 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7efe800cd7a0 tx=0x7efe7801c9a0 comp rx=0 tx=0).stop 2024-01-15T18:18:22.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.233+0000 7efe87e00640 1 -- 172.21.15.27:0/3306667932 shutdown_connections 2024-01-15T18:18:22.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.233+0000 7efe87e00640 1 --2- 172.21.15.27:0/3306667932 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe8006a4f0 0x7efe8006a980 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:22.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.233+0000 7efe87e00640 1 --2- 172.21.15.27:0/3306667932 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe80160cb0 0x7efe801610f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:22.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.233+0000 7efe87e00640 1 --2- 172.21.15.27:0/3306667932 >> v2:172.21.15.93:3300/0 conn(0x7efe80146a60 0x7efe800ac880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:22.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.233+0000 7efe87e00640 1 -- 172.21.15.27:0/3306667932 >> 172.21.15.27:0/3306667932 conn(0x7efe800cce10 msgr2=0x7efe800b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:22.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.233+0000 7efe87e00640 1 -- 172.21.15.27:0/3306667932 shutdown_connections 2024-01-15T18:18:22.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.233+0000 7efe87e00640 1 -- 172.21.15.27:0/3306667932 wait complete. 2024-01-15T18:18:22.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.233+0000 7efe87e00640 1 Processor -- start 2024-01-15T18:18:22.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.233+0000 7efe87e00640 1 -- start start 2024-01-15T18:18:22.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.237+0000 7efe87e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efe8006a4f0 0x7efe800ad1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:22.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.237+0000 7efe87e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe80146a60 0x7efe8015ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:22.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.237+0000 7efe87e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe80160cb0 0x7efe8013a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:22.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.237+0000 7efe87e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe80138b70 con 0x7efe80160cb0 2024-01-15T18:18:22.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.237+0000 7efe87e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe80137cc0 con 0x7efe8006a4f0 2024-01-15T18:18:22.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.237+0000 7efe87e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe8015ea40 con 0x7efe80146a60 2024-01-15T18:18:22.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.237+0000 7efe85e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe80160cb0 0x7efe8013a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:22.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.237+0000 7efe85e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe80160cb0 0x7efe8013a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33830/0 (socket says 172.21.15.27:33830) 2024-01-15T18:18:22.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.237+0000 7efe85e00640 1 -- 172.21.15.27:0/3376638250 learned_addr learned my addr 172.21.15.27:0/3376638250 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:22.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.237+0000 7efe84a00640 1 --2- 172.21.15.27:0/3376638250 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe80146a60 0x7efe8015ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:22.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.237+0000 7efe85400640 1 --2- 172.21.15.27:0/3376638250 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efe8006a4f0 0x7efe800ad1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:22.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.237+0000 7efe85e00640 1 -- 172.21.15.27:0/3376638250 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe80146a60 msgr2=0x7efe8015ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:22.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.237+0000 7efe85e00640 1 --2- 172.21.15.27:0/3376638250 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe80146a60 0x7efe8015ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:22.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.237+0000 7efe85e00640 1 -- 172.21.15.27:0/3376638250 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efe8006a4f0 msgr2=0x7efe800ad1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:22.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.237+0000 7efe85e00640 1 --2- 172.21.15.27:0/3376638250 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efe8006a4f0 0x7efe800ad1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:22.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.237+0000 7efe85e00640 1 -- 172.21.15.27:0/3376638250 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe78009b30 con 0x7efe80160cb0 2024-01-15T18:18:22.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.237+0000 7efe84a00640 1 --2- 172.21.15.27:0/3376638250 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe80146a60 0x7efe8015ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:18:22.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.237+0000 7efe85e00640 1 --2- 172.21.15.27:0/3376638250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe80160cb0 0x7efe8013a6a0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7efe7801b050 tx=0x7efe78019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:22.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.237+0000 7efe7d600640 1 -- 172.21.15.27:0/3376638250 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe780b8670 con 0x7efe80160cb0 2024-01-15T18:18:22.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.237+0000 7efe87e00640 1 -- 172.21.15.27:0/3376638250 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe80137f60 con 0x7efe80160cb0 2024-01-15T18:18:22.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.237+0000 7efe7d600640 1 -- 172.21.15.27:0/3376638250 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe780b87d0 con 0x7efe80160cb0 2024-01-15T18:18:22.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.237+0000 7efe7d600640 1 -- 172.21.15.27:0/3376638250 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe780b8b80 con 0x7efe80160cb0 2024-01-15T18:18:22.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.237+0000 7efe87e00640 1 -- 172.21.15.27:0/3376638250 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe80138520 con 0x7efe80160cb0 2024-01-15T18:18:22.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.237+0000 7efe87e00640 1 -- 172.21.15.27:0/3376638250 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe80073c10 con 0x7efe80160cb0 2024-01-15T18:18:22.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.241+0000 7efe7d600640 1 -- 172.21.15.27:0/3376638250 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7efe780b8ce0 con 0x7efe80160cb0 2024-01-15T18:18:22.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.241+0000 7efe7d600640 1 --2- 172.21.15.27:0/3376638250 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efe58067970 0x7efe58069e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:22.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.241+0000 7efe7d600640 1 -- 172.21.15.27:0/3376638250 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5055+0+0 (secure 0 0 0) 0x7efe7811e3e0 con 0x7efe80160cb0 2024-01-15T18:18:22.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.241+0000 7efe85400640 1 --2- 172.21.15.27:0/3376638250 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efe58067970 0x7efe58069e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:22.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.241+0000 7efe85400640 1 --2- 172.21.15.27:0/3376638250 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efe58067970 0x7efe58069e30 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7efe70009ae0 tx=0x7efe70009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:22.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.245+0000 7efe7d600640 1 -- 172.21.15.27:0/3376638250 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7efe780eccd0 con 0x7efe80160cb0 2024-01-15T18:18:22.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.405+0000 7efe87e00640 1 -- 172.21.15.27:0/3376638250 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efe8004e850 con 0x7efe80160cb0 2024-01-15T18:18:22.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.405+0000 7efe7d600640 1 -- 172.21.15.27:0/3376638250 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) v1 ==== 93+0+10 (secure 0 0 0) 0x7efe780f02f0 con 0x7efe80160cb0 2024-01-15T18:18:22.407 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:18:22.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.405+0000 7efe87e00640 1 -- 172.21.15.27:0/3376638250 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efe58067970 msgr2=0x7efe58069e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:22.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.405+0000 7efe87e00640 1 --2- 172.21.15.27:0/3376638250 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efe58067970 0x7efe58069e30 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7efe70009ae0 tx=0x7efe70009190 comp rx=0 tx=0).stop 2024-01-15T18:18:22.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.405+0000 7efe87e00640 1 -- 172.21.15.27:0/3376638250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe80160cb0 msgr2=0x7efe8013a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:22.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.405+0000 7efe87e00640 1 --2- 172.21.15.27:0/3376638250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe80160cb0 0x7efe8013a6a0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7efe7801b050 tx=0x7efe78019740 comp rx=0 tx=0).stop 2024-01-15T18:18:22.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.405+0000 7efe87e00640 1 -- 172.21.15.27:0/3376638250 shutdown_connections 2024-01-15T18:18:22.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.405+0000 7efe87e00640 1 --2- 172.21.15.27:0/3376638250 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efe58067970 0x7efe58069e30 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:22.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.405+0000 7efe87e00640 1 --2- 172.21.15.27:0/3376638250 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe80160cb0 0x7efe8013a6a0 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:22.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.405+0000 7efe87e00640 1 --2- 172.21.15.27:0/3376638250 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe80146a60 0x7efe8015ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:22.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.405+0000 7efe87e00640 1 --2- 172.21.15.27:0/3376638250 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efe8006a4f0 0x7efe800ad1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:22.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.405+0000 7efe87e00640 1 -- 172.21.15.27:0/3376638250 >> 172.21.15.27:0/3376638250 conn(0x7efe800cce10 msgr2=0x7efe80071a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:22.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.405+0000 7efe87e00640 1 -- 172.21.15.27:0/3376638250 shutdown_connections 2024-01-15T18:18:22.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.405+0000 7efe87e00640 1 -- 172.21.15.27:0/3376638250 wait complete. 2024-01-15T18:18:22.426 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:18:22.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.569+0000 7f9f29000640 1 Processor -- start 2024-01-15T18:18:22.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.569+0000 7f9f29000640 1 -- start start 2024-01-15T18:18:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.569+0000 7f9f29000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f9f240b35a0 0x7f9f240cc700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.569+0000 7f9f29000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f2409a700 con 0x7f9f2409a120 2024-01-15T18:18:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.569+0000 7f9f29000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f240cd590 con 0x7f9f2415e7d0 2024-01-15T18:18:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.569+0000 7f9f29000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f240cd870 con 0x7f9f240b35a0 2024-01-15T18:18:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.569+0000 7f9f21400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9f2409a120 0x7f9f240cbff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44254/0 (socket says 172.21.15.27:44254) 2024-01-15T18:18:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.569+0000 7f9f21400640 1 -- 172.21.15.27:0/2350930000 learned_addr learned my addr 172.21.15.27:0/2350930000 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.569+0000 7f9f22800640 1 --2- 172.21.15.27:0/2350930000 >> v2:172.21.15.93:3300/0 conn(0x7f9f240b35a0 0x7f9f240cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.569+0000 7f9f20a00640 1 -- 172.21.15.27:0/2350930000 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350230276 0 0) 0x7f9f2409a700 con 0x7f9f2409a120 2024-01-15T18:18:22.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.569+0000 7f9f20a00640 1 -- 172.21.15.27:0/2350930000 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f04003530 con 0x7f9f2409a120 2024-01-15T18:18:22.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.569+0000 7f9f22800640 1 -- 172.21.15.27:0/2350930000 >> v1:172.21.15.93:6789/0 conn(0x7f9f2415e7d0 legacy=0x7f9f24099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:22.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.569+0000 7f9f22800640 1 -- 172.21.15.27:0/2350930000 >> v1:172.21.15.27:6789/0 conn(0x7f9f2409a120 legacy=0x7f9f240cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:22.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.569+0000 7f9f22800640 1 -- 172.21.15.27:0/2350930000 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f240cfa00 con 0x7f9f240b35a0 2024-01-15T18:18:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.573+0000 7f9f22800640 1 --2- 172.21.15.27:0/2350930000 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f240b35a0 0x7f9f240cc700 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c009d90 tx=0x7f9f1c01c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f47f98d5f82f3bc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.573+0000 7f9f20a00640 1 -- 172.21.15.27:0/2350930000 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c0a5a00 con 0x7f9f240b35a0 2024-01-15T18:18:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.573+0000 7f9f20a00640 1 -- 172.21.15.27:0/2350930000 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f1c0a5b60 con 0x7f9f240b35a0 2024-01-15T18:18:22.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.573+0000 7f9f20a00640 1 -- 172.21.15.27:0/2350930000 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c0ad440 con 0x7f9f240b35a0 2024-01-15T18:18:22.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.573+0000 7f9f29000640 1 -- 172.21.15.27:0/2350930000 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f240b35a0 msgr2=0x7f9f240cc700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:22.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.573+0000 7f9f29000640 1 --2- 172.21.15.27:0/2350930000 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f240b35a0 0x7f9f240cc700 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c009d90 tx=0x7f9f1c01c920 comp rx=0 tx=0).stop 2024-01-15T18:18:22.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.573+0000 7f9f29000640 1 -- 172.21.15.27:0/2350930000 shutdown_connections 2024-01-15T18:18:22.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.573+0000 7f9f29000640 1 --2- 172.21.15.27:0/2350930000 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f240b35a0 0x7f9f240cc700 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:22.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.573+0000 7f9f29000640 1 -- 172.21.15.27:0/2350930000 >> 172.21.15.27:0/2350930000 conn(0x7f9f240b2a80 msgr2=0x7f9f240b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:22.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.577+0000 7f9f29000640 1 -- 172.21.15.27:0/2350930000 shutdown_connections 2024-01-15T18:18:22.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.577+0000 7f9f29000640 1 -- 172.21.15.27:0/2350930000 wait complete. 2024-01-15T18:18:22.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.577+0000 7f9f29000640 1 Processor -- start 2024-01-15T18:18:22.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.577+0000 7f9f29000640 1 -- start start 2024-01-15T18:18:22.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.577+0000 7f9f29000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f2409a120 0x7f9f2415d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:22.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.577+0000 7f9f29000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f240b35a0 0x7f9f2414fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:22.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.577+0000 7f9f29000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9f2415e7d0 0x7f9f24150330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:22.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.577+0000 7f9f29000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2415cde0 con 0x7f9f2409a120 2024-01-15T18:18:22.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.577+0000 7f9f29000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2415c260 con 0x7f9f240b35a0 2024-01-15T18:18:22.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.577+0000 7f9f29000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2415c540 con 0x7f9f2415e7d0 2024-01-15T18:18:22.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.577+0000 7f9f21e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f2409a120 0x7f9f2415d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:22.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.577+0000 7f9f21e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f2409a120 0x7f9f2415d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33842/0 (socket says 172.21.15.27:33842) 2024-01-15T18:18:22.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.577+0000 7f9f21e00640 1 -- 172.21.15.27:0/3857548342 learned_addr learned my addr 172.21.15.27:0/3857548342 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.577+0000 7f9f22800640 1 --2- 172.21.15.27:0/3857548342 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9f2415e7d0 0x7f9f24150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.577+0000 7f9f21400640 1 --2- 172.21.15.27:0/3857548342 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f240b35a0 0x7f9f2414fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.577+0000 7f9f21e00640 1 -- 172.21.15.27:0/3857548342 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9f2415e7d0 msgr2=0x7f9f24150330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.577+0000 7f9f21e00640 1 --2- 172.21.15.27:0/3857548342 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9f2415e7d0 0x7f9f24150330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.577+0000 7f9f21e00640 1 -- 172.21.15.27:0/3857548342 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f240b35a0 msgr2=0x7f9f2414fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.577+0000 7f9f21e00640 1 --2- 172.21.15.27:0/3857548342 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f240b35a0 0x7f9f2414fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.577+0000 7f9f21e00640 1 -- 172.21.15.27:0/3857548342 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f1c009a70 con 0x7f9f2409a120 2024-01-15T18:18:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.577+0000 7f9f22800640 1 --2- 172.21.15.27:0/3857548342 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9f2415e7d0 0x7f9f24150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:18:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.577+0000 7f9f21e00640 1 --2- 172.21.15.27:0/3857548342 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f2409a120 0x7f9f2415d950 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f9f1400c3c0 tx=0x7f9f1400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.581+0000 7f9f1a000640 1 -- 172.21.15.27:0/3857548342 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f140052d0 con 0x7f9f2409a120 2024-01-15T18:18:22.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.581+0000 7f9f1a000640 1 -- 172.21.15.27:0/3857548342 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f14005430 con 0x7f9f2409a120 2024-01-15T18:18:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.581+0000 7f9f29000640 1 -- 172.21.15.27:0/3857548342 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f2415c820 con 0x7f9f2409a120 2024-01-15T18:18:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.581+0000 7f9f1a000640 1 -- 172.21.15.27:0/3857548342 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f14003840 con 0x7f9f2409a120 2024-01-15T18:18:22.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.581+0000 7f9f29000640 1 -- 172.21.15.27:0/3857548342 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f2415de90 con 0x7f9f2409a120 2024-01-15T18:18:22.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.581+0000 7f9f29000640 1 -- 172.21.15.27:0/3857548342 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f2404e850 con 0x7f9f2409a120 2024-01-15T18:18:22.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.581+0000 7f9f1a000640 1 -- 172.21.15.27:0/3857548342 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f9f140039a0 con 0x7f9f2409a120 2024-01-15T18:18:22.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.581+0000 7f9f1a000640 1 --2- 172.21.15.27:0/3857548342 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9efc067940 0x7f9efc069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:22.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.581+0000 7f9f1a000640 1 -- 172.21.15.27:0/3857548342 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f9f140827f0 con 0x7f9f2409a120 2024-01-15T18:18:22.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.581+0000 7f9f21400640 1 --2- 172.21.15.27:0/3857548342 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9efc067940 0x7f9efc069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:22.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.585+0000 7f9f21400640 1 --2- 172.21.15.27:0/3857548342 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9efc067940 0x7f9efc069e00 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f9f240af440 tx=0x7f9f0c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:22.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.585+0000 7f9f1a000640 1 -- 172.21.15.27:0/3857548342 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9f14051160 con 0x7f9f2409a120 2024-01-15T18:18:22.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.753+0000 7f9f29000640 1 -- 172.21.15.27:0/3857548342 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9f240b6ed0 con 0x7f9f2409a120 2024-01-15T18:18:22.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.757+0000 7f9f1a000640 1 -- 172.21.15.27:0/3857548342 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v100) v1 ==== 80+0+120 (secure 0 0 0) 0x7f9f14017080 con 0x7f9f2409a120 2024-01-15T18:18:22.756 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6835/259930036 2024-01-16T18:18:17.046652+0000 2024-01-15T18:18:22.756 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6833/259930036 2024-01-16T18:18:17.046652+0000 2024-01-15T18:18:22.756 INFO:teuthology.orchestra.run.smithi027.stderr:listed 2 entries 2024-01-15T18:18:22.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.757+0000 7f9f29000640 1 -- 172.21.15.27:0/3857548342 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9efc067940 msgr2=0x7f9efc069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:22.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.757+0000 7f9f29000640 1 --2- 172.21.15.27:0/3857548342 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9efc067940 0x7f9efc069e00 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f9f240af440 tx=0x7f9f0c009040 comp rx=0 tx=0).stop 2024-01-15T18:18:22.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.757+0000 7f9f29000640 1 -- 172.21.15.27:0/3857548342 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f2409a120 msgr2=0x7f9f2415d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:22.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.757+0000 7f9f29000640 1 --2- 172.21.15.27:0/3857548342 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f2409a120 0x7f9f2415d950 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f9f1400c3c0 tx=0x7f9f1400c880 comp rx=0 tx=0).stop 2024-01-15T18:18:22.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.757+0000 7f9f29000640 1 -- 172.21.15.27:0/3857548342 shutdown_connections 2024-01-15T18:18:22.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.757+0000 7f9f29000640 1 --2- 172.21.15.27:0/3857548342 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9efc067940 0x7f9efc069e00 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:22.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.757+0000 7f9f29000640 1 --2- 172.21.15.27:0/3857548342 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9f2415e7d0 0x7f9f24150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:22.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.757+0000 7f9f29000640 1 --2- 172.21.15.27:0/3857548342 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9f240b35a0 0x7f9f2414fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:22.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.757+0000 7f9f29000640 1 --2- 172.21.15.27:0/3857548342 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f2409a120 0x7f9f2415d950 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:22.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.757+0000 7f9f29000640 1 -- 172.21.15.27:0/3857548342 >> 172.21.15.27:0/3857548342 conn(0x7f9f240b2a80 msgr2=0x7f9f240b2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:22.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.757+0000 7f9f29000640 1 -- 172.21.15.27:0/3857548342 shutdown_connections 2024-01-15T18:18:22.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:22.757+0000 7f9f29000640 1 -- 172.21.15.27:0/3857548342 wait complete. 2024-01-15T18:18:22.778 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi093.front.sepia.ceph.com... 2024-01-15T18:18:22.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:22.778 DEBUG:teuthology.orchestra.run.smithi093:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-15T18:18:22.799 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:ceph-fuse[34581]: fuse finished with error 0 and tester_r 0 2024-01-15T18:18:22.802 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T18:18:22.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:22.802 DEBUG:teuthology.orchestra.run.smithi093:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-15T18:18:22.974 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:18:22.974 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:18:22.983 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354580k,nr_inodes=4088645,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9249 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:18:22.983 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T18:18:22.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:22.983 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-15T18:18:23.034 INFO:teuthology.orchestra.run.smithi027.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-15T18:18:23.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:18:23.035 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:18:23.035 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:18:23.045 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354484k,nr_inodes=4088621,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8403 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:18:23.045 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T18:18:23.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:23.045 DEBUG:teuthology.orchestra.run.smithi093:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-15T18:18:23.099 INFO:teuthology.orchestra.run.smithi093.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-15T18:18:23.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:18:23.100 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:18:23.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.245+0000 7f307ce00640 1 Processor -- start 2024-01-15T18:18:23.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.245+0000 7f307ce00640 1 -- start start 2024-01-15T18:18:23.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.245+0000 7f307ce00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f30780701a0 0x7f3078070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:23.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.245+0000 7f307ce00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30780c92b0 0x7f307806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:23.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.245+0000 7f307ce00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f307806c4a0 con 0x7f307806e240 2024-01-15T18:18:23.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.245+0000 7f307ce00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3078070ac0 con 0x7f30780701a0 2024-01-15T18:18:23.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.245+0000 7f307ce00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307815c660 con 0x7f30780c92b0 2024-01-15T18:18:23.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.245+0000 7f3075e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f30780701a0 0x7f3078070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:23.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.245+0000 7f3076800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f307806e240 0x7f307815d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:44258/0 (socket says 172.21.15.27:44258) 2024-01-15T18:18:23.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.245+0000 7f3076800640 1 -- 172.21.15.27:0/1518801120 learned_addr learned my addr 172.21.15.27:0/1518801120 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:23.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.245+0000 7f3075400640 1 --2- 172.21.15.27:0/1518801120 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30780c92b0 0x7f307806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:23.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.245+0000 7f3075e00640 1 -- 172.21.15.27:0/1518801120 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30780c92b0 msgr2=0x7f307806dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:23.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.245+0000 7f3075e00640 1 --2- 172.21.15.27:0/1518801120 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30780c92b0 0x7f307806dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:23.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.245+0000 7f3075e00640 1 -- 172.21.15.27:0/1518801120 >> v1:172.21.15.27:6789/0 conn(0x7f307806e240 legacy=0x7f307815d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:23.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.245+0000 7f3075e00640 1 -- 172.21.15.27:0/1518801120 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307815c940 con 0x7f30780701a0 2024-01-15T18:18:23.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.245+0000 7f3074a00640 1 -- 172.21.15.27:0/1518801120 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 952627166 0 0) 0x7f307806c4a0 con 0x7f307806e240 2024-01-15T18:18:23.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.249+0000 7f3075e00640 1 --2- 172.21.15.27:0/1518801120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30780701a0 0x7f3078070580 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f30680097c0 tx=0x7f306801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffbb75885958ab69 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:23.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.249+0000 7f3074a00640 1 -- 172.21.15.27:0/1518801120 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30680ab650 con 0x7f30780701a0 2024-01-15T18:18:23.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.249+0000 7f3074a00640 1 -- 172.21.15.27:0/1518801120 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30680ab7b0 con 0x7f30780701a0 2024-01-15T18:18:23.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.249+0000 7f3074a00640 1 -- 172.21.15.27:0/1518801120 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30680abb40 con 0x7f30780701a0 2024-01-15T18:18:23.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.249+0000 7f307ce00640 1 -- 172.21.15.27:0/1518801120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30780701a0 msgr2=0x7f3078070580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:23.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.249+0000 7f307ce00640 1 --2- 172.21.15.27:0/1518801120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30780701a0 0x7f3078070580 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f30680097c0 tx=0x7f306801c450 comp rx=0 tx=0).stop 2024-01-15T18:18:23.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.249+0000 7f307ce00640 1 -- 172.21.15.27:0/1518801120 shutdown_connections 2024-01-15T18:18:23.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.249+0000 7f307ce00640 1 --2- 172.21.15.27:0/1518801120 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30780c92b0 0x7f307806dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:23.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.249+0000 7f307ce00640 1 --2- 172.21.15.27:0/1518801120 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30780701a0 0x7f3078070580 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:23.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.249+0000 7f307ce00640 1 -- 172.21.15.27:0/1518801120 >> 172.21.15.27:0/1518801120 conn(0x7f30780cec60 msgr2=0x7f30780cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:23.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.249+0000 7f307ce00640 1 -- 172.21.15.27:0/1518801120 shutdown_connections 2024-01-15T18:18:23.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.249+0000 7f307ce00640 1 -- 172.21.15.27:0/1518801120 wait complete. 2024-01-15T18:18:23.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.253+0000 7f307ce00640 1 Processor -- start 2024-01-15T18:18:23.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.253+0000 7f307ce00640 1 -- start start 2024-01-15T18:18:23.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.253+0000 7f307ce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f307806e240 0x7f30780c87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:23.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.253+0000 7f307ce00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30780701a0 0x7f30780c1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:23.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.253+0000 7f307ce00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30780c92b0 0x7f30780cd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:23.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.253+0000 7f307ce00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30780c7510 con 0x7f307806e240 2024-01-15T18:18:23.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.253+0000 7f307ce00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30780c8350 con 0x7f30780c92b0 2024-01-15T18:18:23.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.253+0000 7f307ce00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30780c7740 con 0x7f30780701a0 2024-01-15T18:18:23.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.253+0000 7f3075e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f307806e240 0x7f30780c87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:23.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.253+0000 7f3075e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f307806e240 0x7f30780c87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33856/0 (socket says 172.21.15.27:33856) 2024-01-15T18:18:23.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.253+0000 7f3075e00640 1 -- 172.21.15.27:0/1872975122 learned_addr learned my addr 172.21.15.27:0/1872975122 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:23.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.253+0000 7f3075400640 1 --2- 172.21.15.27:0/1872975122 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30780701a0 0x7f30780c1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:23.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.253+0000 7f3076800640 1 --2- 172.21.15.27:0/1872975122 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30780c92b0 0x7f30780cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:23.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.253+0000 7f3075e00640 1 -- 172.21.15.27:0/1872975122 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30780701a0 msgr2=0x7f30780c1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:23.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.253+0000 7f3075e00640 1 --2- 172.21.15.27:0/1872975122 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30780701a0 0x7f30780c1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:23.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.253+0000 7f3075e00640 1 -- 172.21.15.27:0/1872975122 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30780c92b0 msgr2=0x7f30780cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:23.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.253+0000 7f3075e00640 1 --2- 172.21.15.27:0/1872975122 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30780c92b0 0x7f30780cd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:23.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.253+0000 7f3075e00640 1 -- 172.21.15.27:0/1872975122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3068009420 con 0x7f307806e240 2024-01-15T18:18:23.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.253+0000 7f3075e00640 1 --2- 172.21.15.27:0/1872975122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f307806e240 0x7f30780c87e0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f30680ad040 tx=0x7f306801ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:23.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.253+0000 7f306e000640 1 -- 172.21.15.27:0/1872975122 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30680bf610 con 0x7f307806e240 2024-01-15T18:18:23.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.253+0000 7f306e000640 1 -- 172.21.15.27:0/1872975122 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30680bf790 con 0x7f307806e240 2024-01-15T18:18:23.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.253+0000 7f306e000640 1 -- 172.21.15.27:0/1872975122 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30680bfb40 con 0x7f307806e240 2024-01-15T18:18:23.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.253+0000 7f307ce00640 1 -- 172.21.15.27:0/1872975122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30780c7a20 con 0x7f307806e240 2024-01-15T18:18:23.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.253+0000 7f307ce00640 1 -- 172.21.15.27:0/1872975122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30780c7fe0 con 0x7f307806e240 2024-01-15T18:18:23.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.257+0000 7f307ce00640 1 -- 172.21.15.27:0/1872975122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f307804e850 con 0x7f307806e240 2024-01-15T18:18:23.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.257+0000 7f306e000640 1 -- 172.21.15.27:0/1872975122 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f30680bfca0 con 0x7f307806e240 2024-01-15T18:18:23.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.257+0000 7f306e000640 1 --2- 172.21.15.27:0/1872975122 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3050067970 0x7f3050069e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:23.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.257+0000 7f306e000640 1 -- 172.21.15.27:0/1872975122 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f306811de60 con 0x7f307806e240 2024-01-15T18:18:23.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.257+0000 7f3075400640 1 --2- 172.21.15.27:0/1872975122 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3050067970 0x7f3050069e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:23.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.257+0000 7f3075400640 1 --2- 172.21.15.27:0/1872975122 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3050067970 0x7f3050069e30 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f3078156e00 tx=0x7f3060009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:23.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.261+0000 7f306e000640 1 -- 172.21.15.27:0/1872975122 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f30680ec9a0 con 0x7f307806e240 2024-01-15T18:18:23.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.449+0000 7f307ce00640 1 -- 172.21.15.27:0/1872975122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3078156aa0 con 0x7f307806e240 2024-01-15T18:18:23.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.449+0000 7f306e000640 1 -- 172.21.15.27:0/1872975122 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f30680effc0 con 0x7f307806e240 2024-01-15T18:18:23.450 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:18:23.450 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":70,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/1905230324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":1905230324},{"type":"v1","addr":"172.21.15.93:6837","nonce":1905230324}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":8269,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/2926806399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":2926806399},{"type":"v1","addr":"172.21.15.93:6836","nonce":2926806399}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":8934,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1107490361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1107490361},{"type":"v1","addr":"172.21.15.27:6834","nonce":1107490361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":9327,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1927211782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1927211782},{"type":"v1","addr":"172.21.15.27:6835","nonce":1927211782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[]} 2024-01-15T18:18:23.450 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 70 2024-01-15T18:18:23.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.453+0000 7f307ce00640 1 -- 172.21.15.27:0/1872975122 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3050067970 msgr2=0x7f3050069e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:23.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.453+0000 7f307ce00640 1 --2- 172.21.15.27:0/1872975122 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3050067970 0x7f3050069e30 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f3078156e00 tx=0x7f3060009190 comp rx=0 tx=0).stop 2024-01-15T18:18:23.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.453+0000 7f307ce00640 1 -- 172.21.15.27:0/1872975122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f307806e240 msgr2=0x7f30780c87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:23.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.453+0000 7f307ce00640 1 --2- 172.21.15.27:0/1872975122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f307806e240 0x7f30780c87e0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f30680ad040 tx=0x7f306801ac90 comp rx=0 tx=0).stop 2024-01-15T18:18:23.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.453+0000 7f307ce00640 1 -- 172.21.15.27:0/1872975122 shutdown_connections 2024-01-15T18:18:23.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.453+0000 7f307ce00640 1 --2- 172.21.15.27:0/1872975122 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3050067970 0x7f3050069e30 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:23.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.453+0000 7f307ce00640 1 --2- 172.21.15.27:0/1872975122 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f30780c92b0 0x7f30780cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:23.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.453+0000 7f307ce00640 1 --2- 172.21.15.27:0/1872975122 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f30780701a0 0x7f30780c1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:23.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.453+0000 7f307ce00640 1 --2- 172.21.15.27:0/1872975122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f307806e240 0x7f30780c87e0 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:23.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.453+0000 7f307ce00640 1 -- 172.21.15.27:0/1872975122 >> 172.21.15.27:0/1872975122 conn(0x7f30780cec60 msgr2=0x7f30780c9690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:23.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.453+0000 7f307ce00640 1 -- 172.21.15.27:0/1872975122 shutdown_connections 2024-01-15T18:18:23.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.453+0000 7f307ce00640 1 -- 172.21.15.27:0/1872975122 wait complete. 2024-01-15T18:18:23.472 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-15T18:18:23.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.613+0000 7f996c200640 1 Processor -- start 2024-01-15T18:18:23.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.613+0000 7f996c200640 1 -- start start 2024-01-15T18:18:23.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.617+0000 7f996c200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f996415f550 0x7f99640cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:23.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.617+0000 7f996c200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f99640cf110 0x7f99640701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:23.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.617+0000 7f996c200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99640b1cb0 con 0x7f9964158900 2024-01-15T18:18:23.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.617+0000 7f996c200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99640cf580 con 0x7f996415f550 2024-01-15T18:18:23.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.617+0000 7f996c200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f996415fa00 con 0x7f99640cf110 2024-01-15T18:18:23.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.617+0000 7f9968e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f996415f550 0x7f99640cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:23.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.617+0000 7f996a200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f99640cf110 0x7f99640701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:23.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.617+0000 7f9968e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f996415f550 0x7f99640cebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:35884/0 (socket says 172.21.15.27:35884) 2024-01-15T18:18:23.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.617+0000 7f9968e00640 1 -- 172.21.15.27:0/4043755422 learned_addr learned my addr 172.21.15.27:0/4043755422 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:23.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.617+0000 7f996a200640 1 -- 172.21.15.27:0/4043755422 >> v2:172.21.15.93:3300/0 conn(0x7f996415f550 msgr2=0x7f99640cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:23.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.617+0000 7f996a200640 1 --2- 172.21.15.27:0/4043755422 >> v2:172.21.15.93:3300/0 conn(0x7f996415f550 0x7f99640cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:23.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.617+0000 7f996a200640 1 -- 172.21.15.27:0/4043755422 >> v1:172.21.15.93:6789/0 conn(0x7f9964158900 legacy=0x7f996415bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:23.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.617+0000 7f996a200640 1 -- 172.21.15.27:0/4043755422 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99640b1f00 con 0x7f99640cf110 2024-01-15T18:18:23.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.621+0000 7f996a200640 1 --2- 172.21.15.27:0/4043755422 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f99640cf110 0x7f99640701e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f995c009d90 tx=0x7f995c01c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=570da9c8a7d64eff server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:23.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.621+0000 7f9963e00640 1 -- 172.21.15.27:0/4043755422 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9964070f30 con 0x7f99640cf110 2024-01-15T18:18:23.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.621+0000 7f9963e00640 1 -- 172.21.15.27:0/4043755422 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f995c0a59e0 con 0x7f99640cf110 2024-01-15T18:18:23.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.621+0000 7f9963e00640 1 -- 172.21.15.27:0/4043755422 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995c0a5d70 con 0x7f99640cf110 2024-01-15T18:18:23.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.621+0000 7f996c200640 1 -- 172.21.15.27:0/4043755422 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f99640cf110 msgr2=0x7f99640701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:23.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.621+0000 7f996c200640 1 --2- 172.21.15.27:0/4043755422 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f99640cf110 0x7f99640701e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f995c009d90 tx=0x7f995c01c920 comp rx=0 tx=0).stop 2024-01-15T18:18:23.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.621+0000 7f996c200640 1 -- 172.21.15.27:0/4043755422 shutdown_connections 2024-01-15T18:18:23.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.621+0000 7f996c200640 1 --2- 172.21.15.27:0/4043755422 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f99640cf110 0x7f99640701e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:23.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.621+0000 7f996c200640 1 --2- 172.21.15.27:0/4043755422 >> v2:172.21.15.93:3300/0 conn(0x7f996415f550 0x7f99640cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:23.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.621+0000 7f996c200640 1 -- 172.21.15.27:0/4043755422 >> 172.21.15.27:0/4043755422 conn(0x7f99640cb700 msgr2=0x7f99640cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:23.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.621+0000 7f996c200640 1 -- 172.21.15.27:0/4043755422 shutdown_connections 2024-01-15T18:18:23.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.621+0000 7f996c200640 1 -- 172.21.15.27:0/4043755422 wait complete. 2024-01-15T18:18:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.621+0000 7f996c200640 1 Processor -- start 2024-01-15T18:18:23.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f996c200640 1 -- start start 2024-01-15T18:18:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f996c200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99640cf110 0x7f99640b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f996c200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9964158900 0x7f996413b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f996c200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f996415f550 0x7f9964154bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f996c200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996414dd40 con 0x7f99640cf110 2024-01-15T18:18:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f996c200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996414d130 con 0x7f996415f550 2024-01-15T18:18:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f996c200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f996414d410 con 0x7f9964158900 2024-01-15T18:18:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f9969800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99640cf110 0x7f99640b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f9969800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99640cf110 0x7f99640b4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57928/0 (socket says 172.21.15.27:57928) 2024-01-15T18:18:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f9969800640 1 -- 172.21.15.27:0/3067760718 learned_addr learned my addr 172.21.15.27:0/3067760718 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f9968e00640 1 --2- 172.21.15.27:0/3067760718 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9964158900 0x7f996413b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:23.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f996a200640 1 --2- 172.21.15.27:0/3067760718 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f996415f550 0x7f9964154bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:23.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f9969800640 1 -- 172.21.15.27:0/3067760718 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9964158900 msgr2=0x7f996413b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:23.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f9969800640 1 --2- 172.21.15.27:0/3067760718 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9964158900 0x7f996413b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:23.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f9969800640 1 -- 172.21.15.27:0/3067760718 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f996415f550 msgr2=0x7f9964154bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:23.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f9969800640 1 --2- 172.21.15.27:0/3067760718 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f996415f550 0x7f9964154bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:23.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f9969800640 1 -- 172.21.15.27:0/3067760718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f995c009a70 con 0x7f99640cf110 2024-01-15T18:18:23.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f996a200640 1 --2- 172.21.15.27:0/3067760718 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f996415f550 0x7f9964154bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:18:23.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f9968e00640 1 --2- 172.21.15.27:0/3067760718 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9964158900 0x7f996413b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:18:23.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f9969800640 1 --2- 172.21.15.27:0/3067760718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99640cf110 0x7f99640b4b50 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f995400c500 tx=0x7f995400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:23.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f9961600640 1 -- 172.21.15.27:0/3067760718 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9954005220 con 0x7f99640cf110 2024-01-15T18:18:23.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f9961600640 1 -- 172.21.15.27:0/3067760718 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9954005380 con 0x7f99640cf110 2024-01-15T18:18:23.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f9961600640 1 -- 172.21.15.27:0/3067760718 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9954007270 con 0x7f99640cf110 2024-01-15T18:18:23.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f996c200640 1 -- 172.21.15.27:0/3067760718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f996414d6f0 con 0x7f99640cf110 2024-01-15T18:18:23.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f996c200640 1 -- 172.21.15.27:0/3067760718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f996414e3c0 con 0x7f99640cf110 2024-01-15T18:18:23.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.625+0000 7f996c200640 1 -- 172.21.15.27:0/3067760718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f996414deb0 con 0x7f99640cf110 2024-01-15T18:18:23.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.629+0000 7f9961600640 1 -- 172.21.15.27:0/3067760718 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f99540073d0 con 0x7f99640cf110 2024-01-15T18:18:23.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.629+0000 7f9961600640 1 --2- 172.21.15.27:0/3067760718 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f993c067940 0x7f993c069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:23.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.629+0000 7f9961600640 1 -- 172.21.15.27:0/3067760718 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f9954082c30 con 0x7f99640cf110 2024-01-15T18:18:23.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.629+0000 7f9968e00640 1 --2- 172.21.15.27:0/3067760718 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f993c067940 0x7f993c069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:23.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.629+0000 7f9968e00640 1 --2- 172.21.15.27:0/3067760718 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f993c067940 0x7f993c069e00 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f9964158050 tx=0x7f994c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:23.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.633+0000 7f9961600640 1 -- 172.21.15.27:0/3067760718 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9954051520 con 0x7f99640cf110 2024-01-15T18:18:23.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.789+0000 7f996c200640 1 -- 172.21.15.27:0/3067760718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7f996404e850 con 0x7f99640cf110 2024-01-15T18:18:23.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.793+0000 7f9961600640 1 -- 172.21.15.27:0/3067760718 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v37) v1 ==== 120+0+0 (secure 0 0 0) 0x7f9954054b40 con 0x7f99640cf110 2024-01-15T18:18:23.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.797+0000 7f996c200640 1 -- 172.21.15.27:0/3067760718 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f993c067940 msgr2=0x7f993c069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:23.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.797+0000 7f996c200640 1 --2- 172.21.15.27:0/3067760718 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f993c067940 0x7f993c069e00 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f9964158050 tx=0x7f994c008040 comp rx=0 tx=0).stop 2024-01-15T18:18:23.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.797+0000 7f996c200640 1 -- 172.21.15.27:0/3067760718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99640cf110 msgr2=0x7f99640b4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:23.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.797+0000 7f996c200640 1 --2- 172.21.15.27:0/3067760718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99640cf110 0x7f99640b4b50 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f995400c500 tx=0x7f995400c9c0 comp rx=0 tx=0).stop 2024-01-15T18:18:23.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.797+0000 7f996c200640 1 -- 172.21.15.27:0/3067760718 shutdown_connections 2024-01-15T18:18:23.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.797+0000 7f996c200640 1 --2- 172.21.15.27:0/3067760718 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f993c067940 0x7f993c069e00 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:23.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.797+0000 7f996c200640 1 --2- 172.21.15.27:0/3067760718 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f996415f550 0x7f9964154bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:23.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.797+0000 7f996c200640 1 --2- 172.21.15.27:0/3067760718 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9964158900 0x7f996413b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:23.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.797+0000 7f996c200640 1 --2- 172.21.15.27:0/3067760718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f99640cf110 0x7f99640b4b50 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:23.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.797+0000 7f996c200640 1 -- 172.21.15.27:0/3067760718 >> 172.21.15.27:0/3067760718 conn(0x7f99640cb700 msgr2=0x7f99640b3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:23.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.797+0000 7f996c200640 1 -- 172.21.15.27:0/3067760718 shutdown_connections 2024-01-15T18:18:23.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.797+0000 7f996c200640 1 -- 172.21.15.27:0/3067760718 wait complete. 2024-01-15T18:18:23.816 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-15T18:18:23.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.957+0000 7f49fd400640 1 Processor -- start 2024-01-15T18:18:23.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49fd400640 1 -- start start 2024-01-15T18:18:23.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49fd400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f49f815b100 0x7f49f8070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:23.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49fd400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f815a610 con 0x7f49f815ad20 2024-01-15T18:18:23.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49fd400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f815e3c0 con 0x7f49f8071420 2024-01-15T18:18:23.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49fd400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f80718b0 con 0x7f49f815b100 2024-01-15T18:18:23.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f6800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f49f815ad20 0x7f49f815d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46496/0 (socket says 172.21.15.27:46496) 2024-01-15T18:18:23.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f6800640 1 -- 172.21.15.27:0/1980434591 learned_addr learned my addr 172.21.15.27:0/1980434591 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:23.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f5e00640 1 --2- 172.21.15.27:0/1980434591 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f49f815b100 0x7f49f8070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:23.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f5400640 1 -- 172.21.15.27:0/1980434591 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688574519 0 0) 0x7f49f815a610 con 0x7f49f815ad20 2024-01-15T18:18:23.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f5400640 1 -- 172.21.15.27:0/1980434591 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49d8005740 con 0x7f49f815ad20 2024-01-15T18:18:23.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f5400640 1 -- 172.21.15.27:0/1980434591 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773458232 0 0) 0x7f49f815e3c0 con 0x7f49f8071420 2024-01-15T18:18:23.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f5400640 1 -- 172.21.15.27:0/1980434591 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49f815a610 con 0x7f49f8071420 2024-01-15T18:18:23.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f5400640 1 -- 172.21.15.27:0/1980434591 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 600851900 0 0) 0x7f49d8005740 con 0x7f49f815ad20 2024-01-15T18:18:23.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f5400640 1 -- 172.21.15.27:0/1980434591 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f49f815b100 msgr2=0x7f49f8070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:23.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f5400640 1 --2- 172.21.15.27:0/1980434591 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f49f815b100 0x7f49f8070ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:23.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f5400640 1 -- 172.21.15.27:0/1980434591 >> v1:172.21.15.93:6789/0 conn(0x7f49f8071420 legacy=0x7f49f815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:23.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f5400640 1 -- 172.21.15.27:0/1980434591 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f815e6a0 con 0x7f49f815ad20 2024-01-15T18:18:23.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f5400640 1 -- 172.21.15.27:0/1980434591 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f49e8001680 con 0x7f49f815ad20 2024-01-15T18:18:23.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f5400640 1 -- 172.21.15.27:0/1980434591 >> v1:172.21.15.27:6789/0 conn(0x7f49f815ad20 legacy=0x7f49f815d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:23.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f5400640 1 --2- 172.21.15.27:0/1980434591 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f49d8005dc0 0x7f49d80061c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:23.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f5400640 1 --2- 172.21.15.27:0/1980434591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49d80069e0 0x7f49d8006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:23.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f5400640 1 --2- 172.21.15.27:0/1980434591 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49d80073a0 0x7f49d800dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:23.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f5400640 1 -- 172.21.15.27:0/1980434591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f80718b0 con 0x7f49d80069e0 2024-01-15T18:18:23.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f5400640 1 -- 172.21.15.27:0/1980434591 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d800e270 con 0x7f49d80073a0 2024-01-15T18:18:23.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f5400640 1 -- 172.21.15.27:0/1980434591 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d800e430 con 0x7f49d8005dc0 2024-01-15T18:18:23.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f5e00640 1 --2- 172.21.15.27:0/1980434591 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f49f815b100 0x7f49f8070ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:18:23.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f5e00640 1 --2- 172.21.15.27:0/1980434591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49d80069e0 0x7f49d8006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:23.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f7200640 1 --2- 172.21.15.27:0/1980434591 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49d80073a0 0x7f49d800dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:23.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f6800640 1 --2- 172.21.15.27:0/1980434591 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f49d8005dc0 0x7f49d80061c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:23.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f5e00640 1 -- 172.21.15.27:0/1980434591 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f49d8005dc0 msgr2=0x7f49d80061c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:23.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f5e00640 1 --2- 172.21.15.27:0/1980434591 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f49d8005dc0 0x7f49d80061c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:23.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f5e00640 1 -- 172.21.15.27:0/1980434591 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49d80073a0 msgr2=0x7f49d800dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:23.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f5e00640 1 --2- 172.21.15.27:0/1980434591 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49d80073a0 0x7f49d800dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:23.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.961+0000 7f49f5e00640 1 -- 172.21.15.27:0/1980434591 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d8005740 con 0x7f49d80069e0 2024-01-15T18:18:23.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.965+0000 7f49f5e00640 1 --2- 172.21.15.27:0/1980434591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49d80069e0 0x7f49d8006e60 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f49e0009b20 tx=0x7f49e001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6061ca02e6a55754 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:23.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.965+0000 7f49f5400640 1 -- 172.21.15.27:0/1980434591 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e0004030 con 0x7f49d80069e0 2024-01-15T18:18:23.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.965+0000 7f49f5400640 1 -- 172.21.15.27:0/1980434591 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e0004190 con 0x7f49d80069e0 2024-01-15T18:18:23.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.965+0000 7f49f5400640 1 -- 172.21.15.27:0/1980434591 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e0004520 con 0x7f49d80069e0 2024-01-15T18:18:23.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.965+0000 7f49fd400640 1 -- 172.21.15.27:0/1980434591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49d80069e0 msgr2=0x7f49d8006e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:23.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.965+0000 7f49fd400640 1 --2- 172.21.15.27:0/1980434591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49d80069e0 0x7f49d8006e60 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f49e0009b20 tx=0x7f49e001ca90 comp rx=0 tx=0).stop 2024-01-15T18:18:23.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.965+0000 7f49f7200640 1 -- 172.21.15.27:0/1980434591 reap_dead start 2024-01-15T18:18:23.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.965+0000 7f49fd400640 1 -- 172.21.15.27:0/1980434591 shutdown_connections 2024-01-15T18:18:23.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.965+0000 7f49fd400640 1 -- 172.21.15.27:0/1980434591 >> 172.21.15.27:0/1980434591 conn(0x7f49f80cb700 msgr2=0x7f49f80cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:23.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.965+0000 7f49fd400640 1 -- 172.21.15.27:0/1980434591 shutdown_connections 2024-01-15T18:18:23.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.965+0000 7f49fd400640 1 -- 172.21.15.27:0/1980434591 wait complete. 2024-01-15T18:18:23.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.969+0000 7f49fd400640 1 Processor -- start 2024-01-15T18:18:23.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.969+0000 7f49fd400640 1 -- start start 2024-01-15T18:18:23.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.969+0000 7f49fd400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49d80069e0 0x7f49f8152510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:23.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.969+0000 7f49fd400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f49f814fc40 0x7f49f8152a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:23.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.969+0000 7f49fd400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49f813f440 0x7f49f813f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:23.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.969+0000 7f49fd400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f813f020 con 0x7f49d80069e0 2024-01-15T18:18:23.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.969+0000 7f49fd400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f813e3e0 con 0x7f49f813f440 2024-01-15T18:18:23.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.969+0000 7f49fd400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f813e680 con 0x7f49f814fc40 2024-01-15T18:18:23.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.969+0000 7f49f6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49d80069e0 0x7f49f8152510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:23.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.969+0000 7f49f6800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49d80069e0 0x7f49f8152510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57954/0 (socket says 172.21.15.27:57954) 2024-01-15T18:18:23.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.969+0000 7f49f6800640 1 -- 172.21.15.27:0/3666899643 learned_addr learned my addr 172.21.15.27:0/3666899643 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:23.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.969+0000 7f49f5e00640 1 --2- 172.21.15.27:0/3666899643 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f49f814fc40 0x7f49f8152a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:23.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.969+0000 7f49f7200640 1 --2- 172.21.15.27:0/3666899643 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49f813f440 0x7f49f813f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:23.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.969+0000 7f49f6800640 1 -- 172.21.15.27:0/3666899643 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f49f814fc40 msgr2=0x7f49f8152a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:23.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.969+0000 7f49f6800640 1 --2- 172.21.15.27:0/3666899643 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f49f814fc40 0x7f49f8152a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:23.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.969+0000 7f49f6800640 1 -- 172.21.15.27:0/3666899643 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49f813f440 msgr2=0x7f49f813f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:23.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.969+0000 7f49f6800640 1 --2- 172.21.15.27:0/3666899643 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49f813f440 0x7f49f813f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:23.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.969+0000 7f49f6800640 1 -- 172.21.15.27:0/3666899643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f815e190 con 0x7f49d80069e0 2024-01-15T18:18:23.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.969+0000 7f49f7200640 1 --2- 172.21.15.27:0/3666899643 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49f813f440 0x7f49f813f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:18:23.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.969+0000 7f49f6800640 1 --2- 172.21.15.27:0/3666899643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49d80069e0 0x7f49f8152510 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f49e8001060 tx=0x7f49e8007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:23.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.969+0000 7f49eea00640 1 -- 172.21.15.27:0/3666899643 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e8001610 con 0x7f49d80069e0 2024-01-15T18:18:23.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.969+0000 7f49eea00640 1 -- 172.21.15.27:0/3666899643 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e800f9c0 con 0x7f49d80069e0 2024-01-15T18:18:23.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.969+0000 7f49eea00640 1 -- 172.21.15.27:0/3666899643 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e800fd70 con 0x7f49d80069e0 2024-01-15T18:18:23.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.969+0000 7f49fd400640 1 -- 172.21.15.27:0/3666899643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49f813e960 con 0x7f49d80069e0 2024-01-15T18:18:23.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.969+0000 7f49fd400640 1 -- 172.21.15.27:0/3666899643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f813fe30 con 0x7f49d80069e0 2024-01-15T18:18:23.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.973+0000 7f49fd400640 1 -- 172.21.15.27:0/3666899643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49f813f160 con 0x7f49d80069e0 2024-01-15T18:18:23.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.973+0000 7f49eea00640 1 -- 172.21.15.27:0/3666899643 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f49e8027430 con 0x7f49d80069e0 2024-01-15T18:18:23.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.973+0000 7f49eea00640 1 --2- 172.21.15.27:0/3666899643 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f49d0067970 0x7f49d0069e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:23.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.973+0000 7f49eea00640 1 -- 172.21.15.27:0/3666899643 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f49e8083c30 con 0x7f49d80069e0 2024-01-15T18:18:23.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.973+0000 7f49f5e00640 1 --2- 172.21.15.27:0/3666899643 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f49d0067970 0x7f49d0069e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:23.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.977+0000 7f49f5e00640 1 --2- 172.21.15.27:0/3666899643 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f49d0067970 0x7f49d0069e30 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f49e00ad000 tx=0x7f49e00af040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:23.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:23.977+0000 7f49eea00640 1 -- 172.21.15.27:0/3666899643 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f49e80525a0 con 0x7f49d80069e0 2024-01-15T18:18:24.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.137+0000 7f49fd400640 1 -- 172.21.15.27:0/3666899643 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f49f813f160 con 0x7f49d80069e0 2024-01-15T18:18:24.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.137+0000 7f49eea00640 1 -- 172.21.15.27:0/3666899643 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v38) v1 ==== 104+0+0 (secure 0 0 0) 0x7f49f813f160 con 0x7f49d80069e0 2024-01-15T18:18:24.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.141+0000 7f49fd400640 1 -- 172.21.15.27:0/3666899643 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f49d0067970 msgr2=0x7f49d0069e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:24.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.141+0000 7f49fd400640 1 --2- 172.21.15.27:0/3666899643 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f49d0067970 0x7f49d0069e30 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f49e00ad000 tx=0x7f49e00af040 comp rx=0 tx=0).stop 2024-01-15T18:18:24.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.141+0000 7f49fd400640 1 -- 172.21.15.27:0/3666899643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49d80069e0 msgr2=0x7f49f8152510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:24.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.141+0000 7f49fd400640 1 --2- 172.21.15.27:0/3666899643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49d80069e0 0x7f49f8152510 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f49e8001060 tx=0x7f49e8007720 comp rx=0 tx=0).stop 2024-01-15T18:18:24.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.141+0000 7f49fd400640 1 -- 172.21.15.27:0/3666899643 shutdown_connections 2024-01-15T18:18:24.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.141+0000 7f49fd400640 1 --2- 172.21.15.27:0/3666899643 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f49d0067970 0x7f49d0069e30 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:24.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.141+0000 7f49fd400640 1 --2- 172.21.15.27:0/3666899643 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f49f813f440 0x7f49f813f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:24.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.141+0000 7f49fd400640 1 --2- 172.21.15.27:0/3666899643 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f49f814fc40 0x7f49f8152a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:24.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.141+0000 7f49fd400640 1 --2- 172.21.15.27:0/3666899643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49d80069e0 0x7f49f8152510 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:24.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.141+0000 7f49fd400640 1 -- 172.21.15.27:0/3666899643 >> 172.21.15.27:0/3666899643 conn(0x7f49f80cb700 msgr2=0x7f49f80b2610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:24.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.141+0000 7f49fd400640 1 -- 172.21.15.27:0/3666899643 shutdown_connections 2024-01-15T18:18:24.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.141+0000 7f49fd400640 1 -- 172.21.15.27:0/3666899643 wait complete. 2024-01-15T18:18:24.161 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2024-01-15T18:18:24.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.305+0000 7f4888c00640 1 Processor -- start 2024-01-15T18:18:24.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.305+0000 7f4888c00640 1 -- start start 2024-01-15T18:18:24.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.305+0000 7f4888c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f48840cf450 0x7f48840d1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:24.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.305+0000 7f4888c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f48840cba60 0x7f48840cf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:24.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.305+0000 7f4888c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48840cefa0 con 0x7f488415ee30 2024-01-15T18:18:24.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.305+0000 7f4888c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48840ce240 con 0x7f48840cba60 2024-01-15T18:18:24.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.305+0000 7f4888c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48840cfd70 con 0x7f48840cf450 2024-01-15T18:18:24.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.305+0000 7f4881400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f48840cba60 0x7f48840cf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:24.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.305+0000 7f4881e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f48840cf450 0x7f48840d1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:24.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.305+0000 7f4881400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f48840cba60 0x7f48840cf830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:35926/0 (socket says 172.21.15.27:35926) 2024-01-15T18:18:24.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.305+0000 7f4881400640 1 -- 172.21.15.27:0/1328455165 learned_addr learned my addr 172.21.15.27:0/1328455165 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:24.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.305+0000 7f4881400640 1 -- 172.21.15.27:0/1328455165 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f48840cf450 msgr2=0x7f48840d1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:24.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.305+0000 7f4881400640 1 --2- 172.21.15.27:0/1328455165 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f48840cf450 0x7f48840d1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:24.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.305+0000 7f4881400640 1 -- 172.21.15.27:0/1328455165 >> v1:172.21.15.93:6789/0 conn(0x7f488415ee30 legacy=0x7f488415f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:24.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.305+0000 7f4881400640 1 -- 172.21.15.27:0/1328455165 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48840ccd70 con 0x7f48840cba60 2024-01-15T18:18:24.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.305+0000 7f4881e00640 1 --2- 172.21.15.27:0/1328455165 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f48840cf450 0x7f48840d1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:18:24.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.309+0000 7f4881400640 1 --2- 172.21.15.27:0/1328455165 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48840cba60 0x7f48840cf830 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f486c009870 tx=0x7f486c01c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=754d1653dab730d9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:24.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.309+0000 7f4880a00640 1 -- 172.21.15.27:0/1328455165 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486c0a7070 con 0x7f48840cba60 2024-01-15T18:18:24.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.309+0000 7f4880a00640 1 -- 172.21.15.27:0/1328455165 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486c0ab430 con 0x7f48840cba60 2024-01-15T18:18:24.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.309+0000 7f4880a00640 1 -- 172.21.15.27:0/1328455165 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486c0ab7c0 con 0x7f48840cba60 2024-01-15T18:18:24.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.309+0000 7f4888c00640 1 -- 172.21.15.27:0/1328455165 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48840cba60 msgr2=0x7f48840cf830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:24.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.309+0000 7f4888c00640 1 --2- 172.21.15.27:0/1328455165 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48840cba60 0x7f48840cf830 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f486c009870 tx=0x7f486c01c540 comp rx=0 tx=0).stop 2024-01-15T18:18:24.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.309+0000 7f4888c00640 1 -- 172.21.15.27:0/1328455165 shutdown_connections 2024-01-15T18:18:24.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.309+0000 7f4888c00640 1 --2- 172.21.15.27:0/1328455165 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48840cba60 0x7f48840cf830 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:24.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.309+0000 7f4888c00640 1 --2- 172.21.15.27:0/1328455165 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f48840cf450 0x7f48840d1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:24.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.309+0000 7f4888c00640 1 -- 172.21.15.27:0/1328455165 >> 172.21.15.27:0/1328455165 conn(0x7f48840d0e70 msgr2=0x7f48840d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:24.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.309+0000 7f4888c00640 1 -- 172.21.15.27:0/1328455165 shutdown_connections 2024-01-15T18:18:24.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.309+0000 7f4888c00640 1 -- 172.21.15.27:0/1328455165 wait complete. 2024-01-15T18:18:24.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.313+0000 7f4888c00640 1 Processor -- start 2024-01-15T18:18:24.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.313+0000 7f4888c00640 1 -- start start 2024-01-15T18:18:24.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.313+0000 7f4888c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48840cba60 0x7f4884156fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:24.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.313+0000 7f4888c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48840cf450 0x7f48840ca480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:24.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.313+0000 7f4888c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f488415ee30 0x7f4884154e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:24.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.313+0000 7f4888c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48840c8600 con 0x7f48840cf450 2024-01-15T18:18:24.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.313+0000 7f4888c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48840c8320 con 0x7f48840cba60 2024-01-15T18:18:24.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.313+0000 7f4888c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4884156590 con 0x7f488415ee30 2024-01-15T18:18:24.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.313+0000 7f4881400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48840cf450 0x7f48840ca480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:24.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.313+0000 7f4881400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48840cf450 0x7f48840ca480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57964/0 (socket says 172.21.15.27:57964) 2024-01-15T18:18:24.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.313+0000 7f4881400640 1 -- 172.21.15.27:0/940273773 learned_addr learned my addr 172.21.15.27:0/940273773 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:24.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.313+0000 7f4881e00640 1 --2- 172.21.15.27:0/940273773 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48840cba60 0x7f4884156fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:24.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.313+0000 7f4882800640 1 --2- 172.21.15.27:0/940273773 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f488415ee30 0x7f4884154e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:24.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.313+0000 7f4881400640 1 -- 172.21.15.27:0/940273773 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f488415ee30 msgr2=0x7f4884154e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:24.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.313+0000 7f4881400640 1 --2- 172.21.15.27:0/940273773 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f488415ee30 0x7f4884154e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:24.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.313+0000 7f4881400640 1 -- 172.21.15.27:0/940273773 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48840cba60 msgr2=0x7f4884156fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:24.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.313+0000 7f4881400640 1 --2- 172.21.15.27:0/940273773 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48840cba60 0x7f4884156fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:24.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.313+0000 7f4881400640 1 -- 172.21.15.27:0/940273773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4874009420 con 0x7f48840cf450 2024-01-15T18:18:24.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.313+0000 7f4881e00640 1 --2- 172.21.15.27:0/940273773 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48840cba60 0x7f4884156fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:18:24.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.313+0000 7f4881400640 1 --2- 172.21.15.27:0/940273773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48840cf450 0x7f48840ca480 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f486c01a170 tx=0x7f486c0192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:24.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.313+0000 7f487a000640 1 -- 172.21.15.27:0/940273773 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486c0ba070 con 0x7f48840cf450 2024-01-15T18:18:24.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.313+0000 7f4888c00640 1 -- 172.21.15.27:0/940273773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f486c0094d0 con 0x7f48840cf450 2024-01-15T18:18:24.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.313+0000 7f487a000640 1 -- 172.21.15.27:0/940273773 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486c0b4420 con 0x7f48840cf450 2024-01-15T18:18:24.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.313+0000 7f487a000640 1 -- 172.21.15.27:0/940273773 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f486c0b47d0 con 0x7f48840cf450 2024-01-15T18:18:24.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.313+0000 7f4888c00640 1 -- 172.21.15.27:0/940273773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4884156b50 con 0x7f48840cf450 2024-01-15T18:18:24.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.317+0000 7f4888c00640 1 -- 172.21.15.27:0/940273773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48841566d0 con 0x7f48840cf450 2024-01-15T18:18:24.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.317+0000 7f487a000640 1 -- 172.21.15.27:0/940273773 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f486c0b4930 con 0x7f48840cf450 2024-01-15T18:18:24.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.317+0000 7f487a000640 1 --2- 172.21.15.27:0/940273773 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f485c067940 0x7f485c069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:24.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.317+0000 7f487a000640 1 -- 172.21.15.27:0/940273773 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f486c11ce70 con 0x7f48840cf450 2024-01-15T18:18:24.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.317+0000 7f4881e00640 1 --2- 172.21.15.27:0/940273773 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f485c067940 0x7f485c069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:24.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.321+0000 7f4881e00640 1 --2- 172.21.15.27:0/940273773 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f485c067940 0x7f485c069e00 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f488406f9a0 tx=0x7f4874008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:24.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.321+0000 7f487a000640 1 -- 172.21.15.27:0/940273773 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f486c0ebb00 con 0x7f48840cf450 2024-01-15T18:18:24.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.481+0000 7f4888c00640 1 -- 172.21.15.27:0/940273773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7f48841566d0 con 0x7f48840cf450 2024-01-15T18:18:24.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.485+0000 7f487a000640 1 -- 172.21.15.27:0/940273773 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v39) v1 ==== 115+0+0 (secure 0 0 0) 0x7f48841566d0 con 0x7f48840cf450 2024-01-15T18:18:24.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.485+0000 7f4888c00640 1 -- 172.21.15.27:0/940273773 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f485c067940 msgr2=0x7f485c069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:24.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.485+0000 7f4888c00640 1 --2- 172.21.15.27:0/940273773 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f485c067940 0x7f485c069e00 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f488406f9a0 tx=0x7f4874008040 comp rx=0 tx=0).stop 2024-01-15T18:18:24.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.485+0000 7f4888c00640 1 -- 172.21.15.27:0/940273773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48840cf450 msgr2=0x7f48840ca480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:24.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.485+0000 7f4888c00640 1 --2- 172.21.15.27:0/940273773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48840cf450 0x7f48840ca480 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f486c01a170 tx=0x7f486c0192e0 comp rx=0 tx=0).stop 2024-01-15T18:18:24.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.485+0000 7f4888c00640 1 -- 172.21.15.27:0/940273773 shutdown_connections 2024-01-15T18:18:24.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.485+0000 7f4888c00640 1 --2- 172.21.15.27:0/940273773 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f485c067940 0x7f485c069e00 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:24.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.485+0000 7f4888c00640 1 --2- 172.21.15.27:0/940273773 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f488415ee30 0x7f4884154e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:24.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.485+0000 7f4888c00640 1 --2- 172.21.15.27:0/940273773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48840cf450 0x7f48840ca480 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:24.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.485+0000 7f4888c00640 1 --2- 172.21.15.27:0/940273773 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f48840cba60 0x7f4884156fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:24.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.485+0000 7f4888c00640 1 -- 172.21.15.27:0/940273773 >> 172.21.15.27:0/940273773 conn(0x7f48840d0e70 msgr2=0x7f488415e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:24.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.489+0000 7f4888c00640 1 -- 172.21.15.27:0/940273773 shutdown_connections 2024-01-15T18:18:24.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.489+0000 7f4888c00640 1 -- 172.21.15.27:0/940273773 wait complete. 2024-01-15T18:18:24.508 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled' 2024-01-15T18:18:24.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.649+0000 7f216f200640 1 Processor -- start 2024-01-15T18:18:24.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.649+0000 7f216f200640 1 -- start start 2024-01-15T18:18:24.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.649+0000 7f216f200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f2168160a20 0x7f21680b1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:24.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.649+0000 7f216f200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f21680cd7d0 0x7f21680cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:24.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.649+0000 7f216f200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21680d1a70 con 0x7f21680ce190 2024-01-15T18:18:24.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.649+0000 7f216f200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2168160e00 con 0x7f21680cd7d0 2024-01-15T18:18:24.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.649+0000 7f216f200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21680d1cc0 con 0x7f2168160a20 2024-01-15T18:18:24.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.649+0000 7f2167400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f21680cd7d0 0x7f21680cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:24.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.649+0000 7f2167400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f21680cd7d0 0x7f21680cdc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57980/0 (socket says 172.21.15.27:57980) 2024-01-15T18:18:24.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.649+0000 7f2167400640 1 -- 172.21.15.27:0/3634215464 learned_addr learned my addr 172.21.15.27:0/3634215464 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:24.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.653+0000 7f2167e00640 1 --2- 172.21.15.27:0/3634215464 >> v2:172.21.15.93:3300/0 conn(0x7f2168160a20 0x7f21680b1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:24.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.653+0000 7f2167400640 1 -- 172.21.15.27:0/3634215464 >> v2:172.21.15.93:3300/0 conn(0x7f2168160a20 msgr2=0x7f21680b1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:24.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.653+0000 7f2167400640 1 --2- 172.21.15.27:0/3634215464 >> v2:172.21.15.93:3300/0 conn(0x7f2168160a20 0x7f21680b1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:24.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.653+0000 7f2167400640 1 -- 172.21.15.27:0/3634215464 >> v1:172.21.15.93:6789/0 conn(0x7f21680ce190 legacy=0x7f21680cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:24.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.653+0000 7f2167400640 1 -- 172.21.15.27:0/3634215464 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216806c010 con 0x7f21680cd7d0 2024-01-15T18:18:24.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.653+0000 7f2167e00640 1 --2- 172.21.15.27:0/3634215464 >> v2:172.21.15.93:3300/0 conn(0x7f2168160a20 0x7f21680b1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:18:24.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.653+0000 7f2167400640 1 --2- 172.21.15.27:0/3634215464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21680cd7d0 0x7f21680cdc50 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f2168073de0 tx=0x7f215001c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=690853469f06d9dc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:24.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.653+0000 7f2166a00640 1 -- 172.21.15.27:0/3634215464 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21500ac410 con 0x7f21680cd7d0 2024-01-15T18:18:24.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.653+0000 7f2166a00640 1 -- 172.21.15.27:0/3634215464 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21500ac570 con 0x7f21680cd7d0 2024-01-15T18:18:24.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.653+0000 7f2166a00640 1 -- 172.21.15.27:0/3634215464 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21500ac900 con 0x7f21680cd7d0 2024-01-15T18:18:24.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f216f200640 1 -- 172.21.15.27:0/3634215464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21680cd7d0 msgr2=0x7f21680cdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:24.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f216f200640 1 --2- 172.21.15.27:0/3634215464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21680cd7d0 0x7f21680cdc50 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f2168073de0 tx=0x7f215001c4c0 comp rx=0 tx=0).stop 2024-01-15T18:18:24.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f216f200640 1 -- 172.21.15.27:0/3634215464 shutdown_connections 2024-01-15T18:18:24.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f216f200640 1 --2- 172.21.15.27:0/3634215464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21680cd7d0 0x7f21680cdc50 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:24.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f216f200640 1 --2- 172.21.15.27:0/3634215464 >> v2:172.21.15.93:3300/0 conn(0x7f2168160a20 0x7f21680b1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:24.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f216f200640 1 -- 172.21.15.27:0/3634215464 >> 172.21.15.27:0/3634215464 conn(0x7f21680d0f50 msgr2=0x7f21680d1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:24.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f216f200640 1 -- 172.21.15.27:0/3634215464 shutdown_connections 2024-01-15T18:18:24.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f216f200640 1 -- 172.21.15.27:0/3634215464 wait complete. 2024-01-15T18:18:24.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f216f200640 1 Processor -- start 2024-01-15T18:18:24.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f216f200640 1 -- start start 2024-01-15T18:18:24.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f216f200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21680cd7d0 0x7f21680caa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:24.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f216f200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f21680ce190 0x7f21681523f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:24.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f216f200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2168160a20 0x7f21680c8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:24.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f216f200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2168152b90 con 0x7f21680cd7d0 2024-01-15T18:18:24.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f216f200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21680cbed0 con 0x7f21680ce190 2024-01-15T18:18:24.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f216f200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f216815a220 con 0x7f2168160a20 2024-01-15T18:18:24.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f2167e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21680cd7d0 0x7f21680caa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:24.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f2167e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21680cd7d0 0x7f21680caa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57992/0 (socket says 172.21.15.27:57992) 2024-01-15T18:18:24.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f2167e00640 1 -- 172.21.15.27:0/2867054340 learned_addr learned my addr 172.21.15.27:0/2867054340 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:24.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f2167400640 1 --2- 172.21.15.27:0/2867054340 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f21680ce190 0x7f21681523f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:24.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f216d200640 1 --2- 172.21.15.27:0/2867054340 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2168160a20 0x7f21680c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:24.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f2167e00640 1 -- 172.21.15.27:0/2867054340 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2168160a20 msgr2=0x7f21680c8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:24.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f2167e00640 1 --2- 172.21.15.27:0/2867054340 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2168160a20 0x7f21680c8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:24.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f2167e00640 1 -- 172.21.15.27:0/2867054340 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f21680ce190 msgr2=0x7f21681523f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:24.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f2167e00640 1 --2- 172.21.15.27:0/2867054340 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f21680ce190 0x7f21681523f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:24.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.657+0000 7f2167e00640 1 -- 172.21.15.27:0/2867054340 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21500094d0 con 0x7f21680cd7d0 2024-01-15T18:18:24.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.661+0000 7f2167e00640 1 --2- 172.21.15.27:0/2867054340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21680cd7d0 0x7f21680caa40 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f215800c7c0 tx=0x7f215800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:24.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.661+0000 7f215fe00640 1 -- 172.21.15.27:0/2867054340 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2158007a30 con 0x7f21680cd7d0 2024-01-15T18:18:24.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.661+0000 7f215fe00640 1 -- 172.21.15.27:0/2867054340 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2158007b90 con 0x7f21680cd7d0 2024-01-15T18:18:24.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.661+0000 7f216f200640 1 -- 172.21.15.27:0/2867054340 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21680cc120 con 0x7f21680cd7d0 2024-01-15T18:18:24.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.661+0000 7f215fe00640 1 -- 172.21.15.27:0/2867054340 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215800ce70 con 0x7f21680cd7d0 2024-01-15T18:18:24.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.661+0000 7f216f200640 1 -- 172.21.15.27:0/2867054340 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2168152930 con 0x7f21680cd7d0 2024-01-15T18:18:24.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.661+0000 7f216f200640 1 -- 172.21.15.27:0/2867054340 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2168152d00 con 0x7f21680cd7d0 2024-01-15T18:18:24.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.661+0000 7f215fe00640 1 -- 172.21.15.27:0/2867054340 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f21580113f0 con 0x7f21680cd7d0 2024-01-15T18:18:24.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.661+0000 7f215fe00640 1 --2- 172.21.15.27:0/2867054340 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2140067970 0x7f2140069e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:24.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.661+0000 7f215fe00640 1 -- 172.21.15.27:0/2867054340 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f21580836e0 con 0x7f21680cd7d0 2024-01-15T18:18:24.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.661+0000 7f2167400640 1 --2- 172.21.15.27:0/2867054340 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2140067970 0x7f2140069e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:24.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.665+0000 7f2167400640 1 --2- 172.21.15.27:0/2867054340 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2140067970 0x7f2140069e30 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f21500ae000 tx=0x7f2150009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:24.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.665+0000 7f215fe00640 1 -- 172.21.15.27:0/2867054340 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2168152d00 con 0x7f21680cd7d0 2024-01-15T18:18:24.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:24.881+0000 7f216f200640 1 -- 172.21.15.27:0/2867054340 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled"]} v 0) v1 -- 0x7f2168152cd0 con 0x7f21680cd7d0 2024-01-15T18:18:25.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.133+0000 7f215fe00640 1 -- 172.21.15.27:0/2867054340 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled"]}]=0 v623) v1 ==== 153+0+0 (secure 0 0 0) 0x7f2158052100 con 0x7f21680cd7d0 2024-01-15T18:18:25.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.137+0000 7f216f200640 1 -- 172.21.15.27:0/2867054340 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2140067970 msgr2=0x7f2140069e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:25.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.137+0000 7f216f200640 1 --2- 172.21.15.27:0/2867054340 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2140067970 0x7f2140069e30 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f21500ae000 tx=0x7f2150009040 comp rx=0 tx=0).stop 2024-01-15T18:18:25.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.137+0000 7f216f200640 1 -- 172.21.15.27:0/2867054340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21680cd7d0 msgr2=0x7f21680caa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:25.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.137+0000 7f216f200640 1 --2- 172.21.15.27:0/2867054340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21680cd7d0 0x7f21680caa40 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f215800c7c0 tx=0x7f215800cc80 comp rx=0 tx=0).stop 2024-01-15T18:18:25.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.137+0000 7f216f200640 1 -- 172.21.15.27:0/2867054340 shutdown_connections 2024-01-15T18:18:25.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.137+0000 7f216f200640 1 --2- 172.21.15.27:0/2867054340 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2140067970 0x7f2140069e30 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:25.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.137+0000 7f216f200640 1 --2- 172.21.15.27:0/2867054340 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2168160a20 0x7f21680c8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:25.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.137+0000 7f216f200640 1 --2- 172.21.15.27:0/2867054340 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f21680ce190 0x7f21681523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:25.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.137+0000 7f216f200640 1 --2- 172.21.15.27:0/2867054340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21680cd7d0 0x7f21680caa40 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:25.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.137+0000 7f216f200640 1 -- 172.21.15.27:0/2867054340 >> 172.21.15.27:0/2867054340 conn(0x7f21680d0f50 msgr2=0x7f21680cf110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:25.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.137+0000 7f216f200640 1 -- 172.21.15.27:0/2867054340 shutdown_connections 2024-01-15T18:18:25.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.137+0000 7f216f200640 1 -- 172.21.15.27:0/2867054340 wait complete. 2024-01-15T18:18:25.158 INFO:tasks.cephfs_test_runner:Test that the progress bars for ongoing clones is removed from the ... ok 2024-01-15T18:18:25.158 INFO:tasks.cephfs_test_runner:Starting test: test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) Test that the progress bars for ongoing clones is removed from the 2024-01-15T18:18:25.159 INFO:tasks.cephfs_test_runner:test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T18:18:25.159 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:18:25.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.301+0000 7fb567000640 1 Processor -- start 2024-01-15T18:18:25.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.301+0000 7fb567000640 1 -- start start 2024-01-15T18:18:25.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.301+0000 7fb567000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb5600b2390 0x7fb5600cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:25.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.301+0000 7fb567000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5600b2770 con 0x7fb5600cdf90 2024-01-15T18:18:25.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.301+0000 7fb567000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb56006c060 con 0x7fb56006bb80 2024-01-15T18:18:25.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.301+0000 7fb567000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb56006c340 con 0x7fb5600b2390 2024-01-15T18:18:25.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.301+0000 7fb55fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb5600b2390 0x7fb5600cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:25.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.301+0000 7fb565000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb5600cdf90 0x7fb5600b0880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46512/0 (socket says 172.21.15.27:46512) 2024-01-15T18:18:25.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.301+0000 7fb565000640 1 -- 172.21.15.27:0/3440957006 learned_addr learned my addr 172.21.15.27:0/3440957006 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:25.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.301+0000 7fb55fe00640 1 -- 172.21.15.27:0/3440957006 >> v1:172.21.15.93:6789/0 conn(0x7fb56006bb80 legacy=0x7fb5600cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:25.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.301+0000 7fb55fe00640 1 -- 172.21.15.27:0/3440957006 >> v1:172.21.15.27:6789/0 conn(0x7fb5600cdf90 legacy=0x7fb5600b0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:25.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.301+0000 7fb55fe00640 1 -- 172.21.15.27:0/3440957006 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb56006c620 con 0x7fb5600b2390 2024-01-15T18:18:25.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.301+0000 7fb55ea00640 1 -- 172.21.15.27:0/3440957006 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 24985363 0 0) 0x7fb5600b2770 con 0x7fb5600cdf90 2024-01-15T18:18:25.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.305+0000 7fb55fe00640 1 --2- 172.21.15.27:0/3440957006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5600b2390 0x7fb5600cf780 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fb5500097c0 tx=0x7fb55001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36a763e674828f7d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:25.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.305+0000 7fb55ea00640 1 -- 172.21.15.27:0/3440957006 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5500ab610 con 0x7fb5600b2390 2024-01-15T18:18:25.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.305+0000 7fb55ea00640 1 -- 172.21.15.27:0/3440957006 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5500ab770 con 0x7fb5600b2390 2024-01-15T18:18:25.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.305+0000 7fb55ea00640 1 -- 172.21.15.27:0/3440957006 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5500abb00 con 0x7fb5600b2390 2024-01-15T18:18:25.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.305+0000 7fb567000640 1 -- 172.21.15.27:0/3440957006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5600b2390 msgr2=0x7fb5600cf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:25.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.305+0000 7fb567000640 1 --2- 172.21.15.27:0/3440957006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5600b2390 0x7fb5600cf780 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fb5500097c0 tx=0x7fb55001c410 comp rx=0 tx=0).stop 2024-01-15T18:18:25.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.305+0000 7fb567000640 1 -- 172.21.15.27:0/3440957006 shutdown_connections 2024-01-15T18:18:25.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.305+0000 7fb567000640 1 --2- 172.21.15.27:0/3440957006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5600b2390 0x7fb5600cf780 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:25.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.305+0000 7fb567000640 1 -- 172.21.15.27:0/3440957006 >> 172.21.15.27:0/3440957006 conn(0x7fb5600b2a10 msgr2=0x7fb5600b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:25.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.305+0000 7fb567000640 1 -- 172.21.15.27:0/3440957006 shutdown_connections 2024-01-15T18:18:25.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.305+0000 7fb567000640 1 -- 172.21.15.27:0/3440957006 wait complete. 2024-01-15T18:18:25.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.309+0000 7fb567000640 1 Processor -- start 2024-01-15T18:18:25.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.309+0000 7fb567000640 1 -- start start 2024-01-15T18:18:25.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.309+0000 7fb567000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb56006bb80 0x7fb56014c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:25.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.309+0000 7fb567000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5600b2390 0x7fb5600b4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:25.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.309+0000 7fb567000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb5600cdf90 0x7fb5600c9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:25.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.309+0000 7fb567000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5600cb550 con 0x7fb5600b2390 2024-01-15T18:18:25.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.309+0000 7fb567000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5600c8e80 con 0x7fb5600cdf90 2024-01-15T18:18:25.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.309+0000 7fb567000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb56014cb60 con 0x7fb56006bb80 2024-01-15T18:18:25.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.309+0000 7fb55f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5600b2390 0x7fb5600b4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:25.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.309+0000 7fb55f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5600b2390 0x7fb5600b4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58020/0 (socket says 172.21.15.27:58020) 2024-01-15T18:18:25.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.309+0000 7fb55f400640 1 -- 172.21.15.27:0/3959545489 learned_addr learned my addr 172.21.15.27:0/3959545489 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:25.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.309+0000 7fb55fe00640 1 --2- 172.21.15.27:0/3959545489 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb56006bb80 0x7fb56014c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:25.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.309+0000 7fb565000640 1 --2- 172.21.15.27:0/3959545489 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb5600cdf90 0x7fb5600c9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:25.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.309+0000 7fb55f400640 1 -- 172.21.15.27:0/3959545489 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb56006bb80 msgr2=0x7fb56014c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:25.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.309+0000 7fb55f400640 1 --2- 172.21.15.27:0/3959545489 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb56006bb80 0x7fb56014c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:25.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.309+0000 7fb55f400640 1 -- 172.21.15.27:0/3959545489 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb5600cdf90 msgr2=0x7fb5600c9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:25.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.309+0000 7fb55f400640 1 --2- 172.21.15.27:0/3959545489 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb5600cdf90 0x7fb5600c9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:25.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.309+0000 7fb55f400640 1 -- 172.21.15.27:0/3959545489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb550009420 con 0x7fb5600b2390 2024-01-15T18:18:25.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.309+0000 7fb55f400640 1 --2- 172.21.15.27:0/3959545489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5600b2390 0x7fb5600b4960 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fb54800b410 tx=0x7fb54800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:25.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.313+0000 7fb557e00640 1 -- 172.21.15.27:0/3959545489 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb548004020 con 0x7fb5600b2390 2024-01-15T18:18:25.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.313+0000 7fb557e00640 1 -- 172.21.15.27:0/3959545489 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb548004180 con 0x7fb5600b2390 2024-01-15T18:18:25.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.313+0000 7fb567000640 1 -- 172.21.15.27:0/3959545489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5600c7f90 con 0x7fb5600b2390 2024-01-15T18:18:25.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.313+0000 7fb557e00640 1 -- 172.21.15.27:0/3959545489 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb548011630 con 0x7fb5600b2390 2024-01-15T18:18:25.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.313+0000 7fb567000640 1 -- 172.21.15.27:0/3959545489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5600c8550 con 0x7fb5600b2390 2024-01-15T18:18:25.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.313+0000 7fb567000640 1 -- 172.21.15.27:0/3959545489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb56004e850 con 0x7fb5600b2390 2024-01-15T18:18:25.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.313+0000 7fb557e00640 1 -- 172.21.15.27:0/3959545489 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7fb548011790 con 0x7fb5600b2390 2024-01-15T18:18:25.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.313+0000 7fb557e00640 1 --2- 172.21.15.27:0/3959545489 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb538067970 0x7fb538069e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:25.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.313+0000 7fb557e00640 1 -- 172.21.15.27:0/3959545489 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5055+0+0 (secure 0 0 0) 0x7fb548082b70 con 0x7fb5600b2390 2024-01-15T18:18:25.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.317+0000 7fb55fe00640 1 --2- 172.21.15.27:0/3959545489 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb538067970 0x7fb538069e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:25.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.317+0000 7fb55fe00640 1 --2- 172.21.15.27:0/3959545489 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb538067970 0x7fb538069e30 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fb5500af000 tx=0x7fb5500a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:25.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.317+0000 7fb557e00640 1 -- 172.21.15.27:0/3959545489 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb5480b7280 con 0x7fb5600b2390 2024-01-15T18:18:25.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.477+0000 7fb567000640 1 -- 172.21.15.27:0/3959545489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb56006f430 con 0x7fb5600b2390 2024-01-15T18:18:25.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.477+0000 7fb557e00640 1 -- 172.21.15.27:0/3959545489 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+13067 (secure 0 0 0) 0x7fb548015020 con 0x7fb5600b2390 2024-01-15T18:18:25.477 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:18:25.477 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":100,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:18:20.055597+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6835/259930036":"2024-01-16T18:18:17.046652+0000","172.21.15.27:6833/259930036":"2024-01-16T18:18:17.046652+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:18:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.481+0000 7fb567000640 1 -- 172.21.15.27:0/3959545489 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb538067970 msgr2=0x7fb538069e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.481+0000 7fb567000640 1 --2- 172.21.15.27:0/3959545489 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb538067970 0x7fb538069e30 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fb5500af000 tx=0x7fb5500a7000 comp rx=0 tx=0).stop 2024-01-15T18:18:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.481+0000 7fb567000640 1 -- 172.21.15.27:0/3959545489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5600b2390 msgr2=0x7fb5600b4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.481+0000 7fb567000640 1 --2- 172.21.15.27:0/3959545489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5600b2390 0x7fb5600b4960 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fb54800b410 tx=0x7fb54800b8d0 comp rx=0 tx=0).stop 2024-01-15T18:18:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.481+0000 7fb567000640 1 -- 172.21.15.27:0/3959545489 shutdown_connections 2024-01-15T18:18:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.481+0000 7fb567000640 1 --2- 172.21.15.27:0/3959545489 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb538067970 0x7fb538069e30 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.481+0000 7fb567000640 1 --2- 172.21.15.27:0/3959545489 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb5600cdf90 0x7fb5600c9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.481+0000 7fb567000640 1 --2- 172.21.15.27:0/3959545489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5600b2390 0x7fb5600b4960 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.481+0000 7fb567000640 1 --2- 172.21.15.27:0/3959545489 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb56006bb80 0x7fb56014c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.481+0000 7fb567000640 1 -- 172.21.15.27:0/3959545489 >> 172.21.15.27:0/3959545489 conn(0x7fb5600b2a10 msgr2=0x7fb5600b2df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.481+0000 7fb567000640 1 -- 172.21.15.27:0/3959545489 shutdown_connections 2024-01-15T18:18:25.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.481+0000 7fb567000640 1 -- 172.21.15.27:0/3959545489 wait complete. 2024-01-15T18:18:25.499 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:18:25.499 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:18:25.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.645+0000 7fc7ec000640 1 Processor -- start 2024-01-15T18:18:25.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.645+0000 7fc7ec000640 1 -- start start 2024-01-15T18:18:25.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.645+0000 7fc7ec000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc7e415b4b0 0x7fc7e40b1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:25.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.645+0000 7fc7ec000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7e40d19a0 con 0x7fc7e40cf460 2024-01-15T18:18:25.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.645+0000 7fc7ec000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7e40d1c80 con 0x7fc7e40cec60 2024-01-15T18:18:25.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.645+0000 7fc7ec000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7e40b07c0 con 0x7fc7e415b4b0 2024-01-15T18:18:25.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.645+0000 7fc7e9600640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc7e40cf460 0x7fc7e40cf840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46520/0 (socket says 172.21.15.27:46520) 2024-01-15T18:18:25.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.645+0000 7fc7e9600640 1 -- 172.21.15.27:0/2542908174 learned_addr learned my addr 172.21.15.27:0/2542908174 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:25.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.645+0000 7fc7e8c00640 1 --2- 172.21.15.27:0/2542908174 >> v2:172.21.15.27:3300/0 conn(0x7fc7e415b4b0 0x7fc7e40b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:25.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.645+0000 7fc7e3e00640 1 -- 172.21.15.27:0/2542908174 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3298798687 0 0) 0x7fc7e40d19a0 con 0x7fc7e40cf460 2024-01-15T18:18:25.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.645+0000 7fc7e3e00640 1 -- 172.21.15.27:0/2542908174 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7c4003530 con 0x7fc7e40cf460 2024-01-15T18:18:25.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.645+0000 7fc7e8c00640 1 -- 172.21.15.27:0/2542908174 >> v1:172.21.15.93:6789/0 conn(0x7fc7e40cec60 legacy=0x7fc7e40b1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:25.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.645+0000 7fc7e3e00640 1 -- 172.21.15.27:0/2542908174 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597635531 0 0) 0x7fc7e40d1c80 con 0x7fc7e40cec60 2024-01-15T18:18:25.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.645+0000 7fc7e8c00640 1 -- 172.21.15.27:0/2542908174 >> v1:172.21.15.27:6789/0 conn(0x7fc7e40cf460 legacy=0x7fc7e40cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:25.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.645+0000 7fc7e8c00640 1 -- 172.21.15.27:0/2542908174 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e40b0aa0 con 0x7fc7e415b4b0 2024-01-15T18:18:25.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.649+0000 7fc7e8c00640 1 --2- 172.21.15.27:0/2542908174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7e415b4b0 0x7fc7e40b1580 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc0097c0 tx=0x7fc7cc01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9891303665d672 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:25.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.649+0000 7fc7e3e00640 1 -- 172.21.15.27:0/2542908174 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc0ab680 con 0x7fc7e415b4b0 2024-01-15T18:18:25.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.649+0000 7fc7e3e00640 1 -- 172.21.15.27:0/2542908174 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7cc0ab7e0 con 0x7fc7e415b4b0 2024-01-15T18:18:25.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.649+0000 7fc7e3e00640 1 -- 172.21.15.27:0/2542908174 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc0abb70 con 0x7fc7e415b4b0 2024-01-15T18:18:25.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.649+0000 7fc7ec000640 1 -- 172.21.15.27:0/2542908174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7e415b4b0 msgr2=0x7fc7e40b1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:25.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.649+0000 7fc7ec000640 1 --2- 172.21.15.27:0/2542908174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7e415b4b0 0x7fc7e40b1580 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc0097c0 tx=0x7fc7cc01c480 comp rx=0 tx=0).stop 2024-01-15T18:18:25.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.649+0000 7fc7ec000640 1 -- 172.21.15.27:0/2542908174 shutdown_connections 2024-01-15T18:18:25.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.649+0000 7fc7ec000640 1 --2- 172.21.15.27:0/2542908174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7e415b4b0 0x7fc7e40b1580 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:25.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.649+0000 7fc7ec000640 1 -- 172.21.15.27:0/2542908174 >> 172.21.15.27:0/2542908174 conn(0x7fc7e40d0e80 msgr2=0x7fc7e40d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:25.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.649+0000 7fc7ec000640 1 -- 172.21.15.27:0/2542908174 shutdown_connections 2024-01-15T18:18:25.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.649+0000 7fc7ec000640 1 -- 172.21.15.27:0/2542908174 wait complete. 2024-01-15T18:18:25.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.653+0000 7fc7ec000640 1 Processor -- start 2024-01-15T18:18:25.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.653+0000 7fc7ec000640 1 -- start start 2024-01-15T18:18:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.653+0000 7fc7ec000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7e40cec60 0x7fc7e415a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.653+0000 7fc7ec000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7e40cf460 0x7fc7e414f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.653+0000 7fc7ec000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7e415b4b0 0x7fc7e414d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:25.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.653+0000 7fc7ec000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e41522e0 con 0x7fc7e40cf460 2024-01-15T18:18:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.653+0000 7fc7ec000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e414faf0 con 0x7fc7e415b4b0 2024-01-15T18:18:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.653+0000 7fc7ec000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e415bac0 con 0x7fc7e40cec60 2024-01-15T18:18:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.653+0000 7fc7e8c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7e40cf460 0x7fc7e414f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.653+0000 7fc7e8c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7e40cf460 0x7fc7e414f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58032/0 (socket says 172.21.15.27:58032) 2024-01-15T18:18:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.653+0000 7fc7e8c00640 1 -- 172.21.15.27:0/1704186724 learned_addr learned my addr 172.21.15.27:0/1704186724 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.653+0000 7fc7e9600640 1 --2- 172.21.15.27:0/1704186724 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7e40cec60 0x7fc7e415a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.653+0000 7fc7ea000640 1 --2- 172.21.15.27:0/1704186724 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7e415b4b0 0x7fc7e414d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.653+0000 7fc7e8c00640 1 -- 172.21.15.27:0/1704186724 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7e40cec60 msgr2=0x7fc7e415a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.653+0000 7fc7e8c00640 1 --2- 172.21.15.27:0/1704186724 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7e40cec60 0x7fc7e415a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.653+0000 7fc7e8c00640 1 -- 172.21.15.27:0/1704186724 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7e415b4b0 msgr2=0x7fc7e414d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.653+0000 7fc7e8c00640 1 --2- 172.21.15.27:0/1704186724 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7e415b4b0 0x7fc7e414d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.653+0000 7fc7e8c00640 1 -- 172.21.15.27:0/1704186724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7cc009420 con 0x7fc7e40cf460 2024-01-15T18:18:25.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.653+0000 7fc7e8c00640 1 --2- 172.21.15.27:0/1704186724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7e40cf460 0x7fc7e414f5b0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc0ad040 tx=0x7fc7cc01acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:25.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.653+0000 7fc7e1600640 1 -- 172.21.15.27:0/1704186724 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc0bf610 con 0x7fc7e40cf460 2024-01-15T18:18:25.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.653+0000 7fc7e1600640 1 -- 172.21.15.27:0/1704186724 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7cc0bf770 con 0x7fc7e40cf460 2024-01-15T18:18:25.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.653+0000 7fc7ec000640 1 -- 172.21.15.27:0/1704186724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7e4151a40 con 0x7fc7e40cf460 2024-01-15T18:18:25.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.653+0000 7fc7e1600640 1 -- 172.21.15.27:0/1704186724 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc0bfb00 con 0x7fc7e40cf460 2024-01-15T18:18:25.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.653+0000 7fc7ec000640 1 -- 172.21.15.27:0/1704186724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7e4152000 con 0x7fc7e40cf460 2024-01-15T18:18:25.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.653+0000 7fc7ec000640 1 -- 172.21.15.27:0/1704186724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7e415b890 con 0x7fc7e40cf460 2024-01-15T18:18:25.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.657+0000 7fc7e1600640 1 -- 172.21.15.27:0/1704186724 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7fc7cc0b5400 con 0x7fc7e40cf460 2024-01-15T18:18:25.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.657+0000 7fc7e1600640 1 --2- 172.21.15.27:0/1704186724 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc7bc067940 0x7fc7bc069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:25.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.657+0000 7fc7e1600640 1 -- 172.21.15.27:0/1704186724 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5055+0+0 (secure 0 0 0) 0x7fc7cc11de10 con 0x7fc7e40cf460 2024-01-15T18:18:25.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.657+0000 7fc7e9600640 1 --2- 172.21.15.27:0/1704186724 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc7bc067940 0x7fc7bc069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:25.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.657+0000 7fc7e9600640 1 --2- 172.21.15.27:0/1704186724 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc7bc067940 0x7fc7bc069e00 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4005890 tx=0x7fc7d4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:25.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.661+0000 7fc7e1600640 1 -- 172.21.15.27:0/1704186724 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc7e415b890 con 0x7fc7e40cf460 2024-01-15T18:18:25.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.821+0000 7fc7ec000640 1 -- 172.21.15.27:0/1704186724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc7e415b890 con 0x7fc7e40cf460 2024-01-15T18:18:25.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.821+0000 7fc7e1600640 1 -- 172.21.15.27:0/1704186724 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc7e415b890 con 0x7fc7e40cf460 2024-01-15T18:18:25.820 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:18:25.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.821+0000 7fc7ec000640 1 -- 172.21.15.27:0/1704186724 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc7bc067940 msgr2=0x7fc7bc069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:25.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.821+0000 7fc7ec000640 1 --2- 172.21.15.27:0/1704186724 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc7bc067940 0x7fc7bc069e00 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4005890 tx=0x7fc7d4009040 comp rx=0 tx=0).stop 2024-01-15T18:18:25.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.821+0000 7fc7ec000640 1 -- 172.21.15.27:0/1704186724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7e40cf460 msgr2=0x7fc7e414f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:25.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.821+0000 7fc7ec000640 1 --2- 172.21.15.27:0/1704186724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7e40cf460 0x7fc7e414f5b0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc0ad040 tx=0x7fc7cc01acc0 comp rx=0 tx=0).stop 2024-01-15T18:18:25.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.821+0000 7fc7ec000640 1 -- 172.21.15.27:0/1704186724 shutdown_connections 2024-01-15T18:18:25.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.821+0000 7fc7ec000640 1 --2- 172.21.15.27:0/1704186724 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc7bc067940 0x7fc7bc069e00 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:25.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.821+0000 7fc7ec000640 1 --2- 172.21.15.27:0/1704186724 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc7e415b4b0 0x7fc7e414d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:25.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.821+0000 7fc7ec000640 1 --2- 172.21.15.27:0/1704186724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc7e40cf460 0x7fc7e414f5b0 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:25.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.821+0000 7fc7ec000640 1 --2- 172.21.15.27:0/1704186724 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc7e40cec60 0x7fc7e415a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:25.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.821+0000 7fc7ec000640 1 -- 172.21.15.27:0/1704186724 >> 172.21.15.27:0/1704186724 conn(0x7fc7e40d0e80 msgr2=0x7fc7e415dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:25.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.821+0000 7fc7ec000640 1 -- 172.21.15.27:0/1704186724 shutdown_connections 2024-01-15T18:18:25.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.821+0000 7fc7ec000640 1 -- 172.21.15.27:0/1704186724 wait complete. 2024-01-15T18:18:25.843 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled' 2024-01-15T18:18:25.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.985+0000 7f2b4d000640 1 Processor -- start 2024-01-15T18:18:25.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.985+0000 7f2b4d000640 1 -- start start 2024-01-15T18:18:25.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.985+0000 7f2b4d000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2b4815d320 0x7f2b480d0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:25.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.985+0000 7f2b4d000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b480b2c90 con 0x7f2b480d34e0 2024-01-15T18:18:25.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.985+0000 7f2b4d000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b4815d700 con 0x7f2b480cdc30 2024-01-15T18:18:25.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.985+0000 7f2b4d000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b480b29e0 con 0x7f2b4815d320 2024-01-15T18:18:25.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.985+0000 7f2b46800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2b480d34e0 0x7f2b480b3120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46530/0 (socket says 172.21.15.27:46530) 2024-01-15T18:18:25.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.985+0000 7f2b46800640 1 -- 172.21.15.27:0/1969865678 learned_addr learned my addr 172.21.15.27:0/1969865678 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:25.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.985+0000 7f2b45e00640 1 --2- 172.21.15.27:0/1969865678 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2b4815d320 0x7f2b480d0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:25.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.985+0000 7f2b44a00640 1 -- 172.21.15.27:0/1969865678 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 600287522 0 0) 0x7f2b480b2c90 con 0x7f2b480d34e0 2024-01-15T18:18:25.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.985+0000 7f2b44a00640 1 -- 172.21.15.27:0/1969865678 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b28003530 con 0x7f2b480d34e0 2024-01-15T18:18:25.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.985+0000 7f2b44a00640 1 -- 172.21.15.27:0/1969865678 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 973762792 0 0) 0x7f2b28003530 con 0x7f2b480d34e0 2024-01-15T18:18:25.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.985+0000 7f2b45e00640 1 -- 172.21.15.27:0/1969865678 >> v1:172.21.15.93:6789/0 conn(0x7f2b480cdc30 legacy=0x7f2b480d2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:25.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.985+0000 7f2b45e00640 1 -- 172.21.15.27:0/1969865678 >> v1:172.21.15.27:6789/0 conn(0x7f2b480d34e0 legacy=0x7f2b480b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:25.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.985+0000 7f2b45e00640 1 -- 172.21.15.27:0/1969865678 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b480d2100 con 0x7f2b4815d320 2024-01-15T18:18:25.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.989+0000 7f2b45e00640 1 --2- 172.21.15.27:0/1969865678 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2b4815d320 0x7f2b480d0cf0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2b380097f0 tx=0x7f2b3801c200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5137bbc7ff34f9b8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:25.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.989+0000 7f2b44a00640 1 -- 172.21.15.27:0/1969865678 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b40001a00 con 0x7f2b4815d320 2024-01-15T18:18:25.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.989+0000 7f2b44a00640 1 -- 172.21.15.27:0/1969865678 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b380a7050 con 0x7f2b4815d320 2024-01-15T18:18:25.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.989+0000 7f2b44a00640 1 -- 172.21.15.27:0/1969865678 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b380ab620 con 0x7f2b4815d320 2024-01-15T18:18:25.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.989+0000 7f2b4d000640 1 -- 172.21.15.27:0/1969865678 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2b4815d320 msgr2=0x7f2b480d0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:25.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.989+0000 7f2b4d000640 1 --2- 172.21.15.27:0/1969865678 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2b4815d320 0x7f2b480d0cf0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2b380097f0 tx=0x7f2b3801c200 comp rx=0 tx=0).stop 2024-01-15T18:18:25.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.989+0000 7f2b4d000640 1 -- 172.21.15.27:0/1969865678 shutdown_connections 2024-01-15T18:18:25.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.989+0000 7f2b4d000640 1 --2- 172.21.15.27:0/1969865678 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2b4815d320 0x7f2b480d0cf0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:25.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.989+0000 7f2b4d000640 1 -- 172.21.15.27:0/1969865678 >> 172.21.15.27:0/1969865678 conn(0x7f2b4808cad0 msgr2=0x7f2b4809c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:25.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.989+0000 7f2b4d000640 1 -- 172.21.15.27:0/1969865678 shutdown_connections 2024-01-15T18:18:25.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.989+0000 7f2b4d000640 1 -- 172.21.15.27:0/1969865678 wait complete. 2024-01-15T18:18:25.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.993+0000 7f2b4d000640 1 Processor -- start 2024-01-15T18:18:25.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.993+0000 7f2b4d000640 1 -- start start 2024-01-15T18:18:25.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.993+0000 7f2b4d000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2b480cdc30 0x7f2b48150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:25.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.993+0000 7f2b4d000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2b480d34e0 0x7f2b4815b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:25.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.993+0000 7f2b4d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b4815d320 0x7f2b481562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:25.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.993+0000 7f2b4d000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4815bb40 con 0x7f2b4815d320 2024-01-15T18:18:25.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.993+0000 7f2b4d000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4815be20 con 0x7f2b480d34e0 2024-01-15T18:18:25.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.993+0000 7f2b4d000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4815c100 con 0x7f2b480cdc30 2024-01-15T18:18:25.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.993+0000 7f2b46800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b4815d320 0x7f2b481562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:25.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.993+0000 7f2b46800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b4815d320 0x7f2b481562c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58040/0 (socket says 172.21.15.27:58040) 2024-01-15T18:18:25.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.993+0000 7f2b46800640 1 -- 172.21.15.27:0/3806405424 learned_addr learned my addr 172.21.15.27:0/3806405424 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:25.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.993+0000 7f2b45e00640 1 --2- 172.21.15.27:0/3806405424 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2b480cdc30 0x7f2b48150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:25.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.993+0000 7f2b45400640 1 --2- 172.21.15.27:0/3806405424 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2b480d34e0 0x7f2b4815b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:25.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.993+0000 7f2b46800640 1 -- 172.21.15.27:0/3806405424 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2b480cdc30 msgr2=0x7f2b48150b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:25.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.993+0000 7f2b46800640 1 --2- 172.21.15.27:0/3806405424 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2b480cdc30 0x7f2b48150b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:25.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.993+0000 7f2b46800640 1 -- 172.21.15.27:0/3806405424 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2b480d34e0 msgr2=0x7f2b4815b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:25.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.993+0000 7f2b46800640 1 --2- 172.21.15.27:0/3806405424 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2b480d34e0 0x7f2b4815b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:25.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.993+0000 7f2b46800640 1 -- 172.21.15.27:0/3806405424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b38009420 con 0x7f2b4815d320 2024-01-15T18:18:25.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.993+0000 7f2b46800640 1 --2- 172.21.15.27:0/3806405424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b4815d320 0x7f2b481562c0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f2b40001570 tx=0x7f2b4000cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:25.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.993+0000 7f2b3e000640 1 -- 172.21.15.27:0/3806405424 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4000a840 con 0x7f2b4815d320 2024-01-15T18:18:25.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.993+0000 7f2b3e000640 1 -- 172.21.15.27:0/3806405424 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b4000a9c0 con 0x7f2b4815d320 2024-01-15T18:18:25.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.993+0000 7f2b4d000640 1 -- 172.21.15.27:0/3806405424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b4815c3e0 con 0x7f2b4815d320 2024-01-15T18:18:25.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.993+0000 7f2b3e000640 1 -- 172.21.15.27:0/3806405424 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4000adf0 con 0x7f2b4815d320 2024-01-15T18:18:25.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.993+0000 7f2b4d000640 1 -- 172.21.15.27:0/3806405424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b4815cbe0 con 0x7f2b4815d320 2024-01-15T18:18:25.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.997+0000 7f2b4d000640 1 -- 172.21.15.27:0/3806405424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b4815c900 con 0x7f2b4815d320 2024-01-15T18:18:25.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.997+0000 7f2b3e000640 1 -- 172.21.15.27:0/3806405424 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f2b400070d0 con 0x7f2b4815d320 2024-01-15T18:18:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.997+0000 7f2b3e000640 1 --2- 172.21.15.27:0/3806405424 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2b20067940 0x7f2b20069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.997+0000 7f2b3e000640 1 -- 172.21.15.27:0/3806405424 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f2b40082e20 con 0x7f2b4815d320 2024-01-15T18:18:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.997+0000 7f2b45e00640 1 --2- 172.21.15.27:0/3806405424 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2b20067940 0x7f2b20069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:25.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:25.997+0000 7f2b45e00640 1 --2- 172.21.15.27:0/3806405424 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2b20067940 0x7f2b20069e00 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f2b380ad000 tx=0x7f2b380a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:26.001+0000 7f2b3e000640 1 -- 172.21.15.27:0/3806405424 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2b4815c900 con 0x7f2b4815d320 2024-01-15T18:18:26.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:26.213+0000 7f2b4d000640 1 -- 172.21.15.27:0/3806405424 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled"]} v 0) v1 -- 0x7f2b4815c900 con 0x7f2b4815d320 2024-01-15T18:18:27.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.145+0000 7f2b3e000640 1 -- 172.21.15.27:0/3806405424 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled"]}]=0 v625) v1 ==== 158+0+0 (secure 0 0 0) 0x7f2b4815c900 con 0x7f2b4815d320 2024-01-15T18:18:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.149+0000 7f2b4d000640 1 -- 172.21.15.27:0/3806405424 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2b20067940 msgr2=0x7f2b20069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.149+0000 7f2b4d000640 1 --2- 172.21.15.27:0/3806405424 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2b20067940 0x7f2b20069e00 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f2b380ad000 tx=0x7f2b380a6000 comp rx=0 tx=0).stop 2024-01-15T18:18:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.149+0000 7f2b4d000640 1 -- 172.21.15.27:0/3806405424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b4815d320 msgr2=0x7f2b481562c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.149+0000 7f2b4d000640 1 --2- 172.21.15.27:0/3806405424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b4815d320 0x7f2b481562c0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f2b40001570 tx=0x7f2b4000cf00 comp rx=0 tx=0).stop 2024-01-15T18:18:27.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.149+0000 7f2b4d000640 1 -- 172.21.15.27:0/3806405424 shutdown_connections 2024-01-15T18:18:27.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.149+0000 7f2b4d000640 1 --2- 172.21.15.27:0/3806405424 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2b20067940 0x7f2b20069e00 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:27.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.149+0000 7f2b4d000640 1 --2- 172.21.15.27:0/3806405424 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b4815d320 0x7f2b481562c0 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:27.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.149+0000 7f2b4d000640 1 --2- 172.21.15.27:0/3806405424 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2b480d34e0 0x7f2b4815b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:27.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.149+0000 7f2b4d000640 1 --2- 172.21.15.27:0/3806405424 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2b480cdc30 0x7f2b48150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:27.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.149+0000 7f2b4d000640 1 -- 172.21.15.27:0/3806405424 >> 172.21.15.27:0/3806405424 conn(0x7f2b4808cad0 msgr2=0x7f2b480ce070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:27.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.149+0000 7f2b4d000640 1 -- 172.21.15.27:0/3806405424 shutdown_connections 2024-01-15T18:18:27.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.149+0000 7f2b4d000640 1 -- 172.21.15.27:0/3806405424 wait complete. 2024-01-15T18:18:27.174 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:18:27.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.321+0000 7f35cc400640 1 Processor -- start 2024-01-15T18:18:27.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.321+0000 7f35cc400640 1 -- start start 2024-01-15T18:18:27.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.325+0000 7f35cc400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35c415b100 0x7f35c4070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:27.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.325+0000 7f35cc400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c415a610 con 0x7f35c415ad20 2024-01-15T18:18:27.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.325+0000 7f35cc400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c415e3c0 con 0x7f35c4071420 2024-01-15T18:18:27.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.325+0000 7f35cc400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35c40718b0 con 0x7f35c415b100 2024-01-15T18:18:27.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.325+0000 7f35c9a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f35c415ad20 0x7f35c415d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46540/0 (socket says 172.21.15.27:46540) 2024-01-15T18:18:27.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.325+0000 7f35c9a00640 1 -- 172.21.15.27:0/487624413 learned_addr learned my addr 172.21.15.27:0/487624413 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:27.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.325+0000 7f35c9000640 1 --2- 172.21.15.27:0/487624413 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35c415b100 0x7f35c4070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:27.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.325+0000 7f35c3e00640 1 -- 172.21.15.27:0/487624413 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 889830218 0 0) 0x7f35c415a610 con 0x7f35c415ad20 2024-01-15T18:18:27.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.325+0000 7f35c3e00640 1 -- 172.21.15.27:0/487624413 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35a4003530 con 0x7f35c415ad20 2024-01-15T18:18:27.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.325+0000 7f35c3e00640 1 -- 172.21.15.27:0/487624413 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618943419 0 0) 0x7f35c415e3c0 con 0x7f35c4071420 2024-01-15T18:18:27.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.325+0000 7f35c3e00640 1 -- 172.21.15.27:0/487624413 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35c415a610 con 0x7f35c4071420 2024-01-15T18:18:27.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.325+0000 7f35c9000640 1 -- 172.21.15.27:0/487624413 >> v1:172.21.15.93:6789/0 conn(0x7f35c4071420 legacy=0x7f35c415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:27.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.325+0000 7f35c9000640 1 -- 172.21.15.27:0/487624413 >> v1:172.21.15.27:6789/0 conn(0x7f35c415ad20 legacy=0x7f35c415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:27.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.325+0000 7f35c9000640 1 -- 172.21.15.27:0/487624413 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c415e6a0 con 0x7f35c415b100 2024-01-15T18:18:27.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.329+0000 7f35c9000640 1 --2- 172.21.15.27:0/487624413 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35c415b100 0x7f35c4070ee0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f35ac0097f0 tx=0x7f35ac01c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18f88ce667f38cc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:27.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.329+0000 7f35c3e00640 1 -- 172.21.15.27:0/487624413 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ac0aa5f0 con 0x7f35c415b100 2024-01-15T18:18:27.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.329+0000 7f35c3e00640 1 -- 172.21.15.27:0/487624413 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35ac0aa750 con 0x7f35c415b100 2024-01-15T18:18:27.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.329+0000 7f35c3e00640 1 -- 172.21.15.27:0/487624413 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ac0aaae0 con 0x7f35c415b100 2024-01-15T18:18:27.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.329+0000 7f35cc400640 1 -- 172.21.15.27:0/487624413 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35c415b100 msgr2=0x7f35c4070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:27.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.329+0000 7f35cc400640 1 --2- 172.21.15.27:0/487624413 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35c415b100 0x7f35c4070ee0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f35ac0097f0 tx=0x7f35ac01c490 comp rx=0 tx=0).stop 2024-01-15T18:18:27.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.329+0000 7f35cc400640 1 -- 172.21.15.27:0/487624413 shutdown_connections 2024-01-15T18:18:27.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.329+0000 7f35cc400640 1 --2- 172.21.15.27:0/487624413 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35c415b100 0x7f35c4070ee0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:27.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.329+0000 7f35cc400640 1 -- 172.21.15.27:0/487624413 >> 172.21.15.27:0/487624413 conn(0x7f35c40cb700 msgr2=0x7f35c40cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:27.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.329+0000 7f35cc400640 1 -- 172.21.15.27:0/487624413 shutdown_connections 2024-01-15T18:18:27.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.329+0000 7f35cc400640 1 -- 172.21.15.27:0/487624413 wait complete. 2024-01-15T18:18:27.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.329+0000 7f35cc400640 1 Processor -- start 2024-01-15T18:18:27.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.329+0000 7f35cc400640 1 -- start start 2024-01-15T18:18:27.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.333+0000 7f35cc400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f35c4071420 0x7f35c414c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:27.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.333+0000 7f35cc400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35c415ad20 0x7f35c40ce3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:27.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.333+0000 7f35cc400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35c415b100 0x7f35c414bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:27.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.333+0000 7f35cc400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c40cead0 con 0x7f35c415b100 2024-01-15T18:18:27.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.333+0000 7f35cc400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c4150710 con 0x7f35c4071420 2024-01-15T18:18:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.333+0000 7f35cc400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35c414c120 con 0x7f35c415ad20 2024-01-15T18:18:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.333+0000 7f35c9000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35c415ad20 0x7f35c40ce3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.333+0000 7f35ca400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35c415b100 0x7f35c414bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.333+0000 7f35c9a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f35c4071420 0x7f35c414c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.333+0000 7f35ca400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35c415b100 0x7f35c414bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58044/0 (socket says 172.21.15.27:58044) 2024-01-15T18:18:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.333+0000 7f35ca400640 1 -- 172.21.15.27:0/4020432473 learned_addr learned my addr 172.21.15.27:0/4020432473 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.333+0000 7f35ca400640 1 -- 172.21.15.27:0/4020432473 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35c415ad20 msgr2=0x7f35c40ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.333+0000 7f35ca400640 1 --2- 172.21.15.27:0/4020432473 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35c415ad20 0x7f35c40ce3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.333+0000 7f35ca400640 1 -- 172.21.15.27:0/4020432473 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f35c4071420 msgr2=0x7f35c414c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.333+0000 7f35ca400640 1 --2- 172.21.15.27:0/4020432473 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f35c4071420 0x7f35c414c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.333+0000 7f35ca400640 1 -- 172.21.15.27:0/4020432473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ac009420 con 0x7f35c415b100 2024-01-15T18:18:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.333+0000 7f35ca400640 1 --2- 172.21.15.27:0/4020432473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35c415b100 0x7f35c414bbe0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f35bc00c730 tx=0x7f35bc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.333+0000 7f35c1600640 1 -- 172.21.15.27:0/4020432473 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35bc003b80 con 0x7f35c415b100 2024-01-15T18:18:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.333+0000 7f35c1600640 1 -- 172.21.15.27:0/4020432473 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35bc003ce0 con 0x7f35c415b100 2024-01-15T18:18:27.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.333+0000 7f35cc400640 1 -- 172.21.15.27:0/4020432473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c40ccf50 con 0x7f35c415b100 2024-01-15T18:18:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.333+0000 7f35c1600640 1 -- 172.21.15.27:0/4020432473 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35bc00aa90 con 0x7f35c415b100 2024-01-15T18:18:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.333+0000 7f35cc400640 1 -- 172.21.15.27:0/4020432473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c40ccc70 con 0x7f35c415b100 2024-01-15T18:18:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.333+0000 7f35cc400640 1 -- 172.21.15.27:0/4020432473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c41504e0 con 0x7f35c415b100 2024-01-15T18:18:27.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.337+0000 7f35c1600640 1 -- 172.21.15.27:0/4020432473 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f35bc00abf0 con 0x7f35c415b100 2024-01-15T18:18:27.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.337+0000 7f35c1600640 1 --2- 172.21.15.27:0/4020432473 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f359c067940 0x7f359c069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:27.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.337+0000 7f35c1600640 1 -- 172.21.15.27:0/4020432473 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f35bc082fa0 con 0x7f35c415b100 2024-01-15T18:18:27.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.337+0000 7f35c9a00640 1 --2- 172.21.15.27:0/4020432473 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f359c067940 0x7f359c069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:27.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.337+0000 7f35c9a00640 1 --2- 172.21.15.27:0/4020432473 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f359c067940 0x7f359c069e00 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f35b4007fd0 tx=0x7f35b4007d60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:27.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.341+0000 7f35c1600640 1 -- 172.21.15.27:0/4020432473 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f35c41504e0 con 0x7f35c415b100 2024-01-15T18:18:27.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.497+0000 7f35cc400640 1 -- 172.21.15.27:0/4020432473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f35c41504e0 con 0x7f35c415b100 2024-01-15T18:18:27.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.501+0000 7f35c1600640 1 -- 172.21.15.27:0/4020432473 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7f35c41504e0 con 0x7f35c415b100 2024-01-15T18:18:27.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.501+0000 7f35cc400640 1 -- 172.21.15.27:0/4020432473 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f359c067940 msgr2=0x7f359c069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:27.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.501+0000 7f35cc400640 1 --2- 172.21.15.27:0/4020432473 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f359c067940 0x7f359c069e00 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f35b4007fd0 tx=0x7f35b4007d60 comp rx=0 tx=0).stop 2024-01-15T18:18:27.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.501+0000 7f35cc400640 1 -- 172.21.15.27:0/4020432473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35c415b100 msgr2=0x7f35c414bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:27.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.501+0000 7f35cc400640 1 --2- 172.21.15.27:0/4020432473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35c415b100 0x7f35c414bbe0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f35bc00c730 tx=0x7f35bc00cbf0 comp rx=0 tx=0).stop 2024-01-15T18:18:27.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.501+0000 7f35cc400640 1 -- 172.21.15.27:0/4020432473 shutdown_connections 2024-01-15T18:18:27.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.501+0000 7f35cc400640 1 --2- 172.21.15.27:0/4020432473 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f359c067940 0x7f359c069e00 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:27.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.501+0000 7f35cc400640 1 --2- 172.21.15.27:0/4020432473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35c415b100 0x7f35c414bbe0 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:27.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.501+0000 7f35cc400640 1 --2- 172.21.15.27:0/4020432473 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f35c415ad20 0x7f35c40ce3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:27.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.501+0000 7f35cc400640 1 --2- 172.21.15.27:0/4020432473 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f35c4071420 0x7f35c414c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:27.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.501+0000 7f35cc400640 1 -- 172.21.15.27:0/4020432473 >> 172.21.15.27:0/4020432473 conn(0x7f35c40cb700 msgr2=0x7f35c406dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:27.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.501+0000 7f35cc400640 1 -- 172.21.15.27:0/4020432473 shutdown_connections 2024-01-15T18:18:27.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.501+0000 7f35cc400640 1 -- 172.21.15.27:0/4020432473 wait complete. 2024-01-15T18:18:27.524 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T18:18:27.524 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:18:27.598 DEBUG:teuthology.orchestra.run.smithi093:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:18:27.669 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T18:18:27.670 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:18:27.670 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:18:27.679 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354580k,nr_inodes=4088645,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9249 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:18:27.679 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:18:27.679 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:18:27.689 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354484k,nr_inodes=4088621,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8403 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:18:27.689 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:18:27.689 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:18:27.730 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354580k,nr_inodes=4088645,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9249 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:18:27.731 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:18:27.731 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:18:27.739 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354484k,nr_inodes=4088621,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8403 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:18:27.739 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-15T18:18:27.740 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:18:27.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.917+0000 7f772ea00640 1 Processor -- start 2024-01-15T18:18:27.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.917+0000 7f772ea00640 1 -- start start 2024-01-15T18:18:27.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.917+0000 7f772ea00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f77280afe70 0x7f77280cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:27.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.917+0000 7f772ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f77280cb7b0 0x7f77280cbc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:27.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.917+0000 7f772ea00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77280d05b0 con 0x7f7728160b10 2024-01-15T18:18:27.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.917+0000 7f772ea00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77280ce7d0 con 0x7f77280cb7b0 2024-01-15T18:18:27.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.917+0000 7f772ea00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77280cfc80 con 0x7f77280afe70 2024-01-15T18:18:27.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.917+0000 7f772ca00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f77280cb7b0 0x7f77280cbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:27.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.917+0000 7f7727e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7728160b10 0x7f77280cd7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46554/0 (socket says 172.21.15.27:46554) 2024-01-15T18:18:27.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.917+0000 7f7727e00640 1 -- 172.21.15.27:0/3839247360 learned_addr learned my addr 172.21.15.27:0/3839247360 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:27.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.917+0000 7f7727400640 1 --2- 172.21.15.27:0/3839247360 >> v2:172.21.15.93:3300/0 conn(0x7f77280afe70 0x7f77280cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:27.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.917+0000 7f772ca00640 1 -- 172.21.15.27:0/3839247360 >> v2:172.21.15.93:3300/0 conn(0x7f77280afe70 msgr2=0x7f77280cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:27.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.917+0000 7f772ca00640 1 --2- 172.21.15.27:0/3839247360 >> v2:172.21.15.93:3300/0 conn(0x7f77280afe70 0x7f77280cdeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:27.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.917+0000 7f772ca00640 1 -- 172.21.15.27:0/3839247360 >> v1:172.21.15.27:6789/0 conn(0x7f7728160b10 legacy=0x7f77280cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:27.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.917+0000 7f772ca00640 1 -- 172.21.15.27:0/3839247360 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77280cff60 con 0x7f77280cb7b0 2024-01-15T18:18:27.919 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:18:27.922 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:18:27.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.921+0000 7f772ca00640 1 --2- 172.21.15.27:0/3839247360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77280cb7b0 0x7f77280cbc00 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f7728067d20 tx=0x7f772001c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0f5aea2df64a6d3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:27.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.921+0000 7f7726a00640 1 -- 172.21.15.27:0/3839247360 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77200a5820 con 0x7f77280cb7b0 2024-01-15T18:18:27.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.921+0000 7f7726a00640 1 -- 172.21.15.27:0/3839247360 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77200a5980 con 0x7f77280cb7b0 2024-01-15T18:18:27.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.921+0000 7f7726a00640 1 -- 172.21.15.27:0/3839247360 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77200a5d10 con 0x7f77280cb7b0 2024-01-15T18:18:27.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.921+0000 7f772ea00640 1 -- 172.21.15.27:0/3839247360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77280cb7b0 msgr2=0x7f77280cbc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:27.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.921+0000 7f772ea00640 1 --2- 172.21.15.27:0/3839247360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77280cb7b0 0x7f77280cbc00 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f7728067d20 tx=0x7f772001c850 comp rx=0 tx=0).stop 2024-01-15T18:18:27.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.921+0000 7f772ea00640 1 -- 172.21.15.27:0/3839247360 shutdown_connections 2024-01-15T18:18:27.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.921+0000 7f772ea00640 1 --2- 172.21.15.27:0/3839247360 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77280cb7b0 0x7f77280cbc00 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:27.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.921+0000 7f772ea00640 1 --2- 172.21.15.27:0/3839247360 >> v2:172.21.15.93:3300/0 conn(0x7f77280afe70 0x7f77280cdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:27.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.921+0000 7f772ea00640 1 -- 172.21.15.27:0/3839247360 >> 172.21.15.27:0/3839247360 conn(0x7f77280d0f20 msgr2=0x7f77280d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:27.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.921+0000 7f772ea00640 1 -- 172.21.15.27:0/3839247360 shutdown_connections 2024-01-15T18:18:27.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.921+0000 7f772ea00640 1 -- 172.21.15.27:0/3839247360 wait complete. 2024-01-15T18:18:27.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.925+0000 7f772ea00640 1 Processor -- start 2024-01-15T18:18:27.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.925+0000 7f772ea00640 1 -- start start 2024-01-15T18:18:27.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.925+0000 7f772ea00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f77280afe70 0x7f772814dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:27.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.925+0000 7f772ea00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f77280cb7b0 0x7f77280c89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:27.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.925+0000 7f772ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7728160b10 0x7f7728151340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:27.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.925+0000 7f772ea00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772814f660 con 0x7f7728160b10 2024-01-15T18:18:27.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.925+0000 7f772ea00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77280c9980 con 0x7f77280afe70 2024-01-15T18:18:27.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.925+0000 7f772ea00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77280c93c0 con 0x7f77280cb7b0 2024-01-15T18:18:27.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.925+0000 7f772ca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7728160b10 0x7f7728151340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:27.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.925+0000 7f772ca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7728160b10 0x7f7728151340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58052/0 (socket says 172.21.15.27:58052) 2024-01-15T18:18:27.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.925+0000 7f772ca00640 1 -- 172.21.15.27:0/2469592167 learned_addr learned my addr 172.21.15.27:0/2469592167 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:27.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.925+0000 7f7727400640 1 --2- 172.21.15.27:0/2469592167 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f77280cb7b0 0x7f77280c89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:27.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.925+0000 7f7727e00640 1 --2- 172.21.15.27:0/2469592167 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f77280afe70 0x7f772814dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:27.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.925+0000 7f772ca00640 1 -- 172.21.15.27:0/2469592167 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f77280cb7b0 msgr2=0x7f77280c89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:27.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.925+0000 7f772ca00640 1 --2- 172.21.15.27:0/2469592167 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f77280cb7b0 0x7f77280c89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:27.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.925+0000 7f772ca00640 1 -- 172.21.15.27:0/2469592167 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f77280afe70 msgr2=0x7f772814dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:27.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.925+0000 7f772ca00640 1 --2- 172.21.15.27:0/2469592167 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f77280afe70 0x7f772814dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:27.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.925+0000 7f772ca00640 1 -- 172.21.15.27:0/2469592167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7720009a70 con 0x7f7728160b10 2024-01-15T18:18:27.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.925+0000 7f772ca00640 1 --2- 172.21.15.27:0/2469592167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7728160b10 0x7f7728151340 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f7720019460 tx=0x7f77200195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:27.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.925+0000 7f771fe00640 1 -- 172.21.15.27:0/2469592167 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77200b9860 con 0x7f7728160b10 2024-01-15T18:18:27.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.925+0000 7f771fe00640 1 -- 172.21.15.27:0/2469592167 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77200b99c0 con 0x7f7728160b10 2024-01-15T18:18:27.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.925+0000 7f771fe00640 1 -- 172.21.15.27:0/2469592167 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77200b9d70 con 0x7f7728160b10 2024-01-15T18:18:27.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.925+0000 7f772ea00640 1 -- 172.21.15.27:0/2469592167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77280c96a0 con 0x7f7728160b10 2024-01-15T18:18:27.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.925+0000 7f772ea00640 1 -- 172.21.15.27:0/2469592167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f772814e190 con 0x7f7728160b10 2024-01-15T18:18:27.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.929+0000 7f772ea00640 1 -- 172.21.15.27:0/2469592167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f772814f4c0 con 0x7f7728160b10 2024-01-15T18:18:27.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.929+0000 7f771fe00640 1 -- 172.21.15.27:0/2469592167 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f77200ad440 con 0x7f7728160b10 2024-01-15T18:18:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.933+0000 7f771fe00640 1 --2- 172.21.15.27:0/2469592167 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7700067990 0x7f7700069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.933+0000 7f771fe00640 1 -- 172.21.15.27:0/2469592167 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f772011e570 con 0x7f7728160b10 2024-01-15T18:18:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.933+0000 7f7727e00640 1 --2- 172.21.15.27:0/2469592167 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7700067990 0x7f7700069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:27.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.933+0000 7f7727e00640 1 --2- 172.21.15.27:0/2469592167 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7700067990 0x7f7700069e50 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f7718005890 tx=0x7f7718009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:27.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:27.937+0000 7f771fe00640 1 -- 172.21.15.27:0/2469592167 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f772814f4c0 con 0x7f7728160b10 2024-01-15T18:18:28.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.125+0000 7f772ea00640 1 -- 172.21.15.27:0/2469592167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f772814fb40 con 0x7f7728160b10 2024-01-15T18:18:28.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.125+0000 7f771fe00640 1 -- 172.21.15.27:0/2469592167 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f77200b2e90 con 0x7f7728160b10 2024-01-15T18:18:28.127 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:18:28.127 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":70,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6837/1905230324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":1905230324},{"type":"v1","addr":"172.21.15.93:6837","nonce":1905230324}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":8269,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/2926806399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":2926806399},{"type":"v1","addr":"172.21.15.93:6836","nonce":2926806399}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":8934,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1107490361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1107490361},{"type":"v1","addr":"172.21.15.27:6834","nonce":1107490361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":9327,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/1927211782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":1927211782},{"type":"v1","addr":"172.21.15.27:6835","nonce":1927211782}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[]} 2024-01-15T18:18:28.127 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 70 2024-01-15T18:18:28.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.129+0000 7f772ea00640 1 -- 172.21.15.27:0/2469592167 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7700067990 msgr2=0x7f7700069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:28.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.129+0000 7f772ea00640 1 --2- 172.21.15.27:0/2469592167 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7700067990 0x7f7700069e50 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f7718005890 tx=0x7f7718009040 comp rx=0 tx=0).stop 2024-01-15T18:18:28.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.129+0000 7f772ea00640 1 -- 172.21.15.27:0/2469592167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7728160b10 msgr2=0x7f7728151340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:28.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.129+0000 7f772ea00640 1 --2- 172.21.15.27:0/2469592167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7728160b10 0x7f7728151340 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f7720019460 tx=0x7f77200195f0 comp rx=0 tx=0).stop 2024-01-15T18:18:28.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.129+0000 7f772ea00640 1 -- 172.21.15.27:0/2469592167 shutdown_connections 2024-01-15T18:18:28.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.129+0000 7f772ea00640 1 --2- 172.21.15.27:0/2469592167 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7700067990 0x7f7700069e50 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:28.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.129+0000 7f772ea00640 1 --2- 172.21.15.27:0/2469592167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7728160b10 0x7f7728151340 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:28.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.129+0000 7f772ea00640 1 --2- 172.21.15.27:0/2469592167 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f77280cb7b0 0x7f77280c89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:28.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.129+0000 7f772ea00640 1 --2- 172.21.15.27:0/2469592167 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f77280afe70 0x7f772814dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:28.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.129+0000 7f772ea00640 1 -- 172.21.15.27:0/2469592167 >> 172.21.15.27:0/2469592167 conn(0x7f77280d0f20 msgr2=0x7f772806fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:28.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.129+0000 7f772ea00640 1 -- 172.21.15.27:0/2469592167 shutdown_connections 2024-01-15T18:18:28.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.129+0000 7f772ea00640 1 -- 172.21.15.27:0/2469592167 wait complete. 2024-01-15T18:18:28.151 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-15T18:18:28.152 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-15T18:18:28.152 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-15T18:18:28.152 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:18:28.152 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-15T18:18:28.152 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-15T18:18:28.153 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-15T18:18:28.153 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:18:28.153 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-15T18:18:28.153 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-15T18:18:28.153 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-15T18:18:28.153 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:18:28.153 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-15T18:18:28.153 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-15T18:18:28.154 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-15T18:18:28.154 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:18:28.165 INFO:tasks.ceph.mds.c:Stopped 2024-01-15T18:18:28.165 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-15T18:18:28.166 INFO:tasks.ceph.mds.a:Stopped 2024-01-15T18:18:28.166 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-15T18:18:28.169 INFO:tasks.ceph.mds.b:Stopped 2024-01-15T18:18:28.169 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-15T18:18:28.170 INFO:tasks.ceph.mds.d:Stopped 2024-01-15T18:18:28.170 DEBUG:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-15T18:18:28.209 INFO:tasks.ceph.mds.c:Started 2024-01-15T18:18:28.210 DEBUG:teuthology.parallel:result is None 2024-01-15T18:18:28.211 INFO:tasks.ceph.mds.a:Started 2024-01-15T18:18:28.211 DEBUG:teuthology.parallel:result is None 2024-01-15T18:18:28.214 INFO:tasks.ceph.mds.b:Started 2024-01-15T18:18:28.214 DEBUG:teuthology.parallel:result is None 2024-01-15T18:18:28.216 INFO:tasks.ceph.mds.d:Started 2024-01-15T18:18:28.216 DEBUG:teuthology.parallel:result is None 2024-01-15T18:18:28.216 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-15T18:18:28.298 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2024-01-15T18:18:28.299 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2024-01-15T18:18:28.306 INFO:tasks.ceph.mds.b.smithi093.stdout:starting mds.b at 2024-01-15T18:18:28.307 INFO:tasks.ceph.mds.d.smithi093.stdout:starting mds.d at 2024-01-15T18:18:28.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.365+0000 7fdd9f600640 1 Processor -- start 2024-01-15T18:18:28.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.365+0000 7fdd9f600640 1 -- start start 2024-01-15T18:18:28.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.365+0000 7fdd9f600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdd980cce90 0x7fdd980cd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:28.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.365+0000 7fdd9f600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fdd9806b810 0x7fdd980cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:28.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.365+0000 7fdd9f600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd980d0310 0x7fdd9815cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:28.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.365+0000 7fdd9f600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd980cf550 con 0x7fdd980cce90 2024-01-15T18:18:28.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.365+0000 7fdd9f600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd980cf830 con 0x7fdd9806b810 2024-01-15T18:18:28.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.365+0000 7fdd9f600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd980cfb10 con 0x7fdd980d0310 2024-01-15T18:18:28.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.365+0000 7fdd9cc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdd980cce90 0x7fdd980cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:28.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.365+0000 7fdd9cc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdd980cce90 0x7fdd980cd710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58076/0 (socket says 172.21.15.27:58076) 2024-01-15T18:18:28.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.365+0000 7fdd9cc00640 1 -- 172.21.15.27:0/467412431 learned_addr learned my addr 172.21.15.27:0/467412431 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:28.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.365+0000 7fdd97e00640 1 --2- 172.21.15.27:0/467412431 >> v2:172.21.15.93:3300/0 conn(0x7fdd9806b810 0x7fdd980cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:28.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.365+0000 7fdd9d600640 1 --2- 172.21.15.27:0/467412431 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd980d0310 0x7fdd9815cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:28.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.365+0000 7fdd9cc00640 1 -- 172.21.15.27:0/467412431 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd980d0310 msgr2=0x7fdd9815cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:28.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.365+0000 7fdd9cc00640 1 --2- 172.21.15.27:0/467412431 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd980d0310 0x7fdd9815cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:28.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.365+0000 7fdd9cc00640 1 -- 172.21.15.27:0/467412431 >> v2:172.21.15.93:3300/0 conn(0x7fdd9806b810 msgr2=0x7fdd980cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:28.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.365+0000 7fdd9cc00640 1 --2- 172.21.15.27:0/467412431 >> v2:172.21.15.93:3300/0 conn(0x7fdd9806b810 0x7fdd980cdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:28.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.365+0000 7fdd9cc00640 1 -- 172.21.15.27:0/467412431 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd980cfdf0 con 0x7fdd980cce90 2024-01-15T18:18:28.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.369+0000 7fdd9cc00640 1 --2- 172.21.15.27:0/467412431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd980cce90 0x7fdd980cd710 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fdd9809aa50 tx=0x7fdd8801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bb1fae794d8a3b8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:28.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.369+0000 7fdd97400640 1 -- 172.21.15.27:0/467412431 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd880aa640 con 0x7fdd980cce90 2024-01-15T18:18:28.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.369+0000 7fdd97400640 1 -- 172.21.15.27:0/467412431 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd880aa7a0 con 0x7fdd980cce90 2024-01-15T18:18:28.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.369+0000 7fdd97400640 1 -- 172.21.15.27:0/467412431 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd880aab30 con 0x7fdd980cce90 2024-01-15T18:18:28.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.369+0000 7fdd9f600640 1 -- 172.21.15.27:0/467412431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd980cce90 msgr2=0x7fdd980cd710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:28.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.369+0000 7fdd9f600640 1 --2- 172.21.15.27:0/467412431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd980cce90 0x7fdd980cd710 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fdd9809aa50 tx=0x7fdd8801c500 comp rx=0 tx=0).stop 2024-01-15T18:18:28.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.369+0000 7fdd9f600640 1 -- 172.21.15.27:0/467412431 shutdown_connections 2024-01-15T18:18:28.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.369+0000 7fdd9f600640 1 --2- 172.21.15.27:0/467412431 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd980d0310 0x7fdd9815cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:28.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.369+0000 7fdd9f600640 1 --2- 172.21.15.27:0/467412431 >> v2:172.21.15.93:3300/0 conn(0x7fdd9806b810 0x7fdd980cdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:28.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.369+0000 7fdd9f600640 1 --2- 172.21.15.27:0/467412431 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd980cce90 0x7fdd980cd710 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:28.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.369+0000 7fdd9f600640 1 -- 172.21.15.27:0/467412431 >> 172.21.15.27:0/467412431 conn(0x7fdd980d0e90 msgr2=0x7fdd980d12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:28.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.369+0000 7fdd9f600640 1 -- 172.21.15.27:0/467412431 shutdown_connections 2024-01-15T18:18:28.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.369+0000 7fdd9f600640 1 -- 172.21.15.27:0/467412431 wait complete. 2024-01-15T18:18:28.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.373+0000 7fdd9f600640 1 Processor -- start 2024-01-15T18:18:28.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.373+0000 7fdd9f600640 1 -- start start 2024-01-15T18:18:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.373+0000 7fdd9f600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd980cce90 0x7fdd98155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.373+0000 7fdd9f600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd980d0310 0x7fdd9814abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.373+0000 7fdd9f600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd98157060 0x7fdd98152830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.373+0000 7fdd9f600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd981513e0 con 0x7fdd980cce90 2024-01-15T18:18:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.373+0000 7fdd9f600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd98155d90 con 0x7fdd98157060 2024-01-15T18:18:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.373+0000 7fdd9f600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd98156bd0 con 0x7fdd980d0310 2024-01-15T18:18:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.373+0000 7fdd9cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd980cce90 0x7fdd98155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.373+0000 7fdd9cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd980cce90 0x7fdd98155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58090/0 (socket says 172.21.15.27:58090) 2024-01-15T18:18:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.373+0000 7fdd9cc00640 1 -- 172.21.15.27:0/2078045883 learned_addr learned my addr 172.21.15.27:0/2078045883 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:28.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.373+0000 7fdd97e00640 1 --2- 172.21.15.27:0/2078045883 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd980d0310 0x7fdd9814abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:28.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.373+0000 7fdd9d600640 1 --2- 172.21.15.27:0/2078045883 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd98157060 0x7fdd98152830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:28.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.373+0000 7fdd9cc00640 1 -- 172.21.15.27:0/2078045883 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd980d0310 msgr2=0x7fdd9814abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:28.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.373+0000 7fdd9cc00640 1 --2- 172.21.15.27:0/2078045883 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd980d0310 0x7fdd9814abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:28.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.373+0000 7fdd9cc00640 1 -- 172.21.15.27:0/2078045883 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd98157060 msgr2=0x7fdd98152830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:28.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.373+0000 7fdd9cc00640 1 --2- 172.21.15.27:0/2078045883 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd98157060 0x7fdd98152830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:28.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.373+0000 7fdd9cc00640 1 -- 172.21.15.27:0/2078045883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd880094d0 con 0x7fdd980cce90 2024-01-15T18:18:28.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.373+0000 7fdd9cc00640 1 --2- 172.21.15.27:0/2078045883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd980cce90 0x7fdd98155850 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fdd8801c9e0 tx=0x7fdd8801ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:28.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.373+0000 7fdd94c00640 1 -- 172.21.15.27:0/2078045883 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd880bd3f0 con 0x7fdd980cce90 2024-01-15T18:18:28.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.373+0000 7fdd94c00640 1 -- 172.21.15.27:0/2078045883 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd880bd570 con 0x7fdd980cce90 2024-01-15T18:18:28.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.373+0000 7fdd94c00640 1 -- 172.21.15.27:0/2078045883 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd880bd920 con 0x7fdd980cce90 2024-01-15T18:18:28.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.373+0000 7fdd9f600640 1 -- 172.21.15.27:0/2078045883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd9814b0f0 con 0x7fdd980cce90 2024-01-15T18:18:28.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.373+0000 7fdd9f600640 1 -- 172.21.15.27:0/2078045883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd981562a0 con 0x7fdd980cce90 2024-01-15T18:18:28.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.373+0000 7fdd9f600640 1 -- 172.21.15.27:0/2078045883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd980ce750 con 0x7fdd980cce90 2024-01-15T18:18:28.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.377+0000 7fdd94c00640 1 -- 172.21.15.27:0/2078045883 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7fdd880a8050 con 0x7fdd980cce90 2024-01-15T18:18:28.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.377+0000 7fdd94c00640 1 --2- 172.21.15.27:0/2078045883 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd70067940 0x7fdd70069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.377+0000 7fdd94c00640 1 -- 172.21.15.27:0/2078045883 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5055+0+0 (secure 0 0 0) 0x7fdd8811d110 con 0x7fdd980cce90 2024-01-15T18:18:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.377+0000 7fdd97e00640 1 --2- 172.21.15.27:0/2078045883 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd70067940 0x7fdd70069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.377+0000 7fdd97e00640 1 --2- 172.21.15.27:0/2078045883 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd70067940 0x7fdd70069e00 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fdd981f9780 tx=0x7fdd80009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.381+0000 7fdd94c00640 1 -- 172.21.15.27:0/2078045883 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fdd880ebc50 con 0x7fdd980cce90 2024-01-15T18:18:28.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:28.549+0000 7fdd9f600640 1 -- 172.21.15.27:0/2078045883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fdd9804e850 con 0x7fdd980cce90 2024-01-15T18:18:29.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.169+0000 7fdd94c00640 1 -- 172.21.15.27:0/2078045883 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v102) v1 ==== 97+0+0 (secure 0 0 0) 0x7fdd880c3080 con 0x7fdd980cce90 2024-01-15T18:18:29.170 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2024-01-15T18:18:29.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.173+0000 7fdd9f600640 1 -- 172.21.15.27:0/2078045883 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd70067940 msgr2=0x7fdd70069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:29.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.173+0000 7fdd9f600640 1 --2- 172.21.15.27:0/2078045883 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd70067940 0x7fdd70069e00 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fdd981f9780 tx=0x7fdd80009190 comp rx=0 tx=0).stop 2024-01-15T18:18:29.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.173+0000 7fdd9f600640 1 -- 172.21.15.27:0/2078045883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd980cce90 msgr2=0x7fdd98155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:29.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.173+0000 7fdd9f600640 1 --2- 172.21.15.27:0/2078045883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd980cce90 0x7fdd98155850 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fdd8801c9e0 tx=0x7fdd8801ad40 comp rx=0 tx=0).stop 2024-01-15T18:18:29.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.173+0000 7fdd9f600640 1 -- 172.21.15.27:0/2078045883 shutdown_connections 2024-01-15T18:18:29.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.173+0000 7fdd9f600640 1 --2- 172.21.15.27:0/2078045883 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fdd70067940 0x7fdd70069e00 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.173+0000 7fdd9f600640 1 --2- 172.21.15.27:0/2078045883 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fdd98157060 0x7fdd98152830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.173+0000 7fdd9f600640 1 --2- 172.21.15.27:0/2078045883 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fdd980d0310 0x7fdd9814abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.173+0000 7fdd9f600640 1 --2- 172.21.15.27:0/2078045883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd980cce90 0x7fdd98155850 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.173+0000 7fdd9f600640 1 -- 172.21.15.27:0/2078045883 >> 172.21.15.27:0/2078045883 conn(0x7fdd980d0e90 msgr2=0x7fdd9815f870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:29.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.173+0000 7fdd9f600640 1 -- 172.21.15.27:0/2078045883 shutdown_connections 2024-01-15T18:18:29.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.173+0000 7fdd9f600640 1 -- 172.21.15.27:0/2078045883 wait complete. 2024-01-15T18:18:29.197 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-15T18:18:29.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.341+0000 7f9a24600640 1 Processor -- start 2024-01-15T18:18:29.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.341+0000 7f9a24600640 1 -- start start 2024-01-15T18:18:29.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.341+0000 7f9a24600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9a1c160b10 0x7f9a1c0afe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:29.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.341+0000 7f9a24600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f9a1c0cd7a0 0x7f9a1c0cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:29.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.341+0000 7f9a24600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9a1c0ce160 0x7f9a1c06b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.341+0000 7f9a24600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a1c0ce5f0 con 0x7f9a1c0ce160 2024-01-15T18:18:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.341+0000 7f9a24600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a1c0d05b0 con 0x7f9a1c0cd7a0 2024-01-15T18:18:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.341+0000 7f9a24600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c0ce7d0 con 0x7f9a1c160b10 2024-01-15T18:18:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.341+0000 7f9a22600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9a1c0ce160 0x7f9a1c06b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.341+0000 7f9a22600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9a1c0ce160 0x7f9a1c06b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58094/0 (socket says 172.21.15.27:58094) 2024-01-15T18:18:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.341+0000 7f9a22600640 1 -- 172.21.15.27:0/3406185191 learned_addr learned my addr 172.21.15.27:0/3406185191 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.341+0000 7f9a21c00640 1 --2- 172.21.15.27:0/3406185191 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9a1c160b10 0x7f9a1c0afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.341+0000 7f9a22600640 1 -- 172.21.15.27:0/3406185191 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9a1c160b10 msgr2=0x7f9a1c0afe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.341+0000 7f9a22600640 1 --2- 172.21.15.27:0/3406185191 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9a1c160b10 0x7f9a1c0afe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.341+0000 7f9a22600640 1 -- 172.21.15.27:0/3406185191 >> v2:172.21.15.93:3300/0 conn(0x7f9a1c0cd7a0 msgr2=0x7f9a1c0cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.341+0000 7f9a22600640 1 --2- 172.21.15.27:0/3406185191 >> v2:172.21.15.93:3300/0 conn(0x7f9a1c0cd7a0 0x7f9a1c0cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.341+0000 7f9a22600640 1 -- 172.21.15.27:0/3406185191 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1c0cfc80 con 0x7f9a1c0ce160 2024-01-15T18:18:29.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.341+0000 7f9a21c00640 1 --2- 172.21.15.27:0/3406185191 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9a1c160b10 0x7f9a1c0afe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:18:29.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.345+0000 7f9a22600640 1 --2- 172.21.15.27:0/3406185191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a1c0ce160 0x7f9a1c06b600 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c071de0 tx=0x7f9a1401c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ff94a668f22fbb2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:29.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.345+0000 7f9a1be00640 1 -- 172.21.15.27:0/3406185191 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a140a64e0 con 0x7f9a1c0ce160 2024-01-15T18:18:29.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.345+0000 7f9a1be00640 1 -- 172.21.15.27:0/3406185191 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a140a6640 con 0x7f9a1c0ce160 2024-01-15T18:18:29.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.345+0000 7f9a1be00640 1 -- 172.21.15.27:0/3406185191 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a140a69d0 con 0x7f9a1c0ce160 2024-01-15T18:18:29.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.345+0000 7f9a24600640 1 -- 172.21.15.27:0/3406185191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a1c0ce160 msgr2=0x7f9a1c06b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:29.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.345+0000 7f9a24600640 1 --2- 172.21.15.27:0/3406185191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a1c0ce160 0x7f9a1c06b600 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c071de0 tx=0x7f9a1401c650 comp rx=0 tx=0).stop 2024-01-15T18:18:29.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.345+0000 7f9a24600640 1 -- 172.21.15.27:0/3406185191 shutdown_connections 2024-01-15T18:18:29.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.345+0000 7f9a24600640 1 --2- 172.21.15.27:0/3406185191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a1c0ce160 0x7f9a1c06b600 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.345+0000 7f9a24600640 1 --2- 172.21.15.27:0/3406185191 >> v2:172.21.15.93:3300/0 conn(0x7f9a1c0cd7a0 0x7f9a1c0cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.345+0000 7f9a24600640 1 --2- 172.21.15.27:0/3406185191 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9a1c160b10 0x7f9a1c0afe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.345+0000 7f9a24600640 1 -- 172.21.15.27:0/3406185191 >> 172.21.15.27:0/3406185191 conn(0x7f9a1c0d0f20 msgr2=0x7f9a1c0d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:29.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.345+0000 7f9a24600640 1 -- 172.21.15.27:0/3406185191 shutdown_connections 2024-01-15T18:18:29.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.345+0000 7f9a24600640 1 -- 172.21.15.27:0/3406185191 wait complete. 2024-01-15T18:18:29.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.349+0000 7f9a24600640 1 Processor -- start 2024-01-15T18:18:29.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.349+0000 7f9a24600640 1 -- start start 2024-01-15T18:18:29.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.349+0000 7f9a24600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a1c0cd7a0 0x7f9a1c0c6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:29.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.349+0000 7f9a24600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9a1c0ce160 0x7f9a1c13b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:29.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.349+0000 7f9a24600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9a1c160b10 0x7f9a1c14c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:29.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.349+0000 7f9a24600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c0c9550 con 0x7f9a1c0cd7a0 2024-01-15T18:18:29.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.349+0000 7f9a24600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c14d8e0 con 0x7f9a1c160b10 2024-01-15T18:18:29.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.349+0000 7f9a24600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1c0c7340 con 0x7f9a1c0ce160 2024-01-15T18:18:29.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.349+0000 7f9a21c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a1c0cd7a0 0x7f9a1c0c6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:29.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.349+0000 7f9a21c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a1c0cd7a0 0x7f9a1c0c6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58098/0 (socket says 172.21.15.27:58098) 2024-01-15T18:18:29.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.349+0000 7f9a21c00640 1 -- 172.21.15.27:0/3032823595 learned_addr learned my addr 172.21.15.27:0/3032823595 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:29.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.349+0000 7f9a22600640 1 --2- 172.21.15.27:0/3032823595 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9a1c160b10 0x7f9a1c14c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:29.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.349+0000 7f9a21200640 1 --2- 172.21.15.27:0/3032823595 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9a1c0ce160 0x7f9a1c13b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:29.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.349+0000 7f9a21c00640 1 -- 172.21.15.27:0/3032823595 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9a1c0ce160 msgr2=0x7f9a1c13b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:29.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.349+0000 7f9a21c00640 1 --2- 172.21.15.27:0/3032823595 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9a1c0ce160 0x7f9a1c13b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.349+0000 7f9a21c00640 1 -- 172.21.15.27:0/3032823595 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9a1c160b10 msgr2=0x7f9a1c14c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:29.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.349+0000 7f9a21c00640 1 --2- 172.21.15.27:0/3032823595 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9a1c160b10 0x7f9a1c14c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.349+0000 7f9a21c00640 1 -- 172.21.15.27:0/3032823595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a14009b30 con 0x7f9a1c0cd7a0 2024-01-15T18:18:29.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.349+0000 7f9a21c00640 1 --2- 172.21.15.27:0/3032823595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a1c0cd7a0 0x7f9a1c0c6d60 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c00c7f0 tx=0x7f9a0c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:29.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.349+0000 7f9a19600640 1 -- 172.21.15.27:0/3032823595 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c007820 con 0x7f9a1c0cd7a0 2024-01-15T18:18:29.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.349+0000 7f9a19600640 1 -- 172.21.15.27:0/3032823595 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a0c007980 con 0x7f9a1c0cd7a0 2024-01-15T18:18:29.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.349+0000 7f9a19600640 1 -- 172.21.15.27:0/3032823595 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c007d30 con 0x7f9a1c0cd7a0 2024-01-15T18:18:29.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.349+0000 7f9a24600640 1 -- 172.21.15.27:0/3032823595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a1c0c7620 con 0x7f9a1c0cd7a0 2024-01-15T18:18:29.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.349+0000 7f9a24600640 1 -- 172.21.15.27:0/3032823595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a1c13b6e0 con 0x7f9a1c0cd7a0 2024-01-15T18:18:29.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.353+0000 7f9a24600640 1 -- 172.21.15.27:0/3032823595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a1c0c7860 con 0x7f9a1c0cd7a0 2024-01-15T18:18:29.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.353+0000 7f9a19600640 1 -- 172.21.15.27:0/3032823595 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f9a0c024480 con 0x7f9a1c0cd7a0 2024-01-15T18:18:29.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.353+0000 7f9a19600640 1 --2- 172.21.15.27:0/3032823595 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f99f4067940 0x7f99f4069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:29.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.353+0000 7f9a19600640 1 -- 172.21.15.27:0/3032823595 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4969+0+0 (secure 0 0 0) 0x7f9a0c083830 con 0x7f9a1c0cd7a0 2024-01-15T18:18:29.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.353+0000 7f9a21200640 1 --2- 172.21.15.27:0/3032823595 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f99f4067940 0x7f99f4069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:29.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.353+0000 7f9a21200640 1 --2- 172.21.15.27:0/3032823595 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f99f4067940 0x7f99f4069e00 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c0c6040 tx=0x7f9a04009110 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:29.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.357+0000 7f9a19600640 1 -- 172.21.15.27:0/3032823595 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9a0c052250 con 0x7f9a1c0cd7a0 2024-01-15T18:18:29.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.569+0000 7f9a24600640 1 -- 172.21.15.27:0/3032823595 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f9a1c0c9040 con 0x7f9a1c0cd7a0 2024-01-15T18:18:29.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.569+0000 7f9a19600640 1 -- 172.21.15.27:0/3032823595 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f9a1c0c9040 con 0x7f9a1c0cd7a0 2024-01-15T18:18:29.571 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:18:29.571 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:18:29.571 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2024-01-15T18:18:29.571 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:29.571 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2024-01-15T18:18:29.571 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlbMZIGBAAhSU4y2DGJd3MH0v6v3LsDg==", 2024-01-15T18:18:29.571 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:18:29.571 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:18:29.571 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:18:29.571 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:18:29.571 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:18:29.571 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:29.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:29.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:29.572 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2024-01-15T18:18:29.572 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlC7B9JBAAsXmBjV4KwPZejOmowXO2Zg==", 2024-01-15T18:18:29.572 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:18:29.572 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:18:29.572 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:18:29.572 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:18:29.572 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:18:29.572 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:29.572 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:29.572 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:29.572 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2024-01-15T18:18:29.573 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6Vla6a3HxAA/AHc6FSTccj0JGwiR4Vl7g==", 2024-01-15T18:18:29.573 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:18:29.573 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:18:29.573 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:18:29.573 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:18:29.573 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:18:29.573 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:29.573 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:29.573 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:29.573 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2024-01-15T18:18:29.573 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6Vl9nqzKxAAx2aDA8UwidAUrU9DBxGldA==", 2024-01-15T18:18:29.573 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:18:29.573 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:18:29.574 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:18:29.574 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:18:29.574 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:18:29.574 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:29.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:29.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:29.574 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2024-01-15T18:18:29.574 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCSc6Vly4TACRAAYBmMIcsZsRp4CZ5a3xCWgA==", 2024-01-15T18:18:29.574 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:18:29.574 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:18:29.574 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:18:29.574 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:18:29.574 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:29.574 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:29.574 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:29.574 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2024-01-15T18:18:29.575 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCVc6VlW/HEERAAOBgPz+eGpHwNXDe8fkYo7w==", 2024-01-15T18:18:29.575 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:18:29.575 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:18:29.575 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:18:29.575 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:18:29.575 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:29.575 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:29.575 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:29.575 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2024-01-15T18:18:29.575 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCYc6Vl+Ec4GBAAWRe/6u7u4RXApd57H8ihng==", 2024-01-15T18:18:29.575 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:18:29.575 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:18:29.575 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:18:29.575 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:18:29.575 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:29.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:29.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:29.576 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2024-01-15T18:18:29.576 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCbc6VlNdMPHxAAHzJvzLltsTl1wrEnI/iyrQ==", 2024-01-15T18:18:29.576 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:18:29.576 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:18:29.576 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:18:29.576 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:18:29.576 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:29.576 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:29.576 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:29.576 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2024-01-15T18:18:29.576 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQChc6Vlg8eVKBAAwZWAncCE2Lra+CxmSIxZkA==", 2024-01-15T18:18:29.576 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:18:29.576 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:18:29.577 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:18:29.577 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:18:29.577 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:29.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:29.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:29.577 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2024-01-15T18:18:29.577 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCkc6VlIXDOLxAAFISn6ILqq/aNFK07RxNnaw==", 2024-01-15T18:18:29.577 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:18:29.577 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:18:29.577 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:18:29.577 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:18:29.577 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:29.577 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:29.577 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:29.577 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2024-01-15T18:18:29.577 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCnc6Vl9Ru2NRAAl996VBDudw23SGiFPusIcw==", 2024-01-15T18:18:29.577 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:18:29.578 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:18:29.578 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:18:29.578 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:18:29.578 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:29.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:29.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:29.578 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2024-01-15T18:18:29.578 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCrc6VlHoAsARAAIoPMridLsIcpU3I9Hr5XLQ==", 2024-01-15T18:18:29.578 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:18:29.578 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2024-01-15T18:18:29.578 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2024-01-15T18:18:29.578 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:18:29.578 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:29.578 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:29.578 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:29.579 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2024-01-15T18:18:29.579 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlH+DCLxAA1sKVut+3WjA/B+aYJ6/Jeg==", 2024-01-15T18:18:29.579 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:18:29.579 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:18:29.579 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow r", 2024-01-15T18:18:29.579 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-15T18:18:29.579 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:29.579 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:29.579 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:29.579 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2024-01-15T18:18:29.579 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlaRBaORAAijn3UJLHbWm8ol2NiKNSaw==", 2024-01-15T18:18:29.579 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:18:29.579 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:18:29.579 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-15T18:18:29.579 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-15T18:18:29.580 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-15T18:18:29.580 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:29.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:29.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:29.580 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2024-01-15T18:18:29.580 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlDbLQNRAA82Lwz+pKY+3nHPAr1DichA==", 2024-01-15T18:18:29.580 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:18:29.580 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:18:29.580 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-15T18:18:29.580 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-15T18:18:29.580 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-15T18:18:29.580 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:29.580 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:29.580 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:29.581 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2024-01-15T18:18:29.581 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCPc6VlOyDaAxAADea0K1OEDTfrzsM7VEkixg==", 2024-01-15T18:18:29.581 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:18:29.581 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2024-01-15T18:18:29.581 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2024-01-15T18:18:29.581 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2024-01-15T18:18:29.581 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2024-01-15T18:18:29.581 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:29.581 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:29.581 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:29.581 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2024-01-15T18:18:29.581 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCNc6VlK70BNxAAgYXkW/pMlzy8GZoZhX9NMA==", 2024-01-15T18:18:29.581 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:18:29.581 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-15T18:18:29.581 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2024-01-15T18:18:29.581 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2024-01-15T18:18:29.582 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:18:29.582 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:29.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:29.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:29.582 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2024-01-15T18:18:29.582 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlQbinExAAWtUdgkUNR3cCKV+ytcbUuQ==", 2024-01-15T18:18:29.582 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:18:29.582 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-15T18:18:29.582 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-15T18:18:29.582 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:18:29.582 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:29.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:29.582 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:29.583 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2024-01-15T18:18:29.583 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOc6VlzafrDxAAVVEBuX8BDT3cB9NfD2xTdQ==", 2024-01-15T18:18:29.583 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2024-01-15T18:18:29.583 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2024-01-15T18:18:29.583 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2024-01-15T18:18:29.583 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2024-01-15T18:18:29.583 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:29.583 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:29.583 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-01-15T18:18:29.583 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:18:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.573+0000 7f9a24600640 1 -- 172.21.15.27:0/3032823595 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f99f4067940 msgr2=0x7f99f4069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.573+0000 7f9a24600640 1 --2- 172.21.15.27:0/3032823595 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f99f4067940 0x7f99f4069e00 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c0c6040 tx=0x7f9a04009110 comp rx=0 tx=0).stop 2024-01-15T18:18:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.573+0000 7f9a24600640 1 -- 172.21.15.27:0/3032823595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a1c0cd7a0 msgr2=0x7f9a1c0c6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.573+0000 7f9a24600640 1 --2- 172.21.15.27:0/3032823595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a1c0cd7a0 0x7f9a1c0c6d60 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c00c7f0 tx=0x7f9a0c00ccb0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.573+0000 7f9a24600640 1 -- 172.21.15.27:0/3032823595 shutdown_connections 2024-01-15T18:18:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.573+0000 7f9a24600640 1 --2- 172.21.15.27:0/3032823595 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f99f4067940 0x7f99f4069e00 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.573+0000 7f9a24600640 1 --2- 172.21.15.27:0/3032823595 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9a1c160b10 0x7f9a1c14c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.573+0000 7f9a24600640 1 --2- 172.21.15.27:0/3032823595 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9a1c0ce160 0x7f9a1c13b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.573+0000 7f9a24600640 1 --2- 172.21.15.27:0/3032823595 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a1c0cd7a0 0x7f9a1c0c6d60 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.573+0000 7f9a24600640 1 -- 172.21.15.27:0/3032823595 >> 172.21.15.27:0/3032823595 conn(0x7f9a1c0d0f20 msgr2=0x7f9a1c1574d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:29.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.573+0000 7f9a24600640 1 -- 172.21.15.27:0/3032823595 shutdown_connections 2024-01-15T18:18:29.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.573+0000 7f9a24600640 1 -- 172.21.15.27:0/3032823595 wait complete. 2024-01-15T18:18:29.594 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:18:29.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.733+0000 7f6a0aa00640 1 Processor -- start 2024-01-15T18:18:29.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.737+0000 7f6a0aa00640 1 -- start start 2024-01-15T18:18:29.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.737+0000 7f6a0aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6a0415c130 0x7f6a0406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:29.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.737+0000 7f6a0aa00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0406c4a0 con 0x7f6a040701a0 2024-01-15T18:18:29.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.737+0000 7f6a0aa00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0415c660 con 0x7f6a0406e240 2024-01-15T18:18:29.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.737+0000 7f6a0aa00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a0415c940 con 0x7f6a0415c130 2024-01-15T18:18:29.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.737+0000 7f6a03e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6a040701a0 0x7f6a04070580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46572/0 (socket says 172.21.15.27:46572) 2024-01-15T18:18:29.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.737+0000 7f6a03e00640 1 -- 172.21.15.27:0/1015682174 learned_addr learned my addr 172.21.15.27:0/1015682174 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:29.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.737+0000 7f6a03400640 1 --2- 172.21.15.27:0/1015682174 >> v2:172.21.15.27:3300/0 conn(0x7f6a0415c130 0x7f6a0406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:29.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.737+0000 7f6a02a00640 1 -- 172.21.15.27:0/1015682174 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266367323 0 0) 0x7f6a0406c4a0 con 0x7f6a040701a0 2024-01-15T18:18:29.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.737+0000 7f6a02a00640 1 -- 172.21.15.27:0/1015682174 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69e4003530 con 0x7f6a040701a0 2024-01-15T18:18:29.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.737+0000 7f6a02a00640 1 -- 172.21.15.27:0/1015682174 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2237465643 0 0) 0x7f6a0415c660 con 0x7f6a0406e240 2024-01-15T18:18:29.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.737+0000 7f6a03400640 1 -- 172.21.15.27:0/1015682174 >> v1:172.21.15.93:6789/0 conn(0x7f6a0406e240 legacy=0x7f6a0415d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:29.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.737+0000 7f6a03400640 1 -- 172.21.15.27:0/1015682174 >> v1:172.21.15.27:6789/0 conn(0x7f6a040701a0 legacy=0x7f6a04070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:29.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.737+0000 7f6a03400640 1 -- 172.21.15.27:0/1015682174 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a0415cc20 con 0x7f6a0415c130 2024-01-15T18:18:29.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.741+0000 7f6a03400640 1 --2- 172.21.15.27:0/1015682174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a0415c130 0x7f6a0406dd00 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f69ec0097c0 tx=0x7f69ec01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=267e5d924a696cfc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:29.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.741+0000 7f6a02a00640 1 -- 172.21.15.27:0/1015682174 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ec0ab650 con 0x7f6a0415c130 2024-01-15T18:18:29.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.741+0000 7f6a02a00640 1 -- 172.21.15.27:0/1015682174 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ec0ab7b0 con 0x7f6a0415c130 2024-01-15T18:18:29.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.741+0000 7f6a02a00640 1 -- 172.21.15.27:0/1015682174 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ec0abb40 con 0x7f6a0415c130 2024-01-15T18:18:29.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.741+0000 7f6a0aa00640 1 -- 172.21.15.27:0/1015682174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a0415c130 msgr2=0x7f6a0406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:29.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.741+0000 7f6a0aa00640 1 --2- 172.21.15.27:0/1015682174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a0415c130 0x7f6a0406dd00 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f69ec0097c0 tx=0x7f69ec01c450 comp rx=0 tx=0).stop 2024-01-15T18:18:29.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.741+0000 7f6a0aa00640 1 -- 172.21.15.27:0/1015682174 shutdown_connections 2024-01-15T18:18:29.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.741+0000 7f6a0aa00640 1 --2- 172.21.15.27:0/1015682174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a0415c130 0x7f6a0406dd00 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.741+0000 7f6a0aa00640 1 -- 172.21.15.27:0/1015682174 >> 172.21.15.27:0/1015682174 conn(0x7f6a040cec60 msgr2=0x7f6a040cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:29.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.741+0000 7f6a0aa00640 1 -- 172.21.15.27:0/1015682174 shutdown_connections 2024-01-15T18:18:29.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.741+0000 7f6a0aa00640 1 -- 172.21.15.27:0/1015682174 wait complete. 2024-01-15T18:18:29.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f6a0aa00640 1 Processor -- start 2024-01-15T18:18:29.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f6a0aa00640 1 -- start start 2024-01-15T18:18:29.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f6a0aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a0406e240 0x7f6a04157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:29.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f6a0aa00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6a040701a0 0x7f6a040c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:29.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f6a0aa00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6a0415c130 0x7f6a0414f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:29.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f6a0aa00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a040c8ac0 con 0x7f6a0406e240 2024-01-15T18:18:29.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f6a0aa00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0415a500 con 0x7f6a0415c130 2024-01-15T18:18:29.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f6a0aa00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a041506b0 con 0x7f6a040701a0 2024-01-15T18:18:29.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f6a03e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a0406e240 0x7f6a04157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:29.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f6a03e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a0406e240 0x7f6a04157420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58112/0 (socket says 172.21.15.27:58112) 2024-01-15T18:18:29.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f6a03e00640 1 -- 172.21.15.27:0/4005554567 learned_addr learned my addr 172.21.15.27:0/4005554567 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:29.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f6a03400640 1 --2- 172.21.15.27:0/4005554567 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6a040701a0 0x7f6a040c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:29.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f6a08a00640 1 --2- 172.21.15.27:0/4005554567 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6a0415c130 0x7f6a0414f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:29.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f6a03e00640 1 -- 172.21.15.27:0/4005554567 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6a040701a0 msgr2=0x7f6a040c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:29.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f6a03e00640 1 --2- 172.21.15.27:0/4005554567 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6a040701a0 0x7f6a040c8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f6a03e00640 1 -- 172.21.15.27:0/4005554567 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6a0415c130 msgr2=0x7f6a0414f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:29.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f6a03e00640 1 --2- 172.21.15.27:0/4005554567 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6a0415c130 0x7f6a0414f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f6a03e00640 1 -- 172.21.15.27:0/4005554567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ec009420 con 0x7f6a0406e240 2024-01-15T18:18:29.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f6a03400640 1 --2- 172.21.15.27:0/4005554567 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6a040701a0 0x7f6a040c8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:18:29.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f6a03e00640 1 --2- 172.21.15.27:0/4005554567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a0406e240 0x7f6a04157420 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f69f400c450 tx=0x7f69f400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:29.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f69fbe00640 1 -- 172.21.15.27:0/4005554567 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f4005cf0 con 0x7f6a0406e240 2024-01-15T18:18:29.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f69fbe00640 1 -- 172.21.15.27:0/4005554567 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f4005e50 con 0x7f6a0406e240 2024-01-15T18:18:29.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f6a0aa00640 1 -- 172.21.15.27:0/4005554567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a04150990 con 0x7f6a0406e240 2024-01-15T18:18:29.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f69fbe00640 1 -- 172.21.15.27:0/4005554567 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f4007290 con 0x7f6a0406e240 2024-01-15T18:18:29.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f6a0aa00640 1 -- 172.21.15.27:0/4005554567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a04150f50 con 0x7f6a0406e240 2024-01-15T18:18:29.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.745+0000 7f6a0aa00640 1 -- 172.21.15.27:0/4005554567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a0404e850 con 0x7f6a0406e240 2024-01-15T18:18:29.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.749+0000 7f69fbe00640 1 -- 172.21.15.27:0/4005554567 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f69f40073f0 con 0x7f6a0406e240 2024-01-15T18:18:29.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.749+0000 7f69fbe00640 1 --2- 172.21.15.27:0/4005554567 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f69dc067970 0x7f69dc069e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:29.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.749+0000 7f69fbe00640 1 -- 172.21.15.27:0/4005554567 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4969+0+0 (secure 0 0 0) 0x7f69f40827a0 con 0x7f6a0406e240 2024-01-15T18:18:29.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.749+0000 7f6a03400640 1 --2- 172.21.15.27:0/4005554567 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f69dc067970 0x7f69dc069e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:29.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.749+0000 7f6a03400640 1 --2- 172.21.15.27:0/4005554567 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f69dc067970 0x7f69dc069e30 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f69ec0af000 tx=0x7f69ec0a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:29.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.753+0000 7f69fbe00640 1 -- 172.21.15.27:0/4005554567 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f69f40511c0 con 0x7f6a0406e240 2024-01-15T18:18:29.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.909+0000 7f6a0aa00640 1 -- 172.21.15.27:0/4005554567 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a04070ec0 con 0x7f6a0406e240 2024-01-15T18:18:29.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.909+0000 7f69fbe00640 1 -- 172.21.15.27:0/4005554567 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+12940 (secure 0 0 0) 0x7f69f4017070 con 0x7f6a0406e240 2024-01-15T18:18:29.911 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:18:29.911 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":104,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:18:29.188773+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:18:29.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.913+0000 7f6a0aa00640 1 -- 172.21.15.27:0/4005554567 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f69dc067970 msgr2=0x7f69dc069e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:29.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.913+0000 7f6a0aa00640 1 --2- 172.21.15.27:0/4005554567 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f69dc067970 0x7f69dc069e30 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f69ec0af000 tx=0x7f69ec0a7000 comp rx=0 tx=0).stop 2024-01-15T18:18:29.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.913+0000 7f6a0aa00640 1 -- 172.21.15.27:0/4005554567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a0406e240 msgr2=0x7f6a04157420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:29.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.913+0000 7f6a0aa00640 1 --2- 172.21.15.27:0/4005554567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a0406e240 0x7f6a04157420 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f69f400c450 tx=0x7f69f400c910 comp rx=0 tx=0).stop 2024-01-15T18:18:29.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.913+0000 7f6a0aa00640 1 -- 172.21.15.27:0/4005554567 shutdown_connections 2024-01-15T18:18:29.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.913+0000 7f6a0aa00640 1 --2- 172.21.15.27:0/4005554567 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f69dc067970 0x7f69dc069e30 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.913+0000 7f6a0aa00640 1 --2- 172.21.15.27:0/4005554567 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6a0415c130 0x7f6a0414f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.913+0000 7f6a0aa00640 1 --2- 172.21.15.27:0/4005554567 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6a040701a0 0x7f6a040c8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.913+0000 7f6a0aa00640 1 --2- 172.21.15.27:0/4005554567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a0406e240 0x7f6a04157420 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.913+0000 7f6a0aa00640 1 -- 172.21.15.27:0/4005554567 >> 172.21.15.27:0/4005554567 conn(0x7f6a040cec60 msgr2=0x7f6a040c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:29.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.913+0000 7f6a0aa00640 1 -- 172.21.15.27:0/4005554567 shutdown_connections 2024-01-15T18:18:29.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:29.913+0000 7f6a0aa00640 1 -- 172.21.15.27:0/4005554567 wait complete. 2024-01-15T18:18:29.933 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:18:29.933 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:18:30.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.077+0000 7f4b9d400640 1 Processor -- start 2024-01-15T18:18:30.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.077+0000 7f4b9d400640 1 -- start start 2024-01-15T18:18:30.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.077+0000 7f4b9d400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4b98158900 0x7f4b9815f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:30.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.077+0000 7f4b9d400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f4b980cf2e0 0x7f4b980701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:30.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.077+0000 7f4b9d400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b980b1cb0 con 0x7f4b9815bf00 2024-01-15T18:18:30.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.077+0000 7f4b9d400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b980b1f00 con 0x7f4b980cf2e0 2024-01-15T18:18:30.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.077+0000 7f4b9d400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b980b21e0 con 0x7f4b98158900 2024-01-15T18:18:30.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.077+0000 7f4b97200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f4b980cf2e0 0x7f4b980701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.077+0000 7f4b96800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4b98158900 0x7f4b9815f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.077+0000 7f4b96800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4b98158900 0x7f4b9815f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:59180/0 (socket says 172.21.15.27:59180) 2024-01-15T18:18:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.077+0000 7f4b96800640 1 -- 172.21.15.27:0/1395424871 learned_addr learned my addr 172.21.15.27:0/1395424871 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.077+0000 7f4b97200640 1 -- 172.21.15.27:0/1395424871 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4b98158900 msgr2=0x7f4b9815f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.077+0000 7f4b97200640 1 --2- 172.21.15.27:0/1395424871 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4b98158900 0x7f4b9815f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.077+0000 7f4b97200640 1 -- 172.21.15.27:0/1395424871 >> v1:172.21.15.93:6789/0 conn(0x7f4b9815bf00 legacy=0x7f4b980cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.077+0000 7f4b95400640 1 -- 172.21.15.27:0/1395424871 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046157821 0 0) 0x7f4b980b1cb0 con 0x7f4b9815bf00 2024-01-15T18:18:30.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.077+0000 7f4b97200640 1 -- 172.21.15.27:0/1395424871 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b9815ea70 con 0x7f4b980cf2e0 2024-01-15T18:18:30.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.081+0000 7f4b97200640 1 --2- 172.21.15.27:0/1395424871 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4b980cf2e0 0x7f4b980701e0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f4b90009d90 tx=0x7f4b9001c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19cfceb04cd66ca6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:30.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.081+0000 7f4b95400640 1 -- 172.21.15.27:0/1395424871 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b98070f30 con 0x7f4b980cf2e0 2024-01-15T18:18:30.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.081+0000 7f4b95400640 1 -- 172.21.15.27:0/1395424871 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b900a5900 con 0x7f4b980cf2e0 2024-01-15T18:18:30.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.081+0000 7f4b95400640 1 -- 172.21.15.27:0/1395424871 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b900a5c90 con 0x7f4b980cf2e0 2024-01-15T18:18:30.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.081+0000 7f4b9d400640 1 -- 172.21.15.27:0/1395424871 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4b980cf2e0 msgr2=0x7f4b980701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:30.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.081+0000 7f4b9d400640 1 --2- 172.21.15.27:0/1395424871 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4b980cf2e0 0x7f4b980701e0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f4b90009d90 tx=0x7f4b9001c950 comp rx=0 tx=0).stop 2024-01-15T18:18:30.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.081+0000 7f4b9d400640 1 -- 172.21.15.27:0/1395424871 shutdown_connections 2024-01-15T18:18:30.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.081+0000 7f4b9d400640 1 --2- 172.21.15.27:0/1395424871 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4b980cf2e0 0x7f4b980701e0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:30.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.081+0000 7f4b9d400640 1 --2- 172.21.15.27:0/1395424871 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4b98158900 0x7f4b9815f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:30.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.081+0000 7f4b9d400640 1 -- 172.21.15.27:0/1395424871 >> 172.21.15.27:0/1395424871 conn(0x7f4b980cb700 msgr2=0x7f4b980cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:30.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.085+0000 7f4b9d400640 1 -- 172.21.15.27:0/1395424871 shutdown_connections 2024-01-15T18:18:30.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.085+0000 7f4b9d400640 1 -- 172.21.15.27:0/1395424871 wait complete. 2024-01-15T18:18:30.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.085+0000 7f4b9d400640 1 Processor -- start 2024-01-15T18:18:30.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.085+0000 7f4b9d400640 1 -- start start 2024-01-15T18:18:30.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.085+0000 7f4b9d400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4b980cf2e0 0x7f4b9814c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:30.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.085+0000 7f4b9d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b98158900 0x7f4b98151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:30.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.085+0000 7f4b9d400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4b9815bf00 0x7f4b9815da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:30.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.085+0000 7f4b9d400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b980cc810 con 0x7f4b98158900 2024-01-15T18:18:30.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.085+0000 7f4b9d400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b980ccaf0 con 0x7f4b9815bf00 2024-01-15T18:18:30.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.085+0000 7f4b9d400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b980ccdd0 con 0x7f4b980cf2e0 2024-01-15T18:18:30.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.085+0000 7f4b95e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b98158900 0x7f4b98151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:30.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.085+0000 7f4b95e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b98158900 0x7f4b98151940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58116/0 (socket says 172.21.15.27:58116) 2024-01-15T18:18:30.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.085+0000 7f4b95e00640 1 -- 172.21.15.27:0/1540359421 learned_addr learned my addr 172.21.15.27:0/1540359421 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:30.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.085+0000 7f4b97200640 1 --2- 172.21.15.27:0/1540359421 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4b9815bf00 0x7f4b9815da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:30.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.085+0000 7f4b96800640 1 --2- 172.21.15.27:0/1540359421 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4b980cf2e0 0x7f4b9814c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:30.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.085+0000 7f4b95e00640 1 -- 172.21.15.27:0/1540359421 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4b980cf2e0 msgr2=0x7f4b9814c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:30.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.085+0000 7f4b95e00640 1 --2- 172.21.15.27:0/1540359421 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4b980cf2e0 0x7f4b9814c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:30.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.085+0000 7f4b95e00640 1 -- 172.21.15.27:0/1540359421 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4b9815bf00 msgr2=0x7f4b9815da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:30.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.085+0000 7f4b95e00640 1 --2- 172.21.15.27:0/1540359421 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4b9815bf00 0x7f4b9815da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:30.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.085+0000 7f4b95e00640 1 -- 172.21.15.27:0/1540359421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b90009a70 con 0x7f4b98158900 2024-01-15T18:18:30.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.085+0000 7f4b97200640 1 --2- 172.21.15.27:0/1540359421 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4b9815bf00 0x7f4b9815da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:18:30.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.085+0000 7f4b95e00640 1 --2- 172.21.15.27:0/1540359421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b98158900 0x7f4b98151940 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f4b80001ec0 tx=0x7f4b8000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:30.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.089+0000 7f4b8ea00640 1 -- 172.21.15.27:0/1540359421 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b800054e0 con 0x7f4b98158900 2024-01-15T18:18:30.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.089+0000 7f4b8ea00640 1 -- 172.21.15.27:0/1540359421 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b80003600 con 0x7f4b98158900 2024-01-15T18:18:30.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.089+0000 7f4b8ea00640 1 -- 172.21.15.27:0/1540359421 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b800039b0 con 0x7f4b98158900 2024-01-15T18:18:30.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.089+0000 7f4b9d400640 1 -- 172.21.15.27:0/1540359421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b980cd0b0 con 0x7f4b98158900 2024-01-15T18:18:30.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.089+0000 7f4b9d400640 1 -- 172.21.15.27:0/1540359421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b9814d360 con 0x7f4b98158900 2024-01-15T18:18:30.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.089+0000 7f4b9d400640 1 -- 172.21.15.27:0/1540359421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b9804e850 con 0x7f4b98158900 2024-01-15T18:18:30.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.089+0000 7f4b8ea00640 1 -- 172.21.15.27:0/1540359421 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f4b80003b10 con 0x7f4b98158900 2024-01-15T18:18:30.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.089+0000 7f4b8ea00640 1 --2- 172.21.15.27:0/1540359421 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4b70067940 0x7f4b70069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:30.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.089+0000 7f4b8ea00640 1 -- 172.21.15.27:0/1540359421 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4969+0+0 (secure 0 0 0) 0x7f4b80082920 con 0x7f4b98158900 2024-01-15T18:18:30.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.089+0000 7f4b96800640 1 --2- 172.21.15.27:0/1540359421 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4b70067940 0x7f4b70069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:30.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.093+0000 7f4b96800640 1 --2- 172.21.15.27:0/1540359421 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4b70067940 0x7f4b70069e00 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f4b88009ae0 tx=0x7f4b88008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:30.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.093+0000 7f4b8ea00640 1 -- 172.21.15.27:0/1540359421 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4b80051200 con 0x7f4b98158900 2024-01-15T18:18:30.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.253+0000 7f4b9d400640 1 -- 172.21.15.27:0/1540359421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4b98073c00 con 0x7f4b98158900 2024-01-15T18:18:30.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.253+0000 7f4b8ea00640 1 -- 172.21.15.27:0/1540359421 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4b98073c00 con 0x7f4b98158900 2024-01-15T18:18:30.254 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:18:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.257+0000 7f4b9d400640 1 -- 172.21.15.27:0/1540359421 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4b70067940 msgr2=0x7f4b70069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.257+0000 7f4b9d400640 1 --2- 172.21.15.27:0/1540359421 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4b70067940 0x7f4b70069e00 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f4b88009ae0 tx=0x7f4b88008040 comp rx=0 tx=0).stop 2024-01-15T18:18:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.257+0000 7f4b9d400640 1 -- 172.21.15.27:0/1540359421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b98158900 msgr2=0x7f4b98151940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.257+0000 7f4b9d400640 1 --2- 172.21.15.27:0/1540359421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b98158900 0x7f4b98151940 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f4b80001ec0 tx=0x7f4b8000c8e0 comp rx=0 tx=0).stop 2024-01-15T18:18:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.257+0000 7f4b9d400640 1 -- 172.21.15.27:0/1540359421 shutdown_connections 2024-01-15T18:18:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.257+0000 7f4b9d400640 1 --2- 172.21.15.27:0/1540359421 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4b70067940 0x7f4b70069e00 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.257+0000 7f4b9d400640 1 --2- 172.21.15.27:0/1540359421 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4b9815bf00 0x7f4b9815da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.257+0000 7f4b9d400640 1 --2- 172.21.15.27:0/1540359421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b98158900 0x7f4b98151940 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.257+0000 7f4b9d400640 1 --2- 172.21.15.27:0/1540359421 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4b980cf2e0 0x7f4b9814c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.257+0000 7f4b9d400640 1 -- 172.21.15.27:0/1540359421 >> 172.21.15.27:0/1540359421 conn(0x7f4b980cb700 msgr2=0x7f4b980b0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.257+0000 7f4b9d400640 1 -- 172.21.15.27:0/1540359421 shutdown_connections 2024-01-15T18:18:30.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.257+0000 7f4b9d400640 1 -- 172.21.15.27:0/1540359421 wait complete. 2024-01-15T18:18:30.277 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-15T18:18:30.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce5c400640 1 Processor -- start 2024-01-15T18:18:30.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce5c400640 1 -- start start 2024-01-15T18:18:30.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce5c400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fce5415e7d0 0x7fce540cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:30.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce5c400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce540b38b0 con 0x7fce540b20d0 2024-01-15T18:18:30.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce5c400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce540cd550 con 0x7fce540cedc0 2024-01-15T18:18:30.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce5c400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce540cd270 con 0x7fce5415e7d0 2024-01-15T18:18:30.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce5a400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fce540b20d0 0x7fce540cc1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46584/0 (socket says 172.21.15.27:46584) 2024-01-15T18:18:30.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce5a400640 1 -- 172.21.15.27:0/515272878 learned_addr learned my addr 172.21.15.27:0/515272878 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:30.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce59a00640 1 --2- 172.21.15.27:0/515272878 >> v2:172.21.15.93:3300/0 conn(0x7fce5415e7d0 0x7fce540cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:30.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce53e00640 1 -- 172.21.15.27:0/515272878 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766574030 0 0) 0x7fce540b38b0 con 0x7fce540b20d0 2024-01-15T18:18:30.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce53e00640 1 -- 172.21.15.27:0/515272878 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce34003530 con 0x7fce540b20d0 2024-01-15T18:18:30.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce53e00640 1 -- 172.21.15.27:0/515272878 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2854081938 0 0) 0x7fce34003530 con 0x7fce540b20d0 2024-01-15T18:18:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce53e00640 1 -- 172.21.15.27:0/515272878 >> v2:172.21.15.93:3300/0 conn(0x7fce5415e7d0 msgr2=0x7fce540cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce53e00640 1 --2- 172.21.15.27:0/515272878 >> v2:172.21.15.93:3300/0 conn(0x7fce5415e7d0 0x7fce540cf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce53e00640 1 -- 172.21.15.27:0/515272878 >> v1:172.21.15.93:6789/0 conn(0x7fce540cedc0 legacy=0x7fce540cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce53e00640 1 -- 172.21.15.27:0/515272878 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce540b35d0 con 0x7fce540b20d0 2024-01-15T18:18:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce53e00640 1 -- 172.21.15.27:0/515272878 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fce4c001a00 con 0x7fce540b20d0 2024-01-15T18:18:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce53e00640 1 -- 172.21.15.27:0/515272878 >> v1:172.21.15.27:6789/0 conn(0x7fce540b20d0 legacy=0x7fce540cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce53e00640 1 --2- 172.21.15.27:0/515272878 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce340039f0 0x7fce34003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce53e00640 1 --2- 172.21.15.27:0/515272878 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce340047d0 0x7fce34004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce53e00640 1 --2- 172.21.15.27:0/515272878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce34005190 0x7fce3400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce53e00640 1 -- 172.21.15.27:0/515272878 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce540cd270 con 0x7fce34005190 2024-01-15T18:18:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce53e00640 1 -- 172.21.15.27:0/515272878 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3400c0c0 con 0x7fce340039f0 2024-01-15T18:18:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce53e00640 1 -- 172.21.15.27:0/515272878 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce3400c280 con 0x7fce340047d0 2024-01-15T18:18:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce59a00640 1 --2- 172.21.15.27:0/515272878 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce340039f0 0x7fce34003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce5a400640 1 --2- 172.21.15.27:0/515272878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce34005190 0x7fce3400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce59000640 1 --2- 172.21.15.27:0/515272878 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce340047d0 0x7fce34004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce59a00640 1 -- 172.21.15.27:0/515272878 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce340047d0 msgr2=0x7fce34004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.421+0000 7fce59a00640 1 --2- 172.21.15.27:0/515272878 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce340047d0 0x7fce34004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.425+0000 7fce59a00640 1 -- 172.21.15.27:0/515272878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce34005190 msgr2=0x7fce3400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.425+0000 7fce59a00640 1 --2- 172.21.15.27:0/515272878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce34005190 0x7fce3400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.425+0000 7fce59a00640 1 -- 172.21.15.27:0/515272878 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce34003530 con 0x7fce340039f0 2024-01-15T18:18:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.425+0000 7fce5a400640 1 --2- 172.21.15.27:0/515272878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce34005190 0x7fce3400b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:18:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.425+0000 7fce5a400640 1 -- 172.21.15.27:0/515272878 reap_dead start 2024-01-15T18:18:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.425+0000 7fce59a00640 1 --2- 172.21.15.27:0/515272878 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce340039f0 0x7fce34003e10 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fce44009ae0 tx=0x7fce4401ca50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=627e88b438894d4e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.425+0000 7fce53e00640 1 -- 172.21.15.27:0/515272878 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce440a5640 con 0x7fce340039f0 2024-01-15T18:18:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.425+0000 7fce53e00640 1 -- 172.21.15.27:0/515272878 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce440a57a0 con 0x7fce340039f0 2024-01-15T18:18:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.425+0000 7fce53e00640 1 -- 172.21.15.27:0/515272878 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce440a5b50 con 0x7fce340039f0 2024-01-15T18:18:30.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.425+0000 7fce5c400640 1 -- 172.21.15.27:0/515272878 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce340039f0 msgr2=0x7fce34003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:30.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.425+0000 7fce5c400640 1 --2- 172.21.15.27:0/515272878 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce340039f0 0x7fce34003e10 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fce44009ae0 tx=0x7fce4401ca50 comp rx=0 tx=0).stop 2024-01-15T18:18:30.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.429+0000 7fce5c400640 1 -- 172.21.15.27:0/515272878 shutdown_connections 2024-01-15T18:18:30.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.429+0000 7fce5c400640 1 --2- 172.21.15.27:0/515272878 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce340039f0 0x7fce34003e10 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:30.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.429+0000 7fce5c400640 1 -- 172.21.15.27:0/515272878 >> 172.21.15.27:0/515272878 conn(0x7fce540b2a10 msgr2=0x7fce540b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:30.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.429+0000 7fce5c400640 1 -- 172.21.15.27:0/515272878 shutdown_connections 2024-01-15T18:18:30.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.429+0000 7fce5c400640 1 -- 172.21.15.27:0/515272878 wait complete. 2024-01-15T18:18:30.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.429+0000 7fce5c400640 1 Processor -- start 2024-01-15T18:18:30.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.429+0000 7fce5c400640 1 -- start start 2024-01-15T18:18:30.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.429+0000 7fce5c400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce340039f0 0x7fce5415a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:30.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.429+0000 7fce5c400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce541503b0 0x7fce540b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:30.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.429+0000 7fce5c400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce54150b70 0x7fce5415c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:30.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.429+0000 7fce5c400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce540b4260 con 0x7fce340039f0 2024-01-15T18:18:30.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.429+0000 7fce5c400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce540b3fe0 con 0x7fce541503b0 2024-01-15T18:18:30.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.429+0000 7fce5c400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce5406bde0 con 0x7fce54150b70 2024-01-15T18:18:30.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.429+0000 7fce59a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce340039f0 0x7fce5415a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:30.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.429+0000 7fce59a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce340039f0 0x7fce5415a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58136/0 (socket says 172.21.15.27:58136) 2024-01-15T18:18:30.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.429+0000 7fce59a00640 1 -- 172.21.15.27:0/4005138814 learned_addr learned my addr 172.21.15.27:0/4005138814 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:30.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.429+0000 7fce5a400640 1 --2- 172.21.15.27:0/4005138814 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce54150b70 0x7fce5415c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:30.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.429+0000 7fce59000640 1 --2- 172.21.15.27:0/4005138814 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce541503b0 0x7fce540b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:30.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.429+0000 7fce59a00640 1 -- 172.21.15.27:0/4005138814 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce54150b70 msgr2=0x7fce5415c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:30.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.429+0000 7fce59a00640 1 --2- 172.21.15.27:0/4005138814 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce54150b70 0x7fce5415c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:30.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.429+0000 7fce59a00640 1 -- 172.21.15.27:0/4005138814 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce541503b0 msgr2=0x7fce540b3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:30.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.429+0000 7fce59a00640 1 --2- 172.21.15.27:0/4005138814 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce541503b0 0x7fce540b3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:30.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.429+0000 7fce59a00640 1 -- 172.21.15.27:0/4005138814 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce5406c0c0 con 0x7fce340039f0 2024-01-15T18:18:30.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.429+0000 7fce59000640 1 --2- 172.21.15.27:0/4005138814 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce541503b0 0x7fce540b3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:18:30.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.433+0000 7fce59a00640 1 --2- 172.21.15.27:0/4005138814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce340039f0 0x7fce5415a6b0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fce440b0040 tx=0x7fce440197f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:30.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.433+0000 7fce51600640 1 -- 172.21.15.27:0/4005138814 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce440a5640 con 0x7fce340039f0 2024-01-15T18:18:30.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.433+0000 7fce51600640 1 -- 172.21.15.27:0/4005138814 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce44004260 con 0x7fce340039f0 2024-01-15T18:18:30.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.433+0000 7fce51600640 1 -- 172.21.15.27:0/4005138814 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce44003620 con 0x7fce340039f0 2024-01-15T18:18:30.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.433+0000 7fce5c400640 1 -- 172.21.15.27:0/4005138814 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce5406c3a0 con 0x7fce340039f0 2024-01-15T18:18:30.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.433+0000 7fce5c400640 1 -- 172.21.15.27:0/4005138814 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce541507d0 con 0x7fce340039f0 2024-01-15T18:18:30.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.433+0000 7fce5c400640 1 -- 172.21.15.27:0/4005138814 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce5406bf20 con 0x7fce340039f0 2024-01-15T18:18:30.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.433+0000 7fce51600640 1 -- 172.21.15.27:0/4005138814 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7fce440b4070 con 0x7fce340039f0 2024-01-15T18:18:30.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.433+0000 7fce51600640 1 --2- 172.21.15.27:0/4005138814 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce2c067940 0x7fce2c069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:30.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.433+0000 7fce51600640 1 -- 172.21.15.27:0/4005138814 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4969+0+0 (secure 0 0 0) 0x7fce4411d080 con 0x7fce340039f0 2024-01-15T18:18:30.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.433+0000 7fce59000640 1 --2- 172.21.15.27:0/4005138814 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce2c067940 0x7fce2c069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:30.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.437+0000 7fce59000640 1 --2- 172.21.15.27:0/4005138814 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce2c067940 0x7fce2c069e00 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fce3c000c10 tx=0x7fce3c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:30.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.437+0000 7fce51600640 1 -- 172.21.15.27:0/4005138814 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fce5406bf20 con 0x7fce340039f0 2024-01-15T18:18:30.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.601+0000 7fce5c400640 1 -- 172.21.15.27:0/4005138814 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fce5406bf20 con 0x7fce340039f0 2024-01-15T18:18:30.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.601+0000 7fce51600640 1 -- 172.21.15.27:0/4005138814 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v104) v1 ==== 84+0+65 (secure 0 0 0) 0x7fce5406bf20 con 0x7fce340039f0 2024-01-15T18:18:30.600 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:18:30.600 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:18:30.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:30.600 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2024-01-15T18:18:30.601 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2024-01-15T18:18:30.601 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:30.601 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:18:30.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.601+0000 7fce5c400640 1 -- 172.21.15.27:0/4005138814 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce2c067940 msgr2=0x7fce2c069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:30.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.601+0000 7fce5c400640 1 --2- 172.21.15.27:0/4005138814 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce2c067940 0x7fce2c069e00 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fce3c000c10 tx=0x7fce3c009040 comp rx=0 tx=0).stop 2024-01-15T18:18:30.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.601+0000 7fce5c400640 1 -- 172.21.15.27:0/4005138814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce340039f0 msgr2=0x7fce5415a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:30.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.601+0000 7fce5c400640 1 --2- 172.21.15.27:0/4005138814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce340039f0 0x7fce5415a6b0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fce440b0040 tx=0x7fce440197f0 comp rx=0 tx=0).stop 2024-01-15T18:18:30.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.601+0000 7fce5c400640 1 -- 172.21.15.27:0/4005138814 shutdown_connections 2024-01-15T18:18:30.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.601+0000 7fce5c400640 1 --2- 172.21.15.27:0/4005138814 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fce2c067940 0x7fce2c069e00 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:30.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.601+0000 7fce5c400640 1 --2- 172.21.15.27:0/4005138814 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fce54150b70 0x7fce5415c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:30.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.601+0000 7fce5c400640 1 --2- 172.21.15.27:0/4005138814 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fce541503b0 0x7fce540b3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:30.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.601+0000 7fce5c400640 1 --2- 172.21.15.27:0/4005138814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fce340039f0 0x7fce5415a6b0 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:30.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.601+0000 7fce5c400640 1 -- 172.21.15.27:0/4005138814 >> 172.21.15.27:0/4005138814 conn(0x7fce540b2a10 msgr2=0x7fce5406fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:30.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.601+0000 7fce5c400640 1 -- 172.21.15.27:0/4005138814 shutdown_connections 2024-01-15T18:18:30.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.601+0000 7fce5c400640 1 -- 172.21.15.27:0/4005138814 wait complete. 2024-01-15T18:18:30.623 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-15T18:18:30.623 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-15T18:18:30.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.769+0000 7f2d9ae00640 1 Processor -- start 2024-01-15T18:18:30.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.769+0000 7f2d9ae00640 1 -- start start 2024-01-15T18:18:30.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.769+0000 7f2d9ae00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f2d94158900 0x7f2d9415f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:30.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.769+0000 7f2d9ae00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d940b1cb0 con 0x7f2d9415bf00 2024-01-15T18:18:30.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.769+0000 7f2d9ae00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d940b1f00 con 0x7f2d940cf2e0 2024-01-15T18:18:30.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.769+0000 7f2d9ae00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d940b21e0 con 0x7f2d94158900 2024-01-15T18:18:30.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.769+0000 7f2d93400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2d9415bf00 0x7f2d940cebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46590/0 (socket says 172.21.15.27:46590) 2024-01-15T18:18:30.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.769+0000 7f2d93400640 1 -- 172.21.15.27:0/271719027 learned_addr learned my addr 172.21.15.27:0/271719027 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:30.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.769+0000 7f2d93e00640 1 --2- 172.21.15.27:0/271719027 >> v2:172.21.15.93:3300/0 conn(0x7f2d94158900 0x7f2d9415f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:30.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.769+0000 7f2d92a00640 1 -- 172.21.15.27:0/271719027 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3479916032 0 0) 0x7f2d940b1cb0 con 0x7f2d9415bf00 2024-01-15T18:18:30.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.769+0000 7f2d92a00640 1 -- 172.21.15.27:0/271719027 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d74003530 con 0x7f2d9415bf00 2024-01-15T18:18:30.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.769+0000 7f2d93e00640 1 -- 172.21.15.27:0/271719027 >> v1:172.21.15.93:6789/0 conn(0x7f2d940cf2e0 legacy=0x7f2d940701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:30.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.769+0000 7f2d93e00640 1 -- 172.21.15.27:0/271719027 >> v1:172.21.15.27:6789/0 conn(0x7f2d9415bf00 legacy=0x7f2d940cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:30.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.769+0000 7f2d93e00640 1 -- 172.21.15.27:0/271719027 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d9415ea70 con 0x7f2d94158900 2024-01-15T18:18:30.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.773+0000 7f2d93e00640 1 --2- 172.21.15.27:0/271719027 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2d94158900 0x7f2d9415f550 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f2d840097c0 tx=0x7f2d8401c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c852d634cb9226a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:30.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.773+0000 7f2d92a00640 1 -- 172.21.15.27:0/271719027 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d840ab600 con 0x7f2d94158900 2024-01-15T18:18:30.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.773+0000 7f2d92a00640 1 -- 172.21.15.27:0/271719027 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d840a7050 con 0x7f2d94158900 2024-01-15T18:18:30.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.773+0000 7f2d92a00640 1 -- 172.21.15.27:0/271719027 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d840ab960 con 0x7f2d94158900 2024-01-15T18:18:30.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.773+0000 7f2d9ae00640 1 -- 172.21.15.27:0/271719027 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2d94158900 msgr2=0x7f2d9415f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:30.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.773+0000 7f2d9ae00640 1 --2- 172.21.15.27:0/271719027 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2d94158900 0x7f2d9415f550 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f2d840097c0 tx=0x7f2d8401c480 comp rx=0 tx=0).stop 2024-01-15T18:18:30.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.773+0000 7f2d9ae00640 1 -- 172.21.15.27:0/271719027 shutdown_connections 2024-01-15T18:18:30.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.773+0000 7f2d9ae00640 1 --2- 172.21.15.27:0/271719027 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2d94158900 0x7f2d9415f550 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:30.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.773+0000 7f2d9ae00640 1 -- 172.21.15.27:0/271719027 >> 172.21.15.27:0/271719027 conn(0x7f2d940cb700 msgr2=0x7f2d940cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:30.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.773+0000 7f2d9ae00640 1 -- 172.21.15.27:0/271719027 shutdown_connections 2024-01-15T18:18:30.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.777+0000 7f2d9ae00640 1 -- 172.21.15.27:0/271719027 wait complete. 2024-01-15T18:18:30.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.777+0000 7f2d9ae00640 1 Processor -- start 2024-01-15T18:18:30.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.777+0000 7f2d9ae00640 1 -- start start 2024-01-15T18:18:30.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.777+0000 7f2d9ae00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2d940cf2e0 0x7f2d940ad880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:30.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.777+0000 7f2d9ae00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2d94158900 0x7f2d9415de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:30.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.777+0000 7f2d9ae00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d9415bf00 0x7f2d9414c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:30.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.777+0000 7f2d9ae00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9415d9c0 con 0x7f2d9415bf00 2024-01-15T18:18:30.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.777+0000 7f2d9ae00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d940aed40 con 0x7f2d94158900 2024-01-15T18:18:30.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.777+0000 7f2d9ae00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d940af020 con 0x7f2d940cf2e0 2024-01-15T18:18:30.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.777+0000 7f2d98e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d9415bf00 0x7f2d9414c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:30.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.777+0000 7f2d98e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d9415bf00 0x7f2d9414c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58146/0 (socket says 172.21.15.27:58146) 2024-01-15T18:18:30.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.777+0000 7f2d98e00640 1 -- 172.21.15.27:0/926951200 learned_addr learned my addr 172.21.15.27:0/926951200 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:30.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.777+0000 7f2d93e00640 1 --2- 172.21.15.27:0/926951200 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2d940cf2e0 0x7f2d940ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:30.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.777+0000 7f2d93400640 1 --2- 172.21.15.27:0/926951200 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2d94158900 0x7f2d9415de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:30.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.777+0000 7f2d98e00640 1 -- 172.21.15.27:0/926951200 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2d940cf2e0 msgr2=0x7f2d940ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:30.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.777+0000 7f2d98e00640 1 --2- 172.21.15.27:0/926951200 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2d940cf2e0 0x7f2d940ad880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:30.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.777+0000 7f2d98e00640 1 -- 172.21.15.27:0/926951200 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2d94158900 msgr2=0x7f2d9415de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:30.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.777+0000 7f2d98e00640 1 --2- 172.21.15.27:0/926951200 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2d94158900 0x7f2d9415de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:30.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.777+0000 7f2d98e00640 1 -- 172.21.15.27:0/926951200 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d84009420 con 0x7f2d9415bf00 2024-01-15T18:18:30.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.777+0000 7f2d98e00640 1 --2- 172.21.15.27:0/926951200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d9415bf00 0x7f2d9414c700 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c00c7a0 tx=0x7f2d8c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:30.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.777+0000 7f2d8be00640 1 -- 172.21.15.27:0/926951200 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c003b80 con 0x7f2d9415bf00 2024-01-15T18:18:30.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.777+0000 7f2d8be00640 1 -- 172.21.15.27:0/926951200 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d8c003ce0 con 0x7f2d9415bf00 2024-01-15T18:18:30.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.781+0000 7f2d8be00640 1 -- 172.21.15.27:0/926951200 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c00aa30 con 0x7f2d9415bf00 2024-01-15T18:18:30.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.781+0000 7f2d9ae00640 1 -- 172.21.15.27:0/926951200 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d940af300 con 0x7f2d9415bf00 2024-01-15T18:18:30.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.781+0000 7f2d9ae00640 1 -- 172.21.15.27:0/926951200 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d9415d650 con 0x7f2d9415bf00 2024-01-15T18:18:30.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.781+0000 7f2d9ae00640 1 -- 172.21.15.27:0/926951200 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d9415db30 con 0x7f2d9415bf00 2024-01-15T18:18:30.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.781+0000 7f2d8be00640 1 -- 172.21.15.27:0/926951200 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f2d8c00ab90 con 0x7f2d9415bf00 2024-01-15T18:18:30.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.781+0000 7f2d8be00640 1 --2- 172.21.15.27:0/926951200 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2d6c067940 0x7f2d6c069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:30.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.781+0000 7f2d8be00640 1 -- 172.21.15.27:0/926951200 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 4969+0+0 (secure 0 0 0) 0x7f2d8c083090 con 0x7f2d9415bf00 2024-01-15T18:18:30.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.781+0000 7f2d93e00640 1 --2- 172.21.15.27:0/926951200 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2d6c067940 0x7f2d6c069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:30.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.785+0000 7f2d93e00640 1 --2- 172.21.15.27:0/926951200 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2d6c067940 0x7f2d6c069e00 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f2d840ad000 tx=0x7f2d840a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:30.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.785+0000 7f2d8be00640 1 -- 172.21.15.27:0/926951200 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2d8c051a30 con 0x7f2d9415bf00 2024-01-15T18:18:30.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:30.941+0000 7f2d9ae00640 1 -- 172.21.15.27:0/926951200 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2d9404e850 con 0x7f2d9415bf00 2024-01-15T18:18:31.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.177+0000 7f2d8be00640 1 -- 172.21.15.27:0/926951200 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v105) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2d8c055050 con 0x7f2d9415bf00 2024-01-15T18:18:31.178 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2024-01-15T18:18:31.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.181+0000 7f2d9ae00640 1 -- 172.21.15.27:0/926951200 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2d6c067940 msgr2=0x7f2d6c069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:31.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.181+0000 7f2d9ae00640 1 --2- 172.21.15.27:0/926951200 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2d6c067940 0x7f2d6c069e00 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f2d840ad000 tx=0x7f2d840a6000 comp rx=0 tx=0).stop 2024-01-15T18:18:31.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.181+0000 7f2d9ae00640 1 -- 172.21.15.27:0/926951200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d9415bf00 msgr2=0x7f2d9414c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:31.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.181+0000 7f2d9ae00640 1 --2- 172.21.15.27:0/926951200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d9415bf00 0x7f2d9414c700 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c00c7a0 tx=0x7f2d8c00cc60 comp rx=0 tx=0).stop 2024-01-15T18:18:31.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.181+0000 7f2d9ae00640 1 -- 172.21.15.27:0/926951200 shutdown_connections 2024-01-15T18:18:31.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.181+0000 7f2d9ae00640 1 --2- 172.21.15.27:0/926951200 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2d6c067940 0x7f2d6c069e00 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:31.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.181+0000 7f2d9ae00640 1 --2- 172.21.15.27:0/926951200 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d9415bf00 0x7f2d9414c700 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:31.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.181+0000 7f2d9ae00640 1 --2- 172.21.15.27:0/926951200 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2d94158900 0x7f2d9415de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:31.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.181+0000 7f2d9ae00640 1 --2- 172.21.15.27:0/926951200 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2d940cf2e0 0x7f2d940ad880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:31.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.181+0000 7f2d9ae00640 1 -- 172.21.15.27:0/926951200 >> 172.21.15.27:0/926951200 conn(0x7f2d940cb700 msgr2=0x7f2d940cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:31.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.181+0000 7f2d9ae00640 1 -- 172.21.15.27:0/926951200 shutdown_connections 2024-01-15T18:18:31.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.181+0000 7f2d9ae00640 1 -- 172.21.15.27:0/926951200 wait complete. 2024-01-15T18:18:31.206 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-15T18:18:31.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.353+0000 7f1afa800640 1 Processor -- start 2024-01-15T18:18:31.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.353+0000 7f1afa800640 1 -- start start 2024-01-15T18:18:31.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.357+0000 7f1afa800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f1af406c3f0 0x7f1af4168250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:31.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.357+0000 7f1afa800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af40b04b0 con 0x7f1af415d3b0 2024-01-15T18:18:31.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.357+0000 7f1afa800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af40b0720 con 0x7f1af406b600 2024-01-15T18:18:31.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.357+0000 7f1afa800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1af40b0a00 con 0x7f1af406c3f0 2024-01-15T18:18:31.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.357+0000 7f1af3400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1af415d3b0 0x7f1af40cfc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46598/0 (socket says 172.21.15.27:46598) 2024-01-15T18:18:31.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.357+0000 7f1af3400640 1 -- 172.21.15.27:0/2937638617 learned_addr learned my addr 172.21.15.27:0/2937638617 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:31.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.357+0000 7f1af3e00640 1 --2- 172.21.15.27:0/2937638617 >> v2:172.21.15.93:3300/0 conn(0x7f1af406c3f0 0x7f1af4168250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:31.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.357+0000 7f1af2000640 1 -- 172.21.15.27:0/2937638617 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828235462 0 0) 0x7f1af40b04b0 con 0x7f1af415d3b0 2024-01-15T18:18:31.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.357+0000 7f1af2000640 1 -- 172.21.15.27:0/2937638617 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad4003530 con 0x7f1af415d3b0 2024-01-15T18:18:31.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.357+0000 7f1af3e00640 1 -- 172.21.15.27:0/2937638617 >> v1:172.21.15.93:6789/0 conn(0x7f1af406b600 legacy=0x7f1af406bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:31.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.357+0000 7f1af3e00640 1 -- 172.21.15.27:0/2937638617 >> v1:172.21.15.27:6789/0 conn(0x7f1af415d3b0 legacy=0x7f1af40cfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:31.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.357+0000 7f1af3e00640 1 -- 172.21.15.27:0/2937638617 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af40b0ce0 con 0x7f1af406c3f0 2024-01-15T18:18:31.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.361+0000 7f1af3e00640 1 --2- 172.21.15.27:0/2937638617 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1af406c3f0 0x7f1af4168250 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f1aec009d90 tx=0x7f1aec01c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67095eb1142ba419 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:31.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.361+0000 7f1af2000640 1 -- 172.21.15.27:0/2937638617 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aec0a59b0 con 0x7f1af406c3f0 2024-01-15T18:18:31.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.361+0000 7f1af2000640 1 -- 172.21.15.27:0/2937638617 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aec0a5b10 con 0x7f1af406c3f0 2024-01-15T18:18:31.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.361+0000 7f1af2000640 1 -- 172.21.15.27:0/2937638617 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aec0a5e80 con 0x7f1af406c3f0 2024-01-15T18:18:31.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.361+0000 7f1afa800640 1 -- 172.21.15.27:0/2937638617 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1af406c3f0 msgr2=0x7f1af4168250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:31.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.361+0000 7f1afa800640 1 --2- 172.21.15.27:0/2937638617 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1af406c3f0 0x7f1af4168250 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f1aec009d90 tx=0x7f1aec01c950 comp rx=0 tx=0).stop 2024-01-15T18:18:31.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.361+0000 7f1afa800640 1 -- 172.21.15.27:0/2937638617 shutdown_connections 2024-01-15T18:18:31.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.361+0000 7f1afa800640 1 --2- 172.21.15.27:0/2937638617 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1af406c3f0 0x7f1af4168250 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:31.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.361+0000 7f1afa800640 1 -- 172.21.15.27:0/2937638617 >> 172.21.15.27:0/2937638617 conn(0x7f1af40d0ed0 msgr2=0x7f1af40d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:31.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.361+0000 7f1afa800640 1 -- 172.21.15.27:0/2937638617 shutdown_connections 2024-01-15T18:18:31.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.361+0000 7f1afa800640 1 -- 172.21.15.27:0/2937638617 wait complete. 2024-01-15T18:18:31.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.361+0000 7f1afa800640 1 Processor -- start 2024-01-15T18:18:31.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.361+0000 7f1afa800640 1 -- start start 2024-01-15T18:18:31.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.365+0000 7f1afa800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1af406b600 0x7f1af4168190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:31.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.365+0000 7f1afa800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1af406c3f0 0x7f1af4159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:31.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.365+0000 7f1afa800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1af415d3b0 0x7f1af4151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:31.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.365+0000 7f1afa800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af4148c20 con 0x7f1af415d3b0 2024-01-15T18:18:31.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.365+0000 7f1afa800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af40c5390 con 0x7f1af406c3f0 2024-01-15T18:18:31.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.365+0000 7f1afa800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1af40c5820 con 0x7f1af406b600 2024-01-15T18:18:31.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.365+0000 7f1af3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1af415d3b0 0x7f1af4151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:31.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.365+0000 7f1af3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1af415d3b0 0x7f1af4151940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58160/0 (socket says 172.21.15.27:58160) 2024-01-15T18:18:31.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.365+0000 7f1af3e00640 1 -- 172.21.15.27:0/639503506 learned_addr learned my addr 172.21.15.27:0/639503506 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:31.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.365+0000 7f1af3400640 1 --2- 172.21.15.27:0/639503506 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1af406b600 0x7f1af4168190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:31.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.365+0000 7f1af2a00640 1 --2- 172.21.15.27:0/639503506 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1af406c3f0 0x7f1af4159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:31.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.365+0000 7f1af3e00640 1 -- 172.21.15.27:0/639503506 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1af406b600 msgr2=0x7f1af4168190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.365+0000 7f1af3e00640 1 --2- 172.21.15.27:0/639503506 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1af406b600 0x7f1af4168190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.365+0000 7f1af3e00640 1 -- 172.21.15.27:0/639503506 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1af406c3f0 msgr2=0x7f1af4159ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.365+0000 7f1af3e00640 1 --2- 172.21.15.27:0/639503506 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1af406c3f0 0x7f1af4159ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.365+0000 7f1af3e00640 1 -- 172.21.15.27:0/639503506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aec009a70 con 0x7f1af415d3b0 2024-01-15T18:18:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.365+0000 7f1af3400640 1 --2- 172.21.15.27:0/639503506 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1af406b600 0x7f1af4168190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:18:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.365+0000 7f1af2a00640 1 --2- 172.21.15.27:0/639503506 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1af406c3f0 0x7f1af4159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:18:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.365+0000 7f1af3e00640 1 --2- 172.21.15.27:0/639503506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1af415d3b0 0x7f1af4151940 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f1aec019560 tx=0x7f1aec0196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.365+0000 7f1aeb400640 1 -- 172.21.15.27:0/639503506 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aec0aee00 con 0x7f1af415d3b0 2024-01-15T18:18:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.365+0000 7f1aeb400640 1 -- 172.21.15.27:0/639503506 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aec009450 con 0x7f1af415d3b0 2024-01-15T18:18:31.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.365+0000 7f1aeb400640 1 -- 172.21.15.27:0/639503506 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aec003bc0 con 0x7f1af415d3b0 2024-01-15T18:18:31.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.365+0000 7f1afa800640 1 -- 172.21.15.27:0/639503506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af4151e80 con 0x7f1af415d3b0 2024-01-15T18:18:31.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.365+0000 7f1afa800640 1 -- 172.21.15.27:0/639503506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af40c4a60 con 0x7f1af415d3b0 2024-01-15T18:18:31.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.365+0000 7f1afa800640 1 -- 172.21.15.27:0/639503506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1af40c5520 con 0x7f1af415d3b0 2024-01-15T18:18:31.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.369+0000 7f1aeb400640 1 -- 172.21.15.27:0/639503506 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f1aec0b4050 con 0x7f1af415d3b0 2024-01-15T18:18:31.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.369+0000 7f1aeb400640 1 --2- 172.21.15.27:0/639503506 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1acc067940 0x7f1acc069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:31.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.369+0000 7f1aeb400640 1 -- 172.21.15.27:0/639503506 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f1aec11d710 con 0x7f1af415d3b0 2024-01-15T18:18:31.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.369+0000 7f1af3400640 1 --2- 172.21.15.27:0/639503506 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1acc067940 0x7f1acc069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:31.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.369+0000 7f1af3400640 1 --2- 172.21.15.27:0/639503506 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1acc067940 0x7f1acc069e00 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f1ae40011f0 tx=0x7f1ae4007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:31.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.373+0000 7f1aeb400640 1 -- 172.21.15.27:0/639503506 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1aec0ec130 con 0x7f1af415d3b0 2024-01-15T18:18:31.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:31.533+0000 7f1afa800640 1 -- 172.21.15.27:0/639503506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f1af404e850 con 0x7f1af415d3b0 2024-01-15T18:18:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.185+0000 7f1aeb400640 1 -- 172.21.15.27:0/639503506 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v106) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1aec0b20d0 con 0x7f1af415d3b0 2024-01-15T18:18:32.187 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2024-01-15T18:18:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.189+0000 7f1afa800640 1 -- 172.21.15.27:0/639503506 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1acc067940 msgr2=0x7f1acc069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:32.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.189+0000 7f1afa800640 1 --2- 172.21.15.27:0/639503506 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1acc067940 0x7f1acc069e00 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f1ae40011f0 tx=0x7f1ae4007d40 comp rx=0 tx=0).stop 2024-01-15T18:18:32.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.189+0000 7f1afa800640 1 -- 172.21.15.27:0/639503506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1af415d3b0 msgr2=0x7f1af4151940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:32.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.189+0000 7f1afa800640 1 --2- 172.21.15.27:0/639503506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1af415d3b0 0x7f1af4151940 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f1aec019560 tx=0x7f1aec0196f0 comp rx=0 tx=0).stop 2024-01-15T18:18:32.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.189+0000 7f1afa800640 1 -- 172.21.15.27:0/639503506 shutdown_connections 2024-01-15T18:18:32.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.189+0000 7f1afa800640 1 --2- 172.21.15.27:0/639503506 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1acc067940 0x7f1acc069e00 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:32.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.189+0000 7f1afa800640 1 --2- 172.21.15.27:0/639503506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1af415d3b0 0x7f1af4151940 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:32.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.189+0000 7f1afa800640 1 --2- 172.21.15.27:0/639503506 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f1af406c3f0 0x7f1af4159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:32.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.189+0000 7f1afa800640 1 --2- 172.21.15.27:0/639503506 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1af406b600 0x7f1af4168190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:32.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.189+0000 7f1afa800640 1 -- 172.21.15.27:0/639503506 >> 172.21.15.27:0/639503506 conn(0x7f1af40d0ed0 msgr2=0x7f1af40af390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:32.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.189+0000 7f1afa800640 1 -- 172.21.15.27:0/639503506 shutdown_connections 2024-01-15T18:18:32.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.189+0000 7f1afa800640 1 -- 172.21.15.27:0/639503506 wait complete. 2024-01-15T18:18:32.232 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-15T18:18:32.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.377+0000 7ff273400640 1 Processor -- start 2024-01-15T18:18:32.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.377+0000 7ff273400640 1 -- start start 2024-01-15T18:18:32.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.377+0000 7ff273400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff26c15b100 0x7ff26c070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:32.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.377+0000 7ff273400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26c071860 con 0x7ff26c071420 2024-01-15T18:18:32.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.377+0000 7ff273400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26c15a610 con 0x7ff26c15ad20 2024-01-15T18:18:32.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.377+0000 7ff273400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff26c15e3c0 con 0x7ff26c15b100 2024-01-15T18:18:32.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.381+0000 7ff271400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff26c071420 0x7ff26c15bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46604/0 (socket says 172.21.15.27:46604) 2024-01-15T18:18:32.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.381+0000 7ff271400640 1 -- 172.21.15.27:0/2391244299 learned_addr learned my addr 172.21.15.27:0/2391244299 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:32.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.381+0000 7ff26be00640 1 --2- 172.21.15.27:0/2391244299 >> v2:172.21.15.93:3300/0 conn(0x7ff26c15b100 0x7ff26c070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:32.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.381+0000 7ff26b400640 1 -- 172.21.15.27:0/2391244299 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834144658 0 0) 0x7ff26c071860 con 0x7ff26c071420 2024-01-15T18:18:32.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.381+0000 7ff26b400640 1 -- 172.21.15.27:0/2391244299 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff24c005740 con 0x7ff26c071420 2024-01-15T18:18:32.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.381+0000 7ff26b400640 1 -- 172.21.15.27:0/2391244299 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1494663796 0 0) 0x7ff24c005740 con 0x7ff26c071420 2024-01-15T18:18:32.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.381+0000 7ff26be00640 1 -- 172.21.15.27:0/2391244299 >> v1:172.21.15.93:6789/0 conn(0x7ff26c15ad20 legacy=0x7ff26c15d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:32.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.381+0000 7ff26be00640 1 -- 172.21.15.27:0/2391244299 >> v1:172.21.15.27:6789/0 conn(0x7ff26c071420 legacy=0x7ff26c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:32.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.381+0000 7ff26be00640 1 -- 172.21.15.27:0/2391244299 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26c15e6a0 con 0x7ff26c15b100 2024-01-15T18:18:32.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.381+0000 7ff26be00640 1 --2- 172.21.15.27:0/2391244299 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff26c15b100 0x7ff26c070ee0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7ff2540097f0 tx=0x7ff25401c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8e1ecf5ad937e02 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:32.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.385+0000 7ff26b400640 1 -- 172.21.15.27:0/2391244299 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff264001a00 con 0x7ff26c15b100 2024-01-15T18:18:32.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.385+0000 7ff26b400640 1 -- 172.21.15.27:0/2391244299 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2540a7050 con 0x7ff26c15b100 2024-01-15T18:18:32.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.385+0000 7ff26b400640 1 -- 172.21.15.27:0/2391244299 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2540ab600 con 0x7ff26c15b100 2024-01-15T18:18:32.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.385+0000 7ff273400640 1 -- 172.21.15.27:0/2391244299 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff26c15b100 msgr2=0x7ff26c070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:32.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.385+0000 7ff273400640 1 --2- 172.21.15.27:0/2391244299 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff26c15b100 0x7ff26c070ee0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7ff2540097f0 tx=0x7ff25401c2e0 comp rx=0 tx=0).stop 2024-01-15T18:18:32.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.385+0000 7ff273400640 1 -- 172.21.15.27:0/2391244299 shutdown_connections 2024-01-15T18:18:32.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.385+0000 7ff273400640 1 --2- 172.21.15.27:0/2391244299 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff26c15b100 0x7ff26c070ee0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:32.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.385+0000 7ff273400640 1 -- 172.21.15.27:0/2391244299 >> 172.21.15.27:0/2391244299 conn(0x7ff26c0cb700 msgr2=0x7ff26c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:32.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.385+0000 7ff273400640 1 -- 172.21.15.27:0/2391244299 shutdown_connections 2024-01-15T18:18:32.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.385+0000 7ff273400640 1 -- 172.21.15.27:0/2391244299 wait complete. 2024-01-15T18:18:32.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.385+0000 7ff273400640 1 Processor -- start 2024-01-15T18:18:32.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.385+0000 7ff273400640 1 -- start start 2024-01-15T18:18:32.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.385+0000 7ff273400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff26c071420 0x7ff26c13d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:32.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.385+0000 7ff273400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff26c15ad20 0x7ff26c1527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:32.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.385+0000 7ff273400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff26c15b100 0x7ff26c14f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:32.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.385+0000 7ff273400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26c151710 con 0x7ff26c15ad20 2024-01-15T18:18:32.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.385+0000 7ff273400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26c1519f0 con 0x7ff26c15b100 2024-01-15T18:18:32.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.389+0000 7ff273400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff26c151cd0 con 0x7ff26c071420 2024-01-15T18:18:32.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.389+0000 7ff26be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff26c15ad20 0x7ff26c1527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:32.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.389+0000 7ff26be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff26c15ad20 0x7ff26c1527b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58172/0 (socket says 172.21.15.27:58172) 2024-01-15T18:18:32.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.389+0000 7ff26be00640 1 -- 172.21.15.27:0/3562356565 learned_addr learned my addr 172.21.15.27:0/3562356565 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:32.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.389+0000 7ff271400640 1 --2- 172.21.15.27:0/3562356565 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff26c15b100 0x7ff26c14f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:32.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.389+0000 7ff270a00640 1 --2- 172.21.15.27:0/3562356565 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff26c071420 0x7ff26c13d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:32.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.389+0000 7ff26be00640 1 -- 172.21.15.27:0/3562356565 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff26c071420 msgr2=0x7ff26c13d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:32.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.389+0000 7ff26be00640 1 --2- 172.21.15.27:0/3562356565 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff26c071420 0x7ff26c13d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:32.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.389+0000 7ff26be00640 1 -- 172.21.15.27:0/3562356565 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff26c15b100 msgr2=0x7ff26c14f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:32.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.389+0000 7ff26be00640 1 --2- 172.21.15.27:0/3562356565 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff26c15b100 0x7ff26c14f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:32.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.389+0000 7ff26be00640 1 -- 172.21.15.27:0/3562356565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff254009420 con 0x7ff26c15ad20 2024-01-15T18:18:32.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.389+0000 7ff270a00640 1 --2- 172.21.15.27:0/3562356565 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff26c071420 0x7ff26c13d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:18:32.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.389+0000 7ff26be00640 1 --2- 172.21.15.27:0/3562356565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff26c15ad20 0x7ff26c1527b0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7ff2540097f0 tx=0x7ff254019080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:32.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.389+0000 7ff268c00640 1 -- 172.21.15.27:0/3562356565 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2540b5410 con 0x7ff26c15ad20 2024-01-15T18:18:32.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.389+0000 7ff268c00640 1 -- 172.21.15.27:0/3562356565 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2540b5570 con 0x7ff26c15ad20 2024-01-15T18:18:32.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.389+0000 7ff273400640 1 -- 172.21.15.27:0/3562356565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff26c151fb0 con 0x7ff26c15ad20 2024-01-15T18:18:32.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.389+0000 7ff268c00640 1 -- 172.21.15.27:0/3562356565 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2540b58e0 con 0x7ff26c15ad20 2024-01-15T18:18:32.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.389+0000 7ff273400640 1 -- 172.21.15.27:0/3562356565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff26c13db10 con 0x7ff26c15ad20 2024-01-15T18:18:32.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.389+0000 7ff273400640 1 -- 172.21.15.27:0/3562356565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff26c1524d0 con 0x7ff26c15ad20 2024-01-15T18:18:32.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.389+0000 7ff268c00640 1 -- 172.21.15.27:0/3562356565 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7ff2540b2050 con 0x7ff26c15ad20 2024-01-15T18:18:32.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.393+0000 7ff268c00640 1 --2- 172.21.15.27:0/3562356565 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff244067940 0x7ff244069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:32.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.393+0000 7ff268c00640 1 -- 172.21.15.27:0/3562356565 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5721+0+0 (secure 0 0 0) 0x7ff25411da50 con 0x7ff26c15ad20 2024-01-15T18:18:32.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.393+0000 7ff270a00640 1 --2- 172.21.15.27:0/3562356565 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff244067940 0x7ff244069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:32.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.393+0000 7ff270a00640 1 --2- 172.21.15.27:0/3562356565 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff244067940 0x7ff244069e00 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ff25c000f70 tx=0x7ff25c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:32.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.397+0000 7ff268c00640 1 -- 172.21.15.27:0/3562356565 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff2540ec5d0 con 0x7ff26c15ad20 2024-01-15T18:18:32.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:32.585+0000 7ff273400640 1 -- 172.21.15.27:0/3562356565 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff26c151e10 con 0x7ff26c15ad20 2024-01-15T18:18:33.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.193+0000 7ff268c00640 1 -- 172.21.15.27:0/3562356565 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v73) v1 ==== 174+0+0 (secure 0 0 0) 0x7ff26c151e10 con 0x7ff26c15ad20 2024-01-15T18:18:33.195 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 16 and data pool 17 2024-01-15T18:18:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.197+0000 7ff273400640 1 -- 172.21.15.27:0/3562356565 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff244067940 msgr2=0x7ff244069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.197+0000 7ff273400640 1 --2- 172.21.15.27:0/3562356565 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff244067940 0x7ff244069e00 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ff25c000f70 tx=0x7ff25c009040 comp rx=0 tx=0).stop 2024-01-15T18:18:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.197+0000 7ff273400640 1 -- 172.21.15.27:0/3562356565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff26c15ad20 msgr2=0x7ff26c1527b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.197+0000 7ff273400640 1 --2- 172.21.15.27:0/3562356565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff26c15ad20 0x7ff26c1527b0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7ff2540097f0 tx=0x7ff254019080 comp rx=0 tx=0).stop 2024-01-15T18:18:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.197+0000 7ff273400640 1 -- 172.21.15.27:0/3562356565 shutdown_connections 2024-01-15T18:18:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.197+0000 7ff273400640 1 --2- 172.21.15.27:0/3562356565 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff244067940 0x7ff244069e00 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.197+0000 7ff273400640 1 --2- 172.21.15.27:0/3562356565 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff26c15b100 0x7ff26c14f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:33.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.197+0000 7ff273400640 1 --2- 172.21.15.27:0/3562356565 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff26c15ad20 0x7ff26c1527b0 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:33.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.197+0000 7ff273400640 1 --2- 172.21.15.27:0/3562356565 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff26c071420 0x7ff26c13d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:33.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.197+0000 7ff273400640 1 -- 172.21.15.27:0/3562356565 >> 172.21.15.27:0/3562356565 conn(0x7ff26c0cb700 msgr2=0x7ff26c0ceb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:33.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.197+0000 7ff273400640 1 -- 172.21.15.27:0/3562356565 shutdown_connections 2024-01-15T18:18:33.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.197+0000 7ff273400640 1 -- 172.21.15.27:0/3562356565 wait complete. 2024-01-15T18:18:33.227 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:18:33.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.369+0000 7f233e200640 1 Processor -- start 2024-01-15T18:18:33.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.369+0000 7f233e200640 1 -- start start 2024-01-15T18:18:33.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.369+0000 7f233e200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f23380701a0 0x7f2338070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:33.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.369+0000 7f233e200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f233806c4a0 con 0x7f233815c130 2024-01-15T18:18:33.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.369+0000 7f233e200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2338070ac0 con 0x7f23380c92b0 2024-01-15T18:18:33.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.369+0000 7f233e200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f233815c660 con 0x7f23380701a0 2024-01-15T18:18:33.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.369+0000 7f2337e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f233815c130 0x7f233815d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46620/0 (socket says 172.21.15.27:46620) 2024-01-15T18:18:33.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.369+0000 7f2337e00640 1 -- 172.21.15.27:0/3791895922 learned_addr learned my addr 172.21.15.27:0/3791895922 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:33.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.369+0000 7f2337400640 1 --2- 172.21.15.27:0/3791895922 >> v2:172.21.15.93:3300/0 conn(0x7f23380701a0 0x7f2338070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:33.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.369+0000 7f2336000640 1 -- 172.21.15.27:0/3791895922 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764884293 0 0) 0x7f233806c4a0 con 0x7f233815c130 2024-01-15T18:18:33.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.373+0000 7f2336000640 1 -- 172.21.15.27:0/3791895922 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2318003530 con 0x7f233815c130 2024-01-15T18:18:33.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.373+0000 7f2336000640 1 -- 172.21.15.27:0/3791895922 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 682599177 0 0) 0x7f2318003530 con 0x7f233815c130 2024-01-15T18:18:33.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.373+0000 7f2337400640 1 -- 172.21.15.27:0/3791895922 >> v1:172.21.15.93:6789/0 conn(0x7f23380c92b0 legacy=0x7f233806dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:33.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.373+0000 7f2337400640 1 -- 172.21.15.27:0/3791895922 >> v1:172.21.15.27:6789/0 conn(0x7f233815c130 legacy=0x7f233815d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:33.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.373+0000 7f2337400640 1 -- 172.21.15.27:0/3791895922 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f233815c940 con 0x7f23380701a0 2024-01-15T18:18:33.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.373+0000 7f2337400640 1 --2- 172.21.15.27:0/3791895922 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f23380701a0 0x7f2338070580 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f23280097c0 tx=0x7f232801c370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3282bc6b803630af server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.373+0000 7f2336000640 1 -- 172.21.15.27:0/3791895922 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2330001a20 con 0x7f23380701a0 2024-01-15T18:18:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.373+0000 7f2336000640 1 -- 172.21.15.27:0/3791895922 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23280a7050 con 0x7f23380701a0 2024-01-15T18:18:33.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.373+0000 7f2336000640 1 -- 172.21.15.27:0/3791895922 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23280ab620 con 0x7f23380701a0 2024-01-15T18:18:33.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.377+0000 7f233e200640 1 -- 172.21.15.27:0/3791895922 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f23380701a0 msgr2=0x7f2338070580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:33.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.377+0000 7f233e200640 1 --2- 172.21.15.27:0/3791895922 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f23380701a0 0x7f2338070580 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f23280097c0 tx=0x7f232801c370 comp rx=0 tx=0).stop 2024-01-15T18:18:33.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.377+0000 7f233e200640 1 -- 172.21.15.27:0/3791895922 shutdown_connections 2024-01-15T18:18:33.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.377+0000 7f233e200640 1 --2- 172.21.15.27:0/3791895922 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f23380701a0 0x7f2338070580 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:33.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.377+0000 7f233e200640 1 -- 172.21.15.27:0/3791895922 >> 172.21.15.27:0/3791895922 conn(0x7f23380cec60 msgr2=0x7f23380cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:33.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.377+0000 7f233e200640 1 -- 172.21.15.27:0/3791895922 shutdown_connections 2024-01-15T18:18:33.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.377+0000 7f233e200640 1 -- 172.21.15.27:0/3791895922 wait complete. 2024-01-15T18:18:33.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.377+0000 7f233e200640 1 Processor -- start 2024-01-15T18:18:33.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.377+0000 7f233e200640 1 -- start start 2024-01-15T18:18:33.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.377+0000 7f233e200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23380701a0 0x7f2338151290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:33.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.377+0000 7f233e200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f23380c92b0 0x7f23380c4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:33.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.377+0000 7f233e200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f233815c130 0x7f23380c49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:33.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.377+0000 7f233e200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23380cdaa0 con 0x7f23380701a0 2024-01-15T18:18:33.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.377+0000 7f233e200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23380cdd80 con 0x7f23380c92b0 2024-01-15T18:18:33.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.377+0000 7f233e200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23380ce060 con 0x7f233815c130 2024-01-15T18:18:33.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.377+0000 7f2337400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23380701a0 0x7f2338151290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:33.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.377+0000 7f2337400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23380701a0 0x7f2338151290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58186/0 (socket says 172.21.15.27:58186) 2024-01-15T18:18:33.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.377+0000 7f2337400640 1 -- 172.21.15.27:0/3400766555 learned_addr learned my addr 172.21.15.27:0/3400766555 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:33.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.377+0000 7f2337e00640 1 --2- 172.21.15.27:0/3400766555 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f233815c130 0x7f23380c49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:33.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.377+0000 7f2336a00640 1 --2- 172.21.15.27:0/3400766555 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f23380c92b0 0x7f23380c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:33.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.381+0000 7f2337400640 1 -- 172.21.15.27:0/3400766555 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f233815c130 msgr2=0x7f23380c49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:33.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.381+0000 7f2337400640 1 --2- 172.21.15.27:0/3400766555 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f233815c130 0x7f23380c49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:33.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.381+0000 7f2337400640 1 -- 172.21.15.27:0/3400766555 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f23380c92b0 msgr2=0x7f23380c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:33.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.381+0000 7f2337400640 1 --2- 172.21.15.27:0/3400766555 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f23380c92b0 0x7f23380c4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:33.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.381+0000 7f2337400640 1 -- 172.21.15.27:0/3400766555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2328009420 con 0x7f23380701a0 2024-01-15T18:18:33.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.381+0000 7f2337400640 1 --2- 172.21.15.27:0/3400766555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23380701a0 0x7f2338151290 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f23280097c0 tx=0x7f232801abb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:33.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.381+0000 7f232f400640 1 -- 172.21.15.27:0/3400766555 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2328004020 con 0x7f23380701a0 2024-01-15T18:18:33.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.381+0000 7f232f400640 1 -- 172.21.15.27:0/3400766555 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2328004180 con 0x7f23380701a0 2024-01-15T18:18:33.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.381+0000 7f232f400640 1 -- 172.21.15.27:0/3400766555 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2328003620 con 0x7f23380701a0 2024-01-15T18:18:33.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.381+0000 7f233e200640 1 -- 172.21.15.27:0/3400766555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23380ce340 con 0x7f23380701a0 2024-01-15T18:18:33.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.381+0000 7f233e200640 1 -- 172.21.15.27:0/3400766555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23381517d0 con 0x7f23380701a0 2024-01-15T18:18:33.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.381+0000 7f233e200640 1 -- 172.21.15.27:0/3400766555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2338151960 con 0x7f23380701a0 2024-01-15T18:18:33.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.381+0000 7f232f400640 1 -- 172.21.15.27:0/3400766555 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f2328003780 con 0x7f23380701a0 2024-01-15T18:18:33.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.381+0000 7f232f400640 1 --2- 172.21.15.27:0/3400766555 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2310067940 0x7f2310069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:33.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.381+0000 7f232f400640 1 -- 172.21.15.27:0/3400766555 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f232811cb40 con 0x7f23380701a0 2024-01-15T18:18:33.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.381+0000 7f2336a00640 1 --2- 172.21.15.27:0/3400766555 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2310067940 0x7f2310069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:33.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.385+0000 7f2336a00640 1 --2- 172.21.15.27:0/3400766555 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2310067940 0x7f2310069e00 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f23380ae9a0 tx=0x7f2320009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:33.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.385+0000 7f232f400640 1 -- 172.21.15.27:0/3400766555 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f23280eb560 con 0x7f23380701a0 2024-01-15T18:18:33.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.545+0000 7f233e200640 1 -- 172.21.15.27:0/3400766555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f233804e850 con 0x7f23380701a0 2024-01-15T18:18:33.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.549+0000 7f232f400640 1 -- 172.21.15.27:0/3400766555 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16621 (secure 0 0 0) 0x7f23280b0090 con 0x7f23380701a0 2024-01-15T18:18:33.548 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:18:33.548 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":107,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:18:32.593006+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:18:30.993882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-15T18:18:31.582649+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:18:33.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.549+0000 7f233e200640 1 -- 172.21.15.27:0/3400766555 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2310067940 msgr2=0x7f2310069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:33.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.549+0000 7f233e200640 1 --2- 172.21.15.27:0/3400766555 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2310067940 0x7f2310069e00 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f23380ae9a0 tx=0x7f2320009040 comp rx=0 tx=0).stop 2024-01-15T18:18:33.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.549+0000 7f233e200640 1 -- 172.21.15.27:0/3400766555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23380701a0 msgr2=0x7f2338151290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:33.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.549+0000 7f233e200640 1 --2- 172.21.15.27:0/3400766555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23380701a0 0x7f2338151290 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f23280097c0 tx=0x7f232801abb0 comp rx=0 tx=0).stop 2024-01-15T18:18:33.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.549+0000 7f233e200640 1 -- 172.21.15.27:0/3400766555 shutdown_connections 2024-01-15T18:18:33.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.549+0000 7f233e200640 1 --2- 172.21.15.27:0/3400766555 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2310067940 0x7f2310069e00 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:33.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.549+0000 7f233e200640 1 --2- 172.21.15.27:0/3400766555 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f233815c130 0x7f23380c49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:33.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.549+0000 7f233e200640 1 --2- 172.21.15.27:0/3400766555 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f23380c92b0 0x7f23380c4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:33.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.549+0000 7f233e200640 1 --2- 172.21.15.27:0/3400766555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23380701a0 0x7f2338151290 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:33.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.549+0000 7f233e200640 1 -- 172.21.15.27:0/3400766555 >> 172.21.15.27:0/3400766555 conn(0x7f23380cec60 msgr2=0x7f23380718e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:33.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.549+0000 7f233e200640 1 -- 172.21.15.27:0/3400766555 shutdown_connections 2024-01-15T18:18:33.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.549+0000 7f233e200640 1 -- 172.21.15.27:0/3400766555 wait complete. 2024-01-15T18:18:33.571 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:18:33.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.713+0000 7efe7d400640 1 Processor -- start 2024-01-15T18:18:33.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.713+0000 7efe7d400640 1 -- start start 2024-01-15T18:18:33.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.713+0000 7efe7d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efe7815e7d0 0x7efe780cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:33.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.713+0000 7efe7d400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe780cedc0 0x7efe780cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:33.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.713+0000 7efe7d400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe780b38b0 con 0x7efe780ce2e0 2024-01-15T18:18:33.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.713+0000 7efe7d400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe780cd550 con 0x7efe7815e7d0 2024-01-15T18:18:33.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.713+0000 7efe7d400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe780cd270 con 0x7efe780cedc0 2024-01-15T18:18:33.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.713+0000 7efe76800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efe7815e7d0 0x7efe780cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:33.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.713+0000 7efe77200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7efe780ce2e0 0x7efe780cba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38638/0 (socket says 172.21.15.27:38638) 2024-01-15T18:18:33.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.713+0000 7efe77200640 1 -- 172.21.15.27:0/2353882125 learned_addr learned my addr 172.21.15.27:0/2353882125 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:33.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.713+0000 7efe75e00640 1 --2- 172.21.15.27:0/2353882125 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe780cedc0 0x7efe780cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:33.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.713+0000 7efe76800640 1 -- 172.21.15.27:0/2353882125 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe780cedc0 msgr2=0x7efe780cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:33.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.713+0000 7efe76800640 1 --2- 172.21.15.27:0/2353882125 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe780cedc0 0x7efe780cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:33.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.713+0000 7efe76800640 1 -- 172.21.15.27:0/2353882125 >> v1:172.21.15.27:6789/0 conn(0x7efe780ce2e0 legacy=0x7efe780cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:33.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.713+0000 7efe76800640 1 -- 172.21.15.27:0/2353882125 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe780b35d0 con 0x7efe7815e7d0 2024-01-15T18:18:33.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.717+0000 7efe76800640 1 --2- 172.21.15.27:0/2353882125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe7815e7d0 0x7efe780cf2a0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7efe680097c0 tx=0x7efe6801c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d66da09a14e2a6f4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:33.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.717+0000 7efe75400640 1 -- 172.21.15.27:0/2353882125 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe680ab600 con 0x7efe7815e7d0 2024-01-15T18:18:33.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.717+0000 7efe75400640 1 -- 172.21.15.27:0/2353882125 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe680a7050 con 0x7efe7815e7d0 2024-01-15T18:18:33.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.717+0000 7efe75400640 1 -- 172.21.15.27:0/2353882125 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe680ab980 con 0x7efe7815e7d0 2024-01-15T18:18:33.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.717+0000 7efe7d400640 1 -- 172.21.15.27:0/2353882125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe7815e7d0 msgr2=0x7efe780cf2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:33.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.717+0000 7efe7d400640 1 --2- 172.21.15.27:0/2353882125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe7815e7d0 0x7efe780cf2a0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7efe680097c0 tx=0x7efe6801c310 comp rx=0 tx=0).stop 2024-01-15T18:18:33.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.717+0000 7efe7d400640 1 -- 172.21.15.27:0/2353882125 shutdown_connections 2024-01-15T18:18:33.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.717+0000 7efe7d400640 1 --2- 172.21.15.27:0/2353882125 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe780cedc0 0x7efe780cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:33.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.717+0000 7efe7d400640 1 --2- 172.21.15.27:0/2353882125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe7815e7d0 0x7efe780cf2a0 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:33.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.717+0000 7efe7d400640 1 -- 172.21.15.27:0/2353882125 >> 172.21.15.27:0/2353882125 conn(0x7efe780b2a10 msgr2=0x7efe780b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:33.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.717+0000 7efe7d400640 1 -- 172.21.15.27:0/2353882125 shutdown_connections 2024-01-15T18:18:33.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.717+0000 7efe7d400640 1 -- 172.21.15.27:0/2353882125 wait complete. 2024-01-15T18:18:33.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.721+0000 7efe7d400640 1 Processor -- start 2024-01-15T18:18:33.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.721+0000 7efe7d400640 1 -- start start 2024-01-15T18:18:33.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.721+0000 7efe7d400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe780ce2e0 0x7efe7815a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:33.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.721+0000 7efe7d400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efe780cedc0 0x7efe780aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:33.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.721+0000 7efe7d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe7815e7d0 0x7efe780aefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:33.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.721+0000 7efe7d400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe780b4710 con 0x7efe7815e7d0 2024-01-15T18:18:33.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.721+0000 7efe7d400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7815a740 con 0x7efe780cedc0 2024-01-15T18:18:33.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.721+0000 7efe7d400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe780b3a90 con 0x7efe780ce2e0 2024-01-15T18:18:33.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.721+0000 7efe77200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe7815e7d0 0x7efe780aefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:33.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.721+0000 7efe77200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe7815e7d0 0x7efe780aefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38078/0 (socket says 172.21.15.27:38078) 2024-01-15T18:18:33.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.721+0000 7efe77200640 1 -- 172.21.15.27:0/2058457213 learned_addr learned my addr 172.21.15.27:0/2058457213 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:33.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.721+0000 7efe76800640 1 --2- 172.21.15.27:0/2058457213 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe780ce2e0 0x7efe7815a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:33.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.721+0000 7efe75e00640 1 --2- 172.21.15.27:0/2058457213 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efe780cedc0 0x7efe780aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:33.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.721+0000 7efe77200640 1 -- 172.21.15.27:0/2058457213 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe780ce2e0 msgr2=0x7efe7815a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:33.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.721+0000 7efe77200640 1 --2- 172.21.15.27:0/2058457213 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe780ce2e0 0x7efe7815a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:33.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.721+0000 7efe77200640 1 -- 172.21.15.27:0/2058457213 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efe780cedc0 msgr2=0x7efe780aff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:33.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.721+0000 7efe77200640 1 --2- 172.21.15.27:0/2058457213 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efe780cedc0 0x7efe780aff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:33.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.721+0000 7efe77200640 1 -- 172.21.15.27:0/2058457213 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe68009420 con 0x7efe7815e7d0 2024-01-15T18:18:33.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.721+0000 7efe77200640 1 --2- 172.21.15.27:0/2058457213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe7815e7d0 0x7efe780aefa0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7efe7000e7b0 tx=0x7efe7000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:33.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.721+0000 7efe6ea00640 1 -- 172.21.15.27:0/2058457213 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7000f930 con 0x7efe7815e7d0 2024-01-15T18:18:33.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.721+0000 7efe6ea00640 1 -- 172.21.15.27:0/2058457213 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe7000fa90 con 0x7efe7815e7d0 2024-01-15T18:18:33.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.721+0000 7efe6ea00640 1 -- 172.21.15.27:0/2058457213 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7000fe00 con 0x7efe7815e7d0 2024-01-15T18:18:33.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.721+0000 7efe7d400640 1 -- 172.21.15.27:0/2058457213 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe780b3d70 con 0x7efe7815e7d0 2024-01-15T18:18:33.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.725+0000 7efe7d400640 1 -- 172.21.15.27:0/2058457213 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe780b4330 con 0x7efe7815e7d0 2024-01-15T18:18:33.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.725+0000 7efe7d400640 1 -- 172.21.15.27:0/2058457213 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe7804e850 con 0x7efe7815e7d0 2024-01-15T18:18:33.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.725+0000 7efe6ea00640 1 -- 172.21.15.27:0/2058457213 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7efe70015070 con 0x7efe7815e7d0 2024-01-15T18:18:33.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.725+0000 7efe6ea00640 1 --2- 172.21.15.27:0/2058457213 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efe50067940 0x7efe50069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:33.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.725+0000 7efe6ea00640 1 -- 172.21.15.27:0/2058457213 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5821+0+0 (secure 0 0 0) 0x7efe70085b20 con 0x7efe7815e7d0 2024-01-15T18:18:33.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.725+0000 7efe76800640 1 --2- 172.21.15.27:0/2058457213 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efe50067940 0x7efe50069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:33.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.725+0000 7efe76800640 1 --2- 172.21.15.27:0/2058457213 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efe50067940 0x7efe50069e00 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7efe680ad000 tx=0x7efe68008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:33.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.729+0000 7efe6ea00640 1 -- 172.21.15.27:0/2058457213 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7efe70054540 con 0x7efe7815e7d0 2024-01-15T18:18:33.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.885+0000 7efe7d400640 1 -- 172.21.15.27:0/2058457213 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe7806eb50 con 0x7efe7815e7d0 2024-01-15T18:18:33.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.889+0000 7efe6ea00640 1 -- 172.21.15.27:0/2058457213 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16621 (secure 0 0 0) 0x7efe70057b60 con 0x7efe7815e7d0 2024-01-15T18:18:33.887 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:18:33.887 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":107,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:18:32.593006+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:18:30.993882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-15T18:18:31.582649+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:18:33.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.889+0000 7efe7d400640 1 -- 172.21.15.27:0/2058457213 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efe50067940 msgr2=0x7efe50069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:33.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.889+0000 7efe7d400640 1 --2- 172.21.15.27:0/2058457213 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efe50067940 0x7efe50069e00 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7efe680ad000 tx=0x7efe68008040 comp rx=0 tx=0).stop 2024-01-15T18:18:33.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.889+0000 7efe7d400640 1 -- 172.21.15.27:0/2058457213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe7815e7d0 msgr2=0x7efe780aefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:33.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.889+0000 7efe7d400640 1 --2- 172.21.15.27:0/2058457213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe7815e7d0 0x7efe780aefa0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7efe7000e7b0 tx=0x7efe7000ec70 comp rx=0 tx=0).stop 2024-01-15T18:18:33.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.889+0000 7efe7d400640 1 -- 172.21.15.27:0/2058457213 shutdown_connections 2024-01-15T18:18:33.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.889+0000 7efe7d400640 1 --2- 172.21.15.27:0/2058457213 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efe50067940 0x7efe50069e00 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:33.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.889+0000 7efe7d400640 1 --2- 172.21.15.27:0/2058457213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe7815e7d0 0x7efe780aefa0 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:33.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.889+0000 7efe7d400640 1 --2- 172.21.15.27:0/2058457213 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efe780cedc0 0x7efe780aff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:33.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.889+0000 7efe7d400640 1 --2- 172.21.15.27:0/2058457213 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe780ce2e0 0x7efe7815a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:33.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.889+0000 7efe7d400640 1 -- 172.21.15.27:0/2058457213 >> 172.21.15.27:0/2058457213 conn(0x7efe780b2a10 msgr2=0x7efe7806eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:33.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.889+0000 7efe7d400640 1 -- 172.21.15.27:0/2058457213 shutdown_connections 2024-01-15T18:18:33.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:33.889+0000 7efe7d400640 1 -- 172.21.15.27:0/2058457213 wait complete. 2024-01-15T18:18:33.909 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-15T18:18:34.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.053+0000 7f0bd9600640 1 Processor -- start 2024-01-15T18:18:34.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.053+0000 7f0bd9600640 1 -- start start 2024-01-15T18:18:34.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.053+0000 7f0bd9600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0bd415b100 0x7f0bd4070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:34.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.053+0000 7f0bd9600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd4071860 con 0x7f0bd4071420 2024-01-15T18:18:34.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.053+0000 7f0bd9600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd415a610 con 0x7f0bd415ad20 2024-01-15T18:18:34.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.053+0000 7f0bd9600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bd415e3c0 con 0x7f0bd415b100 2024-01-15T18:18:34.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.057+0000 7f0bd3400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0bd4071420 0x7f0bd415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38642/0 (socket says 172.21.15.27:38642) 2024-01-15T18:18:34.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.057+0000 7f0bd3400640 1 -- 172.21.15.27:0/3354636190 learned_addr learned my addr 172.21.15.27:0/3354636190 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:34.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.057+0000 7f0bd2000640 1 --2- 172.21.15.27:0/3354636190 >> v2:172.21.15.93:3300/0 conn(0x7f0bd415b100 0x7f0bd4070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.057+0000 7f0bd1600640 1 -- 172.21.15.27:0/3354636190 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 818892006 0 0) 0x7f0bd4071860 con 0x7f0bd4071420 2024-01-15T18:18:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.057+0000 7f0bd1600640 1 -- 172.21.15.27:0/3354636190 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bb4003530 con 0x7f0bd4071420 2024-01-15T18:18:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.057+0000 7f0bd2000640 1 -- 172.21.15.27:0/3354636190 >> v1:172.21.15.93:6789/0 conn(0x7f0bd415ad20 legacy=0x7f0bd415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.057+0000 7f0bd2000640 1 -- 172.21.15.27:0/3354636190 >> v1:172.21.15.27:6789/0 conn(0x7f0bd4071420 legacy=0x7f0bd415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:34.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.057+0000 7f0bd2000640 1 -- 172.21.15.27:0/3354636190 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd415e6a0 con 0x7f0bd415b100 2024-01-15T18:18:34.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.057+0000 7f0bd2000640 1 --2- 172.21.15.27:0/3354636190 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0bd415b100 0x7f0bd4070ee0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc0097f0 tx=0x7f0bbc01c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=826f6ef3b9fefe39 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:34.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.057+0000 7f0bd1600640 1 -- 172.21.15.27:0/3354636190 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc0ac620 con 0x7f0bd415b100 2024-01-15T18:18:34.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.057+0000 7f0bd1600640 1 -- 172.21.15.27:0/3354636190 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bbc0ac780 con 0x7f0bd415b100 2024-01-15T18:18:34.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.057+0000 7f0bd1600640 1 -- 172.21.15.27:0/3354636190 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc0acaf0 con 0x7f0bd415b100 2024-01-15T18:18:34.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.061+0000 7f0bd9600640 1 -- 172.21.15.27:0/3354636190 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0bd415b100 msgr2=0x7f0bd4070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:34.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.061+0000 7f0bd9600640 1 --2- 172.21.15.27:0/3354636190 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0bd415b100 0x7f0bd4070ee0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc0097f0 tx=0x7f0bbc01c3c0 comp rx=0 tx=0).stop 2024-01-15T18:18:34.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.061+0000 7f0bd9600640 1 -- 172.21.15.27:0/3354636190 shutdown_connections 2024-01-15T18:18:34.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.061+0000 7f0bd9600640 1 --2- 172.21.15.27:0/3354636190 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0bd415b100 0x7f0bd4070ee0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:34.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.061+0000 7f0bd9600640 1 -- 172.21.15.27:0/3354636190 >> 172.21.15.27:0/3354636190 conn(0x7f0bd40cb700 msgr2=0x7f0bd40cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:34.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.061+0000 7f0bd9600640 1 -- 172.21.15.27:0/3354636190 shutdown_connections 2024-01-15T18:18:34.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.061+0000 7f0bd9600640 1 -- 172.21.15.27:0/3354636190 wait complete. 2024-01-15T18:18:34.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.061+0000 7f0bd9600640 1 Processor -- start 2024-01-15T18:18:34.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.061+0000 7f0bd9600640 1 -- start start 2024-01-15T18:18:34.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.061+0000 7f0bd9600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bd4071420 0x7f0bd413c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:34.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.061+0000 7f0bd9600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0bd415ad20 0x7f0bd414c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:34.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.061+0000 7f0bd9600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0bd415b100 0x7f0bd4155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:34.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.061+0000 7f0bd9600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd414d2d0 con 0x7f0bd4071420 2024-01-15T18:18:34.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.061+0000 7f0bd9600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd414bba0 con 0x7f0bd415b100 2024-01-15T18:18:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.061+0000 7f0bd9600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd414b300 con 0x7f0bd415ad20 2024-01-15T18:18:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.061+0000 7f0bd2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bd4071420 0x7f0bd413c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.061+0000 7f0bd2a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bd4071420 0x7f0bd413c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38084/0 (socket says 172.21.15.27:38084) 2024-01-15T18:18:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.061+0000 7f0bd2a00640 1 -- 172.21.15.27:0/1307270373 learned_addr learned my addr 172.21.15.27:0/1307270373 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.061+0000 7f0bd2000640 1 --2- 172.21.15.27:0/1307270373 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0bd415ad20 0x7f0bd414c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.065+0000 7f0bd3400640 1 --2- 172.21.15.27:0/1307270373 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0bd415b100 0x7f0bd4155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.065+0000 7f0bd2a00640 1 -- 172.21.15.27:0/1307270373 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0bd415ad20 msgr2=0x7f0bd414c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.065+0000 7f0bd2a00640 1 --2- 172.21.15.27:0/1307270373 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0bd415ad20 0x7f0bd414c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.065+0000 7f0bd2a00640 1 -- 172.21.15.27:0/1307270373 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0bd415b100 msgr2=0x7f0bd4155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.065+0000 7f0bd2a00640 1 --2- 172.21.15.27:0/1307270373 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0bd415b100 0x7f0bd4155190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:34.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.065+0000 7f0bd2a00640 1 -- 172.21.15.27:0/1307270373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bbc009420 con 0x7f0bd4071420 2024-01-15T18:18:34.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.065+0000 7f0bd2a00640 1 --2- 172.21.15.27:0/1307270373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bd4071420 0x7f0bd413c7c0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400c530 tx=0x7f0bc400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:34.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.065+0000 7f0bcaa00640 1 -- 172.21.15.27:0/1307270373 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4005220 con 0x7f0bd4071420 2024-01-15T18:18:34.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.065+0000 7f0bcaa00640 1 -- 172.21.15.27:0/1307270373 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc4005380 con 0x7f0bd4071420 2024-01-15T18:18:34.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.065+0000 7f0bcaa00640 1 -- 172.21.15.27:0/1307270373 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4007270 con 0x7f0bd4071420 2024-01-15T18:18:34.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.065+0000 7f0bd9600640 1 -- 172.21.15.27:0/1307270373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd414b5e0 con 0x7f0bd4071420 2024-01-15T18:18:34.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.065+0000 7f0bd9600640 1 -- 172.21.15.27:0/1307270373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd414cc40 con 0x7f0bd4071420 2024-01-15T18:18:34.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.065+0000 7f0bd9600640 1 -- 172.21.15.27:0/1307270373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd414d030 con 0x7f0bd4071420 2024-01-15T18:18:34.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.065+0000 7f0bcaa00640 1 -- 172.21.15.27:0/1307270373 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f0bc40073d0 con 0x7f0bd4071420 2024-01-15T18:18:34.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.065+0000 7f0bcaa00640 1 --2- 172.21.15.27:0/1307270373 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0bac067940 0x7f0bac069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:34.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.069+0000 7f0bcaa00640 1 -- 172.21.15.27:0/1307270373 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f0bc4082a40 con 0x7f0bd4071420 2024-01-15T18:18:34.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.069+0000 7f0bd2000640 1 --2- 172.21.15.27:0/1307270373 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0bac067940 0x7f0bac069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:34.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.069+0000 7f0bd2000640 1 --2- 172.21.15.27:0/1307270373 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0bac067940 0x7f0bac069e00 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc0ae000 tx=0x7f0bbc0a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:34.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.069+0000 7f0bcaa00640 1 -- 172.21.15.27:0/1307270373 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0bd414d030 con 0x7f0bd4071420 2024-01-15T18:18:34.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:34.261+0000 7f0bd9600640 1 -- 172.21.15.27:0/1307270373 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f0bd414d030 con 0x7f0bd4071420 2024-01-15T18:18:35.171 INFO:tasks.ceph.mds.d.smithi093.stderr:2024-01-15T18:18:35.175+0000 7f5edfa00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-15T18:18:35.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.217+0000 7f0bcaa00640 1 -- 172.21.15.27:0/1307270373 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v76) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0bd414d030 con 0x7f0bd4071420 2024-01-15T18:18:35.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.221+0000 7f0bd9600640 1 -- 172.21.15.27:0/1307270373 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0bac067940 msgr2=0x7f0bac069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:35.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.221+0000 7f0bd9600640 1 --2- 172.21.15.27:0/1307270373 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0bac067940 0x7f0bac069e00 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc0ae000 tx=0x7f0bbc0a7000 comp rx=0 tx=0).stop 2024-01-15T18:18:35.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.221+0000 7f0bd9600640 1 -- 172.21.15.27:0/1307270373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bd4071420 msgr2=0x7f0bd413c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:35.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.221+0000 7f0bd9600640 1 --2- 172.21.15.27:0/1307270373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bd4071420 0x7f0bd413c7c0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400c530 tx=0x7f0bc400c9f0 comp rx=0 tx=0).stop 2024-01-15T18:18:35.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.221+0000 7f0bd9600640 1 -- 172.21.15.27:0/1307270373 shutdown_connections 2024-01-15T18:18:35.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.221+0000 7f0bd9600640 1 --2- 172.21.15.27:0/1307270373 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0bac067940 0x7f0bac069e00 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:35.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.221+0000 7f0bd9600640 1 --2- 172.21.15.27:0/1307270373 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0bd415b100 0x7f0bd4155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:35.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.221+0000 7f0bd9600640 1 --2- 172.21.15.27:0/1307270373 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0bd415ad20 0x7f0bd414c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:35.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.221+0000 7f0bd9600640 1 --2- 172.21.15.27:0/1307270373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bd4071420 0x7f0bd413c7c0 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:35.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.221+0000 7f0bd9600640 1 -- 172.21.15.27:0/1307270373 >> 172.21.15.27:0/1307270373 conn(0x7f0bd40cb700 msgr2=0x7f0bd4157ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:35.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.221+0000 7f0bd9600640 1 -- 172.21.15.27:0/1307270373 shutdown_connections 2024-01-15T18:18:35.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.221+0000 7f0bd9600640 1 -- 172.21.15.27:0/1307270373 wait complete. 2024-01-15T18:18:35.244 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-15T18:18:35.245 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-15T18:18:35.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.389+0000 7fe914c00640 1 Processor -- start 2024-01-15T18:18:35.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.389+0000 7fe914c00640 1 -- start start 2024-01-15T18:18:35.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.393+0000 7fe914c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fe91015f550 0x7fe9100cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:35.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.393+0000 7fe914c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9100cf110 0x7fe9100701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:35.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.393+0000 7fe914c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9100b1cb0 con 0x7fe910158900 2024-01-15T18:18:35.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.393+0000 7fe914c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9100cf580 con 0x7fe91015f550 2024-01-15T18:18:35.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.393+0000 7fe914c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe91015fa00 con 0x7fe9100cf110 2024-01-15T18:18:35.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.393+0000 7fe90d400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fe91015f550 0x7fe9100cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:35.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.393+0000 7fe90e800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9100cf110 0x7fe9100701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:35.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.393+0000 7fe90e800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9100cf110 0x7fe9100701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:45252/0 (socket says 172.21.15.27:45252) 2024-01-15T18:18:35.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.393+0000 7fe90e800640 1 -- 172.21.15.27:0/1921133000 learned_addr learned my addr 172.21.15.27:0/1921133000 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:35.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.393+0000 7fe90d400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fe91015f550 0x7fe9100cebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:37318/0 (socket says 172.21.15.27:37318) 2024-01-15T18:18:35.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.393+0000 7fe90e800640 1 -- 172.21.15.27:0/1921133000 >> v2:172.21.15.93:3300/0 conn(0x7fe91015f550 msgr2=0x7fe9100cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:35.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.393+0000 7fe90e800640 1 --2- 172.21.15.27:0/1921133000 >> v2:172.21.15.93:3300/0 conn(0x7fe91015f550 0x7fe9100cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:35.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.393+0000 7fe90e800640 1 -- 172.21.15.27:0/1921133000 >> v1:172.21.15.93:6789/0 conn(0x7fe910158900 legacy=0x7fe91015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:35.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.393+0000 7fe90e800640 1 -- 172.21.15.27:0/1921133000 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9100b1f00 con 0x7fe9100cf110 2024-01-15T18:18:35.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.397+0000 7fe90e800640 1 --2- 172.21.15.27:0/1921133000 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9100cf110 0x7fe9100701e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe908009e50 tx=0x7fe90801c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb9afbc129a3ea20 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:35.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.397+0000 7fe90ca00640 1 -- 172.21.15.27:0/1921133000 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe910070f30 con 0x7fe9100cf110 2024-01-15T18:18:35.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.397+0000 7fe90ca00640 1 -- 172.21.15.27:0/1921133000 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9080a58a0 con 0x7fe9100cf110 2024-01-15T18:18:35.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.397+0000 7fe90ca00640 1 -- 172.21.15.27:0/1921133000 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9080a5c30 con 0x7fe9100cf110 2024-01-15T18:18:35.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.397+0000 7fe914c00640 1 -- 172.21.15.27:0/1921133000 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9100cf110 msgr2=0x7fe9100701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:35.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.397+0000 7fe914c00640 1 --2- 172.21.15.27:0/1921133000 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9100cf110 0x7fe9100701e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe908009e50 tx=0x7fe90801c9e0 comp rx=0 tx=0).stop 2024-01-15T18:18:35.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.397+0000 7fe914c00640 1 -- 172.21.15.27:0/1921133000 shutdown_connections 2024-01-15T18:18:35.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.397+0000 7fe914c00640 1 --2- 172.21.15.27:0/1921133000 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe9100cf110 0x7fe9100701e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:35.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.397+0000 7fe914c00640 1 --2- 172.21.15.27:0/1921133000 >> v2:172.21.15.93:3300/0 conn(0x7fe91015f550 0x7fe9100cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:35.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.397+0000 7fe914c00640 1 -- 172.21.15.27:0/1921133000 >> 172.21.15.27:0/1921133000 conn(0x7fe9100cb700 msgr2=0x7fe9100cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:35.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.397+0000 7fe914c00640 1 -- 172.21.15.27:0/1921133000 shutdown_connections 2024-01-15T18:18:35.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.397+0000 7fe914c00640 1 -- 172.21.15.27:0/1921133000 wait complete. 2024-01-15T18:18:35.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.397+0000 7fe914c00640 1 Processor -- start 2024-01-15T18:18:35.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.397+0000 7fe914c00640 1 -- start start 2024-01-15T18:18:35.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.401+0000 7fe914c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe9100cf110 0x7fe9101565e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:35.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.401+0000 7fe914c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe910158900 0x7fe9101500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:35.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.401+0000 7fe914c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe91015f550 0x7fe91015bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:35.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.401+0000 7fe914c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9101512f0 con 0x7fe91015f550 2024-01-15T18:18:35.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.401+0000 7fe914c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe910150c20 con 0x7fe9100cf110 2024-01-15T18:18:35.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.401+0000 7fe914c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe91015b080 con 0x7fe910158900 2024-01-15T18:18:35.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.401+0000 7fe90e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe91015f550 0x7fe91015bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:35.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.401+0000 7fe90e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe91015f550 0x7fe91015bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38086/0 (socket says 172.21.15.27:38086) 2024-01-15T18:18:35.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.401+0000 7fe90e800640 1 -- 172.21.15.27:0/2480930984 learned_addr learned my addr 172.21.15.27:0/2480930984 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:35.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.401+0000 7fe90de00640 1 --2- 172.21.15.27:0/2480930984 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe9100cf110 0x7fe9101565e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:35.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.401+0000 7fe90d400640 1 --2- 172.21.15.27:0/2480930984 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe910158900 0x7fe9101500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:35.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.401+0000 7fe90e800640 1 -- 172.21.15.27:0/2480930984 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe910158900 msgr2=0x7fe9101500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:35.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.401+0000 7fe90e800640 1 --2- 172.21.15.27:0/2480930984 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe910158900 0x7fe9101500d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:35.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.401+0000 7fe90e800640 1 -- 172.21.15.27:0/2480930984 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe9100cf110 msgr2=0x7fe9101565e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:35.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.401+0000 7fe90e800640 1 --2- 172.21.15.27:0/2480930984 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe9100cf110 0x7fe9101565e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:35.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.401+0000 7fe90e800640 1 -- 172.21.15.27:0/2480930984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe908009b30 con 0x7fe91015f550 2024-01-15T18:18:35.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.401+0000 7fe90e800640 1 --2- 172.21.15.27:0/2480930984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe91015f550 0x7fe91015bcf0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7fe908009e50 tx=0x7fe908019780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:35.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.401+0000 7fe906000640 1 -- 172.21.15.27:0/2480930984 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9080b8670 con 0x7fe91015f550 2024-01-15T18:18:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.401+0000 7fe906000640 1 -- 172.21.15.27:0/2480930984 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9080b87d0 con 0x7fe91015f550 2024-01-15T18:18:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.401+0000 7fe906000640 1 -- 172.21.15.27:0/2480930984 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9080b8b80 con 0x7fe91015f550 2024-01-15T18:18:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.401+0000 7fe914c00640 1 -- 172.21.15.27:0/2480930984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe910151540 con 0x7fe91015f550 2024-01-15T18:18:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.401+0000 7fe914c00640 1 -- 172.21.15.27:0/2480930984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe910151b00 con 0x7fe91015f550 2024-01-15T18:18:35.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.401+0000 7fe914c00640 1 -- 172.21.15.27:0/2480930984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe910151cf0 con 0x7fe91015f550 2024-01-15T18:18:35.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.405+0000 7fe906000640 1 -- 172.21.15.27:0/2480930984 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7fe9080b8ce0 con 0x7fe91015f550 2024-01-15T18:18:35.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.405+0000 7fe906000640 1 --2- 172.21.15.27:0/2480930984 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe8e8067940 0x7fe8e8069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:35.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.405+0000 7fe906000640 1 -- 172.21.15.27:0/2480930984 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fe90811e790 con 0x7fe91015f550 2024-01-15T18:18:35.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.405+0000 7fe90de00640 1 --2- 172.21.15.27:0/2480930984 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe8e8067940 0x7fe8e8069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:35.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.405+0000 7fe90de00640 1 --2- 172.21.15.27:0/2480930984 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe8e8067940 0x7fe8e8069e00 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fe900007fd0 tx=0x7fe900009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:35.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.409+0000 7fe906000640 1 -- 172.21.15.27:0/2480930984 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe9080ed2d0 con 0x7fe91015f550 2024-01-15T18:18:35.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:35.601+0000 7fe914c00640 1 -- 172.21.15.27:0/2480930984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe91004e850 con 0x7fe91015f550 2024-01-15T18:18:36.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.221+0000 7fe906000640 1 -- 172.21.15.27:0/2480930984 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v77) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe9080bc0c0 con 0x7fe91015f550 2024-01-15T18:18:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.221+0000 7fe914c00640 1 -- 172.21.15.27:0/2480930984 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe8e8067940 msgr2=0x7fe8e8069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.221+0000 7fe914c00640 1 --2- 172.21.15.27:0/2480930984 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe8e8067940 0x7fe8e8069e00 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fe900007fd0 tx=0x7fe900009040 comp rx=0 tx=0).stop 2024-01-15T18:18:36.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.225+0000 7fe914c00640 1 -- 172.21.15.27:0/2480930984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe91015f550 msgr2=0x7fe91015bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.225+0000 7fe914c00640 1 --2- 172.21.15.27:0/2480930984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe91015f550 0x7fe91015bcf0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7fe908009e50 tx=0x7fe908019780 comp rx=0 tx=0).stop 2024-01-15T18:18:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.225+0000 7fe914c00640 1 -- 172.21.15.27:0/2480930984 shutdown_connections 2024-01-15T18:18:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.225+0000 7fe914c00640 1 --2- 172.21.15.27:0/2480930984 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe8e8067940 0x7fe8e8069e00 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.225+0000 7fe914c00640 1 --2- 172.21.15.27:0/2480930984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe91015f550 0x7fe91015bcf0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.225+0000 7fe914c00640 1 --2- 172.21.15.27:0/2480930984 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe910158900 0x7fe9101500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.225+0000 7fe914c00640 1 --2- 172.21.15.27:0/2480930984 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe9100cf110 0x7fe9101565e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.225+0000 7fe914c00640 1 -- 172.21.15.27:0/2480930984 >> 172.21.15.27:0/2480930984 conn(0x7fe9100cb700 msgr2=0x7fe9100b4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.225+0000 7fe914c00640 1 -- 172.21.15.27:0/2480930984 shutdown_connections 2024-01-15T18:18:36.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.225+0000 7fe914c00640 1 -- 172.21.15.27:0/2480930984 wait complete. 2024-01-15T18:18:36.245 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-15T18:18:36.245 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:18:36.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.389+0000 7f8a50200640 1 Processor -- start 2024-01-15T18:18:36.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.389+0000 7f8a50200640 1 -- start start 2024-01-15T18:18:36.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.389+0000 7f8a50200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8a4806a4f0 0x7f8a480d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:36.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.389+0000 7f8a50200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8a4815b470 0x7f8a4815b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:36.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.389+0000 7f8a50200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a4806ece0 con 0x7f8a480ca3f0 2024-01-15T18:18:36.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.389+0000 7f8a50200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a4815be40 con 0x7f8a4806a4f0 2024-01-15T18:18:36.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.389+0000 7f8a50200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4806e6a0 con 0x7f8a4815b470 2024-01-15T18:18:36.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.389+0000 7f8a4d800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8a4806a4f0 0x7f8a480d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:36.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.389+0000 7f8a4ce00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8a480ca3f0 0x7f8a4806a8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38658/0 (socket says 172.21.15.27:38658) 2024-01-15T18:18:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.389+0000 7f8a4ce00640 1 -- 172.21.15.27:0/1683143018 learned_addr learned my addr 172.21.15.27:0/1683143018 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.389+0000 7f8a4e200640 1 --2- 172.21.15.27:0/1683143018 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8a4815b470 0x7f8a4815b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.389+0000 7f8a4d800640 1 -- 172.21.15.27:0/1683143018 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8a4815b470 msgr2=0x7f8a4815b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.389+0000 7f8a4d800640 1 --2- 172.21.15.27:0/1683143018 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8a4815b470 0x7f8a4815b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.389+0000 7f8a4d800640 1 -- 172.21.15.27:0/1683143018 >> v1:172.21.15.27:6789/0 conn(0x7f8a480ca3f0 legacy=0x7f8a4806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.389+0000 7f8a4d800640 1 -- 172.21.15.27:0/1683143018 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4806e940 con 0x7f8a4806a4f0 2024-01-15T18:18:36.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.393+0000 7f8a4d800640 1 --2- 172.21.15.27:0/1683143018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a4806a4f0 0x7f8a480d08c0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f8a380097c0 tx=0x7f8a3801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=553374f2f5b9853c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:36.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.393+0000 7f8a47e00640 1 -- 172.21.15.27:0/1683143018 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a380ab600 con 0x7f8a4806a4f0 2024-01-15T18:18:36.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.393+0000 7f8a47e00640 1 -- 172.21.15.27:0/1683143018 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a380a7050 con 0x7f8a4806a4f0 2024-01-15T18:18:36.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.393+0000 7f8a47e00640 1 -- 172.21.15.27:0/1683143018 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a380ab980 con 0x7f8a4806a4f0 2024-01-15T18:18:36.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.393+0000 7f8a50200640 1 -- 172.21.15.27:0/1683143018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a4806a4f0 msgr2=0x7f8a480d08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:36.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.393+0000 7f8a50200640 1 --2- 172.21.15.27:0/1683143018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a4806a4f0 0x7f8a480d08c0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f8a380097c0 tx=0x7f8a3801c3f0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.393+0000 7f8a50200640 1 -- 172.21.15.27:0/1683143018 shutdown_connections 2024-01-15T18:18:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.393+0000 7f8a50200640 1 --2- 172.21.15.27:0/1683143018 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8a4815b470 0x7f8a4815b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.393+0000 7f8a50200640 1 --2- 172.21.15.27:0/1683143018 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a4806a4f0 0x7f8a480d08c0 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.393+0000 7f8a50200640 1 -- 172.21.15.27:0/1683143018 >> 172.21.15.27:0/1683143018 conn(0x7f8a480cfda0 msgr2=0x7f8a480d01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.393+0000 7f8a50200640 1 -- 172.21.15.27:0/1683143018 shutdown_connections 2024-01-15T18:18:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.393+0000 7f8a50200640 1 -- 172.21.15.27:0/1683143018 wait complete. 2024-01-15T18:18:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.397+0000 7f8a50200640 1 Processor -- start 2024-01-15T18:18:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.397+0000 7f8a50200640 1 -- start start 2024-01-15T18:18:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.397+0000 7f8a50200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8a4806a4f0 0x7f8a480cd970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.397+0000 7f8a50200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8a480ca3f0 0x7f8a481f9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.397+0000 7f8a50200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a4815b470 0x7f8a481fa1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.397+0000 7f8a50200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a480cee80 con 0x7f8a4815b470 2024-01-15T18:18:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.397+0000 7f8a50200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a480cf310 con 0x7f8a480ca3f0 2024-01-15T18:18:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.397+0000 7f8a50200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a480ce270 con 0x7f8a4806a4f0 2024-01-15T18:18:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.397+0000 7f8a4e200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a4815b470 0x7f8a481fa1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:36.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.397+0000 7f8a4e200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a4815b470 0x7f8a481fa1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38108/0 (socket says 172.21.15.27:38108) 2024-01-15T18:18:36.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.397+0000 7f8a4e200640 1 -- 172.21.15.27:0/1815602410 learned_addr learned my addr 172.21.15.27:0/1815602410 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:36.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.397+0000 7f8a4d800640 1 --2- 172.21.15.27:0/1815602410 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8a4806a4f0 0x7f8a480cd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:36.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.397+0000 7f8a4ce00640 1 --2- 172.21.15.27:0/1815602410 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8a480ca3f0 0x7f8a481f9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:36.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.397+0000 7f8a4e200640 1 -- 172.21.15.27:0/1815602410 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8a4806a4f0 msgr2=0x7f8a480cd970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:36.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.397+0000 7f8a4e200640 1 --2- 172.21.15.27:0/1815602410 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8a4806a4f0 0x7f8a480cd970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.397+0000 7f8a4e200640 1 -- 172.21.15.27:0/1815602410 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8a480ca3f0 msgr2=0x7f8a481f9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:36.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.397+0000 7f8a4e200640 1 --2- 172.21.15.27:0/1815602410 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8a480ca3f0 0x7f8a481f9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.397+0000 7f8a4e200640 1 -- 172.21.15.27:0/1815602410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a38009420 con 0x7f8a4815b470 2024-01-15T18:18:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.397+0000 7f8a4e200640 1 --2- 172.21.15.27:0/1815602410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a4815b470 0x7f8a481fa1e0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f8a40007e60 tx=0x7f8a4000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.397+0000 7f8a45600640 1 -- 172.21.15.27:0/1815602410 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a40014420 con 0x7f8a4815b470 2024-01-15T18:18:36.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.397+0000 7f8a45600640 1 -- 172.21.15.27:0/1815602410 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a400145a0 con 0x7f8a4815b470 2024-01-15T18:18:36.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.397+0000 7f8a50200640 1 -- 172.21.15.27:0/1815602410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a480ce550 con 0x7f8a4815b470 2024-01-15T18:18:36.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.397+0000 7f8a45600640 1 -- 172.21.15.27:0/1815602410 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a40014950 con 0x7f8a4815b470 2024-01-15T18:18:36.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.397+0000 7f8a50200640 1 -- 172.21.15.27:0/1815602410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a480ceb10 con 0x7f8a4815b470 2024-01-15T18:18:36.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.397+0000 7f8a50200640 1 -- 172.21.15.27:0/1815602410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a480ceff0 con 0x7f8a4815b470 2024-01-15T18:18:36.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.401+0000 7f8a45600640 1 -- 172.21.15.27:0/1815602410 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f8a40014ab0 con 0x7f8a4815b470 2024-01-15T18:18:36.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.401+0000 7f8a45600640 1 --2- 172.21.15.27:0/1815602410 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8a20067940 0x7f8a20069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:36.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.401+0000 7f8a45600640 1 -- 172.21.15.27:0/1815602410 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f8a40083aa0 con 0x7f8a4815b470 2024-01-15T18:18:36.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.401+0000 7f8a4d800640 1 --2- 172.21.15.27:0/1815602410 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8a20067940 0x7f8a20069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:36.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.401+0000 7f8a4d800640 1 --2- 172.21.15.27:0/1815602410 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8a20067940 0x7f8a20069e00 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f8a380ad000 tx=0x7f8a380a4f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:36.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.405+0000 7f8a45600640 1 -- 172.21.15.27:0/1815602410 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8a400524c0 con 0x7f8a4815b470 2024-01-15T18:18:36.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.593+0000 7f8a50200640 1 -- 172.21.15.27:0/1815602410 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a4804e850 con 0x7f8a4815b470 2024-01-15T18:18:36.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.593+0000 7f8a45600640 1 -- 172.21.15.27:0/1815602410 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f8a40055ae0 con 0x7f8a4815b470 2024-01-15T18:18:36.595 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:18:36.595 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":77,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7721,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3036261071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":3036261071},{"type":"v1","addr":"172.21.15.27:6835","nonce":3036261071}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72},{"gid":7724,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/3646217660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":3646217660},{"type":"v1","addr":"172.21.15.93:6836","nonce":3646217660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72},{"gid":9441,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3131124481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3131124481},{"type":"v1","addr":"172.21.15.27:6834","nonce":3131124481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:18:32.593319+0000","modified":"2024-01-15T18:18:35.606888+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9450},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9450":{"gid":9450,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.93:6837/2040168756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":2040168756},{"type":"v1","addr":"172.21.15.93:6837","nonce":2040168756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-01-15T18:18:36.595 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 77 2024-01-15T18:18:36.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.597+0000 7f8a50200640 1 -- 172.21.15.27:0/1815602410 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8a20067940 msgr2=0x7f8a20069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:36.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.597+0000 7f8a50200640 1 --2- 172.21.15.27:0/1815602410 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8a20067940 0x7f8a20069e00 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f8a380ad000 tx=0x7f8a380a4f90 comp rx=0 tx=0).stop 2024-01-15T18:18:36.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.597+0000 7f8a50200640 1 -- 172.21.15.27:0/1815602410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a4815b470 msgr2=0x7f8a481fa1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:36.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.597+0000 7f8a50200640 1 --2- 172.21.15.27:0/1815602410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a4815b470 0x7f8a481fa1e0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f8a40007e60 tx=0x7f8a4000e4b0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.597+0000 7f8a50200640 1 -- 172.21.15.27:0/1815602410 shutdown_connections 2024-01-15T18:18:36.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.597+0000 7f8a50200640 1 --2- 172.21.15.27:0/1815602410 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8a20067940 0x7f8a20069e00 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.597+0000 7f8a50200640 1 --2- 172.21.15.27:0/1815602410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a4815b470 0x7f8a481fa1e0 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.597+0000 7f8a50200640 1 --2- 172.21.15.27:0/1815602410 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8a480ca3f0 0x7f8a481f9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.597+0000 7f8a50200640 1 --2- 172.21.15.27:0/1815602410 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8a4806a4f0 0x7f8a480cd970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.597+0000 7f8a50200640 1 -- 172.21.15.27:0/1815602410 >> 172.21.15.27:0/1815602410 conn(0x7f8a480cfda0 msgr2=0x7f8a480ad5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:36.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.597+0000 7f8a50200640 1 -- 172.21.15.27:0/1815602410 shutdown_connections 2024-01-15T18:18:36.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.597+0000 7f8a50200640 1 -- 172.21.15.27:0/1815602410 wait complete. 2024-01-15T18:18:36.618 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:18:36.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.757+0000 7f6e72a00640 1 Processor -- start 2024-01-15T18:18:36.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.757+0000 7f6e72a00640 1 -- start start 2024-01-15T18:18:36.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.761+0000 7f6e72a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6e6c0afe70 0x7f6e6c0cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:36.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.761+0000 7f6e72a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6e6c0cb7b0 0x7f6e6c0cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:36.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.761+0000 7f6e72a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6c0d05b0 con 0x7f6e6c160b10 2024-01-15T18:18:36.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.761+0000 7f6e72a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e6c0ce7d0 con 0x7f6e6c0cb7b0 2024-01-15T18:18:36.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.761+0000 7f6e72a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c0cfc80 con 0x7f6e6c0afe70 2024-01-15T18:18:36.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.761+0000 7f6e6be00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6e6c160b10 0x7f6e6c0cd7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38670/0 (socket says 172.21.15.27:38670) 2024-01-15T18:18:36.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.761+0000 7f6e6be00640 1 -- 172.21.15.27:0/1399758639 learned_addr learned my addr 172.21.15.27:0/1399758639 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:36.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.761+0000 7f6e70a00640 1 --2- 172.21.15.27:0/1399758639 >> v2:172.21.15.27:3300/0 conn(0x7f6e6c0cb7b0 0x7f6e6c0cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:36.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.761+0000 7f6e6b400640 1 --2- 172.21.15.27:0/1399758639 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6e6c0afe70 0x7f6e6c0cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:36.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.761+0000 7f6e70a00640 1 -- 172.21.15.27:0/1399758639 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6e6c0afe70 msgr2=0x7f6e6c0cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:36.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.761+0000 7f6e70a00640 1 --2- 172.21.15.27:0/1399758639 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6e6c0afe70 0x7f6e6c0cdeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.761+0000 7f6e70a00640 1 -- 172.21.15.27:0/1399758639 >> v1:172.21.15.27:6789/0 conn(0x7f6e6c160b10 legacy=0x7f6e6c0cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:36.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.761+0000 7f6e70a00640 1 -- 172.21.15.27:0/1399758639 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6c0cff60 con 0x7f6e6c0cb7b0 2024-01-15T18:18:36.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.761+0000 7f6e6aa00640 1 -- 172.21.15.27:0/1399758639 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4175594953 0 0) 0x7f6e6c0d05b0 con 0x7f6e6c160b10 2024-01-15T18:18:36.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.765+0000 7f6e70a00640 1 --2- 172.21.15.27:0/1399758639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e6c0cb7b0 0x7f6e6c0cbc40 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f6e64009d90 tx=0x7f6e6401c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=294ada833d74e085 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:36.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.765+0000 7f6e6aa00640 1 -- 172.21.15.27:0/1399758639 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e640a5860 con 0x7f6e6c0cb7b0 2024-01-15T18:18:36.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.765+0000 7f6e6aa00640 1 -- 172.21.15.27:0/1399758639 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e640a59c0 con 0x7f6e6c0cb7b0 2024-01-15T18:18:36.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.765+0000 7f6e6aa00640 1 -- 172.21.15.27:0/1399758639 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e640a5d50 con 0x7f6e6c0cb7b0 2024-01-15T18:18:36.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.765+0000 7f6e72a00640 1 -- 172.21.15.27:0/1399758639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e6c0cb7b0 msgr2=0x7f6e6c0cbc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:36.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.765+0000 7f6e72a00640 1 --2- 172.21.15.27:0/1399758639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e6c0cb7b0 0x7f6e6c0cbc40 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f6e64009d90 tx=0x7f6e6401c890 comp rx=0 tx=0).stop 2024-01-15T18:18:36.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.765+0000 7f6e72a00640 1 -- 172.21.15.27:0/1399758639 shutdown_connections 2024-01-15T18:18:36.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.765+0000 7f6e72a00640 1 --2- 172.21.15.27:0/1399758639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e6c0cb7b0 0x7f6e6c0cbc40 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.765+0000 7f6e72a00640 1 --2- 172.21.15.27:0/1399758639 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6e6c0afe70 0x7f6e6c0cdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.765+0000 7f6e72a00640 1 -- 172.21.15.27:0/1399758639 >> 172.21.15.27:0/1399758639 conn(0x7f6e6c0d0f20 msgr2=0x7f6e6c0d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:36.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.765+0000 7f6e72a00640 1 -- 172.21.15.27:0/1399758639 shutdown_connections 2024-01-15T18:18:36.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.765+0000 7f6e72a00640 1 -- 172.21.15.27:0/1399758639 wait complete. 2024-01-15T18:18:36.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.765+0000 7f6e72a00640 1 Processor -- start 2024-01-15T18:18:36.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.765+0000 7f6e72a00640 1 -- start start 2024-01-15T18:18:36.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.769+0000 7f6e72a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e6c0afe70 0x7f6e6c151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:36.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.769+0000 7f6e72a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6e6c0cb7b0 0x7f6e6c0cb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.769+0000 7f6e72a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6e6c160b10 0x7f6e6c06ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.769+0000 7f6e72a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c0c9130 con 0x7f6e6c0afe70 2024-01-15T18:18:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.769+0000 7f6e72a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c158c00 con 0x7f6e6c0cb7b0 2024-01-15T18:18:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.769+0000 7f6e72a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6c09e6d0 con 0x7f6e6c160b10 2024-01-15T18:18:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.769+0000 7f6e6be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e6c0afe70 0x7f6e6c151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.769+0000 7f6e6be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e6c0afe70 0x7f6e6c151280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38116/0 (socket says 172.21.15.27:38116) 2024-01-15T18:18:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.769+0000 7f6e6be00640 1 -- 172.21.15.27:0/2469735564 learned_addr learned my addr 172.21.15.27:0/2469735564 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.769+0000 7f6e70a00640 1 --2- 172.21.15.27:0/2469735564 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6e6c160b10 0x7f6e6c06ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.769+0000 7f6e6b400640 1 --2- 172.21.15.27:0/2469735564 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6e6c0cb7b0 0x7f6e6c0cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.769+0000 7f6e6be00640 1 -- 172.21.15.27:0/2469735564 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6e6c160b10 msgr2=0x7f6e6c06ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.769+0000 7f6e6be00640 1 --2- 172.21.15.27:0/2469735564 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6e6c160b10 0x7f6e6c06ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.769+0000 7f6e6be00640 1 -- 172.21.15.27:0/2469735564 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6e6c0cb7b0 msgr2=0x7f6e6c0cb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.769+0000 7f6e6be00640 1 --2- 172.21.15.27:0/2469735564 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6e6c0cb7b0 0x7f6e6c0cb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.769+0000 7f6e6be00640 1 -- 172.21.15.27:0/2469735564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e64009a70 con 0x7f6e6c0afe70 2024-01-15T18:18:36.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.769+0000 7f6e6be00640 1 --2- 172.21.15.27:0/2469735564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e6c0afe70 0x7f6e6c151280 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c00c480 tx=0x7f6e5c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:36.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.769+0000 7f6e63e00640 1 -- 172.21.15.27:0/2469735564 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c0074e0 con 0x7f6e6c0afe70 2024-01-15T18:18:36.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.769+0000 7f6e63e00640 1 -- 172.21.15.27:0/2469735564 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5c007660 con 0x7f6e6c0afe70 2024-01-15T18:18:36.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.769+0000 7f6e72a00640 1 -- 172.21.15.27:0/2469735564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e6c09e9b0 con 0x7f6e6c0afe70 2024-01-15T18:18:36.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.769+0000 7f6e63e00640 1 -- 172.21.15.27:0/2469735564 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c004a30 con 0x7f6e6c0afe70 2024-01-15T18:18:36.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.769+0000 7f6e72a00640 1 -- 172.21.15.27:0/2469735564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e6c1593c0 con 0x7f6e6c0afe70 2024-01-15T18:18:36.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.769+0000 7f6e72a00640 1 -- 172.21.15.27:0/2469735564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e6c04e850 con 0x7f6e6c0afe70 2024-01-15T18:18:36.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.769+0000 7f6e63e00640 1 -- 172.21.15.27:0/2469735564 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f6e5c004b90 con 0x7f6e6c0afe70 2024-01-15T18:18:36.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.773+0000 7f6e63e00640 1 --2- 172.21.15.27:0/2469735564 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6e44067940 0x7f6e44069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:36.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.773+0000 7f6e63e00640 1 -- 172.21.15.27:0/2469735564 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f6e5c0824c0 con 0x7f6e6c0afe70 2024-01-15T18:18:36.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.773+0000 7f6e6b400640 1 --2- 172.21.15.27:0/2469735564 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6e44067940 0x7f6e44069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:36.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.773+0000 7f6e6b400640 1 --2- 172.21.15.27:0/2469735564 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6e44067940 0x7f6e44069e00 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c0cfa60 tx=0x7f6e54008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:36.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.777+0000 7f6e63e00640 1 -- 172.21.15.27:0/2469735564 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6e5c050e60 con 0x7f6e6c0afe70 2024-01-15T18:18:36.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.933+0000 7f6e72a00640 1 -- 172.21.15.27:0/2469735564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e6c15ad40 con 0x7f6e6c0afe70 2024-01-15T18:18:36.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.933+0000 7f6e63e00640 1 -- 172.21.15.27:0/2469735564 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16608 (secure 0 0 0) 0x7f6e5c018020 con 0x7f6e6c0afe70 2024-01-15T18:18:36.933 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:18:36.934 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":108,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:18:34.138239+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:18:30.993882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-15T18:18:31.582649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:18:36.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.937+0000 7f6e72a00640 1 -- 172.21.15.27:0/2469735564 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6e44067940 msgr2=0x7f6e44069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:36.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.937+0000 7f6e72a00640 1 --2- 172.21.15.27:0/2469735564 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6e44067940 0x7f6e44069e00 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c0cfa60 tx=0x7f6e54008040 comp rx=0 tx=0).stop 2024-01-15T18:18:36.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.937+0000 7f6e72a00640 1 -- 172.21.15.27:0/2469735564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e6c0afe70 msgr2=0x7f6e6c151280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:36.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.937+0000 7f6e72a00640 1 --2- 172.21.15.27:0/2469735564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e6c0afe70 0x7f6e6c151280 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c00c480 tx=0x7f6e5c00c940 comp rx=0 tx=0).stop 2024-01-15T18:18:36.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.937+0000 7f6e72a00640 1 -- 172.21.15.27:0/2469735564 shutdown_connections 2024-01-15T18:18:36.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.937+0000 7f6e72a00640 1 --2- 172.21.15.27:0/2469735564 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6e44067940 0x7f6e44069e00 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.937+0000 7f6e72a00640 1 --2- 172.21.15.27:0/2469735564 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6e6c160b10 0x7f6e6c06ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.937+0000 7f6e72a00640 1 --2- 172.21.15.27:0/2469735564 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6e6c0cb7b0 0x7f6e6c0cb000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.937+0000 7f6e72a00640 1 --2- 172.21.15.27:0/2469735564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e6c0afe70 0x7f6e6c151280 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.937+0000 7f6e72a00640 1 -- 172.21.15.27:0/2469735564 >> 172.21.15.27:0/2469735564 conn(0x7f6e6c0d0f20 msgr2=0x7f6e6c06e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:36.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.937+0000 7f6e72a00640 1 -- 172.21.15.27:0/2469735564 shutdown_connections 2024-01-15T18:18:36.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:36.937+0000 7f6e72a00640 1 -- 172.21.15.27:0/2469735564 wait complete. 2024-01-15T18:18:36.956 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-15T18:18:36.956 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:18:37.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.097+0000 7f0343e00640 1 Processor -- start 2024-01-15T18:18:37.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.097+0000 7f0343e00640 1 -- start start 2024-01-15T18:18:37.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.097+0000 7f0343e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f033c0cf9b0 0x7f033c0b10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:37.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.097+0000 7f0343e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f033c0b38b0 con 0x7f033c15e7d0 2024-01-15T18:18:37.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.097+0000 7f0343e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f033c0cd550 con 0x7f033c0cedc0 2024-01-15T18:18:37.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.097+0000 7f0343e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f033c0cd270 con 0x7f033c0cf9b0 2024-01-15T18:18:37.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.097+0000 7f0341400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f033c15e7d0 0x7f033c0cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38686/0 (socket says 172.21.15.27:38686) 2024-01-15T18:18:37.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.097+0000 7f0341400640 1 -- 172.21.15.27:0/1338539142 learned_addr learned my addr 172.21.15.27:0/1338539142 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:37.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.097+0000 7f0340a00640 1 --2- 172.21.15.27:0/1338539142 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f033c0cf9b0 0x7f033c0b10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:37.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.097+0000 7f033be00640 1 -- 172.21.15.27:0/1338539142 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765201334 0 0) 0x7f033c0b38b0 con 0x7f033c15e7d0 2024-01-15T18:18:37.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.097+0000 7f033be00640 1 -- 172.21.15.27:0/1338539142 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f031c003530 con 0x7f033c15e7d0 2024-01-15T18:18:37.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.097+0000 7f033be00640 1 -- 172.21.15.27:0/1338539142 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298052129 0 0) 0x7f033c0cd550 con 0x7f033c0cedc0 2024-01-15T18:18:37.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.097+0000 7f033be00640 1 -- 172.21.15.27:0/1338539142 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f033c0b38b0 con 0x7f033c0cedc0 2024-01-15T18:18:37.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.097+0000 7f0340a00640 1 -- 172.21.15.27:0/1338539142 >> v1:172.21.15.93:6789/0 conn(0x7f033c0cedc0 legacy=0x7f033c0b1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:37.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.097+0000 7f0340a00640 1 -- 172.21.15.27:0/1338539142 >> v1:172.21.15.27:6789/0 conn(0x7f033c15e7d0 legacy=0x7f033c0cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:37.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.097+0000 7f0340a00640 1 -- 172.21.15.27:0/1338539142 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f033c0b35d0 con 0x7f033c0cf9b0 2024-01-15T18:18:37.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.101+0000 7f0340a00640 1 --2- 172.21.15.27:0/1338539142 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f033c0cf9b0 0x7f033c0b10c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f03240097f0 tx=0x7f033c0b2520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e70183b48cd92ec4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:37.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.101+0000 7f033be00640 1 -- 172.21.15.27:0/1338539142 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03240ac410 con 0x7f033c0cf9b0 2024-01-15T18:18:37.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.101+0000 7f033be00640 1 -- 172.21.15.27:0/1338539142 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03240ac570 con 0x7f033c0cf9b0 2024-01-15T18:18:37.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.101+0000 7f033be00640 1 -- 172.21.15.27:0/1338539142 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03240ac900 con 0x7f033c0cf9b0 2024-01-15T18:18:37.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.101+0000 7f0343e00640 1 -- 172.21.15.27:0/1338539142 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f033c0cf9b0 msgr2=0x7f033c0b10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:37.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.101+0000 7f0343e00640 1 --2- 172.21.15.27:0/1338539142 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f033c0cf9b0 0x7f033c0b10c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f03240097f0 tx=0x7f033c0b2520 comp rx=0 tx=0).stop 2024-01-15T18:18:37.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.105+0000 7f0343e00640 1 -- 172.21.15.27:0/1338539142 shutdown_connections 2024-01-15T18:18:37.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.105+0000 7f0343e00640 1 --2- 172.21.15.27:0/1338539142 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f033c0cf9b0 0x7f033c0b10c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:37.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.105+0000 7f0343e00640 1 -- 172.21.15.27:0/1338539142 >> 172.21.15.27:0/1338539142 conn(0x7f033c0b2a10 msgr2=0x7f033c0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:37.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.105+0000 7f0343e00640 1 -- 172.21.15.27:0/1338539142 shutdown_connections 2024-01-15T18:18:37.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.105+0000 7f0343e00640 1 -- 172.21.15.27:0/1338539142 wait complete. 2024-01-15T18:18:37.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.105+0000 7f0343e00640 1 Processor -- start 2024-01-15T18:18:37.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.105+0000 7f0343e00640 1 -- start start 2024-01-15T18:18:37.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.105+0000 7f0343e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f033c0cedc0 0x7f033c0b6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:37.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.105+0000 7f0343e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f033c0cf9b0 0x7f033c0c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:37.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.105+0000 7f0343e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f033c15e7d0 0x7f033c06cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:37.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.105+0000 7f0343e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033c1553b0 con 0x7f033c15e7d0 2024-01-15T18:18:37.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.105+0000 7f0343e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033c0c7f90 con 0x7f033c0cf9b0 2024-01-15T18:18:37.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.105+0000 7f0343e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f033c0c81e0 con 0x7f033c0cedc0 2024-01-15T18:18:37.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.105+0000 7f0341e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f033c15e7d0 0x7f033c06cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:37.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.105+0000 7f0341e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f033c15e7d0 0x7f033c06cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38122/0 (socket says 172.21.15.27:38122) 2024-01-15T18:18:37.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.105+0000 7f0341e00640 1 -- 172.21.15.27:0/4093035144 learned_addr learned my addr 172.21.15.27:0/4093035144 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:37.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.105+0000 7f0340a00640 1 --2- 172.21.15.27:0/4093035144 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f033c0cf9b0 0x7f033c0c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:37.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.105+0000 7f0341400640 1 --2- 172.21.15.27:0/4093035144 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f033c0cedc0 0x7f033c0b6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:37.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.105+0000 7f0341e00640 1 -- 172.21.15.27:0/4093035144 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f033c0cedc0 msgr2=0x7f033c0b6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:37.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.105+0000 7f0341e00640 1 --2- 172.21.15.27:0/4093035144 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f033c0cedc0 0x7f033c0b6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:37.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.105+0000 7f0341e00640 1 -- 172.21.15.27:0/4093035144 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f033c0cf9b0 msgr2=0x7f033c0c75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:37.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.105+0000 7f0341e00640 1 --2- 172.21.15.27:0/4093035144 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f033c0cf9b0 0x7f033c0c75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:37.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.105+0000 7f0341e00640 1 -- 172.21.15.27:0/4093035144 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0324009420 con 0x7f033c15e7d0 2024-01-15T18:18:37.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.109+0000 7f0341e00640 1 --2- 172.21.15.27:0/4093035144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f033c15e7d0 0x7f033c06cce0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f033400c730 tx=0x7f033400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:37.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.109+0000 7f0339600640 1 -- 172.21.15.27:0/4093035144 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0334003b80 con 0x7f033c15e7d0 2024-01-15T18:18:37.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.109+0000 7f0339600640 1 -- 172.21.15.27:0/4093035144 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0334003ce0 con 0x7f033c15e7d0 2024-01-15T18:18:37.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.109+0000 7f0339600640 1 -- 172.21.15.27:0/4093035144 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033400aa90 con 0x7f033c15e7d0 2024-01-15T18:18:37.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.109+0000 7f0343e00640 1 -- 172.21.15.27:0/4093035144 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f033c0c84c0 con 0x7f033c15e7d0 2024-01-15T18:18:37.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.109+0000 7f0343e00640 1 -- 172.21.15.27:0/4093035144 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f033c0c8a80 con 0x7f033c15e7d0 2024-01-15T18:18:37.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.109+0000 7f0343e00640 1 -- 172.21.15.27:0/4093035144 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f033c0b4430 con 0x7f033c15e7d0 2024-01-15T18:18:37.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.109+0000 7f0339600640 1 -- 172.21.15.27:0/4093035144 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f033400abf0 con 0x7f033c15e7d0 2024-01-15T18:18:37.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.109+0000 7f0339600640 1 --2- 172.21.15.27:0/4093035144 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0314067940 0x7f0314069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:37.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.109+0000 7f0339600640 1 -- 172.21.15.27:0/4093035144 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f0334082fa0 con 0x7f033c15e7d0 2024-01-15T18:18:37.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.109+0000 7f0341400640 1 --2- 172.21.15.27:0/4093035144 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0314067940 0x7f0314069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:37.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.113+0000 7f0341400640 1 --2- 172.21.15.27:0/4093035144 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0314067940 0x7f0314069e00 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f032c0011f0 tx=0x7f032c007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:37.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.113+0000 7f0339600640 1 -- 172.21.15.27:0/4093035144 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f033c0b4430 con 0x7f033c15e7d0 2024-01-15T18:18:37.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.269+0000 7f0343e00640 1 -- 172.21.15.27:0/4093035144 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f033c0b4430 con 0x7f0314067940 2024-01-15T18:18:37.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.277+0000 7f0339600640 1 -- 172.21.15.27:0/4093035144 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289387 (secure 0 0 0) 0x7f033c0b4430 con 0x7f0314067940 2024-01-15T18:18:37.282 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:18:37.284 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:18:37.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.281+0000 7f0343e00640 1 -- 172.21.15.27:0/4093035144 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0314067940 msgr2=0x7f0314069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:37.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.281+0000 7f0343e00640 1 --2- 172.21.15.27:0/4093035144 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0314067940 0x7f0314069e00 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f032c0011f0 tx=0x7f032c007d40 comp rx=0 tx=0).stop 2024-01-15T18:18:37.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.281+0000 7f0343e00640 1 -- 172.21.15.27:0/4093035144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f033c15e7d0 msgr2=0x7f033c06cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:37.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.281+0000 7f0343e00640 1 --2- 172.21.15.27:0/4093035144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f033c15e7d0 0x7f033c06cce0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f033400c730 tx=0x7f033400cbf0 comp rx=0 tx=0).stop 2024-01-15T18:18:37.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.281+0000 7f0343e00640 1 -- 172.21.15.27:0/4093035144 shutdown_connections 2024-01-15T18:18:37.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.281+0000 7f0343e00640 1 --2- 172.21.15.27:0/4093035144 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0314067940 0x7f0314069e00 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.281+0000 7f0343e00640 1 --2- 172.21.15.27:0/4093035144 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f033c15e7d0 0x7f033c06cce0 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.281+0000 7f0343e00640 1 --2- 172.21.15.27:0/4093035144 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f033c0cf9b0 0x7f033c0c75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.281+0000 7f0343e00640 1 --2- 172.21.15.27:0/4093035144 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f033c0cedc0 0x7f033c0b6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.281+0000 7f0343e00640 1 -- 172.21.15.27:0/4093035144 >> 172.21.15.27:0/4093035144 conn(0x7f033c0b2a10 msgr2=0x7f033c15cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.281+0000 7f0343e00640 1 -- 172.21.15.27:0/4093035144 shutdown_connections 2024-01-15T18:18:37.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.281+0000 7f0343e00640 1 -- 172.21.15.27:0/4093035144 wait complete. 2024-01-15T18:18:37.303 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2024-01-15T18:18:36.373471+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":88,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":518912,"kb_used_data":11632,"kb_used_omap":12,"kb_used_meta":507123,"kb_avail":754455808,"statfs":{"total":773094113280,"available":772562747392,"internally_reserved":0,"allocated":11911168,"data_stored":5830580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":519294512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.186043"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337389+0000","last_change":"2024-01-15T18:18:33.168792+0000","last_active":"2024-01-15T18:18:34.337389+0000","last_peered":"2024-01-15T18:18:34.337389+0000","last_clean":"2024-01-15T18:18:34.337389+0000","last_became_active":"2024-01-15T18:18:33.168500+0000","last_became_peered":"2024-01-15T18:18:33.168500+0000","last_unstale":"2024-01-15T18:18:34.337389+0000","last_undegraded":"2024-01-15T18:18:34.337389+0000","last_fullsized":"2024-01-15T18:18:34.337389+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:28:41.940111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.144832+0000","last_change":"2024-01-15T18:18:33.151630+0000","last_active":"2024-01-15T18:18:34.144832+0000","last_peered":"2024-01-15T18:18:34.144832+0000","last_clean":"2024-01-15T18:18:34.144832+0000","last_became_active":"2024-01-15T18:18:33.151209+0000","last_became_peered":"2024-01-15T18:18:33.151209+0000","last_unstale":"2024-01-15T18:18:34.144832+0000","last_undegraded":"2024-01-15T18:18:34.144832+0000","last_fullsized":"2024-01-15T18:18:34.144832+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:46:54.687765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336528+0000","last_change":"2024-01-15T18:18:33.163094+0000","last_active":"2024-01-15T18:18:34.336528+0000","last_peered":"2024-01-15T18:18:34.336528+0000","last_clean":"2024-01-15T18:18:34.336528+0000","last_became_active":"2024-01-15T18:18:33.162700+0000","last_became_peered":"2024-01-15T18:18:33.162700+0000","last_unstale":"2024-01-15T18:18:34.336528+0000","last_undegraded":"2024-01-15T18:18:34.336528+0000","last_fullsized":"2024-01-15T18:18:34.336528+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:23:36.167165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337116+0000","last_change":"2024-01-15T18:18:33.160786+0000","last_active":"2024-01-15T18:18:34.337116+0000","last_peered":"2024-01-15T18:18:34.337116+0000","last_clean":"2024-01-15T18:18:34.337116+0000","last_became_active":"2024-01-15T18:18:33.160460+0000","last_became_peered":"2024-01-15T18:18:33.160460+0000","last_unstale":"2024-01-15T18:18:34.337116+0000","last_undegraded":"2024-01-15T18:18:34.337116+0000","last_fullsized":"2024-01-15T18:18:34.337116+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:27:33.980071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337976+0000","last_change":"2024-01-15T18:18:33.155936+0000","last_active":"2024-01-15T18:18:34.337976+0000","last_peered":"2024-01-15T18:18:34.337976+0000","last_clean":"2024-01-15T18:18:34.337976+0000","last_became_active":"2024-01-15T18:18:33.155656+0000","last_became_peered":"2024-01-15T18:18:33.155656+0000","last_unstale":"2024-01-15T18:18:34.337976+0000","last_undegraded":"2024-01-15T18:18:34.337976+0000","last_fullsized":"2024-01-15T18:18:34.337976+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:27:09.469255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145214+0000","last_change":"2024-01-15T18:18:33.152032+0000","last_active":"2024-01-15T18:18:34.145214+0000","last_peered":"2024-01-15T18:18:34.145214+0000","last_clean":"2024-01-15T18:18:34.145214+0000","last_became_active":"2024-01-15T18:18:33.151701+0000","last_became_peered":"2024-01-15T18:18:33.151701+0000","last_unstale":"2024-01-15T18:18:34.145214+0000","last_undegraded":"2024-01-15T18:18:34.145214+0000","last_fullsized":"2024-01-15T18:18:34.145214+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:51.983410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145175+0000","last_change":"2024-01-15T18:18:33.153882+0000","last_active":"2024-01-15T18:18:34.145175+0000","last_peered":"2024-01-15T18:18:34.145175+0000","last_clean":"2024-01-15T18:18:34.145175+0000","last_became_active":"2024-01-15T18:18:33.153264+0000","last_became_peered":"2024-01-15T18:18:33.153264+0000","last_unstale":"2024-01-15T18:18:34.145175+0000","last_undegraded":"2024-01-15T18:18:34.145175+0000","last_fullsized":"2024-01-15T18:18:34.145175+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:37:49.663025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.335705+0000","last_change":"2024-01-15T18:18:33.164714+0000","last_active":"2024-01-15T18:18:34.335705+0000","last_peered":"2024-01-15T18:18:34.335705+0000","last_clean":"2024-01-15T18:18:34.335705+0000","last_became_active":"2024-01-15T18:18:33.163896+0000","last_became_peered":"2024-01-15T18:18:33.163896+0000","last_unstale":"2024-01-15T18:18:34.335705+0000","last_undegraded":"2024-01-15T18:18:34.335705+0000","last_fullsized":"2024-01-15T18:18:34.335705+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:21:18.264946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337455+0000","last_change":"2024-01-15T18:18:33.158682+0000","last_active":"2024-01-15T18:18:34.337455+0000","last_peered":"2024-01-15T18:18:34.337455+0000","last_clean":"2024-01-15T18:18:34.337455+0000","last_became_active":"2024-01-15T18:18:33.158399+0000","last_became_peered":"2024-01-15T18:18:33.158399+0000","last_unstale":"2024-01-15T18:18:34.337455+0000","last_undegraded":"2024-01-15T18:18:34.337455+0000","last_fullsized":"2024-01-15T18:18:34.337455+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:27:12.032383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337118+0000","last_change":"2024-01-15T18:18:33.159936+0000","last_active":"2024-01-15T18:18:34.337118+0000","last_peered":"2024-01-15T18:18:34.337118+0000","last_clean":"2024-01-15T18:18:34.337118+0000","last_became_active":"2024-01-15T18:18:33.159562+0000","last_became_peered":"2024-01-15T18:18:33.159562+0000","last_unstale":"2024-01-15T18:18:34.337118+0000","last_undegraded":"2024-01-15T18:18:34.337118+0000","last_fullsized":"2024-01-15T18:18:34.337118+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:56:32.966770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.144621+0000","last_change":"2024-01-15T18:18:33.155247+0000","last_active":"2024-01-15T18:18:34.144621+0000","last_peered":"2024-01-15T18:18:34.144621+0000","last_clean":"2024-01-15T18:18:34.144621+0000","last_became_active":"2024-01-15T18:18:33.154972+0000","last_became_peered":"2024-01-15T18:18:33.154972+0000","last_unstale":"2024-01-15T18:18:34.144621+0000","last_undegraded":"2024-01-15T18:18:34.144621+0000","last_fullsized":"2024-01-15T18:18:34.144621+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:54:47.923576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337159+0000","last_change":"2024-01-15T18:18:33.158336+0000","last_active":"2024-01-15T18:18:34.337159+0000","last_peered":"2024-01-15T18:18:34.337159+0000","last_clean":"2024-01-15T18:18:34.337159+0000","last_became_active":"2024-01-15T18:18:33.158064+0000","last_became_peered":"2024-01-15T18:18:33.158064+0000","last_unstale":"2024-01-15T18:18:34.337159+0000","last_undegraded":"2024-01-15T18:18:34.337159+0000","last_fullsized":"2024-01-15T18:18:34.337159+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:42:44.182465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336372+0000","last_change":"2024-01-15T18:18:33.163912+0000","last_active":"2024-01-15T18:18:34.336372+0000","last_peered":"2024-01-15T18:18:34.336372+0000","last_clean":"2024-01-15T18:18:34.336372+0000","last_became_active":"2024-01-15T18:18:33.163610+0000","last_became_peered":"2024-01-15T18:18:33.163610+0000","last_unstale":"2024-01-15T18:18:34.336372+0000","last_undegraded":"2024-01-15T18:18:34.336372+0000","last_fullsized":"2024-01-15T18:18:34.336372+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:32:43.468374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336137+0000","last_change":"2024-01-15T18:18:33.169868+0000","last_active":"2024-01-15T18:18:34.336137+0000","last_peered":"2024-01-15T18:18:34.336137+0000","last_clean":"2024-01-15T18:18:34.336137+0000","last_became_active":"2024-01-15T18:18:33.169530+0000","last_became_peered":"2024-01-15T18:18:33.169530+0000","last_unstale":"2024-01-15T18:18:34.336137+0000","last_undegraded":"2024-01-15T18:18:34.336137+0000","last_fullsized":"2024-01-15T18:18:34.336137+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:18:22.473056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145111+0000","last_change":"2024-01-15T18:18:33.151325+0000","last_active":"2024-01-15T18:18:34.145111+0000","last_peered":"2024-01-15T18:18:34.145111+0000","last_clean":"2024-01-15T18:18:34.145111+0000","last_became_active":"2024-01-15T18:18:33.151040+0000","last_became_peered":"2024-01-15T18:18:33.151040+0000","last_unstale":"2024-01-15T18:18:34.145111+0000","last_undegraded":"2024-01-15T18:18:34.145111+0000","last_fullsized":"2024-01-15T18:18:34.145111+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:57:08.165322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336593+0000","last_change":"2024-01-15T18:18:33.166327+0000","last_active":"2024-01-15T18:18:34.336593+0000","last_peered":"2024-01-15T18:18:34.336593+0000","last_clean":"2024-01-15T18:18:34.336593+0000","last_became_active":"2024-01-15T18:18:33.166014+0000","last_became_peered":"2024-01-15T18:18:33.166014+0000","last_unstale":"2024-01-15T18:18:34.336593+0000","last_undegraded":"2024-01-15T18:18:34.336593+0000","last_fullsized":"2024-01-15T18:18:34.336593+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:37:50.428714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145535+0000","last_change":"2024-01-15T18:18:33.168613+0000","last_active":"2024-01-15T18:18:34.145535+0000","last_peered":"2024-01-15T18:18:34.145535+0000","last_clean":"2024-01-15T18:18:34.145535+0000","last_became_active":"2024-01-15T18:18:33.167975+0000","last_became_peered":"2024-01-15T18:18:33.167975+0000","last_unstale":"2024-01-15T18:18:34.145535+0000","last_undegraded":"2024-01-15T18:18:34.145535+0000","last_fullsized":"2024-01-15T18:18:34.145535+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:22:13.319815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.335745+0000","last_change":"2024-01-15T18:18:33.155078+0000","last_active":"2024-01-15T18:18:34.335745+0000","last_peered":"2024-01-15T18:18:34.335745+0000","last_clean":"2024-01-15T18:18:34.335745+0000","last_became_active":"2024-01-15T18:18:33.154816+0000","last_became_peered":"2024-01-15T18:18:33.154816+0000","last_unstale":"2024-01-15T18:18:34.335745+0000","last_undegraded":"2024-01-15T18:18:34.335745+0000","last_fullsized":"2024-01-15T18:18:34.335745+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:48.739168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336626+0000","last_change":"2024-01-15T18:18:33.164028+0000","last_active":"2024-01-15T18:18:34.336626+0000","last_peered":"2024-01-15T18:18:34.336626+0000","last_clean":"2024-01-15T18:18:34.336626+0000","last_became_active":"2024-01-15T18:18:33.163421+0000","last_became_peered":"2024-01-15T18:18:33.163421+0000","last_unstale":"2024-01-15T18:18:34.336626+0000","last_undegraded":"2024-01-15T18:18:34.336626+0000","last_fullsized":"2024-01-15T18:18:34.336626+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:38:31.615856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.144690+0000","last_change":"2024-01-15T18:18:33.160995+0000","last_active":"2024-01-15T18:18:34.144690+0000","last_peered":"2024-01-15T18:18:34.144690+0000","last_clean":"2024-01-15T18:18:34.144690+0000","last_became_active":"2024-01-15T18:18:33.160047+0000","last_became_peered":"2024-01-15T18:18:33.160047+0000","last_unstale":"2024-01-15T18:18:34.144690+0000","last_undegraded":"2024-01-15T18:18:34.144690+0000","last_fullsized":"2024-01-15T18:18:34.144690+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:39:53.520191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.144701+0000","last_change":"2024-01-15T18:18:33.152696+0000","last_active":"2024-01-15T18:18:34.144701+0000","last_peered":"2024-01-15T18:18:34.144701+0000","last_clean":"2024-01-15T18:18:34.144701+0000","last_became_active":"2024-01-15T18:18:33.152128+0000","last_became_peered":"2024-01-15T18:18:33.152128+0000","last_unstale":"2024-01-15T18:18:34.144701+0000","last_undegraded":"2024-01-15T18:18:34.144701+0000","last_fullsized":"2024-01-15T18:18:34.144701+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:08:18.706229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336224+0000","last_change":"2024-01-15T18:18:33.163614+0000","last_active":"2024-01-15T18:18:34.336224+0000","last_peered":"2024-01-15T18:18:34.336224+0000","last_clean":"2024-01-15T18:18:34.336224+0000","last_became_active":"2024-01-15T18:18:33.163252+0000","last_became_peered":"2024-01-15T18:18:33.163252+0000","last_unstale":"2024-01-15T18:18:34.336224+0000","last_undegraded":"2024-01-15T18:18:34.336224+0000","last_fullsized":"2024-01-15T18:18:34.336224+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:48.739168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.144172+0000","last_change":"2024-01-15T18:18:33.159896+0000","last_active":"2024-01-15T18:18:34.144172+0000","last_peered":"2024-01-15T18:18:34.144172+0000","last_clean":"2024-01-15T18:18:34.144172+0000","last_became_active":"2024-01-15T18:18:33.159557+0000","last_became_peered":"2024-01-15T18:18:33.159557+0000","last_unstale":"2024-01-15T18:18:34.144172+0000","last_undegraded":"2024-01-15T18:18:34.144172+0000","last_fullsized":"2024-01-15T18:18:34.144172+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:24.034308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336118+0000","last_change":"2024-01-15T18:18:33.166766+0000","last_active":"2024-01-15T18:18:34.336118+0000","last_peered":"2024-01-15T18:18:34.336118+0000","last_clean":"2024-01-15T18:18:34.336118+0000","last_became_active":"2024-01-15T18:18:33.165600+0000","last_became_peered":"2024-01-15T18:18:33.165600+0000","last_unstale":"2024-01-15T18:18:34.336118+0000","last_undegraded":"2024-01-15T18:18:34.336118+0000","last_fullsized":"2024-01-15T18:18:34.336118+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:03.948102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145680+0000","last_change":"2024-01-15T18:18:33.168720+0000","last_active":"2024-01-15T18:18:34.145680+0000","last_peered":"2024-01-15T18:18:34.145680+0000","last_clean":"2024-01-15T18:18:34.145680+0000","last_became_active":"2024-01-15T18:18:33.168349+0000","last_became_peered":"2024-01-15T18:18:33.168349+0000","last_unstale":"2024-01-15T18:18:34.145680+0000","last_undegraded":"2024-01-15T18:18:34.145680+0000","last_fullsized":"2024-01-15T18:18:34.145680+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:18:22.473056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337215+0000","last_change":"2024-01-15T18:18:33.159262+0000","last_active":"2024-01-15T18:18:34.337215+0000","last_peered":"2024-01-15T18:18:34.337215+0000","last_clean":"2024-01-15T18:18:34.337215+0000","last_became_active":"2024-01-15T18:18:33.158946+0000","last_became_peered":"2024-01-15T18:18:33.158946+0000","last_unstale":"2024-01-15T18:18:34.337215+0000","last_undegraded":"2024-01-15T18:18:34.337215+0000","last_fullsized":"2024-01-15T18:18:34.337215+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:05:55.347898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145408+0000","last_change":"2024-01-15T18:18:33.161814+0000","last_active":"2024-01-15T18:18:34.145408+0000","last_peered":"2024-01-15T18:18:34.145408+0000","last_clean":"2024-01-15T18:18:34.145408+0000","last_became_active":"2024-01-15T18:18:33.161463+0000","last_became_peered":"2024-01-15T18:18:33.161463+0000","last_unstale":"2024-01-15T18:18:34.145408+0000","last_undegraded":"2024-01-15T18:18:34.145408+0000","last_fullsized":"2024-01-15T18:18:34.145408+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:15:36.391855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337021+0000","last_change":"2024-01-15T18:18:33.162952+0000","last_active":"2024-01-15T18:18:34.337021+0000","last_peered":"2024-01-15T18:18:34.337021+0000","last_clean":"2024-01-15T18:18:34.337021+0000","last_became_active":"2024-01-15T18:18:33.162669+0000","last_became_peered":"2024-01-15T18:18:33.162669+0000","last_unstale":"2024-01-15T18:18:34.337021+0000","last_undegraded":"2024-01-15T18:18:34.337021+0000","last_fullsized":"2024-01-15T18:18:34.337021+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:47:59.088407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.144253+0000","last_change":"2024-01-15T18:18:33.160658+0000","last_active":"2024-01-15T18:18:34.144253+0000","last_peered":"2024-01-15T18:18:34.144253+0000","last_clean":"2024-01-15T18:18:34.144253+0000","last_became_active":"2024-01-15T18:18:33.160099+0000","last_became_peered":"2024-01-15T18:18:33.160099+0000","last_unstale":"2024-01-15T18:18:34.144253+0000","last_undegraded":"2024-01-15T18:18:34.144253+0000","last_fullsized":"2024-01-15T18:18:34.144253+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:47:59.088407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145009+0000","last_change":"2024-01-15T18:18:33.161061+0000","last_active":"2024-01-15T18:18:34.145009+0000","last_peered":"2024-01-15T18:18:34.145009+0000","last_clean":"2024-01-15T18:18:34.145009+0000","last_became_active":"2024-01-15T18:18:33.160725+0000","last_became_peered":"2024-01-15T18:18:33.160725+0000","last_unstale":"2024-01-15T18:18:34.145009+0000","last_undegraded":"2024-01-15T18:18:34.145009+0000","last_fullsized":"2024-01-15T18:18:34.145009+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:38:07.236956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336723+0000","last_change":"2024-01-15T18:18:33.159024+0000","last_active":"2024-01-15T18:18:34.336723+0000","last_peered":"2024-01-15T18:18:34.336723+0000","last_clean":"2024-01-15T18:18:34.336723+0000","last_became_active":"2024-01-15T18:18:33.158701+0000","last_became_peered":"2024-01-15T18:18:33.158701+0000","last_unstale":"2024-01-15T18:18:34.336723+0000","last_undegraded":"2024-01-15T18:18:34.336723+0000","last_fullsized":"2024-01-15T18:18:34.336723+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:50:16.189329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145092+0000","last_change":"2024-01-15T18:18:33.152241+0000","last_active":"2024-01-15T18:18:34.145092+0000","last_peered":"2024-01-15T18:18:34.145092+0000","last_clean":"2024-01-15T18:18:34.145092+0000","last_became_active":"2024-01-15T18:18:33.151250+0000","last_became_peered":"2024-01-15T18:18:33.151250+0000","last_unstale":"2024-01-15T18:18:34.145092+0000","last_undegraded":"2024-01-15T18:18:34.145092+0000","last_fullsized":"2024-01-15T18:18:34.145092+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:45:02.525155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145574+0000","last_change":"2024-01-15T18:18:33.154972+0000","last_active":"2024-01-15T18:18:34.145574+0000","last_peered":"2024-01-15T18:18:34.145574+0000","last_clean":"2024-01-15T18:18:34.145574+0000","last_became_active":"2024-01-15T18:18:33.154569+0000","last_became_peered":"2024-01-15T18:18:33.154569+0000","last_unstale":"2024-01-15T18:18:34.145574+0000","last_undegraded":"2024-01-15T18:18:34.145574+0000","last_fullsized":"2024-01-15T18:18:34.145574+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:29:02.332324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"107'4","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145441+0000","last_change":"2024-01-15T18:18:32.217761+0000","last_active":"2024-01-15T18:18:34.145441+0000","last_peered":"2024-01-15T18:18:34.145441+0000","last_clean":"2024-01-15T18:18:34.145441+0000","last_became_active":"2024-01-15T18:18:32.216337+0000","last_became_peered":"2024-01-15T18:18:32.216337+0000","last_unstale":"2024-01-15T18:18:34.145441+0000","last_undegraded":"2024-01-15T18:18:34.145441+0000","last_fullsized":"2024-01-15T18:18:34.145441+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:01:18.991818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.335822+0000","last_change":"2024-01-15T18:18:33.167015+0000","last_active":"2024-01-15T18:18:34.335822+0000","last_peered":"2024-01-15T18:18:34.335822+0000","last_clean":"2024-01-15T18:18:34.335822+0000","last_became_active":"2024-01-15T18:18:33.166726+0000","last_became_peered":"2024-01-15T18:18:33.166726+0000","last_unstale":"2024-01-15T18:18:34.335822+0000","last_undegraded":"2024-01-15T18:18:34.335822+0000","last_fullsized":"2024-01-15T18:18:34.335822+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:48:57.895180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"107'1","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336500+0000","last_change":"2024-01-15T18:18:32.219987+0000","last_active":"2024-01-15T18:18:34.336500+0000","last_peered":"2024-01-15T18:18:34.336500+0000","last_clean":"2024-01-15T18:18:34.336500+0000","last_became_active":"2024-01-15T18:18:32.219596+0000","last_became_peered":"2024-01-15T18:18:32.219596+0000","last_unstale":"2024-01-15T18:18:34.336500+0000","last_undegraded":"2024-01-15T18:18:34.336500+0000","last_fullsized":"2024-01-15T18:18:34.336500+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:23.024473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145427+0000","last_change":"2024-01-15T18:18:33.150743+0000","last_active":"2024-01-15T18:18:34.145427+0000","last_peered":"2024-01-15T18:18:34.145427+0000","last_clean":"2024-01-15T18:18:34.145427+0000","last_became_active":"2024-01-15T18:18:33.150429+0000","last_became_peered":"2024-01-15T18:18:33.150429+0000","last_unstale":"2024-01-15T18:18:34.145427+0000","last_undegraded":"2024-01-15T18:18:34.145427+0000","last_fullsized":"2024-01-15T18:18:34.145427+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:33:18.994479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"107'2","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337825+0000","last_change":"2024-01-15T18:18:32.213207+0000","last_active":"2024-01-15T18:18:34.337825+0000","last_peered":"2024-01-15T18:18:34.337825+0000","last_clean":"2024-01-15T18:18:34.337825+0000","last_became_active":"2024-01-15T18:18:32.212071+0000","last_became_peered":"2024-01-15T18:18:32.212071+0000","last_unstale":"2024-01-15T18:18:34.337825+0000","last_undegraded":"2024-01-15T18:18:34.337825+0000","last_fullsized":"2024-01-15T18:18:34.337825+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:29:14.612143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336611+0000","last_change":"2024-01-15T18:18:33.165545+0000","last_active":"2024-01-15T18:18:34.336611+0000","last_peered":"2024-01-15T18:18:34.336611+0000","last_clean":"2024-01-15T18:18:34.336611+0000","last_became_active":"2024-01-15T18:18:33.165258+0000","last_became_peered":"2024-01-15T18:18:33.165258+0000","last_unstale":"2024-01-15T18:18:34.336611+0000","last_undegraded":"2024-01-15T18:18:34.336611+0000","last_fullsized":"2024-01-15T18:18:34.336611+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:05:07.453809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"107'2","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337766+0000","last_change":"2024-01-15T18:18:32.217238+0000","last_active":"2024-01-15T18:18:34.337766+0000","last_peered":"2024-01-15T18:18:34.337766+0000","last_clean":"2024-01-15T18:18:34.337766+0000","last_became_active":"2024-01-15T18:18:32.216492+0000","last_became_peered":"2024-01-15T18:18:32.216492+0000","last_unstale":"2024-01-15T18:18:34.337766+0000","last_undegraded":"2024-01-15T18:18:34.337766+0000","last_fullsized":"2024-01-15T18:18:34.337766+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:40:25.010807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336803+0000","last_change":"2024-01-15T18:18:33.157996+0000","last_active":"2024-01-15T18:18:34.336803+0000","last_peered":"2024-01-15T18:18:34.336803+0000","last_clean":"2024-01-15T18:18:34.336803+0000","last_became_active":"2024-01-15T18:18:33.157687+0000","last_became_peered":"2024-01-15T18:18:33.157687+0000","last_unstale":"2024-01-15T18:18:34.336803+0000","last_undegraded":"2024-01-15T18:18:34.336803+0000","last_fullsized":"2024-01-15T18:18:34.336803+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:26:16.219599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"107'1","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145053+0000","last_change":"2024-01-15T18:18:32.215007+0000","last_active":"2024-01-15T18:18:34.145053+0000","last_peered":"2024-01-15T18:18:34.145053+0000","last_clean":"2024-01-15T18:18:34.145053+0000","last_became_active":"2024-01-15T18:18:32.214453+0000","last_became_peered":"2024-01-15T18:18:32.214453+0000","last_unstale":"2024-01-15T18:18:34.145053+0000","last_undegraded":"2024-01-15T18:18:34.145053+0000","last_fullsized":"2024-01-15T18:18:34.145053+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:58:50.916786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.144755+0000","last_change":"2024-01-15T18:18:33.153355+0000","last_active":"2024-01-15T18:18:34.144755+0000","last_peered":"2024-01-15T18:18:34.144755+0000","last_clean":"2024-01-15T18:18:34.144755+0000","last_became_active":"2024-01-15T18:18:33.153058+0000","last_became_peered":"2024-01-15T18:18:33.153058+0000","last_unstale":"2024-01-15T18:18:34.144755+0000","last_undegraded":"2024-01-15T18:18:34.144755+0000","last_fullsized":"2024-01-15T18:18:34.144755+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:21:18.264946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"107'1","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336335+0000","last_change":"2024-01-15T18:18:32.215681+0000","last_active":"2024-01-15T18:18:34.336335+0000","last_peered":"2024-01-15T18:18:34.336335+0000","last_clean":"2024-01-15T18:18:34.336335+0000","last_became_active":"2024-01-15T18:18:32.215417+0000","last_became_peered":"2024-01-15T18:18:32.215417+0000","last_unstale":"2024-01-15T18:18:34.336335+0000","last_undegraded":"2024-01-15T18:18:34.336335+0000","last_fullsized":"2024-01-15T18:18:34.336335+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:37:03.114460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336661+0000","last_change":"2024-01-15T18:18:33.161879+0000","last_active":"2024-01-15T18:18:34.336661+0000","last_peered":"2024-01-15T18:18:34.336661+0000","last_clean":"2024-01-15T18:18:34.336661+0000","last_became_active":"2024-01-15T18:18:33.161619+0000","last_became_peered":"2024-01-15T18:18:33.161619+0000","last_unstale":"2024-01-15T18:18:34.336661+0000","last_undegraded":"2024-01-15T18:18:34.336661+0000","last_fullsized":"2024-01-15T18:18:34.336661+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:59:11.477105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.144264+0000","last_change":"2024-01-15T18:18:32.217097+0000","last_active":"2024-01-15T18:18:34.144264+0000","last_peered":"2024-01-15T18:18:34.144264+0000","last_clean":"2024-01-15T18:18:34.144264+0000","last_became_active":"2024-01-15T18:18:32.215465+0000","last_became_peered":"2024-01-15T18:18:32.215465+0000","last_unstale":"2024-01-15T18:18:34.144264+0000","last_undegraded":"2024-01-15T18:18:34.144264+0000","last_fullsized":"2024-01-15T18:18:34.144264+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:47.729333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145835+0000","last_change":"2024-01-15T18:18:33.154067+0000","last_active":"2024-01-15T18:18:34.145835+0000","last_peered":"2024-01-15T18:18:34.145835+0000","last_clean":"2024-01-15T18:18:34.145835+0000","last_became_active":"2024-01-15T18:18:33.153610+0000","last_became_peered":"2024-01-15T18:18:33.153610+0000","last_unstale":"2024-01-15T18:18:34.145835+0000","last_undegraded":"2024-01-15T18:18:34.145835+0000","last_fullsized":"2024-01-15T18:18:34.145835+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:08:41.862115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336688+0000","last_change":"2024-01-15T18:18:32.219909+0000","last_active":"2024-01-15T18:18:34.336688+0000","last_peered":"2024-01-15T18:18:34.336688+0000","last_clean":"2024-01-15T18:18:34.336688+0000","last_became_active":"2024-01-15T18:18:32.219555+0000","last_became_peered":"2024-01-15T18:18:32.219555+0000","last_unstale":"2024-01-15T18:18:34.336688+0000","last_undegraded":"2024-01-15T18:18:34.336688+0000","last_fullsized":"2024-01-15T18:18:34.336688+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:17:21.343043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145567+0000","last_change":"2024-01-15T18:18:33.156038+0000","last_active":"2024-01-15T18:18:34.145567+0000","last_peered":"2024-01-15T18:18:34.145567+0000","last_clean":"2024-01-15T18:18:34.145567+0000","last_became_active":"2024-01-15T18:18:33.155765+0000","last_became_peered":"2024-01-15T18:18:33.155765+0000","last_unstale":"2024-01-15T18:18:34.145567+0000","last_undegraded":"2024-01-15T18:18:34.145567+0000","last_fullsized":"2024-01-15T18:18:34.145567+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:58:43.210443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337824+0000","last_change":"2024-01-15T18:18:32.212880+0000","last_active":"2024-01-15T18:18:34.337824+0000","last_peered":"2024-01-15T18:18:34.337824+0000","last_clean":"2024-01-15T18:18:34.337824+0000","last_became_active":"2024-01-15T18:18:32.212511+0000","last_became_peered":"2024-01-15T18:18:32.212511+0000","last_unstale":"2024-01-15T18:18:34.337824+0000","last_undegraded":"2024-01-15T18:18:34.337824+0000","last_fullsized":"2024-01-15T18:18:34.337824+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:58:38.191140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.335918+0000","last_change":"2024-01-15T18:18:33.164800+0000","last_active":"2024-01-15T18:18:34.335918+0000","last_peered":"2024-01-15T18:18:34.335918+0000","last_clean":"2024-01-15T18:18:34.335918+0000","last_became_active":"2024-01-15T18:18:33.164530+0000","last_became_peered":"2024-01-15T18:18:33.164530+0000","last_unstale":"2024-01-15T18:18:34.335918+0000","last_undegraded":"2024-01-15T18:18:34.335918+0000","last_fullsized":"2024-01-15T18:18:34.335918+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:55.175065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336668+0000","last_change":"2024-01-15T18:18:32.216873+0000","last_active":"2024-01-15T18:18:34.336668+0000","last_peered":"2024-01-15T18:18:34.336668+0000","last_clean":"2024-01-15T18:18:34.336668+0000","last_became_active":"2024-01-15T18:18:32.216337+0000","last_became_peered":"2024-01-15T18:18:32.216337+0000","last_unstale":"2024-01-15T18:18:34.336668+0000","last_undegraded":"2024-01-15T18:18:34.336668+0000","last_fullsized":"2024-01-15T18:18:34.336668+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:02.176575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336850+0000","last_change":"2024-01-15T18:18:33.148278+0000","last_active":"2024-01-15T18:18:34.336850+0000","last_peered":"2024-01-15T18:18:34.336850+0000","last_clean":"2024-01-15T18:18:34.336850+0000","last_became_active":"2024-01-15T18:18:33.148016+0000","last_became_peered":"2024-01-15T18:18:33.148016+0000","last_unstale":"2024-01-15T18:18:34.336850+0000","last_undegraded":"2024-01-15T18:18:34.336850+0000","last_fullsized":"2024-01-15T18:18:34.336850+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:24.034308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"107'3","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337309+0000","last_change":"2024-01-15T18:18:32.214473+0000","last_active":"2024-01-15T18:18:34.337309+0000","last_peered":"2024-01-15T18:18:34.337309+0000","last_clean":"2024-01-15T18:18:34.337309+0000","last_became_active":"2024-01-15T18:18:32.214186+0000","last_became_peered":"2024-01-15T18:18:32.214186+0000","last_unstale":"2024-01-15T18:18:34.337309+0000","last_undegraded":"2024-01-15T18:18:34.337309+0000","last_fullsized":"2024-01-15T18:18:34.337309+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:15:38.528251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145358+0000","last_change":"2024-01-15T18:18:33.159329+0000","last_active":"2024-01-15T18:18:34.145358+0000","last_peered":"2024-01-15T18:18:34.145358+0000","last_clean":"2024-01-15T18:18:34.145358+0000","last_became_active":"2024-01-15T18:18:33.158541+0000","last_became_peered":"2024-01-15T18:18:33.158541+0000","last_unstale":"2024-01-15T18:18:34.145358+0000","last_undegraded":"2024-01-15T18:18:34.145358+0000","last_fullsized":"2024-01-15T18:18:34.145358+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:53:07.912473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"107'6","reported_seq":18,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337256+0000","last_change":"2024-01-15T18:18:32.218321+0000","last_active":"2024-01-15T18:18:34.337256+0000","last_peered":"2024-01-15T18:18:34.337256+0000","last_clean":"2024-01-15T18:18:34.337256+0000","last_became_active":"2024-01-15T18:18:32.218002+0000","last_became_peered":"2024-01-15T18:18:32.218002+0000","last_unstale":"2024-01-15T18:18:34.337256+0000","last_undegraded":"2024-01-15T18:18:34.337256+0000","last_fullsized":"2024-01-15T18:18:34.337256+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:12:40.196083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336362+0000","last_change":"2024-01-15T18:18:33.151414+0000","last_active":"2024-01-15T18:18:34.336362+0000","last_peered":"2024-01-15T18:18:34.336362+0000","last_clean":"2024-01-15T18:18:34.336362+0000","last_became_active":"2024-01-15T18:18:33.151132+0000","last_became_peered":"2024-01-15T18:18:33.151132+0000","last_unstale":"2024-01-15T18:18:34.336362+0000","last_undegraded":"2024-01-15T18:18:34.336362+0000","last_fullsized":"2024-01-15T18:18:34.336362+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:48:57.895180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"107'2","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145300+0000","last_change":"2024-01-15T18:18:32.215335+0000","last_active":"2024-01-15T18:18:34.145300+0000","last_peered":"2024-01-15T18:18:34.145300+0000","last_clean":"2024-01-15T18:18:34.145300+0000","last_became_active":"2024-01-15T18:18:32.215073+0000","last_became_peered":"2024-01-15T18:18:32.215073+0000","last_unstale":"2024-01-15T18:18:34.145300+0000","last_undegraded":"2024-01-15T18:18:34.145300+0000","last_fullsized":"2024-01-15T18:18:34.145300+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:04:07.912459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336493+0000","last_change":"2024-01-15T18:18:33.161775+0000","last_active":"2024-01-15T18:18:34.336493+0000","last_peered":"2024-01-15T18:18:34.336493+0000","last_clean":"2024-01-15T18:18:34.336493+0000","last_became_active":"2024-01-15T18:18:33.161493+0000","last_became_peered":"2024-01-15T18:18:33.161493+0000","last_unstale":"2024-01-15T18:18:34.336493+0000","last_undegraded":"2024-01-15T18:18:34.336493+0000","last_fullsized":"2024-01-15T18:18:34.336493+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:55.175065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336025+0000","last_change":"2024-01-15T18:18:32.215225+0000","last_active":"2024-01-15T18:18:34.336025+0000","last_peered":"2024-01-15T18:18:34.336025+0000","last_clean":"2024-01-15T18:18:34.336025+0000","last_became_active":"2024-01-15T18:18:32.214942+0000","last_became_peered":"2024-01-15T18:18:32.214942+0000","last_unstale":"2024-01-15T18:18:34.336025+0000","last_undegraded":"2024-01-15T18:18:34.336025+0000","last_fullsized":"2024-01-15T18:18:34.336025+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:09:53.884883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336155+0000","last_change":"2024-01-15T18:18:33.146254+0000","last_active":"2024-01-15T18:18:34.336155+0000","last_peered":"2024-01-15T18:18:34.336155+0000","last_clean":"2024-01-15T18:18:34.336155+0000","last_became_active":"2024-01-15T18:18:33.145973+0000","last_became_peered":"2024-01-15T18:18:33.145973+0000","last_unstale":"2024-01-15T18:18:34.336155+0000","last_undegraded":"2024-01-15T18:18:34.336155+0000","last_fullsized":"2024-01-15T18:18:34.336155+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:09:54.894718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"107'8","reported_seq":20,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337489+0000","last_change":"2024-01-15T18:18:32.219579+0000","last_active":"2024-01-15T18:18:34.337489+0000","last_peered":"2024-01-15T18:18:34.337489+0000","last_clean":"2024-01-15T18:18:34.337489+0000","last_became_active":"2024-01-15T18:18:32.219280+0000","last_became_peered":"2024-01-15T18:18:32.219280+0000","last_unstale":"2024-01-15T18:18:34.337489+0000","last_undegraded":"2024-01-15T18:18:34.337489+0000","last_fullsized":"2024-01-15T18:18:34.337489+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:56:09.818155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.338016+0000","last_change":"2024-01-15T18:18:33.161674+0000","last_active":"2024-01-15T18:18:34.338016+0000","last_peered":"2024-01-15T18:18:34.338016+0000","last_clean":"2024-01-15T18:18:34.338016+0000","last_became_active":"2024-01-15T18:18:33.161329+0000","last_became_peered":"2024-01-15T18:18:33.161329+0000","last_unstale":"2024-01-15T18:18:34.338016+0000","last_undegraded":"2024-01-15T18:18:34.338016+0000","last_fullsized":"2024-01-15T18:18:34.338016+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:45:56.254343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"107'1","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337595+0000","last_change":"2024-01-15T18:18:32.215848+0000","last_active":"2024-01-15T18:18:34.337595+0000","last_peered":"2024-01-15T18:18:34.337595+0000","last_clean":"2024-01-15T18:18:34.337595+0000","last_became_active":"2024-01-15T18:18:32.214009+0000","last_became_peered":"2024-01-15T18:18:32.214009+0000","last_unstale":"2024-01-15T18:18:34.337595+0000","last_undegraded":"2024-01-15T18:18:34.337595+0000","last_fullsized":"2024-01-15T18:18:34.337595+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:48:09.722577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145635+0000","last_change":"2024-01-15T18:18:33.162974+0000","last_active":"2024-01-15T18:18:34.145635+0000","last_peered":"2024-01-15T18:18:34.145635+0000","last_clean":"2024-01-15T18:18:34.145635+0000","last_became_active":"2024-01-15T18:18:33.162679+0000","last_became_peered":"2024-01-15T18:18:33.162679+0000","last_unstale":"2024-01-15T18:18:34.145635+0000","last_undegraded":"2024-01-15T18:18:34.145635+0000","last_fullsized":"2024-01-15T18:18:34.145635+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:53:15.036059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"107'2","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337993+0000","last_change":"2024-01-15T18:18:32.214990+0000","last_active":"2024-01-15T18:18:34.337993+0000","last_peered":"2024-01-15T18:18:34.337993+0000","last_clean":"2024-01-15T18:18:34.337993+0000","last_became_active":"2024-01-15T18:18:32.214695+0000","last_became_peered":"2024-01-15T18:18:32.214695+0000","last_unstale":"2024-01-15T18:18:34.337993+0000","last_undegraded":"2024-01-15T18:18:34.337993+0000","last_fullsized":"2024-01-15T18:18:34.337993+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:15:08.445774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336467+0000","last_change":"2024-01-15T18:18:33.168531+0000","last_active":"2024-01-15T18:18:34.336467+0000","last_peered":"2024-01-15T18:18:34.336467+0000","last_clean":"2024-01-15T18:18:34.336467+0000","last_became_active":"2024-01-15T18:18:33.168190+0000","last_became_peered":"2024-01-15T18:18:33.168190+0000","last_unstale":"2024-01-15T18:18:34.336467+0000","last_undegraded":"2024-01-15T18:18:34.336467+0000","last_fullsized":"2024-01-15T18:18:34.336467+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:53:05.325329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"107'1","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.335781+0000","last_change":"2024-01-15T18:18:32.221200+0000","last_active":"2024-01-15T18:18:34.335781+0000","last_peered":"2024-01-15T18:18:34.335781+0000","last_clean":"2024-01-15T18:18:34.335781+0000","last_became_active":"2024-01-15T18:18:32.220884+0000","last_became_peered":"2024-01-15T18:18:32.220884+0000","last_unstale":"2024-01-15T18:18:34.335781+0000","last_undegraded":"2024-01-15T18:18:34.335781+0000","last_fullsized":"2024-01-15T18:18:34.335781+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:02.938267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336678+0000","last_change":"2024-01-15T18:18:33.163114+0000","last_active":"2024-01-15T18:18:34.336678+0000","last_peered":"2024-01-15T18:18:34.336678+0000","last_clean":"2024-01-15T18:18:34.336678+0000","last_became_active":"2024-01-15T18:18:33.162424+0000","last_became_peered":"2024-01-15T18:18:33.162424+0000","last_unstale":"2024-01-15T18:18:34.336678+0000","last_undegraded":"2024-01-15T18:18:34.336678+0000","last_fullsized":"2024-01-15T18:18:34.336678+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:42:08.107694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145019+0000","last_change":"2024-01-15T18:18:32.217252+0000","last_active":"2024-01-15T18:18:34.145019+0000","last_peered":"2024-01-15T18:18:34.145019+0000","last_clean":"2024-01-15T18:18:34.145019+0000","last_became_active":"2024-01-15T18:18:32.216769+0000","last_became_peered":"2024-01-15T18:18:32.216769+0000","last_unstale":"2024-01-15T18:18:34.145019+0000","last_undegraded":"2024-01-15T18:18:34.145019+0000","last_fullsized":"2024-01-15T18:18:34.145019+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:36.807888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336194+0000","last_change":"2024-01-15T18:18:33.163017+0000","last_active":"2024-01-15T18:18:34.336194+0000","last_peered":"2024-01-15T18:18:34.336194+0000","last_clean":"2024-01-15T18:18:34.336194+0000","last_became_active":"2024-01-15T18:18:33.162658+0000","last_became_peered":"2024-01-15T18:18:33.162658+0000","last_unstale":"2024-01-15T18:18:34.336194+0000","last_undegraded":"2024-01-15T18:18:34.336194+0000","last_fullsized":"2024-01-15T18:18:34.336194+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:01:41.498479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"107'2","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145329+0000","last_change":"2024-01-15T18:18:32.207796+0000","last_active":"2024-01-15T18:18:34.145329+0000","last_peered":"2024-01-15T18:18:34.145329+0000","last_clean":"2024-01-15T18:18:34.145329+0000","last_became_active":"2024-01-15T18:18:32.207507+0000","last_became_peered":"2024-01-15T18:18:32.207507+0000","last_unstale":"2024-01-15T18:18:34.145329+0000","last_undegraded":"2024-01-15T18:18:34.145329+0000","last_fullsized":"2024-01-15T18:18:34.145329+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:42.592200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336888+0000","last_change":"2024-01-15T18:18:33.164981+0000","last_active":"2024-01-15T18:18:34.336888+0000","last_peered":"2024-01-15T18:18:34.336888+0000","last_clean":"2024-01-15T18:18:34.336888+0000","last_became_active":"2024-01-15T18:18:33.164206+0000","last_became_peered":"2024-01-15T18:18:33.164206+0000","last_unstale":"2024-01-15T18:18:34.336888+0000","last_undegraded":"2024-01-15T18:18:34.336888+0000","last_fullsized":"2024-01-15T18:18:34.336888+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:53:32.550071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337629+0000","last_change":"2024-01-15T18:18:32.212499+0000","last_active":"2024-01-15T18:18:34.337629+0000","last_peered":"2024-01-15T18:18:34.337629+0000","last_clean":"2024-01-15T18:18:34.337629+0000","last_became_active":"2024-01-15T18:18:32.212201+0000","last_became_peered":"2024-01-15T18:18:32.212201+0000","last_unstale":"2024-01-15T18:18:34.337629+0000","last_undegraded":"2024-01-15T18:18:34.337629+0000","last_fullsized":"2024-01-15T18:18:34.337629+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:25:39.031903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.335829+0000","last_change":"2024-01-15T18:18:33.165046+0000","last_active":"2024-01-15T18:18:34.335829+0000","last_peered":"2024-01-15T18:18:34.335829+0000","last_clean":"2024-01-15T18:18:34.335829+0000","last_became_active":"2024-01-15T18:18:33.164708+0000","last_became_peered":"2024-01-15T18:18:33.164708+0000","last_unstale":"2024-01-15T18:18:34.335829+0000","last_undegraded":"2024-01-15T18:18:34.335829+0000","last_fullsized":"2024-01-15T18:18:34.335829+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:46:54.687765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145456+0000","last_change":"2024-01-15T18:18:32.207872+0000","last_active":"2024-01-15T18:18:34.145456+0000","last_peered":"2024-01-15T18:18:34.145456+0000","last_clean":"2024-01-15T18:18:34.145456+0000","last_became_active":"2024-01-15T18:18:32.207599+0000","last_became_peered":"2024-01-15T18:18:32.207599+0000","last_unstale":"2024-01-15T18:18:34.145456+0000","last_undegraded":"2024-01-15T18:18:34.145456+0000","last_fullsized":"2024-01-15T18:18:34.145456+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:25:20.009100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336145+0000","last_change":"2024-01-15T18:18:33.160618+0000","last_active":"2024-01-15T18:18:34.336145+0000","last_peered":"2024-01-15T18:18:34.336145+0000","last_clean":"2024-01-15T18:18:34.336145+0000","last_became_active":"2024-01-15T18:18:33.160350+0000","last_became_peered":"2024-01-15T18:18:33.160350+0000","last_unstale":"2024-01-15T18:18:34.336145+0000","last_undegraded":"2024-01-15T18:18:34.336145+0000","last_fullsized":"2024-01-15T18:18:34.336145+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:42:11.703397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336755+0000","last_change":"2024-01-15T18:18:32.217214+0000","last_active":"2024-01-15T18:18:34.336755+0000","last_peered":"2024-01-15T18:18:34.336755+0000","last_clean":"2024-01-15T18:18:34.336755+0000","last_became_active":"2024-01-15T18:18:32.216581+0000","last_became_peered":"2024-01-15T18:18:32.216581+0000","last_unstale":"2024-01-15T18:18:34.336755+0000","last_undegraded":"2024-01-15T18:18:34.336755+0000","last_fullsized":"2024-01-15T18:18:34.336755+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:49:26.780801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.335999+0000","last_change":"2024-01-15T18:18:33.165549+0000","last_active":"2024-01-15T18:18:34.335999+0000","last_peered":"2024-01-15T18:18:34.335999+0000","last_clean":"2024-01-15T18:18:34.335999+0000","last_became_active":"2024-01-15T18:18:33.164152+0000","last_became_peered":"2024-01-15T18:18:33.164152+0000","last_unstale":"2024-01-15T18:18:34.335999+0000","last_undegraded":"2024-01-15T18:18:34.335999+0000","last_fullsized":"2024-01-15T18:18:34.335999+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:24.034308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"107'2","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337482+0000","last_change":"2024-01-15T18:18:32.219821+0000","last_active":"2024-01-15T18:18:34.337482+0000","last_peered":"2024-01-15T18:18:34.337482+0000","last_clean":"2024-01-15T18:18:34.337482+0000","last_became_active":"2024-01-15T18:18:32.218858+0000","last_became_peered":"2024-01-15T18:18:32.218858+0000","last_unstale":"2024-01-15T18:18:34.337482+0000","last_undegraded":"2024-01-15T18:18:34.337482+0000","last_fullsized":"2024-01-15T18:18:34.337482+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:33:51.257660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337507+0000","last_change":"2024-01-15T18:18:33.160602+0000","last_active":"2024-01-15T18:18:34.337507+0000","last_peered":"2024-01-15T18:18:34.337507+0000","last_clean":"2024-01-15T18:18:34.337507+0000","last_became_active":"2024-01-15T18:18:33.160315+0000","last_became_peered":"2024-01-15T18:18:33.160315+0000","last_unstale":"2024-01-15T18:18:34.337507+0000","last_undegraded":"2024-01-15T18:18:34.337507+0000","last_fullsized":"2024-01-15T18:18:34.337507+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:34.698586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.335946+0000","last_change":"2024-01-15T18:18:32.217029+0000","last_active":"2024-01-15T18:18:34.335946+0000","last_peered":"2024-01-15T18:18:34.335946+0000","last_clean":"2024-01-15T18:18:34.335946+0000","last_became_active":"2024-01-15T18:18:32.216612+0000","last_became_peered":"2024-01-15T18:18:32.216612+0000","last_unstale":"2024-01-15T18:18:34.335946+0000","last_undegraded":"2024-01-15T18:18:34.335946+0000","last_fullsized":"2024-01-15T18:18:34.335946+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:42:10.693562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337255+0000","last_change":"2024-01-15T18:18:33.165599+0000","last_active":"2024-01-15T18:18:34.337255+0000","last_peered":"2024-01-15T18:18:34.337255+0000","last_clean":"2024-01-15T18:18:34.337255+0000","last_became_active":"2024-01-15T18:18:33.163994+0000","last_became_peered":"2024-01-15T18:18:33.163994+0000","last_unstale":"2024-01-15T18:18:34.337255+0000","last_undegraded":"2024-01-15T18:18:34.337255+0000","last_fullsized":"2024-01-15T18:18:34.337255+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:00:26.779661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337321+0000","last_change":"2024-01-15T18:18:32.216301+0000","last_active":"2024-01-15T18:18:34.337321+0000","last_peered":"2024-01-15T18:18:34.337321+0000","last_clean":"2024-01-15T18:18:34.337321+0000","last_became_active":"2024-01-15T18:18:32.216023+0000","last_became_peered":"2024-01-15T18:18:32.216023+0000","last_unstale":"2024-01-15T18:18:34.337321+0000","last_undegraded":"2024-01-15T18:18:34.337321+0000","last_fullsized":"2024-01-15T18:18:34.337321+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:58:58.730569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.335697+0000","last_change":"2024-01-15T18:18:33.165611+0000","last_active":"2024-01-15T18:18:34.335697+0000","last_peered":"2024-01-15T18:18:34.335697+0000","last_clean":"2024-01-15T18:18:34.335697+0000","last_became_active":"2024-01-15T18:18:33.165297+0000","last_became_peered":"2024-01-15T18:18:33.165297+0000","last_unstale":"2024-01-15T18:18:34.335697+0000","last_undegraded":"2024-01-15T18:18:34.335697+0000","last_fullsized":"2024-01-15T18:18:34.335697+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:18.639432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336949+0000","last_change":"2024-01-15T18:18:32.214224+0000","last_active":"2024-01-15T18:18:34.336949+0000","last_peered":"2024-01-15T18:18:34.336949+0000","last_clean":"2024-01-15T18:18:34.336949+0000","last_became_active":"2024-01-15T18:18:32.213930+0000","last_became_peered":"2024-01-15T18:18:32.213930+0000","last_unstale":"2024-01-15T18:18:34.336949+0000","last_undegraded":"2024-01-15T18:18:34.336949+0000","last_fullsized":"2024-01-15T18:18:34.336949+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:01:15.328258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.144379+0000","last_change":"2024-01-15T18:18:33.150974+0000","last_active":"2024-01-15T18:18:34.144379+0000","last_peered":"2024-01-15T18:18:34.144379+0000","last_clean":"2024-01-15T18:18:34.144379+0000","last_became_active":"2024-01-15T18:18:33.150660+0000","last_became_peered":"2024-01-15T18:18:33.150660+0000","last_unstale":"2024-01-15T18:18:34.144379+0000","last_undegraded":"2024-01-15T18:18:34.144379+0000","last_fullsized":"2024-01-15T18:18:34.144379+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:22:41.967700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145145+0000","last_change":"2024-01-15T18:18:32.215229+0000","last_active":"2024-01-15T18:18:34.145145+0000","last_peered":"2024-01-15T18:18:34.145145+0000","last_clean":"2024-01-15T18:18:34.145145+0000","last_became_active":"2024-01-15T18:18:32.214394+0000","last_became_peered":"2024-01-15T18:18:32.214394+0000","last_unstale":"2024-01-15T18:18:34.145145+0000","last_undegraded":"2024-01-15T18:18:34.145145+0000","last_fullsized":"2024-01-15T18:18:34.145145+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:30:25.477307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.144477+0000","last_change":"2024-01-15T18:18:33.156094+0000","last_active":"2024-01-15T18:18:34.144477+0000","last_peered":"2024-01-15T18:18:34.144477+0000","last_clean":"2024-01-15T18:18:34.144477+0000","last_became_active":"2024-01-15T18:18:33.155797+0000","last_became_peered":"2024-01-15T18:18:33.155797+0000","last_unstale":"2024-01-15T18:18:34.144477+0000","last_undegraded":"2024-01-15T18:18:34.144477+0000","last_fullsized":"2024-01-15T18:18:34.144477+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:21:42.101901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.144235+0000","last_change":"2024-01-15T18:18:32.212760+0000","last_active":"2024-01-15T18:18:34.144235+0000","last_peered":"2024-01-15T18:18:34.144235+0000","last_clean":"2024-01-15T18:18:34.144235+0000","last_became_active":"2024-01-15T18:18:32.212487+0000","last_became_peered":"2024-01-15T18:18:32.212487+0000","last_unstale":"2024-01-15T18:18:34.144235+0000","last_undegraded":"2024-01-15T18:18:34.144235+0000","last_fullsized":"2024-01-15T18:18:34.144235+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:48:56.885346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.335949+0000","last_change":"2024-01-15T18:18:33.167732+0000","last_active":"2024-01-15T18:18:34.335949+0000","last_peered":"2024-01-15T18:18:34.335949+0000","last_clean":"2024-01-15T18:18:34.335949+0000","last_became_active":"2024-01-15T18:18:33.167436+0000","last_became_peered":"2024-01-15T18:18:33.167436+0000","last_unstale":"2024-01-15T18:18:34.335949+0000","last_undegraded":"2024-01-15T18:18:34.335949+0000","last_fullsized":"2024-01-15T18:18:34.335949+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:22:13.319815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337623+0000","last_change":"2024-01-15T18:18:32.213077+0000","last_active":"2024-01-15T18:18:34.337623+0000","last_peered":"2024-01-15T18:18:34.337623+0000","last_clean":"2024-01-15T18:18:34.337623+0000","last_became_active":"2024-01-15T18:18:32.211744+0000","last_became_peered":"2024-01-15T18:18:32.211744+0000","last_unstale":"2024-01-15T18:18:34.337623+0000","last_undegraded":"2024-01-15T18:18:34.337623+0000","last_fullsized":"2024-01-15T18:18:34.337623+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:48:24.967174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336876+0000","last_change":"2024-01-15T18:18:33.163355+0000","last_active":"2024-01-15T18:18:34.336876+0000","last_peered":"2024-01-15T18:18:34.336876+0000","last_clean":"2024-01-15T18:18:34.336876+0000","last_became_active":"2024-01-15T18:18:33.162521+0000","last_became_peered":"2024-01-15T18:18:33.162521+0000","last_unstale":"2024-01-15T18:18:34.336876+0000","last_undegraded":"2024-01-15T18:18:34.336876+0000","last_fullsized":"2024-01-15T18:18:34.336876+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:35:00.923337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337137+0000","last_change":"2024-01-15T18:18:32.217455+0000","last_active":"2024-01-15T18:18:34.337137+0000","last_peered":"2024-01-15T18:18:34.337137+0000","last_clean":"2024-01-15T18:18:34.337137+0000","last_became_active":"2024-01-15T18:18:32.216966+0000","last_became_peered":"2024-01-15T18:18:32.216966+0000","last_unstale":"2024-01-15T18:18:34.337137+0000","last_undegraded":"2024-01-15T18:18:34.337137+0000","last_fullsized":"2024-01-15T18:18:34.337137+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:56:34.922952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336209+0000","last_change":"2024-01-15T18:18:33.166946+0000","last_active":"2024-01-15T18:18:34.336209+0000","last_peered":"2024-01-15T18:18:34.336209+0000","last_clean":"2024-01-15T18:18:34.336209+0000","last_became_active":"2024-01-15T18:18:33.165323+0000","last_became_peered":"2024-01-15T18:18:33.165323+0000","last_unstale":"2024-01-15T18:18:34.336209+0000","last_undegraded":"2024-01-15T18:18:34.336209+0000","last_fullsized":"2024-01-15T18:18:34.336209+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:18.639432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145258+0000","last_change":"2024-01-15T18:18:32.216251+0000","last_active":"2024-01-15T18:18:34.145258+0000","last_peered":"2024-01-15T18:18:34.145258+0000","last_clean":"2024-01-15T18:18:34.145258+0000","last_became_active":"2024-01-15T18:18:32.215973+0000","last_became_peered":"2024-01-15T18:18:32.215973+0000","last_unstale":"2024-01-15T18:18:34.145258+0000","last_undegraded":"2024-01-15T18:18:34.145258+0000","last_fullsized":"2024-01-15T18:18:34.145258+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:41:26.812763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":239,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-01-15T18:18:33.154452+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:18:33.154452+0000","last_peered":"2024-01-15T18:18:33.154452+0000","last_clean":"2024-01-15T18:18:33.154452+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:18:33.154452+0000","last_undegraded":"2024-01-15T18:18:33.154452+0000","last_fullsized":"2024-01-15T18:18:33.154452+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509609,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64944,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306896,"statfs":{"total":96636764160,"available":96570261504,"internally_reserved":0,"allocated":1937408,"data_stored":1173215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64551370},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.84599999999999997,"15min":0.874},"min":{"1min":0.36799999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.95399999999999996,"5min":9.3040000000000003,"15min":10.284000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.85499999999999998,"15min":0.92300000000000004},"min":{"1min":0.42999999999999999,"5min":0.38400000000000001,"15min":0.378},"max":{"1min":0.83499999999999996,"5min":6.6189999999999998,"15min":16.968},"last":0.53600000000000003}]},{"osd":1,"last update":"Mon Jan 15 18:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.90000000000000002,"15min":0.92400000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.33800000000000002},"max":{"1min":0.91600000000000004,"5min":9.3829999999999991,"15min":15.791},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.85799999999999998,"15min":0.91700000000000004},"min":{"1min":0.45000000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.98899999999999999,"5min":5.0899999999999999,"15min":10.842000000000001},"last":0.80200000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":1.1539999999999999,"15min":0.93700000000000006},"min":{"1min":0.45300000000000001,"5min":0.40400000000000003,"15min":0.378},"max":{"1min":0.878,"5min":22.318000000000001,"15min":22.318000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.90900000000000003,"15min":0.90200000000000002},"min":{"1min":0.44900000000000001,"5min":0.433,"15min":0.39100000000000001},"max":{"1min":0.94799999999999995,"5min":9.5169999999999995,"15min":9.7539999999999996},"last":0.97699999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":1.155,"15min":0.997},"min":{"1min":0.501,"5min":0.501,"15min":0.38100000000000001},"max":{"1min":2.1739999999999999,"5min":18.152999999999999,"15min":18.152999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.95199999999999996,"15min":0.97799999999999998},"min":{"1min":0.5,"5min":0.49199999999999999,"15min":0.43099999999999999},"max":{"1min":0.83699999999999997,"5min":9.75,"15min":10.34},"last":0.93400000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.871,"15min":0.93999999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.314},"max":{"1min":0.91000000000000003,"5min":8.4420000000000002,"15min":20.387},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":1.153,"15min":0.97699999999999998},"min":{"1min":0.38400000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82099999999999995,"5min":17.215,"15min":17.215},"last":0.871}]},{"osd":5,"last update":"Mon Jan 15 18:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.93100000000000005,"15min":0.93500000000000005},"min":{"1min":0.38600000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.78400000000000003,"5min":9.6630000000000003,"15min":19.959},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.92200000000000004,"15min":0.97899999999999998},"min":{"1min":0.46500000000000002,"5min":0.39400000000000002,"15min":0.35799999999999998},"max":{"1min":0.84599999999999997,"5min":7.6980000000000004,"15min":19.905000000000001},"last":0.85399999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":1.0029999999999999,"15min":1.1539999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.99099999999999999,"5min":8.6609999999999996,"15min":22.478999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":1.143,"15min":1.0600000000000001},"min":{"1min":0.48299999999999998,"5min":0.40799999999999997,"15min":0.38300000000000001},"max":{"1min":1.012,"5min":17.132000000000001,"15min":21.509},"last":0.91500000000000004}]}]},{"osd":6,"up_from":15,"seq":64424509609,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64368,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307472,"statfs":{"total":96636764160,"available":96570851328,"internally_reserved":0,"allocated":1339392,"data_stored":582301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:18:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":1.0629999999999999,"15min":0.97199999999999998},"min":{"1min":0.42799999999999999,"5min":0.35799999999999998,"15min":0.29099999999999998},"max":{"1min":0.878,"5min":32.587000000000003,"15min":32.587000000000003},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.93300000000000005,"15min":0.86399999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.35399999999999998},"max":{"1min":1.482,"5min":16.193000000000001,"15min":16.193000000000001},"last":0.78200000000000003}]},{"osd":1,"last update":"Mon Jan 15 18:18:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.82799999999999996,"15min":0.98799999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.36599999999999999},"max":{"1min":1.0449999999999999,"5min":7.7649999999999997,"15min":53.451999999999998},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":1.01,"15min":0.91400000000000003},"min":{"1min":0.47299999999999998,"5min":0.437,"15min":0.42899999999999999},"max":{"1min":1.0880000000000001,"5min":32.512,"15min":32.512},"last":0.76100000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:18:01 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":1.0800000000000001,"15min":0.95999999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.38900000000000001},"max":{"1min":0.92900000000000005,"5min":32.472999999999999,"15min":32.472999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.99399999999999999,"15min":0.88700000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.38500000000000001},"max":{"1min":0.94299999999999995,"5min":16.602,"15min":16.602},"last":0.73999999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:18:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.98699999999999999,"15min":0.92500000000000004},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.41399999999999998},"max":{"1min":0.91800000000000004,"5min":16.388000000000002,"15min":16.388000000000002},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.88600000000000001,"15min":1.071},"min":{"1min":0.52800000000000002,"5min":0.52300000000000002,"15min":0.41599999999999998},"max":{"1min":0.96199999999999997,"5min":5.9740000000000002,"15min":53.398000000000003},"last":0.67200000000000004}]},{"osd":4,"last update":"Mon Jan 15 18:18:01 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.99199999999999999,"15min":0.88700000000000001},"min":{"1min":0.45700000000000002,"5min":0.38100000000000001,"15min":0.34999999999999998},"max":{"1min":1.1379999999999999,"5min":32.549999999999997,"15min":32.549999999999997},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.879,"15min":0.83499999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.94699999999999995,"5min":11.852,"15min":11.852},"last":0.629}]},{"osd":5,"last update":"Mon Jan 15 18:18:01 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.80900000000000005,"15min":1.0349999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.30399999999999999},"max":{"1min":0.82099999999999995,"5min":7.5650000000000004,"15min":53.229999999999997},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":1.093,"15min":1.024},"min":{"1min":0.46300000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.45,"5min":32.404000000000003,"15min":32.404000000000003},"last":0.70199999999999996}]},{"osd":7,"last update":"Mon Jan 15 18:18:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.94699999999999995,"15min":1.097},"min":{"1min":0.41099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.97999999999999998,"5min":16.565000000000001,"15min":45.070999999999998},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.72499999999999998,"5min":1.173,"15min":1.2050000000000001},"min":{"1min":0.5,"5min":0.45000000000000001,"15min":0.34999999999999998},"max":{"1min":1.1140000000000001,"5min":40.972999999999999,"15min":53.338000000000001},"last":0.59299999999999997}]}]},{"osd":4,"up_from":15,"seq":64424509609,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64364,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307476,"statfs":{"total":96636764160,"available":96570855424,"internally_reserved":0,"allocated":1335296,"data_stored":582231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.81599999999999995,"15min":0.88200000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.32900000000000001},"max":{"1min":0.91700000000000004,"5min":7.3380000000000001,"15min":23.318000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.68999999999999995,"15min":0.78800000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.32600000000000001},"max":{"1min":0.82099999999999995,"5min":6.444,"15min":14.723000000000001},"last":0.63100000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.82999999999999996,"15min":0.78400000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90500000000000003,"5min":9.5700000000000003,"15min":9.5700000000000003},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.91300000000000003,"15min":0.91400000000000003},"min":{"1min":0.45700000000000002,"5min":0.41399999999999998,"15min":0.36899999999999999},"max":{"1min":0.81100000000000005,"5min":7.9980000000000002,"15min":22.408999999999999},"last":0.77500000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.85999999999999999,"15min":0.80000000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.34399999999999997},"max":{"1min":0.80800000000000005,"5min":9.7089999999999996,"15min":9.7089999999999996},"last":0.874},{"interface":"front","average":{"1min":0.67900000000000005,"5min":1.0629999999999999,"15min":1.014},"min":{"1min":0.44400000000000001,"5min":0.40999999999999998,"15min":0.39100000000000001},"max":{"1min":1.3180000000000001,"5min":14.118,"15min":22.338999999999999},"last":0.72199999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.77300000000000002,"15min":0.86899999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.41199999999999998},"max":{"1min":0.88400000000000001,"5min":6.7930000000000001,"15min":14.763999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.94799999999999995,"15min":0.90500000000000003},"min":{"1min":0.495,"5min":0.495,"15min":0.45500000000000002},"max":{"1min":0.95199999999999996,"5min":9.6289999999999996,"15min":9.6289999999999996},"last":0.82699999999999996}]},{"osd":5,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.93600000000000005,"15min":0.95099999999999996},"min":{"1min":0.39700000000000002,"5min":0.39000000000000001,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":15.787000000000001,"15min":31.617999999999999},"last":0.5},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.746,"15min":0.86099999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.36599999999999999},"max":{"1min":0.79600000000000004,"5min":6.7169999999999996,"15min":17.145},"last":0.79700000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.83899999999999997,"15min":0.82099999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.35499999999999998},"max":{"1min":0.85999999999999999,"5min":9.6709999999999994,"15min":11.124000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":1.159,"15min":1.2170000000000001},"min":{"1min":0.498,"5min":0.438,"15min":0.40200000000000002},"max":{"1min":0.89300000000000002,"5min":32.308,"15min":32.308},"last":0.65600000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.73799999999999999,"15min":0.996},"min":{"1min":0.45700000000000002,"5min":0.434,"15min":0.34799999999999998},"max":{"1min":0.876,"5min":6.8390000000000004,"15min":39.256},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.748,"5min":1.1279999999999999,"15min":1.002},"min":{"1min":0.504,"5min":0.47199999999999998,"15min":0.42199999999999999},"max":{"1min":2.3330000000000002,"5min":16.277000000000001,"15min":26.573},"last":0.73999999999999999}]}]},{"osd":3,"up_from":15,"seq":64424509608,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68532,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303308,"statfs":{"total":96636764160,"available":96566587392,"internally_reserved":0,"allocated":1351680,"data_stored":583913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.45800000000000002,"15min":0.49099999999999999},"min":{"1min":0.29999999999999999,"5min":0.25900000000000001,"15min":0.245},"max":{"1min":0.752,"5min":2.1160000000000001,"15min":5.8849999999999998},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.53900000000000003,"15min":0.61599999999999999},"min":{"1min":0.32900000000000001,"5min":0.29799999999999999,"15min":0.29499999999999998},"max":{"1min":0.72399999999999998,"5min":3.0649999999999999,"15min":17.756},"last":0.38200000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.54500000000000004,"15min":0.59899999999999998},"min":{"1min":0.309,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":5.8819999999999997,"5min":5.8819999999999997,"15min":12.307},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.38600000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.80500000000000005,"5min":3.738,"15min":6.0640000000000001},"last":0.44500000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.66400000000000003,"15min":0.69099999999999995},"min":{"1min":0.375,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.145,"5min":5.431,"15min":12.468999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.70799999999999996,"15min":0.68600000000000005},"min":{"1min":0.433,"5min":0.35399999999999998,"15min":0.34599999999999997},"max":{"1min":0.871,"5min":10.837999999999999,"15min":10.837999999999999},"last":0.46500000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":1.0649999999999999,"15min":1.0089999999999999},"min":{"1min":0.433,"5min":0.41199999999999998,"15min":0.375},"max":{"1min":1.109,"5min":13.863,"15min":17.853999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.96399999999999997,"15min":0.96299999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.2090000000000001,"5min":6.8289999999999997,"15min":13.31},"last":0.66200000000000003}]},{"osd":5,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":1.167,"15min":0.97499999999999998},"min":{"1min":0.53400000000000003,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.042,"5min":17.687999999999999,"15min":17.687999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.88300000000000001,"15min":0.97199999999999998},"min":{"1min":0.53200000000000003,"5min":0.45500000000000002,"15min":0.438},"max":{"1min":1.1870000000000001,"5min":5.0949999999999998,"15min":20.515999999999998},"last":0.54600000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.81100000000000005,"15min":0.97499999999999998},"min":{"1min":0.52200000000000002,"5min":0.442,"15min":0.432},"max":{"1min":1.294,"5min":4.0910000000000002,"15min":23.100999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.91700000000000004,"15min":0.93799999999999994},"min":{"1min":0.56399999999999995,"5min":0.55600000000000005,"15min":0.49399999999999999},"max":{"1min":1.159,"5min":6.7220000000000004,"15min":12.76},"last":0.64800000000000002}]},{"osd":7,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":1.335,"15min":1.1659999999999999},"min":{"1min":0.57699999999999996,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.236,"5min":24.321999999999999,"15min":24.321999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.79600000000000004,"5min":1.147,"15min":1.1000000000000001},"min":{"1min":0.55200000000000005,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.3340000000000001,"5min":23.013000000000002,"15min":23.013000000000002},"last":0.56100000000000005}]}]},{"osd":2,"up_from":15,"seq":64424509608,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64364,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307476,"statfs":{"total":96636764160,"available":96570855424,"internally_reserved":0,"allocated":1335296,"data_stored":582231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:18:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.45900000000000002,"15min":0.50900000000000001},"min":{"1min":0.29799999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.61399999999999999,"5min":0.98199999999999998,"15min":5.1849999999999996},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.48999999999999999,"15min":0.53200000000000003},"min":{"1min":0.32900000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.68500000000000005,"5min":1.1579999999999999,"15min":7.516},"last":0.40000000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:18:07 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.54900000000000004,"15min":0.61299999999999999},"min":{"1min":0.33000000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.72799999999999998,"5min":0.95799999999999996,"15min":10.907},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.53100000000000003,"15min":0.60299999999999998},"min":{"1min":0.32900000000000001,"5min":0.311,"15min":0.27000000000000002},"max":{"1min":2.8799999999999999,"5min":2.8799999999999999,"15min":8.9190000000000005},"last":0.46000000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:18:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.55100000000000005,"15min":0.61899999999999999},"min":{"1min":0.34499999999999997,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.76700000000000002,"5min":1.133,"15min":12.185},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.59399999999999997,"15min":0.66400000000000003},"min":{"1min":0.40100000000000002,"5min":0.373,"15min":0.34000000000000002},"max":{"1min":0.78800000000000003,"5min":0.96699999999999997,"15min":7.6760000000000002},"last":0.47899999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:18:07 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.89100000000000001,"15min":0.877},"min":{"1min":0.54400000000000004,"5min":0.44,"15min":0.44},"max":{"1min":1.26,"5min":4.3150000000000004,"15min":8.6609999999999996},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.79400000000000004,"15min":0.91100000000000003},"min":{"1min":0.45000000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":2.968,"5min":2.968,"15min":19.146999999999998},"last":0.497}]},{"osd":5,"last update":"Mon Jan 15 18:18:07 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.80700000000000005,"15min":0.90600000000000003},"min":{"1min":0.57399999999999995,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.1359999999999999,"5min":5.0309999999999997,"15min":31.710000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.83699999999999997,"15min":0.997},"min":{"1min":0.60999999999999999,"5min":0.49199999999999999,"15min":0.48199999999999998},"max":{"1min":1.115,"5min":3.6459999999999999,"15min":22.539000000000001},"last":0.51700000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:18:07 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.877,"15min":0.91300000000000003},"min":{"1min":0.50700000000000001,"5min":0.499,"15min":0.40899999999999997},"max":{"1min":3.0430000000000001,"5min":4.1959999999999997,"15min":9.4540000000000006},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.79500000000000004,"15min":0.90400000000000003},"min":{"1min":0.54300000000000004,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0589999999999999,"5min":1.633,"15min":18.614999999999998},"last":0.624}]},{"osd":7,"last update":"Mon Jan 15 18:18:07 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.99399999999999999,"15min":0.96899999999999997},"min":{"1min":0.52200000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":3.008,"5min":11.606,"15min":11.606},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.91900000000000004,"15min":1.0009999999999999},"min":{"1min":0.58299999999999996,"5min":0.52500000000000002,"15min":0.502},"max":{"1min":1.151,"5min":6.1260000000000003,"15min":17.013000000000002},"last":0.53900000000000003}]}]},{"osd":1,"up_from":15,"seq":64424509610,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68532,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303308,"statfs":{"total":96636764160,"available":96566587392,"internally_reserved":0,"allocated":1343488,"data_stored":582355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:18:06 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.50900000000000001,"15min":0.65000000000000002},"min":{"1min":0.27100000000000002,"5min":0.25800000000000001,"15min":0.23699999999999999},"max":{"1min":0.60199999999999998,"5min":4.1550000000000002,"15min":19.577000000000002},"last":0.5},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.63100000000000001,"15min":0.76200000000000001},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.251},"max":{"1min":0.81699999999999995,"5min":12.378,"15min":48.277999999999999},"last":0.61899999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:18:06 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.55300000000000005,"15min":0.70499999999999996},"min":{"1min":0.32100000000000001,"5min":0.309,"15min":0.30299999999999999},"max":{"1min":0.73399999999999999,"5min":2.8610000000000002,"15min":40.698},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.58699999999999997,"15min":0.86199999999999999},"min":{"1min":0.38100000000000001,"5min":0.33300000000000002,"15min":0.317},"max":{"1min":0.86299999999999999,"5min":2.5139999999999998,"15min":48.316000000000003},"last":0.58899999999999997}]},{"osd":3,"last update":"Mon Jan 15 18:18:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.628,"15min":0.82699999999999996},"min":{"1min":0.42899999999999999,"5min":0.36099999999999999,"15min":0.33700000000000002},"max":{"1min":0.91200000000000003,"5min":2.8620000000000001,"15min":48.234999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.68700000000000006,"15min":0.67300000000000004},"min":{"1min":0.371,"5min":0.315,"15min":0.311},"max":{"1min":0.79600000000000004,"5min":9.9719999999999995,"15min":9.9719999999999995},"last":0.52800000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:18:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.91100000000000003,"15min":0.89100000000000001},"min":{"1min":0.51400000000000001,"5min":0.46000000000000002,"15min":0.44500000000000001},"max":{"1min":1.0880000000000001,"5min":9.9329999999999998,"15min":9.9329999999999998},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.877,"15min":0.93100000000000005},"min":{"1min":0.496,"5min":0.45800000000000002,"15min":0.42499999999999999},"max":{"1min":1.1819999999999999,"5min":9.8610000000000007,"15min":19.829000000000001},"last":0.999}]},{"osd":5,"last update":"Mon Jan 15 18:18:06 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.85099999999999998,"15min":1.0129999999999999},"min":{"1min":0.56499999999999995,"5min":0.48699999999999999,"15min":0.46100000000000002},"max":{"1min":1.212,"5min":8.5839999999999996,"15min":22.594999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.81699999999999995,"15min":1.1559999999999999},"min":{"1min":0.53400000000000003,"5min":0.44400000000000001,"15min":0.443},"max":{"1min":1.1040000000000001,"5min":2.9609999999999999,"15min":48.198},"last":1.056}]},{"osd":6,"last update":"Mon Jan 15 18:18:06 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.92000000000000004,"15min":1.0409999999999999},"min":{"1min":0.58399999999999996,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.2350000000000001,"5min":9.3989999999999991,"15min":35.112000000000002},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.92700000000000005,"15min":0.96699999999999997},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.1539999999999999,"5min":5.8849999999999998,"15min":19.780999999999999},"last":0.878}]},{"osd":7,"last update":"Mon Jan 15 18:18:06 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.91000000000000003,"15min":1.1639999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.1890000000000001,"5min":11.039,"15min":48.148000000000003},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.98399999999999999,"15min":0.98999999999999999},"min":{"1min":0.57399999999999995,"5min":0.55800000000000005,"15min":0.52100000000000002},"max":{"1min":1.1559999999999999,"5min":9.8640000000000008,"15min":9.8640000000000008},"last":0.95599999999999996}]}]},{"osd":5,"up_from":14,"seq":60129542313,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58856,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312984,"statfs":{"total":96636764160,"available":96576495616,"internally_reserved":0,"allocated":1331200,"data_stored":576423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:18:23 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.77000000000000002,"15min":0.79400000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.28999999999999998},"max":{"1min":0.88700000000000001,"5min":5.1639999999999997,"15min":21.385000000000002},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.77100000000000002,"15min":0.86799999999999999},"min":{"1min":0.45100000000000001,"5min":0.378,"15min":0.34899999999999998},"max":{"1min":0.95099999999999996,"5min":5.3869999999999996,"15min":9.359},"last":0.69499999999999995}]},{"osd":1,"last update":"Mon Jan 15 18:18:23 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.76400000000000001,"15min":0.89200000000000002},"min":{"1min":0.45300000000000001,"5min":0.375,"15min":0.34899999999999998},"max":{"1min":0.92100000000000004,"5min":3.7320000000000002,"15min":21.120000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":1.0029999999999999,"15min":0.96799999999999997},"min":{"1min":0.42499999999999999,"5min":0.39800000000000002,"15min":0.315},"max":{"1min":0.93200000000000005,"5min":15.843,"15min":21.5},"last":0.73099999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:18:23 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.81299999999999994,"15min":0.89700000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.34999999999999998},"max":{"1min":1.7729999999999999,"5min":5.516,"15min":8.7750000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.876,"15min":0.97899999999999998},"min":{"1min":0.46000000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.86499999999999999,"5min":8.3529999999999998,"15min":21.201000000000001},"last":0.77900000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:18:23 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.89700000000000002,"15min":0.90400000000000003},"min":{"1min":0.47199999999999998,"5min":0.42799999999999999,"15min":0.37},"max":{"1min":0.96499999999999997,"5min":7.7199999999999998,"15min":21.457000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.82199999999999995,"15min":0.95399999999999996},"min":{"1min":0.52300000000000002,"5min":0.47699999999999998,"15min":0.45400000000000001},"max":{"1min":0.90800000000000003,"5min":5.6109999999999998,"15min":8.7300000000000004},"last":0.80700000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:18:23 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.90900000000000003,"15min":0.88200000000000001},"min":{"1min":0.41899999999999998,"5min":0.376,"15min":0.30399999999999999},"max":{"1min":0.88900000000000001,"5min":10.962,"15min":10.962},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.749,"15min":1.0089999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.76200000000000001,"5min":4.6639999999999997,"15min":26.146999999999998},"last":0.56200000000000006}]},{"osd":6,"last update":"Mon Jan 15 18:18:23 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.80800000000000005,"15min":0.83199999999999996},"min":{"1min":0.46300000000000002,"5min":0.39700000000000002,"15min":0.32000000000000001},"max":{"1min":1.0169999999999999,"5min":5.2160000000000002,"15min":21.535},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.80000000000000004,"15min":1.0029999999999999},"min":{"1min":0.44,"5min":0.432,"15min":0.36299999999999999},"max":{"1min":0.875,"5min":4.9139999999999997,"15min":30.488},"last":0.751}]},{"osd":7,"last update":"Mon Jan 15 18:18:23 2024","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.95899999999999996,"15min":1.2350000000000001},"min":{"1min":0.52200000000000002,"5min":0.47699999999999998,"15min":0.39700000000000002},"max":{"1min":4.7930000000000001,"5min":8.7539999999999996,"15min":33.499000000000002},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.91100000000000003,"15min":1.0820000000000001},"min":{"1min":0.51300000000000001,"5min":0.46200000000000002,"15min":0.36499999999999999},"max":{"1min":0.99199999999999999,"5min":6.306,"15min":25.437999999999999},"last":0.79400000000000004}]}]},{"osd":0,"up_from":14,"seq":60129542313,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64952,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306888,"statfs":{"total":96636764160,"available":96570253312,"internally_reserved":0,"allocated":1937408,"data_stored":1167911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:18:05 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.52500000000000002,"15min":0.624},"min":{"1min":0.32900000000000001,"5min":0.245,"15min":0.23699999999999999},"max":{"1min":0.67700000000000005,"5min":7.907,"15min":19.843},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.57599999999999996,"15min":0.66900000000000004},"min":{"1min":0.27300000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.69099999999999995,"5min":3.512,"15min":10.154},"last":0.54700000000000004}]},{"osd":2,"last update":"Mon Jan 15 18:18:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.71699999999999997,"15min":0.79700000000000004},"min":{"1min":0.35799999999999998,"5min":0.34100000000000003,"15min":0.30399999999999999},"max":{"1min":0.71999999999999997,"5min":7.0810000000000004,"15min":25.905000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.57699999999999996,"15min":0.61199999999999999},"min":{"1min":0.308,"5min":0.27800000000000002,"15min":0.26700000000000002},"max":{"1min":0.753,"5min":3.8500000000000001,"15min":9.3829999999999991},"last":0.44400000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:18:05 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.69399999999999995,"15min":0.73699999999999999},"min":{"1min":0.379,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66600000000000004,"5min":6.2850000000000001,"15min":9.1780000000000008},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.70699999999999996,"15min":0.75900000000000001},"min":{"1min":0.44400000000000001,"5min":0.38400000000000001,"15min":0.36899999999999999},"max":{"1min":0.77300000000000002,"5min":6.5979999999999999,"15min":10.210000000000001},"last":0.46800000000000003}]},{"osd":4,"last update":"Mon Jan 15 18:18:05 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.83799999999999997,"15min":0.97999999999999998},"min":{"1min":0.40100000000000002,"5min":0.39100000000000001,"15min":0.38},"max":{"1min":0.999,"5min":4.5179999999999998,"15min":22.814},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.82999999999999996,"15min":1.0229999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.41199999999999998},"max":{"1min":1.111,"5min":4.2329999999999997,"15min":23.341999999999999},"last":0.83499999999999996}]},{"osd":5,"last update":"Mon Jan 15 18:18:05 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.85399999999999998,"15min":0.96499999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.0880000000000001,"5min":4.008,"15min":12.428000000000001},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.75800000000000001,"5min":1.248,"15min":1.252},"min":{"1min":0.496,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.99099999999999999,"5min":23.021000000000001,"15min":23.021000000000001},"last":0.80500000000000005}]},{"osd":6,"last update":"Mon Jan 15 18:18:05 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.92100000000000004,"15min":1.21},"min":{"1min":0.52600000000000002,"5min":0.48399999999999999,"15min":0.46200000000000002},"max":{"1min":1.042,"5min":6.5250000000000004,"15min":31.059000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.94899999999999995,"15min":1.083},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.49099999999999999},"max":{"1min":1.0589999999999999,"5min":6.9690000000000003,"15min":40.731000000000002},"last":0.75}]},{"osd":7,"last update":"Mon Jan 15 18:18:05 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":1.1020000000000001,"15min":1.1950000000000001},"min":{"1min":0.50700000000000001,"5min":0.499,"15min":0.42499999999999999},"max":{"1min":1.014,"5min":10.497999999999999,"15min":24.492999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.88700000000000001,"15min":1.099},"min":{"1min":0.54300000000000004,"5min":0.495,"15min":0.46600000000000003},"max":{"1min":1.1459999999999999,"5min":4.6600000000000001,"15min":23.524000000000001},"last":0.68999999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:18:37.309 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:18:37.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.493+0000 7f59f9000640 1 Processor -- start 2024-01-15T18:18:37.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.493+0000 7f59f9000640 1 -- start start 2024-01-15T18:18:37.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.493+0000 7f59f9000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f59f415e7d0 0x7f59f40cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:37.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.493+0000 7f59f9000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59f40b20d0 0x7f59f40cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:37.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.493+0000 7f59f9000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f40b38b0 con 0x7f59f40cedc0 2024-01-15T18:18:37.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.493+0000 7f59f9000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59f40cd550 con 0x7f59f415e7d0 2024-01-15T18:18:37.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.493+0000 7f59f9000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59f40cd270 con 0x7f59f40b20d0 2024-01-15T18:18:37.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.493+0000 7f59f1400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f59f40cedc0 0x7f59f40cba90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38690/0 (socket says 172.21.15.27:38690) 2024-01-15T18:18:37.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.493+0000 7f59f1400640 1 -- 172.21.15.27:0/865287454 learned_addr learned my addr 172.21.15.27:0/865287454 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:37.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.493+0000 7f59f1e00640 1 --2- 172.21.15.27:0/865287454 >> v2:172.21.15.27:3300/0 conn(0x7f59f415e7d0 0x7f59f40cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:37.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.493+0000 7f59f2800640 1 --2- 172.21.15.27:0/865287454 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59f40b20d0 0x7f59f40cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:37.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.493+0000 7f59f1e00640 1 -- 172.21.15.27:0/865287454 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59f40b20d0 msgr2=0x7f59f40cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:37.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.493+0000 7f59f1e00640 1 --2- 172.21.15.27:0/865287454 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59f40b20d0 0x7f59f40cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:37.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.493+0000 7f59f1e00640 1 -- 172.21.15.27:0/865287454 >> v1:172.21.15.27:6789/0 conn(0x7f59f40cedc0 legacy=0x7f59f40cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:37.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.493+0000 7f59f1e00640 1 -- 172.21.15.27:0/865287454 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f40b35d0 con 0x7f59f415e7d0 2024-01-15T18:18:37.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.493+0000 7f59f0a00640 1 -- 172.21.15.27:0/865287454 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018752758 0 0) 0x7f59f40b38b0 con 0x7f59f40cedc0 2024-01-15T18:18:37.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.493+0000 7f59f2800640 1 --2- 172.21.15.27:0/865287454 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59f40b20d0 0x7f59f40cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:18:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.497+0000 7f59f1e00640 1 --2- 172.21.15.27:0/865287454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f415e7d0 0x7f59f40cf2a0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f59e40097c0 tx=0x7f59e401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=694f69ff9434169f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.497+0000 7f59f0a00640 1 -- 172.21.15.27:0/865287454 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e40ab600 con 0x7f59f415e7d0 2024-01-15T18:18:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.497+0000 7f59f0a00640 1 -- 172.21.15.27:0/865287454 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e40a7050 con 0x7f59f415e7d0 2024-01-15T18:18:37.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.497+0000 7f59f0a00640 1 -- 172.21.15.27:0/865287454 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e40ab980 con 0x7f59f415e7d0 2024-01-15T18:18:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.497+0000 7f59f9000640 1 -- 172.21.15.27:0/865287454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f415e7d0 msgr2=0x7f59f40cf2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.497+0000 7f59f9000640 1 --2- 172.21.15.27:0/865287454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f415e7d0 0x7f59f40cf2a0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f59e40097c0 tx=0x7f59e401c480 comp rx=0 tx=0).stop 2024-01-15T18:18:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.497+0000 7f59f9000640 1 -- 172.21.15.27:0/865287454 shutdown_connections 2024-01-15T18:18:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.497+0000 7f59f9000640 1 --2- 172.21.15.27:0/865287454 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59f40b20d0 0x7f59f40cf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.497+0000 7f59f9000640 1 --2- 172.21.15.27:0/865287454 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f415e7d0 0x7f59f40cf2a0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.497+0000 7f59f9000640 1 -- 172.21.15.27:0/865287454 >> 172.21.15.27:0/865287454 conn(0x7f59f40b2a10 msgr2=0x7f59f40b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:37.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.497+0000 7f59f9000640 1 -- 172.21.15.27:0/865287454 shutdown_connections 2024-01-15T18:18:37.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.497+0000 7f59f9000640 1 -- 172.21.15.27:0/865287454 wait complete. 2024-01-15T18:18:37.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.501+0000 7f59f9000640 1 Processor -- start 2024-01-15T18:18:37.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.501+0000 7f59f9000640 1 -- start start 2024-01-15T18:18:37.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.501+0000 7f59f9000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f40b20d0 0x7f59f414b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:37.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.501+0000 7f59f9000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59f40cedc0 0x7f59f41655d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:37.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.501+0000 7f59f9000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59f415e7d0 0x7f59f415ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:37.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.501+0000 7f59f9000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f414b9c0 con 0x7f59f40b20d0 2024-01-15T18:18:37.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.501+0000 7f59f9000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f4163cd0 con 0x7f59f415e7d0 2024-01-15T18:18:37.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.501+0000 7f59f9000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59f4163fb0 con 0x7f59f40cedc0 2024-01-15T18:18:37.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.501+0000 7f59f1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f40b20d0 0x7f59f414b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:37.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.501+0000 7f59f1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f40b20d0 0x7f59f414b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38126/0 (socket says 172.21.15.27:38126) 2024-01-15T18:18:37.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.501+0000 7f59f1e00640 1 -- 172.21.15.27:0/3522054264 learned_addr learned my addr 172.21.15.27:0/3522054264 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:37.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.501+0000 7f59f2800640 1 --2- 172.21.15.27:0/3522054264 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59f415e7d0 0x7f59f415ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:37.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.501+0000 7f59f1400640 1 --2- 172.21.15.27:0/3522054264 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59f40cedc0 0x7f59f41655d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:37.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.501+0000 7f59f1e00640 1 -- 172.21.15.27:0/3522054264 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59f40cedc0 msgr2=0x7f59f41655d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:37.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.501+0000 7f59f1e00640 1 --2- 172.21.15.27:0/3522054264 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59f40cedc0 0x7f59f41655d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:37.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.501+0000 7f59f1e00640 1 -- 172.21.15.27:0/3522054264 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59f415e7d0 msgr2=0x7f59f415ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:37.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.501+0000 7f59f1e00640 1 --2- 172.21.15.27:0/3522054264 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59f415e7d0 0x7f59f415ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:37.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.501+0000 7f59f1e00640 1 -- 172.21.15.27:0/3522054264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e4009420 con 0x7f59f40b20d0 2024-01-15T18:18:37.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.501+0000 7f59f2800640 1 --2- 172.21.15.27:0/3522054264 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59f415e7d0 0x7f59f415ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:18:37.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.501+0000 7f59f1e00640 1 --2- 172.21.15.27:0/3522054264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f40b20d0 0x7f59f414b480 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f59e401a0b0 tx=0x7f59e4019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:37.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.501+0000 7f59ea000640 1 -- 172.21.15.27:0/3522054264 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e40b4610 con 0x7f59f40b20d0 2024-01-15T18:18:37.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.501+0000 7f59ea000640 1 -- 172.21.15.27:0/3522054264 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e40abe90 con 0x7f59f40b20d0 2024-01-15T18:18:37.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.501+0000 7f59ea000640 1 -- 172.21.15.27:0/3522054264 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e40b49d0 con 0x7f59f40b20d0 2024-01-15T18:18:37.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.501+0000 7f59f9000640 1 -- 172.21.15.27:0/3522054264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f4164290 con 0x7f59f40b20d0 2024-01-15T18:18:37.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.501+0000 7f59f9000640 1 -- 172.21.15.27:0/3522054264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f4164850 con 0x7f59f40b20d0 2024-01-15T18:18:37.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.505+0000 7f59f9000640 1 -- 172.21.15.27:0/3522054264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f40d03f0 con 0x7f59f40b20d0 2024-01-15T18:18:37.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.505+0000 7f59ea000640 1 -- 172.21.15.27:0/3522054264 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f59e40b2040 con 0x7f59f40b20d0 2024-01-15T18:18:37.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.505+0000 7f59ea000640 1 --2- 172.21.15.27:0/3522054264 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f59cc067970 0x7f59cc069e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:37.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.505+0000 7f59ea000640 1 -- 172.21.15.27:0/3522054264 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f59e411d050 con 0x7f59f40b20d0 2024-01-15T18:18:37.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.505+0000 7f59f1400640 1 --2- 172.21.15.27:0/3522054264 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f59cc067970 0x7f59cc069e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:37.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.505+0000 7f59f1400640 1 --2- 172.21.15.27:0/3522054264 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f59cc067970 0x7f59cc069e30 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f59f4158250 tx=0x7f59dc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:37.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.509+0000 7f59ea000640 1 -- 172.21.15.27:0/3522054264 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f59e40ebd40 con 0x7f59f40b20d0 2024-01-15T18:18:37.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.661+0000 7f59f9000640 1 -- 172.21.15.27:0/3522054264 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f59f404e850 con 0x7f59cc067970 2024-01-15T18:18:37.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.673+0000 7f59ea000640 1 -- 172.21.15.27:0/3522054264 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289387 (secure 0 0 0) 0x7f59f404e850 con 0x7f59cc067970 2024-01-15T18:18:37.673 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:18:37.676 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:18:37.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.673+0000 7f59f9000640 1 -- 172.21.15.27:0/3522054264 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f59cc067970 msgr2=0x7f59cc069e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:37.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.673+0000 7f59f9000640 1 --2- 172.21.15.27:0/3522054264 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f59cc067970 0x7f59cc069e30 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f59f4158250 tx=0x7f59dc009040 comp rx=0 tx=0).stop 2024-01-15T18:18:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.673+0000 7f59f9000640 1 -- 172.21.15.27:0/3522054264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f40b20d0 msgr2=0x7f59f414b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.673+0000 7f59f9000640 1 --2- 172.21.15.27:0/3522054264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f40b20d0 0x7f59f414b480 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f59e401a0b0 tx=0x7f59e4019220 comp rx=0 tx=0).stop 2024-01-15T18:18:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.673+0000 7f59f9000640 1 -- 172.21.15.27:0/3522054264 shutdown_connections 2024-01-15T18:18:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.673+0000 7f59f9000640 1 --2- 172.21.15.27:0/3522054264 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f59cc067970 0x7f59cc069e30 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.673+0000 7f59f9000640 1 --2- 172.21.15.27:0/3522054264 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59f415e7d0 0x7f59f415ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.673+0000 7f59f9000640 1 --2- 172.21.15.27:0/3522054264 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59f40cedc0 0x7f59f41655d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.673+0000 7f59f9000640 1 --2- 172.21.15.27:0/3522054264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59f40b20d0 0x7f59f414b480 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.673+0000 7f59f9000640 1 -- 172.21.15.27:0/3522054264 >> 172.21.15.27:0/3522054264 conn(0x7f59f40b2a10 msgr2=0x7f59f415daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.673+0000 7f59f9000640 1 -- 172.21.15.27:0/3522054264 shutdown_connections 2024-01-15T18:18:37.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.673+0000 7f59f9000640 1 -- 172.21.15.27:0/3522054264 wait complete. 2024-01-15T18:18:37.695 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2024-01-15T18:18:36.373471+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":88,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":518912,"kb_used_data":11632,"kb_used_omap":12,"kb_used_meta":507123,"kb_avail":754455808,"statfs":{"total":773094113280,"available":772562747392,"internally_reserved":0,"allocated":11911168,"data_stored":5830580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":519294512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.186043"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337389+0000","last_change":"2024-01-15T18:18:33.168792+0000","last_active":"2024-01-15T18:18:34.337389+0000","last_peered":"2024-01-15T18:18:34.337389+0000","last_clean":"2024-01-15T18:18:34.337389+0000","last_became_active":"2024-01-15T18:18:33.168500+0000","last_became_peered":"2024-01-15T18:18:33.168500+0000","last_unstale":"2024-01-15T18:18:34.337389+0000","last_undegraded":"2024-01-15T18:18:34.337389+0000","last_fullsized":"2024-01-15T18:18:34.337389+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:28:41.940111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.144832+0000","last_change":"2024-01-15T18:18:33.151630+0000","last_active":"2024-01-15T18:18:34.144832+0000","last_peered":"2024-01-15T18:18:34.144832+0000","last_clean":"2024-01-15T18:18:34.144832+0000","last_became_active":"2024-01-15T18:18:33.151209+0000","last_became_peered":"2024-01-15T18:18:33.151209+0000","last_unstale":"2024-01-15T18:18:34.144832+0000","last_undegraded":"2024-01-15T18:18:34.144832+0000","last_fullsized":"2024-01-15T18:18:34.144832+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:46:54.687765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336528+0000","last_change":"2024-01-15T18:18:33.163094+0000","last_active":"2024-01-15T18:18:34.336528+0000","last_peered":"2024-01-15T18:18:34.336528+0000","last_clean":"2024-01-15T18:18:34.336528+0000","last_became_active":"2024-01-15T18:18:33.162700+0000","last_became_peered":"2024-01-15T18:18:33.162700+0000","last_unstale":"2024-01-15T18:18:34.336528+0000","last_undegraded":"2024-01-15T18:18:34.336528+0000","last_fullsized":"2024-01-15T18:18:34.336528+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:23:36.167165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337116+0000","last_change":"2024-01-15T18:18:33.160786+0000","last_active":"2024-01-15T18:18:34.337116+0000","last_peered":"2024-01-15T18:18:34.337116+0000","last_clean":"2024-01-15T18:18:34.337116+0000","last_became_active":"2024-01-15T18:18:33.160460+0000","last_became_peered":"2024-01-15T18:18:33.160460+0000","last_unstale":"2024-01-15T18:18:34.337116+0000","last_undegraded":"2024-01-15T18:18:34.337116+0000","last_fullsized":"2024-01-15T18:18:34.337116+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:27:33.980071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337976+0000","last_change":"2024-01-15T18:18:33.155936+0000","last_active":"2024-01-15T18:18:34.337976+0000","last_peered":"2024-01-15T18:18:34.337976+0000","last_clean":"2024-01-15T18:18:34.337976+0000","last_became_active":"2024-01-15T18:18:33.155656+0000","last_became_peered":"2024-01-15T18:18:33.155656+0000","last_unstale":"2024-01-15T18:18:34.337976+0000","last_undegraded":"2024-01-15T18:18:34.337976+0000","last_fullsized":"2024-01-15T18:18:34.337976+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:27:09.469255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145214+0000","last_change":"2024-01-15T18:18:33.152032+0000","last_active":"2024-01-15T18:18:34.145214+0000","last_peered":"2024-01-15T18:18:34.145214+0000","last_clean":"2024-01-15T18:18:34.145214+0000","last_became_active":"2024-01-15T18:18:33.151701+0000","last_became_peered":"2024-01-15T18:18:33.151701+0000","last_unstale":"2024-01-15T18:18:34.145214+0000","last_undegraded":"2024-01-15T18:18:34.145214+0000","last_fullsized":"2024-01-15T18:18:34.145214+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:51.983410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145175+0000","last_change":"2024-01-15T18:18:33.153882+0000","last_active":"2024-01-15T18:18:34.145175+0000","last_peered":"2024-01-15T18:18:34.145175+0000","last_clean":"2024-01-15T18:18:34.145175+0000","last_became_active":"2024-01-15T18:18:33.153264+0000","last_became_peered":"2024-01-15T18:18:33.153264+0000","last_unstale":"2024-01-15T18:18:34.145175+0000","last_undegraded":"2024-01-15T18:18:34.145175+0000","last_fullsized":"2024-01-15T18:18:34.145175+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:37:49.663025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.335705+0000","last_change":"2024-01-15T18:18:33.164714+0000","last_active":"2024-01-15T18:18:34.335705+0000","last_peered":"2024-01-15T18:18:34.335705+0000","last_clean":"2024-01-15T18:18:34.335705+0000","last_became_active":"2024-01-15T18:18:33.163896+0000","last_became_peered":"2024-01-15T18:18:33.163896+0000","last_unstale":"2024-01-15T18:18:34.335705+0000","last_undegraded":"2024-01-15T18:18:34.335705+0000","last_fullsized":"2024-01-15T18:18:34.335705+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:21:18.264946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337455+0000","last_change":"2024-01-15T18:18:33.158682+0000","last_active":"2024-01-15T18:18:34.337455+0000","last_peered":"2024-01-15T18:18:34.337455+0000","last_clean":"2024-01-15T18:18:34.337455+0000","last_became_active":"2024-01-15T18:18:33.158399+0000","last_became_peered":"2024-01-15T18:18:33.158399+0000","last_unstale":"2024-01-15T18:18:34.337455+0000","last_undegraded":"2024-01-15T18:18:34.337455+0000","last_fullsized":"2024-01-15T18:18:34.337455+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:27:12.032383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337118+0000","last_change":"2024-01-15T18:18:33.159936+0000","last_active":"2024-01-15T18:18:34.337118+0000","last_peered":"2024-01-15T18:18:34.337118+0000","last_clean":"2024-01-15T18:18:34.337118+0000","last_became_active":"2024-01-15T18:18:33.159562+0000","last_became_peered":"2024-01-15T18:18:33.159562+0000","last_unstale":"2024-01-15T18:18:34.337118+0000","last_undegraded":"2024-01-15T18:18:34.337118+0000","last_fullsized":"2024-01-15T18:18:34.337118+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:56:32.966770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.144621+0000","last_change":"2024-01-15T18:18:33.155247+0000","last_active":"2024-01-15T18:18:34.144621+0000","last_peered":"2024-01-15T18:18:34.144621+0000","last_clean":"2024-01-15T18:18:34.144621+0000","last_became_active":"2024-01-15T18:18:33.154972+0000","last_became_peered":"2024-01-15T18:18:33.154972+0000","last_unstale":"2024-01-15T18:18:34.144621+0000","last_undegraded":"2024-01-15T18:18:34.144621+0000","last_fullsized":"2024-01-15T18:18:34.144621+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:54:47.923576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337159+0000","last_change":"2024-01-15T18:18:33.158336+0000","last_active":"2024-01-15T18:18:34.337159+0000","last_peered":"2024-01-15T18:18:34.337159+0000","last_clean":"2024-01-15T18:18:34.337159+0000","last_became_active":"2024-01-15T18:18:33.158064+0000","last_became_peered":"2024-01-15T18:18:33.158064+0000","last_unstale":"2024-01-15T18:18:34.337159+0000","last_undegraded":"2024-01-15T18:18:34.337159+0000","last_fullsized":"2024-01-15T18:18:34.337159+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:42:44.182465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336372+0000","last_change":"2024-01-15T18:18:33.163912+0000","last_active":"2024-01-15T18:18:34.336372+0000","last_peered":"2024-01-15T18:18:34.336372+0000","last_clean":"2024-01-15T18:18:34.336372+0000","last_became_active":"2024-01-15T18:18:33.163610+0000","last_became_peered":"2024-01-15T18:18:33.163610+0000","last_unstale":"2024-01-15T18:18:34.336372+0000","last_undegraded":"2024-01-15T18:18:34.336372+0000","last_fullsized":"2024-01-15T18:18:34.336372+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:32:43.468374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336137+0000","last_change":"2024-01-15T18:18:33.169868+0000","last_active":"2024-01-15T18:18:34.336137+0000","last_peered":"2024-01-15T18:18:34.336137+0000","last_clean":"2024-01-15T18:18:34.336137+0000","last_became_active":"2024-01-15T18:18:33.169530+0000","last_became_peered":"2024-01-15T18:18:33.169530+0000","last_unstale":"2024-01-15T18:18:34.336137+0000","last_undegraded":"2024-01-15T18:18:34.336137+0000","last_fullsized":"2024-01-15T18:18:34.336137+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:18:22.473056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145111+0000","last_change":"2024-01-15T18:18:33.151325+0000","last_active":"2024-01-15T18:18:34.145111+0000","last_peered":"2024-01-15T18:18:34.145111+0000","last_clean":"2024-01-15T18:18:34.145111+0000","last_became_active":"2024-01-15T18:18:33.151040+0000","last_became_peered":"2024-01-15T18:18:33.151040+0000","last_unstale":"2024-01-15T18:18:34.145111+0000","last_undegraded":"2024-01-15T18:18:34.145111+0000","last_fullsized":"2024-01-15T18:18:34.145111+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:57:08.165322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336593+0000","last_change":"2024-01-15T18:18:33.166327+0000","last_active":"2024-01-15T18:18:34.336593+0000","last_peered":"2024-01-15T18:18:34.336593+0000","last_clean":"2024-01-15T18:18:34.336593+0000","last_became_active":"2024-01-15T18:18:33.166014+0000","last_became_peered":"2024-01-15T18:18:33.166014+0000","last_unstale":"2024-01-15T18:18:34.336593+0000","last_undegraded":"2024-01-15T18:18:34.336593+0000","last_fullsized":"2024-01-15T18:18:34.336593+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:37:50.428714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145535+0000","last_change":"2024-01-15T18:18:33.168613+0000","last_active":"2024-01-15T18:18:34.145535+0000","last_peered":"2024-01-15T18:18:34.145535+0000","last_clean":"2024-01-15T18:18:34.145535+0000","last_became_active":"2024-01-15T18:18:33.167975+0000","last_became_peered":"2024-01-15T18:18:33.167975+0000","last_unstale":"2024-01-15T18:18:34.145535+0000","last_undegraded":"2024-01-15T18:18:34.145535+0000","last_fullsized":"2024-01-15T18:18:34.145535+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:22:13.319815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.335745+0000","last_change":"2024-01-15T18:18:33.155078+0000","last_active":"2024-01-15T18:18:34.335745+0000","last_peered":"2024-01-15T18:18:34.335745+0000","last_clean":"2024-01-15T18:18:34.335745+0000","last_became_active":"2024-01-15T18:18:33.154816+0000","last_became_peered":"2024-01-15T18:18:33.154816+0000","last_unstale":"2024-01-15T18:18:34.335745+0000","last_undegraded":"2024-01-15T18:18:34.335745+0000","last_fullsized":"2024-01-15T18:18:34.335745+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:48.739168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336626+0000","last_change":"2024-01-15T18:18:33.164028+0000","last_active":"2024-01-15T18:18:34.336626+0000","last_peered":"2024-01-15T18:18:34.336626+0000","last_clean":"2024-01-15T18:18:34.336626+0000","last_became_active":"2024-01-15T18:18:33.163421+0000","last_became_peered":"2024-01-15T18:18:33.163421+0000","last_unstale":"2024-01-15T18:18:34.336626+0000","last_undegraded":"2024-01-15T18:18:34.336626+0000","last_fullsized":"2024-01-15T18:18:34.336626+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:38:31.615856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.144690+0000","last_change":"2024-01-15T18:18:33.160995+0000","last_active":"2024-01-15T18:18:34.144690+0000","last_peered":"2024-01-15T18:18:34.144690+0000","last_clean":"2024-01-15T18:18:34.144690+0000","last_became_active":"2024-01-15T18:18:33.160047+0000","last_became_peered":"2024-01-15T18:18:33.160047+0000","last_unstale":"2024-01-15T18:18:34.144690+0000","last_undegraded":"2024-01-15T18:18:34.144690+0000","last_fullsized":"2024-01-15T18:18:34.144690+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:39:53.520191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.144701+0000","last_change":"2024-01-15T18:18:33.152696+0000","last_active":"2024-01-15T18:18:34.144701+0000","last_peered":"2024-01-15T18:18:34.144701+0000","last_clean":"2024-01-15T18:18:34.144701+0000","last_became_active":"2024-01-15T18:18:33.152128+0000","last_became_peered":"2024-01-15T18:18:33.152128+0000","last_unstale":"2024-01-15T18:18:34.144701+0000","last_undegraded":"2024-01-15T18:18:34.144701+0000","last_fullsized":"2024-01-15T18:18:34.144701+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:08:18.706229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336224+0000","last_change":"2024-01-15T18:18:33.163614+0000","last_active":"2024-01-15T18:18:34.336224+0000","last_peered":"2024-01-15T18:18:34.336224+0000","last_clean":"2024-01-15T18:18:34.336224+0000","last_became_active":"2024-01-15T18:18:33.163252+0000","last_became_peered":"2024-01-15T18:18:33.163252+0000","last_unstale":"2024-01-15T18:18:34.336224+0000","last_undegraded":"2024-01-15T18:18:34.336224+0000","last_fullsized":"2024-01-15T18:18:34.336224+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:48.739168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.144172+0000","last_change":"2024-01-15T18:18:33.159896+0000","last_active":"2024-01-15T18:18:34.144172+0000","last_peered":"2024-01-15T18:18:34.144172+0000","last_clean":"2024-01-15T18:18:34.144172+0000","last_became_active":"2024-01-15T18:18:33.159557+0000","last_became_peered":"2024-01-15T18:18:33.159557+0000","last_unstale":"2024-01-15T18:18:34.144172+0000","last_undegraded":"2024-01-15T18:18:34.144172+0000","last_fullsized":"2024-01-15T18:18:34.144172+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:24.034308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336118+0000","last_change":"2024-01-15T18:18:33.166766+0000","last_active":"2024-01-15T18:18:34.336118+0000","last_peered":"2024-01-15T18:18:34.336118+0000","last_clean":"2024-01-15T18:18:34.336118+0000","last_became_active":"2024-01-15T18:18:33.165600+0000","last_became_peered":"2024-01-15T18:18:33.165600+0000","last_unstale":"2024-01-15T18:18:34.336118+0000","last_undegraded":"2024-01-15T18:18:34.336118+0000","last_fullsized":"2024-01-15T18:18:34.336118+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:03.948102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145680+0000","last_change":"2024-01-15T18:18:33.168720+0000","last_active":"2024-01-15T18:18:34.145680+0000","last_peered":"2024-01-15T18:18:34.145680+0000","last_clean":"2024-01-15T18:18:34.145680+0000","last_became_active":"2024-01-15T18:18:33.168349+0000","last_became_peered":"2024-01-15T18:18:33.168349+0000","last_unstale":"2024-01-15T18:18:34.145680+0000","last_undegraded":"2024-01-15T18:18:34.145680+0000","last_fullsized":"2024-01-15T18:18:34.145680+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:18:22.473056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337215+0000","last_change":"2024-01-15T18:18:33.159262+0000","last_active":"2024-01-15T18:18:34.337215+0000","last_peered":"2024-01-15T18:18:34.337215+0000","last_clean":"2024-01-15T18:18:34.337215+0000","last_became_active":"2024-01-15T18:18:33.158946+0000","last_became_peered":"2024-01-15T18:18:33.158946+0000","last_unstale":"2024-01-15T18:18:34.337215+0000","last_undegraded":"2024-01-15T18:18:34.337215+0000","last_fullsized":"2024-01-15T18:18:34.337215+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:05:55.347898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145408+0000","last_change":"2024-01-15T18:18:33.161814+0000","last_active":"2024-01-15T18:18:34.145408+0000","last_peered":"2024-01-15T18:18:34.145408+0000","last_clean":"2024-01-15T18:18:34.145408+0000","last_became_active":"2024-01-15T18:18:33.161463+0000","last_became_peered":"2024-01-15T18:18:33.161463+0000","last_unstale":"2024-01-15T18:18:34.145408+0000","last_undegraded":"2024-01-15T18:18:34.145408+0000","last_fullsized":"2024-01-15T18:18:34.145408+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:15:36.391855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337021+0000","last_change":"2024-01-15T18:18:33.162952+0000","last_active":"2024-01-15T18:18:34.337021+0000","last_peered":"2024-01-15T18:18:34.337021+0000","last_clean":"2024-01-15T18:18:34.337021+0000","last_became_active":"2024-01-15T18:18:33.162669+0000","last_became_peered":"2024-01-15T18:18:33.162669+0000","last_unstale":"2024-01-15T18:18:34.337021+0000","last_undegraded":"2024-01-15T18:18:34.337021+0000","last_fullsized":"2024-01-15T18:18:34.337021+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:47:59.088407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.144253+0000","last_change":"2024-01-15T18:18:33.160658+0000","last_active":"2024-01-15T18:18:34.144253+0000","last_peered":"2024-01-15T18:18:34.144253+0000","last_clean":"2024-01-15T18:18:34.144253+0000","last_became_active":"2024-01-15T18:18:33.160099+0000","last_became_peered":"2024-01-15T18:18:33.160099+0000","last_unstale":"2024-01-15T18:18:34.144253+0000","last_undegraded":"2024-01-15T18:18:34.144253+0000","last_fullsized":"2024-01-15T18:18:34.144253+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:47:59.088407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145009+0000","last_change":"2024-01-15T18:18:33.161061+0000","last_active":"2024-01-15T18:18:34.145009+0000","last_peered":"2024-01-15T18:18:34.145009+0000","last_clean":"2024-01-15T18:18:34.145009+0000","last_became_active":"2024-01-15T18:18:33.160725+0000","last_became_peered":"2024-01-15T18:18:33.160725+0000","last_unstale":"2024-01-15T18:18:34.145009+0000","last_undegraded":"2024-01-15T18:18:34.145009+0000","last_fullsized":"2024-01-15T18:18:34.145009+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:38:07.236956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336723+0000","last_change":"2024-01-15T18:18:33.159024+0000","last_active":"2024-01-15T18:18:34.336723+0000","last_peered":"2024-01-15T18:18:34.336723+0000","last_clean":"2024-01-15T18:18:34.336723+0000","last_became_active":"2024-01-15T18:18:33.158701+0000","last_became_peered":"2024-01-15T18:18:33.158701+0000","last_unstale":"2024-01-15T18:18:34.336723+0000","last_undegraded":"2024-01-15T18:18:34.336723+0000","last_fullsized":"2024-01-15T18:18:34.336723+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:50:16.189329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145092+0000","last_change":"2024-01-15T18:18:33.152241+0000","last_active":"2024-01-15T18:18:34.145092+0000","last_peered":"2024-01-15T18:18:34.145092+0000","last_clean":"2024-01-15T18:18:34.145092+0000","last_became_active":"2024-01-15T18:18:33.151250+0000","last_became_peered":"2024-01-15T18:18:33.151250+0000","last_unstale":"2024-01-15T18:18:34.145092+0000","last_undegraded":"2024-01-15T18:18:34.145092+0000","last_fullsized":"2024-01-15T18:18:34.145092+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:45:02.525155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145574+0000","last_change":"2024-01-15T18:18:33.154972+0000","last_active":"2024-01-15T18:18:34.145574+0000","last_peered":"2024-01-15T18:18:34.145574+0000","last_clean":"2024-01-15T18:18:34.145574+0000","last_became_active":"2024-01-15T18:18:33.154569+0000","last_became_peered":"2024-01-15T18:18:33.154569+0000","last_unstale":"2024-01-15T18:18:34.145574+0000","last_undegraded":"2024-01-15T18:18:34.145574+0000","last_fullsized":"2024-01-15T18:18:34.145574+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:29:02.332324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"107'4","reported_seq":16,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145441+0000","last_change":"2024-01-15T18:18:32.217761+0000","last_active":"2024-01-15T18:18:34.145441+0000","last_peered":"2024-01-15T18:18:34.145441+0000","last_clean":"2024-01-15T18:18:34.145441+0000","last_became_active":"2024-01-15T18:18:32.216337+0000","last_became_peered":"2024-01-15T18:18:32.216337+0000","last_unstale":"2024-01-15T18:18:34.145441+0000","last_undegraded":"2024-01-15T18:18:34.145441+0000","last_fullsized":"2024-01-15T18:18:34.145441+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:01:18.991818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.335822+0000","last_change":"2024-01-15T18:18:33.167015+0000","last_active":"2024-01-15T18:18:34.335822+0000","last_peered":"2024-01-15T18:18:34.335822+0000","last_clean":"2024-01-15T18:18:34.335822+0000","last_became_active":"2024-01-15T18:18:33.166726+0000","last_became_peered":"2024-01-15T18:18:33.166726+0000","last_unstale":"2024-01-15T18:18:34.335822+0000","last_undegraded":"2024-01-15T18:18:34.335822+0000","last_fullsized":"2024-01-15T18:18:34.335822+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:48:57.895180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"107'1","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336500+0000","last_change":"2024-01-15T18:18:32.219987+0000","last_active":"2024-01-15T18:18:34.336500+0000","last_peered":"2024-01-15T18:18:34.336500+0000","last_clean":"2024-01-15T18:18:34.336500+0000","last_became_active":"2024-01-15T18:18:32.219596+0000","last_became_peered":"2024-01-15T18:18:32.219596+0000","last_unstale":"2024-01-15T18:18:34.336500+0000","last_undegraded":"2024-01-15T18:18:34.336500+0000","last_fullsized":"2024-01-15T18:18:34.336500+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:23.024473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145427+0000","last_change":"2024-01-15T18:18:33.150743+0000","last_active":"2024-01-15T18:18:34.145427+0000","last_peered":"2024-01-15T18:18:34.145427+0000","last_clean":"2024-01-15T18:18:34.145427+0000","last_became_active":"2024-01-15T18:18:33.150429+0000","last_became_peered":"2024-01-15T18:18:33.150429+0000","last_unstale":"2024-01-15T18:18:34.145427+0000","last_undegraded":"2024-01-15T18:18:34.145427+0000","last_fullsized":"2024-01-15T18:18:34.145427+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:33:18.994479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"107'2","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337825+0000","last_change":"2024-01-15T18:18:32.213207+0000","last_active":"2024-01-15T18:18:34.337825+0000","last_peered":"2024-01-15T18:18:34.337825+0000","last_clean":"2024-01-15T18:18:34.337825+0000","last_became_active":"2024-01-15T18:18:32.212071+0000","last_became_peered":"2024-01-15T18:18:32.212071+0000","last_unstale":"2024-01-15T18:18:34.337825+0000","last_undegraded":"2024-01-15T18:18:34.337825+0000","last_fullsized":"2024-01-15T18:18:34.337825+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:29:14.612143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336611+0000","last_change":"2024-01-15T18:18:33.165545+0000","last_active":"2024-01-15T18:18:34.336611+0000","last_peered":"2024-01-15T18:18:34.336611+0000","last_clean":"2024-01-15T18:18:34.336611+0000","last_became_active":"2024-01-15T18:18:33.165258+0000","last_became_peered":"2024-01-15T18:18:33.165258+0000","last_unstale":"2024-01-15T18:18:34.336611+0000","last_undegraded":"2024-01-15T18:18:34.336611+0000","last_fullsized":"2024-01-15T18:18:34.336611+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:05:07.453809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"107'2","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337766+0000","last_change":"2024-01-15T18:18:32.217238+0000","last_active":"2024-01-15T18:18:34.337766+0000","last_peered":"2024-01-15T18:18:34.337766+0000","last_clean":"2024-01-15T18:18:34.337766+0000","last_became_active":"2024-01-15T18:18:32.216492+0000","last_became_peered":"2024-01-15T18:18:32.216492+0000","last_unstale":"2024-01-15T18:18:34.337766+0000","last_undegraded":"2024-01-15T18:18:34.337766+0000","last_fullsized":"2024-01-15T18:18:34.337766+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:40:25.010807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336803+0000","last_change":"2024-01-15T18:18:33.157996+0000","last_active":"2024-01-15T18:18:34.336803+0000","last_peered":"2024-01-15T18:18:34.336803+0000","last_clean":"2024-01-15T18:18:34.336803+0000","last_became_active":"2024-01-15T18:18:33.157687+0000","last_became_peered":"2024-01-15T18:18:33.157687+0000","last_unstale":"2024-01-15T18:18:34.336803+0000","last_undegraded":"2024-01-15T18:18:34.336803+0000","last_fullsized":"2024-01-15T18:18:34.336803+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:26:16.219599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"107'1","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145053+0000","last_change":"2024-01-15T18:18:32.215007+0000","last_active":"2024-01-15T18:18:34.145053+0000","last_peered":"2024-01-15T18:18:34.145053+0000","last_clean":"2024-01-15T18:18:34.145053+0000","last_became_active":"2024-01-15T18:18:32.214453+0000","last_became_peered":"2024-01-15T18:18:32.214453+0000","last_unstale":"2024-01-15T18:18:34.145053+0000","last_undegraded":"2024-01-15T18:18:34.145053+0000","last_fullsized":"2024-01-15T18:18:34.145053+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:58:50.916786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.144755+0000","last_change":"2024-01-15T18:18:33.153355+0000","last_active":"2024-01-15T18:18:34.144755+0000","last_peered":"2024-01-15T18:18:34.144755+0000","last_clean":"2024-01-15T18:18:34.144755+0000","last_became_active":"2024-01-15T18:18:33.153058+0000","last_became_peered":"2024-01-15T18:18:33.153058+0000","last_unstale":"2024-01-15T18:18:34.144755+0000","last_undegraded":"2024-01-15T18:18:34.144755+0000","last_fullsized":"2024-01-15T18:18:34.144755+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:21:18.264946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"107'1","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336335+0000","last_change":"2024-01-15T18:18:32.215681+0000","last_active":"2024-01-15T18:18:34.336335+0000","last_peered":"2024-01-15T18:18:34.336335+0000","last_clean":"2024-01-15T18:18:34.336335+0000","last_became_active":"2024-01-15T18:18:32.215417+0000","last_became_peered":"2024-01-15T18:18:32.215417+0000","last_unstale":"2024-01-15T18:18:34.336335+0000","last_undegraded":"2024-01-15T18:18:34.336335+0000","last_fullsized":"2024-01-15T18:18:34.336335+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:37:03.114460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336661+0000","last_change":"2024-01-15T18:18:33.161879+0000","last_active":"2024-01-15T18:18:34.336661+0000","last_peered":"2024-01-15T18:18:34.336661+0000","last_clean":"2024-01-15T18:18:34.336661+0000","last_became_active":"2024-01-15T18:18:33.161619+0000","last_became_peered":"2024-01-15T18:18:33.161619+0000","last_unstale":"2024-01-15T18:18:34.336661+0000","last_undegraded":"2024-01-15T18:18:34.336661+0000","last_fullsized":"2024-01-15T18:18:34.336661+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:59:11.477105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.144264+0000","last_change":"2024-01-15T18:18:32.217097+0000","last_active":"2024-01-15T18:18:34.144264+0000","last_peered":"2024-01-15T18:18:34.144264+0000","last_clean":"2024-01-15T18:18:34.144264+0000","last_became_active":"2024-01-15T18:18:32.215465+0000","last_became_peered":"2024-01-15T18:18:32.215465+0000","last_unstale":"2024-01-15T18:18:34.144264+0000","last_undegraded":"2024-01-15T18:18:34.144264+0000","last_fullsized":"2024-01-15T18:18:34.144264+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:47.729333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145835+0000","last_change":"2024-01-15T18:18:33.154067+0000","last_active":"2024-01-15T18:18:34.145835+0000","last_peered":"2024-01-15T18:18:34.145835+0000","last_clean":"2024-01-15T18:18:34.145835+0000","last_became_active":"2024-01-15T18:18:33.153610+0000","last_became_peered":"2024-01-15T18:18:33.153610+0000","last_unstale":"2024-01-15T18:18:34.145835+0000","last_undegraded":"2024-01-15T18:18:34.145835+0000","last_fullsized":"2024-01-15T18:18:34.145835+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:08:41.862115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336688+0000","last_change":"2024-01-15T18:18:32.219909+0000","last_active":"2024-01-15T18:18:34.336688+0000","last_peered":"2024-01-15T18:18:34.336688+0000","last_clean":"2024-01-15T18:18:34.336688+0000","last_became_active":"2024-01-15T18:18:32.219555+0000","last_became_peered":"2024-01-15T18:18:32.219555+0000","last_unstale":"2024-01-15T18:18:34.336688+0000","last_undegraded":"2024-01-15T18:18:34.336688+0000","last_fullsized":"2024-01-15T18:18:34.336688+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:17:21.343043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145567+0000","last_change":"2024-01-15T18:18:33.156038+0000","last_active":"2024-01-15T18:18:34.145567+0000","last_peered":"2024-01-15T18:18:34.145567+0000","last_clean":"2024-01-15T18:18:34.145567+0000","last_became_active":"2024-01-15T18:18:33.155765+0000","last_became_peered":"2024-01-15T18:18:33.155765+0000","last_unstale":"2024-01-15T18:18:34.145567+0000","last_undegraded":"2024-01-15T18:18:34.145567+0000","last_fullsized":"2024-01-15T18:18:34.145567+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:58:43.210443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337824+0000","last_change":"2024-01-15T18:18:32.212880+0000","last_active":"2024-01-15T18:18:34.337824+0000","last_peered":"2024-01-15T18:18:34.337824+0000","last_clean":"2024-01-15T18:18:34.337824+0000","last_became_active":"2024-01-15T18:18:32.212511+0000","last_became_peered":"2024-01-15T18:18:32.212511+0000","last_unstale":"2024-01-15T18:18:34.337824+0000","last_undegraded":"2024-01-15T18:18:34.337824+0000","last_fullsized":"2024-01-15T18:18:34.337824+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:58:38.191140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.335918+0000","last_change":"2024-01-15T18:18:33.164800+0000","last_active":"2024-01-15T18:18:34.335918+0000","last_peered":"2024-01-15T18:18:34.335918+0000","last_clean":"2024-01-15T18:18:34.335918+0000","last_became_active":"2024-01-15T18:18:33.164530+0000","last_became_peered":"2024-01-15T18:18:33.164530+0000","last_unstale":"2024-01-15T18:18:34.335918+0000","last_undegraded":"2024-01-15T18:18:34.335918+0000","last_fullsized":"2024-01-15T18:18:34.335918+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:55.175065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336668+0000","last_change":"2024-01-15T18:18:32.216873+0000","last_active":"2024-01-15T18:18:34.336668+0000","last_peered":"2024-01-15T18:18:34.336668+0000","last_clean":"2024-01-15T18:18:34.336668+0000","last_became_active":"2024-01-15T18:18:32.216337+0000","last_became_peered":"2024-01-15T18:18:32.216337+0000","last_unstale":"2024-01-15T18:18:34.336668+0000","last_undegraded":"2024-01-15T18:18:34.336668+0000","last_fullsized":"2024-01-15T18:18:34.336668+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:02.176575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336850+0000","last_change":"2024-01-15T18:18:33.148278+0000","last_active":"2024-01-15T18:18:34.336850+0000","last_peered":"2024-01-15T18:18:34.336850+0000","last_clean":"2024-01-15T18:18:34.336850+0000","last_became_active":"2024-01-15T18:18:33.148016+0000","last_became_peered":"2024-01-15T18:18:33.148016+0000","last_unstale":"2024-01-15T18:18:34.336850+0000","last_undegraded":"2024-01-15T18:18:34.336850+0000","last_fullsized":"2024-01-15T18:18:34.336850+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:24.034308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"107'3","reported_seq":15,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337309+0000","last_change":"2024-01-15T18:18:32.214473+0000","last_active":"2024-01-15T18:18:34.337309+0000","last_peered":"2024-01-15T18:18:34.337309+0000","last_clean":"2024-01-15T18:18:34.337309+0000","last_became_active":"2024-01-15T18:18:32.214186+0000","last_became_peered":"2024-01-15T18:18:32.214186+0000","last_unstale":"2024-01-15T18:18:34.337309+0000","last_undegraded":"2024-01-15T18:18:34.337309+0000","last_fullsized":"2024-01-15T18:18:34.337309+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:15:38.528251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145358+0000","last_change":"2024-01-15T18:18:33.159329+0000","last_active":"2024-01-15T18:18:34.145358+0000","last_peered":"2024-01-15T18:18:34.145358+0000","last_clean":"2024-01-15T18:18:34.145358+0000","last_became_active":"2024-01-15T18:18:33.158541+0000","last_became_peered":"2024-01-15T18:18:33.158541+0000","last_unstale":"2024-01-15T18:18:34.145358+0000","last_undegraded":"2024-01-15T18:18:34.145358+0000","last_fullsized":"2024-01-15T18:18:34.145358+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:53:07.912473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"107'6","reported_seq":18,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337256+0000","last_change":"2024-01-15T18:18:32.218321+0000","last_active":"2024-01-15T18:18:34.337256+0000","last_peered":"2024-01-15T18:18:34.337256+0000","last_clean":"2024-01-15T18:18:34.337256+0000","last_became_active":"2024-01-15T18:18:32.218002+0000","last_became_peered":"2024-01-15T18:18:32.218002+0000","last_unstale":"2024-01-15T18:18:34.337256+0000","last_undegraded":"2024-01-15T18:18:34.337256+0000","last_fullsized":"2024-01-15T18:18:34.337256+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:12:40.196083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336362+0000","last_change":"2024-01-15T18:18:33.151414+0000","last_active":"2024-01-15T18:18:34.336362+0000","last_peered":"2024-01-15T18:18:34.336362+0000","last_clean":"2024-01-15T18:18:34.336362+0000","last_became_active":"2024-01-15T18:18:33.151132+0000","last_became_peered":"2024-01-15T18:18:33.151132+0000","last_unstale":"2024-01-15T18:18:34.336362+0000","last_undegraded":"2024-01-15T18:18:34.336362+0000","last_fullsized":"2024-01-15T18:18:34.336362+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:48:57.895180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"107'2","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145300+0000","last_change":"2024-01-15T18:18:32.215335+0000","last_active":"2024-01-15T18:18:34.145300+0000","last_peered":"2024-01-15T18:18:34.145300+0000","last_clean":"2024-01-15T18:18:34.145300+0000","last_became_active":"2024-01-15T18:18:32.215073+0000","last_became_peered":"2024-01-15T18:18:32.215073+0000","last_unstale":"2024-01-15T18:18:34.145300+0000","last_undegraded":"2024-01-15T18:18:34.145300+0000","last_fullsized":"2024-01-15T18:18:34.145300+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:04:07.912459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336493+0000","last_change":"2024-01-15T18:18:33.161775+0000","last_active":"2024-01-15T18:18:34.336493+0000","last_peered":"2024-01-15T18:18:34.336493+0000","last_clean":"2024-01-15T18:18:34.336493+0000","last_became_active":"2024-01-15T18:18:33.161493+0000","last_became_peered":"2024-01-15T18:18:33.161493+0000","last_unstale":"2024-01-15T18:18:34.336493+0000","last_undegraded":"2024-01-15T18:18:34.336493+0000","last_fullsized":"2024-01-15T18:18:34.336493+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:55.175065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336025+0000","last_change":"2024-01-15T18:18:32.215225+0000","last_active":"2024-01-15T18:18:34.336025+0000","last_peered":"2024-01-15T18:18:34.336025+0000","last_clean":"2024-01-15T18:18:34.336025+0000","last_became_active":"2024-01-15T18:18:32.214942+0000","last_became_peered":"2024-01-15T18:18:32.214942+0000","last_unstale":"2024-01-15T18:18:34.336025+0000","last_undegraded":"2024-01-15T18:18:34.336025+0000","last_fullsized":"2024-01-15T18:18:34.336025+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:09:53.884883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336155+0000","last_change":"2024-01-15T18:18:33.146254+0000","last_active":"2024-01-15T18:18:34.336155+0000","last_peered":"2024-01-15T18:18:34.336155+0000","last_clean":"2024-01-15T18:18:34.336155+0000","last_became_active":"2024-01-15T18:18:33.145973+0000","last_became_peered":"2024-01-15T18:18:33.145973+0000","last_unstale":"2024-01-15T18:18:34.336155+0000","last_undegraded":"2024-01-15T18:18:34.336155+0000","last_fullsized":"2024-01-15T18:18:34.336155+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:09:54.894718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"107'8","reported_seq":20,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337489+0000","last_change":"2024-01-15T18:18:32.219579+0000","last_active":"2024-01-15T18:18:34.337489+0000","last_peered":"2024-01-15T18:18:34.337489+0000","last_clean":"2024-01-15T18:18:34.337489+0000","last_became_active":"2024-01-15T18:18:32.219280+0000","last_became_peered":"2024-01-15T18:18:32.219280+0000","last_unstale":"2024-01-15T18:18:34.337489+0000","last_undegraded":"2024-01-15T18:18:34.337489+0000","last_fullsized":"2024-01-15T18:18:34.337489+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:56:09.818155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.338016+0000","last_change":"2024-01-15T18:18:33.161674+0000","last_active":"2024-01-15T18:18:34.338016+0000","last_peered":"2024-01-15T18:18:34.338016+0000","last_clean":"2024-01-15T18:18:34.338016+0000","last_became_active":"2024-01-15T18:18:33.161329+0000","last_became_peered":"2024-01-15T18:18:33.161329+0000","last_unstale":"2024-01-15T18:18:34.338016+0000","last_undegraded":"2024-01-15T18:18:34.338016+0000","last_fullsized":"2024-01-15T18:18:34.338016+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:45:56.254343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"107'1","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337595+0000","last_change":"2024-01-15T18:18:32.215848+0000","last_active":"2024-01-15T18:18:34.337595+0000","last_peered":"2024-01-15T18:18:34.337595+0000","last_clean":"2024-01-15T18:18:34.337595+0000","last_became_active":"2024-01-15T18:18:32.214009+0000","last_became_peered":"2024-01-15T18:18:32.214009+0000","last_unstale":"2024-01-15T18:18:34.337595+0000","last_undegraded":"2024-01-15T18:18:34.337595+0000","last_fullsized":"2024-01-15T18:18:34.337595+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:48:09.722577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145635+0000","last_change":"2024-01-15T18:18:33.162974+0000","last_active":"2024-01-15T18:18:34.145635+0000","last_peered":"2024-01-15T18:18:34.145635+0000","last_clean":"2024-01-15T18:18:34.145635+0000","last_became_active":"2024-01-15T18:18:33.162679+0000","last_became_peered":"2024-01-15T18:18:33.162679+0000","last_unstale":"2024-01-15T18:18:34.145635+0000","last_undegraded":"2024-01-15T18:18:34.145635+0000","last_fullsized":"2024-01-15T18:18:34.145635+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:53:15.036059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"107'2","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337993+0000","last_change":"2024-01-15T18:18:32.214990+0000","last_active":"2024-01-15T18:18:34.337993+0000","last_peered":"2024-01-15T18:18:34.337993+0000","last_clean":"2024-01-15T18:18:34.337993+0000","last_became_active":"2024-01-15T18:18:32.214695+0000","last_became_peered":"2024-01-15T18:18:32.214695+0000","last_unstale":"2024-01-15T18:18:34.337993+0000","last_undegraded":"2024-01-15T18:18:34.337993+0000","last_fullsized":"2024-01-15T18:18:34.337993+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:15:08.445774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336467+0000","last_change":"2024-01-15T18:18:33.168531+0000","last_active":"2024-01-15T18:18:34.336467+0000","last_peered":"2024-01-15T18:18:34.336467+0000","last_clean":"2024-01-15T18:18:34.336467+0000","last_became_active":"2024-01-15T18:18:33.168190+0000","last_became_peered":"2024-01-15T18:18:33.168190+0000","last_unstale":"2024-01-15T18:18:34.336467+0000","last_undegraded":"2024-01-15T18:18:34.336467+0000","last_fullsized":"2024-01-15T18:18:34.336467+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:53:05.325329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"107'1","reported_seq":13,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.335781+0000","last_change":"2024-01-15T18:18:32.221200+0000","last_active":"2024-01-15T18:18:34.335781+0000","last_peered":"2024-01-15T18:18:34.335781+0000","last_clean":"2024-01-15T18:18:34.335781+0000","last_became_active":"2024-01-15T18:18:32.220884+0000","last_became_peered":"2024-01-15T18:18:32.220884+0000","last_unstale":"2024-01-15T18:18:34.335781+0000","last_undegraded":"2024-01-15T18:18:34.335781+0000","last_fullsized":"2024-01-15T18:18:34.335781+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:02.938267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336678+0000","last_change":"2024-01-15T18:18:33.163114+0000","last_active":"2024-01-15T18:18:34.336678+0000","last_peered":"2024-01-15T18:18:34.336678+0000","last_clean":"2024-01-15T18:18:34.336678+0000","last_became_active":"2024-01-15T18:18:33.162424+0000","last_became_peered":"2024-01-15T18:18:33.162424+0000","last_unstale":"2024-01-15T18:18:34.336678+0000","last_undegraded":"2024-01-15T18:18:34.336678+0000","last_fullsized":"2024-01-15T18:18:34.336678+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:42:08.107694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145019+0000","last_change":"2024-01-15T18:18:32.217252+0000","last_active":"2024-01-15T18:18:34.145019+0000","last_peered":"2024-01-15T18:18:34.145019+0000","last_clean":"2024-01-15T18:18:34.145019+0000","last_became_active":"2024-01-15T18:18:32.216769+0000","last_became_peered":"2024-01-15T18:18:32.216769+0000","last_unstale":"2024-01-15T18:18:34.145019+0000","last_undegraded":"2024-01-15T18:18:34.145019+0000","last_fullsized":"2024-01-15T18:18:34.145019+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:36.807888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336194+0000","last_change":"2024-01-15T18:18:33.163017+0000","last_active":"2024-01-15T18:18:34.336194+0000","last_peered":"2024-01-15T18:18:34.336194+0000","last_clean":"2024-01-15T18:18:34.336194+0000","last_became_active":"2024-01-15T18:18:33.162658+0000","last_became_peered":"2024-01-15T18:18:33.162658+0000","last_unstale":"2024-01-15T18:18:34.336194+0000","last_undegraded":"2024-01-15T18:18:34.336194+0000","last_fullsized":"2024-01-15T18:18:34.336194+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:01:41.498479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"107'2","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145329+0000","last_change":"2024-01-15T18:18:32.207796+0000","last_active":"2024-01-15T18:18:34.145329+0000","last_peered":"2024-01-15T18:18:34.145329+0000","last_clean":"2024-01-15T18:18:34.145329+0000","last_became_active":"2024-01-15T18:18:32.207507+0000","last_became_peered":"2024-01-15T18:18:32.207507+0000","last_unstale":"2024-01-15T18:18:34.145329+0000","last_undegraded":"2024-01-15T18:18:34.145329+0000","last_fullsized":"2024-01-15T18:18:34.145329+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:42.592200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336888+0000","last_change":"2024-01-15T18:18:33.164981+0000","last_active":"2024-01-15T18:18:34.336888+0000","last_peered":"2024-01-15T18:18:34.336888+0000","last_clean":"2024-01-15T18:18:34.336888+0000","last_became_active":"2024-01-15T18:18:33.164206+0000","last_became_peered":"2024-01-15T18:18:33.164206+0000","last_unstale":"2024-01-15T18:18:34.336888+0000","last_undegraded":"2024-01-15T18:18:34.336888+0000","last_fullsized":"2024-01-15T18:18:34.336888+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:53:32.550071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337629+0000","last_change":"2024-01-15T18:18:32.212499+0000","last_active":"2024-01-15T18:18:34.337629+0000","last_peered":"2024-01-15T18:18:34.337629+0000","last_clean":"2024-01-15T18:18:34.337629+0000","last_became_active":"2024-01-15T18:18:32.212201+0000","last_became_peered":"2024-01-15T18:18:32.212201+0000","last_unstale":"2024-01-15T18:18:34.337629+0000","last_undegraded":"2024-01-15T18:18:34.337629+0000","last_fullsized":"2024-01-15T18:18:34.337629+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:25:39.031903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.335829+0000","last_change":"2024-01-15T18:18:33.165046+0000","last_active":"2024-01-15T18:18:34.335829+0000","last_peered":"2024-01-15T18:18:34.335829+0000","last_clean":"2024-01-15T18:18:34.335829+0000","last_became_active":"2024-01-15T18:18:33.164708+0000","last_became_peered":"2024-01-15T18:18:33.164708+0000","last_unstale":"2024-01-15T18:18:34.335829+0000","last_undegraded":"2024-01-15T18:18:34.335829+0000","last_fullsized":"2024-01-15T18:18:34.335829+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:46:54.687765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145456+0000","last_change":"2024-01-15T18:18:32.207872+0000","last_active":"2024-01-15T18:18:34.145456+0000","last_peered":"2024-01-15T18:18:34.145456+0000","last_clean":"2024-01-15T18:18:34.145456+0000","last_became_active":"2024-01-15T18:18:32.207599+0000","last_became_peered":"2024-01-15T18:18:32.207599+0000","last_unstale":"2024-01-15T18:18:34.145456+0000","last_undegraded":"2024-01-15T18:18:34.145456+0000","last_fullsized":"2024-01-15T18:18:34.145456+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:25:20.009100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336145+0000","last_change":"2024-01-15T18:18:33.160618+0000","last_active":"2024-01-15T18:18:34.336145+0000","last_peered":"2024-01-15T18:18:34.336145+0000","last_clean":"2024-01-15T18:18:34.336145+0000","last_became_active":"2024-01-15T18:18:33.160350+0000","last_became_peered":"2024-01-15T18:18:33.160350+0000","last_unstale":"2024-01-15T18:18:34.336145+0000","last_undegraded":"2024-01-15T18:18:34.336145+0000","last_fullsized":"2024-01-15T18:18:34.336145+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:42:11.703397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336755+0000","last_change":"2024-01-15T18:18:32.217214+0000","last_active":"2024-01-15T18:18:34.336755+0000","last_peered":"2024-01-15T18:18:34.336755+0000","last_clean":"2024-01-15T18:18:34.336755+0000","last_became_active":"2024-01-15T18:18:32.216581+0000","last_became_peered":"2024-01-15T18:18:32.216581+0000","last_unstale":"2024-01-15T18:18:34.336755+0000","last_undegraded":"2024-01-15T18:18:34.336755+0000","last_fullsized":"2024-01-15T18:18:34.336755+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:49:26.780801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.335999+0000","last_change":"2024-01-15T18:18:33.165549+0000","last_active":"2024-01-15T18:18:34.335999+0000","last_peered":"2024-01-15T18:18:34.335999+0000","last_clean":"2024-01-15T18:18:34.335999+0000","last_became_active":"2024-01-15T18:18:33.164152+0000","last_became_peered":"2024-01-15T18:18:33.164152+0000","last_unstale":"2024-01-15T18:18:34.335999+0000","last_undegraded":"2024-01-15T18:18:34.335999+0000","last_fullsized":"2024-01-15T18:18:34.335999+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:24.034308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"107'2","reported_seq":14,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337482+0000","last_change":"2024-01-15T18:18:32.219821+0000","last_active":"2024-01-15T18:18:34.337482+0000","last_peered":"2024-01-15T18:18:34.337482+0000","last_clean":"2024-01-15T18:18:34.337482+0000","last_became_active":"2024-01-15T18:18:32.218858+0000","last_became_peered":"2024-01-15T18:18:32.218858+0000","last_unstale":"2024-01-15T18:18:34.337482+0000","last_undegraded":"2024-01-15T18:18:34.337482+0000","last_fullsized":"2024-01-15T18:18:34.337482+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:33:51.257660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337507+0000","last_change":"2024-01-15T18:18:33.160602+0000","last_active":"2024-01-15T18:18:34.337507+0000","last_peered":"2024-01-15T18:18:34.337507+0000","last_clean":"2024-01-15T18:18:34.337507+0000","last_became_active":"2024-01-15T18:18:33.160315+0000","last_became_peered":"2024-01-15T18:18:33.160315+0000","last_unstale":"2024-01-15T18:18:34.337507+0000","last_undegraded":"2024-01-15T18:18:34.337507+0000","last_fullsized":"2024-01-15T18:18:34.337507+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:34.698586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.335946+0000","last_change":"2024-01-15T18:18:32.217029+0000","last_active":"2024-01-15T18:18:34.335946+0000","last_peered":"2024-01-15T18:18:34.335946+0000","last_clean":"2024-01-15T18:18:34.335946+0000","last_became_active":"2024-01-15T18:18:32.216612+0000","last_became_peered":"2024-01-15T18:18:32.216612+0000","last_unstale":"2024-01-15T18:18:34.335946+0000","last_undegraded":"2024-01-15T18:18:34.335946+0000","last_fullsized":"2024-01-15T18:18:34.335946+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:42:10.693562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337255+0000","last_change":"2024-01-15T18:18:33.165599+0000","last_active":"2024-01-15T18:18:34.337255+0000","last_peered":"2024-01-15T18:18:34.337255+0000","last_clean":"2024-01-15T18:18:34.337255+0000","last_became_active":"2024-01-15T18:18:33.163994+0000","last_became_peered":"2024-01-15T18:18:33.163994+0000","last_unstale":"2024-01-15T18:18:34.337255+0000","last_undegraded":"2024-01-15T18:18:34.337255+0000","last_fullsized":"2024-01-15T18:18:34.337255+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:00:26.779661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337321+0000","last_change":"2024-01-15T18:18:32.216301+0000","last_active":"2024-01-15T18:18:34.337321+0000","last_peered":"2024-01-15T18:18:34.337321+0000","last_clean":"2024-01-15T18:18:34.337321+0000","last_became_active":"2024-01-15T18:18:32.216023+0000","last_became_peered":"2024-01-15T18:18:32.216023+0000","last_unstale":"2024-01-15T18:18:34.337321+0000","last_undegraded":"2024-01-15T18:18:34.337321+0000","last_fullsized":"2024-01-15T18:18:34.337321+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:58:58.730569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.335697+0000","last_change":"2024-01-15T18:18:33.165611+0000","last_active":"2024-01-15T18:18:34.335697+0000","last_peered":"2024-01-15T18:18:34.335697+0000","last_clean":"2024-01-15T18:18:34.335697+0000","last_became_active":"2024-01-15T18:18:33.165297+0000","last_became_peered":"2024-01-15T18:18:33.165297+0000","last_unstale":"2024-01-15T18:18:34.335697+0000","last_undegraded":"2024-01-15T18:18:34.335697+0000","last_fullsized":"2024-01-15T18:18:34.335697+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:18.639432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336949+0000","last_change":"2024-01-15T18:18:32.214224+0000","last_active":"2024-01-15T18:18:34.336949+0000","last_peered":"2024-01-15T18:18:34.336949+0000","last_clean":"2024-01-15T18:18:34.336949+0000","last_became_active":"2024-01-15T18:18:32.213930+0000","last_became_peered":"2024-01-15T18:18:32.213930+0000","last_unstale":"2024-01-15T18:18:34.336949+0000","last_undegraded":"2024-01-15T18:18:34.336949+0000","last_fullsized":"2024-01-15T18:18:34.336949+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:01:15.328258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.144379+0000","last_change":"2024-01-15T18:18:33.150974+0000","last_active":"2024-01-15T18:18:34.144379+0000","last_peered":"2024-01-15T18:18:34.144379+0000","last_clean":"2024-01-15T18:18:34.144379+0000","last_became_active":"2024-01-15T18:18:33.150660+0000","last_became_peered":"2024-01-15T18:18:33.150660+0000","last_unstale":"2024-01-15T18:18:34.144379+0000","last_undegraded":"2024-01-15T18:18:34.144379+0000","last_fullsized":"2024-01-15T18:18:34.144379+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:22:41.967700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145145+0000","last_change":"2024-01-15T18:18:32.215229+0000","last_active":"2024-01-15T18:18:34.145145+0000","last_peered":"2024-01-15T18:18:34.145145+0000","last_clean":"2024-01-15T18:18:34.145145+0000","last_became_active":"2024-01-15T18:18:32.214394+0000","last_became_peered":"2024-01-15T18:18:32.214394+0000","last_unstale":"2024-01-15T18:18:34.145145+0000","last_undegraded":"2024-01-15T18:18:34.145145+0000","last_fullsized":"2024-01-15T18:18:34.145145+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:30:25.477307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.144477+0000","last_change":"2024-01-15T18:18:33.156094+0000","last_active":"2024-01-15T18:18:34.144477+0000","last_peered":"2024-01-15T18:18:34.144477+0000","last_clean":"2024-01-15T18:18:34.144477+0000","last_became_active":"2024-01-15T18:18:33.155797+0000","last_became_peered":"2024-01-15T18:18:33.155797+0000","last_unstale":"2024-01-15T18:18:34.144477+0000","last_undegraded":"2024-01-15T18:18:34.144477+0000","last_fullsized":"2024-01-15T18:18:34.144477+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:21:42.101901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.144235+0000","last_change":"2024-01-15T18:18:32.212760+0000","last_active":"2024-01-15T18:18:34.144235+0000","last_peered":"2024-01-15T18:18:34.144235+0000","last_clean":"2024-01-15T18:18:34.144235+0000","last_became_active":"2024-01-15T18:18:32.212487+0000","last_became_peered":"2024-01-15T18:18:32.212487+0000","last_unstale":"2024-01-15T18:18:34.144235+0000","last_undegraded":"2024-01-15T18:18:34.144235+0000","last_fullsized":"2024-01-15T18:18:34.144235+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:48:56.885346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.335949+0000","last_change":"2024-01-15T18:18:33.167732+0000","last_active":"2024-01-15T18:18:34.335949+0000","last_peered":"2024-01-15T18:18:34.335949+0000","last_clean":"2024-01-15T18:18:34.335949+0000","last_became_active":"2024-01-15T18:18:33.167436+0000","last_became_peered":"2024-01-15T18:18:33.167436+0000","last_unstale":"2024-01-15T18:18:34.335949+0000","last_undegraded":"2024-01-15T18:18:34.335949+0000","last_fullsized":"2024-01-15T18:18:34.335949+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:22:13.319815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337623+0000","last_change":"2024-01-15T18:18:32.213077+0000","last_active":"2024-01-15T18:18:34.337623+0000","last_peered":"2024-01-15T18:18:34.337623+0000","last_clean":"2024-01-15T18:18:34.337623+0000","last_became_active":"2024-01-15T18:18:32.211744+0000","last_became_peered":"2024-01-15T18:18:32.211744+0000","last_unstale":"2024-01-15T18:18:34.337623+0000","last_undegraded":"2024-01-15T18:18:34.337623+0000","last_fullsized":"2024-01-15T18:18:34.337623+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:48:24.967174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336876+0000","last_change":"2024-01-15T18:18:33.163355+0000","last_active":"2024-01-15T18:18:34.336876+0000","last_peered":"2024-01-15T18:18:34.336876+0000","last_clean":"2024-01-15T18:18:34.336876+0000","last_became_active":"2024-01-15T18:18:33.162521+0000","last_became_peered":"2024-01-15T18:18:33.162521+0000","last_unstale":"2024-01-15T18:18:34.336876+0000","last_undegraded":"2024-01-15T18:18:34.336876+0000","last_fullsized":"2024-01-15T18:18:34.336876+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:35:00.923337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.337137+0000","last_change":"2024-01-15T18:18:32.217455+0000","last_active":"2024-01-15T18:18:34.337137+0000","last_peered":"2024-01-15T18:18:34.337137+0000","last_clean":"2024-01-15T18:18:34.337137+0000","last_became_active":"2024-01-15T18:18:32.216966+0000","last_became_peered":"2024-01-15T18:18:32.216966+0000","last_unstale":"2024-01-15T18:18:34.337137+0000","last_undegraded":"2024-01-15T18:18:34.337137+0000","last_fullsized":"2024-01-15T18:18:34.337137+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:56:34.922952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":11,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.336209+0000","last_change":"2024-01-15T18:18:33.166946+0000","last_active":"2024-01-15T18:18:34.336209+0000","last_peered":"2024-01-15T18:18:34.336209+0000","last_clean":"2024-01-15T18:18:34.336209+0000","last_became_active":"2024-01-15T18:18:33.165323+0000","last_became_peered":"2024-01-15T18:18:33.165323+0000","last_unstale":"2024-01-15T18:18:34.336209+0000","last_undegraded":"2024-01-15T18:18:34.336209+0000","last_fullsized":"2024-01-15T18:18:34.336209+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:18.639432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-15T18:18:34.145258+0000","last_change":"2024-01-15T18:18:32.216251+0000","last_active":"2024-01-15T18:18:34.145258+0000","last_peered":"2024-01-15T18:18:34.145258+0000","last_clean":"2024-01-15T18:18:34.145258+0000","last_became_active":"2024-01-15T18:18:32.215973+0000","last_became_peered":"2024-01-15T18:18:32.215973+0000","last_unstale":"2024-01-15T18:18:34.145258+0000","last_undegraded":"2024-01-15T18:18:34.145258+0000","last_fullsized":"2024-01-15T18:18:34.145258+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:41:26.812763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":239,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-01-15T18:18:33.154452+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:18:33.154452+0000","last_peered":"2024-01-15T18:18:33.154452+0000","last_clean":"2024-01-15T18:18:33.154452+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:18:33.154452+0000","last_undegraded":"2024-01-15T18:18:33.154452+0000","last_fullsized":"2024-01-15T18:18:33.154452+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509609,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64944,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306896,"statfs":{"total":96636764160,"available":96570261504,"internally_reserved":0,"allocated":1937408,"data_stored":1173215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64551370},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.84599999999999997,"15min":0.874},"min":{"1min":0.36799999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.95399999999999996,"5min":9.3040000000000003,"15min":10.284000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.85499999999999998,"15min":0.92300000000000004},"min":{"1min":0.42999999999999999,"5min":0.38400000000000001,"15min":0.378},"max":{"1min":0.83499999999999996,"5min":6.6189999999999998,"15min":16.968},"last":0.53600000000000003}]},{"osd":1,"last update":"Mon Jan 15 18:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.90000000000000002,"15min":0.92400000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.33800000000000002},"max":{"1min":0.91600000000000004,"5min":9.3829999999999991,"15min":15.791},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.85799999999999998,"15min":0.91700000000000004},"min":{"1min":0.45000000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.98899999999999999,"5min":5.0899999999999999,"15min":10.842000000000001},"last":0.80200000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":1.1539999999999999,"15min":0.93700000000000006},"min":{"1min":0.45300000000000001,"5min":0.40400000000000003,"15min":0.378},"max":{"1min":0.878,"5min":22.318000000000001,"15min":22.318000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.90900000000000003,"15min":0.90200000000000002},"min":{"1min":0.44900000000000001,"5min":0.433,"15min":0.39100000000000001},"max":{"1min":0.94799999999999995,"5min":9.5169999999999995,"15min":9.7539999999999996},"last":0.97699999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":1.155,"15min":0.997},"min":{"1min":0.501,"5min":0.501,"15min":0.38100000000000001},"max":{"1min":2.1739999999999999,"5min":18.152999999999999,"15min":18.152999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.95199999999999996,"15min":0.97799999999999998},"min":{"1min":0.5,"5min":0.49199999999999999,"15min":0.43099999999999999},"max":{"1min":0.83699999999999997,"5min":9.75,"15min":10.34},"last":0.93400000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.871,"15min":0.93999999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.314},"max":{"1min":0.91000000000000003,"5min":8.4420000000000002,"15min":20.387},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":1.153,"15min":0.97699999999999998},"min":{"1min":0.38400000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82099999999999995,"5min":17.215,"15min":17.215},"last":0.871}]},{"osd":5,"last update":"Mon Jan 15 18:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.93100000000000005,"15min":0.93500000000000005},"min":{"1min":0.38600000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.78400000000000003,"5min":9.6630000000000003,"15min":19.959},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.92200000000000004,"15min":0.97899999999999998},"min":{"1min":0.46500000000000002,"5min":0.39400000000000002,"15min":0.35799999999999998},"max":{"1min":0.84599999999999997,"5min":7.6980000000000004,"15min":19.905000000000001},"last":0.85399999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":1.0029999999999999,"15min":1.1539999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.99099999999999999,"5min":8.6609999999999996,"15min":22.478999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":1.143,"15min":1.0600000000000001},"min":{"1min":0.48299999999999998,"5min":0.40799999999999997,"15min":0.38300000000000001},"max":{"1min":1.012,"5min":17.132000000000001,"15min":21.509},"last":0.91500000000000004}]}]},{"osd":6,"up_from":15,"seq":64424509609,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64368,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307472,"statfs":{"total":96636764160,"available":96570851328,"internally_reserved":0,"allocated":1339392,"data_stored":582301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:18:01 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":1.0629999999999999,"15min":0.97199999999999998},"min":{"1min":0.42799999999999999,"5min":0.35799999999999998,"15min":0.29099999999999998},"max":{"1min":0.878,"5min":32.587000000000003,"15min":32.587000000000003},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.93300000000000005,"15min":0.86399999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.35399999999999998},"max":{"1min":1.482,"5min":16.193000000000001,"15min":16.193000000000001},"last":0.78200000000000003}]},{"osd":1,"last update":"Mon Jan 15 18:18:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.82799999999999996,"15min":0.98799999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.36599999999999999},"max":{"1min":1.0449999999999999,"5min":7.7649999999999997,"15min":53.451999999999998},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":1.01,"15min":0.91400000000000003},"min":{"1min":0.47299999999999998,"5min":0.437,"15min":0.42899999999999999},"max":{"1min":1.0880000000000001,"5min":32.512,"15min":32.512},"last":0.76100000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:18:01 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":1.0800000000000001,"15min":0.95999999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.38900000000000001},"max":{"1min":0.92900000000000005,"5min":32.472999999999999,"15min":32.472999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.99399999999999999,"15min":0.88700000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.38500000000000001},"max":{"1min":0.94299999999999995,"5min":16.602,"15min":16.602},"last":0.73999999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:18:01 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.98699999999999999,"15min":0.92500000000000004},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.41399999999999998},"max":{"1min":0.91800000000000004,"5min":16.388000000000002,"15min":16.388000000000002},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.88600000000000001,"15min":1.071},"min":{"1min":0.52800000000000002,"5min":0.52300000000000002,"15min":0.41599999999999998},"max":{"1min":0.96199999999999997,"5min":5.9740000000000002,"15min":53.398000000000003},"last":0.67200000000000004}]},{"osd":4,"last update":"Mon Jan 15 18:18:01 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.99199999999999999,"15min":0.88700000000000001},"min":{"1min":0.45700000000000002,"5min":0.38100000000000001,"15min":0.34999999999999998},"max":{"1min":1.1379999999999999,"5min":32.549999999999997,"15min":32.549999999999997},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.879,"15min":0.83499999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.94699999999999995,"5min":11.852,"15min":11.852},"last":0.629}]},{"osd":5,"last update":"Mon Jan 15 18:18:01 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.80900000000000005,"15min":1.0349999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.30399999999999999},"max":{"1min":0.82099999999999995,"5min":7.5650000000000004,"15min":53.229999999999997},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":1.093,"15min":1.024},"min":{"1min":0.46300000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.45,"5min":32.404000000000003,"15min":32.404000000000003},"last":0.70199999999999996}]},{"osd":7,"last update":"Mon Jan 15 18:18:01 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.94699999999999995,"15min":1.097},"min":{"1min":0.41099999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.97999999999999998,"5min":16.565000000000001,"15min":45.070999999999998},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.72499999999999998,"5min":1.173,"15min":1.2050000000000001},"min":{"1min":0.5,"5min":0.45000000000000001,"15min":0.34999999999999998},"max":{"1min":1.1140000000000001,"5min":40.972999999999999,"15min":53.338000000000001},"last":0.59299999999999997}]}]},{"osd":4,"up_from":15,"seq":64424509609,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64364,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307476,"statfs":{"total":96636764160,"available":96570855424,"internally_reserved":0,"allocated":1335296,"data_stored":582231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.81599999999999995,"15min":0.88200000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.32900000000000001},"max":{"1min":0.91700000000000004,"5min":7.3380000000000001,"15min":23.318000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.68999999999999995,"15min":0.78800000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.32600000000000001},"max":{"1min":0.82099999999999995,"5min":6.444,"15min":14.723000000000001},"last":0.63100000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.82999999999999996,"15min":0.78400000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90500000000000003,"5min":9.5700000000000003,"15min":9.5700000000000003},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.91300000000000003,"15min":0.91400000000000003},"min":{"1min":0.45700000000000002,"5min":0.41399999999999998,"15min":0.36899999999999999},"max":{"1min":0.81100000000000005,"5min":7.9980000000000002,"15min":22.408999999999999},"last":0.77500000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.85999999999999999,"15min":0.80000000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.34399999999999997},"max":{"1min":0.80800000000000005,"5min":9.7089999999999996,"15min":9.7089999999999996},"last":0.874},{"interface":"front","average":{"1min":0.67900000000000005,"5min":1.0629999999999999,"15min":1.014},"min":{"1min":0.44400000000000001,"5min":0.40999999999999998,"15min":0.39100000000000001},"max":{"1min":1.3180000000000001,"5min":14.118,"15min":22.338999999999999},"last":0.72199999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.77300000000000002,"15min":0.86899999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.41199999999999998},"max":{"1min":0.88400000000000001,"5min":6.7930000000000001,"15min":14.763999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.94799999999999995,"15min":0.90500000000000003},"min":{"1min":0.495,"5min":0.495,"15min":0.45500000000000002},"max":{"1min":0.95199999999999996,"5min":9.6289999999999996,"15min":9.6289999999999996},"last":0.82699999999999996}]},{"osd":5,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.93600000000000005,"15min":0.95099999999999996},"min":{"1min":0.39700000000000002,"5min":0.39000000000000001,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":15.787000000000001,"15min":31.617999999999999},"last":0.5},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.746,"15min":0.86099999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.36599999999999999},"max":{"1min":0.79600000000000004,"5min":6.7169999999999996,"15min":17.145},"last":0.79700000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.83899999999999997,"15min":0.82099999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.35499999999999998},"max":{"1min":0.85999999999999999,"5min":9.6709999999999994,"15min":11.124000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":1.159,"15min":1.2170000000000001},"min":{"1min":0.498,"5min":0.438,"15min":0.40200000000000002},"max":{"1min":0.89300000000000002,"5min":32.308,"15min":32.308},"last":0.65600000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.73799999999999999,"15min":0.996},"min":{"1min":0.45700000000000002,"5min":0.434,"15min":0.34799999999999998},"max":{"1min":0.876,"5min":6.8390000000000004,"15min":39.256},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.748,"5min":1.1279999999999999,"15min":1.002},"min":{"1min":0.504,"5min":0.47199999999999998,"15min":0.42199999999999999},"max":{"1min":2.3330000000000002,"5min":16.277000000000001,"15min":26.573},"last":0.73999999999999999}]}]},{"osd":3,"up_from":15,"seq":64424509608,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68532,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303308,"statfs":{"total":96636764160,"available":96566587392,"internally_reserved":0,"allocated":1351680,"data_stored":583913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.45800000000000002,"15min":0.49099999999999999},"min":{"1min":0.29999999999999999,"5min":0.25900000000000001,"15min":0.245},"max":{"1min":0.752,"5min":2.1160000000000001,"15min":5.8849999999999998},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.53900000000000003,"15min":0.61599999999999999},"min":{"1min":0.32900000000000001,"5min":0.29799999999999999,"15min":0.29499999999999998},"max":{"1min":0.72399999999999998,"5min":3.0649999999999999,"15min":17.756},"last":0.38200000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.54500000000000004,"15min":0.59899999999999998},"min":{"1min":0.309,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":5.8819999999999997,"5min":5.8819999999999997,"15min":12.307},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.56200000000000006,"15min":0.58599999999999997},"min":{"1min":0.38600000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.80500000000000005,"5min":3.738,"15min":6.0640000000000001},"last":0.44500000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.66400000000000003,"15min":0.69099999999999995},"min":{"1min":0.375,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.145,"5min":5.431,"15min":12.468999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.70799999999999996,"15min":0.68600000000000005},"min":{"1min":0.433,"5min":0.35399999999999998,"15min":0.34599999999999997},"max":{"1min":0.871,"5min":10.837999999999999,"15min":10.837999999999999},"last":0.46500000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":1.0649999999999999,"15min":1.0089999999999999},"min":{"1min":0.433,"5min":0.41199999999999998,"15min":0.375},"max":{"1min":1.109,"5min":13.863,"15min":17.853999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.96399999999999997,"15min":0.96299999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.2090000000000001,"5min":6.8289999999999997,"15min":13.31},"last":0.66200000000000003}]},{"osd":5,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":1.167,"15min":0.97499999999999998},"min":{"1min":0.53400000000000003,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.042,"5min":17.687999999999999,"15min":17.687999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.88300000000000001,"15min":0.97199999999999998},"min":{"1min":0.53200000000000003,"5min":0.45500000000000002,"15min":0.438},"max":{"1min":1.1870000000000001,"5min":5.0949999999999998,"15min":20.515999999999998},"last":0.54600000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.81100000000000005,"15min":0.97499999999999998},"min":{"1min":0.52200000000000002,"5min":0.442,"15min":0.432},"max":{"1min":1.294,"5min":4.0910000000000002,"15min":23.100999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.91700000000000004,"15min":0.93799999999999994},"min":{"1min":0.56399999999999995,"5min":0.55600000000000005,"15min":0.49399999999999999},"max":{"1min":1.159,"5min":6.7220000000000004,"15min":12.76},"last":0.64800000000000002}]},{"osd":7,"last update":"Mon Jan 15 18:18:14 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":1.335,"15min":1.1659999999999999},"min":{"1min":0.57699999999999996,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.236,"5min":24.321999999999999,"15min":24.321999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.79600000000000004,"5min":1.147,"15min":1.1000000000000001},"min":{"1min":0.55200000000000005,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.3340000000000001,"5min":23.013000000000002,"15min":23.013000000000002},"last":0.56100000000000005}]}]},{"osd":2,"up_from":15,"seq":64424509608,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64364,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307476,"statfs":{"total":96636764160,"available":96570855424,"internally_reserved":0,"allocated":1335296,"data_stored":582231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:18:07 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.45900000000000002,"15min":0.50900000000000001},"min":{"1min":0.29799999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.61399999999999999,"5min":0.98199999999999998,"15min":5.1849999999999996},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.48999999999999999,"15min":0.53200000000000003},"min":{"1min":0.32900000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.68500000000000005,"5min":1.1579999999999999,"15min":7.516},"last":0.40000000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:18:07 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.54900000000000004,"15min":0.61299999999999999},"min":{"1min":0.33000000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.72799999999999998,"5min":0.95799999999999996,"15min":10.907},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.53100000000000003,"15min":0.60299999999999998},"min":{"1min":0.32900000000000001,"5min":0.311,"15min":0.27000000000000002},"max":{"1min":2.8799999999999999,"5min":2.8799999999999999,"15min":8.9190000000000005},"last":0.46000000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:18:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.55100000000000005,"15min":0.61899999999999999},"min":{"1min":0.34499999999999997,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.76700000000000002,"5min":1.133,"15min":12.185},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.59399999999999997,"15min":0.66400000000000003},"min":{"1min":0.40100000000000002,"5min":0.373,"15min":0.34000000000000002},"max":{"1min":0.78800000000000003,"5min":0.96699999999999997,"15min":7.6760000000000002},"last":0.47899999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:18:07 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.89100000000000001,"15min":0.877},"min":{"1min":0.54400000000000004,"5min":0.44,"15min":0.44},"max":{"1min":1.26,"5min":4.3150000000000004,"15min":8.6609999999999996},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.79400000000000004,"15min":0.91100000000000003},"min":{"1min":0.45000000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":2.968,"5min":2.968,"15min":19.146999999999998},"last":0.497}]},{"osd":5,"last update":"Mon Jan 15 18:18:07 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.80700000000000005,"15min":0.90600000000000003},"min":{"1min":0.57399999999999995,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.1359999999999999,"5min":5.0309999999999997,"15min":31.710000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.83699999999999997,"15min":0.997},"min":{"1min":0.60999999999999999,"5min":0.49199999999999999,"15min":0.48199999999999998},"max":{"1min":1.115,"5min":3.6459999999999999,"15min":22.539000000000001},"last":0.51700000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:18:07 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.877,"15min":0.91300000000000003},"min":{"1min":0.50700000000000001,"5min":0.499,"15min":0.40899999999999997},"max":{"1min":3.0430000000000001,"5min":4.1959999999999997,"15min":9.4540000000000006},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.79500000000000004,"15min":0.90400000000000003},"min":{"1min":0.54300000000000004,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0589999999999999,"5min":1.633,"15min":18.614999999999998},"last":0.624}]},{"osd":7,"last update":"Mon Jan 15 18:18:07 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.99399999999999999,"15min":0.96899999999999997},"min":{"1min":0.52200000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":3.008,"5min":11.606,"15min":11.606},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.91900000000000004,"15min":1.0009999999999999},"min":{"1min":0.58299999999999996,"5min":0.52500000000000002,"15min":0.502},"max":{"1min":1.151,"5min":6.1260000000000003,"15min":17.013000000000002},"last":0.53900000000000003}]}]},{"osd":1,"up_from":15,"seq":64424509610,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68532,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303308,"statfs":{"total":96636764160,"available":96566587392,"internally_reserved":0,"allocated":1343488,"data_stored":582355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:18:06 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.50900000000000001,"15min":0.65000000000000002},"min":{"1min":0.27100000000000002,"5min":0.25800000000000001,"15min":0.23699999999999999},"max":{"1min":0.60199999999999998,"5min":4.1550000000000002,"15min":19.577000000000002},"last":0.5},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.63100000000000001,"15min":0.76200000000000001},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.251},"max":{"1min":0.81699999999999995,"5min":12.378,"15min":48.277999999999999},"last":0.61899999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:18:06 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.55300000000000005,"15min":0.70499999999999996},"min":{"1min":0.32100000000000001,"5min":0.309,"15min":0.30299999999999999},"max":{"1min":0.73399999999999999,"5min":2.8610000000000002,"15min":40.698},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.58699999999999997,"15min":0.86199999999999999},"min":{"1min":0.38100000000000001,"5min":0.33300000000000002,"15min":0.317},"max":{"1min":0.86299999999999999,"5min":2.5139999999999998,"15min":48.316000000000003},"last":0.58899999999999997}]},{"osd":3,"last update":"Mon Jan 15 18:18:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.628,"15min":0.82699999999999996},"min":{"1min":0.42899999999999999,"5min":0.36099999999999999,"15min":0.33700000000000002},"max":{"1min":0.91200000000000003,"5min":2.8620000000000001,"15min":48.234999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.68700000000000006,"15min":0.67300000000000004},"min":{"1min":0.371,"5min":0.315,"15min":0.311},"max":{"1min":0.79600000000000004,"5min":9.9719999999999995,"15min":9.9719999999999995},"last":0.52800000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:18:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.91100000000000003,"15min":0.89100000000000001},"min":{"1min":0.51400000000000001,"5min":0.46000000000000002,"15min":0.44500000000000001},"max":{"1min":1.0880000000000001,"5min":9.9329999999999998,"15min":9.9329999999999998},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.877,"15min":0.93100000000000005},"min":{"1min":0.496,"5min":0.45800000000000002,"15min":0.42499999999999999},"max":{"1min":1.1819999999999999,"5min":9.8610000000000007,"15min":19.829000000000001},"last":0.999}]},{"osd":5,"last update":"Mon Jan 15 18:18:06 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.85099999999999998,"15min":1.0129999999999999},"min":{"1min":0.56499999999999995,"5min":0.48699999999999999,"15min":0.46100000000000002},"max":{"1min":1.212,"5min":8.5839999999999996,"15min":22.594999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.81699999999999995,"15min":1.1559999999999999},"min":{"1min":0.53400000000000003,"5min":0.44400000000000001,"15min":0.443},"max":{"1min":1.1040000000000001,"5min":2.9609999999999999,"15min":48.198},"last":1.056}]},{"osd":6,"last update":"Mon Jan 15 18:18:06 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.92000000000000004,"15min":1.0409999999999999},"min":{"1min":0.58399999999999996,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.2350000000000001,"5min":9.3989999999999991,"15min":35.112000000000002},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.92700000000000005,"15min":0.96699999999999997},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.1539999999999999,"5min":5.8849999999999998,"15min":19.780999999999999},"last":0.878}]},{"osd":7,"last update":"Mon Jan 15 18:18:06 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.91000000000000003,"15min":1.1639999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.1890000000000001,"5min":11.039,"15min":48.148000000000003},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.98399999999999999,"15min":0.98999999999999999},"min":{"1min":0.57399999999999995,"5min":0.55800000000000005,"15min":0.52100000000000002},"max":{"1min":1.1559999999999999,"5min":9.8640000000000008,"15min":9.8640000000000008},"last":0.95599999999999996}]}]},{"osd":5,"up_from":14,"seq":60129542313,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58856,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312984,"statfs":{"total":96636764160,"available":96576495616,"internally_reserved":0,"allocated":1331200,"data_stored":576423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:18:23 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.77000000000000002,"15min":0.79400000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.28999999999999998},"max":{"1min":0.88700000000000001,"5min":5.1639999999999997,"15min":21.385000000000002},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.77100000000000002,"15min":0.86799999999999999},"min":{"1min":0.45100000000000001,"5min":0.378,"15min":0.34899999999999998},"max":{"1min":0.95099999999999996,"5min":5.3869999999999996,"15min":9.359},"last":0.69499999999999995}]},{"osd":1,"last update":"Mon Jan 15 18:18:23 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.76400000000000001,"15min":0.89200000000000002},"min":{"1min":0.45300000000000001,"5min":0.375,"15min":0.34899999999999998},"max":{"1min":0.92100000000000004,"5min":3.7320000000000002,"15min":21.120000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":1.0029999999999999,"15min":0.96799999999999997},"min":{"1min":0.42499999999999999,"5min":0.39800000000000002,"15min":0.315},"max":{"1min":0.93200000000000005,"5min":15.843,"15min":21.5},"last":0.73099999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:18:23 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.81299999999999994,"15min":0.89700000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.34999999999999998},"max":{"1min":1.7729999999999999,"5min":5.516,"15min":8.7750000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.876,"15min":0.97899999999999998},"min":{"1min":0.46000000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.86499999999999999,"5min":8.3529999999999998,"15min":21.201000000000001},"last":0.77900000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:18:23 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.89700000000000002,"15min":0.90400000000000003},"min":{"1min":0.47199999999999998,"5min":0.42799999999999999,"15min":0.37},"max":{"1min":0.96499999999999997,"5min":7.7199999999999998,"15min":21.457000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.82199999999999995,"15min":0.95399999999999996},"min":{"1min":0.52300000000000002,"5min":0.47699999999999998,"15min":0.45400000000000001},"max":{"1min":0.90800000000000003,"5min":5.6109999999999998,"15min":8.7300000000000004},"last":0.80700000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:18:23 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.90900000000000003,"15min":0.88200000000000001},"min":{"1min":0.41899999999999998,"5min":0.376,"15min":0.30399999999999999},"max":{"1min":0.88900000000000001,"5min":10.962,"15min":10.962},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.749,"15min":1.0089999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.76200000000000001,"5min":4.6639999999999997,"15min":26.146999999999998},"last":0.56200000000000006}]},{"osd":6,"last update":"Mon Jan 15 18:18:23 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.80800000000000005,"15min":0.83199999999999996},"min":{"1min":0.46300000000000002,"5min":0.39700000000000002,"15min":0.32000000000000001},"max":{"1min":1.0169999999999999,"5min":5.2160000000000002,"15min":21.535},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.80000000000000004,"15min":1.0029999999999999},"min":{"1min":0.44,"5min":0.432,"15min":0.36299999999999999},"max":{"1min":0.875,"5min":4.9139999999999997,"15min":30.488},"last":0.751}]},{"osd":7,"last update":"Mon Jan 15 18:18:23 2024","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.95899999999999996,"15min":1.2350000000000001},"min":{"1min":0.52200000000000002,"5min":0.47699999999999998,"15min":0.39700000000000002},"max":{"1min":4.7930000000000001,"5min":8.7539999999999996,"15min":33.499000000000002},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.91100000000000003,"15min":1.0820000000000001},"min":{"1min":0.51300000000000001,"5min":0.46200000000000002,"15min":0.36499999999999999},"max":{"1min":0.99199999999999999,"5min":6.306,"15min":25.437999999999999},"last":0.79400000000000004}]}]},{"osd":0,"up_from":14,"seq":60129542313,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64952,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306888,"statfs":{"total":96636764160,"available":96570253312,"internally_reserved":0,"allocated":1937408,"data_stored":1167911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:18:05 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.52500000000000002,"15min":0.624},"min":{"1min":0.32900000000000001,"5min":0.245,"15min":0.23699999999999999},"max":{"1min":0.67700000000000005,"5min":7.907,"15min":19.843},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.57599999999999996,"15min":0.66900000000000004},"min":{"1min":0.27300000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.69099999999999995,"5min":3.512,"15min":10.154},"last":0.54700000000000004}]},{"osd":2,"last update":"Mon Jan 15 18:18:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.71699999999999997,"15min":0.79700000000000004},"min":{"1min":0.35799999999999998,"5min":0.34100000000000003,"15min":0.30399999999999999},"max":{"1min":0.71999999999999997,"5min":7.0810000000000004,"15min":25.905000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.57699999999999996,"15min":0.61199999999999999},"min":{"1min":0.308,"5min":0.27800000000000002,"15min":0.26700000000000002},"max":{"1min":0.753,"5min":3.8500000000000001,"15min":9.3829999999999991},"last":0.44400000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:18:05 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.69399999999999995,"15min":0.73699999999999999},"min":{"1min":0.379,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66600000000000004,"5min":6.2850000000000001,"15min":9.1780000000000008},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.70699999999999996,"15min":0.75900000000000001},"min":{"1min":0.44400000000000001,"5min":0.38400000000000001,"15min":0.36899999999999999},"max":{"1min":0.77300000000000002,"5min":6.5979999999999999,"15min":10.210000000000001},"last":0.46800000000000003}]},{"osd":4,"last update":"Mon Jan 15 18:18:05 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.83799999999999997,"15min":0.97999999999999998},"min":{"1min":0.40100000000000002,"5min":0.39100000000000001,"15min":0.38},"max":{"1min":0.999,"5min":4.5179999999999998,"15min":22.814},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.82999999999999996,"15min":1.0229999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.41199999999999998},"max":{"1min":1.111,"5min":4.2329999999999997,"15min":23.341999999999999},"last":0.83499999999999996}]},{"osd":5,"last update":"Mon Jan 15 18:18:05 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.85399999999999998,"15min":0.96499999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.0880000000000001,"5min":4.008,"15min":12.428000000000001},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.75800000000000001,"5min":1.248,"15min":1.252},"min":{"1min":0.496,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.99099999999999999,"5min":23.021000000000001,"15min":23.021000000000001},"last":0.80500000000000005}]},{"osd":6,"last update":"Mon Jan 15 18:18:05 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.92100000000000004,"15min":1.21},"min":{"1min":0.52600000000000002,"5min":0.48399999999999999,"15min":0.46200000000000002},"max":{"1min":1.042,"5min":6.5250000000000004,"15min":31.059000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.94899999999999995,"15min":1.083},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.49099999999999999},"max":{"1min":1.0589999999999999,"5min":6.9690000000000003,"15min":40.731000000000002},"last":0.75}]},{"osd":7,"last update":"Mon Jan 15 18:18:05 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":1.1020000000000001,"15min":1.1950000000000001},"min":{"1min":0.50700000000000001,"5min":0.499,"15min":0.42499999999999999},"max":{"1min":1.014,"5min":10.497999999999999,"15min":24.492999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.88700000000000001,"15min":1.099},"min":{"1min":0.54300000000000004,"5min":0.495,"15min":0.46600000000000003},"max":{"1min":1.1459999999999999,"5min":4.6600000000000001,"15min":23.524000000000001},"last":0.68999999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:18:37.700 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-15T18:18:37.700 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-15T18:18:37.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.885+0000 7f5086000640 1 Processor -- start 2024-01-15T18:18:37.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.885+0000 7f5086000640 1 -- start start 2024-01-15T18:18:37.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f5086000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f508006c740 0x7f5080099a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:37.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f5086000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f508015d860 con 0x7f508015d480 2024-01-15T18:18:37.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f5086000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f508006b6a0 con 0x7f5080099f50 2024-01-15T18:18:37.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f5086000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f508009a400 con 0x7f508006c740 2024-01-15T18:18:37.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507f400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f508015d480 0x7f50800cfbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38698/0 (socket says 172.21.15.27:38698) 2024-01-15T18:18:37.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507f400640 1 -- 172.21.15.27:0/3148665121 learned_addr learned my addr 172.21.15.27:0/3148665121 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:37.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507ea00640 1 --2- 172.21.15.27:0/3148665121 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f508006c740 0x7f5080099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:37.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507e000640 1 -- 172.21.15.27:0/3148665121 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 587552124 0 0) 0x7f508015d860 con 0x7f508015d480 2024-01-15T18:18:37.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507e000640 1 -- 172.21.15.27:0/3148665121 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5060003530 con 0x7f508015d480 2024-01-15T18:18:37.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507e000640 1 -- 172.21.15.27:0/3148665121 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3201940879 0 0) 0x7f508006b6a0 con 0x7f5080099f50 2024-01-15T18:18:37.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507e000640 1 -- 172.21.15.27:0/3148665121 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f508015d860 con 0x7f5080099f50 2024-01-15T18:18:37.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507e000640 1 -- 172.21.15.27:0/3148665121 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2724107187 0 0) 0x7f5060003530 con 0x7f508015d480 2024-01-15T18:18:37.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507e000640 1 -- 172.21.15.27:0/3148665121 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f508006c740 msgr2=0x7f5080099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:37.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507e000640 1 --2- 172.21.15.27:0/3148665121 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f508006c740 0x7f5080099a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:37.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507e000640 1 -- 172.21.15.27:0/3148665121 >> v1:172.21.15.93:6789/0 conn(0x7f5080099f50 legacy=0x7f50800cb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:37.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507e000640 1 -- 172.21.15.27:0/3148665121 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f508006b980 con 0x7f508015d480 2024-01-15T18:18:37.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507e000640 1 -- 172.21.15.27:0/3148665121 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f5070001680 con 0x7f508015d480 2024-01-15T18:18:37.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507e000640 1 -- 172.21.15.27:0/3148665121 >> v1:172.21.15.27:6789/0 conn(0x7f508015d480 legacy=0x7f50800cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:37.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507e000640 1 --2- 172.21.15.27:0/3148665121 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5060003bb0 0x7f5060003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:37.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507e000640 1 --2- 172.21.15.27:0/3148665121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50600047d0 0x7f5060004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:37.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507e000640 1 --2- 172.21.15.27:0/3148665121 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f508006c740 0x7f5060005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:37.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507e000640 1 -- 172.21.15.27:0/3148665121 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508009a400 con 0x7f50600047d0 2024-01-15T18:18:37.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507e000640 1 -- 172.21.15.27:0/3148665121 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506000baa0 con 0x7f5060003bb0 2024-01-15T18:18:37.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507e000640 1 -- 172.21.15.27:0/3148665121 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f506000bc60 con 0x7f508006c740 2024-01-15T18:18:37.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507ea00640 1 --2- 172.21.15.27:0/3148665121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50600047d0 0x7f5060004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:37.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507f400640 1 --2- 172.21.15.27:0/3148665121 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5060003bb0 0x7f5060003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:37.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507fe00640 1 --2- 172.21.15.27:0/3148665121 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f508006c740 0x7f5060005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:37.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507ea00640 1 -- 172.21.15.27:0/3148665121 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f508006c740 msgr2=0x7f5060005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:37.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507ea00640 1 --2- 172.21.15.27:0/3148665121 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f508006c740 0x7f5060005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:37.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507ea00640 1 -- 172.21.15.27:0/3148665121 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5060003bb0 msgr2=0x7f5060003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:37.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507ea00640 1 --2- 172.21.15.27:0/3148665121 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5060003bb0 0x7f5060003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:37.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507ea00640 1 -- 172.21.15.27:0/3148665121 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5060003530 con 0x7f50600047d0 2024-01-15T18:18:37.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.889+0000 7f507fe00640 1 --2- 172.21.15.27:0/3148665121 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f508006c740 0x7f5060005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:18:37.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.893+0000 7f507ea00640 1 --2- 172.21.15.27:0/3148665121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50600047d0 0x7f5060004c50 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f508006cb60 tx=0x7f506801c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b02b1143f64b8b8d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:37.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.893+0000 7f507e000640 1 -- 172.21.15.27:0/3148665121 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50680a55c0 con 0x7f50600047d0 2024-01-15T18:18:37.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.893+0000 7f507e000640 1 -- 172.21.15.27:0/3148665121 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50680a5720 con 0x7f50600047d0 2024-01-15T18:18:37.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.893+0000 7f507e000640 1 -- 172.21.15.27:0/3148665121 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50680a5ad0 con 0x7f50600047d0 2024-01-15T18:18:37.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.893+0000 7f5086000640 1 -- 172.21.15.27:0/3148665121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50600047d0 msgr2=0x7f5060004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:37.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.893+0000 7f5086000640 1 --2- 172.21.15.27:0/3148665121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50600047d0 0x7f5060004c50 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f508006cb60 tx=0x7f506801c9d0 comp rx=0 tx=0).stop 2024-01-15T18:18:37.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.893+0000 7f507fe00640 1 -- 172.21.15.27:0/3148665121 reap_dead start 2024-01-15T18:18:37.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.893+0000 7f5086000640 1 -- 172.21.15.27:0/3148665121 shutdown_connections 2024-01-15T18:18:37.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.893+0000 7f5086000640 1 -- 172.21.15.27:0/3148665121 >> 172.21.15.27:0/3148665121 conn(0x7f50800b2a80 msgr2=0x7f50800b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:37.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.893+0000 7f5086000640 1 -- 172.21.15.27:0/3148665121 shutdown_connections 2024-01-15T18:18:37.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.893+0000 7f5086000640 1 -- 172.21.15.27:0/3148665121 wait complete. 2024-01-15T18:18:37.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.897+0000 7f5086000640 1 Processor -- start 2024-01-15T18:18:37.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.897+0000 7f5086000640 1 -- start start 2024-01-15T18:18:37.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.897+0000 7f5086000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50600047d0 0x7f50801f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:37.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.897+0000 7f5086000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f508009e580 0x7f50801f2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:37.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.897+0000 7f5086000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f508015f560 0x7f508015f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:37.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.897+0000 7f5086000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5080154cd0 con 0x7f50600047d0 2024-01-15T18:18:37.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.897+0000 7f5086000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508015ff20 con 0x7f508009e580 2024-01-15T18:18:37.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.897+0000 7f5086000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50800b52d0 con 0x7f508015f560 2024-01-15T18:18:37.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.897+0000 7f507f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50600047d0 0x7f50801f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:37.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.897+0000 7f507f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50600047d0 0x7f50801f6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38152/0 (socket says 172.21.15.27:38152) 2024-01-15T18:18:37.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.897+0000 7f507f400640 1 -- 172.21.15.27:0/2084459984 learned_addr learned my addr 172.21.15.27:0/2084459984 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:37.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.897+0000 7f507fe00640 1 --2- 172.21.15.27:0/2084459984 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f508015f560 0x7f508015f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:37.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.897+0000 7f507ea00640 1 --2- 172.21.15.27:0/2084459984 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f508009e580 0x7f50801f2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:37.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.897+0000 7f507f400640 1 -- 172.21.15.27:0/2084459984 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f508015f560 msgr2=0x7f508015f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:37.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.897+0000 7f507f400640 1 --2- 172.21.15.27:0/2084459984 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f508015f560 0x7f508015f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:37.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.897+0000 7f507f400640 1 -- 172.21.15.27:0/2084459984 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f508009e580 msgr2=0x7f50801f2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:37.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.897+0000 7f507f400640 1 --2- 172.21.15.27:0/2084459984 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f508009e580 0x7f50801f2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:37.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.897+0000 7f507f400640 1 -- 172.21.15.27:0/2084459984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50800b4070 con 0x7f50600047d0 2024-01-15T18:18:37.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.897+0000 7f507f400640 1 --2- 172.21.15.27:0/2084459984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50600047d0 0x7f50801f6e20 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f5070007310 tx=0x7f50700077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:37.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.897+0000 7f5077400640 1 -- 172.21.15.27:0/2084459984 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507000f9c0 con 0x7f50600047d0 2024-01-15T18:18:37.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.897+0000 7f5077400640 1 -- 172.21.15.27:0/2084459984 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f507000fb40 con 0x7f50600047d0 2024-01-15T18:18:37.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.897+0000 7f5077400640 1 -- 172.21.15.27:0/2084459984 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50700133f0 con 0x7f50600047d0 2024-01-15T18:18:37.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.897+0000 7f5086000640 1 -- 172.21.15.27:0/2084459984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50801f6380 con 0x7f50600047d0 2024-01-15T18:18:37.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.897+0000 7f5086000640 1 -- 172.21.15.27:0/2084459984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50801f6620 con 0x7f50600047d0 2024-01-15T18:18:37.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.901+0000 7f5086000640 1 -- 172.21.15.27:0/2084459984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f508004e850 con 0x7f50600047d0 2024-01-15T18:18:37.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.901+0000 7f5077400640 1 -- 172.21.15.27:0/2084459984 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f5070013550 con 0x7f50600047d0 2024-01-15T18:18:37.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.901+0000 7f5077400640 1 --2- 172.21.15.27:0/2084459984 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5058067970 0x7f5058069e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:37.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.901+0000 7f5077400640 1 -- 172.21.15.27:0/2084459984 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f50700834a0 con 0x7f50600047d0 2024-01-15T18:18:37.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.901+0000 7f507ea00640 1 --2- 172.21.15.27:0/2084459984 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5058067970 0x7f5058069e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:37.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.901+0000 7f507ea00640 1 --2- 172.21.15.27:0/2084459984 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5058067970 0x7f5058069e30 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f50680af000 tx=0x7f50680046d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:37.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:37.905+0000 7f5077400640 1 -- 172.21.15.27:0/2084459984 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5070051e70 con 0x7f50600047d0 2024-01-15T18:18:38.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.121+0000 7f5086000640 1 -- 172.21.15.27:0/2084459984 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f50800d09e0 con 0x7f50600047d0 2024-01-15T18:18:38.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.121+0000 7f5077400640 1 -- 172.21.15.27:0/2084459984 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+147 (secure 0 0 0) 0x7f5070055490 con 0x7f50600047d0 2024-01-15T18:18:38.123 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-01-15T18:18:38.123 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOc6VlH+DCLxAA1sKVut+3WjA/B+aYJ6/Jeg== 2024-01-15T18:18:38.123 INFO:teuthology.orchestra.run.smithi027.stdout: caps mds = "allow" 2024-01-15T18:18:38.123 INFO:teuthology.orchestra.run.smithi027.stdout: caps mon = "allow r" 2024-01-15T18:18:38.123 INFO:teuthology.orchestra.run.smithi027.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-15T18:18:38.123 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2024-01-15T18:18:38.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.125+0000 7f5086000640 1 -- 172.21.15.27:0/2084459984 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5058067970 msgr2=0x7f5058069e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:38.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.125+0000 7f5086000640 1 --2- 172.21.15.27:0/2084459984 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5058067970 0x7f5058069e30 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f50680af000 tx=0x7f50680046d0 comp rx=0 tx=0).stop 2024-01-15T18:18:38.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.125+0000 7f5086000640 1 -- 172.21.15.27:0/2084459984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50600047d0 msgr2=0x7f50801f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:38.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.125+0000 7f5086000640 1 --2- 172.21.15.27:0/2084459984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50600047d0 0x7f50801f6e20 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f5070007310 tx=0x7f50700077d0 comp rx=0 tx=0).stop 2024-01-15T18:18:38.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.125+0000 7f5086000640 1 -- 172.21.15.27:0/2084459984 shutdown_connections 2024-01-15T18:18:38.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.125+0000 7f5086000640 1 --2- 172.21.15.27:0/2084459984 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5058067970 0x7f5058069e30 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:38.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.125+0000 7f5086000640 1 --2- 172.21.15.27:0/2084459984 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f508015f560 0x7f508015f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:38.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.125+0000 7f5086000640 1 --2- 172.21.15.27:0/2084459984 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f508009e580 0x7f50801f2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:38.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.125+0000 7f5086000640 1 --2- 172.21.15.27:0/2084459984 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50600047d0 0x7f50801f6e20 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:38.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.125+0000 7f5086000640 1 -- 172.21.15.27:0/2084459984 >> 172.21.15.27:0/2084459984 conn(0x7f50800b2a80 msgr2=0x7f508015b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:38.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.125+0000 7f5086000640 1 -- 172.21.15.27:0/2084459984 shutdown_connections 2024-01-15T18:18:38.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.125+0000 7f5086000640 1 -- 172.21.15.27:0/2084459984 wait complete. 2024-01-15T18:18:38.146 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:18:38.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.289+0000 7fb6b8000640 1 Processor -- start 2024-01-15T18:18:38.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.289+0000 7fb6b8000640 1 -- start start 2024-01-15T18:18:38.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.289+0000 7fb6b8000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fb6b006ac00 0x7fb6b00b2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:38.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.289+0000 7fb6b8000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb6b006d020 0x7fb6b0071970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:38.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.289+0000 7fb6b8000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b006ea70 con 0x7fb6b00af240 2024-01-15T18:18:38.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.289+0000 7fb6b8000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6b00720c0 con 0x7fb6b006d020 2024-01-15T18:18:38.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.289+0000 7fb6b8000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6b0071eb0 con 0x7fb6b006ac00 2024-01-15T18:18:38.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.289+0000 7fb6b6000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb6b006d020 0x7fb6b0071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:38.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.289+0000 7fb6b6000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb6b006d020 0x7fb6b0071970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38164/0 (socket says 172.21.15.27:38164) 2024-01-15T18:18:38.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.289+0000 7fb6b6000640 1 -- 172.21.15.27:0/363466394 learned_addr learned my addr 172.21.15.27:0/363466394 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:38.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.289+0000 7fb6b4c00640 1 --2- 172.21.15.27:0/363466394 >> v2:172.21.15.93:3300/0 conn(0x7fb6b006ac00 0x7fb6b00b2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.289+0000 7fb6b6000640 1 -- 172.21.15.27:0/363466394 >> v2:172.21.15.93:3300/0 conn(0x7fb6b006ac00 msgr2=0x7fb6b00b2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.289+0000 7fb6b6000640 1 --2- 172.21.15.27:0/363466394 >> v2:172.21.15.93:3300/0 conn(0x7fb6b006ac00 0x7fb6b00b2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.289+0000 7fb6b6000640 1 -- 172.21.15.27:0/363466394 >> v1:172.21.15.93:6789/0 conn(0x7fb6b00af240 legacy=0x7fb6b006a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:38.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.289+0000 7fb6b6000640 1 -- 172.21.15.27:0/363466394 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b006ed50 con 0x7fb6b006d020 2024-01-15T18:18:38.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.293+0000 7fb6b6000640 1 --2- 172.21.15.27:0/363466394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6b006d020 0x7fb6b0071970 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fb6b00cc790 tx=0x7fb6a801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69a2e10a623f2123 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:38.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.293+0000 7fb6afe00640 1 -- 172.21.15.27:0/363466394 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a80a57d0 con 0x7fb6b006d020 2024-01-15T18:18:38.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.293+0000 7fb6afe00640 1 -- 172.21.15.27:0/363466394 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a80a5930 con 0x7fb6b006d020 2024-01-15T18:18:38.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.293+0000 7fb6afe00640 1 -- 172.21.15.27:0/363466394 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a80a5cc0 con 0x7fb6b006d020 2024-01-15T18:18:38.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.293+0000 7fb6b8000640 1 -- 172.21.15.27:0/363466394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6b006d020 msgr2=0x7fb6b0071970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:38.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.293+0000 7fb6b8000640 1 --2- 172.21.15.27:0/363466394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6b006d020 0x7fb6b0071970 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fb6b00cc790 tx=0x7fb6a801c910 comp rx=0 tx=0).stop 2024-01-15T18:18:38.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.293+0000 7fb6b8000640 1 -- 172.21.15.27:0/363466394 shutdown_connections 2024-01-15T18:18:38.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.293+0000 7fb6b8000640 1 --2- 172.21.15.27:0/363466394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6b006d020 0x7fb6b0071970 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:38.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.293+0000 7fb6b8000640 1 --2- 172.21.15.27:0/363466394 >> v2:172.21.15.93:3300/0 conn(0x7fb6b006ac00 0x7fb6b00b2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:38.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.293+0000 7fb6b8000640 1 -- 172.21.15.27:0/363466394 >> 172.21.15.27:0/363466394 conn(0x7fb6b006b600 msgr2=0x7fb6b006b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:38.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.293+0000 7fb6b8000640 1 -- 172.21.15.27:0/363466394 shutdown_connections 2024-01-15T18:18:38.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.293+0000 7fb6b8000640 1 -- 172.21.15.27:0/363466394 wait complete. 2024-01-15T18:18:38.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.297+0000 7fb6b8000640 1 Processor -- start 2024-01-15T18:18:38.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.297+0000 7fb6b8000640 1 -- start start 2024-01-15T18:18:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.297+0000 7fb6b8000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6b006ac00 0x7fb6b00cd150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.297+0000 7fb6b8000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb6b006d020 0x7fb6b00b1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.297+0000 7fb6b8000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6b00af240 0x7fb6b00b1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.297+0000 7fb6b8000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b00cd7c0 con 0x7fb6b00af240 2024-01-15T18:18:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.297+0000 7fb6b8000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b00cdaa0 con 0x7fb6b006d020 2024-01-15T18:18:38.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.297+0000 7fb6b8000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b00cdd80 con 0x7fb6b006ac00 2024-01-15T18:18:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.297+0000 7fb6b6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6b00af240 0x7fb6b00b1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.297+0000 7fb6b6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6b00af240 0x7fb6b00b1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38178/0 (socket says 172.21.15.27:38178) 2024-01-15T18:18:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.297+0000 7fb6b6000640 1 -- 172.21.15.27:0/2602331346 learned_addr learned my addr 172.21.15.27:0/2602331346 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.297+0000 7fb6b5600640 1 --2- 172.21.15.27:0/2602331346 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6b006ac00 0x7fb6b00cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.297+0000 7fb6b4c00640 1 --2- 172.21.15.27:0/2602331346 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb6b006d020 0x7fb6b00b1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.297+0000 7fb6b6000640 1 -- 172.21.15.27:0/2602331346 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6b006ac00 msgr2=0x7fb6b00cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.297+0000 7fb6b6000640 1 --2- 172.21.15.27:0/2602331346 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6b006ac00 0x7fb6b00cd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.297+0000 7fb6b6000640 1 -- 172.21.15.27:0/2602331346 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb6b006d020 msgr2=0x7fb6b00b1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.297+0000 7fb6b6000640 1 --2- 172.21.15.27:0/2602331346 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb6b006d020 0x7fb6b00b1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.297+0000 7fb6b6000640 1 -- 172.21.15.27:0/2602331346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a8009b30 con 0x7fb6b00af240 2024-01-15T18:18:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.297+0000 7fb6b6000640 1 --2- 172.21.15.27:0/2602331346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6b00af240 0x7fb6b00b1590 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fb6a80af040 tx=0x7fb6a80196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:38.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.297+0000 7fb6ad600640 1 -- 172.21.15.27:0/2602331346 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a80b8670 con 0x7fb6b00af240 2024-01-15T18:18:38.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.297+0000 7fb6ad600640 1 -- 172.21.15.27:0/2602331346 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a80b87d0 con 0x7fb6b00af240 2024-01-15T18:18:38.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.297+0000 7fb6b8000640 1 -- 172.21.15.27:0/2602331346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6b00ce060 con 0x7fb6b00af240 2024-01-15T18:18:38.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.297+0000 7fb6ad600640 1 -- 172.21.15.27:0/2602331346 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a80b8b60 con 0x7fb6b00af240 2024-01-15T18:18:38.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.297+0000 7fb6b8000640 1 -- 172.21.15.27:0/2602331346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b014d020 con 0x7fb6b00af240 2024-01-15T18:18:38.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.301+0000 7fb6b8000640 1 -- 172.21.15.27:0/2602331346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6b004e850 con 0x7fb6b00af240 2024-01-15T18:18:38.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.301+0000 7fb6ad600640 1 -- 172.21.15.27:0/2602331346 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7fb6a80b8cc0 con 0x7fb6b00af240 2024-01-15T18:18:38.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.301+0000 7fb6ad600640 1 --2- 172.21.15.27:0/2602331346 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb688067940 0x7fb688069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:38.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.301+0000 7fb6ad600640 1 -- 172.21.15.27:0/2602331346 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fb6a811e750 con 0x7fb6b00af240 2024-01-15T18:18:38.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.301+0000 7fb6b5600640 1 --2- 172.21.15.27:0/2602331346 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb688067940 0x7fb688069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:38.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.301+0000 7fb6b5600640 1 --2- 172.21.15.27:0/2602331346 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb688067940 0x7fb688069e00 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fb6a0007fd0 tx=0x7fb6a0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:38.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.305+0000 7fb6ad600640 1 -- 172.21.15.27:0/2602331346 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb6a80ed230 con 0x7fb6b00af240 2024-01-15T18:18:38.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.493+0000 7fb6b8000640 1 -- 172.21.15.27:0/2602331346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6b00cdec0 con 0x7fb6b00af240 2024-01-15T18:18:38.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.497+0000 7fb6ad600640 1 -- 172.21.15.27:0/2602331346 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+4915 (secure 0 0 0) 0x7fb6b00cdec0 con 0x7fb6b00af240 2024-01-15T18:18:38.495 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:18:38.496 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":77,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7721,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3036261071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":3036261071},{"type":"v1","addr":"172.21.15.27:6835","nonce":3036261071}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72},{"gid":7724,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/3646217660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":3646217660},{"type":"v1","addr":"172.21.15.93:6836","nonce":3646217660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72},{"gid":9441,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3131124481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3131124481},{"type":"v1","addr":"172.21.15.27:6834","nonce":3131124481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:18:32.593319+0000","modified":"2024-01-15T18:18:35.606888+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9450},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9450":{"gid":9450,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.93:6837/2040168756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":2040168756},{"type":"v1","addr":"172.21.15.93:6837","nonce":2040168756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-01-15T18:18:38.496 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 77 2024-01-15T18:18:38.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.497+0000 7fb6b8000640 1 -- 172.21.15.27:0/2602331346 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb688067940 msgr2=0x7fb688069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:38.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.497+0000 7fb6b8000640 1 --2- 172.21.15.27:0/2602331346 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb688067940 0x7fb688069e00 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fb6a0007fd0 tx=0x7fb6a0009040 comp rx=0 tx=0).stop 2024-01-15T18:18:38.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.497+0000 7fb6b8000640 1 -- 172.21.15.27:0/2602331346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6b00af240 msgr2=0x7fb6b00b1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:38.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.497+0000 7fb6b8000640 1 --2- 172.21.15.27:0/2602331346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6b00af240 0x7fb6b00b1590 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fb6a80af040 tx=0x7fb6a80196b0 comp rx=0 tx=0).stop 2024-01-15T18:18:38.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.497+0000 7fb6b8000640 1 -- 172.21.15.27:0/2602331346 shutdown_connections 2024-01-15T18:18:38.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.497+0000 7fb6b8000640 1 --2- 172.21.15.27:0/2602331346 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb688067940 0x7fb688069e00 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:38.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.497+0000 7fb6b8000640 1 --2- 172.21.15.27:0/2602331346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6b00af240 0x7fb6b00b1590 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:38.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.497+0000 7fb6b8000640 1 --2- 172.21.15.27:0/2602331346 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb6b006d020 0x7fb6b00b1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:38.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.497+0000 7fb6b8000640 1 --2- 172.21.15.27:0/2602331346 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb6b006ac00 0x7fb6b00cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:38.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.497+0000 7fb6b8000640 1 -- 172.21.15.27:0/2602331346 >> 172.21.15.27:0/2602331346 conn(0x7fb6b006b600 msgr2=0x7fb6b0156240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:38.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.497+0000 7fb6b8000640 1 -- 172.21.15.27:0/2602331346 shutdown_connections 2024-01-15T18:18:38.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:38.497+0000 7fb6b8000640 1 -- 172.21.15.27:0/2602331346 wait complete. 2024-01-15T18:18:38.920 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-15T18:18:32.593319+0000', 'modified': '2024-01-15T18:18:35.606888+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9450}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9450': {'gid': 9450, 'name': 'd', 'rank': 0, 'incarnation': 73, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.93:6837/2040168756', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.93:6835', 'nonce': 2040168756}, {'type': 'v1', 'addr': '172.21.15.93:6837', 'nonce': 2040168756}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-15T18:18:38.920 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-15T18:18:38.920 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-01-15T18:18:39.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.065+0000 7fac9bc00640 1 Processor -- start 2024-01-15T18:18:39.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.065+0000 7fac9bc00640 1 -- start start 2024-01-15T18:18:39.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.065+0000 7fac9bc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fac9415a320 0x7fac9406cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.065+0000 7fac9bc00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac94070b90 con 0x7fac94153c80 2024-01-15T18:18:39.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.065+0000 7fac9bc00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac9415a730 con 0x7fac9406ca90 2024-01-15T18:18:39.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.065+0000 7fac9bc00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac9406d460 con 0x7fac9415a320 2024-01-15T18:18:39.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.065+0000 7fac99200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fac94153c80 0x7fac94071020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38704/0 (socket says 172.21.15.27:38704) 2024-01-15T18:18:39.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.065+0000 7fac99200640 1 -- 172.21.15.27:0/1039454976 learned_addr learned my addr 172.21.15.27:0/1039454976 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:39.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.065+0000 7fac99c00640 1 --2- 172.21.15.27:0/1039454976 >> v2:172.21.15.27:3300/0 conn(0x7fac9415a320 0x7fac9406cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.065+0000 7fac93400640 1 -- 172.21.15.27:0/1039454976 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3667837864 0 0) 0x7fac94070b90 con 0x7fac94153c80 2024-01-15T18:18:39.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.065+0000 7fac93400640 1 -- 172.21.15.27:0/1039454976 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac74003530 con 0x7fac94153c80 2024-01-15T18:18:39.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.065+0000 7fac99c00640 1 -- 172.21.15.27:0/1039454976 >> v1:172.21.15.93:6789/0 conn(0x7fac9406ca90 legacy=0x7fac94159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:39.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.065+0000 7fac99c00640 1 -- 172.21.15.27:0/1039454976 >> v1:172.21.15.27:6789/0 conn(0x7fac94153c80 legacy=0x7fac94071020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:39.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.065+0000 7fac99c00640 1 -- 172.21.15.27:0/1039454976 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9415a980 con 0x7fac9415a320 2024-01-15T18:18:39.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.069+0000 7fac99c00640 1 --2- 172.21.15.27:0/1039454976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9415a320 0x7fac9406cf20 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fac8c009d90 tx=0x7fac8c01c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fefa565bce91e246 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:39.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.069+0000 7fac93400640 1 -- 172.21.15.27:0/1039454976 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c0a57f0 con 0x7fac9415a320 2024-01-15T18:18:39.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.069+0000 7fac93400640 1 -- 172.21.15.27:0/1039454976 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac8c0a5950 con 0x7fac9415a320 2024-01-15T18:18:39.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.069+0000 7fac93400640 1 -- 172.21.15.27:0/1039454976 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c0a5ce0 con 0x7fac9415a320 2024-01-15T18:18:39.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.069+0000 7fac9bc00640 1 -- 172.21.15.27:0/1039454976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9415a320 msgr2=0x7fac9406cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:39.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.069+0000 7fac9bc00640 1 --2- 172.21.15.27:0/1039454976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9415a320 0x7fac9406cf20 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fac8c009d90 tx=0x7fac8c01c8c0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.069+0000 7fac9bc00640 1 -- 172.21.15.27:0/1039454976 shutdown_connections 2024-01-15T18:18:39.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.069+0000 7fac9bc00640 1 --2- 172.21.15.27:0/1039454976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9415a320 0x7fac9406cf20 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.069+0000 7fac9bc00640 1 -- 172.21.15.27:0/1039454976 >> 172.21.15.27:0/1039454976 conn(0x7fac940b0850 msgr2=0x7fac940b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:39.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.069+0000 7fac9bc00640 1 -- 172.21.15.27:0/1039454976 shutdown_connections 2024-01-15T18:18:39.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.069+0000 7fac9bc00640 1 -- 172.21.15.27:0/1039454976 wait complete. 2024-01-15T18:18:39.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.073+0000 7fac9bc00640 1 Processor -- start 2024-01-15T18:18:39.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.073+0000 7fac9bc00640 1 -- start start 2024-01-15T18:18:39.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.073+0000 7fac9bc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac9406ca90 0x7fac9414ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.073+0000 7fac9bc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fac94153c80 0x7fac941410f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.073+0000 7fac9bc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9415a320 0x7fac9414b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.073+0000 7fac9bc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac94149870 con 0x7fac9415a320 2024-01-15T18:18:39.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.073+0000 7fac9bc00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac94141630 con 0x7fac94153c80 2024-01-15T18:18:39.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.073+0000 7fac9bc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac94149ac0 con 0x7fac9406ca90 2024-01-15T18:18:39.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.073+0000 7fac99c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9415a320 0x7fac9414b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.073+0000 7fac99c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9415a320 0x7fac9414b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38202/0 (socket says 172.21.15.27:38202) 2024-01-15T18:18:39.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.073+0000 7fac99c00640 1 -- 172.21.15.27:0/3776292606 learned_addr learned my addr 172.21.15.27:0/3776292606 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:39.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.073+0000 7fac99200640 1 --2- 172.21.15.27:0/3776292606 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac9406ca90 0x7fac9414ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.073+0000 7fac93e00640 1 --2- 172.21.15.27:0/3776292606 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fac94153c80 0x7fac941410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.073+0000 7fac99c00640 1 -- 172.21.15.27:0/3776292606 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac9406ca90 msgr2=0x7fac9414ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:39.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.073+0000 7fac99c00640 1 --2- 172.21.15.27:0/3776292606 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac9406ca90 0x7fac9414ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.073+0000 7fac99c00640 1 -- 172.21.15.27:0/3776292606 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fac94153c80 msgr2=0x7fac941410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:39.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.073+0000 7fac99c00640 1 --2- 172.21.15.27:0/3776292606 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fac94153c80 0x7fac941410f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.073+0000 7fac99c00640 1 -- 172.21.15.27:0/3776292606 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac8c009a70 con 0x7fac9415a320 2024-01-15T18:18:39.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.073+0000 7fac99c00640 1 --2- 172.21.15.27:0/3776292606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9415a320 0x7fac9414b910 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fac8c0b0040 tx=0x7fac8c019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:39.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.073+0000 7fac90c00640 1 -- 172.21.15.27:0/3776292606 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c0b9650 con 0x7fac9415a320 2024-01-15T18:18:39.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.073+0000 7fac90c00640 1 -- 172.21.15.27:0/3776292606 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac8c0b97b0 con 0x7fac9415a320 2024-01-15T18:18:39.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.073+0000 7fac90c00640 1 -- 172.21.15.27:0/3776292606 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c0b9b60 con 0x7fac9415a320 2024-01-15T18:18:39.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.073+0000 7fac9bc00640 1 -- 172.21.15.27:0/3776292606 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac94149da0 con 0x7fac9415a320 2024-01-15T18:18:39.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.073+0000 7fac9bc00640 1 -- 172.21.15.27:0/3776292606 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac9414a360 con 0x7fac9415a320 2024-01-15T18:18:39.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.077+0000 7fac90c00640 1 -- 172.21.15.27:0/3776292606 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7fac8c0b9cc0 con 0x7fac9415a320 2024-01-15T18:18:39.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.077+0000 7fac90c00640 1 --2- 172.21.15.27:0/3776292606 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fac6c067940 0x7fac6c069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.077+0000 7fac90c00640 1 -- 172.21.15.27:0/3776292606 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fac8c11e340 con 0x7fac9415a320 2024-01-15T18:18:39.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.077+0000 7fac9f09d000 1 Processor -- start 2024-01-15T18:18:39.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.077+0000 7fac99200640 1 --2- 172.21.15.27:0/3776292606 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fac6c067940 0x7fac6c069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.077+0000 7fac9f09d000 1 -- start start 2024-01-15T18:18:39.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.077+0000 7fac99200640 1 --2- 172.21.15.27:0/3776292606 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fac6c067940 0x7fac6c069e00 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fac840011f0 tx=0x7fac84007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:39.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac9f09d000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55b0edaf0c90 0x55b0edaf1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac9f09d000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55b0edaf15b0 0x55b0edaa6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac9f09d000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaa7320 0x55b0edaa7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac9f09d000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0ed911130 con 0x55b0edaa7320 2024-01-15T18:18:39.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac9f09d000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0ed8fe9a0 con 0x55b0edaf0c90 2024-01-15T18:18:39.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac9f09d000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b0ed8fbed0 con 0x55b0edaf15b0 2024-01-15T18:18:39.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac99c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaa7320 0x55b0edaa7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac93e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55b0edaf15b0 0x55b0edaa6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac99c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaa7320 0x55b0edaa7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38206/0 (socket says 172.21.15.27:38206) 2024-01-15T18:18:39.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac99c00640 1 -- 172.21.15.27:0/3949668368 learned_addr learned my addr 172.21.15.27:0/3949668368 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:39.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac99200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55b0edaf0c90 0x55b0edaf1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac99c00640 1 -- 172.21.15.27:0/3949668368 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55b0edaf15b0 msgr2=0x55b0edaa6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:39.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac99c00640 1 --2- 172.21.15.27:0/3949668368 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55b0edaf15b0 0x55b0edaa6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac99c00640 1 -- 172.21.15.27:0/3949668368 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55b0edaf0c90 msgr2=0x55b0edaf1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:39.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac99c00640 1 --2- 172.21.15.27:0/3949668368 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55b0edaf0c90 0x55b0edaf1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac99c00640 1 -- 172.21.15.27:0/3949668368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b0ed5ef890 con 0x55b0edaa7320 2024-01-15T18:18:39.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac99c00640 1 --2- 172.21.15.27:0/3949668368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaa7320 0x55b0edaa7700 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fac8c0f87a0 tx=0x7fac8c0f0350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff9705ed29b41776 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:39.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac89600640 1 -- 172.21.15.27:0/3949668368 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c0f8cd0 con 0x55b0edaa7320 2024-01-15T18:18:39.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac89600640 1 -- 172.21.15.27:0/3949668368 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac8c0ed100 con 0x55b0edaa7320 2024-01-15T18:18:39.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac89600640 1 -- 172.21.15.27:0/3949668368 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c0f0e90 con 0x55b0edaa7320 2024-01-15T18:18:39.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac9f09d000 1 -- 172.21.15.27:0/3949668368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaa7320 msgr2=0x55b0edaa7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:39.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac9f09d000 1 --2- 172.21.15.27:0/3949668368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaa7320 0x55b0edaa7700 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fac8c0f87a0 tx=0x7fac8c0f0350 comp rx=0 tx=0).stop 2024-01-15T18:18:39.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac9f09d000 1 -- 172.21.15.27:0/3949668368 shutdown_connections 2024-01-15T18:18:39.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac9f09d000 1 --2- 172.21.15.27:0/3949668368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaa7320 0x55b0edaa7700 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac9f09d000 1 --2- 172.21.15.27:0/3949668368 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55b0edaf15b0 0x55b0edaa6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac9f09d000 1 --2- 172.21.15.27:0/3949668368 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55b0edaf0c90 0x55b0edaf1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac9f09d000 1 -- 172.21.15.27:0/3949668368 >> 172.21.15.27:0/3949668368 conn(0x55b0edaa32d0 msgr2=0x55b0edaa56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:39.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac9f09d000 1 -- 172.21.15.27:0/3949668368 shutdown_connections 2024-01-15T18:18:39.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac9f09d000 1 -- 172.21.15.27:0/3949668368 wait complete. 2024-01-15T18:18:39.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac9f09d000 1 -- start start 2024-01-15T18:18:39.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.081+0000 7fac9f09d000 1 Processor -- start 2024-01-15T18:18:39.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac9f09d000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55b0edaa7320 0x55b0edaabd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac9f09d000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55b0edaf0c90 0x55b0edaac280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac9f09d000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaf15b0 0x55b0edabf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac9f09d000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0ed8f77c0 con 0x55b0edaf15b0 2024-01-15T18:18:39.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac9f09d000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0ed8afbf0 con 0x55b0edaa7320 2024-01-15T18:18:39.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac9f09d000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b0ed8b5a90 con 0x55b0edaf0c90 2024-01-15T18:18:39.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac9f09d000 10 client.0 _ll_register_callbacks cb 0x55b0eda923c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:18:39.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac99c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaf15b0 0x55b0edabf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac99c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaf15b0 0x55b0edabf910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38210/0 (socket says 172.21.15.27:38210) 2024-01-15T18:18:39.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac99c00640 1 -- 172.21.15.27:0/976058209 learned_addr learned my addr 172.21.15.27:0/976058209 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:39.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac99200640 1 --2- 172.21.15.27:0/976058209 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55b0edaa7320 0x55b0edaabd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac93e00640 1 --2- 172.21.15.27:0/976058209 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55b0edaf0c90 0x55b0edaac280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac99c00640 1 -- 172.21.15.27:0/976058209 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55b0edaf0c90 msgr2=0x55b0edaac280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:39.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac99c00640 1 --2- 172.21.15.27:0/976058209 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55b0edaf0c90 0x55b0edaac280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac99c00640 1 -- 172.21.15.27:0/976058209 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55b0edaa7320 msgr2=0x55b0edaabd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:39.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac99c00640 1 --2- 172.21.15.27:0/976058209 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55b0edaa7320 0x55b0edaabd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac99c00640 1 -- 172.21.15.27:0/976058209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac9414a360 con 0x55b0edaf15b0 2024-01-15T18:18:39.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac99200640 1 --2- 172.21.15.27:0/976058209 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55b0edaa7320 0x55b0edaabd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:18:39.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac99c00640 1 --2- 172.21.15.27:0/976058209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaf15b0 0x55b0edabf910 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fac8c009d90 tx=0x7fac8c0b6000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ef423e840f6c62 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:39.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac82000640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-15T18:18:39.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac82000640 1 -- 172.21.15.27:0/976058209 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c0f00e0 con 0x55b0edaf15b0 2024-01-15T18:18:39.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac82000640 1 -- 172.21.15.27:0/976058209 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fac8c0f8cd0 con 0x55b0edaf15b0 2024-01-15T18:18:39.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac9f09d000 1 -- 172.21.15.27:0/976058209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b0ed8b9470 con 0x55b0edaf15b0 2024-01-15T18:18:39.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac82000640 1 -- 172.21.15.27:0/976058209 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac8c108310 con 0x55b0edaf15b0 2024-01-15T18:18:39.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac9f09d000 10 client.9582 fetch_fsmap learned FSMap version 77 2024-01-15T18:18:39.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac9f09d000 1 -- 172.21.15.27:0/976058209 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=2+}) v3 -- 0x55b0ed8b8190 con 0x55b0edaf15b0 2024-01-15T18:18:39.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac82000640 1 -- 172.21.15.27:0/976058209 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b0ed8b9470 con 0x55b0edaf15b0 2024-01-15T18:18:39.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac82000640 1 -- 172.21.15.27:0/976058209 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 77) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fac8c005d20 con 0x55b0edaf15b0 2024-01-15T18:18:39.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac9f09d000 10 client.9582 fetch_fsmap finished waiting for FSMap version 77 2024-01-15T18:18:39.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac9f09d000 10 client.9582 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-15T18:18:39.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac9f09d000 20 client.9582 populate_metadata read hostname 'smithi027' 2024-01-15T18:18:39.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac9f09d000 1 --2- 172.21.15.27:0/976058209 >> [v2:172.21.15.93:6835/2040168756,v1:172.21.15.93:6837/2040168756] conn(0x55b0edac0320 0x55b0edb02f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac9f09d000 4 client.9582 mds_command: new command op to 9450 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-15T18:18:39.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac9f09d000 1 -- 172.21.15.27:0/976058209 --> [v2:172.21.15.93:6835/2040168756,v1:172.21.15.93:6837/2040168756] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b0ed8bcb50 con 0x55b0edac0320 2024-01-15T18:18:39.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac99200640 1 --2- 172.21.15.27:0/976058209 >> [v2:172.21.15.93:6835/2040168756,v1:172.21.15.93:6837/2040168756] conn(0x55b0edac0320 0x55b0edb02f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac99200640 1 --2- 172.21.15.27:0/976058209 >> [v2:172.21.15.93:6835/2040168756,v1:172.21.15.93:6837/2040168756] conn(0x55b0edac0320 0x55b0edb02f50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=919203e6cc36b6a3 server_cookie=281ef2a5ef1dc270 in_seq=0 out_seq=0 2024-01-15T18:18:39.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.085+0000 7fac82000640 10 client.9582 ms_handle_connect on v2:172.21.15.93:6835/2040168756 2024-01-15T18:18:39.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.089+0000 7fac82000640 1 -- 172.21.15.27:0/976058209 <== mds.0 v2:172.21.15.93:6835/2040168756 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b0ed8bcb50 con 0x55b0edac0320 2024-01-15T18:18:39.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.089+0000 7fac82000640 10 client.9582 handle_command_reply: tid=0 2024-01-15T18:18:39.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.089+0000 7fac9f09d000 1 client.9582 shutdown 2024-01-15T18:18:39.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.089+0000 7fac9f09d000 1 -- 172.21.15.27:0/976058209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaf15b0 msgr2=0x55b0edabf910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:39.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.089+0000 7fac9f09d000 1 --2- 172.21.15.27:0/976058209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaf15b0 0x55b0edabf910 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fac8c009d90 tx=0x7fac8c0b6000 comp rx=0 tx=0).stop 2024-01-15T18:18:39.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.089+0000 7fac9f09d000 1 -- 172.21.15.27:0/976058209 shutdown_connections 2024-01-15T18:18:39.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.089+0000 7fac9f09d000 1 --2- 172.21.15.27:0/976058209 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaf15b0 0x55b0edabf910 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.089+0000 7fac9f09d000 1 --2- 172.21.15.27:0/976058209 >> [v2:172.21.15.93:6835/2040168756,v1:172.21.15.93:6837/2040168756] conn(0x55b0edac0320 0x55b0edb02f50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.089+0000 7fac9f09d000 1 --2- 172.21.15.27:0/976058209 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55b0edaf0c90 0x55b0edaac280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.089+0000 7fac9f09d000 1 --2- 172.21.15.27:0/976058209 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55b0edaa7320 0x55b0edaabd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.089+0000 7fac82000640 0 client.9582 ms_handle_reset on v2:172.21.15.93:6835/2040168756 2024-01-15T18:18:39.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.089+0000 7fac9f09d000 1 -- 172.21.15.27:0/976058209 >> 172.21.15.27:0/976058209 conn(0x55b0edaa32d0 msgr2=0x55b0edaa56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:39.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.089+0000 7fac9f09d000 1 -- 172.21.15.27:0/976058209 shutdown_connections 2024-01-15T18:18:39.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.089+0000 7fac9f09d000 1 -- 172.21.15.27:0/976058209 wait complete. 2024-01-15T18:18:39.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.089+0000 7fac9f09d000 20 client.9582 trim_cache size 0 max 16384 2024-01-15T18:18:39.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.105+0000 7fac9f09d000 1 Processor -- start 2024-01-15T18:18:39.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.105+0000 7fac9f09d000 1 -- start start 2024-01-15T18:18:39.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.105+0000 7fac9f09d000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55b0edaa7320 0x55b0edaefc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.105+0000 7fac9f09d000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaf0c90 0x55b0edaeeb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.105+0000 7fac9f09d000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55b0edaf15b0 0x55b0edaef070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.105+0000 7fac9f09d000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0ed5b0940 con 0x55b0edaf0c90 2024-01-15T18:18:39.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.105+0000 7fac9f09d000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0ed8f6650 con 0x55b0edaf15b0 2024-01-15T18:18:39.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.105+0000 7fac9f09d000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b0ed87ea00 con 0x55b0edaa7320 2024-01-15T18:18:39.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.105+0000 7fac93e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaf0c90 0x55b0edaeeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.105+0000 7fac93e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaf0c90 0x55b0edaeeb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38224/0 (socket says 172.21.15.27:38224) 2024-01-15T18:18:39.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.105+0000 7fac93e00640 1 -- 172.21.15.27:0/1534431775 learned_addr learned my addr 172.21.15.27:0/1534431775 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:39.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.105+0000 7fac99200640 1 --2- 172.21.15.27:0/1534431775 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55b0edaa7320 0x55b0edaefc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.105+0000 7fac99c00640 1 --2- 172.21.15.27:0/1534431775 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55b0edaf15b0 0x55b0edaef070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.105+0000 7fac93e00640 1 -- 172.21.15.27:0/1534431775 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55b0edaa7320 msgr2=0x55b0edaefc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:39.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.105+0000 7fac93e00640 1 --2- 172.21.15.27:0/1534431775 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55b0edaa7320 0x55b0edaefc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.105+0000 7fac93e00640 1 -- 172.21.15.27:0/1534431775 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55b0edaf15b0 msgr2=0x55b0edaef070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:39.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.105+0000 7fac93e00640 1 --2- 172.21.15.27:0/1534431775 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55b0edaf15b0 0x55b0edaef070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.105+0000 7fac93e00640 1 -- 172.21.15.27:0/1534431775 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b0ed89e490 con 0x55b0edaf0c90 2024-01-15T18:18:39.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.105+0000 7fac99c00640 1 --2- 172.21.15.27:0/1534431775 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55b0edaf15b0 0x55b0edaef070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:18:39.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.105+0000 7fac93e00640 1 --2- 172.21.15.27:0/1534431775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaf0c90 0x55b0edaeeb30 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fac7c001750 tx=0x7fac7c007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a063667bba2f83e8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:39.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac8a000640 1 -- 172.21.15.27:0/1534431775 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c0052e0 con 0x55b0edaf0c90 2024-01-15T18:18:39.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac8a000640 1 -- 172.21.15.27:0/1534431775 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac7c005440 con 0x55b0edaf0c90 2024-01-15T18:18:39.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac8a000640 1 -- 172.21.15.27:0/1534431775 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c015b10 con 0x55b0edaf0c90 2024-01-15T18:18:39.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac9f09d000 1 -- 172.21.15.27:0/1534431775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaf0c90 msgr2=0x55b0edaeeb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:39.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac9f09d000 1 --2- 172.21.15.27:0/1534431775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaf0c90 0x55b0edaeeb30 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fac7c001750 tx=0x7fac7c007e40 comp rx=0 tx=0).stop 2024-01-15T18:18:39.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac9f09d000 1 -- 172.21.15.27:0/1534431775 shutdown_connections 2024-01-15T18:18:39.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac9f09d000 1 --2- 172.21.15.27:0/1534431775 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55b0edaf15b0 0x55b0edaef070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac9f09d000 1 --2- 172.21.15.27:0/1534431775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaf0c90 0x55b0edaeeb30 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac9f09d000 1 --2- 172.21.15.27:0/1534431775 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55b0edaa7320 0x55b0edaefc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac9f09d000 1 -- 172.21.15.27:0/1534431775 >> 172.21.15.27:0/1534431775 conn(0x55b0edaa32d0 msgr2=0x55b0edafe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:39.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac9f09d000 1 -- 172.21.15.27:0/1534431775 shutdown_connections 2024-01-15T18:18:39.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac9f09d000 1 -- 172.21.15.27:0/1534431775 wait complete. 2024-01-15T18:18:39.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac9f09d000 1 -- start start 2024-01-15T18:18:39.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac9f09d000 1 Processor -- start 2024-01-15T18:18:39.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac9f09d000 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55b0edaa7320 0x55b0edaa8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac9f09d000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaf0c90 0x55b0edb03fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac9f09d000 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55b0edaf15b0 0x55b0edb044e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac9f09d000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0ed8f4fd0 con 0x55b0edaf0c90 2024-01-15T18:18:39.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac9f09d000 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0edaac7c0 con 0x55b0edaa7320 2024-01-15T18:18:39.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac9f09d000 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b0ed88e420 con 0x55b0edaf15b0 2024-01-15T18:18:39.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac9f09d000 10 client.0 _ll_register_callbacks cb 0x55b0ed945c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:18:39.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac93e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaf0c90 0x55b0edb03fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac93e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaf0c90 0x55b0edb03fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38234/0 (socket says 172.21.15.27:38234) 2024-01-15T18:18:39.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac93e00640 1 -- 172.21.15.27:0/2483988379 learned_addr learned my addr 172.21.15.27:0/2483988379 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:39.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac99200640 1 --2- 172.21.15.27:0/2483988379 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55b0edaa7320 0x55b0edaa8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac99c00640 1 --2- 172.21.15.27:0/2483988379 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55b0edaf15b0 0x55b0edb044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac93e00640 1 -- 172.21.15.27:0/2483988379 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55b0edaf15b0 msgr2=0x55b0edb044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:39.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac93e00640 1 --2- 172.21.15.27:0/2483988379 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55b0edaf15b0 0x55b0edb044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac93e00640 1 -- 172.21.15.27:0/2483988379 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55b0edaa7320 msgr2=0x55b0edaa8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:39.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac93e00640 1 --2- 172.21.15.27:0/2483988379 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55b0edaa7320 0x55b0edaa8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.109+0000 7fac93e00640 1 -- 172.21.15.27:0/2483988379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac9415a730 con 0x55b0edaf0c90 2024-01-15T18:18:39.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac93e00640 1 --2- 172.21.15.27:0/2483988379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaf0c90 0x55b0edb03fa0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x55b0ed96ec50 tx=0x7fac7c000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f97c063a950d8e8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:39.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac82000640 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2024-01-15T18:18:39.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac82000640 1 -- 172.21.15.27:0/2483988379 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c025070 con 0x55b0edaf0c90 2024-01-15T18:18:39.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac82000640 1 -- 172.21.15.27:0/2483988379 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(108..108 src has 1..108) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fac7c0067e0 con 0x55b0edaf0c90 2024-01-15T18:18:39.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac9f09d000 1 -- 172.21.15.27:0/2483988379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b0ed8f6110 con 0x55b0edaf0c90 2024-01-15T18:18:39.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac82000640 1 -- 172.21.15.27:0/2483988379 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac7c0052e0 con 0x55b0edaf0c90 2024-01-15T18:18:39.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac9f09d000 10 client.9588 fetch_fsmap learned FSMap version 77 2024-01-15T18:18:39.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac9f09d000 1 -- 172.21.15.27:0/2483988379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=77,monmap=2+}) v3 -- 0x55b0ed5efcc0 con 0x55b0edaf0c90 2024-01-15T18:18:39.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac82000640 1 -- 172.21.15.27:0/2483988379 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b0ed8f6110 con 0x55b0edaf0c90 2024-01-15T18:18:39.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac82000640 1 -- 172.21.15.27:0/2483988379 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 77) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fac7c012050 con 0x55b0edaf0c90 2024-01-15T18:18:39.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac9f09d000 10 client.9588 fetch_fsmap finished waiting for FSMap version 77 2024-01-15T18:18:39.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac9f09d000 10 client.9588 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-15T18:18:39.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac9f09d000 20 client.9588 populate_metadata read hostname 'smithi027' 2024-01-15T18:18:39.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac9f09d000 1 --2- 172.21.15.27:0/2483988379 >> [v2:172.21.15.93:6835/2040168756,v1:172.21.15.93:6837/2040168756] conn(0x55b0edb51010 0x55b0edb513f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac9f09d000 4 client.9588 mds_command: new command op to 9450 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-15T18:18:39.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac9f09d000 1 -- 172.21.15.27:0/2483988379 --> [v2:172.21.15.93:6835/2040168756,v1:172.21.15.93:6837/2040168756] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b0ed9259a0 con 0x55b0edb51010 2024-01-15T18:18:39.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac99200640 1 --2- 172.21.15.27:0/2483988379 >> [v2:172.21.15.93:6835/2040168756,v1:172.21.15.93:6837/2040168756] conn(0x55b0edb51010 0x55b0edb513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac99200640 1 --2- 172.21.15.27:0/2483988379 >> [v2:172.21.15.93:6835/2040168756,v1:172.21.15.93:6837/2040168756] conn(0x55b0edb51010 0x55b0edb513f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9180275aa3311073 server_cookie=40f5abf67e0421fc in_seq=0 out_seq=0 2024-01-15T18:18:39.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac82000640 10 client.9588 ms_handle_connect on v2:172.21.15.93:6835/2040168756 2024-01-15T18:18:39.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac82000640 1 -- 172.21.15.27:0/2483988379 <== mds.0 v2:172.21.15.93:6835/2040168756 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55b0ed9259a0 con 0x55b0edb51010 2024-01-15T18:18:39.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac82000640 10 client.9588 handle_command_reply: tid=0 2024-01-15T18:18:39.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac9f09d000 1 client.9588 shutdown 2024-01-15T18:18:39.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac9f09d000 1 -- 172.21.15.27:0/2483988379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaf0c90 msgr2=0x55b0edb03fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:39.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac9f09d000 1 --2- 172.21.15.27:0/2483988379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaf0c90 0x55b0edb03fa0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x55b0ed96ec50 tx=0x7fac7c000ee0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac9f09d000 1 -- 172.21.15.27:0/2483988379 shutdown_connections 2024-01-15T18:18:39.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac9f09d000 1 --2- 172.21.15.27:0/2483988379 >> [v2:172.21.15.93:6835/2040168756,v1:172.21.15.93:6837/2040168756] conn(0x55b0edb51010 0x55b0edb513f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac9f09d000 1 --2- 172.21.15.27:0/2483988379 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x55b0edaf15b0 0x55b0edb044e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac9f09d000 1 --2- 172.21.15.27:0/2483988379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0edaf0c90 0x55b0edb03fa0 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac9f09d000 1 --2- 172.21.15.27:0/2483988379 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x55b0edaa7320 0x55b0edaa8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac82000640 0 client.9588 ms_handle_reset on v2:172.21.15.93:6835/2040168756 2024-01-15T18:18:39.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac9f09d000 1 -- 172.21.15.27:0/2483988379 >> 172.21.15.27:0/2483988379 conn(0x55b0edaa32d0 msgr2=0x55b0edaa56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:39.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac9f09d000 1 -- 172.21.15.27:0/2483988379 shutdown_connections 2024-01-15T18:18:39.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac9f09d000 1 -- 172.21.15.27:0/2483988379 wait complete. 2024-01-15T18:18:39.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.113+0000 7fac9f09d000 20 client.9588 trim_cache size 0 max 16384 2024-01-15T18:18:39.118 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:18:39.118 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "fce09647-6947-4a73-bd3d-e3d154552087", 2024-01-15T18:18:39.118 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2024-01-15T18:18:39.118 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 9450, 2024-01-15T18:18:39.118 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2024-01-15T18:18:39.118 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2024-01-15T18:18:39.118 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2024-01-15T18:18:39.118 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 5.9185000719999996, 2024-01-15T18:18:39.118 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 77, 2024-01-15T18:18:39.118 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 108, 2024-01-15T18:18:39.118 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 108, 2024-01-15T18:18:39.118 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 10.801945312999999 2024-01-15T18:18:39.118 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:18:39.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.117+0000 7fac9bc00640 1 -- 172.21.15.27:0/3776292606 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fac6c067940 msgr2=0x7fac6c069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:39.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.117+0000 7fac9bc00640 1 --2- 172.21.15.27:0/3776292606 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fac6c067940 0x7fac6c069e00 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fac840011f0 tx=0x7fac84007d40 comp rx=0 tx=0).stop 2024-01-15T18:18:39.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.117+0000 7fac9bc00640 1 -- 172.21.15.27:0/3776292606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9415a320 msgr2=0x7fac9414b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:39.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.117+0000 7fac9bc00640 1 --2- 172.21.15.27:0/3776292606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9415a320 0x7fac9414b910 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fac8c0b0040 tx=0x7fac8c019660 comp rx=0 tx=0).stop 2024-01-15T18:18:39.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.117+0000 7fac9bc00640 1 -- 172.21.15.27:0/3776292606 shutdown_connections 2024-01-15T18:18:39.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.117+0000 7fac9bc00640 1 --2- 172.21.15.27:0/3776292606 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fac6c067940 0x7fac6c069e00 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.117+0000 7fac9bc00640 1 --2- 172.21.15.27:0/3776292606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9415a320 0x7fac9414b910 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.117+0000 7fac9bc00640 1 --2- 172.21.15.27:0/3776292606 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fac94153c80 0x7fac941410f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.117+0000 7fac9bc00640 1 --2- 172.21.15.27:0/3776292606 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac9406ca90 0x7fac9414ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.117+0000 7fac9bc00640 1 -- 172.21.15.27:0/3776292606 >> 172.21.15.27:0/3776292606 conn(0x7fac940b0850 msgr2=0x7fac94145930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:39.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.117+0000 7fac9bc00640 1 -- 172.21.15.27:0/3776292606 shutdown_connections 2024-01-15T18:18:39.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.117+0000 7fac9bc00640 1 -- 172.21.15.27:0/3776292606 wait complete. 2024-01-15T18:18:39.135 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:18:39.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.281+0000 7f3abb400640 1 Processor -- start 2024-01-15T18:18:39.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.281+0000 7f3abb400640 1 -- start start 2024-01-15T18:18:39.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.281+0000 7f3abb400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ab4159bf0 0x7f3ab4159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.281+0000 7f3abb400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3ab4071e40 0x7f3ab40c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.281+0000 7f3abb400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab4159a50 con 0x7f3ab406d430 2024-01-15T18:18:39.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.281+0000 7f3abb400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ab40cce00 con 0x7f3ab4071e40 2024-01-15T18:18:39.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.281+0000 7f3abb400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab415a510 con 0x7f3ab4159bf0 2024-01-15T18:18:39.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.281+0000 7f3ab9400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3ab406d430 0x7f3ab415d150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38706/0 (socket says 172.21.15.27:38706) 2024-01-15T18:18:39.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.281+0000 7f3ab9400640 1 -- 172.21.15.27:0/660087687 learned_addr learned my addr 172.21.15.27:0/660087687 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:39.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.281+0000 7f3ab3e00640 1 --2- 172.21.15.27:0/660087687 >> v2:172.21.15.27:3300/0 conn(0x7f3ab4071e40 0x7f3ab40c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.281+0000 7f3ab8a00640 1 --2- 172.21.15.27:0/660087687 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ab4159bf0 0x7f3ab4159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.285+0000 7f3ab3400640 1 -- 172.21.15.27:0/660087687 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3057769331 0 0) 0x7f3aac0014a0 con 0x7f3ab406d430 2024-01-15T18:18:39.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.285+0000 7f3ab3400640 1 -- 172.21.15.27:0/660087687 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a94003450 con 0x7f3ab406d430 2024-01-15T18:18:39.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.285+0000 7f3ab3e00640 1 -- 172.21.15.27:0/660087687 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ab4159bf0 msgr2=0x7f3ab4159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:39.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.285+0000 7f3ab3e00640 1 --2- 172.21.15.27:0/660087687 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ab4159bf0 0x7f3ab4159fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.285+0000 7f3ab3e00640 1 -- 172.21.15.27:0/660087687 >> v1:172.21.15.27:6789/0 conn(0x7f3ab406d430 legacy=0x7f3ab415d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:39.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.285+0000 7f3ab3e00640 1 -- 172.21.15.27:0/660087687 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab406d980 con 0x7f3ab4071e40 2024-01-15T18:18:39.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.285+0000 7f3ab8a00640 1 --2- 172.21.15.27:0/660087687 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ab4159bf0 0x7f3ab4159fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:18:39.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.285+0000 7f3ab3e00640 1 --2- 172.21.15.27:0/660087687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ab4071e40 0x7f3ab40c9be0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c0097c0 tx=0x7f3a9c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe3d54265fcddd8d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:39.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.285+0000 7f3ab3400640 1 -- 172.21.15.27:0/660087687 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c0a8970 con 0x7f3ab4071e40 2024-01-15T18:18:39.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.285+0000 7f3ab3400640 1 -- 172.21.15.27:0/660087687 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a9c0a8ad0 con 0x7f3ab4071e40 2024-01-15T18:18:39.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.285+0000 7f3ab3400640 1 -- 172.21.15.27:0/660087687 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c0a8e40 con 0x7f3ab4071e40 2024-01-15T18:18:39.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.285+0000 7f3abb400640 1 -- 172.21.15.27:0/660087687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ab4071e40 msgr2=0x7f3ab40c9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:39.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.285+0000 7f3abb400640 1 --2- 172.21.15.27:0/660087687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ab4071e40 0x7f3ab40c9be0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c0097c0 tx=0x7f3a9c01ec70 comp rx=0 tx=0).stop 2024-01-15T18:18:39.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.289+0000 7f3abb400640 1 -- 172.21.15.27:0/660087687 shutdown_connections 2024-01-15T18:18:39.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.289+0000 7f3abb400640 1 --2- 172.21.15.27:0/660087687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ab4071e40 0x7f3ab40c9be0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.289+0000 7f3abb400640 1 --2- 172.21.15.27:0/660087687 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ab4159bf0 0x7f3ab4159fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.289+0000 7f3abb400640 1 -- 172.21.15.27:0/660087687 >> 172.21.15.27:0/660087687 conn(0x7f3ab40cec60 msgr2=0x7f3ab40cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:39.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.289+0000 7f3abb400640 1 -- 172.21.15.27:0/660087687 shutdown_connections 2024-01-15T18:18:39.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.289+0000 7f3abb400640 1 -- 172.21.15.27:0/660087687 wait complete. 2024-01-15T18:18:39.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.289+0000 7f3abb400640 1 Processor -- start 2024-01-15T18:18:39.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.289+0000 7f3abb400640 1 -- start start 2024-01-15T18:18:39.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.289+0000 7f3abb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ab406d430 0x7f3ab4157500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.289+0000 7f3abb400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ab4159bf0 0x7f3ab40b1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.289+0000 7f3abb400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ab40b2340 0x7f3ab415b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.289+0000 7f3abb400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab40b1f90 con 0x7f3ab406d430 2024-01-15T18:18:39.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.289+0000 7f3abb400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab40c67e0 con 0x7f3ab40b2340 2024-01-15T18:18:39.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.289+0000 7f3abb400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab40c6ac0 con 0x7f3ab4159bf0 2024-01-15T18:18:39.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.289+0000 7f3ab8a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ab406d430 0x7f3ab4157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.289+0000 7f3ab8a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ab406d430 0x7f3ab4157500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38252/0 (socket says 172.21.15.27:38252) 2024-01-15T18:18:39.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.289+0000 7f3ab8a00640 1 -- 172.21.15.27:0/1791290706 learned_addr learned my addr 172.21.15.27:0/1791290706 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:39.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.289+0000 7f3ab3e00640 1 --2- 172.21.15.27:0/1791290706 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ab4159bf0 0x7f3ab40b1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.289+0000 7f3ab9400640 1 --2- 172.21.15.27:0/1791290706 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ab40b2340 0x7f3ab415b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.289+0000 7f3ab8a00640 1 -- 172.21.15.27:0/1791290706 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ab4159bf0 msgr2=0x7f3ab40b1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:39.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.289+0000 7f3ab8a00640 1 --2- 172.21.15.27:0/1791290706 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ab4159bf0 0x7f3ab40b1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.289+0000 7f3ab8a00640 1 -- 172.21.15.27:0/1791290706 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ab40b2340 msgr2=0x7f3ab415b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:39.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.289+0000 7f3ab8a00640 1 --2- 172.21.15.27:0/1791290706 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ab40b2340 0x7f3ab415b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.289+0000 7f3ab8a00640 1 -- 172.21.15.27:0/1791290706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa4009420 con 0x7f3ab406d430 2024-01-15T18:18:39.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.293+0000 7f3ab8a00640 1 --2- 172.21.15.27:0/1791290706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ab406d430 0x7f3ab4157500 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f3aa400e980 tx=0x7f3aa400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:39.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.293+0000 7f3ab0c00640 1 -- 172.21.15.27:0/1791290706 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa400ca00 con 0x7f3ab406d430 2024-01-15T18:18:39.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.293+0000 7f3ab0c00640 1 -- 172.21.15.27:0/1791290706 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa400cb60 con 0x7f3ab406d430 2024-01-15T18:18:39.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.293+0000 7f3abb400640 1 -- 172.21.15.27:0/1791290706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a9c009420 con 0x7f3ab406d430 2024-01-15T18:18:39.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.293+0000 7f3ab0c00640 1 -- 172.21.15.27:0/1791290706 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa4013470 con 0x7f3ab406d430 2024-01-15T18:18:39.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.293+0000 7f3abb400640 1 -- 172.21.15.27:0/1791290706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ab40b27d0 con 0x7f3ab406d430 2024-01-15T18:18:39.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.293+0000 7f3abb400640 1 -- 172.21.15.27:0/1791290706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ab404e850 con 0x7f3ab406d430 2024-01-15T18:18:39.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.293+0000 7f3ab0c00640 1 -- 172.21.15.27:0/1791290706 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f3aa40135d0 con 0x7f3ab406d430 2024-01-15T18:18:39.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.297+0000 7f3ab0c00640 1 --2- 172.21.15.27:0/1791290706 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a8c067970 0x7f3a8c069e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.297+0000 7f3ab0c00640 1 -- 172.21.15.27:0/1791290706 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f3aa4083120 con 0x7f3ab406d430 2024-01-15T18:18:39.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.297+0000 7f3ab3e00640 1 --2- 172.21.15.27:0/1791290706 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a8c067970 0x7f3a8c069e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.297+0000 7f3ab3e00640 1 --2- 172.21.15.27:0/1791290706 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a8c067970 0x7f3a8c069e30 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c0b2000 tx=0x7f3a9c0aa070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:39.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.301+0000 7f3ab0c00640 1 -- 172.21.15.27:0/1791290706 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3aa4051e40 con 0x7f3ab406d430 2024-01-15T18:18:39.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.489+0000 7f3abb400640 1 -- 172.21.15.27:0/1791290706 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3ab40c7210 con 0x7f3ab406d430 2024-01-15T18:18:39.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.489+0000 7f3ab0c00640 1 -- 172.21.15.27:0/1791290706 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v77) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3aa4017070 con 0x7f3ab406d430 2024-01-15T18:18:39.491 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:18:39.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.493+0000 7f3abb400640 1 -- 172.21.15.27:0/1791290706 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a8c067970 msgr2=0x7f3a8c069e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:39.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.493+0000 7f3abb400640 1 --2- 172.21.15.27:0/1791290706 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a8c067970 0x7f3a8c069e30 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c0b2000 tx=0x7f3a9c0aa070 comp rx=0 tx=0).stop 2024-01-15T18:18:39.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.493+0000 7f3abb400640 1 -- 172.21.15.27:0/1791290706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ab406d430 msgr2=0x7f3ab4157500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:39.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.493+0000 7f3abb400640 1 --2- 172.21.15.27:0/1791290706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ab406d430 0x7f3ab4157500 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f3aa400e980 tx=0x7f3aa400ee40 comp rx=0 tx=0).stop 2024-01-15T18:18:39.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.493+0000 7f3abb400640 1 -- 172.21.15.27:0/1791290706 shutdown_connections 2024-01-15T18:18:39.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.493+0000 7f3abb400640 1 --2- 172.21.15.27:0/1791290706 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a8c067970 0x7f3a8c069e30 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.493+0000 7f3abb400640 1 --2- 172.21.15.27:0/1791290706 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3ab40b2340 0x7f3ab415b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.493+0000 7f3abb400640 1 --2- 172.21.15.27:0/1791290706 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3ab4159bf0 0x7f3ab40b1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.493+0000 7f3abb400640 1 --2- 172.21.15.27:0/1791290706 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ab406d430 0x7f3ab4157500 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.493+0000 7f3abb400640 1 -- 172.21.15.27:0/1791290706 >> 172.21.15.27:0/1791290706 conn(0x7f3ab40cec60 msgr2=0x7f3ab4159120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:39.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.493+0000 7f3abb400640 1 -- 172.21.15.27:0/1791290706 shutdown_connections 2024-01-15T18:18:39.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:39.493+0000 7f3abb400640 1 -- 172.21.15.27:0/1791290706 wait complete. 2024-01-15T18:18:39.512 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:18:39.512 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:18:39.512 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-01-15T18:18:39.512 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-15T18:18:39.512 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-15T18:18:39.512 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:18:39.512 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:18:39.512 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:18:39.512 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:18:39.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:39.513 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-01-15T18:18:39.523 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:18:39.523 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:18:39.523 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:18:39.523 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:18:39.523 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-01-15T18:18:39.523 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:18:39.523 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:18:39.523 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:18:39.523 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-01-15T18:18:39.523 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:18:39.523 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:18:39.524 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-01-15T18:18:39.524 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:18:39.524 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-01-15T18:18:39.524 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-01-15T18:18:39.524 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-01-15T18:18:39.524 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 41640sec preferred_lft 41640sec 2024-01-15T18:18:39.524 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2024-01-15T18:18:39.524 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:18:39.524 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:18:39.525 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-01-15T18:18:39.525 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-01-15T18:18:39.525 INFO:teuthology.orchestra.run.smithi027.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-01-15T18:18:39.525 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 02:42:65:20:14:72 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:18:39.525 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-01-15T18:18:39.525 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:18:39.525 INFO:teuthology.orchestra.run.smithi027.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:18:39.525 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-01-15T18:18:39.525 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:18:39.525 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:18:39.525 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-01-15T18:18:39.526 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:18:39.526 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:18:39.526 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:18:39.526 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-01-15T18:18:39.526 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:18:39.526 INFO:teuthology.orchestra.run.smithi027.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:18:39.526 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:18:39.526 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-01-15T18:18:39.526 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:18:39.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:39.527 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:18:39.589 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-01-15T18:18:39.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:39.591 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-01-15T18:18:39.644 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-01-15T18:18:39.645 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:18:39.645 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-01-15T18:18:39.645 INFO:teuthology.orchestra.run.smithi027.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-01-15T18:18:39.645 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-01-15T18:18:39.645 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:18:39.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:39.645 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2024-01-15T18:18:39.645 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-01-15T18:18:39.646 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-15T18:18:39.646 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-15T18:18:39.646 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-15T18:18:39.646 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-01-15T18:18:39.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:39.736 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-15T18:18:39.746 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-15T18:18:39.746 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-15T18:18:39.746 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:18:39.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:39.747 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-01-15T18:18:39.807 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-15T18:18:39.807 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:18:39.807 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-15T18:18:39.859 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:18:39.859 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:18:39.860 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:18:39.911 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-01-15T18:18:39.976 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:18:40.029 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-01-15T18:18:40.029 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-01-15T18:18:40.029 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-01-15T18:18:40.029 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-01-15T18:18:40.029 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-01-15T18:18:40.029 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-01-15T18:18:40.029 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-01-15T18:18:40.029 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-01-15T18:18:40.029 INFO:teuthology.orchestra.run.smithi027.stdout:/run/lock 2024-01-15T18:18:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-01-15T18:18:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-01-15T18:18:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-01-15T18:18:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:18:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-01-15T18:18:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-01-15T18:18:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-01-15T18:18:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-01-15T18:18:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/fs/nfsd 2024-01-15T18:18:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-01-15T18:18:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-01-15T18:18:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:18:40.031 INFO:teuthology.orchestra.run.smithi027.stdout:/run/rpc_pipefs 2024-01-15T18:18:40.031 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-01-15T18:18:40.031 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-01-15T18:18:40.031 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:18:40.031 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:18:40.031 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:18:40.031 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:18:40.031 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-01-15T18:18:40.031 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:18:40.031 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:18:40.031 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:18:40.032 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:18:40.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:40.033 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-01-15T18:18:40.082 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:18:40.083 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-15T18:18:40.130 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-01-15T18:18:40.150 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:18:40.209 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-01-15T18:18:40.209 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-01-15T18:18:40.210 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-01-15T18:18:40.210 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-01-15T18:18:40.210 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-01-15T18:18:40.210 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-01-15T18:18:40.210 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-01-15T18:18:40.210 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-01-15T18:18:40.210 INFO:teuthology.orchestra.run.smithi027.stdout:/run/lock 2024-01-15T18:18:40.210 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-01-15T18:18:40.210 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-01-15T18:18:40.210 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-01-15T18:18:40.211 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:18:40.211 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-01-15T18:18:40.211 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-01-15T18:18:40.211 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-01-15T18:18:40.211 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-01-15T18:18:40.211 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/fs/nfsd 2024-01-15T18:18:40.211 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-01-15T18:18:40.211 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-01-15T18:18:40.211 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:18:40.211 INFO:teuthology.orchestra.run.smithi027.stdout:/run/rpc_pipefs 2024-01-15T18:18:40.211 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-01-15T18:18:40.211 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-01-15T18:18:40.212 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:18:40.212 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:18:40.212 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:18:40.212 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:18:40.212 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-01-15T18:18:40.212 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:18:40.212 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:18:40.212 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:18:40.212 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:18:40.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:40.214 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-01-15T18:18:40.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:2024-01-15T18:18:40.229+0000 7f6bf1016580 -1 init, newargv = 0x559c0f563820 newargc=15 2024-01-15T18:18:40.230 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[52270]: starting ceph client 2024-01-15T18:18:40.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[52270]: starting fuse 2024-01-15T18:18:40.240 INFO:teuthology.orchestra.run.smithi027.stdout:43 2024-01-15T18:18:40.241 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-01-15T18:18:40.241 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:18:40.241 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-01-15T18:18:40.241 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-01-15T18:18:40.241 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-01-15T18:18:40.241 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-01-15T18:18:40.241 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:18:40.241 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-01-15T18:18:40.241 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-15T18:18:40.242 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-01-15T18:18:40.242 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-15T18:18:40.242 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:18:40.242 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-01-15T18:18:40.242 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-01-15T18:18:40.242 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-01-15T18:18:40.242 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-01-15T18:18:40.242 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:18:40.242 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-01-15T18:18:40.242 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:18:40.242 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:18:40.243 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:18:40.243 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-01-15T18:18:40.243 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-01-15T18:18:40.243 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-01-15T18:18:40.243 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:18:40.243 DEBUG:teuthology.orchestra.run.smithi027:> 2024-01-15T18:18:40.243 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.0")) 2024-01-15T18:18:40.243 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-01-15T18:18:40.365 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.0.52270.asok 2024-01-15T18:18:40.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:18:40.376 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.52270.asok 2024-01-15T18:18:40.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:40.377 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.52270.asok status 2024-01-15T18:18:40.550 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:18:40.550 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-01-15T18:18:40.550 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:18:40.550 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:18:40.550 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "0", 2024-01-15T18:18:40.550 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-01-15T18:18:40.551 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-15T18:18:40.551 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "52270", 2024-01-15T18:18:40.551 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-01-15T18:18:40.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:40.551 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-01-15T18:18:40.551 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-01-15T18:18:40.551 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 9603, 2024-01-15T18:18:40.551 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-01-15T18:18:40.551 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-01-15T18:18:40.551 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-01-15T18:18:40.551 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 9603 2024-01-15T18:18:40.551 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:40.551 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-01-15T18:18:40.552 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-01-15T18:18:40.552 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:18:40.552 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1816163441 2024-01-15T18:18:40.552 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:40.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:40.552 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-01-15T18:18:40.552 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-01-15T18:18:40.552 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:18:40.552 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1816163441 2024-01-15T18:18:40.552 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:18:40.552 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.9603 192.168.0.1:0/1816163441", 2024-01-15T18:18:40.552 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.1:0/1816163441", 2024-01-15T18:18:40.552 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-01-15T18:18:40.553 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 77, 2024-01-15T18:18:40.553 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 108, 2024-01-15T18:18:40.553 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:18:40.553 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-01-15T18:18:40.553 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-01-15T18:18:40.553 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:18:40.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:40.566 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-01-15T18:18:40.566 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-15T18:18:40.575 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-01-15T18:18:40.575 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-15T18:18:40.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:40.576 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:18:40.643 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:18:40.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.829+0000 7fbdf8200640 1 Processor -- start 2024-01-15T18:18:40.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.829+0000 7fbdf8200640 1 -- start start 2024-01-15T18:18:40.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.829+0000 7fbdf8200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fbdf0160af0 0x7fbdf006b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:40.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.829+0000 7fbdf8200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbdf00cfaa0 0x7fbdf006bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:40.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.829+0000 7fbdf8200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbdf006c080 0x7fbdf0161210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:40.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.829+0000 7fbdf8200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf00ae280 con 0x7fbdf00cfaa0 2024-01-15T18:18:40.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.829+0000 7fbdf8200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf00adc50 con 0x7fbdf0160af0 2024-01-15T18:18:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.829+0000 7fbdf8200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf00cff80 con 0x7fbdf006c080 2024-01-15T18:18:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.829+0000 7fbdf4e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbdf00cfaa0 0x7fbdf006bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.829+0000 7fbdf4e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbdf00cfaa0 0x7fbdf006bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38260/0 (socket says 172.21.15.27:38260) 2024-01-15T18:18:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.829+0000 7fbdf4e00640 1 -- 172.21.15.27:0/966037530 learned_addr learned my addr 172.21.15.27:0/966037530 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.829+0000 7fbdf6200640 1 --2- 172.21.15.27:0/966037530 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbdf006c080 0x7fbdf0161210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.829+0000 7fbdf5800640 1 --2- 172.21.15.27:0/966037530 >> v2:172.21.15.93:3300/0 conn(0x7fbdf0160af0 0x7fbdf006b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.829+0000 7fbdf4e00640 1 -- 172.21.15.27:0/966037530 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbdf006c080 msgr2=0x7fbdf0161210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.829+0000 7fbdf4e00640 1 --2- 172.21.15.27:0/966037530 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbdf006c080 0x7fbdf0161210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.829+0000 7fbdf4e00640 1 -- 172.21.15.27:0/966037530 >> v2:172.21.15.93:3300/0 conn(0x7fbdf0160af0 msgr2=0x7fbdf006b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.829+0000 7fbdf4e00640 1 --2- 172.21.15.27:0/966037530 >> v2:172.21.15.93:3300/0 conn(0x7fbdf0160af0 0x7fbdf006b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:40.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.829+0000 7fbdf4e00640 1 -- 172.21.15.27:0/966037530 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf00d0260 con 0x7fbdf00cfaa0 2024-01-15T18:18:40.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.833+0000 7fbdf4e00640 1 --2- 172.21.15.27:0/966037530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf00cfaa0 0x7fbdf006bb40 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0071650 tx=0x7fbdd801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfb917fbe2ca58d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:40.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.833+0000 7fbdefe00640 1 -- 172.21.15.27:0/966037530 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd80ab600 con 0x7fbdf00cfaa0 2024-01-15T18:18:40.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.833+0000 7fbdefe00640 1 -- 172.21.15.27:0/966037530 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdd80a7050 con 0x7fbdf00cfaa0 2024-01-15T18:18:40.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.833+0000 7fbdefe00640 1 -- 172.21.15.27:0/966037530 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd80ab980 con 0x7fbdf00cfaa0 2024-01-15T18:18:40.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.833+0000 7fbdf8200640 1 -- 172.21.15.27:0/966037530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf00cfaa0 msgr2=0x7fbdf006bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:40.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.833+0000 7fbdf8200640 1 --2- 172.21.15.27:0/966037530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf00cfaa0 0x7fbdf006bb40 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0071650 tx=0x7fbdd801c500 comp rx=0 tx=0).stop 2024-01-15T18:18:40.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.833+0000 7fbdf8200640 1 -- 172.21.15.27:0/966037530 shutdown_connections 2024-01-15T18:18:40.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.833+0000 7fbdf8200640 1 --2- 172.21.15.27:0/966037530 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbdf006c080 0x7fbdf0161210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:40.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.833+0000 7fbdf8200640 1 --2- 172.21.15.27:0/966037530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf00cfaa0 0x7fbdf006bb40 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:40.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.833+0000 7fbdf8200640 1 --2- 172.21.15.27:0/966037530 >> v2:172.21.15.93:3300/0 conn(0x7fbdf0160af0 0x7fbdf006b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:40.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.833+0000 7fbdf8200640 1 -- 172.21.15.27:0/966037530 >> 172.21.15.27:0/966037530 conn(0x7fbdf00d0f20 msgr2=0x7fbdf00d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:40.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.833+0000 7fbdf8200640 1 -- 172.21.15.27:0/966037530 shutdown_connections 2024-01-15T18:18:40.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.833+0000 7fbdf8200640 1 -- 172.21.15.27:0/966037530 wait complete. 2024-01-15T18:18:40.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.837+0000 7fbdf8200640 1 Processor -- start 2024-01-15T18:18:40.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.837+0000 7fbdf8200640 1 -- start start 2024-01-15T18:18:40.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.837+0000 7fbdf8200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf006c080 0x7fbdf0159510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:40.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.837+0000 7fbdf8200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbdf00cfaa0 0x7fbdf009d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:40.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.837+0000 7fbdf8200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbdf0160af0 0x7fbdf009b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:40.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.837+0000 7fbdf8200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf015af90 con 0x7fbdf006c080 2024-01-15T18:18:40.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.837+0000 7fbdf8200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf0159a50 con 0x7fbdf00cfaa0 2024-01-15T18:18:40.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.837+0000 7fbdf8200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf009d680 con 0x7fbdf0160af0 2024-01-15T18:18:40.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.837+0000 7fbdf5800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf006c080 0x7fbdf0159510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:40.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.837+0000 7fbdf6200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbdf0160af0 0x7fbdf009b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:40.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.837+0000 7fbdf5800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf006c080 0x7fbdf0159510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38272/0 (socket says 172.21.15.27:38272) 2024-01-15T18:18:40.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.837+0000 7fbdf5800640 1 -- 172.21.15.27:0/1758589010 learned_addr learned my addr 172.21.15.27:0/1758589010 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:40.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.837+0000 7fbdf4e00640 1 --2- 172.21.15.27:0/1758589010 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbdf00cfaa0 0x7fbdf009d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:40.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.837+0000 7fbdf5800640 1 -- 172.21.15.27:0/1758589010 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbdf0160af0 msgr2=0x7fbdf009b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:40.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.837+0000 7fbdf5800640 1 --2- 172.21.15.27:0/1758589010 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbdf0160af0 0x7fbdf009b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:40.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.837+0000 7fbdf5800640 1 -- 172.21.15.27:0/1758589010 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbdf00cfaa0 msgr2=0x7fbdf009d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:40.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.837+0000 7fbdf5800640 1 --2- 172.21.15.27:0/1758589010 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbdf00cfaa0 0x7fbdf009d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:40.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.837+0000 7fbdf5800640 1 -- 172.21.15.27:0/1758589010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd80094d0 con 0x7fbdf006c080 2024-01-15T18:18:40.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.837+0000 7fbdf4e00640 1 --2- 172.21.15.27:0/1758589010 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbdf00cfaa0 0x7fbdf009d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:18:40.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.837+0000 7fbdf5800640 1 --2- 172.21.15.27:0/1758589010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf006c080 0x7fbdf0159510 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fbde000d5f0 tx=0x7fbde000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:40.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.837+0000 7fbded600640 1 -- 172.21.15.27:0/1758589010 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde00125f0 con 0x7fbdf006c080 2024-01-15T18:18:40.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.837+0000 7fbded600640 1 -- 172.21.15.27:0/1758589010 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde0012750 con 0x7fbdf006c080 2024-01-15T18:18:40.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.837+0000 7fbded600640 1 -- 172.21.15.27:0/1758589010 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde0012ae0 con 0x7fbdf006c080 2024-01-15T18:18:40.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.837+0000 7fbdf8200640 1 -- 172.21.15.27:0/1758589010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf009d940 con 0x7fbdf006c080 2024-01-15T18:18:40.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.837+0000 7fbdf8200640 1 -- 172.21.15.27:0/1758589010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf009df00 con 0x7fbdf006c080 2024-01-15T18:18:40.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.841+0000 7fbdf8200640 1 -- 172.21.15.27:0/1758589010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf009e830 con 0x7fbdf006c080 2024-01-15T18:18:40.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.841+0000 7fbded600640 1 -- 172.21.15.27:0/1758589010 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7fbde0012c40 con 0x7fbdf006c080 2024-01-15T18:18:40.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.841+0000 7fbded600640 1 --2- 172.21.15.27:0/1758589010 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbdc8067970 0x7fbdc8069e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:40.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.841+0000 7fbded600640 1 -- 172.21.15.27:0/1758589010 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fbde0083300 con 0x7fbdf006c080 2024-01-15T18:18:40.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.841+0000 7fbdf4e00640 1 --2- 172.21.15.27:0/1758589010 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbdc8067970 0x7fbdc8069e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:40.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.841+0000 7fbdf4e00640 1 --2- 172.21.15.27:0/1758589010 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbdc8067970 0x7fbdc8069e30 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fbdd80ad000 tx=0x7fbdd8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:40.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:40.845+0000 7fbded600640 1 -- 172.21.15.27:0/1758589010 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbdf009e830 con 0x7fbdf006c080 2024-01-15T18:18:41.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:41.037+0000 7fbdf8200640 1 -- 172.21.15.27:0/1758589010 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fbdf009e830 con 0x7fbdf006c080 2024-01-15T18:18:41.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:41.037+0000 7fbded600640 1 -- 172.21.15.27:0/1758589010 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v77) v1 ==== 53+0+73 (secure 0 0 0) 0x7fbdf009e830 con 0x7fbdf006c080 2024-01-15T18:18:41.037 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:18:41.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:41.037+0000 7fbdf8200640 1 -- 172.21.15.27:0/1758589010 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbdc8067970 msgr2=0x7fbdc8069e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:41.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:41.037+0000 7fbdf8200640 1 --2- 172.21.15.27:0/1758589010 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbdc8067970 0x7fbdc8069e30 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fbdd80ad000 tx=0x7fbdd8008040 comp rx=0 tx=0).stop 2024-01-15T18:18:41.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:41.037+0000 7fbdf8200640 1 -- 172.21.15.27:0/1758589010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf006c080 msgr2=0x7fbdf0159510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:41.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:41.037+0000 7fbdf8200640 1 --2- 172.21.15.27:0/1758589010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf006c080 0x7fbdf0159510 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7fbde000d5f0 tx=0x7fbde000dab0 comp rx=0 tx=0).stop 2024-01-15T18:18:41.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:41.041+0000 7fbdf8200640 1 -- 172.21.15.27:0/1758589010 shutdown_connections 2024-01-15T18:18:41.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:41.041+0000 7fbdf8200640 1 --2- 172.21.15.27:0/1758589010 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbdc8067970 0x7fbdc8069e30 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:41.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:41.041+0000 7fbdf8200640 1 --2- 172.21.15.27:0/1758589010 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbdf0160af0 0x7fbdf009b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:41.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:41.041+0000 7fbdf8200640 1 --2- 172.21.15.27:0/1758589010 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbdf00cfaa0 0x7fbdf009d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:41.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:41.041+0000 7fbdf8200640 1 --2- 172.21.15.27:0/1758589010 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf006c080 0x7fbdf0159510 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:41.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:41.041+0000 7fbdf8200640 1 -- 172.21.15.27:0/1758589010 >> 172.21.15.27:0/1758589010 conn(0x7fbdf00d0f20 msgr2=0x7fbdf006b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:41.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:41.041+0000 7fbdf8200640 1 -- 172.21.15.27:0/1758589010 shutdown_connections 2024-01-15T18:18:41.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:41.041+0000 7fbdf8200640 1 -- 172.21.15.27:0/1758589010 wait complete. 2024-01-15T18:18:41.059 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:18:41.059 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:18:41.059 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi093.front.sepia.ceph.com 2024-01-15T18:18:41.059 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-15T18:18:41.059 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-15T18:18:41.059 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:18:41.059 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:18:41.060 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:18:41.060 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:18:41.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:41.060 DEBUG:teuthology.orchestra.run.smithi093:> ip addr 2024-01-15T18:18:41.071 INFO:teuthology.orchestra.run.smithi093.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:18:41.071 INFO:teuthology.orchestra.run.smithi093.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:18:41.071 INFO:teuthology.orchestra.run.smithi093.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:18:41.071 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:18:41.071 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 ::1/128 scope host 2024-01-15T18:18:41.071 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:18:41.071 INFO:teuthology.orchestra.run.smithi093.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:18:41.071 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:d8:da:1c brd ff:ff:ff:ff:ff:ff 2024-01-15T18:18:41.072 INFO:teuthology.orchestra.run.smithi093.stdout: altname enp7s0f0 2024-01-15T18:18:41.072 INFO:teuthology.orchestra.run.smithi093.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:18:41.072 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:d8:da:1d brd ff:ff:ff:ff:ff:ff 2024-01-15T18:18:41.072 INFO:teuthology.orchestra.run.smithi093.stdout: altname enp7s0f1 2024-01-15T18:18:41.072 INFO:teuthology.orchestra.run.smithi093.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-01-15T18:18:41.072 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:8f:d6:34 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:18:41.072 INFO:teuthology.orchestra.run.smithi093.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:18:41.072 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0c:c4:7a:8f:d6:35 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:18:41.072 INFO:teuthology.orchestra.run.smithi093.stdout: inet 172.21.15.93/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-01-15T18:18:41.072 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft 41618sec preferred_lft 41618sec 2024-01-15T18:18:41.072 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::ec4:7aff:fe8f:d635/64 scope link 2024-01-15T18:18:41.072 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:18:41.072 INFO:teuthology.orchestra.run.smithi093.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-01-15T18:18:41.073 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 02:42:2a:61:fc:7d brd ff:ff:ff:ff:ff:ff 2024-01-15T18:18:41.073 INFO:teuthology.orchestra.run.smithi093.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-01-15T18:18:41.073 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:18:41.073 INFO:teuthology.orchestra.run.smithi093.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:18:41.073 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-01-15T18:18:41.073 INFO:teuthology.orchestra.run.smithi093.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:18:41.073 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:18:41.073 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-01-15T18:18:41.073 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:18:41.073 INFO:teuthology.orchestra.run.smithi093.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:18:41.073 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:18:41.073 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-01-15T18:18:41.073 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:18:41.073 INFO:teuthology.orchestra.run.smithi093.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:18:41.073 INFO:teuthology.orchestra.run.smithi093.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:18:41.074 INFO:teuthology.orchestra.run.smithi093.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-01-15T18:18:41.074 INFO:teuthology.orchestra.run.smithi093.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:18:41.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:41.074 DEBUG:teuthology.orchestra.run.smithi093:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:18:41.135 INFO:teuthology.orchestra.run.smithi093.stdout:1 2024-01-15T18:18:41.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:41.138 DEBUG:teuthology.orchestra.run.smithi093:> route 2024-01-15T18:18:41.190 INFO:teuthology.orchestra.run.smithi093.stdout:Kernel IP routing table 2024-01-15T18:18:41.190 INFO:teuthology.orchestra.run.smithi093.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:18:41.190 INFO:teuthology.orchestra.run.smithi093.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-01-15T18:18:41.190 INFO:teuthology.orchestra.run.smithi093.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T18:18:41.190 INFO:teuthology.orchestra.run.smithi093.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-01-15T18:18:41.190 INFO:teuthology.orchestra.run.smithi093.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T18:18:41.190 INFO:teuthology.orchestra.run.smithi093.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:18:41.190 INFO:teuthology.orchestra.run.smithi093.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:18:41.190 INFO:teuthology.orchestra.run.smithi093.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-01-15T18:18:41.190 INFO:teuthology.orchestra.run.smithi093.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:18:41.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:41.191 DEBUG:teuthology.orchestra.run.smithi093:> (cd / && exec bash -c ' 2024-01-15T18:18:41.191 DEBUG:teuthology.orchestra.run.smithi093:> set -e 2024-01-15T18:18:41.191 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T18:18:41.191 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T18:18:41.191 DEBUG:teuthology.orchestra.run.smithi093:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T18:18:41.191 DEBUG:teuthology.orchestra.run.smithi093:> ') 2024-01-15T18:18:41.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:41.286 DEBUG:teuthology.orchestra.run.smithi093:> ip netns list 2024-01-15T18:18:41.295 INFO:teuthology.orchestra.run.smithi093.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-15T18:18:41.295 INFO:teuthology.orchestra.run.smithi093.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-15T18:18:41.296 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:18:41.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:41.296 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip link set brx.0 up 2024-01-15T18:18:41.363 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-15T18:18:41.363 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:18:41.363 DEBUG:teuthology.orchestra.run.smithi093:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-15T18:18:41.415 INFO:teuthology.orchestra.run.smithi093.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:18:41.416 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:18:41.416 DEBUG:teuthology.orchestra.run.smithi093:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:18:41.467 DEBUG:teuthology.orchestra.run.smithi093:> sudo modprobe fuse 2024-01-15T18:18:41.532 DEBUG:teuthology.orchestra.run.smithi093:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:18:41.585 INFO:teuthology.orchestra.run.smithi093.stdout:/sys 2024-01-15T18:18:41.585 INFO:teuthology.orchestra.run.smithi093.stdout:/proc 2024-01-15T18:18:41.585 INFO:teuthology.orchestra.run.smithi093.stdout:/dev 2024-01-15T18:18:41.585 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/pts 2024-01-15T18:18:41.585 INFO:teuthology.orchestra.run.smithi093.stdout:/run 2024-01-15T18:18:41.585 INFO:teuthology.orchestra.run.smithi093.stdout:/ 2024-01-15T18:18:41.585 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/security 2024-01-15T18:18:41.585 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/shm 2024-01-15T18:18:41.585 INFO:teuthology.orchestra.run.smithi093.stdout:/run/lock 2024-01-15T18:18:41.586 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/cgroup 2024-01-15T18:18:41.586 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/pstore 2024-01-15T18:18:41.586 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/bpf 2024-01-15T18:18:41.586 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:18:41.586 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/hugepages 2024-01-15T18:18:41.586 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/mqueue 2024-01-15T18:18:41.586 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/debug 2024-01-15T18:18:41.586 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/tracing 2024-01-15T18:18:41.586 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/fs/nfsd 2024-01-15T18:18:41.586 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/fuse/connections 2024-01-15T18:18:41.586 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/config 2024-01-15T18:18:41.587 INFO:teuthology.orchestra.run.smithi093.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:18:41.587 INFO:teuthology.orchestra.run.smithi093.stdout:/run/rpc_pipefs 2024-01-15T18:18:41.587 INFO:teuthology.orchestra.run.smithi093.stdout:/run/user/1000 2024-01-15T18:18:41.587 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph 2024-01-15T18:18:41.587 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:18:41.587 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:18:41.587 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:18:41.587 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:18:41.587 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns 2024-01-15T18:18:41.587 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:18:41.587 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:18:41.587 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:18:41.588 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:18:41.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:41.589 DEBUG:teuthology.orchestra.run.smithi093:> ls /sys/fs/fuse/connections 2024-01-15T18:18:41.639 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:18:41.639 DEBUG:teuthology.orchestra.run.smithi093:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-15T18:18:41.685 DEBUG:teuthology.orchestra.run.smithi093:> sudo modprobe fuse 2024-01-15T18:18:41.707 DEBUG:teuthology.orchestra.run.smithi093:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:18:41.760 INFO:teuthology.orchestra.run.smithi093.stdout:/sys 2024-01-15T18:18:41.760 INFO:teuthology.orchestra.run.smithi093.stdout:/proc 2024-01-15T18:18:41.761 INFO:teuthology.orchestra.run.smithi093.stdout:/dev 2024-01-15T18:18:41.761 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/pts 2024-01-15T18:18:41.761 INFO:teuthology.orchestra.run.smithi093.stdout:/run 2024-01-15T18:18:41.761 INFO:teuthology.orchestra.run.smithi093.stdout:/ 2024-01-15T18:18:41.761 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/security 2024-01-15T18:18:41.761 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/shm 2024-01-15T18:18:41.761 INFO:teuthology.orchestra.run.smithi093.stdout:/run/lock 2024-01-15T18:18:41.761 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/cgroup 2024-01-15T18:18:41.761 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/pstore 2024-01-15T18:18:41.761 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/bpf 2024-01-15T18:18:41.762 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:18:41.762 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/hugepages 2024-01-15T18:18:41.762 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/mqueue 2024-01-15T18:18:41.762 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/debug 2024-01-15T18:18:41.762 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/tracing 2024-01-15T18:18:41.762 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/fs/nfsd 2024-01-15T18:18:41.762 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/fuse/connections 2024-01-15T18:18:41.762 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/config 2024-01-15T18:18:41.762 INFO:teuthology.orchestra.run.smithi093.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:18:41.762 INFO:teuthology.orchestra.run.smithi093.stdout:/run/rpc_pipefs 2024-01-15T18:18:41.762 INFO:teuthology.orchestra.run.smithi093.stdout:/run/user/1000 2024-01-15T18:18:41.763 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph 2024-01-15T18:18:41.763 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:18:41.763 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:18:41.763 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:18:41.763 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:18:41.763 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns 2024-01-15T18:18:41.763 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:18:41.763 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:18:41.763 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:18:41.763 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:18:41.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:41.765 DEBUG:teuthology.orchestra.run.smithi093:> ls /sys/fs/fuse/connections 2024-01-15T18:18:41.783 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:2024-01-15T18:18:41.782+0000 7f278e36c580 -1 init, newargv = 0x55fdc44a3370 newargc=15 2024-01-15T18:18:41.783 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:ceph-fuse[34808]: starting ceph client 2024-01-15T18:18:41.793 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:ceph-fuse[34808]: starting fuse 2024-01-15T18:18:42.892 DEBUG:teuthology.orchestra.run.smithi093:> sudo modprobe fuse 2024-01-15T18:18:42.916 DEBUG:teuthology.orchestra.run.smithi093:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:18:42.970 INFO:teuthology.orchestra.run.smithi093.stdout:/sys 2024-01-15T18:18:42.970 INFO:teuthology.orchestra.run.smithi093.stdout:/proc 2024-01-15T18:18:42.970 INFO:teuthology.orchestra.run.smithi093.stdout:/dev 2024-01-15T18:18:42.970 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/pts 2024-01-15T18:18:42.970 INFO:teuthology.orchestra.run.smithi093.stdout:/run 2024-01-15T18:18:42.970 INFO:teuthology.orchestra.run.smithi093.stdout:/ 2024-01-15T18:18:42.970 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/security 2024-01-15T18:18:42.971 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/shm 2024-01-15T18:18:42.971 INFO:teuthology.orchestra.run.smithi093.stdout:/run/lock 2024-01-15T18:18:42.971 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/cgroup 2024-01-15T18:18:42.971 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/pstore 2024-01-15T18:18:42.971 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/bpf 2024-01-15T18:18:42.971 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:18:42.971 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/hugepages 2024-01-15T18:18:42.971 INFO:teuthology.orchestra.run.smithi093.stdout:/dev/mqueue 2024-01-15T18:18:42.971 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/debug 2024-01-15T18:18:42.971 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/tracing 2024-01-15T18:18:42.971 INFO:teuthology.orchestra.run.smithi093.stdout:/proc/fs/nfsd 2024-01-15T18:18:42.972 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/fs/fuse/connections 2024-01-15T18:18:42.972 INFO:teuthology.orchestra.run.smithi093.stdout:/sys/kernel/config 2024-01-15T18:18:42.972 INFO:teuthology.orchestra.run.smithi093.stdout:/run/credentials/systemd-sysusers.service 2024-01-15T18:18:42.972 INFO:teuthology.orchestra.run.smithi093.stdout:/run/rpc_pipefs 2024-01-15T18:18:42.972 INFO:teuthology.orchestra.run.smithi093.stdout:/run/user/1000 2024-01-15T18:18:42.972 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph 2024-01-15T18:18:42.972 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:18:42.972 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:18:42.973 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:18:42.973 INFO:teuthology.orchestra.run.smithi093.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:18:42.973 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns 2024-01-15T18:18:42.973 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:18:42.973 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:18:42.973 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:18:42.973 INFO:teuthology.orchestra.run.smithi093.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:18:42.973 INFO:teuthology.orchestra.run.smithi093.stdout:/home/ubuntu/cephtest/mnt.1 2024-01-15T18:18:42.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:42.975 DEBUG:teuthology.orchestra.run.smithi093:> ls /sys/fs/fuse/connections 2024-01-15T18:18:43.023 INFO:teuthology.orchestra.run.smithi093.stdout:43 2024-01-15T18:18:43.024 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-01-15T18:18:43.024 DEBUG:teuthology.orchestra.run.smithi093:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:18:43.024 DEBUG:teuthology.orchestra.run.smithi093:> import glob 2024-01-15T18:18:43.024 DEBUG:teuthology.orchestra.run.smithi093:> import re 2024-01-15T18:18:43.024 DEBUG:teuthology.orchestra.run.smithi093:> import os 2024-01-15T18:18:43.024 DEBUG:teuthology.orchestra.run.smithi093:> import subprocess 2024-01-15T18:18:43.025 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:18:43.025 DEBUG:teuthology.orchestra.run.smithi093:> def _find_admin_socket(client_name): 2024-01-15T18:18:43.025 DEBUG:teuthology.orchestra.run.smithi093:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-15T18:18:43.025 DEBUG:teuthology.orchestra.run.smithi093:> files = glob.glob(asok_path) 2024-01-15T18:18:43.025 DEBUG:teuthology.orchestra.run.smithi093:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-15T18:18:43.025 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:18:43.025 DEBUG:teuthology.orchestra.run.smithi093:> # Given a non-glob path, it better be there 2024-01-15T18:18:43.025 DEBUG:teuthology.orchestra.run.smithi093:> if "*" not in asok_path: 2024-01-15T18:18:43.025 DEBUG:teuthology.orchestra.run.smithi093:> assert(len(files) == 1) 2024-01-15T18:18:43.025 DEBUG:teuthology.orchestra.run.smithi093:> return files[0] 2024-01-15T18:18:43.025 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:18:43.025 DEBUG:teuthology.orchestra.run.smithi093:> for f in files: 2024-01-15T18:18:43.026 DEBUG:teuthology.orchestra.run.smithi093:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:18:43.026 DEBUG:teuthology.orchestra.run.smithi093:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:18:43.026 DEBUG:teuthology.orchestra.run.smithi093:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:18:43.026 DEBUG:teuthology.orchestra.run.smithi093:> contents = proc_f.read() 2024-01-15T18:18:43.026 DEBUG:teuthology.orchestra.run.smithi093:> if mountpoint in contents: 2024-01-15T18:18:43.026 DEBUG:teuthology.orchestra.run.smithi093:> return f 2024-01-15T18:18:43.026 DEBUG:teuthology.orchestra.run.smithi093:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:18:43.026 DEBUG:teuthology.orchestra.run.smithi093:> 2024-01-15T18:18:43.026 DEBUG:teuthology.orchestra.run.smithi093:> print(_find_admin_socket("client.1")) 2024-01-15T18:18:43.026 DEBUG:teuthology.orchestra.run.smithi093:> ' 2024-01-15T18:18:43.156 INFO:teuthology.orchestra.run.smithi093.stdout:/var/run/ceph/ceph-client.1.34808.asok 2024-01-15T18:18:43.160 INFO:teuthology.orchestra.run.smithi093.stderr:2024-01-15T18:18:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:18:43.169 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.34808.asok 2024-01-15T18:18:43.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:43.169 DEBUG:teuthology.orchestra.run.smithi093:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.34808.asok status 2024-01-15T18:18:43.342 INFO:teuthology.orchestra.run.smithi093.stdout:{ 2024-01-15T18:18:43.342 INFO:teuthology.orchestra.run.smithi093.stdout: "metadata": { 2024-01-15T18:18:43.342 INFO:teuthology.orchestra.run.smithi093.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:18:43.342 INFO:teuthology.orchestra.run.smithi093.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:18:43.342 INFO:teuthology.orchestra.run.smithi093.stdout: "entity_id": "1", 2024-01-15T18:18:43.342 INFO:teuthology.orchestra.run.smithi093.stdout: "hostname": "smithi093", 2024-01-15T18:18:43.342 INFO:teuthology.orchestra.run.smithi093.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-15T18:18:43.342 INFO:teuthology.orchestra.run.smithi093.stdout: "pid": "34808", 2024-01-15T18:18:43.342 INFO:teuthology.orchestra.run.smithi093.stdout: "root": "/" 2024-01-15T18:18:43.343 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:18:43.343 INFO:teuthology.orchestra.run.smithi093.stdout: "dentry_count": 0, 2024-01-15T18:18:43.343 INFO:teuthology.orchestra.run.smithi093.stdout: "dentry_pinned_count": 0, 2024-01-15T18:18:43.343 INFO:teuthology.orchestra.run.smithi093.stdout: "id": 8893, 2024-01-15T18:18:43.343 INFO:teuthology.orchestra.run.smithi093.stdout: "inst": { 2024-01-15T18:18:43.343 INFO:teuthology.orchestra.run.smithi093.stdout: "name": { 2024-01-15T18:18:43.344 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "client", 2024-01-15T18:18:43.344 INFO:teuthology.orchestra.run.smithi093.stdout: "num": 8893 2024-01-15T18:18:43.344 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:18:43.344 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": { 2024-01-15T18:18:43.344 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "v1", 2024-01-15T18:18:43.344 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:18:43.344 INFO:teuthology.orchestra.run.smithi093.stdout: "nonce": 2839178918 2024-01-15T18:18:43.344 INFO:teuthology.orchestra.run.smithi093.stdout: } 2024-01-15T18:18:43.344 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:18:43.344 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": { 2024-01-15T18:18:43.345 INFO:teuthology.orchestra.run.smithi093.stdout: "type": "v1", 2024-01-15T18:18:43.345 INFO:teuthology.orchestra.run.smithi093.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:18:43.345 INFO:teuthology.orchestra.run.smithi093.stdout: "nonce": 2839178918 2024-01-15T18:18:43.345 INFO:teuthology.orchestra.run.smithi093.stdout: }, 2024-01-15T18:18:43.345 INFO:teuthology.orchestra.run.smithi093.stdout: "inst_str": "client.8893 192.168.0.1:0/2839178918", 2024-01-15T18:18:43.345 INFO:teuthology.orchestra.run.smithi093.stdout: "addr_str": "192.168.0.1:0/2839178918", 2024-01-15T18:18:43.345 INFO:teuthology.orchestra.run.smithi093.stdout: "inode_count": 1, 2024-01-15T18:18:43.345 INFO:teuthology.orchestra.run.smithi093.stdout: "mds_epoch": 77, 2024-01-15T18:18:43.345 INFO:teuthology.orchestra.run.smithi093.stdout: "osd_epoch": 109, 2024-01-15T18:18:43.346 INFO:teuthology.orchestra.run.smithi093.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:18:43.346 INFO:teuthology.orchestra.run.smithi093.stdout: "blocklisted": false, 2024-01-15T18:18:43.346 INFO:teuthology.orchestra.run.smithi093.stdout: "fs_name": "cephfs" 2024-01-15T18:18:43.346 INFO:teuthology.orchestra.run.smithi093.stdout:} 2024-01-15T18:18:43.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:43.358 DEBUG:teuthology.orchestra.run.smithi093:> stat --file-system '--printf=%T 2024-01-15T18:18:43.358 DEBUG:teuthology.orchestra.run.smithi093:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-15T18:18:43.367 INFO:teuthology.orchestra.run.smithi093.stdout:fuseblk 2024-01-15T18:18:43.367 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-15T18:18:43.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:43.367 DEBUG:teuthology.orchestra.run.smithi093:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:18:43.430 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-15T18:18:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.589+0000 7f2ee8400640 1 Processor -- start 2024-01-15T18:18:43.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.589+0000 7f2ee8400640 1 -- start start 2024-01-15T18:18:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.589+0000 7f2ee8400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2ee015d680 0x7f2ee00cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.589+0000 7f2ee8400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee006c750 con 0x7f2ee00cfc30 2024-01-15T18:18:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.589+0000 7f2ee8400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee00cb4d0 con 0x7f2ee00ccf80 2024-01-15T18:18:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.589+0000 7f2ee8400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ee00d0120 con 0x7f2ee015d680 2024-01-15T18:18:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.589+0000 7f2ee5a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2ee015d680 0x7f2ee00cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:43.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.589+0000 7f2ee5a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2ee015d680 0x7f2ee00cd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38282/0 (socket says 172.21.15.27:38282) 2024-01-15T18:18:43.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.589+0000 7f2ee5000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2ee00cfc30 0x7f2ee006b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38714/0 (socket says 172.21.15.27:38714) 2024-01-15T18:18:43.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.589+0000 7f2ee5a00640 1 -- 172.21.15.27:0/1929172080 learned_addr learned my addr 172.21.15.27:0/1929172080 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:43.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.589+0000 7f2ee5a00640 1 -- 172.21.15.27:0/1929172080 >> v1:172.21.15.93:6789/0 conn(0x7f2ee00ccf80 legacy=0x7f2ee006c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:43.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.589+0000 7f2ee5a00640 1 -- 172.21.15.27:0/1929172080 >> v1:172.21.15.27:6789/0 conn(0x7f2ee00cfc30 legacy=0x7f2ee006b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:43.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.589+0000 7f2ee5a00640 1 -- 172.21.15.27:0/1929172080 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee00d0400 con 0x7f2ee015d680 2024-01-15T18:18:43.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.593+0000 7f2ee5a00640 1 --2- 172.21.15.27:0/1929172080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ee015d680 0x7f2ee00cd700 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0009870 tx=0x7f2ed001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19b3d8a975dbb983 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:43.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.593+0000 7f2edfe00640 1 -- 172.21.15.27:0/1929172080 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed00aa670 con 0x7f2ee015d680 2024-01-15T18:18:43.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.593+0000 7f2edfe00640 1 -- 172.21.15.27:0/1929172080 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed00aa7d0 con 0x7f2ee015d680 2024-01-15T18:18:43.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.593+0000 7f2edfe00640 1 -- 172.21.15.27:0/1929172080 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed00aab60 con 0x7f2ee015d680 2024-01-15T18:18:43.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.593+0000 7f2ee8400640 1 -- 172.21.15.27:0/1929172080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ee015d680 msgr2=0x7f2ee00cd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:43.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.593+0000 7f2ee8400640 1 --2- 172.21.15.27:0/1929172080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ee015d680 0x7f2ee00cd700 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0009870 tx=0x7f2ed001c510 comp rx=0 tx=0).stop 2024-01-15T18:18:43.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.593+0000 7f2ee8400640 1 -- 172.21.15.27:0/1929172080 shutdown_connections 2024-01-15T18:18:43.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.593+0000 7f2ee8400640 1 --2- 172.21.15.27:0/1929172080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ee015d680 0x7f2ee00cd700 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:43.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.593+0000 7f2ee8400640 1 -- 172.21.15.27:0/1929172080 >> 172.21.15.27:0/1929172080 conn(0x7f2ee00d0e80 msgr2=0x7f2ee00d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:43.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.597+0000 7f2ee8400640 1 -- 172.21.15.27:0/1929172080 shutdown_connections 2024-01-15T18:18:43.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.597+0000 7f2ee8400640 1 -- 172.21.15.27:0/1929172080 wait complete. 2024-01-15T18:18:43.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.597+0000 7f2ee8400640 1 Processor -- start 2024-01-15T18:18:43.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.597+0000 7f2ee8400640 1 -- start start 2024-01-15T18:18:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.597+0000 7f2ee8400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2ee00ccf80 0x7f2ee0153160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.597+0000 7f2ee8400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ee00cfc30 0x7f2ee00cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.597+0000 7f2ee8400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ee015d680 0x7f2ee00cc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.597+0000 7f2ee8400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee015e2c0 con 0x7f2ee015d680 2024-01-15T18:18:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.597+0000 7f2ee8400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee00c7840 con 0x7f2ee00cfc30 2024-01-15T18:18:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.597+0000 7f2ee8400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee00c7560 con 0x7f2ee00ccf80 2024-01-15T18:18:43.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.597+0000 7f2ee6400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ee015d680 0x7f2ee00cc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:43.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.597+0000 7f2ee6400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ee015d680 0x7f2ee00cc3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38294/0 (socket says 172.21.15.27:38294) 2024-01-15T18:18:43.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.597+0000 7f2ee6400640 1 -- 172.21.15.27:0/1072856863 learned_addr learned my addr 172.21.15.27:0/1072856863 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:43.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.597+0000 7f2ee5a00640 1 --2- 172.21.15.27:0/1072856863 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2ee00ccf80 0x7f2ee0153160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:43.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.597+0000 7f2ee5000640 1 --2- 172.21.15.27:0/1072856863 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ee00cfc30 0x7f2ee00cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:43.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.597+0000 7f2ee6400640 1 -- 172.21.15.27:0/1072856863 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2ee00ccf80 msgr2=0x7f2ee0153160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:43.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.597+0000 7f2ee6400640 1 --2- 172.21.15.27:0/1072856863 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2ee00ccf80 0x7f2ee0153160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:43.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.597+0000 7f2ee6400640 1 -- 172.21.15.27:0/1072856863 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ee00cfc30 msgr2=0x7f2ee00cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:43.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.597+0000 7f2ee6400640 1 --2- 172.21.15.27:0/1072856863 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ee00cfc30 0x7f2ee00cbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:43.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.597+0000 7f2ee6400640 1 -- 172.21.15.27:0/1072856863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed00094d0 con 0x7f2ee015d680 2024-01-15T18:18:43.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.597+0000 7f2ee5000640 1 --2- 172.21.15.27:0/1072856863 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ee00cfc30 0x7f2ee00cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:18:43.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.597+0000 7f2ee6400640 1 --2- 172.21.15.27:0/1072856863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ee015d680 0x7f2ee00cc3b0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f2ed800c770 tx=0x7f2ed800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:43.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.601+0000 7f2edd600640 1 -- 172.21.15.27:0/1072856863 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8003b80 con 0x7f2ee015d680 2024-01-15T18:18:43.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.601+0000 7f2edd600640 1 -- 172.21.15.27:0/1072856863 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed8003ce0 con 0x7f2ee015d680 2024-01-15T18:18:43.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.601+0000 7f2edd600640 1 -- 172.21.15.27:0/1072856863 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed800aa90 con 0x7f2ee015d680 2024-01-15T18:18:43.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.601+0000 7f2ee8400640 1 -- 172.21.15.27:0/1072856863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee015dc00 con 0x7f2ee015d680 2024-01-15T18:18:43.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.601+0000 7f2ee8400640 1 -- 172.21.15.27:0/1072856863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee01517b0 con 0x7f2ee015d680 2024-01-15T18:18:43.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.601+0000 7f2ee8400640 1 -- 172.21.15.27:0/1072856863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee015e400 con 0x7f2ee015d680 2024-01-15T18:18:43.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.601+0000 7f2edd600640 1 -- 172.21.15.27:0/1072856863 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f2ed800abf0 con 0x7f2ee015d680 2024-01-15T18:18:43.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.601+0000 7f2edd600640 1 --2- 172.21.15.27:0/1072856863 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2eb8067970 0x7f2eb8069e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:43.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.601+0000 7f2edd600640 1 -- 172.21.15.27:0/1072856863 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f2ed8082fd0 con 0x7f2ee015d680 2024-01-15T18:18:43.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.601+0000 7f2ee5a00640 1 --2- 172.21.15.27:0/1072856863 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2eb8067970 0x7f2eb8069e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:43.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.605+0000 7f2ee5a00640 1 --2- 172.21.15.27:0/1072856863 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2eb8067970 0x7f2eb8069e30 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f2ed00ae000 tx=0x7f2ed00b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:43.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.605+0000 7f2edd600640 1 -- 172.21.15.27:0/1072856863 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2ee015e400 con 0x7f2ee015d680 2024-01-15T18:18:43.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:43.805+0000 7f2ee8400640 1 -- 172.21.15.27:0/1072856863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2ee015e400 con 0x7f2ee015d680 2024-01-15T18:18:44.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.277+0000 7f2edd600640 1 -- 172.21.15.27:0/1072856863 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v78) v1 ==== 136+0+0 (secure 0 0 0) 0x7f2ee015e400 con 0x7f2ee015d680 2024-01-15T18:18:44.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.277+0000 7f2ee8400640 1 -- 172.21.15.27:0/1072856863 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2eb8067970 msgr2=0x7f2eb8069e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:44.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.277+0000 7f2ee8400640 1 --2- 172.21.15.27:0/1072856863 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2eb8067970 0x7f2eb8069e30 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f2ed00ae000 tx=0x7f2ed00b5040 comp rx=0 tx=0).stop 2024-01-15T18:18:44.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.277+0000 7f2ee8400640 1 -- 172.21.15.27:0/1072856863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ee015d680 msgr2=0x7f2ee00cc3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:44.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.277+0000 7f2ee8400640 1 --2- 172.21.15.27:0/1072856863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ee015d680 0x7f2ee00cc3b0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f2ed800c770 tx=0x7f2ed800cc30 comp rx=0 tx=0).stop 2024-01-15T18:18:44.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.277+0000 7f2ee8400640 1 -- 172.21.15.27:0/1072856863 shutdown_connections 2024-01-15T18:18:44.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.277+0000 7f2ee8400640 1 --2- 172.21.15.27:0/1072856863 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2eb8067970 0x7f2eb8069e30 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.277+0000 7f2ee8400640 1 --2- 172.21.15.27:0/1072856863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ee015d680 0x7f2ee00cc3b0 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.277+0000 7f2ee8400640 1 --2- 172.21.15.27:0/1072856863 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2ee00cfc30 0x7f2ee00cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.277+0000 7f2ee8400640 1 --2- 172.21.15.27:0/1072856863 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2ee00ccf80 0x7f2ee0153160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.277+0000 7f2ee8400640 1 -- 172.21.15.27:0/1072856863 >> 172.21.15.27:0/1072856863 conn(0x7f2ee00d0e80 msgr2=0x7f2ee00ae460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:44.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.277+0000 7f2ee8400640 1 -- 172.21.15.27:0/1072856863 shutdown_connections 2024-01-15T18:18:44.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.277+0000 7f2ee8400640 1 -- 172.21.15.27:0/1072856863 wait complete. 2024-01-15T18:18:44.301 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:18:44.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.445+0000 7f3e3fc00640 1 Processor -- start 2024-01-15T18:18:44.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.445+0000 7f3e3fc00640 1 -- start start 2024-01-15T18:18:44.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.445+0000 7f3e3fc00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f3e3815d680 0x7f3e380cfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:44.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.445+0000 7f3e3fc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e380cde10 0x7f3e380cc150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:44.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.445+0000 7f3e3fc00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e380b3cd0 con 0x7f3e3815d2a0 2024-01-15T18:18:44.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.445+0000 7f3e3fc00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e380d0260 con 0x7f3e3815d680 2024-01-15T18:18:44.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.445+0000 7f3e3fc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e380cb670 con 0x7f3e380cde10 2024-01-15T18:18:44.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.449+0000 7f3e3dc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e380cde10 0x7f3e380cc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:44.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.449+0000 7f3e37e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f3e3815d680 0x7f3e380cfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:44.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.449+0000 7f3e3dc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e380cde10 0x7f3e380cc150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:40986/0 (socket says 172.21.15.27:40986) 2024-01-15T18:18:44.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.449+0000 7f3e3dc00640 1 -- 172.21.15.27:0/577206464 learned_addr learned my addr 172.21.15.27:0/577206464 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:44.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.449+0000 7f3e3dc00640 1 -- 172.21.15.27:0/577206464 >> v2:172.21.15.93:3300/0 conn(0x7f3e3815d680 msgr2=0x7f3e380cfbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:44.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.449+0000 7f3e3dc00640 1 --2- 172.21.15.27:0/577206464 >> v2:172.21.15.93:3300/0 conn(0x7f3e3815d680 0x7f3e380cfbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.449+0000 7f3e3dc00640 1 -- 172.21.15.27:0/577206464 >> v1:172.21.15.93:6789/0 conn(0x7f3e3815d2a0 legacy=0x7f3e380cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:44.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.449+0000 7f3e3dc00640 1 -- 172.21.15.27:0/577206464 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e380cb950 con 0x7f3e380cde10 2024-01-15T18:18:44.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.449+0000 7f3e3dc00640 1 --2- 172.21.15.27:0/577206464 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e380cde10 0x7f3e380cc150 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3e30009e50 tx=0x7f3e3001c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da7f410a47149d26 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:44.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.449+0000 7f3e37400640 1 -- 172.21.15.27:0/577206464 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e300a57f0 con 0x7f3e380cde10 2024-01-15T18:18:44.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.449+0000 7f3e37400640 1 -- 172.21.15.27:0/577206464 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e300a5950 con 0x7f3e380cde10 2024-01-15T18:18:44.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.449+0000 7f3e37400640 1 -- 172.21.15.27:0/577206464 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e300a5ce0 con 0x7f3e380cde10 2024-01-15T18:18:44.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.453+0000 7f3e3fc00640 1 -- 172.21.15.27:0/577206464 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e380cde10 msgr2=0x7f3e380cc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:44.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.453+0000 7f3e3fc00640 1 --2- 172.21.15.27:0/577206464 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e380cde10 0x7f3e380cc150 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3e30009e50 tx=0x7f3e3001c950 comp rx=0 tx=0).stop 2024-01-15T18:18:44.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.453+0000 7f3e3fc00640 1 -- 172.21.15.27:0/577206464 shutdown_connections 2024-01-15T18:18:44.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.453+0000 7f3e3fc00640 1 --2- 172.21.15.27:0/577206464 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e380cde10 0x7f3e380cc150 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.453+0000 7f3e3fc00640 1 --2- 172.21.15.27:0/577206464 >> v2:172.21.15.93:3300/0 conn(0x7f3e3815d680 0x7f3e380cfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.453+0000 7f3e3fc00640 1 -- 172.21.15.27:0/577206464 >> 172.21.15.27:0/577206464 conn(0x7f3e380d0e80 msgr2=0x7f3e380d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:44.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.453+0000 7f3e3fc00640 1 -- 172.21.15.27:0/577206464 shutdown_connections 2024-01-15T18:18:44.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.453+0000 7f3e3fc00640 1 -- 172.21.15.27:0/577206464 wait complete. 2024-01-15T18:18:44.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.453+0000 7f3e3fc00640 1 Processor -- start 2024-01-15T18:18:44.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.453+0000 7f3e3fc00640 1 -- start start 2024-01-15T18:18:44.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.453+0000 7f3e3fc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e380cde10 0x7f3e38152b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:44.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.453+0000 7f3e3fc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3e3815d2a0 0x7f3e380c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:44.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.453+0000 7f3e3fc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e3815d680 0x7f3e380c5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:44.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.453+0000 7f3e3fc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3815fbd0 con 0x7f3e3815d680 2024-01-15T18:18:44.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.453+0000 7f3e3fc00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e38160710 con 0x7f3e3815d2a0 2024-01-15T18:18:44.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.453+0000 7f3e3fc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3815fde0 con 0x7f3e380cde10 2024-01-15T18:18:44.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.453+0000 7f3e3dc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e3815d680 0x7f3e380c5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:44.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.453+0000 7f3e3d200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e380cde10 0x7f3e38152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:44.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.453+0000 7f3e3d200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e380cde10 0x7f3e38152b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:40994/0 (socket says 172.21.15.27:40994) 2024-01-15T18:18:44.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.453+0000 7f3e3d200640 1 -- 172.21.15.27:0/3586606751 learned_addr learned my addr 172.21.15.27:0/3586606751 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:44.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.453+0000 7f3e37e00640 1 --2- 172.21.15.27:0/3586606751 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3e3815d2a0 0x7f3e380c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:44.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.457+0000 7f3e3d200640 1 -- 172.21.15.27:0/3586606751 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3e3815d2a0 msgr2=0x7f3e380c59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:44.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.457+0000 7f3e3d200640 1 --2- 172.21.15.27:0/3586606751 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3e3815d2a0 0x7f3e380c59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.457+0000 7f3e3d200640 1 -- 172.21.15.27:0/3586606751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e3815d680 msgr2=0x7f3e380c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:44.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.457+0000 7f3e3d200640 1 --2- 172.21.15.27:0/3586606751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e3815d680 0x7f3e380c5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.457+0000 7f3e3d200640 1 -- 172.21.15.27:0/3586606751 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e30009b30 con 0x7f3e380cde10 2024-01-15T18:18:44.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.457+0000 7f3e3d200640 1 --2- 172.21.15.27:0/3586606751 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e380cde10 0x7f3e38152b00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3e2800c500 tx=0x7f3e2800c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:44.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.457+0000 7f3e34c00640 1 -- 172.21.15.27:0/3586606751 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e28005220 con 0x7f3e380cde10 2024-01-15T18:18:44.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.457+0000 7f3e34c00640 1 -- 172.21.15.27:0/3586606751 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e28005380 con 0x7f3e380cde10 2024-01-15T18:18:44.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.457+0000 7f3e34c00640 1 -- 172.21.15.27:0/3586606751 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e28007270 con 0x7f3e380cde10 2024-01-15T18:18:44.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.457+0000 7f3e3fc00640 1 -- 172.21.15.27:0/3586606751 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e381600c0 con 0x7f3e380cde10 2024-01-15T18:18:44.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.457+0000 7f3e3fc00640 1 -- 172.21.15.27:0/3586606751 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e38153040 con 0x7f3e380cde10 2024-01-15T18:18:44.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.457+0000 7f3e3fc00640 1 -- 172.21.15.27:0/3586606751 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e380ce740 con 0x7f3e380cde10 2024-01-15T18:18:44.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.457+0000 7f3e34c00640 1 -- 172.21.15.27:0/3586606751 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f3e280073d0 con 0x7f3e380cde10 2024-01-15T18:18:44.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.461+0000 7f3e34c00640 1 --2- 172.21.15.27:0/3586606751 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e10067940 0x7f3e10069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:44.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.461+0000 7f3e34c00640 1 -- 172.21.15.27:0/3586606751 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f3e28082b60 con 0x7f3e380cde10 2024-01-15T18:18:44.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.461+0000 7f3e37e00640 1 --2- 172.21.15.27:0/3586606751 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e10067940 0x7f3e10069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:44.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.461+0000 7f3e37e00640 1 --2- 172.21.15.27:0/3586606751 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e10067940 0x7f3e10069e00 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f3e38159fa0 tx=0x7f3e20009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:44.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.465+0000 7f3e34c00640 1 -- 172.21.15.27:0/3586606751 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3e28051500 con 0x7f3e380cde10 2024-01-15T18:18:44.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.641+0000 7f3e3fc00640 1 -- 172.21.15.27:0/3586606751 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e3816a020 con 0x7f3e10067940 2024-01-15T18:18:44.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.649+0000 7f3e34c00640 1 -- 172.21.15.27:0/3586606751 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f3e20004040 con 0x7f3e10067940 2024-01-15T18:18:44.644 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:18:44.644 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:18:44.644 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-15T18:18:44.644 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:18:44.645 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:18:44.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.649+0000 7f3e3fc00640 1 -- 172.21.15.27:0/3586606751 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e10067940 msgr2=0x7f3e10069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:44.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.649+0000 7f3e3fc00640 1 --2- 172.21.15.27:0/3586606751 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e10067940 0x7f3e10069e00 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f3e38159fa0 tx=0x7f3e20009190 comp rx=0 tx=0).stop 2024-01-15T18:18:44.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.649+0000 7f3e3fc00640 1 -- 172.21.15.27:0/3586606751 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e380cde10 msgr2=0x7f3e38152b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:44.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.649+0000 7f3e3fc00640 1 --2- 172.21.15.27:0/3586606751 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e380cde10 0x7f3e38152b00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3e2800c500 tx=0x7f3e2800c9c0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.649+0000 7f3e3fc00640 1 -- 172.21.15.27:0/3586606751 shutdown_connections 2024-01-15T18:18:44.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.649+0000 7f3e3fc00640 1 --2- 172.21.15.27:0/3586606751 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3e10067940 0x7f3e10069e00 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.649+0000 7f3e3fc00640 1 --2- 172.21.15.27:0/3586606751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e3815d680 0x7f3e380c5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.649+0000 7f3e3fc00640 1 --2- 172.21.15.27:0/3586606751 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3e3815d2a0 0x7f3e380c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.649+0000 7f3e3fc00640 1 --2- 172.21.15.27:0/3586606751 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3e380cde10 0x7f3e38152b00 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.649+0000 7f3e3fc00640 1 -- 172.21.15.27:0/3586606751 >> 172.21.15.27:0/3586606751 conn(0x7f3e380d0e80 msgr2=0x7f3e381596a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:44.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.649+0000 7f3e3fc00640 1 -- 172.21.15.27:0/3586606751 shutdown_connections 2024-01-15T18:18:44.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.649+0000 7f3e3fc00640 1 -- 172.21.15.27:0/3586606751 wait complete. 2024-01-15T18:18:44.667 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:18:44.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.813+0000 7f4a4c400640 1 Processor -- start 2024-01-15T18:18:44.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.813+0000 7f4a4c400640 1 -- start start 2024-01-15T18:18:44.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.813+0000 7f4a4c400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4a440cd070 0x7f4a44168060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:44.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.813+0000 7f4a4c400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a440ad920 con 0x7f4a441468d0 2024-01-15T18:18:44.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.813+0000 7f4a4c400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a440ac880 con 0x7f4a440cf100 2024-01-15T18:18:44.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.813+0000 7f4a4c400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a440cd4e0 con 0x7f4a440cd070 2024-01-15T18:18:44.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.813+0000 7f4a49a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4a441468d0 0x7f4a440ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55430/0 (socket says 172.21.15.27:55430) 2024-01-15T18:18:44.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.813+0000 7f4a49a00640 1 -- 172.21.15.27:0/1649981374 learned_addr learned my addr 172.21.15.27:0/1649981374 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:44.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.813+0000 7f4a4a400640 1 --2- 172.21.15.27:0/1649981374 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4a440cd070 0x7f4a44168060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:44.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.813+0000 7f4a43e00640 1 -- 172.21.15.27:0/1649981374 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306802323 0 0) 0x7f4a34001120 con 0x7f4a441468d0 2024-01-15T18:18:44.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.813+0000 7f4a43e00640 1 -- 172.21.15.27:0/1649981374 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a24003530 con 0x7f4a441468d0 2024-01-15T18:18:44.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.813+0000 7f4a43e00640 1 -- 172.21.15.27:0/1649981374 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641287453 0 0) 0x7f4a440ac880 con 0x7f4a440cf100 2024-01-15T18:18:44.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.813+0000 7f4a43e00640 1 -- 172.21.15.27:0/1649981374 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a34001120 con 0x7f4a440cf100 2024-01-15T18:18:44.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.813+0000 7f4a4a400640 1 -- 172.21.15.27:0/1649981374 >> v1:172.21.15.93:6789/0 conn(0x7f4a440cf100 legacy=0x7f4a440cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:44.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.813+0000 7f4a4a400640 1 -- 172.21.15.27:0/1649981374 >> v1:172.21.15.27:6789/0 conn(0x7f4a441468d0 legacy=0x7f4a440ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:44.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.813+0000 7f4a4a400640 1 -- 172.21.15.27:0/1649981374 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a440acb60 con 0x7f4a440cd070 2024-01-15T18:18:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.817+0000 7f4a4a400640 1 --2- 172.21.15.27:0/1649981374 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4a440cd070 0x7f4a44168060 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c009dc0 tx=0x7f4a3c01c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b291a846275e280 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.817+0000 7f4a43e00640 1 -- 172.21.15.27:0/1649981374 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c0a5910 con 0x7f4a440cd070 2024-01-15T18:18:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.817+0000 7f4a43e00640 1 -- 172.21.15.27:0/1649981374 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a3c0a5a70 con 0x7f4a440cd070 2024-01-15T18:18:44.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.817+0000 7f4a43e00640 1 -- 172.21.15.27:0/1649981374 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c0a5de0 con 0x7f4a440cd070 2024-01-15T18:18:44.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.817+0000 7f4a4c400640 1 -- 172.21.15.27:0/1649981374 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4a440cd070 msgr2=0x7f4a44168060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:44.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.817+0000 7f4a4c400640 1 --2- 172.21.15.27:0/1649981374 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4a440cd070 0x7f4a44168060 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c009dc0 tx=0x7f4a3c01c940 comp rx=0 tx=0).stop 2024-01-15T18:18:44.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.817+0000 7f4a4c400640 1 -- 172.21.15.27:0/1649981374 shutdown_connections 2024-01-15T18:18:44.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.817+0000 7f4a4c400640 1 --2- 172.21.15.27:0/1649981374 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4a440cd070 0x7f4a44168060 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.817+0000 7f4a4c400640 1 -- 172.21.15.27:0/1649981374 >> 172.21.15.27:0/1649981374 conn(0x7f4a440cb700 msgr2=0x7f4a440cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:44.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.817+0000 7f4a4c400640 1 -- 172.21.15.27:0/1649981374 shutdown_connections 2024-01-15T18:18:44.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.817+0000 7f4a4c400640 1 -- 172.21.15.27:0/1649981374 wait complete. 2024-01-15T18:18:44.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.821+0000 7f4a4c400640 1 Processor -- start 2024-01-15T18:18:44.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.821+0000 7f4a4c400640 1 -- start start 2024-01-15T18:18:44.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.821+0000 7f4a4c400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4a440cd070 0x7f4a4413cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:44.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.821+0000 7f4a4c400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a440cf100 0x7f4a4415b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:44.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.821+0000 7f4a4c400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4a441468d0 0x7f4a4415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:44.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.821+0000 7f4a4c400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4413c650 con 0x7f4a440cf100 2024-01-15T18:18:44.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.821+0000 7f4a4c400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4413ba40 con 0x7f4a440cd070 2024-01-15T18:18:44.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.821+0000 7f4a4c400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4413bd20 con 0x7f4a441468d0 2024-01-15T18:18:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.821+0000 7f4a49000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a440cf100 0x7f4a4415b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.821+0000 7f4a49000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a440cf100 0x7f4a4415b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32812/0 (socket says 172.21.15.27:32812) 2024-01-15T18:18:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.821+0000 7f4a49000640 1 -- 172.21.15.27:0/393938033 learned_addr learned my addr 172.21.15.27:0/393938033 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.821+0000 7f4a4a400640 1 --2- 172.21.15.27:0/393938033 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4a441468d0 0x7f4a4415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.821+0000 7f4a49a00640 1 --2- 172.21.15.27:0/393938033 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4a440cd070 0x7f4a4413cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.821+0000 7f4a49000640 1 -- 172.21.15.27:0/393938033 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4a441468d0 msgr2=0x7f4a4415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.821+0000 7f4a49000640 1 --2- 172.21.15.27:0/393938033 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4a441468d0 0x7f4a4415a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.821+0000 7f4a49000640 1 -- 172.21.15.27:0/393938033 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4a440cd070 msgr2=0x7f4a4413cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.821+0000 7f4a49000640 1 --2- 172.21.15.27:0/393938033 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4a440cd070 0x7f4a4413cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.821+0000 7f4a49000640 1 -- 172.21.15.27:0/393938033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3c009a70 con 0x7f4a440cf100 2024-01-15T18:18:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.821+0000 7f4a49a00640 1 --2- 172.21.15.27:0/393938033 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4a440cd070 0x7f4a4413cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:18:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.821+0000 7f4a49000640 1 --2- 172.21.15.27:0/393938033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a440cf100 0x7f4a4415b810 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00c500 tx=0x7f4a2c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.821+0000 7f4a41600640 1 -- 172.21.15.27:0/393938033 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c0052d0 con 0x7f4a440cf100 2024-01-15T18:18:44.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.821+0000 7f4a41600640 1 -- 172.21.15.27:0/393938033 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2c005450 con 0x7f4a440cf100 2024-01-15T18:18:44.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.821+0000 7f4a41600640 1 -- 172.21.15.27:0/393938033 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c012870 con 0x7f4a440cf100 2024-01-15T18:18:44.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.821+0000 7f4a4c400640 1 -- 172.21.15.27:0/393938033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a4413c000 con 0x7f4a440cf100 2024-01-15T18:18:44.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.821+0000 7f4a4c400640 1 -- 172.21.15.27:0/393938033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a4413d020 con 0x7f4a440cf100 2024-01-15T18:18:44.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.825+0000 7f4a4c400640 1 -- 172.21.15.27:0/393938033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a4413d240 con 0x7f4a440cf100 2024-01-15T18:18:44.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.825+0000 7f4a41600640 1 -- 172.21.15.27:0/393938033 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f4a2c0129d0 con 0x7f4a440cf100 2024-01-15T18:18:44.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.825+0000 7f4a41600640 1 --2- 172.21.15.27:0/393938033 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4a1c067940 0x7f4a1c069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:44.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.825+0000 7f4a41600640 1 -- 172.21.15.27:0/393938033 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f4a2c082e90 con 0x7f4a440cf100 2024-01-15T18:18:44.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.825+0000 7f4a49a00640 1 --2- 172.21.15.27:0/393938033 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4a1c067940 0x7f4a1c069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:44.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.825+0000 7f4a49a00640 1 --2- 172.21.15.27:0/393938033 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4a1c067940 0x7f4a1c069e00 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f4a34001380 tx=0x7f4a34009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:44.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.829+0000 7f4a41600640 1 -- 172.21.15.27:0/393938033 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4a2c051780 con 0x7f4a440cf100 2024-01-15T18:18:44.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.989+0000 7f4a4c400640 1 -- 172.21.15.27:0/393938033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4a4413c800 con 0x7f4a440cf100 2024-01-15T18:18:44.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.989+0000 7f4a41600640 1 -- 172.21.15.27:0/393938033 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4a4413c800 con 0x7f4a440cf100 2024-01-15T18:18:44.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.989+0000 7f4a4c400640 1 -- 172.21.15.27:0/393938033 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4a1c067940 msgr2=0x7f4a1c069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:44.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.989+0000 7f4a4c400640 1 --2- 172.21.15.27:0/393938033 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4a1c067940 0x7f4a1c069e00 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f4a34001380 tx=0x7f4a34009040 comp rx=0 tx=0).stop 2024-01-15T18:18:44.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.989+0000 7f4a4c400640 1 -- 172.21.15.27:0/393938033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a440cf100 msgr2=0x7f4a4415b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:44.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.989+0000 7f4a4c400640 1 --2- 172.21.15.27:0/393938033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a440cf100 0x7f4a4415b810 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00c500 tx=0x7f4a2c00c9c0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.989+0000 7f4a4c400640 1 -- 172.21.15.27:0/393938033 shutdown_connections 2024-01-15T18:18:44.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.989+0000 7f4a4c400640 1 --2- 172.21.15.27:0/393938033 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4a1c067940 0x7f4a1c069e00 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.989+0000 7f4a4c400640 1 --2- 172.21.15.27:0/393938033 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4a441468d0 0x7f4a4415a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.989+0000 7f4a4c400640 1 --2- 172.21.15.27:0/393938033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4a440cf100 0x7f4a4415b810 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.989+0000 7f4a4c400640 1 --2- 172.21.15.27:0/393938033 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4a440cd070 0x7f4a4413cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.989+0000 7f4a4c400640 1 -- 172.21.15.27:0/393938033 >> 172.21.15.27:0/393938033 conn(0x7f4a440cb700 msgr2=0x7f4a440ce9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:44.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.993+0000 7f4a4c400640 1 -- 172.21.15.27:0/393938033 shutdown_connections 2024-01-15T18:18:44.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:44.993+0000 7f4a4c400640 1 -- 172.21.15.27:0/393938033 wait complete. 2024-01-15T18:18:45.011 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 10 2024-01-15T18:18:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.157+0000 7fc5b9400640 1 Processor -- start 2024-01-15T18:18:45.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.157+0000 7fc5b9400640 1 -- start start 2024-01-15T18:18:45.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.157+0000 7fc5b9400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fc5b4158900 0x7fc5b415f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:45.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.157+0000 7fc5b9400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5b40cf2e0 0x7fc5b40701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:45.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.157+0000 7fc5b9400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b40b1cb0 con 0x7fc5b415bf00 2024-01-15T18:18:45.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.157+0000 7fc5b9400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5b40b1f00 con 0x7fc5b4158900 2024-01-15T18:18:45.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.157+0000 7fc5b9400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b40b21e0 con 0x7fc5b40cf2e0 2024-01-15T18:18:45.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.157+0000 7fc5b1e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc5b415bf00 0x7fc5b40cebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55440/0 (socket says 172.21.15.27:55440) 2024-01-15T18:18:45.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.157+0000 7fc5b1e00640 1 -- 172.21.15.27:0/3039269007 learned_addr learned my addr 172.21.15.27:0/3039269007 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:45.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.157+0000 7fc5b3200640 1 --2- 172.21.15.27:0/3039269007 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5b40cf2e0 0x7fc5b40701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:45.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.157+0000 7fc5b2800640 1 --2- 172.21.15.27:0/3039269007 >> v2:172.21.15.93:3300/0 conn(0x7fc5b4158900 0x7fc5b415f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:45.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.157+0000 7fc5b1400640 1 -- 172.21.15.27:0/3039269007 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1789071173 0 0) 0x7fc5b40b1cb0 con 0x7fc5b415bf00 2024-01-15T18:18:45.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.157+0000 7fc5b1400640 1 -- 172.21.15.27:0/3039269007 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc594003530 con 0x7fc5b415bf00 2024-01-15T18:18:45.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.157+0000 7fc5b2800640 1 -- 172.21.15.27:0/3039269007 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5b40cf2e0 msgr2=0x7fc5b40701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:45.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.157+0000 7fc5b2800640 1 --2- 172.21.15.27:0/3039269007 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5b40cf2e0 0x7fc5b40701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.157+0000 7fc5b2800640 1 -- 172.21.15.27:0/3039269007 >> v1:172.21.15.27:6789/0 conn(0x7fc5b415bf00 legacy=0x7fc5b40cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:45.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.157+0000 7fc5b2800640 1 -- 172.21.15.27:0/3039269007 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b415ea70 con 0x7fc5b4158900 2024-01-15T18:18:45.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.161+0000 7fc5b2800640 1 --2- 172.21.15.27:0/3039269007 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc5b4158900 0x7fc5b415f550 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fc5a40097c0 tx=0x7fc5a401c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2091e3c411bb515 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:45.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.161+0000 7fc5b1400640 1 -- 172.21.15.27:0/3039269007 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4070f30 con 0x7fc5b4158900 2024-01-15T18:18:45.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.161+0000 7fc5b1400640 1 -- 172.21.15.27:0/3039269007 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5a40a7050 con 0x7fc5b4158900 2024-01-15T18:18:45.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.161+0000 7fc5b1400640 1 -- 172.21.15.27:0/3039269007 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a40ab820 con 0x7fc5b4158900 2024-01-15T18:18:45.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.161+0000 7fc5b9400640 1 -- 172.21.15.27:0/3039269007 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc5b4158900 msgr2=0x7fc5b415f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:45.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.161+0000 7fc5b9400640 1 --2- 172.21.15.27:0/3039269007 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc5b4158900 0x7fc5b415f550 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fc5a40097c0 tx=0x7fc5a401c480 comp rx=0 tx=0).stop 2024-01-15T18:18:45.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.161+0000 7fc5b9400640 1 -- 172.21.15.27:0/3039269007 shutdown_connections 2024-01-15T18:18:45.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.161+0000 7fc5b9400640 1 --2- 172.21.15.27:0/3039269007 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5b40cf2e0 0x7fc5b40701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.161+0000 7fc5b9400640 1 --2- 172.21.15.27:0/3039269007 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc5b4158900 0x7fc5b415f550 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.161+0000 7fc5b9400640 1 -- 172.21.15.27:0/3039269007 >> 172.21.15.27:0/3039269007 conn(0x7fc5b40cb700 msgr2=0x7fc5b40cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:45.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.165+0000 7fc5b9400640 1 -- 172.21.15.27:0/3039269007 shutdown_connections 2024-01-15T18:18:45.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.165+0000 7fc5b9400640 1 -- 172.21.15.27:0/3039269007 wait complete. 2024-01-15T18:18:45.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.165+0000 7fc5b9400640 1 Processor -- start 2024-01-15T18:18:45.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.165+0000 7fc5b9400640 1 -- start start 2024-01-15T18:18:45.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.165+0000 7fc5b9400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5b40cf2e0 0x7fc5b40ae770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:45.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.165+0000 7fc5b9400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc5b4158900 0x7fc5b40ce220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:45.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.165+0000 7fc5b9400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5b415bf00 0x7fc5b415d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:45.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.165+0000 7fc5b9400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b40b0700 con 0x7fc5b40cf2e0 2024-01-15T18:18:45.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.165+0000 7fc5b9400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b40aecb0 con 0x7fc5b4158900 2024-01-15T18:18:45.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.165+0000 7fc5b9400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b40b0420 con 0x7fc5b415bf00 2024-01-15T18:18:45.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.165+0000 7fc5b2800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5b40cf2e0 0x7fc5b40ae770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:45.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.165+0000 7fc5b2800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5b40cf2e0 0x7fc5b40ae770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32822/0 (socket says 172.21.15.27:32822) 2024-01-15T18:18:45.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.165+0000 7fc5b2800640 1 -- 172.21.15.27:0/3732287423 learned_addr learned my addr 172.21.15.27:0/3732287423 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:45.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.165+0000 7fc5b3200640 1 --2- 172.21.15.27:0/3732287423 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5b415bf00 0x7fc5b415d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:45.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.165+0000 7fc5b1e00640 1 --2- 172.21.15.27:0/3732287423 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc5b4158900 0x7fc5b40ce220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:45.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.165+0000 7fc5b2800640 1 -- 172.21.15.27:0/3732287423 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5b415bf00 msgr2=0x7fc5b415d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:45.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.165+0000 7fc5b2800640 1 --2- 172.21.15.27:0/3732287423 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5b415bf00 0x7fc5b415d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.165+0000 7fc5b2800640 1 -- 172.21.15.27:0/3732287423 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc5b4158900 msgr2=0x7fc5b40ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:45.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.165+0000 7fc5b2800640 1 --2- 172.21.15.27:0/3732287423 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc5b4158900 0x7fc5b40ce220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.165+0000 7fc5b2800640 1 -- 172.21.15.27:0/3732287423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a4009420 con 0x7fc5b40cf2e0 2024-01-15T18:18:45.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.165+0000 7fc5b3200640 1 --2- 172.21.15.27:0/3732287423 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5b415bf00 0x7fc5b415d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:18:45.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.165+0000 7fc5b1e00640 1 --2- 172.21.15.27:0/3732287423 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc5b4158900 0x7fc5b40ce220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:18:45.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.165+0000 7fc5b2800640 1 --2- 172.21.15.27:0/3732287423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5b40cf2e0 0x7fc5b40ae770 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fc5a40ae040 tx=0x7fc5a401acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:45.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.169+0000 7fc5aaa00640 1 -- 172.21.15.27:0/3732287423 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a40b7450 con 0x7fc5b40cf2e0 2024-01-15T18:18:45.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.169+0000 7fc5aaa00640 1 -- 172.21.15.27:0/3732287423 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5a40b75d0 con 0x7fc5b40cf2e0 2024-01-15T18:18:45.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.169+0000 7fc5b9400640 1 -- 172.21.15.27:0/3732287423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b4152a80 con 0x7fc5b40cf2e0 2024-01-15T18:18:45.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.169+0000 7fc5aaa00640 1 -- 172.21.15.27:0/3732287423 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a40b7980 con 0x7fc5b40cf2e0 2024-01-15T18:18:45.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.169+0000 7fc5b9400640 1 -- 172.21.15.27:0/3732287423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b4154fe0 con 0x7fc5b40cf2e0 2024-01-15T18:18:45.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.169+0000 7fc5b9400640 1 -- 172.21.15.27:0/3732287423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b41528e0 con 0x7fc5b40cf2e0 2024-01-15T18:18:45.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.169+0000 7fc5aaa00640 1 -- 172.21.15.27:0/3732287423 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7fc5a40b7ae0 con 0x7fc5b40cf2e0 2024-01-15T18:18:45.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.169+0000 7fc5aaa00640 1 --2- 172.21.15.27:0/3732287423 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc58c067970 0x7fc58c069e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:45.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.173+0000 7fc5aaa00640 1 -- 172.21.15.27:0/3732287423 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fc5a411da40 con 0x7fc5b40cf2e0 2024-01-15T18:18:45.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.173+0000 7fc5b1e00640 1 --2- 172.21.15.27:0/3732287423 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc58c067970 0x7fc58c069e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:45.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.173+0000 7fc5b1e00640 1 --2- 172.21.15.27:0/3732287423 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc58c067970 0x7fc58c069e30 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fc5b406d350 tx=0x7fc59c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:45.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.173+0000 7fc5aaa00640 1 -- 172.21.15.27:0/3732287423 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc5b41528e0 con 0x7fc5b40cf2e0 2024-01-15T18:18:45.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.337+0000 7fc5b9400640 1 -- 172.21.15.27:0/3732287423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7fc5b41528e0 con 0x7fc5b40cf2e0 2024-01-15T18:18:45.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.341+0000 7fc5aaa00640 1 -- 172.21.15.27:0/3732287423 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v41)=0 v41) v1 ==== 131+0+0 (secure 0 0 0) 0x7fc5b41528e0 con 0x7fc5b40cf2e0 2024-01-15T18:18:45.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.345+0000 7fc5b9400640 1 -- 172.21.15.27:0/3732287423 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc58c067970 msgr2=0x7fc58c069e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:45.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.345+0000 7fc5b9400640 1 --2- 172.21.15.27:0/3732287423 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc58c067970 0x7fc58c069e30 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fc5b406d350 tx=0x7fc59c009040 comp rx=0 tx=0).stop 2024-01-15T18:18:45.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.345+0000 7fc5b9400640 1 -- 172.21.15.27:0/3732287423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5b40cf2e0 msgr2=0x7fc5b40ae770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:45.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.345+0000 7fc5b9400640 1 --2- 172.21.15.27:0/3732287423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5b40cf2e0 0x7fc5b40ae770 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fc5a40ae040 tx=0x7fc5a401acc0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.345+0000 7fc5b9400640 1 -- 172.21.15.27:0/3732287423 shutdown_connections 2024-01-15T18:18:45.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.345+0000 7fc5b9400640 1 --2- 172.21.15.27:0/3732287423 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc58c067970 0x7fc58c069e30 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.345+0000 7fc5b9400640 1 --2- 172.21.15.27:0/3732287423 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc5b415bf00 0x7fc5b415d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.345+0000 7fc5b9400640 1 --2- 172.21.15.27:0/3732287423 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc5b4158900 0x7fc5b40ce220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.345+0000 7fc5b9400640 1 --2- 172.21.15.27:0/3732287423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5b40cf2e0 0x7fc5b40ae770 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.345+0000 7fc5b9400640 1 -- 172.21.15.27:0/3732287423 >> 172.21.15.27:0/3732287423 conn(0x7fc5b40cb700 msgr2=0x7fc5b40acef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:45.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.345+0000 7fc5b9400640 1 -- 172.21.15.27:0/3732287423 shutdown_connections 2024-01-15T18:18:45.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.345+0000 7fc5b9400640 1 -- 172.21.15.27:0/3732287423 wait complete. 2024-01-15T18:18:45.364 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 120 2024-01-15T18:18:45.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.509+0000 7fa657a00640 1 Processor -- start 2024-01-15T18:18:45.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.509+0000 7fa657a00640 1 -- start start 2024-01-15T18:18:45.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.509+0000 7fa657a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fa65015ad20 0x7fa65015b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:45.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.509+0000 7fa657a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa65015d900 0x7fa65015ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:45.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.509+0000 7fa657a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa650070ee0 0x7fa650071390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:45.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.509+0000 7fa657a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6500718d0 con 0x7fa65015d900 2024-01-15T18:18:45.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.509+0000 7fa657a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa65015a610 con 0x7fa65015ad20 2024-01-15T18:18:45.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.509+0000 7fa657a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa65015e3c0 con 0x7fa650070ee0 2024-01-15T18:18:45.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.509+0000 7fa64fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa65015d900 0x7fa65015ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:45.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.509+0000 7fa64fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa65015d900 0x7fa65015ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32838/0 (socket says 172.21.15.27:32838) 2024-01-15T18:18:45.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.509+0000 7fa64fe00640 1 -- 172.21.15.27:0/1391759962 learned_addr learned my addr 172.21.15.27:0/1391759962 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:45.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.509+0000 7fa655000640 1 --2- 172.21.15.27:0/1391759962 >> v2:172.21.15.93:3300/0 conn(0x7fa65015ad20 0x7fa65015b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:45.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.509+0000 7fa655a00640 1 --2- 172.21.15.27:0/1391759962 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa650070ee0 0x7fa650071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:45.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.509+0000 7fa64fe00640 1 -- 172.21.15.27:0/1391759962 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa650070ee0 msgr2=0x7fa650071390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:45.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.509+0000 7fa64fe00640 1 --2- 172.21.15.27:0/1391759962 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa650070ee0 0x7fa650071390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.509+0000 7fa64fe00640 1 -- 172.21.15.27:0/1391759962 >> v2:172.21.15.93:3300/0 conn(0x7fa65015ad20 msgr2=0x7fa65015b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:45.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.509+0000 7fa64fe00640 1 --2- 172.21.15.27:0/1391759962 >> v2:172.21.15.93:3300/0 conn(0x7fa65015ad20 0x7fa65015b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.509+0000 7fa64fe00640 1 -- 172.21.15.27:0/1391759962 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65015e6a0 con 0x7fa65015d900 2024-01-15T18:18:45.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.513+0000 7fa64fe00640 1 --2- 172.21.15.27:0/1391759962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa65015d900 0x7fa65015ddb0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7fa638009870 tx=0x7fa63801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=148a56663a33c3da server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:45.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.513+0000 7fa64f400640 1 -- 172.21.15.27:0/1391759962 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6380aa5f0 con 0x7fa65015d900 2024-01-15T18:18:45.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.513+0000 7fa64f400640 1 -- 172.21.15.27:0/1391759962 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6380aa750 con 0x7fa65015d900 2024-01-15T18:18:45.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.513+0000 7fa64f400640 1 -- 172.21.15.27:0/1391759962 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6380aaae0 con 0x7fa65015d900 2024-01-15T18:18:45.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.513+0000 7fa657a00640 1 -- 172.21.15.27:0/1391759962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa65015d900 msgr2=0x7fa65015ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:45.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.513+0000 7fa657a00640 1 --2- 172.21.15.27:0/1391759962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa65015d900 0x7fa65015ddb0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7fa638009870 tx=0x7fa63801c4d0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.513+0000 7fa657a00640 1 -- 172.21.15.27:0/1391759962 shutdown_connections 2024-01-15T18:18:45.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.513+0000 7fa657a00640 1 --2- 172.21.15.27:0/1391759962 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa650070ee0 0x7fa650071390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.513+0000 7fa657a00640 1 --2- 172.21.15.27:0/1391759962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa65015d900 0x7fa65015ddb0 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.513+0000 7fa657a00640 1 --2- 172.21.15.27:0/1391759962 >> v2:172.21.15.93:3300/0 conn(0x7fa65015ad20 0x7fa65015b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.513+0000 7fa657a00640 1 -- 172.21.15.27:0/1391759962 >> 172.21.15.27:0/1391759962 conn(0x7fa6500cb700 msgr2=0x7fa6500cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:45.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.513+0000 7fa657a00640 1 -- 172.21.15.27:0/1391759962 shutdown_connections 2024-01-15T18:18:45.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.513+0000 7fa657a00640 1 -- 172.21.15.27:0/1391759962 wait complete. 2024-01-15T18:18:45.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.517+0000 7fa657a00640 1 Processor -- start 2024-01-15T18:18:45.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.517+0000 7fa657a00640 1 -- start start 2024-01-15T18:18:45.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.517+0000 7fa657a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa650070ee0 0x7fa650146080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:45.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.517+0000 7fa657a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa65015ad20 0x7fa6501459a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:45.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.517+0000 7fa657a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa65015d900 0x7fa65014ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:45.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.517+0000 7fa657a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa650145ee0 con 0x7fa650070ee0 2024-01-15T18:18:45.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.517+0000 7fa657a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6500cca90 con 0x7fa65015ad20 2024-01-15T18:18:45.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.517+0000 7fa657a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa650150e80 con 0x7fa65015d900 2024-01-15T18:18:45.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.517+0000 7fa655000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa650070ee0 0x7fa650146080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:45.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.517+0000 7fa655000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa650070ee0 0x7fa650146080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32842/0 (socket says 172.21.15.27:32842) 2024-01-15T18:18:45.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.517+0000 7fa655000640 1 -- 172.21.15.27:0/1609650404 learned_addr learned my addr 172.21.15.27:0/1609650404 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:45.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.517+0000 7fa655a00640 1 --2- 172.21.15.27:0/1609650404 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa65015d900 0x7fa65014ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:45.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.517+0000 7fa64fe00640 1 --2- 172.21.15.27:0/1609650404 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa65015ad20 0x7fa6501459a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:45.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.517+0000 7fa655000640 1 -- 172.21.15.27:0/1609650404 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa65015d900 msgr2=0x7fa65014ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:45.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.517+0000 7fa655000640 1 --2- 172.21.15.27:0/1609650404 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa65015d900 0x7fa65014ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.517+0000 7fa655000640 1 -- 172.21.15.27:0/1609650404 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa65015ad20 msgr2=0x7fa6501459a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:45.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.517+0000 7fa655000640 1 --2- 172.21.15.27:0/1609650404 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa65015ad20 0x7fa6501459a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.517+0000 7fa655000640 1 -- 172.21.15.27:0/1609650404 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6380094d0 con 0x7fa650070ee0 2024-01-15T18:18:45.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.517+0000 7fa655000640 1 --2- 172.21.15.27:0/1609650404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa650070ee0 0x7fa650146080 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fa64000e930 tx=0x7fa64000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:45.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.517+0000 7fa64cc00640 1 -- 172.21.15.27:0/1609650404 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa640009690 con 0x7fa650070ee0 2024-01-15T18:18:45.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.517+0000 7fa64cc00640 1 -- 172.21.15.27:0/1609650404 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64000ca00 con 0x7fa650070ee0 2024-01-15T18:18:45.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.517+0000 7fa64cc00640 1 -- 172.21.15.27:0/1609650404 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64000cd70 con 0x7fa650070ee0 2024-01-15T18:18:45.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.517+0000 7fa657a00640 1 -- 172.21.15.27:0/1609650404 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6500ccce0 con 0x7fa650070ee0 2024-01-15T18:18:45.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.517+0000 7fa657a00640 1 -- 172.21.15.27:0/1609650404 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6500cd2a0 con 0x7fa650070ee0 2024-01-15T18:18:45.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.521+0000 7fa657a00640 1 -- 172.21.15.27:0/1609650404 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa65004e850 con 0x7fa650070ee0 2024-01-15T18:18:45.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.521+0000 7fa64cc00640 1 -- 172.21.15.27:0/1609650404 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7fa640013470 con 0x7fa650070ee0 2024-01-15T18:18:45.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.521+0000 7fa64cc00640 1 --2- 172.21.15.27:0/1609650404 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa628067970 0x7fa628069e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:45.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.521+0000 7fa64cc00640 1 -- 172.21.15.27:0/1609650404 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fa640083040 con 0x7fa650070ee0 2024-01-15T18:18:45.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.521+0000 7fa64fe00640 1 --2- 172.21.15.27:0/1609650404 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa628067970 0x7fa628069e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:45.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.521+0000 7fa64fe00640 1 --2- 172.21.15.27:0/1609650404 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa628067970 0x7fa628069e30 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fa6380ae000 tx=0x7fa6380a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:45.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.525+0000 7fa64cc00640 1 -- 172.21.15.27:0/1609650404 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa640051e40 con 0x7fa650070ee0 2024-01-15T18:18:45.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.681+0000 7fa657a00640 1 -- 172.21.15.27:0/1609650404 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7fa65006cee0 con 0x7fa650070ee0 2024-01-15T18:18:45.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.685+0000 7fa64cc00640 1 -- 172.21.15.27:0/1609650404 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v42)=0 v42) v1 ==== 137+0+0 (secure 0 0 0) 0x7fa6400210d0 con 0x7fa650070ee0 2024-01-15T18:18:45.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.689+0000 7fa657a00640 1 -- 172.21.15.27:0/1609650404 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa628067970 msgr2=0x7fa628069e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:45.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.689+0000 7fa657a00640 1 --2- 172.21.15.27:0/1609650404 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa628067970 0x7fa628069e30 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fa6380ae000 tx=0x7fa6380a7000 comp rx=0 tx=0).stop 2024-01-15T18:18:45.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.689+0000 7fa657a00640 1 -- 172.21.15.27:0/1609650404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa650070ee0 msgr2=0x7fa650146080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:45.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.689+0000 7fa657a00640 1 --2- 172.21.15.27:0/1609650404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa650070ee0 0x7fa650146080 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fa64000e930 tx=0x7fa64000edf0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.689+0000 7fa657a00640 1 -- 172.21.15.27:0/1609650404 shutdown_connections 2024-01-15T18:18:45.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.689+0000 7fa657a00640 1 --2- 172.21.15.27:0/1609650404 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa628067970 0x7fa628069e30 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.689+0000 7fa657a00640 1 --2- 172.21.15.27:0/1609650404 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa65015d900 0x7fa65014ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.689+0000 7fa657a00640 1 --2- 172.21.15.27:0/1609650404 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa65015ad20 0x7fa6501459a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.689+0000 7fa657a00640 1 --2- 172.21.15.27:0/1609650404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa650070ee0 0x7fa650146080 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.689+0000 7fa657a00640 1 -- 172.21.15.27:0/1609650404 >> 172.21.15.27:0/1609650404 conn(0x7fa6500cb700 msgr2=0x7fa650151e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:45.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.689+0000 7fa657a00640 1 -- 172.21.15.27:0/1609650404 shutdown_connections 2024-01-15T18:18:45.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.689+0000 7fa657a00640 1 -- 172.21.15.27:0/1609650404 wait complete. 2024-01-15T18:18:45.709 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-15T18:18:45.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.853+0000 7f9752c00640 1 Processor -- start 2024-01-15T18:18:45.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.853+0000 7f9752c00640 1 -- start start 2024-01-15T18:18:45.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f9752c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f974c0d1a40 0x7f974c0cba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:45.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f9752c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f974c0b0650 0x7f974c0b0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:45.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f9752c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f974c0cbfa0 con 0x7f974c0cf450 2024-01-15T18:18:45.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f9752c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f974c0cefa0 con 0x7f974c0b0650 2024-01-15T18:18:45.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f9752c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974c0ce240 con 0x7f974c0d1a40 2024-01-15T18:18:45.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974be00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f974c0cf450 0x7f974c0cf830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55452/0 (socket says 172.21.15.27:55452) 2024-01-15T18:18:45.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974be00640 1 -- 172.21.15.27:0/4027517131 learned_addr learned my addr 172.21.15.27:0/4027517131 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:45.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f9750c00640 1 --2- 172.21.15.27:0/4027517131 >> v2:172.21.15.93:3300/0 conn(0x7f974c0b0650 0x7f974c0b0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:45.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974b400640 1 --2- 172.21.15.27:0/4027517131 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f974c0d1a40 0x7f974c0cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:45.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974aa00640 1 -- 172.21.15.27:0/4027517131 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688250622 0 0) 0x7f974c0cbfa0 con 0x7f974c0cf450 2024-01-15T18:18:45.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974aa00640 1 -- 172.21.15.27:0/4027517131 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f972c003530 con 0x7f974c0cf450 2024-01-15T18:18:45.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974aa00640 1 -- 172.21.15.27:0/4027517131 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1914819846 0 0) 0x7f972c003530 con 0x7f974c0cf450 2024-01-15T18:18:45.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974aa00640 1 -- 172.21.15.27:0/4027517131 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f974c0d1a40 msgr2=0x7f974c0cba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:45.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974aa00640 1 --2- 172.21.15.27:0/4027517131 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f974c0d1a40 0x7f974c0cba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974aa00640 1 -- 172.21.15.27:0/4027517131 >> v2:172.21.15.93:3300/0 conn(0x7f974c0b0650 msgr2=0x7f974c0b0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:45.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974aa00640 1 --2- 172.21.15.27:0/4027517131 >> v2:172.21.15.93:3300/0 conn(0x7f974c0b0650 0x7f974c0b0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974aa00640 1 -- 172.21.15.27:0/4027517131 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974c0ccd70 con 0x7f974c0cf450 2024-01-15T18:18:45.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974aa00640 1 -- 172.21.15.27:0/4027517131 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f973c001680 con 0x7f974c0cf450 2024-01-15T18:18:45.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974aa00640 1 -- 172.21.15.27:0/4027517131 >> v1:172.21.15.27:6789/0 conn(0x7f974c0cf450 legacy=0x7f974c0cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:45.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974aa00640 1 --2- 172.21.15.27:0/4027517131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f972c0039f0 0x7f972c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:45.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974aa00640 1 --2- 172.21.15.27:0/4027517131 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f972c0047d0 0x7f972c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:45.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974aa00640 1 --2- 172.21.15.27:0/4027517131 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f974c0d1a40 0x7f972c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:45.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974aa00640 1 -- 172.21.15.27:0/4027517131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c0cefa0 con 0x7f972c0039f0 2024-01-15T18:18:45.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974aa00640 1 -- 172.21.15.27:0/4027517131 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c0ce240 con 0x7f972c0047d0 2024-01-15T18:18:45.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974aa00640 1 -- 172.21.15.27:0/4027517131 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f972c00b990 con 0x7f974c0d1a40 2024-01-15T18:18:45.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974be00640 1 --2- 172.21.15.27:0/4027517131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f972c0039f0 0x7f972c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:45.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974b400640 1 --2- 172.21.15.27:0/4027517131 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f972c0047d0 0x7f972c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:45.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f9750c00640 1 --2- 172.21.15.27:0/4027517131 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f974c0d1a40 0x7f972c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:45.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974be00640 1 -- 172.21.15.27:0/4027517131 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f974c0d1a40 msgr2=0x7f972c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:45.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974be00640 1 --2- 172.21.15.27:0/4027517131 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f974c0d1a40 0x7f972c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974be00640 1 -- 172.21.15.27:0/4027517131 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f972c0047d0 msgr2=0x7f972c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:45.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974be00640 1 --2- 172.21.15.27:0/4027517131 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f972c0047d0 0x7f972c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974be00640 1 -- 172.21.15.27:0/4027517131 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f972c003530 con 0x7f972c0039f0 2024-01-15T18:18:45.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.857+0000 7f974b400640 1 --2- 172.21.15.27:0/4027517131 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f972c0047d0 0x7f972c004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:18:45.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.861+0000 7f974be00640 1 --2- 172.21.15.27:0/4027517131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f972c0039f0 0x7f972c003e10 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f973c00c2c0 tx=0x7f973c01ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd56b59f5a049222 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:45.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.861+0000 7f974aa00640 1 -- 172.21.15.27:0/4027517131 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973c0058e0 con 0x7f972c0039f0 2024-01-15T18:18:45.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.861+0000 7f974aa00640 1 -- 172.21.15.27:0/4027517131 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973c005a40 con 0x7f972c0039f0 2024-01-15T18:18:45.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.861+0000 7f974aa00640 1 -- 172.21.15.27:0/4027517131 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973c005df0 con 0x7f972c0039f0 2024-01-15T18:18:45.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.861+0000 7f9752c00640 1 -- 172.21.15.27:0/4027517131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f972c0039f0 msgr2=0x7f972c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:45.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.861+0000 7f9752c00640 1 --2- 172.21.15.27:0/4027517131 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f972c0039f0 0x7f972c003e10 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f973c00c2c0 tx=0x7f973c01ccc0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.861+0000 7f9750c00640 1 -- 172.21.15.27:0/4027517131 reap_dead start 2024-01-15T18:18:45.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.861+0000 7f9752c00640 1 -- 172.21.15.27:0/4027517131 shutdown_connections 2024-01-15T18:18:45.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.861+0000 7f9752c00640 1 -- 172.21.15.27:0/4027517131 >> 172.21.15.27:0/4027517131 conn(0x7f974c0d0e70 msgr2=0x7f974c0d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:45.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.861+0000 7f9752c00640 1 -- 172.21.15.27:0/4027517131 shutdown_connections 2024-01-15T18:18:45.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.861+0000 7f9752c00640 1 -- 172.21.15.27:0/4027517131 wait complete. 2024-01-15T18:18:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.865+0000 7f9752c00640 1 Processor -- start 2024-01-15T18:18:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.865+0000 7f9752c00640 1 -- start start 2024-01-15T18:18:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.865+0000 7f9752c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f972c0039f0 0x7f974c0afd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.865+0000 7f9752c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f974c155240 0x7f974c154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.865+0000 7f9752c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f974c0b2960 0x7f974c0b2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.865+0000 7f9752c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c0b3320 con 0x7f974c155240 2024-01-15T18:18:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.865+0000 7f9752c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c155950 con 0x7f972c0039f0 2024-01-15T18:18:45.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.865+0000 7f9752c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974c06b6a0 con 0x7f974c0b2960 2024-01-15T18:18:45.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.865+0000 7f974b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f974c155240 0x7f974c154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:45.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.865+0000 7f974b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f974c155240 0x7f974c154980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32858/0 (socket says 172.21.15.27:32858) 2024-01-15T18:18:45.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.865+0000 7f974b400640 1 -- 172.21.15.27:0/2967793832 learned_addr learned my addr 172.21.15.27:0/2967793832 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:45.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.865+0000 7f974be00640 1 --2- 172.21.15.27:0/2967793832 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f972c0039f0 0x7f974c0afd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:45.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.865+0000 7f9750c00640 1 --2- 172.21.15.27:0/2967793832 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f974c0b2960 0x7f974c0b2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:45.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.865+0000 7f974b400640 1 -- 172.21.15.27:0/2967793832 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f974c0b2960 msgr2=0x7f974c0b2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:45.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.865+0000 7f974b400640 1 --2- 172.21.15.27:0/2967793832 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f974c0b2960 0x7f974c0b2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.865+0000 7f974b400640 1 -- 172.21.15.27:0/2967793832 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f972c0039f0 msgr2=0x7f974c0afd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:45.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.865+0000 7f974b400640 1 --2- 172.21.15.27:0/2967793832 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f972c0039f0 0x7f974c0afd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:45.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.865+0000 7f974b400640 1 -- 172.21.15.27:0/2967793832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974c0ae4f0 con 0x7f974c155240 2024-01-15T18:18:45.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.865+0000 7f974b400640 1 --2- 172.21.15.27:0/2967793832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f974c155240 0x7f974c154980 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f9734009160 tx=0x7f9734011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:45.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.865+0000 7f9743e00640 1 -- 172.21.15.27:0/2967793832 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9734004220 con 0x7f974c155240 2024-01-15T18:18:45.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.865+0000 7f9743e00640 1 -- 172.21.15.27:0/2967793832 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9734004380 con 0x7f974c155240 2024-01-15T18:18:45.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.865+0000 7f9743e00640 1 -- 172.21.15.27:0/2967793832 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9734015890 con 0x7f974c155240 2024-01-15T18:18:45.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.865+0000 7f9752c00640 1 -- 172.21.15.27:0/2967793832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f974c06b940 con 0x7f974c155240 2024-01-15T18:18:45.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.865+0000 7f9752c00640 1 -- 172.21.15.27:0/2967793832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f974c06bf00 con 0x7f974c155240 2024-01-15T18:18:45.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.865+0000 7f9752c00640 1 -- 172.21.15.27:0/2967793832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f974c0cd030 con 0x7f974c155240 2024-01-15T18:18:45.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.869+0000 7f9743e00640 1 -- 172.21.15.27:0/2967793832 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7f97340159f0 con 0x7f974c155240 2024-01-15T18:18:45.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.869+0000 7f9743e00640 1 --2- 172.21.15.27:0/2967793832 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9724067970 0x7f9724069e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:45.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.869+0000 7f9743e00640 1 -- 172.21.15.27:0/2967793832 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f97340836e0 con 0x7f974c155240 2024-01-15T18:18:45.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.869+0000 7f974be00640 1 --2- 172.21.15.27:0/2967793832 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9724067970 0x7f9724069e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:45.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.869+0000 7f974be00640 1 --2- 172.21.15.27:0/2967793832 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9724067970 0x7f9724069e30 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f973c0aa000 tx=0x7f973c001a10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:45.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:45.873+0000 7f9743e00640 1 -- 172.21.15.27:0/2967793832 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9734052100 con 0x7f974c155240 2024-01-15T18:18:46.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.053+0000 7f9752c00640 1 -- 172.21.15.27:0/2967793832 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f974c0cca90 con 0x7f9724067970 2024-01-15T18:18:46.057 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:18:46.058+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:18:46.058 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:18:46.058+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:18:46.058 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:18:46.058+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:18:46.058 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:18:46.058+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:18:46.058 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:18:46.058+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:18:46.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.093+0000 7f9743e00640 1 -- 172.21.15.27:0/2967793832 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f973c005dd0 con 0x7f9724067970 2024-01-15T18:18:46.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.093+0000 7f9752c00640 1 -- 172.21.15.27:0/2967793832 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9724067970 msgr2=0x7f9724069e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:46.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.093+0000 7f9752c00640 1 --2- 172.21.15.27:0/2967793832 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9724067970 0x7f9724069e30 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f973c0aa000 tx=0x7f973c001a10 comp rx=0 tx=0).stop 2024-01-15T18:18:46.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.093+0000 7f9752c00640 1 -- 172.21.15.27:0/2967793832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f974c155240 msgr2=0x7f974c154980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:46.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.093+0000 7f9752c00640 1 --2- 172.21.15.27:0/2967793832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f974c155240 0x7f974c154980 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f9734009160 tx=0x7f9734011a60 comp rx=0 tx=0).stop 2024-01-15T18:18:46.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.093+0000 7f9752c00640 1 -- 172.21.15.27:0/2967793832 shutdown_connections 2024-01-15T18:18:46.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.093+0000 7f9752c00640 1 --2- 172.21.15.27:0/2967793832 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9724067970 0x7f9724069e30 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.093+0000 7f9752c00640 1 --2- 172.21.15.27:0/2967793832 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f974c0b2960 0x7f974c0b2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.093+0000 7f9752c00640 1 --2- 172.21.15.27:0/2967793832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f974c155240 0x7f974c154980 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.093+0000 7f9752c00640 1 --2- 172.21.15.27:0/2967793832 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f972c0039f0 0x7f974c0afd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.093+0000 7f9752c00640 1 -- 172.21.15.27:0/2967793832 >> 172.21.15.27:0/2967793832 conn(0x7f974c0d0e70 msgr2=0x7f974c06e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:46.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.093+0000 7f9752c00640 1 -- 172.21.15.27:0/2967793832 shutdown_connections 2024-01-15T18:18:46.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.093+0000 7f9752c00640 1 -- 172.21.15.27:0/2967793832 wait complete. 2024-01-15T18:18:46.115 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:18:46.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6d3800640 1 Processor -- start 2024-01-15T18:18:46.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6d3800640 1 -- start start 2024-01-15T18:18:46.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6d3800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc6cc0cba60 0x7fc6cc0b0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:46.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6d3800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6cc0cefa0 con 0x7fc6cc0d1a40 2024-01-15T18:18:46.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6d3800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6cc0ce240 con 0x7fc6cc0cf450 2024-01-15T18:18:46.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6d3800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc0ccd70 con 0x7fc6cc0cba60 2024-01-15T18:18:46.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6cbe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc6cc0d1a40 0x7fc6cc0b0650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:55456/0 (socket says 172.21.15.27:55456) 2024-01-15T18:18:46.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6cbe00640 1 -- 172.21.15.27:0/531428827 learned_addr learned my addr 172.21.15.27:0/531428827 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:18:46.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6d1800640 1 --2- 172.21.15.27:0/531428827 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc6cc0cba60 0x7fc6cc0b0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:46.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6cb400640 1 -- 172.21.15.27:0/531428827 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453666105 0 0) 0x7fc6cc0cefa0 con 0x7fc6cc0d1a40 2024-01-15T18:18:46.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6cb400640 1 -- 172.21.15.27:0/531428827 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6ac003530 con 0x7fc6cc0d1a40 2024-01-15T18:18:46.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6cb400640 1 -- 172.21.15.27:0/531428827 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795313020 0 0) 0x7fc6cc0ce240 con 0x7fc6cc0cf450 2024-01-15T18:18:46.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6cb400640 1 -- 172.21.15.27:0/531428827 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6cc0cefa0 con 0x7fc6cc0cf450 2024-01-15T18:18:46.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6cb400640 1 -- 172.21.15.27:0/531428827 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3098261575 0 0) 0x7fc6ac003530 con 0x7fc6cc0d1a40 2024-01-15T18:18:46.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6cb400640 1 -- 172.21.15.27:0/531428827 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc6cc0cba60 msgr2=0x7fc6cc0b0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:46.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6cb400640 1 --2- 172.21.15.27:0/531428827 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc6cc0cba60 0x7fc6cc0b0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6cb400640 1 -- 172.21.15.27:0/531428827 >> v1:172.21.15.93:6789/0 conn(0x7fc6cc0cf450 legacy=0x7fc6cc0cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:46.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6cb400640 1 -- 172.21.15.27:0/531428827 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6cc0cc1f0 con 0x7fc6cc0d1a40 2024-01-15T18:18:46.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6cb400640 1 -- 172.21.15.27:0/531428827 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fc6b4001680 con 0x7fc6cc0d1a40 2024-01-15T18:18:46.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6d1800640 1 --2- 172.21.15.27:0/531428827 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc6cc0cba60 0x7fc6cc0b0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:18:46.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6cb400640 1 -- 172.21.15.27:0/531428827 >> v1:172.21.15.27:6789/0 conn(0x7fc6cc0d1a40 legacy=0x7fc6cc0b0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:46.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6cb400640 1 --2- 172.21.15.27:0/531428827 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6ac003bb0 0x7fc6ac003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:46.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6cb400640 1 --2- 172.21.15.27:0/531428827 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc6cc0cba60 0x7fc6ac0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6cb400640 1 --2- 172.21.15.27:0/531428827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6ac004d10 0x7fc6ac005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6cb400640 1 -- 172.21.15.27:0/531428827 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc0ccd70 con 0x7fc6ac004d10 2024-01-15T18:18:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6cb400640 1 -- 172.21.15.27:0/531428827 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac00ba40 con 0x7fc6ac003bb0 2024-01-15T18:18:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6cb400640 1 -- 172.21.15.27:0/531428827 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac00bc00 con 0x7fc6cc0cba60 2024-01-15T18:18:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6d0e00640 1 --2- 172.21.15.27:0/531428827 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6ac003bb0 0x7fc6ac003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6d1800640 1 --2- 172.21.15.27:0/531428827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6ac004d10 0x7fc6ac005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6cbe00640 1 --2- 172.21.15.27:0/531428827 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc6cc0cba60 0x7fc6ac0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6cbe00640 1 -- 172.21.15.27:0/531428827 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6ac003bb0 msgr2=0x7fc6ac003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:46.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6cbe00640 1 --2- 172.21.15.27:0/531428827 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6ac003bb0 0x7fc6ac003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6cbe00640 1 -- 172.21.15.27:0/531428827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6ac004d10 msgr2=0x7fc6ac005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:46.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6cbe00640 1 --2- 172.21.15.27:0/531428827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6ac004d10 0x7fc6ac005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6cbe00640 1 -- 172.21.15.27:0/531428827 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c4009a70 con 0x7fc6cc0cba60 2024-01-15T18:18:46.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.261+0000 7fc6d1800640 1 --2- 172.21.15.27:0/531428827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6ac004d10 0x7fc6ac005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:18:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.265+0000 7fc6cbe00640 1 --2- 172.21.15.27:0/531428827 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc6cc0cba60 0x7fc6ac0047d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc6b400c160 tx=0x7fc6b401f2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c174a6e01d2033ad server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.265+0000 7fc6cb400640 1 -- 172.21.15.27:0/531428827 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b40a8d00 con 0x7fc6cc0cba60 2024-01-15T18:18:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.265+0000 7fc6cb400640 1 -- 172.21.15.27:0/531428827 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6b40a8e60 con 0x7fc6cc0cba60 2024-01-15T18:18:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.265+0000 7fc6cb400640 1 -- 172.21.15.27:0/531428827 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b4005ae0 con 0x7fc6cc0cba60 2024-01-15T18:18:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.265+0000 7fc6d3800640 1 -- 172.21.15.27:0/531428827 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc6cc0cba60 msgr2=0x7fc6ac0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.265+0000 7fc6d3800640 1 --2- 172.21.15.27:0/531428827 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc6cc0cba60 0x7fc6ac0047d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc6b400c160 tx=0x7fc6b401f2d0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.265+0000 7fc6d1800640 1 -- 172.21.15.27:0/531428827 reap_dead start 2024-01-15T18:18:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.265+0000 7fc6d3800640 1 -- 172.21.15.27:0/531428827 shutdown_connections 2024-01-15T18:18:46.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.265+0000 7fc6d3800640 1 -- 172.21.15.27:0/531428827 >> 172.21.15.27:0/531428827 conn(0x7fc6cc0d0e70 msgr2=0x7fc6cc0d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:46.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.265+0000 7fc6d3800640 1 -- 172.21.15.27:0/531428827 shutdown_connections 2024-01-15T18:18:46.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.265+0000 7fc6d3800640 1 -- 172.21.15.27:0/531428827 wait complete. 2024-01-15T18:18:46.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.269+0000 7fc6d3800640 1 Processor -- start 2024-01-15T18:18:46.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.269+0000 7fc6d3800640 1 -- start start 2024-01-15T18:18:46.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.269+0000 7fc6d3800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc6cc0cba60 0x7fc6cc1553d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:46.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.269+0000 7fc6d3800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6cc0b3cc0 0x7fc6cc0c58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:46.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.269+0000 7fc6d3800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6cc0c5e00 0x7fc6cc0b5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:46.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.269+0000 7fc6d3800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc0c9f10 con 0x7fc6cc0c5e00 2024-01-15T18:18:46.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.269+0000 7fc6d3800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc0c6380 con 0x7fc6cc0b3cc0 2024-01-15T18:18:46.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.269+0000 7fc6d3800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc155910 con 0x7fc6cc0cba60 2024-01-15T18:18:46.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.269+0000 7fc6d1800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6cc0c5e00 0x7fc6cc0b5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:46.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.269+0000 7fc6d1800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6cc0c5e00 0x7fc6cc0b5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32876/0 (socket says 172.21.15.27:32876) 2024-01-15T18:18:46.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.269+0000 7fc6d1800640 1 -- 172.21.15.27:0/3254364679 learned_addr learned my addr 172.21.15.27:0/3254364679 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:46.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.269+0000 7fc6cbe00640 1 --2- 172.21.15.27:0/3254364679 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6cc0b3cc0 0x7fc6cc0c58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:46.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.269+0000 7fc6d0e00640 1 --2- 172.21.15.27:0/3254364679 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc6cc0cba60 0x7fc6cc1553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:46.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.269+0000 7fc6d1800640 1 -- 172.21.15.27:0/3254364679 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc6cc0cba60 msgr2=0x7fc6cc1553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:46.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.269+0000 7fc6d1800640 1 --2- 172.21.15.27:0/3254364679 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc6cc0cba60 0x7fc6cc1553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.269+0000 7fc6d1800640 1 -- 172.21.15.27:0/3254364679 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6cc0b3cc0 msgr2=0x7fc6cc0c58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:46.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.269+0000 7fc6d1800640 1 --2- 172.21.15.27:0/3254364679 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6cc0b3cc0 0x7fc6cc0c58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.269+0000 7fc6d1800640 1 -- 172.21.15.27:0/3254364679 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6cc0c92f0 con 0x7fc6cc0c5e00 2024-01-15T18:18:46.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.269+0000 7fc6d1800640 1 --2- 172.21.15.27:0/3254364679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6cc0c5e00 0x7fc6cc0b5f20 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc0713a0 tx=0x7fc6c400f7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:46.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.269+0000 7fc6c8c00640 1 -- 172.21.15.27:0/3254364679 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c400e040 con 0x7fc6cc0c5e00 2024-01-15T18:18:46.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.273+0000 7fc6c8c00640 1 -- 172.21.15.27:0/3254364679 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c400f040 con 0x7fc6cc0c5e00 2024-01-15T18:18:46.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.273+0000 7fc6c8c00640 1 -- 172.21.15.27:0/3254364679 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4003e00 con 0x7fc6cc0c5e00 2024-01-15T18:18:46.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.273+0000 7fc6d3800640 1 -- 172.21.15.27:0/3254364679 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6cc0c9570 con 0x7fc6cc0c5e00 2024-01-15T18:18:46.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.273+0000 7fc6d3800640 1 -- 172.21.15.27:0/3254364679 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6cc0c9b30 con 0x7fc6cc0c5e00 2024-01-15T18:18:46.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.273+0000 7fc6d3800640 1 -- 172.21.15.27:0/3254364679 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6cc0ca050 con 0x7fc6cc0c5e00 2024-01-15T18:18:46.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.273+0000 7fc6c8c00640 1 -- 172.21.15.27:0/3254364679 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7fc6c4004030 con 0x7fc6cc0c5e00 2024-01-15T18:18:46.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.277+0000 7fc6c8c00640 1 --2- 172.21.15.27:0/3254364679 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc6a4067970 0x7fc6a4069e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:46.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.277+0000 7fc6c8c00640 1 -- 172.21.15.27:0/3254364679 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fc6c4083d50 con 0x7fc6cc0c5e00 2024-01-15T18:18:46.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.277+0000 7fc6d0e00640 1 --2- 172.21.15.27:0/3254364679 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc6a4067970 0x7fc6a4069e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:46.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.277+0000 7fc6d0e00640 1 --2- 172.21.15.27:0/3254364679 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc6a4067970 0x7fc6a4069e30 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc0b6dc0 tx=0x7fc6bc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:46.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.277+0000 7fc6c8c00640 1 -- 172.21.15.27:0/3254364679 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc6c40527b0 con 0x7fc6cc0c5e00 2024-01-15T18:18:46.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.461+0000 7fc6d3800640 1 -- 172.21.15.27:0/3254364679 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6cc0cca90 con 0x7fc6a4067970 2024-01-15T18:18:46.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.469+0000 7fc6c8c00640 1 -- 172.21.15.27:0/3254364679 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fc6bc012890 con 0x7fc6a4067970 2024-01-15T18:18:46.468 INFO:teuthology.orchestra.run.smithi027.stdout:/volumes/_nogroup/sv1/1a63cb47-42d3-4f34-bbc0-e0bda8e84ca4 2024-01-15T18:18:46.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.469+0000 7fc6d3800640 1 -- 172.21.15.27:0/3254364679 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc6a4067970 msgr2=0x7fc6a4069e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:46.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.469+0000 7fc6d3800640 1 --2- 172.21.15.27:0/3254364679 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc6a4067970 0x7fc6a4069e30 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc0b6dc0 tx=0x7fc6bc009040 comp rx=0 tx=0).stop 2024-01-15T18:18:46.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.469+0000 7fc6d3800640 1 -- 172.21.15.27:0/3254364679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6cc0c5e00 msgr2=0x7fc6cc0b5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:46.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.469+0000 7fc6d3800640 1 --2- 172.21.15.27:0/3254364679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6cc0c5e00 0x7fc6cc0b5f20 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc0713a0 tx=0x7fc6c400f7b0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.469+0000 7fc6d3800640 1 -- 172.21.15.27:0/3254364679 shutdown_connections 2024-01-15T18:18:46.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.469+0000 7fc6d3800640 1 --2- 172.21.15.27:0/3254364679 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc6a4067970 0x7fc6a4069e30 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.469+0000 7fc6d3800640 1 --2- 172.21.15.27:0/3254364679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6cc0c5e00 0x7fc6cc0b5f20 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.469+0000 7fc6d3800640 1 --2- 172.21.15.27:0/3254364679 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc6cc0b3cc0 0x7fc6cc0c58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.469+0000 7fc6d3800640 1 --2- 172.21.15.27:0/3254364679 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc6cc0cba60 0x7fc6cc1553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.469+0000 7fc6d3800640 1 -- 172.21.15.27:0/3254364679 >> 172.21.15.27:0/3254364679 conn(0x7fc6cc0d0e70 msgr2=0x7fc6cc06ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:46.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.469+0000 7fc6d3800640 1 -- 172.21.15.27:0/3254364679 shutdown_connections 2024-01-15T18:18:46.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.469+0000 7fc6d3800640 1 -- 172.21.15.27:0/3254364679 wait complete. 2024-01-15T18:18:46.489 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:18:46.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.637+0000 7fd91f800640 1 Processor -- start 2024-01-15T18:18:46.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.637+0000 7fd91f800640 1 -- start start 2024-01-15T18:18:46.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.637+0000 7fd91f800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd9180cf460 0x7fd9180cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:46.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.637+0000 7fd91f800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fd91815b4b0 0x7fd9180cba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:46.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.637+0000 7fd91f800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9180cfd80 con 0x7fd9180cec60 2024-01-15T18:18:46.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.637+0000 7fd91f800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9180d19a0 con 0x7fd91815b4b0 2024-01-15T18:18:46.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.637+0000 7fd91f800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9180d1c80 con 0x7fd9180cf460 2024-01-15T18:18:46.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.637+0000 7fd91ce00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd9180cf460 0x7fd9180cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:46.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.637+0000 7fd917e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fd91815b4b0 0x7fd9180cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:46.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.637+0000 7fd91ce00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd9180cf460 0x7fd9180cf840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:41102/0 (socket says 172.21.15.27:41102) 2024-01-15T18:18:46.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.637+0000 7fd91ce00640 1 -- 172.21.15.27:0/304675502 learned_addr learned my addr 172.21.15.27:0/304675502 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:46.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.637+0000 7fd91ce00640 1 -- 172.21.15.27:0/304675502 >> v2:172.21.15.93:3300/0 conn(0x7fd91815b4b0 msgr2=0x7fd9180cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:46.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.637+0000 7fd91ce00640 1 --2- 172.21.15.27:0/304675502 >> v2:172.21.15.93:3300/0 conn(0x7fd91815b4b0 0x7fd9180cba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.637+0000 7fd91ce00640 1 -- 172.21.15.27:0/304675502 >> v1:172.21.15.93:6789/0 conn(0x7fd9180cec60 legacy=0x7fd9180cbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:46.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.637+0000 7fd91ce00640 1 -- 172.21.15.27:0/304675502 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9180b07c0 con 0x7fd9180cf460 2024-01-15T18:18:46.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.637+0000 7fd917e00640 1 --2- 172.21.15.27:0/304675502 >> v2:172.21.15.93:3300/0 conn(0x7fd91815b4b0 0x7fd9180cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:18:46.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.641+0000 7fd91ce00640 1 --2- 172.21.15.27:0/304675502 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd9180cf460 0x7fd9180cf840 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd908009870 tx=0x7fd90801c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e341739a2f16f3c2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:46.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.641+0000 7fd917400640 1 -- 172.21.15.27:0/304675502 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9080a9400 con 0x7fd9180cf460 2024-01-15T18:18:46.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.641+0000 7fd917400640 1 -- 172.21.15.27:0/304675502 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9080a9560 con 0x7fd9180cf460 2024-01-15T18:18:46.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.641+0000 7fd917400640 1 -- 172.21.15.27:0/304675502 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9080a98f0 con 0x7fd9180cf460 2024-01-15T18:18:46.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.641+0000 7fd91f800640 1 -- 172.21.15.27:0/304675502 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd9180cf460 msgr2=0x7fd9180cf840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:46.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.641+0000 7fd91f800640 1 --2- 172.21.15.27:0/304675502 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd9180cf460 0x7fd9180cf840 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd908009870 tx=0x7fd90801c540 comp rx=0 tx=0).stop 2024-01-15T18:18:46.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.641+0000 7fd91f800640 1 -- 172.21.15.27:0/304675502 shutdown_connections 2024-01-15T18:18:46.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.641+0000 7fd91f800640 1 --2- 172.21.15.27:0/304675502 >> v2:172.21.15.93:3300/0 conn(0x7fd91815b4b0 0x7fd9180cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.641+0000 7fd91f800640 1 --2- 172.21.15.27:0/304675502 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd9180cf460 0x7fd9180cf840 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.641+0000 7fd91f800640 1 -- 172.21.15.27:0/304675502 >> 172.21.15.27:0/304675502 conn(0x7fd9180d0e80 msgr2=0x7fd9180d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:46.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.641+0000 7fd91f800640 1 -- 172.21.15.27:0/304675502 shutdown_connections 2024-01-15T18:18:46.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.641+0000 7fd91f800640 1 -- 172.21.15.27:0/304675502 wait complete. 2024-01-15T18:18:46.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.645+0000 7fd91f800640 1 Processor -- start 2024-01-15T18:18:46.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.645+0000 7fd91f800640 1 -- start start 2024-01-15T18:18:46.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.645+0000 7fd91f800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9180cec60 0x7fd91814bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:46.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.645+0000 7fd91f800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd9180cf460 0x7fd91814f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:46.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.645+0000 7fd91f800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd91815b4b0 0x7fd91814f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:46.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.645+0000 7fd91f800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91814c420 con 0x7fd9180cec60 2024-01-15T18:18:46.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.645+0000 7fd91f800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd918151b80 con 0x7fd9180cf460 2024-01-15T18:18:46.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.645+0000 7fd91f800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd918150fb0 con 0x7fd91815b4b0 2024-01-15T18:18:46.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.645+0000 7fd91ce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9180cec60 0x7fd91814bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:46.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.645+0000 7fd91ce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9180cec60 0x7fd91814bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:32890/0 (socket says 172.21.15.27:32890) 2024-01-15T18:18:46.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.645+0000 7fd91ce00640 1 -- 172.21.15.27:0/2882852587 learned_addr learned my addr 172.21.15.27:0/2882852587 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:18:46.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.645+0000 7fd91d800640 1 --2- 172.21.15.27:0/2882852587 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd91815b4b0 0x7fd91814f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:46.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.645+0000 7fd917e00640 1 --2- 172.21.15.27:0/2882852587 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd9180cf460 0x7fd91814f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:46.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.645+0000 7fd91ce00640 1 -- 172.21.15.27:0/2882852587 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd91815b4b0 msgr2=0x7fd91814f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:46.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.645+0000 7fd91ce00640 1 --2- 172.21.15.27:0/2882852587 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd91815b4b0 0x7fd91814f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.645+0000 7fd91ce00640 1 -- 172.21.15.27:0/2882852587 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd9180cf460 msgr2=0x7fd91814f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:46.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.645+0000 7fd91ce00640 1 --2- 172.21.15.27:0/2882852587 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd9180cf460 0x7fd91814f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.645+0000 7fd91ce00640 1 -- 172.21.15.27:0/2882852587 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd900009420 con 0x7fd9180cec60 2024-01-15T18:18:46.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.645+0000 7fd91ce00640 1 --2- 172.21.15.27:0/2882852587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9180cec60 0x7fd91814bee0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fd90801a170 tx=0x7fd9080192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:46.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.645+0000 7fd914c00640 1 -- 172.21.15.27:0/2882852587 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9080a9400 con 0x7fd9180cec60 2024-01-15T18:18:46.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.645+0000 7fd914c00640 1 -- 172.21.15.27:0/2882852587 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9080b1800 con 0x7fd9180cec60 2024-01-15T18:18:46.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.649+0000 7fd91f800640 1 -- 172.21.15.27:0/2882852587 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9080094d0 con 0x7fd9180cec60 2024-01-15T18:18:46.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.649+0000 7fd914c00640 1 -- 172.21.15.27:0/2882852587 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9080b1bb0 con 0x7fd9180cec60 2024-01-15T18:18:46.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.649+0000 7fd91f800640 1 -- 172.21.15.27:0/2882852587 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd918151530 con 0x7fd9180cec60 2024-01-15T18:18:46.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.649+0000 7fd91f800640 1 -- 172.21.15.27:0/2882852587 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd91804e850 con 0x7fd9180cec60 2024-01-15T18:18:46.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.649+0000 7fd914c00640 1 -- 172.21.15.27:0/2882852587 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 87639+0+0 (secure 0 0 0) 0x7fd9080b1d10 con 0x7fd9180cec60 2024-01-15T18:18:46.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.649+0000 7fd914c00640 1 --2- 172.21.15.27:0/2882852587 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd8f0067940 0x7fd8f0069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:46.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.649+0000 7fd914c00640 1 -- 172.21.15.27:0/2882852587 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fd90811d660 con 0x7fd9180cec60 2024-01-15T18:18:46.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.649+0000 7fd917e00640 1 --2- 172.21.15.27:0/2882852587 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd8f0067940 0x7fd8f0069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:46.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.653+0000 7fd917e00640 1 --2- 172.21.15.27:0/2882852587 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd8f0067940 0x7fd8f0069e00 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fd91814ca90 tx=0x7fd900008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:46.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.653+0000 7fd914c00640 1 -- 172.21.15.27:0/2882852587 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd9080ec150 con 0x7fd9180cec60 2024-01-15T18:18:46.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.837+0000 7fd91f800640 1 -- 172.21.15.27:0/2882852587 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd918168630 con 0x7fd8f0067940 2024-01-15T18:18:46.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.841+0000 7fd914c00640 1 -- 172.21.15.27:0/2882852587 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fd900011e60 con 0x7fd8f0067940 2024-01-15T18:18:46.841 INFO:teuthology.orchestra.run.smithi027.stdout:/volumes/_nogroup/sv1/1a63cb47-42d3-4f34-bbc0-e0bda8e84ca4 2024-01-15T18:18:46.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.841+0000 7fd91f800640 1 -- 172.21.15.27:0/2882852587 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd8f0067940 msgr2=0x7fd8f0069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:46.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.841+0000 7fd91f800640 1 --2- 172.21.15.27:0/2882852587 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd8f0067940 0x7fd8f0069e00 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fd91814ca90 tx=0x7fd900008040 comp rx=0 tx=0).stop 2024-01-15T18:18:46.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.841+0000 7fd91f800640 1 -- 172.21.15.27:0/2882852587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9180cec60 msgr2=0x7fd91814bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:46.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.841+0000 7fd91f800640 1 --2- 172.21.15.27:0/2882852587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9180cec60 0x7fd91814bee0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fd90801a170 tx=0x7fd9080192e0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.841+0000 7fd91f800640 1 -- 172.21.15.27:0/2882852587 shutdown_connections 2024-01-15T18:18:46.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.841+0000 7fd91f800640 1 --2- 172.21.15.27:0/2882852587 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd8f0067940 0x7fd8f0069e00 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.841+0000 7fd91f800640 1 --2- 172.21.15.27:0/2882852587 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd91815b4b0 0x7fd91814f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.841+0000 7fd91f800640 1 --2- 172.21.15.27:0/2882852587 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd9180cf460 0x7fd91814f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.841+0000 7fd91f800640 1 --2- 172.21.15.27:0/2882852587 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd9180cec60 0x7fd91814bee0 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.841+0000 7fd91f800640 1 -- 172.21.15.27:0/2882852587 >> 172.21.15.27:0/2882852587 conn(0x7fd9180d0e80 msgr2=0x7fd9180cb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:46.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.845+0000 7fd91f800640 1 -- 172.21.15.27:0/2882852587 shutdown_connections 2024-01-15T18:18:46.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:18:46.845+0000 7fd91f800640 1 -- 172.21.15.27:0/2882852587 wait complete. 2024-01-15T18:18:46.863 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/1a63cb47-42d3-4f34-bbc0-e0bda8e84ca4 2024-01-15T18:18:46.863 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:18:46.863 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:18:46.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:46.873 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/1a63cb47-42d3-4f34-bbc0-e0bda8e84ca4/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:18:53.998 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records in 2024-01-15T18:18:53.998 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records out 2024-01-15T18:18:53.998 INFO:teuthology.orchestra.run.smithi027.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.0599 s, 152 MB/s 2024-01-15T18:18:53.999 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:18:53.999 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:18:54.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:18:54.008 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/1a63cb47-42d3-4f34-bbc0-e0bda8e84ca4/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:18:57.974 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:18:57.979 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:19:00.821 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records in 2024-01-15T18:19:00.821 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records out 2024-01-15T18:19:00.821 INFO:teuthology.orchestra.run.smithi027.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.75822 s, 159 MB/s 2024-01-15T18:19:00.822 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:19:00.822 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:19:00.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:19:00.831 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/1a63cb47-42d3-4f34-bbc0-e0bda8e84ca4/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:19:07.792 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records in 2024-01-15T18:19:07.792 INFO:teuthology.orchestra.run.smithi027.stderr:1024+0 records out 2024-01-15T18:19:07.792 INFO:teuthology.orchestra.run.smithi027.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.90509 s, 156 MB/s 2024-01-15T18:19:07.793 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-15T18:19:07.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.941+0000 7f2c69800640 1 Processor -- start 2024-01-15T18:19:07.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.941+0000 7f2c69800640 1 -- start start 2024-01-15T18:19:07.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.941+0000 7f2c69800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f2c6415b100 0x7f2c64070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:07.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.941+0000 7f2c69800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2c64071420 0x7f2c6415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:07.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.941+0000 7f2c69800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c64071860 con 0x7f2c6415ad20 2024-01-15T18:19:07.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.941+0000 7f2c69800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c6415a610 con 0x7f2c64071420 2024-01-15T18:19:07.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.941+0000 7f2c69800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c6415e3c0 con 0x7f2c6415b100 2024-01-15T18:19:07.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.941+0000 7f2c63400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2c64071420 0x7f2c6415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:07.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.941+0000 7f2c62a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2c6415ad20 0x7f2c6415d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59434/0 (socket says 172.21.15.27:59434) 2024-01-15T18:19:07.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.941+0000 7f2c63400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2c64071420 0x7f2c6415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40502/0 (socket says 172.21.15.27:40502) 2024-01-15T18:19:07.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.941+0000 7f2c62a00640 1 -- 172.21.15.27:0/2539180290 learned_addr learned my addr 172.21.15.27:0/2539180290 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:19:07.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.941+0000 7f2c62000640 1 --2- 172.21.15.27:0/2539180290 >> v2:172.21.15.93:3300/0 conn(0x7f2c6415b100 0x7f2c64070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:07.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.941+0000 7f2c63400640 1 -- 172.21.15.27:0/2539180290 >> v2:172.21.15.93:3300/0 conn(0x7f2c6415b100 msgr2=0x7f2c64070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:07.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.941+0000 7f2c63400640 1 --2- 172.21.15.27:0/2539180290 >> v2:172.21.15.93:3300/0 conn(0x7f2c6415b100 0x7f2c64070ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:07.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.941+0000 7f2c63400640 1 -- 172.21.15.27:0/2539180290 >> v1:172.21.15.27:6789/0 conn(0x7f2c6415ad20 legacy=0x7f2c6415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:07.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.941+0000 7f2c61600640 1 -- 172.21.15.27:0/2539180290 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620027617 0 0) 0x7f2c64071860 con 0x7f2c6415ad20 2024-01-15T18:19:07.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.941+0000 7f2c63400640 1 -- 172.21.15.27:0/2539180290 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c6415e6a0 con 0x7f2c64071420 2024-01-15T18:19:07.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.945+0000 7f2c63400640 1 --2- 172.21.15.27:0/2539180290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c64071420 0x7f2c6415bee0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c009d90 tx=0x7f2c5c01c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4db0b163e01333e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:07.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.945+0000 7f2c61600640 1 -- 172.21.15.27:0/2539180290 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c0a5910 con 0x7f2c64071420 2024-01-15T18:19:07.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.945+0000 7f2c61600640 1 -- 172.21.15.27:0/2539180290 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5c0a5a70 con 0x7f2c64071420 2024-01-15T18:19:07.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.945+0000 7f2c61600640 1 -- 172.21.15.27:0/2539180290 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c0a5e00 con 0x7f2c64071420 2024-01-15T18:19:07.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.945+0000 7f2c69800640 1 -- 172.21.15.27:0/2539180290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c64071420 msgr2=0x7f2c6415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:07.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.945+0000 7f2c69800640 1 --2- 172.21.15.27:0/2539180290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c64071420 0x7f2c6415bee0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c009d90 tx=0x7f2c5c01c850 comp rx=0 tx=0).stop 2024-01-15T18:19:07.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.945+0000 7f2c69800640 1 -- 172.21.15.27:0/2539180290 shutdown_connections 2024-01-15T18:19:07.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.945+0000 7f2c69800640 1 --2- 172.21.15.27:0/2539180290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c64071420 0x7f2c6415bee0 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:07.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.945+0000 7f2c69800640 1 --2- 172.21.15.27:0/2539180290 >> v2:172.21.15.93:3300/0 conn(0x7f2c6415b100 0x7f2c64070ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:07.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.945+0000 7f2c69800640 1 -- 172.21.15.27:0/2539180290 >> 172.21.15.27:0/2539180290 conn(0x7f2c640cb700 msgr2=0x7f2c640cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:07.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.945+0000 7f2c69800640 1 -- 172.21.15.27:0/2539180290 shutdown_connections 2024-01-15T18:19:07.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.945+0000 7f2c69800640 1 -- 172.21.15.27:0/2539180290 wait complete. 2024-01-15T18:19:07.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.949+0000 7f2c69800640 1 Processor -- start 2024-01-15T18:19:07.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.949+0000 7f2c69800640 1 -- start start 2024-01-15T18:19:07.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.949+0000 7f2c69800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c64071420 0x7f2c64151ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:07.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.949+0000 7f2c69800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c6415ad20 0x7f2c6414f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:07.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.949+0000 7f2c69800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c6415b100 0x7f2c640b2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:07.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.949+0000 7f2c69800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c64150a20 con 0x7f2c6415b100 2024-01-15T18:19:07.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.949+0000 7f2c69800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c64150d00 con 0x7f2c64071420 2024-01-15T18:19:07.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.949+0000 7f2c69800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c64150fe0 con 0x7f2c6415ad20 2024-01-15T18:19:07.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.949+0000 7f2c63400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c6415b100 0x7f2c640b2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:07.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.949+0000 7f2c63400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c6415b100 0x7f2c640b2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40514/0 (socket says 172.21.15.27:40514) 2024-01-15T18:19:07.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.949+0000 7f2c63400640 1 -- 172.21.15.27:0/2392955254 learned_addr learned my addr 172.21.15.27:0/2392955254 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:19:07.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.949+0000 7f2c62a00640 1 --2- 172.21.15.27:0/2392955254 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c64071420 0x7f2c64151ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:07.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.949+0000 7f2c62000640 1 --2- 172.21.15.27:0/2392955254 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c6415ad20 0x7f2c6414f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:07.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.949+0000 7f2c63400640 1 -- 172.21.15.27:0/2392955254 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c6415ad20 msgr2=0x7f2c6414f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:07.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.949+0000 7f2c63400640 1 --2- 172.21.15.27:0/2392955254 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c6415ad20 0x7f2c6414f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:07.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.949+0000 7f2c63400640 1 -- 172.21.15.27:0/2392955254 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c64071420 msgr2=0x7f2c64151ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:07.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.949+0000 7f2c63400640 1 --2- 172.21.15.27:0/2392955254 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c64071420 0x7f2c64151ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:07.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.949+0000 7f2c63400640 1 -- 172.21.15.27:0/2392955254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5c009a70 con 0x7f2c6415b100 2024-01-15T18:19:07.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.949+0000 7f2c63400640 1 --2- 172.21.15.27:0/2392955254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c6415b100 0x7f2c640b2610 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c0b0040 tx=0x7f2c5c0195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:07.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.949+0000 7f2c5aa00640 1 -- 172.21.15.27:0/2392955254 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c0ad850 con 0x7f2c6415b100 2024-01-15T18:19:07.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.949+0000 7f2c5aa00640 1 -- 172.21.15.27:0/2392955254 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c5c0ad9b0 con 0x7f2c6415b100 2024-01-15T18:19:07.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.949+0000 7f2c69800640 1 -- 172.21.15.27:0/2392955254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c641512c0 con 0x7f2c6415b100 2024-01-15T18:19:07.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.949+0000 7f2c5aa00640 1 -- 172.21.15.27:0/2392955254 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5c0adda0 con 0x7f2c6415b100 2024-01-15T18:19:07.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.949+0000 7f2c69800640 1 -- 172.21.15.27:0/2392955254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c6414f970 con 0x7f2c6415b100 2024-01-15T18:19:07.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.949+0000 7f2c69800640 1 -- 172.21.15.27:0/2392955254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c64152000 con 0x7f2c6415b100 2024-01-15T18:19:07.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.953+0000 7f2c5aa00640 1 -- 172.21.15.27:0/2392955254 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 87690+0+0 (secure 0 0 0) 0x7f2c5c0b4050 con 0x7f2c6415b100 2024-01-15T18:19:07.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.953+0000 7f2c5aa00640 1 --2- 172.21.15.27:0/2392955254 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2c3c0679d0 0x7f2c3c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:07.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.953+0000 7f2c5aa00640 1 -- 172.21.15.27:0/2392955254 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f2c5c11e3d0 con 0x7f2c6415b100 2024-01-15T18:19:07.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.953+0000 7f2c62a00640 1 --2- 172.21.15.27:0/2392955254 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2c3c0679d0 0x7f2c3c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:07.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.953+0000 7f2c62a00640 1 --2- 172.21.15.27:0/2392955254 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2c3c0679d0 0x7f2c3c069e90 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f2c54007fd0 tx=0x7f2c54007d60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:07.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:07.957+0000 7f2c5aa00640 1 -- 172.21.15.27:0/2392955254 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2c5c0ecef0 con 0x7f2c6415b100 2024-01-15T18:19:08.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.141+0000 7f2c69800640 1 -- 172.21.15.27:0/2392955254 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c6404e850 con 0x7f2c3c0679d0 2024-01-15T18:19:08.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.153+0000 7f2c5aa00640 1 -- 172.21.15.27:0/2392955254 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2c6404e850 con 0x7f2c3c0679d0 2024-01-15T18:19:08.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.157+0000 7f2c69800640 1 -- 172.21.15.27:0/2392955254 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2c3c0679d0 msgr2=0x7f2c3c069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:08.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.157+0000 7f2c69800640 1 --2- 172.21.15.27:0/2392955254 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2c3c0679d0 0x7f2c3c069e90 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f2c54007fd0 tx=0x7f2c54007d60 comp rx=0 tx=0).stop 2024-01-15T18:19:08.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.157+0000 7f2c69800640 1 -- 172.21.15.27:0/2392955254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c6415b100 msgr2=0x7f2c640b2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:08.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.157+0000 7f2c69800640 1 --2- 172.21.15.27:0/2392955254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c6415b100 0x7f2c640b2610 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c0b0040 tx=0x7f2c5c0195f0 comp rx=0 tx=0).stop 2024-01-15T18:19:08.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.157+0000 7f2c69800640 1 -- 172.21.15.27:0/2392955254 shutdown_connections 2024-01-15T18:19:08.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.157+0000 7f2c69800640 1 --2- 172.21.15.27:0/2392955254 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2c3c0679d0 0x7f2c3c069e90 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:08.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.157+0000 7f2c69800640 1 --2- 172.21.15.27:0/2392955254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c6415b100 0x7f2c640b2610 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:08.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.157+0000 7f2c69800640 1 --2- 172.21.15.27:0/2392955254 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2c6415ad20 0x7f2c6414f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:08.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.157+0000 7f2c69800640 1 --2- 172.21.15.27:0/2392955254 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2c64071420 0x7f2c64151ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:08.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.157+0000 7f2c69800640 1 -- 172.21.15.27:0/2392955254 >> 172.21.15.27:0/2392955254 conn(0x7f2c640cb700 msgr2=0x7f2c640cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:08.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.157+0000 7f2c69800640 1 -- 172.21.15.27:0/2392955254 shutdown_connections 2024-01-15T18:19:08.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.157+0000 7f2c69800640 1 -- 172.21.15.27:0/2392955254 wait complete. 2024-01-15T18:19:08.177 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:19:08.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.321+0000 7f9264800640 1 Processor -- start 2024-01-15T18:19:08.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.321+0000 7f9264800640 1 -- start start 2024-01-15T18:19:08.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.325+0000 7f9264800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f925c15ad20 0x7f925c15b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:08.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.325+0000 7f9264800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f925c0715f0 0x7f925c15ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:08.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.325+0000 7f9264800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f925c15a610 con 0x7f925c15d900 2024-01-15T18:19:08.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.325+0000 7f9264800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f925c15e3c0 con 0x7f925c0715f0 2024-01-15T18:19:08.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.325+0000 7f9264800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f925c15e6a0 con 0x7f925c15ad20 2024-01-15T18:19:08.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.325+0000 7f9262800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f925c0715f0 0x7f925c15ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:08.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.325+0000 7f9261400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f925c15d900 0x7f925c070ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59450/0 (socket says 172.21.15.27:59450) 2024-01-15T18:19:08.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.325+0000 7f9262800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f925c0715f0 0x7f925c15ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40528/0 (socket says 172.21.15.27:40528) 2024-01-15T18:19:08.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.325+0000 7f9261400640 1 -- 172.21.15.27:0/3465257246 learned_addr learned my addr 172.21.15.27:0/3465257246 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:19:08.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.325+0000 7f9261e00640 1 --2- 172.21.15.27:0/3465257246 >> v2:172.21.15.93:3300/0 conn(0x7f925c15ad20 0x7f925c15b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:08.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.325+0000 7f9262800640 1 -- 172.21.15.27:0/3465257246 >> v2:172.21.15.93:3300/0 conn(0x7f925c15ad20 msgr2=0x7f925c15b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:08.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.325+0000 7f9262800640 1 --2- 172.21.15.27:0/3465257246 >> v2:172.21.15.93:3300/0 conn(0x7f925c15ad20 0x7f925c15b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:08.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.325+0000 7f9262800640 1 -- 172.21.15.27:0/3465257246 >> v1:172.21.15.27:6789/0 conn(0x7f925c15d900 legacy=0x7f925c070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:08.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.325+0000 7f9262800640 1 -- 172.21.15.27:0/3465257246 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f925c15b6e0 con 0x7f925c0715f0 2024-01-15T18:19:08.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.329+0000 7f9262800640 1 --2- 172.21.15.27:0/3465257246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f925c0715f0 0x7f925c15ddb0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f9254009d90 tx=0x7f925401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77f48bd2d03f2748 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:08.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.329+0000 7f9260a00640 1 -- 172.21.15.27:0/3465257246 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92540a59c0 con 0x7f925c0715f0 2024-01-15T18:19:08.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.329+0000 7f9260a00640 1 -- 172.21.15.27:0/3465257246 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92540a5b20 con 0x7f925c0715f0 2024-01-15T18:19:08.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.329+0000 7f9260a00640 1 -- 172.21.15.27:0/3465257246 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92540ad440 con 0x7f925c0715f0 2024-01-15T18:19:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.329+0000 7f9264800640 1 -- 172.21.15.27:0/3465257246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f925c0715f0 msgr2=0x7f925c15ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.329+0000 7f9264800640 1 --2- 172.21.15.27:0/3465257246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f925c0715f0 0x7f925c15ddb0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f9254009d90 tx=0x7f925401c8e0 comp rx=0 tx=0).stop 2024-01-15T18:19:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.329+0000 7f9264800640 1 -- 172.21.15.27:0/3465257246 shutdown_connections 2024-01-15T18:19:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.329+0000 7f9264800640 1 --2- 172.21.15.27:0/3465257246 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f925c0715f0 0x7f925c15ddb0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.329+0000 7f9264800640 1 --2- 172.21.15.27:0/3465257246 >> v2:172.21.15.93:3300/0 conn(0x7f925c15ad20 0x7f925c15b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.329+0000 7f9264800640 1 -- 172.21.15.27:0/3465257246 >> 172.21.15.27:0/3465257246 conn(0x7f925c0cb700 msgr2=0x7f925c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.329+0000 7f9264800640 1 -- 172.21.15.27:0/3465257246 shutdown_connections 2024-01-15T18:19:08.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.329+0000 7f9264800640 1 -- 172.21.15.27:0/3465257246 wait complete. 2024-01-15T18:19:08.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.329+0000 7f9264800640 1 Processor -- start 2024-01-15T18:19:08.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.329+0000 7f9264800640 1 -- start start 2024-01-15T18:19:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f9264800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f925c0715f0 0x7f925c154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f9264800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f925c15ad20 0x7f925c14b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f9264800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f925c15d900 0x7f925c155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f9264800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925c0cdad0 con 0x7f925c15ad20 2024-01-15T18:19:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f9264800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925c155080 con 0x7f925c15d900 2024-01-15T18:19:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f9264800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f925c0b44e0 con 0x7f925c0715f0 2024-01-15T18:19:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f9261400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f925c15ad20 0x7f925c14b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f9261400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f925c15ad20 0x7f925c14b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40542/0 (socket says 172.21.15.27:40542) 2024-01-15T18:19:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f9261400640 1 -- 172.21.15.27:0/1686289308 learned_addr learned my addr 172.21.15.27:0/1686289308 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:19:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f9261e00640 1 --2- 172.21.15.27:0/1686289308 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f925c0715f0 0x7f925c154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f9262800640 1 --2- 172.21.15.27:0/1686289308 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f925c15d900 0x7f925c155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:08.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f9261400640 1 -- 172.21.15.27:0/1686289308 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f925c0715f0 msgr2=0x7f925c154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:08.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f9261400640 1 --2- 172.21.15.27:0/1686289308 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f925c0715f0 0x7f925c154b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:08.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f9261400640 1 -- 172.21.15.27:0/1686289308 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f925c15d900 msgr2=0x7f925c155250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:08.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f9261400640 1 --2- 172.21.15.27:0/1686289308 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f925c15d900 0x7f925c155250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:08.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f9261400640 1 -- 172.21.15.27:0/1686289308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9254009a70 con 0x7f925c15ad20 2024-01-15T18:19:08.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f9262800640 1 --2- 172.21.15.27:0/1686289308 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f925c15d900 0x7f925c155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:19:08.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f9261e00640 1 --2- 172.21.15.27:0/1686289308 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f925c0715f0 0x7f925c154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:19:08.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f9261400640 1 --2- 172.21.15.27:0/1686289308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f925c15ad20 0x7f925c14b060 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f924400c430 tx=0x7f924400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:08.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f925a000640 1 -- 172.21.15.27:0/1686289308 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9244005cf0 con 0x7f925c15ad20 2024-01-15T18:19:08.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f925a000640 1 -- 172.21.15.27:0/1686289308 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9244005e50 con 0x7f925c15ad20 2024-01-15T18:19:08.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f925a000640 1 -- 172.21.15.27:0/1686289308 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92440072d0 con 0x7f925c15ad20 2024-01-15T18:19:08.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f9264800640 1 -- 172.21.15.27:0/1686289308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f925c0b4200 con 0x7f925c15ad20 2024-01-15T18:19:08.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f9264800640 1 -- 172.21.15.27:0/1686289308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f925c0cd6f0 con 0x7f925c15ad20 2024-01-15T18:19:08.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f9264800640 1 -- 172.21.15.27:0/1686289308 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f925c0cdc10 con 0x7f925c15ad20 2024-01-15T18:19:08.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.333+0000 7f925a000640 1 -- 172.21.15.27:0/1686289308 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 87690+0+0 (secure 0 0 0) 0x7f9244007430 con 0x7f925c15ad20 2024-01-15T18:19:08.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.337+0000 7f925a000640 1 --2- 172.21.15.27:0/1686289308 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f92340679d0 0x7f9234069e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:08.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.337+0000 7f925a000640 1 -- 172.21.15.27:0/1686289308 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f92440827a0 con 0x7f925c15ad20 2024-01-15T18:19:08.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.337+0000 7f9261e00640 1 --2- 172.21.15.27:0/1686289308 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f92340679d0 0x7f9234069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:08.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.337+0000 7f9261e00640 1 --2- 172.21.15.27:0/1686289308 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f92340679d0 0x7f9234069e90 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f924c009bb0 tx=0x7f924c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:08.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.341+0000 7f925a000640 1 -- 172.21.15.27:0/1686289308 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f925c0cdc10 con 0x7f925c15ad20 2024-01-15T18:19:08.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.525+0000 7f9264800640 1 -- 172.21.15.27:0/1686289308 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f925c0cdc10 con 0x7f92340679d0 2024-01-15T18:19:08.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.529+0000 7f925a000640 1 -- 172.21.15.27:0/1686289308 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f925c0cdc10 con 0x7f92340679d0 2024-01-15T18:19:08.529 INFO:teuthology.orchestra.run.smithi027.stdout:/volumes/_nogroup/sv1/1a63cb47-42d3-4f34-bbc0-e0bda8e84ca4 2024-01-15T18:19:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.533+0000 7f9264800640 1 -- 172.21.15.27:0/1686289308 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f92340679d0 msgr2=0x7f9234069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.533+0000 7f9264800640 1 --2- 172.21.15.27:0/1686289308 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f92340679d0 0x7f9234069e90 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f924c009bb0 tx=0x7f924c008040 comp rx=0 tx=0).stop 2024-01-15T18:19:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.533+0000 7f9264800640 1 -- 172.21.15.27:0/1686289308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f925c15ad20 msgr2=0x7f925c14b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.533+0000 7f9264800640 1 --2- 172.21.15.27:0/1686289308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f925c15ad20 0x7f925c14b060 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f924400c430 tx=0x7f924400c8f0 comp rx=0 tx=0).stop 2024-01-15T18:19:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.533+0000 7f9264800640 1 -- 172.21.15.27:0/1686289308 shutdown_connections 2024-01-15T18:19:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.533+0000 7f9264800640 1 --2- 172.21.15.27:0/1686289308 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f92340679d0 0x7f9234069e90 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.533+0000 7f9264800640 1 --2- 172.21.15.27:0/1686289308 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f925c15d900 0x7f925c155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:08.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.533+0000 7f9264800640 1 --2- 172.21.15.27:0/1686289308 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f925c15ad20 0x7f925c14b060 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.533+0000 7f9264800640 1 --2- 172.21.15.27:0/1686289308 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f925c0715f0 0x7f925c154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.533+0000 7f9264800640 1 -- 172.21.15.27:0/1686289308 >> 172.21.15.27:0/1686289308 conn(0x7f925c0cb700 msgr2=0x7f925c1595a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.533+0000 7f9264800640 1 -- 172.21.15.27:0/1686289308 shutdown_connections 2024-01-15T18:19:08.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.533+0000 7f9264800640 1 -- 172.21.15.27:0/1686289308 wait complete. 2024-01-15T18:19:08.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:19:08.550 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/1a63cb47-42d3-4f34-bbc0-e0bda8e84ca4) 2024-01-15T18:19:08.565 INFO:teuthology.orchestra.run.smithi027.stdout:# file: volumes/_nogroup/sv1/1a63cb47-42d3-4f34-bbc0-e0bda8e84ca4 2024-01-15T18:19:08.565 INFO:teuthology.orchestra.run.smithi027.stdout:ceph.dir.rbytes="3221225472" 2024-01-15T18:19:08.565 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:19:08.566 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-15T18:19:08.567 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone55241 2024-01-15T18:19:08.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.757+0000 7f9b49400640 1 Processor -- start 2024-01-15T18:19:08.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.757+0000 7f9b49400640 1 -- start start 2024-01-15T18:19:08.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.761+0000 7f9b49400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9b4415d570 0x7f9b4406be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:08.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.761+0000 7f9b49400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b4406b810 0x7f9b4406c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:08.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.761+0000 7f9b49400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b440b0d70 con 0x7f9b4415d190 2024-01-15T18:19:08.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.761+0000 7f9b49400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b440d0540 con 0x7f9b4415d570 2024-01-15T18:19:08.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.761+0000 7f9b49400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b440b0440 con 0x7f9b4406b810 2024-01-15T18:19:08.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.761+0000 7f9b41e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9b4415d570 0x7f9b4406be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:08.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.761+0000 7f9b41e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9b4415d570 0x7f9b4406be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40550/0 (socket says 172.21.15.27:40550) 2024-01-15T18:19:08.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.761+0000 7f9b41e00640 1 -- 172.21.15.27:0/2416801279 learned_addr learned my addr 172.21.15.27:0/2416801279 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:19:08.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.761+0000 7f9b43200640 1 --2- 172.21.15.27:0/2416801279 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b4406b810 0x7f9b4406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:08.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.761+0000 7f9b41e00640 1 -- 172.21.15.27:0/2416801279 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b4406b810 msgr2=0x7f9b4406c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:08.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.761+0000 7f9b41e00640 1 --2- 172.21.15.27:0/2416801279 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b4406b810 0x7f9b4406c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:08.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.761+0000 7f9b41e00640 1 -- 172.21.15.27:0/2416801279 >> v1:172.21.15.93:6789/0 conn(0x7f9b4415d190 legacy=0x7f9b440cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:08.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.761+0000 7f9b41e00640 1 -- 172.21.15.27:0/2416801279 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b440b0720 con 0x7f9b4415d570 2024-01-15T18:19:08.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.765+0000 7f9b41e00640 1 --2- 172.21.15.27:0/2416801279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b4415d570 0x7f9b4406be10 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c009870 tx=0x7f9b2c01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a5813f12fabbb94 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:08.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.765+0000 7f9b41400640 1 -- 172.21.15.27:0/2416801279 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c0a95f0 con 0x7f9b4415d570 2024-01-15T18:19:08.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.765+0000 7f9b41400640 1 -- 172.21.15.27:0/2416801279 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2c0a9750 con 0x7f9b4415d570 2024-01-15T18:19:08.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.765+0000 7f9b41400640 1 -- 172.21.15.27:0/2416801279 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c0a9ae0 con 0x7f9b4415d570 2024-01-15T18:19:08.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.765+0000 7f9b49400640 1 -- 172.21.15.27:0/2416801279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b4415d570 msgr2=0x7f9b4406be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:08.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.765+0000 7f9b49400640 1 --2- 172.21.15.27:0/2416801279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b4415d570 0x7f9b4406be10 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c009870 tx=0x7f9b2c01c510 comp rx=0 tx=0).stop 2024-01-15T18:19:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.765+0000 7f9b49400640 1 -- 172.21.15.27:0/2416801279 shutdown_connections 2024-01-15T18:19:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.765+0000 7f9b49400640 1 --2- 172.21.15.27:0/2416801279 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b4406b810 0x7f9b4406c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.765+0000 7f9b49400640 1 --2- 172.21.15.27:0/2416801279 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b4415d570 0x7f9b4406be10 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.765+0000 7f9b49400640 1 -- 172.21.15.27:0/2416801279 >> 172.21.15.27:0/2416801279 conn(0x7f9b440d0e70 msgr2=0x7f9b440d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.765+0000 7f9b49400640 1 -- 172.21.15.27:0/2416801279 shutdown_connections 2024-01-15T18:19:08.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.765+0000 7f9b49400640 1 -- 172.21.15.27:0/2416801279 wait complete. 2024-01-15T18:19:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.765+0000 7f9b49400640 1 Processor -- start 2024-01-15T18:19:08.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.765+0000 7f9b49400640 1 -- start start 2024-01-15T18:19:08.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.765+0000 7f9b49400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b4406b810 0x7f9b4415b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:08.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.765+0000 7f9b49400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b4415d190 0x7f9b440afa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:08.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.769+0000 7f9b49400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9b4415d570 0x7f9b440c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.769+0000 7f9b49400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b440c5de0 con 0x7f9b4406b810 2024-01-15T18:19:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.769+0000 7f9b49400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b44150c80 con 0x7f9b4415d570 2024-01-15T18:19:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.769+0000 7f9b49400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b44159cd0 con 0x7f9b4415d190 2024-01-15T18:19:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.769+0000 7f9b42800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b4406b810 0x7f9b4415b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.769+0000 7f9b42800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b4406b810 0x7f9b4415b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40560/0 (socket says 172.21.15.27:40560) 2024-01-15T18:19:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.769+0000 7f9b42800640 1 -- 172.21.15.27:0/3888934274 learned_addr learned my addr 172.21.15.27:0/3888934274 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:19:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.769+0000 7f9b41e00640 1 --2- 172.21.15.27:0/3888934274 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b4415d190 0x7f9b440afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.769+0000 7f9b43200640 1 --2- 172.21.15.27:0/3888934274 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9b4415d570 0x7f9b440c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.769+0000 7f9b42800640 1 -- 172.21.15.27:0/3888934274 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b4415d190 msgr2=0x7f9b440afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.769+0000 7f9b42800640 1 --2- 172.21.15.27:0/3888934274 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b4415d190 0x7f9b440afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.769+0000 7f9b42800640 1 -- 172.21.15.27:0/3888934274 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9b4415d570 msgr2=0x7f9b440c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.769+0000 7f9b42800640 1 --2- 172.21.15.27:0/3888934274 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9b4415d570 0x7f9b440c48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.769+0000 7f9b42800640 1 -- 172.21.15.27:0/3888934274 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b2c0094d0 con 0x7f9b4406b810 2024-01-15T18:19:08.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.769+0000 7f9b42800640 1 --2- 172.21.15.27:0/3888934274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b4406b810 0x7f9b4415b8f0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f9b3400c500 tx=0x7f9b3400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:08.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.769+0000 7f9b3aa00640 1 -- 172.21.15.27:0/3888934274 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b34005220 con 0x7f9b4406b810 2024-01-15T18:19:08.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.769+0000 7f9b3aa00640 1 -- 172.21.15.27:0/3888934274 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b34005380 con 0x7f9b4406b810 2024-01-15T18:19:08.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.769+0000 7f9b49400640 1 -- 172.21.15.27:0/3888934274 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b44159fb0 con 0x7f9b4406b810 2024-01-15T18:19:08.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.769+0000 7f9b3aa00640 1 -- 172.21.15.27:0/3888934274 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b34007270 con 0x7f9b4406b810 2024-01-15T18:19:08.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.769+0000 7f9b49400640 1 -- 172.21.15.27:0/3888934274 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b440c5a00 con 0x7f9b4406b810 2024-01-15T18:19:08.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.769+0000 7f9b49400640 1 -- 172.21.15.27:0/3888934274 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b4415a4b0 con 0x7f9b4406b810 2024-01-15T18:19:08.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.769+0000 7f9b3aa00640 1 -- 172.21.15.27:0/3888934274 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 87690+0+0 (secure 0 0 0) 0x7f9b340073d0 con 0x7f9b4406b810 2024-01-15T18:19:08.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.773+0000 7f9b3aa00640 1 --2- 172.21.15.27:0/3888934274 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9b1c0679d0 0x7f9b1c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:08.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.773+0000 7f9b3aa00640 1 -- 172.21.15.27:0/3888934274 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f9b34082310 con 0x7f9b4406b810 2024-01-15T18:19:08.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.773+0000 7f9b41e00640 1 --2- 172.21.15.27:0/3888934274 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9b1c0679d0 0x7f9b1c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:08.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.773+0000 7f9b41e00640 1 --2- 172.21.15.27:0/3888934274 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9b1c0679d0 0x7f9b1c069e90 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c0ae000 tx=0x7f9b2c0a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:08.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.777+0000 7f9b3aa00640 1 -- 172.21.15.27:0/3888934274 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9b34050fd0 con 0x7f9b4406b810 2024-01-15T18:19:08.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:08.961+0000 7f9b49400640 1 -- 172.21.15.27:0/3888934274 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone55241", "target": ["mon-mgr", ""]}) v1 -- 0x7f9b44067880 con 0x7f9b1c0679d0 2024-01-15T18:19:09.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.013+0000 7f9b3aa00640 1 -- 172.21.15.27:0/3888934274 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9b44067880 con 0x7f9b1c0679d0 2024-01-15T18:19:09.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.013+0000 7f9b49400640 1 -- 172.21.15.27:0/3888934274 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9b1c0679d0 msgr2=0x7f9b1c069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:09.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.013+0000 7f9b49400640 1 --2- 172.21.15.27:0/3888934274 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9b1c0679d0 0x7f9b1c069e90 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c0ae000 tx=0x7f9b2c0a7000 comp rx=0 tx=0).stop 2024-01-15T18:19:09.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.013+0000 7f9b49400640 1 -- 172.21.15.27:0/3888934274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b4406b810 msgr2=0x7f9b4415b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:09.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.013+0000 7f9b49400640 1 --2- 172.21.15.27:0/3888934274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b4406b810 0x7f9b4415b8f0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f9b3400c500 tx=0x7f9b3400c9c0 comp rx=0 tx=0).stop 2024-01-15T18:19:09.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.017+0000 7f9b49400640 1 -- 172.21.15.27:0/3888934274 shutdown_connections 2024-01-15T18:19:09.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.017+0000 7f9b49400640 1 --2- 172.21.15.27:0/3888934274 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9b1c0679d0 0x7f9b1c069e90 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:09.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.017+0000 7f9b49400640 1 --2- 172.21.15.27:0/3888934274 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9b4415d570 0x7f9b440c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:09.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.017+0000 7f9b49400640 1 --2- 172.21.15.27:0/3888934274 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9b4415d190 0x7f9b440afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:09.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.017+0000 7f9b49400640 1 --2- 172.21.15.27:0/3888934274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b4406b810 0x7f9b4415b8f0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:09.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.017+0000 7f9b49400640 1 -- 172.21.15.27:0/3888934274 >> 172.21.15.27:0/3888934274 conn(0x7f9b440d0e70 msgr2=0x7f9b440b15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:09.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.017+0000 7f9b49400640 1 -- 172.21.15.27:0/3888934274 shutdown_connections 2024-01-15T18:19:09.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.017+0000 7f9b49400640 1 -- 172.21.15.27:0/3888934274 wait complete. 2024-01-15T18:19:09.019 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:19:09.018+0000 7f59f8800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:19:09.019 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:19:09.018+0000 7f59f8800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:19:09.019 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:19:09.018+0000 7f59f8800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:19:09.019 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:19:09.018+0000 7f59f8800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:19:09.019 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:19:09.018+0000 7f59f8800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:19:09.033 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:19:09.034+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:19:09.033 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:19:09.034+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:19:09.034 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:19:09.034+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:19:09.034 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:19:09.034+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:19:09.034 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:19:09.034+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:19:09.035 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone55242 2024-01-15T18:19:09.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.181+0000 7f3233600640 1 Processor -- start 2024-01-15T18:19:09.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.181+0000 7f3233600640 1 -- start start 2024-01-15T18:19:09.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.185+0000 7f3233600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f322c0cf460 0x7f322c0cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.185+0000 7f3233600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f322c0b2590 0x7f322c0cc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.185+0000 7f3233600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f322c0cfd80 con 0x7f322c15b4b0 2024-01-15T18:19:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.185+0000 7f3233600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f322c0d19a0 con 0x7f322c0cf460 2024-01-15T18:19:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.185+0000 7f3233600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f322c0d1c80 con 0x7f322c0b2590 2024-01-15T18:19:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.185+0000 7f3231600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f322c0b2590 0x7f322c0cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.185+0000 7f3230c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f322c0cf460 0x7f322c0cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.185+0000 7f322be00640 1 --1- >> v1:172.21.15.93:6789/0 conn(0x7f322c15b4b0 0x7f322c0cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:42666/0 (socket says 172.21.15.27:42666) 2024-01-15T18:19:09.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.185+0000 7f322be00640 1 -- 172.21.15.27:0/273169911 learned_addr learned my addr 172.21.15.27:0/273169911 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:19:09.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.185+0000 7f322b400640 1 -- 172.21.15.27:0/273169911 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658984844 0 0) 0x7f322c0cfd80 con 0x7f322c15b4b0 2024-01-15T18:19:09.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.185+0000 7f322b400640 1 -- 172.21.15.27:0/273169911 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f320c003530 con 0x7f322c15b4b0 2024-01-15T18:19:09.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.185+0000 7f3230c00640 1 -- 172.21.15.27:0/273169911 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f322c0b2590 msgr2=0x7f322c0cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:09.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.185+0000 7f3230c00640 1 --2- 172.21.15.27:0/273169911 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f322c0b2590 0x7f322c0cc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:09.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.185+0000 7f3230c00640 1 -- 172.21.15.27:0/273169911 >> v1:172.21.15.93:6789/0 conn(0x7f322c15b4b0 legacy=0x7f322c0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:09.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.185+0000 7f3230c00640 1 -- 172.21.15.27:0/273169911 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c0b07c0 con 0x7f322c0cf460 2024-01-15T18:19:09.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.185+0000 7f3231600640 1 --2- 172.21.15.27:0/273169911 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f322c0b2590 0x7f322c0cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:19:09.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.189+0000 7f3230c00640 1 --2- 172.21.15.27:0/273169911 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f322c0cf460 0x7f322c0cf840 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f321c0097c0 tx=0x7f321c01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=637cf440331d08cb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:09.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.189+0000 7f322b400640 1 -- 172.21.15.27:0/273169911 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c0ab680 con 0x7f322c0cf460 2024-01-15T18:19:09.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.189+0000 7f322b400640 1 -- 172.21.15.27:0/273169911 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321c0ab7e0 con 0x7f322c0cf460 2024-01-15T18:19:09.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.189+0000 7f322b400640 1 -- 172.21.15.27:0/273169911 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c0abb50 con 0x7f322c0cf460 2024-01-15T18:19:09.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.189+0000 7f3233600640 1 -- 172.21.15.27:0/273169911 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f322c0cf460 msgr2=0x7f322c0cf840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:09.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.189+0000 7f3233600640 1 --2- 172.21.15.27:0/273169911 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f322c0cf460 0x7f322c0cf840 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f321c0097c0 tx=0x7f321c01c480 comp rx=0 tx=0).stop 2024-01-15T18:19:09.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.189+0000 7f3233600640 1 -- 172.21.15.27:0/273169911 shutdown_connections 2024-01-15T18:19:09.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.189+0000 7f3233600640 1 --2- 172.21.15.27:0/273169911 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f322c0b2590 0x7f322c0cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:09.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.189+0000 7f3233600640 1 --2- 172.21.15.27:0/273169911 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f322c0cf460 0x7f322c0cf840 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:09.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.189+0000 7f3233600640 1 -- 172.21.15.27:0/273169911 >> 172.21.15.27:0/273169911 conn(0x7f322c0d0e80 msgr2=0x7f322c0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:09.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.189+0000 7f3233600640 1 -- 172.21.15.27:0/273169911 shutdown_connections 2024-01-15T18:19:09.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.189+0000 7f3233600640 1 -- 172.21.15.27:0/273169911 wait complete. 2024-01-15T18:19:09.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.189+0000 7f3233600640 1 Processor -- start 2024-01-15T18:19:09.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.189+0000 7f3233600640 1 -- start start 2024-01-15T18:19:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f3233600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f322c0b2590 0x7f322c164dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f3233600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f322c0cf460 0x7f322c163aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f3233600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f322c15b4b0 0x7f322c159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f3233600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c163680 con 0x7f322c15b4b0 2024-01-15T18:19:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f3233600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c165c60 con 0x7f322c0cf460 2024-01-15T18:19:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f3233600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f322c1656a0 con 0x7f322c0b2590 2024-01-15T18:19:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f3231600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f322c15b4b0 0x7f322c159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f322be00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f322c0cf460 0x7f322c163aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f3230c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f322c0b2590 0x7f322c164dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f322be00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f322c0cf460 0x7f322c163aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:46170/0 (socket says 172.21.15.27:46170) 2024-01-15T18:19:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f322be00640 1 -- 172.21.15.27:0/223194223 learned_addr learned my addr 172.21.15.27:0/223194223 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:19:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f3231600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f322c15b4b0 0x7f322c159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40568/0 (socket says 172.21.15.27:40568) 2024-01-15T18:19:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f3230c00640 1 -- 172.21.15.27:0/223194223 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f322c0cf460 msgr2=0x7f322c163aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f3230c00640 1 --2- 172.21.15.27:0/223194223 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f322c0cf460 0x7f322c163aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f3230c00640 1 -- 172.21.15.27:0/223194223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f322c15b4b0 msgr2=0x7f322c159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f3230c00640 1 --2- 172.21.15.27:0/223194223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f322c15b4b0 0x7f322c159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f3230c00640 1 -- 172.21.15.27:0/223194223 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3224009a70 con 0x7f322c0b2590 2024-01-15T18:19:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f322be00640 1 --2- 172.21.15.27:0/223194223 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f322c0cf460 0x7f322c163aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:19:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f3231600640 1 --2- 172.21.15.27:0/223194223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f322c15b4b0 0x7f322c159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:19:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f3230c00640 1 --2- 172.21.15.27:0/223194223 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f322c0b2590 0x7f322c164dc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f321c0ac000 tx=0x7f321c01acc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f3228c00640 1 -- 172.21.15.27:0/223194223 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c0bf610 con 0x7f322c0b2590 2024-01-15T18:19:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f3228c00640 1 -- 172.21.15.27:0/223194223 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321c0bf770 con 0x7f322c0b2590 2024-01-15T18:19:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f3233600640 1 -- 172.21.15.27:0/223194223 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f321c009420 con 0x7f322c0b2590 2024-01-15T18:19:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f3228c00640 1 -- 172.21.15.27:0/223194223 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c0bfae0 con 0x7f322c0b2590 2024-01-15T18:19:09.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f3233600640 1 -- 172.21.15.27:0/223194223 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f322c1632a0 con 0x7f322c0b2590 2024-01-15T18:19:09.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.193+0000 7f3233600640 1 -- 172.21.15.27:0/223194223 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f322c04e850 con 0x7f322c0b2590 2024-01-15T18:19:09.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.197+0000 7f3228c00640 1 -- 172.21.15.27:0/223194223 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 39) v1 ==== 87690+0+0 (secure 0 0 0) 0x7f321c0bfc40 con 0x7f322c0b2590 2024-01-15T18:19:09.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.197+0000 7f3228c00640 1 --2- 172.21.15.27:0/223194223 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f32040679d0 0x7f3204069e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:09.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.197+0000 7f3228c00640 1 -- 172.21.15.27:0/223194223 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f321c11dcd0 con 0x7f322c0b2590 2024-01-15T18:19:09.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.197+0000 7f322be00640 1 --2- 172.21.15.27:0/223194223 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f32040679d0 0x7f3204069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:09.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.197+0000 7f322be00640 1 --2- 172.21.15.27:0/223194223 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f32040679d0 0x7f3204069e90 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f322c1f9910 tx=0x7f3214009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:09.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.201+0000 7f3228c00640 1 -- 172.21.15.27:0/223194223 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f321c0eca10 con 0x7f322c0b2590 2024-01-15T18:19:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.389+0000 7f3233600640 1 -- 172.21.15.27:0/223194223 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone55242", "target": ["mon-mgr", ""]}) v1 -- 0x7f322c0cf240 con 0x7f32040679d0 2024-01-15T18:19:09.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.437+0000 7f3228c00640 1 -- 172.21.15.27:0/223194223 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f322c0cf240 con 0x7f32040679d0 2024-01-15T18:19:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.441+0000 7f3233600640 1 -- 172.21.15.27:0/223194223 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f32040679d0 msgr2=0x7f3204069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.441+0000 7f3233600640 1 --2- 172.21.15.27:0/223194223 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f32040679d0 0x7f3204069e90 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f322c1f9910 tx=0x7f3214009040 comp rx=0 tx=0).stop 2024-01-15T18:19:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.441+0000 7f3233600640 1 -- 172.21.15.27:0/223194223 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f322c0b2590 msgr2=0x7f322c164dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.441+0000 7f3233600640 1 --2- 172.21.15.27:0/223194223 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f322c0b2590 0x7f322c164dc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f321c0ac000 tx=0x7f321c01acc0 comp rx=0 tx=0).stop 2024-01-15T18:19:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.441+0000 7f3233600640 1 -- 172.21.15.27:0/223194223 shutdown_connections 2024-01-15T18:19:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.441+0000 7f3233600640 1 --2- 172.21.15.27:0/223194223 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f32040679d0 0x7f3204069e90 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.441+0000 7f3233600640 1 --2- 172.21.15.27:0/223194223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f322c15b4b0 0x7f322c159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.441+0000 7f3233600640 1 --2- 172.21.15.27:0/223194223 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f322c0cf460 0x7f322c163aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.441+0000 7f3233600640 1 --2- 172.21.15.27:0/223194223 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f322c0b2590 0x7f322c164dc0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.441+0000 7f3233600640 1 -- 172.21.15.27:0/223194223 >> 172.21.15.27:0/223194223 conn(0x7f322c0d0e80 msgr2=0x7f322c0aee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.441+0000 7f3233600640 1 -- 172.21.15.27:0/223194223 shutdown_connections 2024-01-15T18:19:09.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.441+0000 7f3233600640 1 -- 172.21.15.27:0/223194223 wait complete. 2024-01-15T18:19:09.461 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone55243 2024-01-15T18:19:09.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.609+0000 7f3d0f400640 1 Processor -- start 2024-01-15T18:19:09.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.609+0000 7f3d0f400640 1 -- start start 2024-01-15T18:19:09.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.609+0000 7f3d0f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3d0815d3b0 0x7f3d080cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:09.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.609+0000 7f3d0f400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d080b04b0 con 0x7f3d0806b600 2024-01-15T18:19:09.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.609+0000 7f3d0f400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d080d0160 con 0x7f3d0806c3f0 2024-01-15T18:19:09.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.609+0000 7f3d0f400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d080b0720 con 0x7f3d0815d3b0 2024-01-15T18:19:09.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.609+0000 7f3d0ca00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3d0815d3b0 0x7f3d080cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:09.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.609+0000 7f3d07e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3d0806b600 0x7f3d0806bce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59454/0 (socket says 172.21.15.27:59454) 2024-01-15T18:19:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.609+0000 7f3d07e00640 1 -- 172.21.15.27:0/2691050917 learned_addr learned my addr 172.21.15.27:0/2691050917 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:19:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.609+0000 7f3d07400640 1 -- 172.21.15.27:0/2691050917 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3559695859 0 0) 0x7f3d080b04b0 con 0x7f3d0806b600 2024-01-15T18:19:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.609+0000 7f3d07400640 1 -- 172.21.15.27:0/2691050917 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ce8003530 con 0x7f3d0806b600 2024-01-15T18:19:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.609+0000 7f3d0ca00640 1 -- 172.21.15.27:0/2691050917 >> v1:172.21.15.93:6789/0 conn(0x7f3d0806c3f0 legacy=0x7f3d080b1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.609+0000 7f3d0ca00640 1 -- 172.21.15.27:0/2691050917 >> v1:172.21.15.27:6789/0 conn(0x7f3d0806b600 legacy=0x7f3d0806bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.609+0000 7f3d0ca00640 1 -- 172.21.15.27:0/2691050917 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d080b0a00 con 0x7f3d0815d3b0 2024-01-15T18:19:09.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.613+0000 7f3d0ca00640 1 --2- 172.21.15.27:0/2691050917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d0815d3b0 0x7f3d080cfc20 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f3cf80097c0 tx=0x7f3cf801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7df5133151abaa41 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:09.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.613+0000 7f3d07400640 1 -- 172.21.15.27:0/2691050917 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf80ab5f0 con 0x7f3d0815d3b0 2024-01-15T18:19:09.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.613+0000 7f3d07400640 1 -- 172.21.15.27:0/2691050917 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf80ab750 con 0x7f3d0815d3b0 2024-01-15T18:19:09.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.613+0000 7f3d07400640 1 -- 172.21.15.27:0/2691050917 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf80abae0 con 0x7f3d0815d3b0 2024-01-15T18:19:09.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.613+0000 7f3d0f400640 1 -- 172.21.15.27:0/2691050917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d0815d3b0 msgr2=0x7f3d080cfc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:09.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.613+0000 7f3d0f400640 1 --2- 172.21.15.27:0/2691050917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d0815d3b0 0x7f3d080cfc20 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f3cf80097c0 tx=0x7f3cf801c450 comp rx=0 tx=0).stop 2024-01-15T18:19:09.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.613+0000 7f3d0f400640 1 -- 172.21.15.27:0/2691050917 shutdown_connections 2024-01-15T18:19:09.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.613+0000 7f3d0f400640 1 --2- 172.21.15.27:0/2691050917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d0815d3b0 0x7f3d080cfc20 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:09.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.613+0000 7f3d0f400640 1 -- 172.21.15.27:0/2691050917 >> 172.21.15.27:0/2691050917 conn(0x7f3d080d0ed0 msgr2=0x7f3d080d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:09.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.613+0000 7f3d0f400640 1 -- 172.21.15.27:0/2691050917 shutdown_connections 2024-01-15T18:19:09.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.613+0000 7f3d0f400640 1 -- 172.21.15.27:0/2691050917 wait complete. 2024-01-15T18:19:09.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.617+0000 7f3d0f400640 1 Processor -- start 2024-01-15T18:19:09.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.617+0000 7f3d0f400640 1 -- start start 2024-01-15T18:19:09.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.617+0000 7f3d0f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d0806b600 0x7f3d0814ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:09.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.617+0000 7f3d0f400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3d0806c3f0 0x7f3d08153780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:09.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.617+0000 7f3d0f400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3d0815d3b0 0x7f3d08153cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.617+0000 7f3d0f400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d08151a40 con 0x7f3d0806b600 2024-01-15T18:19:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.617+0000 7f3d0f400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0814f1b0 con 0x7f3d0815d3b0 2024-01-15T18:19:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.617+0000 7f3d0f400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d080c7320 con 0x7f3d0806c3f0 2024-01-15T18:19:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.617+0000 7f3d0ca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d0806b600 0x7f3d0814ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.617+0000 7f3d0ca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d0806b600 0x7f3d0814ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40590/0 (socket says 172.21.15.27:40590) 2024-01-15T18:19:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.617+0000 7f3d0ca00640 1 -- 172.21.15.27:0/1919393758 learned_addr learned my addr 172.21.15.27:0/1919393758 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:19:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.617+0000 7f3d07e00640 1 --2- 172.21.15.27:0/1919393758 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3d0806c3f0 0x7f3d08153780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.617+0000 7f3d0d400640 1 --2- 172.21.15.27:0/1919393758 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3d0815d3b0 0x7f3d08153cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.617+0000 7f3d07e00640 1 -- 172.21.15.27:0/1919393758 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3d0815d3b0 msgr2=0x7f3d08153cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.617+0000 7f3d07e00640 1 --2- 172.21.15.27:0/1919393758 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3d0815d3b0 0x7f3d08153cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.617+0000 7f3d07e00640 1 -- 172.21.15.27:0/1919393758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d0806b600 msgr2=0x7f3d0814ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.617+0000 7f3d07e00640 1 --2- 172.21.15.27:0/1919393758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d0806b600 0x7f3d0814ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.617+0000 7f3d07e00640 1 -- 172.21.15.27:0/1919393758 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf8009420 con 0x7f3d0806c3f0 2024-01-15T18:19:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.617+0000 7f3d07e00640 1 --2- 172.21.15.27:0/1919393758 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3d0806c3f0 0x7f3d08153780 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0001ec0 tx=0x7f3cf000c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.617+0000 7f3d04c00640 1 -- 172.21.15.27:0/1919393758 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf0005cf0 con 0x7f3d0806c3f0 2024-01-15T18:19:09.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.617+0000 7f3d04c00640 1 -- 172.21.15.27:0/1919393758 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf0005e50 con 0x7f3d0806c3f0 2024-01-15T18:19:09.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.617+0000 7f3d04c00640 1 -- 172.21.15.27:0/1919393758 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf00072d0 con 0x7f3d0806c3f0 2024-01-15T18:19:09.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.617+0000 7f3d0f400640 1 -- 172.21.15.27:0/1919393758 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d0814f450 con 0x7f3d0806c3f0 2024-01-15T18:19:09.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.617+0000 7f3d0f400640 1 -- 172.21.15.27:0/1919393758 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d0814fa10 con 0x7f3d0806c3f0 2024-01-15T18:19:09.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.621+0000 7f3d0f400640 1 -- 172.21.15.27:0/1919393758 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d0804e850 con 0x7f3d0806c3f0 2024-01-15T18:19:09.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.621+0000 7f3d04c00640 1 -- 172.21.15.27:0/1919393758 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 39) v1 ==== 87690+0+0 (secure 0 0 0) 0x7f3cf0007430 con 0x7f3d0806c3f0 2024-01-15T18:19:09.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.621+0000 7f3d04c00640 1 --2- 172.21.15.27:0/1919393758 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3ce00679d0 0x7f3ce0069e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:09.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.621+0000 7f3d04c00640 1 -- 172.21.15.27:0/1919393758 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f3cf00827a0 con 0x7f3d0806c3f0 2024-01-15T18:19:09.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.621+0000 7f3d0ca00640 1 --2- 172.21.15.27:0/1919393758 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3ce00679d0 0x7f3ce0069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:09.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.621+0000 7f3d0ca00640 1 --2- 172.21.15.27:0/1919393758 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3ce00679d0 0x7f3ce0069e90 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f3cf801c490 tx=0x7f3cf80a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:09.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.625+0000 7f3d04c00640 1 -- 172.21.15.27:0/1919393758 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3cf00511a0 con 0x7f3d0806c3f0 2024-01-15T18:19:09.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:09.813+0000 7f3d0f400640 1 -- 172.21.15.27:0/1919393758 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone55243", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d080cfcc0 con 0x7f3ce00679d0 2024-01-15T18:19:11.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:11.201+0000 7f3d04c00640 1 -- 172.21.15.27:0/1919393758 <== mon.2 v2:172.21.15.93:3301/0 7 ==== mgrmap(e 40) v1 ==== 87792+0+0 (secure 0 0 0) 0x7f3cf0051570 con 0x7f3d0806c3f0 2024-01-15T18:19:13.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.380+0000 7f3d04c00640 1 -- 172.21.15.27:0/1919393758 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3d080cfcc0 con 0x7f3ce00679d0 2024-01-15T18:19:13.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.384+0000 7f3d0f400640 1 -- 172.21.15.27:0/1919393758 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3ce00679d0 msgr2=0x7f3ce0069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:13.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.384+0000 7f3d0f400640 1 --2- 172.21.15.27:0/1919393758 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3ce00679d0 0x7f3ce0069e90 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f3cf801c490 tx=0x7f3cf80a7000 comp rx=0 tx=0).stop 2024-01-15T18:19:13.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.384+0000 7f3d0f400640 1 -- 172.21.15.27:0/1919393758 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3d0806c3f0 msgr2=0x7f3d08153780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:13.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.384+0000 7f3d0f400640 1 --2- 172.21.15.27:0/1919393758 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3d0806c3f0 0x7f3d08153780 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0001ec0 tx=0x7f3cf000c820 comp rx=0 tx=0).stop 2024-01-15T18:19:13.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.384+0000 7f3d0f400640 1 -- 172.21.15.27:0/1919393758 shutdown_connections 2024-01-15T18:19:13.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.384+0000 7f3d0f400640 1 --2- 172.21.15.27:0/1919393758 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3ce00679d0 0x7f3ce0069e90 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:13.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.384+0000 7f3d0f400640 1 --2- 172.21.15.27:0/1919393758 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3d0815d3b0 0x7f3d08153cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:13.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.384+0000 7f3d0f400640 1 --2- 172.21.15.27:0/1919393758 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3d0806c3f0 0x7f3d08153780 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:13.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.384+0000 7f3d0f400640 1 --2- 172.21.15.27:0/1919393758 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d0806b600 0x7f3d0814ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:13.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.384+0000 7f3d0f400640 1 -- 172.21.15.27:0/1919393758 >> 172.21.15.27:0/1919393758 conn(0x7f3d080d0ed0 msgr2=0x7f3d08154ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:13.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.384+0000 7f3d0f400640 1 -- 172.21.15.27:0/1919393758 shutdown_connections 2024-01-15T18:19:13.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.384+0000 7f3d0f400640 1 -- 172.21.15.27:0/1919393758 wait complete. 2024-01-15T18:19:13.410 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-15T18:19:13.411 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone55243 2024-01-15T18:19:13.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.560+0000 7f2a30800640 1 Processor -- start 2024-01-15T18:19:13.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.560+0000 7f2a30800640 1 -- start start 2024-01-15T18:19:13.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.560+0000 7f2a30800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2a280cf100 0x7f2a280cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:13.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.560+0000 7f2a30800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a280ad920 con 0x7f2a281468d0 2024-01-15T18:19:13.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.560+0000 7f2a30800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a280ac880 con 0x7f2a280ccea0 2024-01-15T18:19:13.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.560+0000 7f2a30800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a280acb60 con 0x7f2a280cf100 2024-01-15T18:19:13.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.560+0000 7f2a2d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2a280cf100 0x7f2a280cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:13.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.560+0000 7f2a2de00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2a281468d0 0x7f2a280ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:59462/0 (socket says 172.21.15.27:59462) 2024-01-15T18:19:13.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.560+0000 7f2a2d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2a280cf100 0x7f2a280cc960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40602/0 (socket says 172.21.15.27:40602) 2024-01-15T18:19:13.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.560+0000 7f2a2de00640 1 -- 172.21.15.27:0/1572942435 learned_addr learned my addr 172.21.15.27:0/1572942435 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:19:13.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.560+0000 7f2a2d400640 1 -- 172.21.15.27:0/1572942435 >> v1:172.21.15.93:6789/0 conn(0x7f2a280ccea0 legacy=0x7f2a2806c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:13.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.560+0000 7f2a2d400640 1 -- 172.21.15.27:0/1572942435 >> v1:172.21.15.27:6789/0 conn(0x7f2a281468d0 legacy=0x7f2a280ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:13.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.560+0000 7f2a2d400640 1 -- 172.21.15.27:0/1572942435 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a280ace40 con 0x7f2a280cf100 2024-01-15T18:19:13.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.564+0000 7f2a2d400640 1 --2- 172.21.15.27:0/1572942435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a280cf100 0x7f2a280cc960 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f2a100097c0 tx=0x7f2a1001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e5adce7c54ef3da server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:13.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.564+0000 7f2a2ca00640 1 -- 172.21.15.27:0/1572942435 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a100ab610 con 0x7f2a280cf100 2024-01-15T18:19:13.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.564+0000 7f2a2ca00640 1 -- 172.21.15.27:0/1572942435 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a100ab770 con 0x7f2a280cf100 2024-01-15T18:19:13.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.564+0000 7f2a2ca00640 1 -- 172.21.15.27:0/1572942435 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a100abb00 con 0x7f2a280cf100 2024-01-15T18:19:13.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.564+0000 7f2a30800640 1 -- 172.21.15.27:0/1572942435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a280cf100 msgr2=0x7f2a280cc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:13.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.564+0000 7f2a30800640 1 --2- 172.21.15.27:0/1572942435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a280cf100 0x7f2a280cc960 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f2a100097c0 tx=0x7f2a1001c410 comp rx=0 tx=0).stop 2024-01-15T18:19:13.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.564+0000 7f2a30800640 1 -- 172.21.15.27:0/1572942435 shutdown_connections 2024-01-15T18:19:13.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.564+0000 7f2a30800640 1 --2- 172.21.15.27:0/1572942435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a280cf100 0x7f2a280cc960 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:13.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.564+0000 7f2a30800640 1 -- 172.21.15.27:0/1572942435 >> 172.21.15.27:0/1572942435 conn(0x7f2a280cb700 msgr2=0x7f2a280cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:13.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.564+0000 7f2a30800640 1 -- 172.21.15.27:0/1572942435 shutdown_connections 2024-01-15T18:19:13.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.564+0000 7f2a30800640 1 -- 172.21.15.27:0/1572942435 wait complete. 2024-01-15T18:19:13.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.568+0000 7f2a30800640 1 Processor -- start 2024-01-15T18:19:13.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.568+0000 7f2a30800640 1 -- start start 2024-01-15T18:19:13.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.568+0000 7f2a30800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a280ccea0 0x7f2a281384a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:13.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.568+0000 7f2a30800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a280cf100 0x7f2a2815aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:13.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.568+0000 7f2a30800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2a281468d0 0x7f2a280afa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:13.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.568+0000 7f2a30800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a28139d20 con 0x7f2a280cf100 2024-01-15T18:19:13.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.568+0000 7f2a30800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a281389e0 con 0x7f2a280ccea0 2024-01-15T18:19:13.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.568+0000 7f2a30800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a28138c80 con 0x7f2a281468d0 2024-01-15T18:19:13.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.568+0000 7f2a2d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a280cf100 0x7f2a2815aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:13.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.568+0000 7f2a2de00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a280ccea0 0x7f2a281384a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:13.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.568+0000 7f2a2d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a280cf100 0x7f2a2815aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40610/0 (socket says 172.21.15.27:40610) 2024-01-15T18:19:13.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.568+0000 7f2a2d400640 1 -- 172.21.15.27:0/125274053 learned_addr learned my addr 172.21.15.27:0/125274053 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:19:13.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.568+0000 7f2a2e800640 1 --2- 172.21.15.27:0/125274053 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2a281468d0 0x7f2a280afa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:13.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.568+0000 7f2a2d400640 1 -- 172.21.15.27:0/125274053 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2a281468d0 msgr2=0x7f2a280afa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:13.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.568+0000 7f2a2d400640 1 --2- 172.21.15.27:0/125274053 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2a281468d0 0x7f2a280afa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:13.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.568+0000 7f2a2d400640 1 -- 172.21.15.27:0/125274053 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a280ccea0 msgr2=0x7f2a281384a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:13.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.568+0000 7f2a2d400640 1 --2- 172.21.15.27:0/125274053 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a280ccea0 0x7f2a281384a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:13.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.568+0000 7f2a2d400640 1 -- 172.21.15.27:0/125274053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a10009420 con 0x7f2a280cf100 2024-01-15T18:19:13.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.568+0000 7f2a2d400640 1 --2- 172.21.15.27:0/125274053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a280cf100 0x7f2a2815aa60 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f2a100ad040 tx=0x7f2a1001ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:13.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.568+0000 7f2a26000640 1 -- 172.21.15.27:0/125274053 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a100b5810 con 0x7f2a280cf100 2024-01-15T18:19:13.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.568+0000 7f2a26000640 1 -- 172.21.15.27:0/125274053 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a100b5990 con 0x7f2a280cf100 2024-01-15T18:19:13.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.568+0000 7f2a30800640 1 -- 172.21.15.27:0/125274053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a28138f60 con 0x7f2a280cf100 2024-01-15T18:19:13.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.568+0000 7f2a26000640 1 -- 172.21.15.27:0/125274053 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a100b5d40 con 0x7f2a280cf100 2024-01-15T18:19:13.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.568+0000 7f2a30800640 1 -- 172.21.15.27:0/125274053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a28139520 con 0x7f2a280cf100 2024-01-15T18:19:13.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.572+0000 7f2a30800640 1 -- 172.21.15.27:0/125274053 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a28142ba0 con 0x7f2a280cf100 2024-01-15T18:19:13.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.572+0000 7f2a26000640 1 -- 172.21.15.27:0/125274053 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 40) v1 ==== 87792+0+0 (secure 0 0 0) 0x7f2a100bf410 con 0x7f2a280cf100 2024-01-15T18:19:13.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.572+0000 7f2a26000640 1 --2- 172.21.15.27:0/125274053 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2a00067af0 0x7f2a00069fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:13.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.572+0000 7f2a26000640 1 -- 172.21.15.27:0/125274053 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f2a1011dd60 con 0x7f2a280cf100 2024-01-15T18:19:13.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.572+0000 7f2a2de00640 1 --2- 172.21.15.27:0/125274053 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2a00067af0 0x7f2a00069fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:13.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.572+0000 7f2a2de00640 1 --2- 172.21.15.27:0/125274053 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2a00067af0 0x7f2a00069fb0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f2a18005890 tx=0x7f2a18009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:13.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.576+0000 7f2a26000640 1 -- 172.21.15.27:0/125274053 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2a100ec730 con 0x7f2a280cf100 2024-01-15T18:19:13.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:13.768+0000 7f2a30800640 1 -- 172.21.15.27:0/125274053 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone55243", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a2816b080 con 0x7f2a00067af0 2024-01-15T18:19:18.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.336+0000 7f2a26000640 1 -- 172.21.15.27:0/125274053 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f2a18001d80 con 0x7f2a00067af0 2024-01-15T18:19:18.334 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:19:18.334 INFO:teuthology.orchestra.run.smithi027.stdout: "status": { 2024-01-15T18:19:18.334 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "pending", 2024-01-15T18:19:18.334 INFO:teuthology.orchestra.run.smithi027.stdout: "source": { 2024-01-15T18:19:18.334 INFO:teuthology.orchestra.run.smithi027.stdout: "volume": "cephfs", 2024-01-15T18:19:18.334 INFO:teuthology.orchestra.run.smithi027.stdout: "subvolume": "sv1", 2024-01-15T18:19:18.334 INFO:teuthology.orchestra.run.smithi027.stdout: "snapshot": "ss1" 2024-01-15T18:19:18.334 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:19:18.334 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:19:18.334 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:19:18.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.336+0000 7f2a30800640 1 -- 172.21.15.27:0/125274053 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2a00067af0 msgr2=0x7f2a00069fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:18.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.336+0000 7f2a30800640 1 --2- 172.21.15.27:0/125274053 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2a00067af0 0x7f2a00069fb0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f2a18005890 tx=0x7f2a18009040 comp rx=0 tx=0).stop 2024-01-15T18:19:18.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.336+0000 7f2a30800640 1 -- 172.21.15.27:0/125274053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a280cf100 msgr2=0x7f2a2815aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:18.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.336+0000 7f2a30800640 1 --2- 172.21.15.27:0/125274053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a280cf100 0x7f2a2815aa60 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f2a100ad040 tx=0x7f2a1001ac50 comp rx=0 tx=0).stop 2024-01-15T18:19:18.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.336+0000 7f2a30800640 1 -- 172.21.15.27:0/125274053 shutdown_connections 2024-01-15T18:19:18.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.336+0000 7f2a30800640 1 --2- 172.21.15.27:0/125274053 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2a00067af0 0x7f2a00069fb0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:18.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.336+0000 7f2a30800640 1 --2- 172.21.15.27:0/125274053 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2a281468d0 0x7f2a280afa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:18.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.336+0000 7f2a30800640 1 --2- 172.21.15.27:0/125274053 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a280cf100 0x7f2a2815aa60 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:18.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.336+0000 7f2a30800640 1 --2- 172.21.15.27:0/125274053 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a280ccea0 0x7f2a281384a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:18.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.336+0000 7f2a30800640 1 -- 172.21.15.27:0/125274053 >> 172.21.15.27:0/125274053 conn(0x7f2a280cb700 msgr2=0x7f2a2815c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:18.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.336+0000 7f2a30800640 1 -- 172.21.15.27:0/125274053 shutdown_connections 2024-01-15T18:19:18.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.336+0000 7f2a30800640 1 -- 172.21.15.27:0/125274053 wait complete. 2024-01-15T18:19:18.363 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone55241 2024-01-15T18:19:18.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.516+0000 7fe44e800640 1 Processor -- start 2024-01-15T18:19:18.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.516+0000 7fe44e800640 1 -- start start 2024-01-15T18:19:18.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.516+0000 7fe44e800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe44815d570 0x7fe44806be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:18.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.516+0000 7fe44e800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe44806b810 0x7fe44806c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:18.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.516+0000 7fe44e800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4480b0d70 con 0x7fe44815d190 2024-01-15T18:19:18.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.516+0000 7fe44e800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4480d0540 con 0x7fe44815d570 2024-01-15T18:19:18.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.516+0000 7fe44e800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4480b0440 con 0x7fe44806b810 2024-01-15T18:19:18.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.516+0000 7fe446a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe44815d570 0x7fe44806be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:18.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.516+0000 7fe447400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe44815d190 0x7fe4480cfcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57980/0 (socket says 172.21.15.27:57980) 2024-01-15T18:19:18.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.516+0000 7fe447400640 1 -- 172.21.15.27:0/1765657294 learned_addr learned my addr 172.21.15.27:0/1765657294 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:19:18.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.516+0000 7fe447e00640 1 --2- 172.21.15.27:0/1765657294 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe44806b810 0x7fe44806c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:18.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.516+0000 7fe446a00640 1 -- 172.21.15.27:0/1765657294 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe44806b810 msgr2=0x7fe44806c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:18.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.516+0000 7fe446a00640 1 --2- 172.21.15.27:0/1765657294 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe44806b810 0x7fe44806c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:18.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.516+0000 7fe446a00640 1 -- 172.21.15.27:0/1765657294 >> v1:172.21.15.27:6789/0 conn(0x7fe44815d190 legacy=0x7fe4480cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:18.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.516+0000 7fe446a00640 1 -- 172.21.15.27:0/1765657294 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4480b0720 con 0x7fe44815d570 2024-01-15T18:19:18.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.520+0000 7fe446a00640 1 --2- 172.21.15.27:0/1765657294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe44815d570 0x7fe44806be10 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fe4300097c0 tx=0x7fe43001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b90d46ed493fc14 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:18.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.520+0000 7fe446000640 1 -- 172.21.15.27:0/1765657294 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4300ab670 con 0x7fe44815d570 2024-01-15T18:19:18.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.520+0000 7fe446000640 1 -- 172.21.15.27:0/1765657294 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4300ab7d0 con 0x7fe44815d570 2024-01-15T18:19:18.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.520+0000 7fe446000640 1 -- 172.21.15.27:0/1765657294 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4300abb60 con 0x7fe44815d570 2024-01-15T18:19:18.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.520+0000 7fe44e800640 1 -- 172.21.15.27:0/1765657294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe44815d570 msgr2=0x7fe44806be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:18.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.520+0000 7fe44e800640 1 --2- 172.21.15.27:0/1765657294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe44815d570 0x7fe44806be10 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fe4300097c0 tx=0x7fe43001c450 comp rx=0 tx=0).stop 2024-01-15T18:19:18.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.520+0000 7fe44e800640 1 -- 172.21.15.27:0/1765657294 shutdown_connections 2024-01-15T18:19:18.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.520+0000 7fe44e800640 1 --2- 172.21.15.27:0/1765657294 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe44806b810 0x7fe44806c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:18.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.520+0000 7fe44e800640 1 --2- 172.21.15.27:0/1765657294 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe44815d570 0x7fe44806be10 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:18.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.520+0000 7fe44e800640 1 -- 172.21.15.27:0/1765657294 >> 172.21.15.27:0/1765657294 conn(0x7fe4480d0e70 msgr2=0x7fe4480d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:18.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.520+0000 7fe44e800640 1 -- 172.21.15.27:0/1765657294 shutdown_connections 2024-01-15T18:19:18.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.520+0000 7fe44e800640 1 -- 172.21.15.27:0/1765657294 wait complete. 2024-01-15T18:19:18.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.524+0000 7fe44e800640 1 Processor -- start 2024-01-15T18:19:18.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.524+0000 7fe44e800640 1 -- start start 2024-01-15T18:19:18.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.524+0000 7fe44e800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe44806b810 0x7fe4480b62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:18.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.524+0000 7fe44e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe44815d190 0x7fe448151430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:18.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.524+0000 7fe44e800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe44815d570 0x7fe448151970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:18.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.524+0000 7fe44e800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4480b56b0 con 0x7fe44815d190 2024-01-15T18:19:18.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.524+0000 7fe44e800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe448152600 con 0x7fe44815d570 2024-01-15T18:19:18.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.524+0000 7fe44e800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4481528e0 con 0x7fe44806b810 2024-01-15T18:19:18.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.524+0000 7fe446a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe44815d190 0x7fe448151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:18.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.524+0000 7fe446a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe44815d190 0x7fe448151430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48036/0 (socket says 172.21.15.27:48036) 2024-01-15T18:19:18.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.524+0000 7fe446a00640 1 -- 172.21.15.27:0/445005784 learned_addr learned my addr 172.21.15.27:0/445005784 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:19:18.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.524+0000 7fe447400640 1 --2- 172.21.15.27:0/445005784 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe44806b810 0x7fe4480b62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:18.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.524+0000 7fe447e00640 1 --2- 172.21.15.27:0/445005784 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe44815d570 0x7fe448151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:18.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.524+0000 7fe446a00640 1 -- 172.21.15.27:0/445005784 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe44806b810 msgr2=0x7fe4480b62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:18.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.524+0000 7fe446a00640 1 --2- 172.21.15.27:0/445005784 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe44806b810 0x7fe4480b62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:18.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.524+0000 7fe446a00640 1 -- 172.21.15.27:0/445005784 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe44815d570 msgr2=0x7fe448151970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:18.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.524+0000 7fe446a00640 1 --2- 172.21.15.27:0/445005784 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe44815d570 0x7fe448151970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:18.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.524+0000 7fe446a00640 1 -- 172.21.15.27:0/445005784 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe430009420 con 0x7fe44815d190 2024-01-15T18:19:18.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.524+0000 7fe447e00640 1 --2- 172.21.15.27:0/445005784 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe44815d570 0x7fe448151970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:19:18.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.524+0000 7fe446a00640 1 --2- 172.21.15.27:0/445005784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe44815d190 0x7fe448151430 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fe43001ab00 tx=0x7fe4300191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:18.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.524+0000 7fe43f400640 1 -- 172.21.15.27:0/445005784 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4300bf650 con 0x7fe44815d190 2024-01-15T18:19:18.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.524+0000 7fe43f400640 1 -- 172.21.15.27:0/445005784 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4300bf7d0 con 0x7fe44815d190 2024-01-15T18:19:18.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.524+0000 7fe43f400640 1 -- 172.21.15.27:0/445005784 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4300bfb80 con 0x7fe44815d190 2024-01-15T18:19:18.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.524+0000 7fe44e800640 1 -- 172.21.15.27:0/445005784 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe448152bc0 con 0x7fe44815d190 2024-01-15T18:19:18.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.524+0000 7fe44e800640 1 -- 172.21.15.27:0/445005784 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe44815bf60 con 0x7fe44815d190 2024-01-15T18:19:18.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.528+0000 7fe44e800640 1 -- 172.21.15.27:0/445005784 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe44804e850 con 0x7fe44815d190 2024-01-15T18:19:18.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.528+0000 7fe43f400640 1 -- 172.21.15.27:0/445005784 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 40) v1 ==== 87792+0+0 (secure 0 0 0) 0x7fe4300bfce0 con 0x7fe44815d190 2024-01-15T18:19:18.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.528+0000 7fe43f400640 1 --2- 172.21.15.27:0/445005784 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe420067af0 0x7fe420069fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:18.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.528+0000 7fe43f400640 1 -- 172.21.15.27:0/445005784 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fe43011dd90 con 0x7fe44815d190 2024-01-15T18:19:18.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.528+0000 7fe447400640 1 --2- 172.21.15.27:0/445005784 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe420067af0 0x7fe420069fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:18.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.528+0000 7fe447400640 1 --2- 172.21.15.27:0/445005784 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe420067af0 0x7fe420069fb0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fe438005890 tx=0x7fe438009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:18.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.532+0000 7fe43f400640 1 -- 172.21.15.27:0/445005784 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe4300ec710 con 0x7fe44815d190 2024-01-15T18:19:18.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:18.720+0000 7fe44e800640 1 -- 172.21.15.27:0/445005784 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone55241", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4480d1a30 con 0x7fe420067af0 2024-01-15T18:19:23.341 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:19:23.341+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:19:23.342 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:19:23.341+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:19:23.342 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:19:23.341+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:19:23.342 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:19:23.341+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:19:23.342 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:19:23.341+0000 7f59f9200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:19:23.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.428+0000 7fe43f400640 1 -- 172.21.15.27:0/445005784 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe4480d1a30 con 0x7fe420067af0 2024-01-15T18:19:23.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.432+0000 7fe44e800640 1 -- 172.21.15.27:0/445005784 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe420067af0 msgr2=0x7fe420069fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:23.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.432+0000 7fe44e800640 1 --2- 172.21.15.27:0/445005784 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe420067af0 0x7fe420069fb0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fe438005890 tx=0x7fe438009040 comp rx=0 tx=0).stop 2024-01-15T18:19:23.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.432+0000 7fe44e800640 1 -- 172.21.15.27:0/445005784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe44815d190 msgr2=0x7fe448151430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:23.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.432+0000 7fe44e800640 1 --2- 172.21.15.27:0/445005784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe44815d190 0x7fe448151430 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fe43001ab00 tx=0x7fe4300191f0 comp rx=0 tx=0).stop 2024-01-15T18:19:23.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.432+0000 7fe44e800640 1 -- 172.21.15.27:0/445005784 shutdown_connections 2024-01-15T18:19:23.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.432+0000 7fe44e800640 1 --2- 172.21.15.27:0/445005784 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe420067af0 0x7fe420069fb0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:23.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.432+0000 7fe44e800640 1 --2- 172.21.15.27:0/445005784 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe44815d570 0x7fe448151970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:23.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.432+0000 7fe44e800640 1 --2- 172.21.15.27:0/445005784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe44815d190 0x7fe448151430 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:23.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.432+0000 7fe44e800640 1 --2- 172.21.15.27:0/445005784 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe44806b810 0x7fe4480b62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:23.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.432+0000 7fe44e800640 1 -- 172.21.15.27:0/445005784 >> 172.21.15.27:0/445005784 conn(0x7fe4480d0e70 msgr2=0x7fe4480adae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:23.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.432+0000 7fe44e800640 1 -- 172.21.15.27:0/445005784 shutdown_connections 2024-01-15T18:19:23.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.432+0000 7fe44e800640 1 -- 172.21.15.27:0/445005784 wait complete. 2024-01-15T18:19:23.451 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone55242 2024-01-15T18:19:23.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.600+0000 7f13dd600640 1 Processor -- start 2024-01-15T18:19:23.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.600+0000 7f13dd600640 1 -- start start 2024-01-15T18:19:23.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.600+0000 7f13dd600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f13d80d1a40 0x7f13d80cba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:23.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.600+0000 7f13dd600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d80cbfa0 con 0x7f13d80b0650 2024-01-15T18:19:23.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.600+0000 7f13dd600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d80cefa0 con 0x7f13d80cf450 2024-01-15T18:19:23.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.600+0000 7f13dd600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13d80ce240 con 0x7f13d80d1a40 2024-01-15T18:19:23.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.600+0000 7f13d6000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f13d80d1a40 0x7f13d80cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:23.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.600+0000 7f13d7400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f13d80b0650 0x7f13d80b0a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57992/0 (socket says 172.21.15.27:57992) 2024-01-15T18:19:23.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.600+0000 7f13d6000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f13d80d1a40 0x7f13d80cba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48038/0 (socket says 172.21.15.27:48038) 2024-01-15T18:19:23.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.600+0000 7f13d7400640 1 -- 172.21.15.27:0/3471897037 learned_addr learned my addr 172.21.15.27:0/3471897037 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:19:23.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.604+0000 7f13d6000640 1 -- 172.21.15.27:0/3471897037 >> v1:172.21.15.93:6789/0 conn(0x7f13d80cf450 legacy=0x7f13d80cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:23.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.604+0000 7f13d6000640 1 -- 172.21.15.27:0/3471897037 >> v1:172.21.15.27:6789/0 conn(0x7f13d80b0650 legacy=0x7f13d80b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:23.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.604+0000 7f13d6000640 1 -- 172.21.15.27:0/3471897037 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d80ccd70 con 0x7f13d80d1a40 2024-01-15T18:19:23.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.604+0000 7f13d6000640 1 --2- 172.21.15.27:0/3471897037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13d80d1a40 0x7f13d80cba60 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f13c00097c0 tx=0x7f13c001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90b433f18aadf848 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:23.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.604+0000 7f13d5600640 1 -- 172.21.15.27:0/3471897037 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c00ab600 con 0x7f13d80d1a40 2024-01-15T18:19:23.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.604+0000 7f13d5600640 1 -- 172.21.15.27:0/3471897037 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c00a7050 con 0x7f13d80d1a40 2024-01-15T18:19:23.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.604+0000 7f13d5600640 1 -- 172.21.15.27:0/3471897037 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c00ab980 con 0x7f13d80d1a40 2024-01-15T18:19:23.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.608+0000 7f13dd600640 1 -- 172.21.15.27:0/3471897037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13d80d1a40 msgr2=0x7f13d80cba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:23.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.608+0000 7f13dd600640 1 --2- 172.21.15.27:0/3471897037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13d80d1a40 0x7f13d80cba60 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f13c00097c0 tx=0x7f13c001c480 comp rx=0 tx=0).stop 2024-01-15T18:19:23.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.608+0000 7f13dd600640 1 -- 172.21.15.27:0/3471897037 shutdown_connections 2024-01-15T18:19:23.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.608+0000 7f13dd600640 1 --2- 172.21.15.27:0/3471897037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13d80d1a40 0x7f13d80cba60 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:23.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.608+0000 7f13dd600640 1 -- 172.21.15.27:0/3471897037 >> 172.21.15.27:0/3471897037 conn(0x7f13d80d0e70 msgr2=0x7f13d80d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:23.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.608+0000 7f13dd600640 1 -- 172.21.15.27:0/3471897037 shutdown_connections 2024-01-15T18:19:23.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.608+0000 7f13dd600640 1 -- 172.21.15.27:0/3471897037 wait complete. 2024-01-15T18:19:23.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.608+0000 7f13dd600640 1 Processor -- start 2024-01-15T18:19:23.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.608+0000 7f13dd600640 1 -- start start 2024-01-15T18:19:23.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.608+0000 7f13dd600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13d80b0650 0x7f13d80caa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:23.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.608+0000 7f13dd600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13d80cf450 0x7f13d80c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:23.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.608+0000 7f13dd600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13d80d1a40 0x7f13d80c4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:23.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.608+0000 7f13dd600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d806e510 con 0x7f13d80cf450 2024-01-15T18:19:23.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.608+0000 7f13dd600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d80c5820 con 0x7f13d80d1a40 2024-01-15T18:19:23.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.608+0000 7f13dd600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d80c5b00 con 0x7f13d80b0650 2024-01-15T18:19:23.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.608+0000 7f13d6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13d80cf450 0x7f13d80c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:23.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.608+0000 7f13d6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13d80cf450 0x7f13d80c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48042/0 (socket says 172.21.15.27:48042) 2024-01-15T18:19:23.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.608+0000 7f13d6000640 1 -- 172.21.15.27:0/2106958365 learned_addr learned my addr 172.21.15.27:0/2106958365 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:19:23.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.608+0000 7f13d7400640 1 --2- 172.21.15.27:0/2106958365 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13d80d1a40 0x7f13d80c4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:23.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.608+0000 7f13d6a00640 1 --2- 172.21.15.27:0/2106958365 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13d80b0650 0x7f13d80caa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:23.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.612+0000 7f13d6000640 1 -- 172.21.15.27:0/2106958365 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13d80b0650 msgr2=0x7f13d80caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:23.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.612+0000 7f13d6000640 1 --2- 172.21.15.27:0/2106958365 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13d80b0650 0x7f13d80caa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:23.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.612+0000 7f13d6000640 1 -- 172.21.15.27:0/2106958365 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13d80d1a40 msgr2=0x7f13d80c4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:23.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.612+0000 7f13d6000640 1 --2- 172.21.15.27:0/2106958365 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13d80d1a40 0x7f13d80c4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:23.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.612+0000 7f13d6000640 1 -- 172.21.15.27:0/2106958365 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c0009420 con 0x7f13d80cf450 2024-01-15T18:19:23.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.612+0000 7f13d6a00640 1 --2- 172.21.15.27:0/2106958365 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13d80b0650 0x7f13d80caa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:19:23.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.612+0000 7f13d6000640 1 --2- 172.21.15.27:0/2106958365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13d80cf450 0x7f13d80c49a0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f13c001a0b0 tx=0x7f13c0019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:23.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.612+0000 7f13cea00640 1 -- 172.21.15.27:0/2106958365 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c00b4810 con 0x7f13d80cf450 2024-01-15T18:19:23.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.612+0000 7f13cea00640 1 -- 172.21.15.27:0/2106958365 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c00b4990 con 0x7f13d80cf450 2024-01-15T18:19:23.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.612+0000 7f13dd600640 1 -- 172.21.15.27:0/2106958365 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d80c5de0 con 0x7f13d80cf450 2024-01-15T18:19:23.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.612+0000 7f13cea00640 1 -- 172.21.15.27:0/2106958365 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c00b4d40 con 0x7f13d80cf450 2024-01-15T18:19:23.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.612+0000 7f13dd600640 1 -- 172.21.15.27:0/2106958365 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d80c5420 con 0x7f13d80cf450 2024-01-15T18:19:23.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.612+0000 7f13dd600640 1 -- 172.21.15.27:0/2106958365 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d806e6c0 con 0x7f13d80cf450 2024-01-15T18:19:23.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.612+0000 7f13cea00640 1 -- 172.21.15.27:0/2106958365 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 40) v1 ==== 87792+0+0 (secure 0 0 0) 0x7f13c00b2040 con 0x7f13d80cf450 2024-01-15T18:19:23.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.616+0000 7f13cea00640 1 --2- 172.21.15.27:0/2106958365 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f13b0067af0 0x7f13b0069fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:23.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.616+0000 7f13cea00640 1 -- 172.21.15.27:0/2106958365 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f13c011d890 con 0x7f13d80cf450 2024-01-15T18:19:23.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.616+0000 7f13d6a00640 1 --2- 172.21.15.27:0/2106958365 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f13b0067af0 0x7f13b0069fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:23.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.616+0000 7f13d6a00640 1 --2- 172.21.15.27:0/2106958365 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f13b0067af0 0x7f13b0069fb0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f13d80c79a0 tx=0x7f13c8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:23.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.620+0000 7f13cea00640 1 -- 172.21.15.27:0/2106958365 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f13d806e6c0 con 0x7f13d80cf450 2024-01-15T18:19:23.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:23.808+0000 7f13dd600640 1 -- 172.21.15.27:0/2106958365 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone55242", "target": ["mon-mgr", ""]}) v1 -- 0x7f13d80c5c70 con 0x7f13b0067af0 2024-01-15T18:19:25.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:19:25.244+0000 7f13cea00640 1 -- 172.21.15.27:0/2106958365 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 41) v1 ==== 87843+0+0 (secure 0 0 0) 0x7f13c00ec630 con 0x7f13d80cf450 2024-01-15T18:19:28.050 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:19:28.054 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:19:58.111 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:19:58.116 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:20:28.172 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:20:28.176 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:20:41.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:20:41.207+0000 7f13cea00640 1 -- 172.21.15.27:0/2106958365 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 42) v1 ==== 87792+0+0 (secure 0 0 0) 0x7f13c00ef410 con 0x7f13d80cf450 2024-01-15T18:20:58.232 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:20:58.236 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:21:23.481 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-15T18:21:23.482 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:21:23.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2734800640 1 Processor -- start 2024-01-15T18:21:23.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2734800640 1 -- start start 2024-01-15T18:21:23.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2734800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f272c06c200 0x7f272c06d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:23.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2734800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f272c06c750 con 0x7f272c143da0 2024-01-15T18:21:23.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2734800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f272c06ca30 con 0x7f272c0cad40 2024-01-15T18:21:23.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2734800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f272c144180 con 0x7f272c06c200 2024-01-15T18:21:23.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2731e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f272c143da0 0x7f272c0cb9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34586/0 (socket says 172.21.15.27:34586) 2024-01-15T18:21:23.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2731e00640 1 -- 172.21.15.27:0/2571857550 learned_addr learned my addr 172.21.15.27:0/2571857550 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:21:23.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2731400640 1 --2- 172.21.15.27:0/2571857550 >> v2:172.21.15.93:3300/0 conn(0x7f272c06c200 0x7f272c06d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:23.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2730a00640 1 -- 172.21.15.27:0/2571857550 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626564000 0 0) 0x7f272c06c750 con 0x7f272c143da0 2024-01-15T18:21:23.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2730a00640 1 -- 172.21.15.27:0/2571857550 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f270c003530 con 0x7f272c143da0 2024-01-15T18:21:23.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2730a00640 1 -- 172.21.15.27:0/2571857550 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1127178749 0 0) 0x7f272c06ca30 con 0x7f272c0cad40 2024-01-15T18:21:23.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2730a00640 1 -- 172.21.15.27:0/2571857550 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f272c06c750 con 0x7f272c0cad40 2024-01-15T18:21:23.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2730a00640 1 -- 172.21.15.27:0/2571857550 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3520272319 0 0) 0x7f270c003530 con 0x7f272c143da0 2024-01-15T18:21:23.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2730a00640 1 -- 172.21.15.27:0/2571857550 >> v2:172.21.15.93:3300/0 conn(0x7f272c06c200 msgr2=0x7f272c06d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:23.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2730a00640 1 --2- 172.21.15.27:0/2571857550 >> v2:172.21.15.93:3300/0 conn(0x7f272c06c200 0x7f272c06d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:23.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2730a00640 1 -- 172.21.15.27:0/2571857550 >> v1:172.21.15.93:6789/0 conn(0x7f272c0cad40 legacy=0x7f272c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:23.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2730a00640 1 -- 172.21.15.27:0/2571857550 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f272c06cd10 con 0x7f272c143da0 2024-01-15T18:21:23.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2730a00640 1 -- 172.21.15.27:0/2571857550 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f271c001680 con 0x7f272c143da0 2024-01-15T18:21:23.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2731400640 1 --2- 172.21.15.27:0/2571857550 >> v2:172.21.15.93:3300/0 conn(0x7f272c06c200 0x7f272c06d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:21:23.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2730a00640 1 -- 172.21.15.27:0/2571857550 >> v1:172.21.15.27:6789/0 conn(0x7f272c143da0 legacy=0x7f272c0cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:23.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2730a00640 1 --2- 172.21.15.27:0/2571857550 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f270c003b80 0x7f270c003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:23.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2730a00640 1 --2- 172.21.15.27:0/2571857550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f270c0047d0 0x7f270c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:23.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2730a00640 1 --2- 172.21.15.27:0/2571857550 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f270c005190 0x7f270c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:23.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2730a00640 1 -- 172.21.15.27:0/2571857550 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272c144180 con 0x7f270c0047d0 2024-01-15T18:21:23.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2730a00640 1 -- 172.21.15.27:0/2571857550 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270c00c090 con 0x7f270c005190 2024-01-15T18:21:23.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2730a00640 1 -- 172.21.15.27:0/2571857550 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270c00c250 con 0x7f270c003b80 2024-01-15T18:21:23.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2731400640 1 --2- 172.21.15.27:0/2571857550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f270c0047d0 0x7f270c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:23.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2731e00640 1 --2- 172.21.15.27:0/2571857550 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f270c003b80 0x7f270c003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:23.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2732800640 1 --2- 172.21.15.27:0/2571857550 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f270c005190 0x7f270c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:23.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2731400640 1 -- 172.21.15.27:0/2571857550 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f270c003b80 msgr2=0x7f270c003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:23.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2731400640 1 --2- 172.21.15.27:0/2571857550 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f270c003b80 0x7f270c003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:23.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2731400640 1 -- 172.21.15.27:0/2571857550 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f270c005190 msgr2=0x7f270c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:23.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2731400640 1 --2- 172.21.15.27:0/2571857550 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f270c005190 0x7f270c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:23.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2731400640 1 -- 172.21.15.27:0/2571857550 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270c003530 con 0x7f270c0047d0 2024-01-15T18:21:23.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.639+0000 7f2732800640 1 -- 172.21.15.27:0/2571857550 reap_dead start 2024-01-15T18:21:23.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.643+0000 7f2731400640 1 --2- 172.21.15.27:0/2571857550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f270c0047d0 0x7f270c004c50 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f2714009cb0 tx=0x7f271401cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef5c643f78054084 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:23.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.643+0000 7f2730a00640 1 -- 172.21.15.27:0/2571857550 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27140a57a0 con 0x7f270c0047d0 2024-01-15T18:21:23.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.643+0000 7f2730a00640 1 -- 172.21.15.27:0/2571857550 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27140a5900 con 0x7f270c0047d0 2024-01-15T18:21:23.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.643+0000 7f2730a00640 1 -- 172.21.15.27:0/2571857550 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27140a5cf0 con 0x7f270c0047d0 2024-01-15T18:21:23.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.643+0000 7f2734800640 1 -- 172.21.15.27:0/2571857550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f270c0047d0 msgr2=0x7f270c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:23.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.643+0000 7f2734800640 1 --2- 172.21.15.27:0/2571857550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f270c0047d0 0x7f270c004c50 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f2714009cb0 tx=0x7f271401cbf0 comp rx=0 tx=0).stop 2024-01-15T18:21:23.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.643+0000 7f2734800640 1 -- 172.21.15.27:0/2571857550 shutdown_connections 2024-01-15T18:21:23.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.643+0000 7f2734800640 1 --2- 172.21.15.27:0/2571857550 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f270c0047d0 0x7f270c004c50 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:23.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.643+0000 7f2734800640 1 -- 172.21.15.27:0/2571857550 >> 172.21.15.27:0/2571857550 conn(0x7f272c0b0850 msgr2=0x7f272c0b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:23.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.647+0000 7f2734800640 1 -- 172.21.15.27:0/2571857550 shutdown_connections 2024-01-15T18:21:23.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.647+0000 7f2734800640 1 -- 172.21.15.27:0/2571857550 wait complete. 2024-01-15T18:21:23.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.647+0000 7f2734800640 1 Processor -- start 2024-01-15T18:21:23.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.647+0000 7f2734800640 1 -- start start 2024-01-15T18:21:23.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.647+0000 7f2734800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f270c0047d0 0x7f272c13cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:23.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.647+0000 7f2734800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f272c14a860 0x7f272c13a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:23.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.647+0000 7f2734800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f272c13c500 0x7f272c13f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:23.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.647+0000 7f2734800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272c13add0 con 0x7f272c13c500 2024-01-15T18:21:23.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.647+0000 7f2734800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272c13d460 con 0x7f270c0047d0 2024-01-15T18:21:23.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.647+0000 7f2734800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f272c0b1e50 con 0x7f272c14a860 2024-01-15T18:21:23.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.647+0000 7f2732800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f272c13c500 0x7f272c13f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:23.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.647+0000 7f2731e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f270c0047d0 0x7f272c13cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:23.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.647+0000 7f2732800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f272c13c500 0x7f272c13f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50998/0 (socket says 172.21.15.27:50998) 2024-01-15T18:21:23.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.647+0000 7f2732800640 1 -- 172.21.15.27:0/898166840 learned_addr learned my addr 172.21.15.27:0/898166840 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:21:23.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.647+0000 7f2731400640 1 --2- 172.21.15.27:0/898166840 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f272c14a860 0x7f272c13a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:23.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.647+0000 7f2732800640 1 -- 172.21.15.27:0/898166840 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f272c14a860 msgr2=0x7f272c13a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:23.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.647+0000 7f2732800640 1 --2- 172.21.15.27:0/898166840 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f272c14a860 0x7f272c13a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:23.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.647+0000 7f2732800640 1 -- 172.21.15.27:0/898166840 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f270c0047d0 msgr2=0x7f272c13cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:23.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.647+0000 7f2732800640 1 --2- 172.21.15.27:0/898166840 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f270c0047d0 0x7f272c13cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:23.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.647+0000 7f2732800640 1 -- 172.21.15.27:0/898166840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f272c0b2130 con 0x7f272c13c500 2024-01-15T18:21:23.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.647+0000 7f2731400640 1 --2- 172.21.15.27:0/898166840 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f272c14a860 0x7f272c13a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:21:23.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.647+0000 7f2731e00640 1 --2- 172.21.15.27:0/898166840 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f270c0047d0 0x7f272c13cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:21:23.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.647+0000 7f2732800640 1 --2- 172.21.15.27:0/898166840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f272c13c500 0x7f272c13f280 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f27240059b0 tx=0x7f2724012d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:23.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.647+0000 7f272a000640 1 -- 172.21.15.27:0/898166840 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2724010ac0 con 0x7f272c13c500 2024-01-15T18:21:23.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.651+0000 7f2734800640 1 -- 172.21.15.27:0/898166840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f272c0b2410 con 0x7f272c13c500 2024-01-15T18:21:23.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.651+0000 7f272a000640 1 -- 172.21.15.27:0/898166840 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2724010c60 con 0x7f272c13c500 2024-01-15T18:21:23.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.651+0000 7f272a000640 1 -- 172.21.15.27:0/898166840 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2724016480 con 0x7f272c13c500 2024-01-15T18:21:23.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.651+0000 7f2734800640 1 -- 172.21.15.27:0/898166840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f272c14ace0 con 0x7f272c13c500 2024-01-15T18:21:23.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.651+0000 7f2734800640 1 -- 172.21.15.27:0/898166840 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f272c04e850 con 0x7f272c13c500 2024-01-15T18:21:23.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.651+0000 7f272a000640 1 -- 172.21.15.27:0/898166840 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 42) v1 ==== 87792+0+0 (secure 0 0 0) 0x7f27240165e0 con 0x7f272c13c500 2024-01-15T18:21:23.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.651+0000 7f272a000640 1 --2- 172.21.15.27:0/898166840 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2704067af0 0x7f2704069fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:23.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.651+0000 7f272a000640 1 -- 172.21.15.27:0/898166840 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f2724086690 con 0x7f272c13c500 2024-01-15T18:21:23.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.651+0000 7f2731e00640 1 --2- 172.21.15.27:0/898166840 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2704067af0 0x7f2704069fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:23.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.655+0000 7f2731e00640 1 --2- 172.21.15.27:0/898166840 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2704067af0 0x7f2704069fb0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f272c069290 tx=0x7f271c001e10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:23.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.655+0000 7f272a000640 1 -- 172.21.15.27:0/898166840 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2724055350 con 0x7f272c13c500 2024-01-15T18:21:23.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.835+0000 7f2734800640 1 -- 172.21.15.27:0/898166840 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f272c06d2d0 con 0x7f2704067af0 2024-01-15T18:21:23.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.835+0000 7f272a000640 1 -- 172.21.15.27:0/898166840 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f271c007cd0 con 0x7f2704067af0 2024-01-15T18:21:23.831 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:21:23.832 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:21:23.832 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "cephfs" 2024-01-15T18:21:23.832 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:21:23.832 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:21:23.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.835+0000 7f2734800640 1 -- 172.21.15.27:0/898166840 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2704067af0 msgr2=0x7f2704069fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:23.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.835+0000 7f2734800640 1 --2- 172.21.15.27:0/898166840 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2704067af0 0x7f2704069fb0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f272c069290 tx=0x7f271c001e10 comp rx=0 tx=0).stop 2024-01-15T18:21:23.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.835+0000 7f2734800640 1 -- 172.21.15.27:0/898166840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f272c13c500 msgr2=0x7f272c13f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:23.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.835+0000 7f2734800640 1 --2- 172.21.15.27:0/898166840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f272c13c500 0x7f272c13f280 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f27240059b0 tx=0x7f2724012d00 comp rx=0 tx=0).stop 2024-01-15T18:21:23.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.839+0000 7f2734800640 1 -- 172.21.15.27:0/898166840 shutdown_connections 2024-01-15T18:21:23.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.839+0000 7f2734800640 1 --2- 172.21.15.27:0/898166840 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2704067af0 0x7f2704069fb0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:23.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.839+0000 7f2734800640 1 --2- 172.21.15.27:0/898166840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f272c13c500 0x7f272c13f280 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:23.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.839+0000 7f2734800640 1 --2- 172.21.15.27:0/898166840 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f272c14a860 0x7f272c13a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:23.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.839+0000 7f2734800640 1 --2- 172.21.15.27:0/898166840 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f270c0047d0 0x7f272c13cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:23.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.839+0000 7f2734800640 1 -- 172.21.15.27:0/898166840 >> 172.21.15.27:0/898166840 conn(0x7f272c0b0850 msgr2=0x7f272c142f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:23.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.839+0000 7f2734800640 1 -- 172.21.15.27:0/898166840 shutdown_connections 2024-01-15T18:21:23.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:23.839+0000 7f2734800640 1 -- 172.21.15.27:0/898166840 wait complete. 2024-01-15T18:21:23.854 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-15T18:21:23.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.003+0000 7f19aae00640 1 Processor -- start 2024-01-15T18:21:23.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.003+0000 7f19aae00640 1 -- start start 2024-01-15T18:21:24.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.003+0000 7f19aae00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19a415d570 0x7f19a406be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:24.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.003+0000 7f19aae00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19a406b810 0x7f19a406c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:24.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.003+0000 7f19aae00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19a40b0d70 con 0x7f19a415d190 2024-01-15T18:21:24.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.003+0000 7f19aae00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19a40d0540 con 0x7f19a415d570 2024-01-15T18:21:24.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.003+0000 7f19aae00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a40b0440 con 0x7f19a406b810 2024-01-15T18:21:24.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.003+0000 7f19a3400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19a415d570 0x7f19a406be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:24.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.003+0000 7f19a3400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19a415d570 0x7f19a406be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51002/0 (socket says 172.21.15.27:51002) 2024-01-15T18:21:24.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.003+0000 7f19a3400640 1 -- 172.21.15.27:0/1175343789 learned_addr learned my addr 172.21.15.27:0/1175343789 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:21:24.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.003+0000 7f19a8e00640 1 --2- 172.21.15.27:0/1175343789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19a406b810 0x7f19a406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:24.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.003+0000 7f19a3400640 1 -- 172.21.15.27:0/1175343789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19a406b810 msgr2=0x7f19a406c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:24.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.003+0000 7f19a3400640 1 --2- 172.21.15.27:0/1175343789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19a406b810 0x7f19a406c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:24.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.003+0000 7f19a3400640 1 -- 172.21.15.27:0/1175343789 >> v1:172.21.15.93:6789/0 conn(0x7f19a415d190 legacy=0x7f19a40cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:24.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.003+0000 7f19a3400640 1 -- 172.21.15.27:0/1175343789 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a40b0720 con 0x7f19a415d570 2024-01-15T18:21:24.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.003+0000 7f19a8e00640 1 --2- 172.21.15.27:0/1175343789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19a406b810 0x7f19a406c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:21:24.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.007+0000 7f19a3400640 1 --2- 172.21.15.27:0/1175343789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19a415d570 0x7f19a406be10 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f198c009870 tx=0x7f198c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=941779aad20e0fe1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:24.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.007+0000 7f19a2a00640 1 -- 172.21.15.27:0/1175343789 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c0aa610 con 0x7f19a415d570 2024-01-15T18:21:24.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.007+0000 7f19a2a00640 1 -- 172.21.15.27:0/1175343789 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f198c0a6040 con 0x7f19a415d570 2024-01-15T18:21:24.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.007+0000 7f19a2a00640 1 -- 172.21.15.27:0/1175343789 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f198c0aa990 con 0x7f19a415d570 2024-01-15T18:21:24.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.007+0000 7f19aae00640 1 -- 172.21.15.27:0/1175343789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19a415d570 msgr2=0x7f19a406be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:24.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.007+0000 7f19aae00640 1 --2- 172.21.15.27:0/1175343789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19a415d570 0x7f19a406be10 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f198c009870 tx=0x7f198c01c4d0 comp rx=0 tx=0).stop 2024-01-15T18:21:24.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.007+0000 7f19aae00640 1 -- 172.21.15.27:0/1175343789 shutdown_connections 2024-01-15T18:21:24.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.007+0000 7f19aae00640 1 --2- 172.21.15.27:0/1175343789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19a406b810 0x7f19a406c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:24.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.007+0000 7f19aae00640 1 --2- 172.21.15.27:0/1175343789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19a415d570 0x7f19a406be10 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:24.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.007+0000 7f19aae00640 1 -- 172.21.15.27:0/1175343789 >> 172.21.15.27:0/1175343789 conn(0x7f19a40d0e70 msgr2=0x7f19a40d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:24.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.007+0000 7f19aae00640 1 -- 172.21.15.27:0/1175343789 shutdown_connections 2024-01-15T18:21:24.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.007+0000 7f19aae00640 1 -- 172.21.15.27:0/1175343789 wait complete. 2024-01-15T18:21:24.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.011+0000 7f19aae00640 1 Processor -- start 2024-01-15T18:21:24.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.011+0000 7f19aae00640 1 -- start start 2024-01-15T18:21:24.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.011+0000 7f19aae00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19a406b810 0x7f19a414d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:24.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.011+0000 7f19aae00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f19a415d190 0x7f19a40b6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:24.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.011+0000 7f19aae00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19a415d570 0x7f19a4150d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:24.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.011+0000 7f19aae00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a40b51d0 con 0x7f19a415d570 2024-01-15T18:21:24.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.011+0000 7f19aae00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a40c5ea0 con 0x7f19a415d190 2024-01-15T18:21:24.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.011+0000 7f19aae00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a40b4970 con 0x7f19a406b810 2024-01-15T18:21:24.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.011+0000 7f19a8e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19a415d570 0x7f19a4150d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:24.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.011+0000 7f19a8e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19a415d570 0x7f19a4150d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51006/0 (socket says 172.21.15.27:51006) 2024-01-15T18:21:24.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.011+0000 7f19a3e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19a406b810 0x7f19a414d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:24.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.011+0000 7f19a3400640 1 --2- 172.21.15.27:0/1811712175 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f19a415d190 0x7f19a40b6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:24.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.011+0000 7f19a8e00640 1 -- 172.21.15.27:0/1811712175 learned_addr learned my addr 172.21.15.27:0/1811712175 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:21:24.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.011+0000 7f19a8e00640 1 -- 172.21.15.27:0/1811712175 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19a406b810 msgr2=0x7f19a414d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:24.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.011+0000 7f19a8e00640 1 --2- 172.21.15.27:0/1811712175 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19a406b810 0x7f19a414d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:24.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.011+0000 7f19a8e00640 1 -- 172.21.15.27:0/1811712175 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f19a415d190 msgr2=0x7f19a40b6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:24.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.011+0000 7f19a8e00640 1 --2- 172.21.15.27:0/1811712175 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f19a415d190 0x7f19a40b6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:24.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.011+0000 7f19a8e00640 1 -- 172.21.15.27:0/1811712175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f198c0094d0 con 0x7f19a415d570 2024-01-15T18:21:24.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.011+0000 7f19a8e00640 1 --2- 172.21.15.27:0/1811712175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19a415d570 0x7f19a4150d10 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f199c00cea0 tx=0x7f199c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:24.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.011+0000 7f199be00640 1 -- 172.21.15.27:0/1811712175 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199c003ba0 con 0x7f19a415d570 2024-01-15T18:21:24.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.011+0000 7f199be00640 1 -- 172.21.15.27:0/1811712175 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f199c003d20 con 0x7f19a415d570 2024-01-15T18:21:24.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.011+0000 7f199be00640 1 -- 172.21.15.27:0/1811712175 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199c00d880 con 0x7f19a415d570 2024-01-15T18:21:24.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.011+0000 7f19aae00640 1 -- 172.21.15.27:0/1811712175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a40b4c10 con 0x7f19a415d570 2024-01-15T18:21:24.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.011+0000 7f19aae00640 1 -- 172.21.15.27:0/1811712175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a40c5b30 con 0x7f19a415d570 2024-01-15T18:21:24.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.015+0000 7f19aae00640 1 -- 172.21.15.27:0/1811712175 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a40c5990 con 0x7f19a415d570 2024-01-15T18:21:24.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.015+0000 7f199be00640 1 -- 172.21.15.27:0/1811712175 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 42) v1 ==== 87792+0+0 (secure 0 0 0) 0x7f199c00d9e0 con 0x7f19a415d570 2024-01-15T18:21:24.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.015+0000 7f199be00640 1 --2- 172.21.15.27:0/1811712175 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f197c067b20 0x7f197c069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:24.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.015+0000 7f199be00640 1 -- 172.21.15.27:0/1811712175 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f199c083860 con 0x7f19a415d570 2024-01-15T18:21:24.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.015+0000 7f19a3e00640 1 --2- 172.21.15.27:0/1811712175 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f197c067b20 0x7f197c069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:24.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.015+0000 7f19a3e00640 1 --2- 172.21.15.27:0/1811712175 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f197c067b20 0x7f197c069fe0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f1994007fd0 tx=0x7f1994007e40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:24.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.019+0000 7f199be00640 1 -- 172.21.15.27:0/1811712175 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f19a40c5990 con 0x7f19a415d570 2024-01-15T18:21:24.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.203+0000 7f19aae00640 1 -- 172.21.15.27:0/1811712175 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f19a406d230 con 0x7f197c067b20 2024-01-15T18:21:24.208 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:21:24.210+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:21:24.208 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:21:24.210+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:21:24.208 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:21:24.210+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:21:24.208 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:21:24.210+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:21:24.208 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:21:24.210+0000 7f59f2400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:21:24.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.227+0000 7f199be00640 1 -- 172.21.15.27:0/1811712175 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+185 (secure 0 0 0) 0x7f19a406d230 con 0x7f197c067b20 2024-01-15T18:21:24.222 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:21:24.222 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:21:24.222 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:21:24.222 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "sv1" 2024-01-15T18:21:24.222 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:21:24.222 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:21:24.222 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone55242" 2024-01-15T18:21:24.223 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:21:24.223 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:21:24.223 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone55243" 2024-01-15T18:21:24.223 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-01-15T18:21:24.223 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:21:24.223 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "subvol_clone55241" 2024-01-15T18:21:24.223 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:21:24.223 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:21:24.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.227+0000 7f19aae00640 1 -- 172.21.15.27:0/1811712175 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f197c067b20 msgr2=0x7f197c069fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:24.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.227+0000 7f19aae00640 1 --2- 172.21.15.27:0/1811712175 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f197c067b20 0x7f197c069fe0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f1994007fd0 tx=0x7f1994007e40 comp rx=0 tx=0).stop 2024-01-15T18:21:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.227+0000 7f19aae00640 1 -- 172.21.15.27:0/1811712175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19a415d570 msgr2=0x7f19a4150d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.227+0000 7f19aae00640 1 --2- 172.21.15.27:0/1811712175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19a415d570 0x7f19a4150d10 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f199c00cea0 tx=0x7f199c00cf90 comp rx=0 tx=0).stop 2024-01-15T18:21:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.227+0000 7f19aae00640 1 -- 172.21.15.27:0/1811712175 shutdown_connections 2024-01-15T18:21:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.227+0000 7f19aae00640 1 --2- 172.21.15.27:0/1811712175 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f197c067b20 0x7f197c069fe0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.227+0000 7f19aae00640 1 --2- 172.21.15.27:0/1811712175 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19a415d570 0x7f19a4150d10 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.227+0000 7f19aae00640 1 --2- 172.21.15.27:0/1811712175 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f19a415d190 0x7f19a40b6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.227+0000 7f19aae00640 1 --2- 172.21.15.27:0/1811712175 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f19a406b810 0x7f19a414d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.227+0000 7f19aae00640 1 -- 172.21.15.27:0/1811712175 >> 172.21.15.27:0/1811712175 conn(0x7f19a40d0e70 msgr2=0x7f19a40adb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.227+0000 7f19aae00640 1 -- 172.21.15.27:0/1811712175 shutdown_connections 2024-01-15T18:21:24.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.227+0000 7f19aae00640 1 -- 172.21.15.27:0/1811712175 wait complete. 2024-01-15T18:21:24.244 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-15T18:21:24.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.395+0000 7f6e65200640 1 Processor -- start 2024-01-15T18:21:24.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.395+0000 7f6e65200640 1 -- start start 2024-01-15T18:21:24.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.395+0000 7f6e65200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f6e6015ad20 0x7f6e6015b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:24.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.395+0000 7f6e65200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6e6015d900 0x7f6e6015ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:24.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.395+0000 7f6e65200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6e60070ee0 0x7f6e60071390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:24.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.395+0000 7f6e65200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e600718d0 con 0x7f6e6015d900 2024-01-15T18:21:24.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.395+0000 7f6e65200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e6015a610 con 0x7f6e6015ad20 2024-01-15T18:21:24.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.395+0000 7f6e65200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6015e3c0 con 0x7f6e60070ee0 2024-01-15T18:21:24.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.395+0000 7f6e5d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6e6015d900 0x7f6e6015ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:24.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.395+0000 7f6e5d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6e6015d900 0x7f6e6015ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51022/0 (socket says 172.21.15.27:51022) 2024-01-15T18:21:24.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.395+0000 7f6e5d400640 1 -- 172.21.15.27:0/2722732755 learned_addr learned my addr 172.21.15.27:0/2722732755 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:21:24.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.395+0000 7f6e5e800640 1 --2- 172.21.15.27:0/2722732755 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6e60070ee0 0x7f6e60071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:24.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.395+0000 7f6e5de00640 1 --2- 172.21.15.27:0/2722732755 >> v2:172.21.15.93:3300/0 conn(0x7f6e6015ad20 0x7f6e6015b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:24.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.395+0000 7f6e5d400640 1 -- 172.21.15.27:0/2722732755 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6e60070ee0 msgr2=0x7f6e60071390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:24.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.395+0000 7f6e5d400640 1 --2- 172.21.15.27:0/2722732755 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6e60070ee0 0x7f6e60071390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:24.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.395+0000 7f6e5d400640 1 -- 172.21.15.27:0/2722732755 >> v2:172.21.15.93:3300/0 conn(0x7f6e6015ad20 msgr2=0x7f6e6015b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:24.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.395+0000 7f6e5d400640 1 --2- 172.21.15.27:0/2722732755 >> v2:172.21.15.93:3300/0 conn(0x7f6e6015ad20 0x7f6e6015b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:24.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.395+0000 7f6e5d400640 1 -- 172.21.15.27:0/2722732755 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6015e6a0 con 0x7f6e6015d900 2024-01-15T18:21:24.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.395+0000 7f6e5e800640 1 --2- 172.21.15.27:0/2722732755 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6e60070ee0 0x7f6e60071390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:21:24.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.399+0000 7f6e5d400640 1 --2- 172.21.15.27:0/2722732755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e6015d900 0x7f6e6015ddb0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f6e48009870 tx=0x7f6e4801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c68a852a4643452 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:24.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.399+0000 7f6e5ca00640 1 -- 172.21.15.27:0/2722732755 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e480aa5f0 con 0x7f6e6015d900 2024-01-15T18:21:24.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.399+0000 7f6e5ca00640 1 -- 172.21.15.27:0/2722732755 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e480aa750 con 0x7f6e6015d900 2024-01-15T18:21:24.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.399+0000 7f6e5ca00640 1 -- 172.21.15.27:0/2722732755 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e480aaae0 con 0x7f6e6015d900 2024-01-15T18:21:24.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.399+0000 7f6e65200640 1 -- 172.21.15.27:0/2722732755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e6015d900 msgr2=0x7f6e6015ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:24.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.399+0000 7f6e65200640 1 --2- 172.21.15.27:0/2722732755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e6015d900 0x7f6e6015ddb0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f6e48009870 tx=0x7f6e4801c4d0 comp rx=0 tx=0).stop 2024-01-15T18:21:24.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.399+0000 7f6e65200640 1 -- 172.21.15.27:0/2722732755 shutdown_connections 2024-01-15T18:21:24.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.399+0000 7f6e65200640 1 --2- 172.21.15.27:0/2722732755 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6e60070ee0 0x7f6e60071390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:24.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.399+0000 7f6e65200640 1 --2- 172.21.15.27:0/2722732755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e6015d900 0x7f6e6015ddb0 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:24.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.399+0000 7f6e65200640 1 --2- 172.21.15.27:0/2722732755 >> v2:172.21.15.93:3300/0 conn(0x7f6e6015ad20 0x7f6e6015b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:24.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.399+0000 7f6e65200640 1 -- 172.21.15.27:0/2722732755 >> 172.21.15.27:0/2722732755 conn(0x7f6e600cb700 msgr2=0x7f6e600cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:24.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.399+0000 7f6e65200640 1 -- 172.21.15.27:0/2722732755 shutdown_connections 2024-01-15T18:21:24.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.399+0000 7f6e65200640 1 -- 172.21.15.27:0/2722732755 wait complete. 2024-01-15T18:21:24.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.403+0000 7f6e65200640 1 Processor -- start 2024-01-15T18:21:24.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.403+0000 7f6e65200640 1 -- start start 2024-01-15T18:21:24.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.403+0000 7f6e65200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6e60070ee0 0x7f6e601f62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:24.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.403+0000 7f6e65200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e6015ad20 0x7f6e6014c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:24.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.403+0000 7f6e65200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6e6015d900 0x7f6e60155160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:24.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.403+0000 7f6e65200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e60155990 con 0x7f6e6015ad20 2024-01-15T18:21:24.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.403+0000 7f6e65200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6014ce90 con 0x7f6e6015d900 2024-01-15T18:21:24.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.403+0000 7f6e65200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6014e890 con 0x7f6e60070ee0 2024-01-15T18:21:24.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.403+0000 7f6e5d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e6015ad20 0x7f6e6014c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:24.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.403+0000 7f6e5d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e6015ad20 0x7f6e6014c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51026/0 (socket says 172.21.15.27:51026) 2024-01-15T18:21:24.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.403+0000 7f6e5d400640 1 -- 172.21.15.27:0/92640986 learned_addr learned my addr 172.21.15.27:0/92640986 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:21:24.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.403+0000 7f6e5e800640 1 --2- 172.21.15.27:0/92640986 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6e6015d900 0x7f6e60155160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:24.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.403+0000 7f6e5de00640 1 --2- 172.21.15.27:0/92640986 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6e60070ee0 0x7f6e601f62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:24.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.403+0000 7f6e5d400640 1 -- 172.21.15.27:0/92640986 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6e60070ee0 msgr2=0x7f6e601f62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:24.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.403+0000 7f6e5d400640 1 --2- 172.21.15.27:0/92640986 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6e60070ee0 0x7f6e601f62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:24.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.403+0000 7f6e5d400640 1 -- 172.21.15.27:0/92640986 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6e6015d900 msgr2=0x7f6e60155160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:24.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.403+0000 7f6e5d400640 1 --2- 172.21.15.27:0/92640986 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6e6015d900 0x7f6e60155160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:24.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.403+0000 7f6e5d400640 1 -- 172.21.15.27:0/92640986 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e480094d0 con 0x7f6e6015ad20 2024-01-15T18:21:24.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.403+0000 7f6e5d400640 1 --2- 172.21.15.27:0/92640986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e6015ad20 0x7f6e6014c950 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f6e480ac040 tx=0x7f6e48019270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:24.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.403+0000 7f6e56000640 1 -- 172.21.15.27:0/92640986 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e480b4810 con 0x7f6e6015ad20 2024-01-15T18:21:24.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.403+0000 7f6e56000640 1 -- 172.21.15.27:0/92640986 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e480b49b0 con 0x7f6e6015ad20 2024-01-15T18:21:24.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.403+0000 7f6e65200640 1 -- 172.21.15.27:0/92640986 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e6014eb70 con 0x7f6e6015ad20 2024-01-15T18:21:24.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.403+0000 7f6e56000640 1 -- 172.21.15.27:0/92640986 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e480b4d60 con 0x7f6e6015ad20 2024-01-15T18:21:24.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.403+0000 7f6e65200640 1 -- 172.21.15.27:0/92640986 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e6014f130 con 0x7f6e6015ad20 2024-01-15T18:21:24.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.407+0000 7f6e65200640 1 -- 172.21.15.27:0/92640986 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e6014f320 con 0x7f6e6015ad20 2024-01-15T18:21:24.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.407+0000 7f6e56000640 1 -- 172.21.15.27:0/92640986 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 42) v1 ==== 87792+0+0 (secure 0 0 0) 0x7f6e480b3390 con 0x7f6e6015ad20 2024-01-15T18:21:24.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.407+0000 7f6e56000640 1 --2- 172.21.15.27:0/92640986 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6e38067af0 0x7f6e38069fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:24.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.407+0000 7f6e56000640 1 -- 172.21.15.27:0/92640986 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f6e480a7070 con 0x7f6e6015ad20 2024-01-15T18:21:24.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.407+0000 7f6e5de00640 1 --2- 172.21.15.27:0/92640986 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6e38067af0 0x7f6e38069fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:24.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.407+0000 7f6e5de00640 1 --2- 172.21.15.27:0/92640986 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6e38067af0 0x7f6e38069fb0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f6e60067870 tx=0x7f6e50008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:24.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.411+0000 7f6e56000640 1 -- 172.21.15.27:0/92640986 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6e480ec540 con 0x7f6e6015ad20 2024-01-15T18:21:24.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.595+0000 7f6e65200640 1 -- 172.21.15.27:0/92640986 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6e60151640 con 0x7f6e38067af0 2024-01-15T18:21:24.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.611+0000 7f6e56000640 1 -- 172.21.15.27:0/92640986 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f6e60151640 con 0x7f6e38067af0 2024-01-15T18:21:24.608 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:21:24.608 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2024-01-15T18:21:24.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-01-15T18:21:24.608 INFO:teuthology.orchestra.run.smithi027.stdout: "name": "ss1" 2024-01-15T18:21:24.608 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-01-15T18:21:24.608 INFO:teuthology.orchestra.run.smithi027.stdout:] 2024-01-15T18:21:24.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.615+0000 7f6e65200640 1 -- 172.21.15.27:0/92640986 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6e38067af0 msgr2=0x7f6e38069fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:24.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.615+0000 7f6e65200640 1 --2- 172.21.15.27:0/92640986 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6e38067af0 0x7f6e38069fb0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f6e60067870 tx=0x7f6e50008040 comp rx=0 tx=0).stop 2024-01-15T18:21:24.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.615+0000 7f6e65200640 1 -- 172.21.15.27:0/92640986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e6015ad20 msgr2=0x7f6e6014c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:24.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.615+0000 7f6e65200640 1 --2- 172.21.15.27:0/92640986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e6015ad20 0x7f6e6014c950 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f6e480ac040 tx=0x7f6e48019270 comp rx=0 tx=0).stop 2024-01-15T18:21:24.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.615+0000 7f6e65200640 1 -- 172.21.15.27:0/92640986 shutdown_connections 2024-01-15T18:21:24.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.615+0000 7f6e65200640 1 --2- 172.21.15.27:0/92640986 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6e38067af0 0x7f6e38069fb0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:24.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.615+0000 7f6e65200640 1 --2- 172.21.15.27:0/92640986 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6e6015d900 0x7f6e60155160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:24.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.615+0000 7f6e65200640 1 --2- 172.21.15.27:0/92640986 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6e6015ad20 0x7f6e6014c950 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:24.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.615+0000 7f6e65200640 1 --2- 172.21.15.27:0/92640986 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6e60070ee0 0x7f6e601f62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:24.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.615+0000 7f6e65200640 1 -- 172.21.15.27:0/92640986 >> 172.21.15.27:0/92640986 conn(0x7f6e600cb700 msgr2=0x7f6e60150f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:24.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.615+0000 7f6e65200640 1 -- 172.21.15.27:0/92640986 shutdown_connections 2024-01-15T18:21:24.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.615+0000 7f6e65200640 1 -- 172.21.15.27:0/92640986 wait complete. 2024-01-15T18:21:24.630 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:21:24.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.779+0000 7f422a200640 1 Processor -- start 2024-01-15T18:21:24.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.779+0000 7f422a200640 1 -- start start 2024-01-15T18:21:24.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.783+0000 7f422a200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f4224160cb0 0x7f4224161120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:24.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.783+0000 7f422a200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42240cde30 con 0x7f4224146a60 2024-01-15T18:21:24.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.783+0000 7f422a200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f422406fa50 con 0x7f422406a4f0 2024-01-15T18:21:24.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.783+0000 7f422a200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4224161820 con 0x7f4224160cb0 2024-01-15T18:21:24.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.783+0000 7f4223400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4224146a60 0x7f42240ac880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34590/0 (socket says 172.21.15.27:34590) 2024-01-15T18:21:24.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.783+0000 7f4223400640 1 -- 172.21.15.27:0/2619188530 learned_addr learned my addr 172.21.15.27:0/2619188530 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:21:24.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.783+0000 7f4222a00640 1 --2- 172.21.15.27:0/2619188530 >> v2:172.21.15.93:3300/0 conn(0x7f4224160cb0 0x7f4224161120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:24.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.783+0000 7f4222000640 1 -- 172.21.15.27:0/2619188530 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561364975 0 0) 0x7f42240cde30 con 0x7f4224146a60 2024-01-15T18:21:24.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.783+0000 7f4222000640 1 -- 172.21.15.27:0/2619188530 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4204003530 con 0x7f4224146a60 2024-01-15T18:21:24.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.783+0000 7f4222a00640 1 -- 172.21.15.27:0/2619188530 >> v1:172.21.15.93:6789/0 conn(0x7f422406a4f0 legacy=0x7f42240ce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:24.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.783+0000 7f4222a00640 1 -- 172.21.15.27:0/2619188530 >> v1:172.21.15.27:6789/0 conn(0x7f4224146a60 legacy=0x7f42240ac880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:24.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.783+0000 7f4222a00640 1 -- 172.21.15.27:0/2619188530 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4224161b00 con 0x7f4224160cb0 2024-01-15T18:21:24.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.787+0000 7f4222a00640 1 --2- 172.21.15.27:0/2619188530 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4224160cb0 0x7f4224161120 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f420c0097c0 tx=0x7f420c01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2498f23b582adee4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:24.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.787+0000 7f4222000640 1 -- 172.21.15.27:0/2619188530 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420c0ab680 con 0x7f4224160cb0 2024-01-15T18:21:24.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.787+0000 7f4222000640 1 -- 172.21.15.27:0/2619188530 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f420c0ab7e0 con 0x7f4224160cb0 2024-01-15T18:21:24.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.787+0000 7f4222000640 1 -- 172.21.15.27:0/2619188530 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420c0abb70 con 0x7f4224160cb0 2024-01-15T18:21:24.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.787+0000 7f422a200640 1 -- 172.21.15.27:0/2619188530 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4224160cb0 msgr2=0x7f4224161120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:24.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.787+0000 7f422a200640 1 --2- 172.21.15.27:0/2619188530 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4224160cb0 0x7f4224161120 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f420c0097c0 tx=0x7f420c01c480 comp rx=0 tx=0).stop 2024-01-15T18:21:24.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.787+0000 7f422a200640 1 -- 172.21.15.27:0/2619188530 shutdown_connections 2024-01-15T18:21:24.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.787+0000 7f422a200640 1 --2- 172.21.15.27:0/2619188530 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4224160cb0 0x7f4224161120 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:24.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.787+0000 7f422a200640 1 -- 172.21.15.27:0/2619188530 >> 172.21.15.27:0/2619188530 conn(0x7f42240cce10 msgr2=0x7f42240b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:24.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.787+0000 7f422a200640 1 -- 172.21.15.27:0/2619188530 shutdown_connections 2024-01-15T18:21:24.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.787+0000 7f422a200640 1 -- 172.21.15.27:0/2619188530 wait complete. 2024-01-15T18:21:24.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.787+0000 7f422a200640 1 Processor -- start 2024-01-15T18:21:24.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.787+0000 7f422a200640 1 -- start start 2024-01-15T18:21:24.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.791+0000 7f422a200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f422406a4f0 0x7f42240b3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:24.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.791+0000 7f422a200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4224146a60 0x7f42240b51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:24.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.791+0000 7f422a200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4224160cb0 0x7f422415ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:24.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.791+0000 7f422a200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42241596c0 con 0x7f4224160cb0 2024-01-15T18:21:24.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.791+0000 7f422a200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42240b4780 con 0x7f4224146a60 2024-01-15T18:21:24.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.791+0000 7f422a200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42240b4290 con 0x7f422406a4f0 2024-01-15T18:21:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.791+0000 7f4223e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4224160cb0 0x7f422415ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.791+0000 7f4223e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4224160cb0 0x7f422415ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51036/0 (socket says 172.21.15.27:51036) 2024-01-15T18:21:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.791+0000 7f4223e00640 1 -- 172.21.15.27:0/3364148073 learned_addr learned my addr 172.21.15.27:0/3364148073 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:21:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.791+0000 7f4222a00640 1 --2- 172.21.15.27:0/3364148073 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4224146a60 0x7f42240b51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.791+0000 7f4223400640 1 --2- 172.21.15.27:0/3364148073 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f422406a4f0 0x7f42240b3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:24.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.791+0000 7f4223e00640 1 -- 172.21.15.27:0/3364148073 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f422406a4f0 msgr2=0x7f42240b3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.791+0000 7f4223e00640 1 --2- 172.21.15.27:0/3364148073 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f422406a4f0 0x7f42240b3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.791+0000 7f4223e00640 1 -- 172.21.15.27:0/3364148073 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4224146a60 msgr2=0x7f42240b51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.791+0000 7f4223e00640 1 --2- 172.21.15.27:0/3364148073 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4224146a60 0x7f42240b51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.791+0000 7f4223e00640 1 -- 172.21.15.27:0/3364148073 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420c009420 con 0x7f4224160cb0 2024-01-15T18:21:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.791+0000 7f4223e00640 1 --2- 172.21.15.27:0/3364148073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4224160cb0 0x7f422415ed30 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f421c00c770 tx=0x7f421c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.791+0000 7f421b400640 1 -- 172.21.15.27:0/3364148073 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c003b80 con 0x7f4224160cb0 2024-01-15T18:21:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.791+0000 7f421b400640 1 -- 172.21.15.27:0/3364148073 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f421c003ce0 con 0x7f4224160cb0 2024-01-15T18:21:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.791+0000 7f422a200640 1 -- 172.21.15.27:0/3364148073 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f422406ae90 con 0x7f4224160cb0 2024-01-15T18:21:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.791+0000 7f421b400640 1 -- 172.21.15.27:0/3364148073 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421c00aa90 con 0x7f4224160cb0 2024-01-15T18:21:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.791+0000 7f422a200640 1 -- 172.21.15.27:0/3364148073 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42240b49f0 con 0x7f4224160cb0 2024-01-15T18:21:24.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.791+0000 7f422a200640 1 -- 172.21.15.27:0/3364148073 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f422404e850 con 0x7f4224160cb0 2024-01-15T18:21:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.795+0000 7f421b400640 1 -- 172.21.15.27:0/3364148073 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 42) v1 ==== 87792+0+0 (secure 0 0 0) 0x7f421c00abf0 con 0x7f4224160cb0 2024-01-15T18:21:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.795+0000 7f421b400640 1 --2- 172.21.15.27:0/3364148073 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f41fc067b20 0x7f41fc069fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.795+0000 7f421b400640 1 -- 172.21.15.27:0/3364148073 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f421c083100 con 0x7f4224160cb0 2024-01-15T18:21:24.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.795+0000 7f4223400640 1 --2- 172.21.15.27:0/3364148073 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f41fc067b20 0x7f41fc069fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:24.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.795+0000 7f4223400640 1 --2- 172.21.15.27:0/3364148073 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f41fc067b20 0x7f41fc069fe0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f42140011f0 tx=0x7f4214009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:24.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.799+0000 7f421b400640 1 -- 172.21.15.27:0/3364148073 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f421c051a20 con 0x7f4224160cb0 2024-01-15T18:21:24.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.983+0000 7f422a200640 1 -- 172.21.15.27:0/3364148073 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f422406d840 con 0x7f41fc067b20 2024-01-15T18:21:24.985 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:21:24.986+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:21:24.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:24.991+0000 7f421b400640 1 -- 172.21.15.27:0/3364148073 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f422406d840 con 0x7f41fc067b20 2024-01-15T18:21:24.986 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:21:24.986 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:21:24.994 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:21:26.995 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:21:27.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.159+0000 7f6b53200640 1 Processor -- start 2024-01-15T18:21:27.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.159+0000 7f6b53200640 1 -- start start 2024-01-15T18:21:27.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.159+0000 7f6b53200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6b4c1468d0 0x7f6b4c0ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:27.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.159+0000 7f6b53200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6b4c0cef30 0x7f6b4c0cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:27.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.159+0000 7f6b53200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b4c0ad920 con 0x7f6b4c0ccea0 2024-01-15T18:21:27.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.159+0000 7f6b53200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b4c0ac880 con 0x7f6b4c0cef30 2024-01-15T18:21:27.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.159+0000 7f6b53200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c0acb60 con 0x7f6b4c1468d0 2024-01-15T18:21:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.159+0000 7f6b4b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6b4c0cef30 0x7f6b4c0cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.159+0000 7f6b51200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6b4c0ccea0 0x7f6b4c160cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34606/0 (socket says 172.21.15.27:34606) 2024-01-15T18:21:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.159+0000 7f6b51200640 1 -- 172.21.15.27:0/2858471508 learned_addr learned my addr 172.21.15.27:0/2858471508 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:21:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.159+0000 7f6b4be00640 1 --2- 172.21.15.27:0/2858471508 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6b4c1468d0 0x7f6b4c0ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.159+0000 7f6b4b400640 1 -- 172.21.15.27:0/2858471508 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6b4c1468d0 msgr2=0x7f6b4c0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.159+0000 7f6b4b400640 1 --2- 172.21.15.27:0/2858471508 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6b4c1468d0 0x7f6b4c0ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.159+0000 7f6b4b400640 1 -- 172.21.15.27:0/2858471508 >> v1:172.21.15.27:6789/0 conn(0x7f6b4c0ccea0 legacy=0x7f6b4c160cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:27.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.159+0000 7f6b4aa00640 1 -- 172.21.15.27:0/2858471508 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660931415 0 0) 0x7f6b440014a0 con 0x7f6b4c0ccea0 2024-01-15T18:21:27.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.159+0000 7f6b4b400640 1 -- 172.21.15.27:0/2858471508 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b4c0ace40 con 0x7f6b4c0cef30 2024-01-15T18:21:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.163+0000 7f6b4b400640 1 --2- 172.21.15.27:0/2858471508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b4c0cef30 0x7f6b4c0cc960 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f6b340097f0 tx=0x7f6b3401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75e832bc6716e12d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.163+0000 7f6b4aa00640 1 -- 172.21.15.27:0/2858471508 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b340ab600 con 0x7f6b4c0cef30 2024-01-15T18:21:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.163+0000 7f6b4aa00640 1 -- 172.21.15.27:0/2858471508 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b340a7050 con 0x7f6b4c0cef30 2024-01-15T18:21:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.163+0000 7f6b4aa00640 1 -- 172.21.15.27:0/2858471508 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b340ab980 con 0x7f6b4c0cef30 2024-01-15T18:21:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.163+0000 7f6b53200640 1 -- 172.21.15.27:0/2858471508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b4c0cef30 msgr2=0x7f6b4c0cc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:27.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.163+0000 7f6b53200640 1 --2- 172.21.15.27:0/2858471508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b4c0cef30 0x7f6b4c0cc960 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f6b340097f0 tx=0x7f6b3401c450 comp rx=0 tx=0).stop 2024-01-15T18:21:27.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.163+0000 7f6b53200640 1 -- 172.21.15.27:0/2858471508 shutdown_connections 2024-01-15T18:21:27.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.163+0000 7f6b53200640 1 --2- 172.21.15.27:0/2858471508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b4c0cef30 0x7f6b4c0cc960 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:27.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.163+0000 7f6b53200640 1 --2- 172.21.15.27:0/2858471508 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6b4c1468d0 0x7f6b4c0ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:27.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.163+0000 7f6b53200640 1 -- 172.21.15.27:0/2858471508 >> 172.21.15.27:0/2858471508 conn(0x7f6b4c0cb700 msgr2=0x7f6b4c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:27.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.163+0000 7f6b53200640 1 -- 172.21.15.27:0/2858471508 shutdown_connections 2024-01-15T18:21:27.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.163+0000 7f6b53200640 1 -- 172.21.15.27:0/2858471508 wait complete. 2024-01-15T18:21:27.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.167+0000 7f6b53200640 1 Processor -- start 2024-01-15T18:21:27.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.167+0000 7f6b53200640 1 -- start start 2024-01-15T18:21:27.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.167+0000 7f6b53200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b4c0ccea0 0x7f6b4c0716b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:27.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.167+0000 7f6b53200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6b4c0cef30 0x7f6b4c15a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:27.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.167+0000 7f6b53200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6b4c1468d0 0x7f6b4c1601e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:27.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.167+0000 7f6b53200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c13cef0 con 0x7f6b4c0ccea0 2024-01-15T18:21:27.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.167+0000 7f6b53200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c070610 con 0x7f6b4c1468d0 2024-01-15T18:21:27.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.167+0000 7f6b53200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c0708f0 con 0x7f6b4c0cef30 2024-01-15T18:21:27.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.167+0000 7f6b4be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b4c0ccea0 0x7f6b4c0716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:27.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.167+0000 7f6b4be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b4c0ccea0 0x7f6b4c0716b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51066/0 (socket says 172.21.15.27:51066) 2024-01-15T18:21:27.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.167+0000 7f6b4be00640 1 -- 172.21.15.27:0/497377137 learned_addr learned my addr 172.21.15.27:0/497377137 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:21:27.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.167+0000 7f6b4b400640 1 --2- 172.21.15.27:0/497377137 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6b4c0cef30 0x7f6b4c15a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:27.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.167+0000 7f6b51200640 1 --2- 172.21.15.27:0/497377137 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6b4c1468d0 0x7f6b4c1601e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:27.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.167+0000 7f6b4be00640 1 -- 172.21.15.27:0/497377137 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6b4c0cef30 msgr2=0x7f6b4c15a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:27.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.167+0000 7f6b4be00640 1 --2- 172.21.15.27:0/497377137 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6b4c0cef30 0x7f6b4c15a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:27.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.167+0000 7f6b4be00640 1 -- 172.21.15.27:0/497377137 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6b4c1468d0 msgr2=0x7f6b4c1601e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:27.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.167+0000 7f6b4be00640 1 --2- 172.21.15.27:0/497377137 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6b4c1468d0 0x7f6b4c1601e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:27.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.167+0000 7f6b4be00640 1 -- 172.21.15.27:0/497377137 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b34009420 con 0x7f6b4c0ccea0 2024-01-15T18:21:27.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.167+0000 7f6b4be00640 1 --2- 172.21.15.27:0/497377137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b4c0ccea0 0x7f6b4c0716b0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c00d5f0 tx=0x7f6b3c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:27.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.167+0000 7f6b43e00640 1 -- 172.21.15.27:0/497377137 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c0125f0 con 0x7f6b4c0ccea0 2024-01-15T18:21:27.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.167+0000 7f6b43e00640 1 -- 172.21.15.27:0/497377137 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b3c012750 con 0x7f6b4c0ccea0 2024-01-15T18:21:27.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.167+0000 7f6b53200640 1 -- 172.21.15.27:0/497377137 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b4c070bd0 con 0x7f6b4c0ccea0 2024-01-15T18:21:27.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.167+0000 7f6b43e00640 1 -- 172.21.15.27:0/497377137 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c012ae0 con 0x7f6b4c0ccea0 2024-01-15T18:21:27.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.167+0000 7f6b53200640 1 -- 172.21.15.27:0/497377137 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b4c071190 con 0x7f6b4c0ccea0 2024-01-15T18:21:27.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.171+0000 7f6b53200640 1 -- 172.21.15.27:0/497377137 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b4c142b80 con 0x7f6b4c0ccea0 2024-01-15T18:21:27.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.171+0000 7f6b43e00640 1 -- 172.21.15.27:0/497377137 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7f6b3c012c40 con 0x7f6b4c0ccea0 2024-01-15T18:21:27.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.171+0000 7f6b43e00640 1 --2- 172.21.15.27:0/497377137 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6b24067b80 0x7f6b2406a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:27.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.171+0000 7f6b43e00640 1 -- 172.21.15.27:0/497377137 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f6b3c083780 con 0x7f6b4c0ccea0 2024-01-15T18:21:27.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.171+0000 7f6b4b400640 1 --2- 172.21.15.27:0/497377137 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6b24067b80 0x7f6b2406a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:27.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.171+0000 7f6b4b400640 1 --2- 172.21.15.27:0/497377137 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6b24067b80 0x7f6b2406a040 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f6b340ad000 tx=0x7f6b34008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:27.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.175+0000 7f6b43e00640 1 -- 172.21.15.27:0/497377137 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6b3c052140 con 0x7f6b4c0ccea0 2024-01-15T18:21:27.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.363+0000 7f6b53200640 1 -- 172.21.15.27:0/497377137 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6b4c13d240 con 0x7f6b24067b80 2024-01-15T18:21:27.365 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:21:27.366+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:21:27.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:27.371+0000 7f6b43e00640 1 -- 172.21.15.27:0/497377137 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6b4c13d240 con 0x7f6b24067b80 2024-01-15T18:21:27.366 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:21:27.366 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:21:27.374 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:21:28.293 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:21:28.297 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:21:29.376 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:21:29.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.591+0000 7f686a400640 1 Processor -- start 2024-01-15T18:21:29.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.591+0000 7f686a400640 1 -- start start 2024-01-15T18:21:29.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.591+0000 7f686a400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6864071ac0 0x7f68640b2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:29.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.591+0000 7f686a400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f68640ac880 0x7f68640b2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:29.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.591+0000 7f686a400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68640cbc80 con 0x7f686406b500 2024-01-15T18:21:29.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.591+0000 7f686a400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f686415d850 con 0x7f6864071ac0 2024-01-15T18:21:29.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.591+0000 7f686a400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f686415db00 con 0x7f68640ac880 2024-01-15T18:21:29.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.591+0000 7f6863400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f686406b500 0x7f686406b8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34622/0 (socket says 172.21.15.27:34622) 2024-01-15T18:21:29.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.591+0000 7f6863400640 1 -- 172.21.15.27:0/2070517398 learned_addr learned my addr 172.21.15.27:0/2070517398 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:21:29.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.591+0000 7f6862a00640 1 --2- 172.21.15.27:0/2070517398 >> v2:172.21.15.27:3300/0 conn(0x7f6864071ac0 0x7f68640b2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:29.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.591+0000 7f6863e00640 1 --2- 172.21.15.27:0/2070517398 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f68640ac880 0x7f68640b2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:29.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.591+0000 7f6862000640 1 -- 172.21.15.27:0/2070517398 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149621417 0 0) 0x7f68640cbc80 con 0x7f686406b500 2024-01-15T18:21:29.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.591+0000 7f6862000640 1 -- 172.21.15.27:0/2070517398 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6844003530 con 0x7f686406b500 2024-01-15T18:21:29.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.591+0000 7f6862a00640 1 -- 172.21.15.27:0/2070517398 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f68640ac880 msgr2=0x7f68640b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:29.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.591+0000 7f6862a00640 1 --2- 172.21.15.27:0/2070517398 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f68640ac880 0x7f68640b2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:29.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.591+0000 7f6862a00640 1 -- 172.21.15.27:0/2070517398 >> v1:172.21.15.27:6789/0 conn(0x7f686406b500 legacy=0x7f686406b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:29.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.591+0000 7f6862a00640 1 -- 172.21.15.27:0/2070517398 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f686415dde0 con 0x7f6864071ac0 2024-01-15T18:21:29.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.595+0000 7f6862a00640 1 --2- 172.21.15.27:0/2070517398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6864071ac0 0x7f68640b2660 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f684c0097f0 tx=0x7f684c01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3d7a62f9663c204 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:29.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.595+0000 7f6862000640 1 -- 172.21.15.27:0/2070517398 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f684c0ab600 con 0x7f6864071ac0 2024-01-15T18:21:29.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.595+0000 7f6862000640 1 -- 172.21.15.27:0/2070517398 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f684c0a7050 con 0x7f6864071ac0 2024-01-15T18:21:29.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.595+0000 7f6862000640 1 -- 172.21.15.27:0/2070517398 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f684c0ab980 con 0x7f6864071ac0 2024-01-15T18:21:29.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.599+0000 7f686a400640 1 -- 172.21.15.27:0/2070517398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6864071ac0 msgr2=0x7f68640b2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:29.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.599+0000 7f686a400640 1 --2- 172.21.15.27:0/2070517398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6864071ac0 0x7f68640b2660 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f684c0097f0 tx=0x7f684c01c3f0 comp rx=0 tx=0).stop 2024-01-15T18:21:29.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.599+0000 7f686a400640 1 -- 172.21.15.27:0/2070517398 shutdown_connections 2024-01-15T18:21:29.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.599+0000 7f686a400640 1 --2- 172.21.15.27:0/2070517398 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f68640ac880 0x7f68640b2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:29.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.599+0000 7f686a400640 1 --2- 172.21.15.27:0/2070517398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6864071ac0 0x7f68640b2660 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:29.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.599+0000 7f686a400640 1 -- 172.21.15.27:0/2070517398 >> 172.21.15.27:0/2070517398 conn(0x7f6864070ad0 msgr2=0x7f6864070ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:29.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.599+0000 7f686a400640 1 -- 172.21.15.27:0/2070517398 shutdown_connections 2024-01-15T18:21:29.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.599+0000 7f686a400640 1 -- 172.21.15.27:0/2070517398 wait complete. 2024-01-15T18:21:29.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.599+0000 7f686a400640 1 Processor -- start 2024-01-15T18:21:29.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.599+0000 7f686a400640 1 -- start start 2024-01-15T18:21:29.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.603+0000 7f686a400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f686406b500 0x7f6864159c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:29.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.603+0000 7f686a400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6864071ac0 0x7f686414eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:29.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.603+0000 7f686a400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68640ac880 0x7f686414f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:29.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.603+0000 7f686a400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686415a1a0 con 0x7f68640ac880 2024-01-15T18:21:29.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.603+0000 7f686a400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686413a7a0 con 0x7f6864071ac0 2024-01-15T18:21:29.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.603+0000 7f686a400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68640cead0 con 0x7f686406b500 2024-01-15T18:21:29.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.603+0000 7f6863e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68640ac880 0x7f686414f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:29.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.603+0000 7f6863e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68640ac880 0x7f686414f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51072/0 (socket says 172.21.15.27:51072) 2024-01-15T18:21:29.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.603+0000 7f6863e00640 1 -- 172.21.15.27:0/173427340 learned_addr learned my addr 172.21.15.27:0/173427340 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:21:29.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.603+0000 7f6863400640 1 --2- 172.21.15.27:0/173427340 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f686406b500 0x7f6864159c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:29.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.603+0000 7f6862a00640 1 --2- 172.21.15.27:0/173427340 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6864071ac0 0x7f686414eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:29.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.603+0000 7f6863e00640 1 -- 172.21.15.27:0/173427340 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f686406b500 msgr2=0x7f6864159c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:29.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.603+0000 7f6863e00640 1 --2- 172.21.15.27:0/173427340 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f686406b500 0x7f6864159c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:29.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.603+0000 7f6863e00640 1 -- 172.21.15.27:0/173427340 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6864071ac0 msgr2=0x7f686414eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:29.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.603+0000 7f6863e00640 1 --2- 172.21.15.27:0/173427340 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6864071ac0 0x7f686414eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:29.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.603+0000 7f6863e00640 1 -- 172.21.15.27:0/173427340 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684c009420 con 0x7f68640ac880 2024-01-15T18:21:29.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.603+0000 7f6863e00640 1 --2- 172.21.15.27:0/173427340 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68640ac880 0x7f686414f0b0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f685c007e30 tx=0x7f685c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:29.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.603+0000 7f685b400640 1 -- 172.21.15.27:0/173427340 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685c014620 con 0x7f68640ac880 2024-01-15T18:21:29.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.603+0000 7f686a400640 1 -- 172.21.15.27:0/173427340 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68640cedb0 con 0x7f68640ac880 2024-01-15T18:21:29.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.603+0000 7f686a400640 1 -- 172.21.15.27:0/173427340 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68641574e0 con 0x7f68640ac880 2024-01-15T18:21:29.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.603+0000 7f685b400640 1 -- 172.21.15.27:0/173427340 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685c00ee90 con 0x7f68640ac880 2024-01-15T18:21:29.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.603+0000 7f685b400640 1 -- 172.21.15.27:0/173427340 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685c014980 con 0x7f68640ac880 2024-01-15T18:21:29.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.607+0000 7f686a400640 1 -- 172.21.15.27:0/173427340 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6834005260 con 0x7f68640ac880 2024-01-15T18:21:29.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.607+0000 7f685b400640 1 -- 172.21.15.27:0/173427340 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7f685c014ae0 con 0x7f68640ac880 2024-01-15T18:21:29.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.611+0000 7f685b400640 1 --2- 172.21.15.27:0/173427340 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f683c067b80 0x7f683c06a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:29.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.611+0000 7f685b400640 1 -- 172.21.15.27:0/173427340 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f685c083e90 con 0x7f68640ac880 2024-01-15T18:21:29.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.611+0000 7f6863400640 1 --2- 172.21.15.27:0/173427340 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f683c067b80 0x7f683c06a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:29.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.611+0000 7f6863400640 1 --2- 172.21.15.27:0/173427340 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f683c067b80 0x7f683c06a040 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f68540011f0 tx=0x7f6854009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:29.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.615+0000 7f685b400640 1 -- 172.21.15.27:0/173427340 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f685c0526d0 con 0x7f68640ac880 2024-01-15T18:21:29.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.895+0000 7f686a400640 1 -- 172.21.15.27:0/173427340 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6834000c80 con 0x7f683c067b80 2024-01-15T18:21:29.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:29.911+0000 7f685b400640 1 -- 172.21.15.27:0/173427340 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6834000c80 con 0x7f683c067b80 2024-01-15T18:21:29.908 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:21:29.906+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:21:29.908 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:21:29.908 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:21:29.923 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:21:31.925 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:21:32.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.086+0000 7f5b31e00640 1 Processor -- start 2024-01-15T18:21:32.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.086+0000 7f5b31e00640 1 -- start start 2024-01-15T18:21:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.090+0000 7f5b31e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5b2c0ce760 0x7f5b2c0b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.090+0000 7f5b31e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2c0712b0 con 0x7f5b2c06c430 2024-01-15T18:21:32.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.090+0000 7f5b31e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2c0ccfb0 con 0x7f5b2c0b1db0 2024-01-15T18:21:32.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.090+0000 7f5b31e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b2c0cd290 con 0x7f5b2c0ce760 2024-01-15T18:21:32.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.090+0000 7f5b2b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5b2c0ce760 0x7f5b2c0b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:32.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.090+0000 7f5b2aa00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5b2c06c430 0x7f5b2c070ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:34632/0 (socket says 172.21.15.27:34632) 2024-01-15T18:21:32.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.090+0000 7f5b2aa00640 1 -- 172.21.15.27:0/876800723 learned_addr learned my addr 172.21.15.27:0/876800723 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:21:32.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.090+0000 7f5b2b400640 1 -- 172.21.15.27:0/876800723 >> v1:172.21.15.93:6789/0 conn(0x7f5b2c0b1db0 legacy=0x7f5b2c0ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:32.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.090+0000 7f5b2b400640 1 -- 172.21.15.27:0/876800723 >> v1:172.21.15.27:6789/0 conn(0x7f5b2c06c430 legacy=0x7f5b2c070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:32.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.090+0000 7f5b29600640 1 -- 172.21.15.27:0/876800723 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2223215796 0 0) 0x7f5b2c0712b0 con 0x7f5b2c06c430 2024-01-15T18:21:32.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.090+0000 7f5b2b400640 1 -- 172.21.15.27:0/876800723 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b2c0cd570 con 0x7f5b2c0ce760 2024-01-15T18:21:32.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.090+0000 7f5b2b400640 1 --2- 172.21.15.27:0/876800723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b2c0ce760 0x7f5b2c0b2240 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f5b24009dc0 tx=0x7f5b2401d980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd38cbe062c23364 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:32.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.094+0000 7f5b29600640 1 -- 172.21.15.27:0/876800723 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b240a69c0 con 0x7f5b2c0ce760 2024-01-15T18:21:32.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.094+0000 7f5b29600640 1 -- 172.21.15.27:0/876800723 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b240a6b20 con 0x7f5b2c0ce760 2024-01-15T18:21:32.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.094+0000 7f5b29600640 1 -- 172.21.15.27:0/876800723 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b240ae440 con 0x7f5b2c0ce760 2024-01-15T18:21:32.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.094+0000 7f5b31e00640 1 -- 172.21.15.27:0/876800723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b2c0ce760 msgr2=0x7f5b2c0b2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:32.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.094+0000 7f5b31e00640 1 --2- 172.21.15.27:0/876800723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b2c0ce760 0x7f5b2c0b2240 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f5b24009dc0 tx=0x7f5b2401d980 comp rx=0 tx=0).stop 2024-01-15T18:21:32.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.094+0000 7f5b31e00640 1 -- 172.21.15.27:0/876800723 shutdown_connections 2024-01-15T18:21:32.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.094+0000 7f5b31e00640 1 --2- 172.21.15.27:0/876800723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b2c0ce760 0x7f5b2c0b2240 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:32.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.094+0000 7f5b31e00640 1 -- 172.21.15.27:0/876800723 >> 172.21.15.27:0/876800723 conn(0x7f5b2c0cb700 msgr2=0x7f5b2c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:32.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.094+0000 7f5b31e00640 1 -- 172.21.15.27:0/876800723 shutdown_connections 2024-01-15T18:21:32.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.094+0000 7f5b31e00640 1 -- 172.21.15.27:0/876800723 wait complete. 2024-01-15T18:21:32.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.094+0000 7f5b31e00640 1 Processor -- start 2024-01-15T18:21:32.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.094+0000 7f5b31e00640 1 -- start start 2024-01-15T18:21:32.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.094+0000 7f5b31e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b2c06c430 0x7f5b2c145f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:32.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.094+0000 7f5b31e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b2c0b1db0 0x7f5b2c153b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:32.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.094+0000 7f5b31e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b2c0ce760 0x7f5b2c1562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:32.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.094+0000 7f5b31e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c153670 con 0x7f5b2c06c430 2024-01-15T18:21:32.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.094+0000 7f5b31e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c152db0 con 0x7f5b2c0b1db0 2024-01-15T18:21:32.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.094+0000 7f5b31e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c153020 con 0x7f5b2c0ce760 2024-01-15T18:21:32.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.098+0000 7f5b2aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b2c06c430 0x7f5b2c145f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:32.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.098+0000 7f5b2aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b2c06c430 0x7f5b2c145f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51102/0 (socket says 172.21.15.27:51102) 2024-01-15T18:21:32.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.098+0000 7f5b2aa00640 1 -- 172.21.15.27:0/3251324233 learned_addr learned my addr 172.21.15.27:0/3251324233 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:21:32.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.098+0000 7f5b2a000640 1 --2- 172.21.15.27:0/3251324233 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b2c0b1db0 0x7f5b2c153b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:32.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.098+0000 7f5b2b400640 1 --2- 172.21.15.27:0/3251324233 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b2c0ce760 0x7f5b2c1562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:32.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.098+0000 7f5b2aa00640 1 -- 172.21.15.27:0/3251324233 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b2c0ce760 msgr2=0x7f5b2c1562d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:32.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.098+0000 7f5b2aa00640 1 --2- 172.21.15.27:0/3251324233 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b2c0ce760 0x7f5b2c1562d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:32.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.098+0000 7f5b2aa00640 1 -- 172.21.15.27:0/3251324233 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b2c0b1db0 msgr2=0x7f5b2c153b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:32.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.098+0000 7f5b2aa00640 1 --2- 172.21.15.27:0/3251324233 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b2c0b1db0 0x7f5b2c153b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:32.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.098+0000 7f5b2aa00640 1 -- 172.21.15.27:0/3251324233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b24009a70 con 0x7f5b2c06c430 2024-01-15T18:21:32.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.098+0000 7f5b2aa00640 1 --2- 172.21.15.27:0/3251324233 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b2c06c430 0x7f5b2c145f30 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c00c4d0 tx=0x7f5b1c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:32.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.098+0000 7f5b23400640 1 -- 172.21.15.27:0/3251324233 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c0074e0 con 0x7f5b2c06c430 2024-01-15T18:21:32.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.098+0000 7f5b23400640 1 -- 172.21.15.27:0/3251324233 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1c007640 con 0x7f5b2c06c430 2024-01-15T18:21:32.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.098+0000 7f5b31e00640 1 -- 172.21.15.27:0/3251324233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b2c153300 con 0x7f5b2c06c430 2024-01-15T18:21:32.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.098+0000 7f5b23400640 1 -- 172.21.15.27:0/3251324233 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c004a30 con 0x7f5b2c06c430 2024-01-15T18:21:32.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.098+0000 7f5b31e00640 1 -- 172.21.15.27:0/3251324233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b2c158b20 con 0x7f5b2c06c430 2024-01-15T18:21:32.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.098+0000 7f5b31e00640 1 -- 172.21.15.27:0/3251324233 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b2c04e850 con 0x7f5b2c06c430 2024-01-15T18:21:32.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.098+0000 7f5b23400640 1 -- 172.21.15.27:0/3251324233 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7f5b1c004b90 con 0x7f5b2c06c430 2024-01-15T18:21:32.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.102+0000 7f5b23400640 1 --2- 172.21.15.27:0/3251324233 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5b04067b80 0x7f5b0406a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:32.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.102+0000 7f5b23400640 1 -- 172.21.15.27:0/3251324233 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f5b1c082870 con 0x7f5b2c06c430 2024-01-15T18:21:32.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.102+0000 7f5b2a000640 1 --2- 172.21.15.27:0/3251324233 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5b04067b80 0x7f5b0406a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:32.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.102+0000 7f5b2a000640 1 --2- 172.21.15.27:0/3251324233 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5b04067b80 0x7f5b0406a040 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c14a5c0 tx=0x7f5b14009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:32.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.102+0000 7f5b23400640 1 -- 172.21.15.27:0/3251324233 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5b1c051230 con 0x7f5b2c06c430 2024-01-15T18:21:32.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.290+0000 7f5b31e00640 1 -- 172.21.15.27:0/3251324233 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5b2c158da0 con 0x7f5b04067b80 2024-01-15T18:21:32.293 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:21:32.294+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:21:32.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:32.298+0000 7f5b23400640 1 -- 172.21.15.27:0/3251324233 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5b2c158da0 con 0x7f5b04067b80 2024-01-15T18:21:32.294 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:21:32.294 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:21:32.303 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:21:34.304 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:21:34.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.466+0000 7f8925400640 1 Processor -- start 2024-01-15T18:21:34.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.466+0000 7f8925400640 1 -- start start 2024-01-15T18:21:34.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.466+0000 7f8925400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89200b1db0 0x7f89200b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:34.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.466+0000 7f8925400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f89200ce050 0x7f89200ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:34.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.466+0000 7f8925400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89200712b0 con 0x7f892006c430 2024-01-15T18:21:34.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.466+0000 7f8925400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89200cea20 con 0x7f89200ce050 2024-01-15T18:21:34.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.466+0000 7f8925400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89200ccfb0 con 0x7f89200b1db0 2024-01-15T18:21:34.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.470+0000 7f891e800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f892006c430 0x7f8920070ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40804/0 (socket says 172.21.15.27:40804) 2024-01-15T18:21:34.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.470+0000 7f891e800640 1 -- 172.21.15.27:0/563011657 learned_addr learned my addr 172.21.15.27:0/563011657 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:21:34.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.470+0000 7f891de00640 1 --2- 172.21.15.27:0/563011657 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89200b1db0 0x7f89200b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:34.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.470+0000 7f891f200640 1 --2- 172.21.15.27:0/563011657 >> v2:172.21.15.93:3300/0 conn(0x7f89200ce050 0x7f89200ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:34.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.470+0000 7f891d400640 1 -- 172.21.15.27:0/563011657 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478659634 0 0) 0x7f89200712b0 con 0x7f892006c430 2024-01-15T18:21:34.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.470+0000 7f891d400640 1 -- 172.21.15.27:0/563011657 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8900003530 con 0x7f892006c430 2024-01-15T18:21:34.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.470+0000 7f891de00640 1 -- 172.21.15.27:0/563011657 >> v2:172.21.15.93:3300/0 conn(0x7f89200ce050 msgr2=0x7f89200ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:34.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.470+0000 7f891de00640 1 --2- 172.21.15.27:0/563011657 >> v2:172.21.15.93:3300/0 conn(0x7f89200ce050 0x7f89200ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:34.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.470+0000 7f891de00640 1 -- 172.21.15.27:0/563011657 >> v1:172.21.15.27:6789/0 conn(0x7f892006c430 legacy=0x7f8920070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:34.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.470+0000 7f891de00640 1 -- 172.21.15.27:0/563011657 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89200cd290 con 0x7f89200b1db0 2024-01-15T18:21:34.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.470+0000 7f891f200640 1 --2- 172.21.15.27:0/563011657 >> v2:172.21.15.93:3300/0 conn(0x7f89200ce050 0x7f89200ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:21:34.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.474+0000 7f891de00640 1 --2- 172.21.15.27:0/563011657 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89200b1db0 0x7f89200b2210 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f89080097c0 tx=0x7f890801c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee831caf8997e005 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:34.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.474+0000 7f891d400640 1 -- 172.21.15.27:0/563011657 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89080ab600 con 0x7f89200b1db0 2024-01-15T18:21:34.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.474+0000 7f891d400640 1 -- 172.21.15.27:0/563011657 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89080a7050 con 0x7f89200b1db0 2024-01-15T18:21:34.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.474+0000 7f891d400640 1 -- 172.21.15.27:0/563011657 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89080ab980 con 0x7f89200b1db0 2024-01-15T18:21:34.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.474+0000 7f8925400640 1 -- 172.21.15.27:0/563011657 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89200b1db0 msgr2=0x7f89200b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:34.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.474+0000 7f8925400640 1 --2- 172.21.15.27:0/563011657 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89200b1db0 0x7f89200b2210 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f89080097c0 tx=0x7f890801c3f0 comp rx=0 tx=0).stop 2024-01-15T18:21:34.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.474+0000 7f8925400640 1 -- 172.21.15.27:0/563011657 shutdown_connections 2024-01-15T18:21:34.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.474+0000 7f8925400640 1 --2- 172.21.15.27:0/563011657 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89200b1db0 0x7f89200b2210 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:34.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.474+0000 7f8925400640 1 --2- 172.21.15.27:0/563011657 >> v2:172.21.15.93:3300/0 conn(0x7f89200ce050 0x7f89200ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:34.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.474+0000 7f8925400640 1 -- 172.21.15.27:0/563011657 >> 172.21.15.27:0/563011657 conn(0x7f89200cb700 msgr2=0x7f89200cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:34.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.474+0000 7f8925400640 1 -- 172.21.15.27:0/563011657 shutdown_connections 2024-01-15T18:21:34.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.474+0000 7f8925400640 1 -- 172.21.15.27:0/563011657 wait complete. 2024-01-15T18:21:34.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.474+0000 7f8925400640 1 Processor -- start 2024-01-15T18:21:34.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.474+0000 7f8925400640 1 -- start start 2024-01-15T18:21:34.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.478+0000 7f8925400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892006c430 0x7f89201558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:34.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.478+0000 7f8925400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f89200b1db0 0x7f892014cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:34.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.478+0000 7f8925400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89200ce050 0x7f892014d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:34.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.478+0000 7f8925400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8920155070 con 0x7f892006c430 2024-01-15T18:21:34.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.478+0000 7f8925400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8920155de0 con 0x7f89200b1db0 2024-01-15T18:21:34.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.478+0000 7f8925400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8920157db0 con 0x7f89200ce050 2024-01-15T18:21:34.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.478+0000 7f891e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892006c430 0x7f89201558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:34.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.478+0000 7f891e800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892006c430 0x7f89201558a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43512/0 (socket says 172.21.15.27:43512) 2024-01-15T18:21:34.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.478+0000 7f891e800640 1 -- 172.21.15.27:0/3573789356 learned_addr learned my addr 172.21.15.27:0/3573789356 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:21:34.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.478+0000 7f891de00640 1 --2- 172.21.15.27:0/3573789356 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f89200b1db0 0x7f892014cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:34.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.478+0000 7f891f200640 1 --2- 172.21.15.27:0/3573789356 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89200ce050 0x7f892014d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:34.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.478+0000 7f891e800640 1 -- 172.21.15.27:0/3573789356 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89200ce050 msgr2=0x7f892014d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:34.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.478+0000 7f891e800640 1 --2- 172.21.15.27:0/3573789356 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f89200ce050 0x7f892014d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:34.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.478+0000 7f891e800640 1 -- 172.21.15.27:0/3573789356 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f89200b1db0 msgr2=0x7f892014cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:34.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.478+0000 7f891e800640 1 --2- 172.21.15.27:0/3573789356 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f89200b1db0 0x7f892014cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:34.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.478+0000 7f891e800640 1 -- 172.21.15.27:0/3573789356 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8918009a70 con 0x7f892006c430 2024-01-15T18:21:34.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.478+0000 7f891e800640 1 --2- 172.21.15.27:0/3573789356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f892006c430 0x7f89201558a0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f891000c500 tx=0x7f891000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:34.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.478+0000 7f8916a00640 1 -- 172.21.15.27:0/3573789356 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8910005cf0 con 0x7f892006c430 2024-01-15T18:21:34.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.478+0000 7f8916a00640 1 -- 172.21.15.27:0/3573789356 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8910005e50 con 0x7f892006c430 2024-01-15T18:21:34.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.478+0000 7f8925400640 1 -- 172.21.15.27:0/3573789356 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8908009420 con 0x7f892006c430 2024-01-15T18:21:34.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.478+0000 7f8916a00640 1 -- 172.21.15.27:0/3573789356 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89100072f0 con 0x7f892006c430 2024-01-15T18:21:34.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.478+0000 7f8925400640 1 -- 172.21.15.27:0/3573789356 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8920154d90 con 0x7f892006c430 2024-01-15T18:21:34.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.478+0000 7f8925400640 1 -- 172.21.15.27:0/3573789356 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f892004e850 con 0x7f892006c430 2024-01-15T18:21:34.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.478+0000 7f8916a00640 1 -- 172.21.15.27:0/3573789356 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7f8910007450 con 0x7f892006c430 2024-01-15T18:21:34.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.482+0000 7f8916a00640 1 --2- 172.21.15.27:0/3573789356 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88f8067b80 0x7f88f806a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:34.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.482+0000 7f8916a00640 1 -- 172.21.15.27:0/3573789356 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f8910082900 con 0x7f892006c430 2024-01-15T18:21:34.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.482+0000 7f891de00640 1 --2- 172.21.15.27:0/3573789356 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88f8067b80 0x7f88f806a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:34.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.482+0000 7f891de00640 1 --2- 172.21.15.27:0/3573789356 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88f8067b80 0x7f88f806a040 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f89080ad000 tx=0x7f8908008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:34.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.486+0000 7f8916a00640 1 -- 172.21.15.27:0/3573789356 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f89100512c0 con 0x7f892006c430 2024-01-15T18:21:34.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.670+0000 7f8925400640 1 -- 172.21.15.27:0/3573789356 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f89200b49c0 con 0x7f88f8067b80 2024-01-15T18:21:34.674 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:21:34.674+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:21:34.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:34.678+0000 7f8916a00640 1 -- 172.21.15.27:0/3573789356 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f89080ab6a0 con 0x7f88f8067b80 2024-01-15T18:21:34.675 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:21:34.675 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:21:34.683 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:21:36.684 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:21:36.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.846+0000 7fd841600640 1 Processor -- start 2024-01-15T18:21:36.842 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.846+0000 7fd841600640 1 -- start start 2024-01-15T18:21:36.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.846+0000 7fd841600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd83c09a5b0 0x7fd83c0cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:36.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.846+0000 7fd841600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd83c0ccd50 con 0x7fd83c099a10 2024-01-15T18:21:36.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.846+0000 7fd841600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd83c0cebc0 con 0x7fd83c15b350 2024-01-15T18:21:36.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.846+0000 7fd841600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd83c0ceea0 con 0x7fd83c09a5b0 2024-01-15T18:21:36.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.846+0000 7fd83a000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd83c09a5b0 0x7fd83c0cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:36.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.846+0000 7fd83b400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd83c099a10 0x7fd83c099e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40806/0 (socket says 172.21.15.27:40806) 2024-01-15T18:21:36.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.846+0000 7fd83b400640 1 -- 172.21.15.27:0/2167677161 learned_addr learned my addr 172.21.15.27:0/2167677161 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:21:36.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.846+0000 7fd83a000640 1 -- 172.21.15.27:0/2167677161 >> v1:172.21.15.93:6789/0 conn(0x7fd83c15b350 legacy=0x7fd83c0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:36.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.846+0000 7fd83a000640 1 -- 172.21.15.27:0/2167677161 >> v1:172.21.15.27:6789/0 conn(0x7fd83c099a10 legacy=0x7fd83c099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:36.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.846+0000 7fd839600640 1 -- 172.21.15.27:0/2167677161 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729254321 0 0) 0x7fd83c0ccd50 con 0x7fd83c099a10 2024-01-15T18:21:36.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.846+0000 7fd83a000640 1 -- 172.21.15.27:0/2167677161 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83c0cf180 con 0x7fd83c09a5b0 2024-01-15T18:21:36.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.850+0000 7fd83a000640 1 --2- 172.21.15.27:0/2167677161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd83c09a5b0 0x7fd83c0cc500 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fd8240097c0 tx=0x7fd82401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff51c748a6a2932 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:36.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.850+0000 7fd839600640 1 -- 172.21.15.27:0/2167677161 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8240ad680 con 0x7fd83c09a5b0 2024-01-15T18:21:36.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.850+0000 7fd839600640 1 -- 172.21.15.27:0/2167677161 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8240ad7e0 con 0x7fd83c09a5b0 2024-01-15T18:21:36.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.850+0000 7fd839600640 1 -- 172.21.15.27:0/2167677161 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8240adb70 con 0x7fd83c09a5b0 2024-01-15T18:21:36.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.850+0000 7fd841600640 1 -- 172.21.15.27:0/2167677161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd83c09a5b0 msgr2=0x7fd83c0cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:36.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.850+0000 7fd841600640 1 --2- 172.21.15.27:0/2167677161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd83c09a5b0 0x7fd83c0cc500 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fd8240097c0 tx=0x7fd82401c3c0 comp rx=0 tx=0).stop 2024-01-15T18:21:36.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.850+0000 7fd841600640 1 -- 172.21.15.27:0/2167677161 shutdown_connections 2024-01-15T18:21:36.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.850+0000 7fd841600640 1 --2- 172.21.15.27:0/2167677161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd83c09a5b0 0x7fd83c0cc500 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:36.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.850+0000 7fd841600640 1 -- 172.21.15.27:0/2167677161 >> 172.21.15.27:0/2167677161 conn(0x7fd83c0b2a80 msgr2=0x7fd83c0b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:36.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.850+0000 7fd841600640 1 -- 172.21.15.27:0/2167677161 shutdown_connections 2024-01-15T18:21:36.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.850+0000 7fd841600640 1 -- 172.21.15.27:0/2167677161 wait complete. 2024-01-15T18:21:36.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.854+0000 7fd841600640 1 Processor -- start 2024-01-15T18:21:36.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.854+0000 7fd841600640 1 -- start start 2024-01-15T18:21:36.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.854+0000 7fd841600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd83c099a10 0x7fd83c14e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:36.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.854+0000 7fd841600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd83c09a5b0 0x7fd83c14fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:36.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.854+0000 7fd841600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd83c15b350 0x7fd83c06e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:36.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.854+0000 7fd841600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c0b4d30 con 0x7fd83c15b350 2024-01-15T18:21:36.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.854+0000 7fd841600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c1511a0 con 0x7fd83c099a10 2024-01-15T18:21:36.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.854+0000 7fd841600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c150900 con 0x7fd83c09a5b0 2024-01-15T18:21:36.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.854+0000 7fd83b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd83c15b350 0x7fd83c06e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:36.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.854+0000 7fd83b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd83c15b350 0x7fd83c06e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43526/0 (socket says 172.21.15.27:43526) 2024-01-15T18:21:36.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.854+0000 7fd83b400640 1 -- 172.21.15.27:0/3599000922 learned_addr learned my addr 172.21.15.27:0/3599000922 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:21:36.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.854+0000 7fd83a000640 1 --2- 172.21.15.27:0/3599000922 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd83c09a5b0 0x7fd83c14fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:36.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.854+0000 7fd83aa00640 1 --2- 172.21.15.27:0/3599000922 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd83c099a10 0x7fd83c14e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:36.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.854+0000 7fd83b400640 1 -- 172.21.15.27:0/3599000922 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd83c09a5b0 msgr2=0x7fd83c14fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:36.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.854+0000 7fd83b400640 1 --2- 172.21.15.27:0/3599000922 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd83c09a5b0 0x7fd83c14fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:36.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.854+0000 7fd83b400640 1 -- 172.21.15.27:0/3599000922 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd83c099a10 msgr2=0x7fd83c14e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:36.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.854+0000 7fd83b400640 1 --2- 172.21.15.27:0/3599000922 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd83c099a10 0x7fd83c14e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:36.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.854+0000 7fd83b400640 1 -- 172.21.15.27:0/3599000922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd824009420 con 0x7fd83c15b350 2024-01-15T18:21:36.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.854+0000 7fd83b400640 1 --2- 172.21.15.27:0/3599000922 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd83c15b350 0x7fd83c06e4d0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fd834001570 tx=0x7fd83400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:36.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.854+0000 7fd832a00640 1 -- 172.21.15.27:0/3599000922 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83400a640 con 0x7fd83c15b350 2024-01-15T18:21:36.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.854+0000 7fd832a00640 1 -- 172.21.15.27:0/3599000922 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83400a7c0 con 0x7fd83c15b350 2024-01-15T18:21:36.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.854+0000 7fd841600640 1 -- 172.21.15.27:0/3599000922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd83c150be0 con 0x7fd83c15b350 2024-01-15T18:21:36.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.854+0000 7fd832a00640 1 -- 172.21.15.27:0/3599000922 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83400ab70 con 0x7fd83c15b350 2024-01-15T18:21:36.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.858+0000 7fd841600640 1 -- 172.21.15.27:0/3599000922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd83c14ea80 con 0x7fd83c15b350 2024-01-15T18:21:36.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.858+0000 7fd841600640 1 -- 172.21.15.27:0/3599000922 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd83c0d06a0 con 0x7fd83c15b350 2024-01-15T18:21:36.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.858+0000 7fd832a00640 1 -- 172.21.15.27:0/3599000922 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7fd83400acd0 con 0x7fd83c15b350 2024-01-15T18:21:36.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.858+0000 7fd832a00640 1 --2- 172.21.15.27:0/3599000922 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd814067b80 0x7fd81406a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:36.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.858+0000 7fd832a00640 1 -- 172.21.15.27:0/3599000922 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fd834082c00 con 0x7fd83c15b350 2024-01-15T18:21:36.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.858+0000 7fd83aa00640 1 --2- 172.21.15.27:0/3599000922 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd814067b80 0x7fd81406a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:36.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.858+0000 7fd83aa00640 1 --2- 172.21.15.27:0/3599000922 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd814067b80 0x7fd81406a040 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fd82c000f70 tx=0x7fd82c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:36.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:36.862+0000 7fd832a00640 1 -- 172.21.15.27:0/3599000922 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd834051510 con 0x7fd83c15b350 2024-01-15T18:21:37.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:37.050+0000 7fd841600640 1 -- 172.21.15.27:0/3599000922 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd83c169080 con 0x7fd814067b80 2024-01-15T18:21:37.052 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:21:37.054+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:21:37.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:37.054+0000 7fd832a00640 1 -- 172.21.15.27:0/3599000922 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd82c0012d0 con 0x7fd814067b80 2024-01-15T18:21:37.052 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:21:37.053 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:21:37.061 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:21:39.063 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:21:39.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.222+0000 7fba5a600640 1 Processor -- start 2024-01-15T18:21:39.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.226+0000 7fba5a600640 1 -- start start 2024-01-15T18:21:39.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.226+0000 7fba5a600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fba5415f550 0x7fba540cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:39.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.226+0000 7fba5a600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fba540cf110 0x7fba540701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:39.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.226+0000 7fba5a600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba540b1cb0 con 0x7fba54158900 2024-01-15T18:21:39.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.226+0000 7fba5a600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba540cf580 con 0x7fba540cf110 2024-01-15T18:21:39.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.226+0000 7fba5a600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba5415fa00 con 0x7fba5415f550 2024-01-15T18:21:39.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.226+0000 7fba53e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fba540cf110 0x7fba540701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:39.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.226+0000 7fba53400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fba54158900 0x7fba5415bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40808/0 (socket says 172.21.15.27:40808) 2024-01-15T18:21:39.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.226+0000 7fba53400640 1 -- 172.21.15.27:0/4121921988 learned_addr learned my addr 172.21.15.27:0/4121921988 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:21:39.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.226+0000 7fba52a00640 1 --2- 172.21.15.27:0/4121921988 >> v2:172.21.15.93:3300/0 conn(0x7fba5415f550 0x7fba540cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:39.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.226+0000 7fba53e00640 1 -- 172.21.15.27:0/4121921988 >> v2:172.21.15.93:3300/0 conn(0x7fba5415f550 msgr2=0x7fba540cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:39.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.226+0000 7fba53e00640 1 --2- 172.21.15.27:0/4121921988 >> v2:172.21.15.93:3300/0 conn(0x7fba5415f550 0x7fba540cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:39.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.226+0000 7fba53e00640 1 -- 172.21.15.27:0/4121921988 >> v1:172.21.15.27:6789/0 conn(0x7fba54158900 legacy=0x7fba5415bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:39.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.226+0000 7fba53e00640 1 -- 172.21.15.27:0/4121921988 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba540b1f00 con 0x7fba540cf110 2024-01-15T18:21:39.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.230+0000 7fba53e00640 1 --2- 172.21.15.27:0/4121921988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba540cf110 0x7fba540701e0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7fba4c009d90 tx=0x7fba4c01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd9a4d66043b95d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:39.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.230+0000 7fba52000640 1 -- 172.21.15.27:0/4121921988 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba54070f30 con 0x7fba540cf110 2024-01-15T18:21:39.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.230+0000 7fba52000640 1 -- 172.21.15.27:0/4121921988 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4c0a59a0 con 0x7fba540cf110 2024-01-15T18:21:39.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.230+0000 7fba52000640 1 -- 172.21.15.27:0/4121921988 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4c0a5d30 con 0x7fba540cf110 2024-01-15T18:21:39.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.230+0000 7fba5a600640 1 -- 172.21.15.27:0/4121921988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba540cf110 msgr2=0x7fba540701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:39.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.230+0000 7fba5a600640 1 --2- 172.21.15.27:0/4121921988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba540cf110 0x7fba540701e0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7fba4c009d90 tx=0x7fba4c01c8e0 comp rx=0 tx=0).stop 2024-01-15T18:21:39.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.230+0000 7fba5a600640 1 -- 172.21.15.27:0/4121921988 shutdown_connections 2024-01-15T18:21:39.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.230+0000 7fba5a600640 1 --2- 172.21.15.27:0/4121921988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba540cf110 0x7fba540701e0 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:39.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.230+0000 7fba5a600640 1 --2- 172.21.15.27:0/4121921988 >> v2:172.21.15.93:3300/0 conn(0x7fba5415f550 0x7fba540cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:39.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.230+0000 7fba5a600640 1 -- 172.21.15.27:0/4121921988 >> 172.21.15.27:0/4121921988 conn(0x7fba540cb700 msgr2=0x7fba540cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:39.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.230+0000 7fba5a600640 1 -- 172.21.15.27:0/4121921988 shutdown_connections 2024-01-15T18:21:39.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.230+0000 7fba5a600640 1 -- 172.21.15.27:0/4121921988 wait complete. 2024-01-15T18:21:39.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.230+0000 7fba5a600640 1 Processor -- start 2024-01-15T18:21:39.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.234+0000 7fba5a600640 1 -- start start 2024-01-15T18:21:39.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.234+0000 7fba5a600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fba540cf110 0x7fba540ade30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:39.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.234+0000 7fba5a600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba54158900 0x7fba54153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:39.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.234+0000 7fba5a600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fba5415f550 0x7fba54154460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:39.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.234+0000 7fba5a600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5413be60 con 0x7fba54158900 2024-01-15T18:21:39.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.234+0000 7fba5a600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5413b190 con 0x7fba540cf110 2024-01-15T18:21:39.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.234+0000 7fba5a600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba540ae370 con 0x7fba5415f550 2024-01-15T18:21:39.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.234+0000 7fba52a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba54158900 0x7fba54153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:39.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.234+0000 7fba52a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba54158900 0x7fba54153f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43546/0 (socket says 172.21.15.27:43546) 2024-01-15T18:21:39.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.234+0000 7fba52a00640 1 -- 172.21.15.27:0/3829200850 learned_addr learned my addr 172.21.15.27:0/3829200850 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:21:39.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.234+0000 7fba53400640 1 --2- 172.21.15.27:0/3829200850 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fba540cf110 0x7fba540ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:39.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.234+0000 7fba53e00640 1 --2- 172.21.15.27:0/3829200850 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fba5415f550 0x7fba54154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:39.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.234+0000 7fba52a00640 1 -- 172.21.15.27:0/3829200850 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fba5415f550 msgr2=0x7fba54154460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:39.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.234+0000 7fba52a00640 1 --2- 172.21.15.27:0/3829200850 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fba5415f550 0x7fba54154460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:39.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.234+0000 7fba52a00640 1 -- 172.21.15.27:0/3829200850 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fba540cf110 msgr2=0x7fba540ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:39.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.234+0000 7fba52a00640 1 --2- 172.21.15.27:0/3829200850 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fba540cf110 0x7fba540ade30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:39.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.234+0000 7fba52a00640 1 -- 172.21.15.27:0/3829200850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba4c009a70 con 0x7fba54158900 2024-01-15T18:21:39.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.234+0000 7fba52a00640 1 --2- 172.21.15.27:0/3829200850 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba54158900 0x7fba54153f20 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fba3c00d620 tx=0x7fba3c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:39.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.234+0000 7fba4b400640 1 -- 172.21.15.27:0/3829200850 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3c00b620 con 0x7fba54158900 2024-01-15T18:21:39.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.234+0000 7fba4b400640 1 -- 172.21.15.27:0/3829200850 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3c00b780 con 0x7fba54158900 2024-01-15T18:21:39.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.234+0000 7fba4b400640 1 -- 172.21.15.27:0/3829200850 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3c00bb10 con 0x7fba54158900 2024-01-15T18:21:39.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.234+0000 7fba5a600640 1 -- 172.21.15.27:0/3829200850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba5415d660 con 0x7fba54158900 2024-01-15T18:21:39.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.234+0000 7fba5a600640 1 -- 172.21.15.27:0/3829200850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba5413b7a0 con 0x7fba54158900 2024-01-15T18:21:39.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.234+0000 7fba5a600640 1 -- 172.21.15.27:0/3829200850 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba5404e850 con 0x7fba54158900 2024-01-15T18:21:39.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.238+0000 7fba4b400640 1 -- 172.21.15.27:0/3829200850 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7fba3c00bc70 con 0x7fba54158900 2024-01-15T18:21:39.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.238+0000 7fba4b400640 1 --2- 172.21.15.27:0/3829200850 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fba2c067b80 0x7fba2c06a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:39.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.238+0000 7fba4b400640 1 -- 172.21.15.27:0/3829200850 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fba3c083790 con 0x7fba54158900 2024-01-15T18:21:39.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.238+0000 7fba53400640 1 --2- 172.21.15.27:0/3829200850 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fba2c067b80 0x7fba2c06a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:39.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.238+0000 7fba53400640 1 --2- 172.21.15.27:0/3829200850 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fba2c067b80 0x7fba2c06a040 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fba44005890 tx=0x7fba44009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:39.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.242+0000 7fba4b400640 1 -- 172.21.15.27:0/3829200850 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fba3c052150 con 0x7fba54158900 2024-01-15T18:21:39.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.430+0000 7fba5a600640 1 -- 172.21.15.27:0/3829200850 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fba5415f030 con 0x7fba2c067b80 2024-01-15T18:21:39.435 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:21:39.434+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:21:39.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:39.438+0000 7fba4b400640 1 -- 172.21.15.27:0/3829200850 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fba44001510 con 0x7fba2c067b80 2024-01-15T18:21:39.436 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:21:39.436 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:21:39.445 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:21:41.446 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:21:41.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.606+0000 7f9635800640 1 Processor -- start 2024-01-15T18:21:41.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.606+0000 7f9635800640 1 -- start start 2024-01-15T18:21:41.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.610+0000 7f9635800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f963015b100 0x7f9630070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:41.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.610+0000 7f9635800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963015a610 con 0x7f963015ad20 2024-01-15T18:21:41.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.610+0000 7f9635800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963015e3c0 con 0x7f9630071420 2024-01-15T18:21:41.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.610+0000 7f9635800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96300718b0 con 0x7f963015b100 2024-01-15T18:21:41.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.610+0000 7f962ea00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f963015ad20 0x7f963015d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40812/0 (socket says 172.21.15.27:40812) 2024-01-15T18:21:41.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.610+0000 7f962ea00640 1 -- 172.21.15.27:0/3329088072 learned_addr learned my addr 172.21.15.27:0/3329088072 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:21:41.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.610+0000 7f962e000640 1 --2- 172.21.15.27:0/3329088072 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f963015b100 0x7f9630070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:41.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.610+0000 7f962d600640 1 -- 172.21.15.27:0/3329088072 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 474024800 0 0) 0x7f963015a610 con 0x7f963015ad20 2024-01-15T18:21:41.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.610+0000 7f962d600640 1 -- 172.21.15.27:0/3329088072 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9610003530 con 0x7f963015ad20 2024-01-15T18:21:41.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.610+0000 7f962d600640 1 -- 172.21.15.27:0/3329088072 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286648510 0 0) 0x7f963015e3c0 con 0x7f9630071420 2024-01-15T18:21:41.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.610+0000 7f962d600640 1 -- 172.21.15.27:0/3329088072 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f963015a610 con 0x7f9630071420 2024-01-15T18:21:41.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.610+0000 7f962e000640 1 -- 172.21.15.27:0/3329088072 >> v1:172.21.15.93:6789/0 conn(0x7f9630071420 legacy=0x7f963015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:41.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.610+0000 7f962e000640 1 -- 172.21.15.27:0/3329088072 >> v1:172.21.15.27:6789/0 conn(0x7f963015ad20 legacy=0x7f963015d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:41.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.610+0000 7f962e000640 1 -- 172.21.15.27:0/3329088072 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963015e6a0 con 0x7f963015b100 2024-01-15T18:21:41.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.614+0000 7f962e000640 1 --2- 172.21.15.27:0/3329088072 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f963015b100 0x7f9630070ee0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f96180097f0 tx=0x7f961801c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bfe25b024c094e3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:41.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.614+0000 7f962d600640 1 -- 172.21.15.27:0/3329088072 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96180aa5f0 con 0x7f963015b100 2024-01-15T18:21:41.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.614+0000 7f962d600640 1 -- 172.21.15.27:0/3329088072 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96180aa750 con 0x7f963015b100 2024-01-15T18:21:41.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.614+0000 7f962d600640 1 -- 172.21.15.27:0/3329088072 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96180aaae0 con 0x7f963015b100 2024-01-15T18:21:41.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.614+0000 7f9635800640 1 -- 172.21.15.27:0/3329088072 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f963015b100 msgr2=0x7f9630070ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:41.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.614+0000 7f9635800640 1 --2- 172.21.15.27:0/3329088072 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f963015b100 0x7f9630070ee0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f96180097f0 tx=0x7f961801c490 comp rx=0 tx=0).stop 2024-01-15T18:21:41.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.614+0000 7f9635800640 1 -- 172.21.15.27:0/3329088072 shutdown_connections 2024-01-15T18:21:41.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.614+0000 7f9635800640 1 --2- 172.21.15.27:0/3329088072 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f963015b100 0x7f9630070ee0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:41.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.614+0000 7f9635800640 1 -- 172.21.15.27:0/3329088072 >> 172.21.15.27:0/3329088072 conn(0x7f96300cb700 msgr2=0x7f96300cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:41.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.614+0000 7f9635800640 1 -- 172.21.15.27:0/3329088072 shutdown_connections 2024-01-15T18:21:41.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.614+0000 7f9635800640 1 -- 172.21.15.27:0/3329088072 wait complete. 2024-01-15T18:21:41.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.618+0000 7f9635800640 1 Processor -- start 2024-01-15T18:21:41.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.618+0000 7f9635800640 1 -- start start 2024-01-15T18:21:41.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.618+0000 7f9635800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9630071420 0x7f963014d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:41.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.618+0000 7f9635800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f963015ad20 0x7f963014cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:41.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.618+0000 7f9635800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f963015b100 0x7f96300cd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:41.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.618+0000 7f9635800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96300cce40 con 0x7f9630071420 2024-01-15T18:21:41.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.618+0000 7f9635800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96300ccb60 con 0x7f963015b100 2024-01-15T18:21:41.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.618+0000 7f9635800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96301518e0 con 0x7f963015ad20 2024-01-15T18:21:41.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.618+0000 7f962ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9630071420 0x7f963014d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:41.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.618+0000 7f962ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9630071420 0x7f963014d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43550/0 (socket says 172.21.15.27:43550) 2024-01-15T18:21:41.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.618+0000 7f962ea00640 1 -- 172.21.15.27:0/1606299156 learned_addr learned my addr 172.21.15.27:0/1606299156 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:21:41.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.618+0000 7f962e000640 1 --2- 172.21.15.27:0/1606299156 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f963015ad20 0x7f963014cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:41.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.618+0000 7f962f400640 1 --2- 172.21.15.27:0/1606299156 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f963015b100 0x7f96300cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:41.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.618+0000 7f962ea00640 1 -- 172.21.15.27:0/1606299156 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f963015ad20 msgr2=0x7f963014cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:41.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.618+0000 7f962ea00640 1 --2- 172.21.15.27:0/1606299156 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f963015ad20 0x7f963014cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:41.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.618+0000 7f962ea00640 1 -- 172.21.15.27:0/1606299156 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f963015b100 msgr2=0x7f96300cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:41.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.618+0000 7f962ea00640 1 --2- 172.21.15.27:0/1606299156 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f963015b100 0x7f96300cd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:41.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.618+0000 7f962ea00640 1 -- 172.21.15.27:0/1606299156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9618009420 con 0x7f9630071420 2024-01-15T18:21:41.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.618+0000 7f962ea00640 1 --2- 172.21.15.27:0/1606299156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9630071420 0x7f963014d800 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f962000c4d0 tx=0x7f962000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:41.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.618+0000 7f9626a00640 1 -- 172.21.15.27:0/1606299156 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9620005cf0 con 0x7f9630071420 2024-01-15T18:21:41.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.618+0000 7f9626a00640 1 -- 172.21.15.27:0/1606299156 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9620005e50 con 0x7f9630071420 2024-01-15T18:21:41.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.618+0000 7f9635800640 1 -- 172.21.15.27:0/1606299156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9630151bc0 con 0x7f9630071420 2024-01-15T18:21:41.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.618+0000 7f9626a00640 1 -- 172.21.15.27:0/1606299156 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9620007270 con 0x7f9630071420 2024-01-15T18:21:41.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.618+0000 7f9635800640 1 -- 172.21.15.27:0/1606299156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f963014d070 con 0x7f9630071420 2024-01-15T18:21:41.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.622+0000 7f9635800640 1 -- 172.21.15.27:0/1606299156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96301520e0 con 0x7f9630071420 2024-01-15T18:21:41.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.622+0000 7f9626a00640 1 -- 172.21.15.27:0/1606299156 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7f96200073d0 con 0x7f9630071420 2024-01-15T18:21:41.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.622+0000 7f9626a00640 1 --2- 172.21.15.27:0/1606299156 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9608067b80 0x7f960806a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:41.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.622+0000 7f9626a00640 1 -- 172.21.15.27:0/1606299156 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f9620082990 con 0x7f9630071420 2024-01-15T18:21:41.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.622+0000 7f962e000640 1 --2- 172.21.15.27:0/1606299156 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9608067b80 0x7f960806a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:41.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.626+0000 7f962e000640 1 --2- 172.21.15.27:0/1606299156 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9608067b80 0x7f960806a040 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f96180ae000 tx=0x7f96180b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:41.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.626+0000 7f9626a00640 1 -- 172.21.15.27:0/1606299156 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f96301520e0 con 0x7f9630071420 2024-01-15T18:21:41.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.814+0000 7f9635800640 1 -- 172.21.15.27:0/1606299156 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f963015dd10 con 0x7f9608067b80 2024-01-15T18:21:41.816 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:21:41.818+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:21:41.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:41.818+0000 7f9626a00640 1 -- 172.21.15.27:0/1606299156 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f963015dd10 con 0x7f9608067b80 2024-01-15T18:21:41.817 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:21:41.817 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:21:41.825 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:21:43.826 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:21:43.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.990+0000 7fb31fc00640 1 Processor -- start 2024-01-15T18:21:43.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.990+0000 7fb31fc00640 1 -- start start 2024-01-15T18:21:43.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.990+0000 7fb31fc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb31815b100 0x7fb318070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:43.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.990+0000 7fb31fc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb318071420 0x7fb31815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:43.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.990+0000 7fb31fc00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31815a610 con 0x7fb31815ad20 2024-01-15T18:21:43.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.990+0000 7fb31fc00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3180718b0 con 0x7fb318071420 2024-01-15T18:21:43.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.990+0000 7fb31fc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31815e3c0 con 0x7fb31815b100 2024-01-15T18:21:43.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.990+0000 7fb31dc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb318071420 0x7fb31815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:43.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.990+0000 7fb31dc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb318071420 0x7fb31815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55994/0 (socket says 172.21.15.27:55994) 2024-01-15T18:21:43.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.990+0000 7fb31d200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb31815ad20 0x7fb31815d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53530/0 (socket says 172.21.15.27:53530) 2024-01-15T18:21:43.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.990+0000 7fb31dc00640 1 -- 172.21.15.27:0/2882552215 learned_addr learned my addr 172.21.15.27:0/2882552215 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:21:43.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.990+0000 7fb317e00640 1 --2- 172.21.15.27:0/2882552215 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb31815b100 0x7fb318070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:43.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.990+0000 7fb31dc00640 1 -- 172.21.15.27:0/2882552215 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb31815b100 msgr2=0x7fb318070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:43.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.990+0000 7fb31dc00640 1 --2- 172.21.15.27:0/2882552215 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb31815b100 0x7fb318070ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:43.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.990+0000 7fb31dc00640 1 -- 172.21.15.27:0/2882552215 >> v1:172.21.15.27:6789/0 conn(0x7fb31815ad20 legacy=0x7fb31815d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:43.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.990+0000 7fb31dc00640 1 -- 172.21.15.27:0/2882552215 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31815e6a0 con 0x7fb318071420 2024-01-15T18:21:43.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.990+0000 7fb317e00640 1 --2- 172.21.15.27:0/2882552215 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb31815b100 0x7fb318070ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:21:43.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.994+0000 7fb31dc00640 1 --2- 172.21.15.27:0/2882552215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318071420 0x7fb31815bee0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fb310009e50 tx=0x7fb31001c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d504e5d1150df40 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:43.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.994+0000 7fb317400640 1 -- 172.21.15.27:0/2882552215 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3100a57c0 con 0x7fb318071420 2024-01-15T18:21:43.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.994+0000 7fb317400640 1 -- 172.21.15.27:0/2882552215 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3100a5920 con 0x7fb318071420 2024-01-15T18:21:43.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.994+0000 7fb317400640 1 -- 172.21.15.27:0/2882552215 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3100a5cb0 con 0x7fb318071420 2024-01-15T18:21:43.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.994+0000 7fb31fc00640 1 -- 172.21.15.27:0/2882552215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318071420 msgr2=0x7fb31815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:43.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.994+0000 7fb31fc00640 1 --2- 172.21.15.27:0/2882552215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318071420 0x7fb31815bee0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fb310009e50 tx=0x7fb31001c9e0 comp rx=0 tx=0).stop 2024-01-15T18:21:43.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.994+0000 7fb31fc00640 1 -- 172.21.15.27:0/2882552215 shutdown_connections 2024-01-15T18:21:43.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.994+0000 7fb31fc00640 1 --2- 172.21.15.27:0/2882552215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318071420 0x7fb31815bee0 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:43.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.994+0000 7fb31fc00640 1 --2- 172.21.15.27:0/2882552215 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb31815b100 0x7fb318070ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:43.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.994+0000 7fb31fc00640 1 -- 172.21.15.27:0/2882552215 >> 172.21.15.27:0/2882552215 conn(0x7fb3180cb700 msgr2=0x7fb3180cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:43.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.994+0000 7fb31fc00640 1 -- 172.21.15.27:0/2882552215 shutdown_connections 2024-01-15T18:21:43.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.994+0000 7fb31fc00640 1 -- 172.21.15.27:0/2882552215 wait complete. 2024-01-15T18:21:43.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.998+0000 7fb31fc00640 1 Processor -- start 2024-01-15T18:21:43.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.998+0000 7fb31fc00640 1 -- start start 2024-01-15T18:21:43.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.998+0000 7fb31fc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318071420 0x7fb3181f5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:43.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.998+0000 7fb31fc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb31815ad20 0x7fb31814ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:43.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.998+0000 7fb31fc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb31815b100 0x7fb318156fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:43.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.998+0000 7fb31fc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3181f6020 con 0x7fb318071420 2024-01-15T18:21:43.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.998+0000 7fb31fc00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31814b2a0 con 0x7fb31815b100 2024-01-15T18:21:43.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.998+0000 7fb31fc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb318155440 con 0x7fb31815ad20 2024-01-15T18:21:43.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.998+0000 7fb31d200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318071420 0x7fb3181f5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:43.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.998+0000 7fb31d200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318071420 0x7fb3181f5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56006/0 (socket says 172.21.15.27:56006) 2024-01-15T18:21:43.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.998+0000 7fb31d200640 1 -- 172.21.15.27:0/1867976704 learned_addr learned my addr 172.21.15.27:0/1867976704 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:21:43.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.998+0000 7fb31dc00640 1 --2- 172.21.15.27:0/1867976704 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb31815b100 0x7fb318156fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:43.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.998+0000 7fb317e00640 1 --2- 172.21.15.27:0/1867976704 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb31815ad20 0x7fb31814ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:43.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.998+0000 7fb31d200640 1 -- 172.21.15.27:0/1867976704 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb31815ad20 msgr2=0x7fb31814ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:43.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.998+0000 7fb31d200640 1 --2- 172.21.15.27:0/1867976704 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb31815ad20 0x7fb31814ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:43.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.998+0000 7fb31d200640 1 -- 172.21.15.27:0/1867976704 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb31815b100 msgr2=0x7fb318156fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:43.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.998+0000 7fb31d200640 1 --2- 172.21.15.27:0/1867976704 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb31815b100 0x7fb318156fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:43.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.998+0000 7fb31d200640 1 -- 172.21.15.27:0/1867976704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb310009b30 con 0x7fb318071420 2024-01-15T18:21:43.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.998+0000 7fb31d200640 1 --2- 172.21.15.27:0/1867976704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb318071420 0x7fb3181f5ae0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fb30800c490 tx=0x7fb30800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:43.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.998+0000 7fb314c00640 1 -- 172.21.15.27:0/1867976704 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb308007480 con 0x7fb318071420 2024-01-15T18:21:43.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.998+0000 7fb314c00640 1 -- 172.21.15.27:0/1867976704 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3080075e0 con 0x7fb318071420 2024-01-15T18:21:43.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.998+0000 7fb314c00640 1 -- 172.21.15.27:0/1867976704 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb308004a10 con 0x7fb318071420 2024-01-15T18:21:43.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.998+0000 7fb31fc00640 1 -- 172.21.15.27:0/1867976704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb318155720 con 0x7fb318071420 2024-01-15T18:21:43.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:43.998+0000 7fb31fc00640 1 -- 172.21.15.27:0/1867976704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3180b3760 con 0x7fb318071420 2024-01-15T18:21:43.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:44.002+0000 7fb31fc00640 1 -- 172.21.15.27:0/1867976704 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3180b3950 con 0x7fb318071420 2024-01-15T18:21:43.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:44.002+0000 7fb314c00640 1 -- 172.21.15.27:0/1867976704 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7fb308004b70 con 0x7fb318071420 2024-01-15T18:21:43.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:44.002+0000 7fb314c00640 1 --2- 172.21.15.27:0/1867976704 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb2f0067b80 0x7fb2f006a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:43.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:44.002+0000 7fb314c00640 1 -- 172.21.15.27:0/1867976704 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fb308082540 con 0x7fb318071420 2024-01-15T18:21:43.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:44.002+0000 7fb317e00640 1 --2- 172.21.15.27:0/1867976704 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb2f0067b80 0x7fb2f006a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:43.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:44.002+0000 7fb317e00640 1 --2- 172.21.15.27:0/1867976704 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb2f0067b80 0x7fb2f006a040 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fb3180b33b0 tx=0x7fb300008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:44.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:44.006+0000 7fb314c00640 1 -- 172.21.15.27:0/1867976704 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb308050f00 con 0x7fb318071420 2024-01-15T18:21:44.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:44.194+0000 7fb31fc00640 1 -- 172.21.15.27:0/1867976704 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb31804e850 con 0x7fb2f0067b80 2024-01-15T18:21:44.195 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:21:44.198+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:21:44.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:44.198+0000 7fb314c00640 1 -- 172.21.15.27:0/1867976704 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb31804e850 con 0x7fb2f0067b80 2024-01-15T18:21:44.196 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:21:44.196 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:21:44.205 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:21:46.206 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:21:46.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.370+0000 7f88f1000640 1 Processor -- start 2024-01-15T18:21:46.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.370+0000 7f88f1000640 1 -- start start 2024-01-15T18:21:46.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.370+0000 7f88f1000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88ec15ad20 0x7f88ec15b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:46.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.370+0000 7f88f1000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f88ec0715f0 0x7f88ec15dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:46.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.370+0000 7f88f1000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88ec15a610 con 0x7f88ec15d900 2024-01-15T18:21:46.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.370+0000 7f88f1000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88ec15e3c0 con 0x7f88ec0715f0 2024-01-15T18:21:46.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.370+0000 7f88f1000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec15e6a0 con 0x7f88ec15ad20 2024-01-15T18:21:46.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.370+0000 7f88ea800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f88ec0715f0 0x7f88ec15dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:46.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.370+0000 7f88ea800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f88ec0715f0 0x7f88ec15dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56018/0 (socket says 172.21.15.27:56018) 2024-01-15T18:21:46.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.370+0000 7f88ea800640 1 -- 172.21.15.27:0/1605647304 learned_addr learned my addr 172.21.15.27:0/1605647304 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:21:46.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.370+0000 7f88e9400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f88ec15d900 0x7f88ec070ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53536/0 (socket says 172.21.15.27:53536) 2024-01-15T18:21:46.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.370+0000 7f88e9e00640 1 --2- 172.21.15.27:0/1605647304 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88ec15ad20 0x7f88ec15b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:46.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.370+0000 7f88ea800640 1 -- 172.21.15.27:0/1605647304 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88ec15ad20 msgr2=0x7f88ec15b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:46.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.370+0000 7f88ea800640 1 --2- 172.21.15.27:0/1605647304 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88ec15ad20 0x7f88ec15b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:46.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.370+0000 7f88ea800640 1 -- 172.21.15.27:0/1605647304 >> v1:172.21.15.27:6789/0 conn(0x7f88ec15d900 legacy=0x7f88ec070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:46.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.370+0000 7f88ea800640 1 -- 172.21.15.27:0/1605647304 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ec15b6e0 con 0x7f88ec0715f0 2024-01-15T18:21:46.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.374+0000 7f88ea800640 1 --2- 172.21.15.27:0/1605647304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ec0715f0 0x7f88ec15dd80 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009e50 tx=0x7f88e401c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1efa42fd049f7735 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:46.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.374+0000 7f88e8a00640 1 -- 172.21.15.27:0/1605647304 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e40a58d0 con 0x7f88ec0715f0 2024-01-15T18:21:46.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.374+0000 7f88e8a00640 1 -- 172.21.15.27:0/1605647304 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e40a5a30 con 0x7f88ec0715f0 2024-01-15T18:21:46.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.374+0000 7f88e8a00640 1 -- 172.21.15.27:0/1605647304 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e40a5dc0 con 0x7f88ec0715f0 2024-01-15T18:21:46.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.374+0000 7f88f1000640 1 -- 172.21.15.27:0/1605647304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ec0715f0 msgr2=0x7f88ec15dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:46.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.374+0000 7f88f1000640 1 --2- 172.21.15.27:0/1605647304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ec0715f0 0x7f88ec15dd80 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009e50 tx=0x7f88e401c9e0 comp rx=0 tx=0).stop 2024-01-15T18:21:46.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.374+0000 7f88f1000640 1 -- 172.21.15.27:0/1605647304 shutdown_connections 2024-01-15T18:21:46.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.374+0000 7f88f1000640 1 --2- 172.21.15.27:0/1605647304 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ec0715f0 0x7f88ec15dd80 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:46.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.374+0000 7f88f1000640 1 --2- 172.21.15.27:0/1605647304 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88ec15ad20 0x7f88ec15b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:46.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.374+0000 7f88f1000640 1 -- 172.21.15.27:0/1605647304 >> 172.21.15.27:0/1605647304 conn(0x7f88ec0cb700 msgr2=0x7f88ec0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:46.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.374+0000 7f88f1000640 1 -- 172.21.15.27:0/1605647304 shutdown_connections 2024-01-15T18:21:46.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.374+0000 7f88f1000640 1 -- 172.21.15.27:0/1605647304 wait complete. 2024-01-15T18:21:46.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.378+0000 7f88f1000640 1 Processor -- start 2024-01-15T18:21:46.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.378+0000 7f88f1000640 1 -- start start 2024-01-15T18:21:46.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.378+0000 7f88f1000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ec0715f0 0x7f88ec14d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:46.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.378+0000 7f88f1000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88ec15ad20 0x7f88ec160430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:46.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.378+0000 7f88f1000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88ec15d900 0x7f88ec160970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:46.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.378+0000 7f88f1000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec0b3bd0 con 0x7f88ec0715f0 2024-01-15T18:21:46.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.378+0000 7f88f1000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec0cf160 con 0x7f88ec15d900 2024-01-15T18:21:46.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.378+0000 7f88f1000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec162370 con 0x7f88ec15ad20 2024-01-15T18:21:46.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.378+0000 7f88e9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ec0715f0 0x7f88ec14d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:46.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.378+0000 7f88e9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ec0715f0 0x7f88ec14d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56028/0 (socket says 172.21.15.27:56028) 2024-01-15T18:21:46.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.378+0000 7f88e9e00640 1 -- 172.21.15.27:0/292973967 learned_addr learned my addr 172.21.15.27:0/292973967 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:21:46.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.378+0000 7f88e9400640 1 --2- 172.21.15.27:0/292973967 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88ec15ad20 0x7f88ec160430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:46.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.378+0000 7f88ea800640 1 --2- 172.21.15.27:0/292973967 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88ec15d900 0x7f88ec160970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:46.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.378+0000 7f88e9e00640 1 -- 172.21.15.27:0/292973967 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88ec15ad20 msgr2=0x7f88ec160430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:46.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.378+0000 7f88e9e00640 1 --2- 172.21.15.27:0/292973967 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f88ec15ad20 0x7f88ec160430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:46.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.378+0000 7f88e9e00640 1 -- 172.21.15.27:0/292973967 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88ec15d900 msgr2=0x7f88ec160970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:46.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.378+0000 7f88e9e00640 1 --2- 172.21.15.27:0/292973967 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f88ec15d900 0x7f88ec160970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:46.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.378+0000 7f88e9e00640 1 -- 172.21.15.27:0/292973967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e4009b30 con 0x7f88ec0715f0 2024-01-15T18:21:46.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.378+0000 7f88e9e00640 1 --2- 172.21.15.27:0/292973967 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f88ec0715f0 0x7f88ec14d700 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f88dc00e930 tx=0x7f88dc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:46.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.378+0000 7f88e2000640 1 -- 172.21.15.27:0/292973967 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88dc009690 con 0x7f88ec0715f0 2024-01-15T18:21:46.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.378+0000 7f88e2000640 1 -- 172.21.15.27:0/292973967 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88dc00ca00 con 0x7f88ec0715f0 2024-01-15T18:21:46.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.378+0000 7f88f1000640 1 -- 172.21.15.27:0/292973967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88ec0cebc0 con 0x7f88ec0715f0 2024-01-15T18:21:46.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.378+0000 7f88e2000640 1 -- 172.21.15.27:0/292973967 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88dc00cd70 con 0x7f88ec0715f0 2024-01-15T18:21:46.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.378+0000 7f88f1000640 1 -- 172.21.15.27:0/292973967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88ec0b3860 con 0x7f88ec0715f0 2024-01-15T18:21:46.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.382+0000 7f88f1000640 1 -- 172.21.15.27:0/292973967 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88ec0cee00 con 0x7f88ec0715f0 2024-01-15T18:21:46.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.382+0000 7f88e2000640 1 -- 172.21.15.27:0/292973967 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7f88dc013470 con 0x7f88ec0715f0 2024-01-15T18:21:46.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.382+0000 7f88e2000640 1 --2- 172.21.15.27:0/292973967 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88c4067b80 0x7f88c406a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:46.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.382+0000 7f88e2000640 1 -- 172.21.15.27:0/292973967 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f88dc0832b0 con 0x7f88ec0715f0 2024-01-15T18:21:46.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.382+0000 7f88e9400640 1 --2- 172.21.15.27:0/292973967 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88c4067b80 0x7f88c406a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:46.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.382+0000 7f88e9400640 1 --2- 172.21.15.27:0/292973967 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f88c4067b80 0x7f88c406a040 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f88ec0b20e0 tx=0x7f88d4007e50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:46.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.386+0000 7f88e2000640 1 -- 172.21.15.27:0/292973967 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f88dc052030 con 0x7f88ec0715f0 2024-01-15T18:21:46.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.574+0000 7f88f1000640 1 -- 172.21.15.27:0/292973967 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f88ec164400 con 0x7f88c4067b80 2024-01-15T18:21:46.577 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:21:46.578+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:21:46.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:46.578+0000 7f88e2000640 1 -- 172.21.15.27:0/292973967 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f88ec164400 con 0x7f88c4067b80 2024-01-15T18:21:46.577 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:21:46.578 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:21:46.585 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:21:48.587 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:21:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.750+0000 7f1691600640 1 Processor -- start 2024-01-15T18:21:48.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.750+0000 7f1691600640 1 -- start start 2024-01-15T18:21:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.754+0000 7f1691600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f168c06c430 0x7f168c070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.754+0000 7f1691600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f168c0b1db0 0x7f168c0b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.754+0000 7f1691600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f168c0ce050 0x7f168c0ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.754+0000 7f1691600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f168c0cea20 con 0x7f168c0b1db0 2024-01-15T18:21:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.754+0000 7f1691600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f168c0ccfb0 con 0x7f168c0ce050 2024-01-15T18:21:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.754+0000 7f1691600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f168c0cd290 con 0x7f168c06c430 2024-01-15T18:21:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.754+0000 7f168a000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f168c0b1db0 0x7f168c0b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.754+0000 7f168a000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f168c0b1db0 0x7f168c0b2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56040/0 (socket says 172.21.15.27:56040) 2024-01-15T18:21:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.754+0000 7f168a000640 1 -- 172.21.15.27:0/2615306332 learned_addr learned my addr 172.21.15.27:0/2615306332 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:21:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.754+0000 7f168aa00640 1 --2- 172.21.15.27:0/2615306332 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f168c06c430 0x7f168c070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:48.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.754+0000 7f168b400640 1 --2- 172.21.15.27:0/2615306332 >> v2:172.21.15.93:3300/0 conn(0x7f168c0ce050 0x7f168c0ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.754+0000 7f168a000640 1 -- 172.21.15.27:0/2615306332 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f168c06c430 msgr2=0x7f168c070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.754+0000 7f168a000640 1 --2- 172.21.15.27:0/2615306332 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f168c06c430 0x7f168c070ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.754+0000 7f168a000640 1 -- 172.21.15.27:0/2615306332 >> v2:172.21.15.93:3300/0 conn(0x7f168c0ce050 msgr2=0x7f168c0ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.754+0000 7f168a000640 1 --2- 172.21.15.27:0/2615306332 >> v2:172.21.15.93:3300/0 conn(0x7f168c0ce050 0x7f168c0ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.754+0000 7f168a000640 1 -- 172.21.15.27:0/2615306332 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f168c0cd570 con 0x7f168c0b1db0 2024-01-15T18:21:48.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.754+0000 7f168a000640 1 --2- 172.21.15.27:0/2615306332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f168c0b1db0 0x7f168c0b2210 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f168c072330 tx=0x7f167401e920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6054629d07194a2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:48.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.758+0000 7f1689600640 1 -- 172.21.15.27:0/2615306332 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1674004230 con 0x7f168c0b1db0 2024-01-15T18:21:48.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.758+0000 7f1689600640 1 -- 172.21.15.27:0/2615306332 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16740ab070 con 0x7f168c0b1db0 2024-01-15T18:21:48.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.758+0000 7f1689600640 1 -- 172.21.15.27:0/2615306332 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1674003620 con 0x7f168c0b1db0 2024-01-15T18:21:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.758+0000 7f1691600640 1 -- 172.21.15.27:0/2615306332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f168c0b1db0 msgr2=0x7f168c0b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.758+0000 7f1691600640 1 --2- 172.21.15.27:0/2615306332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f168c0b1db0 0x7f168c0b2210 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f168c072330 tx=0x7f167401e920 comp rx=0 tx=0).stop 2024-01-15T18:21:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.758+0000 7f1691600640 1 -- 172.21.15.27:0/2615306332 shutdown_connections 2024-01-15T18:21:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.758+0000 7f1691600640 1 --2- 172.21.15.27:0/2615306332 >> v2:172.21.15.93:3300/0 conn(0x7f168c0ce050 0x7f168c0ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.758+0000 7f1691600640 1 --2- 172.21.15.27:0/2615306332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f168c0b1db0 0x7f168c0b2210 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.758+0000 7f1691600640 1 --2- 172.21.15.27:0/2615306332 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f168c06c430 0x7f168c070ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.758+0000 7f1691600640 1 -- 172.21.15.27:0/2615306332 >> 172.21.15.27:0/2615306332 conn(0x7f168c0cb700 msgr2=0x7f168c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.758+0000 7f1691600640 1 -- 172.21.15.27:0/2615306332 shutdown_connections 2024-01-15T18:21:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.758+0000 7f1691600640 1 -- 172.21.15.27:0/2615306332 wait complete. 2024-01-15T18:21:48.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.758+0000 7f1691600640 1 Processor -- start 2024-01-15T18:21:48.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.758+0000 7f1691600640 1 -- start start 2024-01-15T18:21:48.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.758+0000 7f1691600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f168c06c430 0x7f168c1f52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:48.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.758+0000 7f1691600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f168c0b1db0 0x7f168c147540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:48.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.758+0000 7f1691600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f168c0ce050 0x7f168c14cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:48.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.758+0000 7f1691600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168c148100 con 0x7f168c0ce050 2024-01-15T18:21:48.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.758+0000 7f1691600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168c1f5810 con 0x7f168c0b1db0 2024-01-15T18:21:48.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.758+0000 7f1691600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f168c15c9c0 con 0x7f168c06c430 2024-01-15T18:21:48.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.762+0000 7f168b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f168c0ce050 0x7f168c14cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:48.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.762+0000 7f168b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f168c0ce050 0x7f168c14cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56052/0 (socket says 172.21.15.27:56052) 2024-01-15T18:21:48.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.762+0000 7f168b400640 1 -- 172.21.15.27:0/58953009 learned_addr learned my addr 172.21.15.27:0/58953009 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:21:48.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.762+0000 7f168a000640 1 --2- 172.21.15.27:0/58953009 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f168c0b1db0 0x7f168c147540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:48.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.762+0000 7f168aa00640 1 --2- 172.21.15.27:0/58953009 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f168c06c430 0x7f168c1f52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:48.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.762+0000 7f168b400640 1 -- 172.21.15.27:0/58953009 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f168c06c430 msgr2=0x7f168c1f52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:48.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.762+0000 7f168b400640 1 --2- 172.21.15.27:0/58953009 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f168c06c430 0x7f168c1f52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:48.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.762+0000 7f168b400640 1 -- 172.21.15.27:0/58953009 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f168c0b1db0 msgr2=0x7f168c147540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:48.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.762+0000 7f168b400640 1 --2- 172.21.15.27:0/58953009 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f168c0b1db0 0x7f168c147540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:48.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.762+0000 7f168b400640 1 -- 172.21.15.27:0/58953009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167400b780 con 0x7f168c0ce050 2024-01-15T18:21:48.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.762+0000 7f168a000640 1 --2- 172.21.15.27:0/58953009 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f168c0b1db0 0x7f168c147540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:21:48.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.762+0000 7f168b400640 1 --2- 172.21.15.27:0/58953009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f168c0ce050 0x7f168c14cb80 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f1684007e60 tx=0x7f168400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:48.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.762+0000 7f1682a00640 1 -- 172.21.15.27:0/58953009 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1684014420 con 0x7f168c0ce050 2024-01-15T18:21:48.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.762+0000 7f1682a00640 1 -- 172.21.15.27:0/58953009 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16840145a0 con 0x7f168c0ce050 2024-01-15T18:21:48.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.762+0000 7f1691600640 1 -- 172.21.15.27:0/58953009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f168c146780 con 0x7f168c0ce050 2024-01-15T18:21:48.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.762+0000 7f1682a00640 1 -- 172.21.15.27:0/58953009 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1684014950 con 0x7f168c0ce050 2024-01-15T18:21:48.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.762+0000 7f1691600640 1 -- 172.21.15.27:0/58953009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f168c146d40 con 0x7f168c0ce050 2024-01-15T18:21:48.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.762+0000 7f1691600640 1 -- 172.21.15.27:0/58953009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f168c04e850 con 0x7f168c0ce050 2024-01-15T18:21:48.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.762+0000 7f1682a00640 1 -- 172.21.15.27:0/58953009 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7f1684014ab0 con 0x7f168c0ce050 2024-01-15T18:21:48.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.766+0000 7f1682a00640 1 --2- 172.21.15.27:0/58953009 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1664067b80 0x7f166406a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:48.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.766+0000 7f1682a00640 1 -- 172.21.15.27:0/58953009 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f1684083be0 con 0x7f168c0ce050 2024-01-15T18:21:48.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.766+0000 7f168aa00640 1 --2- 172.21.15.27:0/58953009 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1664067b80 0x7f166406a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:48.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.766+0000 7f168aa00640 1 --2- 172.21.15.27:0/58953009 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1664067b80 0x7f166406a040 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f167c007a60 tx=0x7f167c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:48.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.770+0000 7f1682a00640 1 -- 172.21.15.27:0/58953009 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f16840524f0 con 0x7f168c0ce050 2024-01-15T18:21:48.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.958+0000 7f1691600640 1 -- 172.21.15.27:0/58953009 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f168c06c810 con 0x7f1664067b80 2024-01-15T18:21:48.963 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:21:48.962+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:21:48.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:48.966+0000 7f1682a00640 1 -- 172.21.15.27:0/58953009 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f167c010990 con 0x7f1664067b80 2024-01-15T18:21:48.964 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:21:48.964 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:21:48.972 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:21:50.973 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:21:51.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.134+0000 7f4c58e00640 1 Processor -- start 2024-01-15T18:21:51.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.134+0000 7f4c58e00640 1 -- start start 2024-01-15T18:21:51.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.138+0000 7f4c58e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4c540cf450 0x7f4c540d1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:51.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.138+0000 7f4c58e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f4c540b0650 0x7f4c540b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:51.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.138+0000 7f4c58e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c540cefa0 con 0x7f4c540cba60 2024-01-15T18:21:51.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.138+0000 7f4c58e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c540ce240 con 0x7f4c540cf450 2024-01-15T18:21:51.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.138+0000 7f4c58e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c540ccd70 con 0x7f4c540b0650 2024-01-15T18:21:51.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.138+0000 7f4c51e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4c540cf450 0x7f4c540d1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:51.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.138+0000 7f4c51e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4c540cf450 0x7f4c540d1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56054/0 (socket says 172.21.15.27:56054) 2024-01-15T18:21:51.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.138+0000 7f4c51400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4c540cba60 0x7f4c540cf830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53546/0 (socket says 172.21.15.27:53546) 2024-01-15T18:21:51.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.138+0000 7f4c51e00640 1 -- 172.21.15.27:0/1099053067 learned_addr learned my addr 172.21.15.27:0/1099053067 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:21:51.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.138+0000 7f4c51e00640 1 -- 172.21.15.27:0/1099053067 >> v2:172.21.15.93:3300/0 conn(0x7f4c540b0650 msgr2=0x7f4c540b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:51.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.138+0000 7f4c52800640 1 --2- 172.21.15.27:0/1099053067 >> v2:172.21.15.93:3300/0 conn(0x7f4c540b0650 0x7f4c540b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:51.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.138+0000 7f4c51e00640 1 --2- 172.21.15.27:0/1099053067 >> v2:172.21.15.93:3300/0 conn(0x7f4c540b0650 0x7f4c540b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:51.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.138+0000 7f4c51e00640 1 -- 172.21.15.27:0/1099053067 >> v1:172.21.15.27:6789/0 conn(0x7f4c540cba60 legacy=0x7f4c540cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:51.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.138+0000 7f4c51e00640 1 -- 172.21.15.27:0/1099053067 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c540cc1f0 con 0x7f4c540cf450 2024-01-15T18:21:51.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.138+0000 7f4c52800640 1 --2- 172.21.15.27:0/1099053067 >> v2:172.21.15.93:3300/0 conn(0x7f4c540b0650 0x7f4c540b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:21:51.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.142+0000 7f4c51e00640 1 --2- 172.21.15.27:0/1099053067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c540cf450 0x7f4c540d1a40 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f4c540717e0 tx=0x7f4c4401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d428529df62a0e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:51.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.142+0000 7f4c50a00640 1 -- 172.21.15.27:0/1099053067 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c440ab5f0 con 0x7f4c540cf450 2024-01-15T18:21:51.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.142+0000 7f4c50a00640 1 -- 172.21.15.27:0/1099053067 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c440ab750 con 0x7f4c540cf450 2024-01-15T18:21:51.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.142+0000 7f4c50a00640 1 -- 172.21.15.27:0/1099053067 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c440abae0 con 0x7f4c540cf450 2024-01-15T18:21:51.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.142+0000 7f4c58e00640 1 -- 172.21.15.27:0/1099053067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c540cf450 msgr2=0x7f4c540d1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:51.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.142+0000 7f4c58e00640 1 --2- 172.21.15.27:0/1099053067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c540cf450 0x7f4c540d1a40 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f4c540717e0 tx=0x7f4c4401c500 comp rx=0 tx=0).stop 2024-01-15T18:21:51.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.142+0000 7f4c58e00640 1 -- 172.21.15.27:0/1099053067 shutdown_connections 2024-01-15T18:21:51.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.142+0000 7f4c58e00640 1 --2- 172.21.15.27:0/1099053067 >> v2:172.21.15.93:3300/0 conn(0x7f4c540b0650 0x7f4c540b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:51.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.142+0000 7f4c58e00640 1 --2- 172.21.15.27:0/1099053067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c540cf450 0x7f4c540d1a40 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:51.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.142+0000 7f4c58e00640 1 -- 172.21.15.27:0/1099053067 >> 172.21.15.27:0/1099053067 conn(0x7f4c540d0e70 msgr2=0x7f4c540d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:51.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.142+0000 7f4c58e00640 1 -- 172.21.15.27:0/1099053067 shutdown_connections 2024-01-15T18:21:51.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.142+0000 7f4c58e00640 1 -- 172.21.15.27:0/1099053067 wait complete. 2024-01-15T18:21:51.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.142+0000 7f4c58e00640 1 Processor -- start 2024-01-15T18:21:51.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.142+0000 7f4c58e00640 1 -- start start 2024-01-15T18:21:51.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.146+0000 7f4c58e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4c540b0650 0x7f4c540af770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:51.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.146+0000 7f4c58e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c540cba60 0x7f4c541566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:51.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.146+0000 7f4c58e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4c540cf450 0x7f4c54156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:51.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.146+0000 7f4c58e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5415c500 con 0x7f4c540cba60 2024-01-15T18:21:51.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.146+0000 7f4c58e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5415c7e0 con 0x7f4c540b0650 2024-01-15T18:21:51.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.146+0000 7f4c58e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5415cac0 con 0x7f4c540cf450 2024-01-15T18:21:51.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.146+0000 7f4c51400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c540cba60 0x7f4c541566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:51.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.146+0000 7f4c51400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c540cba60 0x7f4c541566b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56070/0 (socket says 172.21.15.27:56070) 2024-01-15T18:21:51.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.146+0000 7f4c51400640 1 -- 172.21.15.27:0/2767599192 learned_addr learned my addr 172.21.15.27:0/2767599192 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:21:51.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.146+0000 7f4c51e00640 1 --2- 172.21.15.27:0/2767599192 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4c540b0650 0x7f4c540af770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:51.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.146+0000 7f4c52800640 1 --2- 172.21.15.27:0/2767599192 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4c540cf450 0x7f4c54156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:51.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.146+0000 7f4c51400640 1 -- 172.21.15.27:0/2767599192 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4c540cf450 msgr2=0x7f4c54156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:51.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.146+0000 7f4c51400640 1 --2- 172.21.15.27:0/2767599192 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4c540cf450 0x7f4c54156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:51.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.146+0000 7f4c51400640 1 -- 172.21.15.27:0/2767599192 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4c540b0650 msgr2=0x7f4c540af770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:51.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.146+0000 7f4c51400640 1 --2- 172.21.15.27:0/2767599192 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4c540b0650 0x7f4c540af770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:51.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.146+0000 7f4c51400640 1 -- 172.21.15.27:0/2767599192 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c440094d0 con 0x7f4c540cba60 2024-01-15T18:21:51.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.146+0000 7f4c52800640 1 --2- 172.21.15.27:0/2767599192 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4c540cf450 0x7f4c54156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:21:51.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.146+0000 7f4c51400640 1 --2- 172.21.15.27:0/2767599192 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c540cba60 0x7f4c541566b0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c00c430 tx=0x7f4c3c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:51.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.146+0000 7f4c4a000640 1 -- 172.21.15.27:0/2767599192 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c0074e0 con 0x7f4c540cba60 2024-01-15T18:21:51.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.146+0000 7f4c4a000640 1 -- 172.21.15.27:0/2767599192 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c3c007640 con 0x7f4c540cba60 2024-01-15T18:21:51.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.146+0000 7f4c58e00640 1 -- 172.21.15.27:0/2767599192 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c5415cda0 con 0x7f4c540cba60 2024-01-15T18:21:51.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.146+0000 7f4c4a000640 1 -- 172.21.15.27:0/2767599192 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c004a10 con 0x7f4c540cba60 2024-01-15T18:21:51.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.146+0000 7f4c58e00640 1 -- 172.21.15.27:0/2767599192 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c540afe80 con 0x7f4c540cba60 2024-01-15T18:21:51.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.146+0000 7f4c58e00640 1 -- 172.21.15.27:0/2767599192 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c5415cf90 con 0x7f4c540cba60 2024-01-15T18:21:51.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.150+0000 7f4c4a000640 1 -- 172.21.15.27:0/2767599192 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7f4c3c004b70 con 0x7f4c540cba60 2024-01-15T18:21:51.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.150+0000 7f4c4a000640 1 --2- 172.21.15.27:0/2767599192 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4c2c067b80 0x7f4c2c06a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:51.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.150+0000 7f4c4a000640 1 -- 172.21.15.27:0/2767599192 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f4c3c082d70 con 0x7f4c540cba60 2024-01-15T18:21:51.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.150+0000 7f4c51e00640 1 --2- 172.21.15.27:0/2767599192 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4c2c067b80 0x7f4c2c06a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:51.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.150+0000 7f4c51e00640 1 --2- 172.21.15.27:0/2767599192 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4c2c067b80 0x7f4c2c06a040 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f4c440af000 tx=0x7f4c440b6040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:51.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.154+0000 7f4c4a000640 1 -- 172.21.15.27:0/2767599192 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4c3c051680 con 0x7f4c540cba60 2024-01-15T18:21:51.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.342+0000 7f4c58e00640 1 -- 172.21.15.27:0/2767599192 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4c540cbe70 con 0x7f4c2c067b80 2024-01-15T18:21:51.345 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:21:51.346+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:21:51.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:51.350+0000 7f4c4a000640 1 -- 172.21.15.27:0/2767599192 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4c440ab9c0 con 0x7f4c2c067b80 2024-01-15T18:21:51.346 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:21:51.346 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:21:51.354 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:21:53.356 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:21:53.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.518+0000 7fbead200640 1 Processor -- start 2024-01-15T18:21:53.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.518+0000 7fbead200640 1 -- start start 2024-01-15T18:21:53.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.518+0000 7fbead200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fbea8158900 0x7fbea815f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:53.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.518+0000 7fbead200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbea80cf2e0 0x7fbea80701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:53.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.518+0000 7fbead200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbea80b1cb0 con 0x7fbea815bf00 2024-01-15T18:21:53.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.518+0000 7fbead200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbea80b1f00 con 0x7fbea8158900 2024-01-15T18:21:53.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.518+0000 7fbead200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbea80b21e0 con 0x7fbea80cf2e0 2024-01-15T18:21:53.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.518+0000 7fbea5400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbea815bf00 0x7fbea80cebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53554/0 (socket says 172.21.15.27:53554) 2024-01-15T18:21:53.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.518+0000 7fbea5400640 1 -- 172.21.15.27:0/4211989708 learned_addr learned my addr 172.21.15.27:0/4211989708 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:21:53.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.518+0000 7fbea6800640 1 --2- 172.21.15.27:0/4211989708 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbea80cf2e0 0x7fbea80701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:53.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.518+0000 7fbea5e00640 1 --2- 172.21.15.27:0/4211989708 >> v2:172.21.15.93:3300/0 conn(0x7fbea8158900 0x7fbea815f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:53.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.518+0000 7fbea4a00640 1 -- 172.21.15.27:0/4211989708 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345234372 0 0) 0x7fbea80b1cb0 con 0x7fbea815bf00 2024-01-15T18:21:53.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.518+0000 7fbea4a00640 1 -- 172.21.15.27:0/4211989708 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe88003530 con 0x7fbea815bf00 2024-01-15T18:21:53.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.522+0000 7fbea6800640 1 -- 172.21.15.27:0/4211989708 >> v2:172.21.15.93:3300/0 conn(0x7fbea8158900 msgr2=0x7fbea815f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:53.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.522+0000 7fbea6800640 1 --2- 172.21.15.27:0/4211989708 >> v2:172.21.15.93:3300/0 conn(0x7fbea8158900 0x7fbea815f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:53.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.522+0000 7fbea4a00640 1 -- 172.21.15.27:0/4211989708 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 645416509 0 0) 0x7fbe88003530 con 0x7fbea815bf00 2024-01-15T18:21:53.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.522+0000 7fbea6800640 1 -- 172.21.15.27:0/4211989708 >> v1:172.21.15.27:6789/0 conn(0x7fbea815bf00 legacy=0x7fbea80cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:53.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.522+0000 7fbea6800640 1 -- 172.21.15.27:0/4211989708 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea815ea70 con 0x7fbea80cf2e0 2024-01-15T18:21:53.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.522+0000 7fbea6800640 1 --2- 172.21.15.27:0/4211989708 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbea80cf2e0 0x7fbea80701e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbea0009d90 tx=0x7fbea001c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25e95d66e0411d50 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:53.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.522+0000 7fbea4a00640 1 -- 172.21.15.27:0/4211989708 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea8070f30 con 0x7fbea80cf2e0 2024-01-15T18:21:53.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.522+0000 7fbea4a00640 1 -- 172.21.15.27:0/4211989708 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbea00a5560 con 0x7fbea80cf2e0 2024-01-15T18:21:53.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.522+0000 7fbea4a00640 1 -- 172.21.15.27:0/4211989708 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe90001680 con 0x7fbea80cf2e0 2024-01-15T18:21:53.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.526+0000 7fbead200640 1 -- 172.21.15.27:0/4211989708 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbea80cf2e0 msgr2=0x7fbea80701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:53.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.526+0000 7fbead200640 1 --2- 172.21.15.27:0/4211989708 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbea80cf2e0 0x7fbea80701e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbea0009d90 tx=0x7fbea001c870 comp rx=0 tx=0).stop 2024-01-15T18:21:53.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.526+0000 7fbead200640 1 -- 172.21.15.27:0/4211989708 shutdown_connections 2024-01-15T18:21:53.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.526+0000 7fbead200640 1 --2- 172.21.15.27:0/4211989708 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbea80cf2e0 0x7fbea80701e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:53.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.526+0000 7fbead200640 1 --2- 172.21.15.27:0/4211989708 >> v2:172.21.15.93:3300/0 conn(0x7fbea8158900 0x7fbea815f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:53.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.526+0000 7fbead200640 1 -- 172.21.15.27:0/4211989708 >> 172.21.15.27:0/4211989708 conn(0x7fbea80cb700 msgr2=0x7fbea80cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:53.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.526+0000 7fbead200640 1 -- 172.21.15.27:0/4211989708 shutdown_connections 2024-01-15T18:21:53.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.526+0000 7fbead200640 1 -- 172.21.15.27:0/4211989708 wait complete. 2024-01-15T18:21:53.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.526+0000 7fbead200640 1 Processor -- start 2024-01-15T18:21:53.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.526+0000 7fbead200640 1 -- start start 2024-01-15T18:21:53.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.526+0000 7fbead200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbea80cf2e0 0x7fbea8144260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:53.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.526+0000 7fbead200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbea8158900 0x7fbea8144800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:53.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.526+0000 7fbead200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbea815bf00 0x7fbea8157300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:53.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.526+0000 7fbead200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea8145a80 con 0x7fbea80cf2e0 2024-01-15T18:21:53.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.526+0000 7fbead200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea8144e70 con 0x7fbea815bf00 2024-01-15T18:21:53.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.526+0000 7fbead200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbea8145150 con 0x7fbea8158900 2024-01-15T18:21:53.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.526+0000 7fbea5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbea80cf2e0 0x7fbea8144260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:53.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.526+0000 7fbea5e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbea80cf2e0 0x7fbea8144260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56084/0 (socket says 172.21.15.27:56084) 2024-01-15T18:21:53.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.526+0000 7fbea5e00640 1 -- 172.21.15.27:0/2852373813 learned_addr learned my addr 172.21.15.27:0/2852373813 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:21:53.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.526+0000 7fbea5400640 1 --2- 172.21.15.27:0/2852373813 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbea8158900 0x7fbea8144800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:53.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.526+0000 7fbea6800640 1 --2- 172.21.15.27:0/2852373813 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbea815bf00 0x7fbea8157300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:53.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.530+0000 7fbea5e00640 1 -- 172.21.15.27:0/2852373813 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbea8158900 msgr2=0x7fbea8144800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:53.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.530+0000 7fbea5e00640 1 --2- 172.21.15.27:0/2852373813 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbea8158900 0x7fbea8144800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:53.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.530+0000 7fbea5e00640 1 -- 172.21.15.27:0/2852373813 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbea815bf00 msgr2=0x7fbea8157300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:53.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.530+0000 7fbea5e00640 1 --2- 172.21.15.27:0/2852373813 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbea815bf00 0x7fbea8157300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:53.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.530+0000 7fbea5e00640 1 -- 172.21.15.27:0/2852373813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea0009a70 con 0x7fbea80cf2e0 2024-01-15T18:21:53.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.530+0000 7fbea5e00640 1 --2- 172.21.15.27:0/2852373813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbea80cf2e0 0x7fbea8144260 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fbe9800e930 tx=0x7fbe9800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:53.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.530+0000 7fbe9e000640 1 -- 172.21.15.27:0/2852373813 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe98009690 con 0x7fbea80cf2e0 2024-01-15T18:21:53.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.530+0000 7fbe9e000640 1 -- 172.21.15.27:0/2852373813 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe9800ca00 con 0x7fbea80cf2e0 2024-01-15T18:21:53.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.530+0000 7fbead200640 1 -- 172.21.15.27:0/2852373813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbea8145430 con 0x7fbea80cf2e0 2024-01-15T18:21:53.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.530+0000 7fbe9e000640 1 -- 172.21.15.27:0/2852373813 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe9800cd70 con 0x7fbea80cf2e0 2024-01-15T18:21:53.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.530+0000 7fbead200640 1 -- 172.21.15.27:0/2852373813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbea8157840 con 0x7fbea80cf2e0 2024-01-15T18:21:53.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.530+0000 7fbead200640 1 -- 172.21.15.27:0/2852373813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbea804e850 con 0x7fbea80cf2e0 2024-01-15T18:21:53.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.530+0000 7fbe9e000640 1 -- 172.21.15.27:0/2852373813 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7fbe98013470 con 0x7fbea80cf2e0 2024-01-15T18:21:53.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.530+0000 7fbe9e000640 1 --2- 172.21.15.27:0/2852373813 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbe80067b80 0x7fbe8006a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:53.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.534+0000 7fbe9e000640 1 -- 172.21.15.27:0/2852373813 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fbe98083250 con 0x7fbea80cf2e0 2024-01-15T18:21:53.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.534+0000 7fbea5400640 1 --2- 172.21.15.27:0/2852373813 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbe80067b80 0x7fbe8006a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:53.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.534+0000 7fbea5400640 1 --2- 172.21.15.27:0/2852373813 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbe80067b80 0x7fbe8006a040 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fbe90001ec0 tx=0x7fbe900019c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:53.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.534+0000 7fbe9e000640 1 -- 172.21.15.27:0/2852373813 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbe98052050 con 0x7fbea80cf2e0 2024-01-15T18:21:53.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.726+0000 7fbead200640 1 -- 172.21.15.27:0/2852373813 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbea80cf990 con 0x7fbe80067b80 2024-01-15T18:21:53.728 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:21:53.730+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:21:53.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:53.734+0000 7fbe9e000640 1 -- 172.21.15.27:0/2852373813 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fbea80cf990 con 0x7fbe80067b80 2024-01-15T18:21:53.729 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:21:53.729 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:21:53.737 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:21:55.738 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:21:55.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f9794200640 1 Processor -- start 2024-01-15T18:21:55.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f9794200640 1 -- start start 2024-01-15T18:21:55.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f9794200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f978c15f550 0x7f978c0cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:55.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f9794200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f978c0cf110 0x7f978c0701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:55.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f9794200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f978c0b1cb0 con 0x7f978c158900 2024-01-15T18:21:55.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f9794200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f978c0cf580 con 0x7f978c15f550 2024-01-15T18:21:55.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f9794200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c15fa00 con 0x7f978c0cf110 2024-01-15T18:21:55.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f9791800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f978c158900 0x7f978c15bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47392/0 (socket says 172.21.15.27:47392) 2024-01-15T18:21:55.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f9791800640 1 -- 172.21.15.27:0/2647652652 learned_addr learned my addr 172.21.15.27:0/2647652652 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:21:55.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f9792200640 1 --2- 172.21.15.27:0/2647652652 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f978c0cf110 0x7f978c0701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:55.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f9790e00640 1 --2- 172.21.15.27:0/2647652652 >> v2:172.21.15.93:3300/0 conn(0x7f978c15f550 0x7f978c0cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:55.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f978be00640 1 -- 172.21.15.27:0/2647652652 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087209473 0 0) 0x7f978c0b1cb0 con 0x7f978c158900 2024-01-15T18:21:55.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f978be00640 1 -- 172.21.15.27:0/2647652652 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f976c003530 con 0x7f978c158900 2024-01-15T18:21:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f978be00640 1 -- 172.21.15.27:0/2647652652 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3382497135 0 0) 0x7f976c003530 con 0x7f978c158900 2024-01-15T18:21:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f978be00640 1 -- 172.21.15.27:0/2647652652 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f978c0cf110 msgr2=0x7f978c0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f978be00640 1 --2- 172.21.15.27:0/2647652652 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f978c0cf110 0x7f978c0701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f978be00640 1 -- 172.21.15.27:0/2647652652 >> v2:172.21.15.93:3300/0 conn(0x7f978c15f550 msgr2=0x7f978c0cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f978be00640 1 --2- 172.21.15.27:0/2647652652 >> v2:172.21.15.93:3300/0 conn(0x7f978c15f550 0x7f978c0cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f978be00640 1 -- 172.21.15.27:0/2647652652 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c0b1f00 con 0x7f978c158900 2024-01-15T18:21:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f978be00640 1 -- 172.21.15.27:0/2647652652 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f977c001680 con 0x7f978c158900 2024-01-15T18:21:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f9790e00640 1 --2- 172.21.15.27:0/2647652652 >> v2:172.21.15.93:3300/0 conn(0x7f978c15f550 0x7f978c0cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:21:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f9792200640 1 --2- 172.21.15.27:0/2647652652 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f978c0cf110 0x7f978c0701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:21:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f978be00640 1 -- 172.21.15.27:0/2647652652 >> v1:172.21.15.27:6789/0 conn(0x7f978c158900 legacy=0x7f978c15bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f978be00640 1 --2- 172.21.15.27:0/2647652652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f976c0039f0 0x7f976c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f978be00640 1 --2- 172.21.15.27:0/2647652652 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f978c0cf110 0x7f976c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f978be00640 1 --2- 172.21.15.27:0/2647652652 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f976c004d10 0x7f976c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:55.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f978be00640 1 -- 172.21.15.27:0/2647652652 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c0cf580 con 0x7f976c0039f0 2024-01-15T18:21:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f978be00640 1 -- 172.21.15.27:0/2647652652 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c15fa00 con 0x7f976c004d10 2024-01-15T18:21:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f978be00640 1 -- 172.21.15.27:0/2647652652 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f976c00b990 con 0x7f978c0cf110 2024-01-15T18:21:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f9791800640 1 --2- 172.21.15.27:0/2647652652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f976c0039f0 0x7f976c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f9790e00640 1 --2- 172.21.15.27:0/2647652652 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f978c0cf110 0x7f976c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f9792200640 1 --2- 172.21.15.27:0/2647652652 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f976c004d10 0x7f976c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f9791800640 1 -- 172.21.15.27:0/2647652652 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f978c0cf110 msgr2=0x7f976c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f9791800640 1 --2- 172.21.15.27:0/2647652652 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f978c0cf110 0x7f976c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.902+0000 7f9791800640 1 -- 172.21.15.27:0/2647652652 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f976c004d10 msgr2=0x7f976c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.906+0000 7f9791800640 1 --2- 172.21.15.27:0/2647652652 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f976c004d10 0x7f976c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.906+0000 7f9791800640 1 -- 172.21.15.27:0/2647652652 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9784009a70 con 0x7f976c0039f0 2024-01-15T18:21:55.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.906+0000 7f9792200640 1 --2- 172.21.15.27:0/2647652652 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f976c004d10 0x7f976c005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:21:55.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.906+0000 7f9791800640 1 --2- 172.21.15.27:0/2647652652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f976c0039f0 0x7f976c003e10 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f977c00c270 tx=0x7f977c01cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4777a43ce548fb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.906+0000 7f978be00640 1 -- 172.21.15.27:0/2647652652 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c0a5910 con 0x7f976c0039f0 2024-01-15T18:21:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.906+0000 7f978be00640 1 -- 172.21.15.27:0/2647652652 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f977c0a5a70 con 0x7f976c0039f0 2024-01-15T18:21:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.906+0000 7f978be00640 1 -- 172.21.15.27:0/2647652652 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c0a5e40 con 0x7f976c0039f0 2024-01-15T18:21:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.906+0000 7f9794200640 1 -- 172.21.15.27:0/2647652652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f976c0039f0 msgr2=0x7f976c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.906+0000 7f9794200640 1 --2- 172.21.15.27:0/2647652652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f976c0039f0 0x7f976c003e10 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f977c00c270 tx=0x7f977c01cc30 comp rx=0 tx=0).stop 2024-01-15T18:21:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.906+0000 7f9792200640 1 -- 172.21.15.27:0/2647652652 reap_dead start 2024-01-15T18:21:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.906+0000 7f9794200640 1 -- 172.21.15.27:0/2647652652 shutdown_connections 2024-01-15T18:21:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.906+0000 7f9794200640 1 -- 172.21.15.27:0/2647652652 >> 172.21.15.27:0/2647652652 conn(0x7f978c0cb700 msgr2=0x7f978c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.910+0000 7f9794200640 1 -- 172.21.15.27:0/2647652652 shutdown_connections 2024-01-15T18:21:55.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.910+0000 7f9794200640 1 -- 172.21.15.27:0/2647652652 wait complete. 2024-01-15T18:21:55.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.910+0000 7f9794200640 1 Processor -- start 2024-01-15T18:21:55.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.910+0000 7f9794200640 1 -- start start 2024-01-15T18:21:55.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.910+0000 7f9794200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f976c0039f0 0x7f978c1475e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:55.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.910+0000 7f9794200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f978c155a90 0x7f978c160540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:55.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.910+0000 7f9794200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f978c160a80 0x7f978c15c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:55.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.910+0000 7f9794200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c1611b0 con 0x7f976c0039f0 2024-01-15T18:21:55.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.910+0000 7f9794200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c160f00 con 0x7f978c155a90 2024-01-15T18:21:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.910+0000 7f9794200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978c0cde70 con 0x7f978c160a80 2024-01-15T18:21:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.910+0000 7f9791800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f976c0039f0 0x7f978c1475e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.910+0000 7f9791800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f976c0039f0 0x7f978c1475e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40556/0 (socket says 172.21.15.27:40556) 2024-01-15T18:21:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.910+0000 7f9791800640 1 -- 172.21.15.27:0/1950297888 learned_addr learned my addr 172.21.15.27:0/1950297888 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:21:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.910+0000 7f9790e00640 1 --2- 172.21.15.27:0/1950297888 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f978c155a90 0x7f978c160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.910+0000 7f9792200640 1 --2- 172.21.15.27:0/1950297888 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f978c160a80 0x7f978c15c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.910+0000 7f9791800640 1 -- 172.21.15.27:0/1950297888 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f978c160a80 msgr2=0x7f978c15c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.910+0000 7f9791800640 1 --2- 172.21.15.27:0/1950297888 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f978c160a80 0x7f978c15c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.910+0000 7f9791800640 1 -- 172.21.15.27:0/1950297888 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f978c155a90 msgr2=0x7f978c160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.910+0000 7f9791800640 1 --2- 172.21.15.27:0/1950297888 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f978c155a90 0x7f978c160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:55.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.910+0000 7f9791800640 1 -- 172.21.15.27:0/1950297888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978c0cd260 con 0x7f976c0039f0 2024-01-15T18:21:55.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.914+0000 7f9791800640 1 --2- 172.21.15.27:0/1950297888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f976c0039f0 0x7f978c1475e0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f977c00c270 tx=0x7f976c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:55.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.914+0000 7f9789600640 1 -- 172.21.15.27:0/1950297888 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c0a5de0 con 0x7f976c0039f0 2024-01-15T18:21:55.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.914+0000 7f9789600640 1 -- 172.21.15.27:0/1950297888 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f977c0b3070 con 0x7f976c0039f0 2024-01-15T18:21:55.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.914+0000 7f9794200640 1 -- 172.21.15.27:0/1950297888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978c0cd540 con 0x7f976c0039f0 2024-01-15T18:21:55.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.914+0000 7f9789600640 1 -- 172.21.15.27:0/1950297888 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c004a10 con 0x7f976c0039f0 2024-01-15T18:21:55.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.914+0000 7f9794200640 1 -- 172.21.15.27:0/1950297888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978c0cdb00 con 0x7f976c0039f0 2024-01-15T18:21:55.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.914+0000 7f9794200640 1 -- 172.21.15.27:0/1950297888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f978c04e850 con 0x7f976c0039f0 2024-01-15T18:21:55.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.914+0000 7f9789600640 1 -- 172.21.15.27:0/1950297888 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7f977c004b70 con 0x7f976c0039f0 2024-01-15T18:21:55.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.914+0000 7f9789600640 1 --2- 172.21.15.27:0/1950297888 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9764067b80 0x7f976406a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:55.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.914+0000 7f9789600640 1 -- 172.21.15.27:0/1950297888 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f977c0b5070 con 0x7f976c0039f0 2024-01-15T18:21:55.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.914+0000 7f9790e00640 1 --2- 172.21.15.27:0/1950297888 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9764067b80 0x7f976406a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:55.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.918+0000 7f9790e00640 1 --2- 172.21.15.27:0/1950297888 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9764067b80 0x7f976406a040 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f978c06d8a0 tx=0x7f9774009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:55.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:55.918+0000 7f9789600640 1 -- 172.21.15.27:0/1950297888 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f977c0ebae0 con 0x7f976c0039f0 2024-01-15T18:21:56.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:56.106+0000 7f9794200640 1 -- 172.21.15.27:0/1950297888 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f978c0cb170 con 0x7f9764067b80 2024-01-15T18:21:56.111 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:21:56.110+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:21:56.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:56.114+0000 7f9789600640 1 -- 172.21.15.27:0/1950297888 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f978c0cb170 con 0x7f9764067b80 2024-01-15T18:21:56.111 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:21:56.112 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:21:56.120 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:21:58.122 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:21:58.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.290+0000 7fd4d1200640 1 Processor -- start 2024-01-15T18:21:58.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.290+0000 7fd4d1200640 1 -- start start 2024-01-15T18:21:58.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.290+0000 7fd4d1200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd4cc0d02c0 0x7fd4cc06db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:58.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.290+0000 7fd4d1200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4cc0af4f0 0x7fd4cc06e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:58.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.290+0000 7fd4d1200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4cc15e790 con 0x7fd4cc159f30 2024-01-15T18:21:58.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.290+0000 7fd4d1200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4cc15a310 con 0x7fd4cc0d02c0 2024-01-15T18:21:58.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.290+0000 7fd4d1200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc0ad730 con 0x7fd4cc0af4f0 2024-01-15T18:21:58.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.290+0000 7fd4c9400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd4cc0d02c0 0x7fd4cc06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:58.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.290+0000 7fd4c9400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd4cc0d02c0 0x7fd4cc06db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40570/0 (socket says 172.21.15.27:40570) 2024-01-15T18:21:58.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.290+0000 7fd4c9400640 1 -- 172.21.15.27:0/2500318525 learned_addr learned my addr 172.21.15.27:0/2500318525 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:21:58.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.290+0000 7fd4ca800640 1 --2- 172.21.15.27:0/2500318525 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4cc0af4f0 0x7fd4cc06e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:58.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.290+0000 7fd4c9400640 1 -- 172.21.15.27:0/2500318525 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4cc0af4f0 msgr2=0x7fd4cc06e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:58.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.290+0000 7fd4c9400640 1 --2- 172.21.15.27:0/2500318525 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4cc0af4f0 0x7fd4cc06e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:58.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.290+0000 7fd4c9400640 1 -- 172.21.15.27:0/2500318525 >> v1:172.21.15.93:6789/0 conn(0x7fd4cc159f30 legacy=0x7fd4cc0cfbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:58.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.290+0000 7fd4c9400640 1 -- 172.21.15.27:0/2500318525 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4cc15db80 con 0x7fd4cc0d02c0 2024-01-15T18:21:58.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.294+0000 7fd4c9400640 1 --2- 172.21.15.27:0/2500318525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4cc0d02c0 0x7fd4cc06db70 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4009870 tx=0x7fd4b401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7624b698f00ed4a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:58.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.294+0000 7fd4c8a00640 1 -- 172.21.15.27:0/2500318525 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b40ab680 con 0x7fd4cc0d02c0 2024-01-15T18:21:58.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.294+0000 7fd4c8a00640 1 -- 172.21.15.27:0/2500318525 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b40ab7e0 con 0x7fd4cc0d02c0 2024-01-15T18:21:58.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.294+0000 7fd4c8a00640 1 -- 172.21.15.27:0/2500318525 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b40abb70 con 0x7fd4cc0d02c0 2024-01-15T18:21:58.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4d1200640 1 -- 172.21.15.27:0/2500318525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4cc0d02c0 msgr2=0x7fd4cc06db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:58.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4d1200640 1 --2- 172.21.15.27:0/2500318525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4cc0d02c0 0x7fd4cc06db70 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4009870 tx=0x7fd4b401c480 comp rx=0 tx=0).stop 2024-01-15T18:21:58.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4d1200640 1 -- 172.21.15.27:0/2500318525 shutdown_connections 2024-01-15T18:21:58.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4d1200640 1 --2- 172.21.15.27:0/2500318525 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4cc0af4f0 0x7fd4cc06e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:58.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4d1200640 1 --2- 172.21.15.27:0/2500318525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4cc0d02c0 0x7fd4cc06db70 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:58.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4d1200640 1 -- 172.21.15.27:0/2500318525 >> 172.21.15.27:0/2500318525 conn(0x7fd4cc06b600 msgr2=0x7fd4cc06ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:58.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4d1200640 1 -- 172.21.15.27:0/2500318525 shutdown_connections 2024-01-15T18:21:58.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4d1200640 1 -- 172.21.15.27:0/2500318525 wait complete. 2024-01-15T18:21:58.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4d1200640 1 Processor -- start 2024-01-15T18:21:58.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4d1200640 1 -- start start 2024-01-15T18:21:58.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4d1200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4cc0af4f0 0x7fd4cc0b0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4d1200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4cc0d02c0 0x7fd4cc06a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4d1200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4cc159f30 0x7fd4cc152b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4d1200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc0b1270 con 0x7fd4cc0af4f0 2024-01-15T18:21:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4d1200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc0b1550 con 0x7fd4cc0d02c0 2024-01-15T18:21:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4d1200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4cc0b1830 con 0x7fd4cc159f30 2024-01-15T18:21:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4c9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4cc0af4f0 0x7fd4cc0b0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4c9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4cc0af4f0 0x7fd4cc0b0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40582/0 (socket says 172.21.15.27:40582) 2024-01-15T18:21:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4c9e00640 1 -- 172.21.15.27:0/202638139 learned_addr learned my addr 172.21.15.27:0/202638139 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:21:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4ca800640 1 --2- 172.21.15.27:0/202638139 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4cc159f30 0x7fd4cc152b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4c9400640 1 --2- 172.21.15.27:0/202638139 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4cc0d02c0 0x7fd4cc06a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4c9e00640 1 -- 172.21.15.27:0/202638139 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4cc159f30 msgr2=0x7fd4cc152b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4c9e00640 1 --2- 172.21.15.27:0/202638139 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4cc159f30 0x7fd4cc152b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4c9e00640 1 -- 172.21.15.27:0/202638139 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4cc0d02c0 msgr2=0x7fd4cc06a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:58.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4c9e00640 1 --2- 172.21.15.27:0/202638139 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd4cc0d02c0 0x7fd4cc06a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.298+0000 7fd4c9e00640 1 -- 172.21.15.27:0/202638139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b40094d0 con 0x7fd4cc0af4f0 2024-01-15T18:21:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.302+0000 7fd4ca800640 1 --2- 172.21.15.27:0/202638139 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd4cc159f30 0x7fd4cc152b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:21:58.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.302+0000 7fd4c9e00640 1 --2- 172.21.15.27:0/202638139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd4cc0af4f0 0x7fd4cc0b0790 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc00c500 tx=0x7fd4bc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.302+0000 7fd4c2000640 1 -- 172.21.15.27:0/202638139 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc005220 con 0x7fd4cc0af4f0 2024-01-15T18:21:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.302+0000 7fd4c2000640 1 -- 172.21.15.27:0/202638139 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4bc005380 con 0x7fd4cc0af4f0 2024-01-15T18:21:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.302+0000 7fd4c2000640 1 -- 172.21.15.27:0/202638139 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4bc007270 con 0x7fd4cc0af4f0 2024-01-15T18:21:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.302+0000 7fd4d1200640 1 -- 172.21.15.27:0/202638139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4cc0b1b10 con 0x7fd4cc0af4f0 2024-01-15T18:21:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.302+0000 7fd4d1200640 1 -- 172.21.15.27:0/202638139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4cc0b0cd0 con 0x7fd4cc0af4f0 2024-01-15T18:21:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.302+0000 7fd4d1200640 1 -- 172.21.15.27:0/202638139 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4cc04e850 con 0x7fd4cc0af4f0 2024-01-15T18:21:58.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.302+0000 7fd4c2000640 1 -- 172.21.15.27:0/202638139 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7fd4bc0073d0 con 0x7fd4cc0af4f0 2024-01-15T18:21:58.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.302+0000 7fd4c2000640 1 --2- 172.21.15.27:0/202638139 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd4a4067b80 0x7fd4a406a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:58.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.302+0000 7fd4c2000640 1 -- 172.21.15.27:0/202638139 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fd4bc082be0 con 0x7fd4cc0af4f0 2024-01-15T18:21:58.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.302+0000 7fd4c9400640 1 --2- 172.21.15.27:0/202638139 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd4a4067b80 0x7fd4a406a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.306+0000 7fd4c9400640 1 --2- 172.21.15.27:0/202638139 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd4a4067b80 0x7fd4a406a040 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fd4b40af000 tx=0x7fd4b40b6040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.306+0000 7fd4c2000640 1 -- 172.21.15.27:0/202638139 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd4bc051580 con 0x7fd4cc0af4f0 2024-01-15T18:21:58.351 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:21:58.354 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:21:58.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.498+0000 7fd4d1200640 1 -- 172.21.15.27:0/202638139 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd4cc0d0150 con 0x7fd4a4067b80 2024-01-15T18:21:58.501 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:21:58.502+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:21:58.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:21:58.506+0000 7fd4c2000640 1 -- 172.21.15.27:0/202638139 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd4cc0d0150 con 0x7fd4a4067b80 2024-01-15T18:21:58.501 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:21:58.502 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:21:58.510 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:00.511 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:22:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.674+0000 7f7eeea00640 1 Processor -- start 2024-01-15T18:22:00.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.674+0000 7f7eeea00640 1 -- start start 2024-01-15T18:22:00.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.674+0000 7f7eeea00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ee80cf100 0x7f7ee80cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:00.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.674+0000 7f7eeea00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee80ad920 con 0x7f7ee81468d0 2024-01-15T18:22:00.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.674+0000 7f7eeea00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee80ac880 con 0x7f7ee80ccea0 2024-01-15T18:22:00.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.674+0000 7f7eeea00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee80acb60 con 0x7f7ee80cf100 2024-01-15T18:22:00.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.674+0000 7f7ee7e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7ee81468d0 0x7f7ee80ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47404/0 (socket says 172.21.15.27:47404) 2024-01-15T18:22:00.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.674+0000 7f7ee7e00640 1 -- 172.21.15.27:0/817711321 learned_addr learned my addr 172.21.15.27:0/817711321 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:22:00.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.674+0000 7f7ee7400640 1 --2- 172.21.15.27:0/817711321 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ee80cf100 0x7f7ee80cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:00.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.674+0000 7f7ee6a00640 1 -- 172.21.15.27:0/817711321 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 541076195 0 0) 0x7f7ed8001120 con 0x7f7ee81468d0 2024-01-15T18:22:00.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.674+0000 7f7ee6a00640 1 -- 172.21.15.27:0/817711321 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ec8003530 con 0x7f7ee81468d0 2024-01-15T18:22:00.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.678+0000 7f7ee6a00640 1 -- 172.21.15.27:0/817711321 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3051756679 0 0) 0x7f7ec8003530 con 0x7f7ee81468d0 2024-01-15T18:22:00.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.678+0000 7f7ee6a00640 1 -- 172.21.15.27:0/817711321 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ee80cf100 msgr2=0x7f7ee80cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:00.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.678+0000 7f7ee6a00640 1 --2- 172.21.15.27:0/817711321 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ee80cf100 0x7f7ee80cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:00.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.678+0000 7f7ee6a00640 1 -- 172.21.15.27:0/817711321 >> v1:172.21.15.93:6789/0 conn(0x7f7ee80ccea0 legacy=0x7f7ee806c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:00.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.678+0000 7f7ee6a00640 1 -- 172.21.15.27:0/817711321 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee80ace40 con 0x7f7ee81468d0 2024-01-15T18:22:00.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.678+0000 7f7ee6a00640 1 -- 172.21.15.27:0/817711321 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f7ed8001810 con 0x7f7ee81468d0 2024-01-15T18:22:00.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.678+0000 7f7ee6a00640 1 -- 172.21.15.27:0/817711321 >> v1:172.21.15.27:6789/0 conn(0x7f7ee81468d0 legacy=0x7f7ee80ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:00.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.678+0000 7f7ee6a00640 1 --2- 172.21.15.27:0/817711321 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ee80cf100 0x7f7ec8003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:00.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.678+0000 7f7ee6a00640 1 --2- 172.21.15.27:0/817711321 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ec8003f60 0x7f7ec80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:00.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.678+0000 7f7ee6a00640 1 --2- 172.21.15.27:0/817711321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec8004d10 0x7f7ec8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:00.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.678+0000 7f7ee6a00640 1 -- 172.21.15.27:0/817711321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee80acb60 con 0x7f7ec8004d10 2024-01-15T18:22:00.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.678+0000 7f7ee6a00640 1 -- 172.21.15.27:0/817711321 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec800bad0 con 0x7f7ec8003f60 2024-01-15T18:22:00.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.678+0000 7f7ee6a00640 1 -- 172.21.15.27:0/817711321 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec800bc90 con 0x7f7ee80cf100 2024-01-15T18:22:00.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.678+0000 7f7eeca00640 1 --2- 172.21.15.27:0/817711321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec8004d10 0x7f7ec8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:00.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.678+0000 7f7ee7e00640 1 --2- 172.21.15.27:0/817711321 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ee80cf100 0x7f7ec8003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:00.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.678+0000 7f7ee7400640 1 --2- 172.21.15.27:0/817711321 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ec8003f60 0x7f7ec80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:00.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.678+0000 7f7eeca00640 1 -- 172.21.15.27:0/817711321 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ee80cf100 msgr2=0x7f7ec8003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:00.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.678+0000 7f7eeca00640 1 --2- 172.21.15.27:0/817711321 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ee80cf100 0x7f7ec8003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:00.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.678+0000 7f7eeca00640 1 -- 172.21.15.27:0/817711321 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ec8003f60 msgr2=0x7f7ec80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:00.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.678+0000 7f7eeca00640 1 --2- 172.21.15.27:0/817711321 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ec8003f60 0x7f7ec80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:00.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.678+0000 7f7eeca00640 1 -- 172.21.15.27:0/817711321 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec8003530 con 0x7f7ec8004d10 2024-01-15T18:22:00.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.682+0000 7f7eeca00640 1 --2- 172.21.15.27:0/817711321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec8004d10 0x7f7ec8005190 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0001f60 tx=0x7f7ee001c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18ad2fb6417a8938 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:00.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.682+0000 7f7ee6a00640 1 -- 172.21.15.27:0/817711321 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee00a5750 con 0x7f7ec8004d10 2024-01-15T18:22:00.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.682+0000 7f7ee6a00640 1 -- 172.21.15.27:0/817711321 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee00a58b0 con 0x7f7ec8004d10 2024-01-15T18:22:00.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.682+0000 7f7ee6a00640 1 -- 172.21.15.27:0/817711321 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee00a5ca0 con 0x7f7ec8004d10 2024-01-15T18:22:00.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.682+0000 7f7eeea00640 1 -- 172.21.15.27:0/817711321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec8004d10 msgr2=0x7f7ec8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:00.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.682+0000 7f7eeea00640 1 --2- 172.21.15.27:0/817711321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec8004d10 0x7f7ec8005190 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0001f60 tx=0x7f7ee001c860 comp rx=0 tx=0).stop 2024-01-15T18:22:00.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.682+0000 7f7eeca00640 1 -- 172.21.15.27:0/817711321 reap_dead start 2024-01-15T18:22:00.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.682+0000 7f7eeea00640 1 -- 172.21.15.27:0/817711321 shutdown_connections 2024-01-15T18:22:00.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.682+0000 7f7eeea00640 1 -- 172.21.15.27:0/817711321 >> 172.21.15.27:0/817711321 conn(0x7f7ee80cb700 msgr2=0x7f7ee80cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:22:00.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.682+0000 7f7eeea00640 1 -- 172.21.15.27:0/817711321 shutdown_connections 2024-01-15T18:22:00.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.682+0000 7f7eeea00640 1 -- 172.21.15.27:0/817711321 wait complete. 2024-01-15T18:22:00.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.682+0000 7f7eeea00640 1 Processor -- start 2024-01-15T18:22:00.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.682+0000 7f7eeea00640 1 -- start start 2024-01-15T18:22:00.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.686+0000 7f7eeea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec8004d10 0x7f7ee815b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:00.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.686+0000 7f7eeea00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ee815d710 0x7f7ee815ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:00.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.686+0000 7f7eeea00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ee8145c80 0x7f7ee8146100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:00.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.686+0000 7f7eeea00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee815f1d0 con 0x7f7ec8004d10 2024-01-15T18:22:00.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.686+0000 7f7eeea00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee815ad40 con 0x7f7ee815d710 2024-01-15T18:22:00.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.686+0000 7f7eeea00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee815dbf0 con 0x7f7ee8145c80 2024-01-15T18:22:00.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.686+0000 7f7ee7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec8004d10 0x7f7ee815b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:00.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.686+0000 7f7ee7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec8004d10 0x7f7ee815b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40592/0 (socket says 172.21.15.27:40592) 2024-01-15T18:22:00.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.686+0000 7f7ee7e00640 1 -- 172.21.15.27:0/2885921786 learned_addr learned my addr 172.21.15.27:0/2885921786 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:00.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.686+0000 7f7ee7400640 1 --2- 172.21.15.27:0/2885921786 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ee815d710 0x7f7ee815ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:00.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.686+0000 7f7eeca00640 1 --2- 172.21.15.27:0/2885921786 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ee8145c80 0x7f7ee8146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:00.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.686+0000 7f7ee7e00640 1 -- 172.21.15.27:0/2885921786 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ee8145c80 msgr2=0x7f7ee8146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:00.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.686+0000 7f7ee7e00640 1 --2- 172.21.15.27:0/2885921786 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f7ee8145c80 0x7f7ee8146100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:00.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.686+0000 7f7ee7e00640 1 -- 172.21.15.27:0/2885921786 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ee815d710 msgr2=0x7f7ee815ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:00.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.686+0000 7f7ee7e00640 1 --2- 172.21.15.27:0/2885921786 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7ee815d710 0x7f7ee815ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:00.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.686+0000 7f7ee7e00640 1 -- 172.21.15.27:0/2885921786 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee815ded0 con 0x7f7ec8004d10 2024-01-15T18:22:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.686+0000 7f7ee7e00640 1 --2- 172.21.15.27:0/2885921786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ec8004d10 0x7f7ee815b640 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f7ed80124c0 tx=0x7f7ed80128c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.686+0000 7f7edfe00640 1 -- 172.21.15.27:0/2885921786 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8005780 con 0x7f7ec8004d10 2024-01-15T18:22:00.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.686+0000 7f7edfe00640 1 -- 172.21.15.27:0/2885921786 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed8014750 con 0x7f7ec8004d10 2024-01-15T18:22:00.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.686+0000 7f7eeea00640 1 -- 172.21.15.27:0/2885921786 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee815e1b0 con 0x7f7ec8004d10 2024-01-15T18:22:00.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.686+0000 7f7edfe00640 1 -- 172.21.15.27:0/2885921786 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8014b00 con 0x7f7ec8004d10 2024-01-15T18:22:00.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.686+0000 7f7eeea00640 1 -- 172.21.15.27:0/2885921786 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee815e770 con 0x7f7ec8004d10 2024-01-15T18:22:00.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.686+0000 7f7eeea00640 1 -- 172.21.15.27:0/2885921786 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee804e850 con 0x7f7ec8004d10 2024-01-15T18:22:00.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.686+0000 7f7edfe00640 1 -- 172.21.15.27:0/2885921786 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7f7ed8014c60 con 0x7f7ec8004d10 2024-01-15T18:22:00.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.690+0000 7f7edfe00640 1 --2- 172.21.15.27:0/2885921786 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7ec0067bb0 0x7f7ec006a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:00.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.690+0000 7f7edfe00640 1 -- 172.21.15.27:0/2885921786 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f7ed8085cb0 con 0x7f7ec8004d10 2024-01-15T18:22:00.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.690+0000 7f7ee7400640 1 --2- 172.21.15.27:0/2885921786 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7ec0067bb0 0x7f7ec006a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:00.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.690+0000 7f7ee7400640 1 --2- 172.21.15.27:0/2885921786 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7ec0067bb0 0x7f7ec006a070 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8160460 tx=0x7f7ed0009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:00.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.694+0000 7f7edfe00640 1 -- 172.21.15.27:0/2885921786 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7ed80545e0 con 0x7f7ec8004d10 2024-01-15T18:22:00.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.878+0000 7f7eeea00640 1 -- 172.21.15.27:0/2885921786 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7ee8146cb0 con 0x7f7ec0067bb0 2024-01-15T18:22:00.882 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:22:00.882+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:22:00.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:00.886+0000 7f7edfe00640 1 -- 172.21.15.27:0/2885921786 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7ed0019070 con 0x7f7ec0067bb0 2024-01-15T18:22:00.882 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:22:00.883 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:22:00.891 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:02.892 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:22:03.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.054+0000 7feb25200640 1 Processor -- start 2024-01-15T18:22:03.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.054+0000 7feb25200640 1 -- start start 2024-01-15T18:22:03.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.058+0000 7feb25200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feb2006c430 0x7feb20070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:03.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.058+0000 7feb25200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7feb200ce760 0x7feb200b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:03.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.058+0000 7feb25200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb200ccfb0 con 0x7feb200b1db0 2024-01-15T18:22:03.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.058+0000 7feb25200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb200cd290 con 0x7feb200ce760 2024-01-15T18:22:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.058+0000 7feb25200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb200cd570 con 0x7feb2006c430 2024-01-15T18:22:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.058+0000 7feb1d400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7feb200b1db0 0x7feb200ce050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47408/0 (socket says 172.21.15.27:47408) 2024-01-15T18:22:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.058+0000 7feb1d400640 1 -- 172.21.15.27:0/4179851495 learned_addr learned my addr 172.21.15.27:0/4179851495 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:22:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.058+0000 7feb1de00640 1 --2- 172.21.15.27:0/4179851495 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feb2006c430 0x7feb20070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:03.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.058+0000 7feb1e800640 1 --2- 172.21.15.27:0/4179851495 >> v2:172.21.15.93:3300/0 conn(0x7feb200ce760 0x7feb200b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.058+0000 7feb1ca00640 1 -- 172.21.15.27:0/4179851495 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1705189450 0 0) 0x7feb200ccfb0 con 0x7feb200b1db0 2024-01-15T18:22:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.058+0000 7feb1ca00640 1 -- 172.21.15.27:0/4179851495 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb00003530 con 0x7feb200b1db0 2024-01-15T18:22:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.058+0000 7feb1e800640 1 -- 172.21.15.27:0/4179851495 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feb2006c430 msgr2=0x7feb20070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.058+0000 7feb1e800640 1 --2- 172.21.15.27:0/4179851495 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feb2006c430 0x7feb20070ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.058+0000 7feb1e800640 1 -- 172.21.15.27:0/4179851495 >> v1:172.21.15.27:6789/0 conn(0x7feb200b1db0 legacy=0x7feb200ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:03.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.058+0000 7feb1e800640 1 -- 172.21.15.27:0/4179851495 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb200cd850 con 0x7feb200ce760 2024-01-15T18:22:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.062+0000 7feb1e800640 1 --2- 172.21.15.27:0/4179851495 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feb200ce760 0x7feb200b2210 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7feb18009d90 tx=0x7feb1801c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cab7a56092cc483 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.062+0000 7feb1ca00640 1 -- 172.21.15.27:0/4179851495 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb180a5810 con 0x7feb200ce760 2024-01-15T18:22:03.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.062+0000 7feb1ca00640 1 -- 172.21.15.27:0/4179851495 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb180a5970 con 0x7feb200ce760 2024-01-15T18:22:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.062+0000 7feb1ca00640 1 -- 172.21.15.27:0/4179851495 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb180a5d00 con 0x7feb200ce760 2024-01-15T18:22:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.062+0000 7feb25200640 1 -- 172.21.15.27:0/4179851495 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feb200ce760 msgr2=0x7feb200b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:03.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.062+0000 7feb25200640 1 --2- 172.21.15.27:0/4179851495 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feb200ce760 0x7feb200b2210 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7feb18009d90 tx=0x7feb1801c8c0 comp rx=0 tx=0).stop 2024-01-15T18:22:03.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.062+0000 7feb25200640 1 -- 172.21.15.27:0/4179851495 shutdown_connections 2024-01-15T18:22:03.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.062+0000 7feb25200640 1 --2- 172.21.15.27:0/4179851495 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feb200ce760 0x7feb200b2210 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:03.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.062+0000 7feb25200640 1 --2- 172.21.15.27:0/4179851495 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feb2006c430 0x7feb20070ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:03.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.062+0000 7feb25200640 1 -- 172.21.15.27:0/4179851495 >> 172.21.15.27:0/4179851495 conn(0x7feb200cb700 msgr2=0x7feb200cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:22:03.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.062+0000 7feb25200640 1 -- 172.21.15.27:0/4179851495 shutdown_connections 2024-01-15T18:22:03.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.062+0000 7feb25200640 1 -- 172.21.15.27:0/4179851495 wait complete. 2024-01-15T18:22:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.062+0000 7feb25200640 1 Processor -- start 2024-01-15T18:22:03.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.062+0000 7feb25200640 1 -- start start 2024-01-15T18:22:03.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.066+0000 7feb25200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feb2006c430 0x7feb2014d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:03.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.066+0000 7feb25200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb200b1db0 0x7feb20150910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:03.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.066+0000 7feb25200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feb200ce760 0x7feb20150e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:03.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.066+0000 7feb25200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb201f58c0 con 0x7feb200b1db0 2024-01-15T18:22:03.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.066+0000 7feb25200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb201f5ba0 con 0x7feb2006c430 2024-01-15T18:22:03.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.066+0000 7feb25200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb201f5e80 con 0x7feb200ce760 2024-01-15T18:22:03.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.066+0000 7feb1d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb200b1db0 0x7feb20150910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:03.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.066+0000 7feb1d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb200b1db0 0x7feb20150910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40608/0 (socket says 172.21.15.27:40608) 2024-01-15T18:22:03.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.066+0000 7feb1d400640 1 -- 172.21.15.27:0/730252574 learned_addr learned my addr 172.21.15.27:0/730252574 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:03.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.066+0000 7feb1e800640 1 --2- 172.21.15.27:0/730252574 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feb200ce760 0x7feb20150e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:03.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.066+0000 7feb1de00640 1 --2- 172.21.15.27:0/730252574 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feb2006c430 0x7feb2014d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:03.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.066+0000 7feb1d400640 1 -- 172.21.15.27:0/730252574 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feb200ce760 msgr2=0x7feb20150e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:03.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.066+0000 7feb1d400640 1 --2- 172.21.15.27:0/730252574 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7feb200ce760 0x7feb20150e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:03.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.066+0000 7feb1d400640 1 -- 172.21.15.27:0/730252574 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feb2006c430 msgr2=0x7feb2014d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:03.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.066+0000 7feb1d400640 1 --2- 172.21.15.27:0/730252574 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7feb2006c430 0x7feb2014d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:03.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.066+0000 7feb1d400640 1 -- 172.21.15.27:0/730252574 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb18009a70 con 0x7feb200b1db0 2024-01-15T18:22:03.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.066+0000 7feb1d400640 1 --2- 172.21.15.27:0/730252574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feb200b1db0 0x7feb20150910 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7feb0800c4d0 tx=0x7feb0800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:03.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.066+0000 7feb16000640 1 -- 172.21.15.27:0/730252574 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb08005cf0 con 0x7feb200b1db0 2024-01-15T18:22:03.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.066+0000 7feb16000640 1 -- 172.21.15.27:0/730252574 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb08005e50 con 0x7feb200b1db0 2024-01-15T18:22:03.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.066+0000 7feb25200640 1 -- 172.21.15.27:0/730252574 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb200712f0 con 0x7feb200b1db0 2024-01-15T18:22:03.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.066+0000 7feb16000640 1 -- 172.21.15.27:0/730252574 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb08007270 con 0x7feb200b1db0 2024-01-15T18:22:03.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.066+0000 7feb25200640 1 -- 172.21.15.27:0/730252574 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb201f6440 con 0x7feb200b1db0 2024-01-15T18:22:03.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.066+0000 7feb25200640 1 -- 172.21.15.27:0/730252574 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb2004e850 con 0x7feb200b1db0 2024-01-15T18:22:03.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.070+0000 7feb16000640 1 -- 172.21.15.27:0/730252574 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7feb080073d0 con 0x7feb200b1db0 2024-01-15T18:22:03.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.070+0000 7feb16000640 1 --2- 172.21.15.27:0/730252574 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7feaf8067bb0 0x7feaf806a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:03.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.070+0000 7feb16000640 1 -- 172.21.15.27:0/730252574 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7feb08082830 con 0x7feb200b1db0 2024-01-15T18:22:03.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.070+0000 7feb1de00640 1 --2- 172.21.15.27:0/730252574 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7feaf8067bb0 0x7feaf806a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:03.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.070+0000 7feb1de00640 1 --2- 172.21.15.27:0/730252574 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7feaf8067bb0 0x7feaf806a070 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7feb10009b60 tx=0x7feb10009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:03.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.074+0000 7feb16000640 1 -- 172.21.15.27:0/730252574 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7feb08051230 con 0x7feb200b1db0 2024-01-15T18:22:03.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.262+0000 7feb25200640 1 -- 172.21.15.27:0/730252574 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7feb2015ced0 con 0x7feaf8067bb0 2024-01-15T18:22:03.267 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:22:03.270+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:22:03.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:03.270+0000 7feb16000640 1 -- 172.21.15.27:0/730252574 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7feb2015ced0 con 0x7feaf8067bb0 2024-01-15T18:22:03.268 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:22:03.268 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:22:03.277 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:05.279 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:22:05.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.442+0000 7fc4c1000640 1 Processor -- start 2024-01-15T18:22:05.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.442+0000 7fc4c1000640 1 -- start start 2024-01-15T18:22:05.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.442+0000 7fc4c1000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc4bc15d190 0x7fc4bc0cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:05.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.442+0000 7fc4c1000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fc4bc06c520 0x7fc4bc0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:05.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.442+0000 7fc4c1000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4bc0b0d70 con 0x7fc4bc15d570 2024-01-15T18:22:05.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.442+0000 7fc4c1000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4bc0d0540 con 0x7fc4bc15d190 2024-01-15T18:22:05.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.442+0000 7fc4c1000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4bc0b0440 con 0x7fc4bc06c520 2024-01-15T18:22:05.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.442+0000 7fc4b9e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc4bc15d190 0x7fc4bc0cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:05.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.442+0000 7fc4b9e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc4bc15d190 0x7fc4bc0cfcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53878/0 (socket says 172.21.15.27:53878) 2024-01-15T18:22:05.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.442+0000 7fc4b9e00640 1 -- 172.21.15.27:0/2680147404 learned_addr learned my addr 172.21.15.27:0/2680147404 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:05.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.442+0000 7fc4ba800640 1 --2- 172.21.15.27:0/2680147404 >> v2:172.21.15.93:3300/0 conn(0x7fc4bc06c520 0x7fc4bc0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:05.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.442+0000 7fc4b9e00640 1 -- 172.21.15.27:0/2680147404 >> v2:172.21.15.93:3300/0 conn(0x7fc4bc06c520 msgr2=0x7fc4bc0cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:05.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.442+0000 7fc4b9e00640 1 --2- 172.21.15.27:0/2680147404 >> v2:172.21.15.93:3300/0 conn(0x7fc4bc06c520 0x7fc4bc0cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:05.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.442+0000 7fc4b9e00640 1 -- 172.21.15.27:0/2680147404 >> v1:172.21.15.93:6789/0 conn(0x7fc4bc15d570 legacy=0x7fc4bc06be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:05.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.442+0000 7fc4b9e00640 1 -- 172.21.15.27:0/2680147404 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4bc0b0720 con 0x7fc4bc15d190 2024-01-15T18:22:05.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.446+0000 7fc4b9e00640 1 --2- 172.21.15.27:0/2680147404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4bc15d190 0x7fc4bc0cfcf0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc0717e0 tx=0x7fc4ac01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e839a701212389a1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:05.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.446+0000 7fc4b8a00640 1 -- 172.21.15.27:0/2680147404 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac0ab650 con 0x7fc4bc15d190 2024-01-15T18:22:05.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.446+0000 7fc4b8a00640 1 -- 172.21.15.27:0/2680147404 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4ac0ab7b0 con 0x7fc4bc15d190 2024-01-15T18:22:05.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.446+0000 7fc4b8a00640 1 -- 172.21.15.27:0/2680147404 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac0abb40 con 0x7fc4bc15d190 2024-01-15T18:22:05.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.446+0000 7fc4c1000640 1 -- 172.21.15.27:0/2680147404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4bc15d190 msgr2=0x7fc4bc0cfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:05.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.446+0000 7fc4c1000640 1 --2- 172.21.15.27:0/2680147404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4bc15d190 0x7fc4bc0cfcf0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc0717e0 tx=0x7fc4ac01c490 comp rx=0 tx=0).stop 2024-01-15T18:22:05.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.446+0000 7fc4c1000640 1 -- 172.21.15.27:0/2680147404 shutdown_connections 2024-01-15T18:22:05.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.446+0000 7fc4c1000640 1 --2- 172.21.15.27:0/2680147404 >> v2:172.21.15.93:3300/0 conn(0x7fc4bc06c520 0x7fc4bc0cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:05.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.446+0000 7fc4c1000640 1 --2- 172.21.15.27:0/2680147404 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4bc15d190 0x7fc4bc0cfcf0 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:05.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.446+0000 7fc4c1000640 1 -- 172.21.15.27:0/2680147404 >> 172.21.15.27:0/2680147404 conn(0x7fc4bc0d0e70 msgr2=0x7fc4bc0d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:22:05.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.446+0000 7fc4c1000640 1 -- 172.21.15.27:0/2680147404 shutdown_connections 2024-01-15T18:22:05.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.446+0000 7fc4c1000640 1 -- 172.21.15.27:0/2680147404 wait complete. 2024-01-15T18:22:05.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.450+0000 7fc4c1000640 1 Processor -- start 2024-01-15T18:22:05.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.450+0000 7fc4c1000640 1 -- start start 2024-01-15T18:22:05.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.450+0000 7fc4c1000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4bc06c520 0x7fc4bc0c68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:05.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.450+0000 7fc4c1000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc4bc15d190 0x7fc4bc148fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:05.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.450+0000 7fc4c1000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc4bc15d570 0x7fc4bc1494f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:05.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.450+0000 7fc4c1000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc0c4ec0 con 0x7fc4bc06c520 2024-01-15T18:22:05.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.450+0000 7fc4c1000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc149a30 con 0x7fc4bc15d570 2024-01-15T18:22:05.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.450+0000 7fc4c1000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4bc0c48d0 con 0x7fc4bc15d190 2024-01-15T18:22:05.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.450+0000 7fc4b9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4bc06c520 0x7fc4bc0c68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:05.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.450+0000 7fc4b9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4bc06c520 0x7fc4bc0c68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53890/0 (socket says 172.21.15.27:53890) 2024-01-15T18:22:05.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.450+0000 7fc4b9e00640 1 -- 172.21.15.27:0/333736283 learned_addr learned my addr 172.21.15.27:0/333736283 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:05.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.450+0000 7fc4ba800640 1 --2- 172.21.15.27:0/333736283 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc4bc15d570 0x7fc4bc1494f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:05.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.450+0000 7fc4b9400640 1 --2- 172.21.15.27:0/333736283 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc4bc15d190 0x7fc4bc148fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:05.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.450+0000 7fc4b9e00640 1 -- 172.21.15.27:0/333736283 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc4bc15d190 msgr2=0x7fc4bc148fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:05.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.450+0000 7fc4b9e00640 1 --2- 172.21.15.27:0/333736283 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc4bc15d190 0x7fc4bc148fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:05.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.450+0000 7fc4b9e00640 1 -- 172.21.15.27:0/333736283 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc4bc15d570 msgr2=0x7fc4bc1494f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:05.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.450+0000 7fc4b9e00640 1 --2- 172.21.15.27:0/333736283 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc4bc15d570 0x7fc4bc1494f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:05.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.450+0000 7fc4b9e00640 1 -- 172.21.15.27:0/333736283 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ac0094d0 con 0x7fc4bc06c520 2024-01-15T18:22:05.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.450+0000 7fc4b9e00640 1 --2- 172.21.15.27:0/333736283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4bc06c520 0x7fc4bc0c68f0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac0ad040 tx=0x7fc4ac01acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:05.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.454+0000 7fc4b2000640 1 -- 172.21.15.27:0/333736283 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac0b4800 con 0x7fc4bc06c520 2024-01-15T18:22:05.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.454+0000 7fc4b2000640 1 -- 172.21.15.27:0/333736283 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4ac0b4960 con 0x7fc4bc06c520 2024-01-15T18:22:05.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.454+0000 7fc4b2000640 1 -- 172.21.15.27:0/333736283 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac0b4cf0 con 0x7fc4bc06c520 2024-01-15T18:22:05.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.454+0000 7fc4c1000640 1 -- 172.21.15.27:0/333736283 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4bc0c44e0 con 0x7fc4bc06c520 2024-01-15T18:22:05.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.454+0000 7fc4c1000640 1 -- 172.21.15.27:0/333736283 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4bc0c7640 con 0x7fc4bc06c520 2024-01-15T18:22:05.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.454+0000 7fc4c1000640 1 -- 172.21.15.27:0/333736283 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4bc04e850 con 0x7fc4bc06c520 2024-01-15T18:22:05.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.454+0000 7fc4b2000640 1 -- 172.21.15.27:0/333736283 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7fc4ac0b3380 con 0x7fc4bc06c520 2024-01-15T18:22:05.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.458+0000 7fc4b2000640 1 --2- 172.21.15.27:0/333736283 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc494067b80 0x7fc49406a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:05.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.458+0000 7fc4b2000640 1 -- 172.21.15.27:0/333736283 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fc4ac11d870 con 0x7fc4bc06c520 2024-01-15T18:22:05.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.458+0000 7fc4b9400640 1 --2- 172.21.15.27:0/333736283 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc494067b80 0x7fc49406a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:05.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.458+0000 7fc4b9400640 1 --2- 172.21.15.27:0/333736283 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc494067b80 0x7fc49406a040 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc1567f0 tx=0x7fc4a4009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:05.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.462+0000 7fc4b2000640 1 -- 172.21.15.27:0/333736283 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc4ac0ec230 con 0x7fc4bc06c520 2024-01-15T18:22:05.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.646+0000 7fc4c1000640 1 -- 172.21.15.27:0/333736283 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc4bc06b020 con 0x7fc494067b80 2024-01-15T18:22:05.650 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:22:05.650+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:22:05.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:05.654+0000 7fc4b2000640 1 -- 172.21.15.27:0/333736283 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc4bc06b020 con 0x7fc494067b80 2024-01-15T18:22:05.651 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:22:05.651 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:22:05.659 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:07.661 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:22:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.822+0000 7fd197a00640 1 Processor -- start 2024-01-15T18:22:07.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.822+0000 7fd197a00640 1 -- start start 2024-01-15T18:22:07.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd197a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd19015b100 0x7fd190070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:07.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd197a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fd190071420 0x7fd19015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:07.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd197a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd19015a610 con 0x7fd19015ad20 2024-01-15T18:22:07.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd197a00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1900718b0 con 0x7fd190071420 2024-01-15T18:22:07.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd197a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd19015e3c0 con 0x7fd19015b100 2024-01-15T18:22:07.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd195000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd19015ad20 0x7fd19015d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:53514/0 (socket says 172.21.15.27:53514) 2024-01-15T18:22:07.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd195000640 1 -- 172.21.15.27:0/1236806995 learned_addr learned my addr 172.21.15.27:0/1236806995 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:22:07.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd195a00640 1 --2- 172.21.15.27:0/1236806995 >> v2:172.21.15.93:3300/0 conn(0x7fd190071420 0x7fd19015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:07.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd18fe00640 1 --2- 172.21.15.27:0/1236806995 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd19015b100 0x7fd190070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:07.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd18f400640 1 -- 172.21.15.27:0/1236806995 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414794463 0 0) 0x7fd19015a610 con 0x7fd19015ad20 2024-01-15T18:22:07.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd18f400640 1 -- 172.21.15.27:0/1236806995 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd170003530 con 0x7fd19015ad20 2024-01-15T18:22:07.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd18f400640 1 -- 172.21.15.27:0/1236806995 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1717259868 0 0) 0x7fd170003530 con 0x7fd19015ad20 2024-01-15T18:22:07.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd18f400640 1 -- 172.21.15.27:0/1236806995 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd19015b100 msgr2=0x7fd190070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:07.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd18f400640 1 --2- 172.21.15.27:0/1236806995 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd19015b100 0x7fd190070ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:07.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd18f400640 1 -- 172.21.15.27:0/1236806995 >> v2:172.21.15.93:3300/0 conn(0x7fd190071420 msgr2=0x7fd19015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:07.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd18f400640 1 --2- 172.21.15.27:0/1236806995 >> v2:172.21.15.93:3300/0 conn(0x7fd190071420 0x7fd19015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:07.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd18f400640 1 -- 172.21.15.27:0/1236806995 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19015e6a0 con 0x7fd19015ad20 2024-01-15T18:22:07.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd18f400640 1 -- 172.21.15.27:0/1236806995 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fd180001680 con 0x7fd19015ad20 2024-01-15T18:22:07.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd195a00640 1 --2- 172.21.15.27:0/1236806995 >> v2:172.21.15.93:3300/0 conn(0x7fd190071420 0x7fd19015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:22:07.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd18f400640 1 -- 172.21.15.27:0/1236806995 >> v1:172.21.15.27:6789/0 conn(0x7fd19015ad20 legacy=0x7fd19015d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:07.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd18f400640 1 --2- 172.21.15.27:0/1236806995 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1700039f0 0x7fd170003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:07.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd18f400640 1 --2- 172.21.15.27:0/1236806995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1700047d0 0x7fd170004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:07.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd18f400640 1 --2- 172.21.15.27:0/1236806995 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd170005190 0x7fd17000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:07.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd18f400640 1 -- 172.21.15.27:0/1236806995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1900718b0 con 0x7fd1700047d0 2024-01-15T18:22:07.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd18f400640 1 -- 172.21.15.27:0/1236806995 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19015e3c0 con 0x7fd1700039f0 2024-01-15T18:22:07.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd18f400640 1 -- 172.21.15.27:0/1236806995 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd17000bf00 con 0x7fd170005190 2024-01-15T18:22:07.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd18fe00640 1 --2- 172.21.15.27:0/1236806995 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd19015b100 0x7fd190070ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:22:07.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd195000640 1 --2- 172.21.15.27:0/1236806995 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1700039f0 0x7fd170003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:07.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd18fe00640 1 --2- 172.21.15.27:0/1236806995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1700047d0 0x7fd170004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:07.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd195a00640 1 --2- 172.21.15.27:0/1236806995 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd170005190 0x7fd17000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:07.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd195000640 1 -- 172.21.15.27:0/1236806995 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd170005190 msgr2=0x7fd17000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:07.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd195000640 1 --2- 172.21.15.27:0/1236806995 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd170005190 0x7fd17000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:07.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd195000640 1 -- 172.21.15.27:0/1236806995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1700047d0 msgr2=0x7fd170004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:07.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd195000640 1 --2- 172.21.15.27:0/1236806995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1700047d0 0x7fd170004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:07.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd195000640 1 -- 172.21.15.27:0/1236806995 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd170003530 con 0x7fd1700039f0 2024-01-15T18:22:07.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.826+0000 7fd18fe00640 1 --2- 172.21.15.27:0/1236806995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1700047d0 0x7fd170004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:22:07.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.830+0000 7fd195000640 1 --2- 172.21.15.27:0/1236806995 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1700039f0 0x7fd170003e10 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fd18000c210 tx=0x7fd18001f3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f8901c0198119ae server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:07.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.830+0000 7fd18f400640 1 -- 172.21.15.27:0/1236806995 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd180005ae0 con 0x7fd1700039f0 2024-01-15T18:22:07.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.830+0000 7fd18f400640 1 -- 172.21.15.27:0/1236806995 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1800ab070 con 0x7fd1700039f0 2024-01-15T18:22:07.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.830+0000 7fd18f400640 1 -- 172.21.15.27:0/1236806995 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd180005ea0 con 0x7fd1700039f0 2024-01-15T18:22:07.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.830+0000 7fd197a00640 1 -- 172.21.15.27:0/1236806995 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1700039f0 msgr2=0x7fd170003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:07.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.830+0000 7fd197a00640 1 --2- 172.21.15.27:0/1236806995 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1700039f0 0x7fd170003e10 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fd18000c210 tx=0x7fd18001f3b0 comp rx=0 tx=0).stop 2024-01-15T18:22:07.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.830+0000 7fd195a00640 1 -- 172.21.15.27:0/1236806995 reap_dead start 2024-01-15T18:22:07.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.830+0000 7fd197a00640 1 -- 172.21.15.27:0/1236806995 shutdown_connections 2024-01-15T18:22:07.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.830+0000 7fd197a00640 1 -- 172.21.15.27:0/1236806995 >> 172.21.15.27:0/1236806995 conn(0x7fd1900cb700 msgr2=0x7fd1900cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:22:07.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.830+0000 7fd197a00640 1 -- 172.21.15.27:0/1236806995 shutdown_connections 2024-01-15T18:22:07.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.830+0000 7fd197a00640 1 -- 172.21.15.27:0/1236806995 wait complete. 2024-01-15T18:22:07.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.834+0000 7fd197a00640 1 Processor -- start 2024-01-15T18:22:07.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.834+0000 7fd197a00640 1 -- start start 2024-01-15T18:22:07.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.834+0000 7fd197a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1700039f0 0x7fd190141730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:07.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.834+0000 7fd197a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd190156070 0x7fd190158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:07.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.834+0000 7fd197a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1901594c0 0x7fd190141db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:07.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.834+0000 7fd197a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd190157340 con 0x7fd1700039f0 2024-01-15T18:22:07.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.834+0000 7fd197a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1901564d0 con 0x7fd1901594c0 2024-01-15T18:22:07.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.834+0000 7fd197a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd190157760 con 0x7fd190156070 2024-01-15T18:22:07.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.834+0000 7fd195000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1700039f0 0x7fd190141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:07.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.834+0000 7fd195000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1700039f0 0x7fd190141730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53908/0 (socket says 172.21.15.27:53908) 2024-01-15T18:22:07.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.834+0000 7fd195000640 1 -- 172.21.15.27:0/3041725727 learned_addr learned my addr 172.21.15.27:0/3041725727 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:07.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.834+0000 7fd18fe00640 1 --2- 172.21.15.27:0/3041725727 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd190156070 0x7fd190158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:07.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.834+0000 7fd195a00640 1 --2- 172.21.15.27:0/3041725727 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1901594c0 0x7fd190141db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:07.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.834+0000 7fd195000640 1 -- 172.21.15.27:0/3041725727 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd190156070 msgr2=0x7fd190158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:07.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.834+0000 7fd195000640 1 --2- 172.21.15.27:0/3041725727 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd190156070 0x7fd190158f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:07.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.834+0000 7fd195000640 1 -- 172.21.15.27:0/3041725727 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1901594c0 msgr2=0x7fd190141db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:07.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.834+0000 7fd195000640 1 --2- 172.21.15.27:0/3041725727 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd1901594c0 0x7fd190141db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:07.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.834+0000 7fd195000640 1 -- 172.21.15.27:0/3041725727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19015e190 con 0x7fd1700039f0 2024-01-15T18:22:07.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.834+0000 7fd195000640 1 --2- 172.21.15.27:0/3041725727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1700039f0 0x7fd190141730 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fd18001f1e0 tx=0x7fd170005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:07.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.834+0000 7fd18cc00640 1 -- 172.21.15.27:0/3041725727 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1800b7070 con 0x7fd1700039f0 2024-01-15T18:22:07.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.834+0000 7fd18cc00640 1 -- 172.21.15.27:0/3041725727 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1800048b0 con 0x7fd1700039f0 2024-01-15T18:22:07.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.834+0000 7fd197a00640 1 -- 172.21.15.27:0/3041725727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1901566c0 con 0x7fd1700039f0 2024-01-15T18:22:07.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.834+0000 7fd18cc00640 1 -- 172.21.15.27:0/3041725727 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd180004c20 con 0x7fd1700039f0 2024-01-15T18:22:07.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.834+0000 7fd197a00640 1 -- 172.21.15.27:0/3041725727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd190156c80 con 0x7fd1700039f0 2024-01-15T18:22:07.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.838+0000 7fd197a00640 1 -- 172.21.15.27:0/3041725727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd19004e850 con 0x7fd1700039f0 2024-01-15T18:22:07.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.838+0000 7fd18cc00640 1 -- 172.21.15.27:0/3041725727 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7fd1800ad020 con 0x7fd1700039f0 2024-01-15T18:22:07.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.838+0000 7fd18cc00640 1 --2- 172.21.15.27:0/3041725727 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd168067bb0 0x7fd16806a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:07.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.838+0000 7fd18cc00640 1 -- 172.21.15.27:0/3041725727 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fd18011f7b0 con 0x7fd1700039f0 2024-01-15T18:22:07.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.838+0000 7fd18fe00640 1 --2- 172.21.15.27:0/3041725727 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd168067bb0 0x7fd16806a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:07.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.838+0000 7fd18fe00640 1 --2- 172.21.15.27:0/3041725727 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd168067bb0 0x7fd16806a070 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fd190159eb0 tx=0x7fd178009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:07.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:07.842+0000 7fd18cc00640 1 -- 172.21.15.27:0/3041725727 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd1800ee4f0 con 0x7fd1700039f0 2024-01-15T18:22:08.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:08.030+0000 7fd197a00640 1 -- 172.21.15.27:0/3041725727 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd190071a10 con 0x7fd168067bb0 2024-01-15T18:22:08.033 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:22:08.034+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:22:08.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:08.038+0000 7fd18cc00640 1 -- 172.21.15.27:0/3041725727 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd190071a10 con 0x7fd168067bb0 2024-01-15T18:22:08.034 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:22:08.035 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:22:08.042 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:10.044 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:22:10.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.206+0000 7fea15600640 1 Processor -- start 2024-01-15T18:22:10.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.206+0000 7fea15600640 1 -- start start 2024-01-15T18:22:10.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.210+0000 7fea15600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fea1006c430 0x7fea10070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:10.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.210+0000 7fea15600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fea100ce760 0x7fea100b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:10.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.210+0000 7fea15600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea100ccfb0 con 0x7fea100b1db0 2024-01-15T18:22:10.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.210+0000 7fea15600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea100cd290 con 0x7fea1006c430 2024-01-15T18:22:10.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.210+0000 7fea15600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea100cd570 con 0x7fea100ce760 2024-01-15T18:22:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.210+0000 7fea0ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fea1006c430 0x7fea10070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.210+0000 7fea0ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fea1006c430 0x7fea10070ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53924/0 (socket says 172.21.15.27:53924) 2024-01-15T18:22:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.210+0000 7fea0ea00640 1 -- 172.21.15.27:0/2242529501 learned_addr learned my addr 172.21.15.27:0/2242529501 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.210+0000 7fea0f400640 1 --2- 172.21.15.27:0/2242529501 >> v2:172.21.15.93:3300/0 conn(0x7fea100ce760 0x7fea100b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.210+0000 7fea0ea00640 1 -- 172.21.15.27:0/2242529501 >> v2:172.21.15.93:3300/0 conn(0x7fea100ce760 msgr2=0x7fea100b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.210+0000 7fea0ea00640 1 --2- 172.21.15.27:0/2242529501 >> v2:172.21.15.93:3300/0 conn(0x7fea100ce760 0x7fea100b2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.210+0000 7fea0ea00640 1 -- 172.21.15.27:0/2242529501 >> v1:172.21.15.93:6789/0 conn(0x7fea100b1db0 legacy=0x7fea100ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.210+0000 7fea0ea00640 1 -- 172.21.15.27:0/2242529501 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea100cd850 con 0x7fea1006c430 2024-01-15T18:22:10.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.210+0000 7fea0ea00640 1 --2- 172.21.15.27:0/2242529501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea1006c430 0x7fea10070ba0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fea00009870 tx=0x7fea0001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfd3b442087398ce server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:10.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.214+0000 7fea0d600640 1 -- 172.21.15.27:0/2242529501 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea000ac640 con 0x7fea1006c430 2024-01-15T18:22:10.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.214+0000 7fea0d600640 1 -- 172.21.15.27:0/2242529501 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea000ac7a0 con 0x7fea1006c430 2024-01-15T18:22:10.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.214+0000 7fea0d600640 1 -- 172.21.15.27:0/2242529501 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea000acb30 con 0x7fea1006c430 2024-01-15T18:22:10.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.214+0000 7fea15600640 1 -- 172.21.15.27:0/2242529501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea1006c430 msgr2=0x7fea10070ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:10.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.214+0000 7fea15600640 1 --2- 172.21.15.27:0/2242529501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea1006c430 0x7fea10070ba0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fea00009870 tx=0x7fea0001c440 comp rx=0 tx=0).stop 2024-01-15T18:22:10.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.214+0000 7fea15600640 1 -- 172.21.15.27:0/2242529501 shutdown_connections 2024-01-15T18:22:10.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.214+0000 7fea15600640 1 --2- 172.21.15.27:0/2242529501 >> v2:172.21.15.93:3300/0 conn(0x7fea100ce760 0x7fea100b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:10.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.214+0000 7fea15600640 1 --2- 172.21.15.27:0/2242529501 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea1006c430 0x7fea10070ba0 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:10.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.214+0000 7fea15600640 1 -- 172.21.15.27:0/2242529501 >> 172.21.15.27:0/2242529501 conn(0x7fea100cb700 msgr2=0x7fea100cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:22:10.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.214+0000 7fea15600640 1 -- 172.21.15.27:0/2242529501 shutdown_connections 2024-01-15T18:22:10.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.214+0000 7fea15600640 1 -- 172.21.15.27:0/2242529501 wait complete. 2024-01-15T18:22:10.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.214+0000 7fea15600640 1 Processor -- start 2024-01-15T18:22:10.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.214+0000 7fea15600640 1 -- start start 2024-01-15T18:22:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.214+0000 7fea15600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fea1006c430 0x7fea10146a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.214+0000 7fea15600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea100b1db0 0x7fea1015a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.214+0000 7fea15600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fea100ce760 0x7fea101682d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.214+0000 7fea15600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea10159b20 con 0x7fea100b1db0 2024-01-15T18:22:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.214+0000 7fea15600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea10147200 con 0x7fea1006c430 2024-01-15T18:22:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.214+0000 7fea15600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea10146fc0 con 0x7fea100ce760 2024-01-15T18:22:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.218+0000 7fea0e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea100b1db0 0x7fea1015a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.218+0000 7fea0e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea100b1db0 0x7fea1015a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53940/0 (socket says 172.21.15.27:53940) 2024-01-15T18:22:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.218+0000 7fea0e000640 1 -- 172.21.15.27:0/3445248927 learned_addr learned my addr 172.21.15.27:0/3445248927 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.218+0000 7fea0ea00640 1 --2- 172.21.15.27:0/3445248927 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fea1006c430 0x7fea10146a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:10.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.218+0000 7fea0f400640 1 --2- 172.21.15.27:0/3445248927 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fea100ce760 0x7fea101682d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:10.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.218+0000 7fea0e000640 1 -- 172.21.15.27:0/3445248927 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fea100ce760 msgr2=0x7fea101682d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:10.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.218+0000 7fea0e000640 1 --2- 172.21.15.27:0/3445248927 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fea100ce760 0x7fea101682d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:10.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.218+0000 7fea0e000640 1 -- 172.21.15.27:0/3445248927 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fea1006c430 msgr2=0x7fea10146a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:10.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.218+0000 7fea0e000640 1 --2- 172.21.15.27:0/3445248927 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fea1006c430 0x7fea10146a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:10.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.218+0000 7fea0e000640 1 -- 172.21.15.27:0/3445248927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea000094d0 con 0x7fea100b1db0 2024-01-15T18:22:10.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.218+0000 7fea0e000640 1 --2- 172.21.15.27:0/3445248927 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea100b1db0 0x7fea1015a530 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fe9f800c420 tx=0x7fe9f800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:10.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.218+0000 7fea06a00640 1 -- 172.21.15.27:0/3445248927 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f80052d0 con 0x7fea100b1db0 2024-01-15T18:22:10.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.218+0000 7fea06a00640 1 -- 172.21.15.27:0/3445248927 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9f8005430 con 0x7fea100b1db0 2024-01-15T18:22:10.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.218+0000 7fea06a00640 1 -- 172.21.15.27:0/3445248927 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8007260 con 0x7fea100b1db0 2024-01-15T18:22:10.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.218+0000 7fea15600640 1 -- 172.21.15.27:0/3445248927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea100712f0 con 0x7fea100b1db0 2024-01-15T18:22:10.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.218+0000 7fea15600640 1 -- 172.21.15.27:0/3445248927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea1015a010 con 0x7fea100b1db0 2024-01-15T18:22:10.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.218+0000 7fea15600640 1 -- 172.21.15.27:0/3445248927 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea1004e850 con 0x7fea100b1db0 2024-01-15T18:22:10.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.218+0000 7fea06a00640 1 -- 172.21.15.27:0/3445248927 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7fe9f80073c0 con 0x7fea100b1db0 2024-01-15T18:22:10.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.222+0000 7fea06a00640 1 --2- 172.21.15.27:0/3445248927 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe9e8067b80 0x7fe9e806a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:10.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.222+0000 7fea06a00640 1 -- 172.21.15.27:0/3445248927 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fe9f80827e0 con 0x7fea100b1db0 2024-01-15T18:22:10.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.222+0000 7fea0ea00640 1 --2- 172.21.15.27:0/3445248927 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe9e8067b80 0x7fe9e806a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:10.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.222+0000 7fea0ea00640 1 --2- 172.21.15.27:0/3445248927 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe9e8067b80 0x7fe9e806a040 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fea000ae000 tx=0x7fea000a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:10.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.226+0000 7fea06a00640 1 -- 172.21.15.27:0/3445248927 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe9f8051120 con 0x7fea100b1db0 2024-01-15T18:22:10.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.414+0000 7fea15600640 1 -- 172.21.15.27:0/3445248927 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fea1015ced0 con 0x7fe9e8067b80 2024-01-15T18:22:10.419 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:22:10.422+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:22:10.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:10.426+0000 7fea06a00640 1 -- 172.21.15.27:0/3445248927 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fea1015ced0 con 0x7fe9e8067b80 2024-01-15T18:22:10.420 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:22:10.420 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:22:10.429 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:12.430 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:22:12.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.594+0000 7f841cc00640 1 Processor -- start 2024-01-15T18:22:12.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.594+0000 7f841cc00640 1 -- start start 2024-01-15T18:22:12.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.594+0000 7f841cc00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f84180cf460 0x7f84180cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:12.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.594+0000 7f841cc00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f841815b4b0 0x7f84180cba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:12.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.594+0000 7f841cc00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84180cfd80 con 0x7f84180cec60 2024-01-15T18:22:12.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.594+0000 7f841cc00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84180d19a0 con 0x7f841815b4b0 2024-01-15T18:22:12.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.594+0000 7f841cc00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84180d1c80 con 0x7f84180cf460 2024-01-15T18:22:12.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.594+0000 7f8415400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f841815b4b0 0x7f84180cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:12.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.594+0000 7f8415400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f841815b4b0 0x7f84180cba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53954/0 (socket says 172.21.15.27:53954) 2024-01-15T18:22:12.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.594+0000 7f8415400640 1 -- 172.21.15.27:0/1911161126 learned_addr learned my addr 172.21.15.27:0/1911161126 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:12.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.594+0000 7f8415e00640 1 --2- 172.21.15.27:0/1911161126 >> v2:172.21.15.93:3300/0 conn(0x7f84180cf460 0x7f84180cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:12.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.594+0000 7f8415400640 1 -- 172.21.15.27:0/1911161126 >> v2:172.21.15.93:3300/0 conn(0x7f84180cf460 msgr2=0x7f84180cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:12.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.594+0000 7f8415400640 1 --2- 172.21.15.27:0/1911161126 >> v2:172.21.15.93:3300/0 conn(0x7f84180cf460 0x7f84180cf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:12.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.594+0000 7f8415400640 1 -- 172.21.15.27:0/1911161126 >> v1:172.21.15.93:6789/0 conn(0x7f84180cec60 legacy=0x7f84180cbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:12.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.594+0000 7f8415400640 1 -- 172.21.15.27:0/1911161126 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84180b07c0 con 0x7f841815b4b0 2024-01-15T18:22:12.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.598+0000 7f8415400640 1 --2- 172.21.15.27:0/1911161126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f841815b4b0 0x7f84180cba20 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f84180681f0 tx=0x7f840001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c58406b3a79ae3e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:12.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.598+0000 7f8414a00640 1 -- 172.21.15.27:0/1911161126 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84000aa660 con 0x7f841815b4b0 2024-01-15T18:22:12.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.598+0000 7f8414a00640 1 -- 172.21.15.27:0/1911161126 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84000aa7c0 con 0x7f841815b4b0 2024-01-15T18:22:12.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.598+0000 7f8414a00640 1 -- 172.21.15.27:0/1911161126 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84000aab50 con 0x7f841815b4b0 2024-01-15T18:22:12.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.598+0000 7f841cc00640 1 -- 172.21.15.27:0/1911161126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f841815b4b0 msgr2=0x7f84180cba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:12.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.598+0000 7f841cc00640 1 --2- 172.21.15.27:0/1911161126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f841815b4b0 0x7f84180cba20 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f84180681f0 tx=0x7f840001c500 comp rx=0 tx=0).stop 2024-01-15T18:22:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.598+0000 7f841cc00640 1 -- 172.21.15.27:0/1911161126 shutdown_connections 2024-01-15T18:22:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.598+0000 7f841cc00640 1 --2- 172.21.15.27:0/1911161126 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f841815b4b0 0x7f84180cba20 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.598+0000 7f841cc00640 1 --2- 172.21.15.27:0/1911161126 >> v2:172.21.15.93:3300/0 conn(0x7f84180cf460 0x7f84180cf840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.598+0000 7f841cc00640 1 -- 172.21.15.27:0/1911161126 >> 172.21.15.27:0/1911161126 conn(0x7f84180d0e80 msgr2=0x7f84180d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:22:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.598+0000 7f841cc00640 1 -- 172.21.15.27:0/1911161126 shutdown_connections 2024-01-15T18:22:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.598+0000 7f841cc00640 1 -- 172.21.15.27:0/1911161126 wait complete. 2024-01-15T18:22:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.602+0000 7f841cc00640 1 Processor -- start 2024-01-15T18:22:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.602+0000 7f841cc00640 1 -- start start 2024-01-15T18:22:12.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.602+0000 7f841cc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f84180cec60 0x7f8418151990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:12.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.602+0000 7f841cc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f84180cf460 0x7f841806c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:12.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.602+0000 7f841cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f841815b4b0 0x7f84181533f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:12.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.602+0000 7f841cc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84181641b0 con 0x7f841815b4b0 2024-01-15T18:22:12.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.602+0000 7f841cc00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8418152810 con 0x7f84180cec60 2024-01-15T18:22:12.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.602+0000 7f841cc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8418153930 con 0x7f84180cf460 2024-01-15T18:22:12.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.602+0000 7f8416800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f841815b4b0 0x7f84181533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:12.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.602+0000 7f8415e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f84180cec60 0x7f8418151990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:12.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.602+0000 7f8415400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f84180cf460 0x7f841806c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:12.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.602+0000 7f8416800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f841815b4b0 0x7f84181533f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53956/0 (socket says 172.21.15.27:53956) 2024-01-15T18:22:12.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.602+0000 7f8416800640 1 -- 172.21.15.27:0/3477162570 learned_addr learned my addr 172.21.15.27:0/3477162570 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:12.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.602+0000 7f8416800640 1 -- 172.21.15.27:0/3477162570 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f84180cf460 msgr2=0x7f841806c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:12.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.602+0000 7f8416800640 1 --2- 172.21.15.27:0/3477162570 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f84180cf460 0x7f841806c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:12.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.602+0000 7f8416800640 1 -- 172.21.15.27:0/3477162570 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f84180cec60 msgr2=0x7f8418151990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:12.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.602+0000 7f8416800640 1 --2- 172.21.15.27:0/3477162570 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f84180cec60 0x7f8418151990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:12.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.602+0000 7f8416800640 1 -- 172.21.15.27:0/3477162570 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84000094d0 con 0x7f841815b4b0 2024-01-15T18:22:12.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.602+0000 7f8415e00640 1 --2- 172.21.15.27:0/3477162570 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f84180cec60 0x7f8418151990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:22:12.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.602+0000 7f8416800640 1 --2- 172.21.15.27:0/3477162570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f841815b4b0 0x7f84181533f0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f841000c760 tx=0x7f841000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:12.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.602+0000 7f840e000640 1 -- 172.21.15.27:0/3477162570 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8410003b80 con 0x7f841815b4b0 2024-01-15T18:22:12.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.602+0000 7f840e000640 1 -- 172.21.15.27:0/3477162570 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8410003ce0 con 0x7f841815b4b0 2024-01-15T18:22:12.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.602+0000 7f840e000640 1 -- 172.21.15.27:0/3477162570 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841000aa30 con 0x7f841815b4b0 2024-01-15T18:22:12.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.602+0000 7f841cc00640 1 -- 172.21.15.27:0/3477162570 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84180c7660 con 0x7f841815b4b0 2024-01-15T18:22:12.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.602+0000 7f841cc00640 1 -- 172.21.15.27:0/3477162570 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8418152250 con 0x7f841815b4b0 2024-01-15T18:22:12.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.606+0000 7f841cc00640 1 -- 172.21.15.27:0/3477162570 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f841804e850 con 0x7f841815b4b0 2024-01-15T18:22:12.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.606+0000 7f840e000640 1 -- 172.21.15.27:0/3477162570 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7f841000ab90 con 0x7f841815b4b0 2024-01-15T18:22:12.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.606+0000 7f840e000640 1 --2- 172.21.15.27:0/3477162570 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f83f0067b80 0x7f83f006a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:12.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.606+0000 7f840e000640 1 -- 172.21.15.27:0/3477162570 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f8410083050 con 0x7f841815b4b0 2024-01-15T18:22:12.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.606+0000 7f8415e00640 1 --2- 172.21.15.27:0/3477162570 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f83f0067b80 0x7f83f006a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:12.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.606+0000 7f8415e00640 1 --2- 172.21.15.27:0/3477162570 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f83f0067b80 0x7f83f006a040 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f8408009f50 tx=0x7f8408009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:12.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.610+0000 7f840e000640 1 -- 172.21.15.27:0/3477162570 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8410051a10 con 0x7f841815b4b0 2024-01-15T18:22:12.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.802+0000 7f841cc00640 1 -- 172.21.15.27:0/3477162570 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8418168630 con 0x7f83f0067b80 2024-01-15T18:22:12.804 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:22:12.806+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:22:12.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:12.810+0000 7f840e000640 1 -- 172.21.15.27:0/3477162570 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8408011e70 con 0x7f83f0067b80 2024-01-15T18:22:12.805 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:22:12.805 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:22:12.813 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:14.815 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:22:14.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.978+0000 7f3b2b000640 1 Processor -- start 2024-01-15T18:22:14.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.978+0000 7f3b2b000640 1 -- start start 2024-01-15T18:22:14.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.978+0000 7f3b2b000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3b2415f550 0x7f3b240cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:14.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.978+0000 7f3b2b000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3b240cf110 0x7f3b240701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:14.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.978+0000 7f3b2b000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b240b1cb0 con 0x7f3b24158900 2024-01-15T18:22:14.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.978+0000 7f3b2b000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b2415fa00 con 0x7f3b240cf110 2024-01-15T18:22:14.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.978+0000 7f3b2b000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b240cf540 con 0x7f3b2415f550 2024-01-15T18:22:14.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.978+0000 7f3b23e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3b24158900 0x7f3b2415bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43354/0 (socket says 172.21.15.27:43354) 2024-01-15T18:22:14.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.978+0000 7f3b23e00640 1 -- 172.21.15.27:0/1426558541 learned_addr learned my addr 172.21.15.27:0/1426558541 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:22:14.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.978+0000 7f3b29000640 1 --2- 172.21.15.27:0/1426558541 >> v2:172.21.15.27:3300/0 conn(0x7f3b240cf110 0x7f3b240701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:14.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.978+0000 7f3b23400640 1 --2- 172.21.15.27:0/1426558541 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3b2415f550 0x7f3b240cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:14.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.978+0000 7f3b22a00640 1 -- 172.21.15.27:0/1426558541 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683991019 0 0) 0x7f3b240b1cb0 con 0x7f3b24158900 2024-01-15T18:22:14.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.978+0000 7f3b22a00640 1 -- 172.21.15.27:0/1426558541 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b04003530 con 0x7f3b24158900 2024-01-15T18:22:14.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.978+0000 7f3b29000640 1 -- 172.21.15.27:0/1426558541 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3b2415f550 msgr2=0x7f3b240cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:14.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.978+0000 7f3b29000640 1 --2- 172.21.15.27:0/1426558541 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3b2415f550 0x7f3b240cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:14.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.978+0000 7f3b29000640 1 -- 172.21.15.27:0/1426558541 >> v1:172.21.15.27:6789/0 conn(0x7f3b24158900 legacy=0x7f3b2415bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:14.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.978+0000 7f3b29000640 1 -- 172.21.15.27:0/1426558541 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b240b1f00 con 0x7f3b240cf110 2024-01-15T18:22:14.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.982+0000 7f3b29000640 1 --2- 172.21.15.27:0/1426558541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b240cf110 0x7f3b240701e0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c009d90 tx=0x7f3b1c01c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eb33627948480b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:14.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.982+0000 7f3b22a00640 1 -- 172.21.15.27:0/1426558541 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b24070f30 con 0x7f3b240cf110 2024-01-15T18:22:14.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.982+0000 7f3b22a00640 1 -- 172.21.15.27:0/1426558541 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1c0a59e0 con 0x7f3b240cf110 2024-01-15T18:22:14.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.982+0000 7f3b22a00640 1 -- 172.21.15.27:0/1426558541 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c0a5d70 con 0x7f3b240cf110 2024-01-15T18:22:14.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.982+0000 7f3b2b000640 1 -- 172.21.15.27:0/1426558541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b240cf110 msgr2=0x7f3b240701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:14.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.982+0000 7f3b2b000640 1 --2- 172.21.15.27:0/1426558541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b240cf110 0x7f3b240701e0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c009d90 tx=0x7f3b1c01c920 comp rx=0 tx=0).stop 2024-01-15T18:22:14.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.982+0000 7f3b2b000640 1 -- 172.21.15.27:0/1426558541 shutdown_connections 2024-01-15T18:22:14.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.982+0000 7f3b2b000640 1 --2- 172.21.15.27:0/1426558541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b240cf110 0x7f3b240701e0 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:14.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.982+0000 7f3b2b000640 1 --2- 172.21.15.27:0/1426558541 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3b2415f550 0x7f3b240cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:14.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.982+0000 7f3b2b000640 1 -- 172.21.15.27:0/1426558541 >> 172.21.15.27:0/1426558541 conn(0x7f3b240cb700 msgr2=0x7f3b240cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:22:14.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.982+0000 7f3b2b000640 1 -- 172.21.15.27:0/1426558541 shutdown_connections 2024-01-15T18:22:14.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.982+0000 7f3b2b000640 1 -- 172.21.15.27:0/1426558541 wait complete. 2024-01-15T18:22:14.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.986+0000 7f3b2b000640 1 Processor -- start 2024-01-15T18:22:14.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.986+0000 7f3b2b000640 1 -- start start 2024-01-15T18:22:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.986+0000 7f3b2b000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b240cf110 0x7f3b2413eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.986+0000 7f3b2b000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3b24158900 0x7f3b2415df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.986+0000 7f3b2b000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3b2415f550 0x7f3b2413c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.986+0000 7f3b2b000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2413cc90 con 0x7f3b240cf110 2024-01-15T18:22:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.986+0000 7f3b2b000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2415c8b0 con 0x7f3b2415f550 2024-01-15T18:22:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.986+0000 7f3b2b000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2413fa90 con 0x7f3b24158900 2024-01-15T18:22:14.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.986+0000 7f3b23e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b240cf110 0x7f3b2413eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:14.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.986+0000 7f3b23e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b240cf110 0x7f3b2413eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55482/0 (socket says 172.21.15.27:55482) 2024-01-15T18:22:14.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.986+0000 7f3b23e00640 1 -- 172.21.15.27:0/1054781425 learned_addr learned my addr 172.21.15.27:0/1054781425 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:14.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.986+0000 7f3b29000640 1 --2- 172.21.15.27:0/1054781425 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3b2415f550 0x7f3b2413c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:14.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.986+0000 7f3b23400640 1 --2- 172.21.15.27:0/1054781425 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3b24158900 0x7f3b2415df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:14.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.986+0000 7f3b23e00640 1 -- 172.21.15.27:0/1054781425 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3b24158900 msgr2=0x7f3b2415df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:14.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.986+0000 7f3b23e00640 1 --2- 172.21.15.27:0/1054781425 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3b24158900 0x7f3b2415df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:14.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.986+0000 7f3b23e00640 1 -- 172.21.15.27:0/1054781425 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3b2415f550 msgr2=0x7f3b2413c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:14.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.986+0000 7f3b23e00640 1 --2- 172.21.15.27:0/1054781425 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3b2415f550 0x7f3b2413c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:14.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.986+0000 7f3b23e00640 1 -- 172.21.15.27:0/1054781425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1c009a70 con 0x7f3b240cf110 2024-01-15T18:22:14.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.986+0000 7f3b23e00640 1 --2- 172.21.15.27:0/1054781425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b240cf110 0x7f3b2413eb40 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f3b1400c4d0 tx=0x7f3b1400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:14.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.986+0000 7f3b1be00640 1 -- 172.21.15.27:0/1054781425 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b14005cf0 con 0x7f3b240cf110 2024-01-15T18:22:14.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.986+0000 7f3b1be00640 1 -- 172.21.15.27:0/1054781425 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b14005e50 con 0x7f3b240cf110 2024-01-15T18:22:14.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.986+0000 7f3b1be00640 1 -- 172.21.15.27:0/1054781425 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b14007270 con 0x7f3b240cf110 2024-01-15T18:22:14.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.986+0000 7f3b2b000640 1 -- 172.21.15.27:0/1054781425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b2415e440 con 0x7f3b240cf110 2024-01-15T18:22:14.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.986+0000 7f3b2b000640 1 -- 172.21.15.27:0/1054781425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2413f4d0 con 0x7f3b240cf110 2024-01-15T18:22:14.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.990+0000 7f3b2b000640 1 -- 172.21.15.27:0/1054781425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b2404e850 con 0x7f3b240cf110 2024-01-15T18:22:14.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.990+0000 7f3b1be00640 1 -- 172.21.15.27:0/1054781425 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7f3b140073d0 con 0x7f3b240cf110 2024-01-15T18:22:14.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.990+0000 7f3b1be00640 1 --2- 172.21.15.27:0/1054781425 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3afc067b80 0x7f3afc06a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:14.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.990+0000 7f3b1be00640 1 -- 172.21.15.27:0/1054781425 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f3b140828b0 con 0x7f3b240cf110 2024-01-15T18:22:14.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.990+0000 7f3b23400640 1 --2- 172.21.15.27:0/1054781425 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3afc067b80 0x7f3afc06a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:14.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.994+0000 7f3b23400640 1 --2- 172.21.15.27:0/1054781425 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3afc067b80 0x7f3afc06a040 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f3b240b47b0 tx=0x7f3b0c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:14.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:14.994+0000 7f3b1be00640 1 -- 172.21.15.27:0/1054781425 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3b14051270 con 0x7f3b240cf110 2024-01-15T18:22:15.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:15.186+0000 7f3b2b000640 1 -- 172.21.15.27:0/1054781425 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3b2415f030 con 0x7f3afc067b80 2024-01-15T18:22:15.188 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:22:15.189+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:22:15.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:15.190+0000 7f3b1be00640 1 -- 172.21.15.27:0/1054781425 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3b0c009820 con 0x7f3afc067b80 2024-01-15T18:22:15.188 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:22:15.189 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:22:15.197 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:17.198 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:22:17.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.362+0000 7fae7d200640 1 Processor -- start 2024-01-15T18:22:17.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.362+0000 7fae7d200640 1 -- start start 2024-01-15T18:22:17.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.362+0000 7fae7d200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fae7815f550 0x7fae780cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:17.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.362+0000 7fae7d200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fae780cf110 0x7fae780701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:17.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.362+0000 7fae7d200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae780b1cb0 con 0x7fae78158900 2024-01-15T18:22:17.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.362+0000 7fae7d200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae780cf580 con 0x7fae7815f550 2024-01-15T18:22:17.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.362+0000 7fae7d200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae7815fa00 con 0x7fae780cf110 2024-01-15T18:22:17.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.362+0000 7fae75400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fae7815f550 0x7fae780cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:17.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.362+0000 7fae75e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fae78158900 0x7fae7815bf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43368/0 (socket says 172.21.15.27:43368) 2024-01-15T18:22:17.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.362+0000 7fae75e00640 1 -- 172.21.15.27:0/3879337366 learned_addr learned my addr 172.21.15.27:0/3879337366 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:22:17.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.362+0000 7fae76800640 1 --2- 172.21.15.27:0/3879337366 >> v2:172.21.15.93:3300/0 conn(0x7fae780cf110 0x7fae780701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:17.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.362+0000 7fae75400640 1 -- 172.21.15.27:0/3879337366 >> v2:172.21.15.93:3300/0 conn(0x7fae780cf110 msgr2=0x7fae780701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:17.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.362+0000 7fae75400640 1 --2- 172.21.15.27:0/3879337366 >> v2:172.21.15.93:3300/0 conn(0x7fae780cf110 0x7fae780701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:17.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.362+0000 7fae75400640 1 -- 172.21.15.27:0/3879337366 >> v1:172.21.15.27:6789/0 conn(0x7fae78158900 legacy=0x7fae7815bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:17.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.362+0000 7fae75400640 1 -- 172.21.15.27:0/3879337366 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae780b1f00 con 0x7fae7815f550 2024-01-15T18:22:17.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.366+0000 7fae75400640 1 --2- 172.21.15.27:0/3879337366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae7815f550 0x7fae780cebd0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fae600097c0 tx=0x7fae6001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2267c25c0a73e92f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:17.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.366+0000 7fae74a00640 1 -- 172.21.15.27:0/3879337366 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae78070f30 con 0x7fae7815f550 2024-01-15T18:22:17.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.366+0000 7fae74a00640 1 -- 172.21.15.27:0/3879337366 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae600ab610 con 0x7fae7815f550 2024-01-15T18:22:17.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.366+0000 7fae74a00640 1 -- 172.21.15.27:0/3879337366 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae600ab9a0 con 0x7fae7815f550 2024-01-15T18:22:17.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.366+0000 7fae7d200640 1 -- 172.21.15.27:0/3879337366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae7815f550 msgr2=0x7fae780cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:17.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.366+0000 7fae7d200640 1 --2- 172.21.15.27:0/3879337366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae7815f550 0x7fae780cebd0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fae600097c0 tx=0x7fae6001c410 comp rx=0 tx=0).stop 2024-01-15T18:22:17.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.366+0000 7fae7d200640 1 -- 172.21.15.27:0/3879337366 shutdown_connections 2024-01-15T18:22:17.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.366+0000 7fae7d200640 1 --2- 172.21.15.27:0/3879337366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae7815f550 0x7fae780cebd0 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:17.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.366+0000 7fae7d200640 1 --2- 172.21.15.27:0/3879337366 >> v2:172.21.15.93:3300/0 conn(0x7fae780cf110 0x7fae780701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:17.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.366+0000 7fae7d200640 1 -- 172.21.15.27:0/3879337366 >> 172.21.15.27:0/3879337366 conn(0x7fae780cb700 msgr2=0x7fae780cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:22:17.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.366+0000 7fae7d200640 1 -- 172.21.15.27:0/3879337366 shutdown_connections 2024-01-15T18:22:17.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.366+0000 7fae7d200640 1 -- 172.21.15.27:0/3879337366 wait complete. 2024-01-15T18:22:17.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.370+0000 7fae7d200640 1 Processor -- start 2024-01-15T18:22:17.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.370+0000 7fae7d200640 1 -- start start 2024-01-15T18:22:17.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.370+0000 7fae7d200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fae780cf110 0x7fae78141810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:17.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.370+0000 7fae7d200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae78158900 0x7fae7806c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:17.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.370+0000 7fae7d200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fae7815f550 0x7fae7806c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:17.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.370+0000 7fae7d200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae78150a70 con 0x7fae78158900 2024-01-15T18:22:17.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.370+0000 7fae7d200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7814fdf0 con 0x7fae7815f550 2024-01-15T18:22:17.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.370+0000 7fae7d200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae781500d0 con 0x7fae780cf110 2024-01-15T18:22:17.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.370+0000 7fae75400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae78158900 0x7fae7806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:17.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.370+0000 7fae75400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae78158900 0x7fae7806c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55494/0 (socket says 172.21.15.27:55494) 2024-01-15T18:22:17.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.370+0000 7fae75400640 1 -- 172.21.15.27:0/584969546 learned_addr learned my addr 172.21.15.27:0/584969546 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:17.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.370+0000 7fae75e00640 1 --2- 172.21.15.27:0/584969546 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fae780cf110 0x7fae78141810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:17.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.370+0000 7fae76800640 1 --2- 172.21.15.27:0/584969546 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fae7815f550 0x7fae7806c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:17.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.370+0000 7fae75400640 1 -- 172.21.15.27:0/584969546 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fae780cf110 msgr2=0x7fae78141810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:17.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.370+0000 7fae75400640 1 --2- 172.21.15.27:0/584969546 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fae780cf110 0x7fae78141810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:17.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.370+0000 7fae75400640 1 -- 172.21.15.27:0/584969546 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fae7815f550 msgr2=0x7fae7806c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:17.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.370+0000 7fae75400640 1 --2- 172.21.15.27:0/584969546 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fae7815f550 0x7fae7806c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:17.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.370+0000 7fae75400640 1 -- 172.21.15.27:0/584969546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae60009420 con 0x7fae78158900 2024-01-15T18:22:17.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.370+0000 7fae76800640 1 --2- 172.21.15.27:0/584969546 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fae7815f550 0x7fae7806c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:22:17.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.370+0000 7fae75400640 1 --2- 172.21.15.27:0/584969546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae78158900 0x7fae7806c430 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fae600ad040 tx=0x7fae6001ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:17.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.370+0000 7fae6e000640 1 -- 172.21.15.27:0/584969546 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae600b5610 con 0x7fae78158900 2024-01-15T18:22:17.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.370+0000 7fae6e000640 1 -- 172.21.15.27:0/584969546 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae600b57b0 con 0x7fae78158900 2024-01-15T18:22:17.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.370+0000 7fae7d200640 1 -- 172.21.15.27:0/584969546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae781503b0 con 0x7fae78158900 2024-01-15T18:22:17.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.370+0000 7fae6e000640 1 -- 172.21.15.27:0/584969546 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae600b5b60 con 0x7fae78158900 2024-01-15T18:22:17.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.370+0000 7fae7d200640 1 -- 172.21.15.27:0/584969546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae7815e180 con 0x7fae78158900 2024-01-15T18:22:17.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.374+0000 7fae7d200640 1 -- 172.21.15.27:0/584969546 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae7804e850 con 0x7fae78158900 2024-01-15T18:22:17.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.374+0000 7fae6e000640 1 -- 172.21.15.27:0/584969546 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7fae600b5cc0 con 0x7fae78158900 2024-01-15T18:22:17.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.374+0000 7fae6e000640 1 --2- 172.21.15.27:0/584969546 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fae50067b80 0x7fae5006a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:17.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.374+0000 7fae6e000640 1 -- 172.21.15.27:0/584969546 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fae6011dea0 con 0x7fae78158900 2024-01-15T18:22:17.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.374+0000 7fae75e00640 1 --2- 172.21.15.27:0/584969546 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fae50067b80 0x7fae5006a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:17.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.374+0000 7fae75e00640 1 --2- 172.21.15.27:0/584969546 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fae50067b80 0x7fae5006a040 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fae68005890 tx=0x7fae68009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:17.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.378+0000 7fae6e000640 1 -- 172.21.15.27:0/584969546 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fae600ec7e0 con 0x7fae78158900 2024-01-15T18:22:17.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.566+0000 7fae7d200640 1 -- 172.21.15.27:0/584969546 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fae7806db80 con 0x7fae50067b80 2024-01-15T18:22:17.569 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:22:17.569+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:22:17.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:17.574+0000 7fae6e000640 1 -- 172.21.15.27:0/584969546 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fae7806db80 con 0x7fae50067b80 2024-01-15T18:22:17.570 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:22:17.570 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:22:17.579 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:19.580 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:22:19.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.742+0000 7fafea800640 1 Processor -- start 2024-01-15T18:22:19.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.742+0000 7fafea800640 1 -- start start 2024-01-15T18:22:19.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.742+0000 7fafea800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fafe41468d0 0x7fafe40ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:19.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.742+0000 7fafea800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fafe40cd070 0x7fafe406c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:19.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.742+0000 7fafea800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe40ad920 con 0x7fafe40cef30 2024-01-15T18:22:19.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.742+0000 7fafea800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafe40ac880 con 0x7fafe40cd070 2024-01-15T18:22:19.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.742+0000 7fafea800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe40acb60 con 0x7fafe41468d0 2024-01-15T18:22:19.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.742+0000 7fafe2a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fafe40cef30 0x7fafe40cc960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43382/0 (socket says 172.21.15.27:43382) 2024-01-15T18:22:19.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.742+0000 7fafe2a00640 1 -- 172.21.15.27:0/4270854900 learned_addr learned my addr 172.21.15.27:0/4270854900 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:22:19.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.742+0000 7fafe3e00640 1 --2- 172.21.15.27:0/4270854900 >> v2:172.21.15.93:3300/0 conn(0x7fafe40cd070 0x7fafe406c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:19.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.742+0000 7fafe3400640 1 --2- 172.21.15.27:0/4270854900 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fafe41468d0 0x7fafe40ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:19.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.742+0000 7fafe2000640 1 -- 172.21.15.27:0/4270854900 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901956662 0 0) 0x7fafcc001120 con 0x7fafe40cef30 2024-01-15T18:22:19.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.742+0000 7fafe2000640 1 -- 172.21.15.27:0/4270854900 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafc4003530 con 0x7fafe40cef30 2024-01-15T18:22:19.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.742+0000 7fafe2000640 1 -- 172.21.15.27:0/4270854900 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2800453506 0 0) 0x7fafc4003530 con 0x7fafe40cef30 2024-01-15T18:22:19.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.742+0000 7fafe2000640 1 -- 172.21.15.27:0/4270854900 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fafe41468d0 msgr2=0x7fafe40ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:19.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.742+0000 7fafe2000640 1 --2- 172.21.15.27:0/4270854900 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fafe41468d0 0x7fafe40ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:19.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.742+0000 7fafe2000640 1 -- 172.21.15.27:0/4270854900 >> v2:172.21.15.93:3300/0 conn(0x7fafe40cd070 msgr2=0x7fafe406c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:19.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.742+0000 7fafe2000640 1 --2- 172.21.15.27:0/4270854900 >> v2:172.21.15.93:3300/0 conn(0x7fafe40cd070 0x7fafe406c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:19.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.742+0000 7fafe2000640 1 -- 172.21.15.27:0/4270854900 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe40ace40 con 0x7fafe40cef30 2024-01-15T18:22:19.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.742+0000 7fafe2000640 1 -- 172.21.15.27:0/4270854900 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fafcc001810 con 0x7fafe40cef30 2024-01-15T18:22:19.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.742+0000 7fafe3e00640 1 --2- 172.21.15.27:0/4270854900 >> v2:172.21.15.93:3300/0 conn(0x7fafe40cd070 0x7fafe406c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:22:19.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.742+0000 7fafe2000640 1 -- 172.21.15.27:0/4270854900 >> v1:172.21.15.27:6789/0 conn(0x7fafe40cef30 legacy=0x7fafe40cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:19.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.746+0000 7fafe2000640 1 --2- 172.21.15.27:0/4270854900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafc40039f0 0x7fafc4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:19.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.746+0000 7fafe2000640 1 --2- 172.21.15.27:0/4270854900 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fafc40047d0 0x7fafc4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:19.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.746+0000 7fafe2000640 1 --2- 172.21.15.27:0/4270854900 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fafe41468d0 0x7fafc4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:19.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.746+0000 7fafe2000640 1 -- 172.21.15.27:0/4270854900 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe40ac880 con 0x7fafc40039f0 2024-01-15T18:22:19.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.746+0000 7fafe2000640 1 -- 172.21.15.27:0/4270854900 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe40acb60 con 0x7fafc40047d0 2024-01-15T18:22:19.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.746+0000 7fafe2000640 1 -- 172.21.15.27:0/4270854900 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc400b990 con 0x7fafe41468d0 2024-01-15T18:22:19.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.746+0000 7fafe3400640 1 --2- 172.21.15.27:0/4270854900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafc40039f0 0x7fafc4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:19.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.746+0000 7fafe2a00640 1 --2- 172.21.15.27:0/4270854900 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fafc40047d0 0x7fafc4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:19.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.746+0000 7fafe3e00640 1 --2- 172.21.15.27:0/4270854900 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fafe41468d0 0x7fafc4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:19.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.746+0000 7fafe3400640 1 -- 172.21.15.27:0/4270854900 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fafe41468d0 msgr2=0x7fafc4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:19.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.746+0000 7fafe3400640 1 --2- 172.21.15.27:0/4270854900 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fafe41468d0 0x7fafc4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:19.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.746+0000 7fafe3400640 1 -- 172.21.15.27:0/4270854900 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fafc40047d0 msgr2=0x7fafc4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:19.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.746+0000 7fafe3400640 1 --2- 172.21.15.27:0/4270854900 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fafc40047d0 0x7fafc4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:19.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.746+0000 7fafe3400640 1 -- 172.21.15.27:0/4270854900 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc4003530 con 0x7fafc40039f0 2024-01-15T18:22:19.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.746+0000 7fafe2a00640 1 --2- 172.21.15.27:0/4270854900 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fafc40047d0 0x7fafc4004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:22:19.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.746+0000 7fafe3400640 1 --2- 172.21.15.27:0/4270854900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafc40039f0 0x7fafc4003e10 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7fafd4009b10 tx=0x7fafd401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c9414ede9852c0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:19.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.746+0000 7fafe2000640 1 -- 172.21.15.27:0/4270854900 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd40a9450 con 0x7fafc40039f0 2024-01-15T18:22:19.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.746+0000 7fafe2000640 1 -- 172.21.15.27:0/4270854900 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd40a95b0 con 0x7fafc40039f0 2024-01-15T18:22:19.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.746+0000 7fafe2000640 1 -- 172.21.15.27:0/4270854900 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd40a9960 con 0x7fafc40039f0 2024-01-15T18:22:19.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.750+0000 7fafea800640 1 -- 172.21.15.27:0/4270854900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafc40039f0 msgr2=0x7fafc4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:19.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.750+0000 7fafea800640 1 --2- 172.21.15.27:0/4270854900 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafc40039f0 0x7fafc4003e10 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7fafd4009b10 tx=0x7fafd401ca50 comp rx=0 tx=0).stop 2024-01-15T18:22:19.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.750+0000 7fafe3e00640 1 -- 172.21.15.27:0/4270854900 reap_dead start 2024-01-15T18:22:19.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.750+0000 7fafea800640 1 -- 172.21.15.27:0/4270854900 shutdown_connections 2024-01-15T18:22:19.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.750+0000 7fafea800640 1 -- 172.21.15.27:0/4270854900 >> 172.21.15.27:0/4270854900 conn(0x7fafe40cb700 msgr2=0x7fafe40cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:22:19.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.750+0000 7fafea800640 1 -- 172.21.15.27:0/4270854900 shutdown_connections 2024-01-15T18:22:19.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.750+0000 7fafea800640 1 -- 172.21.15.27:0/4270854900 wait complete. 2024-01-15T18:22:19.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.750+0000 7fafea800640 1 Processor -- start 2024-01-15T18:22:19.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.750+0000 7fafea800640 1 -- start start 2024-01-15T18:22:19.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.750+0000 7fafea800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafc40039f0 0x7fafe41454e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:19.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.750+0000 7fafea800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fafe40b46b0 0x7fafe40b2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:19.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.750+0000 7fafea800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fafe40b29a0 0x7fafe40700e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:19.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.750+0000 7fafea800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe40b2e30 con 0x7fafc40039f0 2024-01-15T18:22:19.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.750+0000 7fafea800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe40b3650 con 0x7fafe40b46b0 2024-01-15T18:22:19.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.750+0000 7fafea800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe415a720 con 0x7fafe40b29a0 2024-01-15T18:22:19.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.750+0000 7fafe3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafc40039f0 0x7fafe41454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:19.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.750+0000 7fafe3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafc40039f0 0x7fafe41454e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55506/0 (socket says 172.21.15.27:55506) 2024-01-15T18:22:19.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.754+0000 7fafe3400640 1 -- 172.21.15.27:0/1016831714 learned_addr learned my addr 172.21.15.27:0/1016831714 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:19.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.754+0000 7fafe2a00640 1 --2- 172.21.15.27:0/1016831714 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fafe40b46b0 0x7fafe40b2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:19.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.754+0000 7fafe3e00640 1 --2- 172.21.15.27:0/1016831714 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fafe40b29a0 0x7fafe40700e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:19.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.754+0000 7fafe3400640 1 -- 172.21.15.27:0/1016831714 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fafe40b29a0 msgr2=0x7fafe40700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:19.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.754+0000 7fafe3400640 1 --2- 172.21.15.27:0/1016831714 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fafe40b29a0 0x7fafe40700e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:19.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.754+0000 7fafe3400640 1 -- 172.21.15.27:0/1016831714 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fafe40b46b0 msgr2=0x7fafe40b2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:19.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.754+0000 7fafe3400640 1 --2- 172.21.15.27:0/1016831714 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fafe40b46b0 0x7fafe40b2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:19.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.754+0000 7fafe3400640 1 -- 172.21.15.27:0/1016831714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe40cf490 con 0x7fafc40039f0 2024-01-15T18:22:19.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.754+0000 7fafe3400640 1 --2- 172.21.15.27:0/1016831714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafc40039f0 0x7fafe41454e0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fafd4009b10 tx=0x7fafc4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:19.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.754+0000 7fafdb400640 1 -- 172.21.15.27:0/1016831714 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd40b6070 con 0x7fafc40039f0 2024-01-15T18:22:19.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.754+0000 7fafdb400640 1 -- 172.21.15.27:0/1016831714 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd400f000 con 0x7fafc40039f0 2024-01-15T18:22:19.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.754+0000 7fafdb400640 1 -- 172.21.15.27:0/1016831714 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd40bc420 con 0x7fafc40039f0 2024-01-15T18:22:19.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.754+0000 7fafea800640 1 -- 172.21.15.27:0/1016831714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe40b38f0 con 0x7fafc40039f0 2024-01-15T18:22:19.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.754+0000 7fafea800640 1 -- 172.21.15.27:0/1016831714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe40b3eb0 con 0x7fafc40039f0 2024-01-15T18:22:19.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.754+0000 7fafea800640 1 -- 172.21.15.27:0/1016831714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafe40b2db0 con 0x7fafc40039f0 2024-01-15T18:22:19.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.754+0000 7fafdb400640 1 -- 172.21.15.27:0/1016831714 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7fafd40ba070 con 0x7fafc40039f0 2024-01-15T18:22:19.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.754+0000 7fafdb400640 1 --2- 172.21.15.27:0/1016831714 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fafbc067b80 0x7fafbc06a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:19.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.758+0000 7fafdb400640 1 -- 172.21.15.27:0/1016831714 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fafd411ceb0 con 0x7fafc40039f0 2024-01-15T18:22:19.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.758+0000 7fafe2a00640 1 --2- 172.21.15.27:0/1016831714 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fafbc067b80 0x7fafbc06a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:19.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.758+0000 7fafe2a00640 1 --2- 172.21.15.27:0/1016831714 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fafbc067b80 0x7fafbc06a040 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fafe41f9570 tx=0x7fafcc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:19.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.762+0000 7fafdb400640 1 -- 172.21.15.27:0/1016831714 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fafd40ebcc0 con 0x7fafc40039f0 2024-01-15T18:22:19.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.950+0000 7fafea800640 1 -- 172.21.15.27:0/1016831714 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fafe40cf310 con 0x7fafbc067b80 2024-01-15T18:22:19.951 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:22:19.949+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:22:19.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:19.954+0000 7fafdb400640 1 -- 172.21.15.27:0/1016831714 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fafcc001120 con 0x7fafbc067b80 2024-01-15T18:22:19.952 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:22:19.952 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:22:19.960 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:21.961 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:22:22.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.122+0000 7ff832a00640 1 Processor -- start 2024-01-15T18:22:22.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.122+0000 7ff832a00640 1 -- start start 2024-01-15T18:22:22.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.122+0000 7ff832a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff82c06c520 0x7ff82c0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:22.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.122+0000 7ff832a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff82c0b0d70 con 0x7ff82c15d570 2024-01-15T18:22:22.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.122+0000 7ff832a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff82c0d0540 con 0x7ff82c15d190 2024-01-15T18:22:22.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.122+0000 7ff832a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff82c0b0440 con 0x7ff82c06c520 2024-01-15T18:22:22.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82b400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff82c15d570 0x7ff82c06be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43398/0 (socket says 172.21.15.27:43398) 2024-01-15T18:22:22.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82b400640 1 -- 172.21.15.27:0/2740194293 learned_addr learned my addr 172.21.15.27:0/2740194293 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:22:22.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff830a00640 1 --2- 172.21.15.27:0/2740194293 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff82c06c520 0x7ff82c0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:22.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82aa00640 1 -- 172.21.15.27:0/2740194293 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514195736 0 0) 0x7ff82c0b0d70 con 0x7ff82c15d570 2024-01-15T18:22:22.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82aa00640 1 -- 172.21.15.27:0/2740194293 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff80c003530 con 0x7ff82c15d570 2024-01-15T18:22:22.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82aa00640 1 -- 172.21.15.27:0/2740194293 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763276030 0 0) 0x7ff82c0d0540 con 0x7ff82c15d190 2024-01-15T18:22:22.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82aa00640 1 -- 172.21.15.27:0/2740194293 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff82c0b0d70 con 0x7ff82c15d190 2024-01-15T18:22:22.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82aa00640 1 -- 172.21.15.27:0/2740194293 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2556115596 0 0) 0x7ff80c003530 con 0x7ff82c15d570 2024-01-15T18:22:22.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82aa00640 1 -- 172.21.15.27:0/2740194293 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff82c06c520 msgr2=0x7ff82c0cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:22.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82aa00640 1 --2- 172.21.15.27:0/2740194293 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff82c06c520 0x7ff82c0cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:22.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82aa00640 1 -- 172.21.15.27:0/2740194293 >> v1:172.21.15.93:6789/0 conn(0x7ff82c15d190 legacy=0x7ff82c0cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:22.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82aa00640 1 -- 172.21.15.27:0/2740194293 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82c0b0720 con 0x7ff82c15d570 2024-01-15T18:22:22.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82aa00640 1 -- 172.21.15.27:0/2740194293 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7ff814001680 con 0x7ff82c15d570 2024-01-15T18:22:22.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82aa00640 1 -- 172.21.15.27:0/2740194293 >> v1:172.21.15.27:6789/0 conn(0x7ff82c15d570 legacy=0x7ff82c06be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:22.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82aa00640 1 --2- 172.21.15.27:0/2740194293 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff80c003bb0 0x7ff80c003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:22.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82aa00640 1 --2- 172.21.15.27:0/2740194293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff80c0047d0 0x7ff80c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:22.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82aa00640 1 --2- 172.21.15.27:0/2740194293 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff82c06c520 0x7ff80c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:22.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82aa00640 1 -- 172.21.15.27:0/2740194293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82c0b0440 con 0x7ff80c0047d0 2024-01-15T18:22:22.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82aa00640 1 -- 172.21.15.27:0/2740194293 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c00baa0 con 0x7ff80c003bb0 2024-01-15T18:22:22.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82aa00640 1 -- 172.21.15.27:0/2740194293 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c00bc60 con 0x7ff82c06c520 2024-01-15T18:22:22.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82b400640 1 --2- 172.21.15.27:0/2740194293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff80c0047d0 0x7ff80c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:22.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82be00640 1 --2- 172.21.15.27:0/2740194293 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff80c003bb0 0x7ff80c003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:22.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff830a00640 1 --2- 172.21.15.27:0/2740194293 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff82c06c520 0x7ff80c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:22.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82b400640 1 -- 172.21.15.27:0/2740194293 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff82c06c520 msgr2=0x7ff80c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:22.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82b400640 1 --2- 172.21.15.27:0/2740194293 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff82c06c520 0x7ff80c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:22.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82b400640 1 -- 172.21.15.27:0/2740194293 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff80c003bb0 msgr2=0x7ff80c003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:22.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82b400640 1 --2- 172.21.15.27:0/2740194293 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff80c003bb0 0x7ff80c003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:22.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.126+0000 7ff82b400640 1 -- 172.21.15.27:0/2740194293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80c003530 con 0x7ff80c0047d0 2024-01-15T18:22:22.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.130+0000 7ff82b400640 1 --2- 172.21.15.27:0/2740194293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff80c0047d0 0x7ff80c004c50 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7ff814001ec0 tx=0x7ff81401cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5463daac7dc4d886 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:22.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.130+0000 7ff82aa00640 1 -- 172.21.15.27:0/2740194293 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8140058e0 con 0x7ff80c0047d0 2024-01-15T18:22:22.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.130+0000 7ff82aa00640 1 -- 172.21.15.27:0/2740194293 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff814005a60 con 0x7ff80c0047d0 2024-01-15T18:22:22.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.130+0000 7ff82aa00640 1 -- 172.21.15.27:0/2740194293 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff814005e10 con 0x7ff80c0047d0 2024-01-15T18:22:22.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.130+0000 7ff832a00640 1 -- 172.21.15.27:0/2740194293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff80c0047d0 msgr2=0x7ff80c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:22.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.130+0000 7ff832a00640 1 --2- 172.21.15.27:0/2740194293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff80c0047d0 0x7ff80c004c50 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7ff814001ec0 tx=0x7ff81401cb60 comp rx=0 tx=0).stop 2024-01-15T18:22:22.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.130+0000 7ff830a00640 1 -- 172.21.15.27:0/2740194293 reap_dead start 2024-01-15T18:22:22.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.130+0000 7ff832a00640 1 -- 172.21.15.27:0/2740194293 shutdown_connections 2024-01-15T18:22:22.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.130+0000 7ff832a00640 1 -- 172.21.15.27:0/2740194293 >> 172.21.15.27:0/2740194293 conn(0x7ff82c0d0e70 msgr2=0x7ff82c0d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:22:22.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.130+0000 7ff832a00640 1 -- 172.21.15.27:0/2740194293 shutdown_connections 2024-01-15T18:22:22.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.130+0000 7ff832a00640 1 -- 172.21.15.27:0/2740194293 wait complete. 2024-01-15T18:22:22.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.130+0000 7ff832a00640 1 Processor -- start 2024-01-15T18:22:22.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.134+0000 7ff832a00640 1 -- start start 2024-01-15T18:22:22.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.134+0000 7ff832a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff80c0047d0 0x7ff82c0b68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:22.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.134+0000 7ff832a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff82c1544a0 0x7ff82c0b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:22.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.134+0000 7ff832a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff82c1569d0 0x7ff82c156e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:22.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.134+0000 7ff832a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82c157390 con 0x7ff80c0047d0 2024-01-15T18:22:22.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.134+0000 7ff832a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82c0b4c10 con 0x7ff82c1569d0 2024-01-15T18:22:22.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.134+0000 7ff832a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff82c0c53a0 con 0x7ff82c1544a0 2024-01-15T18:22:22.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.134+0000 7ff82be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff80c0047d0 0x7ff82c0b68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:22.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.134+0000 7ff82be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff80c0047d0 0x7ff82c0b68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55522/0 (socket says 172.21.15.27:55522) 2024-01-15T18:22:22.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.134+0000 7ff82be00640 1 -- 172.21.15.27:0/655361436 learned_addr learned my addr 172.21.15.27:0/655361436 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:22.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.134+0000 7ff82b400640 1 --2- 172.21.15.27:0/655361436 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff82c1544a0 0x7ff82c0b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:22.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.134+0000 7ff830a00640 1 --2- 172.21.15.27:0/655361436 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff82c1569d0 0x7ff82c156e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:22.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.134+0000 7ff82be00640 1 -- 172.21.15.27:0/655361436 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff82c1544a0 msgr2=0x7ff82c0b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:22.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.134+0000 7ff82be00640 1 --2- 172.21.15.27:0/655361436 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff82c1544a0 0x7ff82c0b46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:22.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.134+0000 7ff82be00640 1 -- 172.21.15.27:0/655361436 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff82c1569d0 msgr2=0x7ff82c156e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:22.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.134+0000 7ff82be00640 1 --2- 172.21.15.27:0/655361436 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff82c1569d0 0x7ff82c156e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:22.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.134+0000 7ff82be00640 1 -- 172.21.15.27:0/655361436 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff82c154920 con 0x7ff80c0047d0 2024-01-15T18:22:22.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.134+0000 7ff82be00640 1 --2- 172.21.15.27:0/655361436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff80c0047d0 0x7ff82c0b68f0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7ff81c012830 tx=0x7ff81c012cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:22.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.134+0000 7ff823e00640 1 -- 172.21.15.27:0/655361436 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff81c006ff0 con 0x7ff80c0047d0 2024-01-15T18:22:22.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.134+0000 7ff823e00640 1 -- 172.21.15.27:0/655361436 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff81c010b70 con 0x7ff80c0047d0 2024-01-15T18:22:22.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.134+0000 7ff823e00640 1 -- 172.21.15.27:0/655361436 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff81c016480 con 0x7ff80c0047d0 2024-01-15T18:22:22.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.134+0000 7ff832a00640 1 -- 172.21.15.27:0/655361436 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff82c0b4e40 con 0x7ff80c0047d0 2024-01-15T18:22:22.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.134+0000 7ff832a00640 1 -- 172.21.15.27:0/655361436 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff82c0b5400 con 0x7ff80c0047d0 2024-01-15T18:22:22.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.134+0000 7ff832a00640 1 -- 172.21.15.27:0/655361436 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff82c04e850 con 0x7ff80c0047d0 2024-01-15T18:22:22.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.138+0000 7ff823e00640 1 -- 172.21.15.27:0/655361436 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7ff81c0165e0 con 0x7ff80c0047d0 2024-01-15T18:22:22.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.138+0000 7ff823e00640 1 --2- 172.21.15.27:0/655361436 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff804067bb0 0x7ff80406a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:22.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.138+0000 7ff823e00640 1 -- 172.21.15.27:0/655361436 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7ff81c086a70 con 0x7ff80c0047d0 2024-01-15T18:22:22.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.138+0000 7ff82b400640 1 --2- 172.21.15.27:0/655361436 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff804067bb0 0x7ff80406a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:22.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.138+0000 7ff82b400640 1 --2- 172.21.15.27:0/655361436 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff804067bb0 0x7ff80406a070 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7ff8140aa000 tx=0x7ff814001a00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:22.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.142+0000 7ff823e00640 1 -- 172.21.15.27:0/655361436 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff81c055430 con 0x7ff80c0047d0 2024-01-15T18:22:22.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.330+0000 7ff832a00640 1 -- 172.21.15.27:0/655361436 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff82c0c54e0 con 0x7ff804067bb0 2024-01-15T18:22:22.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:22.338+0000 7ff823e00640 1 -- 172.21.15.27:0/655361436 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff814005c50 con 0x7ff804067bb0 2024-01-15T18:22:22.334 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:22:22.333+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:22:22.334 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:22:22.334 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:22:22.342 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:24.344 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:22:24.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.506+0000 7fc945e00640 1 Processor -- start 2024-01-15T18:22:24.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.506+0000 7fc945e00640 1 -- start start 2024-01-15T18:22:24.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.510+0000 7fc945e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc94015d190 0x7fc9400cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:24.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.510+0000 7fc945e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fc94015d570 0x7fc94006be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:24.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.510+0000 7fc945e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9400b0d70 con 0x7fc94006b810 2024-01-15T18:22:24.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.510+0000 7fc945e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9400d0540 con 0x7fc94015d190 2024-01-15T18:22:24.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.510+0000 7fc945e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9400b0440 con 0x7fc94015d570 2024-01-15T18:22:24.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.510+0000 7fc93ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc94015d190 0x7fc9400cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:24.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.510+0000 7fc93ea00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc94015d190 0x7fc9400cfcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59078/0 (socket says 172.21.15.27:59078) 2024-01-15T18:22:24.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.510+0000 7fc93ea00640 1 -- 172.21.15.27:0/2528146410 learned_addr learned my addr 172.21.15.27:0/2528146410 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:24.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.510+0000 7fc93f400640 1 --1- 172.21.15.27:0/2528146410 >> v1:172.21.15.27:6789/0 conn(0x7fc94006b810 0x7fc9400cd5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51548/0 (socket says 172.21.15.27:51548) 2024-01-15T18:22:24.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.510+0000 7fc93e000640 1 --2- 172.21.15.27:0/2528146410 >> v2:172.21.15.93:3300/0 conn(0x7fc94015d570 0x7fc94006be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:24.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.510+0000 7fc93ea00640 1 -- 172.21.15.27:0/2528146410 >> v2:172.21.15.93:3300/0 conn(0x7fc94015d570 msgr2=0x7fc94006be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:24.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.510+0000 7fc93ea00640 1 --2- 172.21.15.27:0/2528146410 >> v2:172.21.15.93:3300/0 conn(0x7fc94015d570 0x7fc94006be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:24.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.510+0000 7fc93ea00640 1 -- 172.21.15.27:0/2528146410 >> v1:172.21.15.27:6789/0 conn(0x7fc94006b810 legacy=0x7fc9400cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:24.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.510+0000 7fc93ea00640 1 -- 172.21.15.27:0/2528146410 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9400b0720 con 0x7fc94015d190 2024-01-15T18:22:24.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.514+0000 7fc93ea00640 1 --2- 172.21.15.27:0/2528146410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94015d190 0x7fc9400cfcf0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fc9400713a0 tx=0x7fc93001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a22ce81ddbea13a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:24.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.514+0000 7fc93d600640 1 -- 172.21.15.27:0/2528146410 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9300ac640 con 0x7fc94015d190 2024-01-15T18:22:24.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.514+0000 7fc93d600640 1 -- 172.21.15.27:0/2528146410 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9300ac7a0 con 0x7fc94015d190 2024-01-15T18:22:24.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.514+0000 7fc93d600640 1 -- 172.21.15.27:0/2528146410 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9300acb30 con 0x7fc94015d190 2024-01-15T18:22:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.514+0000 7fc945e00640 1 -- 172.21.15.27:0/2528146410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94015d190 msgr2=0x7fc9400cfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.514+0000 7fc945e00640 1 --2- 172.21.15.27:0/2528146410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94015d190 0x7fc9400cfcf0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fc9400713a0 tx=0x7fc93001c4d0 comp rx=0 tx=0).stop 2024-01-15T18:22:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.514+0000 7fc945e00640 1 -- 172.21.15.27:0/2528146410 shutdown_connections 2024-01-15T18:22:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.514+0000 7fc945e00640 1 --2- 172.21.15.27:0/2528146410 >> v2:172.21.15.93:3300/0 conn(0x7fc94015d570 0x7fc94006be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.514+0000 7fc945e00640 1 --2- 172.21.15.27:0/2528146410 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94015d190 0x7fc9400cfcf0 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.514+0000 7fc945e00640 1 -- 172.21.15.27:0/2528146410 >> 172.21.15.27:0/2528146410 conn(0x7fc9400d0e70 msgr2=0x7fc9400d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:22:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.514+0000 7fc945e00640 1 -- 172.21.15.27:0/2528146410 shutdown_connections 2024-01-15T18:22:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.514+0000 7fc945e00640 1 -- 172.21.15.27:0/2528146410 wait complete. 2024-01-15T18:22:24.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.514+0000 7fc945e00640 1 Processor -- start 2024-01-15T18:22:24.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.514+0000 7fc945e00640 1 -- start start 2024-01-15T18:22:24.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.518+0000 7fc945e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc94006b810 0x7fc940150480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:24.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.518+0000 7fc945e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94015d190 0x7fc9400b5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:24.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.518+0000 7fc945e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc94015d570 0x7fc940158b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:24.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.518+0000 7fc945e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9401588f0 con 0x7fc94015d190 2024-01-15T18:22:24.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.518+0000 7fc945e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9400b4260 con 0x7fc94006b810 2024-01-15T18:22:24.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.518+0000 7fc945e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9400b4540 con 0x7fc94015d570 2024-01-15T18:22:24.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.518+0000 7fc93e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94015d190 0x7fc9400b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.518+0000 7fc93e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94015d190 0x7fc9400b5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59094/0 (socket says 172.21.15.27:59094) 2024-01-15T18:22:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.518+0000 7fc93e000640 1 -- 172.21.15.27:0/2422819725 learned_addr learned my addr 172.21.15.27:0/2422819725 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.518+0000 7fc93f400640 1 --2- 172.21.15.27:0/2422819725 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc94015d570 0x7fc940158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.518+0000 7fc93ea00640 1 --2- 172.21.15.27:0/2422819725 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc94006b810 0x7fc940150480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.518+0000 7fc93e000640 1 -- 172.21.15.27:0/2422819725 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc94015d570 msgr2=0x7fc940158b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.518+0000 7fc93e000640 1 --2- 172.21.15.27:0/2422819725 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc94015d570 0x7fc940158b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.518+0000 7fc93e000640 1 -- 172.21.15.27:0/2422819725 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc94006b810 msgr2=0x7fc940150480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.518+0000 7fc93e000640 1 --2- 172.21.15.27:0/2422819725 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc94006b810 0x7fc940150480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.518+0000 7fc93e000640 1 -- 172.21.15.27:0/2422819725 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9300094d0 con 0x7fc94015d190 2024-01-15T18:22:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.518+0000 7fc93e000640 1 --2- 172.21.15.27:0/2422819725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc94015d190 0x7fc9400b5300 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fc92800d5f0 tx=0x7fc92800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.518+0000 7fc937400640 1 -- 172.21.15.27:0/2422819725 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9280125f0 con 0x7fc94015d190 2024-01-15T18:22:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.518+0000 7fc937400640 1 -- 172.21.15.27:0/2422819725 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc928012750 con 0x7fc94015d190 2024-01-15T18:22:24.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.518+0000 7fc945e00640 1 -- 172.21.15.27:0/2422819725 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9400b4820 con 0x7fc94015d190 2024-01-15T18:22:24.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.518+0000 7fc937400640 1 -- 172.21.15.27:0/2422819725 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc928012ae0 con 0x7fc94015d190 2024-01-15T18:22:24.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.518+0000 7fc945e00640 1 -- 172.21.15.27:0/2422819725 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9400b4de0 con 0x7fc94015d190 2024-01-15T18:22:24.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.518+0000 7fc945e00640 1 -- 172.21.15.27:0/2422819725 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc940159050 con 0x7fc94015d190 2024-01-15T18:22:24.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.518+0000 7fc937400640 1 -- 172.21.15.27:0/2422819725 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7fc928012c40 con 0x7fc94015d190 2024-01-15T18:22:24.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.522+0000 7fc937400640 1 --2- 172.21.15.27:0/2422819725 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc918067b80 0x7fc91806a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:24.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.522+0000 7fc937400640 1 -- 172.21.15.27:0/2422819725 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fc928083780 con 0x7fc94015d190 2024-01-15T18:22:24.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.522+0000 7fc93ea00640 1 --2- 172.21.15.27:0/2422819725 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc918067b80 0x7fc91806a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:24.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.522+0000 7fc93ea00640 1 --2- 172.21.15.27:0/2422819725 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc918067b80 0x7fc91806a040 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fc9300ae000 tx=0x7fc9300a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:24.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.526+0000 7fc937400640 1 -- 172.21.15.27:0/2422819725 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc928052140 con 0x7fc94015d190 2024-01-15T18:22:24.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.714+0000 7fc945e00640 1 -- 172.21.15.27:0/2422819725 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc94004e850 con 0x7fc918067b80 2024-01-15T18:22:24.716 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:22:24.717+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:22:24.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:24.722+0000 7fc937400640 1 -- 172.21.15.27:0/2422819725 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc94004e850 con 0x7fc918067b80 2024-01-15T18:22:24.717 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:22:24.717 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:22:24.725 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:26.727 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:22:26.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.890+0000 7f1397400640 1 Processor -- start 2024-01-15T18:22:26.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.894+0000 7f1397400640 1 -- start start 2024-01-15T18:22:26.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.894+0000 7f1397400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f13900cf450 0x7f13900d1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:26.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.894+0000 7f1397400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13900cba60 0x7f13900cf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:26.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.894+0000 7f1397400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f139015ee30 0x7f139015f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:26.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.894+0000 7f1397400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13900cefa0 con 0x7f13900cf450 2024-01-15T18:22:26.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.894+0000 7f1397400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13900ce240 con 0x7f139015ee30 2024-01-15T18:22:26.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.894+0000 7f1397400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13900cfd70 con 0x7f13900cba60 2024-01-15T18:22:26.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.894+0000 7f1394a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f13900cf450 0x7f13900d1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:26.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.894+0000 7f1394a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f13900cf450 0x7f13900d1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59096/0 (socket says 172.21.15.27:59096) 2024-01-15T18:22:26.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.894+0000 7f1394a00640 1 -- 172.21.15.27:0/3546081277 learned_addr learned my addr 172.21.15.27:0/3546081277 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:26.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.894+0000 7f1395400640 1 --2- 172.21.15.27:0/3546081277 >> v2:172.21.15.93:3300/0 conn(0x7f139015ee30 0x7f139015f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:26.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.894+0000 7f138fe00640 1 --2- 172.21.15.27:0/3546081277 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13900cba60 0x7f13900cf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:26.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.894+0000 7f1394a00640 1 -- 172.21.15.27:0/3546081277 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13900cba60 msgr2=0x7f13900cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:26.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.894+0000 7f1394a00640 1 --2- 172.21.15.27:0/3546081277 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13900cba60 0x7f13900cf830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:26.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.894+0000 7f1394a00640 1 -- 172.21.15.27:0/3546081277 >> v2:172.21.15.93:3300/0 conn(0x7f139015ee30 msgr2=0x7f139015f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:26.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.894+0000 7f1394a00640 1 --2- 172.21.15.27:0/3546081277 >> v2:172.21.15.93:3300/0 conn(0x7f139015ee30 0x7f139015f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:26.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.894+0000 7f1394a00640 1 -- 172.21.15.27:0/3546081277 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13900ccd70 con 0x7f13900cf450 2024-01-15T18:22:26.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.898+0000 7f1394a00640 1 --2- 172.21.15.27:0/3546081277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13900cf450 0x7f13900d1a40 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f139009aa00 tx=0x7f138001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=464f3c175ea8725d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:26.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.898+0000 7f138f400640 1 -- 172.21.15.27:0/3546081277 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13800ac410 con 0x7f13900cf450 2024-01-15T18:22:26.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.898+0000 7f138f400640 1 -- 172.21.15.27:0/3546081277 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13800ac570 con 0x7f13900cf450 2024-01-15T18:22:26.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.898+0000 7f138f400640 1 -- 172.21.15.27:0/3546081277 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13800ac900 con 0x7f13900cf450 2024-01-15T18:22:26.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.898+0000 7f1397400640 1 -- 172.21.15.27:0/3546081277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13900cf450 msgr2=0x7f13900d1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:26.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.898+0000 7f1397400640 1 --2- 172.21.15.27:0/3546081277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13900cf450 0x7f13900d1a40 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f139009aa00 tx=0x7f138001c500 comp rx=0 tx=0).stop 2024-01-15T18:22:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.898+0000 7f1397400640 1 -- 172.21.15.27:0/3546081277 shutdown_connections 2024-01-15T18:22:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.898+0000 7f1397400640 1 --2- 172.21.15.27:0/3546081277 >> v2:172.21.15.93:3300/0 conn(0x7f139015ee30 0x7f139015f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.898+0000 7f1397400640 1 --2- 172.21.15.27:0/3546081277 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13900cba60 0x7f13900cf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.898+0000 7f1397400640 1 --2- 172.21.15.27:0/3546081277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f13900cf450 0x7f13900d1a40 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.898+0000 7f1397400640 1 -- 172.21.15.27:0/3546081277 >> 172.21.15.27:0/3546081277 conn(0x7f13900d0e70 msgr2=0x7f13900d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:22:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.898+0000 7f1397400640 1 -- 172.21.15.27:0/3546081277 shutdown_connections 2024-01-15T18:22:26.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.898+0000 7f1397400640 1 -- 172.21.15.27:0/3546081277 wait complete. 2024-01-15T18:22:26.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.898+0000 7f1397400640 1 Processor -- start 2024-01-15T18:22:26.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f1397400640 1 -- start start 2024-01-15T18:22:26.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f1397400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13900cba60 0x7f1390153cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f1397400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13900cf450 0x7f13900c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f1397400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f139015ee30 0x7f13900b6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f1397400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13900b5730 con 0x7f139015ee30 2024-01-15T18:22:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f1397400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13900b5fd0 con 0x7f13900cba60 2024-01-15T18:22:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f1397400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13900c7240 con 0x7f13900cf450 2024-01-15T18:22:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f1395400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f139015ee30 0x7f13900b6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f1395400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f139015ee30 0x7f13900b6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59106/0 (socket says 172.21.15.27:59106) 2024-01-15T18:22:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f1395400640 1 -- 172.21.15.27:0/1177084361 learned_addr learned my addr 172.21.15.27:0/1177084361 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f1394a00640 1 --2- 172.21.15.27:0/1177084361 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13900cba60 0x7f1390153cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f138fe00640 1 --2- 172.21.15.27:0/1177084361 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13900cf450 0x7f13900c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:26.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f1395400640 1 -- 172.21.15.27:0/1177084361 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13900cf450 msgr2=0x7f13900c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:26.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f1395400640 1 --2- 172.21.15.27:0/1177084361 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13900cf450 0x7f13900c68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:26.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f1395400640 1 -- 172.21.15.27:0/1177084361 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13900cba60 msgr2=0x7f1390153cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:26.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f1395400640 1 --2- 172.21.15.27:0/1177084361 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13900cba60 0x7f1390153cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:26.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f1395400640 1 -- 172.21.15.27:0/1177084361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13800094d0 con 0x7f139015ee30 2024-01-15T18:22:26.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f1394a00640 1 --2- 172.21.15.27:0/1177084361 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f13900cba60 0x7f1390153cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:22:26.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f138fe00640 1 --2- 172.21.15.27:0/1177084361 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f13900cf450 0x7f13900c68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:22:26.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f1395400640 1 --2- 172.21.15.27:0/1177084361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f139015ee30 0x7f13900b6460 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f1388007e70 tx=0x7f138800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:26.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f138cc00640 1 -- 172.21.15.27:0/1177084361 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1388014620 con 0x7f139015ee30 2024-01-15T18:22:26.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f138cc00640 1 -- 172.21.15.27:0/1177084361 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13880147a0 con 0x7f139015ee30 2024-01-15T18:22:26.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f1397400640 1 -- 172.21.15.27:0/1177084361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13900b5980 con 0x7f139015ee30 2024-01-15T18:22:26.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f138cc00640 1 -- 172.21.15.27:0/1177084361 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1388014b50 con 0x7f139015ee30 2024-01-15T18:22:26.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f1397400640 1 -- 172.21.15.27:0/1177084361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13901541f0 con 0x7f139015ee30 2024-01-15T18:22:26.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.902+0000 7f1397400640 1 -- 172.21.15.27:0/1177084361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f139004e850 con 0x7f139015ee30 2024-01-15T18:22:26.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.906+0000 7f138cc00640 1 -- 172.21.15.27:0/1177084361 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7f1388014cb0 con 0x7f139015ee30 2024-01-15T18:22:26.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.906+0000 7f138cc00640 1 --2- 172.21.15.27:0/1177084361 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1368067c00 0x7f136806a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:26.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.906+0000 7f138cc00640 1 -- 172.21.15.27:0/1177084361 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f1388083d70 con 0x7f139015ee30 2024-01-15T18:22:26.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.906+0000 7f1394a00640 1 --2- 172.21.15.27:0/1177084361 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1368067c00 0x7f136806a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:26.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.906+0000 7f1394a00640 1 --2- 172.21.15.27:0/1177084361 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1368067c00 0x7f136806a0c0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f13800ae000 tx=0x7f13800a6040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:26.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:26.910+0000 7f138cc00640 1 -- 172.21.15.27:0/1177084361 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1388052680 con 0x7f139015ee30 2024-01-15T18:22:27.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:27.098+0000 7f1397400640 1 -- 172.21.15.27:0/1177084361 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f139015fbf0 con 0x7f1368067c00 2024-01-15T18:22:27.101 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:22:27.101+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:22:27.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:27.106+0000 7f138cc00640 1 -- 172.21.15.27:0/1177084361 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f13800ac7e0 con 0x7f1368067c00 2024-01-15T18:22:27.101 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:22:27.102 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:22:27.110 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:28.407 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:22:28.411 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:22:29.110 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:22:29.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.274+0000 7f2deac00640 1 Processor -- start 2024-01-15T18:22:29.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.274+0000 7f2deac00640 1 -- start start 2024-01-15T18:22:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.274+0000 7f2deac00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2de415d0d0 0x7f2de40b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.274+0000 7f2deac00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f2de406e980 0x7f2de406cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.274+0000 7f2deac00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2de406a4f0 0x7f2de406a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.274+0000 7f2deac00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2de406f1d0 con 0x7f2de415d0d0 2024-01-15T18:22:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.274+0000 7f2deac00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2de406ae60 con 0x7f2de406e980 2024-01-15T18:22:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.274+0000 7f2deac00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de4071ac0 con 0x7f2de406a4f0 2024-01-15T18:22:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.274+0000 7f2de3e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2de415d0d0 0x7f2de40b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.274+0000 7f2de3e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2de415d0d0 0x7f2de40b2fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59108/0 (socket says 172.21.15.27:59108) 2024-01-15T18:22:29.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.274+0000 7f2de3e00640 1 -- 172.21.15.27:0/3938539482 learned_addr learned my addr 172.21.15.27:0/3938539482 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:29.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.274+0000 7f2de3400640 1 --2- 172.21.15.27:0/3938539482 >> v2:172.21.15.93:3300/0 conn(0x7f2de406e980 0x7f2de406cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:29.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.274+0000 7f2de8c00640 1 --2- 172.21.15.27:0/3938539482 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2de406a4f0 0x7f2de406a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:29.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.274+0000 7f2de3e00640 1 -- 172.21.15.27:0/3938539482 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2de406a4f0 msgr2=0x7f2de406a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:29.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.274+0000 7f2de3e00640 1 --2- 172.21.15.27:0/3938539482 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2de406a4f0 0x7f2de406a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:29.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.274+0000 7f2de3e00640 1 -- 172.21.15.27:0/3938539482 >> v2:172.21.15.93:3300/0 conn(0x7f2de406e980 msgr2=0x7f2de406cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:29.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.274+0000 7f2de3e00640 1 --2- 172.21.15.27:0/3938539482 >> v2:172.21.15.93:3300/0 conn(0x7f2de406e980 0x7f2de406cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:29.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.274+0000 7f2de3e00640 1 -- 172.21.15.27:0/3938539482 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de4071da0 con 0x7f2de415d0d0 2024-01-15T18:22:29.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.274+0000 7f2de8c00640 1 --2- 172.21.15.27:0/3938539482 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2de406a4f0 0x7f2de406a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:22:29.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.278+0000 7f2de3e00640 1 --2- 172.21.15.27:0/3938539482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2de415d0d0 0x7f2de40b2fa0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f2de4071470 tx=0x7f2dd401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff5f273082933f40 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:29.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.278+0000 7f2de2a00640 1 -- 172.21.15.27:0/3938539482 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd40aa620 con 0x7f2de415d0d0 2024-01-15T18:22:29.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.278+0000 7f2de2a00640 1 -- 172.21.15.27:0/3938539482 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd40aa780 con 0x7f2de415d0d0 2024-01-15T18:22:29.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.278+0000 7f2de2a00640 1 -- 172.21.15.27:0/3938539482 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd40aab10 con 0x7f2de415d0d0 2024-01-15T18:22:29.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.278+0000 7f2deac00640 1 -- 172.21.15.27:0/3938539482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2de415d0d0 msgr2=0x7f2de40b2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:29.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.278+0000 7f2deac00640 1 --2- 172.21.15.27:0/3938539482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2de415d0d0 0x7f2de40b2fa0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f2de4071470 tx=0x7f2dd401c500 comp rx=0 tx=0).stop 2024-01-15T18:22:29.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.278+0000 7f2deac00640 1 -- 172.21.15.27:0/3938539482 shutdown_connections 2024-01-15T18:22:29.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.278+0000 7f2deac00640 1 --2- 172.21.15.27:0/3938539482 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2de406a4f0 0x7f2de406a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:29.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.278+0000 7f2deac00640 1 --2- 172.21.15.27:0/3938539482 >> v2:172.21.15.93:3300/0 conn(0x7f2de406e980 0x7f2de406cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:29.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.278+0000 7f2deac00640 1 --2- 172.21.15.27:0/3938539482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2de415d0d0 0x7f2de40b2fa0 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:29.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.278+0000 7f2deac00640 1 -- 172.21.15.27:0/3938539482 >> 172.21.15.27:0/3938539482 conn(0x7f2de406b600 msgr2=0x7f2de406ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:22:29.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.278+0000 7f2deac00640 1 -- 172.21.15.27:0/3938539482 shutdown_connections 2024-01-15T18:22:29.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.278+0000 7f2deac00640 1 -- 172.21.15.27:0/3938539482 wait complete. 2024-01-15T18:22:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.282+0000 7f2deac00640 1 Processor -- start 2024-01-15T18:22:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.282+0000 7f2deac00640 1 -- start start 2024-01-15T18:22:29.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.282+0000 7f2deac00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2de406a4f0 0x7f2de415f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:29.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.282+0000 7f2deac00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2de406e980 0x7f2de41596a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:29.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.282+0000 7f2deac00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2de415d0d0 0x7f2de4159be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:29.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.282+0000 7f2deac00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de41f7b70 con 0x7f2de406e980 2024-01-15T18:22:29.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.282+0000 7f2deac00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de415f6d0 con 0x7f2de406a4f0 2024-01-15T18:22:29.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.282+0000 7f2deac00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de41f71d0 con 0x7f2de415d0d0 2024-01-15T18:22:29.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.282+0000 7f2de3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2de406e980 0x7f2de41596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:29.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.282+0000 7f2de3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2de406e980 0x7f2de41596a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59114/0 (socket says 172.21.15.27:59114) 2024-01-15T18:22:29.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.282+0000 7f2de3400640 1 -- 172.21.15.27:0/4263253297 learned_addr learned my addr 172.21.15.27:0/4263253297 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:29.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.282+0000 7f2de3e00640 1 --2- 172.21.15.27:0/4263253297 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2de406a4f0 0x7f2de415f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:29.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.282+0000 7f2de8c00640 1 --2- 172.21.15.27:0/4263253297 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2de415d0d0 0x7f2de4159be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:29.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.282+0000 7f2de3400640 1 -- 172.21.15.27:0/4263253297 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2de415d0d0 msgr2=0x7f2de4159be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:29.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.282+0000 7f2de3400640 1 --2- 172.21.15.27:0/4263253297 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2de415d0d0 0x7f2de4159be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:29.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.282+0000 7f2de3400640 1 -- 172.21.15.27:0/4263253297 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2de406a4f0 msgr2=0x7f2de415f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:29.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.282+0000 7f2de3400640 1 --2- 172.21.15.27:0/4263253297 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2de406a4f0 0x7f2de415f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:29.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.282+0000 7f2de3400640 1 -- 172.21.15.27:0/4263253297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd40094d0 con 0x7f2de406e980 2024-01-15T18:22:29.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.282+0000 7f2de3400640 1 --2- 172.21.15.27:0/4263253297 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2de406e980 0x7f2de41596a0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00e960 tx=0x7f2dcc00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:29.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.282+0000 7f2ddbe00640 1 -- 172.21.15.27:0/4263253297 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc009690 con 0x7f2de406e980 2024-01-15T18:22:29.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.282+0000 7f2ddbe00640 1 -- 172.21.15.27:0/4263253297 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dcc00ca00 con 0x7f2de406e980 2024-01-15T18:22:29.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.282+0000 7f2ddbe00640 1 -- 172.21.15.27:0/4263253297 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc00cd70 con 0x7f2de406e980 2024-01-15T18:22:29.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.282+0000 7f2deac00640 1 -- 172.21.15.27:0/4263253297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de41f74b0 con 0x7f2de406e980 2024-01-15T18:22:29.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.282+0000 7f2deac00640 1 -- 172.21.15.27:0/4263253297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de41546b0 con 0x7f2de406e980 2024-01-15T18:22:29.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.286+0000 7f2deac00640 1 -- 172.21.15.27:0/4263253297 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2de41f7cb0 con 0x7f2de406e980 2024-01-15T18:22:29.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.286+0000 7f2ddbe00640 1 -- 172.21.15.27:0/4263253297 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7f2dcc013470 con 0x7f2de406e980 2024-01-15T18:22:29.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.286+0000 7f2ddbe00640 1 --2- 172.21.15.27:0/4263253297 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2dbc067b80 0x7f2dbc06a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:29.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.286+0000 7f2ddbe00640 1 -- 172.21.15.27:0/4263253297 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f2dcc083250 con 0x7f2de406e980 2024-01-15T18:22:29.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.286+0000 7f2de3e00640 1 --2- 172.21.15.27:0/4263253297 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2dbc067b80 0x7f2dbc06a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:29.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.290+0000 7f2de3e00640 1 --2- 172.21.15.27:0/4263253297 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2dbc067b80 0x7f2dbc06a040 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f2dd40ae000 tx=0x7f2dd40a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:29.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.290+0000 7f2ddbe00640 1 -- 172.21.15.27:0/4263253297 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2de41f7cb0 con 0x7f2de406e980 2024-01-15T18:22:29.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.478+0000 7f2deac00640 1 -- 172.21.15.27:0/4263253297 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2de41f7340 con 0x7f2dbc067b80 2024-01-15T18:22:29.479 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:22:29.477+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:22:29.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:29.482+0000 7f2ddbe00640 1 -- 172.21.15.27:0/4263253297 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2de41f7340 con 0x7f2dbc067b80 2024-01-15T18:22:29.479 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:22:29.480 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:22:29.488 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:31.489 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:22:31.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.650+0000 7ff7b3400640 1 Processor -- start 2024-01-15T18:22:31.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.650+0000 7ff7b3400640 1 -- start start 2024-01-15T18:22:31.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.654+0000 7ff7b3400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff7ac15d190 0x7ff7ac0cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:31.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.654+0000 7ff7b3400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff7ac06c520 0x7ff7ac0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:31.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.654+0000 7ff7b3400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7ac0b0d70 con 0x7ff7ac15d570 2024-01-15T18:22:31.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.654+0000 7ff7b3400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7ac0d0540 con 0x7ff7ac15d190 2024-01-15T18:22:31.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.654+0000 7ff7b3400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ac0b0440 con 0x7ff7ac06c520 2024-01-15T18:22:31.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.654+0000 7ff7abe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff7ac15d570 0x7ff7ac06be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51552/0 (socket says 172.21.15.27:51552) 2024-01-15T18:22:31.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.654+0000 7ff7abe00640 1 -- 172.21.15.27:0/3981981624 learned_addr learned my addr 172.21.15.27:0/3981981624 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:22:31.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.654+0000 7ff7b1400640 1 --2- 172.21.15.27:0/3981981624 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff7ac06c520 0x7ff7ac0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:31.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.654+0000 7ff7b0a00640 1 --2- 172.21.15.27:0/3981981624 >> v2:172.21.15.93:3300/0 conn(0x7ff7ac15d190 0x7ff7ac0cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:31.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.654+0000 7ff7ab400640 1 -- 172.21.15.27:0/3981981624 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094530779 0 0) 0x7ff7ac0b0d70 con 0x7ff7ac15d570 2024-01-15T18:22:31.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.654+0000 7ff7ab400640 1 -- 172.21.15.27:0/3981981624 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff78c003530 con 0x7ff7ac15d570 2024-01-15T18:22:31.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.654+0000 7ff7ab400640 1 -- 172.21.15.27:0/3981981624 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1954933644 0 0) 0x7ff78c003530 con 0x7ff7ac15d570 2024-01-15T18:22:31.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.654+0000 7ff7b1400640 1 -- 172.21.15.27:0/3981981624 >> v2:172.21.15.93:3300/0 conn(0x7ff7ac15d190 msgr2=0x7ff7ac0cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:31.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.654+0000 7ff7b1400640 1 --2- 172.21.15.27:0/3981981624 >> v2:172.21.15.93:3300/0 conn(0x7ff7ac15d190 0x7ff7ac0cfcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:31.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.654+0000 7ff7b1400640 1 -- 172.21.15.27:0/3981981624 >> v1:172.21.15.27:6789/0 conn(0x7ff7ac15d570 legacy=0x7ff7ac06be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:31.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.654+0000 7ff7b1400640 1 -- 172.21.15.27:0/3981981624 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ac0b0720 con 0x7ff7ac06c520 2024-01-15T18:22:31.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.654+0000 7ff7b0a00640 1 --2- 172.21.15.27:0/3981981624 >> v2:172.21.15.93:3300/0 conn(0x7ff7ac15d190 0x7ff7ac0cfcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:22:31.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.658+0000 7ff7b1400640 1 --2- 172.21.15.27:0/3981981624 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff7ac06c520 0x7ff7ac0cd5e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4009d90 tx=0x7ff7a401c840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c581499fc00f41af server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:31.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.658+0000 7ff7ab400640 1 -- 172.21.15.27:0/3981981624 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff794001680 con 0x7ff7ac06c520 2024-01-15T18:22:31.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.658+0000 7ff7ab400640 1 -- 172.21.15.27:0/3981981624 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a40a5600 con 0x7ff7ac06c520 2024-01-15T18:22:31.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.658+0000 7ff7ab400640 1 -- 172.21.15.27:0/3981981624 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a40a5990 con 0x7ff7ac06c520 2024-01-15T18:22:31.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.658+0000 7ff7b3400640 1 -- 172.21.15.27:0/3981981624 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff7ac06c520 msgr2=0x7ff7ac0cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:31.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.658+0000 7ff7b3400640 1 --2- 172.21.15.27:0/3981981624 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff7ac06c520 0x7ff7ac0cd5e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4009d90 tx=0x7ff7a401c840 comp rx=0 tx=0).stop 2024-01-15T18:22:31.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.658+0000 7ff7b3400640 1 -- 172.21.15.27:0/3981981624 shutdown_connections 2024-01-15T18:22:31.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.658+0000 7ff7b3400640 1 --2- 172.21.15.27:0/3981981624 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff7ac06c520 0x7ff7ac0cd5e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:31.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.658+0000 7ff7b3400640 1 --2- 172.21.15.27:0/3981981624 >> v2:172.21.15.93:3300/0 conn(0x7ff7ac15d190 0x7ff7ac0cfcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:31.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.658+0000 7ff7b3400640 1 -- 172.21.15.27:0/3981981624 >> 172.21.15.27:0/3981981624 conn(0x7ff7ac0d0e70 msgr2=0x7ff7ac0d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:22:31.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.658+0000 7ff7b3400640 1 -- 172.21.15.27:0/3981981624 shutdown_connections 2024-01-15T18:22:31.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.658+0000 7ff7b3400640 1 -- 172.21.15.27:0/3981981624 wait complete. 2024-01-15T18:22:31.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.662+0000 7ff7b3400640 1 Processor -- start 2024-01-15T18:22:31.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.662+0000 7ff7b3400640 1 -- start start 2024-01-15T18:22:31.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.662+0000 7ff7b3400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff7ac06c520 0x7ff7ac154550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:31.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.662+0000 7ff7b3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7ac15d190 0x7ff7ac0c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:31.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.662+0000 7ff7b3400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff7ac15d570 0x7ff7ac0c4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:31.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.662+0000 7ff7b3400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ac0b67a0 con 0x7ff7ac15d190 2024-01-15T18:22:31.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.662+0000 7ff7b3400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ac0c5440 con 0x7ff7ac06c520 2024-01-15T18:22:31.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.662+0000 7ff7b3400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ac0c5720 con 0x7ff7ac15d570 2024-01-15T18:22:31.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.662+0000 7ff7abe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7ac15d190 0x7ff7ac0c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:31.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.662+0000 7ff7abe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7ac15d190 0x7ff7ac0c48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59130/0 (socket says 172.21.15.27:59130) 2024-01-15T18:22:31.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.662+0000 7ff7abe00640 1 -- 172.21.15.27:0/4197901823 learned_addr learned my addr 172.21.15.27:0/4197901823 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:31.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.662+0000 7ff7b1400640 1 --2- 172.21.15.27:0/4197901823 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff7ac15d570 0x7ff7ac0c4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:31.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.662+0000 7ff7b0a00640 1 --2- 172.21.15.27:0/4197901823 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff7ac06c520 0x7ff7ac154550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:31.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.662+0000 7ff7abe00640 1 -- 172.21.15.27:0/4197901823 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff7ac15d570 msgr2=0x7ff7ac0c4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:31.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.662+0000 7ff7abe00640 1 --2- 172.21.15.27:0/4197901823 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff7ac15d570 0x7ff7ac0c4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:31.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.662+0000 7ff7abe00640 1 -- 172.21.15.27:0/4197901823 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff7ac06c520 msgr2=0x7ff7ac154550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:31.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.662+0000 7ff7abe00640 1 --2- 172.21.15.27:0/4197901823 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff7ac06c520 0x7ff7ac154550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:31.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.662+0000 7ff7abe00640 1 -- 172.21.15.27:0/4197901823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79c009420 con 0x7ff7ac15d190 2024-01-15T18:22:31.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.662+0000 7ff7abe00640 1 --2- 172.21.15.27:0/4197901823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7ac15d190 0x7ff7ac0c48d0 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7ff79400c780 tx=0x7ff79400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:31.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.662+0000 7ff7a8c00640 1 -- 172.21.15.27:0/4197901823 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff794005ae0 con 0x7ff7ac15d190 2024-01-15T18:22:31.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.662+0000 7ff7a8c00640 1 -- 172.21.15.27:0/4197901823 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff794005c40 con 0x7ff7ac15d190 2024-01-15T18:22:31.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.662+0000 7ff7b3400640 1 -- 172.21.15.27:0/4197901823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7a4009a70 con 0x7ff7ac15d190 2024-01-15T18:22:31.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.662+0000 7ff7a8c00640 1 -- 172.21.15.27:0/4197901823 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff794007070 con 0x7ff7ac15d190 2024-01-15T18:22:31.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.662+0000 7ff7b3400640 1 -- 172.21.15.27:0/4197901823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7ac0c5ce0 con 0x7ff7ac15d190 2024-01-15T18:22:31.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.662+0000 7ff7b3400640 1 -- 172.21.15.27:0/4197901823 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7ac04e850 con 0x7ff7ac15d190 2024-01-15T18:22:31.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.666+0000 7ff7a8c00640 1 -- 172.21.15.27:0/4197901823 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7ff7940071d0 con 0x7ff7ac15d190 2024-01-15T18:22:31.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.666+0000 7ff7a8c00640 1 --2- 172.21.15.27:0/4197901823 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff784067b80 0x7ff78406a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:31.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.666+0000 7ff7a8c00640 1 -- 172.21.15.27:0/4197901823 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7ff794082fb0 con 0x7ff7ac15d190 2024-01-15T18:22:31.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.666+0000 7ff7b0a00640 1 --2- 172.21.15.27:0/4197901823 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff784067b80 0x7ff78406a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:31.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.666+0000 7ff7b0a00640 1 --2- 172.21.15.27:0/4197901823 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff784067b80 0x7ff78406a040 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7ff79c009cf0 tx=0x7ff79c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:31.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.670+0000 7ff7a8c00640 1 -- 172.21.15.27:0/4197901823 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff794051a30 con 0x7ff7ac15d190 2024-01-15T18:22:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.858+0000 7ff7b3400640 1 -- 172.21.15.27:0/4197901823 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff7ac0c5890 con 0x7ff784067b80 2024-01-15T18:22:31.860 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:22:31.861+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:22:31.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:31.866+0000 7ff7a8c00640 1 -- 172.21.15.27:0/4197901823 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff7ac0c5890 con 0x7ff784067b80 2024-01-15T18:22:31.861 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:22:31.861 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:22:31.870 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:33.871 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:22:34.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.034+0000 7fbc63e00640 1 Processor -- start 2024-01-15T18:22:34.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.034+0000 7fbc63e00640 1 -- start start 2024-01-15T18:22:34.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.034+0000 7fbc63e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc5c0cf460 0x7fbc5c0cf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:34.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.034+0000 7fbc63e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fbc5c0b2590 0x7fbc5c0cc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:34.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.034+0000 7fbc63e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc5c0cfd80 con 0x7fbc5c15b4b0 2024-01-15T18:22:34.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.034+0000 7fbc63e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc5c0d19a0 con 0x7fbc5c0b2590 2024-01-15T18:22:34.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.034+0000 7fbc63e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c0d1c80 con 0x7fbc5c0cf460 2024-01-15T18:22:34.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.034+0000 7fbc60a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbc5c15b4b0 0x7fbc5c0cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35176/0 (socket says 172.21.15.27:35176) 2024-01-15T18:22:34.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.034+0000 7fbc60a00640 1 -- 172.21.15.27:0/164527514 learned_addr learned my addr 172.21.15.27:0/164527514 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:22:34.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.034+0000 7fbc5be00640 1 -- 172.21.15.27:0/164527514 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325315357 0 0) 0x7fbc5c0cfd80 con 0x7fbc5c15b4b0 2024-01-15T18:22:34.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.034+0000 7fbc61400640 1 --2- 172.21.15.27:0/164527514 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc5c0cf460 0x7fbc5c0cf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:34.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.034+0000 7fbc5be00640 1 -- 172.21.15.27:0/164527514 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc3c003530 con 0x7fbc5c15b4b0 2024-01-15T18:22:34.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.034+0000 7fbc61e00640 1 --2- 172.21.15.27:0/164527514 >> v2:172.21.15.93:3300/0 conn(0x7fbc5c0b2590 0x7fbc5c0cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:34.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.038+0000 7fbc5be00640 1 -- 172.21.15.27:0/164527514 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 22898293 0 0) 0x7fbc3c003530 con 0x7fbc5c15b4b0 2024-01-15T18:22:34.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.038+0000 7fbc5be00640 1 -- 172.21.15.27:0/164527514 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc5c0cf460 msgr2=0x7fbc5c0cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:34.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.038+0000 7fbc5be00640 1 --2- 172.21.15.27:0/164527514 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc5c0cf460 0x7fbc5c0cf840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:34.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.038+0000 7fbc5be00640 1 -- 172.21.15.27:0/164527514 >> v2:172.21.15.93:3300/0 conn(0x7fbc5c0b2590 msgr2=0x7fbc5c0cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:34.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.038+0000 7fbc5be00640 1 --2- 172.21.15.27:0/164527514 >> v2:172.21.15.93:3300/0 conn(0x7fbc5c0b2590 0x7fbc5c0cc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:34.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.038+0000 7fbc5be00640 1 -- 172.21.15.27:0/164527514 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5c0b07c0 con 0x7fbc5c15b4b0 2024-01-15T18:22:34.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.038+0000 7fbc5be00640 1 -- 172.21.15.27:0/164527514 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fbc44001680 con 0x7fbc5c15b4b0 2024-01-15T18:22:34.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.038+0000 7fbc5be00640 1 -- 172.21.15.27:0/164527514 >> v1:172.21.15.27:6789/0 conn(0x7fbc5c15b4b0 legacy=0x7fbc5c0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:34.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.038+0000 7fbc5be00640 1 --2- 172.21.15.27:0/164527514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc3c0039f0 0x7fbc3c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:34.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.038+0000 7fbc5be00640 1 --2- 172.21.15.27:0/164527514 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc5c0cf460 0x7fbc3c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:34.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.038+0000 7fbc5be00640 1 --2- 172.21.15.27:0/164527514 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc3c004d10 0x7fbc3c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:34.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.038+0000 7fbc5be00640 1 -- 172.21.15.27:0/164527514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c0d19a0 con 0x7fbc3c0039f0 2024-01-15T18:22:34.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.038+0000 7fbc5be00640 1 -- 172.21.15.27:0/164527514 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c0d1c80 con 0x7fbc3c004d10 2024-01-15T18:22:34.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.038+0000 7fbc5be00640 1 -- 172.21.15.27:0/164527514 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3c00b990 con 0x7fbc5c0cf460 2024-01-15T18:22:34.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.038+0000 7fbc61400640 1 --2- 172.21.15.27:0/164527514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc3c0039f0 0x7fbc3c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:34.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.038+0000 7fbc60a00640 1 --2- 172.21.15.27:0/164527514 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc5c0cf460 0x7fbc3c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:34.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.038+0000 7fbc61e00640 1 --2- 172.21.15.27:0/164527514 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc3c004d10 0x7fbc3c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:34.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.038+0000 7fbc61400640 1 -- 172.21.15.27:0/164527514 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc5c0cf460 msgr2=0x7fbc3c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:34.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.038+0000 7fbc61400640 1 --2- 172.21.15.27:0/164527514 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc5c0cf460 0x7fbc3c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:34.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.038+0000 7fbc61400640 1 -- 172.21.15.27:0/164527514 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc3c004d10 msgr2=0x7fbc3c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:34.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.038+0000 7fbc61400640 1 --2- 172.21.15.27:0/164527514 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc3c004d10 0x7fbc3c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:34.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.038+0000 7fbc61400640 1 -- 172.21.15.27:0/164527514 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc3c003530 con 0x7fbc3c0039f0 2024-01-15T18:22:34.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.038+0000 7fbc60a00640 1 --2- 172.21.15.27:0/164527514 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc5c0cf460 0x7fbc3c0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:22:34.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.038+0000 7fbc61400640 1 --2- 172.21.15.27:0/164527514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc3c0039f0 0x7fbc3c003e10 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c009ae0 tx=0x7fbc4c01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5c27cba100e0365 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:34.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.038+0000 7fbc5be00640 1 -- 172.21.15.27:0/164527514 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c0a5620 con 0x7fbc3c0039f0 2024-01-15T18:22:34.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.042+0000 7fbc5be00640 1 -- 172.21.15.27:0/164527514 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc4c0a5780 con 0x7fbc3c0039f0 2024-01-15T18:22:34.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.042+0000 7fbc5be00640 1 -- 172.21.15.27:0/164527514 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c0a5b30 con 0x7fbc3c0039f0 2024-01-15T18:22:34.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.042+0000 7fbc63e00640 1 -- 172.21.15.27:0/164527514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc3c0039f0 msgr2=0x7fbc3c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:34.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.042+0000 7fbc63e00640 1 --2- 172.21.15.27:0/164527514 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc3c0039f0 0x7fbc3c003e10 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c009ae0 tx=0x7fbc4c01ca50 comp rx=0 tx=0).stop 2024-01-15T18:22:34.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.042+0000 7fbc61e00640 1 -- 172.21.15.27:0/164527514 reap_dead start 2024-01-15T18:22:34.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.042+0000 7fbc63e00640 1 -- 172.21.15.27:0/164527514 shutdown_connections 2024-01-15T18:22:34.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.042+0000 7fbc63e00640 1 -- 172.21.15.27:0/164527514 >> 172.21.15.27:0/164527514 conn(0x7fbc5c0d0e80 msgr2=0x7fbc5c0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:22:34.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.042+0000 7fbc63e00640 1 -- 172.21.15.27:0/164527514 shutdown_connections 2024-01-15T18:22:34.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.042+0000 7fbc63e00640 1 -- 172.21.15.27:0/164527514 wait complete. 2024-01-15T18:22:34.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.042+0000 7fbc63e00640 1 Processor -- start 2024-01-15T18:22:34.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.042+0000 7fbc63e00640 1 -- start start 2024-01-15T18:22:34.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.042+0000 7fbc63e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc3c0039f0 0x7fbc5c154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:34.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.042+0000 7fbc63e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc5c1553a0 0x7fbc5c15df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:34.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.042+0000 7fbc63e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc5c15e4c0 0x7fbc5c155d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:34.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.042+0000 7fbc63e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c1625e0 con 0x7fbc5c1553a0 2024-01-15T18:22:34.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.042+0000 7fbc63e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c15b890 con 0x7fbc3c0039f0 2024-01-15T18:22:34.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.042+0000 7fbc63e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c15e960 con 0x7fbc5c15e4c0 2024-01-15T18:22:34.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.046+0000 7fbc60a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc5c1553a0 0x7fbc5c15df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:34.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.046+0000 7fbc60a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc5c1553a0 0x7fbc5c15df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53582/0 (socket says 172.21.15.27:53582) 2024-01-15T18:22:34.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.046+0000 7fbc60a00640 1 -- 172.21.15.27:0/3586006962 learned_addr learned my addr 172.21.15.27:0/3586006962 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:34.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.046+0000 7fbc61400640 1 --2- 172.21.15.27:0/3586006962 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc3c0039f0 0x7fbc5c154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:34.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.046+0000 7fbc61e00640 1 --2- 172.21.15.27:0/3586006962 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc5c15e4c0 0x7fbc5c155d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:34.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.046+0000 7fbc60a00640 1 -- 172.21.15.27:0/3586006962 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc5c15e4c0 msgr2=0x7fbc5c155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:34.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.046+0000 7fbc60a00640 1 --2- 172.21.15.27:0/3586006962 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbc5c15e4c0 0x7fbc5c155d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:34.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.046+0000 7fbc60a00640 1 -- 172.21.15.27:0/3586006962 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc3c0039f0 msgr2=0x7fbc5c154ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:34.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.046+0000 7fbc60a00640 1 --2- 172.21.15.27:0/3586006962 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbc3c0039f0 0x7fbc5c154ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:34.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.046+0000 7fbc60a00640 1 -- 172.21.15.27:0/3586006962 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5c161f90 con 0x7fbc5c1553a0 2024-01-15T18:22:34.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.046+0000 7fbc60a00640 1 --2- 172.21.15.27:0/3586006962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc5c1553a0 0x7fbc5c15df80 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fbc44001000 tx=0x7fbc4400ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:34.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.046+0000 7fbc59600640 1 -- 172.21.15.27:0/3586006962 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc4400ee60 con 0x7fbc5c1553a0 2024-01-15T18:22:34.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.046+0000 7fbc59600640 1 -- 172.21.15.27:0/3586006962 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc4400c8a0 con 0x7fbc5c1553a0 2024-01-15T18:22:34.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.046+0000 7fbc63e00640 1 -- 172.21.15.27:0/3586006962 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc5c162270 con 0x7fbc5c1553a0 2024-01-15T18:22:34.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.046+0000 7fbc59600640 1 -- 172.21.15.27:0/3586006962 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc4400cc10 con 0x7fbc5c1553a0 2024-01-15T18:22:34.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.046+0000 7fbc63e00640 1 -- 172.21.15.27:0/3586006962 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc5c156260 con 0x7fbc5c1553a0 2024-01-15T18:22:34.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.046+0000 7fbc63e00640 1 -- 172.21.15.27:0/3586006962 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc5c161df0 con 0x7fbc5c1553a0 2024-01-15T18:22:34.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.046+0000 7fbc59600640 1 -- 172.21.15.27:0/3586006962 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7fbc4401d460 con 0x7fbc5c1553a0 2024-01-15T18:22:34.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.050+0000 7fbc59600640 1 --2- 172.21.15.27:0/3586006962 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbc34067b80 0x7fbc3406a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:34.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.050+0000 7fbc59600640 1 -- 172.21.15.27:0/3586006962 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fbc44083280 con 0x7fbc5c1553a0 2024-01-15T18:22:34.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.050+0000 7fbc61400640 1 --2- 172.21.15.27:0/3586006962 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbc34067b80 0x7fbc3406a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:34.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.050+0000 7fbc61400640 1 --2- 172.21.15.27:0/3586006962 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbc34067b80 0x7fbc3406a040 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c0af000 tx=0x7fbc4c0046d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:34.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.054+0000 7fbc59600640 1 -- 172.21.15.27:0/3586006962 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbc5c161df0 con 0x7fbc5c1553a0 2024-01-15T18:22:34.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.242+0000 7fbc63e00640 1 -- 172.21.15.27:0/3586006962 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbc5c161df0 con 0x7fbc34067b80 2024-01-15T18:22:34.243 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:22:34.241+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:22:34.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:34.246+0000 7fbc59600640 1 -- 172.21.15.27:0/3586006962 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fbc5c161df0 con 0x7fbc34067b80 2024-01-15T18:22:34.244 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:22:34.244 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:22:34.252 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:36.254 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:22:36.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.418+0000 7fb14ba00640 1 Processor -- start 2024-01-15T18:22:36.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.418+0000 7fb14ba00640 1 -- start start 2024-01-15T18:22:36.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.418+0000 7fb14ba00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fb14415d5d0 0x7fb1440d3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:36.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.418+0000 7fb14ba00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb1440cf890 0x7fb1440cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:36.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.418+0000 7fb14ba00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1440cf2d0 con 0x7fb1440cde00 2024-01-15T18:22:36.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.418+0000 7fb14ba00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1440cf5b0 con 0x7fb1440cf890 2024-01-15T18:22:36.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.418+0000 7fb14ba00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1440d0240 con 0x7fb14415d5d0 2024-01-15T18:22:36.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.418+0000 7fb143e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb1440cf890 0x7fb1440cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:36.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.418+0000 7fb143e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb1440cf890 0x7fb1440cfd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53592/0 (socket says 172.21.15.27:53592) 2024-01-15T18:22:36.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.418+0000 7fb143e00640 1 -- 172.21.15.27:0/3514121807 learned_addr learned my addr 172.21.15.27:0/3514121807 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:36.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.418+0000 7fb149000640 1 --2- 172.21.15.27:0/3514121807 >> v2:172.21.15.93:3300/0 conn(0x7fb14415d5d0 0x7fb1440d3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:36.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.418+0000 7fb143e00640 1 -- 172.21.15.27:0/3514121807 >> v2:172.21.15.93:3300/0 conn(0x7fb14415d5d0 msgr2=0x7fb1440d3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:36.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.418+0000 7fb143e00640 1 --2- 172.21.15.27:0/3514121807 >> v2:172.21.15.93:3300/0 conn(0x7fb14415d5d0 0x7fb1440d3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:36.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.418+0000 7fb143e00640 1 -- 172.21.15.27:0/3514121807 >> v1:172.21.15.93:6789/0 conn(0x7fb1440cde00 legacy=0x7fb1440ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:36.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.418+0000 7fb143e00640 1 -- 172.21.15.27:0/3514121807 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1440b2610 con 0x7fb1440cf890 2024-01-15T18:22:36.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.422+0000 7fb143e00640 1 --2- 172.21.15.27:0/3514121807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440cf890 0x7fb1440cfd00 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fb14408d3e0 tx=0x7fb12c01c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d61032d52ab55e48 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:36.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.422+0000 7fb143400640 1 -- 172.21.15.27:0/3514121807 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c0aa620 con 0x7fb1440cf890 2024-01-15T18:22:36.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.422+0000 7fb143400640 1 -- 172.21.15.27:0/3514121807 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12c0aa780 con 0x7fb1440cf890 2024-01-15T18:22:36.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.422+0000 7fb143400640 1 -- 172.21.15.27:0/3514121807 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c0aab10 con 0x7fb1440cf890 2024-01-15T18:22:36.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.422+0000 7fb14ba00640 1 -- 172.21.15.27:0/3514121807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440cf890 msgr2=0x7fb1440cfd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:36.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.422+0000 7fb14ba00640 1 --2- 172.21.15.27:0/3514121807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440cf890 0x7fb1440cfd00 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fb14408d3e0 tx=0x7fb12c01c4c0 comp rx=0 tx=0).stop 2024-01-15T18:22:36.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.422+0000 7fb14ba00640 1 -- 172.21.15.27:0/3514121807 shutdown_connections 2024-01-15T18:22:36.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.422+0000 7fb14ba00640 1 --2- 172.21.15.27:0/3514121807 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440cf890 0x7fb1440cfd00 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:36.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.422+0000 7fb14ba00640 1 --2- 172.21.15.27:0/3514121807 >> v2:172.21.15.93:3300/0 conn(0x7fb14415d5d0 0x7fb1440d3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:36.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.422+0000 7fb14ba00640 1 -- 172.21.15.27:0/3514121807 >> 172.21.15.27:0/3514121807 conn(0x7fb1440d3010 msgr2=0x7fb1440d3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:22:36.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.426+0000 7fb14ba00640 1 -- 172.21.15.27:0/3514121807 shutdown_connections 2024-01-15T18:22:36.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.426+0000 7fb14ba00640 1 -- 172.21.15.27:0/3514121807 wait complete. 2024-01-15T18:22:36.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.426+0000 7fb14ba00640 1 Processor -- start 2024-01-15T18:22:36.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.426+0000 7fb14ba00640 1 -- start start 2024-01-15T18:22:36.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.426+0000 7fb14ba00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1440cde00 0x7fb1441594a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:36.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.426+0000 7fb14ba00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440cf890 0x7fb14415bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:36.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.426+0000 7fb14ba00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb14415d5d0 0x7fb144152550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:36.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.426+0000 7fb14ba00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14415e9c0 con 0x7fb1440cf890 2024-01-15T18:22:36.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.426+0000 7fb14ba00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14415ddb0 con 0x7fb1440cde00 2024-01-15T18:22:36.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.426+0000 7fb14ba00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb14415e090 con 0x7fb14415d5d0 2024-01-15T18:22:36.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.426+0000 7fb143e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440cf890 0x7fb14415bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:36.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.426+0000 7fb143e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440cf890 0x7fb14415bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53600/0 (socket says 172.21.15.27:53600) 2024-01-15T18:22:36.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.426+0000 7fb143e00640 1 -- 172.21.15.27:0/3649899134 learned_addr learned my addr 172.21.15.27:0/3649899134 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:36.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.426+0000 7fb149000640 1 --2- 172.21.15.27:0/3649899134 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1440cde00 0x7fb1441594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:36.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.426+0000 7fb149a00640 1 --2- 172.21.15.27:0/3649899134 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb14415d5d0 0x7fb144152550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:36.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.426+0000 7fb143e00640 1 -- 172.21.15.27:0/3649899134 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb14415d5d0 msgr2=0x7fb144152550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:36.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.426+0000 7fb143e00640 1 --2- 172.21.15.27:0/3649899134 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb14415d5d0 0x7fb144152550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:36.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.426+0000 7fb143e00640 1 -- 172.21.15.27:0/3649899134 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1440cde00 msgr2=0x7fb1441594a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:36.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.426+0000 7fb143e00640 1 --2- 172.21.15.27:0/3649899134 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb1440cde00 0x7fb1441594a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:36.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.426+0000 7fb143e00640 1 -- 172.21.15.27:0/3649899134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb12c0094d0 con 0x7fb1440cf890 2024-01-15T18:22:36.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.426+0000 7fb143e00640 1 --2- 172.21.15.27:0/3649899134 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440cf890 0x7fb14415bd50 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7fb12c01a0f0 tx=0x7fb12c01ad00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:36.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.430+0000 7fb140c00640 1 -- 172.21.15.27:0/3649899134 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c0bd3f0 con 0x7fb1440cf890 2024-01-15T18:22:36.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.430+0000 7fb140c00640 1 -- 172.21.15.27:0/3649899134 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12c0bd550 con 0x7fb1440cf890 2024-01-15T18:22:36.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.430+0000 7fb14ba00640 1 -- 172.21.15.27:0/3649899134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb14415e370 con 0x7fb1440cf890 2024-01-15T18:22:36.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.430+0000 7fb140c00640 1 -- 172.21.15.27:0/3649899134 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c0bd900 con 0x7fb1440cf890 2024-01-15T18:22:36.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.430+0000 7fb14ba00640 1 -- 172.21.15.27:0/3649899134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb14415da40 con 0x7fb1440cf890 2024-01-15T18:22:36.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.430+0000 7fb14ba00640 1 -- 172.21.15.27:0/3649899134 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb14404e850 con 0x7fb1440cf890 2024-01-15T18:22:36.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.430+0000 7fb140c00640 1 -- 172.21.15.27:0/3649899134 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7fb12c0b33f0 con 0x7fb1440cf890 2024-01-15T18:22:36.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.430+0000 7fb140c00640 1 --2- 172.21.15.27:0/3649899134 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb11c067b80 0x7fb11c06a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:36.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.430+0000 7fb140c00640 1 -- 172.21.15.27:0/3649899134 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fb12c11d840 con 0x7fb1440cf890 2024-01-15T18:22:36.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.430+0000 7fb149000640 1 --2- 172.21.15.27:0/3649899134 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb11c067b80 0x7fb11c06a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:36.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.434+0000 7fb149000640 1 --2- 172.21.15.27:0/3649899134 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb11c067b80 0x7fb11c06a040 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fb1340079c0 tx=0x7fb134008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:36.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.434+0000 7fb140c00640 1 -- 172.21.15.27:0/3649899134 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb12c0ec380 con 0x7fb1440cf890 2024-01-15T18:22:36.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.622+0000 7fb14ba00640 1 -- 172.21.15.27:0/3649899134 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb1440b4570 con 0x7fb11c067b80 2024-01-15T18:22:36.626 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:22:36.625+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:22:36.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:36.630+0000 7fb140c00640 1 -- 172.21.15.27:0/3649899134 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb1440b4570 con 0x7fb11c067b80 2024-01-15T18:22:36.626 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:22:36.627 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:22:36.634 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:38.636 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:22:38.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.802+0000 7f32d0e00640 1 Processor -- start 2024-01-15T18:22:38.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.802+0000 7f32d0e00640 1 -- start start 2024-01-15T18:22:38.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.802+0000 7f32d0e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32cc0ce760 0x7f32cc0b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:38.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.802+0000 7f32d0e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32cc0712b0 con 0x7f32cc0b1db0 2024-01-15T18:22:38.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.802+0000 7f32d0e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32cc0ccfb0 con 0x7f32cc06c430 2024-01-15T18:22:38.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.802+0000 7f32d0e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc0cd290 con 0x7f32cc0ce760 2024-01-15T18:22:38.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.802+0000 7f32c9400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f32cc0b1db0 0x7f32cc0ce050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35180/0 (socket says 172.21.15.27:35180) 2024-01-15T18:22:38.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.802+0000 7f32c9400640 1 -- 172.21.15.27:0/1951157658 learned_addr learned my addr 172.21.15.27:0/1951157658 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:22:38.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.802+0000 7f32ca800640 1 --2- 172.21.15.27:0/1951157658 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32cc0ce760 0x7f32cc0b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:38.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.802+0000 7f32c8a00640 1 -- 172.21.15.27:0/1951157658 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370838111 0 0) 0x7f32cc0712b0 con 0x7f32cc0b1db0 2024-01-15T18:22:38.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.802+0000 7f32c8a00640 1 -- 172.21.15.27:0/1951157658 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32ac003450 con 0x7f32cc0b1db0 2024-01-15T18:22:38.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.802+0000 7f32c8a00640 1 -- 172.21.15.27:0/1951157658 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793656440 0 0) 0x7f32cc0ccfb0 con 0x7f32cc06c430 2024-01-15T18:22:38.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.802+0000 7f32c8a00640 1 -- 172.21.15.27:0/1951157658 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32cc0712b0 con 0x7f32cc06c430 2024-01-15T18:22:38.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.802+0000 7f32c8a00640 1 -- 172.21.15.27:0/1951157658 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3227096397 0 0) 0x7f32ac003450 con 0x7f32cc0b1db0 2024-01-15T18:22:38.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.802+0000 7f32ca800640 1 -- 172.21.15.27:0/1951157658 >> v1:172.21.15.93:6789/0 conn(0x7f32cc06c430 legacy=0x7f32cc070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:38.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.802+0000 7f32ca800640 1 -- 172.21.15.27:0/1951157658 >> v1:172.21.15.27:6789/0 conn(0x7f32cc0b1db0 legacy=0x7f32cc0ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:38.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.802+0000 7f32ca800640 1 -- 172.21.15.27:0/1951157658 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32cc0cd570 con 0x7f32cc0ce760 2024-01-15T18:22:38.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.806+0000 7f32ca800640 1 --2- 172.21.15.27:0/1951157658 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32cc0ce760 0x7f32cc0b2240 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f32c4009df0 tx=0x7f32c401c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2983cdc572663979 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:38.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.806+0000 7f32c8a00640 1 -- 172.21.15.27:0/1951157658 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b4001680 con 0x7f32cc0ce760 2024-01-15T18:22:38.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.806+0000 7f32c8a00640 1 -- 172.21.15.27:0/1951157658 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c40a55d0 con 0x7f32cc0ce760 2024-01-15T18:22:38.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.806+0000 7f32c8a00640 1 -- 172.21.15.27:0/1951157658 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c40a5960 con 0x7f32cc0ce760 2024-01-15T18:22:38.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.806+0000 7f32d0e00640 1 -- 172.21.15.27:0/1951157658 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32cc0ce760 msgr2=0x7f32cc0b2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:38.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.806+0000 7f32d0e00640 1 --2- 172.21.15.27:0/1951157658 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32cc0ce760 0x7f32cc0b2240 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f32c4009df0 tx=0x7f32c401c7d0 comp rx=0 tx=0).stop 2024-01-15T18:22:38.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.806+0000 7f32d0e00640 1 -- 172.21.15.27:0/1951157658 shutdown_connections 2024-01-15T18:22:38.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.806+0000 7f32d0e00640 1 --2- 172.21.15.27:0/1951157658 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32cc0ce760 0x7f32cc0b2240 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:38.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.806+0000 7f32d0e00640 1 -- 172.21.15.27:0/1951157658 >> 172.21.15.27:0/1951157658 conn(0x7f32cc0cb700 msgr2=0x7f32cc0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:22:38.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.806+0000 7f32d0e00640 1 -- 172.21.15.27:0/1951157658 shutdown_connections 2024-01-15T18:22:38.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.806+0000 7f32d0e00640 1 -- 172.21.15.27:0/1951157658 wait complete. 2024-01-15T18:22:38.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.810+0000 7f32d0e00640 1 Processor -- start 2024-01-15T18:22:38.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.810+0000 7f32d0e00640 1 -- start start 2024-01-15T18:22:38.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.810+0000 7f32d0e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32cc06c430 0x7f32cc0b3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:38.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.810+0000 7f32d0e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32cc0b1db0 0x7f32cc1460f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:38.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.810+0000 7f32d0e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32cc0ce760 0x7f32cc14f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:38.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.810+0000 7f32d0e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc1530f0 con 0x7f32cc0b1db0 2024-01-15T18:22:38.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.810+0000 7f32d0e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc1533d0 con 0x7f32cc06c430 2024-01-15T18:22:38.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.810+0000 7f32d0e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc1536b0 con 0x7f32cc0ce760 2024-01-15T18:22:38.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.810+0000 7f32c9400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32cc0b1db0 0x7f32cc1460f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:38.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.810+0000 7f32c9400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32cc0b1db0 0x7f32cc1460f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53604/0 (socket says 172.21.15.27:53604) 2024-01-15T18:22:38.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.810+0000 7f32c9400640 1 -- 172.21.15.27:0/2679891970 learned_addr learned my addr 172.21.15.27:0/2679891970 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:38.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.810+0000 7f32ca800640 1 --2- 172.21.15.27:0/2679891970 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32cc0ce760 0x7f32cc14f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:38.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.810+0000 7f32c9e00640 1 --2- 172.21.15.27:0/2679891970 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32cc06c430 0x7f32cc0b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:38.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.810+0000 7f32c9400640 1 -- 172.21.15.27:0/2679891970 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32cc0ce760 msgr2=0x7f32cc14f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:38.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.810+0000 7f32c9400640 1 --2- 172.21.15.27:0/2679891970 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f32cc0ce760 0x7f32cc14f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:38.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.810+0000 7f32c9400640 1 -- 172.21.15.27:0/2679891970 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32cc06c430 msgr2=0x7f32cc0b3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:38.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.810+0000 7f32c9400640 1 --2- 172.21.15.27:0/2679891970 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f32cc06c430 0x7f32cc0b3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:38.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.810+0000 7f32c9400640 1 -- 172.21.15.27:0/2679891970 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c4009a70 con 0x7f32cc0b1db0 2024-01-15T18:22:38.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.810+0000 7f32c9400640 1 --2- 172.21.15.27:0/2679891970 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32cc0b1db0 0x7f32cc1460f0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f32b400c780 tx=0x7f32b400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:38.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.810+0000 7f32c2000640 1 -- 172.21.15.27:0/2679891970 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b4005cf0 con 0x7f32cc0b1db0 2024-01-15T18:22:38.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.810+0000 7f32c2000640 1 -- 172.21.15.27:0/2679891970 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b4005e50 con 0x7f32cc0b1db0 2024-01-15T18:22:38.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.810+0000 7f32c2000640 1 -- 172.21.15.27:0/2679891970 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b40072b0 con 0x7f32cc0b1db0 2024-01-15T18:22:38.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.810+0000 7f32d0e00640 1 -- 172.21.15.27:0/2679891970 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32cc153990 con 0x7f32cc0b1db0 2024-01-15T18:22:38.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.810+0000 7f32d0e00640 1 -- 172.21.15.27:0/2679891970 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32cc0b3a80 con 0x7f32cc0b1db0 2024-01-15T18:22:38.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.814+0000 7f32d0e00640 1 -- 172.21.15.27:0/2679891970 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32cc073bd0 con 0x7f32cc0b1db0 2024-01-15T18:22:38.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.814+0000 7f32c2000640 1 -- 172.21.15.27:0/2679891970 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 43) v1 ==== 87843+0+0 (secure 0 0 0) 0x7f32b4007410 con 0x7f32cc0b1db0 2024-01-15T18:22:38.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.814+0000 7f32c2000640 1 --2- 172.21.15.27:0/2679891970 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f32a4067b80 0x7f32a406a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:38.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.814+0000 7f32c2000640 1 -- 172.21.15.27:0/2679891970 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f32b4082b60 con 0x7f32cc0b1db0 2024-01-15T18:22:38.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.814+0000 7f32c9e00640 1 --2- 172.21.15.27:0/2679891970 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f32a4067b80 0x7f32a406a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:38.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.814+0000 7f32c9e00640 1 --2- 172.21.15.27:0/2679891970 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f32a4067b80 0x7f32a406a040 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f32bc0011f0 tx=0x7f32bc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:38.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:38.818+0000 7f32c2000640 1 -- 172.21.15.27:0/2679891970 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f32b4051520 con 0x7f32cc0b1db0 2024-01-15T18:22:39.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:39.006+0000 7f32d0e00640 1 -- 172.21.15.27:0/2679891970 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f32cc153eb0 con 0x7f32a4067b80 2024-01-15T18:22:39.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:39.014+0000 7f32c2000640 1 -- 172.21.15.27:0/2679891970 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f32cc153eb0 con 0x7f32a4067b80 2024-01-15T18:22:39.011 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:22:39.009+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:22:39.011 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:22:39.011 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:22:39.019 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:41.020 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:22:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.186+0000 7f844c800640 1 Processor -- start 2024-01-15T18:22:41.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.186+0000 7f844c800640 1 -- start start 2024-01-15T18:22:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.186+0000 7f844c800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f84440cba60 0x7f84440b0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.186+0000 7f844c800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84440cefa0 con 0x7f84440d1a40 2024-01-15T18:22:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.186+0000 7f844c800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84440ce240 con 0x7f84440cf450 2024-01-15T18:22:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.186+0000 7f844c800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84440ccd70 con 0x7f84440cba60 2024-01-15T18:22:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.190+0000 7f844a800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f84440cba60 0x7f84440b0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.190+0000 7f8449400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f84440d1a40 0x7f84440b0650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35192/0 (socket says 172.21.15.27:35192) 2024-01-15T18:22:41.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.190+0000 7f844a800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f84440cba60 0x7f84440b0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53620/0 (socket says 172.21.15.27:53620) 2024-01-15T18:22:41.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.190+0000 7f8449400640 1 -- 172.21.15.27:0/1405057129 learned_addr learned my addr 172.21.15.27:0/1405057129 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:22:41.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.190+0000 7f844a800640 1 -- 172.21.15.27:0/1405057129 >> v1:172.21.15.93:6789/0 conn(0x7f84440cf450 legacy=0x7f84440cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:41.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.190+0000 7f844a800640 1 -- 172.21.15.27:0/1405057129 >> v1:172.21.15.27:6789/0 conn(0x7f84440d1a40 legacy=0x7f84440b0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:41.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.190+0000 7f844a800640 1 -- 172.21.15.27:0/1405057129 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84440cc1f0 con 0x7f84440cba60 2024-01-15T18:22:41.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.190+0000 7f8448a00640 1 -- 172.21.15.27:0/1405057129 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019078862 0 0) 0x7f84440cefa0 con 0x7f84440d1a40 2024-01-15T18:22:41.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.190+0000 7f844a800640 1 --2- 172.21.15.27:0/1405057129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84440cba60 0x7f84440b0d60 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f843c009d90 tx=0x7f843c01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=212915a99bf9770 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:41.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.190+0000 7f8448a00640 1 -- 172.21.15.27:0/1405057129 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c0a57d0 con 0x7f84440cba60 2024-01-15T18:22:41.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.190+0000 7f8448a00640 1 -- 172.21.15.27:0/1405057129 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843c0a5930 con 0x7f84440cba60 2024-01-15T18:22:41.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.190+0000 7f8448a00640 1 -- 172.21.15.27:0/1405057129 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c0a5cc0 con 0x7f84440cba60 2024-01-15T18:22:41.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.194+0000 7f844c800640 1 -- 172.21.15.27:0/1405057129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84440cba60 msgr2=0x7f84440b0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:41.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.194+0000 7f844c800640 1 --2- 172.21.15.27:0/1405057129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84440cba60 0x7f84440b0d60 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f843c009d90 tx=0x7f843c01c910 comp rx=0 tx=0).stop 2024-01-15T18:22:41.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.194+0000 7f844c800640 1 -- 172.21.15.27:0/1405057129 shutdown_connections 2024-01-15T18:22:41.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.194+0000 7f844c800640 1 --2- 172.21.15.27:0/1405057129 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84440cba60 0x7f84440b0d60 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:41.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.194+0000 7f844c800640 1 -- 172.21.15.27:0/1405057129 >> 172.21.15.27:0/1405057129 conn(0x7f84440d0e70 msgr2=0x7f84440d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:22:41.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.194+0000 7f844c800640 1 -- 172.21.15.27:0/1405057129 shutdown_connections 2024-01-15T18:22:41.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.194+0000 7f844c800640 1 -- 172.21.15.27:0/1405057129 wait complete. 2024-01-15T18:22:41.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.194+0000 7f844c800640 1 Processor -- start 2024-01-15T18:22:41.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.194+0000 7f844c800640 1 -- start start 2024-01-15T18:22:41.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.194+0000 7f844c800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f84440cba60 0x7f844414f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:41.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.194+0000 7f844c800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84440cf450 0x7f8444165f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:41.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.194+0000 7f844c800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f84440d1a40 0x7f8444166440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:41.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.194+0000 7f844c800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844414fb40 con 0x7f84440cf450 2024-01-15T18:22:41.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.194+0000 7f844c800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84440b6790 con 0x7f84440cba60 2024-01-15T18:22:41.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.194+0000 7f844c800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8444164300 con 0x7f84440d1a40 2024-01-15T18:22:41.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.194+0000 7f8449400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84440cf450 0x7f8444165f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:41.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.194+0000 7f8449400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84440cf450 0x7f8444165f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53632/0 (socket says 172.21.15.27:53632) 2024-01-15T18:22:41.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.194+0000 7f8449400640 1 -- 172.21.15.27:0/4090672973 learned_addr learned my addr 172.21.15.27:0/4090672973 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:41.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.194+0000 7f8449e00640 1 --2- 172.21.15.27:0/4090672973 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f84440cba60 0x7f844414f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:41.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.198+0000 7f844a800640 1 --2- 172.21.15.27:0/4090672973 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f84440d1a40 0x7f8444166440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:41.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.198+0000 7f8449400640 1 -- 172.21.15.27:0/4090672973 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f84440d1a40 msgr2=0x7f8444166440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:41.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.198+0000 7f8449400640 1 --2- 172.21.15.27:0/4090672973 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f84440d1a40 0x7f8444166440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:41.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.198+0000 7f8449400640 1 -- 172.21.15.27:0/4090672973 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f84440cba60 msgr2=0x7f844414f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:41.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.198+0000 7f8449400640 1 --2- 172.21.15.27:0/4090672973 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f84440cba60 0x7f844414f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:41.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.198+0000 7f8449400640 1 -- 172.21.15.27:0/4090672973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843c009a70 con 0x7f84440cf450 2024-01-15T18:22:41.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.198+0000 7f844a800640 1 --2- 172.21.15.27:0/4090672973 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f84440d1a40 0x7f8444166440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:22:41.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.198+0000 7f8449400640 1 --2- 172.21.15.27:0/4090672973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84440cf450 0x7f8444165f00 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f842c0011f0 tx=0x7f842c00c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:41.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.198+0000 7f8442000640 1 -- 172.21.15.27:0/4090672973 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842c0074e0 con 0x7f84440cf450 2024-01-15T18:22:41.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.198+0000 7f8442000640 1 -- 172.21.15.27:0/4090672973 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f842c007640 con 0x7f84440cf450 2024-01-15T18:22:41.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.198+0000 7f8442000640 1 -- 172.21.15.27:0/4090672973 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f842c004a10 con 0x7f84440cf450 2024-01-15T18:22:41.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.198+0000 7f844c800640 1 -- 172.21.15.27:0/4090672973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8444166980 con 0x7f84440cf450 2024-01-15T18:22:41.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.198+0000 7f844c800640 1 -- 172.21.15.27:0/4090672973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844406d1c0 con 0x7f84440cf450 2024-01-15T18:22:41.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.198+0000 7f844c800640 1 -- 172.21.15.27:0/4090672973 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f844406d020 con 0x7f84440cf450 2024-01-15T18:22:41.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.198+0000 7f8442000640 1 -- 172.21.15.27:0/4090672973 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 44) v1 ==== 87792+0+0 (secure 0 0 0) 0x7f842c004b70 con 0x7f84440cf450 2024-01-15T18:22:41.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.198+0000 7f8442000640 1 --2- 172.21.15.27:0/4090672973 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f841c067af0 0x7f841c069fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:41.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.198+0000 7f8442000640 1 -- 172.21.15.27:0/4090672973 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f842c082490 con 0x7f84440cf450 2024-01-15T18:22:41.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.202+0000 7f8449e00640 1 --2- 172.21.15.27:0/4090672973 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f841c067af0 0x7f841c069fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:41.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.202+0000 7f8449e00640 1 --2- 172.21.15.27:0/4090672973 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f841c067af0 0x7f841c069fb0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f8434007fd0 tx=0x7f8434007e40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:41.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.202+0000 7f8442000640 1 -- 172.21.15.27:0/4090672973 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f844406d020 con 0x7f84440cf450 2024-01-15T18:22:41.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.250+0000 7f8442000640 1 -- 172.21.15.27:0/4090672973 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f842c051230 con 0x7f84440cf450 2024-01-15T18:22:41.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.394+0000 7f844c800640 1 -- 172.21.15.27:0/4090672973 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f84440cbe90 con 0x7f841c067af0 2024-01-15T18:22:41.398 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:22:41.397+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:22:41.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:41.402+0000 7f8442000640 1 -- 172.21.15.27:0/4090672973 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8434004c00 con 0x7f841c067af0 2024-01-15T18:22:41.399 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:22:41.399 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:22:41.407 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:43.409 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:22:43.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.562+0000 7ff3f7800640 1 Processor -- start 2024-01-15T18:22:43.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.562+0000 7ff3f7800640 1 -- start start 2024-01-15T18:22:43.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.562+0000 7ff3f7800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff3f015ad20 0x7ff3f015b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:43.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.562+0000 7ff3f7800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff3f015d900 0x7ff3f015dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:43.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.562+0000 7ff3f7800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f015a610 con 0x7ff3f0070ee0 2024-01-15T18:22:43.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.562+0000 7ff3f7800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3f015e3c0 con 0x7ff3f015d900 2024-01-15T18:22:43.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.562+0000 7ff3f7800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f015e6a0 con 0x7ff3f015ad20 2024-01-15T18:22:43.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.566+0000 7ff3f5800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff3f0070ee0 0x7ff3f006d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:35194/0 (socket says 172.21.15.27:35194) 2024-01-15T18:22:43.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.566+0000 7ff3f5800640 1 -- 172.21.15.27:0/4184213190 learned_addr learned my addr 172.21.15.27:0/4184213190 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:22:43.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.566+0000 7ff3efe00640 1 --2- 172.21.15.27:0/4184213190 >> v2:172.21.15.93:3300/0 conn(0x7ff3f015d900 0x7ff3f015dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:43.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.566+0000 7ff3f4e00640 1 --2- 172.21.15.27:0/4184213190 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff3f015ad20 0x7ff3f015b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:43.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.566+0000 7ff3ef400640 1 -- 172.21.15.27:0/4184213190 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4164418775 0 0) 0x7ff3f015a610 con 0x7ff3f0070ee0 2024-01-15T18:22:43.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.566+0000 7ff3ef400640 1 -- 172.21.15.27:0/4184213190 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d0003530 con 0x7ff3f0070ee0 2024-01-15T18:22:43.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.566+0000 7ff3f4e00640 1 -- 172.21.15.27:0/4184213190 >> v2:172.21.15.93:3300/0 conn(0x7ff3f015d900 msgr2=0x7ff3f015dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:43.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.566+0000 7ff3f4e00640 1 --2- 172.21.15.27:0/4184213190 >> v2:172.21.15.93:3300/0 conn(0x7ff3f015d900 0x7ff3f015dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:43.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.566+0000 7ff3f4e00640 1 -- 172.21.15.27:0/4184213190 >> v1:172.21.15.27:6789/0 conn(0x7ff3f0070ee0 legacy=0x7ff3f006d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:43.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.566+0000 7ff3ef400640 1 -- 172.21.15.27:0/4184213190 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2747617668 0 0) 0x7ff3d0003530 con 0x7ff3f0070ee0 2024-01-15T18:22:43.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.566+0000 7ff3f4e00640 1 -- 172.21.15.27:0/4184213190 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f015b6e0 con 0x7ff3f015ad20 2024-01-15T18:22:43.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.566+0000 7ff3f4e00640 1 --2- 172.21.15.27:0/4184213190 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff3f015ad20 0x7ff3f015b100 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00097c0 tx=0x7ff3e001c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=980f20c832287afa server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:43.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.570+0000 7ff3ef400640 1 -- 172.21.15.27:0/4184213190 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e00ab670 con 0x7ff3f015ad20 2024-01-15T18:22:43.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.570+0000 7ff3ef400640 1 -- 172.21.15.27:0/4184213190 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e00ab7d0 con 0x7ff3f015ad20 2024-01-15T18:22:43.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.570+0000 7ff3ef400640 1 -- 172.21.15.27:0/4184213190 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3e00abb60 con 0x7ff3f015ad20 2024-01-15T18:22:43.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.570+0000 7ff3f7800640 1 -- 172.21.15.27:0/4184213190 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff3f015ad20 msgr2=0x7ff3f015b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:43.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.570+0000 7ff3f7800640 1 --2- 172.21.15.27:0/4184213190 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff3f015ad20 0x7ff3f015b100 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00097c0 tx=0x7ff3e001c450 comp rx=0 tx=0).stop 2024-01-15T18:22:43.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.570+0000 7ff3f7800640 1 -- 172.21.15.27:0/4184213190 shutdown_connections 2024-01-15T18:22:43.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.570+0000 7ff3f7800640 1 --2- 172.21.15.27:0/4184213190 >> v2:172.21.15.93:3300/0 conn(0x7ff3f015d900 0x7ff3f015dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:43.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.570+0000 7ff3f7800640 1 --2- 172.21.15.27:0/4184213190 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff3f015ad20 0x7ff3f015b100 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:43.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.570+0000 7ff3f7800640 1 -- 172.21.15.27:0/4184213190 >> 172.21.15.27:0/4184213190 conn(0x7ff3f00cb700 msgr2=0x7ff3f00cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:22:43.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.570+0000 7ff3f7800640 1 -- 172.21.15.27:0/4184213190 shutdown_connections 2024-01-15T18:22:43.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.570+0000 7ff3f7800640 1 -- 172.21.15.27:0/4184213190 wait complete. 2024-01-15T18:22:43.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.570+0000 7ff3f7800640 1 Processor -- start 2024-01-15T18:22:43.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.570+0000 7ff3f7800640 1 -- start start 2024-01-15T18:22:43.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.570+0000 7ff3f7800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3f0070ee0 0x7ff3f01572d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:43.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.574+0000 7ff3f7800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff3f015ad20 0x7ff3f0155100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:43.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.574+0000 7ff3f7800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3f015d900 0x7ff3f0152e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:43.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.574+0000 7ff3f7800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f00b3850 con 0x7ff3f015d900 2024-01-15T18:22:43.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.574+0000 7ff3f7800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f0157810 con 0x7ff3f0070ee0 2024-01-15T18:22:43.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.574+0000 7ff3f7800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f0156f40 con 0x7ff3f015ad20 2024-01-15T18:22:43.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.574+0000 7ff3f5800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3f015d900 0x7ff3f0152e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:43.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.574+0000 7ff3f5800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3f015d900 0x7ff3f0152e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53636/0 (socket says 172.21.15.27:53636) 2024-01-15T18:22:43.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.574+0000 7ff3f5800640 1 -- 172.21.15.27:0/2782799111 learned_addr learned my addr 172.21.15.27:0/2782799111 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:43.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.574+0000 7ff3efe00640 1 --2- 172.21.15.27:0/2782799111 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff3f015ad20 0x7ff3f0155100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:43.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.574+0000 7ff3f4e00640 1 --2- 172.21.15.27:0/2782799111 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3f0070ee0 0x7ff3f01572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:43.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.574+0000 7ff3efe00640 1 -- 172.21.15.27:0/2782799111 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3f0070ee0 msgr2=0x7ff3f01572d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:43.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.574+0000 7ff3efe00640 1 --2- 172.21.15.27:0/2782799111 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff3f0070ee0 0x7ff3f01572d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:43.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.574+0000 7ff3efe00640 1 -- 172.21.15.27:0/2782799111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3f015d900 msgr2=0x7ff3f0152e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:43.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.574+0000 7ff3efe00640 1 --2- 172.21.15.27:0/2782799111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3f015d900 0x7ff3f0152e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:43.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.574+0000 7ff3efe00640 1 -- 172.21.15.27:0/2782799111 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e0009420 con 0x7ff3f015ad20 2024-01-15T18:22:43.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.574+0000 7ff3f5800640 1 --2- 172.21.15.27:0/2782799111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3f015d900 0x7ff3f0152e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:22:43.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.574+0000 7ff3efe00640 1 --2- 172.21.15.27:0/2782799111 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff3f015ad20 0x7ff3f0155100 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff3d800e870 tx=0x7ff3d800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:43.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.574+0000 7ff3ecc00640 1 -- 172.21.15.27:0/2782799111 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d800ee60 con 0x7ff3f015ad20 2024-01-15T18:22:43.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.574+0000 7ff3ecc00640 1 -- 172.21.15.27:0/2782799111 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3d800c940 con 0x7ff3f015ad20 2024-01-15T18:22:43.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.574+0000 7ff3ecc00640 1 -- 172.21.15.27:0/2782799111 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3d800ccd0 con 0x7ff3f015ad20 2024-01-15T18:22:43.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.574+0000 7ff3f7800640 1 -- 172.21.15.27:0/2782799111 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f0155640 con 0x7ff3f015ad20 2024-01-15T18:22:43.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.574+0000 7ff3f7800640 1 -- 172.21.15.27:0/2782799111 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f00b2eb0 con 0x7ff3f015ad20 2024-01-15T18:22:43.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.574+0000 7ff3f7800640 1 -- 172.21.15.27:0/2782799111 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f004e850 con 0x7ff3f015ad20 2024-01-15T18:22:43.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.574+0000 7ff3ecc00640 1 -- 172.21.15.27:0/2782799111 <== mon.2 v2:172.21.15.93:3301/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7ff3d803aab0 con 0x7ff3f015ad20 2024-01-15T18:22:43.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.578+0000 7ff3ecc00640 1 --2- 172.21.15.27:0/2782799111 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff3c8067a90 0x7ff3c8069f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:43.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.578+0000 7ff3ecc00640 1 -- 172.21.15.27:0/2782799111 <== mon.2 v2:172.21.15.93:3301/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7ff3d8083640 con 0x7ff3f015ad20 2024-01-15T18:22:43.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.578+0000 7ff3f4e00640 1 --2- 172.21.15.27:0/2782799111 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff3c8067a90 0x7ff3c8069f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:43.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.578+0000 7ff3f4e00640 1 --2- 172.21.15.27:0/2782799111 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff3c8067a90 0x7ff3c8069f50 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00af000 tx=0x7ff3e00a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:43.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.582+0000 7ff3ecc00640 1 -- 172.21.15.27:0/2782799111 <== mon.2 v2:172.21.15.93:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff3d8052030 con 0x7ff3f015ad20 2024-01-15T18:22:43.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.766+0000 7ff3f7800640 1 -- 172.21.15.27:0/2782799111 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff3f015a340 con 0x7ff3c8067a90 2024-01-15T18:22:43.768 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:22:43.769+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:22:43.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:43.774+0000 7ff3ecc00640 1 -- 172.21.15.27:0/2782799111 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff3f015a340 con 0x7ff3c8067a90 2024-01-15T18:22:43.769 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:22:43.770 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:22:43.777 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:45.779 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:22:45.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.941+0000 7f5925000640 1 Processor -- start 2024-01-15T18:22:45.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.941+0000 7f5925000640 1 -- start start 2024-01-15T18:22:45.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.941+0000 7f5925000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f592015f550 0x7f59200cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:45.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.941+0000 7f5925000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59200cf110 0x7f59200701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:45.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.941+0000 7f5925000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59200b1cb0 con 0x7f5920158900 2024-01-15T18:22:45.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.941+0000 7f5925000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59200cf580 con 0x7f592015f550 2024-01-15T18:22:45.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.941+0000 7f5925000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f592015fa00 con 0x7f59200cf110 2024-01-15T18:22:45.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.941+0000 7f591d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f592015f550 0x7f59200cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:45.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.941+0000 7f591d400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f592015f550 0x7f59200cebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58676/0 (socket says 172.21.15.27:58676) 2024-01-15T18:22:45.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.941+0000 7f591d400640 1 -- 172.21.15.27:0/2471492724 learned_addr learned my addr 172.21.15.27:0/2471492724 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:45.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.945+0000 7f591e800640 1 --2- 172.21.15.27:0/2471492724 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59200cf110 0x7f59200701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:45.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.945+0000 7f591d400640 1 -- 172.21.15.27:0/2471492724 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59200cf110 msgr2=0x7f59200701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:45.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.945+0000 7f591d400640 1 --2- 172.21.15.27:0/2471492724 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59200cf110 0x7f59200701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:45.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.945+0000 7f591d400640 1 -- 172.21.15.27:0/2471492724 >> v1:172.21.15.93:6789/0 conn(0x7f5920158900 legacy=0x7f592015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:45.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.945+0000 7f591d400640 1 -- 172.21.15.27:0/2471492724 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59200b1f00 con 0x7f592015f550 2024-01-15T18:22:45.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.945+0000 7f591d400640 1 --2- 172.21.15.27:0/2471492724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f592015f550 0x7f59200cebd0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f5908009870 tx=0x7f590801c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3ad74fe3304d290 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:45.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.945+0000 7f591ca00640 1 -- 172.21.15.27:0/2471492724 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5920070f30 con 0x7f592015f550 2024-01-15T18:22:45.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.945+0000 7f591ca00640 1 -- 172.21.15.27:0/2471492724 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59080aa670 con 0x7f592015f550 2024-01-15T18:22:45.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.945+0000 7f591ca00640 1 -- 172.21.15.27:0/2471492724 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59080aaa00 con 0x7f592015f550 2024-01-15T18:22:45.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f5925000640 1 -- 172.21.15.27:0/2471492724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f592015f550 msgr2=0x7f59200cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:45.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f5925000640 1 --2- 172.21.15.27:0/2471492724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f592015f550 0x7f59200cebd0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f5908009870 tx=0x7f590801c510 comp rx=0 tx=0).stop 2024-01-15T18:22:45.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f5925000640 1 -- 172.21.15.27:0/2471492724 shutdown_connections 2024-01-15T18:22:45.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f5925000640 1 --2- 172.21.15.27:0/2471492724 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59200cf110 0x7f59200701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:45.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f5925000640 1 --2- 172.21.15.27:0/2471492724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f592015f550 0x7f59200cebd0 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:45.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f5925000640 1 -- 172.21.15.27:0/2471492724 >> 172.21.15.27:0/2471492724 conn(0x7f59200cb700 msgr2=0x7f59200cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:22:45.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f5925000640 1 -- 172.21.15.27:0/2471492724 shutdown_connections 2024-01-15T18:22:45.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f5925000640 1 -- 172.21.15.27:0/2471492724 wait complete. 2024-01-15T18:22:45.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f5925000640 1 Processor -- start 2024-01-15T18:22:45.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f5925000640 1 -- start start 2024-01-15T18:22:45.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f5925000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59200cf110 0x7f592014c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:45.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f5925000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5920158900 0x7f59201540a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:45.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f5925000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f592015f550 0x7f592014d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:45.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f5925000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5920153c80 con 0x7f5920158900 2024-01-15T18:22:45.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f5925000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5920153a80 con 0x7f592015f550 2024-01-15T18:22:45.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f5925000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f592015e490 con 0x7f59200cf110 2024-01-15T18:22:45.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f591d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5920158900 0x7f59201540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:45.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f591d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5920158900 0x7f59201540a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58692/0 (socket says 172.21.15.27:58692) 2024-01-15T18:22:45.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f591d400640 1 -- 172.21.15.27:0/1094120777 learned_addr learned my addr 172.21.15.27:0/1094120777 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:45.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f591e800640 1 --2- 172.21.15.27:0/1094120777 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f592015f550 0x7f592014d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:45.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f591de00640 1 --2- 172.21.15.27:0/1094120777 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59200cf110 0x7f592014c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:45.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f591d400640 1 -- 172.21.15.27:0/1094120777 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59200cf110 msgr2=0x7f592014c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:45.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f591d400640 1 --2- 172.21.15.27:0/1094120777 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59200cf110 0x7f592014c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:45.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f591d400640 1 -- 172.21.15.27:0/1094120777 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f592015f550 msgr2=0x7f592014d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:45.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f591d400640 1 --2- 172.21.15.27:0/1094120777 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f592015f550 0x7f592014d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:45.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f591d400640 1 -- 172.21.15.27:0/1094120777 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59080094d0 con 0x7f5920158900 2024-01-15T18:22:45.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.949+0000 7f591e800640 1 --2- 172.21.15.27:0/1094120777 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f592015f550 0x7f592014d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:22:45.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.953+0000 7f591d400640 1 --2- 172.21.15.27:0/1094120777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5920158900 0x7f59201540a0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f59080ac040 tx=0x7f590801ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:45.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.953+0000 7f5916000640 1 -- 172.21.15.27:0/1094120777 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59080b3800 con 0x7f5920158900 2024-01-15T18:22:45.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.953+0000 7f5916000640 1 -- 172.21.15.27:0/1094120777 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59080b3980 con 0x7f5920158900 2024-01-15T18:22:45.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.953+0000 7f5925000640 1 -- 172.21.15.27:0/1094120777 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59200ad3f0 con 0x7f5920158900 2024-01-15T18:22:45.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.953+0000 7f5916000640 1 -- 172.21.15.27:0/1094120777 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59080b3d30 con 0x7f5920158900 2024-01-15T18:22:45.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.953+0000 7f5925000640 1 -- 172.21.15.27:0/1094120777 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59200ad9b0 con 0x7f5920158900 2024-01-15T18:22:45.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.953+0000 7f5925000640 1 -- 172.21.15.27:0/1094120777 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5920153dc0 con 0x7f5920158900 2024-01-15T18:22:45.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.953+0000 7f5916000640 1 -- 172.21.15.27:0/1094120777 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f59080b2380 con 0x7f5920158900 2024-01-15T18:22:45.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.953+0000 7f5916000640 1 --2- 172.21.15.27:0/1094120777 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f58f8067a60 0x7f58f8069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:45.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.953+0000 7f5916000640 1 -- 172.21.15.27:0/1094120777 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f590811da30 con 0x7f5920158900 2024-01-15T18:22:45.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.953+0000 7f591de00640 1 --2- 172.21.15.27:0/1094120777 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f58f8067a60 0x7f58f8069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:45.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.957+0000 7f591de00640 1 --2- 172.21.15.27:0/1094120777 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f58f8067a60 0x7f58f8069f20 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f5920153870 tx=0x7f5910009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:45.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:45.957+0000 7f5916000640 1 -- 172.21.15.27:0/1094120777 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5920153dc0 con 0x7f5920158900 2024-01-15T18:22:46.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:46.141+0000 7f5925000640 1 -- 172.21.15.27:0/1094120777 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f592015f030 con 0x7f58f8067a60 2024-01-15T18:22:46.147 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:22:46.145+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:22:46.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:46.153+0000 7f5916000640 1 -- 172.21.15.27:0/1094120777 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5910001510 con 0x7f58f8067a60 2024-01-15T18:22:46.148 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:22:46.148 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:22:46.157 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:48.158 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:22:48.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.325+0000 7f3c9b800640 1 Processor -- start 2024-01-15T18:22:48.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.325+0000 7f3c9b800640 1 -- start start 2024-01-15T18:22:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.325+0000 7f3c9b800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f3c9415d680 0x7f3c940cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.325+0000 7f3c9b800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3c940ccf80 0x7f3c940cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.325+0000 7f3c9b800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c940cb4d0 con 0x7f3c940cfc30 2024-01-15T18:22:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.325+0000 7f3c9b800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c940d0120 con 0x7f3c940ccf80 2024-01-15T18:22:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.325+0000 7f3c9b800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c940d0400 con 0x7f3c9415d680 2024-01-15T18:22:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.325+0000 7f3c99800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3c940ccf80 0x7f3c940cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.325+0000 7f3c99800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3c940ccf80 0x7f3c940cdc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58706/0 (socket says 172.21.15.27:58706) 2024-01-15T18:22:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.325+0000 7f3c99800640 1 -- 172.21.15.27:0/2554850331 learned_addr learned my addr 172.21.15.27:0/2554850331 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:48.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.325+0000 7f3c98e00640 1 --2- 172.21.15.27:0/2554850331 >> v2:172.21.15.93:3300/0 conn(0x7f3c9415d680 0x7f3c940cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.325+0000 7f3c99800640 1 -- 172.21.15.27:0/2554850331 >> v2:172.21.15.93:3300/0 conn(0x7f3c9415d680 msgr2=0x7f3c940cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.325+0000 7f3c99800640 1 --2- 172.21.15.27:0/2554850331 >> v2:172.21.15.93:3300/0 conn(0x7f3c9415d680 0x7f3c940cd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.325+0000 7f3c99800640 1 -- 172.21.15.27:0/2554850331 >> v1:172.21.15.93:6789/0 conn(0x7f3c940cfc30 legacy=0x7f3c9406b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:48.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.325+0000 7f3c99800640 1 -- 172.21.15.27:0/2554850331 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c940cb770 con 0x7f3c940ccf80 2024-01-15T18:22:48.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.329+0000 7f3c99800640 1 --2- 172.21.15.27:0/2554850331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c940ccf80 0x7f3c940cdc40 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f3c940d0e50 tx=0x7f3c8c01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6c927dba58d9038 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.329+0000 7f3c93400640 1 -- 172.21.15.27:0/2554850331 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c940ce850 con 0x7f3c940ccf80 2024-01-15T18:22:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.329+0000 7f3c93400640 1 -- 172.21.15.27:0/2554850331 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c8c0a5880 con 0x7f3c940ccf80 2024-01-15T18:22:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.329+0000 7f3c93400640 1 -- 172.21.15.27:0/2554850331 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c0a5c10 con 0x7f3c940ccf80 2024-01-15T18:22:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.329+0000 7f3c9b800640 1 -- 172.21.15.27:0/2554850331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c940ccf80 msgr2=0x7f3c940cdc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.329+0000 7f3c9b800640 1 --2- 172.21.15.27:0/2554850331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c940ccf80 0x7f3c940cdc40 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f3c940d0e50 tx=0x7f3c8c01c9a0 comp rx=0 tx=0).stop 2024-01-15T18:22:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.329+0000 7f3c9b800640 1 -- 172.21.15.27:0/2554850331 shutdown_connections 2024-01-15T18:22:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.329+0000 7f3c9b800640 1 --2- 172.21.15.27:0/2554850331 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c940ccf80 0x7f3c940cdc40 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.329+0000 7f3c9b800640 1 --2- 172.21.15.27:0/2554850331 >> v2:172.21.15.93:3300/0 conn(0x7f3c9415d680 0x7f3c940cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:48.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.329+0000 7f3c9b800640 1 -- 172.21.15.27:0/2554850331 >> 172.21.15.27:0/2554850331 conn(0x7f3c940d0e80 msgr2=0x7f3c940d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:22:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.329+0000 7f3c9b800640 1 -- 172.21.15.27:0/2554850331 shutdown_connections 2024-01-15T18:22:48.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.329+0000 7f3c9b800640 1 -- 172.21.15.27:0/2554850331 wait complete. 2024-01-15T18:22:48.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.333+0000 7f3c9b800640 1 Processor -- start 2024-01-15T18:22:48.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.333+0000 7f3c9b800640 1 -- start start 2024-01-15T18:22:48.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.333+0000 7f3c9b800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c940ccf80 0x7f3c9414c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:48.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.333+0000 7f3c9b800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3c940cfc30 0x7f3c94150860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:48.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.333+0000 7f3c9b800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3c9415d680 0x7f3c9406e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:48.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.333+0000 7f3c9b800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c94157c90 con 0x7f3c940ccf80 2024-01-15T18:22:48.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.333+0000 7f3c9b800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c940c8830 con 0x7f3c9415d680 2024-01-15T18:22:48.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.333+0000 7f3c9b800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c941579b0 con 0x7f3c940cfc30 2024-01-15T18:22:48.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.333+0000 7f3c98e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c940ccf80 0x7f3c9414c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:48.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.333+0000 7f3c98e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c940ccf80 0x7f3c9414c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58712/0 (socket says 172.21.15.27:58712) 2024-01-15T18:22:48.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.333+0000 7f3c98e00640 1 -- 172.21.15.27:0/2584891156 learned_addr learned my addr 172.21.15.27:0/2584891156 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:48.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.333+0000 7f3c99800640 1 --2- 172.21.15.27:0/2584891156 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3c9415d680 0x7f3c9406e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:48.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.333+0000 7f3c93e00640 1 --2- 172.21.15.27:0/2584891156 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3c940cfc30 0x7f3c94150860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:48.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.333+0000 7f3c98e00640 1 -- 172.21.15.27:0/2584891156 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3c940cfc30 msgr2=0x7f3c94150860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:48.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.333+0000 7f3c98e00640 1 --2- 172.21.15.27:0/2584891156 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3c940cfc30 0x7f3c94150860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:48.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.333+0000 7f3c98e00640 1 -- 172.21.15.27:0/2584891156 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3c9415d680 msgr2=0x7f3c9406e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:48.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.333+0000 7f3c98e00640 1 --2- 172.21.15.27:0/2584891156 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3c9415d680 0x7f3c9406e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:48.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.333+0000 7f3c98e00640 1 -- 172.21.15.27:0/2584891156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8c009b30 con 0x7f3c940ccf80 2024-01-15T18:22:48.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.333+0000 7f3c93e00640 1 --2- 172.21.15.27:0/2584891156 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3c940cfc30 0x7f3c94150860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:22:48.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.333+0000 7f3c98e00640 1 --2- 172.21.15.27:0/2584891156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c940ccf80 0x7f3c9414c990 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f3c8400e8a0 tx=0x7f3c8400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:48.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.333+0000 7f3c90c00640 1 -- 172.21.15.27:0/2584891156 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c84009510 con 0x7f3c940ccf80 2024-01-15T18:22:48.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.333+0000 7f3c90c00640 1 -- 172.21.15.27:0/2584891156 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c8400c940 con 0x7f3c940ccf80 2024-01-15T18:22:48.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.333+0000 7f3c90c00640 1 -- 172.21.15.27:0/2584891156 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8400ccb0 con 0x7f3c940ccf80 2024-01-15T18:22:48.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.333+0000 7f3c9b800640 1 -- 172.21.15.27:0/2584891156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c9414dce0 con 0x7f3c940ccf80 2024-01-15T18:22:48.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.333+0000 7f3c9b800640 1 -- 172.21.15.27:0/2584891156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c9414e2a0 con 0x7f3c940ccf80 2024-01-15T18:22:48.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.337+0000 7f3c9b800640 1 -- 172.21.15.27:0/2584891156 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c9414db10 con 0x7f3c940ccf80 2024-01-15T18:22:48.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.337+0000 7f3c90c00640 1 -- 172.21.15.27:0/2584891156 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f3c84013470 con 0x7f3c940ccf80 2024-01-15T18:22:48.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.337+0000 7f3c90c00640 1 --2- 172.21.15.27:0/2584891156 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3c6c067a60 0x7f3c6c069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:48.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.337+0000 7f3c90c00640 1 -- 172.21.15.27:0/2584891156 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f3c84083380 con 0x7f3c940ccf80 2024-01-15T18:22:48.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.337+0000 7f3c93e00640 1 --2- 172.21.15.27:0/2584891156 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3c6c067a60 0x7f3c6c069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:48.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.341+0000 7f3c93e00640 1 --2- 172.21.15.27:0/2584891156 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3c6c067a60 0x7f3c6c069f20 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c000f70 tx=0x7f3c7c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:48.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.341+0000 7f3c90c00640 1 -- 172.21.15.27:0/2584891156 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3c84051d70 con 0x7f3c940ccf80 2024-01-15T18:22:48.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.529+0000 7f3c9b800640 1 -- 172.21.15.27:0/2584891156 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3c940afe70 con 0x7f3c6c067a60 2024-01-15T18:22:48.535 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:22:48.533+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:22:48.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:48.537+0000 7f3c90c00640 1 -- 172.21.15.27:0/2584891156 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3c940afe70 con 0x7f3c6c067a60 2024-01-15T18:22:48.536 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:22:48.536 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:22:48.544 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:50.546 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:22:50.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.709+0000 7f2a0a000640 1 Processor -- start 2024-01-15T18:22:50.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.709+0000 7f2a0a000640 1 -- start start 2024-01-15T18:22:50.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.713+0000 7f2a0a000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2a0415ad20 0x7f2a0415b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:50.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.713+0000 7f2a0a000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f2a040715f0 0x7f2a0415dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:50.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.713+0000 7f2a0a000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0415a610 con 0x7f2a0415d900 2024-01-15T18:22:50.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.713+0000 7f2a0a000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a0415e3c0 con 0x7f2a040715f0 2024-01-15T18:22:50.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.713+0000 7f2a0a000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0415e6a0 con 0x7f2a0415ad20 2024-01-15T18:22:50.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.713+0000 7f2a02a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2a0415d900 0x7f2a04070ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:58336/0 (socket says 172.21.15.27:58336) 2024-01-15T18:22:50.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.713+0000 7f2a02a00640 1 -- 172.21.15.27:0/1280034725 learned_addr learned my addr 172.21.15.27:0/1280034725 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:22:50.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.713+0000 7f2a03400640 1 --2- 172.21.15.27:0/1280034725 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2a0415ad20 0x7f2a0415b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:50.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.713+0000 7f2a03e00640 1 --2- 172.21.15.27:0/1280034725 >> v2:172.21.15.93:3300/0 conn(0x7f2a040715f0 0x7f2a0415dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:50.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.713+0000 7f2a02000640 1 -- 172.21.15.27:0/1280034725 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896264579 0 0) 0x7f2a0415a610 con 0x7f2a0415d900 2024-01-15T18:22:50.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.713+0000 7f2a02000640 1 -- 172.21.15.27:0/1280034725 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29e4003530 con 0x7f2a0415d900 2024-01-15T18:22:50.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.713+0000 7f2a03e00640 1 -- 172.21.15.27:0/1280034725 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2a0415ad20 msgr2=0x7f2a0415b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:50.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.713+0000 7f2a03e00640 1 --2- 172.21.15.27:0/1280034725 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2a0415ad20 0x7f2a0415b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:50.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.713+0000 7f2a03e00640 1 -- 172.21.15.27:0/1280034725 >> v1:172.21.15.27:6789/0 conn(0x7f2a0415d900 legacy=0x7f2a04070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:50.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.713+0000 7f2a03e00640 1 -- 172.21.15.27:0/1280034725 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0415b6e0 con 0x7f2a040715f0 2024-01-15T18:22:50.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.717+0000 7f2a03e00640 1 --2- 172.21.15.27:0/1280034725 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a040715f0 0x7f2a0415dd80 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f29fc009d90 tx=0x7f29fc01c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=baa5fc70e42b984f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:50.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.717+0000 7f2a02000640 1 -- 172.21.15.27:0/1280034725 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29fc0a5a00 con 0x7f2a040715f0 2024-01-15T18:22:50.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.717+0000 7f2a02000640 1 -- 172.21.15.27:0/1280034725 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29fc0a5b60 con 0x7f2a040715f0 2024-01-15T18:22:50.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.717+0000 7f2a02000640 1 -- 172.21.15.27:0/1280034725 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29fc0ad440 con 0x7f2a040715f0 2024-01-15T18:22:50.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.717+0000 7f2a0a000640 1 -- 172.21.15.27:0/1280034725 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a040715f0 msgr2=0x7f2a0415dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:50.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.717+0000 7f2a0a000640 1 --2- 172.21.15.27:0/1280034725 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a040715f0 0x7f2a0415dd80 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f29fc009d90 tx=0x7f29fc01c920 comp rx=0 tx=0).stop 2024-01-15T18:22:50.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.717+0000 7f2a0a000640 1 -- 172.21.15.27:0/1280034725 shutdown_connections 2024-01-15T18:22:50.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.717+0000 7f2a0a000640 1 --2- 172.21.15.27:0/1280034725 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a040715f0 0x7f2a0415dd80 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:50.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.717+0000 7f2a0a000640 1 --2- 172.21.15.27:0/1280034725 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2a0415ad20 0x7f2a0415b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:50.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.717+0000 7f2a0a000640 1 -- 172.21.15.27:0/1280034725 >> 172.21.15.27:0/1280034725 conn(0x7f2a040cb700 msgr2=0x7f2a040cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:22:50.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.717+0000 7f2a0a000640 1 -- 172.21.15.27:0/1280034725 shutdown_connections 2024-01-15T18:22:50.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.717+0000 7f2a0a000640 1 -- 172.21.15.27:0/1280034725 wait complete. 2024-01-15T18:22:50.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.721+0000 7f2a0a000640 1 Processor -- start 2024-01-15T18:22:50.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.721+0000 7f2a0a000640 1 -- start start 2024-01-15T18:22:50.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.721+0000 7f2a0a000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a040715f0 0x7f2a04147670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:50.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.721+0000 7f2a0a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a0415ad20 0x7f2a0414b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:50.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.721+0000 7f2a0a000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2a0415d900 0x7f2a04146e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:50.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.721+0000 7f2a0a000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a04149610 con 0x7f2a0415ad20 2024-01-15T18:22:50.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.721+0000 7f2a0a000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a04149290 con 0x7f2a040715f0 2024-01-15T18:22:50.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.721+0000 7f2a0a000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0414dfb0 con 0x7f2a0415d900 2024-01-15T18:22:50.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.721+0000 7f2a02a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a0415ad20 0x7f2a0414b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:50.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.721+0000 7f2a02a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a0415ad20 0x7f2a0414b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58718/0 (socket says 172.21.15.27:58718) 2024-01-15T18:22:50.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.721+0000 7f2a02a00640 1 -- 172.21.15.27:0/3205757602 learned_addr learned my addr 172.21.15.27:0/3205757602 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:50.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.721+0000 7f2a03e00640 1 --2- 172.21.15.27:0/3205757602 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2a0415d900 0x7f2a04146e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:50.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.721+0000 7f2a03400640 1 --2- 172.21.15.27:0/3205757602 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a040715f0 0x7f2a04147670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:50.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.721+0000 7f2a02a00640 1 -- 172.21.15.27:0/3205757602 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2a0415d900 msgr2=0x7f2a04146e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:50.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.721+0000 7f2a02a00640 1 --2- 172.21.15.27:0/3205757602 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2a0415d900 0x7f2a04146e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:50.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.721+0000 7f2a02a00640 1 -- 172.21.15.27:0/3205757602 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a040715f0 msgr2=0x7f2a04147670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:50.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.721+0000 7f2a02a00640 1 --2- 172.21.15.27:0/3205757602 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2a040715f0 0x7f2a04147670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:50.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.721+0000 7f2a02a00640 1 -- 172.21.15.27:0/3205757602 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29fc009a70 con 0x7f2a0415ad20 2024-01-15T18:22:50.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.721+0000 7f2a02a00640 1 --2- 172.21.15.27:0/3205757602 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a0415ad20 0x7f2a0414b870 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f29ec00c420 tx=0x7f29ec00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:50.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.721+0000 7f29fb400640 1 -- 172.21.15.27:0/3205757602 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec005cf0 con 0x7f2a0415ad20 2024-01-15T18:22:50.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.721+0000 7f29fb400640 1 -- 172.21.15.27:0/3205757602 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29ec005e50 con 0x7f2a0415ad20 2024-01-15T18:22:50.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.721+0000 7f2a0a000640 1 -- 172.21.15.27:0/3205757602 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a0414e290 con 0x7f2a0415ad20 2024-01-15T18:22:50.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.721+0000 7f29fb400640 1 -- 172.21.15.27:0/3205757602 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec0072d0 con 0x7f2a0415ad20 2024-01-15T18:22:50.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.721+0000 7f2a0a000640 1 -- 172.21.15.27:0/3205757602 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0414e850 con 0x7f2a0415ad20 2024-01-15T18:22:50.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.721+0000 7f2a0a000640 1 -- 172.21.15.27:0/3205757602 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a04149750 con 0x7f2a0415ad20 2024-01-15T18:22:50.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.725+0000 7f29fb400640 1 -- 172.21.15.27:0/3205757602 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f29ec007430 con 0x7f2a0415ad20 2024-01-15T18:22:50.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.725+0000 7f29fb400640 1 --2- 172.21.15.27:0/3205757602 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f29dc067a60 0x7f29dc069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:50.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.725+0000 7f29fb400640 1 -- 172.21.15.27:0/3205757602 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f29ec082880 con 0x7f2a0415ad20 2024-01-15T18:22:50.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.725+0000 7f2a03400640 1 --2- 172.21.15.27:0/3205757602 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f29dc067a60 0x7f29dc069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:50.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.725+0000 7f2a03400640 1 --2- 172.21.15.27:0/3205757602 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f29dc067a60 0x7f29dc069f20 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f2a041503d0 tx=0x7f29f4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:50.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.729+0000 7f29fb400640 1 -- 172.21.15.27:0/3205757602 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2a04149750 con 0x7f2a0415ad20 2024-01-15T18:22:50.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.917+0000 7f2a0a000640 1 -- 172.21.15.27:0/3205757602 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2a04149750 con 0x7f29dc067a60 2024-01-15T18:22:50.920 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:22:50.921+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:22:50.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:50.925+0000 7f29fb400640 1 -- 172.21.15.27:0/3205757602 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2a04149750 con 0x7f29dc067a60 2024-01-15T18:22:50.921 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:22:50.921 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:22:50.929 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:52.930 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:22:53.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.093+0000 7fa4db800640 1 Processor -- start 2024-01-15T18:22:53.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.093+0000 7fa4db800640 1 -- start start 2024-01-15T18:22:53.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.093+0000 7fa4db800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4d41468d0 0x7fa4d40ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:53.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.093+0000 7fa4db800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa4d40cef30 0x7fa4d40cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:53.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.093+0000 7fa4db800640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d40ad920 con 0x7fa4d40ccea0 2024-01-15T18:22:53.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.093+0000 7fa4db800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4d40ac880 con 0x7fa4d40cef30 2024-01-15T18:22:53.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.093+0000 7fa4db800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d40acb60 con 0x7fa4d41468d0 2024-01-15T18:22:53.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.093+0000 7fa4d3e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa4d40cef30 0x7fa4d40cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:53.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.093+0000 7fa4d3e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa4d40cef30 0x7fa4d40cc960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58732/0 (socket says 172.21.15.27:58732) 2024-01-15T18:22:53.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.093+0000 7fa4d3e00640 1 -- 172.21.15.27:0/1429629388 learned_addr learned my addr 172.21.15.27:0/1429629388 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:53.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.093+0000 7fa4d8e00640 1 --2- 172.21.15.27:0/1429629388 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4d41468d0 0x7fa4d40ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:53.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.093+0000 7fa4d3e00640 1 -- 172.21.15.27:0/1429629388 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4d41468d0 msgr2=0x7fa4d40ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:53.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.093+0000 7fa4d3e00640 1 --2- 172.21.15.27:0/1429629388 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4d41468d0 0x7fa4d40ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:53.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.093+0000 7fa4d3e00640 1 -- 172.21.15.27:0/1429629388 >> v1:172.21.15.93:6789/0 conn(0x7fa4d40ccea0 legacy=0x7fa4d4160cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:53.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.093+0000 7fa4d3e00640 1 -- 172.21.15.27:0/1429629388 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d40ace40 con 0x7fa4d40cef30 2024-01-15T18:22:53.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.097+0000 7fa4d3e00640 1 --2- 172.21.15.27:0/1429629388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4d40cef30 0x7fa4d40cc960 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc0098a0 tx=0x7fa4bc01c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21251d2812a68adc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:53.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.097+0000 7fa4d3400640 1 -- 172.21.15.27:0/1429629388 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc0ad640 con 0x7fa4d40cef30 2024-01-15T18:22:53.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.097+0000 7fa4d3400640 1 -- 172.21.15.27:0/1429629388 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4bc0ad7a0 con 0x7fa4d40cef30 2024-01-15T18:22:53.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.097+0000 7fa4d3400640 1 -- 172.21.15.27:0/1429629388 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc0adb30 con 0x7fa4d40cef30 2024-01-15T18:22:53.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.097+0000 7fa4db800640 1 -- 172.21.15.27:0/1429629388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4d40cef30 msgr2=0x7fa4d40cc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:53.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.097+0000 7fa4db800640 1 --2- 172.21.15.27:0/1429629388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4d40cef30 0x7fa4d40cc960 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc0098a0 tx=0x7fa4bc01c4b0 comp rx=0 tx=0).stop 2024-01-15T18:22:53.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.097+0000 7fa4db800640 1 -- 172.21.15.27:0/1429629388 shutdown_connections 2024-01-15T18:22:53.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.097+0000 7fa4db800640 1 --2- 172.21.15.27:0/1429629388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4d40cef30 0x7fa4d40cc960 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:53.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.097+0000 7fa4db800640 1 --2- 172.21.15.27:0/1429629388 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4d41468d0 0x7fa4d40ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:53.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.097+0000 7fa4db800640 1 -- 172.21.15.27:0/1429629388 >> 172.21.15.27:0/1429629388 conn(0x7fa4d40cb700 msgr2=0x7fa4d40cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:22:53.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.097+0000 7fa4db800640 1 -- 172.21.15.27:0/1429629388 shutdown_connections 2024-01-15T18:22:53.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.097+0000 7fa4db800640 1 -- 172.21.15.27:0/1429629388 wait complete. 2024-01-15T18:22:53.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.101+0000 7fa4db800640 1 Processor -- start 2024-01-15T18:22:53.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.101+0000 7fa4db800640 1 -- start start 2024-01-15T18:22:53.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.101+0000 7fa4db800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4d40ccea0 0x7fa4d4158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:53.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.101+0000 7fa4db800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa4d40cef30 0x7fa4d413d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:53.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.101+0000 7fa4db800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4d41468d0 0x7fa4d4071210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:53.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.101+0000 7fa4db800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d40b2440 con 0x7fa4d40ccea0 2024-01-15T18:22:53.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.101+0000 7fa4db800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d4071e80 con 0x7fa4d40cef30 2024-01-15T18:22:53.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.101+0000 7fa4db800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d4072160 con 0x7fa4d41468d0 2024-01-15T18:22:53.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.101+0000 7fa4d8e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4d40ccea0 0x7fa4d4158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:53.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.101+0000 7fa4d8e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4d40ccea0 0x7fa4d4158990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58734/0 (socket says 172.21.15.27:58734) 2024-01-15T18:22:53.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.101+0000 7fa4d8e00640 1 -- 172.21.15.27:0/3256356121 learned_addr learned my addr 172.21.15.27:0/3256356121 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:53.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.101+0000 7fa4d3e00640 1 --2- 172.21.15.27:0/3256356121 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa4d40cef30 0x7fa4d413d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:53.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.101+0000 7fa4d9800640 1 --2- 172.21.15.27:0/3256356121 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4d41468d0 0x7fa4d4071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:53.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.101+0000 7fa4d8e00640 1 -- 172.21.15.27:0/3256356121 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4d41468d0 msgr2=0x7fa4d4071210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:53.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.101+0000 7fa4d8e00640 1 --2- 172.21.15.27:0/3256356121 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa4d41468d0 0x7fa4d4071210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:53.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.101+0000 7fa4d8e00640 1 -- 172.21.15.27:0/3256356121 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa4d40cef30 msgr2=0x7fa4d413d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:53.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.101+0000 7fa4d8e00640 1 --2- 172.21.15.27:0/3256356121 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa4d40cef30 0x7fa4d413d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:53.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.101+0000 7fa4d8e00640 1 -- 172.21.15.27:0/3256356121 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4bc0094d0 con 0x7fa4d40ccea0 2024-01-15T18:22:53.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.101+0000 7fa4d8e00640 1 --2- 172.21.15.27:0/3256356121 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4d40ccea0 0x7fa4d4158990 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fa4c400d5f0 tx=0x7fa4c400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:53.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.101+0000 7fa4d0c00640 1 -- 172.21.15.27:0/3256356121 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c40125f0 con 0x7fa4d40ccea0 2024-01-15T18:22:53.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.101+0000 7fa4d0c00640 1 -- 172.21.15.27:0/3256356121 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4c4012750 con 0x7fa4d40ccea0 2024-01-15T18:22:53.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.101+0000 7fa4d0c00640 1 -- 172.21.15.27:0/3256356121 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c4012ae0 con 0x7fa4d40ccea0 2024-01-15T18:22:53.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.101+0000 7fa4db800640 1 -- 172.21.15.27:0/3256356121 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d4072440 con 0x7fa4d40ccea0 2024-01-15T18:22:53.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.101+0000 7fa4db800640 1 -- 172.21.15.27:0/3256356121 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d4158ed0 con 0x7fa4d40ccea0 2024-01-15T18:22:53.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.105+0000 7fa4db800640 1 -- 172.21.15.27:0/3256356121 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d404e850 con 0x7fa4d40ccea0 2024-01-15T18:22:53.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.105+0000 7fa4d0c00640 1 -- 172.21.15.27:0/3256356121 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7fa4c4012c40 con 0x7fa4d40ccea0 2024-01-15T18:22:53.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.105+0000 7fa4d0c00640 1 --2- 172.21.15.27:0/3256356121 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa4ac067a60 0x7fa4ac069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:53.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.105+0000 7fa4d0c00640 1 -- 172.21.15.27:0/3256356121 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fa4c40836e0 con 0x7fa4d40ccea0 2024-01-15T18:22:53.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.105+0000 7fa4d3e00640 1 --2- 172.21.15.27:0/3256356121 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa4ac067a60 0x7fa4ac069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:53.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.105+0000 7fa4d3e00640 1 --2- 172.21.15.27:0/3256356121 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa4ac067a60 0x7fa4ac069f20 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc0af000 tx=0x7fa4bc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:53.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.109+0000 7fa4d0c00640 1 -- 172.21.15.27:0/3256356121 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa4c40520d0 con 0x7fa4d40ccea0 2024-01-15T18:22:53.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.297+0000 7fa4db800640 1 -- 172.21.15.27:0/3256356121 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa4d4146cb0 con 0x7fa4ac067a60 2024-01-15T18:22:53.301 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:22:53.301+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:22:53.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:53.305+0000 7fa4d0c00640 1 -- 172.21.15.27:0/3256356121 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa4bc0a6040 con 0x7fa4ac067a60 2024-01-15T18:22:53.302 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:22:53.302 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:22:53.310 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:55.311 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:22:55.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.473+0000 7f6cf1800640 1 Processor -- start 2024-01-15T18:22:55.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.473+0000 7f6cf1800640 1 -- start start 2024-01-15T18:22:55.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.473+0000 7f6cf1800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6cec15b100 0x7f6cec070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:55.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.473+0000 7f6cf1800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6cec071420 0x7f6cec15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:55.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.473+0000 7f6cf1800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cec15a610 con 0x7f6cec15ad20 2024-01-15T18:22:55.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.473+0000 7f6cf1800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cec0718b0 con 0x7f6cec071420 2024-01-15T18:22:55.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.473+0000 7f6cf1800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec15e3c0 con 0x7f6cec15b100 2024-01-15T18:22:55.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.477+0000 7f6ceb400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6cec071420 0x7f6cec15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:55.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.477+0000 7f6ceb400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6cec071420 0x7f6cec15bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40252/0 (socket says 172.21.15.27:40252) 2024-01-15T18:22:55.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.477+0000 7f6ceb400640 1 -- 172.21.15.27:0/3171027084 learned_addr learned my addr 172.21.15.27:0/3171027084 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:55.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.477+0000 7f6ceaa00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6cec15ad20 0x7f6cec15d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51944/0 (socket says 172.21.15.27:51944) 2024-01-15T18:22:55.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.477+0000 7f6cea000640 1 --2- 172.21.15.27:0/3171027084 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6cec15b100 0x7f6cec070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:55.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.477+0000 7f6ceb400640 1 -- 172.21.15.27:0/3171027084 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6cec15b100 msgr2=0x7f6cec070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:55.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.477+0000 7f6ceb400640 1 --2- 172.21.15.27:0/3171027084 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6cec15b100 0x7f6cec070ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:55.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.477+0000 7f6ceb400640 1 -- 172.21.15.27:0/3171027084 >> v1:172.21.15.27:6789/0 conn(0x7f6cec15ad20 legacy=0x7f6cec15d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:55.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.477+0000 7f6ceb400640 1 -- 172.21.15.27:0/3171027084 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cec15e6a0 con 0x7f6cec071420 2024-01-15T18:22:55.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.477+0000 7f6ce9600640 1 -- 172.21.15.27:0/3171027084 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608867794 0 0) 0x7f6cec15a610 con 0x7f6cec15ad20 2024-01-15T18:22:55.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.477+0000 7f6ceb400640 1 --2- 172.21.15.27:0/3171027084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cec071420 0x7f6cec15bee0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4009e50 tx=0x7f6ce401c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a4cef7d38cc3879 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:55.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.481+0000 7f6ce9600640 1 -- 172.21.15.27:0/3171027084 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce40a57c0 con 0x7f6cec071420 2024-01-15T18:22:55.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.481+0000 7f6ce9600640 1 -- 172.21.15.27:0/3171027084 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce40a5920 con 0x7f6cec071420 2024-01-15T18:22:55.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.481+0000 7f6ce9600640 1 -- 172.21.15.27:0/3171027084 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce40a5cb0 con 0x7f6cec071420 2024-01-15T18:22:55.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.481+0000 7f6cf1800640 1 -- 172.21.15.27:0/3171027084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cec071420 msgr2=0x7f6cec15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:55.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.481+0000 7f6cf1800640 1 --2- 172.21.15.27:0/3171027084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cec071420 0x7f6cec15bee0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f6ce4009e50 tx=0x7f6ce401c9e0 comp rx=0 tx=0).stop 2024-01-15T18:22:55.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.481+0000 7f6cf1800640 1 -- 172.21.15.27:0/3171027084 shutdown_connections 2024-01-15T18:22:55.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.481+0000 7f6cf1800640 1 --2- 172.21.15.27:0/3171027084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cec071420 0x7f6cec15bee0 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:55.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.481+0000 7f6cf1800640 1 --2- 172.21.15.27:0/3171027084 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6cec15b100 0x7f6cec070ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:55.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.481+0000 7f6cf1800640 1 -- 172.21.15.27:0/3171027084 >> 172.21.15.27:0/3171027084 conn(0x7f6cec0cb700 msgr2=0x7f6cec0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:22:55.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.481+0000 7f6cf1800640 1 -- 172.21.15.27:0/3171027084 shutdown_connections 2024-01-15T18:22:55.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.481+0000 7f6cf1800640 1 -- 172.21.15.27:0/3171027084 wait complete. 2024-01-15T18:22:55.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.481+0000 7f6cf1800640 1 Processor -- start 2024-01-15T18:22:55.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.481+0000 7f6cf1800640 1 -- start start 2024-01-15T18:22:55.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.485+0000 7f6cf1800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6cec071420 0x7f6cec14c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:55.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.485+0000 7f6cf1800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cec15ad20 0x7f6cec0b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:55.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.485+0000 7f6cf1800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6cec15b100 0x7f6cec162e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:55.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.485+0000 7f6cf1800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec0b34d0 con 0x7f6cec15ad20 2024-01-15T18:22:55.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.485+0000 7f6cf1800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec14c800 con 0x7f6cec15b100 2024-01-15T18:22:55.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.485+0000 7f6cf1800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec0b3770 con 0x7f6cec071420 2024-01-15T18:22:55.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.485+0000 7f6cea000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cec15ad20 0x7f6cec0b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:55.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.485+0000 7f6cea000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cec15ad20 0x7f6cec0b29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40262/0 (socket says 172.21.15.27:40262) 2024-01-15T18:22:55.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.485+0000 7f6cea000640 1 -- 172.21.15.27:0/2112664464 learned_addr learned my addr 172.21.15.27:0/2112664464 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:55.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.485+0000 7f6ceaa00640 1 --2- 172.21.15.27:0/2112664464 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6cec071420 0x7f6cec14c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:55.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.485+0000 7f6ceb400640 1 --2- 172.21.15.27:0/2112664464 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6cec15b100 0x7f6cec162e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:55.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.485+0000 7f6cea000640 1 -- 172.21.15.27:0/2112664464 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6cec071420 msgr2=0x7f6cec14c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:55.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.485+0000 7f6cea000640 1 --2- 172.21.15.27:0/2112664464 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6cec071420 0x7f6cec14c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:55.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.485+0000 7f6cea000640 1 -- 172.21.15.27:0/2112664464 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6cec15b100 msgr2=0x7f6cec162e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:55.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.485+0000 7f6cea000640 1 --2- 172.21.15.27:0/2112664464 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6cec15b100 0x7f6cec162e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:55.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.485+0000 7f6cea000640 1 -- 172.21.15.27:0/2112664464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce4009b30 con 0x7f6cec15ad20 2024-01-15T18:22:55.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.485+0000 7f6cea000640 1 --2- 172.21.15.27:0/2112664464 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6cec15ad20 0x7f6cec0b29a0 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f6cd400d5f0 tx=0x7f6cd400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:55.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.485+0000 7f6ce2a00640 1 -- 172.21.15.27:0/2112664464 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd40125f0 con 0x7f6cec15ad20 2024-01-15T18:22:55.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.485+0000 7f6ce2a00640 1 -- 172.21.15.27:0/2112664464 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd4012750 con 0x7f6cec15ad20 2024-01-15T18:22:55.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.485+0000 7f6ce2a00640 1 -- 172.21.15.27:0/2112664464 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd4012ae0 con 0x7f6cec15ad20 2024-01-15T18:22:55.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.485+0000 7f6cf1800640 1 -- 172.21.15.27:0/2112664464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cec0b3a50 con 0x7f6cec15ad20 2024-01-15T18:22:55.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.485+0000 7f6cf1800640 1 -- 172.21.15.27:0/2112664464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cec0b4010 con 0x7f6cec15ad20 2024-01-15T18:22:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.485+0000 7f6cf1800640 1 -- 172.21.15.27:0/2112664464 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cec1562f0 con 0x7f6cec15ad20 2024-01-15T18:22:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.485+0000 7f6ce2a00640 1 -- 172.21.15.27:0/2112664464 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f6cd4012c40 con 0x7f6cec15ad20 2024-01-15T18:22:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.489+0000 7f6ce2a00640 1 --2- 172.21.15.27:0/2112664464 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6cc4067a60 0x7f6cc4069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:55.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.489+0000 7f6ce2a00640 1 -- 172.21.15.27:0/2112664464 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f6cd4083860 con 0x7f6cec15ad20 2024-01-15T18:22:55.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.489+0000 7f6ceaa00640 1 --2- 172.21.15.27:0/2112664464 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6cc4067a60 0x7f6cc4069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:55.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.489+0000 7f6ceaa00640 1 --2- 172.21.15.27:0/2112664464 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6cc4067a60 0x7f6cc4069f20 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc000f70 tx=0x7f6cdc007b80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:55.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.493+0000 7f6ce2a00640 1 -- 172.21.15.27:0/2112664464 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6cd40521a0 con 0x7f6cec15ad20 2024-01-15T18:22:55.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.681+0000 7f6cf1800640 1 -- 172.21.15.27:0/2112664464 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6cec0b38e0 con 0x7f6cc4067a60 2024-01-15T18:22:55.686 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:22:55.685+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:22:55.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:55.689+0000 7f6ce2a00640 1 -- 172.21.15.27:0/2112664464 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6cec0b38e0 con 0x7f6cc4067a60 2024-01-15T18:22:55.686 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:22:55.687 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:22:55.695 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:57.696 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:22:57.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.857+0000 7f384c200640 1 Processor -- start 2024-01-15T18:22:57.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.857+0000 7f384c200640 1 -- start start 2024-01-15T18:22:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f384c200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f384415ad20 0x7f384415b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f384c200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f38440715f0 0x7f384415dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f384c200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f384415a610 con 0x7f384415d900 2024-01-15T18:22:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f384c200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f384415e3c0 con 0x7f38440715f0 2024-01-15T18:22:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f384c200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f384415e6a0 con 0x7f384415ad20 2024-01-15T18:22:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3848e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f384415d900 0x7f3844070ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51950/0 (socket says 172.21.15.27:51950) 2024-01-15T18:22:57.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3848e00640 1 -- 172.21.15.27:0/3048159585 learned_addr learned my addr 172.21.15.27:0/3048159585 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:22:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3849800640 1 --2- 172.21.15.27:0/3048159585 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f384415ad20 0x7f384415b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f384a200640 1 --2- 172.21.15.27:0/3048159585 >> v2:172.21.15.93:3300/0 conn(0x7f38440715f0 0x7f384415dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3843e00640 1 -- 172.21.15.27:0/3048159585 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2210563029 0 0) 0x7f384415a610 con 0x7f384415d900 2024-01-15T18:22:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3843e00640 1 -- 172.21.15.27:0/3048159585 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3824005740 con 0x7f384415d900 2024-01-15T18:22:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3843e00640 1 -- 172.21.15.27:0/3048159585 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1819733955 0 0) 0x7f3824005740 con 0x7f384415d900 2024-01-15T18:22:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3843e00640 1 -- 172.21.15.27:0/3048159585 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f384415ad20 msgr2=0x7f384415b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3843e00640 1 --2- 172.21.15.27:0/3048159585 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f384415ad20 0x7f384415b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3843e00640 1 -- 172.21.15.27:0/3048159585 >> v2:172.21.15.93:3300/0 conn(0x7f38440715f0 msgr2=0x7f384415dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3843e00640 1 --2- 172.21.15.27:0/3048159585 >> v2:172.21.15.93:3300/0 conn(0x7f38440715f0 0x7f384415dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3843e00640 1 -- 172.21.15.27:0/3048159585 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f384415b6e0 con 0x7f384415d900 2024-01-15T18:22:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3843e00640 1 -- 172.21.15.27:0/3048159585 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f382c001680 con 0x7f384415d900 2024-01-15T18:22:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3843e00640 1 -- 172.21.15.27:0/3048159585 >> v1:172.21.15.27:6789/0 conn(0x7f384415d900 legacy=0x7f3844070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3843e00640 1 --2- 172.21.15.27:0/3048159585 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f384415ad20 0x7f3824005c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3843e00640 1 --2- 172.21.15.27:0/3048159585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38240061d0 0x7f38240069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3843e00640 1 --2- 172.21.15.27:0/3048159585 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3824006f20 0x7f38240073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3843e00640 1 -- 172.21.15.27:0/3048159585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384415e3c0 con 0x7f38240061d0 2024-01-15T18:22:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3843e00640 1 -- 172.21.15.27:0/3048159585 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384415e6a0 con 0x7f3824006f20 2024-01-15T18:22:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3843e00640 1 -- 172.21.15.27:0/3048159585 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f382400dc20 con 0x7f384415ad20 2024-01-15T18:22:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3848e00640 1 --2- 172.21.15.27:0/3048159585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38240061d0 0x7f38240069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3849800640 1 --2- 172.21.15.27:0/3048159585 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f384415ad20 0x7f3824005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f384a200640 1 --2- 172.21.15.27:0/3048159585 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3824006f20 0x7f38240073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:57.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3848e00640 1 -- 172.21.15.27:0/3048159585 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f384415ad20 msgr2=0x7f3824005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:57.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3848e00640 1 --2- 172.21.15.27:0/3048159585 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f384415ad20 0x7f3824005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:57.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3848e00640 1 -- 172.21.15.27:0/3048159585 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3824006f20 msgr2=0x7f38240073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:22:57.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3848e00640 1 --2- 172.21.15.27:0/3048159585 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3824006f20 0x7f38240073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:57.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.861+0000 7f3848e00640 1 -- 172.21.15.27:0/3048159585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3824005740 con 0x7f38240061d0 2024-01-15T18:22:57.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.865+0000 7f3848e00640 1 --2- 172.21.15.27:0/3048159585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38240061d0 0x7f38240069e0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f382c00c110 tx=0x7f382c01f2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5384e38b2508e8e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:57.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.865+0000 7f3843e00640 1 -- 172.21.15.27:0/3048159585 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c005ae0 con 0x7f38240061d0 2024-01-15T18:22:57.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.865+0000 7f3843e00640 1 -- 172.21.15.27:0/3048159585 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382c0ab070 con 0x7f38240061d0 2024-01-15T18:22:57.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.865+0000 7f3843e00640 1 -- 172.21.15.27:0/3048159585 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c007070 con 0x7f38240061d0 2024-01-15T18:22:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.865+0000 7f384c200640 1 -- 172.21.15.27:0/3048159585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38240061d0 msgr2=0x7f38240069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.865+0000 7f384c200640 1 --2- 172.21.15.27:0/3048159585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38240061d0 0x7f38240069e0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f382c00c110 tx=0x7f382c01f2f0 comp rx=0 tx=0).stop 2024-01-15T18:22:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.865+0000 7f384a200640 1 -- 172.21.15.27:0/3048159585 reap_dead start 2024-01-15T18:22:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.865+0000 7f384c200640 1 -- 172.21.15.27:0/3048159585 shutdown_connections 2024-01-15T18:22:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.865+0000 7f384c200640 1 -- 172.21.15.27:0/3048159585 >> 172.21.15.27:0/3048159585 conn(0x7f38440cb700 msgr2=0x7f38440cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:22:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.865+0000 7f384c200640 1 -- 172.21.15.27:0/3048159585 shutdown_connections 2024-01-15T18:22:57.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.865+0000 7f384c200640 1 -- 172.21.15.27:0/3048159585 wait complete. 2024-01-15T18:22:57.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.869+0000 7f384c200640 1 Processor -- start 2024-01-15T18:22:57.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.869+0000 7f384c200640 1 -- start start 2024-01-15T18:22:57.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.869+0000 7f384c200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38240061d0 0x7f38441f5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:57.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.869+0000 7f384c200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3844153780 0x7f38440a8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:57.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.869+0000 7f384c200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3844153be0 0x7f38440a8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:57.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.869+0000 7f384c200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38441f4a30 con 0x7f3844153780 2024-01-15T18:22:57.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.869+0000 7f384c200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38441f4d10 con 0x7f3844153be0 2024-01-15T18:22:57.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.869+0000 7f384c200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38441f4ff0 con 0x7f38240061d0 2024-01-15T18:22:57.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.869+0000 7f3848e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3844153780 0x7f38440a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:57.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.869+0000 7f3848e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3844153780 0x7f38440a8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40284/0 (socket says 172.21.15.27:40284) 2024-01-15T18:22:57.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.869+0000 7f3848e00640 1 -- 172.21.15.27:0/3800510379 learned_addr learned my addr 172.21.15.27:0/3800510379 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:22:57.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.869+0000 7f384a200640 1 --2- 172.21.15.27:0/3800510379 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3844153be0 0x7f38440a8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:57.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.869+0000 7f3849800640 1 --2- 172.21.15.27:0/3800510379 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38240061d0 0x7f38441f5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:57.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.869+0000 7f3848e00640 1 -- 172.21.15.27:0/3800510379 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38240061d0 msgr2=0x7f38441f5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:57.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.869+0000 7f3848e00640 1 --2- 172.21.15.27:0/3800510379 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38240061d0 0x7f38441f5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:57.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.869+0000 7f3848e00640 1 -- 172.21.15.27:0/3800510379 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3844153be0 msgr2=0x7f38440a8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:22:57.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.869+0000 7f3848e00640 1 --2- 172.21.15.27:0/3800510379 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3844153be0 0x7f38440a8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:22:57.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.869+0000 7f3848e00640 1 -- 172.21.15.27:0/3800510379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38441f52d0 con 0x7f3844153780 2024-01-15T18:22:57.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.869+0000 7f3849800640 1 --2- 172.21.15.27:0/3800510379 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f38240061d0 0x7f38441f5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:22:57.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.869+0000 7f3848e00640 1 --2- 172.21.15.27:0/3800510379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3844153780 0x7f38440a8850 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f382c00c110 tx=0x7f3824007a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:57.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.869+0000 7f3841600640 1 -- 172.21.15.27:0/3800510379 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c0b7070 con 0x7f3844153780 2024-01-15T18:22:57.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.869+0000 7f3841600640 1 -- 172.21.15.27:0/3800510379 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382c005900 con 0x7f3844153780 2024-01-15T18:22:57.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.869+0000 7f3841600640 1 -- 172.21.15.27:0/3800510379 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c005c90 con 0x7f3844153780 2024-01-15T18:22:57.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.869+0000 7f384c200640 1 -- 172.21.15.27:0/3800510379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38441f55b0 con 0x7f3844153780 2024-01-15T18:22:57.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.869+0000 7f384c200640 1 -- 172.21.15.27:0/3800510379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38440cd7a0 con 0x7f3844153780 2024-01-15T18:22:57.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.873+0000 7f384c200640 1 -- 172.21.15.27:0/3800510379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f384404e850 con 0x7f3844153780 2024-01-15T18:22:57.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.873+0000 7f3841600640 1 -- 172.21.15.27:0/3800510379 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f382c0ad020 con 0x7f3844153780 2024-01-15T18:22:57.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.873+0000 7f3841600640 1 --2- 172.21.15.27:0/3800510379 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f381c067a90 0x7f381c069f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:22:57.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.873+0000 7f3841600640 1 -- 172.21.15.27:0/3800510379 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f382c11f5e0 con 0x7f3844153780 2024-01-15T18:22:57.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.873+0000 7f3849800640 1 --2- 172.21.15.27:0/3800510379 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f381c067a90 0x7f381c069f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:22:57.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.873+0000 7f3849800640 1 --2- 172.21.15.27:0/3800510379 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f381c067a90 0x7f381c069f50 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f3834009b00 tx=0x7f3834008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:22:57.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:57.877+0000 7f3841600640 1 -- 172.21.15.27:0/3800510379 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f382c0f1d00 con 0x7f3844153780 2024-01-15T18:22:58.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:58.065+0000 7f384c200640 1 -- 172.21.15.27:0/3800510379 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f38441f4e80 con 0x7f381c067a90 2024-01-15T18:22:58.069 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:22:58.069+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:22:58.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:22:58.077+0000 7f3841600640 1 -- 172.21.15.27:0/3800510379 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f38441f4e80 con 0x7f381c067a90 2024-01-15T18:22:58.070 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:22:58.070 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:22:58.078 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:22:58.600 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:22:58.604 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:23:00.080 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:23:00.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.233+0000 7fd06d800640 1 Processor -- start 2024-01-15T18:23:00.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.233+0000 7fd06d800640 1 -- start start 2024-01-15T18:23:00.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.233+0000 7fd06d800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fd06815d190 0x7fd0680cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:00.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.233+0000 7fd06d800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd06815d570 0x7fd06806be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:00.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.233+0000 7fd06d800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0680b0d70 con 0x7fd06806b810 2024-01-15T18:23:00.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.233+0000 7fd06d800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0680d0540 con 0x7fd06815d570 2024-01-15T18:23:00.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.233+0000 7fd06d800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0680b0440 con 0x7fd06815d190 2024-01-15T18:23:00.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.233+0000 7fd066000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd06815d570 0x7fd06806be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:00.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.233+0000 7fd066000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd06815d570 0x7fd06806be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40296/0 (socket says 172.21.15.27:40296) 2024-01-15T18:23:00.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.233+0000 7fd066000640 1 -- 172.21.15.27:0/1433887208 learned_addr learned my addr 172.21.15.27:0/1433887208 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:00.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.233+0000 7fd067400640 1 --1- 172.21.15.27:0/1433887208 >> v1:172.21.15.27:6789/0 conn(0x7fd06806b810 0x7fd0680cd5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51952/0 (socket says 172.21.15.27:51952) 2024-01-15T18:23:00.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.233+0000 7fd066000640 1 -- 172.21.15.27:0/1433887208 >> v2:172.21.15.93:3300/0 conn(0x7fd06815d190 msgr2=0x7fd0680cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:00.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.233+0000 7fd066a00640 1 --2- 172.21.15.27:0/1433887208 >> v2:172.21.15.93:3300/0 conn(0x7fd06815d190 0x7fd0680cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:00.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.233+0000 7fd066000640 1 --2- 172.21.15.27:0/1433887208 >> v2:172.21.15.93:3300/0 conn(0x7fd06815d190 0x7fd0680cfcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:00.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.233+0000 7fd065600640 1 -- 172.21.15.27:0/1433887208 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3211637437 0 0) 0x7fd0680b0d70 con 0x7fd06806b810 2024-01-15T18:23:00.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.233+0000 7fd066000640 1 -- 172.21.15.27:0/1433887208 >> v1:172.21.15.27:6789/0 conn(0x7fd06806b810 legacy=0x7fd0680cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:00.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.233+0000 7fd066000640 1 -- 172.21.15.27:0/1433887208 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0680b0720 con 0x7fd06815d570 2024-01-15T18:23:00.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.233+0000 7fd066a00640 1 --2- 172.21.15.27:0/1433887208 >> v2:172.21.15.93:3300/0 conn(0x7fd06815d190 0x7fd0680cfcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:23:00.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.237+0000 7fd066000640 1 --2- 172.21.15.27:0/1433887208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd06815d570 0x7fd06806be10 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fd068071f90 tx=0x7fd05001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3fd0c62e47eb15f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:00.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.237+0000 7fd065600640 1 -- 172.21.15.27:0/1433887208 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0500ac640 con 0x7fd06815d570 2024-01-15T18:23:00.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.237+0000 7fd065600640 1 -- 172.21.15.27:0/1433887208 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0500ac7a0 con 0x7fd06815d570 2024-01-15T18:23:00.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.237+0000 7fd065600640 1 -- 172.21.15.27:0/1433887208 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0500acb30 con 0x7fd06815d570 2024-01-15T18:23:00.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.237+0000 7fd06d800640 1 -- 172.21.15.27:0/1433887208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd06815d570 msgr2=0x7fd06806be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.241+0000 7fd06d800640 1 --2- 172.21.15.27:0/1433887208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd06815d570 0x7fd06806be10 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fd068071f90 tx=0x7fd05001c4d0 comp rx=0 tx=0).stop 2024-01-15T18:23:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.241+0000 7fd06d800640 1 -- 172.21.15.27:0/1433887208 shutdown_connections 2024-01-15T18:23:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.241+0000 7fd06d800640 1 --2- 172.21.15.27:0/1433887208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd06815d570 0x7fd06806be10 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.241+0000 7fd06d800640 1 --2- 172.21.15.27:0/1433887208 >> v2:172.21.15.93:3300/0 conn(0x7fd06815d190 0x7fd0680cfcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:00.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.241+0000 7fd06d800640 1 -- 172.21.15.27:0/1433887208 >> 172.21.15.27:0/1433887208 conn(0x7fd0680d0e70 msgr2=0x7fd0680d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:00.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.241+0000 7fd06d800640 1 -- 172.21.15.27:0/1433887208 shutdown_connections 2024-01-15T18:23:00.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.241+0000 7fd06d800640 1 -- 172.21.15.27:0/1433887208 wait complete. 2024-01-15T18:23:00.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.241+0000 7fd06d800640 1 Processor -- start 2024-01-15T18:23:00.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.241+0000 7fd06d800640 1 -- start start 2024-01-15T18:23:00.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.241+0000 7fd06d800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd06806b810 0x7fd068150580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:00.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.241+0000 7fd06d800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd06815d190 0x7fd06815b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:00.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.241+0000 7fd06d800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd06815d570 0x7fd06815b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:00.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.241+0000 7fd06d800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd068150ac0 con 0x7fd06815d570 2024-01-15T18:23:00.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.241+0000 7fd06d800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0681543f0 con 0x7fd06815d190 2024-01-15T18:23:00.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.241+0000 7fd06d800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0681546d0 con 0x7fd06806b810 2024-01-15T18:23:00.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.241+0000 7fd067400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd06815d570 0x7fd06815b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:00.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.241+0000 7fd067400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd06815d570 0x7fd06815b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40298/0 (socket says 172.21.15.27:40298) 2024-01-15T18:23:00.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.241+0000 7fd067400640 1 -- 172.21.15.27:0/2606405450 learned_addr learned my addr 172.21.15.27:0/2606405450 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:00.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.241+0000 7fd066000640 1 --2- 172.21.15.27:0/2606405450 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd06815d190 0x7fd06815b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:00.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.241+0000 7fd066a00640 1 --2- 172.21.15.27:0/2606405450 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd06806b810 0x7fd068150580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:00.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.241+0000 7fd067400640 1 -- 172.21.15.27:0/2606405450 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd06806b810 msgr2=0x7fd068150580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:00.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.241+0000 7fd067400640 1 --2- 172.21.15.27:0/2606405450 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fd06806b810 0x7fd068150580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:00.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.241+0000 7fd067400640 1 -- 172.21.15.27:0/2606405450 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd06815d190 msgr2=0x7fd06815b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:00.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.241+0000 7fd067400640 1 --2- 172.21.15.27:0/2606405450 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fd06815d190 0x7fd06815b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:00.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.241+0000 7fd067400640 1 -- 172.21.15.27:0/2606405450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0500094d0 con 0x7fd06815d570 2024-01-15T18:23:00.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.245+0000 7fd067400640 1 --2- 172.21.15.27:0/2606405450 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd06815d570 0x7fd06815b730 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fd060001570 tx=0x7fd06000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:00.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.245+0000 7fd05ea00640 1 -- 172.21.15.27:0/2606405450 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06000a640 con 0x7fd06815d570 2024-01-15T18:23:00.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.245+0000 7fd05ea00640 1 -- 172.21.15.27:0/2606405450 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06000a7c0 con 0x7fd06815d570 2024-01-15T18:23:00.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.245+0000 7fd05ea00640 1 -- 172.21.15.27:0/2606405450 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06000ab70 con 0x7fd06815d570 2024-01-15T18:23:00.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.245+0000 7fd06d800640 1 -- 172.21.15.27:0/2606405450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0681549b0 con 0x7fd06815d570 2024-01-15T18:23:00.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.245+0000 7fd06d800640 1 -- 172.21.15.27:0/2606405450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd068154f70 con 0x7fd06815d570 2024-01-15T18:23:00.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.245+0000 7fd06d800640 1 -- 172.21.15.27:0/2606405450 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd068155160 con 0x7fd06815d570 2024-01-15T18:23:00.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.245+0000 7fd05ea00640 1 -- 172.21.15.27:0/2606405450 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7fd06000acd0 con 0x7fd06815d570 2024-01-15T18:23:00.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.245+0000 7fd05ea00640 1 --2- 172.21.15.27:0/2606405450 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd040067a60 0x7fd040069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:00.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.245+0000 7fd05ea00640 1 -- 172.21.15.27:0/2606405450 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fd060082880 con 0x7fd06815d570 2024-01-15T18:23:00.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.249+0000 7fd066a00640 1 --2- 172.21.15.27:0/2606405450 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd040067a60 0x7fd040069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:00.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.249+0000 7fd066a00640 1 --2- 172.21.15.27:0/2606405450 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fd040067a60 0x7fd040069f20 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fd05800ab60 tx=0x7fd058008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:00.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.249+0000 7fd05ea00640 1 -- 172.21.15.27:0/2606405450 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fd060051270 con 0x7fd06815d570 2024-01-15T18:23:00.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.437+0000 7fd06d800640 1 -- 172.21.15.27:0/2606405450 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd0680cd450 con 0x7fd040067a60 2024-01-15T18:23:00.441 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:23:00.441+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:23:00.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:00.445+0000 7fd05ea00640 1 -- 172.21.15.27:0/2606405450 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd0680cd450 con 0x7fd040067a60 2024-01-15T18:23:00.442 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:23:00.442 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:23:00.450 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:23:02.452 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:23:02.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.613+0000 7f766dc00640 1 Processor -- start 2024-01-15T18:23:02.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.613+0000 7f766dc00640 1 -- start start 2024-01-15T18:23:02.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f766dc00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f766809a120 0x7f7668168260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:02.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f766dc00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f766815d860 con 0x7f766806c740 2024-01-15T18:23:02.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f766dc00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f766806b6a0 con 0x7f766815d480 2024-01-15T18:23:02.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f766dc00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f766806b980 con 0x7f766809a120 2024-01-15T18:23:02.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7666000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f766806c740 0x7f7668099a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51964/0 (socket says 172.21.15.27:51964) 2024-01-15T18:23:02.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7666000640 1 -- 172.21.15.27:0/501404167 learned_addr learned my addr 172.21.15.27:0/501404167 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:02.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7667400640 1 --2- 172.21.15.27:0/501404167 >> v2:172.21.15.93:3300/0 conn(0x7f766809a120 0x7f7668168260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:02.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7665600640 1 -- 172.21.15.27:0/501404167 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893586604 0 0) 0x7f766815d860 con 0x7f766806c740 2024-01-15T18:23:02.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7665600640 1 -- 172.21.15.27:0/501404167 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7648003530 con 0x7f766806c740 2024-01-15T18:23:02.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7665600640 1 -- 172.21.15.27:0/501404167 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2486630373 0 0) 0x7f7648003530 con 0x7f766806c740 2024-01-15T18:23:02.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7665600640 1 -- 172.21.15.27:0/501404167 >> v2:172.21.15.93:3300/0 conn(0x7f766809a120 msgr2=0x7f7668168260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:02.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7665600640 1 --2- 172.21.15.27:0/501404167 >> v2:172.21.15.93:3300/0 conn(0x7f766809a120 0x7f7668168260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:02.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7665600640 1 -- 172.21.15.27:0/501404167 >> v1:172.21.15.93:6789/0 conn(0x7f766815d480 legacy=0x7f76680cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:02.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7665600640 1 -- 172.21.15.27:0/501404167 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f766806bc60 con 0x7f766806c740 2024-01-15T18:23:02.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7665600640 1 -- 172.21.15.27:0/501404167 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f7650001680 con 0x7f766806c740 2024-01-15T18:23:02.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7665600640 1 -- 172.21.15.27:0/501404167 >> v1:172.21.15.27:6789/0 conn(0x7f766806c740 legacy=0x7f7668099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:02.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7665600640 1 --2- 172.21.15.27:0/501404167 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f76480039f0 0x7f7648003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:02.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7665600640 1 --2- 172.21.15.27:0/501404167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76480047d0 0x7f7648004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:02.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7665600640 1 --2- 172.21.15.27:0/501404167 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7648005190 0x7f764800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:02.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7665600640 1 -- 172.21.15.27:0/501404167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766806b980 con 0x7f76480047d0 2024-01-15T18:23:02.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7665600640 1 -- 172.21.15.27:0/501404167 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764800c0c0 con 0x7f7648005190 2024-01-15T18:23:02.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7665600640 1 -- 172.21.15.27:0/501404167 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f764800c280 con 0x7f76480039f0 2024-01-15T18:23:02.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7666000640 1 --2- 172.21.15.27:0/501404167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76480047d0 0x7f7648004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:02.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7666a00640 1 --2- 172.21.15.27:0/501404167 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f76480039f0 0x7f7648003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:02.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7667400640 1 --2- 172.21.15.27:0/501404167 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7648005190 0x7f764800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:02.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7666000640 1 -- 172.21.15.27:0/501404167 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f76480039f0 msgr2=0x7f7648003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:02.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7666000640 1 --2- 172.21.15.27:0/501404167 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f76480039f0 0x7f7648003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:02.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7666000640 1 -- 172.21.15.27:0/501404167 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7648005190 msgr2=0x7f764800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:02.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7666000640 1 --2- 172.21.15.27:0/501404167 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7648005190 0x7f764800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:02.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7666000640 1 -- 172.21.15.27:0/501404167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7648003530 con 0x7f76480047d0 2024-01-15T18:23:02.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.617+0000 7f7667400640 1 -- 172.21.15.27:0/501404167 reap_dead start 2024-01-15T18:23:02.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.621+0000 7f7666000640 1 --2- 172.21.15.27:0/501404167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76480047d0 0x7f7648004c50 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f765000c240 tx=0x7f765001f330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96b773d0f14dd38e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:02.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.621+0000 7f7665600640 1 -- 172.21.15.27:0/501404167 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7650005ae0 con 0x7f76480047d0 2024-01-15T18:23:02.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.621+0000 7f7665600640 1 -- 172.21.15.27:0/501404167 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76500ab070 con 0x7f76480047d0 2024-01-15T18:23:02.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.621+0000 7f7665600640 1 -- 172.21.15.27:0/501404167 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7650007070 con 0x7f76480047d0 2024-01-15T18:23:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.621+0000 7f766dc00640 1 -- 172.21.15.27:0/501404167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76480047d0 msgr2=0x7f7648004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.621+0000 7f766dc00640 1 --2- 172.21.15.27:0/501404167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76480047d0 0x7f7648004c50 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f765000c240 tx=0x7f765001f330 comp rx=0 tx=0).stop 2024-01-15T18:23:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.621+0000 7f766dc00640 1 -- 172.21.15.27:0/501404167 shutdown_connections 2024-01-15T18:23:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.621+0000 7f766dc00640 1 --2- 172.21.15.27:0/501404167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76480047d0 0x7f7648004c50 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.621+0000 7f766dc00640 1 -- 172.21.15.27:0/501404167 >> 172.21.15.27:0/501404167 conn(0x7f76680b2a80 msgr2=0x7f76680b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.621+0000 7f766dc00640 1 -- 172.21.15.27:0/501404167 shutdown_connections 2024-01-15T18:23:02.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.621+0000 7f766dc00640 1 -- 172.21.15.27:0/501404167 wait complete. 2024-01-15T18:23:02.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.625+0000 7f766dc00640 1 Processor -- start 2024-01-15T18:23:02.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.625+0000 7f766dc00640 1 -- start start 2024-01-15T18:23:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.625+0000 7f766dc00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f76480047d0 0x7f76680c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.625+0000 7f766dc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f766815e420 0x7f7668157990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.625+0000 7f766dc00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76680b5a90 0x7f76680b5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.625+0000 7f766dc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76680b6450 con 0x7f766815e420 2024-01-15T18:23:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.625+0000 7f766dc00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76681584c0 con 0x7f76680b5a90 2024-01-15T18:23:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.625+0000 7f766dc00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76680c7880 con 0x7f76480047d0 2024-01-15T18:23:02.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.625+0000 7f7666000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f766815e420 0x7f7668157990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:02.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.625+0000 7f7666000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f766815e420 0x7f7668157990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40324/0 (socket says 172.21.15.27:40324) 2024-01-15T18:23:02.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.625+0000 7f7666000640 1 -- 172.21.15.27:0/715322647 learned_addr learned my addr 172.21.15.27:0/715322647 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:02.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.625+0000 7f7667400640 1 --2- 172.21.15.27:0/715322647 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76680b5a90 0x7f76680b5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:02.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.625+0000 7f7666a00640 1 --2- 172.21.15.27:0/715322647 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f76480047d0 0x7f76680c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:02.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.625+0000 7f7666000640 1 -- 172.21.15.27:0/715322647 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f76480047d0 msgr2=0x7f76680c60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:02.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.625+0000 7f7666000640 1 --2- 172.21.15.27:0/715322647 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f76480047d0 0x7f76680c60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:02.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.625+0000 7f7666000640 1 -- 172.21.15.27:0/715322647 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76680b5a90 msgr2=0x7f76680b5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:02.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.625+0000 7f7666000640 1 --2- 172.21.15.27:0/715322647 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f76680b5a90 0x7f76680b5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:02.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.625+0000 7f7666000640 1 -- 172.21.15.27:0/715322647 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76680c67e0 con 0x7f766815e420 2024-01-15T18:23:02.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.625+0000 7f7666000640 1 --2- 172.21.15.27:0/715322647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f766815e420 0x7f7668157990 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f765001f160 tx=0x7f7648005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:02.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.625+0000 7f765ea00640 1 -- 172.21.15.27:0/715322647 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7650001680 con 0x7f766815e420 2024-01-15T18:23:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.625+0000 7f765ea00640 1 -- 172.21.15.27:0/715322647 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7650007070 con 0x7f766815e420 2024-01-15T18:23:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.625+0000 7f765ea00640 1 -- 172.21.15.27:0/715322647 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7650007400 con 0x7f766815e420 2024-01-15T18:23:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.625+0000 7f766dc00640 1 -- 172.21.15.27:0/715322647 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76680c6ac0 con 0x7f766815e420 2024-01-15T18:23:02.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.625+0000 7f766dc00640 1 -- 172.21.15.27:0/715322647 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76680c7080 con 0x7f766815e420 2024-01-15T18:23:02.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.625+0000 7f766dc00640 1 -- 172.21.15.27:0/715322647 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f766804e850 con 0x7f766815e420 2024-01-15T18:23:02.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.629+0000 7f765ea00640 1 -- 172.21.15.27:0/715322647 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f76500b9030 con 0x7f766815e420 2024-01-15T18:23:02.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.629+0000 7f765ea00640 1 --2- 172.21.15.27:0/715322647 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7640067a60 0x7f7640069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:02.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.629+0000 7f765ea00640 1 -- 172.21.15.27:0/715322647 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f76500b7070 con 0x7f766815e420 2024-01-15T18:23:02.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.629+0000 7f7666a00640 1 --2- 172.21.15.27:0/715322647 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7640067a60 0x7f7640069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:02.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.629+0000 7f7666a00640 1 --2- 172.21.15.27:0/715322647 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7640067a60 0x7f7640069f20 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f7658000c10 tx=0x7f7658009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:02.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.633+0000 7f765ea00640 1 -- 172.21.15.27:0/715322647 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f76500eeac0 con 0x7f766815e420 2024-01-15T18:23:02.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.825+0000 7f766dc00640 1 -- 172.21.15.27:0/715322647 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7668067db0 con 0x7f7640067a60 2024-01-15T18:23:02.827 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:23:02.825+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:23:02.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:02.833+0000 7f765ea00640 1 -- 172.21.15.27:0/715322647 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7668067db0 con 0x7f7640067a60 2024-01-15T18:23:02.828 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:23:02.828 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:23:02.836 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:23:04.837 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:23:04.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.001+0000 7fea71600640 1 Processor -- start 2024-01-15T18:23:04.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.001+0000 7fea71600640 1 -- start start 2024-01-15T18:23:04.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.001+0000 7fea71600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fea6c15b4b0 0x7fea6c0b1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:04.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.001+0000 7fea71600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea6c0d19a0 con 0x7fea6c0cf460 2024-01-15T18:23:04.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.001+0000 7fea71600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea6c0d1c80 con 0x7fea6c0cec60 2024-01-15T18:23:04.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.001+0000 7fea71600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea6c0b07c0 con 0x7fea6c15b4b0 2024-01-15T18:23:04.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.001+0000 7fea6aa00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fea6c0cf460 0x7fea6c0cf840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57688/0 (socket says 172.21.15.27:57688) 2024-01-15T18:23:04.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.001+0000 7fea6aa00640 1 -- 172.21.15.27:0/582121613 learned_addr learned my addr 172.21.15.27:0/582121613 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:05.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.001+0000 7fea6a000640 1 --2- 172.21.15.27:0/582121613 >> v2:172.21.15.93:3300/0 conn(0x7fea6c15b4b0 0x7fea6c0b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:05.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.001+0000 7fea69600640 1 -- 172.21.15.27:0/582121613 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 105401678 0 0) 0x7fea6c0d19a0 con 0x7fea6c0cf460 2024-01-15T18:23:05.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.001+0000 7fea69600640 1 -- 172.21.15.27:0/582121613 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea4c003530 con 0x7fea6c0cf460 2024-01-15T18:23:05.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.001+0000 7fea69600640 1 -- 172.21.15.27:0/582121613 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2827378305 0 0) 0x7fea4c003530 con 0x7fea6c0cf460 2024-01-15T18:23:05.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.001+0000 7fea69600640 1 -- 172.21.15.27:0/582121613 >> v2:172.21.15.93:3300/0 conn(0x7fea6c15b4b0 msgr2=0x7fea6c0b1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:05.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.001+0000 7fea69600640 1 --2- 172.21.15.27:0/582121613 >> v2:172.21.15.93:3300/0 conn(0x7fea6c15b4b0 0x7fea6c0b1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:05.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.001+0000 7fea69600640 1 -- 172.21.15.27:0/582121613 >> v1:172.21.15.93:6789/0 conn(0x7fea6c0cec60 legacy=0x7fea6c0b1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:05.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.001+0000 7fea69600640 1 -- 172.21.15.27:0/582121613 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea6c0b0aa0 con 0x7fea6c0cf460 2024-01-15T18:23:05.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.001+0000 7fea69600640 1 -- 172.21.15.27:0/582121613 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fea5c001680 con 0x7fea6c0cf460 2024-01-15T18:23:05.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.001+0000 7fea69600640 1 -- 172.21.15.27:0/582121613 >> v1:172.21.15.27:6789/0 conn(0x7fea6c0cf460 legacy=0x7fea6c0cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:05.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.001+0000 7fea69600640 1 --2- 172.21.15.27:0/582121613 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fea4c003a40 0x7fea4c003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:05.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.001+0000 7fea69600640 1 --2- 172.21.15.27:0/582121613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea4c0047d0 0x7fea4c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:05.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.005+0000 7fea69600640 1 --2- 172.21.15.27:0/582121613 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fea4c0051e0 0x7fea4c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:05.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.005+0000 7fea69600640 1 -- 172.21.15.27:0/582121613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c0b07c0 con 0x7fea4c0047d0 2024-01-15T18:23:05.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.005+0000 7fea69600640 1 -- 172.21.15.27:0/582121613 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea4c00c110 con 0x7fea4c003a40 2024-01-15T18:23:05.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.005+0000 7fea69600640 1 -- 172.21.15.27:0/582121613 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea4c00c2d0 con 0x7fea4c0051e0 2024-01-15T18:23:05.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.005+0000 7fea6a000640 1 --2- 172.21.15.27:0/582121613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea4c0047d0 0x7fea4c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:05.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.005+0000 7fea6aa00640 1 --2- 172.21.15.27:0/582121613 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fea4c003a40 0x7fea4c003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:05.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.005+0000 7fea6b400640 1 --2- 172.21.15.27:0/582121613 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fea4c0051e0 0x7fea4c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:05.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.005+0000 7fea6a000640 1 -- 172.21.15.27:0/582121613 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fea4c0051e0 msgr2=0x7fea4c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:05.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.005+0000 7fea6a000640 1 --2- 172.21.15.27:0/582121613 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fea4c0051e0 0x7fea4c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:05.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.005+0000 7fea6a000640 1 -- 172.21.15.27:0/582121613 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fea4c003a40 msgr2=0x7fea4c003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:05.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.005+0000 7fea6a000640 1 --2- 172.21.15.27:0/582121613 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fea4c003a40 0x7fea4c003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:05.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.005+0000 7fea6a000640 1 -- 172.21.15.27:0/582121613 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea4c003530 con 0x7fea4c0047d0 2024-01-15T18:23:05.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.005+0000 7fea6b400640 1 -- 172.21.15.27:0/582121613 reap_dead start 2024-01-15T18:23:05.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.005+0000 7fea6a000640 1 --2- 172.21.15.27:0/582121613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea4c0047d0 0x7fea4c004c50 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fea54009ae0 tx=0x7fea5401ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=debcf78d9550ae03 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:05.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.005+0000 7fea69600640 1 -- 172.21.15.27:0/582121613 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea540a5550 con 0x7fea4c0047d0 2024-01-15T18:23:05.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.005+0000 7fea69600640 1 -- 172.21.15.27:0/582121613 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea540a56b0 con 0x7fea4c0047d0 2024-01-15T18:23:05.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.005+0000 7fea69600640 1 -- 172.21.15.27:0/582121613 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea540a5a60 con 0x7fea4c0047d0 2024-01-15T18:23:05.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.009+0000 7fea71600640 1 -- 172.21.15.27:0/582121613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea4c0047d0 msgr2=0x7fea4c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:05.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.009+0000 7fea71600640 1 --2- 172.21.15.27:0/582121613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea4c0047d0 0x7fea4c004c50 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fea54009ae0 tx=0x7fea5401ca70 comp rx=0 tx=0).stop 2024-01-15T18:23:05.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.009+0000 7fea71600640 1 -- 172.21.15.27:0/582121613 shutdown_connections 2024-01-15T18:23:05.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.009+0000 7fea71600640 1 --2- 172.21.15.27:0/582121613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea4c0047d0 0x7fea4c004c50 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:05.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.009+0000 7fea71600640 1 -- 172.21.15.27:0/582121613 >> 172.21.15.27:0/582121613 conn(0x7fea6c0d0e80 msgr2=0x7fea6c0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:05.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.009+0000 7fea71600640 1 -- 172.21.15.27:0/582121613 shutdown_connections 2024-01-15T18:23:05.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.009+0000 7fea71600640 1 -- 172.21.15.27:0/582121613 wait complete. 2024-01-15T18:23:05.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.009+0000 7fea71600640 1 Processor -- start 2024-01-15T18:23:05.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.009+0000 7fea71600640 1 -- start start 2024-01-15T18:23:05.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.009+0000 7fea71600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea4c0047d0 0x7fea6c14b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:05.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.009+0000 7fea71600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fea6c0cadf0 0x7fea6c09b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:05.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.009+0000 7fea71600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fea6c09bd20 0x7fea6c14bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:05.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.009+0000 7fea71600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c09b3c0 con 0x7fea4c0047d0 2024-01-15T18:23:05.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.009+0000 7fea71600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c14b880 con 0x7fea6c09bd20 2024-01-15T18:23:05.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.009+0000 7fea71600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c0caab0 con 0x7fea6c0cadf0 2024-01-15T18:23:05.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.009+0000 7fea6aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea4c0047d0 0x7fea6c14b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:05.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.013+0000 7fea6aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea4c0047d0 0x7fea6c14b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52496/0 (socket says 172.21.15.27:52496) 2024-01-15T18:23:05.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.013+0000 7fea6aa00640 1 -- 172.21.15.27:0/937047620 learned_addr learned my addr 172.21.15.27:0/937047620 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:05.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.013+0000 7fea6b400640 1 --2- 172.21.15.27:0/937047620 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fea6c09bd20 0x7fea6c14bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:05.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.013+0000 7fea6a000640 1 --2- 172.21.15.27:0/937047620 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fea6c0cadf0 0x7fea6c09b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:05.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.013+0000 7fea6aa00640 1 -- 172.21.15.27:0/937047620 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fea6c0cadf0 msgr2=0x7fea6c09b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:05.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.013+0000 7fea6aa00640 1 --2- 172.21.15.27:0/937047620 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fea6c0cadf0 0x7fea6c09b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:05.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.013+0000 7fea6aa00640 1 -- 172.21.15.27:0/937047620 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fea6c09bd20 msgr2=0x7fea6c14bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:05.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.013+0000 7fea6aa00640 1 --2- 172.21.15.27:0/937047620 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fea6c09bd20 0x7fea6c14bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:05.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.013+0000 7fea6aa00640 1 -- 172.21.15.27:0/937047620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea6c0ca7d0 con 0x7fea4c0047d0 2024-01-15T18:23:05.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.013+0000 7fea6aa00640 1 --2- 172.21.15.27:0/937047620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea4c0047d0 0x7fea6c14b340 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fea5c0122b0 tx=0x7fea5c012770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:05.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.013+0000 7fea62a00640 1 -- 172.21.15.27:0/937047620 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c014750 con 0x7fea4c0047d0 2024-01-15T18:23:05.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.013+0000 7fea62a00640 1 -- 172.21.15.27:0/937047620 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea5c0148d0 con 0x7fea4c0047d0 2024-01-15T18:23:05.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.013+0000 7fea71600640 1 -- 172.21.15.27:0/937047620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea6c09ad00 con 0x7fea4c0047d0 2024-01-15T18:23:05.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.013+0000 7fea62a00640 1 -- 172.21.15.27:0/937047620 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c014c80 con 0x7fea4c0047d0 2024-01-15T18:23:05.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.013+0000 7fea71600640 1 -- 172.21.15.27:0/937047620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea6c0ca460 con 0x7fea4c0047d0 2024-01-15T18:23:05.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.013+0000 7fea71600640 1 -- 172.21.15.27:0/937047620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea6c04e850 con 0x7fea4c0047d0 2024-01-15T18:23:05.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.013+0000 7fea62a00640 1 -- 172.21.15.27:0/937047620 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7fea5c0153e0 con 0x7fea4c0047d0 2024-01-15T18:23:05.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.017+0000 7fea62a00640 1 --2- 172.21.15.27:0/937047620 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fea44067a90 0x7fea44069f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:05.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.017+0000 7fea62a00640 1 -- 172.21.15.27:0/937047620 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fea5c085f00 con 0x7fea4c0047d0 2024-01-15T18:23:05.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.017+0000 7fea6a000640 1 --2- 172.21.15.27:0/937047620 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fea44067a90 0x7fea44069f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:05.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.017+0000 7fea6a000640 1 --2- 172.21.15.27:0/937047620 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fea44067a90 0x7fea44069f50 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fea540af000 tx=0x7fea540046d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:05.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.021+0000 7fea62a00640 1 -- 172.21.15.27:0/937047620 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fea5c054860 con 0x7fea4c0047d0 2024-01-15T18:23:05.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.209+0000 7fea71600640 1 -- 172.21.15.27:0/937047620 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fea6c0b1060 con 0x7fea44067a90 2024-01-15T18:23:05.211 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:23:05.209+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:23:05.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:05.213+0000 7fea62a00640 1 -- 172.21.15.27:0/937047620 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fea540bd0f0 con 0x7fea44067a90 2024-01-15T18:23:05.212 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:23:05.212 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:23:05.220 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:23:07.221 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:23:07.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.385+0000 7f0c67200640 1 Processor -- start 2024-01-15T18:23:07.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.385+0000 7f0c67200640 1 -- start start 2024-01-15T18:23:07.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.385+0000 7f0c67200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0c6015d190 0x7f0c600cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:07.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.385+0000 7f0c67200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0c6015d570 0x7f0c6006be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:07.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.385+0000 7f0c67200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c600b0d70 con 0x7f0c6006b810 2024-01-15T18:23:07.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.385+0000 7f0c67200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c600d0540 con 0x7f0c6015d570 2024-01-15T18:23:07.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.385+0000 7f0c67200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c600b0440 con 0x7f0c6015d190 2024-01-15T18:23:07.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.385+0000 7f0c5f400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0c6015d570 0x7f0c6006be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:07.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.385+0000 7f0c5fe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0c6015d190 0x7f0c600cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:07.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.385+0000 7f0c5f400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0c6015d570 0x7f0c6006be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:45372/0 (socket says 172.21.15.27:45372) 2024-01-15T18:23:07.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.385+0000 7f0c5f400640 1 -- 172.21.15.27:0/375108686 learned_addr learned my addr 172.21.15.27:0/375108686 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:07.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.385+0000 7f0c5fe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0c6015d190 0x7f0c600cfcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:56794/0 (socket says 172.21.15.27:56794) 2024-01-15T18:23:07.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.385+0000 7f0c5fe00640 1 -- 172.21.15.27:0/375108686 >> v2:172.21.15.93:3300/0 conn(0x7f0c6015d570 msgr2=0x7f0c6006be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:07.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.385+0000 7f0c5fe00640 1 --2- 172.21.15.27:0/375108686 >> v2:172.21.15.93:3300/0 conn(0x7f0c6015d570 0x7f0c6006be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:07.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.385+0000 7f0c5fe00640 1 -- 172.21.15.27:0/375108686 >> v1:172.21.15.93:6789/0 conn(0x7f0c6006b810 legacy=0x7f0c600cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:07.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.385+0000 7f0c5fe00640 1 -- 172.21.15.27:0/375108686 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c600b0720 con 0x7f0c6015d190 2024-01-15T18:23:07.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.389+0000 7f0c5fe00640 1 --2- 172.21.15.27:0/375108686 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0c6015d190 0x7f0c600cfcf0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0c500097c0 tx=0x7f0c5001c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be93d3cf3c213928 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:07.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.389+0000 7f0c5ea00640 1 -- 172.21.15.27:0/375108686 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c500ad660 con 0x7f0c6015d190 2024-01-15T18:23:07.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.389+0000 7f0c5ea00640 1 -- 172.21.15.27:0/375108686 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c500ad7c0 con 0x7f0c6015d190 2024-01-15T18:23:07.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.389+0000 7f0c5ea00640 1 -- 172.21.15.27:0/375108686 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c500adb50 con 0x7f0c6015d190 2024-01-15T18:23:07.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.389+0000 7f0c67200640 1 -- 172.21.15.27:0/375108686 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0c6015d190 msgr2=0x7f0c600cfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:07.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.389+0000 7f0c67200640 1 --2- 172.21.15.27:0/375108686 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0c6015d190 0x7f0c600cfcf0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0c500097c0 tx=0x7f0c5001c450 comp rx=0 tx=0).stop 2024-01-15T18:23:07.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.389+0000 7f0c67200640 1 -- 172.21.15.27:0/375108686 shutdown_connections 2024-01-15T18:23:07.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.389+0000 7f0c67200640 1 --2- 172.21.15.27:0/375108686 >> v2:172.21.15.93:3300/0 conn(0x7f0c6015d570 0x7f0c6006be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:07.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.389+0000 7f0c67200640 1 --2- 172.21.15.27:0/375108686 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0c6015d190 0x7f0c600cfcf0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:07.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.389+0000 7f0c67200640 1 -- 172.21.15.27:0/375108686 >> 172.21.15.27:0/375108686 conn(0x7f0c600d0e70 msgr2=0x7f0c600d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:07.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.389+0000 7f0c67200640 1 -- 172.21.15.27:0/375108686 shutdown_connections 2024-01-15T18:23:07.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.389+0000 7f0c67200640 1 -- 172.21.15.27:0/375108686 wait complete. 2024-01-15T18:23:07.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.393+0000 7f0c67200640 1 Processor -- start 2024-01-15T18:23:07.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.393+0000 7f0c67200640 1 -- start start 2024-01-15T18:23:07.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.393+0000 7f0c67200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0c6006b810 0x7f0c600c74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:07.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.393+0000 7f0c67200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c6015d190 0x7f0c6014e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:07.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.393+0000 7f0c67200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0c6015d570 0x7f0c6014f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:07.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.393+0000 7f0c67200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c60154f80 con 0x7f0c6015d190 2024-01-15T18:23:07.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.393+0000 7f0c67200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c600c6120 con 0x7f0c6015d570 2024-01-15T18:23:07.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.393+0000 7f0c67200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c60154990 con 0x7f0c6006b810 2024-01-15T18:23:07.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.393+0000 7f0c5f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c6015d190 0x7f0c6014e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:07.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.393+0000 7f0c5f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c6015d190 0x7f0c6014e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52512/0 (socket says 172.21.15.27:52512) 2024-01-15T18:23:07.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.393+0000 7f0c5f400640 1 -- 172.21.15.27:0/3100010231 learned_addr learned my addr 172.21.15.27:0/3100010231 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:07.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.393+0000 7f0c5fe00640 1 --2- 172.21.15.27:0/3100010231 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0c6006b810 0x7f0c600c74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:07.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.393+0000 7f0c65200640 1 --2- 172.21.15.27:0/3100010231 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0c6015d570 0x7f0c6014f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:07.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.393+0000 7f0c5f400640 1 -- 172.21.15.27:0/3100010231 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0c6006b810 msgr2=0x7f0c600c74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:07.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.393+0000 7f0c5f400640 1 --2- 172.21.15.27:0/3100010231 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0c6006b810 0x7f0c600c74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:07.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.393+0000 7f0c5f400640 1 -- 172.21.15.27:0/3100010231 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0c6015d570 msgr2=0x7f0c6014f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:07.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.393+0000 7f0c5f400640 1 --2- 172.21.15.27:0/3100010231 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0c6015d570 0x7f0c6014f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:07.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.393+0000 7f0c5f400640 1 -- 172.21.15.27:0/3100010231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c50009420 con 0x7f0c6015d190 2024-01-15T18:23:07.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.393+0000 7f0c5f400640 1 --2- 172.21.15.27:0/3100010231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c6015d190 0x7f0c6014e3d0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f0c4800e7f0 tx=0x7f0c4800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:07.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.393+0000 7f0c57e00640 1 -- 172.21.15.27:0/3100010231 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c48009580 con 0x7f0c6015d190 2024-01-15T18:23:07.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.393+0000 7f0c57e00640 1 -- 172.21.15.27:0/3100010231 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4800c940 con 0x7f0c6015d190 2024-01-15T18:23:07.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.393+0000 7f0c67200640 1 -- 172.21.15.27:0/3100010231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c600c5b60 con 0x7f0c6015d190 2024-01-15T18:23:07.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.393+0000 7f0c57e00640 1 -- 172.21.15.27:0/3100010231 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4800ccf0 con 0x7f0c6015d190 2024-01-15T18:23:07.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.397+0000 7f0c67200640 1 -- 172.21.15.27:0/3100010231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c6014e910 con 0x7f0c6015d190 2024-01-15T18:23:07.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.397+0000 7f0c67200640 1 -- 172.21.15.27:0/3100010231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c6004e850 con 0x7f0c6015d190 2024-01-15T18:23:07.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.397+0000 7f0c57e00640 1 -- 172.21.15.27:0/3100010231 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f0c48013470 con 0x7f0c6015d190 2024-01-15T18:23:07.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.397+0000 7f0c57e00640 1 --2- 172.21.15.27:0/3100010231 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c38067a60 0x7f0c38069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:07.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.397+0000 7f0c57e00640 1 -- 172.21.15.27:0/3100010231 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f0c480833f0 con 0x7f0c6015d190 2024-01-15T18:23:07.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.397+0000 7f0c5fe00640 1 --2- 172.21.15.27:0/3100010231 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c38067a60 0x7f0c38069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:07.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.401+0000 7f0c5fe00640 1 --2- 172.21.15.27:0/3100010231 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0c38067a60 0x7f0c38069f20 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f0c500af000 tx=0x7f0c50008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:07.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.401+0000 7f0c57e00640 1 -- 172.21.15.27:0/3100010231 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0c48051d60 con 0x7f0c6015d190 2024-01-15T18:23:07.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.589+0000 7f0c67200640 1 -- 172.21.15.27:0/3100010231 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0c60156640 con 0x7f0c38067a60 2024-01-15T18:23:07.593 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:23:07.593+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:23:07.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:07.601+0000 7f0c57e00640 1 -- 172.21.15.27:0/3100010231 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0c60156640 con 0x7f0c38067a60 2024-01-15T18:23:07.594 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:23:07.594 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:23:07.602 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:23:09.604 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:23:09.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.765+0000 7f4ec1800640 1 Processor -- start 2024-01-15T18:23:09.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.765+0000 7f4ec1800640 1 -- start start 2024-01-15T18:23:09.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.765+0000 7f4ec1800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f4ebc06c430 0x7f4ebc070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:09.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.765+0000 7f4ec1800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4ebc0ce760 0x7f4ebc0b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:09.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.765+0000 7f4ec1800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ebc0ccfb0 con 0x7f4ebc0b1db0 2024-01-15T18:23:09.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.765+0000 7f4ec1800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ebc0cd290 con 0x7f4ebc06c430 2024-01-15T18:23:09.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.765+0000 7f4ec1800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc0cd570 con 0x7f4ebc0ce760 2024-01-15T18:23:09.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.765+0000 7f4eba000640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4ebc0b1db0 0x7f4ebc0ce050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57694/0 (socket says 172.21.15.27:57694) 2024-01-15T18:23:09.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.765+0000 7f4eba000640 1 -- 172.21.15.27:0/903649452 learned_addr learned my addr 172.21.15.27:0/903649452 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:09.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.769+0000 7f4ebaa00640 1 --2- 172.21.15.27:0/903649452 >> v2:172.21.15.93:3300/0 conn(0x7f4ebc06c430 0x7f4ebc070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:09.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.769+0000 7f4ebb400640 1 --2- 172.21.15.27:0/903649452 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4ebc0ce760 0x7f4ebc0b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:09.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.769+0000 7f4eb9600640 1 -- 172.21.15.27:0/903649452 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3410311278 0 0) 0x7f4ebc0ccfb0 con 0x7f4ebc0b1db0 2024-01-15T18:23:09.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.769+0000 7f4eb9600640 1 -- 172.21.15.27:0/903649452 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e9c003530 con 0x7f4ebc0b1db0 2024-01-15T18:23:09.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.769+0000 7f4ebaa00640 1 -- 172.21.15.27:0/903649452 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4ebc0ce760 msgr2=0x7f4ebc0b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:09.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.769+0000 7f4ebaa00640 1 --2- 172.21.15.27:0/903649452 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4ebc0ce760 0x7f4ebc0b2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:09.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.769+0000 7f4ebaa00640 1 -- 172.21.15.27:0/903649452 >> v1:172.21.15.27:6789/0 conn(0x7f4ebc0b1db0 legacy=0x7f4ebc0ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:09.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.769+0000 7f4eb9600640 1 -- 172.21.15.27:0/903649452 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2682107831 0 0) 0x7f4e9c003530 con 0x7f4ebc0b1db0 2024-01-15T18:23:09.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.769+0000 7f4ebaa00640 1 -- 172.21.15.27:0/903649452 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ebc0cd850 con 0x7f4ebc06c430 2024-01-15T18:23:09.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.769+0000 7f4ebaa00640 1 --2- 172.21.15.27:0/903649452 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ebc06c430 0x7f4ebc070ba0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f4eac0097f0 tx=0x7f4eac01c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=140e9c85c65226f6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:09.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.769+0000 7f4eb9600640 1 -- 172.21.15.27:0/903649452 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea4001680 con 0x7f4ebc06c430 2024-01-15T18:23:09.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.769+0000 7f4eb9600640 1 -- 172.21.15.27:0/903649452 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eac0a7050 con 0x7f4ebc06c430 2024-01-15T18:23:09.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.769+0000 7f4eb9600640 1 -- 172.21.15.27:0/903649452 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eac0ab620 con 0x7f4ebc06c430 2024-01-15T18:23:09.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.773+0000 7f4ec1800640 1 -- 172.21.15.27:0/903649452 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ebc06c430 msgr2=0x7f4ebc070ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:09.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.773+0000 7f4ec1800640 1 --2- 172.21.15.27:0/903649452 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ebc06c430 0x7f4ebc070ba0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f4eac0097f0 tx=0x7f4eac01c2e0 comp rx=0 tx=0).stop 2024-01-15T18:23:09.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.773+0000 7f4ec1800640 1 -- 172.21.15.27:0/903649452 shutdown_connections 2024-01-15T18:23:09.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.773+0000 7f4ec1800640 1 --2- 172.21.15.27:0/903649452 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4ebc0ce760 0x7f4ebc0b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:09.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.773+0000 7f4ec1800640 1 --2- 172.21.15.27:0/903649452 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ebc06c430 0x7f4ebc070ba0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:09.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.773+0000 7f4ec1800640 1 -- 172.21.15.27:0/903649452 >> 172.21.15.27:0/903649452 conn(0x7f4ebc0cb700 msgr2=0x7f4ebc0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:09.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.773+0000 7f4ec1800640 1 -- 172.21.15.27:0/903649452 shutdown_connections 2024-01-15T18:23:09.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.773+0000 7f4ec1800640 1 -- 172.21.15.27:0/903649452 wait complete. 2024-01-15T18:23:09.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.773+0000 7f4ec1800640 1 Processor -- start 2024-01-15T18:23:09.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.773+0000 7f4ec1800640 1 -- start start 2024-01-15T18:23:09.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.773+0000 7f4ec1800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4ebc06c430 0x7f4ebc1456c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:09.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.773+0000 7f4ec1800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ebc0b1db0 0x7f4ebc15bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:09.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.773+0000 7f4ec1800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ebc0ce760 0x7f4ebc14e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:09.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.773+0000 7f4ec1800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc14e360 con 0x7f4ebc0ce760 2024-01-15T18:23:09.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.773+0000 7f4ec1800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc14d2c0 con 0x7f4ebc0b1db0 2024-01-15T18:23:09.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.773+0000 7f4ec1800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc14d5a0 con 0x7f4ebc06c430 2024-01-15T18:23:09.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.777+0000 7f4ebb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ebc0ce760 0x7f4ebc14e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:09.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.777+0000 7f4ebb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ebc0ce760 0x7f4ebc14e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52520/0 (socket says 172.21.15.27:52520) 2024-01-15T18:23:09.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.777+0000 7f4ebb400640 1 -- 172.21.15.27:0/1709812051 learned_addr learned my addr 172.21.15.27:0/1709812051 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:09.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.777+0000 7f4eba000640 1 --2- 172.21.15.27:0/1709812051 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ebc0b1db0 0x7f4ebc15bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:09.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.777+0000 7f4ebaa00640 1 --2- 172.21.15.27:0/1709812051 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4ebc06c430 0x7f4ebc1456c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:09.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.777+0000 7f4ebb400640 1 -- 172.21.15.27:0/1709812051 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4ebc06c430 msgr2=0x7f4ebc1456c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:09.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.777+0000 7f4ebb400640 1 --2- 172.21.15.27:0/1709812051 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f4ebc06c430 0x7f4ebc1456c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:09.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.777+0000 7f4ebb400640 1 -- 172.21.15.27:0/1709812051 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ebc0b1db0 msgr2=0x7f4ebc15bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:09.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.777+0000 7f4ebb400640 1 --2- 172.21.15.27:0/1709812051 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f4ebc0b1db0 0x7f4ebc15bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:09.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.777+0000 7f4ebb400640 1 -- 172.21.15.27:0/1709812051 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eac009420 con 0x7f4ebc0ce760 2024-01-15T18:23:09.773 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.777+0000 7f4ebb400640 1 --2- 172.21.15.27:0/1709812051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ebc0ce760 0x7f4ebc14e840 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f4eb400ef90 tx=0x7f4eb400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:09.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.777+0000 7f4eb2a00640 1 -- 172.21.15.27:0/1709812051 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb4009b20 con 0x7f4ebc0ce760 2024-01-15T18:23:09.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.777+0000 7f4eb2a00640 1 -- 172.21.15.27:0/1709812051 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb400f040 con 0x7f4ebc0ce760 2024-01-15T18:23:09.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.777+0000 7f4ec1800640 1 -- 172.21.15.27:0/1709812051 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ebc0712f0 con 0x7f4ebc0ce760 2024-01-15T18:23:09.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.777+0000 7f4eb2a00640 1 -- 172.21.15.27:0/1709812051 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb40146c0 con 0x7f4ebc0ce760 2024-01-15T18:23:09.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.777+0000 7f4ec1800640 1 -- 172.21.15.27:0/1709812051 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ebc14db60 con 0x7f4ebc0ce760 2024-01-15T18:23:09.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.777+0000 7f4ec1800640 1 -- 172.21.15.27:0/1709812051 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ebc04e850 con 0x7f4ebc0ce760 2024-01-15T18:23:09.775 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.777+0000 7f4eb2a00640 1 -- 172.21.15.27:0/1709812051 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f4eb4014820 con 0x7f4ebc0ce760 2024-01-15T18:23:09.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.781+0000 7f4eb2a00640 1 --2- 172.21.15.27:0/1709812051 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4e94067a90 0x7f4e94069f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:09.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.781+0000 7f4eb2a00640 1 -- 172.21.15.27:0/1709812051 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f4eb4083f20 con 0x7f4ebc0ce760 2024-01-15T18:23:09.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.781+0000 7f4ebaa00640 1 --2- 172.21.15.27:0/1709812051 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4e94067a90 0x7f4e94069f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:09.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.781+0000 7f4ebaa00640 1 --2- 172.21.15.27:0/1709812051 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4e94067a90 0x7f4e94069f50 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f4eac0ad000 tx=0x7f4eac0a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:09.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.785+0000 7f4eb2a00640 1 -- 172.21.15.27:0/1709812051 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4eb4052910 con 0x7f4ebc0ce760 2024-01-15T18:23:09.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.969+0000 7f4ec1800640 1 -- 172.21.15.27:0/1709812051 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4ebc151b10 con 0x7f4e94067a90 2024-01-15T18:23:09.971 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:23:09.968+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:23:09.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:09.973+0000 7f4eb2a00640 1 -- 172.21.15.27:0/1709812051 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4ebc151b10 con 0x7f4e94067a90 2024-01-15T18:23:09.972 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:23:09.972 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:23:09.980 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:23:11.982 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:23:12.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.145+0000 7fc414c00640 1 Processor -- start 2024-01-15T18:23:12.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.145+0000 7fc414c00640 1 -- start start 2024-01-15T18:23:12.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.145+0000 7fc414c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fc41006b050 0x7fc41006a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:12.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.145+0000 7fc414c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc410071550 0x7fc41015d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:12.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.145+0000 7fc414c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc410161020 con 0x7fc41006bc50 2024-01-15T18:23:12.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.145+0000 7fc414c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc410161300 con 0x7fc41006b050 2024-01-15T18:23:12.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.145+0000 7fc414c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4101615e0 con 0x7fc410071550 2024-01-15T18:23:12.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.145+0000 7fc40d400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc41006bc50 0x7fc410070e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57702/0 (socket says 172.21.15.27:57702) 2024-01-15T18:23:12.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.145+0000 7fc40d400640 1 -- 172.21.15.27:0/976173910 learned_addr learned my addr 172.21.15.27:0/976173910 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:12.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.145+0000 7fc40de00640 1 --2- 172.21.15.27:0/976173910 >> v2:172.21.15.93:3300/0 conn(0x7fc41006b050 0x7fc41006a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:12.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.145+0000 7fc40e800640 1 --2- 172.21.15.27:0/976173910 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc410071550 0x7fc41015d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:12.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.145+0000 7fc40ca00640 1 -- 172.21.15.27:0/976173910 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2833547291 0 0) 0x7fc410161020 con 0x7fc41006bc50 2024-01-15T18:23:12.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.145+0000 7fc40ca00640 1 -- 172.21.15.27:0/976173910 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3f0003530 con 0x7fc41006bc50 2024-01-15T18:23:12.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.145+0000 7fc40de00640 1 -- 172.21.15.27:0/976173910 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc410071550 msgr2=0x7fc41015d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:12.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.145+0000 7fc40de00640 1 --2- 172.21.15.27:0/976173910 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc410071550 0x7fc41015d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:12.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.145+0000 7fc40de00640 1 -- 172.21.15.27:0/976173910 >> v1:172.21.15.27:6789/0 conn(0x7fc41006bc50 legacy=0x7fc410070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:12.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.145+0000 7fc40de00640 1 -- 172.21.15.27:0/976173910 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4101618c0 con 0x7fc41006b050 2024-01-15T18:23:12.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.149+0000 7fc40de00640 1 --2- 172.21.15.27:0/976173910 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc41006b050 0x7fc41006a4f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fc4000097c0 tx=0x7fc40001c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80ffc068a9656792 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:12.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.149+0000 7fc40ca00640 1 -- 172.21.15.27:0/976173910 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4000ac630 con 0x7fc41006b050 2024-01-15T18:23:12.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.149+0000 7fc40ca00640 1 -- 172.21.15.27:0/976173910 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4000ac790 con 0x7fc41006b050 2024-01-15T18:23:12.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.149+0000 7fc40ca00640 1 -- 172.21.15.27:0/976173910 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4000acb20 con 0x7fc41006b050 2024-01-15T18:23:12.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.149+0000 7fc414c00640 1 -- 172.21.15.27:0/976173910 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc41006b050 msgr2=0x7fc41006a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:12.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.149+0000 7fc414c00640 1 --2- 172.21.15.27:0/976173910 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc41006b050 0x7fc41006a4f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fc4000097c0 tx=0x7fc40001c3f0 comp rx=0 tx=0).stop 2024-01-15T18:23:12.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.149+0000 7fc414c00640 1 -- 172.21.15.27:0/976173910 shutdown_connections 2024-01-15T18:23:12.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.149+0000 7fc414c00640 1 --2- 172.21.15.27:0/976173910 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc410071550 0x7fc41015d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:12.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.149+0000 7fc414c00640 1 --2- 172.21.15.27:0/976173910 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc41006b050 0x7fc41006a4f0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:12.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.149+0000 7fc414c00640 1 -- 172.21.15.27:0/976173910 >> 172.21.15.27:0/976173910 conn(0x7fc4100b15b0 msgr2=0x7fc4100b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:12.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.149+0000 7fc414c00640 1 -- 172.21.15.27:0/976173910 shutdown_connections 2024-01-15T18:23:12.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.149+0000 7fc414c00640 1 -- 172.21.15.27:0/976173910 wait complete. 2024-01-15T18:23:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.153+0000 7fc414c00640 1 Processor -- start 2024-01-15T18:23:12.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.153+0000 7fc414c00640 1 -- start start 2024-01-15T18:23:12.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.153+0000 7fc414c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc41006b050 0x7fc410151c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:12.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.153+0000 7fc414c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc41006bc50 0x7fc4100b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:12.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.153+0000 7fc414c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc410071550 0x7fc4101549e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:12.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.153+0000 7fc414c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc410159660 con 0x7fc41006bc50 2024-01-15T18:23:12.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.153+0000 7fc414c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc410159840 con 0x7fc410071550 2024-01-15T18:23:12.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.153+0000 7fc414c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc410159b20 con 0x7fc41006b050 2024-01-15T18:23:12.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.153+0000 7fc40d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc41006bc50 0x7fc4100b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:12.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.153+0000 7fc40d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc41006bc50 0x7fc4100b04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52522/0 (socket says 172.21.15.27:52522) 2024-01-15T18:23:12.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.153+0000 7fc40d400640 1 -- 172.21.15.27:0/1008523551 learned_addr learned my addr 172.21.15.27:0/1008523551 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:12.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.153+0000 7fc40de00640 1 --2- 172.21.15.27:0/1008523551 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc41006b050 0x7fc410151c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:12.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.153+0000 7fc40e800640 1 --2- 172.21.15.27:0/1008523551 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc410071550 0x7fc4101549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:12.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.153+0000 7fc40d400640 1 -- 172.21.15.27:0/1008523551 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc41006b050 msgr2=0x7fc410151c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:12.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.153+0000 7fc40d400640 1 --2- 172.21.15.27:0/1008523551 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc41006b050 0x7fc410151c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:12.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.153+0000 7fc40d400640 1 -- 172.21.15.27:0/1008523551 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc410071550 msgr2=0x7fc4101549e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:12.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.153+0000 7fc40d400640 1 --2- 172.21.15.27:0/1008523551 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc410071550 0x7fc4101549e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:12.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.153+0000 7fc40d400640 1 -- 172.21.15.27:0/1008523551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc400009420 con 0x7fc41006bc50 2024-01-15T18:23:12.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.153+0000 7fc40d400640 1 --2- 172.21.15.27:0/1008523551 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc41006bc50 0x7fc4100b04d0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8001ec0 tx=0x7fc3f800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:12.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.153+0000 7fc406000640 1 -- 172.21.15.27:0/1008523551 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8005d50 con 0x7fc41006bc50 2024-01-15T18:23:12.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.153+0000 7fc406000640 1 -- 172.21.15.27:0/1008523551 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f8007070 con 0x7fc41006bc50 2024-01-15T18:23:12.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.153+0000 7fc406000640 1 -- 172.21.15.27:0/1008523551 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8007420 con 0x7fc41006bc50 2024-01-15T18:23:12.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.153+0000 7fc414c00640 1 -- 172.21.15.27:0/1008523551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc410159e00 con 0x7fc41006bc50 2024-01-15T18:23:12.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.153+0000 7fc414c00640 1 -- 172.21.15.27:0/1008523551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc410152160 con 0x7fc41006bc50 2024-01-15T18:23:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.157+0000 7fc414c00640 1 -- 172.21.15.27:0/1008523551 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc41004e850 con 0x7fc41006bc50 2024-01-15T18:23:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.157+0000 7fc406000640 1 -- 172.21.15.27:0/1008523551 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7fc3f8007580 con 0x7fc41006bc50 2024-01-15T18:23:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.157+0000 7fc406000640 1 --2- 172.21.15.27:0/1008523551 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc3e8067a60 0x7fc3e8069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.157+0000 7fc406000640 1 -- 172.21.15.27:0/1008523551 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fc3f8082480 con 0x7fc41006bc50 2024-01-15T18:23:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.157+0000 7fc40de00640 1 --2- 172.21.15.27:0/1008523551 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc3e8067a60 0x7fc3e8069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.157+0000 7fc40de00640 1 --2- 172.21.15.27:0/1008523551 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc3e8067a60 0x7fc3e8069f20 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fc4000ae000 tx=0x7fc400008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:12.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.161+0000 7fc406000640 1 -- 172.21.15.27:0/1008523551 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc3f8050f30 con 0x7fc41006bc50 2024-01-15T18:23:12.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.353+0000 7fc414c00640 1 -- 172.21.15.27:0/1008523551 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc41015a2b0 con 0x7fc3e8067a60 2024-01-15T18:23:12.358 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:23:12.356+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:23:12.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:12.361+0000 7fc406000640 1 -- 172.21.15.27:0/1008523551 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc41015a2b0 con 0x7fc3e8067a60 2024-01-15T18:23:12.359 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:23:12.359 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:23:12.367 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:23:14.369 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:23:14.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.529+0000 7f844d600640 1 Processor -- start 2024-01-15T18:23:14.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.529+0000 7f844d600640 1 -- start start 2024-01-15T18:23:14.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.533+0000 7f844d600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f844815ad20 0x7f844815b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:14.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.533+0000 7f844d600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f84480715f0 0x7f844815ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:14.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.533+0000 7f844d600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844815a610 con 0x7f844815d900 2024-01-15T18:23:14.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.533+0000 7f844d600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f844815e3c0 con 0x7f844815ad20 2024-01-15T18:23:14.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.533+0000 7f844d600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f844815e6a0 con 0x7f84480715f0 2024-01-15T18:23:14.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.533+0000 7f8446a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f844815ad20 0x7f844815b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:14.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.533+0000 7f8447400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f84480715f0 0x7f844815ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:14.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.533+0000 7f8446a00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f844815ad20 0x7f844815b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:56132/0 (socket says 172.21.15.27:56132) 2024-01-15T18:23:14.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.533+0000 7f8447400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f84480715f0 0x7f844815ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:50678/0 (socket says 172.21.15.27:50678) 2024-01-15T18:23:14.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.533+0000 7f8446a00640 1 -- 172.21.15.27:0/1175048284 learned_addr learned my addr 172.21.15.27:0/1175048284 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:14.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.533+0000 7f8446a00640 1 -- 172.21.15.27:0/1175048284 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f84480715f0 msgr2=0x7f844815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:14.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.533+0000 7f8446a00640 1 --2- 172.21.15.27:0/1175048284 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f84480715f0 0x7f844815ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:14.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.533+0000 7f8446a00640 1 -- 172.21.15.27:0/1175048284 >> v1:172.21.15.93:6789/0 conn(0x7f844815d900 legacy=0x7f8448070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:14.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.533+0000 7f8446a00640 1 -- 172.21.15.27:0/1175048284 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844815b6e0 con 0x7f844815ad20 2024-01-15T18:23:14.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.537+0000 7f8446a00640 1 --2- 172.21.15.27:0/1175048284 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f844815ad20 0x7f844815b100 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f84380098a0 tx=0x7f843801c4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97611b67731a493d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:14.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.537+0000 7f8445600640 1 -- 172.21.15.27:0/1175048284 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84380aa670 con 0x7f844815ad20 2024-01-15T18:23:14.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.537+0000 7f8445600640 1 -- 172.21.15.27:0/1175048284 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84380aa7d0 con 0x7f844815ad20 2024-01-15T18:23:14.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.537+0000 7f8445600640 1 -- 172.21.15.27:0/1175048284 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84380aab60 con 0x7f844815ad20 2024-01-15T18:23:14.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.537+0000 7f844d600640 1 -- 172.21.15.27:0/1175048284 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f844815ad20 msgr2=0x7f844815b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:14.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.537+0000 7f844d600640 1 --2- 172.21.15.27:0/1175048284 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f844815ad20 0x7f844815b100 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f84380098a0 tx=0x7f843801c4f0 comp rx=0 tx=0).stop 2024-01-15T18:23:14.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.537+0000 7f844d600640 1 -- 172.21.15.27:0/1175048284 shutdown_connections 2024-01-15T18:23:14.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.537+0000 7f844d600640 1 --2- 172.21.15.27:0/1175048284 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f84480715f0 0x7f844815ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:14.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.537+0000 7f844d600640 1 --2- 172.21.15.27:0/1175048284 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f844815ad20 0x7f844815b100 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:14.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.537+0000 7f844d600640 1 -- 172.21.15.27:0/1175048284 >> 172.21.15.27:0/1175048284 conn(0x7f84480cb700 msgr2=0x7f84480cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:14.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.537+0000 7f844d600640 1 -- 172.21.15.27:0/1175048284 shutdown_connections 2024-01-15T18:23:14.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.537+0000 7f844d600640 1 -- 172.21.15.27:0/1175048284 wait complete. 2024-01-15T18:23:14.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.541+0000 7f844d600640 1 Processor -- start 2024-01-15T18:23:14.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.541+0000 7f844d600640 1 -- start start 2024-01-15T18:23:14.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.541+0000 7f844d600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f84480715f0 0x7f8448157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:14.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.541+0000 7f844d600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f844815ad20 0x7f84480b4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:14.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.541+0000 7f844d600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f844815d900 0x7f84481525a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:14.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.541+0000 7f844d600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84480b4980 con 0x7f844815ad20 2024-01-15T18:23:14.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.541+0000 7f844d600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8448154dd0 con 0x7f84480715f0 2024-01-15T18:23:14.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.541+0000 7f844d600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84480b46b0 con 0x7f844815d900 2024-01-15T18:23:14.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.541+0000 7f8446000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f844815ad20 0x7f84480b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:14.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.541+0000 7f8446000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f844815ad20 0x7f84480b4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35064/0 (socket says 172.21.15.27:35064) 2024-01-15T18:23:14.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.541+0000 7f8446000640 1 -- 172.21.15.27:0/1439861353 learned_addr learned my addr 172.21.15.27:0/1439861353 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:14.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.541+0000 7f8446a00640 1 --2- 172.21.15.27:0/1439861353 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f84480715f0 0x7f8448157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:14.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.541+0000 7f8447400640 1 --2- 172.21.15.27:0/1439861353 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f844815d900 0x7f84481525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:14.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.541+0000 7f8446000640 1 -- 172.21.15.27:0/1439861353 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f844815d900 msgr2=0x7f84481525a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:14.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.541+0000 7f8446000640 1 --2- 172.21.15.27:0/1439861353 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f844815d900 0x7f84481525a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:14.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.541+0000 7f8446000640 1 -- 172.21.15.27:0/1439861353 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f84480715f0 msgr2=0x7f8448157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:14.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.541+0000 7f8446000640 1 --2- 172.21.15.27:0/1439861353 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f84480715f0 0x7f8448157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:14.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.541+0000 7f8446000640 1 -- 172.21.15.27:0/1439861353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84380094d0 con 0x7f844815ad20 2024-01-15T18:23:14.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.541+0000 7f8446000640 1 --2- 172.21.15.27:0/1439861353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f844815ad20 0x7f84480b4170 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f843000c3c0 tx=0x7f843000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:14.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.541+0000 7f843ea00640 1 -- 172.21.15.27:0/1439861353 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84300052d0 con 0x7f844815ad20 2024-01-15T18:23:14.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.541+0000 7f843ea00640 1 -- 172.21.15.27:0/1439861353 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8430005430 con 0x7f844815ad20 2024-01-15T18:23:14.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.541+0000 7f843ea00640 1 -- 172.21.15.27:0/1439861353 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8430003840 con 0x7f844815ad20 2024-01-15T18:23:14.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.541+0000 7f844d600640 1 -- 172.21.15.27:0/1439861353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8448154430 con 0x7f844815ad20 2024-01-15T18:23:14.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.541+0000 7f844d600640 1 -- 172.21.15.27:0/1439861353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84481549f0 con 0x7f844815ad20 2024-01-15T18:23:14.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.541+0000 7f844d600640 1 -- 172.21.15.27:0/1439861353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8448154f10 con 0x7f844815ad20 2024-01-15T18:23:14.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.545+0000 7f843ea00640 1 -- 172.21.15.27:0/1439861353 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f84300039a0 con 0x7f844815ad20 2024-01-15T18:23:14.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.545+0000 7f843ea00640 1 --2- 172.21.15.27:0/1439861353 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8420067a60 0x7f8420069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:14.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.545+0000 7f843ea00640 1 -- 172.21.15.27:0/1439861353 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f84300827d0 con 0x7f844815ad20 2024-01-15T18:23:14.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.545+0000 7f8446a00640 1 --2- 172.21.15.27:0/1439861353 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8420067a60 0x7f8420069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:14.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.545+0000 7f8446a00640 1 --2- 172.21.15.27:0/1439861353 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8420067a60 0x7f8420069f20 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f84380ae000 tx=0x7f84380b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:14.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.549+0000 7f843ea00640 1 -- 172.21.15.27:0/1439861353 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8448154f10 con 0x7f844815ad20 2024-01-15T18:23:14.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.737+0000 7f844d600640 1 -- 172.21.15.27:0/1439861353 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8448154f10 con 0x7f8420067a60 2024-01-15T18:23:14.740 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:23:14.740+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:23:14.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:14.745+0000 7f843ea00640 1 -- 172.21.15.27:0/1439861353 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8448154f10 con 0x7f8420067a60 2024-01-15T18:23:14.740 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:23:14.741 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:23:14.749 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:23:16.751 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:23:16.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.913+0000 7f5b73400640 1 Processor -- start 2024-01-15T18:23:16.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.913+0000 7f5b73400640 1 -- start start 2024-01-15T18:23:16.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b73400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f5b6c0cef20 0x7f5b6c161f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:16.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b73400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b6c06c5d0 con 0x7f5b6c0cdb20 2024-01-15T18:23:16.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b73400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b6c06d4a0 con 0x7f5b6c161a90 2024-01-15T18:23:16.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b73400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b6c162b00 con 0x7f5b6c0cef20 2024-01-15T18:23:16.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b70a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5b6c0cdb20 0x7f5b6c06cd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50196/0 (socket says 172.21.15.27:50196) 2024-01-15T18:23:16.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b70a00640 1 -- 172.21.15.27:0/363512640 learned_addr learned my addr 172.21.15.27:0/363512640 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:16.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b71400640 1 --2- 172.21.15.27:0/363512640 >> v2:172.21.15.93:3300/0 conn(0x7f5b6c0cef20 0x7f5b6c161f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:16.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b6b400640 1 -- 172.21.15.27:0/363512640 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1631943160 0 0) 0x7f5b6c06c5d0 con 0x7f5b6c0cdb20 2024-01-15T18:23:16.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b6b400640 1 -- 172.21.15.27:0/363512640 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b4c003530 con 0x7f5b6c0cdb20 2024-01-15T18:23:16.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b6b400640 1 -- 172.21.15.27:0/363512640 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3476458216 0 0) 0x7f5b4c003530 con 0x7f5b6c0cdb20 2024-01-15T18:23:16.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b6b400640 1 -- 172.21.15.27:0/363512640 >> v2:172.21.15.93:3300/0 conn(0x7f5b6c0cef20 msgr2=0x7f5b6c161f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:16.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b6b400640 1 --2- 172.21.15.27:0/363512640 >> v2:172.21.15.93:3300/0 conn(0x7f5b6c0cef20 0x7f5b6c161f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:16.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b6b400640 1 -- 172.21.15.27:0/363512640 >> v1:172.21.15.93:6789/0 conn(0x7f5b6c161a90 legacy=0x7f5b6c0ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:16.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b6b400640 1 -- 172.21.15.27:0/363512640 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6c162540 con 0x7f5b6c0cdb20 2024-01-15T18:23:16.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b6b400640 1 -- 172.21.15.27:0/363512640 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f5b5c001680 con 0x7f5b6c0cdb20 2024-01-15T18:23:16.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b6b400640 1 -- 172.21.15.27:0/363512640 >> v1:172.21.15.27:6789/0 conn(0x7f5b6c0cdb20 legacy=0x7f5b6c06cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:16.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b6b400640 1 --2- 172.21.15.27:0/363512640 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b4c0039f0 0x7f5b4c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:16.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b6b400640 1 --2- 172.21.15.27:0/363512640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b4c0047d0 0x7f5b4c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:16.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b6b400640 1 --2- 172.21.15.27:0/363512640 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b4c005190 0x7f5b4c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:16.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b6b400640 1 -- 172.21.15.27:0/363512640 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c162b00 con 0x7f5b4c0047d0 2024-01-15T18:23:16.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b6b400640 1 -- 172.21.15.27:0/363512640 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4c00c0c0 con 0x7f5b4c0039f0 2024-01-15T18:23:16.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b6b400640 1 -- 172.21.15.27:0/363512640 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4c00c280 con 0x7f5b4c005190 2024-01-15T18:23:16.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b6be00640 1 --2- 172.21.15.27:0/363512640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b4c0047d0 0x7f5b4c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:16.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b70a00640 1 --2- 172.21.15.27:0/363512640 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b4c0039f0 0x7f5b4c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:16.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b71400640 1 --2- 172.21.15.27:0/363512640 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b4c005190 0x7f5b4c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:16.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b6be00640 1 -- 172.21.15.27:0/363512640 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b4c005190 msgr2=0x7f5b4c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:16.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b6be00640 1 --2- 172.21.15.27:0/363512640 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b4c005190 0x7f5b4c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:16.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b6be00640 1 -- 172.21.15.27:0/363512640 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b4c0039f0 msgr2=0x7f5b4c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:16.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b6be00640 1 --2- 172.21.15.27:0/363512640 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b4c0039f0 0x7f5b4c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:16.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b6be00640 1 -- 172.21.15.27:0/363512640 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b4c003530 con 0x7f5b4c0047d0 2024-01-15T18:23:16.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b71400640 1 --2- 172.21.15.27:0/363512640 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b4c005190 0x7f5b4c00b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:23:16.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.917+0000 7f5b71400640 1 -- 172.21.15.27:0/363512640 reap_dead start 2024-01-15T18:23:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.921+0000 7f5b6be00640 1 --2- 172.21.15.27:0/363512640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b4c0047d0 0x7f5b4c004c50 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f5b54001780 tx=0x7f5b5401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54f22ca0fad8a745 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:16.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.921+0000 7f5b6b400640 1 -- 172.21.15.27:0/363512640 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b540052d0 con 0x7f5b4c0047d0 2024-01-15T18:23:16.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.921+0000 7f5b6b400640 1 -- 172.21.15.27:0/363512640 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b540a7050 con 0x7f5b4c0047d0 2024-01-15T18:23:16.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.921+0000 7f5b6b400640 1 -- 172.21.15.27:0/363512640 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b54003600 con 0x7f5b4c0047d0 2024-01-15T18:23:16.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.921+0000 7f5b73400640 1 -- 172.21.15.27:0/363512640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b4c0047d0 msgr2=0x7f5b4c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:16.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.921+0000 7f5b73400640 1 --2- 172.21.15.27:0/363512640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b4c0047d0 0x7f5b4c004c50 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f5b54001780 tx=0x7f5b5401c3e0 comp rx=0 tx=0).stop 2024-01-15T18:23:16.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.921+0000 7f5b73400640 1 -- 172.21.15.27:0/363512640 shutdown_connections 2024-01-15T18:23:16.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.921+0000 7f5b73400640 1 --2- 172.21.15.27:0/363512640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b4c0047d0 0x7f5b4c004c50 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:16.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.921+0000 7f5b73400640 1 -- 172.21.15.27:0/363512640 >> 172.21.15.27:0/363512640 conn(0x7f5b6c0cb700 msgr2=0x7f5b6c0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:16.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.921+0000 7f5b73400640 1 -- 172.21.15.27:0/363512640 shutdown_connections 2024-01-15T18:23:16.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.921+0000 7f5b73400640 1 -- 172.21.15.27:0/363512640 wait complete. 2024-01-15T18:23:16.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b73400640 1 Processor -- start 2024-01-15T18:23:16.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b73400640 1 -- start start 2024-01-15T18:23:16.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b73400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b4c0047d0 0x7f5b6c1644b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:16.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b73400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b6c1f6d50 0x7f5b6c155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:16.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b73400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b6c0afa60 0x7f5b6c0afee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:16.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b73400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c1649f0 con 0x7f5b6c1f6d50 2024-01-15T18:23:16.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b73400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c1f7d20 con 0x7f5b6c0afa60 2024-01-15T18:23:16.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b73400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c0b0420 con 0x7f5b4c0047d0 2024-01-15T18:23:16.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b6be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b6c1f6d50 0x7f5b6c155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:16.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b6be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b6c1f6d50 0x7f5b6c155e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35088/0 (socket says 172.21.15.27:35088) 2024-01-15T18:23:16.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b6be00640 1 -- 172.21.15.27:0/338247104 learned_addr learned my addr 172.21.15.27:0/338247104 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:16.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b70a00640 1 --2- 172.21.15.27:0/338247104 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b4c0047d0 0x7f5b6c1644b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:16.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b71400640 1 --2- 172.21.15.27:0/338247104 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b6c0afa60 0x7f5b6c0afee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:16.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b6be00640 1 -- 172.21.15.27:0/338247104 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b4c0047d0 msgr2=0x7f5b6c1644b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:16.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b6be00640 1 --2- 172.21.15.27:0/338247104 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f5b4c0047d0 0x7f5b6c1644b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:16.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b6be00640 1 -- 172.21.15.27:0/338247104 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b6c0afa60 msgr2=0x7f5b6c0afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:16.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b6be00640 1 --2- 172.21.15.27:0/338247104 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b6c0afa60 0x7f5b6c0afee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:16.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b6be00640 1 -- 172.21.15.27:0/338247104 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6c15f010 con 0x7f5b6c1f6d50 2024-01-15T18:23:16.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b71400640 1 --2- 172.21.15.27:0/338247104 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f5b6c0afa60 0x7f5b6c0afee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:23:16.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b6be00640 1 --2- 172.21.15.27:0/338247104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5b6c1f6d50 0x7f5b6c155e50 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f5b5401c210 tx=0x7f5b4c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:16.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b68c00640 1 -- 172.21.15.27:0/338247104 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b54007500 con 0x7f5b6c1f6d50 2024-01-15T18:23:16.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b68c00640 1 -- 172.21.15.27:0/338247104 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b54007660 con 0x7f5b6c1f6d50 2024-01-15T18:23:16.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b68c00640 1 -- 172.21.15.27:0/338247104 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b54006a00 con 0x7f5b6c1f6d50 2024-01-15T18:23:16.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b73400640 1 -- 172.21.15.27:0/338247104 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b6c1f71b0 con 0x7f5b6c1f6d50 2024-01-15T18:23:16.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b73400640 1 -- 172.21.15.27:0/338247104 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b6c1f7a40 con 0x7f5b6c1f6d50 2024-01-15T18:23:16.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.925+0000 7f5b73400640 1 -- 172.21.15.27:0/338247104 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b6c04e850 con 0x7f5b6c1f6d50 2024-01-15T18:23:16.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.929+0000 7f5b68c00640 1 -- 172.21.15.27:0/338247104 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f5b54006b60 con 0x7f5b6c1f6d50 2024-01-15T18:23:16.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.929+0000 7f5b68c00640 1 --2- 172.21.15.27:0/338247104 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5b44067a60 0x7f5b44069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:16.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.929+0000 7f5b68c00640 1 -- 172.21.15.27:0/338247104 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f5b5411d5e0 con 0x7f5b6c1f6d50 2024-01-15T18:23:16.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.929+0000 7f5b70a00640 1 --2- 172.21.15.27:0/338247104 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5b44067a60 0x7f5b44069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:16.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.929+0000 7f5b70a00640 1 --2- 172.21.15.27:0/338247104 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f5b44067a60 0x7f5b44069f20 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c0018f0 tx=0x7f5b5c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:16.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:16.933+0000 7f5b68c00640 1 -- 172.21.15.27:0/338247104 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5b540ebea0 con 0x7f5b6c1f6d50 2024-01-15T18:23:17.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:17.121+0000 7f5b73400640 1 -- 172.21.15.27:0/338247104 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5b6c06c870 con 0x7f5b44067a60 2024-01-15T18:23:17.122 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:23:17.120+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:23:17.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:17.125+0000 7f5b68c00640 1 -- 172.21.15.27:0/338247104 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5b5c0056d0 con 0x7f5b44067a60 2024-01-15T18:23:17.123 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:23:17.124 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:23:17.132 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:23:19.134 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:23:19.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.297+0000 7efe22a00640 1 Processor -- start 2024-01-15T18:23:19.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.297+0000 7efe22a00640 1 -- start start 2024-01-15T18:23:19.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.297+0000 7efe22a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe1c146a60 0x7efe1c0ac880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:19.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.297+0000 7efe22a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efe1c160cb0 0x7efe1c1610f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:19.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.297+0000 7efe22a00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1c0cde30 con 0x7efe1c06a4f0 2024-01-15T18:23:19.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.297+0000 7efe22a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe1c161630 con 0x7efe1c160cb0 2024-01-15T18:23:19.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.297+0000 7efe22a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe1c06fa50 con 0x7efe1c146a60 2024-01-15T18:23:19.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.297+0000 7efe1b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efe1c160cb0 0x7efe1c1610f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:19.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.297+0000 7efe1b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7efe1c160cb0 0x7efe1c1610f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35094/0 (socket says 172.21.15.27:35094) 2024-01-15T18:23:19.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.297+0000 7efe1b400640 1 -- 172.21.15.27:0/3358526169 learned_addr learned my addr 172.21.15.27:0/3358526169 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:19.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.297+0000 7efe1be00640 1 --2- 172.21.15.27:0/3358526169 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe1c146a60 0x7efe1c0ac880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:19.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.301+0000 7efe1b400640 1 -- 172.21.15.27:0/3358526169 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe1c146a60 msgr2=0x7efe1c0ac880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:19.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.301+0000 7efe1b400640 1 --2- 172.21.15.27:0/3358526169 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe1c146a60 0x7efe1c0ac880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:19.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.301+0000 7efe1b400640 1 -- 172.21.15.27:0/3358526169 >> v1:172.21.15.27:6789/0 conn(0x7efe1c06a4f0 legacy=0x7efe1c0ce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:19.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.301+0000 7efe1b400640 1 -- 172.21.15.27:0/3358526169 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1c161820 con 0x7efe1c160cb0 2024-01-15T18:23:19.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.301+0000 7efe1b400640 1 --2- 172.21.15.27:0/3358526169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe1c160cb0 0x7efe1c1610f0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7efe04009870 tx=0x7efe0401c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4105d886c9c747 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:19.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.301+0000 7efe1aa00640 1 -- 172.21.15.27:0/3358526169 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe040aa670 con 0x7efe1c160cb0 2024-01-15T18:23:19.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.301+0000 7efe1aa00640 1 -- 172.21.15.27:0/3358526169 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe040aa7d0 con 0x7efe1c160cb0 2024-01-15T18:23:19.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.301+0000 7efe1aa00640 1 -- 172.21.15.27:0/3358526169 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe040aab60 con 0x7efe1c160cb0 2024-01-15T18:23:19.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.305+0000 7efe22a00640 1 -- 172.21.15.27:0/3358526169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe1c160cb0 msgr2=0x7efe1c1610f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:19.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.305+0000 7efe22a00640 1 --2- 172.21.15.27:0/3358526169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe1c160cb0 0x7efe1c1610f0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7efe04009870 tx=0x7efe0401c510 comp rx=0 tx=0).stop 2024-01-15T18:23:19.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.305+0000 7efe22a00640 1 -- 172.21.15.27:0/3358526169 shutdown_connections 2024-01-15T18:23:19.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.305+0000 7efe22a00640 1 --2- 172.21.15.27:0/3358526169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe1c160cb0 0x7efe1c1610f0 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:19.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.305+0000 7efe22a00640 1 --2- 172.21.15.27:0/3358526169 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe1c146a60 0x7efe1c0ac880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:19.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.305+0000 7efe22a00640 1 -- 172.21.15.27:0/3358526169 >> 172.21.15.27:0/3358526169 conn(0x7efe1c0cce10 msgr2=0x7efe1c0b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:19.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.305+0000 7efe22a00640 1 -- 172.21.15.27:0/3358526169 shutdown_connections 2024-01-15T18:23:19.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.305+0000 7efe22a00640 1 -- 172.21.15.27:0/3358526169 wait complete. 2024-01-15T18:23:19.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.305+0000 7efe22a00640 1 Processor -- start 2024-01-15T18:23:19.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.305+0000 7efe22a00640 1 -- start start 2024-01-15T18:23:19.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.305+0000 7efe22a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe1c06a4f0 0x7efe1c140470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:19.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.305+0000 7efe22a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efe1c146a60 0x7efe1c14da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:19.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.305+0000 7efe22a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe1c160cb0 0x7efe1c14aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:19.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.305+0000 7efe22a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1c0ada10 con 0x7efe1c160cb0 2024-01-15T18:23:19.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.305+0000 7efe22a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1c14df60 con 0x7efe1c146a60 2024-01-15T18:23:19.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.305+0000 7efe22a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe1c140b00 con 0x7efe1c06a4f0 2024-01-15T18:23:19.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.305+0000 7efe20a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe1c160cb0 0x7efe1c14aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:19.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.305+0000 7efe1be00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe1c06a4f0 0x7efe1c140470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:19.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.305+0000 7efe20a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe1c160cb0 0x7efe1c14aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35104/0 (socket says 172.21.15.27:35104) 2024-01-15T18:23:19.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.305+0000 7efe20a00640 1 -- 172.21.15.27:0/2014362138 learned_addr learned my addr 172.21.15.27:0/2014362138 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:19.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.305+0000 7efe1b400640 1 --2- 172.21.15.27:0/2014362138 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efe1c146a60 0x7efe1c14da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:19.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.309+0000 7efe20a00640 1 -- 172.21.15.27:0/2014362138 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe1c06a4f0 msgr2=0x7efe1c140470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:19.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.309+0000 7efe20a00640 1 --2- 172.21.15.27:0/2014362138 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7efe1c06a4f0 0x7efe1c140470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:19.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.309+0000 7efe20a00640 1 -- 172.21.15.27:0/2014362138 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efe1c146a60 msgr2=0x7efe1c14da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:19.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.309+0000 7efe20a00640 1 --2- 172.21.15.27:0/2014362138 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7efe1c146a60 0x7efe1c14da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:19.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.309+0000 7efe20a00640 1 -- 172.21.15.27:0/2014362138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe040094d0 con 0x7efe1c160cb0 2024-01-15T18:23:19.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.309+0000 7efe20a00640 1 --2- 172.21.15.27:0/2014362138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe1c160cb0 0x7efe1c14aa20 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7efe1400c7a0 tx=0x7efe1400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:19.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.309+0000 7efe13e00640 1 -- 172.21.15.27:0/2014362138 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe14003b80 con 0x7efe1c160cb0 2024-01-15T18:23:19.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.309+0000 7efe13e00640 1 -- 172.21.15.27:0/2014362138 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe14003ce0 con 0x7efe1c160cb0 2024-01-15T18:23:19.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.309+0000 7efe22a00640 1 -- 172.21.15.27:0/2014362138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe1c0ad070 con 0x7efe1c160cb0 2024-01-15T18:23:19.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.309+0000 7efe13e00640 1 -- 172.21.15.27:0/2014362138 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1400aa30 con 0x7efe1c160cb0 2024-01-15T18:23:19.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.309+0000 7efe22a00640 1 -- 172.21.15.27:0/2014362138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe1c0ad630 con 0x7efe1c160cb0 2024-01-15T18:23:19.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.309+0000 7efe22a00640 1 -- 172.21.15.27:0/2014362138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe1c0adb50 con 0x7efe1c160cb0 2024-01-15T18:23:19.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.309+0000 7efe13e00640 1 -- 172.21.15.27:0/2014362138 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7efe1400ab90 con 0x7efe1c160cb0 2024-01-15T18:23:19.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.313+0000 7efe13e00640 1 --2- 172.21.15.27:0/2014362138 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efdf4067a60 0x7efdf4069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:19.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.313+0000 7efe13e00640 1 -- 172.21.15.27:0/2014362138 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7efe14082fe0 con 0x7efe1c160cb0 2024-01-15T18:23:19.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.313+0000 7efe1be00640 1 --2- 172.21.15.27:0/2014362138 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efdf4067a60 0x7efdf4069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:19.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.313+0000 7efe1be00640 1 --2- 172.21.15.27:0/2014362138 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7efdf4067a60 0x7efdf4069f20 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7efe0c009b60 tx=0x7efe0c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:19.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.317+0000 7efe13e00640 1 -- 172.21.15.27:0/2014362138 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7efe140519d0 con 0x7efe1c160cb0 2024-01-15T18:23:19.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.505+0000 7efe22a00640 1 -- 172.21.15.27:0/2014362138 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efe1c0acdc0 con 0x7efdf4067a60 2024-01-15T18:23:19.508 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:23:19.508+0000 7f59f2400640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:23:19.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:19.513+0000 7efe13e00640 1 -- 172.21.15.27:0/2014362138 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7efe0c011ac0 con 0x7efdf4067a60 2024-01-15T18:23:19.509 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:23:19.509 INFO:teuthology.orchestra.run.smithi027.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:23:19.516 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:23:19.522 INFO:tasks.cephfs_test_runner:Test that the progress bars for ongoing clones is removed from the ... ERROR 2024-01-15T18:23:19.522 INFO:tasks.cephfs_test_runner:ERROR 2024-01-15T18:23:19.523 INFO:tasks.cephfs_test_runner: 2024-01-15T18:23:19.523 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-15T18:23:19.523 INFO:tasks.cephfs_test_runner:ERROR: test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T18:23:19.523 INFO:tasks.cephfs_test_runner:Test that the progress bars for ongoing clones is removed from the 2024-01-15T18:23:19.523 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-15T18:23:19.523 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-15T18:23:19.524 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/cephfs/test_volumes.py", line 8163, in test_when_3_clones_are_cancelled 2024-01-15T18:23:19.524 INFO:tasks.cephfs_test_runner: self._cancel_clones_and_assert(c) 2024-01-15T18:23:19.524 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/cephfs/test_volumes.py", line 7984, in _cancel_clones_and_assert 2024-01-15T18:23:19.524 INFO:tasks.cephfs_test_runner: self._cancel_clones(clones, check_status=True) 2024-01-15T18:23:19.524 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/cephfs/test_volumes.py", line 7965, in _cancel_clones 2024-01-15T18:23:19.524 INFO:tasks.cephfs_test_runner: self.run_ceph_cmd(f'fs clone cancel {v} {i}', 2024-01-15T18:23:19.524 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-01-15T18:23:19.524 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-01-15T18:23:19.524 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd 2024-01-15T18:23:19.524 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-01-15T18:23:19.524 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run 2024-01-15T18:23:19.524 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-01-15T18:23:19.525 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run 2024-01-15T18:23:19.525 INFO:tasks.cephfs_test_runner: r.wait() 2024-01-15T18:23:19.525 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait 2024-01-15T18:23:19.525 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-01-15T18:23:19.525 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-01-15T18:23:19.525 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-01-15T18:23:19.525 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone55242' 2024-01-15T18:23:19.525 INFO:tasks.cephfs_test_runner: 2024-01-15T18:23:19.525 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-15T18:23:19.525 INFO:tasks.cephfs_test_runner:ERROR: test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T18:23:19.525 INFO:tasks.cephfs_test_runner:Test that the progress bars for ongoing clones is removed from the 2024-01-15T18:23:19.525 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-15T18:23:19.526 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-15T18:23:19.526 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/cephfs/test_volumes.py", line 7597, in tearDown 2024-01-15T18:23:19.526 INFO:tasks.cephfs_test_runner: self._loop_cmd('fs subvolume snapshot rm --force --format ' 2024-01-15T18:23:19.526 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/cephfs/test_volumes.py", line 7566, in _loop_cmd 2024-01-15T18:23:19.526 INFO:tasks.cephfs_test_runner: raise RuntimeError(f'This cmd failed 50 times: {cmdargs}') 2024-01-15T18:23:19.526 INFO:tasks.cephfs_test_runner:RuntimeError: This cmd failed 50 times: fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:23:19.526 INFO:tasks.cephfs_test_runner: 2024-01-15T18:23:19.526 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-15T18:23:19.526 INFO:tasks.cephfs_test_runner:Ran 7 tests in 1070.338s 2024-01-15T18:23:19.526 INFO:tasks.cephfs_test_runner: 2024-01-15T18:23:19.526 INFO:tasks.cephfs_test_runner:FAILED (errors=2) 2024-01-15T18:23:19.527 INFO:tasks.cephfs_test_runner: 2024-01-15T18:23:19.527 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-15T18:23:19.527 INFO:tasks.cephfs_test_runner:ERROR: test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T18:23:19.527 INFO:tasks.cephfs_test_runner:Test that the progress bars for ongoing clones is removed from the 2024-01-15T18:23:19.527 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-15T18:23:19.527 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-15T18:23:19.527 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/cephfs/test_volumes.py", line 8163, in test_when_3_clones_are_cancelled 2024-01-15T18:23:19.527 INFO:tasks.cephfs_test_runner: self._cancel_clones_and_assert(c) 2024-01-15T18:23:19.527 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/cephfs/test_volumes.py", line 7984, in _cancel_clones_and_assert 2024-01-15T18:23:19.527 INFO:tasks.cephfs_test_runner: self._cancel_clones(clones, check_status=True) 2024-01-15T18:23:19.527 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/cephfs/test_volumes.py", line 7965, in _cancel_clones 2024-01-15T18:23:19.528 INFO:tasks.cephfs_test_runner: self.run_ceph_cmd(f'fs clone cancel {v} {i}', 2024-01-15T18:23:19.528 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-01-15T18:23:19.528 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-01-15T18:23:19.528 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd 2024-01-15T18:23:19.528 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-01-15T18:23:19.528 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run 2024-01-15T18:23:19.528 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-01-15T18:23:19.528 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run 2024-01-15T18:23:19.528 INFO:tasks.cephfs_test_runner: r.wait() 2024-01-15T18:23:19.528 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait 2024-01-15T18:23:19.528 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-01-15T18:23:19.529 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-01-15T18:23:19.529 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-01-15T18:23:19.529 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone55242' 2024-01-15T18:23:19.529 INFO:tasks.cephfs_test_runner: 2024-01-15T18:23:19.529 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-15T18:23:19.529 INFO:tasks.cephfs_test_runner:ERROR: test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T18:23:19.529 INFO:tasks.cephfs_test_runner:Test that the progress bars for ongoing clones is removed from the 2024-01-15T18:23:19.529 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-15T18:23:19.529 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-15T18:23:19.529 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/cephfs/test_volumes.py", line 7597, in tearDown 2024-01-15T18:23:19.529 INFO:tasks.cephfs_test_runner: self._loop_cmd('fs subvolume snapshot rm --force --format ' 2024-01-15T18:23:19.529 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/cephfs/test_volumes.py", line 7566, in _loop_cmd 2024-01-15T18:23:19.530 INFO:tasks.cephfs_test_runner: raise RuntimeError(f'This cmd failed 50 times: {cmdargs}') 2024-01-15T18:23:19.530 INFO:tasks.cephfs_test_runner:RuntimeError: This cmd failed 50 times: fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:23:19.530 INFO:tasks.cephfs_test_runner: 2024-01-15T18:23:19.530 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress), test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T18:23:19.734 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=593ff31fc8a6476abf4d2d6a4e016a4d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress), test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T18:23:19.735 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-01-15T18:23:19.745 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-01-15T18:23:19.752 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-01-15T18:23:19.753 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:23:19.753 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:23:19.775 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:23:19.775 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:23:19.824 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:23:20.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.013+0000 7f4493600640 1 Processor -- start 2024-01-15T18:23:20.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.013+0000 7f4493600640 1 -- start start 2024-01-15T18:23:20.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.013+0000 7f4493600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f448c0cf9b0 0x7f448c0b10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:20.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.013+0000 7f4493600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f448c0cedc0 0x7f448c0b1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:20.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.013+0000 7f4493600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f448c0b38b0 con 0x7f448c15e7d0 2024-01-15T18:23:20.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.013+0000 7f4493600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f448c0cd550 con 0x7f448c0cedc0 2024-01-15T18:23:20.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.013+0000 7f4493600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f448c0cd270 con 0x7f448c0cf9b0 2024-01-15T18:23:20.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.013+0000 7f4490c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f448c15e7d0 0x7f448c0cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50220/0 (socket says 172.21.15.27:50220) 2024-01-15T18:23:20.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.013+0000 7f4490c00640 1 -- 172.21.15.27:0/247453480 learned_addr learned my addr 172.21.15.27:0/247453480 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:20.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.013+0000 7f4491600640 1 --2- 172.21.15.27:0/247453480 >> v2:172.21.15.27:3300/0 conn(0x7f448c0cedc0 0x7f448c0b1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:20.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.013+0000 7f448be00640 1 --2- 172.21.15.27:0/247453480 >> v2:172.21.15.93:3300/0 conn(0x7f448c0cf9b0 0x7f448c0b10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:20.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.013+0000 7f4491600640 1 -- 172.21.15.27:0/247453480 >> v2:172.21.15.93:3300/0 conn(0x7f448c0cf9b0 msgr2=0x7f448c0b10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:20.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.013+0000 7f4491600640 1 --2- 172.21.15.27:0/247453480 >> v2:172.21.15.93:3300/0 conn(0x7f448c0cf9b0 0x7f448c0b10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.013+0000 7f4491600640 1 -- 172.21.15.27:0/247453480 >> v1:172.21.15.27:6789/0 conn(0x7f448c15e7d0 legacy=0x7f448c0cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:20.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.013+0000 7f448b400640 1 -- 172.21.15.27:0/247453480 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4223083480 0 0) 0x7f448c0b38b0 con 0x7f448c15e7d0 2024-01-15T18:23:20.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.013+0000 7f4491600640 1 -- 172.21.15.27:0/247453480 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f448c0b35d0 con 0x7f448c0cedc0 2024-01-15T18:23:20.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.013+0000 7f448be00640 1 --2- 172.21.15.27:0/247453480 >> v2:172.21.15.93:3300/0 conn(0x7f448c0cf9b0 0x7f448c0b10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:23:20.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.017+0000 7f4491600640 1 --2- 172.21.15.27:0/247453480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f448c0cedc0 0x7f448c0b1600 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f448c0718a0 tx=0x7f448401c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54acd924ed161126 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:20.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.017+0000 7f448b400640 1 -- 172.21.15.27:0/247453480 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44840a5830 con 0x7f448c0cedc0 2024-01-15T18:23:20.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.017+0000 7f448b400640 1 -- 172.21.15.27:0/247453480 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44840a5990 con 0x7f448c0cedc0 2024-01-15T18:23:20.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.017+0000 7f448b400640 1 -- 172.21.15.27:0/247453480 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44840a5d20 con 0x7f448c0cedc0 2024-01-15T18:23:20.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.017+0000 7f4493600640 1 -- 172.21.15.27:0/247453480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f448c0cedc0 msgr2=0x7f448c0b1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:20.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.017+0000 7f4493600640 1 --2- 172.21.15.27:0/247453480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f448c0cedc0 0x7f448c0b1600 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f448c0718a0 tx=0x7f448401c880 comp rx=0 tx=0).stop 2024-01-15T18:23:20.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.017+0000 7f4493600640 1 -- 172.21.15.27:0/247453480 shutdown_connections 2024-01-15T18:23:20.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.017+0000 7f4493600640 1 --2- 172.21.15.27:0/247453480 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f448c0cedc0 0x7f448c0b1600 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.017+0000 7f4493600640 1 --2- 172.21.15.27:0/247453480 >> v2:172.21.15.93:3300/0 conn(0x7f448c0cf9b0 0x7f448c0b10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.017+0000 7f4493600640 1 -- 172.21.15.27:0/247453480 >> 172.21.15.27:0/247453480 conn(0x7f448c0b2a10 msgr2=0x7f448c0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:20.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.017+0000 7f4493600640 1 -- 172.21.15.27:0/247453480 shutdown_connections 2024-01-15T18:23:20.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.017+0000 7f4493600640 1 -- 172.21.15.27:0/247453480 wait complete. 2024-01-15T18:23:20.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.021+0000 7f4493600640 1 Processor -- start 2024-01-15T18:23:20.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.021+0000 7f4493600640 1 -- start start 2024-01-15T18:23:20.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.021+0000 7f4493600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f448c0cedc0 0x7f448c1553e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:20.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.021+0000 7f4493600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f448c0cf9b0 0x7f448c0aed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:20.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.021+0000 7f4493600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f448c15e7d0 0x7f448c0af280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:20.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.021+0000 7f4493600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f448c158870 con 0x7f448c0cf9b0 2024-01-15T18:23:20.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.021+0000 7f4493600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f448c0c4ec0 con 0x7f448c15e7d0 2024-01-15T18:23:20.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.021+0000 7f4493600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f448c1582b0 con 0x7f448c0cedc0 2024-01-15T18:23:20.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.021+0000 7f448be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f448c0cf9b0 0x7f448c0aed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:20.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.021+0000 7f448be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f448c0cf9b0 0x7f448c0aed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35122/0 (socket says 172.21.15.27:35122) 2024-01-15T18:23:20.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.021+0000 7f448be00640 1 -- 172.21.15.27:0/3711016867 learned_addr learned my addr 172.21.15.27:0/3711016867 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:20.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.021+0000 7f4491600640 1 --2- 172.21.15.27:0/3711016867 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f448c15e7d0 0x7f448c0af280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:20.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.021+0000 7f4490c00640 1 --2- 172.21.15.27:0/3711016867 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f448c0cedc0 0x7f448c1553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:20.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.021+0000 7f448be00640 1 -- 172.21.15.27:0/3711016867 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f448c0cedc0 msgr2=0x7f448c1553e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:20.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.021+0000 7f448be00640 1 --2- 172.21.15.27:0/3711016867 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f448c0cedc0 0x7f448c1553e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.021+0000 7f448be00640 1 -- 172.21.15.27:0/3711016867 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f448c15e7d0 msgr2=0x7f448c0af280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:20.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.021+0000 7f448be00640 1 --2- 172.21.15.27:0/3711016867 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f448c15e7d0 0x7f448c0af280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.021+0000 7f448be00640 1 -- 172.21.15.27:0/3711016867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4484009a70 con 0x7f448c0cf9b0 2024-01-15T18:23:20.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.021+0000 7f4491600640 1 --2- 172.21.15.27:0/3711016867 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f448c15e7d0 0x7f448c0af280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:23:20.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.021+0000 7f448be00640 1 --2- 172.21.15.27:0/3711016867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f448c0cf9b0 0x7f448c0aed40 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f447400c7c0 tx=0x7f447400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:20.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.021+0000 7f4488c00640 1 -- 172.21.15.27:0/3711016867 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4474007a30 con 0x7f448c0cf9b0 2024-01-15T18:23:20.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.021+0000 7f4488c00640 1 -- 172.21.15.27:0/3711016867 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4474007b90 con 0x7f448c0cf9b0 2024-01-15T18:23:20.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.021+0000 7f4488c00640 1 -- 172.21.15.27:0/3711016867 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447400ce70 con 0x7f448c0cf9b0 2024-01-15T18:23:20.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.021+0000 7f4493600640 1 -- 172.21.15.27:0/3711016867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f448c158590 con 0x7f448c0cf9b0 2024-01-15T18:23:20.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.021+0000 7f4493600640 1 -- 172.21.15.27:0/3711016867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f448c155920 con 0x7f448c0cf9b0 2024-01-15T18:23:20.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.025+0000 7f4493600640 1 -- 172.21.15.27:0/3711016867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f448c0c49b0 con 0x7f448c0cf9b0 2024-01-15T18:23:20.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.025+0000 7f4488c00640 1 -- 172.21.15.27:0/3711016867 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f44740113f0 con 0x7f448c0cf9b0 2024-01-15T18:23:20.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.025+0000 7f4488c00640 1 --2- 172.21.15.27:0/3711016867 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4464067a60 0x7f4464069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:20.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.025+0000 7f4488c00640 1 -- 172.21.15.27:0/3711016867 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f44740836d0 con 0x7f448c0cf9b0 2024-01-15T18:23:20.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.025+0000 7f4490c00640 1 --2- 172.21.15.27:0/3711016867 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4464067a60 0x7f4464069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:20.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.025+0000 7f4490c00640 1 --2- 172.21.15.27:0/3711016867 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4464067a60 0x7f4464069f20 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f447c007fd0 tx=0x7f447c007d60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:20.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.029+0000 7f4488c00640 1 -- 172.21.15.27:0/3711016867 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f448c0c49b0 con 0x7f448c0cf9b0 2024-01-15T18:23:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.189+0000 7f4493600640 1 -- 172.21.15.27:0/3711016867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f448c0c49b0 con 0x7f448c0cf9b0 2024-01-15T18:23:20.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.193+0000 7f4488c00640 1 -- 172.21.15.27:0/3711016867 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16761 (secure 0 0 0) 0x7f448c0c49b0 con 0x7f448c0cf9b0 2024-01-15T18:23:20.189 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:23:20.189 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":110,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:18:42.250892+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:18:30.993882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-15T18:18:31.582649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.5","mappings":[{"from":0,"to":5}]},{"pgid":"17.18","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:23:20.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.193+0000 7f4493600640 1 -- 172.21.15.27:0/3711016867 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4464067a60 msgr2=0x7f4464069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:20.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.193+0000 7f4493600640 1 --2- 172.21.15.27:0/3711016867 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4464067a60 0x7f4464069f20 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f447c007fd0 tx=0x7f447c007d60 comp rx=0 tx=0).stop 2024-01-15T18:23:20.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.193+0000 7f4493600640 1 -- 172.21.15.27:0/3711016867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f448c0cf9b0 msgr2=0x7f448c0aed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:20.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.193+0000 7f4493600640 1 --2- 172.21.15.27:0/3711016867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f448c0cf9b0 0x7f448c0aed40 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f447400c7c0 tx=0x7f447400cc80 comp rx=0 tx=0).stop 2024-01-15T18:23:20.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.193+0000 7f4493600640 1 -- 172.21.15.27:0/3711016867 shutdown_connections 2024-01-15T18:23:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.193+0000 7f4493600640 1 --2- 172.21.15.27:0/3711016867 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4464067a60 0x7f4464069f20 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.193+0000 7f4493600640 1 --2- 172.21.15.27:0/3711016867 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f448c15e7d0 0x7f448c0af280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.193+0000 7f4493600640 1 --2- 172.21.15.27:0/3711016867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f448c0cf9b0 0x7f448c0aed40 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.193+0000 7f4493600640 1 --2- 172.21.15.27:0/3711016867 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f448c0cedc0 0x7f448c1553e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.193+0000 7f4493600640 1 -- 172.21.15.27:0/3711016867 >> 172.21.15.27:0/3711016867 conn(0x7f448c0b2a10 msgr2=0x7f448c1589b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.193+0000 7f4493600640 1 -- 172.21.15.27:0/3711016867 shutdown_connections 2024-01-15T18:23:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.193+0000 7f4493600640 1 -- 172.21.15.27:0/3711016867 wait complete. 2024-01-15T18:23:20.211 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T18:18:30.993882+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T18:18:31.582649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:23:20.212 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:23:20.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.361+0000 7f6145000640 1 Processor -- start 2024-01-15T18:23:20.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.361+0000 7f6145000640 1 -- start start 2024-01-15T18:23:20.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.365+0000 7f6145000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6140158900 0x7f614015f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:20.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.365+0000 7f6145000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f614015bf00 0x7f61400cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:20.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.365+0000 7f6145000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61400b1cb0 con 0x7f61400cf110 2024-01-15T18:23:20.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.365+0000 7f6145000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61400cf510 con 0x7f614015bf00 2024-01-15T18:23:20.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.365+0000 7f6145000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61400b1f00 con 0x7f6140158900 2024-01-15T18:23:20.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.365+0000 7f613de00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6140158900 0x7f614015f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:20.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.365+0000 7f613d400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f614015bf00 0x7f61400cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:20.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.365+0000 7f613d400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f614015bf00 0x7f61400cebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:56206/0 (socket says 172.21.15.27:56206) 2024-01-15T18:23:20.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.365+0000 7f613d400640 1 -- 172.21.15.27:0/1965169384 learned_addr learned my addr 172.21.15.27:0/1965169384 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:20.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.365+0000 7f613e800640 1 --1- 172.21.15.27:0/1965169384 >> v1:172.21.15.93:6789/0 conn(0x7f61400cf110 0x7f61400701e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.93:6789/0 says I am v1:172.21.15.27:38080/0 (socket says 172.21.15.27:38080) 2024-01-15T18:23:20.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.365+0000 7f613d400640 1 -- 172.21.15.27:0/1965169384 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6140158900 msgr2=0x7f614015f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:20.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.365+0000 7f613d400640 1 --2- 172.21.15.27:0/1965169384 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6140158900 0x7f614015f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.365+0000 7f613d400640 1 -- 172.21.15.27:0/1965169384 >> v1:172.21.15.93:6789/0 conn(0x7f61400cf110 legacy=0x7f61400701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:20.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.365+0000 7f613d400640 1 -- 172.21.15.27:0/1965169384 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61400b21e0 con 0x7f614015bf00 2024-01-15T18:23:20.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.365+0000 7f613de00640 1 --2- 172.21.15.27:0/1965169384 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6140158900 0x7f614015f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:23:20.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.369+0000 7f613d400640 1 --2- 172.21.15.27:0/1965169384 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f614015bf00 0x7f61400cebd0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f6128009870 tx=0x7f612801c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e2104cb3afff64b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:20.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.369+0000 7f613ca00640 1 -- 172.21.15.27:0/1965169384 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61280a7070 con 0x7f614015bf00 2024-01-15T18:23:20.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.369+0000 7f613ca00640 1 -- 172.21.15.27:0/1965169384 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61280ab430 con 0x7f614015bf00 2024-01-15T18:23:20.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.369+0000 7f613ca00640 1 -- 172.21.15.27:0/1965169384 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61280ab7c0 con 0x7f614015bf00 2024-01-15T18:23:20.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.369+0000 7f6145000640 1 -- 172.21.15.27:0/1965169384 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f614015bf00 msgr2=0x7f61400cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:20.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.369+0000 7f6145000640 1 --2- 172.21.15.27:0/1965169384 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f614015bf00 0x7f61400cebd0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f6128009870 tx=0x7f612801c540 comp rx=0 tx=0).stop 2024-01-15T18:23:20.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.369+0000 7f6145000640 1 -- 172.21.15.27:0/1965169384 shutdown_connections 2024-01-15T18:23:20.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.369+0000 7f6145000640 1 --2- 172.21.15.27:0/1965169384 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f614015bf00 0x7f61400cebd0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.369+0000 7f6145000640 1 --2- 172.21.15.27:0/1965169384 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6140158900 0x7f614015f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.369+0000 7f6145000640 1 -- 172.21.15.27:0/1965169384 >> 172.21.15.27:0/1965169384 conn(0x7f61400cb700 msgr2=0x7f61400cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:20.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.369+0000 7f6145000640 1 -- 172.21.15.27:0/1965169384 shutdown_connections 2024-01-15T18:23:20.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.369+0000 7f6145000640 1 -- 172.21.15.27:0/1965169384 wait complete. 2024-01-15T18:23:20.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.369+0000 7f6145000640 1 Processor -- start 2024-01-15T18:23:20.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.369+0000 7f6145000640 1 -- start start 2024-01-15T18:23:20.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.373+0000 7f6145000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61400cf110 0x7f614014da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:20.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.373+0000 7f6145000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6140158900 0x7f6140153560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:20.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.373+0000 7f6145000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f614015bf00 0x7f6140153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:20.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.373+0000 7f6145000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614015aa70 con 0x7f6140158900 2024-01-15T18:23:20.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.373+0000 7f6145000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6140153fe0 con 0x7f614015bf00 2024-01-15T18:23:20.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.373+0000 7f6145000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6140152110 con 0x7f61400cf110 2024-01-15T18:23:20.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.373+0000 7f613d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6140158900 0x7f6140153560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:20.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.373+0000 7f613d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6140158900 0x7f6140153560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35124/0 (socket says 172.21.15.27:35124) 2024-01-15T18:23:20.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.373+0000 7f613d400640 1 -- 172.21.15.27:0/2166986574 learned_addr learned my addr 172.21.15.27:0/2166986574 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:20.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.373+0000 7f613de00640 1 --2- 172.21.15.27:0/2166986574 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61400cf110 0x7f614014da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:20.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.373+0000 7f613e800640 1 --2- 172.21.15.27:0/2166986574 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f614015bf00 0x7f6140153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:20.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.373+0000 7f613d400640 1 -- 172.21.15.27:0/2166986574 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61400cf110 msgr2=0x7f614014da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:20.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.373+0000 7f613d400640 1 --2- 172.21.15.27:0/2166986574 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61400cf110 0x7f614014da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.373+0000 7f613d400640 1 -- 172.21.15.27:0/2166986574 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f614015bf00 msgr2=0x7f6140153aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:20.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.373+0000 7f613d400640 1 --2- 172.21.15.27:0/2166986574 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f614015bf00 0x7f6140153aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.373+0000 7f613d400640 1 -- 172.21.15.27:0/2166986574 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6130009420 con 0x7f6140158900 2024-01-15T18:23:20.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.373+0000 7f613e800640 1 --2- 172.21.15.27:0/2166986574 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f614015bf00 0x7f6140153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:23:20.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.373+0000 7f613d400640 1 --2- 172.21.15.27:0/2166986574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6140158900 0x7f6140153560 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f61280ad040 tx=0x7f61280192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:20.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.373+0000 7f6136000640 1 -- 172.21.15.27:0/2166986574 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61280b4600 con 0x7f6140158900 2024-01-15T18:23:20.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.373+0000 7f6136000640 1 -- 172.21.15.27:0/2166986574 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61280b4780 con 0x7f6140158900 2024-01-15T18:23:20.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.373+0000 7f6136000640 1 -- 172.21.15.27:0/2166986574 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61280b4b30 con 0x7f6140158900 2024-01-15T18:23:20.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.373+0000 7f6145000640 1 -- 172.21.15.27:0/2166986574 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61280094d0 con 0x7f6140158900 2024-01-15T18:23:20.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.373+0000 7f6145000640 1 -- 172.21.15.27:0/2166986574 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f614015a4b0 con 0x7f6140158900 2024-01-15T18:23:20.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.373+0000 7f6145000640 1 -- 172.21.15.27:0/2166986574 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f614004e850 con 0x7f6140158900 2024-01-15T18:23:20.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.377+0000 7f6136000640 1 -- 172.21.15.27:0/2166986574 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f61280b4c90 con 0x7f6140158900 2024-01-15T18:23:20.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.377+0000 7f6136000640 1 --2- 172.21.15.27:0/2166986574 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6118067a60 0x7f6118069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:20.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.377+0000 7f6136000640 1 -- 172.21.15.27:0/2166986574 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f612811d1f0 con 0x7f6140158900 2024-01-15T18:23:20.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.377+0000 7f613de00640 1 --2- 172.21.15.27:0/2166986574 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6118067a60 0x7f6118069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:20.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.377+0000 7f613de00640 1 --2- 172.21.15.27:0/2166986574 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6118067a60 0x7f6118069f20 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f6130009c80 tx=0x7f6130008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:20.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.381+0000 7f6136000640 1 -- 172.21.15.27:0/2166986574 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f61280ebb90 con 0x7f6140158900 2024-01-15T18:23:20.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.541+0000 7f6145000640 1 -- 172.21.15.27:0/2166986574 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f61400cbae0 con 0x7f6140158900 2024-01-15T18:23:20.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.541+0000 7f6136000640 1 -- 172.21.15.27:0/2166986574 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v110) v1 ==== 93+0+10 (secure 0 0 0) 0x7f61280ef1b0 con 0x7f6140158900 2024-01-15T18:23:20.536 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:23:20.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.541+0000 7f6145000640 1 -- 172.21.15.27:0/2166986574 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6118067a60 msgr2=0x7f6118069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:20.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.541+0000 7f6145000640 1 --2- 172.21.15.27:0/2166986574 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6118067a60 0x7f6118069f20 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f6130009c80 tx=0x7f6130008040 comp rx=0 tx=0).stop 2024-01-15T18:23:20.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.541+0000 7f6145000640 1 -- 172.21.15.27:0/2166986574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6140158900 msgr2=0x7f6140153560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:20.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.541+0000 7f6145000640 1 --2- 172.21.15.27:0/2166986574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6140158900 0x7f6140153560 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f61280ad040 tx=0x7f61280192e0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.541+0000 7f6145000640 1 -- 172.21.15.27:0/2166986574 shutdown_connections 2024-01-15T18:23:20.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.541+0000 7f6145000640 1 --2- 172.21.15.27:0/2166986574 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6118067a60 0x7f6118069f20 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.541+0000 7f6145000640 1 --2- 172.21.15.27:0/2166986574 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f614015bf00 0x7f6140153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.541+0000 7f6145000640 1 --2- 172.21.15.27:0/2166986574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6140158900 0x7f6140153560 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.541+0000 7f6145000640 1 --2- 172.21.15.27:0/2166986574 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f61400cf110 0x7f614014da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.541+0000 7f6145000640 1 -- 172.21.15.27:0/2166986574 >> 172.21.15.27:0/2166986574 conn(0x7f61400cb700 msgr2=0x7f614006c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:20.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.541+0000 7f6145000640 1 -- 172.21.15.27:0/2166986574 shutdown_connections 2024-01-15T18:23:20.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.541+0000 7f6145000640 1 -- 172.21.15.27:0/2166986574 wait complete. 2024-01-15T18:23:20.558 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T18:23:20.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.705+0000 7f9868000640 1 Processor -- start 2024-01-15T18:23:20.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.705+0000 7f9868000640 1 -- start start 2024-01-15T18:23:20.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.709+0000 7f9868000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f986006c430 0x7f9860070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:20.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.709+0000 7f9868000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f98600b1db0 0x7f98600b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:20.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.709+0000 7f9868000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f98600ce050 0x7f98600ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:20.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.709+0000 7f9868000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98600cea20 con 0x7f98600ce050 2024-01-15T18:23:20.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.709+0000 7f9868000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98600ccfb0 con 0x7f98600b1db0 2024-01-15T18:23:20.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.709+0000 7f9868000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98600cd290 con 0x7f986006c430 2024-01-15T18:23:20.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.709+0000 7f9866000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f98600ce050 0x7f98600ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:20.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.709+0000 7f9866000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f98600ce050 0x7f98600ce4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35138/0 (socket says 172.21.15.27:35138) 2024-01-15T18:23:20.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.709+0000 7f9866000640 1 -- 172.21.15.27:0/520796848 learned_addr learned my addr 172.21.15.27:0/520796848 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:20.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.709+0000 7f9864c00640 1 --2- 172.21.15.27:0/520796848 >> v2:172.21.15.93:3300/0 conn(0x7f98600b1db0 0x7f98600b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:20.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.709+0000 7f9865600640 1 --2- 172.21.15.27:0/520796848 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f986006c430 0x7f9860070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:20.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.709+0000 7f9866000640 1 -- 172.21.15.27:0/520796848 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f986006c430 msgr2=0x7f9860070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:20.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.709+0000 7f9866000640 1 --2- 172.21.15.27:0/520796848 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f986006c430 0x7f9860070ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.709+0000 7f9866000640 1 -- 172.21.15.27:0/520796848 >> v2:172.21.15.93:3300/0 conn(0x7f98600b1db0 msgr2=0x7f98600b2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:20.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.709+0000 7f9866000640 1 --2- 172.21.15.27:0/520796848 >> v2:172.21.15.93:3300/0 conn(0x7f98600b1db0 0x7f98600b2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.709+0000 7f9866000640 1 -- 172.21.15.27:0/520796848 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98600cd570 con 0x7f98600ce050 2024-01-15T18:23:20.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.709+0000 7f9865600640 1 --2- 172.21.15.27:0/520796848 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f986006c430 0x7f9860070ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:23:20.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.709+0000 7f9866000640 1 --2- 172.21.15.27:0/520796848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98600ce050 0x7f98600ce4e0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f98600cc6d0 tx=0x7f985801c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a0494f03d8e856 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:20.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.713+0000 7f985fe00640 1 -- 172.21.15.27:0/520796848 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98580a5630 con 0x7f98600ce050 2024-01-15T18:23:20.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.713+0000 7f985fe00640 1 -- 172.21.15.27:0/520796848 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98580a5790 con 0x7f98600ce050 2024-01-15T18:23:20.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.713+0000 7f985fe00640 1 -- 172.21.15.27:0/520796848 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98580a5b20 con 0x7f98600ce050 2024-01-15T18:23:20.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.713+0000 7f9868000640 1 -- 172.21.15.27:0/520796848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98600ce050 msgr2=0x7f98600ce4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:20.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.713+0000 7f9868000640 1 --2- 172.21.15.27:0/520796848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98600ce050 0x7f98600ce4e0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f98600cc6d0 tx=0x7f985801c940 comp rx=0 tx=0).stop 2024-01-15T18:23:20.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.713+0000 7f9868000640 1 -- 172.21.15.27:0/520796848 shutdown_connections 2024-01-15T18:23:20.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.713+0000 7f9868000640 1 --2- 172.21.15.27:0/520796848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98600ce050 0x7f98600ce4e0 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.713+0000 7f9868000640 1 --2- 172.21.15.27:0/520796848 >> v2:172.21.15.93:3300/0 conn(0x7f98600b1db0 0x7f98600b2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.713+0000 7f9868000640 1 --2- 172.21.15.27:0/520796848 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f986006c430 0x7f9860070ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.713+0000 7f9868000640 1 -- 172.21.15.27:0/520796848 >> 172.21.15.27:0/520796848 conn(0x7f98600cb700 msgr2=0x7f98600cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:20.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.713+0000 7f9868000640 1 -- 172.21.15.27:0/520796848 shutdown_connections 2024-01-15T18:23:20.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.713+0000 7f9868000640 1 -- 172.21.15.27:0/520796848 wait complete. 2024-01-15T18:23:20.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.713+0000 7f9868000640 1 Processor -- start 2024-01-15T18:23:20.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.713+0000 7f9868000640 1 -- start start 2024-01-15T18:23:20.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.713+0000 7f9868000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f986006c430 0x7f98601f4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:20.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.713+0000 7f9868000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f98600b1db0 0x7f9860134f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:20.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.713+0000 7f9868000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98600ce050 0x7f98601f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:20.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.713+0000 7f9868000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98601521c0 con 0x7f98600ce050 2024-01-15T18:23:20.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.713+0000 7f9868000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98600b3b50 con 0x7f98600b1db0 2024-01-15T18:23:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.713+0000 7f9868000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98601f4540 con 0x7f986006c430 2024-01-15T18:23:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.717+0000 7f9866000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98600ce050 0x7f98601f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.717+0000 7f9866000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98600ce050 0x7f98601f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35146/0 (socket says 172.21.15.27:35146) 2024-01-15T18:23:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.717+0000 7f9865600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f986006c430 0x7f98601f4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.717+0000 7f9866000640 1 -- 172.21.15.27:0/1186964122 learned_addr learned my addr 172.21.15.27:0/1186964122 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.717+0000 7f9864c00640 1 --2- 172.21.15.27:0/1186964122 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f98600b1db0 0x7f9860134f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.717+0000 7f9866000640 1 -- 172.21.15.27:0/1186964122 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f986006c430 msgr2=0x7f98601f4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.717+0000 7f9866000640 1 --2- 172.21.15.27:0/1186964122 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f986006c430 0x7f98601f4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.717+0000 7f9866000640 1 -- 172.21.15.27:0/1186964122 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f98600b1db0 msgr2=0x7f9860134f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:20.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.717+0000 7f9866000640 1 --2- 172.21.15.27:0/1186964122 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f98600b1db0 0x7f9860134f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.717+0000 7f9866000640 1 -- 172.21.15.27:0/1186964122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9858009b30 con 0x7f98600ce050 2024-01-15T18:23:20.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.717+0000 7f9866000640 1 --2- 172.21.15.27:0/1186964122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98600ce050 0x7f98601f6b70 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f98580af040 tx=0x7f98580196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:20.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.717+0000 7f985d600640 1 -- 172.21.15.27:0/1186964122 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98580a5630 con 0x7f98600ce050 2024-01-15T18:23:20.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.717+0000 7f985d600640 1 -- 172.21.15.27:0/1186964122 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98580b8670 con 0x7f98600ce050 2024-01-15T18:23:20.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.717+0000 7f9868000640 1 -- 172.21.15.27:0/1186964122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9860135480 con 0x7f98600ce050 2024-01-15T18:23:20.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.717+0000 7f985d600640 1 -- 172.21.15.27:0/1186964122 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98580b8a00 con 0x7f98600ce050 2024-01-15T18:23:20.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.717+0000 7f9868000640 1 -- 172.21.15.27:0/1186964122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9860151b70 con 0x7f98600ce050 2024-01-15T18:23:20.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.717+0000 7f9868000640 1 -- 172.21.15.27:0/1186964122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9860152370 con 0x7f98600ce050 2024-01-15T18:23:20.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.717+0000 7f985d600640 1 -- 172.21.15.27:0/1186964122 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f98580b8b60 con 0x7f98600ce050 2024-01-15T18:23:20.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.721+0000 7f985d600640 1 --2- 172.21.15.27:0/1186964122 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9838067a90 0x7f9838069f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:20.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.721+0000 7f985d600640 1 -- 172.21.15.27:0/1186964122 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f985811de80 con 0x7f98600ce050 2024-01-15T18:23:20.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.721+0000 7f9865600640 1 --2- 172.21.15.27:0/1186964122 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9838067a90 0x7f9838069f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:20.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.721+0000 7f9865600640 1 --2- 172.21.15.27:0/1186964122 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9838067a90 0x7f9838069f50 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f9850006e90 tx=0x7f9850012040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:20.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.725+0000 7f985d600640 1 -- 172.21.15.27:0/1186964122 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9860152370 con 0x7f98600ce050 2024-01-15T18:23:20.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.885+0000 7f9868000640 1 -- 172.21.15.27:0/1186964122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9860152370 con 0x7f98600ce050 2024-01-15T18:23:20.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.885+0000 7f985d600640 1 -- 172.21.15.27:0/1186964122 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v110) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9860152370 con 0x7f98600ce050 2024-01-15T18:23:20.880 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-15T18:23:20.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.885+0000 7f9868000640 1 -- 172.21.15.27:0/1186964122 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9838067a90 msgr2=0x7f9838069f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:20.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.885+0000 7f9868000640 1 --2- 172.21.15.27:0/1186964122 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9838067a90 0x7f9838069f50 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f9850006e90 tx=0x7f9850012040 comp rx=0 tx=0).stop 2024-01-15T18:23:20.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.885+0000 7f9868000640 1 -- 172.21.15.27:0/1186964122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98600ce050 msgr2=0x7f98601f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:20.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.885+0000 7f9868000640 1 --2- 172.21.15.27:0/1186964122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98600ce050 0x7f98601f6b70 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f98580af040 tx=0x7f98580196e0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.885+0000 7f9868000640 1 -- 172.21.15.27:0/1186964122 shutdown_connections 2024-01-15T18:23:20.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.885+0000 7f9868000640 1 --2- 172.21.15.27:0/1186964122 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9838067a90 0x7f9838069f50 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.885+0000 7f9868000640 1 --2- 172.21.15.27:0/1186964122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98600ce050 0x7f98601f6b70 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.885+0000 7f9868000640 1 --2- 172.21.15.27:0/1186964122 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f98600b1db0 0x7f9860134f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.885+0000 7f9868000640 1 --2- 172.21.15.27:0/1186964122 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f986006c430 0x7f98601f4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:20.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.885+0000 7f9868000640 1 -- 172.21.15.27:0/1186964122 >> 172.21.15.27:0/1186964122 conn(0x7f98600cb700 msgr2=0x7f98600ae920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:20.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.885+0000 7f9868000640 1 -- 172.21.15.27:0/1186964122 shutdown_connections 2024-01-15T18:23:20.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:20.885+0000 7f9868000640 1 -- 172.21.15.27:0/1186964122 wait complete. 2024-01-15T18:23:20.901 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T18:23:21.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.049+0000 7f597a800640 1 Processor -- start 2024-01-15T18:23:21.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.049+0000 7f597a800640 1 -- start start 2024-01-15T18:23:21.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.049+0000 7f597a800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f597415b4b0 0x7f59740b1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:21.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.049+0000 7f597a800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f59740cec60 0x7f59740b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:21.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.049+0000 7f597a800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59740d19a0 con 0x7f59740cf460 2024-01-15T18:23:21.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.049+0000 7f597a800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59740d1c80 con 0x7f59740cec60 2024-01-15T18:23:21.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.049+0000 7f597a800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59740b07c0 con 0x7f597415b4b0 2024-01-15T18:23:21.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.049+0000 7f5973e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f59740cec60 0x7f59740b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:21.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.049+0000 7f5973400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f59740cf460 0x7f59740cf840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50224/0 (socket says 172.21.15.27:50224) 2024-01-15T18:23:21.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.049+0000 7f5973400640 1 -- 172.21.15.27:0/3559584583 learned_addr learned my addr 172.21.15.27:0/3559584583 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:21.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.049+0000 7f5972a00640 1 --2- 172.21.15.27:0/3559584583 >> v2:172.21.15.93:3300/0 conn(0x7f597415b4b0 0x7f59740b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:21.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.049+0000 7f5973e00640 1 -- 172.21.15.27:0/3559584583 >> v2:172.21.15.93:3300/0 conn(0x7f597415b4b0 msgr2=0x7f59740b1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:21.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.049+0000 7f5973e00640 1 --2- 172.21.15.27:0/3559584583 >> v2:172.21.15.93:3300/0 conn(0x7f597415b4b0 0x7f59740b1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.049+0000 7f5973e00640 1 -- 172.21.15.27:0/3559584583 >> v1:172.21.15.27:6789/0 conn(0x7f59740cf460 legacy=0x7f59740cf840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:21.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.049+0000 7f5973e00640 1 -- 172.21.15.27:0/3559584583 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59740b0aa0 con 0x7f59740cec60 2024-01-15T18:23:21.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.049+0000 7f5972a00640 1 --2- 172.21.15.27:0/3559584583 >> v2:172.21.15.93:3300/0 conn(0x7f597415b4b0 0x7f59740b1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:23:21.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.049+0000 7f5972000640 1 -- 172.21.15.27:0/3559584583 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727336854 0 0) 0x7f59740d19a0 con 0x7f59740cf460 2024-01-15T18:23:21.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.053+0000 7f5973e00640 1 --2- 172.21.15.27:0/3559584583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59740cec60 0x7f59740b1ac0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f59740717b0 tx=0x7f596c01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36ccde962d201684 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:21.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.053+0000 7f5972000640 1 -- 172.21.15.27:0/3559584583 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59740b2810 con 0x7f59740cec60 2024-01-15T18:23:21.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.053+0000 7f5972000640 1 -- 172.21.15.27:0/3559584583 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596c0a59d0 con 0x7f59740cec60 2024-01-15T18:23:21.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.053+0000 7f5972000640 1 -- 172.21.15.27:0/3559584583 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596c0a5d60 con 0x7f59740cec60 2024-01-15T18:23:21.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.053+0000 7f597a800640 1 -- 172.21.15.27:0/3559584583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59740cec60 msgr2=0x7f59740b1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:21.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.053+0000 7f597a800640 1 --2- 172.21.15.27:0/3559584583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59740cec60 0x7f59740b1ac0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f59740717b0 tx=0x7f596c01c910 comp rx=0 tx=0).stop 2024-01-15T18:23:21.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.053+0000 7f597a800640 1 -- 172.21.15.27:0/3559584583 shutdown_connections 2024-01-15T18:23:21.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.053+0000 7f597a800640 1 --2- 172.21.15.27:0/3559584583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f59740cec60 0x7f59740b1ac0 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.053+0000 7f597a800640 1 --2- 172.21.15.27:0/3559584583 >> v2:172.21.15.93:3300/0 conn(0x7f597415b4b0 0x7f59740b1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.053+0000 7f597a800640 1 -- 172.21.15.27:0/3559584583 >> 172.21.15.27:0/3559584583 conn(0x7f59740d0e80 msgr2=0x7f59740d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:21.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.053+0000 7f597a800640 1 -- 172.21.15.27:0/3559584583 shutdown_connections 2024-01-15T18:23:21.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.053+0000 7f597a800640 1 -- 172.21.15.27:0/3559584583 wait complete. 2024-01-15T18:23:21.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.057+0000 7f597a800640 1 Processor -- start 2024-01-15T18:23:21.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.057+0000 7f597a800640 1 -- start start 2024-01-15T18:23:21.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.057+0000 7f597a800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59740cec60 0x7f5974150850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:21.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.057+0000 7f597a800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59740cf460 0x7f597406b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:21.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.057+0000 7f597a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f597415b4b0 0x7f597406bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:21.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.057+0000 7f597a800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597406e480 con 0x7f597415b4b0 2024-01-15T18:23:21.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.057+0000 7f597a800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597406e760 con 0x7f59740cf460 2024-01-15T18:23:21.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.057+0000 7f597a800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f597406ea40 con 0x7f59740cec60 2024-01-15T18:23:21.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.057+0000 7f5973e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f597415b4b0 0x7f597406bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:21.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.057+0000 7f5973e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f597415b4b0 0x7f597406bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35172/0 (socket says 172.21.15.27:35172) 2024-01-15T18:23:21.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.057+0000 7f5973e00640 1 -- 172.21.15.27:0/3020499000 learned_addr learned my addr 172.21.15.27:0/3020499000 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:21.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.057+0000 7f5972a00640 1 --2- 172.21.15.27:0/3020499000 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59740cf460 0x7f597406b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:21.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.057+0000 7f5973400640 1 --2- 172.21.15.27:0/3020499000 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59740cec60 0x7f5974150850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:21.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.057+0000 7f5973e00640 1 -- 172.21.15.27:0/3020499000 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59740cec60 msgr2=0x7f5974150850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:21.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.057+0000 7f5973e00640 1 --2- 172.21.15.27:0/3020499000 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59740cec60 0x7f5974150850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.057+0000 7f5973e00640 1 -- 172.21.15.27:0/3020499000 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59740cf460 msgr2=0x7f597406b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:21.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.057+0000 7f5973e00640 1 --2- 172.21.15.27:0/3020499000 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59740cf460 0x7f597406b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.057+0000 7f5973e00640 1 -- 172.21.15.27:0/3020499000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596c009a70 con 0x7f597415b4b0 2024-01-15T18:23:21.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.057+0000 7f5973e00640 1 --2- 172.21.15.27:0/3020499000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f597415b4b0 0x7f597406bd70 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f596c019520 tx=0x7f596c0196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:21.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.057+0000 7f596b400640 1 -- 172.21.15.27:0/3020499000 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596c0ad6d0 con 0x7f597415b4b0 2024-01-15T18:23:21.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.057+0000 7f596b400640 1 -- 172.21.15.27:0/3020499000 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596c0ad830 con 0x7f597415b4b0 2024-01-15T18:23:21.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.057+0000 7f597a800640 1 -- 172.21.15.27:0/3020499000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f597406ed20 con 0x7f597415b4b0 2024-01-15T18:23:21.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.057+0000 7f596b400640 1 -- 172.21.15.27:0/3020499000 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596c0adba0 con 0x7f597415b4b0 2024-01-15T18:23:21.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.061+0000 7f597a800640 1 -- 172.21.15.27:0/3020499000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5974150d90 con 0x7f597415b4b0 2024-01-15T18:23:21.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.061+0000 7f597a800640 1 -- 172.21.15.27:0/3020499000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5974150f20 con 0x7f597415b4b0 2024-01-15T18:23:21.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.061+0000 7f596b400640 1 -- 172.21.15.27:0/3020499000 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f596c0add00 con 0x7f597415b4b0 2024-01-15T18:23:21.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.061+0000 7f596b400640 1 --2- 172.21.15.27:0/3020499000 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f594c067a60 0x7f594c069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:21.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.061+0000 7f596b400640 1 -- 172.21.15.27:0/3020499000 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f596c11e2c0 con 0x7f597415b4b0 2024-01-15T18:23:21.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.061+0000 7f5973400640 1 --2- 172.21.15.27:0/3020499000 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f594c067a60 0x7f594c069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:21.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.061+0000 7f5973400640 1 --2- 172.21.15.27:0/3020499000 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f594c067a60 0x7f594c069f20 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f5964007fd0 tx=0x7f5964007d60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:21.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.065+0000 7f596b400640 1 -- 172.21.15.27:0/3020499000 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f596c0ed060 con 0x7f597415b4b0 2024-01-15T18:23:21.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.225+0000 7f597a800640 1 -- 172.21.15.27:0/3020499000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f597404e850 con 0x7f597415b4b0 2024-01-15T18:23:21.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.225+0000 7f596b400640 1 -- 172.21.15.27:0/3020499000 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v110) v1 ==== 100+0+11 (secure 0 0 0) 0x7f596c0b2020 con 0x7f597415b4b0 2024-01-15T18:23:21.222 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-15T18:23:21.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.229+0000 7f597a800640 1 -- 172.21.15.27:0/3020499000 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f594c067a60 msgr2=0x7f594c069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.229+0000 7f597a800640 1 --2- 172.21.15.27:0/3020499000 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f594c067a60 0x7f594c069f20 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f5964007fd0 tx=0x7f5964007d60 comp rx=0 tx=0).stop 2024-01-15T18:23:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.229+0000 7f597a800640 1 -- 172.21.15.27:0/3020499000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f597415b4b0 msgr2=0x7f597406bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.229+0000 7f597a800640 1 --2- 172.21.15.27:0/3020499000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f597415b4b0 0x7f597406bd70 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f596c019520 tx=0x7f596c0196b0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.229+0000 7f597a800640 1 -- 172.21.15.27:0/3020499000 shutdown_connections 2024-01-15T18:23:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.229+0000 7f597a800640 1 --2- 172.21.15.27:0/3020499000 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f594c067a60 0x7f594c069f20 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.229+0000 7f597a800640 1 --2- 172.21.15.27:0/3020499000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f597415b4b0 0x7f597406bd70 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.229+0000 7f597a800640 1 --2- 172.21.15.27:0/3020499000 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f59740cf460 0x7f597406b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.229+0000 7f597a800640 1 --2- 172.21.15.27:0/3020499000 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f59740cec60 0x7f5974150850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.229+0000 7f597a800640 1 -- 172.21.15.27:0/3020499000 >> 172.21.15.27:0/3020499000 conn(0x7f59740d0e80 msgr2=0x7f59740b1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:21.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.229+0000 7f597a800640 1 -- 172.21.15.27:0/3020499000 shutdown_connections 2024-01-15T18:23:21.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.229+0000 7f597a800640 1 -- 172.21.15.27:0/3020499000 wait complete. 2024-01-15T18:23:21.244 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:23:21.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.389+0000 7fa313a00640 1 Processor -- start 2024-01-15T18:23:21.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.389+0000 7fa313a00640 1 -- start start 2024-01-15T18:23:21.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.389+0000 7fa313a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa30c06a4f0 0x7fa30c15ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:21.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.393+0000 7fa313a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa30c0b5360 0x7fa30c0ad640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:21.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.393+0000 7fa313a00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa30c06eef0 con 0x7fa30c159b60 2024-01-15T18:23:21.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.393+0000 7fa313a00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa30c15e410 con 0x7fa30c06a4f0 2024-01-15T18:23:21.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.393+0000 7fa313a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa30c0b5b60 con 0x7fa30c0b5360 2024-01-15T18:23:21.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.393+0000 7fa311000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa30c06a4f0 0x7fa30c15ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:21.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.393+0000 7fa311000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa30c06a4f0 0x7fa30c15ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35178/0 (socket says 172.21.15.27:35178) 2024-01-15T18:23:21.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.393+0000 7fa311000640 1 -- 172.21.15.27:0/1921697689 learned_addr learned my addr 172.21.15.27:0/1921697689 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:21.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.393+0000 7fa312400640 1 --2- 172.21.15.27:0/1921697689 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa30c0b5360 0x7fa30c0ad640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:21.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.393+0000 7fa311000640 1 -- 172.21.15.27:0/1921697689 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa30c0b5360 msgr2=0x7fa30c0ad640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:21.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.393+0000 7fa311000640 1 --2- 172.21.15.27:0/1921697689 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa30c0b5360 0x7fa30c0ad640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.393+0000 7fa311000640 1 -- 172.21.15.27:0/1921697689 >> v1:172.21.15.93:6789/0 conn(0x7fa30c159b60 legacy=0x7fa30c06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:21.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.393+0000 7fa311000640 1 -- 172.21.15.27:0/1921697689 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa30c0b3700 con 0x7fa30c06a4f0 2024-01-15T18:23:21.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.393+0000 7fa311000640 1 --2- 172.21.15.27:0/1921697689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa30c06a4f0 0x7fa30c15ded0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc009870 tx=0x7fa2fc01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=662b0018097bfffb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:21.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.393+0000 7fa30b400640 1 -- 172.21.15.27:0/1921697689 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc0aa5f0 con 0x7fa30c06a4f0 2024-01-15T18:23:21.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.393+0000 7fa30b400640 1 -- 172.21.15.27:0/1921697689 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2fc0aa750 con 0x7fa30c06a4f0 2024-01-15T18:23:21.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.393+0000 7fa30b400640 1 -- 172.21.15.27:0/1921697689 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc0aaae0 con 0x7fa30c06a4f0 2024-01-15T18:23:21.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.397+0000 7fa313a00640 1 -- 172.21.15.27:0/1921697689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa30c06a4f0 msgr2=0x7fa30c15ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:21.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.397+0000 7fa313a00640 1 --2- 172.21.15.27:0/1921697689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa30c06a4f0 0x7fa30c15ded0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc009870 tx=0x7fa2fc01c510 comp rx=0 tx=0).stop 2024-01-15T18:23:21.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.397+0000 7fa313a00640 1 -- 172.21.15.27:0/1921697689 shutdown_connections 2024-01-15T18:23:21.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.397+0000 7fa313a00640 1 --2- 172.21.15.27:0/1921697689 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa30c0b5360 0x7fa30c0ad640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.397+0000 7fa313a00640 1 --2- 172.21.15.27:0/1921697689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa30c06a4f0 0x7fa30c15ded0 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.397+0000 7fa313a00640 1 -- 172.21.15.27:0/1921697689 >> 172.21.15.27:0/1921697689 conn(0x7fa30c0b2360 msgr2=0x7fa30c0b2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:21.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.397+0000 7fa313a00640 1 -- 172.21.15.27:0/1921697689 shutdown_connections 2024-01-15T18:23:21.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.397+0000 7fa313a00640 1 -- 172.21.15.27:0/1921697689 wait complete. 2024-01-15T18:23:21.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.397+0000 7fa313a00640 1 Processor -- start 2024-01-15T18:23:21.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.397+0000 7fa313a00640 1 -- start start 2024-01-15T18:23:21.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.397+0000 7fa313a00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa30c06a4f0 0x7fa30c0ae9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:21.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.397+0000 7fa313a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa30c0b5360 0x7fa30c140c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:21.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.397+0000 7fa313a00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa30c159b60 0x7fa30c141180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:21.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.397+0000 7fa313a00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30c158a20 con 0x7fa30c0b5360 2024-01-15T18:23:21.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.397+0000 7fa313a00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30c158d00 con 0x7fa30c159b60 2024-01-15T18:23:21.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.397+0000 7fa313a00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa30c158fe0 con 0x7fa30c06a4f0 2024-01-15T18:23:21.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.401+0000 7fa30be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa30c0b5360 0x7fa30c140c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:21.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.401+0000 7fa311000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa30c06a4f0 0x7fa30c0ae9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:21.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.401+0000 7fa312400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa30c159b60 0x7fa30c141180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:21.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.401+0000 7fa30be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa30c0b5360 0x7fa30c140c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35194/0 (socket says 172.21.15.27:35194) 2024-01-15T18:23:21.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.401+0000 7fa30be00640 1 -- 172.21.15.27:0/314901032 learned_addr learned my addr 172.21.15.27:0/314901032 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:21.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.401+0000 7fa311000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa30c06a4f0 0x7fa30c0ae9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:50798/0 (socket says 172.21.15.27:50798) 2024-01-15T18:23:21.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.401+0000 7fa312400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa30c159b60 0x7fa30c141180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:56268/0 (socket says 172.21.15.27:56268) 2024-01-15T18:23:21.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.401+0000 7fa30be00640 1 -- 172.21.15.27:0/314901032 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa30c06a4f0 msgr2=0x7fa30c0ae9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:21.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.401+0000 7fa30be00640 1 --2- 172.21.15.27:0/314901032 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa30c06a4f0 0x7fa30c0ae9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.401+0000 7fa30be00640 1 -- 172.21.15.27:0/314901032 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa30c159b60 msgr2=0x7fa30c141180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:21.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.401+0000 7fa30be00640 1 --2- 172.21.15.27:0/314901032 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa30c159b60 0x7fa30c141180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.401+0000 7fa30be00640 1 -- 172.21.15.27:0/314901032 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2fc0094d0 con 0x7fa30c0b5360 2024-01-15T18:23:21.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.401+0000 7fa30be00640 1 --2- 172.21.15.27:0/314901032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa30c0b5360 0x7fa30c140c40 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fa2f400c390 tx=0x7fa2f400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:21.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.401+0000 7fa308c00640 1 -- 172.21.15.27:0/314901032 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f40052d0 con 0x7fa30c0b5360 2024-01-15T18:23:21.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.401+0000 7fa308c00640 1 -- 172.21.15.27:0/314901032 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f4005430 con 0x7fa30c0b5360 2024-01-15T18:23:21.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.401+0000 7fa308c00640 1 -- 172.21.15.27:0/314901032 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4003820 con 0x7fa30c0b5360 2024-01-15T18:23:21.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.401+0000 7fa313a00640 1 -- 172.21.15.27:0/314901032 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa30c141aa0 con 0x7fa30c0b5360 2024-01-15T18:23:21.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.401+0000 7fa313a00640 1 -- 172.21.15.27:0/314901032 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa30c0aef00 con 0x7fa30c0b5360 2024-01-15T18:23:21.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.401+0000 7fa313a00640 1 -- 172.21.15.27:0/314901032 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa30c074820 con 0x7fa30c0b5360 2024-01-15T18:23:21.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.401+0000 7fa308c00640 1 -- 172.21.15.27:0/314901032 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7fa2f4003980 con 0x7fa30c0b5360 2024-01-15T18:23:21.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.405+0000 7fa308c00640 1 --2- 172.21.15.27:0/314901032 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa2e4067a60 0x7fa2e4069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:21.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.405+0000 7fa308c00640 1 -- 172.21.15.27:0/314901032 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fa2f40828b0 con 0x7fa30c0b5360 2024-01-15T18:23:21.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.405+0000 7fa311000640 1 --2- 172.21.15.27:0/314901032 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa2e4067a60 0x7fa2e4069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:21.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.405+0000 7fa311000640 1 --2- 172.21.15.27:0/314901032 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa2e4067a60 0x7fa2e4069f20 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc0ae000 tx=0x7fa2fc0b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:21.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.409+0000 7fa308c00640 1 -- 172.21.15.27:0/314901032 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa2f4051210 con 0x7fa30c0b5360 2024-01-15T18:23:21.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.573+0000 7fa313a00640 1 -- 172.21.15.27:0/314901032 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa30c141f50 con 0x7fa30c0b5360 2024-01-15T18:23:21.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.573+0000 7fa308c00640 1 -- 172.21.15.27:0/314901032 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v110) v1 ==== 80+0+0 (secure 0 0 0) 0x7fa2f4017070 con 0x7fa30c0b5360 2024-01-15T18:23:21.571 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-01-15T18:23:21.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.577+0000 7fa313a00640 1 -- 172.21.15.27:0/314901032 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa2e4067a60 msgr2=0x7fa2e4069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:21.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.577+0000 7fa313a00640 1 --2- 172.21.15.27:0/314901032 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa2e4067a60 0x7fa2e4069f20 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc0ae000 tx=0x7fa2fc0b5040 comp rx=0 tx=0).stop 2024-01-15T18:23:21.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.577+0000 7fa313a00640 1 -- 172.21.15.27:0/314901032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa30c0b5360 msgr2=0x7fa30c140c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:21.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.577+0000 7fa313a00640 1 --2- 172.21.15.27:0/314901032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa30c0b5360 0x7fa30c140c40 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fa2f400c390 tx=0x7fa2f400c850 comp rx=0 tx=0).stop 2024-01-15T18:23:21.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.577+0000 7fa313a00640 1 -- 172.21.15.27:0/314901032 shutdown_connections 2024-01-15T18:23:21.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.577+0000 7fa313a00640 1 --2- 172.21.15.27:0/314901032 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fa2e4067a60 0x7fa2e4069f20 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.577+0000 7fa313a00640 1 --2- 172.21.15.27:0/314901032 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fa30c159b60 0x7fa30c141180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.577+0000 7fa313a00640 1 --2- 172.21.15.27:0/314901032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa30c0b5360 0x7fa30c140c40 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.577+0000 7fa313a00640 1 --2- 172.21.15.27:0/314901032 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fa30c06a4f0 0x7fa30c0ae9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.577+0000 7fa313a00640 1 -- 172.21.15.27:0/314901032 >> 172.21.15.27:0/314901032 conn(0x7fa30c0b2360 msgr2=0x7fa30c0b01d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:21.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.577+0000 7fa313a00640 1 -- 172.21.15.27:0/314901032 shutdown_connections 2024-01-15T18:23:21.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.577+0000 7fa313a00640 1 -- 172.21.15.27:0/314901032 wait complete. 2024-01-15T18:23:21.592 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:23:21.592 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:23:21.601 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:23:21.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.785+0000 7f8a28600640 1 Processor -- start 2024-01-15T18:23:21.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.785+0000 7f8a28600640 1 -- start start 2024-01-15T18:23:21.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.789+0000 7f8a28600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8a20160b10 0x7f8a200afe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:21.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.789+0000 7f8a28600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a200d05b0 con 0x7f8a200cd7a0 2024-01-15T18:23:21.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.789+0000 7f8a28600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a200ce7d0 con 0x7f8a200ce330 2024-01-15T18:23:21.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.789+0000 7f8a28600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a200cfc80 con 0x7f8a20160b10 2024-01-15T18:23:21.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.789+0000 7f8a25200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8a200cd7a0 0x7f8a200cdc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50230/0 (socket says 172.21.15.27:50230) 2024-01-15T18:23:21.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.789+0000 7f8a25200640 1 -- 172.21.15.27:0/281653265 learned_addr learned my addr 172.21.15.27:0/281653265 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:21.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.789+0000 7f8a25c00640 1 --2- 172.21.15.27:0/281653265 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8a20160b10 0x7f8a200afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:21.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.789+0000 7f8a1fe00640 1 -- 172.21.15.27:0/281653265 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691669938 0 0) 0x7f8a200d05b0 con 0x7f8a200cd7a0 2024-01-15T18:23:21.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.789+0000 7f8a1fe00640 1 -- 172.21.15.27:0/281653265 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a00003530 con 0x7f8a200cd7a0 2024-01-15T18:23:21.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.789+0000 7f8a1fe00640 1 -- 172.21.15.27:0/281653265 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042734873 0 0) 0x7f8a200ce7d0 con 0x7f8a200ce330 2024-01-15T18:23:21.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.789+0000 7f8a1fe00640 1 -- 172.21.15.27:0/281653265 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a200d05b0 con 0x7f8a200ce330 2024-01-15T18:23:21.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.789+0000 7f8a25c00640 1 -- 172.21.15.27:0/281653265 >> v1:172.21.15.93:6789/0 conn(0x7f8a200ce330 legacy=0x7f8a200cb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:21.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.789+0000 7f8a25c00640 1 -- 172.21.15.27:0/281653265 >> v1:172.21.15.27:6789/0 conn(0x7f8a200cd7a0 legacy=0x7f8a200cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:21.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.789+0000 7f8a25c00640 1 -- 172.21.15.27:0/281653265 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a200cff60 con 0x7f8a20160b10 2024-01-15T18:23:21.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.793+0000 7f8a25c00640 1 --2- 172.21.15.27:0/281653265 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8a20160b10 0x7f8a200afe70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8a100097f0 tx=0x7f8a1001c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18f6c50afe29f170 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:21.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.793+0000 7f8a1fe00640 1 -- 172.21.15.27:0/281653265 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a100ab620 con 0x7f8a20160b10 2024-01-15T18:23:21.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.793+0000 7f8a1fe00640 1 -- 172.21.15.27:0/281653265 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a100ab780 con 0x7f8a20160b10 2024-01-15T18:23:21.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.793+0000 7f8a1fe00640 1 -- 172.21.15.27:0/281653265 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a100abb10 con 0x7f8a20160b10 2024-01-15T18:23:21.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.793+0000 7f8a28600640 1 -- 172.21.15.27:0/281653265 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8a20160b10 msgr2=0x7f8a200afe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:21.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.793+0000 7f8a28600640 1 --2- 172.21.15.27:0/281653265 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8a20160b10 0x7f8a200afe70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8a100097f0 tx=0x7f8a1001c400 comp rx=0 tx=0).stop 2024-01-15T18:23:21.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.793+0000 7f8a28600640 1 -- 172.21.15.27:0/281653265 shutdown_connections 2024-01-15T18:23:21.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.793+0000 7f8a28600640 1 --2- 172.21.15.27:0/281653265 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8a20160b10 0x7f8a200afe70 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.793+0000 7f8a28600640 1 -- 172.21.15.27:0/281653265 >> 172.21.15.27:0/281653265 conn(0x7f8a200d0f20 msgr2=0x7f8a200d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:21.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.793+0000 7f8a28600640 1 -- 172.21.15.27:0/281653265 shutdown_connections 2024-01-15T18:23:21.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.793+0000 7f8a28600640 1 -- 172.21.15.27:0/281653265 wait complete. 2024-01-15T18:23:21.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.793+0000 7f8a28600640 1 Processor -- start 2024-01-15T18:23:21.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.797+0000 7f8a28600640 1 -- start start 2024-01-15T18:23:21.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.797+0000 7f8a28600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8a200cd7a0 0x7f8a2014e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:21.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.797+0000 7f8a28600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8a200ce330 0x7f8a200c9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:21.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.797+0000 7f8a28600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a20160b10 0x7f8a200adeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:21.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.797+0000 7f8a28600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a200c8f70 con 0x7f8a20160b10 2024-01-15T18:23:21.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.797+0000 7f8a28600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a200c8360 con 0x7f8a200ce330 2024-01-15T18:23:21.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.797+0000 7f8a28600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a200c8640 con 0x7f8a200cd7a0 2024-01-15T18:23:21.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.797+0000 7f8a26600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a20160b10 0x7f8a200adeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:21.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.797+0000 7f8a26600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a20160b10 0x7f8a200adeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35206/0 (socket says 172.21.15.27:35206) 2024-01-15T18:23:21.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.797+0000 7f8a26600640 1 -- 172.21.15.27:0/1103755366 learned_addr learned my addr 172.21.15.27:0/1103755366 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:21.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.797+0000 7f8a25c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8a200cd7a0 0x7f8a2014e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:21.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.797+0000 7f8a25200640 1 --2- 172.21.15.27:0/1103755366 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8a200ce330 0x7f8a200c9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:21.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.797+0000 7f8a26600640 1 -- 172.21.15.27:0/1103755366 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8a200cd7a0 msgr2=0x7f8a2014e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:21.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.797+0000 7f8a26600640 1 --2- 172.21.15.27:0/1103755366 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8a200cd7a0 0x7f8a2014e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.797+0000 7f8a26600640 1 -- 172.21.15.27:0/1103755366 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8a200ce330 msgr2=0x7f8a200c9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:21.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.797+0000 7f8a26600640 1 --2- 172.21.15.27:0/1103755366 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8a200ce330 0x7f8a200c9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.797+0000 7f8a26600640 1 -- 172.21.15.27:0/1103755366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a10009420 con 0x7f8a20160b10 2024-01-15T18:23:21.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.797+0000 7f8a26600640 1 --2- 172.21.15.27:0/1103755366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a20160b10 0x7f8a200adeb0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f8a1800c730 tx=0x7f8a1800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:21.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.797+0000 7f8a1d600640 1 -- 172.21.15.27:0/1103755366 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a18003b80 con 0x7f8a20160b10 2024-01-15T18:23:21.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.797+0000 7f8a1d600640 1 -- 172.21.15.27:0/1103755366 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a18003ce0 con 0x7f8a20160b10 2024-01-15T18:23:21.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.797+0000 7f8a1d600640 1 -- 172.21.15.27:0/1103755366 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1800aa90 con 0x7f8a20160b10 2024-01-15T18:23:21.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.797+0000 7f8a28600640 1 -- 172.21.15.27:0/1103755366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a200c8920 con 0x7f8a20160b10 2024-01-15T18:23:21.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.797+0000 7f8a28600640 1 -- 172.21.15.27:0/1103755366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a200c7ff0 con 0x7f8a20160b10 2024-01-15T18:23:21.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.797+0000 7f8a28600640 1 -- 172.21.15.27:0/1103755366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a200c90e0 con 0x7f8a20160b10 2024-01-15T18:23:21.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.801+0000 7f8a1d600640 1 -- 172.21.15.27:0/1103755366 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f8a1800abf0 con 0x7f8a20160b10 2024-01-15T18:23:21.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.801+0000 7f8a1d600640 1 --2- 172.21.15.27:0/1103755366 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f89f8067a60 0x7f89f8069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:21.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.801+0000 7f8a1d600640 1 -- 172.21.15.27:0/1103755366 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f8a180830a0 con 0x7f8a20160b10 2024-01-15T18:23:21.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.801+0000 7f8a25c00640 1 --2- 172.21.15.27:0/1103755366 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f89f8067a60 0x7f89f8069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:21.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.801+0000 7f8a25c00640 1 --2- 172.21.15.27:0/1103755366 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f89f8067a60 0x7f89f8069f20 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f8a100af000 tx=0x7f8a100a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:21.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.805+0000 7f8a1d600640 1 -- 172.21.15.27:0/1103755366 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8a18051a90 con 0x7f8a20160b10 2024-01-15T18:23:21.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.965+0000 7f8a28600640 1 -- 172.21.15.27:0/1103755366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a2004e850 con 0x7f8a20160b10 2024-01-15T18:23:21.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.965+0000 7f8a1d600640 1 -- 172.21.15.27:0/1103755366 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16761 (secure 0 0 0) 0x7f8a180550b0 con 0x7f8a20160b10 2024-01-15T18:23:21.961 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:23:21.962 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":110,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:18:42.250892+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:18:30.993882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-15T18:18:31.582649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.5","mappings":[{"from":0,"to":5}]},{"pgid":"17.18","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:23:21.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.965+0000 7f8a28600640 1 -- 172.21.15.27:0/1103755366 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f89f8067a60 msgr2=0x7f89f8069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:21.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.965+0000 7f8a28600640 1 --2- 172.21.15.27:0/1103755366 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f89f8067a60 0x7f89f8069f20 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f8a100af000 tx=0x7f8a100a7000 comp rx=0 tx=0).stop 2024-01-15T18:23:21.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.965+0000 7f8a28600640 1 -- 172.21.15.27:0/1103755366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a20160b10 msgr2=0x7f8a200adeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:21.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.965+0000 7f8a28600640 1 --2- 172.21.15.27:0/1103755366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a20160b10 0x7f8a200adeb0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f8a1800c730 tx=0x7f8a1800cbf0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.965+0000 7f8a28600640 1 -- 172.21.15.27:0/1103755366 shutdown_connections 2024-01-15T18:23:21.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.965+0000 7f8a28600640 1 --2- 172.21.15.27:0/1103755366 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f89f8067a60 0x7f89f8069f20 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.965+0000 7f8a28600640 1 --2- 172.21.15.27:0/1103755366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8a20160b10 0x7f8a200adeb0 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.965+0000 7f8a28600640 1 --2- 172.21.15.27:0/1103755366 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8a200ce330 0x7f8a200c9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.965+0000 7f8a28600640 1 --2- 172.21.15.27:0/1103755366 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8a200cd7a0 0x7f8a2014e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:21.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.965+0000 7f8a28600640 1 -- 172.21.15.27:0/1103755366 >> 172.21.15.27:0/1103755366 conn(0x7f8a200d0f20 msgr2=0x7f8a20156250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:21.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.969+0000 7f8a28600640 1 -- 172.21.15.27:0/1103755366 shutdown_connections 2024-01-15T18:23:21.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:21.969+0000 7f8a28600640 1 -- 172.21.15.27:0/1103755366 wait complete. 2024-01-15T18:23:21.984 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T18:18:30.993882+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T18:18:31.582649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:23:21.984 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:23:22.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.129+0000 7f8568000640 1 Processor -- start 2024-01-15T18:23:22.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.129+0000 7f8568000640 1 -- start start 2024-01-15T18:23:22.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.133+0000 7f8568000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8560071550 0x7f856006a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:22.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.133+0000 7f8568000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f856006bc50 0x7f856015d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:22.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.133+0000 7f8568000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8560161020 con 0x7f856006b050 2024-01-15T18:23:22.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.133+0000 7f8568000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8560161300 con 0x7f856006bc50 2024-01-15T18:23:22.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.133+0000 7f8568000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85601615e0 con 0x7f8560071550 2024-01-15T18:23:22.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.133+0000 7f8566000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f856006bc50 0x7f856015d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:22.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.133+0000 7f8566000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f856006bc50 0x7f856015d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35210/0 (socket says 172.21.15.27:35210) 2024-01-15T18:23:22.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.133+0000 7f8566000640 1 -- 172.21.15.27:0/1685635766 learned_addr learned my addr 172.21.15.27:0/1685635766 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:22.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.133+0000 7f8564c00640 1 --2- 172.21.15.27:0/1685635766 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8560071550 0x7f856006a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:22.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.133+0000 7f8566000640 1 -- 172.21.15.27:0/1685635766 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8560071550 msgr2=0x7f856006a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:22.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.133+0000 7f8566000640 1 --2- 172.21.15.27:0/1685635766 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8560071550 0x7f856006a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.133+0000 7f8566000640 1 -- 172.21.15.27:0/1685635766 >> v1:172.21.15.93:6789/0 conn(0x7f856006b050 legacy=0x7f8560070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:22.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.133+0000 7f8566000640 1 -- 172.21.15.27:0/1685635766 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85601618c0 con 0x7f856006bc50 2024-01-15T18:23:22.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.133+0000 7f8564c00640 1 --2- 172.21.15.27:0/1685635766 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8560071550 0x7f856006a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:23:22.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.137+0000 7f8566000640 1 --2- 172.21.15.27:0/1685635766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856006bc50 0x7f856015d9f0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f8558009e50 tx=0x7f855801c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce13566776d31fe server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:22.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.137+0000 7f855fe00640 1 -- 172.21.15.27:0/1685635766 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85580a5820 con 0x7f856006bc50 2024-01-15T18:23:22.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.137+0000 7f855fe00640 1 -- 172.21.15.27:0/1685635766 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85580a5980 con 0x7f856006bc50 2024-01-15T18:23:22.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.137+0000 7f855fe00640 1 -- 172.21.15.27:0/1685635766 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85580a5d10 con 0x7f856006bc50 2024-01-15T18:23:22.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.137+0000 7f8568000640 1 -- 172.21.15.27:0/1685635766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856006bc50 msgr2=0x7f856015d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:22.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.137+0000 7f8568000640 1 --2- 172.21.15.27:0/1685635766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856006bc50 0x7f856015d9f0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f8558009e50 tx=0x7f855801c980 comp rx=0 tx=0).stop 2024-01-15T18:23:22.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.137+0000 7f8568000640 1 -- 172.21.15.27:0/1685635766 shutdown_connections 2024-01-15T18:23:22.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.137+0000 7f8568000640 1 --2- 172.21.15.27:0/1685635766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f856006bc50 0x7f856015d9f0 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.137+0000 7f8568000640 1 --2- 172.21.15.27:0/1685635766 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8560071550 0x7f856006a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.137+0000 7f8568000640 1 -- 172.21.15.27:0/1685635766 >> 172.21.15.27:0/1685635766 conn(0x7f85600b15b0 msgr2=0x7f85600b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:22.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.137+0000 7f8568000640 1 -- 172.21.15.27:0/1685635766 shutdown_connections 2024-01-15T18:23:22.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.137+0000 7f8568000640 1 -- 172.21.15.27:0/1685635766 wait complete. 2024-01-15T18:23:22.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.137+0000 7f8568000640 1 Processor -- start 2024-01-15T18:23:22.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.141+0000 7f8568000640 1 -- start start 2024-01-15T18:23:22.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.141+0000 7f8568000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f856006b050 0x7f85600ae360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:22.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.141+0000 7f8568000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f856006bc50 0x7f856014ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:22.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.141+0000 7f8568000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8560071550 0x7f8560139880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:22.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.141+0000 7f8568000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856014bdf0 con 0x7f8560071550 2024-01-15T18:23:22.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.141+0000 7f8568000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856013a2d0 con 0x7f856006b050 2024-01-15T18:23:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.141+0000 7f8568000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856013a5b0 con 0x7f856006bc50 2024-01-15T18:23:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.141+0000 7f8566000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8560071550 0x7f8560139880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.141+0000 7f8564c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f856006bc50 0x7f856014ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.141+0000 7f8566000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8560071550 0x7f8560139880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35212/0 (socket says 172.21.15.27:35212) 2024-01-15T18:23:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.141+0000 7f8566000640 1 -- 172.21.15.27:0/873196153 learned_addr learned my addr 172.21.15.27:0/873196153 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.141+0000 7f8565600640 1 --2- 172.21.15.27:0/873196153 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f856006b050 0x7f85600ae360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.141+0000 7f8566000640 1 -- 172.21.15.27:0/873196153 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f856006bc50 msgr2=0x7f856014ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.141+0000 7f8566000640 1 --2- 172.21.15.27:0/873196153 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f856006bc50 0x7f856014ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.141+0000 7f8566000640 1 -- 172.21.15.27:0/873196153 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f856006b050 msgr2=0x7f85600ae360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:22.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.141+0000 7f8566000640 1 --2- 172.21.15.27:0/873196153 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f856006b050 0x7f85600ae360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.141+0000 7f8566000640 1 -- 172.21.15.27:0/873196153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8558009b30 con 0x7f8560071550 2024-01-15T18:23:22.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.141+0000 7f8566000640 1 --2- 172.21.15.27:0/873196153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8560071550 0x7f8560139880 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f855801b030 tx=0x7f8558019720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:22.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.141+0000 7f855d600640 1 -- 172.21.15.27:0/873196153 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85580b8670 con 0x7f8560071550 2024-01-15T18:23:22.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.141+0000 7f855d600640 1 -- 172.21.15.27:0/873196153 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85580b87d0 con 0x7f8560071550 2024-01-15T18:23:22.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.141+0000 7f855d600640 1 -- 172.21.15.27:0/873196153 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85580b8b80 con 0x7f8560071550 2024-01-15T18:23:22.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.141+0000 7f8568000640 1 -- 172.21.15.27:0/873196153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f856014ba80 con 0x7f8560071550 2024-01-15T18:23:22.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.141+0000 7f8568000640 1 -- 172.21.15.27:0/873196153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85600ae8a0 con 0x7f8560071550 2024-01-15T18:23:22.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.141+0000 7f8568000640 1 -- 172.21.15.27:0/873196153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f856014bf60 con 0x7f8560071550 2024-01-15T18:23:22.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.145+0000 7f855d600640 1 -- 172.21.15.27:0/873196153 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f85580b8ce0 con 0x7f8560071550 2024-01-15T18:23:22.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.145+0000 7f855d600640 1 --2- 172.21.15.27:0/873196153 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8538067a60 0x7f8538069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:22.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.145+0000 7f855d600640 1 -- 172.21.15.27:0/873196153 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f855811ea10 con 0x7f8560071550 2024-01-15T18:23:22.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.145+0000 7f8565600640 1 --2- 172.21.15.27:0/873196153 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8538067a60 0x7f8538069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:22.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.145+0000 7f8565600640 1 --2- 172.21.15.27:0/873196153 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8538067a60 0x7f8538069f20 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f8550007fd0 tx=0x7f8550009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:22.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.149+0000 7f855d600640 1 -- 172.21.15.27:0/873196153 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f85580ed560 con 0x7f8560071550 2024-01-15T18:23:22.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.309+0000 7f8568000640 1 -- 172.21.15.27:0/873196153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f856004e850 con 0x7f8560071550 2024-01-15T18:23:22.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.309+0000 7f855d600640 1 -- 172.21.15.27:0/873196153 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v110) v1 ==== 93+0+10 (secure 0 0 0) 0x7f85580bc0a0 con 0x7f8560071550 2024-01-15T18:23:22.307 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:23:22.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.313+0000 7f8568000640 1 -- 172.21.15.27:0/873196153 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8538067a60 msgr2=0x7f8538069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:22.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.313+0000 7f8568000640 1 --2- 172.21.15.27:0/873196153 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8538067a60 0x7f8538069f20 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f8550007fd0 tx=0x7f8550009040 comp rx=0 tx=0).stop 2024-01-15T18:23:22.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.313+0000 7f8568000640 1 -- 172.21.15.27:0/873196153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8560071550 msgr2=0x7f8560139880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:22.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.313+0000 7f8568000640 1 --2- 172.21.15.27:0/873196153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8560071550 0x7f8560139880 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f855801b030 tx=0x7f8558019720 comp rx=0 tx=0).stop 2024-01-15T18:23:22.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.313+0000 7f8568000640 1 -- 172.21.15.27:0/873196153 shutdown_connections 2024-01-15T18:23:22.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.313+0000 7f8568000640 1 --2- 172.21.15.27:0/873196153 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8538067a60 0x7f8538069f20 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.313+0000 7f8568000640 1 --2- 172.21.15.27:0/873196153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8560071550 0x7f8560139880 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.313+0000 7f8568000640 1 --2- 172.21.15.27:0/873196153 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f856006bc50 0x7f856014ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.313+0000 7f8568000640 1 --2- 172.21.15.27:0/873196153 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f856006b050 0x7f85600ae360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.313+0000 7f8568000640 1 -- 172.21.15.27:0/873196153 >> 172.21.15.27:0/873196153 conn(0x7f85600b15b0 msgr2=0x7f85600b1990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:22.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.313+0000 7f8568000640 1 -- 172.21.15.27:0/873196153 shutdown_connections 2024-01-15T18:23:22.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.313+0000 7f8568000640 1 -- 172.21.15.27:0/873196153 wait complete. 2024-01-15T18:23:22.329 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T18:23:22.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.477+0000 7f68b2400640 1 Processor -- start 2024-01-15T18:23:22.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.477+0000 7f68b2400640 1 -- start start 2024-01-15T18:23:22.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.477+0000 7f68b2400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f68ac158900 0x7f68ac15f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:22.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.477+0000 7f68b2400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68ac0b1cb0 con 0x7f68ac15bf00 2024-01-15T18:23:22.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.477+0000 7f68b2400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68ac0b1f00 con 0x7f68ac0cf2e0 2024-01-15T18:23:22.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.477+0000 7f68b2400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68ac0b21e0 con 0x7f68ac158900 2024-01-15T18:23:22.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.477+0000 7f68aaa00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f68ac15bf00 0x7f68ac0cebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50238/0 (socket says 172.21.15.27:50238) 2024-01-15T18:23:22.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.477+0000 7f68aaa00640 1 -- 172.21.15.27:0/737921790 learned_addr learned my addr 172.21.15.27:0/737921790 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:22.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.477+0000 7f68ab400640 1 --2- 172.21.15.27:0/737921790 >> v2:172.21.15.93:3300/0 conn(0x7f68ac158900 0x7f68ac15f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:22.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.477+0000 7f68aa000640 1 -- 172.21.15.27:0/737921790 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118604454 0 0) 0x7f68ac0b1cb0 con 0x7f68ac15bf00 2024-01-15T18:23:22.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.477+0000 7f68aa000640 1 -- 172.21.15.27:0/737921790 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f688c003530 con 0x7f68ac15bf00 2024-01-15T18:23:22.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.477+0000 7f68ab400640 1 -- 172.21.15.27:0/737921790 >> v1:172.21.15.93:6789/0 conn(0x7f68ac0cf2e0 legacy=0x7f68ac0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:22.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.477+0000 7f68ab400640 1 -- 172.21.15.27:0/737921790 >> v1:172.21.15.27:6789/0 conn(0x7f68ac15bf00 legacy=0x7f68ac0cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:22.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.477+0000 7f68ab400640 1 -- 172.21.15.27:0/737921790 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ac15ea70 con 0x7f68ac158900 2024-01-15T18:23:22.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.481+0000 7f68ab400640 1 --2- 172.21.15.27:0/737921790 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f68ac158900 0x7f68ac15f550 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f689c0097c0 tx=0x7f689c01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2143396c03d182a3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:22.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.481+0000 7f68aa000640 1 -- 172.21.15.27:0/737921790 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c0ab600 con 0x7f68ac158900 2024-01-15T18:23:22.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.481+0000 7f68aa000640 1 -- 172.21.15.27:0/737921790 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689c0a7050 con 0x7f68ac158900 2024-01-15T18:23:22.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.481+0000 7f68aa000640 1 -- 172.21.15.27:0/737921790 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c0ab980 con 0x7f68ac158900 2024-01-15T18:23:22.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.481+0000 7f68b2400640 1 -- 172.21.15.27:0/737921790 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f68ac158900 msgr2=0x7f68ac15f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:22.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.481+0000 7f68b2400640 1 --2- 172.21.15.27:0/737921790 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f68ac158900 0x7f68ac15f550 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f689c0097c0 tx=0x7f689c01c480 comp rx=0 tx=0).stop 2024-01-15T18:23:22.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.481+0000 7f68b2400640 1 -- 172.21.15.27:0/737921790 shutdown_connections 2024-01-15T18:23:22.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.481+0000 7f68b2400640 1 --2- 172.21.15.27:0/737921790 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f68ac158900 0x7f68ac15f550 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.481+0000 7f68b2400640 1 -- 172.21.15.27:0/737921790 >> 172.21.15.27:0/737921790 conn(0x7f68ac0cb700 msgr2=0x7f68ac0cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:22.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.481+0000 7f68b2400640 1 -- 172.21.15.27:0/737921790 shutdown_connections 2024-01-15T18:23:22.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.481+0000 7f68b2400640 1 -- 172.21.15.27:0/737921790 wait complete. 2024-01-15T18:23:22.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.485+0000 7f68b2400640 1 Processor -- start 2024-01-15T18:23:22.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.485+0000 7f68b2400640 1 -- start start 2024-01-15T18:23:22.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.485+0000 7f68b2400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68ac0cf2e0 0x7f68ac152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:22.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.485+0000 7f68b2400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f68ac158900 0x7f68ac14f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:22.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.485+0000 7f68b2400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f68ac15bf00 0x7f68ac14e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:22.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.485+0000 7f68b2400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac0ae810 con 0x7f68ac0cf2e0 2024-01-15T18:23:22.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.485+0000 7f68b2400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac152270 con 0x7f68ac15bf00 2024-01-15T18:23:22.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.485+0000 7f68b2400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac0aeab0 con 0x7f68ac158900 2024-01-15T18:23:22.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.485+0000 7f68ab400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68ac0cf2e0 0x7f68ac152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:22.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.485+0000 7f68ab400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68ac0cf2e0 0x7f68ac152a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35228/0 (socket says 172.21.15.27:35228) 2024-01-15T18:23:22.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.485+0000 7f68ab400640 1 -- 172.21.15.27:0/712175214 learned_addr learned my addr 172.21.15.27:0/712175214 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:22.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.485+0000 7f68aaa00640 1 --2- 172.21.15.27:0/712175214 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f68ac158900 0x7f68ac14f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:22.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.485+0000 7f68abe00640 1 --2- 172.21.15.27:0/712175214 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f68ac15bf00 0x7f68ac14e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:22.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.485+0000 7f68ab400640 1 -- 172.21.15.27:0/712175214 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f68ac158900 msgr2=0x7f68ac14f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:22.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.485+0000 7f68ab400640 1 --2- 172.21.15.27:0/712175214 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f68ac158900 0x7f68ac14f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.485+0000 7f68ab400640 1 -- 172.21.15.27:0/712175214 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f68ac15bf00 msgr2=0x7f68ac14e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:22.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.485+0000 7f68ab400640 1 --2- 172.21.15.27:0/712175214 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f68ac15bf00 0x7f68ac14e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.485+0000 7f68ab400640 1 -- 172.21.15.27:0/712175214 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f689c009420 con 0x7f68ac0cf2e0 2024-01-15T18:23:22.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.485+0000 7f68ab400640 1 --2- 172.21.15.27:0/712175214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68ac0cf2e0 0x7f68ac152a80 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f689c0ae040 tx=0x7f689c01acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:22.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.485+0000 7f68a3400640 1 -- 172.21.15.27:0/712175214 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c0b7450 con 0x7f68ac0cf2e0 2024-01-15T18:23:22.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.485+0000 7f68a3400640 1 -- 172.21.15.27:0/712175214 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689c0b75b0 con 0x7f68ac0cf2e0 2024-01-15T18:23:22.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.485+0000 7f68a3400640 1 -- 172.21.15.27:0/712175214 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c0b7960 con 0x7f68ac0cf2e0 2024-01-15T18:23:22.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.485+0000 7f68b2400640 1 -- 172.21.15.27:0/712175214 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68ac0aed90 con 0x7f68ac0cf2e0 2024-01-15T18:23:22.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.485+0000 7f68b2400640 1 -- 172.21.15.27:0/712175214 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68ac1575a0 con 0x7f68ac0cf2e0 2024-01-15T18:23:22.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.489+0000 7f68b2400640 1 -- 172.21.15.27:0/712175214 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68ac04e850 con 0x7f68ac0cf2e0 2024-01-15T18:23:22.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.489+0000 7f68a3400640 1 -- 172.21.15.27:0/712175214 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f689c0b7ac0 con 0x7f68ac0cf2e0 2024-01-15T18:23:22.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.489+0000 7f68a3400640 1 --2- 172.21.15.27:0/712175214 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6884067a60 0x7f6884069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:22.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.489+0000 7f68a3400640 1 -- 172.21.15.27:0/712175214 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f689c11dae0 con 0x7f68ac0cf2e0 2024-01-15T18:23:22.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.493+0000 7f68aaa00640 1 --2- 172.21.15.27:0/712175214 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6884067a60 0x7f6884069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:22.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.493+0000 7f68aaa00640 1 --2- 172.21.15.27:0/712175214 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6884067a60 0x7f6884069f20 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f68ac072360 tx=0x7f6894009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:22.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.493+0000 7f68a3400640 1 -- 172.21.15.27:0/712175214 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f689c0ec480 con 0x7f68ac0cf2e0 2024-01-15T18:23:22.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.649+0000 7f68b2400640 1 -- 172.21.15.27:0/712175214 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f68ac0b0720 con 0x7f68ac0cf2e0 2024-01-15T18:23:22.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.649+0000 7f68a3400640 1 -- 172.21.15.27:0/712175214 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v110) v1 ==== 104+0+11 (secure 0 0 0) 0x7f689c0efaa0 con 0x7f68ac0cf2e0 2024-01-15T18:23:22.647 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-15T18:23:22.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.653+0000 7f68b2400640 1 -- 172.21.15.27:0/712175214 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6884067a60 msgr2=0x7f6884069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:22.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.653+0000 7f68b2400640 1 --2- 172.21.15.27:0/712175214 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6884067a60 0x7f6884069f20 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f68ac072360 tx=0x7f6894009040 comp rx=0 tx=0).stop 2024-01-15T18:23:22.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.653+0000 7f68b2400640 1 -- 172.21.15.27:0/712175214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68ac0cf2e0 msgr2=0x7f68ac152a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:22.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.653+0000 7f68b2400640 1 --2- 172.21.15.27:0/712175214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68ac0cf2e0 0x7f68ac152a80 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f689c0ae040 tx=0x7f689c01acc0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.653+0000 7f68b2400640 1 -- 172.21.15.27:0/712175214 shutdown_connections 2024-01-15T18:23:22.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.653+0000 7f68b2400640 1 --2- 172.21.15.27:0/712175214 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6884067a60 0x7f6884069f20 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.653+0000 7f68b2400640 1 --2- 172.21.15.27:0/712175214 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f68ac15bf00 0x7f68ac14e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.653+0000 7f68b2400640 1 --2- 172.21.15.27:0/712175214 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f68ac158900 0x7f68ac14f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.653+0000 7f68b2400640 1 --2- 172.21.15.27:0/712175214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68ac0cf2e0 0x7f68ac152a80 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.653+0000 7f68b2400640 1 -- 172.21.15.27:0/712175214 >> 172.21.15.27:0/712175214 conn(0x7f68ac0cb700 msgr2=0x7f68ac0cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:22.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.653+0000 7f68b2400640 1 -- 172.21.15.27:0/712175214 shutdown_connections 2024-01-15T18:23:22.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.653+0000 7f68b2400640 1 -- 172.21.15.27:0/712175214 wait complete. 2024-01-15T18:23:22.671 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T18:23:22.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.817+0000 7fac1c600640 1 Processor -- start 2024-01-15T18:23:22.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.817+0000 7fac1c600640 1 -- start start 2024-01-15T18:23:22.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.821+0000 7fac1c600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fac1415b100 0x7fac14070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:22.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.821+0000 7fac1c600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fac14071420 0x7fac1415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:22.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.821+0000 7fac1c600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac14071860 con 0x7fac1415ad20 2024-01-15T18:23:22.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.821+0000 7fac1c600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac1415a610 con 0x7fac14071420 2024-01-15T18:23:22.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.821+0000 7fac1c600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac1415e3c0 con 0x7fac1415b100 2024-01-15T18:23:22.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.821+0000 7fac1a600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fac14071420 0x7fac1415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:22.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.821+0000 7fac1a600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fac14071420 0x7fac1415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35244/0 (socket says 172.21.15.27:35244) 2024-01-15T18:23:22.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.821+0000 7fac1a600640 1 -- 172.21.15.27:0/1476726555 learned_addr learned my addr 172.21.15.27:0/1476726555 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:22.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.821+0000 7fac19200640 1 --2- 172.21.15.27:0/1476726555 >> v2:172.21.15.93:3300/0 conn(0x7fac1415b100 0x7fac14070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:22.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.821+0000 7fac1a600640 1 -- 172.21.15.27:0/1476726555 >> v2:172.21.15.93:3300/0 conn(0x7fac1415b100 msgr2=0x7fac14070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:22.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.821+0000 7fac1a600640 1 --2- 172.21.15.27:0/1476726555 >> v2:172.21.15.93:3300/0 conn(0x7fac1415b100 0x7fac14070ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.821+0000 7fac1a600640 1 -- 172.21.15.27:0/1476726555 >> v1:172.21.15.93:6789/0 conn(0x7fac1415ad20 legacy=0x7fac1415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:22.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.821+0000 7fac1a600640 1 -- 172.21.15.27:0/1476726555 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac1415e6a0 con 0x7fac14071420 2024-01-15T18:23:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.825+0000 7fac1a600640 1 --2- 172.21.15.27:0/1476726555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac14071420 0x7fac1415bee0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fac0c009e50 tx=0x7fac0c01c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9e294732d78cfc0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.825+0000 7fac13e00640 1 -- 172.21.15.27:0/1476726555 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c0a5790 con 0x7fac14071420 2024-01-15T18:23:22.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.825+0000 7fac13e00640 1 -- 172.21.15.27:0/1476726555 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac0c0a58f0 con 0x7fac14071420 2024-01-15T18:23:22.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.825+0000 7fac13e00640 1 -- 172.21.15.27:0/1476726555 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c0a5c80 con 0x7fac14071420 2024-01-15T18:23:22.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.825+0000 7fac1c600640 1 -- 172.21.15.27:0/1476726555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac14071420 msgr2=0x7fac1415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:22.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.825+0000 7fac1c600640 1 --2- 172.21.15.27:0/1476726555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac14071420 0x7fac1415bee0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7fac0c009e50 tx=0x7fac0c01c8d0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.825+0000 7fac1c600640 1 -- 172.21.15.27:0/1476726555 shutdown_connections 2024-01-15T18:23:22.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.825+0000 7fac1c600640 1 --2- 172.21.15.27:0/1476726555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac14071420 0x7fac1415bee0 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.825+0000 7fac1c600640 1 --2- 172.21.15.27:0/1476726555 >> v2:172.21.15.93:3300/0 conn(0x7fac1415b100 0x7fac14070ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.825+0000 7fac1c600640 1 -- 172.21.15.27:0/1476726555 >> 172.21.15.27:0/1476726555 conn(0x7fac140cb700 msgr2=0x7fac140cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:22.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.825+0000 7fac1c600640 1 -- 172.21.15.27:0/1476726555 shutdown_connections 2024-01-15T18:23:22.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.825+0000 7fac1c600640 1 -- 172.21.15.27:0/1476726555 wait complete. 2024-01-15T18:23:22.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.825+0000 7fac1c600640 1 Processor -- start 2024-01-15T18:23:22.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.825+0000 7fac1c600640 1 -- start start 2024-01-15T18:23:22.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.829+0000 7fac1c600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac14071420 0x7fac14156c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:22.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.829+0000 7fac1c600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fac1415ad20 0x7fac1415f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:22.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.829+0000 7fac1c600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac1415b100 0x7fac141402e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:22.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.829+0000 7fac1c600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac14162fb0 con 0x7fac1415b100 2024-01-15T18:23:22.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.829+0000 7fac1c600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac14163290 con 0x7fac1415ad20 2024-01-15T18:23:22.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.829+0000 7fac1c600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac14163570 con 0x7fac14071420 2024-01-15T18:23:22.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.829+0000 7fac1a600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac1415b100 0x7fac141402e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:22.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.829+0000 7fac1a600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac1415b100 0x7fac141402e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35246/0 (socket says 172.21.15.27:35246) 2024-01-15T18:23:22.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.829+0000 7fac1a600640 1 -- 172.21.15.27:0/1875882398 learned_addr learned my addr 172.21.15.27:0/1875882398 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:22.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.829+0000 7fac19200640 1 --2- 172.21.15.27:0/1875882398 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fac1415ad20 0x7fac1415f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:22.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.829+0000 7fac19c00640 1 --2- 172.21.15.27:0/1875882398 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac14071420 0x7fac14156c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:22.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.829+0000 7fac1a600640 1 -- 172.21.15.27:0/1875882398 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac14071420 msgr2=0x7fac14156c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:22.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.829+0000 7fac1a600640 1 --2- 172.21.15.27:0/1875882398 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac14071420 0x7fac14156c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.829+0000 7fac1a600640 1 -- 172.21.15.27:0/1875882398 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fac1415ad20 msgr2=0x7fac1415f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:22.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.829+0000 7fac1a600640 1 --2- 172.21.15.27:0/1875882398 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fac1415ad20 0x7fac1415f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.829+0000 7fac1a600640 1 -- 172.21.15.27:0/1875882398 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0c009b30 con 0x7fac1415b100 2024-01-15T18:23:22.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.829+0000 7fac1a600640 1 --2- 172.21.15.27:0/1875882398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac1415b100 0x7fac141402e0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7fac0c0af040 tx=0x7fac0c019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:22.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.829+0000 7fac11600640 1 -- 172.21.15.27:0/1875882398 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c0b8670 con 0x7fac1415b100 2024-01-15T18:23:22.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.829+0000 7fac11600640 1 -- 172.21.15.27:0/1875882398 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac0c0b87d0 con 0x7fac1415b100 2024-01-15T18:23:22.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.829+0000 7fac11600640 1 -- 172.21.15.27:0/1875882398 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac0c0b8b80 con 0x7fac1415b100 2024-01-15T18:23:22.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.829+0000 7fac1c600640 1 -- 172.21.15.27:0/1875882398 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac14163850 con 0x7fac1415b100 2024-01-15T18:23:22.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.829+0000 7fac1c600640 1 -- 172.21.15.27:0/1875882398 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac14157320 con 0x7fac1415b100 2024-01-15T18:23:22.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.829+0000 7fac1c600640 1 -- 172.21.15.27:0/1875882398 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac1414f910 con 0x7fac1415b100 2024-01-15T18:23:22.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.833+0000 7fac11600640 1 -- 172.21.15.27:0/1875882398 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7fac0c0b8ce0 con 0x7fac1415b100 2024-01-15T18:23:22.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.833+0000 7fac11600640 1 --2- 172.21.15.27:0/1875882398 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fabec067a60 0x7fabec069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:22.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.833+0000 7fac11600640 1 -- 172.21.15.27:0/1875882398 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fac0c11e890 con 0x7fac1415b100 2024-01-15T18:23:22.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.833+0000 7fac19c00640 1 --2- 172.21.15.27:0/1875882398 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fabec067a60 0x7fabec069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:22.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.833+0000 7fac19c00640 1 --2- 172.21.15.27:0/1875882398 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fabec067a60 0x7fabec069f20 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fac04007fd0 tx=0x7fac04007e40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:22.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.837+0000 7fac11600640 1 -- 172.21.15.27:0/1875882398 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fac1414f910 con 0x7fac1415b100 2024-01-15T18:23:22.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.993+0000 7fac1c600640 1 -- 172.21.15.27:0/1875882398 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fac1414f910 con 0x7fac1415b100 2024-01-15T18:23:22.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.993+0000 7fac11600640 1 -- 172.21.15.27:0/1875882398 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v110) v1 ==== 100+0+11 (secure 0 0 0) 0x7fac1414f910 con 0x7fac1415b100 2024-01-15T18:23:22.990 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-15T18:23:22.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.993+0000 7fac1c600640 1 -- 172.21.15.27:0/1875882398 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fabec067a60 msgr2=0x7fabec069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:22.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.993+0000 7fac1c600640 1 --2- 172.21.15.27:0/1875882398 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fabec067a60 0x7fabec069f20 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fac04007fd0 tx=0x7fac04007e40 comp rx=0 tx=0).stop 2024-01-15T18:23:22.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.993+0000 7fac1c600640 1 -- 172.21.15.27:0/1875882398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac1415b100 msgr2=0x7fac141402e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:22.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.993+0000 7fac1c600640 1 --2- 172.21.15.27:0/1875882398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac1415b100 0x7fac141402e0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7fac0c0af040 tx=0x7fac0c019670 comp rx=0 tx=0).stop 2024-01-15T18:23:22.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.993+0000 7fac1c600640 1 -- 172.21.15.27:0/1875882398 shutdown_connections 2024-01-15T18:23:22.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.993+0000 7fac1c600640 1 --2- 172.21.15.27:0/1875882398 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fabec067a60 0x7fabec069f20 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.993+0000 7fac1c600640 1 --2- 172.21.15.27:0/1875882398 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac1415b100 0x7fac141402e0 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.993+0000 7fac1c600640 1 --2- 172.21.15.27:0/1875882398 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fac1415ad20 0x7fac1415f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.993+0000 7fac1c600640 1 --2- 172.21.15.27:0/1875882398 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fac14071420 0x7fac14156c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:22.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.993+0000 7fac1c600640 1 -- 172.21.15.27:0/1875882398 >> 172.21.15.27:0/1875882398 conn(0x7fac140cb700 msgr2=0x7fac1406cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:22.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.993+0000 7fac1c600640 1 -- 172.21.15.27:0/1875882398 shutdown_connections 2024-01-15T18:23:22.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:22.997+0000 7fac1c600640 1 -- 172.21.15.27:0/1875882398 wait complete. 2024-01-15T18:23:23.012 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:23:23.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.157+0000 7f98d8400640 1 Processor -- start 2024-01-15T18:23:23.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.157+0000 7f98d8400640 1 -- start start 2024-01-15T18:23:23.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98d8400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f98d00b4d20 0x7f98d00ae560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:23.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98d8400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d006f900 con 0x7f98d0159f30 2024-01-15T18:23:23.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98d8400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d00b3410 con 0x7f98d00aeaa0 2024-01-15T18:23:23.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98d8400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98d015a310 con 0x7f98d00b4d20 2024-01-15T18:23:23.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98d5a00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f98d0159f30 0x7f98d00b2d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:50248/0 (socket says 172.21.15.27:50248) 2024-01-15T18:23:23.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98d5a00640 1 -- 172.21.15.27:0/467070996 learned_addr learned my addr 172.21.15.27:0/467070996 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:23.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98d5000640 1 --2- 172.21.15.27:0/467070996 >> v2:172.21.15.93:3300/0 conn(0x7f98d00b4d20 0x7f98d00ae560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:23.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98cfe00640 1 -- 172.21.15.27:0/467070996 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956110692 0 0) 0x7f98d006f900 con 0x7f98d0159f30 2024-01-15T18:23:23.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98cfe00640 1 -- 172.21.15.27:0/467070996 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98b0003530 con 0x7f98d0159f30 2024-01-15T18:23:23.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98cfe00640 1 -- 172.21.15.27:0/467070996 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1607760253 0 0) 0x7f98b0003530 con 0x7f98d0159f30 2024-01-15T18:23:23.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98cfe00640 1 -- 172.21.15.27:0/467070996 >> v2:172.21.15.93:3300/0 conn(0x7f98d00b4d20 msgr2=0x7f98d00ae560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:23.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98cfe00640 1 --2- 172.21.15.27:0/467070996 >> v2:172.21.15.93:3300/0 conn(0x7f98d00b4d20 0x7f98d00ae560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:23.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98cfe00640 1 -- 172.21.15.27:0/467070996 >> v1:172.21.15.93:6789/0 conn(0x7f98d00aeaa0 legacy=0x7f98d006e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:23.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98cfe00640 1 -- 172.21.15.27:0/467070996 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d00adbd0 con 0x7f98d0159f30 2024-01-15T18:23:23.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98cfe00640 1 -- 172.21.15.27:0/467070996 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f98c0001680 con 0x7f98d0159f30 2024-01-15T18:23:23.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98cfe00640 1 -- 172.21.15.27:0/467070996 >> v1:172.21.15.27:6789/0 conn(0x7f98d0159f30 legacy=0x7f98d00b2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:23.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98cfe00640 1 --2- 172.21.15.27:0/467070996 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f98b00039f0 0x7f98b0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:23.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98cfe00640 1 --2- 172.21.15.27:0/467070996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98b00047d0 0x7f98b0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:23.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98cfe00640 1 --2- 172.21.15.27:0/467070996 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f98b0005190 0x7f98b000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:23.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98cfe00640 1 -- 172.21.15.27:0/467070996 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b000c090 con 0x7f98b00047d0 2024-01-15T18:23:23.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98cfe00640 1 -- 172.21.15.27:0/467070996 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b000c200 con 0x7f98b00039f0 2024-01-15T18:23:23.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98cfe00640 1 -- 172.21.15.27:0/467070996 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b000c3c0 con 0x7f98b0005190 2024-01-15T18:23:23.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98d5000640 1 --2- 172.21.15.27:0/467070996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98b00047d0 0x7f98b0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:23.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98d5a00640 1 --2- 172.21.15.27:0/467070996 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f98b00039f0 0x7f98b0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:23.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98d6400640 1 --2- 172.21.15.27:0/467070996 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f98b0005190 0x7f98b000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:23.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98d5000640 1 -- 172.21.15.27:0/467070996 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f98b0005190 msgr2=0x7f98b000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:23.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98d5000640 1 --2- 172.21.15.27:0/467070996 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f98b0005190 0x7f98b000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:23.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98d5000640 1 -- 172.21.15.27:0/467070996 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f98b00039f0 msgr2=0x7f98b0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:23.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98d5000640 1 --2- 172.21.15.27:0/467070996 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f98b00039f0 0x7f98b0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:23.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98d5000640 1 -- 172.21.15.27:0/467070996 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b0003530 con 0x7f98b00047d0 2024-01-15T18:23:23.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98d6400640 1 -- 172.21.15.27:0/467070996 reap_dead start 2024-01-15T18:23:23.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.161+0000 7f98d5a00640 1 --2- 172.21.15.27:0/467070996 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f98b00039f0 0x7f98b0003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:23:23.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.165+0000 7f98d5000640 1 --2- 172.21.15.27:0/467070996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98b00047d0 0x7f98b0004c50 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f98b8009ae0 tx=0x7f98b801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=377ef09d7fd2e240 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:23.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.165+0000 7f98cfe00640 1 -- 172.21.15.27:0/467070996 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b80a5640 con 0x7f98b00047d0 2024-01-15T18:23:23.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.165+0000 7f98cfe00640 1 -- 172.21.15.27:0/467070996 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b80a57a0 con 0x7f98b00047d0 2024-01-15T18:23:23.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.165+0000 7f98cfe00640 1 -- 172.21.15.27:0/467070996 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b80a5b50 con 0x7f98b00047d0 2024-01-15T18:23:23.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.165+0000 7f98d8400640 1 -- 172.21.15.27:0/467070996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98b00047d0 msgr2=0x7f98b0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:23.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.165+0000 7f98d8400640 1 --2- 172.21.15.27:0/467070996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98b00047d0 0x7f98b0004c50 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f98b8009ae0 tx=0x7f98b801ca50 comp rx=0 tx=0).stop 2024-01-15T18:23:23.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.165+0000 7f98d8400640 1 -- 172.21.15.27:0/467070996 shutdown_connections 2024-01-15T18:23:23.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.165+0000 7f98d8400640 1 --2- 172.21.15.27:0/467070996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98b00047d0 0x7f98b0004c50 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:23.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.165+0000 7f98d8400640 1 -- 172.21.15.27:0/467070996 >> 172.21.15.27:0/467070996 conn(0x7f98d00cfdb0 msgr2=0x7f98d00d01c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:23.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.165+0000 7f98d8400640 1 -- 172.21.15.27:0/467070996 shutdown_connections 2024-01-15T18:23:23.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.165+0000 7f98d8400640 1 -- 172.21.15.27:0/467070996 wait complete. 2024-01-15T18:23:23.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.169+0000 7f98d8400640 1 Processor -- start 2024-01-15T18:23:23.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.169+0000 7f98d8400640 1 -- start start 2024-01-15T18:23:23.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.169+0000 7f98d8400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f98b00047d0 0x7f98d015b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:23.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.169+0000 7f98d8400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f98d015c9e0 0x7f98d0142390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:23.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.169+0000 7f98d8400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98d01e9480 0x7f98d01e6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:23.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.169+0000 7f98d8400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d01428d0 con 0x7f98d01e9480 2024-01-15T18:23:23.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.169+0000 7f98d8400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d015b940 con 0x7f98b00047d0 2024-01-15T18:23:23.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.169+0000 7f98d8400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d015bc20 con 0x7f98d015c9e0 2024-01-15T18:23:23.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.169+0000 7f98d6400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98d01e9480 0x7f98d01e6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:23.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.169+0000 7f98d6400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98d01e9480 0x7f98d01e6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35266/0 (socket says 172.21.15.27:35266) 2024-01-15T18:23:23.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.169+0000 7f98d6400640 1 -- 172.21.15.27:0/1427925777 learned_addr learned my addr 172.21.15.27:0/1427925777 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:23.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.169+0000 7f98d5000640 1 --2- 172.21.15.27:0/1427925777 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f98d015c9e0 0x7f98d0142390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:23.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.169+0000 7f98d5a00640 1 --2- 172.21.15.27:0/1427925777 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f98b00047d0 0x7f98d015b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:23.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.169+0000 7f98d6400640 1 -- 172.21.15.27:0/1427925777 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f98d015c9e0 msgr2=0x7f98d0142390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:23.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.169+0000 7f98d6400640 1 --2- 172.21.15.27:0/1427925777 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f98d015c9e0 0x7f98d0142390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:23.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.169+0000 7f98d6400640 1 -- 172.21.15.27:0/1427925777 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f98b00047d0 msgr2=0x7f98d015b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:23.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.169+0000 7f98d6400640 1 --2- 172.21.15.27:0/1427925777 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f98b00047d0 0x7f98d015b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:23.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.169+0000 7f98d6400640 1 -- 172.21.15.27:0/1427925777 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d015bf00 con 0x7f98d01e9480 2024-01-15T18:23:23.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.169+0000 7f98d6400640 1 --2- 172.21.15.27:0/1427925777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98d01e9480 0x7f98d01e6100 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f98d0067d90 tx=0x7f98c800fc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:23.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.169+0000 7f98cd600640 1 -- 172.21.15.27:0/1427925777 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c800fed0 con 0x7f98d01e9480 2024-01-15T18:23:23.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.169+0000 7f98cd600640 1 -- 172.21.15.27:0/1427925777 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c8003b90 con 0x7f98d01e9480 2024-01-15T18:23:23.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.169+0000 7f98cd600640 1 -- 172.21.15.27:0/1427925777 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c80050d0 con 0x7f98d01e9480 2024-01-15T18:23:23.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.169+0000 7f98d8400640 1 -- 172.21.15.27:0/1427925777 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d015c1e0 con 0x7f98d01e9480 2024-01-15T18:23:23.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.169+0000 7f98d8400640 1 -- 172.21.15.27:0/1427925777 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d01e9910 con 0x7f98d01e9480 2024-01-15T18:23:23.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.173+0000 7f98d8400640 1 -- 172.21.15.27:0/1427925777 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d006fd80 con 0x7f98d01e9480 2024-01-15T18:23:23.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.173+0000 7f98cd600640 1 -- 172.21.15.27:0/1427925777 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f98c8005230 con 0x7f98d01e9480 2024-01-15T18:23:23.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.173+0000 7f98cd600640 1 --2- 172.21.15.27:0/1427925777 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f98a8067a60 0x7f98a8069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:23.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.173+0000 7f98cd600640 1 -- 172.21.15.27:0/1427925777 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f98c8083140 con 0x7f98d01e9480 2024-01-15T18:23:23.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.173+0000 7f98d5a00640 1 --2- 172.21.15.27:0/1427925777 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f98a8067a60 0x7f98a8069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:23.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.173+0000 7f98d5a00640 1 --2- 172.21.15.27:0/1427925777 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f98a8067a60 0x7f98a8069f20 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f98d0069290 tx=0x7f98c0001700 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:23.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.177+0000 7f98cd600640 1 -- 172.21.15.27:0/1427925777 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f98c8051a80 con 0x7f98d01e9480 2024-01-15T18:23:23.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.349+0000 7f98d8400640 1 -- 172.21.15.27:0/1427925777 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f98d004e850 con 0x7f98d01e9480 2024-01-15T18:23:23.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.349+0000 7f98cd600640 1 -- 172.21.15.27:0/1427925777 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v110) v1 ==== 80+0+0 (secure 0 0 0) 0x7f98c80550a0 con 0x7f98d01e9480 2024-01-15T18:23:23.345 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-01-15T18:23:23.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.349+0000 7f98d8400640 1 -- 172.21.15.27:0/1427925777 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f98a8067a60 msgr2=0x7f98a8069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:23.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.349+0000 7f98d8400640 1 --2- 172.21.15.27:0/1427925777 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f98a8067a60 0x7f98a8069f20 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f98d0069290 tx=0x7f98c0001700 comp rx=0 tx=0).stop 2024-01-15T18:23:23.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.349+0000 7f98d8400640 1 -- 172.21.15.27:0/1427925777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98d01e9480 msgr2=0x7f98d01e6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:23.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.349+0000 7f98d8400640 1 --2- 172.21.15.27:0/1427925777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98d01e9480 0x7f98d01e6100 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f98d0067d90 tx=0x7f98c800fc80 comp rx=0 tx=0).stop 2024-01-15T18:23:23.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.349+0000 7f98d8400640 1 -- 172.21.15.27:0/1427925777 shutdown_connections 2024-01-15T18:23:23.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.349+0000 7f98d8400640 1 --2- 172.21.15.27:0/1427925777 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f98a8067a60 0x7f98a8069f20 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:23.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.349+0000 7f98d8400640 1 --2- 172.21.15.27:0/1427925777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98d01e9480 0x7f98d01e6100 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:23.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.349+0000 7f98d8400640 1 --2- 172.21.15.27:0/1427925777 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f98d015c9e0 0x7f98d0142390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:23.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.349+0000 7f98d8400640 1 --2- 172.21.15.27:0/1427925777 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f98b00047d0 0x7f98d015b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:23.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.349+0000 7f98d8400640 1 -- 172.21.15.27:0/1427925777 >> 172.21.15.27:0/1427925777 conn(0x7f98d00cfdb0 msgr2=0x7f98d01563e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:23.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.349+0000 7f98d8400640 1 -- 172.21.15.27:0/1427925777 shutdown_connections 2024-01-15T18:23:23.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:23.353+0000 7f98d8400640 1 -- 172.21.15.27:0/1427925777 wait complete. 2024-01-15T18:23:23.367 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-01-15T18:23:23.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:23:23.368 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-15T18:23:23.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[52270]: fuse finished with error 0 and tester_r 0 2024-01-15T18:23:23.981 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:23:23.981 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T18:23:23.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:23:23.981 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-15T18:23:23.991 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-01-15T18:23:23.991 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:23:24.040 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354580k,nr_inodes=4088645,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9249 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275360k,nr_inodes=818840,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275364k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:23:24.040 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:23:24.040 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:23:24.050 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:23:24.050 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:23:24.099 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:23:24.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.245+0000 7f0d45200640 1 Processor -- start 2024-01-15T18:23:24.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.245+0000 7f0d45200640 1 -- start start 2024-01-15T18:23:24.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.245+0000 7f0d45200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0d400b2390 0x7f0d400cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:24.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.245+0000 7f0d45200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0d400cdf90 0x7f0d400b0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:24.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.245+0000 7f0d45200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d400b2770 con 0x7f0d4006bb80 2024-01-15T18:23:24.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.245+0000 7f0d45200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d4006c060 con 0x7f0d400b2390 2024-01-15T18:23:24.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.245+0000 7f0d45200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d4006c340 con 0x7f0d400cdf90 2024-01-15T18:23:24.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.245+0000 7f0d3de00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0d400b2390 0x7f0d400cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:24.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.245+0000 7f0d3de00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0d400b2390 0x7f0d400cf780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50082/0 (socket says 172.21.15.27:50082) 2024-01-15T18:23:24.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.245+0000 7f0d3de00640 1 -- 172.21.15.27:0/526216439 learned_addr learned my addr 172.21.15.27:0/526216439 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:24.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.245+0000 7f0d3e800640 1 --2- 172.21.15.27:0/526216439 >> v2:172.21.15.93:3300/0 conn(0x7f0d400cdf90 0x7f0d400b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:24.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.245+0000 7f0d3de00640 1 -- 172.21.15.27:0/526216439 >> v2:172.21.15.93:3300/0 conn(0x7f0d400cdf90 msgr2=0x7f0d400b0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:24.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.245+0000 7f0d3de00640 1 --2- 172.21.15.27:0/526216439 >> v2:172.21.15.93:3300/0 conn(0x7f0d400cdf90 0x7f0d400b0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.245+0000 7f0d3de00640 1 -- 172.21.15.27:0/526216439 >> v1:172.21.15.93:6789/0 conn(0x7f0d4006bb80 legacy=0x7f0d400cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:24.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.245+0000 7f0d3de00640 1 -- 172.21.15.27:0/526216439 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4006c620 con 0x7f0d400b2390 2024-01-15T18:23:24.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.249+0000 7f0d3de00640 1 --2- 172.21.15.27:0/526216439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d400b2390 0x7f0d400cf780 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f0d40070dd0 tx=0x7f0d3001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f12b84717b5e26d3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:24.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.249+0000 7f0d3ca00640 1 -- 172.21.15.27:0/526216439 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d300aa5f0 con 0x7f0d400b2390 2024-01-15T18:23:24.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.249+0000 7f0d3ca00640 1 -- 172.21.15.27:0/526216439 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d300aa750 con 0x7f0d400b2390 2024-01-15T18:23:24.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.249+0000 7f0d3ca00640 1 -- 172.21.15.27:0/526216439 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d300aaae0 con 0x7f0d400b2390 2024-01-15T18:23:24.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.249+0000 7f0d45200640 1 -- 172.21.15.27:0/526216439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d400b2390 msgr2=0x7f0d400cf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:24.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.249+0000 7f0d45200640 1 --2- 172.21.15.27:0/526216439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d400b2390 0x7f0d400cf780 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f0d40070dd0 tx=0x7f0d3001c490 comp rx=0 tx=0).stop 2024-01-15T18:23:24.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.249+0000 7f0d45200640 1 -- 172.21.15.27:0/526216439 shutdown_connections 2024-01-15T18:23:24.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.249+0000 7f0d45200640 1 --2- 172.21.15.27:0/526216439 >> v2:172.21.15.93:3300/0 conn(0x7f0d400cdf90 0x7f0d400b0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.249+0000 7f0d45200640 1 --2- 172.21.15.27:0/526216439 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d400b2390 0x7f0d400cf780 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.249+0000 7f0d45200640 1 -- 172.21.15.27:0/526216439 >> 172.21.15.27:0/526216439 conn(0x7f0d400b2a10 msgr2=0x7f0d400b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:24.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.249+0000 7f0d45200640 1 -- 172.21.15.27:0/526216439 shutdown_connections 2024-01-15T18:23:24.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.249+0000 7f0d45200640 1 -- 172.21.15.27:0/526216439 wait complete. 2024-01-15T18:23:24.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.253+0000 7f0d45200640 1 Processor -- start 2024-01-15T18:23:24.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.253+0000 7f0d45200640 1 -- start start 2024-01-15T18:23:24.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.253+0000 7f0d45200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d4006bb80 0x7f0d4014c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:24.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.253+0000 7f0d45200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d400b2390 0x7f0d400b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:24.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.253+0000 7f0d45200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d400cdf90 0x7f0d400ca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:24.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.253+0000 7f0d45200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d40167f80 con 0x7f0d4006bb80 2024-01-15T18:23:24.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.253+0000 7f0d45200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d40168220 con 0x7f0d400b2390 2024-01-15T18:23:24.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.253+0000 7f0d45200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d400b6540 con 0x7f0d400cdf90 2024-01-15T18:23:24.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.253+0000 7f0d3de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d4006bb80 0x7f0d4014c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:24.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.253+0000 7f0d3de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d4006bb80 0x7f0d4014c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50096/0 (socket says 172.21.15.27:50096) 2024-01-15T18:23:24.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.253+0000 7f0d3de00640 1 -- 172.21.15.27:0/385071149 learned_addr learned my addr 172.21.15.27:0/385071149 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:24.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.253+0000 7f0d3d400640 1 --2- 172.21.15.27:0/385071149 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d400b2390 0x7f0d400b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:24.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.253+0000 7f0d3e800640 1 --2- 172.21.15.27:0/385071149 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d400cdf90 0x7f0d400ca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:24.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.253+0000 7f0d3de00640 1 -- 172.21.15.27:0/385071149 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d400cdf90 msgr2=0x7f0d400ca5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:24.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.253+0000 7f0d3de00640 1 --2- 172.21.15.27:0/385071149 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d400cdf90 0x7f0d400ca5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.253+0000 7f0d3de00640 1 -- 172.21.15.27:0/385071149 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d400b2390 msgr2=0x7f0d400b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:24.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.253+0000 7f0d3de00640 1 --2- 172.21.15.27:0/385071149 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d400b2390 0x7f0d400b3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.253+0000 7f0d3de00640 1 -- 172.21.15.27:0/385071149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d300094d0 con 0x7f0d4006bb80 2024-01-15T18:23:24.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.253+0000 7f0d3e800640 1 --2- 172.21.15.27:0/385071149 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d400cdf90 0x7f0d400ca5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:23:24.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.253+0000 7f0d3de00640 1 --2- 172.21.15.27:0/385071149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d4006bb80 0x7f0d4014c2c0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f0d3001a0c0 tx=0x7f0d3001acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:24.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.253+0000 7f0d36000640 1 -- 172.21.15.27:0/385071149 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d300b3800 con 0x7f0d4006bb80 2024-01-15T18:23:24.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.253+0000 7f0d36000640 1 -- 172.21.15.27:0/385071149 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d300b3980 con 0x7f0d4006bb80 2024-01-15T18:23:24.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.253+0000 7f0d45200640 1 -- 172.21.15.27:0/385071149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d400b6820 con 0x7f0d4006bb80 2024-01-15T18:23:24.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.253+0000 7f0d36000640 1 -- 172.21.15.27:0/385071149 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d300b3d30 con 0x7f0d4006bb80 2024-01-15T18:23:24.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.253+0000 7f0d45200640 1 -- 172.21.15.27:0/385071149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d4014c800 con 0x7f0d4006bb80 2024-01-15T18:23:24.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.257+0000 7f0d45200640 1 -- 172.21.15.27:0/385071149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d400d07c0 con 0x7f0d4006bb80 2024-01-15T18:23:24.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.257+0000 7f0d36000640 1 -- 172.21.15.27:0/385071149 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f0d300c6390 con 0x7f0d4006bb80 2024-01-15T18:23:24.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.257+0000 7f0d36000640 1 --2- 172.21.15.27:0/385071149 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0d18067a60 0x7f0d18069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:24.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.257+0000 7f0d36000640 1 -- 172.21.15.27:0/385071149 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f0d300a7070 con 0x7f0d4006bb80 2024-01-15T18:23:24.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.257+0000 7f0d3d400640 1 --2- 172.21.15.27:0/385071149 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0d18067a60 0x7f0d18069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:24.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.257+0000 7f0d3d400640 1 --2- 172.21.15.27:0/385071149 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0d18067a60 0x7f0d18069f20 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f0d400c7ea0 tx=0x7f0d28009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:24.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.261+0000 7f0d36000640 1 -- 172.21.15.27:0/385071149 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0d300ec6c0 con 0x7f0d4006bb80 2024-01-15T18:23:24.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.421+0000 7f0d45200640 1 -- 172.21.15.27:0/385071149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d400b6a60 con 0x7f0d4006bb80 2024-01-15T18:23:24.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.421+0000 7f0d36000640 1 -- 172.21.15.27:0/385071149 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16761 (secure 0 0 0) 0x7f0d400b6a60 con 0x7f0d4006bb80 2024-01-15T18:23:24.417 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:23:24.417 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":110,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:18:42.250892+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:18:30.993882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-15T18:18:31.582649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.5","mappings":[{"from":0,"to":5}]},{"pgid":"17.18","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:23:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.421+0000 7f0d45200640 1 -- 172.21.15.27:0/385071149 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0d18067a60 msgr2=0x7f0d18069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.421+0000 7f0d45200640 1 --2- 172.21.15.27:0/385071149 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0d18067a60 0x7f0d18069f20 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f0d400c7ea0 tx=0x7f0d28009060 comp rx=0 tx=0).stop 2024-01-15T18:23:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.421+0000 7f0d45200640 1 -- 172.21.15.27:0/385071149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d4006bb80 msgr2=0x7f0d4014c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.421+0000 7f0d45200640 1 --2- 172.21.15.27:0/385071149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d4006bb80 0x7f0d4014c2c0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f0d3001a0c0 tx=0x7f0d3001acd0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.421+0000 7f0d45200640 1 -- 172.21.15.27:0/385071149 shutdown_connections 2024-01-15T18:23:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.421+0000 7f0d45200640 1 --2- 172.21.15.27:0/385071149 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0d18067a60 0x7f0d18069f20 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.421+0000 7f0d45200640 1 --2- 172.21.15.27:0/385071149 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0d400cdf90 0x7f0d400ca5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.421+0000 7f0d45200640 1 --2- 172.21.15.27:0/385071149 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0d400b2390 0x7f0d400b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.421+0000 7f0d45200640 1 --2- 172.21.15.27:0/385071149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d4006bb80 0x7f0d4014c2c0 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.421+0000 7f0d45200640 1 -- 172.21.15.27:0/385071149 >> 172.21.15.27:0/385071149 conn(0x7f0d400b2a10 msgr2=0x7f0d400cc570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.421+0000 7f0d45200640 1 -- 172.21.15.27:0/385071149 shutdown_connections 2024-01-15T18:23:24.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.421+0000 7f0d45200640 1 -- 172.21.15.27:0/385071149 wait complete. 2024-01-15T18:23:24.440 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T18:18:30.993882+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T18:18:31.582649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:23:24.440 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:23:24.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.589+0000 7f0fdda00640 1 Processor -- start 2024-01-15T18:23:24.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.589+0000 7f0fdda00640 1 -- start start 2024-01-15T18:23:24.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.589+0000 7f0fdda00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0fd815d190 0x7f0fd80cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:24.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.589+0000 7f0fdda00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0fd815d570 0x7f0fd806be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:24.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.589+0000 7f0fdda00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0fd806b810 0x7f0fd806c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:24.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.589+0000 7f0fdda00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd80b0d70 con 0x7f0fd815d570 2024-01-15T18:23:24.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.589+0000 7f0fdda00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd80d0540 con 0x7f0fd806b810 2024-01-15T18:23:24.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.589+0000 7f0fdda00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd80b0440 con 0x7f0fd815d190 2024-01-15T18:23:24.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.589+0000 7f0fd6000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0fd815d570 0x7f0fd806be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:24.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.589+0000 7f0fd6000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0fd815d570 0x7f0fd806be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50106/0 (socket says 172.21.15.27:50106) 2024-01-15T18:23:24.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.589+0000 7f0fd6000640 1 -- 172.21.15.27:0/1781992285 learned_addr learned my addr 172.21.15.27:0/1781992285 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:24.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.589+0000 7f0fd6a00640 1 --2- 172.21.15.27:0/1781992285 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0fd815d190 0x7f0fd80cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:24.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.589+0000 7f0fd7400640 1 --2- 172.21.15.27:0/1781992285 >> v2:172.21.15.93:3300/0 conn(0x7f0fd806b810 0x7f0fd806c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:24.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.589+0000 7f0fd6000640 1 -- 172.21.15.27:0/1781992285 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0fd815d190 msgr2=0x7f0fd80cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:24.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.589+0000 7f0fd6000640 1 --2- 172.21.15.27:0/1781992285 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0fd815d190 0x7f0fd80cfcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.589+0000 7f0fd6000640 1 -- 172.21.15.27:0/1781992285 >> v2:172.21.15.93:3300/0 conn(0x7f0fd806b810 msgr2=0x7f0fd806c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:24.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.589+0000 7f0fd6000640 1 --2- 172.21.15.27:0/1781992285 >> v2:172.21.15.93:3300/0 conn(0x7f0fd806b810 0x7f0fd806c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.589+0000 7f0fd6000640 1 -- 172.21.15.27:0/1781992285 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd80b0720 con 0x7f0fd815d570 2024-01-15T18:23:24.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.593+0000 7f0fd6000640 1 --2- 172.21.15.27:0/1781992285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd815d570 0x7f0fd806be10 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80710f0 tx=0x7f0fc001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbb68b2cabfe8c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:24.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.593+0000 7f0fd5600640 1 -- 172.21.15.27:0/1781992285 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc00ab650 con 0x7f0fd815d570 2024-01-15T18:23:24.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.593+0000 7f0fd5600640 1 -- 172.21.15.27:0/1781992285 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc00ab7b0 con 0x7f0fd815d570 2024-01-15T18:23:24.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.593+0000 7f0fd5600640 1 -- 172.21.15.27:0/1781992285 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc00abb40 con 0x7f0fd815d570 2024-01-15T18:23:24.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.593+0000 7f0fdda00640 1 -- 172.21.15.27:0/1781992285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd815d570 msgr2=0x7f0fd806be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:24.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.593+0000 7f0fdda00640 1 --2- 172.21.15.27:0/1781992285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd815d570 0x7f0fd806be10 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80710f0 tx=0x7f0fc001c4d0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.593+0000 7f0fdda00640 1 -- 172.21.15.27:0/1781992285 shutdown_connections 2024-01-15T18:23:24.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.593+0000 7f0fdda00640 1 --2- 172.21.15.27:0/1781992285 >> v2:172.21.15.93:3300/0 conn(0x7f0fd806b810 0x7f0fd806c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.593+0000 7f0fdda00640 1 --2- 172.21.15.27:0/1781992285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd815d570 0x7f0fd806be10 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.593+0000 7f0fdda00640 1 --2- 172.21.15.27:0/1781992285 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0fd815d190 0x7f0fd80cfcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.593+0000 7f0fdda00640 1 -- 172.21.15.27:0/1781992285 >> 172.21.15.27:0/1781992285 conn(0x7f0fd80d0e70 msgr2=0x7f0fd80d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:24.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.597+0000 7f0fdda00640 1 -- 172.21.15.27:0/1781992285 shutdown_connections 2024-01-15T18:23:24.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.597+0000 7f0fdda00640 1 -- 172.21.15.27:0/1781992285 wait complete. 2024-01-15T18:23:24.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.597+0000 7f0fdda00640 1 Processor -- start 2024-01-15T18:23:24.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.597+0000 7f0fdda00640 1 -- start start 2024-01-15T18:23:24.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.597+0000 7f0fdda00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0fd806b810 0x7f0fd8157ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:24.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.597+0000 7f0fdda00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd815d190 0x7f0fd81550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:24.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.597+0000 7f0fdda00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0fd815d570 0x7f0fd806e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:24.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.597+0000 7f0fdda00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd80b5430 con 0x7f0fd815d190 2024-01-15T18:23:24.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.597+0000 7f0fdda00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd80c4b40 con 0x7f0fd815d570 2024-01-15T18:23:24.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.597+0000 7f0fdda00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd80b4db0 con 0x7f0fd806b810 2024-01-15T18:23:24.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.597+0000 7f0fd6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd815d190 0x7f0fd81550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:24.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.597+0000 7f0fd6000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd815d190 0x7f0fd81550f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50120/0 (socket says 172.21.15.27:50120) 2024-01-15T18:23:24.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.597+0000 7f0fd6000640 1 -- 172.21.15.27:0/3095762062 learned_addr learned my addr 172.21.15.27:0/3095762062 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:24.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.597+0000 7f0fd6a00640 1 --2- 172.21.15.27:0/3095762062 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0fd806b810 0x7f0fd8157ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:24.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.597+0000 7f0fd7400640 1 --2- 172.21.15.27:0/3095762062 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0fd815d570 0x7f0fd806e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:24.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.597+0000 7f0fd6000640 1 -- 172.21.15.27:0/3095762062 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0fd806b810 msgr2=0x7f0fd8157ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:24.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.597+0000 7f0fd6000640 1 --2- 172.21.15.27:0/3095762062 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0fd806b810 0x7f0fd8157ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.597+0000 7f0fd6000640 1 -- 172.21.15.27:0/3095762062 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0fd815d570 msgr2=0x7f0fd806e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:24.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.597+0000 7f0fd6000640 1 --2- 172.21.15.27:0/3095762062 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0fd815d570 0x7f0fd806e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.597+0000 7f0fd6000640 1 -- 172.21.15.27:0/3095762062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc00094d0 con 0x7f0fd815d190 2024-01-15T18:23:24.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.597+0000 7f0fd6000640 1 --2- 172.21.15.27:0/3095762062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd815d190 0x7f0fd81550f0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f0fc001c9b0 tx=0x7f0fc001ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:24.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.601+0000 7f0fcea00640 1 -- 172.21.15.27:0/3095762062 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc00b4800 con 0x7f0fd815d190 2024-01-15T18:23:24.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.601+0000 7f0fcea00640 1 -- 172.21.15.27:0/3095762062 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc00b4960 con 0x7f0fd815d190 2024-01-15T18:23:24.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.601+0000 7f0fcea00640 1 -- 172.21.15.27:0/3095762062 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc00b4d10 con 0x7f0fd815d190 2024-01-15T18:23:24.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.601+0000 7f0fdda00640 1 -- 172.21.15.27:0/3095762062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd80c74e0 con 0x7f0fd815d190 2024-01-15T18:23:24.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.601+0000 7f0fdda00640 1 -- 172.21.15.27:0/3095762062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd80b5050 con 0x7f0fd815d190 2024-01-15T18:23:24.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.601+0000 7f0fdda00640 1 -- 172.21.15.27:0/3095762062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd80b5570 con 0x7f0fd815d190 2024-01-15T18:23:24.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.601+0000 7f0fcea00640 1 -- 172.21.15.27:0/3095762062 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f0fc00c83a0 con 0x7f0fd815d190 2024-01-15T18:23:24.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.601+0000 7f0fcea00640 1 --2- 172.21.15.27:0/3095762062 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0fb0067a60 0x7f0fb0069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:24.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.601+0000 7f0fcea00640 1 -- 172.21.15.27:0/3095762062 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f0fc011ddc0 con 0x7f0fd815d190 2024-01-15T18:23:24.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.601+0000 7f0fd6a00640 1 --2- 172.21.15.27:0/3095762062 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0fb0067a60 0x7f0fb0069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:24.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.601+0000 7f0fd6a00640 1 --2- 172.21.15.27:0/3095762062 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0fb0067a60 0x7f0fb0069f20 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f0fc80079e0 tx=0x7f0fc8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:24.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.605+0000 7f0fcea00640 1 -- 172.21.15.27:0/3095762062 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0fd80b5570 con 0x7f0fd815d190 2024-01-15T18:23:24.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.765+0000 7f0fdda00640 1 -- 172.21.15.27:0/3095762062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0fd80b5570 con 0x7f0fd815d190 2024-01-15T18:23:24.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.765+0000 7f0fcea00640 1 -- 172.21.15.27:0/3095762062 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v110) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0fd80b5570 con 0x7f0fd815d190 2024-01-15T18:23:24.762 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:23:24.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.769+0000 7f0fdda00640 1 -- 172.21.15.27:0/3095762062 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0fb0067a60 msgr2=0x7f0fb0069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:24.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.769+0000 7f0fdda00640 1 --2- 172.21.15.27:0/3095762062 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0fb0067a60 0x7f0fb0069f20 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f0fc80079e0 tx=0x7f0fc8008040 comp rx=0 tx=0).stop 2024-01-15T18:23:24.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.769+0000 7f0fdda00640 1 -- 172.21.15.27:0/3095762062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd815d190 msgr2=0x7f0fd81550f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:24.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.769+0000 7f0fdda00640 1 --2- 172.21.15.27:0/3095762062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd815d190 0x7f0fd81550f0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f0fc001c9b0 tx=0x7f0fc001ad10 comp rx=0 tx=0).stop 2024-01-15T18:23:24.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.769+0000 7f0fdda00640 1 -- 172.21.15.27:0/3095762062 shutdown_connections 2024-01-15T18:23:24.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.769+0000 7f0fdda00640 1 --2- 172.21.15.27:0/3095762062 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0fb0067a60 0x7f0fb0069f20 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.769+0000 7f0fdda00640 1 --2- 172.21.15.27:0/3095762062 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f0fd815d570 0x7f0fd806e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.769+0000 7f0fdda00640 1 --2- 172.21.15.27:0/3095762062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd815d190 0x7f0fd81550f0 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.769+0000 7f0fdda00640 1 --2- 172.21.15.27:0/3095762062 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f0fd806b810 0x7f0fd8157ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.769+0000 7f0fdda00640 1 -- 172.21.15.27:0/3095762062 >> 172.21.15.27:0/3095762062 conn(0x7f0fd80d0e70 msgr2=0x7f0fd80b1100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:24.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.769+0000 7f0fdda00640 1 -- 172.21.15.27:0/3095762062 shutdown_connections 2024-01-15T18:23:24.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.769+0000 7f0fdda00640 1 -- 172.21.15.27:0/3095762062 wait complete. 2024-01-15T18:23:24.785 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T18:23:24.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.933+0000 7fc2ca200640 1 Processor -- start 2024-01-15T18:23:24.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.933+0000 7fc2ca200640 1 -- start start 2024-01-15T18:23:24.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.933+0000 7fc2ca200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2c40cee10 0x7fc2c40d3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:24.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.933+0000 7fc2ca200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c40cf2d0 con 0x7fc2c40cffa0 2024-01-15T18:23:24.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.933+0000 7fc2ca200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c40b2610 con 0x7fc2c415d5d0 2024-01-15T18:23:24.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.933+0000 7fc2ca200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c40cf5b0 con 0x7fc2c40cee10 2024-01-15T18:23:24.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.933+0000 7fc2c3e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc2c40cffa0 0x7fc2c40b30f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37138/0 (socket says 172.21.15.27:37138) 2024-01-15T18:23:24.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.933+0000 7fc2c3e00640 1 -- 172.21.15.27:0/989842848 learned_addr learned my addr 172.21.15.27:0/989842848 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:24.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.933+0000 7fc2c2a00640 1 --2- 172.21.15.27:0/989842848 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2c40cee10 0x7fc2c40d3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:24.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.933+0000 7fc2c2000640 1 -- 172.21.15.27:0/989842848 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250400439 0 0) 0x7fc2c40cf2d0 con 0x7fc2c40cffa0 2024-01-15T18:23:24.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.933+0000 7fc2c2000640 1 -- 172.21.15.27:0/989842848 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2a4003530 con 0x7fc2c40cffa0 2024-01-15T18:23:24.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.933+0000 7fc2c2000640 1 -- 172.21.15.27:0/989842848 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149118397 0 0) 0x7fc2c40b2610 con 0x7fc2c415d5d0 2024-01-15T18:23:24.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.933+0000 7fc2c2000640 1 -- 172.21.15.27:0/989842848 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2c40cf2d0 con 0x7fc2c415d5d0 2024-01-15T18:23:24.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.933+0000 7fc2c2a00640 1 -- 172.21.15.27:0/989842848 >> v1:172.21.15.93:6789/0 conn(0x7fc2c415d5d0 legacy=0x7fc2c40cf890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:24.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.933+0000 7fc2c2a00640 1 -- 172.21.15.27:0/989842848 >> v1:172.21.15.27:6789/0 conn(0x7fc2c40cffa0 legacy=0x7fc2c40b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:24.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.933+0000 7fc2c2000640 1 -- 172.21.15.27:0/989842848 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3672549003 0 0) 0x7fc2a4003530 con 0x7fc2c40cffa0 2024-01-15T18:23:24.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.933+0000 7fc2c2a00640 1 -- 172.21.15.27:0/989842848 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c40b28f0 con 0x7fc2c40cee10 2024-01-15T18:23:24.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.937+0000 7fc2c2a00640 1 --2- 172.21.15.27:0/989842848 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2c40cee10 0x7fc2c40d3b30 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0097f0 tx=0x7fc2ac01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ae320c11f6aea14 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:24.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.937+0000 7fc2c2000640 1 -- 172.21.15.27:0/989842848 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc001a00 con 0x7fc2c40cee10 2024-01-15T18:23:24.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.937+0000 7fc2c2000640 1 -- 172.21.15.27:0/989842848 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2ac0ad480 con 0x7fc2c40cee10 2024-01-15T18:23:24.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.937+0000 7fc2c2000640 1 -- 172.21.15.27:0/989842848 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac0ad810 con 0x7fc2c40cee10 2024-01-15T18:23:24.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.937+0000 7fc2ca200640 1 -- 172.21.15.27:0/989842848 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2c40cee10 msgr2=0x7fc2c40d3b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:24.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.937+0000 7fc2ca200640 1 --2- 172.21.15.27:0/989842848 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2c40cee10 0x7fc2c40d3b30 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0097f0 tx=0x7fc2ac01c3e0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.937+0000 7fc2ca200640 1 -- 172.21.15.27:0/989842848 shutdown_connections 2024-01-15T18:23:24.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.937+0000 7fc2ca200640 1 --2- 172.21.15.27:0/989842848 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2c40cee10 0x7fc2c40d3b30 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.937+0000 7fc2ca200640 1 -- 172.21.15.27:0/989842848 >> 172.21.15.27:0/989842848 conn(0x7fc2c40d3010 msgr2=0x7fc2c40d3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:24.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.937+0000 7fc2ca200640 1 -- 172.21.15.27:0/989842848 shutdown_connections 2024-01-15T18:23:24.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.937+0000 7fc2ca200640 1 -- 172.21.15.27:0/989842848 wait complete. 2024-01-15T18:23:24.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.941+0000 7fc2ca200640 1 Processor -- start 2024-01-15T18:23:24.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.941+0000 7fc2ca200640 1 -- start start 2024-01-15T18:23:24.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.941+0000 7fc2ca200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2c40cee10 0x7fc2c414f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:24.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.941+0000 7fc2ca200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2c40cffa0 0x7fc2c40b6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:24.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.941+0000 7fc2ca200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2c415d5d0 0x7fc2c40cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:24.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.941+0000 7fc2ca200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c40b81d0 con 0x7fc2c415d5d0 2024-01-15T18:23:24.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.941+0000 7fc2ca200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c40b75c0 con 0x7fc2c40cee10 2024-01-15T18:23:24.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.941+0000 7fc2ca200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c40b78a0 con 0x7fc2c40cffa0 2024-01-15T18:23:24.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.941+0000 7fc2c3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2c415d5d0 0x7fc2c40cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:24.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.941+0000 7fc2c3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2c415d5d0 0x7fc2c40cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50126/0 (socket says 172.21.15.27:50126) 2024-01-15T18:23:24.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.941+0000 7fc2c3e00640 1 -- 172.21.15.27:0/1136955325 learned_addr learned my addr 172.21.15.27:0/1136955325 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:24.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.941+0000 7fc2c2a00640 1 --2- 172.21.15.27:0/1136955325 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2c40cffa0 0x7fc2c40b6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:24.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.941+0000 7fc2c3400640 1 --2- 172.21.15.27:0/1136955325 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2c40cee10 0x7fc2c414f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:24.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.941+0000 7fc2c3e00640 1 -- 172.21.15.27:0/1136955325 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2c40cffa0 msgr2=0x7fc2c40b6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:24.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.941+0000 7fc2c3e00640 1 --2- 172.21.15.27:0/1136955325 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2c40cffa0 0x7fc2c40b6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.941+0000 7fc2c3e00640 1 -- 172.21.15.27:0/1136955325 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2c40cee10 msgr2=0x7fc2c414f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:24.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.941+0000 7fc2c3e00640 1 --2- 172.21.15.27:0/1136955325 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2c40cee10 0x7fc2c414f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:24.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.941+0000 7fc2c3e00640 1 -- 172.21.15.27:0/1136955325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ac009420 con 0x7fc2c415d5d0 2024-01-15T18:23:24.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.941+0000 7fc2c3e00640 1 --2- 172.21.15.27:0/1136955325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2c415d5d0 0x7fc2c40cae10 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc001f50 tx=0x7fc2bc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:24.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.941+0000 7fc2bb400640 1 -- 172.21.15.27:0/1136955325 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc012400 con 0x7fc2c415d5d0 2024-01-15T18:23:24.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.941+0000 7fc2bb400640 1 -- 172.21.15.27:0/1136955325 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2bc012580 con 0x7fc2c415d5d0 2024-01-15T18:23:24.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.941+0000 7fc2bb400640 1 -- 172.21.15.27:0/1136955325 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc012930 con 0x7fc2c415d5d0 2024-01-15T18:23:24.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.941+0000 7fc2ca200640 1 -- 172.21.15.27:0/1136955325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c40b7b80 con 0x7fc2c415d5d0 2024-01-15T18:23:24.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.941+0000 7fc2ca200640 1 -- 172.21.15.27:0/1136955325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c40cb350 con 0x7fc2c415d5d0 2024-01-15T18:23:24.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.945+0000 7fc2ca200640 1 -- 172.21.15.27:0/1136955325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2c40b8340 con 0x7fc2c415d5d0 2024-01-15T18:23:24.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.945+0000 7fc2bb400640 1 -- 172.21.15.27:0/1136955325 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7fc2bc012a90 con 0x7fc2c415d5d0 2024-01-15T18:23:24.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.945+0000 7fc2bb400640 1 --2- 172.21.15.27:0/1136955325 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc29c067a60 0x7fc29c069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:24.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.945+0000 7fc2bb400640 1 -- 172.21.15.27:0/1136955325 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fc2bc0839f0 con 0x7fc2c415d5d0 2024-01-15T18:23:24.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.945+0000 7fc2c3400640 1 --2- 172.21.15.27:0/1136955325 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc29c067a60 0x7fc29c069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:24.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.945+0000 7fc2c3400640 1 --2- 172.21.15.27:0/1136955325 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc29c067a60 0x7fc29c069f20 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fc2b40011f0 tx=0x7fc2b4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:24.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:24.949+0000 7fc2bb400640 1 -- 172.21.15.27:0/1136955325 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc2bc0523e0 con 0x7fc2c415d5d0 2024-01-15T18:23:25.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.105+0000 7fc2ca200640 1 -- 172.21.15.27:0/1136955325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc2c404e850 con 0x7fc2c415d5d0 2024-01-15T18:23:25.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.105+0000 7fc2bb400640 1 -- 172.21.15.27:0/1136955325 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v110) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc2bc055a00 con 0x7fc2c415d5d0 2024-01-15T18:23:25.101 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-15T18:23:25.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.105+0000 7fc2ca200640 1 -- 172.21.15.27:0/1136955325 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc29c067a60 msgr2=0x7fc29c069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:25.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.105+0000 7fc2ca200640 1 --2- 172.21.15.27:0/1136955325 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc29c067a60 0x7fc29c069f20 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fc2b40011f0 tx=0x7fc2b4009040 comp rx=0 tx=0).stop 2024-01-15T18:23:25.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.105+0000 7fc2ca200640 1 -- 172.21.15.27:0/1136955325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2c415d5d0 msgr2=0x7fc2c40cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:25.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.105+0000 7fc2ca200640 1 --2- 172.21.15.27:0/1136955325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2c415d5d0 0x7fc2c40cae10 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc001f50 tx=0x7fc2bc00a430 comp rx=0 tx=0).stop 2024-01-15T18:23:25.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.105+0000 7fc2ca200640 1 -- 172.21.15.27:0/1136955325 shutdown_connections 2024-01-15T18:23:25.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.105+0000 7fc2ca200640 1 --2- 172.21.15.27:0/1136955325 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc29c067a60 0x7fc29c069f20 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.105+0000 7fc2ca200640 1 --2- 172.21.15.27:0/1136955325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2c415d5d0 0x7fc2c40cae10 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.105+0000 7fc2ca200640 1 --2- 172.21.15.27:0/1136955325 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc2c40cffa0 0x7fc2c40b6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.105+0000 7fc2ca200640 1 --2- 172.21.15.27:0/1136955325 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc2c40cee10 0x7fc2c414f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.105+0000 7fc2ca200640 1 -- 172.21.15.27:0/1136955325 >> 172.21.15.27:0/1136955325 conn(0x7fc2c40d3010 msgr2=0x7fc2c40cf6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:25.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.105+0000 7fc2ca200640 1 -- 172.21.15.27:0/1136955325 shutdown_connections 2024-01-15T18:23:25.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.105+0000 7fc2ca200640 1 -- 172.21.15.27:0/1136955325 wait complete. 2024-01-15T18:23:25.123 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T18:23:25.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.273+0000 7f6658200640 1 Processor -- start 2024-01-15T18:23:25.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.273+0000 7f6658200640 1 -- start start 2024-01-15T18:23:25.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.273+0000 7f6658200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f6650158900 0x7f665015f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:25.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.273+0000 7f6658200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f665015bf00 0x7f66500cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:25.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.273+0000 7f6658200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66500b1cb0 con 0x7f66500cf110 2024-01-15T18:23:25.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.273+0000 7f6658200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66500cf580 con 0x7f665015bf00 2024-01-15T18:23:25.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.273+0000 7f6658200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66500b1f00 con 0x7f6650158900 2024-01-15T18:23:25.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.273+0000 7f6654e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f665015bf00 0x7f66500cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:25.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.273+0000 7f6654e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f665015bf00 0x7f66500cebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50140/0 (socket says 172.21.15.27:50140) 2024-01-15T18:23:25.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.273+0000 7f6654e00640 1 -- 172.21.15.27:0/2961403215 learned_addr learned my addr 172.21.15.27:0/2961403215 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:25.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.273+0000 7f6655800640 1 --2- 172.21.15.27:0/2961403215 >> v2:172.21.15.93:3300/0 conn(0x7f6650158900 0x7f665015f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:25.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.273+0000 7f6654e00640 1 -- 172.21.15.27:0/2961403215 >> v2:172.21.15.93:3300/0 conn(0x7f6650158900 msgr2=0x7f665015f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:25.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.273+0000 7f6654e00640 1 --2- 172.21.15.27:0/2961403215 >> v2:172.21.15.93:3300/0 conn(0x7f6650158900 0x7f665015f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.273+0000 7f6654e00640 1 -- 172.21.15.27:0/2961403215 >> v1:172.21.15.93:6789/0 conn(0x7f66500cf110 legacy=0x7f66500701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:25.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.273+0000 7f6654e00640 1 -- 172.21.15.27:0/2961403215 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66500b21e0 con 0x7f665015bf00 2024-01-15T18:23:25.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.277+0000 7f6654e00640 1 --2- 172.21.15.27:0/2961403215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f665015bf00 0x7f66500cebd0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f6638009870 tx=0x7f663801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=993271fc18eb4f87 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:25.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.277+0000 7f664fe00640 1 -- 172.21.15.27:0/2961403215 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66380ab5f0 con 0x7f665015bf00 2024-01-15T18:23:25.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.277+0000 7f664fe00640 1 -- 172.21.15.27:0/2961403215 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66380ab750 con 0x7f665015bf00 2024-01-15T18:23:25.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.277+0000 7f664fe00640 1 -- 172.21.15.27:0/2961403215 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66380abae0 con 0x7f665015bf00 2024-01-15T18:23:25.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.277+0000 7f6658200640 1 -- 172.21.15.27:0/2961403215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f665015bf00 msgr2=0x7f66500cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:25.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.277+0000 7f6658200640 1 --2- 172.21.15.27:0/2961403215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f665015bf00 0x7f66500cebd0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f6638009870 tx=0x7f663801c500 comp rx=0 tx=0).stop 2024-01-15T18:23:25.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.277+0000 7f6658200640 1 -- 172.21.15.27:0/2961403215 shutdown_connections 2024-01-15T18:23:25.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.277+0000 7f6658200640 1 --2- 172.21.15.27:0/2961403215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f665015bf00 0x7f66500cebd0 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.277+0000 7f6658200640 1 --2- 172.21.15.27:0/2961403215 >> v2:172.21.15.93:3300/0 conn(0x7f6650158900 0x7f665015f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.277+0000 7f6658200640 1 -- 172.21.15.27:0/2961403215 >> 172.21.15.27:0/2961403215 conn(0x7f66500cb700 msgr2=0x7f66500cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:25.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.277+0000 7f6658200640 1 -- 172.21.15.27:0/2961403215 shutdown_connections 2024-01-15T18:23:25.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.277+0000 7f6658200640 1 -- 172.21.15.27:0/2961403215 wait complete. 2024-01-15T18:23:25.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.281+0000 7f6658200640 1 Processor -- start 2024-01-15T18:23:25.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.281+0000 7f6658200640 1 -- start start 2024-01-15T18:23:25.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.281+0000 7f6658200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66500cf110 0x7f6650146e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:25.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.281+0000 7f6658200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6650158900 0x7f665014fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:25.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.281+0000 7f6658200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f665015bf00 0x7f6650150150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:25.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.281+0000 7f6658200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66501574a0 con 0x7f66500cf110 2024-01-15T18:23:25.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.281+0000 7f6658200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66500ce620 con 0x7f6650158900 2024-01-15T18:23:25.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.281+0000 7f6658200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66501576f0 con 0x7f665015bf00 2024-01-15T18:23:25.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.281+0000 7f6655800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66500cf110 0x7f6650146e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:25.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.281+0000 7f6655800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66500cf110 0x7f6650146e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50148/0 (socket says 172.21.15.27:50148) 2024-01-15T18:23:25.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.281+0000 7f6655800640 1 -- 172.21.15.27:0/279699015 learned_addr learned my addr 172.21.15.27:0/279699015 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:25.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.281+0000 7f6656200640 1 --2- 172.21.15.27:0/279699015 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f665015bf00 0x7f6650150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:25.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.281+0000 7f6654e00640 1 --2- 172.21.15.27:0/279699015 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6650158900 0x7f665014fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:25.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.281+0000 7f6655800640 1 -- 172.21.15.27:0/279699015 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f665015bf00 msgr2=0x7f6650150150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:25.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.281+0000 7f6655800640 1 --2- 172.21.15.27:0/279699015 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f665015bf00 0x7f6650150150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.281+0000 7f6655800640 1 -- 172.21.15.27:0/279699015 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6650158900 msgr2=0x7f665014fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:25.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.281+0000 7f6655800640 1 --2- 172.21.15.27:0/279699015 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6650158900 0x7f665014fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.281+0000 7f6655800640 1 -- 172.21.15.27:0/279699015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66380094d0 con 0x7f66500cf110 2024-01-15T18:23:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.281+0000 7f6654e00640 1 --2- 172.21.15.27:0/279699015 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6650158900 0x7f665014fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:23:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.281+0000 7f6655800640 1 --2- 172.21.15.27:0/279699015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66500cf110 0x7f6650146e00 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f664000d830 tx=0x7f664000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.281+0000 7f664d600640 1 -- 172.21.15.27:0/279699015 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664000b6c0 con 0x7f66500cf110 2024-01-15T18:23:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.281+0000 7f664d600640 1 -- 172.21.15.27:0/279699015 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664000b860 con 0x7f66500cf110 2024-01-15T18:23:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.281+0000 7f664d600640 1 -- 172.21.15.27:0/279699015 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664000bc10 con 0x7f66500cf110 2024-01-15T18:23:25.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.281+0000 7f6658200640 1 -- 172.21.15.27:0/279699015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66500ce2b0 con 0x7f66500cf110 2024-01-15T18:23:25.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.281+0000 7f6658200640 1 -- 172.21.15.27:0/279699015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6650147340 con 0x7f66500cf110 2024-01-15T18:23:25.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.285+0000 7f6658200640 1 -- 172.21.15.27:0/279699015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66500ce440 con 0x7f66500cf110 2024-01-15T18:23:25.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.285+0000 7f664d600640 1 -- 172.21.15.27:0/279699015 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f664001e460 con 0x7f66500cf110 2024-01-15T18:23:25.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.285+0000 7f664d600640 1 --2- 172.21.15.27:0/279699015 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6628067a60 0x7f6628069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:25.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.285+0000 7f664d600640 1 -- 172.21.15.27:0/279699015 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f6640083b10 con 0x7f66500cf110 2024-01-15T18:23:25.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.285+0000 7f6654e00640 1 --2- 172.21.15.27:0/279699015 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6628067a60 0x7f6628069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:25.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.285+0000 7f6654e00640 1 --2- 172.21.15.27:0/279699015 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6628067a60 0x7f6628069f20 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f66380af000 tx=0x7f66380a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:25.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.289+0000 7f664d600640 1 -- 172.21.15.27:0/279699015 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6640052450 con 0x7f66500cf110 2024-01-15T18:23:25.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.461+0000 7f6658200640 1 -- 172.21.15.27:0/279699015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f66501578b0 con 0x7f66500cf110 2024-01-15T18:23:25.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.461+0000 7f664d600640 1 -- 172.21.15.27:0/279699015 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v110) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6640055a70 con 0x7f66500cf110 2024-01-15T18:23:25.456 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-15T18:23:25.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.461+0000 7f6658200640 1 -- 172.21.15.27:0/279699015 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6628067a60 msgr2=0x7f6628069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:25.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.461+0000 7f6658200640 1 --2- 172.21.15.27:0/279699015 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6628067a60 0x7f6628069f20 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f66380af000 tx=0x7f66380a7000 comp rx=0 tx=0).stop 2024-01-15T18:23:25.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.461+0000 7f6658200640 1 -- 172.21.15.27:0/279699015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66500cf110 msgr2=0x7f6650146e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:25.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.461+0000 7f6658200640 1 --2- 172.21.15.27:0/279699015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66500cf110 0x7f6650146e00 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f664000d830 tx=0x7f664000dcf0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.461+0000 7f6658200640 1 -- 172.21.15.27:0/279699015 shutdown_connections 2024-01-15T18:23:25.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.461+0000 7f6658200640 1 --2- 172.21.15.27:0/279699015 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6628067a60 0x7f6628069f20 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.461+0000 7f6658200640 1 --2- 172.21.15.27:0/279699015 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f665015bf00 0x7f6650150150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.461+0000 7f6658200640 1 --2- 172.21.15.27:0/279699015 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6650158900 0x7f665014fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.461+0000 7f6658200640 1 --2- 172.21.15.27:0/279699015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f66500cf110 0x7f6650146e00 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.461+0000 7f6658200640 1 -- 172.21.15.27:0/279699015 >> 172.21.15.27:0/279699015 conn(0x7f66500cb700 msgr2=0x7f665006c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:25.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.461+0000 7f6658200640 1 -- 172.21.15.27:0/279699015 shutdown_connections 2024-01-15T18:23:25.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.461+0000 7f6658200640 1 -- 172.21.15.27:0/279699015 wait complete. 2024-01-15T18:23:25.479 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:23:25.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.625+0000 7f71ffe00640 1 Processor -- start 2024-01-15T18:23:25.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.625+0000 7f71ffe00640 1 -- start start 2024-01-15T18:23:25.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.625+0000 7f71ffe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71f80b4540 0x7f71f806a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:25.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.625+0000 7f71ffe00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f71f806aa30 0x7f71f80b2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:25.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.625+0000 7f71ffe00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f815dd70 con 0x7f71f815aec0 2024-01-15T18:23:25.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.625+0000 7f71ffe00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71f80acee0 con 0x7f71f806aa30 2024-01-15T18:23:25.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.625+0000 7f71ffe00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f815b2a0 con 0x7f71f80b4540 2024-01-15T18:23:25.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71fd400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f71f815aec0 0x7f71f815e200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37154/0 (socket says 172.21.15.27:37154) 2024-01-15T18:23:25.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71fd400640 1 -- 172.21.15.27:0/2255179257 learned_addr learned my addr 172.21.15.27:0/2255179257 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:25.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71fde00640 1 --2- 172.21.15.27:0/2255179257 >> v2:172.21.15.93:3300/0 conn(0x7f71f806aa30 0x7f71f80b2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:25.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71fca00640 1 --2- 172.21.15.27:0/2255179257 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71f80b4540 0x7f71f806a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:25.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71f7e00640 1 -- 172.21.15.27:0/2255179257 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968397545 0 0) 0x7f71f815dd70 con 0x7f71f815aec0 2024-01-15T18:23:25.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71f7e00640 1 -- 172.21.15.27:0/2255179257 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71d8003530 con 0x7f71f815aec0 2024-01-15T18:23:25.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71f7e00640 1 -- 172.21.15.27:0/2255179257 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1450804446 0 0) 0x7f71d8003530 con 0x7f71f815aec0 2024-01-15T18:23:25.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71f7e00640 1 -- 172.21.15.27:0/2255179257 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71f80b4540 msgr2=0x7f71f806a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:25.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71f7e00640 1 --2- 172.21.15.27:0/2255179257 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71f80b4540 0x7f71f806a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71f7e00640 1 -- 172.21.15.27:0/2255179257 >> v2:172.21.15.93:3300/0 conn(0x7f71f806aa30 msgr2=0x7f71f80b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:25.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71f7e00640 1 --2- 172.21.15.27:0/2255179257 >> v2:172.21.15.93:3300/0 conn(0x7f71f806aa30 0x7f71f80b2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71f7e00640 1 -- 172.21.15.27:0/2255179257 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f80ac920 con 0x7f71f815aec0 2024-01-15T18:23:25.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71f7e00640 1 -- 172.21.15.27:0/2255179257 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f71e8001680 con 0x7f71f815aec0 2024-01-15T18:23:25.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71f7e00640 1 -- 172.21.15.27:0/2255179257 >> v1:172.21.15.27:6789/0 conn(0x7f71f815aec0 legacy=0x7f71f815e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:25.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71f7e00640 1 --2- 172.21.15.27:0/2255179257 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f71d80039f0 0x7f71d8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:25.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71f7e00640 1 --2- 172.21.15.27:0/2255179257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71d80047d0 0x7f71d8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:25.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71f7e00640 1 --2- 172.21.15.27:0/2255179257 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71f80b4540 0x7f71d8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:25.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71f7e00640 1 -- 172.21.15.27:0/2255179257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f80acee0 con 0x7f71d80047d0 2024-01-15T18:23:25.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71f7e00640 1 -- 172.21.15.27:0/2255179257 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f815b2a0 con 0x7f71d80039f0 2024-01-15T18:23:25.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71f7e00640 1 -- 172.21.15.27:0/2255179257 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d800ba40 con 0x7f71f80b4540 2024-01-15T18:23:25.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71fca00640 1 --2- 172.21.15.27:0/2255179257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71d80047d0 0x7f71d8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:25.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71fd400640 1 --2- 172.21.15.27:0/2255179257 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f71d80039f0 0x7f71d8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:25.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71fde00640 1 --2- 172.21.15.27:0/2255179257 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71f80b4540 0x7f71d8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:25.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71fca00640 1 -- 172.21.15.27:0/2255179257 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71f80b4540 msgr2=0x7f71d8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:25.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71fca00640 1 --2- 172.21.15.27:0/2255179257 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71f80b4540 0x7f71d8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71fca00640 1 -- 172.21.15.27:0/2255179257 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f71d80039f0 msgr2=0x7f71d8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:25.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71fca00640 1 --2- 172.21.15.27:0/2255179257 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f71d80039f0 0x7f71d8003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71fca00640 1 -- 172.21.15.27:0/2255179257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d8003530 con 0x7f71d80047d0 2024-01-15T18:23:25.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71fde00640 1 --2- 172.21.15.27:0/2255179257 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71f80b4540 0x7f71d8005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:23:25.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.629+0000 7f71fd400640 1 --2- 172.21.15.27:0/2255179257 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f71d80039f0 0x7f71d8003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:23:25.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.633+0000 7f71fca00640 1 --2- 172.21.15.27:0/2255179257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71d80047d0 0x7f71d8004c50 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f71e0009ab0 tx=0x7f71e001ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f5cf0797fee0e38 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:25.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.633+0000 7f71f7e00640 1 -- 172.21.15.27:0/2255179257 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e00a64e0 con 0x7f71d80047d0 2024-01-15T18:23:25.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.633+0000 7f71f7e00640 1 -- 172.21.15.27:0/2255179257 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e00a6640 con 0x7f71d80047d0 2024-01-15T18:23:25.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.633+0000 7f71f7e00640 1 -- 172.21.15.27:0/2255179257 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e00a69f0 con 0x7f71d80047d0 2024-01-15T18:23:25.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.633+0000 7f71ffe00640 1 -- 172.21.15.27:0/2255179257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71d80047d0 msgr2=0x7f71d8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:25.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.633+0000 7f71ffe00640 1 --2- 172.21.15.27:0/2255179257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71d80047d0 0x7f71d8004c50 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f71e0009ab0 tx=0x7f71e001ca10 comp rx=0 tx=0).stop 2024-01-15T18:23:25.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.633+0000 7f71fde00640 1 -- 172.21.15.27:0/2255179257 reap_dead start 2024-01-15T18:23:25.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.633+0000 7f71ffe00640 1 -- 172.21.15.27:0/2255179257 shutdown_connections 2024-01-15T18:23:25.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.633+0000 7f71ffe00640 1 -- 172.21.15.27:0/2255179257 >> 172.21.15.27:0/2255179257 conn(0x7f71f80cf520 msgr2=0x7f71f80cf930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:25.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.633+0000 7f71ffe00640 1 -- 172.21.15.27:0/2255179257 shutdown_connections 2024-01-15T18:23:25.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.633+0000 7f71ffe00640 1 -- 172.21.15.27:0/2255179257 wait complete. 2024-01-15T18:23:25.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.633+0000 7f71ffe00640 1 Processor -- start 2024-01-15T18:23:25.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.633+0000 7f71ffe00640 1 -- start start 2024-01-15T18:23:25.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.637+0000 7f71ffe00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f71d80047d0 0x7f71f8152d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:25.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.637+0000 7f71ffe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71f80b06e0 0x7f71f8152480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:25.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.637+0000 7f71ffe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71f8166160 0x7f71f81665e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:25.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.637+0000 7f71ffe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f80cbf70 con 0x7f71f80b06e0 2024-01-15T18:23:25.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.637+0000 7f71ffe00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f8166b20 con 0x7f71d80047d0 2024-01-15T18:23:25.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.637+0000 7f71ffe00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f80b0ea0 con 0x7f71f8166160 2024-01-15T18:23:25.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.637+0000 7f71fca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71f80b06e0 0x7f71f8152480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:25.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.637+0000 7f71fca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71f80b06e0 0x7f71f8152480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50164/0 (socket says 172.21.15.27:50164) 2024-01-15T18:23:25.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.637+0000 7f71fca00640 1 -- 172.21.15.27:0/1105022368 learned_addr learned my addr 172.21.15.27:0/1105022368 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:25.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.637+0000 7f71fd400640 1 --2- 172.21.15.27:0/1105022368 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f71d80047d0 0x7f71f8152d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:25.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.637+0000 7f71fde00640 1 --2- 172.21.15.27:0/1105022368 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71f8166160 0x7f71f81665e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:25.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.637+0000 7f71fca00640 1 -- 172.21.15.27:0/1105022368 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71f8166160 msgr2=0x7f71f81665e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:25.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.637+0000 7f71fca00640 1 --2- 172.21.15.27:0/1105022368 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71f8166160 0x7f71f81665e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.637+0000 7f71fca00640 1 -- 172.21.15.27:0/1105022368 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f71d80047d0 msgr2=0x7f71f8152d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:25.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.637+0000 7f71fca00640 1 --2- 172.21.15.27:0/1105022368 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f71d80047d0 0x7f71f8152d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.637+0000 7f71fca00640 1 -- 172.21.15.27:0/1105022368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f815b410 con 0x7f71f80b06e0 2024-01-15T18:23:25.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.637+0000 7f71fca00640 1 --2- 172.21.15.27:0/1105022368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71f80b06e0 0x7f71f8152480 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f71e001c840 tx=0x7f71d8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:25.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.637+0000 7f71f5600640 1 -- 172.21.15.27:0/1105022368 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e00a69b0 con 0x7f71f80b06e0 2024-01-15T18:23:25.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.637+0000 7f71f5600640 1 -- 172.21.15.27:0/1105022368 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e00a6b30 con 0x7f71f80b06e0 2024-01-15T18:23:25.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.637+0000 7f71ffe00640 1 -- 172.21.15.27:0/1105022368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f815b6f0 con 0x7f71f80b06e0 2024-01-15T18:23:25.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.637+0000 7f71f5600640 1 -- 172.21.15.27:0/1105022368 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e000f0c0 con 0x7f71f80b06e0 2024-01-15T18:23:25.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.637+0000 7f71ffe00640 1 -- 172.21.15.27:0/1105022368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f815bcb0 con 0x7f71f80b06e0 2024-01-15T18:23:25.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.637+0000 7f71ffe00640 1 -- 172.21.15.27:0/1105022368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f804e850 con 0x7f71f80b06e0 2024-01-15T18:23:25.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.641+0000 7f71f5600640 1 -- 172.21.15.27:0/1105022368 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f71e0006040 con 0x7f71f80b06e0 2024-01-15T18:23:25.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.641+0000 7f71f5600640 1 --2- 172.21.15.27:0/1105022368 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f71d0067a60 0x7f71d0069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:25.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.641+0000 7f71f5600640 1 -- 172.21.15.27:0/1105022368 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f71e00aa070 con 0x7f71f80b06e0 2024-01-15T18:23:25.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.641+0000 7f71fd400640 1 --2- 172.21.15.27:0/1105022368 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f71d0067a60 0x7f71d0069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:25.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.641+0000 7f71fd400640 1 --2- 172.21.15.27:0/1105022368 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f71d0067a60 0x7f71d0069f20 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f71e8001000 tx=0x7f71e8007040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:25.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.645+0000 7f71f5600640 1 -- 172.21.15.27:0/1105022368 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f71e00eca70 con 0x7f71f80b06e0 2024-01-15T18:23:25.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.813+0000 7f71ffe00640 1 -- 172.21.15.27:0/1105022368 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f71f80b2b90 con 0x7f71f80b06e0 2024-01-15T18:23:25.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.813+0000 7f71f5600640 1 -- 172.21.15.27:0/1105022368 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v110) v1 ==== 80+0+0 (secure 0 0 0) 0x7f71e00f0090 con 0x7f71f80b06e0 2024-01-15T18:23:25.811 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-01-15T18:23:25.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.817+0000 7f71ffe00640 1 -- 172.21.15.27:0/1105022368 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f71d0067a60 msgr2=0x7f71d0069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:25.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.817+0000 7f71ffe00640 1 --2- 172.21.15.27:0/1105022368 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f71d0067a60 0x7f71d0069f20 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f71e8001000 tx=0x7f71e8007040 comp rx=0 tx=0).stop 2024-01-15T18:23:25.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.817+0000 7f71ffe00640 1 -- 172.21.15.27:0/1105022368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71f80b06e0 msgr2=0x7f71f8152480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:25.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.817+0000 7f71ffe00640 1 --2- 172.21.15.27:0/1105022368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71f80b06e0 0x7f71f8152480 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f71e001c840 tx=0x7f71d8005800 comp rx=0 tx=0).stop 2024-01-15T18:23:25.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.817+0000 7f71ffe00640 1 -- 172.21.15.27:0/1105022368 shutdown_connections 2024-01-15T18:23:25.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.817+0000 7f71ffe00640 1 --2- 172.21.15.27:0/1105022368 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f71d0067a60 0x7f71d0069f20 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.817+0000 7f71ffe00640 1 --2- 172.21.15.27:0/1105022368 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f71f8166160 0x7f71f81665e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.817+0000 7f71ffe00640 1 --2- 172.21.15.27:0/1105022368 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71f80b06e0 0x7f71f8152480 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.817+0000 7f71ffe00640 1 --2- 172.21.15.27:0/1105022368 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f71d80047d0 0x7f71f8152d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.817+0000 7f71ffe00640 1 -- 172.21.15.27:0/1105022368 >> 172.21.15.27:0/1105022368 conn(0x7f71f80cf520 msgr2=0x7f71f815e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:25.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.817+0000 7f71ffe00640 1 -- 172.21.15.27:0/1105022368 shutdown_connections 2024-01-15T18:23:25.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.817+0000 7f71ffe00640 1 -- 172.21.15.27:0/1105022368 wait complete. 2024-01-15T18:23:25.833 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:23:25.833 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:23:25.843 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:23:25.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.989+0000 7fc0e7000640 1 Processor -- start 2024-01-15T18:23:25.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.989+0000 7fc0e7000640 1 -- start start 2024-01-15T18:23:25.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.989+0000 7fc0e7000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fc0e00701a0 0x7fc0e0070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:25.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.989+0000 7fc0e7000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc0e00c92b0 0x7fc0e006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:25.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.989+0000 7fc0e7000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e006c4a0 con 0x7fc0e006e240 2024-01-15T18:23:25.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.989+0000 7fc0e7000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0e0070ac0 con 0x7fc0e00c92b0 2024-01-15T18:23:25.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.989+0000 7fc0e7000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0e015c660 con 0x7fc0e00701a0 2024-01-15T18:23:25.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.989+0000 7fc0df400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc0e00c92b0 0x7fc0e006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:25.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.989+0000 7fc0df400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc0e00c92b0 0x7fc0e006dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50174/0 (socket says 172.21.15.27:50174) 2024-01-15T18:23:25.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.989+0000 7fc0df400640 1 -- 172.21.15.27:0/3376541988 learned_addr learned my addr 172.21.15.27:0/3376541988 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:25.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.989+0000 7fc0dfe00640 1 --2- 172.21.15.27:0/3376541988 >> v2:172.21.15.93:3300/0 conn(0x7fc0e00701a0 0x7fc0e0070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:25.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.989+0000 7fc0df400640 1 -- 172.21.15.27:0/3376541988 >> v2:172.21.15.93:3300/0 conn(0x7fc0e00701a0 msgr2=0x7fc0e0070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:25.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.989+0000 7fc0df400640 1 --2- 172.21.15.27:0/3376541988 >> v2:172.21.15.93:3300/0 conn(0x7fc0e00701a0 0x7fc0e0070580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.989+0000 7fc0df400640 1 -- 172.21.15.27:0/3376541988 >> v1:172.21.15.93:6789/0 conn(0x7fc0e006e240 legacy=0x7fc0e015d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:25.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.989+0000 7fc0df400640 1 -- 172.21.15.27:0/3376541988 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e015c940 con 0x7fc0e00c92b0 2024-01-15T18:23:25.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.993+0000 7fc0df400640 1 --2- 172.21.15.27:0/3376541988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e00c92b0 0x7fc0e006dd00 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7fc0e00681f0 tx=0x7fc0c801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b19e58585efd9677 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:25.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.993+0000 7fc0dea00640 1 -- 172.21.15.27:0/3376541988 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c80aa5f0 con 0x7fc0e00c92b0 2024-01-15T18:23:25.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.993+0000 7fc0dea00640 1 -- 172.21.15.27:0/3376541988 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c80aa750 con 0x7fc0e00c92b0 2024-01-15T18:23:25.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.993+0000 7fc0dea00640 1 -- 172.21.15.27:0/3376541988 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c80aaae0 con 0x7fc0e00c92b0 2024-01-15T18:23:25.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.993+0000 7fc0e7000640 1 -- 172.21.15.27:0/3376541988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e00c92b0 msgr2=0x7fc0e006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:25.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.993+0000 7fc0e7000640 1 --2- 172.21.15.27:0/3376541988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e00c92b0 0x7fc0e006dd00 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7fc0e00681f0 tx=0x7fc0c801c490 comp rx=0 tx=0).stop 2024-01-15T18:23:25.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.993+0000 7fc0e7000640 1 -- 172.21.15.27:0/3376541988 shutdown_connections 2024-01-15T18:23:25.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.993+0000 7fc0e7000640 1 --2- 172.21.15.27:0/3376541988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e00c92b0 0x7fc0e006dd00 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.993+0000 7fc0e7000640 1 --2- 172.21.15.27:0/3376541988 >> v2:172.21.15.93:3300/0 conn(0x7fc0e00701a0 0x7fc0e0070580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.993+0000 7fc0e7000640 1 -- 172.21.15.27:0/3376541988 >> 172.21.15.27:0/3376541988 conn(0x7fc0e00cec60 msgr2=0x7fc0e00cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:25.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.993+0000 7fc0e7000640 1 -- 172.21.15.27:0/3376541988 shutdown_connections 2024-01-15T18:23:25.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.993+0000 7fc0e7000640 1 -- 172.21.15.27:0/3376541988 wait complete. 2024-01-15T18:23:25.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.997+0000 7fc0e7000640 1 Processor -- start 2024-01-15T18:23:25.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.997+0000 7fc0e7000640 1 -- start start 2024-01-15T18:23:25.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.997+0000 7fc0e7000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc0e006e240 0x7fc0e014ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:25.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.997+0000 7fc0e7000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e00701a0 0x7fc0e00cc830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:25.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.997+0000 7fc0e7000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc0e00c92b0 0x7fc0e0159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:25.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.997+0000 7fc0e7000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e0158ae0 con 0x7fc0e00701a0 2024-01-15T18:23:25.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.997+0000 7fc0e7000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e00ad440 con 0x7fc0e00c92b0 2024-01-15T18:23:25.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.997+0000 7fc0e7000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e00ad720 con 0x7fc0e006e240 2024-01-15T18:23:25.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.997+0000 7fc0df400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e00701a0 0x7fc0e00cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:25.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.997+0000 7fc0df400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e00701a0 0x7fc0e00cc830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50186/0 (socket says 172.21.15.27:50186) 2024-01-15T18:23:25.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.997+0000 7fc0df400640 1 -- 172.21.15.27:0/411768825 learned_addr learned my addr 172.21.15.27:0/411768825 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:25.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.997+0000 7fc0dfe00640 1 --2- 172.21.15.27:0/411768825 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc0e006e240 0x7fc0e014ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:25.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.997+0000 7fc0e5000640 1 --2- 172.21.15.27:0/411768825 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc0e00c92b0 0x7fc0e0159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:25.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.997+0000 7fc0df400640 1 -- 172.21.15.27:0/411768825 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc0e006e240 msgr2=0x7fc0e014ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:25.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.997+0000 7fc0df400640 1 --2- 172.21.15.27:0/411768825 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc0e006e240 0x7fc0e014ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.997+0000 7fc0df400640 1 -- 172.21.15.27:0/411768825 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc0e00c92b0 msgr2=0x7fc0e0159250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:25.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.997+0000 7fc0df400640 1 --2- 172.21.15.27:0/411768825 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc0e00c92b0 0x7fc0e0159250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:25.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.997+0000 7fc0df400640 1 -- 172.21.15.27:0/411768825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c80094d0 con 0x7fc0e00701a0 2024-01-15T18:23:25.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.997+0000 7fc0df400640 1 --2- 172.21.15.27:0/411768825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e00701a0 0x7fc0e00cc830 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7fc0c801a0c0 tx=0x7fc0c801acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:25.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.997+0000 7fc0d7e00640 1 -- 172.21.15.27:0/411768825 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c80b3800 con 0x7fc0e00701a0 2024-01-15T18:23:25.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.997+0000 7fc0d7e00640 1 -- 172.21.15.27:0/411768825 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c80b3980 con 0x7fc0e00701a0 2024-01-15T18:23:25.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.997+0000 7fc0e7000640 1 -- 172.21.15.27:0/411768825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0e0158d30 con 0x7fc0e00701a0 2024-01-15T18:23:25.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.997+0000 7fc0d7e00640 1 -- 172.21.15.27:0/411768825 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c80b3d30 con 0x7fc0e00701a0 2024-01-15T18:23:25.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:25.997+0000 7fc0e7000640 1 -- 172.21.15.27:0/411768825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0e014ba10 con 0x7fc0e00701a0 2024-01-15T18:23:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.001+0000 7fc0e7000640 1 -- 172.21.15.27:0/411768825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0e004e850 con 0x7fc0e00701a0 2024-01-15T18:23:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.001+0000 7fc0d7e00640 1 -- 172.21.15.27:0/411768825 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7fc0c80b2380 con 0x7fc0e00701a0 2024-01-15T18:23:25.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.001+0000 7fc0d7e00640 1 --2- 172.21.15.27:0/411768825 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc0b8067a60 0x7fc0b8069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:25.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.001+0000 7fc0d7e00640 1 -- 172.21.15.27:0/411768825 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fc0c80a7070 con 0x7fc0e00701a0 2024-01-15T18:23:25.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.001+0000 7fc0dfe00640 1 --2- 172.21.15.27:0/411768825 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc0b8067a60 0x7fc0b8069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:25.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.001+0000 7fc0dfe00640 1 --2- 172.21.15.27:0/411768825 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc0b8067a60 0x7fc0b8069f20 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fc0d000b9d0 tx=0x7fc0d0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:26.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.005+0000 7fc0d7e00640 1 -- 172.21.15.27:0/411768825 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc0c80ec1f0 con 0x7fc0e00701a0 2024-01-15T18:23:26.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.165+0000 7fc0e7000640 1 -- 172.21.15.27:0/411768825 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0e00ca640 con 0x7fc0e00701a0 2024-01-15T18:23:26.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.169+0000 7fc0d7e00640 1 -- 172.21.15.27:0/411768825 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16761 (secure 0 0 0) 0x7fc0c80ef810 con 0x7fc0e00701a0 2024-01-15T18:23:26.165 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:23:26.165 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":110,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:18:42.250892+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:18:30.993882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-15T18:18:31.582649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.5","mappings":[{"from":0,"to":5}]},{"pgid":"17.18","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:23:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.169+0000 7fc0e7000640 1 -- 172.21.15.27:0/411768825 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc0b8067a60 msgr2=0x7fc0b8069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.169+0000 7fc0e7000640 1 --2- 172.21.15.27:0/411768825 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc0b8067a60 0x7fc0b8069f20 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fc0d000b9d0 tx=0x7fc0d0008040 comp rx=0 tx=0).stop 2024-01-15T18:23:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.169+0000 7fc0e7000640 1 -- 172.21.15.27:0/411768825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e00701a0 msgr2=0x7fc0e00cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.169+0000 7fc0e7000640 1 --2- 172.21.15.27:0/411768825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e00701a0 0x7fc0e00cc830 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7fc0c801a0c0 tx=0x7fc0c801acd0 comp rx=0 tx=0).stop 2024-01-15T18:23:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.169+0000 7fc0e7000640 1 -- 172.21.15.27:0/411768825 shutdown_connections 2024-01-15T18:23:26.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.169+0000 7fc0e7000640 1 --2- 172.21.15.27:0/411768825 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc0b8067a60 0x7fc0b8069f20 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:26.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.169+0000 7fc0e7000640 1 --2- 172.21.15.27:0/411768825 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc0e00c92b0 0x7fc0e0159250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:26.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.169+0000 7fc0e7000640 1 --2- 172.21.15.27:0/411768825 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e00701a0 0x7fc0e00cc830 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:26.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.169+0000 7fc0e7000640 1 --2- 172.21.15.27:0/411768825 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc0e006e240 0x7fc0e014ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:26.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.169+0000 7fc0e7000640 1 -- 172.21.15.27:0/411768825 >> 172.21.15.27:0/411768825 conn(0x7fc0e00cec60 msgr2=0x7fc0e00afb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:26.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.169+0000 7fc0e7000640 1 -- 172.21.15.27:0/411768825 shutdown_connections 2024-01-15T18:23:26.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.169+0000 7fc0e7000640 1 -- 172.21.15.27:0/411768825 wait complete. 2024-01-15T18:23:26.187 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T18:04:44.196146+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T18:18:30.993882+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T18:18:31.582649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:23:26.187 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:23:26.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.337+0000 7f44c1e00640 1 Processor -- start 2024-01-15T18:23:26.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.337+0000 7f44c1e00640 1 -- start start 2024-01-15T18:23:26.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.337+0000 7f44c1e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f44bc15b350 0x7f44bc0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:26.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.337+0000 7f44c1e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44bc09a5b0 0x7f44bc0cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:26.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.337+0000 7f44c1e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44bc0cfec0 con 0x7f44bc099a10 2024-01-15T18:23:26.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.337+0000 7f44c1e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44bc0ccd50 con 0x7f44bc15b350 2024-01-15T18:23:26.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.337+0000 7f44c1e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44bc0cebc0 con 0x7f44bc09a5b0 2024-01-15T18:23:26.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.337+0000 7f44ba000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44bc09a5b0 0x7f44bc0cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:26.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.337+0000 7f44baa00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f44bc15b350 0x7f44bc0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:26.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.337+0000 7f44ba000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44bc09a5b0 0x7f44bc0cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:33904/0 (socket says 172.21.15.27:33904) 2024-01-15T18:23:26.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.337+0000 7f44ba000640 1 -- 172.21.15.27:0/2416496229 learned_addr learned my addr 172.21.15.27:0/2416496229 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:26.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.337+0000 7f44ba000640 1 -- 172.21.15.27:0/2416496229 >> v2:172.21.15.93:3300/0 conn(0x7f44bc15b350 msgr2=0x7f44bc0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:26.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.337+0000 7f44ba000640 1 --2- 172.21.15.27:0/2416496229 >> v2:172.21.15.93:3300/0 conn(0x7f44bc15b350 0x7f44bc0cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:26.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.337+0000 7f44ba000640 1 -- 172.21.15.27:0/2416496229 >> v1:172.21.15.93:6789/0 conn(0x7f44bc099a10 legacy=0x7f44bc0b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:26.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.337+0000 7f44ba000640 1 -- 172.21.15.27:0/2416496229 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44bc0ceea0 con 0x7f44bc09a5b0 2024-01-15T18:23:26.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.341+0000 7f44ba000640 1 --2- 172.21.15.27:0/2416496229 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44bc09a5b0 0x7f44bc0cc500 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f44a4009870 tx=0x7f44a401c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6aa44dc1e74a75b6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:26.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.341+0000 7f44b9600640 1 -- 172.21.15.27:0/2416496229 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a40ab680 con 0x7f44bc09a5b0 2024-01-15T18:23:26.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.341+0000 7f44b9600640 1 -- 172.21.15.27:0/2416496229 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a40ab7e0 con 0x7f44bc09a5b0 2024-01-15T18:23:26.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.341+0000 7f44b9600640 1 -- 172.21.15.27:0/2416496229 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a40abb70 con 0x7f44bc09a5b0 2024-01-15T18:23:26.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.341+0000 7f44c1e00640 1 -- 172.21.15.27:0/2416496229 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44bc09a5b0 msgr2=0x7f44bc0cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:26.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.341+0000 7f44c1e00640 1 --2- 172.21.15.27:0/2416496229 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44bc09a5b0 0x7f44bc0cc500 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f44a4009870 tx=0x7f44a401c480 comp rx=0 tx=0).stop 2024-01-15T18:23:26.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.341+0000 7f44c1e00640 1 -- 172.21.15.27:0/2416496229 shutdown_connections 2024-01-15T18:23:26.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.341+0000 7f44c1e00640 1 --2- 172.21.15.27:0/2416496229 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44bc09a5b0 0x7f44bc0cc500 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:26.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.341+0000 7f44c1e00640 1 --2- 172.21.15.27:0/2416496229 >> v2:172.21.15.93:3300/0 conn(0x7f44bc15b350 0x7f44bc0cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:26.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.341+0000 7f44c1e00640 1 -- 172.21.15.27:0/2416496229 >> 172.21.15.27:0/2416496229 conn(0x7f44bc0b2a80 msgr2=0x7f44bc0b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:26.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.341+0000 7f44c1e00640 1 -- 172.21.15.27:0/2416496229 shutdown_connections 2024-01-15T18:23:26.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.341+0000 7f44c1e00640 1 -- 172.21.15.27:0/2416496229 wait complete. 2024-01-15T18:23:26.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44c1e00640 1 Processor -- start 2024-01-15T18:23:26.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44c1e00640 1 -- start start 2024-01-15T18:23:26.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44c1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44bc099a10 0x7f44bc148220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:26.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44c1e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44bc09a5b0 0x7f44bc150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:26.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44c1e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44bc15b350 0x7f44bc1461d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:26.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44c1e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44bc147d90 con 0x7f44bc099a10 2024-01-15T18:23:26.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44c1e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44bc147180 con 0x7f44bc09a5b0 2024-01-15T18:23:26.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44c1e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44bc147460 con 0x7f44bc15b350 2024-01-15T18:23:26.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44baa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44bc099a10 0x7f44bc148220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:26.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44baa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44bc099a10 0x7f44bc148220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50188/0 (socket says 172.21.15.27:50188) 2024-01-15T18:23:26.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44baa00640 1 -- 172.21.15.27:0/603045683 learned_addr learned my addr 172.21.15.27:0/603045683 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:26.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44ba000640 1 --2- 172.21.15.27:0/603045683 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44bc09a5b0 0x7f44bc150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:26.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44baa00640 1 -- 172.21.15.27:0/603045683 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44bc15b350 msgr2=0x7f44bc1461d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:26.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44bb400640 1 --2- 172.21.15.27:0/603045683 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44bc15b350 0x7f44bc1461d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:26.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44baa00640 1 --2- 172.21.15.27:0/603045683 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44bc15b350 0x7f44bc1461d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:26.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44baa00640 1 -- 172.21.15.27:0/603045683 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44bc09a5b0 msgr2=0x7f44bc150950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:26.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44baa00640 1 --2- 172.21.15.27:0/603045683 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44bc09a5b0 0x7f44bc150950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:26.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44baa00640 1 -- 172.21.15.27:0/603045683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a40094d0 con 0x7f44bc099a10 2024-01-15T18:23:26.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44ba000640 1 --2- 172.21.15.27:0/603045683 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44bc09a5b0 0x7f44bc150950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:23:26.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44bb400640 1 --2- 172.21.15.27:0/603045683 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44bc15b350 0x7f44bc1461d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:23:26.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44baa00640 1 --2- 172.21.15.27:0/603045683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44bc099a10 0x7f44bc148220 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f44ac00e870 tx=0x7f44ac00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:26.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44b3400640 1 -- 172.21.15.27:0/603045683 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ac009510 con 0x7f44bc099a10 2024-01-15T18:23:26.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44b3400640 1 -- 172.21.15.27:0/603045683 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44ac00eec0 con 0x7f44bc099a10 2024-01-15T18:23:26.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44b3400640 1 -- 172.21.15.27:0/603045683 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ac00cb40 con 0x7f44bc099a10 2024-01-15T18:23:26.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44c1e00640 1 -- 172.21.15.27:0/603045683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44bc09a3b0 con 0x7f44bc099a10 2024-01-15T18:23:26.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.345+0000 7f44c1e00640 1 -- 172.21.15.27:0/603045683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44bc147a20 con 0x7f44bc099a10 2024-01-15T18:23:26.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.349+0000 7f44c1e00640 1 -- 172.21.15.27:0/603045683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44bc0d0720 con 0x7f44bc099a10 2024-01-15T18:23:26.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.349+0000 7f44b3400640 1 -- 172.21.15.27:0/603045683 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f44ac00cca0 con 0x7f44bc099a10 2024-01-15T18:23:26.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.349+0000 7f44b3400640 1 --2- 172.21.15.27:0/603045683 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4494067a60 0x7f4494069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:26.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.349+0000 7f44b3400640 1 -- 172.21.15.27:0/603045683 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f44ac083840 con 0x7f44bc099a10 2024-01-15T18:23:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.349+0000 7f44ba000640 1 --2- 172.21.15.27:0/603045683 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4494067a60 0x7f4494069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.353+0000 7f44ba000640 1 --2- 172.21.15.27:0/603045683 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4494067a60 0x7f4494069f20 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f44a40af000 tx=0x7f44a4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:26.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.353+0000 7f44b3400640 1 -- 172.21.15.27:0/603045683 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f44ac052180 con 0x7f44bc099a10 2024-01-15T18:23:26.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.513+0000 7f44c1e00640 1 -- 172.21.15.27:0/603045683 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f44bc04e850 con 0x7f44bc099a10 2024-01-15T18:23:26.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.513+0000 7f44b3400640 1 -- 172.21.15.27:0/603045683 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v110) v1 ==== 93+0+10 (secure 0 0 0) 0x7f44ac0170a0 con 0x7f44bc099a10 2024-01-15T18:23:26.509 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-01-15T18:23:26.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.513+0000 7f44c1e00640 1 -- 172.21.15.27:0/603045683 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4494067a60 msgr2=0x7f4494069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:26.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.513+0000 7f44c1e00640 1 --2- 172.21.15.27:0/603045683 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4494067a60 0x7f4494069f20 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f44a40af000 tx=0x7f44a4008040 comp rx=0 tx=0).stop 2024-01-15T18:23:26.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.513+0000 7f44c1e00640 1 -- 172.21.15.27:0/603045683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44bc099a10 msgr2=0x7f44bc148220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:26.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.513+0000 7f44c1e00640 1 --2- 172.21.15.27:0/603045683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44bc099a10 0x7f44bc148220 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f44ac00e870 tx=0x7f44ac00ec70 comp rx=0 tx=0).stop 2024-01-15T18:23:26.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.513+0000 7f44c1e00640 1 -- 172.21.15.27:0/603045683 shutdown_connections 2024-01-15T18:23:26.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.513+0000 7f44c1e00640 1 --2- 172.21.15.27:0/603045683 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4494067a60 0x7f4494069f20 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:26.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.513+0000 7f44c1e00640 1 --2- 172.21.15.27:0/603045683 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f44bc15b350 0x7f44bc1461d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:26.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.513+0000 7f44c1e00640 1 --2- 172.21.15.27:0/603045683 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f44bc09a5b0 0x7f44bc150950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:26.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.513+0000 7f44c1e00640 1 --2- 172.21.15.27:0/603045683 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44bc099a10 0x7f44bc148220 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:26.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.513+0000 7f44c1e00640 1 -- 172.21.15.27:0/603045683 >> 172.21.15.27:0/603045683 conn(0x7f44bc0b2a80 msgr2=0x7f44bc0cbb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:26.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.513+0000 7f44c1e00640 1 -- 172.21.15.27:0/603045683 shutdown_connections 2024-01-15T18:23:26.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.513+0000 7f44c1e00640 1 -- 172.21.15.27:0/603045683 wait complete. 2024-01-15T18:23:26.531 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T18:23:26.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.681+0000 7fc187600640 1 Processor -- start 2024-01-15T18:23:26.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.681+0000 7fc187600640 1 -- start start 2024-01-15T18:23:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.681+0000 7fc187600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc1800cef20 0x7fc1800ce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.681+0000 7fc187600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fc1800cca40 0x7fc1800d19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.681+0000 7fc187600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc18015b6b0 con 0x7fc1800cf300 2024-01-15T18:23:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.681+0000 7fc187600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1800cb9e0 con 0x7fc1800cca40 2024-01-15T18:23:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.681+0000 7fc187600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1800cbc80 con 0x7fc1800cef20 2024-01-15T18:23:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.681+0000 7fc17fe00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fc1800cca40 0x7fc1800d19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.681+0000 7fc184c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc1800cef20 0x7fc1800ce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.681+0000 7fc17fe00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fc1800cca40 0x7fc1800d19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:47788/0 (socket says 172.21.15.27:47788) 2024-01-15T18:23:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.681+0000 7fc17fe00640 1 -- 172.21.15.27:0/2010281355 learned_addr learned my addr 172.21.15.27:0/2010281355 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.681+0000 7fc184c00640 1 -- 172.21.15.27:0/2010281355 >> v2:172.21.15.93:3300/0 conn(0x7fc1800cca40 msgr2=0x7fc1800d19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.681+0000 7fc184c00640 1 --2- 172.21.15.27:0/2010281355 >> v2:172.21.15.93:3300/0 conn(0x7fc1800cca40 0x7fc1800d19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.681+0000 7fc184c00640 1 -- 172.21.15.27:0/2010281355 >> v1:172.21.15.93:6789/0 conn(0x7fc1800cf300 legacy=0x7fc1800b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.681+0000 7fc184c00640 1 -- 172.21.15.27:0/2010281355 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1800cfdb0 con 0x7fc1800cef20 2024-01-15T18:23:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.685+0000 7fc184c00640 1 --2- 172.21.15.27:0/2010281355 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc1800cef20 0x7fc1800ce2c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc1700097c0 tx=0x7fc17001c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa0b89d12edfddae server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.685+0000 7fc17f400640 1 -- 172.21.15.27:0/2010281355 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1700ab600 con 0x7fc1800cef20 2024-01-15T18:23:26.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.685+0000 7fc17f400640 1 -- 172.21.15.27:0/2010281355 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1700a7050 con 0x7fc1800cef20 2024-01-15T18:23:26.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.685+0000 7fc17f400640 1 -- 172.21.15.27:0/2010281355 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1700ab980 con 0x7fc1800cef20 2024-01-15T18:23:26.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.685+0000 7fc187600640 1 -- 172.21.15.27:0/2010281355 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc1800cef20 msgr2=0x7fc1800ce2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:26.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.685+0000 7fc187600640 1 --2- 172.21.15.27:0/2010281355 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc1800cef20 0x7fc1800ce2c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc1700097c0 tx=0x7fc17001c480 comp rx=0 tx=0).stop 2024-01-15T18:23:26.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.685+0000 7fc187600640 1 -- 172.21.15.27:0/2010281355 shutdown_connections 2024-01-15T18:23:26.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.685+0000 7fc187600640 1 --2- 172.21.15.27:0/2010281355 >> v2:172.21.15.93:3300/0 conn(0x7fc1800cca40 0x7fc1800d19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:26.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.685+0000 7fc187600640 1 --2- 172.21.15.27:0/2010281355 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc1800cef20 0x7fc1800ce2c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:26.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.685+0000 7fc187600640 1 -- 172.21.15.27:0/2010281355 >> 172.21.15.27:0/2010281355 conn(0x7fc1800d0ed0 msgr2=0x7fc1800d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:26.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.685+0000 7fc187600640 1 -- 172.21.15.27:0/2010281355 shutdown_connections 2024-01-15T18:23:26.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.685+0000 7fc187600640 1 -- 172.21.15.27:0/2010281355 wait complete. 2024-01-15T18:23:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.689+0000 7fc187600640 1 Processor -- start 2024-01-15T18:23:26.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.689+0000 7fc187600640 1 -- start start 2024-01-15T18:23:26.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.689+0000 7fc187600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc1800cca40 0x7fc180153890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:26.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.689+0000 7fc187600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1800cef20 0x7fc180156400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:26.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.689+0000 7fc187600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc1800cf300 0x7fc180164750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:26.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.689+0000 7fc187600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1801fae70 con 0x7fc1800cef20 2024-01-15T18:23:26.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.689+0000 7fc187600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1801f6960 con 0x7fc1800cca40 2024-01-15T18:23:26.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.689+0000 7fc187600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1801fa2f0 con 0x7fc1800cf300 2024-01-15T18:23:26.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.689+0000 7fc17fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1800cef20 0x7fc180156400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:26.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.689+0000 7fc17fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1800cef20 0x7fc180156400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50192/0 (socket says 172.21.15.27:50192) 2024-01-15T18:23:26.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.689+0000 7fc17fe00640 1 -- 172.21.15.27:0/936801789 learned_addr learned my addr 172.21.15.27:0/936801789 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:26.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.689+0000 7fc184c00640 1 --2- 172.21.15.27:0/936801789 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc1800cca40 0x7fc180153890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:26.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.689+0000 7fc185600640 1 --2- 172.21.15.27:0/936801789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc1800cf300 0x7fc180164750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:26.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.689+0000 7fc17fe00640 1 -- 172.21.15.27:0/936801789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc1800cf300 msgr2=0x7fc180164750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:26.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.689+0000 7fc17fe00640 1 --2- 172.21.15.27:0/936801789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc1800cf300 0x7fc180164750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:26.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.689+0000 7fc17fe00640 1 -- 172.21.15.27:0/936801789 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc1800cca40 msgr2=0x7fc180153890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:26.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.689+0000 7fc17fe00640 1 --2- 172.21.15.27:0/936801789 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc1800cca40 0x7fc180153890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:26.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.689+0000 7fc17fe00640 1 -- 172.21.15.27:0/936801789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc170009420 con 0x7fc1800cef20 2024-01-15T18:23:26.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.689+0000 7fc17fe00640 1 --2- 172.21.15.27:0/936801789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1800cef20 0x7fc180156400 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fc16800e890 tx=0x7fc16800ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:26.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.689+0000 7fc17cc00640 1 -- 172.21.15.27:0/936801789 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc168009620 con 0x7fc1800cef20 2024-01-15T18:23:26.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.689+0000 7fc17cc00640 1 -- 172.21.15.27:0/936801789 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc16800c940 con 0x7fc1800cef20 2024-01-15T18:23:26.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.689+0000 7fc187600640 1 -- 172.21.15.27:0/936801789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1801fa5d0 con 0x7fc1800cef20 2024-01-15T18:23:26.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.689+0000 7fc17cc00640 1 -- 172.21.15.27:0/936801789 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc16800ccf0 con 0x7fc1800cef20 2024-01-15T18:23:26.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.689+0000 7fc187600640 1 -- 172.21.15.27:0/936801789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1801fab90 con 0x7fc1800cef20 2024-01-15T18:23:26.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.693+0000 7fc187600640 1 -- 172.21.15.27:0/936801789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1800d09f0 con 0x7fc1800cef20 2024-01-15T18:23:26.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.693+0000 7fc17cc00640 1 -- 172.21.15.27:0/936801789 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7fc168013470 con 0x7fc1800cef20 2024-01-15T18:23:26.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.693+0000 7fc17cc00640 1 --2- 172.21.15.27:0/936801789 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc158067a60 0x7fc158069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:26.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.693+0000 7fc17cc00640 1 -- 172.21.15.27:0/936801789 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fc168083430 con 0x7fc1800cef20 2024-01-15T18:23:26.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.693+0000 7fc184c00640 1 --2- 172.21.15.27:0/936801789 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc158067a60 0x7fc158069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:26.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.693+0000 7fc184c00640 1 --2- 172.21.15.27:0/936801789 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc158067a60 0x7fc158069f20 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fc1700ad000 tx=0x7fc170004740 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:26.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.697+0000 7fc17cc00640 1 -- 172.21.15.27:0/936801789 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc168051e20 con 0x7fc1800cef20 2024-01-15T18:23:26.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.853+0000 7fc187600640 1 -- 172.21.15.27:0/936801789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc18004e850 con 0x7fc1800cef20 2024-01-15T18:23:26.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.853+0000 7fc17cc00640 1 -- 172.21.15.27:0/936801789 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v110) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc168021020 con 0x7fc1800cef20 2024-01-15T18:23:26.850 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-01-15T18:23:26.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.857+0000 7fc187600640 1 -- 172.21.15.27:0/936801789 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc158067a60 msgr2=0x7fc158069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:26.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.857+0000 7fc187600640 1 --2- 172.21.15.27:0/936801789 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc158067a60 0x7fc158069f20 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fc1700ad000 tx=0x7fc170004740 comp rx=0 tx=0).stop 2024-01-15T18:23:26.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.857+0000 7fc187600640 1 -- 172.21.15.27:0/936801789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1800cef20 msgr2=0x7fc180156400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:26.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.857+0000 7fc187600640 1 --2- 172.21.15.27:0/936801789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1800cef20 0x7fc180156400 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fc16800e890 tx=0x7fc16800ed50 comp rx=0 tx=0).stop 2024-01-15T18:23:26.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.857+0000 7fc187600640 1 -- 172.21.15.27:0/936801789 shutdown_connections 2024-01-15T18:23:26.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.857+0000 7fc187600640 1 --2- 172.21.15.27:0/936801789 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fc158067a60 0x7fc158069f20 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:26.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.857+0000 7fc187600640 1 --2- 172.21.15.27:0/936801789 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fc1800cf300 0x7fc180164750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:26.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.857+0000 7fc187600640 1 --2- 172.21.15.27:0/936801789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc1800cef20 0x7fc180156400 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:26.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.857+0000 7fc187600640 1 --2- 172.21.15.27:0/936801789 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fc1800cca40 0x7fc180153890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:26.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.857+0000 7fc187600640 1 -- 172.21.15.27:0/936801789 >> 172.21.15.27:0/936801789 conn(0x7fc1800d0ed0 msgr2=0x7fc1800cdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:26.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.857+0000 7fc187600640 1 -- 172.21.15.27:0/936801789 shutdown_connections 2024-01-15T18:23:26.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:26.857+0000 7fc187600640 1 -- 172.21.15.27:0/936801789 wait complete. 2024-01-15T18:23:26.872 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T18:23:27.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.021+0000 7fb601200640 1 Processor -- start 2024-01-15T18:23:27.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.021+0000 7fb601200640 1 -- start start 2024-01-15T18:23:27.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.021+0000 7fb601200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb5fc15d570 0x7fb5fc06be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:27.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.021+0000 7fb601200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5fc0b0d70 con 0x7fb5fc15d190 2024-01-15T18:23:27.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.021+0000 7fb601200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5fc0d0540 con 0x7fb5fc06b810 2024-01-15T18:23:27.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.021+0000 7fb601200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5fc0b0440 con 0x7fb5fc15d570 2024-01-15T18:23:27.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.021+0000 7fb5f9400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb5fc15d570 0x7fb5fc06be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:27.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.021+0000 7fb5f9e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb5fc15d190 0x7fb5fc0cfcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37166/0 (socket says 172.21.15.27:37166) 2024-01-15T18:23:27.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.021+0000 7fb5f9e00640 1 -- 172.21.15.27:0/1646192536 learned_addr learned my addr 172.21.15.27:0/1646192536 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:27.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.021+0000 7fb5f9400640 1 -- 172.21.15.27:0/1646192536 >> v1:172.21.15.93:6789/0 conn(0x7fb5fc06b810 legacy=0x7fb5fc0cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:27.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.021+0000 7fb5f9400640 1 -- 172.21.15.27:0/1646192536 >> v1:172.21.15.27:6789/0 conn(0x7fb5fc15d190 legacy=0x7fb5fc0cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:27.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.021+0000 7fb5f9400640 1 -- 172.21.15.27:0/1646192536 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5fc0b0720 con 0x7fb5fc15d570 2024-01-15T18:23:27.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.025+0000 7fb5f9400640 1 --2- 172.21.15.27:0/1646192536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5fc15d570 0x7fb5fc06be10 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7fb5e40097c0 tx=0x7fb5e401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfc8f5d485e9987 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:27.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.025+0000 7fb5f8a00640 1 -- 172.21.15.27:0/1646192536 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e40ab670 con 0x7fb5fc15d570 2024-01-15T18:23:27.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.025+0000 7fb5f8a00640 1 -- 172.21.15.27:0/1646192536 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e40ab7d0 con 0x7fb5fc15d570 2024-01-15T18:23:27.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.025+0000 7fb5f8a00640 1 -- 172.21.15.27:0/1646192536 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e40abb60 con 0x7fb5fc15d570 2024-01-15T18:23:27.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.025+0000 7fb601200640 1 -- 172.21.15.27:0/1646192536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5fc15d570 msgr2=0x7fb5fc06be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:27.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.025+0000 7fb601200640 1 --2- 172.21.15.27:0/1646192536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5fc15d570 0x7fb5fc06be10 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7fb5e40097c0 tx=0x7fb5e401c450 comp rx=0 tx=0).stop 2024-01-15T18:23:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.025+0000 7fb601200640 1 -- 172.21.15.27:0/1646192536 shutdown_connections 2024-01-15T18:23:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.025+0000 7fb601200640 1 --2- 172.21.15.27:0/1646192536 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5fc15d570 0x7fb5fc06be10 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.025+0000 7fb601200640 1 -- 172.21.15.27:0/1646192536 >> 172.21.15.27:0/1646192536 conn(0x7fb5fc0d0e70 msgr2=0x7fb5fc0d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.025+0000 7fb601200640 1 -- 172.21.15.27:0/1646192536 shutdown_connections 2024-01-15T18:23:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.025+0000 7fb601200640 1 -- 172.21.15.27:0/1646192536 wait complete. 2024-01-15T18:23:27.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.029+0000 7fb601200640 1 Processor -- start 2024-01-15T18:23:27.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.029+0000 7fb601200640 1 -- start start 2024-01-15T18:23:27.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.029+0000 7fb601200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb5fc06b810 0x7fb5fc14a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:27.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.029+0000 7fb601200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5fc15d190 0x7fb5fc0b5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:27.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.029+0000 7fb601200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb5fc15d570 0x7fb5fc0b5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:27.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.029+0000 7fb601200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc0b6a30 con 0x7fb5fc15d190 2024-01-15T18:23:27.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.029+0000 7fb601200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc0b5eb0 con 0x7fb5fc15d570 2024-01-15T18:23:27.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.029+0000 7fb601200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc14a940 con 0x7fb5fc06b810 2024-01-15T18:23:27.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.029+0000 7fb5f9400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5fc15d190 0x7fb5fc0b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:27.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.029+0000 7fb5f9400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5fc15d190 0x7fb5fc0b5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50212/0 (socket says 172.21.15.27:50212) 2024-01-15T18:23:27.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.029+0000 7fb5f9400640 1 -- 172.21.15.27:0/2679373035 learned_addr learned my addr 172.21.15.27:0/2679373035 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:27.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.029+0000 7fb5f9e00640 1 --2- 172.21.15.27:0/2679373035 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb5fc06b810 0x7fb5fc14a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:27.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.029+0000 7fb5fa800640 1 --2- 172.21.15.27:0/2679373035 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb5fc15d570 0x7fb5fc0b5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:27.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.029+0000 7fb5f9400640 1 -- 172.21.15.27:0/2679373035 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb5fc06b810 msgr2=0x7fb5fc14a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:27.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.029+0000 7fb5f9400640 1 --2- 172.21.15.27:0/2679373035 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb5fc06b810 0x7fb5fc14a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:27.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.029+0000 7fb5f9400640 1 -- 172.21.15.27:0/2679373035 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb5fc15d570 msgr2=0x7fb5fc0b5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:27.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.029+0000 7fb5f9400640 1 --2- 172.21.15.27:0/2679373035 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb5fc15d570 0x7fb5fc0b5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:27.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.029+0000 7fb5f9400640 1 -- 172.21.15.27:0/2679373035 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e4009420 con 0x7fb5fc15d190 2024-01-15T18:23:27.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.029+0000 7fb5f9400640 1 --2- 172.21.15.27:0/2679373035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5fc15d190 0x7fb5fc0b5430 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4019060 tx=0x7fb5e40191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:27.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.029+0000 7fb5f2000640 1 -- 172.21.15.27:0/2679373035 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e40bf650 con 0x7fb5fc15d190 2024-01-15T18:23:27.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.033+0000 7fb5f2000640 1 -- 172.21.15.27:0/2679373035 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e40bf7d0 con 0x7fb5fc15d190 2024-01-15T18:23:27.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.033+0000 7fb601200640 1 -- 172.21.15.27:0/2679373035 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5fc15c9b0 con 0x7fb5fc15d190 2024-01-15T18:23:27.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.033+0000 7fb5f2000640 1 -- 172.21.15.27:0/2679373035 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e40bfb80 con 0x7fb5fc15d190 2024-01-15T18:23:27.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.033+0000 7fb601200640 1 -- 172.21.15.27:0/2679373035 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5fc0b63e0 con 0x7fb5fc15d190 2024-01-15T18:23:27.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.033+0000 7fb601200640 1 -- 172.21.15.27:0/2679373035 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5fc0b6be0 con 0x7fb5fc15d190 2024-01-15T18:23:27.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.033+0000 7fb5f2000640 1 -- 172.21.15.27:0/2679373035 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7fb5e40bfce0 con 0x7fb5fc15d190 2024-01-15T18:23:27.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.037+0000 7fb5f2000640 1 --2- 172.21.15.27:0/2679373035 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb5d4067a60 0x7fb5d4069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:27.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.037+0000 7fb5f2000640 1 -- 172.21.15.27:0/2679373035 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fb5e411dec0 con 0x7fb5fc15d190 2024-01-15T18:23:27.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.037+0000 7fb5f9e00640 1 --2- 172.21.15.27:0/2679373035 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb5d4067a60 0x7fb5d4069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:27.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.037+0000 7fb5f9e00640 1 --2- 172.21.15.27:0/2679373035 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb5d4067a60 0x7fb5d4069f20 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec005890 tx=0x7fb5ec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:27.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.041+0000 7fb5f2000640 1 -- 172.21.15.27:0/2679373035 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb5fc0b6be0 con 0x7fb5fc15d190 2024-01-15T18:23:27.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.193+0000 7fb601200640 1 -- 172.21.15.27:0/2679373035 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb5fc0b6be0 con 0x7fb5fc15d190 2024-01-15T18:23:27.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.193+0000 7fb5f2000640 1 -- 172.21.15.27:0/2679373035 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v110) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb5fc0b6be0 con 0x7fb5fc15d190 2024-01-15T18:23:27.191 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2024-01-15T18:23:27.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.197+0000 7fb601200640 1 -- 172.21.15.27:0/2679373035 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb5d4067a60 msgr2=0x7fb5d4069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:27.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.197+0000 7fb601200640 1 --2- 172.21.15.27:0/2679373035 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb5d4067a60 0x7fb5d4069f20 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec005890 tx=0x7fb5ec009040 comp rx=0 tx=0).stop 2024-01-15T18:23:27.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.197+0000 7fb601200640 1 -- 172.21.15.27:0/2679373035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5fc15d190 msgr2=0x7fb5fc0b5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:27.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.197+0000 7fb601200640 1 --2- 172.21.15.27:0/2679373035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5fc15d190 0x7fb5fc0b5430 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4019060 tx=0x7fb5e40191f0 comp rx=0 tx=0).stop 2024-01-15T18:23:27.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.197+0000 7fb601200640 1 -- 172.21.15.27:0/2679373035 shutdown_connections 2024-01-15T18:23:27.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.197+0000 7fb601200640 1 --2- 172.21.15.27:0/2679373035 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb5d4067a60 0x7fb5d4069f20 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:27.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.197+0000 7fb601200640 1 --2- 172.21.15.27:0/2679373035 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb5fc15d570 0x7fb5fc0b5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:27.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.197+0000 7fb601200640 1 --2- 172.21.15.27:0/2679373035 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5fc15d190 0x7fb5fc0b5430 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:27.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.197+0000 7fb601200640 1 --2- 172.21.15.27:0/2679373035 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb5fc06b810 0x7fb5fc14a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:27.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.197+0000 7fb601200640 1 -- 172.21.15.27:0/2679373035 >> 172.21.15.27:0/2679373035 conn(0x7fb5fc0d0e70 msgr2=0x7fb5fc0aef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:27.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.197+0000 7fb601200640 1 -- 172.21.15.27:0/2679373035 shutdown_connections 2024-01-15T18:23:27.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.197+0000 7fb601200640 1 -- 172.21.15.27:0/2679373035 wait complete. 2024-01-15T18:23:27.213 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:23:27.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f2506c00640 1 Processor -- start 2024-01-15T18:23:27.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f2506c00640 1 -- start start 2024-01-15T18:23:27.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f2506c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f250015b180 0x7f25000cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:27.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f2506c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25000ccdd0 con 0x7f25000cc9a0 2024-01-15T18:23:27.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f2506c00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25000ce7d0 con 0x7f25000cd7e0 2024-01-15T18:23:27.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f2506c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25000d1e20 con 0x7f250015b180 2024-01-15T18:23:27.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f2504c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f25000cc9a0 0x7f25000b10e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:37172/0 (socket says 172.21.15.27:37172) 2024-01-15T18:23:27.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f2504c00640 1 -- 172.21.15.27:0/4038680714 learned_addr learned my addr 172.21.15.27:0/4038680714 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:27.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f24ffe00640 1 --2- 172.21.15.27:0/4038680714 >> v2:172.21.15.93:3300/0 conn(0x7f250015b180 0x7f25000cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:27.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f24fea00640 1 -- 172.21.15.27:0/4038680714 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 78780385 0 0) 0x7f25000ccdd0 con 0x7f25000cc9a0 2024-01-15T18:23:27.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f24fea00640 1 -- 172.21.15.27:0/4038680714 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24e0003530 con 0x7f25000cc9a0 2024-01-15T18:23:27.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f24fea00640 1 -- 172.21.15.27:0/4038680714 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4085624282 0 0) 0x7f24e0003530 con 0x7f25000cc9a0 2024-01-15T18:23:27.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f24fea00640 1 -- 172.21.15.27:0/4038680714 >> v2:172.21.15.93:3300/0 conn(0x7f250015b180 msgr2=0x7f25000cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:27.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f24fea00640 1 --2- 172.21.15.27:0/4038680714 >> v2:172.21.15.93:3300/0 conn(0x7f250015b180 0x7f25000cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:27.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f24fea00640 1 -- 172.21.15.27:0/4038680714 >> v1:172.21.15.93:6789/0 conn(0x7f25000cd7e0 legacy=0x7f25000b09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:27.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f24fea00640 1 -- 172.21.15.27:0/4038680714 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25000d1b40 con 0x7f25000cc9a0 2024-01-15T18:23:27.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f24fea00640 1 -- 172.21.15.27:0/4038680714 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7f24f8001a00 con 0x7f25000cc9a0 2024-01-15T18:23:27.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f24fea00640 1 -- 172.21.15.27:0/4038680714 >> v1:172.21.15.27:6789/0 conn(0x7f25000cc9a0 legacy=0x7f25000b10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:27.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f24fea00640 1 --2- 172.21.15.27:0/4038680714 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f24e0003a40 0x7f24e0003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:27.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f24fea00640 1 --2- 172.21.15.27:0/4038680714 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f24e00047d0 0x7f24e0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:27.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f24fea00640 1 --2- 172.21.15.27:0/4038680714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24e00051e0 0x7f24e000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:27.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f24fea00640 1 -- 172.21.15.27:0/4038680714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25000d1e20 con 0x7f24e00051e0 2024-01-15T18:23:27.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f24fea00640 1 -- 172.21.15.27:0/4038680714 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e000c110 con 0x7f24e0003a40 2024-01-15T18:23:27.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f24fea00640 1 -- 172.21.15.27:0/4038680714 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e000c2d0 con 0x7f24e00047d0 2024-01-15T18:23:27.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f2504c00640 1 --2- 172.21.15.27:0/4038680714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24e00051e0 0x7f24e000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:27.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f24ffe00640 1 --2- 172.21.15.27:0/4038680714 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f24e0003a40 0x7f24e0003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:27.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f24ff400640 1 --2- 172.21.15.27:0/4038680714 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f24e00047d0 0x7f24e0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:27.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f2504c00640 1 -- 172.21.15.27:0/4038680714 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f24e00047d0 msgr2=0x7f24e0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:27.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f2504c00640 1 --2- 172.21.15.27:0/4038680714 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f24e00047d0 0x7f24e0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:27.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f2504c00640 1 -- 172.21.15.27:0/4038680714 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f24e0003a40 msgr2=0x7f24e0003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f2504c00640 1 --2- 172.21.15.27:0/4038680714 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f24e0003a40 0x7f24e0003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:27.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.361+0000 7f2504c00640 1 -- 172.21.15.27:0/4038680714 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e0003530 con 0x7f24e00051e0 2024-01-15T18:23:27.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.365+0000 7f2504c00640 1 -- 172.21.15.27:0/4038680714 reap_dead start 2024-01-15T18:23:27.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.365+0000 7f2504c00640 1 --2- 172.21.15.27:0/4038680714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24e00051e0 0x7f24e000b990 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f24f800a760 tx=0x7f24f801d260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a5a676c2110309d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:27.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.365+0000 7f24fea00640 1 -- 172.21.15.27:0/4038680714 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f80ad630 con 0x7f24e00051e0 2024-01-15T18:23:27.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.365+0000 7f24fea00640 1 -- 172.21.15.27:0/4038680714 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f80a8070 con 0x7f24e00051e0 2024-01-15T18:23:27.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.365+0000 7f24fea00640 1 -- 172.21.15.27:0/4038680714 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f80ada10 con 0x7f24e00051e0 2024-01-15T18:23:27.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.365+0000 7f2506c00640 1 -- 172.21.15.27:0/4038680714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24e00051e0 msgr2=0x7f24e000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:27.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.365+0000 7f2506c00640 1 --2- 172.21.15.27:0/4038680714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24e00051e0 0x7f24e000b990 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f24f800a760 tx=0x7f24f801d260 comp rx=0 tx=0).stop 2024-01-15T18:23:27.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.365+0000 7f2506c00640 1 -- 172.21.15.27:0/4038680714 shutdown_connections 2024-01-15T18:23:27.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.365+0000 7f2506c00640 1 --2- 172.21.15.27:0/4038680714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24e00051e0 0x7f24e000b990 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:27.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.365+0000 7f2506c00640 1 -- 172.21.15.27:0/4038680714 >> 172.21.15.27:0/4038680714 conn(0x7f25000d0e80 msgr2=0x7f25000d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:27.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.365+0000 7f2506c00640 1 -- 172.21.15.27:0/4038680714 shutdown_connections 2024-01-15T18:23:27.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.365+0000 7f2506c00640 1 -- 172.21.15.27:0/4038680714 wait complete. 2024-01-15T18:23:27.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.369+0000 7f2506c00640 1 Processor -- start 2024-01-15T18:23:27.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.369+0000 7f2506c00640 1 -- start start 2024-01-15T18:23:27.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.369+0000 7f2506c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f24e00051e0 0x7f25001e4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:27.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.369+0000 7f2506c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f25001e74e0 0x7f2500154f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:27.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.369+0000 7f2506c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25000b0900 0x7f25000b0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:27.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.369+0000 7f2506c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25000ceed0 con 0x7f25000b0900 2024-01-15T18:23:27.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.369+0000 7f2506c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25001e5030 con 0x7f24e00051e0 2024-01-15T18:23:27.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.369+0000 7f2506c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25001e5e30 con 0x7f25001e74e0 2024-01-15T18:23:27.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.369+0000 7f2504c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25000b0900 0x7f25000b0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:27.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.369+0000 7f2504c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25000b0900 0x7f25000b0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50232/0 (socket says 172.21.15.27:50232) 2024-01-15T18:23:27.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.369+0000 7f2504c00640 1 -- 172.21.15.27:0/1789236774 learned_addr learned my addr 172.21.15.27:0/1789236774 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:27.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.369+0000 7f24ffe00640 1 --2- 172.21.15.27:0/1789236774 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f24e00051e0 0x7f25001e4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:27.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.369+0000 7f24ff400640 1 --2- 172.21.15.27:0/1789236774 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f25001e74e0 0x7f2500154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:27.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.369+0000 7f2504c00640 1 -- 172.21.15.27:0/1789236774 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f25001e74e0 msgr2=0x7f2500154f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:27.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.369+0000 7f2504c00640 1 --2- 172.21.15.27:0/1789236774 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f25001e74e0 0x7f2500154f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:27.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.369+0000 7f2504c00640 1 -- 172.21.15.27:0/1789236774 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f24e00051e0 msgr2=0x7f25001e4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:27.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.369+0000 7f2504c00640 1 --2- 172.21.15.27:0/1789236774 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f24e00051e0 0x7f25001e4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:27.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.369+0000 7f2504c00640 1 -- 172.21.15.27:0/1789236774 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25000b12a0 con 0x7f25000b0900 2024-01-15T18:23:27.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.369+0000 7f2504c00640 1 --2- 172.21.15.27:0/1789236774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25000b0900 0x7f25000b0d60 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f24f8002240 tx=0x7f24e0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:27.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.369+0000 7f24f7e00640 1 -- 172.21.15.27:0/1789236774 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f8001940 con 0x7f25000b0900 2024-01-15T18:23:27.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.369+0000 7f24f7e00640 1 -- 172.21.15.27:0/1789236774 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f8001aa0 con 0x7f25000b0900 2024-01-15T18:23:27.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.369+0000 7f2506c00640 1 -- 172.21.15.27:0/1789236774 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25001e5220 con 0x7f25000b0900 2024-01-15T18:23:27.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.369+0000 7f24f7e00640 1 -- 172.21.15.27:0/1789236774 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f80ad980 con 0x7f25000b0900 2024-01-15T18:23:27.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.373+0000 7f2506c00640 1 -- 172.21.15.27:0/1789236774 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25001e57e0 con 0x7f25000b0900 2024-01-15T18:23:27.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.373+0000 7f2506c00640 1 -- 172.21.15.27:0/1789236774 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f250015cae0 con 0x7f25000b0900 2024-01-15T18:23:27.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.373+0000 7f24f7e00640 1 -- 172.21.15.27:0/1789236774 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f24f80aa030 con 0x7f25000b0900 2024-01-15T18:23:27.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.373+0000 7f24f7e00640 1 --2- 172.21.15.27:0/1789236774 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f24d8067a60 0x7f24d8069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:27.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.373+0000 7f24f7e00640 1 -- 172.21.15.27:0/1789236774 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f24f80a5e40 con 0x7f25000b0900 2024-01-15T18:23:27.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.373+0000 7f24ffe00640 1 --2- 172.21.15.27:0/1789236774 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f24d8067a60 0x7f24d8069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:27.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.377+0000 7f24ffe00640 1 --2- 172.21.15.27:0/1789236774 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f24d8067a60 0x7f24d8069f20 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f24f0009aa0 tx=0x7f24f0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:27.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.377+0000 7f24f7e00640 1 -- 172.21.15.27:0/1789236774 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f24f80ebc00 con 0x7f25000b0900 2024-01-15T18:23:27.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.545+0000 7f2506c00640 1 -- 172.21.15.27:0/1789236774 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f250004e850 con 0x7f25000b0900 2024-01-15T18:23:27.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.549+0000 7f24f7e00640 1 -- 172.21.15.27:0/1789236774 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v110) v1 ==== 80+0+0 (secure 0 0 0) 0x7f24f80ef220 con 0x7f25000b0900 2024-01-15T18:23:27.544 INFO:teuthology.orchestra.run.smithi027.stderr:listed 0 entries 2024-01-15T18:23:27.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.549+0000 7f2506c00640 1 -- 172.21.15.27:0/1789236774 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f24d8067a60 msgr2=0x7f24d8069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:27.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.549+0000 7f2506c00640 1 --2- 172.21.15.27:0/1789236774 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f24d8067a60 0x7f24d8069f20 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f24f0009aa0 tx=0x7f24f0008040 comp rx=0 tx=0).stop 2024-01-15T18:23:27.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.549+0000 7f2506c00640 1 -- 172.21.15.27:0/1789236774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25000b0900 msgr2=0x7f25000b0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:27.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.549+0000 7f2506c00640 1 --2- 172.21.15.27:0/1789236774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25000b0900 0x7f25000b0d60 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f24f8002240 tx=0x7f24e0005800 comp rx=0 tx=0).stop 2024-01-15T18:23:27.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.549+0000 7f2506c00640 1 -- 172.21.15.27:0/1789236774 shutdown_connections 2024-01-15T18:23:27.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.549+0000 7f2506c00640 1 --2- 172.21.15.27:0/1789236774 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f24d8067a60 0x7f24d8069f20 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:27.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.549+0000 7f2506c00640 1 --2- 172.21.15.27:0/1789236774 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25000b0900 0x7f25000b0d60 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:27.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.549+0000 7f2506c00640 1 --2- 172.21.15.27:0/1789236774 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f25001e74e0 0x7f2500154f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:27.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.549+0000 7f2506c00640 1 --2- 172.21.15.27:0/1789236774 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f24e00051e0 0x7f25001e4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:27.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.549+0000 7f2506c00640 1 -- 172.21.15.27:0/1789236774 >> 172.21.15.27:0/1789236774 conn(0x7f25000d0e80 msgr2=0x7f250015a560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:27.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.549+0000 7f2506c00640 1 -- 172.21.15.27:0/1789236774 shutdown_connections 2024-01-15T18:23:27.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:27.549+0000 7f2506c00640 1 -- 172.21.15.27:0/1789236774 wait complete. 2024-01-15T18:23:27.566 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi093.front.sepia.ceph.com... 2024-01-15T18:23:27.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:23:27.566 DEBUG:teuthology.orchestra.run.smithi093:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-15T18:23:27.586 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi093.stderr:ceph-fuse[34808]: fuse finished with error 0 and tester_r 0 2024-01-15T18:23:27.589 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:23:28.663 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:23:28.667 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:23:33.690 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T18:23:33.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:23:33.692 DEBUG:teuthology.orchestra.run.smithi093:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-15T18:23:33.703 DEBUG:teuthology.orchestra.run.smithi093:> set -ex 2024-01-15T18:23:33.703 DEBUG:teuthology.orchestra.run.smithi093:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:23:33.756 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16354484k,nr_inodes=4088621,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=8403 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275340k,nr_inodes=818835,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275344k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:23:33.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:23:33.757 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-01-15T18:23:33.768 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-15T18:23:33.768 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-15T18:23:33.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:23:33.769 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:23:33.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:23:33.834 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:23:33.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:23:33.933 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link delete ceph-brx 2024-01-15T18:23:34.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:23:34.102 DEBUG:teuthology.orchestra.run.smithi093:> ip netns list 2024-01-15T18:23:34.112 INFO:teuthology.orchestra.run.smithi093.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-15T18:23:34.112 INFO:teuthology.orchestra.run.smithi093.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-15T18:23:34.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:23:34.113 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:23:34.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:23:34.177 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:23:34.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:23:34.251 DEBUG:teuthology.orchestra.run.smithi093:> sudo ip link delete ceph-brx 2024-01-15T18:23:34.474 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-15T18:23:34.484 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-15T18:23:34.484 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:23:34.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.629+0000 7f34f7600640 1 Processor -- start 2024-01-15T18:23:34.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.633+0000 7f34f7600640 1 -- start start 2024-01-15T18:23:34.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.633+0000 7f34f7600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f34f00afe70 0x7f34f00cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:34.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.633+0000 7f34f7600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f00d05b0 con 0x7f34f0160b10 2024-01-15T18:23:34.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.633+0000 7f34f7600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f00ce7d0 con 0x7f34f00cb7b0 2024-01-15T18:23:34.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.633+0000 7f34f7600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34f00cfc80 con 0x7f34f00afe70 2024-01-15T18:23:34.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.633+0000 7f34efe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f34f00afe70 0x7f34f00cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:34.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.633+0000 7f34f4c00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f34f0160b10 0x7f34f00cd7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42844/0 (socket says 172.21.15.27:42844) 2024-01-15T18:23:34.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.633+0000 7f34f4c00640 1 -- 172.21.15.27:0/1234979449 learned_addr learned my addr 172.21.15.27:0/1234979449 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:34.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.633+0000 7f34efe00640 1 -- 172.21.15.27:0/1234979449 >> v1:172.21.15.93:6789/0 conn(0x7f34f00cb7b0 legacy=0x7f34f00cbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:34.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.633+0000 7f34efe00640 1 -- 172.21.15.27:0/1234979449 >> v1:172.21.15.27:6789/0 conn(0x7f34f0160b10 legacy=0x7f34f00cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:34.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.633+0000 7f34efe00640 1 -- 172.21.15.27:0/1234979449 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f00cff60 con 0x7f34f00afe70 2024-01-15T18:23:34.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.637+0000 7f34efe00640 1 --2- 172.21.15.27:0/1234979449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f00afe70 0x7f34f00cdeb0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f34d80097c0 tx=0x7f34d801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c0dabde5ae8514f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:34.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.637+0000 7f34ef400640 1 -- 172.21.15.27:0/1234979449 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f00ccb20 con 0x7f34f00afe70 2024-01-15T18:23:34.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.637+0000 7f34ef400640 1 -- 172.21.15.27:0/1234979449 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d80ac640 con 0x7f34f00afe70 2024-01-15T18:23:34.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.637+0000 7f34ef400640 1 -- 172.21.15.27:0/1234979449 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d80ac9d0 con 0x7f34f00afe70 2024-01-15T18:23:34.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.637+0000 7f34f7600640 1 -- 172.21.15.27:0/1234979449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f00afe70 msgr2=0x7f34f00cdeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:34.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.637+0000 7f34f7600640 1 --2- 172.21.15.27:0/1234979449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f00afe70 0x7f34f00cdeb0 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f34d80097c0 tx=0x7f34d801c3c0 comp rx=0 tx=0).stop 2024-01-15T18:23:34.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.637+0000 7f34f7600640 1 -- 172.21.15.27:0/1234979449 shutdown_connections 2024-01-15T18:23:34.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.637+0000 7f34f7600640 1 --2- 172.21.15.27:0/1234979449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f00afe70 0x7f34f00cdeb0 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:34.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.637+0000 7f34f7600640 1 -- 172.21.15.27:0/1234979449 >> 172.21.15.27:0/1234979449 conn(0x7f34f00d0f20 msgr2=0x7f34f00d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:34.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.637+0000 7f34f7600640 1 -- 172.21.15.27:0/1234979449 shutdown_connections 2024-01-15T18:23:34.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.637+0000 7f34f7600640 1 -- 172.21.15.27:0/1234979449 wait complete. 2024-01-15T18:23:34.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.637+0000 7f34f7600640 1 Processor -- start 2024-01-15T18:23:34.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.637+0000 7f34f7600640 1 -- start start 2024-01-15T18:23:34.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.641+0000 7f34f7600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f00afe70 0x7f34f006f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:34.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.641+0000 7f34f7600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34f00cb7b0 0x7f34f0158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:34.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.641+0000 7f34f7600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f34f0160b10 0x7f34f0150790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:34.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.641+0000 7f34f7600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f01514c0 con 0x7f34f00afe70 2024-01-15T18:23:34.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.641+0000 7f34f7600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f0159330 con 0x7f34f0160b10 2024-01-15T18:23:34.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.641+0000 7f34f7600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f0159610 con 0x7f34f00cb7b0 2024-01-15T18:23:34.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.641+0000 7f34f4c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f00afe70 0x7f34f006f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:34.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.641+0000 7f34f4c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f00afe70 0x7f34f006f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47436/0 (socket says 172.21.15.27:47436) 2024-01-15T18:23:34.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.641+0000 7f34f4c00640 1 -- 172.21.15.27:0/3241901009 learned_addr learned my addr 172.21.15.27:0/3241901009 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:34.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.641+0000 7f34f5600640 1 --2- 172.21.15.27:0/3241901009 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f34f0160b10 0x7f34f0150790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:34.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.641+0000 7f34efe00640 1 --2- 172.21.15.27:0/3241901009 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34f00cb7b0 0x7f34f0158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:34.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.641+0000 7f34f4c00640 1 -- 172.21.15.27:0/3241901009 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34f00cb7b0 msgr2=0x7f34f0158ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:34.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.641+0000 7f34f4c00640 1 --2- 172.21.15.27:0/3241901009 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34f00cb7b0 0x7f34f0158ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:34.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.641+0000 7f34f4c00640 1 -- 172.21.15.27:0/3241901009 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f34f0160b10 msgr2=0x7f34f0150790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:34.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.641+0000 7f34f4c00640 1 --2- 172.21.15.27:0/3241901009 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f34f0160b10 0x7f34f0150790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:34.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.641+0000 7f34f4c00640 1 -- 172.21.15.27:0/3241901009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d8009420 con 0x7f34f00afe70 2024-01-15T18:23:34.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.641+0000 7f34efe00640 1 --2- 172.21.15.27:0/3241901009 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34f00cb7b0 0x7f34f0158ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:23:34.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.641+0000 7f34f5600640 1 --2- 172.21.15.27:0/3241901009 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f34f0160b10 0x7f34f0150790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:23:34.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.641+0000 7f34f4c00640 1 --2- 172.21.15.27:0/3241901009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f00afe70 0x7f34f006f7e0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f34e000c500 tx=0x7f34e000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:34.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.641+0000 7f34ecc00640 1 -- 172.21.15.27:0/3241901009 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e0005cf0 con 0x7f34f00afe70 2024-01-15T18:23:34.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.641+0000 7f34ecc00640 1 -- 172.21.15.27:0/3241901009 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e0005e50 con 0x7f34f00afe70 2024-01-15T18:23:34.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.641+0000 7f34ecc00640 1 -- 172.21.15.27:0/3241901009 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e00072f0 con 0x7f34f00afe70 2024-01-15T18:23:34.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.641+0000 7f34f7600640 1 -- 172.21.15.27:0/3241901009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f01598f0 con 0x7f34f00afe70 2024-01-15T18:23:34.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.641+0000 7f34f7600640 1 -- 172.21.15.27:0/3241901009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f0150cd0 con 0x7f34f00afe70 2024-01-15T18:23:34.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.641+0000 7f34f7600640 1 -- 172.21.15.27:0/3241901009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f0150f80 con 0x7f34f00afe70 2024-01-15T18:23:34.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.645+0000 7f34ecc00640 1 -- 172.21.15.27:0/3241901009 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f34e0007450 con 0x7f34f00afe70 2024-01-15T18:23:34.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.645+0000 7f34ecc00640 1 --2- 172.21.15.27:0/3241901009 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f34c8067a60 0x7f34c8069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:34.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.645+0000 7f34ecc00640 1 -- 172.21.15.27:0/3241901009 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f34e00828b0 con 0x7f34f00afe70 2024-01-15T18:23:34.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.645+0000 7f34efe00640 1 --2- 172.21.15.27:0/3241901009 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f34c8067a60 0x7f34c8069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:34.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.645+0000 7f34efe00640 1 --2- 172.21.15.27:0/3241901009 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f34c8067a60 0x7f34c8069f20 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f34d80ae000 tx=0x7f34d8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:34.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.649+0000 7f34ecc00640 1 -- 172.21.15.27:0/3241901009 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f34e00512a0 con 0x7f34f00afe70 2024-01-15T18:23:34.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.801+0000 7f34f7600640 1 -- 172.21.15.27:0/3241901009 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f34f004e850 con 0x7f34c8067a60 2024-01-15T18:23:34.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.813+0000 7f34ecc00640 1 -- 172.21.15.27:0/3241901009 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+291438 (secure 0 0 0) 0x7f34f004e850 con 0x7f34c8067a60 2024-01-15T18:23:34.810 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:23:34.812 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:23:34.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.813+0000 7f34f7600640 1 -- 172.21.15.27:0/3241901009 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f34c8067a60 msgr2=0x7f34c8069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:34.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.813+0000 7f34f7600640 1 --2- 172.21.15.27:0/3241901009 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f34c8067a60 0x7f34c8069f20 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f34d80ae000 tx=0x7f34d8008040 comp rx=0 tx=0).stop 2024-01-15T18:23:34.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.813+0000 7f34f7600640 1 -- 172.21.15.27:0/3241901009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f00afe70 msgr2=0x7f34f006f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:34.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.813+0000 7f34f7600640 1 --2- 172.21.15.27:0/3241901009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f00afe70 0x7f34f006f7e0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f34e000c500 tx=0x7f34e000c9c0 comp rx=0 tx=0).stop 2024-01-15T18:23:34.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.813+0000 7f34f7600640 1 -- 172.21.15.27:0/3241901009 shutdown_connections 2024-01-15T18:23:34.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.813+0000 7f34f7600640 1 --2- 172.21.15.27:0/3241901009 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f34c8067a60 0x7f34c8069f20 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:34.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.813+0000 7f34f7600640 1 --2- 172.21.15.27:0/3241901009 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f34f0160b10 0x7f34f0150790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:34.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.813+0000 7f34f7600640 1 --2- 172.21.15.27:0/3241901009 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f34f00cb7b0 0x7f34f0158ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:34.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.813+0000 7f34f7600640 1 --2- 172.21.15.27:0/3241901009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f34f00afe70 0x7f34f006f7e0 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:34.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.813+0000 7f34f7600640 1 -- 172.21.15.27:0/3241901009 >> 172.21.15.27:0/3241901009 conn(0x7f34f00d0f20 msgr2=0x7f34f0155c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:34.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.817+0000 7f34f7600640 1 -- 172.21.15.27:0/3241901009 shutdown_connections 2024-01-15T18:23:34.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:34.817+0000 7f34f7600640 1 -- 172.21.15.27:0/3241901009 wait complete. 2024-01-15T18:23:34.832 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":668,"stamp":"2024-01-15T18:23:34.457302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4354848372,"num_objects":1069,"num_object_clones":1,"num_object_copies":2138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1069,"num_whiteouts":1,"num_read":418,"num_read_kb":1142907,"num_write":1483,"num_write_kb":4254748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2926,"ondisk_log_size":2926,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":9070344,"kb_used_data":8532680,"kb_used_omap":12,"kb_used_meta":537587,"kb_avail":745904376,"statfs":{"total":773094113280,"available":763806081024,"internally_reserved":0,"allocated":8737464320,"data_stored":8731229664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":550489648},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":58720646,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":23,"num_read_kb":94208,"num_write":20,"num_write_kb":57348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003319"},"pg_stats":[{"pgid":"17.3f","version":"110'49","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.452952+0000","last_change":"2024-01-15T18:18:33.168792+0000","last_active":"2024-01-15T18:23:23.452952+0000","last_peered":"2024-01-15T18:23:23.452952+0000","last_clean":"2024-01-15T18:23:23.452952+0000","last_became_active":"2024-01-15T18:18:33.168500+0000","last_became_peered":"2024-01-15T18:18:33.168500+0000","last_unstale":"2024-01-15T18:23:23.452952+0000","last_undegraded":"2024-01-15T18:23:23.452952+0000","last_fullsized":"2024-01-15T18:23:23.452952+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:28:41.940111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"110'42","reported_seq":61,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.445536+0000","last_change":"2024-01-15T18:18:33.151630+0000","last_active":"2024-01-15T18:23:23.445536+0000","last_peered":"2024-01-15T18:23:23.445536+0000","last_clean":"2024-01-15T18:23:23.445536+0000","last_became_active":"2024-01-15T18:18:33.151209+0000","last_became_peered":"2024-01-15T18:18:33.151209+0000","last_unstale":"2024-01-15T18:23:23.445536+0000","last_undegraded":"2024-01-15T18:23:23.445536+0000","last_fullsized":"2024-01-15T18:23:23.445536+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:46:54.687765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"110'39","reported_seq":57,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.457520+0000","last_change":"2024-01-15T18:18:33.163094+0000","last_active":"2024-01-15T18:23:23.457520+0000","last_peered":"2024-01-15T18:23:23.457520+0000","last_clean":"2024-01-15T18:23:23.457520+0000","last_became_active":"2024-01-15T18:18:33.162700+0000","last_became_peered":"2024-01-15T18:18:33.162700+0000","last_unstale":"2024-01-15T18:23:23.457520+0000","last_undegraded":"2024-01-15T18:23:23.457520+0000","last_fullsized":"2024-01-15T18:23:23.457520+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:23:36.167165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":16,"num_write_kb":57345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"110'39","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.459915+0000","last_change":"2024-01-15T18:18:33.160786+0000","last_active":"2024-01-15T18:23:23.459915+0000","last_peered":"2024-01-15T18:23:23.459915+0000","last_clean":"2024-01-15T18:23:23.459915+0000","last_became_active":"2024-01-15T18:18:33.160460+0000","last_became_peered":"2024-01-15T18:18:33.160460+0000","last_unstale":"2024-01-15T18:23:23.459915+0000","last_undegraded":"2024-01-15T18:23:23.459915+0000","last_fullsized":"2024-01-15T18:23:23.459915+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:27:33.980071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"110'25","reported_seq":42,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.441694+0000","last_change":"2024-01-15T18:18:33.155936+0000","last_active":"2024-01-15T18:23:23.441694+0000","last_peered":"2024-01-15T18:23:23.441694+0000","last_clean":"2024-01-15T18:23:23.441694+0000","last_became_active":"2024-01-15T18:18:33.155656+0000","last_became_peered":"2024-01-15T18:18:33.155656+0000","last_unstale":"2024-01-15T18:23:23.441694+0000","last_undegraded":"2024-01-15T18:23:23.441694+0000","last_fullsized":"2024-01-15T18:23:23.441694+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:27:09.469255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"110'45","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.452122+0000","last_change":"2024-01-15T18:18:33.152032+0000","last_active":"2024-01-15T18:23:23.452122+0000","last_peered":"2024-01-15T18:23:23.452122+0000","last_clean":"2024-01-15T18:23:23.452122+0000","last_became_active":"2024-01-15T18:18:33.151701+0000","last_became_peered":"2024-01-15T18:18:33.151701+0000","last_unstale":"2024-01-15T18:23:23.452122+0000","last_undegraded":"2024-01-15T18:23:23.452122+0000","last_fullsized":"2024-01-15T18:23:23.452122+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:51.983410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"110'43","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.556273+0000","last_change":"2024-01-15T18:18:33.153882+0000","last_active":"2024-01-15T18:23:31.556273+0000","last_peered":"2024-01-15T18:23:31.556273+0000","last_clean":"2024-01-15T18:23:31.556273+0000","last_became_active":"2024-01-15T18:18:33.153264+0000","last_became_peered":"2024-01-15T18:18:33.153264+0000","last_unstale":"2024-01-15T18:23:31.556273+0000","last_undegraded":"2024-01-15T18:23:31.556273+0000","last_fullsized":"2024-01-15T18:23:31.556273+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:37:49.663025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"110'42","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.451406+0000","last_change":"2024-01-15T18:18:33.164714+0000","last_active":"2024-01-15T18:23:23.451406+0000","last_peered":"2024-01-15T18:23:23.451406+0000","last_clean":"2024-01-15T18:23:23.451406+0000","last_became_active":"2024-01-15T18:18:33.163896+0000","last_became_peered":"2024-01-15T18:18:33.163896+0000","last_unstale":"2024-01-15T18:23:23.451406+0000","last_undegraded":"2024-01-15T18:23:23.451406+0000","last_fullsized":"2024-01-15T18:23:23.451406+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:21:18.264946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":19,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"110'42","reported_seq":60,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.450283+0000","last_change":"2024-01-15T18:18:33.158682+0000","last_active":"2024-01-15T18:23:23.450283+0000","last_peered":"2024-01-15T18:23:23.450283+0000","last_clean":"2024-01-15T18:23:23.450283+0000","last_became_active":"2024-01-15T18:18:33.158399+0000","last_became_peered":"2024-01-15T18:18:33.158399+0000","last_unstale":"2024-01-15T18:23:23.450283+0000","last_undegraded":"2024-01-15T18:23:23.450283+0000","last_fullsized":"2024-01-15T18:23:23.450283+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:27:12.032383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"110'41","reported_seq":60,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.445619+0000","last_change":"2024-01-15T18:18:33.159936+0000","last_active":"2024-01-15T18:23:23.445619+0000","last_peered":"2024-01-15T18:23:23.445619+0000","last_clean":"2024-01-15T18:23:23.445619+0000","last_became_active":"2024-01-15T18:18:33.159562+0000","last_became_peered":"2024-01-15T18:18:33.159562+0000","last_unstale":"2024-01-15T18:23:23.445619+0000","last_undegraded":"2024-01-15T18:23:23.445619+0000","last_fullsized":"2024-01-15T18:23:23.445619+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:56:32.966770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":11,"num_write_kb":45056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"110'41","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.443258+0000","last_change":"2024-01-15T18:18:33.155247+0000","last_active":"2024-01-15T18:23:23.443258+0000","last_peered":"2024-01-15T18:23:23.443258+0000","last_clean":"2024-01-15T18:23:23.443258+0000","last_became_active":"2024-01-15T18:18:33.154972+0000","last_became_peered":"2024-01-15T18:18:33.154972+0000","last_unstale":"2024-01-15T18:23:23.443258+0000","last_undegraded":"2024-01-15T18:23:23.443258+0000","last_fullsized":"2024-01-15T18:23:23.443258+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:54:47.923576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"110'43","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.444775+0000","last_change":"2024-01-15T18:18:33.158336+0000","last_active":"2024-01-15T18:23:23.444775+0000","last_peered":"2024-01-15T18:23:23.444775+0000","last_clean":"2024-01-15T18:23:23.444775+0000","last_became_active":"2024-01-15T18:18:33.158064+0000","last_became_peered":"2024-01-15T18:18:33.158064+0000","last_unstale":"2024-01-15T18:23:23.444775+0000","last_undegraded":"2024-01-15T18:23:23.444775+0000","last_fullsized":"2024-01-15T18:23:23.444775+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:42:44.182465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":6,"num_read_kb":20481,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"110'34","reported_seq":58,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.451044+0000","last_change":"2024-01-15T18:18:33.163912+0000","last_active":"2024-01-15T18:23:23.451044+0000","last_peered":"2024-01-15T18:23:23.451044+0000","last_clean":"2024-01-15T18:23:23.451044+0000","last_became_active":"2024-01-15T18:18:33.163610+0000","last_became_peered":"2024-01-15T18:18:33.163610+0000","last_unstale":"2024-01-15T18:23:23.451044+0000","last_undegraded":"2024-01-15T18:23:23.451044+0000","last_fullsized":"2024-01-15T18:23:23.451044+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:32:43.468374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"110'43","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.451241+0000","last_change":"2024-01-15T18:18:33.169868+0000","last_active":"2024-01-15T18:23:23.451241+0000","last_peered":"2024-01-15T18:23:23.451241+0000","last_clean":"2024-01-15T18:23:23.451241+0000","last_became_active":"2024-01-15T18:18:33.169530+0000","last_became_peered":"2024-01-15T18:18:33.169530+0000","last_unstale":"2024-01-15T18:23:23.451241+0000","last_undegraded":"2024-01-15T18:23:23.451241+0000","last_fullsized":"2024-01-15T18:23:23.451241+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:18:22.473056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"110'29","reported_seq":47,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.453353+0000","last_change":"2024-01-15T18:18:33.151325+0000","last_active":"2024-01-15T18:23:23.453353+0000","last_peered":"2024-01-15T18:23:23.453353+0000","last_clean":"2024-01-15T18:23:23.453353+0000","last_became_active":"2024-01-15T18:18:33.151040+0000","last_became_peered":"2024-01-15T18:18:33.151040+0000","last_unstale":"2024-01-15T18:23:23.453353+0000","last_undegraded":"2024-01-15T18:23:23.453353+0000","last_fullsized":"2024-01-15T18:23:23.453353+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:57:08.165322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":7,"num_write_kb":28672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"110'40","reported_seq":57,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.459812+0000","last_change":"2024-01-15T18:18:33.166327+0000","last_active":"2024-01-15T18:23:23.459812+0000","last_peered":"2024-01-15T18:23:23.459812+0000","last_clean":"2024-01-15T18:23:23.459812+0000","last_became_active":"2024-01-15T18:18:33.166014+0000","last_became_peered":"2024-01-15T18:18:33.166014+0000","last_unstale":"2024-01-15T18:23:23.459812+0000","last_undegraded":"2024-01-15T18:23:23.459812+0000","last_fullsized":"2024-01-15T18:23:23.459812+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:37:50.428714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"110'39","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.456607+0000","last_change":"2024-01-15T18:18:33.168613+0000","last_active":"2024-01-15T18:23:23.456607+0000","last_peered":"2024-01-15T18:23:23.456607+0000","last_clean":"2024-01-15T18:23:23.456607+0000","last_became_active":"2024-01-15T18:18:33.167975+0000","last_became_peered":"2024-01-15T18:18:33.167975+0000","last_unstale":"2024-01-15T18:23:23.456607+0000","last_undegraded":"2024-01-15T18:23:23.456607+0000","last_fullsized":"2024-01-15T18:23:23.456607+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:22:13.319815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"110'47","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.456193+0000","last_change":"2024-01-15T18:18:33.155078+0000","last_active":"2024-01-15T18:23:23.456193+0000","last_peered":"2024-01-15T18:23:23.456193+0000","last_clean":"2024-01-15T18:23:23.456193+0000","last_became_active":"2024-01-15T18:18:33.154816+0000","last_became_peered":"2024-01-15T18:18:33.154816+0000","last_unstale":"2024-01-15T18:23:23.456193+0000","last_undegraded":"2024-01-15T18:23:23.456193+0000","last_fullsized":"2024-01-15T18:23:23.456193+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:48.739168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"110'40","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.454711+0000","last_change":"2024-01-15T18:18:33.164028+0000","last_active":"2024-01-15T18:23:23.454711+0000","last_peered":"2024-01-15T18:23:23.454711+0000","last_clean":"2024-01-15T18:23:23.454711+0000","last_became_active":"2024-01-15T18:18:33.163421+0000","last_became_peered":"2024-01-15T18:18:33.163421+0000","last_unstale":"2024-01-15T18:23:23.454711+0000","last_undegraded":"2024-01-15T18:23:23.454711+0000","last_fullsized":"2024-01-15T18:23:23.454711+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:38:31.615856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"110'46","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.456828+0000","last_change":"2024-01-15T18:18:33.160995+0000","last_active":"2024-01-15T18:23:23.456828+0000","last_peered":"2024-01-15T18:23:23.456828+0000","last_clean":"2024-01-15T18:23:23.456828+0000","last_became_active":"2024-01-15T18:18:33.160047+0000","last_became_peered":"2024-01-15T18:18:33.160047+0000","last_unstale":"2024-01-15T18:23:23.456828+0000","last_undegraded":"2024-01-15T18:23:23.456828+0000","last_fullsized":"2024-01-15T18:23:23.456828+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:39:53.520191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"110'50","reported_seq":76,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.458912+0000","last_change":"2024-01-15T18:18:33.152696+0000","last_active":"2024-01-15T18:23:23.458912+0000","last_peered":"2024-01-15T18:23:23.458912+0000","last_clean":"2024-01-15T18:23:23.458912+0000","last_became_active":"2024-01-15T18:18:33.152128+0000","last_became_peered":"2024-01-15T18:18:33.152128+0000","last_unstale":"2024-01-15T18:23:23.458912+0000","last_undegraded":"2024-01-15T18:23:23.458912+0000","last_fullsized":"2024-01-15T18:23:23.458912+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:08:18.706229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":11,"num_read_kb":36866,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"110'41","reported_seq":60,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.450505+0000","last_change":"2024-01-15T18:18:33.163614+0000","last_active":"2024-01-15T18:23:23.450505+0000","last_peered":"2024-01-15T18:23:23.450505+0000","last_clean":"2024-01-15T18:23:23.450505+0000","last_became_active":"2024-01-15T18:18:33.163252+0000","last_became_peered":"2024-01-15T18:18:33.163252+0000","last_unstale":"2024-01-15T18:23:23.450505+0000","last_undegraded":"2024-01-15T18:23:23.450505+0000","last_fullsized":"2024-01-15T18:23:23.450505+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:48.739168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":4,"num_read_kb":8194,"num_write":17,"num_write_kb":61441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"110'35","reported_seq":52,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.457056+0000","last_change":"2024-01-15T18:18:33.159896+0000","last_active":"2024-01-15T18:23:23.457056+0000","last_peered":"2024-01-15T18:23:23.457056+0000","last_clean":"2024-01-15T18:23:23.457056+0000","last_became_active":"2024-01-15T18:18:33.159557+0000","last_became_peered":"2024-01-15T18:18:33.159557+0000","last_unstale":"2024-01-15T18:23:23.457056+0000","last_undegraded":"2024-01-15T18:23:23.457056+0000","last_fullsized":"2024-01-15T18:23:23.457056+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:24.034308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":11,"num_write_kb":45056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"110'39","reported_seq":57,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.459003+0000","last_change":"2024-01-15T18:18:33.166766+0000","last_active":"2024-01-15T18:23:23.459003+0000","last_peered":"2024-01-15T18:23:23.459003+0000","last_clean":"2024-01-15T18:23:23.459003+0000","last_became_active":"2024-01-15T18:18:33.165600+0000","last_became_peered":"2024-01-15T18:18:33.165600+0000","last_unstale":"2024-01-15T18:23:23.459003+0000","last_undegraded":"2024-01-15T18:23:23.459003+0000","last_fullsized":"2024-01-15T18:23:23.459003+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:03.948102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"110'35","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.457832+0000","last_change":"2024-01-15T18:18:33.168720+0000","last_active":"2024-01-15T18:23:23.457832+0000","last_peered":"2024-01-15T18:23:23.457832+0000","last_clean":"2024-01-15T18:23:23.457832+0000","last_became_active":"2024-01-15T18:18:33.168349+0000","last_became_peered":"2024-01-15T18:18:33.168349+0000","last_unstale":"2024-01-15T18:23:23.457832+0000","last_undegraded":"2024-01-15T18:23:23.457832+0000","last_fullsized":"2024-01-15T18:23:23.457832+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:18:22.473056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":16386,"num_write":17,"num_write_kb":61441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"110'50","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.457272+0000","last_change":"2024-01-15T18:18:33.159262+0000","last_active":"2024-01-15T18:23:23.457272+0000","last_peered":"2024-01-15T18:23:23.457272+0000","last_clean":"2024-01-15T18:23:23.457272+0000","last_became_active":"2024-01-15T18:18:33.158946+0000","last_became_peered":"2024-01-15T18:18:33.158946+0000","last_unstale":"2024-01-15T18:23:23.457272+0000","last_undegraded":"2024-01-15T18:23:23.457272+0000","last_fullsized":"2024-01-15T18:23:23.457272+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:05:55.347898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"110'37","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.447552+0000","last_change":"2024-01-15T18:18:33.161814+0000","last_active":"2024-01-15T18:23:23.447552+0000","last_peered":"2024-01-15T18:23:23.447552+0000","last_clean":"2024-01-15T18:23:23.447552+0000","last_became_active":"2024-01-15T18:18:33.161463+0000","last_became_peered":"2024-01-15T18:18:33.161463+0000","last_unstale":"2024-01-15T18:23:23.447552+0000","last_undegraded":"2024-01-15T18:23:23.447552+0000","last_fullsized":"2024-01-15T18:23:23.447552+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:15:36.391855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886260,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4,"num_read_kb":12289,"num_write":21,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"110'31","reported_seq":49,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.450037+0000","last_change":"2024-01-15T18:18:33.162952+0000","last_active":"2024-01-15T18:23:23.450037+0000","last_peered":"2024-01-15T18:23:23.450037+0000","last_clean":"2024-01-15T18:23:23.450037+0000","last_became_active":"2024-01-15T18:18:33.162669+0000","last_became_peered":"2024-01-15T18:18:33.162669+0000","last_unstale":"2024-01-15T18:23:23.450037+0000","last_undegraded":"2024-01-15T18:23:23.450037+0000","last_fullsized":"2024-01-15T18:23:23.450037+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:47:59.088407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"110'38","reported_seq":57,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.456970+0000","last_change":"2024-01-15T18:18:33.160658+0000","last_active":"2024-01-15T18:23:23.456970+0000","last_peered":"2024-01-15T18:23:23.456970+0000","last_clean":"2024-01-15T18:23:23.456970+0000","last_became_active":"2024-01-15T18:18:33.160099+0000","last_became_peered":"2024-01-15T18:18:33.160099+0000","last_unstale":"2024-01-15T18:23:23.456970+0000","last_undegraded":"2024-01-15T18:23:23.456970+0000","last_fullsized":"2024-01-15T18:23:23.456970+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:47:59.088407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"110'41","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.812331+0000","last_change":"2024-01-15T18:18:33.161061+0000","last_active":"2024-01-15T18:23:31.812331+0000","last_peered":"2024-01-15T18:23:31.812331+0000","last_clean":"2024-01-15T18:23:31.812331+0000","last_became_active":"2024-01-15T18:18:33.160725+0000","last_became_peered":"2024-01-15T18:18:33.160725+0000","last_unstale":"2024-01-15T18:23:31.812331+0000","last_undegraded":"2024-01-15T18:23:31.812331+0000","last_fullsized":"2024-01-15T18:23:31.812331+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:38:07.236956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"110'59","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.456771+0000","last_change":"2024-01-15T18:18:33.159024+0000","last_active":"2024-01-15T18:23:23.456771+0000","last_peered":"2024-01-15T18:23:23.456771+0000","last_clean":"2024-01-15T18:23:23.456771+0000","last_became_active":"2024-01-15T18:18:33.158701+0000","last_became_peered":"2024-01-15T18:18:33.158701+0000","last_unstale":"2024-01-15T18:23:23.456771+0000","last_undegraded":"2024-01-15T18:23:23.456771+0000","last_fullsized":"2024-01-15T18:23:23.456771+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:50:16.189329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"110'50","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.452426+0000","last_change":"2024-01-15T18:18:33.152241+0000","last_active":"2024-01-15T18:23:23.452426+0000","last_peered":"2024-01-15T18:23:23.452426+0000","last_clean":"2024-01-15T18:23:23.452426+0000","last_became_active":"2024-01-15T18:18:33.151250+0000","last_became_peered":"2024-01-15T18:18:33.151250+0000","last_unstale":"2024-01-15T18:23:23.452426+0000","last_undegraded":"2024-01-15T18:23:23.452426+0000","last_fullsized":"2024-01-15T18:23:23.452426+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:45:02.525155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"110'29","reported_seq":48,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.445810+0000","last_change":"2024-01-15T18:18:33.154972+0000","last_active":"2024-01-15T18:23:23.445810+0000","last_peered":"2024-01-15T18:23:23.445810+0000","last_clean":"2024-01-15T18:23:23.445810+0000","last_became_active":"2024-01-15T18:18:33.154569+0000","last_became_peered":"2024-01-15T18:18:33.154569+0000","last_unstale":"2024-01-15T18:23:23.445810+0000","last_undegraded":"2024-01-15T18:23:23.445810+0000","last_fullsized":"2024-01-15T18:23:23.445810+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:29:02.332324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"107'4","reported_seq":20,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:39.170083+0000","last_change":"2024-01-15T18:18:32.217761+0000","last_active":"2024-01-15T18:20:39.170083+0000","last_peered":"2024-01-15T18:20:39.170083+0000","last_clean":"2024-01-15T18:20:39.170083+0000","last_became_active":"2024-01-15T18:18:32.216337+0000","last_became_peered":"2024-01-15T18:18:32.216337+0000","last_unstale":"2024-01-15T18:20:39.170083+0000","last_undegraded":"2024-01-15T18:20:39.170083+0000","last_fullsized":"2024-01-15T18:20:39.170083+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:01:18.991818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"110'44","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.453170+0000","last_change":"2024-01-15T18:18:33.167015+0000","last_active":"2024-01-15T18:23:23.453170+0000","last_peered":"2024-01-15T18:23:23.453170+0000","last_clean":"2024-01-15T18:23:23.453170+0000","last_became_active":"2024-01-15T18:18:33.166726+0000","last_became_peered":"2024-01-15T18:18:33.166726+0000","last_unstale":"2024-01-15T18:23:23.453170+0000","last_undegraded":"2024-01-15T18:23:23.453170+0000","last_fullsized":"2024-01-15T18:23:23.453170+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:48:57.895180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"107'1","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:40.169868+0000","last_change":"2024-01-15T18:18:32.219987+0000","last_active":"2024-01-15T18:20:40.169868+0000","last_peered":"2024-01-15T18:20:40.169868+0000","last_clean":"2024-01-15T18:20:40.169868+0000","last_became_active":"2024-01-15T18:18:32.219596+0000","last_became_peered":"2024-01-15T18:18:32.219596+0000","last_unstale":"2024-01-15T18:20:40.169868+0000","last_undegraded":"2024-01-15T18:20:40.169868+0000","last_fullsized":"2024-01-15T18:20:40.169868+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:23.024473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"110'40","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.451309+0000","last_change":"2024-01-15T18:18:33.150743+0000","last_active":"2024-01-15T18:23:23.451309+0000","last_peered":"2024-01-15T18:23:23.451309+0000","last_clean":"2024-01-15T18:23:23.451309+0000","last_became_active":"2024-01-15T18:18:33.150429+0000","last_became_peered":"2024-01-15T18:18:33.150429+0000","last_unstale":"2024-01-15T18:23:23.451309+0000","last_undegraded":"2024-01-15T18:23:23.451309+0000","last_fullsized":"2024-01-15T18:23:23.451309+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:33:18.994479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"110'3","reported_seq":18,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:19:10.235403+0000","last_change":"2024-01-15T18:18:32.213207+0000","last_active":"2024-01-15T18:19:10.235403+0000","last_peered":"2024-01-15T18:19:10.235403+0000","last_clean":"2024-01-15T18:19:10.235403+0000","last_became_active":"2024-01-15T18:18:32.212071+0000","last_became_peered":"2024-01-15T18:18:32.212071+0000","last_unstale":"2024-01-15T18:19:10.235403+0000","last_undegraded":"2024-01-15T18:19:10.235403+0000","last_fullsized":"2024-01-15T18:19:10.235403+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:29:14.612143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"110'51","reported_seq":75,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.454632+0000","last_change":"2024-01-15T18:18:33.165545+0000","last_active":"2024-01-15T18:23:23.454632+0000","last_peered":"2024-01-15T18:23:23.454632+0000","last_clean":"2024-01-15T18:23:23.454632+0000","last_became_active":"2024-01-15T18:18:33.165258+0000","last_became_peered":"2024-01-15T18:18:33.165258+0000","last_unstale":"2024-01-15T18:23:23.454632+0000","last_undegraded":"2024-01-15T18:23:23.454632+0000","last_fullsized":"2024-01-15T18:23:23.454632+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:05:07.453809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"107'2","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:18:42.257846+0000","last_change":"2024-01-15T18:18:32.217238+0000","last_active":"2024-01-15T18:18:42.257846+0000","last_peered":"2024-01-15T18:18:42.257846+0000","last_clean":"2024-01-15T18:18:42.257846+0000","last_became_active":"2024-01-15T18:18:32.216492+0000","last_became_peered":"2024-01-15T18:18:32.216492+0000","last_unstale":"2024-01-15T18:18:42.257846+0000","last_undegraded":"2024-01-15T18:18:42.257846+0000","last_fullsized":"2024-01-15T18:18:42.257846+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:40:25.010807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"110'36","reported_seq":54,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.447490+0000","last_change":"2024-01-15T18:18:33.157996+0000","last_active":"2024-01-15T18:23:23.447490+0000","last_peered":"2024-01-15T18:23:23.447490+0000","last_clean":"2024-01-15T18:23:23.447490+0000","last_became_active":"2024-01-15T18:18:33.157687+0000","last_became_peered":"2024-01-15T18:18:33.157687+0000","last_unstale":"2024-01-15T18:23:23.447490+0000","last_undegraded":"2024-01-15T18:23:23.447490+0000","last_fullsized":"2024-01-15T18:23:23.447490+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:26:16.219599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"107'1","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:21:04.173272+0000","last_change":"2024-01-15T18:18:32.215007+0000","last_active":"2024-01-15T18:21:04.173272+0000","last_peered":"2024-01-15T18:21:04.173272+0000","last_clean":"2024-01-15T18:21:04.173272+0000","last_became_active":"2024-01-15T18:18:32.214453+0000","last_became_peered":"2024-01-15T18:18:32.214453+0000","last_unstale":"2024-01-15T18:21:04.173272+0000","last_undegraded":"2024-01-15T18:21:04.173272+0000","last_fullsized":"2024-01-15T18:21:04.173272+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:58:50.916786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"110'44","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.457859+0000","last_change":"2024-01-15T18:18:33.153355+0000","last_active":"2024-01-15T18:23:23.457859+0000","last_peered":"2024-01-15T18:23:23.457859+0000","last_clean":"2024-01-15T18:23:23.457859+0000","last_became_active":"2024-01-15T18:18:33.153058+0000","last_became_peered":"2024-01-15T18:18:33.153058+0000","last_unstale":"2024-01-15T18:23:23.457859+0000","last_undegraded":"2024-01-15T18:23:23.457859+0000","last_fullsized":"2024-01-15T18:23:23.457859+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:21:18.264946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"107'1","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:18:43.638172+0000","last_change":"2024-01-15T18:18:32.215681+0000","last_active":"2024-01-15T18:18:43.638172+0000","last_peered":"2024-01-15T18:18:43.638172+0000","last_clean":"2024-01-15T18:18:43.638172+0000","last_became_active":"2024-01-15T18:18:32.215417+0000","last_became_peered":"2024-01-15T18:18:32.215417+0000","last_unstale":"2024-01-15T18:18:43.638172+0000","last_undegraded":"2024-01-15T18:18:43.638172+0000","last_fullsized":"2024-01-15T18:18:43.638172+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:37:03.114460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"110'45","reported_seq":60,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.446039+0000","last_change":"2024-01-15T18:18:43.024618+0000","last_active":"2024-01-15T18:23:23.446039+0000","last_peered":"2024-01-15T18:23:23.446039+0000","last_clean":"2024-01-15T18:23:23.446039+0000","last_became_active":"2024-01-15T18:18:43.024221+0000","last_became_peered":"2024-01-15T18:18:43.024221+0000","last_unstale":"2024-01-15T18:23:23.446039+0000","last_undegraded":"2024-01-15T18:23:23.446039+0000","last_fullsized":"2024-01-15T18:23:23.446039+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:27.549069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303360,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:21:05.181499+0000","last_change":"2024-01-15T18:18:32.217097+0000","last_active":"2024-01-15T18:21:05.181499+0000","last_peered":"2024-01-15T18:21:05.181499+0000","last_clean":"2024-01-15T18:21:05.181499+0000","last_became_active":"2024-01-15T18:18:32.215465+0000","last_became_peered":"2024-01-15T18:18:32.215465+0000","last_unstale":"2024-01-15T18:21:05.181499+0000","last_undegraded":"2024-01-15T18:21:05.181499+0000","last_fullsized":"2024-01-15T18:21:05.181499+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:47.729333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"110'43","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.553963+0000","last_change":"2024-01-15T18:18:33.154067+0000","last_active":"2024-01-15T18:23:31.553963+0000","last_peered":"2024-01-15T18:23:31.553963+0000","last_clean":"2024-01-15T18:23:31.553963+0000","last_became_active":"2024-01-15T18:18:33.153610+0000","last_became_peered":"2024-01-15T18:18:33.153610+0000","last_unstale":"2024-01-15T18:23:31.553963+0000","last_undegraded":"2024-01-15T18:23:31.553963+0000","last_fullsized":"2024-01-15T18:23:31.553963+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:08:41.862115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:59.742931+0000","last_change":"2024-01-15T18:18:32.219909+0000","last_active":"2024-01-15T18:20:59.742931+0000","last_peered":"2024-01-15T18:20:59.742931+0000","last_clean":"2024-01-15T18:20:59.742931+0000","last_became_active":"2024-01-15T18:18:32.219555+0000","last_became_peered":"2024-01-15T18:18:32.219555+0000","last_unstale":"2024-01-15T18:20:59.742931+0000","last_undegraded":"2024-01-15T18:20:59.742931+0000","last_fullsized":"2024-01-15T18:20:59.742931+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:17:21.343043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"110'42","reported_seq":61,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:30.677222+0000","last_change":"2024-01-15T18:18:33.156038+0000","last_active":"2024-01-15T18:23:30.677222+0000","last_peered":"2024-01-15T18:23:30.677222+0000","last_clean":"2024-01-15T18:23:30.677222+0000","last_became_active":"2024-01-15T18:18:33.155765+0000","last_became_peered":"2024-01-15T18:18:33.155765+0000","last_unstale":"2024-01-15T18:23:30.677222+0000","last_undegraded":"2024-01-15T18:23:30.677222+0000","last_fullsized":"2024-01-15T18:23:30.677222+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:58:43.210443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"110'1","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:21:00.168575+0000","last_change":"2024-01-15T18:18:32.212880+0000","last_active":"2024-01-15T18:21:00.168575+0000","last_peered":"2024-01-15T18:21:00.168575+0000","last_clean":"2024-01-15T18:21:00.168575+0000","last_became_active":"2024-01-15T18:18:32.212511+0000","last_became_peered":"2024-01-15T18:18:32.212511+0000","last_unstale":"2024-01-15T18:21:00.168575+0000","last_undegraded":"2024-01-15T18:21:00.168575+0000","last_fullsized":"2024-01-15T18:21:00.168575+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:58:38.191140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"110'35","reported_seq":53,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.444028+0000","last_change":"2024-01-15T18:18:33.164800+0000","last_active":"2024-01-15T18:23:23.444028+0000","last_peered":"2024-01-15T18:23:23.444028+0000","last_clean":"2024-01-15T18:23:23.444028+0000","last_became_active":"2024-01-15T18:18:33.164530+0000","last_became_peered":"2024-01-15T18:18:33.164530+0000","last_unstale":"2024-01-15T18:23:23.444028+0000","last_undegraded":"2024-01-15T18:23:23.444028+0000","last_fullsized":"2024-01-15T18:23:23.444028+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:55.175065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:21:00.437395+0000","last_change":"2024-01-15T18:18:32.216873+0000","last_active":"2024-01-15T18:21:00.437395+0000","last_peered":"2024-01-15T18:21:00.437395+0000","last_clean":"2024-01-15T18:21:00.437395+0000","last_became_active":"2024-01-15T18:18:32.216337+0000","last_became_peered":"2024-01-15T18:18:32.216337+0000","last_unstale":"2024-01-15T18:21:00.437395+0000","last_undegraded":"2024-01-15T18:21:00.437395+0000","last_fullsized":"2024-01-15T18:21:00.437395+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:02.176575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"110'41","reported_seq":61,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.460483+0000","last_change":"2024-01-15T18:18:33.148278+0000","last_active":"2024-01-15T18:23:23.460483+0000","last_peered":"2024-01-15T18:23:23.460483+0000","last_clean":"2024-01-15T18:23:23.460483+0000","last_became_active":"2024-01-15T18:18:33.148016+0000","last_became_peered":"2024-01-15T18:18:33.148016+0000","last_unstale":"2024-01-15T18:23:23.460483+0000","last_undegraded":"2024-01-15T18:23:23.460483+0000","last_fullsized":"2024-01-15T18:23:23.460483+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:24.034308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"110'12","reported_seq":28,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:21:26.352083+0000","last_change":"2024-01-15T18:18:32.214473+0000","last_active":"2024-01-15T18:21:26.352083+0000","last_peered":"2024-01-15T18:21:26.352083+0000","last_clean":"2024-01-15T18:21:26.352083+0000","last_became_active":"2024-01-15T18:18:32.214186+0000","last_became_peered":"2024-01-15T18:18:32.214186+0000","last_unstale":"2024-01-15T18:21:26.352083+0000","last_undegraded":"2024-01-15T18:21:26.352083+0000","last_fullsized":"2024-01-15T18:21:26.352083+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:15:38.528251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"110'43","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.563032+0000","last_change":"2024-01-15T18:18:33.159329+0000","last_active":"2024-01-15T18:23:31.563032+0000","last_peered":"2024-01-15T18:23:31.563032+0000","last_clean":"2024-01-15T18:23:31.563032+0000","last_became_active":"2024-01-15T18:18:33.158541+0000","last_became_peered":"2024-01-15T18:18:33.158541+0000","last_unstale":"2024-01-15T18:23:31.563032+0000","last_undegraded":"2024-01-15T18:23:31.563032+0000","last_fullsized":"2024-01-15T18:23:31.563032+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:53:07.912473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"110'170","reported_seq":186,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.466042+0000","last_change":"2024-01-15T18:18:32.218321+0000","last_active":"2024-01-15T18:23:23.466042+0000","last_peered":"2024-01-15T18:23:23.466042+0000","last_clean":"2024-01-15T18:23:23.466042+0000","last_became_active":"2024-01-15T18:18:32.218002+0000","last_became_peered":"2024-01-15T18:18:32.218002+0000","last_unstale":"2024-01-15T18:23:23.466042+0000","last_undegraded":"2024-01-15T18:23:23.466042+0000","last_fullsized":"2024-01-15T18:23:23.466042+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:12:40.196083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":567317,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":169,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"110'36","reported_seq":53,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.455178+0000","last_change":"2024-01-15T18:18:33.151414+0000","last_active":"2024-01-15T18:23:23.455178+0000","last_peered":"2024-01-15T18:23:23.455178+0000","last_clean":"2024-01-15T18:23:23.455178+0000","last_became_active":"2024-01-15T18:18:33.151132+0000","last_became_peered":"2024-01-15T18:18:33.151132+0000","last_unstale":"2024-01-15T18:23:23.455178+0000","last_undegraded":"2024-01-15T18:23:23.455178+0000","last_fullsized":"2024-01-15T18:23:23.455178+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:48:57.895180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"107'2","reported_seq":18,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:55.470178+0000","last_change":"2024-01-15T18:18:32.215335+0000","last_active":"2024-01-15T18:20:55.470178+0000","last_peered":"2024-01-15T18:20:55.470178+0000","last_clean":"2024-01-15T18:20:55.470178+0000","last_became_active":"2024-01-15T18:18:32.215073+0000","last_became_peered":"2024-01-15T18:18:32.215073+0000","last_unstale":"2024-01-15T18:20:55.470178+0000","last_undegraded":"2024-01-15T18:20:55.470178+0000","last_fullsized":"2024-01-15T18:20:55.470178+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:04:07.912459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"110'46","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.458337+0000","last_change":"2024-01-15T18:18:33.161775+0000","last_active":"2024-01-15T18:23:23.458337+0000","last_peered":"2024-01-15T18:23:23.458337+0000","last_clean":"2024-01-15T18:23:23.458337+0000","last_became_active":"2024-01-15T18:18:33.161493+0000","last_became_peered":"2024-01-15T18:18:33.161493+0000","last_unstale":"2024-01-15T18:23:23.458337+0000","last_undegraded":"2024-01-15T18:23:23.458337+0000","last_fullsized":"2024-01-15T18:23:23.458337+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:55.175065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:55.488186+0000","last_change":"2024-01-15T18:18:32.215225+0000","last_active":"2024-01-15T18:20:55.488186+0000","last_peered":"2024-01-15T18:20:55.488186+0000","last_clean":"2024-01-15T18:20:55.488186+0000","last_became_active":"2024-01-15T18:18:32.214942+0000","last_became_peered":"2024-01-15T18:18:32.214942+0000","last_unstale":"2024-01-15T18:20:55.488186+0000","last_undegraded":"2024-01-15T18:20:55.488186+0000","last_fullsized":"2024-01-15T18:20:55.488186+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:09:53.884883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"110'43","reported_seq":61,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.460487+0000","last_change":"2024-01-15T18:18:33.146254+0000","last_active":"2024-01-15T18:23:23.460487+0000","last_peered":"2024-01-15T18:23:23.460487+0000","last_clean":"2024-01-15T18:23:23.460487+0000","last_became_active":"2024-01-15T18:18:33.145973+0000","last_became_peered":"2024-01-15T18:18:33.145973+0000","last_unstale":"2024-01-15T18:23:23.460487+0000","last_undegraded":"2024-01-15T18:23:23.460487+0000","last_fullsized":"2024-01-15T18:23:23.460487+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:09:54.894718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080645,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":22,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"107'8","reported_seq":24,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:55.167804+0000","last_change":"2024-01-15T18:18:32.219579+0000","last_active":"2024-01-15T18:20:55.167804+0000","last_peered":"2024-01-15T18:20:55.167804+0000","last_clean":"2024-01-15T18:20:55.167804+0000","last_became_active":"2024-01-15T18:18:32.219280+0000","last_became_peered":"2024-01-15T18:18:32.219280+0000","last_unstale":"2024-01-15T18:20:55.167804+0000","last_undegraded":"2024-01-15T18:20:55.167804+0000","last_fullsized":"2024-01-15T18:20:55.167804+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:56:09.818155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"110'21","reported_seq":38,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.451431+0000","last_change":"2024-01-15T18:18:33.161674+0000","last_active":"2024-01-15T18:23:23.451431+0000","last_peered":"2024-01-15T18:23:23.451431+0000","last_clean":"2024-01-15T18:23:23.451431+0000","last_became_active":"2024-01-15T18:18:33.161329+0000","last_became_peered":"2024-01-15T18:18:33.161329+0000","last_unstale":"2024-01-15T18:23:23.451431+0000","last_undegraded":"2024-01-15T18:23:23.451431+0000","last_fullsized":"2024-01-15T18:23:23.451431+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:45:56.254343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":7,"num_write_kb":28672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"107'1","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:49.741619+0000","last_change":"2024-01-15T18:18:32.215848+0000","last_active":"2024-01-15T18:20:49.741619+0000","last_peered":"2024-01-15T18:20:49.741619+0000","last_clean":"2024-01-15T18:20:49.741619+0000","last_became_active":"2024-01-15T18:18:32.214009+0000","last_became_peered":"2024-01-15T18:18:32.214009+0000","last_unstale":"2024-01-15T18:20:49.741619+0000","last_undegraded":"2024-01-15T18:20:49.741619+0000","last_fullsized":"2024-01-15T18:20:49.741619+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:48:09.722577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"110'35","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.453210+0000","last_change":"2024-01-15T18:18:33.162974+0000","last_active":"2024-01-15T18:23:23.453210+0000","last_peered":"2024-01-15T18:23:23.453210+0000","last_clean":"2024-01-15T18:23:23.453210+0000","last_became_active":"2024-01-15T18:18:33.162679+0000","last_became_peered":"2024-01-15T18:18:33.162679+0000","last_unstale":"2024-01-15T18:23:23.453210+0000","last_undegraded":"2024-01-15T18:23:23.453210+0000","last_fullsized":"2024-01-15T18:23:23.453210+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:53:15.036059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"107'2","reported_seq":18,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:50.167169+0000","last_change":"2024-01-15T18:18:32.214990+0000","last_active":"2024-01-15T18:20:50.167169+0000","last_peered":"2024-01-15T18:20:50.167169+0000","last_clean":"2024-01-15T18:20:50.167169+0000","last_became_active":"2024-01-15T18:18:32.214695+0000","last_became_peered":"2024-01-15T18:18:32.214695+0000","last_unstale":"2024-01-15T18:20:50.167169+0000","last_undegraded":"2024-01-15T18:20:50.167169+0000","last_fullsized":"2024-01-15T18:20:50.167169+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:15:08.445774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"110'41","reported_seq":63,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.450293+0000","last_change":"2024-01-15T18:18:33.168531+0000","last_active":"2024-01-15T18:23:23.450293+0000","last_peered":"2024-01-15T18:23:23.450293+0000","last_clean":"2024-01-15T18:23:23.450293+0000","last_became_active":"2024-01-15T18:18:33.168190+0000","last_became_peered":"2024-01-15T18:18:33.168190+0000","last_unstale":"2024-01-15T18:23:23.450293+0000","last_undegraded":"2024-01-15T18:23:23.450293+0000","last_fullsized":"2024-01-15T18:23:23.450293+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:53:05.325329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":7,"num_read_kb":12292,"num_write":19,"num_write_kb":61442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"107'1","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:50.487644+0000","last_change":"2024-01-15T18:18:32.221200+0000","last_active":"2024-01-15T18:20:50.487644+0000","last_peered":"2024-01-15T18:20:50.487644+0000","last_clean":"2024-01-15T18:20:50.487644+0000","last_became_active":"2024-01-15T18:18:32.220884+0000","last_became_peered":"2024-01-15T18:18:32.220884+0000","last_unstale":"2024-01-15T18:20:50.487644+0000","last_undegraded":"2024-01-15T18:20:50.487644+0000","last_fullsized":"2024-01-15T18:20:50.487644+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:02.938267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"110'40","reported_seq":60,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.454892+0000","last_change":"2024-01-15T18:18:33.163114+0000","last_active":"2024-01-15T18:23:23.454892+0000","last_peered":"2024-01-15T18:23:23.454892+0000","last_clean":"2024-01-15T18:23:23.454892+0000","last_became_active":"2024-01-15T18:18:33.162424+0000","last_became_peered":"2024-01-15T18:18:33.162424+0000","last_unstale":"2024-01-15T18:23:23.454892+0000","last_undegraded":"2024-01-15T18:23:23.454892+0000","last_fullsized":"2024-01-15T18:23:23.454892+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:42:08.107694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:49.171310+0000","last_change":"2024-01-15T18:18:32.217252+0000","last_active":"2024-01-15T18:20:49.171310+0000","last_peered":"2024-01-15T18:20:49.171310+0000","last_clean":"2024-01-15T18:20:49.171310+0000","last_became_active":"2024-01-15T18:18:32.216769+0000","last_became_peered":"2024-01-15T18:18:32.216769+0000","last_unstale":"2024-01-15T18:20:49.171310+0000","last_undegraded":"2024-01-15T18:20:49.171310+0000","last_fullsized":"2024-01-15T18:20:49.171310+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:36.807888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"110'34","reported_seq":53,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.456232+0000","last_change":"2024-01-15T18:18:33.163017+0000","last_active":"2024-01-15T18:23:23.456232+0000","last_peered":"2024-01-15T18:23:23.456232+0000","last_clean":"2024-01-15T18:23:23.456232+0000","last_became_active":"2024-01-15T18:18:33.162658+0000","last_became_peered":"2024-01-15T18:18:33.162658+0000","last_unstale":"2024-01-15T18:23:23.456232+0000","last_undegraded":"2024-01-15T18:23:23.456232+0000","last_fullsized":"2024-01-15T18:23:23.456232+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:01:41.498479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"110'3","reported_seq":19,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:50.469624+0000","last_change":"2024-01-15T18:18:32.207796+0000","last_active":"2024-01-15T18:20:50.469624+0000","last_peered":"2024-01-15T18:20:50.469624+0000","last_clean":"2024-01-15T18:20:50.469624+0000","last_became_active":"2024-01-15T18:18:32.207507+0000","last_became_peered":"2024-01-15T18:18:32.207507+0000","last_unstale":"2024-01-15T18:20:50.469624+0000","last_undegraded":"2024-01-15T18:20:50.469624+0000","last_fullsized":"2024-01-15T18:20:50.469624+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:42.592200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"110'39","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.460291+0000","last_change":"2024-01-15T18:18:33.164981+0000","last_active":"2024-01-15T18:23:23.460291+0000","last_peered":"2024-01-15T18:23:23.460291+0000","last_clean":"2024-01-15T18:23:23.460291+0000","last_became_active":"2024-01-15T18:18:33.164206+0000","last_became_peered":"2024-01-15T18:18:33.164206+0000","last_unstale":"2024-01-15T18:23:23.460291+0000","last_undegraded":"2024-01-15T18:23:23.460291+0000","last_fullsized":"2024-01-15T18:23:23.460291+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:53:32.550071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:50.167117+0000","last_change":"2024-01-15T18:18:32.212499+0000","last_active":"2024-01-15T18:20:50.167117+0000","last_peered":"2024-01-15T18:20:50.167117+0000","last_clean":"2024-01-15T18:20:50.167117+0000","last_became_active":"2024-01-15T18:18:32.212201+0000","last_became_peered":"2024-01-15T18:18:32.212201+0000","last_unstale":"2024-01-15T18:20:50.167117+0000","last_undegraded":"2024-01-15T18:20:50.167117+0000","last_fullsized":"2024-01-15T18:20:50.167117+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:25:39.031903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"110'40","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.453577+0000","last_change":"2024-01-15T18:18:33.165046+0000","last_active":"2024-01-15T18:23:23.453577+0000","last_peered":"2024-01-15T18:23:23.453577+0000","last_clean":"2024-01-15T18:23:23.453577+0000","last_became_active":"2024-01-15T18:18:33.164708+0000","last_became_peered":"2024-01-15T18:18:33.164708+0000","last_unstale":"2024-01-15T18:23:23.453577+0000","last_undegraded":"2024-01-15T18:23:23.453577+0000","last_fullsized":"2024-01-15T18:23:23.453577+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:46:54.687765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:45.468959+0000","last_change":"2024-01-15T18:18:32.207872+0000","last_active":"2024-01-15T18:20:45.468959+0000","last_peered":"2024-01-15T18:20:45.468959+0000","last_clean":"2024-01-15T18:20:45.468959+0000","last_became_active":"2024-01-15T18:18:32.207599+0000","last_became_peered":"2024-01-15T18:18:32.207599+0000","last_unstale":"2024-01-15T18:20:45.468959+0000","last_undegraded":"2024-01-15T18:20:45.468959+0000","last_fullsized":"2024-01-15T18:20:45.468959+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:25:20.009100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"110'40","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.450620+0000","last_change":"2024-01-15T18:18:33.160618+0000","last_active":"2024-01-15T18:23:23.450620+0000","last_peered":"2024-01-15T18:23:23.450620+0000","last_clean":"2024-01-15T18:23:23.450620+0000","last_became_active":"2024-01-15T18:18:33.160350+0000","last_became_peered":"2024-01-15T18:18:33.160350+0000","last_unstale":"2024-01-15T18:23:23.450620+0000","last_undegraded":"2024-01-15T18:23:23.450620+0000","last_fullsized":"2024-01-15T18:23:23.450620+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:42:11.703397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720517,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":15,"num_write_kb":57345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:44.740758+0000","last_change":"2024-01-15T18:18:32.217214+0000","last_active":"2024-01-15T18:20:44.740758+0000","last_peered":"2024-01-15T18:20:44.740758+0000","last_clean":"2024-01-15T18:20:44.740758+0000","last_became_active":"2024-01-15T18:18:32.216581+0000","last_became_peered":"2024-01-15T18:18:32.216581+0000","last_unstale":"2024-01-15T18:20:44.740758+0000","last_undegraded":"2024-01-15T18:20:44.740758+0000","last_fullsized":"2024-01-15T18:20:44.740758+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:49:26.780801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"110'33","reported_seq":50,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.444182+0000","last_change":"2024-01-15T18:18:33.165549+0000","last_active":"2024-01-15T18:23:23.444182+0000","last_peered":"2024-01-15T18:23:23.444182+0000","last_clean":"2024-01-15T18:23:23.444182+0000","last_became_active":"2024-01-15T18:18:33.164152+0000","last_became_peered":"2024-01-15T18:18:33.164152+0000","last_unstale":"2024-01-15T18:23:23.444182+0000","last_undegraded":"2024-01-15T18:23:23.444182+0000","last_fullsized":"2024-01-15T18:23:23.444182+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:24.034308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"107'2","reported_seq":18,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:45.435609+0000","last_change":"2024-01-15T18:18:32.219821+0000","last_active":"2024-01-15T18:20:45.435609+0000","last_peered":"2024-01-15T18:20:45.435609+0000","last_clean":"2024-01-15T18:20:45.435609+0000","last_became_active":"2024-01-15T18:18:32.218858+0000","last_became_peered":"2024-01-15T18:18:32.218858+0000","last_unstale":"2024-01-15T18:20:45.435609+0000","last_undegraded":"2024-01-15T18:20:45.435609+0000","last_fullsized":"2024-01-15T18:20:45.435609+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:33:51.257660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"110'37","reported_seq":54,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.462300+0000","last_change":"2024-01-15T18:18:33.160602+0000","last_active":"2024-01-15T18:23:23.462300+0000","last_peered":"2024-01-15T18:23:23.462300+0000","last_clean":"2024-01-15T18:23:23.462300+0000","last_became_active":"2024-01-15T18:18:33.160315+0000","last_became_peered":"2024-01-15T18:18:33.160315+0000","last_unstale":"2024-01-15T18:23:23.462300+0000","last_undegraded":"2024-01-15T18:23:23.462300+0000","last_fullsized":"2024-01-15T18:23:23.462300+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:34.698586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"110'16","reported_seq":32,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:22:55.502681+0000","last_change":"2024-01-15T18:18:32.217029+0000","last_active":"2024-01-15T18:22:55.502681+0000","last_peered":"2024-01-15T18:22:55.502681+0000","last_clean":"2024-01-15T18:22:55.502681+0000","last_became_active":"2024-01-15T18:18:32.216612+0000","last_became_peered":"2024-01-15T18:18:32.216612+0000","last_unstale":"2024-01-15T18:22:55.502681+0000","last_undegraded":"2024-01-15T18:22:55.502681+0000","last_fullsized":"2024-01-15T18:22:55.502681+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:42:10.693562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"110'42","reported_seq":63,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.452715+0000","last_change":"2024-01-15T18:18:33.165599+0000","last_active":"2024-01-15T18:23:23.452715+0000","last_peered":"2024-01-15T18:23:23.452715+0000","last_clean":"2024-01-15T18:23:23.452715+0000","last_became_active":"2024-01-15T18:18:33.163994+0000","last_became_peered":"2024-01-15T18:18:33.163994+0000","last_unstale":"2024-01-15T18:23:23.452715+0000","last_undegraded":"2024-01-15T18:23:23.452715+0000","last_fullsized":"2024-01-15T18:23:23.452715+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:00:26.779661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:39.740198+0000","last_change":"2024-01-15T18:18:32.216301+0000","last_active":"2024-01-15T18:20:39.740198+0000","last_peered":"2024-01-15T18:20:39.740198+0000","last_clean":"2024-01-15T18:20:39.740198+0000","last_became_active":"2024-01-15T18:18:32.216023+0000","last_became_peered":"2024-01-15T18:18:32.216023+0000","last_unstale":"2024-01-15T18:20:39.740198+0000","last_undegraded":"2024-01-15T18:20:39.740198+0000","last_fullsized":"2024-01-15T18:20:39.740198+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:58:58.730569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"110'34","reported_seq":60,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.444257+0000","last_change":"2024-01-15T18:18:42.262610+0000","last_active":"2024-01-15T18:23:23.444257+0000","last_peered":"2024-01-15T18:23:23.444257+0000","last_clean":"2024-01-15T18:23:23.444257+0000","last_became_active":"2024-01-15T18:18:42.262255+0000","last_became_peered":"2024-01-15T18:18:42.262255+0000","last_unstale":"2024-01-15T18:23:23.444257+0000","last_undegraded":"2024-01-15T18:23:23.444257+0000","last_fullsized":"2024-01-15T18:23:23.444257+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:22:41.967700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331770,"num_objects":14,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":5,"num_read_kb":20480,"num_write":14,"num_write_kb":49153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:40.434982+0000","last_change":"2024-01-15T18:18:32.214224+0000","last_active":"2024-01-15T18:20:40.434982+0000","last_peered":"2024-01-15T18:20:40.434982+0000","last_clean":"2024-01-15T18:20:40.434982+0000","last_became_active":"2024-01-15T18:18:32.213930+0000","last_became_peered":"2024-01-15T18:18:32.213930+0000","last_unstale":"2024-01-15T18:20:40.434982+0000","last_undegraded":"2024-01-15T18:20:40.434982+0000","last_fullsized":"2024-01-15T18:20:40.434982+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:01:15.328258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"110'44","reported_seq":60,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.454715+0000","last_change":"2024-01-15T18:18:33.150974+0000","last_active":"2024-01-15T18:23:23.454715+0000","last_peered":"2024-01-15T18:23:23.454715+0000","last_clean":"2024-01-15T18:23:23.454715+0000","last_became_active":"2024-01-15T18:18:33.150660+0000","last_became_peered":"2024-01-15T18:18:33.150660+0000","last_unstale":"2024-01-15T18:23:23.454715+0000","last_undegraded":"2024-01-15T18:23:23.454715+0000","last_fullsized":"2024-01-15T18:23:23.454715+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:22:41.967700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:39.170034+0000","last_change":"2024-01-15T18:18:32.215229+0000","last_active":"2024-01-15T18:20:39.170034+0000","last_peered":"2024-01-15T18:20:39.170034+0000","last_clean":"2024-01-15T18:20:39.170034+0000","last_became_active":"2024-01-15T18:18:32.214394+0000","last_became_peered":"2024-01-15T18:18:32.214394+0000","last_unstale":"2024-01-15T18:20:39.170034+0000","last_undegraded":"2024-01-15T18:20:39.170034+0000","last_fullsized":"2024-01-15T18:20:39.170034+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:30:25.477307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"110'51","reported_seq":72,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.458997+0000","last_change":"2024-01-15T18:18:33.156094+0000","last_active":"2024-01-15T18:23:23.458997+0000","last_peered":"2024-01-15T18:23:23.458997+0000","last_clean":"2024-01-15T18:23:23.458997+0000","last_became_active":"2024-01-15T18:18:33.155797+0000","last_became_peered":"2024-01-15T18:18:33.155797+0000","last_unstale":"2024-01-15T18:23:23.458997+0000","last_undegraded":"2024-01-15T18:23:23.458997+0000","last_fullsized":"2024-01-15T18:23:23.458997+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:21:42.101901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:40.178098+0000","last_change":"2024-01-15T18:18:32.212760+0000","last_active":"2024-01-15T18:20:40.178098+0000","last_peered":"2024-01-15T18:20:40.178098+0000","last_clean":"2024-01-15T18:20:40.178098+0000","last_became_active":"2024-01-15T18:18:32.212487+0000","last_became_peered":"2024-01-15T18:18:32.212487+0000","last_unstale":"2024-01-15T18:20:40.178098+0000","last_undegraded":"2024-01-15T18:20:40.178098+0000","last_fullsized":"2024-01-15T18:20:40.178098+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:48:56.885346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"110'38","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.450335+0000","last_change":"2024-01-15T18:18:33.167732+0000","last_active":"2024-01-15T18:23:23.450335+0000","last_peered":"2024-01-15T18:23:23.450335+0000","last_clean":"2024-01-15T18:23:23.450335+0000","last_became_active":"2024-01-15T18:18:33.167436+0000","last_became_peered":"2024-01-15T18:18:33.167436+0000","last_unstale":"2024-01-15T18:23:23.450335+0000","last_undegraded":"2024-01-15T18:23:23.450335+0000","last_fullsized":"2024-01-15T18:23:23.450335+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:22:13.319815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"110'11","reported_seq":37,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:21:26.348323+0000","last_change":"2024-01-15T18:18:32.213077+0000","last_active":"2024-01-15T18:21:26.348323+0000","last_peered":"2024-01-15T18:21:26.348323+0000","last_clean":"2024-01-15T18:21:26.348323+0000","last_became_active":"2024-01-15T18:18:32.211744+0000","last_became_peered":"2024-01-15T18:18:32.211744+0000","last_unstale":"2024-01-15T18:21:26.348323+0000","last_undegraded":"2024-01-15T18:21:26.348323+0000","last_fullsized":"2024-01-15T18:21:26.348323+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:48:24.967174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1321,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"110'36","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.452282+0000","last_change":"2024-01-15T18:18:33.163355+0000","last_active":"2024-01-15T18:23:23.452282+0000","last_peered":"2024-01-15T18:23:23.452282+0000","last_clean":"2024-01-15T18:23:23.452282+0000","last_became_active":"2024-01-15T18:18:33.162521+0000","last_became_peered":"2024-01-15T18:18:33.162521+0000","last_unstale":"2024-01-15T18:23:23.452282+0000","last_undegraded":"2024-01-15T18:23:23.452282+0000","last_fullsized":"2024-01-15T18:23:23.452282+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:35:00.923337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:34.739313+0000","last_change":"2024-01-15T18:18:32.217455+0000","last_active":"2024-01-15T18:20:34.739313+0000","last_peered":"2024-01-15T18:20:34.739313+0000","last_clean":"2024-01-15T18:20:34.739313+0000","last_became_active":"2024-01-15T18:18:32.216966+0000","last_became_peered":"2024-01-15T18:18:32.216966+0000","last_unstale":"2024-01-15T18:20:34.739313+0000","last_undegraded":"2024-01-15T18:20:34.739313+0000","last_fullsized":"2024-01-15T18:20:34.739313+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:56:34.922952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"110'30","reported_seq":50,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.458512+0000","last_change":"2024-01-15T18:18:33.166946+0000","last_active":"2024-01-15T18:23:23.458512+0000","last_peered":"2024-01-15T18:23:23.458512+0000","last_clean":"2024-01-15T18:23:23.458512+0000","last_became_active":"2024-01-15T18:18:33.165323+0000","last_became_peered":"2024-01-15T18:18:33.165323+0000","last_unstale":"2024-01-15T18:23:23.458512+0000","last_undegraded":"2024-01-15T18:23:23.458512+0000","last_fullsized":"2024-01-15T18:23:23.458512+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:18.639432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:54.171847+0000","last_change":"2024-01-15T18:18:32.216251+0000","last_active":"2024-01-15T18:20:54.171847+0000","last_peered":"2024-01-15T18:20:54.171847+0000","last_clean":"2024-01-15T18:20:54.171847+0000","last_became_active":"2024-01-15T18:18:32.215973+0000","last_became_peered":"2024-01-15T18:18:32.215973+0000","last_unstale":"2024-01-15T18:20:54.171847+0000","last_undegraded":"2024-01-15T18:20:54.171847+0000","last_fullsized":"2024-01-15T18:20:54.171847+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:41:26.812763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":244,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:50.171227+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:20:50.171227+0000","last_peered":"2024-01-15T18:20:50.171227+0000","last_clean":"2024-01-15T18:20:50.171227+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:20:50.171227+0000","last_undegraded":"2024-01-15T18:20:50.171227+0000","last_fullsized":"2024-01-15T18:20:50.171227+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":4353688568,"num_objects":1044,"num_object_clones":1,"num_object_copies":2088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1044,"num_whiteouts":1,"num_read":292,"num_read_kb":1142797,"num_write":1064,"num_write_kb":4251663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8761942016,"data_stored":8761903088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2568,"ondisk_log_size":2568,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":569436,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":260,"num_write_kb":669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1220608,"data_stored":1154557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":241,"ondisk_log_size":241,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509668,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1465860,"kb_used_data":1398652,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92905980,"statfs":{"total":96636764160,"available":95135723520,"internally_reserved":0,"allocated":1432219648,"data_stored":1431443111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:22:31 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.71499999999999997,"15min":0.90300000000000002},"min":{"1min":0.437,"5min":0.36799999999999999,"15min":0.35499999999999998},"max":{"1min":2.8620000000000001,"5min":2.8620000000000001,"15min":10.284000000000001},"last":0.63},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.67600000000000005,"15min":0.92800000000000005},"min":{"1min":0.54900000000000004,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.90000000000000002,"5min":0.92700000000000005,"15min":16.968},"last":0.67900000000000005}]},{"osd":1,"last update":"Mon Jan 15 18:22:31 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.69599999999999995,"15min":0.93400000000000005},"min":{"1min":0.57499999999999996,"5min":0.38600000000000001,"15min":0.33800000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":15.791},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.72699999999999998,"15min":0.92900000000000005},"min":{"1min":0.46999999999999997,"5min":0.44,"15min":0.371},"max":{"1min":1.248,"5min":1.248,"15min":10.842000000000001},"last":0.58299999999999996}]},{"osd":2,"last update":"Mon Jan 15 18:22:31 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.71299999999999997,"15min":0.95999999999999996},"min":{"1min":0.52600000000000002,"5min":0.39500000000000002,"15min":0.378},"max":{"1min":0.98299999999999998,"5min":0.98799999999999999,"15min":22.318000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.878,"5min":0.73399999999999999,"15min":0.91300000000000003},"min":{"1min":0.59699999999999998,"5min":0.42699999999999999,"15min":0.39100000000000001},"max":{"1min":2.2799999999999998,"5min":2.2799999999999998,"15min":9.7539999999999996},"last":0.69299999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:22:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.72799999999999998,"15min":1.0109999999999999},"min":{"1min":0.59599999999999997,"5min":0.46500000000000002,"15min":0.38100000000000001},"max":{"1min":1.2110000000000001,"5min":2.1739999999999999,"15min":18.152999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.76900000000000002,"15min":0.98999999999999999},"min":{"1min":0.497,"5min":0.495,"15min":0.433},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":10.34},"last":0.51600000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:22:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.57199999999999995,"15min":0.93100000000000005},"min":{"1min":0.40799999999999997,"5min":0.34799999999999998,"15min":0.314},"max":{"1min":0.81000000000000005,"5min":0.91000000000000003,"15min":20.387},"last":0.497},{"interface":"front","average":{"1min":0.623,"5min":0.59499999999999997,"15min":0.96299999999999997},"min":{"1min":0.42499999999999999,"5min":0.38400000000000001,"15min":0.34200000000000003},"max":{"1min":0.82699999999999996,"5min":0.98099999999999998,"15min":17.215},"last":0.65800000000000003}]},{"osd":5,"last update":"Mon Jan 15 18:22:31 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.60799999999999998,"15min":0.91500000000000004},"min":{"1min":0.5,"5min":0.38600000000000001,"15min":0.34699999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":19.959},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63500000000000001,"15min":0.95999999999999996},"min":{"1min":0.46300000000000002,"5min":0.40899999999999997,"15min":0.35799999999999998},"max":{"1min":0.82699999999999996,"5min":0.92600000000000005,"15min":19.905000000000001},"last":0.64500000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:22:31 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.73699999999999999,"15min":1.1419999999999999},"min":{"1min":0.57399999999999995,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.91000000000000003,"5min":0.99099999999999999,"15min":22.478999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.68899999999999995,"15min":1.0509999999999999},"min":{"1min":0.53400000000000003,"5min":0.44600000000000001,"15min":0.38300000000000001},"max":{"1min":0.96299999999999997,"5min":1.012,"15min":21.509},"last":0.71599999999999997}]}]},{"osd":6,"up_from":15,"seq":64424509668,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":998272,"kb_used_data":935224,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":93373568,"statfs":{"total":96636764160,"available":95614533632,"internally_reserved":0,"allocated":957669376,"data_stored":956895967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":0.78400000000000003,"15min":0.99399999999999999},"min":{"1min":0.59099999999999997,"5min":0.36299999999999999,"15min":0.29099999999999998},"max":{"1min":6.1900000000000004,"5min":6.1900000000000004,"15min":32.587000000000003},"last":0.753},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.79100000000000004,"15min":0.90100000000000002},"min":{"1min":0.63900000000000001,"5min":0.434,"15min":0.35399999999999998},"max":{"1min":4.9649999999999999,"5min":4.9649999999999999,"15min":16.193000000000001},"last":0.54900000000000004}]},{"osd":1,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.73799999999999999,"15min":0.996},"min":{"1min":0.56599999999999995,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":53.451999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.73399999999999999,"15min":0.91600000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.88400000000000001,"5min":1.0189999999999999,"15min":32.512},"last":0.69199999999999995}]},{"osd":2,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77000000000000002,"15min":0.95099999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.41699999999999998},"max":{"1min":0.98099999999999998,"5min":1.016,"15min":32.472999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.996,"5min":0.84299999999999997,"15min":0.92100000000000004},"min":{"1min":0.57899999999999996,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":5.0410000000000004,"5min":5.0410000000000004,"15min":16.602},"last":0.71099999999999997}]},{"osd":3,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.82699999999999996,"15min":0.93600000000000005},"min":{"1min":0.64900000000000002,"5min":0.52500000000000002,"15min":0.46000000000000002},"max":{"1min":5.0940000000000003,"5min":5.0940000000000003,"15min":16.388000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.76000000000000001,"15min":1.0609999999999999},"min":{"1min":0.67800000000000005,"5min":0.505,"15min":0.432},"max":{"1min":1.004,"5min":1.04,"15min":53.398000000000003},"last":0.64000000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.67000000000000004,"15min":0.88800000000000001},"min":{"1min":0.41499999999999998,"5min":0.39000000000000001,"15min":0.378},"max":{"1min":6.3869999999999996,"5min":6.3869999999999996,"15min":32.549999999999997},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.68799999999999994,"15min":0.83599999999999997},"min":{"1min":0.45800000000000002,"5min":0.41399999999999998,"15min":0.371},"max":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":11.852},"last":0.49099999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.70499999999999996,"15min":1.052},"min":{"1min":0.433,"5min":0.42199999999999999,"15min":0.34300000000000003},"max":{"1min":6.8769999999999998,"5min":6.8769999999999998,"15min":53.229999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.747,"15min":1.0209999999999999},"min":{"1min":0.49199999999999999,"5min":0.44400000000000001,"15min":0.41799999999999998},"max":{"1min":6.1360000000000001,"5min":6.1360000000000001,"15min":32.404000000000003},"last":0.82199999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.749,"15min":1.0820000000000001},"min":{"1min":0.52200000000000002,"5min":0.46000000000000002,"15min":0.38600000000000001},"max":{"1min":5.1319999999999997,"5min":5.1319999999999997,"15min":45.070999999999998},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.71099999999999997,"15min":1.181},"min":{"1min":0.51100000000000001,"5min":0.47999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89100000000000001,"5min":0.999,"15min":53.338000000000001},"last":0.72999999999999998}]}]},{"osd":4,"up_from":15,"seq":64424509668,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1387388,"kb_used_data":1324340,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":92984452,"statfs":{"total":96636764160,"available":95216078848,"internally_reserved":0,"allocated":1356124160,"data_stored":1355354916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.69799999999999995,"15min":0.90600000000000003},"min":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.32900000000000001},"max":{"1min":0.96599999999999997,"5min":1.542,"15min":23.318000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.70299999999999996,"15min":0.81299999999999994},"min":{"1min":0.55000000000000004,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":14.723000000000001},"last":0.75800000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.72599999999999998,"15min":0.81399999999999995},"min":{"1min":0.46300000000000002,"5min":0.39100000000000001,"15min":0.38800000000000001},"max":{"1min":0.96199999999999997,"5min":1.919,"15min":9.5700000000000003},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.74199999999999999,"15min":0.94699999999999995},"min":{"1min":0.52600000000000002,"5min":0.44900000000000001,"15min":0.36899999999999999},"max":{"1min":0.996,"5min":1.429,"15min":22.408999999999999},"last":0.60799999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.79000000000000004,"15min":0.85299999999999998},"min":{"1min":0.50900000000000001,"5min":0.41899999999999998,"15min":0.39700000000000002},"max":{"1min":1.032,"5min":5.6260000000000003,"15min":9.7089999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.76100000000000001,"15min":1.018},"min":{"1min":0.59299999999999997,"5min":0.45000000000000001,"15min":0.39100000000000001},"max":{"1min":1.016,"5min":1.387,"15min":22.338999999999999},"last":0.52900000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.73999999999999999,"15min":0.88700000000000001},"min":{"1min":0.52300000000000002,"5min":0.46200000000000002,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":1.577,"15min":14.763999999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.82599999999999996,"15min":0.93600000000000005},"min":{"1min":0.65300000000000002,"5min":0.53600000000000003,"15min":0.48099999999999998},"max":{"1min":0.996,"5min":1.9610000000000001,"15min":9.6289999999999996},"last":0.56299999999999994}]},{"osd":5,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63300000000000001,"15min":0.94299999999999995},"min":{"1min":0.44,"5min":0.36499999999999999,"15min":0.34799999999999998},"max":{"1min":1.0149999999999999,"5min":1.1970000000000001,"15min":31.617999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.85799999999999998},"min":{"1min":0.40799999999999997,"5min":0.373,"15min":0.36599999999999999},"max":{"1min":0.86699999999999999,"5min":1.4910000000000001,"15min":17.145},"last":0.81799999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.72799999999999998,"15min":0.83899999999999997},"min":{"1min":0.60799999999999998,"5min":0.41799999999999998,"15min":0.37},"max":{"1min":0.89300000000000002,"5min":5.9489999999999998,"15min":11.124000000000001},"last":0.628},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.71099999999999997,"15min":1.2090000000000001},"min":{"1min":0.52300000000000002,"5min":0.42499999999999999,"15min":0.40200000000000002},"max":{"1min":1.0649999999999999,"5min":2.3149999999999999,"15min":32.308},"last":0.83899999999999997}]},{"osd":7,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.72699999999999998,"15min":1.0009999999999999},"min":{"1min":0.48099999999999998,"5min":0.45100000000000001,"15min":0.34799999999999998},"max":{"1min":1,"5min":2.4329999999999998,"15min":39.256},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78200000000000003,"15min":1.002},"min":{"1min":0.60699999999999998,"5min":0.499,"15min":0.46500000000000002},"max":{"1min":1.04,"5min":2.1909999999999998,"15min":26.573},"last":0.71699999999999997}]}]},{"osd":3,"up_from":15,"seq":64424509667,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1154552,"kb_used_data":1087344,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93217288,"statfs":{"total":96636764160,"available":95454502912,"internally_reserved":0,"allocated":1113440256,"data_stored":1112652932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.45100000000000001,"15min":0.498},"min":{"1min":0.28999999999999998,"5min":0.28199999999999997,"15min":0.245},"max":{"1min":0.69099999999999995,"5min":0.69699999999999995,"15min":5.8849999999999998},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.51000000000000001,"15min":0.621},"min":{"1min":0.41799999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.80000000000000004,"5min":1.4750000000000001,"15min":17.756},"last":0.52000000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51300000000000001,"15min":0.60499999999999998},"min":{"1min":0.33500000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60399999999999998,"5min":4.0099999999999998,"15min":12.307},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55200000000000005,"15min":0.59199999999999997},"min":{"1min":0.36199999999999999,"5min":0.34300000000000003,"15min":0.31900000000000001},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":6.0640000000000001},"last":0.54300000000000004}]},{"osd":2,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61399999999999999,"15min":0.71199999999999997},"min":{"1min":0.42799999999999999,"5min":0.35299999999999998,"15min":0.32600000000000001},"max":{"1min":0.82999999999999996,"5min":4.077,"15min":12.468999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.59099999999999997,"15min":0.68000000000000005},"min":{"1min":0.44,"5min":0.39100000000000001,"15min":0.35399999999999998},"max":{"1min":0.78000000000000003,"5min":0.81899999999999995,"15min":10.837999999999999},"last":0.63800000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78500000000000003,"15min":0.999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.375},"max":{"1min":1.1910000000000001,"5min":1.9379999999999999,"15min":17.853999999999999},"last":0.878},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.85999999999999999,"15min":0.98199999999999998},"min":{"1min":0.54900000000000004,"5min":0.49199999999999999,"15min":0.45200000000000001},"max":{"1min":1.0569999999999999,"5min":4.2430000000000003,"15min":13.31},"last":0.93999999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.78400000000000003,"15min":0.98899999999999999},"min":{"1min":0.496,"5min":0.439,"15min":0.439},"max":{"1min":1.1619999999999999,"5min":1.179,"15min":17.687999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.77800000000000002,"15min":0.99099999999999999},"min":{"1min":0.58599999999999997,"5min":0.53800000000000003,"15min":0.438},"max":{"1min":1.048,"5min":1.825,"15min":20.515999999999998},"last":0.78000000000000003}]},{"osd":6,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.80900000000000005,"15min":0.98499999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.432},"max":{"1min":1.1419999999999999,"5min":4.2800000000000002,"15min":23.100999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.80600000000000005,"15min":0.94799999999999995},"min":{"1min":0.69899999999999995,"5min":0.52200000000000002,"15min":0.5},"max":{"1min":1.0720000000000001,"5min":1.149,"15min":12.76},"last":0.80100000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82499999999999996,"15min":1.1850000000000001},"min":{"1min":0.625,"5min":0.56899999999999995,"15min":0.46800000000000003},"max":{"1min":1.0860000000000001,"5min":1.931,"15min":24.321999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.82899999999999996,"15min":1.0820000000000001},"min":{"1min":0.60899999999999999,"5min":0.497,"15min":0.497},"max":{"1min":1.2070000000000001,"5min":4.202,"15min":23.013000000000002},"last":0.90000000000000002}]}]},{"osd":2,"up_from":15,"seq":64424509667,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760768,"kb_used_data":693560,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93611072,"statfs":{"total":96636764160,"available":95857737728,"internally_reserved":0,"allocated":710205440,"data_stored":709433352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45500000000000002,"15min":0.50900000000000001},"min":{"1min":0.32300000000000001,"5min":0.31,"15min":0.247},"max":{"1min":0.55200000000000005,"5min":0.80800000000000005,"15min":5.1849999999999996},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.501,"15min":0.54600000000000004},"min":{"1min":0.36499999999999999,"5min":0.26700000000000002,"15min":0.252},"max":{"1min":0.73999999999999999,"5min":0.83899999999999997,"15min":7.516},"last":0.51700000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.56799999999999995,"15min":0.60999999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.29699999999999999},"max":{"1min":0.71899999999999997,"5min":3.044,"15min":10.907},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.48999999999999999,"15min":0.59599999999999997},"min":{"1min":0.34999999999999998,"5min":0.33100000000000002,"15min":0.28100000000000003},"max":{"1min":0.76400000000000001,"5min":0.76500000000000001,"15min":8.9190000000000005},"last":0.36399999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52500000000000002,"15min":0.60599999999999998},"min":{"1min":0.40799999999999997,"5min":0.35399999999999998,"15min":0.30499999999999999},"max":{"1min":0.70999999999999996,"5min":0.86499999999999999,"15min":12.185},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61899999999999999,"15min":0.67100000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.373},"max":{"1min":0.80700000000000005,"5min":0.89600000000000002,"15min":7.6760000000000002},"last":0.495}]},{"osd":4,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.80300000000000005,"15min":0.879},"min":{"1min":0.59999999999999998,"5min":0.54700000000000004,"15min":0.44},"max":{"1min":1.0660000000000001,"5min":1.22,"15min":8.6609999999999996},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.76800000000000002,"15min":0.91200000000000003},"min":{"1min":0.56399999999999995,"5min":0.46200000000000002,"15min":0.41599999999999998},"max":{"1min":1.008,"5min":1.9670000000000001,"15min":19.146999999999998},"last":0.65000000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.76000000000000001,"15min":0.90200000000000002},"min":{"1min":0.58299999999999996,"5min":0.53600000000000003,"15min":0.41899999999999998},"max":{"1min":1.093,"5min":1.111,"15min":31.710000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.81999999999999995,"15min":0.999},"min":{"1min":0.57399999999999995,"5min":0.51700000000000002,"15min":0.48199999999999998},"max":{"1min":0.99199999999999999,"5min":1.8240000000000001,"15min":22.539000000000001},"last":0.60699999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.78300000000000003,"15min":0.91000000000000003},"min":{"1min":0.56499999999999995,"5min":0.52900000000000003,"15min":0.40899999999999997},"max":{"1min":1.079,"5min":1.3979999999999999,"15min":9.4540000000000006},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.80100000000000005,"15min":0.90800000000000003},"min":{"1min":0.64300000000000002,"5min":0.5,"15min":0.47899999999999998},"max":{"1min":1.0309999999999999,"5min":1.873,"15min":18.614999999999998},"last":0.58599999999999997}]},{"osd":7,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.79800000000000004,"15min":0.96799999999999997},"min":{"1min":0.63200000000000001,"5min":0.50900000000000001,"15min":0.44400000000000001},"max":{"1min":1.0109999999999999,"5min":1.7729999999999999,"15min":11.606},"last":0.746},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.84699999999999998,"15min":0.998},"min":{"1min":0.627,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.105,"5min":1.611,"15min":17.013000000000002},"last":0.73199999999999998}]}]},{"osd":1,"up_from":15,"seq":64424509669,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":941004,"kb_used_data":873796,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93430836,"statfs":{"total":96636764160,"available":95673176064,"internally_reserved":0,"allocated":894767104,"data_stored":893983032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45400000000000001,"15min":0.58499999999999996},"min":{"1min":0.26800000000000002,"5min":0.25900000000000001,"15min":0.23699999999999999},"max":{"1min":0.63500000000000001,"5min":0.88100000000000001,"15min":17.396999999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.51600000000000001,"15min":0.626},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.26900000000000002},"max":{"1min":0.73699999999999999,"5min":0.82999999999999996,"15min":12.378},"last":0.42999999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51700000000000002,"15min":0.57699999999999996},"min":{"1min":0.33000000000000002,"5min":0.313,"15min":0.309},"max":{"1min":0.61699999999999999,"5min":0.878,"15min":6.4569999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.54300000000000004,"15min":0.67400000000000004},"min":{"1min":0.40100000000000002,"5min":0.34499999999999997,"15min":0.33300000000000002},"max":{"1min":0.73899999999999999,"5min":0.81299999999999994,"15min":18.827000000000002},"last":0.47299999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59799999999999998,"15min":0.67200000000000004},"min":{"1min":0.433,"5min":0.40600000000000003,"15min":0.33700000000000002},"max":{"1min":0.872,"5min":1.0580000000000001,"15min":5.915},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.53800000000000003,"15min":0.64800000000000002},"min":{"1min":0.34399999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.77500000000000002,"5min":0.85499999999999998,"15min":9.9719999999999995},"last":0.49399999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.77700000000000002,"15min":0.89400000000000002},"min":{"1min":0.50900000000000001,"5min":0.437,"15min":0.437},"max":{"1min":1.056,"5min":1.1060000000000001,"15min":9.9329999999999998},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.746,"5min":0.79000000000000004,"15min":0.88300000000000001},"min":{"1min":0.55600000000000005,"5min":0.52800000000000002,"15min":0.45800000000000002},"max":{"1min":0.95599999999999996,"5min":1.165,"15min":9.8610000000000007},"last":0.66800000000000004}]},{"osd":5,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.80200000000000005,"15min":0.96299999999999997},"min":{"1min":0.55000000000000004,"5min":0.51200000000000001,"15min":0.46100000000000002},"max":{"1min":0.97499999999999998,"5min":1.2050000000000001,"15min":22.594999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.78200000000000003,"15min":1.0129999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.443},"max":{"1min":0.97499999999999998,"5min":1.077,"15min":26.748999999999999},"last":0.83899999999999997}]},{"osd":6,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.78800000000000003,"15min":1.034},"min":{"1min":0.57899999999999996,"5min":0.53100000000000003,"15min":0.42899999999999999},"max":{"1min":1.095,"5min":1.218,"15min":35.112000000000002},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.81100000000000005,"15min":0.90900000000000003},"min":{"1min":0.62,"5min":0.501,"15min":0.501},"max":{"1min":0.92900000000000005,"5min":1.2090000000000001,"15min":8.3670000000000009},"last":0.68799999999999994}]},{"osd":7,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.80800000000000005,"15min":1.024},"min":{"1min":0.56399999999999995,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.141,"5min":1.1950000000000001,"15min":20.872},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.81599999999999995,"15min":0.97799999999999998},"min":{"1min":0.61499999999999999,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.92300000000000004,"5min":1.139,"15min":9.8640000000000008},"last":0.61899999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542373,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":904260,"kb_used_data":828732,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93467580,"statfs":{"total":96636764160,"available":95710801920,"internally_reserved":0,"allocated":848621568,"data_stored":847844597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.72299999999999998,"15min":0.754},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.29899999999999999},"max":{"1min":1.038,"5min":2.6920000000000002,"15min":5.1639999999999997},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.75800000000000001,"15min":0.83899999999999997},"min":{"1min":0.46899999999999997,"5min":0.36799999999999999,"15min":0.34899999999999998},"max":{"1min":0.98199999999999998,"5min":6.4290000000000003,"15min":9.359},"last":0.70999999999999996}]},{"osd":1,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.76800000000000002,"15min":0.93000000000000005},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.375},"max":{"1min":0.89900000000000002,"5min":6.5069999999999997,"15min":21.120000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.747,"15min":0.93899999999999995},"min":{"1min":0.48299999999999998,"5min":0.39900000000000002,"15min":0.38600000000000001},"max":{"1min":0.96499999999999997,"5min":2.6619999999999999,"15min":15.843},"last":0.64900000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.76800000000000002,"15min":0.86699999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.34999999999999998},"max":{"1min":0.95099999999999996,"5min":6.3339999999999996,"15min":8.7750000000000004},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.79900000000000004,"15min":0.98299999999999998},"min":{"1min":0.501,"5min":0.501,"15min":0.39900000000000002},"max":{"1min":0.95399999999999996,"5min":6.4619999999999997,"15min":21.201000000000001},"last":0.501}]},{"osd":3,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.753,"15min":0.84399999999999997},"min":{"1min":0.498,"5min":0.46000000000000002,"15min":0.37},"max":{"1min":0.99199999999999999,"5min":2.6299999999999999,"15min":7.7199999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.81100000000000005,"15min":0.91900000000000004},"min":{"1min":0.52800000000000002,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.0109999999999999,"5min":6.3730000000000002,"15min":8.7300000000000004},"last":0.61899999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.66500000000000004,"15min":0.83399999999999996},"min":{"1min":0.52600000000000002,"5min":0.42999999999999999,"15min":0.30399999999999999},"max":{"1min":1.012,"5min":3.161,"15min":10.962},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.70699999999999996,"15min":1.012},"min":{"1min":0.48799999999999999,"5min":0.41999999999999998,"15min":0.35099999999999998},"max":{"1min":0.88100000000000001,"5min":6.5460000000000003,"15min":26.146999999999998},"last":0.59599999999999997}]},{"osd":6,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.66500000000000004,"15min":0.76700000000000002},"min":{"1min":0.50700000000000001,"5min":0.41299999999999998,"15min":0.36099999999999999},"max":{"1min":0.871,"5min":2.589,"15min":6.6699999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.82399999999999995,"15min":0.99199999999999999},"min":{"1min":0.53800000000000003,"5min":0.45300000000000001,"15min":0.379},"max":{"1min":0.95699999999999996,"5min":14.007999999999999,"15min":30.488},"last":0.67500000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.72499999999999998,"15min":1.2010000000000001},"min":{"1min":0.56699999999999995,"5min":0.48899999999999999,"15min":0.42599999999999999},"max":{"1min":0.90800000000000003,"5min":3.0870000000000002,"15min":33.499000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.91000000000000003,"15min":1.0680000000000001},"min":{"1min":0.57099999999999995,"5min":0.45700000000000002,"15min":0.42899999999999999},"max":{"1min":0.93000000000000005,"5min":21.178999999999998,"15min":25.437999999999999},"last":0.69199999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542372,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1458240,"kb_used_data":1391032,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92913600,"statfs":{"total":96636764160,"available":95143526400,"internally_reserved":0,"allocated":1424416768,"data_stored":1423621757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.82999999999999996,"15min":0.749},"min":{"1min":0.33100000000000002,"5min":0.25800000000000001,"15min":0.245},"max":{"1min":0.59899999999999998,"5min":38.521999999999998,"15min":38.521999999999998},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.60099999999999998,"15min":0.68799999999999994},"min":{"1min":0.36699999999999999,"5min":0.32900000000000001,"15min":0.27000000000000002},"max":{"1min":0.753,"5min":8.0749999999999993,"15min":10.154},"last":0.34699999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.57699999999999996,"15min":0.80200000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.30399999999999999},"max":{"1min":0.78600000000000003,"5min":2.6909999999999998,"15min":25.905000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.67200000000000004,"15min":0.66900000000000004},"min":{"1min":0.307,"5min":0.307,"15min":0.27800000000000002},"max":{"1min":0.69199999999999995,"5min":18.247,"15min":18.247},"last":0.45300000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.76400000000000001,"15min":0.80300000000000005},"min":{"1min":0.32400000000000001,"5min":0.32000000000000001,"15min":0.29199999999999998},"max":{"1min":0.70999999999999996,"5min":21.451000000000001,"15min":21.451000000000001},"last":0.629},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61799999999999999,"15min":0.76600000000000001},"min":{"1min":0.47099999999999997,"5min":0.38900000000000001,"15min":0.36899999999999999},"max":{"1min":0.80100000000000005,"5min":1.7,"15min":10.210000000000001},"last":0.55600000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.79400000000000004,"15min":0.99199999999999999},"min":{"1min":0.52600000000000002,"5min":0.441,"15min":0.39100000000000001},"max":{"1min":1.1639999999999999,"5min":2.7850000000000001,"15min":22.814},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.80100000000000005,"5min":1.0389999999999999,"15min":1.105},"min":{"1min":0.64000000000000001,"5min":0.45500000000000002,"15min":0.42399999999999999},"max":{"1min":1.1379999999999999,"5min":27.027999999999999,"15min":27.027999999999999},"last":0.41299999999999998}]},{"osd":5,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.83199999999999996,"15min":0.98799999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.38400000000000001},"max":{"1min":1.218,"5min":6.6429999999999998,"15min":12.428000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.77700000000000002,"5min":1.0509999999999999,"15min":1.333},"min":{"1min":0.55400000000000005,"5min":0.47999999999999998,"15min":0.46100000000000002},"max":{"1min":1.0600000000000001,"5min":27.210999999999999,"15min":27.210999999999999},"last":0.66400000000000003}]},{"osd":6,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.86699999999999999,"15min":1.242},"min":{"1min":0.66400000000000003,"5min":0.50900000000000001,"15min":0.46700000000000003},"max":{"1min":0.98399999999999999,"5min":9.3140000000000001,"15min":31.059000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.82699999999999996,"15min":1.089},"min":{"1min":0.58899999999999997,"5min":0.52900000000000003,"15min":0.51500000000000001},"max":{"1min":1.024,"5min":2.8340000000000001,"15min":40.731000000000002},"last":0.68200000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.86299999999999999,"15min":1.2150000000000001},"min":{"1min":0.66500000000000004,"5min":0.52200000000000002,"15min":0.499},"max":{"1min":1.1539999999999999,"5min":5.2089999999999996,"15min":24.492999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.81599999999999995,"5min":1.097,"15min":1.1859999999999999},"min":{"1min":0.63500000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1200000000000001,"5min":27.081,"15min":27.081},"last":0.70099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":585728,"data_stored":567429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":1321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":598016,"data_stored":583072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1421877248,"data_stored":1421869497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":905977856,"data_stored":905970036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":708841472,"data_stored":708837568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1111498752,"data_stored":1111491082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1354764288,"data_stored":1354760453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":847253504,"data_stored":847249530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":981471232,"data_stored":981467258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1430257664,"data_stored":1430257664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:23:34.837 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:23:35.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.017+0000 7f7477000640 1 Processor -- start 2024-01-15T18:23:35.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.017+0000 7f7477000640 1 -- start start 2024-01-15T18:23:35.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.021+0000 7f7477000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f74700cc7c0 0x7f7470168070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:35.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.021+0000 7f7477000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74700d05b0 con 0x7f7470160b10 2024-01-15T18:23:35.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.021+0000 7f7477000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74700ce7d0 con 0x7f74700afe70 2024-01-15T18:23:35.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.021+0000 7f7477000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74700ce5c0 con 0x7f74700cc7c0 2024-01-15T18:23:35.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.021+0000 7f7475000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f74700cc7c0 0x7f7470168070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:35.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.021+0000 7f746fe00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7470160b10 0x7f74700cd7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42856/0 (socket says 172.21.15.27:42856) 2024-01-15T18:23:35.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.021+0000 7f746fe00640 1 -- 172.21.15.27:0/4224104624 learned_addr learned my addr 172.21.15.27:0/4224104624 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:35.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.021+0000 7f7475000640 1 -- 172.21.15.27:0/4224104624 >> v1:172.21.15.93:6789/0 conn(0x7f74700afe70 legacy=0x7f74700cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:35.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.021+0000 7f7475000640 1 -- 172.21.15.27:0/4224104624 >> v1:172.21.15.27:6789/0 conn(0x7f7470160b10 legacy=0x7f74700cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:35.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.021+0000 7f746ea00640 1 -- 172.21.15.27:0/4224104624 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637060586 0 0) 0x7f74700d05b0 con 0x7f7470160b10 2024-01-15T18:23:35.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.021+0000 7f7475000640 1 -- 172.21.15.27:0/4224104624 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74700cfc80 con 0x7f74700cc7c0 2024-01-15T18:23:35.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.025+0000 7f7475000640 1 --2- 172.21.15.27:0/4224104624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74700cc7c0 0x7f7470168070 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f7468009d90 tx=0x7f746801c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2caa838c2a9b5b95 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:35.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.025+0000 7f746ea00640 1 -- 172.21.15.27:0/4224104624 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74680a5910 con 0x7f74700cc7c0 2024-01-15T18:23:35.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.025+0000 7f746ea00640 1 -- 172.21.15.27:0/4224104624 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74680a5a70 con 0x7f74700cc7c0 2024-01-15T18:23:35.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.025+0000 7f746ea00640 1 -- 172.21.15.27:0/4224104624 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74680a5e00 con 0x7f74700cc7c0 2024-01-15T18:23:35.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.025+0000 7f7477000640 1 -- 172.21.15.27:0/4224104624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74700cc7c0 msgr2=0x7f7470168070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:35.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.025+0000 7f7477000640 1 --2- 172.21.15.27:0/4224104624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74700cc7c0 0x7f7470168070 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f7468009d90 tx=0x7f746801c850 comp rx=0 tx=0).stop 2024-01-15T18:23:35.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.025+0000 7f7477000640 1 -- 172.21.15.27:0/4224104624 shutdown_connections 2024-01-15T18:23:35.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.025+0000 7f7477000640 1 --2- 172.21.15.27:0/4224104624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74700cc7c0 0x7f7470168070 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.025+0000 7f7477000640 1 -- 172.21.15.27:0/4224104624 >> 172.21.15.27:0/4224104624 conn(0x7f74700d0f20 msgr2=0x7f74700d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:35.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.025+0000 7f7477000640 1 -- 172.21.15.27:0/4224104624 shutdown_connections 2024-01-15T18:23:35.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.025+0000 7f7477000640 1 -- 172.21.15.27:0/4224104624 wait complete. 2024-01-15T18:23:35.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.025+0000 7f7477000640 1 Processor -- start 2024-01-15T18:23:35.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.025+0000 7f7477000640 1 -- start start 2024-01-15T18:23:35.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.029+0000 7f7477000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f74700afe70 0x7f74700c74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:35.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.029+0000 7f7477000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74700cc7c0 0x7f74700c8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:35.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.029+0000 7f7477000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7470160b10 0x7f74700c8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:35.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.029+0000 7f7477000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7470155fe0 con 0x7f74700cc7c0 2024-01-15T18:23:35.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.029+0000 7f7477000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7470158260 con 0x7f7470160b10 2024-01-15T18:23:35.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.029+0000 7f7477000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7470157c50 con 0x7f74700afe70 2024-01-15T18:23:35.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.029+0000 7f746f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74700cc7c0 0x7f74700c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:35.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.029+0000 7f746f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74700cc7c0 0x7f74700c8410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47462/0 (socket says 172.21.15.27:47462) 2024-01-15T18:23:35.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.029+0000 7f746f400640 1 -- 172.21.15.27:0/769243914 learned_addr learned my addr 172.21.15.27:0/769243914 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:35.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.029+0000 7f746fe00640 1 --2- 172.21.15.27:0/769243914 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f74700afe70 0x7f74700c74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:35.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.029+0000 7f7475000640 1 --2- 172.21.15.27:0/769243914 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7470160b10 0x7f74700c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:35.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.029+0000 7f746f400640 1 -- 172.21.15.27:0/769243914 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f74700afe70 msgr2=0x7f74700c74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:35.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.029+0000 7f746f400640 1 --2- 172.21.15.27:0/769243914 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f74700afe70 0x7f74700c74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.029+0000 7f746f400640 1 -- 172.21.15.27:0/769243914 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7470160b10 msgr2=0x7f74700c8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:35.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.029+0000 7f746f400640 1 --2- 172.21.15.27:0/769243914 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7470160b10 0x7f74700c8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.029+0000 7f746f400640 1 -- 172.21.15.27:0/769243914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7468009a70 con 0x7f74700cc7c0 2024-01-15T18:23:35.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.029+0000 7f746f400640 1 --2- 172.21.15.27:0/769243914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74700cc7c0 0x7f74700c8410 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f745800b440 tx=0x7f745800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:35.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.029+0000 7f7467e00640 1 -- 172.21.15.27:0/769243914 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7458011890 con 0x7f74700cc7c0 2024-01-15T18:23:35.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.029+0000 7f7467e00640 1 -- 172.21.15.27:0/769243914 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74580119f0 con 0x7f74700cc7c0 2024-01-15T18:23:35.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.029+0000 7f7467e00640 1 -- 172.21.15.27:0/769243914 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7458011da0 con 0x7f74700cc7c0 2024-01-15T18:23:35.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.029+0000 7f7477000640 1 -- 172.21.15.27:0/769243914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7470156510 con 0x7f74700cc7c0 2024-01-15T18:23:35.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.029+0000 7f7477000640 1 -- 172.21.15.27:0/769243914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7470157ef0 con 0x7f74700cc7c0 2024-01-15T18:23:35.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.029+0000 7f7477000640 1 -- 172.21.15.27:0/769243914 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7470157a90 con 0x7f74700cc7c0 2024-01-15T18:23:35.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.033+0000 7f7467e00640 1 -- 172.21.15.27:0/769243914 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f7458004020 con 0x7f74700cc7c0 2024-01-15T18:23:35.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.033+0000 7f7467e00640 1 --2- 172.21.15.27:0/769243914 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7448067a60 0x7f7448069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:35.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.033+0000 7f7467e00640 1 -- 172.21.15.27:0/769243914 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f74580833d0 con 0x7f74700cc7c0 2024-01-15T18:23:35.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.033+0000 7f746fe00640 1 --2- 172.21.15.27:0/769243914 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7448067a60 0x7f7448069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:35.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.033+0000 7f746fe00640 1 --2- 172.21.15.27:0/769243914 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7448067a60 0x7f7448069f20 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f7460000f70 tx=0x7f7460007c10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:35.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.037+0000 7f7467e00640 1 -- 172.21.15.27:0/769243914 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7458051cf0 con 0x7f74700cc7c0 2024-01-15T18:23:35.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.189+0000 7f7477000640 1 -- 172.21.15.27:0/769243914 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f747004e850 con 0x7f7448067a60 2024-01-15T18:23:35.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.201+0000 7f7467e00640 1 -- 172.21.15.27:0/769243914 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+291438 (secure 0 0 0) 0x7f747004e850 con 0x7f7448067a60 2024-01-15T18:23:35.198 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:23:35.200 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:23:35.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.201+0000 7f7477000640 1 -- 172.21.15.27:0/769243914 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7448067a60 msgr2=0x7f7448069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:35.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.201+0000 7f7477000640 1 --2- 172.21.15.27:0/769243914 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7448067a60 0x7f7448069f20 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f7460000f70 tx=0x7f7460007c10 comp rx=0 tx=0).stop 2024-01-15T18:23:35.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.201+0000 7f7477000640 1 -- 172.21.15.27:0/769243914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74700cc7c0 msgr2=0x7f74700c8410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:35.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.201+0000 7f7477000640 1 --2- 172.21.15.27:0/769243914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74700cc7c0 0x7f74700c8410 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f745800b440 tx=0x7f745800b900 comp rx=0 tx=0).stop 2024-01-15T18:23:35.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.201+0000 7f7477000640 1 -- 172.21.15.27:0/769243914 shutdown_connections 2024-01-15T18:23:35.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.205+0000 7f7477000640 1 --2- 172.21.15.27:0/769243914 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7448067a60 0x7f7448069f20 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.205+0000 7f7477000640 1 --2- 172.21.15.27:0/769243914 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f7470160b10 0x7f74700c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.205+0000 7f7477000640 1 --2- 172.21.15.27:0/769243914 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74700cc7c0 0x7f74700c8410 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.205+0000 7f7477000640 1 --2- 172.21.15.27:0/769243914 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f74700afe70 0x7f74700c74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.205+0000 7f7477000640 1 -- 172.21.15.27:0/769243914 >> 172.21.15.27:0/769243914 conn(0x7f74700d0f20 msgr2=0x7f74700b0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:35.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.205+0000 7f7477000640 1 -- 172.21.15.27:0/769243914 shutdown_connections 2024-01-15T18:23:35.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.205+0000 7f7477000640 1 -- 172.21.15.27:0/769243914 wait complete. 2024-01-15T18:23:35.220 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":668,"stamp":"2024-01-15T18:23:34.457302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4354848372,"num_objects":1069,"num_object_clones":1,"num_object_copies":2138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1069,"num_whiteouts":1,"num_read":418,"num_read_kb":1142907,"num_write":1483,"num_write_kb":4254748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2926,"ondisk_log_size":2926,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":9070344,"kb_used_data":8532680,"kb_used_omap":12,"kb_used_meta":537587,"kb_avail":745904376,"statfs":{"total":773094113280,"available":763806081024,"internally_reserved":0,"allocated":8737464320,"data_stored":8731229664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":550489648},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":58720646,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":23,"num_read_kb":94208,"num_write":20,"num_write_kb":57348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003319"},"pg_stats":[{"pgid":"17.3f","version":"110'49","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.452952+0000","last_change":"2024-01-15T18:18:33.168792+0000","last_active":"2024-01-15T18:23:23.452952+0000","last_peered":"2024-01-15T18:23:23.452952+0000","last_clean":"2024-01-15T18:23:23.452952+0000","last_became_active":"2024-01-15T18:18:33.168500+0000","last_became_peered":"2024-01-15T18:18:33.168500+0000","last_unstale":"2024-01-15T18:23:23.452952+0000","last_undegraded":"2024-01-15T18:23:23.452952+0000","last_fullsized":"2024-01-15T18:23:23.452952+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:28:41.940111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"110'42","reported_seq":61,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.445536+0000","last_change":"2024-01-15T18:18:33.151630+0000","last_active":"2024-01-15T18:23:23.445536+0000","last_peered":"2024-01-15T18:23:23.445536+0000","last_clean":"2024-01-15T18:23:23.445536+0000","last_became_active":"2024-01-15T18:18:33.151209+0000","last_became_peered":"2024-01-15T18:18:33.151209+0000","last_unstale":"2024-01-15T18:23:23.445536+0000","last_undegraded":"2024-01-15T18:23:23.445536+0000","last_fullsized":"2024-01-15T18:23:23.445536+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:46:54.687765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"110'39","reported_seq":57,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.457520+0000","last_change":"2024-01-15T18:18:33.163094+0000","last_active":"2024-01-15T18:23:23.457520+0000","last_peered":"2024-01-15T18:23:23.457520+0000","last_clean":"2024-01-15T18:23:23.457520+0000","last_became_active":"2024-01-15T18:18:33.162700+0000","last_became_peered":"2024-01-15T18:18:33.162700+0000","last_unstale":"2024-01-15T18:23:23.457520+0000","last_undegraded":"2024-01-15T18:23:23.457520+0000","last_fullsized":"2024-01-15T18:23:23.457520+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:23:36.167165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":16,"num_write_kb":57345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"110'39","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.459915+0000","last_change":"2024-01-15T18:18:33.160786+0000","last_active":"2024-01-15T18:23:23.459915+0000","last_peered":"2024-01-15T18:23:23.459915+0000","last_clean":"2024-01-15T18:23:23.459915+0000","last_became_active":"2024-01-15T18:18:33.160460+0000","last_became_peered":"2024-01-15T18:18:33.160460+0000","last_unstale":"2024-01-15T18:23:23.459915+0000","last_undegraded":"2024-01-15T18:23:23.459915+0000","last_fullsized":"2024-01-15T18:23:23.459915+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:27:33.980071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"110'25","reported_seq":42,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.441694+0000","last_change":"2024-01-15T18:18:33.155936+0000","last_active":"2024-01-15T18:23:23.441694+0000","last_peered":"2024-01-15T18:23:23.441694+0000","last_clean":"2024-01-15T18:23:23.441694+0000","last_became_active":"2024-01-15T18:18:33.155656+0000","last_became_peered":"2024-01-15T18:18:33.155656+0000","last_unstale":"2024-01-15T18:23:23.441694+0000","last_undegraded":"2024-01-15T18:23:23.441694+0000","last_fullsized":"2024-01-15T18:23:23.441694+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:27:09.469255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"110'45","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.452122+0000","last_change":"2024-01-15T18:18:33.152032+0000","last_active":"2024-01-15T18:23:23.452122+0000","last_peered":"2024-01-15T18:23:23.452122+0000","last_clean":"2024-01-15T18:23:23.452122+0000","last_became_active":"2024-01-15T18:18:33.151701+0000","last_became_peered":"2024-01-15T18:18:33.151701+0000","last_unstale":"2024-01-15T18:23:23.452122+0000","last_undegraded":"2024-01-15T18:23:23.452122+0000","last_fullsized":"2024-01-15T18:23:23.452122+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:51.983410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"110'43","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.556273+0000","last_change":"2024-01-15T18:18:33.153882+0000","last_active":"2024-01-15T18:23:31.556273+0000","last_peered":"2024-01-15T18:23:31.556273+0000","last_clean":"2024-01-15T18:23:31.556273+0000","last_became_active":"2024-01-15T18:18:33.153264+0000","last_became_peered":"2024-01-15T18:18:33.153264+0000","last_unstale":"2024-01-15T18:23:31.556273+0000","last_undegraded":"2024-01-15T18:23:31.556273+0000","last_fullsized":"2024-01-15T18:23:31.556273+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:37:49.663025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"110'42","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.451406+0000","last_change":"2024-01-15T18:18:33.164714+0000","last_active":"2024-01-15T18:23:23.451406+0000","last_peered":"2024-01-15T18:23:23.451406+0000","last_clean":"2024-01-15T18:23:23.451406+0000","last_became_active":"2024-01-15T18:18:33.163896+0000","last_became_peered":"2024-01-15T18:18:33.163896+0000","last_unstale":"2024-01-15T18:23:23.451406+0000","last_undegraded":"2024-01-15T18:23:23.451406+0000","last_fullsized":"2024-01-15T18:23:23.451406+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:21:18.264946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":19,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"110'42","reported_seq":60,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.450283+0000","last_change":"2024-01-15T18:18:33.158682+0000","last_active":"2024-01-15T18:23:23.450283+0000","last_peered":"2024-01-15T18:23:23.450283+0000","last_clean":"2024-01-15T18:23:23.450283+0000","last_became_active":"2024-01-15T18:18:33.158399+0000","last_became_peered":"2024-01-15T18:18:33.158399+0000","last_unstale":"2024-01-15T18:23:23.450283+0000","last_undegraded":"2024-01-15T18:23:23.450283+0000","last_fullsized":"2024-01-15T18:23:23.450283+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:27:12.032383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"110'41","reported_seq":60,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.445619+0000","last_change":"2024-01-15T18:18:33.159936+0000","last_active":"2024-01-15T18:23:23.445619+0000","last_peered":"2024-01-15T18:23:23.445619+0000","last_clean":"2024-01-15T18:23:23.445619+0000","last_became_active":"2024-01-15T18:18:33.159562+0000","last_became_peered":"2024-01-15T18:18:33.159562+0000","last_unstale":"2024-01-15T18:23:23.445619+0000","last_undegraded":"2024-01-15T18:23:23.445619+0000","last_fullsized":"2024-01-15T18:23:23.445619+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:56:32.966770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":11,"num_write_kb":45056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"110'41","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.443258+0000","last_change":"2024-01-15T18:18:33.155247+0000","last_active":"2024-01-15T18:23:23.443258+0000","last_peered":"2024-01-15T18:23:23.443258+0000","last_clean":"2024-01-15T18:23:23.443258+0000","last_became_active":"2024-01-15T18:18:33.154972+0000","last_became_peered":"2024-01-15T18:18:33.154972+0000","last_unstale":"2024-01-15T18:23:23.443258+0000","last_undegraded":"2024-01-15T18:23:23.443258+0000","last_fullsized":"2024-01-15T18:23:23.443258+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:54:47.923576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"110'43","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.444775+0000","last_change":"2024-01-15T18:18:33.158336+0000","last_active":"2024-01-15T18:23:23.444775+0000","last_peered":"2024-01-15T18:23:23.444775+0000","last_clean":"2024-01-15T18:23:23.444775+0000","last_became_active":"2024-01-15T18:18:33.158064+0000","last_became_peered":"2024-01-15T18:18:33.158064+0000","last_unstale":"2024-01-15T18:23:23.444775+0000","last_undegraded":"2024-01-15T18:23:23.444775+0000","last_fullsized":"2024-01-15T18:23:23.444775+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:42:44.182465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":6,"num_read_kb":20481,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"110'34","reported_seq":58,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.451044+0000","last_change":"2024-01-15T18:18:33.163912+0000","last_active":"2024-01-15T18:23:23.451044+0000","last_peered":"2024-01-15T18:23:23.451044+0000","last_clean":"2024-01-15T18:23:23.451044+0000","last_became_active":"2024-01-15T18:18:33.163610+0000","last_became_peered":"2024-01-15T18:18:33.163610+0000","last_unstale":"2024-01-15T18:23:23.451044+0000","last_undegraded":"2024-01-15T18:23:23.451044+0000","last_fullsized":"2024-01-15T18:23:23.451044+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:32:43.468374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"110'43","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.451241+0000","last_change":"2024-01-15T18:18:33.169868+0000","last_active":"2024-01-15T18:23:23.451241+0000","last_peered":"2024-01-15T18:23:23.451241+0000","last_clean":"2024-01-15T18:23:23.451241+0000","last_became_active":"2024-01-15T18:18:33.169530+0000","last_became_peered":"2024-01-15T18:18:33.169530+0000","last_unstale":"2024-01-15T18:23:23.451241+0000","last_undegraded":"2024-01-15T18:23:23.451241+0000","last_fullsized":"2024-01-15T18:23:23.451241+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:18:22.473056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"110'29","reported_seq":47,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.453353+0000","last_change":"2024-01-15T18:18:33.151325+0000","last_active":"2024-01-15T18:23:23.453353+0000","last_peered":"2024-01-15T18:23:23.453353+0000","last_clean":"2024-01-15T18:23:23.453353+0000","last_became_active":"2024-01-15T18:18:33.151040+0000","last_became_peered":"2024-01-15T18:18:33.151040+0000","last_unstale":"2024-01-15T18:23:23.453353+0000","last_undegraded":"2024-01-15T18:23:23.453353+0000","last_fullsized":"2024-01-15T18:23:23.453353+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:57:08.165322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":7,"num_write_kb":28672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"110'40","reported_seq":57,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.459812+0000","last_change":"2024-01-15T18:18:33.166327+0000","last_active":"2024-01-15T18:23:23.459812+0000","last_peered":"2024-01-15T18:23:23.459812+0000","last_clean":"2024-01-15T18:23:23.459812+0000","last_became_active":"2024-01-15T18:18:33.166014+0000","last_became_peered":"2024-01-15T18:18:33.166014+0000","last_unstale":"2024-01-15T18:23:23.459812+0000","last_undegraded":"2024-01-15T18:23:23.459812+0000","last_fullsized":"2024-01-15T18:23:23.459812+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:37:50.428714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"110'39","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.456607+0000","last_change":"2024-01-15T18:18:33.168613+0000","last_active":"2024-01-15T18:23:23.456607+0000","last_peered":"2024-01-15T18:23:23.456607+0000","last_clean":"2024-01-15T18:23:23.456607+0000","last_became_active":"2024-01-15T18:18:33.167975+0000","last_became_peered":"2024-01-15T18:18:33.167975+0000","last_unstale":"2024-01-15T18:23:23.456607+0000","last_undegraded":"2024-01-15T18:23:23.456607+0000","last_fullsized":"2024-01-15T18:23:23.456607+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:22:13.319815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"110'47","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.456193+0000","last_change":"2024-01-15T18:18:33.155078+0000","last_active":"2024-01-15T18:23:23.456193+0000","last_peered":"2024-01-15T18:23:23.456193+0000","last_clean":"2024-01-15T18:23:23.456193+0000","last_became_active":"2024-01-15T18:18:33.154816+0000","last_became_peered":"2024-01-15T18:18:33.154816+0000","last_unstale":"2024-01-15T18:23:23.456193+0000","last_undegraded":"2024-01-15T18:23:23.456193+0000","last_fullsized":"2024-01-15T18:23:23.456193+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:48.739168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"110'40","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.454711+0000","last_change":"2024-01-15T18:18:33.164028+0000","last_active":"2024-01-15T18:23:23.454711+0000","last_peered":"2024-01-15T18:23:23.454711+0000","last_clean":"2024-01-15T18:23:23.454711+0000","last_became_active":"2024-01-15T18:18:33.163421+0000","last_became_peered":"2024-01-15T18:18:33.163421+0000","last_unstale":"2024-01-15T18:23:23.454711+0000","last_undegraded":"2024-01-15T18:23:23.454711+0000","last_fullsized":"2024-01-15T18:23:23.454711+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:38:31.615856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"110'46","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.456828+0000","last_change":"2024-01-15T18:18:33.160995+0000","last_active":"2024-01-15T18:23:23.456828+0000","last_peered":"2024-01-15T18:23:23.456828+0000","last_clean":"2024-01-15T18:23:23.456828+0000","last_became_active":"2024-01-15T18:18:33.160047+0000","last_became_peered":"2024-01-15T18:18:33.160047+0000","last_unstale":"2024-01-15T18:23:23.456828+0000","last_undegraded":"2024-01-15T18:23:23.456828+0000","last_fullsized":"2024-01-15T18:23:23.456828+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:39:53.520191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"110'50","reported_seq":76,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.458912+0000","last_change":"2024-01-15T18:18:33.152696+0000","last_active":"2024-01-15T18:23:23.458912+0000","last_peered":"2024-01-15T18:23:23.458912+0000","last_clean":"2024-01-15T18:23:23.458912+0000","last_became_active":"2024-01-15T18:18:33.152128+0000","last_became_peered":"2024-01-15T18:18:33.152128+0000","last_unstale":"2024-01-15T18:23:23.458912+0000","last_undegraded":"2024-01-15T18:23:23.458912+0000","last_fullsized":"2024-01-15T18:23:23.458912+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:08:18.706229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":11,"num_read_kb":36866,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"110'41","reported_seq":60,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.450505+0000","last_change":"2024-01-15T18:18:33.163614+0000","last_active":"2024-01-15T18:23:23.450505+0000","last_peered":"2024-01-15T18:23:23.450505+0000","last_clean":"2024-01-15T18:23:23.450505+0000","last_became_active":"2024-01-15T18:18:33.163252+0000","last_became_peered":"2024-01-15T18:18:33.163252+0000","last_unstale":"2024-01-15T18:23:23.450505+0000","last_undegraded":"2024-01-15T18:23:23.450505+0000","last_fullsized":"2024-01-15T18:23:23.450505+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:48.739168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":4,"num_read_kb":8194,"num_write":17,"num_write_kb":61441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"110'35","reported_seq":52,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.457056+0000","last_change":"2024-01-15T18:18:33.159896+0000","last_active":"2024-01-15T18:23:23.457056+0000","last_peered":"2024-01-15T18:23:23.457056+0000","last_clean":"2024-01-15T18:23:23.457056+0000","last_became_active":"2024-01-15T18:18:33.159557+0000","last_became_peered":"2024-01-15T18:18:33.159557+0000","last_unstale":"2024-01-15T18:23:23.457056+0000","last_undegraded":"2024-01-15T18:23:23.457056+0000","last_fullsized":"2024-01-15T18:23:23.457056+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:24.034308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":11,"num_write_kb":45056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"110'39","reported_seq":57,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.459003+0000","last_change":"2024-01-15T18:18:33.166766+0000","last_active":"2024-01-15T18:23:23.459003+0000","last_peered":"2024-01-15T18:23:23.459003+0000","last_clean":"2024-01-15T18:23:23.459003+0000","last_became_active":"2024-01-15T18:18:33.165600+0000","last_became_peered":"2024-01-15T18:18:33.165600+0000","last_unstale":"2024-01-15T18:23:23.459003+0000","last_undegraded":"2024-01-15T18:23:23.459003+0000","last_fullsized":"2024-01-15T18:23:23.459003+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:03.948102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"110'35","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.457832+0000","last_change":"2024-01-15T18:18:33.168720+0000","last_active":"2024-01-15T18:23:23.457832+0000","last_peered":"2024-01-15T18:23:23.457832+0000","last_clean":"2024-01-15T18:23:23.457832+0000","last_became_active":"2024-01-15T18:18:33.168349+0000","last_became_peered":"2024-01-15T18:18:33.168349+0000","last_unstale":"2024-01-15T18:23:23.457832+0000","last_undegraded":"2024-01-15T18:23:23.457832+0000","last_fullsized":"2024-01-15T18:23:23.457832+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:18:22.473056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":16386,"num_write":17,"num_write_kb":61441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"110'50","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.457272+0000","last_change":"2024-01-15T18:18:33.159262+0000","last_active":"2024-01-15T18:23:23.457272+0000","last_peered":"2024-01-15T18:23:23.457272+0000","last_clean":"2024-01-15T18:23:23.457272+0000","last_became_active":"2024-01-15T18:18:33.158946+0000","last_became_peered":"2024-01-15T18:18:33.158946+0000","last_unstale":"2024-01-15T18:23:23.457272+0000","last_undegraded":"2024-01-15T18:23:23.457272+0000","last_fullsized":"2024-01-15T18:23:23.457272+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:05:55.347898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"110'37","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.447552+0000","last_change":"2024-01-15T18:18:33.161814+0000","last_active":"2024-01-15T18:23:23.447552+0000","last_peered":"2024-01-15T18:23:23.447552+0000","last_clean":"2024-01-15T18:23:23.447552+0000","last_became_active":"2024-01-15T18:18:33.161463+0000","last_became_peered":"2024-01-15T18:18:33.161463+0000","last_unstale":"2024-01-15T18:23:23.447552+0000","last_undegraded":"2024-01-15T18:23:23.447552+0000","last_fullsized":"2024-01-15T18:23:23.447552+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:15:36.391855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886260,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4,"num_read_kb":12289,"num_write":21,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"110'31","reported_seq":49,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.450037+0000","last_change":"2024-01-15T18:18:33.162952+0000","last_active":"2024-01-15T18:23:23.450037+0000","last_peered":"2024-01-15T18:23:23.450037+0000","last_clean":"2024-01-15T18:23:23.450037+0000","last_became_active":"2024-01-15T18:18:33.162669+0000","last_became_peered":"2024-01-15T18:18:33.162669+0000","last_unstale":"2024-01-15T18:23:23.450037+0000","last_undegraded":"2024-01-15T18:23:23.450037+0000","last_fullsized":"2024-01-15T18:23:23.450037+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:47:59.088407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"110'38","reported_seq":57,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.456970+0000","last_change":"2024-01-15T18:18:33.160658+0000","last_active":"2024-01-15T18:23:23.456970+0000","last_peered":"2024-01-15T18:23:23.456970+0000","last_clean":"2024-01-15T18:23:23.456970+0000","last_became_active":"2024-01-15T18:18:33.160099+0000","last_became_peered":"2024-01-15T18:18:33.160099+0000","last_unstale":"2024-01-15T18:23:23.456970+0000","last_undegraded":"2024-01-15T18:23:23.456970+0000","last_fullsized":"2024-01-15T18:23:23.456970+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:47:59.088407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"110'41","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.812331+0000","last_change":"2024-01-15T18:18:33.161061+0000","last_active":"2024-01-15T18:23:31.812331+0000","last_peered":"2024-01-15T18:23:31.812331+0000","last_clean":"2024-01-15T18:23:31.812331+0000","last_became_active":"2024-01-15T18:18:33.160725+0000","last_became_peered":"2024-01-15T18:18:33.160725+0000","last_unstale":"2024-01-15T18:23:31.812331+0000","last_undegraded":"2024-01-15T18:23:31.812331+0000","last_fullsized":"2024-01-15T18:23:31.812331+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:38:07.236956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"110'59","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.456771+0000","last_change":"2024-01-15T18:18:33.159024+0000","last_active":"2024-01-15T18:23:23.456771+0000","last_peered":"2024-01-15T18:23:23.456771+0000","last_clean":"2024-01-15T18:23:23.456771+0000","last_became_active":"2024-01-15T18:18:33.158701+0000","last_became_peered":"2024-01-15T18:18:33.158701+0000","last_unstale":"2024-01-15T18:23:23.456771+0000","last_undegraded":"2024-01-15T18:23:23.456771+0000","last_fullsized":"2024-01-15T18:23:23.456771+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:50:16.189329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"110'50","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.452426+0000","last_change":"2024-01-15T18:18:33.152241+0000","last_active":"2024-01-15T18:23:23.452426+0000","last_peered":"2024-01-15T18:23:23.452426+0000","last_clean":"2024-01-15T18:23:23.452426+0000","last_became_active":"2024-01-15T18:18:33.151250+0000","last_became_peered":"2024-01-15T18:18:33.151250+0000","last_unstale":"2024-01-15T18:23:23.452426+0000","last_undegraded":"2024-01-15T18:23:23.452426+0000","last_fullsized":"2024-01-15T18:23:23.452426+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:45:02.525155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"110'29","reported_seq":48,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.445810+0000","last_change":"2024-01-15T18:18:33.154972+0000","last_active":"2024-01-15T18:23:23.445810+0000","last_peered":"2024-01-15T18:23:23.445810+0000","last_clean":"2024-01-15T18:23:23.445810+0000","last_became_active":"2024-01-15T18:18:33.154569+0000","last_became_peered":"2024-01-15T18:18:33.154569+0000","last_unstale":"2024-01-15T18:23:23.445810+0000","last_undegraded":"2024-01-15T18:23:23.445810+0000","last_fullsized":"2024-01-15T18:23:23.445810+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:29:02.332324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"107'4","reported_seq":20,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:39.170083+0000","last_change":"2024-01-15T18:18:32.217761+0000","last_active":"2024-01-15T18:20:39.170083+0000","last_peered":"2024-01-15T18:20:39.170083+0000","last_clean":"2024-01-15T18:20:39.170083+0000","last_became_active":"2024-01-15T18:18:32.216337+0000","last_became_peered":"2024-01-15T18:18:32.216337+0000","last_unstale":"2024-01-15T18:20:39.170083+0000","last_undegraded":"2024-01-15T18:20:39.170083+0000","last_fullsized":"2024-01-15T18:20:39.170083+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:01:18.991818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"110'44","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.453170+0000","last_change":"2024-01-15T18:18:33.167015+0000","last_active":"2024-01-15T18:23:23.453170+0000","last_peered":"2024-01-15T18:23:23.453170+0000","last_clean":"2024-01-15T18:23:23.453170+0000","last_became_active":"2024-01-15T18:18:33.166726+0000","last_became_peered":"2024-01-15T18:18:33.166726+0000","last_unstale":"2024-01-15T18:23:23.453170+0000","last_undegraded":"2024-01-15T18:23:23.453170+0000","last_fullsized":"2024-01-15T18:23:23.453170+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:48:57.895180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"107'1","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:40.169868+0000","last_change":"2024-01-15T18:18:32.219987+0000","last_active":"2024-01-15T18:20:40.169868+0000","last_peered":"2024-01-15T18:20:40.169868+0000","last_clean":"2024-01-15T18:20:40.169868+0000","last_became_active":"2024-01-15T18:18:32.219596+0000","last_became_peered":"2024-01-15T18:18:32.219596+0000","last_unstale":"2024-01-15T18:20:40.169868+0000","last_undegraded":"2024-01-15T18:20:40.169868+0000","last_fullsized":"2024-01-15T18:20:40.169868+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:23.024473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"110'40","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.451309+0000","last_change":"2024-01-15T18:18:33.150743+0000","last_active":"2024-01-15T18:23:23.451309+0000","last_peered":"2024-01-15T18:23:23.451309+0000","last_clean":"2024-01-15T18:23:23.451309+0000","last_became_active":"2024-01-15T18:18:33.150429+0000","last_became_peered":"2024-01-15T18:18:33.150429+0000","last_unstale":"2024-01-15T18:23:23.451309+0000","last_undegraded":"2024-01-15T18:23:23.451309+0000","last_fullsized":"2024-01-15T18:23:23.451309+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:33:18.994479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"110'3","reported_seq":18,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:19:10.235403+0000","last_change":"2024-01-15T18:18:32.213207+0000","last_active":"2024-01-15T18:19:10.235403+0000","last_peered":"2024-01-15T18:19:10.235403+0000","last_clean":"2024-01-15T18:19:10.235403+0000","last_became_active":"2024-01-15T18:18:32.212071+0000","last_became_peered":"2024-01-15T18:18:32.212071+0000","last_unstale":"2024-01-15T18:19:10.235403+0000","last_undegraded":"2024-01-15T18:19:10.235403+0000","last_fullsized":"2024-01-15T18:19:10.235403+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:29:14.612143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"110'51","reported_seq":75,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.454632+0000","last_change":"2024-01-15T18:18:33.165545+0000","last_active":"2024-01-15T18:23:23.454632+0000","last_peered":"2024-01-15T18:23:23.454632+0000","last_clean":"2024-01-15T18:23:23.454632+0000","last_became_active":"2024-01-15T18:18:33.165258+0000","last_became_peered":"2024-01-15T18:18:33.165258+0000","last_unstale":"2024-01-15T18:23:23.454632+0000","last_undegraded":"2024-01-15T18:23:23.454632+0000","last_fullsized":"2024-01-15T18:23:23.454632+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:05:07.453809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"107'2","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:18:42.257846+0000","last_change":"2024-01-15T18:18:32.217238+0000","last_active":"2024-01-15T18:18:42.257846+0000","last_peered":"2024-01-15T18:18:42.257846+0000","last_clean":"2024-01-15T18:18:42.257846+0000","last_became_active":"2024-01-15T18:18:32.216492+0000","last_became_peered":"2024-01-15T18:18:32.216492+0000","last_unstale":"2024-01-15T18:18:42.257846+0000","last_undegraded":"2024-01-15T18:18:42.257846+0000","last_fullsized":"2024-01-15T18:18:42.257846+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:40:25.010807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"110'36","reported_seq":54,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.447490+0000","last_change":"2024-01-15T18:18:33.157996+0000","last_active":"2024-01-15T18:23:23.447490+0000","last_peered":"2024-01-15T18:23:23.447490+0000","last_clean":"2024-01-15T18:23:23.447490+0000","last_became_active":"2024-01-15T18:18:33.157687+0000","last_became_peered":"2024-01-15T18:18:33.157687+0000","last_unstale":"2024-01-15T18:23:23.447490+0000","last_undegraded":"2024-01-15T18:23:23.447490+0000","last_fullsized":"2024-01-15T18:23:23.447490+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:26:16.219599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"107'1","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:21:04.173272+0000","last_change":"2024-01-15T18:18:32.215007+0000","last_active":"2024-01-15T18:21:04.173272+0000","last_peered":"2024-01-15T18:21:04.173272+0000","last_clean":"2024-01-15T18:21:04.173272+0000","last_became_active":"2024-01-15T18:18:32.214453+0000","last_became_peered":"2024-01-15T18:18:32.214453+0000","last_unstale":"2024-01-15T18:21:04.173272+0000","last_undegraded":"2024-01-15T18:21:04.173272+0000","last_fullsized":"2024-01-15T18:21:04.173272+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:58:50.916786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"110'44","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.457859+0000","last_change":"2024-01-15T18:18:33.153355+0000","last_active":"2024-01-15T18:23:23.457859+0000","last_peered":"2024-01-15T18:23:23.457859+0000","last_clean":"2024-01-15T18:23:23.457859+0000","last_became_active":"2024-01-15T18:18:33.153058+0000","last_became_peered":"2024-01-15T18:18:33.153058+0000","last_unstale":"2024-01-15T18:23:23.457859+0000","last_undegraded":"2024-01-15T18:23:23.457859+0000","last_fullsized":"2024-01-15T18:23:23.457859+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:21:18.264946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"107'1","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:18:43.638172+0000","last_change":"2024-01-15T18:18:32.215681+0000","last_active":"2024-01-15T18:18:43.638172+0000","last_peered":"2024-01-15T18:18:43.638172+0000","last_clean":"2024-01-15T18:18:43.638172+0000","last_became_active":"2024-01-15T18:18:32.215417+0000","last_became_peered":"2024-01-15T18:18:32.215417+0000","last_unstale":"2024-01-15T18:18:43.638172+0000","last_undegraded":"2024-01-15T18:18:43.638172+0000","last_fullsized":"2024-01-15T18:18:43.638172+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:37:03.114460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"110'45","reported_seq":60,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.446039+0000","last_change":"2024-01-15T18:18:43.024618+0000","last_active":"2024-01-15T18:23:23.446039+0000","last_peered":"2024-01-15T18:23:23.446039+0000","last_clean":"2024-01-15T18:23:23.446039+0000","last_became_active":"2024-01-15T18:18:43.024221+0000","last_became_peered":"2024-01-15T18:18:43.024221+0000","last_unstale":"2024-01-15T18:23:23.446039+0000","last_undegraded":"2024-01-15T18:23:23.446039+0000","last_fullsized":"2024-01-15T18:23:23.446039+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:27.549069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303360,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:21:05.181499+0000","last_change":"2024-01-15T18:18:32.217097+0000","last_active":"2024-01-15T18:21:05.181499+0000","last_peered":"2024-01-15T18:21:05.181499+0000","last_clean":"2024-01-15T18:21:05.181499+0000","last_became_active":"2024-01-15T18:18:32.215465+0000","last_became_peered":"2024-01-15T18:18:32.215465+0000","last_unstale":"2024-01-15T18:21:05.181499+0000","last_undegraded":"2024-01-15T18:21:05.181499+0000","last_fullsized":"2024-01-15T18:21:05.181499+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:47.729333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"110'43","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.553963+0000","last_change":"2024-01-15T18:18:33.154067+0000","last_active":"2024-01-15T18:23:31.553963+0000","last_peered":"2024-01-15T18:23:31.553963+0000","last_clean":"2024-01-15T18:23:31.553963+0000","last_became_active":"2024-01-15T18:18:33.153610+0000","last_became_peered":"2024-01-15T18:18:33.153610+0000","last_unstale":"2024-01-15T18:23:31.553963+0000","last_undegraded":"2024-01-15T18:23:31.553963+0000","last_fullsized":"2024-01-15T18:23:31.553963+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:08:41.862115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:59.742931+0000","last_change":"2024-01-15T18:18:32.219909+0000","last_active":"2024-01-15T18:20:59.742931+0000","last_peered":"2024-01-15T18:20:59.742931+0000","last_clean":"2024-01-15T18:20:59.742931+0000","last_became_active":"2024-01-15T18:18:32.219555+0000","last_became_peered":"2024-01-15T18:18:32.219555+0000","last_unstale":"2024-01-15T18:20:59.742931+0000","last_undegraded":"2024-01-15T18:20:59.742931+0000","last_fullsized":"2024-01-15T18:20:59.742931+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:17:21.343043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"110'42","reported_seq":61,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:30.677222+0000","last_change":"2024-01-15T18:18:33.156038+0000","last_active":"2024-01-15T18:23:30.677222+0000","last_peered":"2024-01-15T18:23:30.677222+0000","last_clean":"2024-01-15T18:23:30.677222+0000","last_became_active":"2024-01-15T18:18:33.155765+0000","last_became_peered":"2024-01-15T18:18:33.155765+0000","last_unstale":"2024-01-15T18:23:30.677222+0000","last_undegraded":"2024-01-15T18:23:30.677222+0000","last_fullsized":"2024-01-15T18:23:30.677222+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:58:43.210443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"110'1","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:21:00.168575+0000","last_change":"2024-01-15T18:18:32.212880+0000","last_active":"2024-01-15T18:21:00.168575+0000","last_peered":"2024-01-15T18:21:00.168575+0000","last_clean":"2024-01-15T18:21:00.168575+0000","last_became_active":"2024-01-15T18:18:32.212511+0000","last_became_peered":"2024-01-15T18:18:32.212511+0000","last_unstale":"2024-01-15T18:21:00.168575+0000","last_undegraded":"2024-01-15T18:21:00.168575+0000","last_fullsized":"2024-01-15T18:21:00.168575+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:58:38.191140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"110'35","reported_seq":53,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.444028+0000","last_change":"2024-01-15T18:18:33.164800+0000","last_active":"2024-01-15T18:23:23.444028+0000","last_peered":"2024-01-15T18:23:23.444028+0000","last_clean":"2024-01-15T18:23:23.444028+0000","last_became_active":"2024-01-15T18:18:33.164530+0000","last_became_peered":"2024-01-15T18:18:33.164530+0000","last_unstale":"2024-01-15T18:23:23.444028+0000","last_undegraded":"2024-01-15T18:23:23.444028+0000","last_fullsized":"2024-01-15T18:23:23.444028+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:55.175065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:21:00.437395+0000","last_change":"2024-01-15T18:18:32.216873+0000","last_active":"2024-01-15T18:21:00.437395+0000","last_peered":"2024-01-15T18:21:00.437395+0000","last_clean":"2024-01-15T18:21:00.437395+0000","last_became_active":"2024-01-15T18:18:32.216337+0000","last_became_peered":"2024-01-15T18:18:32.216337+0000","last_unstale":"2024-01-15T18:21:00.437395+0000","last_undegraded":"2024-01-15T18:21:00.437395+0000","last_fullsized":"2024-01-15T18:21:00.437395+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:02.176575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"110'41","reported_seq":61,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.460483+0000","last_change":"2024-01-15T18:18:33.148278+0000","last_active":"2024-01-15T18:23:23.460483+0000","last_peered":"2024-01-15T18:23:23.460483+0000","last_clean":"2024-01-15T18:23:23.460483+0000","last_became_active":"2024-01-15T18:18:33.148016+0000","last_became_peered":"2024-01-15T18:18:33.148016+0000","last_unstale":"2024-01-15T18:23:23.460483+0000","last_undegraded":"2024-01-15T18:23:23.460483+0000","last_fullsized":"2024-01-15T18:23:23.460483+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:24.034308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"110'12","reported_seq":28,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:21:26.352083+0000","last_change":"2024-01-15T18:18:32.214473+0000","last_active":"2024-01-15T18:21:26.352083+0000","last_peered":"2024-01-15T18:21:26.352083+0000","last_clean":"2024-01-15T18:21:26.352083+0000","last_became_active":"2024-01-15T18:18:32.214186+0000","last_became_peered":"2024-01-15T18:18:32.214186+0000","last_unstale":"2024-01-15T18:21:26.352083+0000","last_undegraded":"2024-01-15T18:21:26.352083+0000","last_fullsized":"2024-01-15T18:21:26.352083+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:15:38.528251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"110'43","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.563032+0000","last_change":"2024-01-15T18:18:33.159329+0000","last_active":"2024-01-15T18:23:31.563032+0000","last_peered":"2024-01-15T18:23:31.563032+0000","last_clean":"2024-01-15T18:23:31.563032+0000","last_became_active":"2024-01-15T18:18:33.158541+0000","last_became_peered":"2024-01-15T18:18:33.158541+0000","last_unstale":"2024-01-15T18:23:31.563032+0000","last_undegraded":"2024-01-15T18:23:31.563032+0000","last_fullsized":"2024-01-15T18:23:31.563032+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:53:07.912473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"110'170","reported_seq":186,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.466042+0000","last_change":"2024-01-15T18:18:32.218321+0000","last_active":"2024-01-15T18:23:23.466042+0000","last_peered":"2024-01-15T18:23:23.466042+0000","last_clean":"2024-01-15T18:23:23.466042+0000","last_became_active":"2024-01-15T18:18:32.218002+0000","last_became_peered":"2024-01-15T18:18:32.218002+0000","last_unstale":"2024-01-15T18:23:23.466042+0000","last_undegraded":"2024-01-15T18:23:23.466042+0000","last_fullsized":"2024-01-15T18:23:23.466042+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:12:40.196083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":567317,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":169,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"110'36","reported_seq":53,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.455178+0000","last_change":"2024-01-15T18:18:33.151414+0000","last_active":"2024-01-15T18:23:23.455178+0000","last_peered":"2024-01-15T18:23:23.455178+0000","last_clean":"2024-01-15T18:23:23.455178+0000","last_became_active":"2024-01-15T18:18:33.151132+0000","last_became_peered":"2024-01-15T18:18:33.151132+0000","last_unstale":"2024-01-15T18:23:23.455178+0000","last_undegraded":"2024-01-15T18:23:23.455178+0000","last_fullsized":"2024-01-15T18:23:23.455178+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:48:57.895180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"107'2","reported_seq":18,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:55.470178+0000","last_change":"2024-01-15T18:18:32.215335+0000","last_active":"2024-01-15T18:20:55.470178+0000","last_peered":"2024-01-15T18:20:55.470178+0000","last_clean":"2024-01-15T18:20:55.470178+0000","last_became_active":"2024-01-15T18:18:32.215073+0000","last_became_peered":"2024-01-15T18:18:32.215073+0000","last_unstale":"2024-01-15T18:20:55.470178+0000","last_undegraded":"2024-01-15T18:20:55.470178+0000","last_fullsized":"2024-01-15T18:20:55.470178+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:04:07.912459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"110'46","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.458337+0000","last_change":"2024-01-15T18:18:33.161775+0000","last_active":"2024-01-15T18:23:23.458337+0000","last_peered":"2024-01-15T18:23:23.458337+0000","last_clean":"2024-01-15T18:23:23.458337+0000","last_became_active":"2024-01-15T18:18:33.161493+0000","last_became_peered":"2024-01-15T18:18:33.161493+0000","last_unstale":"2024-01-15T18:23:23.458337+0000","last_undegraded":"2024-01-15T18:23:23.458337+0000","last_fullsized":"2024-01-15T18:23:23.458337+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:55.175065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:55.488186+0000","last_change":"2024-01-15T18:18:32.215225+0000","last_active":"2024-01-15T18:20:55.488186+0000","last_peered":"2024-01-15T18:20:55.488186+0000","last_clean":"2024-01-15T18:20:55.488186+0000","last_became_active":"2024-01-15T18:18:32.214942+0000","last_became_peered":"2024-01-15T18:18:32.214942+0000","last_unstale":"2024-01-15T18:20:55.488186+0000","last_undegraded":"2024-01-15T18:20:55.488186+0000","last_fullsized":"2024-01-15T18:20:55.488186+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:09:53.884883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"110'43","reported_seq":61,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.460487+0000","last_change":"2024-01-15T18:18:33.146254+0000","last_active":"2024-01-15T18:23:23.460487+0000","last_peered":"2024-01-15T18:23:23.460487+0000","last_clean":"2024-01-15T18:23:23.460487+0000","last_became_active":"2024-01-15T18:18:33.145973+0000","last_became_peered":"2024-01-15T18:18:33.145973+0000","last_unstale":"2024-01-15T18:23:23.460487+0000","last_undegraded":"2024-01-15T18:23:23.460487+0000","last_fullsized":"2024-01-15T18:23:23.460487+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:09:54.894718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080645,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":22,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"107'8","reported_seq":24,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:55.167804+0000","last_change":"2024-01-15T18:18:32.219579+0000","last_active":"2024-01-15T18:20:55.167804+0000","last_peered":"2024-01-15T18:20:55.167804+0000","last_clean":"2024-01-15T18:20:55.167804+0000","last_became_active":"2024-01-15T18:18:32.219280+0000","last_became_peered":"2024-01-15T18:18:32.219280+0000","last_unstale":"2024-01-15T18:20:55.167804+0000","last_undegraded":"2024-01-15T18:20:55.167804+0000","last_fullsized":"2024-01-15T18:20:55.167804+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:56:09.818155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"110'21","reported_seq":38,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.451431+0000","last_change":"2024-01-15T18:18:33.161674+0000","last_active":"2024-01-15T18:23:23.451431+0000","last_peered":"2024-01-15T18:23:23.451431+0000","last_clean":"2024-01-15T18:23:23.451431+0000","last_became_active":"2024-01-15T18:18:33.161329+0000","last_became_peered":"2024-01-15T18:18:33.161329+0000","last_unstale":"2024-01-15T18:23:23.451431+0000","last_undegraded":"2024-01-15T18:23:23.451431+0000","last_fullsized":"2024-01-15T18:23:23.451431+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:45:56.254343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":7,"num_write_kb":28672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"107'1","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:49.741619+0000","last_change":"2024-01-15T18:18:32.215848+0000","last_active":"2024-01-15T18:20:49.741619+0000","last_peered":"2024-01-15T18:20:49.741619+0000","last_clean":"2024-01-15T18:20:49.741619+0000","last_became_active":"2024-01-15T18:18:32.214009+0000","last_became_peered":"2024-01-15T18:18:32.214009+0000","last_unstale":"2024-01-15T18:20:49.741619+0000","last_undegraded":"2024-01-15T18:20:49.741619+0000","last_fullsized":"2024-01-15T18:20:49.741619+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:48:09.722577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"110'35","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.453210+0000","last_change":"2024-01-15T18:18:33.162974+0000","last_active":"2024-01-15T18:23:23.453210+0000","last_peered":"2024-01-15T18:23:23.453210+0000","last_clean":"2024-01-15T18:23:23.453210+0000","last_became_active":"2024-01-15T18:18:33.162679+0000","last_became_peered":"2024-01-15T18:18:33.162679+0000","last_unstale":"2024-01-15T18:23:23.453210+0000","last_undegraded":"2024-01-15T18:23:23.453210+0000","last_fullsized":"2024-01-15T18:23:23.453210+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:53:15.036059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"107'2","reported_seq":18,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:50.167169+0000","last_change":"2024-01-15T18:18:32.214990+0000","last_active":"2024-01-15T18:20:50.167169+0000","last_peered":"2024-01-15T18:20:50.167169+0000","last_clean":"2024-01-15T18:20:50.167169+0000","last_became_active":"2024-01-15T18:18:32.214695+0000","last_became_peered":"2024-01-15T18:18:32.214695+0000","last_unstale":"2024-01-15T18:20:50.167169+0000","last_undegraded":"2024-01-15T18:20:50.167169+0000","last_fullsized":"2024-01-15T18:20:50.167169+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:15:08.445774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"110'41","reported_seq":63,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.450293+0000","last_change":"2024-01-15T18:18:33.168531+0000","last_active":"2024-01-15T18:23:23.450293+0000","last_peered":"2024-01-15T18:23:23.450293+0000","last_clean":"2024-01-15T18:23:23.450293+0000","last_became_active":"2024-01-15T18:18:33.168190+0000","last_became_peered":"2024-01-15T18:18:33.168190+0000","last_unstale":"2024-01-15T18:23:23.450293+0000","last_undegraded":"2024-01-15T18:23:23.450293+0000","last_fullsized":"2024-01-15T18:23:23.450293+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:53:05.325329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":7,"num_read_kb":12292,"num_write":19,"num_write_kb":61442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"107'1","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:50.487644+0000","last_change":"2024-01-15T18:18:32.221200+0000","last_active":"2024-01-15T18:20:50.487644+0000","last_peered":"2024-01-15T18:20:50.487644+0000","last_clean":"2024-01-15T18:20:50.487644+0000","last_became_active":"2024-01-15T18:18:32.220884+0000","last_became_peered":"2024-01-15T18:18:32.220884+0000","last_unstale":"2024-01-15T18:20:50.487644+0000","last_undegraded":"2024-01-15T18:20:50.487644+0000","last_fullsized":"2024-01-15T18:20:50.487644+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:02.938267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"110'40","reported_seq":60,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.454892+0000","last_change":"2024-01-15T18:18:33.163114+0000","last_active":"2024-01-15T18:23:23.454892+0000","last_peered":"2024-01-15T18:23:23.454892+0000","last_clean":"2024-01-15T18:23:23.454892+0000","last_became_active":"2024-01-15T18:18:33.162424+0000","last_became_peered":"2024-01-15T18:18:33.162424+0000","last_unstale":"2024-01-15T18:23:23.454892+0000","last_undegraded":"2024-01-15T18:23:23.454892+0000","last_fullsized":"2024-01-15T18:23:23.454892+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:42:08.107694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:49.171310+0000","last_change":"2024-01-15T18:18:32.217252+0000","last_active":"2024-01-15T18:20:49.171310+0000","last_peered":"2024-01-15T18:20:49.171310+0000","last_clean":"2024-01-15T18:20:49.171310+0000","last_became_active":"2024-01-15T18:18:32.216769+0000","last_became_peered":"2024-01-15T18:18:32.216769+0000","last_unstale":"2024-01-15T18:20:49.171310+0000","last_undegraded":"2024-01-15T18:20:49.171310+0000","last_fullsized":"2024-01-15T18:20:49.171310+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:36.807888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"110'34","reported_seq":53,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.456232+0000","last_change":"2024-01-15T18:18:33.163017+0000","last_active":"2024-01-15T18:23:23.456232+0000","last_peered":"2024-01-15T18:23:23.456232+0000","last_clean":"2024-01-15T18:23:23.456232+0000","last_became_active":"2024-01-15T18:18:33.162658+0000","last_became_peered":"2024-01-15T18:18:33.162658+0000","last_unstale":"2024-01-15T18:23:23.456232+0000","last_undegraded":"2024-01-15T18:23:23.456232+0000","last_fullsized":"2024-01-15T18:23:23.456232+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:01:41.498479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"110'3","reported_seq":19,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:50.469624+0000","last_change":"2024-01-15T18:18:32.207796+0000","last_active":"2024-01-15T18:20:50.469624+0000","last_peered":"2024-01-15T18:20:50.469624+0000","last_clean":"2024-01-15T18:20:50.469624+0000","last_became_active":"2024-01-15T18:18:32.207507+0000","last_became_peered":"2024-01-15T18:18:32.207507+0000","last_unstale":"2024-01-15T18:20:50.469624+0000","last_undegraded":"2024-01-15T18:20:50.469624+0000","last_fullsized":"2024-01-15T18:20:50.469624+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:42.592200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"110'39","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.460291+0000","last_change":"2024-01-15T18:18:33.164981+0000","last_active":"2024-01-15T18:23:23.460291+0000","last_peered":"2024-01-15T18:23:23.460291+0000","last_clean":"2024-01-15T18:23:23.460291+0000","last_became_active":"2024-01-15T18:18:33.164206+0000","last_became_peered":"2024-01-15T18:18:33.164206+0000","last_unstale":"2024-01-15T18:23:23.460291+0000","last_undegraded":"2024-01-15T18:23:23.460291+0000","last_fullsized":"2024-01-15T18:23:23.460291+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:53:32.550071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:50.167117+0000","last_change":"2024-01-15T18:18:32.212499+0000","last_active":"2024-01-15T18:20:50.167117+0000","last_peered":"2024-01-15T18:20:50.167117+0000","last_clean":"2024-01-15T18:20:50.167117+0000","last_became_active":"2024-01-15T18:18:32.212201+0000","last_became_peered":"2024-01-15T18:18:32.212201+0000","last_unstale":"2024-01-15T18:20:50.167117+0000","last_undegraded":"2024-01-15T18:20:50.167117+0000","last_fullsized":"2024-01-15T18:20:50.167117+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:25:39.031903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"110'40","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.453577+0000","last_change":"2024-01-15T18:18:33.165046+0000","last_active":"2024-01-15T18:23:23.453577+0000","last_peered":"2024-01-15T18:23:23.453577+0000","last_clean":"2024-01-15T18:23:23.453577+0000","last_became_active":"2024-01-15T18:18:33.164708+0000","last_became_peered":"2024-01-15T18:18:33.164708+0000","last_unstale":"2024-01-15T18:23:23.453577+0000","last_undegraded":"2024-01-15T18:23:23.453577+0000","last_fullsized":"2024-01-15T18:23:23.453577+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:46:54.687765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:45.468959+0000","last_change":"2024-01-15T18:18:32.207872+0000","last_active":"2024-01-15T18:20:45.468959+0000","last_peered":"2024-01-15T18:20:45.468959+0000","last_clean":"2024-01-15T18:20:45.468959+0000","last_became_active":"2024-01-15T18:18:32.207599+0000","last_became_peered":"2024-01-15T18:18:32.207599+0000","last_unstale":"2024-01-15T18:20:45.468959+0000","last_undegraded":"2024-01-15T18:20:45.468959+0000","last_fullsized":"2024-01-15T18:20:45.468959+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:25:20.009100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"110'40","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.450620+0000","last_change":"2024-01-15T18:18:33.160618+0000","last_active":"2024-01-15T18:23:23.450620+0000","last_peered":"2024-01-15T18:23:23.450620+0000","last_clean":"2024-01-15T18:23:23.450620+0000","last_became_active":"2024-01-15T18:18:33.160350+0000","last_became_peered":"2024-01-15T18:18:33.160350+0000","last_unstale":"2024-01-15T18:23:23.450620+0000","last_undegraded":"2024-01-15T18:23:23.450620+0000","last_fullsized":"2024-01-15T18:23:23.450620+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:42:11.703397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720517,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":15,"num_write_kb":57345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:44.740758+0000","last_change":"2024-01-15T18:18:32.217214+0000","last_active":"2024-01-15T18:20:44.740758+0000","last_peered":"2024-01-15T18:20:44.740758+0000","last_clean":"2024-01-15T18:20:44.740758+0000","last_became_active":"2024-01-15T18:18:32.216581+0000","last_became_peered":"2024-01-15T18:18:32.216581+0000","last_unstale":"2024-01-15T18:20:44.740758+0000","last_undegraded":"2024-01-15T18:20:44.740758+0000","last_fullsized":"2024-01-15T18:20:44.740758+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:49:26.780801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"110'33","reported_seq":50,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.444182+0000","last_change":"2024-01-15T18:18:33.165549+0000","last_active":"2024-01-15T18:23:23.444182+0000","last_peered":"2024-01-15T18:23:23.444182+0000","last_clean":"2024-01-15T18:23:23.444182+0000","last_became_active":"2024-01-15T18:18:33.164152+0000","last_became_peered":"2024-01-15T18:18:33.164152+0000","last_unstale":"2024-01-15T18:23:23.444182+0000","last_undegraded":"2024-01-15T18:23:23.444182+0000","last_fullsized":"2024-01-15T18:23:23.444182+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:24.034308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"107'2","reported_seq":18,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:45.435609+0000","last_change":"2024-01-15T18:18:32.219821+0000","last_active":"2024-01-15T18:20:45.435609+0000","last_peered":"2024-01-15T18:20:45.435609+0000","last_clean":"2024-01-15T18:20:45.435609+0000","last_became_active":"2024-01-15T18:18:32.218858+0000","last_became_peered":"2024-01-15T18:18:32.218858+0000","last_unstale":"2024-01-15T18:20:45.435609+0000","last_undegraded":"2024-01-15T18:20:45.435609+0000","last_fullsized":"2024-01-15T18:20:45.435609+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:33:51.257660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"110'37","reported_seq":54,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.462300+0000","last_change":"2024-01-15T18:18:33.160602+0000","last_active":"2024-01-15T18:23:23.462300+0000","last_peered":"2024-01-15T18:23:23.462300+0000","last_clean":"2024-01-15T18:23:23.462300+0000","last_became_active":"2024-01-15T18:18:33.160315+0000","last_became_peered":"2024-01-15T18:18:33.160315+0000","last_unstale":"2024-01-15T18:23:23.462300+0000","last_undegraded":"2024-01-15T18:23:23.462300+0000","last_fullsized":"2024-01-15T18:23:23.462300+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:34.698586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"110'16","reported_seq":32,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:22:55.502681+0000","last_change":"2024-01-15T18:18:32.217029+0000","last_active":"2024-01-15T18:22:55.502681+0000","last_peered":"2024-01-15T18:22:55.502681+0000","last_clean":"2024-01-15T18:22:55.502681+0000","last_became_active":"2024-01-15T18:18:32.216612+0000","last_became_peered":"2024-01-15T18:18:32.216612+0000","last_unstale":"2024-01-15T18:22:55.502681+0000","last_undegraded":"2024-01-15T18:22:55.502681+0000","last_fullsized":"2024-01-15T18:22:55.502681+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:42:10.693562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"110'42","reported_seq":63,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.452715+0000","last_change":"2024-01-15T18:18:33.165599+0000","last_active":"2024-01-15T18:23:23.452715+0000","last_peered":"2024-01-15T18:23:23.452715+0000","last_clean":"2024-01-15T18:23:23.452715+0000","last_became_active":"2024-01-15T18:18:33.163994+0000","last_became_peered":"2024-01-15T18:18:33.163994+0000","last_unstale":"2024-01-15T18:23:23.452715+0000","last_undegraded":"2024-01-15T18:23:23.452715+0000","last_fullsized":"2024-01-15T18:23:23.452715+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:00:26.779661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:39.740198+0000","last_change":"2024-01-15T18:18:32.216301+0000","last_active":"2024-01-15T18:20:39.740198+0000","last_peered":"2024-01-15T18:20:39.740198+0000","last_clean":"2024-01-15T18:20:39.740198+0000","last_became_active":"2024-01-15T18:18:32.216023+0000","last_became_peered":"2024-01-15T18:18:32.216023+0000","last_unstale":"2024-01-15T18:20:39.740198+0000","last_undegraded":"2024-01-15T18:20:39.740198+0000","last_fullsized":"2024-01-15T18:20:39.740198+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:58:58.730569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"110'34","reported_seq":60,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.444257+0000","last_change":"2024-01-15T18:18:42.262610+0000","last_active":"2024-01-15T18:23:23.444257+0000","last_peered":"2024-01-15T18:23:23.444257+0000","last_clean":"2024-01-15T18:23:23.444257+0000","last_became_active":"2024-01-15T18:18:42.262255+0000","last_became_peered":"2024-01-15T18:18:42.262255+0000","last_unstale":"2024-01-15T18:23:23.444257+0000","last_undegraded":"2024-01-15T18:23:23.444257+0000","last_fullsized":"2024-01-15T18:23:23.444257+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:22:41.967700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331770,"num_objects":14,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":5,"num_read_kb":20480,"num_write":14,"num_write_kb":49153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:40.434982+0000","last_change":"2024-01-15T18:18:32.214224+0000","last_active":"2024-01-15T18:20:40.434982+0000","last_peered":"2024-01-15T18:20:40.434982+0000","last_clean":"2024-01-15T18:20:40.434982+0000","last_became_active":"2024-01-15T18:18:32.213930+0000","last_became_peered":"2024-01-15T18:18:32.213930+0000","last_unstale":"2024-01-15T18:20:40.434982+0000","last_undegraded":"2024-01-15T18:20:40.434982+0000","last_fullsized":"2024-01-15T18:20:40.434982+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:01:15.328258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"110'44","reported_seq":60,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.454715+0000","last_change":"2024-01-15T18:18:33.150974+0000","last_active":"2024-01-15T18:23:23.454715+0000","last_peered":"2024-01-15T18:23:23.454715+0000","last_clean":"2024-01-15T18:23:23.454715+0000","last_became_active":"2024-01-15T18:18:33.150660+0000","last_became_peered":"2024-01-15T18:18:33.150660+0000","last_unstale":"2024-01-15T18:23:23.454715+0000","last_undegraded":"2024-01-15T18:23:23.454715+0000","last_fullsized":"2024-01-15T18:23:23.454715+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:22:41.967700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:39.170034+0000","last_change":"2024-01-15T18:18:32.215229+0000","last_active":"2024-01-15T18:20:39.170034+0000","last_peered":"2024-01-15T18:20:39.170034+0000","last_clean":"2024-01-15T18:20:39.170034+0000","last_became_active":"2024-01-15T18:18:32.214394+0000","last_became_peered":"2024-01-15T18:18:32.214394+0000","last_unstale":"2024-01-15T18:20:39.170034+0000","last_undegraded":"2024-01-15T18:20:39.170034+0000","last_fullsized":"2024-01-15T18:20:39.170034+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:30:25.477307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"110'51","reported_seq":72,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.458997+0000","last_change":"2024-01-15T18:18:33.156094+0000","last_active":"2024-01-15T18:23:23.458997+0000","last_peered":"2024-01-15T18:23:23.458997+0000","last_clean":"2024-01-15T18:23:23.458997+0000","last_became_active":"2024-01-15T18:18:33.155797+0000","last_became_peered":"2024-01-15T18:18:33.155797+0000","last_unstale":"2024-01-15T18:23:23.458997+0000","last_undegraded":"2024-01-15T18:23:23.458997+0000","last_fullsized":"2024-01-15T18:23:23.458997+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:21:42.101901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:40.178098+0000","last_change":"2024-01-15T18:18:32.212760+0000","last_active":"2024-01-15T18:20:40.178098+0000","last_peered":"2024-01-15T18:20:40.178098+0000","last_clean":"2024-01-15T18:20:40.178098+0000","last_became_active":"2024-01-15T18:18:32.212487+0000","last_became_peered":"2024-01-15T18:18:32.212487+0000","last_unstale":"2024-01-15T18:20:40.178098+0000","last_undegraded":"2024-01-15T18:20:40.178098+0000","last_fullsized":"2024-01-15T18:20:40.178098+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:48:56.885346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"110'38","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.450335+0000","last_change":"2024-01-15T18:18:33.167732+0000","last_active":"2024-01-15T18:23:23.450335+0000","last_peered":"2024-01-15T18:23:23.450335+0000","last_clean":"2024-01-15T18:23:23.450335+0000","last_became_active":"2024-01-15T18:18:33.167436+0000","last_became_peered":"2024-01-15T18:18:33.167436+0000","last_unstale":"2024-01-15T18:23:23.450335+0000","last_undegraded":"2024-01-15T18:23:23.450335+0000","last_fullsized":"2024-01-15T18:23:23.450335+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:22:13.319815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"110'11","reported_seq":37,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:21:26.348323+0000","last_change":"2024-01-15T18:18:32.213077+0000","last_active":"2024-01-15T18:21:26.348323+0000","last_peered":"2024-01-15T18:21:26.348323+0000","last_clean":"2024-01-15T18:21:26.348323+0000","last_became_active":"2024-01-15T18:18:32.211744+0000","last_became_peered":"2024-01-15T18:18:32.211744+0000","last_unstale":"2024-01-15T18:21:26.348323+0000","last_undegraded":"2024-01-15T18:21:26.348323+0000","last_fullsized":"2024-01-15T18:21:26.348323+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:48:24.967174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1321,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"110'36","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.452282+0000","last_change":"2024-01-15T18:18:33.163355+0000","last_active":"2024-01-15T18:23:23.452282+0000","last_peered":"2024-01-15T18:23:23.452282+0000","last_clean":"2024-01-15T18:23:23.452282+0000","last_became_active":"2024-01-15T18:18:33.162521+0000","last_became_peered":"2024-01-15T18:18:33.162521+0000","last_unstale":"2024-01-15T18:23:23.452282+0000","last_undegraded":"2024-01-15T18:23:23.452282+0000","last_fullsized":"2024-01-15T18:23:23.452282+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:35:00.923337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:34.739313+0000","last_change":"2024-01-15T18:18:32.217455+0000","last_active":"2024-01-15T18:20:34.739313+0000","last_peered":"2024-01-15T18:20:34.739313+0000","last_clean":"2024-01-15T18:20:34.739313+0000","last_became_active":"2024-01-15T18:18:32.216966+0000","last_became_peered":"2024-01-15T18:18:32.216966+0000","last_unstale":"2024-01-15T18:20:34.739313+0000","last_undegraded":"2024-01-15T18:20:34.739313+0000","last_fullsized":"2024-01-15T18:20:34.739313+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:56:34.922952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"110'30","reported_seq":50,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.458512+0000","last_change":"2024-01-15T18:18:33.166946+0000","last_active":"2024-01-15T18:23:23.458512+0000","last_peered":"2024-01-15T18:23:23.458512+0000","last_clean":"2024-01-15T18:23:23.458512+0000","last_became_active":"2024-01-15T18:18:33.165323+0000","last_became_peered":"2024-01-15T18:18:33.165323+0000","last_unstale":"2024-01-15T18:23:23.458512+0000","last_undegraded":"2024-01-15T18:23:23.458512+0000","last_fullsized":"2024-01-15T18:23:23.458512+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:18.639432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:54.171847+0000","last_change":"2024-01-15T18:18:32.216251+0000","last_active":"2024-01-15T18:20:54.171847+0000","last_peered":"2024-01-15T18:20:54.171847+0000","last_clean":"2024-01-15T18:20:54.171847+0000","last_became_active":"2024-01-15T18:18:32.215973+0000","last_became_peered":"2024-01-15T18:18:32.215973+0000","last_unstale":"2024-01-15T18:20:54.171847+0000","last_undegraded":"2024-01-15T18:20:54.171847+0000","last_fullsized":"2024-01-15T18:20:54.171847+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:41:26.812763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":244,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:50.171227+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:20:50.171227+0000","last_peered":"2024-01-15T18:20:50.171227+0000","last_clean":"2024-01-15T18:20:50.171227+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:20:50.171227+0000","last_undegraded":"2024-01-15T18:20:50.171227+0000","last_fullsized":"2024-01-15T18:20:50.171227+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":4353688568,"num_objects":1044,"num_object_clones":1,"num_object_copies":2088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1044,"num_whiteouts":1,"num_read":292,"num_read_kb":1142797,"num_write":1064,"num_write_kb":4251663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8761942016,"data_stored":8761903088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2568,"ondisk_log_size":2568,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":569436,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":260,"num_write_kb":669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1220608,"data_stored":1154557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":241,"ondisk_log_size":241,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509668,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1465860,"kb_used_data":1398652,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92905980,"statfs":{"total":96636764160,"available":95135723520,"internally_reserved":0,"allocated":1432219648,"data_stored":1431443111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:22:31 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.71499999999999997,"15min":0.90300000000000002},"min":{"1min":0.437,"5min":0.36799999999999999,"15min":0.35499999999999998},"max":{"1min":2.8620000000000001,"5min":2.8620000000000001,"15min":10.284000000000001},"last":0.63},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.67600000000000005,"15min":0.92800000000000005},"min":{"1min":0.54900000000000004,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.90000000000000002,"5min":0.92700000000000005,"15min":16.968},"last":0.67900000000000005}]},{"osd":1,"last update":"Mon Jan 15 18:22:31 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.69599999999999995,"15min":0.93400000000000005},"min":{"1min":0.57499999999999996,"5min":0.38600000000000001,"15min":0.33800000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":15.791},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.72699999999999998,"15min":0.92900000000000005},"min":{"1min":0.46999999999999997,"5min":0.44,"15min":0.371},"max":{"1min":1.248,"5min":1.248,"15min":10.842000000000001},"last":0.58299999999999996}]},{"osd":2,"last update":"Mon Jan 15 18:22:31 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.71299999999999997,"15min":0.95999999999999996},"min":{"1min":0.52600000000000002,"5min":0.39500000000000002,"15min":0.378},"max":{"1min":0.98299999999999998,"5min":0.98799999999999999,"15min":22.318000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.878,"5min":0.73399999999999999,"15min":0.91300000000000003},"min":{"1min":0.59699999999999998,"5min":0.42699999999999999,"15min":0.39100000000000001},"max":{"1min":2.2799999999999998,"5min":2.2799999999999998,"15min":9.7539999999999996},"last":0.69299999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:22:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.72799999999999998,"15min":1.0109999999999999},"min":{"1min":0.59599999999999997,"5min":0.46500000000000002,"15min":0.38100000000000001},"max":{"1min":1.2110000000000001,"5min":2.1739999999999999,"15min":18.152999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.76900000000000002,"15min":0.98999999999999999},"min":{"1min":0.497,"5min":0.495,"15min":0.433},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":10.34},"last":0.51600000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:22:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.57199999999999995,"15min":0.93100000000000005},"min":{"1min":0.40799999999999997,"5min":0.34799999999999998,"15min":0.314},"max":{"1min":0.81000000000000005,"5min":0.91000000000000003,"15min":20.387},"last":0.497},{"interface":"front","average":{"1min":0.623,"5min":0.59499999999999997,"15min":0.96299999999999997},"min":{"1min":0.42499999999999999,"5min":0.38400000000000001,"15min":0.34200000000000003},"max":{"1min":0.82699999999999996,"5min":0.98099999999999998,"15min":17.215},"last":0.65800000000000003}]},{"osd":5,"last update":"Mon Jan 15 18:22:31 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.60799999999999998,"15min":0.91500000000000004},"min":{"1min":0.5,"5min":0.38600000000000001,"15min":0.34699999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":19.959},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63500000000000001,"15min":0.95999999999999996},"min":{"1min":0.46300000000000002,"5min":0.40899999999999997,"15min":0.35799999999999998},"max":{"1min":0.82699999999999996,"5min":0.92600000000000005,"15min":19.905000000000001},"last":0.64500000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:22:31 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.73699999999999999,"15min":1.1419999999999999},"min":{"1min":0.57399999999999995,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.91000000000000003,"5min":0.99099999999999999,"15min":22.478999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.68899999999999995,"15min":1.0509999999999999},"min":{"1min":0.53400000000000003,"5min":0.44600000000000001,"15min":0.38300000000000001},"max":{"1min":0.96299999999999997,"5min":1.012,"15min":21.509},"last":0.71599999999999997}]}]},{"osd":6,"up_from":15,"seq":64424509668,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":998272,"kb_used_data":935224,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":93373568,"statfs":{"total":96636764160,"available":95614533632,"internally_reserved":0,"allocated":957669376,"data_stored":956895967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":0.78400000000000003,"15min":0.99399999999999999},"min":{"1min":0.59099999999999997,"5min":0.36299999999999999,"15min":0.29099999999999998},"max":{"1min":6.1900000000000004,"5min":6.1900000000000004,"15min":32.587000000000003},"last":0.753},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.79100000000000004,"15min":0.90100000000000002},"min":{"1min":0.63900000000000001,"5min":0.434,"15min":0.35399999999999998},"max":{"1min":4.9649999999999999,"5min":4.9649999999999999,"15min":16.193000000000001},"last":0.54900000000000004}]},{"osd":1,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.73799999999999999,"15min":0.996},"min":{"1min":0.56599999999999995,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":53.451999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.73399999999999999,"15min":0.91600000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.88400000000000001,"5min":1.0189999999999999,"15min":32.512},"last":0.69199999999999995}]},{"osd":2,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77000000000000002,"15min":0.95099999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.41699999999999998},"max":{"1min":0.98099999999999998,"5min":1.016,"15min":32.472999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.996,"5min":0.84299999999999997,"15min":0.92100000000000004},"min":{"1min":0.57899999999999996,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":5.0410000000000004,"5min":5.0410000000000004,"15min":16.602},"last":0.71099999999999997}]},{"osd":3,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.82699999999999996,"15min":0.93600000000000005},"min":{"1min":0.64900000000000002,"5min":0.52500000000000002,"15min":0.46000000000000002},"max":{"1min":5.0940000000000003,"5min":5.0940000000000003,"15min":16.388000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.76000000000000001,"15min":1.0609999999999999},"min":{"1min":0.67800000000000005,"5min":0.505,"15min":0.432},"max":{"1min":1.004,"5min":1.04,"15min":53.398000000000003},"last":0.64000000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.67000000000000004,"15min":0.88800000000000001},"min":{"1min":0.41499999999999998,"5min":0.39000000000000001,"15min":0.378},"max":{"1min":6.3869999999999996,"5min":6.3869999999999996,"15min":32.549999999999997},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.68799999999999994,"15min":0.83599999999999997},"min":{"1min":0.45800000000000002,"5min":0.41399999999999998,"15min":0.371},"max":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":11.852},"last":0.49099999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.70499999999999996,"15min":1.052},"min":{"1min":0.433,"5min":0.42199999999999999,"15min":0.34300000000000003},"max":{"1min":6.8769999999999998,"5min":6.8769999999999998,"15min":53.229999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.747,"15min":1.0209999999999999},"min":{"1min":0.49199999999999999,"5min":0.44400000000000001,"15min":0.41799999999999998},"max":{"1min":6.1360000000000001,"5min":6.1360000000000001,"15min":32.404000000000003},"last":0.82199999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.749,"15min":1.0820000000000001},"min":{"1min":0.52200000000000002,"5min":0.46000000000000002,"15min":0.38600000000000001},"max":{"1min":5.1319999999999997,"5min":5.1319999999999997,"15min":45.070999999999998},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.71099999999999997,"15min":1.181},"min":{"1min":0.51100000000000001,"5min":0.47999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89100000000000001,"5min":0.999,"15min":53.338000000000001},"last":0.72999999999999998}]}]},{"osd":4,"up_from":15,"seq":64424509668,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1387388,"kb_used_data":1324340,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":92984452,"statfs":{"total":96636764160,"available":95216078848,"internally_reserved":0,"allocated":1356124160,"data_stored":1355354916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.69799999999999995,"15min":0.90600000000000003},"min":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.32900000000000001},"max":{"1min":0.96599999999999997,"5min":1.542,"15min":23.318000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.70299999999999996,"15min":0.81299999999999994},"min":{"1min":0.55000000000000004,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":14.723000000000001},"last":0.75800000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.72599999999999998,"15min":0.81399999999999995},"min":{"1min":0.46300000000000002,"5min":0.39100000000000001,"15min":0.38800000000000001},"max":{"1min":0.96199999999999997,"5min":1.919,"15min":9.5700000000000003},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.74199999999999999,"15min":0.94699999999999995},"min":{"1min":0.52600000000000002,"5min":0.44900000000000001,"15min":0.36899999999999999},"max":{"1min":0.996,"5min":1.429,"15min":22.408999999999999},"last":0.60799999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.79000000000000004,"15min":0.85299999999999998},"min":{"1min":0.50900000000000001,"5min":0.41899999999999998,"15min":0.39700000000000002},"max":{"1min":1.032,"5min":5.6260000000000003,"15min":9.7089999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.76100000000000001,"15min":1.018},"min":{"1min":0.59299999999999997,"5min":0.45000000000000001,"15min":0.39100000000000001},"max":{"1min":1.016,"5min":1.387,"15min":22.338999999999999},"last":0.52900000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.73999999999999999,"15min":0.88700000000000001},"min":{"1min":0.52300000000000002,"5min":0.46200000000000002,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":1.577,"15min":14.763999999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.82599999999999996,"15min":0.93600000000000005},"min":{"1min":0.65300000000000002,"5min":0.53600000000000003,"15min":0.48099999999999998},"max":{"1min":0.996,"5min":1.9610000000000001,"15min":9.6289999999999996},"last":0.56299999999999994}]},{"osd":5,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63300000000000001,"15min":0.94299999999999995},"min":{"1min":0.44,"5min":0.36499999999999999,"15min":0.34799999999999998},"max":{"1min":1.0149999999999999,"5min":1.1970000000000001,"15min":31.617999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.85799999999999998},"min":{"1min":0.40799999999999997,"5min":0.373,"15min":0.36599999999999999},"max":{"1min":0.86699999999999999,"5min":1.4910000000000001,"15min":17.145},"last":0.81799999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.72799999999999998,"15min":0.83899999999999997},"min":{"1min":0.60799999999999998,"5min":0.41799999999999998,"15min":0.37},"max":{"1min":0.89300000000000002,"5min":5.9489999999999998,"15min":11.124000000000001},"last":0.628},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.71099999999999997,"15min":1.2090000000000001},"min":{"1min":0.52300000000000002,"5min":0.42499999999999999,"15min":0.40200000000000002},"max":{"1min":1.0649999999999999,"5min":2.3149999999999999,"15min":32.308},"last":0.83899999999999997}]},{"osd":7,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.72699999999999998,"15min":1.0009999999999999},"min":{"1min":0.48099999999999998,"5min":0.45100000000000001,"15min":0.34799999999999998},"max":{"1min":1,"5min":2.4329999999999998,"15min":39.256},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78200000000000003,"15min":1.002},"min":{"1min":0.60699999999999998,"5min":0.499,"15min":0.46500000000000002},"max":{"1min":1.04,"5min":2.1909999999999998,"15min":26.573},"last":0.71699999999999997}]}]},{"osd":3,"up_from":15,"seq":64424509667,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1154552,"kb_used_data":1087344,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93217288,"statfs":{"total":96636764160,"available":95454502912,"internally_reserved":0,"allocated":1113440256,"data_stored":1112652932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.45100000000000001,"15min":0.498},"min":{"1min":0.28999999999999998,"5min":0.28199999999999997,"15min":0.245},"max":{"1min":0.69099999999999995,"5min":0.69699999999999995,"15min":5.8849999999999998},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.51000000000000001,"15min":0.621},"min":{"1min":0.41799999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.80000000000000004,"5min":1.4750000000000001,"15min":17.756},"last":0.52000000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51300000000000001,"15min":0.60499999999999998},"min":{"1min":0.33500000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60399999999999998,"5min":4.0099999999999998,"15min":12.307},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55200000000000005,"15min":0.59199999999999997},"min":{"1min":0.36199999999999999,"5min":0.34300000000000003,"15min":0.31900000000000001},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":6.0640000000000001},"last":0.54300000000000004}]},{"osd":2,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61399999999999999,"15min":0.71199999999999997},"min":{"1min":0.42799999999999999,"5min":0.35299999999999998,"15min":0.32600000000000001},"max":{"1min":0.82999999999999996,"5min":4.077,"15min":12.468999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.59099999999999997,"15min":0.68000000000000005},"min":{"1min":0.44,"5min":0.39100000000000001,"15min":0.35399999999999998},"max":{"1min":0.78000000000000003,"5min":0.81899999999999995,"15min":10.837999999999999},"last":0.63800000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78500000000000003,"15min":0.999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.375},"max":{"1min":1.1910000000000001,"5min":1.9379999999999999,"15min":17.853999999999999},"last":0.878},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.85999999999999999,"15min":0.98199999999999998},"min":{"1min":0.54900000000000004,"5min":0.49199999999999999,"15min":0.45200000000000001},"max":{"1min":1.0569999999999999,"5min":4.2430000000000003,"15min":13.31},"last":0.93999999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.78400000000000003,"15min":0.98899999999999999},"min":{"1min":0.496,"5min":0.439,"15min":0.439},"max":{"1min":1.1619999999999999,"5min":1.179,"15min":17.687999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.77800000000000002,"15min":0.99099999999999999},"min":{"1min":0.58599999999999997,"5min":0.53800000000000003,"15min":0.438},"max":{"1min":1.048,"5min":1.825,"15min":20.515999999999998},"last":0.78000000000000003}]},{"osd":6,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.80900000000000005,"15min":0.98499999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.432},"max":{"1min":1.1419999999999999,"5min":4.2800000000000002,"15min":23.100999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.80600000000000005,"15min":0.94799999999999995},"min":{"1min":0.69899999999999995,"5min":0.52200000000000002,"15min":0.5},"max":{"1min":1.0720000000000001,"5min":1.149,"15min":12.76},"last":0.80100000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82499999999999996,"15min":1.1850000000000001},"min":{"1min":0.625,"5min":0.56899999999999995,"15min":0.46800000000000003},"max":{"1min":1.0860000000000001,"5min":1.931,"15min":24.321999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.82899999999999996,"15min":1.0820000000000001},"min":{"1min":0.60899999999999999,"5min":0.497,"15min":0.497},"max":{"1min":1.2070000000000001,"5min":4.202,"15min":23.013000000000002},"last":0.90000000000000002}]}]},{"osd":2,"up_from":15,"seq":64424509667,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760768,"kb_used_data":693560,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93611072,"statfs":{"total":96636764160,"available":95857737728,"internally_reserved":0,"allocated":710205440,"data_stored":709433352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45500000000000002,"15min":0.50900000000000001},"min":{"1min":0.32300000000000001,"5min":0.31,"15min":0.247},"max":{"1min":0.55200000000000005,"5min":0.80800000000000005,"15min":5.1849999999999996},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.501,"15min":0.54600000000000004},"min":{"1min":0.36499999999999999,"5min":0.26700000000000002,"15min":0.252},"max":{"1min":0.73999999999999999,"5min":0.83899999999999997,"15min":7.516},"last":0.51700000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.56799999999999995,"15min":0.60999999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.29699999999999999},"max":{"1min":0.71899999999999997,"5min":3.044,"15min":10.907},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.48999999999999999,"15min":0.59599999999999997},"min":{"1min":0.34999999999999998,"5min":0.33100000000000002,"15min":0.28100000000000003},"max":{"1min":0.76400000000000001,"5min":0.76500000000000001,"15min":8.9190000000000005},"last":0.36399999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52500000000000002,"15min":0.60599999999999998},"min":{"1min":0.40799999999999997,"5min":0.35399999999999998,"15min":0.30499999999999999},"max":{"1min":0.70999999999999996,"5min":0.86499999999999999,"15min":12.185},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61899999999999999,"15min":0.67100000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.373},"max":{"1min":0.80700000000000005,"5min":0.89600000000000002,"15min":7.6760000000000002},"last":0.495}]},{"osd":4,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.80300000000000005,"15min":0.879},"min":{"1min":0.59999999999999998,"5min":0.54700000000000004,"15min":0.44},"max":{"1min":1.0660000000000001,"5min":1.22,"15min":8.6609999999999996},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.76800000000000002,"15min":0.91200000000000003},"min":{"1min":0.56399999999999995,"5min":0.46200000000000002,"15min":0.41599999999999998},"max":{"1min":1.008,"5min":1.9670000000000001,"15min":19.146999999999998},"last":0.65000000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.76000000000000001,"15min":0.90200000000000002},"min":{"1min":0.58299999999999996,"5min":0.53600000000000003,"15min":0.41899999999999998},"max":{"1min":1.093,"5min":1.111,"15min":31.710000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.81999999999999995,"15min":0.999},"min":{"1min":0.57399999999999995,"5min":0.51700000000000002,"15min":0.48199999999999998},"max":{"1min":0.99199999999999999,"5min":1.8240000000000001,"15min":22.539000000000001},"last":0.60699999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.78300000000000003,"15min":0.91000000000000003},"min":{"1min":0.56499999999999995,"5min":0.52900000000000003,"15min":0.40899999999999997},"max":{"1min":1.079,"5min":1.3979999999999999,"15min":9.4540000000000006},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.80100000000000005,"15min":0.90800000000000003},"min":{"1min":0.64300000000000002,"5min":0.5,"15min":0.47899999999999998},"max":{"1min":1.0309999999999999,"5min":1.873,"15min":18.614999999999998},"last":0.58599999999999997}]},{"osd":7,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.79800000000000004,"15min":0.96799999999999997},"min":{"1min":0.63200000000000001,"5min":0.50900000000000001,"15min":0.44400000000000001},"max":{"1min":1.0109999999999999,"5min":1.7729999999999999,"15min":11.606},"last":0.746},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.84699999999999998,"15min":0.998},"min":{"1min":0.627,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.105,"5min":1.611,"15min":17.013000000000002},"last":0.73199999999999998}]}]},{"osd":1,"up_from":15,"seq":64424509669,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":941004,"kb_used_data":873796,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93430836,"statfs":{"total":96636764160,"available":95673176064,"internally_reserved":0,"allocated":894767104,"data_stored":893983032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45400000000000001,"15min":0.58499999999999996},"min":{"1min":0.26800000000000002,"5min":0.25900000000000001,"15min":0.23699999999999999},"max":{"1min":0.63500000000000001,"5min":0.88100000000000001,"15min":17.396999999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.51600000000000001,"15min":0.626},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.26900000000000002},"max":{"1min":0.73699999999999999,"5min":0.82999999999999996,"15min":12.378},"last":0.42999999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51700000000000002,"15min":0.57699999999999996},"min":{"1min":0.33000000000000002,"5min":0.313,"15min":0.309},"max":{"1min":0.61699999999999999,"5min":0.878,"15min":6.4569999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.54300000000000004,"15min":0.67400000000000004},"min":{"1min":0.40100000000000002,"5min":0.34499999999999997,"15min":0.33300000000000002},"max":{"1min":0.73899999999999999,"5min":0.81299999999999994,"15min":18.827000000000002},"last":0.47299999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59799999999999998,"15min":0.67200000000000004},"min":{"1min":0.433,"5min":0.40600000000000003,"15min":0.33700000000000002},"max":{"1min":0.872,"5min":1.0580000000000001,"15min":5.915},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.53800000000000003,"15min":0.64800000000000002},"min":{"1min":0.34399999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.77500000000000002,"5min":0.85499999999999998,"15min":9.9719999999999995},"last":0.49399999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.77700000000000002,"15min":0.89400000000000002},"min":{"1min":0.50900000000000001,"5min":0.437,"15min":0.437},"max":{"1min":1.056,"5min":1.1060000000000001,"15min":9.9329999999999998},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.746,"5min":0.79000000000000004,"15min":0.88300000000000001},"min":{"1min":0.55600000000000005,"5min":0.52800000000000002,"15min":0.45800000000000002},"max":{"1min":0.95599999999999996,"5min":1.165,"15min":9.8610000000000007},"last":0.66800000000000004}]},{"osd":5,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.80200000000000005,"15min":0.96299999999999997},"min":{"1min":0.55000000000000004,"5min":0.51200000000000001,"15min":0.46100000000000002},"max":{"1min":0.97499999999999998,"5min":1.2050000000000001,"15min":22.594999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.78200000000000003,"15min":1.0129999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.443},"max":{"1min":0.97499999999999998,"5min":1.077,"15min":26.748999999999999},"last":0.83899999999999997}]},{"osd":6,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.78800000000000003,"15min":1.034},"min":{"1min":0.57899999999999996,"5min":0.53100000000000003,"15min":0.42899999999999999},"max":{"1min":1.095,"5min":1.218,"15min":35.112000000000002},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.81100000000000005,"15min":0.90900000000000003},"min":{"1min":0.62,"5min":0.501,"15min":0.501},"max":{"1min":0.92900000000000005,"5min":1.2090000000000001,"15min":8.3670000000000009},"last":0.68799999999999994}]},{"osd":7,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.80800000000000005,"15min":1.024},"min":{"1min":0.56399999999999995,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.141,"5min":1.1950000000000001,"15min":20.872},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.81599999999999995,"15min":0.97799999999999998},"min":{"1min":0.61499999999999999,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.92300000000000004,"5min":1.139,"15min":9.8640000000000008},"last":0.61899999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542373,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":904260,"kb_used_data":828732,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93467580,"statfs":{"total":96636764160,"available":95710801920,"internally_reserved":0,"allocated":848621568,"data_stored":847844597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.72299999999999998,"15min":0.754},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.29899999999999999},"max":{"1min":1.038,"5min":2.6920000000000002,"15min":5.1639999999999997},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.75800000000000001,"15min":0.83899999999999997},"min":{"1min":0.46899999999999997,"5min":0.36799999999999999,"15min":0.34899999999999998},"max":{"1min":0.98199999999999998,"5min":6.4290000000000003,"15min":9.359},"last":0.70999999999999996}]},{"osd":1,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.76800000000000002,"15min":0.93000000000000005},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.375},"max":{"1min":0.89900000000000002,"5min":6.5069999999999997,"15min":21.120000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.747,"15min":0.93899999999999995},"min":{"1min":0.48299999999999998,"5min":0.39900000000000002,"15min":0.38600000000000001},"max":{"1min":0.96499999999999997,"5min":2.6619999999999999,"15min":15.843},"last":0.64900000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.76800000000000002,"15min":0.86699999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.34999999999999998},"max":{"1min":0.95099999999999996,"5min":6.3339999999999996,"15min":8.7750000000000004},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.79900000000000004,"15min":0.98299999999999998},"min":{"1min":0.501,"5min":0.501,"15min":0.39900000000000002},"max":{"1min":0.95399999999999996,"5min":6.4619999999999997,"15min":21.201000000000001},"last":0.501}]},{"osd":3,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.753,"15min":0.84399999999999997},"min":{"1min":0.498,"5min":0.46000000000000002,"15min":0.37},"max":{"1min":0.99199999999999999,"5min":2.6299999999999999,"15min":7.7199999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.81100000000000005,"15min":0.91900000000000004},"min":{"1min":0.52800000000000002,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.0109999999999999,"5min":6.3730000000000002,"15min":8.7300000000000004},"last":0.61899999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.66500000000000004,"15min":0.83399999999999996},"min":{"1min":0.52600000000000002,"5min":0.42999999999999999,"15min":0.30399999999999999},"max":{"1min":1.012,"5min":3.161,"15min":10.962},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.70699999999999996,"15min":1.012},"min":{"1min":0.48799999999999999,"5min":0.41999999999999998,"15min":0.35099999999999998},"max":{"1min":0.88100000000000001,"5min":6.5460000000000003,"15min":26.146999999999998},"last":0.59599999999999997}]},{"osd":6,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.66500000000000004,"15min":0.76700000000000002},"min":{"1min":0.50700000000000001,"5min":0.41299999999999998,"15min":0.36099999999999999},"max":{"1min":0.871,"5min":2.589,"15min":6.6699999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.82399999999999995,"15min":0.99199999999999999},"min":{"1min":0.53800000000000003,"5min":0.45300000000000001,"15min":0.379},"max":{"1min":0.95699999999999996,"5min":14.007999999999999,"15min":30.488},"last":0.67500000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.72499999999999998,"15min":1.2010000000000001},"min":{"1min":0.56699999999999995,"5min":0.48899999999999999,"15min":0.42599999999999999},"max":{"1min":0.90800000000000003,"5min":3.0870000000000002,"15min":33.499000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.91000000000000003,"15min":1.0680000000000001},"min":{"1min":0.57099999999999995,"5min":0.45700000000000002,"15min":0.42899999999999999},"max":{"1min":0.93000000000000005,"5min":21.178999999999998,"15min":25.437999999999999},"last":0.69199999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542372,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1458240,"kb_used_data":1391032,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92913600,"statfs":{"total":96636764160,"available":95143526400,"internally_reserved":0,"allocated":1424416768,"data_stored":1423621757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.82999999999999996,"15min":0.749},"min":{"1min":0.33100000000000002,"5min":0.25800000000000001,"15min":0.245},"max":{"1min":0.59899999999999998,"5min":38.521999999999998,"15min":38.521999999999998},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.60099999999999998,"15min":0.68799999999999994},"min":{"1min":0.36699999999999999,"5min":0.32900000000000001,"15min":0.27000000000000002},"max":{"1min":0.753,"5min":8.0749999999999993,"15min":10.154},"last":0.34699999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.57699999999999996,"15min":0.80200000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.30399999999999999},"max":{"1min":0.78600000000000003,"5min":2.6909999999999998,"15min":25.905000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.67200000000000004,"15min":0.66900000000000004},"min":{"1min":0.307,"5min":0.307,"15min":0.27800000000000002},"max":{"1min":0.69199999999999995,"5min":18.247,"15min":18.247},"last":0.45300000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.76400000000000001,"15min":0.80300000000000005},"min":{"1min":0.32400000000000001,"5min":0.32000000000000001,"15min":0.29199999999999998},"max":{"1min":0.70999999999999996,"5min":21.451000000000001,"15min":21.451000000000001},"last":0.629},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61799999999999999,"15min":0.76600000000000001},"min":{"1min":0.47099999999999997,"5min":0.38900000000000001,"15min":0.36899999999999999},"max":{"1min":0.80100000000000005,"5min":1.7,"15min":10.210000000000001},"last":0.55600000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.79400000000000004,"15min":0.99199999999999999},"min":{"1min":0.52600000000000002,"5min":0.441,"15min":0.39100000000000001},"max":{"1min":1.1639999999999999,"5min":2.7850000000000001,"15min":22.814},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.80100000000000005,"5min":1.0389999999999999,"15min":1.105},"min":{"1min":0.64000000000000001,"5min":0.45500000000000002,"15min":0.42399999999999999},"max":{"1min":1.1379999999999999,"5min":27.027999999999999,"15min":27.027999999999999},"last":0.41299999999999998}]},{"osd":5,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.83199999999999996,"15min":0.98799999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.38400000000000001},"max":{"1min":1.218,"5min":6.6429999999999998,"15min":12.428000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.77700000000000002,"5min":1.0509999999999999,"15min":1.333},"min":{"1min":0.55400000000000005,"5min":0.47999999999999998,"15min":0.46100000000000002},"max":{"1min":1.0600000000000001,"5min":27.210999999999999,"15min":27.210999999999999},"last":0.66400000000000003}]},{"osd":6,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.86699999999999999,"15min":1.242},"min":{"1min":0.66400000000000003,"5min":0.50900000000000001,"15min":0.46700000000000003},"max":{"1min":0.98399999999999999,"5min":9.3140000000000001,"15min":31.059000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.82699999999999996,"15min":1.089},"min":{"1min":0.58899999999999997,"5min":0.52900000000000003,"15min":0.51500000000000001},"max":{"1min":1.024,"5min":2.8340000000000001,"15min":40.731000000000002},"last":0.68200000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.86299999999999999,"15min":1.2150000000000001},"min":{"1min":0.66500000000000004,"5min":0.52200000000000002,"15min":0.499},"max":{"1min":1.1539999999999999,"5min":5.2089999999999996,"15min":24.492999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.81599999999999995,"5min":1.097,"15min":1.1859999999999999},"min":{"1min":0.63500000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1200000000000001,"5min":27.081,"15min":27.081},"last":0.70099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":585728,"data_stored":567429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":1321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":598016,"data_stored":583072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1421877248,"data_stored":1421869497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":905977856,"data_stored":905970036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":708841472,"data_stored":708837568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1111498752,"data_stored":1111491082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1354764288,"data_stored":1354760453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":847253504,"data_stored":847249530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":981471232,"data_stored":981467258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1430257664,"data_stored":1430257664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:23:35.225 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-15T18:23:35.225 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:23:35.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.409+0000 7f5614c00640 1 Processor -- start 2024-01-15T18:23:35.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.409+0000 7f5614c00640 1 -- start start 2024-01-15T18:23:35.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.413+0000 7f5614c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f56100701a0 0x7f5610070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:35.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.413+0000 7f5614c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f561015c130 0x7f561015d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:35.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.413+0000 7f5614c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f561006c4a0 con 0x7f56100c92b0 2024-01-15T18:23:35.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.413+0000 7f5614c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5610070ac0 con 0x7f56100701a0 2024-01-15T18:23:35.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.413+0000 7f5614c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f561015c660 con 0x7f561015c130 2024-01-15T18:23:35.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.413+0000 7f560d400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f56100c92b0 0x7f561006dd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42864/0 (socket says 172.21.15.27:42864) 2024-01-15T18:23:35.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.413+0000 7f560de00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f56100701a0 0x7f5610070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:35.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.413+0000 7f560d400640 1 -- 172.21.15.27:0/4154801371 learned_addr learned my addr 172.21.15.27:0/4154801371 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:35.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.413+0000 7f560e800640 1 --2- 172.21.15.27:0/4154801371 >> v2:172.21.15.93:3300/0 conn(0x7f561015c130 0x7f561015d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:35.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.413+0000 7f560de00640 1 -- 172.21.15.27:0/4154801371 >> v2:172.21.15.93:3300/0 conn(0x7f561015c130 msgr2=0x7f561015d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:35.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.413+0000 7f560de00640 1 --2- 172.21.15.27:0/4154801371 >> v2:172.21.15.93:3300/0 conn(0x7f561015c130 0x7f561015d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.413+0000 7f560de00640 1 -- 172.21.15.27:0/4154801371 >> v1:172.21.15.27:6789/0 conn(0x7f56100c92b0 legacy=0x7f561006dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:35.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.413+0000 7f560ca00640 1 -- 172.21.15.27:0/4154801371 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998490858 0 0) 0x7f561006c4a0 con 0x7f56100c92b0 2024-01-15T18:23:35.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.413+0000 7f560de00640 1 -- 172.21.15.27:0/4154801371 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561015c940 con 0x7f56100701a0 2024-01-15T18:23:35.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.417+0000 7f560de00640 1 --2- 172.21.15.27:0/4154801371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56100701a0 0x7f5610070580 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f561006c050 tx=0x7f560001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f27210897372ef12 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:35.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.417+0000 7f560ca00640 1 -- 172.21.15.27:0/4154801371 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56000a7070 con 0x7f56100701a0 2024-01-15T18:23:35.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.417+0000 7f560ca00640 1 -- 172.21.15.27:0/4154801371 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56000ab430 con 0x7f56100701a0 2024-01-15T18:23:35.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.417+0000 7f560ca00640 1 -- 172.21.15.27:0/4154801371 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56000ab7c0 con 0x7f56100701a0 2024-01-15T18:23:35.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.417+0000 7f5614c00640 1 -- 172.21.15.27:0/4154801371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56100701a0 msgr2=0x7f5610070580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:35.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.417+0000 7f5614c00640 1 --2- 172.21.15.27:0/4154801371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56100701a0 0x7f5610070580 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f561006c050 tx=0x7f560001c430 comp rx=0 tx=0).stop 2024-01-15T18:23:35.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.417+0000 7f5614c00640 1 -- 172.21.15.27:0/4154801371 shutdown_connections 2024-01-15T18:23:35.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.417+0000 7f5614c00640 1 --2- 172.21.15.27:0/4154801371 >> v2:172.21.15.93:3300/0 conn(0x7f561015c130 0x7f561015d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.417+0000 7f5614c00640 1 --2- 172.21.15.27:0/4154801371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56100701a0 0x7f5610070580 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.417+0000 7f5614c00640 1 -- 172.21.15.27:0/4154801371 >> 172.21.15.27:0/4154801371 conn(0x7f56100cec60 msgr2=0x7f56100cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:35.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.417+0000 7f5614c00640 1 -- 172.21.15.27:0/4154801371 shutdown_connections 2024-01-15T18:23:35.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.417+0000 7f5614c00640 1 -- 172.21.15.27:0/4154801371 wait complete. 2024-01-15T18:23:35.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.417+0000 7f5614c00640 1 Processor -- start 2024-01-15T18:23:35.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.417+0000 7f5614c00640 1 -- start start 2024-01-15T18:23:35.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.421+0000 7f5614c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56100701a0 0x7f56100a86e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:35.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.421+0000 7f5614c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56100c92b0 0x7f56100c5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:35.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.421+0000 7f5614c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f561015c130 0x7f56100c6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:35.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.421+0000 7f5614c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5610159cb0 con 0x7f56100c92b0 2024-01-15T18:23:35.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.421+0000 7f5614c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5610158ac0 con 0x7f56100701a0 2024-01-15T18:23:35.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.421+0000 7f5614c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5610159f00 con 0x7f561015c130 2024-01-15T18:23:35.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.421+0000 7f560d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56100c92b0 0x7f56100c5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:35.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.421+0000 7f560d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56100c92b0 0x7f56100c5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47488/0 (socket says 172.21.15.27:47488) 2024-01-15T18:23:35.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.421+0000 7f560d400640 1 -- 172.21.15.27:0/1124329920 learned_addr learned my addr 172.21.15.27:0/1124329920 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:35.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.421+0000 7f560de00640 1 --2- 172.21.15.27:0/1124329920 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56100701a0 0x7f56100a86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:35.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.421+0000 7f560e800640 1 --2- 172.21.15.27:0/1124329920 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f561015c130 0x7f56100c6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:35.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.421+0000 7f560d400640 1 -- 172.21.15.27:0/1124329920 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f561015c130 msgr2=0x7f56100c6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:35.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.421+0000 7f560d400640 1 --2- 172.21.15.27:0/1124329920 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f561015c130 0x7f56100c6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.421+0000 7f560d400640 1 -- 172.21.15.27:0/1124329920 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56100701a0 msgr2=0x7f56100a86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:35.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.421+0000 7f560d400640 1 --2- 172.21.15.27:0/1124329920 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56100701a0 0x7f56100a86e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.421+0000 7f560d400640 1 -- 172.21.15.27:0/1124329920 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5600009420 con 0x7f56100c92b0 2024-01-15T18:23:35.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.421+0000 7f560d400640 1 --2- 172.21.15.27:0/1124329920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56100c92b0 0x7f56100c5d30 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f55f800c550 tx=0x7f55f800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:35.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.421+0000 7f5606000640 1 -- 172.21.15.27:0/1124329920 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f8005cf0 con 0x7f56100c92b0 2024-01-15T18:23:35.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.421+0000 7f5606000640 1 -- 172.21.15.27:0/1124329920 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f8005e70 con 0x7f56100c92b0 2024-01-15T18:23:35.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.421+0000 7f5606000640 1 -- 172.21.15.27:0/1124329920 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f8007310 con 0x7f56100c92b0 2024-01-15T18:23:35.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.421+0000 7f5614c00640 1 -- 172.21.15.27:0/1124329920 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5610158d60 con 0x7f56100c92b0 2024-01-15T18:23:35.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.421+0000 7f5614c00640 1 -- 172.21.15.27:0/1124329920 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56100a8c20 con 0x7f56100c92b0 2024-01-15T18:23:35.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.421+0000 7f5614c00640 1 -- 172.21.15.27:0/1124329920 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f561004e850 con 0x7f56100c92b0 2024-01-15T18:23:35.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.425+0000 7f5606000640 1 -- 172.21.15.27:0/1124329920 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f55f8007470 con 0x7f56100c92b0 2024-01-15T18:23:35.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.425+0000 7f5606000640 1 --2- 172.21.15.27:0/1124329920 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f55e8067a60 0x7f55e8069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:35.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.425+0000 7f5606000640 1 -- 172.21.15.27:0/1124329920 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f55f8082930 con 0x7f56100c92b0 2024-01-15T18:23:35.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.425+0000 7f560de00640 1 --2- 172.21.15.27:0/1124329920 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f55e8067a60 0x7f55e8069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:35.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.425+0000 7f560de00640 1 --2- 172.21.15.27:0/1124329920 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f55e8067a60 0x7f55e8069f20 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f56000af000 tx=0x7f56000a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:35.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.429+0000 7f5606000640 1 -- 172.21.15.27:0/1124329920 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f55f8051320 con 0x7f56100c92b0 2024-01-15T18:23:35.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.581+0000 7f5614c00640 1 -- 172.21.15.27:0/1124329920 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f561006e410 con 0x7f55e8067a60 2024-01-15T18:23:35.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.593+0000 7f5606000640 1 -- 172.21.15.27:0/1124329920 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+291438 (secure 0 0 0) 0x7f56000ab900 con 0x7f55e8067a60 2024-01-15T18:23:35.591 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:23:35.593 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:23:35.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.593+0000 7f5614c00640 1 -- 172.21.15.27:0/1124329920 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f55e8067a60 msgr2=0x7f55e8069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:35.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.593+0000 7f5614c00640 1 --2- 172.21.15.27:0/1124329920 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f55e8067a60 0x7f55e8069f20 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f56000af000 tx=0x7f56000a7000 comp rx=0 tx=0).stop 2024-01-15T18:23:35.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.593+0000 7f5614c00640 1 -- 172.21.15.27:0/1124329920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56100c92b0 msgr2=0x7f56100c5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:35.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.593+0000 7f5614c00640 1 --2- 172.21.15.27:0/1124329920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56100c92b0 0x7f56100c5d30 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f55f800c550 tx=0x7f55f800ca10 comp rx=0 tx=0).stop 2024-01-15T18:23:35.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.597+0000 7f5614c00640 1 -- 172.21.15.27:0/1124329920 shutdown_connections 2024-01-15T18:23:35.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.597+0000 7f5614c00640 1 --2- 172.21.15.27:0/1124329920 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f55e8067a60 0x7f55e8069f20 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.597+0000 7f5614c00640 1 --2- 172.21.15.27:0/1124329920 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f561015c130 0x7f56100c6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.597+0000 7f5614c00640 1 --2- 172.21.15.27:0/1124329920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56100c92b0 0x7f56100c5d30 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.597+0000 7f5614c00640 1 --2- 172.21.15.27:0/1124329920 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f56100701a0 0x7f56100a86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.597+0000 7f5614c00640 1 -- 172.21.15.27:0/1124329920 >> 172.21.15.27:0/1124329920 conn(0x7f56100cec60 msgr2=0x7f561015b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:35.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.597+0000 7f5614c00640 1 -- 172.21.15.27:0/1124329920 shutdown_connections 2024-01-15T18:23:35.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.597+0000 7f5614c00640 1 -- 172.21.15.27:0/1124329920 wait complete. 2024-01-15T18:23:35.611 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":668,"stamp":"2024-01-15T18:23:34.457302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4354848372,"num_objects":1069,"num_object_clones":1,"num_object_copies":2138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1069,"num_whiteouts":1,"num_read":418,"num_read_kb":1142907,"num_write":1483,"num_write_kb":4254748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2926,"ondisk_log_size":2926,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":9070344,"kb_used_data":8532680,"kb_used_omap":12,"kb_used_meta":537587,"kb_avail":745904376,"statfs":{"total":773094113280,"available":763806081024,"internally_reserved":0,"allocated":8737464320,"data_stored":8731229664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":550489648},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":58720646,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":23,"num_read_kb":94208,"num_write":20,"num_write_kb":57348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003319"},"pg_stats":[{"pgid":"17.3f","version":"110'49","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.452952+0000","last_change":"2024-01-15T18:18:33.168792+0000","last_active":"2024-01-15T18:23:23.452952+0000","last_peered":"2024-01-15T18:23:23.452952+0000","last_clean":"2024-01-15T18:23:23.452952+0000","last_became_active":"2024-01-15T18:18:33.168500+0000","last_became_peered":"2024-01-15T18:18:33.168500+0000","last_unstale":"2024-01-15T18:23:23.452952+0000","last_undegraded":"2024-01-15T18:23:23.452952+0000","last_fullsized":"2024-01-15T18:23:23.452952+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:28:41.940111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"110'42","reported_seq":61,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.445536+0000","last_change":"2024-01-15T18:18:33.151630+0000","last_active":"2024-01-15T18:23:23.445536+0000","last_peered":"2024-01-15T18:23:23.445536+0000","last_clean":"2024-01-15T18:23:23.445536+0000","last_became_active":"2024-01-15T18:18:33.151209+0000","last_became_peered":"2024-01-15T18:18:33.151209+0000","last_unstale":"2024-01-15T18:23:23.445536+0000","last_undegraded":"2024-01-15T18:23:23.445536+0000","last_fullsized":"2024-01-15T18:23:23.445536+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:46:54.687765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"110'39","reported_seq":57,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.457520+0000","last_change":"2024-01-15T18:18:33.163094+0000","last_active":"2024-01-15T18:23:23.457520+0000","last_peered":"2024-01-15T18:23:23.457520+0000","last_clean":"2024-01-15T18:23:23.457520+0000","last_became_active":"2024-01-15T18:18:33.162700+0000","last_became_peered":"2024-01-15T18:18:33.162700+0000","last_unstale":"2024-01-15T18:23:23.457520+0000","last_undegraded":"2024-01-15T18:23:23.457520+0000","last_fullsized":"2024-01-15T18:23:23.457520+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:23:36.167165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":16,"num_write_kb":57345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"110'39","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.459915+0000","last_change":"2024-01-15T18:18:33.160786+0000","last_active":"2024-01-15T18:23:23.459915+0000","last_peered":"2024-01-15T18:23:23.459915+0000","last_clean":"2024-01-15T18:23:23.459915+0000","last_became_active":"2024-01-15T18:18:33.160460+0000","last_became_peered":"2024-01-15T18:18:33.160460+0000","last_unstale":"2024-01-15T18:23:23.459915+0000","last_undegraded":"2024-01-15T18:23:23.459915+0000","last_fullsized":"2024-01-15T18:23:23.459915+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:27:33.980071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"110'25","reported_seq":42,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.441694+0000","last_change":"2024-01-15T18:18:33.155936+0000","last_active":"2024-01-15T18:23:23.441694+0000","last_peered":"2024-01-15T18:23:23.441694+0000","last_clean":"2024-01-15T18:23:23.441694+0000","last_became_active":"2024-01-15T18:18:33.155656+0000","last_became_peered":"2024-01-15T18:18:33.155656+0000","last_unstale":"2024-01-15T18:23:23.441694+0000","last_undegraded":"2024-01-15T18:23:23.441694+0000","last_fullsized":"2024-01-15T18:23:23.441694+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:27:09.469255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"110'45","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.452122+0000","last_change":"2024-01-15T18:18:33.152032+0000","last_active":"2024-01-15T18:23:23.452122+0000","last_peered":"2024-01-15T18:23:23.452122+0000","last_clean":"2024-01-15T18:23:23.452122+0000","last_became_active":"2024-01-15T18:18:33.151701+0000","last_became_peered":"2024-01-15T18:18:33.151701+0000","last_unstale":"2024-01-15T18:23:23.452122+0000","last_undegraded":"2024-01-15T18:23:23.452122+0000","last_fullsized":"2024-01-15T18:23:23.452122+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:51.983410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"110'43","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.556273+0000","last_change":"2024-01-15T18:18:33.153882+0000","last_active":"2024-01-15T18:23:31.556273+0000","last_peered":"2024-01-15T18:23:31.556273+0000","last_clean":"2024-01-15T18:23:31.556273+0000","last_became_active":"2024-01-15T18:18:33.153264+0000","last_became_peered":"2024-01-15T18:18:33.153264+0000","last_unstale":"2024-01-15T18:23:31.556273+0000","last_undegraded":"2024-01-15T18:23:31.556273+0000","last_fullsized":"2024-01-15T18:23:31.556273+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:37:49.663025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"110'42","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.451406+0000","last_change":"2024-01-15T18:18:33.164714+0000","last_active":"2024-01-15T18:23:23.451406+0000","last_peered":"2024-01-15T18:23:23.451406+0000","last_clean":"2024-01-15T18:23:23.451406+0000","last_became_active":"2024-01-15T18:18:33.163896+0000","last_became_peered":"2024-01-15T18:18:33.163896+0000","last_unstale":"2024-01-15T18:23:23.451406+0000","last_undegraded":"2024-01-15T18:23:23.451406+0000","last_fullsized":"2024-01-15T18:23:23.451406+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:21:18.264946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":19,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"110'42","reported_seq":60,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.450283+0000","last_change":"2024-01-15T18:18:33.158682+0000","last_active":"2024-01-15T18:23:23.450283+0000","last_peered":"2024-01-15T18:23:23.450283+0000","last_clean":"2024-01-15T18:23:23.450283+0000","last_became_active":"2024-01-15T18:18:33.158399+0000","last_became_peered":"2024-01-15T18:18:33.158399+0000","last_unstale":"2024-01-15T18:23:23.450283+0000","last_undegraded":"2024-01-15T18:23:23.450283+0000","last_fullsized":"2024-01-15T18:23:23.450283+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:27:12.032383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"110'41","reported_seq":60,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.445619+0000","last_change":"2024-01-15T18:18:33.159936+0000","last_active":"2024-01-15T18:23:23.445619+0000","last_peered":"2024-01-15T18:23:23.445619+0000","last_clean":"2024-01-15T18:23:23.445619+0000","last_became_active":"2024-01-15T18:18:33.159562+0000","last_became_peered":"2024-01-15T18:18:33.159562+0000","last_unstale":"2024-01-15T18:23:23.445619+0000","last_undegraded":"2024-01-15T18:23:23.445619+0000","last_fullsized":"2024-01-15T18:23:23.445619+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:56:32.966770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":11,"num_write_kb":45056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"110'41","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.443258+0000","last_change":"2024-01-15T18:18:33.155247+0000","last_active":"2024-01-15T18:23:23.443258+0000","last_peered":"2024-01-15T18:23:23.443258+0000","last_clean":"2024-01-15T18:23:23.443258+0000","last_became_active":"2024-01-15T18:18:33.154972+0000","last_became_peered":"2024-01-15T18:18:33.154972+0000","last_unstale":"2024-01-15T18:23:23.443258+0000","last_undegraded":"2024-01-15T18:23:23.443258+0000","last_fullsized":"2024-01-15T18:23:23.443258+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:54:47.923576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"110'43","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.444775+0000","last_change":"2024-01-15T18:18:33.158336+0000","last_active":"2024-01-15T18:23:23.444775+0000","last_peered":"2024-01-15T18:23:23.444775+0000","last_clean":"2024-01-15T18:23:23.444775+0000","last_became_active":"2024-01-15T18:18:33.158064+0000","last_became_peered":"2024-01-15T18:18:33.158064+0000","last_unstale":"2024-01-15T18:23:23.444775+0000","last_undegraded":"2024-01-15T18:23:23.444775+0000","last_fullsized":"2024-01-15T18:23:23.444775+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:42:44.182465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":6,"num_read_kb":20481,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"110'34","reported_seq":58,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.451044+0000","last_change":"2024-01-15T18:18:33.163912+0000","last_active":"2024-01-15T18:23:23.451044+0000","last_peered":"2024-01-15T18:23:23.451044+0000","last_clean":"2024-01-15T18:23:23.451044+0000","last_became_active":"2024-01-15T18:18:33.163610+0000","last_became_peered":"2024-01-15T18:18:33.163610+0000","last_unstale":"2024-01-15T18:23:23.451044+0000","last_undegraded":"2024-01-15T18:23:23.451044+0000","last_fullsized":"2024-01-15T18:23:23.451044+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:32:43.468374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"110'43","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.451241+0000","last_change":"2024-01-15T18:18:33.169868+0000","last_active":"2024-01-15T18:23:23.451241+0000","last_peered":"2024-01-15T18:23:23.451241+0000","last_clean":"2024-01-15T18:23:23.451241+0000","last_became_active":"2024-01-15T18:18:33.169530+0000","last_became_peered":"2024-01-15T18:18:33.169530+0000","last_unstale":"2024-01-15T18:23:23.451241+0000","last_undegraded":"2024-01-15T18:23:23.451241+0000","last_fullsized":"2024-01-15T18:23:23.451241+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:18:22.473056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"110'29","reported_seq":47,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.453353+0000","last_change":"2024-01-15T18:18:33.151325+0000","last_active":"2024-01-15T18:23:23.453353+0000","last_peered":"2024-01-15T18:23:23.453353+0000","last_clean":"2024-01-15T18:23:23.453353+0000","last_became_active":"2024-01-15T18:18:33.151040+0000","last_became_peered":"2024-01-15T18:18:33.151040+0000","last_unstale":"2024-01-15T18:23:23.453353+0000","last_undegraded":"2024-01-15T18:23:23.453353+0000","last_fullsized":"2024-01-15T18:23:23.453353+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:57:08.165322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":7,"num_write_kb":28672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"110'40","reported_seq":57,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.459812+0000","last_change":"2024-01-15T18:18:33.166327+0000","last_active":"2024-01-15T18:23:23.459812+0000","last_peered":"2024-01-15T18:23:23.459812+0000","last_clean":"2024-01-15T18:23:23.459812+0000","last_became_active":"2024-01-15T18:18:33.166014+0000","last_became_peered":"2024-01-15T18:18:33.166014+0000","last_unstale":"2024-01-15T18:23:23.459812+0000","last_undegraded":"2024-01-15T18:23:23.459812+0000","last_fullsized":"2024-01-15T18:23:23.459812+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:37:50.428714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"110'39","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.456607+0000","last_change":"2024-01-15T18:18:33.168613+0000","last_active":"2024-01-15T18:23:23.456607+0000","last_peered":"2024-01-15T18:23:23.456607+0000","last_clean":"2024-01-15T18:23:23.456607+0000","last_became_active":"2024-01-15T18:18:33.167975+0000","last_became_peered":"2024-01-15T18:18:33.167975+0000","last_unstale":"2024-01-15T18:23:23.456607+0000","last_undegraded":"2024-01-15T18:23:23.456607+0000","last_fullsized":"2024-01-15T18:23:23.456607+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:22:13.319815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"110'47","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.456193+0000","last_change":"2024-01-15T18:18:33.155078+0000","last_active":"2024-01-15T18:23:23.456193+0000","last_peered":"2024-01-15T18:23:23.456193+0000","last_clean":"2024-01-15T18:23:23.456193+0000","last_became_active":"2024-01-15T18:18:33.154816+0000","last_became_peered":"2024-01-15T18:18:33.154816+0000","last_unstale":"2024-01-15T18:23:23.456193+0000","last_undegraded":"2024-01-15T18:23:23.456193+0000","last_fullsized":"2024-01-15T18:23:23.456193+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:48.739168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"110'40","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.454711+0000","last_change":"2024-01-15T18:18:33.164028+0000","last_active":"2024-01-15T18:23:23.454711+0000","last_peered":"2024-01-15T18:23:23.454711+0000","last_clean":"2024-01-15T18:23:23.454711+0000","last_became_active":"2024-01-15T18:18:33.163421+0000","last_became_peered":"2024-01-15T18:18:33.163421+0000","last_unstale":"2024-01-15T18:23:23.454711+0000","last_undegraded":"2024-01-15T18:23:23.454711+0000","last_fullsized":"2024-01-15T18:23:23.454711+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:38:31.615856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"110'46","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.456828+0000","last_change":"2024-01-15T18:18:33.160995+0000","last_active":"2024-01-15T18:23:23.456828+0000","last_peered":"2024-01-15T18:23:23.456828+0000","last_clean":"2024-01-15T18:23:23.456828+0000","last_became_active":"2024-01-15T18:18:33.160047+0000","last_became_peered":"2024-01-15T18:18:33.160047+0000","last_unstale":"2024-01-15T18:23:23.456828+0000","last_undegraded":"2024-01-15T18:23:23.456828+0000","last_fullsized":"2024-01-15T18:23:23.456828+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:39:53.520191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"110'50","reported_seq":76,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.458912+0000","last_change":"2024-01-15T18:18:33.152696+0000","last_active":"2024-01-15T18:23:23.458912+0000","last_peered":"2024-01-15T18:23:23.458912+0000","last_clean":"2024-01-15T18:23:23.458912+0000","last_became_active":"2024-01-15T18:18:33.152128+0000","last_became_peered":"2024-01-15T18:18:33.152128+0000","last_unstale":"2024-01-15T18:23:23.458912+0000","last_undegraded":"2024-01-15T18:23:23.458912+0000","last_fullsized":"2024-01-15T18:23:23.458912+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:08:18.706229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":11,"num_read_kb":36866,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"110'41","reported_seq":60,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.450505+0000","last_change":"2024-01-15T18:18:33.163614+0000","last_active":"2024-01-15T18:23:23.450505+0000","last_peered":"2024-01-15T18:23:23.450505+0000","last_clean":"2024-01-15T18:23:23.450505+0000","last_became_active":"2024-01-15T18:18:33.163252+0000","last_became_peered":"2024-01-15T18:18:33.163252+0000","last_unstale":"2024-01-15T18:23:23.450505+0000","last_undegraded":"2024-01-15T18:23:23.450505+0000","last_fullsized":"2024-01-15T18:23:23.450505+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:48.739168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":4,"num_read_kb":8194,"num_write":17,"num_write_kb":61441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"110'35","reported_seq":52,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.457056+0000","last_change":"2024-01-15T18:18:33.159896+0000","last_active":"2024-01-15T18:23:23.457056+0000","last_peered":"2024-01-15T18:23:23.457056+0000","last_clean":"2024-01-15T18:23:23.457056+0000","last_became_active":"2024-01-15T18:18:33.159557+0000","last_became_peered":"2024-01-15T18:18:33.159557+0000","last_unstale":"2024-01-15T18:23:23.457056+0000","last_undegraded":"2024-01-15T18:23:23.457056+0000","last_fullsized":"2024-01-15T18:23:23.457056+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:24.034308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":11,"num_write_kb":45056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"110'39","reported_seq":57,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.459003+0000","last_change":"2024-01-15T18:18:33.166766+0000","last_active":"2024-01-15T18:23:23.459003+0000","last_peered":"2024-01-15T18:23:23.459003+0000","last_clean":"2024-01-15T18:23:23.459003+0000","last_became_active":"2024-01-15T18:18:33.165600+0000","last_became_peered":"2024-01-15T18:18:33.165600+0000","last_unstale":"2024-01-15T18:23:23.459003+0000","last_undegraded":"2024-01-15T18:23:23.459003+0000","last_fullsized":"2024-01-15T18:23:23.459003+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:03.948102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"110'35","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.457832+0000","last_change":"2024-01-15T18:18:33.168720+0000","last_active":"2024-01-15T18:23:23.457832+0000","last_peered":"2024-01-15T18:23:23.457832+0000","last_clean":"2024-01-15T18:23:23.457832+0000","last_became_active":"2024-01-15T18:18:33.168349+0000","last_became_peered":"2024-01-15T18:18:33.168349+0000","last_unstale":"2024-01-15T18:23:23.457832+0000","last_undegraded":"2024-01-15T18:23:23.457832+0000","last_fullsized":"2024-01-15T18:23:23.457832+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:18:22.473056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":16386,"num_write":17,"num_write_kb":61441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"110'50","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.457272+0000","last_change":"2024-01-15T18:18:33.159262+0000","last_active":"2024-01-15T18:23:23.457272+0000","last_peered":"2024-01-15T18:23:23.457272+0000","last_clean":"2024-01-15T18:23:23.457272+0000","last_became_active":"2024-01-15T18:18:33.158946+0000","last_became_peered":"2024-01-15T18:18:33.158946+0000","last_unstale":"2024-01-15T18:23:23.457272+0000","last_undegraded":"2024-01-15T18:23:23.457272+0000","last_fullsized":"2024-01-15T18:23:23.457272+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:05:55.347898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"110'37","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.447552+0000","last_change":"2024-01-15T18:18:33.161814+0000","last_active":"2024-01-15T18:23:23.447552+0000","last_peered":"2024-01-15T18:23:23.447552+0000","last_clean":"2024-01-15T18:23:23.447552+0000","last_became_active":"2024-01-15T18:18:33.161463+0000","last_became_peered":"2024-01-15T18:18:33.161463+0000","last_unstale":"2024-01-15T18:23:23.447552+0000","last_undegraded":"2024-01-15T18:23:23.447552+0000","last_fullsized":"2024-01-15T18:23:23.447552+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:15:36.391855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886260,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4,"num_read_kb":12289,"num_write":21,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"110'31","reported_seq":49,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.450037+0000","last_change":"2024-01-15T18:18:33.162952+0000","last_active":"2024-01-15T18:23:23.450037+0000","last_peered":"2024-01-15T18:23:23.450037+0000","last_clean":"2024-01-15T18:23:23.450037+0000","last_became_active":"2024-01-15T18:18:33.162669+0000","last_became_peered":"2024-01-15T18:18:33.162669+0000","last_unstale":"2024-01-15T18:23:23.450037+0000","last_undegraded":"2024-01-15T18:23:23.450037+0000","last_fullsized":"2024-01-15T18:23:23.450037+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:47:59.088407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"110'38","reported_seq":57,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.456970+0000","last_change":"2024-01-15T18:18:33.160658+0000","last_active":"2024-01-15T18:23:23.456970+0000","last_peered":"2024-01-15T18:23:23.456970+0000","last_clean":"2024-01-15T18:23:23.456970+0000","last_became_active":"2024-01-15T18:18:33.160099+0000","last_became_peered":"2024-01-15T18:18:33.160099+0000","last_unstale":"2024-01-15T18:23:23.456970+0000","last_undegraded":"2024-01-15T18:23:23.456970+0000","last_fullsized":"2024-01-15T18:23:23.456970+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:47:59.088407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"110'41","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.812331+0000","last_change":"2024-01-15T18:18:33.161061+0000","last_active":"2024-01-15T18:23:31.812331+0000","last_peered":"2024-01-15T18:23:31.812331+0000","last_clean":"2024-01-15T18:23:31.812331+0000","last_became_active":"2024-01-15T18:18:33.160725+0000","last_became_peered":"2024-01-15T18:18:33.160725+0000","last_unstale":"2024-01-15T18:23:31.812331+0000","last_undegraded":"2024-01-15T18:23:31.812331+0000","last_fullsized":"2024-01-15T18:23:31.812331+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:38:07.236956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"110'59","reported_seq":81,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.456771+0000","last_change":"2024-01-15T18:18:33.159024+0000","last_active":"2024-01-15T18:23:23.456771+0000","last_peered":"2024-01-15T18:23:23.456771+0000","last_clean":"2024-01-15T18:23:23.456771+0000","last_became_active":"2024-01-15T18:18:33.158701+0000","last_became_peered":"2024-01-15T18:18:33.158701+0000","last_unstale":"2024-01-15T18:23:23.456771+0000","last_undegraded":"2024-01-15T18:23:23.456771+0000","last_fullsized":"2024-01-15T18:23:23.456771+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:50:16.189329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"110'50","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.452426+0000","last_change":"2024-01-15T18:18:33.152241+0000","last_active":"2024-01-15T18:23:23.452426+0000","last_peered":"2024-01-15T18:23:23.452426+0000","last_clean":"2024-01-15T18:23:23.452426+0000","last_became_active":"2024-01-15T18:18:33.151250+0000","last_became_peered":"2024-01-15T18:18:33.151250+0000","last_unstale":"2024-01-15T18:23:23.452426+0000","last_undegraded":"2024-01-15T18:23:23.452426+0000","last_fullsized":"2024-01-15T18:23:23.452426+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:45:02.525155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"110'29","reported_seq":48,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.445810+0000","last_change":"2024-01-15T18:18:33.154972+0000","last_active":"2024-01-15T18:23:23.445810+0000","last_peered":"2024-01-15T18:23:23.445810+0000","last_clean":"2024-01-15T18:23:23.445810+0000","last_became_active":"2024-01-15T18:18:33.154569+0000","last_became_peered":"2024-01-15T18:18:33.154569+0000","last_unstale":"2024-01-15T18:23:23.445810+0000","last_undegraded":"2024-01-15T18:23:23.445810+0000","last_fullsized":"2024-01-15T18:23:23.445810+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:29:02.332324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"107'4","reported_seq":20,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:39.170083+0000","last_change":"2024-01-15T18:18:32.217761+0000","last_active":"2024-01-15T18:20:39.170083+0000","last_peered":"2024-01-15T18:20:39.170083+0000","last_clean":"2024-01-15T18:20:39.170083+0000","last_became_active":"2024-01-15T18:18:32.216337+0000","last_became_peered":"2024-01-15T18:18:32.216337+0000","last_unstale":"2024-01-15T18:20:39.170083+0000","last_undegraded":"2024-01-15T18:20:39.170083+0000","last_fullsized":"2024-01-15T18:20:39.170083+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:01:18.991818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"110'44","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.453170+0000","last_change":"2024-01-15T18:18:33.167015+0000","last_active":"2024-01-15T18:23:23.453170+0000","last_peered":"2024-01-15T18:23:23.453170+0000","last_clean":"2024-01-15T18:23:23.453170+0000","last_became_active":"2024-01-15T18:18:33.166726+0000","last_became_peered":"2024-01-15T18:18:33.166726+0000","last_unstale":"2024-01-15T18:23:23.453170+0000","last_undegraded":"2024-01-15T18:23:23.453170+0000","last_fullsized":"2024-01-15T18:23:23.453170+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:48:57.895180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"107'1","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:40.169868+0000","last_change":"2024-01-15T18:18:32.219987+0000","last_active":"2024-01-15T18:20:40.169868+0000","last_peered":"2024-01-15T18:20:40.169868+0000","last_clean":"2024-01-15T18:20:40.169868+0000","last_became_active":"2024-01-15T18:18:32.219596+0000","last_became_peered":"2024-01-15T18:18:32.219596+0000","last_unstale":"2024-01-15T18:20:40.169868+0000","last_undegraded":"2024-01-15T18:20:40.169868+0000","last_fullsized":"2024-01-15T18:20:40.169868+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:23.024473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"110'40","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.451309+0000","last_change":"2024-01-15T18:18:33.150743+0000","last_active":"2024-01-15T18:23:23.451309+0000","last_peered":"2024-01-15T18:23:23.451309+0000","last_clean":"2024-01-15T18:23:23.451309+0000","last_became_active":"2024-01-15T18:18:33.150429+0000","last_became_peered":"2024-01-15T18:18:33.150429+0000","last_unstale":"2024-01-15T18:23:23.451309+0000","last_undegraded":"2024-01-15T18:23:23.451309+0000","last_fullsized":"2024-01-15T18:23:23.451309+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:33:18.994479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"110'3","reported_seq":18,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:19:10.235403+0000","last_change":"2024-01-15T18:18:32.213207+0000","last_active":"2024-01-15T18:19:10.235403+0000","last_peered":"2024-01-15T18:19:10.235403+0000","last_clean":"2024-01-15T18:19:10.235403+0000","last_became_active":"2024-01-15T18:18:32.212071+0000","last_became_peered":"2024-01-15T18:18:32.212071+0000","last_unstale":"2024-01-15T18:19:10.235403+0000","last_undegraded":"2024-01-15T18:19:10.235403+0000","last_fullsized":"2024-01-15T18:19:10.235403+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:29:14.612143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"110'51","reported_seq":75,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.454632+0000","last_change":"2024-01-15T18:18:33.165545+0000","last_active":"2024-01-15T18:23:23.454632+0000","last_peered":"2024-01-15T18:23:23.454632+0000","last_clean":"2024-01-15T18:23:23.454632+0000","last_became_active":"2024-01-15T18:18:33.165258+0000","last_became_peered":"2024-01-15T18:18:33.165258+0000","last_unstale":"2024-01-15T18:23:23.454632+0000","last_undegraded":"2024-01-15T18:23:23.454632+0000","last_fullsized":"2024-01-15T18:23:23.454632+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:05:07.453809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"107'2","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:18:42.257846+0000","last_change":"2024-01-15T18:18:32.217238+0000","last_active":"2024-01-15T18:18:42.257846+0000","last_peered":"2024-01-15T18:18:42.257846+0000","last_clean":"2024-01-15T18:18:42.257846+0000","last_became_active":"2024-01-15T18:18:32.216492+0000","last_became_peered":"2024-01-15T18:18:32.216492+0000","last_unstale":"2024-01-15T18:18:42.257846+0000","last_undegraded":"2024-01-15T18:18:42.257846+0000","last_fullsized":"2024-01-15T18:18:42.257846+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:40:25.010807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"110'36","reported_seq":54,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.447490+0000","last_change":"2024-01-15T18:18:33.157996+0000","last_active":"2024-01-15T18:23:23.447490+0000","last_peered":"2024-01-15T18:23:23.447490+0000","last_clean":"2024-01-15T18:23:23.447490+0000","last_became_active":"2024-01-15T18:18:33.157687+0000","last_became_peered":"2024-01-15T18:18:33.157687+0000","last_unstale":"2024-01-15T18:23:23.447490+0000","last_undegraded":"2024-01-15T18:23:23.447490+0000","last_fullsized":"2024-01-15T18:23:23.447490+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:26:16.219599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"107'1","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:21:04.173272+0000","last_change":"2024-01-15T18:18:32.215007+0000","last_active":"2024-01-15T18:21:04.173272+0000","last_peered":"2024-01-15T18:21:04.173272+0000","last_clean":"2024-01-15T18:21:04.173272+0000","last_became_active":"2024-01-15T18:18:32.214453+0000","last_became_peered":"2024-01-15T18:18:32.214453+0000","last_unstale":"2024-01-15T18:21:04.173272+0000","last_undegraded":"2024-01-15T18:21:04.173272+0000","last_fullsized":"2024-01-15T18:21:04.173272+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:58:50.916786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"110'44","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.457859+0000","last_change":"2024-01-15T18:18:33.153355+0000","last_active":"2024-01-15T18:23:23.457859+0000","last_peered":"2024-01-15T18:23:23.457859+0000","last_clean":"2024-01-15T18:23:23.457859+0000","last_became_active":"2024-01-15T18:18:33.153058+0000","last_became_peered":"2024-01-15T18:18:33.153058+0000","last_unstale":"2024-01-15T18:23:23.457859+0000","last_undegraded":"2024-01-15T18:23:23.457859+0000","last_fullsized":"2024-01-15T18:23:23.457859+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:21:18.264946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"107'1","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:18:43.638172+0000","last_change":"2024-01-15T18:18:32.215681+0000","last_active":"2024-01-15T18:18:43.638172+0000","last_peered":"2024-01-15T18:18:43.638172+0000","last_clean":"2024-01-15T18:18:43.638172+0000","last_became_active":"2024-01-15T18:18:32.215417+0000","last_became_peered":"2024-01-15T18:18:32.215417+0000","last_unstale":"2024-01-15T18:18:43.638172+0000","last_undegraded":"2024-01-15T18:18:43.638172+0000","last_fullsized":"2024-01-15T18:18:43.638172+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:37:03.114460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"110'45","reported_seq":60,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.446039+0000","last_change":"2024-01-15T18:18:43.024618+0000","last_active":"2024-01-15T18:23:23.446039+0000","last_peered":"2024-01-15T18:23:23.446039+0000","last_clean":"2024-01-15T18:23:23.446039+0000","last_became_active":"2024-01-15T18:18:43.024221+0000","last_became_peered":"2024-01-15T18:18:43.024221+0000","last_unstale":"2024-01-15T18:23:23.446039+0000","last_undegraded":"2024-01-15T18:23:23.446039+0000","last_fullsized":"2024-01-15T18:23:23.446039+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:27.549069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303360,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:21:05.181499+0000","last_change":"2024-01-15T18:18:32.217097+0000","last_active":"2024-01-15T18:21:05.181499+0000","last_peered":"2024-01-15T18:21:05.181499+0000","last_clean":"2024-01-15T18:21:05.181499+0000","last_became_active":"2024-01-15T18:18:32.215465+0000","last_became_peered":"2024-01-15T18:18:32.215465+0000","last_unstale":"2024-01-15T18:21:05.181499+0000","last_undegraded":"2024-01-15T18:21:05.181499+0000","last_fullsized":"2024-01-15T18:21:05.181499+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:47.729333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"110'43","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.553963+0000","last_change":"2024-01-15T18:18:33.154067+0000","last_active":"2024-01-15T18:23:31.553963+0000","last_peered":"2024-01-15T18:23:31.553963+0000","last_clean":"2024-01-15T18:23:31.553963+0000","last_became_active":"2024-01-15T18:18:33.153610+0000","last_became_peered":"2024-01-15T18:18:33.153610+0000","last_unstale":"2024-01-15T18:23:31.553963+0000","last_undegraded":"2024-01-15T18:23:31.553963+0000","last_fullsized":"2024-01-15T18:23:31.553963+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:08:41.862115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:59.742931+0000","last_change":"2024-01-15T18:18:32.219909+0000","last_active":"2024-01-15T18:20:59.742931+0000","last_peered":"2024-01-15T18:20:59.742931+0000","last_clean":"2024-01-15T18:20:59.742931+0000","last_became_active":"2024-01-15T18:18:32.219555+0000","last_became_peered":"2024-01-15T18:18:32.219555+0000","last_unstale":"2024-01-15T18:20:59.742931+0000","last_undegraded":"2024-01-15T18:20:59.742931+0000","last_fullsized":"2024-01-15T18:20:59.742931+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:17:21.343043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"110'42","reported_seq":61,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:30.677222+0000","last_change":"2024-01-15T18:18:33.156038+0000","last_active":"2024-01-15T18:23:30.677222+0000","last_peered":"2024-01-15T18:23:30.677222+0000","last_clean":"2024-01-15T18:23:30.677222+0000","last_became_active":"2024-01-15T18:18:33.155765+0000","last_became_peered":"2024-01-15T18:18:33.155765+0000","last_unstale":"2024-01-15T18:23:30.677222+0000","last_undegraded":"2024-01-15T18:23:30.677222+0000","last_fullsized":"2024-01-15T18:23:30.677222+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:58:43.210443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"110'1","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:21:00.168575+0000","last_change":"2024-01-15T18:18:32.212880+0000","last_active":"2024-01-15T18:21:00.168575+0000","last_peered":"2024-01-15T18:21:00.168575+0000","last_clean":"2024-01-15T18:21:00.168575+0000","last_became_active":"2024-01-15T18:18:32.212511+0000","last_became_peered":"2024-01-15T18:18:32.212511+0000","last_unstale":"2024-01-15T18:21:00.168575+0000","last_undegraded":"2024-01-15T18:21:00.168575+0000","last_fullsized":"2024-01-15T18:21:00.168575+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:58:38.191140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"110'35","reported_seq":53,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.444028+0000","last_change":"2024-01-15T18:18:33.164800+0000","last_active":"2024-01-15T18:23:23.444028+0000","last_peered":"2024-01-15T18:23:23.444028+0000","last_clean":"2024-01-15T18:23:23.444028+0000","last_became_active":"2024-01-15T18:18:33.164530+0000","last_became_peered":"2024-01-15T18:18:33.164530+0000","last_unstale":"2024-01-15T18:23:23.444028+0000","last_undegraded":"2024-01-15T18:23:23.444028+0000","last_fullsized":"2024-01-15T18:23:23.444028+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:55.175065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:21:00.437395+0000","last_change":"2024-01-15T18:18:32.216873+0000","last_active":"2024-01-15T18:21:00.437395+0000","last_peered":"2024-01-15T18:21:00.437395+0000","last_clean":"2024-01-15T18:21:00.437395+0000","last_became_active":"2024-01-15T18:18:32.216337+0000","last_became_peered":"2024-01-15T18:18:32.216337+0000","last_unstale":"2024-01-15T18:21:00.437395+0000","last_undegraded":"2024-01-15T18:21:00.437395+0000","last_fullsized":"2024-01-15T18:21:00.437395+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:31:02.176575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"110'41","reported_seq":61,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.460483+0000","last_change":"2024-01-15T18:18:33.148278+0000","last_active":"2024-01-15T18:23:23.460483+0000","last_peered":"2024-01-15T18:23:23.460483+0000","last_clean":"2024-01-15T18:23:23.460483+0000","last_became_active":"2024-01-15T18:18:33.148016+0000","last_became_peered":"2024-01-15T18:18:33.148016+0000","last_unstale":"2024-01-15T18:23:23.460483+0000","last_undegraded":"2024-01-15T18:23:23.460483+0000","last_fullsized":"2024-01-15T18:23:23.460483+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:24.034308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"110'12","reported_seq":28,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:21:26.352083+0000","last_change":"2024-01-15T18:18:32.214473+0000","last_active":"2024-01-15T18:21:26.352083+0000","last_peered":"2024-01-15T18:21:26.352083+0000","last_clean":"2024-01-15T18:21:26.352083+0000","last_became_active":"2024-01-15T18:18:32.214186+0000","last_became_peered":"2024-01-15T18:18:32.214186+0000","last_unstale":"2024-01-15T18:21:26.352083+0000","last_undegraded":"2024-01-15T18:21:26.352083+0000","last_fullsized":"2024-01-15T18:21:26.352083+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:15:38.528251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"110'43","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.563032+0000","last_change":"2024-01-15T18:18:33.159329+0000","last_active":"2024-01-15T18:23:31.563032+0000","last_peered":"2024-01-15T18:23:31.563032+0000","last_clean":"2024-01-15T18:23:31.563032+0000","last_became_active":"2024-01-15T18:18:33.158541+0000","last_became_peered":"2024-01-15T18:18:33.158541+0000","last_unstale":"2024-01-15T18:23:31.563032+0000","last_undegraded":"2024-01-15T18:23:31.563032+0000","last_fullsized":"2024-01-15T18:23:31.563032+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:53:07.912473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"110'170","reported_seq":186,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.466042+0000","last_change":"2024-01-15T18:18:32.218321+0000","last_active":"2024-01-15T18:23:23.466042+0000","last_peered":"2024-01-15T18:23:23.466042+0000","last_clean":"2024-01-15T18:23:23.466042+0000","last_became_active":"2024-01-15T18:18:32.218002+0000","last_became_peered":"2024-01-15T18:18:32.218002+0000","last_unstale":"2024-01-15T18:23:23.466042+0000","last_undegraded":"2024-01-15T18:23:23.466042+0000","last_fullsized":"2024-01-15T18:23:23.466042+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:12:40.196083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":567317,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":169,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"110'36","reported_seq":53,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.455178+0000","last_change":"2024-01-15T18:18:33.151414+0000","last_active":"2024-01-15T18:23:23.455178+0000","last_peered":"2024-01-15T18:23:23.455178+0000","last_clean":"2024-01-15T18:23:23.455178+0000","last_became_active":"2024-01-15T18:18:33.151132+0000","last_became_peered":"2024-01-15T18:18:33.151132+0000","last_unstale":"2024-01-15T18:23:23.455178+0000","last_undegraded":"2024-01-15T18:23:23.455178+0000","last_fullsized":"2024-01-15T18:23:23.455178+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:48:57.895180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"107'2","reported_seq":18,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:55.470178+0000","last_change":"2024-01-15T18:18:32.215335+0000","last_active":"2024-01-15T18:20:55.470178+0000","last_peered":"2024-01-15T18:20:55.470178+0000","last_clean":"2024-01-15T18:20:55.470178+0000","last_became_active":"2024-01-15T18:18:32.215073+0000","last_became_peered":"2024-01-15T18:18:32.215073+0000","last_unstale":"2024-01-15T18:20:55.470178+0000","last_undegraded":"2024-01-15T18:20:55.470178+0000","last_fullsized":"2024-01-15T18:20:55.470178+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:04:07.912459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"110'46","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.458337+0000","last_change":"2024-01-15T18:18:33.161775+0000","last_active":"2024-01-15T18:23:23.458337+0000","last_peered":"2024-01-15T18:23:23.458337+0000","last_clean":"2024-01-15T18:23:23.458337+0000","last_became_active":"2024-01-15T18:18:33.161493+0000","last_became_peered":"2024-01-15T18:18:33.161493+0000","last_unstale":"2024-01-15T18:23:23.458337+0000","last_undegraded":"2024-01-15T18:23:23.458337+0000","last_fullsized":"2024-01-15T18:23:23.458337+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:55.175065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:55.488186+0000","last_change":"2024-01-15T18:18:32.215225+0000","last_active":"2024-01-15T18:20:55.488186+0000","last_peered":"2024-01-15T18:20:55.488186+0000","last_clean":"2024-01-15T18:20:55.488186+0000","last_became_active":"2024-01-15T18:18:32.214942+0000","last_became_peered":"2024-01-15T18:18:32.214942+0000","last_unstale":"2024-01-15T18:20:55.488186+0000","last_undegraded":"2024-01-15T18:20:55.488186+0000","last_fullsized":"2024-01-15T18:20:55.488186+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:09:53.884883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"110'43","reported_seq":61,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.460487+0000","last_change":"2024-01-15T18:18:33.146254+0000","last_active":"2024-01-15T18:23:23.460487+0000","last_peered":"2024-01-15T18:23:23.460487+0000","last_clean":"2024-01-15T18:23:23.460487+0000","last_became_active":"2024-01-15T18:18:33.145973+0000","last_became_peered":"2024-01-15T18:18:33.145973+0000","last_unstale":"2024-01-15T18:23:23.460487+0000","last_undegraded":"2024-01-15T18:23:23.460487+0000","last_fullsized":"2024-01-15T18:23:23.460487+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:09:54.894718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080645,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":22,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"107'8","reported_seq":24,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:55.167804+0000","last_change":"2024-01-15T18:18:32.219579+0000","last_active":"2024-01-15T18:20:55.167804+0000","last_peered":"2024-01-15T18:20:55.167804+0000","last_clean":"2024-01-15T18:20:55.167804+0000","last_became_active":"2024-01-15T18:18:32.219280+0000","last_became_peered":"2024-01-15T18:18:32.219280+0000","last_unstale":"2024-01-15T18:20:55.167804+0000","last_undegraded":"2024-01-15T18:20:55.167804+0000","last_fullsized":"2024-01-15T18:20:55.167804+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:56:09.818155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"110'21","reported_seq":38,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.451431+0000","last_change":"2024-01-15T18:18:33.161674+0000","last_active":"2024-01-15T18:23:23.451431+0000","last_peered":"2024-01-15T18:23:23.451431+0000","last_clean":"2024-01-15T18:23:23.451431+0000","last_became_active":"2024-01-15T18:18:33.161329+0000","last_became_peered":"2024-01-15T18:18:33.161329+0000","last_unstale":"2024-01-15T18:23:23.451431+0000","last_undegraded":"2024-01-15T18:23:23.451431+0000","last_fullsized":"2024-01-15T18:23:23.451431+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:45:56.254343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":7,"num_write_kb":28672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"107'1","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:49.741619+0000","last_change":"2024-01-15T18:18:32.215848+0000","last_active":"2024-01-15T18:20:49.741619+0000","last_peered":"2024-01-15T18:20:49.741619+0000","last_clean":"2024-01-15T18:20:49.741619+0000","last_became_active":"2024-01-15T18:18:32.214009+0000","last_became_peered":"2024-01-15T18:18:32.214009+0000","last_unstale":"2024-01-15T18:20:49.741619+0000","last_undegraded":"2024-01-15T18:20:49.741619+0000","last_fullsized":"2024-01-15T18:20:49.741619+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:48:09.722577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"110'35","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.453210+0000","last_change":"2024-01-15T18:18:33.162974+0000","last_active":"2024-01-15T18:23:23.453210+0000","last_peered":"2024-01-15T18:23:23.453210+0000","last_clean":"2024-01-15T18:23:23.453210+0000","last_became_active":"2024-01-15T18:18:33.162679+0000","last_became_peered":"2024-01-15T18:18:33.162679+0000","last_unstale":"2024-01-15T18:23:23.453210+0000","last_undegraded":"2024-01-15T18:23:23.453210+0000","last_fullsized":"2024-01-15T18:23:23.453210+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:53:15.036059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"107'2","reported_seq":18,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:50.167169+0000","last_change":"2024-01-15T18:18:32.214990+0000","last_active":"2024-01-15T18:20:50.167169+0000","last_peered":"2024-01-15T18:20:50.167169+0000","last_clean":"2024-01-15T18:20:50.167169+0000","last_became_active":"2024-01-15T18:18:32.214695+0000","last_became_peered":"2024-01-15T18:18:32.214695+0000","last_unstale":"2024-01-15T18:20:50.167169+0000","last_undegraded":"2024-01-15T18:20:50.167169+0000","last_fullsized":"2024-01-15T18:20:50.167169+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:15:08.445774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"110'41","reported_seq":63,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.450293+0000","last_change":"2024-01-15T18:18:33.168531+0000","last_active":"2024-01-15T18:23:23.450293+0000","last_peered":"2024-01-15T18:23:23.450293+0000","last_clean":"2024-01-15T18:23:23.450293+0000","last_became_active":"2024-01-15T18:18:33.168190+0000","last_became_peered":"2024-01-15T18:18:33.168190+0000","last_unstale":"2024-01-15T18:23:23.450293+0000","last_undegraded":"2024-01-15T18:23:23.450293+0000","last_fullsized":"2024-01-15T18:23:23.450293+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:53:05.325329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":7,"num_read_kb":12292,"num_write":19,"num_write_kb":61442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"107'1","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:50.487644+0000","last_change":"2024-01-15T18:18:32.221200+0000","last_active":"2024-01-15T18:20:50.487644+0000","last_peered":"2024-01-15T18:20:50.487644+0000","last_clean":"2024-01-15T18:20:50.487644+0000","last_became_active":"2024-01-15T18:18:32.220884+0000","last_became_peered":"2024-01-15T18:18:32.220884+0000","last_unstale":"2024-01-15T18:20:50.487644+0000","last_undegraded":"2024-01-15T18:20:50.487644+0000","last_fullsized":"2024-01-15T18:20:50.487644+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:02.938267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"110'40","reported_seq":60,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.454892+0000","last_change":"2024-01-15T18:18:33.163114+0000","last_active":"2024-01-15T18:23:23.454892+0000","last_peered":"2024-01-15T18:23:23.454892+0000","last_clean":"2024-01-15T18:23:23.454892+0000","last_became_active":"2024-01-15T18:18:33.162424+0000","last_became_peered":"2024-01-15T18:18:33.162424+0000","last_unstale":"2024-01-15T18:23:23.454892+0000","last_undegraded":"2024-01-15T18:23:23.454892+0000","last_fullsized":"2024-01-15T18:23:23.454892+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:42:08.107694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:49.171310+0000","last_change":"2024-01-15T18:18:32.217252+0000","last_active":"2024-01-15T18:20:49.171310+0000","last_peered":"2024-01-15T18:20:49.171310+0000","last_clean":"2024-01-15T18:20:49.171310+0000","last_became_active":"2024-01-15T18:18:32.216769+0000","last_became_peered":"2024-01-15T18:18:32.216769+0000","last_unstale":"2024-01-15T18:20:49.171310+0000","last_undegraded":"2024-01-15T18:20:49.171310+0000","last_fullsized":"2024-01-15T18:20:49.171310+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:36.807888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"110'34","reported_seq":53,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.456232+0000","last_change":"2024-01-15T18:18:33.163017+0000","last_active":"2024-01-15T18:23:23.456232+0000","last_peered":"2024-01-15T18:23:23.456232+0000","last_clean":"2024-01-15T18:23:23.456232+0000","last_became_active":"2024-01-15T18:18:33.162658+0000","last_became_peered":"2024-01-15T18:18:33.162658+0000","last_unstale":"2024-01-15T18:23:23.456232+0000","last_undegraded":"2024-01-15T18:23:23.456232+0000","last_fullsized":"2024-01-15T18:23:23.456232+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:01:41.498479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"110'3","reported_seq":19,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:50.469624+0000","last_change":"2024-01-15T18:18:32.207796+0000","last_active":"2024-01-15T18:20:50.469624+0000","last_peered":"2024-01-15T18:20:50.469624+0000","last_clean":"2024-01-15T18:20:50.469624+0000","last_became_active":"2024-01-15T18:18:32.207507+0000","last_became_peered":"2024-01-15T18:18:32.207507+0000","last_unstale":"2024-01-15T18:20:50.469624+0000","last_undegraded":"2024-01-15T18:20:50.469624+0000","last_fullsized":"2024-01-15T18:20:50.469624+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:42.592200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"110'39","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.460291+0000","last_change":"2024-01-15T18:18:33.164981+0000","last_active":"2024-01-15T18:23:23.460291+0000","last_peered":"2024-01-15T18:23:23.460291+0000","last_clean":"2024-01-15T18:23:23.460291+0000","last_became_active":"2024-01-15T18:18:33.164206+0000","last_became_peered":"2024-01-15T18:18:33.164206+0000","last_unstale":"2024-01-15T18:23:23.460291+0000","last_undegraded":"2024-01-15T18:23:23.460291+0000","last_fullsized":"2024-01-15T18:23:23.460291+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:53:32.550071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:50.167117+0000","last_change":"2024-01-15T18:18:32.212499+0000","last_active":"2024-01-15T18:20:50.167117+0000","last_peered":"2024-01-15T18:20:50.167117+0000","last_clean":"2024-01-15T18:20:50.167117+0000","last_became_active":"2024-01-15T18:18:32.212201+0000","last_became_peered":"2024-01-15T18:18:32.212201+0000","last_unstale":"2024-01-15T18:20:50.167117+0000","last_undegraded":"2024-01-15T18:20:50.167117+0000","last_fullsized":"2024-01-15T18:20:50.167117+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:25:39.031903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"110'40","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.453577+0000","last_change":"2024-01-15T18:18:33.165046+0000","last_active":"2024-01-15T18:23:23.453577+0000","last_peered":"2024-01-15T18:23:23.453577+0000","last_clean":"2024-01-15T18:23:23.453577+0000","last_became_active":"2024-01-15T18:18:33.164708+0000","last_became_peered":"2024-01-15T18:18:33.164708+0000","last_unstale":"2024-01-15T18:23:23.453577+0000","last_undegraded":"2024-01-15T18:23:23.453577+0000","last_fullsized":"2024-01-15T18:23:23.453577+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:46:54.687765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:45.468959+0000","last_change":"2024-01-15T18:18:32.207872+0000","last_active":"2024-01-15T18:20:45.468959+0000","last_peered":"2024-01-15T18:20:45.468959+0000","last_clean":"2024-01-15T18:20:45.468959+0000","last_became_active":"2024-01-15T18:18:32.207599+0000","last_became_peered":"2024-01-15T18:18:32.207599+0000","last_unstale":"2024-01-15T18:20:45.468959+0000","last_undegraded":"2024-01-15T18:20:45.468959+0000","last_fullsized":"2024-01-15T18:20:45.468959+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:25:20.009100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"110'40","reported_seq":59,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.450620+0000","last_change":"2024-01-15T18:18:33.160618+0000","last_active":"2024-01-15T18:23:23.450620+0000","last_peered":"2024-01-15T18:23:23.450620+0000","last_clean":"2024-01-15T18:23:23.450620+0000","last_became_active":"2024-01-15T18:18:33.160350+0000","last_became_peered":"2024-01-15T18:18:33.160350+0000","last_unstale":"2024-01-15T18:23:23.450620+0000","last_undegraded":"2024-01-15T18:23:23.450620+0000","last_fullsized":"2024-01-15T18:23:23.450620+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:42:11.703397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720517,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":15,"num_write_kb":57345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:44.740758+0000","last_change":"2024-01-15T18:18:32.217214+0000","last_active":"2024-01-15T18:20:44.740758+0000","last_peered":"2024-01-15T18:20:44.740758+0000","last_clean":"2024-01-15T18:20:44.740758+0000","last_became_active":"2024-01-15T18:18:32.216581+0000","last_became_peered":"2024-01-15T18:18:32.216581+0000","last_unstale":"2024-01-15T18:20:44.740758+0000","last_undegraded":"2024-01-15T18:20:44.740758+0000","last_fullsized":"2024-01-15T18:20:44.740758+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:49:26.780801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"110'33","reported_seq":50,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.444182+0000","last_change":"2024-01-15T18:18:33.165549+0000","last_active":"2024-01-15T18:23:23.444182+0000","last_peered":"2024-01-15T18:23:23.444182+0000","last_clean":"2024-01-15T18:23:23.444182+0000","last_became_active":"2024-01-15T18:18:33.164152+0000","last_became_peered":"2024-01-15T18:18:33.164152+0000","last_unstale":"2024-01-15T18:23:23.444182+0000","last_undegraded":"2024-01-15T18:23:23.444182+0000","last_fullsized":"2024-01-15T18:23:23.444182+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:24.034308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"107'2","reported_seq":18,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:45.435609+0000","last_change":"2024-01-15T18:18:32.219821+0000","last_active":"2024-01-15T18:20:45.435609+0000","last_peered":"2024-01-15T18:20:45.435609+0000","last_clean":"2024-01-15T18:20:45.435609+0000","last_became_active":"2024-01-15T18:18:32.218858+0000","last_became_peered":"2024-01-15T18:18:32.218858+0000","last_unstale":"2024-01-15T18:20:45.435609+0000","last_undegraded":"2024-01-15T18:20:45.435609+0000","last_fullsized":"2024-01-15T18:20:45.435609+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:33:51.257660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"110'37","reported_seq":54,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.462300+0000","last_change":"2024-01-15T18:18:33.160602+0000","last_active":"2024-01-15T18:23:23.462300+0000","last_peered":"2024-01-15T18:23:23.462300+0000","last_clean":"2024-01-15T18:23:23.462300+0000","last_became_active":"2024-01-15T18:18:33.160315+0000","last_became_peered":"2024-01-15T18:18:33.160315+0000","last_unstale":"2024-01-15T18:23:23.462300+0000","last_undegraded":"2024-01-15T18:23:23.462300+0000","last_fullsized":"2024-01-15T18:23:23.462300+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:34.698586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":2,"num_read_kb":8192,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"110'16","reported_seq":32,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:22:55.502681+0000","last_change":"2024-01-15T18:18:32.217029+0000","last_active":"2024-01-15T18:22:55.502681+0000","last_peered":"2024-01-15T18:22:55.502681+0000","last_clean":"2024-01-15T18:22:55.502681+0000","last_became_active":"2024-01-15T18:18:32.216612+0000","last_became_peered":"2024-01-15T18:18:32.216612+0000","last_unstale":"2024-01-15T18:22:55.502681+0000","last_undegraded":"2024-01-15T18:22:55.502681+0000","last_fullsized":"2024-01-15T18:22:55.502681+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:42:10.693562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"110'42","reported_seq":63,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.452715+0000","last_change":"2024-01-15T18:18:33.165599+0000","last_active":"2024-01-15T18:23:23.452715+0000","last_peered":"2024-01-15T18:23:23.452715+0000","last_clean":"2024-01-15T18:23:23.452715+0000","last_became_active":"2024-01-15T18:18:33.163994+0000","last_became_peered":"2024-01-15T18:18:33.163994+0000","last_unstale":"2024-01-15T18:23:23.452715+0000","last_undegraded":"2024-01-15T18:23:23.452715+0000","last_fullsized":"2024-01-15T18:23:23.452715+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:00:26.779661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:39.740198+0000","last_change":"2024-01-15T18:18:32.216301+0000","last_active":"2024-01-15T18:20:39.740198+0000","last_peered":"2024-01-15T18:20:39.740198+0000","last_clean":"2024-01-15T18:20:39.740198+0000","last_became_active":"2024-01-15T18:18:32.216023+0000","last_became_peered":"2024-01-15T18:18:32.216023+0000","last_unstale":"2024-01-15T18:20:39.740198+0000","last_undegraded":"2024-01-15T18:20:39.740198+0000","last_fullsized":"2024-01-15T18:20:39.740198+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:58:58.730569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"110'34","reported_seq":60,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.444257+0000","last_change":"2024-01-15T18:18:42.262610+0000","last_active":"2024-01-15T18:23:23.444257+0000","last_peered":"2024-01-15T18:23:23.444257+0000","last_clean":"2024-01-15T18:23:23.444257+0000","last_became_active":"2024-01-15T18:18:42.262255+0000","last_became_peered":"2024-01-15T18:18:42.262255+0000","last_unstale":"2024-01-15T18:23:23.444257+0000","last_undegraded":"2024-01-15T18:23:23.444257+0000","last_fullsized":"2024-01-15T18:23:23.444257+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:22:41.967700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331770,"num_objects":14,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":5,"num_read_kb":20480,"num_write":14,"num_write_kb":49153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:40.434982+0000","last_change":"2024-01-15T18:18:32.214224+0000","last_active":"2024-01-15T18:20:40.434982+0000","last_peered":"2024-01-15T18:20:40.434982+0000","last_clean":"2024-01-15T18:20:40.434982+0000","last_became_active":"2024-01-15T18:18:32.213930+0000","last_became_peered":"2024-01-15T18:18:32.213930+0000","last_unstale":"2024-01-15T18:20:40.434982+0000","last_undegraded":"2024-01-15T18:20:40.434982+0000","last_fullsized":"2024-01-15T18:20:40.434982+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:01:15.328258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"110'44","reported_seq":60,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.454715+0000","last_change":"2024-01-15T18:18:33.150974+0000","last_active":"2024-01-15T18:23:23.454715+0000","last_peered":"2024-01-15T18:23:23.454715+0000","last_clean":"2024-01-15T18:23:23.454715+0000","last_became_active":"2024-01-15T18:18:33.150660+0000","last_became_peered":"2024-01-15T18:18:33.150660+0000","last_unstale":"2024-01-15T18:23:23.454715+0000","last_undegraded":"2024-01-15T18:23:23.454715+0000","last_fullsized":"2024-01-15T18:23:23.454715+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:22:41.967700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":4096,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:39.170034+0000","last_change":"2024-01-15T18:18:32.215229+0000","last_active":"2024-01-15T18:20:39.170034+0000","last_peered":"2024-01-15T18:20:39.170034+0000","last_clean":"2024-01-15T18:20:39.170034+0000","last_became_active":"2024-01-15T18:18:32.214394+0000","last_became_peered":"2024-01-15T18:18:32.214394+0000","last_unstale":"2024-01-15T18:20:39.170034+0000","last_undegraded":"2024-01-15T18:20:39.170034+0000","last_fullsized":"2024-01-15T18:20:39.170034+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:30:25.477307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"110'51","reported_seq":72,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.458997+0000","last_change":"2024-01-15T18:18:33.156094+0000","last_active":"2024-01-15T18:23:23.458997+0000","last_peered":"2024-01-15T18:23:23.458997+0000","last_clean":"2024-01-15T18:23:23.458997+0000","last_became_active":"2024-01-15T18:18:33.155797+0000","last_became_peered":"2024-01-15T18:18:33.155797+0000","last_unstale":"2024-01-15T18:23:23.458997+0000","last_undegraded":"2024-01-15T18:23:23.458997+0000","last_fullsized":"2024-01-15T18:23:23.458997+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:21:42.101901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:40.178098+0000","last_change":"2024-01-15T18:18:32.212760+0000","last_active":"2024-01-15T18:20:40.178098+0000","last_peered":"2024-01-15T18:20:40.178098+0000","last_clean":"2024-01-15T18:20:40.178098+0000","last_became_active":"2024-01-15T18:18:32.212487+0000","last_became_peered":"2024-01-15T18:18:32.212487+0000","last_unstale":"2024-01-15T18:20:40.178098+0000","last_undegraded":"2024-01-15T18:20:40.178098+0000","last_fullsized":"2024-01-15T18:20:40.178098+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:48:56.885346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"110'38","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.450335+0000","last_change":"2024-01-15T18:18:33.167732+0000","last_active":"2024-01-15T18:23:23.450335+0000","last_peered":"2024-01-15T18:23:23.450335+0000","last_clean":"2024-01-15T18:23:23.450335+0000","last_became_active":"2024-01-15T18:18:33.167436+0000","last_became_peered":"2024-01-15T18:18:33.167436+0000","last_unstale":"2024-01-15T18:23:23.450335+0000","last_undegraded":"2024-01-15T18:23:23.450335+0000","last_fullsized":"2024-01-15T18:23:23.450335+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:22:13.319815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"110'11","reported_seq":37,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:21:26.348323+0000","last_change":"2024-01-15T18:18:32.213077+0000","last_active":"2024-01-15T18:21:26.348323+0000","last_peered":"2024-01-15T18:21:26.348323+0000","last_clean":"2024-01-15T18:21:26.348323+0000","last_became_active":"2024-01-15T18:18:32.211744+0000","last_became_peered":"2024-01-15T18:18:32.211744+0000","last_unstale":"2024-01-15T18:21:26.348323+0000","last_undegraded":"2024-01-15T18:21:26.348323+0000","last_fullsized":"2024-01-15T18:21:26.348323+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:48:24.967174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1321,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"110'36","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.452282+0000","last_change":"2024-01-15T18:18:33.163355+0000","last_active":"2024-01-15T18:23:23.452282+0000","last_peered":"2024-01-15T18:23:23.452282+0000","last_clean":"2024-01-15T18:23:23.452282+0000","last_became_active":"2024-01-15T18:18:33.162521+0000","last_became_peered":"2024-01-15T18:18:33.162521+0000","last_unstale":"2024-01-15T18:23:23.452282+0000","last_undegraded":"2024-01-15T18:23:23.452282+0000","last_fullsized":"2024-01-15T18:23:23.452282+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:35:00.923337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:34.739313+0000","last_change":"2024-01-15T18:18:32.217455+0000","last_active":"2024-01-15T18:20:34.739313+0000","last_peered":"2024-01-15T18:20:34.739313+0000","last_clean":"2024-01-15T18:20:34.739313+0000","last_became_active":"2024-01-15T18:18:32.216966+0000","last_became_peered":"2024-01-15T18:18:32.216966+0000","last_unstale":"2024-01-15T18:20:34.739313+0000","last_undegraded":"2024-01-15T18:20:34.739313+0000","last_fullsized":"2024-01-15T18:20:34.739313+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:56:34.922952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"110'30","reported_seq":50,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:23.458512+0000","last_change":"2024-01-15T18:18:33.166946+0000","last_active":"2024-01-15T18:23:23.458512+0000","last_peered":"2024-01-15T18:23:23.458512+0000","last_clean":"2024-01-15T18:23:23.458512+0000","last_became_active":"2024-01-15T18:18:33.165323+0000","last_became_peered":"2024-01-15T18:18:33.165323+0000","last_unstale":"2024-01-15T18:23:23.458512+0000","last_undegraded":"2024-01-15T18:23:23.458512+0000","last_fullsized":"2024-01-15T18:23:23.458512+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:18.639432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:54.171847+0000","last_change":"2024-01-15T18:18:32.216251+0000","last_active":"2024-01-15T18:20:54.171847+0000","last_peered":"2024-01-15T18:20:54.171847+0000","last_clean":"2024-01-15T18:20:54.171847+0000","last_became_active":"2024-01-15T18:18:32.215973+0000","last_became_peered":"2024-01-15T18:18:32.215973+0000","last_unstale":"2024-01-15T18:20:54.171847+0000","last_undegraded":"2024-01-15T18:20:54.171847+0000","last_fullsized":"2024-01-15T18:20:54.171847+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:41:26.812763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":244,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:50.171227+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:20:50.171227+0000","last_peered":"2024-01-15T18:20:50.171227+0000","last_clean":"2024-01-15T18:20:50.171227+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:20:50.171227+0000","last_undegraded":"2024-01-15T18:20:50.171227+0000","last_fullsized":"2024-01-15T18:20:50.171227+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":4353688568,"num_objects":1044,"num_object_clones":1,"num_object_copies":2088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1044,"num_whiteouts":1,"num_read":292,"num_read_kb":1142797,"num_write":1064,"num_write_kb":4251663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8761942016,"data_stored":8761903088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2568,"ondisk_log_size":2568,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":569436,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":260,"num_write_kb":669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1220608,"data_stored":1154557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":241,"ondisk_log_size":241,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509668,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1465860,"kb_used_data":1398652,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92905980,"statfs":{"total":96636764160,"available":95135723520,"internally_reserved":0,"allocated":1432219648,"data_stored":1431443111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:22:31 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.71499999999999997,"15min":0.90300000000000002},"min":{"1min":0.437,"5min":0.36799999999999999,"15min":0.35499999999999998},"max":{"1min":2.8620000000000001,"5min":2.8620000000000001,"15min":10.284000000000001},"last":0.63},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.67600000000000005,"15min":0.92800000000000005},"min":{"1min":0.54900000000000004,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.90000000000000002,"5min":0.92700000000000005,"15min":16.968},"last":0.67900000000000005}]},{"osd":1,"last update":"Mon Jan 15 18:22:31 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.69599999999999995,"15min":0.93400000000000005},"min":{"1min":0.57499999999999996,"5min":0.38600000000000001,"15min":0.33800000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":15.791},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.72699999999999998,"15min":0.92900000000000005},"min":{"1min":0.46999999999999997,"5min":0.44,"15min":0.371},"max":{"1min":1.248,"5min":1.248,"15min":10.842000000000001},"last":0.58299999999999996}]},{"osd":2,"last update":"Mon Jan 15 18:22:31 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.71299999999999997,"15min":0.95999999999999996},"min":{"1min":0.52600000000000002,"5min":0.39500000000000002,"15min":0.378},"max":{"1min":0.98299999999999998,"5min":0.98799999999999999,"15min":22.318000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.878,"5min":0.73399999999999999,"15min":0.91300000000000003},"min":{"1min":0.59699999999999998,"5min":0.42699999999999999,"15min":0.39100000000000001},"max":{"1min":2.2799999999999998,"5min":2.2799999999999998,"15min":9.7539999999999996},"last":0.69299999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:22:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.72799999999999998,"15min":1.0109999999999999},"min":{"1min":0.59599999999999997,"5min":0.46500000000000002,"15min":0.38100000000000001},"max":{"1min":1.2110000000000001,"5min":2.1739999999999999,"15min":18.152999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.76900000000000002,"15min":0.98999999999999999},"min":{"1min":0.497,"5min":0.495,"15min":0.433},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":10.34},"last":0.51600000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:22:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.57199999999999995,"15min":0.93100000000000005},"min":{"1min":0.40799999999999997,"5min":0.34799999999999998,"15min":0.314},"max":{"1min":0.81000000000000005,"5min":0.91000000000000003,"15min":20.387},"last":0.497},{"interface":"front","average":{"1min":0.623,"5min":0.59499999999999997,"15min":0.96299999999999997},"min":{"1min":0.42499999999999999,"5min":0.38400000000000001,"15min":0.34200000000000003},"max":{"1min":0.82699999999999996,"5min":0.98099999999999998,"15min":17.215},"last":0.65800000000000003}]},{"osd":5,"last update":"Mon Jan 15 18:22:31 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.60799999999999998,"15min":0.91500000000000004},"min":{"1min":0.5,"5min":0.38600000000000001,"15min":0.34699999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":19.959},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63500000000000001,"15min":0.95999999999999996},"min":{"1min":0.46300000000000002,"5min":0.40899999999999997,"15min":0.35799999999999998},"max":{"1min":0.82699999999999996,"5min":0.92600000000000005,"15min":19.905000000000001},"last":0.64500000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:22:31 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.73699999999999999,"15min":1.1419999999999999},"min":{"1min":0.57399999999999995,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.91000000000000003,"5min":0.99099999999999999,"15min":22.478999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.68899999999999995,"15min":1.0509999999999999},"min":{"1min":0.53400000000000003,"5min":0.44600000000000001,"15min":0.38300000000000001},"max":{"1min":0.96299999999999997,"5min":1.012,"15min":21.509},"last":0.71599999999999997}]}]},{"osd":6,"up_from":15,"seq":64424509668,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":998272,"kb_used_data":935224,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":93373568,"statfs":{"total":96636764160,"available":95614533632,"internally_reserved":0,"allocated":957669376,"data_stored":956895967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":0.78400000000000003,"15min":0.99399999999999999},"min":{"1min":0.59099999999999997,"5min":0.36299999999999999,"15min":0.29099999999999998},"max":{"1min":6.1900000000000004,"5min":6.1900000000000004,"15min":32.587000000000003},"last":0.753},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.79100000000000004,"15min":0.90100000000000002},"min":{"1min":0.63900000000000001,"5min":0.434,"15min":0.35399999999999998},"max":{"1min":4.9649999999999999,"5min":4.9649999999999999,"15min":16.193000000000001},"last":0.54900000000000004}]},{"osd":1,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.73799999999999999,"15min":0.996},"min":{"1min":0.56599999999999995,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":53.451999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.73399999999999999,"15min":0.91600000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.88400000000000001,"5min":1.0189999999999999,"15min":32.512},"last":0.69199999999999995}]},{"osd":2,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77000000000000002,"15min":0.95099999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.41699999999999998},"max":{"1min":0.98099999999999998,"5min":1.016,"15min":32.472999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.996,"5min":0.84299999999999997,"15min":0.92100000000000004},"min":{"1min":0.57899999999999996,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":5.0410000000000004,"5min":5.0410000000000004,"15min":16.602},"last":0.71099999999999997}]},{"osd":3,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.82699999999999996,"15min":0.93600000000000005},"min":{"1min":0.64900000000000002,"5min":0.52500000000000002,"15min":0.46000000000000002},"max":{"1min":5.0940000000000003,"5min":5.0940000000000003,"15min":16.388000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.76000000000000001,"15min":1.0609999999999999},"min":{"1min":0.67800000000000005,"5min":0.505,"15min":0.432},"max":{"1min":1.004,"5min":1.04,"15min":53.398000000000003},"last":0.64000000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.67000000000000004,"15min":0.88800000000000001},"min":{"1min":0.41499999999999998,"5min":0.39000000000000001,"15min":0.378},"max":{"1min":6.3869999999999996,"5min":6.3869999999999996,"15min":32.549999999999997},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.68799999999999994,"15min":0.83599999999999997},"min":{"1min":0.45800000000000002,"5min":0.41399999999999998,"15min":0.371},"max":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":11.852},"last":0.49099999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.70499999999999996,"15min":1.052},"min":{"1min":0.433,"5min":0.42199999999999999,"15min":0.34300000000000003},"max":{"1min":6.8769999999999998,"5min":6.8769999999999998,"15min":53.229999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.747,"15min":1.0209999999999999},"min":{"1min":0.49199999999999999,"5min":0.44400000000000001,"15min":0.41799999999999998},"max":{"1min":6.1360000000000001,"5min":6.1360000000000001,"15min":32.404000000000003},"last":0.82199999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.749,"15min":1.0820000000000001},"min":{"1min":0.52200000000000002,"5min":0.46000000000000002,"15min":0.38600000000000001},"max":{"1min":5.1319999999999997,"5min":5.1319999999999997,"15min":45.070999999999998},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.71099999999999997,"15min":1.181},"min":{"1min":0.51100000000000001,"5min":0.47999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89100000000000001,"5min":0.999,"15min":53.338000000000001},"last":0.72999999999999998}]}]},{"osd":4,"up_from":15,"seq":64424509668,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1387388,"kb_used_data":1324340,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":92984452,"statfs":{"total":96636764160,"available":95216078848,"internally_reserved":0,"allocated":1356124160,"data_stored":1355354916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.69799999999999995,"15min":0.90600000000000003},"min":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.32900000000000001},"max":{"1min":0.96599999999999997,"5min":1.542,"15min":23.318000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.70299999999999996,"15min":0.81299999999999994},"min":{"1min":0.55000000000000004,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":14.723000000000001},"last":0.75800000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.72599999999999998,"15min":0.81399999999999995},"min":{"1min":0.46300000000000002,"5min":0.39100000000000001,"15min":0.38800000000000001},"max":{"1min":0.96199999999999997,"5min":1.919,"15min":9.5700000000000003},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.74199999999999999,"15min":0.94699999999999995},"min":{"1min":0.52600000000000002,"5min":0.44900000000000001,"15min":0.36899999999999999},"max":{"1min":0.996,"5min":1.429,"15min":22.408999999999999},"last":0.60799999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.79000000000000004,"15min":0.85299999999999998},"min":{"1min":0.50900000000000001,"5min":0.41899999999999998,"15min":0.39700000000000002},"max":{"1min":1.032,"5min":5.6260000000000003,"15min":9.7089999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.76100000000000001,"15min":1.018},"min":{"1min":0.59299999999999997,"5min":0.45000000000000001,"15min":0.39100000000000001},"max":{"1min":1.016,"5min":1.387,"15min":22.338999999999999},"last":0.52900000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.73999999999999999,"15min":0.88700000000000001},"min":{"1min":0.52300000000000002,"5min":0.46200000000000002,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":1.577,"15min":14.763999999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.82599999999999996,"15min":0.93600000000000005},"min":{"1min":0.65300000000000002,"5min":0.53600000000000003,"15min":0.48099999999999998},"max":{"1min":0.996,"5min":1.9610000000000001,"15min":9.6289999999999996},"last":0.56299999999999994}]},{"osd":5,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63300000000000001,"15min":0.94299999999999995},"min":{"1min":0.44,"5min":0.36499999999999999,"15min":0.34799999999999998},"max":{"1min":1.0149999999999999,"5min":1.1970000000000001,"15min":31.617999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.85799999999999998},"min":{"1min":0.40799999999999997,"5min":0.373,"15min":0.36599999999999999},"max":{"1min":0.86699999999999999,"5min":1.4910000000000001,"15min":17.145},"last":0.81799999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.72799999999999998,"15min":0.83899999999999997},"min":{"1min":0.60799999999999998,"5min":0.41799999999999998,"15min":0.37},"max":{"1min":0.89300000000000002,"5min":5.9489999999999998,"15min":11.124000000000001},"last":0.628},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.71099999999999997,"15min":1.2090000000000001},"min":{"1min":0.52300000000000002,"5min":0.42499999999999999,"15min":0.40200000000000002},"max":{"1min":1.0649999999999999,"5min":2.3149999999999999,"15min":32.308},"last":0.83899999999999997}]},{"osd":7,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.72699999999999998,"15min":1.0009999999999999},"min":{"1min":0.48099999999999998,"5min":0.45100000000000001,"15min":0.34799999999999998},"max":{"1min":1,"5min":2.4329999999999998,"15min":39.256},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78200000000000003,"15min":1.002},"min":{"1min":0.60699999999999998,"5min":0.499,"15min":0.46500000000000002},"max":{"1min":1.04,"5min":2.1909999999999998,"15min":26.573},"last":0.71699999999999997}]}]},{"osd":3,"up_from":15,"seq":64424509667,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1154552,"kb_used_data":1087344,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93217288,"statfs":{"total":96636764160,"available":95454502912,"internally_reserved":0,"allocated":1113440256,"data_stored":1112652932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.45100000000000001,"15min":0.498},"min":{"1min":0.28999999999999998,"5min":0.28199999999999997,"15min":0.245},"max":{"1min":0.69099999999999995,"5min":0.69699999999999995,"15min":5.8849999999999998},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.51000000000000001,"15min":0.621},"min":{"1min":0.41799999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.80000000000000004,"5min":1.4750000000000001,"15min":17.756},"last":0.52000000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51300000000000001,"15min":0.60499999999999998},"min":{"1min":0.33500000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60399999999999998,"5min":4.0099999999999998,"15min":12.307},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55200000000000005,"15min":0.59199999999999997},"min":{"1min":0.36199999999999999,"5min":0.34300000000000003,"15min":0.31900000000000001},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":6.0640000000000001},"last":0.54300000000000004}]},{"osd":2,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61399999999999999,"15min":0.71199999999999997},"min":{"1min":0.42799999999999999,"5min":0.35299999999999998,"15min":0.32600000000000001},"max":{"1min":0.82999999999999996,"5min":4.077,"15min":12.468999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.59099999999999997,"15min":0.68000000000000005},"min":{"1min":0.44,"5min":0.39100000000000001,"15min":0.35399999999999998},"max":{"1min":0.78000000000000003,"5min":0.81899999999999995,"15min":10.837999999999999},"last":0.63800000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78500000000000003,"15min":0.999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.375},"max":{"1min":1.1910000000000001,"5min":1.9379999999999999,"15min":17.853999999999999},"last":0.878},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.85999999999999999,"15min":0.98199999999999998},"min":{"1min":0.54900000000000004,"5min":0.49199999999999999,"15min":0.45200000000000001},"max":{"1min":1.0569999999999999,"5min":4.2430000000000003,"15min":13.31},"last":0.93999999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.78400000000000003,"15min":0.98899999999999999},"min":{"1min":0.496,"5min":0.439,"15min":0.439},"max":{"1min":1.1619999999999999,"5min":1.179,"15min":17.687999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.77800000000000002,"15min":0.99099999999999999},"min":{"1min":0.58599999999999997,"5min":0.53800000000000003,"15min":0.438},"max":{"1min":1.048,"5min":1.825,"15min":20.515999999999998},"last":0.78000000000000003}]},{"osd":6,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.80900000000000005,"15min":0.98499999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.432},"max":{"1min":1.1419999999999999,"5min":4.2800000000000002,"15min":23.100999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.80600000000000005,"15min":0.94799999999999995},"min":{"1min":0.69899999999999995,"5min":0.52200000000000002,"15min":0.5},"max":{"1min":1.0720000000000001,"5min":1.149,"15min":12.76},"last":0.80100000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82499999999999996,"15min":1.1850000000000001},"min":{"1min":0.625,"5min":0.56899999999999995,"15min":0.46800000000000003},"max":{"1min":1.0860000000000001,"5min":1.931,"15min":24.321999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.82899999999999996,"15min":1.0820000000000001},"min":{"1min":0.60899999999999999,"5min":0.497,"15min":0.497},"max":{"1min":1.2070000000000001,"5min":4.202,"15min":23.013000000000002},"last":0.90000000000000002}]}]},{"osd":2,"up_from":15,"seq":64424509667,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760768,"kb_used_data":693560,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93611072,"statfs":{"total":96636764160,"available":95857737728,"internally_reserved":0,"allocated":710205440,"data_stored":709433352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45500000000000002,"15min":0.50900000000000001},"min":{"1min":0.32300000000000001,"5min":0.31,"15min":0.247},"max":{"1min":0.55200000000000005,"5min":0.80800000000000005,"15min":5.1849999999999996},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.501,"15min":0.54600000000000004},"min":{"1min":0.36499999999999999,"5min":0.26700000000000002,"15min":0.252},"max":{"1min":0.73999999999999999,"5min":0.83899999999999997,"15min":7.516},"last":0.51700000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.56799999999999995,"15min":0.60999999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.29699999999999999},"max":{"1min":0.71899999999999997,"5min":3.044,"15min":10.907},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.48999999999999999,"15min":0.59599999999999997},"min":{"1min":0.34999999999999998,"5min":0.33100000000000002,"15min":0.28100000000000003},"max":{"1min":0.76400000000000001,"5min":0.76500000000000001,"15min":8.9190000000000005},"last":0.36399999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52500000000000002,"15min":0.60599999999999998},"min":{"1min":0.40799999999999997,"5min":0.35399999999999998,"15min":0.30499999999999999},"max":{"1min":0.70999999999999996,"5min":0.86499999999999999,"15min":12.185},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61899999999999999,"15min":0.67100000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.373},"max":{"1min":0.80700000000000005,"5min":0.89600000000000002,"15min":7.6760000000000002},"last":0.495}]},{"osd":4,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.80300000000000005,"15min":0.879},"min":{"1min":0.59999999999999998,"5min":0.54700000000000004,"15min":0.44},"max":{"1min":1.0660000000000001,"5min":1.22,"15min":8.6609999999999996},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.76800000000000002,"15min":0.91200000000000003},"min":{"1min":0.56399999999999995,"5min":0.46200000000000002,"15min":0.41599999999999998},"max":{"1min":1.008,"5min":1.9670000000000001,"15min":19.146999999999998},"last":0.65000000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.76000000000000001,"15min":0.90200000000000002},"min":{"1min":0.58299999999999996,"5min":0.53600000000000003,"15min":0.41899999999999998},"max":{"1min":1.093,"5min":1.111,"15min":31.710000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.81999999999999995,"15min":0.999},"min":{"1min":0.57399999999999995,"5min":0.51700000000000002,"15min":0.48199999999999998},"max":{"1min":0.99199999999999999,"5min":1.8240000000000001,"15min":22.539000000000001},"last":0.60699999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.78300000000000003,"15min":0.91000000000000003},"min":{"1min":0.56499999999999995,"5min":0.52900000000000003,"15min":0.40899999999999997},"max":{"1min":1.079,"5min":1.3979999999999999,"15min":9.4540000000000006},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.80100000000000005,"15min":0.90800000000000003},"min":{"1min":0.64300000000000002,"5min":0.5,"15min":0.47899999999999998},"max":{"1min":1.0309999999999999,"5min":1.873,"15min":18.614999999999998},"last":0.58599999999999997}]},{"osd":7,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.79800000000000004,"15min":0.96799999999999997},"min":{"1min":0.63200000000000001,"5min":0.50900000000000001,"15min":0.44400000000000001},"max":{"1min":1.0109999999999999,"5min":1.7729999999999999,"15min":11.606},"last":0.746},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.84699999999999998,"15min":0.998},"min":{"1min":0.627,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.105,"5min":1.611,"15min":17.013000000000002},"last":0.73199999999999998}]}]},{"osd":1,"up_from":15,"seq":64424509669,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":941004,"kb_used_data":873796,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93430836,"statfs":{"total":96636764160,"available":95673176064,"internally_reserved":0,"allocated":894767104,"data_stored":893983032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45400000000000001,"15min":0.58499999999999996},"min":{"1min":0.26800000000000002,"5min":0.25900000000000001,"15min":0.23699999999999999},"max":{"1min":0.63500000000000001,"5min":0.88100000000000001,"15min":17.396999999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.51600000000000001,"15min":0.626},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.26900000000000002},"max":{"1min":0.73699999999999999,"5min":0.82999999999999996,"15min":12.378},"last":0.42999999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51700000000000002,"15min":0.57699999999999996},"min":{"1min":0.33000000000000002,"5min":0.313,"15min":0.309},"max":{"1min":0.61699999999999999,"5min":0.878,"15min":6.4569999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.54300000000000004,"15min":0.67400000000000004},"min":{"1min":0.40100000000000002,"5min":0.34499999999999997,"15min":0.33300000000000002},"max":{"1min":0.73899999999999999,"5min":0.81299999999999994,"15min":18.827000000000002},"last":0.47299999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59799999999999998,"15min":0.67200000000000004},"min":{"1min":0.433,"5min":0.40600000000000003,"15min":0.33700000000000002},"max":{"1min":0.872,"5min":1.0580000000000001,"15min":5.915},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.53800000000000003,"15min":0.64800000000000002},"min":{"1min":0.34399999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.77500000000000002,"5min":0.85499999999999998,"15min":9.9719999999999995},"last":0.49399999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.77700000000000002,"15min":0.89400000000000002},"min":{"1min":0.50900000000000001,"5min":0.437,"15min":0.437},"max":{"1min":1.056,"5min":1.1060000000000001,"15min":9.9329999999999998},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.746,"5min":0.79000000000000004,"15min":0.88300000000000001},"min":{"1min":0.55600000000000005,"5min":0.52800000000000002,"15min":0.45800000000000002},"max":{"1min":0.95599999999999996,"5min":1.165,"15min":9.8610000000000007},"last":0.66800000000000004}]},{"osd":5,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.80200000000000005,"15min":0.96299999999999997},"min":{"1min":0.55000000000000004,"5min":0.51200000000000001,"15min":0.46100000000000002},"max":{"1min":0.97499999999999998,"5min":1.2050000000000001,"15min":22.594999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.78200000000000003,"15min":1.0129999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.443},"max":{"1min":0.97499999999999998,"5min":1.077,"15min":26.748999999999999},"last":0.83899999999999997}]},{"osd":6,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.78800000000000003,"15min":1.034},"min":{"1min":0.57899999999999996,"5min":0.53100000000000003,"15min":0.42899999999999999},"max":{"1min":1.095,"5min":1.218,"15min":35.112000000000002},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.81100000000000005,"15min":0.90900000000000003},"min":{"1min":0.62,"5min":0.501,"15min":0.501},"max":{"1min":0.92900000000000005,"5min":1.2090000000000001,"15min":8.3670000000000009},"last":0.68799999999999994}]},{"osd":7,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.80800000000000005,"15min":1.024},"min":{"1min":0.56399999999999995,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.141,"5min":1.1950000000000001,"15min":20.872},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.81599999999999995,"15min":0.97799999999999998},"min":{"1min":0.61499999999999999,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.92300000000000004,"5min":1.139,"15min":9.8640000000000008},"last":0.61899999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542373,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":904260,"kb_used_data":828732,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93467580,"statfs":{"total":96636764160,"available":95710801920,"internally_reserved":0,"allocated":848621568,"data_stored":847844597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.72299999999999998,"15min":0.754},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.29899999999999999},"max":{"1min":1.038,"5min":2.6920000000000002,"15min":5.1639999999999997},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.75800000000000001,"15min":0.83899999999999997},"min":{"1min":0.46899999999999997,"5min":0.36799999999999999,"15min":0.34899999999999998},"max":{"1min":0.98199999999999998,"5min":6.4290000000000003,"15min":9.359},"last":0.70999999999999996}]},{"osd":1,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.76800000000000002,"15min":0.93000000000000005},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.375},"max":{"1min":0.89900000000000002,"5min":6.5069999999999997,"15min":21.120000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.747,"15min":0.93899999999999995},"min":{"1min":0.48299999999999998,"5min":0.39900000000000002,"15min":0.38600000000000001},"max":{"1min":0.96499999999999997,"5min":2.6619999999999999,"15min":15.843},"last":0.64900000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.76800000000000002,"15min":0.86699999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.34999999999999998},"max":{"1min":0.95099999999999996,"5min":6.3339999999999996,"15min":8.7750000000000004},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.79900000000000004,"15min":0.98299999999999998},"min":{"1min":0.501,"5min":0.501,"15min":0.39900000000000002},"max":{"1min":0.95399999999999996,"5min":6.4619999999999997,"15min":21.201000000000001},"last":0.501}]},{"osd":3,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.753,"15min":0.84399999999999997},"min":{"1min":0.498,"5min":0.46000000000000002,"15min":0.37},"max":{"1min":0.99199999999999999,"5min":2.6299999999999999,"15min":7.7199999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.81100000000000005,"15min":0.91900000000000004},"min":{"1min":0.52800000000000002,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.0109999999999999,"5min":6.3730000000000002,"15min":8.7300000000000004},"last":0.61899999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.66500000000000004,"15min":0.83399999999999996},"min":{"1min":0.52600000000000002,"5min":0.42999999999999999,"15min":0.30399999999999999},"max":{"1min":1.012,"5min":3.161,"15min":10.962},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.70699999999999996,"15min":1.012},"min":{"1min":0.48799999999999999,"5min":0.41999999999999998,"15min":0.35099999999999998},"max":{"1min":0.88100000000000001,"5min":6.5460000000000003,"15min":26.146999999999998},"last":0.59599999999999997}]},{"osd":6,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.66500000000000004,"15min":0.76700000000000002},"min":{"1min":0.50700000000000001,"5min":0.41299999999999998,"15min":0.36099999999999999},"max":{"1min":0.871,"5min":2.589,"15min":6.6699999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.82399999999999995,"15min":0.99199999999999999},"min":{"1min":0.53800000000000003,"5min":0.45300000000000001,"15min":0.379},"max":{"1min":0.95699999999999996,"5min":14.007999999999999,"15min":30.488},"last":0.67500000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.72499999999999998,"15min":1.2010000000000001},"min":{"1min":0.56699999999999995,"5min":0.48899999999999999,"15min":0.42599999999999999},"max":{"1min":0.90800000000000003,"5min":3.0870000000000002,"15min":33.499000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.91000000000000003,"15min":1.0680000000000001},"min":{"1min":0.57099999999999995,"5min":0.45700000000000002,"15min":0.42899999999999999},"max":{"1min":0.93000000000000005,"5min":21.178999999999998,"15min":25.437999999999999},"last":0.69199999999999995}]}]},{"osd":0,"up_from":14,"seq":60129542372,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1458240,"kb_used_data":1391032,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92913600,"statfs":{"total":96636764160,"available":95143526400,"internally_reserved":0,"allocated":1424416768,"data_stored":1423621757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.82999999999999996,"15min":0.749},"min":{"1min":0.33100000000000002,"5min":0.25800000000000001,"15min":0.245},"max":{"1min":0.59899999999999998,"5min":38.521999999999998,"15min":38.521999999999998},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.60099999999999998,"15min":0.68799999999999994},"min":{"1min":0.36699999999999999,"5min":0.32900000000000001,"15min":0.27000000000000002},"max":{"1min":0.753,"5min":8.0749999999999993,"15min":10.154},"last":0.34699999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.57699999999999996,"15min":0.80200000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.30399999999999999},"max":{"1min":0.78600000000000003,"5min":2.6909999999999998,"15min":25.905000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.67200000000000004,"15min":0.66900000000000004},"min":{"1min":0.307,"5min":0.307,"15min":0.27800000000000002},"max":{"1min":0.69199999999999995,"5min":18.247,"15min":18.247},"last":0.45300000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.76400000000000001,"15min":0.80300000000000005},"min":{"1min":0.32400000000000001,"5min":0.32000000000000001,"15min":0.29199999999999998},"max":{"1min":0.70999999999999996,"5min":21.451000000000001,"15min":21.451000000000001},"last":0.629},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61799999999999999,"15min":0.76600000000000001},"min":{"1min":0.47099999999999997,"5min":0.38900000000000001,"15min":0.36899999999999999},"max":{"1min":0.80100000000000005,"5min":1.7,"15min":10.210000000000001},"last":0.55600000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.79400000000000004,"15min":0.99199999999999999},"min":{"1min":0.52600000000000002,"5min":0.441,"15min":0.39100000000000001},"max":{"1min":1.1639999999999999,"5min":2.7850000000000001,"15min":22.814},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.80100000000000005,"5min":1.0389999999999999,"15min":1.105},"min":{"1min":0.64000000000000001,"5min":0.45500000000000002,"15min":0.42399999999999999},"max":{"1min":1.1379999999999999,"5min":27.027999999999999,"15min":27.027999999999999},"last":0.41299999999999998}]},{"osd":5,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.83199999999999996,"15min":0.98799999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.38400000000000001},"max":{"1min":1.218,"5min":6.6429999999999998,"15min":12.428000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.77700000000000002,"5min":1.0509999999999999,"15min":1.333},"min":{"1min":0.55400000000000005,"5min":0.47999999999999998,"15min":0.46100000000000002},"max":{"1min":1.0600000000000001,"5min":27.210999999999999,"15min":27.210999999999999},"last":0.66400000000000003}]},{"osd":6,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.86699999999999999,"15min":1.242},"min":{"1min":0.66400000000000003,"5min":0.50900000000000001,"15min":0.46700000000000003},"max":{"1min":0.98399999999999999,"5min":9.3140000000000001,"15min":31.059000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.82699999999999996,"15min":1.089},"min":{"1min":0.58899999999999997,"5min":0.52900000000000003,"15min":0.51500000000000001},"max":{"1min":1.024,"5min":2.8340000000000001,"15min":40.731000000000002},"last":0.68200000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.86299999999999999,"15min":1.2150000000000001},"min":{"1min":0.66500000000000004,"5min":0.52200000000000002,"15min":0.499},"max":{"1min":1.1539999999999999,"5min":5.2089999999999996,"15min":24.492999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.81599999999999995,"5min":1.097,"15min":1.1859999999999999},"min":{"1min":0.63500000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1200000000000001,"5min":27.081,"15min":27.081},"last":0.70099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":585728,"data_stored":567429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":1321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":598016,"data_stored":583072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1421877248,"data_stored":1421869497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":905977856,"data_stored":905970036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":708841472,"data_stored":708837568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1111498752,"data_stored":1111491082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1354764288,"data_stored":1354760453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":847253504,"data_stored":847249530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":981471232,"data_stored":981467258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1430257664,"data_stored":1430257664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:23:35.616 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:23:35.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.801+0000 7f4240800640 1 Processor -- start 2024-01-15T18:23:35.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.801+0000 7f4240800640 1 -- start start 2024-01-15T18:23:35.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.805+0000 7f4240800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f42381608f0 0x7f423806a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:35.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.805+0000 7f4240800640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f42380cf0a0 0x7f42380cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:35.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.805+0000 7f4240800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f423806aa30 con 0x7f42380d09a0 2024-01-15T18:23:35.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.805+0000 7f4240800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f423806d040 con 0x7f42381608f0 2024-01-15T18:23:35.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.805+0000 7f4240800640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42380cab70 con 0x7f42380cf0a0 2024-01-15T18:23:35.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.805+0000 7f423de00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f42381608f0 0x7f423806a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:35.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.805+0000 7f423d400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f42380d09a0 0x7f42380ca460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42880/0 (socket says 172.21.15.27:42880) 2024-01-15T18:23:35.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.805+0000 7f423d400640 1 -- 172.21.15.27:0/2976615214 learned_addr learned my addr 172.21.15.27:0/2976615214 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:35.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.805+0000 7f423e800640 1 --2- 172.21.15.27:0/2976615214 >> v2:172.21.15.93:3300/0 conn(0x7f42380cf0a0 0x7f42380cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:35.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.805+0000 7f423de00640 1 -- 172.21.15.27:0/2976615214 >> v2:172.21.15.93:3300/0 conn(0x7f42380cf0a0 msgr2=0x7f42380cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:35.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.805+0000 7f423de00640 1 --2- 172.21.15.27:0/2976615214 >> v2:172.21.15.93:3300/0 conn(0x7f42380cf0a0 0x7f42380cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.805+0000 7f423de00640 1 -- 172.21.15.27:0/2976615214 >> v1:172.21.15.27:6789/0 conn(0x7f42380d09a0 legacy=0x7f42380ca460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:35.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.805+0000 7f423de00640 1 -- 172.21.15.27:0/2976615214 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423806acd0 con 0x7f42381608f0 2024-01-15T18:23:35.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.809+0000 7f423de00640 1 --2- 172.21.15.27:0/2976615214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42381608f0 0x7f423806a4f0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f42380cc5a0 tx=0x7f422801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc42f9caf132881a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:35.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.809+0000 7f423ca00640 1 -- 172.21.15.27:0/2976615214 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42280ab600 con 0x7f42381608f0 2024-01-15T18:23:35.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.809+0000 7f423ca00640 1 -- 172.21.15.27:0/2976615214 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42280a7050 con 0x7f42381608f0 2024-01-15T18:23:35.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.809+0000 7f423ca00640 1 -- 172.21.15.27:0/2976615214 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42280ab980 con 0x7f42381608f0 2024-01-15T18:23:35.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.809+0000 7f4240800640 1 -- 172.21.15.27:0/2976615214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42381608f0 msgr2=0x7f423806a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:35.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.809+0000 7f4240800640 1 --2- 172.21.15.27:0/2976615214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42381608f0 0x7f423806a4f0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f42380cc5a0 tx=0x7f422801c460 comp rx=0 tx=0).stop 2024-01-15T18:23:35.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.809+0000 7f4240800640 1 -- 172.21.15.27:0/2976615214 shutdown_connections 2024-01-15T18:23:35.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.809+0000 7f4240800640 1 --2- 172.21.15.27:0/2976615214 >> v2:172.21.15.93:3300/0 conn(0x7f42380cf0a0 0x7f42380cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.809+0000 7f4240800640 1 --2- 172.21.15.27:0/2976615214 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42381608f0 0x7f423806a4f0 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.809+0000 7f4240800640 1 -- 172.21.15.27:0/2976615214 >> 172.21.15.27:0/2976615214 conn(0x7f42380cfe10 msgr2=0x7f42380d0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:35.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.809+0000 7f4240800640 1 -- 172.21.15.27:0/2976615214 shutdown_connections 2024-01-15T18:23:35.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.809+0000 7f4240800640 1 -- 172.21.15.27:0/2976615214 wait complete. 2024-01-15T18:23:35.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.809+0000 7f4240800640 1 Processor -- start 2024-01-15T18:23:35.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.809+0000 7f4240800640 1 -- start start 2024-01-15T18:23:35.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.813+0000 7f4240800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42380cf0a0 0x7f42380af860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:35.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.813+0000 7f4240800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42380d09a0 0x7f42380afe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:35.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.813+0000 7f4240800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f42381608f0 0x7f4238150170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:35.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.813+0000 7f4240800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42380b0380 con 0x7f42380d09a0 2024-01-15T18:23:35.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.813+0000 7f4240800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f423815a7e0 con 0x7f42380cf0a0 2024-01-15T18:23:35.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.813+0000 7f4240800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f423815aac0 con 0x7f42381608f0 2024-01-15T18:23:35.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.813+0000 7f423d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42380d09a0 0x7f42380afe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:35.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.813+0000 7f423d400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42380d09a0 0x7f42380afe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47516/0 (socket says 172.21.15.27:47516) 2024-01-15T18:23:35.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.813+0000 7f423d400640 1 -- 172.21.15.27:0/1150773201 learned_addr learned my addr 172.21.15.27:0/1150773201 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:35.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.813+0000 7f423e800640 1 --2- 172.21.15.27:0/1150773201 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f42381608f0 0x7f4238150170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:35.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.813+0000 7f423de00640 1 --2- 172.21.15.27:0/1150773201 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42380cf0a0 0x7f42380af860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:35.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.813+0000 7f423d400640 1 -- 172.21.15.27:0/1150773201 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f42381608f0 msgr2=0x7f4238150170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:35.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.813+0000 7f423d400640 1 --2- 172.21.15.27:0/1150773201 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f42381608f0 0x7f4238150170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.813+0000 7f423d400640 1 -- 172.21.15.27:0/1150773201 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42380cf0a0 msgr2=0x7f42380af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:35.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.813+0000 7f423d400640 1 --2- 172.21.15.27:0/1150773201 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42380cf0a0 0x7f42380af860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.813+0000 7f423d400640 1 -- 172.21.15.27:0/1150773201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4228009420 con 0x7f42380d09a0 2024-01-15T18:23:35.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.813+0000 7f423d400640 1 --2- 172.21.15.27:0/1150773201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42380d09a0 0x7f42380afe40 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f422000c400 tx=0x7f422000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:35.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.813+0000 7f4236000640 1 -- 172.21.15.27:0/1150773201 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4220005cf0 con 0x7f42380d09a0 2024-01-15T18:23:35.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.813+0000 7f4236000640 1 -- 172.21.15.27:0/1150773201 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4220005e50 con 0x7f42380d09a0 2024-01-15T18:23:35.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.813+0000 7f4240800640 1 -- 172.21.15.27:0/1150773201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f423815ada0 con 0x7f42380d09a0 2024-01-15T18:23:35.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.813+0000 7f4236000640 1 -- 172.21.15.27:0/1150773201 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42200072d0 con 0x7f42380d09a0 2024-01-15T18:23:35.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.813+0000 7f4240800640 1 -- 172.21.15.27:0/1150773201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4238150880 con 0x7f42380d09a0 2024-01-15T18:23:35.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.813+0000 7f4240800640 1 -- 172.21.15.27:0/1150773201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f423804e850 con 0x7f42380d09a0 2024-01-15T18:23:35.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.817+0000 7f4236000640 1 -- 172.21.15.27:0/1150773201 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f4220007430 con 0x7f42380d09a0 2024-01-15T18:23:35.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.817+0000 7f4236000640 1 --2- 172.21.15.27:0/1150773201 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4210067a90 0x7f4210069f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:35.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.817+0000 7f4236000640 1 -- 172.21.15.27:0/1150773201 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f42200826e0 con 0x7f42380d09a0 2024-01-15T18:23:35.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.817+0000 7f423de00640 1 --2- 172.21.15.27:0/1150773201 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4210067a90 0x7f4210069f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:35.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.817+0000 7f423de00640 1 --2- 172.21.15.27:0/1150773201 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4210067a90 0x7f4210069f50 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f42280ad000 tx=0x7f42280a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:35.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.821+0000 7f4236000640 1 -- 172.21.15.27:0/1150773201 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4220051050 con 0x7f42380d09a0 2024-01-15T18:23:35.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.981+0000 7f4240800640 1 -- 172.21.15.27:0/1150773201 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4238150880 con 0x7f42380d09a0 2024-01-15T18:23:35.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.981+0000 7f4236000640 1 -- 172.21.15.27:0/1150773201 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16761 (secure 0 0 0) 0x7f4238150880 con 0x7f42380d09a0 2024-01-15T18:23:35.977 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:23:35.977 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":110,"fsid":"fce09647-6947-4a73-bd3d-e3d154552087","created":"2024-01-15T18:04:33.010381+0000","modified":"2024-01-15T18:18:42.250892+0000","last_up_change":"2024-01-15T18:04:44.444228+0000","last_in_change":"2024-01-15T18:04:35.557795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T18:04:44.196146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:18:30.993882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-15T18:18:31.582649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ad1c16d0-3fd6-4ca8-8d93-46c40745b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6804","nonce":464948094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6807","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6809","nonce":464948094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6815","nonce":464948094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":464948094},{"type":"v1","addr":"172.21.15.27:6813","nonce":464948094}]},"public_addr":"172.21.15.27:6804/464948094","cluster_addr":"172.21.15.27:6809/464948094","heartbeat_back_addr":"172.21.15.27:6815/464948094","heartbeat_front_addr":"172.21.15.27:6813/464948094","state":["exists","up"]},{"osd":1,"uuid":"619e3be6-1fe3-4d1a-a7ad-a613949aceb7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6801","nonce":3601044148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6803","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6805","nonce":3601044148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6811","nonce":3601044148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":3601044148},{"type":"v1","addr":"172.21.15.27:6808","nonce":3601044148}]},"public_addr":"172.21.15.27:6801/3601044148","cluster_addr":"172.21.15.27:6805/3601044148","heartbeat_back_addr":"172.21.15.27:6811/3601044148","heartbeat_front_addr":"172.21.15.27:6808/3601044148","state":["exists","up"]},{"osd":2,"uuid":"2996e145-a7bb-4e3d-9159-da33ce9f4851","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6825","nonce":366025692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6827","nonce":366025692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6831","nonce":366025692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":366025692},{"type":"v1","addr":"172.21.15.27:6829","nonce":366025692}]},"public_addr":"172.21.15.27:6825/366025692","cluster_addr":"172.21.15.27:6827/366025692","heartbeat_back_addr":"172.21.15.27:6831/366025692","heartbeat_front_addr":"172.21.15.27:6829/366025692","state":["exists","up"]},{"osd":3,"uuid":"d3b5a604-2395-4cef-b9ec-bba975a26f40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6817","nonce":934967897}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6819","nonce":934967897}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6823","nonce":934967897}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":934967897},{"type":"v1","addr":"172.21.15.27:6821","nonce":934967897}]},"public_addr":"172.21.15.27:6817/934967897","cluster_addr":"172.21.15.27:6819/934967897","heartbeat_back_addr":"172.21.15.27:6823/934967897","heartbeat_front_addr":"172.21.15.27:6821/934967897","state":["exists","up"]},{"osd":4,"uuid":"04c19746-6de6-45b1-aaa3-3d8e96469592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6803","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6806","nonce":1254182834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6808","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6811","nonce":1254182834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6819","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6821","nonce":1254182834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6814","nonce":1254182834},{"type":"v1","addr":"172.21.15.93:6817","nonce":1254182834}]},"public_addr":"172.21.15.93:6806/1254182834","cluster_addr":"172.21.15.93:6811/1254182834","heartbeat_back_addr":"172.21.15.93:6821/1254182834","heartbeat_front_addr":"172.21.15.93:6817/1254182834","state":["exists","up"]},{"osd":5,"uuid":"82211717-bb0b-4c7e-9e48-bdc0dd39bd97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6800","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6801","nonce":2348354791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6802","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6804","nonce":2348354791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6812","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6815","nonce":2348354791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6807","nonce":2348354791},{"type":"v1","addr":"172.21.15.93:6810","nonce":2348354791}]},"public_addr":"172.21.15.93:6801/2348354791","cluster_addr":"172.21.15.93:6804/2348354791","heartbeat_back_addr":"172.21.15.93:6815/2348354791","heartbeat_front_addr":"172.21.15.93:6810/2348354791","state":["exists","up"]},{"osd":6,"uuid":"1fabf1d4-0289-42f3-88e4-90fede3a06c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6805","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6809","nonce":838236833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6813","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6816","nonce":838236833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6822","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6823","nonce":838236833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6818","nonce":838236833},{"type":"v1","addr":"172.21.15.93:6820","nonce":838236833}]},"public_addr":"172.21.15.93:6809/838236833","cluster_addr":"172.21.15.93:6816/838236833","heartbeat_back_addr":"172.21.15.93:6823/838236833","heartbeat_front_addr":"172.21.15.93:6820/838236833","state":["exists","up"]},{"osd":7,"uuid":"e3626db0-8611-4ead-86f0-1b9ae0163da4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6824","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6825","nonce":978428062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6826","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6827","nonce":978428062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6830","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6831","nonce":978428062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6828","nonce":978428062},{"type":"v1","addr":"172.21.15.93:6829","nonce":978428062}]},"public_addr":"172.21.15.93:6825/978428062","cluster_addr":"172.21.15.93:6827/978428062","heartbeat_back_addr":"172.21.15.93:6831/978428062","heartbeat_front_addr":"172.21.15.93:6829/978428062","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.110445+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.188611+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.299298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.265015+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.263779+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.101589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.118260+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T18:04:41.235286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.5","mappings":[{"from":0,"to":5}]},{"pgid":"17.18","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":11,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:23:35.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.981+0000 7f4240800640 1 -- 172.21.15.27:0/1150773201 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4210067a90 msgr2=0x7f4210069f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:35.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.981+0000 7f4240800640 1 --2- 172.21.15.27:0/1150773201 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4210067a90 0x7f4210069f50 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f42280ad000 tx=0x7f42280a6000 comp rx=0 tx=0).stop 2024-01-15T18:23:35.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.981+0000 7f4240800640 1 -- 172.21.15.27:0/1150773201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42380d09a0 msgr2=0x7f42380afe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:35.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.981+0000 7f4240800640 1 --2- 172.21.15.27:0/1150773201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42380d09a0 0x7f42380afe40 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f422000c400 tx=0x7f422000c8c0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.981+0000 7f4240800640 1 -- 172.21.15.27:0/1150773201 shutdown_connections 2024-01-15T18:23:35.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.981+0000 7f4240800640 1 --2- 172.21.15.27:0/1150773201 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f4210067a90 0x7f4210069f50 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.981+0000 7f4240800640 1 --2- 172.21.15.27:0/1150773201 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f42381608f0 0x7f4238150170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.981+0000 7f4240800640 1 --2- 172.21.15.27:0/1150773201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42380d09a0 0x7f42380afe40 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.981+0000 7f4240800640 1 --2- 172.21.15.27:0/1150773201 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42380cf0a0 0x7f42380af860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:35.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.981+0000 7f4240800640 1 -- 172.21.15.27:0/1150773201 >> 172.21.15.27:0/1150773201 conn(0x7f42380cfe10 msgr2=0x7f42380cbf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:35.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.981+0000 7f4240800640 1 -- 172.21.15.27:0/1150773201 shutdown_connections 2024-01-15T18:23:35.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:35.981+0000 7f4240800640 1 -- 172.21.15.27:0/1150773201 wait complete. 2024-01-15T18:23:37.000 INFO:tasks.ceph:Scrubbing osd.0 2024-01-15T18:23:37.000 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-01-15T18:23:37.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.157+0000 7f8537800640 1 Processor -- start 2024-01-15T18:23:37.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.157+0000 7f8537800640 1 -- start start 2024-01-15T18:23:37.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.161+0000 7f8537800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f85300cd7d0 0x7f853015a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.161+0000 7f8537800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85300ce680 0x7f85300b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.161+0000 7f8537800640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8530072a40 con 0x7f85300cd3f0 2024-01-15T18:23:37.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.161+0000 7f8537800640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f853006f030 con 0x7f85300cd7d0 2024-01-15T18:23:37.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.161+0000 7f8537800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85300b43f0 con 0x7f85300ce680 2024-01-15T18:23:37.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.161+0000 7f8534e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f85300cd3f0 0x7f853006d9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42884/0 (socket says 172.21.15.27:42884) 2024-01-15T18:23:37.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.161+0000 7f852fe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f85300cd7d0 0x7f853015a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.161+0000 7f8534e00640 1 -- 172.21.15.27:0/1867180336 learned_addr learned my addr 172.21.15.27:0/1867180336 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:37.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.161+0000 7f8536200640 1 --2- 172.21.15.27:0/1867180336 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85300ce680 0x7f85300b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.161+0000 7f852fe00640 1 -- 172.21.15.27:0/1867180336 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85300ce680 msgr2=0x7f85300b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:37.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.161+0000 7f852fe00640 1 --2- 172.21.15.27:0/1867180336 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85300ce680 0x7f85300b4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.161+0000 7f852fe00640 1 -- 172.21.15.27:0/1867180336 >> v1:172.21.15.27:6789/0 conn(0x7f85300cd3f0 legacy=0x7f853006d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:37.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.161+0000 7f852fe00640 1 -- 172.21.15.27:0/1867180336 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85300b46d0 con 0x7f85300cd7d0 2024-01-15T18:23:37.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.165+0000 7f852fe00640 1 --2- 172.21.15.27:0/1867180336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85300cd7d0 0x7f853015a4e0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f85180097c0 tx=0x7f851801ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef67016ee987ca6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:37.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.165+0000 7f852f400640 1 -- 172.21.15.27:0/1867180336 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85180ab670 con 0x7f85300cd7d0 2024-01-15T18:23:37.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.165+0000 7f852f400640 1 -- 172.21.15.27:0/1867180336 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85180ab7d0 con 0x7f85300cd7d0 2024-01-15T18:23:37.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.165+0000 7f852f400640 1 -- 172.21.15.27:0/1867180336 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85180abb60 con 0x7f85300cd7d0 2024-01-15T18:23:37.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.165+0000 7f8537800640 1 -- 172.21.15.27:0/1867180336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85300cd7d0 msgr2=0x7f853015a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:37.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.165+0000 7f8537800640 1 --2- 172.21.15.27:0/1867180336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85300cd7d0 0x7f853015a4e0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f85180097c0 tx=0x7f851801ceb0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.165+0000 7f8537800640 1 -- 172.21.15.27:0/1867180336 shutdown_connections 2024-01-15T18:23:37.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.165+0000 7f8537800640 1 --2- 172.21.15.27:0/1867180336 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85300ce680 0x7f85300b4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.165+0000 7f8537800640 1 --2- 172.21.15.27:0/1867180336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85300cd7d0 0x7f853015a4e0 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.165+0000 7f8537800640 1 -- 172.21.15.27:0/1867180336 >> 172.21.15.27:0/1867180336 conn(0x7f8530099e10 msgr2=0x7f85300b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:37.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.165+0000 7f8537800640 1 -- 172.21.15.27:0/1867180336 shutdown_connections 2024-01-15T18:23:37.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.165+0000 7f8537800640 1 -- 172.21.15.27:0/1867180336 wait complete. 2024-01-15T18:23:37.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.169+0000 7f8537800640 1 Processor -- start 2024-01-15T18:23:37.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.169+0000 7f8537800640 1 -- start start 2024-01-15T18:23:37.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.169+0000 7f8537800640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85300cd3f0 0x7f8530145160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.169+0000 7f8537800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f85300cd7d0 0x7f853006b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.169+0000 7f8537800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85300ce680 0x7f853006ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.169+0000 7f8537800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85300cf0b0 con 0x7f85300ce680 2024-01-15T18:23:37.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.169+0000 7f8537800640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85300cf390 con 0x7f85300cd7d0 2024-01-15T18:23:37.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.169+0000 7f8537800640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85300cf670 con 0x7f85300cd3f0 2024-01-15T18:23:37.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.169+0000 7f8536200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85300ce680 0x7f853006ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.169+0000 7f8536200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85300ce680 0x7f853006ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47528/0 (socket says 172.21.15.27:47528) 2024-01-15T18:23:37.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.169+0000 7f8536200640 1 -- 172.21.15.27:0/3915844483 learned_addr learned my addr 172.21.15.27:0/3915844483 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:37.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.169+0000 7f852fe00640 1 --2- 172.21.15.27:0/3915844483 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f85300cd7d0 0x7f853006b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.169+0000 7f8534e00640 1 --2- 172.21.15.27:0/3915844483 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85300cd3f0 0x7f8530145160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.169+0000 7f8536200640 1 -- 172.21.15.27:0/3915844483 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85300cd3f0 msgr2=0x7f8530145160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:37.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.169+0000 7f8536200640 1 --2- 172.21.15.27:0/3915844483 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f85300cd3f0 0x7f8530145160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.169+0000 7f8536200640 1 -- 172.21.15.27:0/3915844483 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f85300cd7d0 msgr2=0x7f853006b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:37.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.169+0000 7f8536200640 1 --2- 172.21.15.27:0/3915844483 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f85300cd7d0 0x7f853006b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.169+0000 7f8536200640 1 -- 172.21.15.27:0/3915844483 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8518009420 con 0x7f85300ce680 2024-01-15T18:23:37.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.169+0000 7f8536200640 1 --2- 172.21.15.27:0/3915844483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85300ce680 0x7f853006ba80 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f852800efc0 tx=0x7f852800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:37.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.169+0000 7f852cc00640 1 -- 172.21.15.27:0/3915844483 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8528014480 con 0x7f85300ce680 2024-01-15T18:23:37.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.169+0000 7f852cc00640 1 -- 172.21.15.27:0/3915844483 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852800f040 con 0x7f85300ce680 2024-01-15T18:23:37.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.169+0000 7f852cc00640 1 -- 172.21.15.27:0/3915844483 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8528014820 con 0x7f85300ce680 2024-01-15T18:23:37.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.169+0000 7f8537800640 1 -- 172.21.15.27:0/3915844483 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85300b59a0 con 0x7f85300ce680 2024-01-15T18:23:37.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.169+0000 7f8537800640 1 -- 172.21.15.27:0/3915844483 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85300cfc30 con 0x7f85300ce680 2024-01-15T18:23:37.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.169+0000 7f8537800640 1 -- 172.21.15.27:0/3915844483 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f85300b5820 con 0x7f85300ce680 2024-01-15T18:23:37.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.173+0000 7f852cc00640 1 -- 172.21.15.27:0/3915844483 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f8528014980 con 0x7f85300ce680 2024-01-15T18:23:37.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.173+0000 7f852cc00640 1 --2- 172.21.15.27:0/3915844483 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8508067a60 0x7f8508069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.173+0000 7f852cc00640 1 -- 172.21.15.27:0/3915844483 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f8528083d90 con 0x7f85300ce680 2024-01-15T18:23:37.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.173+0000 7f852cc00640 1 --2- 172.21.15.27:0/3915844483 >> [v2:172.21.15.27:6802/464948094,v1:172.21.15.27:6804/464948094] conn(0x7f850806fe60 0x7f8508072300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.173+0000 7f852cc00640 1 -- 172.21.15.27:0/3915844483 --> [v2:172.21.15.27:6802/464948094,v1:172.21.15.27:6804/464948094] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f85080729d0 con 0x7f850806fe60 2024-01-15T18:23:37.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.173+0000 7f852cc00640 1 -- 172.21.15.27:0/3915844483 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7f85280846b0 con 0x7f85300ce680 2024-01-15T18:23:37.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.173+0000 7f8534e00640 1 --2- 172.21.15.27:0/3915844483 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8508067a60 0x7f8508069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.173+0000 7f852fe00640 1 --2- 172.21.15.27:0/3915844483 >> [v2:172.21.15.27:6802/464948094,v1:172.21.15.27:6804/464948094] conn(0x7f850806fe60 0x7f8508072300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.173+0000 7f852fe00640 1 --2- 172.21.15.27:0/3915844483 >> [v2:172.21.15.27:6802/464948094,v1:172.21.15.27:6804/464948094] conn(0x7f850806fe60 0x7f8508072300 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:37.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.173+0000 7f8534e00640 1 --2- 172.21.15.27:0/3915844483 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8508067a60 0x7f8508069f20 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f8520005890 tx=0x7f8520009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:37.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.177+0000 7f852cc00640 1 -- 172.21.15.27:0/3915844483 <== osd.0 v2:172.21.15.27:6802/464948094 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f85080729d0 con 0x7f850806fe60 2024-01-15T18:23:37.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.189+0000 7f8537800640 1 -- 172.21.15.27:0/3915844483 --> [v2:172.21.15.27:6802/464948094,v1:172.21.15.27:6804/464948094] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f853006ec50 con 0x7f850806fe60 2024-01-15T18:23:37.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.189+0000 7f852cc00640 1 -- 172.21.15.27:0/3915844483 <== osd.0 v2:172.21.15.27:6802/464948094 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f85180ab940 con 0x7f850806fe60 2024-01-15T18:23:37.185 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:23:37.185 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-15T18:23:37.186 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:23:37.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.189+0000 7f8537800640 1 -- 172.21.15.27:0/3915844483 >> [v2:172.21.15.27:6802/464948094,v1:172.21.15.27:6804/464948094] conn(0x7f850806fe60 msgr2=0x7f8508072300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:37.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.189+0000 7f8537800640 1 --2- 172.21.15.27:0/3915844483 >> [v2:172.21.15.27:6802/464948094,v1:172.21.15.27:6804/464948094] conn(0x7f850806fe60 0x7f8508072300 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.189+0000 7f8537800640 1 -- 172.21.15.27:0/3915844483 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8508067a60 msgr2=0x7f8508069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:37.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.189+0000 7f8537800640 1 --2- 172.21.15.27:0/3915844483 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8508067a60 0x7f8508069f20 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f8520005890 tx=0x7f8520009040 comp rx=0 tx=0).stop 2024-01-15T18:23:37.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.189+0000 7f8537800640 1 -- 172.21.15.27:0/3915844483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85300ce680 msgr2=0x7f853006ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:37.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.189+0000 7f8537800640 1 --2- 172.21.15.27:0/3915844483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f85300ce680 0x7f853006ba80 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f852800efc0 tx=0x7f852800c490 comp rx=0 tx=0).stop 2024-01-15T18:23:37.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.189+0000 7f8536200640 1 -- 172.21.15.27:0/3915844483 reap_dead start 2024-01-15T18:23:37.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.189+0000 7f8537800640 1 -- 172.21.15.27:0/3915844483 shutdown_connections 2024-01-15T18:23:37.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.189+0000 7f8537800640 1 -- 172.21.15.27:0/3915844483 >> 172.21.15.27:0/3915844483 conn(0x7f8530099e10 msgr2=0x7f85300b3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:37.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.189+0000 7f8537800640 1 -- 172.21.15.27:0/3915844483 shutdown_connections 2024-01-15T18:23:37.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.189+0000 7f8537800640 1 -- 172.21.15.27:0/3915844483 wait complete. 2024-01-15T18:23:37.205 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-01-15T18:23:37.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.353+0000 7fca53e00640 1 Processor -- start 2024-01-15T18:23:37.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.353+0000 7fca53e00640 1 -- start start 2024-01-15T18:23:37.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.353+0000 7fca53e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fca4c15d2c0 0x7fca4c06d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.353+0000 7fca53e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fca4c071d40 0x7fca4c0b49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.353+0000 7fca53e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fca4c0b4f20 0x7fca4c06a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.353+0000 7fca53e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca4c0cfbd0 con 0x7fca4c071d40 2024-01-15T18:23:37.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.353+0000 7fca53e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca4c0aec90 con 0x7fca4c0b4f20 2024-01-15T18:23:37.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.353+0000 7fca53e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca4c0aef70 con 0x7fca4c15d2c0 2024-01-15T18:23:37.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.353+0000 7fca50a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fca4c071d40 0x7fca4c0b49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.353+0000 7fca50a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fca4c071d40 0x7fca4c0b49e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47536/0 (socket says 172.21.15.27:47536) 2024-01-15T18:23:37.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.353+0000 7fca50a00640 1 -- 172.21.15.27:0/2589710133 learned_addr learned my addr 172.21.15.27:0/2589710133 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:37.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.353+0000 7fca51e00640 1 --2- 172.21.15.27:0/2589710133 >> v2:172.21.15.93:3300/0 conn(0x7fca4c0b4f20 0x7fca4c06a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.353+0000 7fca51400640 1 --2- 172.21.15.27:0/2589710133 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fca4c15d2c0 0x7fca4c06d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.353+0000 7fca50a00640 1 -- 172.21.15.27:0/2589710133 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fca4c15d2c0 msgr2=0x7fca4c06d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:37.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.353+0000 7fca50a00640 1 --2- 172.21.15.27:0/2589710133 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fca4c15d2c0 0x7fca4c06d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.353+0000 7fca50a00640 1 -- 172.21.15.27:0/2589710133 >> v2:172.21.15.93:3300/0 conn(0x7fca4c0b4f20 msgr2=0x7fca4c06a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:37.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.353+0000 7fca50a00640 1 --2- 172.21.15.27:0/2589710133 >> v2:172.21.15.93:3300/0 conn(0x7fca4c0b4f20 0x7fca4c06a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.353+0000 7fca50a00640 1 -- 172.21.15.27:0/2589710133 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca4c0af250 con 0x7fca4c071d40 2024-01-15T18:23:37.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.357+0000 7fca50a00640 1 --2- 172.21.15.27:0/2589710133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca4c071d40 0x7fca4c0b49e0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fca4c09e080 tx=0x7fca3401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe55c833ea42520d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:37.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.357+0000 7fca4be00640 1 -- 172.21.15.27:0/2589710133 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca340ac630 con 0x7fca4c071d40 2024-01-15T18:23:37.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.357+0000 7fca4be00640 1 -- 172.21.15.27:0/2589710133 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca340ac790 con 0x7fca4c071d40 2024-01-15T18:23:37.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.357+0000 7fca4be00640 1 -- 172.21.15.27:0/2589710133 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca340acb20 con 0x7fca4c071d40 2024-01-15T18:23:37.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.357+0000 7fca53e00640 1 -- 172.21.15.27:0/2589710133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca4c071d40 msgr2=0x7fca4c0b49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:37.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.357+0000 7fca53e00640 1 --2- 172.21.15.27:0/2589710133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca4c071d40 0x7fca4c0b49e0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fca4c09e080 tx=0x7fca3401c500 comp rx=0 tx=0).stop 2024-01-15T18:23:37.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.357+0000 7fca53e00640 1 -- 172.21.15.27:0/2589710133 shutdown_connections 2024-01-15T18:23:37.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.357+0000 7fca53e00640 1 --2- 172.21.15.27:0/2589710133 >> v2:172.21.15.93:3300/0 conn(0x7fca4c0b4f20 0x7fca4c06a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.357+0000 7fca53e00640 1 --2- 172.21.15.27:0/2589710133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca4c071d40 0x7fca4c0b49e0 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.357+0000 7fca53e00640 1 --2- 172.21.15.27:0/2589710133 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fca4c15d2c0 0x7fca4c06d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.357+0000 7fca53e00640 1 -- 172.21.15.27:0/2589710133 >> 172.21.15.27:0/2589710133 conn(0x7fca4c06b600 msgr2=0x7fca4c06ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:37.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.357+0000 7fca53e00640 1 -- 172.21.15.27:0/2589710133 shutdown_connections 2024-01-15T18:23:37.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.357+0000 7fca53e00640 1 -- 172.21.15.27:0/2589710133 wait complete. 2024-01-15T18:23:37.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.361+0000 7fca53e00640 1 Processor -- start 2024-01-15T18:23:37.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.361+0000 7fca53e00640 1 -- start start 2024-01-15T18:23:37.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.361+0000 7fca53e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca4c071d40 0x7fca4c150130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.361+0000 7fca53e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fca4c0b4f20 0x7fca4c14ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.361+0000 7fca53e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fca4c15d2c0 0x7fca4c1587a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.361+0000 7fca53e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca4c160130 con 0x7fca4c071d40 2024-01-15T18:23:37.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.361+0000 7fca53e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca4c1568a0 con 0x7fca4c15d2c0 2024-01-15T18:23:37.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.361+0000 7fca53e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca4c15fb70 con 0x7fca4c0b4f20 2024-01-15T18:23:37.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.361+0000 7fca51400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca4c071d40 0x7fca4c150130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.361+0000 7fca51400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca4c071d40 0x7fca4c150130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47538/0 (socket says 172.21.15.27:47538) 2024-01-15T18:23:37.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.361+0000 7fca51400640 1 -- 172.21.15.27:0/269614023 learned_addr learned my addr 172.21.15.27:0/269614023 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:37.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.361+0000 7fca51e00640 1 --2- 172.21.15.27:0/269614023 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fca4c15d2c0 0x7fca4c1587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.361+0000 7fca50a00640 1 --2- 172.21.15.27:0/269614023 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fca4c0b4f20 0x7fca4c14ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.361+0000 7fca51400640 1 -- 172.21.15.27:0/269614023 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fca4c0b4f20 msgr2=0x7fca4c14ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:37.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.361+0000 7fca51400640 1 --2- 172.21.15.27:0/269614023 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fca4c0b4f20 0x7fca4c14ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.361+0000 7fca51400640 1 -- 172.21.15.27:0/269614023 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fca4c15d2c0 msgr2=0x7fca4c1587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:37.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.361+0000 7fca51400640 1 --2- 172.21.15.27:0/269614023 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fca4c15d2c0 0x7fca4c1587a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.361+0000 7fca51400640 1 -- 172.21.15.27:0/269614023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca340094d0 con 0x7fca4c071d40 2024-01-15T18:23:37.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.361+0000 7fca51400640 1 --2- 172.21.15.27:0/269614023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca4c071d40 0x7fca4c150130 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7fca3c00d5f0 tx=0x7fca3c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:37.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.361+0000 7fca49600640 1 -- 172.21.15.27:0/269614023 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3c0125f0 con 0x7fca4c071d40 2024-01-15T18:23:37.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.361+0000 7fca49600640 1 -- 172.21.15.27:0/269614023 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3c012750 con 0x7fca4c071d40 2024-01-15T18:23:37.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.361+0000 7fca53e00640 1 -- 172.21.15.27:0/269614023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca4c15fe50 con 0x7fca4c071d40 2024-01-15T18:23:37.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.361+0000 7fca49600640 1 -- 172.21.15.27:0/269614023 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3c012ae0 con 0x7fca4c071d40 2024-01-15T18:23:37.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.361+0000 7fca53e00640 1 -- 172.21.15.27:0/269614023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca4c151920 con 0x7fca4c071d40 2024-01-15T18:23:37.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.365+0000 7fca53e00640 1 -- 172.21.15.27:0/269614023 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca4c072120 con 0x7fca4c071d40 2024-01-15T18:23:37.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.365+0000 7fca49600640 1 -- 172.21.15.27:0/269614023 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7fca3c012c40 con 0x7fca4c071d40 2024-01-15T18:23:37.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.365+0000 7fca49600640 1 --2- 172.21.15.27:0/269614023 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fca24067a60 0x7fca24069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.365+0000 7fca49600640 1 -- 172.21.15.27:0/269614023 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fca3c083630 con 0x7fca4c071d40 2024-01-15T18:23:37.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.365+0000 7fca50a00640 1 --2- 172.21.15.27:0/269614023 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fca24067a60 0x7fca24069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.369+0000 7fca50a00640 1 --2- 172.21.15.27:0/269614023 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fca24067a60 0x7fca24069f20 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fca340ae000 tx=0x7fca34008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:37.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.369+0000 7fca49600640 1 -- 172.21.15.27:0/269614023 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fca3c052020 con 0x7fca4c071d40 2024-01-15T18:23:37.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.529+0000 7fca53e00640 1 -- 172.21.15.27:0/269614023 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fca4c06d3a0 con 0x7fca24067a60 2024-01-15T18:23:37.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.533+0000 7fca49600640 1 -- 172.21.15.27:0/269614023 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fca4c06d3a0 con 0x7fca24067a60 2024-01-15T18:23:37.528 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 0 to deep-scrub 2024-01-15T18:23:37.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.533+0000 7fca53e00640 1 -- 172.21.15.27:0/269614023 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fca24067a60 msgr2=0x7fca24069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:37.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.533+0000 7fca53e00640 1 --2- 172.21.15.27:0/269614023 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fca24067a60 0x7fca24069f20 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fca340ae000 tx=0x7fca34008040 comp rx=0 tx=0).stop 2024-01-15T18:23:37.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.533+0000 7fca53e00640 1 -- 172.21.15.27:0/269614023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca4c071d40 msgr2=0x7fca4c150130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:37.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.533+0000 7fca53e00640 1 --2- 172.21.15.27:0/269614023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca4c071d40 0x7fca4c150130 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7fca3c00d5f0 tx=0x7fca3c00dab0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.533+0000 7fca53e00640 1 -- 172.21.15.27:0/269614023 shutdown_connections 2024-01-15T18:23:37.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.533+0000 7fca53e00640 1 --2- 172.21.15.27:0/269614023 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fca24067a60 0x7fca24069f20 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.533+0000 7fca53e00640 1 --2- 172.21.15.27:0/269614023 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fca4c15d2c0 0x7fca4c1587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.533+0000 7fca53e00640 1 --2- 172.21.15.27:0/269614023 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fca4c0b4f20 0x7fca4c14ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.533+0000 7fca53e00640 1 --2- 172.21.15.27:0/269614023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca4c071d40 0x7fca4c150130 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.533+0000 7fca53e00640 1 -- 172.21.15.27:0/269614023 >> 172.21.15.27:0/269614023 conn(0x7fca4c06b600 msgr2=0x7fca4c0b0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:37.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.533+0000 7fca53e00640 1 -- 172.21.15.27:0/269614023 shutdown_connections 2024-01-15T18:23:37.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.533+0000 7fca53e00640 1 -- 172.21.15.27:0/269614023 wait complete. 2024-01-15T18:23:37.551 INFO:tasks.ceph:Scrubbing osd.1 2024-01-15T18:23:37.551 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-01-15T18:23:37.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.713+0000 7f169e000640 1 Processor -- start 2024-01-15T18:23:37.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.713+0000 7f169e000640 1 -- start start 2024-01-15T18:23:37.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.713+0000 7f169e000640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1698072360 0x7f16980b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.713+0000 7f169e000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f16980cf570 0x7f16980b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.713+0000 7f169e000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16980b5980 con 0x7f169815d5c0 2024-01-15T18:23:37.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.713+0000 7f169e000640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16980ce7b0 con 0x7f1698072360 2024-01-15T18:23:37.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.713+0000 7f169e000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16980cea90 con 0x7f16980cf570 2024-01-15T18:23:37.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.713+0000 7f1696a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1698072360 0x7f16980b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.713+0000 7f1697400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f169815d5c0 0x7f1698071c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42888/0 (socket says 172.21.15.27:42888) 2024-01-15T18:23:37.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.713+0000 7f1697400640 1 -- 172.21.15.27:0/6748151 learned_addr learned my addr 172.21.15.27:0/6748151 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:37.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.713+0000 7f169ca00640 1 --2- 172.21.15.27:0/6748151 >> v2:172.21.15.93:3300/0 conn(0x7f16980cf570 0x7f16980b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.713+0000 7f1696000640 1 -- 172.21.15.27:0/6748151 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1544076689 0 0) 0x7f16980b5980 con 0x7f169815d5c0 2024-01-15T18:23:37.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.713+0000 7f1696000640 1 -- 172.21.15.27:0/6748151 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1678005740 con 0x7f169815d5c0 2024-01-15T18:23:37.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.713+0000 7f169ca00640 1 -- 172.21.15.27:0/6748151 >> v2:172.21.15.27:3300/0 conn(0x7f1698072360 msgr2=0x7f16980b4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:37.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.713+0000 7f169ca00640 1 --2- 172.21.15.27:0/6748151 >> v2:172.21.15.27:3300/0 conn(0x7f1698072360 0x7f16980b4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.713+0000 7f169ca00640 1 -- 172.21.15.27:0/6748151 >> v1:172.21.15.27:6789/0 conn(0x7f169815d5c0 legacy=0x7f1698071c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:37.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.713+0000 7f169ca00640 1 -- 172.21.15.27:0/6748151 --> v2:172.21.15.93:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16980ced70 con 0x7f16980cf570 2024-01-15T18:23:37.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.713+0000 7f1696a00640 1 --2- 172.21.15.27:0/6748151 >> v2:172.21.15.27:3300/0 conn(0x7f1698072360 0x7f16980b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:23:37.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.717+0000 7f169ca00640 1 --2- 172.21.15.27:0/6748151 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f16980cf570 0x7f16980b5050 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f1698072d90 tx=0x7f169001c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b2cf01af9dbe864 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:37.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.717+0000 7f1696000640 1 -- 172.21.15.27:0/6748151 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16900a5850 con 0x7f16980cf570 2024-01-15T18:23:37.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.717+0000 7f1696000640 1 -- 172.21.15.27:0/6748151 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16900a59b0 con 0x7f16980cf570 2024-01-15T18:23:37.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.717+0000 7f1696000640 1 -- 172.21.15.27:0/6748151 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16900a5d40 con 0x7f16980cf570 2024-01-15T18:23:37.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.717+0000 7f169e000640 1 -- 172.21.15.27:0/6748151 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f16980cf570 msgr2=0x7f16980b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:37.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.717+0000 7f169e000640 1 --2- 172.21.15.27:0/6748151 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f16980cf570 0x7f16980b5050 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f1698072d90 tx=0x7f169001c880 comp rx=0 tx=0).stop 2024-01-15T18:23:37.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.717+0000 7f169e000640 1 -- 172.21.15.27:0/6748151 shutdown_connections 2024-01-15T18:23:37.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.717+0000 7f169e000640 1 --2- 172.21.15.27:0/6748151 >> v2:172.21.15.27:3300/0 conn(0x7f1698072360 0x7f16980b4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.717+0000 7f169e000640 1 --2- 172.21.15.27:0/6748151 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f16980cf570 0x7f16980b5050 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.717+0000 7f169e000640 1 -- 172.21.15.27:0/6748151 >> 172.21.15.27:0/6748151 conn(0x7f1698099e10 msgr2=0x7f16980b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:37.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.721+0000 7f169e000640 1 -- 172.21.15.27:0/6748151 shutdown_connections 2024-01-15T18:23:37.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.721+0000 7f169e000640 1 -- 172.21.15.27:0/6748151 wait complete. 2024-01-15T18:23:37.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.721+0000 7f169e000640 1 Processor -- start 2024-01-15T18:23:37.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.721+0000 7f169e000640 1 -- start start 2024-01-15T18:23:37.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.721+0000 7f169e000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1698072360 0x7f169814ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.721+0000 7f169e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16980cf570 0x7f16981f93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.721+0000 7f169e000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f169815d5c0 0x7f1698163a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.721+0000 7f169e000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16981f9c80 con 0x7f16980cf570 2024-01-15T18:23:37.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.721+0000 7f169e000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16981f9f60 con 0x7f169815d5c0 2024-01-15T18:23:37.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.721+0000 7f169e000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16981fa240 con 0x7f1698072360 2024-01-15T18:23:37.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.721+0000 7f1696a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16980cf570 0x7f16981f93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.721+0000 7f1697400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1698072360 0x7f169814ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.721+0000 7f1696a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16980cf570 0x7f16981f93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47564/0 (socket says 172.21.15.27:47564) 2024-01-15T18:23:37.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.721+0000 7f1696a00640 1 -- 172.21.15.27:0/642881648 learned_addr learned my addr 172.21.15.27:0/642881648 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:37.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.721+0000 7f169ca00640 1 --2- 172.21.15.27:0/642881648 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f169815d5c0 0x7f1698163a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.721+0000 7f1696a00640 1 -- 172.21.15.27:0/642881648 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1698072360 msgr2=0x7f169814ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:37.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.721+0000 7f1696a00640 1 --2- 172.21.15.27:0/642881648 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f1698072360 0x7f169814ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.721+0000 7f1696a00640 1 -- 172.21.15.27:0/642881648 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f169815d5c0 msgr2=0x7f1698163a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:37.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.721+0000 7f1696a00640 1 --2- 172.21.15.27:0/642881648 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f169815d5c0 0x7f1698163a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.721+0000 7f1696a00640 1 -- 172.21.15.27:0/642881648 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1680009420 con 0x7f16980cf570 2024-01-15T18:23:37.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.725+0000 7f1696a00640 1 --2- 172.21.15.27:0/642881648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16980cf570 0x7f16981f93c0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f168000ebe0 tx=0x7f168000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:37.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.725+0000 7f168f400640 1 -- 172.21.15.27:0/642881648 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168000eee0 con 0x7f16980cf570 2024-01-15T18:23:37.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.725+0000 7f168f400640 1 -- 172.21.15.27:0/642881648 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168000cc30 con 0x7f16980cf570 2024-01-15T18:23:37.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.725+0000 7f168f400640 1 -- 172.21.15.27:0/642881648 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1680013470 con 0x7f16980cf570 2024-01-15T18:23:37.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.725+0000 7f169e000640 1 -- 172.21.15.27:0/642881648 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1690009a70 con 0x7f16980cf570 2024-01-15T18:23:37.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.725+0000 7f169e000640 1 -- 172.21.15.27:0/642881648 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1698157ff0 con 0x7f16980cf570 2024-01-15T18:23:37.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.725+0000 7f169e000640 1 -- 172.21.15.27:0/642881648 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f169809e950 con 0x7f16980cf570 2024-01-15T18:23:37.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.725+0000 7f168f400640 1 -- 172.21.15.27:0/642881648 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f16800135d0 con 0x7f16980cf570 2024-01-15T18:23:37.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.729+0000 7f168f400640 1 --2- 172.21.15.27:0/642881648 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1670067a90 0x7f1670069f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.729+0000 7f168f400640 1 -- 172.21.15.27:0/642881648 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f1680083530 con 0x7f16980cf570 2024-01-15T18:23:37.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.729+0000 7f168f400640 1 --2- 172.21.15.27:0/642881648 >> [v2:172.21.15.27:6800/3601044148,v1:172.21.15.27:6801/3601044148] conn(0x7f167006fe90 0x7f1670072330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.729+0000 7f1697400640 1 --2- 172.21.15.27:0/642881648 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1670067a90 0x7f1670069f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.729+0000 7f168f400640 1 -- 172.21.15.27:0/642881648 --> [v2:172.21.15.27:6800/3601044148,v1:172.21.15.27:6801/3601044148] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1670072a00 con 0x7f167006fe90 2024-01-15T18:23:37.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.729+0000 7f168f400640 1 -- 172.21.15.27:0/642881648 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16800840f0 con 0x7f16980cf570 2024-01-15T18:23:37.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.729+0000 7f169ca00640 1 --2- 172.21.15.27:0/642881648 >> [v2:172.21.15.27:6800/3601044148,v1:172.21.15.27:6801/3601044148] conn(0x7f167006fe90 0x7f1670072330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.729+0000 7f1697400640 1 --2- 172.21.15.27:0/642881648 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1670067a90 0x7f1670069f50 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f16880011f0 tx=0x7f1688007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:37.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.729+0000 7f169ca00640 1 --2- 172.21.15.27:0/642881648 >> [v2:172.21.15.27:6800/3601044148,v1:172.21.15.27:6801/3601044148] conn(0x7f167006fe90 0x7f1670072330 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:37.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.729+0000 7f168f400640 1 -- 172.21.15.27:0/642881648 <== osd.1 v2:172.21.15.27:6800/3601044148 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f1670072a00 con 0x7f167006fe90 2024-01-15T18:23:37.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.745+0000 7f169e000640 1 -- 172.21.15.27:0/642881648 --> [v2:172.21.15.27:6800/3601044148,v1:172.21.15.27:6801/3601044148] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f169804e930 con 0x7f167006fe90 2024-01-15T18:23:37.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.745+0000 7f168f400640 1 -- 172.21.15.27:0/642881648 <== osd.1 v2:172.21.15.27:6800/3601044148 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f169804e930 con 0x7f167006fe90 2024-01-15T18:23:37.742 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:23:37.742 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-15T18:23:37.742 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:23:37.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.745+0000 7f169e000640 1 -- 172.21.15.27:0/642881648 >> [v2:172.21.15.27:6800/3601044148,v1:172.21.15.27:6801/3601044148] conn(0x7f167006fe90 msgr2=0x7f1670072330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:37.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.745+0000 7f169e000640 1 --2- 172.21.15.27:0/642881648 >> [v2:172.21.15.27:6800/3601044148,v1:172.21.15.27:6801/3601044148] conn(0x7f167006fe90 0x7f1670072330 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.745+0000 7f169e000640 1 -- 172.21.15.27:0/642881648 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1670067a90 msgr2=0x7f1670069f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:37.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.745+0000 7f169e000640 1 --2- 172.21.15.27:0/642881648 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f1670067a90 0x7f1670069f50 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f16880011f0 tx=0x7f1688007d40 comp rx=0 tx=0).stop 2024-01-15T18:23:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.745+0000 7f169e000640 1 -- 172.21.15.27:0/642881648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16980cf570 msgr2=0x7f16981f93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.745+0000 7f169e000640 1 --2- 172.21.15.27:0/642881648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16980cf570 0x7f16981f93c0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f168000ebe0 tx=0x7f168000c520 comp rx=0 tx=0).stop 2024-01-15T18:23:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.745+0000 7f169ca00640 1 -- 172.21.15.27:0/642881648 reap_dead start 2024-01-15T18:23:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.745+0000 7f169e000640 1 -- 172.21.15.27:0/642881648 shutdown_connections 2024-01-15T18:23:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.745+0000 7f169e000640 1 -- 172.21.15.27:0/642881648 >> 172.21.15.27:0/642881648 conn(0x7f1698099e10 msgr2=0x7f16980cfcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.745+0000 7f169e000640 1 -- 172.21.15.27:0/642881648 shutdown_connections 2024-01-15T18:23:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.745+0000 7f169e000640 1 -- 172.21.15.27:0/642881648 wait complete. 2024-01-15T18:23:37.764 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-01-15T18:23:37.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.909+0000 7fe514e00640 1 Processor -- start 2024-01-15T18:23:37.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.909+0000 7fe514e00640 1 -- start start 2024-01-15T18:23:37.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.913+0000 7fe514e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe510153c80 0x7fe510071020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.913+0000 7fe514e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe510070b90 con 0x7fe51015a320 2024-01-15T18:23:37.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.913+0000 7fe514e00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe51015a730 con 0x7fe51006ca90 2024-01-15T18:23:37.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.913+0000 7fe514e00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe51015a980 con 0x7fe510153c80 2024-01-15T18:23:37.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.913+0000 7fe50de00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe510153c80 0x7fe510071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.913+0000 7fe50e800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe51015a320 0x7fe5100b1b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42902/0 (socket says 172.21.15.27:42902) 2024-01-15T18:23:37.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.913+0000 7fe50e800640 1 -- 172.21.15.27:0/2288678739 learned_addr learned my addr 172.21.15.27:0/2288678739 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:37.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.913+0000 7fe50ca00640 1 -- 172.21.15.27:0/2288678739 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 212234487 0 0) 0x7fe510070b90 con 0x7fe51015a320 2024-01-15T18:23:37.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.913+0000 7fe50de00640 1 -- 172.21.15.27:0/2288678739 >> v1:172.21.15.93:6789/0 conn(0x7fe51006ca90 legacy=0x7fe510159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:37.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.913+0000 7fe50de00640 1 -- 172.21.15.27:0/2288678739 >> v1:172.21.15.27:6789/0 conn(0x7fe51015a320 legacy=0x7fe5100b1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:37.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.913+0000 7fe50de00640 1 -- 172.21.15.27:0/2288678739 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe510070260 con 0x7fe510153c80 2024-01-15T18:23:37.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.917+0000 7fe50de00640 1 --2- 172.21.15.27:0/2288678739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe510153c80 0x7fe510071020 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7fe5000097c0 tx=0x7fe50001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f68e5a961d9da05 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:37.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.917+0000 7fe50ca00640 1 -- 172.21.15.27:0/2288678739 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5000ac640 con 0x7fe510153c80 2024-01-15T18:23:37.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.917+0000 7fe50ca00640 1 -- 172.21.15.27:0/2288678739 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5000ac7a0 con 0x7fe510153c80 2024-01-15T18:23:37.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.917+0000 7fe50ca00640 1 -- 172.21.15.27:0/2288678739 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5000acb30 con 0x7fe510153c80 2024-01-15T18:23:37.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.917+0000 7fe514e00640 1 -- 172.21.15.27:0/2288678739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe510153c80 msgr2=0x7fe510071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:37.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.917+0000 7fe514e00640 1 --2- 172.21.15.27:0/2288678739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe510153c80 0x7fe510071020 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7fe5000097c0 tx=0x7fe50001c3c0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.917+0000 7fe514e00640 1 -- 172.21.15.27:0/2288678739 shutdown_connections 2024-01-15T18:23:37.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.917+0000 7fe514e00640 1 --2- 172.21.15.27:0/2288678739 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe510153c80 0x7fe510071020 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.917+0000 7fe514e00640 1 -- 172.21.15.27:0/2288678739 >> 172.21.15.27:0/2288678739 conn(0x7fe5100b0850 msgr2=0x7fe5100b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:37.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.917+0000 7fe514e00640 1 -- 172.21.15.27:0/2288678739 shutdown_connections 2024-01-15T18:23:37.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.917+0000 7fe514e00640 1 -- 172.21.15.27:0/2288678739 wait complete. 2024-01-15T18:23:37.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.917+0000 7fe514e00640 1 Processor -- start 2024-01-15T18:23:37.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.917+0000 7fe514e00640 1 -- start start 2024-01-15T18:23:37.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.921+0000 7fe514e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe51006ca90 0x7fe5100aef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.921+0000 7fe514e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe510153c80 0x7fe510149ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.921+0000 7fe514e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe51015a320 0x7fe51014b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.921+0000 7fe514e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51014a950 con 0x7fe51006ca90 2024-01-15T18:23:37.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.921+0000 7fe514e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5100af5a0 con 0x7fe51015a320 2024-01-15T18:23:37.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.921+0000 7fe514e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5100ad8d0 con 0x7fe510153c80 2024-01-15T18:23:37.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.921+0000 7fe50de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe51006ca90 0x7fe5100aef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.921+0000 7fe50de00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe51006ca90 0x7fe5100aef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47580/0 (socket says 172.21.15.27:47580) 2024-01-15T18:23:37.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.921+0000 7fe50e800640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe51015a320 0x7fe51014b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.921+0000 7fe50de00640 1 -- 172.21.15.27:0/3685514428 learned_addr learned my addr 172.21.15.27:0/3685514428 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:37.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.921+0000 7fe50d400640 1 --2- 172.21.15.27:0/3685514428 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe510153c80 0x7fe510149ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.921+0000 7fe50de00640 1 -- 172.21.15.27:0/3685514428 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe510153c80 msgr2=0x7fe510149ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:37.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.921+0000 7fe50de00640 1 --2- 172.21.15.27:0/3685514428 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe510153c80 0x7fe510149ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.921+0000 7fe50de00640 1 -- 172.21.15.27:0/3685514428 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe51015a320 msgr2=0x7fe51014b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:37.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.921+0000 7fe50de00640 1 --2- 172.21.15.27:0/3685514428 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe51015a320 0x7fe51014b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:37.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.921+0000 7fe50de00640 1 -- 172.21.15.27:0/3685514428 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe500009420 con 0x7fe51006ca90 2024-01-15T18:23:37.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.921+0000 7fe50e800640 1 --2- 172.21.15.27:0/3685514428 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe51015a320 0x7fe51014b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:23:37.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.921+0000 7fe50de00640 1 --2- 172.21.15.27:0/3685514428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe51006ca90 0x7fe5100aef50 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7fe500018fd0 tx=0x7fe500019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:37.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.921+0000 7fe506000640 1 -- 172.21.15.27:0/3685514428 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5000ac640 con 0x7fe51006ca90 2024-01-15T18:23:37.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.921+0000 7fe506000640 1 -- 172.21.15.27:0/3685514428 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5000b5840 con 0x7fe51006ca90 2024-01-15T18:23:37.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.921+0000 7fe506000640 1 -- 172.21.15.27:0/3685514428 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5000b5bf0 con 0x7fe51006ca90 2024-01-15T18:23:37.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.921+0000 7fe514e00640 1 -- 172.21.15.27:0/3685514428 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe51006d430 con 0x7fe51006ca90 2024-01-15T18:23:37.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.921+0000 7fe514e00640 1 -- 172.21.15.27:0/3685514428 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5100ade90 con 0x7fe51006ca90 2024-01-15T18:23:37.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.921+0000 7fe514e00640 1 -- 172.21.15.27:0/3685514428 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe51004e850 con 0x7fe51006ca90 2024-01-15T18:23:37.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.925+0000 7fe506000640 1 -- 172.21.15.27:0/3685514428 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7fe5000b3040 con 0x7fe51006ca90 2024-01-15T18:23:37.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.925+0000 7fe506000640 1 --2- 172.21.15.27:0/3685514428 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe4e8067a90 0x7fe4e8069f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:37.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.925+0000 7fe506000640 1 -- 172.21.15.27:0/3685514428 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fe50011d6c0 con 0x7fe51006ca90 2024-01-15T18:23:37.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.925+0000 7fe50d400640 1 --2- 172.21.15.27:0/3685514428 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe4e8067a90 0x7fe4e8069f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:37.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.925+0000 7fe50d400640 1 --2- 172.21.15.27:0/3685514428 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe4e8067a90 0x7fe4e8069f50 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fe510153100 tx=0x7fe4f8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:37.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:37.929+0000 7fe506000640 1 -- 172.21.15.27:0/3685514428 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe5000ec030 con 0x7fe51006ca90 2024-01-15T18:23:38.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.085+0000 7fe514e00640 1 -- 172.21.15.27:0/3685514428 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe510067c00 con 0x7fe4e8067a90 2024-01-15T18:23:38.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.085+0000 7fe506000640 1 -- 172.21.15.27:0/3685514428 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe510067c00 con 0x7fe4e8067a90 2024-01-15T18:23:38.080 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 1 to deep-scrub 2024-01-15T18:23:38.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.085+0000 7fe514e00640 1 -- 172.21.15.27:0/3685514428 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe4e8067a90 msgr2=0x7fe4e8069f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:38.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.085+0000 7fe514e00640 1 --2- 172.21.15.27:0/3685514428 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe4e8067a90 0x7fe4e8069f50 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fe510153100 tx=0x7fe4f8009040 comp rx=0 tx=0).stop 2024-01-15T18:23:38.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.085+0000 7fe514e00640 1 -- 172.21.15.27:0/3685514428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe51006ca90 msgr2=0x7fe5100aef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:38.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.085+0000 7fe514e00640 1 --2- 172.21.15.27:0/3685514428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe51006ca90 0x7fe5100aef50 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7fe500018fd0 tx=0x7fe500019160 comp rx=0 tx=0).stop 2024-01-15T18:23:38.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.085+0000 7fe514e00640 1 -- 172.21.15.27:0/3685514428 shutdown_connections 2024-01-15T18:23:38.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.085+0000 7fe514e00640 1 --2- 172.21.15.27:0/3685514428 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fe4e8067a90 0x7fe4e8069f50 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.085+0000 7fe514e00640 1 --2- 172.21.15.27:0/3685514428 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fe51015a320 0x7fe51014b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.085+0000 7fe514e00640 1 --2- 172.21.15.27:0/3685514428 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fe510153c80 0x7fe510149ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.085+0000 7fe514e00640 1 --2- 172.21.15.27:0/3685514428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe51006ca90 0x7fe5100aef50 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.085+0000 7fe514e00640 1 -- 172.21.15.27:0/3685514428 >> 172.21.15.27:0/3685514428 conn(0x7fe5100b0850 msgr2=0x7fe5101581e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:38.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.085+0000 7fe514e00640 1 -- 172.21.15.27:0/3685514428 shutdown_connections 2024-01-15T18:23:38.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.085+0000 7fe514e00640 1 -- 172.21.15.27:0/3685514428 wait complete. 2024-01-15T18:23:38.102 INFO:tasks.ceph:Scrubbing osd.2 2024-01-15T18:23:38.103 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-01-15T18:23:38.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.253+0000 7f0095e00640 1 Processor -- start 2024-01-15T18:23:38.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.253+0000 7f0095e00640 1 -- start start 2024-01-15T18:23:38.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.253+0000 7f0095e00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f0090158900 0x7f009015f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.253+0000 7f0095e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f009015bf00 0x7f00900cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.253+0000 7f0095e00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00900b1cb0 con 0x7f00900cf110 2024-01-15T18:23:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.253+0000 7f0095e00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00900cf540 con 0x7f0090158900 2024-01-15T18:23:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.253+0000 7f0095e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00900b1f00 con 0x7f009015bf00 2024-01-15T18:23:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.253+0000 7f008f400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f00900cf110 0x7f00900701e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42918/0 (socket says 172.21.15.27:42918) 2024-01-15T18:23:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.253+0000 7f008f400640 1 -- 172.21.15.27:0/619902206 learned_addr learned my addr 172.21.15.27:0/619902206 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.253+0000 7f008e000640 1 --2- 172.21.15.27:0/619902206 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f009015bf00 0x7f00900cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:38.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.253+0000 7f008ea00640 1 --2- 172.21.15.27:0/619902206 >> v2:172.21.15.93:3300/0 conn(0x7f0090158900 0x7f009015f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:38.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.253+0000 7f008d600640 1 -- 172.21.15.27:0/619902206 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1113180673 0 0) 0x7f00900b1cb0 con 0x7f00900cf110 2024-01-15T18:23:38.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.253+0000 7f008d600640 1 -- 172.21.15.27:0/619902206 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0070003530 con 0x7f00900cf110 2024-01-15T18:23:38.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.253+0000 7f008e000640 1 -- 172.21.15.27:0/619902206 >> v2:172.21.15.93:3300/0 conn(0x7f0090158900 msgr2=0x7f009015f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:38.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.253+0000 7f008e000640 1 --2- 172.21.15.27:0/619902206 >> v2:172.21.15.93:3300/0 conn(0x7f0090158900 0x7f009015f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.253+0000 7f008e000640 1 -- 172.21.15.27:0/619902206 >> v1:172.21.15.27:6789/0 conn(0x7f00900cf110 legacy=0x7f00900701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:38.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.253+0000 7f008e000640 1 -- 172.21.15.27:0/619902206 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00900b21e0 con 0x7f009015bf00 2024-01-15T18:23:38.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.253+0000 7f008ea00640 1 --2- 172.21.15.27:0/619902206 >> v2:172.21.15.93:3300/0 conn(0x7f0090158900 0x7f009015f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:23:38.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.257+0000 7f008e000640 1 --2- 172.21.15.27:0/619902206 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f009015bf00 0x7f00900cebd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f00780097c0 tx=0x7f007801c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b9e69ca8bee989c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:38.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.261+0000 7f008d600640 1 -- 172.21.15.27:0/619902206 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00780ab600 con 0x7f009015bf00 2024-01-15T18:23:38.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.261+0000 7f008d600640 1 -- 172.21.15.27:0/619902206 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00780a7050 con 0x7f009015bf00 2024-01-15T18:23:38.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.261+0000 7f008d600640 1 -- 172.21.15.27:0/619902206 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00780ab980 con 0x7f009015bf00 2024-01-15T18:23:38.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.261+0000 7f0095e00640 1 -- 172.21.15.27:0/619902206 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f009015bf00 msgr2=0x7f00900cebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:38.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.261+0000 7f0095e00640 1 --2- 172.21.15.27:0/619902206 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f009015bf00 0x7f00900cebd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f00780097c0 tx=0x7f007801c480 comp rx=0 tx=0).stop 2024-01-15T18:23:38.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.261+0000 7f0095e00640 1 -- 172.21.15.27:0/619902206 shutdown_connections 2024-01-15T18:23:38.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.261+0000 7f0095e00640 1 --2- 172.21.15.27:0/619902206 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f009015bf00 0x7f00900cebd0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.261+0000 7f0095e00640 1 --2- 172.21.15.27:0/619902206 >> v2:172.21.15.93:3300/0 conn(0x7f0090158900 0x7f009015f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.261+0000 7f0095e00640 1 -- 172.21.15.27:0/619902206 >> 172.21.15.27:0/619902206 conn(0x7f00900cb700 msgr2=0x7f00900cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:38.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.261+0000 7f0095e00640 1 -- 172.21.15.27:0/619902206 shutdown_connections 2024-01-15T18:23:38.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.261+0000 7f0095e00640 1 -- 172.21.15.27:0/619902206 wait complete. 2024-01-15T18:23:38.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.265+0000 7f0095e00640 1 Processor -- start 2024-01-15T18:23:38.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.265+0000 7f0095e00640 1 -- start start 2024-01-15T18:23:38.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.265+0000 7f0095e00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f00900cf110 0x7f00900cd430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:38.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.265+0000 7f0095e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0090158900 0x7f009015a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:38.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.265+0000 7f0095e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f009015bf00 0x7f0090151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:38.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.265+0000 7f0095e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00900af000 con 0x7f0090158900 2024-01-15T18:23:38.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.265+0000 7f0095e00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00900cda10 con 0x7f009015bf00 2024-01-15T18:23:38.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.265+0000 7f0095e00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00900cdcf0 con 0x7f00900cf110 2024-01-15T18:23:38.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.265+0000 7f008e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0090158900 0x7f009015a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:38.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.265+0000 7f008e000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0090158900 0x7f009015a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47596/0 (socket says 172.21.15.27:47596) 2024-01-15T18:23:38.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.265+0000 7f008e000640 1 -- 172.21.15.27:0/3254656101 learned_addr learned my addr 172.21.15.27:0/3254656101 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:38.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.265+0000 7f008f400640 1 --2- 172.21.15.27:0/3254656101 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f009015bf00 0x7f0090151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:38.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.265+0000 7f008ea00640 1 --2- 172.21.15.27:0/3254656101 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f00900cf110 0x7f00900cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:38.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.265+0000 7f008e000640 1 -- 172.21.15.27:0/3254656101 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f00900cf110 msgr2=0x7f00900cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:38.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.265+0000 7f008e000640 1 --2- 172.21.15.27:0/3254656101 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f00900cf110 0x7f00900cd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.265+0000 7f008e000640 1 -- 172.21.15.27:0/3254656101 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f009015bf00 msgr2=0x7f0090151f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:38.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.265+0000 7f008e000640 1 --2- 172.21.15.27:0/3254656101 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f009015bf00 0x7f0090151f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.265+0000 7f008e000640 1 -- 172.21.15.27:0/3254656101 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0080009420 con 0x7f0090158900 2024-01-15T18:23:38.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.265+0000 7f008e000640 1 --2- 172.21.15.27:0/3254656101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0090158900 0x7f009015a620 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f007801ab30 tx=0x7f0078019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:38.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.265+0000 7f0087400640 1 -- 172.21.15.27:0/3254656101 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00780b4400 con 0x7f0090158900 2024-01-15T18:23:38.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.265+0000 7f0087400640 1 -- 172.21.15.27:0/3254656101 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00780b4580 con 0x7f0090158900 2024-01-15T18:23:38.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.265+0000 7f0087400640 1 -- 172.21.15.27:0/3254656101 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00780b4930 con 0x7f0090158900 2024-01-15T18:23:38.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.265+0000 7f0095e00640 1 -- 172.21.15.27:0/3254656101 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0078009420 con 0x7f0090158900 2024-01-15T18:23:38.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.265+0000 7f0095e00640 1 -- 172.21.15.27:0/3254656101 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00900cdfd0 con 0x7f0090158900 2024-01-15T18:23:38.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.265+0000 7f0095e00640 1 -- 172.21.15.27:0/3254656101 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f009004e850 con 0x7f0090158900 2024-01-15T18:23:38.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.269+0000 7f0087400640 1 -- 172.21.15.27:0/3254656101 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f00780b2040 con 0x7f0090158900 2024-01-15T18:23:38.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.269+0000 7f0087400640 1 --2- 172.21.15.27:0/3254656101 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0068067a60 0x7f0068069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:38.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.269+0000 7f0087400640 1 -- 172.21.15.27:0/3254656101 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f007811d990 con 0x7f0090158900 2024-01-15T18:23:38.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.269+0000 7f0087400640 1 --2- 172.21.15.27:0/3254656101 >> [v2:172.21.15.27:6824/366025692,v1:172.21.15.27:6825/366025692] conn(0x7f006806fe60 0x7f0068072300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:38.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.269+0000 7f0087400640 1 -- 172.21.15.27:0/3254656101 --> [v2:172.21.15.27:6824/366025692,v1:172.21.15.27:6825/366025692] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f00680729d0 con 0x7f006806fe60 2024-01-15T18:23:38.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.269+0000 7f0087400640 1 -- 172.21.15.27:0/3254656101 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7f007811dd00 con 0x7f0090158900 2024-01-15T18:23:38.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.269+0000 7f008ea00640 1 --2- 172.21.15.27:0/3254656101 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0068067a60 0x7f0068069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:38.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.269+0000 7f008f400640 1 --2- 172.21.15.27:0/3254656101 >> [v2:172.21.15.27:6824/366025692,v1:172.21.15.27:6825/366025692] conn(0x7f006806fe60 0x7f0068072300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:38.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.269+0000 7f008ea00640 1 --2- 172.21.15.27:0/3254656101 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0068067a60 0x7f0068069f20 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f0080009cf0 tx=0x7f0080009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:38.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.269+0000 7f008f400640 1 --2- 172.21.15.27:0/3254656101 >> [v2:172.21.15.27:6824/366025692,v1:172.21.15.27:6825/366025692] conn(0x7f006806fe60 0x7f0068072300 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:38.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.273+0000 7f0087400640 1 -- 172.21.15.27:0/3254656101 <== osd.2 v2:172.21.15.27:6824/366025692 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f00680729d0 con 0x7f006806fe60 2024-01-15T18:23:38.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.285+0000 7f0095e00640 1 -- 172.21.15.27:0/3254656101 --> [v2:172.21.15.27:6824/366025692,v1:172.21.15.27:6825/366025692] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f009006d540 con 0x7f006806fe60 2024-01-15T18:23:38.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.285+0000 7f0087400640 1 -- 172.21.15.27:0/3254656101 <== osd.2 v2:172.21.15.27:6824/366025692 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f009006d540 con 0x7f006806fe60 2024-01-15T18:23:38.281 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:23:38.281 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-15T18:23:38.282 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:23:38.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.285+0000 7f0095e00640 1 -- 172.21.15.27:0/3254656101 >> [v2:172.21.15.27:6824/366025692,v1:172.21.15.27:6825/366025692] conn(0x7f006806fe60 msgr2=0x7f0068072300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:38.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.285+0000 7f0095e00640 1 --2- 172.21.15.27:0/3254656101 >> [v2:172.21.15.27:6824/366025692,v1:172.21.15.27:6825/366025692] conn(0x7f006806fe60 0x7f0068072300 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.285+0000 7f0095e00640 1 -- 172.21.15.27:0/3254656101 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0068067a60 msgr2=0x7f0068069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:38.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.285+0000 7f0095e00640 1 --2- 172.21.15.27:0/3254656101 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0068067a60 0x7f0068069f20 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f0080009cf0 tx=0x7f0080009190 comp rx=0 tx=0).stop 2024-01-15T18:23:38.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.285+0000 7f0095e00640 1 -- 172.21.15.27:0/3254656101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0090158900 msgr2=0x7f009015a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:38.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.285+0000 7f0095e00640 1 --2- 172.21.15.27:0/3254656101 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0090158900 0x7f009015a620 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f007801ab30 tx=0x7f0078019220 comp rx=0 tx=0).stop 2024-01-15T18:23:38.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.285+0000 7f008f400640 1 -- 172.21.15.27:0/3254656101 reap_dead start 2024-01-15T18:23:38.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.285+0000 7f0095e00640 1 -- 172.21.15.27:0/3254656101 shutdown_connections 2024-01-15T18:23:38.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.285+0000 7f0095e00640 1 -- 172.21.15.27:0/3254656101 >> 172.21.15.27:0/3254656101 conn(0x7f00900cb700 msgr2=0x7f00900cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:38.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.285+0000 7f0095e00640 1 -- 172.21.15.27:0/3254656101 shutdown_connections 2024-01-15T18:23:38.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.285+0000 7f0095e00640 1 -- 172.21.15.27:0/3254656101 wait complete. 2024-01-15T18:23:38.300 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-01-15T18:23:38.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.445+0000 7f8e91c00640 1 Processor -- start 2024-01-15T18:23:38.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.445+0000 7f8e91c00640 1 -- start start 2024-01-15T18:23:38.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.449+0000 7f8e91c00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8e8c15b180 0x7f8e8c0cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:38.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.449+0000 7f8e91c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f8e8c0cd7e0 0x7f8e8c0cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:38.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.449+0000 7f8e91c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e8c0ce180 con 0x7f8e8c0b09d0 2024-01-15T18:23:38.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.449+0000 7f8e91c00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e8c0d1e20 con 0x7f8e8c15b180 2024-01-15T18:23:38.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.449+0000 7f8e91c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e8c0ceed0 con 0x7f8e8c0cd7e0 2024-01-15T18:23:38.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.449+0000 7f8e8aa00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8e8c15b180 0x7f8e8c0cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:38.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.449+0000 7f8e8b400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8e8c0b09d0 0x7f8e8c0b0e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42928/0 (socket says 172.21.15.27:42928) 2024-01-15T18:23:38.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.449+0000 7f8e8b400640 1 -- 172.21.15.27:0/2888494393 learned_addr learned my addr 172.21.15.27:0/2888494393 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:38.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.449+0000 7f8e8a000640 1 --2- 172.21.15.27:0/2888494393 >> v2:172.21.15.93:3300/0 conn(0x7f8e8c0cd7e0 0x7f8e8c0cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:38.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.449+0000 7f8e8aa00640 1 -- 172.21.15.27:0/2888494393 >> v2:172.21.15.93:3300/0 conn(0x7f8e8c0cd7e0 msgr2=0x7f8e8c0cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:38.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.449+0000 7f8e8aa00640 1 --2- 172.21.15.27:0/2888494393 >> v2:172.21.15.93:3300/0 conn(0x7f8e8c0cd7e0 0x7f8e8c0cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.449+0000 7f8e8aa00640 1 -- 172.21.15.27:0/2888494393 >> v1:172.21.15.27:6789/0 conn(0x7f8e8c0b09d0 legacy=0x7f8e8c0b0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:38.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.449+0000 7f8e8aa00640 1 -- 172.21.15.27:0/2888494393 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8c0ce7d0 con 0x7f8e8c15b180 2024-01-15T18:23:38.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.453+0000 7f8e8aa00640 1 --2- 172.21.15.27:0/2888494393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e8c15b180 0x7f8e8c0cfa90 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c071d00 tx=0x7f8e7c01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3112d84e542367d4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:38.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.453+0000 7f8e89600640 1 -- 172.21.15.27:0/2888494393 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c0ab600 con 0x7f8e8c15b180 2024-01-15T18:23:38.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.453+0000 7f8e89600640 1 -- 172.21.15.27:0/2888494393 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7c0a7050 con 0x7f8e8c15b180 2024-01-15T18:23:38.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.453+0000 7f8e89600640 1 -- 172.21.15.27:0/2888494393 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c0ab980 con 0x7f8e8c15b180 2024-01-15T18:23:38.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.453+0000 7f8e91c00640 1 -- 172.21.15.27:0/2888494393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e8c15b180 msgr2=0x7f8e8c0cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:38.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.453+0000 7f8e91c00640 1 --2- 172.21.15.27:0/2888494393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e8c15b180 0x7f8e8c0cfa90 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c071d00 tx=0x7f8e7c01c430 comp rx=0 tx=0).stop 2024-01-15T18:23:38.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.453+0000 7f8e91c00640 1 -- 172.21.15.27:0/2888494393 shutdown_connections 2024-01-15T18:23:38.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.453+0000 7f8e91c00640 1 --2- 172.21.15.27:0/2888494393 >> v2:172.21.15.93:3300/0 conn(0x7f8e8c0cd7e0 0x7f8e8c0cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.453+0000 7f8e91c00640 1 --2- 172.21.15.27:0/2888494393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e8c15b180 0x7f8e8c0cfa90 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.453+0000 7f8e91c00640 1 -- 172.21.15.27:0/2888494393 >> 172.21.15.27:0/2888494393 conn(0x7f8e8c0d0e80 msgr2=0x7f8e8c0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:38.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.453+0000 7f8e91c00640 1 -- 172.21.15.27:0/2888494393 shutdown_connections 2024-01-15T18:23:38.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.453+0000 7f8e91c00640 1 -- 172.21.15.27:0/2888494393 wait complete. 2024-01-15T18:23:38.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.453+0000 7f8e91c00640 1 Processor -- start 2024-01-15T18:23:38.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.453+0000 7f8e91c00640 1 -- start start 2024-01-15T18:23:38.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.457+0000 7f8e91c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8e8c0b09d0 0x7f8e8c151660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:38.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.457+0000 7f8e91c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e8c0cd7e0 0x7f8e8c150ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:38.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.457+0000 7f8e91c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8e8c15b180 0x7f8e8c157530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:38.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.457+0000 7f8e91c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c0ca570 con 0x7f8e8c0cd7e0 2024-01-15T18:23:38.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.457+0000 7f8e91c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c1558b0 con 0x7f8e8c0b09d0 2024-01-15T18:23:38.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.457+0000 7f8e91c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c155b90 con 0x7f8e8c15b180 2024-01-15T18:23:38.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.457+0000 7f8e8a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e8c0cd7e0 0x7f8e8c150ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:38.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.457+0000 7f8e8a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e8c0cd7e0 0x7f8e8c150ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47622/0 (socket says 172.21.15.27:47622) 2024-01-15T18:23:38.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.457+0000 7f8e8a000640 1 -- 172.21.15.27:0/4230760303 learned_addr learned my addr 172.21.15.27:0/4230760303 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:38.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.457+0000 7f8e8b400640 1 --2- 172.21.15.27:0/4230760303 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8e8c15b180 0x7f8e8c157530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:38.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.457+0000 7f8e8aa00640 1 --2- 172.21.15.27:0/4230760303 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8e8c0b09d0 0x7f8e8c151660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:38.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.457+0000 7f8e8a000640 1 -- 172.21.15.27:0/4230760303 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8e8c15b180 msgr2=0x7f8e8c157530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:38.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.457+0000 7f8e8a000640 1 --2- 172.21.15.27:0/4230760303 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8e8c15b180 0x7f8e8c157530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.457+0000 7f8e8a000640 1 -- 172.21.15.27:0/4230760303 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8e8c0b09d0 msgr2=0x7f8e8c151660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:38.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.457+0000 7f8e8a000640 1 --2- 172.21.15.27:0/4230760303 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8e8c0b09d0 0x7f8e8c151660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.457+0000 7f8e8a000640 1 -- 172.21.15.27:0/4230760303 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e7c009420 con 0x7f8e8c0cd7e0 2024-01-15T18:23:38.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.457+0000 7f8e8a000640 1 --2- 172.21.15.27:0/4230760303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e8c0cd7e0 0x7f8e8c150ec0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f8e7400e870 tx=0x7f8e7400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:38.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.457+0000 7f8e82a00640 1 -- 172.21.15.27:0/4230760303 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e74009510 con 0x7f8e8c0cd7e0 2024-01-15T18:23:38.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.457+0000 7f8e82a00640 1 -- 172.21.15.27:0/4230760303 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7400eec0 con 0x7f8e8c0cd7e0 2024-01-15T18:23:38.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.457+0000 7f8e82a00640 1 -- 172.21.15.27:0/4230760303 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7400cb40 con 0x7f8e8c0cd7e0 2024-01-15T18:23:38.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.457+0000 7f8e91c00640 1 -- 172.21.15.27:0/4230760303 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e8c155e70 con 0x7f8e8c0cd7e0 2024-01-15T18:23:38.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.457+0000 7f8e91c00640 1 -- 172.21.15.27:0/4230760303 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e8c151400 con 0x7f8e8c0cd7e0 2024-01-15T18:23:38.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.457+0000 7f8e91c00640 1 -- 172.21.15.27:0/4230760303 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e8c04e850 con 0x7f8e8c0cd7e0 2024-01-15T18:23:38.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.461+0000 7f8e82a00640 1 -- 172.21.15.27:0/4230760303 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f8e7400cca0 con 0x7f8e8c0cd7e0 2024-01-15T18:23:38.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.461+0000 7f8e82a00640 1 --2- 172.21.15.27:0/4230760303 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8e64067a60 0x7f8e64069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:38.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.461+0000 7f8e82a00640 1 -- 172.21.15.27:0/4230760303 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f8e74083bb0 con 0x7f8e8c0cd7e0 2024-01-15T18:23:38.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.461+0000 7f8e8aa00640 1 --2- 172.21.15.27:0/4230760303 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8e64067a60 0x7f8e64069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:38.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.461+0000 7f8e8aa00640 1 --2- 172.21.15.27:0/4230760303 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8e64067a60 0x7f8e64069f20 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c073d70 tx=0x7f8e7c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:38.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.465+0000 7f8e82a00640 1 -- 172.21.15.27:0/4230760303 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8e740525e0 con 0x7f8e8c0cd7e0 2024-01-15T18:23:38.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.621+0000 7f8e91c00640 1 -- 172.21.15.27:0/4230760303 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e8c167e20 con 0x7f8e64067a60 2024-01-15T18:23:38.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.621+0000 7f8e82a00640 1 -- 172.21.15.27:0/4230760303 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8e7c0ab7e0 con 0x7f8e64067a60 2024-01-15T18:23:38.617 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 2 to deep-scrub 2024-01-15T18:23:38.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.621+0000 7f8e91c00640 1 -- 172.21.15.27:0/4230760303 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8e64067a60 msgr2=0x7f8e64069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:38.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.621+0000 7f8e91c00640 1 --2- 172.21.15.27:0/4230760303 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8e64067a60 0x7f8e64069f20 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c073d70 tx=0x7f8e7c008040 comp rx=0 tx=0).stop 2024-01-15T18:23:38.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.621+0000 7f8e91c00640 1 -- 172.21.15.27:0/4230760303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e8c0cd7e0 msgr2=0x7f8e8c150ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:38.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.621+0000 7f8e91c00640 1 --2- 172.21.15.27:0/4230760303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e8c0cd7e0 0x7f8e8c150ec0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f8e7400e870 tx=0x7f8e7400ec70 comp rx=0 tx=0).stop 2024-01-15T18:23:38.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.621+0000 7f8e91c00640 1 -- 172.21.15.27:0/4230760303 shutdown_connections 2024-01-15T18:23:38.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.621+0000 7f8e91c00640 1 --2- 172.21.15.27:0/4230760303 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8e64067a60 0x7f8e64069f20 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.625+0000 7f8e91c00640 1 --2- 172.21.15.27:0/4230760303 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8e8c15b180 0x7f8e8c157530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.625+0000 7f8e91c00640 1 --2- 172.21.15.27:0/4230760303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e8c0cd7e0 0x7f8e8c150ec0 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.625+0000 7f8e91c00640 1 --2- 172.21.15.27:0/4230760303 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8e8c0b09d0 0x7f8e8c151660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.625+0000 7f8e91c00640 1 -- 172.21.15.27:0/4230760303 >> 172.21.15.27:0/4230760303 conn(0x7f8e8c0d0e80 msgr2=0x7f8e8c0af540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:38.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.625+0000 7f8e91c00640 1 -- 172.21.15.27:0/4230760303 shutdown_connections 2024-01-15T18:23:38.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.625+0000 7f8e91c00640 1 -- 172.21.15.27:0/4230760303 wait complete. 2024-01-15T18:23:38.642 INFO:tasks.ceph:Scrubbing osd.3 2024-01-15T18:23:38.643 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-01-15T18:23:38.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.805+0000 7f256c000640 1 Processor -- start 2024-01-15T18:23:38.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.805+0000 7f256c000640 1 -- start start 2024-01-15T18:23:38.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.805+0000 7f256c000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f256406c430 0x7f2564070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:38.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.805+0000 7f256c000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f25640ce760 0x7f25640b2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:38.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.805+0000 7f256c000640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25640ccfb0 con 0x7f25640b1db0 2024-01-15T18:23:38.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.805+0000 7f256c000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25640cd290 con 0x7f25640ce760 2024-01-15T18:23:38.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.805+0000 7f256c000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25640cd570 con 0x7f256406c430 2024-01-15T18:23:38.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.805+0000 7f2569600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f256406c430 0x7f2564070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:38.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.805+0000 7f256a000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f25640ce760 0x7f25640b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:38.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.805+0000 7f256a000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f25640ce760 0x7f25640b2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:38080/0 (socket says 172.21.15.27:38080) 2024-01-15T18:23:38.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.805+0000 7f2569600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f256406c430 0x7f2564070ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:48148/0 (socket says 172.21.15.27:48148) 2024-01-15T18:23:38.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.805+0000 7f256a000640 1 -- 172.21.15.27:0/2459340951 learned_addr learned my addr 172.21.15.27:0/2459340951 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:38.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.805+0000 7f2569600640 1 -- 172.21.15.27:0/2459340951 >> v2:172.21.15.93:3300/0 conn(0x7f25640ce760 msgr2=0x7f25640b2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:38.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.805+0000 7f2569600640 1 --2- 172.21.15.27:0/2459340951 >> v2:172.21.15.93:3300/0 conn(0x7f25640ce760 0x7f25640b2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.805+0000 7f2569600640 1 -- 172.21.15.27:0/2459340951 >> v1:172.21.15.93:6789/0 conn(0x7f25640b1db0 legacy=0x7f25640ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:38.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.805+0000 7f2569600640 1 -- 172.21.15.27:0/2459340951 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25640cd850 con 0x7f256406c430 2024-01-15T18:23:38.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.805+0000 7f256a000640 1 --2- 172.21.15.27:0/2459340951 >> v2:172.21.15.93:3300/0 conn(0x7f25640ce760 0x7f25640b2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:23:38.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.809+0000 7f2569600640 1 --2- 172.21.15.27:0/2459340951 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f256406c430 0x7f2564070ba0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f25540097c0 tx=0x7f255401c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b110965fabc118eb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:38.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.809+0000 7f2563e00640 1 -- 172.21.15.27:0/2459340951 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25540ab600 con 0x7f256406c430 2024-01-15T18:23:38.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.809+0000 7f2563e00640 1 -- 172.21.15.27:0/2459340951 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25540a7050 con 0x7f256406c430 2024-01-15T18:23:38.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.809+0000 7f2563e00640 1 -- 172.21.15.27:0/2459340951 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25540ab980 con 0x7f256406c430 2024-01-15T18:23:38.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.809+0000 7f256c000640 1 -- 172.21.15.27:0/2459340951 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f256406c430 msgr2=0x7f2564070ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:38.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.809+0000 7f256c000640 1 --2- 172.21.15.27:0/2459340951 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f256406c430 0x7f2564070ba0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f25540097c0 tx=0x7f255401c3f0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.809+0000 7f256c000640 1 -- 172.21.15.27:0/2459340951 shutdown_connections 2024-01-15T18:23:38.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.809+0000 7f256c000640 1 --2- 172.21.15.27:0/2459340951 >> v2:172.21.15.93:3300/0 conn(0x7f25640ce760 0x7f25640b2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.809+0000 7f256c000640 1 --2- 172.21.15.27:0/2459340951 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f256406c430 0x7f2564070ba0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.809+0000 7f256c000640 1 -- 172.21.15.27:0/2459340951 >> 172.21.15.27:0/2459340951 conn(0x7f25640cb700 msgr2=0x7f25640cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:38.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.809+0000 7f256c000640 1 -- 172.21.15.27:0/2459340951 shutdown_connections 2024-01-15T18:23:38.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.809+0000 7f256c000640 1 -- 172.21.15.27:0/2459340951 wait complete. 2024-01-15T18:23:38.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.813+0000 7f256c000640 1 Processor -- start 2024-01-15T18:23:38.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.813+0000 7f256c000640 1 -- start start 2024-01-15T18:23:38.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.813+0000 7f256c000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f256406c430 0x7f25640ade50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:38.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.813+0000 7f256c000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f25640b1db0 0x7f2564159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:38.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.813+0000 7f256c000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f25640ce760 0x7f25641440d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:38.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.813+0000 7f256c000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256414e9a0 con 0x7f256406c430 2024-01-15T18:23:38.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.813+0000 7f256c000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2564134800 con 0x7f25640b1db0 2024-01-15T18:23:38.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.813+0000 7f256c000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2564134ae0 con 0x7f25640ce760 2024-01-15T18:23:38.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.813+0000 7f2569600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f256406c430 0x7f25640ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:38.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.813+0000 7f2569600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f256406c430 0x7f25640ade50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47626/0 (socket says 172.21.15.27:47626) 2024-01-15T18:23:38.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.813+0000 7f2569600640 1 -- 172.21.15.27:0/2864130285 learned_addr learned my addr 172.21.15.27:0/2864130285 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:38.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.813+0000 7f256a000640 1 --2- 172.21.15.27:0/2864130285 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f25640ce760 0x7f25641440d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:38.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.813+0000 7f2568c00640 1 --2- 172.21.15.27:0/2864130285 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f25640b1db0 0x7f2564159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:38.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.813+0000 7f2569600640 1 -- 172.21.15.27:0/2864130285 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f25640ce760 msgr2=0x7f25641440d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:38.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.813+0000 7f2569600640 1 --2- 172.21.15.27:0/2864130285 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f25640ce760 0x7f25641440d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.813+0000 7f2569600640 1 -- 172.21.15.27:0/2864130285 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f25640b1db0 msgr2=0x7f2564159d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:38.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.813+0000 7f2569600640 1 --2- 172.21.15.27:0/2864130285 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f25640b1db0 0x7f2564159d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.813+0000 7f2569600640 1 -- 172.21.15.27:0/2864130285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f255c00ac40 con 0x7f256406c430 2024-01-15T18:23:38.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.813+0000 7f2568c00640 1 --2- 172.21.15.27:0/2864130285 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f25640b1db0 0x7f2564159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:23:38.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.813+0000 7f2569600640 1 --2- 172.21.15.27:0/2864130285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f256406c430 0x7f25640ade50 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f255401a020 tx=0x7f255401ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:38.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.813+0000 7f2561600640 1 -- 172.21.15.27:0/2864130285 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25540b7450 con 0x7f256406c430 2024-01-15T18:23:38.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.813+0000 7f256c000640 1 -- 172.21.15.27:0/2864130285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2554009420 con 0x7f256406c430 2024-01-15T18:23:38.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.813+0000 7f2561600640 1 -- 172.21.15.27:0/2864130285 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25540b75b0 con 0x7f256406c430 2024-01-15T18:23:38.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.813+0000 7f2561600640 1 -- 172.21.15.27:0/2864130285 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25540b7960 con 0x7f256406c430 2024-01-15T18:23:38.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.813+0000 7f256c000640 1 -- 172.21.15.27:0/2864130285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25640712f0 con 0x7f256406c430 2024-01-15T18:23:38.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.817+0000 7f256c000640 1 -- 172.21.15.27:0/2864130285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f256404e850 con 0x7f256406c430 2024-01-15T18:23:38.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.817+0000 7f2561600640 1 -- 172.21.15.27:0/2864130285 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f25540b7ac0 con 0x7f256406c430 2024-01-15T18:23:38.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.817+0000 7f2561600640 1 --2- 172.21.15.27:0/2864130285 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f253c067a90 0x7f253c069f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:38.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.817+0000 7f2561600640 1 -- 172.21.15.27:0/2864130285 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f255411db40 con 0x7f256406c430 2024-01-15T18:23:38.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.817+0000 7f2561600640 1 --2- 172.21.15.27:0/2864130285 >> [v2:172.21.15.27:6816/934967897,v1:172.21.15.27:6817/934967897] conn(0x7f253c06fe90 0x7f253c072330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:38.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.817+0000 7f2561600640 1 -- 172.21.15.27:0/2864130285 --> [v2:172.21.15.27:6816/934967897,v1:172.21.15.27:6817/934967897] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f253c072a00 con 0x7f253c06fe90 2024-01-15T18:23:38.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.817+0000 7f2561600640 1 -- 172.21.15.27:0/2864130285 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7f255411e4e0 con 0x7f256406c430 2024-01-15T18:23:38.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.817+0000 7f2568c00640 1 --2- 172.21.15.27:0/2864130285 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f253c067a90 0x7f253c069f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:38.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.817+0000 7f256a000640 1 --2- 172.21.15.27:0/2864130285 >> [v2:172.21.15.27:6816/934967897,v1:172.21.15.27:6817/934967897] conn(0x7f253c06fe90 0x7f253c072330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:38.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.817+0000 7f2568c00640 1 --2- 172.21.15.27:0/2864130285 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f253c067a90 0x7f253c069f50 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f256406e640 tx=0x7f254c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:38.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.817+0000 7f256a000640 1 --2- 172.21.15.27:0/2864130285 >> [v2:172.21.15.27:6816/934967897,v1:172.21.15.27:6817/934967897] conn(0x7f253c06fe90 0x7f253c072330 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:38.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.821+0000 7f2561600640 1 -- 172.21.15.27:0/2864130285 <== osd.3 v2:172.21.15.27:6816/934967897 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f253c072a00 con 0x7f253c06fe90 2024-01-15T18:23:38.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.833+0000 7f256c000640 1 -- 172.21.15.27:0/2864130285 --> [v2:172.21.15.27:6816/934967897,v1:172.21.15.27:6817/934967897] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f256415ced0 con 0x7f253c06fe90 2024-01-15T18:23:38.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.833+0000 7f2561600640 1 -- 172.21.15.27:0/2864130285 <== osd.3 v2:172.21.15.27:6816/934967897 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f256415ced0 con 0x7f253c06fe90 2024-01-15T18:23:38.830 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:23:38.830 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-15T18:23:38.830 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:23:38.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.833+0000 7f256c000640 1 -- 172.21.15.27:0/2864130285 >> [v2:172.21.15.27:6816/934967897,v1:172.21.15.27:6817/934967897] conn(0x7f253c06fe90 msgr2=0x7f253c072330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:38.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.833+0000 7f256c000640 1 --2- 172.21.15.27:0/2864130285 >> [v2:172.21.15.27:6816/934967897,v1:172.21.15.27:6817/934967897] conn(0x7f253c06fe90 0x7f253c072330 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.833+0000 7f256c000640 1 -- 172.21.15.27:0/2864130285 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f253c067a90 msgr2=0x7f253c069f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:38.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.833+0000 7f256c000640 1 --2- 172.21.15.27:0/2864130285 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f253c067a90 0x7f253c069f50 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f256406e640 tx=0x7f254c009040 comp rx=0 tx=0).stop 2024-01-15T18:23:38.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.833+0000 7f256c000640 1 -- 172.21.15.27:0/2864130285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f256406c430 msgr2=0x7f25640ade50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:38.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.833+0000 7f256c000640 1 --2- 172.21.15.27:0/2864130285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f256406c430 0x7f25640ade50 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f255401a020 tx=0x7f255401ac30 comp rx=0 tx=0).stop 2024-01-15T18:23:38.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.833+0000 7f256a000640 1 -- 172.21.15.27:0/2864130285 reap_dead start 2024-01-15T18:23:38.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.833+0000 7f256c000640 1 -- 172.21.15.27:0/2864130285 shutdown_connections 2024-01-15T18:23:38.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.833+0000 7f256c000640 1 -- 172.21.15.27:0/2864130285 >> 172.21.15.27:0/2864130285 conn(0x7f25640cb700 msgr2=0x7f25640b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:38.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.833+0000 7f256c000640 1 -- 172.21.15.27:0/2864130285 shutdown_connections 2024-01-15T18:23:38.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.833+0000 7f256c000640 1 -- 172.21.15.27:0/2864130285 wait complete. 2024-01-15T18:23:38.848 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-01-15T18:23:38.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.997+0000 7f7310c00640 1 Processor -- start 2024-01-15T18:23:38.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.997+0000 7f7310c00640 1 -- start start 2024-01-15T18:23:38.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.997+0000 7f7310c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f730c071020 0x7f730c071460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:38.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.997+0000 7f7310c00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f730c06c6d0 0x7f730c06cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:38.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.997+0000 7f7310c00640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f730c070b90 con 0x7f730c15ac50 2024-01-15T18:23:38.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.997+0000 7f7310c00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f730c072020 con 0x7f730c06c6d0 2024-01-15T18:23:38.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.997+0000 7f7310c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f730c072300 con 0x7f730c071020 2024-01-15T18:23:38.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.997+0000 7f7309e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f730c15ac50 0x7f730c0b2660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42934/0 (socket says 172.21.15.27:42934) 2024-01-15T18:23:38.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.997+0000 7f7309e00640 1 -- 172.21.15.27:0/450969757 learned_addr learned my addr 172.21.15.27:0/450969757 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:38.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.997+0000 7f730a800640 1 --2- 172.21.15.27:0/450969757 >> v2:172.21.15.93:3300/0 conn(0x7f730c06c6d0 0x7f730c06cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:38.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.997+0000 7f7309400640 1 --2- 172.21.15.27:0/450969757 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f730c071020 0x7f730c071460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:38.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.997+0000 7f7308a00640 1 -- 172.21.15.27:0/450969757 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647537398 0 0) 0x7f730c070b90 con 0x7f730c15ac50 2024-01-15T18:23:38.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.997+0000 7f7308a00640 1 -- 172.21.15.27:0/450969757 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72ec003530 con 0x7f730c15ac50 2024-01-15T18:23:38.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.997+0000 7f7309400640 1 -- 172.21.15.27:0/450969757 >> v2:172.21.15.93:3300/0 conn(0x7f730c06c6d0 msgr2=0x7f730c06cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:38.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.997+0000 7f7309400640 1 --2- 172.21.15.27:0/450969757 >> v2:172.21.15.93:3300/0 conn(0x7f730c06c6d0 0x7f730c06cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:38.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.997+0000 7f7309400640 1 -- 172.21.15.27:0/450969757 >> v1:172.21.15.27:6789/0 conn(0x7f730c15ac50 legacy=0x7f730c0b2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:38.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.997+0000 7f7309400640 1 -- 172.21.15.27:0/450969757 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c070820 con 0x7f730c071020 2024-01-15T18:23:38.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:38.997+0000 7f730a800640 1 --2- 172.21.15.27:0/450969757 >> v2:172.21.15.93:3300/0 conn(0x7f730c06c6d0 0x7f730c06cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:23:38.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.001+0000 7f7309400640 1 --2- 172.21.15.27:0/450969757 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f730c071020 0x7f730c071460 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f72f40097c0 tx=0x7f72f401c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f44eaf4662738a9d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:38.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.001+0000 7f7308a00640 1 -- 172.21.15.27:0/450969757 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f40a6040 con 0x7f730c071020 2024-01-15T18:23:38.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.001+0000 7f7308a00640 1 -- 172.21.15.27:0/450969757 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f40aa600 con 0x7f730c071020 2024-01-15T18:23:38.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.001+0000 7f7308a00640 1 -- 172.21.15.27:0/450969757 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f40aa990 con 0x7f730c071020 2024-01-15T18:23:38.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.001+0000 7f7310c00640 1 -- 172.21.15.27:0/450969757 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f730c071020 msgr2=0x7f730c071460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:38.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.001+0000 7f7310c00640 1 --2- 172.21.15.27:0/450969757 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f730c071020 0x7f730c071460 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f72f40097c0 tx=0x7f72f401c480 comp rx=0 tx=0).stop 2024-01-15T18:23:38.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.001+0000 7f7310c00640 1 -- 172.21.15.27:0/450969757 shutdown_connections 2024-01-15T18:23:38.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.001+0000 7f7310c00640 1 --2- 172.21.15.27:0/450969757 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f730c071020 0x7f730c071460 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.001+0000 7f7310c00640 1 --2- 172.21.15.27:0/450969757 >> v2:172.21.15.93:3300/0 conn(0x7f730c06c6d0 0x7f730c06cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.001+0000 7f7310c00640 1 -- 172.21.15.27:0/450969757 >> 172.21.15.27:0/450969757 conn(0x7f730c0b0850 msgr2=0x7f730c0b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.001+0000 7f7310c00640 1 -- 172.21.15.27:0/450969757 shutdown_connections 2024-01-15T18:23:39.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.001+0000 7f7310c00640 1 -- 172.21.15.27:0/450969757 wait complete. 2024-01-15T18:23:39.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.005+0000 7f7310c00640 1 Processor -- start 2024-01-15T18:23:39.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.005+0000 7f7310c00640 1 -- start start 2024-01-15T18:23:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.005+0000 7f7310c00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f730c06c6d0 0x7f730c14dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.005+0000 7f7310c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c071020 0x7f730c0ac880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.005+0000 7f7310c00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f730c15ac50 0x7f730c0acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.005+0000 7f7310c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c14f620 con 0x7f730c071020 2024-01-15T18:23:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.005+0000 7f7310c00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c14e1f0 con 0x7f730c06c6d0 2024-01-15T18:23:39.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.005+0000 7f7310c00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f730c14e9a0 con 0x7f730c15ac50 2024-01-15T18:23:39.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.005+0000 7f7309400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c071020 0x7f730c0ac880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.005+0000 7f7309400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c071020 0x7f730c0ac880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47632/0 (socket says 172.21.15.27:47632) 2024-01-15T18:23:39.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.005+0000 7f7309400640 1 -- 172.21.15.27:0/9811660 learned_addr learned my addr 172.21.15.27:0/9811660 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:39.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.005+0000 7f7309e00640 1 --2- 172.21.15.27:0/9811660 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f730c06c6d0 0x7f730c14dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.005+0000 7f730a800640 1 --2- 172.21.15.27:0/9811660 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f730c15ac50 0x7f730c0acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.005+0000 7f7309400640 1 -- 172.21.15.27:0/9811660 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f730c15ac50 msgr2=0x7f730c0acdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:39.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.005+0000 7f7309400640 1 --2- 172.21.15.27:0/9811660 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f730c15ac50 0x7f730c0acdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.005+0000 7f7309400640 1 -- 172.21.15.27:0/9811660 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f730c06c6d0 msgr2=0x7f730c14dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:39.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.005+0000 7f7309400640 1 --2- 172.21.15.27:0/9811660 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f730c06c6d0 0x7f730c14dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.005+0000 7f7309400640 1 -- 172.21.15.27:0/9811660 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7304009a70 con 0x7f730c071020 2024-01-15T18:23:39.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.005+0000 7f7309400640 1 --2- 172.21.15.27:0/9811660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c071020 0x7f730c0ac880 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f72f401a0b0 tx=0x7f72f401acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:39.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.009+0000 7f7302000640 1 -- 172.21.15.27:0/9811660 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f4006e20 con 0x7f730c071020 2024-01-15T18:23:39.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.009+0000 7f7302000640 1 -- 172.21.15.27:0/9811660 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f40b3600 con 0x7f730c071020 2024-01-15T18:23:39.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.009+0000 7f7302000640 1 -- 172.21.15.27:0/9811660 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f40b3990 con 0x7f730c071020 2024-01-15T18:23:39.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.009+0000 7f7310c00640 1 -- 172.21.15.27:0/9811660 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f4009420 con 0x7f730c071020 2024-01-15T18:23:39.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.009+0000 7f7310c00640 1 -- 172.21.15.27:0/9811660 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f730c14ef60 con 0x7f730c071020 2024-01-15T18:23:39.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.009+0000 7f7310c00640 1 -- 172.21.15.27:0/9811660 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f730c14f760 con 0x7f730c071020 2024-01-15T18:23:39.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.009+0000 7f7302000640 1 -- 172.21.15.27:0/9811660 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f72f40b3af0 con 0x7f730c071020 2024-01-15T18:23:39.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.009+0000 7f7302000640 1 --2- 172.21.15.27:0/9811660 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f72e4067a60 0x7f72e4069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.009+0000 7f7302000640 1 -- 172.21.15.27:0/9811660 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f72f411d330 con 0x7f730c071020 2024-01-15T18:23:39.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.009+0000 7f7309e00640 1 --2- 172.21.15.27:0/9811660 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f72e4067a60 0x7f72e4069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.013+0000 7f7309e00640 1 --2- 172.21.15.27:0/9811660 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f72e4067a60 0x7f72e4069f20 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f72fc0011f0 tx=0x7f72fc007d40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:39.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.013+0000 7f7302000640 1 -- 172.21.15.27:0/9811660 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f72f40ebd20 con 0x7f730c071020 2024-01-15T18:23:39.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.173+0000 7f7310c00640 1 -- 172.21.15.27:0/9811660 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f730c06e2b0 con 0x7f72e4067a60 2024-01-15T18:23:39.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.173+0000 7f7302000640 1 -- 172.21.15.27:0/9811660 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f730c06e2b0 con 0x7f72e4067a60 2024-01-15T18:23:39.169 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 3 to deep-scrub 2024-01-15T18:23:39.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.173+0000 7f7310c00640 1 -- 172.21.15.27:0/9811660 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f72e4067a60 msgr2=0x7f72e4069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:39.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.173+0000 7f7310c00640 1 --2- 172.21.15.27:0/9811660 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f72e4067a60 0x7f72e4069f20 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f72fc0011f0 tx=0x7f72fc007d40 comp rx=0 tx=0).stop 2024-01-15T18:23:39.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.173+0000 7f7310c00640 1 -- 172.21.15.27:0/9811660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c071020 msgr2=0x7f730c0ac880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:39.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.173+0000 7f7310c00640 1 --2- 172.21.15.27:0/9811660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c071020 0x7f730c0ac880 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f72f401a0b0 tx=0x7f72f401acc0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.173+0000 7f7310c00640 1 -- 172.21.15.27:0/9811660 shutdown_connections 2024-01-15T18:23:39.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.173+0000 7f7310c00640 1 --2- 172.21.15.27:0/9811660 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f72e4067a60 0x7f72e4069f20 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.173+0000 7f7310c00640 1 --2- 172.21.15.27:0/9811660 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f730c15ac50 0x7f730c0acdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.173+0000 7f7310c00640 1 --2- 172.21.15.27:0/9811660 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c071020 0x7f730c0ac880 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.173+0000 7f7310c00640 1 --2- 172.21.15.27:0/9811660 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f730c06c6d0 0x7f730c14dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.177+0000 7f7310c00640 1 -- 172.21.15.27:0/9811660 >> 172.21.15.27:0/9811660 conn(0x7f730c0b0850 msgr2=0x7f730c0ad930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:39.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.177+0000 7f7310c00640 1 -- 172.21.15.27:0/9811660 shutdown_connections 2024-01-15T18:23:39.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.177+0000 7f7310c00640 1 -- 172.21.15.27:0/9811660 wait complete. 2024-01-15T18:23:39.192 INFO:tasks.ceph:Scrubbing osd.4 2024-01-15T18:23:39.192 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-01-15T18:23:39.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.341+0000 7f74bf600640 1 Processor -- start 2024-01-15T18:23:39.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.341+0000 7f74bf600640 1 -- start start 2024-01-15T18:23:39.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.341+0000 7f74bf600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f74b8158900 0x7f74b815f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.341+0000 7f74bf600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f74b80cf2e0 0x7f74b80701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.341+0000 7f74bf600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b80b1cb0 con 0x7f74b815bf00 2024-01-15T18:23:39.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.341+0000 7f74bf600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74b80b1f00 con 0x7f74b8158900 2024-01-15T18:23:39.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.341+0000 7f74bf600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74b80b21e0 con 0x7f74b80cf2e0 2024-01-15T18:23:39.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.341+0000 7f74b7e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f74b815bf00 0x7f74b80cebd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42942/0 (socket says 172.21.15.27:42942) 2024-01-15T18:23:39.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.341+0000 7f74b7e00640 1 -- 172.21.15.27:0/2442220529 learned_addr learned my addr 172.21.15.27:0/2442220529 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:39.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.341+0000 7f74bcc00640 1 --2- 172.21.15.27:0/2442220529 >> v2:172.21.15.27:3300/0 conn(0x7f74b8158900 0x7f74b815f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.341+0000 7f74bd600640 1 --2- 172.21.15.27:0/2442220529 >> v2:172.21.15.93:3300/0 conn(0x7f74b80cf2e0 0x7f74b80701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.345+0000 7f74b7400640 1 -- 172.21.15.27:0/2442220529 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215770072 0 0) 0x7f74b80b1cb0 con 0x7f74b815bf00 2024-01-15T18:23:39.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.345+0000 7f74bcc00640 1 -- 172.21.15.27:0/2442220529 >> v2:172.21.15.93:3300/0 conn(0x7f74b80cf2e0 msgr2=0x7f74b80701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:39.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.345+0000 7f74bcc00640 1 --2- 172.21.15.27:0/2442220529 >> v2:172.21.15.93:3300/0 conn(0x7f74b80cf2e0 0x7f74b80701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.345+0000 7f74bcc00640 1 -- 172.21.15.27:0/2442220529 >> v1:172.21.15.27:6789/0 conn(0x7f74b815bf00 legacy=0x7f74b80cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:39.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.345+0000 7f74bcc00640 1 -- 172.21.15.27:0/2442220529 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b815ea70 con 0x7f74b8158900 2024-01-15T18:23:39.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.345+0000 7f74bcc00640 1 --2- 172.21.15.27:0/2442220529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b8158900 0x7f74b815f550 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f74b80b30c0 tx=0x7f74a801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef064b10d03f2d3b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:39.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.345+0000 7f74b7400640 1 -- 172.21.15.27:0/2442220529 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b8070f30 con 0x7f74b8158900 2024-01-15T18:23:39.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.345+0000 7f74b7400640 1 -- 172.21.15.27:0/2442220529 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a80a7050 con 0x7f74b8158900 2024-01-15T18:23:39.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.345+0000 7f74b7400640 1 -- 172.21.15.27:0/2442220529 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a8004450 con 0x7f74b8158900 2024-01-15T18:23:39.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.345+0000 7f74bf600640 1 -- 172.21.15.27:0/2442220529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b8158900 msgr2=0x7f74b815f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:39.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.345+0000 7f74bf600640 1 --2- 172.21.15.27:0/2442220529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b8158900 0x7f74b815f550 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f74b80b30c0 tx=0x7f74a801c460 comp rx=0 tx=0).stop 2024-01-15T18:23:39.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.349+0000 7f74bf600640 1 -- 172.21.15.27:0/2442220529 shutdown_connections 2024-01-15T18:23:39.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.349+0000 7f74bf600640 1 --2- 172.21.15.27:0/2442220529 >> v2:172.21.15.93:3300/0 conn(0x7f74b80cf2e0 0x7f74b80701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.349+0000 7f74bf600640 1 --2- 172.21.15.27:0/2442220529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b8158900 0x7f74b815f550 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.349+0000 7f74bf600640 1 -- 172.21.15.27:0/2442220529 >> 172.21.15.27:0/2442220529 conn(0x7f74b80cb700 msgr2=0x7f74b80cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:39.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.349+0000 7f74bf600640 1 -- 172.21.15.27:0/2442220529 shutdown_connections 2024-01-15T18:23:39.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.349+0000 7f74bf600640 1 -- 172.21.15.27:0/2442220529 wait complete. 2024-01-15T18:23:39.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.349+0000 7f74bf600640 1 Processor -- start 2024-01-15T18:23:39.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.349+0000 7f74bf600640 1 -- start start 2024-01-15T18:23:39.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.349+0000 7f74bf600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f74b80cf2e0 0x7f74b814aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.349+0000 7f74bf600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b8158900 0x7f74b814a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.349+0000 7f74bf600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f74b815bf00 0x7f74b8151240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.349+0000 7f74bf600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b80aeec0 con 0x7f74b8158900 2024-01-15T18:23:39.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.349+0000 7f74bf600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b80aebe0 con 0x7f74b815bf00 2024-01-15T18:23:39.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.349+0000 7f74bf600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b80adb50 con 0x7f74b80cf2e0 2024-01-15T18:23:39.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.349+0000 7f74b7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b8158900 0x7f74b814a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.349+0000 7f74b7e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b8158900 0x7f74b814a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47648/0 (socket says 172.21.15.27:47648) 2024-01-15T18:23:39.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.349+0000 7f74b7e00640 1 -- 172.21.15.27:0/756536278 learned_addr learned my addr 172.21.15.27:0/756536278 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:39.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.349+0000 7f74bcc00640 1 --2- 172.21.15.27:0/756536278 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f74b80cf2e0 0x7f74b814aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.349+0000 7f74bd600640 1 --2- 172.21.15.27:0/756536278 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f74b815bf00 0x7f74b8151240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.349+0000 7f74b7e00640 1 -- 172.21.15.27:0/756536278 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f74b80cf2e0 msgr2=0x7f74b814aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:39.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.349+0000 7f74b7e00640 1 --2- 172.21.15.27:0/756536278 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f74b80cf2e0 0x7f74b814aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.349+0000 7f74b7e00640 1 -- 172.21.15.27:0/756536278 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f74b815bf00 msgr2=0x7f74b8151240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:39.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.349+0000 7f74b7e00640 1 --2- 172.21.15.27:0/756536278 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f74b815bf00 0x7f74b8151240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.349+0000 7f74b7e00640 1 -- 172.21.15.27:0/756536278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a8009420 con 0x7f74b8158900 2024-01-15T18:23:39.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.353+0000 7f74b7e00640 1 --2- 172.21.15.27:0/756536278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b8158900 0x7f74b814a210 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f74a000c480 tx=0x7f74a000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:39.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.353+0000 7f74b4c00640 1 -- 172.21.15.27:0/756536278 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a00074e0 con 0x7f74b8158900 2024-01-15T18:23:39.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.353+0000 7f74b4c00640 1 -- 172.21.15.27:0/756536278 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a0007660 con 0x7f74b8158900 2024-01-15T18:23:39.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.353+0000 7f74bf600640 1 -- 172.21.15.27:0/756536278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b80ade30 con 0x7f74b8158900 2024-01-15T18:23:39.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.353+0000 7f74b4c00640 1 -- 172.21.15.27:0/756536278 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a0004a30 con 0x7f74b8158900 2024-01-15T18:23:39.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.353+0000 7f74bf600640 1 -- 172.21.15.27:0/756536278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b814a750 con 0x7f74b8158900 2024-01-15T18:23:39.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.353+0000 7f74bf600640 1 -- 172.21.15.27:0/756536278 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f74b806e170 con 0x7f74b8158900 2024-01-15T18:23:39.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.353+0000 7f74b4c00640 1 -- 172.21.15.27:0/756536278 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f74a0004b90 con 0x7f74b8158900 2024-01-15T18:23:39.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.353+0000 7f74b4c00640 1 --2- 172.21.15.27:0/756536278 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7490067a60 0x7f7490069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.353+0000 7f74b4c00640 1 -- 172.21.15.27:0/756536278 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f74a00827b0 con 0x7f74b8158900 2024-01-15T18:23:39.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.353+0000 7f74b4c00640 1 --2- 172.21.15.27:0/756536278 >> [v2:172.21.15.93:6803/1254182834,v1:172.21.15.93:6806/1254182834] conn(0x7f749006fe60 0x7f7490072300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.353+0000 7f74b4c00640 1 -- 172.21.15.27:0/756536278 --> [v2:172.21.15.93:6803/1254182834,v1:172.21.15.93:6806/1254182834] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f74900729d0 con 0x7f749006fe60 2024-01-15T18:23:39.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.353+0000 7f74b4c00640 1 -- 172.21.15.27:0/756536278 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74a0083340 con 0x7f74b8158900 2024-01-15T18:23:39.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.353+0000 7f74bcc00640 1 --2- 172.21.15.27:0/756536278 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7490067a60 0x7f7490069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.357+0000 7f74bd600640 1 --2- 172.21.15.27:0/756536278 >> [v2:172.21.15.93:6803/1254182834,v1:172.21.15.93:6806/1254182834] conn(0x7f749006fe60 0x7f7490072300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.357+0000 7f74bcc00640 1 --2- 172.21.15.27:0/756536278 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7490067a60 0x7f7490069f20 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f74a80ad000 tx=0x7f74a80a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:39.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.357+0000 7f74bd600640 1 --2- 172.21.15.27:0/756536278 >> [v2:172.21.15.93:6803/1254182834,v1:172.21.15.93:6806/1254182834] conn(0x7f749006fe60 0x7f7490072300 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:39.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.357+0000 7f74b4c00640 1 -- 172.21.15.27:0/756536278 <== osd.4 v2:172.21.15.93:6803/1254182834 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f74900729d0 con 0x7f749006fe60 2024-01-15T18:23:39.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.369+0000 7f74bf600640 1 -- 172.21.15.27:0/756536278 --> [v2:172.21.15.93:6803/1254182834,v1:172.21.15.93:6806/1254182834] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f74b80ad9b0 con 0x7f749006fe60 2024-01-15T18:23:39.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.369+0000 7f74b4c00640 1 -- 172.21.15.27:0/756536278 <== osd.4 v2:172.21.15.93:6803/1254182834 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f74b80ad9b0 con 0x7f749006fe60 2024-01-15T18:23:39.367 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:23:39.367 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-15T18:23:39.367 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:23:39.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.369+0000 7f74bf600640 1 -- 172.21.15.27:0/756536278 >> [v2:172.21.15.93:6803/1254182834,v1:172.21.15.93:6806/1254182834] conn(0x7f749006fe60 msgr2=0x7f7490072300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:39.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.369+0000 7f74bf600640 1 --2- 172.21.15.27:0/756536278 >> [v2:172.21.15.93:6803/1254182834,v1:172.21.15.93:6806/1254182834] conn(0x7f749006fe60 0x7f7490072300 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.369+0000 7f74bf600640 1 -- 172.21.15.27:0/756536278 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7490067a60 msgr2=0x7f7490069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:39.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.369+0000 7f74bf600640 1 --2- 172.21.15.27:0/756536278 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f7490067a60 0x7f7490069f20 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f74a80ad000 tx=0x7f74a80a6000 comp rx=0 tx=0).stop 2024-01-15T18:23:39.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.369+0000 7f74bf600640 1 -- 172.21.15.27:0/756536278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b8158900 msgr2=0x7f74b814a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:39.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.369+0000 7f74bf600640 1 --2- 172.21.15.27:0/756536278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74b8158900 0x7f74b814a210 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f74a000c480 tx=0x7f74a000c940 comp rx=0 tx=0).stop 2024-01-15T18:23:39.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.369+0000 7f74bd600640 1 -- 172.21.15.27:0/756536278 reap_dead start 2024-01-15T18:23:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.369+0000 7f74bf600640 1 -- 172.21.15.27:0/756536278 shutdown_connections 2024-01-15T18:23:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.369+0000 7f74bf600640 1 -- 172.21.15.27:0/756536278 >> 172.21.15.27:0/756536278 conn(0x7f74b80cb700 msgr2=0x7f74b80718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.373+0000 7f74bf600640 1 -- 172.21.15.27:0/756536278 shutdown_connections 2024-01-15T18:23:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.373+0000 7f74bf600640 1 -- 172.21.15.27:0/756536278 wait complete. 2024-01-15T18:23:39.386 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-01-15T18:23:39.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2d0400640 1 Processor -- start 2024-01-15T18:23:39.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2d0400640 1 -- start start 2024-01-15T18:23:39.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2d0400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7ff2c815b180 0x7ff2c80cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2d0400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2c80cd7e0 0x7ff2c80cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2d0400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2c80ce180 con 0x7ff2c80b09d0 2024-01-15T18:23:39.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2d0400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2c80d1e20 con 0x7ff2c815b180 2024-01-15T18:23:39.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2d0400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c80ceed0 con 0x7ff2c80cd7e0 2024-01-15T18:23:39.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2ce400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff2c80b09d0 0x7ff2c80b0e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42958/0 (socket says 172.21.15.27:42958) 2024-01-15T18:23:39.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2ce400640 1 -- 172.21.15.27:0/1412942091 learned_addr learned my addr 172.21.15.27:0/1412942091 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:39.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2cda00640 1 --2- 172.21.15.27:0/1412942091 >> v2:172.21.15.93:3300/0 conn(0x7ff2c815b180 0x7ff2c80cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2cd000640 1 --2- 172.21.15.27:0/1412942091 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2c80cd7e0 0x7ff2c80cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2c7e00640 1 -- 172.21.15.27:0/1412942091 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741067276 0 0) 0x7ff2c80ce180 con 0x7ff2c80b09d0 2024-01-15T18:23:39.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2c7e00640 1 -- 172.21.15.27:0/1412942091 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2a8003530 con 0x7ff2c80b09d0 2024-01-15T18:23:39.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2c7e00640 1 -- 172.21.15.27:0/1412942091 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 731423843 0 0) 0x7ff2a8003530 con 0x7ff2c80b09d0 2024-01-15T18:23:39.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2c7e00640 1 -- 172.21.15.27:0/1412942091 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2c80cd7e0 msgr2=0x7ff2c80cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:39.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2c7e00640 1 --2- 172.21.15.27:0/1412942091 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2c80cd7e0 0x7ff2c80cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2c7e00640 1 -- 172.21.15.27:0/1412942091 >> v2:172.21.15.93:3300/0 conn(0x7ff2c815b180 msgr2=0x7ff2c80cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:39.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2c7e00640 1 --2- 172.21.15.27:0/1412942091 >> v2:172.21.15.93:3300/0 conn(0x7ff2c815b180 0x7ff2c80cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2c7e00640 1 -- 172.21.15.27:0/1412942091 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c80ce7d0 con 0x7ff2c80b09d0 2024-01-15T18:23:39.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2c7e00640 1 -- 172.21.15.27:0/1412942091 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7ff2c0001a00 con 0x7ff2c80b09d0 2024-01-15T18:23:39.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2c7e00640 1 -- 172.21.15.27:0/1412942091 >> v1:172.21.15.27:6789/0 conn(0x7ff2c80b09d0 legacy=0x7ff2c80b0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:39.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2c7e00640 1 --2- 172.21.15.27:0/1412942091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2a80039f0 0x7ff2a8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2c7e00640 1 --2- 172.21.15.27:0/1412942091 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2a80047d0 0x7ff2a8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2c7e00640 1 --2- 172.21.15.27:0/1412942091 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2c80cd7e0 0x7ff2a8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2c7e00640 1 -- 172.21.15.27:0/1412942091 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c80d1e20 con 0x7ff2a80039f0 2024-01-15T18:23:39.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2c7e00640 1 -- 172.21.15.27:0/1412942091 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c80ceed0 con 0x7ff2a80047d0 2024-01-15T18:23:39.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2c7e00640 1 -- 172.21.15.27:0/1412942091 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a800b990 con 0x7ff2c80cd7e0 2024-01-15T18:23:39.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2cda00640 1 --2- 172.21.15.27:0/1412942091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2a80039f0 0x7ff2a8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2cd000640 1 --2- 172.21.15.27:0/1412942091 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2a80047d0 0x7ff2a8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2ce400640 1 --2- 172.21.15.27:0/1412942091 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2c80cd7e0 0x7ff2a8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2cda00640 1 -- 172.21.15.27:0/1412942091 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2c80cd7e0 msgr2=0x7ff2a8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:39.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2cda00640 1 --2- 172.21.15.27:0/1412942091 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2c80cd7e0 0x7ff2a8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2cda00640 1 -- 172.21.15.27:0/1412942091 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2a80047d0 msgr2=0x7ff2a8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:39.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2cda00640 1 --2- 172.21.15.27:0/1412942091 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2a80047d0 0x7ff2a8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.533+0000 7ff2cda00640 1 -- 172.21.15.27:0/1412942091 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a8003530 con 0x7ff2a80039f0 2024-01-15T18:23:39.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.537+0000 7ff2cda00640 1 --2- 172.21.15.27:0/1412942091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2a80039f0 0x7ff2a8003e10 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8009b10 tx=0x7ff2b801ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d14f735d29035712 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:39.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.537+0000 7ff2c7e00640 1 -- 172.21.15.27:0/1412942091 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b80a6470 con 0x7ff2a80039f0 2024-01-15T18:23:39.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.537+0000 7ff2c7e00640 1 -- 172.21.15.27:0/1412942091 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b80a65d0 con 0x7ff2a80039f0 2024-01-15T18:23:39.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.537+0000 7ff2c7e00640 1 -- 172.21.15.27:0/1412942091 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b80a6980 con 0x7ff2a80039f0 2024-01-15T18:23:39.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.537+0000 7ff2d0400640 1 -- 172.21.15.27:0/1412942091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2a80039f0 msgr2=0x7ff2a8003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:39.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.537+0000 7ff2d0400640 1 --2- 172.21.15.27:0/1412942091 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2a80039f0 0x7ff2a8003e10 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8009b10 tx=0x7ff2b801ca60 comp rx=0 tx=0).stop 2024-01-15T18:23:39.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.537+0000 7ff2ce400640 1 -- 172.21.15.27:0/1412942091 reap_dead start 2024-01-15T18:23:39.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.537+0000 7ff2d0400640 1 -- 172.21.15.27:0/1412942091 shutdown_connections 2024-01-15T18:23:39.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.537+0000 7ff2d0400640 1 -- 172.21.15.27:0/1412942091 >> 172.21.15.27:0/1412942091 conn(0x7ff2c80d0e80 msgr2=0x7ff2c80d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:39.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.537+0000 7ff2d0400640 1 -- 172.21.15.27:0/1412942091 shutdown_connections 2024-01-15T18:23:39.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.537+0000 7ff2d0400640 1 -- 172.21.15.27:0/1412942091 wait complete. 2024-01-15T18:23:39.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2d0400640 1 Processor -- start 2024-01-15T18:23:39.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2d0400640 1 -- start start 2024-01-15T18:23:39.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2d0400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2a80039f0 0x7ff2c809e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2d0400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2c80c8960 0x7ff2c81534c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2d0400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c8158000 0x7ff2c81584b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2d0400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c81589f0 con 0x7ff2c8158000 2024-01-15T18:23:39.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2d0400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c80ca170 con 0x7ff2c80c8960 2024-01-15T18:23:39.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2d0400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c8153a00 con 0x7ff2a80039f0 2024-01-15T18:23:39.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2ce400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c8158000 0x7ff2c81584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2ce400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c8158000 0x7ff2c81584b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47666/0 (socket says 172.21.15.27:47666) 2024-01-15T18:23:39.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2ce400640 1 -- 172.21.15.27:0/2963332729 learned_addr learned my addr 172.21.15.27:0/2963332729 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:39.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2cd000640 1 --2- 172.21.15.27:0/2963332729 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2c80c8960 0x7ff2c81534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2cda00640 1 --2- 172.21.15.27:0/2963332729 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2a80039f0 0x7ff2c809e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2ce400640 1 -- 172.21.15.27:0/2963332729 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2a80039f0 msgr2=0x7ff2c809e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:39.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2ce400640 1 --2- 172.21.15.27:0/2963332729 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2a80039f0 0x7ff2c809e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2ce400640 1 -- 172.21.15.27:0/2963332729 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2c80c8960 msgr2=0x7ff2c81534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:39.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2ce400640 1 --2- 172.21.15.27:0/2963332729 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2c80c8960 0x7ff2c81534c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2ce400640 1 -- 172.21.15.27:0/2963332729 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c815e360 con 0x7ff2c8158000 2024-01-15T18:23:39.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2cd000640 1 --2- 172.21.15.27:0/2963332729 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2c80c8960 0x7ff2c81534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:23:39.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2cda00640 1 --2- 172.21.15.27:0/2963332729 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2a80039f0 0x7ff2c809e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:23:39.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2ce400640 1 --2- 172.21.15.27:0/2963332729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c8158000 0x7ff2c81584b0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7ff2c80716f0 tx=0x7ff2c0007b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:39.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2c5600640 1 -- 172.21.15.27:0/2963332729 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c000abc0 con 0x7ff2c8158000 2024-01-15T18:23:39.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2c5600640 1 -- 172.21.15.27:0/2963332729 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2c000ad40 con 0x7ff2c8158000 2024-01-15T18:23:39.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2c5600640 1 -- 172.21.15.27:0/2963332729 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c0013630 con 0x7ff2c8158000 2024-01-15T18:23:39.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2d0400640 1 -- 172.21.15.27:0/2963332729 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2c815e640 con 0x7ff2c8158000 2024-01-15T18:23:39.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.541+0000 7ff2d0400640 1 -- 172.21.15.27:0/2963332729 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2c81f6910 con 0x7ff2c8158000 2024-01-15T18:23:39.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.545+0000 7ff2d0400640 1 -- 172.21.15.27:0/2963332729 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2c804e850 con 0x7ff2c8158000 2024-01-15T18:23:39.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.545+0000 7ff2c5600640 1 -- 172.21.15.27:0/2963332729 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7ff2c0013790 con 0x7ff2c8158000 2024-01-15T18:23:39.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.545+0000 7ff2c5600640 1 --2- 172.21.15.27:0/2963332729 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff2a0067a90 0x7ff2a0069f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.545+0000 7ff2c5600640 1 -- 172.21.15.27:0/2963332729 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7ff2c0083990 con 0x7ff2c8158000 2024-01-15T18:23:39.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.545+0000 7ff2cda00640 1 --2- 172.21.15.27:0/2963332729 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff2a0067a90 0x7ff2a0069f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.549+0000 7ff2cda00640 1 --2- 172.21.15.27:0/2963332729 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff2a0067a90 0x7ff2a0069f50 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7ff2b80ad000 tx=0x7ff2b80b2040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:39.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.549+0000 7ff2c5600640 1 -- 172.21.15.27:0/2963332729 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff2c00522d0 con 0x7ff2c8158000 2024-01-15T18:23:39.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.709+0000 7ff2d0400640 1 -- 172.21.15.27:0/2963332729 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2c80ccdb0 con 0x7ff2a0067a90 2024-01-15T18:23:39.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.709+0000 7ff2c5600640 1 -- 172.21.15.27:0/2963332729 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff2b80a6740 con 0x7ff2a0067a90 2024-01-15T18:23:39.705 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 4 to deep-scrub 2024-01-15T18:23:39.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.709+0000 7ff2d0400640 1 -- 172.21.15.27:0/2963332729 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff2a0067a90 msgr2=0x7ff2a0069f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:39.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.709+0000 7ff2d0400640 1 --2- 172.21.15.27:0/2963332729 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff2a0067a90 0x7ff2a0069f50 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7ff2b80ad000 tx=0x7ff2b80b2040 comp rx=0 tx=0).stop 2024-01-15T18:23:39.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.713+0000 7ff2d0400640 1 -- 172.21.15.27:0/2963332729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c8158000 msgr2=0x7ff2c81584b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:39.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.713+0000 7ff2d0400640 1 --2- 172.21.15.27:0/2963332729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c8158000 0x7ff2c81584b0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7ff2c80716f0 tx=0x7ff2c0007b00 comp rx=0 tx=0).stop 2024-01-15T18:23:39.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.713+0000 7ff2d0400640 1 -- 172.21.15.27:0/2963332729 shutdown_connections 2024-01-15T18:23:39.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.713+0000 7ff2d0400640 1 --2- 172.21.15.27:0/2963332729 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7ff2a0067a90 0x7ff2a0069f50 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.713+0000 7ff2d0400640 1 --2- 172.21.15.27:0/2963332729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2c8158000 0x7ff2c81584b0 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.713+0000 7ff2d0400640 1 --2- 172.21.15.27:0/2963332729 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7ff2c80c8960 0x7ff2c81534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.713+0000 7ff2d0400640 1 --2- 172.21.15.27:0/2963332729 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7ff2a80039f0 0x7ff2c809e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.713+0000 7ff2d0400640 1 -- 172.21.15.27:0/2963332729 >> 172.21.15.27:0/2963332729 conn(0x7ff2c80d0e80 msgr2=0x7ff2c80adbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:39.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.713+0000 7ff2d0400640 1 -- 172.21.15.27:0/2963332729 shutdown_connections 2024-01-15T18:23:39.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.713+0000 7ff2d0400640 1 -- 172.21.15.27:0/2963332729 wait complete. 2024-01-15T18:23:39.728 INFO:tasks.ceph:Scrubbing osd.5 2024-01-15T18:23:39.728 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-01-15T18:23:39.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.877+0000 7f96a4200640 1 Processor -- start 2024-01-15T18:23:39.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.877+0000 7f96a4200640 1 -- start start 2024-01-15T18:23:39.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.877+0000 7f96a4200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f969c15d190 0x7f969c0cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.877+0000 7f96a4200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969c0b0d70 con 0x7f969c15d570 2024-01-15T18:23:39.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.877+0000 7f96a4200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969c0d0540 con 0x7f969c06c520 2024-01-15T18:23:39.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.877+0000 7f96a4200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f969c0b0440 con 0x7f969c15d190 2024-01-15T18:23:39.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.877+0000 7f96a1800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f969c15d190 0x7f969c0cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.877+0000 7f96a0e00640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f969c15d570 0x7f969c06be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42972/0 (socket says 172.21.15.27:42972) 2024-01-15T18:23:39.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.877+0000 7f96a0e00640 1 -- 172.21.15.27:0/3626443012 learned_addr learned my addr 172.21.15.27:0/3626443012 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:39.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.877+0000 7f96a1800640 1 -- 172.21.15.27:0/3626443012 >> v1:172.21.15.93:6789/0 conn(0x7f969c06c520 legacy=0x7f969c0cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:39.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.877+0000 7f96a1800640 1 -- 172.21.15.27:0/3626443012 >> v1:172.21.15.27:6789/0 conn(0x7f969c15d570 legacy=0x7f969c06be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:39.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.877+0000 7f96a1800640 1 -- 172.21.15.27:0/3626443012 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969c0b0720 con 0x7f969c15d190 2024-01-15T18:23:39.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.877+0000 7f969be00640 1 -- 172.21.15.27:0/3626443012 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2788316249 0 0) 0x7f969c0b0d70 con 0x7f969c15d570 2024-01-15T18:23:39.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.881+0000 7f96a1800640 1 --2- 172.21.15.27:0/3626443012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f969c15d190 0x7f969c0cfcf0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f968c0097c0 tx=0x7f968c01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e1739d87551d8c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:39.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.881+0000 7f969be00640 1 -- 172.21.15.27:0/3626443012 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c0ac630 con 0x7f969c15d190 2024-01-15T18:23:39.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.881+0000 7f969be00640 1 -- 172.21.15.27:0/3626443012 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f968c0ac790 con 0x7f969c15d190 2024-01-15T18:23:39.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.881+0000 7f969be00640 1 -- 172.21.15.27:0/3626443012 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c0acb20 con 0x7f969c15d190 2024-01-15T18:23:39.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.881+0000 7f96a4200640 1 -- 172.21.15.27:0/3626443012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f969c15d190 msgr2=0x7f969c0cfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:39.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.881+0000 7f96a4200640 1 --2- 172.21.15.27:0/3626443012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f969c15d190 0x7f969c0cfcf0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f968c0097c0 tx=0x7f968c01c410 comp rx=0 tx=0).stop 2024-01-15T18:23:39.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.881+0000 7f96a4200640 1 -- 172.21.15.27:0/3626443012 shutdown_connections 2024-01-15T18:23:39.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.881+0000 7f96a4200640 1 --2- 172.21.15.27:0/3626443012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f969c15d190 0x7f969c0cfcf0 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.881+0000 7f96a4200640 1 -- 172.21.15.27:0/3626443012 >> 172.21.15.27:0/3626443012 conn(0x7f969c0d0e70 msgr2=0x7f969c0d1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:39.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.881+0000 7f96a4200640 1 -- 172.21.15.27:0/3626443012 shutdown_connections 2024-01-15T18:23:39.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.881+0000 7f96a4200640 1 -- 172.21.15.27:0/3626443012 wait complete. 2024-01-15T18:23:39.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.885+0000 7f96a4200640 1 Processor -- start 2024-01-15T18:23:39.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.885+0000 7f96a4200640 1 -- start start 2024-01-15T18:23:39.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.885+0000 7f96a4200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f969c06c520 0x7f969c1510b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.885+0000 7f96a4200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f969c15d190 0x7f969c0b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.885+0000 7f96a4200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f969c15d570 0x7f969c06f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.885+0000 7f96a4200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c1515f0 con 0x7f969c15d190 2024-01-15T18:23:39.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.885+0000 7f96a4200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c0b5200 con 0x7f969c15d570 2024-01-15T18:23:39.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.885+0000 7f96a4200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f969c0c6610 con 0x7f969c06c520 2024-01-15T18:23:39.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.885+0000 7f96a0e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f969c15d190 0x7f969c0b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.885+0000 7f96a0e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f969c15d190 0x7f969c0b5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47688/0 (socket says 172.21.15.27:47688) 2024-01-15T18:23:39.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.885+0000 7f96a0e00640 1 -- 172.21.15.27:0/2609084178 learned_addr learned my addr 172.21.15.27:0/2609084178 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:39.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.885+0000 7f96a1800640 1 --2- 172.21.15.27:0/2609084178 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f969c06c520 0x7f969c1510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.885+0000 7f96a2200640 1 --2- 172.21.15.27:0/2609084178 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f969c15d570 0x7f969c06f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.885+0000 7f96a0e00640 1 -- 172.21.15.27:0/2609084178 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f969c06c520 msgr2=0x7f969c1510b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:39.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.885+0000 7f96a0e00640 1 --2- 172.21.15.27:0/2609084178 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f969c06c520 0x7f969c1510b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.885+0000 7f96a0e00640 1 -- 172.21.15.27:0/2609084178 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f969c15d570 msgr2=0x7f969c06f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:39.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.885+0000 7f96a0e00640 1 --2- 172.21.15.27:0/2609084178 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f969c15d570 0x7f969c06f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.885+0000 7f96a0e00640 1 -- 172.21.15.27:0/2609084178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f968c009420 con 0x7f969c15d190 2024-01-15T18:23:39.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.885+0000 7f96a2200640 1 --2- 172.21.15.27:0/2609084178 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f969c15d570 0x7f969c06f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:23:39.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.885+0000 7f96a0e00640 1 --2- 172.21.15.27:0/2609084178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f969c15d190 0x7f969c0b5f30 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f968400c480 tx=0x7f968400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:39.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.885+0000 7f9699600640 1 -- 172.21.15.27:0/2609084178 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96840074e0 con 0x7f969c15d190 2024-01-15T18:23:39.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.885+0000 7f9699600640 1 -- 172.21.15.27:0/2609084178 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9684007660 con 0x7f969c15d190 2024-01-15T18:23:39.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.885+0000 7f9699600640 1 -- 172.21.15.27:0/2609084178 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9684004a30 con 0x7f969c15d190 2024-01-15T18:23:39.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.885+0000 7f96a4200640 1 -- 172.21.15.27:0/2609084178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f969c0b5450 con 0x7f969c15d190 2024-01-15T18:23:39.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.885+0000 7f96a4200640 1 -- 172.21.15.27:0/2609084178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f969c0b5a10 con 0x7f969c15d190 2024-01-15T18:23:39.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.889+0000 7f96a4200640 1 -- 172.21.15.27:0/2609084178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f969c04e850 con 0x7f969c15d190 2024-01-15T18:23:39.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.889+0000 7f9699600640 1 -- 172.21.15.27:0/2609084178 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f968401b360 con 0x7f969c15d190 2024-01-15T18:23:39.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.889+0000 7f9699600640 1 --2- 172.21.15.27:0/2609084178 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9674067a60 0x7f9674069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.889+0000 7f9699600640 1 -- 172.21.15.27:0/2609084178 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f9684082c10 con 0x7f969c15d190 2024-01-15T18:23:39.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.889+0000 7f9699600640 1 --2- 172.21.15.27:0/2609084178 >> [v2:172.21.15.93:6800/2348354791,v1:172.21.15.93:6801/2348354791] conn(0x7f967406fe60 0x7f9674072300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:39.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.889+0000 7f9699600640 1 -- 172.21.15.27:0/2609084178 --> [v2:172.21.15.93:6800/2348354791,v1:172.21.15.93:6801/2348354791] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96740729d0 con 0x7f967406fe60 2024-01-15T18:23:39.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.889+0000 7f9699600640 1 -- 172.21.15.27:0/2609084178 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9684082f80 con 0x7f969c15d190 2024-01-15T18:23:39.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.889+0000 7f96a1800640 1 --2- 172.21.15.27:0/2609084178 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9674067a60 0x7f9674069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.889+0000 7f96a2200640 1 --2- 172.21.15.27:0/2609084178 >> [v2:172.21.15.93:6800/2348354791,v1:172.21.15.93:6801/2348354791] conn(0x7f967406fe60 0x7f9674072300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:39.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.889+0000 7f96a1800640 1 --2- 172.21.15.27:0/2609084178 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9674067a60 0x7f9674069f20 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f968c0ae000 tx=0x7f968c0a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:39.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.889+0000 7f96a2200640 1 --2- 172.21.15.27:0/2609084178 >> [v2:172.21.15.93:6800/2348354791,v1:172.21.15.93:6801/2348354791] conn(0x7f967406fe60 0x7f9674072300 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:39.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.893+0000 7f9699600640 1 -- 172.21.15.27:0/2609084178 <== osd.5 v2:172.21.15.93:6800/2348354791 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f96740729d0 con 0x7f967406fe60 2024-01-15T18:23:39.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.905+0000 7f96a4200640 1 -- 172.21.15.27:0/2609084178 --> [v2:172.21.15.93:6800/2348354791,v1:172.21.15.93:6801/2348354791] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f969c0b5c30 con 0x7f967406fe60 2024-01-15T18:23:39.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.905+0000 7f9699600640 1 -- 172.21.15.27:0/2609084178 <== osd.5 v2:172.21.15.93:6800/2348354791 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f969c0b5c30 con 0x7f967406fe60 2024-01-15T18:23:39.902 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:23:39.902 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-15T18:23:39.902 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:23:39.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.905+0000 7f96a4200640 1 -- 172.21.15.27:0/2609084178 >> [v2:172.21.15.93:6800/2348354791,v1:172.21.15.93:6801/2348354791] conn(0x7f967406fe60 msgr2=0x7f9674072300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:39.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.905+0000 7f96a4200640 1 --2- 172.21.15.27:0/2609084178 >> [v2:172.21.15.93:6800/2348354791,v1:172.21.15.93:6801/2348354791] conn(0x7f967406fe60 0x7f9674072300 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:39.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.905+0000 7f96a4200640 1 -- 172.21.15.27:0/2609084178 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9674067a60 msgr2=0x7f9674069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:39.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.905+0000 7f96a4200640 1 --2- 172.21.15.27:0/2609084178 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9674067a60 0x7f9674069f20 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f968c0ae000 tx=0x7f968c0a7000 comp rx=0 tx=0).stop 2024-01-15T18:23:39.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.905+0000 7f96a4200640 1 -- 172.21.15.27:0/2609084178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f969c15d190 msgr2=0x7f969c0b5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:39.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.905+0000 7f96a4200640 1 --2- 172.21.15.27:0/2609084178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f969c15d190 0x7f969c0b5f30 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f968400c480 tx=0x7f968400c940 comp rx=0 tx=0).stop 2024-01-15T18:23:39.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.905+0000 7f96a2200640 1 -- 172.21.15.27:0/2609084178 reap_dead start 2024-01-15T18:23:39.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.905+0000 7f96a4200640 1 -- 172.21.15.27:0/2609084178 shutdown_connections 2024-01-15T18:23:39.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.905+0000 7f96a4200640 1 -- 172.21.15.27:0/2609084178 >> 172.21.15.27:0/2609084178 conn(0x7f969c0d0e70 msgr2=0x7f969c0cd370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:39.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.905+0000 7f96a4200640 1 -- 172.21.15.27:0/2609084178 shutdown_connections 2024-01-15T18:23:39.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:39.905+0000 7f96a4200640 1 -- 172.21.15.27:0/2609084178 wait complete. 2024-01-15T18:23:39.921 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-01-15T18:23:40.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.065+0000 7f427b400640 1 Processor -- start 2024-01-15T18:23:40.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.065+0000 7f427b400640 1 -- start start 2024-01-15T18:23:40.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.069+0000 7f427b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f427415b180 0x7f42740cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.069+0000 7f427b400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f42740cd7e0 0x7f42740cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.069+0000 7f427b400640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42740ce180 con 0x7f42740b09d0 2024-01-15T18:23:40.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.069+0000 7f427b400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42740d1e20 con 0x7f427415b180 2024-01-15T18:23:40.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.069+0000 7f427b400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42740ceed0 con 0x7f42740cd7e0 2024-01-15T18:23:40.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.069+0000 7f4278a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f427415b180 0x7f42740cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.069+0000 7f4279400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f42740b09d0 0x7f42740b0e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42980/0 (socket says 172.21.15.27:42980) 2024-01-15T18:23:40.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.069+0000 7f4279400640 1 -- 172.21.15.27:0/914051699 learned_addr learned my addr 172.21.15.27:0/914051699 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:40.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.069+0000 7f4273e00640 1 --2- 172.21.15.27:0/914051699 >> v2:172.21.15.93:3300/0 conn(0x7f42740cd7e0 0x7f42740cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.069+0000 7f4278a00640 1 -- 172.21.15.27:0/914051699 >> v2:172.21.15.93:3300/0 conn(0x7f42740cd7e0 msgr2=0x7f42740cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:40.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.069+0000 7f4278a00640 1 --2- 172.21.15.27:0/914051699 >> v2:172.21.15.93:3300/0 conn(0x7f42740cd7e0 0x7f42740cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.069+0000 7f4278a00640 1 -- 172.21.15.27:0/914051699 >> v1:172.21.15.27:6789/0 conn(0x7f42740b09d0 legacy=0x7f42740b0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:40.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.069+0000 7f4278a00640 1 -- 172.21.15.27:0/914051699 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42740ce7d0 con 0x7f427415b180 2024-01-15T18:23:40.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.069+0000 7f4273400640 1 -- 172.21.15.27:0/914051699 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123956586 0 0) 0x7f42740ce180 con 0x7f42740b09d0 2024-01-15T18:23:40.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.073+0000 7f4278a00640 1 --2- 172.21.15.27:0/914051699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f427415b180 0x7f42740cfa90 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f4274071d00 tx=0x7f426401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48a232ffff0022f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:40.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.073+0000 7f4273400640 1 -- 172.21.15.27:0/914051699 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42640ab600 con 0x7f427415b180 2024-01-15T18:23:40.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.073+0000 7f4273400640 1 -- 172.21.15.27:0/914051699 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42640a7050 con 0x7f427415b180 2024-01-15T18:23:40.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.073+0000 7f4273400640 1 -- 172.21.15.27:0/914051699 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42640ab980 con 0x7f427415b180 2024-01-15T18:23:40.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.073+0000 7f427b400640 1 -- 172.21.15.27:0/914051699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f427415b180 msgr2=0x7f42740cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:40.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.073+0000 7f427b400640 1 --2- 172.21.15.27:0/914051699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f427415b180 0x7f42740cfa90 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f4274071d00 tx=0x7f426401c430 comp rx=0 tx=0).stop 2024-01-15T18:23:40.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.073+0000 7f427b400640 1 -- 172.21.15.27:0/914051699 shutdown_connections 2024-01-15T18:23:40.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.073+0000 7f427b400640 1 --2- 172.21.15.27:0/914051699 >> v2:172.21.15.93:3300/0 conn(0x7f42740cd7e0 0x7f42740cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.073+0000 7f427b400640 1 --2- 172.21.15.27:0/914051699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f427415b180 0x7f42740cfa90 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.073+0000 7f427b400640 1 -- 172.21.15.27:0/914051699 >> 172.21.15.27:0/914051699 conn(0x7f42740d0e80 msgr2=0x7f42740d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:40.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.073+0000 7f427b400640 1 -- 172.21.15.27:0/914051699 shutdown_connections 2024-01-15T18:23:40.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.073+0000 7f427b400640 1 -- 172.21.15.27:0/914051699 wait complete. 2024-01-15T18:23:40.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.073+0000 7f427b400640 1 Processor -- start 2024-01-15T18:23:40.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.073+0000 7f427b400640 1 -- start start 2024-01-15T18:23:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.077+0000 7f427b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42740b09d0 0x7f427406cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.077+0000 7f427b400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42740cd7e0 0x7f427406d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.077+0000 7f427b400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f427415b180 0x7f427415c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.077+0000 7f427b400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f427415c270 con 0x7f42740b09d0 2024-01-15T18:23:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.077+0000 7f427b400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f427415c550 con 0x7f42740cd7e0 2024-01-15T18:23:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.077+0000 7f427b400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42740ca700 con 0x7f427415b180 2024-01-15T18:23:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.077+0000 7f4278a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42740b09d0 0x7f427406cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.077+0000 7f4278a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42740b09d0 0x7f427406cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47714/0 (socket says 172.21.15.27:47714) 2024-01-15T18:23:40.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.077+0000 7f4278a00640 1 -- 172.21.15.27:0/2577962352 learned_addr learned my addr 172.21.15.27:0/2577962352 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:40.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.077+0000 7f4273e00640 1 --2- 172.21.15.27:0/2577962352 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42740cd7e0 0x7f427406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.077+0000 7f4279400640 1 --2- 172.21.15.27:0/2577962352 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f427415b180 0x7f427415c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.077+0000 7f4278a00640 1 -- 172.21.15.27:0/2577962352 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f427415b180 msgr2=0x7f427415c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:40.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.077+0000 7f4278a00640 1 --2- 172.21.15.27:0/2577962352 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f427415b180 0x7f427415c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.077+0000 7f4278a00640 1 -- 172.21.15.27:0/2577962352 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42740cd7e0 msgr2=0x7f427406d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:40.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.077+0000 7f4278a00640 1 --2- 172.21.15.27:0/2577962352 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42740cd7e0 0x7f427406d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.077+0000 7f4278a00640 1 -- 172.21.15.27:0/2577962352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4264009420 con 0x7f42740b09d0 2024-01-15T18:23:40.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.077+0000 7f4278a00640 1 --2- 172.21.15.27:0/2577962352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42740b09d0 0x7f427406cc70 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f426401c910 tx=0x7f42640191d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:40.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.077+0000 7f4270c00640 1 -- 172.21.15.27:0/2577962352 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42640b4400 con 0x7f42740b09d0 2024-01-15T18:23:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.077+0000 7f4270c00640 1 -- 172.21.15.27:0/2577962352 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42640b4560 con 0x7f42740b09d0 2024-01-15T18:23:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.077+0000 7f427b400640 1 -- 172.21.15.27:0/2577962352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42740ca9e0 con 0x7f42740b09d0 2024-01-15T18:23:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.077+0000 7f4270c00640 1 -- 172.21.15.27:0/2577962352 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42640b48d0 con 0x7f42740b09d0 2024-01-15T18:23:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.077+0000 7f427b400640 1 -- 172.21.15.27:0/2577962352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4274157820 con 0x7f42740b09d0 2024-01-15T18:23:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.077+0000 7f427b400640 1 -- 172.21.15.27:0/2577962352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f427404e850 con 0x7f42740b09d0 2024-01-15T18:23:40.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.081+0000 7f4270c00640 1 -- 172.21.15.27:0/2577962352 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f42640b2040 con 0x7f42740b09d0 2024-01-15T18:23:40.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.081+0000 7f4270c00640 1 --2- 172.21.15.27:0/2577962352 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f424c067a60 0x7f424c069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.081+0000 7f4270c00640 1 -- 172.21.15.27:0/2577962352 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f426411d2c0 con 0x7f42740b09d0 2024-01-15T18:23:40.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.081+0000 7f4273e00640 1 --2- 172.21.15.27:0/2577962352 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f424c067a60 0x7f424c069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.081+0000 7f4273e00640 1 --2- 172.21.15.27:0/2577962352 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f424c067a60 0x7f424c069f20 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f42740b0480 tx=0x7f425c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:40.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.085+0000 7f4270c00640 1 -- 172.21.15.27:0/2577962352 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f42640ebc90 con 0x7f42740b09d0 2024-01-15T18:23:40.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.241+0000 7f427b400640 1 -- 172.21.15.27:0/2577962352 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f427415ad60 con 0x7f424c067a60 2024-01-15T18:23:40.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.241+0000 7f4270c00640 1 -- 172.21.15.27:0/2577962352 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f427415ad60 con 0x7f424c067a60 2024-01-15T18:23:40.238 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 5 to deep-scrub 2024-01-15T18:23:40.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.245+0000 7f427b400640 1 -- 172.21.15.27:0/2577962352 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f424c067a60 msgr2=0x7f424c069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:40.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.245+0000 7f427b400640 1 --2- 172.21.15.27:0/2577962352 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f424c067a60 0x7f424c069f20 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f42740b0480 tx=0x7f425c008040 comp rx=0 tx=0).stop 2024-01-15T18:23:40.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.245+0000 7f427b400640 1 -- 172.21.15.27:0/2577962352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42740b09d0 msgr2=0x7f427406cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:40.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.245+0000 7f427b400640 1 --2- 172.21.15.27:0/2577962352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42740b09d0 0x7f427406cc70 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f426401c910 tx=0x7f42640191d0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.245+0000 7f427b400640 1 -- 172.21.15.27:0/2577962352 shutdown_connections 2024-01-15T18:23:40.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.245+0000 7f427b400640 1 --2- 172.21.15.27:0/2577962352 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f424c067a60 0x7f424c069f20 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.245+0000 7f427b400640 1 --2- 172.21.15.27:0/2577962352 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f427415b180 0x7f427415c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.245+0000 7f427b400640 1 --2- 172.21.15.27:0/2577962352 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f42740cd7e0 0x7f427406d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.245+0000 7f427b400640 1 --2- 172.21.15.27:0/2577962352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42740b09d0 0x7f427406cc70 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.245+0000 7f427b400640 1 -- 172.21.15.27:0/2577962352 >> 172.21.15.27:0/2577962352 conn(0x7f42740d0e80 msgr2=0x7f42740733c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:40.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.245+0000 7f427b400640 1 -- 172.21.15.27:0/2577962352 shutdown_connections 2024-01-15T18:23:40.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.245+0000 7f427b400640 1 -- 172.21.15.27:0/2577962352 wait complete. 2024-01-15T18:23:40.260 INFO:tasks.ceph:Scrubbing osd.6 2024-01-15T18:23:40.261 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-01-15T18:23:40.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.409+0000 7f2db9000640 1 Processor -- start 2024-01-15T18:23:40.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.409+0000 7f2db9000640 1 -- start start 2024-01-15T18:23:40.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.413+0000 7f2db9000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2db4146a60 0x7f2db40ac880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.413+0000 7f2db9000640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f2db4160cb0 0x7f2db41610f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.413+0000 7f2db9000640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db40cde30 con 0x7f2db406a4f0 2024-01-15T18:23:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.413+0000 7f2db9000640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2db4161630 con 0x7f2db4160cb0 2024-01-15T18:23:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.413+0000 7f2db9000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db406fa50 con 0x7f2db4146a60 2024-01-15T18:23:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.413+0000 7f2db2800640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2db406a4f0 0x7f2db40ce8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:42996/0 (socket says 172.21.15.27:42996) 2024-01-15T18:23:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.413+0000 7f2db2800640 1 -- 172.21.15.27:0/3367377317 learned_addr learned my addr 172.21.15.27:0/3367377317 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.413+0000 7f2db1e00640 1 --2- 172.21.15.27:0/3367377317 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2db4146a60 0x7f2db40ac880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.413+0000 7f2db1400640 1 --2- 172.21.15.27:0/3367377317 >> v2:172.21.15.93:3300/0 conn(0x7f2db4160cb0 0x7f2db41610f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.413+0000 7f2db0a00640 1 -- 172.21.15.27:0/3367377317 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874603247 0 0) 0x7f2db40cde30 con 0x7f2db406a4f0 2024-01-15T18:23:40.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.413+0000 7f2db0a00640 1 -- 172.21.15.27:0/3367377317 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d94003530 con 0x7f2db406a4f0 2024-01-15T18:23:40.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.413+0000 7f2db1e00640 1 -- 172.21.15.27:0/3367377317 >> v2:172.21.15.93:3300/0 conn(0x7f2db4160cb0 msgr2=0x7f2db41610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:40.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.413+0000 7f2db1e00640 1 --2- 172.21.15.27:0/3367377317 >> v2:172.21.15.93:3300/0 conn(0x7f2db4160cb0 0x7f2db41610f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.413+0000 7f2db1e00640 1 -- 172.21.15.27:0/3367377317 >> v1:172.21.15.27:6789/0 conn(0x7f2db406a4f0 legacy=0x7f2db40ce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:40.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.413+0000 7f2db1e00640 1 -- 172.21.15.27:0/3367377317 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db4161820 con 0x7f2db4146a60 2024-01-15T18:23:40.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.417+0000 7f2db1e00640 1 --2- 172.21.15.27:0/3367377317 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2db4146a60 0x7f2db40ac880 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2da40097c0 tx=0x7f2da401c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76fac57ab78cc8d8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:40.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.417+0000 7f2db0a00640 1 -- 172.21.15.27:0/3367377317 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da40ab650 con 0x7f2db4146a60 2024-01-15T18:23:40.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.417+0000 7f2db0a00640 1 -- 172.21.15.27:0/3367377317 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da40ab7b0 con 0x7f2db4146a60 2024-01-15T18:23:40.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.417+0000 7f2db0a00640 1 -- 172.21.15.27:0/3367377317 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da40abb40 con 0x7f2db4146a60 2024-01-15T18:23:40.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.417+0000 7f2db9000640 1 -- 172.21.15.27:0/3367377317 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2db4146a60 msgr2=0x7f2db40ac880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:40.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.417+0000 7f2db9000640 1 --2- 172.21.15.27:0/3367377317 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2db4146a60 0x7f2db40ac880 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2da40097c0 tx=0x7f2da401c450 comp rx=0 tx=0).stop 2024-01-15T18:23:40.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.417+0000 7f2db9000640 1 -- 172.21.15.27:0/3367377317 shutdown_connections 2024-01-15T18:23:40.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.417+0000 7f2db9000640 1 --2- 172.21.15.27:0/3367377317 >> v2:172.21.15.93:3300/0 conn(0x7f2db4160cb0 0x7f2db41610f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.417+0000 7f2db9000640 1 --2- 172.21.15.27:0/3367377317 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2db4146a60 0x7f2db40ac880 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.417+0000 7f2db9000640 1 -- 172.21.15.27:0/3367377317 >> 172.21.15.27:0/3367377317 conn(0x7f2db40cce10 msgr2=0x7f2db40b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:40.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.417+0000 7f2db9000640 1 -- 172.21.15.27:0/3367377317 shutdown_connections 2024-01-15T18:23:40.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.417+0000 7f2db9000640 1 -- 172.21.15.27:0/3367377317 wait complete. 2024-01-15T18:23:40.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.417+0000 7f2db9000640 1 Processor -- start 2024-01-15T18:23:40.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.417+0000 7f2db9000640 1 -- start start 2024-01-15T18:23:40.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.421+0000 7f2db9000640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2db406a4f0 0x7f2db4137240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.421+0000 7f2db9000640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2db4146a60 0x7f2db41f9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.421+0000 7f2db9000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2db4160cb0 0x7f2db41fa090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.421+0000 7f2db9000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db415d780 con 0x7f2db4160cb0 2024-01-15T18:23:40.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.421+0000 7f2db9000640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db4136eb0 con 0x7f2db406a4f0 2024-01-15T18:23:40.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.421+0000 7f2db9000640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db40b42f0 con 0x7f2db4146a60 2024-01-15T18:23:40.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.421+0000 7f2db2800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2db4160cb0 0x7f2db41fa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.421+0000 7f2db1e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2db406a4f0 0x7f2db4137240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.421+0000 7f2db1e00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2db406a4f0 0x7f2db4137240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3300/0 says I am v2:172.21.15.27:38198/0 (socket says 172.21.15.27:38198) 2024-01-15T18:23:40.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.421+0000 7f2db1e00640 1 -- 172.21.15.27:0/1114949814 learned_addr learned my addr 172.21.15.27:0/1114949814 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:40.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.421+0000 7f2db1400640 1 --2- 172.21.15.27:0/1114949814 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2db4146a60 0x7f2db41f9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.421+0000 7f2db1e00640 1 -- 172.21.15.27:0/1114949814 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2db4146a60 msgr2=0x7f2db41f9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:40.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.421+0000 7f2db1e00640 1 --2- 172.21.15.27:0/1114949814 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f2db4146a60 0x7f2db41f9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.421+0000 7f2db1e00640 1 -- 172.21.15.27:0/1114949814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2db4160cb0 msgr2=0x7f2db41fa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:40.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.421+0000 7f2db1e00640 1 --2- 172.21.15.27:0/1114949814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2db4160cb0 0x7f2db41fa090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.421+0000 7f2db1e00640 1 -- 172.21.15.27:0/1114949814 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da4009420 con 0x7f2db406a4f0 2024-01-15T18:23:40.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.421+0000 7f2db2800640 1 --2- 172.21.15.27:0/1114949814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2db4160cb0 0x7f2db41fa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:23:40.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.421+0000 7f2db1e00640 1 --2- 172.21.15.27:0/1114949814 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2db406a4f0 0x7f2db4137240 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f2da401a080 tx=0x7f2da401ac90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:40.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.421+0000 7f2daa000640 1 -- 172.21.15.27:0/1114949814 <== mon.1 v2:172.21.15.93:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da40bf610 con 0x7f2db406a4f0 2024-01-15T18:23:40.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.421+0000 7f2daa000640 1 -- 172.21.15.27:0/1114949814 <== mon.1 v2:172.21.15.93:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da40bf790 con 0x7f2db406a4f0 2024-01-15T18:23:40.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.421+0000 7f2db9000640 1 -- 172.21.15.27:0/1114949814 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db4137780 con 0x7f2db406a4f0 2024-01-15T18:23:40.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.421+0000 7f2daa000640 1 -- 172.21.15.27:0/1114949814 <== mon.1 v2:172.21.15.93:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da40bfb40 con 0x7f2db406a4f0 2024-01-15T18:23:40.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.421+0000 7f2db9000640 1 -- 172.21.15.27:0/1114949814 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db41fa8d0 con 0x7f2db406a4f0 2024-01-15T18:23:40.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.421+0000 7f2db9000640 1 -- 172.21.15.27:0/1114949814 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2db40acdc0 con 0x7f2db406a4f0 2024-01-15T18:23:40.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.425+0000 7f2daa000640 1 -- 172.21.15.27:0/1114949814 <== mon.1 v2:172.21.15.93:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f2da40ca430 con 0x7f2db406a4f0 2024-01-15T18:23:40.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.425+0000 7f2daa000640 1 --2- 172.21.15.27:0/1114949814 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2d8c067a60 0x7f2d8c069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.425+0000 7f2daa000640 1 -- 172.21.15.27:0/1114949814 <== mon.1 v2:172.21.15.93:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f2da411e130 con 0x7f2db406a4f0 2024-01-15T18:23:40.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.425+0000 7f2daa000640 1 --2- 172.21.15.27:0/1114949814 >> [v2:172.21.15.93:6805/838236833,v1:172.21.15.93:6809/838236833] conn(0x7f2d8c06fe60 0x7f2d8c072300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.425+0000 7f2daa000640 1 -- 172.21.15.27:0/1114949814 --> [v2:172.21.15.93:6805/838236833,v1:172.21.15.93:6809/838236833] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2d8c0729d0 con 0x7f2d8c06fe60 2024-01-15T18:23:40.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.425+0000 7f2daa000640 1 -- 172.21.15.27:0/1114949814 <== mon.1 v2:172.21.15.93:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2da411e4a0 con 0x7f2db406a4f0 2024-01-15T18:23:40.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.425+0000 7f2db1400640 1 --2- 172.21.15.27:0/1114949814 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2d8c067a60 0x7f2d8c069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.425+0000 7f2db2800640 1 --2- 172.21.15.27:0/1114949814 >> [v2:172.21.15.93:6805/838236833,v1:172.21.15.93:6809/838236833] conn(0x7f2d8c06fe60 0x7f2d8c072300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.425+0000 7f2db1400640 1 --2- 172.21.15.27:0/1114949814 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2d8c067a60 0x7f2d8c069f20 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f2db41f9ab0 tx=0x7f2d9c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:40.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.425+0000 7f2db2800640 1 --2- 172.21.15.27:0/1114949814 >> [v2:172.21.15.93:6805/838236833,v1:172.21.15.93:6809/838236833] conn(0x7f2d8c06fe60 0x7f2d8c072300 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:40.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.425+0000 7f2daa000640 1 -- 172.21.15.27:0/1114949814 <== osd.6 v2:172.21.15.93:6805/838236833 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f2d8c0729d0 con 0x7f2d8c06fe60 2024-01-15T18:23:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.441+0000 7f2db9000640 1 -- 172.21.15.27:0/1114949814 --> [v2:172.21.15.93:6805/838236833,v1:172.21.15.93:6809/838236833] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2db40ce3d0 con 0x7f2d8c06fe60 2024-01-15T18:23:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.441+0000 7f2daa000640 1 -- 172.21.15.27:0/1114949814 <== osd.6 v2:172.21.15.93:6805/838236833 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f2dac004b70 con 0x7f2d8c06fe60 2024-01-15T18:23:40.437 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:23:40.437 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-15T18:23:40.437 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:23:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.441+0000 7f2db9000640 1 -- 172.21.15.27:0/1114949814 >> [v2:172.21.15.93:6805/838236833,v1:172.21.15.93:6809/838236833] conn(0x7f2d8c06fe60 msgr2=0x7f2d8c072300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.441+0000 7f2db9000640 1 --2- 172.21.15.27:0/1114949814 >> [v2:172.21.15.93:6805/838236833,v1:172.21.15.93:6809/838236833] conn(0x7f2d8c06fe60 0x7f2d8c072300 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.441+0000 7f2db9000640 1 -- 172.21.15.27:0/1114949814 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2d8c067a60 msgr2=0x7f2d8c069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.441+0000 7f2db9000640 1 --2- 172.21.15.27:0/1114949814 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f2d8c067a60 0x7f2d8c069f20 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f2db41f9ab0 tx=0x7f2d9c009190 comp rx=0 tx=0).stop 2024-01-15T18:23:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.441+0000 7f2db9000640 1 -- 172.21.15.27:0/1114949814 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2db406a4f0 msgr2=0x7f2db4137240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.441+0000 7f2db9000640 1 --2- 172.21.15.27:0/1114949814 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f2db406a4f0 0x7f2db4137240 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f2da401a080 tx=0x7f2da401ac90 comp rx=0 tx=0).stop 2024-01-15T18:23:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.441+0000 7f2db2800640 1 -- 172.21.15.27:0/1114949814 reap_dead start 2024-01-15T18:23:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.441+0000 7f2db9000640 1 -- 172.21.15.27:0/1114949814 shutdown_connections 2024-01-15T18:23:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.441+0000 7f2db9000640 1 -- 172.21.15.27:0/1114949814 >> 172.21.15.27:0/1114949814 conn(0x7f2db40cce10 msgr2=0x7f2db40cbe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.441+0000 7f2db9000640 1 -- 172.21.15.27:0/1114949814 shutdown_connections 2024-01-15T18:23:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.441+0000 7f2db9000640 1 -- 172.21.15.27:0/1114949814 wait complete. 2024-01-15T18:23:40.456 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-01-15T18:23:40.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.609+0000 7f3a3f200640 1 Processor -- start 2024-01-15T18:23:40.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.609+0000 7f3a3f200640 1 -- start start 2024-01-15T18:23:40.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.609+0000 7f3a3f200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a3815d7c0 0x7f3a380b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.609+0000 7f3a3f200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3a380b1d90 0x7f3a38099a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.609+0000 7f3a3f200640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a380b1aa0 con 0x7f3a3806bc70 2024-01-15T18:23:40.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.609+0000 7f3a3f200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a380b1300 con 0x7f3a380b1d90 2024-01-15T18:23:40.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.609+0000 7f3a3f200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a380b25a0 con 0x7f3a3815d7c0 2024-01-15T18:23:40.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.609+0000 7f3a3d200640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3a3806bc70 0x7f3a38161140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:43010/0 (socket says 172.21.15.27:43010) 2024-01-15T18:23:40.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.609+0000 7f3a3d200640 1 -- 172.21.15.27:0/935150112 learned_addr learned my addr 172.21.15.27:0/935150112 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:23:40.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.609+0000 7f3a37400640 1 --2- 172.21.15.27:0/935150112 >> v2:172.21.15.27:3300/0 conn(0x7f3a380b1d90 0x7f3a38099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.609+0000 7f3a37e00640 1 --2- 172.21.15.27:0/935150112 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a3815d7c0 0x7f3a380b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.609+0000 7f3a37400640 1 -- 172.21.15.27:0/935150112 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a3815d7c0 msgr2=0x7f3a380b35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:40.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.609+0000 7f3a37400640 1 --2- 172.21.15.27:0/935150112 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a3815d7c0 0x7f3a380b35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.609+0000 7f3a36a00640 1 -- 172.21.15.27:0/935150112 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152012364 0 0) 0x7f3a380b1aa0 con 0x7f3a3806bc70 2024-01-15T18:23:40.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.609+0000 7f3a37400640 1 -- 172.21.15.27:0/935150112 >> v1:172.21.15.27:6789/0 conn(0x7f3a3806bc70 legacy=0x7f3a38161140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:40.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.609+0000 7f3a37400640 1 -- 172.21.15.27:0/935150112 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a380b14e0 con 0x7f3a380b1d90 2024-01-15T18:23:40.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.613+0000 7f3a37400640 1 --2- 172.21.15.27:0/935150112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a380b1d90 0x7f3a38099a10 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f3a200097c0 tx=0x7f3a2001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6686964201cce67d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:40.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.613+0000 7f3a36a00640 1 -- 172.21.15.27:0/935150112 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a200ab600 con 0x7f3a380b1d90 2024-01-15T18:23:40.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.613+0000 7f3a36a00640 1 -- 172.21.15.27:0/935150112 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a200a7050 con 0x7f3a380b1d90 2024-01-15T18:23:40.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.613+0000 7f3a36a00640 1 -- 172.21.15.27:0/935150112 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a200ab980 con 0x7f3a380b1d90 2024-01-15T18:23:40.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.613+0000 7f3a3f200640 1 -- 172.21.15.27:0/935150112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a380b1d90 msgr2=0x7f3a38099a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:40.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.613+0000 7f3a3f200640 1 --2- 172.21.15.27:0/935150112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a380b1d90 0x7f3a38099a10 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f3a200097c0 tx=0x7f3a2001c360 comp rx=0 tx=0).stop 2024-01-15T18:23:40.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.613+0000 7f3a3f200640 1 -- 172.21.15.27:0/935150112 shutdown_connections 2024-01-15T18:23:40.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.613+0000 7f3a3f200640 1 --2- 172.21.15.27:0/935150112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a380b1d90 0x7f3a38099a10 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.613+0000 7f3a3f200640 1 --2- 172.21.15.27:0/935150112 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a3815d7c0 0x7f3a380b35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.613+0000 7f3a3f200640 1 -- 172.21.15.27:0/935150112 >> 172.21.15.27:0/935150112 conn(0x7f3a380b2a80 msgr2=0x7f3a380b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:40.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.613+0000 7f3a3f200640 1 -- 172.21.15.27:0/935150112 shutdown_connections 2024-01-15T18:23:40.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.613+0000 7f3a3f200640 1 -- 172.21.15.27:0/935150112 wait complete. 2024-01-15T18:23:40.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.617+0000 7f3a3f200640 1 Processor -- start 2024-01-15T18:23:40.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.617+0000 7f3a3f200640 1 -- start start 2024-01-15T18:23:40.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.617+0000 7f3a3f200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3a3806bc70 0x7f3a380ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.617+0000 7f3a3f200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a380b1d90 0x7f3a381527a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.617+0000 7f3a3f200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a3815d7c0 0x7f3a38155480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.617+0000 7f3a3f200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a38152ce0 con 0x7f3a380b1d90 2024-01-15T18:23:40.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.617+0000 7f3a3f200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a38154c20 con 0x7f3a3806bc70 2024-01-15T18:23:40.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.617+0000 7f3a3f200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a38154190 con 0x7f3a3815d7c0 2024-01-15T18:23:40.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.617+0000 7f3a37400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a380b1d90 0x7f3a381527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.617+0000 7f3a37400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a380b1d90 0x7f3a381527a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47744/0 (socket says 172.21.15.27:47744) 2024-01-15T18:23:40.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.617+0000 7f3a37400640 1 -- 172.21.15.27:0/4033046264 learned_addr learned my addr 172.21.15.27:0/4033046264 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:40.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.617+0000 7f3a37e00640 1 --2- 172.21.15.27:0/4033046264 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3a3806bc70 0x7f3a380ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.617+0000 7f3a3d200640 1 --2- 172.21.15.27:0/4033046264 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a3815d7c0 0x7f3a38155480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.617+0000 7f3a37400640 1 -- 172.21.15.27:0/4033046264 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a3815d7c0 msgr2=0x7f3a38155480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:40.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.617+0000 7f3a37400640 1 --2- 172.21.15.27:0/4033046264 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a3815d7c0 0x7f3a38155480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.617+0000 7f3a37400640 1 -- 172.21.15.27:0/4033046264 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3a3806bc70 msgr2=0x7f3a380ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:40.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.617+0000 7f3a37400640 1 --2- 172.21.15.27:0/4033046264 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3a3806bc70 0x7f3a380ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.617+0000 7f3a37400640 1 -- 172.21.15.27:0/4033046264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a20009420 con 0x7f3a380b1d90 2024-01-15T18:23:40.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.617+0000 7f3a37400640 1 --2- 172.21.15.27:0/4033046264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a380b1d90 0x7f3a381527a0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f3a20019f90 tx=0x7f3a20019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:40.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.617+0000 7f3a2fe00640 1 -- 172.21.15.27:0/4033046264 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a200abdd0 con 0x7f3a380b1d90 2024-01-15T18:23:40.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.617+0000 7f3a2fe00640 1 -- 172.21.15.27:0/4033046264 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a200b4410 con 0x7f3a380b1d90 2024-01-15T18:23:40.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.617+0000 7f3a3f200640 1 -- 172.21.15.27:0/4033046264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a38156140 con 0x7f3a380b1d90 2024-01-15T18:23:40.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.617+0000 7f3a2fe00640 1 -- 172.21.15.27:0/4033046264 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a200b47c0 con 0x7f3a380b1d90 2024-01-15T18:23:40.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.621+0000 7f3a3f200640 1 -- 172.21.15.27:0/4033046264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a38153bd0 con 0x7f3a380b1d90 2024-01-15T18:23:40.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.621+0000 7f3a3f200640 1 -- 172.21.15.27:0/4033046264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a3804e850 con 0x7f3a380b1d90 2024-01-15T18:23:40.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.621+0000 7f3a2fe00640 1 -- 172.21.15.27:0/4033046264 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f3a200b2040 con 0x7f3a380b1d90 2024-01-15T18:23:40.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.621+0000 7f3a2fe00640 1 --2- 172.21.15.27:0/4033046264 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a10067a60 0x7f3a10069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.621+0000 7f3a2fe00640 1 -- 172.21.15.27:0/4033046264 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f3a2011d0f0 con 0x7f3a380b1d90 2024-01-15T18:23:40.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.621+0000 7f3a37e00640 1 --2- 172.21.15.27:0/4033046264 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a10067a60 0x7f3a10069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.625+0000 7f3a37e00640 1 --2- 172.21.15.27:0/4033046264 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a10067a60 0x7f3a10069f20 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009ac0 tx=0x7f3a28009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:40.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.625+0000 7f3a2fe00640 1 -- 172.21.15.27:0/4033046264 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3a200ebb80 con 0x7f3a380b1d90 2024-01-15T18:23:40.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.789+0000 7f3a3f200640 1 -- 172.21.15.27:0/4033046264 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a380d0bf0 con 0x7f3a10067a60 2024-01-15T18:23:40.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.789+0000 7f3a2fe00640 1 -- 172.21.15.27:0/4033046264 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3a380d0bf0 con 0x7f3a10067a60 2024-01-15T18:23:40.786 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 6 to deep-scrub 2024-01-15T18:23:40.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.793+0000 7f3a3f200640 1 -- 172.21.15.27:0/4033046264 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a10067a60 msgr2=0x7f3a10069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:40.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.793+0000 7f3a3f200640 1 --2- 172.21.15.27:0/4033046264 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a10067a60 0x7f3a10069f20 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009ac0 tx=0x7f3a28009190 comp rx=0 tx=0).stop 2024-01-15T18:23:40.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.793+0000 7f3a3f200640 1 -- 172.21.15.27:0/4033046264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a380b1d90 msgr2=0x7f3a381527a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:40.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.793+0000 7f3a3f200640 1 --2- 172.21.15.27:0/4033046264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a380b1d90 0x7f3a381527a0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f3a20019f90 tx=0x7f3a20019100 comp rx=0 tx=0).stop 2024-01-15T18:23:40.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.793+0000 7f3a3f200640 1 -- 172.21.15.27:0/4033046264 shutdown_connections 2024-01-15T18:23:40.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.793+0000 7f3a3f200640 1 --2- 172.21.15.27:0/4033046264 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f3a10067a60 0x7f3a10069f20 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.793+0000 7f3a3f200640 1 --2- 172.21.15.27:0/4033046264 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f3a3815d7c0 0x7f3a38155480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.793+0000 7f3a3f200640 1 --2- 172.21.15.27:0/4033046264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a380b1d90 0x7f3a381527a0 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.793+0000 7f3a3f200640 1 --2- 172.21.15.27:0/4033046264 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f3a3806bc70 0x7f3a380ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.793+0000 7f3a3f200640 1 -- 172.21.15.27:0/4033046264 >> 172.21.15.27:0/4033046264 conn(0x7f3a380b2a80 msgr2=0x7f3a3806f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:40.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.793+0000 7f3a3f200640 1 -- 172.21.15.27:0/4033046264 shutdown_connections 2024-01-15T18:23:40.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.793+0000 7f3a3f200640 1 -- 172.21.15.27:0/4033046264 wait complete. 2024-01-15T18:23:40.808 INFO:tasks.ceph:Scrubbing osd.7 2024-01-15T18:23:40.809 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-01-15T18:23:40.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.957+0000 7f0556400640 1 Processor -- start 2024-01-15T18:23:40.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.957+0000 7f0556400640 1 -- start start 2024-01-15T18:23:40.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.961+0000 7f0556400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f055015ad20 0x7f055015b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.961+0000 7f0556400640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f05500715f0 0x7f055015ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.961+0000 7f0556400640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f055015a610 con 0x7f055015d900 2024-01-15T18:23:40.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.961+0000 7f0556400640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f055015e3c0 con 0x7f055015ad20 2024-01-15T18:23:40.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.961+0000 7f0556400640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f055015e6a0 con 0x7f05500715f0 2024-01-15T18:23:40.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.961+0000 7f054f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f055015ad20 0x7f055015b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.961+0000 7f054f400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f055015ad20 0x7f055015b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47760/0 (socket says 172.21.15.27:47760) 2024-01-15T18:23:40.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.961+0000 7f054f400640 1 -- 172.21.15.27:0/2393743920 learned_addr learned my addr 172.21.15.27:0/2393743920 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:40.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.961+0000 7f054fe00640 1 --2- 172.21.15.27:0/2393743920 >> v2:172.21.15.93:3300/0 conn(0x7f05500715f0 0x7f055015ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.961+0000 7f054f400640 1 -- 172.21.15.27:0/2393743920 >> v2:172.21.15.93:3300/0 conn(0x7f05500715f0 msgr2=0x7f055015ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:40.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.961+0000 7f054f400640 1 --2- 172.21.15.27:0/2393743920 >> v2:172.21.15.93:3300/0 conn(0x7f05500715f0 0x7f055015ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.961+0000 7f054f400640 1 -- 172.21.15.27:0/2393743920 >> v1:172.21.15.93:6789/0 conn(0x7f055015d900 legacy=0x7f0550070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:40.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.961+0000 7f054f400640 1 -- 172.21.15.27:0/2393743920 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f055015b6e0 con 0x7f055015ad20 2024-01-15T18:23:40.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.961+0000 7f054fe00640 1 --2- 172.21.15.27:0/2393743920 >> v2:172.21.15.93:3300/0 conn(0x7f05500715f0 0x7f055015ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:23:40.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.961+0000 7f054f400640 1 --2- 172.21.15.27:0/2393743920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f055015ad20 0x7f055015b100 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f054000bb10 tx=0x7f054001eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f25ae47291306b6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:40.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.961+0000 7f054e000640 1 -- 172.21.15.27:0/2393743920 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05400a7b50 con 0x7f055015ad20 2024-01-15T18:23:40.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.965+0000 7f054e000640 1 -- 172.21.15.27:0/2393743920 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05400a7cb0 con 0x7f055015ad20 2024-01-15T18:23:40.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.965+0000 7f054e000640 1 -- 172.21.15.27:0/2393743920 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05400af660 con 0x7f055015ad20 2024-01-15T18:23:40.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.965+0000 7f0556400640 1 -- 172.21.15.27:0/2393743920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f055015ad20 msgr2=0x7f055015b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:40.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.965+0000 7f0556400640 1 --2- 172.21.15.27:0/2393743920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f055015ad20 0x7f055015b100 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f054000bb10 tx=0x7f054001eb60 comp rx=0 tx=0).stop 2024-01-15T18:23:40.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.965+0000 7f0556400640 1 -- 172.21.15.27:0/2393743920 shutdown_connections 2024-01-15T18:23:40.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.965+0000 7f0556400640 1 --2- 172.21.15.27:0/2393743920 >> v2:172.21.15.93:3300/0 conn(0x7f05500715f0 0x7f055015ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.965+0000 7f0556400640 1 --2- 172.21.15.27:0/2393743920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f055015ad20 0x7f055015b100 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.965+0000 7f0556400640 1 -- 172.21.15.27:0/2393743920 >> 172.21.15.27:0/2393743920 conn(0x7f05500cb700 msgr2=0x7f05500cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:40.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.965+0000 7f0556400640 1 -- 172.21.15.27:0/2393743920 shutdown_connections 2024-01-15T18:23:40.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.965+0000 7f0556400640 1 -- 172.21.15.27:0/2393743920 wait complete. 2024-01-15T18:23:40.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.965+0000 7f0556400640 1 Processor -- start 2024-01-15T18:23:40.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.965+0000 7f0556400640 1 -- start start 2024-01-15T18:23:40.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.965+0000 7f0556400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05500715f0 0x7f0550156f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.965+0000 7f0556400640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f055015ad20 0x7f05501f5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.965+0000 7f0556400640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f055015d900 0x7f0550155400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.965+0000 7f0556400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0550150300 con 0x7f05500715f0 2024-01-15T18:23:40.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.965+0000 7f0556400640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0550157590 con 0x7f055015ad20 2024-01-15T18:23:40.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.965+0000 7f0556400640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f055014fcb0 con 0x7f055015d900 2024-01-15T18:23:40.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.969+0000 7f054f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05500715f0 0x7f0550156f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.969+0000 7f054f400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05500715f0 0x7f0550156f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47768/0 (socket says 172.21.15.27:47768) 2024-01-15T18:23:40.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.969+0000 7f054f400640 1 -- 172.21.15.27:0/1291079940 learned_addr learned my addr 172.21.15.27:0/1291079940 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:40.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.969+0000 7f054fe00640 1 --2- 172.21.15.27:0/1291079940 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f055015d900 0x7f0550155400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.969+0000 7f054ea00640 1 --2- 172.21.15.27:0/1291079940 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f055015ad20 0x7f05501f5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.969+0000 7f054f400640 1 -- 172.21.15.27:0/1291079940 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f055015d900 msgr2=0x7f0550155400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:40.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.969+0000 7f054f400640 1 --2- 172.21.15.27:0/1291079940 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f055015d900 0x7f0550155400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.969+0000 7f054f400640 1 -- 172.21.15.27:0/1291079940 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f055015ad20 msgr2=0x7f05501f5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:40.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.969+0000 7f054f400640 1 --2- 172.21.15.27:0/1291079940 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f055015ad20 0x7f05501f5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.969+0000 7f054f400640 1 -- 172.21.15.27:0/1291079940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054000b770 con 0x7f05500715f0 2024-01-15T18:23:40.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.969+0000 7f054f400640 1 --2- 172.21.15.27:0/1291079940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05500715f0 0x7f0550156f20 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f054000b2c0 tx=0x7f054001b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:40.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.969+0000 7f0547400640 1 -- 172.21.15.27:0/1291079940 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05400ba640 con 0x7f05500715f0 2024-01-15T18:23:40.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.969+0000 7f0547400640 1 -- 172.21.15.27:0/1291079940 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05400ba7c0 con 0x7f05500715f0 2024-01-15T18:23:40.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.969+0000 7f0547400640 1 -- 172.21.15.27:0/1291079940 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05400bab70 con 0x7f05500715f0 2024-01-15T18:23:40.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.969+0000 7f0556400640 1 -- 172.21.15.27:0/1291079940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f055014ff90 con 0x7f05500715f0 2024-01-15T18:23:40.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.969+0000 7f0556400640 1 -- 172.21.15.27:0/1291079940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0550155940 con 0x7f05500715f0 2024-01-15T18:23:40.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.969+0000 7f0556400640 1 -- 172.21.15.27:0/1291079940 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0550155b30 con 0x7f05500715f0 2024-01-15T18:23:40.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.969+0000 7f0547400640 1 -- 172.21.15.27:0/1291079940 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f05400c33f0 con 0x7f05500715f0 2024-01-15T18:23:40.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.973+0000 7f0547400640 1 --2- 172.21.15.27:0/1291079940 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0528067a60 0x7f0528069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.973+0000 7f0547400640 1 -- 172.21.15.27:0/1291079940 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f054011e850 con 0x7f05500715f0 2024-01-15T18:23:40.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.973+0000 7f0547400640 1 --2- 172.21.15.27:0/1291079940 >> [v2:172.21.15.93:6824/978428062,v1:172.21.15.93:6825/978428062] conn(0x7f052806fe60 0x7f0528072300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:40.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.973+0000 7f0547400640 1 -- 172.21.15.27:0/1291079940 --> [v2:172.21.15.93:6824/978428062,v1:172.21.15.93:6825/978428062] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f05280729d0 con 0x7f052806fe60 2024-01-15T18:23:40.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.973+0000 7f0547400640 1 -- 172.21.15.27:0/1291079940 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7f054011ebc0 con 0x7f05500715f0 2024-01-15T18:23:40.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.973+0000 7f054ea00640 1 --2- 172.21.15.27:0/1291079940 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0528067a60 0x7f0528069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.973+0000 7f054fe00640 1 --2- 172.21.15.27:0/1291079940 >> [v2:172.21.15.93:6824/978428062,v1:172.21.15.93:6825/978428062] conn(0x7f052806fe60 0x7f0528072300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:40.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.973+0000 7f054ea00640 1 --2- 172.21.15.27:0/1291079940 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0528067a60 0x7f0528069f20 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f05501f9070 tx=0x7f0538009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:40.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.973+0000 7f054fe00640 1 --2- 172.21.15.27:0/1291079940 >> [v2:172.21.15.93:6824/978428062,v1:172.21.15.93:6825/978428062] conn(0x7f052806fe60 0x7f0528072300 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:40.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.973+0000 7f0547400640 1 -- 172.21.15.27:0/1291079940 <== osd.7 v2:172.21.15.93:6824/978428062 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f05280729d0 con 0x7f052806fe60 2024-01-15T18:23:40.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.985+0000 7f0556400640 1 -- 172.21.15.27:0/1291079940 --> [v2:172.21.15.93:6824/978428062,v1:172.21.15.93:6825/978428062] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f055014fb30 con 0x7f052806fe60 2024-01-15T18:23:40.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.989+0000 7f0547400640 1 -- 172.21.15.27:0/1291079940 <== osd.7 v2:172.21.15.93:6824/978428062 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f05480039a0 con 0x7f052806fe60 2024-01-15T18:23:40.984 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-01-15T18:23:40.984 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-15T18:23:40.984 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-01-15T18:23:40.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.989+0000 7f0556400640 1 -- 172.21.15.27:0/1291079940 >> [v2:172.21.15.93:6824/978428062,v1:172.21.15.93:6825/978428062] conn(0x7f052806fe60 msgr2=0x7f0528072300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:40.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.989+0000 7f0556400640 1 --2- 172.21.15.27:0/1291079940 >> [v2:172.21.15.93:6824/978428062,v1:172.21.15.93:6825/978428062] conn(0x7f052806fe60 0x7f0528072300 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:40.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.989+0000 7f0556400640 1 -- 172.21.15.27:0/1291079940 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0528067a60 msgr2=0x7f0528069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:40.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.989+0000 7f0556400640 1 --2- 172.21.15.27:0/1291079940 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f0528067a60 0x7f0528069f20 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f05501f9070 tx=0x7f0538009060 comp rx=0 tx=0).stop 2024-01-15T18:23:40.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.989+0000 7f0556400640 1 -- 172.21.15.27:0/1291079940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05500715f0 msgr2=0x7f0550156f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:40.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.989+0000 7f0556400640 1 --2- 172.21.15.27:0/1291079940 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05500715f0 0x7f0550156f20 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f054000b2c0 tx=0x7f054001b900 comp rx=0 tx=0).stop 2024-01-15T18:23:40.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.989+0000 7f054fe00640 1 -- 172.21.15.27:0/1291079940 reap_dead start 2024-01-15T18:23:40.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.989+0000 7f0556400640 1 -- 172.21.15.27:0/1291079940 shutdown_connections 2024-01-15T18:23:40.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.989+0000 7f0556400640 1 -- 172.21.15.27:0/1291079940 >> 172.21.15.27:0/1291079940 conn(0x7f05500cb700 msgr2=0x7f055006dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:40.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.989+0000 7f0556400640 1 -- 172.21.15.27:0/1291079940 shutdown_connections 2024-01-15T18:23:40.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:40.989+0000 7f0556400640 1 -- 172.21.15.27:0/1291079940 wait complete. 2024-01-15T18:23:41.003 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-01-15T18:23:41.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.149+0000 7faedbe00640 1 Processor -- start 2024-01-15T18:23:41.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.149+0000 7faedbe00640 1 -- start start 2024-01-15T18:23:41.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.149+0000 7faedbe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faed4143da0 0x7faed406d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:41.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.149+0000 7faedbe00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faed406c200 0x7faed40cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:41.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.149+0000 7faedbe00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7faed40cad40 0x7faed40cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:41.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.149+0000 7faedbe00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed40cb710 con 0x7faed406c200 2024-01-15T18:23:41.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.149+0000 7faedbe00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed4144180 con 0x7faed40cad40 2024-01-15T18:23:41.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.149+0000 7faedbe00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed406c750 con 0x7faed4143da0 2024-01-15T18:23:41.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.149+0000 7faed8a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faed406c200 0x7faed40cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:41.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.149+0000 7faed8a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faed406c200 0x7faed40cb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47776/0 (socket says 172.21.15.27:47776) 2024-01-15T18:23:41.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.149+0000 7faed8a00640 1 -- 172.21.15.27:0/1758692861 learned_addr learned my addr 172.21.15.27:0/1758692861 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:41.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.149+0000 7faed9400640 1 --2- 172.21.15.27:0/1758692861 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faed4143da0 0x7faed406d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:41.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.149+0000 7faed9e00640 1 --2- 172.21.15.27:0/1758692861 >> v2:172.21.15.93:3300/0 conn(0x7faed40cad40 0x7faed40cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:41.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.149+0000 7faed8a00640 1 -- 172.21.15.27:0/1758692861 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faed4143da0 msgr2=0x7faed406d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:41.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.149+0000 7faed8a00640 1 --2- 172.21.15.27:0/1758692861 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faed4143da0 0x7faed406d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:41.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.149+0000 7faed8a00640 1 -- 172.21.15.27:0/1758692861 >> v2:172.21.15.93:3300/0 conn(0x7faed40cad40 msgr2=0x7faed40cb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:41.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.149+0000 7faed8a00640 1 --2- 172.21.15.27:0/1758692861 >> v2:172.21.15.93:3300/0 conn(0x7faed40cad40 0x7faed40cb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:41.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.149+0000 7faed8a00640 1 -- 172.21.15.27:0/1758692861 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed406ca30 con 0x7faed406c200 2024-01-15T18:23:41.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.153+0000 7faed8a00640 1 --2- 172.21.15.27:0/1758692861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faed406c200 0x7faed40cb9c0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7faed40ae790 tx=0x7faebc01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85a134299e4ca51c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:41.149 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.153+0000 7faed3e00640 1 -- 172.21.15.27:0/1758692861 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc0aa5f0 con 0x7faed406c200 2024-01-15T18:23:41.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.153+0000 7faed3e00640 1 -- 172.21.15.27:0/1758692861 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faebc0aa750 con 0x7faed406c200 2024-01-15T18:23:41.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.153+0000 7faed3e00640 1 -- 172.21.15.27:0/1758692861 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc0aaae0 con 0x7faed406c200 2024-01-15T18:23:41.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.153+0000 7faedbe00640 1 -- 172.21.15.27:0/1758692861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faed406c200 msgr2=0x7faed40cb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:41.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.153+0000 7faedbe00640 1 --2- 172.21.15.27:0/1758692861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faed406c200 0x7faed40cb9c0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7faed40ae790 tx=0x7faebc01c4d0 comp rx=0 tx=0).stop 2024-01-15T18:23:41.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.153+0000 7faedbe00640 1 -- 172.21.15.27:0/1758692861 shutdown_connections 2024-01-15T18:23:41.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.153+0000 7faedbe00640 1 --2- 172.21.15.27:0/1758692861 >> v2:172.21.15.93:3300/0 conn(0x7faed40cad40 0x7faed40cb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:41.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.153+0000 7faedbe00640 1 --2- 172.21.15.27:0/1758692861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faed406c200 0x7faed40cb9c0 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:41.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.153+0000 7faedbe00640 1 --2- 172.21.15.27:0/1758692861 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faed4143da0 0x7faed406d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:41.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.153+0000 7faedbe00640 1 -- 172.21.15.27:0/1758692861 >> 172.21.15.27:0/1758692861 conn(0x7faed40b0850 msgr2=0x7faed40b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:41.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.153+0000 7faedbe00640 1 -- 172.21.15.27:0/1758692861 shutdown_connections 2024-01-15T18:23:41.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.153+0000 7faedbe00640 1 -- 172.21.15.27:0/1758692861 wait complete. 2024-01-15T18:23:41.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.157+0000 7faedbe00640 1 Processor -- start 2024-01-15T18:23:41.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.157+0000 7faedbe00640 1 -- start start 2024-01-15T18:23:41.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.157+0000 7faedbe00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faed406c200 0x7faed413f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:41.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.157+0000 7faedbe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faed40cad40 0x7faed40b3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:41.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.157+0000 7faedbe00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faed4143da0 0x7faed4138640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:41.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.157+0000 7faedbe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed4141260 con 0x7faed40cad40 2024-01-15T18:23:41.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.157+0000 7faedbe00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed4141500 con 0x7faed406c200 2024-01-15T18:23:41.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.157+0000 7faedbe00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed41417e0 con 0x7faed4143da0 2024-01-15T18:23:41.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.157+0000 7faed8a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faed40cad40 0x7faed40b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:41.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.157+0000 7faed8a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faed40cad40 0x7faed40b3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47788/0 (socket says 172.21.15.27:47788) 2024-01-15T18:23:41.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.157+0000 7faed8a00640 1 -- 172.21.15.27:0/643247112 learned_addr learned my addr 172.21.15.27:0/643247112 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:41.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.157+0000 7faed9400640 1 --2- 172.21.15.27:0/643247112 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faed406c200 0x7faed413f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:41.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.157+0000 7faed9e00640 1 --2- 172.21.15.27:0/643247112 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faed4143da0 0x7faed4138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:41.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.157+0000 7faed8a00640 1 -- 172.21.15.27:0/643247112 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faed4143da0 msgr2=0x7faed4138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:41.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.157+0000 7faed8a00640 1 --2- 172.21.15.27:0/643247112 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faed4143da0 0x7faed4138640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:41.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.157+0000 7faed8a00640 1 -- 172.21.15.27:0/643247112 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faed406c200 msgr2=0x7faed413f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:41.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.157+0000 7faed8a00640 1 --2- 172.21.15.27:0/643247112 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faed406c200 0x7faed413f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:41.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.157+0000 7faed8a00640 1 -- 172.21.15.27:0/643247112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faebc0094d0 con 0x7faed40cad40 2024-01-15T18:23:41.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.157+0000 7faed8a00640 1 --2- 172.21.15.27:0/643247112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faed40cad40 0x7faed40b3290 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7faebc01ab80 tx=0x7faebc01ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:41.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.157+0000 7faed1600640 1 -- 172.21.15.27:0/643247112 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc0bd5f0 con 0x7faed40cad40 2024-01-15T18:23:41.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.157+0000 7faed1600640 1 -- 172.21.15.27:0/643247112 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faebc0bd750 con 0x7faed40cad40 2024-01-15T18:23:41.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.157+0000 7faed1600640 1 -- 172.21.15.27:0/643247112 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc0bdb00 con 0x7faed40cad40 2024-01-15T18:23:41.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.157+0000 7faedbe00640 1 -- 172.21.15.27:0/643247112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed4141ac0 con 0x7faed40cad40 2024-01-15T18:23:41.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.157+0000 7faedbe00640 1 -- 172.21.15.27:0/643247112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed4139920 con 0x7faed40cad40 2024-01-15T18:23:41.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.161+0000 7faedbe00640 1 -- 172.21.15.27:0/643247112 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faed404e850 con 0x7faed40cad40 2024-01-15T18:23:41.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.161+0000 7faed1600640 1 -- 172.21.15.27:0/643247112 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7faebc0b33f0 con 0x7faed40cad40 2024-01-15T18:23:41.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.161+0000 7faed1600640 1 --2- 172.21.15.27:0/643247112 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faeac067a90 0x7faeac069f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:41.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.161+0000 7faed1600640 1 -- 172.21.15.27:0/643247112 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7faebc0a7070 con 0x7faed40cad40 2024-01-15T18:23:41.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.161+0000 7faed9400640 1 --2- 172.21.15.27:0/643247112 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faeac067a90 0x7faeac069f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:41.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.161+0000 7faed9400640 1 --2- 172.21.15.27:0/643247112 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faeac067a90 0x7faeac069f50 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7faec4009a40 tx=0x7faec4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:41.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.165+0000 7faed1600640 1 -- 172.21.15.27:0/643247112 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7faebc0ec7f0 con 0x7faed40cad40 2024-01-15T18:23:41.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.321+0000 7faedbe00640 1 -- 172.21.15.27:0/643247112 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7faed4141670 con 0x7faeac067a90 2024-01-15T18:23:41.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.325+0000 7faed1600640 1 -- 172.21.15.27:0/643247112 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7faed4141670 con 0x7faeac067a90 2024-01-15T18:23:41.320 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 7 to deep-scrub 2024-01-15T18:23:41.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.325+0000 7faedbe00640 1 -- 172.21.15.27:0/643247112 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faeac067a90 msgr2=0x7faeac069f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:41.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.325+0000 7faedbe00640 1 --2- 172.21.15.27:0/643247112 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faeac067a90 0x7faeac069f50 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7faec4009a40 tx=0x7faec4009190 comp rx=0 tx=0).stop 2024-01-15T18:23:41.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.325+0000 7faedbe00640 1 -- 172.21.15.27:0/643247112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faed40cad40 msgr2=0x7faed40b3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:41.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.325+0000 7faedbe00640 1 --2- 172.21.15.27:0/643247112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faed40cad40 0x7faed40b3290 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7faebc01ab80 tx=0x7faebc01ad10 comp rx=0 tx=0).stop 2024-01-15T18:23:41.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.325+0000 7faedbe00640 1 -- 172.21.15.27:0/643247112 shutdown_connections 2024-01-15T18:23:41.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.325+0000 7faedbe00640 1 --2- 172.21.15.27:0/643247112 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7faeac067a90 0x7faeac069f50 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:41.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.325+0000 7faedbe00640 1 --2- 172.21.15.27:0/643247112 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7faed4143da0 0x7faed4138640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:41.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.325+0000 7faedbe00640 1 --2- 172.21.15.27:0/643247112 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faed40cad40 0x7faed40b3290 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:41.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.325+0000 7faedbe00640 1 --2- 172.21.15.27:0/643247112 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7faed406c200 0x7faed413f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:41.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.325+0000 7faedbe00640 1 -- 172.21.15.27:0/643247112 >> 172.21.15.27:0/643247112 conn(0x7faed40b0850 msgr2=0x7faed41423b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:41.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.325+0000 7faedbe00640 1 -- 172.21.15.27:0/643247112 shutdown_connections 2024-01-15T18:23:41.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.325+0000 7faedbe00640 1 -- 172.21.15.27:0/643247112 wait complete. 2024-01-15T18:23:41.343 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:23:41.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.489+0000 7f9a8ec00640 1 Processor -- start 2024-01-15T18:23:41.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.489+0000 7f9a8ec00640 1 -- start start 2024-01-15T18:23:41.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.489+0000 7f9a8ec00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9a880701a0 0x7f9a88070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:41.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.489+0000 7f9a8ec00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9a880c92b0 0x7f9a8806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:41.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.489+0000 7f9a8ec00640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a8806c4a0 con 0x7f9a8806e240 2024-01-15T18:23:41.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.489+0000 7f9a8ec00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a88070ac0 con 0x7f9a880701a0 2024-01-15T18:23:41.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.489+0000 7f9a8ec00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8815c660 con 0x7f9a880c92b0 2024-01-15T18:23:41.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.489+0000 7f9a87e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9a880701a0 0x7f9a88070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:41.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.489+0000 7f9a87e00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9a880701a0 0x7f9a88070580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47794/0 (socket says 172.21.15.27:47794) 2024-01-15T18:23:41.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.489+0000 7f9a87e00640 1 -- 172.21.15.27:0/3012420023 learned_addr learned my addr 172.21.15.27:0/3012420023 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:41.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.489+0000 7f9a87400640 1 --2- 172.21.15.27:0/3012420023 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9a880c92b0 0x7f9a8806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:41.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.489+0000 7f9a87e00640 1 -- 172.21.15.27:0/3012420023 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9a880c92b0 msgr2=0x7f9a8806dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:23:41.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.489+0000 7f9a87e00640 1 --2- 172.21.15.27:0/3012420023 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9a880c92b0 0x7f9a8806dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:41.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.489+0000 7f9a87e00640 1 -- 172.21.15.27:0/3012420023 >> v1:172.21.15.93:6789/0 conn(0x7f9a8806e240 legacy=0x7f9a8815d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:41.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.489+0000 7f9a87e00640 1 -- 172.21.15.27:0/3012420023 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a8815c940 con 0x7f9a880701a0 2024-01-15T18:23:41.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.489+0000 7f9a87400640 1 --2- 172.21.15.27:0/3012420023 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9a880c92b0 0x7f9a8806dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:23:41.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.493+0000 7f9a87e00640 1 --2- 172.21.15.27:0/3012420023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a880701a0 0x7f9a88070580 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f9a78009870 tx=0x7f9a7801c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b157af9422343fe1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:41.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.493+0000 7f9a86a00640 1 -- 172.21.15.27:0/3012420023 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a780aa670 con 0x7f9a880701a0 2024-01-15T18:23:41.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.493+0000 7f9a86a00640 1 -- 172.21.15.27:0/3012420023 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a780aa7d0 con 0x7f9a880701a0 2024-01-15T18:23:41.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.493+0000 7f9a86a00640 1 -- 172.21.15.27:0/3012420023 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a780aab60 con 0x7f9a880701a0 2024-01-15T18:23:41.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.493+0000 7f9a8ec00640 1 -- 172.21.15.27:0/3012420023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a880701a0 msgr2=0x7f9a88070580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:41.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.493+0000 7f9a8ec00640 1 --2- 172.21.15.27:0/3012420023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a880701a0 0x7f9a88070580 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f9a78009870 tx=0x7f9a7801c510 comp rx=0 tx=0).stop 2024-01-15T18:23:41.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.493+0000 7f9a8ec00640 1 -- 172.21.15.27:0/3012420023 shutdown_connections 2024-01-15T18:23:41.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.493+0000 7f9a8ec00640 1 --2- 172.21.15.27:0/3012420023 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9a880c92b0 0x7f9a8806dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:41.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.493+0000 7f9a8ec00640 1 --2- 172.21.15.27:0/3012420023 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a880701a0 0x7f9a88070580 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:41.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.493+0000 7f9a8ec00640 1 -- 172.21.15.27:0/3012420023 >> 172.21.15.27:0/3012420023 conn(0x7f9a880cec60 msgr2=0x7f9a880cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:41.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.493+0000 7f9a8ec00640 1 -- 172.21.15.27:0/3012420023 shutdown_connections 2024-01-15T18:23:41.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.493+0000 7f9a8ec00640 1 -- 172.21.15.27:0/3012420023 wait complete. 2024-01-15T18:23:41.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.497+0000 7f9a8ec00640 1 Processor -- start 2024-01-15T18:23:41.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.497+0000 7f9a8ec00640 1 -- start start 2024-01-15T18:23:41.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.497+0000 7f9a8ec00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9a8806e240 0x7f9a881590c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:41.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.497+0000 7f9a8ec00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9a880701a0 0x7f9a880acaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:41.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.497+0000 7f9a8ec00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a880c92b0 0x7f9a88159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:41.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.497+0000 7f9a8ec00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a88158c30 con 0x7f9a880c92b0 2024-01-15T18:23:41.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.497+0000 7f9a8ec00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a88159600 con 0x7f9a8806e240 2024-01-15T18:23:41.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.497+0000 7f9a8ec00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a88158340 con 0x7f9a880701a0 2024-01-15T18:23:41.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.497+0000 7f9a8cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a880c92b0 0x7f9a88159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:41.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.497+0000 7f9a8cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a880c92b0 0x7f9a88159d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:47796/0 (socket says 172.21.15.27:47796) 2024-01-15T18:23:41.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.497+0000 7f9a8cc00640 1 -- 172.21.15.27:0/3718592508 learned_addr learned my addr 172.21.15.27:0/3718592508 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:23:41.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.497+0000 7f9a87e00640 1 --2- 172.21.15.27:0/3718592508 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9a8806e240 0x7f9a881590c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:41.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.497+0000 7f9a87400640 1 --2- 172.21.15.27:0/3718592508 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9a880701a0 0x7f9a880acaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:41.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.497+0000 7f9a8cc00640 1 -- 172.21.15.27:0/3718592508 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9a880701a0 msgr2=0x7f9a880acaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:41.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.497+0000 7f9a8cc00640 1 --2- 172.21.15.27:0/3718592508 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9a880701a0 0x7f9a880acaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:41.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.497+0000 7f9a8cc00640 1 -- 172.21.15.27:0/3718592508 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9a8806e240 msgr2=0x7f9a881590c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:41.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.497+0000 7f9a8cc00640 1 --2- 172.21.15.27:0/3718592508 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9a8806e240 0x7f9a881590c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:41.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.497+0000 7f9a8cc00640 1 -- 172.21.15.27:0/3718592508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a780094d0 con 0x7f9a880c92b0 2024-01-15T18:23:41.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.497+0000 7f9a87e00640 1 --2- 172.21.15.27:0/3718592508 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9a8806e240 0x7f9a881590c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:23:41.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.497+0000 7f9a8cc00640 1 --2- 172.21.15.27:0/3718592508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a880c92b0 0x7f9a88159d80 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f9a8000c7a0 tx=0x7f9a8000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:41.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.497+0000 7f9a7fe00640 1 -- 172.21.15.27:0/3718592508 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a80003b80 con 0x7f9a880c92b0 2024-01-15T18:23:41.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.497+0000 7f9a7fe00640 1 -- 172.21.15.27:0/3718592508 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a80003ce0 con 0x7f9a880c92b0 2024-01-15T18:23:41.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.497+0000 7f9a8ec00640 1 -- 172.21.15.27:0/3718592508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a8815a820 con 0x7f9a880c92b0 2024-01-15T18:23:41.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.497+0000 7f9a7fe00640 1 -- 172.21.15.27:0/3718592508 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8000aa30 con 0x7f9a880c92b0 2024-01-15T18:23:41.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.497+0000 7f9a8ec00640 1 -- 172.21.15.27:0/3718592508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a881588c0 con 0x7f9a880c92b0 2024-01-15T18:23:41.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.501+0000 7f9a8ec00640 1 -- 172.21.15.27:0/3718592508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a88158da0 con 0x7f9a880c92b0 2024-01-15T18:23:41.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.501+0000 7f9a7fe00640 1 -- 172.21.15.27:0/3718592508 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f9a8000ab90 con 0x7f9a880c92b0 2024-01-15T18:23:41.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.501+0000 7f9a7fe00640 1 --2- 172.21.15.27:0/3718592508 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9a60067a60 0x7f9a60069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:23:41.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.501+0000 7f9a7fe00640 1 -- 172.21.15.27:0/3718592508 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f9a80083050 con 0x7f9a880c92b0 2024-01-15T18:23:41.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.501+0000 7f9a87e00640 1 --2- 172.21.15.27:0/3718592508 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9a60067a60 0x7f9a60069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:23:41.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.501+0000 7f9a87e00640 1 --2- 172.21.15.27:0/3718592508 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9a60067a60 0x7f9a60069f20 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f9a780ae000 tx=0x7f9a780b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:23:41.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.505+0000 7f9a7fe00640 1 -- 172.21.15.27:0/3718592508 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9a80051a40 con 0x7f9a880c92b0 2024-01-15T18:23:41.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.657+0000 7f9a8ec00640 1 -- 172.21.15.27:0/3718592508 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9a8815d4c0 con 0x7f9a60067a60 2024-01-15T18:23:41.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.665+0000 7f9a7fe00640 1 -- 172.21.15.27:0/3718592508 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289456 (secure 0 0 0) 0x7f9a780aa4f0 con 0x7f9a60067a60 2024-01-15T18:23:41.666 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:23:41.667 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:23:41.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.669+0000 7f9a8ec00640 1 -- 172.21.15.27:0/3718592508 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9a60067a60 msgr2=0x7f9a60069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:41.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.669+0000 7f9a8ec00640 1 --2- 172.21.15.27:0/3718592508 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9a60067a60 0x7f9a60069f20 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f9a780ae000 tx=0x7f9a780b5040 comp rx=0 tx=0).stop 2024-01-15T18:23:41.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.669+0000 7f9a8ec00640 1 -- 172.21.15.27:0/3718592508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a880c92b0 msgr2=0x7f9a88159d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:23:41.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.669+0000 7f9a8ec00640 1 --2- 172.21.15.27:0/3718592508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a880c92b0 0x7f9a88159d80 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f9a8000c7a0 tx=0x7f9a8000cc60 comp rx=0 tx=0).stop 2024-01-15T18:23:41.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.669+0000 7f9a8ec00640 1 -- 172.21.15.27:0/3718592508 shutdown_connections 2024-01-15T18:23:41.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.669+0000 7f9a8ec00640 1 --2- 172.21.15.27:0/3718592508 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f9a60067a60 0x7f9a60069f20 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:41.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.669+0000 7f9a8ec00640 1 --2- 172.21.15.27:0/3718592508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a880c92b0 0x7f9a88159d80 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:41.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.669+0000 7f9a8ec00640 1 --2- 172.21.15.27:0/3718592508 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f9a880701a0 0x7f9a880acaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:41.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.669+0000 7f9a8ec00640 1 --2- 172.21.15.27:0/3718592508 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f9a8806e240 0x7f9a881590c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:23:41.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.669+0000 7f9a8ec00640 1 -- 172.21.15.27:0/3718592508 >> 172.21.15.27:0/3718592508 conn(0x7f9a880cec60 msgr2=0x7f9a88071a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:23:41.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.669+0000 7f9a8ec00640 1 -- 172.21.15.27:0/3718592508 shutdown_connections 2024-01-15T18:23:41.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:23:41.669+0000 7f9a8ec00640 1 -- 172.21.15.27:0/3718592508 wait complete. 2024-01-15T18:23:41.687 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":671,"stamp":"2024-01-15T18:23:40.458738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5369913756,"num_objects":1311,"num_object_clones":1,"num_object_copies":2622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1311,"num_whiteouts":1,"num_read":651,"num_read_kb":2097275,"num_write":1741,"num_write_kb":5246027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3224,"ondisk_log_size":3224,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":11036512,"kb_used_data":10498848,"kb_used_omap":12,"kb_used_meta":537587,"kb_avail":743938208,"statfs":{"total":773094113280,"available":761792724992,"internally_reserved":0,"allocated":10750820352,"data_stored":10744583216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":550489648},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1073785781,"num_objects":256,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":256,"num_whiteouts":0,"num_read":256,"num_read_kb":1048576,"num_write":275,"num_write_kb":1048624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003125"},"pg_stats":[{"pgid":"17.3f","version":"110'53","reported_seq":78,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.666198+0000","last_change":"2024-01-15T18:18:33.168792+0000","last_active":"2024-01-15T18:23:31.666198+0000","last_peered":"2024-01-15T18:23:31.666198+0000","last_clean":"2024-01-15T18:23:31.666198+0000","last_became_active":"2024-01-15T18:18:33.168500+0000","last_became_peered":"2024-01-15T18:18:33.168500+0000","last_unstale":"2024-01-15T18:23:31.666198+0000","last_undegraded":"2024-01-15T18:23:31.666198+0000","last_fullsized":"2024-01-15T18:23:31.666198+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"110'45","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.710301+0000","last_change":"2024-01-15T18:18:33.151630+0000","last_active":"2024-01-15T18:23:31.710301+0000","last_peered":"2024-01-15T18:23:31.710301+0000","last_clean":"2024-01-15T18:23:31.710301+0000","last_became_active":"2024-01-15T18:18:33.151209+0000","last_became_peered":"2024-01-15T18:18:33.151209+0000","last_unstale":"2024-01-15T18:23:31.710301+0000","last_undegraded":"2024-01-15T18:23:31.710301+0000","last_fullsized":"2024-01-15T18:23:31.710301+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"110'44","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.820489+0000","last_change":"2024-01-15T18:18:33.163094+0000","last_active":"2024-01-15T18:23:31.820489+0000","last_peered":"2024-01-15T18:23:31.820489+0000","last_clean":"2024-01-15T18:23:31.820489+0000","last_became_active":"2024-01-15T18:18:33.162700+0000","last_became_peered":"2024-01-15T18:18:33.162700+0000","last_unstale":"2024-01-15T18:23:31.820489+0000","last_undegraded":"2024-01-15T18:23:31.820489+0000","last_fullsized":"2024-01-15T18:23:31.820489+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:23:36.167165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"110'62","reported_seq":92,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:37.761746+0000","last_change":"2024-01-15T18:23:37.761794+0000","last_active":"2024-01-15T18:23:37.761746+0000","last_peered":"2024-01-15T18:23:37.761746+0000","last_clean":"2024-01-15T18:23:37.761746+0000","last_became_active":"2024-01-15T18:18:33.160460+0000","last_became_peered":"2024-01-15T18:18:33.160460+0000","last_unstale":"2024-01-15T18:23:37.761746+0000","last_undegraded":"2024-01-15T18:23:37.761746+0000","last_fullsized":"2024-01-15T18:23:37.761746+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'62","last_scrub_stamp":"2024-01-15T18:23:37.761665+0000","last_deep_scrub":"110'62","last_deep_scrub_stamp":"2024-01-15T18:23:37.761665+0000","last_clean_scrub_stamp":"2024-01-15T18:23:37.761665+0000","objects_scrubbed":23,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:24:28.403935+0000","scrub_duration":0.14250365100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"110'27","reported_seq":50,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:35.184846+0000","last_change":"2024-01-15T18:18:33.155936+0000","last_active":"2024-01-15T18:23:35.184846+0000","last_peered":"2024-01-15T18:23:35.184846+0000","last_clean":"2024-01-15T18:23:35.184846+0000","last_became_active":"2024-01-15T18:18:33.155656+0000","last_became_peered":"2024-01-15T18:18:33.155656+0000","last_unstale":"2024-01-15T18:23:35.184846+0000","last_undegraded":"2024-01-15T18:23:35.184846+0000","last_fullsized":"2024-01-15T18:23:35.184846+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"110'48","reported_seq":74,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.845242+0000","last_change":"2024-01-15T18:18:33.152032+0000","last_active":"2024-01-15T18:23:31.845242+0000","last_peered":"2024-01-15T18:23:31.845242+0000","last_clean":"2024-01-15T18:23:31.845242+0000","last_became_active":"2024-01-15T18:18:33.151701+0000","last_became_peered":"2024-01-15T18:18:33.151701+0000","last_unstale":"2024-01-15T18:23:31.845242+0000","last_undegraded":"2024-01-15T18:23:31.845242+0000","last_fullsized":"2024-01-15T18:23:31.845242+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:51.983410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"110'43","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.556273+0000","last_change":"2024-01-15T18:18:33.153882+0000","last_active":"2024-01-15T18:23:31.556273+0000","last_peered":"2024-01-15T18:23:31.556273+0000","last_clean":"2024-01-15T18:23:31.556273+0000","last_became_active":"2024-01-15T18:18:33.153264+0000","last_became_peered":"2024-01-15T18:18:33.153264+0000","last_unstale":"2024-01-15T18:23:31.556273+0000","last_undegraded":"2024-01-15T18:23:31.556273+0000","last_fullsized":"2024-01-15T18:23:31.556273+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:37:49.663025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"110'44","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.520555+0000","last_change":"2024-01-15T18:18:33.164714+0000","last_active":"2024-01-15T18:23:31.520555+0000","last_peered":"2024-01-15T18:23:31.520555+0000","last_clean":"2024-01-15T18:23:31.520555+0000","last_became_active":"2024-01-15T18:18:33.163896+0000","last_became_peered":"2024-01-15T18:18:33.163896+0000","last_unstale":"2024-01-15T18:23:31.520555+0000","last_undegraded":"2024-01-15T18:23:31.520555+0000","last_fullsized":"2024-01-15T18:23:31.520555+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:21:18.264946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"110'47","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.030521+0000","last_change":"2024-01-15T18:18:33.158682+0000","last_active":"2024-01-15T18:23:32.030521+0000","last_peered":"2024-01-15T18:23:32.030521+0000","last_clean":"2024-01-15T18:23:32.030521+0000","last_became_active":"2024-01-15T18:18:33.158399+0000","last_became_peered":"2024-01-15T18:18:33.158399+0000","last_unstale":"2024-01-15T18:23:32.030521+0000","last_undegraded":"2024-01-15T18:23:32.030521+0000","last_fullsized":"2024-01-15T18:23:32.030521+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"110'47","reported_seq":73,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.054699+0000","last_change":"2024-01-15T18:18:33.159936+0000","last_active":"2024-01-15T18:23:32.054699+0000","last_peered":"2024-01-15T18:23:32.054699+0000","last_clean":"2024-01-15T18:23:32.054699+0000","last_became_active":"2024-01-15T18:18:33.159562+0000","last_became_peered":"2024-01-15T18:18:33.159562+0000","last_unstale":"2024-01-15T18:23:32.054699+0000","last_undegraded":"2024-01-15T18:23:32.054699+0000","last_fullsized":"2024-01-15T18:23:32.054699+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"110'47","reported_seq":75,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.998202+0000","last_change":"2024-01-15T18:18:33.155247+0000","last_active":"2024-01-15T18:23:31.998202+0000","last_peered":"2024-01-15T18:23:31.998202+0000","last_clean":"2024-01-15T18:23:31.998202+0000","last_became_active":"2024-01-15T18:18:33.154972+0000","last_became_peered":"2024-01-15T18:18:33.154972+0000","last_unstale":"2024-01-15T18:23:31.998202+0000","last_undegraded":"2024-01-15T18:23:31.998202+0000","last_fullsized":"2024-01-15T18:23:31.998202+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"110'48","reported_seq":80,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.959268+0000","last_change":"2024-01-15T18:18:33.158336+0000","last_active":"2024-01-15T18:23:31.959268+0000","last_peered":"2024-01-15T18:23:31.959268+0000","last_clean":"2024-01-15T18:23:31.959268+0000","last_became_active":"2024-01-15T18:18:33.158064+0000","last_became_peered":"2024-01-15T18:18:33.158064+0000","last_unstale":"2024-01-15T18:23:31.959268+0000","last_undegraded":"2024-01-15T18:23:31.959268+0000","last_fullsized":"2024-01-15T18:23:31.959268+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":16,"num_read_kb":61441,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"110'38","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.273591+0000","last_change":"2024-01-15T18:18:33.163912+0000","last_active":"2024-01-15T18:23:31.273591+0000","last_peered":"2024-01-15T18:23:31.273591+0000","last_clean":"2024-01-15T18:23:31.273591+0000","last_became_active":"2024-01-15T18:18:33.163610+0000","last_became_peered":"2024-01-15T18:18:33.163610+0000","last_unstale":"2024-01-15T18:23:31.273591+0000","last_undegraded":"2024-01-15T18:23:31.273591+0000","last_fullsized":"2024-01-15T18:23:31.273591+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:32:43.468374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"110'51","reported_seq":73,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.936664+0000","last_change":"2024-01-15T18:18:33.169868+0000","last_active":"2024-01-15T18:23:31.936664+0000","last_peered":"2024-01-15T18:23:31.936664+0000","last_clean":"2024-01-15T18:23:31.936664+0000","last_became_active":"2024-01-15T18:18:33.169530+0000","last_became_peered":"2024-01-15T18:18:33.169530+0000","last_unstale":"2024-01-15T18:23:31.936664+0000","last_undegraded":"2024-01-15T18:23:31.936664+0000","last_fullsized":"2024-01-15T18:23:31.936664+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:18:22.473056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"110'34","reported_seq":53,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.867842+0000","last_change":"2024-01-15T18:18:33.151325+0000","last_active":"2024-01-15T18:23:31.867842+0000","last_peered":"2024-01-15T18:23:31.867842+0000","last_clean":"2024-01-15T18:23:31.867842+0000","last_became_active":"2024-01-15T18:18:33.151040+0000","last_became_peered":"2024-01-15T18:18:33.151040+0000","last_unstale":"2024-01-15T18:23:31.867842+0000","last_undegraded":"2024-01-15T18:23:31.867842+0000","last_fullsized":"2024-01-15T18:23:31.867842+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:57:08.165322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"110'42","reported_seq":61,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.667801+0000","last_change":"2024-01-15T18:18:33.166327+0000","last_active":"2024-01-15T18:23:31.667801+0000","last_peered":"2024-01-15T18:23:31.667801+0000","last_clean":"2024-01-15T18:23:31.667801+0000","last_became_active":"2024-01-15T18:18:33.166014+0000","last_became_peered":"2024-01-15T18:18:33.166014+0000","last_unstale":"2024-01-15T18:23:31.667801+0000","last_undegraded":"2024-01-15T18:23:31.667801+0000","last_fullsized":"2024-01-15T18:23:31.667801+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"110'44","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.912923+0000","last_change":"2024-01-15T18:18:33.168613+0000","last_active":"2024-01-15T18:23:31.912923+0000","last_peered":"2024-01-15T18:23:31.912923+0000","last_clean":"2024-01-15T18:23:31.912923+0000","last_became_active":"2024-01-15T18:18:33.167975+0000","last_became_peered":"2024-01-15T18:18:33.167975+0000","last_unstale":"2024-01-15T18:23:31.912923+0000","last_undegraded":"2024-01-15T18:23:31.912923+0000","last_fullsized":"2024-01-15T18:23:31.912923+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"110'48","reported_seq":76,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.863728+0000","last_change":"2024-01-15T18:18:33.155078+0000","last_active":"2024-01-15T18:23:31.863728+0000","last_peered":"2024-01-15T18:23:31.863728+0000","last_clean":"2024-01-15T18:23:31.863728+0000","last_became_active":"2024-01-15T18:18:33.154816+0000","last_became_peered":"2024-01-15T18:18:33.154816+0000","last_unstale":"2024-01-15T18:23:31.863728+0000","last_undegraded":"2024-01-15T18:23:31.863728+0000","last_fullsized":"2024-01-15T18:23:31.863728+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:24:48.739168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"110'44","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.653674+0000","last_change":"2024-01-15T18:18:33.164028+0000","last_active":"2024-01-15T18:23:31.653674+0000","last_peered":"2024-01-15T18:23:31.653674+0000","last_clean":"2024-01-15T18:23:31.653674+0000","last_became_active":"2024-01-15T18:18:33.163421+0000","last_became_peered":"2024-01-15T18:18:33.163421+0000","last_unstale":"2024-01-15T18:23:31.653674+0000","last_undegraded":"2024-01-15T18:23:31.653674+0000","last_fullsized":"2024-01-15T18:23:31.653674+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:38:31.615856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"110'46","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.956031+0000","last_change":"2024-01-15T18:18:33.160995+0000","last_active":"2024-01-15T18:23:31.956031+0000","last_peered":"2024-01-15T18:23:31.956031+0000","last_clean":"2024-01-15T18:23:31.956031+0000","last_became_active":"2024-01-15T18:18:33.160047+0000","last_became_peered":"2024-01-15T18:18:33.160047+0000","last_unstale":"2024-01-15T18:23:31.956031+0000","last_undegraded":"2024-01-15T18:23:31.956031+0000","last_fullsized":"2024-01-15T18:23:31.956031+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"110'56","reported_seq":85,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.026650+0000","last_change":"2024-01-15T18:18:33.152696+0000","last_active":"2024-01-15T18:23:32.026650+0000","last_peered":"2024-01-15T18:23:32.026650+0000","last_clean":"2024-01-15T18:23:32.026650+0000","last_became_active":"2024-01-15T18:18:33.152128+0000","last_became_peered":"2024-01-15T18:18:33.152128+0000","last_unstale":"2024-01-15T18:23:32.026650+0000","last_undegraded":"2024-01-15T18:23:32.026650+0000","last_fullsized":"2024-01-15T18:23:32.026650+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"110'47","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.064950+0000","last_change":"2024-01-15T18:18:33.163614+0000","last_active":"2024-01-15T18:23:32.064950+0000","last_peered":"2024-01-15T18:23:32.064950+0000","last_clean":"2024-01-15T18:23:32.064950+0000","last_became_active":"2024-01-15T18:18:33.163252+0000","last_became_peered":"2024-01-15T18:18:33.163252+0000","last_unstale":"2024-01-15T18:23:32.064950+0000","last_undegraded":"2024-01-15T18:23:32.064950+0000","last_fullsized":"2024-01-15T18:23:32.064950+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":24578,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"110'39","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.847727+0000","last_change":"2024-01-15T18:18:33.159896+0000","last_active":"2024-01-15T18:23:31.847727+0000","last_peered":"2024-01-15T18:23:31.847727+0000","last_clean":"2024-01-15T18:23:31.847727+0000","last_became_active":"2024-01-15T18:18:33.159557+0000","last_became_peered":"2024-01-15T18:18:33.159557+0000","last_unstale":"2024-01-15T18:23:31.847727+0000","last_undegraded":"2024-01-15T18:23:31.847727+0000","last_fullsized":"2024-01-15T18:23:31.847727+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"110'43","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.014236+0000","last_change":"2024-01-15T18:18:33.166766+0000","last_active":"2024-01-15T18:23:32.014236+0000","last_peered":"2024-01-15T18:23:32.014236+0000","last_clean":"2024-01-15T18:23:32.014236+0000","last_became_active":"2024-01-15T18:18:33.165600+0000","last_became_peered":"2024-01-15T18:18:33.165600+0000","last_unstale":"2024-01-15T18:23:32.014236+0000","last_undegraded":"2024-01-15T18:23:32.014236+0000","last_fullsized":"2024-01-15T18:23:32.014236+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"110'38","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.985981+0000","last_change":"2024-01-15T18:18:33.168720+0000","last_active":"2024-01-15T18:23:31.985981+0000","last_peered":"2024-01-15T18:23:31.985981+0000","last_clean":"2024-01-15T18:23:31.985981+0000","last_became_active":"2024-01-15T18:18:33.168349+0000","last_became_peered":"2024-01-15T18:18:33.168349+0000","last_unstale":"2024-01-15T18:23:31.985981+0000","last_undegraded":"2024-01-15T18:23:31.985981+0000","last_fullsized":"2024-01-15T18:23:31.985981+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"110'53","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.034275+0000","last_change":"2024-01-15T18:18:33.159262+0000","last_active":"2024-01-15T18:23:32.034275+0000","last_peered":"2024-01-15T18:23:32.034275+0000","last_clean":"2024-01-15T18:23:32.034275+0000","last_became_active":"2024-01-15T18:18:33.158946+0000","last_became_peered":"2024-01-15T18:18:33.158946+0000","last_unstale":"2024-01-15T18:23:32.034275+0000","last_undegraded":"2024-01-15T18:23:32.034275+0000","last_fullsized":"2024-01-15T18:23:32.034275+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"110'37","reported_seq":60,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.223711+0000","last_change":"2024-01-15T18:18:33.161814+0000","last_active":"2024-01-15T18:23:31.223711+0000","last_peered":"2024-01-15T18:23:31.223711+0000","last_clean":"2024-01-15T18:23:31.223711+0000","last_became_active":"2024-01-15T18:18:33.161463+0000","last_became_peered":"2024-01-15T18:18:33.161463+0000","last_unstale":"2024-01-15T18:23:31.223711+0000","last_undegraded":"2024-01-15T18:23:31.223711+0000","last_fullsized":"2024-01-15T18:23:31.223711+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:15:36.391855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886260,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":28673,"num_write":21,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"110'38","reported_seq":57,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.030549+0000","last_change":"2024-01-15T18:18:33.162952+0000","last_active":"2024-01-15T18:23:32.030549+0000","last_peered":"2024-01-15T18:23:32.030549+0000","last_clean":"2024-01-15T18:23:32.030549+0000","last_became_active":"2024-01-15T18:18:33.162669+0000","last_became_peered":"2024-01-15T18:18:33.162669+0000","last_unstale":"2024-01-15T18:23:32.030549+0000","last_undegraded":"2024-01-15T18:23:32.030549+0000","last_fullsized":"2024-01-15T18:23:32.030549+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"110'46","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.968407+0000","last_change":"2024-01-15T18:18:33.160658+0000","last_active":"2024-01-15T18:23:31.968407+0000","last_peered":"2024-01-15T18:23:31.968407+0000","last_clean":"2024-01-15T18:23:31.968407+0000","last_became_active":"2024-01-15T18:18:33.160099+0000","last_became_peered":"2024-01-15T18:18:33.160099+0000","last_unstale":"2024-01-15T18:23:31.968407+0000","last_undegraded":"2024-01-15T18:23:31.968407+0000","last_fullsized":"2024-01-15T18:23:31.968407+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"110'41","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.812331+0000","last_change":"2024-01-15T18:18:33.161061+0000","last_active":"2024-01-15T18:23:31.812331+0000","last_peered":"2024-01-15T18:23:31.812331+0000","last_clean":"2024-01-15T18:23:31.812331+0000","last_became_active":"2024-01-15T18:18:33.160725+0000","last_became_peered":"2024-01-15T18:18:33.160725+0000","last_unstale":"2024-01-15T18:23:31.812331+0000","last_undegraded":"2024-01-15T18:23:31.812331+0000","last_fullsized":"2024-01-15T18:23:31.812331+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:38:07.236956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"110'62","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.918200+0000","last_change":"2024-01-15T18:18:33.159024+0000","last_active":"2024-01-15T18:23:31.918200+0000","last_peered":"2024-01-15T18:23:31.918200+0000","last_clean":"2024-01-15T18:23:31.918200+0000","last_became_active":"2024-01-15T18:18:33.158701+0000","last_became_peered":"2024-01-15T18:18:33.158701+0000","last_unstale":"2024-01-15T18:23:31.918200+0000","last_undegraded":"2024-01-15T18:23:31.918200+0000","last_fullsized":"2024-01-15T18:23:31.918200+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"110'54","reported_seq":76,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.735592+0000","last_change":"2024-01-15T18:18:33.152241+0000","last_active":"2024-01-15T18:23:31.735592+0000","last_peered":"2024-01-15T18:23:31.735592+0000","last_clean":"2024-01-15T18:23:31.735592+0000","last_became_active":"2024-01-15T18:18:33.151250+0000","last_became_peered":"2024-01-15T18:18:33.151250+0000","last_unstale":"2024-01-15T18:23:31.735592+0000","last_undegraded":"2024-01-15T18:23:31.735592+0000","last_fullsized":"2024-01-15T18:23:31.735592+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:45:02.525155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"110'34","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.551784+0000","last_change":"2024-01-15T18:18:33.154972+0000","last_active":"2024-01-15T18:23:31.551784+0000","last_peered":"2024-01-15T18:23:31.551784+0000","last_clean":"2024-01-15T18:23:31.551784+0000","last_became_active":"2024-01-15T18:18:33.154569+0000","last_became_peered":"2024-01-15T18:18:33.154569+0000","last_unstale":"2024-01-15T18:23:31.551784+0000","last_undegraded":"2024-01-15T18:23:31.551784+0000","last_fullsized":"2024-01-15T18:23:31.551784+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:29:02.332324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"107'4","reported_seq":20,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:39.170083+0000","last_change":"2024-01-15T18:18:32.217761+0000","last_active":"2024-01-15T18:20:39.170083+0000","last_peered":"2024-01-15T18:20:39.170083+0000","last_clean":"2024-01-15T18:20:39.170083+0000","last_became_active":"2024-01-15T18:18:32.216337+0000","last_became_peered":"2024-01-15T18:18:32.216337+0000","last_unstale":"2024-01-15T18:20:39.170083+0000","last_undegraded":"2024-01-15T18:20:39.170083+0000","last_fullsized":"2024-01-15T18:20:39.170083+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:01:18.991818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"110'47","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.245614+0000","last_change":"2024-01-15T18:18:33.167015+0000","last_active":"2024-01-15T18:23:31.245614+0000","last_peered":"2024-01-15T18:23:31.245614+0000","last_clean":"2024-01-15T18:23:31.245614+0000","last_became_active":"2024-01-15T18:18:33.166726+0000","last_became_peered":"2024-01-15T18:18:33.166726+0000","last_unstale":"2024-01-15T18:23:31.245614+0000","last_undegraded":"2024-01-15T18:23:31.245614+0000","last_fullsized":"2024-01-15T18:23:31.245614+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:48:57.895180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"107'1","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:40.169868+0000","last_change":"2024-01-15T18:18:32.219987+0000","last_active":"2024-01-15T18:20:40.169868+0000","last_peered":"2024-01-15T18:20:40.169868+0000","last_clean":"2024-01-15T18:20:40.169868+0000","last_became_active":"2024-01-15T18:18:32.219596+0000","last_became_peered":"2024-01-15T18:18:32.219596+0000","last_unstale":"2024-01-15T18:20:40.169868+0000","last_undegraded":"2024-01-15T18:20:40.169868+0000","last_fullsized":"2024-01-15T18:20:40.169868+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:23.024473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"110'43","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.034689+0000","last_change":"2024-01-15T18:18:33.150743+0000","last_active":"2024-01-15T18:23:32.034689+0000","last_peered":"2024-01-15T18:23:32.034689+0000","last_clean":"2024-01-15T18:23:32.034689+0000","last_became_active":"2024-01-15T18:18:33.150429+0000","last_became_peered":"2024-01-15T18:18:33.150429+0000","last_unstale":"2024-01-15T18:23:32.034689+0000","last_undegraded":"2024-01-15T18:23:32.034689+0000","last_fullsized":"2024-01-15T18:23:32.034689+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:33:18.994479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"110'3","reported_seq":20,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:35.184876+0000","last_change":"2024-01-15T18:18:32.213207+0000","last_active":"2024-01-15T18:23:35.184876+0000","last_peered":"2024-01-15T18:23:35.184876+0000","last_clean":"2024-01-15T18:23:35.184876+0000","last_became_active":"2024-01-15T18:18:32.212071+0000","last_became_peered":"2024-01-15T18:18:32.212071+0000","last_unstale":"2024-01-15T18:23:35.184876+0000","last_undegraded":"2024-01-15T18:23:35.184876+0000","last_fullsized":"2024-01-15T18:23:35.184876+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"110'56","reported_seq":82,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.898547+0000","last_change":"2024-01-15T18:18:33.165545+0000","last_active":"2024-01-15T18:23:31.898547+0000","last_peered":"2024-01-15T18:23:31.898547+0000","last_clean":"2024-01-15T18:23:31.898547+0000","last_became_active":"2024-01-15T18:18:33.165258+0000","last_became_peered":"2024-01-15T18:18:33.165258+0000","last_unstale":"2024-01-15T18:23:31.898547+0000","last_undegraded":"2024-01-15T18:23:31.898547+0000","last_fullsized":"2024-01-15T18:23:31.898547+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:05:07.453809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"107'2","reported_seq":19,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:35.184865+0000","last_change":"2024-01-15T18:18:32.217238+0000","last_active":"2024-01-15T18:23:35.184865+0000","last_peered":"2024-01-15T18:23:35.184865+0000","last_clean":"2024-01-15T18:23:35.184865+0000","last_became_active":"2024-01-15T18:18:32.216492+0000","last_became_peered":"2024-01-15T18:18:32.216492+0000","last_unstale":"2024-01-15T18:23:35.184865+0000","last_undegraded":"2024-01-15T18:23:35.184865+0000","last_fullsized":"2024-01-15T18:23:35.184865+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"110'39","reported_seq":63,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:34.763315+0000","last_change":"2024-01-15T18:18:33.157996+0000","last_active":"2024-01-15T18:23:34.763315+0000","last_peered":"2024-01-15T18:23:34.763315+0000","last_clean":"2024-01-15T18:23:34.763315+0000","last_became_active":"2024-01-15T18:18:33.157687+0000","last_became_peered":"2024-01-15T18:18:33.157687+0000","last_unstale":"2024-01-15T18:23:34.763315+0000","last_undegraded":"2024-01-15T18:23:34.763315+0000","last_fullsized":"2024-01-15T18:23:34.763315+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"107'1","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:21:04.173272+0000","last_change":"2024-01-15T18:18:32.215007+0000","last_active":"2024-01-15T18:21:04.173272+0000","last_peered":"2024-01-15T18:21:04.173272+0000","last_clean":"2024-01-15T18:21:04.173272+0000","last_became_active":"2024-01-15T18:18:32.214453+0000","last_became_peered":"2024-01-15T18:18:32.214453+0000","last_unstale":"2024-01-15T18:21:04.173272+0000","last_undegraded":"2024-01-15T18:21:04.173272+0000","last_fullsized":"2024-01-15T18:21:04.173272+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:58:50.916786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"110'49","reported_seq":78,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:35.200051+0000","last_change":"2024-01-15T18:18:33.153355+0000","last_active":"2024-01-15T18:23:35.200051+0000","last_peered":"2024-01-15T18:23:35.200051+0000","last_clean":"2024-01-15T18:23:35.200051+0000","last_became_active":"2024-01-15T18:18:33.153058+0000","last_became_peered":"2024-01-15T18:18:33.153058+0000","last_unstale":"2024-01-15T18:23:35.200051+0000","last_undegraded":"2024-01-15T18:23:35.200051+0000","last_fullsized":"2024-01-15T18:23:35.200051+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"107'1","reported_seq":18,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:35.455720+0000","last_change":"2024-01-15T18:18:32.215681+0000","last_active":"2024-01-15T18:23:35.455720+0000","last_peered":"2024-01-15T18:23:35.455720+0000","last_clean":"2024-01-15T18:23:35.455720+0000","last_became_active":"2024-01-15T18:18:32.215417+0000","last_became_peered":"2024-01-15T18:18:32.215417+0000","last_unstale":"2024-01-15T18:23:35.455720+0000","last_undegraded":"2024-01-15T18:23:35.455720+0000","last_fullsized":"2024-01-15T18:23:35.455720+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"110'53","reported_seq":75,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:35.184808+0000","last_change":"2024-01-15T18:18:43.024618+0000","last_active":"2024-01-15T18:23:35.184808+0000","last_peered":"2024-01-15T18:23:35.184808+0000","last_clean":"2024-01-15T18:23:35.184808+0000","last_became_active":"2024-01-15T18:18:43.024221+0000","last_became_peered":"2024-01-15T18:18:43.024221+0000","last_unstale":"2024-01-15T18:23:35.184808+0000","last_undegraded":"2024-01-15T18:23:35.184808+0000","last_fullsized":"2024-01-15T18:23:35.184808+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857792,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:30.199432+0000","last_change":"2024-01-15T18:18:32.217097+0000","last_active":"2024-01-15T18:23:30.199432+0000","last_peered":"2024-01-15T18:23:30.199432+0000","last_clean":"2024-01-15T18:23:30.199432+0000","last_became_active":"2024-01-15T18:18:32.215465+0000","last_became_peered":"2024-01-15T18:18:32.215465+0000","last_unstale":"2024-01-15T18:23:30.199432+0000","last_undegraded":"2024-01-15T18:23:30.199432+0000","last_fullsized":"2024-01-15T18:23:30.199432+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"110'43","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.553963+0000","last_change":"2024-01-15T18:18:33.154067+0000","last_active":"2024-01-15T18:23:31.553963+0000","last_peered":"2024-01-15T18:23:31.553963+0000","last_clean":"2024-01-15T18:23:31.553963+0000","last_became_active":"2024-01-15T18:18:33.153610+0000","last_became_peered":"2024-01-15T18:18:33.153610+0000","last_unstale":"2024-01-15T18:23:31.553963+0000","last_undegraded":"2024-01-15T18:23:31.553963+0000","last_fullsized":"2024-01-15T18:23:31.553963+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:08:41.862115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:24.762059+0000","last_change":"2024-01-15T18:18:32.219909+0000","last_active":"2024-01-15T18:23:24.762059+0000","last_peered":"2024-01-15T18:23:24.762059+0000","last_clean":"2024-01-15T18:23:24.762059+0000","last_became_active":"2024-01-15T18:18:32.219555+0000","last_became_peered":"2024-01-15T18:18:32.219555+0000","last_unstale":"2024-01-15T18:23:24.762059+0000","last_undegraded":"2024-01-15T18:23:24.762059+0000","last_fullsized":"2024-01-15T18:23:24.762059+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"110'42","reported_seq":61,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:30.677222+0000","last_change":"2024-01-15T18:18:33.156038+0000","last_active":"2024-01-15T18:23:30.677222+0000","last_peered":"2024-01-15T18:23:30.677222+0000","last_clean":"2024-01-15T18:23:30.677222+0000","last_became_active":"2024-01-15T18:18:33.155765+0000","last_became_peered":"2024-01-15T18:18:33.155765+0000","last_unstale":"2024-01-15T18:23:30.677222+0000","last_undegraded":"2024-01-15T18:23:30.677222+0000","last_fullsized":"2024-01-15T18:23:30.677222+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:58:43.210443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"110'1","reported_seq":18,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:25.183788+0000","last_change":"2024-01-15T18:18:32.212880+0000","last_active":"2024-01-15T18:23:25.183788+0000","last_peered":"2024-01-15T18:23:25.183788+0000","last_clean":"2024-01-15T18:23:25.183788+0000","last_became_active":"2024-01-15T18:18:32.212511+0000","last_became_peered":"2024-01-15T18:18:32.212511+0000","last_unstale":"2024-01-15T18:23:25.183788+0000","last_undegraded":"2024-01-15T18:23:25.183788+0000","last_fullsized":"2024-01-15T18:23:25.183788+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"110'40","reported_seq":60,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.817223+0000","last_change":"2024-01-15T18:18:33.164800+0000","last_active":"2024-01-15T18:23:31.817223+0000","last_peered":"2024-01-15T18:23:31.817223+0000","last_clean":"2024-01-15T18:23:31.817223+0000","last_became_active":"2024-01-15T18:18:33.164530+0000","last_became_peered":"2024-01-15T18:18:33.164530+0000","last_unstale":"2024-01-15T18:23:31.817223+0000","last_undegraded":"2024-01-15T18:23:31.817223+0000","last_fullsized":"2024-01-15T18:23:31.817223+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:55.175065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:25.454687+0000","last_change":"2024-01-15T18:18:32.216873+0000","last_active":"2024-01-15T18:23:25.454687+0000","last_peered":"2024-01-15T18:23:25.454687+0000","last_clean":"2024-01-15T18:23:25.454687+0000","last_became_active":"2024-01-15T18:18:32.216337+0000","last_became_peered":"2024-01-15T18:18:32.216337+0000","last_unstale":"2024-01-15T18:23:25.454687+0000","last_undegraded":"2024-01-15T18:23:25.454687+0000","last_fullsized":"2024-01-15T18:23:25.454687+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"110'43","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.038280+0000","last_change":"2024-01-15T18:18:33.148278+0000","last_active":"2024-01-15T18:23:32.038280+0000","last_peered":"2024-01-15T18:23:32.038280+0000","last_clean":"2024-01-15T18:23:32.038280+0000","last_became_active":"2024-01-15T18:18:33.148016+0000","last_became_peered":"2024-01-15T18:18:33.148016+0000","last_unstale":"2024-01-15T18:23:32.038280+0000","last_undegraded":"2024-01-15T18:23:32.038280+0000","last_fullsized":"2024-01-15T18:23:32.038280+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"110'12","reported_seq":29,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:24.762133+0000","last_change":"2024-01-15T18:18:32.214473+0000","last_active":"2024-01-15T18:23:24.762133+0000","last_peered":"2024-01-15T18:23:24.762133+0000","last_clean":"2024-01-15T18:23:24.762133+0000","last_became_active":"2024-01-15T18:18:32.214186+0000","last_became_peered":"2024-01-15T18:18:32.214186+0000","last_unstale":"2024-01-15T18:23:24.762133+0000","last_undegraded":"2024-01-15T18:23:24.762133+0000","last_fullsized":"2024-01-15T18:23:24.762133+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"110'43","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.563032+0000","last_change":"2024-01-15T18:18:33.159329+0000","last_active":"2024-01-15T18:23:31.563032+0000","last_peered":"2024-01-15T18:23:31.563032+0000","last_clean":"2024-01-15T18:23:31.563032+0000","last_became_active":"2024-01-15T18:18:33.158541+0000","last_became_peered":"2024-01-15T18:18:33.158541+0000","last_unstale":"2024-01-15T18:23:31.563032+0000","last_undegraded":"2024-01-15T18:23:31.563032+0000","last_fullsized":"2024-01-15T18:23:31.563032+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:53:07.912473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"110'184","reported_seq":201,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:38.332673+0000","last_change":"2024-01-15T18:18:32.218321+0000","last_active":"2024-01-15T18:23:38.332673+0000","last_peered":"2024-01-15T18:23:38.332673+0000","last_clean":"2024-01-15T18:23:38.332673+0000","last_became_active":"2024-01-15T18:18:32.218002+0000","last_became_peered":"2024-01-15T18:18:32.218002+0000","last_unstale":"2024-01-15T18:23:38.332673+0000","last_undegraded":"2024-01-15T18:23:38.332673+0000","last_fullsized":"2024-01-15T18:23:38.332673+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":611133,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"110'44","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.064489+0000","last_change":"2024-01-15T18:18:33.151414+0000","last_active":"2024-01-15T18:23:32.064489+0000","last_peered":"2024-01-15T18:23:32.064489+0000","last_clean":"2024-01-15T18:23:32.064489+0000","last_became_active":"2024-01-15T18:18:33.151132+0000","last_became_peered":"2024-01-15T18:18:33.151132+0000","last_unstale":"2024-01-15T18:23:32.064489+0000","last_undegraded":"2024-01-15T18:23:32.064489+0000","last_fullsized":"2024-01-15T18:23:32.064489+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"107'2","reported_seq":18,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:55.470178+0000","last_change":"2024-01-15T18:18:32.215335+0000","last_active":"2024-01-15T18:20:55.470178+0000","last_peered":"2024-01-15T18:20:55.470178+0000","last_clean":"2024-01-15T18:20:55.470178+0000","last_became_active":"2024-01-15T18:18:32.215073+0000","last_became_peered":"2024-01-15T18:18:32.215073+0000","last_unstale":"2024-01-15T18:20:55.470178+0000","last_undegraded":"2024-01-15T18:20:55.470178+0000","last_fullsized":"2024-01-15T18:20:55.470178+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:04:07.912459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"110'52","reported_seq":73,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.897145+0000","last_change":"2024-01-15T18:18:33.161775+0000","last_active":"2024-01-15T18:23:31.897145+0000","last_peered":"2024-01-15T18:23:31.897145+0000","last_clean":"2024-01-15T18:23:31.897145+0000","last_became_active":"2024-01-15T18:18:33.161493+0000","last_became_peered":"2024-01-15T18:18:33.161493+0000","last_unstale":"2024-01-15T18:23:31.897145+0000","last_undegraded":"2024-01-15T18:23:31.897145+0000","last_fullsized":"2024-01-15T18:23:31.897145+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:55.488186+0000","last_change":"2024-01-15T18:18:32.215225+0000","last_active":"2024-01-15T18:20:55.488186+0000","last_peered":"2024-01-15T18:20:55.488186+0000","last_clean":"2024-01-15T18:20:55.488186+0000","last_became_active":"2024-01-15T18:18:32.214942+0000","last_became_peered":"2024-01-15T18:18:32.214942+0000","last_unstale":"2024-01-15T18:20:55.488186+0000","last_undegraded":"2024-01-15T18:20:55.488186+0000","last_fullsized":"2024-01-15T18:20:55.488186+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:09:53.884883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"110'46","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.844423+0000","last_change":"2024-01-15T18:18:33.146254+0000","last_active":"2024-01-15T18:23:31.844423+0000","last_peered":"2024-01-15T18:23:31.844423+0000","last_clean":"2024-01-15T18:23:31.844423+0000","last_became_active":"2024-01-15T18:18:33.145973+0000","last_became_peered":"2024-01-15T18:18:33.145973+0000","last_unstale":"2024-01-15T18:23:31.844423+0000","last_undegraded":"2024-01-15T18:23:31.844423+0000","last_fullsized":"2024-01-15T18:23:31.844423+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663557,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"110'12","reported_seq":37,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:38.931283+0000","last_change":"2024-01-15T18:23:38.931329+0000","last_active":"2024-01-15T18:23:38.931283+0000","last_peered":"2024-01-15T18:23:38.931283+0000","last_clean":"2024-01-15T18:23:38.931283+0000","last_became_active":"2024-01-15T18:18:32.219280+0000","last_became_peered":"2024-01-15T18:18:32.219280+0000","last_unstale":"2024-01-15T18:23:38.931283+0000","last_undegraded":"2024-01-15T18:23:38.931283+0000","last_fullsized":"2024-01-15T18:23:38.931283+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'12","last_scrub_stamp":"2024-01-15T18:23:38.931162+0000","last_deep_scrub":"110'12","last_deep_scrub_stamp":"2024-01-15T18:23:38.931162+0000","last_clean_scrub_stamp":"2024-01-15T18:23:38.931162+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:57:50.305539+0000","scrub_duration":0.0047879089999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"110'24","reported_seq":44,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.993163+0000","last_change":"2024-01-15T18:18:33.161674+0000","last_active":"2024-01-15T18:23:31.993163+0000","last_peered":"2024-01-15T18:23:31.993163+0000","last_clean":"2024-01-15T18:23:31.993163+0000","last_became_active":"2024-01-15T18:18:33.161329+0000","last_became_peered":"2024-01-15T18:18:33.161329+0000","last_unstale":"2024-01-15T18:23:31.993163+0000","last_undegraded":"2024-01-15T18:23:31.993163+0000","last_fullsized":"2024-01-15T18:23:31.993163+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"107'1","reported_seq":18,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:04.760007+0000","last_change":"2024-01-15T18:18:32.215848+0000","last_active":"2024-01-15T18:23:04.760007+0000","last_peered":"2024-01-15T18:23:04.760007+0000","last_clean":"2024-01-15T18:23:04.760007+0000","last_became_active":"2024-01-15T18:18:32.214009+0000","last_became_peered":"2024-01-15T18:18:32.214009+0000","last_unstale":"2024-01-15T18:23:04.760007+0000","last_undegraded":"2024-01-15T18:23:04.760007+0000","last_fullsized":"2024-01-15T18:23:04.760007+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"110'36","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.822441+0000","last_change":"2024-01-15T18:18:33.162974+0000","last_active":"2024-01-15T18:23:31.822441+0000","last_peered":"2024-01-15T18:23:31.822441+0000","last_clean":"2024-01-15T18:23:31.822441+0000","last_became_active":"2024-01-15T18:18:33.162679+0000","last_became_peered":"2024-01-15T18:18:33.162679+0000","last_unstale":"2024-01-15T18:23:31.822441+0000","last_undegraded":"2024-01-15T18:23:31.822441+0000","last_fullsized":"2024-01-15T18:23:31.822441+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:53:15.036059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"107'2","reported_seq":19,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:05.181760+0000","last_change":"2024-01-15T18:18:32.214990+0000","last_active":"2024-01-15T18:23:05.181760+0000","last_peered":"2024-01-15T18:23:05.181760+0000","last_clean":"2024-01-15T18:23:05.181760+0000","last_became_active":"2024-01-15T18:18:32.214695+0000","last_became_peered":"2024-01-15T18:18:32.214695+0000","last_unstale":"2024-01-15T18:23:05.181760+0000","last_undegraded":"2024-01-15T18:23:05.181760+0000","last_fullsized":"2024-01-15T18:23:05.181760+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"110'44","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.739913+0000","last_change":"2024-01-15T18:18:33.168531+0000","last_active":"2024-01-15T18:23:31.739913+0000","last_peered":"2024-01-15T18:23:31.739913+0000","last_clean":"2024-01-15T18:23:31.739913+0000","last_became_active":"2024-01-15T18:18:33.168190+0000","last_became_peered":"2024-01-15T18:18:33.168190+0000","last_unstale":"2024-01-15T18:23:31.739913+0000","last_undegraded":"2024-01-15T18:23:31.739913+0000","last_fullsized":"2024-01-15T18:23:31.739913+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:53:05.325329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":11,"num_read_kb":28676,"num_write":22,"num_write_kb":73730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"107'1","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:50.487644+0000","last_change":"2024-01-15T18:18:32.221200+0000","last_active":"2024-01-15T18:20:50.487644+0000","last_peered":"2024-01-15T18:20:50.487644+0000","last_clean":"2024-01-15T18:20:50.487644+0000","last_became_active":"2024-01-15T18:18:32.220884+0000","last_became_peered":"2024-01-15T18:18:32.220884+0000","last_unstale":"2024-01-15T18:20:50.487644+0000","last_undegraded":"2024-01-15T18:20:50.487644+0000","last_fullsized":"2024-01-15T18:20:50.487644+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:02.938267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"110'41","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.846141+0000","last_change":"2024-01-15T18:18:33.163114+0000","last_active":"2024-01-15T18:23:31.846141+0000","last_peered":"2024-01-15T18:23:31.846141+0000","last_clean":"2024-01-15T18:23:31.846141+0000","last_became_active":"2024-01-15T18:18:33.162424+0000","last_became_peered":"2024-01-15T18:18:33.162424+0000","last_unstale":"2024-01-15T18:23:31.846141+0000","last_undegraded":"2024-01-15T18:23:31.846141+0000","last_fullsized":"2024-01-15T18:23:31.846141+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:42:08.107694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:49.171310+0000","last_change":"2024-01-15T18:18:32.217252+0000","last_active":"2024-01-15T18:20:49.171310+0000","last_peered":"2024-01-15T18:20:49.171310+0000","last_clean":"2024-01-15T18:20:49.171310+0000","last_became_active":"2024-01-15T18:18:32.216769+0000","last_became_peered":"2024-01-15T18:18:32.216769+0000","last_unstale":"2024-01-15T18:20:49.171310+0000","last_undegraded":"2024-01-15T18:20:49.171310+0000","last_fullsized":"2024-01-15T18:20:49.171310+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:36.807888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"110'45","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.023971+0000","last_change":"2024-01-15T18:18:33.163017+0000","last_active":"2024-01-15T18:23:32.023971+0000","last_peered":"2024-01-15T18:23:32.023971+0000","last_clean":"2024-01-15T18:23:32.023971+0000","last_became_active":"2024-01-15T18:18:33.162658+0000","last_became_peered":"2024-01-15T18:18:33.162658+0000","last_unstale":"2024-01-15T18:23:32.023971+0000","last_undegraded":"2024-01-15T18:23:32.023971+0000","last_fullsized":"2024-01-15T18:23:32.023971+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:01:41.498479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"110'3","reported_seq":19,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:50.469624+0000","last_change":"2024-01-15T18:18:32.207796+0000","last_active":"2024-01-15T18:20:50.469624+0000","last_peered":"2024-01-15T18:20:50.469624+0000","last_clean":"2024-01-15T18:20:50.469624+0000","last_became_active":"2024-01-15T18:18:32.207507+0000","last_became_peered":"2024-01-15T18:18:32.207507+0000","last_unstale":"2024-01-15T18:20:50.469624+0000","last_undegraded":"2024-01-15T18:20:50.469624+0000","last_fullsized":"2024-01-15T18:20:50.469624+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:36:42.592200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"110'43","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.490371+0000","last_change":"2024-01-15T18:18:33.164981+0000","last_active":"2024-01-15T18:23:31.490371+0000","last_peered":"2024-01-15T18:23:31.490371+0000","last_clean":"2024-01-15T18:23:31.490371+0000","last_became_active":"2024-01-15T18:18:33.164206+0000","last_became_peered":"2024-01-15T18:18:33.164206+0000","last_unstale":"2024-01-15T18:23:31.490371+0000","last_undegraded":"2024-01-15T18:23:31.490371+0000","last_fullsized":"2024-01-15T18:23:31.490371+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:00.181319+0000","last_change":"2024-01-15T18:18:32.212499+0000","last_active":"2024-01-15T18:23:00.181319+0000","last_peered":"2024-01-15T18:23:00.181319+0000","last_clean":"2024-01-15T18:23:00.181319+0000","last_became_active":"2024-01-15T18:18:32.212201+0000","last_became_peered":"2024-01-15T18:18:32.212201+0000","last_unstale":"2024-01-15T18:23:00.181319+0000","last_undegraded":"2024-01-15T18:23:00.181319+0000","last_fullsized":"2024-01-15T18:23:00.181319+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"110'44","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.774067+0000","last_change":"2024-01-15T18:18:33.165046+0000","last_active":"2024-01-15T18:23:31.774067+0000","last_peered":"2024-01-15T18:23:31.774067+0000","last_clean":"2024-01-15T18:23:31.774067+0000","last_became_active":"2024-01-15T18:18:33.164708+0000","last_became_peered":"2024-01-15T18:18:33.164708+0000","last_unstale":"2024-01-15T18:23:31.774067+0000","last_undegraded":"2024-01-15T18:23:31.774067+0000","last_fullsized":"2024-01-15T18:23:31.774067+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:46:54.687765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:45.468959+0000","last_change":"2024-01-15T18:18:32.207872+0000","last_active":"2024-01-15T18:20:45.468959+0000","last_peered":"2024-01-15T18:20:45.468959+0000","last_clean":"2024-01-15T18:20:45.468959+0000","last_became_active":"2024-01-15T18:18:32.207599+0000","last_became_peered":"2024-01-15T18:18:32.207599+0000","last_unstale":"2024-01-15T18:20:45.468959+0000","last_undegraded":"2024-01-15T18:20:45.468959+0000","last_fullsized":"2024-01-15T18:20:45.468959+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:25:20.009100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"110'42","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.029364+0000","last_change":"2024-01-15T18:18:33.160618+0000","last_active":"2024-01-15T18:23:32.029364+0000","last_peered":"2024-01-15T18:23:32.029364+0000","last_clean":"2024-01-15T18:23:32.029364+0000","last_became_active":"2024-01-15T18:18:33.160350+0000","last_became_peered":"2024-01-15T18:18:33.160350+0000","last_unstale":"2024-01-15T18:23:32.029364+0000","last_undegraded":"2024-01-15T18:23:32.029364+0000","last_fullsized":"2024-01-15T18:23:32.029364+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67109125,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":17,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:22:54.758458+0000","last_change":"2024-01-15T18:18:32.217214+0000","last_active":"2024-01-15T18:22:54.758458+0000","last_peered":"2024-01-15T18:22:54.758458+0000","last_clean":"2024-01-15T18:22:54.758458+0000","last_became_active":"2024-01-15T18:18:32.216581+0000","last_became_peered":"2024-01-15T18:18:32.216581+0000","last_unstale":"2024-01-15T18:22:54.758458+0000","last_undegraded":"2024-01-15T18:22:54.758458+0000","last_fullsized":"2024-01-15T18:22:54.758458+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"110'36","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.738324+0000","last_change":"2024-01-15T18:18:33.165549+0000","last_active":"2024-01-15T18:23:31.738324+0000","last_peered":"2024-01-15T18:23:31.738324+0000","last_clean":"2024-01-15T18:23:31.738324+0000","last_became_active":"2024-01-15T18:18:33.164152+0000","last_became_peered":"2024-01-15T18:18:33.164152+0000","last_unstale":"2024-01-15T18:23:31.738324+0000","last_undegraded":"2024-01-15T18:23:31.738324+0000","last_fullsized":"2024-01-15T18:23:31.738324+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:16:24.034308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"107'2","reported_seq":19,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:22:55.451907+0000","last_change":"2024-01-15T18:18:32.219821+0000","last_active":"2024-01-15T18:22:55.451907+0000","last_peered":"2024-01-15T18:22:55.451907+0000","last_clean":"2024-01-15T18:22:55.451907+0000","last_became_active":"2024-01-15T18:18:32.218858+0000","last_became_peered":"2024-01-15T18:18:32.218858+0000","last_unstale":"2024-01-15T18:22:55.451907+0000","last_undegraded":"2024-01-15T18:22:55.451907+0000","last_fullsized":"2024-01-15T18:22:55.451907+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"110'41","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.577271+0000","last_change":"2024-01-15T18:18:33.160602+0000","last_active":"2024-01-15T18:23:31.577271+0000","last_peered":"2024-01-15T18:23:31.577271+0000","last_clean":"2024-01-15T18:23:31.577271+0000","last_became_active":"2024-01-15T18:18:33.160315+0000","last_became_peered":"2024-01-15T18:18:33.160315+0000","last_unstale":"2024-01-15T18:23:31.577271+0000","last_undegraded":"2024-01-15T18:23:31.577271+0000","last_fullsized":"2024-01-15T18:23:31.577271+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"110'17","reported_seq":33,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:28.328002+0000","last_change":"2024-01-15T18:18:32.217029+0000","last_active":"2024-01-15T18:23:28.328002+0000","last_peered":"2024-01-15T18:23:28.328002+0000","last_clean":"2024-01-15T18:23:28.328002+0000","last_became_active":"2024-01-15T18:18:32.216612+0000","last_became_peered":"2024-01-15T18:18:32.216612+0000","last_unstale":"2024-01-15T18:23:28.328002+0000","last_undegraded":"2024-01-15T18:23:28.328002+0000","last_fullsized":"2024-01-15T18:23:28.328002+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:42:10.693562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"110'46","reported_seq":76,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.927021+0000","last_change":"2024-01-15T18:18:33.165599+0000","last_active":"2024-01-15T18:23:31.927021+0000","last_peered":"2024-01-15T18:23:31.927021+0000","last_clean":"2024-01-15T18:23:31.927021+0000","last_became_active":"2024-01-15T18:18:33.163994+0000","last_became_peered":"2024-01-15T18:18:33.163994+0000","last_unstale":"2024-01-15T18:23:31.927021+0000","last_undegraded":"2024-01-15T18:23:31.927021+0000","last_fullsized":"2024-01-15T18:23:31.927021+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:22:44.757088+0000","last_change":"2024-01-15T18:18:32.216301+0000","last_active":"2024-01-15T18:22:44.757088+0000","last_peered":"2024-01-15T18:22:44.757088+0000","last_clean":"2024-01-15T18:22:44.757088+0000","last_became_active":"2024-01-15T18:18:32.216023+0000","last_became_peered":"2024-01-15T18:18:32.216023+0000","last_unstale":"2024-01-15T18:22:44.757088+0000","last_undegraded":"2024-01-15T18:22:44.757088+0000","last_fullsized":"2024-01-15T18:22:44.757088+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"110'39","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.097184+0000","last_change":"2024-01-15T18:18:42.262610+0000","last_active":"2024-01-15T18:23:31.097184+0000","last_peered":"2024-01-15T18:23:31.097184+0000","last_clean":"2024-01-15T18:23:31.097184+0000","last_became_active":"2024-01-15T18:18:42.262255+0000","last_became_peered":"2024-01-15T18:18:42.262255+0000","last_unstale":"2024-01-15T18:23:31.097184+0000","last_undegraded":"2024-01-15T18:23:31.097184+0000","last_fullsized":"2024-01-15T18:23:31.097184+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:22:41.967700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303290,"num_objects":19,"num_object_clones":1,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:22:45.450880+0000","last_change":"2024-01-15T18:18:32.214224+0000","last_active":"2024-01-15T18:22:45.450880+0000","last_peered":"2024-01-15T18:22:45.450880+0000","last_clean":"2024-01-15T18:22:45.450880+0000","last_became_active":"2024-01-15T18:18:32.213930+0000","last_became_peered":"2024-01-15T18:18:32.213930+0000","last_unstale":"2024-01-15T18:22:45.450880+0000","last_undegraded":"2024-01-15T18:22:45.450880+0000","last_fullsized":"2024-01-15T18:22:45.450880+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"110'44","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.004572+0000","last_change":"2024-01-15T18:18:33.150974+0000","last_active":"2024-01-15T18:23:32.004572+0000","last_peered":"2024-01-15T18:23:32.004572+0000","last_clean":"2024-01-15T18:23:32.004572+0000","last_became_active":"2024-01-15T18:18:33.150660+0000","last_became_peered":"2024-01-15T18:18:33.150660+0000","last_unstale":"2024-01-15T18:23:32.004572+0000","last_undegraded":"2024-01-15T18:23:32.004572+0000","last_fullsized":"2024-01-15T18:23:32.004572+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:39.170034+0000","last_change":"2024-01-15T18:18:32.215229+0000","last_active":"2024-01-15T18:20:39.170034+0000","last_peered":"2024-01-15T18:20:39.170034+0000","last_clean":"2024-01-15T18:20:39.170034+0000","last_became_active":"2024-01-15T18:18:32.214394+0000","last_became_peered":"2024-01-15T18:18:32.214394+0000","last_unstale":"2024-01-15T18:20:39.170034+0000","last_undegraded":"2024-01-15T18:20:39.170034+0000","last_fullsized":"2024-01-15T18:20:39.170034+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:30:25.477307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"110'56","reported_seq":80,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.020041+0000","last_change":"2024-01-15T18:18:33.156094+0000","last_active":"2024-01-15T18:23:32.020041+0000","last_peered":"2024-01-15T18:23:32.020041+0000","last_clean":"2024-01-15T18:23:32.020041+0000","last_became_active":"2024-01-15T18:18:33.155797+0000","last_became_peered":"2024-01-15T18:18:33.155797+0000","last_unstale":"2024-01-15T18:23:32.020041+0000","last_undegraded":"2024-01-15T18:23:32.020041+0000","last_fullsized":"2024-01-15T18:23:32.020041+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:22:45.193573+0000","last_change":"2024-01-15T18:18:32.212760+0000","last_active":"2024-01-15T18:22:45.193573+0000","last_peered":"2024-01-15T18:22:45.193573+0000","last_clean":"2024-01-15T18:22:45.193573+0000","last_became_active":"2024-01-15T18:18:32.212487+0000","last_became_peered":"2024-01-15T18:18:32.212487+0000","last_unstale":"2024-01-15T18:22:45.193573+0000","last_undegraded":"2024-01-15T18:22:45.193573+0000","last_fullsized":"2024-01-15T18:22:45.193573+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"110'46","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.912814+0000","last_change":"2024-01-15T18:18:33.167732+0000","last_active":"2024-01-15T18:23:31.912814+0000","last_peered":"2024-01-15T18:23:31.912814+0000","last_clean":"2024-01-15T18:23:31.912814+0000","last_became_active":"2024-01-15T18:18:33.167436+0000","last_became_peered":"2024-01-15T18:18:33.167436+0000","last_unstale":"2024-01-15T18:23:31.912814+0000","last_undegraded":"2024-01-15T18:23:31.912814+0000","last_fullsized":"2024-01-15T18:23:31.912814+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:22:13.319815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"110'11","reported_seq":38,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:22:45.179665+0000","last_change":"2024-01-15T18:18:32.213077+0000","last_active":"2024-01-15T18:22:45.179665+0000","last_peered":"2024-01-15T18:22:45.179665+0000","last_clean":"2024-01-15T18:22:45.179665+0000","last_became_active":"2024-01-15T18:18:32.211744+0000","last_became_peered":"2024-01-15T18:18:32.211744+0000","last_unstale":"2024-01-15T18:22:45.179665+0000","last_undegraded":"2024-01-15T18:22:45.179665+0000","last_fullsized":"2024-01-15T18:22:45.179665+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1321,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"110'59","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:38.786673+0000","last_change":"2024-01-15T18:23:38.786716+0000","last_active":"2024-01-15T18:23:38.786673+0000","last_peered":"2024-01-15T18:23:38.786673+0000","last_clean":"2024-01-15T18:23:38.786673+0000","last_became_active":"2024-01-15T18:18:33.162521+0000","last_became_peered":"2024-01-15T18:18:33.162521+0000","last_unstale":"2024-01-15T18:23:38.786673+0000","last_undegraded":"2024-01-15T18:23:38.786673+0000","last_fullsized":"2024-01-15T18:23:38.786673+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'59","last_scrub_stamp":"2024-01-15T18:23:38.786600+0000","last_deep_scrub":"110'59","last_deep_scrub_stamp":"2024-01-15T18:23:38.786600+0000","last_clean_scrub_stamp":"2024-01-15T18:23:38.786600+0000","objects_scrubbed":23,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:26:13.413987+0000","scrub_duration":0.160345389,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:22:34.755651+0000","last_change":"2024-01-15T18:18:32.217455+0000","last_active":"2024-01-15T18:22:34.755651+0000","last_peered":"2024-01-15T18:22:34.755651+0000","last_clean":"2024-01-15T18:22:34.755651+0000","last_became_active":"2024-01-15T18:18:32.216966+0000","last_became_peered":"2024-01-15T18:18:32.216966+0000","last_unstale":"2024-01-15T18:22:34.755651+0000","last_undegraded":"2024-01-15T18:22:34.755651+0000","last_fullsized":"2024-01-15T18:22:34.755651+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"110'31","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.034409+0000","last_change":"2024-01-15T18:18:33.166946+0000","last_active":"2024-01-15T18:23:32.034409+0000","last_peered":"2024-01-15T18:23:32.034409+0000","last_clean":"2024-01-15T18:23:32.034409+0000","last_became_active":"2024-01-15T18:18:33.165323+0000","last_became_peered":"2024-01-15T18:18:33.165323+0000","last_unstale":"2024-01-15T18:23:32.034409+0000","last_undegraded":"2024-01-15T18:23:32.034409+0000","last_fullsized":"2024-01-15T18:23:32.034409+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:54.171847+0000","last_change":"2024-01-15T18:18:32.216251+0000","last_active":"2024-01-15T18:20:54.171847+0000","last_peered":"2024-01-15T18:20:54.171847+0000","last_clean":"2024-01-15T18:20:54.171847+0000","last_became_active":"2024-01-15T18:18:32.215973+0000","last_became_peered":"2024-01-15T18:18:32.215973+0000","last_unstale":"2024-01-15T18:20:54.171847+0000","last_undegraded":"2024-01-15T18:20:54.171847+0000","last_fullsized":"2024-01-15T18:20:54.171847+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:41:26.812763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":244,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:20:50.171227+0000","last_change":"2024-01-15T18:04:45.459535+0000","last_active":"2024-01-15T18:20:50.171227+0000","last_peered":"2024-01-15T18:20:50.171227+0000","last_clean":"2024-01-15T18:20:50.171227+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:20:50.171227+0000","last_undegraded":"2024-01-15T18:20:50.171227+0000","last_fullsized":"2024-01-15T18:20:50.171227+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:04:44.444228+0000","last_clean_scrub_stamp":"2024-01-15T18:04:44.444228+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:20.029308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":5368710136,"num_objects":1286,"num_object_clones":1,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":1,"num_read":525,"num_read_kb":2097165,"num_write":1306,"num_write_kb":5242895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10737459200,"data_stored":10737420272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2847,"ondisk_log_size":2847,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":613252,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":276,"num_write_kb":716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1294336,"data_stored":1226504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":260,"ondisk_log_size":260,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509670,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1834500,"kb_used_data":1767292,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92537340,"statfs":{"total":96636764160,"available":94758236160,"internally_reserved":0,"allocated":1809707008,"data_stored":1808930471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.72799999999999998,"15min":0.90300000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92800000000000005,"5min":2.8620000000000001,"15min":10.284000000000001},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.70699999999999996,"15min":0.93700000000000006},"min":{"1min":0.54200000000000004,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":16.968},"last":0.54400000000000004}]},{"osd":1,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.72399999999999998,"15min":0.93899999999999995},"min":{"1min":0.47599999999999998,"5min":0.38600000000000001,"15min":0.33800000000000002},"max":{"1min":1.028,"5min":1.028,"15min":15.791},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.73499999999999999,"15min":0.93100000000000005},"min":{"1min":0.52100000000000002,"5min":0.44,"15min":0.371},"max":{"1min":0.94799999999999995,"5min":1.248,"15min":10.842000000000001},"last":0.59599999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.72899999999999998,"15min":0.94499999999999995},"min":{"1min":0.49099999999999999,"5min":0.39500000000000002,"15min":0.378},"max":{"1min":0.96499999999999997,"5min":0.98799999999999999,"15min":22.318000000000001},"last":0.621},{"interface":"front","average":{"1min":0.753,"5min":0.752,"15min":0.91500000000000004},"min":{"1min":0.46899999999999997,"5min":0.42699999999999999,"15min":0.39100000000000001},"max":{"1min":0.91400000000000003,"5min":2.2799999999999998,"15min":9.7539999999999996},"last":0.77800000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.72699999999999998,"15min":0.996},"min":{"1min":0.56999999999999995,"5min":0.46500000000000002,"15min":0.38100000000000001},"max":{"1min":0.875,"5min":1.2110000000000001,"15min":18.152999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.78700000000000003,"15min":0.99199999999999999},"min":{"1min":0.51600000000000001,"5min":0.495,"15min":0.433},"max":{"1min":0.98099999999999998,"5min":2.2389999999999999,"15min":10.34},"last":0.745}]},{"osd":4,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.57599999999999996,"15min":0.93000000000000005},"min":{"1min":0.40600000000000003,"5min":0.34799999999999998,"15min":0.314},"max":{"1min":0.82799999999999996,"5min":0.877,"15min":20.387},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.94099999999999995},"min":{"1min":0.438,"5min":0.39400000000000002,"15min":0.34200000000000003},"max":{"1min":0.84199999999999997,"5min":0.98099999999999998,"15min":17.215},"last":0.72699999999999998}]},{"osd":5,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.622,"15min":0.91500000000000004},"min":{"1min":0.42399999999999999,"5min":0.40999999999999998,"15min":0.34699999999999998},"max":{"1min":0.91500000000000004,"5min":0.96999999999999997,"15min":19.959},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.64200000000000002,"15min":0.95999999999999996},"min":{"1min":0.40999999999999998,"5min":0.40899999999999997,"15min":0.35799999999999998},"max":{"1min":0.92300000000000004,"5min":0.92600000000000005,"15min":19.905000000000001},"last":0.70899999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.748,"15min":1.1439999999999999},"min":{"1min":0.49299999999999999,"5min":0.44,"15min":0.40799999999999997},"max":{"1min":0.995,"5min":0.995,"15min":22.478999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.69599999999999995,"15min":1.0269999999999999},"min":{"1min":0.48599999999999999,"5min":0.44600000000000001,"15min":0.40799999999999997},"max":{"1min":0.92200000000000004,"5min":0.96299999999999997,"15min":21.509},"last":0.79100000000000004}]}]},{"osd":6,"up_from":15,"seq":64424509669,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1256320,"kb_used_data":1193272,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":93115520,"statfs":{"total":96636764160,"available":95350292480,"internally_reserved":0,"allocated":1221910528,"data_stored":1221137119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":0.78400000000000003,"15min":0.99399999999999999},"min":{"1min":0.59099999999999997,"5min":0.36299999999999999,"15min":0.29099999999999998},"max":{"1min":6.1900000000000004,"5min":6.1900000000000004,"15min":32.587000000000003},"last":0.753},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.79100000000000004,"15min":0.90100000000000002},"min":{"1min":0.63900000000000001,"5min":0.434,"15min":0.35399999999999998},"max":{"1min":4.9649999999999999,"5min":4.9649999999999999,"15min":16.193000000000001},"last":0.54900000000000004}]},{"osd":1,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.73799999999999999,"15min":0.996},"min":{"1min":0.56599999999999995,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":53.451999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.73399999999999999,"15min":0.91600000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.88400000000000001,"5min":1.0189999999999999,"15min":32.512},"last":0.69199999999999995}]},{"osd":2,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77000000000000002,"15min":0.95099999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.41699999999999998},"max":{"1min":0.98099999999999998,"5min":1.016,"15min":32.472999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.996,"5min":0.84299999999999997,"15min":0.92100000000000004},"min":{"1min":0.57899999999999996,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":5.0410000000000004,"5min":5.0410000000000004,"15min":16.602},"last":0.71099999999999997}]},{"osd":3,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.82699999999999996,"15min":0.93600000000000005},"min":{"1min":0.64900000000000002,"5min":0.52500000000000002,"15min":0.46000000000000002},"max":{"1min":5.0940000000000003,"5min":5.0940000000000003,"15min":16.388000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.76000000000000001,"15min":1.0609999999999999},"min":{"1min":0.67800000000000005,"5min":0.505,"15min":0.432},"max":{"1min":1.004,"5min":1.04,"15min":53.398000000000003},"last":0.64000000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.67000000000000004,"15min":0.88800000000000001},"min":{"1min":0.41499999999999998,"5min":0.39000000000000001,"15min":0.378},"max":{"1min":6.3869999999999996,"5min":6.3869999999999996,"15min":32.549999999999997},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.68799999999999994,"15min":0.83599999999999997},"min":{"1min":0.45800000000000002,"5min":0.41399999999999998,"15min":0.371},"max":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":11.852},"last":0.49099999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.70499999999999996,"15min":1.052},"min":{"1min":0.433,"5min":0.42199999999999999,"15min":0.34300000000000003},"max":{"1min":6.8769999999999998,"5min":6.8769999999999998,"15min":53.229999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.747,"15min":1.0209999999999999},"min":{"1min":0.49199999999999999,"5min":0.44400000000000001,"15min":0.41799999999999998},"max":{"1min":6.1360000000000001,"5min":6.1360000000000001,"15min":32.404000000000003},"last":0.82199999999999995}]},{"osd":7,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.749,"15min":1.0820000000000001},"min":{"1min":0.52200000000000002,"5min":0.46000000000000002,"15min":0.38600000000000001},"max":{"1min":5.1319999999999997,"5min":5.1319999999999997,"15min":45.070999999999998},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.71099999999999997,"15min":1.181},"min":{"1min":0.51100000000000001,"5min":0.47999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89100000000000001,"5min":0.999,"15min":53.338000000000001},"last":0.72999999999999998}]}]},{"osd":4,"up_from":15,"seq":64424509670,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1706876,"kb_used_data":1643828,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":92664964,"statfs":{"total":96636764160,"available":94888923136,"internally_reserved":0,"allocated":1683279872,"data_stored":1682510628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.69799999999999995,"15min":0.90600000000000003},"min":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.32900000000000001},"max":{"1min":0.96599999999999997,"5min":1.542,"15min":23.318000000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.70299999999999996,"15min":0.81299999999999994},"min":{"1min":0.55000000000000004,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":14.723000000000001},"last":0.79400000000000004}]},{"osd":1,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.72599999999999998,"15min":0.81399999999999995},"min":{"1min":0.46300000000000002,"5min":0.39100000000000001,"15min":0.38800000000000001},"max":{"1min":0.96199999999999997,"5min":1.919,"15min":9.5700000000000003},"last":0.755},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.74199999999999999,"15min":0.94699999999999995},"min":{"1min":0.52600000000000002,"5min":0.44900000000000001,"15min":0.36899999999999999},"max":{"1min":0.996,"5min":1.429,"15min":22.408999999999999},"last":0.84599999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.79000000000000004,"15min":0.85299999999999998},"min":{"1min":0.50900000000000001,"5min":0.41899999999999998,"15min":0.39700000000000002},"max":{"1min":1.032,"5min":5.6260000000000003,"15min":9.7089999999999996},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.76100000000000001,"15min":1.018},"min":{"1min":0.59299999999999997,"5min":0.45000000000000001,"15min":0.39100000000000001},"max":{"1min":1.016,"5min":1.387,"15min":22.338999999999999},"last":0.878}]},{"osd":3,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.73999999999999999,"15min":0.88700000000000001},"min":{"1min":0.52300000000000002,"5min":0.46200000000000002,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":1.577,"15min":14.763999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.82599999999999996,"15min":0.93600000000000005},"min":{"1min":0.65300000000000002,"5min":0.53600000000000003,"15min":0.48099999999999998},"max":{"1min":0.996,"5min":1.9610000000000001,"15min":9.6289999999999996},"last":0.83199999999999996}]},{"osd":5,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63300000000000001,"15min":0.94299999999999995},"min":{"1min":0.44,"5min":0.36499999999999999,"15min":0.34799999999999998},"max":{"1min":1.0149999999999999,"5min":1.1970000000000001,"15min":31.617999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.85799999999999998},"min":{"1min":0.40799999999999997,"5min":0.373,"15min":0.36599999999999999},"max":{"1min":0.86699999999999999,"5min":1.4910000000000001,"15min":17.145},"last":0.53800000000000003}]},{"osd":6,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.72799999999999998,"15min":0.83899999999999997},"min":{"1min":0.60799999999999998,"5min":0.41799999999999998,"15min":0.37},"max":{"1min":0.89300000000000002,"5min":5.9489999999999998,"15min":11.124000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.71099999999999997,"15min":1.2090000000000001},"min":{"1min":0.52300000000000002,"5min":0.42499999999999999,"15min":0.40200000000000002},"max":{"1min":1.0649999999999999,"5min":2.3149999999999999,"15min":32.308},"last":0.66500000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.72699999999999998,"15min":1.0009999999999999},"min":{"1min":0.48099999999999998,"5min":0.45100000000000001,"15min":0.34799999999999998},"max":{"1min":1,"5min":2.4329999999999998,"15min":39.256},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78200000000000003,"15min":1.002},"min":{"1min":0.60699999999999998,"5min":0.499,"15min":0.46500000000000002},"max":{"1min":1.04,"5min":2.1909999999999998,"15min":26.573},"last":0.59699999999999998}]}]},{"osd":3,"up_from":15,"seq":64424509669,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1453604,"kb_used_data":1386396,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92918236,"statfs":{"total":96636764160,"available":95148273664,"internally_reserved":0,"allocated":1419669504,"data_stored":1418880940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.45100000000000001,"15min":0.498},"min":{"1min":0.28999999999999998,"5min":0.28199999999999997,"15min":0.245},"max":{"1min":0.69099999999999995,"5min":0.69699999999999995,"15min":5.8849999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.51000000000000001,"15min":0.621},"min":{"1min":0.41799999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.80000000000000004,"5min":1.4750000000000001,"15min":17.756},"last":0.46700000000000003}]},{"osd":1,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51300000000000001,"15min":0.60499999999999998},"min":{"1min":0.33500000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60399999999999998,"5min":4.0099999999999998,"15min":12.307},"last":0.5},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55200000000000005,"15min":0.59199999999999997},"min":{"1min":0.36199999999999999,"5min":0.34300000000000003,"15min":0.31900000000000001},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":6.0640000000000001},"last":0.44800000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61399999999999999,"15min":0.71199999999999997},"min":{"1min":0.42799999999999999,"5min":0.35299999999999998,"15min":0.32600000000000001},"max":{"1min":0.82999999999999996,"5min":4.077,"15min":12.468999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.59099999999999997,"15min":0.68000000000000005},"min":{"1min":0.44,"5min":0.39100000000000001,"15min":0.35399999999999998},"max":{"1min":0.78000000000000003,"5min":0.81899999999999995,"15min":10.837999999999999},"last":0.48199999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78500000000000003,"15min":0.999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.375},"max":{"1min":1.1910000000000001,"5min":1.9379999999999999,"15min":17.853999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.85999999999999999,"15min":0.98199999999999998},"min":{"1min":0.54900000000000004,"5min":0.49199999999999999,"15min":0.45200000000000001},"max":{"1min":1.0569999999999999,"5min":4.2430000000000003,"15min":13.31},"last":0.88200000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.78400000000000003,"15min":0.98899999999999999},"min":{"1min":0.496,"5min":0.439,"15min":0.439},"max":{"1min":1.1619999999999999,"5min":1.179,"15min":17.687999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.77800000000000002,"15min":0.99099999999999999},"min":{"1min":0.58599999999999997,"5min":0.53800000000000003,"15min":0.438},"max":{"1min":1.048,"5min":1.825,"15min":20.515999999999998},"last":0.86799999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.80900000000000005,"15min":0.98499999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.432},"max":{"1min":1.1419999999999999,"5min":4.2800000000000002,"15min":23.100999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.80600000000000005,"15min":0.94799999999999995},"min":{"1min":0.69899999999999995,"5min":0.52200000000000002,"15min":0.5},"max":{"1min":1.0720000000000001,"5min":1.149,"15min":12.76},"last":0.72099999999999997}]},{"osd":7,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82499999999999996,"15min":1.1850000000000001},"min":{"1min":0.625,"5min":0.56899999999999995,"15min":0.46800000000000003},"max":{"1min":1.0860000000000001,"5min":1.931,"15min":24.321999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.82899999999999996,"15min":1.0820000000000001},"min":{"1min":0.60899999999999999,"5min":0.497,"15min":0.497},"max":{"1min":1.2070000000000001,"5min":4.202,"15min":23.013000000000002},"last":0.75700000000000001}]}]},{"osd":2,"up_from":15,"seq":64424509669,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":977856,"kb_used_data":910648,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93393984,"statfs":{"total":96636764160,"available":95635439616,"internally_reserved":0,"allocated":932503552,"data_stored":931731464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45500000000000002,"15min":0.50900000000000001},"min":{"1min":0.32300000000000001,"5min":0.31,"15min":0.247},"max":{"1min":0.55200000000000005,"5min":0.80800000000000005,"15min":5.1849999999999996},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.501,"15min":0.54600000000000004},"min":{"1min":0.36499999999999999,"5min":0.26700000000000002,"15min":0.252},"max":{"1min":0.73999999999999999,"5min":0.83899999999999997,"15min":7.516},"last":0.56699999999999995}]},{"osd":1,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.56799999999999995,"15min":0.60999999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.29699999999999999},"max":{"1min":0.71899999999999997,"5min":3.044,"15min":10.907},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.48999999999999999,"15min":0.59599999999999997},"min":{"1min":0.34999999999999998,"5min":0.33100000000000002,"15min":0.28100000000000003},"max":{"1min":0.76400000000000001,"5min":0.76500000000000001,"15min":8.9190000000000005},"last":0.47699999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52500000000000002,"15min":0.60599999999999998},"min":{"1min":0.40799999999999997,"5min":0.35399999999999998,"15min":0.30499999999999999},"max":{"1min":0.70999999999999996,"5min":0.86499999999999999,"15min":12.185},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61899999999999999,"15min":0.67100000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.373},"max":{"1min":0.80700000000000005,"5min":0.89600000000000002,"15min":7.6760000000000002},"last":0.58899999999999997}]},{"osd":4,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.80300000000000005,"15min":0.879},"min":{"1min":0.59999999999999998,"5min":0.54700000000000004,"15min":0.44},"max":{"1min":1.0660000000000001,"5min":1.22,"15min":8.6609999999999996},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.76800000000000002,"15min":0.91200000000000003},"min":{"1min":0.56399999999999995,"5min":0.46200000000000002,"15min":0.41599999999999998},"max":{"1min":1.008,"5min":1.9670000000000001,"15min":19.146999999999998},"last":0.752}]},{"osd":5,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.76000000000000001,"15min":0.90200000000000002},"min":{"1min":0.58299999999999996,"5min":0.53600000000000003,"15min":0.41899999999999998},"max":{"1min":1.093,"5min":1.111,"15min":31.710000000000001},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.81999999999999995,"15min":0.999},"min":{"1min":0.57399999999999995,"5min":0.51700000000000002,"15min":0.48199999999999998},"max":{"1min":0.99199999999999999,"5min":1.8240000000000001,"15min":22.539000000000001},"last":0.85399999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.78300000000000003,"15min":0.91000000000000003},"min":{"1min":0.56499999999999995,"5min":0.52900000000000003,"15min":0.40899999999999997},"max":{"1min":1.079,"5min":1.3979999999999999,"15min":9.4540000000000006},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.80100000000000005,"15min":0.90800000000000003},"min":{"1min":0.64300000000000002,"5min":0.5,"15min":0.47899999999999998},"max":{"1min":1.0309999999999999,"5min":1.873,"15min":18.614999999999998},"last":0.83599999999999997}]},{"osd":7,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.79800000000000004,"15min":0.96799999999999997},"min":{"1min":0.63200000000000001,"5min":0.50900000000000001,"15min":0.44400000000000001},"max":{"1min":1.0109999999999999,"5min":1.7729999999999999,"15min":11.606},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.84699999999999998,"15min":0.998},"min":{"1min":0.627,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.105,"5min":1.611,"15min":17.013000000000002},"last":0.89000000000000001}]}]},{"osd":1,"up_from":15,"seq":64424509670,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1133516,"kb_used_data":1066308,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93238324,"statfs":{"total":96636764160,"available":95476043776,"internally_reserved":0,"allocated":1091899392,"data_stored":1091115320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45400000000000001,"15min":0.58499999999999996},"min":{"1min":0.26800000000000002,"5min":0.25900000000000001,"15min":0.23699999999999999},"max":{"1min":0.63500000000000001,"5min":0.88100000000000001,"15min":17.396999999999998},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.51600000000000001,"15min":0.626},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.26900000000000002},"max":{"1min":0.73699999999999999,"5min":0.82999999999999996,"15min":12.378},"last":0.56200000000000006}]},{"osd":2,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51700000000000002,"15min":0.57699999999999996},"min":{"1min":0.33000000000000002,"5min":0.313,"15min":0.309},"max":{"1min":0.61699999999999999,"5min":0.878,"15min":6.4569999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.54300000000000004,"15min":0.67400000000000004},"min":{"1min":0.40100000000000002,"5min":0.34499999999999997,"15min":0.33300000000000002},"max":{"1min":0.73899999999999999,"5min":0.81299999999999994,"15min":18.827000000000002},"last":0.48299999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59799999999999998,"15min":0.67200000000000004},"min":{"1min":0.433,"5min":0.40600000000000003,"15min":0.33700000000000002},"max":{"1min":0.872,"5min":1.0580000000000001,"15min":5.915},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.53800000000000003,"15min":0.64800000000000002},"min":{"1min":0.34399999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.77500000000000002,"5min":0.85499999999999998,"15min":9.9719999999999995},"last":0.438}]},{"osd":4,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.77700000000000002,"15min":0.89400000000000002},"min":{"1min":0.50900000000000001,"5min":0.437,"15min":0.437},"max":{"1min":1.056,"5min":1.1060000000000001,"15min":9.9329999999999998},"last":0.627},{"interface":"front","average":{"1min":0.746,"5min":0.79000000000000004,"15min":0.88300000000000001},"min":{"1min":0.55600000000000005,"5min":0.52800000000000002,"15min":0.45800000000000002},"max":{"1min":0.95599999999999996,"5min":1.165,"15min":9.8610000000000007},"last":0.64200000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.80200000000000005,"15min":0.96299999999999997},"min":{"1min":0.55000000000000004,"5min":0.51200000000000001,"15min":0.46100000000000002},"max":{"1min":0.97499999999999998,"5min":1.2050000000000001,"15min":22.594999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.78200000000000003,"15min":1.0129999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.443},"max":{"1min":0.97499999999999998,"5min":1.077,"15min":26.748999999999999},"last":0.59299999999999997}]},{"osd":6,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.78800000000000003,"15min":1.034},"min":{"1min":0.57899999999999996,"5min":0.53100000000000003,"15min":0.42899999999999999},"max":{"1min":1.095,"5min":1.218,"15min":35.112000000000002},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.81100000000000005,"15min":0.90900000000000003},"min":{"1min":0.62,"5min":0.501,"15min":0.501},"max":{"1min":0.92900000000000005,"5min":1.2090000000000001,"15min":8.3670000000000009},"last":0.60999999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.80800000000000005,"15min":1.024},"min":{"1min":0.56399999999999995,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.141,"5min":1.1950000000000001,"15min":20.872},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.81599999999999995,"15min":0.97799999999999998},"min":{"1min":0.61499999999999999,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.92300000000000004,"5min":1.139,"15min":9.8640000000000008},"last":0.73199999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542374,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":904260,"kb_used_data":828732,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93467580,"statfs":{"total":96636764160,"available":95710801920,"internally_reserved":0,"allocated":848621568,"data_stored":847844597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.72299999999999998,"15min":0.754},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.29899999999999999},"max":{"1min":1.038,"5min":2.6920000000000002,"15min":5.1639999999999997},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.75800000000000001,"15min":0.83899999999999997},"min":{"1min":0.46899999999999997,"5min":0.36799999999999999,"15min":0.34899999999999998},"max":{"1min":0.98199999999999998,"5min":6.4290000000000003,"15min":9.359},"last":0.54300000000000004}]},{"osd":1,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.76800000000000002,"15min":0.93000000000000005},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.375},"max":{"1min":0.89900000000000002,"5min":6.5069999999999997,"15min":21.120000000000001},"last":0.628},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.747,"15min":0.93899999999999995},"min":{"1min":0.48299999999999998,"5min":0.39900000000000002,"15min":0.38600000000000001},"max":{"1min":0.96499999999999997,"5min":2.6619999999999999,"15min":15.843},"last":0.51300000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.76800000000000002,"15min":0.86699999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.34999999999999998},"max":{"1min":0.95099999999999996,"5min":6.3339999999999996,"15min":8.7750000000000004},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.79900000000000004,"15min":0.98299999999999998},"min":{"1min":0.501,"5min":0.501,"15min":0.39900000000000002},"max":{"1min":0.95399999999999996,"5min":6.4619999999999997,"15min":21.201000000000001},"last":0.56299999999999994}]},{"osd":3,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.753,"15min":0.84399999999999997},"min":{"1min":0.498,"5min":0.46000000000000002,"15min":0.37},"max":{"1min":0.99199999999999999,"5min":2.6299999999999999,"15min":7.7199999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.755,"5min":0.81100000000000005,"15min":0.91900000000000004},"min":{"1min":0.52800000000000002,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.0109999999999999,"5min":6.3730000000000002,"15min":8.7300000000000004},"last":0.68200000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.66500000000000004,"15min":0.83399999999999996},"min":{"1min":0.52600000000000002,"5min":0.42999999999999999,"15min":0.30399999999999999},"max":{"1min":1.012,"5min":3.161,"15min":10.962},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.70699999999999996,"15min":1.012},"min":{"1min":0.48799999999999999,"5min":0.41999999999999998,"15min":0.35099999999999998},"max":{"1min":0.88100000000000001,"5min":6.5460000000000003,"15min":26.146999999999998},"last":0.77100000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.66500000000000004,"15min":0.76700000000000002},"min":{"1min":0.50700000000000001,"5min":0.41299999999999998,"15min":0.36099999999999999},"max":{"1min":0.871,"5min":2.589,"15min":6.6699999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.82399999999999995,"15min":0.99199999999999999},"min":{"1min":0.53800000000000003,"5min":0.45300000000000001,"15min":0.379},"max":{"1min":0.95699999999999996,"5min":14.007999999999999,"15min":30.488},"last":0.754}]},{"osd":7,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.72499999999999998,"15min":1.2010000000000001},"min":{"1min":0.56699999999999995,"5min":0.48899999999999999,"15min":0.42599999999999999},"max":{"1min":0.90800000000000003,"5min":3.0870000000000002,"15min":33.499000000000002},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.91000000000000003,"15min":1.0680000000000001},"min":{"1min":0.57099999999999995,"5min":0.45700000000000002,"15min":0.42899999999999999},"max":{"1min":0.93000000000000005,"5min":21.178999999999998,"15min":25.437999999999999},"last":0.78600000000000003}]}]},{"osd":0,"up_from":14,"seq":60129542374,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1769580,"kb_used_data":1702372,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92602260,"statfs":{"total":96636764160,"available":94824714240,"internally_reserved":0,"allocated":1743228928,"data_stored":1742432677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.82999999999999996,"15min":0.749},"min":{"1min":0.33100000000000002,"5min":0.25800000000000001,"15min":0.245},"max":{"1min":0.59899999999999998,"5min":38.521999999999998,"15min":38.521999999999998},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.60099999999999998,"15min":0.68799999999999994},"min":{"1min":0.36699999999999999,"5min":0.32900000000000001,"15min":0.27000000000000002},"max":{"1min":0.753,"5min":8.0749999999999993,"15min":10.154},"last":0.51300000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.57699999999999996,"15min":0.80200000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.30399999999999999},"max":{"1min":0.78600000000000003,"5min":2.6909999999999998,"15min":25.905000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.67200000000000004,"15min":0.66900000000000004},"min":{"1min":0.307,"5min":0.307,"15min":0.27800000000000002},"max":{"1min":0.69199999999999995,"5min":18.247,"15min":18.247},"last":0.496}]},{"osd":3,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.76400000000000001,"15min":0.80300000000000005},"min":{"1min":0.32400000000000001,"5min":0.32000000000000001,"15min":0.29199999999999998},"max":{"1min":0.70999999999999996,"5min":21.451000000000001,"15min":21.451000000000001},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61799999999999999,"15min":0.76600000000000001},"min":{"1min":0.47099999999999997,"5min":0.38900000000000001,"15min":0.36899999999999999},"max":{"1min":0.80100000000000005,"5min":1.7,"15min":10.210000000000001},"last":0.54500000000000004}]},{"osd":4,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.79400000000000004,"15min":0.99199999999999999},"min":{"1min":0.52600000000000002,"5min":0.441,"15min":0.39100000000000001},"max":{"1min":1.1639999999999999,"5min":2.7850000000000001,"15min":22.814},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.80100000000000005,"5min":1.0389999999999999,"15min":1.105},"min":{"1min":0.64000000000000001,"5min":0.45500000000000002,"15min":0.42399999999999999},"max":{"1min":1.1379999999999999,"5min":27.027999999999999,"15min":27.027999999999999},"last":0.61899999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.83199999999999996,"15min":0.98799999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.38400000000000001},"max":{"1min":1.218,"5min":6.6429999999999998,"15min":12.428000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.77700000000000002,"5min":1.0509999999999999,"15min":1.333},"min":{"1min":0.55400000000000005,"5min":0.47999999999999998,"15min":0.46100000000000002},"max":{"1min":1.0600000000000001,"5min":27.210999999999999,"15min":27.210999999999999},"last":0.73499999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.86699999999999999,"15min":1.242},"min":{"1min":0.66400000000000003,"5min":0.50900000000000001,"15min":0.46700000000000003},"max":{"1min":0.98399999999999999,"5min":9.3140000000000001,"15min":31.059000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.82699999999999996,"15min":1.089},"min":{"1min":0.58899999999999997,"5min":0.52900000000000003,"15min":0.51500000000000001},"max":{"1min":1.024,"5min":2.8340000000000001,"15min":40.731000000000002},"last":0.75700000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.86299999999999999,"15min":1.2150000000000001},"min":{"1min":0.66500000000000004,"5min":0.52200000000000002,"15min":0.499},"max":{"1min":1.1539999999999999,"5min":5.2089999999999996,"15min":24.492999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.81599999999999995,"5min":1.097,"15min":1.1859999999999999},"min":{"1min":0.63500000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1200000000000001,"5min":27.081,"15min":27.081},"last":0.77200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":630784,"data_stored":611245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":1321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":626688,"data_stored":611203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1740644352,"data_stored":1740636601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1090527232,"data_stored":1090519412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":931139584,"data_stored":931135680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1417682944,"data_stored":1417675274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1681920000,"data_stored":1681916165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":847253504,"data_stored":847249530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1220546560,"data_stored":1220542586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1807745024,"data_stored":1807745024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:23:41.694 INFO:tasks.ceph:pgid 17.3f last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.694 INFO:tasks.ceph:pgid 17.3e last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.694 INFO:tasks.ceph:pgid 17.3d last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.694 INFO:tasks.ceph:pgid 17.3b last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.694 INFO:tasks.ceph:pgid 17.3a last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.695 INFO:tasks.ceph:pgid 17.39 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.695 INFO:tasks.ceph:pgid 17.38 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.695 INFO:tasks.ceph:pgid 17.37 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.695 INFO:tasks.ceph:pgid 17.36 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.695 INFO:tasks.ceph:pgid 17.35 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.695 INFO:tasks.ceph:pgid 17.34 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.696 INFO:tasks.ceph:pgid 17.33 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.696 INFO:tasks.ceph:pgid 17.32 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.696 INFO:tasks.ceph:pgid 17.31 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.696 INFO:tasks.ceph:pgid 17.30 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.696 INFO:tasks.ceph:pgid 17.2f last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.696 INFO:tasks.ceph:pgid 17.2e last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.697 INFO:tasks.ceph:pgid 17.2d last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.697 INFO:tasks.ceph:pgid 17.2c last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.697 INFO:tasks.ceph:pgid 17.2b last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.697 INFO:tasks.ceph:pgid 17.2a last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.697 INFO:tasks.ceph:pgid 17.29 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.697 INFO:tasks.ceph:pgid 17.28 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.698 INFO:tasks.ceph:pgid 17.27 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.698 INFO:tasks.ceph:pgid 17.26 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.698 INFO:tasks.ceph:pgid 17.25 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.698 INFO:tasks.ceph:pgid 17.24 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.698 INFO:tasks.ceph:pgid 17.23 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.698 INFO:tasks.ceph:pgid 17.22 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.699 INFO:tasks.ceph:pgid 17.21 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.699 INFO:tasks.ceph:pgid 17.20 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.699 INFO:tasks.ceph:pgid 17.1e last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.699 INFO:tasks.ceph:pgid 16.1f last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.699 INFO:tasks.ceph:pgid 17.1f last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.699 INFO:tasks.ceph:pgid 16.1e last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.700 INFO:tasks.ceph:pgid 17.1c last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.700 INFO:tasks.ceph:pgid 16.1d last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.700 INFO:tasks.ceph:pgid 17.1d last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.700 INFO:tasks.ceph:pgid 16.1c last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.700 INFO:tasks.ceph:pgid 17.1a last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.701 INFO:tasks.ceph:pgid 16.1b last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.701 INFO:tasks.ceph:pgid 17.1b last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.701 INFO:tasks.ceph:pgid 16.1a last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.701 INFO:tasks.ceph:pgid 17.18 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.701 INFO:tasks.ceph:pgid 16.19 last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.701 INFO:tasks.ceph:pgid 17.19 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.701 INFO:tasks.ceph:pgid 16.18 last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.702 INFO:tasks.ceph:pgid 17.16 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.702 INFO:tasks.ceph:pgid 16.17 last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.702 INFO:tasks.ceph:pgid 17.17 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.702 INFO:tasks.ceph:pgid 16.16 last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.702 INFO:tasks.ceph:pgid 17.14 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.702 INFO:tasks.ceph:pgid 16.15 last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.703 INFO:tasks.ceph:pgid 17.15 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.703 INFO:tasks.ceph:pgid 16.14 last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.703 INFO:tasks.ceph:pgid 17.12 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.703 INFO:tasks.ceph:pgid 16.13 last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.703 INFO:tasks.ceph:pgid 17.13 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.703 INFO:tasks.ceph:pgid 16.12 last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.703 INFO:tasks.ceph:pgid 17.11 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.704 INFO:tasks.ceph:pgid 17.e last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.704 INFO:tasks.ceph:pgid 16.f last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.704 INFO:tasks.ceph:pgid 17.f last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.704 INFO:tasks.ceph:pgid 16.e last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.704 INFO:tasks.ceph:pgid 17.c last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.704 INFO:tasks.ceph:pgid 16.d last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.705 INFO:tasks.ceph:pgid 17.d last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.705 INFO:tasks.ceph:pgid 16.c last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.705 INFO:tasks.ceph:pgid 17.a last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.705 INFO:tasks.ceph:pgid 16.b last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.705 INFO:tasks.ceph:pgid 17.b last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.705 INFO:tasks.ceph:pgid 16.a last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.706 INFO:tasks.ceph:pgid 17.8 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.706 INFO:tasks.ceph:pgid 16.9 last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.706 INFO:tasks.ceph:pgid 17.9 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.706 INFO:tasks.ceph:pgid 16.8 last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.706 INFO:tasks.ceph:pgid 17.6 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.706 INFO:tasks.ceph:pgid 16.7 last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.706 INFO:tasks.ceph:pgid 17.7 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.707 INFO:tasks.ceph:pgid 16.6 last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.707 INFO:tasks.ceph:pgid 17.4 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.707 INFO:tasks.ceph:pgid 16.5 last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.707 INFO:tasks.ceph:pgid 17.5 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.707 INFO:tasks.ceph:pgid 16.4 last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.707 INFO:tasks.ceph:pgid 17.2 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.708 INFO:tasks.ceph:pgid 16.3 last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.708 INFO:tasks.ceph:pgid 17.3 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.708 INFO:tasks.ceph:pgid 16.2 last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.708 INFO:tasks.ceph:pgid 17.0 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.708 INFO:tasks.ceph:pgid 16.1 last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.709 INFO:tasks.ceph:pgid 16.0 last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.709 INFO:tasks.ceph:pgid 17.10 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.709 INFO:tasks.ceph:pgid 16.11 last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.709 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-15T18:04:44.444228+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=4, tm_sec=44, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:23:41.709 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-15T18:23:58.710 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:23:58.714 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:24:01.710 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:24:01.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.872+0000 7f6d32600640 1 Processor -- start 2024-01-15T18:24:01.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.872+0000 7f6d32600640 1 -- start start 2024-01-15T18:24:01.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.872+0000 7f6d32600640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6d2c0701a0 0x7f6d2c070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:01.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.872+0000 7f6d32600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6d2c0c92b0 0x7f6d2c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:01.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.872+0000 7f6d32600640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f6d2c06e240 0x7f6d2c15d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:01.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.872+0000 7f6d32600640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d2c06c4a0 con 0x7f6d2c0701a0 2024-01-15T18:24:01.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.872+0000 7f6d32600640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d2c070ac0 con 0x7f6d2c06e240 2024-01-15T18:24:01.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.872+0000 7f6d32600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c15c660 con 0x7f6d2c0c92b0 2024-01-15T18:24:01.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.872+0000 7f6d2b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6d2c0701a0 0x7f6d2c070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:01.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.872+0000 7f6d2b400640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6d2c0701a0 0x7f6d2c070580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49400/0 (socket says 172.21.15.27:49400) 2024-01-15T18:24:01.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.872+0000 7f6d2b400640 1 -- 172.21.15.27:0/1524964827 learned_addr learned my addr 172.21.15.27:0/1524964827 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:24:01.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.872+0000 7f6d2be00640 1 --2- 172.21.15.27:0/1524964827 >> v2:172.21.15.93:3300/0 conn(0x7f6d2c06e240 0x7f6d2c15d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:01.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.872+0000 7f6d2aa00640 1 --2- 172.21.15.27:0/1524964827 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6d2c0c92b0 0x7f6d2c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:01.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.872+0000 7f6d2b400640 1 -- 172.21.15.27:0/1524964827 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6d2c0c92b0 msgr2=0x7f6d2c06dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:24:01.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.872+0000 7f6d2b400640 1 --2- 172.21.15.27:0/1524964827 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6d2c0c92b0 0x7f6d2c06dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:01.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.872+0000 7f6d2b400640 1 -- 172.21.15.27:0/1524964827 >> v2:172.21.15.93:3300/0 conn(0x7f6d2c06e240 msgr2=0x7f6d2c15d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:24:01.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.872+0000 7f6d2b400640 1 --2- 172.21.15.27:0/1524964827 >> v2:172.21.15.93:3300/0 conn(0x7f6d2c06e240 0x7f6d2c15d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:01.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.872+0000 7f6d2b400640 1 -- 172.21.15.27:0/1524964827 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d2c15c940 con 0x7f6d2c0701a0 2024-01-15T18:24:01.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.876+0000 7f6d2b400640 1 --2- 172.21.15.27:0/1524964827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d2c0701a0 0x7f6d2c070580 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c072aa0 tx=0x7f6d1c01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6021d0351784776 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:24:01.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.876+0000 7f6d2a000640 1 -- 172.21.15.27:0/1524964827 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c0aa5f0 con 0x7f6d2c0701a0 2024-01-15T18:24:01.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.876+0000 7f6d2a000640 1 -- 172.21.15.27:0/1524964827 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d1c0aa750 con 0x7f6d2c0701a0 2024-01-15T18:24:01.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.876+0000 7f6d2a000640 1 -- 172.21.15.27:0/1524964827 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c0aaae0 con 0x7f6d2c0701a0 2024-01-15T18:24:01.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.876+0000 7f6d32600640 1 -- 172.21.15.27:0/1524964827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d2c0701a0 msgr2=0x7f6d2c070580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:01.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.876+0000 7f6d32600640 1 --2- 172.21.15.27:0/1524964827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d2c0701a0 0x7f6d2c070580 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c072aa0 tx=0x7f6d1c01c4d0 comp rx=0 tx=0).stop 2024-01-15T18:24:01.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.876+0000 7f6d32600640 1 -- 172.21.15.27:0/1524964827 shutdown_connections 2024-01-15T18:24:01.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.876+0000 7f6d32600640 1 --2- 172.21.15.27:0/1524964827 >> v2:172.21.15.93:3300/0 conn(0x7f6d2c06e240 0x7f6d2c15d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:01.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.876+0000 7f6d32600640 1 --2- 172.21.15.27:0/1524964827 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6d2c0c92b0 0x7f6d2c06dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:01.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.876+0000 7f6d32600640 1 --2- 172.21.15.27:0/1524964827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d2c0701a0 0x7f6d2c070580 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:01.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.876+0000 7f6d32600640 1 -- 172.21.15.27:0/1524964827 >> 172.21.15.27:0/1524964827 conn(0x7f6d2c0cec60 msgr2=0x7f6d2c0cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:24:01.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.876+0000 7f6d32600640 1 -- 172.21.15.27:0/1524964827 shutdown_connections 2024-01-15T18:24:01.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.876+0000 7f6d32600640 1 -- 172.21.15.27:0/1524964827 wait complete. 2024-01-15T18:24:01.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.880+0000 7f6d32600640 1 Processor -- start 2024-01-15T18:24:01.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.880+0000 7f6d32600640 1 -- start start 2024-01-15T18:24:01.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.880+0000 7f6d32600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6d2c06e240 0x7f6d2c155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:01.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.880+0000 7f6d32600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d2c0701a0 0x7f6d2c0cc830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:01.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.880+0000 7f6d32600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6d2c0c92b0 0x7f6d2c13fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:01.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.880+0000 7f6d32600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c153370 con 0x7f6d2c0701a0 2024-01-15T18:24:01.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.880+0000 7f6d32600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c0cd220 con 0x7f6d2c0c92b0 2024-01-15T18:24:01.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.880+0000 7f6d32600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c0cd4e0 con 0x7f6d2c06e240 2024-01-15T18:24:01.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.880+0000 7f6d2aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d2c0701a0 0x7f6d2c0cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:01.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.880+0000 7f6d2aa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d2c0701a0 0x7f6d2c0cc830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49404/0 (socket says 172.21.15.27:49404) 2024-01-15T18:24:01.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.880+0000 7f6d2aa00640 1 -- 172.21.15.27:0/2522460366 learned_addr learned my addr 172.21.15.27:0/2522460366 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:24:01.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.880+0000 7f6d2b400640 1 --2- 172.21.15.27:0/2522460366 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6d2c06e240 0x7f6d2c155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:01.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.880+0000 7f6d2be00640 1 --2- 172.21.15.27:0/2522460366 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6d2c0c92b0 0x7f6d2c13fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:01.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.880+0000 7f6d2aa00640 1 -- 172.21.15.27:0/2522460366 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6d2c06e240 msgr2=0x7f6d2c155250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:01.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.880+0000 7f6d2aa00640 1 --2- 172.21.15.27:0/2522460366 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6d2c06e240 0x7f6d2c155250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:01.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.880+0000 7f6d2aa00640 1 -- 172.21.15.27:0/2522460366 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6d2c0c92b0 msgr2=0x7f6d2c13fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:01.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.880+0000 7f6d2aa00640 1 --2- 172.21.15.27:0/2522460366 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6d2c0c92b0 0x7f6d2c13fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:01.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.880+0000 7f6d2aa00640 1 -- 172.21.15.27:0/2522460366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1c0094d0 con 0x7f6d2c0701a0 2024-01-15T18:24:01.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.880+0000 7f6d2aa00640 1 --2- 172.21.15.27:0/2522460366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d2c0701a0 0x7f6d2c0cc830 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f6d1400d5f0 tx=0x7f6d1400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:24:01.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.880+0000 7f6d23400640 1 -- 172.21.15.27:0/2522460366 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d140125f0 con 0x7f6d2c0701a0 2024-01-15T18:24:01.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.880+0000 7f6d23400640 1 -- 172.21.15.27:0/2522460366 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d14012750 con 0x7f6d2c0701a0 2024-01-15T18:24:01.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.880+0000 7f6d32600640 1 -- 172.21.15.27:0/2522460366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d2c0cd7c0 con 0x7f6d2c0701a0 2024-01-15T18:24:01.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.880+0000 7f6d23400640 1 -- 172.21.15.27:0/2522460366 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d14012ae0 con 0x7f6d2c0701a0 2024-01-15T18:24:01.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.880+0000 7f6d32600640 1 -- 172.21.15.27:0/2522460366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d2c0cdd80 con 0x7f6d2c0701a0 2024-01-15T18:24:01.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.884+0000 7f6d32600640 1 -- 172.21.15.27:0/2522460366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d2c0cdf40 con 0x7f6d2c0701a0 2024-01-15T18:24:01.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.884+0000 7f6d23400640 1 -- 172.21.15.27:0/2522460366 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f6d14012c40 con 0x7f6d2c0701a0 2024-01-15T18:24:01.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.884+0000 7f6d23400640 1 --2- 172.21.15.27:0/2522460366 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6d04067a60 0x7f6d04069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:01.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.884+0000 7f6d23400640 1 -- 172.21.15.27:0/2522460366 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f6d14083630 con 0x7f6d2c0701a0 2024-01-15T18:24:01.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.884+0000 7f6d2b400640 1 --2- 172.21.15.27:0/2522460366 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6d04067a60 0x7f6d04069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:01.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.884+0000 7f6d2b400640 1 --2- 172.21.15.27:0/2522460366 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6d04067a60 0x7f6d04069f20 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c0ae000 tx=0x7f6d1c0b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:24:01.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:01.888+0000 7f6d23400640 1 -- 172.21.15.27:0/2522460366 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6d14052020 con 0x7f6d2c0701a0 2024-01-15T18:24:02.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:02.044+0000 7f6d32600640 1 -- 172.21.15.27:0/2522460366 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6d2c15d4c0 con 0x7f6d04067a60 2024-01-15T18:24:02.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:02.052+0000 7f6d23400640 1 -- 172.21.15.27:0/2522460366 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290543 (secure 0 0 0) 0x7f6d1c0aa8c0 con 0x7f6d04067a60 2024-01-15T18:24:02.049 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:24:02.051 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:24:02.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:02.056+0000 7f6d32600640 1 -- 172.21.15.27:0/2522460366 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6d04067a60 msgr2=0x7f6d04069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:02.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:02.056+0000 7f6d32600640 1 --2- 172.21.15.27:0/2522460366 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6d04067a60 0x7f6d04069f20 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c0ae000 tx=0x7f6d1c0b5040 comp rx=0 tx=0).stop 2024-01-15T18:24:02.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:02.056+0000 7f6d32600640 1 -- 172.21.15.27:0/2522460366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d2c0701a0 msgr2=0x7f6d2c0cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:02.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:02.056+0000 7f6d32600640 1 --2- 172.21.15.27:0/2522460366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d2c0701a0 0x7f6d2c0cc830 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f6d1400d5f0 tx=0x7f6d1400dab0 comp rx=0 tx=0).stop 2024-01-15T18:24:02.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:02.056+0000 7f6d32600640 1 -- 172.21.15.27:0/2522460366 shutdown_connections 2024-01-15T18:24:02.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:02.056+0000 7f6d32600640 1 --2- 172.21.15.27:0/2522460366 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f6d04067a60 0x7f6d04069f20 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:02.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:02.056+0000 7f6d32600640 1 --2- 172.21.15.27:0/2522460366 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f6d2c0c92b0 0x7f6d2c13fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:02.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:02.056+0000 7f6d32600640 1 --2- 172.21.15.27:0/2522460366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6d2c0701a0 0x7f6d2c0cc830 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:02.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:02.056+0000 7f6d32600640 1 --2- 172.21.15.27:0/2522460366 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f6d2c06e240 0x7f6d2c155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:02.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:02.056+0000 7f6d32600640 1 -- 172.21.15.27:0/2522460366 >> 172.21.15.27:0/2522460366 conn(0x7f6d2c0cec60 msgr2=0x7f6d2c0ae4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:24:02.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:02.056+0000 7f6d32600640 1 -- 172.21.15.27:0/2522460366 shutdown_connections 2024-01-15T18:24:02.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:02.056+0000 7f6d32600640 1 -- 172.21.15.27:0/2522460366 wait complete. 2024-01-15T18:24:02.071 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":681,"stamp":"2024-01-15T18:24:00.464360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5369916698,"num_objects":1311,"num_object_clones":1,"num_object_copies":2622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1311,"num_whiteouts":1,"num_read":651,"num_read_kb":2097275,"num_write":1745,"num_write_kb":5246031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5506,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3454,"ondisk_log_size":3454,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":11036512,"kb_used_data":10498848,"kb_used_omap":12,"kb_used_meta":537587,"kb_avail":743938208,"statfs":{"total":773094113280,"available":761792724992,"internally_reserved":0,"allocated":10750820352,"data_stored":10744589100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":550489648},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2942,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":506,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003414"},"pg_stats":[{"pgid":"17.3f","version":"110'53","reported_seq":78,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.666198+0000","last_change":"2024-01-15T18:18:33.168792+0000","last_active":"2024-01-15T18:23:31.666198+0000","last_peered":"2024-01-15T18:23:31.666198+0000","last_clean":"2024-01-15T18:23:31.666198+0000","last_became_active":"2024-01-15T18:18:33.168500+0000","last_became_peered":"2024-01-15T18:18:33.168500+0000","last_unstale":"2024-01-15T18:23:31.666198+0000","last_undegraded":"2024-01-15T18:23:31.666198+0000","last_fullsized":"2024-01-15T18:23:31.666198+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"110'45","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.710301+0000","last_change":"2024-01-15T18:18:33.151630+0000","last_active":"2024-01-15T18:23:31.710301+0000","last_peered":"2024-01-15T18:23:31.710301+0000","last_clean":"2024-01-15T18:23:31.710301+0000","last_became_active":"2024-01-15T18:18:33.151209+0000","last_became_peered":"2024-01-15T18:18:33.151209+0000","last_unstale":"2024-01-15T18:23:31.710301+0000","last_undegraded":"2024-01-15T18:23:31.710301+0000","last_fullsized":"2024-01-15T18:23:31.710301+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"110'44","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.851607+0000","last_change":"2024-01-15T18:18:33.163094+0000","last_active":"2024-01-15T18:23:31.851607+0000","last_peered":"2024-01-15T18:23:31.851607+0000","last_clean":"2024-01-15T18:23:31.851607+0000","last_became_active":"2024-01-15T18:18:33.162700+0000","last_became_peered":"2024-01-15T18:18:33.162700+0000","last_unstale":"2024-01-15T18:23:31.851607+0000","last_undegraded":"2024-01-15T18:23:31.851607+0000","last_fullsized":"2024-01-15T18:23:31.851607+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"110'62","reported_seq":92,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:37.761746+0000","last_change":"2024-01-15T18:23:37.761794+0000","last_active":"2024-01-15T18:23:37.761746+0000","last_peered":"2024-01-15T18:23:37.761746+0000","last_clean":"2024-01-15T18:23:37.761746+0000","last_became_active":"2024-01-15T18:18:33.160460+0000","last_became_peered":"2024-01-15T18:18:33.160460+0000","last_unstale":"2024-01-15T18:23:37.761746+0000","last_undegraded":"2024-01-15T18:23:37.761746+0000","last_fullsized":"2024-01-15T18:23:37.761746+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'62","last_scrub_stamp":"2024-01-15T18:23:37.761665+0000","last_deep_scrub":"110'62","last_deep_scrub_stamp":"2024-01-15T18:23:37.761665+0000","last_clean_scrub_stamp":"2024-01-15T18:23:37.761665+0000","objects_scrubbed":23,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:24:28.403935+0000","scrub_duration":0.14250365100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"110'27","reported_seq":50,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:35.184846+0000","last_change":"2024-01-15T18:18:33.155936+0000","last_active":"2024-01-15T18:23:35.184846+0000","last_peered":"2024-01-15T18:23:35.184846+0000","last_clean":"2024-01-15T18:23:35.184846+0000","last_became_active":"2024-01-15T18:18:33.155656+0000","last_became_peered":"2024-01-15T18:18:33.155656+0000","last_unstale":"2024-01-15T18:23:35.184846+0000","last_undegraded":"2024-01-15T18:23:35.184846+0000","last_fullsized":"2024-01-15T18:23:35.184846+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"110'48","reported_seq":75,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:35.487820+0000","last_change":"2024-01-15T18:18:33.152032+0000","last_active":"2024-01-15T18:23:35.487820+0000","last_peered":"2024-01-15T18:23:35.487820+0000","last_clean":"2024-01-15T18:23:35.487820+0000","last_became_active":"2024-01-15T18:18:33.151701+0000","last_became_peered":"2024-01-15T18:18:33.151701+0000","last_unstale":"2024-01-15T18:23:35.487820+0000","last_undegraded":"2024-01-15T18:23:35.487820+0000","last_fullsized":"2024-01-15T18:23:35.487820+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"110'43","reported_seq":63,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:34.192520+0000","last_change":"2024-01-15T18:18:33.153882+0000","last_active":"2024-01-15T18:23:34.192520+0000","last_peered":"2024-01-15T18:23:34.192520+0000","last_clean":"2024-01-15T18:23:34.192520+0000","last_became_active":"2024-01-15T18:18:33.153264+0000","last_became_peered":"2024-01-15T18:18:33.153264+0000","last_unstale":"2024-01-15T18:23:34.192520+0000","last_undegraded":"2024-01-15T18:23:34.192520+0000","last_fullsized":"2024-01-15T18:23:34.192520+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"110'61","reported_seq":93,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:56.297341+0000","last_change":"2024-01-15T18:23:56.297378+0000","last_active":"2024-01-15T18:23:56.297341+0000","last_peered":"2024-01-15T18:23:56.297341+0000","last_clean":"2024-01-15T18:23:56.297341+0000","last_became_active":"2024-01-15T18:18:33.163896+0000","last_became_peered":"2024-01-15T18:18:33.163896+0000","last_unstale":"2024-01-15T18:23:56.297341+0000","last_undegraded":"2024-01-15T18:23:56.297341+0000","last_fullsized":"2024-01-15T18:23:56.297341+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'61","last_scrub_stamp":"2024-01-15T18:23:56.297250+0000","last_deep_scrub":"110'61","last_deep_scrub_stamp":"2024-01-15T18:23:56.297250+0000","last_clean_scrub_stamp":"2024-01-15T18:23:56.297250+0000","objects_scrubbed":19,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:44:20.320452+0000","scrub_duration":0.120656875,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"110'47","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.030521+0000","last_change":"2024-01-15T18:18:33.158682+0000","last_active":"2024-01-15T18:23:32.030521+0000","last_peered":"2024-01-15T18:23:32.030521+0000","last_clean":"2024-01-15T18:23:32.030521+0000","last_became_active":"2024-01-15T18:18:33.158399+0000","last_became_peered":"2024-01-15T18:18:33.158399+0000","last_unstale":"2024-01-15T18:23:32.030521+0000","last_undegraded":"2024-01-15T18:23:32.030521+0000","last_fullsized":"2024-01-15T18:23:32.030521+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"110'47","reported_seq":73,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.054699+0000","last_change":"2024-01-15T18:18:33.159936+0000","last_active":"2024-01-15T18:23:32.054699+0000","last_peered":"2024-01-15T18:23:32.054699+0000","last_clean":"2024-01-15T18:23:32.054699+0000","last_became_active":"2024-01-15T18:18:33.159562+0000","last_became_peered":"2024-01-15T18:18:33.159562+0000","last_unstale":"2024-01-15T18:23:32.054699+0000","last_undegraded":"2024-01-15T18:23:32.054699+0000","last_fullsized":"2024-01-15T18:23:32.054699+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"110'47","reported_seq":75,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.998202+0000","last_change":"2024-01-15T18:18:33.155247+0000","last_active":"2024-01-15T18:23:31.998202+0000","last_peered":"2024-01-15T18:23:31.998202+0000","last_clean":"2024-01-15T18:23:31.998202+0000","last_became_active":"2024-01-15T18:18:33.154972+0000","last_became_peered":"2024-01-15T18:18:33.154972+0000","last_unstale":"2024-01-15T18:23:31.998202+0000","last_undegraded":"2024-01-15T18:23:31.998202+0000","last_fullsized":"2024-01-15T18:23:31.998202+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"110'48","reported_seq":80,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.959268+0000","last_change":"2024-01-15T18:18:33.158336+0000","last_active":"2024-01-15T18:23:31.959268+0000","last_peered":"2024-01-15T18:23:31.959268+0000","last_clean":"2024-01-15T18:23:31.959268+0000","last_became_active":"2024-01-15T18:18:33.158064+0000","last_became_peered":"2024-01-15T18:18:33.158064+0000","last_unstale":"2024-01-15T18:23:31.959268+0000","last_undegraded":"2024-01-15T18:23:31.959268+0000","last_fullsized":"2024-01-15T18:23:31.959268+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":16,"num_read_kb":61441,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"110'38","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.553796+0000","last_change":"2024-01-15T18:18:33.163912+0000","last_active":"2024-01-15T18:23:31.553796+0000","last_peered":"2024-01-15T18:23:31.553796+0000","last_clean":"2024-01-15T18:23:31.553796+0000","last_became_active":"2024-01-15T18:18:33.163610+0000","last_became_peered":"2024-01-15T18:18:33.163610+0000","last_unstale":"2024-01-15T18:23:31.553796+0000","last_undegraded":"2024-01-15T18:23:31.553796+0000","last_fullsized":"2024-01-15T18:23:31.553796+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"110'70","reported_seq":102,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:52.349564+0000","last_change":"2024-01-15T18:23:52.349588+0000","last_active":"2024-01-15T18:23:52.349564+0000","last_peered":"2024-01-15T18:23:52.349564+0000","last_clean":"2024-01-15T18:23:52.349564+0000","last_became_active":"2024-01-15T18:18:33.169530+0000","last_became_peered":"2024-01-15T18:18:33.169530+0000","last_unstale":"2024-01-15T18:23:52.349564+0000","last_undegraded":"2024-01-15T18:23:52.349564+0000","last_fullsized":"2024-01-15T18:23:52.349564+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'70","last_scrub_stamp":"2024-01-15T18:23:52.349480+0000","last_deep_scrub":"110'70","last_deep_scrub_stamp":"2024-01-15T18:23:52.349480+0000","last_clean_scrub_stamp":"2024-01-15T18:23:52.349480+0000","objects_scrubbed":27,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:02.669418+0000","scrub_duration":0.19608498999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"110'41","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:53.303856+0000","last_change":"2024-01-15T18:23:53.303892+0000","last_active":"2024-01-15T18:23:53.303856+0000","last_peered":"2024-01-15T18:23:53.303856+0000","last_clean":"2024-01-15T18:23:53.303856+0000","last_became_active":"2024-01-15T18:18:33.151040+0000","last_became_peered":"2024-01-15T18:18:33.151040+0000","last_unstale":"2024-01-15T18:23:53.303856+0000","last_undegraded":"2024-01-15T18:23:53.303856+0000","last_fullsized":"2024-01-15T18:23:53.303856+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'41","last_scrub_stamp":"2024-01-15T18:23:53.303773+0000","last_deep_scrub":"110'41","last_deep_scrub_stamp":"2024-01-15T18:23:53.303773+0000","last_clean_scrub_stamp":"2024-01-15T18:23:53.303773+0000","objects_scrubbed":12,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:12:23.550874+0000","scrub_duration":0.088010931000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"110'42","reported_seq":61,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.667801+0000","last_change":"2024-01-15T18:18:33.166327+0000","last_active":"2024-01-15T18:23:31.667801+0000","last_peered":"2024-01-15T18:23:31.667801+0000","last_clean":"2024-01-15T18:23:31.667801+0000","last_became_active":"2024-01-15T18:18:33.166014+0000","last_became_peered":"2024-01-15T18:18:33.166014+0000","last_unstale":"2024-01-15T18:23:31.667801+0000","last_undegraded":"2024-01-15T18:23:31.667801+0000","last_fullsized":"2024-01-15T18:23:31.667801+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"110'44","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.912923+0000","last_change":"2024-01-15T18:18:33.168613+0000","last_active":"2024-01-15T18:23:31.912923+0000","last_peered":"2024-01-15T18:23:31.912923+0000","last_clean":"2024-01-15T18:23:31.912923+0000","last_became_active":"2024-01-15T18:18:33.167975+0000","last_became_peered":"2024-01-15T18:18:33.167975+0000","last_unstale":"2024-01-15T18:23:31.912923+0000","last_undegraded":"2024-01-15T18:23:31.912923+0000","last_fullsized":"2024-01-15T18:23:31.912923+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"110'67","reported_seq":104,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:49.598996+0000","last_change":"2024-01-15T18:23:49.599036+0000","last_active":"2024-01-15T18:23:49.598996+0000","last_peered":"2024-01-15T18:23:49.598996+0000","last_clean":"2024-01-15T18:23:49.598996+0000","last_became_active":"2024-01-15T18:18:33.154816+0000","last_became_peered":"2024-01-15T18:18:33.154816+0000","last_unstale":"2024-01-15T18:23:49.598996+0000","last_undegraded":"2024-01-15T18:23:49.598996+0000","last_fullsized":"2024-01-15T18:23:49.598996+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'67","last_scrub_stamp":"2024-01-15T18:23:49.598911+0000","last_deep_scrub":"110'67","last_deep_scrub_stamp":"2024-01-15T18:23:49.598911+0000","last_clean_scrub_stamp":"2024-01-15T18:23:49.598911+0000","objects_scrubbed":21,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:43:00.465860+0000","scrub_duration":0.149097972,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"110'44","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.730734+0000","last_change":"2024-01-15T18:18:33.164028+0000","last_active":"2024-01-15T18:23:31.730734+0000","last_peered":"2024-01-15T18:23:31.730734+0000","last_clean":"2024-01-15T18:23:31.730734+0000","last_became_active":"2024-01-15T18:18:33.163421+0000","last_became_peered":"2024-01-15T18:18:33.163421+0000","last_unstale":"2024-01-15T18:23:31.730734+0000","last_undegraded":"2024-01-15T18:23:31.730734+0000","last_fullsized":"2024-01-15T18:23:31.730734+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"110'46","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.956031+0000","last_change":"2024-01-15T18:18:33.160995+0000","last_active":"2024-01-15T18:23:31.956031+0000","last_peered":"2024-01-15T18:23:31.956031+0000","last_clean":"2024-01-15T18:23:31.956031+0000","last_became_active":"2024-01-15T18:18:33.160047+0000","last_became_peered":"2024-01-15T18:18:33.160047+0000","last_unstale":"2024-01-15T18:23:31.956031+0000","last_undegraded":"2024-01-15T18:23:31.956031+0000","last_fullsized":"2024-01-15T18:23:31.956031+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"110'76","reported_seq":114,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:56.964314+0000","last_change":"2024-01-15T18:23:56.964335+0000","last_active":"2024-01-15T18:23:56.964314+0000","last_peered":"2024-01-15T18:23:56.964314+0000","last_clean":"2024-01-15T18:23:56.964314+0000","last_became_active":"2024-01-15T18:18:33.152128+0000","last_became_peered":"2024-01-15T18:18:33.152128+0000","last_unstale":"2024-01-15T18:23:56.964314+0000","last_undegraded":"2024-01-15T18:23:56.964314+0000","last_fullsized":"2024-01-15T18:23:56.964314+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'76","last_scrub_stamp":"2024-01-15T18:23:56.964221+0000","last_deep_scrub":"110'76","last_deep_scrub_stamp":"2024-01-15T18:23:56.964221+0000","last_clean_scrub_stamp":"2024-01-15T18:23:56.964221+0000","objects_scrubbed":26,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:39:29.172538+0000","scrub_duration":0.16674940499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"110'47","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.064950+0000","last_change":"2024-01-15T18:18:33.163614+0000","last_active":"2024-01-15T18:23:32.064950+0000","last_peered":"2024-01-15T18:23:32.064950+0000","last_clean":"2024-01-15T18:23:32.064950+0000","last_became_active":"2024-01-15T18:18:33.163252+0000","last_became_peered":"2024-01-15T18:18:33.163252+0000","last_unstale":"2024-01-15T18:23:32.064950+0000","last_undegraded":"2024-01-15T18:23:32.064950+0000","last_fullsized":"2024-01-15T18:23:32.064950+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":24578,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"110'50","reported_seq":83,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:50.982646+0000","last_change":"2024-01-15T18:23:50.982684+0000","last_active":"2024-01-15T18:23:50.982646+0000","last_peered":"2024-01-15T18:23:50.982646+0000","last_clean":"2024-01-15T18:23:50.982646+0000","last_became_active":"2024-01-15T18:18:33.159557+0000","last_became_peered":"2024-01-15T18:18:33.159557+0000","last_unstale":"2024-01-15T18:23:50.982646+0000","last_undegraded":"2024-01-15T18:23:50.982646+0000","last_fullsized":"2024-01-15T18:23:50.982646+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'50","last_scrub_stamp":"2024-01-15T18:23:50.982553+0000","last_deep_scrub":"110'50","last_deep_scrub_stamp":"2024-01-15T18:23:50.982553+0000","last_clean_scrub_stamp":"2024-01-15T18:23:50.982553+0000","objects_scrubbed":15,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:10:19.036866+0000","scrub_duration":0.10536667600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"110'43","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.014236+0000","last_change":"2024-01-15T18:18:33.166766+0000","last_active":"2024-01-15T18:23:32.014236+0000","last_peered":"2024-01-15T18:23:32.014236+0000","last_clean":"2024-01-15T18:23:32.014236+0000","last_became_active":"2024-01-15T18:18:33.165600+0000","last_became_peered":"2024-01-15T18:18:33.165600+0000","last_unstale":"2024-01-15T18:23:32.014236+0000","last_undegraded":"2024-01-15T18:23:32.014236+0000","last_fullsized":"2024-01-15T18:23:32.014236+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"110'38","reported_seq":64,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.985981+0000","last_change":"2024-01-15T18:18:33.168720+0000","last_active":"2024-01-15T18:23:31.985981+0000","last_peered":"2024-01-15T18:23:31.985981+0000","last_clean":"2024-01-15T18:23:31.985981+0000","last_became_active":"2024-01-15T18:18:33.168349+0000","last_became_peered":"2024-01-15T18:18:33.168349+0000","last_unstale":"2024-01-15T18:23:31.985981+0000","last_undegraded":"2024-01-15T18:23:31.985981+0000","last_fullsized":"2024-01-15T18:23:31.985981+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"110'53","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.034275+0000","last_change":"2024-01-15T18:18:33.159262+0000","last_active":"2024-01-15T18:23:32.034275+0000","last_peered":"2024-01-15T18:23:32.034275+0000","last_clean":"2024-01-15T18:23:32.034275+0000","last_became_active":"2024-01-15T18:18:33.158946+0000","last_became_peered":"2024-01-15T18:18:33.158946+0000","last_unstale":"2024-01-15T18:23:32.034275+0000","last_undegraded":"2024-01-15T18:23:32.034275+0000","last_fullsized":"2024-01-15T18:23:32.034275+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"110'37","reported_seq":61,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.417591+0000","last_change":"2024-01-15T18:18:33.161814+0000","last_active":"2024-01-15T18:23:31.417591+0000","last_peered":"2024-01-15T18:23:31.417591+0000","last_clean":"2024-01-15T18:23:31.417591+0000","last_became_active":"2024-01-15T18:18:33.161463+0000","last_became_peered":"2024-01-15T18:18:33.161463+0000","last_unstale":"2024-01-15T18:23:31.417591+0000","last_undegraded":"2024-01-15T18:23:31.417591+0000","last_fullsized":"2024-01-15T18:23:31.417591+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886260,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":28673,"num_write":21,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"110'38","reported_seq":57,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.030549+0000","last_change":"2024-01-15T18:18:33.162952+0000","last_active":"2024-01-15T18:23:32.030549+0000","last_peered":"2024-01-15T18:23:32.030549+0000","last_clean":"2024-01-15T18:23:32.030549+0000","last_became_active":"2024-01-15T18:18:33.162669+0000","last_became_peered":"2024-01-15T18:18:33.162669+0000","last_unstale":"2024-01-15T18:23:32.030549+0000","last_undegraded":"2024-01-15T18:23:32.030549+0000","last_fullsized":"2024-01-15T18:23:32.030549+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"110'46","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.968407+0000","last_change":"2024-01-15T18:18:33.160658+0000","last_active":"2024-01-15T18:23:31.968407+0000","last_peered":"2024-01-15T18:23:31.968407+0000","last_clean":"2024-01-15T18:23:31.968407+0000","last_became_active":"2024-01-15T18:18:33.160099+0000","last_became_peered":"2024-01-15T18:18:33.160099+0000","last_unstale":"2024-01-15T18:23:31.968407+0000","last_undegraded":"2024-01-15T18:23:31.968407+0000","last_fullsized":"2024-01-15T18:23:31.968407+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"110'41","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.892179+0000","last_change":"2024-01-15T18:18:33.161061+0000","last_active":"2024-01-15T18:23:31.892179+0000","last_peered":"2024-01-15T18:23:31.892179+0000","last_clean":"2024-01-15T18:23:31.892179+0000","last_became_active":"2024-01-15T18:18:33.160725+0000","last_became_peered":"2024-01-15T18:18:33.160725+0000","last_unstale":"2024-01-15T18:23:31.892179+0000","last_undegraded":"2024-01-15T18:23:31.892179+0000","last_fullsized":"2024-01-15T18:23:31.892179+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"110'85","reported_seq":122,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:40.714947+0000","last_change":"2024-01-15T18:23:40.714970+0000","last_active":"2024-01-15T18:23:40.714947+0000","last_peered":"2024-01-15T18:23:40.714947+0000","last_clean":"2024-01-15T18:23:40.714947+0000","last_became_active":"2024-01-15T18:18:33.158701+0000","last_became_peered":"2024-01-15T18:18:33.158701+0000","last_unstale":"2024-01-15T18:23:40.714947+0000","last_undegraded":"2024-01-15T18:23:40.714947+0000","last_fullsized":"2024-01-15T18:23:40.714947+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'85","last_scrub_stamp":"2024-01-15T18:23:40.714867+0000","last_deep_scrub":"110'85","last_deep_scrub_stamp":"2024-01-15T18:23:40.714867+0000","last_clean_scrub_stamp":"2024-01-15T18:23:40.714867+0000","objects_scrubbed":26,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:21.790890+0000","scrub_duration":0.16315633500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"110'54","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.065509+0000","last_change":"2024-01-15T18:18:33.152241+0000","last_active":"2024-01-15T18:23:32.065509+0000","last_peered":"2024-01-15T18:23:32.065509+0000","last_clean":"2024-01-15T18:23:32.065509+0000","last_became_active":"2024-01-15T18:18:33.151250+0000","last_became_peered":"2024-01-15T18:18:33.151250+0000","last_unstale":"2024-01-15T18:23:32.065509+0000","last_undegraded":"2024-01-15T18:23:32.065509+0000","last_fullsized":"2024-01-15T18:23:32.065509+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"110'34","reported_seq":56,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.735539+0000","last_change":"2024-01-15T18:18:33.154972+0000","last_active":"2024-01-15T18:23:31.735539+0000","last_peered":"2024-01-15T18:23:31.735539+0000","last_clean":"2024-01-15T18:23:31.735539+0000","last_became_active":"2024-01-15T18:18:33.154569+0000","last_became_peered":"2024-01-15T18:18:33.154569+0000","last_unstale":"2024-01-15T18:23:31.735539+0000","last_undegraded":"2024-01-15T18:23:31.735539+0000","last_fullsized":"2024-01-15T18:23:31.735539+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"110'6","reported_seq":31,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:52.319457+0000","last_change":"2024-01-15T18:23:52.319495+0000","last_active":"2024-01-15T18:23:52.319457+0000","last_peered":"2024-01-15T18:23:52.319457+0000","last_clean":"2024-01-15T18:23:52.319457+0000","last_became_active":"2024-01-15T18:18:32.216337+0000","last_became_peered":"2024-01-15T18:18:32.216337+0000","last_unstale":"2024-01-15T18:23:52.319457+0000","last_undegraded":"2024-01-15T18:23:52.319457+0000","last_fullsized":"2024-01-15T18:23:52.319457+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'6","last_scrub_stamp":"2024-01-15T18:23:52.319346+0000","last_deep_scrub":"110'6","last_deep_scrub_stamp":"2024-01-15T18:23:52.319346+0000","last_clean_scrub_stamp":"2024-01-15T18:23:52.319346+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:00:46.740483+0000","scrub_duration":0.003716685,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"110'47","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.688853+0000","last_change":"2024-01-15T18:18:33.167015+0000","last_active":"2024-01-15T18:23:31.688853+0000","last_peered":"2024-01-15T18:23:31.688853+0000","last_clean":"2024-01-15T18:23:31.688853+0000","last_became_active":"2024-01-15T18:18:33.166726+0000","last_became_peered":"2024-01-15T18:18:33.166726+0000","last_unstale":"2024-01-15T18:23:31.688853+0000","last_undegraded":"2024-01-15T18:23:31.688853+0000","last_fullsized":"2024-01-15T18:23:31.688853+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"107'1","reported_seq":26,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:49.184521+0000","last_change":"2024-01-15T18:23:49.184542+0000","last_active":"2024-01-15T18:23:49.184521+0000","last_peered":"2024-01-15T18:23:49.184521+0000","last_clean":"2024-01-15T18:23:49.184521+0000","last_became_active":"2024-01-15T18:18:32.219596+0000","last_became_peered":"2024-01-15T18:18:32.219596+0000","last_unstale":"2024-01-15T18:23:49.184521+0000","last_undegraded":"2024-01-15T18:23:49.184521+0000","last_fullsized":"2024-01-15T18:23:49.184521+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'1","last_scrub_stamp":"2024-01-15T18:23:49.184382+0000","last_deep_scrub":"107'1","last_deep_scrub_stamp":"2024-01-15T18:23:49.184382+0000","last_clean_scrub_stamp":"2024-01-15T18:23:49.184382+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:22:53.818633+0000","scrub_duration":0.0021746040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"110'43","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:35.487859+0000","last_change":"2024-01-15T18:18:33.150743+0000","last_active":"2024-01-15T18:23:35.487859+0000","last_peered":"2024-01-15T18:23:35.487859+0000","last_clean":"2024-01-15T18:23:35.487859+0000","last_became_active":"2024-01-15T18:18:33.150429+0000","last_became_peered":"2024-01-15T18:18:33.150429+0000","last_unstale":"2024-01-15T18:23:35.487859+0000","last_undegraded":"2024-01-15T18:23:35.487859+0000","last_fullsized":"2024-01-15T18:23:35.487859+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"110'4","reported_seq":29,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:52.947456+0000","last_change":"2024-01-15T18:23:52.947480+0000","last_active":"2024-01-15T18:23:52.947456+0000","last_peered":"2024-01-15T18:23:52.947456+0000","last_clean":"2024-01-15T18:23:52.947456+0000","last_became_active":"2024-01-15T18:18:32.212071+0000","last_became_peered":"2024-01-15T18:18:32.212071+0000","last_unstale":"2024-01-15T18:23:52.947456+0000","last_undegraded":"2024-01-15T18:23:52.947456+0000","last_fullsized":"2024-01-15T18:23:52.947456+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'4","last_scrub_stamp":"2024-01-15T18:23:52.947355+0000","last_deep_scrub":"110'4","last_deep_scrub_stamp":"2024-01-15T18:23:52.947355+0000","last_clean_scrub_stamp":"2024-01-15T18:23:52.947355+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:46.837594+0000","scrub_duration":0.0037854780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"110'56","reported_seq":83,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:35.193344+0000","last_change":"2024-01-15T18:18:33.165545+0000","last_active":"2024-01-15T18:23:35.193344+0000","last_peered":"2024-01-15T18:23:35.193344+0000","last_clean":"2024-01-15T18:23:35.193344+0000","last_became_active":"2024-01-15T18:18:33.165258+0000","last_became_peered":"2024-01-15T18:18:33.165258+0000","last_unstale":"2024-01-15T18:23:35.193344+0000","last_undegraded":"2024-01-15T18:23:35.193344+0000","last_fullsized":"2024-01-15T18:23:35.193344+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"110'3","reported_seq":28,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:56.916317+0000","last_change":"2024-01-15T18:23:56.916341+0000","last_active":"2024-01-15T18:23:56.916317+0000","last_peered":"2024-01-15T18:23:56.916317+0000","last_clean":"2024-01-15T18:23:56.916317+0000","last_became_active":"2024-01-15T18:18:32.216492+0000","last_became_peered":"2024-01-15T18:18:32.216492+0000","last_unstale":"2024-01-15T18:23:56.916317+0000","last_undegraded":"2024-01-15T18:23:56.916317+0000","last_fullsized":"2024-01-15T18:23:56.916317+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'3","last_scrub_stamp":"2024-01-15T18:23:56.916206+0000","last_deep_scrub":"110'3","last_deep_scrub_stamp":"2024-01-15T18:23:56.916206+0000","last_clean_scrub_stamp":"2024-01-15T18:23:56.916206+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:52:35.385555+0000","scrub_duration":0.0031224619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"110'39","reported_seq":63,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:34.763315+0000","last_change":"2024-01-15T18:18:33.157996+0000","last_active":"2024-01-15T18:23:34.763315+0000","last_peered":"2024-01-15T18:23:34.763315+0000","last_clean":"2024-01-15T18:23:34.763315+0000","last_became_active":"2024-01-15T18:18:33.157687+0000","last_became_peered":"2024-01-15T18:18:33.157687+0000","last_unstale":"2024-01-15T18:23:34.763315+0000","last_undegraded":"2024-01-15T18:23:34.763315+0000","last_fullsized":"2024-01-15T18:23:34.763315+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"107'1","reported_seq":26,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:56.330972+0000","last_change":"2024-01-15T18:23:56.330992+0000","last_active":"2024-01-15T18:23:56.330972+0000","last_peered":"2024-01-15T18:23:56.330972+0000","last_clean":"2024-01-15T18:23:56.330972+0000","last_became_active":"2024-01-15T18:18:32.214453+0000","last_became_peered":"2024-01-15T18:18:32.214453+0000","last_unstale":"2024-01-15T18:23:56.330972+0000","last_undegraded":"2024-01-15T18:23:56.330972+0000","last_fullsized":"2024-01-15T18:23:56.330972+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'1","last_scrub_stamp":"2024-01-15T18:23:56.330895+0000","last_deep_scrub":"107'1","last_deep_scrub_stamp":"2024-01-15T18:23:56.330895+0000","last_clean_scrub_stamp":"2024-01-15T18:23:56.330895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:22:46.630675+0000","scrub_duration":0.0018859580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"110'49","reported_seq":78,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:35.200051+0000","last_change":"2024-01-15T18:18:33.153355+0000","last_active":"2024-01-15T18:23:35.200051+0000","last_peered":"2024-01-15T18:23:35.200051+0000","last_clean":"2024-01-15T18:23:35.200051+0000","last_became_active":"2024-01-15T18:18:33.153058+0000","last_became_peered":"2024-01-15T18:18:33.153058+0000","last_unstale":"2024-01-15T18:23:35.200051+0000","last_undegraded":"2024-01-15T18:23:35.200051+0000","last_fullsized":"2024-01-15T18:23:35.200051+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"107'1","reported_seq":26,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:42.849182+0000","last_change":"2024-01-15T18:23:42.849208+0000","last_active":"2024-01-15T18:23:42.849182+0000","last_peered":"2024-01-15T18:23:42.849182+0000","last_clean":"2024-01-15T18:23:42.849182+0000","last_became_active":"2024-01-15T18:18:32.215417+0000","last_became_peered":"2024-01-15T18:18:32.215417+0000","last_unstale":"2024-01-15T18:23:42.849182+0000","last_undegraded":"2024-01-15T18:23:42.849182+0000","last_fullsized":"2024-01-15T18:23:42.849182+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'1","last_scrub_stamp":"2024-01-15T18:23:42.849065+0000","last_deep_scrub":"107'1","last_deep_scrub_stamp":"2024-01-15T18:23:42.849065+0000","last_clean_scrub_stamp":"2024-01-15T18:23:42.849065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:40:12.007683+0000","scrub_duration":0.0030182830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"110'53","reported_seq":75,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:35.184808+0000","last_change":"2024-01-15T18:18:43.024618+0000","last_active":"2024-01-15T18:23:35.184808+0000","last_peered":"2024-01-15T18:23:35.184808+0000","last_clean":"2024-01-15T18:23:35.184808+0000","last_became_active":"2024-01-15T18:18:43.024221+0000","last_became_peered":"2024-01-15T18:18:43.024221+0000","last_unstale":"2024-01-15T18:23:35.184808+0000","last_undegraded":"2024-01-15T18:23:35.184808+0000","last_fullsized":"2024-01-15T18:23:35.184808+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857792,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:41.955122+0000","last_change":"2024-01-15T18:23:41.955145+0000","last_active":"2024-01-15T18:23:41.955122+0000","last_peered":"2024-01-15T18:23:41.955122+0000","last_clean":"2024-01-15T18:23:41.955122+0000","last_became_active":"2024-01-15T18:18:32.215465+0000","last_became_peered":"2024-01-15T18:18:32.215465+0000","last_unstale":"2024-01-15T18:23:41.955122+0000","last_undegraded":"2024-01-15T18:23:41.955122+0000","last_fullsized":"2024-01-15T18:23:41.955122+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:41.954982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:41.954982+0000","last_clean_scrub_stamp":"2024-01-15T18:23:41.954982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:42:52.821931+0000","scrub_duration":0.002690853,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"110'43","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.931569+0000","last_change":"2024-01-15T18:18:33.154067+0000","last_active":"2024-01-15T18:23:31.931569+0000","last_peered":"2024-01-15T18:23:31.931569+0000","last_clean":"2024-01-15T18:23:31.931569+0000","last_became_active":"2024-01-15T18:18:33.153610+0000","last_became_peered":"2024-01-15T18:18:33.153610+0000","last_unstale":"2024-01-15T18:23:31.931569+0000","last_undegraded":"2024-01-15T18:23:31.931569+0000","last_fullsized":"2024-01-15T18:23:31.931569+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:50.513004+0000","last_change":"2024-01-15T18:23:50.513024+0000","last_active":"2024-01-15T18:23:50.513004+0000","last_peered":"2024-01-15T18:23:50.513004+0000","last_clean":"2024-01-15T18:23:50.513004+0000","last_became_active":"2024-01-15T18:18:32.219555+0000","last_became_peered":"2024-01-15T18:18:32.219555+0000","last_unstale":"2024-01-15T18:23:50.513004+0000","last_undegraded":"2024-01-15T18:23:50.513004+0000","last_fullsized":"2024-01-15T18:23:50.513004+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:50.512904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:50.512904+0000","last_clean_scrub_stamp":"2024-01-15T18:23:50.512904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:03:40.746495+0000","scrub_duration":0.0020306230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"110'42","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.136220+0000","last_change":"2024-01-15T18:18:33.156038+0000","last_active":"2024-01-15T18:23:31.136220+0000","last_peered":"2024-01-15T18:23:31.136220+0000","last_clean":"2024-01-15T18:23:31.136220+0000","last_became_active":"2024-01-15T18:18:33.155765+0000","last_became_peered":"2024-01-15T18:18:33.155765+0000","last_unstale":"2024-01-15T18:23:31.136220+0000","last_undegraded":"2024-01-15T18:23:31.136220+0000","last_fullsized":"2024-01-15T18:23:31.136220+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"110'1","reported_seq":26,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:54.901556+0000","last_change":"2024-01-15T18:23:54.901577+0000","last_active":"2024-01-15T18:23:54.901556+0000","last_peered":"2024-01-15T18:23:54.901556+0000","last_clean":"2024-01-15T18:23:54.901556+0000","last_became_active":"2024-01-15T18:18:32.212511+0000","last_became_peered":"2024-01-15T18:18:32.212511+0000","last_unstale":"2024-01-15T18:23:54.901556+0000","last_undegraded":"2024-01-15T18:23:54.901556+0000","last_fullsized":"2024-01-15T18:23:54.901556+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1","last_scrub_stamp":"2024-01-15T18:23:54.901458+0000","last_deep_scrub":"110'1","last_deep_scrub_stamp":"2024-01-15T18:23:54.901458+0000","last_clean_scrub_stamp":"2024-01-15T18:23:54.901458+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:49:11.641921+0000","scrub_duration":0.0018983540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"110'53","reported_seq":82,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:51.617489+0000","last_change":"2024-01-15T18:23:51.617525+0000","last_active":"2024-01-15T18:23:51.617489+0000","last_peered":"2024-01-15T18:23:51.617489+0000","last_clean":"2024-01-15T18:23:51.617489+0000","last_became_active":"2024-01-15T18:18:33.164530+0000","last_became_peered":"2024-01-15T18:18:33.164530+0000","last_unstale":"2024-01-15T18:23:51.617489+0000","last_undegraded":"2024-01-15T18:23:51.617489+0000","last_fullsized":"2024-01-15T18:23:51.617489+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'53","last_scrub_stamp":"2024-01-15T18:23:51.617414+0000","last_deep_scrub":"110'53","last_deep_scrub_stamp":"2024-01-15T18:23:51.617414+0000","last_clean_scrub_stamp":"2024-01-15T18:23:51.617414+0000","objects_scrubbed":18,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:57:57.307137+0000","scrub_duration":0.11477269399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:55.893196+0000","last_change":"2024-01-15T18:23:55.893219+0000","last_active":"2024-01-15T18:23:55.893196+0000","last_peered":"2024-01-15T18:23:55.893196+0000","last_clean":"2024-01-15T18:23:55.893196+0000","last_became_active":"2024-01-15T18:18:32.216337+0000","last_became_peered":"2024-01-15T18:18:32.216337+0000","last_unstale":"2024-01-15T18:23:55.893196+0000","last_undegraded":"2024-01-15T18:23:55.893196+0000","last_fullsized":"2024-01-15T18:23:55.893196+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:55.893099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:55.893099+0000","last_clean_scrub_stamp":"2024-01-15T18:23:55.893099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:45:26.429594+0000","scrub_duration":0.002804001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"110'43","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.038280+0000","last_change":"2024-01-15T18:18:33.148278+0000","last_active":"2024-01-15T18:23:32.038280+0000","last_peered":"2024-01-15T18:23:32.038280+0000","last_clean":"2024-01-15T18:23:32.038280+0000","last_became_active":"2024-01-15T18:18:33.148016+0000","last_became_peered":"2024-01-15T18:18:33.148016+0000","last_unstale":"2024-01-15T18:23:32.038280+0000","last_undegraded":"2024-01-15T18:23:32.038280+0000","last_fullsized":"2024-01-15T18:23:32.038280+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"110'14","reported_seq":39,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:55.542471+0000","last_change":"2024-01-15T18:23:55.542525+0000","last_active":"2024-01-15T18:23:55.542471+0000","last_peered":"2024-01-15T18:23:55.542471+0000","last_clean":"2024-01-15T18:23:55.542471+0000","last_became_active":"2024-01-15T18:18:32.214186+0000","last_became_peered":"2024-01-15T18:18:32.214186+0000","last_unstale":"2024-01-15T18:23:55.542471+0000","last_undegraded":"2024-01-15T18:23:55.542471+0000","last_fullsized":"2024-01-15T18:23:55.542471+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'14","last_scrub_stamp":"2024-01-15T18:23:55.542363+0000","last_deep_scrub":"110'14","last_deep_scrub_stamp":"2024-01-15T18:23:55.542363+0000","last_clean_scrub_stamp":"2024-01-15T18:23:55.542363+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:36:43.100265+0000","scrub_duration":0.0043649800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"110'43","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.580933+0000","last_change":"2024-01-15T18:18:33.159329+0000","last_active":"2024-01-15T18:23:31.580933+0000","last_peered":"2024-01-15T18:23:31.580933+0000","last_clean":"2024-01-15T18:23:31.580933+0000","last_became_active":"2024-01-15T18:18:33.158541+0000","last_became_peered":"2024-01-15T18:18:33.158541+0000","last_unstale":"2024-01-15T18:23:31.580933+0000","last_undegraded":"2024-01-15T18:23:31.580933+0000","last_fullsized":"2024-01-15T18:23:31.580933+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"110'188","reported_seq":213,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:53.472690+0000","last_change":"2024-01-15T18:23:53.472714+0000","last_active":"2024-01-15T18:23:53.472690+0000","last_peered":"2024-01-15T18:23:53.472690+0000","last_clean":"2024-01-15T18:23:53.472690+0000","last_became_active":"2024-01-15T18:18:32.218002+0000","last_became_peered":"2024-01-15T18:18:32.218002+0000","last_unstale":"2024-01-15T18:23:53.472690+0000","last_undegraded":"2024-01-15T18:23:53.472690+0000","last_fullsized":"2024-01-15T18:23:53.472690+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'188","last_scrub_stamp":"2024-01-15T18:23:53.472606+0000","last_deep_scrub":"110'188","last_deep_scrub_stamp":"2024-01-15T18:23:53.472606+0000","last_clean_scrub_stamp":"2024-01-15T18:23:53.472606+0000","objects_scrubbed":4,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:11:55.352315+0000","scrub_duration":0.0071479380000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":614075,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":185,"num_write_kb":681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"110'44","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.064489+0000","last_change":"2024-01-15T18:18:33.151414+0000","last_active":"2024-01-15T18:23:32.064489+0000","last_peered":"2024-01-15T18:23:32.064489+0000","last_clean":"2024-01-15T18:23:32.064489+0000","last_became_active":"2024-01-15T18:18:33.151132+0000","last_became_peered":"2024-01-15T18:18:33.151132+0000","last_unstale":"2024-01-15T18:23:32.064489+0000","last_undegraded":"2024-01-15T18:23:32.064489+0000","last_fullsized":"2024-01-15T18:23:32.064489+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"110'4","reported_seq":29,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:42.234836+0000","last_change":"2024-01-15T18:23:42.234860+0000","last_active":"2024-01-15T18:23:42.234836+0000","last_peered":"2024-01-15T18:23:42.234836+0000","last_clean":"2024-01-15T18:23:42.234836+0000","last_became_active":"2024-01-15T18:18:32.215073+0000","last_became_peered":"2024-01-15T18:18:32.215073+0000","last_unstale":"2024-01-15T18:23:42.234836+0000","last_undegraded":"2024-01-15T18:23:42.234836+0000","last_fullsized":"2024-01-15T18:23:42.234836+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'4","last_scrub_stamp":"2024-01-15T18:23:42.234754+0000","last_deep_scrub":"110'4","last_deep_scrub_stamp":"2024-01-15T18:23:42.234754+0000","last_clean_scrub_stamp":"2024-01-15T18:23:42.234754+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:48:58.975217+0000","scrub_duration":0.004651957,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"110'52","reported_seq":73,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.897145+0000","last_change":"2024-01-15T18:18:33.161775+0000","last_active":"2024-01-15T18:23:31.897145+0000","last_peered":"2024-01-15T18:23:31.897145+0000","last_clean":"2024-01-15T18:23:31.897145+0000","last_became_active":"2024-01-15T18:18:33.161493+0000","last_became_peered":"2024-01-15T18:18:33.161493+0000","last_unstale":"2024-01-15T18:23:31.897145+0000","last_undegraded":"2024-01-15T18:23:31.897145+0000","last_fullsized":"2024-01-15T18:23:31.897145+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:43.624818+0000","last_change":"2024-01-15T18:23:43.624838+0000","last_active":"2024-01-15T18:23:43.624818+0000","last_peered":"2024-01-15T18:23:43.624818+0000","last_clean":"2024-01-15T18:23:43.624818+0000","last_became_active":"2024-01-15T18:18:32.214942+0000","last_became_peered":"2024-01-15T18:18:32.214942+0000","last_unstale":"2024-01-15T18:23:43.624818+0000","last_undegraded":"2024-01-15T18:23:43.624818+0000","last_fullsized":"2024-01-15T18:23:43.624818+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:43.624692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:43.624692+0000","last_clean_scrub_stamp":"2024-01-15T18:23:43.624692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:36:51.484541+0000","scrub_duration":0.0025066049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"110'68","reported_seq":100,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:57.083832+0000","last_change":"2024-01-15T18:23:57.083851+0000","last_active":"2024-01-15T18:23:57.083832+0000","last_peered":"2024-01-15T18:23:57.083832+0000","last_clean":"2024-01-15T18:23:57.083832+0000","last_became_active":"2024-01-15T18:18:33.145973+0000","last_became_peered":"2024-01-15T18:18:33.145973+0000","last_unstale":"2024-01-15T18:23:57.083832+0000","last_undegraded":"2024-01-15T18:23:57.083832+0000","last_fullsized":"2024-01-15T18:23:57.083832+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'68","last_scrub_stamp":"2024-01-15T18:23:57.083768+0000","last_deep_scrub":"110'68","last_deep_scrub_stamp":"2024-01-15T18:23:57.083768+0000","last_clean_scrub_stamp":"2024-01-15T18:23:57.083768+0000","objects_scrubbed":25,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:22:28.677434+0000","scrub_duration":0.150822226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663557,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"110'12","reported_seq":37,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:38.931283+0000","last_change":"2024-01-15T18:23:38.931329+0000","last_active":"2024-01-15T18:23:38.931283+0000","last_peered":"2024-01-15T18:23:38.931283+0000","last_clean":"2024-01-15T18:23:38.931283+0000","last_became_active":"2024-01-15T18:18:32.219280+0000","last_became_peered":"2024-01-15T18:18:32.219280+0000","last_unstale":"2024-01-15T18:23:38.931283+0000","last_undegraded":"2024-01-15T18:23:38.931283+0000","last_fullsized":"2024-01-15T18:23:38.931283+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'12","last_scrub_stamp":"2024-01-15T18:23:38.931162+0000","last_deep_scrub":"110'12","last_deep_scrub_stamp":"2024-01-15T18:23:38.931162+0000","last_clean_scrub_stamp":"2024-01-15T18:23:38.931162+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:57:50.305539+0000","scrub_duration":0.0047879089999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"110'24","reported_seq":44,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.993163+0000","last_change":"2024-01-15T18:18:33.161674+0000","last_active":"2024-01-15T18:23:31.993163+0000","last_peered":"2024-01-15T18:23:31.993163+0000","last_clean":"2024-01-15T18:23:31.993163+0000","last_became_active":"2024-01-15T18:18:33.161329+0000","last_became_peered":"2024-01-15T18:18:33.161329+0000","last_unstale":"2024-01-15T18:23:31.993163+0000","last_undegraded":"2024-01-15T18:23:31.993163+0000","last_fullsized":"2024-01-15T18:23:31.993163+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"107'1","reported_seq":18,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:04.760007+0000","last_change":"2024-01-15T18:18:32.215848+0000","last_active":"2024-01-15T18:23:04.760007+0000","last_peered":"2024-01-15T18:23:04.760007+0000","last_clean":"2024-01-15T18:23:04.760007+0000","last_became_active":"2024-01-15T18:18:32.214009+0000","last_became_peered":"2024-01-15T18:18:32.214009+0000","last_unstale":"2024-01-15T18:23:04.760007+0000","last_undegraded":"2024-01-15T18:23:04.760007+0000","last_fullsized":"2024-01-15T18:23:04.760007+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:31.173930+0000","last_clean_scrub_stamp":"2024-01-15T18:18:31.173930+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"110'36","reported_seq":63,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.943479+0000","last_change":"2024-01-15T18:18:33.162974+0000","last_active":"2024-01-15T18:23:31.943479+0000","last_peered":"2024-01-15T18:23:31.943479+0000","last_clean":"2024-01-15T18:23:31.943479+0000","last_became_active":"2024-01-15T18:18:33.162679+0000","last_became_peered":"2024-01-15T18:18:33.162679+0000","last_unstale":"2024-01-15T18:23:31.943479+0000","last_undegraded":"2024-01-15T18:23:31.943479+0000","last_fullsized":"2024-01-15T18:23:31.943479+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"110'3","reported_seq":28,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:43.939236+0000","last_change":"2024-01-15T18:23:43.939260+0000","last_active":"2024-01-15T18:23:43.939236+0000","last_peered":"2024-01-15T18:23:43.939236+0000","last_clean":"2024-01-15T18:23:43.939236+0000","last_became_active":"2024-01-15T18:18:32.214695+0000","last_became_peered":"2024-01-15T18:18:32.214695+0000","last_unstale":"2024-01-15T18:23:43.939236+0000","last_undegraded":"2024-01-15T18:23:43.939236+0000","last_fullsized":"2024-01-15T18:23:43.939236+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'3","last_scrub_stamp":"2024-01-15T18:23:43.939137+0000","last_deep_scrub":"110'3","last_deep_scrub_stamp":"2024-01-15T18:23:43.939137+0000","last_clean_scrub_stamp":"2024-01-15T18:23:43.939137+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:24:14.419417+0000","scrub_duration":0.0041419650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"110'44","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.746060+0000","last_change":"2024-01-15T18:18:33.168531+0000","last_active":"2024-01-15T18:23:31.746060+0000","last_peered":"2024-01-15T18:23:31.746060+0000","last_clean":"2024-01-15T18:23:31.746060+0000","last_became_active":"2024-01-15T18:18:33.168190+0000","last_became_peered":"2024-01-15T18:18:33.168190+0000","last_unstale":"2024-01-15T18:23:31.746060+0000","last_undegraded":"2024-01-15T18:23:31.746060+0000","last_fullsized":"2024-01-15T18:23:31.746060+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":11,"num_read_kb":28676,"num_write":22,"num_write_kb":73730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"110'2","reported_seq":27,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:47.483359+0000","last_change":"2024-01-15T18:23:47.483404+0000","last_active":"2024-01-15T18:23:47.483359+0000","last_peered":"2024-01-15T18:23:47.483359+0000","last_clean":"2024-01-15T18:23:47.483359+0000","last_became_active":"2024-01-15T18:18:32.220884+0000","last_became_peered":"2024-01-15T18:18:32.220884+0000","last_unstale":"2024-01-15T18:23:47.483359+0000","last_undegraded":"2024-01-15T18:23:47.483359+0000","last_fullsized":"2024-01-15T18:23:47.483359+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2","last_scrub_stamp":"2024-01-15T18:23:47.483235+0000","last_deep_scrub":"110'2","last_deep_scrub_stamp":"2024-01-15T18:23:47.483235+0000","last_clean_scrub_stamp":"2024-01-15T18:23:47.483235+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:48:48.450236+0000","scrub_duration":0.0045499140000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"110'41","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.898859+0000","last_change":"2024-01-15T18:18:33.163114+0000","last_active":"2024-01-15T18:23:31.898859+0000","last_peered":"2024-01-15T18:23:31.898859+0000","last_clean":"2024-01-15T18:23:31.898859+0000","last_became_active":"2024-01-15T18:18:33.162424+0000","last_became_peered":"2024-01-15T18:18:33.162424+0000","last_unstale":"2024-01-15T18:23:31.898859+0000","last_undegraded":"2024-01-15T18:23:31.898859+0000","last_fullsized":"2024-01-15T18:23:31.898859+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:44.345346+0000","last_change":"2024-01-15T18:23:44.345376+0000","last_active":"2024-01-15T18:23:44.345346+0000","last_peered":"2024-01-15T18:23:44.345346+0000","last_clean":"2024-01-15T18:23:44.345346+0000","last_became_active":"2024-01-15T18:18:32.216769+0000","last_became_peered":"2024-01-15T18:18:32.216769+0000","last_unstale":"2024-01-15T18:23:44.345346+0000","last_undegraded":"2024-01-15T18:23:44.345346+0000","last_fullsized":"2024-01-15T18:23:44.345346+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:44.345256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:44.345256+0000","last_clean_scrub_stamp":"2024-01-15T18:23:44.345256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:35:12.856197+0000","scrub_duration":0.0019326370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"110'45","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.055574+0000","last_change":"2024-01-15T18:18:33.163017+0000","last_active":"2024-01-15T18:23:32.055574+0000","last_peered":"2024-01-15T18:23:32.055574+0000","last_clean":"2024-01-15T18:23:32.055574+0000","last_became_active":"2024-01-15T18:18:33.162658+0000","last_became_peered":"2024-01-15T18:18:33.162658+0000","last_unstale":"2024-01-15T18:23:32.055574+0000","last_undegraded":"2024-01-15T18:23:32.055574+0000","last_fullsized":"2024-01-15T18:23:32.055574+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"110'4","reported_seq":29,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:48.233481+0000","last_change":"2024-01-15T18:23:48.233504+0000","last_active":"2024-01-15T18:23:48.233481+0000","last_peered":"2024-01-15T18:23:48.233481+0000","last_clean":"2024-01-15T18:23:48.233481+0000","last_became_active":"2024-01-15T18:18:32.207507+0000","last_became_peered":"2024-01-15T18:18:32.207507+0000","last_unstale":"2024-01-15T18:23:48.233481+0000","last_undegraded":"2024-01-15T18:23:48.233481+0000","last_fullsized":"2024-01-15T18:23:48.233481+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'4","last_scrub_stamp":"2024-01-15T18:23:48.233388+0000","last_deep_scrub":"110'4","last_deep_scrub_stamp":"2024-01-15T18:23:48.233388+0000","last_clean_scrub_stamp":"2024-01-15T18:23:48.233388+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:50.599913+0000","scrub_duration":0.0034152789999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"110'43","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.490371+0000","last_change":"2024-01-15T18:18:33.164981+0000","last_active":"2024-01-15T18:23:31.490371+0000","last_peered":"2024-01-15T18:23:31.490371+0000","last_clean":"2024-01-15T18:23:31.490371+0000","last_became_active":"2024-01-15T18:18:33.164206+0000","last_became_peered":"2024-01-15T18:18:33.164206+0000","last_unstale":"2024-01-15T18:23:31.490371+0000","last_undegraded":"2024-01-15T18:23:31.490371+0000","last_fullsized":"2024-01-15T18:23:31.490371+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:48.035163+0000","last_change":"2024-01-15T18:23:48.035184+0000","last_active":"2024-01-15T18:23:48.035163+0000","last_peered":"2024-01-15T18:23:48.035163+0000","last_clean":"2024-01-15T18:23:48.035163+0000","last_became_active":"2024-01-15T18:18:32.212201+0000","last_became_peered":"2024-01-15T18:18:32.212201+0000","last_unstale":"2024-01-15T18:23:48.035163+0000","last_undegraded":"2024-01-15T18:23:48.035163+0000","last_fullsized":"2024-01-15T18:23:48.035163+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:48.035058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:48.035058+0000","last_clean_scrub_stamp":"2024-01-15T18:23:48.035058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:49:40.411498+0000","scrub_duration":0.0017447160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"110'44","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.953968+0000","last_change":"2024-01-15T18:18:33.165046+0000","last_active":"2024-01-15T18:23:31.953968+0000","last_peered":"2024-01-15T18:23:31.953968+0000","last_clean":"2024-01-15T18:23:31.953968+0000","last_became_active":"2024-01-15T18:18:33.164708+0000","last_became_peered":"2024-01-15T18:18:33.164708+0000","last_unstale":"2024-01-15T18:23:31.953968+0000","last_undegraded":"2024-01-15T18:23:31.953968+0000","last_fullsized":"2024-01-15T18:23:31.953968+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:44.292443+0000","last_change":"2024-01-15T18:23:44.292477+0000","last_active":"2024-01-15T18:23:44.292443+0000","last_peered":"2024-01-15T18:23:44.292443+0000","last_clean":"2024-01-15T18:23:44.292443+0000","last_became_active":"2024-01-15T18:18:32.207599+0000","last_became_peered":"2024-01-15T18:18:32.207599+0000","last_unstale":"2024-01-15T18:23:44.292443+0000","last_undegraded":"2024-01-15T18:23:44.292443+0000","last_fullsized":"2024-01-15T18:23:44.292443+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:44.292288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:44.292288+0000","last_clean_scrub_stamp":"2024-01-15T18:23:44.292288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:52:22.761638+0000","scrub_duration":0.0021733669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"110'42","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.029364+0000","last_change":"2024-01-15T18:18:33.160618+0000","last_active":"2024-01-15T18:23:32.029364+0000","last_peered":"2024-01-15T18:23:32.029364+0000","last_clean":"2024-01-15T18:23:32.029364+0000","last_became_active":"2024-01-15T18:18:33.160350+0000","last_became_peered":"2024-01-15T18:18:33.160350+0000","last_unstale":"2024-01-15T18:23:32.029364+0000","last_undegraded":"2024-01-15T18:23:32.029364+0000","last_fullsized":"2024-01-15T18:23:32.029364+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67109125,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":17,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:47.491435+0000","last_change":"2024-01-15T18:23:47.491458+0000","last_active":"2024-01-15T18:23:47.491435+0000","last_peered":"2024-01-15T18:23:47.491435+0000","last_clean":"2024-01-15T18:23:47.491435+0000","last_became_active":"2024-01-15T18:18:32.216581+0000","last_became_peered":"2024-01-15T18:18:32.216581+0000","last_unstale":"2024-01-15T18:23:47.491435+0000","last_undegraded":"2024-01-15T18:23:47.491435+0000","last_fullsized":"2024-01-15T18:23:47.491435+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:47.491336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:47.491336+0000","last_clean_scrub_stamp":"2024-01-15T18:23:47.491336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:06:23.583464+0000","scrub_duration":0.0018469020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"110'46","reported_seq":74,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:48.560733+0000","last_change":"2024-01-15T18:23:48.560769+0000","last_active":"2024-01-15T18:23:48.560733+0000","last_peered":"2024-01-15T18:23:48.560733+0000","last_clean":"2024-01-15T18:23:48.560733+0000","last_became_active":"2024-01-15T18:18:33.164152+0000","last_became_peered":"2024-01-15T18:18:33.164152+0000","last_unstale":"2024-01-15T18:23:48.560733+0000","last_undegraded":"2024-01-15T18:23:48.560733+0000","last_fullsized":"2024-01-15T18:23:48.560733+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'46","last_scrub_stamp":"2024-01-15T18:23:48.560650+0000","last_deep_scrub":"110'46","last_deep_scrub_stamp":"2024-01-15T18:23:48.560650+0000","last_clean_scrub_stamp":"2024-01-15T18:23:48.560650+0000","objects_scrubbed":13,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:42:57.049428+0000","scrub_duration":0.082496549000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"110'3","reported_seq":28,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:44.865576+0000","last_change":"2024-01-15T18:23:44.865597+0000","last_active":"2024-01-15T18:23:44.865576+0000","last_peered":"2024-01-15T18:23:44.865576+0000","last_clean":"2024-01-15T18:23:44.865576+0000","last_became_active":"2024-01-15T18:18:32.218858+0000","last_became_peered":"2024-01-15T18:18:32.218858+0000","last_unstale":"2024-01-15T18:23:44.865576+0000","last_undegraded":"2024-01-15T18:23:44.865576+0000","last_fullsized":"2024-01-15T18:23:44.865576+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'3","last_scrub_stamp":"2024-01-15T18:23:44.865505+0000","last_deep_scrub":"110'3","last_deep_scrub_stamp":"2024-01-15T18:23:44.865505+0000","last_clean_scrub_stamp":"2024-01-15T18:23:44.865505+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:03:03.103591+0000","scrub_duration":0.0027720739999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"110'41","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.577271+0000","last_change":"2024-01-15T18:18:33.160602+0000","last_active":"2024-01-15T18:23:31.577271+0000","last_peered":"2024-01-15T18:23:31.577271+0000","last_clean":"2024-01-15T18:23:31.577271+0000","last_became_active":"2024-01-15T18:18:33.160315+0000","last_became_peered":"2024-01-15T18:18:33.160315+0000","last_unstale":"2024-01-15T18:23:31.577271+0000","last_undegraded":"2024-01-15T18:23:31.577271+0000","last_fullsized":"2024-01-15T18:23:31.577271+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"110'18","reported_seq":43,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:45.538237+0000","last_change":"2024-01-15T18:23:45.538258+0000","last_active":"2024-01-15T18:23:45.538237+0000","last_peered":"2024-01-15T18:23:45.538237+0000","last_clean":"2024-01-15T18:23:45.538237+0000","last_became_active":"2024-01-15T18:18:32.216612+0000","last_became_peered":"2024-01-15T18:18:32.216612+0000","last_unstale":"2024-01-15T18:23:45.538237+0000","last_undegraded":"2024-01-15T18:23:45.538237+0000","last_fullsized":"2024-01-15T18:23:45.538237+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'18","last_scrub_stamp":"2024-01-15T18:23:45.538139+0000","last_deep_scrub":"110'18","last_deep_scrub_stamp":"2024-01-15T18:23:45.538139+0000","last_clean_scrub_stamp":"2024-01-15T18:23:45.538139+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:40:14.696757+0000","scrub_duration":0.0021555250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":506,"num_omap_keys":10,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"110'66","reported_seq":104,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:44.623824+0000","last_change":"2024-01-15T18:23:44.623866+0000","last_active":"2024-01-15T18:23:44.623824+0000","last_peered":"2024-01-15T18:23:44.623824+0000","last_clean":"2024-01-15T18:23:44.623824+0000","last_became_active":"2024-01-15T18:18:33.163994+0000","last_became_peered":"2024-01-15T18:18:33.163994+0000","last_unstale":"2024-01-15T18:23:44.623824+0000","last_undegraded":"2024-01-15T18:23:44.623824+0000","last_fullsized":"2024-01-15T18:23:44.623824+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'66","last_scrub_stamp":"2024-01-15T18:23:44.623747+0000","last_deep_scrub":"110'66","last_deep_scrub_stamp":"2024-01-15T18:23:44.623747+0000","last_clean_scrub_stamp":"2024-01-15T18:23:44.623747+0000","objects_scrubbed":24,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:47:40.548332+0000","scrub_duration":0.162659633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:54.492408+0000","last_change":"2024-01-15T18:23:54.492428+0000","last_active":"2024-01-15T18:23:54.492408+0000","last_peered":"2024-01-15T18:23:54.492408+0000","last_clean":"2024-01-15T18:23:54.492408+0000","last_became_active":"2024-01-15T18:18:32.216023+0000","last_became_peered":"2024-01-15T18:18:32.216023+0000","last_unstale":"2024-01-15T18:23:54.492408+0000","last_undegraded":"2024-01-15T18:23:54.492408+0000","last_fullsized":"2024-01-15T18:23:54.492408+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:54.492304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:54.492304+0000","last_clean_scrub_stamp":"2024-01-15T18:23:54.492304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:21:06.895168+0000","scrub_duration":0.0023113349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"110'39","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.233008+0000","last_change":"2024-01-15T18:18:42.262610+0000","last_active":"2024-01-15T18:23:31.233008+0000","last_peered":"2024-01-15T18:23:31.233008+0000","last_clean":"2024-01-15T18:23:31.233008+0000","last_became_active":"2024-01-15T18:18:42.262255+0000","last_became_peered":"2024-01-15T18:18:42.262255+0000","last_unstale":"2024-01-15T18:23:31.233008+0000","last_undegraded":"2024-01-15T18:23:31.233008+0000","last_fullsized":"2024-01-15T18:23:31.233008+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303290,"num_objects":19,"num_object_clones":1,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:54.899638+0000","last_change":"2024-01-15T18:23:54.899657+0000","last_active":"2024-01-15T18:23:54.899638+0000","last_peered":"2024-01-15T18:23:54.899638+0000","last_clean":"2024-01-15T18:23:54.899638+0000","last_became_active":"2024-01-15T18:18:32.213930+0000","last_became_peered":"2024-01-15T18:18:32.213930+0000","last_unstale":"2024-01-15T18:23:54.899638+0000","last_undegraded":"2024-01-15T18:23:54.899638+0000","last_fullsized":"2024-01-15T18:23:54.899638+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:54.899542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:54.899542+0000","last_clean_scrub_stamp":"2024-01-15T18:23:54.899542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:23:19.753410+0000","scrub_duration":0.0018061939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"110'60","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:52.993684+0000","last_change":"2024-01-15T18:23:52.993718+0000","last_active":"2024-01-15T18:23:52.993684+0000","last_peered":"2024-01-15T18:23:52.993684+0000","last_clean":"2024-01-15T18:23:52.993684+0000","last_became_active":"2024-01-15T18:18:33.150660+0000","last_became_peered":"2024-01-15T18:18:33.150660+0000","last_unstale":"2024-01-15T18:23:52.993684+0000","last_undegraded":"2024-01-15T18:23:52.993684+0000","last_fullsized":"2024-01-15T18:23:52.993684+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'60","last_scrub_stamp":"2024-01-15T18:23:52.993598+0000","last_deep_scrub":"110'60","last_deep_scrub_stamp":"2024-01-15T18:23:52.993598+0000","last_clean_scrub_stamp":"2024-01-15T18:23:52.993598+0000","objects_scrubbed":16,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:28:50.475455+0000","scrub_duration":0.111747205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:53.298745+0000","last_change":"2024-01-15T18:23:53.298763+0000","last_active":"2024-01-15T18:23:53.298745+0000","last_peered":"2024-01-15T18:23:53.298745+0000","last_clean":"2024-01-15T18:23:53.298745+0000","last_became_active":"2024-01-15T18:18:32.214394+0000","last_became_peered":"2024-01-15T18:18:32.214394+0000","last_unstale":"2024-01-15T18:23:53.298745+0000","last_undegraded":"2024-01-15T18:23:53.298745+0000","last_fullsized":"2024-01-15T18:23:53.298745+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:53.298655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:53.298655+0000","last_clean_scrub_stamp":"2024-01-15T18:23:53.298655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:46:41.201522+0000","scrub_duration":0.00139146,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"110'56","reported_seq":80,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.020041+0000","last_change":"2024-01-15T18:18:33.156094+0000","last_active":"2024-01-15T18:23:32.020041+0000","last_peered":"2024-01-15T18:23:32.020041+0000","last_clean":"2024-01-15T18:23:32.020041+0000","last_became_active":"2024-01-15T18:18:33.155797+0000","last_became_peered":"2024-01-15T18:18:33.155797+0000","last_unstale":"2024-01-15T18:23:32.020041+0000","last_undegraded":"2024-01-15T18:23:32.020041+0000","last_fullsized":"2024-01-15T18:23:32.020041+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:47.949655+0000","last_change":"2024-01-15T18:23:47.949688+0000","last_active":"2024-01-15T18:23:47.949655+0000","last_peered":"2024-01-15T18:23:47.949655+0000","last_clean":"2024-01-15T18:23:47.949655+0000","last_became_active":"2024-01-15T18:18:32.212487+0000","last_became_peered":"2024-01-15T18:18:32.212487+0000","last_unstale":"2024-01-15T18:23:47.949655+0000","last_undegraded":"2024-01-15T18:23:47.949655+0000","last_fullsized":"2024-01-15T18:23:47.949655+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:47.949523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:47.949523+0000","last_clean_scrub_stamp":"2024-01-15T18:23:47.949523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:31:41.981969+0000","scrub_duration":0.0029090320000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"110'46","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.056164+0000","last_change":"2024-01-15T18:18:33.167732+0000","last_active":"2024-01-15T18:23:32.056164+0000","last_peered":"2024-01-15T18:23:32.056164+0000","last_clean":"2024-01-15T18:23:32.056164+0000","last_became_active":"2024-01-15T18:18:33.167436+0000","last_became_peered":"2024-01-15T18:18:33.167436+0000","last_unstale":"2024-01-15T18:23:32.056164+0000","last_undegraded":"2024-01-15T18:23:32.056164+0000","last_fullsized":"2024-01-15T18:23:32.056164+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"110'12","reported_seq":47,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:40.933714+0000","last_change":"2024-01-15T18:23:40.933751+0000","last_active":"2024-01-15T18:23:40.933714+0000","last_peered":"2024-01-15T18:23:40.933714+0000","last_clean":"2024-01-15T18:23:40.933714+0000","last_became_active":"2024-01-15T18:18:32.211744+0000","last_became_peered":"2024-01-15T18:18:32.211744+0000","last_unstale":"2024-01-15T18:23:40.933714+0000","last_undegraded":"2024-01-15T18:23:40.933714+0000","last_fullsized":"2024-01-15T18:23:40.933714+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'12","last_scrub_stamp":"2024-01-15T18:23:40.933627+0000","last_deep_scrub":"110'12","last_deep_scrub_stamp":"2024-01-15T18:23:40.933627+0000","last_clean_scrub_stamp":"2024-01-15T18:23:40.933627+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:56:38.633122+0000","scrub_duration":0.0031438920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1321,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"110'59","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:38.786673+0000","last_change":"2024-01-15T18:23:38.786716+0000","last_active":"2024-01-15T18:23:38.786673+0000","last_peered":"2024-01-15T18:23:38.786673+0000","last_clean":"2024-01-15T18:23:38.786673+0000","last_became_active":"2024-01-15T18:18:33.162521+0000","last_became_peered":"2024-01-15T18:18:33.162521+0000","last_unstale":"2024-01-15T18:23:38.786673+0000","last_undegraded":"2024-01-15T18:23:38.786673+0000","last_fullsized":"2024-01-15T18:23:38.786673+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'59","last_scrub_stamp":"2024-01-15T18:23:38.786600+0000","last_deep_scrub":"110'59","last_deep_scrub_stamp":"2024-01-15T18:23:38.786600+0000","last_clean_scrub_stamp":"2024-01-15T18:23:38.786600+0000","objects_scrubbed":23,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:26:13.413987+0000","scrub_duration":0.160345389,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:45.474238+0000","last_change":"2024-01-15T18:23:45.474259+0000","last_active":"2024-01-15T18:23:45.474238+0000","last_peered":"2024-01-15T18:23:45.474238+0000","last_clean":"2024-01-15T18:23:45.474238+0000","last_became_active":"2024-01-15T18:18:32.216966+0000","last_became_peered":"2024-01-15T18:18:32.216966+0000","last_unstale":"2024-01-15T18:23:45.474238+0000","last_undegraded":"2024-01-15T18:23:45.474238+0000","last_fullsized":"2024-01-15T18:23:45.474238+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:45.474139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:45.474139+0000","last_clean_scrub_stamp":"2024-01-15T18:23:45.474139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:43:14.897532+0000","scrub_duration":0.0025295999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"110'44","reported_seq":76,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:59.012816+0000","last_change":"2024-01-15T18:23:59.012849+0000","last_active":"2024-01-15T18:23:59.012816+0000","last_peered":"2024-01-15T18:23:59.012816+0000","last_clean":"2024-01-15T18:23:59.012816+0000","last_became_active":"2024-01-15T18:18:33.165323+0000","last_became_peered":"2024-01-15T18:18:33.165323+0000","last_unstale":"2024-01-15T18:23:59.012816+0000","last_undegraded":"2024-01-15T18:23:59.012816+0000","last_fullsized":"2024-01-15T18:23:59.012816+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'44","last_scrub_stamp":"2024-01-15T18:23:59.012731+0000","last_deep_scrub":"110'44","last_deep_scrub_stamp":"2024-01-15T18:23:59.012731+0000","last_clean_scrub_stamp":"2024-01-15T18:23:59.012731+0000","objects_scrubbed":14,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:30:06.603978+0000","scrub_duration":0.088778124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:42.312415+0000","last_change":"2024-01-15T18:23:42.312436+0000","last_active":"2024-01-15T18:23:42.312415+0000","last_peered":"2024-01-15T18:23:42.312415+0000","last_clean":"2024-01-15T18:23:42.312415+0000","last_became_active":"2024-01-15T18:18:32.215973+0000","last_became_peered":"2024-01-15T18:18:32.215973+0000","last_unstale":"2024-01-15T18:23:42.312415+0000","last_undegraded":"2024-01-15T18:23:42.312415+0000","last_fullsized":"2024-01-15T18:23:42.312415+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:42.312313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:42.312313+0000","last_clean_scrub_stamp":"2024-01-15T18:23:42.312313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:17:28.948911+0000","scrub_duration":0.0020901840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"110'119","reported_seq":255,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:44.359592+0000","last_change":"2024-01-15T18:23:44.359635+0000","last_active":"2024-01-15T18:23:44.359592+0000","last_peered":"2024-01-15T18:23:44.359592+0000","last_clean":"2024-01-15T18:23:44.359592+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:23:44.359592+0000","last_undegraded":"2024-01-15T18:23:44.359592+0000","last_fullsized":"2024-01-15T18:23:44.359592+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'119","last_scrub_stamp":"2024-01-15T18:23:44.359514+0000","last_deep_scrub":"110'119","last_deep_scrub_stamp":"2024-01-15T18:23:44.359514+0000","last_clean_scrub_stamp":"2024-01-15T18:23:44.359514+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:41:00.611426+0000","scrub_duration":0.18253596899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":5368710136,"num_objects":1286,"num_object_clones":1,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":1,"num_read":525,"num_read_kb":2097165,"num_write":1306,"num_write_kb":5242895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10737459200,"data_stored":10737420272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3057,"ondisk_log_size":3057,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":616194,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":280,"num_write_kb":720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5506,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1294336,"data_stored":1232388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":278,"ondisk_log_size":278,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509674,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1834500,"kb_used_data":1767292,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92537340,"statfs":{"total":96636764160,"available":94758236160,"internally_reserved":0,"allocated":1809707008,"data_stored":1808930471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.72799999999999998,"15min":0.90300000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92800000000000005,"5min":2.8620000000000001,"15min":10.284000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.70699999999999996,"15min":0.93700000000000006},"min":{"1min":0.54200000000000004,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":16.968},"last":0.74099999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.72399999999999998,"15min":0.93899999999999995},"min":{"1min":0.47599999999999998,"5min":0.38600000000000001,"15min":0.33800000000000002},"max":{"1min":1.028,"5min":1.028,"15min":15.791},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.73499999999999999,"15min":0.93100000000000005},"min":{"1min":0.52100000000000002,"5min":0.44,"15min":0.371},"max":{"1min":0.94799999999999995,"5min":1.248,"15min":10.842000000000001},"last":0.71899999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.72899999999999998,"15min":0.94499999999999995},"min":{"1min":0.49099999999999999,"5min":0.39500000000000002,"15min":0.378},"max":{"1min":0.96499999999999997,"5min":0.98799999999999999,"15min":22.318000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.752,"15min":0.91500000000000004},"min":{"1min":0.46899999999999997,"5min":0.42699999999999999,"15min":0.39100000000000001},"max":{"1min":0.91400000000000003,"5min":2.2799999999999998,"15min":9.7539999999999996},"last":0.65800000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.72699999999999998,"15min":0.996},"min":{"1min":0.56999999999999995,"5min":0.46500000000000002,"15min":0.38100000000000001},"max":{"1min":0.875,"5min":1.2110000000000001,"15min":18.152999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.78700000000000003,"15min":0.99199999999999999},"min":{"1min":0.51600000000000001,"5min":0.495,"15min":0.433},"max":{"1min":0.98099999999999998,"5min":2.2389999999999999,"15min":10.34},"last":0.754}]},{"osd":4,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.57599999999999996,"15min":0.93000000000000005},"min":{"1min":0.40600000000000003,"5min":0.34799999999999998,"15min":0.314},"max":{"1min":0.82799999999999996,"5min":0.877,"15min":20.387},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.94099999999999995},"min":{"1min":0.438,"5min":0.39400000000000002,"15min":0.34200000000000003},"max":{"1min":0.84199999999999997,"5min":0.98099999999999998,"15min":17.215},"last":0.54400000000000004}]},{"osd":5,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.622,"15min":0.91500000000000004},"min":{"1min":0.42399999999999999,"5min":0.40999999999999998,"15min":0.34699999999999998},"max":{"1min":0.91500000000000004,"5min":0.96999999999999997,"15min":19.959},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.64200000000000002,"15min":0.95999999999999996},"min":{"1min":0.40999999999999998,"5min":0.40899999999999997,"15min":0.35799999999999998},"max":{"1min":0.92300000000000004,"5min":0.92600000000000005,"15min":19.905000000000001},"last":0.68899999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.748,"15min":1.1439999999999999},"min":{"1min":0.49299999999999999,"5min":0.44,"15min":0.40799999999999997},"max":{"1min":0.995,"5min":0.995,"15min":22.478999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.69599999999999995,"15min":1.0269999999999999},"min":{"1min":0.48599999999999999,"5min":0.44600000000000001,"15min":0.40799999999999997},"max":{"1min":0.92200000000000004,"5min":0.96299999999999997,"15min":21.509},"last":0.58999999999999997}]}]},{"osd":6,"up_from":15,"seq":64424509673,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1256320,"kb_used_data":1193272,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":93115520,"statfs":{"total":96636764160,"available":95350292480,"internally_reserved":0,"allocated":1221910528,"data_stored":1221137119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":0.78400000000000003,"15min":0.99399999999999999},"min":{"1min":0.59099999999999997,"5min":0.36299999999999999,"15min":0.29099999999999998},"max":{"1min":6.1900000000000004,"5min":6.1900000000000004,"15min":32.587000000000003},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.79100000000000004,"15min":0.90100000000000002},"min":{"1min":0.63900000000000001,"5min":0.434,"15min":0.35399999999999998},"max":{"1min":4.9649999999999999,"5min":4.9649999999999999,"15min":16.193000000000001},"last":0.51200000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.73799999999999999,"15min":0.996},"min":{"1min":0.56599999999999995,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":53.451999999999998},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.73399999999999999,"15min":0.91600000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.88400000000000001,"5min":1.0189999999999999,"15min":32.512},"last":0.71399999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77000000000000002,"15min":0.95099999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.41699999999999998},"max":{"1min":0.98099999999999998,"5min":1.016,"15min":32.472999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.996,"5min":0.84299999999999997,"15min":0.92100000000000004},"min":{"1min":0.57899999999999996,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":5.0410000000000004,"5min":5.0410000000000004,"15min":16.602},"last":0.69399999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.82699999999999996,"15min":0.93600000000000005},"min":{"1min":0.64900000000000002,"5min":0.52500000000000002,"15min":0.46000000000000002},"max":{"1min":5.0940000000000003,"5min":5.0940000000000003,"15min":16.388000000000002},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.76000000000000001,"15min":1.0609999999999999},"min":{"1min":0.67800000000000005,"5min":0.505,"15min":0.432},"max":{"1min":1.004,"5min":1.04,"15min":53.398000000000003},"last":0.72699999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.67000000000000004,"15min":0.88800000000000001},"min":{"1min":0.41499999999999998,"5min":0.39000000000000001,"15min":0.378},"max":{"1min":6.3869999999999996,"5min":6.3869999999999996,"15min":32.549999999999997},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.68799999999999994,"15min":0.83599999999999997},"min":{"1min":0.45800000000000002,"5min":0.41399999999999998,"15min":0.371},"max":{"1min":5.1719999999999997,"5min":5.1719999999999997,"15min":11.852},"last":0.65000000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.70499999999999996,"15min":1.052},"min":{"1min":0.433,"5min":0.42199999999999999,"15min":0.34300000000000003},"max":{"1min":6.8769999999999998,"5min":6.8769999999999998,"15min":53.229999999999997},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.747,"15min":1.0209999999999999},"min":{"1min":0.49199999999999999,"5min":0.44400000000000001,"15min":0.41799999999999998},"max":{"1min":6.1360000000000001,"5min":6.1360000000000001,"15min":32.404000000000003},"last":0.79800000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:23:08 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.749,"15min":1.0820000000000001},"min":{"1min":0.52200000000000002,"5min":0.46000000000000002,"15min":0.38600000000000001},"max":{"1min":5.1319999999999997,"5min":5.1319999999999997,"15min":45.070999999999998},"last":0.753},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.71099999999999997,"15min":1.181},"min":{"1min":0.51100000000000001,"5min":0.47999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89100000000000001,"5min":0.999,"15min":53.338000000000001},"last":0.77100000000000002}]}]},{"osd":4,"up_from":15,"seq":64424509674,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1706876,"kb_used_data":1643828,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":92664964,"statfs":{"total":96636764160,"available":94888923136,"internally_reserved":0,"allocated":1683279872,"data_stored":1682510628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.69799999999999995,"15min":0.90600000000000003},"min":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.32900000000000001},"max":{"1min":0.96599999999999997,"5min":1.542,"15min":23.318000000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.70299999999999996,"15min":0.81299999999999994},"min":{"1min":0.55000000000000004,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":14.723000000000001},"last":0.73899999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.72599999999999998,"15min":0.81399999999999995},"min":{"1min":0.46300000000000002,"5min":0.39100000000000001,"15min":0.38800000000000001},"max":{"1min":0.96199999999999997,"5min":1.919,"15min":9.5700000000000003},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.74199999999999999,"15min":0.94699999999999995},"min":{"1min":0.52600000000000002,"5min":0.44900000000000001,"15min":0.36899999999999999},"max":{"1min":0.996,"5min":1.429,"15min":22.408999999999999},"last":0.81299999999999994}]},{"osd":2,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.79000000000000004,"15min":0.85299999999999998},"min":{"1min":0.50900000000000001,"5min":0.41899999999999998,"15min":0.39700000000000002},"max":{"1min":1.032,"5min":5.6260000000000003,"15min":9.7089999999999996},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.76100000000000001,"15min":1.018},"min":{"1min":0.59299999999999997,"5min":0.45000000000000001,"15min":0.39100000000000001},"max":{"1min":1.016,"5min":1.387,"15min":22.338999999999999},"last":0.91600000000000004}]},{"osd":3,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.73999999999999999,"15min":0.88700000000000001},"min":{"1min":0.52300000000000002,"5min":0.46200000000000002,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":1.577,"15min":14.763999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.82599999999999996,"15min":0.93600000000000005},"min":{"1min":0.65300000000000002,"5min":0.53600000000000003,"15min":0.48099999999999998},"max":{"1min":0.996,"5min":1.9610000000000001,"15min":9.6289999999999996},"last":0.93300000000000005}]},{"osd":5,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63300000000000001,"15min":0.94299999999999995},"min":{"1min":0.44,"5min":0.36499999999999999,"15min":0.34799999999999998},"max":{"1min":1.0149999999999999,"5min":1.1970000000000001,"15min":31.617999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.85799999999999998},"min":{"1min":0.40799999999999997,"5min":0.373,"15min":0.36599999999999999},"max":{"1min":0.86699999999999999,"5min":1.4910000000000001,"15min":17.145},"last":0.60099999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.72799999999999998,"15min":0.83899999999999997},"min":{"1min":0.60799999999999998,"5min":0.41799999999999998,"15min":0.37},"max":{"1min":0.89300000000000002,"5min":5.9489999999999998,"15min":11.124000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.71099999999999997,"15min":1.2090000000000001},"min":{"1min":0.52300000000000002,"5min":0.42499999999999999,"15min":0.40200000000000002},"max":{"1min":1.0649999999999999,"5min":2.3149999999999999,"15min":32.308},"last":0.65700000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.72699999999999998,"15min":1.0009999999999999},"min":{"1min":0.48099999999999998,"5min":0.45100000000000001,"15min":0.34799999999999998},"max":{"1min":1,"5min":2.4329999999999998,"15min":39.256},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78200000000000003,"15min":1.002},"min":{"1min":0.60699999999999998,"5min":0.499,"15min":0.46500000000000002},"max":{"1min":1.04,"5min":2.1909999999999998,"15min":26.573},"last":0.63200000000000001}]}]},{"osd":3,"up_from":15,"seq":64424509673,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1453604,"kb_used_data":1386396,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92918236,"statfs":{"total":96636764160,"available":95148273664,"internally_reserved":0,"allocated":1419669504,"data_stored":1418883882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.45100000000000001,"15min":0.498},"min":{"1min":0.28999999999999998,"5min":0.28199999999999997,"15min":0.245},"max":{"1min":0.69099999999999995,"5min":0.69699999999999995,"15min":5.8849999999999998},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.51000000000000001,"15min":0.621},"min":{"1min":0.41799999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.80000000000000004,"5min":1.4750000000000001,"15min":17.756},"last":0.34599999999999997}]},{"osd":1,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51300000000000001,"15min":0.60499999999999998},"min":{"1min":0.33500000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60399999999999998,"5min":4.0099999999999998,"15min":12.307},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55200000000000005,"15min":0.59199999999999997},"min":{"1min":0.36199999999999999,"5min":0.34300000000000003,"15min":0.31900000000000001},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":6.0640000000000001},"last":0.437}]},{"osd":2,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61399999999999999,"15min":0.71199999999999997},"min":{"1min":0.42799999999999999,"5min":0.35299999999999998,"15min":0.32600000000000001},"max":{"1min":0.82999999999999996,"5min":4.077,"15min":12.468999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.59099999999999997,"15min":0.68000000000000005},"min":{"1min":0.44,"5min":0.39100000000000001,"15min":0.35399999999999998},"max":{"1min":0.78000000000000003,"5min":0.81899999999999995,"15min":10.837999999999999},"last":0.39400000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78500000000000003,"15min":0.999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.375},"max":{"1min":1.1910000000000001,"5min":1.9379999999999999,"15min":17.853999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.85999999999999999,"15min":0.98199999999999998},"min":{"1min":0.54900000000000004,"5min":0.49199999999999999,"15min":0.45200000000000001},"max":{"1min":1.0569999999999999,"5min":4.2430000000000003,"15min":13.31},"last":0.63600000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.78400000000000003,"15min":0.98899999999999999},"min":{"1min":0.496,"5min":0.439,"15min":0.439},"max":{"1min":1.1619999999999999,"5min":1.179,"15min":17.687999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.77800000000000002,"15min":0.99099999999999999},"min":{"1min":0.58599999999999997,"5min":0.53800000000000003,"15min":0.438},"max":{"1min":1.048,"5min":1.825,"15min":20.515999999999998},"last":0.56200000000000006}]},{"osd":6,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.80900000000000005,"15min":0.98499999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.432},"max":{"1min":1.1419999999999999,"5min":4.2800000000000002,"15min":23.100999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.80600000000000005,"15min":0.94799999999999995},"min":{"1min":0.69899999999999995,"5min":0.52200000000000002,"15min":0.5},"max":{"1min":1.0720000000000001,"5min":1.149,"15min":12.76},"last":0.65900000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82499999999999996,"15min":1.1850000000000001},"min":{"1min":0.625,"5min":0.56899999999999995,"15min":0.46800000000000003},"max":{"1min":1.0860000000000001,"5min":1.931,"15min":24.321999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.82899999999999996,"15min":1.0820000000000001},"min":{"1min":0.60899999999999999,"5min":0.497,"15min":0.497},"max":{"1min":1.2070000000000001,"5min":4.202,"15min":23.013000000000002},"last":0.81699999999999995}]}]},{"osd":2,"up_from":15,"seq":64424509673,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":977856,"kb_used_data":910648,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93393984,"statfs":{"total":96636764160,"available":95635439616,"internally_reserved":0,"allocated":932503552,"data_stored":931731464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45500000000000002,"15min":0.50900000000000001},"min":{"1min":0.32300000000000001,"5min":0.31,"15min":0.247},"max":{"1min":0.55200000000000005,"5min":0.80800000000000005,"15min":5.1849999999999996},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.503,"5min":0.501,"15min":0.54600000000000004},"min":{"1min":0.36499999999999999,"5min":0.26700000000000002,"15min":0.252},"max":{"1min":0.73999999999999999,"5min":0.83899999999999997,"15min":7.516},"last":0.63200000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.56799999999999995,"15min":0.60999999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.29699999999999999},"max":{"1min":0.71899999999999997,"5min":3.044,"15min":10.907},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.48999999999999999,"15min":0.59599999999999997},"min":{"1min":0.34999999999999998,"5min":0.33100000000000002,"15min":0.28100000000000003},"max":{"1min":0.76400000000000001,"5min":0.76500000000000001,"15min":8.9190000000000005},"last":0.52900000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52500000000000002,"15min":0.60599999999999998},"min":{"1min":0.40799999999999997,"5min":0.35399999999999998,"15min":0.30499999999999999},"max":{"1min":0.70999999999999996,"5min":0.86499999999999999,"15min":12.185},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61899999999999999,"15min":0.67100000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.373},"max":{"1min":0.80700000000000005,"5min":0.89600000000000002,"15min":7.6760000000000002},"last":0.82399999999999995}]},{"osd":4,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.80300000000000005,"15min":0.879},"min":{"1min":0.59999999999999998,"5min":0.54700000000000004,"15min":0.44},"max":{"1min":1.0660000000000001,"5min":1.22,"15min":8.6609999999999996},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.76800000000000002,"15min":0.91200000000000003},"min":{"1min":0.56399999999999995,"5min":0.46200000000000002,"15min":0.41599999999999998},"max":{"1min":1.008,"5min":1.9670000000000001,"15min":19.146999999999998},"last":0.91300000000000003}]},{"osd":5,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.76000000000000001,"15min":0.90200000000000002},"min":{"1min":0.58299999999999996,"5min":0.53600000000000003,"15min":0.41899999999999998},"max":{"1min":1.093,"5min":1.111,"15min":31.710000000000001},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.81999999999999995,"15min":0.999},"min":{"1min":0.57399999999999995,"5min":0.51700000000000002,"15min":0.48199999999999998},"max":{"1min":0.99199999999999999,"5min":1.8240000000000001,"15min":22.539000000000001},"last":0.93000000000000005}]},{"osd":6,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.78300000000000003,"15min":0.91000000000000003},"min":{"1min":0.56499999999999995,"5min":0.52900000000000003,"15min":0.40899999999999997},"max":{"1min":1.079,"5min":1.3979999999999999,"15min":9.4540000000000006},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.80100000000000005,"15min":0.90800000000000003},"min":{"1min":0.64300000000000002,"5min":0.5,"15min":0.47899999999999998},"max":{"1min":1.0309999999999999,"5min":1.873,"15min":18.614999999999998},"last":0.79800000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.79800000000000004,"15min":0.96799999999999997},"min":{"1min":0.63200000000000001,"5min":0.50900000000000001,"15min":0.44400000000000001},"max":{"1min":1.0109999999999999,"5min":1.7729999999999999,"15min":11.606},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.84699999999999998,"15min":0.998},"min":{"1min":0.627,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.105,"5min":1.611,"15min":17.013000000000002},"last":0.97399999999999998}]}]},{"osd":1,"up_from":15,"seq":64424509674,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1133516,"kb_used_data":1066308,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93238324,"statfs":{"total":96636764160,"available":95476043776,"internally_reserved":0,"allocated":1091899392,"data_stored":1091115320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45400000000000001,"15min":0.58499999999999996},"min":{"1min":0.26800000000000002,"5min":0.25900000000000001,"15min":0.23699999999999999},"max":{"1min":0.63500000000000001,"5min":0.88100000000000001,"15min":17.396999999999998},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.51600000000000001,"15min":0.626},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.26900000000000002},"max":{"1min":0.73699999999999999,"5min":0.82999999999999996,"15min":12.378},"last":0.50900000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.51700000000000002,"15min":0.57699999999999996},"min":{"1min":0.33000000000000002,"5min":0.313,"15min":0.309},"max":{"1min":0.61699999999999999,"5min":0.878,"15min":6.4569999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.54300000000000004,"15min":0.67400000000000004},"min":{"1min":0.40100000000000002,"5min":0.34499999999999997,"15min":0.33300000000000002},"max":{"1min":0.73899999999999999,"5min":0.81299999999999994,"15min":18.827000000000002},"last":0.63500000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.59799999999999998,"15min":0.67200000000000004},"min":{"1min":0.433,"5min":0.40600000000000003,"15min":0.33700000000000002},"max":{"1min":0.872,"5min":1.0580000000000001,"15min":5.915},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.53800000000000003,"15min":0.64800000000000002},"min":{"1min":0.34399999999999997,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.77500000000000002,"5min":0.85499999999999998,"15min":9.9719999999999995},"last":0.45900000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.77700000000000002,"15min":0.89400000000000002},"min":{"1min":0.50900000000000001,"5min":0.437,"15min":0.437},"max":{"1min":1.056,"5min":1.1060000000000001,"15min":9.9329999999999998},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.746,"5min":0.79000000000000004,"15min":0.88300000000000001},"min":{"1min":0.55600000000000005,"5min":0.52800000000000002,"15min":0.45800000000000002},"max":{"1min":0.95599999999999996,"5min":1.165,"15min":9.8610000000000007},"last":0.79800000000000004}]},{"osd":5,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.80200000000000005,"15min":0.96299999999999997},"min":{"1min":0.55000000000000004,"5min":0.51200000000000001,"15min":0.46100000000000002},"max":{"1min":0.97499999999999998,"5min":1.2050000000000001,"15min":22.594999999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.78200000000000003,"15min":1.0129999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.443},"max":{"1min":0.97499999999999998,"5min":1.077,"15min":26.748999999999999},"last":0.81299999999999994}]},{"osd":6,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.78800000000000003,"15min":1.034},"min":{"1min":0.57899999999999996,"5min":0.53100000000000003,"15min":0.42899999999999999},"max":{"1min":1.095,"5min":1.218,"15min":35.112000000000002},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.81100000000000005,"15min":0.90900000000000003},"min":{"1min":0.62,"5min":0.501,"15min":0.501},"max":{"1min":0.92900000000000005,"5min":1.2090000000000001,"15min":8.3670000000000009},"last":0.73299999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.80800000000000005,"15min":1.024},"min":{"1min":0.56399999999999995,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.141,"5min":1.1950000000000001,"15min":20.872},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.81599999999999995,"15min":0.97799999999999998},"min":{"1min":0.61499999999999999,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.92300000000000004,"5min":1.139,"15min":9.8640000000000008},"last":0.77800000000000002}]}]},{"osd":5,"up_from":14,"seq":60129542378,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":904260,"kb_used_data":828732,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93467580,"statfs":{"total":96636764160,"available":95710801920,"internally_reserved":0,"allocated":848621568,"data_stored":847844597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.72299999999999998,"15min":0.754},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.29899999999999999},"max":{"1min":1.038,"5min":2.6920000000000002,"15min":5.1639999999999997},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.75800000000000001,"15min":0.83899999999999997},"min":{"1min":0.46899999999999997,"5min":0.36799999999999999,"15min":0.34899999999999998},"max":{"1min":0.98199999999999998,"5min":6.4290000000000003,"15min":9.359},"last":0.88600000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.76800000000000002,"15min":0.93000000000000005},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.375},"max":{"1min":0.89900000000000002,"5min":6.5069999999999997,"15min":21.120000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.747,"15min":0.93899999999999995},"min":{"1min":0.48299999999999998,"5min":0.39900000000000002,"15min":0.38600000000000001},"max":{"1min":0.96499999999999997,"5min":2.6619999999999999,"15min":15.843},"last":0.86699999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.76800000000000002,"15min":0.86699999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.34999999999999998},"max":{"1min":0.95099999999999996,"5min":6.3339999999999996,"15min":8.7750000000000004},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.79900000000000004,"15min":0.98299999999999998},"min":{"1min":0.501,"5min":0.501,"15min":0.39900000000000002},"max":{"1min":0.95399999999999996,"5min":6.4619999999999997,"15min":21.201000000000001},"last":0.84699999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.753,"15min":0.84399999999999997},"min":{"1min":0.498,"5min":0.46000000000000002,"15min":0.37},"max":{"1min":0.99199999999999999,"5min":2.6299999999999999,"15min":7.7199999999999998},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.755,"5min":0.81100000000000005,"15min":0.91900000000000004},"min":{"1min":0.52800000000000002,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.0109999999999999,"5min":6.3730000000000002,"15min":8.7300000000000004},"last":0.92800000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.66500000000000004,"15min":0.83399999999999996},"min":{"1min":0.52600000000000002,"5min":0.42999999999999999,"15min":0.30399999999999999},"max":{"1min":1.012,"5min":3.161,"15min":10.962},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.70699999999999996,"15min":1.012},"min":{"1min":0.48799999999999999,"5min":0.41999999999999998,"15min":0.35099999999999998},"max":{"1min":0.88100000000000001,"5min":6.5460000000000003,"15min":26.146999999999998},"last":0.70599999999999996}]},{"osd":6,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.66500000000000004,"15min":0.76700000000000002},"min":{"1min":0.50700000000000001,"5min":0.41299999999999998,"15min":0.36099999999999999},"max":{"1min":0.871,"5min":2.589,"15min":6.6699999999999999},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.82399999999999995,"15min":0.99199999999999999},"min":{"1min":0.53800000000000003,"5min":0.45300000000000001,"15min":0.379},"max":{"1min":0.95699999999999996,"5min":14.007999999999999,"15min":30.488},"last":0.96199999999999997}]},{"osd":7,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.72499999999999998,"15min":1.2010000000000001},"min":{"1min":0.56699999999999995,"5min":0.48899999999999999,"15min":0.42599999999999999},"max":{"1min":0.90800000000000003,"5min":3.0870000000000002,"15min":33.499000000000002},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.91000000000000003,"15min":1.0680000000000001},"min":{"1min":0.57099999999999995,"5min":0.45700000000000002,"15min":0.42899999999999999},"max":{"1min":0.93000000000000005,"5min":21.178999999999998,"15min":25.437999999999999},"last":0.82699999999999996}]}]},{"osd":0,"up_from":14,"seq":60129542378,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1769580,"kb_used_data":1702372,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92602260,"statfs":{"total":96636764160,"available":94824714240,"internally_reserved":0,"allocated":1743228928,"data_stored":1742435619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.82999999999999996,"15min":0.749},"min":{"1min":0.33100000000000002,"5min":0.25800000000000001,"15min":0.245},"max":{"1min":0.59899999999999998,"5min":38.521999999999998,"15min":38.521999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.60099999999999998,"15min":0.68799999999999994},"min":{"1min":0.36699999999999999,"5min":0.32900000000000001,"15min":0.27000000000000002},"max":{"1min":0.753,"5min":8.0749999999999993,"15min":10.154},"last":0.54700000000000004}]},{"osd":2,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.57699999999999996,"15min":0.80200000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.30399999999999999},"max":{"1min":0.78600000000000003,"5min":2.6909999999999998,"15min":25.905000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.67200000000000004,"15min":0.66900000000000004},"min":{"1min":0.307,"5min":0.307,"15min":0.27800000000000002},"max":{"1min":0.69199999999999995,"5min":18.247,"15min":18.247},"last":0.48399999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.76400000000000001,"15min":0.80300000000000005},"min":{"1min":0.32400000000000001,"5min":0.32000000000000001,"15min":0.29199999999999998},"max":{"1min":0.70999999999999996,"5min":21.451000000000001,"15min":21.451000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61799999999999999,"15min":0.76600000000000001},"min":{"1min":0.47099999999999997,"5min":0.38900000000000001,"15min":0.36899999999999999},"max":{"1min":0.80100000000000005,"5min":1.7,"15min":10.210000000000001},"last":0.59999999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.79400000000000004,"15min":0.99199999999999999},"min":{"1min":0.52600000000000002,"5min":0.441,"15min":0.39100000000000001},"max":{"1min":1.1639999999999999,"5min":2.7850000000000001,"15min":22.814},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.80100000000000005,"5min":1.0389999999999999,"15min":1.105},"min":{"1min":0.64000000000000001,"5min":0.45500000000000002,"15min":0.42399999999999999},"max":{"1min":1.1379999999999999,"5min":27.027999999999999,"15min":27.027999999999999},"last":0.97299999999999998}]},{"osd":5,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.83199999999999996,"15min":0.98799999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.38400000000000001},"max":{"1min":1.218,"5min":6.6429999999999998,"15min":12.428000000000001},"last":0.747},{"interface":"front","average":{"1min":0.77700000000000002,"5min":1.0509999999999999,"15min":1.333},"min":{"1min":0.55400000000000005,"5min":0.47999999999999998,"15min":0.46100000000000002},"max":{"1min":1.0600000000000001,"5min":27.210999999999999,"15min":27.210999999999999},"last":1.024}]},{"osd":6,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.86699999999999999,"15min":1.242},"min":{"1min":0.66400000000000003,"5min":0.50900000000000001,"15min":0.46700000000000003},"max":{"1min":0.98399999999999999,"5min":9.3140000000000001,"15min":31.059000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.82699999999999996,"15min":1.089},"min":{"1min":0.58899999999999997,"5min":0.52900000000000003,"15min":0.51500000000000001},"max":{"1min":1.024,"5min":2.8340000000000001,"15min":40.731000000000002},"last":0.88200000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.86299999999999999,"15min":1.2150000000000001},"min":{"1min":0.66500000000000004,"5min":0.52200000000000002,"15min":0.499},"max":{"1min":1.1539999999999999,"5min":5.2089999999999996,"15min":24.492999999999999},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.81599999999999995,"5min":1.097,"15min":1.1859999999999999},"min":{"1min":0.63500000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1200000000000001,"5min":27.081,"15min":27.081},"last":1.0580000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":630784,"data_stored":614187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":1321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":626688,"data_stored":614145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1740644352,"data_stored":1740636601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1090527232,"data_stored":1090519412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":931139584,"data_stored":931135680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1417682944,"data_stored":1417675274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1681920000,"data_stored":1681916165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":847253504,"data_stored":847249530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1220546560,"data_stored":1220542586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1807745024,"data_stored":1807745024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:24:02.077 INFO:tasks.ceph:pgid 17.3f last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.077 INFO:tasks.ceph:pgid 17.3e last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.077 INFO:tasks.ceph:pgid 17.3d last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.078 INFO:tasks.ceph:pgid 17.3b last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.078 INFO:tasks.ceph:pgid 17.3a last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.078 INFO:tasks.ceph:pgid 17.39 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.078 INFO:tasks.ceph:pgid 17.37 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.078 INFO:tasks.ceph:pgid 17.36 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.078 INFO:tasks.ceph:pgid 17.35 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.079 INFO:tasks.ceph:pgid 17.34 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.079 INFO:tasks.ceph:pgid 17.33 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.079 INFO:tasks.ceph:pgid 17.30 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.079 INFO:tasks.ceph:pgid 17.2f last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.079 INFO:tasks.ceph:pgid 17.2d last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.080 INFO:tasks.ceph:pgid 17.2c last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.080 INFO:tasks.ceph:pgid 17.2a last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.080 INFO:tasks.ceph:pgid 17.28 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.080 INFO:tasks.ceph:pgid 17.27 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.080 INFO:tasks.ceph:pgid 17.26 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.080 INFO:tasks.ceph:pgid 17.25 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.081 INFO:tasks.ceph:pgid 17.24 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.081 INFO:tasks.ceph:pgid 17.23 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.081 INFO:tasks.ceph:pgid 17.22 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.081 INFO:tasks.ceph:pgid 17.20 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.081 INFO:tasks.ceph:pgid 17.1e last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.081 INFO:tasks.ceph:pgid 17.1f last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.082 INFO:tasks.ceph:pgid 17.1c last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.082 INFO:tasks.ceph:pgid 17.1d last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.082 INFO:tasks.ceph:pgid 17.1a last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.082 INFO:tasks.ceph:pgid 17.1b last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.082 INFO:tasks.ceph:pgid 17.18 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.082 INFO:tasks.ceph:pgid 17.19 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.083 INFO:tasks.ceph:pgid 17.16 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.083 INFO:tasks.ceph:pgid 17.14 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.083 INFO:tasks.ceph:pgid 17.15 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.083 INFO:tasks.ceph:pgid 17.12 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.083 INFO:tasks.ceph:pgid 17.13 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.084 INFO:tasks.ceph:pgid 17.e last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.084 INFO:tasks.ceph:pgid 16.f last_scrub_stamp 2024-01-15T18:18:31.173930+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=31, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.084 INFO:tasks.ceph:pgid 17.f last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.084 INFO:tasks.ceph:pgid 17.c last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.084 INFO:tasks.ceph:pgid 17.d last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.084 INFO:tasks.ceph:pgid 17.a last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.085 INFO:tasks.ceph:pgid 17.b last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.085 INFO:tasks.ceph:pgid 17.8 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.085 INFO:tasks.ceph:pgid 17.9 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.085 INFO:tasks.ceph:pgid 17.7 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.085 INFO:tasks.ceph:pgid 17.5 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.086 INFO:tasks.ceph:pgid 17.3 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.086 INFO:tasks.ceph:pgid 17.0 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:02.086 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-15T18:24:22.087 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:24:22.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.239+0000 7f8ad3200640 1 Processor -- start 2024-01-15T18:24:22.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.239+0000 7f8ad3200640 1 -- start start 2024-01-15T18:24:22.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.239+0000 7f8ad3200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f8acc15e810 0x7f8acc0afc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:22.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.239+0000 7f8ad3200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8acc0cf6c0 0x7f8acc0adc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:22.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.239+0000 7f8ad3200640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8acc0b18a0 con 0x7f8acc0b2070 2024-01-15T18:24:22.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.239+0000 7f8ad3200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8acc0b27f0 con 0x7f8acc15e810 2024-01-15T18:24:22.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.239+0000 7f8ad3200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8acc0b2510 con 0x7f8acc0cf6c0 2024-01-15T18:24:22.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.239+0000 7f8ad1200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8acc0cf6c0 0x7f8acc0adc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:22.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.239+0000 7f8acbe00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f8acc15e810 0x7f8acc0afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:22.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.239+0000 7f8ad1200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8acc0cf6c0 0x7f8acc0adc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.93:3301/0 says I am v2:172.21.15.27:43976/0 (socket says 172.21.15.27:43976) 2024-01-15T18:24:22.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.239+0000 7f8ad1200640 1 -- 172.21.15.27:0/2096967449 learned_addr learned my addr 172.21.15.27:0/2096967449 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:24:22.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.243+0000 7f8ad1200640 1 -- 172.21.15.27:0/2096967449 >> v2:172.21.15.93:3300/0 conn(0x7f8acc15e810 msgr2=0x7f8acc0afc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:24:22.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.243+0000 7f8ad1200640 1 --2- 172.21.15.27:0/2096967449 >> v2:172.21.15.93:3300/0 conn(0x7f8acc15e810 0x7f8acc0afc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:22.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.243+0000 7f8ad1200640 1 -- 172.21.15.27:0/2096967449 >> v1:172.21.15.93:6789/0 conn(0x7f8acc0b2070 legacy=0x7f8acc0cefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:22.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.243+0000 7f8ad1200640 1 -- 172.21.15.27:0/2096967449 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8acc0b1250 con 0x7f8acc0cf6c0 2024-01-15T18:24:22.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.243+0000 7f8ad1200640 1 --2- 172.21.15.27:0/2096967449 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8acc0cf6c0 0x7f8acc0adc50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8ac4009e50 tx=0x7f8ac401c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce2f1de5b5f14b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:24:22.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.243+0000 7f8acaa00640 1 -- 172.21.15.27:0/2096967449 <== mon.2 v2:172.21.15.93:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac40a58d0 con 0x7f8acc0cf6c0 2024-01-15T18:24:22.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.243+0000 7f8acaa00640 1 -- 172.21.15.27:0/2096967449 <== mon.2 v2:172.21.15.93:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac40a5a30 con 0x7f8acc0cf6c0 2024-01-15T18:24:22.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.243+0000 7f8acaa00640 1 -- 172.21.15.27:0/2096967449 <== mon.2 v2:172.21.15.93:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac40a5dc0 con 0x7f8acc0cf6c0 2024-01-15T18:24:22.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.247+0000 7f8ad3200640 1 -- 172.21.15.27:0/2096967449 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8acc0cf6c0 msgr2=0x7f8acc0adc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:22.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.247+0000 7f8ad3200640 1 --2- 172.21.15.27:0/2096967449 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8acc0cf6c0 0x7f8acc0adc50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8ac4009e50 tx=0x7f8ac401c9e0 comp rx=0 tx=0).stop 2024-01-15T18:24:22.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.247+0000 7f8ad3200640 1 -- 172.21.15.27:0/2096967449 shutdown_connections 2024-01-15T18:24:22.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.247+0000 7f8ad3200640 1 --2- 172.21.15.27:0/2096967449 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8acc0cf6c0 0x7f8acc0adc50 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:22.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.247+0000 7f8ad3200640 1 --2- 172.21.15.27:0/2096967449 >> v2:172.21.15.93:3300/0 conn(0x7f8acc15e810 0x7f8acc0afc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:22.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.247+0000 7f8ad3200640 1 -- 172.21.15.27:0/2096967449 >> 172.21.15.27:0/2096967449 conn(0x7f8acc0d0f20 msgr2=0x7f8acc0d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:24:22.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.247+0000 7f8ad3200640 1 -- 172.21.15.27:0/2096967449 shutdown_connections 2024-01-15T18:24:22.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.247+0000 7f8ad3200640 1 -- 172.21.15.27:0/2096967449 wait complete. 2024-01-15T18:24:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.247+0000 7f8ad3200640 1 Processor -- start 2024-01-15T18:24:22.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.247+0000 7f8ad3200640 1 -- start start 2024-01-15T18:24:22.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.247+0000 7f8ad3200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8acc0b2070 0x7f8acc0c4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:22.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.247+0000 7f8ad3200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8acc0cf6c0 0x7f8acc151270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:22.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.247+0000 7f8ad3200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8acc15e810 0x7f8acc152c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:22.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.247+0000 7f8ad3200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8acc1568b0 con 0x7f8acc0cf6c0 2024-01-15T18:24:22.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.247+0000 7f8ad3200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8acc156b90 con 0x7f8acc0b2070 2024-01-15T18:24:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.247+0000 7f8ad3200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8acc156e70 con 0x7f8acc15e810 2024-01-15T18:24:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.247+0000 7f8acb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8acc0cf6c0 0x7f8acc151270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.247+0000 7f8acb400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8acc0cf6c0 0x7f8acc151270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34956/0 (socket says 172.21.15.27:34956) 2024-01-15T18:24:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.247+0000 7f8acb400640 1 -- 172.21.15.27:0/3055721624 learned_addr learned my addr 172.21.15.27:0/3055721624 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:24:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.247+0000 7f8ad1200640 1 --2- 172.21.15.27:0/3055721624 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8acc15e810 0x7f8acc152c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.247+0000 7f8acbe00640 1 --2- 172.21.15.27:0/3055721624 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8acc0b2070 0x7f8acc0c4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:22.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.251+0000 7f8acb400640 1 -- 172.21.15.27:0/3055721624 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8acc15e810 msgr2=0x7f8acc152c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.251+0000 7f8acb400640 1 --2- 172.21.15.27:0/3055721624 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8acc15e810 0x7f8acc152c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.251+0000 7f8acb400640 1 -- 172.21.15.27:0/3055721624 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8acc0b2070 msgr2=0x7f8acc0c4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.251+0000 7f8acb400640 1 --2- 172.21.15.27:0/3055721624 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8acc0b2070 0x7f8acc0c4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.251+0000 7f8acb400640 1 -- 172.21.15.27:0/3055721624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac4009b30 con 0x7f8acc0cf6c0 2024-01-15T18:24:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.251+0000 7f8ad1200640 1 --2- 172.21.15.27:0/3055721624 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8acc15e810 0x7f8acc152c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:24:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.251+0000 7f8acbe00640 1 --2- 172.21.15.27:0/3055721624 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8acc0b2070 0x7f8acc0c4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:24:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.251+0000 7f8acb400640 1 --2- 172.21.15.27:0/3055721624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8acc0cf6c0 0x7f8acc151270 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f8ab400c500 tx=0x7f8ab400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:24:22.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.251+0000 7f8ac3e00640 1 -- 172.21.15.27:0/3055721624 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab4005220 con 0x7f8acc0cf6c0 2024-01-15T18:24:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.251+0000 7f8ac3e00640 1 -- 172.21.15.27:0/3055721624 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ab4005380 con 0x7f8acc0cf6c0 2024-01-15T18:24:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.251+0000 7f8ac3e00640 1 -- 172.21.15.27:0/3055721624 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab4007270 con 0x7f8acc0cf6c0 2024-01-15T18:24:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.251+0000 7f8ad3200640 1 -- 172.21.15.27:0/3055721624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8acc157150 con 0x7f8acc0cf6c0 2024-01-15T18:24:22.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.251+0000 7f8ad3200640 1 -- 172.21.15.27:0/3055721624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8acc0c4eb0 con 0x7f8acc0cf6c0 2024-01-15T18:24:22.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.251+0000 7f8ad3200640 1 -- 172.21.15.27:0/3055721624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8acc0d0870 con 0x7f8acc0cf6c0 2024-01-15T18:24:22.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.251+0000 7f8ac3e00640 1 -- 172.21.15.27:0/3055721624 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 45) v1 ==== 87741+0+0 (secure 0 0 0) 0x7f8ab40073d0 con 0x7f8acc0cf6c0 2024-01-15T18:24:22.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.251+0000 7f8ac3e00640 1 --2- 172.21.15.27:0/3055721624 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8aa4067a60 0x7f8aa4069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:22.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.251+0000 7f8ac3e00640 1 -- 172.21.15.27:0/3055721624 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f8ab4082cd0 con 0x7f8acc0cf6c0 2024-01-15T18:24:22.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.251+0000 7f8acbe00640 1 --2- 172.21.15.27:0/3055721624 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8aa4067a60 0x7f8aa4069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:22.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.255+0000 7f8acbe00640 1 --2- 172.21.15.27:0/3055721624 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8aa4067a60 0x7f8aa4069f20 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f8acc067940 tx=0x7f8abc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:24:22.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.255+0000 7f8ac3e00640 1 -- 172.21.15.27:0/3055721624 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8ab4051590 con 0x7f8acc0cf6c0 2024-01-15T18:24:22.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.415+0000 7f8ad3200640 1 -- 172.21.15.27:0/3055721624 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8acc0ae9a0 con 0x7f8aa4067a60 2024-01-15T18:24:22.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.423+0000 7f8ac3e00640 1 -- 172.21.15.27:0/3055721624 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+293010 (secure 0 0 0) 0x7f8abc0119a0 con 0x7f8aa4067a60 2024-01-15T18:24:22.421 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:24:22.422 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:24:22.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.427+0000 7f8ad3200640 1 -- 172.21.15.27:0/3055721624 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8aa4067a60 msgr2=0x7f8aa4069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:22.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.427+0000 7f8ad3200640 1 --2- 172.21.15.27:0/3055721624 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8aa4067a60 0x7f8aa4069f20 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f8acc067940 tx=0x7f8abc008040 comp rx=0 tx=0).stop 2024-01-15T18:24:22.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.427+0000 7f8ad3200640 1 -- 172.21.15.27:0/3055721624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8acc0cf6c0 msgr2=0x7f8acc151270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:22.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.427+0000 7f8ad3200640 1 --2- 172.21.15.27:0/3055721624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8acc0cf6c0 0x7f8acc151270 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f8ab400c500 tx=0x7f8ab400c9c0 comp rx=0 tx=0).stop 2024-01-15T18:24:22.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.427+0000 7f8ad3200640 1 -- 172.21.15.27:0/3055721624 shutdown_connections 2024-01-15T18:24:22.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.427+0000 7f8ad3200640 1 --2- 172.21.15.27:0/3055721624 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f8aa4067a60 0x7f8aa4069f20 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:22.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.427+0000 7f8ad3200640 1 --2- 172.21.15.27:0/3055721624 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f8acc15e810 0x7f8acc152c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:22.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.427+0000 7f8ad3200640 1 --2- 172.21.15.27:0/3055721624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8acc0cf6c0 0x7f8acc151270 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:22.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.427+0000 7f8ad3200640 1 --2- 172.21.15.27:0/3055721624 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f8acc0b2070 0x7f8acc0c4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:22.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.427+0000 7f8ad3200640 1 -- 172.21.15.27:0/3055721624 >> 172.21.15.27:0/3055721624 conn(0x7f8acc0d0f20 msgr2=0x7f8acc0cd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:24:22.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.427+0000 7f8ad3200640 1 -- 172.21.15.27:0/3055721624 shutdown_connections 2024-01-15T18:24:22.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:22.427+0000 7f8ad3200640 1 -- 172.21.15.27:0/3055721624 wait complete. 2024-01-15T18:24:22.441 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":691,"stamp":"2024-01-15T18:24:20.469677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5369916698,"num_objects":1311,"num_object_clones":1,"num_object_copies":2622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1311,"num_whiteouts":1,"num_read":651,"num_read_kb":2097275,"num_write":1745,"num_write_kb":5246031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5506,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4143,"ondisk_log_size":4143,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":11048992,"kb_used_data":10498848,"kb_used_omap":12,"kb_used_meta":550067,"kb_avail":743925728,"statfs":{"total":773094113280,"available":761779945472,"internally_reserved":0,"allocated":10750820352,"data_stored":10744589100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":563269168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002946"},"pg_stats":[{"pgid":"17.3f","version":"110'53","reported_seq":78,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.666198+0000","last_change":"2024-01-15T18:18:33.168792+0000","last_active":"2024-01-15T18:23:31.666198+0000","last_peered":"2024-01-15T18:23:31.666198+0000","last_clean":"2024-01-15T18:23:31.666198+0000","last_became_active":"2024-01-15T18:18:33.168500+0000","last_became_peered":"2024-01-15T18:18:33.168500+0000","last_unstale":"2024-01-15T18:23:31.666198+0000","last_undegraded":"2024-01-15T18:23:31.666198+0000","last_fullsized":"2024-01-15T18:23:31.666198+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"110'64","reported_seq":96,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:18.004675+0000","last_change":"2024-01-15T18:24:18.004707+0000","last_active":"2024-01-15T18:24:18.004675+0000","last_peered":"2024-01-15T18:24:18.004675+0000","last_clean":"2024-01-15T18:24:18.004675+0000","last_became_active":"2024-01-15T18:18:33.151209+0000","last_became_peered":"2024-01-15T18:18:33.151209+0000","last_unstale":"2024-01-15T18:24:18.004675+0000","last_undegraded":"2024-01-15T18:24:18.004675+0000","last_fullsized":"2024-01-15T18:24:18.004675+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'64","last_scrub_stamp":"2024-01-15T18:24:18.004594+0000","last_deep_scrub":"110'64","last_deep_scrub_stamp":"2024-01-15T18:24:18.004594+0000","last_clean_scrub_stamp":"2024-01-15T18:24:18.004594+0000","objects_scrubbed":19,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:58:47.994108+0000","scrub_duration":0.121021875,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"110'63","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:15.376299+0000","last_change":"2024-01-15T18:24:15.376335+0000","last_active":"2024-01-15T18:24:15.376299+0000","last_peered":"2024-01-15T18:24:15.376299+0000","last_clean":"2024-01-15T18:24:15.376299+0000","last_became_active":"2024-01-15T18:18:33.162700+0000","last_became_peered":"2024-01-15T18:18:33.162700+0000","last_unstale":"2024-01-15T18:24:15.376299+0000","last_undegraded":"2024-01-15T18:24:15.376299+0000","last_fullsized":"2024-01-15T18:24:15.376299+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'63","last_scrub_stamp":"2024-01-15T18:24:15.376222+0000","last_deep_scrub":"110'63","last_deep_scrub_stamp":"2024-01-15T18:24:15.376222+0000","last_clean_scrub_stamp":"2024-01-15T18:24:15.376222+0000","objects_scrubbed":19,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:50:26.909076+0000","scrub_duration":0.130268359,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"110'62","reported_seq":92,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:37.761746+0000","last_change":"2024-01-15T18:23:37.761794+0000","last_active":"2024-01-15T18:23:37.761746+0000","last_peered":"2024-01-15T18:23:37.761746+0000","last_clean":"2024-01-15T18:23:37.761746+0000","last_became_active":"2024-01-15T18:18:33.160460+0000","last_became_peered":"2024-01-15T18:18:33.160460+0000","last_unstale":"2024-01-15T18:23:37.761746+0000","last_undegraded":"2024-01-15T18:23:37.761746+0000","last_fullsized":"2024-01-15T18:23:37.761746+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'62","last_scrub_stamp":"2024-01-15T18:23:37.761665+0000","last_deep_scrub":"110'62","last_deep_scrub_stamp":"2024-01-15T18:23:37.761665+0000","last_clean_scrub_stamp":"2024-01-15T18:23:37.761665+0000","objects_scrubbed":23,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:24:28.403935+0000","scrub_duration":0.14250365100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"110'35","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:01.029619+0000","last_change":"2024-01-15T18:24:01.029682+0000","last_active":"2024-01-15T18:24:01.029619+0000","last_peered":"2024-01-15T18:24:01.029619+0000","last_clean":"2024-01-15T18:24:01.029619+0000","last_became_active":"2024-01-15T18:18:33.155656+0000","last_became_peered":"2024-01-15T18:18:33.155656+0000","last_unstale":"2024-01-15T18:24:01.029619+0000","last_undegraded":"2024-01-15T18:24:01.029619+0000","last_fullsized":"2024-01-15T18:24:01.029619+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'35","last_scrub_stamp":"2024-01-15T18:24:01.029508+0000","last_deep_scrub":"110'35","last_deep_scrub_stamp":"2024-01-15T18:24:01.029508+0000","last_clean_scrub_stamp":"2024-01-15T18:24:01.029508+0000","objects_scrubbed":10,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:30:03.396033+0000","scrub_duration":0.072926557000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"110'69","reported_seq":104,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:09.399789+0000","last_change":"2024-01-15T18:24:09.399859+0000","last_active":"2024-01-15T18:24:09.399789+0000","last_peered":"2024-01-15T18:24:09.399789+0000","last_clean":"2024-01-15T18:24:09.399789+0000","last_became_active":"2024-01-15T18:18:33.151701+0000","last_became_peered":"2024-01-15T18:18:33.151701+0000","last_unstale":"2024-01-15T18:24:09.399789+0000","last_undegraded":"2024-01-15T18:24:09.399789+0000","last_fullsized":"2024-01-15T18:24:09.399789+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'69","last_scrub_stamp":"2024-01-15T18:24:09.399699+0000","last_deep_scrub":"110'69","last_deep_scrub_stamp":"2024-01-15T18:24:09.399699+0000","last_clean_scrub_stamp":"2024-01-15T18:24:09.399699+0000","objects_scrubbed":21,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:03:09.635967+0000","scrub_duration":0.161292621,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"110'58","reported_seq":86,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:59.512912+0000","last_change":"2024-01-15T18:23:59.512955+0000","last_active":"2024-01-15T18:23:59.512912+0000","last_peered":"2024-01-15T18:23:59.512912+0000","last_clean":"2024-01-15T18:23:59.512912+0000","last_became_active":"2024-01-15T18:18:33.153264+0000","last_became_peered":"2024-01-15T18:18:33.153264+0000","last_unstale":"2024-01-15T18:23:59.512912+0000","last_undegraded":"2024-01-15T18:23:59.512912+0000","last_fullsized":"2024-01-15T18:23:59.512912+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'58","last_scrub_stamp":"2024-01-15T18:23:59.512827+0000","last_deep_scrub":"110'58","last_deep_scrub_stamp":"2024-01-15T18:23:59.512827+0000","last_clean_scrub_stamp":"2024-01-15T18:23:59.512827+0000","objects_scrubbed":17,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:46:09.671279+0000","scrub_duration":0.12555955899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"110'61","reported_seq":93,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:56.297341+0000","last_change":"2024-01-15T18:23:56.297378+0000","last_active":"2024-01-15T18:23:56.297341+0000","last_peered":"2024-01-15T18:23:56.297341+0000","last_clean":"2024-01-15T18:23:56.297341+0000","last_became_active":"2024-01-15T18:18:33.163896+0000","last_became_peered":"2024-01-15T18:18:33.163896+0000","last_unstale":"2024-01-15T18:23:56.297341+0000","last_undegraded":"2024-01-15T18:23:56.297341+0000","last_fullsized":"2024-01-15T18:23:56.297341+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'61","last_scrub_stamp":"2024-01-15T18:23:56.297250+0000","last_deep_scrub":"110'61","last_deep_scrub_stamp":"2024-01-15T18:23:56.297250+0000","last_clean_scrub_stamp":"2024-01-15T18:23:56.297250+0000","objects_scrubbed":19,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:44:20.320452+0000","scrub_duration":0.120656875,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"110'47","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.030521+0000","last_change":"2024-01-15T18:18:33.158682+0000","last_active":"2024-01-15T18:23:32.030521+0000","last_peered":"2024-01-15T18:23:32.030521+0000","last_clean":"2024-01-15T18:23:32.030521+0000","last_became_active":"2024-01-15T18:18:33.158399+0000","last_became_peered":"2024-01-15T18:18:33.158399+0000","last_unstale":"2024-01-15T18:23:32.030521+0000","last_undegraded":"2024-01-15T18:23:32.030521+0000","last_fullsized":"2024-01-15T18:23:32.030521+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"110'47","reported_seq":73,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.054699+0000","last_change":"2024-01-15T18:18:33.159936+0000","last_active":"2024-01-15T18:23:32.054699+0000","last_peered":"2024-01-15T18:23:32.054699+0000","last_clean":"2024-01-15T18:23:32.054699+0000","last_became_active":"2024-01-15T18:18:33.159562+0000","last_became_peered":"2024-01-15T18:18:33.159562+0000","last_unstale":"2024-01-15T18:23:32.054699+0000","last_undegraded":"2024-01-15T18:23:32.054699+0000","last_fullsized":"2024-01-15T18:23:32.054699+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"110'75","reported_seq":112,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:11.106393+0000","last_change":"2024-01-15T18:24:11.106416+0000","last_active":"2024-01-15T18:24:11.106393+0000","last_peered":"2024-01-15T18:24:11.106393+0000","last_clean":"2024-01-15T18:24:11.106393+0000","last_became_active":"2024-01-15T18:18:33.154972+0000","last_became_peered":"2024-01-15T18:18:33.154972+0000","last_unstale":"2024-01-15T18:24:11.106393+0000","last_undegraded":"2024-01-15T18:24:11.106393+0000","last_fullsized":"2024-01-15T18:24:11.106393+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'75","last_scrub_stamp":"2024-01-15T18:24:11.106322+0000","last_deep_scrub":"110'75","last_deep_scrub_stamp":"2024-01-15T18:24:11.106322+0000","last_clean_scrub_stamp":"2024-01-15T18:24:11.106322+0000","objects_scrubbed":28,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:48:48.772386+0000","scrub_duration":0.19587069500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"110'48","reported_seq":84,"reported_epoch":110,"state":"active+clean+scrubbing+deep","last_fresh":"2024-01-15T18:24:19.683538+0000","last_change":"2024-01-15T18:24:19.682754+0000","last_active":"2024-01-15T18:24:19.683538+0000","last_peered":"2024-01-15T18:24:19.683538+0000","last_clean":"2024-01-15T18:24:19.683538+0000","last_became_active":"2024-01-15T18:18:33.158064+0000","last_became_peered":"2024-01-15T18:18:33.158064+0000","last_unstale":"2024-01-15T18:24:19.683538+0000","last_undegraded":"2024-01-15T18:24:19.683538+0000","last_fullsized":"2024-01-15T18:24:19.683538+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":16,"num_read_kb":61441,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"110'38","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.553796+0000","last_change":"2024-01-15T18:18:33.163912+0000","last_active":"2024-01-15T18:23:31.553796+0000","last_peered":"2024-01-15T18:23:31.553796+0000","last_clean":"2024-01-15T18:23:31.553796+0000","last_became_active":"2024-01-15T18:18:33.163610+0000","last_became_peered":"2024-01-15T18:18:33.163610+0000","last_unstale":"2024-01-15T18:23:31.553796+0000","last_undegraded":"2024-01-15T18:23:31.553796+0000","last_fullsized":"2024-01-15T18:23:31.553796+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"110'70","reported_seq":102,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:52.349564+0000","last_change":"2024-01-15T18:23:52.349588+0000","last_active":"2024-01-15T18:23:52.349564+0000","last_peered":"2024-01-15T18:23:52.349564+0000","last_clean":"2024-01-15T18:23:52.349564+0000","last_became_active":"2024-01-15T18:18:33.169530+0000","last_became_peered":"2024-01-15T18:18:33.169530+0000","last_unstale":"2024-01-15T18:23:52.349564+0000","last_undegraded":"2024-01-15T18:23:52.349564+0000","last_fullsized":"2024-01-15T18:23:52.349564+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'70","last_scrub_stamp":"2024-01-15T18:23:52.349480+0000","last_deep_scrub":"110'70","last_deep_scrub_stamp":"2024-01-15T18:23:52.349480+0000","last_clean_scrub_stamp":"2024-01-15T18:23:52.349480+0000","objects_scrubbed":27,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:02.669418+0000","scrub_duration":0.19608498999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"110'41","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:53.303856+0000","last_change":"2024-01-15T18:23:53.303892+0000","last_active":"2024-01-15T18:23:53.303856+0000","last_peered":"2024-01-15T18:23:53.303856+0000","last_clean":"2024-01-15T18:23:53.303856+0000","last_became_active":"2024-01-15T18:18:33.151040+0000","last_became_peered":"2024-01-15T18:18:33.151040+0000","last_unstale":"2024-01-15T18:23:53.303856+0000","last_undegraded":"2024-01-15T18:23:53.303856+0000","last_fullsized":"2024-01-15T18:23:53.303856+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'41","last_scrub_stamp":"2024-01-15T18:23:53.303773+0000","last_deep_scrub":"110'41","last_deep_scrub_stamp":"2024-01-15T18:23:53.303773+0000","last_clean_scrub_stamp":"2024-01-15T18:23:53.303773+0000","objects_scrubbed":12,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:12:23.550874+0000","scrub_duration":0.088010931000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"110'42","reported_seq":61,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.667801+0000","last_change":"2024-01-15T18:18:33.166327+0000","last_active":"2024-01-15T18:23:31.667801+0000","last_peered":"2024-01-15T18:23:31.667801+0000","last_clean":"2024-01-15T18:23:31.667801+0000","last_became_active":"2024-01-15T18:18:33.166014+0000","last_became_peered":"2024-01-15T18:18:33.166014+0000","last_unstale":"2024-01-15T18:23:31.667801+0000","last_undegraded":"2024-01-15T18:23:31.667801+0000","last_fullsized":"2024-01-15T18:23:31.667801+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"110'66","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:15.009110+0000","last_change":"2024-01-15T18:24:15.009148+0000","last_active":"2024-01-15T18:24:15.009110+0000","last_peered":"2024-01-15T18:24:15.009110+0000","last_clean":"2024-01-15T18:24:15.009110+0000","last_became_active":"2024-01-15T18:18:33.167975+0000","last_became_peered":"2024-01-15T18:18:33.167975+0000","last_unstale":"2024-01-15T18:24:15.009110+0000","last_undegraded":"2024-01-15T18:24:15.009110+0000","last_fullsized":"2024-01-15T18:24:15.009110+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'66","last_scrub_stamp":"2024-01-15T18:24:15.009030+0000","last_deep_scrub":"110'66","last_deep_scrub_stamp":"2024-01-15T18:24:15.009030+0000","last_clean_scrub_stamp":"2024-01-15T18:24:15.009030+0000","objects_scrubbed":22,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:29.066710+0000","scrub_duration":0.16366056400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"110'67","reported_seq":104,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:49.598996+0000","last_change":"2024-01-15T18:23:49.599036+0000","last_active":"2024-01-15T18:23:49.598996+0000","last_peered":"2024-01-15T18:23:49.598996+0000","last_clean":"2024-01-15T18:23:49.598996+0000","last_became_active":"2024-01-15T18:18:33.154816+0000","last_became_peered":"2024-01-15T18:18:33.154816+0000","last_unstale":"2024-01-15T18:23:49.598996+0000","last_undegraded":"2024-01-15T18:23:49.598996+0000","last_fullsized":"2024-01-15T18:23:49.598996+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'67","last_scrub_stamp":"2024-01-15T18:23:49.598911+0000","last_deep_scrub":"110'67","last_deep_scrub_stamp":"2024-01-15T18:23:49.598911+0000","last_clean_scrub_stamp":"2024-01-15T18:23:49.598911+0000","objects_scrubbed":21,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:43:00.465860+0000","scrub_duration":0.149097972,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"110'44","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.730734+0000","last_change":"2024-01-15T18:18:33.164028+0000","last_active":"2024-01-15T18:23:31.730734+0000","last_peered":"2024-01-15T18:23:31.730734+0000","last_clean":"2024-01-15T18:23:31.730734+0000","last_became_active":"2024-01-15T18:18:33.163421+0000","last_became_peered":"2024-01-15T18:18:33.163421+0000","last_unstale":"2024-01-15T18:23:31.730734+0000","last_undegraded":"2024-01-15T18:23:31.730734+0000","last_fullsized":"2024-01-15T18:23:31.730734+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"110'63","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:02.045282+0000","last_change":"2024-01-15T18:24:02.045321+0000","last_active":"2024-01-15T18:24:02.045282+0000","last_peered":"2024-01-15T18:24:02.045282+0000","last_clean":"2024-01-15T18:24:02.045282+0000","last_became_active":"2024-01-15T18:18:33.160047+0000","last_became_peered":"2024-01-15T18:18:33.160047+0000","last_unstale":"2024-01-15T18:24:02.045282+0000","last_undegraded":"2024-01-15T18:24:02.045282+0000","last_fullsized":"2024-01-15T18:24:02.045282+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'63","last_scrub_stamp":"2024-01-15T18:24:02.045207+0000","last_deep_scrub":"110'63","last_deep_scrub_stamp":"2024-01-15T18:24:02.045207+0000","last_clean_scrub_stamp":"2024-01-15T18:24:02.045207+0000","objects_scrubbed":17,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:23:06.679457+0000","scrub_duration":0.11837629199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"110'76","reported_seq":114,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:56.964314+0000","last_change":"2024-01-15T18:23:56.964335+0000","last_active":"2024-01-15T18:23:56.964314+0000","last_peered":"2024-01-15T18:23:56.964314+0000","last_clean":"2024-01-15T18:23:56.964314+0000","last_became_active":"2024-01-15T18:18:33.152128+0000","last_became_peered":"2024-01-15T18:18:33.152128+0000","last_unstale":"2024-01-15T18:23:56.964314+0000","last_undegraded":"2024-01-15T18:23:56.964314+0000","last_fullsized":"2024-01-15T18:23:56.964314+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'76","last_scrub_stamp":"2024-01-15T18:23:56.964221+0000","last_deep_scrub":"110'76","last_deep_scrub_stamp":"2024-01-15T18:23:56.964221+0000","last_clean_scrub_stamp":"2024-01-15T18:23:56.964221+0000","objects_scrubbed":26,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:39:29.172538+0000","scrub_duration":0.16674940499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"110'68","reported_seq":100,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:14.112722+0000","last_change":"2024-01-15T18:24:14.112757+0000","last_active":"2024-01-15T18:24:14.112722+0000","last_peered":"2024-01-15T18:24:14.112722+0000","last_clean":"2024-01-15T18:24:14.112722+0000","last_became_active":"2024-01-15T18:18:33.163252+0000","last_became_peered":"2024-01-15T18:18:33.163252+0000","last_unstale":"2024-01-15T18:24:14.112722+0000","last_undegraded":"2024-01-15T18:24:14.112722+0000","last_fullsized":"2024-01-15T18:24:14.112722+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'68","last_scrub_stamp":"2024-01-15T18:24:14.112643+0000","last_deep_scrub":"110'68","last_deep_scrub_stamp":"2024-01-15T18:24:14.112643+0000","last_clean_scrub_stamp":"2024-01-15T18:24:14.112643+0000","objects_scrubbed":21,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:08:51.346952+0000","scrub_duration":0.15134489700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":24578,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"110'50","reported_seq":83,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:50.982646+0000","last_change":"2024-01-15T18:23:50.982684+0000","last_active":"2024-01-15T18:23:50.982646+0000","last_peered":"2024-01-15T18:23:50.982646+0000","last_clean":"2024-01-15T18:23:50.982646+0000","last_became_active":"2024-01-15T18:18:33.159557+0000","last_became_peered":"2024-01-15T18:18:33.159557+0000","last_unstale":"2024-01-15T18:23:50.982646+0000","last_undegraded":"2024-01-15T18:23:50.982646+0000","last_fullsized":"2024-01-15T18:23:50.982646+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'50","last_scrub_stamp":"2024-01-15T18:23:50.982553+0000","last_deep_scrub":"110'50","last_deep_scrub_stamp":"2024-01-15T18:23:50.982553+0000","last_clean_scrub_stamp":"2024-01-15T18:23:50.982553+0000","objects_scrubbed":15,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:10:19.036866+0000","scrub_duration":0.10536667600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"110'61","reported_seq":92,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:03.046163+0000","last_change":"2024-01-15T18:24:03.046204+0000","last_active":"2024-01-15T18:24:03.046163+0000","last_peered":"2024-01-15T18:24:03.046163+0000","last_clean":"2024-01-15T18:24:03.046163+0000","last_became_active":"2024-01-15T18:18:33.165600+0000","last_became_peered":"2024-01-15T18:18:33.165600+0000","last_unstale":"2024-01-15T18:24:03.046163+0000","last_undegraded":"2024-01-15T18:24:03.046163+0000","last_fullsized":"2024-01-15T18:24:03.046163+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'61","last_scrub_stamp":"2024-01-15T18:24:03.046071+0000","last_deep_scrub":"110'61","last_deep_scrub_stamp":"2024-01-15T18:24:03.046071+0000","last_clean_scrub_stamp":"2024-01-15T18:24:03.046071+0000","objects_scrubbed":18,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:10:31.100384+0000","scrub_duration":0.11690018100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"110'56","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:10.038734+0000","last_change":"2024-01-15T18:24:10.038767+0000","last_active":"2024-01-15T18:24:10.038734+0000","last_peered":"2024-01-15T18:24:10.038734+0000","last_clean":"2024-01-15T18:24:10.038734+0000","last_became_active":"2024-01-15T18:18:33.168349+0000","last_became_peered":"2024-01-15T18:18:33.168349+0000","last_unstale":"2024-01-15T18:24:10.038734+0000","last_undegraded":"2024-01-15T18:24:10.038734+0000","last_fullsized":"2024-01-15T18:24:10.038734+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'56","last_scrub_stamp":"2024-01-15T18:24:10.038661+0000","last_deep_scrub":"110'56","last_deep_scrub_stamp":"2024-01-15T18:24:10.038661+0000","last_clean_scrub_stamp":"2024-01-15T18:24:10.038661+0000","objects_scrubbed":18,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:47:47.509239+0000","scrub_duration":0.12652001600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"110'53","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.034275+0000","last_change":"2024-01-15T18:18:33.159262+0000","last_active":"2024-01-15T18:23:32.034275+0000","last_peered":"2024-01-15T18:23:32.034275+0000","last_clean":"2024-01-15T18:23:32.034275+0000","last_became_active":"2024-01-15T18:18:33.158946+0000","last_became_peered":"2024-01-15T18:18:33.158946+0000","last_unstale":"2024-01-15T18:23:32.034275+0000","last_undegraded":"2024-01-15T18:23:32.034275+0000","last_fullsized":"2024-01-15T18:23:32.034275+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"110'58","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:06.379649+0000","last_change":"2024-01-15T18:24:06.379683+0000","last_active":"2024-01-15T18:24:06.379649+0000","last_peered":"2024-01-15T18:24:06.379649+0000","last_clean":"2024-01-15T18:24:06.379649+0000","last_became_active":"2024-01-15T18:18:33.161463+0000","last_became_peered":"2024-01-15T18:18:33.161463+0000","last_unstale":"2024-01-15T18:24:06.379649+0000","last_undegraded":"2024-01-15T18:24:06.379649+0000","last_fullsized":"2024-01-15T18:24:06.379649+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'58","last_scrub_stamp":"2024-01-15T18:24:06.379571+0000","last_deep_scrub":"110'58","last_deep_scrub_stamp":"2024-01-15T18:24:06.379571+0000","last_clean_scrub_stamp":"2024-01-15T18:24:06.379571+0000","objects_scrubbed":21,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:37:14.239420+0000","scrub_duration":0.134830067,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886260,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":28673,"num_write":21,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"110'55","reported_seq":82,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:07.024489+0000","last_change":"2024-01-15T18:24:07.024520+0000","last_active":"2024-01-15T18:24:07.024489+0000","last_peered":"2024-01-15T18:24:07.024489+0000","last_clean":"2024-01-15T18:24:07.024489+0000","last_became_active":"2024-01-15T18:18:33.162669+0000","last_became_peered":"2024-01-15T18:18:33.162669+0000","last_unstale":"2024-01-15T18:24:07.024489+0000","last_undegraded":"2024-01-15T18:24:07.024489+0000","last_fullsized":"2024-01-15T18:24:07.024489+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'55","last_scrub_stamp":"2024-01-15T18:24:07.024419+0000","last_deep_scrub":"110'55","last_deep_scrub_stamp":"2024-01-15T18:24:07.024419+0000","last_clean_scrub_stamp":"2024-01-15T18:24:07.024419+0000","objects_scrubbed":17,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:23:11.658669+0000","scrub_duration":0.12702018900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"110'72","reported_seq":106,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:14.039459+0000","last_change":"2024-01-15T18:24:14.039500+0000","last_active":"2024-01-15T18:24:14.039459+0000","last_peered":"2024-01-15T18:24:14.039459+0000","last_clean":"2024-01-15T18:24:14.039459+0000","last_became_active":"2024-01-15T18:18:33.160099+0000","last_became_peered":"2024-01-15T18:18:33.160099+0000","last_unstale":"2024-01-15T18:24:14.039459+0000","last_undegraded":"2024-01-15T18:24:14.039459+0000","last_fullsized":"2024-01-15T18:24:14.039459+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'72","last_scrub_stamp":"2024-01-15T18:24:14.039336+0000","last_deep_scrub":"110'72","last_deep_scrub_stamp":"2024-01-15T18:24:14.039336+0000","last_clean_scrub_stamp":"2024-01-15T18:24:14.039336+0000","objects_scrubbed":26,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:00:56.503350+0000","scrub_duration":0.19647456399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"110'57","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:12.460267+0000","last_change":"2024-01-15T18:24:12.460306+0000","last_active":"2024-01-15T18:24:12.460267+0000","last_peered":"2024-01-15T18:24:12.460267+0000","last_clean":"2024-01-15T18:24:12.460267+0000","last_became_active":"2024-01-15T18:18:33.160725+0000","last_became_peered":"2024-01-15T18:18:33.160725+0000","last_unstale":"2024-01-15T18:24:12.460267+0000","last_undegraded":"2024-01-15T18:24:12.460267+0000","last_fullsized":"2024-01-15T18:24:12.460267+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'57","last_scrub_stamp":"2024-01-15T18:24:12.460185+0000","last_deep_scrub":"110'57","last_deep_scrub_stamp":"2024-01-15T18:24:12.460185+0000","last_clean_scrub_stamp":"2024-01-15T18:24:12.460185+0000","objects_scrubbed":16,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:02:30.578846+0000","scrub_duration":0.105574268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"110'85","reported_seq":122,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:40.714947+0000","last_change":"2024-01-15T18:23:40.714970+0000","last_active":"2024-01-15T18:23:40.714947+0000","last_peered":"2024-01-15T18:23:40.714947+0000","last_clean":"2024-01-15T18:23:40.714947+0000","last_became_active":"2024-01-15T18:18:33.158701+0000","last_became_peered":"2024-01-15T18:18:33.158701+0000","last_unstale":"2024-01-15T18:23:40.714947+0000","last_undegraded":"2024-01-15T18:23:40.714947+0000","last_fullsized":"2024-01-15T18:23:40.714947+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'85","last_scrub_stamp":"2024-01-15T18:23:40.714867+0000","last_deep_scrub":"110'85","last_deep_scrub_stamp":"2024-01-15T18:23:40.714867+0000","last_clean_scrub_stamp":"2024-01-15T18:23:40.714867+0000","objects_scrubbed":26,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:21.790890+0000","scrub_duration":0.16315633500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"110'76","reported_seq":107,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:03.367042+0000","last_change":"2024-01-15T18:24:03.367103+0000","last_active":"2024-01-15T18:24:03.367042+0000","last_peered":"2024-01-15T18:24:03.367042+0000","last_clean":"2024-01-15T18:24:03.367042+0000","last_became_active":"2024-01-15T18:18:33.151250+0000","last_became_peered":"2024-01-15T18:18:33.151250+0000","last_unstale":"2024-01-15T18:24:03.367042+0000","last_undegraded":"2024-01-15T18:24:03.367042+0000","last_fullsized":"2024-01-15T18:24:03.367042+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'76","last_scrub_stamp":"2024-01-15T18:24:03.366959+0000","last_deep_scrub":"110'76","last_deep_scrub_stamp":"2024-01-15T18:24:03.366959+0000","last_clean_scrub_stamp":"2024-01-15T18:24:03.366959+0000","objects_scrubbed":22,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:07:35.792913+0000","scrub_duration":0.156425128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"110'49","reported_seq":79,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:05.312754+0000","last_change":"2024-01-15T18:24:05.312788+0000","last_active":"2024-01-15T18:24:05.312754+0000","last_peered":"2024-01-15T18:24:05.312754+0000","last_clean":"2024-01-15T18:24:05.312754+0000","last_became_active":"2024-01-15T18:18:33.154569+0000","last_became_peered":"2024-01-15T18:18:33.154569+0000","last_unstale":"2024-01-15T18:24:05.312754+0000","last_undegraded":"2024-01-15T18:24:05.312754+0000","last_fullsized":"2024-01-15T18:24:05.312754+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'49","last_scrub_stamp":"2024-01-15T18:24:05.312682+0000","last_deep_scrub":"110'49","last_deep_scrub_stamp":"2024-01-15T18:24:05.312682+0000","last_clean_scrub_stamp":"2024-01-15T18:24:05.312682+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:06:44.457178+0000","scrub_duration":0.10915641800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"110'6","reported_seq":31,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:52.319457+0000","last_change":"2024-01-15T18:23:52.319495+0000","last_active":"2024-01-15T18:23:52.319457+0000","last_peered":"2024-01-15T18:23:52.319457+0000","last_clean":"2024-01-15T18:23:52.319457+0000","last_became_active":"2024-01-15T18:18:32.216337+0000","last_became_peered":"2024-01-15T18:18:32.216337+0000","last_unstale":"2024-01-15T18:23:52.319457+0000","last_undegraded":"2024-01-15T18:23:52.319457+0000","last_fullsized":"2024-01-15T18:23:52.319457+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'6","last_scrub_stamp":"2024-01-15T18:23:52.319346+0000","last_deep_scrub":"110'6","last_deep_scrub_stamp":"2024-01-15T18:23:52.319346+0000","last_clean_scrub_stamp":"2024-01-15T18:23:52.319346+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:00:46.740483+0000","scrub_duration":0.003716685,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"110'66","reported_seq":98,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:57.637128+0000","last_change":"2024-01-15T18:23:57.637165+0000","last_active":"2024-01-15T18:23:57.637128+0000","last_peered":"2024-01-15T18:23:57.637128+0000","last_clean":"2024-01-15T18:23:57.637128+0000","last_became_active":"2024-01-15T18:18:33.166726+0000","last_became_peered":"2024-01-15T18:18:33.166726+0000","last_unstale":"2024-01-15T18:23:57.637128+0000","last_undegraded":"2024-01-15T18:23:57.637128+0000","last_fullsized":"2024-01-15T18:23:57.637128+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'66","last_scrub_stamp":"2024-01-15T18:23:57.637053+0000","last_deep_scrub":"110'66","last_deep_scrub_stamp":"2024-01-15T18:23:57.637053+0000","last_clean_scrub_stamp":"2024-01-15T18:23:57.637053+0000","objects_scrubbed":22,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:28:55.118910+0000","scrub_duration":0.15067504100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"107'1","reported_seq":26,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:49.184521+0000","last_change":"2024-01-15T18:23:49.184542+0000","last_active":"2024-01-15T18:23:49.184521+0000","last_peered":"2024-01-15T18:23:49.184521+0000","last_clean":"2024-01-15T18:23:49.184521+0000","last_became_active":"2024-01-15T18:18:32.219596+0000","last_became_peered":"2024-01-15T18:18:32.219596+0000","last_unstale":"2024-01-15T18:23:49.184521+0000","last_undegraded":"2024-01-15T18:23:49.184521+0000","last_fullsized":"2024-01-15T18:23:49.184521+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'1","last_scrub_stamp":"2024-01-15T18:23:49.184382+0000","last_deep_scrub":"107'1","last_deep_scrub_stamp":"2024-01-15T18:23:49.184382+0000","last_clean_scrub_stamp":"2024-01-15T18:23:49.184382+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:22:53.818633+0000","scrub_duration":0.0021746040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"110'59","reported_seq":94,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:57.284418+0000","last_change":"2024-01-15T18:23:57.284475+0000","last_active":"2024-01-15T18:23:57.284418+0000","last_peered":"2024-01-15T18:23:57.284418+0000","last_clean":"2024-01-15T18:23:57.284418+0000","last_became_active":"2024-01-15T18:18:33.150429+0000","last_became_peered":"2024-01-15T18:18:33.150429+0000","last_unstale":"2024-01-15T18:23:57.284418+0000","last_undegraded":"2024-01-15T18:23:57.284418+0000","last_fullsized":"2024-01-15T18:23:57.284418+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'59","last_scrub_stamp":"2024-01-15T18:23:57.284332+0000","last_deep_scrub":"110'59","last_deep_scrub_stamp":"2024-01-15T18:23:57.284332+0000","last_clean_scrub_stamp":"2024-01-15T18:23:57.284332+0000","objects_scrubbed":19,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:18:31.335937+0000","scrub_duration":0.13278458600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"110'4","reported_seq":29,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:52.947456+0000","last_change":"2024-01-15T18:23:52.947480+0000","last_active":"2024-01-15T18:23:52.947456+0000","last_peered":"2024-01-15T18:23:52.947456+0000","last_clean":"2024-01-15T18:23:52.947456+0000","last_became_active":"2024-01-15T18:18:32.212071+0000","last_became_peered":"2024-01-15T18:18:32.212071+0000","last_unstale":"2024-01-15T18:23:52.947456+0000","last_undegraded":"2024-01-15T18:23:52.947456+0000","last_fullsized":"2024-01-15T18:23:52.947456+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'4","last_scrub_stamp":"2024-01-15T18:23:52.947355+0000","last_deep_scrub":"110'4","last_deep_scrub_stamp":"2024-01-15T18:23:52.947355+0000","last_clean_scrub_stamp":"2024-01-15T18:23:52.947355+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:46.837594+0000","scrub_duration":0.0037854780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"110'56","reported_seq":83,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:35.193344+0000","last_change":"2024-01-15T18:18:33.165545+0000","last_active":"2024-01-15T18:23:35.193344+0000","last_peered":"2024-01-15T18:23:35.193344+0000","last_clean":"2024-01-15T18:23:35.193344+0000","last_became_active":"2024-01-15T18:18:33.165258+0000","last_became_peered":"2024-01-15T18:18:33.165258+0000","last_unstale":"2024-01-15T18:23:35.193344+0000","last_undegraded":"2024-01-15T18:23:35.193344+0000","last_fullsized":"2024-01-15T18:23:35.193344+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"110'3","reported_seq":28,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:56.916317+0000","last_change":"2024-01-15T18:23:56.916341+0000","last_active":"2024-01-15T18:23:56.916317+0000","last_peered":"2024-01-15T18:23:56.916317+0000","last_clean":"2024-01-15T18:23:56.916317+0000","last_became_active":"2024-01-15T18:18:32.216492+0000","last_became_peered":"2024-01-15T18:18:32.216492+0000","last_unstale":"2024-01-15T18:23:56.916317+0000","last_undegraded":"2024-01-15T18:23:56.916317+0000","last_fullsized":"2024-01-15T18:23:56.916317+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'3","last_scrub_stamp":"2024-01-15T18:23:56.916206+0000","last_deep_scrub":"110'3","last_deep_scrub_stamp":"2024-01-15T18:23:56.916206+0000","last_clean_scrub_stamp":"2024-01-15T18:23:56.916206+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:52:35.385555+0000","scrub_duration":0.0031224619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"110'56","reported_seq":88,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:04.618644+0000","last_change":"2024-01-15T18:24:04.618679+0000","last_active":"2024-01-15T18:24:04.618644+0000","last_peered":"2024-01-15T18:24:04.618644+0000","last_clean":"2024-01-15T18:24:04.618644+0000","last_became_active":"2024-01-15T18:18:33.157687+0000","last_became_peered":"2024-01-15T18:18:33.157687+0000","last_unstale":"2024-01-15T18:24:04.618644+0000","last_undegraded":"2024-01-15T18:24:04.618644+0000","last_fullsized":"2024-01-15T18:24:04.618644+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'56","last_scrub_stamp":"2024-01-15T18:24:04.618572+0000","last_deep_scrub":"110'56","last_deep_scrub_stamp":"2024-01-15T18:24:04.618572+0000","last_clean_scrub_stamp":"2024-01-15T18:24:04.618572+0000","objects_scrubbed":17,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:14:20.064593+0000","scrub_duration":0.11968945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"107'1","reported_seq":26,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:56.330972+0000","last_change":"2024-01-15T18:23:56.330992+0000","last_active":"2024-01-15T18:23:56.330972+0000","last_peered":"2024-01-15T18:23:56.330972+0000","last_clean":"2024-01-15T18:23:56.330972+0000","last_became_active":"2024-01-15T18:18:32.214453+0000","last_became_peered":"2024-01-15T18:18:32.214453+0000","last_unstale":"2024-01-15T18:23:56.330972+0000","last_undegraded":"2024-01-15T18:23:56.330972+0000","last_fullsized":"2024-01-15T18:23:56.330972+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'1","last_scrub_stamp":"2024-01-15T18:23:56.330895+0000","last_deep_scrub":"107'1","last_deep_scrub_stamp":"2024-01-15T18:23:56.330895+0000","last_clean_scrub_stamp":"2024-01-15T18:23:56.330895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:22:46.630675+0000","scrub_duration":0.0018859580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"110'73","reported_seq":110,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:13.026892+0000","last_change":"2024-01-15T18:24:13.026930+0000","last_active":"2024-01-15T18:24:13.026892+0000","last_peered":"2024-01-15T18:24:13.026892+0000","last_clean":"2024-01-15T18:24:13.026892+0000","last_became_active":"2024-01-15T18:18:33.153058+0000","last_became_peered":"2024-01-15T18:18:33.153058+0000","last_unstale":"2024-01-15T18:24:13.026892+0000","last_undegraded":"2024-01-15T18:24:13.026892+0000","last_fullsized":"2024-01-15T18:24:13.026892+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'73","last_scrub_stamp":"2024-01-15T18:24:13.026816+0000","last_deep_scrub":"110'73","last_deep_scrub_stamp":"2024-01-15T18:24:13.026816+0000","last_clean_scrub_stamp":"2024-01-15T18:24:13.026816+0000","objects_scrubbed":24,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:43:03.776881+0000","scrub_duration":0.14808554800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"107'1","reported_seq":26,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:42.849182+0000","last_change":"2024-01-15T18:23:42.849208+0000","last_active":"2024-01-15T18:23:42.849182+0000","last_peered":"2024-01-15T18:23:42.849182+0000","last_clean":"2024-01-15T18:23:42.849182+0000","last_became_active":"2024-01-15T18:18:32.215417+0000","last_became_peered":"2024-01-15T18:18:32.215417+0000","last_unstale":"2024-01-15T18:23:42.849182+0000","last_undegraded":"2024-01-15T18:23:42.849182+0000","last_fullsized":"2024-01-15T18:23:42.849182+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'1","last_scrub_stamp":"2024-01-15T18:23:42.849065+0000","last_deep_scrub":"107'1","last_deep_scrub_stamp":"2024-01-15T18:23:42.849065+0000","last_clean_scrub_stamp":"2024-01-15T18:23:42.849065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:40:12.007683+0000","scrub_duration":0.0030182830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"110'79","reported_seq":110,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:11.111901+0000","last_change":"2024-01-15T18:24:11.111943+0000","last_active":"2024-01-15T18:24:11.111901+0000","last_peered":"2024-01-15T18:24:11.111901+0000","last_clean":"2024-01-15T18:24:11.111901+0000","last_became_active":"2024-01-15T18:18:43.024221+0000","last_became_peered":"2024-01-15T18:18:43.024221+0000","last_unstale":"2024-01-15T18:24:11.111901+0000","last_undegraded":"2024-01-15T18:24:11.111901+0000","last_fullsized":"2024-01-15T18:24:11.111901+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'79","last_scrub_stamp":"2024-01-15T18:24:11.111818+0000","last_deep_scrub":"110'79","last_deep_scrub_stamp":"2024-01-15T18:24:11.111818+0000","last_clean_scrub_stamp":"2024-01-15T18:24:11.111818+0000","objects_scrubbed":26,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:18:45.163423+0000","scrub_duration":0.18158712699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857792,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:41.955122+0000","last_change":"2024-01-15T18:23:41.955145+0000","last_active":"2024-01-15T18:23:41.955122+0000","last_peered":"2024-01-15T18:23:41.955122+0000","last_clean":"2024-01-15T18:23:41.955122+0000","last_became_active":"2024-01-15T18:18:32.215465+0000","last_became_peered":"2024-01-15T18:18:32.215465+0000","last_unstale":"2024-01-15T18:23:41.955122+0000","last_undegraded":"2024-01-15T18:23:41.955122+0000","last_fullsized":"2024-01-15T18:23:41.955122+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:41.954982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:41.954982+0000","last_clean_scrub_stamp":"2024-01-15T18:23:41.954982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:42:52.821931+0000","scrub_duration":0.002690853,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"110'59","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:07.422578+0000","last_change":"2024-01-15T18:24:07.422633+0000","last_active":"2024-01-15T18:24:07.422578+0000","last_peered":"2024-01-15T18:24:07.422578+0000","last_clean":"2024-01-15T18:24:07.422578+0000","last_became_active":"2024-01-15T18:18:33.153610+0000","last_became_peered":"2024-01-15T18:18:33.153610+0000","last_unstale":"2024-01-15T18:24:07.422578+0000","last_undegraded":"2024-01-15T18:24:07.422578+0000","last_fullsized":"2024-01-15T18:24:07.422578+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'59","last_scrub_stamp":"2024-01-15T18:24:07.422480+0000","last_deep_scrub":"110'59","last_deep_scrub_stamp":"2024-01-15T18:24:07.422480+0000","last_clean_scrub_stamp":"2024-01-15T18:24:07.422480+0000","objects_scrubbed":16,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:44:16.329452+0000","scrub_duration":0.11664002800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:50.513004+0000","last_change":"2024-01-15T18:23:50.513024+0000","last_active":"2024-01-15T18:23:50.513004+0000","last_peered":"2024-01-15T18:23:50.513004+0000","last_clean":"2024-01-15T18:23:50.513004+0000","last_became_active":"2024-01-15T18:18:32.219555+0000","last_became_peered":"2024-01-15T18:18:32.219555+0000","last_unstale":"2024-01-15T18:23:50.513004+0000","last_undegraded":"2024-01-15T18:23:50.513004+0000","last_fullsized":"2024-01-15T18:23:50.513004+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:50.512904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:50.512904+0000","last_clean_scrub_stamp":"2024-01-15T18:23:50.512904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:03:40.746495+0000","scrub_duration":0.0020306230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"110'50","reported_seq":78,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:05.416297+0000","last_change":"2024-01-15T18:24:05.416324+0000","last_active":"2024-01-15T18:24:05.416297+0000","last_peered":"2024-01-15T18:24:05.416297+0000","last_clean":"2024-01-15T18:24:05.416297+0000","last_became_active":"2024-01-15T18:18:33.155765+0000","last_became_peered":"2024-01-15T18:18:33.155765+0000","last_unstale":"2024-01-15T18:24:05.416297+0000","last_undegraded":"2024-01-15T18:24:05.416297+0000","last_fullsized":"2024-01-15T18:24:05.416297+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'50","last_scrub_stamp":"2024-01-15T18:24:05.416219+0000","last_deep_scrub":"110'50","last_deep_scrub_stamp":"2024-01-15T18:24:05.416219+0000","last_clean_scrub_stamp":"2024-01-15T18:24:05.416219+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:10:03.756727+0000","scrub_duration":0.058782918000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"110'1","reported_seq":26,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:54.901556+0000","last_change":"2024-01-15T18:23:54.901577+0000","last_active":"2024-01-15T18:23:54.901556+0000","last_peered":"2024-01-15T18:23:54.901556+0000","last_clean":"2024-01-15T18:23:54.901556+0000","last_became_active":"2024-01-15T18:18:32.212511+0000","last_became_peered":"2024-01-15T18:18:32.212511+0000","last_unstale":"2024-01-15T18:23:54.901556+0000","last_undegraded":"2024-01-15T18:23:54.901556+0000","last_fullsized":"2024-01-15T18:23:54.901556+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1","last_scrub_stamp":"2024-01-15T18:23:54.901458+0000","last_deep_scrub":"110'1","last_deep_scrub_stamp":"2024-01-15T18:23:54.901458+0000","last_clean_scrub_stamp":"2024-01-15T18:23:54.901458+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:49:11.641921+0000","scrub_duration":0.0018983540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"110'53","reported_seq":82,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:51.617489+0000","last_change":"2024-01-15T18:23:51.617525+0000","last_active":"2024-01-15T18:23:51.617489+0000","last_peered":"2024-01-15T18:23:51.617489+0000","last_clean":"2024-01-15T18:23:51.617489+0000","last_became_active":"2024-01-15T18:18:33.164530+0000","last_became_peered":"2024-01-15T18:18:33.164530+0000","last_unstale":"2024-01-15T18:23:51.617489+0000","last_undegraded":"2024-01-15T18:23:51.617489+0000","last_fullsized":"2024-01-15T18:23:51.617489+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'53","last_scrub_stamp":"2024-01-15T18:23:51.617414+0000","last_deep_scrub":"110'53","last_deep_scrub_stamp":"2024-01-15T18:23:51.617414+0000","last_clean_scrub_stamp":"2024-01-15T18:23:51.617414+0000","objects_scrubbed":18,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:57:57.307137+0000","scrub_duration":0.11477269399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:55.893196+0000","last_change":"2024-01-15T18:23:55.893219+0000","last_active":"2024-01-15T18:23:55.893196+0000","last_peered":"2024-01-15T18:23:55.893196+0000","last_clean":"2024-01-15T18:23:55.893196+0000","last_became_active":"2024-01-15T18:18:32.216337+0000","last_became_peered":"2024-01-15T18:18:32.216337+0000","last_unstale":"2024-01-15T18:23:55.893196+0000","last_undegraded":"2024-01-15T18:23:55.893196+0000","last_fullsized":"2024-01-15T18:23:55.893196+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:55.893099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:55.893099+0000","last_clean_scrub_stamp":"2024-01-15T18:23:55.893099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:45:26.429594+0000","scrub_duration":0.002804001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"110'65","reported_seq":100,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:15.115218+0000","last_change":"2024-01-15T18:24:15.115251+0000","last_active":"2024-01-15T18:24:15.115218+0000","last_peered":"2024-01-15T18:24:15.115218+0000","last_clean":"2024-01-15T18:24:15.115218+0000","last_became_active":"2024-01-15T18:18:33.148016+0000","last_became_peered":"2024-01-15T18:18:33.148016+0000","last_unstale":"2024-01-15T18:24:15.115218+0000","last_undegraded":"2024-01-15T18:24:15.115218+0000","last_fullsized":"2024-01-15T18:24:15.115218+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'65","last_scrub_stamp":"2024-01-15T18:24:15.115148+0000","last_deep_scrub":"110'65","last_deep_scrub_stamp":"2024-01-15T18:24:15.115148+0000","last_clean_scrub_stamp":"2024-01-15T18:24:15.115148+0000","objects_scrubbed":22,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:23:51.814230+0000","scrub_duration":0.13686477899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"110'14","reported_seq":39,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:55.542471+0000","last_change":"2024-01-15T18:23:55.542525+0000","last_active":"2024-01-15T18:23:55.542471+0000","last_peered":"2024-01-15T18:23:55.542471+0000","last_clean":"2024-01-15T18:23:55.542471+0000","last_became_active":"2024-01-15T18:18:32.214186+0000","last_became_peered":"2024-01-15T18:18:32.214186+0000","last_unstale":"2024-01-15T18:23:55.542471+0000","last_undegraded":"2024-01-15T18:23:55.542471+0000","last_fullsized":"2024-01-15T18:23:55.542471+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'14","last_scrub_stamp":"2024-01-15T18:23:55.542363+0000","last_deep_scrub":"110'14","last_deep_scrub_stamp":"2024-01-15T18:23:55.542363+0000","last_clean_scrub_stamp":"2024-01-15T18:23:55.542363+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:36:43.100265+0000","scrub_duration":0.0043649800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"110'63","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:09.440301+0000","last_change":"2024-01-15T18:24:09.440341+0000","last_active":"2024-01-15T18:24:09.440301+0000","last_peered":"2024-01-15T18:24:09.440301+0000","last_clean":"2024-01-15T18:24:09.440301+0000","last_became_active":"2024-01-15T18:18:33.158541+0000","last_became_peered":"2024-01-15T18:18:33.158541+0000","last_unstale":"2024-01-15T18:24:09.440301+0000","last_undegraded":"2024-01-15T18:24:09.440301+0000","last_fullsized":"2024-01-15T18:24:09.440301+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'63","last_scrub_stamp":"2024-01-15T18:24:09.440226+0000","last_deep_scrub":"110'63","last_deep_scrub_stamp":"2024-01-15T18:24:09.440226+0000","last_clean_scrub_stamp":"2024-01-15T18:24:09.440226+0000","objects_scrubbed":20,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:33:00.283608+0000","scrub_duration":0.16288201199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"110'188","reported_seq":213,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:53.472690+0000","last_change":"2024-01-15T18:23:53.472714+0000","last_active":"2024-01-15T18:23:53.472690+0000","last_peered":"2024-01-15T18:23:53.472690+0000","last_clean":"2024-01-15T18:23:53.472690+0000","last_became_active":"2024-01-15T18:18:32.218002+0000","last_became_peered":"2024-01-15T18:18:32.218002+0000","last_unstale":"2024-01-15T18:23:53.472690+0000","last_undegraded":"2024-01-15T18:23:53.472690+0000","last_fullsized":"2024-01-15T18:23:53.472690+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'188","last_scrub_stamp":"2024-01-15T18:23:53.472606+0000","last_deep_scrub":"110'188","last_deep_scrub_stamp":"2024-01-15T18:23:53.472606+0000","last_clean_scrub_stamp":"2024-01-15T18:23:53.472606+0000","objects_scrubbed":4,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:11:55.352315+0000","scrub_duration":0.0071479380000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":614075,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":185,"num_write_kb":681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"110'65","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:05.066433+0000","last_change":"2024-01-15T18:24:05.066465+0000","last_active":"2024-01-15T18:24:05.066433+0000","last_peered":"2024-01-15T18:24:05.066433+0000","last_clean":"2024-01-15T18:24:05.066433+0000","last_became_active":"2024-01-15T18:18:33.151132+0000","last_became_peered":"2024-01-15T18:18:33.151132+0000","last_unstale":"2024-01-15T18:24:05.066433+0000","last_undegraded":"2024-01-15T18:24:05.066433+0000","last_fullsized":"2024-01-15T18:24:05.066433+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'65","last_scrub_stamp":"2024-01-15T18:24:05.066365+0000","last_deep_scrub":"110'65","last_deep_scrub_stamp":"2024-01-15T18:24:05.066365+0000","last_clean_scrub_stamp":"2024-01-15T18:24:05.066365+0000","objects_scrubbed":21,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:22:28.584664+0000","scrub_duration":0.13404809200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"110'4","reported_seq":29,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:42.234836+0000","last_change":"2024-01-15T18:23:42.234860+0000","last_active":"2024-01-15T18:23:42.234836+0000","last_peered":"2024-01-15T18:23:42.234836+0000","last_clean":"2024-01-15T18:23:42.234836+0000","last_became_active":"2024-01-15T18:18:32.215073+0000","last_became_peered":"2024-01-15T18:18:32.215073+0000","last_unstale":"2024-01-15T18:23:42.234836+0000","last_undegraded":"2024-01-15T18:23:42.234836+0000","last_fullsized":"2024-01-15T18:23:42.234836+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'4","last_scrub_stamp":"2024-01-15T18:23:42.234754+0000","last_deep_scrub":"110'4","last_deep_scrub_stamp":"2024-01-15T18:23:42.234754+0000","last_clean_scrub_stamp":"2024-01-15T18:23:42.234754+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:48:58.975217+0000","scrub_duration":0.004651957,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"110'73","reported_seq":102,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:06.069897+0000","last_change":"2024-01-15T18:24:06.069934+0000","last_active":"2024-01-15T18:24:06.069897+0000","last_peered":"2024-01-15T18:24:06.069897+0000","last_clean":"2024-01-15T18:24:06.069897+0000","last_became_active":"2024-01-15T18:18:33.161493+0000","last_became_peered":"2024-01-15T18:18:33.161493+0000","last_unstale":"2024-01-15T18:24:06.069897+0000","last_undegraded":"2024-01-15T18:24:06.069897+0000","last_fullsized":"2024-01-15T18:24:06.069897+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'73","last_scrub_stamp":"2024-01-15T18:24:06.069802+0000","last_deep_scrub":"110'73","last_deep_scrub_stamp":"2024-01-15T18:24:06.069802+0000","last_clean_scrub_stamp":"2024-01-15T18:24:06.069802+0000","objects_scrubbed":21,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:28.365170+0000","scrub_duration":0.14167099799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:43.624818+0000","last_change":"2024-01-15T18:23:43.624838+0000","last_active":"2024-01-15T18:23:43.624818+0000","last_peered":"2024-01-15T18:23:43.624818+0000","last_clean":"2024-01-15T18:23:43.624818+0000","last_became_active":"2024-01-15T18:18:32.214942+0000","last_became_peered":"2024-01-15T18:18:32.214942+0000","last_unstale":"2024-01-15T18:23:43.624818+0000","last_undegraded":"2024-01-15T18:23:43.624818+0000","last_fullsized":"2024-01-15T18:23:43.624818+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:43.624692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:43.624692+0000","last_clean_scrub_stamp":"2024-01-15T18:23:43.624692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:36:51.484541+0000","scrub_duration":0.0025066049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"110'68","reported_seq":100,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:57.083832+0000","last_change":"2024-01-15T18:23:57.083851+0000","last_active":"2024-01-15T18:23:57.083832+0000","last_peered":"2024-01-15T18:23:57.083832+0000","last_clean":"2024-01-15T18:23:57.083832+0000","last_became_active":"2024-01-15T18:18:33.145973+0000","last_became_peered":"2024-01-15T18:18:33.145973+0000","last_unstale":"2024-01-15T18:23:57.083832+0000","last_undegraded":"2024-01-15T18:23:57.083832+0000","last_fullsized":"2024-01-15T18:23:57.083832+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'68","last_scrub_stamp":"2024-01-15T18:23:57.083768+0000","last_deep_scrub":"110'68","last_deep_scrub_stamp":"2024-01-15T18:23:57.083768+0000","last_clean_scrub_stamp":"2024-01-15T18:23:57.083768+0000","objects_scrubbed":25,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:22:28.677434+0000","scrub_duration":0.150822226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663557,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"110'12","reported_seq":37,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:38.931283+0000","last_change":"2024-01-15T18:23:38.931329+0000","last_active":"2024-01-15T18:23:38.931283+0000","last_peered":"2024-01-15T18:23:38.931283+0000","last_clean":"2024-01-15T18:23:38.931283+0000","last_became_active":"2024-01-15T18:18:32.219280+0000","last_became_peered":"2024-01-15T18:18:32.219280+0000","last_unstale":"2024-01-15T18:23:38.931283+0000","last_undegraded":"2024-01-15T18:23:38.931283+0000","last_fullsized":"2024-01-15T18:23:38.931283+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'12","last_scrub_stamp":"2024-01-15T18:23:38.931162+0000","last_deep_scrub":"110'12","last_deep_scrub_stamp":"2024-01-15T18:23:38.931162+0000","last_clean_scrub_stamp":"2024-01-15T18:23:38.931162+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:57:50.305539+0000","scrub_duration":0.0047879089999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"110'34","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:07.968432+0000","last_change":"2024-01-15T18:24:07.968469+0000","last_active":"2024-01-15T18:24:07.968432+0000","last_peered":"2024-01-15T18:24:07.968432+0000","last_clean":"2024-01-15T18:24:07.968432+0000","last_became_active":"2024-01-15T18:18:33.161329+0000","last_became_peered":"2024-01-15T18:18:33.161329+0000","last_unstale":"2024-01-15T18:24:07.968432+0000","last_undegraded":"2024-01-15T18:24:07.968432+0000","last_fullsized":"2024-01-15T18:24:07.968432+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'34","last_scrub_stamp":"2024-01-15T18:24:07.968352+0000","last_deep_scrub":"110'34","last_deep_scrub_stamp":"2024-01-15T18:24:07.968352+0000","last_clean_scrub_stamp":"2024-01-15T18:24:07.968352+0000","objects_scrubbed":10,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:12:38.215453+0000","scrub_duration":0.071078854999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"110'2","reported_seq":27,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:03.485991+0000","last_change":"2024-01-15T18:24:03.486011+0000","last_active":"2024-01-15T18:24:03.485991+0000","last_peered":"2024-01-15T18:24:03.485991+0000","last_clean":"2024-01-15T18:24:03.485991+0000","last_became_active":"2024-01-15T18:18:32.214009+0000","last_became_peered":"2024-01-15T18:18:32.214009+0000","last_unstale":"2024-01-15T18:24:03.485991+0000","last_undegraded":"2024-01-15T18:24:03.485991+0000","last_fullsized":"2024-01-15T18:24:03.485991+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2","last_scrub_stamp":"2024-01-15T18:24:03.485914+0000","last_deep_scrub":"110'2","last_deep_scrub_stamp":"2024-01-15T18:24:03.485914+0000","last_clean_scrub_stamp":"2024-01-15T18:24:03.485914+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:51:10.423070+0000","scrub_duration":0.0039431259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"110'52","reported_seq":87,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:59.256424+0000","last_change":"2024-01-15T18:23:59.256459+0000","last_active":"2024-01-15T18:23:59.256424+0000","last_peered":"2024-01-15T18:23:59.256424+0000","last_clean":"2024-01-15T18:23:59.256424+0000","last_became_active":"2024-01-15T18:18:33.162679+0000","last_became_peered":"2024-01-15T18:18:33.162679+0000","last_unstale":"2024-01-15T18:23:59.256424+0000","last_undegraded":"2024-01-15T18:23:59.256424+0000","last_fullsized":"2024-01-15T18:23:59.256424+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'52","last_scrub_stamp":"2024-01-15T18:23:59.256325+0000","last_deep_scrub":"110'52","last_deep_scrub_stamp":"2024-01-15T18:23:59.256325+0000","last_clean_scrub_stamp":"2024-01-15T18:23:59.256325+0000","objects_scrubbed":17,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:05:12.786176+0000","scrub_duration":0.11336687600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"110'3","reported_seq":28,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:43.939236+0000","last_change":"2024-01-15T18:23:43.939260+0000","last_active":"2024-01-15T18:23:43.939236+0000","last_peered":"2024-01-15T18:23:43.939236+0000","last_clean":"2024-01-15T18:23:43.939236+0000","last_became_active":"2024-01-15T18:18:32.214695+0000","last_became_peered":"2024-01-15T18:18:32.214695+0000","last_unstale":"2024-01-15T18:23:43.939236+0000","last_undegraded":"2024-01-15T18:23:43.939236+0000","last_fullsized":"2024-01-15T18:23:43.939236+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'3","last_scrub_stamp":"2024-01-15T18:23:43.939137+0000","last_deep_scrub":"110'3","last_deep_scrub_stamp":"2024-01-15T18:23:43.939137+0000","last_clean_scrub_stamp":"2024-01-15T18:23:43.939137+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:24:14.419417+0000","scrub_duration":0.0041419650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"110'44","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.746060+0000","last_change":"2024-01-15T18:18:33.168531+0000","last_active":"2024-01-15T18:23:31.746060+0000","last_peered":"2024-01-15T18:23:31.746060+0000","last_clean":"2024-01-15T18:23:31.746060+0000","last_became_active":"2024-01-15T18:18:33.168190+0000","last_became_peered":"2024-01-15T18:18:33.168190+0000","last_unstale":"2024-01-15T18:23:31.746060+0000","last_undegraded":"2024-01-15T18:23:31.746060+0000","last_fullsized":"2024-01-15T18:23:31.746060+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":11,"num_read_kb":28676,"num_write":22,"num_write_kb":73730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"110'2","reported_seq":27,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:47.483359+0000","last_change":"2024-01-15T18:23:47.483404+0000","last_active":"2024-01-15T18:23:47.483359+0000","last_peered":"2024-01-15T18:23:47.483359+0000","last_clean":"2024-01-15T18:23:47.483359+0000","last_became_active":"2024-01-15T18:18:32.220884+0000","last_became_peered":"2024-01-15T18:18:32.220884+0000","last_unstale":"2024-01-15T18:23:47.483359+0000","last_undegraded":"2024-01-15T18:23:47.483359+0000","last_fullsized":"2024-01-15T18:23:47.483359+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2","last_scrub_stamp":"2024-01-15T18:23:47.483235+0000","last_deep_scrub":"110'2","last_deep_scrub_stamp":"2024-01-15T18:23:47.483235+0000","last_clean_scrub_stamp":"2024-01-15T18:23:47.483235+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:48:48.450236+0000","scrub_duration":0.0045499140000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"110'41","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.898859+0000","last_change":"2024-01-15T18:18:33.163114+0000","last_active":"2024-01-15T18:23:31.898859+0000","last_peered":"2024-01-15T18:23:31.898859+0000","last_clean":"2024-01-15T18:23:31.898859+0000","last_became_active":"2024-01-15T18:18:33.162424+0000","last_became_peered":"2024-01-15T18:18:33.162424+0000","last_unstale":"2024-01-15T18:23:31.898859+0000","last_undegraded":"2024-01-15T18:23:31.898859+0000","last_fullsized":"2024-01-15T18:23:31.898859+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:44.345346+0000","last_change":"2024-01-15T18:23:44.345376+0000","last_active":"2024-01-15T18:23:44.345346+0000","last_peered":"2024-01-15T18:23:44.345346+0000","last_clean":"2024-01-15T18:23:44.345346+0000","last_became_active":"2024-01-15T18:18:32.216769+0000","last_became_peered":"2024-01-15T18:18:32.216769+0000","last_unstale":"2024-01-15T18:23:44.345346+0000","last_undegraded":"2024-01-15T18:23:44.345346+0000","last_fullsized":"2024-01-15T18:23:44.345346+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:44.345256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:44.345256+0000","last_clean_scrub_stamp":"2024-01-15T18:23:44.345256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:35:12.856197+0000","scrub_duration":0.0019326370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"110'75","reported_seq":108,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:06.470407+0000","last_change":"2024-01-15T18:24:06.470439+0000","last_active":"2024-01-15T18:24:06.470407+0000","last_peered":"2024-01-15T18:24:06.470407+0000","last_clean":"2024-01-15T18:24:06.470407+0000","last_became_active":"2024-01-15T18:18:33.162658+0000","last_became_peered":"2024-01-15T18:18:33.162658+0000","last_unstale":"2024-01-15T18:24:06.470407+0000","last_undegraded":"2024-01-15T18:24:06.470407+0000","last_fullsized":"2024-01-15T18:24:06.470407+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'75","last_scrub_stamp":"2024-01-15T18:24:06.470318+0000","last_deep_scrub":"110'75","last_deep_scrub_stamp":"2024-01-15T18:24:06.470318+0000","last_clean_scrub_stamp":"2024-01-15T18:24:06.470318+0000","objects_scrubbed":30,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:41:38.067402+0000","scrub_duration":0.20391261699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"110'4","reported_seq":29,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:48.233481+0000","last_change":"2024-01-15T18:23:48.233504+0000","last_active":"2024-01-15T18:23:48.233481+0000","last_peered":"2024-01-15T18:23:48.233481+0000","last_clean":"2024-01-15T18:23:48.233481+0000","last_became_active":"2024-01-15T18:18:32.207507+0000","last_became_peered":"2024-01-15T18:18:32.207507+0000","last_unstale":"2024-01-15T18:23:48.233481+0000","last_undegraded":"2024-01-15T18:23:48.233481+0000","last_fullsized":"2024-01-15T18:23:48.233481+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'4","last_scrub_stamp":"2024-01-15T18:23:48.233388+0000","last_deep_scrub":"110'4","last_deep_scrub_stamp":"2024-01-15T18:23:48.233388+0000","last_clean_scrub_stamp":"2024-01-15T18:23:48.233388+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:50.599913+0000","scrub_duration":0.0034152789999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"110'43","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:31.490371+0000","last_change":"2024-01-15T18:18:33.164981+0000","last_active":"2024-01-15T18:23:31.490371+0000","last_peered":"2024-01-15T18:23:31.490371+0000","last_clean":"2024-01-15T18:23:31.490371+0000","last_became_active":"2024-01-15T18:18:33.164206+0000","last_became_peered":"2024-01-15T18:18:33.164206+0000","last_unstale":"2024-01-15T18:23:31.490371+0000","last_undegraded":"2024-01-15T18:23:31.490371+0000","last_fullsized":"2024-01-15T18:23:31.490371+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:48.035163+0000","last_change":"2024-01-15T18:23:48.035184+0000","last_active":"2024-01-15T18:23:48.035163+0000","last_peered":"2024-01-15T18:23:48.035163+0000","last_clean":"2024-01-15T18:23:48.035163+0000","last_became_active":"2024-01-15T18:18:32.212201+0000","last_became_peered":"2024-01-15T18:18:32.212201+0000","last_unstale":"2024-01-15T18:23:48.035163+0000","last_undegraded":"2024-01-15T18:23:48.035163+0000","last_fullsized":"2024-01-15T18:23:48.035163+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:48.035058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:48.035058+0000","last_clean_scrub_stamp":"2024-01-15T18:23:48.035058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:49:40.411498+0000","scrub_duration":0.0017447160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"110'61","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:09.459306+0000","last_change":"2024-01-15T18:24:09.459337+0000","last_active":"2024-01-15T18:24:09.459306+0000","last_peered":"2024-01-15T18:24:09.459306+0000","last_clean":"2024-01-15T18:24:09.459306+0000","last_became_active":"2024-01-15T18:18:33.164708+0000","last_became_peered":"2024-01-15T18:18:33.164708+0000","last_unstale":"2024-01-15T18:24:09.459306+0000","last_undegraded":"2024-01-15T18:24:09.459306+0000","last_fullsized":"2024-01-15T18:24:09.459306+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'61","last_scrub_stamp":"2024-01-15T18:24:09.459228+0000","last_deep_scrub":"110'61","last_deep_scrub_stamp":"2024-01-15T18:24:09.459228+0000","last_clean_scrub_stamp":"2024-01-15T18:24:09.459228+0000","objects_scrubbed":17,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:00:51.923242+0000","scrub_duration":0.14356816999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:44.292443+0000","last_change":"2024-01-15T18:23:44.292477+0000","last_active":"2024-01-15T18:23:44.292443+0000","last_peered":"2024-01-15T18:23:44.292443+0000","last_clean":"2024-01-15T18:23:44.292443+0000","last_became_active":"2024-01-15T18:18:32.207599+0000","last_became_peered":"2024-01-15T18:18:32.207599+0000","last_unstale":"2024-01-15T18:23:44.292443+0000","last_undegraded":"2024-01-15T18:23:44.292443+0000","last_fullsized":"2024-01-15T18:23:44.292443+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:44.292288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:44.292288+0000","last_clean_scrub_stamp":"2024-01-15T18:23:44.292288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:52:22.761638+0000","scrub_duration":0.0021733669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"110'59","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:04.017805+0000","last_change":"2024-01-15T18:24:04.017845+0000","last_active":"2024-01-15T18:24:04.017805+0000","last_peered":"2024-01-15T18:24:04.017805+0000","last_clean":"2024-01-15T18:24:04.017805+0000","last_became_active":"2024-01-15T18:18:33.160350+0000","last_became_peered":"2024-01-15T18:18:33.160350+0000","last_unstale":"2024-01-15T18:24:04.017805+0000","last_undegraded":"2024-01-15T18:24:04.017805+0000","last_fullsized":"2024-01-15T18:24:04.017805+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'59","last_scrub_stamp":"2024-01-15T18:24:04.017732+0000","last_deep_scrub":"110'59","last_deep_scrub_stamp":"2024-01-15T18:24:04.017732+0000","last_clean_scrub_stamp":"2024-01-15T18:24:04.017732+0000","objects_scrubbed":17,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:29:01.499590+0000","scrub_duration":0.112111222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67109125,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":17,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:47.491435+0000","last_change":"2024-01-15T18:23:47.491458+0000","last_active":"2024-01-15T18:23:47.491435+0000","last_peered":"2024-01-15T18:23:47.491435+0000","last_clean":"2024-01-15T18:23:47.491435+0000","last_became_active":"2024-01-15T18:18:32.216581+0000","last_became_peered":"2024-01-15T18:18:32.216581+0000","last_unstale":"2024-01-15T18:23:47.491435+0000","last_undegraded":"2024-01-15T18:23:47.491435+0000","last_fullsized":"2024-01-15T18:23:47.491435+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:47.491336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:47.491336+0000","last_clean_scrub_stamp":"2024-01-15T18:23:47.491336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:06:23.583464+0000","scrub_duration":0.0018469020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"110'46","reported_seq":74,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:48.560733+0000","last_change":"2024-01-15T18:23:48.560769+0000","last_active":"2024-01-15T18:23:48.560733+0000","last_peered":"2024-01-15T18:23:48.560733+0000","last_clean":"2024-01-15T18:23:48.560733+0000","last_became_active":"2024-01-15T18:18:33.164152+0000","last_became_peered":"2024-01-15T18:18:33.164152+0000","last_unstale":"2024-01-15T18:23:48.560733+0000","last_undegraded":"2024-01-15T18:23:48.560733+0000","last_fullsized":"2024-01-15T18:23:48.560733+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'46","last_scrub_stamp":"2024-01-15T18:23:48.560650+0000","last_deep_scrub":"110'46","last_deep_scrub_stamp":"2024-01-15T18:23:48.560650+0000","last_clean_scrub_stamp":"2024-01-15T18:23:48.560650+0000","objects_scrubbed":13,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:42:57.049428+0000","scrub_duration":0.082496549000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"110'3","reported_seq":28,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:44.865576+0000","last_change":"2024-01-15T18:23:44.865597+0000","last_active":"2024-01-15T18:23:44.865576+0000","last_peered":"2024-01-15T18:23:44.865576+0000","last_clean":"2024-01-15T18:23:44.865576+0000","last_became_active":"2024-01-15T18:18:32.218858+0000","last_became_peered":"2024-01-15T18:18:32.218858+0000","last_unstale":"2024-01-15T18:23:44.865576+0000","last_undegraded":"2024-01-15T18:23:44.865576+0000","last_fullsized":"2024-01-15T18:23:44.865576+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'3","last_scrub_stamp":"2024-01-15T18:23:44.865505+0000","last_deep_scrub":"110'3","last_deep_scrub_stamp":"2024-01-15T18:23:44.865505+0000","last_clean_scrub_stamp":"2024-01-15T18:23:44.865505+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:03:03.103591+0000","scrub_duration":0.0027720739999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"110'63","reported_seq":99,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:16.773337+0000","last_change":"2024-01-15T18:24:16.773374+0000","last_active":"2024-01-15T18:24:16.773337+0000","last_peered":"2024-01-15T18:24:16.773337+0000","last_clean":"2024-01-15T18:24:16.773337+0000","last_became_active":"2024-01-15T18:18:33.160315+0000","last_became_peered":"2024-01-15T18:18:33.160315+0000","last_unstale":"2024-01-15T18:24:16.773337+0000","last_undegraded":"2024-01-15T18:24:16.773337+0000","last_fullsized":"2024-01-15T18:24:16.773337+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'63","last_scrub_stamp":"2024-01-15T18:24:16.773261+0000","last_deep_scrub":"110'63","last_deep_scrub_stamp":"2024-01-15T18:24:16.773261+0000","last_clean_scrub_stamp":"2024-01-15T18:24:16.773261+0000","objects_scrubbed":22,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:21:25.069967+0000","scrub_duration":0.13882724199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"110'18","reported_seq":43,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:45.538237+0000","last_change":"2024-01-15T18:23:45.538258+0000","last_active":"2024-01-15T18:23:45.538237+0000","last_peered":"2024-01-15T18:23:45.538237+0000","last_clean":"2024-01-15T18:23:45.538237+0000","last_became_active":"2024-01-15T18:18:32.216612+0000","last_became_peered":"2024-01-15T18:18:32.216612+0000","last_unstale":"2024-01-15T18:23:45.538237+0000","last_undegraded":"2024-01-15T18:23:45.538237+0000","last_fullsized":"2024-01-15T18:23:45.538237+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'18","last_scrub_stamp":"2024-01-15T18:23:45.538139+0000","last_deep_scrub":"110'18","last_deep_scrub_stamp":"2024-01-15T18:23:45.538139+0000","last_clean_scrub_stamp":"2024-01-15T18:23:45.538139+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:40:14.696757+0000","scrub_duration":0.0021555250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":506,"num_omap_keys":10,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"110'66","reported_seq":104,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:44.623824+0000","last_change":"2024-01-15T18:23:44.623866+0000","last_active":"2024-01-15T18:23:44.623824+0000","last_peered":"2024-01-15T18:23:44.623824+0000","last_clean":"2024-01-15T18:23:44.623824+0000","last_became_active":"2024-01-15T18:18:33.163994+0000","last_became_peered":"2024-01-15T18:18:33.163994+0000","last_unstale":"2024-01-15T18:23:44.623824+0000","last_undegraded":"2024-01-15T18:23:44.623824+0000","last_fullsized":"2024-01-15T18:23:44.623824+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'66","last_scrub_stamp":"2024-01-15T18:23:44.623747+0000","last_deep_scrub":"110'66","last_deep_scrub_stamp":"2024-01-15T18:23:44.623747+0000","last_clean_scrub_stamp":"2024-01-15T18:23:44.623747+0000","objects_scrubbed":24,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:47:40.548332+0000","scrub_duration":0.162659633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:54.492408+0000","last_change":"2024-01-15T18:23:54.492428+0000","last_active":"2024-01-15T18:23:54.492408+0000","last_peered":"2024-01-15T18:23:54.492408+0000","last_clean":"2024-01-15T18:23:54.492408+0000","last_became_active":"2024-01-15T18:18:32.216023+0000","last_became_peered":"2024-01-15T18:18:32.216023+0000","last_unstale":"2024-01-15T18:23:54.492408+0000","last_undegraded":"2024-01-15T18:23:54.492408+0000","last_fullsized":"2024-01-15T18:23:54.492408+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:54.492304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:54.492304+0000","last_clean_scrub_stamp":"2024-01-15T18:23:54.492304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:21:06.895168+0000","scrub_duration":0.0023113349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"110'53","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:56.632803+0000","last_change":"2024-01-15T18:23:56.632855+0000","last_active":"2024-01-15T18:23:56.632803+0000","last_peered":"2024-01-15T18:23:56.632803+0000","last_clean":"2024-01-15T18:23:56.632803+0000","last_became_active":"2024-01-15T18:18:42.262255+0000","last_became_peered":"2024-01-15T18:18:42.262255+0000","last_unstale":"2024-01-15T18:23:56.632803+0000","last_undegraded":"2024-01-15T18:23:56.632803+0000","last_fullsized":"2024-01-15T18:23:56.632803+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'53","last_scrub_stamp":"2024-01-15T18:23:56.632712+0000","last_deep_scrub":"110'53","last_deep_scrub_stamp":"2024-01-15T18:23:56.632712+0000","last_clean_scrub_stamp":"2024-01-15T18:23:56.632712+0000","objects_scrubbed":19,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:10:24.687025+0000","scrub_duration":0.126095914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303290,"num_objects":19,"num_object_clones":1,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:54.899638+0000","last_change":"2024-01-15T18:23:54.899657+0000","last_active":"2024-01-15T18:23:54.899638+0000","last_peered":"2024-01-15T18:23:54.899638+0000","last_clean":"2024-01-15T18:23:54.899638+0000","last_became_active":"2024-01-15T18:18:32.213930+0000","last_became_peered":"2024-01-15T18:18:32.213930+0000","last_unstale":"2024-01-15T18:23:54.899638+0000","last_undegraded":"2024-01-15T18:23:54.899638+0000","last_fullsized":"2024-01-15T18:23:54.899638+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:54.899542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:54.899542+0000","last_clean_scrub_stamp":"2024-01-15T18:23:54.899542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:23:19.753410+0000","scrub_duration":0.0018061939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"110'60","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:52.993684+0000","last_change":"2024-01-15T18:23:52.993718+0000","last_active":"2024-01-15T18:23:52.993684+0000","last_peered":"2024-01-15T18:23:52.993684+0000","last_clean":"2024-01-15T18:23:52.993684+0000","last_became_active":"2024-01-15T18:18:33.150660+0000","last_became_peered":"2024-01-15T18:18:33.150660+0000","last_unstale":"2024-01-15T18:23:52.993684+0000","last_undegraded":"2024-01-15T18:23:52.993684+0000","last_fullsized":"2024-01-15T18:23:52.993684+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'60","last_scrub_stamp":"2024-01-15T18:23:52.993598+0000","last_deep_scrub":"110'60","last_deep_scrub_stamp":"2024-01-15T18:23:52.993598+0000","last_clean_scrub_stamp":"2024-01-15T18:23:52.993598+0000","objects_scrubbed":16,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:28:50.475455+0000","scrub_duration":0.111747205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:53.298745+0000","last_change":"2024-01-15T18:23:53.298763+0000","last_active":"2024-01-15T18:23:53.298745+0000","last_peered":"2024-01-15T18:23:53.298745+0000","last_clean":"2024-01-15T18:23:53.298745+0000","last_became_active":"2024-01-15T18:18:32.214394+0000","last_became_peered":"2024-01-15T18:18:32.214394+0000","last_unstale":"2024-01-15T18:23:53.298745+0000","last_undegraded":"2024-01-15T18:23:53.298745+0000","last_fullsized":"2024-01-15T18:23:53.298745+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:53.298655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:53.298655+0000","last_clean_scrub_stamp":"2024-01-15T18:23:53.298655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:46:41.201522+0000","scrub_duration":0.00139146,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"110'85","reported_seq":118,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:06.148602+0000","last_change":"2024-01-15T18:24:06.148627+0000","last_active":"2024-01-15T18:24:06.148602+0000","last_peered":"2024-01-15T18:24:06.148602+0000","last_clean":"2024-01-15T18:24:06.148602+0000","last_became_active":"2024-01-15T18:18:33.155797+0000","last_became_peered":"2024-01-15T18:18:33.155797+0000","last_unstale":"2024-01-15T18:24:06.148602+0000","last_undegraded":"2024-01-15T18:24:06.148602+0000","last_fullsized":"2024-01-15T18:24:06.148602+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'85","last_scrub_stamp":"2024-01-15T18:24:06.148530+0000","last_deep_scrub":"110'85","last_deep_scrub_stamp":"2024-01-15T18:24:06.148530+0000","last_clean_scrub_stamp":"2024-01-15T18:24:06.148530+0000","objects_scrubbed":29,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:13.689014+0000","scrub_duration":0.19150456099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:47.949655+0000","last_change":"2024-01-15T18:23:47.949688+0000","last_active":"2024-01-15T18:23:47.949655+0000","last_peered":"2024-01-15T18:23:47.949655+0000","last_clean":"2024-01-15T18:23:47.949655+0000","last_became_active":"2024-01-15T18:18:32.212487+0000","last_became_peered":"2024-01-15T18:18:32.212487+0000","last_unstale":"2024-01-15T18:23:47.949655+0000","last_undegraded":"2024-01-15T18:23:47.949655+0000","last_fullsized":"2024-01-15T18:23:47.949655+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:47.949523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:47.949523+0000","last_clean_scrub_stamp":"2024-01-15T18:23:47.949523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:31:41.981969+0000","scrub_duration":0.0029090320000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"110'46","reported_seq":70,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:32.056164+0000","last_change":"2024-01-15T18:18:33.167732+0000","last_active":"2024-01-15T18:23:32.056164+0000","last_peered":"2024-01-15T18:23:32.056164+0000","last_clean":"2024-01-15T18:23:32.056164+0000","last_became_active":"2024-01-15T18:18:33.167436+0000","last_became_peered":"2024-01-15T18:18:33.167436+0000","last_unstale":"2024-01-15T18:23:32.056164+0000","last_undegraded":"2024-01-15T18:23:32.056164+0000","last_fullsized":"2024-01-15T18:23:32.056164+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:18:32.183765+0000","last_clean_scrub_stamp":"2024-01-15T18:18:32.183765+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"110'12","reported_seq":47,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:40.933714+0000","last_change":"2024-01-15T18:23:40.933751+0000","last_active":"2024-01-15T18:23:40.933714+0000","last_peered":"2024-01-15T18:23:40.933714+0000","last_clean":"2024-01-15T18:23:40.933714+0000","last_became_active":"2024-01-15T18:18:32.211744+0000","last_became_peered":"2024-01-15T18:18:32.211744+0000","last_unstale":"2024-01-15T18:23:40.933714+0000","last_undegraded":"2024-01-15T18:23:40.933714+0000","last_fullsized":"2024-01-15T18:23:40.933714+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'12","last_scrub_stamp":"2024-01-15T18:23:40.933627+0000","last_deep_scrub":"110'12","last_deep_scrub_stamp":"2024-01-15T18:23:40.933627+0000","last_clean_scrub_stamp":"2024-01-15T18:23:40.933627+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:56:38.633122+0000","scrub_duration":0.0031438920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1321,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"110'59","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:38.786673+0000","last_change":"2024-01-15T18:23:38.786716+0000","last_active":"2024-01-15T18:23:38.786673+0000","last_peered":"2024-01-15T18:23:38.786673+0000","last_clean":"2024-01-15T18:23:38.786673+0000","last_became_active":"2024-01-15T18:18:33.162521+0000","last_became_peered":"2024-01-15T18:18:33.162521+0000","last_unstale":"2024-01-15T18:23:38.786673+0000","last_undegraded":"2024-01-15T18:23:38.786673+0000","last_fullsized":"2024-01-15T18:23:38.786673+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'59","last_scrub_stamp":"2024-01-15T18:23:38.786600+0000","last_deep_scrub":"110'59","last_deep_scrub_stamp":"2024-01-15T18:23:38.786600+0000","last_clean_scrub_stamp":"2024-01-15T18:23:38.786600+0000","objects_scrubbed":23,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:26:13.413987+0000","scrub_duration":0.160345389,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:45.474238+0000","last_change":"2024-01-15T18:23:45.474259+0000","last_active":"2024-01-15T18:23:45.474238+0000","last_peered":"2024-01-15T18:23:45.474238+0000","last_clean":"2024-01-15T18:23:45.474238+0000","last_became_active":"2024-01-15T18:18:32.216966+0000","last_became_peered":"2024-01-15T18:18:32.216966+0000","last_unstale":"2024-01-15T18:23:45.474238+0000","last_undegraded":"2024-01-15T18:23:45.474238+0000","last_fullsized":"2024-01-15T18:23:45.474238+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:45.474139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:45.474139+0000","last_clean_scrub_stamp":"2024-01-15T18:23:45.474139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:43:14.897532+0000","scrub_duration":0.0025295999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"110'44","reported_seq":76,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:59.012816+0000","last_change":"2024-01-15T18:23:59.012849+0000","last_active":"2024-01-15T18:23:59.012816+0000","last_peered":"2024-01-15T18:23:59.012816+0000","last_clean":"2024-01-15T18:23:59.012816+0000","last_became_active":"2024-01-15T18:18:33.165323+0000","last_became_peered":"2024-01-15T18:18:33.165323+0000","last_unstale":"2024-01-15T18:23:59.012816+0000","last_undegraded":"2024-01-15T18:23:59.012816+0000","last_fullsized":"2024-01-15T18:23:59.012816+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'44","last_scrub_stamp":"2024-01-15T18:23:59.012731+0000","last_deep_scrub":"110'44","last_deep_scrub_stamp":"2024-01-15T18:23:59.012731+0000","last_clean_scrub_stamp":"2024-01-15T18:23:59.012731+0000","objects_scrubbed":14,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:30:06.603978+0000","scrub_duration":0.088778124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:42.312415+0000","last_change":"2024-01-15T18:23:42.312436+0000","last_active":"2024-01-15T18:23:42.312415+0000","last_peered":"2024-01-15T18:23:42.312415+0000","last_clean":"2024-01-15T18:23:42.312415+0000","last_became_active":"2024-01-15T18:18:32.215973+0000","last_became_peered":"2024-01-15T18:18:32.215973+0000","last_unstale":"2024-01-15T18:23:42.312415+0000","last_undegraded":"2024-01-15T18:23:42.312415+0000","last_fullsized":"2024-01-15T18:23:42.312415+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:42.312313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:42.312313+0000","last_clean_scrub_stamp":"2024-01-15T18:23:42.312313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:17:28.948911+0000","scrub_duration":0.0020901840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"110'119","reported_seq":255,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:44.359592+0000","last_change":"2024-01-15T18:23:44.359635+0000","last_active":"2024-01-15T18:23:44.359592+0000","last_peered":"2024-01-15T18:23:44.359592+0000","last_clean":"2024-01-15T18:23:44.359592+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:23:44.359592+0000","last_undegraded":"2024-01-15T18:23:44.359592+0000","last_fullsized":"2024-01-15T18:23:44.359592+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'119","last_scrub_stamp":"2024-01-15T18:23:44.359514+0000","last_deep_scrub":"110'119","last_deep_scrub_stamp":"2024-01-15T18:23:44.359514+0000","last_clean_scrub_stamp":"2024-01-15T18:23:44.359514+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:41:00.611426+0000","scrub_duration":0.18253596899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":5368710136,"num_objects":1286,"num_object_clones":1,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":1,"num_read":525,"num_read_kb":2097165,"num_write":1306,"num_write_kb":5242895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10737459200,"data_stored":10737420272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3745,"ondisk_log_size":3745,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":616194,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":280,"num_write_kb":720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5506,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1294336,"data_stored":1232388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279,"ondisk_log_size":279,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509678,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1834500,"kb_used_data":1767292,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92537340,"statfs":{"total":96636764160,"available":94758236160,"internally_reserved":0,"allocated":1809707008,"data_stored":1808930471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.72799999999999998,"15min":0.90300000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92800000000000005,"5min":2.8620000000000001,"15min":10.284000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.70699999999999996,"15min":0.93700000000000006},"min":{"1min":0.54200000000000004,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":16.968},"last":0.72699999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.72399999999999998,"15min":0.93899999999999995},"min":{"1min":0.47599999999999998,"5min":0.38600000000000001,"15min":0.33800000000000002},"max":{"1min":1.028,"5min":1.028,"15min":15.791},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.73499999999999999,"15min":0.93100000000000005},"min":{"1min":0.52100000000000002,"5min":0.44,"15min":0.371},"max":{"1min":0.94799999999999995,"5min":1.248,"15min":10.842000000000001},"last":0.78100000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.72899999999999998,"15min":0.94499999999999995},"min":{"1min":0.49099999999999999,"5min":0.39500000000000002,"15min":0.378},"max":{"1min":0.96499999999999997,"5min":0.98799999999999999,"15min":22.318000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.752,"15min":0.91500000000000004},"min":{"1min":0.46899999999999997,"5min":0.42699999999999999,"15min":0.39100000000000001},"max":{"1min":0.91400000000000003,"5min":2.2799999999999998,"15min":9.7539999999999996},"last":0.82999999999999996}]},{"osd":3,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.72699999999999998,"15min":0.996},"min":{"1min":0.56999999999999995,"5min":0.46500000000000002,"15min":0.38100000000000001},"max":{"1min":0.875,"5min":1.2110000000000001,"15min":18.152999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.78700000000000003,"15min":0.99199999999999999},"min":{"1min":0.51600000000000001,"5min":0.495,"15min":0.433},"max":{"1min":0.98099999999999998,"5min":2.2389999999999999,"15min":10.34},"last":0.68200000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.57599999999999996,"15min":0.93000000000000005},"min":{"1min":0.40600000000000003,"5min":0.34799999999999998,"15min":0.314},"max":{"1min":0.82799999999999996,"5min":0.877,"15min":20.387},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.94099999999999995},"min":{"1min":0.438,"5min":0.39400000000000002,"15min":0.34200000000000003},"max":{"1min":0.84199999999999997,"5min":0.98099999999999998,"15min":17.215},"last":0.69999999999999996}]},{"osd":5,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.622,"15min":0.91500000000000004},"min":{"1min":0.42399999999999999,"5min":0.40999999999999998,"15min":0.34699999999999998},"max":{"1min":0.91500000000000004,"5min":0.96999999999999997,"15min":19.959},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.64200000000000002,"15min":0.95999999999999996},"min":{"1min":0.40999999999999998,"5min":0.40899999999999997,"15min":0.35799999999999998},"max":{"1min":0.92300000000000004,"5min":0.92600000000000005,"15min":19.905000000000001},"last":0.54200000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.748,"15min":1.1439999999999999},"min":{"1min":0.49299999999999999,"5min":0.44,"15min":0.40799999999999997},"max":{"1min":0.995,"5min":0.995,"15min":22.478999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.69599999999999995,"15min":1.0269999999999999},"min":{"1min":0.48599999999999999,"5min":0.44600000000000001,"15min":0.40799999999999997},"max":{"1min":0.92200000000000004,"5min":0.96299999999999997,"15min":21.509},"last":0.57099999999999995}]}]},{"osd":6,"up_from":15,"seq":64424509677,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1260480,"kb_used_data":1193272,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93111360,"statfs":{"total":96636764160,"available":95346032640,"internally_reserved":0,"allocated":1221910528,"data_stored":1221137119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:24:08 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.77600000000000002,"15min":0.93899999999999995},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.86799999999999999,"5min":6.1900000000000004,"15min":32.587000000000003},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.78100000000000003,"15min":0.84499999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.35399999999999998},"max":{"1min":0.82199999999999995,"5min":4.9649999999999999,"15min":16.193000000000001},"last":0.60399999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:24:08 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.72899999999999998,"15min":0.78600000000000003},"min":{"1min":0.41199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81200000000000006,"5min":1.0049999999999999,"15min":7.7649999999999997},"last":0.755},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.72599999999999998,"15min":0.875},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.82799999999999996,"5min":1.0189999999999999,"15min":32.512},"last":0.55100000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:24:08 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76400000000000001,"15min":0.92700000000000005},"min":{"1min":0.47599999999999998,"5min":0.46400000000000002,"15min":0.41699999999999998},"max":{"1min":0.98799999999999999,"5min":1.016,"15min":32.472999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.80400000000000005,"15min":0.88500000000000001},"min":{"1min":0.46000000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.84399999999999997,"5min":5.0410000000000004,"15min":16.602},"last":0.627}]},{"osd":3,"last update":"Mon Jan 15 18:24:08 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.82099999999999995,"15min":0.91100000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.46000000000000002},"max":{"1min":0.89200000000000002,"5min":5.0940000000000003,"15min":16.388000000000002},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.752,"15min":0.86199999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.432},"max":{"1min":0.86899999999999999,"5min":1.04,"15min":9.3719999999999999},"last":0.72299999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:24:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.67100000000000004,"15min":0.81699999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.378},"max":{"1min":0.77800000000000002,"5min":6.3869999999999996,"15min":32.549999999999997},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.68999999999999995,"15min":0.79300000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.84999999999999998,"5min":5.1719999999999997,"15min":11.852},"last":0.45600000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:24:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.69899999999999995,"15min":0.80800000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.34300000000000003},"max":{"1min":0.83199999999999996,"5min":6.8769999999999998,"15min":13.087999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.749,"15min":0.94299999999999995},"min":{"1min":0.436,"5min":0.436,"15min":0.41799999999999998},"max":{"1min":0.95099999999999996,"5min":6.1360000000000001,"15min":32.404000000000003},"last":0.48899999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:24:08 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.747,"15min":0.91700000000000004},"min":{"1min":0.49199999999999999,"5min":0.46700000000000003,"15min":0.38600000000000001},"max":{"1min":0.93300000000000005,"5min":5.1319999999999997,"15min":23.991},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.92100000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.40999999999999998},"max":{"1min":0.91300000000000003,"5min":0.999,"15min":40.972999999999999},"last":0.52800000000000002}]}]},{"osd":4,"up_from":15,"seq":64424509678,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1711036,"kb_used_data":1643828,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92660804,"statfs":{"total":96636764160,"available":94884663296,"internally_reserved":0,"allocated":1683279872,"data_stored":1682510628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.69799999999999995,"15min":0.90600000000000003},"min":{"1min":0.40699999999999997,"5min":0.39200000000000002,"15min":0.32900000000000001},"max":{"1min":0.96599999999999997,"5min":1.542,"15min":23.318000000000001},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.70299999999999996,"15min":0.81299999999999994},"min":{"1min":0.55000000000000004,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":14.723000000000001},"last":0.81499999999999995}]},{"osd":1,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.72599999999999998,"15min":0.81399999999999995},"min":{"1min":0.46300000000000002,"5min":0.39100000000000001,"15min":0.38800000000000001},"max":{"1min":0.96199999999999997,"5min":1.919,"15min":9.5700000000000003},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.74199999999999999,"15min":0.94699999999999995},"min":{"1min":0.52600000000000002,"5min":0.44900000000000001,"15min":0.36899999999999999},"max":{"1min":0.996,"5min":1.429,"15min":22.408999999999999},"last":0.83699999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.79000000000000004,"15min":0.85299999999999998},"min":{"1min":0.50900000000000001,"5min":0.41899999999999998,"15min":0.39700000000000002},"max":{"1min":1.032,"5min":5.6260000000000003,"15min":9.7089999999999996},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.76100000000000001,"15min":1.018},"min":{"1min":0.59299999999999997,"5min":0.45000000000000001,"15min":0.39100000000000001},"max":{"1min":1.016,"5min":1.387,"15min":22.338999999999999},"last":0.93000000000000005}]},{"osd":3,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.73999999999999999,"15min":0.88700000000000001},"min":{"1min":0.52300000000000002,"5min":0.46200000000000002,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":1.577,"15min":14.763999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.82599999999999996,"15min":0.93600000000000005},"min":{"1min":0.65300000000000002,"5min":0.53600000000000003,"15min":0.48099999999999998},"max":{"1min":0.996,"5min":1.9610000000000001,"15min":9.6289999999999996},"last":0.95599999999999996}]},{"osd":5,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63300000000000001,"15min":0.94299999999999995},"min":{"1min":0.44,"5min":0.36499999999999999,"15min":0.34799999999999998},"max":{"1min":1.0149999999999999,"5min":1.1970000000000001,"15min":31.617999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.85799999999999998},"min":{"1min":0.40799999999999997,"5min":0.373,"15min":0.36599999999999999},"max":{"1min":0.86699999999999999,"5min":1.4910000000000001,"15min":17.145},"last":0.54800000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.72799999999999998,"15min":0.83899999999999997},"min":{"1min":0.60799999999999998,"5min":0.41799999999999998,"15min":0.37},"max":{"1min":0.89300000000000002,"5min":5.9489999999999998,"15min":11.124000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.71099999999999997,"15min":1.2090000000000001},"min":{"1min":0.52300000000000002,"5min":0.42499999999999999,"15min":0.40200000000000002},"max":{"1min":1.0649999999999999,"5min":2.3149999999999999,"15min":32.308},"last":0.70399999999999996}]},{"osd":7,"last update":"Mon Jan 15 18:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.72699999999999998,"15min":1.0009999999999999},"min":{"1min":0.48099999999999998,"5min":0.45100000000000001,"15min":0.34799999999999998},"max":{"1min":1,"5min":2.4329999999999998,"15min":39.256},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78200000000000003,"15min":1.002},"min":{"1min":0.60699999999999998,"5min":0.499,"15min":0.46500000000000002},"max":{"1min":1.04,"5min":2.1909999999999998,"15min":26.573},"last":0.78900000000000003}]}]},{"osd":3,"up_from":15,"seq":64424509677,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1453604,"kb_used_data":1386396,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92918236,"statfs":{"total":96636764160,"available":95148273664,"internally_reserved":0,"allocated":1419669504,"data_stored":1418883882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68811207},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.45100000000000001,"15min":0.498},"min":{"1min":0.28999999999999998,"5min":0.28199999999999997,"15min":0.245},"max":{"1min":0.69099999999999995,"5min":0.69699999999999995,"15min":5.8849999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.51000000000000001,"15min":0.621},"min":{"1min":0.41799999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.80000000000000004,"5min":1.4750000000000001,"15min":17.756},"last":0.374}]},{"osd":1,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51300000000000001,"15min":0.60499999999999998},"min":{"1min":0.33500000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60399999999999998,"5min":4.0099999999999998,"15min":12.307},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55200000000000005,"15min":0.59199999999999997},"min":{"1min":0.36199999999999999,"5min":0.34300000000000003,"15min":0.31900000000000001},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":6.0640000000000001},"last":0.48599999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61399999999999999,"15min":0.71199999999999997},"min":{"1min":0.42799999999999999,"5min":0.35299999999999998,"15min":0.32600000000000001},"max":{"1min":0.82999999999999996,"5min":4.077,"15min":12.468999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.59099999999999997,"15min":0.68000000000000005},"min":{"1min":0.44,"5min":0.39100000000000001,"15min":0.35399999999999998},"max":{"1min":0.78000000000000003,"5min":0.81899999999999995,"15min":10.837999999999999},"last":0.52800000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78500000000000003,"15min":0.999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.375},"max":{"1min":1.1910000000000001,"5min":1.9379999999999999,"15min":17.853999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.85999999999999999,"15min":0.98199999999999998},"min":{"1min":0.54900000000000004,"5min":0.49199999999999999,"15min":0.45200000000000001},"max":{"1min":1.0569999999999999,"5min":4.2430000000000003,"15min":13.31},"last":0.76000000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.78400000000000003,"15min":0.98899999999999999},"min":{"1min":0.496,"5min":0.439,"15min":0.439},"max":{"1min":1.1619999999999999,"5min":1.179,"15min":17.687999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.77800000000000002,"15min":0.99099999999999999},"min":{"1min":0.58599999999999997,"5min":0.53800000000000003,"15min":0.438},"max":{"1min":1.048,"5min":1.825,"15min":20.515999999999998},"last":0.81599999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.80900000000000005,"15min":0.98499999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.432},"max":{"1min":1.1419999999999999,"5min":4.2800000000000002,"15min":23.100999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.80600000000000005,"15min":0.94799999999999995},"min":{"1min":0.69899999999999995,"5min":0.52200000000000002,"15min":0.5},"max":{"1min":1.0720000000000001,"5min":1.149,"15min":12.76},"last":0.80500000000000005}]},{"osd":7,"last update":"Mon Jan 15 18:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82499999999999996,"15min":1.1850000000000001},"min":{"1min":0.625,"5min":0.56899999999999995,"15min":0.46800000000000003},"max":{"1min":1.0860000000000001,"5min":1.931,"15min":24.321999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.82899999999999996,"15min":1.0820000000000001},"min":{"1min":0.60899999999999999,"5min":0.497,"15min":0.497},"max":{"1min":1.2070000000000001,"5min":4.202,"15min":23.013000000000002},"last":0.66600000000000004}]}]},{"osd":2,"up_from":15,"seq":64424509677,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":977856,"kb_used_data":910648,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93393984,"statfs":{"total":96636764160,"available":95635439616,"internally_reserved":0,"allocated":932503552,"data_stored":931731464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46200000000000002,"15min":0.47499999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.247},"max":{"1min":0.65900000000000003,"5min":0.80800000000000005,"15min":1.857},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.499,"15min":0.50600000000000001},"min":{"1min":0.318,"5min":0.26700000000000002,"15min":0.252},"max":{"1min":0.67400000000000004,"5min":0.73999999999999999,"15min":2.3759999999999999},"last":0.53300000000000003}]},{"osd":1,"last update":"Mon Jan 15 18:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55700000000000005,"15min":0.60099999999999998},"min":{"1min":0.34200000000000003,"5min":0.314,"15min":0.29699999999999999},"max":{"1min":0.79300000000000004,"5min":0.94599999999999995,"15min":10.907},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.49299999999999999,"15min":0.56100000000000005},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.28100000000000003},"max":{"1min":0.64200000000000002,"5min":0.76500000000000001,"15min":7.2119999999999997},"last":0.58299999999999996}]},{"osd":3,"last update":"Mon Jan 15 18:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.52900000000000003,"15min":0.55700000000000005},"min":{"1min":0.39000000000000001,"5min":0.35699999999999998,"15min":0.30499999999999999},"max":{"1min":0.748,"5min":0.86499999999999999,"15min":2.7290000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.62,"15min":0.64200000000000002},"min":{"1min":0.41799999999999998,"5min":0.38600000000000001,"15min":0.373},"max":{"1min":0.82399999999999995,"5min":0.89600000000000002,"15min":7.6760000000000002},"last":0.65300000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.81000000000000005,"15min":0.84199999999999997},"min":{"1min":0.60699999999999998,"5min":0.55800000000000005,"15min":0.44},"max":{"1min":0.98699999999999999,"5min":1.22,"15min":4.3150000000000004},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.77700000000000002,"15min":0.80200000000000005},"min":{"1min":0.55900000000000005,"5min":0.47699999999999998,"15min":0.41599999999999998},"max":{"1min":0.91300000000000003,"5min":1.9670000000000001,"15min":8.3190000000000008},"last":0.83199999999999996}]},{"osd":5,"last update":"Mon Jan 15 18:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.76800000000000002,"15min":0.78500000000000003},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.41899999999999998},"max":{"1min":0.84699999999999998,"5min":1.111,"15min":5.0309999999999997},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.82599999999999996,"15min":0.86399999999999999},"min":{"1min":0.58499999999999996,"5min":0.57399999999999995,"15min":0.48199999999999998},"max":{"1min":0.93999999999999995,"5min":1.8240000000000001,"15min":10.460000000000001},"last":0.86699999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.79400000000000004,"15min":0.86599999999999999},"min":{"1min":0.60999999999999999,"5min":0.55800000000000005,"15min":0.40899999999999997},"max":{"1min":0.93799999999999994,"5min":1.3979999999999999,"15min":8.3580000000000005},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.80800000000000005,"15min":0.80600000000000005},"min":{"1min":0.54700000000000004,"5min":0.5,"15min":0.47899999999999998},"max":{"1min":0.91100000000000003,"5min":1.873,"15min":2.669},"last":0.89100000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.80900000000000005,"15min":0.91800000000000004},"min":{"1min":0.65100000000000002,"5min":0.60699999999999998,"15min":0.44400000000000001},"max":{"1min":0.96999999999999997,"5min":1.7729999999999999,"15min":11.606},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.85799999999999998,"15min":0.88},"min":{"1min":0.67300000000000004,"5min":0.627,"15min":0.47899999999999998},"max":{"1min":0.97399999999999998,"5min":1.611,"15min":6.1260000000000003},"last":0.92500000000000004}]}]},{"osd":1,"up_from":15,"seq":64424509678,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1133516,"kb_used_data":1066308,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93238324,"statfs":{"total":96636764160,"available":95476043776,"internally_reserved":0,"allocated":1091899392,"data_stored":1091115320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:24:13 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.45200000000000001,"15min":0.53400000000000003},"min":{"1min":0.26800000000000002,"5min":0.26700000000000002,"15min":0.23699999999999999},"max":{"1min":0.54000000000000004,"5min":0.69699999999999995,"15min":17.396999999999998},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.502,"15min":0.57999999999999996},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.26900000000000002},"max":{"1min":0.61199999999999999,"5min":0.82999999999999996,"15min":12.378},"last":0.316}]},{"osd":2,"last update":"Mon Jan 15 18:24:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.54300000000000004},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.72699999999999998,"5min":0.878,"15min":2.8610000000000002},"last":0.377},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54500000000000004,"15min":0.57799999999999996},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":3.0449999999999999},"last":0.41999999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:24:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58499999999999996,"15min":0.622},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.33700000000000002},"max":{"1min":0.79300000000000004,"5min":1.0580000000000001,"15min":3.0899999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.52700000000000002,"15min":0.626},"min":{"1min":0.34699999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.754,"5min":0.85499999999999998,"15min":9.9719999999999995},"last":0.40200000000000002}]},{"osd":4,"last update":"Mon Jan 15 18:24:13 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.77300000000000002,"15min":0.86099999999999999},"min":{"1min":0.51600000000000001,"5min":0.50900000000000001,"15min":0.437},"max":{"1min":0.92800000000000005,"5min":1.1060000000000001,"15min":9.9329999999999998},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.77400000000000002,"15min":0.84999999999999998},"min":{"1min":0.496,"5min":0.496,"15min":0.45800000000000002},"max":{"1min":0.98999999999999999,"5min":1.165,"15min":9.8610000000000007},"last":0.82299999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:24:13 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.79300000000000004,"15min":0.82999999999999996},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.46100000000000002},"max":{"1min":0.83599999999999997,"5min":1.2050000000000001,"15min":8.5839999999999996},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.75,"5min":0.78100000000000003,"15min":0.92700000000000005},"min":{"1min":0.53100000000000003,"5min":0.52300000000000002,"15min":0.443},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":26.748999999999999},"last":0.84899999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:24:13 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.78600000000000003,"15min":0.88100000000000001},"min":{"1min":0.59099999999999997,"5min":0.56799999999999995,"15min":0.42899999999999999},"max":{"1min":1.0209999999999999,"5min":1.218,"15min":9.6349999999999998},"last":0.874},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.80300000000000005,"15min":0.85699999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.501},"max":{"1min":0.99199999999999999,"5min":1.2090000000000001,"15min":5.8849999999999998},"last":0.70899999999999996}]},{"osd":7,"last update":"Mon Jan 15 18:24:13 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.80900000000000005,"15min":0.96799999999999997},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.46899999999999997},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":20.872},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.81399999999999995,"15min":0.94099999999999995},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.081,"5min":1.139,"15min":9.8640000000000008},"last":0.92600000000000005}]}]},{"osd":5,"up_from":14,"seq":60129542382,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":908420,"kb_used_data":828732,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":93463420,"statfs":{"total":96636764160,"available":95706542080,"internally_reserved":0,"allocated":848621568,"data_stored":847844597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81590725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.72299999999999998,"15min":0.754},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.29899999999999999},"max":{"1min":1.038,"5min":2.6920000000000002,"15min":5.1639999999999997},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.75800000000000001,"15min":0.83899999999999997},"min":{"1min":0.46899999999999997,"5min":0.36799999999999999,"15min":0.34899999999999998},"max":{"1min":0.98199999999999998,"5min":6.4290000000000003,"15min":9.359},"last":0.82999999999999996}]},{"osd":1,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.76800000000000002,"15min":0.93000000000000005},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.375},"max":{"1min":0.89900000000000002,"5min":6.5069999999999997,"15min":21.120000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.747,"15min":0.93899999999999995},"min":{"1min":0.48299999999999998,"5min":0.39900000000000002,"15min":0.38600000000000001},"max":{"1min":0.96499999999999997,"5min":2.6619999999999999,"15min":15.843},"last":0.79300000000000004}]},{"osd":2,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.76800000000000002,"15min":0.86699999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.34999999999999998},"max":{"1min":0.95099999999999996,"5min":6.3339999999999996,"15min":8.7750000000000004},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.79900000000000004,"15min":0.98299999999999998},"min":{"1min":0.501,"5min":0.501,"15min":0.39900000000000002},"max":{"1min":0.95399999999999996,"5min":6.4619999999999997,"15min":21.201000000000001},"last":0.81499999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.753,"15min":0.84399999999999997},"min":{"1min":0.498,"5min":0.46000000000000002,"15min":0.37},"max":{"1min":0.99199999999999999,"5min":2.6299999999999999,"15min":7.7199999999999998},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.755,"5min":0.81100000000000005,"15min":0.91900000000000004},"min":{"1min":0.52800000000000002,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.0109999999999999,"5min":6.3730000000000002,"15min":8.7300000000000004},"last":0.76300000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.66500000000000004,"15min":0.83399999999999996},"min":{"1min":0.52600000000000002,"5min":0.42999999999999999,"15min":0.30399999999999999},"max":{"1min":1.012,"5min":3.161,"15min":10.962},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.70699999999999996,"15min":1.012},"min":{"1min":0.48799999999999999,"5min":0.41999999999999998,"15min":0.35099999999999998},"max":{"1min":0.88100000000000001,"5min":6.5460000000000003,"15min":26.146999999999998},"last":0.51700000000000002}]},{"osd":6,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.66500000000000004,"15min":0.76700000000000002},"min":{"1min":0.50700000000000001,"5min":0.41299999999999998,"15min":0.36099999999999999},"max":{"1min":0.871,"5min":2.589,"15min":6.6699999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.82399999999999995,"15min":0.99199999999999999},"min":{"1min":0.53800000000000003,"5min":0.45300000000000001,"15min":0.379},"max":{"1min":0.95699999999999996,"5min":14.007999999999999,"15min":30.488},"last":0.49199999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.72499999999999998,"15min":1.2010000000000001},"min":{"1min":0.56699999999999995,"5min":0.48899999999999999,"15min":0.42599999999999999},"max":{"1min":0.90800000000000003,"5min":3.0870000000000002,"15min":33.499000000000002},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.91000000000000003,"15min":1.0680000000000001},"min":{"1min":0.57099999999999995,"5min":0.45700000000000002,"15min":0.42899999999999999},"max":{"1min":0.93000000000000005,"5min":21.178999999999998,"15min":25.437999999999999},"last":0.54700000000000004}]}]},{"osd":0,"up_from":14,"seq":60129542382,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1769580,"kb_used_data":1702372,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92602260,"statfs":{"total":96636764160,"available":94824714240,"internally_reserved":0,"allocated":1743228928,"data_stored":1742435619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.82999999999999996,"15min":0.749},"min":{"1min":0.33100000000000002,"5min":0.25800000000000001,"15min":0.245},"max":{"1min":0.59899999999999998,"5min":38.521999999999998,"15min":38.521999999999998},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.60099999999999998,"15min":0.68799999999999994},"min":{"1min":0.36699999999999999,"5min":0.32900000000000001,"15min":0.27000000000000002},"max":{"1min":0.753,"5min":8.0749999999999993,"15min":10.154},"last":0.65000000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.57699999999999996,"15min":0.80200000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.30399999999999999},"max":{"1min":0.78600000000000003,"5min":2.6909999999999998,"15min":25.905000000000001},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.67200000000000004,"15min":0.66900000000000004},"min":{"1min":0.307,"5min":0.307,"15min":0.27800000000000002},"max":{"1min":0.69199999999999995,"5min":18.247,"15min":18.247},"last":0.629}]},{"osd":3,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.76400000000000001,"15min":0.80300000000000005},"min":{"1min":0.32400000000000001,"5min":0.32000000000000001,"15min":0.29199999999999998},"max":{"1min":0.70999999999999996,"5min":21.451000000000001,"15min":21.451000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.61799999999999999,"15min":0.76600000000000001},"min":{"1min":0.47099999999999997,"5min":0.38900000000000001,"15min":0.36899999999999999},"max":{"1min":0.80100000000000005,"5min":1.7,"15min":10.210000000000001},"last":0.57899999999999996}]},{"osd":4,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.79400000000000004,"15min":0.99199999999999999},"min":{"1min":0.52600000000000002,"5min":0.441,"15min":0.39100000000000001},"max":{"1min":1.1639999999999999,"5min":2.7850000000000001,"15min":22.814},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.80100000000000005,"5min":1.0389999999999999,"15min":1.105},"min":{"1min":0.64000000000000001,"5min":0.45500000000000002,"15min":0.42399999999999999},"max":{"1min":1.1379999999999999,"5min":27.027999999999999,"15min":27.027999999999999},"last":0.47899999999999998}]},{"osd":5,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.83199999999999996,"15min":0.98799999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.38400000000000001},"max":{"1min":1.218,"5min":6.6429999999999998,"15min":12.428000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.77700000000000002,"5min":1.0509999999999999,"15min":1.333},"min":{"1min":0.55400000000000005,"5min":0.47999999999999998,"15min":0.46100000000000002},"max":{"1min":1.0600000000000001,"5min":27.210999999999999,"15min":27.210999999999999},"last":0.60599999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.86699999999999999,"15min":1.242},"min":{"1min":0.66400000000000003,"5min":0.50900000000000001,"15min":0.46700000000000003},"max":{"1min":0.98399999999999999,"5min":9.3140000000000001,"15min":31.059000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.82699999999999996,"15min":1.089},"min":{"1min":0.58899999999999997,"5min":0.52900000000000003,"15min":0.51500000000000001},"max":{"1min":1.024,"5min":2.8340000000000001,"15min":40.731000000000002},"last":0.72399999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.86299999999999999,"15min":1.2150000000000001},"min":{"1min":0.66500000000000004,"5min":0.52200000000000002,"15min":0.499},"max":{"1min":1.1539999999999999,"5min":5.2089999999999996,"15min":24.492999999999999},"last":0.745},{"interface":"front","average":{"1min":0.81599999999999995,"5min":1.097,"15min":1.1859999999999999},"min":{"1min":0.63500000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1200000000000001,"5min":27.081,"15min":27.081},"last":0.67800000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":630784,"data_stored":614187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":1321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":626688,"data_stored":614145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1740644352,"data_stored":1740636601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1090527232,"data_stored":1090519412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":931139584,"data_stored":931135680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1417682944,"data_stored":1417675274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1681920000,"data_stored":1681916165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":847253504,"data_stored":847249530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1220546560,"data_stored":1220542586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1807745024,"data_stored":1807745024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:24:22.446 INFO:tasks.ceph:pgid 17.3f last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:22.447 INFO:tasks.ceph:pgid 17.37 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:22.447 INFO:tasks.ceph:pgid 17.36 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:22.447 INFO:tasks.ceph:pgid 17.34 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:22.447 INFO:tasks.ceph:pgid 17.33 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:22.447 INFO:tasks.ceph:pgid 17.30 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:22.448 INFO:tasks.ceph:pgid 17.2d last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:22.448 INFO:tasks.ceph:pgid 17.26 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:22.448 INFO:tasks.ceph:pgid 17.1d last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:22.449 INFO:tasks.ceph:pgid 17.c last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:22.449 INFO:tasks.ceph:pgid 17.d last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:22.449 INFO:tasks.ceph:pgid 17.b last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:22.450 INFO:tasks.ceph:pgid 17.0 last_scrub_stamp 2024-01-15T18:18:32.183765+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=18, tm_sec=32, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=23, tm_sec=35, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:24:22.450 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-15T18:24:28.766 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:24:28.770 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:24:42.451 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:24:42.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.606+0000 7fb30d600640 1 Processor -- start 2024-01-15T18:24:42.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb30d600640 1 -- start start 2024-01-15T18:24:42.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb30d600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb3080cfaa0 0x7fb30806bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:42.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb30d600640 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3080ae280 con 0x7fb3080ae710 2024-01-15T18:24:42.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb30d600640 1 -- --> v1:172.21.15.93:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3080adc50 con 0x7fb308160af0 2024-01-15T18:24:42.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb30d600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3080cff80 con 0x7fb3080cfaa0 2024-01-15T18:24:42.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb307400640 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb3080ae710 0x7fb3080aebc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46746/0 (socket says 172.21.15.27:46746) 2024-01-15T18:24:42.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb307400640 1 -- 172.21.15.27:0/3188369877 learned_addr learned my addr 172.21.15.27:0/3188369877 (peer_addr_for_me v1:172.21.15.27:0/0) 2024-01-15T18:24:42.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb306000640 1 --2- 172.21.15.27:0/3188369877 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb3080cfaa0 0x7fb30806bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:42.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb305600640 1 -- 172.21.15.27:0/3188369877 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2878189457 0 0) 0x7fb3080ae280 con 0x7fb3080ae710 2024-01-15T18:24:42.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb305600640 1 -- 172.21.15.27:0/3188369877 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2e8003530 con 0x7fb3080ae710 2024-01-15T18:24:42.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb305600640 1 -- 172.21.15.27:0/3188369877 <== mon.1 v1:172.21.15.93:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922655132 0 0) 0x7fb3080adc50 con 0x7fb308160af0 2024-01-15T18:24:42.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb305600640 1 -- 172.21.15.27:0/3188369877 --> v1:172.21.15.93:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3080ae280 con 0x7fb308160af0 2024-01-15T18:24:42.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb305600640 1 -- 172.21.15.27:0/3188369877 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 672280076 0 0) 0x7fb2e8003530 con 0x7fb3080ae710 2024-01-15T18:24:42.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb305600640 1 -- 172.21.15.27:0/3188369877 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb3080cfaa0 msgr2=0x7fb30806bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:24:42.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb305600640 1 --2- 172.21.15.27:0/3188369877 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb3080cfaa0 0x7fb30806bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:42.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb305600640 1 -- 172.21.15.27:0/3188369877 >> v1:172.21.15.93:6789/0 conn(0x7fb308160af0 legacy=0x7fb30806b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:42.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb305600640 1 -- 172.21.15.27:0/3188369877 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3080d0260 con 0x7fb3080ae710 2024-01-15T18:24:42.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb305600640 1 -- 172.21.15.27:0/3188369877 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1938298234 0 0) 0x7fb300001a00 con 0x7fb3080ae710 2024-01-15T18:24:42.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb305600640 1 -- 172.21.15.27:0/3188369877 >> v1:172.21.15.27:6789/0 conn(0x7fb3080ae710 legacy=0x7fb3080aebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:42.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb305600640 1 --2- 172.21.15.27:0/3188369877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2e8003bb0 0x7fb2e8003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:42.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb305600640 1 --2- 172.21.15.27:0/3188369877 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb2e80047d0 0x7fb2e8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:42.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb305600640 1 --2- 172.21.15.27:0/3188369877 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb3080cfaa0 0x7fb2e8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:42.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb305600640 1 -- 172.21.15.27:0/3188369877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3080cff80 con 0x7fb2e8003bb0 2024-01-15T18:24:42.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb305600640 1 -- 172.21.15.27:0/3188369877 --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e800ba70 con 0x7fb2e80047d0 2024-01-15T18:24:42.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb305600640 1 -- 172.21.15.27:0/3188369877 --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e800bc30 con 0x7fb3080cfaa0 2024-01-15T18:24:42.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb306a00640 1 --2- 172.21.15.27:0/3188369877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2e8003bb0 0x7fb2e8003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:42.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb306000640 1 --2- 172.21.15.27:0/3188369877 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb2e80047d0 0x7fb2e8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:42.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb307400640 1 --2- 172.21.15.27:0/3188369877 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb3080cfaa0 0x7fb2e8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:42.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb306a00640 1 -- 172.21.15.27:0/3188369877 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb3080cfaa0 msgr2=0x7fb2e8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:24:42.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb306a00640 1 --2- 172.21.15.27:0/3188369877 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb3080cfaa0 0x7fb2e8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:42.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb306a00640 1 -- 172.21.15.27:0/3188369877 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb2e80047d0 msgr2=0x7fb2e8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:24:42.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb306a00640 1 --2- 172.21.15.27:0/3188369877 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb2e80047d0 0x7fb2e8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:42.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb306a00640 1 -- 172.21.15.27:0/3188369877 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e8003530 con 0x7fb2e8003bb0 2024-01-15T18:24:42.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.610+0000 7fb306000640 1 --2- 172.21.15.27:0/3188369877 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb2e80047d0 0x7fb2e8004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:24:42.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.614+0000 7fb306a00640 1 --2- 172.21.15.27:0/3188369877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2e8003bb0 0x7fb2e8003fb0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8007fd0 tx=0x7fb2f801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e39a85ebdd4afb6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:24:42.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.614+0000 7fb305600640 1 -- 172.21.15.27:0/3188369877 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f80052d0 con 0x7fb2e8003bb0 2024-01-15T18:24:42.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.614+0000 7fb305600640 1 -- 172.21.15.27:0/3188369877 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f80a7050 con 0x7fb2e8003bb0 2024-01-15T18:24:42.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.614+0000 7fb305600640 1 -- 172.21.15.27:0/3188369877 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f8003600 con 0x7fb2e8003bb0 2024-01-15T18:24:42.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.614+0000 7fb30d600640 1 -- 172.21.15.27:0/3188369877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2e8003bb0 msgr2=0x7fb2e8003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:42.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.614+0000 7fb30d600640 1 --2- 172.21.15.27:0/3188369877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2e8003bb0 0x7fb2e8003fb0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8007fd0 tx=0x7fb2f801c520 comp rx=0 tx=0).stop 2024-01-15T18:24:42.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.614+0000 7fb307400640 1 -- 172.21.15.27:0/3188369877 reap_dead start 2024-01-15T18:24:42.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.614+0000 7fb30d600640 1 -- 172.21.15.27:0/3188369877 shutdown_connections 2024-01-15T18:24:42.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.614+0000 7fb30d600640 1 -- 172.21.15.27:0/3188369877 >> 172.21.15.27:0/3188369877 conn(0x7fb3080d0f20 msgr2=0x7fb3080d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:24:42.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.614+0000 7fb30d600640 1 -- 172.21.15.27:0/3188369877 shutdown_connections 2024-01-15T18:24:42.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.614+0000 7fb30d600640 1 -- 172.21.15.27:0/3188369877 wait complete. 2024-01-15T18:24:42.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.618+0000 7fb30d600640 1 Processor -- start 2024-01-15T18:24:42.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.618+0000 7fb30d600640 1 -- start start 2024-01-15T18:24:42.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.618+0000 7fb30d600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2e8003bb0 0x7fb3080ca0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:42.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.618+0000 7fb30d600640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb308150060 0x7fb3081504e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:42.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.618+0000 7fb30d600640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb308163970 0x7fb308163e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:42.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.618+0000 7fb30d600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3081626f0 con 0x7fb2e8003bb0 2024-01-15T18:24:42.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.618+0000 7fb30d600640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3080ccdf0 con 0x7fb308163970 2024-01-15T18:24:42.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.618+0000 7fb30d600640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3081628d0 con 0x7fb308150060 2024-01-15T18:24:42.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.618+0000 7fb306a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2e8003bb0 0x7fb3080ca0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:42.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.618+0000 7fb306a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2e8003bb0 0x7fb3080ca0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55068/0 (socket says 172.21.15.27:55068) 2024-01-15T18:24:42.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.618+0000 7fb306a00640 1 -- 172.21.15.27:0/1832756241 learned_addr learned my addr 172.21.15.27:0/1832756241 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:24:42.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.618+0000 7fb306000640 1 --2- 172.21.15.27:0/1832756241 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb308150060 0x7fb3081504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:42.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.618+0000 7fb307400640 1 --2- 172.21.15.27:0/1832756241 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb308163970 0x7fb308163e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:42.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.618+0000 7fb306a00640 1 -- 172.21.15.27:0/1832756241 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb308150060 msgr2=0x7fb3081504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:42.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.618+0000 7fb306a00640 1 --2- 172.21.15.27:0/1832756241 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb308150060 0x7fb3081504e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:42.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.618+0000 7fb306a00640 1 -- 172.21.15.27:0/1832756241 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb308163970 msgr2=0x7fb308163e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:42.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.618+0000 7fb306a00640 1 --2- 172.21.15.27:0/1832756241 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb308163970 0x7fb308163e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:42.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.618+0000 7fb306a00640 1 -- 172.21.15.27:0/1832756241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb308162bb0 con 0x7fb2e8003bb0 2024-01-15T18:24:42.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.618+0000 7fb306a00640 1 --2- 172.21.15.27:0/1832756241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2e8003bb0 0x7fb3080ca0e0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8005040 tx=0x7fb2e8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:24:42.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.618+0000 7fb2fea00640 1 -- 172.21.15.27:0/1832756241 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f80069e0 con 0x7fb2e8003bb0 2024-01-15T18:24:42.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.618+0000 7fb30d600640 1 -- 172.21.15.27:0/1832756241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb308162e90 con 0x7fb2e8003bb0 2024-01-15T18:24:42.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.618+0000 7fb2fea00640 1 -- 172.21.15.27:0/1832756241 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f8006b40 con 0x7fb2e8003bb0 2024-01-15T18:24:42.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.618+0000 7fb2fea00640 1 -- 172.21.15.27:0/1832756241 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f8007030 con 0x7fb2e8003bb0 2024-01-15T18:24:42.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.618+0000 7fb30d600640 1 -- 172.21.15.27:0/1832756241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb308163450 con 0x7fb2e8003bb0 2024-01-15T18:24:42.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.622+0000 7fb30d600640 1 -- 172.21.15.27:0/1832756241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb30804e850 con 0x7fb2e8003bb0 2024-01-15T18:24:42.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.622+0000 7fb2fea00640 1 -- 172.21.15.27:0/1832756241 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 87690+0+0 (secure 0 0 0) 0x7fb2f8007190 con 0x7fb2e8003bb0 2024-01-15T18:24:42.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.622+0000 7fb2fea00640 1 --2- 172.21.15.27:0/1832756241 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb2e00679d0 0x7fb2e0069e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:42.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.622+0000 7fb2fea00640 1 -- 172.21.15.27:0/1832756241 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fb2f80a4e40 con 0x7fb2e8003bb0 2024-01-15T18:24:42.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.626+0000 7fb306000640 1 --2- 172.21.15.27:0/1832756241 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb2e00679d0 0x7fb2e0069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:42.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.626+0000 7fb306000640 1 --2- 172.21.15.27:0/1832756241 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb2e00679d0 0x7fb2e0069e90 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fb308150010 tx=0x7fb2f0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:24:42.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.626+0000 7fb2fea00640 1 -- 172.21.15.27:0/1832756241 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb2f80ec4e0 con 0x7fb2e8003bb0 2024-01-15T18:24:42.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.782+0000 7fb30d600640 1 -- 172.21.15.27:0/1832756241 --> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb308072620 con 0x7fb2e00679d0 2024-01-15T18:24:42.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.790+0000 7fb2fea00640 1 -- 172.21.15.27:0/1832756241 <== mgr.4106 v2:172.21.15.93:6832/1477920209 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+293971 (secure 0 0 0) 0x7fb308072620 con 0x7fb2e00679d0 2024-01-15T18:24:42.789 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:24:42.790 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-01-15T18:24:42.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.794+0000 7fb30d600640 1 -- 172.21.15.27:0/1832756241 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb2e00679d0 msgr2=0x7fb2e0069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:42.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.794+0000 7fb30d600640 1 --2- 172.21.15.27:0/1832756241 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb2e00679d0 0x7fb2e0069e90 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fb308150010 tx=0x7fb2f0008040 comp rx=0 tx=0).stop 2024-01-15T18:24:42.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.794+0000 7fb30d600640 1 -- 172.21.15.27:0/1832756241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2e8003bb0 msgr2=0x7fb3080ca0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:42.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.794+0000 7fb30d600640 1 --2- 172.21.15.27:0/1832756241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2e8003bb0 0x7fb3080ca0e0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8005040 tx=0x7fb2e8005800 comp rx=0 tx=0).stop 2024-01-15T18:24:42.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.794+0000 7fb30d600640 1 -- 172.21.15.27:0/1832756241 shutdown_connections 2024-01-15T18:24:42.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.794+0000 7fb30d600640 1 --2- 172.21.15.27:0/1832756241 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fb2e00679d0 0x7fb2e0069e90 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:42.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.794+0000 7fb30d600640 1 --2- 172.21.15.27:0/1832756241 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fb308163970 0x7fb308163e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:42.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.794+0000 7fb30d600640 1 --2- 172.21.15.27:0/1832756241 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fb308150060 0x7fb3081504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:42.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.794+0000 7fb30d600640 1 --2- 172.21.15.27:0/1832756241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb2e8003bb0 0x7fb3080ca0e0 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:42.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.794+0000 7fb30d600640 1 -- 172.21.15.27:0/1832756241 >> 172.21.15.27:0/1832756241 conn(0x7fb3080d0f20 msgr2=0x7fb3080d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:24:42.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.794+0000 7fb30d600640 1 -- 172.21.15.27:0/1832756241 shutdown_connections 2024-01-15T18:24:42.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.794+0000 7fb30d600640 1 -- 172.21.15.27:0/1832756241 wait complete. 2024-01-15T18:24:42.810 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":702,"stamp":"2024-01-15T18:24:42.475520+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5369916839,"num_objects":1311,"num_object_clones":1,"num_object_copies":2622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1311,"num_whiteouts":1,"num_read":651,"num_read_kb":2097275,"num_write":1747,"num_write_kb":5246033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5506,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4425,"ondisk_log_size":4425,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":11053152,"kb_used_data":10498848,"kb_used_omap":12,"kb_used_meta":554227,"kb_avail":743921568,"statfs":{"total":773094113280,"available":761775685632,"internally_reserved":0,"allocated":10750820352,"data_stored":10744589241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":567529008},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003039"},"pg_stats":[{"pgid":"17.3f","version":"110'78","reported_seq":112,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:26.914577+0000","last_change":"2024-01-15T18:24:26.914597+0000","last_active":"2024-01-15T18:24:26.914577+0000","last_peered":"2024-01-15T18:24:26.914577+0000","last_clean":"2024-01-15T18:24:26.914577+0000","last_became_active":"2024-01-15T18:18:33.168500+0000","last_became_peered":"2024-01-15T18:18:33.168500+0000","last_unstale":"2024-01-15T18:24:26.914577+0000","last_undegraded":"2024-01-15T18:24:26.914577+0000","last_fullsized":"2024-01-15T18:24:26.914577+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'78","last_scrub_stamp":"2024-01-15T18:24:26.914501+0000","last_deep_scrub":"110'78","last_deep_scrub_stamp":"2024-01-15T18:24:26.914501+0000","last_clean_scrub_stamp":"2024-01-15T18:24:26.914501+0000","objects_scrubbed":25,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:26:00.463315+0000","scrub_duration":0.159949852,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"110'64","reported_seq":96,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:18.004675+0000","last_change":"2024-01-15T18:24:18.004707+0000","last_active":"2024-01-15T18:24:18.004675+0000","last_peered":"2024-01-15T18:24:18.004675+0000","last_clean":"2024-01-15T18:24:18.004675+0000","last_became_active":"2024-01-15T18:18:33.151209+0000","last_became_peered":"2024-01-15T18:18:33.151209+0000","last_unstale":"2024-01-15T18:24:18.004675+0000","last_undegraded":"2024-01-15T18:24:18.004675+0000","last_fullsized":"2024-01-15T18:24:18.004675+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'64","last_scrub_stamp":"2024-01-15T18:24:18.004594+0000","last_deep_scrub":"110'64","last_deep_scrub_stamp":"2024-01-15T18:24:18.004594+0000","last_clean_scrub_stamp":"2024-01-15T18:24:18.004594+0000","objects_scrubbed":19,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:58:47.994108+0000","scrub_duration":0.121021875,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"110'63","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:15.376299+0000","last_change":"2024-01-15T18:24:15.376335+0000","last_active":"2024-01-15T18:24:15.376299+0000","last_peered":"2024-01-15T18:24:15.376299+0000","last_clean":"2024-01-15T18:24:15.376299+0000","last_became_active":"2024-01-15T18:18:33.162700+0000","last_became_peered":"2024-01-15T18:18:33.162700+0000","last_unstale":"2024-01-15T18:24:15.376299+0000","last_undegraded":"2024-01-15T18:24:15.376299+0000","last_fullsized":"2024-01-15T18:24:15.376299+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'63","last_scrub_stamp":"2024-01-15T18:24:15.376222+0000","last_deep_scrub":"110'63","last_deep_scrub_stamp":"2024-01-15T18:24:15.376222+0000","last_clean_scrub_stamp":"2024-01-15T18:24:15.376222+0000","objects_scrubbed":19,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:50:26.909076+0000","scrub_duration":0.130268359,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"110'62","reported_seq":93,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:37.761794+0000","last_change":"2024-01-15T18:23:37.761794+0000","last_active":"2024-01-15T18:23:37.761794+0000","last_peered":"2024-01-15T18:23:37.761794+0000","last_clean":"2024-01-15T18:23:37.761794+0000","last_became_active":"2024-01-15T18:18:33.160460+0000","last_became_peered":"2024-01-15T18:18:33.160460+0000","last_unstale":"2024-01-15T18:23:37.761794+0000","last_undegraded":"2024-01-15T18:23:37.761794+0000","last_fullsized":"2024-01-15T18:23:37.761794+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'62","last_scrub_stamp":"2024-01-15T18:23:37.761665+0000","last_deep_scrub":"110'62","last_deep_scrub_stamp":"2024-01-15T18:23:37.761665+0000","last_clean_scrub_stamp":"2024-01-15T18:23:37.761665+0000","objects_scrubbed":23,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:24:28.403935+0000","scrub_duration":0.14250365100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"110'35","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:01.029619+0000","last_change":"2024-01-15T18:24:01.029682+0000","last_active":"2024-01-15T18:24:01.029619+0000","last_peered":"2024-01-15T18:24:01.029619+0000","last_clean":"2024-01-15T18:24:01.029619+0000","last_became_active":"2024-01-15T18:18:33.155656+0000","last_became_peered":"2024-01-15T18:18:33.155656+0000","last_unstale":"2024-01-15T18:24:01.029619+0000","last_undegraded":"2024-01-15T18:24:01.029619+0000","last_fullsized":"2024-01-15T18:24:01.029619+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'35","last_scrub_stamp":"2024-01-15T18:24:01.029508+0000","last_deep_scrub":"110'35","last_deep_scrub_stamp":"2024-01-15T18:24:01.029508+0000","last_clean_scrub_stamp":"2024-01-15T18:24:01.029508+0000","objects_scrubbed":10,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:30:03.396033+0000","scrub_duration":0.072926557000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"110'69","reported_seq":104,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:09.399789+0000","last_change":"2024-01-15T18:24:09.399859+0000","last_active":"2024-01-15T18:24:09.399789+0000","last_peered":"2024-01-15T18:24:09.399789+0000","last_clean":"2024-01-15T18:24:09.399789+0000","last_became_active":"2024-01-15T18:18:33.151701+0000","last_became_peered":"2024-01-15T18:18:33.151701+0000","last_unstale":"2024-01-15T18:24:09.399789+0000","last_undegraded":"2024-01-15T18:24:09.399789+0000","last_fullsized":"2024-01-15T18:24:09.399789+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'69","last_scrub_stamp":"2024-01-15T18:24:09.399699+0000","last_deep_scrub":"110'69","last_deep_scrub_stamp":"2024-01-15T18:24:09.399699+0000","last_clean_scrub_stamp":"2024-01-15T18:24:09.399699+0000","objects_scrubbed":21,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:03:09.635967+0000","scrub_duration":0.161292621,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"110'58","reported_seq":86,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:59.512912+0000","last_change":"2024-01-15T18:23:59.512955+0000","last_active":"2024-01-15T18:23:59.512912+0000","last_peered":"2024-01-15T18:23:59.512912+0000","last_clean":"2024-01-15T18:23:59.512912+0000","last_became_active":"2024-01-15T18:18:33.153264+0000","last_became_peered":"2024-01-15T18:18:33.153264+0000","last_unstale":"2024-01-15T18:23:59.512912+0000","last_undegraded":"2024-01-15T18:23:59.512912+0000","last_fullsized":"2024-01-15T18:23:59.512912+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'58","last_scrub_stamp":"2024-01-15T18:23:59.512827+0000","last_deep_scrub":"110'58","last_deep_scrub_stamp":"2024-01-15T18:23:59.512827+0000","last_clean_scrub_stamp":"2024-01-15T18:23:59.512827+0000","objects_scrubbed":17,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:46:09.671279+0000","scrub_duration":0.12555955899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"110'61","reported_seq":93,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:56.297341+0000","last_change":"2024-01-15T18:23:56.297378+0000","last_active":"2024-01-15T18:23:56.297341+0000","last_peered":"2024-01-15T18:23:56.297341+0000","last_clean":"2024-01-15T18:23:56.297341+0000","last_became_active":"2024-01-15T18:18:33.163896+0000","last_became_peered":"2024-01-15T18:18:33.163896+0000","last_unstale":"2024-01-15T18:23:56.297341+0000","last_undegraded":"2024-01-15T18:23:56.297341+0000","last_fullsized":"2024-01-15T18:23:56.297341+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'61","last_scrub_stamp":"2024-01-15T18:23:56.297250+0000","last_deep_scrub":"110'61","last_deep_scrub_stamp":"2024-01-15T18:23:56.297250+0000","last_clean_scrub_stamp":"2024-01-15T18:23:56.297250+0000","objects_scrubbed":19,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:44:20.320452+0000","scrub_duration":0.120656875,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"110'68","reported_seq":100,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:25.873450+0000","last_change":"2024-01-15T18:24:25.873484+0000","last_active":"2024-01-15T18:24:25.873450+0000","last_peered":"2024-01-15T18:24:25.873450+0000","last_clean":"2024-01-15T18:24:25.873450+0000","last_became_active":"2024-01-15T18:18:33.158399+0000","last_became_peered":"2024-01-15T18:18:33.158399+0000","last_unstale":"2024-01-15T18:24:25.873450+0000","last_undegraded":"2024-01-15T18:24:25.873450+0000","last_fullsized":"2024-01-15T18:24:25.873450+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'68","last_scrub_stamp":"2024-01-15T18:24:25.873379+0000","last_deep_scrub":"110'68","last_deep_scrub_stamp":"2024-01-15T18:24:25.873379+0000","last_clean_scrub_stamp":"2024-01-15T18:24:25.873379+0000","objects_scrubbed":21,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:24:41.709267+0000","scrub_duration":0.14373503100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"110'64","reported_seq":98,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:30.896983+0000","last_change":"2024-01-15T18:24:30.897016+0000","last_active":"2024-01-15T18:24:30.896983+0000","last_peered":"2024-01-15T18:24:30.896983+0000","last_clean":"2024-01-15T18:24:30.896983+0000","last_became_active":"2024-01-15T18:18:33.159562+0000","last_became_peered":"2024-01-15T18:18:33.159562+0000","last_unstale":"2024-01-15T18:24:30.896983+0000","last_undegraded":"2024-01-15T18:24:30.896983+0000","last_fullsized":"2024-01-15T18:24:30.896983+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'64","last_scrub_stamp":"2024-01-15T18:24:30.896894+0000","last_deep_scrub":"110'64","last_deep_scrub_stamp":"2024-01-15T18:24:30.896894+0000","last_clean_scrub_stamp":"2024-01-15T18:24:30.896894+0000","objects_scrubbed":17,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:37:43.146095+0000","scrub_duration":0.12550940899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"110'75","reported_seq":112,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:11.106393+0000","last_change":"2024-01-15T18:24:11.106416+0000","last_active":"2024-01-15T18:24:11.106393+0000","last_peered":"2024-01-15T18:24:11.106393+0000","last_clean":"2024-01-15T18:24:11.106393+0000","last_became_active":"2024-01-15T18:18:33.154972+0000","last_became_peered":"2024-01-15T18:18:33.154972+0000","last_unstale":"2024-01-15T18:24:11.106393+0000","last_undegraded":"2024-01-15T18:24:11.106393+0000","last_fullsized":"2024-01-15T18:24:11.106393+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'75","last_scrub_stamp":"2024-01-15T18:24:11.106322+0000","last_deep_scrub":"110'75","last_deep_scrub_stamp":"2024-01-15T18:24:11.106322+0000","last_clean_scrub_stamp":"2024-01-15T18:24:11.106322+0000","objects_scrubbed":28,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:48:48.772386+0000","scrub_duration":0.19587069500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"110'76","reported_seq":118,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:19.874336+0000","last_change":"2024-01-15T18:24:19.874360+0000","last_active":"2024-01-15T18:24:19.874336+0000","last_peered":"2024-01-15T18:24:19.874336+0000","last_clean":"2024-01-15T18:24:19.874336+0000","last_became_active":"2024-01-15T18:18:33.158064+0000","last_became_peered":"2024-01-15T18:18:33.158064+0000","last_unstale":"2024-01-15T18:24:19.874336+0000","last_undegraded":"2024-01-15T18:24:19.874336+0000","last_fullsized":"2024-01-15T18:24:19.874336+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'76","last_scrub_stamp":"2024-01-15T18:24:19.874264+0000","last_deep_scrub":"110'76","last_deep_scrub_stamp":"2024-01-15T18:24:19.874264+0000","last_clean_scrub_stamp":"2024-01-15T18:24:19.874264+0000","objects_scrubbed":28,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:00:28.748941+0000","scrub_duration":0.19137116100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":16,"num_read_kb":61441,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"110'57","reported_seq":92,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:25.401362+0000","last_change":"2024-01-15T18:24:25.401403+0000","last_active":"2024-01-15T18:24:25.401362+0000","last_peered":"2024-01-15T18:24:25.401362+0000","last_clean":"2024-01-15T18:24:25.401362+0000","last_became_active":"2024-01-15T18:18:33.163610+0000","last_became_peered":"2024-01-15T18:18:33.163610+0000","last_unstale":"2024-01-15T18:24:25.401362+0000","last_undegraded":"2024-01-15T18:24:25.401362+0000","last_fullsized":"2024-01-15T18:24:25.401362+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'57","last_scrub_stamp":"2024-01-15T18:24:25.401279+0000","last_deep_scrub":"110'57","last_deep_scrub_stamp":"2024-01-15T18:24:25.401279+0000","last_clean_scrub_stamp":"2024-01-15T18:24:25.401279+0000","objects_scrubbed":19,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:02:39.696998+0000","scrub_duration":0.12195774299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"110'70","reported_seq":102,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:52.349564+0000","last_change":"2024-01-15T18:23:52.349588+0000","last_active":"2024-01-15T18:23:52.349564+0000","last_peered":"2024-01-15T18:23:52.349564+0000","last_clean":"2024-01-15T18:23:52.349564+0000","last_became_active":"2024-01-15T18:18:33.169530+0000","last_became_peered":"2024-01-15T18:18:33.169530+0000","last_unstale":"2024-01-15T18:23:52.349564+0000","last_undegraded":"2024-01-15T18:23:52.349564+0000","last_fullsized":"2024-01-15T18:23:52.349564+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'70","last_scrub_stamp":"2024-01-15T18:23:52.349480+0000","last_deep_scrub":"110'70","last_deep_scrub_stamp":"2024-01-15T18:23:52.349480+0000","last_clean_scrub_stamp":"2024-01-15T18:23:52.349480+0000","objects_scrubbed":27,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:02.669418+0000","scrub_duration":0.19608498999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"110'41","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:53.303856+0000","last_change":"2024-01-15T18:23:53.303892+0000","last_active":"2024-01-15T18:23:53.303856+0000","last_peered":"2024-01-15T18:23:53.303856+0000","last_clean":"2024-01-15T18:23:53.303856+0000","last_became_active":"2024-01-15T18:18:33.151040+0000","last_became_peered":"2024-01-15T18:18:33.151040+0000","last_unstale":"2024-01-15T18:23:53.303856+0000","last_undegraded":"2024-01-15T18:23:53.303856+0000","last_fullsized":"2024-01-15T18:23:53.303856+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'41","last_scrub_stamp":"2024-01-15T18:23:53.303773+0000","last_deep_scrub":"110'41","last_deep_scrub_stamp":"2024-01-15T18:23:53.303773+0000","last_clean_scrub_stamp":"2024-01-15T18:23:53.303773+0000","objects_scrubbed":12,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:12:23.550874+0000","scrub_duration":0.088010931000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"110'60","reported_seq":87,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:35.900714+0000","last_change":"2024-01-15T18:24:35.900749+0000","last_active":"2024-01-15T18:24:35.900714+0000","last_peered":"2024-01-15T18:24:35.900714+0000","last_clean":"2024-01-15T18:24:35.900714+0000","last_became_active":"2024-01-15T18:18:33.166014+0000","last_became_peered":"2024-01-15T18:18:33.166014+0000","last_unstale":"2024-01-15T18:24:35.900714+0000","last_undegraded":"2024-01-15T18:24:35.900714+0000","last_fullsized":"2024-01-15T18:24:35.900714+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'60","last_scrub_stamp":"2024-01-15T18:24:35.900638+0000","last_deep_scrub":"110'60","last_deep_scrub_stamp":"2024-01-15T18:24:35.900638+0000","last_clean_scrub_stamp":"2024-01-15T18:24:35.900638+0000","objects_scrubbed":18,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:13:02.021725+0000","scrub_duration":0.11303044800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"110'66","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:15.009110+0000","last_change":"2024-01-15T18:24:15.009148+0000","last_active":"2024-01-15T18:24:15.009110+0000","last_peered":"2024-01-15T18:24:15.009110+0000","last_clean":"2024-01-15T18:24:15.009110+0000","last_became_active":"2024-01-15T18:18:33.167975+0000","last_became_peered":"2024-01-15T18:18:33.167975+0000","last_unstale":"2024-01-15T18:24:15.009110+0000","last_undegraded":"2024-01-15T18:24:15.009110+0000","last_fullsized":"2024-01-15T18:24:15.009110+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'66","last_scrub_stamp":"2024-01-15T18:24:15.009030+0000","last_deep_scrub":"110'66","last_deep_scrub_stamp":"2024-01-15T18:24:15.009030+0000","last_clean_scrub_stamp":"2024-01-15T18:24:15.009030+0000","objects_scrubbed":22,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:29.066710+0000","scrub_duration":0.16366056400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"110'67","reported_seq":104,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:49.598996+0000","last_change":"2024-01-15T18:23:49.599036+0000","last_active":"2024-01-15T18:23:49.598996+0000","last_peered":"2024-01-15T18:23:49.598996+0000","last_clean":"2024-01-15T18:23:49.598996+0000","last_became_active":"2024-01-15T18:18:33.154816+0000","last_became_peered":"2024-01-15T18:18:33.154816+0000","last_unstale":"2024-01-15T18:23:49.598996+0000","last_undegraded":"2024-01-15T18:23:49.598996+0000","last_fullsized":"2024-01-15T18:23:49.598996+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'67","last_scrub_stamp":"2024-01-15T18:23:49.598911+0000","last_deep_scrub":"110'67","last_deep_scrub_stamp":"2024-01-15T18:23:49.598911+0000","last_clean_scrub_stamp":"2024-01-15T18:23:49.598911+0000","objects_scrubbed":21,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:43:00.465860+0000","scrub_duration":0.149097972,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"110'65","reported_seq":94,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:21.470285+0000","last_change":"2024-01-15T18:24:21.470325+0000","last_active":"2024-01-15T18:24:21.470285+0000","last_peered":"2024-01-15T18:24:21.470285+0000","last_clean":"2024-01-15T18:24:21.470285+0000","last_became_active":"2024-01-15T18:18:33.163421+0000","last_became_peered":"2024-01-15T18:18:33.163421+0000","last_unstale":"2024-01-15T18:24:21.470285+0000","last_undegraded":"2024-01-15T18:24:21.470285+0000","last_fullsized":"2024-01-15T18:24:21.470285+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'65","last_scrub_stamp":"2024-01-15T18:24:21.470200+0000","last_deep_scrub":"110'65","last_deep_scrub_stamp":"2024-01-15T18:24:21.470200+0000","last_clean_scrub_stamp":"2024-01-15T18:24:21.470200+0000","objects_scrubbed":21,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:45:19.514008+0000","scrub_duration":0.14111131800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"110'63","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:02.045282+0000","last_change":"2024-01-15T18:24:02.045321+0000","last_active":"2024-01-15T18:24:02.045282+0000","last_peered":"2024-01-15T18:24:02.045282+0000","last_clean":"2024-01-15T18:24:02.045282+0000","last_became_active":"2024-01-15T18:18:33.160047+0000","last_became_peered":"2024-01-15T18:18:33.160047+0000","last_unstale":"2024-01-15T18:24:02.045282+0000","last_undegraded":"2024-01-15T18:24:02.045282+0000","last_fullsized":"2024-01-15T18:24:02.045282+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'63","last_scrub_stamp":"2024-01-15T18:24:02.045207+0000","last_deep_scrub":"110'63","last_deep_scrub_stamp":"2024-01-15T18:24:02.045207+0000","last_clean_scrub_stamp":"2024-01-15T18:24:02.045207+0000","objects_scrubbed":17,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:23:06.679457+0000","scrub_duration":0.11837629199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"110'76","reported_seq":114,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:56.964314+0000","last_change":"2024-01-15T18:23:56.964335+0000","last_active":"2024-01-15T18:23:56.964314+0000","last_peered":"2024-01-15T18:23:56.964314+0000","last_clean":"2024-01-15T18:23:56.964314+0000","last_became_active":"2024-01-15T18:18:33.152128+0000","last_became_peered":"2024-01-15T18:18:33.152128+0000","last_unstale":"2024-01-15T18:23:56.964314+0000","last_undegraded":"2024-01-15T18:23:56.964314+0000","last_fullsized":"2024-01-15T18:23:56.964314+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'76","last_scrub_stamp":"2024-01-15T18:23:56.964221+0000","last_deep_scrub":"110'76","last_deep_scrub_stamp":"2024-01-15T18:23:56.964221+0000","last_clean_scrub_stamp":"2024-01-15T18:23:56.964221+0000","objects_scrubbed":26,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:39:29.172538+0000","scrub_duration":0.16674940499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"110'68","reported_seq":100,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:14.112722+0000","last_change":"2024-01-15T18:24:14.112757+0000","last_active":"2024-01-15T18:24:14.112722+0000","last_peered":"2024-01-15T18:24:14.112722+0000","last_clean":"2024-01-15T18:24:14.112722+0000","last_became_active":"2024-01-15T18:18:33.163252+0000","last_became_peered":"2024-01-15T18:18:33.163252+0000","last_unstale":"2024-01-15T18:24:14.112722+0000","last_undegraded":"2024-01-15T18:24:14.112722+0000","last_fullsized":"2024-01-15T18:24:14.112722+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'68","last_scrub_stamp":"2024-01-15T18:24:14.112643+0000","last_deep_scrub":"110'68","last_deep_scrub_stamp":"2024-01-15T18:24:14.112643+0000","last_clean_scrub_stamp":"2024-01-15T18:24:14.112643+0000","objects_scrubbed":21,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:08:51.346952+0000","scrub_duration":0.15134489700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":24578,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"110'50","reported_seq":83,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:50.982646+0000","last_change":"2024-01-15T18:23:50.982684+0000","last_active":"2024-01-15T18:23:50.982646+0000","last_peered":"2024-01-15T18:23:50.982646+0000","last_clean":"2024-01-15T18:23:50.982646+0000","last_became_active":"2024-01-15T18:18:33.159557+0000","last_became_peered":"2024-01-15T18:18:33.159557+0000","last_unstale":"2024-01-15T18:23:50.982646+0000","last_undegraded":"2024-01-15T18:23:50.982646+0000","last_fullsized":"2024-01-15T18:23:50.982646+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'50","last_scrub_stamp":"2024-01-15T18:23:50.982553+0000","last_deep_scrub":"110'50","last_deep_scrub_stamp":"2024-01-15T18:23:50.982553+0000","last_clean_scrub_stamp":"2024-01-15T18:23:50.982553+0000","objects_scrubbed":15,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:10:19.036866+0000","scrub_duration":0.10536667600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"110'61","reported_seq":92,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:03.046163+0000","last_change":"2024-01-15T18:24:03.046204+0000","last_active":"2024-01-15T18:24:03.046163+0000","last_peered":"2024-01-15T18:24:03.046163+0000","last_clean":"2024-01-15T18:24:03.046163+0000","last_became_active":"2024-01-15T18:18:33.165600+0000","last_became_peered":"2024-01-15T18:18:33.165600+0000","last_unstale":"2024-01-15T18:24:03.046163+0000","last_undegraded":"2024-01-15T18:24:03.046163+0000","last_fullsized":"2024-01-15T18:24:03.046163+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'61","last_scrub_stamp":"2024-01-15T18:24:03.046071+0000","last_deep_scrub":"110'61","last_deep_scrub_stamp":"2024-01-15T18:24:03.046071+0000","last_clean_scrub_stamp":"2024-01-15T18:24:03.046071+0000","objects_scrubbed":18,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:10:31.100384+0000","scrub_duration":0.11690018100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"110'56","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:10.038734+0000","last_change":"2024-01-15T18:24:10.038767+0000","last_active":"2024-01-15T18:24:10.038734+0000","last_peered":"2024-01-15T18:24:10.038734+0000","last_clean":"2024-01-15T18:24:10.038734+0000","last_became_active":"2024-01-15T18:18:33.168349+0000","last_became_peered":"2024-01-15T18:18:33.168349+0000","last_unstale":"2024-01-15T18:24:10.038734+0000","last_undegraded":"2024-01-15T18:24:10.038734+0000","last_fullsized":"2024-01-15T18:24:10.038734+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'56","last_scrub_stamp":"2024-01-15T18:24:10.038661+0000","last_deep_scrub":"110'56","last_deep_scrub_stamp":"2024-01-15T18:24:10.038661+0000","last_clean_scrub_stamp":"2024-01-15T18:24:10.038661+0000","objects_scrubbed":18,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:47:47.509239+0000","scrub_duration":0.12652001600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"110'76","reported_seq":108,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:20.860412+0000","last_change":"2024-01-15T18:24:20.860446+0000","last_active":"2024-01-15T18:24:20.860412+0000","last_peered":"2024-01-15T18:24:20.860412+0000","last_clean":"2024-01-15T18:24:20.860412+0000","last_became_active":"2024-01-15T18:18:33.158946+0000","last_became_peered":"2024-01-15T18:18:33.158946+0000","last_unstale":"2024-01-15T18:24:20.860412+0000","last_undegraded":"2024-01-15T18:24:20.860412+0000","last_fullsized":"2024-01-15T18:24:20.860412+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'76","last_scrub_stamp":"2024-01-15T18:24:20.860342+0000","last_deep_scrub":"110'76","last_deep_scrub_stamp":"2024-01-15T18:24:20.860342+0000","last_clean_scrub_stamp":"2024-01-15T18:24:20.860342+0000","objects_scrubbed":23,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:07:04.860496+0000","scrub_duration":0.14412594100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"110'58","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:06.379649+0000","last_change":"2024-01-15T18:24:06.379683+0000","last_active":"2024-01-15T18:24:06.379649+0000","last_peered":"2024-01-15T18:24:06.379649+0000","last_clean":"2024-01-15T18:24:06.379649+0000","last_became_active":"2024-01-15T18:18:33.161463+0000","last_became_peered":"2024-01-15T18:18:33.161463+0000","last_unstale":"2024-01-15T18:24:06.379649+0000","last_undegraded":"2024-01-15T18:24:06.379649+0000","last_fullsized":"2024-01-15T18:24:06.379649+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'58","last_scrub_stamp":"2024-01-15T18:24:06.379571+0000","last_deep_scrub":"110'58","last_deep_scrub_stamp":"2024-01-15T18:24:06.379571+0000","last_clean_scrub_stamp":"2024-01-15T18:24:06.379571+0000","objects_scrubbed":21,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:37:14.239420+0000","scrub_duration":0.134830067,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886260,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":28673,"num_write":21,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"110'55","reported_seq":82,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:07.024489+0000","last_change":"2024-01-15T18:24:07.024520+0000","last_active":"2024-01-15T18:24:07.024489+0000","last_peered":"2024-01-15T18:24:07.024489+0000","last_clean":"2024-01-15T18:24:07.024489+0000","last_became_active":"2024-01-15T18:18:33.162669+0000","last_became_peered":"2024-01-15T18:18:33.162669+0000","last_unstale":"2024-01-15T18:24:07.024489+0000","last_undegraded":"2024-01-15T18:24:07.024489+0000","last_fullsized":"2024-01-15T18:24:07.024489+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'55","last_scrub_stamp":"2024-01-15T18:24:07.024419+0000","last_deep_scrub":"110'55","last_deep_scrub_stamp":"2024-01-15T18:24:07.024419+0000","last_clean_scrub_stamp":"2024-01-15T18:24:07.024419+0000","objects_scrubbed":17,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:23:11.658669+0000","scrub_duration":0.12702018900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"110'72","reported_seq":106,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:14.039459+0000","last_change":"2024-01-15T18:24:14.039500+0000","last_active":"2024-01-15T18:24:14.039459+0000","last_peered":"2024-01-15T18:24:14.039459+0000","last_clean":"2024-01-15T18:24:14.039459+0000","last_became_active":"2024-01-15T18:18:33.160099+0000","last_became_peered":"2024-01-15T18:18:33.160099+0000","last_unstale":"2024-01-15T18:24:14.039459+0000","last_undegraded":"2024-01-15T18:24:14.039459+0000","last_fullsized":"2024-01-15T18:24:14.039459+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'72","last_scrub_stamp":"2024-01-15T18:24:14.039336+0000","last_deep_scrub":"110'72","last_deep_scrub_stamp":"2024-01-15T18:24:14.039336+0000","last_clean_scrub_stamp":"2024-01-15T18:24:14.039336+0000","objects_scrubbed":26,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:00:56.503350+0000","scrub_duration":0.19647456399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"110'57","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:12.460267+0000","last_change":"2024-01-15T18:24:12.460306+0000","last_active":"2024-01-15T18:24:12.460267+0000","last_peered":"2024-01-15T18:24:12.460267+0000","last_clean":"2024-01-15T18:24:12.460267+0000","last_became_active":"2024-01-15T18:18:33.160725+0000","last_became_peered":"2024-01-15T18:18:33.160725+0000","last_unstale":"2024-01-15T18:24:12.460267+0000","last_undegraded":"2024-01-15T18:24:12.460267+0000","last_fullsized":"2024-01-15T18:24:12.460267+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'57","last_scrub_stamp":"2024-01-15T18:24:12.460185+0000","last_deep_scrub":"110'57","last_deep_scrub_stamp":"2024-01-15T18:24:12.460185+0000","last_clean_scrub_stamp":"2024-01-15T18:24:12.460185+0000","objects_scrubbed":16,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:02:30.578846+0000","scrub_duration":0.105574268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"110'85","reported_seq":122,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:40.714947+0000","last_change":"2024-01-15T18:23:40.714970+0000","last_active":"2024-01-15T18:23:40.714947+0000","last_peered":"2024-01-15T18:23:40.714947+0000","last_clean":"2024-01-15T18:23:40.714947+0000","last_became_active":"2024-01-15T18:18:33.158701+0000","last_became_peered":"2024-01-15T18:18:33.158701+0000","last_unstale":"2024-01-15T18:23:40.714947+0000","last_undegraded":"2024-01-15T18:23:40.714947+0000","last_fullsized":"2024-01-15T18:23:40.714947+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'85","last_scrub_stamp":"2024-01-15T18:23:40.714867+0000","last_deep_scrub":"110'85","last_deep_scrub_stamp":"2024-01-15T18:23:40.714867+0000","last_clean_scrub_stamp":"2024-01-15T18:23:40.714867+0000","objects_scrubbed":26,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:21.790890+0000","scrub_duration":0.16315633500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"110'76","reported_seq":107,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:03.367042+0000","last_change":"2024-01-15T18:24:03.367103+0000","last_active":"2024-01-15T18:24:03.367042+0000","last_peered":"2024-01-15T18:24:03.367042+0000","last_clean":"2024-01-15T18:24:03.367042+0000","last_became_active":"2024-01-15T18:18:33.151250+0000","last_became_peered":"2024-01-15T18:18:33.151250+0000","last_unstale":"2024-01-15T18:24:03.367042+0000","last_undegraded":"2024-01-15T18:24:03.367042+0000","last_fullsized":"2024-01-15T18:24:03.367042+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'76","last_scrub_stamp":"2024-01-15T18:24:03.366959+0000","last_deep_scrub":"110'76","last_deep_scrub_stamp":"2024-01-15T18:24:03.366959+0000","last_clean_scrub_stamp":"2024-01-15T18:24:03.366959+0000","objects_scrubbed":22,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:07:35.792913+0000","scrub_duration":0.156425128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"110'49","reported_seq":80,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:05.312788+0000","last_change":"2024-01-15T18:24:05.312788+0000","last_active":"2024-01-15T18:24:05.312788+0000","last_peered":"2024-01-15T18:24:05.312788+0000","last_clean":"2024-01-15T18:24:05.312788+0000","last_became_active":"2024-01-15T18:18:33.154569+0000","last_became_peered":"2024-01-15T18:18:33.154569+0000","last_unstale":"2024-01-15T18:24:05.312788+0000","last_undegraded":"2024-01-15T18:24:05.312788+0000","last_fullsized":"2024-01-15T18:24:05.312788+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'49","last_scrub_stamp":"2024-01-15T18:24:05.312682+0000","last_deep_scrub":"110'49","last_deep_scrub_stamp":"2024-01-15T18:24:05.312682+0000","last_clean_scrub_stamp":"2024-01-15T18:24:05.312682+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:06:44.457178+0000","scrub_duration":0.10915641800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"110'6","reported_seq":31,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:52.319457+0000","last_change":"2024-01-15T18:23:52.319495+0000","last_active":"2024-01-15T18:23:52.319457+0000","last_peered":"2024-01-15T18:23:52.319457+0000","last_clean":"2024-01-15T18:23:52.319457+0000","last_became_active":"2024-01-15T18:18:32.216337+0000","last_became_peered":"2024-01-15T18:18:32.216337+0000","last_unstale":"2024-01-15T18:23:52.319457+0000","last_undegraded":"2024-01-15T18:23:52.319457+0000","last_fullsized":"2024-01-15T18:23:52.319457+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'6","last_scrub_stamp":"2024-01-15T18:23:52.319346+0000","last_deep_scrub":"110'6","last_deep_scrub_stamp":"2024-01-15T18:23:52.319346+0000","last_clean_scrub_stamp":"2024-01-15T18:23:52.319346+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:00:46.740483+0000","scrub_duration":0.003716685,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"110'66","reported_seq":98,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:57.637128+0000","last_change":"2024-01-15T18:23:57.637165+0000","last_active":"2024-01-15T18:23:57.637128+0000","last_peered":"2024-01-15T18:23:57.637128+0000","last_clean":"2024-01-15T18:23:57.637128+0000","last_became_active":"2024-01-15T18:18:33.166726+0000","last_became_peered":"2024-01-15T18:18:33.166726+0000","last_unstale":"2024-01-15T18:23:57.637128+0000","last_undegraded":"2024-01-15T18:23:57.637128+0000","last_fullsized":"2024-01-15T18:23:57.637128+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'66","last_scrub_stamp":"2024-01-15T18:23:57.637053+0000","last_deep_scrub":"110'66","last_deep_scrub_stamp":"2024-01-15T18:23:57.637053+0000","last_clean_scrub_stamp":"2024-01-15T18:23:57.637053+0000","objects_scrubbed":22,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:28:55.118910+0000","scrub_duration":0.15067504100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"107'1","reported_seq":27,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:49.184542+0000","last_change":"2024-01-15T18:23:49.184542+0000","last_active":"2024-01-15T18:23:49.184542+0000","last_peered":"2024-01-15T18:23:49.184542+0000","last_clean":"2024-01-15T18:23:49.184542+0000","last_became_active":"2024-01-15T18:18:32.219596+0000","last_became_peered":"2024-01-15T18:18:32.219596+0000","last_unstale":"2024-01-15T18:23:49.184542+0000","last_undegraded":"2024-01-15T18:23:49.184542+0000","last_fullsized":"2024-01-15T18:23:49.184542+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'1","last_scrub_stamp":"2024-01-15T18:23:49.184382+0000","last_deep_scrub":"107'1","last_deep_scrub_stamp":"2024-01-15T18:23:49.184382+0000","last_clean_scrub_stamp":"2024-01-15T18:23:49.184382+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:22:53.818633+0000","scrub_duration":0.0021746040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"110'59","reported_seq":94,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:57.284418+0000","last_change":"2024-01-15T18:23:57.284475+0000","last_active":"2024-01-15T18:23:57.284418+0000","last_peered":"2024-01-15T18:23:57.284418+0000","last_clean":"2024-01-15T18:23:57.284418+0000","last_became_active":"2024-01-15T18:18:33.150429+0000","last_became_peered":"2024-01-15T18:18:33.150429+0000","last_unstale":"2024-01-15T18:23:57.284418+0000","last_undegraded":"2024-01-15T18:23:57.284418+0000","last_fullsized":"2024-01-15T18:23:57.284418+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'59","last_scrub_stamp":"2024-01-15T18:23:57.284332+0000","last_deep_scrub":"110'59","last_deep_scrub_stamp":"2024-01-15T18:23:57.284332+0000","last_clean_scrub_stamp":"2024-01-15T18:23:57.284332+0000","objects_scrubbed":19,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:18:31.335937+0000","scrub_duration":0.13278458600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"110'4","reported_seq":29,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:52.947456+0000","last_change":"2024-01-15T18:23:52.947480+0000","last_active":"2024-01-15T18:23:52.947456+0000","last_peered":"2024-01-15T18:23:52.947456+0000","last_clean":"2024-01-15T18:23:52.947456+0000","last_became_active":"2024-01-15T18:18:32.212071+0000","last_became_peered":"2024-01-15T18:18:32.212071+0000","last_unstale":"2024-01-15T18:23:52.947456+0000","last_undegraded":"2024-01-15T18:23:52.947456+0000","last_fullsized":"2024-01-15T18:23:52.947456+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'4","last_scrub_stamp":"2024-01-15T18:23:52.947355+0000","last_deep_scrub":"110'4","last_deep_scrub_stamp":"2024-01-15T18:23:52.947355+0000","last_clean_scrub_stamp":"2024-01-15T18:23:52.947355+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:38:46.837594+0000","scrub_duration":0.0037854780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"110'79","reported_seq":114,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:32.452932+0000","last_change":"2024-01-15T18:24:32.452972+0000","last_active":"2024-01-15T18:24:32.452932+0000","last_peered":"2024-01-15T18:24:32.452932+0000","last_clean":"2024-01-15T18:24:32.452932+0000","last_became_active":"2024-01-15T18:18:33.165258+0000","last_became_peered":"2024-01-15T18:18:33.165258+0000","last_unstale":"2024-01-15T18:24:32.452932+0000","last_undegraded":"2024-01-15T18:24:32.452932+0000","last_fullsized":"2024-01-15T18:24:32.452932+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'79","last_scrub_stamp":"2024-01-15T18:24:32.452859+0000","last_deep_scrub":"110'79","last_deep_scrub_stamp":"2024-01-15T18:24:32.452859+0000","last_clean_scrub_stamp":"2024-01-15T18:24:32.452859+0000","objects_scrubbed":23,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:47:24.604748+0000","scrub_duration":0.14125531199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"110'3","reported_seq":28,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:56.916317+0000","last_change":"2024-01-15T18:23:56.916341+0000","last_active":"2024-01-15T18:23:56.916317+0000","last_peered":"2024-01-15T18:23:56.916317+0000","last_clean":"2024-01-15T18:23:56.916317+0000","last_became_active":"2024-01-15T18:18:32.216492+0000","last_became_peered":"2024-01-15T18:18:32.216492+0000","last_unstale":"2024-01-15T18:23:56.916317+0000","last_undegraded":"2024-01-15T18:23:56.916317+0000","last_fullsized":"2024-01-15T18:23:56.916317+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'3","last_scrub_stamp":"2024-01-15T18:23:56.916206+0000","last_deep_scrub":"110'3","last_deep_scrub_stamp":"2024-01-15T18:23:56.916206+0000","last_clean_scrub_stamp":"2024-01-15T18:23:56.916206+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:52:35.385555+0000","scrub_duration":0.0031224619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"110'56","reported_seq":88,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:04.618644+0000","last_change":"2024-01-15T18:24:04.618679+0000","last_active":"2024-01-15T18:24:04.618644+0000","last_peered":"2024-01-15T18:24:04.618644+0000","last_clean":"2024-01-15T18:24:04.618644+0000","last_became_active":"2024-01-15T18:18:33.157687+0000","last_became_peered":"2024-01-15T18:18:33.157687+0000","last_unstale":"2024-01-15T18:24:04.618644+0000","last_undegraded":"2024-01-15T18:24:04.618644+0000","last_fullsized":"2024-01-15T18:24:04.618644+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'56","last_scrub_stamp":"2024-01-15T18:24:04.618572+0000","last_deep_scrub":"110'56","last_deep_scrub_stamp":"2024-01-15T18:24:04.618572+0000","last_clean_scrub_stamp":"2024-01-15T18:24:04.618572+0000","objects_scrubbed":17,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:14:20.064593+0000","scrub_duration":0.11968945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"107'1","reported_seq":26,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:56.330972+0000","last_change":"2024-01-15T18:23:56.330992+0000","last_active":"2024-01-15T18:23:56.330972+0000","last_peered":"2024-01-15T18:23:56.330972+0000","last_clean":"2024-01-15T18:23:56.330972+0000","last_became_active":"2024-01-15T18:18:32.214453+0000","last_became_peered":"2024-01-15T18:18:32.214453+0000","last_unstale":"2024-01-15T18:23:56.330972+0000","last_undegraded":"2024-01-15T18:23:56.330972+0000","last_fullsized":"2024-01-15T18:23:56.330972+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'1","last_scrub_stamp":"2024-01-15T18:23:56.330895+0000","last_deep_scrub":"107'1","last_deep_scrub_stamp":"2024-01-15T18:23:56.330895+0000","last_clean_scrub_stamp":"2024-01-15T18:23:56.330895+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:22:46.630675+0000","scrub_duration":0.0018859580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"110'73","reported_seq":110,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:13.026892+0000","last_change":"2024-01-15T18:24:13.026930+0000","last_active":"2024-01-15T18:24:13.026892+0000","last_peered":"2024-01-15T18:24:13.026892+0000","last_clean":"2024-01-15T18:24:13.026892+0000","last_became_active":"2024-01-15T18:18:33.153058+0000","last_became_peered":"2024-01-15T18:18:33.153058+0000","last_unstale":"2024-01-15T18:24:13.026892+0000","last_undegraded":"2024-01-15T18:24:13.026892+0000","last_fullsized":"2024-01-15T18:24:13.026892+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'73","last_scrub_stamp":"2024-01-15T18:24:13.026816+0000","last_deep_scrub":"110'73","last_deep_scrub_stamp":"2024-01-15T18:24:13.026816+0000","last_clean_scrub_stamp":"2024-01-15T18:24:13.026816+0000","objects_scrubbed":24,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:43:03.776881+0000","scrub_duration":0.14808554800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"107'1","reported_seq":26,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:42.849182+0000","last_change":"2024-01-15T18:23:42.849208+0000","last_active":"2024-01-15T18:23:42.849182+0000","last_peered":"2024-01-15T18:23:42.849182+0000","last_clean":"2024-01-15T18:23:42.849182+0000","last_became_active":"2024-01-15T18:18:32.215417+0000","last_became_peered":"2024-01-15T18:18:32.215417+0000","last_unstale":"2024-01-15T18:23:42.849182+0000","last_undegraded":"2024-01-15T18:23:42.849182+0000","last_fullsized":"2024-01-15T18:23:42.849182+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'1","last_scrub_stamp":"2024-01-15T18:23:42.849065+0000","last_deep_scrub":"107'1","last_deep_scrub_stamp":"2024-01-15T18:23:42.849065+0000","last_clean_scrub_stamp":"2024-01-15T18:23:42.849065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:40:12.007683+0000","scrub_duration":0.0030182830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"110'79","reported_seq":110,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:11.111901+0000","last_change":"2024-01-15T18:24:11.111943+0000","last_active":"2024-01-15T18:24:11.111901+0000","last_peered":"2024-01-15T18:24:11.111901+0000","last_clean":"2024-01-15T18:24:11.111901+0000","last_became_active":"2024-01-15T18:18:43.024221+0000","last_became_peered":"2024-01-15T18:18:43.024221+0000","last_unstale":"2024-01-15T18:24:11.111901+0000","last_undegraded":"2024-01-15T18:24:11.111901+0000","last_fullsized":"2024-01-15T18:24:11.111901+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'79","last_scrub_stamp":"2024-01-15T18:24:11.111818+0000","last_deep_scrub":"110'79","last_deep_scrub_stamp":"2024-01-15T18:24:11.111818+0000","last_clean_scrub_stamp":"2024-01-15T18:24:11.111818+0000","objects_scrubbed":26,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:18:45.163423+0000","scrub_duration":0.18158712699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857792,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:41.955122+0000","last_change":"2024-01-15T18:23:41.955145+0000","last_active":"2024-01-15T18:23:41.955122+0000","last_peered":"2024-01-15T18:23:41.955122+0000","last_clean":"2024-01-15T18:23:41.955122+0000","last_became_active":"2024-01-15T18:18:32.215465+0000","last_became_peered":"2024-01-15T18:18:32.215465+0000","last_unstale":"2024-01-15T18:23:41.955122+0000","last_undegraded":"2024-01-15T18:23:41.955122+0000","last_fullsized":"2024-01-15T18:23:41.955122+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:41.954982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:41.954982+0000","last_clean_scrub_stamp":"2024-01-15T18:23:41.954982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:42:52.821931+0000","scrub_duration":0.002690853,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"110'59","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:07.422578+0000","last_change":"2024-01-15T18:24:07.422633+0000","last_active":"2024-01-15T18:24:07.422578+0000","last_peered":"2024-01-15T18:24:07.422578+0000","last_clean":"2024-01-15T18:24:07.422578+0000","last_became_active":"2024-01-15T18:18:33.153610+0000","last_became_peered":"2024-01-15T18:18:33.153610+0000","last_unstale":"2024-01-15T18:24:07.422578+0000","last_undegraded":"2024-01-15T18:24:07.422578+0000","last_fullsized":"2024-01-15T18:24:07.422578+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'59","last_scrub_stamp":"2024-01-15T18:24:07.422480+0000","last_deep_scrub":"110'59","last_deep_scrub_stamp":"2024-01-15T18:24:07.422480+0000","last_clean_scrub_stamp":"2024-01-15T18:24:07.422480+0000","objects_scrubbed":16,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:44:16.329452+0000","scrub_duration":0.11664002800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:50.513004+0000","last_change":"2024-01-15T18:23:50.513024+0000","last_active":"2024-01-15T18:23:50.513004+0000","last_peered":"2024-01-15T18:23:50.513004+0000","last_clean":"2024-01-15T18:23:50.513004+0000","last_became_active":"2024-01-15T18:18:32.219555+0000","last_became_peered":"2024-01-15T18:18:32.219555+0000","last_unstale":"2024-01-15T18:23:50.513004+0000","last_undegraded":"2024-01-15T18:23:50.513004+0000","last_fullsized":"2024-01-15T18:23:50.513004+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:50.512904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:50.512904+0000","last_clean_scrub_stamp":"2024-01-15T18:23:50.512904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:03:40.746495+0000","scrub_duration":0.0020306230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"110'50","reported_seq":78,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:05.416297+0000","last_change":"2024-01-15T18:24:05.416324+0000","last_active":"2024-01-15T18:24:05.416297+0000","last_peered":"2024-01-15T18:24:05.416297+0000","last_clean":"2024-01-15T18:24:05.416297+0000","last_became_active":"2024-01-15T18:18:33.155765+0000","last_became_peered":"2024-01-15T18:18:33.155765+0000","last_unstale":"2024-01-15T18:24:05.416297+0000","last_undegraded":"2024-01-15T18:24:05.416297+0000","last_fullsized":"2024-01-15T18:24:05.416297+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'50","last_scrub_stamp":"2024-01-15T18:24:05.416219+0000","last_deep_scrub":"110'50","last_deep_scrub_stamp":"2024-01-15T18:24:05.416219+0000","last_clean_scrub_stamp":"2024-01-15T18:24:05.416219+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:10:03.756727+0000","scrub_duration":0.058782918000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"110'1","reported_seq":26,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:54.901556+0000","last_change":"2024-01-15T18:23:54.901577+0000","last_active":"2024-01-15T18:23:54.901556+0000","last_peered":"2024-01-15T18:23:54.901556+0000","last_clean":"2024-01-15T18:23:54.901556+0000","last_became_active":"2024-01-15T18:18:32.212511+0000","last_became_peered":"2024-01-15T18:18:32.212511+0000","last_unstale":"2024-01-15T18:23:54.901556+0000","last_undegraded":"2024-01-15T18:23:54.901556+0000","last_fullsized":"2024-01-15T18:23:54.901556+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'1","last_scrub_stamp":"2024-01-15T18:23:54.901458+0000","last_deep_scrub":"110'1","last_deep_scrub_stamp":"2024-01-15T18:23:54.901458+0000","last_clean_scrub_stamp":"2024-01-15T18:23:54.901458+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:49:11.641921+0000","scrub_duration":0.0018983540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"110'53","reported_seq":82,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:51.617489+0000","last_change":"2024-01-15T18:23:51.617525+0000","last_active":"2024-01-15T18:23:51.617489+0000","last_peered":"2024-01-15T18:23:51.617489+0000","last_clean":"2024-01-15T18:23:51.617489+0000","last_became_active":"2024-01-15T18:18:33.164530+0000","last_became_peered":"2024-01-15T18:18:33.164530+0000","last_unstale":"2024-01-15T18:23:51.617489+0000","last_undegraded":"2024-01-15T18:23:51.617489+0000","last_fullsized":"2024-01-15T18:23:51.617489+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'53","last_scrub_stamp":"2024-01-15T18:23:51.617414+0000","last_deep_scrub":"110'53","last_deep_scrub_stamp":"2024-01-15T18:23:51.617414+0000","last_clean_scrub_stamp":"2024-01-15T18:23:51.617414+0000","objects_scrubbed":18,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:57:57.307137+0000","scrub_duration":0.11477269399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:55.893196+0000","last_change":"2024-01-15T18:23:55.893219+0000","last_active":"2024-01-15T18:23:55.893196+0000","last_peered":"2024-01-15T18:23:55.893196+0000","last_clean":"2024-01-15T18:23:55.893196+0000","last_became_active":"2024-01-15T18:18:32.216337+0000","last_became_peered":"2024-01-15T18:18:32.216337+0000","last_unstale":"2024-01-15T18:23:55.893196+0000","last_undegraded":"2024-01-15T18:23:55.893196+0000","last_fullsized":"2024-01-15T18:23:55.893196+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:55.893099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:55.893099+0000","last_clean_scrub_stamp":"2024-01-15T18:23:55.893099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:45:26.429594+0000","scrub_duration":0.002804001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"110'65","reported_seq":100,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:15.115218+0000","last_change":"2024-01-15T18:24:15.115251+0000","last_active":"2024-01-15T18:24:15.115218+0000","last_peered":"2024-01-15T18:24:15.115218+0000","last_clean":"2024-01-15T18:24:15.115218+0000","last_became_active":"2024-01-15T18:18:33.148016+0000","last_became_peered":"2024-01-15T18:18:33.148016+0000","last_unstale":"2024-01-15T18:24:15.115218+0000","last_undegraded":"2024-01-15T18:24:15.115218+0000","last_fullsized":"2024-01-15T18:24:15.115218+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'65","last_scrub_stamp":"2024-01-15T18:24:15.115148+0000","last_deep_scrub":"110'65","last_deep_scrub_stamp":"2024-01-15T18:24:15.115148+0000","last_clean_scrub_stamp":"2024-01-15T18:24:15.115148+0000","objects_scrubbed":22,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:23:51.814230+0000","scrub_duration":0.13686477899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"110'14","reported_seq":39,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:55.542471+0000","last_change":"2024-01-15T18:23:55.542525+0000","last_active":"2024-01-15T18:23:55.542471+0000","last_peered":"2024-01-15T18:23:55.542471+0000","last_clean":"2024-01-15T18:23:55.542471+0000","last_became_active":"2024-01-15T18:18:32.214186+0000","last_became_peered":"2024-01-15T18:18:32.214186+0000","last_unstale":"2024-01-15T18:23:55.542471+0000","last_undegraded":"2024-01-15T18:23:55.542471+0000","last_fullsized":"2024-01-15T18:23:55.542471+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'14","last_scrub_stamp":"2024-01-15T18:23:55.542363+0000","last_deep_scrub":"110'14","last_deep_scrub_stamp":"2024-01-15T18:23:55.542363+0000","last_clean_scrub_stamp":"2024-01-15T18:23:55.542363+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:36:43.100265+0000","scrub_duration":0.0043649800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"110'63","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:09.440301+0000","last_change":"2024-01-15T18:24:09.440341+0000","last_active":"2024-01-15T18:24:09.440301+0000","last_peered":"2024-01-15T18:24:09.440301+0000","last_clean":"2024-01-15T18:24:09.440301+0000","last_became_active":"2024-01-15T18:18:33.158541+0000","last_became_peered":"2024-01-15T18:18:33.158541+0000","last_unstale":"2024-01-15T18:24:09.440301+0000","last_undegraded":"2024-01-15T18:24:09.440301+0000","last_fullsized":"2024-01-15T18:24:09.440301+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'63","last_scrub_stamp":"2024-01-15T18:24:09.440226+0000","last_deep_scrub":"110'63","last_deep_scrub_stamp":"2024-01-15T18:24:09.440226+0000","last_clean_scrub_stamp":"2024-01-15T18:24:09.440226+0000","objects_scrubbed":20,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:33:00.283608+0000","scrub_duration":0.16288201199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"110'190","reported_seq":215,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:39.263702+0000","last_change":"2024-01-15T18:23:53.472714+0000","last_active":"2024-01-15T18:24:39.263702+0000","last_peered":"2024-01-15T18:24:39.263702+0000","last_clean":"2024-01-15T18:24:39.263702+0000","last_became_active":"2024-01-15T18:18:32.218002+0000","last_became_peered":"2024-01-15T18:18:32.218002+0000","last_unstale":"2024-01-15T18:24:39.263702+0000","last_undegraded":"2024-01-15T18:24:39.263702+0000","last_fullsized":"2024-01-15T18:24:39.263702+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'188","last_scrub_stamp":"2024-01-15T18:23:53.472606+0000","last_deep_scrub":"110'188","last_deep_scrub_stamp":"2024-01-15T18:23:53.472606+0000","last_clean_scrub_stamp":"2024-01-15T18:23:53.472606+0000","objects_scrubbed":4,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:11:55.352315+0000","scrub_duration":0.0071479380000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":614216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":187,"num_write_kb":683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"110'65","reported_seq":95,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:05.066433+0000","last_change":"2024-01-15T18:24:05.066465+0000","last_active":"2024-01-15T18:24:05.066433+0000","last_peered":"2024-01-15T18:24:05.066433+0000","last_clean":"2024-01-15T18:24:05.066433+0000","last_became_active":"2024-01-15T18:18:33.151132+0000","last_became_peered":"2024-01-15T18:18:33.151132+0000","last_unstale":"2024-01-15T18:24:05.066433+0000","last_undegraded":"2024-01-15T18:24:05.066433+0000","last_fullsized":"2024-01-15T18:24:05.066433+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'65","last_scrub_stamp":"2024-01-15T18:24:05.066365+0000","last_deep_scrub":"110'65","last_deep_scrub_stamp":"2024-01-15T18:24:05.066365+0000","last_clean_scrub_stamp":"2024-01-15T18:24:05.066365+0000","objects_scrubbed":21,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:22:28.584664+0000","scrub_duration":0.13404809200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"110'4","reported_seq":29,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:42.234836+0000","last_change":"2024-01-15T18:23:42.234860+0000","last_active":"2024-01-15T18:23:42.234836+0000","last_peered":"2024-01-15T18:23:42.234836+0000","last_clean":"2024-01-15T18:23:42.234836+0000","last_became_active":"2024-01-15T18:18:32.215073+0000","last_became_peered":"2024-01-15T18:18:32.215073+0000","last_unstale":"2024-01-15T18:23:42.234836+0000","last_undegraded":"2024-01-15T18:23:42.234836+0000","last_fullsized":"2024-01-15T18:23:42.234836+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'4","last_scrub_stamp":"2024-01-15T18:23:42.234754+0000","last_deep_scrub":"110'4","last_deep_scrub_stamp":"2024-01-15T18:23:42.234754+0000","last_clean_scrub_stamp":"2024-01-15T18:23:42.234754+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:48:58.975217+0000","scrub_duration":0.004651957,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"110'73","reported_seq":102,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:06.069897+0000","last_change":"2024-01-15T18:24:06.069934+0000","last_active":"2024-01-15T18:24:06.069897+0000","last_peered":"2024-01-15T18:24:06.069897+0000","last_clean":"2024-01-15T18:24:06.069897+0000","last_became_active":"2024-01-15T18:18:33.161493+0000","last_became_peered":"2024-01-15T18:18:33.161493+0000","last_unstale":"2024-01-15T18:24:06.069897+0000","last_undegraded":"2024-01-15T18:24:06.069897+0000","last_fullsized":"2024-01-15T18:24:06.069897+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'73","last_scrub_stamp":"2024-01-15T18:24:06.069802+0000","last_deep_scrub":"110'73","last_deep_scrub_stamp":"2024-01-15T18:24:06.069802+0000","last_clean_scrub_stamp":"2024-01-15T18:24:06.069802+0000","objects_scrubbed":21,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:28.365170+0000","scrub_duration":0.14167099799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:43.624818+0000","last_change":"2024-01-15T18:23:43.624838+0000","last_active":"2024-01-15T18:23:43.624818+0000","last_peered":"2024-01-15T18:23:43.624818+0000","last_clean":"2024-01-15T18:23:43.624818+0000","last_became_active":"2024-01-15T18:18:32.214942+0000","last_became_peered":"2024-01-15T18:18:32.214942+0000","last_unstale":"2024-01-15T18:23:43.624818+0000","last_undegraded":"2024-01-15T18:23:43.624818+0000","last_fullsized":"2024-01-15T18:23:43.624818+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:43.624692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:43.624692+0000","last_clean_scrub_stamp":"2024-01-15T18:23:43.624692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:36:51.484541+0000","scrub_duration":0.0025066049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"110'68","reported_seq":100,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:57.083832+0000","last_change":"2024-01-15T18:23:57.083851+0000","last_active":"2024-01-15T18:23:57.083832+0000","last_peered":"2024-01-15T18:23:57.083832+0000","last_clean":"2024-01-15T18:23:57.083832+0000","last_became_active":"2024-01-15T18:18:33.145973+0000","last_became_peered":"2024-01-15T18:18:33.145973+0000","last_unstale":"2024-01-15T18:23:57.083832+0000","last_undegraded":"2024-01-15T18:23:57.083832+0000","last_fullsized":"2024-01-15T18:23:57.083832+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'68","last_scrub_stamp":"2024-01-15T18:23:57.083768+0000","last_deep_scrub":"110'68","last_deep_scrub_stamp":"2024-01-15T18:23:57.083768+0000","last_clean_scrub_stamp":"2024-01-15T18:23:57.083768+0000","objects_scrubbed":25,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:22:28.677434+0000","scrub_duration":0.150822226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663557,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"110'12","reported_seq":37,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:38.931283+0000","last_change":"2024-01-15T18:23:38.931329+0000","last_active":"2024-01-15T18:23:38.931283+0000","last_peered":"2024-01-15T18:23:38.931283+0000","last_clean":"2024-01-15T18:23:38.931283+0000","last_became_active":"2024-01-15T18:18:32.219280+0000","last_became_peered":"2024-01-15T18:18:32.219280+0000","last_unstale":"2024-01-15T18:23:38.931283+0000","last_undegraded":"2024-01-15T18:23:38.931283+0000","last_fullsized":"2024-01-15T18:23:38.931283+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'12","last_scrub_stamp":"2024-01-15T18:23:38.931162+0000","last_deep_scrub":"110'12","last_deep_scrub_stamp":"2024-01-15T18:23:38.931162+0000","last_clean_scrub_stamp":"2024-01-15T18:23:38.931162+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:57:50.305539+0000","scrub_duration":0.0047879089999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"110'34","reported_seq":62,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:07.968432+0000","last_change":"2024-01-15T18:24:07.968469+0000","last_active":"2024-01-15T18:24:07.968432+0000","last_peered":"2024-01-15T18:24:07.968432+0000","last_clean":"2024-01-15T18:24:07.968432+0000","last_became_active":"2024-01-15T18:18:33.161329+0000","last_became_peered":"2024-01-15T18:18:33.161329+0000","last_unstale":"2024-01-15T18:24:07.968432+0000","last_undegraded":"2024-01-15T18:24:07.968432+0000","last_fullsized":"2024-01-15T18:24:07.968432+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'34","last_scrub_stamp":"2024-01-15T18:24:07.968352+0000","last_deep_scrub":"110'34","last_deep_scrub_stamp":"2024-01-15T18:24:07.968352+0000","last_clean_scrub_stamp":"2024-01-15T18:24:07.968352+0000","objects_scrubbed":10,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:12:38.215453+0000","scrub_duration":0.071078854999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"110'2","reported_seq":27,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:03.485991+0000","last_change":"2024-01-15T18:24:03.486011+0000","last_active":"2024-01-15T18:24:03.485991+0000","last_peered":"2024-01-15T18:24:03.485991+0000","last_clean":"2024-01-15T18:24:03.485991+0000","last_became_active":"2024-01-15T18:18:32.214009+0000","last_became_peered":"2024-01-15T18:18:32.214009+0000","last_unstale":"2024-01-15T18:24:03.485991+0000","last_undegraded":"2024-01-15T18:24:03.485991+0000","last_fullsized":"2024-01-15T18:24:03.485991+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2","last_scrub_stamp":"2024-01-15T18:24:03.485914+0000","last_deep_scrub":"110'2","last_deep_scrub_stamp":"2024-01-15T18:24:03.485914+0000","last_clean_scrub_stamp":"2024-01-15T18:24:03.485914+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:51:10.423070+0000","scrub_duration":0.0039431259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"110'52","reported_seq":87,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:59.256424+0000","last_change":"2024-01-15T18:23:59.256459+0000","last_active":"2024-01-15T18:23:59.256424+0000","last_peered":"2024-01-15T18:23:59.256424+0000","last_clean":"2024-01-15T18:23:59.256424+0000","last_became_active":"2024-01-15T18:18:33.162679+0000","last_became_peered":"2024-01-15T18:18:33.162679+0000","last_unstale":"2024-01-15T18:23:59.256424+0000","last_undegraded":"2024-01-15T18:23:59.256424+0000","last_fullsized":"2024-01-15T18:23:59.256424+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'52","last_scrub_stamp":"2024-01-15T18:23:59.256325+0000","last_deep_scrub":"110'52","last_deep_scrub_stamp":"2024-01-15T18:23:59.256325+0000","last_clean_scrub_stamp":"2024-01-15T18:23:59.256325+0000","objects_scrubbed":17,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:05:12.786176+0000","scrub_duration":0.11336687600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"110'3","reported_seq":28,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:43.939236+0000","last_change":"2024-01-15T18:23:43.939260+0000","last_active":"2024-01-15T18:23:43.939236+0000","last_peered":"2024-01-15T18:23:43.939236+0000","last_clean":"2024-01-15T18:23:43.939236+0000","last_became_active":"2024-01-15T18:18:32.214695+0000","last_became_peered":"2024-01-15T18:18:32.214695+0000","last_unstale":"2024-01-15T18:23:43.939236+0000","last_undegraded":"2024-01-15T18:23:43.939236+0000","last_fullsized":"2024-01-15T18:23:43.939236+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'3","last_scrub_stamp":"2024-01-15T18:23:43.939137+0000","last_deep_scrub":"110'3","last_deep_scrub_stamp":"2024-01-15T18:23:43.939137+0000","last_clean_scrub_stamp":"2024-01-15T18:23:43.939137+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:24:14.419417+0000","scrub_duration":0.0041419650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"110'62","reported_seq":97,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:33.408611+0000","last_change":"2024-01-15T18:24:33.408643+0000","last_active":"2024-01-15T18:24:33.408611+0000","last_peered":"2024-01-15T18:24:33.408611+0000","last_clean":"2024-01-15T18:24:33.408611+0000","last_became_active":"2024-01-15T18:18:33.168190+0000","last_became_peered":"2024-01-15T18:18:33.168190+0000","last_unstale":"2024-01-15T18:24:33.408611+0000","last_undegraded":"2024-01-15T18:24:33.408611+0000","last_fullsized":"2024-01-15T18:24:33.408611+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'62","last_scrub_stamp":"2024-01-15T18:24:33.408535+0000","last_deep_scrub":"110'62","last_deep_scrub_stamp":"2024-01-15T18:24:33.408535+0000","last_clean_scrub_stamp":"2024-01-15T18:24:33.408535+0000","objects_scrubbed":18,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:32:39.374083+0000","scrub_duration":0.111003042,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":11,"num_read_kb":28676,"num_write":22,"num_write_kb":73730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"110'2","reported_seq":27,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:47.483359+0000","last_change":"2024-01-15T18:23:47.483404+0000","last_active":"2024-01-15T18:23:47.483359+0000","last_peered":"2024-01-15T18:23:47.483359+0000","last_clean":"2024-01-15T18:23:47.483359+0000","last_became_active":"2024-01-15T18:18:32.220884+0000","last_became_peered":"2024-01-15T18:18:32.220884+0000","last_unstale":"2024-01-15T18:23:47.483359+0000","last_undegraded":"2024-01-15T18:23:47.483359+0000","last_fullsized":"2024-01-15T18:23:47.483359+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'2","last_scrub_stamp":"2024-01-15T18:23:47.483235+0000","last_deep_scrub":"110'2","last_deep_scrub_stamp":"2024-01-15T18:23:47.483235+0000","last_clean_scrub_stamp":"2024-01-15T18:23:47.483235+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:48:48.450236+0000","scrub_duration":0.0045499140000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"110'63","reported_seq":99,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:35.432050+0000","last_change":"2024-01-15T18:24:35.432088+0000","last_active":"2024-01-15T18:24:35.432050+0000","last_peered":"2024-01-15T18:24:35.432050+0000","last_clean":"2024-01-15T18:24:35.432050+0000","last_became_active":"2024-01-15T18:18:33.162424+0000","last_became_peered":"2024-01-15T18:18:33.162424+0000","last_unstale":"2024-01-15T18:24:35.432050+0000","last_undegraded":"2024-01-15T18:24:35.432050+0000","last_fullsized":"2024-01-15T18:24:35.432050+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'63","last_scrub_stamp":"2024-01-15T18:24:35.431981+0000","last_deep_scrub":"110'63","last_deep_scrub_stamp":"2024-01-15T18:24:35.431981+0000","last_clean_scrub_stamp":"2024-01-15T18:24:35.431981+0000","objects_scrubbed":22,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:17:18.631859+0000","scrub_duration":0.15189919499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:44.345346+0000","last_change":"2024-01-15T18:23:44.345376+0000","last_active":"2024-01-15T18:23:44.345346+0000","last_peered":"2024-01-15T18:23:44.345346+0000","last_clean":"2024-01-15T18:23:44.345346+0000","last_became_active":"2024-01-15T18:18:32.216769+0000","last_became_peered":"2024-01-15T18:18:32.216769+0000","last_unstale":"2024-01-15T18:23:44.345346+0000","last_undegraded":"2024-01-15T18:23:44.345346+0000","last_fullsized":"2024-01-15T18:23:44.345346+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:44.345256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:44.345256+0000","last_clean_scrub_stamp":"2024-01-15T18:23:44.345256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:35:12.856197+0000","scrub_duration":0.0019326370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"110'75","reported_seq":108,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:06.470407+0000","last_change":"2024-01-15T18:24:06.470439+0000","last_active":"2024-01-15T18:24:06.470407+0000","last_peered":"2024-01-15T18:24:06.470407+0000","last_clean":"2024-01-15T18:24:06.470407+0000","last_became_active":"2024-01-15T18:18:33.162658+0000","last_became_peered":"2024-01-15T18:18:33.162658+0000","last_unstale":"2024-01-15T18:24:06.470407+0000","last_undegraded":"2024-01-15T18:24:06.470407+0000","last_fullsized":"2024-01-15T18:24:06.470407+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'75","last_scrub_stamp":"2024-01-15T18:24:06.470318+0000","last_deep_scrub":"110'75","last_deep_scrub_stamp":"2024-01-15T18:24:06.470318+0000","last_clean_scrub_stamp":"2024-01-15T18:24:06.470318+0000","objects_scrubbed":30,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:41:38.067402+0000","scrub_duration":0.20391261699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"110'4","reported_seq":29,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:48.233481+0000","last_change":"2024-01-15T18:23:48.233504+0000","last_active":"2024-01-15T18:23:48.233481+0000","last_peered":"2024-01-15T18:23:48.233481+0000","last_clean":"2024-01-15T18:23:48.233481+0000","last_became_active":"2024-01-15T18:18:32.207507+0000","last_became_peered":"2024-01-15T18:18:32.207507+0000","last_unstale":"2024-01-15T18:23:48.233481+0000","last_undegraded":"2024-01-15T18:23:48.233481+0000","last_fullsized":"2024-01-15T18:23:48.233481+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'4","last_scrub_stamp":"2024-01-15T18:23:48.233388+0000","last_deep_scrub":"110'4","last_deep_scrub_stamp":"2024-01-15T18:23:48.233388+0000","last_clean_scrub_stamp":"2024-01-15T18:23:48.233388+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:50.599913+0000","scrub_duration":0.0034152789999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"110'62","reported_seq":93,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:31.875831+0000","last_change":"2024-01-15T18:24:31.875870+0000","last_active":"2024-01-15T18:24:31.875831+0000","last_peered":"2024-01-15T18:24:31.875831+0000","last_clean":"2024-01-15T18:24:31.875831+0000","last_became_active":"2024-01-15T18:18:33.164206+0000","last_became_peered":"2024-01-15T18:18:33.164206+0000","last_unstale":"2024-01-15T18:24:31.875831+0000","last_undegraded":"2024-01-15T18:24:31.875831+0000","last_fullsized":"2024-01-15T18:24:31.875831+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'62","last_scrub_stamp":"2024-01-15T18:24:31.875741+0000","last_deep_scrub":"110'62","last_deep_scrub_stamp":"2024-01-15T18:24:31.875741+0000","last_clean_scrub_stamp":"2024-01-15T18:24:31.875741+0000","objects_scrubbed":19,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:24:06.719428+0000","scrub_duration":0.118668572,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:48.035163+0000","last_change":"2024-01-15T18:23:48.035184+0000","last_active":"2024-01-15T18:23:48.035163+0000","last_peered":"2024-01-15T18:23:48.035163+0000","last_clean":"2024-01-15T18:23:48.035163+0000","last_became_active":"2024-01-15T18:18:32.212201+0000","last_became_peered":"2024-01-15T18:18:32.212201+0000","last_unstale":"2024-01-15T18:23:48.035163+0000","last_undegraded":"2024-01-15T18:23:48.035163+0000","last_fullsized":"2024-01-15T18:23:48.035163+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:48.035058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:48.035058+0000","last_clean_scrub_stamp":"2024-01-15T18:23:48.035058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:49:40.411498+0000","scrub_duration":0.0017447160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"110'61","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:09.459306+0000","last_change":"2024-01-15T18:24:09.459337+0000","last_active":"2024-01-15T18:24:09.459306+0000","last_peered":"2024-01-15T18:24:09.459306+0000","last_clean":"2024-01-15T18:24:09.459306+0000","last_became_active":"2024-01-15T18:18:33.164708+0000","last_became_peered":"2024-01-15T18:18:33.164708+0000","last_unstale":"2024-01-15T18:24:09.459306+0000","last_undegraded":"2024-01-15T18:24:09.459306+0000","last_fullsized":"2024-01-15T18:24:09.459306+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'61","last_scrub_stamp":"2024-01-15T18:24:09.459228+0000","last_deep_scrub":"110'61","last_deep_scrub_stamp":"2024-01-15T18:24:09.459228+0000","last_clean_scrub_stamp":"2024-01-15T18:24:09.459228+0000","objects_scrubbed":17,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:00:51.923242+0000","scrub_duration":0.14356816999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:44.292443+0000","last_change":"2024-01-15T18:23:44.292477+0000","last_active":"2024-01-15T18:23:44.292443+0000","last_peered":"2024-01-15T18:23:44.292443+0000","last_clean":"2024-01-15T18:23:44.292443+0000","last_became_active":"2024-01-15T18:18:32.207599+0000","last_became_peered":"2024-01-15T18:18:32.207599+0000","last_unstale":"2024-01-15T18:23:44.292443+0000","last_undegraded":"2024-01-15T18:23:44.292443+0000","last_fullsized":"2024-01-15T18:23:44.292443+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:44.292288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:44.292288+0000","last_clean_scrub_stamp":"2024-01-15T18:23:44.292288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:52:22.761638+0000","scrub_duration":0.0021733669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"110'59","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:04.017805+0000","last_change":"2024-01-15T18:24:04.017845+0000","last_active":"2024-01-15T18:24:04.017805+0000","last_peered":"2024-01-15T18:24:04.017805+0000","last_clean":"2024-01-15T18:24:04.017805+0000","last_became_active":"2024-01-15T18:18:33.160350+0000","last_became_peered":"2024-01-15T18:18:33.160350+0000","last_unstale":"2024-01-15T18:24:04.017805+0000","last_undegraded":"2024-01-15T18:24:04.017805+0000","last_fullsized":"2024-01-15T18:24:04.017805+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'59","last_scrub_stamp":"2024-01-15T18:24:04.017732+0000","last_deep_scrub":"110'59","last_deep_scrub_stamp":"2024-01-15T18:24:04.017732+0000","last_clean_scrub_stamp":"2024-01-15T18:24:04.017732+0000","objects_scrubbed":17,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:29:01.499590+0000","scrub_duration":0.112111222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67109125,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":17,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:47.491435+0000","last_change":"2024-01-15T18:23:47.491458+0000","last_active":"2024-01-15T18:23:47.491435+0000","last_peered":"2024-01-15T18:23:47.491435+0000","last_clean":"2024-01-15T18:23:47.491435+0000","last_became_active":"2024-01-15T18:18:32.216581+0000","last_became_peered":"2024-01-15T18:18:32.216581+0000","last_unstale":"2024-01-15T18:23:47.491435+0000","last_undegraded":"2024-01-15T18:23:47.491435+0000","last_fullsized":"2024-01-15T18:23:47.491435+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:47.491336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:47.491336+0000","last_clean_scrub_stamp":"2024-01-15T18:23:47.491336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:06:23.583464+0000","scrub_duration":0.0018469020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"110'46","reported_seq":74,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:48.560733+0000","last_change":"2024-01-15T18:23:48.560769+0000","last_active":"2024-01-15T18:23:48.560733+0000","last_peered":"2024-01-15T18:23:48.560733+0000","last_clean":"2024-01-15T18:23:48.560733+0000","last_became_active":"2024-01-15T18:18:33.164152+0000","last_became_peered":"2024-01-15T18:18:33.164152+0000","last_unstale":"2024-01-15T18:23:48.560733+0000","last_undegraded":"2024-01-15T18:23:48.560733+0000","last_fullsized":"2024-01-15T18:23:48.560733+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'46","last_scrub_stamp":"2024-01-15T18:23:48.560650+0000","last_deep_scrub":"110'46","last_deep_scrub_stamp":"2024-01-15T18:23:48.560650+0000","last_clean_scrub_stamp":"2024-01-15T18:23:48.560650+0000","objects_scrubbed":13,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:42:57.049428+0000","scrub_duration":0.082496549000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"110'3","reported_seq":28,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:44.865576+0000","last_change":"2024-01-15T18:23:44.865597+0000","last_active":"2024-01-15T18:23:44.865576+0000","last_peered":"2024-01-15T18:23:44.865576+0000","last_clean":"2024-01-15T18:23:44.865576+0000","last_became_active":"2024-01-15T18:18:32.218858+0000","last_became_peered":"2024-01-15T18:18:32.218858+0000","last_unstale":"2024-01-15T18:23:44.865576+0000","last_undegraded":"2024-01-15T18:23:44.865576+0000","last_fullsized":"2024-01-15T18:23:44.865576+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'3","last_scrub_stamp":"2024-01-15T18:23:44.865505+0000","last_deep_scrub":"110'3","last_deep_scrub_stamp":"2024-01-15T18:23:44.865505+0000","last_clean_scrub_stamp":"2024-01-15T18:23:44.865505+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:03:03.103591+0000","scrub_duration":0.0027720739999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"110'63","reported_seq":99,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:16.773337+0000","last_change":"2024-01-15T18:24:16.773374+0000","last_active":"2024-01-15T18:24:16.773337+0000","last_peered":"2024-01-15T18:24:16.773337+0000","last_clean":"2024-01-15T18:24:16.773337+0000","last_became_active":"2024-01-15T18:18:33.160315+0000","last_became_peered":"2024-01-15T18:18:33.160315+0000","last_unstale":"2024-01-15T18:24:16.773337+0000","last_undegraded":"2024-01-15T18:24:16.773337+0000","last_fullsized":"2024-01-15T18:24:16.773337+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'63","last_scrub_stamp":"2024-01-15T18:24:16.773261+0000","last_deep_scrub":"110'63","last_deep_scrub_stamp":"2024-01-15T18:24:16.773261+0000","last_clean_scrub_stamp":"2024-01-15T18:24:16.773261+0000","objects_scrubbed":22,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:21:25.069967+0000","scrub_duration":0.13882724199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"110'18","reported_seq":43,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:45.538237+0000","last_change":"2024-01-15T18:23:45.538258+0000","last_active":"2024-01-15T18:23:45.538237+0000","last_peered":"2024-01-15T18:23:45.538237+0000","last_clean":"2024-01-15T18:23:45.538237+0000","last_became_active":"2024-01-15T18:18:32.216612+0000","last_became_peered":"2024-01-15T18:18:32.216612+0000","last_unstale":"2024-01-15T18:23:45.538237+0000","last_undegraded":"2024-01-15T18:23:45.538237+0000","last_fullsized":"2024-01-15T18:23:45.538237+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'18","last_scrub_stamp":"2024-01-15T18:23:45.538139+0000","last_deep_scrub":"110'18","last_deep_scrub_stamp":"2024-01-15T18:23:45.538139+0000","last_clean_scrub_stamp":"2024-01-15T18:23:45.538139+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:40:14.696757+0000","scrub_duration":0.0021555250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":506,"num_omap_keys":10,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"110'66","reported_seq":104,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:44.623824+0000","last_change":"2024-01-15T18:23:44.623866+0000","last_active":"2024-01-15T18:23:44.623824+0000","last_peered":"2024-01-15T18:23:44.623824+0000","last_clean":"2024-01-15T18:23:44.623824+0000","last_became_active":"2024-01-15T18:18:33.163994+0000","last_became_peered":"2024-01-15T18:18:33.163994+0000","last_unstale":"2024-01-15T18:23:44.623824+0000","last_undegraded":"2024-01-15T18:23:44.623824+0000","last_fullsized":"2024-01-15T18:23:44.623824+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'66","last_scrub_stamp":"2024-01-15T18:23:44.623747+0000","last_deep_scrub":"110'66","last_deep_scrub_stamp":"2024-01-15T18:23:44.623747+0000","last_clean_scrub_stamp":"2024-01-15T18:23:44.623747+0000","objects_scrubbed":24,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:47:40.548332+0000","scrub_duration":0.162659633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:54.492408+0000","last_change":"2024-01-15T18:23:54.492428+0000","last_active":"2024-01-15T18:23:54.492408+0000","last_peered":"2024-01-15T18:23:54.492408+0000","last_clean":"2024-01-15T18:23:54.492408+0000","last_became_active":"2024-01-15T18:18:32.216023+0000","last_became_peered":"2024-01-15T18:18:32.216023+0000","last_unstale":"2024-01-15T18:23:54.492408+0000","last_undegraded":"2024-01-15T18:23:54.492408+0000","last_fullsized":"2024-01-15T18:23:54.492408+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:54.492304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:54.492304+0000","last_clean_scrub_stamp":"2024-01-15T18:23:54.492304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:21:06.895168+0000","scrub_duration":0.0023113349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"110'53","reported_seq":90,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:56.632803+0000","last_change":"2024-01-15T18:23:56.632855+0000","last_active":"2024-01-15T18:23:56.632803+0000","last_peered":"2024-01-15T18:23:56.632803+0000","last_clean":"2024-01-15T18:23:56.632803+0000","last_became_active":"2024-01-15T18:18:42.262255+0000","last_became_peered":"2024-01-15T18:18:42.262255+0000","last_unstale":"2024-01-15T18:23:56.632803+0000","last_undegraded":"2024-01-15T18:23:56.632803+0000","last_fullsized":"2024-01-15T18:23:56.632803+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'53","last_scrub_stamp":"2024-01-15T18:23:56.632712+0000","last_deep_scrub":"110'53","last_deep_scrub_stamp":"2024-01-15T18:23:56.632712+0000","last_clean_scrub_stamp":"2024-01-15T18:23:56.632712+0000","objects_scrubbed":19,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:10:24.687025+0000","scrub_duration":0.126095914,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303290,"num_objects":19,"num_object_clones":1,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:54.899638+0000","last_change":"2024-01-15T18:23:54.899657+0000","last_active":"2024-01-15T18:23:54.899638+0000","last_peered":"2024-01-15T18:23:54.899638+0000","last_clean":"2024-01-15T18:23:54.899638+0000","last_became_active":"2024-01-15T18:18:32.213930+0000","last_became_peered":"2024-01-15T18:18:32.213930+0000","last_unstale":"2024-01-15T18:23:54.899638+0000","last_undegraded":"2024-01-15T18:23:54.899638+0000","last_fullsized":"2024-01-15T18:23:54.899638+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:54.899542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:54.899542+0000","last_clean_scrub_stamp":"2024-01-15T18:23:54.899542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:23:19.753410+0000","scrub_duration":0.0018061939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"110'60","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:52.993684+0000","last_change":"2024-01-15T18:23:52.993718+0000","last_active":"2024-01-15T18:23:52.993684+0000","last_peered":"2024-01-15T18:23:52.993684+0000","last_clean":"2024-01-15T18:23:52.993684+0000","last_became_active":"2024-01-15T18:18:33.150660+0000","last_became_peered":"2024-01-15T18:18:33.150660+0000","last_unstale":"2024-01-15T18:23:52.993684+0000","last_undegraded":"2024-01-15T18:23:52.993684+0000","last_fullsized":"2024-01-15T18:23:52.993684+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'60","last_scrub_stamp":"2024-01-15T18:23:52.993598+0000","last_deep_scrub":"110'60","last_deep_scrub_stamp":"2024-01-15T18:23:52.993598+0000","last_clean_scrub_stamp":"2024-01-15T18:23:52.993598+0000","objects_scrubbed":16,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:28:50.475455+0000","scrub_duration":0.111747205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:53.298745+0000","last_change":"2024-01-15T18:23:53.298763+0000","last_active":"2024-01-15T18:23:53.298745+0000","last_peered":"2024-01-15T18:23:53.298745+0000","last_clean":"2024-01-15T18:23:53.298745+0000","last_became_active":"2024-01-15T18:18:32.214394+0000","last_became_peered":"2024-01-15T18:18:32.214394+0000","last_unstale":"2024-01-15T18:23:53.298745+0000","last_undegraded":"2024-01-15T18:23:53.298745+0000","last_fullsized":"2024-01-15T18:23:53.298745+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:53.298655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:53.298655+0000","last_clean_scrub_stamp":"2024-01-15T18:23:53.298655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:46:41.201522+0000","scrub_duration":0.00139146,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"110'85","reported_seq":118,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:06.148602+0000","last_change":"2024-01-15T18:24:06.148627+0000","last_active":"2024-01-15T18:24:06.148602+0000","last_peered":"2024-01-15T18:24:06.148602+0000","last_clean":"2024-01-15T18:24:06.148602+0000","last_became_active":"2024-01-15T18:18:33.155797+0000","last_became_peered":"2024-01-15T18:18:33.155797+0000","last_unstale":"2024-01-15T18:24:06.148602+0000","last_undegraded":"2024-01-15T18:24:06.148602+0000","last_fullsized":"2024-01-15T18:24:06.148602+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'85","last_scrub_stamp":"2024-01-15T18:24:06.148530+0000","last_deep_scrub":"110'85","last_deep_scrub_stamp":"2024-01-15T18:24:06.148530+0000","last_clean_scrub_stamp":"2024-01-15T18:24:06.148530+0000","objects_scrubbed":29,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:13.689014+0000","scrub_duration":0.19150456099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:47.949655+0000","last_change":"2024-01-15T18:23:47.949688+0000","last_active":"2024-01-15T18:23:47.949655+0000","last_peered":"2024-01-15T18:23:47.949655+0000","last_clean":"2024-01-15T18:23:47.949655+0000","last_became_active":"2024-01-15T18:18:32.212487+0000","last_became_peered":"2024-01-15T18:18:32.212487+0000","last_unstale":"2024-01-15T18:23:47.949655+0000","last_undegraded":"2024-01-15T18:23:47.949655+0000","last_fullsized":"2024-01-15T18:23:47.949655+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:47.949523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:47.949523+0000","last_clean_scrub_stamp":"2024-01-15T18:23:47.949523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:31:41.981969+0000","scrub_duration":0.0029090320000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"110'72","reported_seq":106,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:24:20.478495+0000","last_change":"2024-01-15T18:24:20.478495+0000","last_active":"2024-01-15T18:24:20.478495+0000","last_peered":"2024-01-15T18:24:20.478495+0000","last_clean":"2024-01-15T18:24:20.478495+0000","last_became_active":"2024-01-15T18:18:33.167436+0000","last_became_peered":"2024-01-15T18:18:33.167436+0000","last_unstale":"2024-01-15T18:24:20.478495+0000","last_undegraded":"2024-01-15T18:24:20.478495+0000","last_fullsized":"2024-01-15T18:24:20.478495+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'72","last_scrub_stamp":"2024-01-15T18:24:20.478392+0000","last_deep_scrub":"110'72","last_deep_scrub_stamp":"2024-01-15T18:24:20.478392+0000","last_clean_scrub_stamp":"2024-01-15T18:24:20.478392+0000","objects_scrubbed":26,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:13:07.948090+0000","scrub_duration":0.188420635,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"110'12","reported_seq":47,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:40.933714+0000","last_change":"2024-01-15T18:23:40.933751+0000","last_active":"2024-01-15T18:23:40.933714+0000","last_peered":"2024-01-15T18:23:40.933714+0000","last_clean":"2024-01-15T18:23:40.933714+0000","last_became_active":"2024-01-15T18:18:32.211744+0000","last_became_peered":"2024-01-15T18:18:32.211744+0000","last_unstale":"2024-01-15T18:23:40.933714+0000","last_undegraded":"2024-01-15T18:23:40.933714+0000","last_fullsized":"2024-01-15T18:23:40.933714+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'12","last_scrub_stamp":"2024-01-15T18:23:40.933627+0000","last_deep_scrub":"110'12","last_deep_scrub_stamp":"2024-01-15T18:23:40.933627+0000","last_clean_scrub_stamp":"2024-01-15T18:23:40.933627+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:56:38.633122+0000","scrub_duration":0.0031438920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1321,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"110'59","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:38.786673+0000","last_change":"2024-01-15T18:23:38.786716+0000","last_active":"2024-01-15T18:23:38.786673+0000","last_peered":"2024-01-15T18:23:38.786673+0000","last_clean":"2024-01-15T18:23:38.786673+0000","last_became_active":"2024-01-15T18:18:33.162521+0000","last_became_peered":"2024-01-15T18:18:33.162521+0000","last_unstale":"2024-01-15T18:23:38.786673+0000","last_undegraded":"2024-01-15T18:23:38.786673+0000","last_fullsized":"2024-01-15T18:23:38.786673+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'59","last_scrub_stamp":"2024-01-15T18:23:38.786600+0000","last_deep_scrub":"110'59","last_deep_scrub_stamp":"2024-01-15T18:23:38.786600+0000","last_clean_scrub_stamp":"2024-01-15T18:23:38.786600+0000","objects_scrubbed":23,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:26:13.413987+0000","scrub_duration":0.160345389,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":26,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:45.474259+0000","last_change":"2024-01-15T18:23:45.474259+0000","last_active":"2024-01-15T18:23:45.474259+0000","last_peered":"2024-01-15T18:23:45.474259+0000","last_clean":"2024-01-15T18:23:45.474259+0000","last_became_active":"2024-01-15T18:18:32.216966+0000","last_became_peered":"2024-01-15T18:18:32.216966+0000","last_unstale":"2024-01-15T18:23:45.474259+0000","last_undegraded":"2024-01-15T18:23:45.474259+0000","last_fullsized":"2024-01-15T18:23:45.474259+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:45.474139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:45.474139+0000","last_clean_scrub_stamp":"2024-01-15T18:23:45.474139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:43:14.897532+0000","scrub_duration":0.0025295999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"110'44","reported_seq":76,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:59.012816+0000","last_change":"2024-01-15T18:23:59.012849+0000","last_active":"2024-01-15T18:23:59.012816+0000","last_peered":"2024-01-15T18:23:59.012816+0000","last_clean":"2024-01-15T18:23:59.012816+0000","last_became_active":"2024-01-15T18:18:33.165323+0000","last_became_peered":"2024-01-15T18:18:33.165323+0000","last_unstale":"2024-01-15T18:23:59.012816+0000","last_undegraded":"2024-01-15T18:23:59.012816+0000","last_fullsized":"2024-01-15T18:23:59.012816+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'44","last_scrub_stamp":"2024-01-15T18:23:59.012731+0000","last_deep_scrub":"110'44","last_deep_scrub_stamp":"2024-01-15T18:23:59.012731+0000","last_clean_scrub_stamp":"2024-01-15T18:23:59.012731+0000","objects_scrubbed":14,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:30:06.603978+0000","scrub_duration":0.088778124,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":25,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:42.312415+0000","last_change":"2024-01-15T18:23:42.312436+0000","last_active":"2024-01-15T18:23:42.312415+0000","last_peered":"2024-01-15T18:23:42.312415+0000","last_clean":"2024-01-15T18:23:42.312415+0000","last_became_active":"2024-01-15T18:18:32.215973+0000","last_became_peered":"2024-01-15T18:18:32.215973+0000","last_unstale":"2024-01-15T18:23:42.312415+0000","last_undegraded":"2024-01-15T18:23:42.312415+0000","last_fullsized":"2024-01-15T18:23:42.312415+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:23:42.312313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:23:42.312313+0000","last_clean_scrub_stamp":"2024-01-15T18:23:42.312313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:17:28.948911+0000","scrub_duration":0.0020901840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"110'119","reported_seq":255,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:23:44.359592+0000","last_change":"2024-01-15T18:23:44.359635+0000","last_active":"2024-01-15T18:23:44.359592+0000","last_peered":"2024-01-15T18:23:44.359592+0000","last_clean":"2024-01-15T18:23:44.359592+0000","last_became_active":"2024-01-15T18:04:45.458399+0000","last_became_peered":"2024-01-15T18:04:45.458399+0000","last_unstale":"2024-01-15T18:23:44.359592+0000","last_undegraded":"2024-01-15T18:23:44.359592+0000","last_fullsized":"2024-01-15T18:23:44.359592+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'119","last_scrub_stamp":"2024-01-15T18:23:44.359514+0000","last_deep_scrub":"110'119","last_deep_scrub_stamp":"2024-01-15T18:23:44.359514+0000","last_clean_scrub_stamp":"2024-01-15T18:23:44.359514+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:41:00.611426+0000","scrub_duration":0.18253596899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":5368710136,"num_objects":1286,"num_object_clones":1,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":1,"num_read":525,"num_read_kb":2097165,"num_write":1306,"num_write_kb":5242895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10737459200,"data_stored":10737420272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4025,"ondisk_log_size":4025,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":616335,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":282,"num_write_kb":722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5506,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1294336,"data_stored":1232670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":281,"ondisk_log_size":281,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509682,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1834500,"kb_used_data":1767292,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92537340,"statfs":{"total":96636764160,"available":94758236160,"internally_reserved":0,"allocated":1809707008,"data_stored":1808930471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.73399999999999999,"15min":0.84299999999999997},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.35499999999999998},"max":{"1min":0.85699999999999998,"5min":2.8620000000000001,"15min":9.6989999999999998},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.72199999999999998,"15min":0.85799999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.34599999999999997},"max":{"1min":0.92700000000000005,"5min":1.0029999999999999,"15min":16.968},"last":0.80900000000000005}]},{"osd":1,"last update":"Mon Jan 15 18:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.73699999999999999,"15min":0.91100000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94999999999999996,"5min":1.028,"15min":15.791},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.73199999999999998,"15min":0.85899999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.88100000000000001,"5min":1.248,"15min":10.842000000000001},"last":0.76500000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.73399999999999999,"15min":0.89100000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.39500000000000002},"max":{"1min":0.81499999999999995,"5min":0.98299999999999998,"15min":22.318000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.76400000000000001,"15min":0.878},"min":{"1min":0.54600000000000004,"5min":0.42699999999999999,"15min":0.39100000000000001},"max":{"1min":0.90300000000000002,"5min":2.2799999999999998,"15min":9.7539999999999996},"last":0.90700000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.73899999999999999,"15min":0.93400000000000005},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.38100000000000001},"max":{"1min":0.85899999999999999,"5min":1.2110000000000001,"15min":18.152999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.78400000000000003,"15min":0.92300000000000004},"min":{"1min":0.52700000000000002,"5min":0.497,"15min":0.433},"max":{"1min":0.872,"5min":2.2389999999999999,"15min":9.7959999999999994},"last":0.85799999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58899999999999997,"15min":0.80900000000000005},"min":{"1min":0.38100000000000001,"5min":0.34799999999999998,"15min":0.314},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":20.387},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.60499999999999998,"15min":0.86299999999999999},"min":{"1min":0.45100000000000001,"5min":0.42499999999999999,"15min":0.34200000000000003},"max":{"1min":0.72699999999999998,"5min":0.98099999999999998,"15min":17.215},"last":0.82299999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.625,"15min":0.83299999999999996},"min":{"1min":0.46200000000000002,"5min":0.40999999999999998,"15min":0.34699999999999998},"max":{"1min":0.83699999999999997,"5min":0.96999999999999997,"15min":9.8450000000000006},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.63600000000000001,"15min":0.86799999999999999},"min":{"1min":0.438,"5min":0.40899999999999997,"15min":0.35799999999999998},"max":{"1min":0.77500000000000002,"5min":0.92600000000000005,"15min":19.905000000000001},"last":0.69799999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:24:35 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.753,"15min":0.99199999999999999},"min":{"1min":0.55500000000000005,"5min":0.44,"15min":0.40799999999999997},"max":{"1min":0.86699999999999999,"5min":0.995,"15min":20.492000000000001},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.63,"5min":0.69199999999999995,"15min":0.89400000000000002},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.40799999999999997},"max":{"1min":0.80600000000000005,"5min":0.96299999999999997,"15min":17.132000000000001},"last":0.75}]}]},{"osd":6,"up_from":15,"seq":64424509682,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1260480,"kb_used_data":1193272,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93111360,"statfs":{"total":96636764160,"available":95346032640,"internally_reserved":0,"allocated":1221910528,"data_stored":1221137119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:24:08 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.77600000000000002,"15min":0.93899999999999995},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.86799999999999999,"5min":6.1900000000000004,"15min":32.587000000000003},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.78100000000000003,"15min":0.84499999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.35399999999999998},"max":{"1min":0.82199999999999995,"5min":4.9649999999999999,"15min":16.193000000000001},"last":0.82999999999999996}]},{"osd":1,"last update":"Mon Jan 15 18:24:08 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.72899999999999998,"15min":0.78600000000000003},"min":{"1min":0.41199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81200000000000006,"5min":1.0049999999999999,"15min":7.7649999999999997},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.72599999999999998,"15min":0.875},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.82799999999999996,"5min":1.0189999999999999,"15min":32.512},"last":0.80600000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:24:08 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76400000000000001,"15min":0.92700000000000005},"min":{"1min":0.47599999999999998,"5min":0.46400000000000002,"15min":0.41699999999999998},"max":{"1min":0.98799999999999999,"5min":1.016,"15min":32.472999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.80400000000000005,"15min":0.88500000000000001},"min":{"1min":0.46000000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.84399999999999997,"5min":5.0410000000000004,"15min":16.602},"last":0.86199999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:24:08 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.82099999999999995,"15min":0.91100000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.46000000000000002},"max":{"1min":0.89200000000000002,"5min":5.0940000000000003,"15min":16.388000000000002},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.752,"15min":0.86199999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.432},"max":{"1min":0.86899999999999999,"5min":1.04,"15min":9.3719999999999999},"last":0.76600000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:24:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.67100000000000004,"15min":0.81699999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.378},"max":{"1min":0.77800000000000002,"5min":6.3869999999999996,"15min":32.549999999999997},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.68999999999999995,"15min":0.79300000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.84999999999999998,"5min":5.1719999999999997,"15min":11.852},"last":0.69399999999999995}]},{"osd":5,"last update":"Mon Jan 15 18:24:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.69899999999999995,"15min":0.80800000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.34300000000000003},"max":{"1min":0.83199999999999996,"5min":6.8769999999999998,"15min":13.087999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.749,"15min":0.94299999999999995},"min":{"1min":0.436,"5min":0.436,"15min":0.41799999999999998},"max":{"1min":0.95099999999999996,"5min":6.1360000000000001,"15min":32.404000000000003},"last":0.66600000000000004}]},{"osd":7,"last update":"Mon Jan 15 18:24:08 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.747,"15min":0.91700000000000004},"min":{"1min":0.49199999999999999,"5min":0.46700000000000003,"15min":0.38600000000000001},"max":{"1min":0.93300000000000005,"5min":5.1319999999999997,"15min":23.991},"last":0.753},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.92100000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.40999999999999998},"max":{"1min":0.91300000000000003,"5min":0.999,"15min":40.972999999999999},"last":0.72699999999999998}]}]},{"osd":4,"up_from":15,"seq":64424509682,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1711036,"kb_used_data":1643828,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92660804,"statfs":{"total":96636764160,"available":94884663296,"internally_reserved":0,"allocated":1683279872,"data_stored":1682510628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:24:24 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":0.878,"15min":0.86499999999999999},"min":{"1min":0.436,"5min":0.40200000000000002,"15min":0.32900000000000001},"max":{"1min":17.082000000000001,"5min":17.082000000000001,"15min":17.082000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.72499999999999998,"15min":0.74299999999999999},"min":{"1min":0.45800000000000002,"5min":0.44400000000000001,"15min":0.34000000000000002},"max":{"1min":1.036,"5min":1.036,"15min":14.723000000000001},"last":0.92900000000000005}]},{"osd":1,"last update":"Mon Jan 15 18:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.751,"15min":0.79300000000000004},"min":{"1min":0.43099999999999999,"5min":0.40400000000000003,"15min":0.38800000000000001},"max":{"1min":0.95499999999999996,"5min":1.919,"15min":9.5700000000000003},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.4950000000000001,"5min":0.91200000000000003,"15min":0.90700000000000003},"min":{"1min":0.46999999999999997,"5min":0.46400000000000002,"15min":0.36899999999999999},"max":{"1min":16.989999999999998,"5min":16.989999999999998,"15min":16.989999999999998},"last":0.91300000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.76100000000000001,"15min":0.82999999999999996},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.39700000000000002},"max":{"1min":0.99399999999999999,"5min":1.9950000000000001,"15min":9.7089999999999996},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.619,"5min":0.95799999999999996,"15min":0.98299999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.40999999999999998},"max":{"1min":19.265000000000001,"5min":19.265000000000001,"15min":19.265000000000001},"last":0.86199999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.75900000000000001,"15min":0.81599999999999995},"min":{"1min":0.54400000000000004,"5min":0.46200000000000002,"15min":0.41199999999999998},"max":{"1min":1.0880000000000001,"5min":1.577,"15min":14.763999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.83899999999999997,"15min":0.90300000000000002},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.48099999999999998},"max":{"1min":1.0229999999999999,"5min":1.9610000000000001,"15min":9.6289999999999996},"last":1.1020000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:24:24 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":0.80700000000000005,"15min":0.85599999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.34799999999999998},"max":{"1min":19.349,"5min":19.349,"15min":19.349},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.64400000000000002,"15min":0.73699999999999999},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.95399999999999996,"5min":1.4910000000000001,"15min":16.388000000000002},"last":0.69899999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.73699999999999999,"15min":0.78300000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.37},"max":{"1min":1.0720000000000001,"5min":5.9489999999999998,"15min":11.124000000000001},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.52,"5min":0.88300000000000001,"15min":0.98199999999999998},"min":{"1min":0.435,"5min":0.435,"15min":0.40200000000000002},"max":{"1min":19.423999999999999,"5min":19.423999999999999,"15min":32.308},"last":0.95799999999999996}]},{"osd":7,"last update":"Mon Jan 15 18:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.72499999999999998,"15min":0.79700000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.34799999999999998},"max":{"1min":0.91300000000000003,"5min":2.4329999999999998,"15min":14.653},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.79100000000000004,"15min":0.91300000000000003},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.46500000000000002},"max":{"1min":1.4610000000000001,"5min":2.1909999999999998,"15min":16.277000000000001},"last":1.0720000000000001}]}]},{"osd":3,"up_from":15,"seq":64424509681,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1457764,"kb_used_data":1386396,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92914076,"statfs":{"total":96636764160,"available":95144013824,"internally_reserved":0,"allocated":1419669504,"data_stored":1418884023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73071047},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45600000000000002,"15min":0.48999999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.245},"max":{"1min":0.67000000000000004,"5min":0.69699999999999995,"15min":5.8849999999999998},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.50800000000000001,"15min":0.55900000000000005},"min":{"1min":0.34599999999999997,"5min":0.30199999999999999,"15min":0.25600000000000001},"max":{"1min":0.64400000000000002,"5min":0.82099999999999995,"15min":4.7990000000000004},"last":0.38900000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.48299999999999998,"15min":0.58499999999999996},"min":{"1min":0.29899999999999999,"5min":0.28399999999999997,"15min":0.26400000000000001},"max":{"1min":0.70199999999999996,"5min":0.76400000000000001,"15min":12.307},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56000000000000005,"15min":0.58899999999999997},"min":{"1min":0.34499999999999997,"5min":0.34300000000000003,"15min":0.31900000000000001},"max":{"1min":0.79300000000000004,"5min":0.80200000000000005,"15min":6.0640000000000001},"last":0.436}]},{"osd":2,"last update":"Mon Jan 15 18:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.57399999999999995,"15min":0.66100000000000003},"min":{"1min":0.36599999999999999,"5min":0.35299999999999998,"15min":0.32600000000000001},"max":{"1min":0.72399999999999998,"5min":1.0329999999999999,"15min":12.468999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.59999999999999998,"15min":0.67400000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.35399999999999998},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":10.837999999999999},"last":0.56899999999999995}]},{"osd":4,"last update":"Mon Jan 15 18:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.78200000000000003,"15min":0.93100000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0429999999999999,"5min":1.1910000000000001,"15min":13.863},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.84299999999999997,"15min":0.96799999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.8460000000000001,"5min":1.8460000000000001,"15min":13.31},"last":0.59699999999999998}]},{"osd":5,"last update":"Mon Jan 15 18:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.79500000000000004,"15min":0.96299999999999997},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.439},"max":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":17.687999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.77200000000000002,"15min":0.89700000000000002},"min":{"1min":0.497,"5min":0.497,"15min":0.438},"max":{"1min":1.006,"5min":1.1379999999999999,"15min":20.515999999999998},"last":0.53600000000000003}]},{"osd":6,"last update":"Mon Jan 15 18:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.80000000000000004,"15min":0.89700000000000002},"min":{"1min":0.438,"5min":0.438,"15min":0.432},"max":{"1min":1.806,"5min":1.806,"15min":13.250999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.81799999999999995,"15min":0.92600000000000005},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.163,"5min":1.163,"15min":12.76},"last":0.52300000000000002}]},{"osd":7,"last update":"Mon Jan 15 18:24:31 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.81399999999999995,"15min":1.123},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.46800000000000003},"max":{"1min":1.222,"5min":1.222,"15min":24.321999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82399999999999995,"15min":1.0640000000000001},"min":{"1min":0.57499999999999996,"5min":0.55000000000000004,"15min":0.497},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":23.013000000000002},"last":0.55600000000000005}]}]},{"osd":2,"up_from":15,"seq":64424509681,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":977856,"kb_used_data":910648,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93393984,"statfs":{"total":96636764160,"available":95635439616,"internally_reserved":0,"allocated":932503552,"data_stored":931731464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46200000000000002,"15min":0.47499999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.247},"max":{"1min":0.65900000000000003,"5min":0.80800000000000005,"15min":1.857},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.499,"15min":0.50600000000000001},"min":{"1min":0.318,"5min":0.26700000000000002,"15min":0.252},"max":{"1min":0.67400000000000004,"5min":0.73999999999999999,"15min":2.3759999999999999},"last":0.55700000000000005}]},{"osd":1,"last update":"Mon Jan 15 18:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55700000000000005,"15min":0.60099999999999998},"min":{"1min":0.34200000000000003,"5min":0.314,"15min":0.29699999999999999},"max":{"1min":0.79300000000000004,"5min":0.94599999999999995,"15min":10.907},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.49299999999999999,"15min":0.56100000000000005},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.28100000000000003},"max":{"1min":0.64200000000000002,"5min":0.76500000000000001,"15min":7.2119999999999997},"last":0.59999999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.52900000000000003,"15min":0.55700000000000005},"min":{"1min":0.39000000000000001,"5min":0.35699999999999998,"15min":0.30499999999999999},"max":{"1min":0.748,"5min":0.86499999999999999,"15min":2.7290000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.62,"15min":0.64200000000000002},"min":{"1min":0.41799999999999998,"5min":0.38600000000000001,"15min":0.373},"max":{"1min":0.82399999999999995,"5min":0.89600000000000002,"15min":7.6760000000000002},"last":0.81899999999999995}]},{"osd":4,"last update":"Mon Jan 15 18:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.81000000000000005,"15min":0.84199999999999997},"min":{"1min":0.60699999999999998,"5min":0.55800000000000005,"15min":0.44},"max":{"1min":0.98699999999999999,"5min":1.22,"15min":4.3150000000000004},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.77700000000000002,"15min":0.80200000000000005},"min":{"1min":0.55900000000000005,"5min":0.47699999999999998,"15min":0.41599999999999998},"max":{"1min":0.91300000000000003,"5min":1.9670000000000001,"15min":8.3190000000000008},"last":0.71799999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.76800000000000002,"15min":0.78500000000000003},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.41899999999999998},"max":{"1min":0.84699999999999998,"5min":1.111,"15min":5.0309999999999997},"last":0.745},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.82599999999999996,"15min":0.86399999999999999},"min":{"1min":0.58499999999999996,"5min":0.57399999999999995,"15min":0.48199999999999998},"max":{"1min":0.93999999999999995,"5min":1.8240000000000001,"15min":10.460000000000001},"last":0.73699999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.79400000000000004,"15min":0.86599999999999999},"min":{"1min":0.60999999999999999,"5min":0.55800000000000005,"15min":0.40899999999999997},"max":{"1min":0.93799999999999994,"5min":1.3979999999999999,"15min":8.3580000000000005},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.80800000000000005,"15min":0.80600000000000005},"min":{"1min":0.54700000000000004,"5min":0.5,"15min":0.47899999999999998},"max":{"1min":0.91100000000000003,"5min":1.873,"15min":2.669},"last":0.83099999999999996}]},{"osd":7,"last update":"Mon Jan 15 18:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.80900000000000005,"15min":0.91800000000000004},"min":{"1min":0.65100000000000002,"5min":0.60699999999999998,"15min":0.44400000000000001},"max":{"1min":0.96999999999999997,"5min":1.7729999999999999,"15min":11.606},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.85799999999999998,"15min":0.88},"min":{"1min":0.67300000000000004,"5min":0.627,"15min":0.47899999999999998},"max":{"1min":0.97399999999999998,"5min":1.611,"15min":6.1260000000000003},"last":0.78400000000000003}]}]},{"osd":1,"up_from":15,"seq":64424509683,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1133516,"kb_used_data":1066308,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93238324,"statfs":{"total":96636764160,"available":95476043776,"internally_reserved":0,"allocated":1091899392,"data_stored":1091115320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:24:13 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.45200000000000001,"15min":0.53400000000000003},"min":{"1min":0.26800000000000002,"5min":0.26700000000000002,"15min":0.23699999999999999},"max":{"1min":0.54000000000000004,"5min":0.69699999999999995,"15min":17.396999999999998},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.502,"15min":0.57999999999999996},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.26900000000000002},"max":{"1min":0.61199999999999999,"5min":0.82999999999999996,"15min":12.378},"last":0.53500000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:24:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.54300000000000004},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.72699999999999998,"5min":0.878,"15min":2.8610000000000002},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54500000000000004,"15min":0.57799999999999996},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":3.0449999999999999},"last":0.49399999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:24:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58499999999999996,"15min":0.622},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.33700000000000002},"max":{"1min":0.79300000000000004,"5min":1.0580000000000001,"15min":3.0899999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.52700000000000002,"15min":0.626},"min":{"1min":0.34699999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.754,"5min":0.85499999999999998,"15min":9.9719999999999995},"last":0.55900000000000005}]},{"osd":4,"last update":"Mon Jan 15 18:24:13 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.77300000000000002,"15min":0.86099999999999999},"min":{"1min":0.51600000000000001,"5min":0.50900000000000001,"15min":0.437},"max":{"1min":0.92800000000000005,"5min":1.1060000000000001,"15min":9.9329999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.77400000000000002,"15min":0.84999999999999998},"min":{"1min":0.496,"5min":0.496,"15min":0.45800000000000002},"max":{"1min":0.98999999999999999,"5min":1.165,"15min":9.8610000000000007},"last":0.66200000000000003}]},{"osd":5,"last update":"Mon Jan 15 18:24:13 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.79300000000000004,"15min":0.82999999999999996},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.46100000000000002},"max":{"1min":0.83599999999999997,"5min":1.2050000000000001,"15min":8.5839999999999996},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.78100000000000003,"15min":0.92700000000000005},"min":{"1min":0.53100000000000003,"5min":0.52300000000000002,"15min":0.443},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":26.748999999999999},"last":0.625}]},{"osd":6,"last update":"Mon Jan 15 18:24:13 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.78600000000000003,"15min":0.88100000000000001},"min":{"1min":0.59099999999999997,"5min":0.56799999999999995,"15min":0.42899999999999999},"max":{"1min":1.0209999999999999,"5min":1.218,"15min":9.6349999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.80300000000000005,"15min":0.85699999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.501},"max":{"1min":0.99199999999999999,"5min":1.2090000000000001,"15min":5.8849999999999998},"last":0.70799999999999996}]},{"osd":7,"last update":"Mon Jan 15 18:24:13 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.80900000000000005,"15min":0.96799999999999997},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.46899999999999997},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":20.872},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.81399999999999995,"15min":0.94099999999999995},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.081,"5min":1.139,"15min":9.8640000000000008},"last":0.64400000000000002}]}]},{"osd":5,"up_from":14,"seq":60129542386,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":908420,"kb_used_data":828732,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":93463420,"statfs":{"total":96636764160,"available":95706542080,"internally_reserved":0,"allocated":848621568,"data_stored":847844597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81590725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:24:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.71599999999999997,"15min":0.74299999999999999},"min":{"1min":0.48599999999999999,"5min":0.39400000000000002,"15min":0.29899999999999999},"max":{"1min":0.879,"5min":2.6920000000000002,"15min":5.1639999999999997},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.76200000000000001,"15min":0.80700000000000005},"min":{"1min":0.45000000000000001,"5min":0.36799999999999999,"15min":0.34899999999999998},"max":{"1min":0.93799999999999994,"5min":6.4290000000000003,"15min":9.359},"last":0.81999999999999995}]},{"osd":1,"last update":"Mon Jan 15 18:24:36 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.77600000000000002,"15min":0.91400000000000003},"min":{"1min":0.47999999999999998,"5min":0.45100000000000001,"15min":0.375},"max":{"1min":0.89300000000000002,"5min":6.5069999999999997,"15min":21.120000000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.746,"15min":0.92000000000000004},"min":{"1min":0.42299999999999999,"5min":0.39900000000000002,"15min":0.38600000000000001},"max":{"1min":0.89300000000000002,"5min":2.6619999999999999,"15min":15.843},"last":0.71199999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:24:36 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.76900000000000002,"15min":0.81899999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.34999999999999998},"max":{"1min":0.879,"5min":6.3339999999999996,"15min":6.3339999999999996},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.80600000000000005,"15min":0.95199999999999996},"min":{"1min":0.56299999999999994,"5min":0.501,"15min":0.39900000000000002},"max":{"1min":0.86899999999999999,"5min":6.4619999999999997,"15min":21.201000000000001},"last":0.748}]},{"osd":3,"last update":"Mon Jan 15 18:24:36 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.75800000000000001,"15min":0.81499999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.37},"max":{"1min":0.94199999999999995,"5min":2.6299999999999999,"15min":7.7199999999999998},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.82399999999999995,"15min":0.85699999999999998},"min":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.47099999999999997},"max":{"1min":0.92800000000000005,"5min":6.3730000000000002,"15min":6.8650000000000002},"last":0.68799999999999994}]},{"osd":4,"last update":"Mon Jan 15 18:24:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.65900000000000003,"15min":0.80100000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.30399999999999999},"max":{"1min":0.74199999999999999,"5min":3.161,"15min":10.962},"last":0.505},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.70399999999999996,"15min":0.92200000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.35099999999999998},"max":{"1min":0.91000000000000003,"5min":6.5460000000000003,"15min":21.724},"last":0.47199999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:24:36 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.65700000000000003,"15min":0.72799999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.36099999999999999},"max":{"1min":0.97799999999999998,"5min":2.589,"15min":5.2160000000000002},"last":0.621},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.82299999999999995,"15min":0.86099999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.379},"max":{"1min":0.96199999999999997,"5min":14.007999999999999,"15min":14.007999999999999},"last":0.59699999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:24:36 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.72499999999999998,"15min":1.179},"min":{"1min":0.55000000000000004,"5min":0.48899999999999999,"15min":0.42599999999999999},"max":{"1min":0.92200000000000004,"5min":3.0870000000000002,"15min":33.499000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.90200000000000002,"15min":1.0549999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.42899999999999999},"max":{"1min":0.83799999999999997,"5min":21.178999999999998,"15min":25.437999999999999},"last":0.54400000000000004}]}]},{"osd":0,"up_from":14,"seq":60129542386,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1769580,"kb_used_data":1702372,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92602260,"statfs":{"total":96636764160,"available":94824714240,"internally_reserved":0,"allocated":1743228928,"data_stored":1742435619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:24:23 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.443,"15min":0.72399999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.245},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":38.521999999999998},"last":0.54200000000000004},{"interface":"front","average":{"1min":1.5029999999999999,"5min":0.73199999999999998,"15min":0.747},"min":{"1min":0.34699999999999998,"5min":0.32900000000000001,"15min":0.27000000000000002},"max":{"1min":21.501000000000001,"5min":21.501000000000001,"15min":21.501000000000001},"last":0.51600000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:24:23 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":0.75800000000000001,"15min":0.85899999999999999},"min":{"1min":0.34399999999999997,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":21.545999999999999,"5min":21.545999999999999,"15min":25.905000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.496,"5min":0.502,"15min":0.65900000000000003},"min":{"1min":0.32200000000000001,"5min":0.307,"15min":0.27800000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":18.247},"last":0.38500000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:24:23 2024","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":0.66200000000000003,"15min":0.81299999999999994},"min":{"1min":0.35199999999999998,"5min":0.32400000000000001,"15min":0.29199999999999998},"max":{"1min":8.8740000000000006,"5min":8.8740000000000006,"15min":21.451000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":1.633,"5min":0.82499999999999996,"15min":0.80600000000000005},"min":{"1min":0.40899999999999997,"5min":0.38900000000000001,"15min":0.38400000000000001},"max":{"1min":21.428000000000001,"5min":21.428000000000001,"15min":21.428000000000001},"last":0.60599999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:24:23 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.78700000000000003,"15min":0.88700000000000001},"min":{"1min":0.52000000000000002,"5min":0.441,"15min":0.39100000000000001},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":10.691000000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.746,"5min":0.77400000000000002,"15min":1.054},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.502,"5min":1.502,"15min":27.027999999999999},"last":0.74199999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:24:23 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.78200000000000003,"15min":0.91400000000000003},"min":{"1min":0.50700000000000001,"5min":0.48299999999999998,"15min":0.38400000000000001},"max":{"1min":1.464,"5min":1.464,"15min":10.737},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.78400000000000003,"15min":1.2569999999999999},"min":{"1min":0.48899999999999999,"5min":0.47999999999999998,"15min":0.46100000000000002},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":27.210999999999999},"last":0.81899999999999995}]},{"osd":6,"last update":"Mon Jan 15 18:24:23 2024","interfaces":[{"interface":"back","average":{"1min":1.794,"5min":0.98599999999999999,"15min":1.175},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.46700000000000003},"max":{"1min":21.593,"5min":21.593,"15min":26.036000000000001},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.81799999999999995,"15min":0.90200000000000002},"min":{"1min":0.56599999999999995,"5min":0.52900000000000003,"15min":0.51500000000000001},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":6.9690000000000003},"last":0.72199999999999998}]},{"osd":7,"last update":"Mon Jan 15 18:24:23 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82399999999999995,"15min":1.1599999999999999},"min":{"1min":0.54600000000000004,"5min":0.52200000000000002,"15min":0.499},"max":{"1min":1.524,"5min":1.524,"15min":24.492999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.82899999999999996,"15min":1.0449999999999999},"min":{"1min":0.58699999999999997,"5min":0.53200000000000003,"15min":0.49199999999999999},"max":{"1min":1.859,"5min":1.859,"15min":27.081},"last":0.70899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":630784,"data_stored":614328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":1321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":626688,"data_stored":614286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1740644352,"data_stored":1740636601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1090527232,"data_stored":1090519412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":931139584,"data_stored":931135680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1417682944,"data_stored":1417675274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1681920000,"data_stored":1681916165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":847253504,"data_stored":847249530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1220546560,"data_stored":1220542586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1807745024,"data_stored":1807745024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:24:42.817 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-01-15T18:24:42.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.998+0000 7fbe58200640 1 Processor -- start 2024-01-15T18:24:42.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.998+0000 7fbe58200640 1 -- start start 2024-01-15T18:24:42.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.998+0000 7fbe58200640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbe5006c430 0x7fbe50070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:42.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.998+0000 7fbe58200640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7fbe500b1db0 0x7fbe500b2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:42.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.998+0000 7fbe58200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbe500ce050 0x7fbe500ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:42.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.998+0000 7fbe58200640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe500cea20 con 0x7fbe5006c430 2024-01-15T18:24:42.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.998+0000 7fbe58200640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe500ccfb0 con 0x7fbe500b1db0 2024-01-15T18:24:42.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.998+0000 7fbe58200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe500cd290 con 0x7fbe500ce050 2024-01-15T18:24:42.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:42.998+0000 7fbe55800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbe5006c430 0x7fbe50070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:42.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.002+0000 7fbe55800640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbe5006c430 0x7fbe50070ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55082/0 (socket says 172.21.15.27:55082) 2024-01-15T18:24:42.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.002+0000 7fbe55800640 1 -- 172.21.15.27:0/1675575630 learned_addr learned my addr 172.21.15.27:0/1675575630 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:24:42.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.002+0000 7fbe54e00640 1 --2- 172.21.15.27:0/1675575630 >> v2:172.21.15.93:3300/0 conn(0x7fbe500b1db0 0x7fbe500b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:42.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.002+0000 7fbe56200640 1 --2- 172.21.15.27:0/1675575630 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbe500ce050 0x7fbe500ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:42.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.002+0000 7fbe55800640 1 -- 172.21.15.27:0/1675575630 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbe500ce050 msgr2=0x7fbe500ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:24:42.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.002+0000 7fbe55800640 1 --2- 172.21.15.27:0/1675575630 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbe500ce050 0x7fbe500ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:42.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.002+0000 7fbe55800640 1 -- 172.21.15.27:0/1675575630 >> v2:172.21.15.93:3300/0 conn(0x7fbe500b1db0 msgr2=0x7fbe500b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:24:42.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.002+0000 7fbe55800640 1 --2- 172.21.15.27:0/1675575630 >> v2:172.21.15.93:3300/0 conn(0x7fbe500b1db0 0x7fbe500b2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:42.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.002+0000 7fbe55800640 1 -- 172.21.15.27:0/1675575630 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe500cd570 con 0x7fbe5006c430 2024-01-15T18:24:42.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.002+0000 7fbe55800640 1 --2- 172.21.15.27:0/1675575630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe5006c430 0x7fbe50070ba0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fbe40009870 tx=0x7fbe4001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8fb9b299924d79d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:24:43.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.002+0000 7fbe4fe00640 1 -- 172.21.15.27:0/1675575630 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe400ad680 con 0x7fbe5006c430 2024-01-15T18:24:43.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.002+0000 7fbe4fe00640 1 -- 172.21.15.27:0/1675575630 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe400ad7e0 con 0x7fbe5006c430 2024-01-15T18:24:43.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.002+0000 7fbe4fe00640 1 -- 172.21.15.27:0/1675575630 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe400adb70 con 0x7fbe5006c430 2024-01-15T18:24:43.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.006+0000 7fbe58200640 1 -- 172.21.15.27:0/1675575630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe5006c430 msgr2=0x7fbe50070ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:43.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.006+0000 7fbe58200640 1 --2- 172.21.15.27:0/1675575630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe5006c430 0x7fbe50070ba0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fbe40009870 tx=0x7fbe4001c440 comp rx=0 tx=0).stop 2024-01-15T18:24:43.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.006+0000 7fbe58200640 1 -- 172.21.15.27:0/1675575630 shutdown_connections 2024-01-15T18:24:43.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.006+0000 7fbe58200640 1 --2- 172.21.15.27:0/1675575630 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbe500ce050 0x7fbe500ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:43.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.006+0000 7fbe58200640 1 --2- 172.21.15.27:0/1675575630 >> v2:172.21.15.93:3300/0 conn(0x7fbe500b1db0 0x7fbe500b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:43.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.006+0000 7fbe58200640 1 --2- 172.21.15.27:0/1675575630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe5006c430 0x7fbe50070ba0 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:43.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.006+0000 7fbe58200640 1 -- 172.21.15.27:0/1675575630 >> 172.21.15.27:0/1675575630 conn(0x7fbe500cb700 msgr2=0x7fbe500cbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:24:43.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.006+0000 7fbe58200640 1 -- 172.21.15.27:0/1675575630 shutdown_connections 2024-01-15T18:24:43.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.006+0000 7fbe58200640 1 -- 172.21.15.27:0/1675575630 wait complete. 2024-01-15T18:24:43.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.006+0000 7fbe58200640 1 Processor -- start 2024-01-15T18:24:43.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.006+0000 7fbe58200640 1 -- start start 2024-01-15T18:24:43.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.006+0000 7fbe58200640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbe5006c430 0x7fbe500a91d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:43.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.006+0000 7fbe58200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe500b1db0 0x7fbe50153670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:43.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.006+0000 7fbe58200640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbe500ce050 0x7fbe50141880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:43.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.006+0000 7fbe58200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe50153f00 con 0x7fbe500b1db0 2024-01-15T18:24:43.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.006+0000 7fbe58200640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe501541e0 con 0x7fbe5006c430 2024-01-15T18:24:43.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.006+0000 7fbe58200640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe501544c0 con 0x7fbe500ce050 2024-01-15T18:24:43.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.006+0000 7fbe54e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe500b1db0 0x7fbe50153670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:43.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.006+0000 7fbe54e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe500b1db0 0x7fbe50153670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55086/0 (socket says 172.21.15.27:55086) 2024-01-15T18:24:43.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.006+0000 7fbe54e00640 1 -- 172.21.15.27:0/1028770358 learned_addr learned my addr 172.21.15.27:0/1028770358 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:24:43.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.006+0000 7fbe55800640 1 --2- 172.21.15.27:0/1028770358 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbe5006c430 0x7fbe500a91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:43.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.006+0000 7fbe56200640 1 --2- 172.21.15.27:0/1028770358 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbe500ce050 0x7fbe50141880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:43.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.006+0000 7fbe54e00640 1 -- 172.21.15.27:0/1028770358 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbe500ce050 msgr2=0x7fbe50141880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:43.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.006+0000 7fbe54e00640 1 --2- 172.21.15.27:0/1028770358 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbe500ce050 0x7fbe50141880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:43.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.010+0000 7fbe54e00640 1 -- 172.21.15.27:0/1028770358 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbe5006c430 msgr2=0x7fbe500a91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:43.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.010+0000 7fbe54e00640 1 --2- 172.21.15.27:0/1028770358 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbe5006c430 0x7fbe500a91d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:43.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.010+0000 7fbe54e00640 1 -- 172.21.15.27:0/1028770358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe400094d0 con 0x7fbe500b1db0 2024-01-15T18:24:43.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.010+0000 7fbe54e00640 1 --2- 172.21.15.27:0/1028770358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe500b1db0 0x7fbe50153670 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7fbe3800d800 tx=0x7fbe3800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:24:43.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.010+0000 7fbe4d600640 1 -- 172.21.15.27:0/1028770358 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3800b6c0 con 0x7fbe500b1db0 2024-01-15T18:24:43.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.010+0000 7fbe4d600640 1 -- 172.21.15.27:0/1028770358 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe3800b820 con 0x7fbe500b1db0 2024-01-15T18:24:43.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.010+0000 7fbe58200640 1 -- 172.21.15.27:0/1028770358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe501547a0 con 0x7fbe500b1db0 2024-01-15T18:24:43.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.010+0000 7fbe4d600640 1 -- 172.21.15.27:0/1028770358 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3800bb90 con 0x7fbe500b1db0 2024-01-15T18:24:43.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.010+0000 7fbe58200640 1 -- 172.21.15.27:0/1028770358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe50153bb0 con 0x7fbe500b1db0 2024-01-15T18:24:43.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.010+0000 7fbe58200640 1 -- 172.21.15.27:0/1028770358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe5004e850 con 0x7fbe500b1db0 2024-01-15T18:24:43.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.010+0000 7fbe4d600640 1 -- 172.21.15.27:0/1028770358 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 87690+0+0 (secure 0 0 0) 0x7fbe3800bcf0 con 0x7fbe500b1db0 2024-01-15T18:24:43.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.014+0000 7fbe4d600640 1 --2- 172.21.15.27:0/1028770358 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbe28067a00 0x7fbe28069ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:43.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.014+0000 7fbe4d600640 1 -- 172.21.15.27:0/1028770358 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fbe38083350 con 0x7fbe500b1db0 2024-01-15T18:24:43.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.014+0000 7fbe55800640 1 --2- 172.21.15.27:0/1028770358 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbe28067a00 0x7fbe28069ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:43.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.014+0000 7fbe55800640 1 --2- 172.21.15.27:0/1028770358 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbe28067a00 0x7fbe28069ec0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fbe400af000 tx=0x7fbe400a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:24:43.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.018+0000 7fbe4d600640 1 -- 172.21.15.27:0/1028770358 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbe380520f0 con 0x7fbe500b1db0 2024-01-15T18:24:43.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.174+0000 7fbe58200640 1 -- 172.21.15.27:0/1028770358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fbe5015ced0 con 0x7fbe500b1db0 2024-01-15T18:24:43.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.178+0000 7fbe4d600640 1 -- 172.21.15.27:0/1028770358 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v43)=0 v43) v1 ==== 123+0+0 (secure 0 0 0) 0x7fbe38055710 con 0x7fbe500b1db0 2024-01-15T18:24:43.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.178+0000 7fbe4d600640 1 -- 172.21.15.27:0/1028770358 <== mon.0 v2:172.21.15.27:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fbe380520f0 con 0x7fbe500b1db0 2024-01-15T18:24:43.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.182+0000 7fbe58200640 1 -- 172.21.15.27:0/1028770358 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbe28067a00 msgr2=0x7fbe28069ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:43.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.182+0000 7fbe58200640 1 --2- 172.21.15.27:0/1028770358 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbe28067a00 0x7fbe28069ec0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fbe400af000 tx=0x7fbe400a7000 comp rx=0 tx=0).stop 2024-01-15T18:24:43.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.182+0000 7fbe58200640 1 -- 172.21.15.27:0/1028770358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe500b1db0 msgr2=0x7fbe50153670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:43.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.182+0000 7fbe58200640 1 --2- 172.21.15.27:0/1028770358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe500b1db0 0x7fbe50153670 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7fbe3800d800 tx=0x7fbe3800dcc0 comp rx=0 tx=0).stop 2024-01-15T18:24:43.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.182+0000 7fbe58200640 1 -- 172.21.15.27:0/1028770358 shutdown_connections 2024-01-15T18:24:43.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.182+0000 7fbe58200640 1 --2- 172.21.15.27:0/1028770358 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7fbe28067a00 0x7fbe28069ec0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:43.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.182+0000 7fbe58200640 1 --2- 172.21.15.27:0/1028770358 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7fbe500ce050 0x7fbe50141880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:43.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.182+0000 7fbe58200640 1 --2- 172.21.15.27:0/1028770358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe500b1db0 0x7fbe50153670 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:43.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.182+0000 7fbe58200640 1 --2- 172.21.15.27:0/1028770358 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7fbe5006c430 0x7fbe500a91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:43.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.182+0000 7fbe58200640 1 -- 172.21.15.27:0/1028770358 >> 172.21.15.27:0/1028770358 conn(0x7fbe500cb700 msgr2=0x7fbe500b2240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:24:43.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.182+0000 7fbe58200640 1 -- 172.21.15.27:0/1028770358 shutdown_connections 2024-01-15T18:24:43.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.182+0000 7fbe58200640 1 -- 172.21.15.27:0/1028770358 wait complete. 2024-01-15T18:24:43.192 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:24:43.195+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:24:43.192 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:24:43.195+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:24:43.192 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:24:43.195+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:24:43.193 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:24:43.195+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:24:43.193 INFO:tasks.ceph.mgr.x.smithi093.stderr:2024-01-15T18:24:43.195+0000 7f59f7e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:24:43.198 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-15T18:24:43.199 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:24:43.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.346+0000 7f320be00640 1 Processor -- start 2024-01-15T18:24:43.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.346+0000 7f320be00640 1 -- start start 2024-01-15T18:24:43.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.346+0000 7f320be00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f320413e000 0x7f3204141170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:43.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.346+0000 7f320be00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f32040cd3e0 0x7f320415ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:43.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.346+0000 7f320be00640 1 --2- >> v2:172.21.15.93:3300/0 conn(0x7f320415efd0 0x7f320415b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:43.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.346+0000 7f320be00640 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f320415f460 con 0x7f32040cd3e0 2024-01-15T18:24:43.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.346+0000 7f320be00640 1 -- --> v2:172.21.15.93:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32040ce7b0 con 0x7f320415efd0 2024-01-15T18:24:43.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.346+0000 7f320be00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32040cdc30 con 0x7f320413e000 2024-01-15T18:24:43.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.346+0000 7f3208a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f32040cd3e0 0x7f320415ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:43.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.346+0000 7f3208a00640 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f32040cd3e0 0x7f320415ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55088/0 (socket says 172.21.15.27:55088) 2024-01-15T18:24:43.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.346+0000 7f3208a00640 1 -- 172.21.15.27:0/4004933244 learned_addr learned my addr 172.21.15.27:0/4004933244 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:24:43.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.346+0000 7f3209e00640 1 --2- 172.21.15.27:0/4004933244 >> v2:172.21.15.93:3300/0 conn(0x7f320415efd0 0x7f320415b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:43.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.346+0000 7f3208a00640 1 -- 172.21.15.27:0/4004933244 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f320413e000 msgr2=0x7f3204141170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:24:43.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.346+0000 7f3209400640 1 --2- 172.21.15.27:0/4004933244 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f320413e000 0x7f3204141170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:43.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.346+0000 7f3208a00640 1 --2- 172.21.15.27:0/4004933244 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f320413e000 0x7f3204141170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:43.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.346+0000 7f3208a00640 1 -- 172.21.15.27:0/4004933244 >> v2:172.21.15.93:3300/0 conn(0x7f320415efd0 msgr2=0x7f320415b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:24:43.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.346+0000 7f3208a00640 1 --2- 172.21.15.27:0/4004933244 >> v2:172.21.15.93:3300/0 conn(0x7f320415efd0 0x7f320415b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:43.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.346+0000 7f3208a00640 1 -- 172.21.15.27:0/4004933244 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32040cdf10 con 0x7f32040cd3e0 2024-01-15T18:24:43.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.346+0000 7f3209e00640 1 --2- 172.21.15.27:0/4004933244 >> v2:172.21.15.93:3300/0 conn(0x7f320415efd0 0x7f320415b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:24:43.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.346+0000 7f3209400640 1 --2- 172.21.15.27:0/4004933244 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f320413e000 0x7f3204141170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:24:43.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.350+0000 7f3208a00640 1 --2- 172.21.15.27:0/4004933244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32040cd3e0 0x7f320415ea90 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f320406e600 tx=0x7f31ec01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ba6187f7c8db7f6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:24:43.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.350+0000 7f3203e00640 1 -- 172.21.15.27:0/4004933244 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec0ab600 con 0x7f32040cd3e0 2024-01-15T18:24:43.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.350+0000 7f3203e00640 1 -- 172.21.15.27:0/4004933244 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f31ec0a7050 con 0x7f32040cd3e0 2024-01-15T18:24:43.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.350+0000 7f3203e00640 1 -- 172.21.15.27:0/4004933244 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec0ab9a0 con 0x7f32040cd3e0 2024-01-15T18:24:43.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.350+0000 7f320be00640 1 -- 172.21.15.27:0/4004933244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32040cd3e0 msgr2=0x7f320415ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:43.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.350+0000 7f320be00640 1 --2- 172.21.15.27:0/4004933244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32040cd3e0 0x7f320415ea90 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f320406e600 tx=0x7f31ec01c470 comp rx=0 tx=0).stop 2024-01-15T18:24:43.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.350+0000 7f320be00640 1 -- 172.21.15.27:0/4004933244 shutdown_connections 2024-01-15T18:24:43.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.350+0000 7f320be00640 1 --2- 172.21.15.27:0/4004933244 >> v2:172.21.15.93:3300/0 conn(0x7f320415efd0 0x7f320415b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:43.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.350+0000 7f320be00640 1 --2- 172.21.15.27:0/4004933244 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32040cd3e0 0x7f320415ea90 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:43.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.350+0000 7f320be00640 1 --2- 172.21.15.27:0/4004933244 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f320413e000 0x7f3204141170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:43.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.350+0000 7f320be00640 1 -- 172.21.15.27:0/4004933244 >> 172.21.15.27:0/4004933244 conn(0x7f320406b0c0 msgr2=0x7f32040cec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:24:43.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.350+0000 7f320be00640 1 -- 172.21.15.27:0/4004933244 shutdown_connections 2024-01-15T18:24:43.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.350+0000 7f320be00640 1 -- 172.21.15.27:0/4004933244 wait complete. 2024-01-15T18:24:43.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.354+0000 7f320be00640 1 Processor -- start 2024-01-15T18:24:43.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.354+0000 7f320be00640 1 -- start start 2024-01-15T18:24:43.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.354+0000 7f320be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32040cd3e0 0x7f3204153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.354+0000 7f320be00640 1 --2- >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f320413e000 0x7f3204153940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.354+0000 7f320be00640 1 --2- >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f320415efd0 0x7f320414ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.354+0000 7f320be00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32040aea50 con 0x7f32040cd3e0 2024-01-15T18:24:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.354+0000 7f320be00640 1 -- --> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32040aed30 con 0x7f320413e000 2024-01-15T18:24:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.354+0000 7f320be00640 1 -- --> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32040af010 con 0x7f320415efd0 2024-01-15T18:24:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.354+0000 7f3209400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32040cd3e0 0x7f3204153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.354+0000 7f3209400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32040cd3e0 0x7f3204153400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55090/0 (socket says 172.21.15.27:55090) 2024-01-15T18:24:43.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.354+0000 7f3209400640 1 -- 172.21.15.27:0/70502303 learned_addr learned my addr 172.21.15.27:0/70502303 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-01-15T18:24:43.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.354+0000 7f3208a00640 1 --2- 172.21.15.27:0/70502303 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f320413e000 0x7f3204153940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:43.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.354+0000 7f3209e00640 1 --2- 172.21.15.27:0/70502303 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f320415efd0 0x7f320414ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:43.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.354+0000 7f3209400640 1 -- 172.21.15.27:0/70502303 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f320415efd0 msgr2=0x7f320414ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:43.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.354+0000 7f3209400640 1 --2- 172.21.15.27:0/70502303 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f320415efd0 0x7f320414ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:43.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.354+0000 7f3209400640 1 -- 172.21.15.27:0/70502303 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f320413e000 msgr2=0x7f3204153940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:43.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.354+0000 7f3209400640 1 --2- 172.21.15.27:0/70502303 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f320413e000 0x7f3204153940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:43.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.354+0000 7f3209400640 1 -- 172.21.15.27:0/70502303 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ec0094d0 con 0x7f32040cd3e0 2024-01-15T18:24:43.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.354+0000 7f3209e00640 1 --2- 172.21.15.27:0/70502303 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f320415efd0 0x7f320414ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:24:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.354+0000 7f3209400640 1 --2- 172.21.15.27:0/70502303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32040cd3e0 0x7f3204153400 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f31f400c7c0 tx=0x7f31f400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:24:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.354+0000 7f3201600640 1 -- 172.21.15.27:0/70502303 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f4005ba0 con 0x7f32040cd3e0 2024-01-15T18:24:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.354+0000 7f3201600640 1 -- 172.21.15.27:0/70502303 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f31f4005d00 con 0x7f32040cd3e0 2024-01-15T18:24:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.354+0000 7f3201600640 1 -- 172.21.15.27:0/70502303 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f4007a70 con 0x7f32040cd3e0 2024-01-15T18:24:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.354+0000 7f320be00640 1 -- 172.21.15.27:0/70502303 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32040af2f0 con 0x7f32040cd3e0 2024-01-15T18:24:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.354+0000 7f320be00640 1 -- 172.21.15.27:0/70502303 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3204153e80 con 0x7f32040cd3e0 2024-01-15T18:24:43.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.358+0000 7f320be00640 1 -- 172.21.15.27:0/70502303 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f320404e850 con 0x7f32040cd3e0 2024-01-15T18:24:43.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.358+0000 7f3201600640 1 -- 172.21.15.27:0/70502303 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 46) v1 ==== 87690+0+0 (secure 0 0 0) 0x7f31f4007bd0 con 0x7f32040cd3e0 2024-01-15T18:24:43.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.358+0000 7f3201600640 1 --2- 172.21.15.27:0/70502303 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f31dc0679d0 0x7f31dc069e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:24:43.355 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.358+0000 7f3201600640 1 -- 172.21.15.27:0/70502303 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f31f40831d0 con 0x7f32040cd3e0 2024-01-15T18:24:43.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.358+0000 7f3208a00640 1 --2- 172.21.15.27:0/70502303 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f31dc0679d0 0x7f31dc069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:24:43.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.358+0000 7f3208a00640 1 --2- 172.21.15.27:0/70502303 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f31dc0679d0 0x7f31dc069e90 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f31ec0ad000 tx=0x7f31ec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:24:43.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.362+0000 7f3201600640 1 -- 172.21.15.27:0/70502303 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f31f4051bd0 con 0x7f32040cd3e0 2024-01-15T18:24:43.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.558+0000 7f320be00640 1 -- 172.21.15.27:0/70502303 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3204165c00 con 0x7f32040cd3e0 2024-01-15T18:24:43.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.558+0000 7f3201600640 1 -- 172.21.15.27:0/70502303 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f31f401e070 con 0x7f32040cd3e0 2024-01-15T18:24:43.554 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-01-15T18:24:43.554 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":78,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7721,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/3036261071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":3036261071},{"type":"v1","addr":"172.21.15.27:6835","nonce":3036261071}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72},{"gid":7724,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.93:6836/3646217660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6834","nonce":3646217660},{"type":"v1","addr":"172.21.15.93:6836","nonce":3646217660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":72},{"gid":9441,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/3131124481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":3131124481},{"type":"v1","addr":"172.21.15.27:6834","nonce":3131124481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":71}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:18:32.593319+0000","modified":"2024-01-15T18:18:35.606888+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9450},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9450":{"gid":9450,"name":"d","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.93:6837/2040168756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.93:6835","nonce":2040168756},{"type":"v1","addr":"172.21.15.93:6837","nonce":2040168756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-01-15T18:24:43.554 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 78 2024-01-15T18:24:43.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.558+0000 7f320be00640 1 -- 172.21.15.27:0/70502303 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f31dc0679d0 msgr2=0x7f31dc069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:43.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.558+0000 7f320be00640 1 --2- 172.21.15.27:0/70502303 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f31dc0679d0 0x7f31dc069e90 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f31ec0ad000 tx=0x7f31ec008040 comp rx=0 tx=0).stop 2024-01-15T18:24:43.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.558+0000 7f320be00640 1 -- 172.21.15.27:0/70502303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32040cd3e0 msgr2=0x7f3204153400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:24:43.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.558+0000 7f320be00640 1 --2- 172.21.15.27:0/70502303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32040cd3e0 0x7f3204153400 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f31f400c7c0 tx=0x7f31f400cc80 comp rx=0 tx=0).stop 2024-01-15T18:24:43.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.558+0000 7f320be00640 1 -- 172.21.15.27:0/70502303 shutdown_connections 2024-01-15T18:24:43.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.558+0000 7f320be00640 1 --2- 172.21.15.27:0/70502303 >> [v2:172.21.15.93:6832/1477920209,v1:172.21.15.93:6833/1477920209] conn(0x7f31dc0679d0 0x7f31dc069e90 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:43.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.558+0000 7f320be00640 1 --2- 172.21.15.27:0/70502303 >> [v2:172.21.15.93:3301/0,v1:172.21.15.93:6790/0] conn(0x7f320415efd0 0x7f320414ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:43.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.558+0000 7f320be00640 1 --2- 172.21.15.27:0/70502303 >> [v2:172.21.15.93:3300/0,v1:172.21.15.93:6789/0] conn(0x7f320413e000 0x7f3204153940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:43.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.558+0000 7f320be00640 1 --2- 172.21.15.27:0/70502303 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32040cd3e0 0x7f3204153400 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:24:43.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.558+0000 7f320be00640 1 -- 172.21.15.27:0/70502303 >> 172.21.15.27:0/70502303 conn(0x7f320406b0c0 msgr2=0x7f320413d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:24:43.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.558+0000 7f320be00640 1 -- 172.21.15.27:0/70502303 shutdown_connections 2024-01-15T18:24:43.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-01-15T18:24:43.558+0000 7f320be00640 1 -- 172.21.15.27:0/70502303 wait complete. 2024-01-15T18:24:43.576 DEBUG:tasks.cephfs.filesystem:already dead... 2024-01-15T18:24:43.576 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-15T18:24:43.576 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-15T18:24:43.577 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:24:43.589 INFO:tasks.ceph.mds.a:Stopped 2024-01-15T18:24:43.589 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-15T18:24:43.589 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:24:43.644 INFO:tasks.ceph.mds.c:Stopped 2024-01-15T18:24:43.644 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-15T18:24:43.644 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:24:43.658 INFO:tasks.ceph.mds.b:Stopped 2024-01-15T18:24:43.658 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-15T18:24:43.659 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:24:43.715 INFO:tasks.ceph.mds.d:Stopped 2024-01-15T18:24:43.715 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-15T18:24:43.715 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-15T18:24:43.716 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:24:45.967 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-01-15T18:24:49.818 INFO:tasks.ceph.osd.0:Stopped 2024-01-15T18:24:49.819 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-15T18:24:49.819 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:24:49.909 INFO:tasks.ceph.osd.1:Stopped 2024-01-15T18:24:49.910 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-15T18:24:49.910 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:24:50.004 INFO:tasks.ceph.osd.2:Stopped 2024-01-15T18:24:50.004 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-01-15T18:24:50.004 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:24:50.091 INFO:tasks.ceph.osd.3:Stopped 2024-01-15T18:24:50.092 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-01-15T18:24:50.092 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:24:51.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-01-15T18:24:51.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-01-15T18:24:56.194 INFO:tasks.ceph.osd.4:Stopped 2024-01-15T18:24:56.194 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-01-15T18:24:56.195 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:24:56.265 INFO:tasks.ceph.osd.5:Stopped 2024-01-15T18:24:56.265 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-01-15T18:24:56.265 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:24:57.379 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-01-15T18:24:57.380 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-01-15T18:24:58.823 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:24:58.827 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:25:02.367 INFO:tasks.ceph.osd.6:Stopped 2024-01-15T18:25:02.367 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-01-15T18:25:02.367 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:25:02.465 INFO:tasks.ceph.osd.7:Stopped 2024-01-15T18:25:02.465 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-15T18:25:02.466 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-01-15T18:25:02.466 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:25:02.521 INFO:tasks.ceph.mgr.y:Stopped 2024-01-15T18:25:02.522 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-15T18:25:02.522 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:25:02.768 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-01-15T18:25:02.768 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2024-01-15T18:25:02.768 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-01-15T18:25:08.072 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~5s 2024-01-15T18:25:08.072 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-01-15T18:25:08.623 INFO:tasks.ceph.mgr.x:Stopped 2024-01-15T18:25:08.623 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-15T18:25:08.624 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-15T18:25:08.624 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:25:08.649 INFO:tasks.ceph.mon.a:Stopped 2024-01-15T18:25:08.650 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-01-15T18:25:08.650 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:25:08.668 INFO:tasks.ceph.mon.b:Stopped 2024-01-15T18:25:08.669 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-01-15T18:25:08.669 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:25:08.733 INFO:tasks.ceph.mon.c:Stopped 2024-01-15T18:25:08.734 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-15T18:25:08.734 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2024-01-15T18:25:08.771 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T18:25:08.771 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-01-15T18:25:09.117 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T18:25:09.118 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-01-15T18:25:09.476 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T18:25:09.476 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-01-15T18:25:09.675 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T18:25:09.675 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-01-15T18:25:09.902 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T18:25:09.902 DEBUG:teuthology.orchestra.run.smithi093:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-01-15T18:25:10.188 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T18:25:10.189 DEBUG:teuthology.orchestra.run.smithi093:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-01-15T18:25:10.352 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T18:25:10.352 DEBUG:teuthology.orchestra.run.smithi093:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-01-15T18:25:10.536 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T18:25:10.537 DEBUG:teuthology.orchestra.run.smithi093:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-01-15T18:25:10.704 INFO:tasks.ceph:Archiving mon data... 2024-01-15T18:25:10.706 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2024-01-15_17:34:44-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517095/data/mon.a.tgz 2024-01-15T18:25:10.706 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2024-01-15T18:25:10.716 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.d3Jggxu6Zh 2024-01-15T18:25:10.716 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.d3Jggxu6Zh 2024-01-15T18:25:10.990 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.d3Jggxu6Zh 2024-01-15T18:25:11.069 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.d3Jggxu6Zh is 1MB 2024-01-15T18:25:11.144 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.d3Jggxu6Zh 2024-01-15T18:25:11.153 DEBUG:teuthology.misc:Transferring archived files from smithi093:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2024-01-15_17:34:44-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517095/data/mon.b.tgz 2024-01-15T18:25:11.153 DEBUG:teuthology.orchestra.run.smithi093:> mktemp 2024-01-15T18:25:11.161 INFO:teuthology.orchestra.run.smithi093.stdout:/tmp/tmp.Mi22RW7jir 2024-01-15T18:25:11.162 DEBUG:teuthology.orchestra.run.smithi093:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.Mi22RW7jir 2024-01-15T18:25:11.430 DEBUG:teuthology.orchestra.run.smithi093:> sudo chmod 0666 /tmp/tmp.Mi22RW7jir 2024-01-15T18:25:11.503 DEBUG:teuthology.orchestra.remote:smithi093:/tmp/tmp.Mi22RW7jir is 1MB 2024-01-15T18:25:11.578 DEBUG:teuthology.orchestra.run.smithi093:> rm -fr /tmp/tmp.Mi22RW7jir 2024-01-15T18:25:11.586 DEBUG:teuthology.misc:Transferring archived files from smithi093:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2024-01-15_17:34:44-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517095/data/mon.c.tgz 2024-01-15T18:25:11.586 DEBUG:teuthology.orchestra.run.smithi093:> mktemp 2024-01-15T18:25:11.637 INFO:teuthology.orchestra.run.smithi093.stdout:/tmp/tmp.PVlqgZ5wyM 2024-01-15T18:25:11.638 DEBUG:teuthology.orchestra.run.smithi093:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.PVlqgZ5wyM 2024-01-15T18:25:11.906 DEBUG:teuthology.orchestra.run.smithi093:> sudo chmod 0666 /tmp/tmp.PVlqgZ5wyM 2024-01-15T18:25:11.978 DEBUG:teuthology.orchestra.remote:smithi093:/tmp/tmp.PVlqgZ5wyM is 1MB 2024-01-15T18:25:12.054 DEBUG:teuthology.orchestra.run.smithi093:> rm -fr /tmp/tmp.PVlqgZ5wyM 2024-01-15T18:25:12.062 INFO:tasks.ceph:Cleaning ceph cluster... 2024-01-15T18:25:12.062 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-15T18:25:12.065 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-15T18:25:12.137 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-01-15T18:25:12.174 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-01-15T18:25:12.174 INFO:tasks.ceph:Archiving crash dumps... 2024-01-15T18:25:12.176 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-01-15_17:34:44-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517095/remote/smithi027/crash 2024-01-15T18:25:12.177 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-15T18:25:12.205 DEBUG:teuthology.misc:Transferring archived files from smithi093:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-01-15_17:34:44-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517095/remote/smithi093/crash 2024-01-15T18:25:12.206 DEBUG:teuthology.orchestra.run.smithi093:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-15T18:25:12.231 INFO:tasks.ceph:Shutting down logrotate 2024-01-15T18:25:12.232 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:25:12.251 DEBUG:teuthology.orchestra.run.smithi093:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:25:12.342 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-01-15T18:25:12.362 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-01-15T18:25:12.381 INFO:tasks.ceph:Compressing logs... 2024-01-15T18:25:12.382 DEBUG:teuthology.orchestra.run.smithi027:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-15T18:25:12.407 DEBUG:teuthology.orchestra.run.smithi093:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-15T18:25:12.432 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47382.log 2024-01-15T18:25:12.433 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38945.log 2024-01-15T18:25:12.434 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54153.log 2024-01-15T18:25:12.434 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47382.log.gz 2024-01-15T18:25:12.435 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32344.log 2024-01-15T18:25:12.435 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38945.log.gz 2024-01-15T18:25:12.436 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45836.log 2024-01-15T18:25:12.436 INFO:teuthology.orchestra.run.smithi027.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.54153.log.gz 2024-01-15T18:25:12.437 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35466.log 2024-01-15T18:25:12.438 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32344.log.gz 2024-01-15T18:25:12.438 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45836.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45836.log.gz 2024-01-15T18:25:12.438 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48123.log 2024-01-15T18:25:12.439 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38280.log 2024-01-15T18:25:12.439 INFO:teuthology.orchestra.run.smithi027.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35466.log.gz 2024-01-15T18:25:12.440 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52474.log 2024-01-15T18:25:12.440 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48123.log.gz 2024-01-15T18:25:12.441 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52669.log 2024-01-15T18:25:12.441 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38280.log.gz 2024-01-15T18:25:12.442 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52165.log 2024-01-15T18:25:12.443 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52474.log.gz 2024-01-15T18:25:12.444 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52669.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.52669.log.gz 2024-01-15T18:25:12.444 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38530.log 2024-01-15T18:25:12.445 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54728.log 2024-01-15T18:25:12.445 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38530.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.52165.log.gz 2024-01-15T18:25:12.446 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38530.log.gz 2024-01-15T18:25:12.446 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48357.log 2024-01-15T18:25:12.447 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48383.log 2024-01-15T18:25:12.447 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54728.log.gz 2024-01-15T18:25:12.448 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42106.log 2024-01-15T18:25:12.448 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48357.log.gz 2024-01-15T18:25:12.449 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51378.log 2024-01-15T18:25:12.449 INFO:teuthology.orchestra.run.smithi027.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.48383.log.gz 2024-01-15T18:25:12.450 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54310.log 2024-01-15T18:25:12.450 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42106.log.gz 2024-01-15T18:25:12.451 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52589.log 2024-01-15T18:25:12.451 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51378.log.gz 2024-01-15T18:25:12.452 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48488.log 2024-01-15T18:25:12.452 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54310.log.gz 2024-01-15T18:25:12.453 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44792.log 2024-01-15T18:25:12.453 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52589.log.gz 2024-01-15T18:25:12.454 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.48488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52643.log 2024-01-15T18:25:12.455 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34091.log 2024-01-15T18:25:12.455 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44792.log.gz 2024-01-15T18:25:12.455 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37195.log 2024-01-15T18:25:12.456 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52643.log.gz 2024-01-15T18:25:12.456 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47702.log 2024-01-15T18:25:12.457 INFO:teuthology.orchestra.run.smithi093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27399.log 2024-01-15T18:25:12.458 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34091.log.gz 2024-01-15T18:25:12.458 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37195.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37195.log.gz 2024-01-15T18:25:12.458 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37436.log 2024-01-15T18:25:12.460 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44525.log 2024-01-15T18:25:12.460 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47702.log.gz 2024-01-15T18:25:12.461 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52035.log 2024-01-15T18:25:12.461 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37436.log.gz 2024-01-15T18:25:12.462 INFO:teuthology.orchestra.run.smithi093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27303.log 2024-01-15T18:25:12.462 INFO:teuthology.orchestra.run.smithi093.stderr:/var/log/ceph/ceph-client.admin.27399.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-01-15T18:25:12.462 INFO:teuthology.orchestra.run.smithi093.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.27399.log.gz 2024-01-15T18:25:12.462 INFO:teuthology.orchestra.run.smithi093.stderr:/var/log/ceph/ceph-client.admin.27303.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-01-15T18:25:12.462 INFO:teuthology.orchestra.run.smithi093.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.27303.log.gz 2024-01-15T18:25:12.462 INFO:teuthology.orchestra.run.smithi093.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27231.log 2024-01-15T18:25:12.463 INFO:teuthology.orchestra.run.smithi093.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.34808.log 2024-01-15T18:25:12.465 INFO:teuthology.orchestra.run.smithi093.stderr:/var/log/ceph/ceph-client.admin.27231.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27231.log.gz 2024-01-15T18:25:12.465 INFO:teuthology.orchestra.run.smithi093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-01-15T18:25:12.465 INFO:teuthology.orchestra.run.smithi093.stderr:/var/log/ceph/ceph-client.1.34808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27375.log 2024-01-15T18:25:12.465 INFO:teuthology.orchestra.run.smithi093.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.33741.log 2024-01-15T18:25:12.467 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33311.log 2024-01-15T18:25:12.467 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44525.log.gz 2024-01-15T18:25:12.467 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52035.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52035.log.gz 2024-01-15T18:25:12.467 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44073.log 2024-01-15T18:25:12.468 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43943.log 2024-01-15T18:25:12.468 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33311.log.gz 2024-01-15T18:25:12.468 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44073.log: gzip -5 --verbose 88.7% -- -- replaced with /var/log/ceph/ceph-client.admin.44073.log.gz /var/log/ceph/ceph-mgr.y.log 2024-01-15T18:25:12.468 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-01-15T18:25:12.468 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46021.log 2024-01-15T18:25:12.468 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43943.log.gz 2024-01-15T18:25:12.468 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54676.log 2024-01-15T18:25:12.469 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51031.log 2024-01-15T18:25:12.469 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46021.log.gz 2024-01-15T18:25:12.473 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54676.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54676.log.gz 2024-01-15T18:25:12.473 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36287.log 2024-01-15T18:25:12.473 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40800.log 2024-01-15T18:25:12.473 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51031.log.gz 2024-01-15T18:25:12.473 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36287.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36287.log.gz 2024-01-15T18:25:12.473 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38072.log 2024-01-15T18:25:12.473 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40112.log 2024-01-15T18:25:12.474 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40800.log.gz 2024-01-15T18:25:12.475 INFO:teuthology.orchestra.run.smithi093.stderr:/var/log/ceph/ceph-client.admin.27375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.33018.log 2024-01-15T18:25:12.475 INFO:teuthology.orchestra.run.smithi093.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.27375.log.gz 2024-01-15T18:25:12.475 INFO:teuthology.orchestra.run.smithi093.stderr:/var/log/ceph/ceph-client.1.33741.log: 96.3% -- replaced with /var/log/ceph/ceph-client.1.34808.log.gz 2024-01-15T18:25:12.475 INFO:teuthology.orchestra.run.smithi093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27351.log 2024-01-15T18:25:12.475 INFO:teuthology.orchestra.run.smithi093.stderr:/var/log/ceph/ceph-client.1.33018.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-01-15T18:25:12.475 INFO:teuthology.orchestra.run.smithi093.stderr:/var/log/ceph/ceph-client.admin.27351.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.27351.log.gz 2024-01-15T18:25:12.475 INFO:teuthology.orchestra.run.smithi093.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.1.33018.log.gz 2024-01-15T18:25:12.475 INFO:teuthology.orchestra.run.smithi093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-15T18:25:12.476 INFO:teuthology.orchestra.run.smithi093.stderr: 95.3% -- replaced with /var/log/ceph/ceph-client.1.33741.log.gz 2024-01-15T18:25:12.476 INFO:teuthology.orchestra.run.smithi093.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.33287.log 2024-01-15T18:25:12.476 INFO:teuthology.orchestra.run.smithi093.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-01-15T18:25:12.476 INFO:teuthology.orchestra.run.smithi093.stderr:/var/log/ceph/ceph-client.1.33287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.33503.log 2024-01-15T18:25:12.478 INFO:teuthology.orchestra.run.smithi093.stderr:/var/log/ceph/ceph-mon.c.log: 92.8% -- replaced with /var/log/ceph/ceph-client.1.33287.log.gz 2024-01-15T18:25:12.478 INFO:teuthology.orchestra.run.smithi093.stderr:/var/log/ceph/ceph-client.1.33503.log: 92.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-15T18:25:12.478 INFO:teuthology.orchestra.run.smithi093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.34346.log 2024-01-15T18:25:12.478 INFO:teuthology.orchestra.run.smithi093.stderr: 94.1% -- replaced with /var/log/ceph/ceph-client.1.33503.log.gz 2024-01-15T18:25:12.480 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38072.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38072.log.gz 2024-01-15T18:25:12.480 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37886.log 2024-01-15T18:25:12.480 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38740.log 2024-01-15T18:25:12.480 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40112.log.gz 2024-01-15T18:25:12.480 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37886.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37886.log.gz 2024-01-15T18:25:12.481 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.36928.log 2024-01-15T18:25:12.481 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37941.log 2024-01-15T18:25:12.481 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38740.log.gz 2024-01-15T18:25:12.481 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.36928.log: 94.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-01-15T18:25:12.481 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46339.log 2024-01-15T18:25:12.481 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53719.log 2024-01-15T18:25:12.481 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37941.log.gz 2024-01-15T18:25:12.484 INFO:teuthology.orchestra.run.smithi093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.34581.log 2024-01-15T18:25:12.484 INFO:teuthology.orchestra.run.smithi093.stderr:/var/log/ceph/ceph-client.1.34346.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-15T18:25:12.484 INFO:teuthology.orchestra.run.smithi093.stderr:/var/log/ceph/ceph-client.1.34581.log: 93.5% -- replaced with /var/log/ceph/ceph-client.1.34346.log.gz 2024-01-15T18:25:12.484 INFO:teuthology.orchestra.run.smithi093.stderr:/var/log/ceph/ceph.audit.log: 92.8% -- replaced with /var/log/ceph/ceph-client.1.34581.log.gz 2024-01-15T18:25:12.484 INFO:teuthology.orchestra.run.smithi093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27279.log 2024-01-15T18:25:12.485 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46339.log: gzip -5 88.3% --verbose -- replaced with /var/log/ceph/ceph-client.admin.46339.log.gz -- 2024-01-15T18:25:12.485 INFO:teuthology.orchestra.run.smithi027.stderr: /var/log/ceph/ceph-client.admin.33441.log 2024-01-15T18:25:12.485 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42382.log 2024-01-15T18:25:12.485 INFO:teuthology.orchestra.run.smithi027.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.53719.log.gz 2024-01-15T18:25:12.485 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33441.log: 89.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.33441.log.gz 2024-01-15T18:25:12.485 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34646.log 2024-01-15T18:25:12.486 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45471.log 2024-01-15T18:25:12.486 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42382.log.gz 2024-01-15T18:25:12.486 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34646.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34646.log.gz 2024-01-15T18:25:12.486 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35778.log 2024-01-15T18:25:12.487 INFO:teuthology.orchestra.run.smithi093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-01-15T18:25:12.488 INFO:teuthology.orchestra.run.smithi093.stderr:/var/log/ceph/ceph-client.admin.27279.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.27279.log.gz 2024-01-15T18:25:12.489 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49341.log 2024-01-15T18:25:12.489 INFO:teuthology.orchestra.run.smithi027.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45471.log.gz 2024-01-15T18:25:12.489 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35778.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35778.log.gz 2024-01-15T18:25:12.489 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54472.log 2024-01-15T18:25:12.489 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44383.log 2024-01-15T18:25:12.489 INFO:teuthology.orchestra.run.smithi027.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.49341.log.gz 2024-01-15T18:25:12.491 INFO:teuthology.orchestra.run.smithi093.stderr:/var/log/ceph/ceph-mds.d.log: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-15T18:25:12.491 INFO:teuthology.orchestra.run.smithi093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.34052.log 2024-01-15T18:25:12.491 INFO:teuthology.orchestra.run.smithi093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27255.log 2024-01-15T18:25:12.492 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41736.log 2024-01-15T18:25:12.492 INFO:teuthology.orchestra.run.smithi027.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.54472.log.gz 2024-01-15T18:25:12.492 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38098.log 2024-01-15T18:25:12.492 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44383.log.gz 2024-01-15T18:25:12.492 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41736.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41736.log.gz 2024-01-15T18:25:12.493 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49429.log 2024-01-15T18:25:12.493 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54049.log 2024-01-15T18:25:12.494 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38098.log.gz 2024-01-15T18:25:12.494 INFO:teuthology.orchestra.run.smithi093.stderr:/var/log/ceph/ceph-client.1.34052.log: /var/log/ceph/ceph-client.admin.27255.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.27255.log.gz 2024-01-15T18:25:12.494 INFO:teuthology.orchestra.run.smithi093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27327.log 2024-01-15T18:25:12.495 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49429.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.49429.log.gz 2024-01-15T18:25:12.495 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54858.log 2024-01-15T18:25:12.495 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50977.log 2024-01-15T18:25:12.496 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54049.log.gz 2024-01-15T18:25:12.497 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32042.log 87.9% 2024-01-15T18:25:12.497 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54858.log.gz 2024-01-15T18:25:12.497 INFO:teuthology.orchestra.run.smithi093.stderr: 95.8% -- replaced with /var/log/ceph/ceph-client.1.34052.log.gz 2024-01-15T18:25:12.497 INFO:teuthology.orchestra.run.smithi093.stderr:/var/log/ceph/ceph-client.admin.27327.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.27327.log.gz 2024-01-15T18:25:12.498 INFO:teuthology.orchestra.run.smithi093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-01-15T18:25:12.498 INFO:teuthology.orchestra.run.smithi093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-01-15T18:25:12.500 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49512.log 2024-01-15T18:25:12.500 INFO:teuthology.orchestra.run.smithi027.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50977.log.gz 2024-01-15T18:25:12.500 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32042.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.32042.log.gz 2024-01-15T18:25:12.500 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42716.log 2024-01-15T18:25:12.500 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53694.log 2024-01-15T18:25:12.500 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49512.log.gz 2024-01-15T18:25:12.503 INFO:teuthology.orchestra.run.smithi093.stderr:/var/log/ceph/ceph-mds.b.log: /var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-01-15T18:25:12.505 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42716.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42716.log.gzgzip 2024-01-15T18:25:12.505 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46313.log 2024-01-15T18:25:12.505 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.38897.log 2024-01-15T18:25:12.505 INFO:teuthology.orchestra.run.smithi027.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.53694.log.gz 2024-01-15T18:25:12.505 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46313.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.46313.log.gz 2024-01-15T18:25:12.506 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39642.log 2024-01-15T18:25:12.506 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.38897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44331.log 2024-01-15T18:25:12.506 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39642.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39642.log.gz 2024-01-15T18:25:12.507 INFO:teuthology.orchestra.run.smithi093.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.33130.log 2024-01-15T18:25:12.508 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50075.log 2024-01-15T18:25:12.508 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35830.log 2024-01-15T18:25:12.508 INFO:teuthology.orchestra.run.smithi027.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44331.log.gz 2024-01-15T18:25:12.508 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50075.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50075.log.gz 2024-01-15T18:25:12.508 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53417.log 2024-01-15T18:25:12.509 INFO:teuthology.orchestra.run.smithi093.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.3.33130.log: 90.7% -- replaced with /var/log/ceph/ceph-client.3.33130.log.gz 2024-01-15T18:25:12.509 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35830.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35830.log.gz 2024-01-15T18:25:12.510 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54446.log 2024-01-15T18:25:12.511 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53417.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-01-15T18:25:12.512 INFO:teuthology.orchestra.run.smithi027.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.53417.log.gz 2024-01-15T18:25:12.512 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54446.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54446.log.gz 2024-01-15T18:25:12.512 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41279.log 2024-01-15T18:25:12.513 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47119.log 2024-01-15T18:25:12.514 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34223.log 2024-01-15T18:25:12.514 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41279.log.gz 2024-01-15T18:25:12.515 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53366.log 2024-01-15T18:25:12.516 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47119.log.gz 2024-01-15T18:25:12.517 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51718.log 2024-01-15T18:25:12.517 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34223.log.gz 2024-01-15T18:25:12.518 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43749.log 2024-01-15T18:25:12.519 INFO:teuthology.orchestra.run.smithi027.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.53366.log.gz 2024-01-15T18:25:12.520 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39850.log 2024-01-15T18:25:12.520 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51718.log.gz 2024-01-15T18:25:12.521 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48642.log 2024-01-15T18:25:12.521 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43749.log.gz 2024-01-15T18:25:12.522 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47767.log 2024-01-15T18:25:12.522 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39850.log.gz 2024-01-15T18:25:12.524 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39616.log 2024-01-15T18:25:12.524 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48642.log.gz 2024-01-15T18:25:12.525 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44579.log 2024-01-15T18:25:12.525 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47767.log.gz 2024-01-15T18:25:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45677.log 2024-01-15T18:25:12.526 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39616.log.gz 2024-01-15T18:25:12.528 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34249.log 2024-01-15T18:25:12.528 INFO:teuthology.orchestra.run.smithi027.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44579.log.gz 2024-01-15T18:25:12.529 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45677.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.45677.log.gz -5 2024-01-15T18:25:12.530 INFO:teuthology.orchestra.run.smithi027.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39564.log 2024-01-15T18:25:12.531 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42051.log 2024-01-15T18:25:12.531 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34249.log.gz 2024-01-15T18:25:12.532 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39564.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39564.log.gz 2024-01-15T18:25:12.532 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40407.log 2024-01-15T18:25:12.534 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42051.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42051.log.gz 2024-01-15T18:25:12.534 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33981.log 2024-01-15T18:25:12.535 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35622.log 2024-01-15T18:25:12.536 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40407.log.gz 2024-01-15T18:25:12.536 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52810.log 2024-01-15T18:25:12.537 INFO:teuthology.orchestra.run.smithi027.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.33981.log.gz 2024-01-15T18:25:12.538 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50816.log 2024-01-15T18:25:12.538 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35622.log.gz 2024-01-15T18:25:12.539 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55198.log 2024-01-15T18:25:12.539 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52810.log.gz 2024-01-15T18:25:12.540 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46182.log 2024-01-15T18:25:12.540 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50816.log.gz 2024-01-15T18:25:12.542 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43517.log 2024-01-15T18:25:12.542 INFO:teuthology.orchestra.run.smithi027.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55198.log.gz 2024-01-15T18:25:12.543 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54550.log 2024-01-15T18:25:12.543 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46182.log.gz 2024-01-15T18:25:12.545 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49619.log 2024-01-15T18:25:12.545 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43517.log.gz 2024-01-15T18:25:12.545 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45995.log 2024-01-15T18:25:12.546 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54550.log.gz 2024-01-15T18:25:12.547 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51482.log 2024-01-15T18:25:12.547 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49619.log.gz 2024-01-15T18:25:12.548 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36037.log 2024-01-15T18:25:12.549 INFO:teuthology.orchestra.run.smithi027.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45995.log.gz 2024-01-15T18:25:12.550 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33018.log 2024-01-15T18:25:12.550 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51482.log.gz 2024-01-15T18:25:12.551 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52617.log 2024-01-15T18:25:12.551 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36037.log.gz 2024-01-15T18:25:12.552 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33018.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.22129.log 2024-01-15T18:25:12.553 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.33018.log.gz 2024-01-15T18:25:12.554 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39458.log 2024-01-15T18:25:12.554 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52617.log.gz 2024-01-15T18:25:12.555 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph.tmp-client.admin.22129.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.22129.log.gz 2024-01-15T18:25:12.555 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32811.log 2024-01-15T18:25:12.556 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40163.log 2024-01-15T18:25:12.557 INFO:teuthology.orchestra.run.smithi027.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39458.log.gz 2024-01-15T18:25:12.558 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34964.log 2024-01-15T18:25:12.558 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32811.log.gz 2024-01-15T18:25:12.559 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44551.log 2024-01-15T18:25:12.559 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40163.log.gz 2024-01-15T18:25:12.560 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50622.log 2024-01-15T18:25:12.561 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34964.log.gz 2024-01-15T18:25:12.562 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46286.log 2024-01-15T18:25:12.562 INFO:teuthology.orchestra.run.smithi027.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44551.log.gz 2024-01-15T18:25:12.563 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40217.log 2024-01-15T18:25:12.563 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50622.log.gz 2024-01-15T18:25:12.564 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36535.log 2024-01-15T18:25:12.564 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46286.log.gz 2024-01-15T18:25:12.566 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45783.log 2024-01-15T18:25:12.566 INFO:teuthology.orchestra.run.smithi027.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40217.log.gz 2024-01-15T18:25:12.567 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45601.log 2024-01-15T18:25:12.567 INFO:teuthology.orchestra.run.smithi027.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36535.log.gz 2024-01-15T18:25:12.568 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44631.log 2024-01-15T18:25:12.568 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45783.log.gz 2024-01-15T18:25:12.569 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32809.log 2024-01-15T18:25:12.570 INFO:teuthology.orchestra.run.smithi027.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.45601.log.gz 2024-01-15T18:25:12.571 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51560.log 2024-01-15T18:25:12.571 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44631.log.gz 2024-01-15T18:25:12.572 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32200.log 2024-01-15T18:25:12.572 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32809.log.gz 2024-01-15T18:25:12.573 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40433.log 2024-01-15T18:25:12.573 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51560.log.gz 2024-01-15T18:25:12.575 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44922.log 2024-01-15T18:25:12.575 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32200.log.gz 2024-01-15T18:25:12.576 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38503.log 2024-01-15T18:25:12.576 INFO:teuthology.orchestra.run.smithi027.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40433.log.gz 2024-01-15T18:25:12.577 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41460.log 2024-01-15T18:25:12.578 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44922.log.gz 2024-01-15T18:25:12.579 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33002.log 2024-01-15T18:25:12.579 INFO:teuthology.orchestra.run.smithi027.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38503.log.gz 2024-01-15T18:25:12.580 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36822.log 2024-01-15T18:25:12.581 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41460.log.gz 2024-01-15T18:25:12.582 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38792.log 2024-01-15T18:25:12.582 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33002.log.gz 2024-01-15T18:25:12.583 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39249.log 2024-01-15T18:25:12.583 INFO:teuthology.orchestra.run.smithi027.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.36822.log.gz 2024-01-15T18:25:12.584 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46104.log 2024-01-15T18:25:12.585 INFO:teuthology.orchestra.run.smithi027.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.38792.log.gz 2024-01-15T18:25:12.585 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45367.log 2024-01-15T18:25:12.585 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39249.log.gz 2024-01-15T18:25:12.587 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43406.log 2024-01-15T18:25:12.587 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46104.log.gz 2024-01-15T18:25:12.588 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52318.log 2024-01-15T18:25:12.588 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45367.log.gz 2024-01-15T18:25:12.589 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43328.log 2024-01-15T18:25:12.589 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43406.log.gz 2024-01-15T18:25:12.591 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53341.log 2024-01-15T18:25:12.591 INFO:teuthology.orchestra.run.smithi027.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.52318.log.gz 2024-01-15T18:25:12.592 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42453.log 2024-01-15T18:25:12.592 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43328.log.gz 2024-01-15T18:25:12.593 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44657.log 2024-01-15T18:25:12.593 INFO:teuthology.orchestra.run.smithi027.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.53341.log.gz 2024-01-15T18:25:12.595 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35253.log 2024-01-15T18:25:12.595 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42453.log.gz 2024-01-15T18:25:12.596 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44605.log 2024-01-15T18:25:12.596 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44657.log.gz 2024-01-15T18:25:12.597 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40888.log 2024-01-15T18:25:12.597 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35253.log.gz 2024-01-15T18:25:12.598 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52722.log 2024-01-15T18:25:12.599 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44605.log.gz 2024-01-15T18:25:12.600 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42690.log 2024-01-15T18:25:12.600 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40888.log.gz 2024-01-15T18:25:12.601 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47093.log 2024-01-15T18:25:12.601 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52722.log.gz 2024-01-15T18:25:12.602 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33653.log 2024-01-15T18:25:12.602 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42690.log.gz 2024-01-15T18:25:12.604 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34561.log 2024-01-15T18:25:12.604 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47093.log.gz 2024-01-15T18:25:12.605 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49285.log 2024-01-15T18:25:12.605 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33653.log.gz 2024-01-15T18:25:12.606 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46859.log 2024-01-15T18:25:12.606 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34561.log.gz 2024-01-15T18:25:12.608 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54127.log 2024-01-15T18:25:12.608 INFO:teuthology.orchestra.run.smithi027.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49285.log.gz 2024-01-15T18:25:12.608 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35700.log 2024-01-15T18:25:12.609 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46859.log.gz 2024-01-15T18:25:12.610 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36872.log 2024-01-15T18:25:12.610 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54127.log.gz 2024-01-15T18:25:12.611 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53973.log 2024-01-15T18:25:12.612 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35700.log.gz 2024-01-15T18:25:12.612 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37700.log 2024-01-15T18:25:12.613 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36872.log.gz 2024-01-15T18:25:12.614 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41920.log 2024-01-15T18:25:12.614 INFO:teuthology.orchestra.run.smithi027.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.53973.log.gz 2024-01-15T18:25:12.615 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52500.log 2024-01-15T18:25:12.615 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37700.log.gz 2024-01-15T18:25:12.617 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37028.log 2024-01-15T18:25:12.617 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41920.log.gz 2024-01-15T18:25:12.618 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32605.log 2024-01-15T18:25:12.618 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52500.log.gz 2024-01-15T18:25:12.619 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34431.log 2024-01-15T18:25:12.619 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37028.log.gz 2024-01-15T18:25:12.621 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42770.log 2024-01-15T18:25:12.621 INFO:teuthology.orchestra.run.smithi027.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.32605.log.gz 2024-01-15T18:25:12.622 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34431.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-01-15T18:25:12.622 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34431.log.gz 2024-01-15T18:25:12.623 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35226.log 2024-01-15T18:25:12.623 INFO:teuthology.orchestra.run.smithi027.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42770.log.gz 2024-01-15T18:25:12.625 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39023.log 2024-01-15T18:25:12.626 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42531.log 2024-01-15T18:25:12.626 INFO:teuthology.orchestra.run.smithi027.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35226.log.gz 2024-01-15T18:25:12.627 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35148.log 2024-01-15T18:25:12.628 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39023.log.gz 2024-01-15T18:25:12.629 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43666.log 2024-01-15T18:25:12.629 INFO:teuthology.orchestra.run.smithi027.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42531.log.gz 2024-01-15T18:25:12.630 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35148.log: gzip -5 89.1% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.35148.log.gz /var/log/ceph/ceph-client.admin.33027.log 2024-01-15T18:25:12.630 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-01-15T18:25:12.632 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50285.log 2024-01-15T18:25:12.632 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43666.log.gz 2024-01-15T18:25:12.633 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.50522.log 2024-01-15T18:25:12.634 INFO:teuthology.orchestra.run.smithi027.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33027.log.gz 2024-01-15T18:25:12.634 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50285.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50285.log.gz 2024-01-15T18:25:12.635 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35016.log 2024-01-15T18:25:12.636 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.50522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54832.log 2024-01-15T18:25:12.637 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35016.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35016.log.gz 2024-01-15T18:25:12.638 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54832.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54832.log.gz 2024-01-15T18:25:12.639 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41842.log 2024-01-15T18:25:12.640 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49181.log 2024-01-15T18:25:12.642 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41842.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41842.log.gz 2024-01-15T18:25:12.643 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49181.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49181.log.gz 2024-01-15T18:25:12.643 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31889.log 2024-01-15T18:25:12.645 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55040.log 2024-01-15T18:25:12.646 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.31889.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31889.log.gz 2024-01-15T18:25:12.647 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55040.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55040.log.gz 2024-01-15T18:25:12.648 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34143.log 2024-01-15T18:25:12.650 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53241.log 2024-01-15T18:25:12.651 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34143.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34143.log.gz 2024-01-15T18:25:12.652 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53241.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53241.log.gz 2024-01-15T18:25:12.653 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47249.log 2024-01-15T18:25:12.654 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37464.log 2024-01-15T18:25:12.656 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47249.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47249.log.gz 2024-01-15T18:25:12.657 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47650.log 2024-01-15T18:25:12.657 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37464.log.gz 2024-01-15T18:25:12.659 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37594.log 2024-01-15T18:25:12.660 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47650.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47650.log.gz 2024-01-15T18:25:12.661 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37594.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37594.log.gz 2024-01-15T18:25:12.662 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54780.log 2024-01-15T18:25:12.663 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41710.log 2024-01-15T18:25:12.665 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54780.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54780.log.gz 2024-01-15T18:25:12.665 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35726.log 2024-01-15T18:25:12.666 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41710.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41710.log.gz 2024-01-15T18:25:12.669 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35726.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35726.log.gz 2024-01-15T18:25:12.669 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51619.log 2024-01-15T18:25:12.671 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40381.log 2024-01-15T18:25:12.672 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51619.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51619.log.gz 2024-01-15T18:25:12.673 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35388.log 2024-01-15T18:25:12.675 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40381.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.40381.log.gz 2024-01-15T18:25:12.675 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34509.log 2024-01-15T18:25:12.676 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35388.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35388.log.gz 2024-01-15T18:25:12.677 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34509.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34509.log.gz 2024-01-15T18:25:12.678 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32479.log 2024-01-15T18:25:12.680 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42427.log 2024-01-15T18:25:12.681 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32479.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32479.log.gz 2024-01-15T18:25:12.682 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42427.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42427.log.gz 2024-01-15T18:25:12.683 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40602.log 2024-01-15T18:25:12.684 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54524.log 2024-01-15T18:25:12.685 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40602.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40602.log.gz 2024-01-15T18:25:12.686 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54524.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54524.log.gz 2024-01-15T18:25:12.687 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46729.log 2024-01-15T18:25:12.689 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40995.log 2024-01-15T18:25:12.690 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46729.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46729.log.gz 2024-01-15T18:25:12.691 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40995.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40995.log.gz 2024-01-15T18:25:12.692 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-01-15T18:25:12.693 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54336.log 2024-01-15T18:25:12.696 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.54336.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54336.log.gz 2024-01-15T18:25:12.696 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44710.log 2024-01-15T18:25:12.703 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44710.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44710.log.gz 2024-01-15T18:25:12.703 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54101.log 2024-01-15T18:25:12.703 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54101.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54101.log.gz 2024-01-15T18:25:12.703 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45916.log 2024-01-15T18:25:12.706 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45916.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45916.log.gz 2024-01-15T18:25:12.706 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53266.log 2024-01-15T18:25:12.709 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38163.log 2024-01-15T18:25:12.710 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34672.log 2024-01-15T18:25:12.713 INFO:teuthology.orchestra.run.smithi027.stderr: 86.1%/var/log/ceph/ceph-client.admin.38163.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38163.log.gz 2024-01-15T18:25:12.713 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53266.log.gz 2024-01-15T18:25:12.716 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34672.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34672.log.gz 2024-01-15T18:25:12.716 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46391.log 2024-01-15T18:25:12.719 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46391.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46391.log.gz 2024-01-15T18:25:12.719 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54284.log 2024-01-15T18:25:12.721 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39167.log 2024-01-15T18:25:12.722 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54284.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54284.log.gz 2024-01-15T18:25:12.725 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39167.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39167.log.gz 2024-01-15T18:25:12.726 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45235.log 2024-01-15T18:25:12.729 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45235.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45235.log.gz 2024-01-15T18:25:12.729 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33026.log 2024-01-15T18:25:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33026.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33026.log.gz 2024-01-15T18:25:12.732 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43276.log 2024-01-15T18:25:12.735 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37410.log 2024-01-15T18:25:12.735 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43276.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43276.log.gz 2024-01-15T18:25:12.738 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37410.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37410.log.gz 2024-01-15T18:25:12.738 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35174.log 2024-01-15T18:25:12.744 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35174.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35174.log.gz 2024-01-15T18:25:12.745 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34938.log 2024-01-15T18:25:12.753 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34938.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34938.log.gz 2024-01-15T18:25:12.753 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35357.log 2024-01-15T18:25:12.759 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41684.log 2024-01-15T18:25:12.760 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35357.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35357.log.gz 2024-01-15T18:25:12.761 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50363.log 2024-01-15T18:25:12.764 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46208.log 2024-01-15T18:25:12.765 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50363.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50363.log.gz 2024-01-15T18:25:12.766 INFO:teuthology.orchestra.run.smithi027.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41684.log.gz 2024-01-15T18:25:12.768 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43695.log 2024-01-15T18:25:12.769 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46208.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46208.log.gz 2024-01-15T18:25:12.771 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48756.log 2024-01-15T18:25:12.775 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43695.log: /var/log/ceph/ceph-client.admin.48756.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48756.log.gz 2024-01-15T18:25:12.775 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43695.log.gz 2024-01-15T18:25:12.775 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44948.log 2024-01-15T18:25:12.779 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44948.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44948.log.gz 2024-01-15T18:25:12.779 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55225.log 2024-01-15T18:25:12.785 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55225.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55225.log.gz 2024-01-15T18:25:12.785 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26980.log 2024-01-15T18:25:12.785 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.26980.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.26980.log.gz 2024-01-15T18:25:12.785 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39193.log 2024-01-15T18:25:12.788 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39193.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39193.log.gz 2024-01-15T18:25:12.788 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46049.log 2024-01-15T18:25:12.792 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43969.log 2024-01-15T18:25:12.792 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46049.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46049.log.gz 2024-01-15T18:25:12.795 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43969.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43969.log.gz 2024-01-15T18:25:12.795 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53089.log 2024-01-15T18:25:12.797 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41073.log 2024-01-15T18:25:12.801 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41073.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41073.log.gz 2024-01-15T18:25:12.801 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53089.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.53089.log.gz 2024-01-15T18:25:12.801 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33003.log 2024-01-15T18:25:12.804 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33003.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33003.log.gz 2024-01-15T18:25:12.804 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41177.log 2024-01-15T18:25:12.811 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41177.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41177.log.gz 2024-01-15T18:25:12.811 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54179.log 2024-01-15T18:25:12.811 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54179.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54179.log.gz 2024-01-15T18:25:12.811 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41658.log 2024-01-15T18:25:12.813 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41125.log 2024-01-15T18:25:12.814 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41658.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41658.log.gz 2024-01-15T18:25:12.816 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48920.log 2024-01-15T18:25:12.820 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48920.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48920.log.gz 2024-01-15T18:25:12.821 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41125.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41125.log.gz 2024-01-15T18:25:12.821 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49076.log 2024-01-15T18:25:12.824 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49076.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49076.log.gz 2024-01-15T18:25:12.824 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55092.log 2024-01-15T18:25:12.827 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55092.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55092.log.gz 2024-01-15T18:25:12.827 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46599.log 2024-01-15T18:25:12.830 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46599.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.46599.log.gz 2024-01-15T18:25:12.830 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38660.log 2024-01-15T18:25:12.834 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41550.log 2024-01-15T18:25:12.834 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38660.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38660.log.gz 2024-01-15T18:25:12.837 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41550.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41550.log.gz 2024-01-15T18:25:12.837 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45131.log 2024-01-15T18:25:12.840 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45131.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45131.log.gz 2024-01-15T18:25:12.840 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50700.log 2024-01-15T18:25:12.843 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50700.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50700.log.gz 2024-01-15T18:25:12.843 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45001.log 2024-01-15T18:25:12.846 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35414.log 2024-01-15T18:25:12.846 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45001.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45001.log.gz 2024-01-15T18:25:12.849 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35414.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35414.log.gz 2024-01-15T18:25:12.850 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49486.log 2024-01-15T18:25:12.853 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49486.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49486.log.gz 2024-01-15T18:25:12.853 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51879.log 2024-01-15T18:25:12.856 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51879.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51879.log.gz 2024-01-15T18:25:12.856 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39301.log 2024-01-15T18:25:12.859 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40485.log 2024-01-15T18:25:12.859 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39301.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39301.log.gz 2024-01-15T18:25:12.862 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41602.log 2024-01-15T18:25:12.863 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40485.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40485.log.gz 2024-01-15T18:25:12.866 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41602.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41602.log.gz 2024-01-15T18:25:12.866 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35122.log 2024-01-15T18:25:12.872 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35122.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35122.log.gz 2024-01-15T18:25:12.872 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33019.log 2024-01-15T18:25:12.873 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33019.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33019.log.gz 2024-01-15T18:25:12.873 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27095.log 2024-01-15T18:25:12.874 INFO:teuthology.orchestra.run.smithi093.stderr: 93.3% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-01-15T18:25:12.875 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.27095.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.27095.log.gz 2024-01-15T18:25:12.876 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49050.log 2024-01-15T18:25:12.879 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43164.log 2024-01-15T18:25:12.879 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49050.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49050.log.gz 2024-01-15T18:25:12.882 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43164.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43164.log.gz 2024-01-15T18:25:12.882 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39590.log 2024-01-15T18:25:12.885 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39590.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39590.log.gz 2024-01-15T18:25:12.885 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51214.log 2024-01-15T18:25:12.888 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51214.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51214.log.gz 2024-01-15T18:25:12.888 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45575.log 2024-01-15T18:25:12.891 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38997.log 2024-01-15T18:25:12.892 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45575.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45575.log.gz 2024-01-15T18:25:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38997.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38997.log.gz 2024-01-15T18:25:12.895 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39772.log 2024-01-15T18:25:12.898 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39772.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39772.log.gz 2024-01-15T18:25:12.898 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37542.log 2024-01-15T18:25:12.901 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37803.log 2024-01-15T18:25:12.901 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37542.log.gz 2024-01-15T18:25:12.904 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43777.log 2024-01-15T18:25:12.904 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37803.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37803.log.gz 2024-01-15T18:25:12.907 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40274.log 2024-01-15T18:25:12.907 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43777.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43777.log.gz 2024-01-15T18:25:12.909 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40274.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40274.log.gz 2024-01-15T18:25:12.910 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33014.log 2024-01-15T18:25:12.913 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33014.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33014.log.gz 2024-01-15T18:25:12.913 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34698.log 2024-01-15T18:25:12.915 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37254.log 2024-01-15T18:25:12.920 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37254.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37254.log.gz 2024-01-15T18:25:12.920 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34698.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34698.log.gz 2024-01-15T18:25:12.920 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35596.log 2024-01-15T18:25:12.923 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35596.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35596.log.gz 2024-01-15T18:25:12.923 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42978.log 2024-01-15T18:25:12.926 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42978.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42978.log.gz 2024-01-15T18:25:12.926 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47067.log 2024-01-15T18:25:12.929 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53544.log 2024-01-15T18:25:12.930 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47067.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47067.log.gz 2024-01-15T18:25:12.932 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33389.log 2024-01-15T18:25:12.933 INFO:teuthology.orchestra.run.smithi027.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53544.log.gz 2024-01-15T18:25:12.936 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37777.log 2024-01-15T18:25:12.937 INFO:teuthology.orchestra.run.smithi027.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33389.log.gz 2024-01-15T18:25:12.940 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37777.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37777.log.gz 2024-01-15T18:25:12.940 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38449.log 2024-01-15T18:25:12.943 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38449.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38449.log.gz 2024-01-15T18:25:12.943 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44818.log 2024-01-15T18:25:12.946 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44818.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44818.log.gz 2024-01-15T18:25:12.946 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42025.log 2024-01-15T18:25:12.949 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42297.log 2024-01-15T18:25:12.949 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42025.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42025.log.gz 2024-01-15T18:25:12.952 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42297.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42297.log.gz 2024-01-15T18:25:12.953 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46625.log 2024-01-15T18:25:12.956 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46625.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46625.log.gz 2024-01-15T18:25:12.956 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40537.log 2024-01-15T18:25:12.959 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40537.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40537.log.gz 2024-01-15T18:25:12.959 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32122.log 2024-01-15T18:25:12.962 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42664.log 2024-01-15T18:25:12.962 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32122.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32122.log.gz 2024-01-15T18:25:12.965 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42664.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42664.log.gz 2024-01-15T18:25:12.965 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45105.log 2024-01-15T18:25:12.968 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45105.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45105.log.gz 2024-01-15T18:25:12.969 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55014.log 2024-01-15T18:25:12.972 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55014.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55014.log.gz 2024-01-15T18:25:12.972 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46547.log 2024-01-15T18:25:12.975 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52448.log 2024-01-15T18:25:12.975 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46547.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46547.log.gz 2024-01-15T18:25:12.978 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52448.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52448.log.gz 2024-01-15T18:25:12.978 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38228.log 2024-01-15T18:25:12.981 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38228.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38228.log.gz 2024-01-15T18:25:12.981 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45703.log 2024-01-15T18:25:12.984 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45703.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45703.log.gz 2024-01-15T18:25:12.984 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41408.log 2024-01-15T18:25:12.991 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41408.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41408.log.gz 2024-01-15T18:25:12.991 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39746.log 2024-01-15T18:25:12.991 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39746.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39746.log.gz 2024-01-15T18:25:12.991 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32799.log 2024-01-15T18:25:12.994 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32799.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32799.log.gz 2024-01-15T18:25:12.994 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53164.log 2024-01-15T18:25:12.997 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53164.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.53164.log.gz 2024-01-15T18:25:12.997 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37169.log 2024-01-15T18:25:13.000 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49377.log 2024-01-15T18:25:13.001 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37169.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37169.log.gz 2024-01-15T18:25:13.004 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49377.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49377.log.gz 2024-01-15T18:25:13.004 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45183.log 2024-01-15T18:25:13.007 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45183.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45183.log.gz 2024-01-15T18:25:13.007 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49024.log 2024-01-15T18:25:13.008 INFO:teuthology.orchestra.run.smithi093.stderr: 93.3% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-01-15T18:25:13.010 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49024.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49024.log.gz 2024-01-15T18:25:13.011 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34912.log 2024-01-15T18:25:13.014 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42952.log 2024-01-15T18:25:13.014 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34912.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34912.log.gz 2024-01-15T18:25:13.017 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42952.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42952.log.gz 2024-01-15T18:25:13.017 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39954.log 2024-01-15T18:25:13.020 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39954.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39954.log.gz 2024-01-15T18:25:13.020 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34275.log 2024-01-15T18:25:13.023 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34275.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34275.log.gz 2024-01-15T18:25:13.024 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55066.log 2024-01-15T18:25:13.027 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44409.log 2024-01-15T18:25:13.027 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55066.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55066.log.gz 2024-01-15T18:25:13.030 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44409.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44409.log.gz 2024-01-15T18:25:13.030 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50596.log 2024-01-15T18:25:13.033 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50596.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50596.log.gz 2024-01-15T18:25:13.033 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34327.log 2024-01-15T18:25:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34327.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34327.log.gz 2024-01-15T18:25:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40355.log 2024-01-15T18:25:13.039 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40355.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40355.log.gz 2024-01-15T18:25:13.039 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51653.log 2024-01-15T18:25:13.042 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37672.log 2024-01-15T18:25:13.043 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51653.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51653.log.gz 2024-01-15T18:25:13.046 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37672.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37672.log.gz 2024-01-15T18:25:13.046 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38254.log 2024-01-15T18:25:13.049 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38254.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38254.log.gz 2024-01-15T18:25:13.049 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43112.log 2024-01-15T18:25:13.052 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43112.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43112.log.gz 2024-01-15T18:25:13.052 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45626.log 2024-01-15T18:25:13.055 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48045.log 2024-01-15T18:25:13.055 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45626.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.45626.log.gz 2024-01-15T18:25:13.058 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48045.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48045.log.gz 2024-01-15T18:25:13.059 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54988.log 2024-01-15T18:25:13.061 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47356.log 2024-01-15T18:25:13.062 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54988.log.gz 2024-01-15T18:25:13.063 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37080.log 2024-01-15T18:25:13.069 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37080.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37080.log.gz 2024-01-15T18:25:13.069 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47356.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47356.log.gz 2024-01-15T18:25:13.069 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35648.log 2024-01-15T18:25:13.072 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35648.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35648.log.gz 2024-01-15T18:25:13.072 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39824.log 2024-01-15T18:25:13.072 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2024-01-15T18:25:13.075 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39824.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39824.log.gz 2024-01-15T18:25:13.075 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48590.log 2024-01-15T18:25:13.078 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48590.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48590.log.gz 2024-01-15T18:25:13.078 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36207.log 2024-01-15T18:25:13.081 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33598.log 2024-01-15T18:25:13.082 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36207.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36207.log.gz 2024-01-15T18:25:13.083 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41434.log/var/log/ceph/ceph-client.admin.33598.log: 2024-01-15T18:25:13.086 INFO:teuthology.orchestra.run.smithi027.stderr: 91.1%/var/log/ceph/ceph-client.admin.41434.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41434.log.gz 2024-01-15T18:25:13.087 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37568.log 2024-01-15T18:25:13.087 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33598.log.gz 2024-01-15T18:25:13.090 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37568.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37568.log.gz 2024-01-15T18:25:13.090 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52396.log 2024-01-15T18:25:13.093 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43891.log 2024-01-15T18:25:13.093 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52396.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52396.log.gz 2024-01-15T18:25:13.096 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43891.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43891.log.gz 2024-01-15T18:25:13.096 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50950.log 2024-01-15T18:25:13.099 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50950.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50950.log.gz 2024-01-15T18:25:13.099 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54754.log 2024-01-15T18:25:13.102 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54754.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54754.log.gz 2024-01-15T18:25:13.102 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35856.log 2024-01-15T18:25:13.105 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53619.log 2024-01-15T18:25:13.105 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35856.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35856.log.gz 2024-01-15T18:25:13.108 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53619.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.53619.log.gz 2024-01-15T18:25:13.108 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46443.log 2024-01-15T18:25:13.111 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46443.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46443.log.gz 2024-01-15T18:25:13.111 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35544.log 2024-01-15T18:25:13.114 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35544.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35544.log.gz 2024-01-15T18:25:13.114 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-01-15T18:25:13.117 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42900.log 2024-01-15T18:25:13.120 INFO:teuthology.orchestra.run.smithi093.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-01-15T18:25:13.121 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.42900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41021.log 2024-01-15T18:25:13.126 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42900.log.gz 2024-01-15T18:25:13.126 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31945.log 2024-01-15T18:25:13.127 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42271.log 2024-01-15T18:25:13.127 INFO:teuthology.orchestra.run.smithi027.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41021.log.gz 2024-01-15T18:25:13.131 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42271.log: /var/log/ceph/ceph-client.admin.31945.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42271.log.gz 2024-01-15T18:25:13.131 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.31945.log.gz 2024-01-15T18:25:13.132 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47624.log 2024-01-15T18:25:13.139 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47624.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47624.log.gz 2024-01-15T18:25:13.140 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47408.log 2024-01-15T18:25:13.147 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47408.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47408.log.gz 2024-01-15T18:25:13.148 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53948.log 2024-01-15T18:25:13.155 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53948.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.53948.log.gz 2024-01-15T18:25:13.156 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35492.log 2024-01-15T18:25:13.158 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38556.log 2024-01-15T18:25:13.162 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32453.log 2024-01-15T18:25:13.162 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38556.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35492.log.gz 2024-01-15T18:25:13.163 INFO:teuthology.orchestra.run.smithi027.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38556.log.gz 2024-01-15T18:25:13.167 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32453.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32453.log.gz 2024-01-15T18:25:13.168 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48868.log 2024-01-15T18:25:13.175 INFO:teuthology.orchestra.run.smithi093.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-01-15T18:25:13.175 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48868.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48868.log.gz 2024-01-15T18:25:13.176 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52836.log 2024-01-15T18:25:13.182 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37054.log 2024-01-15T18:25:13.184 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52836.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52836.log.gz 2024-01-15T18:25:13.186 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39876.log 2024-01-15T18:25:13.189 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37054.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37054.log.gz 2024-01-15T18:25:13.193 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39876.log: 89.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38634.log 2024-01-15T18:25:13.197 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39876.log.gz 2024-01-15T18:25:13.198 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51931.log 2024-01-15T18:25:13.199 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38634.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38634.log.gz 2024-01-15T18:25:13.201 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53492.log 2024-01-15T18:25:13.203 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51931.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51931.log.gz 2024-01-15T18:25:13.204 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53492.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.53492.log.gz 2024-01-15T18:25:13.209 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36313.log 2024-01-15T18:25:13.214 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36744.log 2024-01-15T18:25:13.215 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36313.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36313.log.gz 2024-01-15T18:25:13.218 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40329.log 2024-01-15T18:25:13.219 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36744.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36744.log.gz 2024-01-15T18:25:13.220 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52964.log 2024-01-15T18:25:13.222 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40329.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40329.log.gz 2024-01-15T18:25:13.226 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44896.log 2024-01-15T18:25:13.227 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52964.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.52964.log.gz 2024-01-15T18:25:13.230 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48788.log 2024-01-15T18:25:13.231 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44896.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44896.log.gz 2024-01-15T18:25:13.233 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43917.log 2024-01-15T18:25:13.234 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48788.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48788.log.gz 2024-01-15T18:25:13.238 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41973.log 2024-01-15T18:25:13.239 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43917.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43917.log.gz 2024-01-15T18:25:13.242 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52784.log 2024-01-15T18:25:13.243 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41973.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41973.log.gz 2024-01-15T18:25:13.246 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52784.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52784.log.gz 2024-01-15T18:25:13.249 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40459.log 2024-01-15T18:25:13.254 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35570.log 2024-01-15T18:25:13.256 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40459.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40459.log.gz 2024-01-15T18:25:13.258 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41946.log 2024-01-15T18:25:13.261 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35570.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35570.log.gz 2024-01-15T18:25:13.262 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34379.log 2024-01-15T18:25:13.263 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41946.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.41946.log.gz 2024-01-15T18:25:13.265 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45888.log 2024-01-15T18:25:13.266 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34379.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34379.log.gz 2024-01-15T18:25:13.270 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40006.log 2024-01-15T18:25:13.271 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45888.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45888.log.gz 2024-01-15T18:25:13.272 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44125.log 2024-01-15T18:25:13.278 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47546.log 2024-01-15T18:25:13.278 INFO:teuthology.orchestra.run.smithi027.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40006.log.gz 2024-01-15T18:25:13.279 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44125.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44125.log.gz 2024-01-15T18:25:13.282 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46807.log 2024-01-15T18:25:13.283 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47546.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47546.log.gz 2024-01-15T18:25:13.286 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46807.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.46807.log.gz 2024-01-15T18:25:13.289 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40300.log 2024-01-15T18:25:13.294 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32148.log 2024-01-15T18:25:13.295 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40300.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40300.log.gz 2024-01-15T18:25:13.298 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41868.log 2024-01-15T18:25:13.299 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32148.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32148.log.gz 2024-01-15T18:25:13.302 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33955.log 2024-01-15T18:25:13.303 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41868.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41868.log.gz 2024-01-15T18:25:13.310 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51083.log 2024-01-15T18:25:13.311 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33955.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33955.log.gz 2024-01-15T18:25:13.314 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45209.log 2024-01-15T18:25:13.315 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51083.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51083.log.gz 2024-01-15T18:25:13.318 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45209.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45209.log.gz 2024-01-15T18:25:13.319 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36102.log 2024-01-15T18:25:13.323 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36102.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36102.log.gz 2024-01-15T18:25:13.325 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33337.log 2024-01-15T18:25:13.330 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53517.log 2024-01-15T18:25:13.331 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33337.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33337.log.gz 2024-01-15T18:25:13.334 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32096.log 2024-01-15T18:25:13.335 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53517.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53517.log.gz 2024-01-15T18:25:13.338 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45523.log 2024-01-15T18:25:13.339 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32096.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32096.log.gz 2024-01-15T18:25:13.341 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46260.log 2024-01-15T18:25:13.342 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45523.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45523.log.gz 2024-01-15T18:25:13.346 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32258.log 2024-01-15T18:25:13.347 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46260.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46260.log.gz 2024-01-15T18:25:13.350 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36339.log 2024-01-15T18:25:13.351 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32258.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32258.log.gz 2024-01-15T18:25:13.354 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36339.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36339.log.gz 2024-01-15T18:25:13.357 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50129.log 2024-01-15T18:25:13.362 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38686.log 2024-01-15T18:25:13.363 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50129.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50129.log.gz 2024-01-15T18:25:13.366 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48149.log 2024-01-15T18:25:13.367 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38686.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38686.log.gz 2024-01-15T18:25:13.369 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45651.log 2024-01-15T18:25:13.377 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48149.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48149.log.gz 2024-01-15T18:25:13.379 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45651.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45651.log.gz 2024-01-15T18:25:13.381 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44151.log 2024-01-15T18:25:13.386 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55170.log 2024-01-15T18:25:13.387 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44151.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44151.log.gz 2024-01-15T18:25:13.390 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49880.log 2024-01-15T18:25:13.391 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55170.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55170.log.gz 2024-01-15T18:25:13.394 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45393.log 2024-01-15T18:25:13.395 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49880.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49880.log.gz 2024-01-15T18:25:13.397 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38131.log 2024-01-15T18:25:13.398 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45393.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45393.log.gz 2024-01-15T18:25:13.403 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38131.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38131.log.gz 2024-01-15T18:25:13.405 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47598.log 2024-01-15T18:25:13.410 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37751.log 2024-01-15T18:25:13.411 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47598.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47598.log.gz 2024-01-15T18:25:13.414 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47197.log 2024-01-15T18:25:13.417 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37751.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37751.log.gz 2024-01-15T18:25:13.417 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41486.log 2024-01-15T18:25:13.419 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47197.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-01-15T18:25:13.420 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41486.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41486.log.gz 2024-01-15T18:25:13.425 INFO:teuthology.orchestra.run.smithi027.stderr: 87.8%/var/log/ceph/ceph-osd.1.log: -- replaced with /var/log/ceph/ceph-client.admin.47197.log.gz 2024-01-15T18:25:13.425 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40969.log 2024-01-15T18:25:13.430 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53442.log 2024-01-15T18:25:13.431 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40969.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40969.log.gz 2024-01-15T18:25:13.437 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52696.log 2024-01-15T18:25:13.437 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53442.log.gz 2024-01-15T18:25:13.439 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36613.log 2024-01-15T18:25:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52696.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52696.log.gz 2024-01-15T18:25:13.442 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48814.log 2024-01-15T18:25:13.443 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36613.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36613.log.gz 2024-01-15T18:25:13.446 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38307.log 2024-01-15T18:25:13.450 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48814.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48814.log.gz 2024-01-15T18:25:13.450 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36365.log 2024-01-15T18:25:13.453 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38307.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38307.log.gz 2024-01-15T18:25:13.462 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32611.log 2024-01-15T18:25:13.463 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36365.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36365.log.gz 2024-01-15T18:25:13.466 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51005.log 2024-01-15T18:25:13.467 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32611.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32611.log.gz 2024-01-15T18:25:13.469 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51586.log 2024-01-15T18:25:13.470 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51005.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51005.log.gz 2024-01-15T18:25:13.471 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49987.log 2024-01-15T18:25:13.473 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51586.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51586.log.gz 2024-01-15T18:25:13.474 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43837.log 2024-01-15T18:25:13.475 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49987.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49987.log.gz 2024-01-15T18:25:13.477 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53064.log 2024-01-15T18:25:13.478 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43837.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43837.log.gz 2024-01-15T18:25:13.480 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53569.log 2024-01-15T18:25:13.481 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53064.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.53064.log.gz 2024-01-15T18:25:13.483 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36261.log 2024-01-15T18:25:13.484 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53569.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.53569.log.gz 2024-01-15T18:25:13.486 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33820.log 2024-01-15T18:25:13.493 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36261.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36261.log.gz 2024-01-15T18:25:13.493 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36393.log 2024-01-15T18:25:13.493 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33820.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33820.log.gz 2024-01-15T18:25:13.493 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52370.log 2024-01-15T18:25:13.493 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36393.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36393.log.gz 2024-01-15T18:25:13.494 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43195.log 2024-01-15T18:25:13.495 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52370.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52370.log.gz 2024-01-15T18:25:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38333.log 2024-01-15T18:25:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43195.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43195.log.gz 2024-01-15T18:25:13.499 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48972.log 2024-01-15T18:25:13.501 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38333.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38333.log.gz 2024-01-15T18:25:13.503 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48972.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48972.log.gz 2024-01-15T18:25:13.504 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35068.log 2024-01-15T18:25:13.507 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.33779.log 2024-01-15T18:25:13.510 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.35068.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.47491.log 2024-01-15T18:25:13.510 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-01-15T18:25:13.512 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.33779.log: 90.6% -- replaced with /var/log/ceph/ceph-client.0.33779.log.gz 2024-01-15T18:25:13.515 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45079.log 2024-01-15T18:25:13.516 INFO:teuthology.orchestra.run.smithi027.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47491.log.gz 2024-01-15T18:25:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40084.log 2024-01-15T18:25:13.526 INFO:teuthology.orchestra.run.smithi027.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45079.log.gz 2024-01-15T18:25:13.526 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41576.log 2024-01-15T18:25:13.527 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40084.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40084.log.gz 2024-01-15T18:25:13.530 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32372.log 2024-01-15T18:25:13.531 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41576.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41576.log.gz 2024-01-15T18:25:13.534 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5/var/log/ceph/ceph-client.admin.32372.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32372.log.gz 2024-01-15T18:25:13.537 INFO:teuthology.orchestra.run.smithi027.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41382.log 2024-01-15T18:25:13.542 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45313.log 2024-01-15T18:25:13.543 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41382.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41382.log.gz 2024-01-15T18:25:13.546 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50726.log 2024-01-15T18:25:13.547 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45313.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.45313.log.gz 2024-01-15T18:25:13.550 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46833.log 2024-01-15T18:25:13.551 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50726.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50726.log.gz 2024-01-15T18:25:13.554 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49959.log 2024-01-15T18:25:13.555 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46833.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46833.log.gz 2024-01-15T18:25:13.559 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49959.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49959.log.gz 2024-01-15T18:25:13.561 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54498.log 2024-01-15T18:25:13.566 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38842.log 2024-01-15T18:25:13.567 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54498.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54498.log.gz 2024-01-15T18:25:13.570 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46989.log 2024-01-15T18:25:13.571 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38842.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38842.log.gz 2024-01-15T18:25:13.574 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33233.log 2024-01-15T18:25:13.575 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46989.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46989.log.gz 2024-01-15T18:25:13.578 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43458.log 2024-01-15T18:25:13.579 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33233.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33233.log.gz 2024-01-15T18:25:13.582 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39432.log 2024-01-15T18:25:13.583 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43458.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43458.log.gz 2024-01-15T18:25:13.586 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44974.log 2024-01-15T18:25:13.587 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39432.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39432.log.gz 2024-01-15T18:25:13.590 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53846.log 2024-01-15T18:25:13.593 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44974.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44974.log.gz 2024-01-15T18:25:13.595 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53846.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53846.log.gz 2024-01-15T18:25:13.597 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36587.log 2024-01-15T18:25:13.602 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46703.log 2024-01-15T18:25:13.603 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36587.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36587.log.gz 2024-01-15T18:25:13.606 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40032.log 2024-01-15T18:25:13.607 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46703.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46703.log.gz 2024-01-15T18:25:13.610 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50389.log 2024-01-15T18:25:13.611 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40032.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40032.log.gz 2024-01-15T18:25:13.614 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39694.log 2024-01-15T18:25:13.615 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50389.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50389.log.gz 2024-01-15T18:25:13.621 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48668.log 2024-01-15T18:25:13.621 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39694.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39694.log.gz 2024-01-15T18:25:13.623 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48668.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48668.log.gz 2024-01-15T18:25:13.625 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54258.log 2024-01-15T18:25:13.630 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41229.log 2024-01-15T18:25:13.631 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54258.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54258.log.gz 2024-01-15T18:25:13.634 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50207.log 2024-01-15T18:25:13.636 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41229.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.41229.log.gz 2024-01-15T18:25:13.637 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45497.log 2024-01-15T18:25:13.638 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50207.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50207.log.gz 2024-01-15T18:25:13.643 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45497.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45497.log.gz 2024-01-15T18:25:13.645 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36976.log 2024-01-15T18:25:13.650 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43721.log 2024-01-15T18:25:13.650 INFO:teuthology.orchestra.run.smithi027.stderr: 96.5% -- replaced with /var/log/ceph/ceph-client.0.48488.log.gz 2024-01-15T18:25:13.652 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36976.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36976.log.gz 2024-01-15T18:25:13.652 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36481.log 2024-01-15T18:25:13.653 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43721.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43721.log.gz 2024-01-15T18:25:13.658 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34725.log 2024-01-15T18:25:13.659 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36481.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36481.log.gz 2024-01-15T18:25:13.662 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44283.log 2024-01-15T18:25:13.663 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34725.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34725.log.gz 2024-01-15T18:25:13.666 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43354.log 2024-01-15T18:25:13.674 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.44283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51349.log 2024-01-15T18:25:13.676 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43354.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43354.log.gz 2024-01-15T18:25:13.679 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51349.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51349.log.gz 2024-01-15T18:25:13.681 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39141.log 2024-01-15T18:25:13.686 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48433.log 2024-01-15T18:25:13.687 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39141.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39141.log.gz 2024-01-15T18:25:13.690 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36507.log 2024-01-15T18:25:13.692 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48433.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48433.log.gz 2024-01-15T18:25:13.698 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50898.log 2024-01-15T18:25:13.699 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36507.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36507.log.gz 2024-01-15T18:25:13.702 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33467.log 2024-01-15T18:25:13.703 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50898.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50898.log.gz 2024-01-15T18:25:13.706 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37516.log 2024-01-15T18:25:13.707 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33467.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33467.log.gz 2024-01-15T18:25:13.710 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35674.log 2024-01-15T18:25:13.711 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37516.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37516.log.gz 2024-01-15T18:25:13.714 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35674.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.35674.log.gz 2024-01-15T18:25:13.715 INFO:teuthology.orchestra.run.smithi027.stderr: 96.4% -- replaced with /var/log/ceph/ceph-client.0.38897.log.gz 2024-01-15T18:25:13.716 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51240.log 96.3% -- replaced with /var/log/ceph/ceph-client.0.36928.log.gz 2024-01-15T18:25:13.716 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-01-15T18:25:13.718 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34886.log 2024-01-15T18:25:13.723 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34886.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34886.log.gz 2024-01-15T18:25:13.723 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51240.log.gz 2024-01-15T18:25:13.723 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47145.log 2024-01-15T18:25:13.726 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47145.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47145.log.gz 2024-01-15T18:25:13.726 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44461.log 2024-01-15T18:25:13.729 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44461.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44461.log.gz 2024-01-15T18:25:13.729 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42848.log 2024-01-15T18:25:13.736 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42848.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42848.log.gz 2024-01-15T18:25:13.736 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49723.log 2024-01-15T18:25:13.736 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49723.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49723.log.gz 2024-01-15T18:25:13.736 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35305.log 2024-01-15T18:25:13.739 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35305.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35305.log.gz 2024-01-15T18:25:13.739 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49103.log 2024-01-15T18:25:13.745 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49103.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49103.log.gz 2024-01-15T18:25:13.745 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53898.log 2024-01-15T18:25:13.745 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47832.log 2024-01-15T18:25:13.746 INFO:teuthology.orchestra.run.smithi027.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53898.log.gz 2024-01-15T18:25:13.748 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41790.log 2024-01-15T18:25:13.750 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41790.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41790.log.gz 2024-01-15T18:25:13.752 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34065.log 2024-01-15T18:25:13.756 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34065.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34065.log.gz 2024-01-15T18:25:13.756 INFO:teuthology.orchestra.run.smithi027.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.47832.log.gz 2024-01-15T18:25:13.757 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47676.log 2024-01-15T18:25:13.760 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47676.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47676.log.gz 2024-01-15T18:25:13.760 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53923.log 2024-01-15T18:25:13.763 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53923.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53923.log.gz 2024-01-15T18:25:13.763 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40191.log 2024-01-15T18:25:13.766 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40191.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40191.log.gz 2024-01-15T18:25:13.766 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51297.log 2024-01-15T18:25:13.769 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33572.log 2024-01-15T18:25:13.770 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51297.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51297.log.gz 2024-01-15T18:25:13.773 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33572.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.33572.log.gz 2024-01-15T18:25:13.773 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49155.log 2024-01-15T18:25:13.776 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49155.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49155.log.gz 2024-01-15T18:25:13.776 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54024.log 2024-01-15T18:25:13.783 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54024.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.54024.log.gz 2024-01-15T18:25:13.783 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47171.log 2024-01-15T18:25:13.783 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47171.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47171.log.gz 2024-01-15T18:25:13.783 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43995.log 2024-01-15T18:25:13.786 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43995.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43995.log.gz 2024-01-15T18:25:13.786 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54806.log 2024-01-15T18:25:13.793 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54806.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54806.log.gz 2024-01-15T18:25:13.793 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45968.log 2024-01-15T18:25:13.793 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45968.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45968.log.gz 2024-01-15T18:25:13.793 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40719.log 2024-01-15T18:25:13.796 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40719.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40719.log.gz 2024-01-15T18:25:13.796 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49593.log 2024-01-15T18:25:13.799 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49593.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49593.log.gz 2024-01-15T18:25:13.799 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34620.log 2024-01-15T18:25:13.801 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53316.log 2024-01-15T18:25:13.805 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53316.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53316.log.gz 2024-01-15T18:25:13.806 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34620.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34620.log.gz 2024-01-15T18:25:13.806 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35979.log 2024-01-15T18:25:13.809 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35979.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35979.log.gz 2024-01-15T18:25:13.809 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48019.log 2024-01-15T18:25:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48019.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48019.log.gz 2024-01-15T18:25:13.812 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39049.log 2024-01-15T18:25:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39049.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39049.log.gz 2024-01-15T18:25:13.815 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52422.log 2024-01-15T18:25:13.818 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45729.log 2024-01-15T18:25:13.819 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52422.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52422.log.gz 2024-01-15T18:25:13.821 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38046.log 2024-01-15T18:25:13.822 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45729.log.gz 2024-01-15T18:25:13.824 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50259.log 2024-01-15T18:25:13.826 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38046.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38046.log.gz 2024-01-15T18:25:13.829 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50259.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50259.log.gz 2024-01-15T18:25:13.829 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35752.log 2024-01-15T18:25:13.832 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35752.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35752.log.gz 2024-01-15T18:25:13.832 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40828.log 2024-01-15T18:25:13.835 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40828.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40828.log.gz 2024-01-15T18:25:13.836 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32230.log 2024-01-15T18:25:13.839 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39980.log 2024-01-15T18:25:13.839 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32230.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32230.log.gz 2024-01-15T18:25:13.842 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39980.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39980.log.gz 2024-01-15T18:25:13.842 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49207.log 2024-01-15T18:25:13.845 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49207.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49207.log.gz 2024-01-15T18:25:13.845 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54576.log 2024-01-15T18:25:13.848 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54576.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54576.log.gz 2024-01-15T18:25:13.848 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36421.log 2024-01-15T18:25:13.851 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-15T18:25:13.852 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36421.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36421.log.gz 2024-01-15T18:25:13.857 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52887.log 2024-01-15T18:25:13.859 INFO:teuthology.orchestra.run.smithi027.stderr: 86.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-15T18:25:13.860 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33259.log 2024-01-15T18:25:13.862 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52887.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.52887.log.gz 2024-01-15T18:25:13.865 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50101.log 2024-01-15T18:25:13.868 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50101.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50101.log.gz 2024-01-15T18:25:13.869 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37620.log 2024-01-15T18:25:13.869 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33259.log.gz 2024-01-15T18:25:13.872 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.37620.log: /var/log/ceph/ceph-client.admin.32068.log 2024-01-15T18:25:13.876 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32068.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32068.log.gz 2024-01-15T18:25:13.876 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37620.log.gz 2024-01-15T18:25:13.877 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53291.log 2024-01-15T18:25:13.880 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53291.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.53291.log.gz 2024-01-15T18:25:13.880 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51430.log 2024-01-15T18:25:13.883 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51430.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51430.log.gz 2024-01-15T18:25:13.883 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48946.log 2024-01-15T18:25:13.886 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48946.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48946.log.gz 2024-01-15T18:25:13.887 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38714.log 2024-01-15T18:25:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43432.log 2024-01-15T18:25:13.890 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38714.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38714.log.gz 2024-01-15T18:25:13.893 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43432.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43432.log.gz 2024-01-15T18:25:13.893 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44047.log 2024-01-15T18:25:13.896 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44047.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44047.log.gz 2024-01-15T18:25:13.896 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40746.log 2024-01-15T18:25:13.900 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40746.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40746.log.gz 2024-01-15T18:25:13.900 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39668.log 2024-01-15T18:25:13.903 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53014.log 2024-01-15T18:25:13.903 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39668.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39668.log.gz 2024-01-15T18:25:13.906 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53014.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.53014.log.gz 2024-01-15T18:25:13.906 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37002.log 2024-01-15T18:25:13.909 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37002.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37002.log.gz 2024-01-15T18:25:13.909 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42167.log 2024-01-15T18:25:13.912 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42167.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42167.log.gz 2024-01-15T18:25:13.913 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51983.log 2024-01-15T18:25:13.916 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40914.log 2024-01-15T18:25:13.916 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51983.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51983.log.gz 2024-01-15T18:25:13.919 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40914.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40914.log.gz 2024-01-15T18:25:13.919 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37358.log 2024-01-15T18:25:13.921 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36692.log 2024-01-15T18:25:13.926 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36692.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36692.log.gz 2024-01-15T18:25:13.926 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37358.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37358.log.gz 2024-01-15T18:25:13.926 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47735.log 2024-01-15T18:25:13.929 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47735.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47735.log.gz 2024-01-15T18:25:13.929 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51534.log 2024-01-15T18:25:13.932 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51534.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51534.log.gz 2024-01-15T18:25:13.932 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40862.log 2024-01-15T18:25:13.935 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39510.log 2024-01-15T18:25:13.935 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40862.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40862.log.gz 2024-01-15T18:25:13.939 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39510.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39510.log.gz 2024-01-15T18:25:13.939 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40772.log 2024-01-15T18:25:13.942 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40772.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40772.log.gz 2024-01-15T18:25:13.942 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39406.log 2024-01-15T18:25:13.945 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39406.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39406.log.gz 2024-01-15T18:25:13.945 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54074.log 2024-01-15T18:25:13.948 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38766.log 2024-01-15T18:25:13.948 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54074.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.54074.log.gz 2024-01-15T18:25:13.952 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38766.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38766.log.gz 2024-01-15T18:25:13.952 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37915.log 2024-01-15T18:25:13.955 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37915.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37915.log.gz 2024-01-15T18:25:13.955 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50465.log 2024-01-15T18:25:13.958 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50465.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50465.log.gz 2024-01-15T18:25:13.958 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53644.log 2024-01-15T18:25:13.961 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44870.log 2024-01-15T18:25:13.961 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53644.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.53644.log.gz 2024-01-15T18:25:13.965 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44870.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44870.log.gz 2024-01-15T18:25:13.965 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48998.log 2024-01-15T18:25:13.968 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48998.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48998.log.gz 2024-01-15T18:25:13.968 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42479.log 2024-01-15T18:25:13.968 INFO:teuthology.orchestra.run.smithi027.stderr: 96.4% -- replaced with /var/log/ceph/ceph-client.0.50522.log.gz 2024-01-15T18:25:13.971 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42479.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42479.log.gz 2024-01-15T18:25:13.971 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46651.log 2024-01-15T18:25:13.972 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46937.log 2024-01-15T18:25:13.974 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46651.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46651.log.gz 2024-01-15T18:25:13.974 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46911.log 2024-01-15T18:25:13.975 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46937.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46937.log.gz 2024-01-15T18:25:13.976 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34860.log 2024-01-15T18:25:13.978 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46911.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.46911.log.gz 2024-01-15T18:25:13.978 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44764.log 2024-01-15T18:25:13.979 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34860.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34860.log.gz 2024-01-15T18:25:13.979 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36455.log 2024-01-15T18:25:13.981 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44764.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44764.log.gz 2024-01-15T18:25:13.982 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34353.log 2024-01-15T18:25:13.982 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36455.log.gz 2024-01-15T18:25:13.983 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53998.log 2024-01-15T18:25:13.985 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34353.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34353.log.gz 2024-01-15T18:25:13.985 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34301.log 2024-01-15T18:25:13.987 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53998.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.53998.log.gz 2024-01-15T18:25:13.988 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44227.log 2024-01-15T18:25:13.988 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34301.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34301.log.gz 2024-01-15T18:25:13.989 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48331.log 2024-01-15T18:25:13.990 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44227.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44227.log.gz 2024-01-15T18:25:13.992 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48331.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48331.log.gz 2024-01-15T18:25:13.993 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48842.log 2024-01-15T18:25:13.994 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44021.log 2024-01-15T18:25:13.996 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48842.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48842.log.gz 2024-01-15T18:25:13.997 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32598.log 2024-01-15T18:25:13.998 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44021.log.gz 2024-01-15T18:25:13.999 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35042.log 2024-01-15T18:25:14.000 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32598.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.32598.log.gz 2024-01-15T18:25:14.002 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35042.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35042.log.gz 2024-01-15T18:25:14.002 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39115.log 2024-01-15T18:25:14.003 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37860.log 2024-01-15T18:25:14.005 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39115.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39115.log.gz 2024-01-15T18:25:14.006 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37860.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37860.log.gz 2024-01-15T18:25:14.007 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52113.log 2024-01-15T18:25:14.008 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42585.log 2024-01-15T18:25:14.010 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52113.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52113.log.gz 2024-01-15T18:25:14.011 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42585.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42585.log.gz 2024-01-15T18:25:14.011 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37143.log 2024-01-15T18:25:14.013 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47223.log 2024-01-15T18:25:14.014 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37143.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37143.log.gz 2024-01-15T18:25:14.016 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47223.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47223.log.gz 2024-01-15T18:25:14.016 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49541.log 2024-01-15T18:25:14.018 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52139.log 2024-01-15T18:25:14.019 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49541.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49541.log.gz 2024-01-15T18:25:14.020 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52139.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52139.log.gz 2024-01-15T18:25:14.021 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32505.log 2024-01-15T18:25:14.022 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39327.log 2024-01-15T18:25:14.024 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32505.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32505.log.gz 2024-01-15T18:25:14.025 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33285.log 2024-01-15T18:25:14.026 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39327.log.gz 2024-01-15T18:25:14.027 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42219.log 2024-01-15T18:25:14.028 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33285.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33285.log.gz 2024-01-15T18:25:14.029 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42219.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42219.log.gz 2024-01-15T18:25:14.030 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.52270.log 2024-01-15T18:25:14.031 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38020.log 2024-01-15T18:25:14.037 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.52270.log: /var/log/ceph/ceph-client.admin.38020.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38020.log.gz 2024-01-15T18:25:14.037 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52862.log 2024-01-15T18:25:14.038 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52862.log: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.52862.log.gz 2024-01-15T18:25:14.038 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51744.log 2024-01-15T18:25:14.041 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53769.log 2024-01-15T18:25:14.042 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51744.log.gz 2024-01-15T18:25:14.044 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36235.log 2024-01-15T18:25:14.046 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53769.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.53769.log.gz 2024-01-15T18:25:14.049 INFO:teuthology.orchestra.run.smithi027.stderr:gzip/var/log/ceph/ceph-client.admin.36235.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.52989.log 2024-01-15T18:25:14.053 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52989.log: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.52989.log.gz 2024-01-15T18:25:14.053 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36235.log.gz 2024-01-15T18:25:14.054 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52563.log 2024-01-15T18:25:14.057 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52563.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52563.log.gz 2024-01-15T18:25:14.057 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42356.log 2024-01-15T18:25:14.061 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42356.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42356.log.gz 2024-01-15T18:25:14.061 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45809.log 2024-01-15T18:25:14.064 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45809.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45809.log.gz 2024-01-15T18:25:14.064 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49403.log 2024-01-15T18:25:14.067 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36666.log 2024-01-15T18:25:14.067 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49403.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49403.log.gz 2024-01-15T18:25:14.070 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36666.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36666.log.gz 2024-01-15T18:25:14.071 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43380.log 2024-01-15T18:25:14.074 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43380.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43380.log.gz 2024-01-15T18:25:14.074 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40667.log 2024-01-15T18:25:14.077 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40667.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40667.log.gz 2024-01-15T18:25:14.077 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41512.log 2024-01-15T18:25:14.084 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48175.log 2024-01-15T18:25:14.085 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41512.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41512.log.gz 2024-01-15T18:25:14.085 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48175.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48175.log.gz 2024-01-15T18:25:14.085 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37106.log 2024-01-15T18:25:14.088 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37106.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37106.log.gz 2024-01-15T18:25:14.088 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34457.log 2024-01-15T18:25:14.091 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34457.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34457.log.gz 2024-01-15T18:25:14.091 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32818.log 2024-01-15T18:25:14.094 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32818.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32818.log.gz 2024-01-15T18:25:14.095 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39720.log 2024-01-15T18:25:14.097 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48564.log 2024-01-15T18:25:14.097 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39720.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39720.log.gz 2024-01-15T18:25:14.100 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39928.log 2024-01-15T18:25:14.101 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48564.log.gz 2024-01-15T18:25:14.105 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39928.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39928.log.gz 2024-01-15T18:25:14.105 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46495.log 2024-01-15T18:25:14.108 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46495.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46495.log.gz 2024-01-15T18:25:14.108 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55253.log 2024-01-15T18:25:14.115 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55253.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55253.log.gz 2024-01-15T18:25:14.115 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41816.log 2024-01-15T18:25:14.115 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41816.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41816.log.gz 2024-01-15T18:25:14.115 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53467.log 2024-01-15T18:25:14.118 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53467.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53467.log.gz 2024-01-15T18:25:14.119 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37993.log 2024-01-15T18:25:14.121 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33012.log 2024-01-15T18:25:14.122 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37993.log.gz 2024-01-15T18:25:14.124 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47858.log 2024-01-15T18:25:14.126 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33012.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33012.log.gz 2024-01-15T18:25:14.129 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47858.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47858.log.gz 2024-01-15T18:25:14.130 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34990.log 2024-01-15T18:25:14.135 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34990.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34990.log.gz 2024-01-15T18:25:14.136 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44499.log 2024-01-15T18:25:14.138 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44499.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44499.log.gz 2024-01-15T18:25:14.138 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40943.log 2024-01-15T18:25:14.140 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40943.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40943.log.gz 2024-01-15T18:25:14.140 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46573.log 2024-01-15T18:25:14.143 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44099.log 2024-01-15T18:25:14.143 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46573.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46573.log.gz 2024-01-15T18:25:14.147 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44099.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44099.log.gz 2024-01-15T18:25:14.147 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43484.log 2024-01-15T18:25:14.150 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43484.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43484.log.gz 2024-01-15T18:25:14.150 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36128.log 2024-01-15T18:25:14.153 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36128.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36128.log.gz 2024-01-15T18:25:14.153 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35200.log 2024-01-15T18:25:14.156 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43640.log 2024-01-15T18:25:14.157 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35200.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35200.log.gz 2024-01-15T18:25:14.160 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43640.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43640.log.gz 2024-01-15T18:25:14.160 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46417.log 2024-01-15T18:25:14.164 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46417.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46417.log.gz 2024-01-15T18:25:14.164 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45757.log 2024-01-15T18:25:14.167 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45757.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45757.log.gz 2024-01-15T18:25:14.167 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49756.log 2024-01-15T18:25:14.169 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50337.log 2024-01-15T18:25:14.173 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49756.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49756.log.gz 2024-01-15T18:25:14.173 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50337.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50337.log.gz 2024-01-15T18:25:14.174 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50570.log 2024-01-15T18:25:14.177 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45549.log 2024-01-15T18:25:14.178 INFO:teuthology.orchestra.run.smithi027.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50570.log.gz 2024-01-15T18:25:14.180 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46365.log 2024-01-15T18:25:14.185 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46365.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46365.log.gz 2024-01-15T18:25:14.186 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52009.log 2024-01-15T18:25:14.186 INFO:teuthology.orchestra.run.smithi027.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45549.log.gz 2024-01-15T18:25:14.188 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36640.log 2024-01-15T18:25:14.193 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36640.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36640.log.gz 2024-01-15T18:25:14.193 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52009.log.gz 2024-01-15T18:25:14.193 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37490.log 2024-01-15T18:25:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37490.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37490.log.gz 2024-01-15T18:25:14.197 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35094.log 2024-01-15T18:25:14.200 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35094.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35094.log.gz 2024-01-15T18:25:14.200 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47465.log 2024-01-15T18:25:14.203 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47465.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47465.log.gz 2024-01-15T18:25:14.203 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44177.log 2024-01-15T18:25:14.205 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52526.log 2024-01-15T18:25:14.208 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44177.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.44177.log.gz 2024-01-15T18:25:14.211 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52526.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52526.log.gz 2024-01-15T18:25:14.211 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34535.log 2024-01-15T18:25:14.214 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34535.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34535.log.gz 2024-01-15T18:25:14.215 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32292.log 2024-01-15T18:25:14.218 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32292.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32292.log.gz 2024-01-15T18:25:14.218 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46130.log 2024-01-15T18:25:14.221 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51266.log 2024-01-15T18:25:14.224 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46130.log: /var/log/ceph/ceph-client.admin.51266.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51266.log.gz 2024-01-15T18:25:14.224 INFO:teuthology.orchestra.run.smithi027.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46130.log.gz 2024-01-15T18:25:14.225 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35331.log 2024-01-15T18:25:14.229 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35331.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35331.log.gz 2024-01-15T18:25:14.229 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39798.log 2024-01-15T18:25:14.232 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39798.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39798.log.gz 2024-01-15T18:25:14.232 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51135.log 2024-01-15T18:25:14.234 INFO:teuthology.orchestra.run.smithi027.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-01-15T18:25:14.234 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50778.log 2024-01-15T18:25:14.235 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51135.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51135.log.gz 2024-01-15T18:25:14.236 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34039.log 2024-01-15T18:25:14.238 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50778.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50778.log.gz 2024-01-15T18:25:14.238 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47305.log 2024-01-15T18:25:14.239 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34039.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34039.log.gz 2024-01-15T18:25:14.240 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53819.log 2024-01-15T18:25:14.241 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47305.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.47305.log.gz 2024-01-15T18:25:14.242 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32810.log 2024-01-15T18:25:14.243 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53819.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.53819.log.gz 2024-01-15T18:25:14.244 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47884.log 2024-01-15T18:25:14.245 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32810.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32810.log.gz 2024-01-15T18:25:14.246 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32608.log 2024-01-15T18:25:14.247 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47884.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47884.log.gz 2024-01-15T18:25:14.248 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38475.log 2024-01-15T18:25:14.249 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32608.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32608.log.gz 2024-01-15T18:25:14.250 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36005.log 2024-01-15T18:25:14.251 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38475.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38475.log.gz 2024-01-15T18:25:14.252 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49854.log 2024-01-15T18:25:14.253 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36005.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36005.log.gz 2024-01-15T18:25:14.253 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47277.log 2024-01-15T18:25:14.255 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49854.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49854.log.gz 2024-01-15T18:25:14.255 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47041.log 2024-01-15T18:25:14.256 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47277.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47277.log.gz 2024-01-15T18:25:14.257 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38389.log 2024-01-15T18:25:14.259 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47041.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47041.log.gz 2024-01-15T18:25:14.259 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41999.log 2024-01-15T18:25:14.260 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38389.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38389.log.gz 2024-01-15T18:25:14.262 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48227.log 2024-01-15T18:25:14.262 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41999.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41999.log.gz 2024-01-15T18:25:14.263 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32427.log 2024-01-15T18:25:14.265 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48227.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48227.log.gz 2024-01-15T18:25:14.265 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-15T18:25:14.267 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32427.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32427.log.gz 2024-01-15T18:25:14.267 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48253.log 2024-01-15T18:25:14.271 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.48253.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48253.log.gz 2024-01-15T18:25:14.271 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36796.log 2024-01-15T18:25:14.272 INFO:teuthology.orchestra.run.smithi027.stderr: 90.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-15T18:25:14.273 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40511.log 2024-01-15T18:25:14.274 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36796.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36796.log.gz 2024-01-15T18:25:14.274 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-01-15T18:25:14.276 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40511.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40511.log.gz 2024-01-15T18:25:14.277 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37332.log 2024-01-15T18:25:14.280 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.37332.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37332.log.gz 2024-01-15T18:25:14.280 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51057.log 2024-01-15T18:25:14.283 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51057.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51057.log.gz 2024-01-15T18:25:14.284 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-01-15T18:25:14.289 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42796.log 2024-01-15T18:25:14.294 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48279.log 2024-01-15T18:25:14.295 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42796.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42796.log.gz 2024-01-15T18:25:14.297 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49129.log 2024-01-15T18:25:14.298 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48279.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48279.log.gz 2024-01-15T18:25:14.304 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49129.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49129.log.gz 2024-01-15T18:25:14.305 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47015.log 2024-01-15T18:25:14.310 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45053.log 2024-01-15T18:25:14.311 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47015.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47015.log.gz 2024-01-15T18:25:14.314 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54205.log 2024-01-15T18:25:14.315 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45053.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45053.log.gz 2024-01-15T18:25:14.318 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54205.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54205.log.gz 2024-01-15T18:25:14.321 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37967.log 2024-01-15T18:25:14.326 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46963.log 2024-01-15T18:25:14.327 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37967.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37967.log.gz 2024-01-15T18:25:14.330 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51323.log 2024-01-15T18:25:14.331 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46963.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46963.log.gz 2024-01-15T18:25:14.333 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42744.log 2024-01-15T18:25:14.334 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51323.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.51323.log.gz 2024-01-15T18:25:14.338 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47439.log 2024-01-15T18:25:14.339 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42744.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42744.log.gz 2024-01-15T18:25:14.343 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47439.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47439.log.gz 2024-01-15T18:25:14.345 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37646.log 2024-01-15T18:25:14.350 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34483.log 2024-01-15T18:25:14.351 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37646.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37646.log.gz 2024-01-15T18:25:14.354 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35518.log 2024-01-15T18:25:14.358 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34483.log: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32531.log 2024-01-15T18:25:14.359 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35518.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35518.log.gz 2024-01-15T18:25:14.361 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34483.log.gz 2024-01-15T18:25:14.363 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32531.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.32531.log.gz 2024-01-15T18:25:14.365 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36181.log 2024-01-15T18:25:14.370 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54936.log 2024-01-15T18:25:14.371 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36181.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36181.log.gz 2024-01-15T18:25:14.374 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37834.log 2024-01-15T18:25:14.375 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54936.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54936.log.gz 2024-01-15T18:25:14.377 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43811.log 2024-01-15T18:25:14.382 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39223.log 2024-01-15T18:25:14.382 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37834.log.gz 2024-01-15T18:25:14.383 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43811.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43811.log.gz 2024-01-15T18:25:14.389 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53114.log 2024-01-15T18:25:14.389 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39223.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39223.log.gz 2024-01-15T18:25:14.389 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32174.log 2024-01-15T18:25:14.390 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53114.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53114.log.gz 2024-01-15T18:25:14.395 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32174.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32174.log.gz 2024-01-15T18:25:14.397 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48097.log 2024-01-15T18:25:14.402 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46156.log 2024-01-15T18:25:14.403 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48097.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48097.log.gz 2024-01-15T18:25:14.406 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49567.log 2024-01-15T18:25:14.407 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46156.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46156.log.gz 2024-01-15T18:25:14.409 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41099.log 2024-01-15T18:25:14.410 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49567.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49567.log.gz 2024-01-15T18:25:14.414 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43549.log 2024-01-15T18:25:14.415 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41099.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41099.log.gz 2024-01-15T18:25:14.417 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42557.log 2024-01-15T18:25:14.418 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43549.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43549.log.gz 2024-01-15T18:25:14.422 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39536.log 2024-01-15T18:25:14.423 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42557.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42557.log.gz 2024-01-15T18:25:14.426 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39536.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39536.log.gz 2024-01-15T18:25:14.433 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51508.log 2024-01-15T18:25:14.438 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49311.log 2024-01-15T18:25:14.439 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51508.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51508.log.gz 2024-01-15T18:25:14.442 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51823.log 2024-01-15T18:25:14.443 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49311.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49311.log.gz 2024-01-15T18:25:14.449 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42611.log 2024-01-15T18:25:14.450 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51823.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51823.log.gz 2024-01-15T18:25:14.454 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34803.log 2024-01-15T18:25:14.455 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42611.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42611.log.gz 2024-01-15T18:25:14.458 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45287.log 2024-01-15T18:25:14.459 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34803.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34803.log.gz 2024-01-15T18:25:14.461 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53744.log 2024-01-15T18:25:14.466 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45287.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45287.log.gz 2024-01-15T18:25:14.467 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53744.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.53744.log.gz 2024-01-15T18:25:14.468 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43030.log 2024-01-15T18:25:14.477 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51109.log 2024-01-15T18:25:14.477 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43030.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43030.log.gz 2024-01-15T18:25:14.478 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46234.log 2024-01-15T18:25:14.479 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51109.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51109.log.gz 2024-01-15T18:25:14.482 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32401.log 2024-01-15T18:25:14.486 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46234.log: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49933.log 2024-01-15T18:25:14.487 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32401.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32401.log.gz 2024-01-15T18:25:14.489 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46234.log.gz 2024-01-15T18:25:14.490 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45942.log 2024-01-15T18:25:14.491 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49933.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49933.log.gz 2024-01-15T18:25:14.495 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45942.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45942.log.gz 2024-01-15T18:25:14.497 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39902.log 2024-01-15T18:25:14.502 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51163.log 2024-01-15T18:25:14.504 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39902.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39902.log.gz 2024-01-15T18:25:14.506 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37384.log 2024-01-15T18:25:14.509 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51163.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51163.log.gz 2024-01-15T18:25:14.510 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51770.log 2024-01-15T18:25:14.511 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37384.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37384.log.gz 2024-01-15T18:25:14.513 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39484.log 2024-01-15T18:25:14.514 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51770.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51770.log.gz 2024-01-15T18:25:14.519 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39484.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39484.log.gz 2024-01-15T18:25:14.521 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44736.log 2024-01-15T18:25:14.526 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32610.log 2024-01-15T18:25:14.528 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44736.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44736.log.gz 2024-01-15T18:25:14.530 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55277.log 2024-01-15T18:25:14.531 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32610.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32610.log.gz 2024-01-15T18:25:14.534 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50924.log 2024-01-15T18:25:14.535 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55277.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55277.log.gz 2024-01-15T18:25:14.538 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43060.log 2024-01-15T18:25:14.539 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50924.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50924.log.gz 2024-01-15T18:25:14.542 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53039.log 2024-01-15T18:25:14.544 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43060.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43060.log.gz 2024-01-15T18:25:14.547 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52344.log 2024-01-15T18:25:14.548 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53039.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.53039.log.gz 2024-01-15T18:25:14.550 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48730.log 2024-01-15T18:25:14.551 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52344.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52344.log.gz 2024-01-15T18:25:14.557 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48730.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48730.log.gz 2024-01-15T18:25:14.557 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54650.log 2024-01-15T18:25:14.562 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53794.log 2024-01-15T18:25:14.563 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54650.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54650.log.gz 2024-01-15T18:25:14.566 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.33911.log 2024-01-15T18:25:14.567 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53794.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.53794.log.gz 2024-01-15T18:25:14.569 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51957.log 2024-01-15T18:25:14.575 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.2.33911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52087.log 2024-01-15T18:25:14.576 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51957.log: 91.1% -- replaced with /var/log/ceph/ceph-client.2.33911.log.gz 2024-01-15T18:25:14.576 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51957.log.gz 2024-01-15T18:25:14.578 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52939.log 2024-01-15T18:25:14.579 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52087.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52087.log.gz 2024-01-15T18:25:14.581 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34777.log 2024-01-15T18:25:14.582 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52939.log: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.52939.log.gz 2024-01-15T18:25:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34777.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34777.log.gz 2024-01-15T18:25:14.589 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48894.log 2024-01-15T18:25:14.594 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50674.log 2024-01-15T18:25:14.595 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48894.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48894.log.gz 2024-01-15T18:25:14.598 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53669.log 2024-01-15T18:25:14.599 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50674.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50674.log.gz 2024-01-15T18:25:14.601 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46885.log 2024-01-15T18:25:14.606 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53669.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.53669.log.gz 2024-01-15T18:25:14.606 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33493.log 2024-01-15T18:25:14.608 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46885.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46885.log.gz 2024-01-15T18:25:14.611 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33493.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33493.log.gz 2024-01-15T18:25:14.613 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34117.log 2024-01-15T18:25:14.618 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33363.log 2024-01-15T18:25:14.619 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34117.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34117.log.gz 2024-01-15T18:25:14.622 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48704.log 2024-01-15T18:25:14.624 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33363.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33363.log.gz 2024-01-15T18:25:14.626 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50648.log 2024-01-15T18:25:14.627 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48704.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48704.log.gz 2024-01-15T18:25:14.630 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52748.log 2024-01-15T18:25:14.631 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50648.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50648.log.gz 2024-01-15T18:25:14.634 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34013.log 2024-01-15T18:25:14.635 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52748.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.52748.log.gz 2024-01-15T18:25:14.637 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54420.log 2024-01-15T18:25:14.641 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34013.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34013.log.gz 2024-01-15T18:25:14.643 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54420.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54420.log.gz 2024-01-15T18:25:14.645 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37280.log 2024-01-15T18:25:14.650 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48071.log 2024-01-15T18:25:14.651 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37280.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37280.log.gz 2024-01-15T18:25:14.654 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54702.log 2024-01-15T18:25:14.655 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48071.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48071.log.gz 2024-01-15T18:25:14.657 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42245.log 2024-01-15T18:25:14.662 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54702.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54702.log.gz 2024-01-15T18:25:14.662 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53189.log 2024-01-15T18:25:14.664 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42245.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42245.log.gz 2024-01-15T18:25:14.667 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53189.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53189.log.gz 2024-01-15T18:25:14.669 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53594.log 2024-01-15T18:25:14.674 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50181.log 2024-01-15T18:25:14.675 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53594.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53594.log.gz 2024-01-15T18:25:14.678 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37306.log 2024-01-15T18:25:14.679 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50181.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50181.log.gz 2024-01-15T18:25:14.682 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43004.log 2024-01-15T18:25:14.683 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37306.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37306.log.gz 2024-01-15T18:25:14.685 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51851.log 2024-01-15T18:25:14.685 INFO:teuthology.orchestra.run.smithi093.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-01-15T18:25:14.691 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43004.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43004.log.gz 2024-01-15T18:25:14.691 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51851.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51851.log.gz 2024-01-15T18:25:14.693 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37228.log 2024-01-15T18:25:14.698 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42505.log 2024-01-15T18:25:14.699 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.37228.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37228.log.gz 2024-01-15T18:25:14.702 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38361.log 2024-01-15T18:25:14.703 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42505.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42505.log.gz 2024-01-15T18:25:14.705 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45157.log 2024-01-15T18:25:14.710 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38361.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38361.log.gz 2024-01-15T18:25:14.712 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45157.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45157.log.gz 2024-01-15T18:25:14.713 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47993.log 2024-01-15T18:25:14.718 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47937.log 2024-01-15T18:25:14.719 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47993.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47993.log.gz 2024-01-15T18:25:14.722 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50155.log 2024-01-15T18:25:14.723 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47937.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47937.log.gz 2024-01-15T18:25:14.726 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50870.log 2024-01-15T18:25:14.728 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50155.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50155.log.gz 2024-01-15T18:25:14.730 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50415.log 2024-01-15T18:25:14.731 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50870.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50870.log.gz 2024-01-15T18:25:14.734 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54232.log 2024-01-15T18:25:14.736 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50415.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.50415.log.gz 2024-01-15T18:25:14.738 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55303.log 2024-01-15T18:25:14.739 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54232.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54232.log.gz 2024-01-15T18:25:14.744 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55303.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.55303.log.gz 2024-01-15T18:25:14.745 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40693.log 2024-01-15T18:25:14.750 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49259.log 2024-01-15T18:25:14.752 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40693.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40693.log.gz 2024-01-15T18:25:14.754 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49697.log 2024-01-15T18:25:14.755 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49259.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49259.log.gz 2024-01-15T18:25:14.758 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34594.log 2024-01-15T18:25:14.760 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49697.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49697.log.gz 2024-01-15T18:25:14.762 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45862.log 2024-01-15T18:25:14.763 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34594.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34594.log.gz 2024-01-15T18:25:14.766 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35882.log 2024-01-15T18:25:14.767 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45862.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45862.log.gz 2024-01-15T18:25:14.769 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38423.log 2024-01-15T18:25:14.773 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35882.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35882.log.gz 2024-01-15T18:25:14.775 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38423.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38423.log.gz 2024-01-15T18:25:14.777 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44684.log 2024-01-15T18:25:14.782 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34829.log 2024-01-15T18:25:14.783 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44684.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44684.log.gz 2024-01-15T18:25:14.786 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39354.log 2024-01-15T18:25:14.787 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34829.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34829.log.gz 2024-01-15T18:25:14.789 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43302.log 2024-01-15T18:25:14.794 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45261.log 2024-01-15T18:25:14.795 INFO:teuthology.orchestra.run.smithi027.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39354.log.gz 2024-01-15T18:25:14.795 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43302.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43302.log.gz 2024-01-15T18:25:14.801 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45261.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41047.log 2024-01-15T18:25:14.805 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45261.log.gz 2024-01-15T18:25:14.806 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36718.log 2024-01-15T18:25:14.807 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41047.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41047.log.gz 2024-01-15T18:25:14.810 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33546.log 2024-01-15T18:25:14.811 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36718.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36718.log.gz 2024-01-15T18:25:14.813 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39075.log 2024-01-15T18:25:14.818 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33546.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33546.log.gz 2024-01-15T18:25:14.820 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39075.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39075.log.gz 2024-01-15T18:25:14.821 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54884.log 2024-01-15T18:25:14.826 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46755.log 2024-01-15T18:25:14.827 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54884.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54884.log.gz 2024-01-15T18:25:14.830 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41203.log 2024-01-15T18:25:14.832 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46755.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46755.log.gz 2024-01-15T18:25:14.834 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49460.log 2024-01-15T18:25:14.838 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41203.log: 89.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43138.log 2024-01-15T18:25:14.839 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49460.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49460.log.gz 2024-01-15T18:25:14.841 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41203.log.gz 2024-01-15T18:25:14.845 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43138.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43138.log.gz 2024-01-15T18:25:14.845 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47572.log 2024-01-15T18:25:14.850 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41628.log 2024-01-15T18:25:14.852 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47572.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47572.log.gz 2024-01-15T18:25:14.854 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34405.log 2024-01-15T18:25:14.856 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41628.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41628.log.gz 2024-01-15T18:25:14.858 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32806.log 2024-01-15T18:25:14.859 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34405.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34405.log.gz 2024-01-15T18:25:14.862 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40248.log 2024-01-15T18:25:14.864 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32806.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32806.log.gz 2024-01-15T18:25:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40248.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40248.log.gz 2024-01-15T18:25:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46078.log 2024-01-15T18:25:14.874 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34195.log 2024-01-15T18:25:14.875 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46078.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46078.log.gz 2024-01-15T18:25:14.878 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38608.log 2024-01-15T18:25:14.879 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34195.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34195.log.gz 2024-01-15T18:25:14.882 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33207.log 2024-01-15T18:25:14.883 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38608.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38608.log.gz 2024-01-15T18:25:14.890 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45027.log 2024-01-15T18:25:14.891 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33207.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33207.log.gz 2024-01-15T18:25:14.894 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43614.log 2024-01-15T18:25:14.895 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45027.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.45027.log.gz 2024-01-15T18:25:14.897 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45339.log 2024-01-15T18:25:14.898 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43614.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43614.log.gz 2024-01-15T18:25:14.902 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53392.log 2024-01-15T18:25:14.904 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45339.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45339.log.gz 2024-01-15T18:25:14.911 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53392.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.53392.log.gz 2024-01-15T18:25:14.913 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48305.log 2024-01-15T18:25:14.918 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.41334.log 2024-01-15T18:25:14.920 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48305.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48305.log.gz 2024-01-15T18:25:14.922 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53214.log 2024-01-15T18:25:14.926 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.41334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32819.log 2024-01-15T18:25:14.927 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53214.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.53214.log.gz 2024-01-15T18:25:14.932 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32819.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32819.log.gz 2024-01-15T18:25:14.933 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53139.log 2024-01-15T18:25:14.938 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35440.log 2024-01-15T18:25:14.940 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53139.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.53139.log.gz 2024-01-15T18:25:14.943 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49906.log 2024-01-15T18:25:14.943 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35440.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35440.log.gz 2024-01-15T18:25:14.946 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50842.log 2024-01-15T18:25:14.947 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49906.log: 96.4% 87.8% -- replaced with /var/log/ceph/ceph-client.0.44283.log.gz -- replaced with /var/log/ceph/ceph-client.admin.49906.log.gz 2024-01-15T18:25:14.948 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-01-15T18:25:14.950 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50233.log 2024-01-15T18:25:14.951 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50842.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50842.log.gz 2024-01-15T18:25:14.954 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45419.log 2024-01-15T18:25:14.955 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50233.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50233.log.gz 2024-01-15T18:25:14.958 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51797.log 2024-01-15T18:25:14.959 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45419.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45419.log.gz 2024-01-15T18:25:14.962 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51797.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51797.log.gz 2024-01-15T18:25:14.965 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42822.log 2024-01-15T18:25:14.970 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45445.log 2024-01-15T18:25:14.971 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42822.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42822.log.gz 2024-01-15T18:25:14.973 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49233.log 2024-01-15T18:25:14.978 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46677.log 2024-01-15T18:25:14.979 INFO:teuthology.orchestra.run.smithi027.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45445.log.gz 2024-01-15T18:25:14.979 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49233.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49233.log.gz 2024-01-15T18:25:14.982 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55144.log 2024-01-15T18:25:14.983 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46677.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46677.log.gz 2024-01-15T18:25:14.984 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53873.log 2024-01-15T18:25:14.990 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55144.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55144.log.gz 2024-01-15T18:25:14.992 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49788.log 2024-01-15T18:25:14.993 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.53873.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.53873.log.gz 2024-01-15T18:25:14.998 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51404.log 2024-01-15T18:25:14.999 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49788.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49788.log.gz 2024-01-15T18:25:15.002 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54602.log 2024-01-15T18:25:15.003 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51404.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51404.log.gz 2024-01-15T18:25:15.007 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54602.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54602.log.gz 2024-01-15T18:25:15.009 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51905.log 2024-01-15T18:25:15.014 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41151.log 2024-01-15T18:25:15.015 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51905.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51905.log.gz 2024-01-15T18:25:15.018 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47965.log 2024-01-15T18:25:15.019 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41151.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41151.log.gz 2024-01-15T18:25:15.022 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42077.log 2024-01-15T18:25:15.024 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47965.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47965.log.gz 2024-01-15T18:25:15.026 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39275.log 2024-01-15T18:25:15.027 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42077.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42077.log.gz 2024-01-15T18:25:15.030 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43221.log 2024-01-15T18:25:15.031 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39275.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39275.log.gz 2024-01-15T18:25:15.034 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42874.log 2024-01-15T18:25:15.035 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43221.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43221.log.gz 2024-01-15T18:25:15.038 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27215.log 2024-01-15T18:25:15.039 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42874.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42874.log.gz 2024-01-15T18:25:15.043 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.27215.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.27215.log.gz 2024-01-15T18:25:15.045 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49671.log 2024-01-15T18:25:15.046 INFO:teuthology.orchestra.run.smithi093.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-01-15T18:25:15.050 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32596.log 2024-01-15T18:25:15.051 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49671.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49671.log.gz 2024-01-15T18:25:15.054 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50311.log 2024-01-15T18:25:15.055 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32596.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32596.log.gz 2024-01-15T18:25:15.057 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48201.log 2024-01-15T18:25:15.061 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50311.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50311.log.gz 2024-01-15T18:25:15.062 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44435.log 2024-01-15T18:25:15.063 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48201.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48201.log.gz 2024-01-15T18:25:15.065 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47911.log 2024-01-15T18:25:15.071 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35279.log 2024-01-15T18:25:15.071 INFO:teuthology.orchestra.run.smithi027.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44435.log.gz 2024-01-15T18:25:15.071 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47911.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47911.log.gz 2024-01-15T18:25:15.075 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35279.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35279.log.gz 2024-01-15T18:25:15.077 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36770.log 2024-01-15T18:25:15.082 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40570.log 2024-01-15T18:25:15.084 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36770.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36770.log.gz 2024-01-15T18:25:15.086 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43863.log 2024-01-15T18:25:15.087 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40570.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40570.log.gz 2024-01-15T18:25:15.090 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49645.log 2024-01-15T18:25:15.091 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43863.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43863.log.gz 2024-01-15T18:25:15.093 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42926.log 2024-01-15T18:25:15.098 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.49645.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49645.log.gz 2024-01-15T18:25:15.099 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42926.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42926.log.gz 2024-01-15T18:25:15.101 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35804.log 2024-01-15T18:25:15.106 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32557.log 2024-01-15T18:25:15.107 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35804.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35804.log.gz 2024-01-15T18:25:15.110 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41764.log 2024-01-15T18:25:15.111 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32557.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.32557.log.gz 2024-01-15T18:25:15.113 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46469.log 2024-01-15T18:25:15.118 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41764.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41764.log.gz 2024-01-15T18:25:15.120 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46469.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46469.log.gz 2024-01-15T18:25:15.121 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40058.log 2024-01-15T18:25:15.126 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47520.log 2024-01-15T18:25:15.127 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40058.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40058.log.gz 2024-01-15T18:25:15.130 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48616.log 2024-01-15T18:25:15.131 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47520.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47520.log.gz 2024-01-15T18:25:15.134 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50049.log 2024-01-15T18:25:15.135 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48616.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48616.log.gz 2024-01-15T18:25:15.138 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42637.log 2024-01-15T18:25:15.139 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50049.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50049.log.gz 2024-01-15T18:25:15.142 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38582.log 2024-01-15T18:25:15.143 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42637.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42637.log.gz 2024-01-15T18:25:15.145 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32318.log 2024-01-15T18:25:15.151 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38582.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38582.log.gz 2024-01-15T18:25:15.151 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32318.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32318.log.gz 2024-01-15T18:25:15.153 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34751.log 2024-01-15T18:25:15.158 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44357.log 2024-01-15T18:25:15.159 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34751.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34751.log.gz 2024-01-15T18:25:15.163 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54910.log 2024-01-15T18:25:15.163 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44357.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44357.log.gz 2024-01-15T18:25:15.165 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46521.log 2024-01-15T18:25:15.170 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54910.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54910.log.gz 2024-01-15T18:25:15.170 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41894.log 2024-01-15T18:25:15.171 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46521.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46521.log.gz 2024-01-15T18:25:15.174 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52215.log 2024-01-15T18:25:15.175 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41894.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41894.log.gz 2024-01-15T18:25:15.179 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52215.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52215.log.gz 2024-01-15T18:25:15.181 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39380.log 2024-01-15T18:25:15.186 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44844.log 2024-01-15T18:25:15.187 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39380.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39380.log.gz 2024-01-15T18:25:15.190 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43247.log 2024-01-15T18:25:15.191 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.44844.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44844.log.gz 2024-01-15T18:25:15.194 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50015.log 2024-01-15T18:25:15.195 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.43247.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43247.log.gz 2024-01-15T18:25:15.198 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51456.log 2024-01-15T18:25:15.199 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50015.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50015.log.gz 2024-01-15T18:25:15.202 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50752.log 2024-01-15T18:25:15.204 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.51456.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51456.log.gz 2024-01-15T18:25:15.210 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36561.log 2024-01-15T18:25:15.211 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.50752.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50752.log.gz 2024-01-15T18:25:15.216 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36561.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36561.log.gz 2024-01-15T18:25:15.217 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55118.log 2024-01-15T18:25:15.222 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46781.log 2024-01-15T18:25:15.224 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.55118.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55118.log.gz 2024-01-15T18:25:15.226 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33519.log 2024-01-15T18:25:15.227 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46781.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46781.log.gz 2024-01-15T18:25:15.230 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35934.log 2024-01-15T18:25:15.231 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33519.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33519.log.gz 2024-01-15T18:25:15.234 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33415.log 2024-01-15T18:25:15.235 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35934.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35934.log.gz 2024-01-15T18:25:15.242 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42193.log 2024-01-15T18:25:15.243 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33415.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.33415.log.gz 2024-01-15T18:25:15.253 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42193.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42193.log.gz 2024-01-15T18:25:15.253 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32798.log 2024-01-15T18:25:15.254 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34169.log 2024-01-15T18:25:15.256 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32798.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32798.log.gz 2024-01-15T18:25:15.258 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38971.log 2024-01-15T18:25:15.259 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34169.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34169.log.gz 2024-01-15T18:25:15.264 INFO:teuthology.orchestra.run.smithi027.stderr: 96.5% -- replaced with /var/log/ceph/ceph-client.0.52270.log.gz 2024-01-15T18:25:15.265 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36154.log 2024-01-15T18:25:15.268 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31978.log 2024-01-15T18:25:15.269 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.36154.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36154.log.gz 2024-01-15T18:25:15.269 INFO:teuthology.orchestra.run.smithi027.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38971.log.gz 2024-01-15T18:25:15.271 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32595.log 2024-01-15T18:25:15.273 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.31978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52914.log 2024-01-15T18:25:15.275 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32595.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32595.log.gz 2024-01-15T18:25:15.275 INFO:teuthology.orchestra.run.smithi027.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31978.log.gz 2024-01-15T18:25:15.277 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54368.log 2024-01-15T18:25:15.280 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52914.log: /var/log/ceph/ceph-client.admin.54368.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54368.log.gz 2024-01-15T18:25:15.281 INFO:teuthology.orchestra.run.smithi027.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.52914.log.gz 2024-01-15T18:25:15.282 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52061.log 2024-01-15T18:25:15.283 INFO:teuthology.orchestra.run.smithi093.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-01-15T18:25:15.285 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.52061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48536.log 2024-01-15T18:25:15.287 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32606.log 2024-01-15T18:25:15.289 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.48536.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48536.log.gz 2024-01-15T18:25:15.289 INFO:teuthology.orchestra.run.smithi027.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52061.log.gz 2024-01-15T18:25:15.292 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32606.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32606.log.gz 2024-01-15T18:25:15.292 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54394.log 2024-01-15T18:25:15.295 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54394.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.54394.log.gz 2024-01-15T18:25:15.295 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54962.log 2024-01-15T18:25:15.301 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.54962.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54962.log.gz 2024-01-15T18:25:15.301 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35908.log 2024-01-15T18:25:15.302 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.35908.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35908.log.gz 2024-01-15T18:25:15.367 INFO:teuthology.orchestra.run.smithi093.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-01-15T18:25:15.697 INFO:teuthology.orchestra.run.smithi027.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-01-15T18:25:16.002 INFO:teuthology.orchestra.run.smithi093.stderr: 93.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-01-15T18:25:16.004 INFO:teuthology.orchestra.run.smithi093.stderr: 2024-01-15T18:25:16.004 INFO:teuthology.orchestra.run.smithi093.stderr:real 0m3.572s 2024-01-15T18:25:16.004 INFO:teuthology.orchestra.run.smithi093.stderr:user 0m14.778s 2024-01-15T18:25:16.005 INFO:teuthology.orchestra.run.smithi093.stderr:sys 0m1.211s 2024-01-15T18:25:16.025 INFO:teuthology.orchestra.run.smithi027.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-01-15T18:25:16.179 INFO:teuthology.orchestra.run.smithi027.stderr: 96.5% -- replaced with /var/log/ceph/ceph-client.0.41334.log.gz 2024-01-15T18:25:16.435 INFO:teuthology.orchestra.run.smithi027.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-01-15T18:25:16.697 INFO:teuthology.orchestra.run.smithi027.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-01-15T18:25:16.757 INFO:teuthology.orchestra.run.smithi027.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-01-15T18:25:17.641 INFO:teuthology.orchestra.run.smithi027.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-01-15T18:25:17.750 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-01-15T18:25:17.751 INFO:teuthology.orchestra.run.smithi027.stderr:real 0m5.232s 2024-01-15T18:25:17.751 INFO:teuthology.orchestra.run.smithi027.stderr:user 0m27.204s 2024-01-15T18:25:17.751 INFO:teuthology.orchestra.run.smithi027.stderr:sys 0m3.326s 2024-01-15T18:25:17.751 INFO:tasks.ceph:Archiving logs... 2024-01-15T18:25:17.752 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-01-15_17:34:44-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517095/remote/smithi027/log 2024-01-15T18:25:17.753 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-15T18:25:20.535 DEBUG:teuthology.misc:Transferring archived files from smithi093:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-01-15_17:34:44-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517095/remote/smithi093/log 2024-01-15T18:25:20.536 DEBUG:teuthology.orchestra.run.smithi093:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-15T18:25:22.051 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-15T18:25:22.060 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-15T18:25:22.060 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-15T18:25:22.064 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-15T18:25:22.150 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-15T18:25:22.159 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-15T18:25:22.160 DEBUG:teuthology.orchestra.run.smithi027:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-15T18:25:22.163 DEBUG:teuthology.orchestra.run.smithi093:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-15T18:25:22.176 INFO:teuthology.orchestra.run.smithi027.stdout: remote refid st t when poll reach delay offset jitter 2024-01-15T18:25:22.176 INFO:teuthology.orchestra.run.smithi027.stdout:============================================================================== 2024-01-15T18:25:22.176 INFO:teuthology.orchestra.run.smithi027.stdout:*hv01.front.sepi 44.4.53.2 2 u 39 64 377 0.090 -11.422 2.047 2024-01-15T18:25:22.176 INFO:teuthology.orchestra.run.smithi027.stdout:+hv02.front.sepi 63.231.80.2 3 u 42 64 377 0.075 -8.863 2.755 2024-01-15T18:25:22.176 INFO:teuthology.orchestra.run.smithi027.stdout:+hv03.front.sepi 74.6.168.72 3 u 39 64 377 0.082 -7.617 2.346 2024-01-15T18:25:22.176 INFO:teuthology.orchestra.run.smithi027.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 +0.000 0.000 2024-01-15T18:25:22.178 INFO:teuthology.orchestra.run.smithi093.stdout: remote refid st t when poll reach delay offset jitter 2024-01-15T18:25:22.178 INFO:teuthology.orchestra.run.smithi093.stdout:============================================================================== 2024-01-15T18:25:22.179 INFO:teuthology.orchestra.run.smithi093.stdout:*hv01.front.sepi 44.4.53.2 2 u 24 64 377 0.103 -8.278 0.572 2024-01-15T18:25:22.179 INFO:teuthology.orchestra.run.smithi093.stdout:+hv02.front.sepi 63.231.80.2 3 u 44 64 377 0.098 -4.474 0.285 2024-01-15T18:25:22.179 INFO:teuthology.orchestra.run.smithi093.stdout:+hv03.front.sepi 74.6.168.72 3 u 41 64 377 0.089 -6.772 0.383 2024-01-15T18:25:22.179 INFO:teuthology.orchestra.run.smithi093.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 +0.000 0.000 2024-01-15T18:25:22.180 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-15T18:25:22.189 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-15T18:25:22.189 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-15T18:25:22.198 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-15T18:25:22.206 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-15T18:25:22.214 INFO:teuthology.task.internal:Duration was 1938.930742 seconds 2024-01-15T18:25:22.215 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-15T18:25:22.224 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-15T18:25:22.224 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-15T18:25:22.227 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-15T18:25:22.322 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-15T18:25:22.323 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi027.front.sepia.ceph.com 2024-01-15T18:25:22.323 DEBUG:teuthology.orchestra.run.smithi027:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-15T18:25:22.343 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi093.front.sepia.ceph.com 2024-01-15T18:25:22.343 DEBUG:teuthology.orchestra.run.smithi093:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-15T18:25:22.364 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-15T18:25:22.364 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-15T18:25:22.391 DEBUG:teuthology.orchestra.run.smithi093:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-15T18:25:22.443 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-15T18:25:22.443 DEBUG:teuthology.orchestra.run.smithi027:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-15T18:25:22.476 DEBUG:teuthology.orchestra.run.smithi093:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-15T18:25:22.563 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-15T18:25:22.574 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-15T18:25:22.574 DEBUG:teuthology.orchestra.run.smithi027:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-15T18:25:22.594 DEBUG:teuthology.orchestra.run.smithi093:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-15T18:25:22.614 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-15T18:25:22.624 DEBUG:teuthology.orchestra.run.smithi027:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-15T18:25:22.640 DEBUG:teuthology.orchestra.run.smithi093:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-15T18:25:22.654 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = core 2024-01-15T18:25:22.675 INFO:teuthology.orchestra.run.smithi093.stdout:kernel.core_pattern = core 2024-01-15T18:25:22.693 DEBUG:teuthology.orchestra.run.smithi027:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-15T18:25:22.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:25:22.728 DEBUG:teuthology.orchestra.run.smithi093:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-15T18:25:22.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:25:22.741 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-15T18:25:22.751 INFO:teuthology.task.internal:Transferring archived files... 2024-01-15T18:25:22.752 DEBUG:teuthology.misc:Transferring archived files from smithi027:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-01-15_17:34:44-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517095/remote/smithi027 2024-01-15T18:25:22.752 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-15T18:25:22.795 DEBUG:teuthology.misc:Transferring archived files from smithi093:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-01-15_17:34:44-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517095/remote/smithi093 2024-01-15T18:25:22.795 DEBUG:teuthology.orchestra.run.smithi093:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-15T18:25:22.821 INFO:teuthology.task.internal:Removing archive directory... 2024-01-15T18:25:22.821 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-15T18:25:22.835 DEBUG:teuthology.orchestra.run.smithi093:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-15T18:25:22.870 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-15T18:25:22.881 INFO:teuthology.task.internal:Not uploading archives. 2024-01-15T18:25:22.881 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-15T18:25:22.891 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-15T18:25:22.891 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-15T18:25:22.894 DEBUG:teuthology.orchestra.run.smithi093:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-15T18:25:22.899 INFO:teuthology.orchestra.run.smithi027.stdout: 658073 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 15 18:25 /home/ubuntu/cephtest 2024-01-15T18:25:22.922 INFO:teuthology.orchestra.run.smithi093.stdout: 658072 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 15 18:25 /home/ubuntu/cephtest 2024-01-15T18:25:22.924 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-15T18:25:22.934 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-15T18:25:22.956 INFO:teuthology.nuke:Checking targets against current locks 2024-01-15T18:25:22.985 DEBUG:teuthology.nuke:shortname: smithi027 2024-01-15T18:25:22.985 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-15T18:25:23.008 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-15_17:34:44-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517095', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-15 17:35:48.121859', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAI2qU5KCRsR9Mi+3w/gmSQ0CY1fK8zX3xasJHG5TMacJFxEA5yy5YXPW72BFKwHcZiTD1GhG0NIszB++jnF8Zg='} 2024-01-15T18:25:23.012 DEBUG:teuthology.nuke:shortname: smithi093 2024-01-15T18:25:23.012 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-15T18:25:23.036 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi093.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-15_17:34:44-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517095', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-15 17:35:48.124229', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDogDNdQ6bvThoHNs52dfNHRHe7uqvJnSoKggoh25Kct9v5aHtzSGiaGbpXRFTsuZc7w2kiZGssL/Ml4pOFOSnw='} 2024-01-15T18:25:23.065 INFO:teuthology.orchestra.console.smithi027:Power off 2024-01-15T18:25:23.066 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-15T18:25:23.080 INFO:teuthology.orchestra.console.smithi093:Power off 2024-01-15T18:25:23.080 DEBUG:teuthology.orchestra.console.smithi093:pexpect command: ipmitool -H smithi093.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-15T18:25:23.090 DEBUG:teuthology.orchestra.console.smithi027:power off output: Chassis Power Control: Down/Off 2024-01-15T18:25:23.090 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-15T18:25:23.102 DEBUG:teuthology.orchestra.console.smithi093:power off output: Chassis Power Control: Down/Off 2024-01-15T18:25:23.103 DEBUG:teuthology.orchestra.console.smithi093:pexpect command: ipmitool -H smithi093.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-15T18:25:23.111 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2024-01-15T18:25:23.124 DEBUG:teuthology.orchestra.console.smithi093:check power output: Chassis Power is on 2024-01-15T18:25:27.112 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-15T18:25:27.125 DEBUG:teuthology.orchestra.console.smithi093:pexpect command: ipmitool -H smithi093.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-15T18:25:27.224 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2024-01-15T18:25:27.234 DEBUG:teuthology.orchestra.console.smithi093:check power output: Chassis Power is on 2024-01-15T18:25:31.229 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-15T18:25:31.237 DEBUG:teuthology.orchestra.console.smithi093:pexpect command: ipmitool -H smithi093.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-15T18:25:31.339 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is off 2024-01-15T18:25:31.346 DEBUG:teuthology.orchestra.console.smithi093:check power output: Chassis Power is on 2024-01-15T18:25:31.440 INFO:teuthology.orchestra.console.smithi027:Power off completed 2024-01-15T18:25:35.349 DEBUG:teuthology.orchestra.console.smithi093:pexpect command: ipmitool -H smithi093.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-15T18:25:35.459 DEBUG:teuthology.orchestra.console.smithi093:check power output: Chassis Power is off 2024-01-15T18:25:35.560 INFO:teuthology.orchestra.console.smithi093:Power off completed 2024-01-15T18:25:35.666 INFO:teuthology.run:Summary data: description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} duration: 1938.9307417869568 failure_reason: 'Test failure: test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress), test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress)' flavor: default mon.a-kernel-sha1: 2668949e0971849893a3fb7f177ce9032fc65780 mon.b-kernel-sha1: 2668949e0971849893a3fb7f177ce9032fc65780 owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=593ff31fc8a6476abf4d2d6a4e016a4d status: fail success: false 2024-01-15T18:25:35.666 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-15T18:25:35.751 INFO:teuthology.run:FAIL